From 73f987a0133107cfdf9137a13fecd67a6fd6f205 Mon Sep 17 00:00:00 2001 From: mhjensen Date: Tue, 13 Aug 2019 23:10:37 +0200 Subject: [PATCH] update on getting started --- README.md~ | 122 + doc/.DS_Store | Bin 0 -> 10244 bytes doc/DataFiles/.DS_Store | Bin 0 -> 6148 bytes doc/DataFiles/Finance/.DS_Store | Bin 0 -> 6148 bytes doc/LectureNotes/.book.copyright | 1 + doc/LectureNotes/book.dlog | 231 + doc/LectureNotes/book.do.txt~ | 12249 ++++++++++++++++ doc/LectureNotes/src/Hudson_Bay.py~ | 43 + doc/LectureNotes/src/plot_Hudson.py~ | 19 + doc/Programs/.DS_Store | Bin 0 -> 6148 bytes doc/Programs/ANN/cnnkeras.py~ | 330 + doc/Programs/DimRed/covariance.py~ | 34 + doc/Programs/Finance/agents.py~ | 38 + doc/Programs/RandomWalks/OneDimParticle.py~ | 45 + .../ResamplingAnalysisScripts/README.md~ | 19 + doc/Programs/SVD/Fortran/simplefit.dat~ | 9 + doc/Programs/Sampling/.DS_Store | Bin 0 -> 6148 bytes doc/Projects/2018/Project1/pdf/.DS_Store | Bin 0 -> 6148 bytes .../2018/Project2/html/._Project2-bs000.html | 611 + doc/Projects/2018/Project2/pdf/Project2.tex~ | 557 + doc/Projects/2018/hw1/html/._hw1-bs000.html | 285 + doc/Projects/2018/hw2/html/._hw2-bs000.html | 211 + doc/Textbooks/.DS_Store | Bin 0 -> 6148 bytes .../html/._Autoencoders-bs002.html | 116 + .../html/reveal.js/plugin/leap/leap.js | 159 + .../html/reveal.js/plugin/remotes/remotes.js | 39 + doc/pub/Bayesian/html/._Bayesian-bs002.html | 218 + doc/pub/Bayesian/html/._Bayesian-bs003.html | 213 + doc/pub/Bayesian/html/._Bayesian-bs004.html | 210 + doc/pub/Bayesian/html/._Bayesian-bs005.html | 201 + doc/pub/Bayesian/html/._Bayesian-bs006.html | 205 + doc/pub/Bayesian/html/._Bayesian-bs007.html | 209 + doc/pub/Bayesian/html/._Bayesian-bs008.html | 208 + doc/pub/Bayesian/html/._Bayesian-bs009.html | 208 + doc/pub/Bayesian/html/._Bayesian-bs010.html | 233 + doc/pub/Bayesian/html/._Bayesian-bs011.html | 229 + doc/pub/Bayesian/html/._Bayesian-bs012.html | 221 + doc/pub/Bayesian/html/._Bayesian-bs013.html | 228 + doc/pub/Bayesian/html/._Bayesian-bs014.html | 238 + doc/pub/Bayesian/html/._Bayesian-bs015.html | 222 + doc/pub/Bayesian/html/._Bayesian-bs016.html | 216 + doc/pub/Bayesian/html/._Bayesian-bs017.html | 313 + doc/pub/Bayesian/html/._Bayesian-bs018.html | 208 + doc/pub/Bayesian/html/._Bayesian-bs019.html | 213 + doc/pub/Bayesian/html/._Bayesian-bs020.html | 470 + doc/pub/Bayesian/html/._Bayesian-bs021.html | 210 + doc/pub/Bayesian/html/._Bayesian-bs022.html | 206 + .../reveal.js/css/theme/template/mixins.scss | 29 + .../css/theme/template/settings.scss | 34 + .../reveal.js/css/theme/template/theme.scss | 171 + .../How2ReadData/html/How2ReadData-bs.html | 1852 +-- .../html/How2ReadData-reveal.html | 1830 +-- .../html/How2ReadData-solarized.html | 1799 ++- doc/pub/How2ReadData/html/How2ReadData.html | 1799 ++- doc/pub/How2ReadData/html/fig/pandas.jpg | Bin 0 -> 103233 bytes .../reveal.js/css/theme/template/mixins.scss | 29 + .../css/theme/template/settings.scss | 34 + .../reveal.js/css/theme/template/theme.scss | 171 + doc/pub/How2ReadData/html/reveal.js/demo.html | 410 + .../reveal.js/plugin/multiplex/package.json | 19 + .../html/reveal.js/test/simple.md | 12 + .../test/test-markdown-external.html | 36 + .../reveal.js/test/test-markdown-external.js | 24 + .../reveal.js/test/test-markdown-options.html | 41 + .../reveal.js/test/test-markdown-options.js | 26 + doc/pub/How2ReadData/html/src/Hudson_Bay.py~ | 43 + doc/pub/How2ReadData/html/src/plot_Hudson.py~ | 19 + doc/pub/How2ReadData/ipynb/.DS_Store | Bin 0 -> 6148 bytes doc/pub/How2ReadData/ipynb/How2ReadData.ipynb | 2525 ++-- doc/pub/How2ReadData/ipynb/fig/pandas.jpg | Bin 0 -> 103233 bytes .../ipynb/ipynb-How2ReadData-src.tar.gz | Bin 113365 -> 103580 bytes doc/pub/How2ReadData/ipynb/src/Hudson_Bay.py~ | 43 + .../How2ReadData/ipynb/src/plot_Hudson.py~ | 19 + .../How2ReadData/pdf/How2ReadData-minted.pdf | Bin 406775 -> 464361 bytes .../html/reveal.js/plugin/leap/leap.js | 159 + .../html/reveal.js/plugin/remotes/remotes.js | 39 + .../reveal.js/css/theme/template/mixins.scss | 29 + .../css/theme/template/settings.scss | 34 + .../reveal.js/css/theme/template/theme.scss | 171 + doc/pub/Linalg/html/reveal.js/demo.html | 410 + .../reveal.js/plugin/multiplex/package.json | 19 + doc/pub/Linalg/html/reveal.js/test/simple.md | 12 + .../test/test-markdown-external.html | 36 + .../reveal.js/test/test-markdown-external.js | 24 + .../reveal.js/test/test-markdown-options.html | 41 + .../reveal.js/test/test-markdown-options.js | 26 + .../Linalg-checkpoint.ipynb | 2029 +++ .../LogReg-checkpoint.ipynb | 502 + .../reveal.js/css/theme/template/mixins.scss | 29 + .../css/theme/template/settings.scss | 34 + .../reveal.js/css/theme/template/theme.scss | 171 + doc/pub/NeuralNet/html/reveal.js/demo.html | 410 + .../reveal.js/plugin/multiplex/package.json | 19 + .../NeuralNet/html/reveal.js/test/simple.md | 12 + .../test/test-markdown-external.html | 36 + .../reveal.js/test/test-markdown-external.js | 24 + .../reveal.js/test/test-markdown-options.html | 41 + .../reveal.js/test/test-markdown-options.js | 26 + .../NeuralNet-checkpoint.ipynb | 1163 ++ .../Untitled-checkpoint.ipynb | 6 + doc/pub/NeuralNet/ipynb/Untitled.ipynb | 48 + doc/pub/NeuralNet/ipynb/notebook.tex | 3332 +++++ doc/pub/Recurrent/html/._Recurrent-bs002.html | 146 + .../html/reveal.js/plugin/leap/leap.js | 159 + .../html/reveal.js/plugin/remotes/remotes.js | 39 + .../reveal.js/css/theme/template/mixins.scss | 29 + .../css/theme/template/settings.scss | 34 + .../reveal.js/css/theme/template/theme.scss | 171 + doc/pub/Regression/html/reveal.js/demo.html | 410 + .../reveal.js/plugin/multiplex/package.json | 19 + .../Regression/html/reveal.js/test/simple.md | 12 + .../test/test-markdown-external.html | 36 + .../reveal.js/test/test-markdown-external.js | 24 + .../reveal.js/test/test-markdown-options.html | 41 + .../reveal.js/test/test-markdown-options.js | 26 + .../Regression-checkpoint.ipynb | 939 ++ doc/pub/Reinforce/html/._Reinforce-bs002.html | 227 + .../html/reveal.js/plugin/leap/leap.js | 159 + .../html/reveal.js/plugin/remotes/remotes.js | 39 + .../reveal.js/css/theme/template/mixins.scss | 29 + .../css/theme/template/settings.scss | 34 + .../reveal.js/css/theme/template/theme.scss | 171 + doc/pub/Splines/html/reveal.js/demo.html | 410 + .../reveal.js/plugin/multiplex/package.json | 19 + doc/pub/Splines/html/reveal.js/test/simple.md | 12 + .../test/test-markdown-external.html | 36 + .../reveal.js/test/test-markdown-external.js | 24 + .../reveal.js/test/test-markdown-options.html | 41 + .../reveal.js/test/test-markdown-options.js | 26 + .../Splines-checkpoint.ipynb | 1341 ++ doc/src/.DS_Store | Bin 0 -> 6148 bytes doc/src/Bayesian/Bayesian.do.txt~ | 433 + doc/src/BoltzmannMachines/BM.do.txt~ | 1440 ++ .../BoltzmannMachines/figures/RMBenergy.dat~ | 100 + doc/src/BoltzmannMachines/figures/plot.py~ | 14 + .../figures/plotEnergies.py~ | 62 + doc/src/BoltzmannMachines/src/Hudson_Bay.py~ | 43 + doc/src/How2ReadData/How2ReadData.do.txt | 1577 +- doc/src/How2ReadData/fig/pandas.jpg | Bin 0 -> 103233 bytes doc/src/How2ReadData/src/Hudson_Bay.py~ | 43 + doc/src/How2ReadData/src/plot_Hudson.py~ | 19 + doc/src/Intro2Course/#Intro2Course.do.txt# | 167 + doc/src/Intro2Course/Intro2Course.do.txt~ | 171 + doc/src/Intro2Course/back.do.txt | 161 + doc/src/Linalg/Linalg.do.txt~ | 1121 ++ doc/src/NeuralNet/.DS_Store | Bin 0 -> 6148 bytes doc/src/NeuralNet/test.py~ | 335 + .../2018/Project1/.Project1.copyright | 1 + doc/src/Projects/2018/Project1/Project1.dlog | 2 + .../.Recurrent-bs_html_file_collection | 4 + .../.Recurrent-reveal_html_file_collection | 2 + .../.Recurrent-solarized_html_file_collection | 1 + doc/src/Recurrent/.Recurrent.copyright | 1 + .../Recurrent/.Recurrent_html_file_collection | 1 + doc/src/Recurrent/._Recurrent-bs000.html | 162 + doc/src/Recurrent/._Recurrent-bs001.html | 158 + doc/src/Recurrent/._Recurrent-bs002.html | 146 + doc/src/Recurrent/Recurrent.aux | 18 + doc/src/Recurrent/Recurrent.dlog | 12 + doc/src/Recurrent/Recurrent.do.txt~ | 38 + doc/src/Recurrent/Recurrent.idx | 0 doc/src/Recurrent/Recurrent.log | 819 ++ doc/src/Recurrent/Recurrent.out | 0 doc/src/Recurrent/Recurrent.tex.old~~ | 171 + .../reveal.js/css/theme/template/mixins.scss | 29 + .../css/theme/template/settings.scss | 34 + .../reveal.js/css/theme/template/theme.scss | 171 + .../.Regression-bs_html_file_collection | 101 + .../.Regression-reveal_html_file_collection | 2 + ....Regression-solarized_html_file_collection | 1 + doc/src/Regression/.Regression.copyright | 1 + .../.Regression_html_file_collection | 1 + doc/src/Regression/._Regression-bs000.html | 452 + doc/src/Regression/._Regression-bs001.html | 446 + doc/src/Regression/._Regression-bs002.html | 450 + doc/src/Regression/._Regression-bs003.html | 459 + doc/src/Regression/._Regression-bs004.html | 456 + doc/src/Regression/._Regression-bs005.html | 449 + doc/src/Regression/._Regression-bs006.html | 449 + doc/src/Regression/._Regression-bs007.html | 473 + doc/src/Regression/._Regression-bs008.html | 463 + doc/src/Regression/._Regression-bs009.html | 460 + doc/src/Regression/._Regression-bs010.html | 462 + doc/src/Regression/._Regression-bs011.html | 523 + doc/src/Regression/._Regression-bs012.html | 469 + doc/src/Regression/._Regression-bs013.html | 489 + doc/src/Regression/._Regression-bs014.html | 471 + doc/src/Regression/._Regression-bs015.html | 464 + doc/src/Regression/._Regression-bs016.html | 474 + doc/src/Regression/._Regression-bs017.html | 473 + doc/src/Regression/._Regression-bs018.html | 465 + doc/src/Regression/._Regression-bs019.html | 461 + doc/src/Regression/._Regression-bs020.html | 459 + doc/src/Regression/._Regression-bs021.html | 464 + doc/src/Regression/._Regression-bs022.html | 457 + doc/src/Regression/._Regression-bs023.html | 491 + doc/src/Regression/._Regression-bs024.html | 457 + doc/src/Regression/._Regression-bs025.html | 535 + doc/src/Regression/._Regression-bs026.html | 517 + doc/src/Regression/._Regression-bs027.html | 464 + doc/src/Regression/._Regression-bs028.html | 489 + doc/src/Regression/._Regression-bs029.html | 482 + doc/src/Regression/._Regression-bs030.html | 480 + doc/src/Regression/._Regression-bs031.html | 519 + doc/src/Regression/._Regression-bs032.html | 483 + doc/src/Regression/._Regression-bs033.html | 460 + doc/src/Regression/._Regression-bs034.html | 471 + doc/src/Regression/._Regression-bs035.html | 463 + doc/src/Regression/._Regression-bs036.html | 470 + doc/src/Regression/._Regression-bs037.html | 457 + doc/src/Regression/._Regression-bs038.html | 484 + doc/src/Regression/._Regression-bs039.html | 490 + doc/src/Regression/._Regression-bs040.html | 491 + doc/src/Regression/._Regression-bs041.html | 456 + doc/src/Regression/._Regression-bs042.html | 469 + doc/src/Regression/._Regression-bs043.html | 449 + doc/src/Regression/._Regression-bs044.html | 453 + doc/src/Regression/._Regression-bs045.html | 462 + doc/src/Regression/._Regression-bs046.html | 449 + doc/src/Regression/._Regression-bs047.html | 455 + doc/src/Regression/._Regression-bs048.html | 464 + doc/src/Regression/._Regression-bs049.html | 456 + doc/src/Regression/._Regression-bs050.html | 471 + doc/src/Regression/._Regression-bs051.html | 464 + doc/src/Regression/._Regression-bs052.html | 465 + doc/src/Regression/._Regression-bs053.html | 459 + doc/src/Regression/._Regression-bs054.html | 465 + doc/src/Regression/._Regression-bs055.html | 461 + doc/src/Regression/._Regression-bs056.html | 459 + doc/src/Regression/._Regression-bs057.html | 454 + doc/src/Regression/._Regression-bs058.html | 465 + doc/src/Regression/._Regression-bs059.html | 455 + doc/src/Regression/._Regression-bs060.html | 454 + doc/src/Regression/._Regression-bs061.html | 458 + doc/src/Regression/._Regression-bs062.html | 462 + doc/src/Regression/._Regression-bs063.html | 460 + doc/src/Regression/._Regression-bs064.html | 472 + doc/src/Regression/._Regression-bs065.html | 468 + doc/src/Regression/._Regression-bs066.html | 462 + doc/src/Regression/._Regression-bs067.html | 455 + doc/src/Regression/._Regression-bs068.html | 466 + doc/src/Regression/._Regression-bs069.html | 466 + doc/src/Regression/._Regression-bs070.html | 458 + doc/src/Regression/._Regression-bs071.html | 470 + doc/src/Regression/._Regression-bs072.html | 452 + doc/src/Regression/._Regression-bs073.html | 467 + doc/src/Regression/._Regression-bs074.html | 517 + doc/src/Regression/._Regression-bs075.html | 466 + doc/src/Regression/._Regression-bs076.html | 444 + doc/src/Regression/._Regression-bs077.html | 453 + doc/src/Regression/._Regression-bs078.html | 468 + doc/src/Regression/._Regression-bs079.html | 455 + doc/src/Regression/._Regression-bs080.html | 451 + doc/src/Regression/._Regression-bs081.html | 468 + doc/src/Regression/._Regression-bs082.html | 454 + doc/src/Regression/._Regression-bs083.html | 448 + doc/src/Regression/._Regression-bs084.html | 454 + doc/src/Regression/._Regression-bs085.html | 453 + doc/src/Regression/._Regression-bs086.html | 457 + doc/src/Regression/._Regression-bs087.html | 496 + doc/src/Regression/._Regression-bs088.html | 532 + doc/src/Regression/._Regression-bs089.html | 498 + doc/src/Regression/._Regression-bs090.html | 492 + doc/src/Regression/._Regression-bs091.html | 483 + doc/src/Regression/._Regression-bs092.html | 462 + doc/src/Regression/._Regression-bs093.html | 506 + doc/src/Regression/._Regression-bs094.html | 560 + doc/src/Regression/._Regression-bs095.html | 460 + doc/src/Regression/._Regression-bs096.html | 462 + doc/src/Regression/._Regression-bs097.html | 477 + doc/src/Regression/._Regression-bs098.html | 474 + doc/src/Regression/._Regression-bs099.html | 650 + doc/src/Regression/README.txt | 2 + doc/src/Regression/Regression-bs.html | 452 + doc/src/Regression/Regression-minted.pdf | Bin 0 -> 459187 bytes .../Regression/Regression-plain-minted.tex | 3673 +++++ doc/src/Regression/Regression-reveal.html | 4558 ++++++ doc/src/Regression/Regression-solarized.html | 4279 ++++++ doc/src/Regression/Regression.aux | 119 + doc/src/Regression/Regression.dlog | 146 + doc/src/Regression/Regression.html | 4284 ++++++ doc/src/Regression/Regression.idx | 0 doc/src/Regression/Regression.ipynb | 5838 ++++++++ doc/src/Regression/Regression.log | 2482 ++++ doc/src/Regression/Regression.out | 0 doc/src/Regression/Regression.p.tex | 3703 +++++ doc/src/Regression/Regression.tex | 3673 +++++ doc/src/Regression/Regression.tex.old~~ | 3673 +++++ .../_minted-Regression/default.pygstyle | 0 .../Regression/ipynb-Regression-src.tar.gz | Bin 0 -> 211 bytes doc/src/Regression/reveal.js/.gitignore | 8 + doc/src/Regression/reveal.js/.travis.yml | 5 + doc/src/Regression/reveal.js/CONTRIBUTING.md | 23 + doc/src/Regression/reveal.js/Gruntfile.js | 140 + doc/src/Regression/reveal.js/LICENSE | 19 + doc/src/Regression/reveal.js/README.md | 1052 ++ doc/src/Regression/reveal.js/bower.json | 27 + .../reveal.js/css/images/cbc_footer.png | Bin 0 -> 10008 bytes .../reveal.js/css/images/cbc_symbol.png | Bin 0 -> 2946 bytes .../reveal.js/css/images/simula_footer.png | Bin 0 -> 2513 bytes .../reveal.js/css/images/simula_logo.png | Bin 0 -> 2138 bytes .../reveal.js/css/images/simula_symbol.png | Bin 0 -> 2138 bytes .../reveal.js/css/images/uio_footer.png | Bin 0 -> 18189 bytes .../reveal.js/css/images/uio_symbol.png | Bin 0 -> 11352 bytes .../Regression/reveal.js/css/print/paper.css | 202 + .../Regression/reveal.js/css/print/pdf.css | 157 + doc/src/Regression/reveal.js/css/reveal.css | 1886 +++ doc/src/Regression/reveal.js/css/reveal.scss | 1319 ++ .../Regression/reveal.js/css/theme/README.md | 23 + .../Regression/reveal.js/css/theme/beige.css | 154 + .../reveal.js/css/theme/beigesmall.css | 155 + .../Regression/reveal.js/css/theme/black.css | 273 + .../Regression/reveal.js/css/theme/blood.css | 180 + .../Regression/reveal.js/css/theme/cbc.css | 144 + .../reveal.js/css/theme/darkgray.css | 153 + .../reveal.js/css/theme/default.css | 153 + .../Regression/reveal.js/css/theme/league.css | 279 + .../Regression/reveal.js/css/theme/moon.css | 153 + .../Regression/reveal.js/css/theme/night.css | 141 + .../Regression/reveal.js/css/theme/serif.css | 143 + .../Regression/reveal.js/css/theme/simple.css | 144 + .../Regression/reveal.js/css/theme/simula.css | 144 + .../Regression/reveal.js/css/theme/sky.css | 150 + .../reveal.js/css/theme/solarized.css | 153 + .../reveal.js/css/theme/source/beige.scss | 50 + .../css/theme/source/beigesmall.scss | 51 + .../reveal.js/css/theme/source/black.scss | 49 + .../reveal.js/css/theme/source/blood.scss | 91 + .../reveal.js/css/theme/source/cbc.scss | 39 + .../reveal.js/css/theme/source/darkgray.scss | 42 + .../reveal.js/css/theme/source/default.scss | 42 + .../reveal.js/css/theme/source/league.scss | 34 + .../reveal.js/css/theme/source/moon.scss | 68 + .../reveal.js/css/theme/source/night.scss | 35 + .../reveal.js/css/theme/source/serif.scss | 35 + .../reveal.js/css/theme/source/simple.scss | 38 + .../reveal.js/css/theme/source/simula.scss | 39 + .../reveal.js/css/theme/source/sky.scss | 46 + .../reveal.js/css/theme/source/solarized.scss | 74 + .../reveal.js/css/theme/source/white.scss | 49 + .../reveal.js/css/theme/template/mixins.scss | 29 + .../css/theme/template/settings.scss | 34 + .../reveal.js/css/theme/template/theme.scss | 171 + .../Regression/reveal.js/css/theme/white.css | 273 + doc/src/Regression/reveal.js/index.html | 411 + doc/src/Regression/reveal.js/js/reveal.js | 4508 ++++++ .../Regression/reveal.js/lib/css/zenburn.css | 117 + .../reveal.js/lib/font/league-gothic/LICENSE | 2 + .../lib/font/league-gothic/league-gothic.css | 10 + .../lib/font/league-gothic/league-gothic.eot | Bin 0 -> 25696 bytes .../lib/font/league-gothic/league-gothic.ttf | Bin 0 -> 64256 bytes .../lib/font/league-gothic/league-gothic.woff | Bin 0 -> 30764 bytes .../lib/font/source-sans-pro/LICENSE | 45 + .../source-sans-pro-italic.eot | Bin 0 -> 75720 bytes .../source-sans-pro-italic.ttf | Bin 0 -> 238084 bytes .../source-sans-pro-italic.woff | Bin 0 -> 98556 bytes .../source-sans-pro-regular.eot | Bin 0 -> 88070 bytes .../source-sans-pro-regular.ttf | Bin 0 -> 288008 bytes .../source-sans-pro-regular.woff | Bin 0 -> 114324 bytes .../source-sans-pro-semibold.eot | Bin 0 -> 89897 bytes .../source-sans-pro-semibold.ttf | Bin 0 -> 284640 bytes .../source-sans-pro-semibold.woff | Bin 0 -> 115648 bytes .../source-sans-pro-semibolditalic.eot | Bin 0 -> 75706 bytes .../source-sans-pro-semibolditalic.ttf | Bin 0 -> 240944 bytes .../source-sans-pro-semibolditalic.woff | Bin 0 -> 98816 bytes .../font/source-sans-pro/source-sans-pro.css | 39 + .../Regression/reveal.js/lib/js/classList.js | 2 + .../Regression/reveal.js/lib/js/head.min.js | 8 + .../Regression/reveal.js/lib/js/html5shiv.js | 7 + doc/src/Regression/reveal.js/package.json | 45 + .../reveal.js/plugin/highlight/highlight.js | 30 + .../Regression/reveal.js/plugin/leap/leap.js | 159 + .../reveal.js/plugin/markdown/example.html | 129 + .../reveal.js/plugin/markdown/example.md | 31 + .../reveal.js/plugin/markdown/markdown.js | 393 + .../reveal.js/plugin/markdown/marked.js | 6 + .../Regression/reveal.js/plugin/math/math.js | 64 + .../reveal.js/plugin/multiplex/client.js | 13 + .../reveal.js/plugin/multiplex/index.js | 56 + .../reveal.js/plugin/multiplex/master.js | 51 + .../reveal.js/plugin/notes-server/client.js | 60 + .../reveal.js/plugin/notes-server/index.js | 66 + .../reveal.js/plugin/notes-server/notes.html | 396 + .../reveal.js/plugin/notes/notes.html | 406 + .../reveal.js/plugin/notes/notes.js | 122 + .../reveal.js/plugin/print-pdf/print-pdf.js | 48 + .../reveal.js/plugin/remotes/remotes.js | 39 + .../reveal.js/plugin/search/search.js | 196 + .../reveal.js/plugin/zoom-js/zoom.js | 278 + .../reveal.js/test/examples/assets/image1.png | Bin 0 -> 21991 bytes .../reveal.js/test/examples/assets/image2.png | Bin 0 -> 10237 bytes .../reveal.js/test/examples/barebones.html | 41 + .../test/examples/embedded-media.html | 49 + .../reveal.js/test/examples/math.html | 185 + .../test/examples/slide-backgrounds.html | 144 + .../test/examples/slide-transitions.html | 101 + .../reveal.js/test/qunit-1.12.0.css | 244 + .../Regression/reveal.js/test/qunit-1.12.0.js | 2212 +++ .../test-markdown-element-attributes.html | 134 + .../test/test-markdown-element-attributes.js | 46 + .../test/test-markdown-slide-attributes.html | 128 + .../test/test-markdown-slide-attributes.js | 47 + .../reveal.js/test/test-markdown.html | 52 + .../reveal.js/test/test-markdown.js | 15 + .../Regression/reveal.js/test/test-pdf.html | 83 + doc/src/Regression/reveal.js/test/test-pdf.js | 15 + doc/src/Regression/reveal.js/test/test.html | 85 + doc/src/Regression/reveal.js/test/test.js | 589 + doc/src/Splines/Splines.do.txt~ | 1864 +++ doc/src/Statistics/Statistics.do.txt~ | 1867 +++ doc/src/SupportVMachines/svm.do.txt~ | 1120 ++ doc/web/.course_html_file_collection | 1 + doc/web/.schedule_html_file_collection | 1 + doc/web/course.dlog | 108 + doc/web/course.do.txt~ | 277 + doc/web/schedule.dlog | 6 + doc/web/schedule.do.txt~ | 58 + doc/web/tmp_mako__course.do.txt | 496 + doc/web/tmp_preprocess__course.do.txt | 271 + doc/web/tmp_preprocess__schedule.do.txt | 52 + 420 files changed, 157728 insertions(+), 5042 deletions(-) create mode 100644 README.md~ create mode 100644 doc/.DS_Store create mode 100644 doc/DataFiles/.DS_Store create mode 100644 doc/DataFiles/Finance/.DS_Store create mode 100644 doc/LectureNotes/.book.copyright create mode 100644 doc/LectureNotes/book.dlog create mode 100644 doc/LectureNotes/book.do.txt~ create mode 100644 doc/LectureNotes/src/Hudson_Bay.py~ create mode 100644 doc/LectureNotes/src/plot_Hudson.py~ create mode 100644 doc/Programs/.DS_Store create mode 100644 doc/Programs/ANN/cnnkeras.py~ create mode 100644 doc/Programs/DimRed/covariance.py~ create mode 100644 doc/Programs/Finance/agents.py~ create mode 100644 doc/Programs/RandomWalks/OneDimParticle.py~ create mode 100644 doc/Programs/ResamplingAnalysisScripts/README.md~ create mode 100755 doc/Programs/SVD/Fortran/simplefit.dat~ create mode 100644 doc/Programs/Sampling/.DS_Store create mode 100644 doc/Projects/2018/Project1/pdf/.DS_Store create mode 100644 doc/Projects/2018/Project2/html/._Project2-bs000.html create mode 100644 doc/Projects/2018/Project2/pdf/Project2.tex~ create mode 100644 doc/Projects/2018/hw1/html/._hw1-bs000.html create mode 100644 doc/Projects/2018/hw2/html/._hw2-bs000.html create mode 100644 doc/Textbooks/.DS_Store create mode 100644 doc/pub/Autoencoders/html/._Autoencoders-bs002.html create mode 100644 doc/pub/Autoencoders/html/reveal.js/plugin/leap/leap.js create mode 100644 doc/pub/Autoencoders/html/reveal.js/plugin/remotes/remotes.js create mode 100644 doc/pub/Bayesian/html/._Bayesian-bs002.html create mode 100644 doc/pub/Bayesian/html/._Bayesian-bs003.html create mode 100644 doc/pub/Bayesian/html/._Bayesian-bs004.html create mode 100644 doc/pub/Bayesian/html/._Bayesian-bs005.html create mode 100644 doc/pub/Bayesian/html/._Bayesian-bs006.html create mode 100644 doc/pub/Bayesian/html/._Bayesian-bs007.html create mode 100644 doc/pub/Bayesian/html/._Bayesian-bs008.html create mode 100644 doc/pub/Bayesian/html/._Bayesian-bs009.html create mode 100644 doc/pub/Bayesian/html/._Bayesian-bs010.html create mode 100644 doc/pub/Bayesian/html/._Bayesian-bs011.html create mode 100644 doc/pub/Bayesian/html/._Bayesian-bs012.html create mode 100644 doc/pub/Bayesian/html/._Bayesian-bs013.html create mode 100644 doc/pub/Bayesian/html/._Bayesian-bs014.html create mode 100644 doc/pub/Bayesian/html/._Bayesian-bs015.html create mode 100644 doc/pub/Bayesian/html/._Bayesian-bs016.html create mode 100644 doc/pub/Bayesian/html/._Bayesian-bs017.html create mode 100644 doc/pub/Bayesian/html/._Bayesian-bs018.html create mode 100644 doc/pub/Bayesian/html/._Bayesian-bs019.html create mode 100644 doc/pub/Bayesian/html/._Bayesian-bs020.html create mode 100644 doc/pub/Bayesian/html/._Bayesian-bs021.html create mode 100644 doc/pub/Bayesian/html/._Bayesian-bs022.html create mode 100644 doc/pub/Bayesian/html/reveal.js/css/theme/template/mixins.scss create mode 100644 doc/pub/Bayesian/html/reveal.js/css/theme/template/settings.scss create mode 100644 doc/pub/Bayesian/html/reveal.js/css/theme/template/theme.scss create mode 100644 doc/pub/How2ReadData/html/fig/pandas.jpg create mode 100644 doc/pub/How2ReadData/html/reveal.js/css/theme/template/mixins.scss create mode 100644 doc/pub/How2ReadData/html/reveal.js/css/theme/template/settings.scss create mode 100644 doc/pub/How2ReadData/html/reveal.js/css/theme/template/theme.scss create mode 100644 doc/pub/How2ReadData/html/reveal.js/demo.html create mode 100644 doc/pub/How2ReadData/html/reveal.js/plugin/multiplex/package.json create mode 100644 doc/pub/How2ReadData/html/reveal.js/test/simple.md create mode 100644 doc/pub/How2ReadData/html/reveal.js/test/test-markdown-external.html create mode 100644 doc/pub/How2ReadData/html/reveal.js/test/test-markdown-external.js create mode 100644 doc/pub/How2ReadData/html/reveal.js/test/test-markdown-options.html create mode 100644 doc/pub/How2ReadData/html/reveal.js/test/test-markdown-options.js create mode 100644 doc/pub/How2ReadData/html/src/Hudson_Bay.py~ create mode 100644 doc/pub/How2ReadData/html/src/plot_Hudson.py~ create mode 100644 doc/pub/How2ReadData/ipynb/.DS_Store create mode 100644 doc/pub/How2ReadData/ipynb/fig/pandas.jpg create mode 100644 doc/pub/How2ReadData/ipynb/src/Hudson_Bay.py~ create mode 100644 doc/pub/How2ReadData/ipynb/src/plot_Hudson.py~ create mode 100644 doc/pub/Intro2Course/html/reveal.js/plugin/leap/leap.js create mode 100644 doc/pub/Intro2Course/html/reveal.js/plugin/remotes/remotes.js create mode 100644 doc/pub/Linalg/html/reveal.js/css/theme/template/mixins.scss create mode 100644 doc/pub/Linalg/html/reveal.js/css/theme/template/settings.scss create mode 100644 doc/pub/Linalg/html/reveal.js/css/theme/template/theme.scss create mode 100644 doc/pub/Linalg/html/reveal.js/demo.html create mode 100644 doc/pub/Linalg/html/reveal.js/plugin/multiplex/package.json create mode 100644 doc/pub/Linalg/html/reveal.js/test/simple.md create mode 100644 doc/pub/Linalg/html/reveal.js/test/test-markdown-external.html create mode 100644 doc/pub/Linalg/html/reveal.js/test/test-markdown-external.js create mode 100644 doc/pub/Linalg/html/reveal.js/test/test-markdown-options.html create mode 100644 doc/pub/Linalg/html/reveal.js/test/test-markdown-options.js create mode 100644 doc/pub/Linalg/ipynb/.ipynb_checkpoints/Linalg-checkpoint.ipynb create mode 100644 doc/pub/LogReg/ipynb/.ipynb_checkpoints/LogReg-checkpoint.ipynb create mode 100644 doc/pub/NeuralNet/html/reveal.js/css/theme/template/mixins.scss create mode 100644 doc/pub/NeuralNet/html/reveal.js/css/theme/template/settings.scss create mode 100644 doc/pub/NeuralNet/html/reveal.js/css/theme/template/theme.scss create mode 100644 doc/pub/NeuralNet/html/reveal.js/demo.html create mode 100644 doc/pub/NeuralNet/html/reveal.js/plugin/multiplex/package.json create mode 100644 doc/pub/NeuralNet/html/reveal.js/test/simple.md create mode 100644 doc/pub/NeuralNet/html/reveal.js/test/test-markdown-external.html create mode 100644 doc/pub/NeuralNet/html/reveal.js/test/test-markdown-external.js create mode 100644 doc/pub/NeuralNet/html/reveal.js/test/test-markdown-options.html create mode 100644 doc/pub/NeuralNet/html/reveal.js/test/test-markdown-options.js create mode 100644 doc/pub/NeuralNet/ipynb/.ipynb_checkpoints/NeuralNet-checkpoint.ipynb create mode 100644 doc/pub/NeuralNet/ipynb/.ipynb_checkpoints/Untitled-checkpoint.ipynb create mode 100644 doc/pub/NeuralNet/ipynb/Untitled.ipynb create mode 100644 doc/pub/NeuralNet/ipynb/notebook.tex create mode 100644 doc/pub/Recurrent/html/._Recurrent-bs002.html create mode 100644 doc/pub/Recurrent/html/reveal.js/plugin/leap/leap.js create mode 100644 doc/pub/Recurrent/html/reveal.js/plugin/remotes/remotes.js create mode 100644 doc/pub/Regression/html/reveal.js/css/theme/template/mixins.scss create mode 100644 doc/pub/Regression/html/reveal.js/css/theme/template/settings.scss create mode 100644 doc/pub/Regression/html/reveal.js/css/theme/template/theme.scss create mode 100644 doc/pub/Regression/html/reveal.js/demo.html create mode 100644 doc/pub/Regression/html/reveal.js/plugin/multiplex/package.json create mode 100644 doc/pub/Regression/html/reveal.js/test/simple.md create mode 100644 doc/pub/Regression/html/reveal.js/test/test-markdown-external.html create mode 100644 doc/pub/Regression/html/reveal.js/test/test-markdown-external.js create mode 100644 doc/pub/Regression/html/reveal.js/test/test-markdown-options.html create mode 100644 doc/pub/Regression/html/reveal.js/test/test-markdown-options.js create mode 100644 doc/pub/Regression/ipynb/.ipynb_checkpoints/Regression-checkpoint.ipynb create mode 100644 doc/pub/Reinforce/html/._Reinforce-bs002.html create mode 100644 doc/pub/Reinforce/html/reveal.js/plugin/leap/leap.js create mode 100644 doc/pub/Reinforce/html/reveal.js/plugin/remotes/remotes.js create mode 100644 doc/pub/Splines/html/reveal.js/css/theme/template/mixins.scss create mode 100644 doc/pub/Splines/html/reveal.js/css/theme/template/settings.scss create mode 100644 doc/pub/Splines/html/reveal.js/css/theme/template/theme.scss create mode 100644 doc/pub/Splines/html/reveal.js/demo.html create mode 100644 doc/pub/Splines/html/reveal.js/plugin/multiplex/package.json create mode 100644 doc/pub/Splines/html/reveal.js/test/simple.md create mode 100644 doc/pub/Splines/html/reveal.js/test/test-markdown-external.html create mode 100644 doc/pub/Splines/html/reveal.js/test/test-markdown-external.js create mode 100644 doc/pub/Splines/html/reveal.js/test/test-markdown-options.html create mode 100644 doc/pub/Splines/html/reveal.js/test/test-markdown-options.js create mode 100644 doc/pub/Splines/ipynb/.ipynb_checkpoints/Splines-checkpoint.ipynb create mode 100644 doc/src/.DS_Store create mode 100644 doc/src/Bayesian/Bayesian.do.txt~ create mode 100644 doc/src/BoltzmannMachines/BM.do.txt~ create mode 100644 doc/src/BoltzmannMachines/figures/RMBenergy.dat~ create mode 100644 doc/src/BoltzmannMachines/figures/plot.py~ create mode 100644 doc/src/BoltzmannMachines/figures/plotEnergies.py~ create mode 100644 doc/src/BoltzmannMachines/src/Hudson_Bay.py~ create mode 100644 doc/src/How2ReadData/fig/pandas.jpg create mode 100644 doc/src/How2ReadData/src/Hudson_Bay.py~ create mode 100644 doc/src/How2ReadData/src/plot_Hudson.py~ create mode 100644 doc/src/Intro2Course/#Intro2Course.do.txt# create mode 100644 doc/src/Intro2Course/Intro2Course.do.txt~ create mode 100644 doc/src/Intro2Course/back.do.txt create mode 100644 doc/src/Linalg/Linalg.do.txt~ create mode 100644 doc/src/NeuralNet/.DS_Store create mode 100644 doc/src/NeuralNet/test.py~ create mode 100644 doc/src/Projects/2018/Project1/.Project1.copyright create mode 100644 doc/src/Projects/2018/Project1/Project1.dlog create mode 100644 doc/src/Recurrent/.Recurrent-bs_html_file_collection create mode 100644 doc/src/Recurrent/.Recurrent-reveal_html_file_collection create mode 100644 doc/src/Recurrent/.Recurrent-solarized_html_file_collection create mode 100644 doc/src/Recurrent/.Recurrent.copyright create mode 100644 doc/src/Recurrent/.Recurrent_html_file_collection create mode 100644 doc/src/Recurrent/._Recurrent-bs000.html create mode 100644 doc/src/Recurrent/._Recurrent-bs001.html create mode 100644 doc/src/Recurrent/._Recurrent-bs002.html create mode 100644 doc/src/Recurrent/Recurrent.aux create mode 100644 doc/src/Recurrent/Recurrent.dlog create mode 100644 doc/src/Recurrent/Recurrent.do.txt~ create mode 100644 doc/src/Recurrent/Recurrent.idx create mode 100644 doc/src/Recurrent/Recurrent.log create mode 100644 doc/src/Recurrent/Recurrent.out create mode 100644 doc/src/Recurrent/Recurrent.tex.old~~ create mode 100644 doc/src/Recurrent/reveal.js/css/theme/template/mixins.scss create mode 100644 doc/src/Recurrent/reveal.js/css/theme/template/settings.scss create mode 100644 doc/src/Recurrent/reveal.js/css/theme/template/theme.scss create mode 100644 doc/src/Regression/.Regression-bs_html_file_collection create mode 100644 doc/src/Regression/.Regression-reveal_html_file_collection create mode 100644 doc/src/Regression/.Regression-solarized_html_file_collection create mode 100644 doc/src/Regression/.Regression.copyright create mode 100644 doc/src/Regression/.Regression_html_file_collection create mode 100644 doc/src/Regression/._Regression-bs000.html create mode 100644 doc/src/Regression/._Regression-bs001.html create mode 100644 doc/src/Regression/._Regression-bs002.html create mode 100644 doc/src/Regression/._Regression-bs003.html create mode 100644 doc/src/Regression/._Regression-bs004.html create mode 100644 doc/src/Regression/._Regression-bs005.html create mode 100644 doc/src/Regression/._Regression-bs006.html create mode 100644 doc/src/Regression/._Regression-bs007.html create mode 100644 doc/src/Regression/._Regression-bs008.html create mode 100644 doc/src/Regression/._Regression-bs009.html create mode 100644 doc/src/Regression/._Regression-bs010.html create mode 100644 doc/src/Regression/._Regression-bs011.html create mode 100644 doc/src/Regression/._Regression-bs012.html create mode 100644 doc/src/Regression/._Regression-bs013.html create mode 100644 doc/src/Regression/._Regression-bs014.html create mode 100644 doc/src/Regression/._Regression-bs015.html create mode 100644 doc/src/Regression/._Regression-bs016.html create mode 100644 doc/src/Regression/._Regression-bs017.html create mode 100644 doc/src/Regression/._Regression-bs018.html create mode 100644 doc/src/Regression/._Regression-bs019.html create mode 100644 doc/src/Regression/._Regression-bs020.html create mode 100644 doc/src/Regression/._Regression-bs021.html create mode 100644 doc/src/Regression/._Regression-bs022.html create mode 100644 doc/src/Regression/._Regression-bs023.html create mode 100644 doc/src/Regression/._Regression-bs024.html create mode 100644 doc/src/Regression/._Regression-bs025.html create mode 100644 doc/src/Regression/._Regression-bs026.html create mode 100644 doc/src/Regression/._Regression-bs027.html create mode 100644 doc/src/Regression/._Regression-bs028.html create mode 100644 doc/src/Regression/._Regression-bs029.html create mode 100644 doc/src/Regression/._Regression-bs030.html create mode 100644 doc/src/Regression/._Regression-bs031.html create mode 100644 doc/src/Regression/._Regression-bs032.html create mode 100644 doc/src/Regression/._Regression-bs033.html create mode 100644 doc/src/Regression/._Regression-bs034.html create mode 100644 doc/src/Regression/._Regression-bs035.html create mode 100644 doc/src/Regression/._Regression-bs036.html create mode 100644 doc/src/Regression/._Regression-bs037.html create mode 100644 doc/src/Regression/._Regression-bs038.html create mode 100644 doc/src/Regression/._Regression-bs039.html create mode 100644 doc/src/Regression/._Regression-bs040.html create mode 100644 doc/src/Regression/._Regression-bs041.html create mode 100644 doc/src/Regression/._Regression-bs042.html create mode 100644 doc/src/Regression/._Regression-bs043.html create mode 100644 doc/src/Regression/._Regression-bs044.html create mode 100644 doc/src/Regression/._Regression-bs045.html create mode 100644 doc/src/Regression/._Regression-bs046.html create mode 100644 doc/src/Regression/._Regression-bs047.html create mode 100644 doc/src/Regression/._Regression-bs048.html create mode 100644 doc/src/Regression/._Regression-bs049.html create mode 100644 doc/src/Regression/._Regression-bs050.html create mode 100644 doc/src/Regression/._Regression-bs051.html create mode 100644 doc/src/Regression/._Regression-bs052.html create mode 100644 doc/src/Regression/._Regression-bs053.html create mode 100644 doc/src/Regression/._Regression-bs054.html create mode 100644 doc/src/Regression/._Regression-bs055.html create mode 100644 doc/src/Regression/._Regression-bs056.html create mode 100644 doc/src/Regression/._Regression-bs057.html create mode 100644 doc/src/Regression/._Regression-bs058.html create mode 100644 doc/src/Regression/._Regression-bs059.html create mode 100644 doc/src/Regression/._Regression-bs060.html create mode 100644 doc/src/Regression/._Regression-bs061.html create mode 100644 doc/src/Regression/._Regression-bs062.html create mode 100644 doc/src/Regression/._Regression-bs063.html create mode 100644 doc/src/Regression/._Regression-bs064.html create mode 100644 doc/src/Regression/._Regression-bs065.html create mode 100644 doc/src/Regression/._Regression-bs066.html create mode 100644 doc/src/Regression/._Regression-bs067.html create mode 100644 doc/src/Regression/._Regression-bs068.html create mode 100644 doc/src/Regression/._Regression-bs069.html create mode 100644 doc/src/Regression/._Regression-bs070.html create mode 100644 doc/src/Regression/._Regression-bs071.html create mode 100644 doc/src/Regression/._Regression-bs072.html create mode 100644 doc/src/Regression/._Regression-bs073.html create mode 100644 doc/src/Regression/._Regression-bs074.html create mode 100644 doc/src/Regression/._Regression-bs075.html create mode 100644 doc/src/Regression/._Regression-bs076.html create mode 100644 doc/src/Regression/._Regression-bs077.html create mode 100644 doc/src/Regression/._Regression-bs078.html create mode 100644 doc/src/Regression/._Regression-bs079.html create mode 100644 doc/src/Regression/._Regression-bs080.html create mode 100644 doc/src/Regression/._Regression-bs081.html create mode 100644 doc/src/Regression/._Regression-bs082.html create mode 100644 doc/src/Regression/._Regression-bs083.html create mode 100644 doc/src/Regression/._Regression-bs084.html create mode 100644 doc/src/Regression/._Regression-bs085.html create mode 100644 doc/src/Regression/._Regression-bs086.html create mode 100644 doc/src/Regression/._Regression-bs087.html create mode 100644 doc/src/Regression/._Regression-bs088.html create mode 100644 doc/src/Regression/._Regression-bs089.html create mode 100644 doc/src/Regression/._Regression-bs090.html create mode 100644 doc/src/Regression/._Regression-bs091.html create mode 100644 doc/src/Regression/._Regression-bs092.html create mode 100644 doc/src/Regression/._Regression-bs093.html create mode 100644 doc/src/Regression/._Regression-bs094.html create mode 100644 doc/src/Regression/._Regression-bs095.html create mode 100644 doc/src/Regression/._Regression-bs096.html create mode 100644 doc/src/Regression/._Regression-bs097.html create mode 100644 doc/src/Regression/._Regression-bs098.html create mode 100644 doc/src/Regression/._Regression-bs099.html create mode 100644 doc/src/Regression/README.txt create mode 100644 doc/src/Regression/Regression-bs.html create mode 100644 doc/src/Regression/Regression-minted.pdf create mode 100644 doc/src/Regression/Regression-plain-minted.tex create mode 100644 doc/src/Regression/Regression-reveal.html create mode 100644 doc/src/Regression/Regression-solarized.html create mode 100644 doc/src/Regression/Regression.aux create mode 100644 doc/src/Regression/Regression.dlog create mode 100644 doc/src/Regression/Regression.html create mode 100644 doc/src/Regression/Regression.idx create mode 100644 doc/src/Regression/Regression.ipynb create mode 100644 doc/src/Regression/Regression.log create mode 100644 doc/src/Regression/Regression.out create mode 100644 doc/src/Regression/Regression.p.tex create mode 100644 doc/src/Regression/Regression.tex create mode 100644 doc/src/Regression/Regression.tex.old~~ create mode 100644 doc/src/Regression/_minted-Regression/default.pygstyle create mode 100644 doc/src/Regression/ipynb-Regression-src.tar.gz create mode 100644 doc/src/Regression/reveal.js/.gitignore create mode 100644 doc/src/Regression/reveal.js/.travis.yml create mode 100644 doc/src/Regression/reveal.js/CONTRIBUTING.md create mode 100644 doc/src/Regression/reveal.js/Gruntfile.js create mode 100644 doc/src/Regression/reveal.js/LICENSE create mode 100644 doc/src/Regression/reveal.js/README.md create mode 100644 doc/src/Regression/reveal.js/bower.json create mode 100644 doc/src/Regression/reveal.js/css/images/cbc_footer.png create mode 100644 doc/src/Regression/reveal.js/css/images/cbc_symbol.png create mode 100644 doc/src/Regression/reveal.js/css/images/simula_footer.png create mode 100644 doc/src/Regression/reveal.js/css/images/simula_logo.png create mode 100644 doc/src/Regression/reveal.js/css/images/simula_symbol.png create mode 100644 doc/src/Regression/reveal.js/css/images/uio_footer.png create mode 100644 doc/src/Regression/reveal.js/css/images/uio_symbol.png create mode 100644 doc/src/Regression/reveal.js/css/print/paper.css create mode 100644 doc/src/Regression/reveal.js/css/print/pdf.css create mode 100644 doc/src/Regression/reveal.js/css/reveal.css create mode 100644 doc/src/Regression/reveal.js/css/reveal.scss create mode 100644 doc/src/Regression/reveal.js/css/theme/README.md create mode 100644 doc/src/Regression/reveal.js/css/theme/beige.css create mode 100644 doc/src/Regression/reveal.js/css/theme/beigesmall.css create mode 100644 doc/src/Regression/reveal.js/css/theme/black.css create mode 100644 doc/src/Regression/reveal.js/css/theme/blood.css create mode 100644 doc/src/Regression/reveal.js/css/theme/cbc.css create mode 100644 doc/src/Regression/reveal.js/css/theme/darkgray.css create mode 100644 doc/src/Regression/reveal.js/css/theme/default.css create mode 100644 doc/src/Regression/reveal.js/css/theme/league.css create mode 100644 doc/src/Regression/reveal.js/css/theme/moon.css create mode 100644 doc/src/Regression/reveal.js/css/theme/night.css create mode 100644 doc/src/Regression/reveal.js/css/theme/serif.css create mode 100644 doc/src/Regression/reveal.js/css/theme/simple.css create mode 100644 doc/src/Regression/reveal.js/css/theme/simula.css create mode 100644 doc/src/Regression/reveal.js/css/theme/sky.css create mode 100644 doc/src/Regression/reveal.js/css/theme/solarized.css create mode 100644 doc/src/Regression/reveal.js/css/theme/source/beige.scss create mode 100644 doc/src/Regression/reveal.js/css/theme/source/beigesmall.scss create mode 100644 doc/src/Regression/reveal.js/css/theme/source/black.scss create mode 100644 doc/src/Regression/reveal.js/css/theme/source/blood.scss create mode 100644 doc/src/Regression/reveal.js/css/theme/source/cbc.scss create mode 100644 doc/src/Regression/reveal.js/css/theme/source/darkgray.scss create mode 100644 doc/src/Regression/reveal.js/css/theme/source/default.scss create mode 100644 doc/src/Regression/reveal.js/css/theme/source/league.scss create mode 100644 doc/src/Regression/reveal.js/css/theme/source/moon.scss create mode 100644 doc/src/Regression/reveal.js/css/theme/source/night.scss create mode 100644 doc/src/Regression/reveal.js/css/theme/source/serif.scss create mode 100644 doc/src/Regression/reveal.js/css/theme/source/simple.scss create mode 100644 doc/src/Regression/reveal.js/css/theme/source/simula.scss create mode 100644 doc/src/Regression/reveal.js/css/theme/source/sky.scss create mode 100644 doc/src/Regression/reveal.js/css/theme/source/solarized.scss create mode 100644 doc/src/Regression/reveal.js/css/theme/source/white.scss create mode 100644 doc/src/Regression/reveal.js/css/theme/template/mixins.scss create mode 100644 doc/src/Regression/reveal.js/css/theme/template/settings.scss create mode 100644 doc/src/Regression/reveal.js/css/theme/template/theme.scss create mode 100644 doc/src/Regression/reveal.js/css/theme/white.css create mode 100644 doc/src/Regression/reveal.js/index.html create mode 100644 doc/src/Regression/reveal.js/js/reveal.js create mode 100644 doc/src/Regression/reveal.js/lib/css/zenburn.css create mode 100644 doc/src/Regression/reveal.js/lib/font/league-gothic/LICENSE create mode 100644 doc/src/Regression/reveal.js/lib/font/league-gothic/league-gothic.css create mode 100644 doc/src/Regression/reveal.js/lib/font/league-gothic/league-gothic.eot create mode 100644 doc/src/Regression/reveal.js/lib/font/league-gothic/league-gothic.ttf create mode 100644 doc/src/Regression/reveal.js/lib/font/league-gothic/league-gothic.woff create mode 100644 doc/src/Regression/reveal.js/lib/font/source-sans-pro/LICENSE create mode 100644 doc/src/Regression/reveal.js/lib/font/source-sans-pro/source-sans-pro-italic.eot create mode 100644 doc/src/Regression/reveal.js/lib/font/source-sans-pro/source-sans-pro-italic.ttf create mode 100644 doc/src/Regression/reveal.js/lib/font/source-sans-pro/source-sans-pro-italic.woff create mode 100644 doc/src/Regression/reveal.js/lib/font/source-sans-pro/source-sans-pro-regular.eot create mode 100644 doc/src/Regression/reveal.js/lib/font/source-sans-pro/source-sans-pro-regular.ttf create mode 100644 doc/src/Regression/reveal.js/lib/font/source-sans-pro/source-sans-pro-regular.woff create mode 100644 doc/src/Regression/reveal.js/lib/font/source-sans-pro/source-sans-pro-semibold.eot create mode 100644 doc/src/Regression/reveal.js/lib/font/source-sans-pro/source-sans-pro-semibold.ttf create mode 100644 doc/src/Regression/reveal.js/lib/font/source-sans-pro/source-sans-pro-semibold.woff create mode 100644 doc/src/Regression/reveal.js/lib/font/source-sans-pro/source-sans-pro-semibolditalic.eot create mode 100644 doc/src/Regression/reveal.js/lib/font/source-sans-pro/source-sans-pro-semibolditalic.ttf create mode 100644 doc/src/Regression/reveal.js/lib/font/source-sans-pro/source-sans-pro-semibolditalic.woff create mode 100644 doc/src/Regression/reveal.js/lib/font/source-sans-pro/source-sans-pro.css create mode 100644 doc/src/Regression/reveal.js/lib/js/classList.js create mode 100644 doc/src/Regression/reveal.js/lib/js/head.min.js create mode 100644 doc/src/Regression/reveal.js/lib/js/html5shiv.js create mode 100644 doc/src/Regression/reveal.js/package.json create mode 100644 doc/src/Regression/reveal.js/plugin/highlight/highlight.js create mode 100644 doc/src/Regression/reveal.js/plugin/leap/leap.js create mode 100644 doc/src/Regression/reveal.js/plugin/markdown/example.html create mode 100644 doc/src/Regression/reveal.js/plugin/markdown/example.md create mode 100644 doc/src/Regression/reveal.js/plugin/markdown/markdown.js create mode 100644 doc/src/Regression/reveal.js/plugin/markdown/marked.js create mode 100644 doc/src/Regression/reveal.js/plugin/math/math.js create mode 100644 doc/src/Regression/reveal.js/plugin/multiplex/client.js create mode 100644 doc/src/Regression/reveal.js/plugin/multiplex/index.js create mode 100644 doc/src/Regression/reveal.js/plugin/multiplex/master.js create mode 100644 doc/src/Regression/reveal.js/plugin/notes-server/client.js create mode 100644 doc/src/Regression/reveal.js/plugin/notes-server/index.js create mode 100644 doc/src/Regression/reveal.js/plugin/notes-server/notes.html create mode 100644 doc/src/Regression/reveal.js/plugin/notes/notes.html create mode 100644 doc/src/Regression/reveal.js/plugin/notes/notes.js create mode 100644 doc/src/Regression/reveal.js/plugin/print-pdf/print-pdf.js create mode 100644 doc/src/Regression/reveal.js/plugin/remotes/remotes.js create mode 100644 doc/src/Regression/reveal.js/plugin/search/search.js create mode 100644 doc/src/Regression/reveal.js/plugin/zoom-js/zoom.js create mode 100644 doc/src/Regression/reveal.js/test/examples/assets/image1.png create mode 100644 doc/src/Regression/reveal.js/test/examples/assets/image2.png create mode 100644 doc/src/Regression/reveal.js/test/examples/barebones.html create mode 100644 doc/src/Regression/reveal.js/test/examples/embedded-media.html create mode 100644 doc/src/Regression/reveal.js/test/examples/math.html create mode 100644 doc/src/Regression/reveal.js/test/examples/slide-backgrounds.html create mode 100644 doc/src/Regression/reveal.js/test/examples/slide-transitions.html create mode 100644 doc/src/Regression/reveal.js/test/qunit-1.12.0.css create mode 100644 doc/src/Regression/reveal.js/test/qunit-1.12.0.js create mode 100644 doc/src/Regression/reveal.js/test/test-markdown-element-attributes.html create mode 100644 doc/src/Regression/reveal.js/test/test-markdown-element-attributes.js create mode 100644 doc/src/Regression/reveal.js/test/test-markdown-slide-attributes.html create mode 100644 doc/src/Regression/reveal.js/test/test-markdown-slide-attributes.js create mode 100644 doc/src/Regression/reveal.js/test/test-markdown.html create mode 100644 doc/src/Regression/reveal.js/test/test-markdown.js create mode 100644 doc/src/Regression/reveal.js/test/test-pdf.html create mode 100644 doc/src/Regression/reveal.js/test/test-pdf.js create mode 100644 doc/src/Regression/reveal.js/test/test.html create mode 100644 doc/src/Regression/reveal.js/test/test.js create mode 100644 doc/src/Splines/Splines.do.txt~ create mode 100644 doc/src/Statistics/Statistics.do.txt~ create mode 100644 doc/src/SupportVMachines/svm.do.txt~ create mode 100644 doc/web/.course_html_file_collection create mode 100644 doc/web/.schedule_html_file_collection create mode 100644 doc/web/course.dlog create mode 100644 doc/web/course.do.txt~ create mode 100644 doc/web/schedule.dlog create mode 100644 doc/web/schedule.do.txt~ create mode 100644 doc/web/tmp_mako__course.do.txt create mode 100644 doc/web/tmp_preprocess__course.do.txt create mode 100644 doc/web/tmp_preprocess__schedule.do.txt diff --git a/README.md~ b/README.md~ new file mode 100644 index 000000000..6d606bff3 --- /dev/null +++ b/README.md~ @@ -0,0 +1,122 @@ +# FYS-STK3155/4155 Applied Data Analysis and Machine Learning, http://www.uio.no/studier/emner/matnat/fys/FYS-STK4155/index-eng.html + + +This site contains all material relevant for the course on Data Analysis and Machine Learning FYS-STK3155/4155. + +## Course content + +Probability theory and statistical methods play a central role in science. Nowadays we are +surrounded by huge amounts of data. For example, there are about one trillion web pages; more than one +hour of video is uploaded to YouTube every second, amounting to 10 years of content every +day; the genomes of 1000s of people, each of which has a length of more than a billion base pairs, have +been sequenced by various labs and so on. +This deluge of data calls for automated methods of data analysis, +which is exactly what machine +learning provides. In this course the approach is to define machine learning as a set of methods that can +automatically detect patterns in data, and then use the uncovered patterns to predict future +data, or to perform other kinds of decision making under uncertainty. Since many of these problems can be studied using +tools of probability theory, the aim of this course is to expose you to central methods in probability theory linked with machine learning. + +This course covers thus topics like Monte Carlo methods and Markov chains, Bayesian statistics, error estimates, various regression methods, optimization of data and error analysis and central algorithms in machine learning. +The course has several numerical projects and numerical exercises that are meant to illustrate the theory. + + + +## Learning outcomes + +The course introduces a variety of central algorithms and methods +essential for studies of data analysis and machine learning. The course is project based and through the various projects, normally three, the students will be exposed to fundamental research problems in these fields, with the aim to reproduce state of the art scientific results. Both supervised and unsupervised methods will be covered. You will learn to develop and structure large codes for studying these systems, get acquainted with computing facilities and learn to handle large scientific projects. A good scientific and ethical conduct is emphasized throughout the course. More specifically, after this course you will + +- Learn about basic data analysis, statistical analysis, Monte Carlo sampling, data optimization and machine learning; +- Be capable of extending the acquired knowledge to other systems and cases; +- Have an understanding of central algorithms used in data analysis and machine learning; +- Gain knowledge of central aspects of Monte Carlo methods, Markov chains, Gibbs samplers and their possible applications; +- Understand linear methods for regression and classification, from ordinary least squares, via Lasso and Ridge to Logistic regression; +- Learn about various neural networks and deep learning methods for supervised and unsupervised learning; +- Learn about about decision trees and random forests +- Learn about support vector machines and kernel transformations +- Reduction of data sets, from PCA to clustering, supervised and unsupervided methods +- Work on numerical projects to illustrate the theory. The projects play a central role and students are expected to know modern programming languages like Python or C++. + +## Prerequisites + +Basic knowledge in programming and mathematics, with an emphasis on linear algebra. Knowledge of Python or/and C++ as programming languages is required and experience with Jupiter notebook is recommended. Required courses are the equivalents to the University of Oslo mathematics courses MAT1100, MAT1110, MAT1120 and at least one of the corresponding computing and programming courses INF1000/INF1110 or MAT-INF1100/MAT-INF1100L/BIOS1100/KJM-INF1100. + + +## The course has two central parts + +1. Statistical analysis and optimization of data +2. Machine learning + +### Statistical analysis and optimization of data + +The following topics will be covered +- Basic concepts, expectation values, variance, covariance, correlation functions and errors; +- Simpler models, binomial distribution, the Poisson distribution, simple and multivariate normal distributions; +- Central elements of Bayesian statistics and modeling; +- Central elements from linear algebra +- Gradient methods for data optimization +- Monte Carlo methods, Markov chains, Metropolis-Hastings algorithm; +- Linear methods for regression and classification; +- Estimation of errors using cross-validation, blocking, bootstrapping and jackknife methods; +- Practical optimization using Singular-value decomposition and least squares for parameterizing data. + + +### Machine learning, mainly supervised learning + +The following topics will be covered +- Linear Regression and Logistic Regression; +- Neural networks and deep learning; +- Decisions trees and nearest neighbor algorithms +- Support vector machines + +All the above topics will be supported by examples, hands-on exercises and project work. + +Computational aspects play a central role and the students are +expected to work on numerical examples and projects which illustrate +the theory and methods. Some of the projects can be coordinated with the high-performance programming course IN4200. + + + +## Practicalities + +1. Four lectures per week, Fall semester, 10 ECTS; +2. Four hours of laboratory sessions for work on computational projects; +3. Three projects which are graded and count 1/3 each of the final grade; +4. A selected number of weekly assignments; +6. The course is part of the CS Master of Science program, but is open to other bachelor and Master of Science students at the University of Oslo; +7. Grading scale: Grades are awarded on a scale from A to F, where A is the best grade and F is a fail; +8. The course will be offered as a FYS-MAT4155 (Master of Science level) and a FYS-MAT3155 (senior undergraduate) course. + +## Possible textbooks + +_Recommended textbooks_: +- Trevor Hastie, Robert Tibshirani, Jerome H. Friedman, The Elements of Statistical Learning, Springer +- Aurelien Geron, Hands‑On Machine Learning with Scikit‑Learn and TensorFlow, O'Reilly + +_General learning book on statistical analysis_: +- Christian Robert and George Casella, Monte Carlo Statistical Methods, Springer +- Peter Hoff, A first course in Bayesian statistical models, Springer + +_General Machine Learning Books_: +- Kevin Murphy, Machine Learning: A Probabilistic Perspective, MIT Press +- Christopher M. Bishop, Pattern Recognition and Machine Learning, Springer +- David J.C. MacKay, Information Theory, Inference, and Learning Algorithms, Cambridge University Press +- David Barber, Bayesian Reasoning and Machine Learning, Cambridge University Press + +## Links to relevant courses at the University of Oslo +The link here https://www.mn.uio.no/english/research/about/centre-focus/innovation/data-science/studies/ gives an excellent overview of courses on Machine learning at UiO. + +- _STK2100 Machine learning and statistical methods for prediction and classification_ http://www.uio.no/studier/emner/matnat/math/STK2100/index-eng.html. +- _IN3050 Introduction to Artificial Intelligence and Machine Learning_ https://www.uio.no/studier/emner/matnat/ifi/IN3050/index-eng.html. Introductory course in machine learning and AI with an algorithmic approach. +- _STK-INF3000/4000 Selected Topics in Data Science_ http://www.uio.no/studier/emner/matnat/math/STK-INF3000/index-eng.html. The course provides insight into selected contemporary relevant topics within Data Science. +- _IN4080 Natural Language Processing_ https://www.uio.no/studier/emner/matnat/ifi/IN4080/index.html. Probabilistic and machine learning techniques applied to natural language processing. +- _STK-IN4300 Statistical learning methods in Data Science_ https://www.uio.no/studier/emner/matnat/math/STK-IN4300/index-eng.html. An advanced introduction to statistical and machine learning. For students with a good mathematics and statistics background. +- _INF4490 Biologically Inspired Computing_ http://www.uio.no/studier/emner/matnat/ifi/INF4490/. An introduction to self-adapting methods also called artificial intelligence or machine learning. +- _IN-STK5000 Adaptive Methods for Data-Based Decision Making_ https://www.uio.no/studier/emner/matnat/ifi/IN-STK5000/index-eng.html. Methods for adaptive collection and processing of data based on machine learning techniques. +- _IN5400/INF5860 Machine Learning for Image Analysis_ https://www.uio.no/studier/emner/matnat/ifi/IN5400/. An introduction to deep learning with particular emphasis on applications within Image analysis, but useful for other application areas too. +- _TEK5040 Deep learning for autonomous systems_ https://www.uio.no/studier/emner/matnat/its/TEK5040/. The course addresses advanced algorithms and architectures for deep learning with neural networks. The course provides an introduction to how deep-learning techniques can be used in the construction of key parts of advanced autonomous systems that exist in physical environments and cyber environments. +- _STK4051 Computational Statistics_ https://www.uio.no/studier/emner/matnat/math/STK4051/index-eng.html +- _STK4021 Applied Bayesian Analysis and Numerical Methods_ https://www.uio.no/studier/emner/matnat/math/STK4021/index-eng.html + + diff --git a/doc/.DS_Store b/doc/.DS_Store new file mode 100644 index 0000000000000000000000000000000000000000..c2e3c9d2f796fb5adaa9029518f18a0a75adff7a GIT binary patch literal 10244 zcmeHMUu+ab82`R&DZ8#qw}7@He-0~9;gq|!6cB-i!P1u1*CE8VcSd))3_ zfm-Z~F^a?&W5oaZB9Zte5RLj`j9`o}5Q#=!Oo(r4B2f|_^qZM&(_RlXJ{W>K&CEAD z-}lYz{Ps69{pJ7w+lxj5APN8yWhS*dsi=^cp5-Yi60~R}k^BKRfdw|0;MoVKSceTE z0wDq+0wDq+0wDso0s?eqvm_}(hGmFAh(L(II09^ch*D-U8OTu~`PM-N4*^J4Q!`Ig zXW;)DbT^EJRyEhq_b`=Kf$4bRnASNjG~_f`1-0>bL~duzH+Yd53KTo zQ%01a3ku-EK`M2~f!Arf*^KRGXdL$5YbZDU!4uWkB}JKiDP3@E*I{=NY}=@UJQN`_ zxfiS_&BR`+z@zc9WmZpBw#p27`QVK%X7aY>`SJJ_j8s)mtC=pz^ruRXr3c-iw6FU` zuU#+fWlvqYKbQ&X-L7ko*Y(V9%N!ceW_P%buUn2;;Les|lF7ba%Q4)ccDLvl9`7eD z6l@%MKx-HtZflN36Y&)zvFLCjo@k9lSFCCo8Ih&>MTxbY2XaS`4L>n*{1i=v5(Yad z)Qacv@@1+8Ad~?pLpNRS93iS?=(D(RsQi=QFXTHB%nBa27-X%hTVm0c=4OW3%9Zh0 zw6&$}UqM!3sukZb?Jn1|e9LwA^!erypV_dE zCaL=RJ-6U@aK}E2`WIF+*%&+X%-P#*7Ce%&ES;%sn0MdeW$_gcuTNdp>SoTGeV3y2 z6Ym_~dfYM#Jwuk^5AM;utmzn*bFjB$c~-_YyB$4eYS(4?@~pa{SwWJ+GFb()7A`GxV5cR)ZUVqlC~{qT=R&c_R$uXqnUVh7z>({8&q{a z8+p6#`8)HbLvM9~hUc?bpl+6=Jxsg7FifLyY{@hxn?$uS85b3e{Xa7g;?NGeVL$CV zN8tp#2yejqa2C$NdH58*ge!0ret@6gI{XU1!3{)=U^Py|8CZ|^<6>NbO_;#7xDMCj z7Tke5u?ru?ejGp@Gnhpi3+Q7JOZX%{jnCk-cmiL**YHhz3*W{w_#u9TALA$ZIbOhv zcnQD5YxtvhzsT6)-##`6A@OIDm6;subCHubXL9}4rME7Vzo`}Uy}Q0);o@b>TUW2^ z82|odUwjfNC-RV!Pq2%#Zj4o&?U(PFbMIVrz7ic<1LbR@hj?lpkr>&u>B72ET_h2| zG>JbdP3lsa7J-IB+oV1uNm@kc5^9S~yw|FPu1#4b5yQ1oNANg~a;iRuC-HfF2~XiG_$t1R@8D^Cmq_~_p2PF_0VnTg_*EHQ zx0TWLN@cp1om*8fMCVr7!5`7dcj8$!x$E#LL?A>UL?A>UMBp|>pi0au%I^O+-u(Cf z+jREfv_b?z1Q-HXol2+LS>+~pu)Fpsudmvf>S5hYN;$pwZHk#*65Ow1yS9QU}T6}{1J{W#AD-fsAh4Cv)9 z>7HJ2cdh#?>T26cIH8_TRcBebe=Ud(pSy>T1AIq(%3po9PxHKY);Z34rVVo1P(dsC zHT~?-$>nPKj$C?sRCOzpQIC;4TCeX@XTTY722P3r)NGODQqf0ez!`7`jtt26AwUIF z!$vWFIxxf*062v?3g*&FNK7zH4I4$QKv+Y88p_sUu!h4P%r7-;6g8aKnh&;<**X-C zr(^#R-HB60ADsbbpv%CSKF*~6-+bTycZ2-N8E^(piUFQxdA7hSS#2G>oYdL?y@QH~ nU!!;o!6cSq_)00hhDL!s$OM=gHj1!7{EtAS!3SsHPZ{_E?j}pk literal 0 HcmV?d00001 diff --git a/doc/DataFiles/Finance/.DS_Store b/doc/DataFiles/Finance/.DS_Store new file mode 100644 index 0000000000000000000000000000000000000000..f27267b2a20e5b38155436b0094e47dfdf01337e GIT binary patch literal 6148 zcmeHKy=ucS5WZ_O7`kNa&_S4=C|$G#<65>bOD$f8t4rmLnab6xI0HY< z0Pbv&=0?$bXTTY720j^(^C6%KM#HRFjt+FC1OUo2x(IZsB_t*oM#HQK3xqWksG)2n z25UIxgZV|ntf=9{R(!Cn{87AcSsnR9aVL(7-a7-%z?^|=9WLemU*ngl7WwlKA2|cg zz&~Sv7pY4te3ad-7oR70Z9scQ6A`~G3IzJ>5rBc5Be%t<_8>a^qG48)EHYohf&LIE Lgm~u+`~m}KY1k?7 literal 0 HcmV?d00001 diff --git a/doc/LectureNotes/.book.copyright b/doc/LectureNotes/.book.copyright new file mode 100644 index 000000000..4a0005b5d --- /dev/null +++ b/doc/LectureNotes/.book.copyright @@ -0,0 +1 @@ +{'holder': ['Morten Hjorth-Jensen'], 'year': '1999-2018', 'license': 'Released under CC Attribution-NonCommercial 4.0 license', 'cite doconce': False} \ No newline at end of file diff --git a/doc/LectureNotes/book.dlog b/doc/LectureNotes/book.dlog new file mode 100644 index 000000000..07cf15422 --- /dev/null +++ b/doc/LectureNotes/book.dlog @@ -0,0 +1,231 @@ +translating doconce text in book.do.txt to ipynb +*** error: could not open the file src/plot_Hudson.py used in +@@@CODE src/plot_Hudson.py +translating doconce text in book.do.txt to ipynb +copy complete file src/plot_Hudson.py (format: pypro) +copy complete file src/Hudson_Bay.py (format: pypro) +*** error: figure file "fig/Hudson_Bay_sim" does not exist! +translating doconce text in book.do.txt to ipynb +copy complete file src/plot_Hudson.py (format: pypro) +copy complete file src/Hudson_Bay.py (format: pypro) +figure file fig/Hudson_Bay_data: + can use fig/Hudson_Bay_data.png for format ipynb +figure file fig/Hudson_Bay_sim: + can use fig/Hudson_Bay_sim.png for format ipynb +collected all required additional files in ipynb-book-src.tar.gz which must be distributed with the notebook +*** warning: latex envir \begin{bmatrix} does not work well in Markdown. + Stick to \[ ... \], equation, equation*, align, or align* + environments in math environments. + +*** warning: latex envir \begin{bmatrix} does not work well in Markdown. + Stick to \[ ... \], equation, equation*, align, or align* + environments in math environments. + +*** warning: latex envir \begin{bmatrix} does not work well in Markdown. + Stick to \[ ... \], equation, equation*, align, or align* + environments in math environments. + +*** warning: latex envir \begin{bmatrix} does not work well in Markdown. + Stick to \[ ... \], equation, equation*, align, or align* + environments in math environments. + +*** warning: latex envir \begin{bmatrix} does not work well in Markdown. + Stick to \[ ... \], equation, equation*, align, or align* + environments in math environments. + +*** warning: latex envir \begin{bmatrix} does not work well in Markdown. + Stick to \[ ... \], equation, equation*, align, or align* + environments in math environments. + +*** warning: latex envir \begin{bmatrix} does not work well in Markdown. + Stick to \[ ... \], equation, equation*, align, or align* + environments in math environments. + +*** warning: latex envir \begin{bmatrix} does not work well in Markdown. + Stick to \[ ... \], equation, equation*, align, or align* + environments in math environments. + +*** warning: latex envir \begin{bmatrix} does not work well in Markdown. + Stick to \[ ... \], equation, equation*, align, or align* + environments in math environments. + +*** warning: latex envir \begin{bmatrix} does not work well in Markdown. + Stick to \[ ... \], equation, equation*, align, or align* + environments in math environments. + +*** warning: latex envir \begin{bmatrix} does not work well in Markdown. + Stick to \[ ... \], equation, equation*, align, or align* + environments in math environments. + +*** warning: latex envir \begin{bmatrix} does not work well in Markdown. + Stick to \[ ... \], equation, equation*, align, or align* + environments in math environments. + +*** warning: latex envir \begin{bmatrix} does not work well in Markdown. + Stick to \[ ... \], equation, equation*, align, or align* + environments in math environments. + +output in book.ipynb +*** error: file has a mako construction ${\bf \hat{J}' + but seemingly no definition in <%...%>' + (it is not a command-line given mako variable either). + However, if this is a variable in a Makefile or Bash script + run with --no_mako - and you cannot use mako and Makefile or Bash variables + in the same document! + +*** error: file has a mako construction ${\bf \hat{J}' + but seemingly no definition in <%...%>' + (it is not a command-line given mako variable either). + However, if this is a variable in a Makefile or Bash script + run with --no_mako - and you cannot use mako and Makefile or Bash variables + in the same document! + +translating doconce text in book.do.txt to ipynb +*** error: found multiple labels: + eq:def_covariance eq:autocorrelation_time eq:error_estimate_corr_time +translating doconce text in book.do.txt to ipynb +copy complete file src/plot_Hudson.py (format: pypro) +copy complete file src/Hudson_Bay.py (format: pypro) +figure file fig/Hudson_Bay_data: + can use fig/Hudson_Bay_data.png for format ipynb +figure file fig/Hudson_Bay_sim: + can use fig/Hudson_Bay_sim.png for format ipynb +collected all required additional files in ipynb-book-src.tar.gz which must be distributed with the notebook +*** warning: latex envir \begin{bmatrix} does not work well in Markdown. + Stick to \[ ... \], equation, equation*, align, or align* + environments in math environments. + +*** warning: latex envir \begin{bmatrix} does not work well in Markdown. + Stick to \[ ... \], equation, equation*, align, or align* + environments in math environments. + +*** warning: latex envir \begin{bmatrix} does not work well in Markdown. + Stick to \[ ... \], equation, equation*, align, or align* + environments in math environments. + +*** warning: latex envir \begin{bmatrix} does not work well in Markdown. + Stick to \[ ... \], equation, equation*, align, or align* + environments in math environments. + +*** warning: latex envir \begin{bmatrix} does not work well in Markdown. + Stick to \[ ... \], equation, equation*, align, or align* + environments in math environments. + +*** warning: latex envir \begin{bmatrix} does not work well in Markdown. + Stick to \[ ... \], equation, equation*, align, or align* + environments in math environments. + +*** warning: latex envir \begin{bmatrix} does not work well in Markdown. + Stick to \[ ... \], equation, equation*, align, or align* + environments in math environments. + +*** warning: latex envir \begin{bmatrix} does not work well in Markdown. + Stick to \[ ... \], equation, equation*, align, or align* + environments in math environments. + +*** warning: latex envir \begin{bmatrix} does not work well in Markdown. + Stick to \[ ... \], equation, equation*, align, or align* + environments in math environments. + +*** warning: latex envir \begin{bmatrix} does not work well in Markdown. + Stick to \[ ... \], equation, equation*, align, or align* + environments in math environments. + +*** warning: latex envir \begin{bmatrix} does not work well in Markdown. + Stick to \[ ... \], equation, equation*, align, or align* + environments in math environments. + +*** warning: latex envir \begin{bmatrix} does not work well in Markdown. + Stick to \[ ... \], equation, equation*, align, or align* + environments in math environments. + +*** warning: latex envir \begin{bmatrix} does not work well in Markdown. + Stick to \[ ... \], equation, equation*, align, or align* + environments in math environments. + +*** warning: latex envir \begin{bmatrix} does not work well in Markdown. + Stick to \[ ... \], equation, equation*, align, or align* + environments in math environments. + +*** warning: latex envir \begin{bmatrix} does not work well in Markdown. + Stick to \[ ... \], equation, equation*, align, or align* + environments in math environments. + +*** warning: latex envir \begin{bmatrix} does not work well in Markdown. + Stick to \[ ... \], equation, equation*, align, or align* + environments in math environments. + +output in book.ipynb +translating doconce text in book.do.txt to ipynb +copy complete file src/plot_Hudson.py (format: pypro) +copy complete file src/Hudson_Bay.py (format: pypro) +figure file fig/Hudson_Bay_data: + can use fig/Hudson_Bay_data.png for format ipynb +figure file fig/Hudson_Bay_sim: + can use fig/Hudson_Bay_sim.png for format ipynb +collected all required additional files in ipynb-book-src.tar.gz which must be distributed with the notebook +*** warning: latex envir \begin{bmatrix} does not work well in Markdown. + Stick to \[ ... \], equation, equation*, align, or align* + environments in math environments. + +*** warning: latex envir \begin{bmatrix} does not work well in Markdown. + Stick to \[ ... \], equation, equation*, align, or align* + environments in math environments. + +*** warning: latex envir \begin{bmatrix} does not work well in Markdown. + Stick to \[ ... \], equation, equation*, align, or align* + environments in math environments. + +*** warning: latex envir \begin{bmatrix} does not work well in Markdown. + Stick to \[ ... \], equation, equation*, align, or align* + environments in math environments. + +*** warning: latex envir \begin{bmatrix} does not work well in Markdown. + Stick to \[ ... \], equation, equation*, align, or align* + environments in math environments. + +*** warning: latex envir \begin{bmatrix} does not work well in Markdown. + Stick to \[ ... \], equation, equation*, align, or align* + environments in math environments. + +*** warning: latex envir \begin{bmatrix} does not work well in Markdown. + Stick to \[ ... \], equation, equation*, align, or align* + environments in math environments. + +*** warning: latex envir \begin{bmatrix} does not work well in Markdown. + Stick to \[ ... \], equation, equation*, align, or align* + environments in math environments. + +*** warning: latex envir \begin{bmatrix} does not work well in Markdown. + Stick to \[ ... \], equation, equation*, align, or align* + environments in math environments. + +*** warning: latex envir \begin{bmatrix} does not work well in Markdown. + Stick to \[ ... \], equation, equation*, align, or align* + environments in math environments. + +*** warning: latex envir \begin{bmatrix} does not work well in Markdown. + Stick to \[ ... \], equation, equation*, align, or align* + environments in math environments. + +*** warning: latex envir \begin{bmatrix} does not work well in Markdown. + Stick to \[ ... \], equation, equation*, align, or align* + environments in math environments. + +*** warning: latex envir \begin{bmatrix} does not work well in Markdown. + Stick to \[ ... \], equation, equation*, align, or align* + environments in math environments. + +*** warning: latex envir \begin{bmatrix} does not work well in Markdown. + Stick to \[ ... \], equation, equation*, align, or align* + environments in math environments. + +*** warning: latex envir \begin{bmatrix} does not work well in Markdown. + Stick to \[ ... \], equation, equation*, align, or align* + environments in math environments. + +*** warning: latex envir \begin{bmatrix} does not work well in Markdown. + Stick to \[ ... \], equation, equation*, align, or align* + environments in math environments. + +output in book.ipynb diff --git a/doc/LectureNotes/book.do.txt~ b/doc/LectureNotes/book.do.txt~ new file mode 100644 index 000000000..0f267c0d1 --- /dev/null +++ b/doc/LectureNotes/book.do.txt~ @@ -0,0 +1,12249 @@ +TITLE: Data Analysis and Machine Learning +AUTHOR: Morten Hjorth-Jensen {copyright, 1999-present|CC BY-NC} at Department of Physics, University of Oslo & Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University +DATE: today + +TOC: on + + + +======= Introduction ======= + +Statistics, data science and machine learning form important fields of +research in modern science. They describe how to learn and make +predictions from data, as well as allowing us to extract important +correlations about physical process and the underlying laws of motion +in large data sets. The latter, big data sets, appear frequently in +essentially all disciplines, from the traditional Science, Technology, +Mathematics and Engineering fields to Life Science, Law, education +research, the Humanities and the Social Sciences. + +It has become more +and more common to see research projects on big data in for example +the Social Sciences where extracting patterns from complicated survey +data is one of many research directions. Having a solid grasp of data +analysis and machine learning is thus becoming central to scientific +computing in many fields, and competences and skills within the fields +of machine learning and scientific computing are nowadays strongly +requested by many potential employers. The latter cannot be +overstated, familiarity with machine learning has almost become a +prerequisite for many of the most exciting employment opportunities, +whether they are in bioinformatics, life science, physics or finance, +in the private or the public sector. This author has had several +students or met students who have been hired recently based on their +skills and competences in scientific computing and data science, often +with marginal knowledge of machine learning. + +Machine learning is a subfield of computer science, and is closely +related to computational statistics. It evolved from the study of +pattern recognition in artificial intelligence (AI) research, and has +made contributions to AI tasks like computer vision, natural language +processing and speech recognition. Many of the methods we will study are also +strongly rooted in basic mathematics and physics research. + +Ideally, machine learning represents the science of giving computers +the ability to learn without being explicitly programmed. The idea is +that there exist generic algorithms which can be used to find patterns +in a broad class of data sets without having to write code +specifically for each problem. The algorithm will build its own logic +based on the data. You should however always keep in mind that +machines and algorithms are to a large extent developed by humans. The +insights and knowledge we have about a specific system, play a central +role when we develop a specific machine learning algorithm. + +Machine learning is an extremely rich field, in spite of its young +age. The increases we have seen during the last three decades in +computational capabilities have been followed by developments of +methods and techniques for analyzing and handling large date sets, +relying heavily on statistics, computer science and mathematics. The +field is rather new and developing rapidly. Popular software packages +written in Python for machine learning like +"Scikit-learn":"http://scikit-learn.org/stable/", +"Tensorflow":"https://www.tensorflow.org/", +"PyTorch":"http://pytorch.org/" and "Keras":"https://keras.io/", all +freely available at their respective GitHub sites, encompass +communities of developers in the thousands or more. And the number of +code developers and contributors keeps increasing. Not all the +algorithms and methods can be given a rigorous mathematical +justification, opening up thereby large rooms for experimenting and +trial and error and thereby exciting new developments. However, a +solid command of linear algebra, multivariate theory, probability +theory, statistical data analysis, understanding errors and Monte +Carlo methods are central elements in a proper understanding of many +of algorithms and methods we will discuss. + + + +===== Learning outcomes ===== + +These setsof lectures aim at giving you an overview of central aspects of +statistical data analysis as well as some of the central algorithms +used in machine learning. We will introduce a variety of central +algorithms and methods essential for studies of data analysis and +machine learning. + +Hands-on projects and experimenting with data and algorithms plays a central role in +these lectures, and our hope is, through the various +projects and exercies, to expose you to fundamental +research problems in these fields, with the aim to reproduce state of +the art scientific results. You will learn to develop and +structure large codes for studying these systems, get acquainted with +computing facilities and learn to handle large scientific projects. A +good scientific and ethical conduct is emphasized throughout the +course. More specifically, you will + +o learn about basic data analysis, Bayesian statistics, Monte Carlo methods, data optimization and machine learning; +o be capable of extending the acquired knowledge to other systems and cases; +o Have an understanding of central algorithms used in data analysis and machine learning; +o Gain knowledge of central aspects of Monte Carlo methods, Markov chains, Gibbs samplers and their possible applications, from numerical integration to simulation of stock markets; +o Understand methods for regression and classification; +o Learn about neural network, genetic algorithms and Boltzmann machines; +o Work on numerical projects to illustrate the theory. The projects play a central role and you are expected to know modern programming languages like Python or C++, in addition to a basic knowledge of linear algebra (typically taught during the first one or two years of undergraduate studies). + +There are several topics we will cover here, spanning from a +statistical data analysis and its basic concepts such expectation +values, variance, covariance, correlation functions and errors, via +well-known probability distribution functions like uniform +distribution, the binomial distribution, the Poisson distribution and +simple and multivariate normal distributions to central elements of +Bayesian statistics and modeling. We will also remind the reader about +central elements from linear algebra and standard methods based on +linear algebra used to fit functions such Cubic splines and gradient +methods for data optimization and the Singular-value decomposition and +least square methods for parameterizing data. + +We will also cover Monte Carlo methods, Markov chains, well-known +algorithms for sampling stochastic events like the Metropolis-Hastings +and Gibbs sampling methods. An important aspect of all our +calculations is a proper estimation of errors. Here we will also +discuss famous resampling techniques like the blocking, bootstrapping +and jackknife methods. + +The second part of the material covers several algorithms used in +machine learning. + + + + + + +===== Types of Machine Learning ===== + + +The approaches to machine learning are many, but are often split into +two main categories. In *supervised learning* we know the answer to a +problem, and let the computer deduce the logic behind it. On the other +hand, *unsupervised learning* is a method for finding patterns and +relationship in data sets without any prior knowledge of the system. +Some authours also operate with a third category, namely +*reinforcement learning*. This is a paradigm of learning inspired by +behavioral psychology, where learning is achieved by trial-and-error, +solely from rewards and punishment. + +Another way to categorize machine learning tasks is to consider the +desired output of a system. Some of the most common tasks are: + + * Classification: Outputs are divided into two or more classes. The goal is to produce a model that assigns inputs into one of these classes. An example is to identify digits based on pictures of hand-written ones. Classification is typically supervised learning. + + * Regression: Finding a functional relationship between an input data set and a reference data set. The goal is to construct a function that maps input data to continuous output values. + + * Clustering: Data are divided into groups with certain common traits, without knowing the different groups beforehand. It is thus a form of unsupervised learning. + + +The methods we cover have three main topics in common, irrespective of +whether we deal with supervised or unsupervised learning. The first +ingredient is normally our data set (which can be subdivided into +training and test data), the second item is a model which is normally a +function of some parameters. The model reflects our knowledge of the system (or lack thereof). As an example, if we know that our data show a behavior similar to what would be predicted by a polynomial, fitting our data to a polynomial of some degree would then determin our model. + +The last ingredient is a so-called _cost_ +function which allows us to present an estimate on how good our model +is in reproducing the data it is supposed to train. + +Here we will build our machine learning approach on elements of the +statistical foundation discussed above, with elements from data +analysis, stochastic processes etc. We will discuss the following +machine learning algorithms + +o Linear regression and its variants, in essence polynomial regression +o Decision tree algorithms, from simpler to more complex ones +o Nearest neighbors models +o Bayesian statistics and regression +o Support vector machines and finally various variants of +o Artifical neural networks and deep learning +o Networks for unsupervised learning using for example reduced Boltzmann machines. + + + +===== Choice of programming language ===== + +Python plays nowadays a central role in the development of machine +learning techniques and tools for data analysis. In particular, seen +the wealth of machine learning and data analysis packages written in +Python, easy to use libraries with immediate visualization(and not the +least impressive galleries of existing example), the popularity of the +Jupyter notebook framework with the possibility to run _R_ codes or +compiled programs written in C++, and much more made our choice of +programming language for this series of lectures of easy. However, +since the focus here is not only on using existing Python tools such +as _scikit-learn_ or _tensorflow_, but also on developing your own +algorithms and codes, we will as far as possible present many of these +algorithms eithers a Python codes or C++ codes. Finally, we will, as +far as possible keep parallel versions of the data analysis and +machine larning programming aspects in _R_ as +well. "R":"https://www.r-project.org/" is a language and environment +for statistical computing and graphics which is widely used in +statistics and mathematics applications. + +The reason we also focus on compiled languages like C++ (or +Fortran), is that Python is still notoriously slow when we do not +utilize highly streamlined computational libraries like +"Lapack":"http://www.netlib.org/lapack/" or other numerical libraries +written in compiled languages (many of these libraries are written in +Fortran). Although a project like "Numba":"https://numba.pydata.org/" +holds great promise for speeding up the unrolling of lengthy loops, C+ +and Fortran are presently still the performance winners. Numba gives +you potentially the power to speed up your applications with high +performance functions written directly in Python. In particular, +array-oriented and math-heavy Python code can achieve similar +performance to C, C++ and Fortran. However, even with these speed-ups, +for codes involving heavy Markov Chain Monte Carlo analyses and +optimizations of cost functions, C++/C or Fortran codes tend to +outperform Python codes. + +Presently thus, the community tends to let +code written in C++/C or Fortran do the heavy duty numerical +number crunching and leave the post-analysis of the data to the above +mentioned Python modules or software packages. However, with the developments taking place in for example the Python community, and seen +the changes during the last decade, the above situation may change swiftly in the not too distant future. + +Many of the examples we discuss in this series of lectures come with +existing data files or provide code examples which produce the data to +be analyzed. Most of the applications we will discuss deal with +small data sets (less than a terabyte of information) and can easily +be analyzed and tested on standard off the shelf laptops you find in general +grocery stores. + +===== Data handling, machine learning and ethical aspects ===== + +In most of the cases we will study, we will either generate the data +to analyze ourselves (both for supervised learning and unsupervised +learning) or we will recur again and again to data present in say +_scikit-learn_ or _tensorflow_. Many of the examples we end up +dealing with are from a privacy and data protection point of view, +rather inoccuous and boring results of numerical +calculations. However, this does not hinder us from developing a sound +ethical attitude to the data we use, how we analyze the data and how +we handle the data. + +The most immediate and simplest possible ethical aspects deal with our +approach to the scientific process. Nowadays, with version control +software like "Git":"https://git-scm.com/" and various online +repositories like "Github":"https://github.com/", +"Gitlab":"https://about.gitlab.com/" etc, we can easily make our codes +and data sets we have used, freely and easily accessible to a wider +community. This helps us almost automagically in making our science +reproducible. The large open-source development communities involved +in say "Scikit-learn":"http://scikit-learn.org/stable/", +"Tensorflow":"https://www.tensorflow.org/", +"PyTorch":"http://pytorch.org/" and "Keras":"https://keras.io/", are +all excellent examples of this. The codes can be tested and improved +upon continuosly, helping thereby our scientific community at large in +developing data analysis and machine learning tools. It is much +easier today to gain traction and acceptance for making your science +reproducible. From a societal stand, this is an important element +since many of the developers are employees of large public institutions like +universities and research labs. Our taxpayer do deserve to get +something back for their bucks. + +However, this more mechanical aspect of the ethics of science (in +particular the reproducibility of scientific results) is something +which is obvious and everybody should do as part of the dialectics of +science. The fact that many scientists are not willing to share their codes or +data is detrimental to the scientific discourse. + +Before we proceed, we should add a disclaimer. Even though +we may dream of computers developing some kind of higher learning +capabilities, at the end (even if the artificial intelligence +community keeps touting our ears full of fancy futuristic avenues), it is we +who end up constructing and instructing, via various algorithms, the +computers. Self-driving cars for example, rely on sofisticated +programs which take into account all possible situations a car can +encounter. In addition, extensive usage of training datas from GPS +information, maps etc, are typically fed into the software for +self-driving cars. Adding to this various sensors and cameras that +feed information to the programs, there are zillions of ethical issues +which arise from this. + +For self-driving cars, where basically many of the standard machine +learning algorithms discussed here enter into the codes, at a certain +stage we have to make choices. Yes, we , the lads and lasses who wrote +a program for a specific brand of a self-driving car. As an example, +a most carmakers have as their utmost priority the security of the +driver and the accompanying passengers. A famous carmaker, which is +one of the leaders in the market of self-driving cars, had _if_ +statements of the following type: suppose there are two obstacles in +front of you and you cannot avoid to collide with one of them. One of +the obstacles is a monstertruck while the other one is a kindergarten +class trying to cross the road. The self-driving car algo would then +opt for the hitting the small folks instead of the monstertruck, since +the likelihood of surving a collision with our future citizens, is +much higher. + +This brings us leads then to serious ethical aspects. Why should we +opt for such an option? Who decides and who is entitled to make such +choices? Keep in mind that many of the algorithms you will about in +this series of lectures or hear about later, are indeed based on +simple programming instructions. And you are very likely to be one of +the people who may end up writing such a code. Thus, developing a +sound ethical attitude to what we do, an approach well beyond the +simple mechanistic one of making our science available and +reproducible, is much needed. The example of the self-driving cars is +just one of infinitely many cases where we have to make choices. When +you analyze data on economic inequalities, who guarantees that you are +not weighting some data in a particular way, perhaps because you dearly want a +specific conclusion which may support your political views? + +We do not have the answers here, but we want you think over these +topics in a more overarching way. A statistical data analysis with +its dry numbers and graphs meant to guide the eye, do not necessarily +reflect the truth, whatever that is. As a scientist, and after a +university education, you are supposedly a better citizen, with an +improved critical view and understanding of the scientific method, and +perhaps some deeper understandings of the ethics of science at +large. Use these insights. Be a critical citizen. You owe it to our +societies. + + + + + +===== Software ===== + +Our emphasis throughout this series of lectures +is on understanding the mathematical aspects of +different algorithms used in the fields of data analysis and machine learning. + +However, where possible we will emphasize the +importance of using available software. We start thus with a hands-on +and top-down approach to machine learning. The aim is thus to start with +relevant data or data we have produced +and use these to introduce statistical data analysis +concepts and machine learning algorithms before we delve into the +algorithms themselves. The examples we will use in the beginning, start with simple +polynomials with random noise added. We will use the Python +software package "Scikit-learn":"http://scikit-learn.org/stable/" and +introduce various machine learning algorithms to make fits of +the data and predictions. We move thereafter to more interesting +cases such as the simulation of financial transactions or disease +models. These are examples where we can easily set up the data and +then use machine learning algorithms included in for example +_scikit-learn_. + +These examples will serve us the purpose of getting +started. Furthermore, they allow us to catch more than two birds with +a stone. They will allow us to bring in some programming specific +topics and tools as well as showing the power of various Python (and +R) packages for machine learning and statistical data analysis. In the +lectures on linear algebra we cover in more detail various programming +features of languages like Python and C++ (and other), we will also +look into more specific linear functions which are relevant for the +various algorithms we will discuss. Here, we will mainly focus on two +specific Python packages for Machine Learning, scikit-learn and +tensorflow (see below for links etc). Moreover, the examples we +introduce will serve as inputs to many of our discussions later, as +well as allowing you to set up models and produce your own data and +get started with programming. + + + + + +===== Software and needed installations ===== + +We will make extensive use of Python as programming language and its +myriad of available libraries. You will find +IPython/Jupyter notebooks invaluable in your work. You can run _R_ +codes in the Jupyter/IPython notebooks, with the immediate benefit of +visualizing your data. You can also use compiled languages like C++, +Rust, Fortran etc if you prefer. The focus in these lectures will be +on Python, but we will provide many code examples for those of you who +prefer R or compiled languages. You can integrate C++ codes and R in for example +a Jupyter notebook. + + +If you have Python installed (we recommend Python3) and you feel +pretty familiar with installing different packages, we recommend that +you install the following Python packages via _pip_ as + +o pip install numpy scipy matplotlib ipython scikit-learn mglearn sympy pandas pillow + +For Python3, replace _pip_ with _pip3_. + +For OSX users we recommend, after having installed Xcode, to +install _brew_. Brew allows for a seamless installation of additional +software via for example + +o brew install python3 + +For Linux users, with its variety of distributions like for example the widely popular Ubuntu distribution, +you can use _pip_ as well and simply install Python as + +o sudo apt-get install python3 (or python for pyhton2.7) + +etc etc. + + +===== Python installers ===== + +If you don't want to perform these operations separately and venture +into the hassle of exploring how to set up dependencies and paths, we +recommend two widely used distrubutions which set up all relevant +dependencies for Python, namely + +* "Anaconda":"https://docs.anaconda.com/", + +which is an open source +distribution of the Python and R programming languages for large-scale +data processing, predictive analytics, and scientific computing, that +aims to simplify package management and deployment. Package versions +are managed by the package management system _conda_. + +* "Enthought canopy":"https://www.enthought.com/product/canopy/" + +is a Python +distribution for scientific and analytic computing distribution and +analysis environment, available for free and under a commercial +license. + + + +===== Installing R, C++, cython or Julia ===== + +You will also find it convenient to utilize R. Although we will mainly +use Python during lectures and in various projects and exercises, we +provide a full R set of codes for the same examples. Those of you +already familiar with R should feel free to continue using R, keeping +however an eye on the parallel Python set ups. Similarly, if you are a +Python afecionado, feel free to explore R as well. Jupyter/Ipython +notebook allows you to run _R_ codes interactively in your +browser. The software library _R_ is tuned to statistically analysis +and allows for an easy usage of the tools we will discuss in these +texts. + +To install _R_ with Jupyter notebook +"follow the link here":"https://mpacer.org/maths/r-kernel-for-ipython-notebook" + + + + +===== Installing R, C++, cython, Numba etc ===== + + +For the C++ aficionados, Jupyter/IPython notebook allows you also to +install C++ and run codes written in this language interactively in +the browser. Since we will emphasize writing many of the algorithms +yourself, you can thus opt for either Python or C++ (or Fortran or other compiled languages) as programming +languages. + +To add more entropy, _cython_ can also be used when running your +notebooks. It means that Python with the Jupyter/IPython notebook +setup allows you to integrate widely popular softwares and tools for +scientific computing. Similarly, the +"Numba Python package":"https://numba.pydata.org/" delivers increased performance +capabilities with minimal rewrites of your codes. With its +versatility, including symbolic operations, Python offers a unique +computational environment. Your Jupyter/IPython notebook can easily be +converted into a nicely rendered _PDF_ file or a Latex file for +further processing. For example, convert to latex as + +!bc +pycod jupyter nbconvert filename.ipynb --to latex +!ec + +And to add more versatility, the Python package "SymPy":"http://www.sympy.org/en/index.html" is a Python library for symbolic mathematics. It aims to become a full-featured computer algebra system (CAS) and is entirely written in Python. + +Finally, if you wish to use the light mark-up language +"doconce":"https://github.com/hplgit/doconce" you can convert a standard ascii text file into various HTML +formats, ipython notebooks, latex files, pdf files etc with minimal edits. + + +===== Simple linear regression model using _scikit-learn_ ===== + +We start with perhaps our simplest possible example, using _scikit-learn_ to perform linear regression analysis on a data set produced by us. +What follows is a simple Python code where we have defined function $y$ in terms of the variable $x$. Both are defined as vectors of dimension $1\times 100$. The entries to the vector $\hat{x}$ are given by random numbers generated with a uniform distribution with entries $x_i \in [0,1]$ (more about probability distribution functions later). These values are then used to define a function $y(x)$ (tabulated again as a vector) with a linear dependence on $x$ plus a random noise added via the normal distribution. + + +The Numpy functions are imported used the _import numpy as np_ +statement and the random number generator for the uniform distribution +is called using the function _np.random.rand()_, where we specificy +that we want $100$ random variables. Using Numpy we define +automatically an array with the specified number of elements, $100$ in +our case. With the Numpy function _randn()_ we can compute random +numbers with the normal distribution (mean value $\mu$ equal to zero and +variance $\sigma^2$ set to one) and produce the values of $y$ assuming a linear +dependence as function of $x$ + +!bt +\[ +y = 2x+N(0,1), +\] +!et + +where $N(0,1)$ represents random numbers generated by the normal +distribution. From _scikit-learn_ we import then the +_LinearRegression_ functionality and make a prediction $\tilde{y} = +\alpha + \beta x$ using the function _fit(x,y)_. We call the set of +data $(\hat{x},\hat{y})$ for our training data. The Python package +_scikit-learn_ has also a functionality which extracts the above +fitting parameters $\alpha$ and $\beta$ (see below). Later we will +distinguish between training data and test data. + +For plotting we use the Python package +"matplotlib":"https://matplotlib.org/" which produces publication +quality figures. Feel free to explore the extensive +"gallery":"https://matplotlib.org/gallery/index.html" of examples. In +this example we plot our original values of $x$ and $y$ as well as the +prediction _ypredict_ ($\tilde{y}$), which attempts at fitting our +data with a straight line. + +The Python code follows here. +!bc pycod +# Importing various packages +import numpy as np +import matplotlib.pyplot as plt +from sklearn.linear_model import LinearRegression + +x = np.random.rand(100,1) +y = 2*x+np.random.randn(100,1) +linreg = LinearRegression() +linreg.fit(x,y) +xnew = np.array([[0],[1]]) +ypredict = linreg.predict(xnew) + +plt.plot(xnew, ypredict, "r-") +plt.plot(x, y ,'ro') +plt.axis([0,1.0,0, 5.0]) +plt.xlabel(r'$x$') +plt.ylabel(r'$y$') +plt.title(r'Simple Linear Regression') +plt.show() +!ec + +This example serves several aims. It allows us to demonstrate several +aspects of data analysis and later machine learning algorithms. The +immediate visualization shows that our linear fit is not +impressive. It goes through the data points, but there are many +outliers which are not reproduced by our linear regression. We could +now play around with this small program and change for example the +factor in front of $x$ and the normal distribution. Try to change the +function $y$ to + +!bt +\[ +y = 10x+0.01 \times N(0,1), +\] +!et + +where $x$ is defined as before. Does the fit look better? Indeed, by +reducing the role of the normal distribution we see immediately that +our linear prediction seemingly reproduces better the training +set. However, this testing 'by the eye' is obviouly not satisfactory in the +long run. Here we have only defined the training data and our model, and +have not discussed a more rigorous approach to the _cost_ function. + +We need more rigorous criteria in defining whether we have succeeded or +not in modeling our training data. You will be surprised to see that +many scientists seldomly venture beyond this 'by the eye' approach. A +standard approach for the *cost* function is the so-called $\chi^2$ +function + +!bt +\[ \chi^2 = \frac{1}{n} +\sum_{i=0}^{n-1}\frac{(y_i-\tilde{y}_i)^2}{\sigma_i^2}, +\] +!et + +where $\sigma_i^2$ is the variance (to be defined later) of the entry +$y_i$. We may not know the explicit value of $\sigma_i^2$, it serves +however the aim of scaling the equations and make the cost function +dimensionless. + +Minimizing the cost function is a central aspect of +our discussions to come. Finding its minima as function of the model +parameters ($\alpha$ and $\beta$ in our case) will be a recurring +theme in these series of lectures. Essentially all machine learning +algorithms we will discuss center around the minimization of the +chosen cost function. This depends in turn on our specific +model for describing the data, a typical situation in supervised +learning. Automatizing the search for the minima of the cost function is a +central ingredient in all algorithms. Typical methods which are +employed are various variants of _gradient_ methods. These will be +discussed in more detail later. Again, you'll be surprised to hear that +many practitioners minimize the above function ''by the eye', popularly dubbed as +'chi by the eye'. That is, change a parameter and see (visually and numerically) that +the $\chi^2$ function becomes smaller. + +There are many ways to define the cost function. A simpler approach is to look at the relative difference between the training data and the predicted data, that is we define +the relative error as + +!bt +\[ +\epsilon_{\mathrm{relative}}= \frac{\vert \hat{y} -\hat{\tilde{y}}\vert}{\vert \hat{y}\vert}. +\] +!et +We can modify easily the above Python code and plot the relative instead +!bc pycod +import numpy as np +import matplotlib.pyplot as plt +from sklearn.linear_model import LinearRegression + +x = np.random.rand(100,1) +y = 5*x+0.01*np.random.randn(100,1) +linreg = LinearRegression() +linreg.fit(x,y) +ypredict = linreg.predict(x) + +plt.plot(x, np.abs(ypredict-y)/abs(y), "ro") +plt.axis([0,1.0,0.0, 0.5]) +plt.xlabel(r'$x$') +plt.ylabel(r'$\epsilon_{\mathrm{relative}}$') +plt.title(r'Relative error') +plt.show() +!ec + +Depending on the parameter in front of the normal distribution, we may +have a small or larger relative error. Try to play around with +different training data sets and study (graphically) the value of the +relative error. + +As mentioned above, _scikit-learn_ has an impressive functionality. +We can for example extract the values of $\alpha$ and $\beta$ and +their error estimates, or the variance and standard deviation and many +other properties from the statistical data analysis. + +Here we show an +example of the functionality of scikit-learn. +!bc pycod +import numpy as np +import matplotlib.pyplot as plt +from sklearn.linear_model import LinearRegression +from sklearn.metrics import mean_squared_error, r2_score, mean_squared_log_error, mean_absolute_error + +x = np.random.rand(100,1) +y = 2.0+ 5*x+0.5*np.random.randn(100,1) +linreg = LinearRegression() +linreg.fit(x,y) +ypredict = linreg.predict(x) +print('The intercept alpha: \n', linreg.intercept_) +print('Coefficient beta : \n', linreg.coef_) +# The mean squared error +print("Mean squared error: %.2f" % mean_squared_error(y, ypredict)) +# Explained variance score: 1 is perfect prediction +print('Variance score: %.2f' % r2_score(y, ypredict)) +# Mean squared log error +print('Mean squared log error: %.2f' % mean_squared_log_error(y, ypredict) ) +# Mean absolute error +print('Mean absolute error: %.2f' % mean_absolute_error(y, ypredict)) +plt.plot(x, ypredict, "r-") +plt.plot(x, y ,'ro') +plt.axis([0.0,1.0,1.5, 7.0]) +plt.xlabel(r'$x$') +plt.ylabel(r'$y$') +plt.title(r'Linear Regression fit ') +plt.show() + +!ec +The function _coef_ gives us the parameter $\beta$ of our fit while _intercept_ yields +$\alpha$. Depending on the constant in front of the normal distribution, we get values near or far from $alpha =2$ and $\beta =5$. Try to play around with different parameters in front of the normal distribution. The function _meansquarederror_ gives us the mean square error, a risk metric corresponding to the expected value of the squared (quadratic) error or loss defined as +!bt +\[ MSE(\hat{y},\hat{\tilde{y}}) = \frac{1}{n} +\sum_{i=0}^{n-1}(y_i-\tilde{y}_i)^2, +\] +!et + +The smaller the value, the better the fit. Ideally we would like to +have an MSE equal zero. The attentive reader has probably recognized +this function as being similar to the $\chi^2$ function defined above. + +The _r2score_ function computes $R^2$, the coefficient of +determination. It provides a measure of how well future samples are +likely to be predicted by the model. Best possible score is 1.0 and it +can be negative (because the model can be arbitrarily worse). A +constant model that always predicts the expected value of $\hat{y}$, +disregarding the input features, would get a $R^2$ score of $0.0$. + +If $\tilde{\hat{y}}_i$ is the predicted value of the $i-th$ sample and $y_i$ is the corresponding true value, then the score $R^2$ is defined as +!bt +\[ +R^2(\hat{y}, \tilde{\hat{y}}) = 1 - \frac{\sum_{i=0}^{n - 1} (y_i - \tilde{y}_i)^2}{\sum_{i=0}^{n - 1} (y_i - \bar{y})^2}, +\] +!et +where we have defined the mean value of $\hat{y}$ as +!bt +\[ +\bar{y} = \frac{1}{n} \sum_{i=0}^{n - 1} y_i. +\] +!et +Another quantity will meet again in our discussions of regression analysis is + mean absolute error (MAE), a risk metric corresponding to the expected value of the absolute error loss or what we call the $l1$-norm loss. In our discussion above we presented the relative error. +The MAE is defined as follows +!bt +\[ +\text{MAE}(\hat{y}, \hat{\tilde{y}}) = \frac{1}{n} \sum_{i=0}^{n-1} \left| y_i - \tilde{y}_i \right|. +\] +!et +Finally we present the +squared logarithmic (quadratic) error +!bt +\[ +\text{MSLE}(\hat{y}, \hat{\tilde{y}}) = \frac{1}{n} \sum_{i=0}^{n - 1} (\log_e (1 + y_i) - \log_e (1 + \tilde{y}_i) )^2, +\] +!et + +where $\log_e (x)$ stands for the natural logarithm of $x$. This error +estimate is best to use when targets having exponential growth, such +as population counts, average sales of a commodity over a span of +years etc. + +We will discuss in more +detail these and other functions in the various lectures. We conclude this part with another example. Instead of +a linear $x$-dependence we study now a cubic polynomial and use the polynomial regression analysis tools of scikit-learn. + +!bc pycod +import matplotlib.pyplot as plt +import numpy as np +import random +from sklearn.linear_model import Ridge +from sklearn.preprocessing import PolynomialFeatures +from sklearn.pipeline import make_pipeline +from sklearn.linear_model import LinearRegression + +x=np.linspace(0.02,0.98,200) +noise = np.asarray(random.sample((range(200)),200)) +y=x**3*noise +yn=x**3*100 +poly3 = PolynomialFeatures(degree=3) +X = poly3.fit_transform(x[:,np.newaxis]) +clf3 = LinearRegression() +clf3.fit(X,y) + +Xplot=poly3.fit_transform(x[:,np.newaxis]) +poly3_plot=plt.plot(x, clf3.predict(Xplot), label='Cubic Fit') +plt.plot(x,yn, color='red', label="True Cubic") +plt.scatter(x, y, label='Data', color='orange', s=15) +plt.legend() +plt.show() + +def error(a): + for i in y: + err=(y-yn)/yn + return abs(np.sum(err))/len(err) + +print (error(y)) +!ec + +Similarly, using _R_, we can perform similar studies. The following _R_ code illustrates this. +(more details on _R_ will be inserted later). + +===== Non-Linear Least squares in R ===== +!bblock +!bc r +set.seed(1485) +len = 24 +x = runif(len) +y = x^3+rnorm(len, 0,0.06) +ds = data.frame(x = x, y = y) +str(ds) +plot( y ~ x, main ="Known cubic with noise") +s = seq(0,1,length =100) +lines(s, s^3, lty =2, col ="green") +m = nls(y ~ I(x^power), data = ds, start = list(power=1), trace = T) +class(m) +summary(m) +power = round(summary(m)$coefficients[1], 3) +power.se = round(summary(m)$coefficients[2], 3) +plot(y ~ x, main = "Fitted power model", sub = "Blue: fit; green: known") +s = seq(0, 1, length = 100) +lines(s, s^3, lty = 2, col = "green") +lines(s, predict(m, list(x = s)), lty = 1, col = "blue") +text(0, 0.5, paste("y =x^ (", power, " +/- ", power.se, ")", sep = ""), pos = 4) +!ec +!eblock + +In our lectures on regression analysis (and other ones as well), we will discuss in more details various _R_ functionalities. + + +Another useful Python package is +"pandas":"https://pandas.pydata.org/", which is an open source library +providing high-performance, easy-to-use data structures and data +analysis tools for Python. The following simple example shows how we can, in an easy way make tables of our data. Here we define a data set which includes names, city of residence and age, and displays the data in an easy to read way. We will see repeated use of _pandas_, in particular in connection with classification of data. + +!bc pycod +import pandas as pd +from IPython.display import display +data = {'Name': ["John", "Anna", "Peter", "Linda"], 'Location': ["Nairobi", "Napoli", "London", "Buenos Aires"], 'Age':[51, 21, 34, 45]} +data_pandas = pd.DataFrame(data) +display(data_pandas) +!ec + + + +===== Examples ===== + +We present here several examples, with pertinent Python codes that we +will use to illustrate various machine learning methods and ways to +analyze, from simple to complex, various data sets. Many of these +examples allow us to generate the data we want to analyze, following +much of the same philosophy we discussed above when +fitting various polynomials. + +We start with a simple exponential growth model that is meant to mimick an ecoli lab experiment. +We can easily model this system and then produce the data used to train various machine learning algorithms. +Another model from the life sciences is the so-called predator-prey model from ecology. Thereafter we present +a simple model for financial transactions before moving to a random walk model and ending with +the simulation of velocities of a non-interacting atom or molecule confined to move in a one-dimensional region. + + +=== Ecoli lab experiment === + + +A typical pattern seen in population models is that the population grows faster and faster. "Why? Is there an underlying (general) mechanism":"http://www.zo.utexas.edu/courses/Thoc/PopGrowth.html"? +Here we will construct a model for cell growth based on a simple difference equation for the growth. We make the following assumptions +!bblock + o Cells divide after $T$ seconds on average (one generation) + o $2N$ celles divide into twice as many new cells $\Delta N$ in a time + interval $\Delta t$ as $N$ cells would: $\Delta N \propto N$ + o $N$ cells result in twice as many new individuals $\Delta N$ in + time $2\Delta t$ as in time $\Delta t$: $\Delta N \propto\Delta t$ + o Same proportionality with respect to death + o Proposed model: $\Delta N = b\Delta t N - d\Delta tN$ for some unknown + constants $b$ (births) and $d$ (deaths) + o Describe evolution in discrete time: $t_n=n\Delta t$ + o Program-friendly notation: $N$ at $t_n$ is $N^n$ + o Math model: $N^{n+1} = N^n + r\Delta t\, N$ (with $\ r=b-d$) + o Program model: `N[n+1] = N[n] + r*dt*N[n]` +!eblock + +The difference equation can be programmed in a simple way, and in order to get started we +set $r=1.5$, $N^0=1$, $\Delta t=0.5$. The program reads + +!bc pycod +import numpy as np + +t = np.linspace(0, 10, 21) # 20 intervals in [0, 10] +dt = t[1] - t[0] +N = np.zeros(t.size) +N[0] = 1 +r = 0.5 + +for n in range(0, N.size-1, 1): + N[n+1] = N[n] + r*dt*N[n] + print('N[%d]=%.1f' % (n+1, N[n+1])) +!ec +and it generates the following output +!bc +N[1]=1.2 +N[2]=1.6 +N[3]=2.0 +N[4]=2.4 +N[5]=3.1 +N[6]=3.8 +N[7]=4.8 +N[8]=6.0 +N[9]=7.5 +N[10]=9.3 +N[11]=11.6 +N[12]=14.6 +N[13]=18.2 +N[14]=22.7 +N[15]=28.4 +N[16]=35.5 +N[17]=44.4 +N[18]=55.5 +N[19]=69.4 +N[20]=86.7 +!ec +This forms our data which later will define our training set. +In this case we defined the value of the parameter $r$. We could alternatively assume that we just received the +above data file and where asked to find $r$. How can we estimate $r$ from data? This will be one of our tasks later. + +We can use the difference equation with the experimental data +!bt +\[ N^{n+1} = N^n + r\Delta t N^n\] +!et +Suppose now that $N^{n+1}$ and $N^n$ are known from data. Then we could solve with respect to $r$ as follows +!bt +\[ r = \frac{N^{n+1}-N^n}{N^n\Delta t} \] +!et +Suppose we set $t_1=600$, $t_2=1200$, +$N^1=140$ and $N^2=250$. +The following code plots the data +!bc pycod +import numpy as np +import matplotlib.pyplot as plt + +# Estimate r +data = np.loadtxt('ecoli.csv', delimiter=',') +t_e = data[:,0] +N_e = data[:,1] +i = 2 # Data point (i,i+1) used to estimate r +r = (N_e[i+1] - N_e[i])/(N_e[i]*(t_e[i+1] - t_e[i])) +print('Estimated r=%.5f' % r) +# Can experiment with r values and see if the model can +# match the data better +T = 1200 # cell can divide after T sec +t_max = 5*T # 5 generations in experiment +t = np.linspace(0, t_max, 1000) +dt = t[1] - t[0] +N = np.zeros(t.size) + +N[0] = 100 +for n in range(0, len(t)-1, 1): + N[n+1] = N[n] + r*dt*N[n] + +plt.plot(t, N, 'r-', t_e, N_e, 'bo') +plt.xlabel('time [s]'); plt.ylabel('N') +plt.legend(['model', 'experiment'], loc='upper left') +plt.show() + +!ec +We can then change the parameter $r$ in the program and play around to make a better fit. By now we know that this +'search bythe eye' approach is not the most optimal one. + + +=== Predator-Prey model from ecology === + + +The population dynamics of a simple predator-prey system is a +classical example shown in many biology textbooks when ecological +systems are discussed. The system contains all elements of the +scientific method: + + * The set up of a specific hypothesis combined with + * the experimental methods needed (one can study existing data or perform experiments) + * analyzing and interpreting the data and performing further experiments if needed + * trying to extract general behaviors and extract eventual laws or patterns + * develop mathematical relations for the uncovered regularities/laws and test these by per forming new experiments + +Lots of data about populations of hares and lynx collected from furs in Hudson Bay, Canada, are available. It is known that the populations oscillate. Why? +Here we start by + + o plotting the data + o derive a simple model for the population dynamics + o (fitting parameters in the model to the data) + o using the model predict the evolution other predator-pray systems + +Most mammalian predators rely on a variety of prey, which complicates mathematical modeling; however, a few predators have become highly specialized and seek almost exclusively a single prey species. An example of this simplified predator-prey interaction is seen in Canadian northern forests, where the populations of the lynx and the snowshoe hare are intertwined in a life and death struggle. + +One reason that this particular system has been so extensively studied is that the Hudson Bay company kept careful records of all furs from the early 1800s into the 1900s. The records for the furs collected by the Hudson Bay company showed distinct oscillations (approximately 12 year periods), suggesting that these species caused almost periodic fluctuations of each other's populations. The table here shows data from 1900 to 1920. + + +|------------------------------------------------------| +| Year | Hares (x1000) | Lynx (x1000)| +|---------l-----------------------r--------------r------| +| 1900 | 30.0 | 4.0 | +| 1901 | 47.2 | 6.1 | +| 1902 | 70.2 | 9.8 | +| 1903 | 77.4 | 35.2 | +| 1904 | 36.3 | 59.4 | +| 1905 | 20.6 | 41.7 | +| 1906 | 18.1 | 19.0 | +| 1907 | 21.4 | 13.0 | +| 1908 | 22.0 | 8.3 | +| 1909 | 25.4 | 9.1 | +| 1910 | 27.1 | 7.4 | +| 1911 | 40.3 | 8.0 | +| 1912 | 57 | 12.3 | +| 1913 | 76.6 | 19.5 | +| 1914 | 52.3 | 45.7 | +| 1915 | 19.5 | 51.1 | +| 1916 | 11.2 | 29.7 | +| 1917 | 7.6 | 15.8 | +| 1918 | 14.6 | 9.7 | +| 1919 | 16.2 | 10.1 | +| 1920 | 24.7 | 8.6 | +|------------------------------------------------------| + + + +@@@CODE src/plot_Hudson.py + +FIGURE: [fig/Hudson_Bay_data, width=700 frac=0.9] + + +We see from the plot that there are indeed fluctuations. +We would like to create a mathematical model that explains these +population fluctuations. Ecologists have predicted that in a simple +predator-prey system that a rise in prey population is followed (with +a lag) by a rise in the predator population. When the predator +population is sufficiently high, then the prey population begins +dropping. After the prey population falls, then the predator +population falls, which allows the prey population to recover and +complete one cycle of this interaction. Thus, we see that +qualitatively oscillations occur. Can a mathematical model predict +this? What causes cycles to slow or speed up? What affects the +amplitude of the oscillation or do you expect to see the oscillations +damp to a stable equilibrium? The models tend to ignore factors like +climate and other complicating factors. How significant are these? + + * We see oscillations in the data + * What causes cycles to slow or speed up? + * What affects the amplitude of the oscillation or do you expect to see the oscillations damp to a stable equilibrium? + * With a model we can better *understand the data* + * More important: Can we understand the ecology dynamics of predator-pray populations? + +The classical way (in all books) is to present the Lotka-Volterra equations: + +!bt +\begin{align*} +\frac{dH}{dt} &= H(a - b L)\\ +\frac{dL}{dt} &= - L(d - c H) +\end{align*} +!et + +Here, + + * $H$ is the number of preys + * $L$ the number of predators + * $a$, $b$, $d$, $c$ are parameters + + +The population of hares evolves due to births and deaths exactly as a bacteria population: + +!bt +\[ +\Delta H = a \Delta t H^n +\] +!et +However, hares have an additional loss in the population because +they are eaten by lynx. +All the hares and lynx can form +$H\cdot L$ pairs in total. When such pairs meet during a time +interval $\Delta t$, there is some +small probablity that the lynx will eat the hare. +So in fraction $b\Delta t HL$, the lynx eat hares. This +loss of hares must be accounted for. Subtracted in the equation for hares: + +!bt +\[ \Delta H = a\Delta t H^n - b \Delta t H^nL^n\] +!et + +We assume that the primary growth for the lynx population depends on sufficient food for raising lynx kittens, which implies an adequate source of nutrients from predation on hares. Thus, the growth of the lynx population does not only depend of how many lynx there are, but on how many hares they can eat. +In a time interval $\Delta t HL$ hares and lynx can meet, and in a +fraction $b\Delta t HL$ the lynx eats the hare. All of this does not +contribute to the growth of lynx, again just a fraction of +$b\Delta t HL$ that we write as +$d\Delta t HL$. In addition, lynx die just as in the population +dynamics with one isolated animal population, leading to a loss +$-c\Delta t L$. +The accounting of lynx then looks like +!bt +\[ \Delta L = d\Delta t H^nL^n - c\Delta t L^n\] +!et + +By writing up the definition of $\Delta H$ and $\Delta L$, and putting +all assumed known terms $H^n$ and $L^n$ on the right-hand side, we have + +!bt +\[ H^{n+1} = H^n + a\Delta t H^n - b\Delta t H^n L^n \] +!et + +!bt +\[ L^{n+1} = L^n + d\Delta t H^nL^n - c\Delta t L^n \] +!et + +Note: + + * These equations are ready to be implemented! + * But to start, we need $H^0$ and $L^0$ (which we can get from the data) + * We also need values for $a$, $b$, $d$, $c$ + + * As always, models tend to be general - as here, applicable + to ``all'' predator-pray systems + * The critical issue is whether the *interaction* between hares and lynx + is sufficiently well modeled by $\hbox{const}HL$ + * The parameters $a$, $b$, $d$, and $c$ must be + estimated from data + +!bblock +@@@CODE src/Hudson_Bay.py +!eblock + +FIGURE: [fig/Hudson_Bay_sim, width=700 frac=0.9] + +We will later perform a least-square fitting. Then we can find optimal +values for the parameters $a$, $b$, $d$, $c$. In our calculations here +we set $a=0.4807$, $b=0.02482$, $d=0.9272$ and $c=0.02756$. These +parameters result in a slightly modified initial conditions, namely +$H(0) = 34.91$ and $L(0)=3.857$. + + +The following Python code demonstrates how we can use linear regression to fit for example the population of lynx. +Similarly, we have also used a decision tree algorithm to fit the lynx population data. As expected, the linear regression is not exactly impressive +!bc pycod +import numpy as np +import matplotlib.pyplot as plt +from IPython.display import display +import sklearn +from sklearn.linear_model import LinearRegression +from sklearn.tree import DecisionTreeRegressor + + +data = np.loadtxt('src/Hudson_Bay.csv', delimiter=',', skiprows=1) +x = data[:,0] +y = data[:,1] +line = np.linspace(1900,1920,1000,endpoint=False).reshape(-1,1) +reg = DecisionTreeRegressor(min_samples_split=3).fit(x.reshape(-1,1),y.reshape(-1,1)) +plt.plot(line, reg.predict(line), label="decision tree") +regline = LinearRegression().fit(x.reshape(-1,1),y.reshape(-1,1)) +plt.plot(line, regline.predict(line), label= "Linear Regression") +plt.plot(x, y, label= "Linear Regression") +plt.show() +!ec + + + +The similar code for linear regression in _R_ reads (more details to come) +!bc r +HudsonBay = read.csv("src/Hudson_Bay.csv",header=T) +fix(HudsonBay) +dim(HudsonBay) +names(HudsonBay) +plot(HudsonBay$Year, HudsonBay$Hares..x1000.) +attach(HudsonBay) +plot(Year, Hares..x1000.) +plot(Year, Hares..x1000., col="red", varwidth=T, xlab="Years", ylab="Haresx 1000") +summary(HudsonBay) +summary(Hares..x1000.) +library(MASS) +library(ISLR) +scatter.smooth(x=Year, y = Hares..x1000.) +linearMod = lm(Hares..x1000. ~ Year) +print(linearMod) +summary(linearMod) +plot(linearMod) +confint(linearMod) +predict(linearMod,data.frame(Year=c(1910,1914,1920)),interval="confidence") +!ec + +=== Simulating financial transactions === + +The aim here is to simulate financial transactions among financial agents +using Monte Carlo methods. The final goal is to extract a distribution of income as function +of the income $m$. From Pareto's work ("V.~Pareto, 1897":"http://www.institutcoppet.org/2012/05/08/cours-deconomie-politique-1896-de-vilfredo-pareto") it is known from empirical studies +that the higher end of the distribution of money follows a distribution +!bt +\[ +w_m\propto m^{-1-\alpha}, +\] +!et +with $\alpha\in [1,2]$. We will here follow the analysis made by "Patriarca and collaborators":"http://www.sciencedirect.com/science/article/pii/S0378437104004327". + +Here we will study numerically the relation between the micro-dynamic relations among financial +agents and the resulting macroscopic money distribution. + +We assume we have $N$ agents that exchange money in pairs $(i,j)$. We assume also that all agents +start with the same amount of money $m_0 > 0$. At a given 'time step', we choose randomly a pair +of agents $(i,j)$ and let a transaction take place. This means that agent $i$'s money $m_i$ changes +to $m_i'$ and similarly we have $m_j\rightarrow m_j'$. +Money is conserved during a transaction, meaning that +!bt +\begin{equation} + m_i+m_j=m_i'+m_j'. + label{eq:conserve} +\end{equation} +!et +The change is done via a random reassignement (a random number) $\epsilon$, meaning that + +!bt +\begin{equation*} +m_i' = \epsilon(m_i+m_j), +\end{equation*} +!et +leading to + +!bt +\begin{equation*} +m_j'= (1-\epsilon)(m_i+m_j). +\end{equation*} +!et +The number $\epsilon$ is extracted from a uniform distribution. +In this simple model, no agents are left with a debt, that is $m\ge 0$. +Due to the conservation law above, one can show that the system relaxes toward an equilibrium +state given by a Gibbs distribution + +!bt +\begin{equation*} +w_m=\beta \exp{(-\beta m)}, +\end{equation*} +!et +with + +!bt +\begin{equation*} +\beta = \frac{1}{\langle m\rangle}, +\end{equation*} +!et +and $\langle m\rangle=\sum_i m_i/N=m_0$, the average money. +It means that after equilibrium has been reached that the majority of agents is left with a small +number of money, while the number of richest agents, those with $m$ larger than a specific value $m'$, +exponentially decreases with $m'$. + +We assume that we have $N=500$ agents. In each simulation, we need a sufficiently large number of transactions, say $10^7$. Our aim is find the final equilibrium distribution $w_m$. In order to do that we would need +several runs of the above simulations, at least $10^3-10^4$ runs (experiments). + +Our task is to first set up an algorithm which simulates the above transactions with an initial + amount $m_0$. + The challenge here is to figure out a Monte Carlo simulation based on the + above equations. + You will in particular need to make an algorithm which sets up a histogram as function of $m$. + This histogram contains the number of times a value $m$ is registered and represents + $w_m\Delta m$. You will need to set up a value for the interval $\Delta m$ (typically $0.01-0.05$). + That means you need to account for the number of times you register an income in the interval + $m,m+\Delta m$. The number of times you register this income, represents the value that enters the histogram. + +!bc pycod +#!/usr/bin/env python +import numpy as np +import matplotlib.mlab as mlab +import matplotlib.pyplot as plt +import random + +# initialize the rng with a seed +random.seed() +# Hard coding of input parameters +Agents = 500 +MCcounts = 1000 +Transactions = 100000 +startMoney = 1.0 +Lambda = 0.0 +FinancialAgents = startMoney*np.ones(Agents) +for i in range (1, MCcounts, 1): + for j in range (1, Transactions, 1): + agent_i = int(Agents*random.random()) + agent_j = int(Agents*random.random()) + epsilon = random.random() + if agent_i != agent_j: + m1 = Lambda*FinancialAgents[agent_i] + (1-Lambda)*epsilon*(FinancialAgents[agent_i] + FinancialAgents[agent_j]) + m2 = Lambda*FinancialAgents[agent_j] + (1-Lambda)*(1-epsilon)*(FinancialAgents[agent_i] + FinancialAgents[agent_j]) + FinancialAgents[agent_i] = m1 + FinancialAgents[agent_j] = m2 + +# the histogram of the data +n, bins, patches = plt.hist(FinancialAgents, 50, facecolor='green') + +plt.xlabel('$x$') +plt.ylabel('Distribution of wealth') +plt.title(r'Money') +plt.axis([0, 10, 0, 500]) +plt.grid(True) +plt.show() + +!ec + + +We can then change our model to allow for a saving criterion, meaning that the agents save + a fraction $\lambda$ of the money they have before the transaction is made. The final distribution will then no longer be given by Gibbs distribution. It could also include a taxation on financial transactions. + + The conservation law of Eq. (ref{eq:conserve}) holds, but the money to be shared in a transaction between + agent $i$ and agent $j$ is now $(1-\lambda)(m_i+m_j)$. This means that we have + +!bt +\begin{equation*} + m_i' = \lambda m_i+\epsilon(1-\lambda)(m_i+m_j), + \end{equation*} +!et + and + +!bt +\begin{equation*} + m_j' = \lambda m_j+(1-\epsilon)(1-\lambda)(m_i+m_j), + \end{equation*} +!et + which can be written as + +!bt +\begin{equation*} + m_i'=m_i+\delta m + \end{equation*} +!et + and + +!bt +\begin{equation*} + m_j'=m_j-\delta m, + \end{equation*} +!et + with + +!bt +\begin{equation*} + \delta m=(1-\lambda)(\epsilon m_j-(1-\epsilon)m_i), + \end{equation*} +!et + showing how money is conserved during a transaction. + Select values of $\lambda =0.25,0.5$ and $\lambda=0.9$ and try to extract the corresponding + equilibrium distributions and compare these with the Gibbs distribution. We will use this model to +extract a parametrization of the above curves, see for example "Patriarca and collaborators":"http://www.sciencedirect.com/science/article/pii/S0378437104004327". + + +=== Particle in one dimension and velocity distribution === +!bc pycod +# Program to test the Metropolis algorithm with one particle at given temp in one dimension +import numpy as np +import matplotlib.mlab as mlab +import matplotlib.pyplot as plt +import random +from math import sqrt, exp, log +# initialize the rng with a seed +random.seed() +# Hard coding of input parameters +MCcycles = 100000 +Temperature = 2.0 +beta = 1./Temperature +InitialVelocity = -2.0 +CurrentVelocity = InitialVelocity +Energy = 0.5*InitialVelocity*InitialVelocity +VelocityRange = 10*sqrt(Temperature) +VelocityStep = 2*VelocityRange/10. +AverageEnergy = Energy +AverageEnergy2 = Energy*Energy +VelocityValues = np.zeros(MCcycles) +# The Monte Carlo sampling with Metropolis starts here +for i in range (1, MCcycles, 1): + TrialVelocity = CurrentVelocity + (2.0*random.random() - 1.0)*VelocityStep + EnergyChange = 0.5*(TrialVelocity*TrialVelocity -CurrentVelocity*CurrentVelocity); + if random.random() <= exp(-beta*EnergyChange): + CurrentVelocity = TrialVelocity + Energy += EnergyChange + VelocityValues[i] = CurrentVelocity + AverageEnergy += Energy + AverageEnergy2 += Energy*Energy +#Final averages +AverageEnergy = AverageEnergy/MCcycles +AverageEnergy2 = AverageEnergy2/MCcycles +Variance = AverageEnergy2 - AverageEnergy*AverageEnergy +print(AverageEnergy, Variance) +n, bins, patches = plt.hist(VelocityValues, 400, facecolor='green') + +plt.xlabel('$v$') +plt.ylabel('Velocity distribution P(v)') +plt.title(r'Velocity histogram at $k_BT=2$') +plt.axis([-5, 5, 0, 600]) +plt.grid(True) +plt.show() + +!ec + + + + +=== Random walk model === +!bc pycod +import numpy as np +import matplotlib.pyplot as plt +from sklearn.preprocessing import PolynomialFeatures +from sklearn.linear_model import LinearRegression + +steps=250 + +distance=0 +x=0 +distance_list=[] +steps_list=[] +while x j$ + * Lower triangular if $a_{ij}=0$ for $i < j$ + * Upper Hessenberg if $a_{ij}=0$ for $i > j+1$ + * Lower Hessenberg if $a_{ij}=0$ for $i < j+1$ + * Tridiagonal if $a_{ij}=0$ for $|i -j| > 1$ + * Lower banded with bandwidth $p$: $a_{ij}=0$ for $i > j+p$ + * Upper banded with bandwidth $p$: $a_{ij}=0$ for $i < j+p$ + * Banded, block upper triangular, block lower triangular.... + +!split +===== Basic Matrix Features ===== + +!bblock Some Equivalent Statements +For an $N\times N$ matrix $\mathbf{A}$ the following properties are all equivalent + + * If the inverse of $\mathbf{A}$ exists, $\mathbf{A}$ is nonsingular. + * The equation $\mathbf{Ax}=0$ implies $\mathbf{x}=0$. + * The rows of $\mathbf{A}$ form a basis of $R^N$. + * The columns of $\mathbf{A}$ form a basis of $R^N$. + * $\mathbf{A}$ is a product of elementary matrices. + * $0$ is not eigenvalue of $\mathbf{A}$. +!eblock + +!split +===== Numpy and arrays ===== +"Numpy":"http://www.numpy.org/" provides an easy way to handle arrays in Python. The standard way to import this library is as +!bc pycod +import numpy as np +n = 10 +x = np.random.normal(size=n) +print(x) +!ec +Here we have defined a vector $x$ with $n=10$ elements with its values given by the Normal distribution $N(0,1)$. +Another alternative is to declare a vector as follows +!bc pycod +import numpy as np +x = np.array([1, 2, 3]) +print(x) +!ec +Here we have defined a vector with three elements, with $x_0=1$, $x_1=2$ and $x_2=3$. Note that both Python and C++ +start numbering array elements from $0$ and on. This means that a vector with $n$ elements has a sequence of entities $x_0, x_1, x_2, \dots, x_{n-1}$. We could also let (recommended) Numpy to compute the logarithms of a specific array as +!bc pycod +import numpy as np +x = np.log(np.array([4, 7, 8])) +print(x) +!ec + +Here we have used Numpy's unary function $np.log$. This function is +highly tuned to compute array elements since the code is vectorized +and does not require looping. We normaly recommend that you use the +Numpy intrinsic functions instead of the corresponding _log_ function +from Python's _math_ module. The looping is done explicitely by the +_np.log_ function. The alternative, and slower way to compute the +logarithms of a vector would be to write + +!bc pycod +import numpy as np +from math import log +x = np.array([4, 7, 8]) +for i in range(0, len(x)): + x[i] = log(x[i]) +print(x) +!ec +We note that our code is much longer already and we need to import the _log_ function from the _math_ module. +The attentive reader will also notice that the output is $[1, 1, 2]$. Python interprets automacally our numbers as integers (like the _automatic_ keyword in C++). To change this we could define our array elements to be double precision numbers as +!bc pycod +import numpy as np +x = np.log(np.array([4, 7, 8], dtype = np.float64)) +print(x) +!ec +or simply write them as double precision numbers (Python uses 64 bits as default for floating point type variables), that is +!bc pycod +import numpy as np +x = np.log(np.array([4.0, 7.0, 8.0]) +print(x) +!ec +To check the number of bytes (remember that one byte contains eight bits for double precision variables), you can use simple use the _itemsize_ functionality (the array $x$ is actually an object which inherits the functionalities defined in Numpy) as +!bc pycod +import numpy as np +x = np.log(np.array([4.0, 7.0, 8.0]) +print(x.itemsize) +!ec + +!split +===== Matrices in Python ===== +Having defined vectors, we are now ready to try out matrices. We can define a $3 \times 3 $ real matrix $\hat{A}$ +as (recall that we user lowercase letters for vectors and uppercase letters for matrices) +!bc pycod +import numpy as np +A = np.log(np.array([ [4.0, 7.0, 8.0], [3.0, 10.0, 11.0], [4.0, 5.0, 7.0] ])) +print(A) +!ec +If we use the _shape_ function we would get $(3, 3)$ as output, that is verifying that our matrix is a $3\times 3$ matrix. We can slice the matrix and print for example the first column (Python organized matrix elements in a row-major order, see below) as +!bc pycod +import numpy as np +A = np.log(np.array([ [4.0, 7.0, 8.0], [3.0, 10.0, 11.0], [4.0, 5.0, 7.0] ])) +# print the first column, row-major order and elements start with 0 +print(A[:,0]) +!ec +We can continue this was by printing out other columns or rows. The example here prints out the second column +!bc pycod +import numpy as np +A = np.log(np.array([ [4.0, 7.0, 8.0], [3.0, 10.0, 11.0], [4.0, 5.0, 7.0] ])) +# print the first column, row-major order and elements start with 0 +print(A[1,:]) +!ec +Numpy contains many other functionalities that allow us to slice, subdivide etc etc arrays. We strongly recommend that you look up the "Numpy website for more details":"http://www.numpy.org/". Useful functions when defining a matrix are the _np.zeros_ function which declares a matrix of a given dimension and sets all elements to zero +!bc pycod +import numpy as np +n = 10 +# define a matrix of dimension 10 x 10 and set all elements to zero +A = np.zeros( (n, n) ) +print(A) +!ec +or initializing all elements to +!bc pycod +import numpy as np +n = 10 +# define a matrix of dimension 10 x 10 and set all elements to one +A = np.ones( (n, n) ) +print(A) +!ec +or as unitarily distributed random numbers (see the material on random number generators in the statistics part) +!bc pycod +import numpy as np +n = 10 +# define a matrix of dimension 10 x 10 and set all elements to random numbers with x \in [0, 1] +A = np.random.rand(n, n) +print(A) +!ec + +As we will see throughout these lectures, there are several extremely useful functionalities in Numpy. +As an example, consider the discussion of the covariance matrix. Suppose we have defined three vectors +$\hat{x}, \hat{y}, \hat{z}$ with $n$ elements each. The covariance matrix is defined as +!bt +\[ +\hat{\Sigma} = \begin{bmatrix} \sigma_{xx} & \sigma_{xy} & \sigma_{xz} \\ + \sigma_{yx} & \sigma_{yy} & \sigma_{yz} \\ + \sigma_{zx} & \sigma_{zy} & \sigma_{zz} + \end{bmatrix}, +\] +!et +where for example +!bt +\[ +\sigma_{xy} =\frac{1}{n} \sum_{i=0}^{n-1}(x_i- \overline{x})(y_i- \overline{y}). +\] +!et +The Numpy function _np.cov_ calculates the covariance elements using the factor $1/(n-1)$ instead of $1/n$ since it assumes we do not have the exact mean values. For a more in-depth discussion of the covariance and covariance matrix and its meaning, we refer you to the lectures on statistics. +The following simple function uses the _np.vstack_ function which takes each vector of dimension $1\times n$ and produces a $ 3\times n$ matrix $\hat{W}$ +!bt +\[ +\hat{W} = \begin{bmatrix} x_0 & y_0 & z_0 \\ + x_1 & y_1 & z_1 \\ + x_2 & y_2 & z_2 \\ + \dots & \dots & \dots \\ + x_{n-2} & y_{n-2} & z_{n-2} \\ + x_{n-1} & y_{n-1} & z_{n-1} + \end{bmatrix}, +\] +!et + +which in turn is converted into into the $3 times 3$ covariance matrix +$\hat{\Sigma}$ via the Numpy function _np.cov()_. In our review of +statistical functions and quantities we will discuss more about the +meaning of the covariance matrix. Here we note that we can calculate +the mean value of each set of samples $\hat{x}$ etc using the Numpy +function _np.mean(x)_. We can also extract the eigenvalues of the +covariance matrix through the _np.linalg.eig()_ function. + +!bc pycod +# Importing various packages +import numpy as np + +n = 100 +x = np.random.normal(size=n) +print(np.mean(x)) +y = 4+3*x+np.random.normal(size=n) +print(np.mean(y)) +z = x**3+np.random.normal(size=n) +print(np.mean(z)) +W = np.vstack((x, y, z)) +Sigma = np.cov(W) +print(Sigma) +Eigvals, Eigvecs = np.linalg.eig(Sigma) +print(Eigvals) +!ec + + +!bc pycod +import numpy as np +import matplotlib.pyplot as plt +from scipy import sparse +eye = np.eye(4) +print(eye) +sparse_mtx = sparse.csr_matrix(eye) +print(sparse_mtx) +x = np.linspace(-10,10,100) +y = np.sin(x) +plt.plot(x,y,marker='x') +plt.show() +!ec + + + + +!split +===== Matrix Handling in C/C++, Static and Dynamical allocation ===== + +!bblock Static +We have an $N\times N$ matrix A with $N=100$ +In C/C++ this would be defined as + +!bc cppcod + int N = 100; + double A[100][100]; + // initialize all elements to zero + for(i=0 ; i < N ; i++) { + for(j=0 ; j < N ; j++) { + A[i][j] = 0.0; + +!ec +Note the way the matrix is organized, row-major order. +!eblock + +!split +===== Matrix Handling in C/C++ ===== + +!bblock Row Major Order, Addition +We have $N\times N$ matrices A, B and C and we wish to +evaluate $A=B+C$. + +!bt +\[ +\mathbf{A}= \mathbf{B}\pm\mathbf{C} \Longrightarrow a_{ij} = b_{ij}\pm c_{ij}, +\] +!et +In C/C++ this would be coded like + +!bc cppcod + for(i=0 ; i < N ; i++) { + for(j=0 ; j < N ; j++) { + a[i][j] = b[i][j]+c[i][j] + +!ec +!eblock + +!split +===== Matrix Handling in C/C++ ===== + +!bblock Row Major Order, Multiplication +We have $N\times N$ matrices A, B and C and we wish to +evaluate $A=BC$. + +!bt +\[ +\mathbf{A}=\mathbf{BC} \Longrightarrow a_{ij} = \sum_{k=1}^{n} b_{ik}c_{kj}, +\] +!et +In C/C++ this would be coded like + +!bc cppcod + for(i=0 ; i < N ; i++) { + for(j=0 ; j < N ; j++) { + for(k=0 ; k < N ; k++) { + a[i][j]+=b[i][k]*c[k][j]; + +!ec +!eblock + + +!split +===== Dynamic memory allocation in C/C++ ===== + +At least three possibilities in this course + + * Do it yourself + * Use the functions provided in the library package lib.cpp + * Use Armadillo URL: "http://arma.sourceforgenet" (a C++ linear algebra library, discussion both here and at lab). + +!split +===== Matrix Handling in C/C++, Dynamic Allocation ===== + +!bblock Do it yourself +!bc cppcod +int N; +double ** A; +A = new double*[N] +for ( i = 0; i < N; i++) + A[i] = new double[N]; +!ec +Always free space when you don't need an array anymore. + +!bc cppcod +for ( i = 0; i < N; i++) + delete[] A[i]; +delete[] A; +!ec +!eblock + +!split +===== Armadillo, recommended!! ===== + + * Armadillo is a C++ linear algebra library (matrix maths) aiming towards a good balance between speed and ease of use. The syntax is deliberately similar to Matlab. + * Integer, floating point and complex numbers are supported, as well as a subset of trigonometric and statistics functions. Various matrix decompositions are provided through optional integration with LAPACK, or one of its high performance drop-in replacements (such as the multi-threaded MKL or ACML libraries). + * A delayed evaluation approach is employed (at compile-time) to combine several operations into one and reduce (or eliminate) the need for temporaries. This is accomplished through recursive templates and template meta-programming. + * Useful for conversion of research code into production environments, or if C++ has been decided as the language of choice, due to speed and/or integration capabilities. + * The library is open-source software, and is distributed under a license that is useful in both open-source and commercial/proprietary contexts. + +!split +===== Armadillo, simple examples ===== + +!bc cppcod +#include +#include + +using namespace std; +using namespace arma; + +int main(int argc, char** argv) + { + mat A = randu(5,5); + mat B = randu(5,5); + + cout << A*B << endl; + + return 0; + +!ec + +!split +===== Armadillo, how to compile and install ===== + +For people using Ubuntu, Debian, Linux Mint, simply go to the synaptic package manager and install +armadillo from there. +You may have to install Lapack as well. +For Mac and Windows users, follow the instructions from the webpage +URL: "http://arma.sourceforge.net". +To compile, use for example (linux/ubuntu) + +!bc cppcod +c++ -O2 -o program.x program.cpp -larmadillo -llapack -lblas +!ec +where the `-l` option indicates the library you wish to link to. + +For OS X users you may have to declare the paths to the include files and the libraries as +!bc cppcod +c++ -O2 -o program.x program.cpp -L/usr/local/lib -I/usr/local/include -larmadillo -llapack -lblas +!ec + +!split +===== Armadillo, simple examples ===== + +!bc cppcod +#include +#include "armadillo" +using namespace arma; +using namespace std; + +int main(int argc, char** argv) + { + // directly specify the matrix size (elements are uninitialised) + mat A(2,3); + // .n_rows = number of rows (read only) + // .n_cols = number of columns (read only) + cout << "A.n_rows = " << A.n_rows << endl; + cout << "A.n_cols = " << A.n_cols << endl; + // directly access an element (indexing starts at 0) + A(1,2) = 456.0; + A.print("A:"); + // scalars are treated as a 1x1 matrix, + // hence the code below will set A to have a size of 1x1 + A = 5.0; + A.print("A:"); + // if you want a matrix with all elements set to a particular value + // the .fill() member function can be used + A.set_size(3,3); + A.fill(5.0); A.print("A:"); +!ec + +!split +===== Armadillo, simple examples ===== + +!bc cppcod + mat B; + + // endr indicates "end of row" + B << 0.555950 << 0.274690 << 0.540605 << 0.798938 << endr + << 0.108929 << 0.830123 << 0.891726 << 0.895283 << endr + << 0.948014 << 0.973234 << 0.216504 << 0.883152 << endr + << 0.023787 << 0.675382 << 0.231751 << 0.450332 << endr; + + // print to the cout stream + // with an optional string before the contents of the matrix + B.print("B:"); + + // the << operator can also be used to print the matrix + // to an arbitrary stream (cout in this case) + cout << "B:" << endl << B << endl; + // save to disk + B.save("B.txt", raw_ascii); + // load from disk + mat C; + C.load("B.txt"); + C += 2.0 * B; + C.print("C:"); +!ec + +!split +===== Armadillo, simple examples ===== + +!bc cppcod + // submatrix types: + // + // .submat(first_row, first_column, last_row, last_column) + // .row(row_number) + // .col(column_number) + // .cols(first_column, last_column) + // .rows(first_row, last_row) + + cout << "C.submat(0,0,3,1) =" << endl; + cout << C.submat(0,0,3,1) << endl; + + // generate the identity matrix + mat D = eye(4,4); + + D.submat(0,0,3,1) = C.cols(1,2); + D.print("D:"); + + // transpose + cout << "trans(B) =" << endl; + cout << trans(B) << endl; + + // maximum from each column (traverse along rows) + cout << "max(B) =" << endl; + cout << max(B) << endl; + +!ec + +!split +===== Armadillo, simple examples ===== + +!bc cppcod + // maximum from each row (traverse along columns) + cout << "max(B,1) =" << endl; + cout << max(B,1) << endl; + // maximum value in B + cout << "max(max(B)) = " << max(max(B)) << endl; + // sum of each column (traverse along rows) + cout << "sum(B) =" << endl; + cout << sum(B) << endl; + // sum of each row (traverse along columns) + cout << "sum(B,1) =" << endl; + cout << sum(B,1) << endl; + // sum of all elements + cout << "sum(sum(B)) = " << sum(sum(B)) << endl; + cout << "accu(B) = " << accu(B) << endl; + // trace = sum along diagonal + cout << "trace(B) = " << trace(B) << endl; + // random matrix -- values are uniformly distributed in the [0,1] interval + mat E = randu(4,4); + E.print("E:"); + +!ec + +!split +===== Armadillo, simple examples ===== + +!bc cppcod + // row vectors are treated like a matrix with one row + rowvec r; + r << 0.59499 << 0.88807 << 0.88532 << 0.19968; + r.print("r:"); + + // column vectors are treated like a matrix with one column + colvec q; + q << 0.81114 << 0.06256 << 0.95989 << 0.73628; + q.print("q:"); + + // dot or inner product + cout << "as_scalar(r*q) = " << as_scalar(r*q) << endl; + + // outer product + cout << "q*r =" << endl; + cout << q*r << endl; + + + // sum of three matrices (no temporary matrices are created) + mat F = B + C + D; + F.print("F:"); + + return 0; + +!ec + +!split +===== Armadillo, simple examples ===== + +!bc cppcod +#include +#include "armadillo" +using namespace arma; +using namespace std; + +int main(int argc, char** argv) + { + cout << "Armadillo version: " << arma_version::as_string() << endl; + + mat A; + + A << 0.165300 << 0.454037 << 0.995795 << 0.124098 << 0.047084 << endr + << 0.688782 << 0.036549 << 0.552848 << 0.937664 << 0.866401 << endr + << 0.348740 << 0.479388 << 0.506228 << 0.145673 << 0.491547 << endr + << 0.148678 << 0.682258 << 0.571154 << 0.874724 << 0.444632 << endr + << 0.245726 << 0.595218 << 0.409327 << 0.367827 << 0.385736 << endr; + + A.print("A ="); + + // determinant + cout << "det(A) = " << det(A) << endl; +!ec + +!split +===== Armadillo, simple examples ===== + +!bc cppcod + // inverse + cout << "inv(A) = " << endl << inv(A) << endl; + double k = 1.23; + + mat B = randu(5,5); + mat C = randu(5,5); + + rowvec r = randu(5); + colvec q = randu(5); + + + // examples of some expressions + // for which optimised implementations exist + // optimised implementation of a trinary expression + // that results in a scalar + cout << "as_scalar( r*inv(diagmat(B))*q ) = "; + cout << as_scalar( r*inv(diagmat(B))*q ) << endl; + + // example of an expression which is optimised + // as a call to the dgemm() function in BLAS: + cout << "k*trans(B)*C = " << endl << k*trans(B)*C; + + return 0; + +!ec + +!split +===== Gaussian Elimination ===== + +We start with the linear set of equations + +!bt +\[ + \mathbf{A}\mathbf{x} = \mathbf{w}. +\] +!et +We assume also that the matrix $\mathbf{A}$ is non-singular and that the +matrix elements along the diagonal satisfy $a_{ii} \ne 0$. Simple $4\times 4 $ example + +!bt +\[ +\begin{bmatrix} + a_{11}& a_{12} &a_{13}& a_{14}\\ + a_{21}& a_{22} &a_{23}& a_{24}\\ + a_{31}& a_{32} &a_{33}& a_{34}\\ + a_{41}& a_{42} &a_{43}& a_{44}\\ + \end{bmatrix} \begin{bmatrix} + x_1\\ + x_2\\ + x_3 \\ + x_4 \\ + \end{bmatrix} + =\begin{bmatrix} + w_1\\ + w_2\\ + w_3 \\ + w_4\\ + \end{bmatrix}. +\] +!et + +!split +===== Gaussian Elimination ===== +or + +!bt +\begin{align} + a_{11}x_1 +a_{12}x_2 +a_{13}x_3 + a_{14}x_4=&w_1 \nonumber \\ +a_{21}x_1 + a_{22}x_2 + a_{23}x_3 + a_{24}x_4=&w_2 \nonumber \\ +a_{31}x_1 + a_{32}x_2 + a_{33}x_3 + a_{34}x_4=&w_3 \nonumber \\ +a_{41}x_1 + a_{42}x_2 + a_{43}x_3 + a_{44}x_4=&w_4. \nonumber +\end{align} +!et + +!split +===== Gaussian Elimination ===== + +The basic idea of Gaussian elimination is to use the first equation to eliminate the first unknown $x_1$ +from the remaining $n-1$ equations. Then we use the new second equation to eliminate the second unknown +$x_2$ from the remaining $n-2$ equations. With $n-1$ such eliminations +we obtain a so-called upper triangular set of equations of the form + +!bt +\begin{align} + b_{11}x_1 +b_{12}x_2 +b_{13}x_3 + b_{14}x_4=&y_1 \nonumber \\ + b_{22}x_2 + b_{23}x_3 + b_{24}x_4=&y_2 \nonumber \\ +b_{33}x_3 + b_{34}x_4=&y_3 \nonumber \\ +b_{44}x_4=&y_4. \nonumber +label{eq:gaussbacksub} +\end{align} +!et +We can solve this system of equations recursively starting from $x_n$ (in our case $x_4$) and proceed with +what is called a backward substitution. + +!split +===== Gaussian Elimination ===== +This process can be expressed mathematically as + +!bt +\begin{equation} + x_m = \frac{1}{b_{mm}}\left(y_m-\sum_{k=m+1}^nb_{mk}x_k\right)\quad m=n-1,n-2,\dots,1. +\end{equation} +!et +To arrive at such an upper triangular system of equations, we start by eliminating +the unknown $x_1$ for $j=2,n$. We achieve this by multiplying the first equation by $a_{j1}/a_{11}$ and then subtract +the result from the $j$th equation. We assume obviously that $a_{11}\ne 0$ and that +$\mathbf{A}$ is not singular. + +!split +===== Gaussian Elimination ===== + +Our actual $4\times 4$ example reads after the first operation + +!bt +\[ +\begin{bmatrix} + a_{11}& a_{12} &a_{13}& a_{14}\\ + 0& (a_{22}-\frac{a_{21}a_{12}}{a_{11}}) &(a_{23}-\frac{a_{21}a_{13}}{a_{11}}) & (a_{24}-\frac{a_{21}a_{14}}{a_{11}})\\ +0& (a_{32}-\frac{a_{31}a_{12}}{a_{11}})& (a_{33}-\frac{a_{31}a_{13}}{a_{11}})& (a_{34}-\frac{a_{31}a_{14}}{a_{11}})\\ +0&(a_{42}-\frac{a_{41}a_{12}}{a_{11}}) &(a_{43}-\frac{a_{41}a_{13}}{a_{11}}) & (a_{44}-\frac{a_{41}a_{14}}{a_{11}}) \\ + \end{bmatrix} \begin{bmatrix} + x_1\\ + x_2\\ + x_3 \\ + x_4 \\ + \end{bmatrix} + =\begin{bmatrix} + y_1\\ + w_2^{(2)}\\ + w_3^{(2)} \\ + w_4^{(2)}\\ + \end{bmatrix}, +\] +!et +or + +!bt +\begin{align} + b_{11}x_1 +b_{12}x_2 +b_{13}x_3 + b_{14}x_4=&y_1 \nonumber \\ + a^{(2)}_{22}x_2 + a^{(2)}_{23}x_3 + a^{(2)}_{24}x_4=&w^{(2)}_2 \nonumber \\ + a^{(2)}_{32}x_2 + a^{(2)}_{33}x_3 + a^{(2)}_{34}x_4=&w^{(2)}_3 \nonumber \\ + a^{(2)}_{42}x_2 + a^{(2)}_{43}x_3 + a^{(2)}_{44}x_4=&w^{(2)}_4, \nonumber \\ +\end{align} +!et + +!split +===== Gaussian Elimination ===== + +The new coefficients are + +!bt +\begin{equation} + b_{1k} = a_{1k}^{(1)} \quad k=1,\dots,n, +\end{equation} +!et +where each $a_{1k}^{(1)}$ is equal to the original $a_{1k}$ element. The other coefficients are + +!bt +\begin{equation} +a_{jk}^{(2)} = a_{jk}^{(1)}-\frac{a_{j1}^{(1)}a_{1k}^{(1)}}{a_{11}^{(1)}} \quad j,k=2,\dots,n, +\end{equation} +!et +with a new right-hand side given by + +!bt +\begin{equation} +y_{1}=w_1^{(1)}, \quad w_j^{(2)} =w_j^{(1)}-\frac{a_{j1}^{(1)}w_1^{(1)}}{a_{11}^{(1)}} \quad j=2,\dots,n. +\end{equation} +!et +We have also set $w_1^{(1)}=w_1$, the original vector element. +We see that the system of unknowns $x_1,\dots,x_n$ is transformed into an $(n-1)\times (n-1)$ problem. + +!split +===== Gaussian Elimination ===== + +This step is called forward substitution. +Proceeding with these substitutions, we obtain the +general expressions for the new coefficients + +!bt +\begin{equation} + a_{jk}^{(m+1)} = a_{jk}^{(m)}-\frac{a_{jm}^{(m)}a_{mk}^{(m)}}{a_{mm}^{(m)}} \quad j,k=m+1,\dots,n, +\end{equation} +!et +with $m=1,\dots,n-1$ and a +right-hand side given by + +!bt +\begin{equation} + w_j^{(m+1)} =w_j^{(m)}-\frac{a_{jm}^{(m)}w_m^{(m)}}{a_{mm}^{(m)}}\quad j=m+1,\dots,n. +\end{equation} +!et +This set of $n-1$ elimations leads us to an equations which is solved by back substitution. +If the arithmetics is exact and the matrix $\mathbf{A}$ is not singular, then the computed answer will be exact. + +Even though the matrix elements along the diagonal are not zero, +numerically small numbers may appear and subsequent divisions may lead to large numbers, which, if added +to a small number may yield losses of precision. Suppose for example that our first division in $(a_{22}-a_{21}a_{12}/a_{11})$ +results in $-10^{-7}$ and that $a_{22}$ is one. +one. We are then +adding $10^7+1$. With single precision this results in $10^7$. + + + +!split +===== Linear Algebra Methods ===== + + * Gaussian elimination, $O(2/3n^3)$ flops, general matrix + * LU decomposition, upper triangular and lower tridiagonal matrices, $O(2/3n^3)$ flops, general matrix. Get easily the inverse, determinant and can solve linear equations with back-substitution only, $O(n^2)$ flops + * Cholesky decomposition. Real symmetric or hermitian positive definite matrix, $O(1/3n^3)$ flops. + * Tridiagonal linear systems, important for differential equations. Normally positive definite and non-singular. $O(8n)$ flops for symmetric. Special case of banded matrices. + * Singular value decomposition + * the QR method will be discussed in chapter 7 in connection with eigenvalue systems. $O(4/3n^3)$ flops. + +!split +===== LU Decomposition ===== + +The LU decomposition method means that we can rewrite +this matrix as the product of two matrices $\mathbf{L}$ and $\mathbf{U}$ +where + +!bt +\[ + \begin{bmatrix} + a_{11} & a_{12} & a_{13} & a_{14} \\ + a_{21} & a_{22} & a_{23} & a_{24} \\ + a_{31} & a_{32} & a_{33} & a_{34} \\ + a_{41} & a_{42} & a_{43} & a_{44} + \end{bmatrix} + = \begin{bmatrix} + 1 & 0 & 0 & 0 \\ + l_{21} & 1 & 0 & 0 \\ + l_{31} & l_{32} & 1 & 0 \\ + l_{41} & l_{42} & l_{43} & 1 + \end{bmatrix} + \begin{bmatrix} + u_{11} & u_{12} & u_{13} & u_{14} \\ + 0 & u_{22} & u_{23} & u_{24} \\ + 0 & 0 & u_{33} & u_{34} \\ + 0 & 0 & 0 & u_{44} + \end{bmatrix}. +\] +!et + +!split +===== LU Decomposition ===== + +LU decomposition forms the backbone of other algorithms in linear algebra, such as the +solution of linear equations given by + +!bt +\begin{align} + a_{11}x_1 +a_{12}x_2 +a_{13}x_3 + a_{14}x_4=&w_1 \nonumber \\ +a_{21}x_1 + a_{22}x_2 + a_{23}x_3 + a_{24}x_4=&w_2 \nonumber \\ +a_{31}x_1 + a_{32}x_2 + a_{33}x_3 + a_{34}x_4=&w_3 \nonumber \\ +a_{41}x_1 + a_{42}x_2 + a_{43}x_3 + a_{44}x_4=&w_4. \nonumber +\end{align} +!et +The above set of equations is conveniently solved by using LU decomposition as an intermediate step. + +The matrix $\mathbf{A}\in \mathbb{R}^{n\times n}$ has an LU factorization if the determinant +is different from zero. If the LU factorization exists and $\mathbf{A}$ is non-singular, then the LU factorization +is unique and the determinant is given by + +!bt +\[ +det\{\mathbf{A}\}=det\{\mathbf{LU}\}= det\{\mathbf{L}\}det\{\mathbf{U}\}=u_{11}u_{22}\dots u_{nn}. +\] +!et + +!split +===== LU Decomposition, why? ===== + +There are at least three main advantages with LU decomposition compared with standard Gaussian elimination: + + * It is straightforward to compute the determinant of a matrix + * If we have to solve sets of linear equations with the same matrix but with different vectors $\mathbf{y}$, the number of FLOPS is of the order $n^3$. + * The inverse is such an operation + +!split +===== LU Decomposition, linear equations ===== + +With the LU decomposition it is rather +simple to solve a system of linear equations + +!bt +\begin{align} + a_{11}x_1 +a_{12}x_2 +a_{13}x_3 + a_{14}x_4=&w_1 \nonumber \\ +a_{21}x_1 + a_{22}x_2 + a_{23}x_3 + a_{24}x_4=&w_2 \nonumber \\ +a_{31}x_1 + a_{32}x_2 + a_{33}x_3 + a_{34}x_4=&w_3 \nonumber \\ +a_{41}x_1 + a_{42}x_2 + a_{43}x_3 + a_{44}x_4=&w_4. \nonumber +\end{align} +!et + +This can be written in matrix form as + +!bt +\[ \mathbf{Ax}=\mathbf{w}. \] +!et + +where $\mathbf{A}$ and $\mathbf{w}$ are known and we have to solve for +$\mathbf{x}$. Using the LU dcomposition we write + +!bt +\[ \mathbf{A} \mathbf{x} \equiv \mathbf{L} \mathbf{U} \mathbf{x} =\mathbf{w}. \] +!et + +!split +===== LU Decomposition, linear equations ===== + +The previous equation can be calculated in two steps + +!bt +\[ \mathbf{L} \mathbf{y} = \mathbf{w};\qquad \mathbf{Ux}=\mathbf{y}. \] +!et + +To show that this is correct we use to the LU decomposition +to rewrite our system of linear equations as + +!bt +\[ \mathbf{LUx}=\mathbf{w}, \] +!et +and since the determinant of $\mathbf{L}$ is equal to 1 (by construction +since the diagonals of $\mathbf{L}$ equal 1) we can use the inverse of +$\mathbf{L}$ to obtain + +!bt +\[ + \mathbf{Ux}=\mathbf{L^{-1}w}=\mathbf{y}, +\] +!et +which yields the intermediate step + +!bt +\[ + \mathbf{L^{-1}w}=\mathbf{y} +\] +!et +and as soon as we have $\mathbf{y}$ we can obtain $\mathbf{x}$ +through $\mathbf{Ux}=\mathbf{y}$. + +!split +===== LU Decomposition, why? ===== + +For our four-dimentional example this takes the form + +!bt +\begin{align} + y_1=&w_1 \nonumber\\ +l_{21}y_1 + y_2=&w_2\nonumber \\ +l_{31}y_1 + l_{32}y_2 + y_3 =&w_3\nonumber \\ +l_{41}y_1 + l_{42}y_2 + l_{43}y_3 + y_4=&w_4. \nonumber +\end{align} +!et + +and + +!bt +\begin{align} + u_{11}x_1 +u_{12}x_2 +u_{13}x_3 + u_{14}x_4=&y_1 \nonumber\\ +u_{22}x_2 + u_{23}x_3 + u_{24}x_4=&y_2\nonumber \\ +u_{33}x_3 + u_{34}x_4=&y_3\nonumber \\ +u_{44}x_4=&y_4 \nonumber +\end{align} +!et + +This example shows the basis for the algorithm +needed to solve the set of $n$ linear equations. + +!split +===== LU Decomposition, linear equations ===== + +The algorithm goes as follows + + * Set up the matrix $\bf A$ and the vector $\bf w$ with their correct dimensions. This determines the dimensionality of the unknown vector $\bf x$. + * Then LU decompose the matrix $\bf A$ through a call to the function `ludcmp(double a, int n, int indx, double &d)`. This functions returns the LU decomposed matrix $\bf A$, its determinant and the vector indx which keeps track of the number of interchanges of rows. If the determinant is zero, the solution is malconditioned. + * Thereafter you call the function `lubksb(double a, int n, int indx, double w)` which uses the LU decomposed matrix $\bf A$ and the vector $\bf w$ and returns $\bf x$ in the same place as $\bf w$. Upon exit the original content in $\bf w$ is destroyed. If you wish to keep this information, you should make a backup of it in your calling function. + +!split +===== LU Decomposition, the inverse of a matrix ===== + +If the inverse exists then + +!bt +\[ + \mathbf{A}^{-1}\mathbf{A}=\mathbf{I}, +\] +!et +the identity matrix. With an LU decomposed matrix we can rewrite the last equation as + +!bt +\[ + \mathbf{LU}\mathbf{A}^{-1}=\mathbf{I}. +\] +!et + +!split +===== LU Decomposition, the inverse of a matrix ===== + +If we assume that the first column (that is column 1) of the inverse matrix +can be written as a vector with unknown entries + +!bt +\[ + \mathbf{A}_1^{-1}= \begin{bmatrix} + + a_{11}^{-1} \\ + a_{21}^{-1} \\ + \dots \\ + a_{n1}^{-1} \\ + \end{bmatrix}, +\] +!et +then we have a linear set of equations + +!bt +\[ + \mathbf{LU}\begin{bmatrix} + + a_{11}^{-1} \\ + a_{21}^{-1} \\ + \dots \\ + a_{n1}^{-1} \\ + \end{bmatrix} =\begin{bmatrix} + 1 \\ + 0 \\ + \dots \\ + 0 \\ + \end{bmatrix}. +\] +!et + +!split +===== LU Decomposition, the inverse ===== + +In a similar way we can compute the unknow entries of the second column, + +!bt +\[ + \mathbf{LU}\begin{bmatrix} + + a_{12}^{-1} \\ + a_{22}^{-1} \\ + \dots \\ + a_{n2}^{-1} \\ + \end{bmatrix}=\begin{bmatrix} + 0 \\ + 1 \\ + \dots \\ + 0 \\ + \end{bmatrix}, +\] +!et +and continue till we have solved all $n$ sets of linear equations. + + +!split +===== "Using Armadillo to perform an LU decomposition":"https://github.com/CompPhysics/ComputationalPhysicsMSU/blob/master/doc/Programs/CppQtCodesLectures/MatrixTest/main.cpp" ===== +!bc cppcod +#include +#include "armadillo" +using namespace arma; +using namespace std; + +int main() + { + mat A = randu(5,5); + vec b = randu(5); + + A.print("A ="); + b.print("b="); + // solve Ax = b + vec x = solve(A,b); + // print x + x.print("x="); + // find LU decomp of A, if needed, P is the permutation matrix + mat L, U; + lu(L,U,A); + // print l + L.print(" L= "); + // print U + U.print(" U= "); + //Check that A = LU + (A-L*U).print("Test of LU decomposition"); + return 0; + } +!ec + + +======= Review of Statistics ======= + + +===== Domains and probabilities ===== +!bblock +Consider the following simple example, namely the tossing of two dice, resulting in the following possible values +!bt +\begin{equation*} +\{2,3,4,5,6,7,8,9,10,11,12\}. +\end{equation*} +!et +These values are called the *domain*. +To this domain we have the corresponding *probabilities* +!bt +\begin{equation*} +\{1/36,2/36/,3/36,4/36,5/36,6/36,5/36,4/36,3/36,2/36,1/36\}. +\end{equation*} +!et +!eblock + + +===== Tossing the dice ===== +!bblock +The numbers in the domain are the outcomes of the physical process of tossing say two dice. +We cannot tell beforehand whether the outcome is 3 or 5 or any other number in this domain. +This defines the randomness of the outcome, or unexpectedness or any other synonimous word which +encompasses the uncertitude of the final outcome. + +The only thing we can tell beforehand +is that say the outcome 2 has a certain probability. +If our favorite hobby is to spend an hour every evening throwing dice and +registering the sequence of outcomes, we will note that the numbers in the above domain +!bt +\begin{equation*} +\{2,3,4,5,6,7,8,9,10,11,12\}, +\end{equation*} +!et +appear in a random order. After 11 throws the results may look like + +!bt +\begin{equation*} +\{10,8,6,3,6,9,11,8,12,4,5\}. +\end{equation*} +!et +!eblock + + +===== Stochastic variables ===== +!bblock + +_Random variables are characterized by a domain which contains all possible values that the random value may take. This domain has a corresponding probability distribution function(PDF)_. +!eblock + + +===== Stochastic variables and the main concepts, the discrete case ===== +!bblock +There are two main concepts associated with a stochastic variable. The +*domain* is the set $\mathbb D = \{x\}$ of all accessible values +the variable can assume, so that $X \in \mathbb D$. An example of a +discrete domain is the set of six different numbers that we may get by +throwing of a dice, $x\in\{1,\,2,\,3,\,4,\,5,\,6\}$. + +The *probability distribution function (PDF)* is a function +$p(x)$ on the domain which, in the discrete case, gives us the +probability or relative frequency with which these values of $X$ +occur +!bt +\begin{equation*} +p(x) = \mathrm{Prob}(X=x). +\end{equation*} +!et +!eblock + + + +===== Stochastic variables and the main concepts, the continuous case ===== +!bblock +In the continuous case, the PDF does not directly depict the +actual probability. Instead we define the probability for the +stochastic variable to assume any value on an infinitesimal interval +around $x$ to be $p(x)dx$. The continuous function $p(x)$ then gives us +the *density* of the probability rather than the probability +itself. The probability for a stochastic variable to assume any value +on a non-infinitesimal interval $[a,\,b]$ is then just the integral + +!bt +\begin{equation*} +\mathrm{Prob}(a\leq X\leq b) = \int_a^b p(x)dx. +\end{equation*} +!et +Qualitatively speaking, a stochastic variable represents the values of +numbers chosen as if by chance from some specified PDF so that the +selection of a large set of these numbers reproduces this PDF. +!eblock + + +===== The cumulative probability ===== +!bblock +Of interest to us is the *cumulative probability +distribution function* (_CDF_), $P(x)$, which is just the probability +for a stochastic variable $X$ to assume any value less than $x$ +!bt +\begin{equation*} +P(x)=\mathrm{Prob(}X\leq x\mathrm{)} = +\int_{-\infty}^x p(x^{\prime})dx^{\prime}. +\end{equation*} +!et +The relation between a CDF and its corresponding PDF is then + +!bt +\begin{equation*} +p(x) = \frac{d}{dx}P(x). +\end{equation*} +!et +!eblock + + +===== Properties of PDFs ===== +!bblock + +There are two properties that all PDFs must satisfy. The first one is +positivity (assuming that the PDF is normalized) + +!bt +\begin{equation*} +0 \leq p(x) \leq 1. +\end{equation*} +!et +Naturally, it would be nonsensical for any of the values of the domain +to occur with a probability greater than $1$ or less than $0$. Also, +the PDF must be normalized. That is, all the probabilities must add up +to unity. The probability of ``anything'' to happen is always unity. For +both discrete and continuous PDFs, this condition is +!bt +\begin{align*} +\sum_{x_i\in\mathbb D} p(x_i) & = 1,\\ +\int_{x\in\mathbb D} p(x)\,dx & = 1. +\end{align*} +!et +!eblock + + +===== Important distributions, the uniform distribution ===== +!bblock +The first one +is the most basic PDF; namely the uniform distribution +!bt +\begin{equation} +p(x) = \frac{1}{b-a}\theta(x-a)\theta(b-x). +label{eq:unifromPDF} +\end{equation} +!et +For $a=0$ and $b=1$ we have +!bt +\[ +\begin{array}{ll} +p(x)dx = dx & \in [0,1]. +\end{array} +\] +!et +The latter distribution is used to generate random numbers. For other PDFs, one needs normally a mapping from this distribution to say for example the exponential distribution. +!eblock + + +===== Gaussian distribution ===== +!bblock +The second one is the Gaussian Distribution +!bt +\begin{equation*} +p(x) = \frac{1}{\sigma\sqrt{2\pi}} \exp{(-\frac{(x-\mu)^2}{2\sigma^2})}, +\end{equation*} +!et +with mean value $\mu$ and standard deviation $\sigma$. If $\mu=0$ and $\sigma=1$, it is normally called the _standard normal distribution_ +!bt +\begin{equation*} +p(x) = \frac{1}{\sqrt{2\pi}} \exp{(-\frac{x^2}{2})}, +\end{equation*} +!et + +The following simple Python code plots the above distribution for different values of $\mu$ and $\sigma$. +!bc pyscpro +import numpy as np +from math import acos, exp, sqrt +from matplotlib import pyplot as plt +from matplotlib import rc, rcParams +import matplotlib.units as units +import matplotlib.ticker as ticker +rc('text',usetex=True) +rc('font',**{'family':'serif','serif':['Gaussian distribution']}) +font = {'family' : 'serif', + 'color' : 'darkred', + 'weight' : 'normal', + 'size' : 16, + } +pi = acos(-1.0) +mu0 = 0.0 +sigma0 = 1.0 +mu1= 1.0 +sigma1 = 2.0 +mu2 = 2.0 +sigma2 = 4.0 + +x = np.linspace(-20.0, 20.0) +v0 = np.exp(-(x*x-2*x*mu0+mu0*mu0)/(2*sigma0*sigma0))/sqrt(2*pi*sigma0*sigma0) +v1 = np.exp(-(x*x-2*x*mu1+mu1*mu1)/(2*sigma1*sigma1))/sqrt(2*pi*sigma1*sigma1) +v2 = np.exp(-(x*x-2*x*mu2+mu2*mu2)/(2*sigma2*sigma2))/sqrt(2*pi*sigma2*sigma2) +plt.plot(x, v0, 'b-', x, v1, 'r-', x, v2, 'g-') +plt.title(r'{\bf Gaussian distributions}', fontsize=20) +plt.text(-19, 0.3, r'Parameters: $\mu = 0$, $\sigma = 1$', fontdict=font) +plt.text(-19, 0.18, r'Parameters: $\mu = 1$, $\sigma = 2$', fontdict=font) +plt.text(-19, 0.08, r'Parameters: $\mu = 2$, $\sigma = 4$', fontdict=font) +plt.xlabel(r'$x$',fontsize=20) +plt.ylabel(r'$p(x)$ [MeV]',fontsize=20) + +# Tweak spacing to prevent clipping of ylabel +plt.subplots_adjust(left=0.15) +plt.savefig('gaussian.pdf', format='pdf') +plt.show() +!ec +!eblock + + + +===== Exponential distribution ===== +!bblock +Another important distribution in science is the exponential distribution +!bt +\begin{equation*} +p(x) = \alpha\exp{-(\alpha x)}. +\end{equation*} +!et +!eblock + + +===== Expectation values ===== +!bblock +Let $h(x)$ be an arbitrary continuous function on the domain of the stochastic +variable $X$ whose PDF is $p(x)$. We define the *expectation value* +of $h$ with respect to $p$ as follows + +!bt +\begin{equation} +\langle h \rangle_X \equiv \int\! h(x)p(x)\,dx +label{eq:expectation_value_of_h_wrt_p} +\end{equation} +!et +Whenever the PDF is known implicitly, like in this case, we will drop +the index $X$ for clarity. +A particularly useful class of special expectation values are the +*moments*. The $n$-th moment of the PDF $p$ is defined as +follows +!bt +\begin{equation*} +\langle x^n \rangle \equiv \int\! x^n p(x)\,dx +\end{equation*} +!et +!eblock + + +===== Stochastic variables and the main concepts, mean values ===== +!bblock +The zero-th moment $\langle 1\rangle$ is just the normalization condition of +$p$. The first moment, $\langle x\rangle$, is called the *mean* of $p$ +and often denoted by the letter $\mu$ +!bt +\begin{equation*} +\langle x\rangle = \mu \equiv \int x p(x)dx, +\end{equation*} +!et +for a continuous distribution and +!bt +\begin{equation*} +\langle x\rangle = \mu \equiv \sum_{i=1}^N x_i p(x_i), +\end{equation*} +!et +for a discrete distribution. +Qualitatively it represents the centroid or the average value of the +PDF and is therefore simply called the expectation value of $p(x)$. +!eblock + + +===== Stochastic variables and the main concepts, central moments, the variance ===== +!bblock + +A special version of the moments is the set of *central moments*, the n-th central moment defined as +!bt +\begin{equation*} +\langle (x-\langle x\rangle )^n\rangle \equiv \int\! (x-\langle x\rangle)^n p(x)\,dx +\end{equation*} +!et +The zero-th and first central moments are both trivial, equal $1$ and +$0$, respectively. But the second central moment, known as the +*variance* of $p$, is of particular interest. For the stochastic +variable $X$, the variance is denoted as $\sigma^2_X$ or $\mathrm{Var}(X)$ +!bt +\begin{align*} +\sigma^2_X &=\mathrm{Var}(X) = \langle (x-\langle x\rangle)^2\rangle = +\int (x-\langle x\rangle)^2 p(x)dx\\ +& = \int\left(x^2 - 2 x \langle x\rangle^{2} +\langle x\rangle^2\right)p(x)dx\\ +& = \langle x^2\rangle - 2 \langle x\rangle\langle x\rangle + \langle x\rangle^2\\ +& = \langle x^2 \rangle - \langle x\rangle^2 +\end{align*} +!et +The square root of the variance, $\sigma =\sqrt{\langle (x-\langle x\rangle)^2\rangle}$ is called the +_standard deviation_ of $p$. It is the RMS (root-mean-square) +value of the deviation of the PDF from its mean value, interpreted +qualitatively as the ``spread'' of $p$ around its mean. +!eblock + + + + +===== Probability Distribution Functions ===== +!bblock + +The following table collects properties of probability distribution functions. +In our notation we reserve the label $p(x)$ for the probability of a certain event, +while $P(x)$ is the cumulative probability. + + +|--------------------------------------------------------------------------------------------------------------------------------------| +| | Discrete PDF | Continuous PDF | +|---------------------l-------------------------------------------c-------------------------------------------c------------------------| +| Domain | $\left\{x_1, x_2, x_3, \dots, x_N\right\}$ | $[a,b]$ | +| Probability | $p(x_i)$ | $p(x)dx$ | +| Cumulative | $P_i=\sum_{l=1}^ip(x_l)$ | $P(x)=\int_a^xp(t)dt$ | +| Positivity | $0 \le p(x_i) \le 1$ | $p(x) \ge 0$ | +| Positivity | $0 \le P_i \le 1$ | $0 \le P(x) \le 1$ | +| Monotonic | $P_i \ge P_j$ if $x_i \ge x_j$ | $P(x_i) \ge P(x_j)$ if $x_i \ge x_j$ | +| Normalization | $P_N=1$ | $P(b)=1$ | +|--------------------------------------------------------------------------------------------------------------------------------------| + +!eblock + + + +===== Probability Distribution Functions ===== +!bblock +With a PDF we can compute expectation values of selected quantities such as + +!bt +\begin{equation*} + \langle x^k\rangle=\sum_{i=1}^{N}x_i^kp(x_i), +\end{equation*} +!et +if we have a discrete PDF or + +!bt +\begin{equation*} + \langle x^k\rangle=\int_a^b x^kp(x)dx, +\end{equation*} +!et +in the case of a continuous PDF. We have already defined the mean value $\mu$ +and the variance $\sigma^2$. +!eblock + + +===== The three famous Probability Distribution Functions ===== +!bblock + +There are at least three PDFs which one may encounter. These are the + +_Uniform distribution_ +!bt +\begin{equation*} +p(x)=\frac{1}{b-a}\Theta(x-a)\Theta(b-x), +\end{equation*} +!et +yielding probabilities different from zero in the interval $[a,b]$. + +_The exponential distribution_ +!bt +\begin{equation*} +p(x)=\alpha \exp{(-\alpha x)}, +\end{equation*} +!et +yielding probabilities different from zero in the interval $[0,\infty)$ and with mean value +!bt +\begin{equation*} +\mu = \int_0^{\infty}xp(x)dx=\int_0^{\infty}x\alpha \exp{(-\alpha x)}dx=\frac{1}{\alpha}, +\end{equation*} +!et +!eblock +with variance +!bt +\begin{equation*} +\sigma^2=\int_0^{\infty}x^2p(x)dx-\mu^2 = \frac{1}{\alpha^2}. +\end{equation*} +!et + + +===== Probability Distribution Functions, the normal distribution ===== +!bblock +Finally, we have the so-called univariate normal distribution, or just the _normal distribution_ +!bt +\begin{equation*} +p(x)=\frac{1}{b\sqrt{2\pi}}\exp{\left(-\frac{(x-a)^2}{2b^2}\right)} +\end{equation*} +!et +with probabilities different from zero in the interval $(-\infty,\infty)$. +The integral $\int_{-\infty}^{\infty}\exp{\left(-(x^2\right)}dx$ appears in many calculations, its value +is $\sqrt{\pi}$, a result we will need when we compute the mean value and the variance. +The mean value is +!bt +\begin{equation*} + \mu = \int_0^{\infty}xp(x)dx=\frac{1}{b\sqrt{2\pi}}\int_{-\infty}^{\infty}x \exp{\left(-\frac{(x-a)^2}{2b^2}\right)}dx, +\end{equation*} +!et +which becomes with a suitable change of variables +!bt +\begin{equation*} + \mu =\frac{1}{b\sqrt{2\pi}}\int_{-\infty}^{\infty}b\sqrt{2}(a+b\sqrt{2}y)\exp{-y^2}dy=a. +\end{equation*} +!et +!eblock + + +===== Probability Distribution Functions, the normal distribution ===== +!bblock +Similarly, the variance becomes +!bt +\begin{equation*} + \sigma^2 = \frac{1}{b\sqrt{2\pi}}\int_{-\infty}^{\infty}(x-\mu)^2 \exp{\left(-\frac{(x-a)^2}{2b^2}\right)}dx, +\end{equation*} +!et +and inserting the mean value and performing a variable change we obtain + +!bt +\begin{equation*} + \sigma^2 = \frac{1}{b\sqrt{2\pi}}\int_{-\infty}^{\infty}b\sqrt{2}(b\sqrt{2}y)^2\exp{\left(-y^2\right)}dy= +\frac{2b^2}{\sqrt{\pi}}\int_{-\infty}^{\infty}y^2\exp{\left(-y^2\right)}dy, +\end{equation*} +!et +and performing a final integration by parts we obtain the well-known result $\sigma^2=b^2$. +It is useful to introduce the standard normal distribution as well, defined by $\mu=a=0$, viz. a distribution +centered around zero and with a variance $\sigma^2=1$, leading to + +!bt +\begin{equation} + p(x)=\frac{1}{\sqrt{2\pi}}\exp{\left(-\frac{x^2}{2}\right)}. +\end{equation} +!et +!eblock + + +===== Probability Distribution Functions, the cumulative distribution ===== +!bblock + +The exponential and uniform distributions have simple cumulative functions, +whereas the normal distribution does not, being proportional to the so-called +error function $erf(x)$, given by + +!bt +\begin{equation*} +P(x) = \frac{1}{\sqrt{2\pi}}\int_{-\infty}^x\exp{\left(-\frac{t^2}{2}\right)}dt, +\end{equation*} +!et +which is difficult to evaluate in a quick way. +!eblock + + + +===== Probability Distribution Functions, other important distribution ===== +!bblock + +Some other PDFs which one encounters often in the natural sciences are the binomial distribution +!bt +\begin{equation*} + p(x) = \left(\begin{array}{c} n \\ x\end{array}\right)y^x(1-y)^{n-x} \hspace{0.5cm}x=0,1,\dots,n, +\end{equation*} +!et +where $y$ is the probability for a specific event, such as the tossing of a coin or moving left or right +in case of a random walker. Note that $x$ is a discrete stochastic variable. + +The sequence of binomial trials is characterized by the following definitions + + * Every experiment is thought to consist of $N$ independent trials. + + * In every independent trial one registers if a specific situation happens or not, such as the jump to the left or right of a random walker. + + * The probability for every outcome in a single trial has the same value, for example the outcome of tossing (either heads or tails) a coin is always $1/2$. +!eblock + + +===== Probability Distribution Functions, the binomial distribution ===== +!bblock + +In order to compute the mean and variance we need to recall Newton's binomial +formula +!bt +\begin{equation*} + (a+b)^m=\sum_{n=0}^m \left(\begin{array}{c} m \\ n\end{array}\right)a^nb^{m-n}, +\end{equation*} +!et +which can be used to show that + +!bt +\begin{equation*} +\sum_{x=0}^n\left(\begin{array}{c} n \\ x\end{array}\right)y^x(1-y)^{n-x} = (y+1-y)^n = 1, +\end{equation*} +!et +the PDF is normalized to one. +The mean value is +!bt +\begin{equation*} +\mu = \sum_{x=0}^n x\left(\begin{array}{c} n \\ x\end{array}\right)y^x(1-y)^{n-x} = +\sum_{x=0}^n x\frac{n!}{x!(n-x)!}y^x(1-y)^{n-x}, +\end{equation*} +!et +resulting in +!bt +\begin{equation*} +\mu = +\sum_{x=0}^n x\frac{(n-1)!}{(x-1)!(n-1-(x-1))!}y^{x-1}(1-y)^{n-1-(x-1)}, +\end{equation*} +!et +which we rewrite as + +!bt +\begin{equation*} +\mu=ny\sum_{\nu=0}^n\left(\begin{array}{c} n-1 \\ \nu\end{array}\right)y^{\nu}(1-y)^{n-1-\nu} =ny(y+1-y)^{n-1}=ny. +\end{equation*} +!et +!eblock +The variance is slightly trickier to get. It reads $\sigma^2=ny(1-y)$. + + +===== Probability Distribution Functions, Poisson's distribution ===== +!bblock + +Another important distribution with discrete stochastic variables $x$ is +the Poisson model, which resembles the exponential distribution and reads +!bt +\begin{equation*} + p(x) = \frac{\lambda^x}{x!} e^{-\lambda} \hspace{0.5cm}x=0,1,\dots,;\lambda > 0. +\end{equation*} +!et +In this case both the mean value and the variance are easier to calculate, + +!bt +\begin{equation*} +\mu = \sum_{x=0}^{\infty} x \frac{\lambda^x}{x!} e^{-\lambda} = \lambda e^{-\lambda}\sum_{x=1}^{\infty} +\frac{\lambda^{x-1}}{(x-1)!}=\lambda, +\end{equation*} +!et +and the variance is $\sigma^2=\lambda$. +!eblock + + + + +===== Probability Distribution Functions, Poisson's distribution ===== +!bblock +An example of applications of the Poisson distribution could be the counting +of the number of $\alpha$-particles emitted from a radioactive source in a given time interval. +In the limit of $n\rightarrow \infty$ and for small probabilities $y$, the binomial distribution +approaches the Poisson distribution. Setting $\lambda = ny$, with $y$ the probability for an event in +the binomial distribution we can show that + +!bt +\begin{equation*} +\lim_{n\rightarrow \infty}\left(\begin{array}{c} n \\ x\end{array}\right)y^x(1-y)^{n-x} e^{-\lambda}=\sum_{x=1}^{\infty}\frac{\lambda^x}{x!} e^{-\lambda}. +\end{equation*} +!et +!eblock + + + +===== Meet the covariance! ===== +!bblock +An important quantity in a statistical analysis is the so-called covariance. + +Consider the set $\{X_i\}$ of $n$ +stochastic variables (not necessarily uncorrelated) with the +multivariate PDF $P(x_1,\dots,x_n)$. The *covariance* of two +of the stochastic variables, $X_i$ and $X_j$, is defined as follows + +!bt +\begin{align} +\mathrm{Cov}(X_i,\,X_j) & = \langle (x_i-\langle x_i\rangle)(x_j-\langle x_j\rangle)\rangle \\ +&=\int\cdots\int (x_i-\langle x_i\rangle)(x_j-\langle x_j\rangle)P(x_1,\dots,x_n)\,dx_1\dots dx_n, +label{eq:def_covariance} +\end{align} +!et +with +!bt +\begin{equation*} +\langle x_i\rangle = +\int\cdots\int x_i P(x_1,\dots,x_n)\,dx_1\dots dx_n. +\end{equation*} +!et +!eblock + + + + +===== Meet the covariance in matrix disguise ===== +!bblock +If we consider the above covariance as a matrix +!bt +\[ +C_{ij} =\mathrm{Cov}(X_i,\,X_j), +\] +!et +then the diagonal elements are just the familiar +variances, $C_{ii} = \mathrm{Cov}(X_i,\,X_i) = \mathrm{Var}(X_i)$. It turns out that +all the off-diagonal elements are zero if the stochastic variables are +uncorrelated. +!eblock + + +===== Covariance ===== +!bc pycod +# Importing various packages +from math import exp, sqrt +from random import random, seed +import numpy as np +import matplotlib.pyplot as plt + +def covariance(x, y, n): + sum = 0.0 + mean_x = np.mean(x) + mean_y = np.mean(y) + for i in range(0, n): + sum += (x[(i)]-mean_x)*(y[i]-mean_y) + return sum/n + +n = 10 + +x=np.random.normal(size=n) +y = 4+3*x+np.random.normal(size=n) +covxy = covariance(x,y,n) +print(covxy) +z = np.vstack((x, y)) +c = np.cov(z.T) + +print(c) + +!ec + + + + +===== Meet the covariance, uncorrelated events ===== +!bblock + +Consider the stochastic variables $X_i$ and $X_j$, ($i\neq j$). We have +!bt +\begin{align*} +Cov(X_i,\,X_j) &= \langle (x_i-\langle x_i\rangle)(x_j-\langle x_j\rangle)\rangle\\ +&=\langle x_i x_j - x_i\langle x_j\rangle - \langle x_i\rangle x_j + \langle x_i\rangle\langle x_j\rangle\rangle\\ +&=\langle x_i x_j\rangle - \langle x_i\langle x_j\rangle\rangle - \langle \langle x_i\rangle x_j \rangle + +\langle \langle x_i\rangle\langle x_j\rangle\rangle \\ +&=\langle x_i x_j\rangle - \langle x_i\rangle\langle x_j\rangle - \langle x_i\rangle\langle x_j\rangle + +\langle x_i\rangle\langle x_j\rangle \\ +&=\langle x_i x_j\rangle - \langle x_i\rangle\langle x_j\rangle +\end{align*} +!et +If $X_i$ and $X_j$ are independent (assuming $i \neq j$), we have that +!bt +\[ +\langle x_i x_j\rangle = \langle x_i\rangle\langle x_j\rangle, +\] +!et +leading to +!bt +\[ +Cov(X_i, X_j) = 0 \hspace{0.1cm} (i\neq j). +\] +!et +!eblock + + + + +===== Numerical experiments and the covariance ===== +!bblock + +Now that we have constructed an idealized mathematical framework, let +us try to apply it to empirical observations. Examples of relevant +physical phenomena may be spontaneous decays of nuclei, or a purely +mathematical set of numbers produced by some deterministic +mechanism. It is the latter we will deal with, using so-called pseudo-random +number generators. In general our observations will contain only a limited set of +observables. We remind the reader that +a *stochastic process* is a process that produces sequentially a +chain of values +!bt +\begin{equation*} +\{x_1, x_2,\dots\,x_k,\dots\}. +\end{equation*} +!et +!eblock + + + +===== Numerical experiments and the covariance ===== +!bblock +We will call these +values our *measurements* and the entire set as our measured +*sample*. The action of measuring all the elements of a sample +we will call a stochastic *experiment* (since, operationally, +they are often associated with results of empirical observation of +some physical or mathematical phenomena; precisely an experiment). We +assume that these values are distributed according to some +PDF $p_X^{\phantom X}(x)$, where $X$ is just the formal symbol for the +stochastic variable whose PDF is $p_X^{\phantom X}(x)$. Instead of +trying to determine the full distribution $p$ we are often only +interested in finding the few lowest moments, like the mean +$\mu_X^{\phantom X}$ and the variance $\sigma_X^{\phantom X}$. +!eblock + + + +===== Numerical experiments and the covariance, actual situations ===== +!bblock +In practical situations however, a sample is always of finite size. Let that +size be $n$. The expectation value of a sample $\alpha$, the _sample mean_, is then defined as follows +!bt +\begin{equation*} +\langle x_{\alpha} \rangle \equiv \frac{1}{n}\sum_{k=1}^n x_{\alpha,k}. +\end{equation*} +!et +The *sample variance* is: +!bt +\begin{equation*} +\mathrm{Var}(x) \equiv \frac{1}{n}\sum_{k=1}^n (x_{\alpha,k} - \langle x_{\alpha} \rangle)^2, +\end{equation*} +!et +with its square root being the *standard deviation of the sample*. +!eblock + + + +===== Numerical experiments and the covariance, our observables ===== +!bblock +You can think of the above observables as a set of quantities which define +a given experiment. This experiment is then repeated several times, say $m$ times. +The total average is then +!bt +\begin{equation} +\langle X_m \rangle= \frac{1}{m}\sum_{\alpha=1}^mx_{\alpha}=\frac{1}{mn}\sum_{\alpha, k} x_{\alpha,k}, +label{eq:exptmean} +\end{equation} +!et +where the last sums end at $m$ and $n$. +The total variance is +!bt +\begin{equation*} +\sigma^2_m= \frac{1}{mn^2}\sum_{\alpha=1}^m(\langle x_{\alpha} \rangle-\langle X_m \rangle)^2, +\end{equation*} +!et +which we rewrite as +!bt +\begin{equation} +\sigma^2_m=\frac{1}{m}\sum_{\alpha=1}^m\sum_{kl=1}^n (x_{\alpha,k}-\langle X_m \rangle)(x_{\alpha,l}-\langle X_m \rangle). +label{eq:exptvariance} +\end{equation} +!et +!eblock + + +===== Numerical experiments and the covariance, the sample variance ===== +!bblock + +We define also the sample variance $\sigma^2$ of all $mn$ individual experiments as +!bt +\begin{equation} +\sigma^2=\frac{1}{mn}\sum_{\alpha=1}^m\sum_{k=1}^n (x_{\alpha,k}-\langle X_m \rangle)^2. +label{eq:sampleexptvariance} +\end{equation} +!et + + + +These quantities, being known experimental values or the results from our calculations, +may differ, in some cases +significantly, from the similarly named +exact values for the mean value $\mu_X$, the variance $\mathrm{Var}(X)$ +and the covariance $\mathrm{Cov}(X,Y)$. +!eblock + + +===== Numerical experiments and the covariance, central limit theorem ===== +!bblock + +The central limit theorem states that the PDF $\tilde{p}(z)$ of +the average of $m$ random values corresponding to a PDF $p(x)$ +is a normal distribution whose mean is the +mean value of the PDF $p(x)$ and whose variance is the variance +of the PDF $p(x)$ divided by $m$, the number of values used to compute $z$. + +The central limit theorem leads then to the well-known expression for the +standard deviation, given by +!bt +\begin{equation*} + \sigma_m= +\frac{\sigma}{\sqrt{m}}. +\end{equation*} +!et + +In many cases the above estimate for the standard deviation, in particular if correlations are strong, may be too simplistic. We need therefore a more precise defintion of the error and the variance in our results. +!eblock + + +===== Definition of Correlation Functions and Standard Deviation ===== +!bblock +Our estimate of the true average $\mu_{X}$ is the sample mean $\langle X_m \rangle$ + +!bt +\begin{equation*} +\mu_{X}^{\phantom X} \approx X_m=\frac{1}{mn}\sum_{\alpha=1}^m\sum_{k=1}^n x_{\alpha,k}. +\end{equation*} +!et + + +We can then use Eq. (ref{eq:exptvariance}) +!bt +\begin{equation*} +\sigma^2_m=\frac{1}{mn^2}\sum_{\alpha=1}^m\sum_{kl=1}^n (x_{\alpha,k}-\langle X_m \rangle)(x_{\alpha,l}-\langle X_m \rangle), +\end{equation*} +!et +and rewrite it as +!bt +\begin{equation*} +\sigma^2_m=\frac{\sigma^2}{n}+\frac{2}{mn^2}\sum_{\alpha=1}^m\sum_{k +#include +#include +#include +using namespace std; +// output file as global variable +ofstream ofile; + +// Main function begins here +int main(int argc, char* argv[]) +{ + int n; + char *outfilename; + + cin >> n; + double MCint = 0.; double MCintsqr2=0.; + double invers_period = 1./RAND_MAX; // initialise the random number generator + srand(time(NULL)); // This produces the so-called seed in MC jargon + // Compute the variance and the mean value of the uniform distribution + // Compute also the specific values x for each cycle in order to be able to + // the covariance and the correlation function + // Read in output file, abort if there are too few command-line arguments + if( argc <= 2 ){ + cout << "Bad Usage: " << argv[0] << + " read also output file and number of cycles on same line" << endl; + exit(1); + } + else{ + outfilename=argv[1]; + } + ofile.open(outfilename); + // Get the number of Monte-Carlo samples + n = atoi(argv[2]); + double *X; + X = new double[n]; + for (int i = 0; i < n; i++){ + double x = double(rand())*invers_period; + X[i] = x; + MCint += x; + MCintsqr2 += x*x; + } + double Mean = MCint/((double) n ); + MCintsqr2 = MCintsqr2/((double) n ); + double STDev = sqrt(MCintsqr2-Mean*Mean); + double Variance = MCintsqr2-Mean*Mean; +// Write mean value and standard deviation + cout << " Standard deviation= " << STDev << " Integral = " << Mean << endl; + + // Now we compute the autocorrelation function + double *autocor; autocor = new double[n]; + for (int j = 0; j < n; j++){ + double sum = 0.0; + for (int k = 0; k < (n-j); k++){ + sum += (X[k]-Mean)*(X[k+j]-Mean); + } + autocor[j] = sum/Variance/((double) n ); + ofile << setiosflags(ios::showpoint | ios::uppercase); + ofile << setw(15) << setprecision(8) << j; + ofile << setw(15) << setprecision(8) << autocor[j] << endl; + } + ofile.close(); // close output file + return 0; +} // end of main program +!ec +!eblock + + + + +======= Which RNG should I use? ======= +!bblock +* C++ has a class called _random_. The "random class":"http://www.cplusplus.com/reference/random/" contains a large selection of RNGs and is highly recommended. Some of these RNGs have very large periods making it thereby very safe to use these RNGs in case one is performing large calculations. In particular, the "Mersenne twister random number engine":"http://www.cplusplus.com/reference/random/mersenne_twister_engine/" has a period of $2^{19937}$. +* Add RNGs in Python + +!eblock + + + +===== How to use the Mersenne generator ===== +!bblock +The following part of a c++ code (from project 4) sets up the uniform distribution for $x\in [0,1]$. +!bc cppcod +/* + +// You need this +#include + +// Initialize the seed and call the Mersienne algo +std::random_device rd; +std::mt19937_64 gen(rd()); +// Set up the uniform distribution for x \in [[0, 1] +std::uniform_real_distribution RandomNumberGenerator(0.0,1.0); + +// Now use the RNG +int ix = (int) (RandomNumberGenerator(gen)*NSpins); +!ec +!eblock + + + + + +===== Why blocking? ===== +!bblock Statistical analysis + * Monte Carlo simulations can be treated as *computer experiments* + * The results can be analysed with the same statistical tools as we would use analysing experimental data. + * As in all experiments, we are looking for expectation values and an estimate of how accurate they are, i.e., possible sources for errors. + +A very good article which explains blocking is H. Flyvbjerg and H. G. Petersen, *Error estimates on averages of correlated data*, "Journal of Chemical Physics 91, 461-466 (1989)":"http://scitation.aip.org/content/aip/journal/jcp/91/1/10.1063/1.457480". + +!eblock + + + + +===== Why blocking? ===== +!bblock Statistical analysis + * As in other experiments, Monte Carlo experiments have two classes of errors: + * Statistical errors + * Systematical errors + * Statistical errors can be estimated using standard tools from statistics + * Systematical errors are method specific and must be treated differently from case to case. (In VMC a common source is the step length or time step in importance sampling) +!eblock + + + +===== Code to demonstrate the calculation of the autocorrelation function ===== +The following code computes the autocorrelation function, the covariance and the standard deviation +for standard RNG. +The "following file":"https://github.com/CompPhysics/ComputationalPhysics2/tree/gh-pages/doc/Programs/LecturePrograms/programs/Blocking/autocorrelation.cpp" gives the code. +!bc cppcod +// This function computes the autocorrelation function for +// the Mersenne random number generator with a uniform distribution +#include +#include +#include +#include +#include +#include +#include +#include +using namespace std; +using namespace arma; +// output file +ofstream ofile; + +// Main function begins here +int main(int argc, char* argv[]) +{ + int MonteCarloCycles; + string filename; + if (argc > 1) { + filename=argv[1]; + MonteCarloCycles = atoi(argv[2]); + string fileout = filename; + string argument = to_string(MonteCarloCycles); + fileout.append(argument); + ofile.open(fileout); + } + + // Compute the variance and the mean value of the uniform distribution + // Compute also the specific values x for each cycle in order to be able to + // compute the covariance and the correlation function + + vec X = zeros(MonteCarloCycles); + double MCint = 0.; double MCintsqr2=0.; + std::random_device rd; + std::mt19937_64 gen(rd()); + // Set up the uniform distribution for x \in [[0, 1] + std::uniform_real_distribution RandomNumberGenerator(0.0,1.0); + for (int i = 0; i < MonteCarloCycles; i++){ + double x = RandomNumberGenerator(gen); + X(i) = x; + MCint += x; + MCintsqr2 += x*x; + } + double Mean = MCint/((double) MonteCarloCycles ); + MCintsqr2 = MCintsqr2/((double) MonteCarloCycles ); + double STDev = sqrt(MCintsqr2-Mean*Mean); + double Variance = MCintsqr2-Mean*Mean; + // Write mean value and variance + cout << " Sample variance= " << Variance << " Mean value = " << Mean << endl; + // Now we compute the autocorrelation function + vec autocorrelation = zeros(MonteCarloCycles); + for (int j = 0; j < MonteCarloCycles; j++){ + double sum = 0.0; + for (int k = 0; k < (MonteCarloCycles-j); k++){ + sum += (X(k)-Mean)*(X(k+j)-Mean); + } + autocorrelation(j) = sum/Variance/((double) MonteCarloCycles ); + ofile << setiosflags(ios::showpoint | ios::uppercase); + ofile << setw(15) << setprecision(8) << j; + ofile << setw(15) << setprecision(8) << autocorrelation(j) << endl; + } + // Now compute the exact covariance using the autocorrelation function + double Covariance = 0.0; + for (int j = 0; j < MonteCarloCycles; j++){ + Covariance += autocorrelation(j); + } + Covariance *= 2.0/((double) MonteCarloCycles); + // Compute now the total variance, including the covariance, and obtain the standard deviation + double TotalVariance = (Variance/((double) MonteCarloCycles ))+Covariance; + cout << "Covariance =" << Covariance << "Totalvariance= " << TotalVariance << "Sample Variance/n= " << (Variance/((double) MonteCarloCycles )) << endl; + cout << " STD from sample variance= " << sqrt(Variance/((double) MonteCarloCycles )) << " STD with covariance = " << sqrt(TotalVariance) << endl; + + ofile.close(); // close output file + return 0; +} // end of main program + + +!ec + + + +===== What is blocking? ===== +!bblock Blocking + * Say that we have a set of samples from a Monte Carlo experiment + * Assuming (wrongly) that our samples are uncorrelated our best estimate of the standard deviation of the mean $\langle \mathbf{M}\rangle$ is given by +!bt +\[ +\sigma=\sqrt{\frac{1}{n}\left(\langle \mathbf{M}^2\rangle-\langle \mathbf{M}\rangle^2\right)} +\] +!et + * If the samples are correlated we can rewrite our results to show that +!bt +\[ +\sigma=\sqrt{\frac{1+2\tau/\Delta t}{n}\left(\langle \mathbf{M}^2\rangle-\langle \mathbf{M}\rangle^2\right)} +\] +!et + where $\tau$ is the correlation time (the time between a sample and the next uncorrelated sample) and $\Delta t$ is time between each sample +!eblock + + +===== What is blocking? ===== +!bblock Blocking + * If $\Delta t\gg\tau$ our first estimate of $\sigma$ still holds + * Much more common that $\Delta t<\tau$ + * In the method of data blocking we divide the sequence of samples into blocks + * We then take the mean $\langle \mathbf{M}_i\rangle$ of block $i=1\ldots n_{blocks}$ to calculate the total mean and variance + * The size of each block must be so large that sample $j$ of block $i$ is not correlated with sample $j$ of block $i+1$ + * The correlation time $\tau$ would be a good choice +!eblock + + +===== What is blocking? ===== +!bblock Blocking + * Problem: We don't know $\tau$ or it is too expensive to compute + * Solution: Make a plot of std. dev. as a function of blocksize + * The estimate of std. dev. of correlated data is too low $\to$ the error will increase with increasing block size until the blocks are uncorrelated, where we reach a plateau + * When the std. dev. stops increasing the blocks are uncorrelated +!eblock + + +===== Implementation ===== +!bblock + * Do a Monte Carlo simulation, storing all samples to file + * Do the statistical analysis on this file, independently of your Monte Carlo program + * Read the file into an array + * Loop over various block sizes + * For each block size $n_b$, loop over the array in steps of $n_b$ taking the mean of elements $i n_b,\ldots,(i+1) n_b$ + * Take the mean and variance of the resulting array + * Write the results for each block size to file for later + analysis +!eblock + + + + + + +===== Actual implementation with code, main function ===== +When the file gets large, it can be useful to write your data in binary mode instead of ascii characters. +The "following python file":"https://github.com/CompPhysics/MachineLearning/blob/master/doc/Programs/Sampling/analysis.py" reads data from file with the output from every Monte Carlo cycle. +!bc pycod +# Blocking + @timeFunction + def blocking(self, blockSizeMax = 500): + blockSizeMin = 1 + + self.blockSizes = [] + self.meanVec = [] + self.varVec = [] + + for i in range(blockSizeMin, blockSizeMax): + if(len(self.data) % i != 0): + pass#continue + blockSize = i + meanTempVec = [] + varTempVec = [] + startPoint = 0 + endPoint = blockSize + + while endPoint <= len(self.data): + meanTempVec.append(np.average(self.data[startPoint:endPoint])) + startPoint = endPoint + endPoint += blockSize + mean, var = np.average(meanTempVec), np.var(meanTempVec)/len(meanTempVec) + self.meanVec.append(mean) + self.varVec.append(var) + self.blockSizes.append(blockSize) + + self.blockingAvg = np.average(self.meanVec[-200:]) + self.blockingVar = (np.average(self.varVec[-200:])) + self.blockingStd = np.sqrt(self.blockingVar) + +!ec + + + + + +===== The Bootstrap method ===== + +The Bootstrap resampling method is also very popular. It is very simple: + +o Start with your sample of measurements and compute the sample variance and the mean values +o Then start again but pick in a random way the numbers in the sample and recalculate the mean and the sample variance. +o Repeat this $K$ times. + +It can be shown, see the article by "Efron":"https://projecteuclid.org/download/pdf_1/euclid.aos/1176344552" +that it produces the correct standard deviation. + +This method is very useful for small ensembles of data points. + + +===== Bootstrapping ===== +Given a set of $N$ data, assume that we are interested in some +observable $\theta$ which may be estimated from that set. This observable can also be for example the result of a fit based on all $N$ raw data. +Let us call the value of the observable obtained from the original +data set $\hat{\theta}$. One recreates from the sample repeatedly +other samples by choosing randomly $N$ data out of the original set. +This costs essentially nothing, since we just recycle the original data set for the building of new sets. + + +===== Bootstrapping, recipe ===== +Let us assume we have done this $K$ times and thus have $K$ sets of $N$ +data values each. +Of course some values will enter more than once in the new sets. For each of these sets one computes the observable $\theta$ resulting in values $\theta_k$ with $k = 1,...,K$. Then one determines +!bt +\[ +\tilde{\theta} = \frac{1}{K} \sum_{k=1}^K \theta_k, +\] +!et +and +!bt +\[ +sigma^2_{\tilde{\theta}} = \frac{1}{K} \sum_{k=1}^K \left(\theta_k-\tilde{\theta}\right)^2. +\] +!et + +These are estimators for $\angle\theta\rangle$ and its variance. They are not unbiased and therefore +$\tilde{\theta}\neq\hat{\theta}$ for finite K. + +The difference is called bias and gives an idea on how far away the result may be from +the true $\angle\theta\rangle$. As final result for the observable one quotes $\angle\theta\rangle = \tilde{\theta} \pm \sigma_{\tilde{\theta}}$ . + + + +===== Bootstrapping, "code":"https://github.com/CompPhysics/MachineLearning/blob/master/doc/Programs/Sampling/analysis.py" ===== +!bc +# Bootstrap + @timeFunction + def bootstrap(self, nBoots = 1000): + bootVec = np.zeros(nBoots) + for k in range(0,nBoots): + bootVec[k] = np.average(np.random.choice(self.data, len(self.data))) + self.bootAvg = np.average(bootVec) + self.bootVar = np.var(bootVec) + self.bootStd = np.std(bootVec) +!ec + + +===== Jackknife, "code":"https://github.com/CompPhysics/MachineLearning/blob/master/doc/Programs/Sampling/analysis.py" ===== +!bc +# Jackknife + @timeFunction + def jackknife(self): + jackknVec = np.zeros(len(self.data)) + for k in range(0,len(self.data)): + jackknVec[k] = np.average(np.delete(self.data, k)) + self.jackknAvg = self.avg - (len(self.data) - 1) * (np.average(jackknVec) - self.avg) + self.jackknVar = float(len(self.data) - 1) * np.var(jackknVec) + self.jackknStd = np.sqrt(self.jackknVar) +!ec + + + + + + +======= Regression analysis, overarching aims ======= +!bblock + +Regression modeling deals with the description of the sampling distribution of a given random variable $y$ varies as function of another variable or a set of such variables $\hat{x} =[x_0, x_1,\dots, x_p]^T$. +The first variable is called the _dependent_, the _outcome_ or the _response_ variable while the set of variables $\hat{x}$ is called the independent variable, or the predictor variable or the explanatory variable. + +A regression model aims at finding a likelihood function $p(y\vert \hat{x})$, that is the conditional distribution for $y$ with a given $\hat{x}$. The estimation of $p(y\vert \hat{x})$ is made using a data set with +* $n$ cases $i = 0, 1, 2, \dots, n-1$ +* Response (dependent or outcome) variable $y_i$ with $i = 0, 1, 2, \dots, n-1$ +* $p$ Explanatory (independent or predictor) variables $\hat{x}_i=[x_{i0}, x_{i1}, \dots, x_{ip}]$ with $i = 0, 1, 2, \dots, n-1$ + The goal of the regression analysis is to extract/exploit relationship between $y_i$ and $\hat{x}_i$ in or to infer causal dependencies, approximations to the likelihood functions, functional relationships and to make predictions . +!eblock + + +===== Regression analysis, overarching aims II ===== +!bblock + + +Consider an experiment in which $p$ characteristics of $n$ samples are +measured. The data from this experiment are denoted $\mathbf{X}$, with +$\mathbf{X}$ as above. The matrix $\mathbf{X}$ is called the *design +matrix*. Additional information of the samples is available in the +form of $\mathbf{Y}$ (also as above). The variable $\mathbf{Y}$ is +generally referred to as the *response variable*. The aim of +regression analysis is to explain $\mathbf{Y}$ in terms of +$\mathbf{X}$ through a functional relationship like $Y_i = +f(\mathbf{X}_{i,\ast})$. When no prior knowledge on the form of +$f(\cdot)$ is available, it is common to assume a linear relationship +between $\mathbf{X}$ and $\mathbf{Y}$. This assumption gives rise to +the *linear regression model* where $\beta = (\beta_1, \ldots, +\beta_p)^{\top}$ is the *regression parameter*. The parameter +$\beta_j$, $j=1, \ldots, p$, represents the effect size of covariate +$j$ on the response. That is, for each unit change in covariate $j$ +(while keeping the other covariates fixed) the observed change in the +response is equal to $\beta_j$. + +!eblock + + +===== General linear models ===== +!bblock +Before we proceed let us study a case from linear algebra where we aim at fitting a set of data $\hat{y}=[y_0,y_1,\dots,y_{n-1}]$. We could think of these data as a result of an experiment or a complicated numerical experiment. These data are functions of a series of variables $\hat{x}=[x_0,x_1,\dots,x_{n-1}]$, that is $y_i = y(x_i)$ with $i=0,1,2,\dots,n-1$. The variables $x_i$ could represent physical quantities like time, temperature, position etc. We assume that $y(x)$ is a smooth function. + +Since obtaining these data points may not be trivial, we want to use these data to fit a function which can allow us to make predictions for values of $y$ which are not in the present set. The perhaps simplest approach is to assume we can parametrize our function in terms of a polynomial of degree $n-1$ with $n$ points, that is +!bt +\[ +y=y(x) \rightarrow y(x_i)=\tilde{y}_i+\epsilon_i=\sum_{j=0}^{n-1} \beta_i x_i^j+\epsilon_i, +\] +!et +where $\epsilon_i$ is the error in our approximation. + +!eblock + + + +===== Rewriting the fitting procedure as a linear algebra problem ===== +!bblock +For every set of values $y_i,x_i$ we have thus the corresponding set of equations +!bt +\begin{align*} +y_0&=\beta_0+\beta_1x_0^1+\beta_2x_0^2+\dots+\beta_{n-1}x_0^{n-1}+\epsilon_0\\ +y_1&=\beta_0+\beta_1x_1^1+\beta_2x_1^2+\dots+\beta_{n-1}x_1^{n-1}+\epsilon_1\\ +y_2&=\beta_0+\beta_1x_2^1+\beta_2x_2^2+\dots+\beta_{n-1}x_2^{n-1}+\epsilon_2\\ +\dots & \dots \\ +y_{n-1}&=\beta_0+\beta_1x_{n-1}^1+\beta_2x_{n-1}^2+\dots+\beta_1x_{n-1}^{n-1}+\epsilon_{n-1}.\\ +\end{align*} +!et +!eblock + + + +===== Rewriting the fitting procedure as a linear algebra problem, follows ===== +!bblock +Defining the vectors +!bt +\[ +\hat{y} = [y_0,y_1, y_2,\dots, y_{n-1}]^T, +\] +!et +and +!bt +\[ +\hat{\beta} = [\beta_0,\beta_1, \beta_2,\dots, \beta_{n-1}]^T, +\] +!et +and +!bt +\[ +\hat{\epsilon} = [\epsilon_0,\epsilon_1, \epsilon_2,\dots, \epsilon_{n-1}]^T, +\] +!et +and the matrix +!bt +\[ +\hat{X}= +\begin{bmatrix} +1& x_{0}^1 &x_{0}^2& \dots & \dots &x_{0}^{n-1}\\ +1& x_{1}^1 &x_{1}^2& \dots & \dots &x_{1}^{n-1}\\ +1& x_{2}^1 &x_{2}^2& \dots & \dots &x_{2}^{n-1}\\ +\dots& \dots &\dots& \dots & \dots &\dots\\ +1& x_{n-1}^1 &x_{n-1}^2& \dots & \dots &x_{n-1}^{n-1}\\ +\end{bmatrix} +\] +!et +we can rewrite our equations as +!bt +\[ +\hat{y} = \hat{X}\hat{\beta}+\hat{\epsilon}. +\] +!et +!eblock + + + +===== Generalizing the fitting procedure as a linear algebra problem ===== +!bblock +We are obviously not limited to the above polynomial. We could replace the various powers of $x$ with elements of Fourier series, that is, instead of $x_i^j$ we could have $\cos{(j x_i)}$ or $\sin{(j x_i)}$, or time series or other orthogonal functions. +For every set of values $y_i,x_i$ we can then generalize the equations to +!bt +\begin{align*} +y_0&=\beta_0x_{00}+\beta_1x_{01}+\beta_2x_{02}+\dots+\beta_{n-1}x_{0n-1}+\epsilon_0\\ +y_1&=\beta_0x_{10}+\beta_1x_{11}+\beta_2x_{12}+\dots+\beta_{n-1}x_{1n-1}+\epsilon_1\\ +y_2&=\beta_0x_{20}+\beta_1x_{21}+\beta_2x_{22}+\dots+\beta_{n-1}x_{2n-1}+\epsilon_2\\ +\dots & \dots \\ +y_{i}&=\beta_0x_{i0}+\beta_1x_{i1}+\beta_2x_{i2}+\dots+\beta_{n-1}x_{in-1}+\epsilon_i\\ +\dots & \dots \\ +y_{n-1}&=\beta_0x_{n-1,0}+\beta_1x_{n-1,2}+\beta_2x_{n-1,2}+\dots+\beta_1x_{n-1,n-1}+\epsilon_{n-1}.\\ +\end{align*} +!et +!eblock + + + +===== Generalizing the fitting procedure as a linear algebra problem ===== +!bblock +We redefine in turn the matrix $\hat{X}$ as +!bt +\[ +\hat{X}= +\begin{bmatrix} +x_{00}& x_{01} &x_{02}& \dots & \dots &x_{0,n-1}\\ +x_{10}& x_{11} &x_{12}& \dots & \dots &x_{1,n-1}\\ +x_{20}& x_{21} &x_{22}& \dots & \dots &x_{2,n-1}\\ +\dots& \dots &\dots& \dots & \dots &\dots\\ +x_{n-1,0}& x_{n-1,1} &x_{n-1,2}& \dots & \dots &x_{n-1,n-1}\\ +\end{bmatrix} +\] +!et +and without loss of generality we rewrite again our equations as +!bt +\[ +\hat{y} = \hat{X}\hat{\beta}+\hat{\epsilon}. +\] +!et +The left-hand side of this equation forms know. Our error vector $\hat{\epsilon}$ and the parameter vector $\hat{\beta}$ are our unknow quantities. How can we obtain the optimal set of $\beta_i$ values? +!eblock + + + +===== Optimizing our parameters ===== +!bblock +We have defined the matrix $\hat{X}$ +!bt +\begin{align*} +y_0&=\beta_0x_{00}+\beta_1x_{01}+\beta_2x_{02}+\dots+\beta_{n-1}x_{0n-1}+\epsilon_0\\ +y_1&=\beta_0x_{10}+\beta_1x_{11}+\beta_2x_{12}+\dots+\beta_{n-1}x_{1n-1}+\epsilon_1\\ +y_2&=\beta_0x_{20}+\beta_1x_{21}+\beta_2x_{22}+\dots+\beta_{n-1}x_{2n-1}+\epsilon_1\\ +\dots & \dots \\ +y_{i}&=\beta_0x_{i0}+\beta_1x_{i1}+\beta_2x_{i2}+\dots+\beta_{n-1}x_{in-1}+\epsilon_1\\ +\dots & \dots \\ +y_{n-1}&=\beta_0x_{n-1,0}+\beta_1x_{n-1,2}+\beta_2x_{n-1,2}+\dots+\beta_1x_{n-1,n-1}+\epsilon_{n-1}.\\ +\end{align*} +!et +!eblock + + + +===== Optimizing our parameters, more details ===== +!bblock +We well use this matrix to define the approximation $\hat{\tilde{y}}$ via the unknown quantity $\hat{\beta}$ as +!bt +\[ +\hat{\tilde{y}}= \hat{X}\hat{\beta}, +\] +!et +and in order to find the optimal parameters $\beta_i$ instead of solving the above linear algebra problem, we define a function which gives a measure of the spread between the values $y_i$ (which represent hopefully the exact values) and the parametrized values $\tilde{y}_i$, namely +!bt +\[ +Q(\hat{\beta})=\sum_{i=0}^{n-1}\left(y_i-\tilde{y}_i\right)^2=\left(\hat{y}-\hat{\tilde{y}}\right)^T\left(\hat{y}-\hat{\tilde{y}}\right), +\] +!et +or using the matrix $\hat{X}$ as +!bt +\[ +Q(\hat{\beta})=\left(\hat{y}-\hat{X}\hat{\beta}\right)^T\left(\hat{y}-\hat{X}\hat{\beta}\right). +\] +!et +!eblock + + + +===== Interpretations and optimizing our parameters ===== +!bblock +The function +!bt +\[ +Q(\hat{\beta})=\left(\hat{y}-\hat{X}\hat{\beta}\right)^T\left(\hat{y}-\hat{X}\hat{\beta}\right), +\] +!et +can be linked to the variance of the quantity $y_i$ if we interpret the latter as the mean value of for example a numerical experiment. When linking below with the maximum likelihood approach below, we will indeed interpret $y_i$ as a mean value +!bt +\[ +y_{i}=\langle y_i \rangle = \beta_0x_{i,0}+\beta_1x_{i,1}+\beta_2x_{i,2}+\dots+\beta_{n-1}x_{i,n-1}+\epsilon_i, +\] +!et +where $\langle y_i \rangle$ is the mean value. Keep in mind also that till now we have treated $y_i$ as the exact value. Normally, the response (dependent or outcome) variable $y_i$ the outcome of a numerical experiment or another type of experiment and is thus only an approximation to the true value. It is then always accompanied by an error estimate, often limited to a statistical error estimate given by the standard deviation discussed earlier. In the discussion here we will treat $y_i$ as our exact value for the response variable. + +In order to find the parameters $\beta_i$ we will then minimize the spread of $Q(\hat{\beta})$ by requiring +!bt +\[ +\frac{\partial Q(\hat{\beta})}{\partial \beta_j} = \frac{\partial }{\partial \beta_j}\left[ \sum_{i=0}^{n-1}\left(y_i-\beta_0x_{i,0}-\beta_1x_{i,1}-\beta_2x_{i,2}-\dots-\beta_{n-1}x_{i,n-1}\right)^2\right]=0, +\] +!et +which results in +!bt +\[ +\frac{\partial Q(\hat{\beta})}{\partial \beta_j} = -2\left[ \sum_{i=0}^{n-1}x_{ij}\left(y_i-\beta_0x_{i,0}-\beta_1x_{i,1}-\beta_2x_{i,2}-\dots-\beta_{n-1}x_{i,n-1}\right)\right]=0, +\] +!et +or in a matrix-vector form as +!bt +\[ +\frac{\partial Q(\hat{\beta})}{\partial \hat{\beta}} = 0 = \hat{X}^T\left( \hat{y}-\hat{X}\hat{\beta}\right). +\] +!et + + +!eblock + + + +===== Interpretations and optimizing our parameters ===== +!bblock +We can rewrite +!bt +\[ +\frac{\partial Q(\hat{\beta})}{\partial \hat{\beta}} = 0 = \hat{X}^T\left( \hat{y}-\hat{X}\hat{\beta}\right), +\] +!et +as +!bt +\[ +\hat{X}^T\hat{y} = \hat{X}^T\hat{X}\hat{\beta}, +\] +!et +and if the matrix $\hat{X}^T\hat{X}$ is invertible we have the solution +!bt +\[ +\hat{\beta} =\left(\hat{X}^T\hat{X}\right)^{-1}\hat{X}^T\hat{y}. +\] +!et + +!eblock + + +===== Interpretations and optimizing our parameters ===== +!bblock +The residuals $\hat{\epsilon}$ are in turn given by +!bt +\[ +\hat{\epsilon} = \hat{y}-\hat{\tilde{y}} = \hat{y}-\hat{X}\hat{\beta}, +\] +!et +and with +!bt +\[ +\hat{X}^T\left( \hat{y}-\hat{X}\hat{\beta}\right)= 0, +\] +!et +we have +!bt +\[ +\hat{X}^T\hat{\epsilon}=\hat{X}^T\left( \hat{y}-\hat{X}\hat{\beta}\right)= 0, +\] +!et +meaning that the solution for $\hat{\beta}$ is the one which minimizes the residuals. Later we will link this with the maximum likelihood approach. + +!eblock + + + +===== The $\chi^2$ function ===== +!bblock + +Normally, the response (dependent or outcome) variable $y_i$ the outcome of a numerical experiment or another type of experiment and is thus only an approximation to the true value. It is then always accompanied by an error estimate, often limited to a statistical error estimate given by the standard deviation discussed earlier. In the discussion here we will treat $y_i$ as our exact value for the response variable. + +Introducing the standard deviation $\sigma_i$ for each measurement $y_i$, we define now the $\chi^2$ function as +!bt +\[ +\chi^2(\hat{\beta})=\sum_{i=0}^{n-1}\frac{\left(y_i-\tilde{y}_i\right)^2}{\sigma_i^2}=\left(\hat{y}-\hat{\tilde{y}}\right)^T\frac{1}{\hat{\Sigma^2}}\left(\hat{y}-\hat{\tilde{y}}\right), +\] +!et +where the matrix $\hat{\Sigma}$ is a diagonal matrix with $\sigma_i$ as matrix elements. + +!eblock + + +===== The $\chi^2$ function ===== +!bblock + +In order to find the parameters $\beta_i$ we will then minimize the spread of $\chi^2(\hat{\beta})$ by requiring +!bt +\[ +\frac{\partial \chi^2(\hat{\beta})}{\partial \beta_j} = \frac{\partial }{\partial \beta_j}\left[ \sum_{i=0}^{n-1}\left(\frac{y_i-\beta_0x_{i,0}-\beta_1x_{i,1}-\beta_2x_{i,2}-\dots-\beta_{n-1}x_{i,n-1}}{\sigma_i}\right)^2\right]=0, +\] +!et +which results in +!bt +\[ +\frac{\partial \chi^2(\hat{\beta})}{\partial \beta_j} = -2\left[ \sum_{i=0}^{n-1}\frac{x_{ij}}{\sigma_i}\left(\frac{y_i-\beta_0x_{i,0}-\beta_1x_{i,1}-\beta_2x_{i,2}-\dots-\beta_{n-1}x_{i,n-1}}{\sigma_i}\right)\right]=0, +\] +!et +or in a matrix-vector form as +!bt +\[ +\frac{\partial \chi^2(\hat{\beta})}{\partial \hat{\beta}} = 0 = \hat{A}^T\left( \hat{b}-\hat{A}\hat{\beta}\right). +\] +!et +where we have defined the matrix $\hat{A} =\hat{X}/\hat{\Sigma}$ with matrix elements $a_{ij} = x_{ij}/\sigma_i$ and the vector $\hat{b}$ with elements $b_i = y_i/\sigma_i$. +!eblock + + +===== The $\chi^2$ function ===== +!bblock + +We can rewrite +!bt +\[ +\frac{\partial \chi^2(\hat{\beta})}{\partial \hat{\beta}} = 0 = \hat{A}^T\left( \hat{b}-\hat{A}\hat{\beta}\right), +\] +!et +as +!bt +\[ +\hat{A}^T\hat{b} = \hat{A}^T\hat{A}\hat{\beta}, +\] +!et +and if the matrix $\hat{A}^T\hat{A}$ is invertible we have the solution +!bt +\[ +\hat{\beta} =\left(\hat{A}^T\hat{A}\right)^{-1}\hat{A}^T\hat{b}. +\] +!et +!eblock + + +===== The $\chi^2$ function ===== +!bblock + +If we then introduce the matrix +!bt +\[ +\hat{H} = \left(\hat{A}^T\hat{A}\right)^{-1}, +\] +!et +we have then the following expression for the parameters $\beta_j$ (the matrix elements of $\hat{H}$ are $h_{ij}$) +!bt +\[ +\beta_j = \sum_{k=0}^{p-1}h_{jk}\sum_{i=0}^{n-1}\frac{y_i}{\sigma_i}\frac{x_{ik}}{\sigma_i} = \sum_{k=0}^{p-1}h_{jk}\sum_{i=0}^{n-1}b_ia_{ik} +\] +!et +We state without proof the expression for the uncertainty in the parameters $\beta_j$ as (we leave this as an exercise) +!bt +\[ +\sigma^2(\beta_j) = \sum_{i=0}^{n-1}\sigma_i^2\left( \frac{\partial \beta_j}{\partial y_i}\right)^2, +\] +!et +resulting in +!bt +\[ +\sigma^2(\beta_j) = \left(\sum_{k=0}^{p-1}h_{jk}\sum_{i=0}^{n-1}a_{ik}\right)\left(\sum_{l=0}^{p-1}h_{jl}\sum_{m=0}^{n-1}a_{ml}\right) = h_{jj}! +\] +!et +!eblock + + +===== The $\chi^2$ function ===== +!bblock +The first step here is to approximate the function $y$ with a first-order polynomial, that is we write +!bt +\[ +y=y(x) \rightarrow y(x_i) \approx \beta_0+\beta_1 x_i. +\] +!et +By computing the derivatives of $\chi^2$ with respect to $\beta_0$ and $\beta_1$ show that these are given by +!bt +\[ +\frac{\partial \chi^2(\hat{\beta})}{\partial \beta_0} = -2\left[ \sum_{i=0}^{n-1}\left(\frac{y_i-\beta_0-\beta_1x_{i}}{\sigma_i^2}\right)\right]=0, +\] +!et +and +!bt +\[ +\frac{\partial \chi^2(\hat{\beta})}{\partial \beta_0} = -2\left[ \sum_{i=0}^{n-1}x_i\left(\frac{y_i-\beta_0-\beta_1x_{i}}{\sigma_i^2}\right)\right]=0. +\] +!et +!eblock + + +===== The $\chi^2$ function ===== +!bblock + +For a linear fit we don't need to invert a matrix!! +Defining +!bt +\[ +\gamma = \sum_{i=0}^{n-1}\frac{1}{\sigma_i^2}, +\] +!et + +!bt +\[ +\gamma_x = \sum_{i=0}^{n-1}\frac{x_{i}}{\sigma_i^2}, +\] +!et +!bt +\[ +\gamma_y = \sum_{i=0}^{n-1}\left(\frac{y_i}{\sigma_i^2}\right), +\] +!et +!bt +\[ +\gamma_{xx} = \sum_{i=0}^{n-1}\frac{x_ix_{i}}{\sigma_i^2}, +\] +!et +!bt +\[ +\gamma_{xy} = \sum_{i=0}^{n-1}\frac{y_ix_{i}}{\sigma_i^2}, +\] +!et +we obtain +!bt +\[ +\beta_0 = \frac{\gamma_{xx}\gamma_y-\gamma_x\gamma_y}{\gamma\gamma_{xx}-\gamma_x^2}, +\] +!et +!bt +\[ +\beta_1 = \frac{\gamma_{xy}\gamma-\gamma_x\gamma_y}{\gamma\gamma_{xx}-\gamma_x^2}. +\] +!et + +This approach (different linear and non-linear regression) suffers often from both being underdetermined and overdetermined in the unknown coefficients $\beta_i$. A better approach is to use the Singular Value Decomposition (SVD) method discussed below. Or using Lasso and Ridge regression. See below. +!eblock + + + + + +===== Simple regression model ===== +We are now ready to write our first program which aims at solving the above linear regression equations. We start with data we have produced ourselves, in this case normally distributed random numbers along the $x$-axis. These numbers define then the value of a function $y(x)=4+3x+N(0,1)$. Thereafter we order the $x$ values and employ our linear regression algorithm to set up the best fit. Here we find it useful to use the numpy function $c\_$ arrays where arrays are stacked along their last axis after being upgraded to at least two dimensions with ones post-pended to the shape. The following examples help in understanding what happens +!bc pycod +import numpy as np +print(np.c_[np.array([1,2,3]), np.array([4,5,6])]) +print(np.c_[np.array([[1,2,3]]), 0, 0, np.array([[4,5,6]])]) +!ec + +!bc pycod +# Importing various packages +from random import random, seed +import numpy as np +import matplotlib.pyplot as plt + +x = 2*np.random.rand(100,1) +y = 4+3*x+np.random.randn(100,1) + +xb = np.c_[np.ones((100,1)), x] +beta = np.linalg.inv(xb.T.dot(xb)).dot(xb.T).dot(y) +xnew = np.array([[0],[2]]) +xbnew = np.c_[np.ones((2,1)), xnew] +ypredict = xbnew.dot(beta) + +plt.plot(xnew, ypredict, "r-") +plt.plot(x, y ,'ro') +plt.axis([0,2.0,0, 15.0]) +plt.xlabel(r'$x$') +plt.ylabel(r'$y$') +plt.title(r'Linear Regression') +plt.show() + +!ec + +We see that, as expected, a linear fit gives a seemingly (from the graph) good representation of the data. + + + + + +===== Simple regression model, now using _scikit-learn_ ===== + + +We can repeat the above algorithm using _scikit-learn_ as follows +!bc pycod +# Importing various packages +from random import random, seed +import numpy as np +import matplotlib.pyplot as plt +from sklearn.linear_model import LinearRegression + +x = 2*np.random.rand(100,1) +y = 4+3*x+np.random.randn(100,1) +linreg = LinearRegression() +linreg.fit(x,y) +xnew = np.array([[0],[2]]) +ypredict = linreg.predict(xnew) + +plt.plot(xnew, ypredict, "r-") +plt.plot(x, y ,'ro') +plt.axis([0,2.0,0, 15.0]) +plt.xlabel(r'$x$') +plt.ylabel(r'$y$') +plt.title(r'Random numbers ') +plt.show() +!ec + + + +===== Simple linear regression model using _scikit-learn_ ===== + +We start with perhaps our simplest possible example, using _scikit-learn_ to perform linear regression analysis on a data set produced by us. +What follows is a simple Python code where we have defined function $y$ in terms of the variable $x$. Both are defined as vectors of dimension $1\times 100$. The entries to the vector $\hat{x}$ are given by random numbers generated with a uniform distribution with entries $x_i \in [0,1]$ (more about probability distribution functions later). These values are then used to define a function $y(x)$ (tabulated again as a vector) with a linear dependence on $x$ plus a random noise added via the normal distribution. + + +The Numpy functions are imported used the _import numpy as np_ +statement and the random number generator for the uniform distribution +is called using the function _np.random.rand()_, where we specificy +that we want $100$ random variables. Using Numpy we define +automatically an array with the specified number of elements, $100$ in +our case. With the Numpy function _randn()_ we can compute random +numbers with the normal distribution (mean value $\mu$ equal to zero and +variance $\sigma^2$ set to one) and produce the values of $y$ assuming a linear +dependence as function of $x$ + +!bt +\[ +y = 2x+N(0,1), +\] +!et + +where $N(0,1)$ represents random numbers generated by the normal +distribution. From _scikit-learn_ we import then the +_LinearRegression_ functionality and make a prediction $\tilde{y} = +\alpha + \beta x$ using the function _fit(x,y)_. We call the set of +data $(\hat{x},\hat{y})$ for our training data. The Python package +_scikit-learn_ has also a functionality which extracts the above +fitting parameters $\alpha$ and $\beta$ (see below). Later we will +distinguish between training data and test data. + +For plotting we use the Python package +"matplotlib":"https://matplotlib.org/" which produces publication +quality figures. Feel free to explore the extensive +"gallery":"https://matplotlib.org/gallery/index.html" of examples. In +this example we plot our original values of $x$ and $y$ as well as the +prediction _ypredict_ ($\tilde{y}$), which attempts at fitting our +data with a straight line. + +The Python code follows here. +!bc pycod +# Importing various packages +import numpy as np +import matplotlib.pyplot as plt +from sklearn.linear_model import LinearRegression + +x = np.random.rand(100,1) +y = 2*x+np.random.randn(100,1) +linreg = LinearRegression() +linreg.fit(x,y) +xnew = np.array([[0],[1]]) +ypredict = linreg.predict(xnew) + +plt.plot(xnew, ypredict, "r-") +plt.plot(x, y ,'ro') +plt.axis([0,1.0,0, 5.0]) +plt.xlabel(r'$x$') +plt.ylabel(r'$y$') +plt.title(r'Simple Linear Regression') +plt.show() +!ec + + + +===== Simple linear regression model ===== + +This example serves several aims. It allows us to demonstrate several +aspects of data analysis and later machine learning algorithms. The +immediate visualization shows that our linear fit is not +impressive. It goes through the data points, but there are many +outliers which are not reproduced by our linear regression. We could +now play around with this small program and change for example the +factor in front of $x$ and the normal distribution. Try to change the +function $y$ to + +!bt +\[ +y = 10x+0.01 \times N(0,1), +\] +!et + +where $x$ is defined as before. + + + +===== Less noise ===== + +Does the fit look better? Indeed, by +reducing the role of the normal distribution we see immediately that +our linear prediction seemingly reproduces better the training +set. However, this testing 'by the eye' is obviouly not satisfactory in the +long run. Here we have only defined the training data and our model, and +have not discussed a more rigorous approach to the _cost_ function. + + + +===== How to study our fits ===== + +We need more rigorous criteria in defining whether we have succeeded or +not in modeling our training data. You will be surprised to see that +many scientists seldomly venture beyond this 'by the eye' approach. A +standard approach for the *cost* function is the so-called $\chi^2$ +function + +!bt +\[ \chi^2 = \frac{1}{n} +\sum_{i=0}^{n-1}\frac{(y_i-\tilde{y}_i)^2}{\sigma_i^2}, +\] +!et + +where $\sigma_i^2$ is the variance (to be defined later) of the entry +$y_i$. We may not know the explicit value of $\sigma_i^2$, it serves +however the aim of scaling the equations and make the cost function +dimensionless. + + + +===== Minimizing the cost function ===== + +Minimizing the cost function is a central aspect of +our discussions to come. Finding its minima as function of the model +parameters ($\alpha$ and $\beta$ in our case) will be a recurring +theme in these series of lectures. Essentially all machine learning +algorithms we will discuss center around the minimization of the +chosen cost function. This depends in turn on our specific +model for describing the data, a typical situation in supervised +learning. Automatizing the search for the minima of the cost function is a +central ingredient in all algorithms. Typical methods which are +employed are various variants of _gradient_ methods. These will be +discussed in more detail later. Again, you'll be surprised to hear that +many practitioners minimize the above function ''by the eye', popularly dubbed as +'chi by the eye'. That is, change a parameter and see (visually and numerically) that +the $\chi^2$ function becomes smaller. + + +===== Relative error ===== + +There are many ways to define the cost function. A simpler approach is to look at the relative difference between the training data and the predicted data, that is we define +the relative error as + +!bt +\[ +\epsilon_{\mathrm{relative}}= \frac{\vert \hat{y} -\hat{\tilde{y}}\vert}{\vert \hat{y}\vert}. +\] +!et +We can modify easily the above Python code and plot the relative error instead +!bc pycod +import numpy as np +import matplotlib.pyplot as plt +from sklearn.linear_model import LinearRegression + +x = np.random.rand(100,1) +y = 5*x+0.01*np.random.randn(100,1) +linreg = LinearRegression() +linreg.fit(x,y) +ypredict = linreg.predict(x) + +plt.plot(x, np.abs(ypredict-y)/abs(y), "ro") +plt.axis([0,1.0,0.0, 0.5]) +plt.xlabel(r'$x$') +plt.ylabel(r'$\epsilon_{\mathrm{relative}}$') +plt.title(r'Relative error') +plt.show() +!ec + +Depending on the parameter in front of the normal distribution, we may +have a small or larger relative error. Try to play around with +different training data sets and study (graphically) the value of the +relative error. + + + +===== The richness of _scikit-learn_ ===== + +As mentioned above, _scikit-learn_ has an impressive functionality. +We can for example extract the values of $\alpha$ and $\beta$ and +their error estimates, or the variance and standard deviation and many +other properties from the statistical data analysis. + +Here we show an +example of the functionality of scikit-learn. +!bc pycod +import numpy as np +import matplotlib.pyplot as plt +from sklearn.linear_model import LinearRegression +from sklearn.metrics import mean_squared_error, r2_score, mean_squared_log_error, mean_absolute_error + +x = np.random.rand(100,1) +y = 2.0+ 5*x+0.5*np.random.randn(100,1) +linreg = LinearRegression() +linreg.fit(x,y) +ypredict = linreg.predict(x) +print('The intercept alpha: \n', linreg.intercept_) +print('Coefficient beta : \n', linreg.coef_) +# The mean squared error +print("Mean squared error: %.2f" % mean_squared_error(y, ypredict)) +# Explained variance score: 1 is perfect prediction +print('Variance score: %.2f' % r2_score(y, ypredict)) +# Mean squared log error +print('Mean squared log error: %.2f' % mean_squared_log_error(y, ypredict) ) +# Mean absolute error +print('Mean absolute error: %.2f' % mean_absolute_error(y, ypredict)) +plt.plot(x, ypredict, "r-") +plt.plot(x, y ,'ro') +plt.axis([0.0,1.0,1.5, 7.0]) +plt.xlabel(r'$x$') +plt.ylabel(r'$y$') +plt.title(r'Linear Regression fit ') +plt.show() + +!ec + + + +===== Functions in _scikit-learn_ ===== + +The function _coef_ gives us the parameter $\beta$ of our fit while _intercept_ yields +$\alpha$. Depending on the constant in front of the normal distribution, we get values near or far from $alpha =2$ and $\beta =5$. Try to play around with different parameters in front of the normal distribution. The function _meansquarederror_ gives us the mean square error, a risk metric corresponding to the expected value of the squared (quadratic) error or loss defined as +!bt +\[ MSE(\hat{y},\hat{\tilde{y}}) = \frac{1}{n} +\sum_{i=0}^{n-1}(y_i-\tilde{y}_i)^2, +\] +!et + +The smaller the value, the better the fit. Ideally we would like to +have an MSE equal zero. The attentive reader has probably recognized +this function as being similar to the $\chi^2$ function defined above. + + +===== Other functions in _scikit-learn_ ===== + +The _r2score_ function computes $R^2$, the coefficient of +determination. It provides a measure of how well future samples are +likely to be predicted by the model. Best possible score is 1.0 and it +can be negative (because the model can be arbitrarily worse). A +constant model that always predicts the expected value of $\hat{y}$, +disregarding the input features, would get a $R^2$ score of $0.0$. + +If $\tilde{\hat{y}}_i$ is the predicted value of the $i-th$ sample and $y_i$ is the corresponding true value, then the score $R^2$ is defined as +!bt +\[ +R^2(\hat{y}, \tilde{\hat{y}}) = 1 - \frac{\sum_{i=0}^{n - 1} (y_i - \tilde{y}_i)^2}{\sum_{i=0}^{n - 1} (y_i - \bar{y})^2}, +\] +!et +where we have defined the mean value of $\hat{y}$ as +!bt +\[ +\bar{y} = \frac{1}{n} \sum_{i=0}^{n - 1} y_i. +\] +!et + + +===== The mean absolute error and other functions in _scikit-learn_ ===== + +Another quantity will meet again in our discussions of regression analysis is + mean absolute error (MAE), a risk metric corresponding to the expected value of the absolute error loss or what we call the $l1$-norm loss. In our discussion above we presented the relative error. +The MAE is defined as follows +!bt +\[ +\text{MAE}(\hat{y}, \hat{\tilde{y}}) = \frac{1}{n} \sum_{i=0}^{n-1} \left| y_i - \tilde{y}_i \right|. +\] +!et +Finally we present the +squared logarithmic (quadratic) error +!bt +\[ +\text{MSLE}(\hat{y}, \hat{\tilde{y}}) = \frac{1}{n} \sum_{i=0}^{n - 1} (\log_e (1 + y_i) - \log_e (1 + \tilde{y}_i) )^2, +\] +!et + +where $\log_e (x)$ stands for the natural logarithm of $x$. This error +estimate is best to use when targets having exponential growth, such +as population counts, average sales of a commodity over a span of +years etc. + + + +===== Cubic polynomial in _scikit-learn_ ===== + +We will discuss in more +detail these and other functions in the various lectures. We conclude this part with another example. Instead of +a linear $x$-dependence we study now a cubic polynomial and use the polynomial regression analysis tools of scikit-learn. +Add description of the various python commands. + +!bc pycod +import matplotlib.pyplot as plt +import numpy as np +import random +from sklearn.linear_model import Ridge +from sklearn.preprocessing import PolynomialFeatures +from sklearn.pipeline import make_pipeline +from sklearn.linear_model import LinearRegression + +x=np.linspace(0.02,0.98,200) +noise = np.asarray(random.sample((range(200)),200)) +y=x**3*noise +yn=x**3*100 +poly3 = PolynomialFeatures(degree=3) +X = poly3.fit_transform(x[:,np.newaxis]) +clf3 = LinearRegression() +clf3.fit(X,y) + +Xplot=poly3.fit_transform(x[:,np.newaxis]) +poly3_plot=plt.plot(x, clf3.predict(Xplot), label='Cubic Fit') +plt.plot(x,yn, color='red', label="True Cubic") +plt.scatter(x, y, label='Data', color='orange', s=15) +plt.legend() +plt.show() + +def error(a): + for i in y: + err=(y-yn)/yn + return abs(np.sum(err))/len(err) + +print (error(y)) +!ec + +Using _R_, we can perform similar studies. + + + + + + + +===== Polynomial Regression ===== +!bc pycod +# Importing various packages +from math import exp, sqrt +from random import random, seed +import numpy as np +import matplotlib.pyplot as plt + +m = 100 +x = 2*np.random.rand(m,1)+4. +y = 4+3*x*x+ +x-np.random.randn(m,1) + +xb = np.c_[np.ones((m,1)), x] +theta = np.linalg.inv(xb.T.dot(xb)).dot(xb.T).dot(y) +xnew = np.array([[0],[2]]) +xbnew = np.c_[np.ones((2,1)), xnew] +ypredict = xbnew.dot(theta) + +plt.plot(xnew, ypredict, "r-") +plt.plot(x, y ,'ro') +plt.axis([0,2.0,0, 15.0]) +plt.xlabel(r'$x$') +plt.ylabel(r'$y$') +plt.title(r'Random numbers ') +plt.show() + +!ec + + +===== Linking the regression analysis with a statistical interpretation ===== + +Before we proceed, and to link with our discussions of Bayesian statistics to come, it is useful the derive the standard regression analysis equations using a statistical interpretation. This allows us also to derive quantities like the variance and other expectation values in a rather straightforward way. + +It is assumed that $\varepsilon_i +\sim \mathcal{N}(0, \sigma^2)$ and the $\varepsilon_{i}$ are +independent, i.e.: +!bt +\begin{align*} +\mbox{Cov}(\varepsilon_{i_1}, +\varepsilon_{i_2}) & = \left\{ \begin{array}{lcc} \sigma^2 & \mbox{if} +& i_1 = i_2, \\ 0 & \mbox{if} & i_1 \not= i_2. \end{array} \right. +\end{align*} +!et +The randomness of $\varepsilon_i$ implies that +$\mathbf{Y}_i$ is also a random variable. In particular, +$\mathbf{Y}_i$ is normally distributed, because $\varepsilon_i \sim +\mathcal{N}(0, \sigma^2)$ and $\mathbf{X}_{i,\ast} \, \beta$ is a +non-random scalar. To specify the parameters of the distribution of +$\mathbf{Y}_i$ we need to calculate its first two moments. + + +===== Expectation value and variance ===== + +Its expectation equals: +!bt +\begin{align*} +\mathbb{E}(Y_i) & = +\mathbb{E}(\mathbf{X}_{i, \ast} \, \beta) + \mathbb{E}(\varepsilon_i) +\, \, \, = \, \, \, \mathbf{X}_{i, \ast} \, \beta, +\end{align*} +!et +while +its variance is +!bt +\begin{align*} \mbox{Var}(Y_i) & = \mathbb{E} \{ [Y_i +- \mathbb{E}(Y_i)]^2 \} \, \, \, = \, \, \, \mathbb{E} ( Y_i^2 ) - +[\mathbb{E}(Y_i)]^2 \\ & = \mathbb{E} [ ( \mathbf{X}_{i, \ast} \, +\beta + \varepsilon_i )^2] - ( \mathbf{X}_{i, \ast} \, \beta)^2 \\ & += \mathbb{E} [ ( \mathbf{X}_{i, \ast} \, \beta)^2 + 2 \varepsilon_i +\mathbf{X}_{i, \ast} \, \beta + \varepsilon_i^2 ] - ( \mathbf{X}_{i, +\ast} \, \beta)^2 \\ & = ( \mathbf{X}_{i, \ast} \, \beta)^2 + 2 +\mathbb{E}(\varepsilon_i) \mathbf{X}_{i, \ast} \, \beta + +\mathbb{E}(\varepsilon_i^2 ) - ( \mathbf{X}_{i, \ast} \, \beta)^2 +\\ & = \mathbb{E}(\varepsilon_i^2 ) \, \, \, = \, \, \, +\mbox{Var}(\varepsilon_i) \, \, \, = \, \, \, \sigma^2. +\end{align*} +!et +Hence, $Y_i \sim \mathcal{N}( \mathbf{X}_{i, \ast} \, \beta, \sigma^2)$. + + + + + + + +===== The singular value decompostion ===== +!bblock + + +A general +$m\times n$ matrix $\hat{A}$ can be written in terms of a diagonal +matrix $\hat{D}$ of dimensionality $n\times n$ and two orthognal +matrices $\hat{U}$ and $\hat{V}$, where the first has dimensionality +$m \times m$ and the last dimensionality $n\times n$. +We have then +!bt +\[ +\hat{A} = \hat{U}\hat{D}\hat{V}^T +\] +!et +!eblock + + + + + + + + + + + + + + + + +===== From standard regression to Ridge regressions ===== + +One of the typical problems we encounter with linear regression, in particular +when the matrix $\hat{X}$ (our so-called design matrix) is high-dimensional, +are problems with near singular or singular matrices. The column vectors of $\hat{X}$ +may be linearly dependent, normally referred to as super-collinearity. +This means that the matrix may be rank deficient and it is basically impossible to +to model the data using linear regression. As an example, consider the matrix +!bt +\begin{align*} +\mathbf{X} & = \left[ +\begin{array}{rrr} +1 & -1 & 2 +\\ +1 & 0 & 1 +\\ +1 & 2 & -1 +\\ +1 & 1 & 0 +\end{array} \right] +\end{align*} +!et + +The columns of $\hat{X}$ are linearly dependent. We se this easily since the +the first column is the row-wise sum of the other two columns. The rank (more correct, +the column rank) of a matrix is the dimension of the space spanned by the +column vectors. Hence, the rank of $\mathbf{X}$ is equal to the number +of linearly independent columns. In this particular case the matrix has rank 2. + +Super-collinearity of an $(n \times p)$-dimensional design matrix $\mathbf{X}$ implies +that the inverse of the matrix $\hat{X}^T\hat{x}$ (the matrix we needto invert to solve the linear regression equations) is non-invertible. If we have a square matrix that does not have an inverse, we say this matrix singular. The example here demonstrates this +!bt +\begin{align*} +\hat{X} & = \left[ +\begin{array}{rr} +1 & -1 +\\ +1 & -1 +\end{array} \right]. +\end{align*} +!et +We see easily that $\mbox{det}(\hat{X}) = x_{11} x_{22} - x_{12} x_{21} = 1 \times (-1) - 1 \times (-1) = 0$. Hence, $\mathbf{X}$ is singular and its inverse is undefined. +This is equivalent to saying that the matrix $\hat{X}$ has at least an eigenvalue which is zero. + + +===== Fixing the singularity ===== + +If our design matrix $\hat{X}$ which enters the linear regression problem +!bt +\begin{align} +\hat{\beta} & = (\hat{X}^{T} \hat{X})^{-1} \hat{X}^{T} \hat{y}, +\end{align} +!et +has linearly dependent column vectors, we will not be able to compute the inverse +of $\hat{X}^T\hat{X}$ and we cannot find the parameters (estimators) $\beta_i$. +The estimators are only well-defined if $(\hat{X}^{T}\hat{X})^{-1}$ exits. +This is more likely to happen when the matrix $\hat{X}$ is high-dimensional. In this case it is likely to encounter a situation where +the regression parameters $\beta_i$ cannot be estimated. + +The *ad hoc* approach which was introduced in the 70s was simply to add a diagonal component to the matrix to invert, that is we change +!bt +\[ +\hat{X}^{T} \hat{X} \rightarrow \hat{X}^{T} \hat{X}+\lambda \hat{I}, +\] +!et +where $\hat{I}$ is the identity matrix. + + + + + + +===== Fitting vs. predicting when data is in the model class ===== + +We start by considering the case +$f(x)=2x$. + +Then the data is clearly generated by a model that is contained within +all three model classes we are using to make predictions (linear +models, third order polynomials, and tenth order polynomials). + +Run the code for the following cases: + +o For $f(x)=2x$ , $Ntrain=10$ and $\sigma =0$ (noiseless case), train the three classes of models (linear, third-order polynomial, and tenth order polynomial) for a training set when $x \in [0,1]$ . Make graphs comparing fits for different order of polynomials. Which model fits the data the best? +o Do you think that the data that has the least error on the training set will also make the best predictions? Why or why not? Can you try to discuss and formalize your intuition? What can go right and what can go wrong? +o Check your answer by seeing how well your fits predict newly generated test data (including on data outside the range you fit on, for example $x \in [0,1.2]$ ) using the code below. How well do you do on points in the range of x where you trained the model? How about points outside the original training data set? +o Repeat the above for $f(x)=2x$ , $Ntrain=10$ , and $\sigma=1$ . What changes? +Repeat the exercises above for $f(x)=2x$ , $Ntrain=100$ , and $\sigma=1$ . What changes? +Summarize what you have learned about the relationship between model complexity (number of parameters), goodness of fit on training data, and the ability to predict well. + + + +===== Fitting versus predicting when data is not in the model class ===== + +Thus far, we have considered the case where the data is generated using a model contained in the model class. Now consider $f(x)=2x-10x^5+15x^{10}$ . Notice that the for linear and third-order polynomial the true model $f(x)$ is not contained in model class. + +o Do better fits lead to better predictions? +o What is the relationship between the true model for generating the data and the model class that has the most predictive power? How is this related to the model complexity? How does this depend on the number of data points $Ntrain$ and $\sigma$? +Summarize what you think you learned about the relationship of knowing the true model class and predictive power. + + +===== An example code without the model assessment part ===== + +!bc pycod +import numpy as np +import sklearn as sk +from sklearn import datasets, linear_model +from sklearn.preprocessing import PolynomialFeatures + +import matplotlib as mpl +from matplotlib import pyplot as plt + +%matplotlib notebook + +# The Training Data + +N_train=100 + +sigma_train=1; + +# Train on integers +x=np.linspace(0.05,0.95,N_train) +# Draw random noise +s = sigma_train*np.random.randn(N_train) + +#linear +y=2*x+s + +#Tenth Order +#y=2*x-10*x**5+15*x**10+s + +p1=plt.plot(x,y, "o",ms=15, label='Training') + +#Linear Regression +# Create linear regression object +clf = linear_model.LinearRegression() + +# Train the model using the training sets +clf.fit(x[:, np.newaxis], y) +# The coefficients + +xplot=np.linspace(0.02,0.98,200) +linear_plot=plt.plot(xplot, clf.predict(xplot[:, np.newaxis]),label='Linear') + +#Polynomial Regression + + +poly3 = PolynomialFeatures(degree=3) +X = poly3.fit_transform(x[:,np.newaxis]) +clf3 = linear_model.LinearRegression() +clf3.fit(X,y) + + +Xplot=poly3.fit_transform(xplot[:,np.newaxis]) +poly3_plot=plt.plot(xplot, clf3.predict(Xplot), label='Poly 3') + + + +#poly5 = PolynomialFeatures(degree=5) +#X = poly5.fit_transform(x[:,np.newaxis]) +#clf5 = linear_model.LinearRegression() +#clf5.fit(X,y) + +#Xplot=poly5.fit_transform(xplot[:,np.newaxis]) +#plt.plot(xplot, clf5.predict(Xplot), 'r--',linewidth=1) + +poly10 = PolynomialFeatures(degree=10) +X = poly10.fit_transform(x[:,np.newaxis]) +clf10 = linear_model.LinearRegression() +clf10.fit(X,y) + +Xplot=poly10.fit_transform(xplot[:,np.newaxis]) +poly10_plot=plt.plot(xplot, clf10.predict(Xplot), label='Poly 10') + +axes = plt.gca() +axes.set_ylim([-7,7]) + +handles, labels=axes.get_legend_handles_labels() +plt.legend(handles,labels, loc='lower center') +plt.xlabel("$x$") +plt.ylabel("$y$") +Title="$N=$"+str(N_train)+", $\sigma=$"+str(sigma_train) +plt.title(Title+" (train)") +plt.tight_layout() +plt.show() + +!ec + + +===== Generating test data ===== +!bc pycod +# Generate Test Data + +#Number of test data +N_test=20 + +sigma_test=sigma_train + +max_x=1.2 +x_test=max_x*np.random.random(N_test) +# Draw random noise +s_test = sigma_test*np.random.randn(N_test) + +#Linear +y_test=2*x_test+s_test +#Tenth order +#y_test=2*x_test-10*x_test**5+15*x_test**10+s_test + +#Make design matrices for prediction +x_plot=np.linspace(0,max_x, 200) +X3 = poly3.fit_transform(x_plot[:,np.newaxis]) +X10 = poly10.fit_transform(x_plot[:,np.newaxis]) + +%matplotlib notebook + +fig = plt.figure() +p1=plt.plot(x_test,y_test.transpose(), 'o', ms=12, label='data') +p2=plt.plot(x_plot,clf.predict(x_plot[:,np.newaxis]), label='linear') +p3=plt.plot(x_plot,clf3.predict(X3), label='3rd order') +p10=plt.plot(x_plot,clf10.predict(X10), label='10th order') + + +plt.legend(loc=2) +plt.xlabel('$x$') +plt.ylabel('$y$') +plt.legend(loc='best') +plt.title(Title+" (pred.)") +plt.tight_layout() +plt.show() + + +!ec + + +===== How can we effectively evaluate the various models? ===== + +In Ridge regression and the subsequent discussion of its properties +the bias or penalty parameter is considered known or `given'. In +practice, it is unknown and the user needs to make an informed +decision on its value. How do we do that? Much of the same considerations apply to the Lasso method. + + +===== Code examples for Ridge and Lasso Regression ===== + +!bc pycod +import matplotlib.pyplot as plt +import numpy as np +from sklearn import linear_model +from sklearn.linear_model import LinearRegression +from sklearn.metrics import mean_squared_error, r2_score + +#creating data with random noise +x=np.arange(50) + +delta=np.random.uniform(-2.5,2.5, size=(50)) +np.random.shuffle(delta) +y =0.5*x+5+delta + +#arranging data into 2x50 matrix +a=np.array(x) #inputs +b=np.array(y) #outputs + +#Split into training and test +X_train=a[:37, np.newaxis] +X_test=a[37:, np.newaxis] +y_train=b[:37] +y_test=b[37:] + +print ("X_train: ", X_train.shape) +print ("y_train: ", y_train.shape) +print ("X_test: ", X_test.shape) +print ("y_test: ", y_test.shape) + +print ("------------------------------------") + +print ("Ordinary Least Squares") +#Add Ordinary Least Squares fit +reg=LinearRegression() +reg.fit(X_train, y_train) +pred=reg.predict(X_test) +print ("Prediction Shape: ", pred.shape) + +print('Coefficients: \n', reg.coef_) +# The mean squared error +print("Mean squared error: %.2f" + % mean_squared_error(y_test, pred)) +# Explained variance score: 1 is perfect prediction +print('Variance score: %.2f' % r2_score(y_test, pred)) + +#plot +plt.scatter(X_test,y_test,color='green', label="Training Data") +plt.plot(X_test, pred, color='black', label="Fit Line") +plt.legend() +plt.show() + +print ("------------------------------------") + +print ("Ridge Regression") + +ridge=linear_model.RidgeCV(alphas=[0.1,1.0,10.0]) +ridge.fit(X_train,y_train) +print ("Ridge Coefficient: ",ridge.coef_) +print ("Ridge Intercept: ", ridge.intercept_) +#Look into graphing with Ridge fit + +print ("------------------------------------") + +print ("Lasso") +lasso=linear_model.Lasso(alpha=0.1) +lasso.fit(X_train,y_train) +predl=lasso.predict(X_test) +print("Lasso Coefficient: ", lasso.coef_) +print("Lasso Intercept: ", lasso.intercept_) +plt.scatter(X_test,y_test,color='green', label="Training Data") +plt.plot(X_test, predl, color='blue', label="Lasso") +plt.legend() +plt.show() +!ec + + + + + + +===== A second-order polynomial with Ridge and Lasso ===== +!bc pycod +import numpy as np +import matplotlib.pyplot as plt +from sklearn.linear_model import Ridge +from sklearn.metrics import r2_score + +np.random.seed(4155) + +n_samples = 100 + +x = np.random.rand(n_samples,1) +y = 5*x*x + 0.1*np.random.rand(n_samples,1) + +# Centering x and y. +x_ = x - np.mean(x) +y_ = y - np.mean(y) # beta_0 = mean(y) + +X = np.c_[np.ones((n_samples,1)), x, x**2] +X_ = np.c_[x_, x_**2] + + +### 1. +lmb_values = [1e-4, 1e-3, 1e-2, 10, 1e2, 1e4] +num_values = len(lmb_values) + +## Ridge-regression of centered and not centered data +beta_ridge = np.zeros((3,num_values)) +beta_ridge_centered = np.zeros((3,num_values)) + +I3 = np.eye(3) +I2 = np.eye(2) + +for i,lmb in enumerate(lmb_values): + beta_ridge[:,i] = (np.linalg.inv( X.T @ X + lmb*I3) @ X.T @ y).flatten() + beta_ridge_centered[1:,i] = (np.linalg.inv( X_.T @ X_ + lmb*I2) @ X_.T @ y_).flatten() + +# sett beta_0 = np.mean(y) +beta_ridge_centered[0,:] = np.mean(y) + +## OLS (ordinary least squares) solution +beta_ls = np.linalg.inv( X.T @ X ) @ X.T @ y + +## Evaluate the models +pred_ls = X @ beta_ls +pred_ridge = X @ beta_ridge +pred_ridge_centered = X_ @ beta_ridge_centered[1:] + beta_ridge_centered[0,:] + +## Plot the results + +# Sorting +sort_ind = np.argsort(x[:,0]) + +x_plot = x[sort_ind,0] +x_centered_plot = x_[sort_ind,0] + +pred_ls_plot = pred_ls[sort_ind,0] +pred_ridge_plot = pred_ridge[sort_ind,:] +pred_ridge_centered_plot = pred_ridge_centered[sort_ind,:] + +# Plott not centered +plt.plot(x_plot,pred_ls_plot,label='ls') + +for i in range(num_values): + plt.plot(x_plot,pred_ridge_plot[:,i],label='ridge, lmb=%g'%lmb_values[i]) + +plt.plot(x,y,'ro') + +plt.title('linear regression on un-centered data') +plt.legend() + +# Plott centered +plt.figure() + +for i in range(num_values): + plt.plot(x_centered_plot,pred_ridge_centered_plot[:,i],label='ridge, lmb=%g'%lmb_values[i]) + +plt.plot(x_,y,'ro') + +plt.title('linear regression on centered data') +plt.legend() + + +# 2. + +pred_ridge_scikit = np.zeros((n_samples,num_values)) +for i,lmb in enumerate(lmb_values): + pred_ridge_scikit[:,i] = (Ridge(alpha=lmb,fit_intercept=False).fit(X,y).predict(X)).flatten() # fit_intercept=False fordi bias er allerede i X + +plt.figure() + +plt.plot(x_plot,pred_ls_plot,label='ls') + +for i in range(num_values): + plt.plot(x_plot,pred_ridge_scikit[sort_ind,i],label='scikit-ridge, lmb=%g'%lmb_values[i]) + +plt.plot(x,y,'ro') +plt.legend() +plt.title('linear regression using scikit') + +plt.show() + +### R2-score of the results +for i in range(num_values): + print('lambda = %g'%lmb_values[i]) + print('r2 for scikit: %g'%r2_score(y,pred_ridge_scikit[:,i])) + print('r2 for own code, not centered: %g'%r2_score(y,pred_ridge[:,i])) + print('r2 for own, centered: %g\n'%r2_score(y,pred_ridge_centered[:,i])) + + +!ec + + + + +===== Resampling methods ===== +!bblock +Resampling methods are an indispensable tool in modern +statistics. They involve repeatedly drawing samples from a training +set and refitting a model of interest on each sample in order to +obtain additional information about the fitted model. For example, in +order to estimate the variability of a linear regression fit, we can +repeatedly draw different samples from the training data, fit a linear +regression to each new sample, and then examine the extent to which +the resulting fits differ. Such an approach may allow us to obtain +information that would not be available from fitting the model only +once using the original training sample. +!eblock + + +===== Resampling approaches can be computationally expensive ===== +!bblock +Resampling approaches can be computationally expensive, because they +involve fitting the same statistical method multiple times using +different subsets of the training data. However, due to recent +advances in computing power, the computational requirements of +resampling methods generally are not prohibitive. In this chapter, we +discuss two of the most commonly used resampling methods, +cross-validation and the bootstrap. Both methods are important tools +in the practical application of many statistical learning +procedures. For example, cross-validation can be used to estimate the +test error associated with a given statistical learning method in +order to evaluate its performance, or to select the appropriate level +of flexibility. The process of evaluating a model’s performance is +known as model assessment, whereas the process of selecting the proper +level of flexibility for a model is known as model selection. The +bootstrap is widely used. +!eblock + + +===== Why resampling methods ? ===== +!bblock Statistical analysis + * Our simulations can be treated as *computer experiments*. This is particularly the case for Monte Carlo methods + * The results can be analysed with the same statistical tools as we would use analysing experimental data. + * As in all experiments, we are looking for expectation values and an estimate of how accurate they are, i.e., possible sources for errors. + + +!eblock + + +===== Statistical analysis ===== +!bblock + * As in other experiments, many numerical experiments have two classes of errors: + * Statistical errors + * Systematical errors + * Statistical errors can be estimated using standard tools from statistics + * Systematical errors are method specific and must be treated differently from case to case. +!eblock + + +===== Statistics ===== +!bblock +The *probability distribution function (PDF)* is a function +$p(x)$ on the domain which, in the discrete case, gives us the +probability or relative frequency with which these values of $X$ occur: +!bt +\[ +p(x) = \mathrm{prob}(X=x) +\] +!et +In the continuous case, the PDF does not directly depict the +actual probability. Instead we define the probability for the +stochastic variable to assume any value on an infinitesimal interval +around $x$ to be $p(x)dx$. The continuous function $p(x)$ then gives us +the *density* of the probability rather than the probability +itself. The probability for a stochastic variable to assume any value +on a non-infinitesimal interval $[a,\,b]$ is then just the integral: +!bt +\[ +\mathrm{prob}(a\leq X\leq b) = \int_a^b p(x)dx +\] +!et +Qualitatively speaking, a stochastic variable represents the values of +numbers chosen as if by chance from some specified PDF so that the +selection of a large set of these numbers reproduces this PDF. +!eblock + + + + + + +===== Log-likelihood ===== + +A popular strategy is to choose a penalty parameter that yields a good +but parsimonious model. Information criteria measure the balance +between model fit and model complexity. One possibility is Aikaike's +information criterion (AIC). +The AIC measures model fit by the log-likelihood +and model complexity is measured by the number of parameters used by +the model. The number of model parameters in regular regression simply +corresponds to the number of covariates in the model. Or, by the +degrees of freedom consumed by the model, which is equivalent to the +trace of the hat matrix. For ridge regression it thus seems natural to +define model complexity analogously by the trace of the ridge hat +matrix. This yields the AIC for the linear regression model with ridge +estimates: + + +!bt +\begin{align*} +\mbox{AIC}(\lambda) & = 2 \, p - 2 \log(\hat{L}) +\\ +& = 2 \, \mbox{tr} [\mathbf{H}(\lambda)] - 2 \log\{L[\hat{\beta}(\lambda), \hat{\sigma}^2(\lambda)]\} +\\ +& = 2 \, \sum_{j=1}^p \frac{d_{jj}^2}{d_{jj}^2 + \lambda} ++ 2 n \, \log[\sqrt{2 \, \pi} \, \hat{\sigma}(\lambda)] + \frac{1}{\hat{\sigma}^2(\lambda)} \sum_{i=1}^n [y_i - \mathbf{X}_{i, \ast} \, \hat{\beta}(\lambda)]^2. +\end{align*} +!et +The value of $\lambda$ which minimizes $\mbox{AIC}(\lambda)$ corresponds to the `optimal' balance of model complexity and overfitting. + + + +===== Cross-validation ===== + +Instead of choosing the penalty parameter to balance model fit with +model complexity, cross-validation requires it (i.e. the penalty +parameter) to yield a model with good prediction +performance. Commonly, this performance is evaluated on novel +data. Novel data need not be easy to come by and one has to make do +with the data at hand. The setting of `original' and novel data is +then mimicked by sample splitting: the data set is divided into two +(groups of samples). One of these two data sets, called the *training +set*, plays the role of `original' data on which the model is +built. The second of these data sets, called the *test set*, plays the +role of the `novel' data and is used to evaluate the prediction +performance (often operationalized as the log-likelihood or the +prediction error or its square or the R2 score) of the model built on the training data set. This +procedure (model building and prediction evaluation on training and +test set, respectively) is done for a collection of possible penalty +parameter choices. The penalty parameter that yields the model with +the best prediction performance is to be preferred. The thus obtained +performance evaluation depends on the actual split of the data set. To +remove this dependence the data set is split many times into a +training and test set. For each split the model parameters are +estimated for all choices of $\lambda$ using the training data and +estimated parameters are evaluated on the corresponding test set. The +penalty parameter that on average over the test sets performs best (in +some sense) is then selected. + + + +===== Computationally expensive ===== + +The validation set approach is conceptually simple and is easy to implement. But it has two potential drawbacks: + +* The validation estimate of the test error rate can be highly variable, depending on precisely which observations are included in the training set and which observations are included in the validation set. + +* In the validation approach, only a subset of the observations, those that are included in the training set rather than in the validation set are used to fit the model. Since statistical methods tend to perform worse when trained on fewer observations, this suggests that the validation set error rate may tend to overestimate the test error rate for the model fit on the entire data set. + + + + +===== Various steps in cross-validation ===== + +When the repetitive splitting of the data set is done randomly, +samples may accidently end up in a fast majority of the splits in +either training or test set. Such samples may have an unbalanced +influence on either model building or prediction evaluation. To avoid +this $k$-fold cross-validation structures the data splitting. The +samples are divided into $k$ more or less equally sized exhaustive and +mutually exclusive subsets. In turn (at each split) one of these +subsets plays the role of the test set while the union of the +remaining subsets constitutes the training set. Such a splitting +warrants a balanced representation of each sample in both training and +test set over the splits. Still the division into the $k$ subsets +involves a degree of randomness. This may be fully excluded when +choosing $k=n$. This particular case is referred to as leave-one-out +cross-validation (LOOCV). + + +===== How to set up the cross-validation for Ridge and/or Lasso ===== + +* Define a range of interest for the penalty parameter. + +* Divide the data set into training and test set comprising samples $\{1, \ldots, n\} \setminus i$ and $\{ i \}$, respectively. + +* Fit the linear regression model by means of ridge estimation for each $\lambda$ in the grid using the training set, and the corresponding estimate of the error variance $\hat{\sigma}_{-i}^2(\lambda)$, as +!bt +\begin{align*} +\hat{\beta}_{-i}(\lambda) & = ( \hat{X}_{-i, \ast}^{\top} +\hat{X}_{-i, \ast} + \lambda \hat{I}_{pp})^{-1} +\hat{X}_{-i, \ast}^{\top} \hat{y}_{-i} +\end{align*} +!et + +* Evaluate the prediction performance of these models on the test set by $\log\{L[y_i, \hat{X}_{i, \ast}; \hat{\beta}_{-i}(\lambda), \hat{\sigma}_{-i}^2(\lambda)]\}$. Or, by the prediction error $|y_i - \hat{X}_{i, \ast} \hat{\beta}_{-i}(\lambda)|$, the relative error, the error squared or the R2 score function. + +* Repeat the first three steps such that each sample plays the role of the test set once. + +* Average the prediction performances of the test sets at each grid point of the penalty bias/parameter by computing the *cross-validated log-likelihood*. It is an estimate of the prediction performance of the model corresponding to this value of the penalty parameter on novel data. It is defined as +!bt +\begin{align*} +\frac{1}{n} \sum_{i = 1}^n \log\{L[y_i, \mathbf{X}_{i, \ast}; \hat{\beta}_{-i}(\lambda), \hat{\sigma}_{-i}^2(\lambda)]\}. +\end{align*} +!et + +* The value of the penalty parameter that maximizes the cross-validated log-likelihood is the value of choice. Or we can use the MSE or the R2 score functions. + + +===== Predicted Residual Error Sum of Squares ===== +!bblock +Another approach in the LOOCV scheme is to the use the so-called Predicted Residual Error Sum of Squares (PRESS). + +We can define the optimal penalty parameter to minimize +!bt +\begin{align*} +\lambda_{\mbox{{\tiny opt}}} = \arg \min_{\lambda} \frac{1}{n} \sum_{i=1}^n [y_i - \hat{X}_{i, \ast} \hat{\beta}_{-i}(\lambda)]^2. +\end{align*} +!et + +The LOOCV prediction performance can be +expressed analytically in terms of the known quantities derived from +the design matrix and the parameters $\beta$. +!eblock + + + +===== Resampling methods: Jackknife and Bootstrap ===== + +Two famous +resampling methods are the _independent bootstrap_ and _the jackknife_. + +The jackknife is a special case of the independent bootstrap. Still, the jackknife was made +popular prior to the independent bootstrap. And as the popularity of +the independent bootstrap soared, new variants, such as _the dependent bootstrap_. + +The Jackknife and independent bootstrap work for +independent, identically distributed random variables. +If these conditions are not +satisfied, the methods will fail. Yet, it should be said that if the data are +independent, identically distributed, and we only want to estimate the +variance of $\overline{X}$ (which often is the case), then there is no +need for bootstrapping. + + +===== Resampling methods: Jackknife ===== + +The Jackknife works by making many replicas of the estimator $\widehat{\theta}$. +The jackknife is a resampling method, we explained that this happens by scrambling the data in some way. When using the jackknife, this is done by systematically leaving out one observation from the vector of observed values $\hat{x} = (x_1,x_2,\cdots,X_n)$. +Let $\hat{x}_i$ denote the vector +!bt +\[ +\hat{x}_i = (x_1,x_2,\cdots,x_{i-1},x_{i+1},\cdots,x_n), +\] +!et + +which equals the vector $\hat{x}$ with the exception that observation +number $i$ is left out. Using this notation, define +$\widehat{\theta}_i$ to be the estimator +$\widehat{\theta}$ computed using $\vec{X}_i$. + + +===== Resampling methods: Jackknife estimator ===== + +To get an estimate for the bias and +standard error of $\widehat{\theta}$, use the following +estimators for each component of $\widehat{\theta}$ + +!bt +\[ +\widehat{\mathrm{Bias}}(\widehat \theta,\theta) = (n-1)\left( - \widehat{\theta} + \frac{1}{n}\sum_{i=1}^{n} \widehat \theta_i \right) \qquad \text{and} \qquad \widehat{\sigma}^2_{\widehat{\theta} } = \frac{n-1}{n}\sum_{i=1}^{n}( \widehat{\theta}_i - \frac{1}{n}\sum_{j=1}^{n}\widehat \theta_j )^2. +\] +!et + + + +===== Jackknife code example ===== +!bc pycod +from numpy import * +from numpy.random import randint, randn +from time import time + +def jackknife(data, stat): + n = len(data);t = zeros(n); inds = arange(n); t0 = time() + ## 'jackknifing' by leaving out an observation for each i + for i in range(n): + t[i] = stat(delete(data,i) ) + + # analysis + print("Runtime: %g sec" % (time()-t0)); print("Jackknife Statistics :") + print("original bias std. error") + print("%8g %14g %15g" % (stat(data),(n-1)*mean(t)/n, (n*var(t))**.5)) + + return t + + +# Returns mean of data samples +def stat(data): + return mean(data) + + +mu, sigma = 100, 15 +datapoints = 10000 +x = mu + sigma*random.randn(datapoints) +# jackknife returns the data sample +t = jackknife(x, stat) + +!ec + + + +===== Resampling methods: Bootstrap ===== +!bblock +Bootstrapping is a nonparametric approach to statistical inference +that substitutes computation for more traditional distributional +assumptions and asymptotic results. Bootstrapping offers a number of +advantages: +o The bootstrap is quite general, although there are some cases in which it fails. +o Because it does not require distributional assumptions (such as normally distributed errors), the bootstrap can provide more accurate inferences when the data are not well behaved or when the sample size is small. +o It is possible to apply the bootstrap to statistics with sampling distributions that are difficult to derive, even asymptotically. +o It is relatively simple to apply the bootstrap to complex data-collection plans (such as stratified and clustered samples). +!eblock + + + +===== Resampling methods: Bootstrap background ===== + +Since $\widehat{\theta} = \widehat{\theta}(\hat{X})$ is a function of random variables, +$\widehat{\theta}$ itself must be a random variable. Thus it has +a pdf, call this function $p(\hat{t})$. The aim of the bootstrap is to +estimate $p(\hat{t})$ by the relative frequency of +$\widehat{\theta}$. You can think of this as using a histogram +in the place of $p(\hat{t})$. If the relative frequency closely +resembles $p(\vec{t})$, then using numerics, it is straight forward to +estimate all the interesting parameters of $p(\hat{t})$ using point +estimators. + + + +===== Resampling methods: More Bootstrap background ===== + +In the case that $\widehat{\theta}$ has +more than one component, and the components are independent, we use the +same estimator on each component separately. If the probability +density function of $X_i$, $p(x)$, had been known, then it would have +been straight forward to do this by: +o Drawing lots of numbers from $p(x)$, suppose we call one such set of numbers $(X_1^*, X_2^*, \cdots, X_n^*)$. +o Then using these numbers, we could compute a replica of $\widehat{\theta}$ called $\widehat{\theta}^*$. + +By repeated use of (1) and (2), many +estimates of $\widehat{\theta}$ could have been obtained. The +idea is to use the relative frequency of $\widehat{\theta}^*$ +(think of a histogram) as an estimate of $p(\hat{t})$. + + +===== Resampling methods: Bootstrap approach ===== + +But +unless there is enough information available about the process that +generated $X_1,X_2,\cdots,X_n$, $p(x)$ is in general +unknown. Therefore, "Efron in 1979":"https://projecteuclid.org/euclid.aos/1176344552" asked the +question: What if we replace $p(x)$ by the relative frequency +of the observation $X_i$; if we draw observations in accordance with +the relative frequency of the observations, will we obtain the same +result in some asymptotic sense? The answer is yes. + + +Instead of generating the histogram for the relative +frequency of the observation $X_i$, just draw the values +$(X_1^*,X_2^*,\cdots,X_n^*)$ with replacement from the vector +$\hat{X}$. + + +===== Resampling methods: Bootstrap steps ===== + +The independent bootstrap works like this: + +o Draw with replacement $n$ numbers for the observed variables $\hat{x} = (x_1,x_2,\cdots,x_n)$. +o Define a vector $\hat{x}^*$ containing the values which were drawn from $\hat{x}$. +o Using the vector $\hat{x}^*$ compute $\widehat{\theta}^*$ by evaluating $\widehat \theta$ under the observations $\hat{x}^*$. +o Repeat this process $k$ times. + +When you are done, you can draw a histogram of the relative frequency of $\widehat \theta^*$. This is your estimate of the probability distribution $p(t)$. Using this probability distribution you can estimate any statistics thereof. In principle you never draw the histogram of the relative frequency of $\widehat{\theta}^*$. Instead you use the estimators corresponding to the statistic of interest. For example, if you are interested in estimating the variance of $\widehat \theta$, apply the etsimator $\widehat \sigma^2$ to the values $\widehat \theta ^*$. + + + +===== Code example for the Bootstrap method ===== +The following code starts with a Gaussian distribution with mean value $\mu =100$ and variance $\sigma=15$. We use this to generate the data used in the bootstrap analysis. The bootstrap analysis returns a data set after a given number of bootstrap operations (as many as we have data points). This data set consists of estimated mean values for each bootstrap operation. The histogram generated by the bootstrap method shows that the distribution for these mean values is also a Gaussian, centered around the mean value $\mu=100$ but with standard deviation $\sigma/\sqrt{n}$, where $n$ is the number of bootstrap samples (in this case the same as the number of original data points). The value of the standard deviation is what we expect from the central limit theorem. +!bc pycod +from numpy import * +from numpy.random import randint, randn +from time import time +import matplotlib.mlab as mlab +import matplotlib.pyplot as plt + +# Returns mean of bootstrap samples +def stat(data): + return mean(data) + +# Bootstrap algorithm +def bootstrap(data, statistic, R): + t = zeros(R); n = len(data); inds = arange(n); t0 = time() + + # non-parametric bootstrap + for i in range(R): + t[i] = statistic(data[randint(0,n,n)]) + + # analysis + print("Runtime: %g sec" % (time()-t0)); print("Bootstrap Statistics :") + print("original bias std. error") + print("%8g %8g %14g %15g" % (statistic(data), std(data),\ + mean(t), \ + std(t))) + return t + + +mu, sigma = 100, 15 +datapoints = 10000 +x = mu + sigma*random.randn(datapoints) +# bootstrap returns the data sample t = bootstrap(x, stat, datapoints) +# the histogram of the bootstrapped data n, binsboot, patches = plt.hist(t, 50, normed=1, facecolor='red', alpha=0.75) + +# add a 'best fit' line +y = mlab.normpdf( binsboot, mean(t), std(t)) +lt = plt.plot(binsboot, y, 'r--', linewidth=1) +plt.xlabel('Smarts') +plt.ylabel('Probability') +plt.axis([99.5, 100.6, 0, 3.0]) +plt.grid(True) + +plt.show() + +!ec + + + +===== Resampling methods: Blocking ===== + +The blocking method was made popular by "Flyvbjerg and Pedersen (1989)":"https://aip.scitation.org/doi/10.1063/1.457480" +and has become one of the standard ways to estimate +$V(\widehat{\theta})$ for exactly one $\widehat{\theta}$, namely +$\widehat{\theta} = \overline{X}$. + +Assume $n = 2^d$ for some integer $d>1$ and $X_1,X_2,\cdots, X_n$ is a stationary time series to begin with. +Moreover, assume that the time series is asymptotically uncorrelated. We switch to vector notation by arranging $X_1,X_2,\cdots,X_n$ in an $n$-tuple. Define: +!bt +\begin{align*} +\hat{X} = (X_1,X_2,\cdots,X_n). +\end{align*} +!et + +The strength of the blocking method is when the number of +observations, $n$ is large. For large $n$, the complexity of dependent +bootstrapping scales poorly, but the blocking method does not, +moreover, it becomes more accurate the larger $n$ is. + + +===== Blocking Transformations ===== + We now define +blocking transformations. The idea is to take the mean of subsequent +pair of elements from $\vec{X}$ and form a new vector +$\vec{X}_1$. Continuing in the same way by taking the mean of +subsequent pairs of elements of $\vec{X}_1$ we obtain $\vec{X}_2$, and +so on. +Define $\vec{X}_i$ recursively by: + +!bt +\begin{align} +(\vec{X}_0)_k &\equiv (\vec{X})_k \nonumber \\ +(\vec{X}_{i+1})_k &\equiv \frac{1}{2}\Big( (\vec{X}_i)_{2k-1} + +(\vec{X}_i)_{2k} \Big) \qquad \text{for all} \qquad 1 \leq i \leq d-1 +\end{align} +!et + +The quantity $\vec{X}_k$ is +subject to $k$ _blocking transformations_. We now have $d$ vectors +$\vec{X}_0, \vec{X}_1,\cdots,\vec X_{d-1}$ containing the subsequent +averages of observations. It turns out that if the components of +$\vec{X}$ is a stationary time series, then the components of +$\vec{X}_i$ is a stationary time series for all $0 \leq i \leq d-1$ + +We can then compute the autocovariance, the variance, sample mean, and +number of observations for each $i$. +Let $\gamma_i, \sigma_i^2, +\overline{X}_i$ denote the autocovariance, variance and average of the +elements of $\vec{X}_i$ and let $n_i$ be the number of elements of +$\vec{X}_i$. It follows by induction that $n_i = n/2^i$. + + +===== Blocking Transformations ===== + +Using the +definition of the blocking transformation and the distributive +property of the covariance, it is clear that since $h =|i-j|$ +we can define +!bt +\begin{align} +\gamma_{k+1}(h) &= cov\left( ({X}_{k+1})_{i}, ({X}_{k+1})_{j} \right) \nonumber \\ +&= \frac{1}{4}cov\left( ({X}_{k})_{2i-1} + ({X}_{k})_{2i}, ({X}_{k})_{2j-1} + ({X}_{k})_{2j} \right) \nonumber \\ +&= \frac{1}{2}\gamma_{k}(2h) + \frac{1}{2}\gamma_k(2h+1) \hspace{0.1cm} \mathrm{h = 0} \\ +&=\frac{1}{4}\gamma_k(2h-1) + \frac{1}{2}\gamma_k(2h) + \frac{1}{4}\gamma_k(2h+1) \quad \mathrm{else} +\end{align} +!et + +The quantity $\hat{X}$ is asymptotic uncorrelated by assumption, $\hat{X}_k$ is also asymptotic uncorrelated. Let's turn our attention to the variance of the sample mean $V(\overline{X})$. + + +===== Blocking Transformations, getting there ===== +We have +!bt +\begin{align} +V(\overline{X}_k) = \frac{\sigma_k^2}{n_k} + \underbrace{\frac{2}{n_k} \sum_{h=1}^{n_k-1}\left( 1 - \frac{h}{n_k} \right)\gamma_k(h)}_{\equiv e_k} = \frac{\sigma^2_k}{n_k} + e_k \quad \text{if} \quad \gamma_k(0) = \sigma_k^2. +\end{align} +!et +The term $e_k$ is called the _truncation error_: +!bt +\begin{equation} +e_k = \frac{2}{n_k} \sum_{h=1}^{n_k-1}\left( 1 - \frac{h}{n_k} \right)\gamma_k(h). +\end{equation} +!et +We can show that $V(\overline{X}_i) = V(\overline{X}_j)$ for all $0 \leq i \leq d-1$ and $0 \leq j \leq d-1$. + + +===== Blocking Transformations, final expressions ===== + +We can then wrap up +!bt +\begin{align} +n_{j+1} \overline{X}_{j+1} &= \sum_{i=1}^{n_{j+1}} (\hat{X}_{j+1})_i = \frac{1}{2}\sum_{i=1}^{n_{j}/2} (\hat{X}_{j})_{2i-1} + (\hat{X}_{j})_{2i} \nonumber \\ +&= \frac{1}{2}\left[ (\hat{X}_j)_1 + (\hat{X}_j)_2 + \cdots + (\hat{X}_j)_{n_j} \right] = \underbrace{\frac{n_j}{2}}_{=n_{j+1}} \overline{X}_j = n_{j+1}\overline{X}_j. +\end{align} +!et +By repeated use of this equation we get $V(\overline{X}_i) = V(\overline{X}_0) = V(\overline{X})$ for all $0 \leq i \leq d-1$. This has the consequence that +!bt +\begin{align} +V(\overline{X}) = \frac{\sigma_k^2}{n_k} + e_k \qquad \text{for all} \qquad 0 \leq k \leq d-1. \label{eq:convergence} +\end{align} +!et + +Fyvbjerg and Petersen demonstrated that the sequence +$\{e_k\}_{k=0}^{d-1}$ is decreasing, and conjecture that the term +$e_k$ can be made as small as we would like by making $k$ (and hence +$d$) sufficiently large. The sequence is decreasing (Master of Science thesis by Marius Jonsson, UiO 2018). +It means we can apply blocking transformations until +$e_k$ is sufficiently small, and then estimate $V(\overline{X})$ by +$\widehat{\sigma}^2_k/n_k$. + + + +===== "Code examples for Blocking, Jackknife and bootstrap":"https://github.com/CompPhysics/MachineLearning/tree/master/doc/Programs/ResamplingAnalysisScripts" ===== + +!bc pycod +from sys import argv +from os import mkdir, path +import time +import numpy as np +import matplotlib.pyplot as plt +from matplotlib.ticker import FormatStrFormatter +from matplotlib.font_manager import FontProperties + +# Timing Decorator +def timeFunction(f): + def wrap(*args): + time1 = time.time() + ret = f(*args) + time2 = time.time() + print '%s Function Took: \t %0.3f s' % (f.func_name.title(), (time2-time1)) + return ret + return wrap + +class dataAnalysisClass: + # General Init functions + def __init__(self, fileName, size=0): + self.inputFileName = fileName + self.loadData(size) + self.createOutputFolder() + self.avg = np.average(self.data) + self.var = np.var(self.data) + self.std = np.std(self.data) + + def loadData(self, size=0): + if size != 0: + with open(self.inputFileName) as inputFile: + self.data = np.zeros(size) + for x in xrange(size): + self.data[x] = float(next(inputFile)) + else: + self.data = np.loadtxt(self.inputFileName) + + # Statistical Analysis with Multiple Methods + def runAllAnalyses(self): + if len(self.data) <= 100000: + print "Autocorrelation..." + self.autocorrelation() + print "Bootstrap..." + self.bootstrap() + print "Jackknife..." + self.jackknife() + print "Blocking..." + self.blocking() + + # Standard Autocorrelation + @timeFunction + def autocorrelation(self): + self.acf = np.zeros(len(self.data)/2) + for k in range(0, len(self.data)/2): + self.acf[k] = np.corrcoef(np.array([self.data[0:len(self.data)-k], \ + self.data[k:len(self.data)]]))[0,1] + + # Bootstrap + @timeFunction + def bootstrap(self, nBoots = 1000): + bootVec = np.zeros(nBoots) + for k in range(0,nBoots): + bootVec[k] = np.average(np.random.choice(self.data, len(self.data))) + self.bootAvg = np.average(bootVec) + self.bootVar = np.var(bootVec) + self.bootStd = np.std(bootVec) + + # Jackknife + @timeFunction + def jackknife(self): + jackknVec = np.zeros(len(self.data)) + for k in range(0,len(self.data)): + jackknVec[k] = np.average(np.delete(self.data, k)) + self.jackknAvg = self.avg - (len(self.data) - 1) * (np.average(jackknVec) - self.avg) + self.jackknVar = float(len(self.data) - 1) * np.var(jackknVec) + self.jackknStd = np.sqrt(self.jackknVar) + + # Blocking + @timeFunction + def blocking(self, blockSizeMax = 500): + blockSizeMin = 1 + + self.blockSizes = [] + self.meanVec = [] + self.varVec = [] + + for i in range(blockSizeMin, blockSizeMax): + if(len(self.data) % i != 0): + pass#continue + blockSize = i + meanTempVec = [] + varTempVec = [] + startPoint = 0 + endPoint = blockSize + + while endPoint <= len(self.data): + meanTempVec.append(np.average(self.data[startPoint:endPoint])) + startPoint = endPoint + endPoint += blockSize + mean, var = np.average(meanTempVec), np.var(meanTempVec)/len(meanTempVec) + self.meanVec.append(mean) + self.varVec.append(var) + self.blockSizes.append(blockSize) + + self.blockingAvg = np.average(self.meanVec[-200:]) + self.blockingVar = (np.average(self.varVec[-200:])) + self.blockingStd = np.sqrt(self.blockingVar) + + + + # Plot of Data, Autocorrelation Function and Histogram + def plotAll(self): + self.createOutputFolder() + if len(self.data) <= 100000: + self.plotAutocorrelation() + self.plotData() + self.plotHistogram() + self.plotBlocking() + + # Create Output Plots Folder + def createOutputFolder(self): + self.outName = self.inputFileName[:-4] + if not path.exists(self.outName): + mkdir(self.outName) + + # Plot the Dataset, Mean and Std + def plotData(self): + # Far away plot + font = {'fontname':'serif'} + plt.plot(range(0, len(self.data)), self.data, 'r-', linewidth=1) + plt.plot([0, len(self.data)], [self.avg, self.avg], 'b-', linewidth=1) + plt.plot([0, len(self.data)], [self.avg + self.std, self.avg + self.std], 'g--', linewidth=1) + plt.plot([0, len(self.data)], [self.avg - self.std, self.avg - self.std], 'g--', linewidth=1) + plt.ylim(self.avg - 5*self.std, self.avg + 5*self.std) + plt.gca().yaxis.set_major_formatter(FormatStrFormatter('%.4f')) + plt.xlim(0, len(self.data)) + plt.ylabel(self.outName.title() + ' Monte Carlo Evolution', **font) + plt.xlabel('MonteCarlo History', **font) + plt.title(self.outName.title(), **font) + plt.savefig(self.outName + "/data.eps") + plt.savefig(self.outName + "/data.png") + plt.clf() + + # Plot Histogram of Dataset and Gaussian around it + def plotHistogram(self): + binNumber = 50 + font = {'fontname':'serif'} + count, bins, ignore = plt.hist(self.data, bins=np.linspace(self.avg - 5*self.std, self.avg + 5*self.std, binNumber)) + plt.plot([self.avg, self.avg], [0,np.max(count)+10], 'b-', linewidth=1) + plt.ylim(0,np.max(count)+10) + plt.ylabel(self.outName.title() + ' Histogram', **font) + plt.xlabel(self.outName.title() , **font) + plt.title('Counts', **font) + + #gaussian + norm = 0 + for i in range(0,len(bins)-1): + norm += (bins[i+1]-bins[i])*count[i] + plt.plot(bins, norm/(self.std * np.sqrt(2 * np.pi)) * np.exp( - (bins - self.avg)**2 / (2 * self.std**2) ), linewidth=1, color='r') + plt.savefig(self.outName + "/hist.eps") + plt.savefig(self.outName + "/hist.png") + plt.clf() + + # Plot the Autocorrelation Function + def plotAutocorrelation(self): + font = {'fontname':'serif'} + plt.plot(range(1, len(self.data)/2), self.acf[1:], 'r-') + plt.ylim(-1, 1) + plt.xlim(0, len(self.data)/2) + plt.ylabel('Autocorrelation Function', **font) + plt.xlabel('Lag', **font) + plt.title('Autocorrelation', **font) + plt.savefig(self.outName + "/autocorrelation.eps") + plt.savefig(self.outName + "/autocorrelation.png") + plt.clf() + + def plotBlocking(self): + font = {'fontname':'serif'} + plt.plot(self.blockSizes, self.varVec, 'r-') + plt.ylabel('Variance', **font) + plt.xlabel('Block Size', **font) + plt.title('Blocking', **font) + plt.savefig(self.outName + "/blocking.eps") + plt.savefig(self.outName + "/blocking.png") + plt.clf() + + # Print Stuff to the Terminal + def printOutput(self): + print "\nSample Size: \t", len(self.data) + print "\n=========================================\n" + print "Sample Average: \t", self.avg + print "Sample Variance:\t", self.var + print "Sample Std: \t", self.std + print "\n=========================================\n" + print "Bootstrap Average: \t", self.bootAvg + print "Bootstrap Variance:\t", self.bootVar + print "Bootstrap Error: \t", self.bootStd + print "\n=========================================\n" + print "Jackknife Average: \t", self.jackknAvg + print "Jackknife Variance:\t", self.jackknVar + print "Jackknife Error: \t", self.jackknStd + print "\n=========================================\n" + print "Blocking Average: \t", self.blockingAvg + print "Blocking Variance:\t", self.blockingVar + print "Blocking Error: \t", self.blockingStd, "\n" + +# Initialize the class +if len(argv) > 2: + dataAnalysis = dataAnalysisClass(argv[1], int(argv[2])) +else: + dataAnalysis = dataAnalysisClass(argv[1]) + +# Run Analyses +dataAnalysis.runAllAnalyses() + +# Plot the data +dataAnalysis.plotAll() + +# Print Some Output +dataAnalysis.printOutput() + +!ec + + + +===== The bias-variance tradeoff ===== + +We begin with an unknown function $y=f(x)$ and fix a \emph{hypothesis set} + $\mathcal{H}$ consisting of all functions we are willing to consider, + defined also on the domain of $f$. This set may be uncountably + infinite (e.g.~if there are real-valued parameters to fit). +The + choice of which functions to include in $\mathcal{H}$ usually depends + on our intuition about the problem of interest. The function $f(x)$ + produces a set of pairs $(x_i,y_i)$, $i=1\dots N$, which serve as the + observable data. Our goal is to select a function from the hypothesis + set $h\in\mathcal{H}$ which approximates $f(x)$ as best as possible, + namely, we would like to find $h\in\mathcal{H}$ such that $h\approx + f$ in some strict mathematical sense which we specify below. If this + is possible, we say that we \emph{learned} $f(x)$. But if the + function $f(x)$ can, in principle, take any value on + \emph{unobserved} inputs, how is it possible to learn in any + meaningful sense? + + +===== Training and testing data ===== + +We will discuss the bias-variance tradeoff in the context of continuous predictions such as regression. However, many of the intuitions and ideas discussed here also carry over to classification tasks. Consider a dataset $\mathcal{L}$ consisting of the data $\mathbf{X}_\mathcal{L}=\{(y_j, \boldsymbol{x}_j), j=1\ldots N\}$. Let us assume that the true data is generated from a noisy model +!bt +\[ +y=f(\boldsymbol{x}) + \epsilon +\] +!et +where $\epsilon$ is normally distributed with mean zero and standard deviation $\sigma_\epsilon$. + + +===== Procedure to find a predictor ===== + +We have a statistical procedure (e.g. least-squares regression) for +forming a predictor $\hat{g}_{\mathcal{L}}(\boldsymbol{x})$ that gives the +prediction of our model for a new data point $\boldsymbol{x}$. This estimator +is chosen by minimizing a cost function which we take to be the +squared error + +!bt +\[ + \mathcal{C}( \boldsymbol{X}, \hat{g}(\boldsymbol{x})) = \sum_i (y_i - \hat{g}_\mathcal{L}(\boldsymbol{x}_i))^2. +\] +!et + + +===== What we want ===== + +We are interested in the generalization error on all data drawn from +the true model, not just the error on the particular training dataset +$\mathcal{L}$ that we have in hand. This is just the expectation of +the cost function over many different data sets +$\{\mathcal{L}_j\}$. Denote this expectation value by +$E_{\mathcal{L}}$. In other words, we can view $\hat{g}_{\mathcal{L}}$ +as a stochastic functional that depends on the dataset $\mathcal{L}$ +and we can think of $E_{\mathcal{L}}$ as the expected value of the +functional if we drew an infinite number of datasets $\{\mathcal{L}_1, +\mathcal{L}_2, \ldots \}$. + + + +===== The expected generalization error ===== + +We would also like to average over different instances of the +``noise'' $\epsilon$ and we denote the expectation value over the +noise by $E_\epsilon$. Thus, we can decompose the expected +generalization error as + + +!bt +\begin{align} +E_\mathcal{L, \epsilon}[\mathcal{C}( \boldsymbol{X}, \hat{g}(\boldsymbol{x})) ]&= E_\mathcal{L,\epsilon}\left[ \sum_i ({y}_i - \hat{g}_\mathcal{L}(\boldsymbol{x}_i))^2 \right] \nonumber \\ + &= E_\mathcal{L, \epsilon}\left[ \sum_{i}({y}_i -f(\boldsymbol{x}_i) +f(\boldsymbol{x}_i)- \hat{g}_\mathcal{L}(\boldsymbol{x}_i))^2\right] \nonumber \\ + &= \sum_i E_\epsilon[ ({y}_i -f(\boldsymbol{x}_i))^2 ]+ E_\mathcal{L, \epsilon}[(f(\boldsymbol{x}_i)- \hat{g}_\mathcal{L}(\boldsymbol{x}_i))^2] + 2E_\epsilon[{y}_i -f(\boldsymbol{x}_i)]E_\mathcal{L}[f(\boldsymbol{x}_i)- \hat{g}_\mathcal{L}(\boldsymbol{x}_i)] \nonumber \\ + &=\sum_i \sigma_\epsilon^2 + E_\mathcal{L}[(f(\boldsymbol{x}_i)- \hat{g}_\mathcal{L}(\boldsymbol{x}_i))^2], +\end{align} +!et + +where in the last line we used the fact that our noise has zero mean +and variance $\sigma_\epsilon^2$ and the sum over $i$ applies to all +terms. + + +===== Elaborating a little bit more ===== + +It is also helpful to further decompose the second term as +follows: + +!bt +\begin{align} +E_\mathcal{L}[(f(\boldsymbol{x}_i)- \hat{g}_\mathcal{L}(\boldsymbol{x}_i))^2] &=E_\mathcal{L}[(f(\mathbf{x}_i)-E_\mathcal{L}[\hat{g}_\mathcal{L}(\boldsymbol{x}_i)]+ E_\mathcal{L}[\hat{g}_\mathcal{L}(\boldsymbol{x}_i)]- \hat{g}_\mathcal{L}(\boldsymbol{x}_i))^2] \nonumber \\ +&=E_\mathcal{L}[(f(\boldsymbol{x}_i)-E_\mathcal{L}[\hat{g}_\mathcal{L}(\boldsymbol{x}_i)])^2] + E_\mathcal{L}[( \hat{g}_\mathcal{L}(\boldsymbol{x}_i)-E_\mathcal{L}[\hat{g}_\mathcal{L}(\boldsymbol{x}_i)])^2] \nonumber \\ +&+2E_\mathcal{L}[(f(\boldsymbol{x}_i)-E_\mathcal{L}[\hat{g}_\mathcal{L}(\boldsymbol{x}_i)])( \hat{g}_\mathcal{L}(\boldsymbol{x}_i)-E_\mathcal{L}[\hat{g}_\mathcal{L}(\boldsymbol{x}_i)])] \nonumber \\ +&=(f(\boldsymbol{x}_i)-E_\mathcal{L}[\hat{g}_\mathcal{L}(\boldsymbol{x}_i)])^2+E_\mathcal{L}[( \hat{g}_\mathcal{L}(\boldsymbol{x}_i)-E_\mathcal{L}[\hat{g}_\mathcal{L}(\boldsymbol{x}_i)])^2]. +\end{align} +!et + + +===== The bias ===== + +The first term is called the bias +!bt +\[ +Bias^2= \sum_i (f(\boldsymbol{x}_i)-E_\mathcal{L}[\hat{g}_\mathcal{L}(\boldsymbol{x}_i)])^2 +\] +!et +and measures the deviation of the expectation value of our estimator (i.e. the asymptotic value of our estimator in the infinite data limit) from the true value. + + +===== The variance ===== +The second term is called the variance +!bt +\[ +Var=\sum_i E_\mathcal{L}[( \hat{g}_\mathcal{L}(\boldsymbol{x}_i)-E_\mathcal{L}[\hat{g}_\mathcal{L}(\boldsymbol{x}_i)])^2], +\] +!et + +and measures how much our estimator fluctuates due to finite-sample effects. Combining these expressions, we see that the expected out-of-sample error of our model can be decomposed as +!bt +\[ +E_\mathrm{out}=E_\mathcal{L, \epsilon}[\mathcal{C}( \boldsymbol{X}, \hat{g}(\boldsymbol{x})) ] = Bias^2 + Var + Noise. +\] +!et + +The bias-variance tradeoff summarizes the fundamental tension in +machine learning, particularly supervised learning, between the +complexity of a model and the amount of training data needed to train +it. Since data is often limited, in practice it is often useful to +use a less-complex model with higher bias -- a model whose asymptotic +performance is worse than another model -- because it is easier to +train and less sensitive to sampling noise arising from having a +finite-sized training dataset (smaller variance). + + +===== Summing up ===== + +The above equations tell us that in +order to minimize the expected test error, we need to select a +statistical learning method that simultaneously achieves low variance +and low bias. Note that variance is inherently a nonnegative quantity, +and squared bias is also nonnegative. Hence, we see that the expected +test MSE can never lie below $Var(\epsilon)$, the irreducible error. + + +What do we mean by the variance and bias of a statistical learning +method? The variance refers to the amount by which our model would change if we +estimated it using a different training data set. Since the training +data are used to fit the statistical learning method, different +training data sets will result in a different estimate. But ideally the +estimate for our model should not vary too much between training +sets. However, if a method has high variance then small changes in +the training data can result in large changes in the model. In general, more +flexible statistical methods have higher variance. + + + +===== The one-dimensional Ising model, project 2 ===== + +The one-dimensional Ising model with nearest neighbor interaction, no external field and a constant coupling constant $J$ is given by + +!bt +\begin{align} + H = -J \sum_{k}^L s_k s_{k + 1}, +\end{align} +!et +where $s_i \in \{-1, 1\}$ and $s_{N + 1} = s_1$. The number of spins in the system is determined by $L$. For the one-dimensional system there is no phase transition. + +We will look at a system of $L = 40$ spins with a coupling constant of $J = 1$. To get enough training data we will generate 10000 states with their respective energies. + + +!bc pycod +import numpy as np +import matplotlib.pyplot as plt +from mpl_toolkits.axes_grid1 import make_axes_locatable +import seaborn as sns +import scipy.linalg as scl +from sklearn.model_selection import train_test_split +import sklearn.linear_model as skl +import tqdm +sns.set(color_codes=True) +cmap_args=dict(vmin=-1., vmax=1., cmap='seismic') + +L = 40 +n = int(1e4) + +spins = np.random.choice([-1, 1], size=(n, L)) +J = 1.0 + +energies = np.zeros(n) + +for i in range(n): + energies[i] = - J * np.dot(spins[i], np.roll(spins[i], 1)) +!ec + +Here we use linear (ordinary least squares), ridge and LASSO +regression to predict the energy in the nearest neighbor +one-dimensional Ising model on a ring, i.e., the endpoints wrap +around. We will use the linear regression models to fit a value for +the coupling constant to achieve this. + + +===== Reformulating the problem to suit regression ===== + +A more general form for the one-dimensional Ising model is + +!bt +\begin{align} + H = - \sum_j^L \sum_k^L s_j s_k J_{jk}. +\end{align} +!et + +Here we allow for interactions beyond the nearest neighbors and a more +adaptive coupling matrix. This latter expression can be formulated as +a matrix-product on the form +!bt +\begin{align} + H = X J, +\end{align} +!et + +where $X_{jk} = s_j s_k$ and $J$ is the matrix consisting of the +elements $-J_{jk}$. This form of writing the energy fits perfectly +with the form utilized in linear regression, viz. +!bt +\begin{align} + y = X\omega + \epsilon, +\end{align} +!et + + + +!bc pycod +X = np.zeros((n, L ** 2)) +for i in range(n): + X[i] = np.outer(spins[i], spins[i]).ravel() +y = energies +X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.96) + +X_train_own = np.concatenate( + (np.ones(len(X_train))[:, np.newaxis], X_train), + axis=1 +) + +X_test_own = np.concatenate( + (np.ones(len(X_test))[:, np.newaxis], X_test), + axis=1 +) +!ec + + +===== Linear regression ===== + +The problem at hand is to try to fit the equation +!bt +\begin{align} + y = f(x) + \epsilon, +\end{align} +!et + +where $f(x)$ is some unknown function of the data $x$ and $\epsilon$ +is normally distributed with mean zero noise with standard deviation +$\sigma_{\epsilon}$. Our job is to try to find a predictor which +estimates the function $f(x)$. In linear regression we assume that we +can formulate the problem as + +!bt +\begin{align} + y = X\omega + \epsilon, +\end{align} +!et + +where $X$ and $\omega$ are now matrices. Our job at hand is now to +find a _cost function_ $C$, which we wish to minimize in order to find +the best estimate of $\omega$. + + +===== Ordinary least squares ===== + +In the ordinary least squares method we choose the cost function +!bt +\begin{align} + C(X, \omega) = ||X\omega - y||^2 + = (X\omega - y)^T(X\omega - y) +\end{align} +!et +We then find the extremal point of $C$ by taking the derivative with respect to $\omega$ and setting it to zero, i.e., + +!bt +\begin{align} + \dfrac{\mathrm{d}C}{\mathrm{d}\omega} + = 0. +\end{align} +!et +This yields the expression for $\omega$ to be +!bt +\begin{align} + \omega = \frac{X^T y}{X^T X}, +\end{align} +!et + +which immediately imposes some requirements on $X$ as there must exist +an inverse of $X^T X$. If the expression we are modelling contains an +intercept, i.e., a constant expression we must make sure that the +first column of $X$ consists of $1$. + + +!bc pycod +def get_ols_weights_naive(x: np.ndarray, y: np.ndarray) -> np.ndarray: + return scl.inv(x.T @ x) @ (x.T @ y) +omega = get_ols_weights_naive(X_train_own, y_train) +!ec + + + +===== Singular Value decomposition ===== +Doing the inversion directly turns out to be a bad idea as the matrix +$X^TX$ is singular. An alternative approach is to use the _singular +value decomposition_. Using the definition of the Moore-Penrose +pseudoinverse we can write the equation for $\omega$ as + +!bt +\begin{align} + \omega = X^{+}y, +\end{align} +!et +where the pseudoinverse of $X$ is given by +!bt +\begin{align} + X^{+} = \frac{X^T}{X^T X}. +\end{align} +!et + +Using singular value decomposition we have that $X = U\Sigma V^T$, +where $X^{+} = V\Sigma^{+} U^T$. This reduces the equation for +$\omega$ to +!bt +\begin{align} + \omega = V\Sigma^{+} U^T y. +\end{align} +!et + +Note that solving this equation by actually doing the pseudoinverse +(which is what we will do) is not a good idea as this operation scales +as $\mathcal{O}(n^3)$, where $n$ is the number of elements in a +general matrix. Instead, doing $QR$-factorization and solving the +linear system as an equation would reduce this down to +$\mathcal{O}(n^2)$ operations. + + +!bc pycod +def get_ols_weights(x: np.ndarray, y: np.ndarray) -> np.ndarray: + u, s, v = scl.svd(x) + return v.T @ scl.pinv(scl.diagsvd(s, u.shape[0], v.shape[0])) @ u.T @ y +!ec +Before passing in the data to the function we append a column with ones to the training data. + + +!bc pycod +omega = get_ols_weights(X_train_own,y_train) +!ec + + +===== Fitting with scikit-learn ===== + +Next we fit a `LinearRegression`-model from Scikit-learn for comparison. + +!bc pycod +clf = skl.LinearRegression().fit(X_train, y_train) +!ec + +Extracting the $J$-matrix from both our own method and the Scikit-learn model where we make sure to remove the intercept. + + +!bc pycod +J_own = omega[1:].reshape(L, L) +J_sk = clf.coef_.reshape(L, L) +!ec + +A way of looking at the coefficients in $J$ is to plot the matrices as images. + + +!bc pycod +fig = plt.figure(figsize=(20, 14)) +im = plt.imshow(J_own, **cmap_args) +plt.title("Home-made OLS", fontsize=18) +plt.xticks(fontsize=18) +plt.yticks(fontsize=18) +cb = fig.colorbar(im) +cb.ax.set_yticklabels(cb.ax.get_yticklabels(), fontsize=18) + +fig = plt.figure(figsize=(20, 14)) +im = plt.imshow(J_sk, **cmap_args) +plt.title("LinearRegression from Scikit-learn", fontsize=18) +plt.xticks(fontsize=18) +plt.yticks(fontsize=18) +cb = fig.colorbar(im) +cb.ax.set_yticklabels(cb.ax.get_yticklabels(), fontsize=18) +plt.show() +!ec + +We can see that our model for the least squares method performes close +to the benchmark from Scikit-learn. It is interesting to note that OLS +considers both $J_{j, j + 1} = -0.5$ and $J_{j, j - 1} = -0.5$ as +valid matrix elements for $J$. + + +===== Ridge regression ===== + +Having explored the ordinary least squares we move on to ridge +regression. In ridge regression we include a _regularizer_. This +involves a new cost function which leads to a new estimate for the +weights $\omega$. This results in a penalized regression problem. The +cost function is given by + +!bt +\begin{align} + C(X, \omega; \lambda) = ||X\omega - y||^2 + \lambda ||\omega||^2 + = (X\omega - y)^T(X\omega - y) + \lambda \omega^T\omega. +\end{align} +!et +Finding the extremum of this function yields the weights + +!bt +\begin{align} + \omega(\lambda) = \frac{X^Ty}{X^TX + \lambda} \to \frac{\omega_{\text{LS}}}{1 + \lambda}, +\end{align} +!et + +where $\omega_{\text{LS}}$ is the weights from ordinary least +squares. The last assumption assumes that $X$ is orthogonal, which it +is not. We will therefore resort to solving the equation as it stands +on the left hand side. + + +!bc pycod +def get_ridge_weights(x: np.ndarray, y: np.ndarray, _lambda: float) -> np.ndarray: + return x.T @ y @ scl.inv( + x.T @ x + np.eye(x.shape[1], x.shape[1]) * _lambda + ) +lambda = 0.1 +omega_ridge = get_ridge_weights(X_train_own, y_train, np.array([_lambda])) +clf_ridge = skl.Ridge(alpha=_lambda).fit(X_train, y_train) +J_ridge_own = omega_ridge[1:].reshape(L, L) +J_ridge_sk = clf_ridge.coef_.reshape(L, L) +fig = plt.figure(figsize=(20, 14)) +im = plt.imshow(J_ridge_own, **cmap_args) +plt.title("Home-made ridge regression", fontsize=18) +plt.xticks(fontsize=18) +plt.yticks(fontsize=18) +cb = fig.colorbar(im) +cb.ax.set_yticklabels(cb.ax.get_yticklabels(), fontsize=18) + +fig = plt.figure(figsize=(20, 14)) +im = plt.imshow(J_ridge_sk, **cmap_args) +plt.title("Ridge from Scikit-learn", fontsize=18) +plt.xticks(fontsize=18) +plt.yticks(fontsize=18) +cb = fig.colorbar(im) +cb.ax.set_yticklabels(cb.ax.get_yticklabels(), fontsize=18) + +plt.show() +!ec + + +===== LASSO regression ===== + +In the _Least Absolute Shrinkage and Selection Operator_ (LASSO)-method we get a third cost function. +!bt +\begin{align} + C(X, \omega; \lambda) = + ||X\omega - y||^2 + \lambda ||\omega|| + = (X\omega - y)^T(X\omega - y) + \lambda \sqrt{\omega^T\omega}. +\end{align} +!et +Finding the extremal point of this cost function is not so straight-forward as in least squares and ridge. We will therefore rely solely on the function ``Lasso`` from Scikit-learn. + +!bc pycod +clf_lasso = skl.Lasso(alpha=_lambda).fit(X_train, y_train) +J_lasso_sk = clf_lasso.coef_.reshape(L, L) +fig = plt.figure(figsize=(20, 14)) +im = plt.imshow(J_lasso_sk, **cmap_args) +plt.title("Lasso from Scikit-learn", fontsize=18) +plt.xticks(fontsize=18) +plt.yticks(fontsize=18) +cb = fig.colorbar(im) +cb.ax.set_yticklabels(cb.ax.get_yticklabels(), fontsize=18) + +plt.show() +!ec + +It is quite striking how LASSO breaks the symmetry of the coupling +constant as opposed to ridge and OLS. We get a sparse solution with +$J_{j, j + 1} = -1$. + + + +===== Performance of the different models ===== + +In order to judge which model performs best at varying values of $\lambda$ (for ridge and LASSO) we compute $R^2$ which is given by + +!bt +\begin{align} + R^2 = 1 - \frac{(y - \hat{y})^2}{(y - \bar{y})^2}, +\end{align} +!et +where $y$ is a vector with the true values of the energy, $\hat{y}$ is the predicted values of $y$ from the models and $\bar{y}$ is the mean of $\hat{y}$. + + +!bc pycod +def r_squared(y, y_hat): + return 1 - np.sum((y - y_hat) ** 2) / np.sum((y - np.mean(y_hat)) ** 2) +!ec + +This is the same metric used by Scikit-learn for their regression models when scoring. +!bc pycod +y_hat = clf.predict(X_test) +r_test = r_squared(y_test, y_hat) +sk_r_test = clf.score(X_test, y_test) + +assert abs(r_test - sk_r_test) < 1e-2 +!ec + + + +===== Performance as function of the regularization parameter ===== + +We see how the different models perform for a different set of values for $\lambda$. + + +!bc pycod +lambdas = np.logspace(-4, 5, 10) + +train_errors = { + "ols_own": np.zeros(lambdas.size), + "ols_sk": np.zeros(lambdas.size), + "ridge_own": np.zeros(lambdas.size), + "ridge_sk": np.zeros(lambdas.size), + "lasso_sk": np.zeros(lambdas.size) +} + +test_errors = { + "ols_own": np.zeros(lambdas.size), + "ols_sk": np.zeros(lambdas.size), + "ridge_own": np.zeros(lambdas.size), + "ridge_sk": np.zeros(lambdas.size), + "lasso_sk": np.zeros(lambdas.size) +} + +plot_counter = 1 + +fig = plt.figure(figsize=(32, 54)) + +for i, _lambda in enumerate(tqdm.tqdm(lambdas)): + omega = get_ols_weights(X_train_own, y_train) + y_hat_train = X_train_own @ omega + y_hat_test = X_test_own @ omega + + train_errors["ols_own"][i] = r_squared(y_train, y_hat_train) + test_errors["ols_own"][i] = r_squared(y_test, y_hat_test) + + plt.subplot(10, 5, plot_counter) + plt.imshow(omega[1:].reshape(L, L), **cmap_args) + plt.title("Home made OLS") + plot_counter += 1 + + omega = get_ridge_weights(X_train_own, y_train, _lambda) + y_hat_train = X_train_own @ omega + y_hat_test = X_test_own @ omega + + train_errors["ridge_own"][i] = r_squared(y_train, y_hat_train) + test_errors["ridge_own"][i] = r_squared(y_test, y_hat_test) + + plt.subplot(10, 5, plot_counter) + plt.imshow(omega[1:].reshape(L, L), **cmap_args) + plt.title(r"Home made ridge, $\lambda = %.4f$" % _lambda) + plot_counter += 1 + + for key, method in zip( + ["ols_sk", "ridge_sk", "lasso_sk"], + [skl.LinearRegression(), skl.Ridge(alpha=_lambda), skl.Lasso(alpha=_lambda)] + ): + method = method.fit(X_train, y_train) + + train_errors[key][i] = method.score(X_train, y_train) + test_errors[key][i] = method.score(X_test, y_test) + + omega = method.coef_.reshape(L, L) + + plt.subplot(10, 5, plot_counter) + plt.imshow(omega, **cmap_args) + plt.title(r"%s, $\lambda = %.4f$" % (key, _lambda)) + plot_counter += 1 + +plt.show() +!ec + +We can see that LASSO quite fast reaches a good solution for low +values of $\lambda$, but will "wither" when we increase $\lambda$ too +much. Ridge is more stable over a larger range of values for +$\lambda$, but eventually also fades away. + + +===== Finding the optimal value of $\lambda$ ===== + +To determine which value of $\lambda$ is best we plot the accuracy of +the models when predicting the training and the testing set. We expect +the accuracy of the training set to be quite good, but if the accuracy +of the testing set is much lower this tells us that we might be +subject to an overfit model. The ideal scenario is an accuracy on the +testing set that is close to the accuracy of the training set. + + +!bc pycod +fig = plt.figure(figsize=(20, 14)) + +colors = { + "ols_own": "b", + "ridge_own": "g", + "ols_sk": "r", + "ridge_sk": "y", + "lasso_sk": "c" +} + +for key in train_errors: + plt.semilogx( + lambdas, + train_errors[key], + colors[key], + label="Train {0}".format(key), + linewidth=4.0 + ) + +for key in test_errors: + plt.semilogx( + lambdas, + test_errors[key], + colors[key] + "--", + label="Test {0}".format(key), + linewidth=4.0 + ) +#plt.semilogx(lambdas, train_errors["ols_own"], label="Train (OLS own)") +#plt.semilogx(lambdas, test_errors["ols_own"], label="Test (OLS own)") + +plt.legend(loc="best", fontsize=18) +plt.xlabel(r"$\lambda$", fontsize=18) +plt.ylabel(r"$R^2$", fontsize=18) +plt.tick_params(labelsize=18) +plt.show() +!ec + +From the above figure we can see that LASSO with $\lambda = 10^{-2}$ +achieve a very good accuracy on the test set. This by far surpases the +other models for all values of $\lambda$. + + + +======= Optimization and Gradient Methods ======= + + + +===== Optimization, the central part of any Machine Learning algortithm ===== + +Almost every problem in machine learning and data science starts with +a dataset $X$, a model $g(\beta)$, which is a function of the +parameters $\beta$ and a cost function $C(X, g(\beta))$ that allows +us to judge how well the model $g(\beta)$ explains the observations +$X$. The model is fit by finding the values of $\beta$ that minimize +the cost function. Ideally we would be able to solve for $\beta$ +analytically, however this is not possible in general and we must use +some approximative/numerical method to compute the minimum. + + + +===== Revisiting our Logistic Regression case ===== + +In our discussion on Logistic Regression we studied the +case of +two classes, with $y_i$ either +$0$ or $1$. Furthermore we assumed also that we have only two +parameters $\beta$ in our fitting, that is we +defined probabilities + +!bt +\begin{align*} +p(y_i=1|x_i,\hat{\beta}) &= \frac{\exp{(\beta_0+\beta_1x_i)}}{1+\exp{(\beta_0+\beta_1x_i)}},\nonumber\\ +p(y_i=0|x_i,\hat{\beta}) &= 1 - p(y_i=1|x_i,\hat{\beta}), +\end{align*} +!et +where $\hat{\beta}$ are the weights we wish to extract from data, in our case $\beta_0$ and $\beta_1$. + + +===== The equations to solve ===== + +Our compact equations used a definition of a vector $\hat{y}$ with $n$ +elements $y_i$, an $n\times p$ matrix $\hat{X}$ which contains the +$x_i$ values and a vector $\hat{p}$ of fitted probabilities +$p(y_i\vert x_i,\hat{\beta})$. We rewrote in a more compact form +the first derivative of the cost function as + +!bt +\[ +\frac{\partial \mathcal{C}(\hat{\beta})}{\partial \hat{\beta}} = -\hat{X}^T\left(\hat{y}-\hat{p}\right). +\] +!et + +If we in addition define a diagonal matrix $\hat{W}$ with elements +$p(y_i\vert x_i,\hat{\beta})(1-p(y_i\vert x_i,\hat{\beta})$, we can obtain a compact expression of the second derivative as + +!bt +\[ +\frac{\partial^2 \mathcal{C}(\hat{\beta})}{\partial \hat{\beta}\partial \hat{\beta}^T} = \hat{X}^T\hat{W}\hat{X}. +\] +!et +This defines what is called the Hessian matrix. + + +===== Solving using Newton-Raphson's method ===== + +If we can set up these equations, Newton-Raphson's iterative method is normally the method of choice. It requires however that we can compute in an efficient way the matrices that define the first and second derivatives. + +Our iterative scheme is then given by + +!bt +\[ +\hat{\beta}^{\mathrm{new}} = \hat{\beta}^{\mathrm{old}}-\left(\frac{\partial^2 \mathcal{C}(\hat{\beta})}{\partial \hat{\beta}\partial \hat{\beta}^T}\right)^{-1}_{\hat{\beta}^{\mathrm{old}}}\times \left(\frac{\partial \mathcal{C}(\hat{\beta})}{\partial \hat{\beta}}\right)_{\hat{\beta}^{\mathrm{old}}}, +\] +!et +or in matrix form as + +!bt +\[ +\hat{\beta}^{\mathrm{new}} = \hat{\beta}^{\mathrm{old}}-\left(\hat{X}^T\hat{W}\hat{X} \right)^{-1}\times \left(-\hat{X}^T(\hat{y}-\hat{p}) \right)_{\hat{\beta}^{\mathrm{old}}}. +\] +!et +The right-hand side is computed with the old values of $\beta$. + +If we can compute these matrices, in particular the Hessian, the above is often the easiest method to implement. + + + +===== Brief reminder on Newton-Raphson's method ===== + +Let us quickly remind ourselves how we derive the above method. + +Perhaps the most celebrated of all one-dimensional root-finding +routines is Newton's method, also called the Newton-Raphson +method. This method requires the evaluation of both the +function $f$ and its derivative $f'$ at arbitrary points. +If you can only calculate the derivative +numerically and/or your function is not of the smooth type, we +normally discourage the use of this method. + + +===== The equations ===== + +The Newton-Raphson formula consists geometrically of extending the +tangent line at a current point until it crosses zero, then setting +the next guess to the abscissa of that zero-crossing. The mathematics +behind this method is rather simple. Employing a Taylor expansion for +$x$ sufficiently close to the solution $s$, we have + + +!bt +\[ + f(s)=0=f(x)+(s-x)f'(x)+\frac{(s-x)^2}{2}f''(x) +\dots. + \label{eq:taylornr} +\] +!et + +For small enough values of the function and for well-behaved +functions, the terms beyond linear are unimportant, hence we obtain + + +!bt +\[ + f(x)+(s-x)f'(x)\approx 0, +\] +!et +yielding +!bt +\[ + s\approx x-\frac{f(x)}{f'(x)}. +\] +!et + +Having in mind an iterative procedure, it is natural to start iterating with +!bt +\[ + x_{n+1}=x_n-\frac{f(x_n)}{f'(x_n)}. +\] +!et + + +===== Simple geometric interpretation ===== + +The above is Newton-Raphson's method. It has a simple geometric +interpretation, namely $x_{n+1}$ is the point where the tangent from +$(x_n,f(x_n))$ crosses the $x$-axis. Close to the solution, +Newton-Raphson converges fast to the desired result. However, if we +are far from a root, where the higher-order terms in the series are +important, the Newton-Raphson formula can give grossly inaccurate +results. For instance, the initial guess for the root might be so far +from the true root as to let the search interval include a local +maximum or minimum of the function. If an iteration places a trial +guess near such a local extremum, so that the first derivative nearly +vanishes, then Newton-Raphson may fail totally + + + +===== Extending to more than one variable ===== + +Newton's method can be generalized to systems of several non-linear equations +and variables. Consider the case with two equations +!bt +\[ + \begin{array}{cc} f_1(x_1,x_2) &=0\\ + f_2(x_1,x_2) &=0,\end{array} +\] +!et +which we Taylor expand to obtain + +!bt +\[ + \begin{array}{cc} 0=f_1(x_1+h_1,x_2+h_2)=&f_1(x_1,x_2)+h_1 + \partial f_1/\partial x_1+h_2 + \partial f_1/\partial x_2+\dots\\ + 0=f_2(x_1+h_1,x_2+h_2)=&f_2(x_1,x_2)+h_1 + \partial f_2/\partial x_1+h_2 + \partial f_2/\partial x_2+\dots + \end{array}. +\] +!et +Defining the Jacobian matrix $\hat{J}$ we have +!bt +\[ + \hat{J}=\left( \begin{array}{cc} + \partial f_1/\partial x_1 & \partial f_1/\partial x_2 \\ + \partial f_2/\partial x_1 &\partial f_2/\partial x_2 + \end{array} \right), +\] +!et +we can rephrase Newton's method as +!bt +\[ +\left(\begin{array}{c} x_1^{n+1} \\ x_2^{n+1} \end{array} \right)= +\left(\begin{array}{c} x_1^{n} \\ x_2^{n} \end{array} \right)+ +\left(\begin{array}{c} h_1^{n} \\ h_2^{n} \end{array} \right), +\] +!et +where we have defined +!bt +\[ + \left(\begin{array}{c} h_1^{n} \\ h_2^{n} \end{array} \right)= + -\hat{J}^{-1} + \left(\begin{array}{c} f_1(x_1^{n},x_2^{n}) \\ f_2(x_1^{n},x_2^{n}) \end{array} \right). +\] +!et +We need thus to compute the inverse of the Jacobian matrix and it +is to understand that difficulties may +arise in case $\hat{J}$ is nearly singular. + +It is rather straightforward to extend the above scheme to systems of +more than two non-linear equations. In our case, the Jacobian matrix is given by the Hessian that represents the second derivative of cost function. + + + + +===== Steepest descent ===== + +The basic idea of gradient descent is +that a function $F(\mathbf{x})$, +$\mathbf{x} \equiv (x_1,\cdots,x_n)$, decreases fastest if one goes from $\bf {x}$ in the +direction of the negative gradient $-\nabla F(\mathbf{x})$. + +It can be shown that if +!bt +\[ +\mathbf{x}_{k+1} = \mathbf{x}_k - \gamma_k \nabla F(\mathbf{x}_k), +\] +!et +with $\gamma_k > 0$. + +For $\gamma_k$ small enough, then $F(\mathbf{x}_{k+1}) \leq +F(\mathbf{x}_k)$. This means that for a sufficiently small $\gamma_k$ +we are always moving towards smaller function values, i.e a minimum. + + +===== More on Steepest descent ===== + +The previous observation is the basis of the method of steepest +descent, which is also referred to as just gradient descent (GD). One +starts with an initial guess $\mathbf{x}_0$ for a minimum of $F$ and +computes new approximations according to + +!bt +\[ +\mathbf{x}_{k+1} = \mathbf{x}_k - \gamma_k \nabla F(\mathbf{x}_k), \ \ k \geq 0. +\] +!et + +The parameter $\gamma_k$ is often referred to as the step length or +the learning rate within the context of Machine Learning. + + +===== The ideal ===== + +Ideally the sequence $\{\mathbf{x}_k \}_{k=0}$ converges to a global +minimum of the function $F$. In general we do not know if we are in a +global or local minimum. In the special case when $F$ is a convex +function, all local minima are also global minima, so in this case +gradient descent can converge to the global solution. The advantage of +this scheme is that it is conceptually simple and straightforward to +implement. However the method in this form has some severe +limitations: + +In machine learing we are often faced with non-convex high dimensional +cost functions with many local minima. Since GD is deterministic we +will get stuck in a local minimum, if the method converges, unless we +have a very good intial guess. This also implies that the scheme is +sensitive to the chosen initial condition. + +Note that the gradient is a function of $\mathbf{x} = +(x_1,\cdots,x_n)$ which makes it expensive to compute numerically. + + + +===== The sensitiveness of the gradient descent ===== + +The gradient descent method +is sensitive to the choice of learning rate $\gamma_k$. This is due +to the fact that we are only guaranteed that $F(\mathbf{x}_{k+1}) \leq +F(\mathbf{x}_k)$ for sufficiently small $\gamma_k$. The problem is to +determine an optimal learning rate. If the learning rate is chosen too +small the method will take a long time to converge and if it is too +large we can experience erratic behavior. + +Many of these shortcomings can be alleviated by introducing +randomness. One such method is that of Stochastic Gradient Descent +(SGD), see below. + + + +===== Convex functions ===== + +Ideally we want our cost/loss function to be convex(concave). + +First we give the definition of a convex set: A set $C$ in +$\mathbb{R}^n$ is said to be convex if, for all $x$ and $y$ in $C$ and +all $t \in (0,1)$ , the point $(1 − t)x + ty$ also belongs to +C. Geometrically this means that every point on the line segment +connecting $x$ and $y$ is in $C$ as discussed below. + +The convex subsets of $\mathbb{R}$ are the intervals of +$\mathbb{R}$. Examples of convex sets of $\mathbb{R}^2$ are the +regular polygons (triangles, rectangles, pentagons, etc...). + + +===== Convex function ===== + +_Convex function_: Let $X \subset \mathbb{R}^n$ be a convex set. Assume that the function $f: X \rightarrow \mathbb{R}$ is continuous, then $f$ is said to be convex if $$f(tx_1 + (1-t)x_2) \leq tf(x_1) + (1-t)f(x_2) $$ for all $x_1, x_2 \in X$ and for all $t \in [0,1]$. If $\leq$ is replaced with a strict inequaltiy in the definition, we demand $x_1 \neq x_2$ and $t\in(0,1)$ then $f$ is said to be strictly convex. For a single variable function, convexity means that if you draw a straight line connecting $f(x_1)$ and $f(x_2)$, the value of the function on the interval $[x_1,x_2]$ is always below the line as illustrated below. + + +===== Conditions on convex functions ===== + +In the following we state first and second-order conditions which +ensures convexity of a function $f$. We write $D_f$ to denote the +domain of $f$, i.e the subset of $R^n$ where $f$ is defined. For more +details and proofs we refer to: "S. Boyd and L. Vandenberghe. Convex Optimization. Cambridge University Press":"http://stanford.edu/boyd/cvxbook/, 2004". + +!bblock First order condition +Suppose $f$ is differentiable (i.e $\nabla f(x)$ is well defined for +all $x$ in the domain of $f$). Then $f$ is convex if and only if $D_f$ +is a convex set and $$f(y) \geq f(x) + \nabla f(x)^T (y-x) $$ holds +for all $x,y \in D_f$. This condition means that for a convex function +the first order Taylor expansion (right hand side above) at any point +a global under estimator of the function. To convince yourself you can +make a drawing of $f(x) = x^2+1$ and draw the tangent line to $f(x)$ and +note that it is always below the graph. +!eblock + +!bblock Second order condition +Assume that $f$ is twice +differentiable, i.e the Hessian matrix exists at each point in +$D_f$. Then $f$ is convex if and only if $D_f$ is a convex set and its +Hessian is positive semi-definite for all $x\in D_f$. For a +single-variable function this reduces to $f''(x) \geq 0$. Geometrically this means that $f$ has nonnegative curvature +everywhere. +!eblock + +This condition is particularly useful since it gives us an procedure for determining if the function under consideration is convex, apart from using the definition. + + +===== More on convex functions ===== + +The next result is of great importance to us and the reason why we are +going on about convex functions. In machine learning we frequently +have to minimize a loss/cost function in order to find the best +parameters for the model we are considering. + +Ideally we want the +global minimum (for high-dimensional models it is hard to know +if we have local or global minimum). However, if the cost/loss function +is convex the following result provides invaluable information: + +!bblock Any minimum is global for convex functions +Consider the problem of finding $x \in \mathbb{R}^n$ such that $f(x)$ +is minimal, where $f$ is convex and differentiable. Then, any point +$x^*$ that satisfies $\nabla f(x^*) = 0$ is a global minimum. +!eblock + +This result means that if we know that the cost/loss function is convex and we are able to find a minimum, we are guaranteed that it is a global minimum. + + +===== Some simple problems ===== + +o Show that $f(x)=x^2$ is convex for $x \in \mathbb{R}$ using the definition of convexity. Hint: If you re-write the definition, $f$ is convex if the following holds for all $x,y \in D_f$ and any $\lambda \in [0,1]$ $\lambda f(x)+(1-\lambda)f(y)-f(\lambda x + (1-\lambda) y ) \geq 0$. + +o Using the second order condition show that the following functions are convex on the specified domain. + * $f(x) = e^x$ is convex for $x \in \mathbb{R}$. + * $g(x) = -\ln(x)$ is convex for $x \in (0,\infty)$. +o Let $f(x) = x^2$ and $g(x) = e^x$. Show that $f(g(x))$ and $g(f(x))$ is convex for $x \in \mathbb{R}$. Also show that if $f(x)$ is any convex function than $h(x) = e^{f(x)}$ is convex. + +o A norm is any function that satisfy the following properties + * $f(\alpha x) = |\alpha| f(x)$ for all $\alpha \in \mathbb{R}$. + * $f(x+y) \leq f(x) + f(y)$ + * $f(x) \leq 0$ for all $x \in \mathbb{R}^n$ with equality if and only if $x = 0$ + +Using the definition of convexity, try to show that a function satisfying the properties above is convex (the third condition is not needed to show this). + + + +===== Standard steepest descent ===== + + +Before we proceed, we would like to discuss the approach called the +_standard Steepest descent_, which again leads to us having to be able +to compute a matrix. It belongs to the class of Conjugate Gradient methods (CG). + +"The success of the CG method":"https://www.cs.cmu.edu/~quake-papers/painless-conjugate-gradient.pdf" +for finding solutions of non-linear problems is based on the theory +of conjugate gradients for linear systems of equations. It belongs to +the class of iterative methods for solving problems from linear +algebra of the type +!bt +\begin{equation*} +\hat{A}\hat{x} = \hat{b}. +\end{equation*} +!et + +In the iterative process we end up with a problem like + +!bt +\begin{equation*} + \hat{r}= \hat{b}-\hat{A}\hat{x}, +\end{equation*} +!et +where $\hat{r}$ is the so-called residual or error in the iterative process. + +When we have found the exact solution, $\hat{r}=0$. + + +===== Gradient method ===== + +The residual is zero when we reach the minimum of the quadratic equation +!bt +\begin{equation*} + P(\hat{x})=\frac{1}{2}\hat{x}^T\hat{A}\hat{x} - \hat{x}^T\hat{b}, +\end{equation*} +!et + +with the constraint that the matrix $\hat{A}$ is positive definite and +symmetric. This defines also the Hessian and we want it to be positive definite. + + + +===== Steepest descent method ===== + +We denote the initial guess for $\hat{x}$ as $\hat{x}_0$. +We can assume without loss of generality that +!bt +\begin{equation*} +\hat{x}_0=0, +\end{equation*} +!et +or consider the system +!bt +\begin{equation*} +\hat{A}\hat{z} = \hat{b}-\hat{A}\hat{x}_0, +\end{equation*} +!et +instead. + + + +===== Steepest descent method ===== +!bblock +One can show that the solution $\hat{x}$ is also the unique minimizer of the quadratic form +!bt +\begin{equation*} + f(\hat{x}) = \frac{1}{2}\hat{x}^T\hat{A}\hat{x} - \hat{x}^T \hat{x} , \quad \hat{x}\in\mathbf{R}^n. +\end{equation*} +!et +This suggests taking the first basis vector $\hat{r}_1$ (see below for definition) +to be the gradient of $f$ at $\hat{x}=\hat{x}_0$, +which equals +!bt +\begin{equation*} +\hat{A}\hat{x}_0-\hat{b}, +\end{equation*} +!et +and +$\hat{x}_0=0$ it is equal $-\hat{b}$. + +!eblock + + +===== Final expressions ===== +!bblock +We can compute the residual iteratively as +!bt +\begin{equation*} +\hat{r}_{k+1}=\hat{b}-\hat{A}\hat{x}_{k+1}, + \end{equation*} +!et +which equals +!bt +\begin{equation*} +\hat{b}-\hat{A}(\hat{x}_k+\alpha_k\hat{r}_k), + \end{equation*} +!et +or +!bt +\begin{equation*} +(\hat{b}-\hat{A}\hat{x}_k)-\alpha_k\hat{A}\hat{r}_k, + \end{equation*} +!et +which gives + +!bt +\[ +\alpha_k = \frac{\hat{r}_k^T\hat{r}_k}{\hat{r}_k^T\hat{A}\hat{r}_k} +\] +!et +leading to the iterative scheme +!bt +\begin{equation*} +\hat{x}_{k+1}=\hat{x}_k-\alpha_k\hat{r}_{k}, + \end{equation*} +!et +!eblock + + + +===== Code examples for steepest descent ===== + + +===== Simple codes for steepest descent and conjugate gradient using a $2\times 2$ matrix, in c++, Python code to come ===== +!bblock +!bc cppcod +#include +#include +#include +#include +#include "vectormatrixclass.h" +using namespace std; +// Main function begins here +int main(int argc, char * argv[]){ + int dim = 2; + Vector x(dim),xsd(dim), b(dim),x0(dim); + Matrix A(dim,dim); + + // Set our initial guess + x0(0) = x0(1) = 0; + // Set the matrix + A(0,0) = 3; A(1,0) = 2; A(0,1) = 2; A(1,1) = 6; + b(0) = 2; b(1) = -8; + cout << "The Matrix A that we are using: " << endl; + A.Print(); + cout << endl; + xsd = SteepestDescent(A,b,x0); + cout << "The approximate solution using Steepest Descent is: " << endl; + xsd.Print(); + cout << endl; +} +!ec +!eblock + + +===== The routine for the steepest descent method ===== +!bblock +!bc cppcod +Vector SteepestDescent(Matrix A, Vector b, Vector x0){ + int IterMax, i; + int dim = x0.Dimension(); + const double tolerance = 1.0e-14; + Vector x(dim),f(dim),z(dim); + double c,alpha,d; + IterMax = 30; + x = x0; + r = A*x-b; + i = 0; + while (i <= IterMax){ + z = A*r; + c = dot(r,r); + alpha = c/dot(r,z); + x = x - alpha*r; + r = A*x-b; + if(sqrt(dot(r,r)) < tolerance) break; + i++; + } + return x; +} +!ec +!eblock + + + +===== Steepest descent example ===== + +!bc pycod +import numpy as np +import numpy.linalg as la + +import scipy.optimize as sopt + +import matplotlib.pyplot as pt +from mpl_toolkits.mplot3d import axes3d + +def f(x): + return 0.5*x[0]**2 + 2.5*x[1]**2 + +def df(x): + return np.array([x[0], 5*x[1]]) + +fig = pt.figure() +ax = fig.gca(projection="3d") + +xmesh, ymesh = np.mgrid[-2:2:50j,-2:2:50j] +fmesh = f(np.array([xmesh, ymesh])) +ax.plot_surface(xmesh, ymesh, fmesh) +!ec +And then as countor plot +!bc pycod +pt.axis("equal") +pt.contour(xmesh, ymesh, fmesh) +guesses = [np.array([2, 2./5])] +!ec +Find guesses +!bc pycod +x = guesses[-1] +s = -df(x) +!ec +Run it! +!bc pycod +def f1d(alpha): + return f(x + alpha*s) + +alpha_opt = sopt.golden(f1d) +next_guess = x + alpha_opt * s +guesses.append(next_guess) +print(next_guess) +!ec +What happened? +!bc pycod +pt.axis("equal") +pt.contour(xmesh, ymesh, fmesh, 50) +it_array = np.array(guesses) +pt.plot(it_array.T[0], it_array.T[1], "x-") +!ec + + +===== Conjugate gradient method ===== +!bblock +In the CG method we define so-called conjugate directions and two vectors +$\hat{s}$ and $\hat{t}$ +are said to be +conjugate if +!bt +\begin{equation*} +\hat{s}^T\hat{A}\hat{t}= 0. +\end{equation*} +!et +The philosophy of the CG method is to perform searches in various conjugate directions +of our vectors $\hat{x}_i$ obeying the above criterion, namely +!bt +\begin{equation*} +\hat{x}_i^T\hat{A}\hat{x}_j= 0. +\end{equation*} +!et +Two vectors are conjugate if they are orthogonal with respect to +this inner product. Being conjugate is a symmetric relation: if $\hat{s}$ is conjugate to $\hat{t}$, then $\hat{t}$ is conjugate to $\hat{s}$. +!eblock + + +===== Conjugate gradient method ===== +!bblock +An example is given by the eigenvectors of the matrix +!bt +\begin{equation*} +\hat{v}_i^T\hat{A}\hat{v}_j= \lambda\hat{v}_i^T\hat{v}_j, +\end{equation*} +!et +which is zero unless $i=j$. +!eblock + + + +===== Conjugate gradient method ===== +!bblock +Assume now that we have a symmetric positive-definite matrix $\hat{A}$ of size +$n\times n$. At each iteration $i+1$ we obtain the conjugate direction of a vector +!bt +\begin{equation*} +\hat{x}_{i+1}=\hat{x}_{i}+\alpha_i\hat{p}_{i}. +\end{equation*} +!et +We assume that $\hat{p}_{i}$ is a sequence of $n$ mutually conjugate directions. +Then the $\hat{p}_{i}$ form a basis of $R^n$ and we can expand the solution +$ \hat{A}\hat{x} = \hat{b}$ in this basis, namely + +!bt +\begin{equation*} + \hat{x} = \sum^{n}_{i=1} \alpha_i \hat{p}_i. +\end{equation*} +!et +!eblock + + +===== Conjugate gradient method ===== +!bblock +The coefficients are given by +!bt +\begin{equation*} + \mathbf{A}\mathbf{x} = \sum^{n}_{i=1} \alpha_i \mathbf{A} \mathbf{p}_i = \mathbf{b}. +\end{equation*} +!et +Multiplying with $\hat{p}_k^T$ from the left gives + +!bt +\begin{equation*} + \hat{p}_k^T \hat{A}\hat{x} = \sum^{n}_{i=1} \alpha_i\hat{p}_k^T \hat{A}\hat{p}_i= \hat{p}_k^T \hat{b}, +\end{equation*} +!et +and we can define the coefficients $\alpha_k$ as + +!bt +\begin{equation*} + \alpha_k = \frac{\hat{p}_k^T \hat{b}}{\hat{p}_k^T \hat{A} \hat{p}_k} +\end{equation*} +!et +!eblock + + +===== Conjugate gradient method and iterations ===== +!bblock + +If we choose the conjugate vectors $\hat{p}_k$ carefully, +then we may not need all of them to obtain a good approximation to the solution +$\hat{x}$. +We want to regard the conjugate gradient method as an iterative method. +This will us to solve systems where $n$ is so large that the direct +method would take too much time. + +We denote the initial guess for $\hat{x}$ as $\hat{x}_0$. +We can assume without loss of generality that +!bt +\begin{equation*} +\hat{x}_0=0, +\end{equation*} +!et +or consider the system +!bt +\begin{equation*} +\hat{A}\hat{z} = \hat{b}-\hat{A}\hat{x}_0, +\end{equation*} +!et +instead. +!eblock + + + +===== Conjugate gradient method ===== +!bblock +One can show that the solution $\hat{x}$ is also the unique minimizer of the quadratic form +!bt +\begin{equation*} + f(\hat{x}) = \frac{1}{2}\hat{x}^T\hat{A}\hat{x} - \hat{x}^T \hat{x} , \quad \hat{x}\in\mathbf{R}^n. +\end{equation*} +!et +This suggests taking the first basis vector $\hat{p}_1$ +to be the gradient of $f$ at $\hat{x}=\hat{x}_0$, +which equals +!bt +\begin{equation*} +\hat{A}\hat{x}_0-\hat{b}, +\end{equation*} +!et +and +$\hat{x}_0=0$ it is equal $-\hat{b}$. +The other vectors in the basis will be conjugate to the gradient, +hence the name conjugate gradient method. +!eblock + + + +===== Conjugate gradient method ===== +!bblock +Let $\hat{r}_k$ be the residual at the $k$-th step: +!bt +\begin{equation*} +\hat{r}_k=\hat{b}-\hat{A}\hat{x}_k. +\end{equation*} +!et +Note that $\hat{r}_k$ is the negative gradient of $f$ at +$\hat{x}=\hat{x}_k$, +so the gradient descent method would be to move in the direction $\hat{r}_k$. +Here, we insist that the directions $\hat{p}_k$ are conjugate to each other, +so we take the direction closest to the gradient $\hat{r}_k$ +under the conjugacy constraint. +This gives the following expression +!bt +\begin{equation*} +\hat{p}_{k+1}=\hat{r}_k-\frac{\hat{p}_k^T \hat{A}\hat{r}_k}{\hat{p}_k^T\hat{A}\hat{p}_k} \hat{p}_k. +\end{equation*} +!et +!eblock + + +===== Conjugate gradient method ===== +!bblock +We can also compute the residual iteratively as +!bt +\begin{equation*} +\hat{r}_{k+1}=\hat{b}-\hat{A}\hat{x}_{k+1}, + \end{equation*} +!et +which equals +!bt +\begin{equation*} +\hat{b}-\hat{A}(\hat{x}_k+\alpha_k\hat{p}_k), + \end{equation*} +!et +or +!bt +\begin{equation*} +(\hat{b}-\hat{A}\hat{x}_k)-\alpha_k\hat{A}\hat{p}_k, + \end{equation*} +!et +which gives + +!bt +\begin{equation*} +\hat{r}_{k+1}=\hat{r}_k-\hat{A}\hat{p}_{k}, + \end{equation*} +!et +!eblock + + + + +===== Simple implementation of the Conjugate gradient algorithm ===== +!bblock +!bc cppcod + Vector ConjugateGradient(Matrix A, Vector b, Vector x0){ + int dim = x0.Dimension(); + const double tolerance = 1.0e-14; + Vector x(dim),r(dim),v(dim),z(dim); + double c,t,d; + + x = x0; + r = b - A*x; + v = r; + c = dot(r,r); + int i = 0; IterMax = dim; + while(i <= IterMax){ + z = A*v; + t = c/dot(v,z); + x = x + t*v; + r = r - t*z; + d = dot(r,r); + if(sqrt(d) < tolerance) + break; + v = r + (d/c)*v; + c = d; i++; + } + return x; +} +!ec +!eblock + + + +===== Broyden–Fletcher–Goldfarb–Shanno algorithm ===== +!bblock +The optimization problem is to minimize $f(\mathbf {x} )$ where $\mathbf {x}$ is a vector in $R^{n}$, and $f$ is a differentiable scalar function. There are no constraints on the values that $\mathbf {x}$ can take. + +The algorithm begins at an initial estimate for the optimal value $\mathbf {x}_{0}$ and proceeds iteratively to get a better estimate at each stage. + +The search direction $p_k$ at stage $k$ is given by the solution of the analogue of the Newton equation +!bt +\[ +B_{k}\mathbf {p} _{k}=-\nabla f(\mathbf {x}_{k}), +\] +!et + +where $B_{k}$ is an approximation to the Hessian matrix, which is +updated iteratively at each stage, and $\nabla f(\mathbf {x} _{k})$ +is the gradient of the function +evaluated at $x_k$. +A line search in the direction $p_k$ is then used to +find the next point $x_{k+1}$ by minimising +!bt +\[ +f(\mathbf {x}_{k}+\alpha \mathbf {p}_{k}), +\] +!et +over the scalar $\alpha > 0$. + +!eblock + + + + + + +===== Revisiting our first homework ===== + +We will use linear regression as a case study for the gradient descent +methods. Linear regression is a great test case for the gradient +descent methods discussed in the lectures since it has several +desirable properties such as: + +o An analytical solution (recall homework set 1). +o The gradient can be computed analytically. +o The cost function is convex which guarantees that gradient descent converges for small enough learning rates + +We revisit the example from homework set 1 where we had +!bt +\[ +y_i = 5x_i^2 + 0.1\xi_i, \ i=1,\cdots,100 +\] +!et +with $x_i \in [0,1] $ chosen randomly with a uniform distribution. Additionally $\xi_i$ represents stochastic noise chosen according to a normal distribution $\cal {N}(0,1)$. +The linear regression model is given by +!bt +\[ +h_\beta(x) = \hat{y} = \beta_0 + \beta_1 x, +\] +!et +such that +!bt +\[ +\hat{y}_i = \beta_0 + \beta_1 x_i. +\] +!et + + +===== Gradient descent example ===== + +Let $\mathbf{y} = (y_1,\cdots,y_n)^T$, $\mathbf{\hat{y}} = (\hat{y}_1,\cdots,\hat{y}_n)^T$ and $\beta = (\beta_0, \beta_1)^T$ + +It is convenient to write $\mathbf{\hat{y}} = X\beta$ where $X \in \mathbb{R}^{100 \times 2} $ is the design matrix given by +!bt +\[ +X \equiv \begin{bmatrix} +1 & x_1 \\ +\vdots & \vdots \\ +1 & x_{100} & \\ +\end{bmatrix}. +\] +!et +The loss function is given by +!bt +\[ +C(\beta) = ||X\beta-\mathbf{y}||^2 = ||X\beta||^2 - 2 \mathbf{y}^T X\beta + ||\mathbf{y}||^2 = \sum_{i=1}^{100} (\beta_0 + \beta_1 x_i)^2 - 2 y_i (\beta_0 + \beta_1 x_i) + y_i^2 +\] +!et +and we want to find $\beta$ such that $C(\beta)$ is minimized. + + +===== The derivative of the cost/loss function ===== + +Computing $\partial C(\beta) / \partial \beta_0$ and $\partial C(\beta) / \partial \beta_1$ we can show that the gradient can be written as +!bt +\[ +\nabla_{\beta} C(\beta) = (\partial C(\beta) / \partial \beta_0, \partial C(\beta) / \partial \beta_1)^T = 2\begin{bmatrix} \sum_{i=1}^{100} \left(\beta_0+\beta_1x_i-y_i\right) \\ +\sum_{i=1}^{100}\left( x_i (\beta_0+\beta_1x_i)-y_ix_i\right) \\ +\end{bmatrix} = 2X^T(X\beta - \mathbf{y}), +\] +!et +where $X$ is the design matrix defined above. + + +===== The Hessian matrix ===== +The Hessian matrix of $C(\beta)$ is given by +!bt +\[ +\hat{H} \equiv \begin{bmatrix} +\frac{\partial^2 C(\beta)}{\partial \beta_0^2} & \frac{\partial^2 C(\beta)}{\partial \beta_0 \partial \beta_1} \\ +\frac{\partial^2 C(\beta)}{\partial \beta_0 \partial \beta_1} & \frac{\partial^2 C(\beta)}{\partial \beta_1^2} & \\ +\end{bmatrix} = 2X^T X. +\] +!et +This result implies that $C(\beta)$ is a convex function since the matrix $X^T X$ always is positive semi-definite. + + + + + +===== Simple program ===== + +We can now write a program that minimizes $C(\beta)$ using the gradient descent method with a constant learning rate $\gamma$ according to +!bt +\[ +\beta_{k+1} = \beta_k - \gamma \nabla_\beta C(\beta_k), \ k=0,1,\cdots +\] +!et + +We can use the expression we computed for the gradient and let use a +$\beta_0$ be chosen randomly and let $\gamma = 0.001$. Stop iterating +when $||\nabla_\beta C(\beta_k) || \leq \epsilon = 10^{-8}$. + +And finally we can compare our solution for $\beta$ with the analytic result given by +$\beta= (X^TX)^{-1} X^T \mathbf{y}$. +!bc pycod +import numpy as np + +""" +The following setup is just a suggestion, feel free to write it the way you like. +""" + +#Setup problem described in the exercise +N = 100 #Nr of datapoints +M = 2 #Nr of features +x = np.random.rand(N) #Uniformly generated x-values in [0,1] +y = 5*x**2 + 0.1*np.random.randn(N) +X = np.c_[np.ones(N),x] #Construct design matrix + +#Compute beta according to normal equations to compare with GD solution +Xt_X_inv = np.linalg.inv(np.dot(X.T,X)) +Xt_y = np.dot(X.transpose(),y) +beta_NE = np.dot(Xt_X_inv,Xt_y) +print(beta_NE) +!ec + + +===== Gradient Descent Example ===== + +Another simple example is here +!bc pycod + +# Importing various packages +from random import random, seed +import numpy as np +import matplotlib.pyplot as plt +from mpl_toolkits.mplot3d import Axes3D +from matplotlib import cm +from matplotlib.ticker import LinearLocator, FormatStrFormatter +import sys + +x = 2*np.random.rand(100,1) +y = 4+3*x+np.random.randn(100,1) + +xb = np.c_[np.ones((100,1)), x] +beta_linreg = np.linalg.inv(xb.T.dot(xb)).dot(xb.T).dot(y) +print(beta_linreg) +beta = np.random.randn(2,1) + +eta = 0.1 +Niterations = 1000 +m = 100 + +for iter in range(Niterations): + gradients = 2.0/m*xb.T.dot(xb.dot(beta)-y) + beta -= eta*gradients + +print(beta) +xnew = np.array([[0],[2]]) +xbnew = np.c_[np.ones((2,1)), xnew] +ypredict = xbnew.dot(beta) +ypredict2 = xbnew.dot(beta_linreg) +plt.plot(xnew, ypredict, "r-") +plt.plot(xnew, ypredict2, "b-") +plt.plot(x, y ,'ro') +plt.axis([0,2.0,0, 15.0]) +plt.xlabel(r'$x$') +plt.ylabel(r'$y$') +plt.title(r'Gradient descent example') +plt.show() + +!ec + + +===== And a corresponding example using _scikit-learn_ ===== + +!bc pycod +# Importing various packages +from random import random, seed +import numpy as np +import matplotlib.pyplot as plt +from sklearn.linear_model import SGDRegressor + +x = 2*np.random.rand(100,1) +y = 4+3*x+np.random.randn(100,1) + +xb = np.c_[np.ones((100,1)), x] +beta_linreg = np.linalg.inv(xb.T.dot(xb)).dot(xb.T).dot(y) +print(beta_linreg) +sgdreg = SGDRegressor(n_iter = 50, penalty=None, eta0=0.1) +sgdreg.fit(x,y.ravel()) +print(sgdreg.intercept_, sgdreg.coef_) + +!ec + + + + +===== Gradient descent and Ridge ===== + +We have also discussed Ridge regression where the loss function contains a regularized given by the $L_2$ norm of $\beta$, +!bt +\[ +C_{\text{ridge}}(\beta) = ||X\beta -\mathbf{y}||^2 + \lambda ||\beta||^2, \ \lambda \geq 0. +\] +!et + +In order to minimize $C_{\text{ridge}}(\beta)$ using GD we only have adjust the gradient as follows +!bt +\[ +\nabla_\beta C_{\text{ridge}}(\beta) = 2\begin{bmatrix} \sum_{i=1}^{100} \left(\beta_0+\beta_1x_i-y_i\right) \\ +\sum_{i=1}^{100}\left( x_i (\beta_0+\beta_1x_i)-y_ix_i\right) \\ +\end{bmatrix} + 2\lambda\begin{bmatrix} \beta_0 \\ \beta_1\end{bmatrix} = 2 (X^T(X\beta - \mathbf{y})+\lambda \beta). +\] +!et + +We can now extend our program to minimize $C_{\text{ridge}}(\beta)$ using gradient descent and compare with the analytical solution given by +!bt +\[ +\beta_{\text{ridge}} = \left(X^T X + \lambda I_{2 \times 2} \right)^{-1} X^T \mathbf{y}, +\] +!et +for $\lambda = {0,1,10,50,100}$ ($\lambda = 0$ corresponds to ordinary least squares). +We can then compute $||\beta_{\text{ridge}}||$ for each $\lambda$. + +!bc pycod +import numpy as np + +""" +The following setup is just a suggestion, feel free to write it the way you like. +""" + +#Setup problem described in the exercise +N = 100 #Nr of datapoints +M = 2 #Nr of features +x = np.random.rand(N) +y = 5*x**2 + 0.1*np.random.randn(N) + + +#Compute analytic beta for Ridge regression +X = np.c_[np.ones(N),x] +XT_X = np.dot(X.T,X) + +l = 0.1 #Ridge parameter lambda +Id = np.eye(XT_X.shape[0]) + +Z = np.linalg.inv(XT_X+l*Id) +beta_ridge = np.dot(Z,np.dot(X.T,y)) + +print(beta_ridge) +print(np.linalg.norm(beta_ridge)) #||beta|| +!ec + + +===== Automatic differentiation ===== +Python has tools for so-called _automatic differentiation_. +Consider the following example +!bt +\[ +f(x) = \sin\left(2\pi x + x^2\right) +\] +!et +which has the following derivative +!bt +\[ +f'(x) = \cos\left(2\pi x + x^2\right)\left(2\pi + 2x\right) +\] +!et +Using _autograd_ we have + +!bc pycod +import autograd.numpy as np + +# To do elementwise differentiation: +from autograd import elementwise_grad as egrad + +# To plot: +import matplotlib.pyplot as plt + + +def f(x): + return np.sin(2*np.pi*x + x**2) + +def f_grad_analytic(x): + return np.cos(2*np.pi*x + x**2)*(2*np.pi + 2*x) + +# Do the comparison: +x = np.linspace(0,1,1000) + +f_grad = egrad(f) + +computed = f_grad(x) +analytic = f_grad_analytic(x) + +plt.title('Derivative computed from Autograd compared with the analytical derivative') +plt.plot(x,computed,label='autograd') +plt.plot(x,analytic,label='analytic') + +plt.xlabel('x') +plt.ylabel('y') +plt.legend() + +plt.show() + +print("The max absolute difference is: %g"%(np.max(np.abs(computed - analytic)))) +!ec + + +===== Using autograd ===== + +Here we +experiment with what kind of functions Autograd is capable +of finding the gradient of. The following Python functions are just +meant to illustrate what Autograd can do, but please feel free to +experiment with other, possibly more complicated, functions as well. + +!bc pycod +import autograd.numpy as np +from autograd import grad + +def f1(x): + return x**3 + 1 + +f1_grad = grad(f1) + +# Remember to send in float as argument to the computed gradient from Autograd! +a = 1.0 + +# See the evaluated gradient at a using autograd: +print("The gradient of f1 evaluated at a = %g using autograd is: %g"%(a,f1_grad(a))) + +# Compare with the analytical derivative, that is f1'(x) = 3*x**2 +grad_analytical = 3*a**2 +print("The gradient of f1 evaluated at a = %g by finding the analytic expression is: %g"%(a,grad_analytical)) +!ec + + + +===== Autograd with more complicated functions ===== + +To differentiate with respect to two (or more) arguments of a Python +function, Autograd need to know at which variable the function if +being differentiated with respect to. + +!bc pycod +import autograd.numpy as np +from autograd import grad +def f2(x1,x2): + return 3*x1**3 + x2*(x1 - 5) + 1 + +# By sending the argument 0, Autograd will compute the derivative w.r.t the first variable, in this case x1 +f2_grad_x1 = grad(f2,0) + +# ... and differentiate w.r.t x2 by sending 1 as an additional arugment to grad +f2_grad_x2 = grad(f2,1) + +x1 = 1.0 +x2 = 3.0 + +print("Evaluating at x1 = %g, x2 = %g"%(x1,x2)) +print("-"*30) + +# Compare with the analytical derivatives: + +# Derivative of f2 w.r.t x1 is: 9*x1**2 + x2: +f2_grad_x1_analytical = 9*x1**2 + x2 + +# Derivative of f2 w.r.t x2 is: x1 - 5: +f2_grad_x2_analytical = x1 - 5 + +# See the evaluated derivations: +print("The derivative of f2 w.r.t x1: %g"%( f2_grad_x1(x1,x2) )) +print("The analytical derivative of f2 w.r.t x1: %g"%( f2_grad_x1(x1,x2) )) + +print() + +print("The derivative of f2 w.r.t x2: %g"%( f2_grad_x2(x1,x2) )) +print("The analytical derivative of f2 w.r.t x2: %g"%( f2_grad_x2(x1,x2) )) +!ec + +Note that the grad function will not produce the true gradient of the function. The true gradient of a function with two or more variables will produce a vector, where each element is the function differentiated w.r.t a variable. + + + +===== More complicated functions using the elements of their arguments directly ===== + +!bc pycod +import autograd.numpy as np +from autograd import grad +def f3(x): # Assumes x is an array of length 5 or higher + return 2*x[0] + 3*x[1] + 5*x[2] + 7*x[3] + 11*x[4]**2 + +f3_grad = grad(f3) + +x = np.linspace(0,4,5) + +# Print the computed gradient: +print("The computed gradient of f3 is: ", f3_grad(x)) + +# The analytical gradient is: (2, 3, 5, 7, 22*x[4]) +f3_grad_analytical = np.array([2, 3, 5, 7, 22*x[4]]) + +# Print the analytical gradient: +print("The analytical gradient of f3 is: ", f3_grad_analytical) +!ec + +Note that in this case, when sending an array as input argument, the +output from Autograd is another array. This is the true gradient of +the function, as opposed to the function in the previous example. By +using arrays to represent the variables, the output from Autograd +might be easier to work with, as the output is closer to what one +could expect form a gradient-evaluting function. + + +===== Functions using mathematical functions from Numpy ===== + +!bc pycod +import autograd.numpy as np +from autograd import grad +def f4(x): + return np.sqrt(1+x**2) + np.exp(x) + np.sin(2*np.pi*x) + +f4_grad = grad(f4) + +x = 2.7 + +# Print the computed derivative: +print("The computed derivative of f4 at x = %g is: %g"%(x,f4_grad(x))) + +# The analytical derivative is: x/sqrt(1 + x**2) + exp(x) + cos(2*pi*x)*2*pi +f4_grad_analytical = x/np.sqrt(1 + x**2) + np.exp(x) + np.cos(2*np.pi*x)*2*np.pi + +# Print the analytical gradient: +print("The analytical gradient of f4 at x = %g is: %g"%(x,f4_grad_analytical)) +!ec + + + +===== More autograd ===== + +!bc pycod +import autograd.numpy as np +from autograd import grad +def f5(x): + if x >= 0: + return x**2 + else: + return -3*x + 1 + +f5_grad = grad(f5) + +x = 2.7 + +# Print the computed derivative: +print("The computed derivative of f5 at x = %g is: %g"%(x,f5_grad(x))) +!ec + + + +===== And with loops ===== + +!bc pycod +import autograd.numpy as np +from autograd import grad +def f6_for(x): + val = 0 + for i in range(10): + val = val + x**i + return val + +def f6_while(x): + val = 0 + i = 0 + while i < 10: + val = val + x**i + i = i + 1 + return val + +f6_for_grad = grad(f6_for) +f6_while_grad = grad(f6_while) + +x = 0.5 + +# Print the computed derivaties of f6_for and f6_while +print("The computed derivative of f6_for at x = %g is: %g"%(x,f6_for_grad(x))) +print("The computed derivative of f6_while at x = %g is: %g"%(x,f6_while_grad(x))) +!ec +!bc pycod +import autograd.numpy as np +from autograd import grad +# Both of the functions are implementation of the sum: sum(x**i) for i = 0, ..., 9 +# The analytical derivative is: sum(i*x**(i-1)) +f6_grad_analytical = 0 +for i in range(10): + f6_grad_analytical += i*x**(i-1) + +print("The analytical derivative of f6 at x = %g is: %g"%(x,f6_grad_analytical)) +!ec + + +===== Using recursion ===== +!bc pycod +import autograd.numpy as np +from autograd import grad + +def f7(n): # Assume that n is an integer + if n == 1 or n == 0: + return 1 + else: + return n*f7(n-1) + +f7_grad = grad(f7) + +n = 2.0 + +print("The computed derivative of f7 at n = %d is: %g"%(n,f7_grad(n))) + +# The function f7 is an implementation of the factorial of n. +# By using the product rule, one can find that the derivative is: + +f7_grad_analytical = 0 +for i in range(int(n)-1): + tmp = 1 + for k in range(int(n)-1): + if k != i: + tmp *= (n - k) + f7_grad_analytical += tmp + +print("The analytical derivative of f7 at n = %d is: %g"%(n,f7_grad_analytical)) + +!ec +Note that if n is equal to zero or one, Autograd will give an error message. This message appears when the output is independent on input. + + +===== Unsupported functions ===== +Autograd supports many features. However, there are some functions that is not supported (yet) by Autograd. + +Assigning a value to the variable being differentiated with respect to +!bc pycod +import autograd.numpy as np +from autograd import grad +def f8(x): # Assume x is an array + x[2] = 3 + return x*2 + +f8_grad = grad(f8) + +x = 8.4 + +print("The derivative of f8 is:",f8_grad(x)) +!ec +Here, Autograd tells us that an 'ArrayBox' does not support item assignment. The item assignment is done when the program tries to assign x[2] to the value 3. However, Autograd has implemented the computation of the derivative such that this assignment is not possible. + + +===== The syntax a.dot(b) when finding the dot product ===== +!bc pycod +import autograd.numpy as np +from autograd import grad +def f9(a): # Assume a is an array with 2 elements + b = np.array([1.0,2.0]) + return a.dot(b) + +f9_grad = grad(f9) + +x = np.array([1.0,0.0]) + +print("The derivative of f9 is:",f9_grad(x)) +!ec + +Here we are told that the 'dot' function does not belong to Autograd's +version of a Numpy array. To overcome this, an alternative syntax +which also computed the dot product can be used: + +!bc pycod +import autograd.numpy as np +from autograd import grad +def f9_alternative(x): # Assume a is an array with 2 elements + b = np.array([1.0,2.0]) + return np.dot(x,b) # The same as x_1*b_1 + x_2*b_2 + +f9_alternative_grad = grad(f9_alternative) + +x = np.array([3.0,0.0]) + +print("The gradient of f9 is:",f9_alternative_grad(x)) + +# The analytical gradient of the dot product of vectors x and b with two elements (x_1,x_2) and (b_1, b_2) respectively +# w.r.t x is (b_1, b_2). +!ec + + +===== Recommended to avoid ===== +The documentation recommends to avoid inplace operations such as +!bc pycod +a += b +a -= b +a*= b +a /=b +!ec + + +===== Stochastic Gradient Descent ===== + +Stochastic gradient descent (SGD) and variants thereof address some of +the shortcomings of the Gradient descent method discussed above. + +The underlying idea of SGD comes from the observation that the cost +function, which we want to minimize, can almost always be written as a +sum over $n$ data points $\{\mathbf{x}_i\}_{i=1}^n$, +!bt +\[ +C(\mathbf{\beta}) = \sum_{i=1}^n c_i(\mathbf{x}_i, +\mathbf{\beta}). +\] +!et + + +===== Computation of gradients ===== + +This in turn means that the gradient can be +computed as a sum over $i$-gradients +!bt +\[ +\nabla_\beta C(\mathbf{\beta}) = \sum_i^n \nabla_\beta c_i(\mathbf{x}_i, +\mathbf{\beta}). +\] +!et + +Stochasticity/randomness is introduced by only taking the +gradient on a subset of the data called minibatches. If there are $n$ +data points and the size of each minibatch is $M$, there will be $n/M$ +minibatches. We denote these minibatches by $B_k$ where +$k=1,\cdots,n/M$. + + +===== SGD example ===== +As an example, suppose we have $10$ data points $(\mathbf{x}_1,\cdots, \mathbf{x}_{10})$ +and we choose to have $M=5$ minibathces, +then each minibatch contains two data points. In particular we have +$B_1 = (\mathbf{x}_1,\mathbf{x}_2), \cdots, B_5 = +(\mathbf{x}_9,\mathbf{x}_{10})$. Note that if you choose $M=1$ you +have only a single batch with all data points and on the other extreme, +you may choose $M=n$ resulting in a minibatch for each datapoint, i.e +$B_k = \mathbf{x}_k$. + +The idea is now to approximate the gradient by replacing the sum over +all data points with a sum over the data points in one the minibatches +picked at random in each gradient descent step +!bt +\[ +\nabla_{\beta} +C(\mathbf{\beta}) = \sum_{i=1}^n \nabla_\beta c_i(\mathbf{x}_i, +\mathbf{\beta}) \rightarrow \sum_{i \in B_k}^n \nabla_\beta +c_i(\mathbf{x}_i, \mathbf{\beta}). +\] +!et + + +===== The gradient step ===== + +Thus a gradient descent step now looks like +!bt +\[ +\beta_{j+1} = \beta_j - \gamma_j \sum_{i \in B_k}^n \nabla_\beta c_i(\mathbf{x}_i, +\mathbf{\beta}) +\] +!et + +where $k$ is picked at random with equal +probability from $[1,n/M]$. An iteration over the number of +minibathces (n/M) is commonly referred to as an epoch. Thus it is +typical to choose a number of epochs and for each epoch iterate over +the number of minibatches, as exemplified in the code below. + + +===== Simple example code ===== + +!bc pycod +import numpy as np + +n = 100 #100 datapoints +M = 5 #size of each minibatch +m = int(n/M) #number of minibatches +n_epochs = 10 #number of epochs + +j = 0 +for epoch in range(1,n_epochs+1): + for i in range(m): + k = np.random.randint(m) #Pick the k-th minibatch at random + #Compute the gradient using the data in minibatch Bk + #Compute new suggestion for + j += 1 +!ec + +Taking the gradient only on a subset of the data has two important +benefits. First, it introduces randomness which decreases the chance +that our opmization scheme gets stuck in a local minima. Second, if +the size of the minibatches are small relative to the number of +datapoints ($M < n$), the computation of the gradient is much +cheaper since we sum over the datapoints in the $k-th$ minibatch and not +all $n$ datapoints. + + +===== When do we stop? ===== + +A natural question is when do we stop the search for a new minimum? +One possibility is to compute the full gradient after a given number +of epochs and check if the norm of the gradient is smaller than some +threshold and stop if true. However, the condition that the gradient +is zero is valid also for local minima, so this would only tell us +that we are close to a local/global minimum. However, we could also +evaluate the cost function at this point, store the result and +continue the search. If the test kicks in at a later stage we can +compare the values of the cost function and keep the $\beta$ that +gave the lowest value. + + +===== Slightly different approach ===== + +Another approach is to let the step length $\gamma_j$ depend on the +number of epochs in such a way that it becomes very small after a +reasonable time such that we do not move at all. + +As an example, let $e = 0,1,2,3,\cdots$ denote the current epoch and let $t_0, t_1 > 0$ be two fixed numbers. Furthermore, let $t = e \cdot m + i$ where $m$ is the number of minibatches and $i=0,\cdots,m-1$. Then the function $$\gamma_j(t; t_0, t_1) = \frac{t_0}{t+t_1} $$ goes to zero as the number of epochs gets large. I.e. we start with a step length $\gamma_j (0; t_0, t_1) = t_0/t_1$ which decays in *time* $t$. + +In this way we can fix the number of epochs, compute $\beta$ and +evaluate the cost function at the end. Repeating the computation will +give a different result since the scheme is random by design. Then we +pick the final $\beta$ that gives the lowest value of the cost +function. + +!bc pycod +import numpy as np + +def step_length(t,t0,t1): + return t0/(t+t1) + +n = 100 #100 datapoints +M = 5 #size of each minibatch +m = int(n/M) #number of minibatches +n_epochs = 500 #number of epochs +t0 = 1.0 +t1 = 10 + +gamma_j = t0/t1 +j = 0 +for epoch in range(1,n_epochs+1): + for i in range(m): + k = np.random.randint(m) #Pick the k-th minibatch at random + #Compute the gradient using the data in minibatch Bk + #Compute new suggestion for beta + t = epoch*m+i + gamma_j = step_length(t,t0,t1) + j += 1 + +print("gamma_j after %d epochs: %g" % (n_epochs,gamma_j)) +!ec + + + + + + +===== Program for stochastic gradient ===== + +!bc pycod +# Importing various packages +from math import exp, sqrt +from random import random, seed +import numpy as np +import matplotlib.pyplot as plt +from sklearn.linear_model import SGDRegressor + +x = 2*np.random.rand(100,1) +y = 4+3*x+np.random.randn(100,1) + +xb = np.c_[np.ones((100,1)), x] +theta_linreg = np.linalg.inv(xb.T.dot(xb)).dot(xb.T).dot(y) +print("Own inversion") +print(theta_linreg) +sgdreg = SGDRegressor(n_iter = 50, penalty=None, eta0=0.1) +sgdreg.fit(x,y.ravel()) +print("sgdreg from scikit") +print(sgdreg.intercept_, sgdreg.coef_) + + +theta = np.random.randn(2,1) + +eta = 0.1 +Niterations = 1000 +m = 100 + +for iter in range(Niterations): + gradients = 2.0/m*xb.T.dot(xb.dot(theta)-y) + theta -= eta*gradients +print("theta frm own gd") +print(theta) + +xnew = np.array([[0],[2]]) +xbnew = np.c_[np.ones((2,1)), xnew] +ypredict = xbnew.dot(theta) +ypredict2 = xbnew.dot(theta_linreg) + + +n_epochs = 50 +t0, t1 = 5, 50 +m = 100 +def learning_schedule(t): + return t0/(t+t1) + +theta = np.random.randn(2,1) + +for epoch in range(n_epochs): + for i in range(m): + random_index = np.random.randint(m) + xi = xb[random_index:random_index+1] + yi = y[random_index:random_index+1] + gradients = 2 * xi.T.dot(xi.dot(theta)-yi) + eta = learning_schedule(epoch*m+i) + theta = theta - eta*gradients +print("theta from own sdg") +print(theta) + + + + + + +plt.plot(xnew, ypredict, "r-") +plt.plot(xnew, ypredict2, "b-") +plt.plot(x, y ,'ro') +plt.axis([0,2.0,0, 15.0]) +plt.xlabel(r'$x$') +plt.ylabel(r'$y$') +plt.title(r'Random numbers ') +plt.show() + +!ec + + + + + + +===== Using gradient descent methods, limitations ===== + +* _Gradient descent (GD) finds local minima of our function_. Since the GD algorithm is deterministic, if it converges, it will converge to a local minimum of our energy function. Because in ML we are often dealing with extremely rugged landscapes with many local minima, this can lead to poor performance. + +* _GD is sensitive to initial conditions_. One consequence of the local nature of GD is that initial conditions matter. Depending on where one starts, one will end up at a different local minima. Therefore, it is very important to think about how one initializes the training process. This is true for GD as well as more complicated variants of GD. + +* _Gradients are computationally expensive to calculate for large datasets_. In many cases in statistics and ML, the energy function is a sum of terms, with one term for each data point. For example, in linear regression, $E \propto \sum_{i=1}^n (y_i - \mathbf{w}^T\cdot\mathbf{x}_i)^2$; for logistic regression, the square error is replaced by the cross entropy. To calculate the gradient we have to sum over *all* $n$ data points. Doing this at every GD step becomes extremely computationally expensive. An ingenious solution to this, is to calculate the gradients using small subsets of the data called ``mini batches''. This has the added benefit of introducing stochasticity into our algorithm. + +* _GD is very sensitive to choices of learning rates_. GD is extremely sensitive to the choice of learning rates. If the learning rate is very small, the training process take an extremely long time. For larger learning rates, GD can diverge and give poor results. Furthermore, depending on what the local landscape looks like, we have to modify the learning rates to ensure convergence. Ideally, we would *adaptively* choose the learning rates to match the landscape. + +* _GD treats all directions in parameter space uniformly._ Another major drawback of GD is that unlike Newton's method, the learning rate for GD is the same in all directions in parameter space. For this reason, the maximum learning rate is set by the behavior of the steepest direction and this can significantly slow down training. Ideally, we would like to take large steps in flat directions and small steps in steep directions. Since we are exploring rugged landscapes where curvatures change, this requires us to keep track of not only the gradient but second derivatives. The ideal scenario would be to calculate the Hessian but this proves to be too computationally expensive. + +* GD can take exponential time to escape saddle points, even with random initialization. As we mentioned, GD is extremely sensitive to initial condition since it determines the particular local minimum GD would eventually reach. However, even with a good initialization scheme, through the introduction of randomness, GD can still take exponential time to escape saddle points. + + + +===== Momentum based GD ===== + +The stochastic gradient descent (SGD) is almost always used with a *momentum* or inertia term that serves as a memory of the direction we are moving in parameter space. This is typically +implemented as follows +!bt +\begin{align} +\mathbf{v}_{t}&=\gamma \mathbf{v}_{t-1}+\eta_{t}\nabla_\theta E(\boldsymbol{\theta}_t) \nonumber \\ +\boldsymbol{\theta}_{t+1}&= \boldsymbol{\theta}_t -\mathbf{v}_{t}, +\end{align} +!et +where we have introduced a momentum parameter $\gamma$, with $0\le\gamma\le 1$, and for brevity we dropped the explicit notation to indicate the gradient is to be taken over a different mini-batch at each step. We call this algorithm gradient descent with momentum (GDM). From these equations, it is clear that $\mathbf{v}_t$ is a running average of recently encountered gradients and $(1-\gamma)^{-1}$ sets the characteristic time scale for the memory used in the averaging procedure. Consistent with this, when $\gamma=0$, this just reduces down to ordinary SGD as discussed earlier. An equivalent way of writing the updates is +!bt +\[ +\Delta \boldsymbol{\theta}_{t+1} = \gamma \Delta \boldsymbol{\theta}_t -\ \eta_{t}\nabla_\theta E(\boldsymbol{\theta}_t), +\] +!et +where we have defined $\Delta \boldsymbol{\theta}_{t}= \boldsymbol{\theta}_t-\boldsymbol{\theta}_{t-1}$. + + +===== More on momentum based approaches ===== + +Let us try to get more intuition from these equations. It is helpful to consider a simple physical analogy with a particle of mass $m$ moving in a viscous medium with drag coefficient $\mu$ and potential +$E(\mathbf{w})$. If we denote the particle's position by $\mathbf{w}$, then its motion is described by +!bt +\[ +m {d^2 \mathbf{w} \over dt^2} + \mu {d \mathbf{w} \over dt }= -\nabla_w E(\mathbf{w}). +\] +!et +We can discretize this equation in the usual way to get +!bt +\[ +m { \mathbf{w}_{t+\Delta t}-2 \mathbf{w}_{t} +\mathbf{w}_{t-\Delta t} \over (\Delta t)^2}+\mu {\mathbf{w}_{t+\Delta t}- \mathbf{w}_{t} \over \Delta t} = -\nabla_w E(\mathbf{w}). +\] +!et +Rearranging this equation, we can rewrite this as +!bt +\[ +\Delta \mathbf{w}_{t +\Delta t}= - { (\Delta t)^2 \over m +\mu \Delta t} \nabla_w E(\mathbf{w})+ {m \over m +\mu \Delta t} \Delta \mathbf{w}_t. +\] +!et + + +===== Momentum parameter ===== +Notice that this equation is identical to previous one if we identify the position of the particle, $\mathbf{w}$, with the parameters $\boldsymbol{\theta}$. This allows +us to identify the momentum parameter and learning rate with the mass of the particle and the viscous drag as: +!bt +\[ +\gamma= {m \over m +\mu \Delta t }, \qquad \eta = {(\Delta t)^2 \over m +\mu \Delta t}. +\] +!et +Thus, as the name suggests, the momentum parameter is proportional to the mass of the particle and effectively provides inertia. Furthermore, in the large viscosity/small learning rate limit, our memory time scales as $(1-\gamma)^{-1} \approx m/(\mu \Delta t)$. + +Why is momentum useful? SGD momentum helps the gradient descent algorithm gain speed in directions with persistent but small gradients even in the presence of stochasticity, while suppressing oscillations in high-curvature directions. This becomes especially important in situations where the landscape is shallow and flat in some directions and narrow and steep in others. It has been argued that first-order methods (with appropriate initial conditions) can perform comparable to more expensive second order methods, especially in the context of complex deep learning models. + +These beneficial properties of momentum can sometimes become even more pronounced by using a slight modification of the classical momentum algorithm called Nesterov Accelerated Gradient (NAG). + +In the NAG algorithm, rather than calculating the gradient at the current parameters, $\nabla_\theta E(\boldsymbol{\theta}_t)$, one calculates the gradient at the expected value of the parameters given our current momentum, $\nabla_\theta E(\boldsymbol{\theta}_t +\gamma \mathbf{v}_{t-1})$. This yields the NAG update rule +!bt +\begin{align} +\mathbf{v}_{t}&=\gamma \mathbf{v}_{t-1}+\eta_{t}\nabla_\theta E(\boldsymbol{\theta}_t +\gamma \mathbf{v}_{t-1}) \nonumber \\ +\boldsymbol{\theta}_{t+1}&= \boldsymbol{\theta}_t -\mathbf{v}_{t}. +\end{align} +!et +One of the major advantages of NAG is that it allows for the use of a larger learning rate than GDM for the same choice of $\gamma$. + + + +===== Second moment of the gradient ===== + + +In stochastic gradient descent, with and without momentum, we still +have to specify a schedule for tuning the learning rates $\eta_t$ +as a function of time. As discussed in the context of Newton's +method, this presents a number of dilemmas. The learning rate is +limited by the steepest direction which can change depending on the +current position in the landscape. To circumvent this problem, ideally +our algorithm would keep track of curvature and take large steps in +shallow, flat directions and small steps in steep, narrow directions. +Second-order methods accomplish this by calculating or approximating +the Hessian and normalizing the learning rate by the +curvature. However, this is very computationally expensive for +extremely large models. Ideally, we would like to be able to +adaptively change the step size to match the landscape without paying +the steep computational price of calculating or approximating +Hessians. + +Recently, a number of methods have been introduced that accomplish this by tracking not only the gradient, but also the second moment of the gradient. These methods include AdaGrad, AdaDelta, RMS-Prop, and ADAM. + + +===== RMS prop ===== + +In RMS prop, in addition to keeping a running average of the first moment of the gradient, we also keep track of the second moment denoted by $\mathbf{s}_t=\mathbb{E}[\mathbf{g}_t^2]$. The update rule for RMS prop is given by +!bt +\begin{align} +\mathbf{g}_t &= \nabla_\theta E(\boldsymbol{\theta}) \\ +\mathbf{s}_t &=\beta \mathbf{s}_{t-1} +(1-\beta)\mathbf{g}_t^2 \nonumber \\ +\boldsymbol{\theta}_{t+1}&=&\boldsymbol{\theta}_t - \eta_t { \mathbf{g}_t \over \sqrt{\mathbf{s}_t +\epsilon}}, \nonumber +\end{align} +!et +where $\beta$ controls the averaging time of the second moment and is typically taken to be about $\beta=0.9$, $\eta_t$ is a learning rate typically chosen to be $10^{-3}$, and $\epsilon\sim 10^{-8} $ is a small regularization constant to prevent divergences. Multiplication and division by vectors is understood as an element-wise operation. It is clear from this formula that the learning rate is reduced in directions where the norm of the gradient is consistently large. This greatly speeds up the convergence by allowing us to use a larger learning rate for flat directions. + + + +===== ADAM optimizer ===== + +A related algorithm is the ADAM optimizer. In ADAM, we keep a running average of both the first and second moment of the gradient and use this information to adaptively change the learning rate for different parameters. In addition to keeping a running average of the first and second moments of the gradient (i.e. $\mathbf{m}_t=\mathbb{E}[\mathbf{g}_t]$ and $\mathbf{s}_t=\mathbb{E}[\mathbf{g}^2_t]$, respectively), ADAM performs an additional bias correction to account for the fact that we are estimating the first two moments of the gradient using a running average (denoted by the hats in the update rule below). The update rule for ADAM is given by (where multiplication and division are once again understood to be element-wise operations below) +!bt +\begin{align} +\mathbf{g}_t &= \nabla_\theta E(\boldsymbol{\theta}) \\ +\mathbf{m}_t &= \beta_1 \mathbf{m}_{t-1} + (1-\beta_1) \mathbf{g}_t \nonumber \\ +\mathbf{s}_t &=\beta_2 \mathbf{s}_{t-1} +(1-\beta_2)\mathbf{g}_t^2 \nonumber \\ +\hat{\mathbf{m}}_t&={\mathbf{m}_t \over 1-\beta_1^t} \nonumber \\ +\hat{\mathbf{s}}_t &={\mathbf{s}_t \over1-\beta_2^t} \nonumber \\ +\boldsymbol{\theta}_{t+1}&=\boldsymbol{\theta}_t - \eta_t { \hat{\mathbf{m}}_t \over \sqrt{\hat{\mathbf{s}}_t} +\epsilon}, \nonumber \\ +\end{align} +!et +where $\beta_1$ and $\beta_2$ set the memory lifetime of the first and second moment and are typically taken to be $0.9$ and $0.99$ respectively, and $\eta$ and $\epsilon$ are identical to RMSprop. + +Like in RMSprop, the effective step size of a parameter depends on the magnitude of its gradient squared. To understand this better, let us rewrite this expression in terms of the variance $\boldsymbol{\sigma}_t^2 = \hat{\mathbf{s}}_t - (\hat{\mathbf{m}}_t)^2$. Consider a single parameter $\theta_t$. The update rule for this parameter is given by +!bt +\[ +\Delta \theta_{t+1}= -\eta_t { \hat{m}_t \over \sqrt{\sigma_t^2 + m_t^2 }+\epsilon}. +\] +!et + + + + + +===== Practical tips ===== + +* _Randomize the data when making mini-batches_. It is always important to randomly shuffle the data when forming mini-batches. Otherwise, the gradient descent method can fit spurious correlations resulting from the order in which data is presented. + +* _Transform your inputs_. Learning becomes difficult when our landscape has a mixture of steep and flat directions. One simple trick for minimizing these situations is to standardize the data by subtracting the mean and normalizing the variance of input variables. Whenever possible, also decorrelate the inputs. To understand why this is helpful, consider the case of linear regression. It is easy to show that for the squared error cost function, the Hessian of the energy matrix is just the correlation matrix between the inputs. Thus, by standardizing the inputs, we are ensuring that the landscape looks homogeneous in all directions in parameter space. Since most deep networks can be viewed as linear transformations followed by a non-linearity at each layer, we expect this intuition to hold beyond the linear case. + +* _Monitor the out-of-sample performance._ Always monitor the performance of your model on a validation set (a small portion of the training data that is held out of the training process to serve as a proxy for the test set. If the validation error starts increasing, then the model is beginning to overfit. Terminate the learning process. This *early stopping* significantly improves performance in many settings. + +* _Adaptive optimization methods don't always have good generalization._ Recent studies have shown that adaptive methods such as ADAM, RMSPorp, and AdaGrad tend to have poor generalization compared to SGD or SGD with momentum, particularly in the high-dimensional limit (i.e. the number of parameters exceeds the number of data points). Although it is not clear at this stage why these methods perform so well in training deep neural networks, simpler procedures like properly-tuned SGD may work as well or better in these applications. + + +======= Logistic Regression ======= + +In linear regression our main interest was centered on learning the +coefficients of a functional fit (say a polynomial) in order to be +able to predict the response of a continuous variable on some unseen +data. The fit to the continuous variable $y_i$ is based on some +independent variables $\hat{x}_i$. Linear regression resulted in +analytical expressions (in terms of matrices to invert) for several +quantities, ranging from the variance and thereby the confidence +intervals of the parameters $\hat{\beta}$ to the mean squared +error. If we can invert the product of the design matrices, linear +regression gives then a simple recipe for fitting our data. + + +Classification problems, however, are concerned with outcomes taking +the form of discrete variables (i.e. categories). We may for example, +on the basis of DNA sequencing for a number of patients, like to find +out which mutations are important for a certain disease; or based on +scans of various patients' brains, figure out if there is a tumor or +not; or given a specific physical system, we'd like to identify its +state, say whether it is an ordered or disordered system (typical +situation in solid state physics); or classify the status of a +patient, whether she/he has a stroke or not and many other similar +situations. + +The most common situation we encounter when we apply logistic +regression is that of two possible outcomes, normally denoted as a +binary outcome, true or false, positive or negative, success or +failure etc. + + +===== Optimization and Deep learning ===== + +Logistic regression will also serve as our stepping stone towards neural +network algorithms and supervised deep learning. For logistic +learning, the minimization of the cost function leads to a non-linear +equation in the parameters $\hat{\beta}$. The optmization of the problem calls therefore for minimization algorithms. This forms the bottle neck of all machine learning algorithms, namely how to find reliable minima of a multi-variable function. This leads us to the family of gradient descent methods. The latter are the working horses of basically all modern machine learning algorithms. + +We note also that many of the topics discussed here +regression are also commonly used in modern supervised Deep Learning +models, as we will see later. + + + +===== Basics ===== + +We consider the case where the dependent variables, also called the +responses or the outcomes, $y_i$ are discrete and only take values +from $k=0,\dots,K-1$ (i.e. $K$ classes). + +The goal is to predict the +output classes from the design matrix $\hat{X}\in\mathbb{R}^{n\times p}$ +made of $n$ samples, each of which carries $p$ features or predictors. The +primary goal is to identify the classes to which new unseen samples +belong. + +Let us specialize to the case of two classes only, with outputs $y_i=0$ and $y_i=1$. Our outcomes could represent the status of a credit card user who could default or not on her/his credit card debt. That is +!bt +\[ +y_i = \begin{bmatrix} 0 & \mathrm{no}\\ 1 & \mathrm{yes} \end{bmatrix}. +\] +!et + + + + +===== Linear classifier ===== + +Before moving to the logistic model, let us try to use our linear regression model to classify these two outcomes. We could for example fit a linear model to the default case if $y_i > 0.5$ and the no default case $y_i \leq 0.5$. + +We would then have our +weighted linear combination, namely +!bt +\begin{equation} +\hat{y} = \hat{X}^T\hat{\beta} + \hat{\epsilon}, +\end{equation} +!et +where $\hat{y}$ is a vector representing the possible outcomes, $\hat{X}$ is our +$n\times p$ design matrix and $\hat{\beta}$ represents our estimators/predictors. + + +===== Some selected properties ===== + +The main problem with our function is that it +takes values on the entire real axis. In the case of +logistic regression, however, the labels $y_i$ are discrete +variables. + +One simple way to get a discrete output is to have sign +functions that map the output of a linear regressor to values $\{0,1\}$, +$f(s_i)=sign(s_i)=1$ if $s_i\ge 0$ and 0 if otherwise. +We will encounter this model in our first demonstration of neural networks. Historically it is called the ``perceptron" model in the machine learning +literature. This model is extremely simple. However, in many cases it is more +favorable to use a ``soft" classifier that outputs +the probability of a given category. This leads us to the logistic function. + +The code for plotting the perceptron can be seen here. This si nothing but the standard "Heaviside step function":"https://en.wikipedia.org/wiki/Heaviside_step_function". +!bc pycod + +!ec + + + +===== The logistic function ===== + +The perceptron is an example of a ``hard classification'' model. We +will encounter this model when we discuss neural networks as +well. Each datapoint is deterministically assigned to a category (i.e +$y_i=0$ or $y_i=1$). In many cases, it is favorable to have a ``soft'' +classifier that outputs the probability of a given category rather +than a single value. For example, given $x_i$, the classifier +outputs the probability of being in a category $k$. Logistic regression +is the most common example of a so-called soft classifier. In logistic +regression, the probability that a data point $x_i$ +belongs to a category $y_i=\{0,1\}$ is given by the so-called logit function (or Sigmoid) which is meant to represent the likelihood for a given event, +!bt +\[ +p(t) = \frac{1}{1+\mathrm \exp{-t}}=\frac{\exp{t}}{1+\mathrm \exp{t}}. +\] +!et +Note that $1-p(t)= p(-t)$. +The following code plots the logistic function. +!bc pycod + +!ec + + + + +===== Two parameters ===== + +We assume now that we have two classes with $y_i$ either $0$ or $1$. Furthermore we assume also that we have only two parameters $\beta$ in our fitting of the Sigmoid function, that is we define probabilities +!bt +\begin{align*} +p(y_i=1|x_i,\hat{\beta}) &= \frac{\exp{(\beta_0+\beta_1x_i)}}{1+\exp{(\beta_0+\beta_1x_i)}},\nonumber\\ +p(y_i=0|x_i,\hat{\beta}) &= 1 - p(y_i=1|x_i,\hat{\beta}), +\end{align*} +!et +where $\hat{\beta}$ are the weights we wish to extract from data, in our case $\beta_0$ and $\beta_1$. + +Note that we used +!bt +\[ +p(y_i=0\vert x_i, \hat{\beta}) = 1-p(y_i=1\vert x_i, \hat{\beta}). +\] +!et + + +===== Maximum likelihood ===== + +In order to define the total likelihood for all possible outcomes from a +dataset $\mathcal{D}=\{(y_i,x_i)\}$, with the binary labels +$y_i\in\{0,1\}$ and where the data points are drawn independently, we use the so-called "Maximum Likelihood Estimation":"https://en.wikipedia.org/wiki/Maximum_likelihood_estimation" (MLE) principle. +We aim thus at maximizing +the probability of seeing the observed data. We can then approximate the +likelihood in terms of the product of the individual probabilities of a specific outcome $y_i$, that is +!bt +\begin{align*} +P(\mathcal{D}|\hat{\beta})& = \prod_{i=1}^n \left[p(y_i=1|x_i,\hat{\beta})\right]^{y_i}\left[1-p(y_i=1|x_i,\hat{\beta}))\right]^{1-y_i}\nonumber \\ +\end{align*} +!et +from which we obtain the log-likelihood and our _cost/loss_ function +!bt +\[ +\mathcal{C}(\hat{\beta}) = \sum_{i=1}^n \left( y_i\log{p(y_i=1|x_i,\hat{\beta})} + (1-y_i)\log\left[1-p(y_i=1|x_i,\hat{\beta}))\right]\right). +\] +!et + + +===== The cost function rewritten ===== + +Reordering the logarithms, we can rewrite the _cost/loss_ function as +!bt +\[ +\mathcal{C}(\hat{\beta}) = \sum_{i=1}^n \left(y_i(\beta_0+\beta_1x_i) -\log{(1+\exp{(\beta_0+\beta_1x_i)})}\right). +\] +!et + +The maximum likelihood estimator is defined as the set of parameters that maximize the log-likelihood where we maximize with respect to $\beta$. +Since the cost (error) function is just the negative log-likelihood, for logistic regression we have that +!bt +\[ +\mathcal{C}(\hat{\beta})=-\sum_{i=1}^n \left(y_i(\beta_0+\beta_1x_i) -\log{(1+\exp{(\beta_0+\beta_1x_i)})}\right). +\] +!et +This equation is known in statistics as the _cross entropy_. Finally, we note that just as in linear regression, +in practice we often supplement the cross-entropy with additional regularization terms, usually $L_1$ and $L_2$ regularization as we did for Ridge and Lasso regression. + + +===== Minimizing the cross entropy ===== + +The cross entropy is a convex function of the weights $\hat{\beta}$ and, +therefore, any local minimizer is a global minimizer. + + +Minimizing this +cost function with respect to the two parameters $\beta_0$ and $\beta_1$ we obtain + +!bt +\[ +\frac{\partial \mathcal{C}(\hat{\beta})}{\partial \beta_0} = -\sum_{i=1}^n \left(y_i -\frac{\exp{(\beta_0+\beta_1x_i)}}{1+\exp{(\beta_0+\beta_1x_i)}}\right), +\] +!et +and +!bt +\[ +\frac{\partial \mathcal{C}(\hat{\beta})}{\partial \beta_1} = -\sum_{i=1}^n \left(y_ix_i -x_i\frac{\exp{(\beta_0+\beta_1x_i)}}{1+\exp{(\beta_0+\beta_1x_i)}}\right). +\] +!et + + +===== A more compact expression ===== + +Let us now define a vector $\hat{y}$ with $n$ elements $y_i$, an +$n\times p$ matrix $\hat{X}$ which contains the $x_i$ values and a +vector $\hat{p}$ of fitted probabilities $p(y_i\vert x_i,\hat{\beta})$. We can rewrite in a more compact form the first +derivative of cost function as + +!bt +\[ +\frac{\partial \mathcal{C}(\hat{\beta})}{\partial \hat{\beta}} = -\hat{X}^T\left(\hat{y}-\hat{p}\right). +\] +!et + +If we in addition define a diagonal matrix $\hat{W}$ with elements +$p(y_i\vert x_i,\hat{\beta})(1-p(y_i\vert x_i,\hat{\beta})$, we can obtain a compact expression of the second derivative as + +!bt +\[ +\frac{\partial^2 \mathcal{C}(\hat{\beta})}{\partial \hat{\beta}\partial \hat{\beta}^T} = \hat{X}^T\hat{W}\hat{X}. +\] +!et + + +===== Extending to more predictors ===== + +Within a binary classification problem, we can easily expand our model to include multiple predictors. Our ratio between likelihoods is then with $p$ predictors +!bt +\[ +\log{ \frac{p(\hat{\beta}\hat{x})}{1-p(\hat{\beta}\hat{x})}} = \beta_0+\beta_1x_1+\beta_2x_2+\dots+\beta_px_p. +\] +!et +Here we defined $\hat{x}=[1,x_1,x_2,\dots,x_p]$ and $\hat{\beta}=[\beta_0, \beta_1, \dots, \beta_p]$ leading to +!bt +\[ +p(\hat{\beta}\hat{x})=\frac{ \exp{(\beta_0+\beta_1x_1+\beta_2x_2+\dots+\beta_px_p)}}{1+\exp{(\beta_0+\beta_1x_1+\beta_2x_2+\dots+\beta_px_p)}}. +\] +!et + + +===== Including more classes ===== + +Till now we have mainly focused on two classes, the so-called binary system. Suppose we wish to extend to $K$ classes. +Let us for the sake of simplicity assume we have only two predictors. We have then following model +!bt +\[ +\log{\frac{p(C=1\vert x)}{p(K\vert x)}} = \beta_{10}+\beta_{11}x_1, +\] +!et +!bt +\[ +\log{\frac{p(C=2\vert x)}{p(K\vert x)}} = \beta_{20}+\beta_{21}x_1, +\] +!et +and so on till the class $C=K-1$ class +!bt +\[ +\log{\frac{p(C=K-1\vert x)}{p(K\vert x)}} = \beta_{(K-1)0}+\beta_{(K-1)1}x_1, +\] +!et +and the model is specified in term of $K-1$ so-called log-odds or _logit_ transformations. + + + +===== The Softmax function ===== + +In our discussion of neural networks we will encounter the above again in terms of the so-called _Softmax_ function. + +The softmax function is used in various multiclass classification +methods, such as multinomial logistic regression (also known as +softmax regression), multiclass linear discriminant +analysis, naive Bayes classifiers, and artificial neural networks. +Specifically, in multinomial logistic regression and linear +discriminant analysis, the input to the function is the result of $K$ +distinct linear functions, and the predicted probability for the $k$-th +class given a sample vector $\hat{x}$ and a weighting vector $\hat{\beta}$ is (with two predictors): + +!bt +\[ +p(C=k\vert \mathbf {x} )=\frac{\exp{(\beta_{k0}+\beta_{k1}x_1)}}{1+\sum_{l=1}^{K-1}\exp{(\beta_{l0}+\beta_{l1}x_1)}}. +\] +!et +It is easy to extend to more predictors. The final class is +!bt +\[ +p(C=K\vert \mathbf {x} )=\frac{1}{1+\sum_{l=1}^{K-1}\exp{(\beta_{l0}+\beta_{l1}x_1)}}, +\] +!et +and they sum to one. Our earlier discussions were all specialized to the case with two classes only. It is easy to see from the above that what we derived earlier is compatible with these equations. + +To find the optimal parameters we would typically use a gradient descent method. +Newton's method and gradient descent methods are discussed in the material on "optimization methods":"https://compphysics.github.io/MachineLearning/doc/pub/Splines/html/Splines-bs.html". + + + + +===== A _scikit-learn_ example ===== + +!bc pycod +import numpy as np +import matplotlib.pyplot as plt +from sklearn import datasets +iris = datasets.load_iris() +list(iris.keys()) +['data', 'target_names', 'feature_names', 'target', 'DESCR'] +X = iris["data"][:, 3:] # petal width +y = (iris["target"] == 2).astype(np.int) # 1 if Iris-Virginica, else 0 + +from sklearn.linear_model import LogisticRegression +log_reg = LogisticRegression() +log_reg.fit(X, y) + +X_new = np.linspace(0, 3, 1000).reshape(-1, 1) +y_proba = log_reg.predict_proba(X_new) +plt.plot(X_new, y_proba[:, 1], "g-", label="Iris-Virginica") +plt.plot(X_new, y_proba[:, 0], "b--", label="Not Iris-Virginica") +plt.show() + +!ec + + + +===== A simple classification problem ===== +!bc pycod +import numpy as np +from sklearn import datasets, linear_model +import matplotlib.pyplot as plt + + +def generate_data(): + np.random.seed(0) + X, y = datasets.make_moons(200, noise=0.20) + return X, y + + +def visualize(X, y, clf): + # plt.scatter(X[:, 0], X[:, 1], s=40, c=y, cmap=plt.cm.Spectral) + # plt.show() + plot_decision_boundary(lambda x: clf.predict(x), X, y) + plt.title("Logistic Regression") + + +def plot_decision_boundary(pred_func, X, y): + # Set min and max values and give it some padding + x_min, x_max = X[:, 0].min() - .5, X[:, 0].max() + .5 + y_min, y_max = X[:, 1].min() - .5, X[:, 1].max() + .5 + h = 0.01 + # Generate a grid of points with distance h between them + xx, yy = np.meshgrid(np.arange(x_min, x_max, h), np.arange(y_min, y_max, h)) + # Predict the function value for the whole gid + Z = pred_func(np.c_[xx.ravel(), yy.ravel()]) + Z = Z.reshape(xx.shape) + # Plot the contour and training examples + plt.contourf(xx, yy, Z, cmap=plt.cm.Spectral) + plt.scatter(X[:, 0], X[:, 1], c=y, cmap=plt.cm.Spectral) + plt.show() + + +def classify(X, y): + clf = linear_model.LogisticRegressionCV() + clf.fit(X, y) + return clf + + +def main(): + X, y = generate_data() + # visualize(X, y) + clf = classify(X, y) + visualize(X, y, clf) + + +if __name__ == "__main__": + main() +!ec + + +===== The two-dimensional Ising model, Predicting phase transition of the two-dimensional Ising model ===== + +The Hamiltonian of the two-dimensional Ising model without an external field for a constant coupling constant $J$ is given by +!bt +\begin{align} + H = -J \sum_{\langle ij\rangle} S_i S_j, +\end{align} +!et +where $S_i \in \{-1, 1\}$ and $\langle ij \rangle$ signifies that we only iterate over the nearest neighbors in the lattice. We will be looking at a system of $L = 40$ spins in each dimension, i.e., $L^2 = 1600$ spins in total. Opposed to the one-dimensional Ising model we will get a phase transition from an _ordered_ phase to a _disordered_ phase at the critical temperature + +!bt +\begin{align} + \frac{T_c}{J} = \frac{2}{\log\left(1 + \sqrt{2}\right)} \approx 2.26, +\end{align} +!et +as shown by Lars Onsager. + + +Here we use _logistic regression_ to predict when a phase transition +occurs. The data we will look at is a set of spin configurations, +i.e., individual lattices with spins, labeled _ordered_ `1` or +_disordered_ `0`. Our job is to build a model which will take in a +spin configuration and predict whether or not the spin configuration +constitutes an ordered or a disordered phase. To achieve this we will +represent the lattices as flattened arrays with $1600$ elements +instead of a matrix of $40 \times 40$ elements. As an extra test of +the performance of the algorithms we will divide the dataset into +three pieces. We will do a conventional train-test-split on a +combination of totally ordered and totally disordered phases. The +remaining "critical-like" states will be used as test data which we +hope the model will be able to make good extrapolated predictions on. + + +!bc pycod +import pickle +import os +import glob +import numpy as np +import pandas as pd +import matplotlib.pyplot as plt +import seaborn as sns +import sklearn.model_selection as skms +import sklearn.linear_model as skl +import sklearn.metrics as skm +import tqdm +import copy +import time +from IPython.display import display + +%matplotlib inline + +sns.set(color_codes=True) +!ec + + +===== Reading in the data ===== + +Using the data from "Mehta et al.":"https://physics.bu.edu/~pankajm/ML-Review-Datasets/isingMC/" (specifically the two datasets named `Ising2DFM_reSample_L40_T=All.pkl` and `Ising2DFM_reSample_L40_T=All_labels.pkl`) we have to unpack the data into numpy arrays. + + +!bc pycod +filenames = glob.glob(os.path.join("..", "dat", "*")) +label_filename = list(filter(lambda x: "label" in x, filenames))[0] +dat_filename = list(filter(lambda x: "label" not in x, filenames))[0] + +# Read in the labels +with open(label_filename, "rb") as f: + labels = pickle.load(f) + +# Read in the corresponding configurations +with open(dat_filename, "rb") as f: + data = np.unpackbits(pickle.load(f)).reshape(-1, 1600).astype("int") + +# Set spin-down to -1 +data[data == 0] = -1 +!ec + +This dataset consists of $10000$ samples, i.e., $10000$ spin +configurations with $40 \times 40$ spins each, for $16$ temperatures +between $0.25$ to $4.0$. Next we create a train/test-split and keep +the data in the critical phase as a separate dataset for +extrapolation-testing. + + +!bc pycod +# Set up slices of the dataset +ordered = slice(0, 70000) +critical = slice(70000, 100000) +disordered = slice(100000, 160000) + +X_train, X_test, y_train, y_test = skms.train_test_split( + np.concatenate((data[ordered], data[disordered])), + np.concatenate((labels[ordered], labels[disordered])), + test_size=0.95 +) +!ec + +Using a small training set yields a better accuracy. This will be discussed in the end. + + +===== Logistic regression ===== + +Logistic regression is a linear model for classification. Recalling +the cost function for ordinary least squares with both L2 (ridge) and +L1 (LASSO) penalties we will see that the logistic cost function is +very similar. In OLS we wish to predict a continuous variable +$\hat{y}$ using +!bt +\begin{align} + \hat{y} = X\omega, +\end{align} +!et + +where $X \in \mathbb{R}^{n \times p}$ is the input data and $\omega^{p +\times d}$ are the weights of the regression. In a classification +setting (binary classification in our situation) we are interested in +a positive or negative answer. We can thus define either answer to be +above or below some threshold. But, in order to limit the size of the +answer and also to get a probability interpretation on how sure we are +for either answer we can compute the sigmoid function of OLS. That is, + +!bt +\begin{align} + f(X\omega) = \frac{1}{1 + \exp(-X\omega)}. +\end{align} +!et +We are thus interested in minizming the following cost function +!bt +\begin{align} + C(X, \omega) = \sum_{i = 1}^n \left\{ + - y_i\log\left( f(x_i^T\omega) \right) + - (1 - y_i)\log\left[1 - f(x_i^T\omega)\right] + \right\}, +\end{align} +!et + +where we will restrict ourselves to a value for $f(z)$ as the sigmoid +described above. We can also tack on a L2 (Ridge) or L1 (LASSO) +penalization to this cost function in the same manner we did for +linear regression. + + +===== Exploring the logistic regression ===== + +The penalization factor $\lambda$ is inverted in the case of the +logistic regression model we use. We will explore several values of +$\lambda$ using both L1 and L2 penalization. We do this using a grid +search over different parameters and run a 3-fold cross validation for +each configuration. In other words, we fit a model 3 times for each +configuration of the hyper parameters. + + +!bc pycod +lambdas = np.logspace(-7, -1, 7) + +param_grid = { + "C": list(1.0/lambdas), + "penalty": ["l1", "l2"] +} +clf = skms.GridSearchCV( + skl.LogisticRegression(), + param_grid=param_grid, + n_jobs=-1, + return_train_score=True +) +t0 = time.time() +clf.fit(X_train, y_train) +t1 = time.time() + +print ( + "Time spent fitting GridSearchCV(LogisticRegression): {0:.3f} sec".format( + t1 - t0 + ) +) +!ec + +We can see that logistic regression is quite slow and using the grid +search and cross validation results in quite a heavy +computation. Below we show the results of the different +configurations. + + +!bc pycod +logreg_df = pd.DataFrame(clf.cv_results_) + +display(logreg_df) +!ec + + +===== Accuracy of a classification model ===== + +To determine how well a classification model is performing we count +the number of correctly labeled classes and divide by the number of +classes in total. The accuracy is thus given by + +!bt +\begin{align} + a(y, \hat{y}) = \frac{1}{n}\sum_{i = 1}^{n} I(y_i = \hat{y}_i), +\end{align} +!et + +where $I(y_i = \hat{y}_i)$ is the indicator function given by + +!bt +\begin{align} + I(x = y) = \begin{array}{cc} + 1 & x = y, \\ + 0 & x \neq y. + \end{array} +\end{align} +!et + +This is the accuracy provided by Scikit-learn when using _sklearn.metrics.accuracyscore_. + +Below we compute the accuracy of the best fit model on the training data (which should give a good accuracy), the test data (which has not been shown to the model) and the critical data (completely new data that needs to be extrapolated). + + +!bc pycod +train_accuracy = skm.accuracy_score(y_train, clf.predict(X_train)) +test_accuracy = skm.accuracy_score(y_test, clf.predict(X_test)) +critical_accuracy = skm.accuracy_score(labels[critical], clf.predict(data[critical])) + +print ("Accuracy on train data: {0}".format(train_accuracy)) +print ("Accuracy on test data: {0}".format(test_accuracy)) +print ("Accuracy on critical data: {0}".format(critical_accuracy)) +!ec + +We can see that we get quite good accuracy on the training data, but gradually worsening accuracy on the test and critical data. + + +===== Analyzing the results ===== + +Below we show a different metric for determining the quality of our +model, namely the _reciever operating characteristic_ (ROC). The ROC +curve tells us how well the model correctly classifies the different +labels. We plot the _true positive rate_ (the rate of predicted +positive classes that are positive) versus the _false positive rate_ +(the rate of predicted positive classes that are negative). The ROC +curve is built by computing the true positive rate and the false +positive rate for varying _thresholds_, i.e, which probability we +should acredit a certain class. + +By computing the _area under the curve_ (AUC) of the ROC curve we get an estimate of how well our model is performing. Pure guessing will get an AUC of $0.5$. A perfect score will get an AUC of $1.0$. + + +!bc pycod +fig = plt.figure(figsize=(20, 14)) + +for (_X, _y), label in zip( + [ + (X_train, y_train), + (X_test, y_test), + (data[critical], labels[critical]) + ], + ["Train", "Test", "Critical"] +): + proba = clf.predict_proba(_X) + fpr, tpr, _ = skm.roc_curve(_y, proba[:, 1]) + roc_auc = skm.auc(fpr, tpr) + + print ("LogisticRegression AUC ({0}): {1}".format(label, roc_auc)) + + plt.plot(fpr, tpr, label="{0} (AUC = {1})".format(label, roc_auc), linewidth=4.0) + +plt.plot([0, 1], [0, 1], "--", label="Guessing (AUC = 0.5)", linewidth=4.0) + +plt.title(r"The ROC curve for LogisticRegression", fontsize=18) +plt.xlabel(r"False positive rate", fontsize=18) +plt.ylabel(r"True positive rate", fontsize=18) +plt.axis([-0.01, 1.01, -0.01, 1.01]) +plt.xticks(fontsize=18) +plt.yticks(fontsize=18) +plt.legend(loc="best", fontsize=18) +plt.show() +!ec + +We can see that this plot of the ROC looks very strange. This tells us +that logistic regression is quite inept at predicting the Ising model +transition and is therefore highly non-linear. The ROC curve for the +training data looks quite good, but as the testing data is so far off +we see that we are dealing with an overfit model. + +A previous run with $50\%$ of the data used for training yielded a +worse performance than using a smaller training set. This again gives +confidence to the fact that logistic regression is not able to +correctly fit the Ising model as it is not a linear model. + + + + + + + +======= Neural networks ======= + +Artificial neural networks are computational systems that can learn to +perform tasks by considering examples, generally without being +programmed with any task-specific rules. It is supposed to mimic a +biological system, wherein neurons interact by sending signals in the +form of mathematical functions between layers. All layers can contain +an arbitrary number of neurons, and each connection is represented by +a weight variable. + + + +===== Artificial neurons ===== + +The field of artificial neural networks has a long history of +development, and is closely connected with the advancement of computer +science and computers in general. A model of artificial neurons was +first developed by McCulloch and Pitts in 1943 to study signal +processing in the brain and has later been refined by others. The +general idea is to mimic neural networks in the human brain, which is +composed of billions of neurons that communicate with each other by +sending electrical signals. Each neuron accumulates its incoming +signals, which must exceed an activation threshold to yield an +output. If the threshold is not overcome, the neuron remains inactive, +i.e. has zero output. + +This behaviour has inspired a simple mathematical model for an artificial neuron. + +!bt +\begin{equation} + y = f\left(\sum_{i=1}^n w_ix_i\right) = f(u) + label{artificialNeuron} +\end{equation} +!et +Here, the output $y$ of the neuron is the value of its activation function, which have as input +a weighted sum of signals $x_i, \dots ,x_n$ received by $n$ other neurons. + +Conceptually, it is helpful to divide neural networks into four +categories: +o general purpose neural networks for supervised learning, +o neural networks designed specifically for image processing, the most prominent example of this class being Convolutional Neural Networks (CNNs), +o neural networks for sequential data such as Recurrent Neural Networks (RNNs), and +o neural networks for unsupervised learning such as Deep Boltzmann Machines. + + +In natural science, DNNs and CNNs have already found numerous +applications. In statistical physics, they have been applied to detect +phase transitions in 2D Ising and Potts models, lattice gauge +theories, and different phases of polymers, or solving the +Navier-Stokes equation in weather forecasting. Deep learning has also +found interesting applications in quantum physics. Various quantum +phase transitions can be detected and studied using DNNs and CNNs, +topological phases, and even non-equilibrium many-body +localization. Representing quantum states as DNNs quantum state +tomography are among some of the impressive achievements to reveal the +potential of DNNs to facilitate the study of quantum systems. + +In quantum information theory, it has been shown that one can perform +gate decompositions with the help of neural. + +The applications are not limited to the natural sciences. There is a +plethora of applications in essentially all disciplines, from the +humanities to life science and medicine. + + +===== Neural network types ===== + +An artificial neural network (ANN), is a computational model that +consists of layers of connected neurons, or nodes or units. We will +refer to these interchangeably as units or nodes, and sometimes as +neurons. + +It is supposed to mimic a biological nervous system by letting each +neuron interact with other neurons by sending signals in the form of +mathematical functions between layers. A wide variety of different +ANNs have been developed, but most of them consist of an input layer, +an output layer and eventual layers in-between, called *hidden +layers*. All layers can contain an arbitrary number of nodes, and each +connection between two nodes is associated with a weight variable. + +Neural networks (also called neural nets) are neural-inspired +nonlinear models for supervised learning. As we will see, neural nets +can be viewed as natural, more powerful extensions of supervised +learning methods such as linear and logistic regression and soft-max +methods we discussed earlier. + + + +===== Feed-forward neural networks ===== + +The feed-forward neural network (FFNN) was the first and simplest type +of ANNs that were devised. In this network, the information moves in +only one direction: forward through the layers. + +Nodes are represented by circles, while the arrows display the +connections between the nodes, including the direction of information +flow. Additionally, each arrow corresponds to a weight variable +(figure to come). We observe that each node in a layer is connected +to *all* nodes in the subsequent layer, making this a so-called +*fully-connected* FFNN. + + + + +===== Convolutional Neural Network ===== + +A different variant of FFNNs are *convolutional neural networks* +(CNNs), which have a connectivity pattern inspired by the animal +visual cortex. Individual neurons in the visual cortex only respond to +stimuli from small sub-regions of the visual field, called a receptive +field. This makes the neurons well-suited to exploit the strong +spatially local correlation present in natural images. The response of +each neuron can be approximated mathematically as a convolution +operation. (figure to come) + +Convolutional neural networks emulate the behaviour of neurons in the +visual cortex by enforcing a *local* connectivity pattern between +nodes of adjacent layers: Each node in a convolutional layer is +connected only to a subset of the nodes in the previous layer, in +contrast to the fully-connected FFNN. Often, CNNs consist of several +convolutional layers that learn local features of the input, with a +fully-connected layer at the end, which gathers all the local data and +produces the outputs. They have wide applications in image and video +recognition. + + +===== Recurrent neural networks ===== + +So far we have only mentioned ANNs where information flows in one +direction: forward. *Recurrent neural networks* on the other hand, +have connections between nodes that form directed *cycles*. This +creates a form of internal memory which are able to capture +information on what has been calculated before; the output is +dependent on the previous computations. Recurrent NNs make use of +sequential information by performing the same task for every element +in a sequence, where each element depends on previous elements. An +example of such information is sentences, making recurrent NNs +especially well-suited for handwriting and speech recognition. + + +===== Other types of networks ===== + +There are many other kinds of ANNs that have been developed. One type +that is specifically designed for interpolation in multidimensional +space is the radial basis function (RBF) network. RBFs are typically +made up of three layers: an input layer, a hidden layer with +non-linear radial symmetric activation functions and a linear output +layer (''linear'' here means that each node in the output layer has a +linear activation function). The layers are normally fully-connected +and there are no cycles, thus RBFs can be viewed as a type of +fully-connected FFNN. They are however usually treated as a separate +type of NN due the unusual activation functions. + + +===== Multilayer perceptrons ===== + +One uses often so-called fully-connected feed-forward neural networks +with three or more layers (an input layer, one or more hidden layers +and an output layer) consisting of neurons that have non-linear +activation functions. + +Such networks are often called *multilayer perceptrons* (MLPs). + + +===== Why multilayer perceptrons? ===== + +According to the *Universal approximation theorem*, a feed-forward +neural network with just a single hidden layer containing a finite +number of neurons can approximate a continuous multidimensional +function to arbitrary accuracy, assuming the activation function for +the hidden layer is a _non-constant, bounded and +monotonically-increasing continuous function_. + +Note that the requirements on the activation function only applies to +the hidden layer, the output nodes are always assumed to be linear, so +as to not restrict the range of output values. + + + +===== Mathematical model ===== + +The output $y$ is produced via the activation function $f$ +!bt +\[ + y = f\left(\sum_{i=1}^n w_ix_i + b_i\right) = f(z), +\] +!et +This function receives $x_i$ as inputs. +Here the activation $z=(\sum_{i=1}^n w_ix_i+b_i)$. +In an FFNN of such neurons, the *inputs* $x_i$ are the *outputs* of +the neurons in the preceding layer. Furthermore, an MLP is +fully-connected, which means that each neuron receives a weighted sum +of the outputs of *all* neurons in the previous layer. + + +===== Mathematical model ===== + +First, for each node $i$ in the first hidden layer, we calculate a weighted sum $z_i^1$ of the input coordinates $x_j$, + +!bt +\begin{equation} z_i^1 = \sum_{j=1}^{M} w_{ij}^1 x_j + b_i^1 +\end{equation} +!et + +Here $b_i$ is the so-called bias which is normally needed in +case of zero activation weights or inputs. How to fix the biases and +the weights will be discussed below. The value of $z_i^1$ is the +argument to the activation function $f_i$ of each node $i$, The +variable $M$ stands for all possible inputs to a given node $i$ in the +first layer. We define the output $y_i^1$ of all neurons in layer 1 as + +!bt +\begin{equation} + y_i^1 = f(z_i^1) = f\left(\sum_{j=1}^M w_{ij}^1 x_j + b_i^1\right) + label{outputLayer1} +\end{equation} +!et + +where we assume that all nodes in the same layer have identical +activation functions, hence the notation $f$. In general, we could assume in the more general case that different layers have different activation functions. +In this case we would identify these functions with a superscript $l$ for the $l$-th layer, + +!bt +\begin{equation} + y_i^l = f^l(u_i^l) = f^l\left(\sum_{j=1}^{N_{l-1}} w_{ij}^l y_j^{l-1} + b_i^l\right) + label{generalLayer} +\end{equation} +!et + +where $N_l$ is the number of nodes in layer $l$. When the output of +all the nodes in the first hidden layer are computed, the values of +the subsequent layer can be calculated and so forth until the output +is obtained. + + + + +===== Mathematical model ===== + +The output of neuron $i$ in layer 2 is thus, + +!bt +\begin{align} + y_i^2 &= f^2\left(\sum_{j=1}^N w_{ij}^2 y_j^1 + b_i^2\right) \\ + &= f^2\left[\sum_{j=1}^N w_{ij}^2f^1\left(\sum_{k=1}^M w_{jk}^1 x_k + b_j^1\right) + b_i^2\right] + label{outputLayer2} +\end{align} +!et +where we have substituted $y_k^1$ with the inputs $x_k$. Finally, the ANN output reads + +!bt +\begin{align} + y_i^3 &= f^3\left(\sum_{j=1}^N w_{ij}^3 y_j^2 + b_i^3\right) \\ + &= f_3\left[\sum_{j} w_{ij}^3 f^2\left(\sum_{k} w_{jk}^2 f^1\left(\sum_{m} w_{km}^1 x_m + b_k^1\right) + b_j^2\right) + + b_1^3\right] +\end{align} +!et + + +===== Mathematical model ===== + +We can generalize this expression to an MLP with $l$ hidden +layers. The complete functional form is, + +!bt +\begin{align} +&y^{l+1}_i = f^{l+1}\left[\!\sum_{j=1}^{N_l} w_{ij}^3 f^l\left(\sum_{k=1}^{N_{l-1}}w_{jk}^{l-1}\left(\dots f^1\left(\sum_{n=1}^{N_0} w_{mn}^1 x_n+ b_m^1\right)\dots\right)+b_k^2\right)+b_1^3\right] && + label{completeNN} +\end{align} +!et + +which illustrates a basic property of MLPs: The only independent +variables are the input values $x_n$. + + +===== Mathematical model ===== + +This confirms that an MLP, despite its quite convoluted mathematical +form, is nothing more than an analytic function, specifically a +mapping of real-valued vectors $\hat{x} \in \mathbb{R}^n \rightarrow +\hat{y} \in \mathbb{R}^m$. + +Furthermore, the flexibility and universality of an MLP can be +illustrated by realizing that the expression is essentially a nested +sum of scaled activation functions of the form + +!bt +\begin{equation} + f(x) = c_1 f(c_2 x + c_3) + c_4 +\end{equation} +!et + +where the parameters $c_i$ are weights and biases. By adjusting these +parameters, the activation functions can be shifted up and down or +left and right, change slope or be rescaled which is the key to the +flexibility of a neural network. + + +=== Matrix-vector notation === + +We can introduce a more convenient notation for the activations in an A NN. + +Additionally, we can represent the biases and activations +as layer-wise column vectors $\hat{b}_l$ and $\hat{y}_l$, so that the $i$-th element of each vector +is the bias $b_i^l$ and activation $y_i^l$ of node $i$ in layer $l$ respectively. + +We have that $\mathrm{W}_l$ is an $N_{l-1} \times N_l$ matrix, while $\hat{b}_l$ and $\hat{y}_l$ are $N_l \times 1$ column vectors. +With this notation, the sum becomes a matrix-vector multiplication, and we can write +the equation for the activations of hidden layer 2 (assuming three nodes for simplicity) as +!bt +\begin{equation} + \hat{y}_2 = f_2(\mathrm{W}_2 \hat{y}_{1} + \hat{b}_{2}) = + f_2\left(\left[\begin{array}{ccc} + w^2_{11} &w^2_{12} &w^2_{13} \\ + w^2_{21} &w^2_{22} &w^2_{23} \\ + w^2_{31} &w^2_{32} &w^2_{33} \\ + \end{array} \right] \cdot + \left[\begin{array}{c} + y^1_1 \\ + y^1_2 \\ + y^1_3 \\ + \end{array}\right] + + \left[\begin{array}{c} + b^2_1 \\ + b^2_2 \\ + b^2_3 \\ + \end{array}\right]\right). +\end{equation} +!et + + +=== Matrix-vector notation and activation === + +The activation of node $i$ in layer 2 is + +!bt +\begin{equation} + y^2_i = f_2\Bigr(w^2_{i1}y^1_1 + w^2_{i2}y^1_2 + w^2_{i3}y^1_3 + b^2_i\Bigr) = + f_2\left(\sum_{j=1}^3 w^2_{ij} y_j^1 + b^2_i\right). +\end{equation} +!et + +This is not just a convenient and compact notation, but also a useful +and intuitive way to think about MLPs: The output is calculated by a +series of matrix-vector multiplications and vector additions that are +used as input to the activation functions. For each operation +$\mathrm{W}_l \hat{y}_{l-1}$ we move forward one layer. + + + +=== Activation functions === + + +A property that characterizes a neural network, other than its +connectivity, is the choice of activation function(s). As described +in, the following restrictions are imposed on an activation function +for a FFNN to fulfill the universal approximation theorem + + * Non-constant + + * Bounded + + * Monotonically-increasing + + * Continuous + + +=== Activation functions, Logistic and Hyperbolic ones === + +The second requirement excludes all linear functions. Furthermore, in +a MLP with only linear activation functions, each layer simply +performs a linear transformation of its inputs. + +Regardless of the number of layers, the output of the NN will be +nothing but a linear function of the inputs. Thus we need to introduce +some kind of non-linearity to the NN to be able to fit non-linear +functions Typical examples are the logistic *Sigmoid* + +!bt +\[ + f(x) = \frac{1}{1 + e^{-x}}, +\] +!et +and the *hyperbolic tangent* function +!bt +\[ + f(x) = \tanh(x) +\] +!et + + +=== Relevance === + +The *sigmoid* function are more biologically plausible because the +output of inactive neurons are zero. Such activation function are +called *one-sided*. However, it has been shown that the hyperbolic +tangent performs better than the sigmoid for training MLPs. has +become the most popular for *deep neural networks* + +!bc pycod +"""The sigmoid function (or the logistic curve) is a +function that takes any real number, z, and outputs a number (0,1). +It is useful in neural networks for assigning weights on a relative scale. +The value z is the weighted sum of parameters involved in the learning algorithm.""" + +import numpy +import matplotlib.pyplot as plt +import math as mt + +z = numpy.arange(-5, 5, .1) +sigma_fn = numpy.vectorize(lambda z: 1/(1+numpy.exp(-z))) +sigma = sigma_fn(z) + +fig = plt.figure() +ax = fig.add_subplot(111) +ax.plot(z, sigma) +ax.set_ylim([-0.1, 1.1]) +ax.set_xlim([-5,5]) +ax.grid(True) +ax.set_xlabel('z') +ax.set_title('sigmoid function') + +plt.show() + +"""Step Function""" +z = numpy.arange(-5, 5, .02) +step_fn = numpy.vectorize(lambda z: 1.0 if z >= 0.0 else 0.0) +step = step_fn(z) + +fig = plt.figure() +ax = fig.add_subplot(111) +ax.plot(z, step) +ax.set_ylim([-0.5, 1.5]) +ax.set_xlim([-5,5]) +ax.grid(True) +ax.set_xlabel('z') +ax.set_title('step function') + +plt.show() + +"""Sine Function""" +z = numpy.arange(-2*mt.pi, 2*mt.pi, 0.1) +t = numpy.sin(z) + +fig = plt.figure() +ax = fig.add_subplot(111) +ax.plot(z, t) +ax.set_ylim([-1.0, 1.0]) +ax.set_xlim([-2*mt.pi,2*mt.pi]) +ax.grid(True) +ax.set_xlabel('z') +ax.set_title('sine function') + +plt.show() + +"""Plots a graph of the squashing function used by a rectified linear +unit""" +z = numpy.arange(-2, 2, .1) +zero = numpy.zeros(len(z)) +y = numpy.max([zero, z], axis=0) + +fig = plt.figure() +ax = fig.add_subplot(111) +ax.plot(z, y) +ax.set_ylim([-2.0, 2.0]) +ax.set_xlim([-2.0, 2.0]) +ax.grid(True) +ax.set_xlabel('z') +ax.set_title('Rectified linear unit') + +plt.show() +!ec + + + +===== The multilayer perceptron (MLP) ===== + +The multilayer perceptron is a very popular, and easy to implement approach, to deep learning. It consists of +o A neural network with one or more layers of nodes between the input and the output nodes. +o The multilayer network structure, or architecture, or topology, consists of an input layer, one or more hidden layers, and one output layer. +o The input nodes pass values to the first hidden layer, its nodes pass the information on to the second and so on till we reach the output layer. + +As a convention it is normal to call a network with one layer of input units, one layer of hidden +units and one layer of output units as a two-layer network. A network with two layers of hidden units is called a three-layer network etc etc. + +For an MLP network there is no direct connection between the output nodes/neurons/units and the input nodes/neurons/units. +Hereafter we will call the various entities of a layer for nodes. +There are also no connections within a single layer. + +The number of input nodes does not need to equal the number of output +nodes. This applies also to the hidden layers. Each layer may have its +own number of nodes and activation functions. + +The hidden layers have their name from the fact that they are not +linked to observables and as we will see below when we define the +so-called activation $\hat{z}$, we can think of this as a basis +expansion of the original inputs $\hat{x}$. The difference however +between neural networks and say linear regression is that now these +basis functions (which will correspond to the weights in the network) +are learned from data. This results in an important difference between +neural networks and deep learning approaches on one side and methods +like logistic regression or linear regression and their modifications on the other side. + + + +===== From one to many layers, the universal approximation theorem ===== + + +A neural network with only one layer, what we called the simple +perceptron, is best suited if we have a standard binary model with +clear (linear) boundaries between the outcomes. As such it could +equally well be replaced by standard linear regression or logistic +regression. Networks with one or more hidden layers approximate +systems with more complex boundaries. + +As stated earlier, +an important theorem in studies of neural networks, restated without +proof here, is the "universal approximation +theorem":"http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.441.7873&rep=rep1&type=pdf". + +It states that a feed-forward network with a single hidden layer +containing a finite number of neurons can approximate continuous +functions on compact subsets of real functions. The theorem thus +states that simple neural networks can represent a wide variety of +interesting functions when given appropriate parameters. It is the +multilayer feedforward architecture itself which gives neural networks +the potential of being universal approximators. + + + +===== Deriving the back propagation code for a multilayer perceptron model ===== + + +_Note: figures will be inserted later!_ + +As we have seen now in a feed forward network, we can express the final output of our network in terms of basic matrix-vector multiplications. +The unknowwn quantities are our weights $w_{ij}$ and we need to find an algorithm for changing them so that our errors are as small as possible. +This leads us to the famous "back propagation algorithm":"https://www.nature.com/articles/323533a0". + +The questions we want to ask are how do changes in the biases and the +weights in our network change the cost function and how can we use the +final output to modify the weights? + +To derive these equations let us start with a plain regression problem +and define our cost function as + +!bt +\[ +{\cal C}(\hat{W}) = \frac{1}{2}\sum_{i=1}^n\left(y_i - t_i\right)^2, +\] +!et + +where the $t_i$s are our $n$ targets (the values we want to +reproduce), while the outputs of the network after having propagated +all inputs $\hat{x}$ are given by $y_i$. Below we will demonstrate +how the basic equations arising from the back propagation algorithm +can be modified in order to study classification problems with $K$ +classes. + + +===== Definitions ===== + +With our definition of the targets $\hat{t}$, the outputs of the +network $\hat{y}$ and the inputs $\hat{x}$ we +define now the activation $z_j^l$ of node/neuron/unit $j$ of the +$l$-th layer as a function of the bias, the weights which add up from +the previous layer $l-1$ and the forward passes/outputs +$\hat{a}^{l-1}$ from the previous layer as + + +!bt +\[ +z_j^l = \sum_{i=1}^{M_{l-1}}w_{ij}^la_i^{l-1}+b_j^l, +\] +!et + +where $b_k^l$ are the biases from layer $l$. Here $M_{l-1}$ +represents the total number of nodes/neurons/units of layer $l-1$. The +figure here illustrates this equation. We can rewrite this in a more +compact form as the matrix-vector products we discussed earlier, + +!bt +\[ +\hat{z}^l = \left(\hat{W}^l\right)^T\hat{a}^{l-1}+\hat{b}^l. +\] +!et + +With the activation values $\hat{z}^l$ we can in turn define the +output of layer $l$ as $\hat{a}^l = f(\hat{z}^l)$ where $f$ is our +activation function. In the examples here we will use the sigmoid +function discussed in our logistic regression lectures. We will also use the same activation function $f$ for all layers +and their nodes. It means we have + +!bt +\[ +a_j^l = f(z_j^l) = \frac{1}{1+\exp{-(z_j^l)}}. +\] +!et + + + +===== Derivatives and the chain rule ===== + +From the definition of the activation $z_j^l$ we have +!bt +\[ +\frac{\partial z_j^l}{\partial w_{ij}^l} = a_i^{l-1}, +\] +!et +and +!bt +\[ +\frac{\partial z_j^l}{\partial a_i^{l-1}} = w_{ji}^l. +\] +!et + +With our definition of the activation function we have that (note that this function depends only on $z_j^l$) +!bt +\[ +\frac{\partial a_j^l}{\partial z_j^{l}} = a_j^l(1-a_j^l)=f(z_j^l)(1-f(z_j^l)). +\] +!et + + + +===== Derivative of the cost function ===== + +With these definitions we can now compute the derivative of the cost function in terms of the weights. + +Let us specialize to the output layer $l=L$. Our cost function is +!bt +\[ +{\cal C}(\hat{W^L}) = \frac{1}{2}\sum_{i=1}^n\left(y_i - t_i\right)^2=\frac{1}{2}\sum_{i=1}^n\left(a_i^L - t_i\right)^2, +\] +!et +The derivative of this function with respect to the weights is + +!bt +\[ +\frac{\partial{\cal C}(\hat{W^L})}{\partial w_{jk}^L} = \left(a_j^L - t_j\right)\frac{\partial a_j^L}{\partial w_{jk}^{L}}, +\] +!et +The last partial derivative can easily be computed and reads (by applying the chain rule) +!bt +\[ +\frac{\partial a_j^L}{\partial w_{jk}^{L}} = \frac{\partial a_j^L}{\partial z_{j}^{L}}\frac{\partial z_j^L}{\partial w_{jk}^{L}}=a_j^L(1-a_j^L)a_k^{L-1}, +\] +!et + + + + +===== Bringing it together, first back propagation equation ===== + +We have thus +!bt +\[ +\frac{\partial{\cal C}(\hat{W^L})}{\partial w_{jk}^L} = \left(a_j^L - t_j\right)a_j^L(1-a_j^L)a_k^{L-1}, +\] +!et + +Defining +!bt +\[ +\delta_j^L = a_j^L(1-a_j^L)\left(a_j^L - t_j\right) = f'(z_j^L)\frac{\partial {\cal C}}{\partial (a_j^L)}, +\] +!et +and using the Hadamard product of two vectors we can write this as +!bt +\[ +\hat{\delta}^L = f'(\hat{z}^L)\circ\frac{\partial {\cal C}}{\partial (\hat{a}L)}. +\] +!et + +This is an important expression. The second term on the right handside +measures how fast the cost function is changing as a function of the $j$th +output activation. If, for example, the cost function doesn't depend +much on a particular output node $j$, then $\delta_j^L$ will be small, +which is what we would expect. The first term on the right, measures +how fast the activation function $f$ is changing at a given activation +value $z_j^L$. + +Notice that everything in the above equations is easily computed. In +particular, we compute $z_j^L$ while computing the behaviour of the +network, and it is only a small additional overhead to compute +$f'(z^L_j)$. The exact form of the derivative with respect to the +output depends on the form of the cost function. +However, provided the cost function is known there should be little +trouble in calculating + +!bt +\[ +\frac{\partial {\cal C}}{\partial (a_j^L)} +\] +!et + +With the definition of $\delta_j^L$ we have a more compact definition of the derivative of the cost function in terms of the weights, namely +!bt +\[ +\frac{\partial{\cal C}(\hat{W^L})}{\partial w_{jk}^L} = \delta_j^La_k^{L-1}. +\] +!et + + +===== Derivatives in terms of $z_j^L$ ===== + +It is also easy to see that our previous equation can be written as + +!bt +\[ +\delta_j^L =\frac{\partial {\cal C}}{\partial z_j^L}= \frac{\partial {\cal C}}{\partial a_j^L}\frac{\partial a_j^L}{\partial z_j^L}, +\] +!et +which can also be interpreted as the partial derivative of the cost function with respect to the biases $b_j^L$, namely +!bt +\[ +\delta_j^L = \frac{\partial {\cal C}}{\partial b_j^L}\frac{\partial b_j^L}{\partial z_j^L}=\frac{\partial {\cal C}}{\partial b_j^L}, +\] +!et +That is, the error $\delta_j^L$ is exactly equal to the rate of change of the cost function as a function of the bias. + +===== Bringing it together ===== + +We have now three equations that are essential for the computations of the derivatives of the cost function at the output layer. These equations are needed to start the algorithm and they are + +!bblock The starting equations + +!bt +\begin{equation} +\frac{\partial{\cal C}(\hat{W^L})}{\partial w_{jk}^L} = \delta_j^La_k^{L-1}, +\end{equation} +!et +and +!bt +\begin{equation} +\delta_j^L = f'(z_j^L)\frac{\partial {\cal C}}{\partial (a_j^L)}, +\end{equation} +!et +and + +!bt +\begin{equation} +\delta_j^L = \frac{\partial {\cal C}}{\partial b_j^L}, +\end{equation} +!et +!eblock + + +An interesting consequence of the above equations is that when the +activation $a_k^{L-1}$ is small, the gradient term, that is the +derivative of the cost function with respect to the weights, will also +tend to be small. We say then that the weight learns slowly, meaning +that it changes slowly when we minimize the weights via say gradient +descent. In this case we say the system learns slowly. + +Another interesting feature is that is when the activation function, +represented by the sigmoid function here, is rather flat when we move towards +its end values $0$ and $1$ (see the above Python codes). In these +cases, the derivatives of the activation function will also be close +to zero, meaning again that the gradients will be small and the +network learns slowly again. + + + +We need a fourth equation and we are set. We are going to propagate +backwards in order to the determine the weights and biases. In order +to do so we need to represent the error in the layer before the final +one $L-1$ in terms of the errors in the final output layer. + + +===== Final back propagating equation ===== + +We have that (replacing $L$ with a general layer $l$) +!bt +\[ +\delta_j^l =\frac{\partial {\cal C}}{\partial z_j^l}. +\] +!et +We want to express this in terms of the equations for layer $l+1$. Using the chain rule and summing over all $k$ entries we have + +!bt +\[ +\delta_j^l =\sum_k \frac{\partial {\cal C}}{\partial z_k^{l+1}}\frac{\partial z_k^{l+1}}{\partial z_j^{l}}=\sum_k \delta_k^{l+1}\frac{\partial z_k^{l+1}}{\partial z_j^{l}}, +\] +!et +and recalling that +!bt +\[ +z_j^{l+1} = \sum_{i=1}^{M_{l}}w_{ij}^{l+1}a_j^{l}+b_j^{l+1}, +\] +!et +with $M_l$ being the number of nodes in layer $l$, we obtain +!bt +\[ +\delta_j^l =\sum_k \delta_k^{l+1}w_{kj}^{l+1}f'(z_j^l), +\] +!et +This is our final equation. + +We are now ready to set up the algorithm for back propagation and learning the weights and biases. + + +===== Setting up the Back propagation algorithm ===== + + + +The four equations provide us with a way of computing the gradient of the cost function. Let us write this out in the form of an algorithm. + +!bblock +First, we set up the input data $\hat{x}$ and the activations +$\hat{z}_1$ of the input layer and compute the activation function and +the pertinent outputs $\hat{a}^1$. +!eblock + +!bblock +Secondly, we perform then the feed forward till we reach the output +layer and compute all $\hat{z}_l$ of the input layer and compute the +activation function and the pertinent outputs $\hat{a}^l$ for +$l=2,3,\dots,L$. +!eblock + +!bblock +Thereafter we compute the ouput error $\hat{\delta}^L$ by computing all +!bt +\[ +\delta_j^L = f'(z_j^L)\frac{\partial {\cal C}}{\partial (a_j^L)}. +\] +!et +!eblock + +!bblock +Then we compute the back propagate error for each $l=L-1,L-2,\dots,2$ as +!bt +\[ +\delta_j^l = \sum_k \delta_k^{l+1}w_{kj}^{l+1}f'(z_j^l). +\] +!et +!eblock + +!bblock +Finally, we update the weights and the biases using gradient descent for each $l=L-1,L-2,\dots,2$ and update the weights and biases according to the rules +!bt +\[ +w_{jk}^l\leftarrow = w_{jk}^l- \eta \delta_j^la_k^{l-1}, +\] +!et + +!bt +\[ +b_j^l \leftarrow b_j^l-\eta \frac{\partial {\cal C}}{\partial b_j^l}=b_j^l-\eta \delta_j^l, +\] +!et +!eblock + +The parameter $\eta$ is the learning parameter discussed in connection with the gradient descent methods. +Here it is convenient to use stochastic gradient descent (see the examples below) with mini-batches with an outer loop that steps through multiple epochs of training. + + + +===== Setting up a Multi-layer perceptron model for classification ===== + +We are now gong to develop an example based on the MNIST data +base. This is a classification problem and we need to use our +cross-entropy function we discussed in connection with logistic +regression. The cross-entropy defines our cost function for the +classificaton problems with neural networks. + +In binary classification with two classes $(0, 1)$ we define the +logistic/sigmoid function as the probability that a particular input +is in class $0$ or $1$. This is possible because the logistic +function takes any input from the real numbers and inputs a number +between 0 and 1, and can therefore be interpreted as a probability. It +also has other nice properties, such as a derivative that is simple to +calculate. + +For an input $\boldsymbol{a}$ from the hidden layer, the probability that the input $\boldsymbol{x}$ +is in class 0 or 1 is just. We let $\theta$ represent the unknown weights and biases to be adjusted by our equations). The variable $x$ +represents our activation values $z$. We have +!bt +\[ +P(y = 0 \mid \hat{x}, \hat{\theta}) = \frac{1}{1 + \exp{(- \hat{x}})} , +\] +!et +and +!bt +\[ +P(y = 1 \mid \hat{x}, \hat{\theta}) = 1 - P(y = 0 \mid \hat{x}, \hat{\theta}) , +\] +!et + +where $y \in \{0, 1\}$ and $\hat{\theta}$ represents the weights and biases +of our network. + + + +===== Defining the cost function ===== + +Our cost function is given as (see the Logistic regression lectures) +!bt +\[ +\mathcal{C}(\hat{\theta}) = - \ln P(\mathcal{D} \mid \hat{\theta}) = - \sum_{i=1}^n +y_i \ln[P(y_i = 0)] + (1 - y_i) \ln [1 - P(y_i = 0)] = \sum_{i=1}^n \mathcal{L}_i(\hat{\theta}) . +\] +!et + +This last equality means that we can interpret our *cost* function as a sum over the *loss* function +for each point in the dataset $\mathcal{L}_i(\hat{\theta})$. +The negative sign is just so that we can think about our algorithm as minimizing a positive number, rather +than maximizing a negative number. + +In *multiclass* classification it is common to treat each integer label as a so called *one-hot* vector: + +$y = 5 \quad \rightarrow \quad \hat{y} = (0, 0, 0, 0, 0, 1, 0, 0, 0, 0) ,$ and + + +$y = 1 \quad \rightarrow \quad \hat{y} = (0, 1, 0, 0, 0, 0, 0, 0, 0, 0) ,$ + + +i.e. a binary bit string of length $C$, where $C = 10$ is the number of classes in the MNIST dataset (numbers from $0$ to $9$).. + +If $\hat{x}_i$ is the $i$-th input (image), $y_{ic}$ refers to the $c$-th component of the $i$-th +output vector $\hat{y}_i$. +The probability of $\hat{x}_i$ being in class $c$ will be given by the softmax function: + +!bt +\[ +P(y_{ic} = 1 \mid \hat{x}_i, \hat{\theta}) = \frac{\exp{((\hat{a}_i^{hidden})^T \hat{w}_c)}} +{\sum_{c'=0}^{C-1} \exp{((\hat{a}_i^{hidden})^T \hat{w}_{c'})}} , +\] +!et + +which reduces to the logistic function in the binary case. +The likelihood of this $C$-class classifier +is now given as: + +!bt +\[ +P(\mathcal{D} \mid \hat{\theta}) = \prod_{i=1}^n \prod_{c=0}^{C-1} [P(y_{ic} = 1)]^{y_{ic}} . +\] +!et +Again we take the negative log-likelihood to define our cost function: + +!bt +\[ +\mathcal{C}(\hat{\theta}) = - \log{P(\mathcal{D} \mid \hat{\theta})}. +\] +!et +See the logistic regression lectures for a full definition of the cost function. + +The back propagation equations need now only a small change, namely the definition of a new cost function. We are thus ready to use the same equations as before! + + +===== Example: binary classification problem ===== + +As an example of the above, relevant for project 2 as well, let us consider a binary class. As discussed in our logistic regression lectures, we defined a cost function in terms of the parameters $\beta$ as +!bt +\[ +\mathcal{C}(\hat{\beta}) = - \sum_{i=1}^n \left(y_i\log{p(y_i \vert x_i,\hat{\beta})}+(i-y_i)\log{1-p(y_i \vert x_i,\hat{\beta})}\right), +\] +!et +where we had defined the logistic (sigmoid) function +!bt +\[ +p(y_i =1\vert x_i,\hat{\beta})=\frac{\exp{(\beta_0+\beta_1 x_i)}}{1+\exp{(\beta_0+\beta_1 x_i)}}, +\] +!et +and +!bt +\[ +p(y_i =0\vert x_i,\hat{\beta})=1-p(y_i =1\vert x_i,\hat{\beta}). +\] +!et +The parameters $\hat{\beta}$ were defined using a minimization method like gradient descent or Newton-Raphson's method. + +Now we replace $x_i$ with the activation $z_i^l$ for a given layer $l$ and the outputs as $y_i=a_i^l=f(z_i^l)$, with $z_i^l$ now being a function of the weights $w_{ij}^l$ and biases $b_i^l$. +We have then +!bt +\[ +a_i^l = y_i = \frac{\exp{(z_i^l)}}{1+\exp{(z_i^l)}}, +\] +!et +with +!bt +\[ +z_i^l = \sum_{j}w_{ij}^l a_j^{l-1}+b_i^l, +\] +!et +where the superscript $l-1$ indicates that these are the outputs from layer $l-1$. +Our cost function at the final layer $l=L$ is now +!bt +\[ +\mathcal{C}(\hat{W}) = - \sum_{i=1}^n \left(t_i\log{a_i^L}+(i-t_i)\log{(1-a_i^L)}\right), +\] +!et +where we have defined the targets $t_i$. The derivatives of the cost function with respect to the output $a_i^L$ are then easily calculated and we get +!bt +\[ +\frac{\partial \mathcal{C}(\hat{W})}{\partial a_i^L} = \frac{a_i^L-t_i}{a_i^L(1-a_i^L)}. +\] +!et +In case we use another activation function than the logistic one, we need to evaluate other derivatives. + + + +===== The Softmax function ===== +In case we employ the more general case given by the Softmax equation, we need to evaluate the derivative of the activation function with respect to the activation $z_i^l$, that is we need +!bt +\[ +\frac{\partial f(z_i^l)}{\partial w_{jk}^l} = +\frac{\partial f(z_i^l)}{\partial z_j^l} \frac{\partial z_j^l}{\partial w_{jk}^l}= \frac{\partial f(z_i^l)}{\partial z_j^l}a_k^{l-1}. +\] +!et +For the Softmax function we have +!bt +\[ +f(z_i^l) = \frac{\exp{(z_i^l)}}{\sum_{m=1}^K\exp{(z_m^l)}}. +\] +!et +Its derivative with respect to $z_j^l$ gives +!bt +\[ +\frac{\partial f(z_i^l)}{\partial z_j^l}= f(z_i^l)\left(\delta_{ij}-f(z_j^l)\right), +\] +!et +which in case of the simply binary model reduces to having $i=j$. + + +===== Developing a code for doing neural networks with back propagation ===== + + +One can identify a set of key steps when using neural networks to solve supervised learning problems: + +o Collect and pre-process data +o Define model and architecture +o Choose cost function and optimizer +o Train the model +o Evaluate model performance on test data +o Adjust hyperparameters (if necessary, network architecture) + + +===== Collect and pre-process data ===== + +Here we will be using the MNIST dataset, which is readily available through the _scikit-learn_ +package. You may also find it for example "here":"http://yann.lecun.com/exdb/mnist/". +The *MNIST* (Modified National Institute of Standards and Technology) database is a large database +of handwritten digits that is commonly used for training various image processing systems. +The MNIST dataset consists of 70 000 images of size 28x28 pixels, each labeled from 0 to 9. +The scikit-learn dataset we will use consists of a selection of 1797 images of size $8\times 8$ collected and processed from this database. + +To feed data into a feed-forward neural network we need to represent +the inputs as a feature matrix $X = (n_{inputs}, n_{features})$. Each +row represents an *input*, in this case a handwritten digit, and +each column represents a *feature*, in this case a pixel. The +correct answers, also known as *labels* or *targets* are +represented as a 1D array of integers +$Y = (n_{inputs}) = (5, 3, 1, 8,...)$. + +As an example, say we want to build a neural network using supervised learning to predict Body-Mass Index (BMI) from +measurements of height (in m) +and weight (in kg). If we have measurements of 5 people the feature matrix could be for example: + +$$ X = \begin{bmatrix} +1.85 & 81\\ +1.71 & 65\\ +1.95 & 103\\ +1.55 & 42\\ +1.63 & 56 +\end{bmatrix} ,$$ + +and the targets would be: + +$$ Y = (23.7, 22.2, 27.1, 17.5, 21.1) $$ + +Since each input image is a 2D matrix, we need to flatten the image +(i.e. "unravel" the 2D matrix into a 1D array) to turn the data into a +feature matrix. This means we lose all spatial information in the +image, such as locality and translational invariance. More complicated +architectures such as Convolutional Neural Networks can take advantage +of such information, and are most commonly applied when analyzing +images. + + +!bc pycod +# import necessary packages +import numpy as np +import matplotlib.pyplot as plt +from sklearn import datasets + + +# ensure the same random numbers appear every time +np.random.seed(0) + +# display images in notebook +%matplotlib inline +plt.rcParams['figure.figsize'] = (12,12) + + +# download MNIST dataset +digits = datasets.load_digits() + +# define inputs and labels +inputs = digits.images +labels = digits.target + +print("inputs = (n_inputs, pixel_width, pixel_height) = " + str(inputs.shape)) +print("labels = (n_inputs) = " + str(labels.shape)) + + +# flatten the image +# the value -1 means dimension is inferred from the remaining dimensions: 8x8 = 64 +n_inputs = len(inputs) +inputs = inputs.reshape(n_inputs, -1) +print("X = (n_inputs, n_features) = " + str(inputs.shape)) + + +# choose some random images to display +indices = np.arange(n_inputs) +random_indices = np.random.choice(indices, size=5) + +for i, image in enumerate(digits.images[random_indices]): + plt.subplot(1, 5, i+1) + plt.axis('off') + plt.imshow(image, cmap=plt.cm.gray_r, interpolation='nearest') + plt.title("Label: %d" % digits.target[random_indices[i]]) +plt.show() +!ec + + +===== Train and test datasets ===== + +Performing analysis before partitioning the dataset is a major error, that can lead to incorrect conclusions. + +We will reserve $80 \%$ of our dataset for training and $20 \%$ for testing. + +It is important that the train and test datasets are drawn randomly from our dataset, to ensure +no bias in the sampling. +Say you are taking measurements of weather data to predict the weather in the coming 5 days. +You don't want to train your model on measurements taken from the hours 00.00 to 12.00, and then test it on data +collected from 12.00 to 24.00. + + +!bc pycod +from sklearn.model_selection import train_test_split + +# one-liner from scikit-learn library +train_size = 0.8 +test_size = 1 - train_size +X_train, X_test, Y_train, Y_test = train_test_split(inputs, labels, train_size=train_size, + test_size=test_size) + +# equivalently in numpy +def train_test_split_numpy(inputs, labels, train_size, test_size): + n_inputs = len(inputs) + inputs_shuffled = inputs.copy() + labels_shuffled = labels.copy() + + np.random.shuffle(inputs_shuffled) + np.random.shuffle(labels_shuffled) + + train_end = int(n_inputs*train_size) + X_train, X_test = inputs_shuffled[:train_end], inputs_shuffled[train_end:] + Y_train, Y_test = labels_shuffled[:train_end], labels_shuffled[train_end:] + + return X_train, X_test, Y_train, Y_test + +#X_train, X_test, Y_train, Y_test = train_test_split_numpy(inputs, labels, train_size, test_size) + +print("Number of training images: " + str(len(X_train))) +print("Number of test images: " + str(len(X_test))) +!ec + + +===== Define model and architecture ===== + +Our simple feed-forward neural network will consist of an *input* layer, a single *hidden* layer and an *output* layer. The activation $y$ of each neuron is a weighted sum of inputs, passed through an activation function. In case of the simple perceptron model we have + +$$ z = \sum_{i=1}^n w_i a_i ,$$ + +$$ y = f(z) ,$$ + +where $f$ is the activation function, $a_i$ represents input from neuron $i$ in the preceding layer +and $w_i$ is the weight to input $i$. +The activation of the neurons in the input layer is just the features (e.g. a pixel value). + +The simplest activation function for a neuron is the *Heaviside* function: + +$$ f(z) = +\begin{cases} +1, & z > 0\\ +0, & \text{otherwise} +\end{cases} +$$ + +A feed-forward neural network with this activation is known as a *perceptron*. +For a binary classifier (i.e. two classes, 0 or 1, dog or not-dog) we can also use this in our output layer. +This activation can be generalized to $k$ classes (using e.g. the *one-against-all* strategy), +and we call these architectures *multiclass perceptrons*. + +However, it is now common to use the terms Single Layer Perceptron (SLP) (1 hidden layer) and +Multilayer Perceptron (MLP) (2 or more hidden layers) to refer to feed-forward neural networks with any activation function. + +Typical choices for activation functions include the sigmoid function, hyperbolic tangent, and Rectified Linear Unit (ReLU). +We will be using the sigmoid function $\sigma(x)$: + +$$ f(x) = \sigma(x) = \frac{1}{1 + e^{-x}} ,$$ + +which is inspired by probability theory (see logistic regression) and was most commonly used until about 2011. See the discussion below concerning other activation functions. + + +===== Layers ===== + +* Input +Since each input image has 8x8 = 64 pixels or features, we have an input layer of 64 neurons. + +* Hidden layer +We will use 50 neurons in the hidden layer receiving input from the neurons in the input layer. +Since each neuron in the hidden layer is connected to the 64 inputs we have 64x50 = 3200 weights to the hidden layer. + +* Output +If we were building a binary classifier, it would be sufficient with a single neuron in the output layer, +which could output 0 or 1 according to the Heaviside function. This would be an example of a *hard* classifier, meaning it outputs the class of the input directly. However, if we are dealing with noisy data it is often beneficial to use a *soft* classifier, which outputs the probability of being in class 0 or 1. + +For a soft binary classifier, we could use a single neuron and interpret the output as either being the probability of being in class 0 or the probability of being in class 1. Alternatively we could use 2 neurons, and interpret each neuron as the probability of being in each class. + +Since we are doing multiclass classification, with 10 categories, it is natural to use 10 neurons in the output layer. We number the neurons $j = 0,1,...,9$. The activation of each output neuron $j$ will be according to the *softmax* function: + +$$ P(\text{class $j$} \mid \text{input $\hat{a}$}) = \frac{\exp{(\hat{a}^T \hat{w}_j)}} +{\sum_{c=0}^{9} \exp{(\hat{a}^T \hat{w}_c)}} ,$$ + +i.e. each neuron $j$ outputs the probability of being in class $j$ given an input from the hidden layer $\hat{a}$, with $\hat{w}_j$ the weights of neuron $j$ to the inputs. +The denominator is a normalization factor to ensure the outputs (probabilities) sum up to 1. +The exponent is just the weighted sum of inputs as before: + +$$ z_j = \sum_{i=1}^n w_ {ij} a_i+b_j.$$ + +Since each neuron in the output layer is connected to the 50 inputs from the hidden layer we have 50x10 = 500 +weights to the output layer. + + +===== Weights and biases ===== + +Typically weights are initialized with small values distributed around zero, drawn from a uniform +or normal distribution. Setting all weights to zero means all neurons give the same output, making the network useless. + +Adding a bias value to the weighted sum of inputs allows the neural network to represent a greater range +of values. Without it, any input with the value 0 will be mapped to zero (before being passed through the activation). The bias unit has an output of 1, and a weight to each neuron $j$, $b_j$: + +$$ z_j = \sum_{i=1}^n w_ {ij} a_i + b_j.$$ + +The bias weights $\hat{b}$ are often initialized to zero, but a small value like $0.01$ ensures all neurons have some output which can be backpropagated in the first training cycle. +!bc pycod +# building our neural network + +n_inputs, n_features = X_train.shape +n_hidden_neurons = 50 +n_categories = 10 + +# we make the weights normally distributed using numpy.random.randn + +# weights and bias in the hidden layer +hidden_weights = np.random.randn(n_features, n_hidden_neurons) +hidden_bias = np.zeros(n_hidden_neurons) + 0.01 + +# weights and bias in the output layer +output_weights = np.random.randn(n_hidden_neurons, n_categories) +output_bias = np.zeros(n_categories) + 0.01 +!ec + + +===== Feed-forward pass ===== + +Denote $F$ the number of features, $H$ the number of hidden neurons and $C$ the number of categories. +For each input image we calculate a weighted sum of input features (pixel values) to each neuron $j$ in the hidden layer $l$: + +$$ z_{j}^{l} = \sum_{i=1}^{F} w_{ij}^{l} x_i + b_{j}^{l},$$ + +this is then passed through our activation function + +$$ a_{j}^{l} = f(z_{j}^{l}) .$$ + +We calculate a weighted sum of inputs (activations in the hidden layer) to each neuron $j$ in the output layer: + +$$ z_{j}^{L} = \sum_{i=1}^{H} w_{ij}^{L} a_{i}^{l} + b_{j}^{L}.$$ + +Finally we calculate the output of neuron $j$ in the output layer using the softmax function: + +$$ a_{j}^{L} = \frac{\exp{(z_j^{L})}} +{\sum_{c=0}^{C-1} \exp{(z_c^{L})}} .$$ + + +===== Matrix multiplications ===== + +Since our data has the dimensions $X = (n_{inputs}, n_{features})$ and our weights to the hidden +layer have the dimensions +$W_{hidden} = (n_{features}, n_{hidden})$, +we can easily feed the network all our training data in one go by taking the matrix product + +$$ X W^{h} = (n_{inputs}, n_{hidden}),$$ + +and obtain a matrix that holds the weighted sum of inputs to the hidden layer +for each input image and each hidden neuron. +We also add the bias to obtain a matrix of weighted sums to the hidden layer $Z^{h}$: + +$$ \hat{z}^{l} = \hat{X} \hat{W}^{l} + \hat{b}^{l} ,$$ + +meaning the same bias (1D array with size equal number of hidden neurons) is added to each input image. +This is then passed through the activation: + +$$ \hat{a}^{l} = f(\hat{z}^l) .$$ + +This is fed to the output layer: + +$$ \hat{z}^{L} = \hat{a}^{L} \hat{W}^{L} + \hat{b}^{L} .$$ + +Finally we receive our output values for each image and each category by passing it through the softmax function: + +$$ output = softmax (\hat{z}^{L}) = (n_{inputs}, n_{categories}) .$$ + + +!bc pycod +# setup the feed-forward pass, subscript h = hidden layer + +def sigmoid(x): + return 1/(1 + np.exp(-x)) + +def feed_forward(X): + # weighted sum of inputs to the hidden layer + z_h = np.matmul(X, hidden_weights) + hidden_bias + # activation in the hidden layer + a_h = sigmoid(z_h) + + # weighted sum of inputs to the output layer + z_o = np.matmul(a_h, output_weights) + output_bias + # softmax output + # axis 0 holds each input and axis 1 the probabilities of each category + exp_term = np.exp(z_o) + probabilities = exp_term / np.sum(exp_term, axis=1, keepdims=True) + + return probabilities + +probabilities = feed_forward(X_train) +print("probabilities = (n_inputs, n_categories) = " + str(probabilities.shape)) +print("probability that image 0 is in category 0,1,2,...,9 = \n" + str(probabilities[0])) +print("probabilities sum up to: " + str(probabilities[0].sum())) +print() + +# we obtain a prediction by taking the class with the highest likelihood +def predict(X): + probabilities = feed_forward(X) + return np.argmax(probabilities, axis=1) + +predictions = predict(X_train) +print("predictions = (n_inputs) = " + str(predictions.shape)) +print("prediction for image 0: " + str(predictions[0])) +print("correct label for image 0: " + str(Y_train[0])) +!ec + + +===== Choose cost function and optimizer ===== + +To measure how well our neural network is doing we need to introduce a cost function. +We will call the function that gives the error of a single sample output the *loss* function, and the function +that gives the total error of our network across all samples the *cost* function. +A typical choice for multiclass classification is the *cross-entropy* loss, also known as the negative log likelihood. + +In *multiclass* classification it is common to treat each integer label as a so called *one-hot* vector: + +$$ y = 5 \quad \rightarrow \quad \hat{y} = (0, 0, 0, 0, 0, 1, 0, 0, 0, 0) ,$$ + + +$$ y = 1 \quad \rightarrow \quad \hat{y} = (0, 1, 0, 0, 0, 0, 0, 0, 0, 0) ,$$ + + +i.e. a binary bit string of length $C$, where $C = 10$ is the number of classes in the MNIST dataset. + +Let $y_{ic}$ denote the $c$-th component of the $i$-th one-hot vector. +We define the cost function $\mathcal{C}$ as a sum over the cross-entropy loss for each point $\hat{x}_i$ in the dataset. + +In the one-hot representation only one of the terms in the loss function is non-zero, namely the +probability of the correct category $c'$ +(i.e. the category $c'$ such that $y_{ic'} = 1$). This means that the cross entropy loss only punishes you for how wrong +you got the correct label. The probability of category $c$ is given by the softmax function. The vector $\hat{\theta}$ represents the parameters of our network, i.e. all the weights and biases. + + + +===== Optimizing the cost function ===== + +The network is trained by finding the weights and biases that minimize the cost function. One of the most widely used classes of methods is *gradient descent* and its generalizations. The idea behind gradient descent +is simply to adjust the weights in the direction where the gradient of the cost function is large and negative. This ensures we flow toward a *local* minimum of the cost function. +Each parameter $\theta$ is iteratively adjusted according to the rule + +$$ \theta_{i+1} = \theta_i - \eta \nabla \mathcal{C}(\theta_i) ,$$ + +where $\eta$ is known as the *learning rate*, which controls how big a step we take towards the minimum. +This update can be repeated for any number of iterations, or until we are satisfied with the result. + +A simple and effective improvement is a variant called *Batch Gradient Descent*. +Instead of calculating the gradient on the whole dataset, we calculate an approximation of the gradient +on a subset of the data called a *minibatch*. +If there are $N$ data points and we have a minibatch size of $M$, the total number of batches +is $N/M$. +We denote each minibatch $B_k$, with $k = 1, 2,...,N/M$. The gradient then becomes: + +$$ \nabla \mathcal{C}(\theta) = \frac{1}{N} \sum_{i=1}^N \nabla \mathcal{L}_i(\theta) \quad \rightarrow \quad +\frac{1}{M} \sum_{i \in B_k} \nabla \mathcal{L}_i(\theta) ,$$ + +i.e. instead of averaging the loss over the entire dataset, we average over a minibatch. + +This has two important benefits: +o Introducing stochasticity decreases the chance that the algorithm becomes stuck in a local minima. +o It significantly speeds up the calculation, since we do not have to use the entire dataset to calculate the gradient. + +The various optmization methods, with codes and algorithms, are discussed in our lectures on "Gradient descent approaches":"https://compphysics.github.io/MachineLearning/doc/pub/Splines/html/Splines-bs.html". + + +===== Regularization ===== + +It is common to add an extra term to the cost function, proportional +to the size of the weights. This is equivalent to constraining the +size of the weights, so that they do not grow out of control. +Constraining the size of the weights means that the weights cannot +grow arbitrarily large to fit the training data, and in this way +reduces *overfitting*. + +We will measure the size of the weights using the so called *L2-norm*, meaning our cost function becomes: + +$$ \nabla \mathcal{C}(\theta) = \frac{1}{N} \sum_{i=1}^N \nabla \mathcal{L}_i(\theta) \quad \rightarrow \quad +\frac{1}{N} \sum_{i=1}^N \nabla \mathcal{L}_i(\theta) + \lambda \lvert \lvert \hat{w} \rvert \rvert_2^2 += \frac{1}{N} \sum_{i=1}^N \nabla \mathcal{L}(\theta) + \lambda \sum_{ij} w_{ij}^2,$$ + +i.e. we sum up all the weights squared. The factor $\lambda$ is known as a regularization parameter. + + +In order to train the model, we need to calculate the derivative of +the cost function with respect to every bias and weight in the +network. In total our network has $(64 + 1)\times 50=3250$ weights in +the hidden layer and $(50 + 1)\times 10=510$ weights to the output +layer ($+1$ for the bias), and the gradient must be calculated for +every parameter. We use the *backpropagation* algorithm discussed +above. This is a clever use of the chain rule that allows us to +calculate the gradient efficently. + + + +===== Matrix multiplication ===== + +To more efficently train our network these equations are implemented using matrix operations. +The error in the output layer is calculated simply as, with $\hat{t}$ being our targets, + +$$ \delta_L = \hat{t} - \hat{y} = (n_{inputs}, n_{categories}) .$$ + +The gradient for the output weights is calculated as + +$$ \nabla W_{L} = \hat{a}^T \delta_L = (n_{hidden}, n_{categories}) ,$$ + +where $\hat{a} = (n_{inputs}, n_{hidden})$. This simply means that we are summing up the gradients for each input. +Since we are going backwards we have to transpose the activation matrix. + +The gradient with respect to the output bias is then + +$$ \nabla \hat{b}_{L} = \sum_{i=1}^{n_{inputs}} \delta_L = (n_{categories}) .$$ + +The error in the hidden layer is + +$$ \Delta_h = \delta_L W_{L}^T \circ f'(z_{h}) = \delta_L W_{L}^T \circ a_{h} \circ (1 - a_{h}) = (n_{inputs}, n_{hidden}) ,$$ + +where $f'(a_{h})$ is the derivative of the activation in the hidden layer. The matrix products mean +that we are summing up the products for each neuron in the output layer. The symbol $\circ$ denotes +the *Hadamard product*, meaning element-wise multiplication. + +This again gives us the gradients in the hidden layer: + +$$ \nabla W_{h} = X^T \delta_h = (n_{features}, n_{hidden}) ,$$ + +$$ \nabla b_{h} = \sum_{i=1}^{n_{inputs}} \delta_h = (n_{hidden}) .$$ + + +!bc pycod +# to categorical turns our integer vector into a onehot representation +from sklearn.metrics import accuracy_score + +# one-hot in numpy +def to_categorical_numpy(integer_vector): + n_inputs = len(integer_vector) + n_categories = np.max(integer_vector) + 1 + onehot_vector = np.zeros((n_inputs, n_categories)) + onehot_vector[range(n_inputs), integer_vector] = 1 + + return onehot_vector + +#Y_train_onehot, Y_test_onehot = to_categorical(Y_train), to_categorical(Y_test) +Y_train_onehot, Y_test_onehot = to_categorical_numpy(Y_train), to_categorical_numpy(Y_test) + +def feed_forward_train(X): + # weighted sum of inputs to the hidden layer + z_h = np.matmul(X, hidden_weights) + hidden_bias + # activation in the hidden layer + a_h = sigmoid(z_h) + + # weighted sum of inputs to the output layer + z_o = np.matmul(a_h, output_weights) + output_bias + # softmax output + # axis 0 holds each input and axis 1 the probabilities of each category + exp_term = np.exp(z_o) + probabilities = exp_term / np.sum(exp_term, axis=1, keepdims=True) + + # for backpropagation need activations in hidden and output layers + return a_h, probabilities + +def backpropagation(X, Y): + a_h, probabilities = feed_forward_train(X) + + # error in the output layer + error_output = probabilities - Y + # error in the hidden layer + error_hidden = np.matmul(error_output, output_weights.T) * a_h * (1 - a_h) + + # gradients for the output layer + output_weights_gradient = np.matmul(a_h.T, error_output) + output_bias_gradient = np.sum(error_output, axis=0) + + # gradient for the hidden layer + hidden_weights_gradient = np.matmul(X.T, error_hidden) + hidden_bias_gradient = np.sum(error_hidden, axis=0) + + return output_weights_gradient, output_bias_gradient, hidden_weights_gradient, hidden_bias_gradient + +print("Old accuracy on training data: " + str(accuracy_score(predict(X_train), Y_train))) + +eta = 0.01 +lmbd = 0.01 +for i in range(1000): + # calculate gradients + dWo, dBo, dWh, dBh = backpropagation(X_train, Y_train_onehot) + + # regularization term gradients + dWo += lmbd * output_weights + dWh += lmbd * hidden_weights + + # update weights and biases + output_weights -= eta * dWo + output_bias -= eta * dBo + hidden_weights -= eta * dWh + hidden_bias -= eta * dBh + +print("New accuracy on training data: " + str(accuracy_score(predict(X_train), Y_train))) +!ec + + +===== Improving performance ===== + +As we can see the network does not seem to be learning at all. It seems to be just guessing the label for each image. +In order to obtain a network that does something useful, we will have to do a bit more work. + +The choice of *hyperparameters* such as learning rate and regularization parameter is hugely influential for the performance of the network. Typically a *grid-search* is performed, wherein we test different hyperparameters separated by orders of magnitude. For example we could test the learning rates $\eta = 10^{-6}, 10^{-5},...,10^{-1}$ with different regularization parameters $\lambda = 10^{-6},...,10^{-0}$. + +Next, we haven't implemented minibatching yet, which introduces stochasticity and is though to act as an important regularizer on the weights. We call a feed-forward + backward pass with a minibatch an *iteration*, and a full training period +going through the entire dataset ($n/M$ batches) an *epoch*. + +If this does not improve network performance, you may want to consider altering the network architecture, adding more neurons or hidden layers. +Andrew Ng goes through some of these considerations in this "video":"https://youtu.be/F1ka6a13S9I". You can find a summary of the video "here":"https://kevinzakka.github.io/2016/09/26/applying-deep-learning/". + + +===== Full object-oriented implementation ===== + +It is very natural to think of the network as an object, with specific instances of the network +being realizations of this object with different hyperparameters. An implementation using Python classes provides a clean structure and interface, and the full implementation of our neural network is given below. + + +!bc pycod +class NeuralNetwork: + def __init__( + self, + X_data, + Y_data, + n_hidden_neurons=50, + n_categories=10, + epochs=10, + batch_size=100, + eta=0.1, + lmbd=0.0, + + ): + self.X_data_full = X_data + self.Y_data_full = Y_data + + self.n_inputs = X_data.shape[0] + self.n_features = X_data.shape[1] + self.n_hidden_neurons = n_hidden_neurons + self.n_categories = n_categories + + self.epochs = epochs + self.batch_size = batch_size + self.iterations = self.n_inputs // self.batch_size + self.eta = eta + self.lmbd = lmbd + + self.create_biases_and_weights() + + def create_biases_and_weights(self): + self.hidden_weights = np.random.randn(self.n_features, self.n_hidden_neurons) + self.hidden_bias = np.zeros(self.n_hidden_neurons) + 0.01 + + self.output_weights = np.random.randn(self.n_hidden_neurons, self.n_categories) + self.output_bias = np.zeros(self.n_categories) + 0.01 + + def feed_forward(self): + # feed-forward for training + self.z_h = np.matmul(self.X_data, self.hidden_weights) + self.hidden_bias + self.a_h = sigmoid(self.z_h) + + self.z_o = np.matmul(self.a_h, self.output_weights) + self.output_bias + + exp_term = np.exp(self.z_o) + self.probabilities = exp_term / np.sum(exp_term, axis=1, keepdims=True) + + def feed_forward_out(self, X): + # feed-forward for output + z_h = np.matmul(X, self.hidden_weights) + self.hidden_bias + a_h = sigmoid(z_h) + + z_o = np.matmul(a_h, self.output_weights) + self.output_bias + + exp_term = np.exp(z_o) + probabilities = exp_term / np.sum(exp_term, axis=1, keepdims=True) + return probabilities + + def backpropagation(self): + error_output = self.probabilities - self.Y_data + error_hidden = np.matmul(error_output, self.output_weights.T) * self.a_h * (1 - self.a_h) + + self.output_weights_gradient = np.matmul(self.a_h.T, error_output) + self.output_bias_gradient = np.sum(error_output, axis=0) + + self.hidden_weights_gradient = np.matmul(self.X_data.T, error_hidden) + self.hidden_bias_gradient = np.sum(error_hidden, axis=0) + + if self.lmbd > 0.0: + self.output_weights_gradient += self.lmbd * self.output_weights + self.hidden_weights_gradient += self.lmbd * self.hidden_weights + + self.output_weights -= self.eta * self.output_weights_gradient + self.output_bias -= self.eta * self.output_bias_gradient + self.hidden_weights -= self.eta * self.hidden_weights_gradient + self.hidden_bias -= self.eta * self.hidden_bias_gradient + + def predict(self, X): + probabilities = self.feed_forward_out(X) + return np.argmax(probabilities, axis=1) + + def predict_probabilities(self, X): + probabilities = self.feed_forward_out(X) + return probabilities + + def train(self): + data_indices = np.arange(self.n_inputs) + + for i in range(self.epochs): + for j in range(self.iterations): + # pick datapoints with replacement + chosen_datapoints = np.random.choice( + data_indices, size=self.batch_size, replace=False + ) + + # minibatch training data + self.X_data = self.X_data_full[chosen_datapoints] + self.Y_data = self.Y_data_full[chosen_datapoints] + + self.feed_forward() + self.backpropagation() +!ec + + +===== Evaluate model performance on test data ===== + +To measure the performance of our network we evaluate how well it does it data it has never seen before, i.e. the test data. +We measure the performance of the network using the *accuracy* score. +The accuracy is as you would expect just the number of images correctly labeled divided by the total number of images. A perfect classifier will have an accuracy score of $1$. + +$$ \text{Accuracy} = \frac{\sum_{i=1}^n I(\hat{y}_i = y_i)}{n} ,$$ + +where $I$ is the indicator function, $1$ if $\hat{y}_i = y_i$ and $0$ otherwise. + + +!bc pycod +epochs = 100 +batch_size = 100 + +dnn = NeuralNetwork(X_train, Y_train_onehot, eta=eta, lmbd=lmbd, epochs=epochs, batch_size=batch_size, + n_hidden_neurons=n_hidden_neurons, n_categories=n_categories) +dnn.train() +test_predict = dnn.predict(X_test) + +# accuracy score from scikit library +print("Accuracy score on test set: ", accuracy_score(Y_test, test_predict)) + +# equivalent in numpy +def accuracy_score_numpy(Y_test, Y_pred): + return np.sum(Y_test == Y_pred) / len(Y_test) + +#print("Accuracy score on test set: ", accuracy_score_numpy(Y_test, test_predict)) +!ec + + +===== Adjust hyperparameters ===== + +We now perform a grid search to find the optimal hyperparameters for the network. +Note that we are only using 1 layer with 50 neurons, and human performance is estimated to be around $98\%$ ($2\%$ error rate). + +!bc pycod +eta_vals = np.logspace(-5, 1, 7) +lmbd_vals = np.logspace(-5, 1, 7) +# store the models for later use +DNN_numpy = np.zeros((len(eta_vals), len(lmbd_vals)), dtype=object) + +# grid search +for i, eta in enumerate(eta_vals): + for j, lmbd in enumerate(lmbd_vals): + dnn = NeuralNetwork(X_train, Y_train_onehot, eta=eta, lmbd=lmbd, epochs=epochs, batch_size=batch_size, + n_hidden_neurons=n_hidden_neurons, n_categories=n_categories) + dnn.train() + + DNN_numpy[i][j] = dnn + + test_predict = dnn.predict(X_test) + + print("Learning rate = ", eta) + print("Lambda = ", lmbd) + print("Accuracy score on test set: ", accuracy_score(Y_test, test_predict)) + print() +!ec + + +===== Visualization ===== + +!bc pycod +# visual representation of grid search +# uses seaborn heatmap, you can also do this with matplotlib imshow +import seaborn as sns + +sns.set() + +train_accuracy = np.zeros((len(eta_vals), len(lmbd_vals))) +test_accuracy = np.zeros((len(eta_vals), len(lmbd_vals))) + +for i in range(len(eta_vals)): + for j in range(len(lmbd_vals)): + dnn = DNN_numpy[i][j] + + train_pred = dnn.predict(X_train) + test_pred = dnn.predict(X_test) + + train_accuracy[i][j] = accuracy_score(Y_train, train_pred) + test_accuracy[i][j] = accuracy_score(Y_test, test_pred) + + +fig, ax = plt.subplots(figsize = (10, 10)) +sns.heatmap(train_accuracy, annot=True, ax=ax, cmap="viridis") +ax.set_title("Training Accuracy") +ax.set_ylabel("$\eta$") +ax.set_xlabel("$\lambda$") +plt.show() + +fig, ax = plt.subplots(figsize = (10, 10)) +sns.heatmap(test_accuracy, annot=True, ax=ax, cmap="viridis") +ax.set_title("Test Accuracy") +ax.set_ylabel("$\eta$") +ax.set_xlabel("$\lambda$") +plt.show() +!ec + + +===== scikit-learn implementation ===== + +_scikit-learn_ focuses more +on traditional machine learning methods, such as regression, +clustering, decision trees, etc. As such, it has only two types of +neural networks: Multi Layer Perceptron outputting continuous values, +*MPLRegressor*, and Multi Layer Perceptron outputting labels, +*MLPClassifier*. We will see how simple it is to use these classes. + +_scikit-learn_ implements a few improvements from our neural network, +such as early stopping, a varying learning rate, different +optimization methods, etc. We would therefore expect a better +performance overall. + +!bc pycod +from sklearn.neural_network import MLPClassifier +# store models for later use +DNN_scikit = np.zeros((len(eta_vals), len(lmbd_vals)), dtype=object) + +for i, eta in enumerate(eta_vals): + for j, lmbd in enumerate(lmbd_vals): + dnn = MLPClassifier(hidden_layer_sizes=(n_hidden_neurons), activation='logistic', + alpha=lmbd, learning_rate_init=eta, max_iter=epochs) + dnn.fit(X_train, Y_train) + + DNN_scikit[i][j] = dnn + + print("Learning rate = ", eta) + print("Lambda = ", lmbd) + print("Accuracy score on test set: ", dnn.score(X_test, Y_test)) + print() +!ec + + + +===== Visualization ===== +!bc pycod +# optional +# visual representation of grid search +# uses seaborn heatmap, could probably do this in matplotlib +import seaborn as sns + +sns.set() + +train_accuracy = np.zeros((len(eta_vals), len(lmbd_vals))) +test_accuracy = np.zeros((len(eta_vals), len(lmbd_vals))) + +for i in range(len(eta_vals)): + for j in range(len(lmbd_vals)): + dnn = DNN_scikit[i][j] + + train_pred = dnn.predict(X_train) + test_pred = dnn.predict(X_test) + + train_accuracy[i][j] = accuracy_score(Y_train, train_pred) + test_accuracy[i][j] = accuracy_score(Y_test, test_pred) + + +fig, ax = plt.subplots(figsize = (10, 10)) +sns.heatmap(train_accuracy, annot=True, ax=ax, cmap="viridis") +ax.set_title("Training Accuracy") +ax.set_ylabel("$\eta$") +ax.set_xlabel("$\lambda$") +plt.show() + +fig, ax = plt.subplots(figsize = (10, 10)) +sns.heatmap(test_accuracy, annot=True, ax=ax, cmap="viridis") +ax.set_title("Test Accuracy") +ax.set_ylabel("$\eta$") +ax.set_xlabel("$\lambda$") +plt.show() +!ec + + + +===== Building neural networks in Tensorflow and Keras ===== + +Now we want to build on the experience gained from our neural network implementation in NumPy and scikit-learn +and use it to construct a neural network in Tensorflow. Once we have constructed a neural network in NumPy +and Tensorflow, building one in Keras is really quite trivial, though the performance may suffer. + +In our previous example we used only one hidden layer, and in this we will use two. From this it should be quite +clear how to build one using an arbitrary number of hidden layers, using data structures such as Python lists or +NumPy arrays. + + +===== Tensorflow ===== + +Tensorflow is an open source library machine learning library +developed by the Google Brain team for internal use. It was released +under the Apache 2.0 open source license in November 9, 2015. + +Tensorflow is a computational framework that allows you to construct +machine learning models at different levels of abstraction, from +high-level, object-oriented APIs like Keras, down to the C++ kernels +that Tensorflow is built upon. The higher levels of abstraction are +simpler to use, but less flexible, and our choice of implementation +should reflect the problems we are trying to solve. + +"Tensorflow uses":"https://www.tensorflow.org/guide/graphs" so-called graphs to represent your computation +in terms of the dependencies between individual operations, such that you first build a Tensorflow *graph* +to represent your model, and then create a Tensorflow *session* to run the graph. + +In this guide we will analyze the same data as we did in our NumPy and +scikit-learn tutorial, gathered from the MNIST database of images. We +will give an introduction to the lower level Python Application +Program Interfaces (APIs), and see how we use them to build our graph. +Then we will build (effectively) the same graph in Keras, to see just +how simple solving a machine learning problem can be. + +To install tensorflow on Unix/Linux systems, use pip as +!bc pycod +pip3 install tensorflow +!ec +and/or if you use _anaconda_, just write (or install from the graphical user interface) +!bc pycod +conda install tensorflow +!ec + + +===== Collect and pre-process data ===== + +!bc pycod +# import necessary packages +import numpy as np +import matplotlib.pyplot as plt +from sklearn import datasets + + +# ensure the same random numbers appear every time +np.random.seed(0) + +# display images in notebook +%matplotlib inline +plt.rcParams['figure.figsize'] = (12,12) + + +# download MNIST dataset +digits = datasets.load_digits() + +# define inputs and labels +inputs = digits.images +labels = digits.target + +print("inputs = (n_inputs, pixel_width, pixel_height) = " + str(inputs.shape)) +print("labels = (n_inputs) = " + str(labels.shape)) + + +# flatten the image +# the value -1 means dimension is inferred from the remaining dimensions: 8x8 = 64 +n_inputs = len(inputs) +inputs = inputs.reshape(n_inputs, -1) +print("X = (n_inputs, n_features) = " + str(inputs.shape)) + + +# choose some random images to display +indices = np.arange(n_inputs) +random_indices = np.random.choice(indices, size=5) + +for i, image in enumerate(digits.images[random_indices]): + plt.subplot(1, 5, i+1) + plt.axis('off') + plt.imshow(image, cmap=plt.cm.gray_r, interpolation='nearest') + plt.title("Label: %d" % digits.target[random_indices[i]]) +plt.show() +!ec + +!bc pycod +from keras.utils import to_categorical +from sklearn.model_selection import train_test_split + +# one-hot representation of labels +labels = to_categorical(labels) + +# split into train and test data +train_size = 0.8 +test_size = 1 - train_size +X_train, X_test, Y_train, Y_test = train_test_split(inputs, labels, train_size=train_size, + test_size=test_size) +!ec + + +===== Using TensorFlow backend ===== + +o Define model and architecture +o Choose cost function and optimizer + +!bc pycod +import tensorflow as tf + +class NeuralNetworkTensorflow: + def __init__( + self, + X_train, + Y_train, + X_test, + Y_test, + n_neurons_layer1=100, + n_neurons_layer2=50, + n_categories=2, + epochs=10, + batch_size=100, + eta=0.1, + lmbd=0.0, + ): + + # keep track of number of steps + self.global_step = tf.Variable(0, dtype=tf.int32, trainable=False, name='global_step') + + self.X_train = X_train + self.Y_train = Y_train + self.X_test = X_test + self.Y_test = Y_test + + self.n_inputs = X_train.shape[0] + self.n_features = X_train.shape[1] + self.n_neurons_layer1 = n_neurons_layer1 + self.n_neurons_layer2 = n_neurons_layer2 + self.n_categories = n_categories + + self.epochs = epochs + self.batch_size = batch_size + self.iterations = self.n_inputs // self.batch_size + self.eta = eta + self.lmbd = lmbd + + # build network piece by piece + # name scopes (with) are used to enforce creation of new variables + # https://www.tensorflow.org/guide/variables + self.create_placeholders() + self.create_DNN() + self.create_loss() + self.create_optimiser() + self.create_accuracy() + + def create_placeholders(self): + # placeholders are fine here, but "Datasets" are the preferred method + # of streaming data into a model + with tf.name_scope('data'): + self.X = tf.placeholder(tf.float32, shape=(None, self.n_features), name='X_data') + self.Y = tf.placeholder(tf.float32, shape=(None, self.n_categories), name='Y_data') + + def create_DNN(self): + with tf.name_scope('DNN'): + # the weights are stored to calculate regularization loss later + + # Fully connected layer 1 + self.W_fc1 = self.weight_variable([self.n_features, self.n_neurons_layer1], name='fc1', dtype=tf.float32) + b_fc1 = self.bias_variable([self.n_neurons_layer1], name='fc1', dtype=tf.float32) + a_fc1 = tf.nn.sigmoid(tf.matmul(self.X, self.W_fc1) + b_fc1) + + # Fully connected layer 2 + self.W_fc2 = self.weight_variable([self.n_neurons_layer1, self.n_neurons_layer2], name='fc2', dtype=tf.float32) + b_fc2 = self.bias_variable([self.n_neurons_layer2], name='fc2', dtype=tf.float32) + a_fc2 = tf.nn.sigmoid(tf.matmul(a_fc1, self.W_fc2) + b_fc2) + + # Output layer + self.W_out = self.weight_variable([self.n_neurons_layer2, self.n_categories], name='out', dtype=tf.float32) + b_out = self.bias_variable([self.n_categories], name='out', dtype=tf.float32) + self.z_out = tf.matmul(a_fc2, self.W_out) + b_out + + def create_loss(self): + with tf.name_scope('loss'): + softmax_loss = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits_v2(labels=self.Y, logits=self.z_out)) + + regularizer_loss_fc1 = tf.nn.l2_loss(self.W_fc1) + regularizer_loss_fc2 = tf.nn.l2_loss(self.W_fc2) + regularizer_loss_out = tf.nn.l2_loss(self.W_out) + regularizer_loss = self.lmbd*(regularizer_loss_fc1 + regularizer_loss_fc2 + regularizer_loss_out) + + self.loss = softmax_loss + regularizer_loss + + def create_accuracy(self): + with tf.name_scope('accuracy'): + probabilities = tf.nn.softmax(self.z_out) + predictions = tf.argmax(probabilities, axis=1) + labels = tf.argmax(self.Y, axis=1) + + correct_predictions = tf.equal(predictions, labels) + correct_predictions = tf.cast(correct_predictions, tf.float32) + self.accuracy = tf.reduce_mean(correct_predictions) + + def create_optimiser(self): + with tf.name_scope('optimizer'): + self.optimizer = tf.train.GradientDescentOptimizer(learning_rate=self.eta).minimize(self.loss, global_step=self.global_step) + + def weight_variable(self, shape, name='', dtype=tf.float32): + initial = tf.truncated_normal(shape, stddev=0.1) + return tf.Variable(initial, name=name, dtype=dtype) + + def bias_variable(self, shape, name='', dtype=tf.float32): + initial = tf.constant(0.1, shape=shape) + return tf.Variable(initial, name=name, dtype=dtype) + + def fit(self): + data_indices = np.arange(self.n_inputs) + + with tf.Session() as sess: + sess.run(tf.global_variables_initializer()) + for i in range(self.epochs): + for j in range(self.iterations): + chosen_datapoints = np.random.choice(data_indices, size=self.batch_size, replace=False) + batch_X, batch_Y = self.X_train[chosen_datapoints], self.Y_train[chosen_datapoints] + + sess.run([DNN.loss, DNN.optimizer], + feed_dict={DNN.X: batch_X, + DNN.Y: batch_Y}) + accuracy = sess.run(DNN.accuracy, + feed_dict={DNN.X: batch_X, + DNN.Y: batch_Y}) + step = sess.run(DNN.global_step) + + self.train_loss, self.train_accuracy = sess.run([DNN.loss, DNN.accuracy], + feed_dict={DNN.X: self.X_train, + DNN.Y: self.Y_train}) + + self.test_loss, self.test_accuracy = sess.run([DNN.loss, DNN.accuracy], + feed_dict={DNN.X: self.X_test, + DNN.Y: self.Y_test}) +!ec + + + +===== Optimizing and using gradient descent ===== + +!bc pycod +epochs = 100 +batch_size = 100 +n_neurons_layer1 = 100 +n_neurons_layer2 = 50 +n_categories = 10 +eta_vals = np.logspace(-5, 1, 7) +lmbd_vals = np.logspace(-5, 1, 7) +!ec + + +!bc pycod +DNN_tf = np.zeros((len(eta_vals), len(lmbd_vals)), dtype=object) + +for i, eta in enumerate(eta_vals): + for j, lmbd in enumerate(lmbd_vals): + DNN = NeuralNetworkTensorflow(X_train, Y_train, X_test, Y_test, + n_neurons_layer1, n_neurons_layer2, n_categories, + epochs=epochs, batch_size=batch_size, eta=eta, lmbd=lmbd) + DNN.fit() + + DNN_tf[i][j] = DNN + + print("Learning rate = ", eta) + print("Lambda = ", lmbd) + print("Test accuracy: %.3f" % DNN.test_accuracy) + print() +!ec + +!bc pycod +# optional +# visual representation of grid search +# uses seaborn heatmap, could probably do this in matplotlib +import seaborn as sns + +sns.set() + +train_accuracy = np.zeros((len(eta_vals), len(lmbd_vals))) +test_accuracy = np.zeros((len(eta_vals), len(lmbd_vals))) + +for i in range(len(eta_vals)): + for j in range(len(lmbd_vals)): + DNN = DNN_tf[i][j] + + train_accuracy[i][j] = DNN.train_accuracy + test_accuracy[i][j] = DNN.test_accuracy + + +fig, ax = plt.subplots(figsize = (10, 10)) +sns.heatmap(train_accuracy, annot=True, ax=ax, cmap="viridis") +ax.set_title("Training Accuracy") +ax.set_ylabel("$\eta$") +ax.set_xlabel("$\lambda$") +plt.show() + +fig, ax = plt.subplots(figsize = (10, 10)) +sns.heatmap(test_accuracy, annot=True, ax=ax, cmap="viridis") +ax.set_title("Test Accuracy") +ax.set_ylabel("$\eta$") +ax.set_xlabel("$\lambda$") +plt.show() +!ec + +!bc pycod +# optional +# we can use log files to visualize our graph in Tensorboard +writer = tf.summary.FileWriter('logs/') +writer.add_graph(tf.get_default_graph()) +!ec + + + +===== Using Keras ===== + +Keras is a high level "neural network":"https://en.wikipedia.org/wiki/Application_programming_interface" +that supports Tensorflow, CTNK and Theano as backends. +If you have Tensorflow installed Keras is available through the *tf.keras* module. +If you have Anaconda installed you may run the following command +!bc pycod +conda install keras +!ec + +Alternatively, if you have Tensorflow or one of the other supported backends install you may use the pip package manager: + +!bc pycod +pip3 install keras +!ec +or look up the "instructions here":"https://keras.io/". + +!bc pycod +from keras.models import Sequential +from keras.layers import Dense +from keras.regularizers import l2 +from keras.optimizers import SGD + +def create_neural_network_keras(n_neurons_layer1, n_neurons_layer2, n_categories, eta, lmbd): + model = Sequential() + model.add(Dense(n_neurons_layer1, activation='sigmoid', kernel_regularizer=l2(lmbd))) + model.add(Dense(n_neurons_layer2, activation='sigmoid', kernel_regularizer=l2(lmbd))) + model.add(Dense(n_categories, activation='softmax')) + + sgd = SGD(lr=eta) + model.compile(loss='categorical_crossentropy', optimizer=sgd, metrics=['accuracy']) + + return model +!ec + +!bc pycod +DNN_keras = np.zeros((len(eta_vals), len(lmbd_vals)), dtype=object) + +for i, eta in enumerate(eta_vals): + for j, lmbd in enumerate(lmbd_vals): + DNN = create_neural_network_keras(n_neurons_layer1, n_neurons_layer2, n_categories, + eta=eta, lmbd=lmbd) + DNN.fit(X_train, Y_train, epochs=epochs, batch_size=batch_size, verbose=0) + scores = DNN.evaluate(X_test, Y_test) + + DNN_keras[i][j] = DNN + + print("Learning rate = ", eta) + print("Lambda = ", lmbd) + print("Test accuracy: %.3f" % scores[1]) + print() +!ec + +!bc pycod +# optional +# visual representation of grid search +# uses seaborn heatmap, could probably do this in matplotlib +import seaborn as sns + +sns.set() + +train_accuracy = np.zeros((len(eta_vals), len(lmbd_vals))) +test_accuracy = np.zeros((len(eta_vals), len(lmbd_vals))) + +for i in range(len(eta_vals)): + for j in range(len(lmbd_vals)): + DNN = DNN_keras[i][j] + + train_accuracy[i][j] = DNN.evaluate(X_train, Y_train)[1] + test_accuracy[i][j] = DNN.evaluate(X_test, Y_test)[1] + + +fig, ax = plt.subplots(figsize = (10, 10)) +sns.heatmap(train_accuracy, annot=True, ax=ax, cmap="viridis") +ax.set_title("Training Accuracy") +ax.set_ylabel("$\eta$") +ax.set_xlabel("$\lambda$") +plt.show() + +fig, ax = plt.subplots(figsize = (10, 10)) +sns.heatmap(test_accuracy, annot=True, ax=ax, cmap="viridis") +ax.set_title("Test Accuracy") +ax.set_ylabel("$\eta$") +ax.set_xlabel("$\lambda$") +plt.show() +!ec + + + + + +===== Which activation function should I use? ===== + +The Back propagation algorithm we derived above works by going from +the output layer to the input layer, propagating the error gradient on +the way. Once the algorithm has computed the gradient of the cost +function with regards to each parameter in the network, it uses these +gradients to update each parameter with a Gradient Descent (GD) step. + + +Unfortunately for us, the gradients often get smaller and smaller as the +algorithm progresses down to the first hidden layers. As a result, the +GD update leaves the lower layer connection weights +virtually unchanged, and training never converges to a good +solution. This is known in the literature as +_the vanishing gradients problem_. + +In other cases, the opposite can happen, namely the the gradients can grow bigger and +bigger. The result is that many of the layers get large updates of the +weights the +algorithm diverges. This is the _exploding gradients problem_, which is +mostly encountered in recurrent neural networks. More generally, deep +neural networks suffer from unstable gradients, different layers may +learn at widely different speeds + + +===== Is the Logistic activation function (Sigmoid) our choice? ===== + +Although this unfortunate behavior has been empirically observed for +quite a while (it was one of the reasons why deep neural networks were +mostly abandoned for a long time), it is only around 2010 that +significant progress was made in understanding it. + +A paper titled "Understanding the Difficulty of Training Deep +Feedforward Neural Networks by Xavier Glorot and Yoshua Bengio":"http://proceedings.mlr.press/v9/glorot10a.html" found that +the problems with the popular logistic +sigmoid activation function and the weight initialization technique +that was most popular at the time, namely random initialization using +a normal distribution with a mean of 0 and a standard deviation of +1. + +They showed that with this activation function and this +initialization scheme, the variance of the outputs of each layer is +much greater than the variance of its inputs. Going forward in the +network, the variance keeps increasing after each layer until the +activation function saturates at the top layers. This is actually made +worse by the fact that the logistic function has a mean of 0.5, not 0 +(the hyperbolic tangent function has a mean of 0 and behaves slightly +better than the logistic function in deep networks). + + + +===== The derivative of the Logistic funtion ===== + +Looking at the logistic activation function, when inputs become large +(negative or positive), the function saturates at 0 or 1, with a +derivative extremely close to 0. Thus when backpropagation kicks in, +it has virtually no gradient to propagate back through the network, +and what little gradient exists keeps getting diluted as +backpropagation progresses down through the top layers, so there is +really nothing left for the lower layers. + +In their paper, Glorot and Bengio propose a way to significantly +alleviate this problem. We need the signal to flow properly in both +directions: in the forward direction when making predictions, and in +the reverse direction when backpropagating gradients. We don’t want +the signal to die out, nor do we want it to explode and saturate. For +the signal to flow properly, the authors argue that we need the +variance of the outputs of each layer to be equal to the variance of +its inputs, and we also need the gradients to have equal variance +before and after flowing through a layer in the reverse direction. + + + +One of the insights in the 2010 paper by Glorot and Bengio was that +the vanishing/exploding gradients problems were in part due to a poor +choice of activation function. Until then most people had assumed that +if Nature had chosen to use roughly sigmoid activation functions in +biological neurons, they must be an excellent choice. But it turns out +that other activation functions behave much better in deep neural +networks, in particular the ReLU activation function, mostly because +it does not saturate for positive values (and also because it is quite +fast to compute). + + + +===== The RELU function family ===== + +The ReLU activation function suffers from a problem known as the dying +ReLUs: during training, some neurons effectively die, meaning they +stop outputting anything other than 0. + +In some cases, you may find that half of your network’s neurons are +dead, especially if you used a large learning rate. During training, +if a neuron’s weights get updated such that the weighted sum of the +neuron’s inputs is negative, it will start outputting 0. When this +happen, the neuron is unlikely to come back to life since the gradient +of the ReLU function is 0 when its input is negative. + +To solve this problem, nowadays practitioners use a variant of the ReLU +function, such as the leaky ReLU discussed above or the so-called +exponential linear unit (ELU) function + + +!bt +\[ +ELU(z) = \left\{\begin{array}{cc} \alpha\left( \exp{(z)}-1\right) & z < 0,\\ z & z \ge 0.\end{array}\right. +\] +!et + + +===== Which activation function should we use? ===== + +In general it seems that the ELU activation function is better than +the leaky ReLU function (and its variants), which is better than +ReLU. ReLU performs better than $\tanh$ which in turn performs better +than the logistic function. + +If runtime +performance is an issue, then you may opt for the leaky ReLU function over the +ELU function If you don’t +want to tweak yet another hyperparameter, you may just use the default +$\alpha$ of $0.01$ for the leaky ReLU, and $1$ for ELU. If you have +spare time and computing power, you can use cross-validation or +bootstrap to evaluate other activation functions. + + + +===== A top-down perspective on Neural networks ===== + + +The first thing we would like to do is divide the data into two or three +parts. A training set, a validation or dev (development) set, and a +test set. The test set is the data on which we want to make +predictions. The dev set is a subset of the training data we use to +check how well we are doing out-of-sample, after training the model on +the training dataset. We use the validation error as a proxy for the +test error in order to make tweaks to our model. It is crucial that we +do not use any of the test data to train the algorithm. This is a +cardinal sin in ML. Then: + + +* Estimate optimal error rate + +* Minimize underfitting (bias) on training data set. + +* Make sure you are not overfitting. + +If the validation and test sets are drawn from the same distributions, +then a good performance on the validation set should lead to similarly +good performance on the test set. + +However, sometimes +the training data and test data differ in subtle ways because, for +example, they are collected using slightly different methods, or +because it is cheaper to collect data in one way versus another. In +this case, there can be a mismatch between the training and test +data. This can lead to the neural network overfitting these small +differences between the test and training sets, and a poor performance +on the test set despite having a good performance on the validation +set. To rectify this, Andrew Ng suggests making two validation or dev +sets, one constructed from the training data and one constructed from +the test data. The difference between the performance of the algorithm +on these two validation sets quantifies the train-test mismatch. This +can serve as another important diagnostic when using DNNs for +supervised learning. + + +===== Limitations of supervised learning with deep networks ===== + +Like all statistical methods, supervised learning using neural +networks has important limitations. This is especially important when +one seeks to apply these methods, especially to physics problems. Like +all tools, DNNs are not a universal solution. Often, the same or +better performance on a task can be achieved by using a few +hand-engineered features (or even a collection of random +features). + +Here we list some of the important limitations of supervised neural network based models. + + + +* _Need labeled data_. All supervised learning methods, DNNs for supervised learning require labeled data. Often, labeled data is harder to acquire than unlabeled data (e.g. one must pay for human experts to label images). +* _Supervised neural networks are extremely data intensive._ DNNs are data hungry. They perform best when data is plentiful. This is doubly so for supervised methods where the data must also be labeled. The utility of DNNs is extremely limited if data is hard to acquire or the datasets are small (hundreds to a few thousand samples). In this case, the performance of other methods that utilize hand-engineered features can exceed that of DNNs. +* _Homogeneous data._ Almost all DNNs deal with homogeneous data of one type. It is very hard to design architectures that mix and match data types (i.e.~some continuous variables, some discrete variables, some time series). In applications beyond images, video, and language, this is often what is required. In contrast, ensemble models like random forests or gradient-boosted trees have no difficulty handling mixed data types. +* _Many problems are not about prediction._ In natural science we are often interested in learning something about the underlying distribution that generates the data. In this case, it is often difficult to cast these ideas in a supervised learning setting. While the problems are related, it is possible to make good predictions with a *wrong* model. The model might or might not be useful for understanding the underlying science. + +Some of these remarks are particular to DNNs, others are shared by all supervised learning methods. This motivates the use of unsupervised methods which in part circumnavigate these problems. + + + + + + diff --git a/doc/LectureNotes/src/Hudson_Bay.py~ b/doc/LectureNotes/src/Hudson_Bay.py~ new file mode 100644 index 000000000..acd44518e --- /dev/null +++ b/doc/LectureNotes/src/Hudson_Bay.py~ @@ -0,0 +1,43 @@ +import numpy as np +import matplotlib.pyplot as plt + +def solver(m, H0, L0, dt, a, b, c, d, t0): + """Solve the difference equations for H and L over m years + with time step dt (measured in years.""" + + num_intervals = int(m/float(dt)) + t = np.linspace(t0, t0 + m, num_intervals+1) + H = np.zeros(t.size) + L = np.zeros(t.size) + + print 'Init:', H0, L0, dt + H[0] = H0 + L[0] = L0 + + for n in range(0, len(t)-1): + H[n+1] = H[n] + a*dt*H[n] - b*dt*H[n]*L[n] + L[n+1] = L[n] + d*dt*H[n]*L[n] - c*dt*L[n] + return H, L, t + +# Load in data file +data = np.loadtxt('Hudson_Bay.csv', delimiter=',', skiprows=1) +# Make arrays containing x-axis and hares and lynx populations +t_e = data[:,0] +H_e = data[:,1] +L_e = data[:,2] + +# Simulate using the model +H, L, t = solver(m=20, H0=34.91, L0=3.857, dt=0.1, + a=0.4807, b=0.02482, c=0.9272, d=0.02756, + t0=1900) + +# Visualize simulations and data +plt.plot(t_e, H_e, 'b-+', t_e, L_e, 'r-o', t, H, 'm--', t, L, 'k--') +plt.xlabel('Year') +plt.ylabel('Numbers of hares and lynx') +plt.axis([1900, 1920, 0, 140]) +plt.title(r'Population of hares and lynx 1900-1920 (x1000)') +plt.legend(('H_e', 'L_e', 'H', 'L'), loc='upper left') +plt.savefig('Hudson_Bay_sim.pdf') +plt.savefig('Hudson_Bay_sim.png') +plt.show() diff --git a/doc/LectureNotes/src/plot_Hudson.py~ b/doc/LectureNotes/src/plot_Hudson.py~ new file mode 100644 index 000000000..3b57c3277 --- /dev/null +++ b/doc/LectureNotes/src/plot_Hudson.py~ @@ -0,0 +1,19 @@ +import numpy as np +from matplotlib import pyplot as plt + +# Load in data file +data = np.loadtxt('src/Hudson_Bay.dat', delimiter=',', skiprows=1) +# Make arrays containing x-axis and hares and lynx populations +year = data[:,0] +hares = data[:,1] +lynx = data[:,2] + +plt.plot(year, hares ,'b-+', year, lynx, 'r-o') +plt.axis([1900,1920,0, 100.0]) +plt.xlabel(r'Year') +plt.ylabel(r'Numbers of hares and lynx ') +plt.legend(('Hares','Lynx'), loc='upper right') +plt.title(r'Population of hares and lynx from 1900-1920 (x1000)}') +plt.savefig('Hudson_Bay_data.pdf') +plt.savefig('Hudson_Bay_data.png') +plt.show() diff --git a/doc/Programs/.DS_Store b/doc/Programs/.DS_Store new file mode 100644 index 0000000000000000000000000000000000000000..b185d0e0321e52c9660918cf1c2c17da30a43239 GIT binary patch literal 6148 zcmeHK%}T>S5T0$TZYW|8f<5lVTMwaDTsEQ#;#xqmh&wn~lSh+lQy;;mhdt&9E!*N7S-raSZQh z%q{fM?2p>E#?L>f+9>QqA3;ji6Klm`jyRdE#?MIIS4&7j$>yQ_Jtz!?C4iI z9fWI;TV{Y6m}Q`7x)nPAkAHsu&lYiy8DIt$iUCn-`b`g)WY5;Q#nD--P;XI5C@(iS lDM3RY#h6P+aTQey`V|?7uEpFSdQkWxplRTS8TeBMz5un7Qx^aL literal 0 HcmV?d00001 diff --git a/doc/Programs/ANN/cnnkeras.py~ b/doc/Programs/ANN/cnnkeras.py~ new file mode 100644 index 000000000..f81b35353 --- /dev/null +++ b/doc/Programs/ANN/cnnkeras.py~ @@ -0,0 +1,330 @@ +# import necessary packages +import numpy as np +import matplotlib.pyplot as plt +from sklearn import datasets + + +# ensure the same random numbers appear every time +np.random.seed(0) + +# display images in notebook +plt.rcParams['figure.figsize'] = (12,12) + + +# download MNIST dataset +digits = datasets.load_digits() + +# define inputs and labels +inputs = digits.images +labels = digits.target + +# RGB images have a depth of 3 +# our images are grayscale so they should have a depth of 1 +inputs = inputs[:,:,:,np.newaxis] + +print("inputs = (n_inputs, pixel_width, pixel_height, depth) = " + str(inputs.shape)) +print("labels = (n_inputs) = " + str(labels.shape)) + + +# choose some random images to display +n_inputs = len(inputs) +indices = np.arange(n_inputs) +random_indices = np.random.choice(indices, size=5) + +for i, image in enumerate(digits.images[random_indices]): + plt.subplot(1, 5, i+1) + plt.axis('off') + plt.imshow(image, cmap=plt.cm.gray_r, interpolation='nearest') + plt.title("Label: %d" % digits.target[random_indices[i]]) +plt.show() + +from keras.utils import to_categorical +from sklearn.model_selection import train_test_split + +# representation of labels +labels = to_categorical(labels) + +# split into train and test data +# one-liner from scikit-learn library +train_size = 0.8 +test_size = 1 - train_size +X_train, X_test, Y_train, Y_test = train_test_split(inputs, labels, train_size=train_size, + test_size=test_size) + +import tensorflow as tf + +class ConvolutionalNeuralNetworkTensorflow: + def __init__( + self, + X_train, + Y_train, + X_test, + Y_test, + n_filters=10, + n_neurons_connected=50, + n_categories=10, + receptive_field=3, + stride=1, + padding=1, + epochs=10, + batch_size=100, + eta=0.1, + lmbd=0.0, + ): + + self.global_step = tf.Variable(0, dtype=tf.int32, trainable=False, name='global_step') + + self.X_train = X_train + self.Y_train = Y_train + self.X_test = X_test + self.Y_test = Y_test + + self.n_inputs, self.input_width, self.input_height, self.depth = X_train.shape + + self.n_filters = n_filters + self.n_downsampled = int(self.input_width*self.input_height*n_filters / 4) + self.n_neurons_connected = n_neurons_connected + self.n_categories = n_categories + + self.receptive_field = receptive_field + self.stride = stride + self.strides = [stride, stride, stride, stride] + self.padding = padding + + self.epochs = epochs + self.batch_size = batch_size + self.iterations = self.n_inputs // self.batch_size + self.eta = eta + self.lmbd = lmbd + + self.create_placeholders() + self.create_CNN() + self.create_loss() + self.create_optimiser() + self.create_accuracy() + + def create_placeholders(self): + with tf.name_scope('data'): + self.X = tf.placeholder(tf.float32, shape=(None, self.input_width, self.input_height, self.depth), name='X_data') + self.Y = tf.placeholder(tf.float32, shape=(None, self.n_categories), name='Y_data') + + def create_CNN(self): + with tf.name_scope('CNN'): + + # Convolutional layer + self.W_conv = self.weight_variable([self.receptive_field, self.receptive_field, self.depth, self.n_filters], name='conv', dtype=tf.float32) + b_conv = self.weight_variable([self.n_filters], name='conv', dtype=tf.float32) + z_conv = tf.nn.conv2d(self.X, self.W_conv, self.strides, padding='SAME', name='conv') + b_conv + a_conv = tf.nn.relu(z_conv) + + # 2x2 max pooling + a_pool = tf.nn.max_pool(a_conv, [1, 2, 2, 1], [1, 2, 2, 1], padding='SAME', name='pool') + + # Fully connected layer + a_pool_flat = tf.reshape(a_pool, [-1, self.n_downsampled]) + self.W_fc = self.weight_variable([self.n_downsampled, self.n_neurons_connected], name='fc', dtype=tf.float32) + b_fc = self.bias_variable([self.n_neurons_connected], name='fc', dtype=tf.float32) + a_fc = tf.nn.relu(tf.matmul(a_pool_flat, self.W_fc) + b_fc) + + # Output layer + self.W_out = self.weight_variable([self.n_neurons_connected, self.n_categories], name='out', dtype=tf.float32) + b_out = self.bias_variable([self.n_categories], name='out', dtype=tf.float32) + self.z_out = tf.matmul(a_fc, self.W_out) + b_out + + def create_loss(self): + with tf.name_scope('loss'): + softmax_loss = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits_v2(labels=self.Y, logits=self.z_out)) + + regularizer_loss_conv = tf.nn.l2_loss(self.W_conv) + regularizer_loss_fc = tf.nn.l2_loss(self.W_fc) + regularizer_loss_out = tf.nn.l2_loss(self.W_out) + regularizer_loss = self.lmbd*(regularizer_loss_conv + regularizer_loss_fc + regularizer_loss_out) + + self.loss = softmax_loss + regularizer_loss + + def create_accuracy(self): + with tf.name_scope('accuracy'): + probabilities = tf.nn.softmax(self.z_out) + predictions = tf.argmax(probabilities, 1) + labels = tf.argmax(self.Y, 1) + + correct_predictions = tf.equal(predictions, labels) + correct_predictions = tf.cast(correct_predictions, tf.float32) + self.accuracy = tf.reduce_mean(correct_predictions) + + def create_optimiser(self): + with tf.name_scope('optimizer'): + self.optimizer = tf.train.GradientDescentOptimizer(learning_rate=self.eta).minimize(self.loss, global_step=self.global_step) + + def weight_variable(self, shape, name='', dtype=tf.float32): + initial = tf.truncated_normal(shape, stddev=0.1) + return tf.Variable(initial, name=name, dtype=dtype) + + def bias_variable(self, shape, name='', dtype=tf.float32): + initial = tf.constant(0.1, shape=shape) + return tf.Variable(initial, name=name, dtype=dtype) + + def fit(self): + data_indices = np.arange(self.n_inputs) + + with tf.Session() as sess: + sess.run(tf.global_variables_initializer()) + for i in range(self.epochs): + for j in range(self.iterations): + chosen_datapoints = np.random.choice(data_indices, size=self.batch_size, replace=False) + batch_X, batch_Y = self.X_train[chosen_datapoints], self.Y_train[chosen_datapoints] + + sess.run([CNN.loss, CNN.optimizer], + feed_dict={CNN.X: batch_X, + CNN.Y: batch_Y}) + accuracy = sess.run(CNN.accuracy, + feed_dict={CNN.X: batch_X, + CNN.Y: batch_Y}) + step = sess.run(CNN.global_step) + + self.train_loss, self.train_accuracy = sess.run([CNN.loss, CNN.accuracy], + feed_dict={CNN.X: self.X_train, + CNN.Y: self.Y_train}) + + self.test_loss, self.test_accuracy = sess.run([CNN.loss, CNN.accuracy], + feed_dict={CNN.X: self.X_test, + CNN.Y: self.Y_test}) + +epochs = 100 +batch_size = 100 +n_filters = 10 +n_neurons_connected = 50 +n_categories = 10 + +eta_vals = np.logspace(-5, 1, 7) +lmbd_vals = np.logspace(-5, 1, 7) +CNN_tf = np.zeros((len(eta_vals), len(lmbd_vals)), dtype=object) + +for i, eta in enumerate(eta_vals): + for j, lmbd in enumerate(lmbd_vals): + CNN = ConvolutionalNeuralNetworkTensorflow(X_train, Y_train, X_test, Y_test, + n_filters=n_filters, n_neurons_connected=n_neurons_connected, + n_categories=n_categories, epochs=epochs, batch_size=batch_size, + eta=eta, lmbd=lmbd) + CNN.fit() + + print("Learning rate = ", eta) + print("Lambda = ", lmbd) + print("Test accuracy: %.3f" % CNN.test_accuracy) + print() + + CNN_tf[i][j] = CNN + +# visual representation of grid search +# uses seaborn heatmap, could probably do this in matplotlib +import seaborn as sns + +sns.set() + +train_accuracy = np.zeros((len(eta_vals), len(lmbd_vals))) +test_accuracy = np.zeros((len(eta_vals), len(lmbd_vals))) + +for i in range(len(eta_vals)): + for j in range(len(lmbd_vals)): + CNN = CNN_tf[i][j] + + train_accuracy[i][j] = CNN.train_accuracy + test_accuracy[i][j] = CNN.test_accuracy + + +fig, ax = plt.subplots(figsize = (10, 10)) +sns.heatmap(train_accuracy, annot=True, ax=ax, cmap="viridis") +ax.set_title("Training Accuracy") +ax.set_ylabel("$\eta$") +ax.set_xlabel("$\lambda$") +plt.show() + +fig, ax = plt.subplots(figsize = (10, 10)) +sns.heatmap(test_accuracy, annot=True, ax=ax, cmap="viridis") +ax.set_title("Test Accuracy") +ax.set_ylabel("$\eta$") +ax.set_xlabel("$\lambda$") +plt.show() + +from keras.models import Sequential +from keras.layers.convolutional import Conv2D +from keras.layers.convolutional import MaxPooling2D +from keras.layers import Flatten +from keras.layers import Dense +from keras.regularizers import l2 +from keras.optimizers import SGD + +def create_convolutional_neural_network_keras(input_shape, receptive_field, + n_filters, n_neurons_connected, n_categories, + eta, lmbd): + model = Sequential() + model.add(Conv2D(n_filters, (receptive_field, receptive_field), input_shape=input_shape, padding='same', + activation='relu', kernel_regularizer=l2(lmbd))) + model.add(MaxPooling2D(pool_size=(2, 2))) + model.add(Flatten()) + model.add(Dense(n_neurons_connected, activation='relu', kernel_regularizer=l2(lmbd))) + model.add(Dense(n_categories, activation='softmax', kernel_regularizer=l2(lmbd))) + + sgd = SGD(lr=eta) + model.compile(loss='categorical_crossentropy', optimizer=sgd, metrics=['accuracy']) + + return model + +epochs = 100 +batch_size = 100 +input_shape = X_train.shape[1:4] +receptive_field = 3 +n_filters = 10 +n_neurons_connected = 50 +n_categories = 10 + +eta_vals = np.logspace(-5, 1, 7) +lmbd_vals = np.logspace(-5, 1, 7) + +CNN_keras = np.zeros((len(eta_vals), len(lmbd_vals)), dtype=object) + +for i, eta in enumerate(eta_vals): + for j, lmbd in enumerate(lmbd_vals): + CNN = create_convolutional_neural_network_keras(input_shape, receptive_field, + n_filters, n_neurons_connected, n_categories, + eta, lmbd) + CNN.fit(X_train, Y_train, epochs=epochs, batch_size=batch_size, verbose=0) + scores = CNN.evaluate(X_test, Y_test) + + CNN_keras[i][j] = CNN + + print("Learning rate = ", eta) + print("Lambda = ", lmbd) + print("Test accuracy: %.3f" % scores[1]) + print() + +# visual representation of grid search +# uses seaborn heatmap, could probably do this in matplotlib +import seaborn as sns + +sns.set() + +train_accuracy = np.zeros((len(eta_vals), len(lmbd_vals))) +test_accuracy = np.zeros((len(eta_vals), len(lmbd_vals))) + +for i in range(len(eta_vals)): + for j in range(len(lmbd_vals)): + CNN = CNN_keras[i][j] + + train_accuracy[i][j] = CNN.evaluate(X_train, Y_train)[1] + test_accuracy[i][j] = CNN.evaluate(X_test, Y_test)[1] + + +fig, ax = plt.subplots(figsize = (10, 10)) +sns.heatmap(train_accuracy, annot=True, ax=ax, cmap="viridis") +ax.set_title("Training Accuracy") +ax.set_ylabel("$\eta$") +ax.set_xlabel("$\lambda$") +plt.show() + +fig, ax = plt.subplots(figsize = (10, 10)) +sns.heatmap(test_accuracy, annot=True, ax=ax, cmap="viridis") +ax.set_title("Test Accuracy") +ax.set_ylabel("$\eta$") +ax.set_xlabel("$\lambda$") +plt.show() diff --git a/doc/Programs/DimRed/covariance.py~ b/doc/Programs/DimRed/covariance.py~ new file mode 100644 index 000000000..38d1c7380 --- /dev/null +++ b/doc/Programs/DimRed/covariance.py~ @@ -0,0 +1,34 @@ +#The covariance matrix and its eigenvalues the hard way +from random import random, seed +import numpy as np + +def covariance(x, y, n): + sum = 0.0 + mean_x = np.mean(x) + mean_y = np.mean(y) + for i in range(0, n): + sum += (x[(i)]-mean_x)*(y[i]-mean_y) + return sum/n + +n = 10 + +x = np.random.normal(size=n) +y = np.random.normal(size=n) +z = x*x*x+y*y 0.5*np.random.normal(size=n) +covxx = covariance(x,x,n) +covxy = covariance(x,y,n) +covxz = covariance(x,z,n) +covyy = covariance(y,y,n) +covyz = covariance(y,z,n) +covzz = covariance(z,z,n) + +SigmaCov = np.array([ [covxx, covxy, covxz], [covxy, covyy, covyz], [covxz, covyz, covzz]]) +print(SigmaCov) + +EigValues, EigVectors = np.linalg.eig(SigmaCov) +# sort eigenvectors and eigenvalues +permute = EigValues.argsort() +EigValues = EigValues[permute] +EigVectors = EigVectors[:,permute] +print(EigValues) +print(EigVectors) diff --git a/doc/Programs/Finance/agents.py~ b/doc/Programs/Finance/agents.py~ new file mode 100644 index 000000000..73904afc8 --- /dev/null +++ b/doc/Programs/Finance/agents.py~ @@ -0,0 +1,38 @@ +# Simulation of financial transations with or without saving/taxation on transaction +# If lambda =0.0, no saving/taxation +# See Patriarca et al http://www.sciencedirect.com/science/article/pii/S0378437104004327 +#!/usr/bin/env python +import numpy as np +import matplotlib.mlab as mlab +import matplotlib.pyplot as plt +import random + +# initialize the rng with a seed +random.seed() +# Hard coding of input parameters +Agents = 500 +MCcounts = 1000 +Transactions = 100000 +startMoney = 1.0 +Lambda = 0.5 +FinancialAgents = startMoney*np.ones(Agents) +for i in range (1, MCcounts, 1): + for j in range (1, Transactions, 1): + agent_i = int(Agents*random.random()) + agent_j = int(Agents*random.random()) + epsilon = random.random() + if agent_i != agent_j: + m1 = Lambda*FinancialAgents[agent_i] + (1-Lambda)*epsilon*(FinancialAgents[agent_i] + FinancialAgents[agent_j]) + m2 = Lambda*FinancialAgents[agent_j] + (1-Lambda)*(1-epsilon)*(FinancialAgents[agent_i] + FinancialAgents[agent_j]) + FinancialAgents[agent_i] = m1 + FinancialAgents[agent_j] = m2 + +# the histogram of the data +n, bins, patches = plt.hist(FinancialAgents, 50, facecolor='green') + +plt.xlabel('$x$') +plt.ylabel('Distribution of wealth') +plt.title(r'Money') +plt.axis([0, 10, 0, 100]) +plt.grid(True) +plt.show() diff --git a/doc/Programs/RandomWalks/OneDimParticle.py~ b/doc/Programs/RandomWalks/OneDimParticle.py~ new file mode 100644 index 000000000..ec77cdb42 --- /dev/null +++ b/doc/Programs/RandomWalks/OneDimParticle.py~ @@ -0,0 +1,45 @@ +# Program to test the Metropolis algorithm with one particle at given temp in +# one dimension +#!/usr/bin/env python +import numpy as np +import matplotlib.mlab as mlab +import matplotlib.pyplot as plt +import random +from math import sqrt, exp, log +# initialize the rng with a seed +random.seed() +# Hard coding of input parameters +MCcycles = 100000 +Temperature = 2.0 +beta = 1./Temperature +InitialVelocity = -2.0 +CurrentVelocity = InitialVelocity +Energy = 0.5*InitialVelocity*InitialVelocity +VelocityRange = 10*sqrt(Temperature) +VelocityStep = 2*VelocityRange/10. +AverageEnergy = Energy +AverageEnergy2 = Energy*Energy +VelocityValues = np.zeros(MCcycles) +# The Monte Carlo sampling with Metropolis starts here +for i in range (1, MCcycles, 1): + TrialVelocity = CurrentVelocity + (2.0*random.random() - 1.0)*VelocityStep + EnergyChange = 0.5*(TrialVelocity*TrialVelocity -CurrentVelocity*CurrentVelocity); + if random.random() <= exp(-beta*EnergyChange): + CurrentVelocity = TrialVelocity + Energy += EnergyChange + VelocityValues[i] = CurrentVelocity + AverageEnergy += Energy + AverageEnergy2 += Energy*Energy +#Final averages +AverageEnergy = AverageEnergy/MCcycles +AverageEnergy2 = AverageEnergy2/MCcycles +Variance = AverageEnergy2 - AverageEnergy*AverageEnergy +print(AverageEnergy, Variance) +n, bins, patches = plt.hist(VelocityValues, 400, facecolor='green') + +plt.xlabel('$v$') +plt.ylabel('Velocity distribution P(v)') +plt.title(r'Velocity histogram at $k_BT=2$') +plt.axis([-5, 5, 0, 600]) +plt.grid(True) +plt.show() diff --git a/doc/Programs/ResamplingAnalysisScripts/README.md~ b/doc/Programs/ResamplingAnalysisScripts/README.md~ new file mode 100644 index 000000000..16a5f15ee --- /dev/null +++ b/doc/Programs/ResamplingAnalysisScripts/README.md~ @@ -0,0 +1,19 @@ +# ResamplingAnalysisScripts + +## Sample Scripts for data Analysis +So far this is a simple python script (should be made parallel...) to perform resampling of a data set. Methods used are __Bootstrapping__, __Jackknife__ and __Blocking__. + +## Usage +Simply run `python analysis.py FILENAME.xxx [NLINES]` + +Where `FILENAME` is expected to have a 3 charachter extension `NLINES` (optional) is the number of lines in the file to read and process (default is the whole file, but it gets very slow above 2-3 hundred thousand entries) + +Ouput is located into the `FILENAME/` folder. + +If more than 10⁵ lines are specified the autocorrelation function won't be computed, as it would take too long. + +The `gaussian.dat` dataset has been generated with numpy, as a proof of concept. It represents a normally distributed set of 5x10⁵ elements with `std = 0.05`. One will notice that the estimate on the error of the central value is greatly improved by all resampling methods. + +`energy.dat` is an autocorrelated data set, with autocorrelation time of roughly 200. It is useful to see the use of blocking on this dataset as a convenient method to estimate the autocorrelation time (compare the elapsed time on the different methods). + +In the `plaquette.dat` file there is a small data set (just 1000 samples) and it shows the strenght of using resampling methods to better estimate the error on the central value as opposed to the standard deviation. diff --git a/doc/Programs/SVD/Fortran/simplefit.dat~ b/doc/Programs/SVD/Fortran/simplefit.dat~ new file mode 100755 index 000000000..69cb54c8b --- /dev/null +++ b/doc/Programs/SVD/Fortran/simplefit.dat~ @@ -0,0 +1,9 @@ +8 2 +0.001 -2.89017 0.00073621 +0.002 -2.88946 0.00052732 +0.005 -2.89067 0.00055038 +0.010 -2.89091 0.00040973 +0.015 -2.89084 0.00034278 +0.02 -2.89086 0.00029315 +0.025 -2.89059 0.00034278 +0.03 -2.89077 0.00025017 \ No newline at end of file diff --git a/doc/Programs/Sampling/.DS_Store b/doc/Programs/Sampling/.DS_Store new file mode 100644 index 0000000000000000000000000000000000000000..5008ddfcf53c02e82d7eee2e57c38e5672ef89f6 GIT binary patch literal 6148 zcmeH~Jr2S!425mzP>H1@V-^m;4Wg<&0T*E43hX&L&p$$qDprKhvt+--jT7}7np#A3 zem<@ulZcFPQ@L2!n>{z**++&mCkOWA81W14cNZlEfg7;MkzE(HCqgga^y>{tEnwC%0;vJ&^%eQ zLs35+`xjp>T0H1@V-^m;4Wg<&0T*E43hX&L&p$$qDprKhvt+--jT7}7np#A3 zem<@ulZcFPQ@L2!n>{z**++&mCkOWA81W14cNZlEfg7;MkzE(HCqgga^y>{tEnwC%0;vJ&^%eQ zLs35+`xjp>T0 + + + + + + +Project 2 on Machine Learning, deadline November 12 + + + + + + + + + + + + + + + + + + + + + + + +
+
+ + Project 2 on Machine Learning, deadline November 12 +
+ +
+ +
+
+ + +
+ +

 

 

 

+ + + + + + +
+

Project 2 on Machine Learning, deadline November 12

+ +

+ + +

+Data Analysis and Machine Learning FYS-STK3155/FYS4155 +
+ +

+ + +

Department of Physics, University of Oslo, Norway
+
+

+

Nov 2, 2018

+
+

+

+ +

Classification and Regression, from linear and logistic regression to neural networks

+ +

+The main aim of this project is to study both classification and +regression problems, starting with the regression algorithms studied +in project 1. We will include logistic regresion for classification +problems and write our own multilayer perceptron code for studying +both regression and classification problems. The codes developed in +project 1, including bootstrap and/or cross-validation as well as the +computation of the mean-squared error and the R2 score function can +also be utilized (and included in logistic regression and the neural +network codes) in the present analysis. + +

+We will use the so-called Ising model for our training data and will +focus on supervised training. We will follow closely the recent +article of Mehta et al, arXiv +1803.08823. This article stands +out as an excellent review on machine learning (ML) algorithms. +The added benefit is that each figure and +model presented in this article is accompanied by its jupyter +notebook. This +means that we can start using these and compare with our own results. +They provide also the data set for the regression and classification +analysis that we will explore. In this sense, with their available +notebooks, it makes life easier since we can compare our own codes +with their codes. + +

+With the abovementioned configurations we will determine, using first +various regression methods, the value of the coupling constant for the +energy of the one-dimensional Ising model. Thereafter, we will use the +two-dimensional data, but now computed at different temperatures, in +order to classify the phase of the Ising model. Below the critical +temperature, the system will be in a so-called ferromagnetic +phase. Close to the critical temperature, the final magnetization +becomes smaller and smaller in absolute value while above the critical +temperature, the net magnetization is zero. This classification case, +that is the two-dimensional Ising model, will be studied using +logistic regression and deep neural networks. The aim is to develop +your own logistic regression code for the classification of the phases +(this is a binary model) and your multilayer perceptron code for the +classification and regression case. You can compare your own results with those obtained +using scikit-learn or tensorflow or other Python packages such as keras or other. + +

+Feel free to use the notebooks to benchmark your code. If you wish to +write your own C++ or Fortran program for say a multilayer neural network +model and a logistic regression model, please feel free to do so. + +

Part a): Producing the data for the one-dimensional Ising model

+ +

+The model we will employ in our studies is the so-called Ising +model. Together with +models like the Potts +model and similar +so-called lattice models, the Ising model has been widely studied in +mathematics (in statistics in particular), physics, life +science, +chemistry and even in the social sciences in order to model social +behavior. It is a +simple binary value system where the variables of the model (spins often in +physics) can take two values only, for example \( \pm 1 \) or \( 0 \) and \( 1 \). +The system exhibits a phase transition in two or higher dimensions and +the first person to find the analytical expressions for various +expectation values was the Norwegian chemist Lars +Onsager (Nobel prize in +chemistry) after a tour de force mathematics exercise. + +

+In our discussions here we will stay with a physicist's approach and +call the variables for spin. You could replace this with any other +type of binary variables, ranging from a two political parties to blue +and red spheres. In its simplest form we define the energy of the +system as + +$$ +\begin{equation*} + E=-J\sum_{< kl>}^{N}s_ks_l, +\end{equation*} +$$ + +with \( s_k=\pm 1 \), \( N \) is the total number of spins, +\( J \) is a coupling constant expressing the strength of the interaction +between neighboring spins. + +

+The symbol \( < kl> \) indicates that we sum over nearest +neighbors only. +Notice that for \( J>0 \) it is energetically favorable for neighboring spins +to be aligned. This feature leads to, at low enough temperatures, +a cooperative phenomenon called spontaneous magnetization. That is, +through interactions between nearest neighbors, a given magnetic +moment can influence the alignment of spins that are separated +from the given spin by a macroscopic distance. These long range correlations +between spins are associated with a long-range order in which +the lattice has a net magnetization in the absence of a magnetic field. + +

+We start by considering the one-dimensional Ising model with nearest neighbor interactions. This model does not exhibit any phase transition. + +

+Consider the 1D Ising model with nearest-neighbor interactions + +$$ +\begin{equation*} + E[\hat{s}]=-J\sum_{j=1}^{N}s_{j}s_{j+1}, +\end{equation*} +$$ + +

+on a chain of length \( N \) with so-called periodic boundary conditions and \( S_j=\pm 1 \) Ising spin variables. +In one dimension, this model has no phase transition at finite temperature. + +

+In the Python code below we generate, with a coupling coefficient set to \( J=1 \), a large number of spin configurations say \( 10000 \) as shown in the code below. +It means that our data will be a set of \( i=1\ldots n \) points of the form +\( \{(E[\boldsymbol{s}^i],\boldsymbol{s}^i)\} \). +Our task is to find the value of \( J \) from the data set using linear regression. + +

+Here is the Python code you need to generate the training data, see +also the notebook of Mehta et +al. + +

+ + +

import numpy as np
+import scipy.sparse as sp
+np.random.seed(12)
+
+import warnings
+#Comment this to turn on warnings
+warnings.filterwarnings('ignore')
+
+### define Ising model aprams
+# system size
+L=40
+
+# create 10000 random Ising states
+states=np.random.choice([-1, 1], size=(10000,L))
+
+def ising_energies(states,L):
+    """
+    This function calculates the energies of the states in the nn Ising Hamiltonian
+    """
+    J=np.zeros((L,L),)
+    for i in range(L):
+        J[i,(i+1)%L]-=1.0
+    # compute energies
+    E = np.einsum('...i,ij,...j->...',states,J,states)
+
+    return E
+# calculate Ising energies
+energies=ising_energies(states,L)
+
+

+We can now recast the problem as a linear regression model using our codes from project 1. +The way we are going to build our model mimicks the way we could think of finding say the gravitional constant for the graviational force between two planets. +In the absence of any prior knowledge, one sensible choice is the all-to-all Ising model + +$$ +E_\mathrm{model}[\boldsymbol{s}^i] = - \sum_{j=1}^N \sum_{k=1}^N J_{j,k}s_{j}^is_{k}^i. +$$ + +

+Here \( i \) represents a particular spin configuration (one of the possible \( n \) configurations we generated with the code above). + +

+This model is uniquely defined by the non-local coupling strengths \( J_{jk} \) which we want to learn. +The model is linear in \( \mathbf{J} \) which makes it possible to use linear regression. + +

+To apply linear regression, we recast this model in the form +$$ +E_\mathrm{model}^i \equiv \mathbf{X}^i \cdot \mathbf{J}, +$$ + +

+where the vectors \( \mathbf{X}^i \) represent all two-body interactions +\( \{s_{j}^is_{k}^i \}_{j,k=1}^N \), and the index \( i \) runs over the +samples in the data set. To make the analogy complete, we can also +represent the dot product by a single index \( p = \{j,k\} \), +i.e. \( \mathbf{X}^i \cdot \mathbf{J}=X^i_pJ_p \). Note that the +regression model does not include the minus sign, so we expect to +learn negative \( J \)'s. + +

+With these preliminaries, we are now ready to reutilize our codes from project 1. + +

Part b): Estimating the coupling constant of the one-dimensional Ising model using linear regression

+ +

+We start with the one-dimensional Ising model and use the data we have +generated with \( J=1 \) in the previous point. Use linear regression, +Lasso and Ridge regression as done in project 1. You can compare your +results with those of Mehta +et al.. +Make sure it is the 1D data which is used. + +

+Discuss the methods and how they perform in computing the coupling +constant \( J \) and include a bias-variance analysis using either +cross-validation or bootstrap. Discuss also the mean squared error and +the \( R2 \) score as measures to assess your model. + +

+Give a critical analysis of your results. + +

Part c): Determine the phase of the two-dimensional Ising model

+ +

+We switch now to binary classification methods and use logistic +regression to define the phases of the Ising model. This means that we switch to the two-dimensional Ising model +and use the data sets generated by Mehta et al +These energies and their corresponding spin orientation configurations +represent then your data. We will use a fixed lattice of \( L\times L = +40 \times 40 \) spins in two dimensions. The link above contains data for several temperatures. +The theoretical critical temperature for a phase transition is \( T_C\approx 2.269 \) in units of energy. +However, for a finite lattice the results representing the critical temperature are slightly higher (\( T_C \approx 2.3 \)). + +

+Our goal here, using logistic regression, is to train our model to +predict the phase of a sample given the spin configuration, whether it +represents a state above the critical temperature or below. The +configurations representing states below the critical temperature are +called ordered states (the spins tend to point in one direction, +resulting in a net magnetic moment) while those above the critical +temperature are called disordered. Since a finite lattice like this +does not exhibit a clear sign of a phase transition we will mainly +stay with either orderer or disordered phases. You could include the +critical phase if you want. + +

+Your aim here is thus to read in these data (use the examples from +Mehta et +al) +and write your own code for doing logistic regression, see the lecture +notes on logistic +regression. + +

+In this case, to evaluate the model, we will use the so-called accuracy score +instead of the bootstrap or cross-validation as done in the standard linear regression part discussed in b). Examples of how to define the accuracy score can be found under the neural network slides, see for example the slides here. + +

+To measure the performance of our network we evaluate how well it does +it data it has never seen before, i.e. the test data. We measure the +performance of the network using the accuracy score. The accuracy +is as you would expect just the number of images correctly labeled +divided by the total number of images. A perfect classifier will have +an accuracy score of \( 1 \). + +$$ +\text{Accuracy} = \frac{\sum_{i=1}^n I(t_i = y_i)}{n} , +$$ + +

+where \( I \) is the indicator function, \( 1 \) if \( t_i = y_i \) and \( 0 \) +otherwise, where \( t_i \) represents the target and \( y_i \) the outputs. + +

+In order to find the optimal parameters of your logistic regressor you should +include a gradient descent solver, as discussed in the gradient +descent +lectures. +Since we don't have so many data points, you may just code the +standard gradient descent with a given learning rate, or even attempt +to use the Newton-Raphson method. Alternatively, it may be useful for +the next part on neural networks to implement a stochastic gradient +descent with and without mini-batches. Stochastic gradient with mini-batches may give the best results. You could finally compare your code with the output from scikit-learn's toolbox for +optimization methods applied to logistic regression. + +

+The notebook of Mehta et al is highly recommended in order to benchmark your code and results. + +

Part d): Regression analysis of the one-dimensional Ising model using neural networks

+ +

+Your aim now, and this is the central part of this project, is to +write to your own multilayer perceptron model implementing the back +propagation algorithm discussed in the lecture +slides. We +start with the regression case discussed in parts a) and b) but train +now the network to find the optimal weights and biases. You are free +to use the codes in the above lecture slides as starting points. + +

+Train your network and compare the results with those from your linear regression code. +You can test your results against a similar code using _scikit_learn_ (see the examples in the above lecture notes) or tensorflow/keras. + +

+A useful reference on the back progagation algorithm is Nielsen's book. It is an excellent read. + +

Part e): Classifying the Ising model phase using neural networks

+ +

+Finally, change now your cost function to the \( log \) cross-entropy classification cost function for the case discussed in part c). Train your network again and +compare the results with those from your logistic regression code i c). +Here again you can compare your results with those of Mehta et al. There they used tensorflow to classify the phases. + +

Part f) Critical evaluation of the various algorithms

+ +

+After all these glorious calculations, you should now summarize the various algorithms and come with a critical evaluation of their pros and cons. Which algorithm works best for the regression case and which is best for the classification case. These codes will also be part of your final project 3, but now applied to other data sets. + +

Background literature

+ +
    +
  1. The text of Michael Nielsen is highly recommended, see Nielsen's book. It is an excellent read.
    2. +
  2. The textbook of Trevor Hastie, Robert Tibshirani, Jerome H. Friedman, The Elements of Statistical Learning, Springer, chapters 3 and 7 are the most relevant ones for the analysis here.
    3. +
  3. Mehta et al, arXiv 1803.08823, A high-bias, low-variance introduction to Machine Learning for physicists, ArXiv:1803.08823.
    4. +
+ +If you wish to read more about the Ising model and statistical physics here are three suggestions. + +
    +
  1. M. Plischke and B. Bergersen, Equilibrium Statistical Physics, World Scientific, see chapters 5 and 6.
    2. +
  2. D. P. Landau and K. Binder, A Guide to Monte Carlo Simulations in Statistical Physics, Cambridge, see chapters 2,3 and 4.
    3. +
  3. M. E. J. Newman and T. Barkema, Monte Carlo Methods in Statistical Physics, Oxford, see chapters 3 and 4.
    4. +
+ +

Introduction to numerical projects

+ +

+Here follows a brief recipe and recommendation on how to write a report for each +project. + +

    +
  • Give a short description of the nature of the problem and the eventual numerical methods you have used.
    • +
  • Describe the algorithm you have used and/or developed. Here you may find it convenient to use pseudocoding. In many cases you can describe the algorithm in the program itself.
    • +
  • Include the source code of your program. Comment your program properly.
    • +
  • If possible, try to find analytic solutions, or known limits in order to test your program when developing the code.
    • +
  • Include your results either in figure form or in a table. Remember to label your results. All tables and figures should have relevant captions and labels on the axes.
    • +
  • Try to evaluate the reliabilty and numerical stability/precision of your results. If possible, include a qualitative and/or quantitative discussion of the numerical stability, eventual loss of precision etc.
    • +
  • Try to give an interpretation of you results in your answers to the problems.
    • +
  • Critique: if possible include your comments and reflections about the exercise, whether you felt you learnt something, ideas for improvements and other thoughts you've made when solving the exercise. We wish to keep this course at the interactive level and your comments can help us improve it.
    • +
  • Try to establish a practice where you log your work at the computerlab. You may find such a logbook very handy at later stages in your work, especially when you don't properly remember what a previous test version of your program did. Here you could also record the time spent on solving the exercise, various algorithms you may have tested or other topics which you feel worthy of mentioning.
    • +
+ +

Format for electronic delivery of report and programs

+ +

+The preferred format for the report is a PDF file. You can also use DOC or postscript formats or as an ipython notebook file. As programming language we prefer that you choose between C/C++, Fortran2008 or Python. The following prescription should be followed when preparing the report: + +

    +
  • Use Devilry to hand in your projects, log in at http://devilry.ifi.uio.no with your normal UiO username and password and choose either 'fysstk3155' or 'fysstk4155'. There you can load up the files within the deadline.
    • +
  • Upload only the report file! For the source code file(s) you have developed please provide us with your link to your github domain. The report file should include all of your discussions and a list of the codes you have developed. Do not include library files which are available at the course homepage, unless you have made specific changes to them.
    • +
  • In your git repository, please include a folder which contains selected results. These can be in the form of output from your code for a selected set of runs and input parameters.
    • +
  • In this and all later projects, you should include tests (for example unit tests) of your code(s).
    • +
  • Comments from us on your projects, approval or not, corrections to be made etc can be found under your Devilry domain and are only visible to you and the teachers of the course.
    • +
+ +Finally, +we encourage you to collaborate. Optimal working groups consist of +2-3 students. You can then hand in a common report. + +

Software and needed installations

+ +

+If you have Python installed (we recommend Python3) and you feel pretty familiar with installing different packages, +we recommend that you install the following Python packages via pip as + +

    +
  1. pip install numpy scipy matplotlib ipython scikit-learn tensorflow sympy pandas pillow
    2. +
+ +For Python3, replace pip with pip3. + +

+See below for a discussion of tensorflow and scikit-learn. + +

+For OSX users we recommend also, after having installed Xcode, to install brew. Brew allows +for a seamless installation of additional software via for example + +

    +
  1. brew install python3
    2. +
+ +For Linux users, with its variety of distributions like for example the widely popular Ubuntu distribution +you can use pip as well and simply install Python as + +
    +
  1. sudo apt-get install python3 (or python for python2.7)
    2. +
+ +etc etc. + +

+If you don't want to install various Python packages with their dependencies separately, we recommend two widely used distrubutions which set up all relevant dependencies for Python, namely + +

    +
  1. Anaconda Anaconda is an open source distribution of the Python and R programming languages for large-scale data processing, predictive analytics, and scientific computing, that aims to simplify package management and deployment. Package versions are managed by the package management system conda
    2. +
  2. Enthought canopy is a Python distribution for scientific and analytic computing distribution and analysis environment, available for free and under a commercial license.
    3. +
+ +Popular software packages written in Python for ML are + + + +These are all freely available at their respective GitHub sites. They +encompass communities of developers in the thousands or more. And the number +of code developers and contributors keeps increasing. + +

+ +

+ +

    +
  • 1
    • +
+ + +
+ + + + + + + +
+ © 1999-2018, "Data Analysis and Machine Learning FYS-STK3155/FYS4155":"http://www.uio.no/studier/emner/matnat/fys/FYS3155/index-eng.html". Released under CC Attribution-NonCommercial 4.0 license +
+ + + + + + diff --git a/doc/Projects/2018/Project2/pdf/Project2.tex~ b/doc/Projects/2018/Project2/pdf/Project2.tex~ new file mode 100644 index 000000000..08a9e82ca --- /dev/null +++ b/doc/Projects/2018/Project2/pdf/Project2.tex~ @@ -0,0 +1,557 @@ +%% +%% Automatically generated file from DocOnce source +%% (https://github.com/hplgit/doconce/) +%% +%% + + +%-------------------- begin preamble ---------------------- + +\documentclass[% +oneside, % oneside: electronic viewing, twoside: printing +final, % draft: marks overfull hboxes, figures with paths +10pt]{article} + +\listfiles % print all files needed to compile this document + +\usepackage{relsize,makeidx,color,setspace,amsmath,amsfonts,amssymb} +\usepackage[table]{xcolor} +\usepackage{bm,ltablex,microtype} + +\usepackage[pdftex]{graphicx} + +\usepackage{fancyvrb} % packages needed for verbatim environments + +\usepackage[T1]{fontenc} +%\usepackage[latin1]{inputenc} +\usepackage{ucs} +\usepackage[utf8x]{inputenc} + +\usepackage{lmodern} % Latin Modern fonts derived from Computer Modern + +% Hyperlinks in PDF: +\definecolor{linkcolor}{rgb}{0,0,0.4} +\usepackage{hyperref} +\hypersetup{ + breaklinks=true, + colorlinks=true, + linkcolor=linkcolor, + urlcolor=linkcolor, + citecolor=black, + filecolor=black, + %filecolor=blue, + pdfmenubar=true, + pdftoolbar=true, + bookmarksdepth=3 % Uncomment (and tweak) for PDF bookmarks with more levels than the TOC + } +%\hyperbaseurl{} % hyperlinks are relative to this root + +\setcounter{tocdepth}{2} % levels in table of contents + +% --- fancyhdr package for fancy headers --- +\usepackage{fancyhdr} +\fancyhf{} % sets both header and footer to nothing +\renewcommand{\headrulewidth}{0pt} +\fancyfoot[LE,RO]{\thepage} +% Ensure copyright on titlepage (article style) and chapter pages (book style) +\fancypagestyle{plain}{ + \fancyhf{} + \fancyfoot[C]{{\footnotesize \copyright\ 1999-2018, "Data Analysis and Machine Learning FYS-STK3155/FYS4155":"http://www.uio.no/studier/emner/matnat/fys/FYS3155/index-eng.html". Released under CC Attribution-NonCommercial 4.0 license}} +% \renewcommand{\footrulewidth}{0mm} + \renewcommand{\headrulewidth}{0mm} +} +% Ensure copyright on titlepages with \thispagestyle{empty} +\fancypagestyle{empty}{ + \fancyhf{} + \fancyfoot[C]{{\footnotesize \copyright\ 1999-2018, "Data Analysis and Machine Learning FYS-STK3155/FYS4155":"http://www.uio.no/studier/emner/matnat/fys/FYS3155/index-eng.html". Released under CC Attribution-NonCommercial 4.0 license}} + \renewcommand{\footrulewidth}{0mm} + \renewcommand{\headrulewidth}{0mm} +} + +\pagestyle{fancy} + + +% prevent orhpans and widows +\clubpenalty = 10000 +\widowpenalty = 10000 + +% --- end of standard preamble for documents --- + + +% insert custom LaTeX commands... + +\raggedbottom +\makeindex +\usepackage[totoc]{idxlayout} % for index in the toc +\usepackage[nottoc]{tocbibind} % for references/bibliography in the toc + +%-------------------- end preamble ---------------------- + +\begin{document} + +% matching end for #ifdef PREAMBLE + +\newcommand{\exercisesection}[1]{\subsection*{#1}} + + +% ------------------- main content ---------------------- + + + +% ----------------- title ------------------------- + +\thispagestyle{empty} + +\begin{center} +{\LARGE\bf +\begin{spacing}{1.25} +Project 2 on Machine Learning, deadline November 12 +\end{spacing} +} +\end{center} + +% ----------------- author(s) ------------------------- + +\begin{center} +{\bf \href{{http://www.uio.no/studier/emner/matnat/fys/FYS3155/index-eng.html}}{Data Analysis and Machine Learning FYS-STK3155/FYS4155}} +\end{center} + + \begin{center} +% List of all institutions: +\centerline{{\small Department of Physics, University of Oslo, Norway}} +\end{center} + +% ----------------- end author(s) ------------------------- + +% --- begin date --- +\begin{center} +Nov 2, 2018 +\end{center} +% --- end date --- + +\vspace{1cm} + + +\subsection*{Classification and Regression, from linear and logistic regression to neural networks} + +The main aim of this project is to study both classification and +regression problems, starting with the regression algorithms studied +in project 1. We will include logistic regresion for classification +problems and write our own multilayer perceptron code for studying +both regression and classification problems. The codes developed in +project 1, including bootstrap and/or cross-validation as well as the +computation of the mean-squared error and the R2 score function can +also be utilized (and included in logistic regression and the neural +network codes) in the present analysis. + +We will use the so-called Ising model for our training data and will +focus on supervised training. We will follow closely the recent +article of \href{{https://arxiv.org/abs/1803.08823}}{Mehta et al, arXiv +1803.08823}. This article stands +out as an excellent review on machine learning (ML) algorithms. +The added benefit is that each figure and +model presented in \href{{https://physics.bu.edu/~pankajm/MLnotebooks.html}}{this article is accompanied by its jupyter +notebook}. This +means that we can start using these and compare with our own results. +They provide also the data set for the regression and classification +analysis that we will explore. In this sense, with their available +notebooks, it makes life easier since we can compare our own codes +with their codes. + + + + +With the abovementioned configurations we will determine, using first +various regression methods, the value of the coupling constant for the +energy of the one-dimensional Ising model. Thereafter, we will use the +two-dimensional data, but now computed at different temperatures, in +order to classify the phase of the Ising model. Below the critical +temperature, the system will be in a so-called ferromagnetic +phase. Close to the critical temperature, the final magnetization +becomes smaller and smaller in absolute value while above the critical +temperature, the net magnetization is zero. This classification case, +that is the two-dimensional Ising model, will be studied using +logistic regression and deep neural networks. The aim is to develop +your own logistic regression code for the classification of the phases +(this is a binary model) and your multilayer perceptron code for the +classification and regression case. You can compare your own results with those obtained +using \textbf{scikit-learn} or \textbf{tensorflow} or other Python packages such as \textbf{keras} or other. + + +Feel free to use the notebooks to benchmark your code. If you wish to +write your own C++ or Fortran program for say a multilayer neural network +model and a logistic regression model, please feel free to do so. + + +\paragraph{Part a): Producing the data for the one-dimensional Ising model.} +The model we will employ in our studies is the so-called \href{{https://en.wikipedia.org/wiki/Ising_model}}{Ising +model}. Together with +models like the \href{{https://en.wikipedia.org/wiki/Potts_model}}{Potts +model} and similar +so-called lattice models, the Ising model has been widely studied in +mathematics (in statistics in particular), physics, \href{{https://journals.aps.org/pre/abstract/10.1103/PhysRevE.93.062402}}{life +science}, +chemistry and even in the \href{{https://www.springer.com/gp/book/9781461420316}}{social sciences in order to model social +behavior}. It is a +simple binary value system where the variables of the model (spins often in +physics) can take two values only, for example $\pm 1$ or $0$ and $1$. +The system exhibits a phase transition in two or higher dimensions and +the first person to find the analytical expressions for various +expectation values was the Norwegian chemist \href{{https://en.wikipedia.org/wiki/Lars_Onsager}}{Lars +Onsager} (Nobel prize in +chemistry) after a tour de force mathematics exercise. + +In our discussions here we will stay with a physicist's approach and +call the variables for spin. You could replace this with any other +type of binary variables, ranging from a two political parties to blue +and red spheres. In its simplest form we define the energy of the +system as + +\begin{equation*} + E=-J\sum_{}^{N}s_ks_l, +\end{equation*} +with $s_k=\pm 1$, $N$ is the total number of spins, +$J$ is a coupling constant expressing the strength of the interaction +between neighboring spins. + +The symbol $$ indicates that we sum over nearest +neighbors only. +Notice that for $J>0$ it is energetically favorable for neighboring spins +to be aligned. This feature leads to, at low enough temperatures, +a cooperative phenomenon called spontaneous magnetization. That is, +through interactions between nearest neighbors, a given magnetic +moment can influence the alignment of spins that are separated +from the given spin by a macroscopic distance. These long range correlations +between spins are associated with a long-range order in which +the lattice has a net magnetization in the absence of a magnetic field. + + + +We start by considering the one-dimensional Ising model with nearest neighbor interactions. This model does not exhibit any phase transition. + +Consider the 1D Ising model with nearest-neighbor interactions + +\begin{equation*} + E[\hat{s}]=-J\sum_{j=1}^{N}s_{j}s_{j+1}, +\end{equation*} + +on a chain of length $N$ with so-called periodic boundary conditions and $S_j=\pm 1$ Ising spin variables. +In one dimension, this model has no phase transition at finite temperature. + +In the Python code below we generate, with a coupling coefficient set to $J=1$, a large number of spin configurations say $10000$ as shown in the code below. +It means that our data will be a set of $i=1\ldots n$ points of the form +$\{(E[\boldsymbol{s}^i],\boldsymbol{s}^i)\}$. +Our task is to find the value of $J$ from the data set using linear regression. + +Here is the Python code you need to generate the training data, see +also the \href{{https://physics.bu.edu/~pankajm/ML-Notebooks/HTML/NB_CVI-linreg_ising.html}}{notebook of Mehta et +al}. + +\begin{print} +import numpy as np +import scipy.sparse as sp +np.random.seed(12) + +import warnings +#Comment this to turn on warnings +warnings.filterwarnings('ignore') + +### define Ising model aprams +# system size +L=40 + +# create 10000 random Ising states +states=np.random.choice([-1, 1], size=(10000,L)) + +def ising_energies(states,L): + """ + This function calculates the energies of the states in the nn Ising Hamiltonian + """ + J=np.zeros((L,L),) + for i in range(L): + J[i,(i+1)%L]-=1.0 + # compute energies + E = np.einsum('...i,ij,...j->...',states,J,states) + + return E +# calculate Ising energies +energies=ising_energies(states,L) +\end{print} + +We can now recast the problem as a linear regression model using our codes from project 1. +The way we are going to build our model mimicks the way we could think of finding say the gravitional constant for the graviational force between two planets. +In the absence of any prior knowledge, one sensible choice is the all-to-all Ising model + +\[ +E_\mathrm{model}[\boldsymbol{s}^i] = - \sum_{j=1}^N \sum_{k=1}^N J_{j,k}s_{j}^is_{k}^i. +\] + +Here $i$ represents a particular spin configuration (one of the possible $n$ configurations we generated with the code above). + +This model is uniquely defined by the non-local coupling strengths $J_{jk}$ which we want to learn. +The model is linear in $\mathbf{J}$ which makes it possible to use linear regression. + +To apply linear regression, we recast this model in the form +\[ +E_\mathrm{model}^i \equiv \mathbf{X}^i \cdot \mathbf{J}, +\] + +where the vectors $\mathbf{X}^i$ represent all two-body interactions +$\{s_{j}^is_{k}^i \}_{j,k=1}^N$, and the index $i$ runs over the +samples in the data set. To make the analogy complete, we can also +represent the dot product by a single index $p = \{j,k\}$, +i.e.~$\mathbf{X}^i \cdot \mathbf{J}=X^i_pJ_p$. Note that the +regression model does not include the minus sign, so we expect to +learn negative $J$'s. + +With these preliminaries, we are now ready to reutilize our codes from project 1. + + +\paragraph{Part b): Estimating the coupling constant of the one-dimensional Ising model using linear regression.} +We start with the one-dimensional Ising model and use the data we have +generated with $J=1$ in the previous point. Use linear regression, +Lasso and Ridge regression as done in project 1. You can compare your +results with those of \href{{https://physics.bu.edu/~pankajm/ML-Notebooks/HTML/NB_CVI-linreg_ising.html}}{Mehta +et al.}. +Make sure it is the 1D data which is used. + +Discuss the methods and how they perform in computing the coupling +constant $J$ and include a bias-variance analysis using either +cross-validation or bootstrap. Discuss also the mean squared error and +the $R2$ score as measures to assess your model. + +Give a critical analysis of your results. + + +\paragraph{Part c): Determine the phase of the two-dimensional Ising model.} +We switch now to binary classification methods and use logistic +regression to define the phases of the Ising model. This means that we switch to the two-dimensional Ising model +and use the data sets generated by \href{{https://physics.bu.edu/~pankajm/ML-Review-Datasets/isingMC/}}{Mehta et al} +These energies and their corresponding spin orientation configurations +represent then your data. We will use a fixed lattice of $L\times L = +40 \times 40$ spins in two dimensions. The link above contains data for several temperatures. +The theoretical critical temperature for a phase transition is $T_C\approx 2.269$ in units of energy. +However, for a finite lattice the results representing the critical temperature are slightly higher ($T_C \approx 2.3$). + +Our goal here, using logistic regression, is to train our model to +predict the phase of a sample given the spin configuration, whether it +represents a state above the critical temperature or below. The +configurations representing states below the critical temperature are +called ordered states (the spins tend to point in one direction, +resulting in a net magnetic moment) while those above the critical +temperature are called disordered. Since a finite lattice like this +does not exhibit a clear sign of a phase transition we will mainly +stay with either orderer or disordered phases. You could include the +critical phase if you want. + + +Your aim here is thus to read in these data (use the examples from +\href{{https://physics.bu.edu/~pankajm/ML-Notebooks/HTML/NB_CVII-logreg_ising.html}}{Mehta et +al}) +and write your own code for doing logistic regression, see the lecture +notes on \href{{https://compphysics.github.io/MachineLearning/doc/pub/LogReg/html/LogReg-bs.html}}{logistic +regression}. + + +In this case, to evaluate the model, we will use the so-called accuracy score +instead of the bootstrap or cross-validation as done in the standard linear regression part discussed in b). Examples of how to define the accuracy score can be found under the neural network slides, see for example the \href{{https://compphysics.github.io/MachineLearning/doc/pub/NeuralNet/html/._NeuralNet-bs047.html}}{slides here}. + +To measure the performance of our network we evaluate how well it does +it data it has never seen before, i.e.~the test data. We measure the +performance of the network using the \emph{accuracy} score. The accuracy +is as you would expect just the number of images correctly labeled +divided by the total number of images. A perfect classifier will have +an accuracy score of $1$. + +\[ +\text{Accuracy} = \frac{\sum_{i=1}^n I(t_i = y_i)}{n} , +\] + +where $I$ is the indicator function, $1$ if $t_i = y_i$ and $0$ +otherwise, where $t_i$ represents the target and $y_i$ the outputs. + + + +In order to find the optimal parameters of your logistic regressor you should +include a gradient descent solver, as discussed in the \href{{https://compphysics.github.io/MachineLearning/doc/pub/Splines/html/Splines-bs.html}}{gradient +descent +lectures}. +Since we don't have so many data points, you may just code the +standard gradient descent with a given learning rate, or even attempt +to use the Newton-Raphson method. Alternatively, it may be useful for +the next part on neural networks to implement a stochastic gradient +descent with and without mini-batches. Stochastic gradient with mini-batches may give the best results. You could finally compare your code with the output from \textbf{scikit-learn}'s toolbox for +optimization methods applied to logistic regression. + + +The notebook of \href{{https://physics.bu.edu/~pankajm/ML-Notebooks/HTML/NB_CVII-logreg_ising.html}}{Mehta et al} is highly recommended in order to benchmark your code and results. + +\paragraph{Part d): Regression analysis of the one-dimensional Ising model using neural networks.} +Your aim now, and this is the central part of this project, is to +write to your own multilayer perceptron model implementing the back +propagation algorithm discussed in the \href{{https://compphysics.github.io/MachineLearning/doc/pub/NeuralNet/html/NeuralNet-bs.html}}{lecture +slides}. We +start with the regression case discussed in parts a) and b) but train +now the network to find the optimal weights and biases. You are free +to use the codes in the above lecture slides as starting points. + +Train your network and compare the results with those from your linear regression code. +You can test your results against a similar code using _scikit_learn_ (see the examples in the above lecture notes) or \textbf{tensorflow/keras}. + + +A useful reference on the back progagation algorithm is \href{{http://neuralnetworksanddeeplearning.com/}}{Nielsen's book}. It is an excellent read. + +\paragraph{Part e): Classifying the Ising model phase using neural networks.} +Finally, change now your cost function to the $log$ cross-entropy classification cost function for the case discussed in part c). Train your network again and +compare the results with those from your logistic regression code i c). +Here again you can compare your results with those of \href{{https://physics.bu.edu/~pankajm/ML-Notebooks/HTML/NB_CIX-DNN_ising_TFlow.html}}{Mehta et al}. There they used \textbf{tensorflow} to classify the phases. + + + +\paragraph{Part f) Critical evaluation of the various algorithms.} +After all these glorious calculations, you should now summarize the various algorithms and come with a critical evaluation of their pros and cons. Which algorithm works best for the regression case and which is best for the classification case. These codes will also be part of your final project 3, but now applied to other data sets. + + + + +\subsection*{Background literature} + +\begin{enumerate} +\item The text of Michael Nielsen is highly recommended, see \href{{http://neuralnetworksanddeeplearning.com/}}{Nielsen's book}. It is an excellent read. + +\item The textbook of \href{{https://www.springer.com/gp/book/9780387848570}}{Trevor Hastie, Robert Tibshirani, Jerome H. Friedman, The Elements of Statistical Learning, Springer}, chapters 3 and 7 are the most relevant ones for the analysis here. + +\item \href{{https://arxiv.org/abs/1803.08823}}{Mehta et al, arXiv 1803.08823}, \emph{A high-bias, low-variance introduction to Machine Learning for physicists}, ArXiv:1803.08823. +\end{enumerate} + +\noindent +If you wish to read more about the Ising model and statistical physics here are three suggestions. + +\begin{enumerate} +\item \href{{http://www.worldscientific.com/worldscibooks/10.1142/5660}}{M. Plischke and B. Bergersen}, \emph{Equilibrium Statistical Physics}, World Scientific, see chapters 5 and 6. + +\item \href{{http://www.cambridge.org/no/academic/subjects/physics/computational-science-and-modelling/guide-monte-carlo-simulations-statistical-physics-4th-edition?format=HB}}{D. P. Landau and K. Binder}, \emph{A Guide to Monte Carlo Simulations in Statistical Physics}, Cambridge, see chapters 2,3 and 4. + +\item \href{{https://global.oup.com/academic/product/monte-carlo-methods-in-statistical-physics-9780198517979?cc=no&lang=en&}}{M. E. J. Newman and T. Barkema}, \emph{Monte Carlo Methods in Statistical Physics}, Oxford, see chapters 3 and 4. +\end{enumerate} + +\noindent +\subsection*{Introduction to numerical projects} + +Here follows a brief recipe and recommendation on how to write a report for each +project. + +\begin{itemize} + \item Give a short description of the nature of the problem and the eventual numerical methods you have used. + + \item Describe the algorithm you have used and/or developed. Here you may find it convenient to use pseudocoding. In many cases you can describe the algorithm in the program itself. + + \item Include the source code of your program. Comment your program properly. + + \item If possible, try to find analytic solutions, or known limits in order to test your program when developing the code. + + \item Include your results either in figure form or in a table. Remember to label your results. All tables and figures should have relevant captions and labels on the axes. + + \item Try to evaluate the reliabilty and numerical stability/precision of your results. If possible, include a qualitative and/or quantitative discussion of the numerical stability, eventual loss of precision etc. + + \item Try to give an interpretation of you results in your answers to the problems. + + \item Critique: if possible include your comments and reflections about the exercise, whether you felt you learnt something, ideas for improvements and other thoughts you've made when solving the exercise. We wish to keep this course at the interactive level and your comments can help us improve it. + + \item Try to establish a practice where you log your work at the computerlab. You may find such a logbook very handy at later stages in your work, especially when you don't properly remember what a previous test version of your program did. Here you could also record the time spent on solving the exercise, various algorithms you may have tested or other topics which you feel worthy of mentioning. +\end{itemize} + +\noindent +\subsection*{Format for electronic delivery of report and programs} + +The preferred format for the report is a PDF file. You can also use DOC or postscript formats or as an ipython notebook file. As programming language we prefer that you choose between C/C++, Fortran2008 or Python. The following prescription should be followed when preparing the report: + +\begin{itemize} + \item Use Devilry to hand in your projects, log in at \href{{http://devilry.ifi.uio.no}}{\nolinkurl{http://devilry.ifi.uio.no}} with your normal UiO username and password and choose either 'fysstk3155' or 'fysstk4155'. There you can load up the files within the deadline. + + \item Upload \textbf{only} the report file! For the source code file(s) you have developed please provide us with your link to your github domain. The report file should include all of your discussions and a list of the codes you have developed. Do not include library files which are available at the course homepage, unless you have made specific changes to them. + + \item In your git repository, please include a folder which contains selected results. These can be in the form of output from your code for a selected set of runs and input parameters. + + \item In this and all later projects, you should include tests (for example unit tests) of your code(s). + + \item Comments from us on your projects, approval or not, corrections to be made etc can be found under your Devilry domain and are only visible to you and the teachers of the course. +\end{itemize} + +\noindent +Finally, +we encourage you to collaborate. Optimal working groups consist of +2-3 students. You can then hand in a common report. + + + +\subsection*{Software and needed installations} + +If you have Python installed (we recommend Python3) and you feel pretty familiar with installing different packages, +we recommend that you install the following Python packages via \textbf{pip} as +\begin{enumerate} +\item pip install numpy scipy matplotlib ipython scikit-learn tensorflow sympy pandas pillow +\end{enumerate} + +\noindent +For Python3, replace \textbf{pip} with \textbf{pip3}. + +See below for a discussion of \textbf{tensorflow} and \textbf{scikit-learn}. + +For OSX users we recommend also, after having installed Xcode, to install \textbf{brew}. Brew allows +for a seamless installation of additional software via for example +\begin{enumerate} +\item brew install python3 +\end{enumerate} + +\noindent +For Linux users, with its variety of distributions like for example the widely popular Ubuntu distribution +you can use \textbf{pip} as well and simply install Python as +\begin{enumerate} +\item sudo apt-get install python3 (or python for python2.7) +\end{enumerate} + +\noindent +etc etc. + +If you don't want to install various Python packages with their dependencies separately, we recommend two widely used distrubutions which set up all relevant dependencies for Python, namely +\begin{enumerate} +\item \href{{https://docs.anaconda.com/}}{Anaconda} Anaconda is an open source distribution of the Python and R programming languages for large-scale data processing, predictive analytics, and scientific computing, that aims to simplify package management and deployment. Package versions are managed by the package management system \textbf{conda} + +\item \href{{https://www.enthought.com/product/canopy/}}{Enthought canopy} is a Python distribution for scientific and analytic computing distribution and analysis environment, available for free and under a commercial license. +\end{enumerate} + +\noindent +Popular software packages written in Python for ML are + +\begin{itemize} +\item \href{{http://scikit-learn.org/stable/}}{Scikit-learn}, + +\item \href{{https://www.tensorflow.org/}}{Tensorflow}, + +\item \href{{http://pytorch.org/}}{PyTorch} and + +\item \href{{https://keras.io/}}{Keras}. +\end{itemize} + +\noindent +These are all freely available at their respective GitHub sites. They +encompass communities of developers in the thousands or more. And the number +of code developers and contributors keeps increasing. + + + + + + + + + + + + + +% ------------------- end of main content --------------- + +\end{document} + diff --git a/doc/Projects/2018/hw1/html/._hw1-bs000.html b/doc/Projects/2018/hw1/html/._hw1-bs000.html new file mode 100644 index 000000000..2b32af321 --- /dev/null +++ b/doc/Projects/2018/hw1/html/._hw1-bs000.html @@ -0,0 +1,285 @@ + + + + + + + +Homework 1 + + + + + + + + + + + + + + + + + + + + + + + +
+
+ + Homework 1 +
+ +
+ +
+
+ + +
+ +

 

 

 

+ + + + + + +
+

Homework 1

+ +

+ + +

+Data Analysis and Machine Learning FYS-STK3155/FYS4155 +
+ +

+ + +

Department of Physics, University of Oslo, Norway
+
+

+

Aug 30, 2018

+
+

+

+ +

Exercise 1

+ +

+The first exercise here is of a mere technical art. We want you to have + +

    +
  • git as a version control software and to establish a user account on a provider like GitHub. Other providers like GitLab etc are equally fine. You can also use the University of Oslo GitHub facilities.
    • +
  • Install various Python packages
    • +
+ +We will make extensive use of Python as programming language and its +myriad of available libraries. You will find +IPython/Jupyter notebooks invaluable in your work. You can run R +codes in the Jupyter/IPython notebooks, with the immediate benefit of +visualizing your data. You can also use compiled languages like C++, +Rust, Fortran etc if you prefer. The focus in these lectures will be +on Python. + +

+If you have Python installed (we recommend Python3) and you feel +pretty familiar with installing different packages, we recommend that +you install the following Python packages via pip as + +

    +
  1. pip install numpy scipy matplotlib ipython scikit-learn sympy pandas pillow
    2. +
+ +For Tensorflow, we recommend following the instructions in the text of +Aurelien Geron, Hands‑On Machine Learning with Scikit‑Learn and TensorFlow, O'Reilly + +

+We will come back to tensorflow later. + +

+For Python3, replace pip with pip3. + +

+For OSX users we recommend, after having installed Xcode, to +install brew. Brew allows for a seamless installation of additional +software via for example + +

    +
  1. brew install python3
    2. +
+ +For Linux users, with its variety of distributions like for example the widely popular Ubuntu distribution, +you can use pip as well and simply install Python as + +
    +
  1. sudo apt-get install python3 (or python for pyhton2.7)
    2. +
+ +If you don't want to perform these operations separately and venture +into the hassle of exploring how to set up dependencies and paths, we +recommend two widely used distrubutions which set up all relevant +dependencies for Python, namely + + + +which is an open source +distribution of the Python and R programming languages for large-scale +data processing, predictive analytics, and scientific computing, that +aims to simplify package management and deployment. Package versions +are managed by the package management system conda. + + + +is a Python +distribution for scientific and analytic computing distribution and +analysis environment, available for free and under a commercial +license. + +

+We recommend using Anaconda. + +

Exercise 2

+ +

+We will generate our own dataset for a function \( y(x) \) where \( x \in [0,1] \) and defined by random numbers computed with the uniform distribution. The function \( y \) is a quadratic polynomial in \( x \) with added stochastic noise according to the normal distribution \( \cal {N}(0,1) \). +The following simple Python instructions define our \( x \) and \( y \) values (with 100 data points). +

+ + +

x = np.random.rand(100,1)
+y = 5*x*x+0.1*np.random.randn(100,1)
+
+
    +
  1. Write your own code (following the examples under the regression slides) for computing the parametrization of the data set fitting a second-order polynomial.
    2. +
  2. Use thereafter scikit-learn (see again the examples in the regression slides) and compare with your own code.
    3. +
  3. Using scikit-learn, compute also the mean square error, a risk metric corresponding to the expected value of the squared (quadratic) error defined as
    4. +
+ +$$ MSE(\hat{y},\hat{\tilde{y}}) = \frac{1}{n} +\sum_{i=0}^{n-1}(y_i-\tilde{y}_i)^2, +$$ + +and the \( R^2 \) score function. +If \( \tilde{\hat{y}}_i \) is the predicted value of the \( i-th \) sample and \( y_i \) is the corresponding true value, then the score \( R^2 \) is defined as +$$ +R^2(\hat{y}, \tilde{\hat{y}}) = 1 - \frac{\sum_{i=0}^{n - 1} (y_i - \tilde{y}_i)^2}{\sum_{i=0}^{n - 1} (y_i - \bar{y})^2}, +$$ + +where we have defined the mean value of \( \hat{y} \) as +$$ +\bar{y} = \frac{1}{n} \sum_{i=0}^{n - 1} y_i. +$$ + +You can use the functionality included in scikit-learn. If you feel for it, you can use your own program and define functions which compute the above two functions. +Discuss the meaning of these results. Try also to vary the coefficient in front of the added stochastic noise term and discuss the quality of the fits. + +

Exercise 3, variance of the parameters \( \beta \) in linear regression

+ +

+Show that the variance of the parameters \( \beta \) in the linear regression method (chapter 3, equation (3.8) of Trevor Hastie, Robert Tibshirani, Jerome H. Friedman, The Elements of Statistical Learning, Springer) is given as + +$$ +\mathrm{Var}(\hat{\beta}) = \left(\hat{X}^T\hat{X}\right)^{-1}\sigma^2, +$$ + +with +$$ +\sigma^2 = \frac{1}{N-p-1}\sum_{i=1}^{N} (y_i-\tilde{y}_i)^2, +$$ + +where we have assumed that we fit a function of degree \( p-1 \) (for example a polynomial in \( x \)). + +

+ +

    +
  • 1
    • +
+ + +
+ + + + + + + +
+ © 1999-2018, "Data Analysis and Machine Learning FYS-STK3155/FYS4155":"http://www.uio.no/studier/emner/matnat/fys/FYS3155/index-eng.html". Released under CC Attribution-NonCommercial 4.0 license +
+ + + + + + diff --git a/doc/Projects/2018/hw2/html/._hw2-bs000.html b/doc/Projects/2018/hw2/html/._hw2-bs000.html new file mode 100644 index 000000000..2cac701d8 --- /dev/null +++ b/doc/Projects/2018/hw2/html/._hw2-bs000.html @@ -0,0 +1,211 @@ + + + + + + + +Homework 2 + + + + + + + + + + + + + + + + + + + + + + + +
+
+ + Homework 2 +
+ +
+ +
+
+ + +
+ +

 

 

 

+ + + + + + +
+

Homework 2

+ +

+ + +

+Data Analysis and Machine Learning FYS-STK3155/FYS4155 +
+ +

+ + +

Department of Physics, University of Oslo, Norway
+
+

+

Sep 5, 2018

+
+

+

+ +

Exercise 4

+ +

+This exercise is a continuation of exercise 2 from homework 1. We will +use the same function to generate our data set, still staying with a +simple function \( y(x) \) which we want to fit using linear regression, +but now extending the analysis to include the Ridge and the Lasso +regression methods. You can use the code under the Regression as an example on how to use the Ridge and the Lasso methods, see the regression slides). + +

+We will thus again generate our own dataset for a function \( y(x) \) where +\( x \in [0,1] \) and defined by random numbers computed with the uniform +distribution. The function \( y \) is a quadratic polynomial in \( x \) with +added stochastic noise according to the normal distribution \( \cal{N}(0,1) \). + +

+The following simple Python instructions define our \( x \) and \( y \) values (with 100 data points). +

+ + +

x = np.random.rand(100,1)
+y = 5*x*x+0.1*np.random.randn(100,1)
+
+
    +
  1. Write your own code for the Ridge method (see chapter 3.4 of Hastie et al., equations (3.43) and (3.44)) and compute the parametrization for different values of \( \lambda \). Compare and analyze your results with those from exercise 2. Study the dependence on \( \lambda \) while also varying the strength of the noise in your expression for \( y(x) \).
    2. +
  2. Repeat the above but using the functionality of scikit-learn. Compare your code with the results from scikit-learn. Remember to run with the same random numbers for generating \( x \) and \( y \).
    3. +
  3. Our next step is to study the variance of the parameters \( \beta_1 \) and \( \beta_2 \) (assuming that we are parametrizing our function with a second-order polynomial. We will use standard linear regression and the Ridge regression. You can now opt for either writing your own function that calculates the variance of these paramaters (recall that this is equal to the diagonal elements of the matrix \( (\hat{X}^T\hat{X})+\lambda\hat{I})^{-1} \)) or use the functionality of scikit-learn and compute their variances. Discuss the results of these variances as functions of \( \lambda \). In particular, try to link your discussion with the discussion in Hastie et al. and their figure 3.11.
    4. +
  4. Repeat the previous step but add now the Lasso method, see equation (3.53) of Hastie et al.. Discuss your results and compare with standard regression and the Ridge regression results. You can write your own code or use the functionality of scikit-learn.
    5. +
  5. Finally, using scikit-learn or your own code, compute also the mean square error, a risk metric corresponding to the expected value of the squared (quadratic) error defined as
    6. +
+ +$$ MSE(\hat{y},\hat{\tilde{y}}) = \frac{1}{n} +\sum_{i=0}^{n-1}(y_i-\tilde{y}_i)^2, +$$ + +and the \( R^2 \) score function. +If \( \tilde{\hat{y}}_i \) is the predicted value of the \( i-th \) sample and \( y_i \) is the corresponding true value, then the score \( R^2 \) is defined as +$$ +R^2(\hat{y}, \tilde{\hat{y}}) = 1 - \frac{\sum_{i=0}^{n - 1} (y_i - \tilde{y}_i)^2}{\sum_{i=0}^{n - 1} (y_i - \bar{y})^2}, +$$ + +where we have defined the mean value of \( \hat{y} \) as +$$ +\bar{y} = \frac{1}{n} \sum_{i=0}^{n - 1} y_i. +$$ + +Discuss these quantities as functions of the variable \( \lambda \) in the Ridge and Lasso regression methods. + +

Exercise 5

+ +

+Using the singular value decomposition, show that the variance of the direction vector +\( \hat{z}_i=\hat{X}\hat{v}_i=\hat{u}_1d_1 \) is equal to (equation (3.49) of Hastie et al.) +$$ +\mathrm{Var}(\hat{z}_i)=\frac{d_i^2}{N}, +$$ + +where \( d_i \) are the singular values of the matrix \( \hat{X} \). In Hastie et al, the matrix elements of \( X \) are centered. The consequence is that the mean values of for example \( \hat{u}_i \) are zero. + +

+Give an interpretation of these results, in particular in connection with the variance of the coefficients you obtained in the previous exercise. + +

+ +

    +
  • 1
    • +
+ + +
+ + + + + + + +
+ © 1999-2018, "Data Analysis and Machine Learning FYS-STK3155/FYS4155":"http://www.uio.no/studier/emner/matnat/fys/FYS3155/index-eng.html". Released under CC Attribution-NonCommercial 4.0 license +
+ + + + + + diff --git a/doc/Textbooks/.DS_Store b/doc/Textbooks/.DS_Store new file mode 100644 index 0000000000000000000000000000000000000000..5113b495cb415f0799e6b41e7211536fcbd9370d GIT binary patch literal 6148 zcmeHKOG*Pl5PhuyMMbjlx0_`)ict~7%^2b$U_i_T%uf@DlaNUScZ1*^yn;vZJRU$_ zbvGfI8Mh)@6{=o$O;vTz8@f9Uz*MH4GEfAtN);@u(d39c7tP3$_pA_&tr4S-CL$bQ zl!;b{GN26nHU?zvZlJ|g`-Cp$*KY@RwZX968Vp;EkqzRUTKcrYmiW3DP#>rfCd`g$ zn^al<+5ERueu=w<r<_vqBV}8YtRK;yv+vpfKoY5mLn0JO# z+C!Y+2xb0`^K04f@;dijHZ70xRbuLxj7ckZ9(d + + + + + + + +Data Analysis and Machine Learning: Autoencoders + + + + + + + + + + + + + + + + +
+
+ + Data Analysis and Machine Learning: Autoencoders +
+ +
+ +
+
+ + +
+ +

 

 

 

+ + + + +

Simple examples of Autoencoders

+ +

+ +

+ + +
+ + + + + + + +
+ +
+ + + + + + diff --git a/doc/pub/Autoencoders/html/reveal.js/plugin/leap/leap.js b/doc/pub/Autoencoders/html/reveal.js/plugin/leap/leap.js new file mode 100644 index 000000000..48084ffb0 --- /dev/null +++ b/doc/pub/Autoencoders/html/reveal.js/plugin/leap/leap.js @@ -0,0 +1,159 @@ +/* + * Copyright (c) 2013, Leap Motion, Inc. + * All rights reserved. + * + * Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: + * + * Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. + * Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + * + * Version 0.2.0 - http://js.leapmotion.com/0.2.0/leap.min.js + * Grab latest versions from http://js.leapmotion.com/ + */ + +!function(e,t,n){function i(n,s){if(!t[n]){if(!e[n]){var o=typeof require=="function"&&require;if(!s&&o)return o(n,!0);if(r)return r(n,!0);throw new Error("Cannot find module '"+n+"'")}var u=t[n]={exports:{}};e[n][0].call(u.exports,function(t){var r=e[n][1][t];return i(r?r:t)},u,u.exports)}return t[n].exports}var r=typeof require=="function"&&require;for(var s=0;s=this.size)return undefined;if(i>=this._buf.length)return undefined;return this._buf[(this.pos-i-1)%this.size]};CircularBuffer.prototype.push=function(o){this._buf[this.pos%this.size]=o;return this.pos++}},{}],3:[function(require,module,exports){var Connection=module.exports=require("./base_connection");Connection.prototype.setupSocket=function(){var connection=this;var socket=new WebSocket(this.getUrl());socket.onopen=function(){connection.handleOpen()};socket.onmessage=function(message){connection.handleData(message.data)};socket.onclose=function(){connection.handleClose()};return socket};Connection.prototype.startHeartbeat=function(){if(!this.protocol.sendHeartbeat||this.heartbeatTimer)return;var connection=this;var propertyName=null;if(typeof document.hidden!=="undefined"){propertyName="hidden"}else if(typeof document.mozHidden!=="undefined"){propertyName="mozHidden"}else if(typeof document.msHidden!=="undefined"){propertyName="msHidden"}else if(typeof document.webkitHidden!=="undefined"){propertyName="webkitHidden"}else{propertyName=undefined}var windowVisible=true;var focusListener=window.addEventListener("focus",function(e){windowVisible=true});var blurListener=window.addEventListener("blur",function(e){windowVisible=false});this.on("disconnect",function(){if(connection.heartbeatTimer){clearTimeout(connection.heartbeatTimer);delete connection.heartbeatTimer}window.removeEventListener(focusListener);window.removeEventListener(blurListener)});this.heartbeatTimer=setInterval(function(){var isVisible=propertyName===undefined?true:document[propertyName]===false;if(isVisible&&windowVisible){connection.sendHeartbeat()}else{connection.setHeartbeatState(false)}},this.opts.heartbeatInterval)}},{"./base_connection":1}],4:[function(require,module,exports){!function(process){var Frame=require("./frame"),CircularBuffer=require("./circular_buffer"),Pipeline=require("./pipeline"),EventEmitter=require("events").EventEmitter,gestureListener=require("./gesture").gestureListener,_=require("underscore");var Controller=module.exports=function(opts){var inNode=typeof process!=="undefined"&&process.title==="node";opts=_.defaults(opts||{},{inNode:inNode});this.inNode=opts.inNode;opts=_.defaults(opts||{},{frameEventName:this.useAnimationLoop()?"animationFrame":"deviceFrame",supressAnimationLoop:false});this.supressAnimationLoop=opts.supressAnimationLoop;this.frameEventName=opts.frameEventName;this.history=new CircularBuffer(200);this.lastFrame=Frame.Invalid;this.lastValidFrame=Frame.Invalid;this.lastConnectionFrame=Frame.Invalid;this.accumulatedGestures=[];if(opts.connectionType===undefined){this.connectionType=this.inBrowser()?require("./connection"):require("./node_connection")}else{this.connectionType=opts.connectionType}this.connection=new this.connectionType(opts);this.setupConnectionEvents()};Controller.prototype.gesture=function(type,cb){var creator=gestureListener(this,type);if(cb!==undefined){creator.stop(cb)}return creator};Controller.prototype.inBrowser=function(){return!this.inNode};Controller.prototype.useAnimationLoop=function(){return this.inBrowser()&&typeof chrome==="undefined"};Controller.prototype.connect=function(){var controller=this;if(this.connection.connect()&&this.inBrowser()&&!controller.supressAnimationLoop){var callback=function(){controller.emit("animationFrame",controller.lastConnectionFrame);window.requestAnimFrame(callback)};window.requestAnimFrame(callback)}};Controller.prototype.disconnect=function(){this.connection.disconnect()};Controller.prototype.frame=function(num){return this.history.get(num)||Frame.Invalid};Controller.prototype.loop=function(callback){switch(callback.length){case 1:this.on(this.frameEventName,callback);break;case 2:var controller=this;var scheduler=null;var immediateRunnerCallback=function(frame){callback(frame,function(){if(controller.lastFrame!=frame){immediateRunnerCallback(controller.lastFrame)}else{controller.once(controller.frameEventName,immediateRunnerCallback)}})};this.once(this.frameEventName,immediateRunnerCallback);break}this.connect()};Controller.prototype.addStep=function(step){if(!this.pipeline)this.pipeline=new Pipeline(this);this.pipeline.addStep(step)};Controller.prototype.processFrame=function(frame){if(frame.gestures){this.accumulatedGestures=this.accumulatedGestures.concat(frame.gestures)}if(this.pipeline){frame=this.pipeline.run(frame);if(!frame)frame=Frame.Invalid}this.lastConnectionFrame=frame;this.emit("deviceFrame",frame)};Controller.prototype.processFinishedFrame=function(frame){this.lastFrame=frame;if(frame.valid){this.lastValidFrame=frame}frame.controller=this;frame.historyIdx=this.history.push(frame);if(frame.gestures){frame.gestures=this.accumulatedGestures;this.accumulatedGestures=[];for(var gestureIdx=0;gestureIdx!=frame.gestures.length;gestureIdx++){this.emit("gesture",frame.gestures[gestureIdx],frame)}}this.emit("frame",frame)};Controller.prototype.setupConnectionEvents=function(){var controller=this;this.connection.on("frame",function(frame){controller.processFrame(frame)});this.on(this.frameEventName,function(frame){controller.processFinishedFrame(frame)});this.connection.on("disconnect",function(){controller.emit("disconnect")});this.connection.on("ready",function(){controller.emit("ready")});this.connection.on("connect",function(){controller.emit("connect")});this.connection.on("focus",function(){controller.emit("focus")});this.connection.on("blur",function(){controller.emit("blur")});this.connection.on("protocol",function(protocol){controller.emit("protocol",protocol)});this.connection.on("deviceConnect",function(evt){controller.emit(evt.state?"deviceConnected":"deviceDisconnected")})};_.extend(Controller.prototype,EventEmitter.prototype)}(require("__browserify_process"))},{"./circular_buffer":2,"./connection":3,"./frame":5,"./gesture":6,"./node_connection":16,"./pipeline":10,__browserify_process:18,events:17,underscore:20}],5:[function(require,module,exports){var Hand=require("./hand"),Pointable=require("./pointable"),createGesture=require("./gesture").createGesture,glMatrix=require("gl-matrix"),mat3=glMatrix.mat3,vec3=glMatrix.vec3,InteractionBox=require("./interaction_box"),_=require("underscore");var Frame=module.exports=function(data){this.valid=true;this.id=data.id;this.timestamp=data.timestamp;this.hands=[];this.handsMap={};this.pointables=[];this.tools=[];this.fingers=[];if(data.interactionBox){this.interactionBox=new InteractionBox(data.interactionBox)}this.gestures=[];this.pointablesMap={};this._translation=data.t;this._rotation=_.flatten(data.r);this._scaleFactor=data.s;this.data=data;this.type="frame";this.currentFrameRate=data.currentFrameRate;var handMap={};for(var handIdx=0,handCount=data.hands.length;handIdx!=handCount;handIdx++){var hand=new Hand(data.hands[handIdx]);hand.frame=this;this.hands.push(hand);this.handsMap[hand.id]=hand;handMap[hand.id]=handIdx}for(var pointableIdx=0,pointableCount=data.pointables.length;pointableIdx!=pointableCount;pointableIdx++){var pointable=new Pointable(data.pointables[pointableIdx]);pointable.frame=this;this.pointables.push(pointable);this.pointablesMap[pointable.id]=pointable;(pointable.tool?this.tools:this.fingers).push(pointable);if(pointable.handId!==undefined&&handMap.hasOwnProperty(pointable.handId)){var hand=this.hands[handMap[pointable.handId]];hand.pointables.push(pointable);(pointable.tool?hand.tools:hand.fingers).push(pointable)}}if(data.gestures){for(var gestureIdx=0,gestureCount=data.gestures.length;gestureIdx!=gestureCount;gestureIdx++){this.gestures.push(createGesture(data.gestures[gestureIdx]))}}};Frame.prototype.tool=function(id){var pointable=this.pointable(id);return pointable.tool?pointable:Pointable.Invalid};Frame.prototype.pointable=function(id){return this.pointablesMap[id]||Pointable.Invalid};Frame.prototype.finger=function(id){var pointable=this.pointable(id);return!pointable.tool?pointable:Pointable.Invalid};Frame.prototype.hand=function(id){return this.handsMap[id]||Hand.Invalid};Frame.prototype.rotationAngle=function(sinceFrame,axis){if(!this.valid||!sinceFrame.valid)return 0;var rot=this.rotationMatrix(sinceFrame);var cs=(rot[0]+rot[4]+rot[8]-1)*.5;var angle=Math.acos(cs);angle=isNaN(angle)?0:angle;if(axis!==undefined){var rotAxis=this.rotationAxis(sinceFrame);angle*=vec3.dot(rotAxis,vec3.normalize(vec3.create(),axis))}return angle};Frame.prototype.rotationAxis=function(sinceFrame){if(!this.valid||!sinceFrame.valid)return vec3.create();return vec3.normalize(vec3.create(),[this._rotation[7]-sinceFrame._rotation[5],this._rotation[2]-sinceFrame._rotation[6],this._rotation[3]-sinceFrame._rotation[1]])};Frame.prototype.rotationMatrix=function(sinceFrame){if(!this.valid||!sinceFrame.valid)return mat3.create();var transpose=mat3.transpose(mat3.create(),this._rotation);return mat3.multiply(mat3.create(),sinceFrame._rotation,transpose)};Frame.prototype.scaleFactor=function(sinceFrame){if(!this.valid||!sinceFrame.valid)return 1;return Math.exp(this._scaleFactor-sinceFrame._scaleFactor)};Frame.prototype.translation=function(sinceFrame){if(!this.valid||!sinceFrame.valid)return vec3.create();return vec3.subtract(vec3.create(),this._translation,sinceFrame._translation)};Frame.prototype.toString=function(){var str="Frame [ id:"+this.id+" | timestamp:"+this.timestamp+" | Hand count:("+this.hands.length+") | Pointable count:("+this.pointables.length+")";if(this.gestures)str+=" | Gesture count:("+this.gestures.length+")";str+=" ]";return str};Frame.prototype.dump=function(){var out="";out+="Frame Info:
";out+=this.toString();out+="

Hands:
";for(var handIdx=0,handCount=this.hands.length;handIdx!=handCount;handIdx++){out+=" "+this.hands[handIdx].toString()+"
"}out+="

Pointables:
";for(var pointableIdx=0,pointableCount=this.pointables.length;pointableIdx!=pointableCount;pointableIdx++){out+=" "+this.pointables[pointableIdx].toString()+"
"}if(this.gestures){out+="

Gestures:
";for(var gestureIdx=0,gestureCount=this.gestures.length;gestureIdx!=gestureCount;gestureIdx++){out+=" "+this.gestures[gestureIdx].toString()+"
"}}out+="

Raw JSON:
";out+=JSON.stringify(this.data);return out};Frame.Invalid={valid:false,hands:[],fingers:[],tools:[],gestures:[],pointables:[],pointable:function(){return Pointable.Invalid},finger:function(){return Pointable.Invalid},hand:function(){return Hand.Invalid},toString:function(){return"invalid frame"},dump:function(){return this.toString()},rotationAngle:function(){return 0},rotationMatrix:function(){return mat3.create()},rotationAxis:function(){return vec3.create()},scaleFactor:function(){return 1},translation:function(){return vec3.create()}}},{"./gesture":6,"./hand":7,"./interaction_box":9,"./pointable":11,"gl-matrix":19,underscore:20}],6:[function(require,module,exports){var glMatrix=require("gl-matrix"),vec3=glMatrix.vec3,EventEmitter=require("events").EventEmitter,_=require("underscore");var createGesture=exports.createGesture=function(data){var gesture;switch(data.type){case"circle":gesture=new CircleGesture(data);break;case"swipe":gesture=new SwipeGesture(data);break;case"screenTap":gesture=new ScreenTapGesture(data);break;case"keyTap":gesture=new KeyTapGesture(data);break;default:throw"unkown gesture type"}gesture.id=data.id;gesture.handIds=data.handIds;gesture.pointableIds=data.pointableIds;gesture.duration=data.duration;gesture.state=data.state;gesture.type=data.type;return gesture};var gestureListener=exports.gestureListener=function(controller,type){var handlers={};var gestureMap={};var gestureCreator=function(){var candidateGesture=gestureMap[gesture.id];if(candidateGesture!==undefined)gesture.update(gesture,frame);if(gesture.state=="start"||gesture.state=="stop"){if(type==gesture.type&&gestureMap[gesture.id]===undefined){gestureMap[gesture.id]=new Gesture(gesture,frame);gesture.update(gesture,frame)}if(gesture.state=="stop"){delete gestureMap[gesture.id]}}};controller.on("gesture",function(gesture,frame){if(gesture.type==type){if(gesture.state=="start"||gesture.state=="stop"){if(gestureMap[gesture.id]===undefined){var gestureTracker=new Gesture(gesture,frame);gestureMap[gesture.id]=gestureTracker;_.each(handlers,function(cb,name){gestureTracker.on(name,cb)})}}gestureMap[gesture.id].update(gesture,frame);if(gesture.state=="stop"){delete gestureMap[gesture.id]}}});var builder={start:function(cb){handlers["start"]=cb;return builder},stop:function(cb){handlers["stop"]=cb;return builder},complete:function(cb){handlers["stop"]=cb;return builder},update:function(cb){handlers["update"]=cb;return builder}};return builder};var Gesture=exports.Gesture=function(gesture,frame){this.gestures=[gesture];this.frames=[frame]};Gesture.prototype.update=function(gesture,frame){this.gestures.push(gesture);this.frames.push(frame);this.emit(gesture.state,this)};_.extend(Gesture.prototype,EventEmitter.prototype);var CircleGesture=function(data){this.center=data.center;this.normal=data.normal;this.progress=data.progress;this.radius=data.radius};CircleGesture.prototype.toString=function(){return"CircleGesture ["+JSON.stringify(this)+"]"};var SwipeGesture=function(data){this.startPosition=data.startPosition;this.position=data.position;this.direction=data.direction;this.speed=data.speed};SwipeGesture.prototype.toString=function(){return"SwipeGesture ["+JSON.stringify(this)+"]"};var ScreenTapGesture=function(data){this.position=data.position;this.direction=data.direction;this.progress=data.progress};ScreenTapGesture.prototype.toString=function(){return"ScreenTapGesture ["+JSON.stringify(this)+"]"};var KeyTapGesture=function(data){this.position=data.position;this.direction=data.direction;this.progress=data.progress};KeyTapGesture.prototype.toString=function(){return"KeyTapGesture ["+JSON.stringify(this)+"]"}},{events:17,"gl-matrix":19,underscore:20}],7:[function(require,module,exports){var Pointable=require("./pointable"),glMatrix=require("gl-matrix"),mat3=glMatrix.mat3,vec3=glMatrix.vec3,_=require("underscore");var Hand=module.exports=function(data){this.id=data.id;this.palmPosition=data.palmPosition;this.direction=data.direction;this.palmVelocity=data.palmVelocity;this.palmNormal=data.palmNormal;this.sphereCenter=data.sphereCenter;this.sphereRadius=data.sphereRadius;this.valid=true;this.pointables=[];this.fingers=[];this.tools=[];this._translation=data.t;this._rotation=_.flatten(data.r);this._scaleFactor=data.s;this.timeVisible=data.timeVisible;this.stabilizedPalmPosition=data.stabilizedPalmPosition};Hand.prototype.finger=function(id){var finger=this.frame.finger(id);return finger&&finger.handId==this.id?finger:Pointable.Invalid};Hand.prototype.rotationAngle=function(sinceFrame,axis){if(!this.valid||!sinceFrame.valid)return 0;var sinceHand=sinceFrame.hand(this.id);if(!sinceHand.valid)return 0;var rot=this.rotationMatrix(sinceFrame);var cs=(rot[0]+rot[4]+rot[8]-1)*.5;var angle=Math.acos(cs);angle=isNaN(angle)?0:angle;if(axis!==undefined){var rotAxis=this.rotationAxis(sinceFrame);angle*=vec3.dot(rotAxis,vec3.normalize(vec3.create(),axis))}return angle};Hand.prototype.rotationAxis=function(sinceFrame){if(!this.valid||!sinceFrame.valid)return vec3.create();var sinceHand=sinceFrame.hand(this.id);if(!sinceHand.valid)return vec3.create();return vec3.normalize(vec3.create(),[this._rotation[7]-sinceHand._rotation[5],this._rotation[2]-sinceHand._rotation[6],this._rotation[3]-sinceHand._rotation[1]])};Hand.prototype.rotationMatrix=function(sinceFrame){if(!this.valid||!sinceFrame.valid)return mat3.create();var sinceHand=sinceFrame.hand(this.id);if(!sinceHand.valid)return mat3.create();var transpose=mat3.transpose(mat3.create(),this._rotation);var m=mat3.multiply(mat3.create(),sinceHand._rotation,transpose);return m};Hand.prototype.scaleFactor=function(sinceFrame){if(!this.valid||!sinceFrame.valid)return 1;var sinceHand=sinceFrame.hand(this.id);if(!sinceHand.valid)return 1;return Math.exp(this._scaleFactor-sinceHand._scaleFactor)};Hand.prototype.translation=function(sinceFrame){if(!this.valid||!sinceFrame.valid)return vec3.create();var sinceHand=sinceFrame.hand(this.id);if(!sinceHand.valid)return vec3.create();return[this._translation[0]-sinceHand._translation[0],this._translation[1]-sinceHand._translation[1],this._translation[2]-sinceHand._translation[2]]};Hand.prototype.toString=function(){return"Hand [ id: "+this.id+" | palm velocity:"+this.palmVelocity+" | sphere center:"+this.sphereCenter+" ] "};Hand.Invalid={valid:false,fingers:[],tools:[],pointables:[],pointable:function(){return Pointable.Invalid},finger:function(){return Pointable.Invalid},toString:function(){return"invalid frame"},dump:function(){return this.toString()},rotationAngle:function(){return 0},rotationMatrix:function(){return mat3.create()},rotationAxis:function(){return vec3.create()},scaleFactor:function(){return 1},translation:function(){return vec3.create()}}},{"./pointable":11,"gl-matrix":19,underscore:20}],8:[function(require,module,exports){!function(){module.exports={Controller:require("./controller"),Frame:require("./frame"),Gesture:require("./gesture"),Hand:require("./hand"),Pointable:require("./pointable"),InteractionBox:require("./interaction_box"),Connection:require("./connection"),CircularBuffer:require("./circular_buffer"),UI:require("./ui"),glMatrix:require("gl-matrix"),mat3:require("gl-matrix").mat3,vec3:require("gl-matrix").vec3,loopController:undefined,loop:function(opts,callback){if(callback===undefined){callback=opts;opts={}}if(!this.loopController)this.loopController=new this.Controller(opts);this.loopController.loop(callback)}}}()},{"./circular_buffer":2,"./connection":3,"./controller":4,"./frame":5,"./gesture":6,"./hand":7,"./interaction_box":9,"./pointable":11,"./ui":13,"gl-matrix":19}],9:[function(require,module,exports){var glMatrix=require("gl-matrix"),vec3=glMatrix.vec3;var InteractionBox=module.exports=function(data){this.valid=true;this.center=data.center;this.size=data.size;this.width=data.size[0];this.height=data.size[1];this.depth=data.size[2]};InteractionBox.prototype.denormalizePoint=function(normalizedPosition){return vec3.fromValues((normalizedPosition[0]-.5)*this.size[0]+this.center[0],(normalizedPosition[1]-.5)*this.size[1]+this.center[1],(normalizedPosition[2]-.5)*this.size[2]+this.center[2])};InteractionBox.prototype.normalizePoint=function(position,clamp){var vec=vec3.fromValues((position[0]-this.center[0])/this.size[0]+.5,(position[1]-this.center[1])/this.size[1]+.5,(position[2]-this.center[2])/this.size[2]+.5);if(clamp){vec[0]=Math.min(Math.max(vec[0],0),1);vec[1]=Math.min(Math.max(vec[1],0),1);vec[2]=Math.min(Math.max(vec[2],0),1)}return vec};InteractionBox.prototype.toString=function(){return"InteractionBox [ width:"+this.width+" | height:"+this.height+" | depth:"+this.depth+" ]"};InteractionBox.Invalid={valid:false}},{"gl-matrix":19}],10:[function(require,module,exports){var Pipeline=module.exports=function(){this.steps=[]};Pipeline.prototype.addStep=function(step){this.steps.push(step)};Pipeline.prototype.run=function(frame){var stepsLength=this.steps.length;for(var i=0;i!=stepsLength;i++){if(!frame)break;frame=this.steps[i](frame)}return frame}},{}],11:[function(require,module,exports){var glMatrix=require("gl-matrix"),vec3=glMatrix.vec3;var Pointable=module.exports=function(data){this.valid=true;this.id=data.id;this.handId=data.handId;this.length=data.length;this.tool=data.tool;this.width=data.width;this.direction=data.direction;this.stabilizedTipPosition=data.stabilizedTipPosition;this.tipPosition=data.tipPosition;this.tipVelocity=data.tipVelocity;this.touchZone=data.touchZone;this.touchDistance=data.touchDistance;this.timeVisible=data.timeVisible};Pointable.prototype.toString=function(){if(this.tool==true){return"Pointable [ id:"+this.id+" "+this.length+"mmx | with:"+this.width+"mm | direction:"+this.direction+" ]"}else{return"Pointable [ id:"+this.id+" "+this.length+"mmx | direction: "+this.direction+" ]"}};Pointable.Invalid={valid:false}},{"gl-matrix":19}],12:[function(require,module,exports){var Frame=require("./frame");var Event=function(data){this.type=data.type;this.state=data.state};var chooseProtocol=exports.chooseProtocol=function(header){var protocol;switch(header.version){case 1:protocol=JSONProtocol(1,function(data){return new Frame(data)});break;case 2:protocol=JSONProtocol(2,function(data){return new Frame(data)});protocol.sendHeartbeat=function(connection){connection.send(protocol.encode({heartbeat:true}))};break;case 3:protocol=JSONProtocol(3,function(data){return data.event?new Event(data.event):new Frame(data)});protocol.sendHeartbeat=function(connection){connection.send(protocol.encode({heartbeat:true}))};break;default:throw"unrecognized version"}return protocol};var JSONProtocol=function(version,cb){var protocol=cb;protocol.encode=function(message){return JSON.stringify(message)};protocol.version=version;protocol.versionLong="Version "+version;protocol.type="protocol";return protocol}},{"./frame":5}],13:[function(require,module,exports){exports.UI={Region:require("./ui/region"),Cursor:require("./ui/cursor")}},{"./ui/cursor":14,"./ui/region":15}],14:[function(require,module,exports){var Cursor=module.exports=function(){return function(frame){var pointable=frame.pointables.sort(function(a,b){return a.z-b.z})[0];if(pointable&&pointable.valid){frame.cursorPosition=pointable.tipPosition}return frame}}},{}],15:[function(require,module,exports){var EventEmitter=require("events").EventEmitter,_=require("underscore");var Region=module.exports=function(start,end){this.start=new Vector(start);this.end=new Vector(end);this.enteredFrame=null};Region.prototype.hasPointables=function(frame){for(var i=0;i!=frame.pointables.length;i++){var position=frame.pointables[i].tipPosition;if(position.x>=this.start.x&&position.x<=this.end.x&&position.y>=this.start.y&&position.y<=this.end.y&&position.z>=this.start.z&&position.z<=this.end.z){return true}}return false};Region.prototype.listener=function(opts){var region=this;if(opts&&opts.nearThreshold)this.setupNearRegion(opts.nearThreshold);return function(frame){return region.updatePosition(frame)}};Region.prototype.clipper=function(){var region=this;return function(frame){region.updatePosition(frame);return region.enteredFrame?frame:null}};Region.prototype.setupNearRegion=function(distance){var nearRegion=this.nearRegion=new Region([this.start.x-distance,this.start.y-distance,this.start.z-distance],[this.end.x+distance,this.end.y+distance,this.end.z+distance]);var region=this;nearRegion.on("enter",function(frame){region.emit("near",frame)});nearRegion.on("exit",function(frame){region.emit("far",frame)});region.on("exit",function(frame){region.emit("near",frame)})};Region.prototype.updatePosition=function(frame){if(this.nearRegion)this.nearRegion.updatePosition(frame);if(this.hasPointables(frame)&&this.enteredFrame==null){this.enteredFrame=frame;this.emit("enter",this.enteredFrame)}else if(!this.hasPointables(frame)&&this.enteredFrame!=null){this.enteredFrame=null;this.emit("exit",this.enteredFrame)}return frame};Region.prototype.normalize=function(position){return new Vector([(position.x-this.start.x)/(this.end.x-this.start.x),(position.y-this.start.y)/(this.end.y-this.start.y),(position.z-this.start.z)/(this.end.z-this.start.z)])};Region.prototype.mapToXY=function(position,width,height){var normalized=this.normalize(position);var x=normalized.x,y=normalized.y;if(x>1)x=1;else if(x<-1)x=-1;if(y>1)y=1;else if(y<-1)y=-1;return[(x+1)/2*width,(1-y)/2*height,normalized.z]};_.extend(Region.prototype,EventEmitter.prototype)},{events:17,underscore:20}],16:[function(require,module,exports){},{}],17:[function(require,module,exports){!function(process){if(!process.EventEmitter)process.EventEmitter=function(){};var EventEmitter=exports.EventEmitter=process.EventEmitter;var isArray=typeof Array.isArray==="function"?Array.isArray:function(xs){return Object.prototype.toString.call(xs)==="[object Array]"};function indexOf(xs,x){if(xs.indexOf)return xs.indexOf(x);for(var i=0;i0&&this._events[type].length>m){this._events[type].warned=true;console.error("(node) warning: possible EventEmitter memory "+"leak detected. %d listeners added. "+"Use emitter.setMaxListeners() to increase limit.",this._events[type].length);console.trace()}}this._events[type].push(listener)}else{this._events[type]=[this._events[type],listener]}return this};EventEmitter.prototype.on=EventEmitter.prototype.addListener;EventEmitter.prototype.once=function(type,listener){var self=this;self.on(type,function g(){self.removeListener(type,g);listener.apply(this,arguments)});return this};EventEmitter.prototype.removeListener=function(type,listener){if("function"!==typeof listener){throw new Error("removeListener only takes instances of Function")}if(!this._events||!this._events[type])return this;var list=this._events[type];if(isArray(list)){var i=indexOf(list,listener);if(i<0)return this;list.splice(i,1);if(list.length==0)delete this._events[type]}else if(this._events[type]===listener){delete this._events[type]}return this};EventEmitter.prototype.removeAllListeners=function(type){if(arguments.length===0){this._events={};return this}if(type&&this._events&&this._events[type])this._events[type]=null;return this};EventEmitter.prototype.listeners=function(type){if(!this._events)this._events={};if(!this._events[type])this._events[type]=[];if(!isArray(this._events[type])){this._events[type]=[this._events[type]]}return this._events[type]}}(require("__browserify_process"))},{__browserify_process:18}],18:[function(require,module,exports){var process=module.exports={};process.nextTick=function(){var canSetImmediate=typeof window!=="undefined"&&window.setImmediate;var canPost=typeof window!=="undefined"&&window.postMessage&&window.addEventListener;if(canSetImmediate){return function(f){return window.setImmediate(f)}}if(canPost){var queue=[];window.addEventListener("message",function(ev){if(ev.source===window&&ev.data==="process-tick"){ev.stopPropagation();if(queue.length>0){var fn=queue.shift();fn()}}},true);return function nextTick(fn){queue.push(fn);window.postMessage("process-tick","*")}}return function nextTick(fn){setTimeout(fn,0)}}();process.title="browser";process.browser=true;process.env={};process.argv=[];process.binding=function(name){throw new Error("process.binding is not supported")};process.cwd=function(){return"/"};process.chdir=function(dir){throw new Error("process.chdir is not supported")}},{}],19:[function(require,module,exports){!function(){!function(){"use strict";var shim={};if(typeof exports==="undefined"){if(typeof define=="function"&&typeof define.amd=="object"&&define.amd){shim.exports={};define(function(){return shim.exports})}else{shim.exports=window}}else{shim.exports=exports}!function(exports){var vec2={};if(!GLMAT_EPSILON){var GLMAT_EPSILON=1e-6}vec2.create=function(){return new Float32Array(2)};vec2.clone=function(a){var out=new Float32Array(2);out[0]=a[0];out[1]=a[1];return out};vec2.fromValues=function(x,y){var out=new Float32Array(2);out[0]=x;out[1]=y;return out};vec2.copy=function(out,a){out[0]=a[0];out[1]=a[1];return out};vec2.set=function(out,x,y){out[0]=x;out[1]=y;return out};vec2.add=function(out,a,b){out[0]=a[0]+b[0];out[1]=a[1]+b[1];return out};vec2.sub=vec2.subtract=function(out,a,b){out[0]=a[0]-b[0];out[1]=a[1]-b[1];return out};vec2.mul=vec2.multiply=function(out,a,b){out[0]=a[0]*b[0];out[1]=a[1]*b[1];return out};vec2.div=vec2.divide=function(out,a,b){out[0]=a[0]/b[0];out[1]=a[1]/b[1];return out};vec2.min=function(out,a,b){out[0]=Math.min(a[0],b[0]); +out[1]=Math.min(a[1],b[1]);return out};vec2.max=function(out,a,b){out[0]=Math.max(a[0],b[0]);out[1]=Math.max(a[1],b[1]);return out};vec2.scale=function(out,a,b){out[0]=a[0]*b;out[1]=a[1]*b;return out};vec2.dist=vec2.distance=function(a,b){var x=b[0]-a[0],y=b[1]-a[1];return Math.sqrt(x*x+y*y)};vec2.sqrDist=vec2.squaredDistance=function(a,b){var x=b[0]-a[0],y=b[1]-a[1];return x*x+y*y};vec2.len=vec2.length=function(a){var x=a[0],y=a[1];return Math.sqrt(x*x+y*y)};vec2.sqrLen=vec2.squaredLength=function(a){var x=a[0],y=a[1];return x*x+y*y};vec2.negate=function(out,a){out[0]=-a[0];out[1]=-a[1];return out};vec2.normalize=function(out,a){var x=a[0],y=a[1];var len=x*x+y*y;if(len>0){len=1/Math.sqrt(len);out[0]=a[0]*len;out[1]=a[1]*len}return out};vec2.dot=function(a,b){return a[0]*b[0]+a[1]*b[1]};vec2.cross=function(out,a,b){var z=a[0]*b[1]-a[1]*b[0];out[0]=out[1]=0;out[2]=z;return out};vec2.lerp=function(out,a,b,t){var ax=a[0],ay=a[1];out[0]=ax+t*(b[0]-ax);out[1]=ay+t*(b[1]-ay);return out};vec2.transformMat2=function(out,a,m){var x=a[0],y=a[1];out[0]=x*m[0]+y*m[1];out[1]=x*m[2]+y*m[3];return out};vec2.forEach=function(){var vec=new Float32Array(2);return function(a,stride,offset,count,fn,arg){var i,l;if(!stride){stride=2}if(!offset){offset=0}if(count){l=Math.min(count*stride+offset,a.length)}else{l=a.length}for(i=offset;i0){len=1/Math.sqrt(len);out[0]=a[0]*len;out[1]=a[1]*len;out[2]=a[2]*len}return out};vec3.dot=function(a,b){return a[0]*b[0]+a[1]*b[1]+a[2]*b[2]};vec3.cross=function(out,a,b){var ax=a[0],ay=a[1],az=a[2],bx=b[0],by=b[1],bz=b[2];out[0]=ay*bz-az*by;out[1]=az*bx-ax*bz;out[2]=ax*by-ay*bx;return out};vec3.lerp=function(out,a,b,t){var ax=a[0],ay=a[1],az=a[2];out[0]=ax+t*(b[0]-ax);out[1]=ay+t*(b[1]-ay);out[2]=az+t*(b[2]-az);return out};vec3.transformMat4=function(out,a,m){var x=a[0],y=a[1],z=a[2];out[0]=m[0]*x+m[4]*y+m[8]*z+m[12];out[1]=m[1]*x+m[5]*y+m[9]*z+m[13];out[2]=m[2]*x+m[6]*y+m[10]*z+m[14];return out};vec3.transformQuat=function(out,a,q){var x=a[0],y=a[1],z=a[2],qx=q[0],qy=q[1],qz=q[2],qw=q[3],ix=qw*x+qy*z-qz*y,iy=qw*y+qz*x-qx*z,iz=qw*z+qx*y-qy*x,iw=-qx*x-qy*y-qz*z;out[0]=ix*qw+iw*-qx+iy*-qz-iz*-qy;out[1]=iy*qw+iw*-qy+iz*-qx-ix*-qz;out[2]=iz*qw+iw*-qz+ix*-qy-iy*-qx;return out};vec3.forEach=function(){var vec=new Float32Array(3);return function(a,stride,offset,count,fn,arg){var i,l;if(!stride){stride=3}if(!offset){offset=0}if(count){l=Math.min(count*stride+offset,a.length)}else{l=a.length}for(i=offset;i0){len=1/Math.sqrt(len);out[0]=a[0]*len;out[1]=a[1]*len;out[2]=a[2]*len;out[3]=a[3]*len}return out};vec4.dot=function(a,b){return a[0]*b[0]+a[1]*b[1]+a[2]*b[2]+a[3]*b[3]};vec4.lerp=function(out,a,b,t){var ax=a[0],ay=a[1],az=a[2],aw=a[3];out[0]=ax+t*(b[0]-ax);out[1]=ay+t*(b[1]-ay);out[2]=az+t*(b[2]-az);out[3]=aw+t*(b[3]-aw);return out};vec4.transformMat4=function(out,a,m){var x=a[0],y=a[1],z=a[2],w=a[3];out[0]=m[0]*x+m[4]*y+m[8]*z+m[12]*w;out[1]=m[1]*x+m[5]*y+m[9]*z+m[13]*w;out[2]=m[2]*x+m[6]*y+m[10]*z+m[14]*w;out[3]=m[3]*x+m[7]*y+m[11]*z+m[15]*w;return out};vec4.transformQuat=function(out,a,q){var x=a[0],y=a[1],z=a[2],qx=q[0],qy=q[1],qz=q[2],qw=q[3],ix=qw*x+qy*z-qz*y,iy=qw*y+qz*x-qx*z,iz=qw*z+qx*y-qy*x,iw=-qx*x-qy*y-qz*z;out[0]=ix*qw+iw*-qx+iy*-qz-iz*-qy;out[1]=iy*qw+iw*-qy+iz*-qx-ix*-qz;out[2]=iz*qw+iw*-qz+ix*-qy-iy*-qx;return out};vec4.forEach=function(){var vec=new Float32Array(4);return function(a,stride,offset,count,fn,arg){var i,l;if(!stride){stride=4}if(!offset){offset=0}if(count){l=Math.min(count*stride+offset,a.length)}else{l=a.length}for(i=offset;i=1){if(out!==a){out[0]=ax;out[1]=ay;out[2]=az;out[3]=aw}return out}halfTheta=Math.acos(cosHalfTheta);sinHalfTheta=Math.sqrt(1-cosHalfTheta*cosHalfTheta);if(Math.abs(sinHalfTheta)<.001){out[0]=ax*.5+bx*.5;out[1]=ay*.5+by*.5;out[2]=az*.5+bz*.5;out[3]=aw*.5+bw*.5;return out}ratioA=Math.sin((1-t)*halfTheta)/sinHalfTheta;ratioB=Math.sin(t*halfTheta)/sinHalfTheta;out[0]=ax*ratioA+bx*ratioB;out[1]=ay*ratioA+by*ratioB;out[2]=az*ratioA+bz*ratioB;out[3]=aw*ratioA+bw*ratioB;return out};quat.invert=function(out,a){var a0=a[0],a1=a[1],a2=a[2],a3=a[3],dot=a0*a0+a1*a1+a2*a2+a3*a3,invDot=dot?1/dot:0;out[0]=-a0*invDot;out[1]=-a1*invDot;out[2]=-a2*invDot;out[3]=a3*invDot;return out};quat.conjugate=function(out,a){out[0]=-a[0];out[1]=-a[1];out[2]=-a[2];out[3]=a[3];return out};quat.len=quat.length=vec4.length;quat.sqrLen=quat.squaredLength=vec4.squaredLength;quat.normalize=vec4.normalize;quat.str=function(a){return"quat("+a[0]+", "+a[1]+", "+a[2]+", "+a[3]+")"};if(typeof exports!=="undefined"){exports.quat=quat}}(shim.exports)}()}()},{}],20:[function(require,module,exports){!function(){!function(){var root=this;var previousUnderscore=root._;var breaker={};var ArrayProto=Array.prototype,ObjProto=Object.prototype,FuncProto=Function.prototype;var push=ArrayProto.push,slice=ArrayProto.slice,concat=ArrayProto.concat,toString=ObjProto.toString,hasOwnProperty=ObjProto.hasOwnProperty;var nativeForEach=ArrayProto.forEach,nativeMap=ArrayProto.map,nativeReduce=ArrayProto.reduce,nativeReduceRight=ArrayProto.reduceRight,nativeFilter=ArrayProto.filter,nativeEvery=ArrayProto.every,nativeSome=ArrayProto.some,nativeIndexOf=ArrayProto.indexOf,nativeLastIndexOf=ArrayProto.lastIndexOf,nativeIsArray=Array.isArray,nativeKeys=Object.keys,nativeBind=FuncProto.bind;var _=function(obj){if(obj instanceof _)return obj;if(!(this instanceof _))return new _(obj);this._wrapped=obj};if(typeof exports!=="undefined"){if(typeof module!=="undefined"&&module.exports){exports=module.exports=_}exports._=_}else{root._=_}_.VERSION="1.4.4";var each=_.each=_.forEach=function(obj,iterator,context){if(obj==null)return;if(nativeForEach&&obj.forEach===nativeForEach){obj.forEach(iterator,context)}else if(obj.length===+obj.length){for(var i=0,l=obj.length;i2;if(obj==null)obj=[];if(nativeReduce&&obj.reduce===nativeReduce){if(context)iterator=_.bind(iterator,context);return initial?obj.reduce(iterator,memo):obj.reduce(iterator)}each(obj,function(value,index,list){if(!initial){memo=value;initial=true}else{memo=iterator.call(context,memo,value,index,list)}});if(!initial)throw new TypeError(reduceError);return memo};_.reduceRight=_.foldr=function(obj,iterator,memo,context){var initial=arguments.length>2;if(obj==null)obj=[];if(nativeReduceRight&&obj.reduceRight===nativeReduceRight){if(context)iterator=_.bind(iterator,context);return initial?obj.reduceRight(iterator,memo):obj.reduceRight(iterator)}var length=obj.length;if(length!==+length){var keys=_.keys(obj);length=keys.length}each(obj,function(value,index,list){index=keys?keys[--length]:--length;if(!initial){memo=obj[index];initial=true}else{memo=iterator.call(context,memo,obj[index],index,list)}});if(!initial)throw new TypeError(reduceError);return memo};_.find=_.detect=function(obj,iterator,context){var result;any(obj,function(value,index,list){if(iterator.call(context,value,index,list)){result=value;return true}});return result};_.filter=_.select=function(obj,iterator,context){var results=[];if(obj==null)return results;if(nativeFilter&&obj.filter===nativeFilter)return obj.filter(iterator,context);each(obj,function(value,index,list){if(iterator.call(context,value,index,list))results[results.length]=value});return results};_.reject=function(obj,iterator,context){return _.filter(obj,function(value,index,list){return!iterator.call(context,value,index,list)},context)};_.every=_.all=function(obj,iterator,context){iterator||(iterator=_.identity);var result=true;if(obj==null)return result;if(nativeEvery&&obj.every===nativeEvery)return obj.every(iterator,context);each(obj,function(value,index,list){if(!(result=result&&iterator.call(context,value,index,list)))return breaker});return!!result};var any=_.some=_.any=function(obj,iterator,context){iterator||(iterator=_.identity);var result=false;if(obj==null)return result;if(nativeSome&&obj.some===nativeSome)return obj.some(iterator,context);each(obj,function(value,index,list){if(result||(result=iterator.call(context,value,index,list)))return breaker});return!!result};_.contains=_.include=function(obj,target){if(obj==null)return false;if(nativeIndexOf&&obj.indexOf===nativeIndexOf)return obj.indexOf(target)!=-1;return any(obj,function(value){return value===target})};_.invoke=function(obj,method){var args=slice.call(arguments,2);var isFunc=_.isFunction(method);return _.map(obj,function(value){return(isFunc?method:value[method]).apply(value,args)})};_.pluck=function(obj,key){return _.map(obj,function(value){return value[key]})};_.where=function(obj,attrs,first){if(_.isEmpty(attrs))return first?null:[];return _[first?"find":"filter"](obj,function(value){for(var key in attrs){if(attrs[key]!==value[key])return false}return true})};_.findWhere=function(obj,attrs){return _.where(obj,attrs,true)};_.max=function(obj,iterator,context){if(!iterator&&_.isArray(obj)&&obj[0]===+obj[0]&&obj.length<65535){return Math.max.apply(Math,obj)}if(!iterator&&_.isEmpty(obj))return-Infinity;var result={computed:-Infinity,value:-Infinity};each(obj,function(value,index,list){var computed=iterator?iterator.call(context,value,index,list):value;computed>=result.computed&&(result={value:value,computed:computed})});return result.value};_.min=function(obj,iterator,context){if(!iterator&&_.isArray(obj)&&obj[0]===+obj[0]&&obj.length<65535){return Math.min.apply(Math,obj)}if(!iterator&&_.isEmpty(obj))return Infinity;var result={computed:Infinity,value:Infinity};each(obj,function(value,index,list){var computed=iterator?iterator.call(context,value,index,list):value;computedb||a===void 0)return 1;if(a>>1;iterator.call(context,array[mid])=0})})};_.difference=function(array){var rest=concat.apply(ArrayProto,slice.call(arguments,1));return _.filter(array,function(value){return!_.contains(rest,value)})};_.zip=function(){var args=slice.call(arguments);var length=_.max(_.pluck(args,"length"));var results=new Array(length);for(var i=0;i=0;i--){args=[funcs[i].apply(this,args)]}return args[0]}};_.after=function(times,func){if(times<=0)return func();return function(){if(--times<1){return func.apply(this,arguments)}}};_.keys=nativeKeys||function(obj){if(obj!==Object(obj))throw new TypeError("Invalid object");var keys=[];for(var key in obj)if(_.has(obj,key))keys[keys.length]=key;return keys};_.values=function(obj){var values=[];for(var key in obj)if(_.has(obj,key))values.push(obj[key]);return values};_.pairs=function(obj){var pairs=[];for(var key in obj)if(_.has(obj,key))pairs.push([key,obj[key]]);return pairs};_.invert=function(obj){var result={};for(var key in obj)if(_.has(obj,key))result[obj[key]]=key;return result};_.functions=_.methods=function(obj){var names=[];for(var key in obj){if(_.isFunction(obj[key]))names.push(key)}return names.sort()};_.extend=function(obj){each(slice.call(arguments,1),function(source){if(source){for(var prop in source){obj[prop]=source[prop]}}});return obj};_.pick=function(obj){var copy={};var keys=concat.apply(ArrayProto,slice.call(arguments,1));each(keys,function(key){if(key in obj)copy[key]=obj[key]});return copy};_.omit=function(obj){var copy={};var keys=concat.apply(ArrayProto,slice.call(arguments,1));for(var key in obj){if(!_.contains(keys,key))copy[key]=obj[key]}return copy};_.defaults=function(obj){each(slice.call(arguments,1),function(source){if(source){for(var prop in source){if(obj[prop]==null)obj[prop]=source[prop]}}});return obj};_.clone=function(obj){if(!_.isObject(obj))return obj;return _.isArray(obj)?obj.slice():_.extend({},obj)};_.tap=function(obj,interceptor){interceptor(obj);return obj};var eq=function(a,b,aStack,bStack){if(a===b)return a!==0||1/a==1/b;if(a==null||b==null)return a===b;if(a instanceof _)a=a._wrapped;if(b instanceof _)b=b._wrapped;var className=toString.call(a);if(className!=toString.call(b))return false;switch(className){case"[object String]":return a==String(b);case"[object Number]":return a!=+a?b!=+b:a==0?1/a==1/b:a==+b;case"[object Date]":case"[object Boolean]":return+a==+b;case"[object RegExp]":return a.source==b.source&&a.global==b.global&&a.multiline==b.multiline&&a.ignoreCase==b.ignoreCase}if(typeof a!="object"||typeof b!="object")return false;var length=aStack.length;while(length--){if(aStack[length]==a)return bStack[length]==b}aStack.push(a);bStack.push(b);var size=0,result=true;if(className=="[object Array]"){size=a.length;result=size==b.length;if(result){while(size--){if(!(result=eq(a[size],b[size],aStack,bStack)))break}}}else{var aCtor=a.constructor,bCtor=b.constructor;if(aCtor!==bCtor&&!(_.isFunction(aCtor)&&aCtor instanceof aCtor&&_.isFunction(bCtor)&&bCtor instanceof bCtor)){return false}for(var key in a){if(_.has(a,key)){size++;if(!(result=_.has(b,key)&&eq(a[key],b[key],aStack,bStack)))break}}if(result){for(key in b){if(_.has(b,key)&&!size--)break}result=!size}}aStack.pop();bStack.pop();return result};_.isEqual=function(a,b){return eq(a,b,[],[])};_.isEmpty=function(obj){if(obj==null)return true;if(_.isArray(obj)||_.isString(obj))return obj.length===0;for(var key in obj)if(_.has(obj,key))return false;return true};_.isElement=function(obj){return!!(obj&&obj.nodeType===1)};_.isArray=nativeIsArray||function(obj){return toString.call(obj)=="[object Array]"};_.isObject=function(obj){return obj===Object(obj)};each(["Arguments","Function","String","Number","Date","RegExp"],function(name){_["is"+name]=function(obj){return toString.call(obj)=="[object "+name+"]"}});if(!_.isArguments(arguments)){_.isArguments=function(obj){return!!(obj&&_.has(obj,"callee"))}}if(typeof/./!=="function"){_.isFunction=function(obj){return typeof obj==="function"}}_.isFinite=function(obj){return isFinite(obj)&&!isNaN(parseFloat(obj))};_.isNaN=function(obj){return _.isNumber(obj)&&obj!=+obj};_.isBoolean=function(obj){return obj===true||obj===false||toString.call(obj)=="[object Boolean]"};_.isNull=function(obj){return obj===null};_.isUndefined=function(obj){return obj===void 0};_.has=function(obj,key){return hasOwnProperty.call(obj,key)};_.noConflict=function(){root._=previousUnderscore;return this};_.identity=function(value){return value};_.times=function(n,iterator,context){var accum=Array(n);for(var i=0;i":">",'"':""","'":"'","/":"/"}};entityMap.unescape=_.invert(entityMap.escape);var entityRegexes={escape:new RegExp("["+_.keys(entityMap.escape).join("")+"]","g"),unescape:new RegExp("("+_.keys(entityMap.unescape).join("|")+")","g")};_.each(["escape","unescape"],function(method){_[method]=function(string){if(string==null)return"";return(""+string).replace(entityRegexes[method],function(match){return entityMap[method][match]})}});_.result=function(object,property){if(object==null)return null;var value=object[property];return _.isFunction(value)?value.call(object):value};_.mixin=function(obj){each(_.functions(obj),function(name){var func=_[name]=obj[name];_.prototype[name]=function(){var args=[this._wrapped];push.apply(args,arguments);return result.call(this,func.apply(_,args))}})};var idCounter=0;_.uniqueId=function(prefix){var id=++idCounter+"";return prefix?prefix+id:id};_.templateSettings={evaluate:/<%([\s\S]+?)%>/g,interpolate:/<%=([\s\S]+?)%>/g,escape:/<%-([\s\S]+?)%>/g};var noMatch=/(.)^/;var escapes={"'":"'","\\":"\\","\r":"r","\n":"n"," ":"t","\u2028":"u2028","\u2029":"u2029"};var escaper=/\\|'|\r|\n|\t|\u2028|\u2029/g;_.template=function(text,data,settings){var render;settings=_.defaults({},settings,_.templateSettings);var matcher=new RegExp([(settings.escape||noMatch).source,(settings.interpolate||noMatch).source,(settings.evaluate||noMatch).source].join("|")+"|$","g");var index=0;var source="__p+='";text.replace(matcher,function(match,escape,interpolate,evaluate,offset){source+=text.slice(index,offset).replace(escaper,function(match){return"\\"+escapes[match]});if(escape){source+="'+\n((__t=("+escape+"))==null?'':_.escape(__t))+\n'"}if(interpolate){source+="'+\n((__t=("+interpolate+"))==null?'':__t)+\n'"}if(evaluate){source+="';\n"+evaluate+"\n__p+='"}index=offset+match.length;return match});source+="';\n";if(!settings.variable)source="with(obj||{}){\n"+source+"}\n";source="var __t,__p='',__j=Array.prototype.join,"+"print=function(){__p+=__j.call(arguments,'');};\n"+source+"return __p;\n";try{render=new Function(settings.variable||"obj","_",source)}catch(e){e.source=source;throw e}if(data)return render(data,_);var template=function(data){return render.call(this,data,_)};template.source="function("+(settings.variable||"obj")+"){\n"+source+"}";return template};_.chain=function(obj){return _(obj).chain()};var result=function(obj){return this._chain?_(obj).chain():obj};_.mixin(_);each(["pop","push","reverse","shift","sort","splice","unshift"],function(name){var method=ArrayProto[name];_.prototype[name]=function(){var obj=this._wrapped;method.apply(obj,arguments);if((name=="shift"||name=="splice")&&obj.length===0)delete obj[0];return result.call(this,obj)}});each(["concat","join","slice"],function(name){var method=ArrayProto[name];_.prototype[name]=function(){return result.call(this,method.apply(this._wrapped,arguments))}});_.extend(_.prototype,{chain:function(){this._chain=true;return this},value:function(){return this._wrapped}})}.call(this)}()},{}],21:[function(require,module,exports){window.requestAnimFrame=function(){return window.requestAnimationFrame||window.webkitRequestAnimationFrame||window.mozRequestAnimationFrame||window.oRequestAnimationFrame||window.msRequestAnimationFrame||function(callback){window.setTimeout(callback,1e3/60)}}();Leap=require("../lib/index")},{"../lib/index":8}]},{},[21]); + +/* + * Leap Motion integration for Reveal.js. + * James Sun [sun16] + * Rory Hardy [gneatgeek] + */ + +(function () { + var body = document.body, + controller = new Leap.Controller({ enableGestures: true }), + lastGesture = 0, + leapConfig = Reveal.getConfig().leap, + pointer = document.createElement( 'div' ), + config = { + autoCenter : true, // Center pointer around detected position. + gestureDelay : 500, // How long to delay between gestures. + naturalSwipe : true, // Swipe as if it were a touch screen. + pointerColor : '#00aaff', // Default color of the pointer. + pointerOpacity : 0.7, // Default opacity of the pointer. + pointerSize : 15, // Default minimum height/width of the pointer. + pointerTolerance : 120 // Bigger = slower pointer. + }, + entered, enteredPosition, now, size, tipPosition; // Other vars we need later, but don't need to redeclare. + + // Merge user defined settings with defaults + if( leapConfig ) { + for( key in leapConfig ) { + config[key] = leapConfig[key]; + } + } + + pointer.id = 'leap'; + + pointer.style.position = 'absolute'; + pointer.style.visibility = 'hidden'; + pointer.style.zIndex = 50; + pointer.style.opacity = config.pointerOpacity; + pointer.style.backgroundColor = config.pointerColor; + + body.appendChild( pointer ); + + // Leap's loop + controller.on( 'frame', function ( frame ) { + // Timing code to rate limit gesture execution + now = new Date().getTime(); + + // Pointer: 1 to 2 fingers. Strictly one finger works but may cause innaccuracies. + // The innaccuracies were observed on a development model and may not be an issue with consumer models. + if( frame.fingers.length > 0 && frame.fingers.length < 3 ) { + // Invert direction and multiply by 3 for greater effect. + size = -3 * frame.fingers[0].tipPosition[2]; + + if( size < config.pointerSize ) { + size = config.pointerSize; + } + + pointer.style.width = size + 'px'; + pointer.style.height = size + 'px'; + pointer.style.borderRadius = size - 5 + 'px'; + pointer.style.visibility = 'visible'; + + tipPosition = frame.fingers[0].tipPosition; + + if( config.autoCenter ) { + + + // Check whether the finger has entered the z range of the Leap Motion. Used for the autoCenter option. + if( !entered ) { + entered = true; + enteredPosition = frame.fingers[0].tipPosition; + } + + pointer.style.top = + (-1 * (( tipPosition[1] - enteredPosition[1] ) * body.offsetHeight / config.pointerTolerance )) + + ( body.offsetHeight / 2 ) + 'px'; + + pointer.style.left = + (( tipPosition[0] - enteredPosition[0] ) * body.offsetWidth / config.pointerTolerance ) + + ( body.offsetWidth / 2 ) + 'px'; + } + else { + pointer.style.top = ( 1 - (( tipPosition[1] - 50) / config.pointerTolerance )) * + body.offsetHeight + 'px'; + + pointer.style.left = ( tipPosition[0] * body.offsetWidth / config.pointerTolerance ) + + ( body.offsetWidth / 2 ) + 'px'; + } + } + else { + // Hide pointer on exit + entered = false; + pointer.style.visibility = 'hidden'; + } + + // Gestures + if( frame.gestures.length > 0 && (now - lastGesture) > config.gestureDelay ) { + var gesture = frame.gestures[0]; + + // One hand gestures + if( frame.hands.length === 1 ) { + // Swipe gestures. 3+ fingers. + if( frame.fingers.length > 2 && gesture.type === 'swipe' ) { + // Define here since some gestures will throw undefined for these. + var x = gesture.direction[0], + y = gesture.direction[1]; + + // Left/right swipe gestures + if( Math.abs( x ) > Math.abs( y )) { + if( x > 0 ) { + config.naturalSwipe ? Reveal.left() : Reveal.right(); + } + else { + config.naturalSwipe ? Reveal.right() : Reveal.left(); + } + } + // Up/down swipe gestures + else { + if( y > 0 ) { + config.naturalSwipe ? Reveal.down() : Reveal.up(); + } + else { + config.naturalSwipe ? Reveal.up() : Reveal.down(); + } + } + + lastGesture = now; + } + } + // Two hand gestures + else if( frame.hands.length === 2 ) { + // Upward two hand swipe gesture + if( gesture.type === 'swipe' && gesture.direction[1] > 0 ) { + Reveal.toggleOverview(); + } + + lastGesture = now; + } + } + }); + + controller.connect(); +})(); diff --git a/doc/pub/Autoencoders/html/reveal.js/plugin/remotes/remotes.js b/doc/pub/Autoencoders/html/reveal.js/plugin/remotes/remotes.js new file mode 100644 index 000000000..ba0dbad7b --- /dev/null +++ b/doc/pub/Autoencoders/html/reveal.js/plugin/remotes/remotes.js @@ -0,0 +1,39 @@ +/** + * Touch-based remote controller for your presentation courtesy + * of the folks at http://remotes.io + */ + +(function(window){ + + /** + * Detects if we are dealing with a touch enabled device (with some false positives) + * Borrowed from modernizr: https://github.com/Modernizr/Modernizr/blob/master/feature-detects/touch.js + */ + var hasTouch = (function(){ + return ('ontouchstart' in window) || window.DocumentTouch && document instanceof DocumentTouch; + })(); + + /** + * Detects if notes are enable and the current page is opened inside an /iframe + * this prevents loading Remotes.io several times + */ + var isNotesAndIframe = (function(){ + return window.RevealNotes && !(self == top); + })(); + + if(!hasTouch && !isNotesAndIframe){ + head.ready( 'remotes.ne.min.js', function() { + new Remotes("preview") + .on("swipe-left", function(e){ Reveal.right(); }) + .on("swipe-right", function(e){ Reveal.left(); }) + .on("swipe-up", function(e){ Reveal.down(); }) + .on("swipe-down", function(e){ Reveal.up(); }) + .on("tap", function(e){ Reveal.next(); }) + .on("zoom-out", function(e){ Reveal.toggleOverview(true); }) + .on("zoom-in", function(e){ Reveal.toggleOverview(false); }) + ; + } ); + + head.js('https://hakim-static.s3.amazonaws.com/reveal-js/remotes.ne.min.js'); + } +})(window); \ No newline at end of file diff --git a/doc/pub/Bayesian/html/._Bayesian-bs002.html b/doc/pub/Bayesian/html/._Bayesian-bs002.html new file mode 100644 index 000000000..6a2fb1711 --- /dev/null +++ b/doc/pub/Bayesian/html/._Bayesian-bs002.html @@ -0,0 +1,218 @@ + + + + + + + + +Data Analysis and Machine Learning: Elements of Bayesian theory and Bayesian Neural Networks + + + + + + + + + + + + + + + + + + + + + + + +
+
+ + Data Analysis and Machine Learning: Elements of Bayesian theory and Bayesian Neural Networks +
+ +
+ +
+
+ + +
+ +

 

 

 

+ + + + +

Inference

+
+
+

+ +

+
Inference:
+ "the act of passing from one proposition, statement or judgment considered as true to another whose truth is believed to follow from that of the former" (Webster)
+ Do premises \( A, B, \ldots \to \) hypothesis, \( H \)? +
Deductive inference:
+ Premises allow definite determination of truth/falsity of H (syllogisms, symbolic logic, Boolean algebra)
+ \( B(H|A,B,...) = 0 \) or \( 1 \) +
Inductive inference:
+ Premises bear on truth/falsity of H, but don’t allow its definite determination (weak syllogisms, analogies)
+ \( A, B, C, D \) share properties \( x, y, z \); \( E \) has properties \( x, y \)
+ \( \to \) $E$ probably has property \( z \). +
+
+
+ + +

+

+ +

+ + +
+ + + + + + + +
+ +
+ + + + + + diff --git a/doc/pub/Bayesian/html/._Bayesian-bs003.html b/doc/pub/Bayesian/html/._Bayesian-bs003.html new file mode 100644 index 000000000..4c1f7f601 --- /dev/null +++ b/doc/pub/Bayesian/html/._Bayesian-bs003.html @@ -0,0 +1,213 @@ + + + + + + + + +Data Analysis and Machine Learning: Elements of Bayesian theory and Bayesian Neural Networks + + + + + + + + + + + + + + + + + + + + + + + +
+
+ + Data Analysis and Machine Learning: Elements of Bayesian theory and Bayesian Neural Networks +
+ +
+ +
+
+ + +
+ +

 

 

 

+ + + + +

Statistical Inference

+
+
+

+ +

    +
  • Quantify the strength of inductive inferences from facts, in the form of data (\( D \)), and other premises, e.g. models, to hypotheses about the phenomena producing the data.
    • +
  • Quantify via probabilities, or averages calculated using probabilities. Frequentists (\( \mathcal{F} \)) and Bayesians (\( \mathcal{B} \)) use probabilities very differently for this.
    • +
  • To the pioneers such as Bernoulli, Bayes and Laplace, a probability represented a degree-of-belief or plausability: how much they thought that something as true based on the evidence at hand. This is the Bayesian approach.
    • +
  • To the 19th century scholars, this seemed too vague and subjective. They redefined probability as the long run relative frequency with which an event occurred, given (infinitely) many repeated (experimental) trials.
    • +
+
+
+ + +

+

+ +

+ + +
+ + + + + + + +
+ +
+ + + + + + diff --git a/doc/pub/Bayesian/html/._Bayesian-bs004.html b/doc/pub/Bayesian/html/._Bayesian-bs004.html new file mode 100644 index 000000000..2decbca37 --- /dev/null +++ b/doc/pub/Bayesian/html/._Bayesian-bs004.html @@ -0,0 +1,210 @@ + + + + + + + + +Data Analysis and Machine Learning: Elements of Bayesian theory and Bayesian Neural Networks + + + + + + + + + + + + + + + + + + + + + + + +
+
+ + Data Analysis and Machine Learning: Elements of Bayesian theory and Bayesian Neural Networks +
+ +
+ +
+
+ + +
+ +

 

 

 

+ + + + +

Some history

+Adapted from D.S. Sivia : + +

1: Sivia, Devinderjit, and John Skilling. Data Analysis : A Bayesian Tutorial, OUP Oxford, 2006

+ +

+

+ Although the frequency definition appears to be more objective, its range of validity is also far more limited. For example, Laplace used (his) probability theory to estimate the mass of Saturn, given orbital data that were available to him from various astronomical observatories. In essence, he computed the posterior pdf for the mass M , given the data and all the relevant background information I (such as a knowledge of the laws of classical mechanics): prob(M|{data},I); this is shown schematically in the figure [Fig. 1.2]. +
+ + +

+

+ +

+ + +
+ + + + + + + +
+ +
+ + + + + + diff --git a/doc/pub/Bayesian/html/._Bayesian-bs005.html b/doc/pub/Bayesian/html/._Bayesian-bs005.html new file mode 100644 index 000000000..c954262b4 --- /dev/null +++ b/doc/pub/Bayesian/html/._Bayesian-bs005.html @@ -0,0 +1,201 @@ + + + + + + + + +Data Analysis and Machine Learning: Elements of Bayesian theory and Bayesian Neural Networks + + + + + + + + + + + + + + + + + + + + + + + +
+
+ + Data Analysis and Machine Learning: Elements of Bayesian theory and Bayesian Neural Networks +
+ +
+ +
+
+ + +
+ +

 

 

 

+ + + +



+ +

+

+ +

+ + +
+ + + + + + + +
+ +
+ + + + + + diff --git a/doc/pub/Bayesian/html/._Bayesian-bs006.html b/doc/pub/Bayesian/html/._Bayesian-bs006.html new file mode 100644 index 000000000..52f41d2bc --- /dev/null +++ b/doc/pub/Bayesian/html/._Bayesian-bs006.html @@ -0,0 +1,205 @@ + + + + + + + + +Data Analysis and Machine Learning: Elements of Bayesian theory and Bayesian Neural Networks + + + + + + + + + + + + + + + + + + + + + + + +
+
+ + Data Analysis and Machine Learning: Elements of Bayesian theory and Bayesian Neural Networks +
+ +
+ +
+
+ + +
+ +

 

 

 

+ + + +
+ To Laplace, the (shaded) area under the posterior pdf curve between \( m_1 \) and \( m_2 \) was a measure of how much he believed that the mass of Saturn lay in the range \( m_1 \le M \le m_2 \). As such, the position of the maximum of the posterior pdf represents a best estimate of the mass; its width, or spread, about this optimal value gives an indication of the uncertainty in the estimate. Laplace stated that: ‘ . . . it is a bet of 11,000 to 1 that the error of this result is not 1/100th of its value.’ He would have won the bet, as another 150 years’ accumulation of data has changed the estimate by only 0.63%! +
+ + +

+

+ +

+ + +
+ + + + + + + +
+ +
+ + + + + + diff --git a/doc/pub/Bayesian/html/._Bayesian-bs007.html b/doc/pub/Bayesian/html/._Bayesian-bs007.html new file mode 100644 index 000000000..033bd5b2b --- /dev/null +++ b/doc/pub/Bayesian/html/._Bayesian-bs007.html @@ -0,0 +1,209 @@ + + + + + + + + +Data Analysis and Machine Learning: Elements of Bayesian theory and Bayesian Neural Networks + + + + + + + + + + + + + + + + + + + + + + + +
+
+ + Data Analysis and Machine Learning: Elements of Bayesian theory and Bayesian Neural Networks +
+ +
+ +
+
+ + +
+ +

 

 

 

+ + + +
+ According to the frequency definition, however, we are not permitted to use probability theory to tackle this problem. This is because the mass of Saturn is a constant and not a random variable; therefore, it has no frequency distribution and so probability theory cannot be used. + +

+ If the pdf [of Fig. 1.2] had to be interpreted in terms of the frequency definition, we would have to imagine a large ensemble of universes in which everything remains constant apart from the mass of Saturn. +

+ + +

+

+ +

+ + +
+ + + + + + + +
+ +
+ + + + + + diff --git a/doc/pub/Bayesian/html/._Bayesian-bs008.html b/doc/pub/Bayesian/html/._Bayesian-bs008.html new file mode 100644 index 000000000..51f161022 --- /dev/null +++ b/doc/pub/Bayesian/html/._Bayesian-bs008.html @@ -0,0 +1,208 @@ + + + + + + + + +Data Analysis and Machine Learning: Elements of Bayesian theory and Bayesian Neural Networks + + + + + + + + + + + + + + + + + + + + + + + +
+
+ + Data Analysis and Machine Learning: Elements of Bayesian theory and Bayesian Neural Networks +
+ +
+ +
+
+ + +
+ +

 

 

 

+ + + +
+ As this scenario appears quite far-fetched, we might be inclined to think of [Fig. 1.2] in terms of the distribution of the measurements of the mass in many repetitions of the experiment. Although we are at liberty to think about a problem in any way that facilitates its solution, or our understanding of it, having to seek a frequency interpretation for every data analysis problem seems rather perverse. + For example, what do we mean by the ‘measurement of the mass’ when the data consist of orbital periods? Besides, why should we have to think about many repetitions of an experiment that never happened? What we really want to do is to make the best inference of the mass given the (few) data that we actually have; this is precisely the Bayes and Laplace view of probability. +
+ + +

+

+ +

+ + +
+ + + + + + + +
+ +
+ + + + + + diff --git a/doc/pub/Bayesian/html/._Bayesian-bs009.html b/doc/pub/Bayesian/html/._Bayesian-bs009.html new file mode 100644 index 000000000..44f9ca95e --- /dev/null +++ b/doc/pub/Bayesian/html/._Bayesian-bs009.html @@ -0,0 +1,208 @@ + + + + + + + + +Data Analysis and Machine Learning: Elements of Bayesian theory and Bayesian Neural Networks + + + + + + + + + + + + + + + + + + + + + + + +
+
+ + Data Analysis and Machine Learning: Elements of Bayesian theory and Bayesian Neural Networks +
+ +
+ +
+
+ + +
+ +

 

 

 

+ + + +
+ Faced with the realization that the frequency definition of probability theory did not permit most real-life scientific problems to be addressed, a new subject was invented — statistics! To estimate the mass of Saturn, for example, one has to relate the mass to the data through some function called the statistic; since the data are subject to ‘random’ noise, the statistic becomes the random variable to which the rules of probability the- ory can be applied. But now the question arises: How should we choose the statistic? The frequentist approach does not yield a natural way of doing this and has, therefore, led to the development of several alternative schools of orthodox or conventional statis- tics. The masters, such as Fisher, Neyman and Pearson, provided a variety of different principles, which has merely resulted in a plethora of tests and procedures without any clear underlying rationale. This lack of unifying principles is, perhaps, at the heart of the shortcomings of the cook-book approach to statistics that students are often taught even today. +
+ + +

+

+ +

+ + +
+ + + + + + + +
+ +
+ + + + + + diff --git a/doc/pub/Bayesian/html/._Bayesian-bs010.html b/doc/pub/Bayesian/html/._Bayesian-bs010.html new file mode 100644 index 000000000..55794dd2d --- /dev/null +++ b/doc/pub/Bayesian/html/._Bayesian-bs010.html @@ -0,0 +1,233 @@ + + + + + + + + +Data Analysis and Machine Learning: Elements of Bayesian theory and Bayesian Neural Networks + + + + + + + + + + + + + + + + + + + + + + + +
+
+ + Data Analysis and Machine Learning: Elements of Bayesian theory and Bayesian Neural Networks +
+ +
+ +
+
+ + +
+ +

 

 

 

+ + + + +

The Bayesian recipe

+
+
+

+Assess hypotheses by calculating their probabilities \( p(H_i | \ldots) \) conditional on known and/or presumed information using the rules of probability theory. +

+
+ +
+
+

+Probability Theory Axioms: + +

+
Product (AND) rule :
+ \( p(A, B | I) = p(A|I) p(B|A, I) = p(B|I)p(A|B,I) \)
+ Should read \( p(A,B|I) \) as the probability for propositions \( A \) AND \( B \) being true given that \( I \) is true. +
Sum (OR) rule:
+ \( p(A + B | I) = p(A | I) + p(B | I) - p(A, B | I) \)
+ \( p(A+B|I) \) is the probability that proposition \( A \) OR \( B \) is true given that \( I \) is true. +
Normalization:
+ \( p(A|I) + p(\bar{A}|I) = 1 \)
+ \( \bar{A} \) denotes the proposition that \( A \) is false. +
+
+
+ + +

+

+ +

+ + +
+ + + + + + + +
+ +
+ + + + + + diff --git a/doc/pub/Bayesian/html/._Bayesian-bs011.html b/doc/pub/Bayesian/html/._Bayesian-bs011.html new file mode 100644 index 000000000..efed96d9d --- /dev/null +++ b/doc/pub/Bayesian/html/._Bayesian-bs011.html @@ -0,0 +1,229 @@ + + + + + + + + +Data Analysis and Machine Learning: Elements of Bayesian theory and Bayesian Neural Networks + + + + + + + + + + + + + + + + + + + + + + + +
+
+ + Data Analysis and Machine Learning: Elements of Bayesian theory and Bayesian Neural Networks +
+ +
+ +
+
+ + +
+ +

 

 

 

+ + + + +

Bayes' theorem

+
+
+

+Bayes' theorem follows directly from the product rule +$$ +$$ +p(A|B,I) = \frac{p(B|A,I) p(A|I)}{p(B|I)}. +$$ +$$ + +The importance of this property to data analysis becomes apparent if we replace \( A \) and \( B \) by hypothesis(\( H \)) and data(\( D \)): +$$ +\begin{align} +p(H|D,I) &= \frac{p(D|H,I) p(H|I)}{p(D|I)}. +\tag{1} +\end{align} +$$ + +The power of Bayes’ theorem lies in the fact that it relates the quantity of interest, the probability that the hypothesis is true given the data, to the term we have a better chance of being able to assign, the probability that we would have observed the measured data if the hypothesis was true. +

+
+ + +

+

+ +

+ + +
+ + + + + + + +
+ +
+ + + + + + diff --git a/doc/pub/Bayesian/html/._Bayesian-bs012.html b/doc/pub/Bayesian/html/._Bayesian-bs012.html new file mode 100644 index 000000000..d0a773446 --- /dev/null +++ b/doc/pub/Bayesian/html/._Bayesian-bs012.html @@ -0,0 +1,221 @@ + + + + + + + + +Data Analysis and Machine Learning: Elements of Bayesian theory and Bayesian Neural Networks + + + + + + + + + + + + + + + + + + + + + + + +
+
+ + Data Analysis and Machine Learning: Elements of Bayesian theory and Bayesian Neural Networks +
+ +
+ +
+
+ + +
+ +

 

 

 

+ + + +
+
+

+The various terms in Bayes’ theorem have formal names. + +

    +
  • The quantity on the far right, \( p(H|I) \), is called the prior probability; it represents our state of knowledge (or ignorance) about the truth of the hypothesis before we have analysed the current data.
    • +
  • This is modified by the experimental measurements through \( p(D|H,I) \), the likelihood function,
    • +
  • The denominator \( p(D|I) \) is called the evidence. It does not depend on the hypothesis and can be regarded as a normalization constant.
    • +
  • Together, these yield the posterior probability, \( p(H|D, I ) \), representing our state of knowledge about the truth of the hypothesis in the light of the data.
    • +
+ +In a sense, Bayes’ theorem encapsulates the process of learning. +
+
+ + +

+

+ +

+ + +
+ + + + + + + +
+ +
+ + + + + + diff --git a/doc/pub/Bayesian/html/._Bayesian-bs013.html b/doc/pub/Bayesian/html/._Bayesian-bs013.html new file mode 100644 index 000000000..d02cb1077 --- /dev/null +++ b/doc/pub/Bayesian/html/._Bayesian-bs013.html @@ -0,0 +1,228 @@ + + + + + + + + +Data Analysis and Machine Learning: Elements of Bayesian theory and Bayesian Neural Networks + + + + + + + + + + + + + + + + + + + + + + + +
+
+ + Data Analysis and Machine Learning: Elements of Bayesian theory and Bayesian Neural Networks +
+ +
+ +
+
+ + +
+ +

 

 

 

+ + + + +

The friends of Bayes' theorem

+
+
+

+ +

+
Normalization:
+ \( \sum_i p(H_i|\ldots) = 1 \). +
Marginalization:
+ \( \sum_i p(A,H_i|I) = \sum_i p(H_i|A,I) p(A|I) = p(A|I) \). +
Marginalization (continuum limit):
+ \( \int dx p(A,H(x)|I) = p(A|I) \). +
+ +In the above, \( H_i \) is an exclusive and exhaustive list of hypotheses. For example,let’s imagine that there are five candidates in a presidential election; then \( H_1 \) could be the proposition that the first candidate will win, and so on. The probability that \( A \) is true, for example that unemployment will be lower in a year’s time (given all relevant information \( I \), but irrespective of whoever becomes president) is then given by \( \sum_i p(A,H_i|I) \). + +

+In the continuum limit of propositions we must understand \( p(\ldots) \) as a pdf (probability density function). + +

+Marginalization is a very powerful device in data analysis because it enables us to deal with nuisance parameters; that is, quantities which necessarily enter the analysis but are of no intrinsic interest. The unwanted background signal present in many experimental measurements are examples of nuisance parameters. +

+
+ + +

+

+ +

+ + +
+ + + + + + + +
+ +
+ + + + + + diff --git a/doc/pub/Bayesian/html/._Bayesian-bs014.html b/doc/pub/Bayesian/html/._Bayesian-bs014.html new file mode 100644 index 000000000..53f77119c --- /dev/null +++ b/doc/pub/Bayesian/html/._Bayesian-bs014.html @@ -0,0 +1,238 @@ + + + + + + + + +Data Analysis and Machine Learning: Elements of Bayesian theory and Bayesian Neural Networks + + + + + + + + + + + + + + + + + + + + + + + +
+
+ + Data Analysis and Machine Learning: Elements of Bayesian theory and Bayesian Neural Networks +
+ +
+ +
+
+ + +
+ +

 

 

 

+ + + + +

Inference With Parametric Models

+
+
+

+Inductive inference with parametric models is a very important tool in the natural sciences. + +

    +
  • Consider \( N \) different models \( M_i \) (\( i = 1, \ldots, N \)), each with parameters \( \boldsymbol{\alpha}_i \). Each of them implies a sampling distribution (conditional predictive distribution for possible data)
    • +
+ +$$ +$$ +p(D|\boldsymbol{\alpha}_i, M_i) +$$ +$$ + + +
    +
  • The \( \boldsymbol{\alpha}_i \) dependence when we fix attention on the actual, observed data (\( D_\mathrm{obs} \)) is the likelihood function:
    • +
+ +$$ +$$ +\mathcal{L}_i (\boldsymbol{\alpha}_i) \equiv p(D_\mathrm{obs}|\boldsymbol{\alpha}_i, M_i) +$$ +$$ + + +
    +
  • We may be uncertain about \( i \) (model uncertainty),
    • +
  • or uncertain about \( \boldsymbol{\alpha}_i \) (parameter uncertainty).
    • +
+
+
+ + +

+

+ +

+ + +
+ + + + + + + +
+ +
+ + + + + + diff --git a/doc/pub/Bayesian/html/._Bayesian-bs015.html b/doc/pub/Bayesian/html/._Bayesian-bs015.html new file mode 100644 index 000000000..61fe7625a --- /dev/null +++ b/doc/pub/Bayesian/html/._Bayesian-bs015.html @@ -0,0 +1,222 @@ + + + + + + + + +Data Analysis and Machine Learning: Elements of Bayesian theory and Bayesian Neural Networks + + + + + + + + + + + + + + + + + + + + + + + +
+
+ + Data Analysis and Machine Learning: Elements of Bayesian theory and Bayesian Neural Networks +
+ +
+ +
+
+ + +
+ +

 

 

 

+ + + +
+
+

+ +

+
Parameter Estimation:
+ Premise = choice of model (pick specific \( i \))
+ \( \Rightarrow \) What can we say about \( \boldsymbol{\alpha}_i \)? +
Model comparison:
+ Premise = \( \{M_i\} \)
+ \( \Rightarrow \) What can we say about \( i \)? +
Model adequacy:
+ Premise = \( M_1 \)
+ \( \Rightarrow \) Is \( M_1 \) adequate? +
Hybrid Uncertainty:
+ Models share some common params: \( \boldsymbol{\alpha}_1 = \{ \boldsymbol{\varphi}, \boldsymbol{\eta}_i\} \)
+ \( \Rightarrow \) What can we say about \( \boldsymbol{\varphi} \)? (Systematic error is an example) +
+
+
+ + +

+

+ +

+ + +
+ + + + + + + +
+ +
+ + + + + + diff --git a/doc/pub/Bayesian/html/._Bayesian-bs016.html b/doc/pub/Bayesian/html/._Bayesian-bs016.html new file mode 100644 index 000000000..2cd99f83a --- /dev/null +++ b/doc/pub/Bayesian/html/._Bayesian-bs016.html @@ -0,0 +1,216 @@ + + + + + + + + +Data Analysis and Machine Learning: Elements of Bayesian theory and Bayesian Neural Networks + + + + + + + + + + + + + + + + + + + + + + + +
+
+ + Data Analysis and Machine Learning: Elements of Bayesian theory and Bayesian Neural Networks +
+ +
+ +
+
+ + +
+ +

 

 

 

+ + + + +

Illustrative examples with python code

+
+
+

+ +

    +
  • Is this a fair coin? (analytical)
    • +
  • Flux from a star (single parameter, MCMC)
    • +
  • The lighthouse problem (two parameters, MCMC)
    • +
  • Linear fit with outliers (nuisance parameters)
    • +
  • ...
    • +
+
+
+ + +

+

+ +

+ + +
+ + + + + + + +
+ +
+ + + + + + diff --git a/doc/pub/Bayesian/html/._Bayesian-bs017.html b/doc/pub/Bayesian/html/._Bayesian-bs017.html new file mode 100644 index 000000000..d85c83fcb --- /dev/null +++ b/doc/pub/Bayesian/html/._Bayesian-bs017.html @@ -0,0 +1,313 @@ + + + + + + + + +Data Analysis and Machine Learning: Elements of Bayesian theory and Bayesian Neural Networks + + + + + + + + + + + + + + + + + + + + + + + +
+
+ + Data Analysis and Machine Learning: Elements of Bayesian theory and Bayesian Neural Networks +
+ +
+ +
+
+ + +
+ +

 

 

 

+ + + + +

Example: Is this a fair coin?

+Let us begin with the analysis of data from a simple coin-tossing experiment. +Given that we had observed 6 heads in 8 flips, would you think it was a fair coin? By fair, we mean that we would be prepared to lay an even 1 : 1 bet on the outcome of a flip being a head or a tail. If we decide that the coin was fair, the question which follows naturally is how sure are we that this was so; if it was not fair, how unfair do we think it was? Furthermore, if we were to continue collecting data for this particular coin, observing the outcomes of additional flips, how would we update our belief on the fairness of the coin? + +

+A sensible way of formulating this problem is to consider a large number of hypotheses about the range in which the bias-weighting of the coin might lie. If we denote the bias-weighting by \( H \), then \( H = 0 \) and \( H = 1 \) can represent a coin which produces a tail or a head on every flip, respectively. There is a continuum of possibilities for the value of H between these limits, with \( H = 0.5 \) indicating a fair coin. Our state of knowledge about the fairness, or the degree of unfairness, of the coin is then completely summarized by specifying how much we believe these various propositions to be true. + +

+Let us perform a computer simulation of a coin-tossing experiment. This provides the data that we will be analysing. + +

+ + +

import numpy as np
+import matplotlib.pyplot as plt
+
+

+ + +

np.random.seed(999)         # for reproducibility
+a=0.6                       # biased coin
+flips=np.random.rand(2**12) # simulates 4096 coin flips
+heads=flips<a               # boolean array, heads[i]=True if flip i is heads
+
+

+In the light of this data, our inference about the fairness of this coin is summarized by the conditional pdf: \( p(H|D,I) \). This is, of course, shorthand for the limiting case of a continuum of propositions for the value of \( H \); that is to say, the probability that \( H \) lies in an infinitesimally narrow range is given by \( p(H|D,I) dH \). + +

+To estimate this posterior pdf, we need to use Bayes’ theorem (1). We will ignore the denominator \( p(D|I) \) as it does not involve bias-weighting explicitly, and it will therefore not affect the shape of the desired pdf. At the end we can evaluate the missing constant subsequently from the normalization condition +$$ +\begin{equation} +\int_0^1 p(H|D,I) dH = 1. +\tag{2} +\end{equation} +$$ + +

+The prior pdf, \( p(H|I) \), represents what we know about the coin given only the information \( I \) that we are dealing with a ‘strange coin’. We could keep a very open mind about the nature of the coin; a simple probability assignment which reflects this is a uniform, or flat, prior +$$ +\begin{equation} +p(H|I) = \left\{ \begin{array}{ll} +1 & 0 \le H \le 1, \\ +0 & \mathrm{otherwise}. +\end{array} \right. +\tag{3} +\end{equation} +$$ + +We will get back later to the choice of prior and its effect on the analysis. + +

+This prior state of knowledge, or ignorance, is modified by the data through the likelihood function \( p(D|H,I) \). It is a measure of the chance that we would have obtained the data that we actually observed, if the value of the bias-weighting was given (as known). If, in the conditioning information \( I \), we assume that the flips of the coin were independent events, so that the outcome of one did not influence that of another, then the probability of obtaining the data `R heads in N tosses' is given by the binomial distribution (we leave a formal definition of this to a statistics textbook) +$$ +\begin{equation} +p(D|H,I) \propto H^R (1-H)^{N-R}. +\tag{4} +\end{equation} +$$ + +It seems reasonable because \( H \) is the chance of obtaining a head on any flip, and there were \( R \) of them, and \( 1-H \) is the corresponding probability for a tail, of which there were \( N-R \). We note that this binomial distribution also contains a normalization factor, but we will ignore it since it does not depend explicitly on \( H \), the quantity of interest. It will be absorbed by the normalization condition (2). + +

+We perform the setup of this Bayesian framework on the computer. + +

+ + +

def prior(H):
+    p=np.zeros_like(H)
+    p[(0<=x)&(x<=1)]=1      # allowed range: 0<=H<=1
+    return p                # uniform prior
+def likelihood(H,data):
+    N = len(data)
+    no_of_heads = sum(data)
+    no_of_tails = N - no_of_heads
+    return H**no_of_heads * (1-H)**no_of_tails
+def posterior(H,data):
+    p=prior(H)*likelihood(H,data)
+    norm=np.trapz(p,H)
+    return p/norm
+
+

+The next step is to confront this setup with the simulated data. To get a feel for the result, it is instructive to see how the posterior pdf evolves as we obtain more and more data pertaining to the coin. The results of such an analyses is shown in Fig. 1. + +

+ + +

x=np.linspace(0,1,100)
+fig, axs = plt.subplots(nrows=4,ncols=3,sharex=True,sharey='row')
+axs_vec=np.reshape(axs,-1)
+axs_vec[0].plot(x,prior(x))
+for ndouble in range(11):
+    ax=axs_vec[1+ndouble]
+    ax.plot(x,posterior(x,heads[:2**ndouble]))
+    ax.text(0.1, 0.8, '$N={0}$'.format(2**ndouble), transform=ax.transAxes)
+for row in range(4): axs[row,0].set_ylabel('$p(H|D_\mathrm{obs},I)$')
+for col in range(3): axs[-1,col].set_xlabel('$H$')
+
+

+

+
+

Figure 1: The evolution of the posterior pdf for the bias-weighting of a coin, as the number of data available increases. The figure on the top left-hand corner of each panel shows the number of data included in the analysis.

+

+
+ +

+The panel in the top left-hand corner shows the posterior pdf for \( H \) given no data, i.e., it is the same as the prior pdf of Eq. (3). It indicates that we have no more reason to believe that the coin is fair than we have to think that it is double-headed, double-tailed, or of any other intermediate bias-weighting. + +

+The first flip is obviously tails. At this point we have no evidence that the coin has a side with heads, as indicated by the pdf going to zero as \( H \to 1 \). The second flip is obviously heads and we have now excluded both extreme options \( H=0 \) (double-tailed) and \( H=1 \) (double-headed). We can note that the posterior at this point has the simple form \( p(H|D,I) = H(1-H) \) for \( 0 \le H \le 1 \). + +

+The remainder of Fig. 1 shows how the posterior pdf evolves as the number of data analysed becomes larger and larger. We see that the position of the maximum moves around, but that the amount by which it does so decreases with the increasing number of observations. The width of the posterior pdf also becomes narrower with more data, indicating that we are becoming increasingly confident in our estimate of the bias-weighting. For the coin in this example, the best estimate of \( H \) eventually converges to 0.6, which, of course, was the value chosen to simulate the flips. + +

+

+ +

+ + +
+ + + + + + + +
+ +
+ + + + + + diff --git a/doc/pub/Bayesian/html/._Bayesian-bs018.html b/doc/pub/Bayesian/html/._Bayesian-bs018.html new file mode 100644 index 000000000..66b5901d5 --- /dev/null +++ b/doc/pub/Bayesian/html/._Bayesian-bs018.html @@ -0,0 +1,208 @@ + + + + + + + + +Data Analysis and Machine Learning: Elements of Bayesian theory and Bayesian Neural Networks + + + + + + + + + + + + + + + + + + + + + + + +
+
+ + Data Analysis and Machine Learning: Elements of Bayesian theory and Bayesian Neural Networks +
+ +
+ +
+
+ + +
+ +

 

 

 

+ + + + +

A few words on different priors

+ +
    +
  • uniform
    • +
  • Gaussian
    • +
  • Jeffrey's prior
    • +
+ +Repeat the coin flipping experiment with other priors. + +

+

+ +

+ + +
+ + + + + + + +
+ +
+ + + + + + diff --git a/doc/pub/Bayesian/html/._Bayesian-bs019.html b/doc/pub/Bayesian/html/._Bayesian-bs019.html new file mode 100644 index 000000000..153763ece --- /dev/null +++ b/doc/pub/Bayesian/html/._Bayesian-bs019.html @@ -0,0 +1,213 @@ + + + + + + + + +Data Analysis and Machine Learning: Elements of Bayesian theory and Bayesian Neural Networks + + + + + + + + + + + + + + + + + + + + + + + +
+
+ + Data Analysis and Machine Learning: Elements of Bayesian theory and Bayesian Neural Networks +
+ +
+ +
+
+ + +
+ +

 

 

 

+ + + + +

Bayesian parameter estimation (single parameter)

+
+
+

+We will now consider the very important task of model parameter estimation using statistical inference. + + +(CF 1: maybe stress that model parameters are not random variables, and the meaning of parameter estimation is therefore very different between frequentist and bayesian approaches.) + + +

+Throughout this section we will consider a specific example that involves a model with a single parameter: "Measured flux from a star". +

+
+ + +

+

+ +

+ + +
+ + + + + + + +
+ +
+ + + + + + diff --git a/doc/pub/Bayesian/html/._Bayesian-bs020.html b/doc/pub/Bayesian/html/._Bayesian-bs020.html new file mode 100644 index 000000000..74b131463 --- /dev/null +++ b/doc/pub/Bayesian/html/._Bayesian-bs020.html @@ -0,0 +1,470 @@ + + + + + + + + +Data Analysis and Machine Learning: Elements of Bayesian theory and Bayesian Neural Networks + + + + + + + + + + + + + + + + + + + + + + + +
+
+ + Data Analysis and Machine Learning: Elements of Bayesian theory and Bayesian Neural Networks +
+ +
+ +
+
+ + +
+ +

 

 

 

+ + + + +

Example: Measured flux from a star

+ +Adapted from the blog Pythonic Perambulations by Jake VanderPlas. + +

+Imagine that we point our telescope to the sky, and observe the light coming from a single star. For the time being, we'll assume that the star's true flux is constant with time, i.e. that is it has a fixed value \( F_\mathrm{true} \) (we'll also ignore effects like sky noise and other sources of systematic error). We'll assume that we perform a series of \( N \) measurements with our telescope, where the ith measurement reports the observed photon flux \( F_i \) and error \( e_i \) . +The question is, given this set of measurements \( D = \{F_i, e_i\} \), what is our best estimate of the true flux \( F_\mathrm{true} \)? + +

2: We'll make the reasonable assumption that errors are Gaussian. In a Frequentist perspective, \( e_i \) is the standard deviation of the results of a single measurement event in the limit of repetitions of that event. In the Bayesian perspective, \( e_i \) is the standard deviation of the (Gaussian) probability distribution describing our knowledge of that particular measurement given its observed value.

+ +

+Because the measurements are number counts, a Poisson distribution is a good approximation to the measurement process: + +

+ + +

np.random.seed(1)      # for repeatability
+F_true = 1000          # true flux, say number of photons measured in 1 second
+N = 50                 # number of measurements
+F = stats.poisson(F_true).rvs(N)
+                       # N measurements of the flux
+e = np.sqrt(F)         # errors on Poisson counts estimated via square root
+
+

+Now let's make a simple visualization of the "observed" data, see Fig. 2. + +

+ + +

fig, ax = plt.subplots()
+ax.errorbar(F, np.arange(N), xerr=e, fmt='ok', ecolor='gray', alpha=0.5)
+ax.vlines([F_true], 0, N, linewidth=5, alpha=0.2)
+ax.set_xlabel("Flux");ax.set_ylabel("measurement number");
+
+

+

+
+

Figure 2: Single photon counts (flux measurements).

+

+
+ +

+These measurements each have a different error \( e_i \) which is estimated from Poisson statistics using the standard square-root rule. In this toy example we already know the true flux \( F_\mathrm{true} \), but the question is this: given our measurements and errors, what is our best estimate of the true flux? + +

+Let's take a look at the frequentist and Bayesian approaches to solving this. + +

Simple Photon Counts: Frequentist Approach

+ +We'll start with the classical frequentist maximum likelihood approach. Given a single observation \( D_i = (F_i, e_i) \), we can compute the probability distribution of the measurement given the true flux Ftrue given our assumption of Gaussian errors +$$ +\begin{equation} +p(D_i | F_\mathrm{true}, I) = \frac{1}{\sqrt{2\pi e_i^2}} \exp \left( \frac{-(F_i-F_\mathrm{true})^2}{2e_i^2} \right). +\tag{5} +\end{equation} +$$ + +This should be read "the probability of \( D_i \) given \( F_\mathrm{true} \) +equals ...". You should recognize this as a normal distribution with mean \( F_\mathrm{true} \) and standard deviation \( e_i \). + +

+We construct the likelihood function by computing the product of the probabilities for each data point +$$ +\begin{equation} +\mathcal{L}(D | F_\mathrm{true}, I) = \prod_{i=1}^N p(D_i | F_\mathrm{true}, I), +\tag{6} +\end{equation} +$$ + +here \( D = \{D_i\} \) represents the entire set of measurements. Because the value of the likelihood can become very small, it is often more convenient to instead compute the log-likelihood. Combining the previous two equations and computing the log, we have +$$ +\begin{equation} +\log\mathcal{L} = -\frac{1}{2} \sum_{i=1}^N \left[ \log(2\pi e_i^2) + \frac{(F_i-F_\mathrm{true})^2}{e_i^2} \right]. +\tag{7} +\end{equation} +$$ + +

+What we'd like to do is determine \( F_\mathrm{true} \) such that the likelihood is maximized. For this simple problem, the maximization can be computed analytically (i.e. by setting \( d\log\mathcal{L}/d F_\mathrm{true} = 0 \)). This results in the following observed estimate of \( F_\mathrm{true} \) +$$ +\begin{equation} +F_\mathrm{est} = \sum_{i=1}^N w_i F_i; \quad w_i = 1/e_i^2. +\tag{8} +\end{equation} +$$ + +Notice that in the special case of all errors \( e_i \) being equal, this reduces to +$$ +\begin{equation} +F_\mathrm{est} = \frac{1}{N} \sum_{i=1} F_i. +\tag{9} +\end{equation} +$$ + +That is, in agreement with intuition, \( F_\mathrm{est} \) is simply the mean of the observed data when errors are equal. + +

+We can go further and ask what the error of our estimate is. In the frequentist approach, this can be accomplished by fitting a Gaussian approximation to the likelihood curve at maximum; in this simple case this can also be solved analytically (the sum of Gaussians is also a Gaussian). It can be shown that the standard deviation of this Gaussian approximation is +$$ +\begin{equation} +\sigma_\mathrm{est} = \sum_{i=1}^N w_i. +\tag{10} +\end{equation} +$$ + +These results are fairly simple calculations; let's evaluate them for our toy dataset: + +

+ + +

w=1./e**2
+print("""
+F_true = {0}
+F_est = {1:.0f} +/- {2:.0f} (based on {3} measurements) """\ 
+          .format(F_true, (w * F).sum() / w.sum(), w.sum() ** -0.5, N))
+
+

+F_true = 1000
+F_est = 998 +/- 4 (based on 50 measurements)
+ +

+We find that for 50 measurements of the flux, our estimate has an error of about 0.4% and is consistent with the input value. + +

Simple Photon Counts: Bayesian Approach

+ +The Bayesian approach, as you might expect, begins and ends with probabilities. Our hypothesis is that the star has a constant flux \( F_\mathrm{true} \). It recognizes that what we fundamentally want to compute is our knowledge of the parameters in question given the data and other information (such as our knowledge of uncertainties for the observed values), i.e. in this case, \( p(F_\mathrm{true} | D,I) \). +Note that this formulation of the problem is fundamentally contrary to the frequentist philosophy, which says that probabilities have no meaning for model parameters like \( F_\mathrm{true} \). Nevertheless, within the Bayesian philosophy this is perfectly acceptable. + +

+To compute this result, Bayesians next apply Bayes' Theorem (1). +If we set the prior \( p(F_\mathrm{true}|I) \propto 1 \) (a flat prior), we find +\( p(F_\mathrm{true}|D,I) \propto p(D | F_\mathrm{true},I) \equiv \mathcal{L}(D | F_\mathrm{true},I) \) +and the Bayesian probability is maximized at precisely the same value as the frequentist result! So despite the philosophical differences, we see that (for this simple problem at least) the Bayesian and frequentist point estimates are equivalent. + +

A note about priors

+ +The prior allows inclusion of other information into the computation, which becomes very useful in cases where multiple measurement strategies are being combined to constrain a single model. The necessity to specify a prior, however, is one of the more controversial pieces of Bayesian analysis. +A frequentist will point out that the prior is problematic when no true prior information is available. Though it might seem straightforward to use a noninformative prior like the flat prior mentioned above, there are some surprisingly subtleties involved. It turns out that in many situations, a truly noninformative prior does not exist! Frequentists point out that the subjective choice of a prior which necessarily biases your result has no place in statistical data analysis. +A Bayesian would counter that frequentism doesn't solve this problem, but simply skirts the question. Frequentism can often be viewed as simply a special case of the Bayesian approach for some (implicit) choice of the prior: a Bayesian would say that it's better to make this implicit choice explicit, even if the choice might include some subjectivity. + +

Simple Photon Counts: Bayesian approach in practice

+ +Leaving these philosophical debates aside for the time being, let's address how Bayesian results are generally computed in practice. For a one parameter problem like the one considered here, it's as simple as computing the posterior probability \( p(F_\mathrm{true} | D,I) \) as a function of \( F_\mathrm{true} \): this is the distribution reflecting our knowledge of the parameter \( F_\mathrm{true} \). +But as the dimension of the model grows, this direct approach becomes increasingly intractable. For this reason, Bayesian calculations often depend on sampling methods such as Markov Chain Monte Carlo (MCMC). For this practical example, let us apply an MCMC approach using Dan Foreman-Mackey's emcee package. Keep in mind here that the goal is to generate a set of points drawn from the posterior probability distribution, and to use those points to determine the answer we seek. +To perform this MCMC, we start by defining Python functions for the prior \( p(F_\mathrm{true} | I) \), the likelihood \( p(D | F_\mathrm{true},I) \), and the posterior \( p(F_\mathrm{true} | D,I) \), noting that none of these need be properly normalized. Our model here is one-dimensional, but to handle multi-dimensional models we'll define the model in terms of an array of parameters \( \boldsymbol{\alpha} \), which in this case is \( \boldsymbol{\alpha} = [F_\mathrm{true}] \) + +

+ + +

def log_prior(alpha):
+    return 0 # flat prior
+
+def log_likelihood(alpha, F, e):
+    return -0.5 * np.sum(np.log(2 * np.pi * e ** 2) \ 
+                             + (F - alpha[0]) ** 2 / e ** 2)
+                             
+def log_posterior(alpha, F, e):
+    return log_prior(alpha) + log_likelihood(alpha, F, e)
+
+

+Now we set up the problem, including generating some random starting guesses for the multiple chains of points. + +

+ + +

ndim = 1      # number of parameters in the model
+nwalkers = 50 # number of MCMC walkers
+nburn = 1000  # "burn-in" period to let chains stabilize
+nsteps = 2000 # number of MCMC steps to take
+# we'll start at random locations between 0 and 2000
+starting_guesses = 2000 * np.random.rand(nwalkers, ndim)
+sampler = emcee.EnsembleSampler(nwalkers, ndim, log_posterior, args=[F,e])
+sampler.run_mcmc(starting_guesses, nsteps)
+# Shape of sampler.chain  = (nwalkers, nsteps, ndim)
+# Flatten the sampler chain and discard burn-in points:
+samples = sampler.chain[:, nburn:, :].reshape((-1, ndim))
+
+

+If this all worked correctly, the array sample should contain a series of 50,000 points drawn from the posterior. Let's plot them and check. See results in Fig. 3. + +

+ + +

fig, ax = plt.subplots()
+ax.hist(samples, bins=50, histtype="stepfilled", alpha=0.3, normed=True)
+ax.set_xlabel(r'$F_\mathrm{est}$')
+ax.set_ylabel(r'$p(F_\mathrm{est}|D,I)$')
+
+

+

+
+

Figure 3: Bayesian posterior pdf (represented by a histogram of MCMC samples) from flux measurements.

+

+
+ +

Best estimates and confidence intervals

+ +The posterior distribution from our Bayesian data analysis is the key quantity that encodes our inference about the values of the model parameters, given the data and the relevant background information. Often, however, we wish to summarize this result with just a few numbers: the best estimate and a measure of its reliability. + +

+There are a few different options for this. The choice of the most appropriate one depends mainly on the shape of the posterior distribution: + +

+Symmetric posterior pdfs: Since the probability (density) associated with any particular value of the parameter is a measure of how much we believe that it lies in the neighbourhood of that point, our best estimate is given by the maximum of the posterior pdf. If we denote the quantity of interest by \( X \), with a posterior pdf \( P =p(X|D,I) \), then the best estimate of its value \( X_0 \) is given by the condition \( dP/dX|_{X=X_0}=0 \). Strictly speaking, we should also check the sign of the second derivative to ensure that \( X_0 \) represents a maximum. + +

+To obtain a measure of the reliability of this best estimate, we need to look at the width or spread of the posterior pdf about \( X_0 \). When considering the behaviour of any function in the neighbourhood of a particular point, it is often helpful to carry out a Taylor series expansion; this is simply a standard tool for (locally) approximating a complicated function by a low-order polynomial. The linear term is zero at the maximum and the quadratic term is often the dominating one determining the width of the posterior pdf. Ignoring all the higher-order terms we arrive at the Gaussian approximation +$$ +\begin{equation} +p(X|D,I) \approx \frac{1}{\sigma\sqrt{2\pi}} \exp \left[ -\frac{(x-\mu)^2}{2\sigma^2} \right], +\tag{11} +\end{equation} +$$ + +where the mean \( \mu = X_0 \) and the variance \( \sigma = \left( - \left. \frac{d^2L}{dX^2} \right|_{X_0} \right)^{-1/2} \), where \( L \) is the logarithm of the posterior \( P \). Our inference about the quantity of interest is conveyed very concisely, therefore, by the statement \( X = X_0 \pm \sigma \), and +$$ +$$ +p(X_0-\sigma < X < X_0+\sigma | D,I) = \int_{X_0-\sigma}^{X_0+\sigma} p(X|D,I) dX \approx 0.67. +$$ +$$ + +

+Asymmetric posterior pdfs: While the maximum of the posterior (\( X_0 \)) can still be regarded as giving the best estimate, the true value is now more likely to be on one side of this rather than the other. Alternatively one can compute the mean value, \( \langle X \rangle = \int X p(X|D,I) dX \), although this tends to overemphasise very long tails. The best option is probably a compromise that can be employed when having access to a large sample from the posterior (as provided by an MCMC), namely to give the median of this ensamble. + +

+Furthermore, the concept of an error-bar does not seem appropriate in this case, as it implicitly entails the idea of symmetry. A good way of expressing the reliability with which a parameter can be inferred, for an asymmetric posterior pdf, is rather through a confidence interval. Since the area under the posterior pdf between \( X_1 \) and \( X_2 \) is proportional to how much we believe that \( X \) lies in that range, the shortest interval that encloses 67% of the area represents a sensible measure of the uncertainty of the estimate. Obviously we can choose to provide some other degree-of-belief that we think is relevant for the case at hand. Assuming that the posterior pdf has been normalized, to have unit area, we need to find \( X_1 \) and \( X_2 \) such that: +$$ +$$ +p(X_1 < X < X_2 | D,I) = \int_{X_1}^{X_2} p(X|D,I) dX \approx 0.67, +$$ +$$ + +where the difference \( X_2 - X_1 \) is as small as possible. The region \( X_1 < X < X_2 \) is then called the shortest 67% confidence interval. + +

+Multimodal posterior pdfs: We can sometimes obtain posteriors which are multimodal; i.e. contains several disconnected regions with large probabilities. There is no difficulty when one of the maxima is very much larger than the others: we can simply ignore the subsidiary solutions, to a good approximation, and concentrate on the global maximum. The problem arises when there are several maxima of comparable magnitude. What do we now mean by a best estimate, and how should we quantify its reliability? The idea of a best estimate and an error-bar, or even a confidence interval, is merely an attempt to summarize the posterior with just two or three numbers; sometimes this just can’t be done, and so these concepts are not valid. For the bimodal case we might be able to characterize the posterior in terms of a few numbers: two best estimates and their associated error-bars, or disjoint confidence intervals. For a general multimodal pdf, the most honest thing we can do is just display the posterior itself. + +

Simple Photon Counts: Best estimates and confidence intervals

+ +To compute these numbers for our example, you would run: + +

+ + +

sampper=np.percentile(samples, [2.5, 16.5, 50, 83.5, 97.5],axis=0).flatten()
+print("""
+F_true = {0}
+Based on {1} measurements the posterior point estimates are:
+...F_est = {2:.0f} +/- {3:.0f}
+or using credible intervals:
+...F_est = {4:.0f}          (posterior median) 
+...F_est in [{5:.0f}, {6:.0f}] (67% credible interval) 
+...F_est in [{7:.0f}, {8:.0f}] (95% credible interval) """\ 
+          .format(F_true, N, np.mean(samples), np.std(samples), \ 
+                      sampper[2], sampper[1], sampper[3], sampper[0], sampper[4]))
+
+

+F_true = 1000
+Based on 50 measurements the posterior point estimates are:
+...F_est = 998 +/- 4
+or using credible intervals:
+...F_est = 998 (posterior median)
+...F_est in [993, 1002] (67% credible interval)
+...F_est in [989, 1006] (95% credible interval)
+ +

+In this particular example, the posterior pdf is actually a Gaussian (since it is constructed as a product of Gaussians), and the mean and variance from the quadratic approximation will agree exactly with the frequentist approach. + +

+From this final result you might come away with the impression that the Bayesian method is unnecessarily complicated, and in this case it certainly is. Using an MCMC sampler to characterize a one-dimensional normal distribution is a bit like using the Death Star to destroy a beach ball, but we did this here because it demonstrates an approach that can scale to complicated posteriors in many, many dimensions, and can provide nice results in more complicated situations where an analytic likelihood approach is not possible. + +

+Furthermore, as data and models grow in complexity, the two approaches can diverge greatly. + +

+

+ +

+ + +
+ + + + + + + +
+ +
+ + + + + + diff --git a/doc/pub/Bayesian/html/._Bayesian-bs021.html b/doc/pub/Bayesian/html/._Bayesian-bs021.html new file mode 100644 index 000000000..60c66e3c2 --- /dev/null +++ b/doc/pub/Bayesian/html/._Bayesian-bs021.html @@ -0,0 +1,210 @@ + + + + + + + + +Data Analysis and Machine Learning: Elements of Bayesian theory and Bayesian Neural Networks + + + + + + + + + + + + + + + + + + + + + + + +
+
+ + Data Analysis and Machine Learning: Elements of Bayesian theory and Bayesian Neural Networks +
+ +
+ +
+
+ + +
+ +

 

 

 

+ + + + +

Bayesian parameter estimation (multiple parameters, covariance)

+
+
+

+ +

    +
  • multidimensional posterior pdf:s
    • +
  • nuisance parameters (e.g. background subtraction?)
    • +
  • corner plots, covariance, correlations
    • +
  • best example?
    • +
+
+
+ + +

+

+ +

+ + +
+ + + + + + + +
+ +
+ + + + + + diff --git a/doc/pub/Bayesian/html/._Bayesian-bs022.html b/doc/pub/Bayesian/html/._Bayesian-bs022.html new file mode 100644 index 000000000..e65b8c033 --- /dev/null +++ b/doc/pub/Bayesian/html/._Bayesian-bs022.html @@ -0,0 +1,206 @@ + + + + + + + + +Data Analysis and Machine Learning: Elements of Bayesian theory and Bayesian Neural Networks + + + + + + + + + + + + + + + + + + + + + + + +
+
+ + Data Analysis and Machine Learning: Elements of Bayesian theory and Bayesian Neural Networks +
+ +
+ +
+
+ + +
+ +

 

 

 

+ + + + +

Bayesian model selection

+
+
+

+ +

    +
  • Bayesian evidence
    • +
  • Occam's razor
    • +
  • Best example? How many spectral lines are there?
    • +
+
+
+ + +

+ +

+ + +
+ + + + + + + +
+ +
+ + + + + + diff --git a/doc/pub/Bayesian/html/reveal.js/css/theme/template/mixins.scss b/doc/pub/Bayesian/html/reveal.js/css/theme/template/mixins.scss new file mode 100644 index 000000000..e0c560692 --- /dev/null +++ b/doc/pub/Bayesian/html/reveal.js/css/theme/template/mixins.scss @@ -0,0 +1,29 @@ +@mixin vertical-gradient( $top, $bottom ) { + background: $top; + background: -moz-linear-gradient( top, $top 0%, $bottom 100% ); + background: -webkit-gradient( linear, left top, left bottom, color-stop(0%,$top), color-stop(100%,$bottom) ); + background: -webkit-linear-gradient( top, $top 0%, $bottom 100% ); + background: -o-linear-gradient( top, $top 0%, $bottom 100% ); + background: -ms-linear-gradient( top, $top 0%, $bottom 100% ); + background: linear-gradient( top, $top 0%, $bottom 100% ); +} + +@mixin horizontal-gradient( $top, $bottom ) { + background: $top; + background: -moz-linear-gradient( left, $top 0%, $bottom 100% ); + background: -webkit-gradient( linear, left top, right top, color-stop(0%,$top), color-stop(100%,$bottom) ); + background: -webkit-linear-gradient( left, $top 0%, $bottom 100% ); + background: -o-linear-gradient( left, $top 0%, $bottom 100% ); + background: -ms-linear-gradient( left, $top 0%, $bottom 100% ); + background: linear-gradient( left, $top 0%, $bottom 100% ); +} + +@mixin radial-gradient( $outer, $inner, $type: circle ) { + background: $outer; + background: -moz-radial-gradient( center, $type cover, $inner 0%, $outer 100% ); + background: -webkit-gradient( radial, center center, 0px, center center, 100%, color-stop(0%,$inner), color-stop(100%,$outer) ); + background: -webkit-radial-gradient( center, $type cover, $inner 0%, $outer 100% ); + background: -o-radial-gradient( center, $type cover, $inner 0%, $outer 100% ); + background: -ms-radial-gradient( center, $type cover, $inner 0%, $outer 100% ); + background: radial-gradient( center, $type cover, $inner 0%, $outer 100% ); +} \ No newline at end of file diff --git a/doc/pub/Bayesian/html/reveal.js/css/theme/template/settings.scss b/doc/pub/Bayesian/html/reveal.js/css/theme/template/settings.scss new file mode 100644 index 000000000..e706e19c5 --- /dev/null +++ b/doc/pub/Bayesian/html/reveal.js/css/theme/template/settings.scss @@ -0,0 +1,34 @@ +// Base settings for all themes that can optionally be +// overridden by the super-theme + +// Background of the presentation +$backgroundColor: #2b2b2b; + +// Primary/body text +$mainFont: 'Lato', sans-serif; +$mainFontSize: 30px; /* changed (by hpl) from 36px; */ +$mainColor: #eee; + +// Headings +$headingMargin: 0 0 20px 0; +$headingFont: 'League Gothic', Impact, sans-serif; +$headingColor: #eee; +$headingLineHeight: 0.9em; +$headingLetterSpacing: 0.02em; +$headingTextTransform: none; /* changed (by hpl) from uppercase; */ +$headingTextShadow: 0px 0px 6px rgba(0,0,0,0.2); +$heading1TextShadow: $headingTextShadow; + +// Links and actions +$linkColor: #13DAEC; +$linkColorHover: lighten( $linkColor, 20% ); + +// Text selection +$selectionBackgroundColor: #FF5E99; +$selectionColor: #fff; + +// Generates the presentation background, can be overridden +// to return a background image or gradient +@mixin bodyBackground() { + background: $backgroundColor; +} \ No newline at end of file diff --git a/doc/pub/Bayesian/html/reveal.js/css/theme/template/theme.scss b/doc/pub/Bayesian/html/reveal.js/css/theme/template/theme.scss new file mode 100644 index 000000000..b1dbc2736 --- /dev/null +++ b/doc/pub/Bayesian/html/reveal.js/css/theme/template/theme.scss @@ -0,0 +1,171 @@ +// Base theme template for reveal.js + +/********************************************* + * GLOBAL STYLES + *********************************************/ + +body { + @include bodyBackground(); + background-color: $backgroundColor; +} + +.reveal { + font-family: $mainFont; + font-size: $mainFontSize; + font-weight: normal; + letter-spacing: -0.02em; + color: $mainColor; +} + +::selection { + color: $selectionColor; + background: $selectionBackgroundColor; + text-shadow: none; +} + +/********************************************* + * HEADERS + *********************************************/ + +.reveal h1, +.reveal h2, +.reveal h3, +.reveal h4, +.reveal h5, +.reveal h6 { + margin: $headingMargin; + color: $headingColor; + + font-family: $headingFont; + line-height: $headingLineHeight; + letter-spacing: $headingLetterSpacing; + + text-transform: $headingTextTransform; + text-shadow: $headingTextShadow; +} + +.reveal h1 { + line-height: 1.2em; /* added by hpl */ + text-shadow: $heading1TextShadow; +} + + +/********************************************* + * LINKS + *********************************************/ + +.reveal a:not(.image) { + color: $linkColor; + text-decoration: none; + + -webkit-transition: color .15s ease; + -moz-transition: color .15s ease; + -ms-transition: color .15s ease; + -o-transition: color .15s ease; + transition: color .15s ease; +} + .reveal a:not(.image):hover { + color: $linkColorHover; + + text-shadow: none; + border: none; + } + +.reveal .roll span:after { + color: #fff; + background: darken( $linkColor, 15% ); +} + + +/********************************************* + * IMAGES + *********************************************/ + +.reveal section img { + margin: 15px 0px; + background: rgba(255,255,255,0.12); + border: 4px solid $mainColor; + + box-shadow: 0 0 10px rgba(0, 0, 0, 0.15); + + -webkit-transition: all .2s linear; + -moz-transition: all .2s linear; + -ms-transition: all .2s linear; + -o-transition: all .2s linear; + transition: all .2s linear; +} + + .reveal a:hover img { + background: rgba(255,255,255,0.2); + border-color: $linkColor; + + box-shadow: 0 0 20px rgba(0, 0, 0, 0.55); + } + + +/********************************************* + * NAVIGATION CONTROLS + *********************************************/ + +.reveal .controls div.navigate-left, +.reveal .controls div.navigate-left.enabled { + border-right-color: $linkColor; +} + +.reveal .controls div.navigate-right, +.reveal .controls div.navigate-right.enabled { + border-left-color: $linkColor; +} + +.reveal .controls div.navigate-up, +.reveal .controls div.navigate-up.enabled { + border-bottom-color: $linkColor; +} + +.reveal .controls div.navigate-down, +.reveal .controls div.navigate-down.enabled { + border-top-color: $linkColor; +} + +.reveal .controls div.navigate-left.enabled:hover { + border-right-color: $linkColorHover; +} + +.reveal .controls div.navigate-right.enabled:hover { + border-left-color: $linkColorHover; +} + +.reveal .controls div.navigate-up.enabled:hover { + border-bottom-color: $linkColorHover; +} + +.reveal .controls div.navigate-down.enabled:hover { + border-top-color: $linkColorHover; +} + + +/********************************************* + * PROGRESS BAR + *********************************************/ + +.reveal .progress { + background: rgba(0,0,0,0.2); +} + .reveal .progress span { + background: $linkColor; + + -webkit-transition: width 800ms cubic-bezier(0.260, 0.860, 0.440, 0.985); + -moz-transition: width 800ms cubic-bezier(0.260, 0.860, 0.440, 0.985); + -ms-transition: width 800ms cubic-bezier(0.260, 0.860, 0.440, 0.985); + -o-transition: width 800ms cubic-bezier(0.260, 0.860, 0.440, 0.985); + transition: width 800ms cubic-bezier(0.260, 0.860, 0.440, 0.985); + } + +/********************************************* + * SLIDE NUMBER + *********************************************/ +.reveal .slide-number { + color: $linkColor; +} + + diff --git a/doc/pub/How2ReadData/html/How2ReadData-bs.html b/doc/pub/How2ReadData/html/How2ReadData-bs.html index 5ac65ab37..a47e06046 100644 --- a/doc/pub/How2ReadData/html/How2ReadData-bs.html +++ b/doc/pub/How2ReadData/html/How2ReadData-bs.html @@ -42,25 +42,40 @@ Automatically generated HTML file from DocOnce source @@ -99,19 +114,29 @@ MathJax.Hub.Config({ Contents @@ -145,7 +170,7 @@ MathJax.Hub.Config({
[2] Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University

-

Jan 14, 2019

+

Aug 13, 2019


@@ -166,45 +191,147 @@ and use these to introduce statistical data analysis concepts and machine learning algorithms before we delve into the algorithms themselves. The examples we will use in the beginning, start with simple polynomials with random noise added. We will use the Python -software package Scikit-learn and +software package Scikit-Learn and introduce various machine learning algorithms to make fits of the data and predictions. We move thereafter to more interesting -cases such as the simulation of financial transactions or disease -models. These are examples where we can easily set up the data and +cases such as nuclear binding energies. +These are examples where we can easily set up the data and then use machine learning algorithms included in for example -scikit-learn. +Scikit-Learn.

These examples will serve us the purpose of getting started. Furthermore, they allow us to catch more than two birds with a stone. They will allow us to bring in some programming specific -topics and tools as well as showing the power of various Python (and -R) packages for machine learning and statistical data analysis. In the -lectures on linear algebra we cover in more detail various programming -features of languages like Python and C++ (and other), we will also -look into more specific linear functions which are relevant for the -various algorithms we will discuss. Here, we will mainly focus on two -specific Python packages for Machine Learning, scikit-learn and -tensorflow (see below for links etc). Moreover, the examples we +topics and tools as well as showing the power of various Python +packages for machine learning and statistical data analysis. + +

+Here, we will mainly focus on two +specific Python packages for Machine Learning, Scikit-Learn and +Tensorflow (see below for links etc). Moreover, the examples we introduce will serve as inputs to many of our discussions later, as well as allowing you to set up models and produce your own data and get started with programming. -

Software and needed installations

+

What is Machine Learning?

+ +

+Statistics, data science and machine learning form important fields of +research in modern science. They describe how to learn and make +predictions from data, as well as allowing us to extract important +correlations about physical process and the underlying laws of motion +in large data sets. The latter, big data sets, appear frequently in +essentially all disciplines, from the traditional Science, Technology, +Mathematics and Engineering fields to Life Science, Law, education +research, the Humanities and the Social Sciences. + +

+It has become more +and more common to see research projects on big data in for example +the Social Sciences where extracting patterns from complicated survey +data is one of many research directions. Having a solid grasp of data +analysis and machine learning is thus becoming central to scientific +computing in many fields, and competences and skills within the fields +of machine learning and scientific computing are nowadays strongly +requested by many potential employers. The latter cannot be +overstated, familiarity with machine learning has almost become a +prerequisite for many of the most exciting employment opportunities, +whether they are in bioinformatics, life science, physics or finance, +in the private or the public sector. This author has had several +students or met students who have been hired recently based on their +skills and competences in scientific computing and data science, often +with marginal knowledge of machine learning. + +

+Machine learning is a subfield of computer science, and is closely +related to computational statistics. It evolved from the study of +pattern recognition in artificial intelligence (AI) research, and has +made contributions to AI tasks like computer vision, natural language +processing and speech recognition. Many of the methods we will study are also +strongly rooted in basic mathematics and physics research. + +

+Ideally, machine learning represents the science of giving computers +the ability to learn without being explicitly programmed. The idea is +that there exist generic algorithms which can be used to find patterns +in a broad class of data sets without having to write code +specifically for each problem. The algorithm will build its own logic +based on the data. You should however always keep in mind that +machines and algorithms are to a large extent developed by humans. The +insights and knowledge we have about a specific system, play a central +role when we develop a specific machine learning algorithm. + +

+Machine learning is an extremely rich field, in spite of its young +age. The increases we have seen during the last three decades in +computational capabilities have been followed by developments of +methods and techniques for analyzing and handling large date sets, +relying heavily on statistics, computer science and mathematics. The +field is rather new and developing rapidly. Popular software packages +written in Python for machine learning like +Scikit-learn, +Tensorflow, +PyTorch and Keras, all +freely available at their respective GitHub sites, encompass +communities of developers in the thousands or more. And the number of +code developers and contributors keeps increasing. Not all the +algorithms and methods can be given a rigorous mathematical +justification, opening up thereby large rooms for experimenting and +trial and error and thereby exciting new developments. However, a +solid command of linear algebra, multivariate theory, probability +theory, statistical data analysis, understanding errors and Monte +Carlo methods are central elements in a proper understanding of many +of algorithms and methods we will discuss. + +

Types of Machine Learning

+ +

+The approaches to machine learning are many, but are often split into +two main categories. In supervised learning we know the answer to a +problem, and let the computer deduce the logic behind it. On the other +hand, unsupervised learning is a method for finding patterns and +relationship in data sets without any prior knowledge of the system. +Some authours also operate with a third category, namely +reinforcement learning. This is a paradigm of learning inspired by +behavioral psychology, where learning is achieved by trial-and-error, +solely from rewards and punishment. + +

+Another way to categorize machine learning tasks is to consider the +desired output of a system. Some of the most common tasks are: + +

    +
  • Classification: Outputs are divided into two or more classes. The goal is to produce a model that assigns inputs into one of these classes. An example is to identify digits based on pictures of hand-written ones. Classification is typically supervised learning.
    • +
  • Regression: Finding a functional relationship between an input data set and a reference data set. The goal is to construct a function that maps input data to continuous output values.
    • +
  • Clustering: Data are divided into groups with certain common traits, without knowing the different groups beforehand. It is thus a form of unsupervised learning.
    • +
+ +The methods we cover have three main topics in common, irrespective of +whether we deal with supervised or unsupervised learning. The first +ingredient is normally our data set (which can be subdivided into +training and test data), the second item is a model which is normally a +function of some parameters. The model reflects our knowledge of the system (or lack thereof). As an example, if we know that our data show a behavior similar to what would be predicted by a polynomial, fitting our data to a polynomial of some degree would then determin our model. + +

+The last ingredient is a so-called cost +function which allows us to present an estimate on how good our model +is in reproducing the data it is supposed to train. +At the heart of basically all ML algorithms there are so-called minimization algorithms, often we end up with various variants of gradient methods. + +

Software and needed installations

We will make extensive use of Python as programming language and its myriad of available libraries. You will find -IPython/Jupyter notebooks invaluable in your work. You can run R +Jupyter notebooks invaluable in your work. You can run R codes in the Jupyter/IPython notebooks, with the immediate benefit of visualizing your data. You can also use compiled languages like C++, -Rust, Fortran etc if you prefer. The focus in these lectures will be -on Python, but we will provide many code examples for those of you who -prefer R or compiled languages. You can integrate C++ codes and R in for example -a Jupyter notebook. +Rust, Julia, Fortran etc if you prefer. The focus in these lectures will be +on Python.

-If you have Python installed (we recommend Python3) and you feel +If you have Python installed (we strongly recommend Python3) and you feel pretty familiar with installing different packages, we recommend that you install the following Python packages via pip as @@ -232,7 +359,7 @@ you can use pip as well and simply install Python as etc etc. -

Python installers

+

Python installers

If you don't want to perform these operations separately and venture @@ -259,28 +386,46 @@ distribution for scientific and analytic computing distribution and analysis environment, available for free and under a commercial license. -

Useful Python packages

-Here we list several useful Python packages. +

+Furthermore, Google's Colab is a free Jupyter notebook environment that requires +no setup and runs entirely in the cloud. Try it out! -

Installing R, C++, cython or Julia

+

Useful Python libraries

+Here we list several useful Python libraries we strongly recommend (if you use anaconda many of these are already there) + +
    +
  • NumPy is a highly popular library for large, multi-dimensional arrays and matrices, along with a large collection of high-level mathematical functions to operate on these arrays
    • +
  • The pandas library provides high-performance, easy-to-use data structures and data analysis tools
    • +
  • Xarray is a Python package that makes working with labelled multi-dimensional arrays simple, efficient, and fun!
    • +
  • Scipy (pronounced “Sigh Pie”) is a Python-based ecosystem of open-source software for mathematics, science, and engineering.
    • +
  • Matplotlib is a Python 2D plotting library which produces publication quality figures in a variety of hardcopy formats and interactive environments across platforms.
    • +
  • Autograd can automatically differentiate native Python and Numpy code. It can handle a large subset of Python's features, including loops, ifs, recursion and closures, and it can even take derivatives of derivatives of derivatives
    • +
  • SymPy is a Python library for symbolic mathematics.
    • +
  • scikit-learn has simple and efficient tools for machine learning, data mining and data analysis
    • +
  • TensorFlow is a Python library for fast numerical computing created and released by Google
    • +
  • Keras is a high-level neural networks API, written in Python and capable of running on top of TensorFlow, CNTK, or Theano
    • +
  • And many more such as pytorch, Theano etc
    • +
+ +

Installing R, C++, cython or Julia

-You will also find it convenient to utilize R. Although we will mainly -use Python during lectures and in various projects and exercises, we -provide a full R set of codes for the same examples. Those of you -already familiar with R should feel free to continue using R, keeping +You will also find it convenient to utilize R. We will mainly +use Python during lectures and in various projects and exercises. +Those of you +already familiar with R should feel free to continue using R, keeping however an eye on the parallel Python set ups. Similarly, if you are a -Python afecionado, feel free to explore R as well. Jupyter/Ipython +Python afecionado, feel free to explore R as well. Jupyter/Ipython notebook allows you to run R codes interactively in your browser. The software library R is tuned to statistically analysis and allows for an easy usage of the tools we will discuss in these -texts. +lectures.

To install R with Jupyter notebook follow the link here -

Installing R, C++, cython, Numba etc

+

Installing R, C++, cython, Numba etc

For the C++ aficionados, Jupyter/IPython notebook allows you also to @@ -291,13 +436,13 @@ languages.

To add more entropy, cython can also be used when running your -notebooks. It means that Python with the Jupyter/IPython notebook +notebooks. It means that Python with the jupyter notebook setup allows you to integrate widely popular softwares and tools for scientific computing. Similarly, the Numba Python package delivers increased performance capabilities with minimal rewrites of your codes. With its versatility, including symbolic operations, Python offers a unique -computational environment. Your Jupyter/IPython notebook can easily be +computational environment. Your jupyter notebook can easily be converted into a nicely rendered PDF file or a Latex file for further processing. For example, convert to latex as @@ -312,13 +457,522 @@ And to add more versatility, the Python package doconce you can convert a standard ascii text file into various HTML -formats, ipython notebooks, latex files, pdf files etc with minimal edits. +formats, ipython notebooks, latex files, pdf files etc with minimal edits. These lectures were generated using doconce. -

Simple linear regression model using scikit-learn

+

Numpy examples and Important Matrix and vector handling packages

-We start with perhaps our simplest possible example, using scikit-learn to perform linear regression analysis on a data set produced by us. -What follows is a simple Python code where we have defined function \( y \) in terms of the variable \( x \). Both are defined as vectors of dimension \( 1\times 100 \). The entries to the vector \( \hat{x} \) are given by random numbers generated with a uniform distribution with entries \( x_i \in [0,1] \) (more about probability distribution functions later). These values are then used to define a function \( y(x) \) (tabulated again as a vector) with a linear dependence on \( x \) plus a random noise added via the normal distribution. +There are several central software packages for linear algebra and eigenvalue problems. Several of the more +popular ones have been wrapped into ofter software packages like those from the widely used text Numerical Recipes. The original source codes in many of the available packages are often taken from the widely used +software package LAPACK, which follows two other popular packages +developed in the 1970s, namely EISPACK and LINPACK. We describe them shortly here. + +

    +
  • LINPACK: package for linear equations and least square problems.
    • +
  • LAPACK:package for solving symmetric, unsymmetric and generalized eigenvalue problems. From LAPACK's website http://www.netlib.org it is possible to download for free all source codes from this library. Both C/C++ and Fortran versions are available.
    • +
  • BLAS (I, II and III): (Basic Linear Algebra Subprograms) are routines that provide standard building blocks for performing basic vector and matrix operations. Blas I is vector operations, II vector-matrix operations and III matrix-matrix operations. Highly parallelized and efficient codes, all available for download from http://www.netlib.org.
    • +
+ +

Basic Matrix Features

+ +

+

+
+

+$$ + \mathbf{A} = + \begin{bmatrix} a_{11} & a_{12} & a_{13} & a_{14} \\ + a_{21} & a_{22} & a_{23} & a_{24} \\ + a_{31} & a_{32} & a_{33} & a_{34} \\ + a_{41} & a_{42} & a_{43} & a_{44} + \end{bmatrix}\qquad +\mathbf{I} = + \begin{bmatrix} 1 & 0 & 0 & 0 \\ + 0 & 1 & 0 & 0 \\ + 0 & 0 & 1 & 0 \\ + 0 & 0 & 0 & 1 + \end{bmatrix} +$$ +

+
+ + +

Basic Matrix Features

+
+
+

+ +

+The inverse of a matrix is defined by + +$$ +\mathbf{A}^{-1} \cdot \mathbf{A} = I +$$ +

+
+ + +

Basic Matrix Features

+ +

+

+
+

+ +

+ +

+
+ + + + + + + + + + + +
Relations Name matrix elements
\( A = A^{T} \) symmetric \( a_{ij} = a_{ji} \)
\( A = \left (A^{T} \right )^{-1} \) real orthogonal \( \sum_k a_{ik} a_{jk} = \sum_k a_{ki} a_{kj} = \delta_{ij} \)
\( A = A^{ * } \) real matrix \( a_{ij} = a_{ij}^{ * } \)
\( A = A^{\dagger} \) hermitian \( a_{ij} = a_{ji}^{ * } \)
\( A = \left (A^{\dagger} \right )^{-1} \) unitary \( \sum_k a_{ik} a_{jk}^{ * } = \sum_k a_{ki}^{ * } a_{kj} = \delta_{ij} \)
+
+
+

+

+
+ + +

Some famous Matrices

+ +
    +
  • Diagonal if \( a_{ij}=0 \) for \( i\ne j \)
    • +
  • Upper triangular if \( a_{ij}=0 \) for \( i > j \)
    • +
  • Lower triangular if \( a_{ij}=0 \) for \( i < j \)
    • +
  • Upper Hessenberg if \( a_{ij}=0 \) for \( i > j+1 \)
    • +
  • Lower Hessenberg if \( a_{ij}=0 \) for \( i < j+1 \)
    • +
  • Tridiagonal if \( a_{ij}=0 \) for \( |i -j| > 1 \)
    • +
  • Lower banded with bandwidth \( p \): \( a_{ij}=0 \) for \( i > j+p \)
    • +
  • Upper banded with bandwidth \( p \): \( a_{ij}=0 \) for \( i < j+p \)
    • +
  • Banded, block upper triangular, block lower triangular....
    • +
+ +

Basic Matrix Features

+ +

+

+
+

+For an \( N\times N \) matrix \( \mathbf{A} \) the following properties are all equivalent + +

    +
  • If the inverse of \( \mathbf{A} \) exists, \( \mathbf{A} \) is nonsingular.
    • +
  • The equation \( \mathbf{Ax}=0 \) implies \( \mathbf{x}=0 \).
    • +
  • The rows of \( \mathbf{A} \) form a basis of \( R^N \).
    • +
  • The columns of \( \mathbf{A} \) form a basis of \( R^N \).
    • +
  • \( \mathbf{A} \) is a product of elementary matrices.
    • +
  • \( 0 \) is not eigenvalue of \( \mathbf{A} \).
    • +
+
+
+ + +

Numpy and arrays

+Numpy provides an easy way to handle arrays in Python. The standard way to import this library is as + +

+ + +

import numpy as np
+
+

+Here follows a simple example where we set up an array of ten elements, all determined by random numbers drawn according to the normal distribution, +

+ + +

n = 10
+x = np.random.normal(size=n)
+print(x)
+
+

+We defined a vector \( x \) with \( n=10 \) elements with its values given by the Normal distribution \( N(0,1) \). +Another alternative is to declare a vector as follows +

+ + +

import numpy as np
+x = np.array([1, 2, 3])
+print(x)
+
+

+Here we have defined a vector with three elements, with \( x_0=1 \), \( x_1=2 \) and \( x_2=3 \). Note that both Python and C++ +start numbering array elements from \( 0 \) and on. This means that a vector with \( n \) elements has a sequence of entities \( x_0, x_1, x_2, \dots, x_{n-1} \). We could also let (recommended) Numpy to compute the logarithms of a specific array as +

+ + +

import numpy as np
+x = np.log(np.array([4, 7, 8]))
+print(x)
+
+

+In the last example we used Numpy's unary function \( np.log \). This function is +highly tuned to compute array elements since the code is vectorized +and does not require looping. We normaly recommend that you use the +Numpy intrinsic functions instead of the corresponding log function +from Python's math module. The looping is done explicitely by the +np.log function. The alternative, and slower way to compute the +logarithms of a vector would be to write + +

+ + +

import numpy as np
+from math import log
+x = np.array([4, 7, 8])
+for i in range(0, len(x)):
+    x[i] = log(x[i])
+print(x)
+
+

+We note that our code is much longer already and we need to import the log function from the math module. +The attentive reader will also notice that the output is \( [1, 1, 2] \). Python interprets automagically our numbers as integers (like the automatic keyword in C++). To change this we could define our array elements to be double precision numbers as +

+ + +

import numpy as np
+x = np.log(np.array([4, 7, 8], dtype = np.float64))
+print(x)
+
+

+or simply write them as double precision numbers (Python uses 64 bits as default for floating point type variables), that is +

+ + +

import numpy as np
+x = np.log(np.array([4.0, 7.0, 8.0])
+print(x)
+
+

+To check the number of bytes (remember that one byte contains eight bits for double precision variables), you can use simple use the itemsize functionality (the array \( x \) is actually an object which inherits the functionalities defined in Numpy) as +

+ + +

import numpy as np
+x = np.log(np.array([4.0, 7.0, 8.0])
+print(x.itemsize)
+
+ +

Matrices in Python

+ +

+Having defined vectors, we are now ready to try out matrices. We can +define a \( 3 \times 3 \) real matrix \( \hat{A} \) as (recall that we user +lowercase letters for vectors and uppercase letters for matrices) + +

+ + +

import numpy as np
+A = np.log(np.array([ [4.0, 7.0, 8.0], [3.0, 10.0, 11.0], [4.0, 5.0, 7.0] ]))
+print(A)
+
+

+If we use the shape function we would get \( (3, 3) \) as output, that is verifying that our matrix is a \( 3\times 3 \) matrix. We can slice the matrix and print for example the first column (Python organized matrix elements in a row-major order, see below) as +

+ + +

import numpy as np
+A = np.log(np.array([ [4.0, 7.0, 8.0], [3.0, 10.0, 11.0], [4.0, 5.0, 7.0] ]))
+# print the first column, row-major order and elements start with 0
+print(A[:,0]) 
+
+

+We can continue this was by printing out other columns or rows. The example here prints out the second column +

+ + +

import numpy as np
+A = np.log(np.array([ [4.0, 7.0, 8.0], [3.0, 10.0, 11.0], [4.0, 5.0, 7.0] ]))
+# print the first column, row-major order and elements start with 0
+print(A[1,:]) 
+
+

+Numpy contains many other functionalities that allow us to slice, subdivide etc etc arrays. We strongly recommend that you look up the Numpy website for more details. Useful functions when defining a matrix are the np.zeros function which declares a matrix of a given dimension and sets all elements to zero +

+ + +

import numpy as np
+n = 10
+# define a matrix of dimension 10 x 10 and set all elements to zero
+A = np.zeros( (n, n) )
+print(A) 
+
+

+or initializing all elements to +

+ + +

import numpy as np
+n = 10
+# define a matrix of dimension 10 x 10 and set all elements to one
+A = np.ones( (n, n) )
+print(A) 
+
+

+or as unitarily distributed random numbers (see the material on random number generators in the statistics part) +

+ + +

import numpy as np
+n = 10
+# define a matrix of dimension 10 x 10 and set all elements to random numbers with x \in [0, 1]
+A = np.random.rand(n, n)
+print(A) 
+
+

+As we will see throughout these lectures, there are several extremely useful functionalities in Numpy. +As an example, consider the discussion of the covariance matrix. Suppose we have defined three vectors +\( \hat{x}, \hat{y}, \hat{z} \) with \( n \) elements each. The covariance matrix is defined as +$$ +\hat{\Sigma} = \begin{bmatrix} \sigma_{xx} & \sigma_{xy} & \sigma_{xz} \\ + \sigma_{yx} & \sigma_{yy} & \sigma_{yz} \\ + \sigma_{zx} & \sigma_{zy} & \sigma_{zz} + \end{bmatrix}, +$$ + +where for example +$$ +\sigma_{xy} =\frac{1}{n} \sum_{i=0}^{n-1}(x_i- \overline{x})(y_i- \overline{y}). +$$ + +The Numpy function np.cov calculates the covariance elements using the factor \( 1/(n-1) \) instead of \( 1/n \) since it assumes we do not have the exact mean values. +The following simple function uses the np.vstack function which takes each vector of dimension \( 1\times n \) and produces a \( 3\times n \) matrix \( \hat{W} \) +$$ +\hat{W} = \begin{bmatrix} x_0 & y_0 & z_0 \\ + x_1 & y_1 & z_1 \\ + x_2 & y_2 & z_2 \\ + \dots & \dots & \dots \\ + x_{n-2} & y_{n-2} & z_{n-2} \\ + x_{n-1} & y_{n-1} & z_{n-1} + \end{bmatrix}, +$$ + +

+which in turn is converted into into the \( 3\times 3 \) covariance matrix +\( \hat{\Sigma} \) via the Numpy function np.cov(). We note that we can also calculate +the mean value of each set of samples \( \hat{x} \) etc using the Numpy +function np.mean(x). We can also extract the eigenvalues of the +covariance matrix through the np.linalg.eig() function. + +

+ + +

# Importing various packages
+import numpy as np
+
+n = 100
+x = np.random.normal(size=n)
+print(np.mean(x))
+y = 4+3*x+np.random.normal(size=n)
+print(np.mean(y))
+z = x**3+np.random.normal(size=n)
+print(np.mean(z))
+W = np.vstack((x, y, z))
+Sigma = np.cov(W)
+print(Sigma)
+Eigvals, Eigvecs = np.linalg.eig(Sigma)
+print(Eigvals)
+
+

+ + +

import numpy as np
+import matplotlib.pyplot as plt
+from scipy import sparse
+eye = np.eye(4)
+print(eye)
+sparse_mtx = sparse.csr_matrix(eye)
+print(sparse_mtx)
+x = np.linspace(-10,10,100)
+y = np.sin(x)
+plt.plot(x,y,marker='x')
+plt.show()
+
+ +

Meet the Pandas

+ +

+



+ +

+Another useful Python package is +pandas, which is an open source library +providing high-performance, easy-to-use data structures and data +analysis tools for Python. pandas stands for panel data, a term borrowed from econometrics and is an efficient library for data analysis with an emphasis on tabular data. +pandas has two major classes, the DataFrame class with two-dimensional data objects and tabular data organized in columns and the class Series with a focus on one-dimensional data objects. Both classes allow you to index data easily as we will see in the examples below. +pandas allows you also to perform mathematical operations on the data, spanning from simple reshapings of vectors and matrices to statistical operations. + +

+The following simple example shows how we can, in an easy way make tables of our data. Here we define a data set which includes names, place of birth and date of birth, and displays the data in an easy to read way. We will see repeated use of pandas, in particular in connection with classification of data. + +

+ + +

import pandas as pd
+from IPython.display import display
+data = {'First Name': ["Frodo", "Bilbo", "Aragorn II", "Samwise"],
+        'Last Name': ["Baggins", "Baggins","Elessar","Gamgee"],
+        'Place of birth': ["Shire", "Shire", "Eriador", "Shire"],
+        'Date of Birth T.A.': [2968, 2890, 2931, 2980]
+        }
+data_pandas = pd.DataFrame(data)
+display(data_pandas)
+
+

+In the above we have imported pandas with the shorthand pd, the latter has become the standard way we import pandas. We make then a list of various variables +and reorganize the aboves lists into a DataFrame and then print out a neat table with specific column labels as Name, place of birth and date of birth. +Displaying these results, we see that the indices are given by the default numbers from zero to three. +pandas is extremely flexible and we can easily change the above indices by defining a new type of indexing as +

+ + +

data_pandas = pd.DataFrame(data,index=['Frodo','Bilbo','Aragorn','Sam'])
+display(data_pandas)
+
+

+Thereafter we display the content of the row which begins with the index Aragorn +

+ + +

display(data_pandas.loc['Aragorn'])
+
+

+We can easily append data to this, for example +

+ + +

new_hobbit = {'First Name': ["Peregrin"],
+              'Last Name': ["Took"],
+              'Place of birth': ["Shire"],
+              'Date of Birth T.A.': [2990]
+              }
+data_pandas=data_pandas.append(pd.DataFrame(new_hobbit, index=['Pippin']))
+display(data_pandas)
+
+

+Here are other examples where we use the DataFrame functionality to handle arrays, now with more interesting features for us, namely numbers. We set up a matrix +of dimensionality \( 10\times 5 \) and compute the mean value and standard deviation of each column. Similarly, we can perform mathematial operations like squaring the matrix elements and many other operations. +

+ + +

import numpy as np
+import pandas as pd
+from IPython.display import display
+np.random.seed(100)
+# setting up a 10 x 5 matrix
+rows = 10
+cols = 5
+a = np.random.randn(rows,cols)
+df = pd.DataFrame(a)
+display(df)
+print(df.mean())
+print(df.std())
+display(df**2)
+
+

+Thereafter we can select specific columns only and plot final results +

+ + +

df.columns = ['First', 'Second', 'Third', 'Fourth', 'Fifth']
+df.index = np.arange(10)
+
+display(df)
+print(df['Second'].mean() )
+print(df.info())
+print(df.describe())
+
+from pylab import plt, mpl
+plt.style.use('seaborn')
+mpl.rcParams['font.family'] = 'serif'
+
+df.cumsum().plot(lw=2.0, figsize=(10,6))
+plt.show()
+
+
+df.plot.bar(figsize=(10,6), rot=15)
+plt.show()
+
+

+We can produce a \( 4\times 4 \) matrix +

+ + +

b = np.arange(16).reshape((4,4))
+print(b)
+df1 = pd.DataFrame(b)
+print(df1)
+
+

+and many other operations. + +

+The Series class is another important class included in +pandas. You can view it as a specialization of DataFrame but where +we have just a single column of data. It shares many of the same features as _DataFrame. As with DataFrame, +most operations are vectorized, achieving thereby a high performance when dealing with computations of arrays, in particular labeled arrays. +As we will see below it leads also to a very concice code close to the mathematical operations we may be interested in. +For multidimensional arrays, we recommend strongly xarray. xarray has much of the same flexibility as pandas, but allows for the extension to higher dimensions than two. We will see examples later of the usage of both pandas and xarray. + +

Reading Data and fitting

+ +

+In order to study various Machine Learning algorithms, we need to +access data. Acccessing data is an essential step in all machine +learning algorithms. In particular, setting up the so-called design +matrix (to be defined below) is often the first element we need in +order to perform our calculations. To set up the design matrix means +reading (and later, when the calculations are done, writing) data +in various formats, The formats span from reading files from disk, +loading data from databases and interacting with online sources +like web application programming interfaces (APIs). + +

+In handling various input formats, as discussed above, we will mainly stay with pandas, +a Python package which allows us, in a seamless and painless way, to +deal with a multitude of formats, from standard csv (comma separated +values) files, via excel, html to hdf5 formats. With pandas +and the DataFrame and Series functionalities we are able to convert text data +into the calculational formats we need for a specific algorithm. And our code is going to be +pretty close the basic mathematical expressions. + +

+Our first data set is going to be a classic from nuclear physics, namely all +available data on binding energies. Don't be intimidated if you are not familiar with nuclear physics. It serves simply as an example here of a data set. + +

+We will show some of the +strengths of packages like Scikit-Learn in fitting nuclear binding energies to +specific functions using linear regression first. Then, as a teaser, we will show you how +you can easily implement other algorithms like decision trees and random forests and neural networks. + +

+But before we really start with nuclear physics data, let's just look at some simpler polynomial fitting cases, such as, +(don't be offended) fitting straight lines! + +

Simple linear regression model using scikit-learn

+ +

+We start with perhaps our simplest possible example, using Scikit-Learn to perform linear regression analysis on a data set produced by us. + +

+What follows is a simple Python code where we have defined a function +\( y \) in terms of the variable \( x \). Both are defined as vectors with \( 100 \) entries. +The numbers in the vector \( \hat{x} \) are given +by random numbers generated with a uniform distribution with entries +\( x_i \in [0,1] \) (more about probability distribution functions +later). These values are then used to define a function \( y(x) \) +(tabulated again as a vector) with a linear dependence on \( x \) plus a +random noise added via the normal distribution.

The Numpy functions are imported used the import numpy as np @@ -337,7 +991,7 @@ $$

where \( N(0,1) \) represents random numbers generated by the normal -distribution. From scikit-learn we import then the +distribution. From Scikit-Learn we import then the LinearRegression functionality and make a prediction \( \tilde{y} = \alpha + \beta x \) using the function fit(x,y). We call the set of data \( (\hat{x},\hat{y}) \) for our training data. The Python package @@ -395,7 +1049,7 @@ $$

where \( x \) is defined as before. Does the fit look better? Indeed, by -reducing the role of the normal distribution we see immediately that +reducing the role of the noise given by the normal distribution we see immediately that our linear prediction seemingly reproduces better the training set. However, this testing 'by the eye' is obviouly not satisfactory in the long run. Here we have only defined the training data and our model, and @@ -406,7 +1060,7 @@ We need more rigorous criteria in defining whether we have succeeded or not in modeling our training data. You will be surprised to see that many scientists seldomly venture beyond this 'by the eye' approach. A standard approach for the cost function is the so-called \( \chi^2 \) -function +function (a variant of the mean-squared error (MSE)) $$ \chi^2 = \frac{1}{n} \sum_{i=0}^{n-1}\frac{(y_i-\tilde{y}_i)^2}{\sigma_i^2}, @@ -436,13 +1090,13 @@ the \( \chi^2 \) function becomes smaller.

There are many ways to define the cost function. A simpler approach is to look at the relative difference between the training data and the predicted data, that is we define -the relative error as +the relative error (why would we prefer the MSE instead of the relative error?) as $$ \epsilon_{\mathrm{relative}}= \frac{\vert \hat{y} -\hat{\tilde{y}}\vert}{\vert \hat{y}\vert}. $$ -We can modify easily the above Python code and plot the relative instead +We can modify easily the above Python code and plot the relative error instead

@@ -470,14 +1124,14 @@ different training data sets and study (graphically) the value of the relative error.

-As mentioned above, scikit-learn has an impressive functionality. +As mentioned above, Scikit-Learn has an impressive functionality. We can for example extract the values of \( \alpha \) and \( \beta \) and their error estimates, or the variance and standard deviation and many other properties from the statistical data analysis.

Here we show an -example of the functionality of scikit-learn. +example of the functionality of Scikit-Learn.

@@ -540,8 +1194,8 @@ $$ \bar{y} = \frac{1}{n} \sum_{i=0}^{n - 1} y_i. $$ -Another quantity will meet again in our discussions of regression analysis is - mean absolute error (MAE), a risk metric corresponding to the expected value of the absolute error loss or what we call the \( l1 \)-norm loss. In our discussion above we presented the relative error. +Another quantity taht we will meet again in our discussions of regression analysis is + the mean absolute error (MAE), a risk metric corresponding to the expected value of the absolute error loss or what we call the \( l1 \)-norm loss. In our discussion above we presented the relative error. The MAE is defined as follows $$ \text{MAE}(\hat{y}, \hat{\tilde{y}}) = \frac{1}{n} \sum_{i=0}^{n-1} \left| y_i - \tilde{y}_i \right|. @@ -598,820 +1252,362 @@ plt.show() print (error(y)) -

-Similarly, using R, we can perform similar studies. The following R code illustrates this. -(more details on R will be inserted later). - -

Non-Linear Least squares in R

-
-
-

-

- - -

set.seed(1485)
-len = 24
-x = runif(len)
-y = x^3+rnorm(len, 0,0.06)
-ds = data.frame(x = x, y = y)
-str(ds)
-plot( y ~ x, main ="Known cubic with noise")
-s  = seq(0,1,length =100)
-lines(s, s^3, lty =2, col ="green")
-m = nls(y ~ I(x^power), data = ds, start = list(power=1), trace = T)
-class(m)
-summary(m)
-power = round(summary(m)$coefficients[1], 3)
-power.se = round(summary(m)$coefficients[2], 3)
-plot(y ~ x, main = "Fitted power model", sub = "Blue: fit; green: known")
-s = seq(0, 1, length = 100)
-lines(s, s^3, lty = 2, col = "green")
-lines(s, predict(m, list(x = s)), lty = 1, col = "blue")
-text(0, 0.5, paste("y =x^ (", power, " +/- ", power.se, ")", sep = ""), pos = 4)
-
-

-

-
+

To our real data: nuclear binding energies. Brief reminder on masses and binding energies

-In our lectures on regression analysis (and other ones as well), we will discuss in more details various R functionalities. +Let us now dive into nuclear physics and remind ourselves briefly about some basic features about binding +energies. A basic quantity which can be measured for the ground +states of nuclei is the atomic mass \( M(N, Z) \) of the neutral atom with +atomic mass number \( A \) and charge \( Z \). The number of neutrons is \( N \). There are indeed several sophisticated experiments worldwide which allow us to measure this quantity to high precision (parts per million even).

-Another useful Python package is -pandas, which is an open source library -providing high-performance, easy-to-use data structures and data -analysis tools for Python. The following simple example shows how we can, in an easy way make tables of our data. Here we define a data set which includes names, city of residence and age, and displays the data in an easy to read way. We will see repeated use of pandas, in particular in connection with classification of data. +Atomic masses are usually tabulated in terms of the mass excess defined by +$$ +\Delta M(N, Z) = M(N, Z) - uA, +$$ + +where \( u \) is the Atomic Mass Unit +$$ +u = M(^{12}\mathrm{C})/12 = 931.4940954(57) \hspace{0.1cm} \mathrm{MeV}/c^2. +$$ + +The nucleon masses are +$$ +m_p = 1.00727646693(9)u, +$$ + +and +$$ +m_n = 939.56536(8)\hspace{0.1cm} \mathrm{MeV}/c^2 = 1.0086649156(6)u. +$$

- - -

import pandas as pd
-from IPython.display import display
-data = {'Name': ["John", "Anna", "Peter", "Linda"], 'Location': ["Nairobi", "Napoli", "London", "Buenos Aires"], 'Age':[51, 21, 34, 45]}
-data_pandas = pd.DataFrame(data)
-display(data_pandas)
-
- -

Examples

+In the 2016 mass evaluation of by W.J.Huang, G.Audi, M.Wang, F.G.Kondev, S.Naimi and X.Xu +there are data on masses and decays of 3437 nuclei.

-We present here several examples, with pertinent Python codes that we -will use to illustrate various machine learning methods and ways to -analyze, from simple to complex, various data sets. Many of these -examples allow us to generate the data we want to analyze, following -much of the same philosophy we discussed above when -fitting various polynomials. +The nuclear binding energy is defined as the energy required to break +up a given nucleus into its constituent parts of \( N \) neutrons and \( Z \) +protons. In terms of the atomic masses \( M(N, Z) \) the binding energy is +defined by + +$$ +BE(N, Z) = ZM_H c^2 + Nm_n c^2 - M(N, Z)c^2 , +$$ + +where \( M_H \) is the mass of the hydrogen atom and \( m_n \) is the mass of the neutron. +In terms of the mass excess the binding energy is given by +$$ +BE(N, Z) = Z\Delta_H c^2 + N\Delta_n c^2 -\Delta(N, Z)c^2 , +$$ + +where \( \Delta_H c^2 = 7.2890 \) MeV and \( \Delta_n c^2 = 8.0713 \) MeV.

-We start with a simple exponential growth model that is meant to mimick an ecoli lab experiment. -We can easily model this system and then produce the data used to train various machine learning algorithms. -Another model from the life sciences is the so-called predator-prey model from ecology. Thereafter we present -a simple model for financial transactions before moving to a random walk model and ending with -the simulation of velocities of a non-interacting atom or molecule confined to move in a one-dimensional region. +A popular and physically intuitive model which can be used to parametrize +the experimental binding energies as function of \( A \), is the so-called +liquid drop model. The ansatz is based on the following expression -

Ecoli lab experiment

+$$ +BE(N,Z) = a_1A-a_2A^{2/3}-a_3\frac{Z^2}{A^{1/3}}-a_4\frac{(N-Z)^2}{A}, +$$

-A typical pattern seen in population models is that the population grows faster and faster. Why? Is there an underlying (general) mechanism? -Here we will construct a model for cell growth based on a simple difference equation for the growth. We make the following assumptions -

-
-

- -

    -
  1. Cells divide after \( T \) seconds on average (one generation)
    2. -
  2. \( 2N \) celles divide into twice as many new cells \( \Delta N \) in a time - interval \( \Delta t \) as \( N \) cells would: \( \Delta N \propto N \)
    3. -
  3. \( N \) cells result in twice as many new individuals \( \Delta N \) in - time \( 2\Delta t \) as in time \( \Delta t \): \( \Delta N \propto\Delta t \)
    4. -
  4. Same proportionality with respect to death
    5. -
  5. Proposed model: \( \Delta N = b\Delta t N - d\Delta tN \) for some unknown - constants \( b \) (births) and \( d \) (deaths)
    6. -
  6. Describe evolution in discrete time: \( t_n=n\Delta t \)
    7. -
  7. Program-friendly notation: \( N \) at \( t_n \) is \( N^n \)
    8. -
  8. Math model: \( N^{n+1} = N^n + r\Delta t\, N \) (with \( \ r=b-d \))
    9. -
  9. Program model: N[n+1] = N[n] + r*dt*N[n]
    10. -
-
-
- +where \( A \) stands for the number of nucleons and the $a_i$s are parameters which are determined by a fit +to the experimental data.

-The difference equation can be programmed in a simple way, and in order to get started we -set \( r=1.5 \), \( N^0=1 \), \( \Delta t=0.5 \). The program reads - -

- - -

import numpy as np
-
-t = np.linspace(0, 10, 21)  # 20 intervals in [0, 10]
-dt = t[1] - t[0]
-N = np.zeros(t.size)
-N[0] = 1
-r = 0.5
-
-for n in range(0, N.size-1, 1):
-    N[n+1] = N[n] + r*dt*N[n]
-    print('N[%d]=%.1f' % (n+1, N[n+1]))
-
-

-and it generates the following output -

- - -

N[1]=1.2
-N[2]=1.6
-N[3]=2.0
-N[4]=2.4
-N[5]=3.1
-N[6]=3.8
-N[7]=4.8
-N[8]=6.0
-N[9]=7.5
-N[10]=9.3
-N[11]=11.6
-N[12]=14.6
-N[13]=18.2
-N[14]=22.7
-N[15]=28.4
-N[16]=35.5
-N[17]=44.4
-N[18]=55.5
-N[19]=69.4
-N[20]=86.7
-
-

-This forms our data which later will define our training set. -In this case we defined the value of the parameter \( r \). We could alternatively assume that we just received the -above data file and where asked to find \( r \). How can we estimate \( r \) from data? This will be one of our tasks later. - -

-We can use the difference equation with the experimental data -$$ N^{n+1} = N^n + r\Delta t N^n$$ - -Suppose now that \( N^{n+1} \) and \( N^n \) are known from data. Then we could solve with respect to \( r \) as follows -$$ r = \frac{N^{n+1}-N^n}{N^n\Delta t} $$ - -Suppose we set \( t_1=600 \), \( t_2=1200 \), -\( N^1=140 \) and \( N^2=250 \). -The following code plots the data -

- - -

import numpy as np
-import matplotlib.pyplot as plt
-
-# Estimate r
-data = np.loadtxt('ecoli.csv', delimiter=',')
-t_e = data[:,0]
-N_e = data[:,1]
-i = 2  # Data point (i,i+1) used to estimate r
-r = (N_e[i+1] - N_e[i])/(N_e[i]*(t_e[i+1] - t_e[i]))
-print('Estimated r=%.5f' % r)
-# Can experiment with r values and see if the model can
-# match the data better
-T = 1200     # cell can divide after T sec
-t_max = 5*T  # 5 generations in experiment
-t = np.linspace(0, t_max, 1000)
-dt = t[1] - t[0]
-N = np.zeros(t.size)
-
-N[0] = 100
-for n in range(0, len(t)-1, 1):
-    N[n+1] = N[n] + r*dt*N[n]
-
-plt.plot(t, N, 'r-', t_e, N_e, 'bo')
-plt.xlabel('time [s]');  plt.ylabel('N')
-plt.legend(['model', 'experiment'], loc='upper left')
-plt.show()
-
-

-We can then change the parameter \( r \) in the program and play around to make a better fit. By now we know that this -'search bythe eye' approach is not the most optimal one. - -

Predator-Prey model from ecology

- -

-The population dynamics of a simple predator-prey system is a -classical example shown in many biology textbooks when ecological -systems are discussed. The system contains all elements of the -scientific method: +To arrive at the above expression we have assumed that we can make the following assumptions:

    -
  • The set up of a specific hypothesis combined with
    • -
  • the experimental methods needed (one can study existing data or perform experiments)
    • -
  • analyzing and interpreting the data and performing further experiments if needed
    • -
  • trying to extract general behaviors and extract eventual laws or patterns
    • -
  • develop mathematical relations for the uncovered regularities/laws and test these by per forming new experiments
    • +
  • There is a volume term \( a_1A \) proportional with the number of nucleons (the energy is also an extensive quantity). When an assembly of nucleons of the same size is packed together into the smallest volume, each interior nucleon has a certain number of other nucleons in contact with it. This contribution is proportional to the volume.
    • +
  • There is a surface energy term \( a_2A^{2/3} \). The assumption here is that a nucleon at the surface of a nucleus interacts with fewer other nucleons than one in the interior of the nucleus and hence its binding energy is less. This surface energy term takes that into account and is therefore negative and is proportional to the surface area.
    • +
  • There is a Coulomb energy term \( a_3\frac{Z^2}{A^{1/3}} \). The electric repulsion between each pair of protons in a nucleus yields less binding.
    • +
  • There is an asymmetry term \( a_4\frac{(N-Z)^2}{A} \). This term is associated with the Pauli exclusion principle and reflects the fact that the proton-neutron interaction is more attractive on the average than the neutron-neutron and proton-proton interactions.
-Lots of data about populations of hares and lynx collected from furs in Hudson Bay, Canada, are available. It is known that the populations oscillate. Why? -Here we start by +We could also add a so-called pairing term, which is a correction term that +arises from the tendency of proton pairs and neutron pairs to +occur. An even number of particles is more stable than an odd number. -
    -
  1. plotting the data
    2. -
  2. derive a simple model for the population dynamics
    3. -
  3. (fitting parameters in the model to the data)
    4. -
  4. using the model predict the evolution other predator-pray systems
    5. -
- -Most mammalian predators rely on a variety of prey, which complicates mathematical modeling; however, a few predators have become highly specialized and seek almost exclusively a single prey species. An example of this simplified predator-prey interaction is seen in Canadian northern forests, where the populations of the lynx and the snowshoe hare are intertwined in a life and death struggle. +

Organizing our data

-One reason that this particular system has been so extensively studied is that the Hudson Bay company kept careful records of all furs from the early 1800s into the 1900s. The records for the furs collected by the Hudson Bay company showed distinct oscillations (approximately 12 year periods), suggesting that these species caused almost periodic fluctuations of each other's populations. The table here shows data from 1900 to 1920. +Let us start with reading and organizing our data. +We start with the compilation of masses and binding energies from 2016. +After having downloaded this file to our own computer, we are now ready to read the file and start structuring our data.

- -

-
- - - - - - - - - - - - - - - - - - - - - - - - - - - -
Year Hares (x1000) Lynx (x1000)
1900 30.0 4.0
1901 47.2 6.1
1902 70.2 9.8
1903 77.4 35.2
1904 36.3 59.4
1905 20.6 41.7
1906 18.1 19.0
1907 21.4 13.0
1908 22.0 8.3
1909 25.4 9.1
1910 27.1 7.4
1911 40.3 8.0
1912 57 12.3
1913 76.6 19.5
1914 52.3 45.7
1915 19.5 51.1
1916 11.2 29.7
1917 7.6 15.8
1918 14.6 9.7
1919 16.2 10.1
1920 24.7 8.6
-
-
-

- - -

import numpy as np
-from  matplotlib import pyplot as plt
-
-# Load in data file
-data = np.loadtxt('src/Hudson_Bay.csv', delimiter=',', skiprows=1)
-# Make arrays containing x-axis and hares and lynx populations
-year = data[:,0]
-hares = data[:,1]
-lynx = data[:,2]
-
-plt.plot(year, hares ,'b-+', year, lynx, 'r-o')
-plt.axis([1900,1920,0, 100.0])
-plt.xlabel(r'Year')
-plt.ylabel(r'Numbers of hares and lynx ')
-plt.legend(('Hares','Lynx'), loc='upper right')
-plt.title(r'Population of hares and lynx from 1900-1920 (x1000)}')
-plt.savefig('Hudson_Bay_data.pdf')
-plt.savefig('Hudson_Bay_data.png')
-plt.show()
-
-

-



- -

-We see from the plot that there are indeed fluctuations. -We would like to create a mathematical model that explains these -population fluctuations. Ecologists have predicted that in a simple -predator-prey system that a rise in prey population is followed (with -a lag) by a rise in the predator population. When the predator -population is sufficiently high, then the prey population begins -dropping. After the prey population falls, then the predator -population falls, which allows the prey population to recover and -complete one cycle of this interaction. Thus, we see that -qualitatively oscillations occur. Can a mathematical model predict -this? What causes cycles to slow or speed up? What affects the -amplitude of the oscillation or do you expect to see the oscillations -damp to a stable equilibrium? The models tend to ignore factors like -climate and other complicating factors. How significant are these? - -

    -
  • We see oscillations in the data
    • -
  • What causes cycles to slow or speed up?
    • -
  • What affects the amplitude of the oscillation or do you expect to see the oscillations damp to a stable equilibrium?
    • -
  • With a model we can better understand the data
    • -
  • More important: Can we understand the ecology dynamics of predator-pray populations?
    • -
- -The classical way (in all books) is to present the Lotka-Volterra equations: - -$$ -\begin{align*} -\frac{dH}{dt} &= H(a - b L)\\ -\frac{dL}{dt} &= - L(d - c H) -\end{align*} -$$ - -

-Here, - -

    -
  • \( H \) is the number of preys
    • -
  • \( L \) the number of predators
    • -
  • \( a \), \( b \), \( d \), \( c \) are parameters
    • -
- -The population of hares evolves due to births and deaths exactly as a bacteria population: - -$$ -\Delta H = a \Delta t H^n -$$ - -However, hares have an additional loss in the population because -they are eaten by lynx. -All the hares and lynx can form -\( H\cdot L \) pairs in total. When such pairs meet during a time -interval \( \Delta t \), there is some -small probablity that the lynx will eat the hare. -So in fraction \( b\Delta t HL \), the lynx eat hares. This -loss of hares must be accounted for. Subtracted in the equation for hares: - -$$ \Delta H = a\Delta t H^n - b \Delta t H^nL^n$$ - -

-We assume that the primary growth for the lynx population depends on sufficient food for raising lynx kittens, which implies an adequate source of nutrients from predation on hares. Thus, the growth of the lynx population does not only depend of how many lynx there are, but on how many hares they can eat. -In a time interval \( \Delta t HL \) hares and lynx can meet, and in a -fraction \( b\Delta t HL \) the lynx eats the hare. All of this does not -contribute to the growth of lynx, again just a fraction of -\( b\Delta t HL \) that we write as -\( d\Delta t HL \). In addition, lynx die just as in the population -dynamics with one isolated animal population, leading to a loss -\( -c\Delta t L \). -The accounting of lynx then looks like -$$ \Delta L = d\Delta t H^nL^n - c\Delta t L^n$$ - -

-By writing up the definition of \( \Delta H \) and \( \Delta L \), and putting -all assumed known terms \( H^n \) and \( L^n \) on the right-hand side, we have - -$$ H^{n+1} = H^n + a\Delta t H^n - b\Delta t H^n L^n $$ - - -$$ L^{n+1} = L^n + d\Delta t H^nL^n - c\Delta t L^n $$ - -

-Note: - -

    -
  • These equations are ready to be implemented!
    • -
  • But to start, we need \( H^0 \) and \( L^0 \) (which we can get from the data)
    • -
  • We also need values for \( a \), \( b \), \( d \), \( c \)
    • -
  • As always, models tend to be general - as here, applicable - to "all" predator-pray systems
    • -
  • The critical issue is whether the interaction between hares and lynx - is sufficiently well modeled by \( \hbox{const}HL \)
    • -
  • The parameters \( a \), \( b \), \( d \), and \( c \) must be - estimated from data
    • -
- -
-
-

-

- - -

import numpy as np
-import matplotlib.pyplot as plt
-
-def solver(m, H0, L0, dt, a, b, c, d, t0):
-    """Solve the difference equations for H and L over m years
-    with time step dt (measured in years."""
-
-    num_intervals = int(m/float(dt))
-    t = np.linspace(t0, t0 + m, num_intervals+1)
-    H = np.zeros(t.size)
-    L = np.zeros(t.size)
-
-    print('Init:', H0, L0, dt)
-    H[0] = H0
-    L[0] = L0
-
-    for n in range(0, len(t)-1):
-        H[n+1] = H[n] + a*dt*H[n] - b*dt*H[n]*L[n]
-        L[n+1] = L[n] + d*dt*H[n]*L[n] - c*dt*L[n]
-    return H, L, t
-
-# Load in data file
-data = np.loadtxt('src/Hudson_Bay.csv', delimiter=',', skiprows=1)
-# Make arrays containing x-axis and hares and lynx populations
-t_e = data[:,0]
-H_e = data[:,1]
-L_e = data[:,2]
-
-# Simulate using the model
-H, L, t = solver(m=20, H0=34.91, L0=3.857, dt=0.1,
-                 a=0.4807, b=0.02482, c=0.9272, d=0.02756,
-                 t0=1900)
-
-# Visualize simulations and data
-plt.plot(t_e, H_e, 'b-+', t_e, L_e, 'r-o', t, H, 'm--', t, L, 'k--')
-plt.xlabel('Year')
-plt.ylabel('Numbers of hares and lynx')
-plt.axis([1900, 1920, 0, 140])
-plt.title(r'Population of hares and lynx 1900-1920 (x1000)')
-plt.legend(('H_e', 'L_e', 'H', 'L'), loc='upper left')
-plt.savefig('Hudson_Bay_sim.pdf')
-plt.savefig('Hudson_Bay_sim.png')
-plt.show()
-
-

-

-
- - -

-



- -

-We will later perform a least-square fitting. Then we can find optimal -values for the parameters \( a \), \( b \), \( d \), \( c \). In our calculations here -we set \( a=0.4807 \), \( b=0.02482 \), \( d=0.9272 \) and \( c=0.02756 \). These -parameters result in a slightly modified initial conditions, namely -\( H(0) = 34.91 \) and \( L(0)=3.857 \). - -

-The following Python code demonstrates how we can use linear regression to fit for example the population of lynx. -Similarly, we have also used a decision tree algorithm to fit the lynx population data. As expected, the linear regression is not exactly impressive +We start with preparing folders for storing our calculations and the data file over masses and binding energies. We import also various modules that we will find useful in order to present various Machine Learning methods. Here we focus mainly on the functionality of scikit-learn.

-

import numpy as np
-import matplotlib.pyplot as plt
-from IPython.display import display
-import sklearn
-from sklearn.linear_model import LinearRegression
-from sklearn.tree import DecisionTreeRegressor
-
-
-data = np.loadtxt('src/Hudson_Bay.csv', delimiter=',', skiprows=1)
-x = data[:,0]
-y = data[:,1]
-line = np.linspace(1900,1920,1000,endpoint=False).reshape(-1,1)
-reg = DecisionTreeRegressor(min_samples_split=3).fit(x.reshape(-1,1),y.reshape(-1,1))
-plt.plot(line, reg.predict(line), label="decision tree")
-regline = LinearRegression().fit(x.reshape(-1,1),y.reshape(-1,1))
-plt.plot(line, regline.predict(line), label= "Linear Regression")
-plt.plot(x, y, label= "Linear Regression")
-plt.show()
-
-

-The similar code for linear regression in R reads (more details to come) -

- - -

HudsonBay = read.csv("src/Hudson_Bay.csv",header=T)
-fix(HudsonBay)
-dim(HudsonBay)
-names(HudsonBay)
-plot(HudsonBay$Year, HudsonBay$Hares..x1000.)
-attach(HudsonBay)
-plot(Year, Hares..x1000.)
-plot(Year, Hares..x1000., col="red", varwidth=T, xlab="Years", ylab="Haresx 1000")
-summary(HudsonBay)
-summary(Hares..x1000.)
-library(MASS)
-library(ISLR)
-scatter.smooth(x=Year, y = Hares..x1000.)
-linearMod = lm(Hares..x1000. ~ Year)
-print(linearMod)
-summary(linearMod)
-plot(linearMod)
-confint(linearMod)
-predict(linearMod,data.frame(Year=c(1910,1914,1920)),interval="confidence")
-
- -

Simulating financial transactions

- -

-The aim here is to simulate financial transactions among financial agents -using Monte Carlo methods. The final goal is to extract a distribution of income as function -of the income \( m \). From Pareto's work (V. Pareto, 1897) it is known from empirical studies -that the higher end of the distribution of money follows a distribution -$$ -w_m\propto m^{-1-\alpha}, -$$ - -with \( \alpha\in [1,2] \). We will here follow the analysis made by Patriarca and collaborators. - -

-Here we will study numerically the relation between the micro-dynamic relations among financial -agents and the resulting macroscopic money distribution. - -

-We assume we have \( N \) agents that exchange money in pairs \( (i,j) \). We assume also that all agents -start with the same amount of money \( m_0 > 0 \). At a given 'time step', we choose randomly a pair -of agents \( (i,j) \) and let a transaction take place. This means that agent \( i \)'s money \( m_i \) changes -to \( m_i' \) and similarly we have \( m_j\rightarrow m_j' \). -Money is conserved during a transaction, meaning that -$$ -\begin{equation} - m_i+m_j=m_i'+m_j'. -\label{eq:conserve} -\end{equation} -$$ - -The change is done via a random reassignement (a random number) \( \epsilon \), meaning that - -$$ -\begin{equation*} -m_i' = \epsilon(m_i+m_j), -\end{equation*} -$$ - -leading to - -$$ -\begin{equation*} -m_j'= (1-\epsilon)(m_i+m_j). -\end{equation*} -$$ - -The number \( \epsilon \) is extracted from a uniform distribution. -In this simple model, no agents are left with a debt, that is \( m\ge 0 \). -Due to the conservation law above, one can show that the system relaxes toward an equilibrium -state given by a Gibbs distribution - -$$ -\begin{equation*} -w_m=\beta \exp{(-\beta m)}, -\end{equation*} -$$ - -with - -$$ -\begin{equation*} -\beta = \frac{1}{\langle m\rangle}, -\end{equation*} -$$ - -and \( \langle m\rangle=\sum_i m_i/N=m_0 \), the average money. -It means that after equilibrium has been reached that the majority of agents is left with a small -number of money, while the number of richest agents, those with \( m \) larger than a specific value \( m' \), -exponentially decreases with \( m' \). - -

-We assume that we have \( N=500 \) agents. In each simulation, we need a sufficiently large number of transactions, say \( 10^7 \). Our aim is find the final equilibrium distribution \( w_m \). In order to do that we would need -several runs of the above simulations, at least \( 10^3-10^4 \) runs (experiments). - -

-Our task is to first set up an algorithm which simulates the above transactions with an initial - amount \( m_0 \). - The challenge here is to figure out a Monte Carlo simulation based on the - above equations. - You will in particular need to make an algorithm which sets up a histogram as function of \( m \). - This histogram contains the number of times a value \( m \) is registered and represents - \( w_m\Delta m \). You will need to set up a value for the interval \( \Delta m \) (typically \( 0.01-0.05 \)). - That means you need to account for the number of times you register an income in the interval - \( m,m+\Delta m \). The number of times you register this income, represents the value that enters the histogram. - -

- - -

#!/usr/bin/env python
+
# Common imports
 import numpy as np
-import matplotlib.mlab as mlab
+import pandas as pd
 import matplotlib.pyplot as plt
-import random
+import sklearn.linear_model as skl
+from sklearn.model_selection import train_test_split
+from sklearn.metrics import mean_squared_error, r2_score, mean_absolute_error
+import os
 
-# initialize the rng with a seed
-random.seed()
-# Hard coding of input parameters
-Agents  = 500
-MCcounts = 1000
-Transactions = 100000
-startMoney = 1.0
-Lambda = 0.0
-FinancialAgents = startMoney*np.ones(Agents)
-for i in range (1, MCcounts, 1):
-    for j in range (1, Transactions, 1):
-        agent_i = int(Agents*random.random())
-        agent_j = int(Agents*random.random())
-        epsilon = random.random()
-        if agent_i != agent_j:
-           m1 = Lambda*FinancialAgents[agent_i] + (1-Lambda)*epsilon*(FinancialAgents[agent_i] + FinancialAgents[agent_j])
-           m2 = Lambda*FinancialAgents[agent_j] + (1-Lambda)*(1-epsilon)*(FinancialAgents[agent_i] + FinancialAgents[agent_j])
-           FinancialAgents[agent_i] = m1
-           FinancialAgents[agent_j] = m2
+# Where to save the figures and data files
+PROJECT_ROOT_DIR = "Results"
+FIGURE_ID = "Results/FigureFiles"
+DATA_ID = "DataFiles/"
 
-# the histogram of the data
-n, bins, patches = plt.hist(FinancialAgents, 50, facecolor='green')
+if not os.path.exists(PROJECT_ROOT_DIR):
+    os.mkdir(PROJECT_ROOT_DIR)
 
-plt.xlabel('$x$')
-plt.ylabel('Distribution of wealth')
-plt.title(r'Money')
-plt.axis([0, 10, 0, 500])
-plt.grid(True)
-plt.show()
+if not os.path.exists(FIGURE_ID):
+    os.makedirs(FIGURE_ID)
+
+if not os.path.exists(DATA_ID):
+    os.makedirs(DATA_ID)
+
+def image_path(fig_id):
+    return os.path.join(FIGURE_ID, fig_id)
+
+def data_path(dat_id):
+    return os.path.join(DATA_ID, dat_id)
+
+def save_fig(fig_id):
+    plt.savefig(image_path(fig_id) + ".png", format='png')
+
+infile = open(data_path("MassEval2016.dat"),'r')
 

 

-We can then change our model to allow for a saving criterion, meaning that the agents save
-  a fraction \( \lambda \) of the money they have before the transaction is made. The final distribution will then no longer be given by Gibbs distribution. It could also include a taxation on financial transactions.
+Before we proceed, we define also a function for making our plots. You can obviously avoid this and simply set up various matplotlib commands every time you need them. You may however find it convenient to collect all such commands in one function and simply call this function. 
+

+
+
+
from pylab import plt, mpl
+plt.style.use('seaborn')
+mpl.rcParams['font.family'] = 'serif'
+
+def MakePlot(x,y, styles, labels, axlabels):
+    plt.figure(figsize=(10,6))
+    for i in range(len(x)):
+        plt.plot(x[i], y[i], styles[i], label = labels[i])
+        plt.xlabel(axlabels[0])
+        plt.ylabel(axlabels[1])
+    plt.legend(loc=0)
+

+

+Our next step is to read the data on experimental binding energies and
+reorganize them as functions of the mass number \( A \), the number of
+protons \( Z \) and neutrons \( N \) using pandas.  Before we do this it is
+always useful (unless you have a binary file or other types of compressed
+data) to actually open the file and simply take a look at it!
 
 

-  The conservation law of Eq. \eqref{eq:conserve} holds, but the money to be shared in a transaction between
-  agent \( i \) and agent \( j \) is now \( (1-\lambda)(m_i+m_j) \). This means that we have
+In particular, the program that outputs the final nuclear masses is written in Fortran with a specific format. It means that we need to figure out the format and which columns contain the data we are interested in. Pandas comes with a function that reads formatted output. After having admired the file, we are now ready to start massaging it with pandas. The file begins with some basic format information.
+

 
-$$
-\begin{equation*}
-  m_i' = \lambda m_i+\epsilon(1-\lambda)(m_i+m_j),
-  \end{equation*}
-$$
-
-  and
-
-$$
-\begin{equation*}
-  m_j' = \lambda m_j+(1-\epsilon)(1-\lambda)(m_i+m_j),
-  \end{equation*}
-$$
-
-  which can be written as
-
-$$
-\begin{equation*}
-  m_i'=m_i+\delta m
-  \end{equation*}
-$$
-
-  and
-
-$$
-\begin{equation*}
-  m_j'=m_j-\delta m,
-  \end{equation*}
-$$
-
-  with
-
-$$
-\begin{equation*}
-  \delta m=(1-\lambda)(\epsilon m_j-(1-\epsilon)m_i),
-  \end{equation*}
-$$
-
-  showing how money is conserved during a transaction.
-  Select values of \( \lambda =0.25,0.5 \) and \( \lambda=0.9 \) and try to extract the corresponding
-  equilibrium distributions and compare these with the Gibbs distribution. We will use this model to 
-extract a parametrization of the above curves, see for example Patriarca and collaborators.
-
-
Particle in one dimension and velocity distribution 

+
+
"""                                                                                                                         
+This is taken from the data file of the mass 2016 evaluation.                                                               
+All files are 3436 lines long with 124 character per line.                                                                  
+       Headers are 39 lines long.                                                                                           
+   col 1     :  Fortran character control: 1 = page feed  0 = line feed                                                     
+   format    :  a1,i3,i5,i5,i5,1x,a3,a4,1x,f13.5,f11.5,f11.3,f9.3,1x,a2,f11.3,f9.3,1x,i3,1x,f12.5,f11.5                     
+   These formats are reflected in the pandas widths variable below, see the statement                                       
+   widths=(1,3,5,5,5,1,3,4,1,13,11,11,9,1,2,11,9,1,3,1,12,11,1),                                                            
+   Pandas has also a variable header, with length 39 in this case.                                                          
+"""
+

+

+The data we are interested in are in columns 2, 3, 4 and 11, giving us
+the number of neutrons, protons, mass numbers and binding energies,
+respectively. We add also for the sake of completeness the element name. The data are in fixed-width formatted lines and we will
+covert them into the pandas DataFrame structure.
 
 

 
 
-
# Program to test the Metropolis algorithm with one particle at given temp in one dimension
-import numpy as np
-import matplotlib.mlab as mlab
-import matplotlib.pyplot as plt
-import random
-from math import sqrt, exp, log
-# initialize the rng with a seed
-random.seed()
-# Hard coding of input parameters
-MCcycles = 100000
-Temperature = 2.0
-beta = 1./Temperature
-InitialVelocity = -2.0
-CurrentVelocity = InitialVelocity
-Energy = 0.5*InitialVelocity*InitialVelocity
-VelocityRange = 10*sqrt(Temperature)
-VelocityStep = 2*VelocityRange/10.
-AverageEnergy = Energy
-AverageEnergy2 = Energy*Energy
-VelocityValues = np.zeros(MCcycles)
-# The Monte Carlo sampling with Metropolis starts here
-for i in range (1, MCcycles, 1):
-    TrialVelocity = CurrentVelocity + (2.0*random.random() - 1.0)*VelocityStep
-    EnergyChange = 0.5*(TrialVelocity*TrialVelocity -CurrentVelocity*CurrentVelocity);
-    if random.random() <= exp(-beta*EnergyChange):
-        CurrentVelocity = TrialVelocity
-        Energy += EnergyChange
-        VelocityValues[i] = CurrentVelocity
-    AverageEnergy += Energy
-    AverageEnergy2 += Energy*Energy
-#Final averages
-AverageEnergy = AverageEnergy/MCcycles
-AverageEnergy2 = AverageEnergy2/MCcycles
-Variance = AverageEnergy2 - AverageEnergy*AverageEnergy
-print(AverageEnergy, Variance)
-n, bins, patches = plt.hist(VelocityValues, 400, facecolor='green')
+
# Read the experimental data with Pandas
+Masses = pd.read_fwf(infile, usecols=(2,3,4,6,11),
+              names=('N', 'Z', 'A', 'Element', 'Ebinding'),
+              widths=(1,3,5,5,5,1,3,4,1,13,11,11,9,1,2,11,9,1,3,1,12,11,1),
+              header=39,
+              index_col=False)
 
-plt.xlabel('$v$')
-plt.ylabel('Velocity distribution P(v)')
-plt.title(r'Velocity histogram at $k_BT=2$')
-plt.axis([-5, 5, 0, 600])
-plt.grid(True)
+# Extrapolated values are indicated by '#' in place of the decimal place, so
+# the Ebinding column won't be numeric. Coerce to float and drop these entries.
+Masses['Ebinding'] = pd.to_numeric(Masses['Ebinding'], errors='coerce')
+Masses = Masses.dropna()
+# Convert from keV to MeV.
+Masses['Ebinding'] /= 1000
+
+# Group the DataFrame by nucleon number, A.
+Masses = Masses.groupby('A')
+# Find the rows of the grouped DataFrame with the maximum binding energy.
+Masses = Masses.apply(lambda t: t[t.Ebinding==t.Ebinding.max()])
+

+

+We have now read in the data, grouped them according to the variables we are interested in. 
+We see how easy it is to reorganize the data using pandas. If we
+were to do these operations in C/C++ or Fortran, we would have had to
+write various functions/subroutines which perform the above
+reorganizations for us.  Having reorganized the data, we can now start
+to make some simple fits using both the functionalities in numpy and
+Scikit-Learn afterwards.
+
+

+Now we define five variables which contain
+the number of nucleons \( A \), the number of protons \( Z \) and the number of neutrons \( N \), the element name and finally the energies themselves.
+

+
+
+
A = Masses['A']
+Z = Masses['Z']
+N = Masses['N']
+Element = Masses['Element']
+Energies = Masses['Ebinding']
+print(Masses)
+

+

+The next step, and we will define this mathematically later, is to set up the so-called design matrix. We will throughout call this matrix \( \boldsymbol{X} \).
+It has dimensionality \( p\times n \), where \( n \) is the number of data points and \( p \) are the so-called predictors. In our case here they are given by the number of polynomials in \( A \) we wish to include in the fit. 
+

+
+
+
# Now we set up the design matrix X
+X = np.zeros((len(A),5))
+X[:,0] = 1
+X[:,1] = A
+X[:,2] = A**(2.0/3.0)
+X[:,3] = A**(-1.0/3.0)
+X[:,4] = A**(-1.0)
+

+

+With scikitlearn we are now ready to use linear regression and fit our data.
+

+
+
+
clf = skl.LinearRegression().fit(X, Energies)
+fity = clf.predict(X)
+

+

+Pretty simple!  
+Now we can print measures of how our fit is doing, the coefficients from the fits and plot the final fit together with our data.
+

+
+
+
# The mean squared error                               
+print("Mean squared error: %.2f" % mean_squared_error(Energies, fity))
+# Explained variance score: 1 is perfect prediction                                 
+print('Variance score: %.2f' % r2_score(Energies, fity))
+# Mean absolute error                                                           
+print('Mean absolute error: %.2f' % mean_absolute_error(Energies, fity))
+print(clf.coef_, clf.intercept_)
+
+Masses['Eapprox']  = fity
+# Generate a plot comparing the experimental with the fitted values values.
+fig, ax = plt.subplots()
+ax.set_xlabel(r'$A = N + Z$')
+ax.set_ylabel(r'$E_\mathrm{bind}\,/\mathrm{MeV}$')
+ax.plot(Masses['A'], Masses['Ebinding'], alpha=0.7, lw=2,
+            label='Ame2016')
+ax.plot(Masses['A'], Masses['Eapprox'], alpha=0.7, lw=2, c='m',
+            label='Fit')
+ax.legend()
+save_fig("Masses2016")
 plt.show()
 

 
-
Random walk model 

+
Seeing the wood for the trees 

+
+

+As a teaser, let us now see how we can do this with decision trees using scikit-learn. Later we will switch to so-called random forests!
 
 

 
 
-
import numpy as np
-import matplotlib.pyplot as plt
-from sklearn.preprocessing import PolynomialFeatures
-from sklearn.linear_model import LinearRegression
-
-steps=250
-
-distance=0
-x=0
-distance_list=[]
-steps_list=[]
-while x<steps:
-    distance+=np.random.randint(-1,2)
-    distance_list.append(distance)
-    x+=1
-    steps_list.append(x)
-plt.plot(steps_list,distance_list, color='green', label="Random Walk Data")
-
-steps_list=np.asarray(steps_list)
-distance_list=np.asarray(distance_list)
-
-X=steps_list[:,np.newaxis]
-
-#Polynomial fits
-
-#Degree 2
-poly_features=PolynomialFeatures(degree=2, include_bias=False)
-X_poly=poly_features.fit_transform(X)
-
-lin_reg=LinearRegression()
-poly_fit=lin_reg.fit(X_poly,distance_list)
-b=lin_reg.coef_
-c=lin_reg.intercept_
-print ("2nd degree coefficients:")
-print ("zero power: ",c)
-print ("first power: ", b[0])
-print ("second power: ",b[1])
-
-z = np.arange(0, steps, .01)
-z_mod=b[1]*z**2+b[0]*z+c
-
-fit_mod=b[1]*X**2+b[0]*X+c
-plt.plot(z, z_mod, color='r', label="2nd Degree Fit")
-plt.title("Polynomial Regression")
-
-plt.xlabel("Steps")
-plt.ylabel("Distance")
-
-#Degree 10
-poly_features10=PolynomialFeatures(degree=10, include_bias=False)
-X_poly10=poly_features10.fit_transform(X)
-
-poly_fit10=lin_reg.fit(X_poly10,distance_list)
-
-y_plot=poly_fit10.predict(X_poly10)
-plt.plot(X, y_plot, color='black', label="10th Degree Fit")
-
-plt.legend()
-plt.show()
-
-
-#Decision Tree Regression
+
#Decision Tree Regression
 from sklearn.tree import DecisionTreeRegressor
-regr_1=DecisionTreeRegressor(max_depth=2)
-regr_2=DecisionTreeRegressor(max_depth=5)
-regr_3=DecisionTreeRegressor(max_depth=7)
-regr_1.fit(X, distance_list)
-regr_2.fit(X, distance_list)
-regr_3.fit(X, distance_list)
+regr_1=DecisionTreeRegressor(max_depth=5)
+regr_2=DecisionTreeRegressor(max_depth=7)
+regr_3=DecisionTreeRegressor(max_depth=9)
+regr_1.fit(X, Energies)
+regr_2.fit(X, Energies)
+regr_3.fit(X, Energies)
 
-X_test = np.arange(0.0, steps, 0.01)[:, np.newaxis]
-y_1 = regr_1.predict(X_test)
-y_2 = regr_2.predict(X_test)
-y_3=regr_3.predict(X_test)
 
+y_1 = regr_1.predict(X)
+y_2 = regr_2.predict(X)
+y_3=regr_3.predict(X)
+Masses['Eapprox'] = y_3
 # Plot the results
 plt.figure()
-plt.scatter(X, distance_list, s=2.5, c="black", label="data")
-plt.plot(X_test, y_1, color="red",
-         label="max_depth=2", linewidth=2)
-plt.plot(X_test, y_2, color="green", label="max_depth=5", linewidth=2)
-plt.plot(X_test, y_3, color="m", label="max_depth=7", linewidth=2)
+plt.plot(A, Energies, color="blue", label="Data", linewidth=2)
+plt.plot(A, y_1, color="red", label="max_depth=5", linewidth=2)
+plt.plot(A, y_2, color="green", label="max_depth=7", linewidth=2)
+plt.plot(A, y_3, color="m", label="max_depth=9", linewidth=2)
 
-plt.xlabel("Data")
-plt.ylabel("Darget")
+plt.xlabel("$A$")
+plt.ylabel("$E$[MeV]")
 plt.title("Decision Tree Regression")
 plt.legend()
+save_fig("Masses2016Trees")
+plt.show()
+print(Masses)
+print(np.mean( (Energies-y_1)**2))
+

+
+
And what about using neural networks? 

+
+The seaborn package allows us to visualize data in an efficient way. Note that we use scikit-learn's multi-layer perceptron (or feed forward neural network) 
+functionality.
+

+
+
+
from sklearn.neural_network import MLPRegressor
+from sklearn.metrics import accuracy_score
+import seaborn as sns
+
+X_train = X
+Y_train = Energies
+n_hidden_neurons = 100
+epochs = 100
+# store models for later use
+eta_vals = np.logspace(-5, 1, 7)
+lmbd_vals = np.logspace(-5, 1, 7)
+# store the models for later use
+DNN_scikit = np.zeros((len(eta_vals), len(lmbd_vals)), dtype=object)
+train_accuracy = np.zeros((len(eta_vals), len(lmbd_vals)))
+sns.set()
+for i, eta in enumerate(eta_vals):
+    for j, lmbd in enumerate(lmbd_vals):
+        dnn = MLPRegressor(hidden_layer_sizes=(n_hidden_neurons), activation='logistic',
+                            alpha=lmbd, learning_rate_init=eta, max_iter=epochs)
+        dnn.fit(X_train, Y_train)
+        DNN_scikit[i][j] = dnn
+        train_accuracy[i][j] = dnn.score(X_train, Y_train)
+
+fig, ax = plt.subplots(figsize = (10, 10))
+sns.heatmap(train_accuracy, annot=True, ax=ax, cmap="viridis")
+ax.set_title("Training Accuracy")
+ax.set_ylabel("$\eta$")
+ax.set_xlabel("$\lambda$")
 plt.show()
 

+
+
A first summary 

+
+

+The aim behind these introductory words was to present to you various
+Python libraries and their functionalities, in particular libraries like
+numpy, pandas, xarray and matplotlib and other that make our life much easier
+in handling various data sets and visualizing data.
+
+

+Furthermore,
+Scikit-Learn allows us with few lines of code to implement popular
+Machine Learning algorithms for supervised learning. Later we will meet Tensorflow, a powerful library for deep learning. 
+Now it is time to dive more into the details of various methods. We will start with linear regression and try to take a deeper look at what it entails.
+
 

 
 
diff --git a/doc/pub/How2ReadData/html/How2ReadData-reveal.html b/doc/pub/How2ReadData/html/How2ReadData-reveal.html
index cce217bde..3bc2a9175 100644
--- a/doc/pub/How2ReadData/html/How2ReadData-reveal.html
+++ b/doc/pub/How2ReadData/html/How2ReadData-reveal.html
@@ -148,7 +148,7 @@ MathJax.Hub.Config({
 
[2] Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University

 

 
 

-
Jan 14, 2019
 
+
Aug 13, 2019
 
 

 
 
Introduction 

@@ -167,45 +167,151 @@ and use these to introduce statistical data analysis
 concepts and machine learning algorithms before we delve into the
 algorithms themselves. The examples we will use in the beginning, start with simple
 polynomials with random noise added. We will use the Python
-software package Scikit-learn and
+software package Scikit-Learn and
 introduce various machine learning algorithms to make fits of
 the data and predictions. We move thereafter to more interesting
-cases such as the simulation of financial transactions or disease
-models. These are examples where we can easily set up the data and
+cases such as nuclear binding energies.
+These are examples where we can easily set up the data and
 then use machine learning algorithms included in for example
-scikit-learn.
+Scikit-Learn.
 
 

 These examples will serve us the purpose of getting
 started. Furthermore, they allow us to catch more than two birds with
 a stone. They will allow us to bring in some programming specific
-topics and tools as well as showing the power of various Python (and
-R) packages for machine learning and statistical data analysis. In the
-lectures on linear algebra we cover in more detail various programming
-features of languages like Python and C++ (and other), we will also
-look into more specific linear functions which are relevant for the
-various algorithms we will discuss. Here, we will mainly focus on two
-specific Python packages for Machine Learning, scikit-learn and
-tensorflow (see below for links etc).  Moreover, the examples we
+topics and tools as well as showing the power of various Python 
+packages for machine learning and statistical data analysis.
+
+

+Here, we will mainly focus on two
+specific Python packages for Machine Learning, Scikit-Learn and
+Tensorflow (see below for links etc).  Moreover, the examples we
 introduce will serve as inputs to many of our discussions later, as
 well as allowing you to set up models and produce your own data and
 get started with programming.
 
-
Software and needed installations 

+
What is Machine Learning?  

+
+

+Statistics, data science and machine learning form important fields of
+research in modern science.  They describe how to learn and make
+predictions from data, as well as allowing us to extract important
+correlations about physical process and the underlying laws of motion
+in large data sets. The latter, big data sets, appear frequently in
+essentially all disciplines, from the traditional Science, Technology,
+Mathematics and Engineering fields to Life Science, Law, education
+research, the Humanities and the Social Sciences.
+
+

+It has become more
+and more common to see research projects on big data in for example
+the Social Sciences where extracting patterns from complicated survey
+data is one of many research directions.  Having a solid grasp of data
+analysis and machine learning is thus becoming central to scientific
+computing in many fields, and competences and skills within the fields
+of machine learning and scientific computing are nowadays strongly
+requested by many potential employers. The latter cannot be
+overstated, familiarity with machine learning has almost become a
+prerequisite for many of the most exciting employment opportunities,
+whether they are in bioinformatics, life science, physics or finance,
+in the private or the public sector. This author has had several
+students or met students who have been hired recently based on their
+skills and competences in scientific computing and data science, often
+with marginal knowledge of machine learning.
+
+

+Machine learning is a subfield of computer science, and is closely
+related to computational statistics.  It evolved from the study of
+pattern recognition in artificial intelligence (AI) research, and has
+made contributions to AI tasks like computer vision, natural language
+processing and speech recognition. Many of the methods we will study are also 
+strongly rooted in basic mathematics and physics research.
+
+

+Ideally, machine learning represents the science of giving computers
+the ability to learn without being explicitly programmed.  The idea is
+that there exist generic algorithms which can be used to find patterns
+in a broad class of data sets without having to write code
+specifically for each problem. The algorithm will build its own logic
+based on the data.  You should however always keep in mind that
+machines and algorithms are to a large extent developed by humans. The
+insights and knowledge we have about a specific system, play a central
+role when we develop a specific machine learning algorithm.
+
+

+Machine learning is an extremely rich field, in spite of its young
+age. The increases we have seen during the last three decades in
+computational capabilities have been followed by developments of
+methods and techniques for analyzing and handling large date sets,
+relying heavily on statistics, computer science and mathematics.  The
+field is rather new and developing rapidly. Popular software packages
+written in Python for machine learning like
+Scikit-learn,
+Tensorflow,
+PyTorch and Keras, all
+freely available at their respective GitHub sites, encompass
+communities of developers in the thousands or more. And the number of
+code developers and contributors keeps increasing. Not all the
+algorithms and methods can be given a rigorous mathematical
+justification, opening up thereby large rooms for experimenting and
+trial and error and thereby exciting new developments.  However, a
+solid command of linear algebra, multivariate theory, probability
+theory, statistical data analysis, understanding errors and Monte
+Carlo methods are central elements in a proper understanding of many
+of algorithms and methods we will discuss.
+
+
Types of Machine Learning 

+
+

+The approaches to machine learning are many, but are often split into
+two main categories.  In supervised learning we know the answer to a
+problem, and let the computer deduce the logic behind it. On the other
+hand, unsupervised learning is a method for finding patterns and
+relationship in data sets without any prior knowledge of the system.
+Some authours also operate with a third category, namely
+reinforcement learning. This is a paradigm of learning inspired by
+behavioral psychology, where learning is achieved by trial-and-error,
+solely from rewards and punishment.
+
+

+Another way to categorize machine learning tasks is to consider the
+desired output of a system.  Some of the most common tasks are:
+
+
    
+
+
  •  Classification: Outputs are divided into two or more classes. The goal is to   produce a model that assigns inputs into one of these classes. An example is to identify  digits based on pictures of hand-written ones. Classification is typically supervised learning.
    • 
+
+
  •  Regression: Finding a functional relationship between an input data set and a reference data set.   The goal is to construct a function that maps input data to continuous output values.
    • 
+
+
  •  Clustering: Data are divided into groups with certain common traits, without knowing the different groups beforehand.  It is thus a form of unsupervised learning.
    • 
+

+

+
+The methods we cover have three main topics in common, irrespective of
+whether we deal with supervised or unsupervised learning. The first
+ingredient is normally our data set (which can be subdivided into
+training and test data), the second item is a model which is normally a
+function of some parameters.  The model reflects our knowledge of the system (or lack thereof). As an example, if we know that our data show a behavior similar to what would be predicted by a polynomial, fitting our data to a polynomial of some degree would then determin our model.
+
+

+The last ingredient is a so-called cost
+function which allows us to present an estimate on how good our model
+is in reproducing the data it is supposed to train.  
+At the heart of basically all ML algorithms there are so-called minimization algorithms, often we end up with various variants of gradient methods.
+
+
Software and needed installations 

 
 

 We will make extensive use of Python as programming language and its
 myriad of available libraries.  You will find
-IPython/Jupyter notebooks invaluable in your work.  You can run R
+Jupyter notebooks invaluable in your work.  You can run R
 codes in the Jupyter/IPython notebooks, with the immediate benefit of
 visualizing your data. You can also use compiled languages like C++,
-Rust, Fortran etc if you prefer. The focus in these lectures will be
-on Python, but we will provide many code examples for those of you who
-prefer R or compiled languages. You can integrate C++ codes and R in for example
-a Jupyter notebook.
+Rust, Julia, Fortran etc if you prefer. The focus in these lectures will be
+on Python.
 
 

-If you have Python installed (we recommend Python3) and you feel
+If you have Python installed (we strongly recommend Python3) and you feel
 pretty familiar with installing different packages, we recommend that
 you install the following Python packages via pip as 
 
@@ -236,7 +342,7 @@ you can use pip as well and simply install Python as
 
 etc etc.
 
-
Python installers 

+
Python installers 

 
 

 If you don't want to perform these operations separately and venture
@@ -265,28 +371,46 @@ distribution for scientific and analytic computing distribution and
 analysis environment, available for free and under a commercial
 license.
 
-
Useful Python packages 

-Here we list several useful Python packages.
+

+Furthermore, Google's Colab is a free Jupyter notebook environment that requires 
+no setup and runs entirely in the cloud. Try it out!
 
-
Installing R, C++, cython or Julia 

+
Useful Python libraries 

+Here we list several useful Python libraries we strongly recommend (if you use anaconda many of these are already there)
+
+
    
+
  •  NumPy is a highly popular library for large, multi-dimensional arrays and matrices, along with a large collection of high-level mathematical functions to operate on these arrays
    • 
+
  •  The pandas library provides high-performance, easy-to-use data structures and data analysis tools
    •  
+
  •  Xarray is a Python package that makes working with labelled multi-dimensional arrays simple, efficient, and fun!
    • 
+
  •  Scipy (pronounced “Sigh Pie”) is a Python-based ecosystem of open-source software for mathematics, science, and engineering.
    •  
+
  •  Matplotlib is a Python 2D plotting library which produces publication quality figures in a variety of hardcopy formats and interactive environments across platforms.
    • 
+
  •  Autograd can automatically differentiate native Python and Numpy code. It can handle a large subset of Python's features, including loops, ifs, recursion and closures, and it can even take derivatives of derivatives of derivatives
    • 
+
  •  SymPy is a Python library for symbolic mathematics.
    •  
+
  •  scikit-learn has simple and efficient tools for machine learning, data mining and data analysis
    • 
+
  •  TensorFlow is a Python library for fast numerical computing created and released by Google
    • 
+
  •  Keras is a high-level neural networks API, written in Python and capable of running on top of TensorFlow, CNTK, or Theano
    • 
+
  •  And many more such as pytorch,  Theano etc
    •  
+

+
+
Installing R, C++, cython or Julia 

 
 

-You will also find it convenient to utilize R. Although we will mainly
-use Python during lectures and in various projects and exercises, we
-provide a full R set of codes for the same examples. Those of you
-already familiar with R should feel free to continue using R, keeping
+You will also find it convenient to utilize R. We will mainly
+use Python during lectures and in various projects and exercises.
+Those of you
+already familiar with R should feel free to continue using R, keeping
 however an eye on the parallel Python set ups. Similarly, if you are a
-Python afecionado, feel free to explore R as well.  Jupyter/Ipython
+Python afecionado, feel free to explore R as well.  Jupyter/Ipython
 notebook allows you to run R codes interactively in your
 browser. The software library R is tuned to statistically analysis
 and allows for an easy usage of the tools we will discuss in these
-texts.
+lectures.
 
 

 To install R with Jupyter notebook 
 follow the link here
 
-
Installing R, C++, cython, Numba etc 

+
Installing R, C++, cython, Numba etc 

 
 

 For the C++ aficionados, Jupyter/IPython notebook allows you also to
@@ -297,13 +421,13 @@ languages.
 
 

 To add more entropy, cython can also be used when running your
-notebooks. It means that Python with the Jupyter/IPython notebook
+notebooks. It means that Python with the jupyter notebook
 setup allows you to integrate widely popular softwares and tools for
 scientific computing. Similarly, the 
 Numba Python package delivers increased performance
 capabilities with minimal rewrites of your codes.  With its
 versatility, including symbolic operations, Python offers a unique
-computational environment. Your Jupyter/IPython notebook can easily be
+computational environment. Your jupyter notebook can easily be
 converted into a nicely rendered PDF file or a Latex file for
 further processing. For example, convert to latex as
 
@@ -318,13 +442,532 @@ And to add more versatility, the Python package doconce you can convert a standard ascii text file into various HTML 
-formats, ipython notebooks, latex files, pdf files etc with minimal edits.
+formats, ipython notebooks, latex files, pdf files etc with minimal edits. These lectures were generated using doconce.
 
-
Simple linear regression model using scikit-learn 

+
Numpy examples and Important Matrix and vector handling packages 

 
 

-We start with perhaps our simplest possible example, using scikit-learn to perform linear regression analysis on a data set produced by us. 
-What follows is a simple Python code where we have defined  function \( y \) in terms of the variable \( x \). Both are defined as vectors of dimension \( 1\times 100 \). The entries to the vector \( \hat{x} \)  are given by random numbers generated with a uniform distribution with entries \( x_i \in [0,1] \) (more about probability distribution functions later). These values are then used to define a function \( y(x) \) (tabulated again as a vector) with a linear dependence on \( x \) plus a random noise added via the normal distribution.
+There are several central software packages for linear algebra and eigenvalue problems. Several of the more
+popular ones have been wrapped into ofter software packages like those from the widely used text Numerical Recipes. The original source codes in many of the available packages are often taken from the widely used
+software package LAPACK, which follows two other popular packages
+developed in the 1970s, namely EISPACK and LINPACK.  We describe them shortly here.
+
+
    
+
+
  •  LINPACK: package for linear equations and least square problems.
    • 
+
+
  •  LAPACK:package for solving symmetric, unsymmetric and generalized eigenvalue problems. From LAPACK's website http://www.netlib.org it is possible to download for free all source codes from this library. Both C/C++ and Fortran versions are available.
    • 
+
+
  •  BLAS (I, II and III): (Basic Linear Algebra Subprograms) are routines that provide standard building blocks for performing basic vector and matrix operations. Blas I is vector operations, II vector-matrix operations and III matrix-matrix operations. Highly parallelized and efficient codes, all available for download from http://www.netlib.org.
    • 
+

+
+
Basic Matrix Features 

+
+

+

+Matrix properties reminder.
+
 

+$$
+ \mathbf{A} =
+      \begin{bmatrix} a_{11} & a_{12} & a_{13} & a_{14} \\
+                                 a_{21} & a_{22} & a_{23} & a_{24} \\
+                                   a_{31} & a_{32} & a_{33} & a_{34} \\
+                                  a_{41} & a_{42} & a_{43} & a_{44}
+             \end{bmatrix}\qquad
+\mathbf{I} =
+      \begin{bmatrix} 1 & 0 & 0 & 0 \\
+                                 0 & 1 & 0 & 0 \\
+                                 0 & 0 & 1 & 0 \\
+                                 0 & 0 & 0 & 1
+             \end{bmatrix}
+$$
+
 

+

+
+
Basic Matrix Features 

+

+
+

+The inverse of a matrix is defined by
+
+
 

+$$
+\mathbf{A}^{-1} \cdot \mathbf{A} = I
+$$
+
 

+

+
+
Basic Matrix Features 

+
+

+

+Matrix Properties Reminder.
+

+
+
+
+
+
+
+
+
+
+
+
+
                Relations                        Name                                    matrix elements                               
   \( A = A^{T} \)                                   symmetric              \( a_{ij} = a_{ji} \)                                                           
   \( A = \left (A^{T} \right )^{-1} \)              real orthogonal        \( \sum_k a_{ik} a_{jk} = \sum_k a_{ki} a_{kj} = \delta_{ij} \)                 
   \( A = A^{ * } \)                                 real matrix            \( a_{ij} = a_{ij}^{ * } \)                                                     
   \( A = A^{\dagger} \)                             hermitian              \( a_{ij} = a_{ji}^{ * } \)                                                     
   \( A = \left (A^{\dagger} \right )^{-1} \)        unitary                \( \sum_k a_{ik} a_{jk}^{ * } = \sum_k a_{ki}^{ * } a_{kj} = \delta_{ij} \)     

+
+

+
+
Some famous Matrices 

+
+
    
+
+
  •  Diagonal if \( a_{ij}=0 \) for \( i\ne j \)
    • 
+
+
  •  Upper triangular if \( a_{ij}=0 \) for \( i > j \)
    • 
+
+
  •  Lower triangular if \( a_{ij}=0 \) for \( i < j \)
    • 
+
+
  •  Upper Hessenberg if \( a_{ij}=0 \) for \( i > j+1 \)
    • 
+
+
  •  Lower Hessenberg if \( a_{ij}=0 \) for \( i < j+1 \)
    • 
+
+
  •  Tridiagonal if \( a_{ij}=0 \) for \( |i -j| > 1 \)
    • 
+
+
  •  Lower banded with bandwidth \( p \): \( a_{ij}=0 \) for \( i > j+p \)
    • 
+
+
  •  Upper banded with bandwidth \( p \): \( a_{ij}=0 \) for \( i < j+p \)
    • 
+
+
  •  Banded, block upper triangular, block lower triangular....
    • 
+

+
+
Basic Matrix Features 

+
+

+

+Some Equivalent Statements.
+

+For an \( N\times N \) matrix  \( \mathbf{A} \) the following properties are all equivalent
+
+
    
+
+
  •  If the inverse of \( \mathbf{A} \) exists, \( \mathbf{A} \) is nonsingular.
    • 
+
+
  •  The equation \( \mathbf{Ax}=0 \) implies \( \mathbf{x}=0 \).
    • 
+
+
  •  The rows of \( \mathbf{A} \) form a basis of \( R^N \).
    • 
+
+
  •  The columns of \( \mathbf{A} \) form a basis of \( R^N \).
    • 
+
+
  •  \( \mathbf{A} \) is a product of elementary matrices.
    • 
+
+
  •  \( 0 \) is not eigenvalue of \( \mathbf{A} \).
    • 
+

+

+
+
Numpy and arrays 

+Numpy provides an easy way to handle arrays in Python. The standard way to import this library is as
+
+

+
+
+
import numpy as np
+

+

+Here follows a simple example where we set up an array of ten elements, all determined by random numbers drawn according to the normal distribution,
+

+
+
+
n = 10
+x = np.random.normal(size=n)
+print(x)
+

+

+We defined a vector \( x \) with \( n=10 \) elements with its values given by the Normal distribution \( N(0,1) \).
+Another alternative is to declare a vector as follows
+

+
+
+
import numpy as np
+x = np.array([1, 2, 3])
+print(x)
+

+

+Here we have defined a vector with three elements, with \( x_0=1 \), \( x_1=2 \) and \( x_2=3 \). Note that both Python and C++
+start numbering array elements from \( 0 \) and on. This means that a vector with \( n \) elements has a sequence of entities \( x_0, x_1, x_2, \dots, x_{n-1} \). We could also let (recommended) Numpy to compute the logarithms of a specific array as
+

+
+
+
import numpy as np
+x = np.log(np.array([4, 7, 8]))
+print(x)
+

+

+In the last example we used Numpy's unary function \( np.log \). This function is
+highly tuned to compute array elements since the code is vectorized
+and does not require looping. We normaly recommend that you use the
+Numpy intrinsic functions instead of the corresponding log function
+from Python's math module. The looping is done explicitely by the
+np.log function. The alternative, and slower way to compute the
+logarithms of a vector would be to write
+
+

+
+
+
import numpy as np
+from math import log
+x = np.array([4, 7, 8])
+for i in range(0, len(x)):
+    x[i] = log(x[i])
+print(x)
+

+

+We note that our code is much longer already and we need to import the log function from the math module. 
+The attentive reader will also notice that the output is \( [1, 1, 2] \). Python interprets automagically our numbers as integers (like the automatic keyword in C++). To change this we could define our array elements to be double precision numbers as
+

+
+
+
import numpy as np
+x = np.log(np.array([4, 7, 8], dtype = np.float64))
+print(x)
+

+

+or simply write them as double precision numbers (Python uses 64 bits as default for floating point type variables), that is
+

+
+
+
import numpy as np
+x = np.log(np.array([4.0, 7.0, 8.0])
+print(x)
+

+

+To check the number of bytes (remember that one byte contains eight bits for double precision variables), you can use simple use the itemsize functionality (the array \( x \) is actually an object which inherits the functionalities defined in Numpy) as 
+

+
+
+
import numpy as np
+x = np.log(np.array([4.0, 7.0, 8.0])
+print(x.itemsize)
+

+
+
Matrices in Python 

+
+

+Having defined vectors, we are now ready to try out matrices. We can
+define a \( 3 \times 3  \) real matrix \( \hat{A} \) as (recall that we user
+lowercase letters for vectors and uppercase letters for matrices)
+
+

+
+
+
import numpy as np
+A = np.log(np.array([ [4.0, 7.0, 8.0], [3.0, 10.0, 11.0], [4.0, 5.0, 7.0] ]))
+print(A)
+

+

+If we use the shape function we would get \( (3, 3) \) as output, that is verifying that our matrix is a \( 3\times 3 \) matrix. We can slice the matrix and print for example the first column (Python organized matrix elements in a row-major order, see below) as
+

+
+
+
import numpy as np
+A = np.log(np.array([ [4.0, 7.0, 8.0], [3.0, 10.0, 11.0], [4.0, 5.0, 7.0] ]))
+# print the first column, row-major order and elements start with 0
+print(A[:,0]) 
+

+

+We can continue this was by printing out other columns or rows. The example here prints out the second column
+

+
+
+
import numpy as np
+A = np.log(np.array([ [4.0, 7.0, 8.0], [3.0, 10.0, 11.0], [4.0, 5.0, 7.0] ]))
+# print the first column, row-major order and elements start with 0
+print(A[1,:]) 
+

+

+Numpy contains many other functionalities that allow us to slice, subdivide etc etc arrays. We strongly recommend that you look up the Numpy website for more details. Useful functions when defining a matrix are the np.zeros function which declares a matrix of a given dimension and sets all elements to zero
+

+
+
+
import numpy as np
+n = 10
+# define a matrix of dimension 10 x 10 and set all elements to zero
+A = np.zeros( (n, n) )
+print(A) 
+

+

+or initializing all elements to 
+

+
+
+
import numpy as np
+n = 10
+# define a matrix of dimension 10 x 10 and set all elements to one
+A = np.ones( (n, n) )
+print(A) 
+

+

+or as unitarily distributed random numbers (see the material on random number generators in the statistics part)
+

+
+
+
import numpy as np
+n = 10
+# define a matrix of dimension 10 x 10 and set all elements to random numbers with x \in [0, 1]
+A = np.random.rand(n, n)
+print(A) 
+

+

+As we will see throughout these lectures, there are several extremely useful functionalities in Numpy.
+As an example, consider the discussion of the covariance matrix. Suppose we have defined three vectors
+\( \hat{x}, \hat{y}, \hat{z} \) with \( n \) elements each. The covariance matrix is defined as 
+
 

+$$
+\hat{\Sigma} = \begin{bmatrix} \sigma_{xx} & \sigma_{xy} & \sigma_{xz} \\
+                              \sigma_{yx} & \sigma_{yy} & \sigma_{yz} \\
+                              \sigma_{zx} & \sigma_{zy} & \sigma_{zz} 
+             \end{bmatrix},
+$$
+
 

+
+where for example
+
 

+$$
+\sigma_{xy} =\frac{1}{n} \sum_{i=0}^{n-1}(x_i- \overline{x})(y_i- \overline{y}).
+$$
+
 

+
+The Numpy function np.cov calculates the covariance elements using the factor \( 1/(n-1) \) instead of \( 1/n \) since it assumes we do not have the exact mean values. 
+The following simple function uses the np.vstack function which takes each vector of dimension \( 1\times n \) and produces a \( 3\times n \) matrix \( \hat{W} \)
+
 

+$$
+\hat{W} = \begin{bmatrix} x_0 & y_0 & z_0 \\
+                          x_1 & y_1 & z_1 \\
+                          x_2 & y_2 & z_2 \\
+                          \dots & \dots & \dots \\
+                          x_{n-2} & y_{n-2} & z_{n-2} \\
+                          x_{n-1} & y_{n-1} & z_{n-1}
+             \end{bmatrix},
+$$
+
 

+
+

+which in turn is converted into into the \( 3\times 3 \) covariance matrix
+\( \hat{\Sigma} \) via the Numpy function np.cov(). We note that we can also calculate
+the mean value of each set of samples \( \hat{x} \) etc using the Numpy
+function np.mean(x). We can also extract the eigenvalues of the
+covariance matrix through the np.linalg.eig() function.
+
+

+
+
+
# Importing various packages
+import numpy as np
+
+n = 100
+x = np.random.normal(size=n)
+print(np.mean(x))
+y = 4+3*x+np.random.normal(size=n)
+print(np.mean(y))
+z = x**3+np.random.normal(size=n)
+print(np.mean(z))
+W = np.vstack((x, y, z))
+Sigma = np.cov(W)
+print(Sigma)
+Eigvals, Eigvecs = np.linalg.eig(Sigma)
+print(Eigvals)
+

+

+
+
+
import numpy as np
+import matplotlib.pyplot as plt
+from scipy import sparse
+eye = np.eye(4)
+print(eye)
+sparse_mtx = sparse.csr_matrix(eye)
+print(sparse_mtx)
+x = np.linspace(-10,10,100)
+y = np.sin(x)
+plt.plot(x,y,marker='x')
+plt.show()
+

+
+
Meet the Pandas 

+
+

+




+
+

+Another useful Python package is
+pandas, which is an open source library
+providing high-performance, easy-to-use data structures and data
+analysis tools for Python. pandas stands for panel data, a term borrowed from econometrics and is an efficient library for data analysis with an emphasis on tabular data.
+pandas has two major classes, the DataFrame class with two-dimensional data objects and tabular data organized in columns and the class Series with a focus on one-dimensional data objects. Both classes allow you to index data easily as we will see in the examples below. 
+pandas allows you also to perform mathematical operations on the data, spanning from simple reshapings of vectors and matrices to statistical operations.
+
+

+The following simple example shows how we can, in an easy way make tables of our data. Here we define a data set which includes names, place of birth and date of birth, and displays the data in an easy to read way. We will see repeated use of pandas, in particular in connection with classification of data.
+
+

+
+
+
import pandas as pd
+from IPython.display import display
+data = {'First Name': ["Frodo", "Bilbo", "Aragorn II", "Samwise"],
+        'Last Name': ["Baggins", "Baggins","Elessar","Gamgee"],
+        'Place of birth': ["Shire", "Shire", "Eriador", "Shire"],
+        'Date of Birth T.A.': [2968, 2890, 2931, 2980]
+        }
+data_pandas = pd.DataFrame(data)
+display(data_pandas)
+

+

+In the above we have imported pandas with the shorthand pd, the latter has become the standard way we import pandas. We make then a list of various variables
+and reorganize the aboves lists into a DataFrame and then print out  a neat table with specific column labels as Name, place of birth and date of birth.
+Displaying these results, we see that the indices are given by the default numbers from zero to three.
+pandas is extremely flexible and we can easily change the above indices by defining a new type of indexing as
+

+
+
+
data_pandas = pd.DataFrame(data,index=['Frodo','Bilbo','Aragorn','Sam'])
+display(data_pandas)
+

+

+Thereafter we display the content of the row which begins with the index Aragorn
+

+
+
+
display(data_pandas.loc['Aragorn'])
+

+

+We can easily append data to this, for example
+

+
+
+
new_hobbit = {'First Name': ["Peregrin"],
+              'Last Name': ["Took"],
+              'Place of birth': ["Shire"],
+              'Date of Birth T.A.': [2990]
+              }
+data_pandas=data_pandas.append(pd.DataFrame(new_hobbit, index=['Pippin']))
+display(data_pandas)
+

+

+Here are other examples where we use the DataFrame functionality to handle arrays, now with more interesting features for us, namely numbers. We set up a matrix 
+of dimensionality \( 10\times 5 \) and compute the mean value and standard deviation of each column. Similarly, we can perform mathematial operations like squaring the matrix elements and many other operations. 
+

+
+
+
import numpy as np
+import pandas as pd
+from IPython.display import display
+np.random.seed(100)
+# setting up a 10 x 5 matrix
+rows = 10
+cols = 5
+a = np.random.randn(rows,cols)
+df = pd.DataFrame(a)
+display(df)
+print(df.mean())
+print(df.std())
+display(df**2)
+

+

+Thereafter we can select specific columns only and plot final results
+

+
+
+
df.columns = ['First', 'Second', 'Third', 'Fourth', 'Fifth']
+df.index = np.arange(10)
+
+display(df)
+print(df['Second'].mean() )
+print(df.info())
+print(df.describe())
+
+from pylab import plt, mpl
+plt.style.use('seaborn')
+mpl.rcParams['font.family'] = 'serif'
+
+df.cumsum().plot(lw=2.0, figsize=(10,6))
+plt.show()
+
+
+df.plot.bar(figsize=(10,6), rot=15)
+plt.show()
+

+

+We can produce a \( 4\times 4 \) matrix
+

+
+
+
b = np.arange(16).reshape((4,4))
+print(b)
+df1 = pd.DataFrame(b)
+print(df1)
+

+

+and many other operations.
+
+

+The Series class is another important class included in
+pandas. You can view it as a specialization of DataFrame but where
+we have just a single column of data. It shares many of the same features as _DataFrame. As with DataFrame,
+most operations are vectorized, achieving thereby a high performance when dealing with computations of arrays, in particular labeled arrays.
+As we will see below it leads also to a very concice code close to the mathematical operations we may be interested in.
+For multidimensional arrays, we recommend strongly xarray. xarray has much of the same flexibility as pandas, but allows for the extension to higher dimensions than two. We will see examples later of the usage of both pandas and xarray.
+
+
Reading Data and fitting  

+
+

+In order to study various Machine Learning algorithms, we need to
+access data. Acccessing data is an essential step in all machine
+learning algorithms. In particular, setting up the so-called design
+matrix (to be defined below) is often the first element we need in
+order to perform our calculations. To set up the design matrix means
+reading (and later, when the calculations are done, writing) data
+in various formats, The formats span from reading files from disk,
+loading data from databases and interacting with online sources
+like web application programming interfaces (APIs).
+
+

+In handling various input formats, as discussed above, we will mainly stay with pandas,
+a Python package which allows us, in a seamless and painless way, to
+deal with a multitude of formats, from standard csv (comma separated
+values) files, via excel, html to hdf5 formats.  With pandas
+and the DataFrame  and Series functionalities we are able to convert text data
+into the calculational formats we need for a specific algorithm. And our code is going to be 
+pretty close the basic mathematical expressions.
+
+

+Our first data set is going to be a classic from nuclear physics, namely all
+available data on binding energies. Don't be intimidated if you are not familiar with nuclear physics. It serves simply as an example here of a data set.
+
+

+We will show some of the
+strengths of packages like Scikit-Learn in fitting nuclear binding energies to
+specific functions using linear regression first. Then, as a teaser, we will show you how 
+you can easily implement other algorithms like decision trees and random forests and neural networks.
+
+

+But before we really start with nuclear physics data, let's just look at some simpler polynomial fitting cases, such as,
+(don't be offended) fitting straight lines!
+
+
Simple linear regression model using scikit-learn 

+
+

+We start with perhaps our simplest possible example, using Scikit-Learn to perform linear regression analysis on a data set produced by us.
+
+

+What follows is a simple Python code where we have defined a function
+\( y \) in terms of the variable \( x \). Both are defined as vectors with  \( 100 \) entries. 
+The numbers in the vector \( \hat{x} \) are given
+by random numbers generated with a uniform distribution with entries
+\( x_i \in [0,1] \) (more about probability distribution functions
+later). These values are then used to define a function \( y(x) \)
+(tabulated again as a vector) with a linear dependence on \( x \) plus a
+random noise added via the normal distribution.
 
 

 The Numpy functions are imported used the import numpy as np
@@ -345,7 +988,7 @@ $$
 
 

 where \( N(0,1) \) represents random numbers generated by the normal
-distribution.  From scikit-learn we import then the
+distribution.  From Scikit-Learn we import then the
 LinearRegression functionality and make a prediction \( \tilde{y} =
 \alpha + \beta x \) using the function fit(x,y). We call the set of
 data \( (\hat{x},\hat{y}) \) for our training data. The Python package
@@ -405,7 +1048,7 @@ $$
 
 

 where \( x \) is defined as before.  Does the fit look better? Indeed, by
-reducing the role of the normal distribution we see immediately that
+reducing the role of the noise given by the normal distribution we see immediately that
 our linear prediction seemingly reproduces better the training
 set. However, this testing 'by the eye' is obviouly not satisfactory in the
 long run. Here we have only defined the training data and our model, and 
@@ -416,7 +1059,7 @@ We need more rigorous criteria in defining whether we have succeeded or
 not in modeling our training data.  You will be surprised to see that
 many scientists seldomly venture beyond this 'by the eye' approach. A
 standard approach for the cost function is the so-called \( \chi^2 \)
-function
+function (a variant of the mean-squared error (MSE))
 
 
 

 $$ \chi^2 = \frac{1}{n}
@@ -448,7 +1091,7 @@ the  \( \chi^2 \) function becomes smaller.
 
 

 There are many ways to define the cost function. A simpler approach is to look at the relative difference between the training data and the predicted data, that is we define 
-the relative error as
+the relative error (why would we prefer the MSE instead of the relative error?) as
 
 
 

 $$
@@ -456,7 +1099,7 @@ $$
 $$
 
 

 
-We can modify easily the above Python code and plot the relative instead
+We can modify easily the above Python code and plot the relative error instead
 

 
 
@@ -484,14 +1127,14 @@ different training data sets and study (graphically) the value of the
 relative error.
 
 

-As mentioned above, scikit-learn has an impressive functionality.
+As mentioned above, Scikit-Learn has an impressive functionality.
 We can for example extract the values of \( \alpha \) and \( \beta \) and
 their error estimates, or the variance and standard deviation and many
 other properties from the statistical data analysis.
 
 

 Here we show an
-example of the functionality of scikit-learn.
+example of the functionality of Scikit-Learn.
 

 
 
@@ -560,8 +1203,8 @@ $$
 $$
 
 

 
-Another quantity will meet again in our discussions of regression analysis is 
- mean absolute error (MAE), a risk metric corresponding to the expected value of the absolute error loss or what we call the \( l1 \)-norm loss. In our discussion above we presented the relative error.
+Another quantity taht we will meet again in our discussions of regression analysis is 
+ the mean absolute error (MAE), a risk metric corresponding to the expected value of the absolute error loss or what we call the \( l1 \)-norm loss. In our discussion above we presented the relative error.
 The MAE is defined as follows
 
 

 $$
@@ -622,847 +1265,378 @@ plt.show()
 
 print (error(y))
 

-

-Similarly, using R, we can perform similar studies. The following R code illustrates this.
-(more details on R will be inserted later).
 
-
Non-Linear Least squares in R  

-

-
-

-
-
-
set.seed(1485)
-len = 24
-x = runif(len)
-y = x^3+rnorm(len, 0,0.06)
-ds = data.frame(x = x, y = y)
-str(ds)
-plot( y ~ x, main ="Known cubic with noise")
-s  = seq(0,1,length =100)
-lines(s, s^3, lty =2, col ="green")
-m = nls(y ~ I(x^power), data = ds, start = list(power=1), trace = T)
-class(m)
-summary(m)
-power = round(summary(m)$coefficients[1], 3)
-power.se = round(summary(m)$coefficients[2], 3)
-plot(y ~ x, main = "Fitted power model", sub = "Blue: fit; green: known")
-s = seq(0, 1, length = 100)
-lines(s, s^3, lty = 2, col = "green")
-lines(s, predict(m, list(x = s)), lty = 1, col = "blue")
-text(0, 0.5, paste("y =x^ (", power, " +/- ", power.se, ")", sep = ""), pos = 4)
-

-
-

+
To our real data: nuclear binding energies. Brief reminder on masses and binding energies 

 
 

-In our lectures on regression analysis (and other ones as well), we will discuss in more details various R functionalities.
+Let us now dive into  nuclear physics and remind ourselves briefly about some basic features about binding
+energies.  A basic quantity which can be measured for the ground
+states of nuclei is the atomic mass \( M(N, Z) \) of the neutral atom with
+atomic mass number \( A \) and charge \( Z \). The number of neutrons is \( N \). There are indeed several sophisticated experiments worldwide which allow us to measure this quantity to high precision (parts per million even).
 
 

-Another useful Python package is
-pandas, which is an open source library
-providing high-performance, easy-to-use data structures and data
-analysis tools for Python. The following simple example shows how we can, in an easy way make tables of our data. Here we define a data set which includes names, city of residence and age, and displays the data in an easy to read way. We will see repeated use of pandas, in particular in connection with classification of data.
-
-

-
-
-
import pandas as pd
-from IPython.display import display
-data = {'Name': ["John", "Anna", "Peter", "Linda"], 'Location': ["Nairobi", "Napoli", "London", "Buenos Aires"], 'Age':[51, 21, 34, 45]}
-data_pandas = pd.DataFrame(data)
-display(data_pandas)
-

-
-
Examples 

-
-

-We present here several examples, with pertinent Python codes that we
-will use to illustrate various machine learning methods and ways to
-analyze, from simple to complex, various data sets. Many of these
-examples allow us to generate the data we want to analyze, following
-much of the same philosophy we discussed above when
-fitting various polynomials.
-
-

-We start with a simple exponential growth model that is meant to mimick an ecoli lab experiment.
-We can easily model this system and then produce the data used to train various machine learning algorithms.
-Another model from the life sciences is the so-called predator-prey model from ecology. Thereafter we present 
-a simple model for financial transactions before moving to a random walk model and ending with 
-the simulation of velocities of a non-interacting atom or molecule confined to move in  a one-dimensional region.
-
-
Ecoli lab experiment 

-
-

-A typical pattern seen in population models is that the population grows faster and faster. Why? Is there an underlying (general) mechanism?
-Here we will construct a model for cell growth based on a simple difference equation for the growth. We make the following assumptions
-

-
-
    
-
  1.  Cells divide after \( T \) seconds on average (one generation)
    2. 
-
  2.  \( 2N \) celles divide into twice as many new cells \( \Delta N \) in a time
-   interval \( \Delta t \) as \( N \) cells would: \( \Delta N \propto N \)
    3. 
-
  3.  \( N \) cells result in twice as many new individuals \( \Delta N \) in
-   time \( 2\Delta t \) as in time \( \Delta t \): \( \Delta N \propto\Delta t \)
    4. 
-
  4.  Same proportionality with respect to  death
    5.  
-
  5.  Proposed model: \( \Delta N = b\Delta t N - d\Delta tN \) for some unknown
-   constants \( b \) (births) and \( d \) (deaths)
    6. 
-
  6.  Describe evolution in discrete time: \( t_n=n\Delta t \)
    7. 
-
  7.  Program-friendly notation: \( N \) at \( t_n \) is \( N^n \)
    8. 
-
  8.  Math model: \( N^{n+1} = N^n + r\Delta t\, N \) (with \( \ r=b-d \))
    9. 
-
  9.  Program model: N[n+1] = N[n] + r*dt*N[n]
    10. 
-

-

-
-

-The difference equation can be programmed in a simple way, and in order to get started we
-set \( r=1.5 \), \( N^0=1 \), \( \Delta t=0.5 \). The program reads
-
-

-
-
-
import numpy as np
-
-t = np.linspace(0, 10, 21)  # 20 intervals in [0, 10]
-dt = t[1] - t[0]
-N = np.zeros(t.size)
-N[0] = 1
-r = 0.5
-
-for n in range(0, N.size-1, 1):
-    N[n+1] = N[n] + r*dt*N[n]
-    print('N[%d]=%.1f' % (n+1, N[n+1]))
-

-

-and it generates the following output
-

-
-
-
N[1]=1.2
-N[2]=1.6
-N[3]=2.0
-N[4]=2.4
-N[5]=3.1
-N[6]=3.8
-N[7]=4.8
-N[8]=6.0
-N[9]=7.5
-N[10]=9.3
-N[11]=11.6
-N[12]=14.6
-N[13]=18.2
-N[14]=22.7
-N[15]=28.4
-N[16]=35.5
-N[17]=44.4
-N[18]=55.5
-N[19]=69.4
-N[20]=86.7
-

-

-This forms our data which later will define our training set. 
-In this case we defined the value of the parameter \( r \). We could alternatively assume that we just received the 
-above data file and where asked to find \( r \). How can we estimate \( r \) from data? This will be one of our tasks later.
-
-

-We can use the difference equation with the experimental data
+Atomic masses are usually tabulated in terms of the mass excess defined by
 
 

-$$ N^{n+1} = N^n + r\Delta t N^n$$
+$$
+\Delta M(N, Z) =  M(N, Z) - uA,
+$$
 
 

 
-Suppose now that  \( N^{n+1} \) and \( N^n \) are known from data. Then we could solve with respect to  \( r \) as follows
+where \( u \) is the Atomic Mass Unit 
 
 

-$$ r = \frac{N^{n+1}-N^n}{N^n\Delta t} $$
+$$
+u = M(^{12}\mathrm{C})/12 = 931.4940954(57) \hspace{0.1cm} \mathrm{MeV}/c^2.
+$$
 
 

 
-Suppose we set \( t_1=600 \), \( t_2=1200 \),
-\( N^1=140 \) and \( N^2=250 \). 
-The following code plots the data 
-

+The nucleon masses are
+
 

+$$
+m_p =  1.00727646693(9)u,
+$$
+
 

 
-
-
import numpy as np
-import matplotlib.pyplot as plt
-
-# Estimate r
-data = np.loadtxt('ecoli.csv', delimiter=',')
-t_e = data[:,0]
-N_e = data[:,1]
-i = 2  # Data point (i,i+1) used to estimate r
-r = (N_e[i+1] - N_e[i])/(N_e[i]*(t_e[i+1] - t_e[i]))
-print('Estimated r=%.5f' % r)
-# Can experiment with r values and see if the model can
-# match the data better
-T = 1200     # cell can divide after T sec
-t_max = 5*T  # 5 generations in experiment
-t = np.linspace(0, t_max, 1000)
-dt = t[1] - t[0]
-N = np.zeros(t.size)
-
-N[0] = 100
-for n in range(0, len(t)-1, 1):
-    N[n+1] = N[n] + r*dt*N[n]
-
-plt.plot(t, N, 'r-', t_e, N_e, 'bo')
-plt.xlabel('time [s]');  plt.ylabel('N')
-plt.legend(['model', 'experiment'], loc='upper left')
-plt.show()
-

-

-We can then change the parameter \( r \) in the program and play around to make a better fit. By now we know that this
-'search bythe eye' approach is not the most optimal one.
-
-
Predator-Prey model from ecology 

+and
+
 

+$$
+m_n = 939.56536(8)\hspace{0.1cm} \mathrm{MeV}/c^2 = 1.0086649156(6)u.
+$$
+
 

 
 

-The population dynamics of a simple predator-prey system is a
-classical example shown in many biology textbooks when ecological
-systems are discussed. The system contains all elements of the
-scientific method:
+In the 2016 mass evaluation of by W.J.Huang, G.Audi, M.Wang, F.G.Kondev, S.Naimi and X.Xu
+there are data on masses and decays of 3437 nuclei.
+
+

+The nuclear binding energy is defined as the energy required to break
+up a given nucleus into its constituent parts of \( N \) neutrons and \( Z \)
+protons. In terms of the atomic masses \( M(N, Z) \) the binding energy is
+defined by
+
+
 

+$$
+BE(N, Z) = ZM_H c^2 + Nm_n c^2 - M(N, Z)c^2 ,
+$$
+
 

+
+where \( M_H \) is the mass of the hydrogen atom and \( m_n \) is the mass of the neutron.
+In terms of the mass excess the binding energy is given by
+
 

+$$
+BE(N, Z) = Z\Delta_H c^2 + N\Delta_n c^2 -\Delta(N, Z)c^2 ,
+$$
+
 

+
+where \( \Delta_H c^2 = 7.2890 \) MeV and \( \Delta_n c^2 = 8.0713 \) MeV.
+
+

+A popular and physically intuitive model which can be used to parametrize 
+the experimental binding energies as function of \( A \), is the so-called 
+liquid drop model. The ansatz is based on the following expression
+
+
 

+$$ 
+BE(N,Z) = a_1A-a_2A^{2/3}-a_3\frac{Z^2}{A^{1/3}}-a_4\frac{(N-Z)^2}{A},
+$$
+
 

+
+

+where \( A \) stands for the number of nucleons and the $a_i$s are parameters which are determined by a fit 
+to the experimental data.
+
+

+To arrive at the above expression we have assumed that we can make the following assumptions:
 
 
    
- 
  •  The set up of a specific hypothesis combined with
    • 
- 
  •  the experimental methods needed (one can study existing data or perform experiments)
    • 
- 
  •  analyzing and interpreting the data and performing further experiments if needed
    • 
- 
  •  trying to extract general behaviors and extract eventual laws or patterns
    • 
- 
  •  develop mathematical relations for the uncovered regularities/laws and test these by per forming new experiments
    • 
+ 
  •  There is a volume term \( a_1A \) proportional with the number of nucleons (the energy is also an extensive quantity). When an assembly of nucleons of the same size is packed together into the smallest volume, each interior nucleon has a certain number of other nucleons in contact with it. This contribution is proportional to the volume.
    • 
+ 
  •  There is a surface energy term \( a_2A^{2/3} \). The assumption here is that a nucleon at the surface of a nucleus interacts with fewer other nucleons than one in the interior of the nucleus and hence its binding energy is less. This surface energy term takes that into account and is therefore negative and is proportional to the surface area.
    • 
+ 
  •  There is a Coulomb energy term \( a_3\frac{Z^2}{A^{1/3}} \). The electric repulsion between each pair of protons in a nucleus yields less binding.
    • 
+
+
  •  There is an asymmetry term \( a_4\frac{(N-Z)^2}{A} \). This term is associated with the Pauli exclusion principle and reflects the fact that the proton-neutron interaction is more attractive on the average than the neutron-neutron and proton-proton interactions.
    • 
 

 

 
-Lots of data about populations of hares and lynx collected from furs in Hudson Bay, Canada, are available. It is known that the populations oscillate. Why?
-Here we start by
+We could also add a so-called pairing term, which is a correction term that
+arises from the tendency of proton pairs and neutron pairs to
+occur. An even number of particles is more stable than an odd number.
 
-
    
-
  1.  plotting the data
    2. 
-
  2.  derive a simple model for the population dynamics
    3. 
-
  3.  (fitting parameters in the model to the data)
    4. 
-
  4.  using the model predict the evolution other predator-pray systems
    5. 
-

-

-
-Most mammalian predators rely on a variety of prey, which complicates mathematical modeling; however, a few predators have become highly specialized and seek almost exclusively a single prey species. An example of this simplified predator-prey interaction is seen in Canadian northern forests, where the populations of the lynx and the snowshoe hare are intertwined in a life and death struggle.
+
Organizing our data 

 
 

-One reason that this particular system has been so extensively studied is that the Hudson Bay company kept careful records of all furs from the early 1800s into the 1900s. The records for the furs collected by the Hudson Bay company showed distinct oscillations (approximately 12 year periods), suggesting that these species caused almost periodic fluctuations of each other's populations. The table here shows data from 1900 to 1920.
+Let us start with reading and organizing our data. 
+We start with the compilation of masses and binding energies from 2016.
+After having downloaded this file to our own computer, we are now ready to read the file and start structuring our data.
 
 

-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
Year Hares (x1000) Lynx (x1000) 
   1900        30.0                 4.0              
   1901        47.2                 6.1              
   1902        70.2                 9.8              
   1903        77.4                 35.2             
   1904        36.3                 59.4             
   1905        20.6                 41.7             
   1906        18.1                 19.0             
   1907        21.4                 13.0             
   1908        22.0                 8.3              
   1909        25.4                 9.1              
   1910        27.1                 7.4              
   1911        40.3                 8.0              
   1912        57                   12.3             
   1913        76.6                 19.5             
   1914        52.3                 45.7             
   1915        19.5                 51.1             
   1916        11.2                 29.7             
   1917        7.6                  15.8             
   1918        14.6                 9.7              
   1919        16.2                 10.1             
   1920        24.7                 8.6              

-

-
-
-
import numpy as np
-from  matplotlib import pyplot as plt
-
-# Load in data file
-data = np.loadtxt('src/Hudson_Bay.csv', delimiter=',', skiprows=1)
-# Make arrays containing x-axis and hares and lynx populations
-year = data[:,0]
-hares = data[:,1]
-lynx = data[:,2]
-
-plt.plot(year, hares ,'b-+', year, lynx, 'r-o')
-plt.axis([1900,1920,0, 100.0])
-plt.xlabel(r'Year')
-plt.ylabel(r'Numbers of hares and lynx ')
-plt.legend(('Hares','Lynx'), loc='upper right')
-plt.title(r'Population of hares and lynx from 1900-1920 (x1000)}')
-plt.savefig('Hudson_Bay_data.pdf')
-plt.savefig('Hudson_Bay_data.png')
-plt.show()
-

-

-




-
-

-We see from the plot that there are indeed fluctuations.
-We would like to create a mathematical model that explains these
-population fluctuations. Ecologists have predicted that in a simple
-predator-prey system that a rise in prey population is followed (with
-a lag) by a rise in the predator population. When the predator
-population is sufficiently high, then the prey population begins
-dropping. After the prey population falls, then the predator
-population falls, which allows the prey population to recover and
-complete one cycle of this interaction. Thus, we see that
-qualitatively oscillations occur. Can a mathematical model predict
-this? What causes cycles to slow or speed up? What affects the
-amplitude of the oscillation or do you expect to see the oscillations
-damp to a stable equilibrium? The models tend to ignore factors like
-climate and other complicating factors. How significant are these?
-
-
    
- 
  •  We see oscillations in the data
    • 
- 
  •  What causes cycles to slow or speed up?
    • 
- 
  •  What affects the amplitude of the oscillation or do you expect to see the oscillations damp to a stable equilibrium?
    • 
- 
  •  With a model we can better understand the data
    • 
- 
  •  More important: Can we understand the ecology dynamics of predator-pray populations?
    • 
-

-

-
-The classical way (in all books) is to present the Lotka-Volterra equations:
-
-
 

-$$
-\begin{align*}
-\frac{dH}{dt} &= H(a - b L)\\
-\frac{dL}{dt} &= - L(d - c  H)
-\end{align*}
-$$
-
 

-
-

-Here,
-
-
    
- 
  •  \( H \) is the number of preys
    • 
- 
  •  \( L \) the number of predators
    • 
- 
  •  \( a \), \( b \), \( d \), \( c \) are parameters
    • 
-

-

-
-The population of hares evolves due to births and deaths exactly as a bacteria population:
-
-
 

-$$
-\Delta H = a \Delta t H^n
-$$
-
 

-
-However, hares have an additional loss in the population because
-they are eaten by lynx.
-All the hares and lynx can form
-\( H\cdot L \) pairs in total. When such pairs meet during a time
-interval \( \Delta t \), there is some
-small probablity that the lynx will eat the hare.
-So in fraction \( b\Delta t HL \), the lynx eat hares. This
-loss of hares must be accounted for. Subtracted in the equation for hares:
-
-
 

-$$ \Delta H = a\Delta t H^n - b \Delta t H^nL^n$$
-
 

-
-

-We assume that the primary growth for the lynx population depends on sufficient food for raising lynx kittens, which implies an adequate source of nutrients from predation on hares. Thus, the growth of the lynx population does not only depend of how many lynx there are, but on how many hares they can eat.
-In a time interval \( \Delta t HL \) hares and lynx can meet, and in a
-fraction \( b\Delta t HL \) the lynx eats the hare. All of this does not
-contribute to the growth of lynx, again just a fraction of
-\( b\Delta t HL \) that we write as
-\( d\Delta t HL \). In addition, lynx die just as in the population
-dynamics with one isolated animal population, leading to a loss
-\( -c\Delta t L \).
-The accounting of lynx then looks like
-
 

-$$ \Delta L = d\Delta t H^nL^n - c\Delta t L^n$$
-
 

-
-

-By writing up the definition of \( \Delta H \) and \( \Delta L \), and putting
-all assumed known terms \( H^n \) and \( L^n \) on the right-hand side, we have
-
-
 

-$$ H^{n+1} = H^n + a\Delta t H^n - b\Delta t H^n L^n $$
-
 

-
-
 

-$$ L^{n+1} = L^n + d\Delta t H^nL^n - c\Delta t L^n $$
-
 

-
-

-Note:
-
-
    
- 
  •  These equations are ready to be implemented!
    • 
- 
  •  But to start, we need \( H^0 \) and \( L^0 \) (which we can get from the data)
    • 
- 
  •  We also need values for \( a \), \( b \), \( d \), \( c \)
    • 
- 
  •  As always, models tend to be general - as here, applicable
-   to "all" predator-pray systems
    • 
- 
  •  The critical issue is whether the interaction between hares and lynx
-   is sufficiently well modeled by \( \hbox{const}HL \)
    • 
- 
  •  The parameters \( a \), \( b \), \( d \), and \( c \) must be
-   estimated from data
    • 
-

-

-
-

-
-

-
-
-
import numpy as np
-import matplotlib.pyplot as plt
-
-def solver(m, H0, L0, dt, a, b, c, d, t0):
-    """Solve the difference equations for H and L over m years
-    with time step dt (measured in years."""
-
-    num_intervals = int(m/float(dt))
-    t = np.linspace(t0, t0 + m, num_intervals+1)
-    H = np.zeros(t.size)
-    L = np.zeros(t.size)
-
-    print('Init:', H0, L0, dt)
-    H[0] = H0
-    L[0] = L0
-
-    for n in range(0, len(t)-1):
-        H[n+1] = H[n] + a*dt*H[n] - b*dt*H[n]*L[n]
-        L[n+1] = L[n] + d*dt*H[n]*L[n] - c*dt*L[n]
-    return H, L, t
-
-# Load in data file
-data = np.loadtxt('src/Hudson_Bay.csv', delimiter=',', skiprows=1)
-# Make arrays containing x-axis and hares and lynx populations
-t_e = data[:,0]
-H_e = data[:,1]
-L_e = data[:,2]
-
-# Simulate using the model
-H, L, t = solver(m=20, H0=34.91, L0=3.857, dt=0.1,
-                 a=0.4807, b=0.02482, c=0.9272, d=0.02756,
-                 t0=1900)
-
-# Visualize simulations and data
-plt.plot(t_e, H_e, 'b-+', t_e, L_e, 'r-o', t, H, 'm--', t, L, 'k--')
-plt.xlabel('Year')
-plt.ylabel('Numbers of hares and lynx')
-plt.axis([1900, 1920, 0, 140])
-plt.title(r'Population of hares and lynx 1900-1920 (x1000)')
-plt.legend(('H_e', 'L_e', 'H', 'L'), loc='upper left')
-plt.savefig('Hudson_Bay_sim.pdf')
-plt.savefig('Hudson_Bay_sim.png')
-plt.show()
-

-
-

-
-

-




-
-

-We will later perform a least-square fitting. Then we can find optimal
-values for the parameters \( a \), \( b \), \( d \), \( c \). In our calculations here
-we set \( a=0.4807 \), \( b=0.02482 \), \( d=0.9272 \) and \( c=0.02756 \). These
-parameters result in a slightly modified initial conditions, namely
-\( H(0) = 34.91 \) and \( L(0)=3.857 \).
-
-

-The following Python code demonstrates how we can use linear regression to fit for example the population of lynx.
-Similarly, we have also used a decision tree algorithm to fit the lynx population data. As expected, the linear regression is not exactly impressive 
+We start with preparing folders for storing our calculations and the data file over masses and binding energies. We import also various modules that we will find useful in order to present various Machine Learning methods. Here we focus mainly on the functionality of scikit-learn.
 

 
 
-
import numpy as np
-import matplotlib.pyplot as plt
-from IPython.display import display
-import sklearn
-from sklearn.linear_model import LinearRegression
-from sklearn.tree import DecisionTreeRegressor
-
-
-data = np.loadtxt('src/Hudson_Bay.csv', delimiter=',', skiprows=1)
-x = data[:,0]
-y = data[:,1]
-line = np.linspace(1900,1920,1000,endpoint=False).reshape(-1,1)
-reg = DecisionTreeRegressor(min_samples_split=3).fit(x.reshape(-1,1),y.reshape(-1,1))
-plt.plot(line, reg.predict(line), label="decision tree")
-regline = LinearRegression().fit(x.reshape(-1,1),y.reshape(-1,1))
-plt.plot(line, regline.predict(line), label= "Linear Regression")
-plt.plot(x, y, label= "Linear Regression")
-plt.show()
-

-

-The similar code for linear regression in R reads (more details to come)
-

-
-
-
HudsonBay = read.csv("src/Hudson_Bay.csv",header=T)
-fix(HudsonBay)
-dim(HudsonBay)
-names(HudsonBay)
-plot(HudsonBay$Year, HudsonBay$Hares..x1000.)
-attach(HudsonBay)
-plot(Year, Hares..x1000.)
-plot(Year, Hares..x1000., col="red", varwidth=T, xlab="Years", ylab="Haresx 1000")
-summary(HudsonBay)
-summary(Hares..x1000.)
-library(MASS)
-library(ISLR)
-scatter.smooth(x=Year, y = Hares..x1000.)
-linearMod = lm(Hares..x1000. ~ Year)
-print(linearMod)
-summary(linearMod)
-plot(linearMod)
-confint(linearMod)
-predict(linearMod,data.frame(Year=c(1910,1914,1920)),interval="confidence")
-

-
-
Simulating financial transactions 

-
-

-The aim here is to simulate financial transactions among financial agents
-using Monte Carlo methods. The final goal is to extract a distribution of income  as function
-of the income \( m \).   From Pareto's work (V. Pareto, 1897) it is known from empirical studies
-that the higher end of the distribution of money follows a distribution 
-
 

-$$
-w_m\propto m^{-1-\alpha},
-$$
-
 

-
-with \( \alpha\in [1,2] \). We will here follow the analysis made by Patriarca and collaborators.
-
-

-Here we will study numerically the relation between the micro-dynamic relations among financial 
-agents and the  resulting macroscopic money distribution.
-
-

-We assume we have \( N \) agents that exchange money in pairs \( (i,j) \). We assume also that all agents
-start with the same amount of money \( m_0 > 0 \). At a given 'time step', we choose randomly a pair
-of agents \( (i,j) \) and let a transaction take place. This means that agent \( i \)'s money \( m_i \) changes
-to \( m_i' \) and similarly we have \( m_j\rightarrow m_j' \). 
-Money is conserved during a transaction, meaning that
-
 

-$$
-\begin{equation}
-  m_i+m_j=m_i'+m_j'.
-\tag{1}
-\end{equation}
-$$
-
 

-
-The change is done via a random reassignement (a random number) \( \epsilon \), meaning that
-
-
 

-$$
-\begin{equation*}
-m_i' = \epsilon(m_i+m_j),
-\end{equation*}
-$$
-
 

-
-leading to
-
-
 

-$$
-\begin{equation*}
-m_j'= (1-\epsilon)(m_i+m_j).
-\end{equation*}
-$$
-
 

-
-The number \( \epsilon \) is extracted from a uniform distribution.
-In this simple model, no agents are left with a debt, that is \( m\ge 0 \).
-Due to the conservation law above, one can show that the system relaxes toward an equilibrium
-state given by a Gibbs distribution
-
-
 

-$$
-\begin{equation*}
-w_m=\beta \exp{(-\beta m)},
-\end{equation*}
-$$
-
 

-
-with
-
-
 

-$$
-\begin{equation*}
-\beta = \frac{1}{\langle m\rangle},
-\end{equation*}
-$$
-
 

-
-and \( \langle m\rangle=\sum_i m_i/N=m_0 \), the average money.
-It means that after equilibrium has been reached that the majority of agents is left with a small
-number of money, while the number of richest agents, those with \( m \) larger than a specific value \( m' \),
-exponentially decreases with \( m' \).
-
-

-We assume that we have \( N=500 \) agents.   In each simulation, we need a sufficiently large number of transactions, say \( 10^7 \). Our aim is find the final equilibrium distribution \( w_m \). In order to do that we would need
-several runs of the above simulations, at least \( 10^3-10^4 \) runs (experiments).
-
-

-Our task is to first set up an algorithm which simulates the above transactions with an initial
-  amount \( m_0 \).
-  The challenge here is to figure out a Monte Carlo  simulation  based on the
-  above equations.
-  You will in particular need to make an algorithm which sets up a histogram as function of \( m \).
-  This histogram contains the number of times a value \( m \) is registered and represents
-  \( w_m\Delta m \). You will need to set up a value for the interval \( \Delta m \)  (typically \( 0.01-0.05 \)).
-  That means you need to account for the number of times you register an income in the interval
-  \( m,m+\Delta m \). The number of times you register this income, represents the value that enters the histogram.
-
-

-
-
-
#!/usr/bin/env python
+
# Common imports
 import numpy as np
-import matplotlib.mlab as mlab
+import pandas as pd
 import matplotlib.pyplot as plt
-import random
+import sklearn.linear_model as skl
+from sklearn.model_selection import train_test_split
+from sklearn.metrics import mean_squared_error, r2_score, mean_absolute_error
+import os
 
-# initialize the rng with a seed
-random.seed()
-# Hard coding of input parameters
-Agents  = 500
-MCcounts = 1000
-Transactions = 100000
-startMoney = 1.0
-Lambda = 0.0
-FinancialAgents = startMoney*np.ones(Agents)
-for i in range (1, MCcounts, 1):
-    for j in range (1, Transactions, 1):
-        agent_i = int(Agents*random.random())
-        agent_j = int(Agents*random.random())
-        epsilon = random.random()
-        if agent_i != agent_j:
-           m1 = Lambda*FinancialAgents[agent_i] + (1-Lambda)*epsilon*(FinancialAgents[agent_i] + FinancialAgents[agent_j])
-           m2 = Lambda*FinancialAgents[agent_j] + (1-Lambda)*(1-epsilon)*(FinancialAgents[agent_i] + FinancialAgents[agent_j])
-           FinancialAgents[agent_i] = m1
-           FinancialAgents[agent_j] = m2
+# Where to save the figures and data files
+PROJECT_ROOT_DIR = "Results"
+FIGURE_ID = "Results/FigureFiles"
+DATA_ID = "DataFiles/"
 
-# the histogram of the data
-n, bins, patches = plt.hist(FinancialAgents, 50, facecolor='green')
+if not os.path.exists(PROJECT_ROOT_DIR):
+    os.mkdir(PROJECT_ROOT_DIR)
 
-plt.xlabel('$x$')
-plt.ylabel('Distribution of wealth')
-plt.title(r'Money')
-plt.axis([0, 10, 0, 500])
-plt.grid(True)
-plt.show()
+if not os.path.exists(FIGURE_ID):
+    os.makedirs(FIGURE_ID)
+
+if not os.path.exists(DATA_ID):
+    os.makedirs(DATA_ID)
+
+def image_path(fig_id):
+    return os.path.join(FIGURE_ID, fig_id)
+
+def data_path(dat_id):
+    return os.path.join(DATA_ID, dat_id)
+
+def save_fig(fig_id):
+    plt.savefig(image_path(fig_id) + ".png", format='png')
+
+infile = open(data_path("MassEval2016.dat"),'r')
 

 

-We can then change our model to allow for a saving criterion, meaning that the agents save
-  a fraction \( \lambda \) of the money they have before the transaction is made. The final distribution will then no longer be given by Gibbs distribution. It could also include a taxation on financial transactions.
+Before we proceed, we define also a function for making our plots. You can obviously avoid this and simply set up various matplotlib commands every time you need them. You may however find it convenient to collect all such commands in one function and simply call this function. 
+

+
+
+
from pylab import plt, mpl
+plt.style.use('seaborn')
+mpl.rcParams['font.family'] = 'serif'
+
+def MakePlot(x,y, styles, labels, axlabels):
+    plt.figure(figsize=(10,6))
+    for i in range(len(x)):
+        plt.plot(x[i], y[i], styles[i], label = labels[i])
+        plt.xlabel(axlabels[0])
+        plt.ylabel(axlabels[1])
+    plt.legend(loc=0)
+

+

+Our next step is to read the data on experimental binding energies and
+reorganize them as functions of the mass number \( A \), the number of
+protons \( Z \) and neutrons \( N \) using pandas.  Before we do this it is
+always useful (unless you have a binary file or other types of compressed
+data) to actually open the file and simply take a look at it!
 
 

-  The conservation law of Eq. (1) holds, but the money to be shared in a transaction between
-  agent \( i \) and agent \( j \) is now \( (1-\lambda)(m_i+m_j) \). This means that we have
+In particular, the program that outputs the final nuclear masses is written in Fortran with a specific format. It means that we need to figure out the format and which columns contain the data we are interested in. Pandas comes with a function that reads formatted output. After having admired the file, we are now ready to start massaging it with pandas. The file begins with some basic format information.
+

 
-
 

-$$
-\begin{equation*}
-  m_i' = \lambda m_i+\epsilon(1-\lambda)(m_i+m_j),
-  \end{equation*}
-$$
-
 

-
-  and
-
-
 

-$$
-\begin{equation*}
-  m_j' = \lambda m_j+(1-\epsilon)(1-\lambda)(m_i+m_j),
-  \end{equation*}
-$$
-
 

-
-  which can be written as
-
-
 

-$$
-\begin{equation*}
-  m_i'=m_i+\delta m
-  \end{equation*}
-$$
-
 

-
-  and
-
-
 

-$$
-\begin{equation*}
-  m_j'=m_j-\delta m,
-  \end{equation*}
-$$
-
 

-
-  with
-
-
 

-$$
-\begin{equation*}
-  \delta m=(1-\lambda)(\epsilon m_j-(1-\epsilon)m_i),
-  \end{equation*}
-$$
-
 

-
-  showing how money is conserved during a transaction.
-  Select values of \( \lambda =0.25,0.5 \) and \( \lambda=0.9 \) and try to extract the corresponding
-  equilibrium distributions and compare these with the Gibbs distribution. We will use this model to 
-extract a parametrization of the above curves, see for example Patriarca and collaborators.
-
-
Particle in one dimension and velocity distribution 

+
+
"""                                                                                                                         
+This is taken from the data file of the mass 2016 evaluation.                                                               
+All files are 3436 lines long with 124 character per line.                                                                  
+       Headers are 39 lines long.                                                                                           
+   col 1     :  Fortran character control: 1 = page feed  0 = line feed                                                     
+   format    :  a1,i3,i5,i5,i5,1x,a3,a4,1x,f13.5,f11.5,f11.3,f9.3,1x,a2,f11.3,f9.3,1x,i3,1x,f12.5,f11.5                     
+   These formats are reflected in the pandas widths variable below, see the statement                                       
+   widths=(1,3,5,5,5,1,3,4,1,13,11,11,9,1,2,11,9,1,3,1,12,11,1),                                                            
+   Pandas has also a variable header, with length 39 in this case.                                                          
+"""
+

+

+The data we are interested in are in columns 2, 3, 4 and 11, giving us
+the number of neutrons, protons, mass numbers and binding energies,
+respectively. We add also for the sake of completeness the element name. The data are in fixed-width formatted lines and we will
+covert them into the pandas DataFrame structure.
 
 

 
 
-
# Program to test the Metropolis algorithm with one particle at given temp in one dimension
-import numpy as np
-import matplotlib.mlab as mlab
-import matplotlib.pyplot as plt
-import random
-from math import sqrt, exp, log
-# initialize the rng with a seed
-random.seed()
-# Hard coding of input parameters
-MCcycles = 100000
-Temperature = 2.0
-beta = 1./Temperature
-InitialVelocity = -2.0
-CurrentVelocity = InitialVelocity
-Energy = 0.5*InitialVelocity*InitialVelocity
-VelocityRange = 10*sqrt(Temperature)
-VelocityStep = 2*VelocityRange/10.
-AverageEnergy = Energy
-AverageEnergy2 = Energy*Energy
-VelocityValues = np.zeros(MCcycles)
-# The Monte Carlo sampling with Metropolis starts here
-for i in range (1, MCcycles, 1):
-    TrialVelocity = CurrentVelocity + (2.0*random.random() - 1.0)*VelocityStep
-    EnergyChange = 0.5*(TrialVelocity*TrialVelocity -CurrentVelocity*CurrentVelocity);
-    if random.random() <= exp(-beta*EnergyChange):
-        CurrentVelocity = TrialVelocity
-        Energy += EnergyChange
-        VelocityValues[i] = CurrentVelocity
-    AverageEnergy += Energy
-    AverageEnergy2 += Energy*Energy
-#Final averages
-AverageEnergy = AverageEnergy/MCcycles
-AverageEnergy2 = AverageEnergy2/MCcycles
-Variance = AverageEnergy2 - AverageEnergy*AverageEnergy
-print(AverageEnergy, Variance)
-n, bins, patches = plt.hist(VelocityValues, 400, facecolor='green')
+
# Read the experimental data with Pandas
+Masses = pd.read_fwf(infile, usecols=(2,3,4,6,11),
+              names=('N', 'Z', 'A', 'Element', 'Ebinding'),
+              widths=(1,3,5,5,5,1,3,4,1,13,11,11,9,1,2,11,9,1,3,1,12,11,1),
+              header=39,
+              index_col=False)
 
-plt.xlabel('$v$')
-plt.ylabel('Velocity distribution P(v)')
-plt.title(r'Velocity histogram at $k_BT=2$')
-plt.axis([-5, 5, 0, 600])
-plt.grid(True)
+# Extrapolated values are indicated by '#' in place of the decimal place, so
+# the Ebinding column won't be numeric. Coerce to float and drop these entries.
+Masses['Ebinding'] = pd.to_numeric(Masses['Ebinding'], errors='coerce')
+Masses = Masses.dropna()
+# Convert from keV to MeV.
+Masses['Ebinding'] /= 1000
+
+# Group the DataFrame by nucleon number, A.
+Masses = Masses.groupby('A')
+# Find the rows of the grouped DataFrame with the maximum binding energy.
+Masses = Masses.apply(lambda t: t[t.Ebinding==t.Ebinding.max()])
+

+

+We have now read in the data, grouped them according to the variables we are interested in. 
+We see how easy it is to reorganize the data using pandas. If we
+were to do these operations in C/C++ or Fortran, we would have had to
+write various functions/subroutines which perform the above
+reorganizations for us.  Having reorganized the data, we can now start
+to make some simple fits using both the functionalities in numpy and
+Scikit-Learn afterwards.
+
+

+Now we define five variables which contain
+the number of nucleons \( A \), the number of protons \( Z \) and the number of neutrons \( N \), the element name and finally the energies themselves.
+

+
+
+
A = Masses['A']
+Z = Masses['Z']
+N = Masses['N']
+Element = Masses['Element']
+Energies = Masses['Ebinding']
+print(Masses)
+

+

+The next step, and we will define this mathematically later, is to set up the so-called design matrix. We will throughout call this matrix \( \boldsymbol{X} \).
+It has dimensionality \( p\times n \), where \( n \) is the number of data points and \( p \) are the so-called predictors. In our case here they are given by the number of polynomials in \( A \) we wish to include in the fit. 
+

+
+
+
# Now we set up the design matrix X
+X = np.zeros((len(A),5))
+X[:,0] = 1
+X[:,1] = A
+X[:,2] = A**(2.0/3.0)
+X[:,3] = A**(-1.0/3.0)
+X[:,4] = A**(-1.0)
+

+

+With scikitlearn we are now ready to use linear regression and fit our data.
+

+
+
+
clf = skl.LinearRegression().fit(X, Energies)
+fity = clf.predict(X)
+

+

+Pretty simple!  
+Now we can print measures of how our fit is doing, the coefficients from the fits and plot the final fit together with our data.
+

+
+
+
# The mean squared error                               
+print("Mean squared error: %.2f" % mean_squared_error(Energies, fity))
+# Explained variance score: 1 is perfect prediction                                 
+print('Variance score: %.2f' % r2_score(Energies, fity))
+# Mean absolute error                                                           
+print('Mean absolute error: %.2f' % mean_absolute_error(Energies, fity))
+print(clf.coef_, clf.intercept_)
+
+Masses['Eapprox']  = fity
+# Generate a plot comparing the experimental with the fitted values values.
+fig, ax = plt.subplots()
+ax.set_xlabel(r'$A = N + Z$')
+ax.set_ylabel(r'$E_\mathrm{bind}\,/\mathrm{MeV}$')
+ax.plot(Masses['A'], Masses['Ebinding'], alpha=0.7, lw=2,
+            label='Ame2016')
+ax.plot(Masses['A'], Masses['Eapprox'], alpha=0.7, lw=2, c='m',
+            label='Fit')
+ax.legend()
+save_fig("Masses2016")
 plt.show()
 

 
-
Random walk model 

+
Seeing the wood for the trees 

+
+

+As a teaser, let us now see how we can do this with decision trees using scikit-learn. Later we will switch to so-called random forests!
 
 

 
 
-
import numpy as np
-import matplotlib.pyplot as plt
-from sklearn.preprocessing import PolynomialFeatures
-from sklearn.linear_model import LinearRegression
-
-steps=250
-
-distance=0
-x=0
-distance_list=[]
-steps_list=[]
-while x<steps:
-    distance+=np.random.randint(-1,2)
-    distance_list.append(distance)
-    x+=1
-    steps_list.append(x)
-plt.plot(steps_list,distance_list, color='green', label="Random Walk Data")
-
-steps_list=np.asarray(steps_list)
-distance_list=np.asarray(distance_list)
-
-X=steps_list[:,np.newaxis]
-
-#Polynomial fits
-
-#Degree 2
-poly_features=PolynomialFeatures(degree=2, include_bias=False)
-X_poly=poly_features.fit_transform(X)
-
-lin_reg=LinearRegression()
-poly_fit=lin_reg.fit(X_poly,distance_list)
-b=lin_reg.coef_
-c=lin_reg.intercept_
-print ("2nd degree coefficients:")
-print ("zero power: ",c)
-print ("first power: ", b[0])
-print ("second power: ",b[1])
-
-z = np.arange(0, steps, .01)
-z_mod=b[1]*z**2+b[0]*z+c
-
-fit_mod=b[1]*X**2+b[0]*X+c
-plt.plot(z, z_mod, color='r', label="2nd Degree Fit")
-plt.title("Polynomial Regression")
-
-plt.xlabel("Steps")
-plt.ylabel("Distance")
-
-#Degree 10
-poly_features10=PolynomialFeatures(degree=10, include_bias=False)
-X_poly10=poly_features10.fit_transform(X)
-
-poly_fit10=lin_reg.fit(X_poly10,distance_list)
-
-y_plot=poly_fit10.predict(X_poly10)
-plt.plot(X, y_plot, color='black', label="10th Degree Fit")
-
-plt.legend()
-plt.show()
-
-
-#Decision Tree Regression
+
#Decision Tree Regression
 from sklearn.tree import DecisionTreeRegressor
-regr_1=DecisionTreeRegressor(max_depth=2)
-regr_2=DecisionTreeRegressor(max_depth=5)
-regr_3=DecisionTreeRegressor(max_depth=7)
-regr_1.fit(X, distance_list)
-regr_2.fit(X, distance_list)
-regr_3.fit(X, distance_list)
+regr_1=DecisionTreeRegressor(max_depth=5)
+regr_2=DecisionTreeRegressor(max_depth=7)
+regr_3=DecisionTreeRegressor(max_depth=9)
+regr_1.fit(X, Energies)
+regr_2.fit(X, Energies)
+regr_3.fit(X, Energies)
 
-X_test = np.arange(0.0, steps, 0.01)[:, np.newaxis]
-y_1 = regr_1.predict(X_test)
-y_2 = regr_2.predict(X_test)
-y_3=regr_3.predict(X_test)
 
+y_1 = regr_1.predict(X)
+y_2 = regr_2.predict(X)
+y_3=regr_3.predict(X)
+Masses['Eapprox'] = y_3
 # Plot the results
 plt.figure()
-plt.scatter(X, distance_list, s=2.5, c="black", label="data")
-plt.plot(X_test, y_1, color="red",
-         label="max_depth=2", linewidth=2)
-plt.plot(X_test, y_2, color="green", label="max_depth=5", linewidth=2)
-plt.plot(X_test, y_3, color="m", label="max_depth=7", linewidth=2)
+plt.plot(A, Energies, color="blue", label="Data", linewidth=2)
+plt.plot(A, y_1, color="red", label="max_depth=5", linewidth=2)
+plt.plot(A, y_2, color="green", label="max_depth=7", linewidth=2)
+plt.plot(A, y_3, color="m", label="max_depth=9", linewidth=2)
 
-plt.xlabel("Data")
-plt.ylabel("Darget")
+plt.xlabel("$A$")
+plt.ylabel("$E$[MeV]")
 plt.title("Decision Tree Regression")
 plt.legend()
+save_fig("Masses2016Trees")
+plt.show()
+print(Masses)
+print(np.mean( (Energies-y_1)**2))
+

+
+
And what about using neural networks? 

+
+The seaborn package allows us to visualize data in an efficient way. Note that we use scikit-learn's multi-layer perceptron (or feed forward neural network) 
+functionality.
+

+
+
+
from sklearn.neural_network import MLPRegressor
+from sklearn.metrics import accuracy_score
+import seaborn as sns
+
+X_train = X
+Y_train = Energies
+n_hidden_neurons = 100
+epochs = 100
+# store models for later use
+eta_vals = np.logspace(-5, 1, 7)
+lmbd_vals = np.logspace(-5, 1, 7)
+# store the models for later use
+DNN_scikit = np.zeros((len(eta_vals), len(lmbd_vals)), dtype=object)
+train_accuracy = np.zeros((len(eta_vals), len(lmbd_vals)))
+sns.set()
+for i, eta in enumerate(eta_vals):
+    for j, lmbd in enumerate(lmbd_vals):
+        dnn = MLPRegressor(hidden_layer_sizes=(n_hidden_neurons), activation='logistic',
+                            alpha=lmbd, learning_rate_init=eta, max_iter=epochs)
+        dnn.fit(X_train, Y_train)
+        DNN_scikit[i][j] = dnn
+        train_accuracy[i][j] = dnn.score(X_train, Y_train)
+
+fig, ax = plt.subplots(figsize = (10, 10))
+sns.heatmap(train_accuracy, annot=True, ax=ax, cmap="viridis")
+ax.set_title("Training Accuracy")
+ax.set_ylabel("$\eta$")
+ax.set_xlabel("$\lambda$")
 plt.show()
 

+
+
A first summary 

+
+

+The aim behind these introductory words was to present to you various
+Python libraries and their functionalities, in particular libraries like
+numpy, pandas, xarray and matplotlib and other that make our life much easier
+in handling various data sets and visualizing data.
+
+

+Furthermore,
+Scikit-Learn allows us with few lines of code to implement popular
+Machine Learning algorithms for supervised learning. Later we will meet Tensorflow, a powerful library for deep learning. 
+Now it is time to dive more into the details of various methods. We will start with linear regression and try to take a deeper look at what it entails.
+
 

 
 
diff --git a/doc/pub/How2ReadData/html/How2ReadData-solarized.html b/doc/pub/How2ReadData/html/How2ReadData-solarized.html
index 3854aebc7..38f4ae343 100644
--- a/doc/pub/How2ReadData/html/How2ReadData-solarized.html
+++ b/doc/pub/How2ReadData/html/How2ReadData-solarized.html
@@ -62,25 +62,40 @@ div { text-align: justify; text-justify: inter-word; }
 
 
 
@@ -122,7 +137,7 @@ MathJax.Hub.Config({
 
[2] Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University

 

 

-
Jan 14, 2019
 
+
Aug 13, 2019
 
 

 
 
Introduction 

@@ -141,45 +156,147 @@ and use these to introduce statistical data analysis
 concepts and machine learning algorithms before we delve into the
 algorithms themselves. The examples we will use in the beginning, start with simple
 polynomials with random noise added. We will use the Python
-software package Scikit-learn and
+software package Scikit-Learn and
 introduce various machine learning algorithms to make fits of
 the data and predictions. We move thereafter to more interesting
-cases such as the simulation of financial transactions or disease
-models. These are examples where we can easily set up the data and
+cases such as nuclear binding energies.
+These are examples where we can easily set up the data and
 then use machine learning algorithms included in for example
-scikit-learn.
+Scikit-Learn.
 
 

 These examples will serve us the purpose of getting
 started. Furthermore, they allow us to catch more than two birds with
 a stone. They will allow us to bring in some programming specific
-topics and tools as well as showing the power of various Python (and
-R) packages for machine learning and statistical data analysis. In the
-lectures on linear algebra we cover in more detail various programming
-features of languages like Python and C++ (and other), we will also
-look into more specific linear functions which are relevant for the
-various algorithms we will discuss. Here, we will mainly focus on two
-specific Python packages for Machine Learning, scikit-learn and
-tensorflow (see below for links etc).  Moreover, the examples we
+topics and tools as well as showing the power of various Python 
+packages for machine learning and statistical data analysis.
+
+

+Here, we will mainly focus on two
+specific Python packages for Machine Learning, Scikit-Learn and
+Tensorflow (see below for links etc).  Moreover, the examples we
 introduce will serve as inputs to many of our discussions later, as
 well as allowing you to set up models and produce your own data and
 get started with programming.
 
-
Software and needed installations 

+
What is Machine Learning?  

+
+

+Statistics, data science and machine learning form important fields of
+research in modern science.  They describe how to learn and make
+predictions from data, as well as allowing us to extract important
+correlations about physical process and the underlying laws of motion
+in large data sets. The latter, big data sets, appear frequently in
+essentially all disciplines, from the traditional Science, Technology,
+Mathematics and Engineering fields to Life Science, Law, education
+research, the Humanities and the Social Sciences.
+
+

+It has become more
+and more common to see research projects on big data in for example
+the Social Sciences where extracting patterns from complicated survey
+data is one of many research directions.  Having a solid grasp of data
+analysis and machine learning is thus becoming central to scientific
+computing in many fields, and competences and skills within the fields
+of machine learning and scientific computing are nowadays strongly
+requested by many potential employers. The latter cannot be
+overstated, familiarity with machine learning has almost become a
+prerequisite for many of the most exciting employment opportunities,
+whether they are in bioinformatics, life science, physics or finance,
+in the private or the public sector. This author has had several
+students or met students who have been hired recently based on their
+skills and competences in scientific computing and data science, often
+with marginal knowledge of machine learning.
+
+

+Machine learning is a subfield of computer science, and is closely
+related to computational statistics.  It evolved from the study of
+pattern recognition in artificial intelligence (AI) research, and has
+made contributions to AI tasks like computer vision, natural language
+processing and speech recognition. Many of the methods we will study are also 
+strongly rooted in basic mathematics and physics research.
+
+

+Ideally, machine learning represents the science of giving computers
+the ability to learn without being explicitly programmed.  The idea is
+that there exist generic algorithms which can be used to find patterns
+in a broad class of data sets without having to write code
+specifically for each problem. The algorithm will build its own logic
+based on the data.  You should however always keep in mind that
+machines and algorithms are to a large extent developed by humans. The
+insights and knowledge we have about a specific system, play a central
+role when we develop a specific machine learning algorithm.
+
+

+Machine learning is an extremely rich field, in spite of its young
+age. The increases we have seen during the last three decades in
+computational capabilities have been followed by developments of
+methods and techniques for analyzing and handling large date sets,
+relying heavily on statistics, computer science and mathematics.  The
+field is rather new and developing rapidly. Popular software packages
+written in Python for machine learning like
+Scikit-learn,
+Tensorflow,
+PyTorch and Keras, all
+freely available at their respective GitHub sites, encompass
+communities of developers in the thousands or more. And the number of
+code developers and contributors keeps increasing. Not all the
+algorithms and methods can be given a rigorous mathematical
+justification, opening up thereby large rooms for experimenting and
+trial and error and thereby exciting new developments.  However, a
+solid command of linear algebra, multivariate theory, probability
+theory, statistical data analysis, understanding errors and Monte
+Carlo methods are central elements in a proper understanding of many
+of algorithms and methods we will discuss.
+
+
Types of Machine Learning 

+
+

+The approaches to machine learning are many, but are often split into
+two main categories.  In supervised learning we know the answer to a
+problem, and let the computer deduce the logic behind it. On the other
+hand, unsupervised learning is a method for finding patterns and
+relationship in data sets without any prior knowledge of the system.
+Some authours also operate with a third category, namely
+reinforcement learning. This is a paradigm of learning inspired by
+behavioral psychology, where learning is achieved by trial-and-error,
+solely from rewards and punishment.
+
+

+Another way to categorize machine learning tasks is to consider the
+desired output of a system.  Some of the most common tasks are:
+
+
    
+  
  •  Classification: Outputs are divided into two or more classes. The goal is to   produce a model that assigns inputs into one of these classes. An example is to identify  digits based on pictures of hand-written ones. Classification is typically supervised learning.
    • 
+  
  •  Regression: Finding a functional relationship between an input data set and a reference data set.   The goal is to construct a function that maps input data to continuous output values.
    • 
+  
  •  Clustering: Data are divided into groups with certain common traits, without knowing the different groups beforehand.  It is thus a form of unsupervised learning.
    • 
+

+
+The methods we cover have three main topics in common, irrespective of
+whether we deal with supervised or unsupervised learning. The first
+ingredient is normally our data set (which can be subdivided into
+training and test data), the second item is a model which is normally a
+function of some parameters.  The model reflects our knowledge of the system (or lack thereof). As an example, if we know that our data show a behavior similar to what would be predicted by a polynomial, fitting our data to a polynomial of some degree would then determin our model.
+
+

+The last ingredient is a so-called cost
+function which allows us to present an estimate on how good our model
+is in reproducing the data it is supposed to train.  
+At the heart of basically all ML algorithms there are so-called minimization algorithms, often we end up with various variants of gradient methods.
+
+
Software and needed installations 

 
 

 We will make extensive use of Python as programming language and its
 myriad of available libraries.  You will find
-IPython/Jupyter notebooks invaluable in your work.  You can run R
+Jupyter notebooks invaluable in your work.  You can run R
 codes in the Jupyter/IPython notebooks, with the immediate benefit of
 visualizing your data. You can also use compiled languages like C++,
-Rust, Fortran etc if you prefer. The focus in these lectures will be
-on Python, but we will provide many code examples for those of you who
-prefer R or compiled languages. You can integrate C++ codes and R in for example
-a Jupyter notebook.
+Rust, Julia, Fortran etc if you prefer. The focus in these lectures will be
+on Python.
 
 

-If you have Python installed (we recommend Python3) and you feel
+If you have Python installed (we strongly recommend Python3) and you feel
 pretty familiar with installing different packages, we recommend that
 you install the following Python packages via pip as 
 
@@ -207,7 +324,7 @@ you can use pip as well and simply install Python as
 
 etc etc.
 
-
Python installers 

+
Python installers 

 
 

 If you don't want to perform these operations separately and venture
@@ -234,28 +351,46 @@ distribution for scientific and analytic computing distribution and
 analysis environment, available for free and under a commercial
 license.
 
-
Useful Python packages 

-Here we list several useful Python packages.
+

+Furthermore, Google's Colab is a free Jupyter notebook environment that requires 
+no setup and runs entirely in the cloud. Try it out!
 
-
Installing R, C++, cython or Julia 

+
Useful Python libraries 

+Here we list several useful Python libraries we strongly recommend (if you use anaconda many of these are already there)
+
+
    
+
  •  NumPy is a highly popular library for large, multi-dimensional arrays and matrices, along with a large collection of high-level mathematical functions to operate on these arrays
    • 
+
  •  The pandas library provides high-performance, easy-to-use data structures and data analysis tools
    •  
+
  •  Xarray is a Python package that makes working with labelled multi-dimensional arrays simple, efficient, and fun!
    • 
+
  •  Scipy (pronounced “Sigh Pie”) is a Python-based ecosystem of open-source software for mathematics, science, and engineering.
    •  
+
  •  Matplotlib is a Python 2D plotting library which produces publication quality figures in a variety of hardcopy formats and interactive environments across platforms.
    • 
+
  •  Autograd can automatically differentiate native Python and Numpy code. It can handle a large subset of Python's features, including loops, ifs, recursion and closures, and it can even take derivatives of derivatives of derivatives
    • 
+
  •  SymPy is a Python library for symbolic mathematics.
    •  
+
  •  scikit-learn has simple and efficient tools for machine learning, data mining and data analysis
    • 
+
  •  TensorFlow is a Python library for fast numerical computing created and released by Google
    • 
+
  •  Keras is a high-level neural networks API, written in Python and capable of running on top of TensorFlow, CNTK, or Theano
    • 
+
  •  And many more such as pytorch,  Theano etc
    •  
+

+
+
Installing R, C++, cython or Julia 

 
 

-You will also find it convenient to utilize R. Although we will mainly
-use Python during lectures and in various projects and exercises, we
-provide a full R set of codes for the same examples. Those of you
-already familiar with R should feel free to continue using R, keeping
+You will also find it convenient to utilize R. We will mainly
+use Python during lectures and in various projects and exercises.
+Those of you
+already familiar with R should feel free to continue using R, keeping
 however an eye on the parallel Python set ups. Similarly, if you are a
-Python afecionado, feel free to explore R as well.  Jupyter/Ipython
+Python afecionado, feel free to explore R as well.  Jupyter/Ipython
 notebook allows you to run R codes interactively in your
 browser. The software library R is tuned to statistically analysis
 and allows for an easy usage of the tools we will discuss in these
-texts.
+lectures.
 
 

 To install R with Jupyter notebook 
 follow the link here
 
-
Installing R, C++, cython, Numba etc 

+
Installing R, C++, cython, Numba etc 

 
 

 For the C++ aficionados, Jupyter/IPython notebook allows you also to
@@ -266,13 +401,13 @@ languages.
 
 

 To add more entropy, cython can also be used when running your
-notebooks. It means that Python with the Jupyter/IPython notebook
+notebooks. It means that Python with the jupyter notebook
 setup allows you to integrate widely popular softwares and tools for
 scientific computing. Similarly, the 
 Numba Python package delivers increased performance
 capabilities with minimal rewrites of your codes.  With its
 versatility, including symbolic operations, Python offers a unique
-computational environment. Your Jupyter/IPython notebook can easily be
+computational environment. Your jupyter notebook can easily be
 converted into a nicely rendered PDF file or a Latex file for
 further processing. For example, convert to latex as
 
@@ -287,13 +422,513 @@ And to add more versatility, the Python package doconce you can convert a standard ascii text file into various HTML 
-formats, ipython notebooks, latex files, pdf files etc with minimal edits.
+formats, ipython notebooks, latex files, pdf files etc with minimal edits. These lectures were generated using doconce.
 
-
Simple linear regression model using scikit-learn 

+
Numpy examples and Important Matrix and vector handling packages 

 
 

-We start with perhaps our simplest possible example, using scikit-learn to perform linear regression analysis on a data set produced by us. 
-What follows is a simple Python code where we have defined  function \( y \) in terms of the variable \( x \). Both are defined as vectors of dimension \( 1\times 100 \). The entries to the vector \( \hat{x} \)  are given by random numbers generated with a uniform distribution with entries \( x_i \in [0,1] \) (more about probability distribution functions later). These values are then used to define a function \( y(x) \) (tabulated again as a vector) with a linear dependence on \( x \) plus a random noise added via the normal distribution.
+There are several central software packages for linear algebra and eigenvalue problems. Several of the more
+popular ones have been wrapped into ofter software packages like those from the widely used text Numerical Recipes. The original source codes in many of the available packages are often taken from the widely used
+software package LAPACK, which follows two other popular packages
+developed in the 1970s, namely EISPACK and LINPACK.  We describe them shortly here.
+
+
    
+  
  •  LINPACK: package for linear equations and least square problems.
    • 
+  
  •  LAPACK:package for solving symmetric, unsymmetric and generalized eigenvalue problems. From LAPACK's website http://www.netlib.org it is possible to download for free all source codes from this library. Both C/C++ and Fortran versions are available.
    • 
+  
  •  BLAS (I, II and III): (Basic Linear Algebra Subprograms) are routines that provide standard building blocks for performing basic vector and matrix operations. Blas I is vector operations, II vector-matrix operations and III matrix-matrix operations. Highly parallelized and efficient codes, all available for download from http://www.netlib.org.
    • 
+

+
+
Basic Matrix Features 

+
+

+

+Matrix properties reminder.
+

+$$
+ \mathbf{A} =
+      \begin{bmatrix} a_{11} & a_{12} & a_{13} & a_{14} \\
+                                 a_{21} & a_{22} & a_{23} & a_{24} \\
+                                   a_{31} & a_{32} & a_{33} & a_{34} \\
+                                  a_{41} & a_{42} & a_{43} & a_{44}
+             \end{bmatrix}\qquad
+\mathbf{I} =
+      \begin{bmatrix} 1 & 0 & 0 & 0 \\
+                                 0 & 1 & 0 & 0 \\
+                                 0 & 0 & 1 & 0 \\
+                                 0 & 0 & 0 & 1
+             \end{bmatrix}
+$$
+

+
+
+
Basic Matrix Features 

+

+
+

+
+

+The inverse of a matrix is defined by
+
+$$
+\mathbf{A}^{-1} \cdot \mathbf{A} = I
+$$
+

+
+
+
Basic Matrix Features 

+
+

+

+Matrix Properties Reminder.
+

+
+

+
+
+
+
+
+
+
+
+
+
+
+
                Relations                        Name                                    matrix elements                               
   \( A = A^{T} \)                                   symmetric              \( a_{ij} = a_{ji} \)                                                           
   \( A = \left (A^{T} \right )^{-1} \)              real orthogonal        \( \sum_k a_{ik} a_{jk} = \sum_k a_{ki} a_{kj} = \delta_{ij} \)                 
   \( A = A^{ * } \)                                 real matrix            \( a_{ij} = a_{ij}^{ * } \)                                                     
   \( A = A^{\dagger} \)                             hermitian              \( a_{ij} = a_{ji}^{ * } \)                                                     
   \( A = \left (A^{\dagger} \right )^{-1} \)        unitary                \( \sum_k a_{ik} a_{jk}^{ * } = \sum_k a_{ki}^{ * } a_{kj} = \delta_{ij} \)     

+
+

+
+
+
Some famous Matrices 

+
+
    
+  
  •  Diagonal if \( a_{ij}=0 \) for \( i\ne j \)
    • 
+  
  •  Upper triangular if \( a_{ij}=0 \) for \( i > j \)
    • 
+  
  •  Lower triangular if \( a_{ij}=0 \) for \( i < j \)
    • 
+  
  •  Upper Hessenberg if \( a_{ij}=0 \) for \( i > j+1 \)
    • 
+  
  •  Lower Hessenberg if \( a_{ij}=0 \) for \( i < j+1 \)
    • 
+  
  •  Tridiagonal if \( a_{ij}=0 \) for \( |i -j| > 1 \)
    • 
+  
  •  Lower banded with bandwidth \( p \): \( a_{ij}=0 \) for \( i > j+p \)
    • 
+  
  •  Upper banded with bandwidth \( p \): \( a_{ij}=0 \) for \( i < j+p \)
    • 
+  
  •  Banded, block upper triangular, block lower triangular....
    • 
+

+
+
Basic Matrix Features 

+
+

+

+Some Equivalent Statements.
+

+For an \( N\times N \) matrix  \( \mathbf{A} \) the following properties are all equivalent
+
+
    
+  
  •  If the inverse of \( \mathbf{A} \) exists, \( \mathbf{A} \) is nonsingular.
    • 
+  
  •  The equation \( \mathbf{Ax}=0 \) implies \( \mathbf{x}=0 \).
    • 
+  
  •  The rows of \( \mathbf{A} \) form a basis of \( R^N \).
    • 
+  
  •  The columns of \( \mathbf{A} \) form a basis of \( R^N \).
    • 
+  
  •  \( \mathbf{A} \) is a product of elementary matrices.
    • 
+  
  •  \( 0 \) is not eigenvalue of \( \mathbf{A} \).
    • 
+

+

+
+
+
Numpy and arrays 

+Numpy provides an easy way to handle arrays in Python. The standard way to import this library is as
+
+

+
+
+
import numpy as np
+

+

+Here follows a simple example where we set up an array of ten elements, all determined by random numbers drawn according to the normal distribution,
+

+
+
+
n = 10
+x = np.random.normal(size=n)
+print(x)
+

+

+We defined a vector \( x \) with \( n=10 \) elements with its values given by the Normal distribution \( N(0,1) \).
+Another alternative is to declare a vector as follows
+

+
+
+
import numpy as np
+x = np.array([1, 2, 3])
+print(x)
+

+

+Here we have defined a vector with three elements, with \( x_0=1 \), \( x_1=2 \) and \( x_2=3 \). Note that both Python and C++
+start numbering array elements from \( 0 \) and on. This means that a vector with \( n \) elements has a sequence of entities \( x_0, x_1, x_2, \dots, x_{n-1} \). We could also let (recommended) Numpy to compute the logarithms of a specific array as
+

+
+
+
import numpy as np
+x = np.log(np.array([4, 7, 8]))
+print(x)
+

+

+In the last example we used Numpy's unary function \( np.log \). This function is
+highly tuned to compute array elements since the code is vectorized
+and does not require looping. We normaly recommend that you use the
+Numpy intrinsic functions instead of the corresponding log function
+from Python's math module. The looping is done explicitely by the
+np.log function. The alternative, and slower way to compute the
+logarithms of a vector would be to write
+
+

+
+
+
import numpy as np
+from math import log
+x = np.array([4, 7, 8])
+for i in range(0, len(x)):
+    x[i] = log(x[i])
+print(x)
+

+

+We note that our code is much longer already and we need to import the log function from the math module. 
+The attentive reader will also notice that the output is \( [1, 1, 2] \). Python interprets automagically our numbers as integers (like the automatic keyword in C++). To change this we could define our array elements to be double precision numbers as
+

+
+
+
import numpy as np
+x = np.log(np.array([4, 7, 8], dtype = np.float64))
+print(x)
+

+

+or simply write them as double precision numbers (Python uses 64 bits as default for floating point type variables), that is
+

+
+
+
import numpy as np
+x = np.log(np.array([4.0, 7.0, 8.0])
+print(x)
+

+

+To check the number of bytes (remember that one byte contains eight bits for double precision variables), you can use simple use the itemsize functionality (the array \( x \) is actually an object which inherits the functionalities defined in Numpy) as 
+

+
+
+
import numpy as np
+x = np.log(np.array([4.0, 7.0, 8.0])
+print(x.itemsize)
+

+
+
Matrices in Python 

+
+

+Having defined vectors, we are now ready to try out matrices. We can
+define a \( 3 \times 3  \) real matrix \( \hat{A} \) as (recall that we user
+lowercase letters for vectors and uppercase letters for matrices)
+
+

+
+
+
import numpy as np
+A = np.log(np.array([ [4.0, 7.0, 8.0], [3.0, 10.0, 11.0], [4.0, 5.0, 7.0] ]))
+print(A)
+

+

+If we use the shape function we would get \( (3, 3) \) as output, that is verifying that our matrix is a \( 3\times 3 \) matrix. We can slice the matrix and print for example the first column (Python organized matrix elements in a row-major order, see below) as
+

+
+
+
import numpy as np
+A = np.log(np.array([ [4.0, 7.0, 8.0], [3.0, 10.0, 11.0], [4.0, 5.0, 7.0] ]))
+# print the first column, row-major order and elements start with 0
+print(A[:,0]) 
+

+

+We can continue this was by printing out other columns or rows. The example here prints out the second column
+

+
+
+
import numpy as np
+A = np.log(np.array([ [4.0, 7.0, 8.0], [3.0, 10.0, 11.0], [4.0, 5.0, 7.0] ]))
+# print the first column, row-major order and elements start with 0
+print(A[1,:]) 
+

+

+Numpy contains many other functionalities that allow us to slice, subdivide etc etc arrays. We strongly recommend that you look up the Numpy website for more details. Useful functions when defining a matrix are the np.zeros function which declares a matrix of a given dimension and sets all elements to zero
+

+
+
+
import numpy as np
+n = 10
+# define a matrix of dimension 10 x 10 and set all elements to zero
+A = np.zeros( (n, n) )
+print(A) 
+

+

+or initializing all elements to 
+

+
+
+
import numpy as np
+n = 10
+# define a matrix of dimension 10 x 10 and set all elements to one
+A = np.ones( (n, n) )
+print(A) 
+

+

+or as unitarily distributed random numbers (see the material on random number generators in the statistics part)
+

+
+
+
import numpy as np
+n = 10
+# define a matrix of dimension 10 x 10 and set all elements to random numbers with x \in [0, 1]
+A = np.random.rand(n, n)
+print(A) 
+

+

+As we will see throughout these lectures, there are several extremely useful functionalities in Numpy.
+As an example, consider the discussion of the covariance matrix. Suppose we have defined three vectors
+\( \hat{x}, \hat{y}, \hat{z} \) with \( n \) elements each. The covariance matrix is defined as 
+$$
+\hat{\Sigma} = \begin{bmatrix} \sigma_{xx} & \sigma_{xy} & \sigma_{xz} \\
+                              \sigma_{yx} & \sigma_{yy} & \sigma_{yz} \\
+                              \sigma_{zx} & \sigma_{zy} & \sigma_{zz} 
+             \end{bmatrix},
+$$
+
+where for example
+$$
+\sigma_{xy} =\frac{1}{n} \sum_{i=0}^{n-1}(x_i- \overline{x})(y_i- \overline{y}).
+$$
+
+The Numpy function np.cov calculates the covariance elements using the factor \( 1/(n-1) \) instead of \( 1/n \) since it assumes we do not have the exact mean values. 
+The following simple function uses the np.vstack function which takes each vector of dimension \( 1\times n \) and produces a \( 3\times n \) matrix \( \hat{W} \)
+$$
+\hat{W} = \begin{bmatrix} x_0 & y_0 & z_0 \\
+                          x_1 & y_1 & z_1 \\
+                          x_2 & y_2 & z_2 \\
+                          \dots & \dots & \dots \\
+                          x_{n-2} & y_{n-2} & z_{n-2} \\
+                          x_{n-1} & y_{n-1} & z_{n-1}
+             \end{bmatrix},
+$$
+
+

+which in turn is converted into into the \( 3\times 3 \) covariance matrix
+\( \hat{\Sigma} \) via the Numpy function np.cov(). We note that we can also calculate
+the mean value of each set of samples \( \hat{x} \) etc using the Numpy
+function np.mean(x). We can also extract the eigenvalues of the
+covariance matrix through the np.linalg.eig() function.
+
+

+
+
+
# Importing various packages
+import numpy as np
+
+n = 100
+x = np.random.normal(size=n)
+print(np.mean(x))
+y = 4+3*x+np.random.normal(size=n)
+print(np.mean(y))
+z = x**3+np.random.normal(size=n)
+print(np.mean(z))
+W = np.vstack((x, y, z))
+Sigma = np.cov(W)
+print(Sigma)
+Eigvals, Eigvecs = np.linalg.eig(Sigma)
+print(Eigvals)
+

+

+
+
+
import numpy as np
+import matplotlib.pyplot as plt
+from scipy import sparse
+eye = np.eye(4)
+print(eye)
+sparse_mtx = sparse.csr_matrix(eye)
+print(sparse_mtx)
+x = np.linspace(-10,10,100)
+y = np.sin(x)
+plt.plot(x,y,marker='x')
+plt.show()
+

+
+
Meet the Pandas 

+
+

+




+
+

+Another useful Python package is
+pandas, which is an open source library
+providing high-performance, easy-to-use data structures and data
+analysis tools for Python. pandas stands for panel data, a term borrowed from econometrics and is an efficient library for data analysis with an emphasis on tabular data.
+pandas has two major classes, the DataFrame class with two-dimensional data objects and tabular data organized in columns and the class Series with a focus on one-dimensional data objects. Both classes allow you to index data easily as we will see in the examples below. 
+pandas allows you also to perform mathematical operations on the data, spanning from simple reshapings of vectors and matrices to statistical operations.
+
+

+The following simple example shows how we can, in an easy way make tables of our data. Here we define a data set which includes names, place of birth and date of birth, and displays the data in an easy to read way. We will see repeated use of pandas, in particular in connection with classification of data.
+
+

+
+
+
import pandas as pd
+from IPython.display import display
+data = {'First Name': ["Frodo", "Bilbo", "Aragorn II", "Samwise"],
+        'Last Name': ["Baggins", "Baggins","Elessar","Gamgee"],
+        'Place of birth': ["Shire", "Shire", "Eriador", "Shire"],
+        'Date of Birth T.A.': [2968, 2890, 2931, 2980]
+        }
+data_pandas = pd.DataFrame(data)
+display(data_pandas)
+

+

+In the above we have imported pandas with the shorthand pd, the latter has become the standard way we import pandas. We make then a list of various variables
+and reorganize the aboves lists into a DataFrame and then print out  a neat table with specific column labels as Name, place of birth and date of birth.
+Displaying these results, we see that the indices are given by the default numbers from zero to three.
+pandas is extremely flexible and we can easily change the above indices by defining a new type of indexing as
+

+
+
+
data_pandas = pd.DataFrame(data,index=['Frodo','Bilbo','Aragorn','Sam'])
+display(data_pandas)
+

+

+Thereafter we display the content of the row which begins with the index Aragorn
+

+
+
+
display(data_pandas.loc['Aragorn'])
+

+

+We can easily append data to this, for example
+

+
+
+
new_hobbit = {'First Name': ["Peregrin"],
+              'Last Name': ["Took"],
+              'Place of birth': ["Shire"],
+              'Date of Birth T.A.': [2990]
+              }
+data_pandas=data_pandas.append(pd.DataFrame(new_hobbit, index=['Pippin']))
+display(data_pandas)
+

+

+Here are other examples where we use the DataFrame functionality to handle arrays, now with more interesting features for us, namely numbers. We set up a matrix 
+of dimensionality \( 10\times 5 \) and compute the mean value and standard deviation of each column. Similarly, we can perform mathematial operations like squaring the matrix elements and many other operations. 
+

+
+
+
import numpy as np
+import pandas as pd
+from IPython.display import display
+np.random.seed(100)
+# setting up a 10 x 5 matrix
+rows = 10
+cols = 5
+a = np.random.randn(rows,cols)
+df = pd.DataFrame(a)
+display(df)
+print(df.mean())
+print(df.std())
+display(df**2)
+

+

+Thereafter we can select specific columns only and plot final results
+

+
+
+
df.columns = ['First', 'Second', 'Third', 'Fourth', 'Fifth']
+df.index = np.arange(10)
+
+display(df)
+print(df['Second'].mean() )
+print(df.info())
+print(df.describe())
+
+from pylab import plt, mpl
+plt.style.use('seaborn')
+mpl.rcParams['font.family'] = 'serif'
+
+df.cumsum().plot(lw=2.0, figsize=(10,6))
+plt.show()
+
+
+df.plot.bar(figsize=(10,6), rot=15)
+plt.show()
+

+

+We can produce a \( 4\times 4 \) matrix
+

+
+
+
b = np.arange(16).reshape((4,4))
+print(b)
+df1 = pd.DataFrame(b)
+print(df1)
+

+

+and many other operations.
+
+

+The Series class is another important class included in
+pandas. You can view it as a specialization of DataFrame but where
+we have just a single column of data. It shares many of the same features as _DataFrame. As with DataFrame,
+most operations are vectorized, achieving thereby a high performance when dealing with computations of arrays, in particular labeled arrays.
+As we will see below it leads also to a very concice code close to the mathematical operations we may be interested in.
+For multidimensional arrays, we recommend strongly xarray. xarray has much of the same flexibility as pandas, but allows for the extension to higher dimensions than two. We will see examples later of the usage of both pandas and xarray.
+
+
Reading Data and fitting  

+
+

+In order to study various Machine Learning algorithms, we need to
+access data. Acccessing data is an essential step in all machine
+learning algorithms. In particular, setting up the so-called design
+matrix (to be defined below) is often the first element we need in
+order to perform our calculations. To set up the design matrix means
+reading (and later, when the calculations are done, writing) data
+in various formats, The formats span from reading files from disk,
+loading data from databases and interacting with online sources
+like web application programming interfaces (APIs).
+
+

+In handling various input formats, as discussed above, we will mainly stay with pandas,
+a Python package which allows us, in a seamless and painless way, to
+deal with a multitude of formats, from standard csv (comma separated
+values) files, via excel, html to hdf5 formats.  With pandas
+and the DataFrame  and Series functionalities we are able to convert text data
+into the calculational formats we need for a specific algorithm. And our code is going to be 
+pretty close the basic mathematical expressions.
+
+

+Our first data set is going to be a classic from nuclear physics, namely all
+available data on binding energies. Don't be intimidated if you are not familiar with nuclear physics. It serves simply as an example here of a data set.
+
+

+We will show some of the
+strengths of packages like Scikit-Learn in fitting nuclear binding energies to
+specific functions using linear regression first. Then, as a teaser, we will show you how 
+you can easily implement other algorithms like decision trees and random forests and neural networks.
+
+

+But before we really start with nuclear physics data, let's just look at some simpler polynomial fitting cases, such as,
+(don't be offended) fitting straight lines!
+
+
Simple linear regression model using scikit-learn 

+
+

+We start with perhaps our simplest possible example, using Scikit-Learn to perform linear regression analysis on a data set produced by us.
+
+

+What follows is a simple Python code where we have defined a function
+\( y \) in terms of the variable \( x \). Both are defined as vectors with  \( 100 \) entries. 
+The numbers in the vector \( \hat{x} \) are given
+by random numbers generated with a uniform distribution with entries
+\( x_i \in [0,1] \) (more about probability distribution functions
+later). These values are then used to define a function \( y(x) \)
+(tabulated again as a vector) with a linear dependence on \( x \) plus a
+random noise added via the normal distribution.
 
 

 The Numpy functions are imported used the import numpy as np
@@ -312,7 +947,7 @@ $$
 
 

 where \( N(0,1) \) represents random numbers generated by the normal
-distribution.  From scikit-learn we import then the
+distribution.  From Scikit-Learn we import then the
 LinearRegression functionality and make a prediction \( \tilde{y} =
 \alpha + \beta x \) using the function fit(x,y). We call the set of
 data \( (\hat{x},\hat{y}) \) for our training data. The Python package
@@ -370,7 +1005,7 @@ $$
 
 

 where \( x \) is defined as before.  Does the fit look better? Indeed, by
-reducing the role of the normal distribution we see immediately that
+reducing the role of the noise given by the normal distribution we see immediately that
 our linear prediction seemingly reproduces better the training
 set. However, this testing 'by the eye' is obviouly not satisfactory in the
 long run. Here we have only defined the training data and our model, and 
@@ -381,7 +1016,7 @@ We need more rigorous criteria in defining whether we have succeeded or
 not in modeling our training data.  You will be surprised to see that
 many scientists seldomly venture beyond this 'by the eye' approach. A
 standard approach for the cost function is the so-called \( \chi^2 \)
-function 
+function (a variant of the mean-squared error (MSE))
 
 $$ \chi^2 = \frac{1}{n}
 \sum_{i=0}^{n-1}\frac{(y_i-\tilde{y}_i)^2}{\sigma_i^2}, 
@@ -411,13 +1046,13 @@ the  \( \chi^2 \) function becomes smaller.
 
 

 There are many ways to define the cost function. A simpler approach is to look at the relative difference between the training data and the predicted data, that is we define 
-the relative error as
+the relative error (why would we prefer the MSE instead of the relative error?) as
 
 $$
 \epsilon_{\mathrm{relative}}= \frac{\vert \hat{y} -\hat{\tilde{y}}\vert}{\vert \hat{y}\vert}.
 $$
 
-We can modify easily the above Python code and plot the relative instead
+We can modify easily the above Python code and plot the relative error instead
 

 
 
@@ -445,14 +1080,14 @@ different training data sets and study (graphically) the value of the
 relative error.
 
 

-As mentioned above, scikit-learn has an impressive functionality.
+As mentioned above, Scikit-Learn has an impressive functionality.
 We can for example extract the values of \( \alpha \) and \( \beta \) and
 their error estimates, or the variance and standard deviation and many
 other properties from the statistical data analysis.
 
 

 Here we show an
-example of the functionality of scikit-learn.
+example of the functionality of Scikit-Learn.
 

 
 
@@ -515,8 +1150,8 @@ $$
 \bar{y} =  \frac{1}{n} \sum_{i=0}^{n - 1} y_i.
 $$
 
-Another quantity will meet again in our discussions of regression analysis is 
- mean absolute error (MAE), a risk metric corresponding to the expected value of the absolute error loss or what we call the \( l1 \)-norm loss. In our discussion above we presented the relative error.
+Another quantity taht we will meet again in our discussions of regression analysis is 
+ the mean absolute error (MAE), a risk metric corresponding to the expected value of the absolute error loss or what we call the \( l1 \)-norm loss. In our discussion above we presented the relative error.
 The MAE is defined as follows
 $$
 \text{MAE}(\hat{y}, \hat{\tilde{y}}) = \frac{1}{n} \sum_{i=0}^{n-1} \left| y_i - \tilde{y}_i \right|.
@@ -573,812 +1208,362 @@ plt.show()
 
 print (error(y))
 

-

-Similarly, using R, we can perform similar studies. The following R code illustrates this.
-(more details on R will be inserted later).
-
-
Non-Linear Least squares in R  

-

-
-

-

-
-
-
set.seed(1485)
-len = 24
-x = runif(len)
-y = x^3+rnorm(len, 0,0.06)
-ds = data.frame(x = x, y = y)
-str(ds)
-plot( y ~ x, main ="Known cubic with noise")
-s  = seq(0,1,length =100)
-lines(s, s^3, lty =2, col ="green")
-m = nls(y ~ I(x^power), data = ds, start = list(power=1), trace = T)
-class(m)
-summary(m)
-power = round(summary(m)$coefficients[1], 3)
-power.se = round(summary(m)$coefficients[2], 3)
-plot(y ~ x, main = "Fitted power model", sub = "Blue: fit; green: known")
-s = seq(0, 1, length = 100)
-lines(s, s^3, lty = 2, col = "green")
-lines(s, predict(m, list(x = s)), lty = 1, col = "blue")
-text(0, 0.5, paste("y =x^ (", power, " +/- ", power.se, ")", sep = ""), pos = 4)
-

-
-

 
+
To our real data: nuclear binding energies. Brief reminder on masses and binding energies 

 
 

-In our lectures on regression analysis (and other ones as well), we will discuss in more details various R functionalities.
+Let us now dive into  nuclear physics and remind ourselves briefly about some basic features about binding
+energies.  A basic quantity which can be measured for the ground
+states of nuclei is the atomic mass \( M(N, Z) \) of the neutral atom with
+atomic mass number \( A \) and charge \( Z \). The number of neutrons is \( N \). There are indeed several sophisticated experiments worldwide which allow us to measure this quantity to high precision (parts per million even).
 
 

-Another useful Python package is
-pandas, which is an open source library
-providing high-performance, easy-to-use data structures and data
-analysis tools for Python. The following simple example shows how we can, in an easy way make tables of our data. Here we define a data set which includes names, city of residence and age, and displays the data in an easy to read way. We will see repeated use of pandas, in particular in connection with classification of data.
+Atomic masses are usually tabulated in terms of the mass excess defined by
+$$
+\Delta M(N, Z) =  M(N, Z) - uA,
+$$
+
+where \( u \) is the Atomic Mass Unit 
+$$
+u = M(^{12}\mathrm{C})/12 = 931.4940954(57) \hspace{0.1cm} \mathrm{MeV}/c^2.
+$$
+
+The nucleon masses are
+$$
+m_p =  1.00727646693(9)u,
+$$
+
+and
+$$
+m_n = 939.56536(8)\hspace{0.1cm} \mathrm{MeV}/c^2 = 1.0086649156(6)u.
+$$
 
 

-
-
-
import pandas as pd
-from IPython.display import display
-data = {'Name': ["John", "Anna", "Peter", "Linda"], 'Location': ["Nairobi", "Napoli", "London", "Buenos Aires"], 'Age':[51, 21, 34, 45]}
-data_pandas = pd.DataFrame(data)
-display(data_pandas)
-

-
-
Examples 

+In the 2016 mass evaluation of by W.J.Huang, G.Audi, M.Wang, F.G.Kondev, S.Naimi and X.Xu
+there are data on masses and decays of 3437 nuclei.
 
 

-We present here several examples, with pertinent Python codes that we
-will use to illustrate various machine learning methods and ways to
-analyze, from simple to complex, various data sets. Many of these
-examples allow us to generate the data we want to analyze, following
-much of the same philosophy we discussed above when
-fitting various polynomials.
+The nuclear binding energy is defined as the energy required to break
+up a given nucleus into its constituent parts of \( N \) neutrons and \( Z \)
+protons. In terms of the atomic masses \( M(N, Z) \) the binding energy is
+defined by
+
+$$
+BE(N, Z) = ZM_H c^2 + Nm_n c^2 - M(N, Z)c^2 ,
+$$
+
+where \( M_H \) is the mass of the hydrogen atom and \( m_n \) is the mass of the neutron.
+In terms of the mass excess the binding energy is given by
+$$
+BE(N, Z) = Z\Delta_H c^2 + N\Delta_n c^2 -\Delta(N, Z)c^2 ,
+$$
+
+where \( \Delta_H c^2 = 7.2890 \) MeV and \( \Delta_n c^2 = 8.0713 \) MeV.
 
 

-We start with a simple exponential growth model that is meant to mimick an ecoli lab experiment.
-We can easily model this system and then produce the data used to train various machine learning algorithms.
-Another model from the life sciences is the so-called predator-prey model from ecology. Thereafter we present 
-a simple model for financial transactions before moving to a random walk model and ending with 
-the simulation of velocities of a non-interacting atom or molecule confined to move in  a one-dimensional region.
+A popular and physically intuitive model which can be used to parametrize 
+the experimental binding energies as function of \( A \), is the so-called 
+liquid drop model. The ansatz is based on the following expression
 
-
Ecoli lab experiment 

+$$ 
+BE(N,Z) = a_1A-a_2A^{2/3}-a_3\frac{Z^2}{A^{1/3}}-a_4\frac{(N-Z)^2}{A},
+$$
 
 

-A typical pattern seen in population models is that the population grows faster and faster. Why? Is there an underlying (general) mechanism?
-Here we will construct a model for cell growth based on a simple difference equation for the growth. We make the following assumptions
-

-
-

-
-
    
-
  1.  Cells divide after \( T \) seconds on average (one generation)
    2. 
-
  2.  \( 2N \) celles divide into twice as many new cells \( \Delta N \) in a time
-   interval \( \Delta t \) as \( N \) cells would: \( \Delta N \propto N \)
    3. 
-
  3.  \( N \) cells result in twice as many new individuals \( \Delta N \) in
-   time \( 2\Delta t \) as in time \( \Delta t \): \( \Delta N \propto\Delta t \)
    4. 
-
  4.  Same proportionality with respect to  death
    5.  
-
  5.  Proposed model: \( \Delta N = b\Delta t N - d\Delta tN \) for some unknown
-   constants \( b \) (births) and \( d \) (deaths)
    6. 
-
  6.  Describe evolution in discrete time: \( t_n=n\Delta t \)
    7. 
-
  7.  Program-friendly notation: \( N \) at \( t_n \) is \( N^n \)
    8. 
-
  8.  Math model: \( N^{n+1} = N^n + r\Delta t\, N \) (with \( \ r=b-d \))
    9. 
-
  9.  Program model: N[n+1] = N[n] + r*dt*N[n]
    10. 
-

-

-
+where \( A \) stands for the number of nucleons and the $a_i$s are parameters which are determined by a fit 
+to the experimental data.
 
 

-The difference equation can be programmed in a simple way, and in order to get started we
-set \( r=1.5 \), \( N^0=1 \), \( \Delta t=0.5 \). The program reads
-
-

-
-
-
import numpy as np
-
-t = np.linspace(0, 10, 21)  # 20 intervals in [0, 10]
-dt = t[1] - t[0]
-N = np.zeros(t.size)
-N[0] = 1
-r = 0.5
-
-for n in range(0, N.size-1, 1):
-    N[n+1] = N[n] + r*dt*N[n]
-    print('N[%d]=%.1f' % (n+1, N[n+1]))
-

-

-and it generates the following output
-

-
-
-
N[1]=1.2
-N[2]=1.6
-N[3]=2.0
-N[4]=2.4
-N[5]=3.1
-N[6]=3.8
-N[7]=4.8
-N[8]=6.0
-N[9]=7.5
-N[10]=9.3
-N[11]=11.6
-N[12]=14.6
-N[13]=18.2
-N[14]=22.7
-N[15]=28.4
-N[16]=35.5
-N[17]=44.4
-N[18]=55.5
-N[19]=69.4
-N[20]=86.7
-

-

-This forms our data which later will define our training set. 
-In this case we defined the value of the parameter \( r \). We could alternatively assume that we just received the 
-above data file and where asked to find \( r \). How can we estimate \( r \) from data? This will be one of our tasks later.
-
-

-We can use the difference equation with the experimental data
-$$ N^{n+1} = N^n + r\Delta t N^n$$
-
-Suppose now that  \( N^{n+1} \) and \( N^n \) are known from data. Then we could solve with respect to  \( r \) as follows
-$$ r = \frac{N^{n+1}-N^n}{N^n\Delta t} $$
-
-Suppose we set \( t_1=600 \), \( t_2=1200 \),
-\( N^1=140 \) and \( N^2=250 \). 
-The following code plots the data 
-

-
-
-
import numpy as np
-import matplotlib.pyplot as plt
-
-# Estimate r
-data = np.loadtxt('ecoli.csv', delimiter=',')
-t_e = data[:,0]
-N_e = data[:,1]
-i = 2  # Data point (i,i+1) used to estimate r
-r = (N_e[i+1] - N_e[i])/(N_e[i]*(t_e[i+1] - t_e[i]))
-print('Estimated r=%.5f' % r)
-# Can experiment with r values and see if the model can
-# match the data better
-T = 1200     # cell can divide after T sec
-t_max = 5*T  # 5 generations in experiment
-t = np.linspace(0, t_max, 1000)
-dt = t[1] - t[0]
-N = np.zeros(t.size)
-
-N[0] = 100
-for n in range(0, len(t)-1, 1):
-    N[n+1] = N[n] + r*dt*N[n]
-
-plt.plot(t, N, 'r-', t_e, N_e, 'bo')
-plt.xlabel('time [s]');  plt.ylabel('N')
-plt.legend(['model', 'experiment'], loc='upper left')
-plt.show()
-

-

-We can then change the parameter \( r \) in the program and play around to make a better fit. By now we know that this
-'search bythe eye' approach is not the most optimal one.
-
-
Predator-Prey model from ecology 

-
-

-The population dynamics of a simple predator-prey system is a
-classical example shown in many biology textbooks when ecological
-systems are discussed. The system contains all elements of the
-scientific method:
+To arrive at the above expression we have assumed that we can make the following assumptions:
 
 
    
- 
  •  The set up of a specific hypothesis combined with
    • 
- 
  •  the experimental methods needed (one can study existing data or perform experiments)
    • 
- 
  •  analyzing and interpreting the data and performing further experiments if needed
    • 
- 
  •  trying to extract general behaviors and extract eventual laws or patterns
    • 
- 
  •  develop mathematical relations for the uncovered regularities/laws and test these by per forming new experiments
    • 
+ 
  •  There is a volume term \( a_1A \) proportional with the number of nucleons (the energy is also an extensive quantity). When an assembly of nucleons of the same size is packed together into the smallest volume, each interior nucleon has a certain number of other nucleons in contact with it. This contribution is proportional to the volume.
    • 
+ 
  •  There is a surface energy term \( a_2A^{2/3} \). The assumption here is that a nucleon at the surface of a nucleus interacts with fewer other nucleons than one in the interior of the nucleus and hence its binding energy is less. This surface energy term takes that into account and is therefore negative and is proportional to the surface area.
    • 
+ 
  •  There is a Coulomb energy term \( a_3\frac{Z^2}{A^{1/3}} \). The electric repulsion between each pair of protons in a nucleus yields less binding.
    •  
+ 
  •  There is an asymmetry term \( a_4\frac{(N-Z)^2}{A} \). This term is associated with the Pauli exclusion principle and reflects the fact that the proton-neutron interaction is more attractive on the average than the neutron-neutron and proton-proton interactions.
    • 
 

 
-Lots of data about populations of hares and lynx collected from furs in Hudson Bay, Canada, are available. It is known that the populations oscillate. Why?
-Here we start by
+We could also add a so-called pairing term, which is a correction term that
+arises from the tendency of proton pairs and neutron pairs to
+occur. An even number of particles is more stable than an odd number.
 
-
    
-
  1.  plotting the data
    2. 
-
  2.  derive a simple model for the population dynamics
    3. 
-
  3.  (fitting parameters in the model to the data)
    4. 
-
  4.  using the model predict the evolution other predator-pray systems
    5. 
-

-
-Most mammalian predators rely on a variety of prey, which complicates mathematical modeling; however, a few predators have become highly specialized and seek almost exclusively a single prey species. An example of this simplified predator-prey interaction is seen in Canadian northern forests, where the populations of the lynx and the snowshoe hare are intertwined in a life and death struggle.
+
Organizing our data 

 
 

-One reason that this particular system has been so extensively studied is that the Hudson Bay company kept careful records of all furs from the early 1800s into the 1900s. The records for the furs collected by the Hudson Bay company showed distinct oscillations (approximately 12 year periods), suggesting that these species caused almost periodic fluctuations of each other's populations. The table here shows data from 1900 to 1920.
+Let us start with reading and organizing our data. 
+We start with the compilation of masses and binding energies from 2016.
+After having downloaded this file to our own computer, we are now ready to read the file and start structuring our data.
 
 

-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
Year Hares (x1000) Lynx (x1000) 
   1900        30.0                 4.0              
   1901        47.2                 6.1              
   1902        70.2                 9.8              
   1903        77.4                 35.2             
   1904        36.3                 59.4             
   1905        20.6                 41.7             
   1906        18.1                 19.0             
   1907        21.4                 13.0             
   1908        22.0                 8.3              
   1909        25.4                 9.1              
   1910        27.1                 7.4              
   1911        40.3                 8.0              
   1912        57                   12.3             
   1913        76.6                 19.5             
   1914        52.3                 45.7             
   1915        19.5                 51.1             
   1916        11.2                 29.7             
   1917        7.6                  15.8             
   1918        14.6                 9.7              
   1919        16.2                 10.1             
   1920        24.7                 8.6              

-

-
-
-
import numpy as np
-from  matplotlib import pyplot as plt
-
-# Load in data file
-data = np.loadtxt('src/Hudson_Bay.csv', delimiter=',', skiprows=1)
-# Make arrays containing x-axis and hares and lynx populations
-year = data[:,0]
-hares = data[:,1]
-lynx = data[:,2]
-
-plt.plot(year, hares ,'b-+', year, lynx, 'r-o')
-plt.axis([1900,1920,0, 100.0])
-plt.xlabel(r'Year')
-plt.ylabel(r'Numbers of hares and lynx ')
-plt.legend(('Hares','Lynx'), loc='upper right')
-plt.title(r'Population of hares and lynx from 1900-1920 (x1000)}')
-plt.savefig('Hudson_Bay_data.pdf')
-plt.savefig('Hudson_Bay_data.png')
-plt.show()
-

-

-




-
-

-We see from the plot that there are indeed fluctuations.
-We would like to create a mathematical model that explains these
-population fluctuations. Ecologists have predicted that in a simple
-predator-prey system that a rise in prey population is followed (with
-a lag) by a rise in the predator population. When the predator
-population is sufficiently high, then the prey population begins
-dropping. After the prey population falls, then the predator
-population falls, which allows the prey population to recover and
-complete one cycle of this interaction. Thus, we see that
-qualitatively oscillations occur. Can a mathematical model predict
-this? What causes cycles to slow or speed up? What affects the
-amplitude of the oscillation or do you expect to see the oscillations
-damp to a stable equilibrium? The models tend to ignore factors like
-climate and other complicating factors. How significant are these?
-
-
    
- 
  •  We see oscillations in the data
    • 
- 
  •  What causes cycles to slow or speed up?
    • 
- 
  •  What affects the amplitude of the oscillation or do you expect to see the oscillations damp to a stable equilibrium?
    • 
- 
  •  With a model we can better understand the data
    • 
- 
  •  More important: Can we understand the ecology dynamics of predator-pray populations?
    • 
-

-
-The classical way (in all books) is to present the Lotka-Volterra equations:
-
-$$
-\begin{align*}
-\frac{dH}{dt} &= H(a - b L)\\
-\frac{dL}{dt} &= - L(d - c  H)
-\end{align*}
-$$
-
-

-Here,
-
-
    
- 
  •  \( H \) is the number of preys
    • 
- 
  •  \( L \) the number of predators
    • 
- 
  •  \( a \), \( b \), \( d \), \( c \) are parameters
    • 
-

-
-The population of hares evolves due to births and deaths exactly as a bacteria population:
-
-$$
-\Delta H = a \Delta t H^n
-$$
-
-However, hares have an additional loss in the population because
-they are eaten by lynx.
-All the hares and lynx can form
-\( H\cdot L \) pairs in total. When such pairs meet during a time
-interval \( \Delta t \), there is some
-small probablity that the lynx will eat the hare.
-So in fraction \( b\Delta t HL \), the lynx eat hares. This
-loss of hares must be accounted for. Subtracted in the equation for hares:
-
-$$ \Delta H = a\Delta t H^n - b \Delta t H^nL^n$$
-
-

-We assume that the primary growth for the lynx population depends on sufficient food for raising lynx kittens, which implies an adequate source of nutrients from predation on hares. Thus, the growth of the lynx population does not only depend of how many lynx there are, but on how many hares they can eat.
-In a time interval \( \Delta t HL \) hares and lynx can meet, and in a
-fraction \( b\Delta t HL \) the lynx eats the hare. All of this does not
-contribute to the growth of lynx, again just a fraction of
-\( b\Delta t HL \) that we write as
-\( d\Delta t HL \). In addition, lynx die just as in the population
-dynamics with one isolated animal population, leading to a loss
-\( -c\Delta t L \).
-The accounting of lynx then looks like
-$$ \Delta L = d\Delta t H^nL^n - c\Delta t L^n$$
-
-

-By writing up the definition of \( \Delta H \) and \( \Delta L \), and putting
-all assumed known terms \( H^n \) and \( L^n \) on the right-hand side, we have
-
-$$ H^{n+1} = H^n + a\Delta t H^n - b\Delta t H^n L^n $$
-
-
-$$ L^{n+1} = L^n + d\Delta t H^nL^n - c\Delta t L^n $$
-
-

-Note:
-
-
    
- 
  •  These equations are ready to be implemented!
    • 
- 
  •  But to start, we need \( H^0 \) and \( L^0 \) (which we can get from the data)
    • 
- 
  •  We also need values for \( a \), \( b \), \( d \), \( c \)
    • 
- 
  •  As always, models tend to be general - as here, applicable
-   to "all" predator-pray systems
    • 
- 
  •  The critical issue is whether the interaction between hares and lynx
-   is sufficiently well modeled by \( \hbox{const}HL \)
    • 
- 
  •  The parameters \( a \), \( b \), \( d \), and \( c \) must be
-   estimated from data
    • 
-

-
-

-
-

-

-
-
-
import numpy as np
-import matplotlib.pyplot as plt
-
-def solver(m, H0, L0, dt, a, b, c, d, t0):
-    """Solve the difference equations for H and L over m years
-    with time step dt (measured in years."""
-
-    num_intervals = int(m/float(dt))
-    t = np.linspace(t0, t0 + m, num_intervals+1)
-    H = np.zeros(t.size)
-    L = np.zeros(t.size)
-
-    print('Init:', H0, L0, dt)
-    H[0] = H0
-    L[0] = L0
-
-    for n in range(0, len(t)-1):
-        H[n+1] = H[n] + a*dt*H[n] - b*dt*H[n]*L[n]
-        L[n+1] = L[n] + d*dt*H[n]*L[n] - c*dt*L[n]
-    return H, L, t
-
-# Load in data file
-data = np.loadtxt('src/Hudson_Bay.csv', delimiter=',', skiprows=1)
-# Make arrays containing x-axis and hares and lynx populations
-t_e = data[:,0]
-H_e = data[:,1]
-L_e = data[:,2]
-
-# Simulate using the model
-H, L, t = solver(m=20, H0=34.91, L0=3.857, dt=0.1,
-                 a=0.4807, b=0.02482, c=0.9272, d=0.02756,
-                 t0=1900)
-
-# Visualize simulations and data
-plt.plot(t_e, H_e, 'b-+', t_e, L_e, 'r-o', t, H, 'm--', t, L, 'k--')
-plt.xlabel('Year')
-plt.ylabel('Numbers of hares and lynx')
-plt.axis([1900, 1920, 0, 140])
-plt.title(r'Population of hares and lynx 1900-1920 (x1000)')
-plt.legend(('H_e', 'L_e', 'H', 'L'), loc='upper left')
-plt.savefig('Hudson_Bay_sim.pdf')
-plt.savefig('Hudson_Bay_sim.png')
-plt.show()
-

-
-

-
-
-

-




-
-

-We will later perform a least-square fitting. Then we can find optimal
-values for the parameters \( a \), \( b \), \( d \), \( c \). In our calculations here
-we set \( a=0.4807 \), \( b=0.02482 \), \( d=0.9272 \) and \( c=0.02756 \). These
-parameters result in a slightly modified initial conditions, namely
-\( H(0) = 34.91 \) and \( L(0)=3.857 \).
-
-

-The following Python code demonstrates how we can use linear regression to fit for example the population of lynx.
-Similarly, we have also used a decision tree algorithm to fit the lynx population data. As expected, the linear regression is not exactly impressive 
+We start with preparing folders for storing our calculations and the data file over masses and binding energies. We import also various modules that we will find useful in order to present various Machine Learning methods. Here we focus mainly on the functionality of scikit-learn.
 

 
 
-
import numpy as np
-import matplotlib.pyplot as plt
-from IPython.display import display
-import sklearn
-from sklearn.linear_model import LinearRegression
-from sklearn.tree import DecisionTreeRegressor
-
-
-data = np.loadtxt('src/Hudson_Bay.csv', delimiter=',', skiprows=1)
-x = data[:,0]
-y = data[:,1]
-line = np.linspace(1900,1920,1000,endpoint=False).reshape(-1,1)
-reg = DecisionTreeRegressor(min_samples_split=3).fit(x.reshape(-1,1),y.reshape(-1,1))
-plt.plot(line, reg.predict(line), label="decision tree")
-regline = LinearRegression().fit(x.reshape(-1,1),y.reshape(-1,1))
-plt.plot(line, regline.predict(line), label= "Linear Regression")
-plt.plot(x, y, label= "Linear Regression")
-plt.show()
-

-

-The similar code for linear regression in R reads (more details to come)
-

-
-
-
HudsonBay = read.csv("src/Hudson_Bay.csv",header=T)
-fix(HudsonBay)
-dim(HudsonBay)
-names(HudsonBay)
-plot(HudsonBay$Year, HudsonBay$Hares..x1000.)
-attach(HudsonBay)
-plot(Year, Hares..x1000.)
-plot(Year, Hares..x1000., col="red", varwidth=T, xlab="Years", ylab="Haresx 1000")
-summary(HudsonBay)
-summary(Hares..x1000.)
-library(MASS)
-library(ISLR)
-scatter.smooth(x=Year, y = Hares..x1000.)
-linearMod = lm(Hares..x1000. ~ Year)
-print(linearMod)
-summary(linearMod)
-plot(linearMod)
-confint(linearMod)
-predict(linearMod,data.frame(Year=c(1910,1914,1920)),interval="confidence")
-

-
-
Simulating financial transactions 

-
-

-The aim here is to simulate financial transactions among financial agents
-using Monte Carlo methods. The final goal is to extract a distribution of income  as function
-of the income \( m \).   From Pareto's work (V. Pareto, 1897) it is known from empirical studies
-that the higher end of the distribution of money follows a distribution 
-$$
-w_m\propto m^{-1-\alpha},
-$$
-
-with \( \alpha\in [1,2] \). We will here follow the analysis made by Patriarca and collaborators.
-
-

-Here we will study numerically the relation between the micro-dynamic relations among financial 
-agents and the  resulting macroscopic money distribution.
-
-

-We assume we have \( N \) agents that exchange money in pairs \( (i,j) \). We assume also that all agents
-start with the same amount of money \( m_0 > 0 \). At a given 'time step', we choose randomly a pair
-of agents \( (i,j) \) and let a transaction take place. This means that agent \( i \)'s money \( m_i \) changes
-to \( m_i' \) and similarly we have \( m_j\rightarrow m_j' \). 
-Money is conserved during a transaction, meaning that
-$$
-\begin{equation}
-  m_i+m_j=m_i'+m_j'.
-\label{eq:conserve}
-\end{equation}
-$$
-
-The change is done via a random reassignement (a random number) \( \epsilon \), meaning that
-
-$$
-\begin{equation*}
-m_i' = \epsilon(m_i+m_j),
-\end{equation*}
-$$
-
-leading to
-
-$$
-\begin{equation*}
-m_j'= (1-\epsilon)(m_i+m_j).
-\end{equation*}
-$$
-
-The number \( \epsilon \) is extracted from a uniform distribution.
-In this simple model, no agents are left with a debt, that is \( m\ge 0 \).
-Due to the conservation law above, one can show that the system relaxes toward an equilibrium
-state given by a Gibbs distribution
-
-$$
-\begin{equation*}
-w_m=\beta \exp{(-\beta m)},
-\end{equation*}
-$$
-
-with
-
-$$
-\begin{equation*}
-\beta = \frac{1}{\langle m\rangle},
-\end{equation*}
-$$
-
-and \( \langle m\rangle=\sum_i m_i/N=m_0 \), the average money.
-It means that after equilibrium has been reached that the majority of agents is left with a small
-number of money, while the number of richest agents, those with \( m \) larger than a specific value \( m' \),
-exponentially decreases with \( m' \).
-
-

-We assume that we have \( N=500 \) agents.   In each simulation, we need a sufficiently large number of transactions, say \( 10^7 \). Our aim is find the final equilibrium distribution \( w_m \). In order to do that we would need
-several runs of the above simulations, at least \( 10^3-10^4 \) runs (experiments).
-
-

-Our task is to first set up an algorithm which simulates the above transactions with an initial
-  amount \( m_0 \).
-  The challenge here is to figure out a Monte Carlo  simulation  based on the
-  above equations.
-  You will in particular need to make an algorithm which sets up a histogram as function of \( m \).
-  This histogram contains the number of times a value \( m \) is registered and represents
-  \( w_m\Delta m \). You will need to set up a value for the interval \( \Delta m \)  (typically \( 0.01-0.05 \)).
-  That means you need to account for the number of times you register an income in the interval
-  \( m,m+\Delta m \). The number of times you register this income, represents the value that enters the histogram.
-
-

-
-
-
#!/usr/bin/env python
+
# Common imports
 import numpy as np
-import matplotlib.mlab as mlab
+import pandas as pd
 import matplotlib.pyplot as plt
-import random
+import sklearn.linear_model as skl
+from sklearn.model_selection import train_test_split
+from sklearn.metrics import mean_squared_error, r2_score, mean_absolute_error
+import os
 
-# initialize the rng with a seed
-random.seed()
-# Hard coding of input parameters
-Agents  = 500
-MCcounts = 1000
-Transactions = 100000
-startMoney = 1.0
-Lambda = 0.0
-FinancialAgents = startMoney*np.ones(Agents)
-for i in range (1, MCcounts, 1):
-    for j in range (1, Transactions, 1):
-        agent_i = int(Agents*random.random())
-        agent_j = int(Agents*random.random())
-        epsilon = random.random()
-        if agent_i != agent_j:
-           m1 = Lambda*FinancialAgents[agent_i] + (1-Lambda)*epsilon*(FinancialAgents[agent_i] + FinancialAgents[agent_j])
-           m2 = Lambda*FinancialAgents[agent_j] + (1-Lambda)*(1-epsilon)*(FinancialAgents[agent_i] + FinancialAgents[agent_j])
-           FinancialAgents[agent_i] = m1
-           FinancialAgents[agent_j] = m2
+# Where to save the figures and data files
+PROJECT_ROOT_DIR = "Results"
+FIGURE_ID = "Results/FigureFiles"
+DATA_ID = "DataFiles/"
 
-# the histogram of the data
-n, bins, patches = plt.hist(FinancialAgents, 50, facecolor='green')
+if not os.path.exists(PROJECT_ROOT_DIR):
+    os.mkdir(PROJECT_ROOT_DIR)
 
-plt.xlabel('$x$')
-plt.ylabel('Distribution of wealth')
-plt.title(r'Money')
-plt.axis([0, 10, 0, 500])
-plt.grid(True)
-plt.show()
+if not os.path.exists(FIGURE_ID):
+    os.makedirs(FIGURE_ID)
+
+if not os.path.exists(DATA_ID):
+    os.makedirs(DATA_ID)
+
+def image_path(fig_id):
+    return os.path.join(FIGURE_ID, fig_id)
+
+def data_path(dat_id):
+    return os.path.join(DATA_ID, dat_id)
+
+def save_fig(fig_id):
+    plt.savefig(image_path(fig_id) + ".png", format='png')
+
+infile = open(data_path("MassEval2016.dat"),'r')
 

 

-We can then change our model to allow for a saving criterion, meaning that the agents save
-  a fraction \( \lambda \) of the money they have before the transaction is made. The final distribution will then no longer be given by Gibbs distribution. It could also include a taxation on financial transactions.
+Before we proceed, we define also a function for making our plots. You can obviously avoid this and simply set up various matplotlib commands every time you need them. You may however find it convenient to collect all such commands in one function and simply call this function. 
+

+
+
+
from pylab import plt, mpl
+plt.style.use('seaborn')
+mpl.rcParams['font.family'] = 'serif'
+
+def MakePlot(x,y, styles, labels, axlabels):
+    plt.figure(figsize=(10,6))
+    for i in range(len(x)):
+        plt.plot(x[i], y[i], styles[i], label = labels[i])
+        plt.xlabel(axlabels[0])
+        plt.ylabel(axlabels[1])
+    plt.legend(loc=0)
+

+

+Our next step is to read the data on experimental binding energies and
+reorganize them as functions of the mass number \( A \), the number of
+protons \( Z \) and neutrons \( N \) using pandas.  Before we do this it is
+always useful (unless you have a binary file or other types of compressed
+data) to actually open the file and simply take a look at it!
 
 

-  The conservation law of Eq. \eqref{eq:conserve} holds, but the money to be shared in a transaction between
-  agent \( i \) and agent \( j \) is now \( (1-\lambda)(m_i+m_j) \). This means that we have
+In particular, the program that outputs the final nuclear masses is written in Fortran with a specific format. It means that we need to figure out the format and which columns contain the data we are interested in. Pandas comes with a function that reads formatted output. After having admired the file, we are now ready to start massaging it with pandas. The file begins with some basic format information.
+

 
-$$
-\begin{equation*}
-  m_i' = \lambda m_i+\epsilon(1-\lambda)(m_i+m_j),
-  \end{equation*}
-$$
-
-  and
-
-$$
-\begin{equation*}
-  m_j' = \lambda m_j+(1-\epsilon)(1-\lambda)(m_i+m_j),
-  \end{equation*}
-$$
-
-  which can be written as
-
-$$
-\begin{equation*}
-  m_i'=m_i+\delta m
-  \end{equation*}
-$$
-
-  and
-
-$$
-\begin{equation*}
-  m_j'=m_j-\delta m,
-  \end{equation*}
-$$
-
-  with
-
-$$
-\begin{equation*}
-  \delta m=(1-\lambda)(\epsilon m_j-(1-\epsilon)m_i),
-  \end{equation*}
-$$
-
-  showing how money is conserved during a transaction.
-  Select values of \( \lambda =0.25,0.5 \) and \( \lambda=0.9 \) and try to extract the corresponding
-  equilibrium distributions and compare these with the Gibbs distribution. We will use this model to 
-extract a parametrization of the above curves, see for example Patriarca and collaborators.
-
-
Particle in one dimension and velocity distribution 

+
+
"""                                                                                                                         
+This is taken from the data file of the mass 2016 evaluation.                                                               
+All files are 3436 lines long with 124 character per line.                                                                  
+       Headers are 39 lines long.                                                                                           
+   col 1     :  Fortran character control: 1 = page feed  0 = line feed                                                     
+   format    :  a1,i3,i5,i5,i5,1x,a3,a4,1x,f13.5,f11.5,f11.3,f9.3,1x,a2,f11.3,f9.3,1x,i3,1x,f12.5,f11.5                     
+   These formats are reflected in the pandas widths variable below, see the statement                                       
+   widths=(1,3,5,5,5,1,3,4,1,13,11,11,9,1,2,11,9,1,3,1,12,11,1),                                                            
+   Pandas has also a variable header, with length 39 in this case.                                                          
+"""
+

+

+The data we are interested in are in columns 2, 3, 4 and 11, giving us
+the number of neutrons, protons, mass numbers and binding energies,
+respectively. We add also for the sake of completeness the element name. The data are in fixed-width formatted lines and we will
+covert them into the pandas DataFrame structure.
 
 

 
 
-
# Program to test the Metropolis algorithm with one particle at given temp in one dimension
-import numpy as np
-import matplotlib.mlab as mlab
-import matplotlib.pyplot as plt
-import random
-from math import sqrt, exp, log
-# initialize the rng with a seed
-random.seed()
-# Hard coding of input parameters
-MCcycles = 100000
-Temperature = 2.0
-beta = 1./Temperature
-InitialVelocity = -2.0
-CurrentVelocity = InitialVelocity
-Energy = 0.5*InitialVelocity*InitialVelocity
-VelocityRange = 10*sqrt(Temperature)
-VelocityStep = 2*VelocityRange/10.
-AverageEnergy = Energy
-AverageEnergy2 = Energy*Energy
-VelocityValues = np.zeros(MCcycles)
-# The Monte Carlo sampling with Metropolis starts here
-for i in range (1, MCcycles, 1):
-    TrialVelocity = CurrentVelocity + (2.0*random.random() - 1.0)*VelocityStep
-    EnergyChange = 0.5*(TrialVelocity*TrialVelocity -CurrentVelocity*CurrentVelocity);
-    if random.random() <= exp(-beta*EnergyChange):
-        CurrentVelocity = TrialVelocity
-        Energy += EnergyChange
-        VelocityValues[i] = CurrentVelocity
-    AverageEnergy += Energy
-    AverageEnergy2 += Energy*Energy
-#Final averages
-AverageEnergy = AverageEnergy/MCcycles
-AverageEnergy2 = AverageEnergy2/MCcycles
-Variance = AverageEnergy2 - AverageEnergy*AverageEnergy
-print(AverageEnergy, Variance)
-n, bins, patches = plt.hist(VelocityValues, 400, facecolor='green')
+
# Read the experimental data with Pandas
+Masses = pd.read_fwf(infile, usecols=(2,3,4,6,11),
+              names=('N', 'Z', 'A', 'Element', 'Ebinding'),
+              widths=(1,3,5,5,5,1,3,4,1,13,11,11,9,1,2,11,9,1,3,1,12,11,1),
+              header=39,
+              index_col=False)
 
-plt.xlabel('$v$')
-plt.ylabel('Velocity distribution P(v)')
-plt.title(r'Velocity histogram at $k_BT=2$')
-plt.axis([-5, 5, 0, 600])
-plt.grid(True)
+# Extrapolated values are indicated by '#' in place of the decimal place, so
+# the Ebinding column won't be numeric. Coerce to float and drop these entries.
+Masses['Ebinding'] = pd.to_numeric(Masses['Ebinding'], errors='coerce')
+Masses = Masses.dropna()
+# Convert from keV to MeV.
+Masses['Ebinding'] /= 1000
+
+# Group the DataFrame by nucleon number, A.
+Masses = Masses.groupby('A')
+# Find the rows of the grouped DataFrame with the maximum binding energy.
+Masses = Masses.apply(lambda t: t[t.Ebinding==t.Ebinding.max()])
+

+

+We have now read in the data, grouped them according to the variables we are interested in. 
+We see how easy it is to reorganize the data using pandas. If we
+were to do these operations in C/C++ or Fortran, we would have had to
+write various functions/subroutines which perform the above
+reorganizations for us.  Having reorganized the data, we can now start
+to make some simple fits using both the functionalities in numpy and
+Scikit-Learn afterwards.
+
+

+Now we define five variables which contain
+the number of nucleons \( A \), the number of protons \( Z \) and the number of neutrons \( N \), the element name and finally the energies themselves.
+

+
+
+
A = Masses['A']
+Z = Masses['Z']
+N = Masses['N']
+Element = Masses['Element']
+Energies = Masses['Ebinding']
+print(Masses)
+

+

+The next step, and we will define this mathematically later, is to set up the so-called design matrix. We will throughout call this matrix \( \boldsymbol{X} \).
+It has dimensionality \( p\times n \), where \( n \) is the number of data points and \( p \) are the so-called predictors. In our case here they are given by the number of polynomials in \( A \) we wish to include in the fit. 
+

+
+
+
# Now we set up the design matrix X
+X = np.zeros((len(A),5))
+X[:,0] = 1
+X[:,1] = A
+X[:,2] = A**(2.0/3.0)
+X[:,3] = A**(-1.0/3.0)
+X[:,4] = A**(-1.0)
+

+

+With scikitlearn we are now ready to use linear regression and fit our data.
+

+
+
+
clf = skl.LinearRegression().fit(X, Energies)
+fity = clf.predict(X)
+

+

+Pretty simple!  
+Now we can print measures of how our fit is doing, the coefficients from the fits and plot the final fit together with our data.
+

+
+
+
# The mean squared error                               
+print("Mean squared error: %.2f" % mean_squared_error(Energies, fity))
+# Explained variance score: 1 is perfect prediction                                 
+print('Variance score: %.2f' % r2_score(Energies, fity))
+# Mean absolute error                                                           
+print('Mean absolute error: %.2f' % mean_absolute_error(Energies, fity))
+print(clf.coef_, clf.intercept_)
+
+Masses['Eapprox']  = fity
+# Generate a plot comparing the experimental with the fitted values values.
+fig, ax = plt.subplots()
+ax.set_xlabel(r'$A = N + Z$')
+ax.set_ylabel(r'$E_\mathrm{bind}\,/\mathrm{MeV}$')
+ax.plot(Masses['A'], Masses['Ebinding'], alpha=0.7, lw=2,
+            label='Ame2016')
+ax.plot(Masses['A'], Masses['Eapprox'], alpha=0.7, lw=2, c='m',
+            label='Fit')
+ax.legend()
+save_fig("Masses2016")
 plt.show()
 

 
-
Random walk model 

+
Seeing the wood for the trees 

+
+

+As a teaser, let us now see how we can do this with decision trees using scikit-learn. Later we will switch to so-called random forests!
 
 

 
 
-
import numpy as np
-import matplotlib.pyplot as plt
-from sklearn.preprocessing import PolynomialFeatures
-from sklearn.linear_model import LinearRegression
-
-steps=250
-
-distance=0
-x=0
-distance_list=[]
-steps_list=[]
-while x<steps:
-    distance+=np.random.randint(-1,2)
-    distance_list.append(distance)
-    x+=1
-    steps_list.append(x)
-plt.plot(steps_list,distance_list, color='green', label="Random Walk Data")
-
-steps_list=np.asarray(steps_list)
-distance_list=np.asarray(distance_list)
-
-X=steps_list[:,np.newaxis]
-
-#Polynomial fits
-
-#Degree 2
-poly_features=PolynomialFeatures(degree=2, include_bias=False)
-X_poly=poly_features.fit_transform(X)
-
-lin_reg=LinearRegression()
-poly_fit=lin_reg.fit(X_poly,distance_list)
-b=lin_reg.coef_
-c=lin_reg.intercept_
-print ("2nd degree coefficients:")
-print ("zero power: ",c)
-print ("first power: ", b[0])
-print ("second power: ",b[1])
-
-z = np.arange(0, steps, .01)
-z_mod=b[1]*z**2+b[0]*z+c
-
-fit_mod=b[1]*X**2+b[0]*X+c
-plt.plot(z, z_mod, color='r', label="2nd Degree Fit")
-plt.title("Polynomial Regression")
-
-plt.xlabel("Steps")
-plt.ylabel("Distance")
-
-#Degree 10
-poly_features10=PolynomialFeatures(degree=10, include_bias=False)
-X_poly10=poly_features10.fit_transform(X)
-
-poly_fit10=lin_reg.fit(X_poly10,distance_list)
-
-y_plot=poly_fit10.predict(X_poly10)
-plt.plot(X, y_plot, color='black', label="10th Degree Fit")
-
-plt.legend()
-plt.show()
-
-
-#Decision Tree Regression
+
#Decision Tree Regression
 from sklearn.tree import DecisionTreeRegressor
-regr_1=DecisionTreeRegressor(max_depth=2)
-regr_2=DecisionTreeRegressor(max_depth=5)
-regr_3=DecisionTreeRegressor(max_depth=7)
-regr_1.fit(X, distance_list)
-regr_2.fit(X, distance_list)
-regr_3.fit(X, distance_list)
+regr_1=DecisionTreeRegressor(max_depth=5)
+regr_2=DecisionTreeRegressor(max_depth=7)
+regr_3=DecisionTreeRegressor(max_depth=9)
+regr_1.fit(X, Energies)
+regr_2.fit(X, Energies)
+regr_3.fit(X, Energies)
 
-X_test = np.arange(0.0, steps, 0.01)[:, np.newaxis]
-y_1 = regr_1.predict(X_test)
-y_2 = regr_2.predict(X_test)
-y_3=regr_3.predict(X_test)
 
+y_1 = regr_1.predict(X)
+y_2 = regr_2.predict(X)
+y_3=regr_3.predict(X)
+Masses['Eapprox'] = y_3
 # Plot the results
 plt.figure()
-plt.scatter(X, distance_list, s=2.5, c="black", label="data")
-plt.plot(X_test, y_1, color="red",
-         label="max_depth=2", linewidth=2)
-plt.plot(X_test, y_2, color="green", label="max_depth=5", linewidth=2)
-plt.plot(X_test, y_3, color="m", label="max_depth=7", linewidth=2)
+plt.plot(A, Energies, color="blue", label="Data", linewidth=2)
+plt.plot(A, y_1, color="red", label="max_depth=5", linewidth=2)
+plt.plot(A, y_2, color="green", label="max_depth=7", linewidth=2)
+plt.plot(A, y_3, color="m", label="max_depth=9", linewidth=2)
 
-plt.xlabel("Data")
-plt.ylabel("Darget")
+plt.xlabel("$A$")
+plt.ylabel("$E$[MeV]")
 plt.title("Decision Tree Regression")
 plt.legend()
+save_fig("Masses2016Trees")
+plt.show()
+print(Masses)
+print(np.mean( (Energies-y_1)**2))
+

+
+
And what about using neural networks? 

+
+The seaborn package allows us to visualize data in an efficient way. Note that we use scikit-learn's multi-layer perceptron (or feed forward neural network) 
+functionality.
+

+
+
+
from sklearn.neural_network import MLPRegressor
+from sklearn.metrics import accuracy_score
+import seaborn as sns
+
+X_train = X
+Y_train = Energies
+n_hidden_neurons = 100
+epochs = 100
+# store models for later use
+eta_vals = np.logspace(-5, 1, 7)
+lmbd_vals = np.logspace(-5, 1, 7)
+# store the models for later use
+DNN_scikit = np.zeros((len(eta_vals), len(lmbd_vals)), dtype=object)
+train_accuracy = np.zeros((len(eta_vals), len(lmbd_vals)))
+sns.set()
+for i, eta in enumerate(eta_vals):
+    for j, lmbd in enumerate(lmbd_vals):
+        dnn = MLPRegressor(hidden_layer_sizes=(n_hidden_neurons), activation='logistic',
+                            alpha=lmbd, learning_rate_init=eta, max_iter=epochs)
+        dnn.fit(X_train, Y_train)
+        DNN_scikit[i][j] = dnn
+        train_accuracy[i][j] = dnn.score(X_train, Y_train)
+
+fig, ax = plt.subplots(figsize = (10, 10))
+sns.heatmap(train_accuracy, annot=True, ax=ax, cmap="viridis")
+ax.set_title("Training Accuracy")
+ax.set_ylabel("$\eta$")
+ax.set_xlabel("$\lambda$")
 plt.show()
 

+
+
A first summary 

+
+

+The aim behind these introductory words was to present to you various
+Python libraries and their functionalities, in particular libraries like
+numpy, pandas, xarray and matplotlib and other that make our life much easier
+in handling various data sets and visualizing data.
+
+

+Furthermore,
+Scikit-Learn allows us with few lines of code to implement popular
+Machine Learning algorithms for supervised learning. Later we will meet Tensorflow, a powerful library for deep learning. 
+Now it is time to dive more into the details of various methods. We will start with linear regression and try to take a deeper look at what it entails.
+
 

 
 
diff --git a/doc/pub/How2ReadData/html/How2ReadData.html b/doc/pub/How2ReadData/html/How2ReadData.html
index ed3295b88..230e16f8a 100644
--- a/doc/pub/How2ReadData/html/How2ReadData.html
+++ b/doc/pub/How2ReadData/html/How2ReadData.html
@@ -67,25 +67,40 @@ div { text-align: justify; text-justify: inter-word; }
 
 
 
@@ -127,7 +142,7 @@ MathJax.Hub.Config({
 
[2] Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University

 

 

-
Jan 14, 2019
 
+
Aug 13, 2019
 
 

 
 
Introduction 

@@ -146,45 +161,147 @@ and use these to introduce statistical data analysis
 concepts and machine learning algorithms before we delve into the
 algorithms themselves. The examples we will use in the beginning, start with simple
 polynomials with random noise added. We will use the Python
-software package Scikit-learn and
+software package Scikit-Learn and
 introduce various machine learning algorithms to make fits of
 the data and predictions. We move thereafter to more interesting
-cases such as the simulation of financial transactions or disease
-models. These are examples where we can easily set up the data and
+cases such as nuclear binding energies.
+These are examples where we can easily set up the data and
 then use machine learning algorithms included in for example
-scikit-learn.
+Scikit-Learn.
 
 

 These examples will serve us the purpose of getting
 started. Furthermore, they allow us to catch more than two birds with
 a stone. They will allow us to bring in some programming specific
-topics and tools as well as showing the power of various Python (and
-R) packages for machine learning and statistical data analysis. In the
-lectures on linear algebra we cover in more detail various programming
-features of languages like Python and C++ (and other), we will also
-look into more specific linear functions which are relevant for the
-various algorithms we will discuss. Here, we will mainly focus on two
-specific Python packages for Machine Learning, scikit-learn and
-tensorflow (see below for links etc).  Moreover, the examples we
+topics and tools as well as showing the power of various Python 
+packages for machine learning and statistical data analysis.
+
+

+Here, we will mainly focus on two
+specific Python packages for Machine Learning, Scikit-Learn and
+Tensorflow (see below for links etc).  Moreover, the examples we
 introduce will serve as inputs to many of our discussions later, as
 well as allowing you to set up models and produce your own data and
 get started with programming.
 
-
Software and needed installations 

+
What is Machine Learning?  

+
+

+Statistics, data science and machine learning form important fields of
+research in modern science.  They describe how to learn and make
+predictions from data, as well as allowing us to extract important
+correlations about physical process and the underlying laws of motion
+in large data sets. The latter, big data sets, appear frequently in
+essentially all disciplines, from the traditional Science, Technology,
+Mathematics and Engineering fields to Life Science, Law, education
+research, the Humanities and the Social Sciences.
+
+

+It has become more
+and more common to see research projects on big data in for example
+the Social Sciences where extracting patterns from complicated survey
+data is one of many research directions.  Having a solid grasp of data
+analysis and machine learning is thus becoming central to scientific
+computing in many fields, and competences and skills within the fields
+of machine learning and scientific computing are nowadays strongly
+requested by many potential employers. The latter cannot be
+overstated, familiarity with machine learning has almost become a
+prerequisite for many of the most exciting employment opportunities,
+whether they are in bioinformatics, life science, physics or finance,
+in the private or the public sector. This author has had several
+students or met students who have been hired recently based on their
+skills and competences in scientific computing and data science, often
+with marginal knowledge of machine learning.
+
+

+Machine learning is a subfield of computer science, and is closely
+related to computational statistics.  It evolved from the study of
+pattern recognition in artificial intelligence (AI) research, and has
+made contributions to AI tasks like computer vision, natural language
+processing and speech recognition. Many of the methods we will study are also 
+strongly rooted in basic mathematics and physics research.
+
+

+Ideally, machine learning represents the science of giving computers
+the ability to learn without being explicitly programmed.  The idea is
+that there exist generic algorithms which can be used to find patterns
+in a broad class of data sets without having to write code
+specifically for each problem. The algorithm will build its own logic
+based on the data.  You should however always keep in mind that
+machines and algorithms are to a large extent developed by humans. The
+insights and knowledge we have about a specific system, play a central
+role when we develop a specific machine learning algorithm.
+
+

+Machine learning is an extremely rich field, in spite of its young
+age. The increases we have seen during the last three decades in
+computational capabilities have been followed by developments of
+methods and techniques for analyzing and handling large date sets,
+relying heavily on statistics, computer science and mathematics.  The
+field is rather new and developing rapidly. Popular software packages
+written in Python for machine learning like
+Scikit-learn,
+Tensorflow,
+PyTorch and Keras, all
+freely available at their respective GitHub sites, encompass
+communities of developers in the thousands or more. And the number of
+code developers and contributors keeps increasing. Not all the
+algorithms and methods can be given a rigorous mathematical
+justification, opening up thereby large rooms for experimenting and
+trial and error and thereby exciting new developments.  However, a
+solid command of linear algebra, multivariate theory, probability
+theory, statistical data analysis, understanding errors and Monte
+Carlo methods are central elements in a proper understanding of many
+of algorithms and methods we will discuss.
+
+
Types of Machine Learning 

+
+

+The approaches to machine learning are many, but are often split into
+two main categories.  In supervised learning we know the answer to a
+problem, and let the computer deduce the logic behind it. On the other
+hand, unsupervised learning is a method for finding patterns and
+relationship in data sets without any prior knowledge of the system.
+Some authours also operate with a third category, namely
+reinforcement learning. This is a paradigm of learning inspired by
+behavioral psychology, where learning is achieved by trial-and-error,
+solely from rewards and punishment.
+
+

+Another way to categorize machine learning tasks is to consider the
+desired output of a system.  Some of the most common tasks are:
+
+
    
+  
  •  Classification: Outputs are divided into two or more classes. The goal is to   produce a model that assigns inputs into one of these classes. An example is to identify  digits based on pictures of hand-written ones. Classification is typically supervised learning.
    • 
+  
  •  Regression: Finding a functional relationship between an input data set and a reference data set.   The goal is to construct a function that maps input data to continuous output values.
    • 
+  
  •  Clustering: Data are divided into groups with certain common traits, without knowing the different groups beforehand.  It is thus a form of unsupervised learning.
    • 
+

+
+The methods we cover have three main topics in common, irrespective of
+whether we deal with supervised or unsupervised learning. The first
+ingredient is normally our data set (which can be subdivided into
+training and test data), the second item is a model which is normally a
+function of some parameters.  The model reflects our knowledge of the system (or lack thereof). As an example, if we know that our data show a behavior similar to what would be predicted by a polynomial, fitting our data to a polynomial of some degree would then determin our model.
+
+

+The last ingredient is a so-called cost
+function which allows us to present an estimate on how good our model
+is in reproducing the data it is supposed to train.  
+At the heart of basically all ML algorithms there are so-called minimization algorithms, often we end up with various variants of gradient methods.
+
+
Software and needed installations 

 
 

 We will make extensive use of Python as programming language and its
 myriad of available libraries.  You will find
-IPython/Jupyter notebooks invaluable in your work.  You can run R
+Jupyter notebooks invaluable in your work.  You can run R
 codes in the Jupyter/IPython notebooks, with the immediate benefit of
 visualizing your data. You can also use compiled languages like C++,
-Rust, Fortran etc if you prefer. The focus in these lectures will be
-on Python, but we will provide many code examples for those of you who
-prefer R or compiled languages. You can integrate C++ codes and R in for example
-a Jupyter notebook.
+Rust, Julia, Fortran etc if you prefer. The focus in these lectures will be
+on Python.
 
 

-If you have Python installed (we recommend Python3) and you feel
+If you have Python installed (we strongly recommend Python3) and you feel
 pretty familiar with installing different packages, we recommend that
 you install the following Python packages via pip as 
 
@@ -212,7 +329,7 @@ you can use pip as well and simply install Python as
 
 etc etc.
 
-
Python installers 

+
Python installers 

 
 

 If you don't want to perform these operations separately and venture
@@ -239,28 +356,46 @@ distribution for scientific and analytic computing distribution and
 analysis environment, available for free and under a commercial
 license.
 
-
Useful Python packages 

-Here we list several useful Python packages.
+

+Furthermore, Google's Colab is a free Jupyter notebook environment that requires 
+no setup and runs entirely in the cloud. Try it out!
 
-
Installing R, C++, cython or Julia 

+
Useful Python libraries 

+Here we list several useful Python libraries we strongly recommend (if you use anaconda many of these are already there)
+
+
    
+
  •  NumPy is a highly popular library for large, multi-dimensional arrays and matrices, along with a large collection of high-level mathematical functions to operate on these arrays
    • 
+
  •  The pandas library provides high-performance, easy-to-use data structures and data analysis tools
    •  
+
  •  Xarray is a Python package that makes working with labelled multi-dimensional arrays simple, efficient, and fun!
    • 
+
  •  Scipy (pronounced “Sigh Pie”) is a Python-based ecosystem of open-source software for mathematics, science, and engineering.
    •  
+
  •  Matplotlib is a Python 2D plotting library which produces publication quality figures in a variety of hardcopy formats and interactive environments across platforms.
    • 
+
  •  Autograd can automatically differentiate native Python and Numpy code. It can handle a large subset of Python's features, including loops, ifs, recursion and closures, and it can even take derivatives of derivatives of derivatives
    • 
+
  •  SymPy is a Python library for symbolic mathematics.
    •  
+
  •  scikit-learn has simple and efficient tools for machine learning, data mining and data analysis
    • 
+
  •  TensorFlow is a Python library for fast numerical computing created and released by Google
    • 
+
  •  Keras is a high-level neural networks API, written in Python and capable of running on top of TensorFlow, CNTK, or Theano
    • 
+
  •  And many more such as pytorch,  Theano etc
    •  
+

+
+
Installing R, C++, cython or Julia 

 
 

-You will also find it convenient to utilize R. Although we will mainly
-use Python during lectures and in various projects and exercises, we
-provide a full R set of codes for the same examples. Those of you
-already familiar with R should feel free to continue using R, keeping
+You will also find it convenient to utilize R. We will mainly
+use Python during lectures and in various projects and exercises.
+Those of you
+already familiar with R should feel free to continue using R, keeping
 however an eye on the parallel Python set ups. Similarly, if you are a
-Python afecionado, feel free to explore R as well.  Jupyter/Ipython
+Python afecionado, feel free to explore R as well.  Jupyter/Ipython
 notebook allows you to run R codes interactively in your
 browser. The software library R is tuned to statistically analysis
 and allows for an easy usage of the tools we will discuss in these
-texts.
+lectures.
 
 

 To install R with Jupyter notebook 
 follow the link here
 
-
Installing R, C++, cython, Numba etc 

+
Installing R, C++, cython, Numba etc 

 
 

 For the C++ aficionados, Jupyter/IPython notebook allows you also to
@@ -271,13 +406,13 @@ languages.
 
 

 To add more entropy, cython can also be used when running your
-notebooks. It means that Python with the Jupyter/IPython notebook
+notebooks. It means that Python with the jupyter notebook
 setup allows you to integrate widely popular softwares and tools for
 scientific computing. Similarly, the 
 Numba Python package delivers increased performance
 capabilities with minimal rewrites of your codes.  With its
 versatility, including symbolic operations, Python offers a unique
-computational environment. Your Jupyter/IPython notebook can easily be
+computational environment. Your jupyter notebook can easily be
 converted into a nicely rendered PDF file or a Latex file for
 further processing. For example, convert to latex as
 
@@ -292,13 +427,513 @@ And to add more versatility, the Python package doconce you can convert a standard ascii text file into various HTML 
-formats, ipython notebooks, latex files, pdf files etc with minimal edits.
+formats, ipython notebooks, latex files, pdf files etc with minimal edits. These lectures were generated using doconce.
 
-
Simple linear regression model using scikit-learn 

+
Numpy examples and Important Matrix and vector handling packages 

 
 

-We start with perhaps our simplest possible example, using scikit-learn to perform linear regression analysis on a data set produced by us. 
-What follows is a simple Python code where we have defined  function \( y \) in terms of the variable \( x \). Both are defined as vectors of dimension \( 1\times 100 \). The entries to the vector \( \hat{x} \)  are given by random numbers generated with a uniform distribution with entries \( x_i \in [0,1] \) (more about probability distribution functions later). These values are then used to define a function \( y(x) \) (tabulated again as a vector) with a linear dependence on \( x \) plus a random noise added via the normal distribution.
+There are several central software packages for linear algebra and eigenvalue problems. Several of the more
+popular ones have been wrapped into ofter software packages like those from the widely used text Numerical Recipes. The original source codes in many of the available packages are often taken from the widely used
+software package LAPACK, which follows two other popular packages
+developed in the 1970s, namely EISPACK and LINPACK.  We describe them shortly here.
+
+
    
+  
  •  LINPACK: package for linear equations and least square problems.
    • 
+  
  •  LAPACK:package for solving symmetric, unsymmetric and generalized eigenvalue problems. From LAPACK's website http://www.netlib.org it is possible to download for free all source codes from this library. Both C/C++ and Fortran versions are available.
    • 
+  
  •  BLAS (I, II and III): (Basic Linear Algebra Subprograms) are routines that provide standard building blocks for performing basic vector and matrix operations. Blas I is vector operations, II vector-matrix operations and III matrix-matrix operations. Highly parallelized and efficient codes, all available for download from http://www.netlib.org.
    • 
+

+
+
Basic Matrix Features 

+
+

+

+Matrix properties reminder.
+

+$$
+ \mathbf{A} =
+      \begin{bmatrix} a_{11} & a_{12} & a_{13} & a_{14} \\
+                                 a_{21} & a_{22} & a_{23} & a_{24} \\
+                                   a_{31} & a_{32} & a_{33} & a_{34} \\
+                                  a_{41} & a_{42} & a_{43} & a_{44}
+             \end{bmatrix}\qquad
+\mathbf{I} =
+      \begin{bmatrix} 1 & 0 & 0 & 0 \\
+                                 0 & 1 & 0 & 0 \\
+                                 0 & 0 & 1 & 0 \\
+                                 0 & 0 & 0 & 1
+             \end{bmatrix}
+$$
+

+
+
+
Basic Matrix Features 

+

+
+

+
+

+The inverse of a matrix is defined by
+
+$$
+\mathbf{A}^{-1} \cdot \mathbf{A} = I
+$$
+

+
+
+
Basic Matrix Features 

+
+

+

+Matrix Properties Reminder.
+

+
+

+
+
+
+
+
+
+
+
+
+
+
+
                Relations                        Name                                    matrix elements                               
   \( A = A^{T} \)                                   symmetric              \( a_{ij} = a_{ji} \)                                                           
   \( A = \left (A^{T} \right )^{-1} \)              real orthogonal        \( \sum_k a_{ik} a_{jk} = \sum_k a_{ki} a_{kj} = \delta_{ij} \)                 
   \( A = A^{ * } \)                                 real matrix            \( a_{ij} = a_{ij}^{ * } \)                                                     
   \( A = A^{\dagger} \)                             hermitian              \( a_{ij} = a_{ji}^{ * } \)                                                     
   \( A = \left (A^{\dagger} \right )^{-1} \)        unitary                \( \sum_k a_{ik} a_{jk}^{ * } = \sum_k a_{ki}^{ * } a_{kj} = \delta_{ij} \)     

+
+

+
+
+
Some famous Matrices 

+
+
    
+  
  •  Diagonal if \( a_{ij}=0 \) for \( i\ne j \)
    • 
+  
  •  Upper triangular if \( a_{ij}=0 \) for \( i > j \)
    • 
+  
  •  Lower triangular if \( a_{ij}=0 \) for \( i < j \)
    • 
+  
  •  Upper Hessenberg if \( a_{ij}=0 \) for \( i > j+1 \)
    • 
+  
  •  Lower Hessenberg if \( a_{ij}=0 \) for \( i < j+1 \)
    • 
+  
  •  Tridiagonal if \( a_{ij}=0 \) for \( |i -j| > 1 \)
    • 
+  
  •  Lower banded with bandwidth \( p \): \( a_{ij}=0 \) for \( i > j+p \)
    • 
+  
  •  Upper banded with bandwidth \( p \): \( a_{ij}=0 \) for \( i < j+p \)
    • 
+  
  •  Banded, block upper triangular, block lower triangular....
    • 
+

+
+
Basic Matrix Features 

+
+

+

+Some Equivalent Statements.
+

+For an \( N\times N \) matrix  \( \mathbf{A} \) the following properties are all equivalent
+
+
    
+  
  •  If the inverse of \( \mathbf{A} \) exists, \( \mathbf{A} \) is nonsingular.
    • 
+  
  •  The equation \( \mathbf{Ax}=0 \) implies \( \mathbf{x}=0 \).
    • 
+  
  •  The rows of \( \mathbf{A} \) form a basis of \( R^N \).
    • 
+  
  •  The columns of \( \mathbf{A} \) form a basis of \( R^N \).
    • 
+  
  •  \( \mathbf{A} \) is a product of elementary matrices.
    • 
+  
  •  \( 0 \) is not eigenvalue of \( \mathbf{A} \).
    • 
+

+

+
+
+
Numpy and arrays 

+Numpy provides an easy way to handle arrays in Python. The standard way to import this library is as
+
+

+
+
+
import numpy as np
+

+

+Here follows a simple example where we set up an array of ten elements, all determined by random numbers drawn according to the normal distribution,
+

+
+
+
n = 10
+x = np.random.normal(size=n)
+print(x)
+

+

+We defined a vector \( x \) with \( n=10 \) elements with its values given by the Normal distribution \( N(0,1) \).
+Another alternative is to declare a vector as follows
+

+
+
+
import numpy as np
+x = np.array([1, 2, 3])
+print(x)
+

+

+Here we have defined a vector with three elements, with \( x_0=1 \), \( x_1=2 \) and \( x_2=3 \). Note that both Python and C++
+start numbering array elements from \( 0 \) and on. This means that a vector with \( n \) elements has a sequence of entities \( x_0, x_1, x_2, \dots, x_{n-1} \). We could also let (recommended) Numpy to compute the logarithms of a specific array as
+

+
+
+
import numpy as np
+x = np.log(np.array([4, 7, 8]))
+print(x)
+

+

+In the last example we used Numpy's unary function \( np.log \). This function is
+highly tuned to compute array elements since the code is vectorized
+and does not require looping. We normaly recommend that you use the
+Numpy intrinsic functions instead of the corresponding log function
+from Python's math module. The looping is done explicitely by the
+np.log function. The alternative, and slower way to compute the
+logarithms of a vector would be to write
+
+

+
+
+
import numpy as np
+from math import log
+x = np.array([4, 7, 8])
+for i in range(0, len(x)):
+    x[i] = log(x[i])
+print(x)
+

+

+We note that our code is much longer already and we need to import the log function from the math module. 
+The attentive reader will also notice that the output is \( [1, 1, 2] \). Python interprets automagically our numbers as integers (like the automatic keyword in C++). To change this we could define our array elements to be double precision numbers as
+

+
+
+
import numpy as np
+x = np.log(np.array([4, 7, 8], dtype = np.float64))
+print(x)
+

+

+or simply write them as double precision numbers (Python uses 64 bits as default for floating point type variables), that is
+

+
+
+
import numpy as np
+x = np.log(np.array([4.0, 7.0, 8.0])
+print(x)
+

+

+To check the number of bytes (remember that one byte contains eight bits for double precision variables), you can use simple use the itemsize functionality (the array \( x \) is actually an object which inherits the functionalities defined in Numpy) as 
+

+
+
+
import numpy as np
+x = np.log(np.array([4.0, 7.0, 8.0])
+print(x.itemsize)
+

+
+
Matrices in Python 

+
+

+Having defined vectors, we are now ready to try out matrices. We can
+define a \( 3 \times 3  \) real matrix \( \hat{A} \) as (recall that we user
+lowercase letters for vectors and uppercase letters for matrices)
+
+

+
+
+
import numpy as np
+A = np.log(np.array([ [4.0, 7.0, 8.0], [3.0, 10.0, 11.0], [4.0, 5.0, 7.0] ]))
+print(A)
+

+

+If we use the shape function we would get \( (3, 3) \) as output, that is verifying that our matrix is a \( 3\times 3 \) matrix. We can slice the matrix and print for example the first column (Python organized matrix elements in a row-major order, see below) as
+

+
+
+
import numpy as np
+A = np.log(np.array([ [4.0, 7.0, 8.0], [3.0, 10.0, 11.0], [4.0, 5.0, 7.0] ]))
+# print the first column, row-major order and elements start with 0
+print(A[:,0]) 
+

+

+We can continue this was by printing out other columns or rows. The example here prints out the second column
+

+
+
+
import numpy as np
+A = np.log(np.array([ [4.0, 7.0, 8.0], [3.0, 10.0, 11.0], [4.0, 5.0, 7.0] ]))
+# print the first column, row-major order and elements start with 0
+print(A[1,:]) 
+

+

+Numpy contains many other functionalities that allow us to slice, subdivide etc etc arrays. We strongly recommend that you look up the Numpy website for more details. Useful functions when defining a matrix are the np.zeros function which declares a matrix of a given dimension and sets all elements to zero
+

+
+
+
import numpy as np
+n = 10
+# define a matrix of dimension 10 x 10 and set all elements to zero
+A = np.zeros( (n, n) )
+print(A) 
+

+

+or initializing all elements to 
+

+
+
+
import numpy as np
+n = 10
+# define a matrix of dimension 10 x 10 and set all elements to one
+A = np.ones( (n, n) )
+print(A) 
+

+

+or as unitarily distributed random numbers (see the material on random number generators in the statistics part)
+

+
+
+
import numpy as np
+n = 10
+# define a matrix of dimension 10 x 10 and set all elements to random numbers with x \in [0, 1]
+A = np.random.rand(n, n)
+print(A) 
+

+

+As we will see throughout these lectures, there are several extremely useful functionalities in Numpy.
+As an example, consider the discussion of the covariance matrix. Suppose we have defined three vectors
+\( \hat{x}, \hat{y}, \hat{z} \) with \( n \) elements each. The covariance matrix is defined as 
+$$
+\hat{\Sigma} = \begin{bmatrix} \sigma_{xx} & \sigma_{xy} & \sigma_{xz} \\
+                              \sigma_{yx} & \sigma_{yy} & \sigma_{yz} \\
+                              \sigma_{zx} & \sigma_{zy} & \sigma_{zz} 
+             \end{bmatrix},
+$$
+
+where for example
+$$
+\sigma_{xy} =\frac{1}{n} \sum_{i=0}^{n-1}(x_i- \overline{x})(y_i- \overline{y}).
+$$
+
+The Numpy function np.cov calculates the covariance elements using the factor \( 1/(n-1) \) instead of \( 1/n \) since it assumes we do not have the exact mean values. 
+The following simple function uses the np.vstack function which takes each vector of dimension \( 1\times n \) and produces a \( 3\times n \) matrix \( \hat{W} \)
+$$
+\hat{W} = \begin{bmatrix} x_0 & y_0 & z_0 \\
+                          x_1 & y_1 & z_1 \\
+                          x_2 & y_2 & z_2 \\
+                          \dots & \dots & \dots \\
+                          x_{n-2} & y_{n-2} & z_{n-2} \\
+                          x_{n-1} & y_{n-1} & z_{n-1}
+             \end{bmatrix},
+$$
+
+

+which in turn is converted into into the \( 3\times 3 \) covariance matrix
+\( \hat{\Sigma} \) via the Numpy function np.cov(). We note that we can also calculate
+the mean value of each set of samples \( \hat{x} \) etc using the Numpy
+function np.mean(x). We can also extract the eigenvalues of the
+covariance matrix through the np.linalg.eig() function.
+
+

+
+
+
# Importing various packages
+import numpy as np
+
+n = 100
+x = np.random.normal(size=n)
+print(np.mean(x))
+y = 4+3*x+np.random.normal(size=n)
+print(np.mean(y))
+z = x**3+np.random.normal(size=n)
+print(np.mean(z))
+W = np.vstack((x, y, z))
+Sigma = np.cov(W)
+print(Sigma)
+Eigvals, Eigvecs = np.linalg.eig(Sigma)
+print(Eigvals)
+

+

+
+
+
import numpy as np
+import matplotlib.pyplot as plt
+from scipy import sparse
+eye = np.eye(4)
+print(eye)
+sparse_mtx = sparse.csr_matrix(eye)
+print(sparse_mtx)
+x = np.linspace(-10,10,100)
+y = np.sin(x)
+plt.plot(x,y,marker='x')
+plt.show()
+

+
+
Meet the Pandas 

+
+

+




+
+

+Another useful Python package is
+pandas, which is an open source library
+providing high-performance, easy-to-use data structures and data
+analysis tools for Python. pandas stands for panel data, a term borrowed from econometrics and is an efficient library for data analysis with an emphasis on tabular data.
+pandas has two major classes, the DataFrame class with two-dimensional data objects and tabular data organized in columns and the class Series with a focus on one-dimensional data objects. Both classes allow you to index data easily as we will see in the examples below. 
+pandas allows you also to perform mathematical operations on the data, spanning from simple reshapings of vectors and matrices to statistical operations.
+
+

+The following simple example shows how we can, in an easy way make tables of our data. Here we define a data set which includes names, place of birth and date of birth, and displays the data in an easy to read way. We will see repeated use of pandas, in particular in connection with classification of data.
+
+

+
+
+
import pandas as pd
+from IPython.display import display
+data = {'First Name': ["Frodo", "Bilbo", "Aragorn II", "Samwise"],
+        'Last Name': ["Baggins", "Baggins","Elessar","Gamgee"],
+        'Place of birth': ["Shire", "Shire", "Eriador", "Shire"],
+        'Date of Birth T.A.': [2968, 2890, 2931, 2980]
+        }
+data_pandas = pd.DataFrame(data)
+display(data_pandas)
+

+

+In the above we have imported pandas with the shorthand pd, the latter has become the standard way we import pandas. We make then a list of various variables
+and reorganize the aboves lists into a DataFrame and then print out  a neat table with specific column labels as Name, place of birth and date of birth.
+Displaying these results, we see that the indices are given by the default numbers from zero to three.
+pandas is extremely flexible and we can easily change the above indices by defining a new type of indexing as
+

+
+
+
data_pandas = pd.DataFrame(data,index=['Frodo','Bilbo','Aragorn','Sam'])
+display(data_pandas)
+

+

+Thereafter we display the content of the row which begins with the index Aragorn
+

+
+
+
display(data_pandas.loc['Aragorn'])
+

+

+We can easily append data to this, for example
+

+
+
+
new_hobbit = {'First Name': ["Peregrin"],
+              'Last Name': ["Took"],
+              'Place of birth': ["Shire"],
+              'Date of Birth T.A.': [2990]
+              }
+data_pandas=data_pandas.append(pd.DataFrame(new_hobbit, index=['Pippin']))
+display(data_pandas)
+

+

+Here are other examples where we use the DataFrame functionality to handle arrays, now with more interesting features for us, namely numbers. We set up a matrix 
+of dimensionality \( 10\times 5 \) and compute the mean value and standard deviation of each column. Similarly, we can perform mathematial operations like squaring the matrix elements and many other operations. 
+

+
+
+
import numpy as np
+import pandas as pd
+from IPython.display import display
+np.random.seed(100)
+# setting up a 10 x 5 matrix
+rows = 10
+cols = 5
+a = np.random.randn(rows,cols)
+df = pd.DataFrame(a)
+display(df)
+print(df.mean())
+print(df.std())
+display(df**2)
+

+

+Thereafter we can select specific columns only and plot final results
+

+
+
+
df.columns = ['First', 'Second', 'Third', 'Fourth', 'Fifth']
+df.index = np.arange(10)
+
+display(df)
+print(df['Second'].mean() )
+print(df.info())
+print(df.describe())
+
+from pylab import plt, mpl
+plt.style.use('seaborn')
+mpl.rcParams['font.family'] = 'serif'
+
+df.cumsum().plot(lw=2.0, figsize=(10,6))
+plt.show()
+
+
+df.plot.bar(figsize=(10,6), rot=15)
+plt.show()
+

+

+We can produce a \( 4\times 4 \) matrix
+

+
+
+
b = np.arange(16).reshape((4,4))
+print(b)
+df1 = pd.DataFrame(b)
+print(df1)
+

+

+and many other operations.
+
+

+The Series class is another important class included in
+pandas. You can view it as a specialization of DataFrame but where
+we have just a single column of data. It shares many of the same features as _DataFrame. As with DataFrame,
+most operations are vectorized, achieving thereby a high performance when dealing with computations of arrays, in particular labeled arrays.
+As we will see below it leads also to a very concice code close to the mathematical operations we may be interested in.
+For multidimensional arrays, we recommend strongly xarray. xarray has much of the same flexibility as pandas, but allows for the extension to higher dimensions than two. We will see examples later of the usage of both pandas and xarray.
+
+
Reading Data and fitting  

+
+

+In order to study various Machine Learning algorithms, we need to
+access data. Acccessing data is an essential step in all machine
+learning algorithms. In particular, setting up the so-called design
+matrix (to be defined below) is often the first element we need in
+order to perform our calculations. To set up the design matrix means
+reading (and later, when the calculations are done, writing) data
+in various formats, The formats span from reading files from disk,
+loading data from databases and interacting with online sources
+like web application programming interfaces (APIs).
+
+

+In handling various input formats, as discussed above, we will mainly stay with pandas,
+a Python package which allows us, in a seamless and painless way, to
+deal with a multitude of formats, from standard csv (comma separated
+values) files, via excel, html to hdf5 formats.  With pandas
+and the DataFrame  and Series functionalities we are able to convert text data
+into the calculational formats we need for a specific algorithm. And our code is going to be 
+pretty close the basic mathematical expressions.
+
+

+Our first data set is going to be a classic from nuclear physics, namely all
+available data on binding energies. Don't be intimidated if you are not familiar with nuclear physics. It serves simply as an example here of a data set.
+
+

+We will show some of the
+strengths of packages like Scikit-Learn in fitting nuclear binding energies to
+specific functions using linear regression first. Then, as a teaser, we will show you how 
+you can easily implement other algorithms like decision trees and random forests and neural networks.
+
+

+But before we really start with nuclear physics data, let's just look at some simpler polynomial fitting cases, such as,
+(don't be offended) fitting straight lines!
+
+
Simple linear regression model using scikit-learn 

+
+

+We start with perhaps our simplest possible example, using Scikit-Learn to perform linear regression analysis on a data set produced by us.
+
+

+What follows is a simple Python code where we have defined a function
+\( y \) in terms of the variable \( x \). Both are defined as vectors with  \( 100 \) entries. 
+The numbers in the vector \( \hat{x} \) are given
+by random numbers generated with a uniform distribution with entries
+\( x_i \in [0,1] \) (more about probability distribution functions
+later). These values are then used to define a function \( y(x) \)
+(tabulated again as a vector) with a linear dependence on \( x \) plus a
+random noise added via the normal distribution.
 
 

 The Numpy functions are imported used the import numpy as np
@@ -317,7 +952,7 @@ $$
 
 

 where \( N(0,1) \) represents random numbers generated by the normal
-distribution.  From scikit-learn we import then the
+distribution.  From Scikit-Learn we import then the
 LinearRegression functionality and make a prediction \( \tilde{y} =
 \alpha + \beta x \) using the function fit(x,y). We call the set of
 data \( (\hat{x},\hat{y}) \) for our training data. The Python package
@@ -375,7 +1010,7 @@ $$
 
 

 where \( x \) is defined as before.  Does the fit look better? Indeed, by
-reducing the role of the normal distribution we see immediately that
+reducing the role of the noise given by the normal distribution we see immediately that
 our linear prediction seemingly reproduces better the training
 set. However, this testing 'by the eye' is obviouly not satisfactory in the
 long run. Here we have only defined the training data and our model, and 
@@ -386,7 +1021,7 @@ We need more rigorous criteria in defining whether we have succeeded or
 not in modeling our training data.  You will be surprised to see that
 many scientists seldomly venture beyond this 'by the eye' approach. A
 standard approach for the cost function is the so-called \( \chi^2 \)
-function 
+function (a variant of the mean-squared error (MSE))
 
 $$ \chi^2 = \frac{1}{n}
 \sum_{i=0}^{n-1}\frac{(y_i-\tilde{y}_i)^2}{\sigma_i^2}, 
@@ -416,13 +1051,13 @@ the  \( \chi^2 \) function becomes smaller.
 
 

 There are many ways to define the cost function. A simpler approach is to look at the relative difference between the training data and the predicted data, that is we define 
-the relative error as
+the relative error (why would we prefer the MSE instead of the relative error?) as
 
 $$
 \epsilon_{\mathrm{relative}}= \frac{\vert \hat{y} -\hat{\tilde{y}}\vert}{\vert \hat{y}\vert}.
 $$
 
-We can modify easily the above Python code and plot the relative instead
+We can modify easily the above Python code and plot the relative error instead
 

 
 
@@ -450,14 +1085,14 @@ different training data sets and study (graphically) the value of the
 relative error.
 
 

-As mentioned above, scikit-learn has an impressive functionality.
+As mentioned above, Scikit-Learn has an impressive functionality.
 We can for example extract the values of \( \alpha \) and \( \beta \) and
 their error estimates, or the variance and standard deviation and many
 other properties from the statistical data analysis.
 
 

 Here we show an
-example of the functionality of scikit-learn.
+example of the functionality of Scikit-Learn.
 

 
 
@@ -520,8 +1155,8 @@ $$
 \bar{y} =  \frac{1}{n} \sum_{i=0}^{n - 1} y_i.
 $$
 
-Another quantity will meet again in our discussions of regression analysis is 
- mean absolute error (MAE), a risk metric corresponding to the expected value of the absolute error loss or what we call the \( l1 \)-norm loss. In our discussion above we presented the relative error.
+Another quantity taht we will meet again in our discussions of regression analysis is 
+ the mean absolute error (MAE), a risk metric corresponding to the expected value of the absolute error loss or what we call the \( l1 \)-norm loss. In our discussion above we presented the relative error.
 The MAE is defined as follows
 $$
 \text{MAE}(\hat{y}, \hat{\tilde{y}}) = \frac{1}{n} \sum_{i=0}^{n-1} \left| y_i - \tilde{y}_i \right|.
@@ -578,812 +1213,362 @@ plt.show()
 
 print (error(y))
 

-

-Similarly, using R, we can perform similar studies. The following R code illustrates this.
-(more details on R will be inserted later).
-
-
Non-Linear Least squares in R  

-

-
-

-

-
-
-
set.seed(1485)
-len = 24
-x = runif(len)
-y = x^3+rnorm(len, 0,0.06)
-ds = data.frame(x = x, y = y)
-str(ds)
-plot( y ~ x, main ="Known cubic with noise")
-s  = seq(0,1,length =100)
-lines(s, s^3, lty =2, col ="green")
-m = nls(y ~ I(x^power), data = ds, start = list(power=1), trace = T)
-class(m)
-summary(m)
-power = round(summary(m)$coefficients[1], 3)
-power.se = round(summary(m)$coefficients[2], 3)
-plot(y ~ x, main = "Fitted power model", sub = "Blue: fit; green: known")
-s = seq(0, 1, length = 100)
-lines(s, s^3, lty = 2, col = "green")
-lines(s, predict(m, list(x = s)), lty = 1, col = "blue")
-text(0, 0.5, paste("y =x^ (", power, " +/- ", power.se, ")", sep = ""), pos = 4)
-

-
-

 
+
To our real data: nuclear binding energies. Brief reminder on masses and binding energies 

 
 

-In our lectures on regression analysis (and other ones as well), we will discuss in more details various R functionalities.
+Let us now dive into  nuclear physics and remind ourselves briefly about some basic features about binding
+energies.  A basic quantity which can be measured for the ground
+states of nuclei is the atomic mass \( M(N, Z) \) of the neutral atom with
+atomic mass number \( A \) and charge \( Z \). The number of neutrons is \( N \). There are indeed several sophisticated experiments worldwide which allow us to measure this quantity to high precision (parts per million even).
 
 

-Another useful Python package is
-pandas, which is an open source library
-providing high-performance, easy-to-use data structures and data
-analysis tools for Python. The following simple example shows how we can, in an easy way make tables of our data. Here we define a data set which includes names, city of residence and age, and displays the data in an easy to read way. We will see repeated use of pandas, in particular in connection with classification of data.
+Atomic masses are usually tabulated in terms of the mass excess defined by
+$$
+\Delta M(N, Z) =  M(N, Z) - uA,
+$$
+
+where \( u \) is the Atomic Mass Unit 
+$$
+u = M(^{12}\mathrm{C})/12 = 931.4940954(57) \hspace{0.1cm} \mathrm{MeV}/c^2.
+$$
+
+The nucleon masses are
+$$
+m_p =  1.00727646693(9)u,
+$$
+
+and
+$$
+m_n = 939.56536(8)\hspace{0.1cm} \mathrm{MeV}/c^2 = 1.0086649156(6)u.
+$$
 
 

-
-
-
import pandas as pd
-from IPython.display import display
-data = {'Name': ["John", "Anna", "Peter", "Linda"], 'Location': ["Nairobi", "Napoli", "London", "Buenos Aires"], 'Age':[51, 21, 34, 45]}
-data_pandas = pd.DataFrame(data)
-display(data_pandas)
-

-
-
Examples 

+In the 2016 mass evaluation of by W.J.Huang, G.Audi, M.Wang, F.G.Kondev, S.Naimi and X.Xu
+there are data on masses and decays of 3437 nuclei.
 
 

-We present here several examples, with pertinent Python codes that we
-will use to illustrate various machine learning methods and ways to
-analyze, from simple to complex, various data sets. Many of these
-examples allow us to generate the data we want to analyze, following
-much of the same philosophy we discussed above when
-fitting various polynomials.
+The nuclear binding energy is defined as the energy required to break
+up a given nucleus into its constituent parts of \( N \) neutrons and \( Z \)
+protons. In terms of the atomic masses \( M(N, Z) \) the binding energy is
+defined by
+
+$$
+BE(N, Z) = ZM_H c^2 + Nm_n c^2 - M(N, Z)c^2 ,
+$$
+
+where \( M_H \) is the mass of the hydrogen atom and \( m_n \) is the mass of the neutron.
+In terms of the mass excess the binding energy is given by
+$$
+BE(N, Z) = Z\Delta_H c^2 + N\Delta_n c^2 -\Delta(N, Z)c^2 ,
+$$
+
+where \( \Delta_H c^2 = 7.2890 \) MeV and \( \Delta_n c^2 = 8.0713 \) MeV.
 
 

-We start with a simple exponential growth model that is meant to mimick an ecoli lab experiment.
-We can easily model this system and then produce the data used to train various machine learning algorithms.
-Another model from the life sciences is the so-called predator-prey model from ecology. Thereafter we present 
-a simple model for financial transactions before moving to a random walk model and ending with 
-the simulation of velocities of a non-interacting atom or molecule confined to move in  a one-dimensional region.
+A popular and physically intuitive model which can be used to parametrize 
+the experimental binding energies as function of \( A \), is the so-called 
+liquid drop model. The ansatz is based on the following expression
 
-
Ecoli lab experiment 

+$$ 
+BE(N,Z) = a_1A-a_2A^{2/3}-a_3\frac{Z^2}{A^{1/3}}-a_4\frac{(N-Z)^2}{A},
+$$
 
 

-A typical pattern seen in population models is that the population grows faster and faster. Why? Is there an underlying (general) mechanism?
-Here we will construct a model for cell growth based on a simple difference equation for the growth. We make the following assumptions
-

-
-

-
-
    
-
  1.  Cells divide after \( T \) seconds on average (one generation)
    2. 
-
  2.  \( 2N \) celles divide into twice as many new cells \( \Delta N \) in a time
-   interval \( \Delta t \) as \( N \) cells would: \( \Delta N \propto N \)
    3. 
-
  3.  \( N \) cells result in twice as many new individuals \( \Delta N \) in
-   time \( 2\Delta t \) as in time \( \Delta t \): \( \Delta N \propto\Delta t \)
    4. 
-
  4.  Same proportionality with respect to  death
    5.  
-
  5.  Proposed model: \( \Delta N = b\Delta t N - d\Delta tN \) for some unknown
-   constants \( b \) (births) and \( d \) (deaths)
    6. 
-
  6.  Describe evolution in discrete time: \( t_n=n\Delta t \)
    7. 
-
  7.  Program-friendly notation: \( N \) at \( t_n \) is \( N^n \)
    8. 
-
  8.  Math model: \( N^{n+1} = N^n + r\Delta t\, N \) (with \( \ r=b-d \))
    9. 
-
  9.  Program model: N[n+1] = N[n] + r*dt*N[n]
    10. 
-

-

-
+where \( A \) stands for the number of nucleons and the $a_i$s are parameters which are determined by a fit 
+to the experimental data.
 
 

-The difference equation can be programmed in a simple way, and in order to get started we
-set \( r=1.5 \), \( N^0=1 \), \( \Delta t=0.5 \). The program reads
-
-

-
-
-
import numpy as np
-
-t = np.linspace(0, 10, 21)  # 20 intervals in [0, 10]
-dt = t[1] - t[0]
-N = np.zeros(t.size)
-N[0] = 1
-r = 0.5
-
-for n in range(0, N.size-1, 1):
-    N[n+1] = N[n] + r*dt*N[n]
-    print('N[%d]=%.1f' % (n+1, N[n+1]))
-

-

-and it generates the following output
-

-
-
-
N[1]=1.2
-N[2]=1.6
-N[3]=2.0
-N[4]=2.4
-N[5]=3.1
-N[6]=3.8
-N[7]=4.8
-N[8]=6.0
-N[9]=7.5
-N[10]=9.3
-N[11]=11.6
-N[12]=14.6
-N[13]=18.2
-N[14]=22.7
-N[15]=28.4
-N[16]=35.5
-N[17]=44.4
-N[18]=55.5
-N[19]=69.4
-N[20]=86.7
-

-

-This forms our data which later will define our training set. 
-In this case we defined the value of the parameter \( r \). We could alternatively assume that we just received the 
-above data file and where asked to find \( r \). How can we estimate \( r \) from data? This will be one of our tasks later.
-
-

-We can use the difference equation with the experimental data
-$$ N^{n+1} = N^n + r\Delta t N^n$$
-
-Suppose now that  \( N^{n+1} \) and \( N^n \) are known from data. Then we could solve with respect to  \( r \) as follows
-$$ r = \frac{N^{n+1}-N^n}{N^n\Delta t} $$
-
-Suppose we set \( t_1=600 \), \( t_2=1200 \),
-\( N^1=140 \) and \( N^2=250 \). 
-The following code plots the data 
-

-
-
-
import numpy as np
-import matplotlib.pyplot as plt
-
-# Estimate r
-data = np.loadtxt('ecoli.csv', delimiter=',')
-t_e = data[:,0]
-N_e = data[:,1]
-i = 2  # Data point (i,i+1) used to estimate r
-r = (N_e[i+1] - N_e[i])/(N_e[i]*(t_e[i+1] - t_e[i]))
-print('Estimated r=%.5f' % r)
-# Can experiment with r values and see if the model can
-# match the data better
-T = 1200     # cell can divide after T sec
-t_max = 5*T  # 5 generations in experiment
-t = np.linspace(0, t_max, 1000)
-dt = t[1] - t[0]
-N = np.zeros(t.size)
-
-N[0] = 100
-for n in range(0, len(t)-1, 1):
-    N[n+1] = N[n] + r*dt*N[n]
-
-plt.plot(t, N, 'r-', t_e, N_e, 'bo')
-plt.xlabel('time [s]');  plt.ylabel('N')
-plt.legend(['model', 'experiment'], loc='upper left')
-plt.show()
-

-

-We can then change the parameter \( r \) in the program and play around to make a better fit. By now we know that this
-'search bythe eye' approach is not the most optimal one.
-
-
Predator-Prey model from ecology 

-
-

-The population dynamics of a simple predator-prey system is a
-classical example shown in many biology textbooks when ecological
-systems are discussed. The system contains all elements of the
-scientific method:
+To arrive at the above expression we have assumed that we can make the following assumptions:
 
 
    
- 
  •  The set up of a specific hypothesis combined with
    • 
- 
  •  the experimental methods needed (one can study existing data or perform experiments)
    • 
- 
  •  analyzing and interpreting the data and performing further experiments if needed
    • 
- 
  •  trying to extract general behaviors and extract eventual laws or patterns
    • 
- 
  •  develop mathematical relations for the uncovered regularities/laws and test these by per forming new experiments
    • 
+ 
  •  There is a volume term \( a_1A \) proportional with the number of nucleons (the energy is also an extensive quantity). When an assembly of nucleons of the same size is packed together into the smallest volume, each interior nucleon has a certain number of other nucleons in contact with it. This contribution is proportional to the volume.
    • 
+ 
  •  There is a surface energy term \( a_2A^{2/3} \). The assumption here is that a nucleon at the surface of a nucleus interacts with fewer other nucleons than one in the interior of the nucleus and hence its binding energy is less. This surface energy term takes that into account and is therefore negative and is proportional to the surface area.
    • 
+ 
  •  There is a Coulomb energy term \( a_3\frac{Z^2}{A^{1/3}} \). The electric repulsion between each pair of protons in a nucleus yields less binding.
    •  
+ 
  •  There is an asymmetry term \( a_4\frac{(N-Z)^2}{A} \). This term is associated with the Pauli exclusion principle and reflects the fact that the proton-neutron interaction is more attractive on the average than the neutron-neutron and proton-proton interactions.
    • 
 

 
-Lots of data about populations of hares and lynx collected from furs in Hudson Bay, Canada, are available. It is known that the populations oscillate. Why?
-Here we start by
+We could also add a so-called pairing term, which is a correction term that
+arises from the tendency of proton pairs and neutron pairs to
+occur. An even number of particles is more stable than an odd number.
 
-
    
-
  1.  plotting the data
    2. 
-
  2.  derive a simple model for the population dynamics
    3. 
-
  3.  (fitting parameters in the model to the data)
    4. 
-
  4.  using the model predict the evolution other predator-pray systems
    5. 
-

-
-Most mammalian predators rely on a variety of prey, which complicates mathematical modeling; however, a few predators have become highly specialized and seek almost exclusively a single prey species. An example of this simplified predator-prey interaction is seen in Canadian northern forests, where the populations of the lynx and the snowshoe hare are intertwined in a life and death struggle.
+
Organizing our data 

 
 

-One reason that this particular system has been so extensively studied is that the Hudson Bay company kept careful records of all furs from the early 1800s into the 1900s. The records for the furs collected by the Hudson Bay company showed distinct oscillations (approximately 12 year periods), suggesting that these species caused almost periodic fluctuations of each other's populations. The table here shows data from 1900 to 1920.
+Let us start with reading and organizing our data. 
+We start with the compilation of masses and binding energies from 2016.
+After having downloaded this file to our own computer, we are now ready to read the file and start structuring our data.
 
 

-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
Year Hares (x1000) Lynx (x1000) 
   1900        30.0                 4.0              
   1901        47.2                 6.1              
   1902        70.2                 9.8              
   1903        77.4                 35.2             
   1904        36.3                 59.4             
   1905        20.6                 41.7             
   1906        18.1                 19.0             
   1907        21.4                 13.0             
   1908        22.0                 8.3              
   1909        25.4                 9.1              
   1910        27.1                 7.4              
   1911        40.3                 8.0              
   1912        57                   12.3             
   1913        76.6                 19.5             
   1914        52.3                 45.7             
   1915        19.5                 51.1             
   1916        11.2                 29.7             
   1917        7.6                  15.8             
   1918        14.6                 9.7              
   1919        16.2                 10.1             
   1920        24.7                 8.6              

-

-
-
-
import numpy as np
-from  matplotlib import pyplot as plt
-
-# Load in data file
-data = np.loadtxt('src/Hudson_Bay.csv', delimiter=',', skiprows=1)
-# Make arrays containing x-axis and hares and lynx populations
-year = data[:,0]
-hares = data[:,1]
-lynx = data[:,2]
-
-plt.plot(year, hares ,'b-+', year, lynx, 'r-o')
-plt.axis([1900,1920,0, 100.0])
-plt.xlabel(r'Year')
-plt.ylabel(r'Numbers of hares and lynx ')
-plt.legend(('Hares','Lynx'), loc='upper right')
-plt.title(r'Population of hares and lynx from 1900-1920 (x1000)}')
-plt.savefig('Hudson_Bay_data.pdf')
-plt.savefig('Hudson_Bay_data.png')
-plt.show()
-

-

-




-
-

-We see from the plot that there are indeed fluctuations.
-We would like to create a mathematical model that explains these
-population fluctuations. Ecologists have predicted that in a simple
-predator-prey system that a rise in prey population is followed (with
-a lag) by a rise in the predator population. When the predator
-population is sufficiently high, then the prey population begins
-dropping. After the prey population falls, then the predator
-population falls, which allows the prey population to recover and
-complete one cycle of this interaction. Thus, we see that
-qualitatively oscillations occur. Can a mathematical model predict
-this? What causes cycles to slow or speed up? What affects the
-amplitude of the oscillation or do you expect to see the oscillations
-damp to a stable equilibrium? The models tend to ignore factors like
-climate and other complicating factors. How significant are these?
-
-
    
- 
  •  We see oscillations in the data
    • 
- 
  •  What causes cycles to slow or speed up?
    • 
- 
  •  What affects the amplitude of the oscillation or do you expect to see the oscillations damp to a stable equilibrium?
    • 
- 
  •  With a model we can better understand the data
    • 
- 
  •  More important: Can we understand the ecology dynamics of predator-pray populations?
    • 
-

-
-The classical way (in all books) is to present the Lotka-Volterra equations:
-
-$$
-\begin{align*}
-\frac{dH}{dt} &= H(a - b L)\\
-\frac{dL}{dt} &= - L(d - c  H)
-\end{align*}
-$$
-
-

-Here,
-
-
    
- 
  •  \( H \) is the number of preys
    • 
- 
  •  \( L \) the number of predators
    • 
- 
  •  \( a \), \( b \), \( d \), \( c \) are parameters
    • 
-

-
-The population of hares evolves due to births and deaths exactly as a bacteria population:
-
-$$
-\Delta H = a \Delta t H^n
-$$
-
-However, hares have an additional loss in the population because
-they are eaten by lynx.
-All the hares and lynx can form
-\( H\cdot L \) pairs in total. When such pairs meet during a time
-interval \( \Delta t \), there is some
-small probablity that the lynx will eat the hare.
-So in fraction \( b\Delta t HL \), the lynx eat hares. This
-loss of hares must be accounted for. Subtracted in the equation for hares:
-
-$$ \Delta H = a\Delta t H^n - b \Delta t H^nL^n$$
-
-

-We assume that the primary growth for the lynx population depends on sufficient food for raising lynx kittens, which implies an adequate source of nutrients from predation on hares. Thus, the growth of the lynx population does not only depend of how many lynx there are, but on how many hares they can eat.
-In a time interval \( \Delta t HL \) hares and lynx can meet, and in a
-fraction \( b\Delta t HL \) the lynx eats the hare. All of this does not
-contribute to the growth of lynx, again just a fraction of
-\( b\Delta t HL \) that we write as
-\( d\Delta t HL \). In addition, lynx die just as in the population
-dynamics with one isolated animal population, leading to a loss
-\( -c\Delta t L \).
-The accounting of lynx then looks like
-$$ \Delta L = d\Delta t H^nL^n - c\Delta t L^n$$
-
-

-By writing up the definition of \( \Delta H \) and \( \Delta L \), and putting
-all assumed known terms \( H^n \) and \( L^n \) on the right-hand side, we have
-
-$$ H^{n+1} = H^n + a\Delta t H^n - b\Delta t H^n L^n $$
-
-
-$$ L^{n+1} = L^n + d\Delta t H^nL^n - c\Delta t L^n $$
-
-

-Note:
-
-
    
- 
  •  These equations are ready to be implemented!
    • 
- 
  •  But to start, we need \( H^0 \) and \( L^0 \) (which we can get from the data)
    • 
- 
  •  We also need values for \( a \), \( b \), \( d \), \( c \)
    • 
- 
  •  As always, models tend to be general - as here, applicable
-   to "all" predator-pray systems
    • 
- 
  •  The critical issue is whether the interaction between hares and lynx
-   is sufficiently well modeled by \( \hbox{const}HL \)
    • 
- 
  •  The parameters \( a \), \( b \), \( d \), and \( c \) must be
-   estimated from data
    • 
-

-
-

-
-

-

-
-
-
import numpy as np
-import matplotlib.pyplot as plt
-
-def solver(m, H0, L0, dt, a, b, c, d, t0):
-    """Solve the difference equations for H and L over m years
-    with time step dt (measured in years."""
-
-    num_intervals = int(m/float(dt))
-    t = np.linspace(t0, t0 + m, num_intervals+1)
-    H = np.zeros(t.size)
-    L = np.zeros(t.size)
-
-    print('Init:', H0, L0, dt)
-    H[0] = H0
-    L[0] = L0
-
-    for n in range(0, len(t)-1):
-        H[n+1] = H[n] + a*dt*H[n] - b*dt*H[n]*L[n]
-        L[n+1] = L[n] + d*dt*H[n]*L[n] - c*dt*L[n]
-    return H, L, t
-
-# Load in data file
-data = np.loadtxt('src/Hudson_Bay.csv', delimiter=',', skiprows=1)
-# Make arrays containing x-axis and hares and lynx populations
-t_e = data[:,0]
-H_e = data[:,1]
-L_e = data[:,2]
-
-# Simulate using the model
-H, L, t = solver(m=20, H0=34.91, L0=3.857, dt=0.1,
-                 a=0.4807, b=0.02482, c=0.9272, d=0.02756,
-                 t0=1900)
-
-# Visualize simulations and data
-plt.plot(t_e, H_e, 'b-+', t_e, L_e, 'r-o', t, H, 'm--', t, L, 'k--')
-plt.xlabel('Year')
-plt.ylabel('Numbers of hares and lynx')
-plt.axis([1900, 1920, 0, 140])
-plt.title(r'Population of hares and lynx 1900-1920 (x1000)')
-plt.legend(('H_e', 'L_e', 'H', 'L'), loc='upper left')
-plt.savefig('Hudson_Bay_sim.pdf')
-plt.savefig('Hudson_Bay_sim.png')
-plt.show()
-

-
-

-
-
-

-




-
-

-We will later perform a least-square fitting. Then we can find optimal
-values for the parameters \( a \), \( b \), \( d \), \( c \). In our calculations here
-we set \( a=0.4807 \), \( b=0.02482 \), \( d=0.9272 \) and \( c=0.02756 \). These
-parameters result in a slightly modified initial conditions, namely
-\( H(0) = 34.91 \) and \( L(0)=3.857 \).
-
-

-The following Python code demonstrates how we can use linear regression to fit for example the population of lynx.
-Similarly, we have also used a decision tree algorithm to fit the lynx population data. As expected, the linear regression is not exactly impressive 
+We start with preparing folders for storing our calculations and the data file over masses and binding energies. We import also various modules that we will find useful in order to present various Machine Learning methods. Here we focus mainly on the functionality of scikit-learn.
 

 
 
-
import numpy as np
-import matplotlib.pyplot as plt
-from IPython.display import display
-import sklearn
-from sklearn.linear_model import LinearRegression
-from sklearn.tree import DecisionTreeRegressor
-
-
-data = np.loadtxt('src/Hudson_Bay.csv', delimiter=',', skiprows=1)
-x = data[:,0]
-y = data[:,1]
-line = np.linspace(1900,1920,1000,endpoint=False).reshape(-1,1)
-reg = DecisionTreeRegressor(min_samples_split=3).fit(x.reshape(-1,1),y.reshape(-1,1))
-plt.plot(line, reg.predict(line), label="decision tree")
-regline = LinearRegression().fit(x.reshape(-1,1),y.reshape(-1,1))
-plt.plot(line, regline.predict(line), label= "Linear Regression")
-plt.plot(x, y, label= "Linear Regression")
-plt.show()
-

-

-The similar code for linear regression in R reads (more details to come)
-

-
-
-
HudsonBay = read.csv("src/Hudson_Bay.csv",header=T)
-fix(HudsonBay)
-dim(HudsonBay)
-names(HudsonBay)
-plot(HudsonBay$Year, HudsonBay$Hares..x1000.)
-attach(HudsonBay)
-plot(Year, Hares..x1000.)
-plot(Year, Hares..x1000., col="red", varwidth=T, xlab="Years", ylab="Haresx 1000")
-summary(HudsonBay)
-summary(Hares..x1000.)
-library(MASS)
-library(ISLR)
-scatter.smooth(x=Year, y = Hares..x1000.)
-linearMod = lm(Hares..x1000. ~ Year)
-print(linearMod)
-summary(linearMod)
-plot(linearMod)
-confint(linearMod)
-predict(linearMod,data.frame(Year=c(1910,1914,1920)),interval="confidence")
-

-
-
Simulating financial transactions 

-
-

-The aim here is to simulate financial transactions among financial agents
-using Monte Carlo methods. The final goal is to extract a distribution of income  as function
-of the income \( m \).   From Pareto's work (V. Pareto, 1897) it is known from empirical studies
-that the higher end of the distribution of money follows a distribution 
-$$
-w_m\propto m^{-1-\alpha},
-$$
-
-with \( \alpha\in [1,2] \). We will here follow the analysis made by Patriarca and collaborators.
-
-

-Here we will study numerically the relation between the micro-dynamic relations among financial 
-agents and the  resulting macroscopic money distribution.
-
-

-We assume we have \( N \) agents that exchange money in pairs \( (i,j) \). We assume also that all agents
-start with the same amount of money \( m_0 > 0 \). At a given 'time step', we choose randomly a pair
-of agents \( (i,j) \) and let a transaction take place. This means that agent \( i \)'s money \( m_i \) changes
-to \( m_i' \) and similarly we have \( m_j\rightarrow m_j' \). 
-Money is conserved during a transaction, meaning that
-$$
-\begin{equation}
-  m_i+m_j=m_i'+m_j'.
-\label{eq:conserve}
-\end{equation}
-$$
-
-The change is done via a random reassignement (a random number) \( \epsilon \), meaning that
-
-$$
-\begin{equation*}
-m_i' = \epsilon(m_i+m_j),
-\end{equation*}
-$$
-
-leading to
-
-$$
-\begin{equation*}
-m_j'= (1-\epsilon)(m_i+m_j).
-\end{equation*}
-$$
-
-The number \( \epsilon \) is extracted from a uniform distribution.
-In this simple model, no agents are left with a debt, that is \( m\ge 0 \).
-Due to the conservation law above, one can show that the system relaxes toward an equilibrium
-state given by a Gibbs distribution
-
-$$
-\begin{equation*}
-w_m=\beta \exp{(-\beta m)},
-\end{equation*}
-$$
-
-with
-
-$$
-\begin{equation*}
-\beta = \frac{1}{\langle m\rangle},
-\end{equation*}
-$$
-
-and \( \langle m\rangle=\sum_i m_i/N=m_0 \), the average money.
-It means that after equilibrium has been reached that the majority of agents is left with a small
-number of money, while the number of richest agents, those with \( m \) larger than a specific value \( m' \),
-exponentially decreases with \( m' \).
-
-

-We assume that we have \( N=500 \) agents.   In each simulation, we need a sufficiently large number of transactions, say \( 10^7 \). Our aim is find the final equilibrium distribution \( w_m \). In order to do that we would need
-several runs of the above simulations, at least \( 10^3-10^4 \) runs (experiments).
-
-

-Our task is to first set up an algorithm which simulates the above transactions with an initial
-  amount \( m_0 \).
-  The challenge here is to figure out a Monte Carlo  simulation  based on the
-  above equations.
-  You will in particular need to make an algorithm which sets up a histogram as function of \( m \).
-  This histogram contains the number of times a value \( m \) is registered and represents
-  \( w_m\Delta m \). You will need to set up a value for the interval \( \Delta m \)  (typically \( 0.01-0.05 \)).
-  That means you need to account for the number of times you register an income in the interval
-  \( m,m+\Delta m \). The number of times you register this income, represents the value that enters the histogram.
-
-

-
-
-
#!/usr/bin/env python
+
# Common imports
 import numpy as np
-import matplotlib.mlab as mlab
+import pandas as pd
 import matplotlib.pyplot as plt
-import random
+import sklearn.linear_model as skl
+from sklearn.model_selection import train_test_split
+from sklearn.metrics import mean_squared_error, r2_score, mean_absolute_error
+import os
 
-# initialize the rng with a seed
-random.seed()
-# Hard coding of input parameters
-Agents  = 500
-MCcounts = 1000
-Transactions = 100000
-startMoney = 1.0
-Lambda = 0.0
-FinancialAgents = startMoney*np.ones(Agents)
-for i in range (1, MCcounts, 1):
-    for j in range (1, Transactions, 1):
-        agent_i = int(Agents*random.random())
-        agent_j = int(Agents*random.random())
-        epsilon = random.random()
-        if agent_i != agent_j:
-           m1 = Lambda*FinancialAgents[agent_i] + (1-Lambda)*epsilon*(FinancialAgents[agent_i] + FinancialAgents[agent_j])
-           m2 = Lambda*FinancialAgents[agent_j] + (1-Lambda)*(1-epsilon)*(FinancialAgents[agent_i] + FinancialAgents[agent_j])
-           FinancialAgents[agent_i] = m1
-           FinancialAgents[agent_j] = m2
+# Where to save the figures and data files
+PROJECT_ROOT_DIR = "Results"
+FIGURE_ID = "Results/FigureFiles"
+DATA_ID = "DataFiles/"
 
-# the histogram of the data
-n, bins, patches = plt.hist(FinancialAgents, 50, facecolor='green')
+if not os.path.exists(PROJECT_ROOT_DIR):
+    os.mkdir(PROJECT_ROOT_DIR)
 
-plt.xlabel('$x$')
-plt.ylabel('Distribution of wealth')
-plt.title(r'Money')
-plt.axis([0, 10, 0, 500])
-plt.grid(True)
-plt.show()
+if not os.path.exists(FIGURE_ID):
+    os.makedirs(FIGURE_ID)
+
+if not os.path.exists(DATA_ID):
+    os.makedirs(DATA_ID)
+
+def image_path(fig_id):
+    return os.path.join(FIGURE_ID, fig_id)
+
+def data_path(dat_id):
+    return os.path.join(DATA_ID, dat_id)
+
+def save_fig(fig_id):
+    plt.savefig(image_path(fig_id) + ".png", format='png')
+
+infile = open(data_path("MassEval2016.dat"),'r')
 

 

-We can then change our model to allow for a saving criterion, meaning that the agents save
-  a fraction \( \lambda \) of the money they have before the transaction is made. The final distribution will then no longer be given by Gibbs distribution. It could also include a taxation on financial transactions.
+Before we proceed, we define also a function for making our plots. You can obviously avoid this and simply set up various matplotlib commands every time you need them. You may however find it convenient to collect all such commands in one function and simply call this function. 
+

+
+
+
from pylab import plt, mpl
+plt.style.use('seaborn')
+mpl.rcParams['font.family'] = 'serif'
+
+def MakePlot(x,y, styles, labels, axlabels):
+    plt.figure(figsize=(10,6))
+    for i in range(len(x)):
+        plt.plot(x[i], y[i], styles[i], label = labels[i])
+        plt.xlabel(axlabels[0])
+        plt.ylabel(axlabels[1])
+    plt.legend(loc=0)
+

+

+Our next step is to read the data on experimental binding energies and
+reorganize them as functions of the mass number \( A \), the number of
+protons \( Z \) and neutrons \( N \) using pandas.  Before we do this it is
+always useful (unless you have a binary file or other types of compressed
+data) to actually open the file and simply take a look at it!
 
 

-  The conservation law of Eq. \eqref{eq:conserve} holds, but the money to be shared in a transaction between
-  agent \( i \) and agent \( j \) is now \( (1-\lambda)(m_i+m_j) \). This means that we have
+In particular, the program that outputs the final nuclear masses is written in Fortran with a specific format. It means that we need to figure out the format and which columns contain the data we are interested in. Pandas comes with a function that reads formatted output. After having admired the file, we are now ready to start massaging it with pandas. The file begins with some basic format information.
+

 
-$$
-\begin{equation*}
-  m_i' = \lambda m_i+\epsilon(1-\lambda)(m_i+m_j),
-  \end{equation*}
-$$
-
-  and
-
-$$
-\begin{equation*}
-  m_j' = \lambda m_j+(1-\epsilon)(1-\lambda)(m_i+m_j),
-  \end{equation*}
-$$
-
-  which can be written as
-
-$$
-\begin{equation*}
-  m_i'=m_i+\delta m
-  \end{equation*}
-$$
-
-  and
-
-$$
-\begin{equation*}
-  m_j'=m_j-\delta m,
-  \end{equation*}
-$$
-
-  with
-
-$$
-\begin{equation*}
-  \delta m=(1-\lambda)(\epsilon m_j-(1-\epsilon)m_i),
-  \end{equation*}
-$$
-
-  showing how money is conserved during a transaction.
-  Select values of \( \lambda =0.25,0.5 \) and \( \lambda=0.9 \) and try to extract the corresponding
-  equilibrium distributions and compare these with the Gibbs distribution. We will use this model to 
-extract a parametrization of the above curves, see for example Patriarca and collaborators.
-
-
Particle in one dimension and velocity distribution 

+
+
"""                                                                                                                         
+This is taken from the data file of the mass 2016 evaluation.                                                               
+All files are 3436 lines long with 124 character per line.                                                                  
+       Headers are 39 lines long.                                                                                           
+   col 1     :  Fortran character control: 1 = page feed  0 = line feed                                                     
+   format    :  a1,i3,i5,i5,i5,1x,a3,a4,1x,f13.5,f11.5,f11.3,f9.3,1x,a2,f11.3,f9.3,1x,i3,1x,f12.5,f11.5                     
+   These formats are reflected in the pandas widths variable below, see the statement                                       
+   widths=(1,3,5,5,5,1,3,4,1,13,11,11,9,1,2,11,9,1,3,1,12,11,1),                                                            
+   Pandas has also a variable header, with length 39 in this case.                                                          
+"""
+

+

+The data we are interested in are in columns 2, 3, 4 and 11, giving us
+the number of neutrons, protons, mass numbers and binding energies,
+respectively. We add also for the sake of completeness the element name. The data are in fixed-width formatted lines and we will
+covert them into the pandas DataFrame structure.
 
 

 
 
-
# Program to test the Metropolis algorithm with one particle at given temp in one dimension
-import numpy as np
-import matplotlib.mlab as mlab
-import matplotlib.pyplot as plt
-import random
-from math import sqrt, exp, log
-# initialize the rng with a seed
-random.seed()
-# Hard coding of input parameters
-MCcycles = 100000
-Temperature = 2.0
-beta = 1./Temperature
-InitialVelocity = -2.0
-CurrentVelocity = InitialVelocity
-Energy = 0.5*InitialVelocity*InitialVelocity
-VelocityRange = 10*sqrt(Temperature)
-VelocityStep = 2*VelocityRange/10.
-AverageEnergy = Energy
-AverageEnergy2 = Energy*Energy
-VelocityValues = np.zeros(MCcycles)
-# The Monte Carlo sampling with Metropolis starts here
-for i in range (1, MCcycles, 1):
-    TrialVelocity = CurrentVelocity + (2.0*random.random() - 1.0)*VelocityStep
-    EnergyChange = 0.5*(TrialVelocity*TrialVelocity -CurrentVelocity*CurrentVelocity);
-    if random.random() <= exp(-beta*EnergyChange):
-        CurrentVelocity = TrialVelocity
-        Energy += EnergyChange
-        VelocityValues[i] = CurrentVelocity
-    AverageEnergy += Energy
-    AverageEnergy2 += Energy*Energy
-#Final averages
-AverageEnergy = AverageEnergy/MCcycles
-AverageEnergy2 = AverageEnergy2/MCcycles
-Variance = AverageEnergy2 - AverageEnergy*AverageEnergy
-print(AverageEnergy, Variance)
-n, bins, patches = plt.hist(VelocityValues, 400, facecolor='green')
+
# Read the experimental data with Pandas
+Masses = pd.read_fwf(infile, usecols=(2,3,4,6,11),
+              names=('N', 'Z', 'A', 'Element', 'Ebinding'),
+              widths=(1,3,5,5,5,1,3,4,1,13,11,11,9,1,2,11,9,1,3,1,12,11,1),
+              header=39,
+              index_col=False)
 
-plt.xlabel('$v$')
-plt.ylabel('Velocity distribution P(v)')
-plt.title(r'Velocity histogram at $k_BT=2$')
-plt.axis([-5, 5, 0, 600])
-plt.grid(True)
+# Extrapolated values are indicated by '#' in place of the decimal place, so
+# the Ebinding column won't be numeric. Coerce to float and drop these entries.
+Masses['Ebinding'] = pd.to_numeric(Masses['Ebinding'], errors='coerce')
+Masses = Masses.dropna()
+# Convert from keV to MeV.
+Masses['Ebinding'] /= 1000
+
+# Group the DataFrame by nucleon number, A.
+Masses = Masses.groupby('A')
+# Find the rows of the grouped DataFrame with the maximum binding energy.
+Masses = Masses.apply(lambda t: t[t.Ebinding==t.Ebinding.max()])
+

+

+We have now read in the data, grouped them according to the variables we are interested in. 
+We see how easy it is to reorganize the data using pandas. If we
+were to do these operations in C/C++ or Fortran, we would have had to
+write various functions/subroutines which perform the above
+reorganizations for us.  Having reorganized the data, we can now start
+to make some simple fits using both the functionalities in numpy and
+Scikit-Learn afterwards.
+
+

+Now we define five variables which contain
+the number of nucleons \( A \), the number of protons \( Z \) and the number of neutrons \( N \), the element name and finally the energies themselves.
+

+
+
+
A = Masses['A']
+Z = Masses['Z']
+N = Masses['N']
+Element = Masses['Element']
+Energies = Masses['Ebinding']
+print(Masses)
+

+

+The next step, and we will define this mathematically later, is to set up the so-called design matrix. We will throughout call this matrix \( \boldsymbol{X} \).
+It has dimensionality \( p\times n \), where \( n \) is the number of data points and \( p \) are the so-called predictors. In our case here they are given by the number of polynomials in \( A \) we wish to include in the fit. 
+

+
+
+
# Now we set up the design matrix X
+X = np.zeros((len(A),5))
+X[:,0] = 1
+X[:,1] = A
+X[:,2] = A**(2.0/3.0)
+X[:,3] = A**(-1.0/3.0)
+X[:,4] = A**(-1.0)
+

+

+With scikitlearn we are now ready to use linear regression and fit our data.
+

+
+
+
clf = skl.LinearRegression().fit(X, Energies)
+fity = clf.predict(X)
+

+

+Pretty simple!  
+Now we can print measures of how our fit is doing, the coefficients from the fits and plot the final fit together with our data.
+

+
+
+
# The mean squared error                               
+print("Mean squared error: %.2f" % mean_squared_error(Energies, fity))
+# Explained variance score: 1 is perfect prediction                                 
+print('Variance score: %.2f' % r2_score(Energies, fity))
+# Mean absolute error                                                           
+print('Mean absolute error: %.2f' % mean_absolute_error(Energies, fity))
+print(clf.coef_, clf.intercept_)
+
+Masses['Eapprox']  = fity
+# Generate a plot comparing the experimental with the fitted values values.
+fig, ax = plt.subplots()
+ax.set_xlabel(r'$A = N + Z$')
+ax.set_ylabel(r'$E_\mathrm{bind}\,/\mathrm{MeV}$')
+ax.plot(Masses['A'], Masses['Ebinding'], alpha=0.7, lw=2,
+            label='Ame2016')
+ax.plot(Masses['A'], Masses['Eapprox'], alpha=0.7, lw=2, c='m',
+            label='Fit')
+ax.legend()
+save_fig("Masses2016")
 plt.show()
 

 
-
Random walk model 

+
Seeing the wood for the trees 

+
+

+As a teaser, let us now see how we can do this with decision trees using scikit-learn. Later we will switch to so-called random forests!
 
 

 
 
-
import numpy as np
-import matplotlib.pyplot as plt
-from sklearn.preprocessing import PolynomialFeatures
-from sklearn.linear_model import LinearRegression
-
-steps=250
-
-distance=0
-x=0
-distance_list=[]
-steps_list=[]
-while x<steps:
-    distance+=np.random.randint(-1,2)
-    distance_list.append(distance)
-    x+=1
-    steps_list.append(x)
-plt.plot(steps_list,distance_list, color='green', label="Random Walk Data")
-
-steps_list=np.asarray(steps_list)
-distance_list=np.asarray(distance_list)
-
-X=steps_list[:,np.newaxis]
-
-#Polynomial fits
-
-#Degree 2
-poly_features=PolynomialFeatures(degree=2, include_bias=False)
-X_poly=poly_features.fit_transform(X)
-
-lin_reg=LinearRegression()
-poly_fit=lin_reg.fit(X_poly,distance_list)
-b=lin_reg.coef_
-c=lin_reg.intercept_
-print ("2nd degree coefficients:")
-print ("zero power: ",c)
-print ("first power: ", b[0])
-print ("second power: ",b[1])
-
-z = np.arange(0, steps, .01)
-z_mod=b[1]*z**2+b[0]*z+c
-
-fit_mod=b[1]*X**2+b[0]*X+c
-plt.plot(z, z_mod, color='r', label="2nd Degree Fit")
-plt.title("Polynomial Regression")
-
-plt.xlabel("Steps")
-plt.ylabel("Distance")
-
-#Degree 10
-poly_features10=PolynomialFeatures(degree=10, include_bias=False)
-X_poly10=poly_features10.fit_transform(X)
-
-poly_fit10=lin_reg.fit(X_poly10,distance_list)
-
-y_plot=poly_fit10.predict(X_poly10)
-plt.plot(X, y_plot, color='black', label="10th Degree Fit")
-
-plt.legend()
-plt.show()
-
-
-#Decision Tree Regression
+
#Decision Tree Regression
 from sklearn.tree import DecisionTreeRegressor
-regr_1=DecisionTreeRegressor(max_depth=2)
-regr_2=DecisionTreeRegressor(max_depth=5)
-regr_3=DecisionTreeRegressor(max_depth=7)
-regr_1.fit(X, distance_list)
-regr_2.fit(X, distance_list)
-regr_3.fit(X, distance_list)
+regr_1=DecisionTreeRegressor(max_depth=5)
+regr_2=DecisionTreeRegressor(max_depth=7)
+regr_3=DecisionTreeRegressor(max_depth=9)
+regr_1.fit(X, Energies)
+regr_2.fit(X, Energies)
+regr_3.fit(X, Energies)
 
-X_test = np.arange(0.0, steps, 0.01)[:, np.newaxis]
-y_1 = regr_1.predict(X_test)
-y_2 = regr_2.predict(X_test)
-y_3=regr_3.predict(X_test)
 
+y_1 = regr_1.predict(X)
+y_2 = regr_2.predict(X)
+y_3=regr_3.predict(X)
+Masses['Eapprox'] = y_3
 # Plot the results
 plt.figure()
-plt.scatter(X, distance_list, s=2.5, c="black", label="data")
-plt.plot(X_test, y_1, color="red",
-         label="max_depth=2", linewidth=2)
-plt.plot(X_test, y_2, color="green", label="max_depth=5", linewidth=2)
-plt.plot(X_test, y_3, color="m", label="max_depth=7", linewidth=2)
+plt.plot(A, Energies, color="blue", label="Data", linewidth=2)
+plt.plot(A, y_1, color="red", label="max_depth=5", linewidth=2)
+plt.plot(A, y_2, color="green", label="max_depth=7", linewidth=2)
+plt.plot(A, y_3, color="m", label="max_depth=9", linewidth=2)
 
-plt.xlabel("Data")
-plt.ylabel("Darget")
+plt.xlabel("$A$")
+plt.ylabel("$E$[MeV]")
 plt.title("Decision Tree Regression")
 plt.legend()
+save_fig("Masses2016Trees")
+plt.show()
+print(Masses)
+print(np.mean( (Energies-y_1)**2))
+

+
+
And what about using neural networks? 

+
+The seaborn package allows us to visualize data in an efficient way. Note that we use scikit-learn's multi-layer perceptron (or feed forward neural network) 
+functionality.
+

+
+
+
from sklearn.neural_network import MLPRegressor
+from sklearn.metrics import accuracy_score
+import seaborn as sns
+
+X_train = X
+Y_train = Energies
+n_hidden_neurons = 100
+epochs = 100
+# store models for later use
+eta_vals = np.logspace(-5, 1, 7)
+lmbd_vals = np.logspace(-5, 1, 7)
+# store the models for later use
+DNN_scikit = np.zeros((len(eta_vals), len(lmbd_vals)), dtype=object)
+train_accuracy = np.zeros((len(eta_vals), len(lmbd_vals)))
+sns.set()
+for i, eta in enumerate(eta_vals):
+    for j, lmbd in enumerate(lmbd_vals):
+        dnn = MLPRegressor(hidden_layer_sizes=(n_hidden_neurons), activation='logistic',
+                            alpha=lmbd, learning_rate_init=eta, max_iter=epochs)
+        dnn.fit(X_train, Y_train)
+        DNN_scikit[i][j] = dnn
+        train_accuracy[i][j] = dnn.score(X_train, Y_train)
+
+fig, ax = plt.subplots(figsize = (10, 10))
+sns.heatmap(train_accuracy, annot=True, ax=ax, cmap="viridis")
+ax.set_title("Training Accuracy")
+ax.set_ylabel("$\eta$")
+ax.set_xlabel("$\lambda$")
 plt.show()
 

+
+
A first summary 

+
+

+The aim behind these introductory words was to present to you various
+Python libraries and their functionalities, in particular libraries like
+numpy, pandas, xarray and matplotlib and other that make our life much easier
+in handling various data sets and visualizing data.
+
+

+Furthermore,
+Scikit-Learn allows us with few lines of code to implement popular
+Machine Learning algorithms for supervised learning. Later we will meet Tensorflow, a powerful library for deep learning. 
+Now it is time to dive more into the details of various methods. We will start with linear regression and try to take a deeper look at what it entails.
+
 

 
 
diff --git a/doc/pub/How2ReadData/html/fig/pandas.jpg b/doc/pub/How2ReadData/html/fig/pandas.jpg
new file mode 100644
index 0000000000000000000000000000000000000000..4236914f8e0a657e3a4bf0e80f23f270988d440f
GIT binary patch
literal 103233
zcmbUIWpEul%rJ_W@ct=@PwI}>4Z5AGxG^EGeg77Ibm*?n}!-{H_yBK&F2!vE_OanHbEIT
zTP89NE;4x!hyUqL#?B$Y#VNqeMkcGIA;|Xc8p=k+#@XE&Xyfcc#>v74!6zZ71Oo%{
z2~Y||P_PgPT>R|ZY}~x&L{Rt;FbbN=vi~7aGMfJtuqxupQsVz%kevTR|Ky10e+cRt
za*B%oVQ~M0ssD$xwCZU^#3!@|80x^hkcJvS-foBZJa&a$y_Ym9BqL>TW2R0A}DW&s{g-rfQqE_
zAN!vo`;bYxS$--`fb4%PfXvK??EgU>WOV<_5a>~n{y#MF|JCq+s0I}Y6^RJy(~=-y
zApUJZNJ7BDz{0`8z`?=7!NbELAfY26AtEASe?db*|B8)=_Z1r#mw=d>lz@BL=>C6(e?t%$2vD5RXV8!o
z5KtJ9&=`>aMj(hj*@XV@@cn1{{{zr4ke~Js2aoVc#fOCWkNCex|CNwX&@iz7HX%@;
zAt9j9q0v8E9)Kl^OQr!Wz|Skf+s*-$R44En~$C{P_@4Ni|=kMBQs}PbN$|;gxQ)v>wuoK
zhf?pEiqNfO(Sh{#a{E4x6DV-HXgzjjGg;rcETx9uld!$XC{SJ2?I-sEtrwNM8W~-oB3k=?c
z|Ngd%WX$!NftgB@@zsg{CnBUS{j$_?Tbsp*a|5vk!!C`z=?V_|7%GZ7)oSx%DbchC
z0p?F{%!Ru*9e(8kA8UPPm5^AW$bf`})vd(3BCxG1Rn
zf>AZ_uRZm9#eoMAz{<&lo;uOTj_ru-gv@W_b$sk3L
z4Sa2#@=y>F8!Uy{xfb)On
z(EXtqO6mynP)t}1(*S>TcpOdoM#T17i=ggsJ(U?pce8KGjCrC(ZnZ9^0f8RqlUTr4
z#?=d3f^Syo3uL+8BR>-|veNFkCz(>$j$Usa)Ni~T;HLJGh*2FcxHH!LJ^D&Sa7BH;
zr+f538RV5-vgI}y;$qgSrntvudjBBUmE@EB+Oe@lnXa$FNx#uCnW?E)YtF6pUIyz@
zoPCI;k2Qi5yHw1nHHaM|Oeg82K}3FRW0B25y92y8dK8}BxomP5+UO`UhpZ3GgU}l^
z%p&H+3u?Up}$JJNbinW2Ugt3PFpqXop7;3e>v@cncl4{DXHJHri-H2
zF{S{lc*h0Rx80)^`rKAfM<0pg9Su*s7+y^Y!#DjViQ#zUW&*HXjI(>Yi4&ZmswL?~
zHBkAjsAO4qMA|Z<)3RXDLz8mFQUx;JlgQm`=rJ9O6
zNAn>q>8S6mK*yvb)Pv(Q?)%tPg@0tZK_GU2<_vuE9l;BPyO33Afg7>|ZR&A=b`F-hd?%cUXV6SrCdKxZZ51CK(CW4211(W*8GyT-#Y%^hGgu4Kky_{-9*K7!vK((VV>J^zh@
zumnSDXlQB502Pn@%J=g5`t<7NHu0GfIk7q^^l1_qZ3c#nmC3a>A0kWb17On2P8`YK
z-o4WI`D>yN-d-`cU28;=ldiYPKSVzO$5I`=FKe|#K0i$F-zGDf8WQreZy6Qc9|Ili
zZINwH#y0%Qgz}#;0>5`0$9J6yIG>NbWcHRQcS+;MJbu$S!^-%zb$5uSM7XC@Y03Hl
zI1(8;+?unc%#?}Vt#l8@GOI6P)|-}LLnXJ+l(bMJFXDeh?e&;CFaC1Rz~Wv@1;+xg
ztVj6*%A3NMpP|@vRI;+^Q-wj2(qvyZlc674MYepI3&F@HF(*4?TPvfLLDu8WJjup9
zgsV?B%R5juE6KG%p1*#1xk_d9Ia}QLMW`8B%~s5hr&80tnuMD7HPQkgi%d&tAkrb5
z^yW1j*fqM`6y6g?%JQg}6Mc44c8dO7b_YOSVVM)V_b08s%lYZK;z=WA+2ZHBksmjH
z>b=`DFB0x-zmsb{zs7wbp0jFCv$^iO`ZqG}!f3D8G5q5Gm)%CFEs?midQ)a$KCRQs
zb8C>AGk>o)BKZlrJE5T}%h+V9mq1IlZXaaDtPK-7cagY_pjeN=C2@_Oo4B>M#t(Ea
zHcKCMEv+SXi%#iuor(&}e7vgapA8(Ma!EC)b(Hke*IW*o!c<_b)$XLN>J>(C@tv}1
zEmFugsG>tE-&k;hr4OQ=k9;eb+PIjUBpA*3s4wQ!tytiQ><&igSeMlkTi>pvtTx@~%_tShUBh)v8Sv3Sb#%u%9!oB;=ay}oy
zKXboNyB^Zg7@<+XP)nK7%EriLZ{`kx{E0M*kp~=i|TXNrU9J8iFcL
zA6gE)GP)yB8f`A&hBb=~qr!`3p(3!79xzX{>Z^dLU!2*>`0>fs425p^7!*|qNZsRc
zv9*pWR9D;d=6ZLIyzGE`C>!UcaFn1`yF3@TDa1u`47L!D*A--T+)zsJ|
zSSuex7}g(EVBe+GL8U80*n(2LN}p?19qICReC0fK=d`0@_&%>f$(Sx$WU5|RgcYjE
zN=z^Z=4yHof~f3gy?6GGY*d98mMG@pDcFxxV!GJm;m#T!PCRsfoyy=keFR-#P|#Vc
zeP6mXolbu3vS2$_QF)k^hZ%BUu(a+-P)%!;D$A>-wn;f`%B+`Xg_hc$J(5XK%c`7U
zQ3w+TirQ1Ebwl&7(!^`gD_5EReA=Tt-d&A*?6!Wzu_wawXK(lBzV-8Yvx!vah)3qN
zIi}7@F17g*;LfNIV-)>^=|p$)*g%j~i8v;Sid!gR!aUOC{06P~#@}mu`jJbFo-HZ~
z;}u}l7PEAx
znNqZRNKLJrMdjHtG`bjp?z|e4hvle=QnN!-dTk1H>A&}&y##mG?!%Rm&VlxVv)YP0KzF~R^w{Pqx(;>2H#W9~oR{+Y=xs_EGE>qb#rnICjM&x}_7E`&8
zR=%4p7DvT=DXw!_lNOf0Y%Ql{sQQ^PJ#T+${_(BBcQ&jb>Q|7Ce~{2!f!a(dh0{Lp
zJU8;(|A|$h*-x=QF}W2@A#pZ!aeo9BYmHxJ@U%*(qwM~1VHS_bEDN8$#?zYLj@n*U
zcw3ddSD1v*%y{7Bhp>zO(nyuCp*fEtY+bzw@eK=nfB3zyk>AS1M1L|i^R@E#+BvHn
zgQ28>jm8E9j8OsmIx5&?l$k!hX@Q~=~l{)({zSF};Cf=4pKW+W+mC`)+0%urn8(biv(X_++q@bi)3R$F?K4XS
zd}PX?C2iyCKhMkk9FF}@<*p(%%RUA9I1j9X_YHO=JBc`#)XT8{AVf}QB9gWf76unn
zp@`YmddB9Q*mq|h`l-%dt(*SvE{3j}j=2SDI{O@Lp;eA6&I$?K2w8z^PN>V0>1_+y
z#E7({kIrGzii@KRKspwVXY_pC3;_@mIeL#Mk9m5VF=udNT<@ct4USERU$l$w^0SpI
z&H;DxZkOP*pBMyZ#aT6{T+Py_e~1-4qq;3T7Xflx<`8ARqy7hkG@eOaJk_(uwQuYr
zPjo80(g+Gia89PvHk)t;m`b7%=jv1SDLIhc_e%LW6gx<5<^Ipq3m4n78o8GSr(=CzBjq$v-W!-r43=IT2S
zxIU!CZ&2k`4HNVCffJDT4y4uZ-qEm7H@*zBc|l(_`gVg-Dm$8?OIKUC!^e3lxBGAG
zCsly6kcDDdsQWPD8*xUCD%D?(ju&p~6%4kzxtw5%R~*A5}pEG
z%IR1nQ^as$sA?3V6GmB9uI=P$f{5FqJR-q_&^kF2TxFp=mpp46g#sFAnpo`T6iv`8
z$dX<}D>#9X_l|2FKj3tXrGHP6{m6RUE=U3T5w{dYCmf{!TA@TLx`=mBZqB4NZRGwK
z=LaC4))p$lbzb`oyyOYb>AbfE5Wt)&_vp@PwF@ntG4*JUkHLmBm0EPBofMVMP_mkE
z7|%DdbVtrNt>d+n#9>I{pa$1QM6Flk`@kCxL`&zQ#&&^x>O|?!*m$~A^2_o)xhbsi
z99IhOUgnyFcS-YBPHQjQf8SpCe<@q4e08le+aDX#I^{lGD62jDiHlXJ>Z@lbQbO_G
zf-tni7?&`?Xo#L<7;$c234;(Rt=}OuHwCNC8_R*LhOA1FKtAi>D(K|UXuYmcq7Im3
za`(&ij$+&~GZC9>MNJt99A^HQ+s{h)>S5wpEB5U*YU0oHj*xhn27kRr
zj-Saf%aKDLU?NeHw#e@4Wp-#*mva-kq??!);VDO|m_}-c|F8NZBQTJ)8fPXd{THd+
zwf0!6@N?GnIAX@wez)1e*aqhxKKl^R*8mo_F0sPz<4YQ_CId#jqsk(@)pC2FYlp|`U+*~iEr=`K(p|e*YyH}am&iJc7nU2Tr`ZbZ!+n=j@D{h46nm-cpo03?5V^bcHu{X)%$e7oOI
z4|kBs6czv@`3@I*B4r9KylRF{33AO_=CVdXwvytrW9Ew+|J6P`U3&HVo>0o=aVe#H
znh@8gRAIgFPJfsSNoJfv+>%Pn*5E!{NMA3={*rs=mQ(#Cgeq=Xl{R~^UL*Co{go|FQR)LYVw@TK5^2dudazbyLzTL1bd
z&}|bq`vIOn1}^!8I3|>m@HVY)Vhk7X1D5G_vUIU|;FhgqvPuQ!g$HCb
zO`FNGyee%+B>Y!l0C#f+CP~k9?&&MWLCvN#d3<+lWe0MY4N(Z03vSGSq88MmuC-oD
zbdHO|hyC$C2+5eWcE_BhuzmdMB6!9E8(AHlu2B`m`=+l%IOk2jR_lSGQsAl$DZB4ksH?e)sLgB}0n8
zwZ6XvaG%Y`xvUhzr)8u@V^k!k_h%TL7XI$iaRnO}fi$VLIrChV^S`f`3++Ic;M1(t
zZgxmLR)Xs_b1Eu-`#lTb*HP0L_x&ch>FPsETOI>FOlIlUwgxCGUd9X`_lC5l(XZVz
zsb)`HLY`-Tty`Xsi`JLqh~{4a*V%7o$BBZLtQfrJ@&uA4NW2vITe}jfrVh4BieLGI
zECG%a5#8;0L%9gkR;?By*2xb)a!#}VsOB6g=2wP7T#-eWsTNdToami-fUM7#_A_G2
z=VB$oxqjx!8okuFwZ2Cd;>fJXsp>P$r7gRXR@*73
zKatc6)i6j<6!f^Ob={U-Z*Y;<3qqOp>yo8370@b&!VGt7B~~4F`_j=mi&l|sTN+zw
z#-}LY4n!@Vwd_*+!5*bi3RujUuxEDhL$CRwFE;1CF`Q8V$*@$5lB&yKyV}6h!y&}O
zv!7R~=))I%zyqtJv4yy)mFlfcm~VR3jdSw43OnxT+wphh_g)o^_=HTwyeqeK?5)sz
zCGH7Upeg@7x63Te3QP^Fo?7UZ+r73pPn?aNK_0KBj}|dfo?@EKEO!pRVSNDg$Iv}^
zv#@tkm>c%qAkp-`gjkz>=uk4G)~Y`V8)5DtDi$gL(|qm}n#}YNjUhobrlCZ4De+Vv
z)gv?|6@b8`X9E4L1jIxEQJ)24{=9{8$x9M9{
z#gBfZk%NCtD;vg*+?0gLK~Tn{YiES+p5jDfm%t{EQc^J(>&PzHqDz_eIQ=pm7tGNh$VzXnc6oQxQc
z3(Xo{qKlTINHrxw8r$C4I>JO^XWf)lKHoY?*(Lt_iQ)8!Z#hWiOvtN#>7^DK3p0T;
zx#|(0ZfBM$T6IB@{g>=VKY=mx5uR7&lM+^QjtrL>B)7mLl2xuK!8|KALVN`6<>^3!
zH4~DntjuaQrtdA8#k*mv8-;h?M%-wPfBxq05k87$HLtC^m}*%4{7^WZ%fIh_q?%(6
zF5D=l#%j?vrg#ztu?pbADzX*X+RZ@XG0-6@2)mvZ_t-z&X89$(vRjZDGoSV2!abzf
z7(LHjzd!Pix;tkdY*^11sUfos<${+!Y|V9kIyovJV0fF*jl%tCpTB0MgOegn|J*mGotXO|@AD1K(>ebPS)J6$p6trf2DA~H(_z01QdHzp#;0NuEMuKd
zl!4~D)-;k9+TyT~H9Rb6^eovGF$XQjQuEmtku=}ggtTRylz!9AY0Bo%Jw
z2;o1M+sbx+ys7mM0t$#WqGo1zeCdA#??ov^toDiVtD&O&J?*9*Br_dPtO-n*V;qjXV>`&o_
zne0l--Fg=C2cvHfd+0*Ccr3~igC-hTSgNeNQGKx3QD~RF=CiM8)MuA~DP(8%xxlFG
zcwfi{0iUau^5;wSN@{!Zy7S(Iqn|BW{P|vrTCO&oxVOwf(qZ5=zVKi@#<%fy
zPc0F%jkE)6W4WMXt%b)wx|bmXts&odMy*|u`7T!JiQda39bpN;E|f;Gb0~|^>bmm{y8mlJ^#r@L0XbR-IpJJ
zs-ooAc#Y=X{~(GP5)<%e%qdX5`=64K$Oo1!Hw^%N$S;hV{Dj&$F8Y;bx*Q8)6Vu&lbY19q!B2jdFL-o
z5rjk#BwN!hy$1bKL0uqlBqe_UeQk(KfxNMgk#WKi+w|_0ZImPaI=!ho!$&wpVzv5}
zV%Z2u`U1PMb_+vJ>X8Y{C^%*}uFVPqoYUc~d8&Ax7BMJfag{5vKt|h=CZT?=sjY>M
zSb44dGpG1eWkN_%zV_F+-J`Ct@o9kE^8JXR<^{uMoXz(Yw2D_n;xPWZ#Ku1d@3*Tj
zw<@Qj0l*VuIYm+02RsvxFD?1s7tQ5U>ZIJmvkkkl^Ro??Ct`Mke-Kt3+rX^8J_}Wu
zqHk{ppqxh|w(9VKJ9Fxc)C;UkLo(LvS{A7%739edQ|rsE8J~19RSwWJWBAb9GwGCy){Hn(E`I4xq^_pRAcZ
z#rFNW@RE1%L8vfcD|dE?uLb@R7#u*!J4Gcb3Qzzsfj;8kgptxH>nRahc@#5EqD|&<
zu%KgfMWTam5{Ps4hA%KPDPrMPikmr7I%szEk*uiHeug9TouM#I3}D%nDj+Q>N=qiR
zIMnxBSeI-^(_AOlZL$gnDVH^HopwrRg)Nx^m>rsl$k}DRzjE0{>>GWBSF#j|3a0k0
zWR@i#;?qw^aPm$0+F3@Uog2)>8ME6@XhjlKG*B&k;|WCS@|>fx7E}Z^ZGeC08VwPR
zQwSFEokwUfLsL9r73w^Cfv$H7ji_n12;7C
zeO}}&C+rYD1K0x>OvwR#=%ji0FuMf2Z$z+{dL4naUp|zfq*Pd4CUD`&?4&VvbSeTX
zV2|8lJ@}aIzF~z=^f|iV=-_D-CvQ?fJ;6W=^%gEMB5l*-n`Ow#YT{lDjLSdvxDP1R
zSjZOMzoB9ndJrA9tSYhJ9HAI4VDiv$`!3|sjYE_#Oo3>mPx_E{bdVp(SivX8k%JXG
zYhki3Vs8#6vi55dzO&%)79NXBxa9tvhLZlCidli3bGOZx9mp
zZ*DC8I+=GS-Mx6hH$?Mpf<50ExA(k&ls%z6UAPJRxn8h{+eA+1Y9jkYUc8Ij7{s&B
zUKG}yf<3_*H|aVzM6xc-JK$;J$?d#Jjey}t?wd{kXD8mu4JoL>&9cEg{pRS)$4t&M
zjD8n}@F4p(vC-#}L|!gIpII*K=nzm)P>?W?&@i7R6!d>dE)dW#7?`l=Smf;3WE7Na
z9O7`CI8Xj~u03|1cLo$3SU3(3EIxH)rTMBV0oU{aVsBeIE==4Vp367~vW
zRc0x&*RbegC=r5AOLvz^oJl?tWO@(#EtgFD|E;F*!8+=vBpBgd
z7oGtgOxRGobtD#kC%`E&E9Lvn!wD$&W{<;59oGX?ONCY{)~;~JLW07mB+=}C_Aq_D
z@qQ9A8I~AKWF`o_A&+daL-pE*1IF#VcrJnz@9;FDE4$;zY?L`w`?Y`CcT_nkaVsX&
z)^G@8JRyQf8w7jsw4X
zPkDuhnOnxfyExwXq6d9QHiMO`j2Rk*Ev+?w_WD2g@zVefm9V*=G{wp|jMF^tcEIPS
z#^3rq{6*~PSDN@;bYps(gYYlg%i6W|KvV8+JD)F`Vogfy(_5qSaIS!|a9mNd}n54vEj>g~X
z(^lfXEck-Ko(NrevToCIe8ep8%b5#|uX?^;^z*Zq|Hf3MFE{;0dza>J_=uxV^ffB!
zem}Q96qbO9@xw1dxcLVm>it5T;~_}?XSgP}&PC1R(DtY8m>wvps)Vun@P&P5pkp*3
z_PDC^zzj64bC_?x%6LF`=*eS=7M>iRl8ybyQgEwSL2wr!CeNzph7%fLDHo(#Mm9Ck
z&#LT)s-v+651Vj@o$AS8vPsosPPj
znM(9qf2Y1639Z^UZBjDhm=$6#Q#7kTMw_9)?IWuh^QLiEd#oso^8PVSunwVkf$91v
zaC01?Ya`yrT!052yd8t#$F;3`J<{Hl8n>bw>U*>{X5YS#wf+2r))d*8w8jS@IuN4q
zM)`$5WXHEK>Eu;;wC;V(sAlz9qelL21V!SPOu2fI}f
z9!;q|QD{DBrUvoy!}dZGfcZKCBMrBdSOhGH%ve!m97}~Q7aiR)pB?*+lvtZJwJGb7
zuzs}6rlLpYkwGv*s8KPT@ACmJ3lU`(^SQS?Hbqo
zkw8gX)w{cI7Bx}2*Aux4C)SNHa`L=0P%=3yjd`uFf)sPQoCVeWX&Wox`*&f4-7OPk
zb@0NKguqI54G8jkXZ(oZG`0{|FsF?qQz-($NWO);aIMV`k(0M8v1Jh1Y<3Gz;Hu+S
z$3boplN6*ZYhh&$q;vZ26;^|y<<*B1hv^CXF
zG5=Nv*8~wLr#lVi^}F&pG;=Oe_dwzEK$8h-iP11&5~Hq^iVg_@io
zHy2dMj_>g&`_EuyujeZ$04I+6u~w6?2dTzNiG+~)EneS2-z{ufr@drW8J!8st){m<
zpgW;24xN{3w37r=gTO$a^rC`VV=7VbBPO2k!avs
zM`d3ZF+5w>W>eUFKq5Vu&}(0FZh#xE=Zm08Fg)0vmw@E@t{{xWy#^_+VVWq2Nl?h^
zetRQ_&ADAOU*O6yX9PZGQF3>E{Zir|gcZx<3jmn>eS!(Y=qnBGBVz7H-oQdroMP7`
z*QzN=i#_NxE2Z7(3o{|HBHxkp@2WZ@90GejPWCAioUgZRZy?&JVO{j-tnCI)SfOn0
z;Hu5bjI!Cw%9OuVsX*<5r1-GEy0(lQIaf2WuaYc6=Z>F`FF}_Vc6Rpd75G*4D{CzW
zm={qXYdqN^?yE1`eXIP0k&r>hP6XZ;B>Qi1Cg#w>^FY<6@ud1O!$8KrXbP6ysd|;U
z1$$~O&cCO)*_&qPl@!#*IMYxsr-d53azl^MWWlHWUVDynssW?5wlxHde9xV)#z#Le
zF16r%AS_#@n~`a=q9y)vtS!uR;tWq)VsG}2uIN5Nc2yN|(PC%VD2>}D?Ung_7`~qW
zc!}M7Lt@pMX@29Jn_V1$z$
zRf;SE_nzJ?yZbbArT~SmEh|#>>hl!=J7YrJ$L6>o$kBBYfOe}EnwR4Is}~BwIwk
zWko@k6Y8JQFaPHau3?al3)*6?MI$=$)#Igj>=9+#b_}TlPUzK}9=wQ^oi8`#mJ6!j
zpBXmI|CZtRTf4@=utwQ1`qA_V9FN#F1=O9i)}B4u!>TysQ3nxu))2yz68E$y5Lm2c
zBCRws>3gB(qXW&Zkyjyqv74-NvBQHpZ{DYVLiuy?y
z0Z<8BZf@?C;t|=QDBhhEVLW63h9Cb!#cy>Unvv}jh`HGh;Ta)ddce1~3Ug#vCd+m^
z5hXv7LzczF;JP=QJiU_m)2
zFggG$$k4xE<<0XFch!IvC-n;@SFNBBPNp
z@9WFV^GAE*_#KMPv%yJA?&{4xi63M6G_60DETyJGnX;`U(kI+zEaBsI*5a+`^#i7>
z71J41t2$S#c;t^Y=VjA&qbCnZ>12Mm^0o({Z2h77lU6Wx`*^5l=kKTW8jKYMl8FL)
zJ|BT%GaiTr{8)-y`A3`KwvWhcYEk++%urFsMV(Gg-hA_m@jL5zn|PXH=E>p%zXz5T
zFVM1!;z7g|yM`XUyXKdVwbSfxi&XGWxrMrhx#H@4Lo??yNIHaO9do=Oz}cGg!GIoi
z_v@0^7ziAF=#$*81a`L_8qj!%l+%a21}o>Ba^A5ME8ur!_=CU8x+0c()KeKE!1sv&tqRP5y`q?A{!RgU~IknMV
zgDWdr`WiM}=iJtVW?VP969Fo%R-$+H4iPgr9bws0v%zBwottnYJ-5aB5F=)_Sb$9P=JLy6VZRzivg=TJ@y
z@TOf;P7V@M>$${XY^a0Sh+{Y)b1f9-Aq)K@G6`}l=>%>>F&49?T<~AT{9wWyPT3Yl
z{1<#Tg8K8!mnK@x5q`$#B~3y`%3;wXAs|bB*3TLjwl%0Nf;EoQY&72XPx%=CL68If
zK_ryBx37RUJ;*8;$+?E?T(=>rN@(K(5H{PG)Z&E)&7+T+F#)b3*3U6kvTq}m=QU39
zFpS*7W9Ix>Hw!w;6H>p~KOc~XVao|kn<+yW_ZKm{(tqBq4NGTm)~?450kJ?|tS0kM
zHi0$@Qs!T`s|d|A#_^2$P8$yl$!OzbRLe~$iN|8MRH8rc%5XVsH(y-bVtKLd{UaE8
zIhjwb>#y&Bxo%BsU<8Z{S$e(R5Buax@;hn!pj-?#U;IvpOg-r2t(!Ko;@RNaP;lRt
zBqfL`b9{Yj?K^SI5r^6MP5T#Cl$i6QofpR=tyYJFQ)+i2
zURV+$1m-6VKL_TmVLHqtsb(rY)!6=oHjHRd;0=$jGr((ey{
zk{90bNTrwr<{C0xq*1C?p7;f3sCS(f9PIi
z;h;J|8q-Y!)c~}tf|{vD{z1sU=I`f=(b7+EWor}jj>Ufn^oAq6m&x=dkK
z4RQPB^IKZ`f?&0%JksH
zYq0cN7&$UDar47QtYj7ZdB9&5-AUT3`NaZm6o(;xM2PY1n$7C|ztJ2fRnN9CrrMH_pgZagJVlU`^`pWh+|ARnYe>I^lMj(4rJlh~mx-yN&
z8|N6y`0HsE#78Q$DLTsv_&CuBi#69
zS5@v-uukG#rft&%nesB0+K!bpfo9aNQQ+J_2S?=PuM273X@P~Q7$WZTEk3i>Njmq1{UNraOV|KpEIv|>Np+}w~E
z9z@IUgtVfjGxzu>`8H15r~LGxVe4UHFo%4@2v;?imA+;sM1o3lZZ1kO?PR!`aB6N?z_bqB9LvD
z!d`K?v%w3|*OfmtZ*~J~RToZ9zPJ5PSp`@wty|(2$B<#O^V!y;=G$OKw5m>rhzWh5
zKFFH1@o6KvS
zHi7zV=KlDjK}W&EFUQO>)Ui&Zh1gKK{V=CaA);Kt2~6PWBrgZ=msoW*y3s(of@f`?
z|D&{Vn0+nPUy><)^T#oo`S0C>$0w&RCyY4U)`7fMg5nuNxJ+EXhZa`;FpVi?w5j;K
zw`<3i*$r5zDVgOAXEb&mB5r6c9Cyt`5eAu>P^^cg79B}4y#Nkj{WHHCT3HUhK?={i
z!eERIz1j$px~95~X(GCHteX^UDI4}%%JI$7Kt6eIVnYRT)U4)IkJ}sA;_M32Fj#sp
zvYL&|c$5oM}^QXKe>#mJLsD27wph0w2ydx6EfI(l=u
zWPMBxn8We`uJvX|C^4~$egvKN=6E={I@Nk`x}CWZL039-)uWAAS2J;yqDfj=UNlkL
z3-P#iye{tcuJypA8xQ^=xkSdgS_`TA_72tSqm@gN`l2+q`e46*5I8kpPw)>qs$ZJP
z#(J;vnNU}l&ljKnj@Mqb-<%%}XTAIRtymA8Su39K0Ah&;W?>~K6j1o&808#y^~1zP
z!e?qr)v7eY*5uuV#|gBL_e>=F*B{R90xu2q-D$enhZ(0%!u5B7mc~O){>nnTVbxiy
zoN2BiwRF6=LPhKtSMF#1I=(d6{~+S()X!?a4K$bq_!!~|3Nkj2HRS+pX4}rSHQ3yH
zIvb12>LQ4rqug
zKp6&J5(()3t?#6&>S^29+so%^3T{oVkt*~4dfE}{*ZrDvvY-G5Tkf^LL5_7*Efr?
z%P?3KBPYHr+I8PHuBeN<_)xuW@9jxC@W>12-pw?VA>ch+aj<)r@?q=&H_-O9>>t_(q=
zbr9J)?@{X)#ME@{aa+D}9bYRR1z65TiG2;Tb&0(&*c%!bT26#>D1t8v%S!Ij9cxrtYnODmS<{~!`sV=Et?2V3ANrbsdO
zDoZ$t9q_dX66ehV_wKG*8Ns$_PR>*$`Y>bt9m2f0wY70I(s+LqIlDWIhbEhE?`ric
z6*sYH3NM#;_qd;r)TU!MV|)st`X@=+;OEN2Lhd<43M518$o~Jj5hwPp~TkJ
zgoJeB4Mzm>Xu5-oLS9`_C}dfX;|qWlVnaXPCf&TPsa76N`n{y#C-!J`t2ATC=9;+5%H<%^VoV$nzZ2lFd);J!O$0!
zEn9*WmSktApPucYnafWij$(Pjj?$*
z)dL?fSzS*fx5N-9Hm7Q#_4ShP=jK_%;=2Lru$;CggF?}#YkgEG+g1W+lcqBp|2<7H
zuFEn!`o-pH)LQq5qrs8?LTBLcr;AS4dg%@zXQl}t@7mb9H7;1J4@f+Qfu(KrM*WmuSi$0+}u|_
z-B*uCD88Zh2X&v{uFvy6J2zo5tCw!`T{D0qhrVgPABzGbF2*VFPGbqS2putcliO>R2=
zIKRpW$hN0K!x;7T8$4J>e~)D%V_jd9sb^&`O2W(QIak8W#|E#Yr=?{Jt|!&4GG9Y`
zJBPB?%?>5vA4Xv|jf=_hvPBpq%5F8;h1FK|d!skJq^l;oG5EC%5@uxqHrW=BK=x6&
z6C-eZ$iYP>q<
zQZJ^PUYjtwm~b*do_N;BvRd-PGMn-^^t=m^j#n@-^48F=DOze*z)OC5$NoMG?vohm
zJ(?dU`#ab(l-wUHu0#V7IXhgHo9*Ikb73=cefJN*)T>%E$i=K}#%Wc2r!loYw>K_2
zuMmD+m)lajD(W@)Fj)vV__to&FB-f#nkmU!FmRS!n6(Vd?(0uA1u_mf}%t$mkfhN9a#XxHSy+7bmwpN_vfz#Of88*=R%L)+M>In7d=tD^9{G
zgZQ*>g1U%CnYA~#%N0abg=1(y`;~~0x4vGP%<>Yf+lWU89d*^q_u7PS$pmQ@JU0gP0Rvc~)sOs9TAXArTD`&7GE{F74|ytroTW
z7|hGXG-O)WT1?+!KI~OqTi;{Z0iObV%Od)PhLc7e?OG%<2jMNg$aTGEedAZpU2qk_jGS0?y
z*W;Oy{>weqD}g5lp4JBRymXd}jbF~g{#Ms*_C%p
z>Gj~%_nH1mzoQH@0FNTmMn!jYl!vD3HVUkCP-(|vZmD(i6#J>UAa2ZiEVJC@=*rAv
z?!rVw+;SFgWoGB}PhfoyaYzyKYe;jLnsPaVWIrd={+u(U@o%i+wx+=r+Wh(MZVAJ3lp@U0`iDY(5Zpu-
zshJyKk(eS9T5{BJ%2KchwH{n2xnmI#JF%OG*{{@XlD}Y|#G?k5S)+vh8vLThveM$~
zdDYp6AP|Z9ebu3{x-U%_mHyG6>V)r$Vw-N3*?cLe^lVj*eAL?D4San`Ws^zFTPTev3ucHFtY6@m}I5*}S&jMjf3
zX5$z?k(>*;$P172T{pyme&RT`NT`RfSb4ZbMUZl;7XEzqFp+KN#I^Jz|=>lQ%cmpL26?n8pgr
zMy3xLL{=qS$0O@V#N>u_H6Jf$^4~1I!zT#)S_iEn8Wg5x-(ysY()1I
z31uOp)y6vUn}E>KvDKrgL;nEv+}sv!8&zRDpWpURV=MO6Kk24HN&43gHKe(~YY!%i
zDbH4wWlyNxl;nEmjSY3Qd|8q=b(Q(wAO5|V{{T)`KKk+K`nTrStH(l)M;dwVD)x8v
zu2B)}MqwX8Uip=KF4L
zZ|taTzsIrl_83ocdlLzkWY&_*$K8K{`pVvA<&F(ym&);iLg6LZ0yTxARk-C=Y`Fj=
zvH2%taZ!VS)jJ=e8&(rqmaRSULIa6gRjgi$x
z@iVzUQOffvim`6|SeIp8LkmS+#Sr=q<)BH%V|s6a&s9{uOLK7ic5rR@eBE
z>QTk{*I?Ypvd!*pZ|&b>H#avo_6dkK{Wx&PYD~cl$9LR-P&GOA@?A=~70
zyM95fkEyLv>*d$`N}ke`rBmwooi(v*N~M-hn7!Y%9E}>>$vbM!pB$=NZv%fJkX6kW
z<-Lt(nU)*a&U4KYzrwWREFM8>uJs*%E*?3309%ilU2T#n$fs_DlpABgfC6!c$yI~YW|e?pqtMiQeQ1~lp@hQGPrqzd
zjRw}W8^$p(`ilzgOnmD6RnY8O$>nv8bF>qEw>S14$Myv2zwNE3I2s0=p=e$;qAHdH
zDQlT|yn~jqGOO5Ia>w^&g&+@Us~u83%7N@_O5+_hfg>?u1k?lHXZ|w?fr*Z)mF4J_
z9GcvJrd06>kU2r_awNaxTjX(zU0Sd5eM<+YACvra{9mle%{s}V-WIsx*k_Hq(^#OQ
zuIG+ZMM$-dF;OF9qsGRrEeIg1V?wpI*|#99+}zwI7^BxAKgVpY>^)1WRvkyCErIJ`
z;WalmxT^q*XBlH{l_r3UVKd~F_7+lvvsRTTU7rb#!GNoqEqLeQnPdaWIVNtl!#hq@J6Ok~30G|<_aV9EFMrcg=ILNW-Ce_Ar6^M)N
zykf9_0Zjtd$hXFi$DS5arB`(adoma>s9DR#49&({THIS}HderX-lmMy6CL!`-RT~A
zCyl~@IX+6)FUp^*kocWF5v;Pc8Mn)>mYzzg$Q$FAHA?)hmE+^3okQKW7hylX``3F<
zZlB46AgS^XIgO3x4`{^4#_DzH$TsR%###QWp1oGf2$cv)+g$5BuE73L*2M?68T?^*
zeZ{D1ik+U+6)Ln-6zqG7S_U%JZXUsL#PS;7ew~QfnHbcq70p8$G1O1aTT-(19+}YdzRyH>{=I$-IZk(q+9Aykz-jk7L!__0D1nF|$8Sap1ZP&4dX%5lQL*D$;#p#7jcip>U#F@JHG>S#
z9V;w2;5X$fZUyWu0z!sxGl>K&3oFSUB*+0@EAn=HsZKR@#<&4yR)yJ9^R%S^0?L7}
zW{Fc6eMY-unVQ@JYHPCyJ|>LeJS{wNfD_z6z^D@`!qzJ-844VGlAvznMXL>>7Tjs5
zyN~f!O_jlxh8w9f*8R&zu%kkdV&ekePfA3zR>sD8^o+&F1*;`#7F=#MqLJYfi3Zc4
zhf8Dfj;Csyo9)r4^JgcaH!`y8$4t(>V;}h}qTMeovazW;9b2h=BGJ@7ro6ySwSOCq
z+QKeBLmX@s0yoAwmb1mPRXFmaijwTA=OQ7q
zaIt~eN@W>AYi9}&-pbL&p?_-Kg#Q2;*PM2=@lK_X_ubJG5LKU>Mquf<#}^XYg<`yF
zI)$|l9%bLjp@h$vsVE^3{Oi<5DJ#Y{uO(HrefGNE$~hjBh%vFdh-X{MTxEQl7}3|ID=fX9r)Ox_J=pJwO?TY+iC*9lE=tnNc^{ct0^d%8aa9I3y9Nm;D%IG9
zoce@rYJ|pN?Pk3y13Y6^)9j@9?_bOy5NS!ota!6gV?*+wn5noaUB8+YuoY)BfzwcD
z9e4i#8f|5RIYuCi{*h;rj+1FUKc>@$EaN(b0jo}QH3RkU5!x}_192*WXT;X!6{=R`
zGk^;oE
z=#4XiOKr~s1#r|S8RHX#B71<%`gK=^y~4UYwF{FqH#x+tv`lYFNE5~caG+UjQ_Ec1
zH)8SV{b@K;n6QtH8`Hc`IOS`-%PcymbISDA|FL{025H4@6oOmW~)uceN4>drFt%H
zciq=~je~>MUX{D4jEMMuQB8Dsjm|g|xm8VJ@dWSlD)i0dy2G_0j{J^bi8nHzmA#D)
z*todwk$y68su`J|jf~Y=uDxV2r`Zmiu=+kH)LQfkUmn&qWL;Ee^`1Vl7D-}I6EM#Nz^_7xm
zR>gJ16&XiWbW>J4+f6}3KNZNx{M?o*o0V->Q;gICRwG;x9qd40Fh_1EetUn6fY`#M
zS@GMgbrFYM7`YkE{yNOAv`(jwKpDXe7|hiHWx5tWkeRr!kIS<|1pd8(%v-wB);CN2
zO1hpeg5t1=sjqdRbC?yDoZrI_9r}fX#f@G^Qyc#Pm?Bfu>b764WDhE1CY(ERZ_=%d
z_=9^VzbLw9b6h9cO?LMmjtk8bkQxT>o{2rjcQ~IYR6sXXVdjKzF!+yeALlO*B;P<
z{GMHxqT(8iG1HF`*-&5CW
z@vO;fIzg$|8oFUM2&0_UqIoP*mj{pTG||`)jl?BmzUR0}if1S-ocA6Q)AF2Wh{at8
z8YucI+Z?AjZ=P9i78YaH(LC7YGyNx1Z9aKQpSe{ib8L8$dki+Lg^p;adfksXehUtjRw`iH4b3tV3O?U+Y_giH56oUN$(ttCyB#
zSkbIIEjOsPu!&%3phA<3Z3xG3dW6l*L+~{WOnJ$5qQBns7|rGcC{jk)|`
zZT4jQ2W(r#H+kPFtx=C(Ex`cnPNKKw&Rm7uT@!1TIXe&BdOZ&_jO7`oP7OITAEsl_
z*)O4M9~Jw#loIEcDs{gddV}q=;;zO%yO?7S6Z(g+8-lR~U``?rQ*&{QIRL|WS!MqKi(ASw
z!b$AY@+0ODHvltdVzy(4ZIoEn7a|VDyMIY8{0m2!%lci9WMYS>XA9Gm#K6zE_qSA!#Q*(23asCkdw(?)Ges?)W(Wi3n)10HfuAIq-)AF(z+7^!Up3dSO
zM`&7O;9T|dA-~3znvW2c<0FjTmp>$J#_S#gxYLv*_Vqb#C)i<|g3|??oafJGY4Z_F
zPp90of{jH&#cNzu@P2kNX{aK>_8CdjXL`tU>6>t`o2UJs)o#@`Cs7VOpvJajHz-RA
z==t)26ZJ3Gt@=U(n4zJg3N1w<@|To>k_a
zGXet^uqBFoZahgwPX>J_WTVq@Q#Ca>okeXoP_xTs;cv~YC3mt+V;I2!pVJ-Ai-G5|
zs+AS`ek<6Ot!oT*-0F4Rk7`@8-GC5AFh-i1F=b|Ma-63qT63Ir-{KRwsZCtlNm#bV
zXaqzoRILCsU*yBtfJK66>)&csF0RVWEs<|Iy-qCC9QfNIcwN7_BLoK4a
z6R7lISAD=?>`di3!tp-V?0v%vPnZO*u2aYs3CCx~wGEgA1Gf8UPC{~m#-2k1EY!Nz
zvQ|R9$`s--66ye4zi00i?4Wzo12Z;_Sbfc^W!O2ZKbi&
zkEo)IYb%~3mTja~s#2*!mZ`T`OA{umssYSe*{1|pW-%%ve0B~GW6Bij-W7i_?+4oXiqsu>0RgjJmuwQA6=8st$Eqg6NO9LtQPv*
zY8xWkdM#8@jQ%{9KdaR;{+f$#YZ;i?i}!&5`;29uGPEp3e#Vj=u|?!oq@`}VpV}gP
z!#DSaI^_Van}xOejwv*B~RP90W+f0T2poHacc=oVfuZy
z_BYSi4}Hze!5grM2`^h9J*%>YqaYjaxK<85o;SOVsb8A5Dr_J_+f$4`vrVR^ve5O~
zio<+Mt!5Fazj$w{RncvN*?vxgt+@%%GrS|M@OUPN9#*A=aXb;(s
z%aw>6E)KB9KaKCRlYOe3rxZxu$?v|vO?}(gGQVY-6WkfT#LdI*R^<;W?2O!7lQg!j
za_nYUX{vo1mHKc9$E$^XJy=tYaP7*Wxzv?cb)Z|3kixpA=H}+$hv)6zW8ZUq!;u>2
zVBN9r@z-Sm@h0XAZo4r#Eddensq)!kCqEuMp%%u-1uU&d4tX=t#~1J3r#SxMQXwiq
zNk}nSk}dG$N`c6>gsmvetLg0e+dOGnEWof*nYq5QZt_{!8M(gLLe;Yy4~9M2FXgu~
z`zulR0!L%q_aAzg%LdkQ3<^zlA8Ig{C%MXU6Py;XhMMhPr6AWz-nQcSrFJ+AXXlZV
zWtK%w6wUV&``q6CoN|VfO#R*DQT(A9Ft#fOO
z*Tml}#lK{p+k^u&L_&lEs7&%oGCXX(Pu5OQwsR0z1Z<5W@!Wl<;PyR-W7>~&G_&B$Z}vCZ%`XAmW=L&Q!RCK>vOd^Zi_ywPhH#Y60+iYFU9VG9?FWtWRL7n;O-R7Ke?ncUClPX7Rj3D-=T7xY=R
zpT!&gkfzs@)fT7Oxn{=;?DW|e*L+?q?=J}OcZf&j@|0}aS0$Q7WfaF^v|iR~c8xi7
zYIWHz{{WIM#WuPl{SKFG?ng*+5;l!tIuwn@=2|TlgxYwDa&(9I@8OculzgmRV)13>
z3g59ex4^Xq?
zHyg`C6KlZ-x+$NHPR)mCp4og32=rw94m)u|Lsg9~E3xBMb(#_=Rq`~7%4hmEvtwtZ
z&5YYwvJ$jvDtC=F*(XG$e0*Z8*v+g(#*VP@y1WYG+99hk&E+%2b#6CPAW1X9C8<3(XEjW{{T
zO1fD5hU*=R6l!UI+5}yB;eCyaUP%0!pRoyWs>aj7V`9X-n)}HQ#Ke0>b-kD{gNW065yTm%?#>kSo
z=$4+(IqNCqVv6Tw^N@sTc^uKbzXg4iQ5jJ^ILL5u#E$|ZMPEdeQ~o02;>0J#DW#yIB#@6#=q2;L`Eofnu&E6Cer8on@(B6yCT;VK@CWJ~ed
z@M%wHLi@zyxN+L+G10YTG@}xzo|>QgAswo*s_tBx#9_OnM@Ef^qJ_mu)p4Y4BMWqm
zk%H{9u)G~HjS4)qA9(Ot7&&&0Ju2>5N5L&L-SC`uBTu>ZOHMKpu7bItbAiY>pk;&YVV8alWqGND*N|XyB!HE*YWn2Wh46^czDuv
zn;fs)vQSbQtUNId5RTPcYl0bN;pzOLLATkd1q`p#+-}Ol7E)w=*739CiL2+05khyt
zB8ITBDIRfW`#g9xBe`~ZdwY0-eS3DDj9&PfVx>}8>MH3M11P*zlhYz1E7$l@3_D6D-=u!ME8s>?}vBdZ(!(*q_;Dk^3xoqQvO%
z@wYZ_L)14L*#7gm;g<1u-Nwt}#5OiwB{94uh(~c4DCNa6T0cjpG&a|g&pa$r#bpW>
zZLSj3_(PPWDv7w>S$KKA8?k&Z*!d)Q9JpEJ(~i~0F*5HmW3DR^)A?eGTx!YbeI7lp
zP;(m@zv$jAPqJFS1LL9Ri#0VipSq*JbdW)O?ehkgXy6Y-QdUmpwUlFK1Cx%@xJ5%bjtG6R!eynMFxc92`9HzhOGz
zw4$?FMx39|6@}olYJCkgm%_&Hm6es4vc$yiEYz*v#IItw(a^DF@{2bLSIHyfeUgn~
zi;I5B;T9${a~z~Ob=aL^v0m?A1q+&Gj!aT4-A9vfvnbLRd|qt7X-M2|EOP$sKP
zD5HXImLRm3o(uXtHDq|>6XBLoq^40=Omlg%iYp5qpyXYjlFCQpiyp|tQA5O)^RXha
zW}@?WvTn~DZP{3SWpH24e4P>H;T++kjk-=zW6hLGv1d3|HJhWck&fo9Q8etVC7MMI
zBz8qt8Eb-sKJbx5qhuDDbB$C}eUVt9qoT@8PK{V=4-Qqx{Qm%GQ%@FIdA!yZ=IBz{
zWtXzgBjE8z=XDKP^PZ37zoIb8EYxs3Yb?}QhE7d0&uWc-XW-8li%Pc2LzH6a=7fsE
zf0b;llFb!_v0FU6w2-4piYTG1EJ;wYHa>VG4I(JgQ1FpO2_u2x10N^$dTK;`5s5_-
zikD>kUYp=vNv_4^Um|zWLPzcLEWNYn>uqN9{{VWisMKD|8nG%U^7cYBuPCy_p5}*+
zmMM{LSghG4sZ%`{7IoRBd=d37kdsT2PCi}}Aw6C8KfTRbBg(~=a6PtNj71F#be@z(
z4ZC?_8WaT@Lt7ph5^ol1(c`7H3`)sDEPe&>cwwZHwMV4j>~Ejsg!ORyD|AP?@kJFw
zBZrF}WgoeVELJSbGUF665_abZH+2?0hjdxvbp@4oP+?*?!gG<(G;p6lgRO;~p|YWLG$qE7bDv3Vn^x3?JfYQ9nx;qr7|7T&ur2v~SzkrKxtY-tJz
zLsmZMjBByFKfI#PBsCs4@HqVjfkm&AQQ4(4N+#puS0;MWoR*%8;8V6plx>bO
zgCWqZbZQeXeeJX7+p%ScL(Jx&xh3j(Iea70e`lpN(-)F)?2OajR8Ylb{T^RFmr+*5
z;+K)?@)1?h>DOl$hKWjYvg3RUZInHW4;AgFxoo=SkC&;I*bIx6mt3;BrG*f#fdr=7nL#j
zIJADAZg+UHZ}V7jvN7Y>OBI#bTWbybGxMnQB{}mW)KUCJe$8ywQh2Ti?f#ixXz=+x
zHh+<;t_JLs<6V)HoVhBF&aqgbAuP9E)SU@$f2of{l)Rh@F@aK<>7|A_4VE@1wPKi-
z{Y3u&Nf*%k5{f-JeG%}r%@JNk>2!J$bEz&5N;hj|ixhNrGO2J?_3b`h>3Jz|O5}Pe
zJ7TxO6cl-ozl`=fFPh55?7h@BPtPmh^(FR3(xe
z%PzlY;=kzSmsmu^xv~B2ye*bFn#bpgJxe79*%eD;!Wn0Hq3~ZAZi~(PkINr{%Fc*+
zH9kbX&X4SJa^eJGr?I@Dlrfekv25cEyehKy&dU|iMPV#Uy^UtX_jzR{`%Cgmfpy7$
zLWTAv718~j;`Xdw%ECsRC&8&2udT?vh+-cGkw;^Ae#PR>PH@oEV)6Zz^d$cP=ytj&
zy}e?~y_b$QS*W6gheJXue2Uwoy7nF~7q*JTq$K2UjtVOg%GspoVwTD}H4T&`JgIvX
zip_0!SVeACTPpi5`aLs7J_NQ|Wr>pVMklMQ;OFQ|HHtKpP}&;2Q4iFnxfhl{W1rU5
zweLb7ShDeWMTy@7R6peCNTP?f@j{Q;%=+kFB(}KWExZ?>@u-i4QAky>ZGPPpQCX~f
zRvHr7L&M5P?|;VX#bMzUiYTmB8nI%9vhe+h7nhpK_MXoguX2hW7TVtNW08B=MHEq9
z`&(AWhAppU;{BJFzDpOqU+Ax4YwHcu3GC%3doCkD40Idd)o2dCVR2knZ@KImN
zB^Mm0hgU=gB929el5(PepjlxBta4r%UMmH8<)Hex6a%g@8CyNXDWdpTU(&QK_>i|f
zZ4N5LB6O8*3#;W(k!y;iD`a{fl%G0XWLt%9s-{%>F#44rr#}>PJk+_Np6gDjJn|*%
z+gF&=$}d<-9C2+d3+Ry@5^^CY2*_QHaPYipq1cx0hcl
zna{&vweo8RGJ_fAf2hLc>K#*YFzj-#k!5ZhaamQX(w$1>Cc`VLaKn?C^qV96xgyfE
z8x%)1Ci0~<%l$?V-3J4d#d3!Qvdcg;P!yj^(#n;%{g+1dR+HDx(}>VQ(80A^ILfMv
za|+l<_f{$#EAwT@l53H~a1-V`1o|IVt167I!(~vZ!Bu>(#u_f60Ts^b>NCE6BRxnS
zVKk%sQALxMtSfUII$EfJULFdD)U8;RDx6hR@{U)^oE8efS_;#Apd7D0>w7;7Y2tpL
z+Z3Mub67KIH{%&q7T}FXs`!lrc0(5m?c5U{IggOP9ntKkrdL+^7=<*GO;sB(Zx-T=zBM-LD7hpFHA2muWi9&J;yWX;e5*~^FlMhMm2
zddT))xZ~KSKd6;3Q2jOsxhdGVcd!fa>c_xRk%4WUPX9$=VfH+3btE#Xs)E?La)WF
zTx8e%1qTaeN-9m)Wwd8yWo+%m$$cxAn@$~gE{F6y*8C$KMOAX9)ZJ%Gf{zqWMYd0{
z%TT%v#@yAf4hUDLLbo;*50ch`os%?@45pl_i!r^E*1%{>a47-$7O82YK2l2RSK?NrF7+}^WnO_SI(-OL>n6;fsQtt
z72)Dn2-v9^IOW+uZ<}RhWo2(RPZXlYF{k{YOe;Z5X$8$Dp=`rwUXZmto4YN#g7|I@
zRR%e%pqxa72*n1byn<9?n7A^z5bhV4jV{-Tj)^+22F
ziQRvv$MpR!^0(z>ZVGV1jvMe@T~w+n@YPjm#BQdowP!`?MtS*nS65fg%G`AsZSxKe
z4Y`$7CXDK|I|-yMSXf{lyithRSJi@ltfdlFGNCG}@UJKisA?(LN1c&tvhevicHku+
z)L{BeALA%lzdIn=AGsvN1Eg0h8me5;j575w@Y3LI6`
zuE7r9CsL+W{Q9Hem>pE0(dp)cS|D&vez2q!RsI~iWm>8_J=axSsHhLRZBeb$qTzBF
zfkUno${1fz3cMxF*zU@>i$)mE1%eqtSMu-Y77s|eWGFzNQ|rVNS%>O*gmDfl3x&cAf1zq=X$XRz4Ia@x`Ow0w
z6=DjmL)}$sX}a+giedy>;RR}EB_q}p9H_TtMJF!jG7}JcZWCTP@xThEy%Y@V!+fe$
zK1%0;)>C^XgwfkF%qqQj#*F^}Dk&le
zFmf6lrQpj-#MdM~sQBHEZ`Sw=e3<}(L&b&18B(J0Hng*J4
zMoV5#CGlrZWu=xqe5gi=zUXD$%UsD#!+bXj`7JH5sU=esCgI^rk5mHmH}I*Gz49OQ
z1&RLvSj!Y*G!?Pr4(Zr-jP{j0)U9l>{*+>VR5@)c;wfoYS6iI61-R!ZCwt`H^ve81
zMOAF5tHFK?D|uAova+#Ki`S`Ks><8)Hp;8F0BdegfxE7CD!>Fm;pz?Z#h#i4PxP6Dc_{6fVa<(4P5MZ=xc@KC67r=YD5`xT_;
z>MPW*#1pavA;)TYHC9$szIRv3{!NOq^o9FN_S7}7N&>l5t1D$*D=RB!T)^tQDiAs4
z0>gw5gNAkEg0(QyqV`7t_B;Uq@Ekj-@Yco&P4~_@oa@nJ`{e%sWh!0IjyYxiV}{v!
zN~p?sS?$ApM=cOM$|&H!mqC@)(Ek7&>+(hVt;q|h7bHRaH?_$qr=*P-y;~dPT|s
zIPt?pD^n82p)
z;eMhRN-b#B?59Ky>M&&@WivnE?Yvv7)if24PbTw&1^%C~e9YE5BMC26wpas6C^
zH9C4Kl-6?QgBTzTor!?$-3o37Ea9H$IoQ~N#@x^ishTO%tEgI-(WPx9DF&zj2DPj=
zYN1k9I2;xrJkzU1r#VEU$1eDIT~9R9+0bo1ZqQtNDa9iUce;Tmx`_c?l_5vdlnjHc
zw>XoOH3N+AvWzjLa7DtA!o!4TNkdsJ39Zy!ht*O40F~k5ZX=qs1*{0=lbiq|)yr_p
zU@F`@j4g(9A*-_4yPPJn54l$0zn4LrHCD0BQnQi9#`^-N>CNws*~zkNo&n|x!)87a
z#&^>t4}#nrL1!p<9_XK>tLjuhe-&Vf{XM!Z!!C-sLUZP6997Ev6cS3+i4S*9L2G}Y
ze{0)NbHDdYe!(IUv+diAY=#C(Q0wl<)}6S_=Y`~!fbadaVId=HcHj{
zhsXG}dR^};P7{b!A`IQlib^
zKKVdc`aN)f$Hig8JTy}Mt?snpw7q>S2dH1
zF5zK>q-&rf-+qR3|jtWe0+IspL^;Wg3S74HYOm
z8*{oYLpNb_lnyQ_)_DbaVX;89%Tz&Bqm&!@1%{Y|pH;24ij>FU;sIN8Q{qoLsKq*c
zxuJ{U{5(5PH2g#KgdO^$JmGvCF#IxP5Gp$?ZyJp;3Ctuyal-v5bN>LCDcFu_)b7cp
z-l3mm997|rL0+m`Y~&TcDMdiHITZ6MO?RMI*3?~mertDL#UuS$MT$6QEBVJY`bt%$&c-6V$?=*LZbvB
zisr+e{Vi_$1YGxhkY+K}h5rBwMT%}V0QijM)nQ@=IgxYGHwle)+n7@74yZAf%UJaf
zA+K*FAFopiYhoZ|3$9iiC*e$L7g8}=JE8ug^$x`CWr>A@h#F%}IW0S$4>8SrLsQEq
zR#Uc#GEM`>*`3+$Y^wA7#2iqWEt6EjXJVsEb}DrGG!uIQ|yA
zH%L%(hOPI@SR2}jD^DYq>vdgMLZm3wk*c}@qEBV79vtqRg4(N`HBrS?PHh)4QnhJ1
z7AY`BnGKZja{xGBdz|B
zkMQWaE{HCw*;W*?-4r>sR#Y7Td}o$v#*mrKnp`XBPuF6Qu$a)zXj=*rm08y3n^iO
z6h4m1{9g|fN$#Xc_FoV|{3;bds;s6VVQI9K{4>g69QC-Mv-(rvxcMxoRgW
zDKu&VA&Qxv025j{EK%86Mi@ux#OCGU;?yZRk7-)8HJW*qrw(JPanC@`Fq(hrK;xeg
z-_qE|x7cOB38(C~>SwYi1>xT7P9tH;UBM3A6-+POUb#@kT9x0*rgm1%(e0HKggPq9
z=y>}sUzITch5M}gIa7DBM+X*%
zl=WDSJs}t>sk}j3C<)rGPznWtrc$rLOPv1DuQ|>oFvs?$5X=ndnLTr#4MrFZ9hTCF
z$CopdQ9}T4adod0mpL?8T+c-ONPumXE~KaeN^U3XDgFpGjRB*l6Q1r!y#R|qbiemL
zRJl{+4yz~@Rn95wtA8i5s)P?t8>*fSqH?bmhU
zd022mRJ%Q+ZeXu`<@~O#4HgQ@naMyk2Vy>4c?CoY96B*avL&+MziIyfg;w}X=hUFa
zSx42;7dZxDw^ND_#D`;b>Et!NPX!f6k>5M{GoGC3mwd0CZnoDtWe7WiV@Y0v!>N{1
z(*;SosvJydu}7pfv%!Cd;eOSq)a
zcTmf9S{X)g45+EdK@qQuuBBs+_|ZB$W_6L+ULF_(TbT&kqWmL?xTR@R<2D`$y!S3
zms6BzYWPwW1p6p(gOj4%Ej{vrsG=ypifmgT!U5`{!=`Tv>SBuEDhckO1MkN{E>$zd
zOsJoXY$!d^{Gc1?B?r>Bbw^FzBiNmf3E
z$f{8|I>w-?Da^&tIG@rFc_Mz5A)u`~cB)w0A*58Pax_MHWLCF+6#oFi!jGv~X%qvh
zZ~C{Ae~9FUgDoOQ4yRG78)QpbNEc7k*^xujikSvYCQ+r<-!6;C#1wC*&4XnHfJa5G
zmr%;Dh*ULqTc|QHikBw@Jc=nf3RxHFiR8O>Q_qZWhV(;#Wo1U!s;s=C9`>+w3W3f&
z*G#t+RZyjEl@6+mM*dOonguQu%r6o22xXY`5}A_^MAsWM
z7bcfhl2Glzsr5XXzwi`ktooC%@Q#rG09SQ}@Z&QM2+Vsb+-22UE
zAd*#0dIh-sg4#5(cg~A(P}RGd95hkKQw#SM!BSO3s88MsyHgCQ{+jYD!=`C
zs>Ty~3$LZ#<$vi;cUMiDWw0pU>H3|v?nJaG)
zC@D^!O13mamx4QIMdCCP*{hW5-TYA3WMSHmMK3q9fn@&x5DIftx%n!ij(vxjQ}{vb
zg2{!i8x#aXIVsg0VMY*v)k~^-zN4xyYh(quN0nLyO|1)H*8c$MPNEf01qM`d8ihjIs#R1q#3N8?%?T)sHp+<7
zmya}sXiNIlNIRxPFL-`P@
zfoONxZNn$rpvQpG&?H?Rla00goQVP?^FS7ScEGJhr2KfL5R1p!C)%Xb(>T-ly8Xnk^2~*#akngd|
z#ak+H_EZ_|3*tUcK)xOj_j34;G1AM_w>eo;h7}QUKy-3iT+-9sNO%G&wIKE%cJx$1pK_)ouXP$i^shivU|l_x6%`m(^BhuD6zBo%
zp~|fV7hw%9MXnI;C|h``mdmZ%u{w0hf>s+NNNS8QXQB;zjvg#qkSVXlaC5qfNKE>g
z$_~ntTEItZzQ~L+1PO;4sBj?VUyyGW+O5vgj+J$(q8r^4`hzY*!?lVBn^v}c(HyK1
zW2`cUtyqMG4Xc!L@`HZMMC`KDi0wb*j*_rgM%b#Qrm6?0qVzLgC9TAh&4C*?qVA{=
zjE11Kw{v?hNy=(77|z5FB*va((4e@-6x$D(
zO&_N9H~U9ZbbFzO+MZ{CVlY#m6?kASMD*&P@)w4DX+*508c(kc~H=eSHlQ9^%X53B|U{bcZjEyY8WkjO&yG
z3v-?MS1R%|y7eoZq{;SLK!AXmJqI1Aj){`DQB#00dU=!>!2od6A_x$UCJ@`xBIk+(
z*60qzcPO%(JEt7Hs9=(}8gx`%CM&e`9P+B5$l_8aGWX`8zL8}DsvQ8wGMxir(m*5y
zx|wY=odUAlKvi<*ot3yX>e9i@Y;%(}nW`fy2pvkCXGvQ!AyQPmY5Of9ONADuFMl(-
zI*s`R=Cngxnl;B&CC+f{f$3m5D4()23y(!6oZ$yL1tN;bAFnIr(5j9aaMpnc!Y8|w
zVq!JF;>Apw=ZI?UoW!m5?6$N8Cj0;dKbx=QltN;jXp(Ki0cd4zk%%F0#V6Tp1eIlFYG-}{hsZ99^+3jx%}J?LSI%8i
zlr>AJrz&|=3aa69T#($GzUjUjiKS84;#Fv=S9e68Bl!-6R}FItj$D=3sZs4)o<7TS
zm%?cGE8=1jwSUTuO_f$^e^oCzQ#$8d+0a*mRwJMyFdL+GDtu=maXoD?iUE*bQPoF<
zApmaEMW|FNTRJWO0EMNr9kK!?PCLKtQHCHb>MY)w(Oqf6s`!Ij)fnHx_?Td=T2JL|
z5UEg(qYBd2)YW5usu1s>CO1^Eo|y|vn8yj5Elk3A9Jz(yvACLX1!tNi->1X?b1IGU
z=M&Hr#d4~;0CVQI-BD3X?Ug*LsB2W?80?{qRa9r?dO`*V;R5Xg`ao!5M@EqDkc_&X
z>Ww4civC41x}j8C-WU~eNniU!VK#mED5|6d@m((Hwul^0_RP5QPA)%XiH-WGac?B-
z91qgALE849*NyI}MLF2m#khv8Y;r}dG6ar2zP_rcWwz^HA=1^O39vfwLvjPE5(kKH
z#Ze@p<`O$2=`+=8AW2Z$uZgTbqH8DNw!s6em_>s~5xY@Z+~!a5P+($pmA0opM6cQk
z(VzJ!H&;}qRa8*xj%vAR^COamMCTb$a2aTH^+l`^H-RbIU8n_FjHR>U{{UQ6`#s?#)65#`&4nOiD0RN}l2
z$}rm}sa%!#ZJP<;HBQz0+^3>G5r{SRS~0h=X~74%C=hhwHEttPaLO7KT<}^~QlRC2
z1DVBZXfB$-4&s>3O%a})Tu66CuMOIoQFA?QvKsCLDd%2c^*n0IBJGkol7pIHqlfyTt-Q)U9wCEffVPMV0*2?LQ--7k
zrvyiSXNd4(hd~myv=40ImBn?tEKEVlu5*gUz`r^vb=QdeCY1~j7RL~^YIBr0{k*y=
zna^PRql5aQ4@psmei`M%NTMDay*69M<*yc)K>*Mi(6*oauS$mi-*qi?EjWYO$}ub#
z!`KfdsOCqIB*0r`Ivcd97S7>Fud;9s(%)CQCD%DwL%lwIN~aZ7IAahVl(
zFtDqu)e9Bf8Upyvsjx!}39#8kZuE+kv_nOdoB8g;f7N6Tqu~>Nh`3^d?sJS>PYu7y
zB>>!F_H%G*_jA<1-S2A3VHv0C0kW0_4l!Kx9{jsntdptQZ`lBNIw207YU
zKZOT6C~HfUKOLUJrBjzOw|*4(McpZ7`!4(~%_Fg0{E8~@(D`t3s8pi(WbH9?+i&N1++TMdY<-J9RMK)?A4ZTnt40z<4;m;>GvT$%vo2rj}
zN**lBd4;Gs&wn7V5}XG$Un0IhDhUVvrTcJuBh!Jw&W6GmW=6Y0GJxW*C58Zsesa>m
zsPt9hl_dktQDAJa{YLBg6VB+o>7JQVjq0^y?>*Lx5#ckkhA)qap`DhjIDc}gMDz+U
zu;Fe$jIOWd*UFL=36&I8RXIS)c!6-#-eCaSEBsC%O_Y@;%3nq|l9Tx2@LYYfmr(Uu
zqf@)(t)RM-3SU=~C$ea5)IQ-s61lhELR)c?4ebsoa#f*FBCQ
zR_ZgAwS7MwZVan&~zeL~6DPpc#GLF>6&G^
zpsiR`T44EADxO7^LZYrz%7RyQrIqiL0IOV6{VjNhaXys<)5!;+MD?nioH@6M78MVn
z;H%&*XfX3oXxpjfkS5BWlTq>#6J;PQ>MQCCVKBFS
zveasZ(F^_uO!WiySdJz%jvDPMl~o)RuNY`Sqly-nv=EE`08eh8FD8xZwBb{}R8xsp
ze;+FOR4HLYUnRIm=C+v#f{7l5Jp76(sWws#CCg(yQ27Tz`bGK;b_DfaX;lZD3FO3KPm}b2TY{UD8d!$tH)j{WxAMy
z`^Xn5!f^H~bm^b($?rG#xHI1Oll&udZ
zXpkc1VB%oWxPoJ%cgWR4Xfx>ErA8H9)k}{U^Mu(E>eN393hm`-tg5q0*UDq&qZ1Ru
zZ&d9rE^NnS!dgR%XCUW#6lyWi8WZvWfBX}_PV=Zm26tZ|WLq!XdG6FgK^srjnkE!#?g{VI@Gb-A=(QkK6VwCd`rwJ_@JhK+&WN{n|%
z?1;e|Q*fGn{#uG0WM=ldt-N<$J{!p-?q^Kn7tu6YnxErOrW9M~y+bTFM!!X`UKoo@hC;EPpybMo_&b-#fLGFsbk?^$*RHD_53BETE8)MUoH+-&v2Er1;1~yp^@EC3GAkYVd<5m-(x!t-fW|@}XB%JUU>1v8uVN
zN%uqyCjHSZx%oMc8f8%-SzS~8r6bF#r^<$k1yAc)9RpDM2{{ZAUqy0J7i2YjAi}UKg3aE3?J*2a55D_X&Nb12bkLkp@+SCd5}}Vq-z^
z6+(oPyPNNlakfirGv$lmqi@n$+&+^dz(!S@v2o6zQaN}&kv`%C5*`*rOn6fDGhmrO
zuKK$9A;ZvX>S+H0K!~#m5PN3sW$4Vs|sMCU6>N$oW++uE7qHt8lc#1(o!71V?%3Rbj;#l!2Qc)UiRM0=U
zOtafBL_6`aS*-C5F>5M
z&PBtpOjbcW#hA+v18H|rzi=@+Qa2!jJkCt8Sli@M12I%OW0tV$V2DYxkrmUaLk(V7
zAM9g?OLKsVjwf!=oBCzu7eXpwgzcy(CDzTDR`_NbhFqc*DRTAkCw6sH0?)YM-?$gy
z5yiOf<(Nm*M=)7{Ai7}ai}5(i1Et^=@Lc8)P0V08nVURIrXwTRcsyAK4q!DcU&H~$
zhH9X;c;wXeB8?@Lp*uktQ?$E|W=%mnMXg3@6T$S4mAmSGPiZO{{UF}
zg7b}T^K`^@D(kVr<_!yW?E8q;-@1!`t*yeSw>JT`$D)0pHlrA+M$-hNxso3dD{)Ac
zki^8SM&&bGmP%~_Sc&0*8{EVQfs7Hb1j`U+trFQTY=FUpmy{ni1UdMfY~hu+34;aW
z)l~5tYSl5G3_#)uk`fm?nAoXFG6N3d)rqVp`I#8~xvPa(S0oa|JEYibzS_C&ELX^E9j**2jGG7PCbog7R%Hf7Gytx6u^sTaqA&<57en$v)LcA
z`eJm0>ND`fp&IyRJ6@=x40kx0A*Jl8Ej1ji3-_#$Q$?mUFGQB^sB21t%P6oW1U
z3S7j8?pwK#xYJN>D`6<*j@emMwxB8|ctCbC!y1l-77WL?mQoa%tFI7#Qqw3-6@AC<
zmgF%XfN?%=pmMN)29jE&X>+kS!yL2d%YDQHw}~^7d{H-rB%KoJ1$krcW7Hu|ptN`aN4jL(SHS?~
zgNml{h%b_0MTO2Fqd8SnMdeFurUo2+d_TeP(U?Bv5(Fb~6?IH>zfj853<6kIBQP<~
zWWl(J@$ef$qVZ-1skzj%p3McBmJwUrdc~b{E^V2V3YEMQ9a7_3+X})GR$G-Csht?E
z;O#pyvRuFcz08j(e80HW-;(ID6a>P2YB_lY7rPRr?JTOfU9vKUb*;KeyVJIBX
zzR;obsKWmMRK#HTpSQ$S@mS$ABrW$dD!PkV2fUzoSE-!0a06_@FP1lPB2g&XPnoj$GY>4J_*Wo2u3ps_r`jO1A{dg)`Oi@G?
zl9OZ^$4*Hva`hK6f(=5s9m~!%O8SY4B3xsb03@+!W$JAco5c@L~
z-#A;#Qq_+E%M3lD9n86ia|AUxS#=l9NF+iYHp-Ql_#w}gsR@A&wq^X)3Y&rtyPWCi
z?ki<98H_4=a~t{yEwdXQ1EA_Y!TuSbs(R}E?|RL68ur|Hqx0@N4dxA)}Z$nBB_5VqVe#mQ`EGyX~a9kZiri_
zx-pGHy#AwxJQzq?)jrvyG3HTU5ZW+b@hP*eW^ND{z#
ziNsxz*#SWE1RjXS;}6s}Vtu)rW19+`B@9?pQz+HRGlRI7Uloych!=wSW)2zZRfeT)
zw=
ztwi@T-;NSY9KH+9lC7aNWiQh(WSZ<=C*-5a8jaR!+{^&=;UXvE4gfR|CakAfCnUi&
z{&{dffK)ov;Mj(cZq#P0>J%IVZe*bCOO$Q$22`z
zhb#|%7b?aB*wo4{$$YIt-4V(qMlPB
zGCiV^=!omWo+8u~A-p~(%6C(m)A2Fw_=z~{+(U1|bR+6Gcx9ZS8e|`cTZ{q{<6W}b
zjY@!>XhV6JsIPB3QxzCSxMlo7Foy|KID@34)b%5bg)n`Jbw8|+WZ74G;Y-#?WR&KSbml4Ot
zL0Oi|DcYuAMd-|!8%4x0^h%ZolbJ=lSz`<;xRJZ&o54oMPl
zl)IW34PX<6F2Qot5v29YIh^aC7AT%shdP61nQL$!XDyg;C%Sg0w-C)nDLb`<5lWBT
z?w*VsjO`HWHgLm!MCx2K`)Y7d`;=TDTWrC`Sxr<{T)}r0zM&L#D(>eFE7D^DqqqkV
zLz%TytWk?HhS-CcHlaCSA8a65Q-sW9dnKj!E6QQd2X_j1lwv&c$@*W^M4X-=!%Kak
zLJm_un5*p*;FmXe5|DSv5+aNUrtTWk7$DyfE9Fp4mnmX;nDHOfPd}tO*X=!kji?1~
z4X7eFWWx~{Dj??UKkRHTSaR`kD;c({_iwq=TMpeYwMlb4)&?Cy+7}5_ZGXB+U5f!
zRBA6-h~`>~nSdjSkqr2TTtb+KGb{_O9Y*?qaLRm5sQmajc-+j)qd0Xv)7>lC78OA;
zuGsd2&5K;uF&t5s>Iz`;r}imH0XH#E<&+f>oMM@`TfjG)AzmPu7JWg{Pg6iw)U?G%
zN(Q&o36>(f
zvo3J}>6GOxWvZ@Wk49U&mBlK!_&`?k4Vc7Vn7Z-9nElPS6{kmuLAgr|b6JvOh}_&z
z+&dxEN1D)Jzmza*7Wzt+%=1wK{7XRh7&@6KfWN4eD|G-wZV;rGiD);9aU2M-FDy~X
z@L(Ua7kv{Q7>Djs+MQu=l_SHyOfy=SaCdELE5J7@#6rnT5=sKe?gXPYHkv^g;|yf%
zz_~sHn&S9|a`P>x%tW0^-_B*0Uvi4*cy~hr29t{%DZvYcT|iUu981_$D+RIh{1DQ$
zh!vSbaV9cjF^oY1jeH|QD-wNpxI^s|+!9?Q=?b`jI-G+WaRV0Ibz|cK@C|+UXsH)JxxEhWKcSg
zXC{{{+)c_&F)K>IW{$UA4;WEY-
z3ic~N_XViuF}R`W3|7`}hGY#o@ixq!p^Es1fV_n0+}W5hR7SEP?YUkS3q)ze&U(yZ
zdH&(b9-*uhx72Q&`-(EbFeI|1c*$lnj54CRbiB%)2<-Zn9uvVa{lx6UqJYJ3P>dT*
zvZI)Z^R8Iea2yPw#i`#H?1n<+m<)z2IWpT_#}Zz`Ma8XyAg#=LiuD&SJ-ii_;U96&
zEUmkT&SRl6Ga0p0u3he67THoo9Wy>G`%E;;@|fmmxWj%q=ueg%(v0Me=y(i*F&3j&
zGUNq!714SnMYg+EKM=Lxt6VIpK%k$wnLtMNOD@utgWLd^z97tZ3h%;^M@(@G+>cP6
z=7u|p(aGU}6vPsUR#|W%6P+*p>R&qdx44>p;-+54rXX3}WQ*Gdh!ArX{HVG;N=%9P
zB=Er~(Y_!G=mOyRYCKFqgVf^BQ3qaL;n{6*8Ham=YIfHXOYy_puvAM#&*BaTFyO*H
z7~CHjBZX6_M^I$9DFTi@a^|#TlxfFsgjn2kT;)^h52=?BfVY02nfwSipS
zDx;0UBG?WQZ&77?cP!q#Zrrh`H>euS$qHv-Dk9j8vZaf*ir9k2gAHjgvNZECaa92IZ0U8&QW)=P|meUWw|B3vh&Ea-%#=o;b@Y73(u3
zt2csWD!I1d3DKQKQ-Fb{n1i1J5o>TN>_4eOE3*tlChbGLwyB2a#kvwvG7kO9l^q8J
z(!^mbKg7B-+5JmGX6n`fs@v7nr@?*3%JQ(t<=jKnFQZwcTAYcRRtu{;lmk#gIHjSb
zjybaa4kmK8O16@-3L8ktSS6QhGhqg0K@u_{m1VeUzV1F|TL_G{c1%$&;_Gt9GL9fJ
zy%g{Uv_+pCJ%;CrSp4Qa
zq18(;e$t|{H!@i?MOO}`T&L7c5e_5t2!|1AtTf8rF&?no!*Lfu1)?g{Q^=O^%AtZt
zP3~*77+gr^J6yW|0Btro#C7vD{{WeWnIO0lW;o6zH<-k!UI}*-kV7m~>=A9Vv6fHe
zC$N(W+FZUR{Y@9c1DA~i^x=k@oePZMA56%TwES9BGXpONOhFBE^*O$iDmW7ve}yfV
z6C0a@4t`0dyy)f=ij4+(%qgVFGr#)5N|{fzNTpL;O?Tmjpndq(tVsf&vTy1OFv|Iw
zOfqU2n)#QODCSbwLq`#}RS}vN?z1exP9-~k5%*)>;9bgKK&BY5#$m3^Wvs{b4Q>%w
zvzVwF>M3qoDM-r9M;n4N3~WMh4m>gowd*8#;q*K9CJ3Bn`S+A&$+qwGh@Whx@?7V>xk;^T+U|`4r4h%PyK>HRZs5!
z0OWQk4PSruV8{dNQP&y2b1zmj{?V^8@q*8AtZba5v2!#1HwK$vg03Q)(ZEY8_tZ6!
z4G*Ni^%gvGkIYGlyTXV<>sn{kn@aHolB09P_rL(FO%Tik!oG>bKX8nYH59
zC9L6=wy_5lxm-&&Iw~$1_RTF=@6izP^V%{lGR_qhxTl5>mw|HJRorVWMs)zz;)dMD
zpahl@ic^N>bu0Oohj9{fznHBf=5q*b5Hz@zfZl$kC>6;ThM~ro4jEH8tgT%~Rq~gb
zoP!VvCgfpRe$aS$XyRMJ$eW!J+=V!0r9*JpZaqW`qdD#i2&Sgp4xe(x@tA_B%-hs)
zV9jsPCB4oF&@}MfZXPP(J;=*fE~49(Iv^TpPx?-Rt~{X`4q%)ikC}P6A91LM`hhir
z+Ce*F(4UabZVl}T*rXezD=eAM^
zf;u3(ge_(U>4j6ce>2db*jG(i=JG^@iL)4({0
z%)&#L8-TAnr9>9fL;i@;jLgFPFyc@wD6ZP)GbKQz7?uMqvPP$vU-^f;f^tvXP9}6<
zr7;cxNdutzCAGaFO@Z!cu46WXz)nQP%nPOhF22ym7W~Rw&mI@j1|Xhex3*RC`GdH5+-o#b
zDXeeOBX~x{;%zW=#*AUatw_gdM3~IP7#8B8UEJ;0scM;8%-htkK%Rcc!wgDj
z>KPY)VR`A)II&()1E@x;5Jm|`5)evou3-#?c=Aj#jhI|^HnM;)V1To!QPIp`l<0us
zSO<%Y+DUsN`JxetYd%u$R7|(ISVM6RV5S+L;xF7(w7(xRkQY5XOKE2bMqUxD--z{L
zUL`%wMp(q?2tJTpz!~Vxxs|H^%mwOQ!Z1RmLS={=#I;KrBg*)eBmO&$gOX9Bpd~lk
z7vf?)WT>yFNL*bHS+u@Rl~^yZnRo6LrCpnW%Sc0E+x?R6o!t6NipXekcNmsxRG$gy
zf`G>JC>IkD2kKtM_$HD|qw3&wTJQ=o2(2}7rm-2k3^6irdzu@R*DxuBea5xXs+DeF
zhjSnz8K=QqxNZ$@QxdU1aK&Q;&>y)=7npYDa9sCCa^G&HOKr>rO*85Qy9j0G9$9-r
zu(1prpKyF7wiPzuvZBI&o&|S#3ywmoN=(!2aekR;_bQ4q&JU%Tdfx>R_rij1Y2n^o2k?d5;b)Wy|26
zm;sK|mC0>VDWX68`-layw-F)4|Q%tdQjfv_I~xqYC*($-f;
zgw%$h#J=DouoJoD3=dUTfYSye#Ec>RKqw8E|0P46w_vX)bPFE#Gi}angU+
z4AiG6a+G6J7%Ez1OJWO`LzKb_4cs%6=37~q)+icSQu-Vdf*GSdL?y{|mHz;gVJ*De
z#w+bx?qr9-4HCy*V#&Ch7$2EINzHKrI7hql1M2B=sGQmbCH*8uPYp2X_(n5Lx
zVauh~Ov70FC7Foh%(cTAlkh=84S&N7V$PC~9MI-Dp8*!8UgfbfDwtwU#j`T8?887F
zt}&G&MOmG>m<6@0=>g@aKO_q30W3Sd0t(T*XERayqYo%mt14;r0iMcb?rag~6Fk2p
zxEsDRXl4&WU?8dD?(xLsDSiEOmA
z#ljK1r|vcbJ_$fS2O0_2SWP9x%xEv9mZ+Bv!ok^`a8F@JptYOpm^8(ykeADlviLr1!q42$UGc#K2@MR4T}P)=@*z2w?H5Wj;E1?~
zKcajK#2-kiSzkCM0Oic9VRH69=Q8+evI`u(qOSl|EtbVRGAc913;0cfIyn%gQL?Xc
ztRZ95Hk3`HPjfL&n3b^bTe6p!Pg39v!Hq>gnnJTra){99gpa^A2?`5P$*DsFsZC}T
z_@Y_YztHM;v16u}1}w-vQiCJ@E?B2>N^kO|pcrEOHsIlBGcCu~CO#IniD^LVv}(9)
ztl1ZO6PLKR3?M~YvRZKrn*RW4x7?|y-7nZ|*TR
zN7Q*@%Tlui%apuVJw$aI%2qTE^ozYiM787f0WaD(5;p-bYIY+KbU+XRa_`^l%(*pn
zb8vc@nXSj+G1&J2iW8Co!Mh?wM?(sVQyEEuDfbXHZWBqA{-H1kQcr)1AFoqkBNmTR
zzz?lHPGD-|n=>(cD&Oh`Ypv6mX#o4tsB;(O98BFdtEpZx-r1--`Gz+djm?DTAI>2b
z7gUu~eI$JassbB+BeGfWmV~#6DTJuOijSY-Tbbd*
zQJ0
zukg5mS8Jn4TG#Z&yv}24$7~9NL>MJ|%NubG=^QXcjt_%qbi~LI!)}SPismS8Tee*Q
z%q;Q2$ILsk!JI4UotEH=SXB3m{@yo(@+bIF{{S$82P5IL{ltox9dwZB0`V$JMj@os
zX$AWx^#QOhMYA^Ee-mM@cN0$WRcG}se-y3{{WP+
zP~W&{e}IiOM)1ov!{R)D!YF*t_(WnJoU*9j$V0~r>RvfbE}{<)Qxd!%r4l!BML6_L
z!03{*D6!d(jXHS}tCG8kck=-^8;OB|1G$e4fQ;?{?4Hlu9@$Er%kb1IFv^n#-h_U-
zJ?-}sxYfcvGe9K$?m3h!Ru2fLE3U>pW1)bJT`_K1Amflo?c-;na>o|zR;!Br#i}w&
zTOA4SVrs!LLP0;gkFEGoY*BoZ{H=CMUmlNB8C(V&)j|td!_=VRWZc~l9ZPHs9Hrxg
zxs7GZ_>BO@9}w}$dsNFRmFrUuV0|eR`OC>N)8OWb0%xVcZV0Ahof-JJ}F3n4>
zB{Ll^(HCH0!55WZL>9+hMsSl@ho`BOM9pIfd#aT=OI1fJPt>I!lS~rS3i|^v1<*iY
zCyPG$?t?QibzLMl&zoP0qSLuUDtyoPGe|v0JuU7ntn-L6)(MQt?BhTAEo!{#=5U`t
zKH{R{?1Gy)m!DG-yi*ZorAc_(;yo1}D6eqNJYRv~#fZf+3`-p`3DC;Gn1PGTNiXmu
zMhpky2{QgPRe4A?JC(SzJc9a(0IwL)2lOwY)OT-^I7-QV!|xjMJGi$O#pY+&zyZ~B@T&Mw|FTf^c}Lj?ACD)9QI*FxeMwQaIdouPodluqY~Rb9ByK5EQ_``
z5ST2&9tu^#?pnU5+a77J4t`;^3PM@m$JS4Q{JbEf?*~1Cc-MLBWBBU=3e+
zsn4lN@(F%F@)A6VV3n@85NazKTa1tIAl&g*0(+rUwuG<~2M%Fi2#qSI{HO&E5~k41
zY35xtCZf<$tB#(BZ5X;1wTWd*qQaMmxHt&oz!l()2H}4ag+g(1OK6K+CPpoQ=2WsK
zJ4%7&uM31)Gc3G*V!De}LA4>DPf-TnO_Sj#gMgYv`hoFiGDGxC;aKpm_M_poMKue_
zl_jq381-EBFy~JcyU+WZ3h4K>g+CvRxuku|C5K%lNux)<0=Z*xEQ@Yt-W7Xd^q->P
z7l-cyQx8nCC1uRP&gqq6RL2v(S?ydNp`7y(=3ZIO3*3w4099>MGGGb_daigS3#aul
z4i5`wDp$p9)K;Aln(j!OO-X+7UlO`8P1?+CU$$)4xPes8v-c=5Tf2h*=)p1qpbi15
zjb_#BxLc|lhPJnfTeX6R4q~k}1HnLQ@ehD1IsVd=uf)5W?}Bfl89$;g8uNy#R24;9w*nh06F-h&
zK4+dU>L49UN^=~(&&(yj%MUCB0m0d*ry1}{bWhyeSvPl_OxLU#Y#mH|s>c^_OA0=r
zLwviHIUeQ!KQRkoeUIBL&c*?~_ZqJZ?C@ul1Q!fOZDlCB%(HT?kh(zVfUTkxw9XYO
zTCB>P3|kK&vkNS%w#FIUCJ5K5*vz1H6O7)jJR?=lGn1K4nPM{)8c=2piAO2LWtGbF
zFKE!T1#2zldu7qLsP-^8A8>ELGYEc{!#Iz?3Hp^ZI)E^X@62VKGqctZ}9T49yL!MTkLQG&9^4#IgsPXlx=Ixp|Gk!$Jc#
z$Ahz(_7+nGNaCXU5G6mV_KM600mL;T1t&2py~ZnRXehd)qy>Z#j%;U(#0~0^6of7|
zTO5LAKh8z=%Os9jVbEvjf>&ixX}%$-v(!r^vgDJ3I7rxe`I!44e**Y)Y$7RtqEX`R
z<_MkmWx{4KtqcIjWw#a~ZiXUTTQ1zkFH!LgGYJSD=0+h6Einm+NErge?Q@Mi%b1q+
z+^I)5Gh2p4M`gL0Yn|#*uTs=->UYH2WUeA<#HH4r<(O;gTYY~?iOlX%eSfKiGXpMM
zvCBGCB9dFo$1Gm87p3@0!cz#-m_Ox+E?ez}(DhQGQl82+d4=%YhT|L7{^9fuZd%82
z{*&&7ehUShZh4qlyhXiFW)5eN%d>3`!<@bCu_sdz*P2v*BU^SFf7)EPzi{$+!r%O>#LMjRvsLaofG
z6=f_vx`0QinC`y>vx)m$m_Q5xFHET0%Mx-Z2t+S7{-4P_4iCCM8o^sQz6Z=pWsqi6
z3gKmwbi>zjl)zlQ7=?`D#8d_uW~+$~?g$A0ScSq#gmozHVjXRl1@Qw5ilw6IDZXG8
zth1Svy?U1^i&r`3J>T3A+tO?T{mOB*adUcCF49;zIF#N|
zp=fAoFxt54UZ0|B{{UdzGz|9>5Cik-4Q?yeV&)vp)Uf!1zDQ597WEWz;uYcmR8TNS
zPjdV926nGh2U6o#(w_k-z^zP@eOwM;IE(R)(@;yGh5b(ETXobo+M(OxQ`4Ein2yAu
z1xzI<*CwQ^W{6%D9^iqaduD937_>e{A;Hypg2EgPuyclqvFlR(rQFLRwj9nTg1{Rg
zNl&Pnoc6_SYZOW<`hx&N=l*Lmj__hPsd;E+00ljmmrxxfYk!0a-IkoYmaaLBL4s}=
z>pYQxUsEl~xQt@+NE8H5P`jd~beR*t{^C1AD)B2*{mkSZ?pWRzZN>8Hr2{5KM#^uv
zTD7TbojGO&)b|w`%ACZqu4^sjIa@3m!@l66s{4Yiv#6=gH2XoxhH@F|SsTN`35~j!
zkMW;zg$po-y~)J)pQJ9^AKd+^D434WK@;PsMPikNS(V9C^iOso+{OAZumu8M!OBzz
z+yqA>P?RpjHCmFm9e!R|4LIuk)(aaFpOl<|KP9@2rP+lVf?l=Ueeh>KP|
ziAG$o>ZSw2#jRl|d&fj@0RmJAEsi~^dd3w`GuD3PP-$;vgUAB{RdhpVxQV#kL!oL}
zRFpIMC0{G@!|Gx$QD(xNLa3MH{1~{GnQ^&ssDBX6<_jE4QE)&@d#G-5GC^32aUPT1
zBMrEYElf&w4%w+zAxIo?FL&Bp{lhH6TAA);>1cPDHHoy;&?jihVVNO!D-AGLW&V;=
zxD`r8_RImk;^Ol$Q>4T;>K+49fheGL++qvqG>zmT+L)MpPBOx#7|pEN1}cNMiBo^N
z+y&Rk61n#U68A(Fn2_T@BDgt6$&TmDK{)0{aeM>x80NY45df+{RlmQ)8q5d-OSI)=
zG{11!xdK?&<1sBW+X~d-Vr3|<`j=UwWy|#}ZIoY9fE*+t2J=GZSSme1XnHTWt*bf?
zRQ`}PHpj$tW5n5l*{NSFW^jJY+TB5@MShE_-N6+&hCCqQExRLa5(X)1Q*aKQI*6MO
zh^K4$n~PmVNAmG^6bP_Y?H!Z+hrc%#E+@Bxkr&+FVj_=;LeQ=l7lJf^v4Xu+wMG4C
zg$f@L%5I}9yXZ#k`sN;2a>%byOj|2ZO}UkU7GGqq8lBWG%nLU#8M(WGR3KanYY?zP
zt0AECQxlB;00iELTZGg})ysaO#ol}!^#pm25sl6Y38l^@W|{H^HloFB8cwE=Le<^~
z7_Vkx%XK{wTEO;Cw`o%%kA&NBJ1Szz;r<~N-|B6F^~A%W_bPt65MQSgDIO6ltl;K3
zR=v>T18fqlN}F$i+bl;F7*oR@QCiZ}F-vU;YV;maMCuEIIK^u9D;Y5Xq*Akx9_0kd
zeXu;Dli#?iUKDzrb8cmj2d*Y%qd|gRuXn?#)v+Loa_x7B3dm?N-;pueA2Np#^-EUn
zI>pS^Wy{RHPt0)4{5&ygEc_Dg2)dsH8|Z7pE$6`65!ZmRg|Q_VfoTC0Gma%_I%HuB
z&&Fm6k#^_10o?6kVAi{3iUAQ>;K#U>eUkAAn}+K(ObHozRJ$s#aR{0mFSuW=l~{JC^A}gB
zwV0_JIGk!fmygmtrIOxTi{hpS&`~eAv0{l*;-n2l{LgQS*tfSThjRG`9_}`nRstqX
z5aoR_2Nk~D%l`Pythv%ZR%5$n5+1qS0~_v8YgbbF$cuC5#;Ug($WxhKW$ed651QnQ
zrqN#`Q)PsvlUQ73O(HO30n!P!A;Na1$LL(K7t(%XTGfJ)Mq#W{ljRqdIGxkEdxi;6
z0*OtesvSR!dw7lr?*9Opix@p%n_-(~w4UY6#RqJpX)_zmUgKUeZ@GHes60vUAT7!v
zI0M_bvzD*rg(!7qqHY_bOx%#Z(=Kz|zV4EMfqFm_0&$y>wyMS$C5py{C&so_9&j=dh+^g7*lwh|UN|kFq)kpaT
z?8D0=m8r{z4NC9|;Fv1rB+DEpu<=HJM4d&{aAhyZ24{(mOGCDaZ3?_20O(Zo0D6Tn
ziwg+08X;i#8Z9F>PDyd9%EZ^u606PpM$;^HA2L*6_Z}g{E8#s8m@M@Z-{PL;Jw>dV
zKa@NPr@2U9gaO&xEqCDIY+M8#LNF%X(S*36_Y514QQHsPpkNc)H~N>H%c!)nr%)E;
zdb|X(;tD21a^~0&8%K20@aP7^rHM&9H8zRYBz(STCuz;^TOW8;U*0v*5A0ean{z
zxJTT>=4v~hjwXq6GJ|IBCuQ{m5pWTejp`^;#IY_^FQG8r@%REF!lk&z3z+K*GKO{)
z;v)feIwt7Nw97rOLp2cq+n&f`w|7r74-2kE{-Y{6>O1yUU-1c%w%9|CO%-g(8?REb
z!SF!fl6xNz>CuW1P)nVbFIkxsQ^A%PVm2x`OcCc0NmutEFyg@9xP;av0FQ`NHrEWU
zA-nKs5o;4Xu2<>Dj&?wjzX6Q@01*30oMr;bI=(1k96{|5gvQsezF5~+2rZ~1PnQh|
zI-C0@NQP4fc@YLFnytX9|^DqV;7euB&4j2H&)otz)_`6Cx2L~F8
zEv$nV1?lY2WzeSX6f0{t+`l3N0k<pj
zZpw!UL5sr?GFiHrWUFjdUcs5a`BQnM_&{|LXd&E=Lx`xL=WJu(1_%lyCH`XKCVxk=p)E&Zv#YD{ukzS6%n&F24Zb62n
zwsQkrmLnl;7FvrhnP$N4U-viit3Q$@Z7&Alaexj^f>z@Vi736D@=BYBW^QbJOO+)Z
zT@0}~THaG8kDhl0}c%Hc8VS(#amBGH4PF6fn_
z$g6VC5~xrydyRk>8+r;MFO#77GaH*S5
zXy3R+7SnnqY9E=0GTf$RXaP6b09043r&+n&C3hatyMlu4x7@JINBUN
z122EHoEU7W+Ho@qjgZ6S5#Iot;*!pR7VmHkT~6pQ8J1d=HZKGLZ)Dtvt!2omVy5TM
zxDkuxm8@bR7ITpWsH)h7(Cq?qWv>!T^J+5AqXd=&VaC?PIhh%J@o>YcGIq?rdjo^+_a1Ld?;#o{CTNfMx&Y~A(
z%8DgBMPM!i$1s>9_oYyKmIB5j^8vYK#pv9`cs3BaV9|vgPk{o(7t~)kLZ+@*aUSMr
z$HY-R^;0OTxQh8c48Zb9Vvhd+sp-Jsh@z}ya(zTfFhzkZSnRq#nPUj{A;&KYW%DiZ
zsnZz27GbaI7jBqzGUWG^kHm18p^(x{rq1I_g1?qkS-Xp1V3qQ(xo!|}(5ldA&$R6f
zQCF#SOrT9>Gl;n`ygUA;njl=jxt1?-@YGuA04aS!f6FK4iHv54sY}1*FccpIMJzjq
zP>f=s6s62-3l#7j+;WI=-rk_Q`>ZIqh(*CLhcczyS}~-Gqb*B=5|An-#WmQ?!2C>V
zA5i61A)lH}uYe%-<@em)6V8!BvgV}?ekJLoF;;{KuW<^ln?DnAF2^>7O1>A|Df!ks
z9t5R=
z2r*1y)3g!o4%mCJEwRMzHE>9$?_#gaQFgH@)HcacjCk=1=wSpY&90?P2WoQ?tk%mY
z`oWt}Ytx9X7uhWqX%qi-LNzR1<@N0{X|GI*Z9)`b8-5D)l4R^_lfE*S^7$b
zS=2Eg3@mkAuK@?7-XnwqNfa@oa(qQ}1?u04rvk}j;mqFNGD8S(^)Pd(
zouQ*pS|lTeIk{Ahd~!{akYfJ;@=>e^9(UoEq0B_+@RX?t%NtJ2M#88Fp_}awsNDs-
zSe$WshT-6?CNsevhZ7JRJQ4)~!miiq8KKJ{wqJD{LaT-$zGpg(gXNe*-e8)Rqp973
zbgBZq$NNYbvG%YUt>Ee!`IWkXKyfnZa)FP;0gTEja>FXv;q@v2rGpOj+IaOfxI$%y
zSi@um;}&|zw?w;m`FkgC$fJ$Q#Bpd$KBEfyI*YXrGQp^IpK(Kn_=j+XcXK8Q4dw%x
zU1k(=#(1LAt*Jtp`s|k|4+AZ5TUx(}ZLl#*%X6w>bWYA?O+`3|^KZn$n@q$QB|WQ|
zn^*WQTx1}{H~Wa0i!&iCi;cMZQaENOlAsaCVJKL&6^GJrsH@-(Q9@O0m%t2#H|`i2
zHM}YHECMlxzG7nAbuLiQ%|)49Ad~1rGQ2QTQl1nVh3z>-co4;`+su>4u)iuQa5pPV
zrf5Af!vsNUyNp>d7#oWfsDo0PsQ&umwh;N_(ylpSn3X7IpmjB
zb?87o*dEd@yCQ(8p8w@GM;DPskaRu`2g5)3Z3~
z6v?R2xy&pkr7e*lsd8pHj<>!HzEPtt>Ln}tpnxd#NntYqPU+O5zqGG|+_z#*WHgKk
zK@!=5<)nK|2uD&4loT%2z`%j+KN*uzQQ)}T=L~Bqy|U{N+{<$Q^B9b6;{se`&yqmK
zDw0}!0n9;Zb8^8H3(bWd?=X<0SEn`ZSi!jh4#Yi+e17{x(J1lt{R99--j&fKC|hF7aG!UB7VS|9nC)(h_>uot7atzfnD
zEk&z!MM2HQWsV@5ZYmuT%Z9uZIh8EN=8^fc73v-SW9l!No=NUNAXhFZmpfQE!YX%?
z2z*Ovl*yitQ6P=WBZRKd(TF+Bt-0?bE~6Qrm=|QKr!i1K+ce7zQ)EcGGHxz*OuK~V
zL&ZvcNabqwl!PI@?K_1ql3Y#>AW(6?nG%arG~FM*t&jVGWB1(MLMpfAm)xZdA(4p9
ztC+V!nYR9A+w4sciLzLpn9OsyimG@y%W~>#j^fIe%qCh{?k$sS!h1sHq8ATD*a@gb1+!%8B3I^i!s|Ow;JU(#&6e#z1=eR
z011#Ft5m^~KWEhJgDyk}P(o^0{{Wcn
z*e}B2a)|i~SXL`tTzVTn7gFy|s10Fq>NtWs8GNlY{$LXbcX1eL3c{HoZOUp>l?*8d
zW8@VT+{wvL5DJf6qsEYeBl0D1%k-H1O`P(}pHOkqS$s+jmvtzQL|J?x?{gDOVBr^6
z1ySXxfMwT+wCVyc_M7lZ$AKR(YttLB-N$BcNY1>uK9eyFfHYtWxXEs!lLPlTnfgv&
z1f|FLH47I#yu(SH$}NUX`Fmr#quDG0?99dNFKot(c`6muSSx
z1Tf4sw5l@H5f|LC@#fis5vEijq8VTaVl1*agz8Hqh7O|+3Cwh)s{RAOwK)gSO=_+>
zidMOqcu!DrVbEC3M1Nca-(AL_rFG_L&s+#V_Di{K+}zlmWacs~gv?9nO@Fo5!#yN0
zIgaUvI`s()zzLER)Z(Uy54h%M{3e-^6EdYrhLky!N>pjVGjf@0il@wv%(}xIn1lo7
zu*`^$!yG^24hUa^;#I)4grTYf7oTh`h!IS0UUrAY2
zM^PxusZb_OK*6{wI5D!|P_wz!)`XY=~h=if7aoP7!UV3H3f3
zWxO~D5{;b^i4il=hmoYLV-!IcBW^JJO~NUCOVr)ABoQ^MAknHRWo@WjPk|X;WhSCwyFn(;Qn!y4r3u$JWu?8xsh1qUsq%;;
zoK#rhDGnt@g~REB6S!s#*XDj>{{TdIM1)_0eLfL|a`NJ6$(geSn28Gn>N2@XmjId$
zcd2=eFtuZt**fJ3vK@S;4AU|*_=3lTisPt@I-cT?x6)DFA~IQm*~ebxI4NO=LscHw
z#<#c87MsMN9YWB_*mj4^38`>aC^0GVQj(rySC13;4Qj?4SpS8~E}+mdVsA|L&d*wo{=iliyQ7Tg$#xrd}0!E+Zf^)4vYTtz}G
zZblI&)T}P9E+dtQqeSa2@C_aY1~mzLwidcG58>A`_WO*Gw`8$hmBhR^vNFeyXf*gv
z(w&msMAggp<(93+`-(7UUglN47Zz2u5TQZQNpaz0!D0dKV=tM3G
zCMJ?TV--cgrpbgKa4$bpsK66aQy%`ig`SsJ7&(HuxCR>M6Hqgl?h@{FZf6$tG)EHx
zxC0zp(It`fB~iQ;s6N6|EnUxU;ch+GhFpAX+;aB`K#u8Bel1
z6#V8XTDRYUi)q;z?C=j}Hf;(Zom>gTvR%aiZ*sgzXbQMZLZwfALIrYWO3GSZp>HqL
z9&-;|&1yME@twJzv&28eOP9i%mk}PwFPkhol?ABL#8SwJdPfOpC>7?i{4oXqAZAuP
zsXzqqVt9As)!K2y1$PldsHfoH%mWtYVVt1l#4-W3ZJ81KGNIk%zUDR1WrN}of|v}m
zD%JB-C;cf!GVoI)503Tb-cSukY9^EODtigHMzJY@duj+2;^wC2O6KJ`%AuShm5773
z@qR4Ms$g2Z1QkGD$zvMa5%<9N9Ef*un6|5&Uop6mJXEQB2=R|K1WqPYpbKIZQuFF8
z=iZ9KtWns@hF@>`ge$JjaZ?@ryvz)}$Fyx$UM1$qK`CZBhK>B8AVY)*=wV=>AdC0{
zbAPokh4e)8u$m?K6Fp0p8SyO@Sg(j~6?3ViUo|leGV;x)(?+F<;#3|YH)C1CQbcLK
zBbBttsSDTm`Gb00ai~~FPgsYO^6Qy%(%e>dO0zqdZY-$ysuH49JltZ;R;^_l)8KEw
z%kw~sh&M|$?had{tBbAr6M8i+EV+1#n4sDkAo&iZ%fufBkhEMx_zjoik{ZH{vgM7M
zircusXty6zVBF1KSMSuNtz=BN2N+amgD@hl
z3unW>7ntAb0%8}GTZMu)JMdrOSA)M2)Cib|IA4$TVn35n@07tvwp4J&WuSv2Z@7baBo0xkqGSZ4eFVR662Cwu
zF9feQDnFv-?r=qAaVj-a)5s6xQR;6EF9i3?VEJDpS25HKmq_9pFX@)tdSEyS?j2!W
ztCAtW52?Q1tC-d+h?>s%*k)M^IGNG7_9f&awc@4@51dXy)ytzMSuevmw9-2Bu$zNX
zi-OCSFUfZXq85?$EoY)H7n;J4Oa-Ry#Mgn|gQ9q1Ufsjmi(%ZjDFScGxLaHNO($^b
zI*PQRi;szd5!BlQ6S5gUi{MKTcWL;PZU{$`enTTBH7Qk|t_)lq&vd~VigT5fdOqhW
z4a8~-6@(;w;7T)?TehoFB4JQ9|lQSl+8{jF+;%2sQEJz
zrAjd2l$L8m0Vg2AYwOKGNdh~V0@~_)doPIPBX!MdOc%ItIK(osPtHLO{Q`5Q6R{Kwy^|%vA~cwX$)!Lg<iRaUL$M3crIJqSGU5YyN$$Dd|@>yWtv_cr5N=^^W$;xau2~0DO7Im^C%uJT)AY&o0TiYK`I_&
z3h9*ll39%*mIsHGd=!C(+jDN>X)avr;6=-qE?l{C<;#Omzmf77zu|Ka0W}=OUJEYe
z%a`Ki%a;^jmjYBtaYYSyxIO}D;KIKQxpIQzWtaQ`@JC(g^R+---vI?w#N^}VVDvyh
zA5xB21wIL0p{7$;Q1vLw!~AYsUo3nb_;i{cMvop-9|qqpd@%59{{W%mpBz3P{vq(r
zfB2j5H93cYuMBnPJU_(wzr$yN{GOafv0T}QC}6TVBAO4Vl^7%dL&Zuw2!pa=jCu1P
z2mb&M|Jncy0|5X600RI301&5%*hUOv&g&3x&Gwm8h2ch;*mmv*l4$2DMV+pp@oaGm
zWJHM0IdL)F>3!fW%?GUeHInus+AT`s0)0kRua=B4n9%4iv9B`LDR18K>H*-TA%L@7sbF
z{>qboO}8-d@-6YEAR>cZj}?OIi~J;dA`F;592K7uDi8;cCJItpE0vi!9^6>0^e
zOT(ZUNjDjFWuZw3Lm5!)Utn6%a9tFLbvo$deIx7{4cMZ
zeR}X(y@PJ`$Rl>hm@Dw5$HR({>dK(>;CK*>Xtpt5H^RohqtvJ;RgW9wLq~nBig?RY
zF5G0STuE)BJQfo|p#9uY1+mNTbSFl7jjTrR@^Djhgr!j36OM{oCw4fVk*H(WOCrAI#=0B?^vN#XjZFy
zg7ygpdK0mYGkpNLwp0vV!2I6+X(U{YH=u|XH%C$rcg7m&;iGEMob}&RpJ02Px&gw(
z^3$wP<#+iqb>BIt?7t1CR%oe8V+bHcC7!WgHexlol)@#Z*(jvraavn$`XGe+TL=sf
zrF>7=I2f!H;wYgRr_zC4mN*)-Q1X^0$z6(wsxD`)3Zi
zFDXC`e@0s_NxBevn^bhbM<#KPzo?jjs%+jLR5i8m{jEh~Cjy6TW?WOx80@`W28e2c
zRM0Se=8(HZ+N-;KIeRtq?A#>vlu4oa6mZ
z-Mx3p!$+1Zrgw{SO1(Lfg#arE5KJxVfOYa!JWkUZaV0$e03wD5QRpaTa;C#l`rs$y
z{7f+tev+&%eKGM9lsB}g?fojN2j|oh)ENw>KIi@b7ladHH?PiPrg3C3zSL$JT`Ogn
zCzT+HrO+;bp{O&`>&1om_)kGDbTNkauVf=3m56_cvK9
zz|`<+-*7*ZvqO8Kh$n#EdE@xnUMGqm>{)xE`O1zZJithKNTCtV=RZ;2yLIeUz
zz$bTat&D%nCDDe&yTG+NiBYd|jJ<8e8Y;r6+tQ$2;on*JY-O_tml&U_!wuux^qNiX
zJM2l{c=*O&ikC6DI%g!V%66VS3gw80@6nwxT~)}s=X(xR!b3maFMGs;zg(l`C0`?&
zr1nF^f$qQ^sPobdNbv_rbW83bQ&)2Lq71L2rvwJ4w9zC|WfA4#1`4;X?~IH6WT_&C
zrJGw%>U>Vk^a>@U`C`-9vsQ@8NSLIx!v6re6v2&TgQ{+wzrlCaOAz)`kz?CE@vdza
z)N?MeaC>ftK`4f(43z%>2OF;lcu+jb@RY+D=`(iUt-*s#6|kYSnF_{U;@jynaD2(2
zV2O2M-XgZ0{{Yu1(}YOjtd9~nlMDkBj5Z%72r!FtU0uzZCAM&DJr28l9Xk+}YRD6E
zmPE8cy{g#NHPoqVj4&5N^WPx!0D=4fqI=-NcmN{+FTQcdzInhy0pCFUCm(+k*K@QMAxNc*2rx5J!rJZ?t0nA6q3w|coQWA$I${Dfwl^(0h-B|)i3;SG{4P!lU7JMFxlYDCM*5!^KNr{J
z^D^0oWNUJ5voR+e<%86Fr-%|$rT6=*sNHFl}Y8B@p?UyiuIx^eT>x((Qkk`O8@$KvVNiNQ^
zbUk@ojj?42!Qf_0BKH7j#w}Z|(_%zo086psBzT6>c*COxk`u%AV*GyXyXqY@yYcQ5
z+>ciAWgjg23xw>m1Zdj&BxGhwkgR+H!RkL2dXRXF;%7UP
z>~_t1um)n=wgr%QDn`{Wt2O9=`U{H*9xZnX;bScDH+nuM;w|`ZoqDhs5R&<{
zPu06`z_MNLWSh0K)%Lc|HV&5jLZ>mq6Uha4CzF<;*O@tl_v2+gA$o@aXq?ZEEMA~`
zzGn2G?z8tP0f5#C&inDzvaC5R%iXQG5kHIz)Ixh)
z;d_^g@d2g8yPwJIDO$3$Ip0;IvcX
z0^Lp)mS9PoT>F#bO1W_znqW41B)ft2bGhR+;_Z#Xb+~^20Md3|ABZe$9xbOgaW&H6
znL-#tUms|Y0SzC<6s|XoL(Y1&Sbln
z9I}QIGyMoi9EJRrgI;V$CwCi>dbcms8*p&XvN6<|9Kr+F{HJ#Nnc-s%F7~n=L1nq_5>PFp
zjniikk_qHxw`Q_*iBe?V?l0U@KjR^DG7=8}IOYS@yyPc4Oc8^&S;d5mCVY4hc!g=*
z`n&Ulami@&eZXvko?>`I`;mu-_dN36^UKn}_1)`V1hV0>>u<%I+_m2yFR39TyJY&m
zcIQ^_J_lYhSP}V)iIA=C>9Y@GY{pp|gL1fo8yA8mM9Vw3;ual@L*q+kJWt76@?Qc7
zT$smM%z>{{9mTiqQ*^mL9PM8GI7}Qx$g=DAzfkRt9L{>Vv$JnGJ=t%L%#$nQXP)Ja
zZ9;f_CYs$7sVlJQdXmfP&m*wLkg+9Bt9Qn>=K>Jl%0F@n&~!m{>cP
zUM`_J?hbX&h~3CzCh`|T+int!@t)as&%uc&4(vqpBFDcctE3%;Y#EkWV)ZP)cXz3A
zER#26-({K>Kzp}jW2t^@-w7RfV~(KS?xUG=iB@$Zcf$1tXOk?wNh!>4koM0%mh2Wh
zTWL8>T#pt>EcGD+)rTwqPj-070$U%u?oi>k!3!U|B#)#%$qS{E*ngCm#&B@PxJ=t*
zoX=P844;In3v+_V>nDhp54b;8KBJ3#?E03xerFq!UCu*jbb^>>cj#IRej$8*VZkleCF0^*=CVJnl~{nm}0u5Y|_ge||y=
z_!|+)Hd~R+mjpd*mDWhL@B^inr!kzk!>1pH+F9^J5J%`19^u0g;PO~Bj(CtJjcyVx
zmx;l#o`?`V*g-$S2O)6@)?0L$;%9wY?=TLJ1QBzV9s_1Jd=SsVOQ`r;+An|t!wPUV
zAS7rYOFCm5H_}_8bJMxw;4P(RsJx<`Fu#!PFATZH4ZLxy_aCZuqkAi{zc*{M<3iFQdm`
zT>9Z$WY%`Gyjp+ST4q8vbGS42byS6x-V%{Yt{ZEW{XFT1%b~*3Ie8FU}+tChDPFmqXtU7WaN#5
zj^K*xz)x=(0rMSM4v8FQN3wm1ClXQF<9`6M;!X3H0|(WK-QR`-b>JSdb1w4QwsEXC
zxQ@OL9`<9C8>!}Z2dFPryfS_~Ws{uwFKK)2F)j^;dAY1PmtPwUo`}x9+;h}eSWkp@
zA@973#t#f!d$_mC&5UMGh+8k;!%togqo@|l
z@WY`ckuZ43A+atAeX^~YGHH-_XMqD7eaY@_eP9}R+Z@~nFZQ$^rMBM;!V8bbvRWJv
z=Tp6JGF}+R5~Ro{+r|tzv}Z{TQ}A45xLS(bkm`2d3H&wSWU)5byIQg~nK`o{9BxM0
zB7ZRSdGWF{nQRUV3&o#?N#&49bJfg)v2ynuTE5Wj>dym?mYJL|&rp3O$Za8C)EVyf
zUq}NPbMEhme^MU2Z;r2m4YF6m;Nn(T%0&E@!Y!u@adhqv+i1;faF`aH&YhgT8F2gp
zJY|Xkks#tS@V$2Qqx>KiPZH^Dp#2tC>HWtLgsJ-$^W
zBbPRgo9wrHm(};O?`;D)usE~ES>s`p5sw)YK5VkfEQ2hvW?2Q6StXtUWtLgtxonUY
zcx9GB<2+
z%jMu~Ks`WvWC=smFH*Mr+Zdaer`ZZOc!n&$&-bAd?)tKht~|WI
z`-^=`IELHCTU+4>aXPT1sj-gjBPL6FAZK>wSCq&etUM%P(Q3q=lHIdp9x^#|Zx~PB
z)uzbXR%~>+AM)iZ=&d*I5?3w@W{gMMT8k4
z8EGcmb~W+a00oIVYmDa7bKp8d&Rkv|gzkDu`<}mWo!~-sIK0N53xkNx%x9_UIbsM*
zB0#(rCtS!(VLjcI@?mjf978)~_FH}2!=9xl69feYE}JaAH^OwjLQ!t*cZor1g$eF*
z+=plT-sD>R%cnN6lktFQ4r;;XdXx)p^X?nRW)qjW%!CWJ#v~rDPX~tNaLk5XAmz!K
zxn#mx4x>B?J4yKv`j$8?PnI5?@vPYyXKXy*+W!FT@I!>V!5=C5ha;I8%y9oBA{{YK_WOxfB&U{DRvCK~;#z=rmmr?5-(s!E~4}rvqq7a)?*XWmq9YH}3
zV%P(-!yQj0y!7)J;JI8`c|4sWVB9Cc@a?vIp5@iD%ugmUctSpNr~d$YljNV-`?AYC
zceumg9}w8drTa%$2x||B?{|5Iox^hV8v`sb5x5C@xm+Iuq=FU-2Z93a{Er)bqywuP
z2Ur_of7uVrm*>D|z~SIW5!5a4dX9PUgTvIp!`D%QAaJzxbMqvgID4C)Y>p?nERO^o
zk_@^`J{yI;i?qHn~
z5bAiw%Nfk_xdTjLH(Uk)5ky2R?TXiw4@
z349^ngP1v&S;fn6Bd9gLH^LZE^KLyezU;lWK4SJ`n_kO_q+2jH1?op7Vd-JUH!>rJ
zGfWxBeRw@?iFvtuiPwX-e*)tPyKtI$&j&mgd&p*4obdykwqGX7EG})oJ{Wo6?q75r
z)tz}Ik5?;xu2N%st_>m0!sN*hBzNVI+XKZ)43N=Kr%o$
zb18E((#BePnC1xsW&q$rpGiMS=6q=&4~wlimTj;b0g*7+<;BZ7U#K4L%=IIzk3z)a
z6yn%G-tMfOPPv#Px$DE!P)tG=PY!MR;>_i_U2U@rw+IKT?2hh%I$a-fH;x$*yJ2v$
z_c+W(F`MV*Kjafv0kYmMBzv$4gtttG;T!DoIqv7ic;hvY!>15uHZb#NdAHN31@WF3
zU9&lkWdbp026!$*axCY9g*LckW%JT)Evd@8Sbd^$`0d
zffPHq8vxCe^F7CK6T^YVGbgz0j}0LMAe7eM2M-p$;8+Jf19_J&Ui@nuws=GW>J8`d
z9%Gl7lpS*!hCB{Bf!}t9{YS)>MX(MdWF{QuP$N=hK0-KY%RA=8a}Lh~mKXOUT*pqQ
zczd&vEI)APv5aEkPng#-bpdj?MjRIG&l~Y?jQnx$y91v8042j%>N)=a
zR~qUkUReDoH&KTci3Z`07Y7!EJ7k^*JLA=01(sNAvC#jNu
zmlgpP?s9N5zU~g7pMnkcl9%0|;Hgexr
zLz6Eb@&5qr)BaAl~g&co}&}*R{dOU#IwVeFY0oA
z_|>hJF<6Slpvli9^@U^dGPSh#BfhB$8z*WIccPDEp
zlW~jz$B=wOo%vjx!>!r_mBpM~f7*LPY|aZTCq3I7Sll|>lLd#ldRPoR8v>Ev7g2f{
zek{4$PjS_-@jXS@F&57Wb7h$ulJR%r^P#h}vz#yY3aaCnkJZz1YZc%t*qW+@?zB#C6Y}=ZH7EImvoF
zHru8bHZcQEU{)B_#OZ`w(r9{E=ELM
zrOr#0$)Wp#n|{1xn{l2n*s)^9cx9e42y*Vv0$Bs++2diJxVm^;A{>ps9s}kMEp71M
z8yn!KbEq=kDYj>J+v2_(G1>lXCbGlo^{3#xdB`oYe6k7RIkP3Y@}4`g%O%4l#}@3G
zU3qzgE|z&4Wu8{!*4uvs-y=^9vC|dkcOlAgY1x+#P#c15vdN6WWVyY>-A6Mf+mR`L
zXNTj5a(vF`7JSL^1n$W8j|}P6zdwk#`bY3i8~x?HX)XpF?)qB>!Jc+apD6l=!b=k}
zb-6C=@t#KV**-JQS>-2#&QFZ+c|Ql6ENt_ZS>!(#z*%LUa^D6l@X0K)_mE+h%MSi7
zJZ;=g{AHhm{Q^8c(0>ANe~+&Z;PvJF1?$7z>N@aVuHS>#;CTPU045Lt00II60s;a8
z0|5X40000101+WEK~Z6GfsvuH!O;-m@bUlJ00;pA00BP`tzi{3XK|K$L5Nio5BikF
zj6uZog3>Wv5pHy0mUSD-brf({?s}eLjAjZ8x)*+;8)gfOLU7=yY}5(}_?hOSZ#jU?
z2iy>_!L*6za=-Hv!Su>qf$k|xOZ6{$+-_#0t<>#c9LiQpi8z6*K<^Q`sMAd6AO|A*
zg!r1MYNq}o*G?hhaCNBo<{b#a)e?YD<_IH~QHB{V0#O%@!4xCrC~nzg*BneXs}g`W
zS!EdkPF0OH6yvyT2qEq{c~;ysI}8y=-#KE`zGNY2<5AF*gl3ATF*T)JUR6@$a@trV
z%aO&k-#{xQw=G?q?>OY8b+C*hM@Uz~Sa!bh6^%tiqtA2A(*S
z0Du@<9Bgn4ADGx?D?whR1+HJjbJVvwh1PIH<$rNN)8ak?@t}v`*gpg_>{0`wlv_m
zQ#B32ETD{1SAL*aMIFEp3rn~InZhsPbz15@iN2t-n!S@>Gt)3^%D$$YY^xDUks5E0
z?q#aW4HclwP1ZpKonGN?fOo62vITgl!9xHI7wQ7(lw}N%V;`8;GMY$u>LDKIQxRL<
zBOS+x@GM;64q%0FJwR5jV(TCvQKldVGQhkh8KfpUX>0%ugL_zJ;-3*4M(=fqy#e^B
z_{#AIJl@oYQ2xn$jQR5jL6UAiuEnvJh%1+VVUvSMWLhap%#qu&kIZDP
zNXmntMqPrunVUCi_XR-;=6sT&cFiY}4pJx7)LKH1aaetvOf|4CWTDbrEYA6qy`J2Gtzx+BzYtWz?AsPgZ&9I9eIk$AR1;u(xqE5j7v^?OQ5b@#@ghnky~BA0G{nQk
zNmPYe#@89_FcT}OlOcgXWS9;j;v>k^R0GJ$EK$vWu2Pb$V64B%GY7*L;2px@4VU5w
zb$%t=iB-hR2H*>37{V$S=4?^yiQxAoOI3nZ`4apBxrQD{3kmTv1wBipxYTh1z#gM?
zUS*0#@fTFY?mHoitCe*RO~UUo&VO>1+u|eJ+$n5%h-D225ZLBb%0kBsJkSE9BeoMk
zyLybLQ3tQYqLcAo%wQoLm}}e-YD7`4_UZIYsPNi;qQ#_BB5vXhvtT&V`{DqxO5u+(
zBDFKnKB36{LD~6_lNoZ{JAd_t-M(v|h$aYz8(&u)u!0RjUSO1%62q9jpK!D$E^eFY
z=4j~oi&z@)!qG-G6<%0{=Hjl1YZe^fAV@EnY5RaJxa8ho
z8Y43+!NegY3mExBU!+|_n6$MXBktoRmKD0CVc_ya@Yar4t?A>-7T#StjH=zKXxV@2Ezs2YoSM+$0TH${w9m$Z2mHgdAM433nA*aPq;u7KLk#wx21K#%`{eGW**Bsf7h-6^R2@sknlQIUn%jP`=>lZ_GzXlmZd#gcL5xuV2)}V)ViH
zmh3v)^Ft8H^&_6B%%Uome-J5)x|TS>GgX;?ij7-bMny`sYBV%_%TA@%M^qB>*-=J1
zlm)!Z^#oCuP?{ECa1#b9s
zh0231Em{fV6=Ijc6F$7I8)r+Km9D69P+B49%g>fYa`hcVhF0l6;KkYwD7`$B&%(MY60*v?AMZ|2`s_Fq1eGp
zA=GLhU#WK{Qmfpjgb#NJ4M9&Z?8>-ZnZy^xj28)TKayNy^%B5U@<5y*BM>}Pr#Q$F
zawl+iCAFBK0jQKV1XMmkH*|+_b)^$Fgk}?zroCLbLDe}W9L6l>pWtOsyyM(n*F3W<
zHRB7F3iWYY1u3mciXg`vbtfJm4?uchD|a6rU|W{&RVpD(e4gVQBdu1QAq(pgQ8wMG
zKB)N8JfV1SLYYAE%Ea>`3{yBLh0N@ah$^QVP1S5cQumSFV;I;e2Z5#5Y
z2+sSEj+yuqls{315t@b;2=l`#uHXsW21~O!hZ1=L7*d-fDo&9mu(3->REiiQmHFhJ<321#%0f|e79-uIpgFp`wlsAcVbyrihL)ht
zipv6%4`k%6*@&~3xB`}UFz*D=j8qgen5?Y<4*^x6BWNkK$Ax3d0{|586*MtwsyJCa
zMsa7TR40}-dncqtv1ZFDvGAe_WNeJhf~W>db;d;6C;>;f1u}DnJ8INCz;3jOEy~dL
zKl-Q=E)KSouUbKEfv@rBV>LjE*-5hd2_*Vic{h
zGa@X&n40{eVx!XGg{KuWZ@QTn(LLduc5am|s8g~g?B+Qo+N$*}zU6kLpbL1+7$?BO
zDVi=&#}TLsr)B`#PUG7bk~x8KW)H+ol@H8*%X^9pgNfk1F{cCSBIHi(_<-#Qc!r><
zTE^gkfZPtH!{EN)(v=ytVCocmeBTbEUh#Ox^#Y^izI!A^9JtgDFVVZ`#Fyd#;N=08S%1R`Gtse?P&@XvU
zl@vzgAo&0Vj43K(BK(joqgacw5bQ*^E%+guT8yG!QiXjkYXa30l_oUE+#oKzMKHZ)
zEwx{Xu%-79+OTt&2TU1ZHl`Ce91-rYO%_Uq>IP-a;sg`8<-rYJQQhuhJqxKtP|hjp
zv0vbXz?piKJ0K0A+y^nk{XqsRxDFMTL&=`)4$qioJ9&f+@eBnA;wvikD4?G)vXl<3
zi#2(Z0tjL^J$+qNF=b^7DhZ(ZfuI~-8;z`bOWf*V0#+8JTIW0n>|
z`J!c`Ogsx*q%$O46nY~Gw!)xf4X-mK4`K~T395${GfBbbS)XBr0kOvfs4=|e1bw8S
zSx#0YS_5HA2uF?TJZ4!wJNHu3IX*1ZOR6nJ#W*7)6`Nj3|a%tQp5Vc-MeQ-aSEXFMXZG-QU19w?P5
zNw;yVj=o_$Cx#hVW!y!TS(hV2eZY&(%8kb3Ar3Y7am`9QV5>kml?qGd6~Vp|7{yBn
z2gr$OK9Zl(wzz7do2H>%Kw+s?U`VUI3=KSX%(V&d-{*55$*?}R`le!@VtcoZKm*9Dh
zRrxA)C?j?hh#lyw`X(MS+!4m6L?m|xb#5=
z>A6Y=_$LBCaobTY!ihkmb53893jx0nz8@D9zd&W^DNxb}BXG)djKGaYf8huOJ7uLk
z63U_KR93piV|D|~VXDh=W67Ceyp;fO76e>TSYyi-aD3q~teT6OY`EwcA)EuJfC3pE$^iGqMEcNem~Fc%+*V;~mSQC47Cc!NiS
zJb&z?gJEVmvrvzqfqX(xhn#MZN~kie$>Ww^p#arKxCYfg=W&Pu4wBqerL!3{(MOrV
z{lZ=gv9ZOA8sGTEEf|MH-n{D-c7Emywe>RdJ`fuJ01>-PP}3M^!*Dr=f&)y}2y%+v
zZxL)-y+I`&_<$&_cNPE_s)@A;#0{#hVpQ430~=Lm=z+{V5U>=)QRWZ@OI(vGmEt)A
zy<$Rx(Fhp&ggHmlQ^d)Ojhoa1d5AP`Qrd)rRs_@BIFdsA+@N9Zh#e?YQy-8cKxw3e
zoWPGH)k~^{Z58SuI1BMBpdw#jtV5LpETAit5Mj9>TmZ^*?krd82nJr|5Fc{cNa|p}
z@fVd(aoEZRobxJpiPu+CRi-yGGT#1Ww>lVR*4qeT;=Jlv0eFN|`$K4gZ%hS&
z1d5{UW&Lv~P~1y_07nxMVNFUTuF8dhk{=B~0BelE;cJVc8mT}S6w4{O%QQ?&5wA5J
z+e4T8jsPKJoZ}NlZU?aMJ~K3-RC>;Qw*m}Y+kG)O1gmDyM|B?K8t!NX!;ra@P3eU;
zlhmdj!4LE@tjDXEATgIYiFE>26%@YB%1pK)A4l_usk1}%Ew3;JK)6QWq2R9G*yI7B
zwp`A`t&2kS)TJB^kxR3ei~54CR-)SOVnOmp50ZeXx_IpJZUalF09$Ba@mMZbq|Y5k
zi9_<13$U?9z0C$#t<78zYv~8cUr@f^q`QxCWqYrt8K6E~)ER_dQTl^gA=&YZy?V?P
zB%Ke;pm716d1i9I8HsE!h|L!L62uCtfdsSh@c>tdxk`|AQI@h^qa5)R%?_K6bX!Rb
z!3NB9AIc)zVk;mxhTY+X{e-A4UKXG+xha)&RcP#js38jlmNl3BK}%nR4OOz8#Yayy
z$r=W|Jdl{#q-)d_wAoA_A83}uK4J1uiZEZ5veFNTH2Gm8nr_>Nub!7P6EF
z%(G~&mMC!^GXUFT)XYf1Y>Sk2_=TUagtdH#B(eB}We1W|pxu^1M&HU?LeAN6{)DQ_
zXNZ>a8QT`B&>XNrY2}*7?GdF9Q3ojBmg*^NG^m47E&X{Dxr~
zn^Yq&YCJiO$|_~cmEQj{?w4W5q^byMF8D4&f1Y
zuAZlZuF}7AAJiIsZW>;`)WBg+n4vQ}cziOkS6*Ni-X_5%_!T8WUM}mYOGc<1D5+OD
zw>6875)74jhWVz$Wx~8a?gKy(zNM5;MV^c$0ocKW04hj@f0%n-W#yQvgI4sl4h;<9
z-^>LBSHkg81+IZZ<}KK_Dty6Xe4gKIz>JSRFU7#XOhaN;klb2_R2V~L_x#C
zOE|fd75jt0FI$`K{l?m#id3x~kz}j8oDfhEL`pl~;soWsQQ2L9*Y^xdUs=pWMyFwL
zbt3kflrC>YhBl^$M823-ASM~UBfg|Pf!hF_J)WYgt*uaRWB7=`0i
zQ1W04Fhv63Fa%pHv#1CSM--7r18bl>u;?qcACs66f}5(V2u;|C(yKj7UYK@w1fkq0
z@o|C~cZ|B!wc9>xa-xyWQ!K_cJ{>^=r)lBle6hj*0JAl5i-BzoUx@hc81}P9<(O07
z_XH2XzdD30DcIsEsN8fE<|`r}bsYHx(+Ye%4eQYzS%=B)0J#BpW&<{n(k=rglr_M0
z#Hmec1WqijKfi+^*8mLWCWw$k076v%VZdZ0J{K)PPB0OFf(!-J4+Vars!lfZ1!+jVhtUV!J8Cpi@05=2_^?rfxtc`8
z=#?k|Q>|QF71imLp-8TlFZl^{xt`%vO|g>_E`s@qg@@dFS?FSUpuDk>HI2hS!Gfdt
zOQolx2nZM4GB77|9sWaPV~Nm=h%=IlDKYdIK~&K|CSg!vXeBS>+ZRBE#Y{5onG||h
zsIYGhcN21ZitfqSgm*syEV9t?C@4l~F{cpkJ|NX_6BPI%{{WDJDvI$eN+*Lc4TIFx
zC&ceyk(tahX!jqn=B160F@#}e#V}cnANFzeeQ4Ztf0*)>RV(y`I#RI*VL|p_q4UIZpYaZ_)Q`#v?lA%@hlZ)uK{=_
zM{)+4eT_)|=1>82_Q&uD8X60KgSJvblFUtkx4EBn4F?ba7O>RF4((gF9KR5bY^?*`
zoB4zoz`KYlTZpFPD`C4nOt^N}s7No&xWzSlo5kK?uHH?n{{UsQ_mKYpnur9$%(c1_
zS+KjN5V>DuKs0y)24zajG0ADu#5@%*#S;9~YPYy+g0UQZ%$P=$@<-kr
zyLIXfw;PW4%qthXzou7kR*$&gT3oRuJgR2$BC~7GW=Iv4{{ZS;stqfC63C-$@z=Qe
zre!yb{@^QHP!^S6+$-?K>m4yt)}1kh2*7C>RUN->&i5L=V6GkYDu-`!pePhvwt|eE
z#l*RAC*k!mWnP7VavEE=$z@f+Qq;XmhA7**cL75;LO}9QOh$kJFFc3LDuyHWJ&=cw
zaK9Eo;9#bMA9FEcac~6D#vuy%(Eus7K^+zmw>KYxEuo=>2BkGKYgYZiYK6SNmL=2C
z`i^QX@pC9w$q5ZFxH#RtOEqi~+hN;bcf3bZ+kiAeO$Gs_P5@M>AepRl+-S<&-c>=p
zGmcd(CBdA=L_p>^KMAUl!bQ#pGNc@NxoR7EbC{{Cw{hl`;wWpo)Zn+ts7#`jM-f44
zd5j;XYh8R1G2gaW7Ed8E5eJ+=9hD5_V7C>##pry_IGko&k#b(37@M|5f&fwj1_rdL
zMPqXFHbvyH#i~X1Fb-LNxk8FPOhqr;b##=dn5{BRB6P>AgUD_Qv}>5H2gFAU9I(r?
z42LWFKP)$Zw~W>ya#HBKe1C8$@Xlm7JwUh=Kx@>(57ZS~ZAUn&0WOvR-M%2iteTLu
zCo?;sa`<>47zI+UeHffNLSrPYSQY@g95E{dx?zTmTq33249%y!bLz;HZ0i#BaZcWI
zcJ5z8ZxL9nlD$As5{3MqGO$1arp^BV61cmT^
zT204Nshb;u7VR#iM(Yf8XZHcDk<<3Bp-?-PrdHFUMQ3*iA%-a9+#m+*d_sps>~mlH
z7*GMt_I%1E2}{$-6=`eT#TiiQ=2&C67C3}(d_cORx-Dm<<KQUn!=@irP
z0T*%et@Rbl@`-LlV&798i$xDL4p(kuwXx64TdW$i(x|TU4X8n|rQnA`F>=>gc$D0r
zSx%)*gpa2|2X*O-SU7{R#p#rk!5{H`z`g>*oc{pT6d4K=Rpp3kQWc{++#;%o`>CV=
zuAtUgmC56{HlS^10Ss~vUAqwQO%-bs2&QBQ3?x+o#5#0tV#?0VJw(Unfz+ZPqr^fo
zZ1CbUuLuanA0qB8O^EUne&K8q-$>k5a({B61W>SH3LZxVK>!XFr5ah{7``C7(Ureb
zA1`^8j4PwLUuChs?8*T5;5?Gq3$j2>_b9XO`JWUz{Om1Xqu+MiWELAygnQ8M_$6fyb-en57^GsE=k<$M0&igK=
zMXu16hhXHs5?W*ljUY%U*o0qDD-b+6gDJr~lvL{bm5>~-bxi-BZb0RTYP*>4b3p7?u-|LBT_9wOo$aU+;JaK5JFYIv_P@}S#brt
z%3)sBdzfA~jGE>c`9CF`L#ttN>I<#Uj?oItDr78UcMj7wkS32u)HYEqWmZ!x4LM==
zQ3pk|mG3aOmI6c8o|71Nj6@a9l`$kthqw!p@J~=Ucj$<9m58I%x4XpWCr-!A0peLa
z5BGA`Qnq9bf8OQ@sDgm^d`88An`%6D(c&CXyDcOu6$j=59D2I;JY3yD@Q)Bn0m*y0
zN01nH{{Xl32B^z__jCTL*x^tDn87#2z&(yH6%zq$;6Zi?Z{lNqVRAhkM1@|jpK%za
zwSB^imv?%Z$r!p!KXn3|8tg!~ZE;*(A5uB$9GVY;IHB^i)qmX49m)O>OLs^OFor&PN3%C&IYPfL5PgQ
z45$*age+>M8Xp%qsQH39eaEFI%>LnskzmE2MpO!4xn@FqM1~X0O4Gikn9or9(zQwn
zc)3Yu;th&EmH}5+6&CSpyiAq$;ytheUKMEWYt``sXg?C|C#sj+{lpGR@I!-NF{#t(
zi^3UEWV~Sz4Kp+L6QNI0jy{N@p=qo{jy)KhkTR>pS{)4&aTr7cIQ&7@m|0vx003?m
zx&a+70LtKjLEs{q4q^qVnQBN0S>ka+Dr-LA7O@4YmcT6>u06~Y^BGuv1*%o&_?VM@
zbjSFa^AzbZh83az0F6MXDvo#pqEn`}MDLYsW)-N=wks65Kp&vIMXDf2f&~|2=?oqq
zQjSjMX=nw^rt)v*RR9B|du}h()vu);yXsN={=+cUH_0rl2C*I?^M?>IcpO;?DW?$u
zS$A+T3B#r}4J;$4xrVnWt>Q5R^N$7jl;L+A`G#`ySba;N%+UIY^i`QsDZQlmv;QRzO4NggZ<^WXl0m@ck^Bv0CQCQ5**>o@!2K>Pfwg}y?a;ZVgIk?G;
zBr|r@dg1phmU?4)Q~|PF#XDon)=+tcdppD}g|BlW6|tIRGaE;!w5q66PVgQZnX5BL=cCTPsifrppN4a&geUuQQu?6WaLyhma@pX$3YR=OKKqdhRSGa$#LKuMj0;#2^b6s6$(&3>R}-n
z<{WaBG&OzZQHZeJY$mnlGk~`15yn5>UHXh|fdkiq+$t*IVxT`=InK$FQ+ptHa;M=754_0IncM-B#z8W69>eoQgS$flq%}66AVt5+9k^p)>)mvYUP9i;R6Q{u8mB1e38o)_>G#H
z1}nr<`=F}c5S
zXqk#wm6OCarD~vzD5HM^iJXfd~b!^AL6xx{CN8FKl~(cy47Ic@Z|&(fmuzbt#iHe{$iL
zn;%R}aM(~$o0lP%;nOp7*dn~N;t^(6RHCwx%nfeMU0eh#y;NsHC)~P{m;SiQAqovK
zGHT?23$&t2C2h;KcQOw+FjRb5#hgF@9+Mlys__8>HgPzZEfEFfIqqcHT%o~v(_Upn
zjgDo@7u6Ug*r55SfB-$cN{X5$GB?2}?k$hSSX3(5}@RBp#8X_%niakD_(SyGEw$Imm_&(8RJ+&C2%5
z8yP?dF;mFL0Yf+6E9Yd3*YFC
zMI}k+mNXHY$&G^3l2m&UH=Xwq*>>L*+(99B*95_R@HVejM9yWJD(N;kmp~HAWu?;+
z-xc9*U^4=WxR!#zyHNq{_Yf^rZYrD6t{D59w3Z*I^Df@P(5rgE6cbG%x$;}7p23CV
z*ZPLFXNa4(Eqj`J;dPl_#c5Q2Na?T5AeIHSh2|IlH)_O&)y<)JgH%ibdrV1yL!#Ed
ze2`{qS%2!5s?1^Dp?L(sjbi>_noQj>4yYJ09z78DdU-hv^XUJpt9s7web}p4Tk=sIsvMMk>c4jQ+>x0d2TN4
zCwI(2R(;EC@7!IB9tl8!aQ&oi?^4}{WH6PBTTWxK2LY$*TB}wZ3WH0o
z%f1WMyhX*A^Di_l;Y>gT2FBa|TlV|u|QoAl<`{fGQ$gMw)tQf0=wcRQbMZ&GJ%=Z
zaRIA${E#?*pvU@!ODSI!EBZ?Jh__YL>{XbEtBGcd;AKIE++rjRiRKtBq+$TEO!|d(
zYW5;kmKDd;7S1@9l)8au4aJX}xMhaCOsvANY3?1Uqn2f$Xgm$>Fh$5S&ZQ}NtTM1I
zrVlchfclDJ4=-$ELY<@;H+muhrxt1@?*uq{NM*K)Hxh`Z`pm0RfF5~ZjqU??e-oXv
z1|k$j$g>%FBh_1(>l~yqR#Q2S-IvZ~kPVfhUgn=sD+lERf<4MjReFxGaEdb(S5%vm
z?j2=#g?E$8rGKdOaohwsGpGlnsfq;boa~k>2rOu?6ND?yB{y5d3fy^ss=*b2MJ2|w
zR~5qiK&5`7wio6l0uC_8alpP92O|2oK7Pf78$YF1xmk}lU4N=
zIx~<3$_QH4A%M^>%9bPLps&&(%~M$
z0b4uqi$94-4UaW1fHr2~Tc}6?TCTbu;}P^7*eY0`p@8|7+y>SBMCmg?vkj+ce&*A|
z-fmn*X^bBc0I4mq^Ai+RZ@AR3kT`+`Ist|cnT6XMYS+AxxXeqziIQcPV(~U6#Teq@
z@NH{%#jL%=N(_%vKSf)ZudKlUDoob|cw^T9>&)j?0~{~8kQmL=#B!0t*Ze>x#2D4b
zL10c67O4ZvCobi~FUq1-EfmKw56K&aZJAy(xpWl@mh(m>)Er&c#IeBM-k|{%e+w=)
zK4tM;nGDWaY;7O{bu2FC(Wj;fX@Qwk{6e**SJbwoREGqlwc!QaE{Lqyb2$`V5!m^0
z95aF-fXa39ZYcgo5{*h%PK%UElo~N`L%>nN9P1c8czcejEW48!UAu`)pJaZ_cq3gX
z^%2TOB+TjV2}?t`1h`K|F_LG61~%@`DlJyAv%E5nv)oeHdgcMS8X}ehFnL5U
zaR&l?#7M*4##{<}f!Xs9D`U(y6gVyrc6*D!KFNK6jWpUUS%eEKxrJHi%N%x(aRD>P
z%*z2f983+Fy1JMZK4o0cXuV77x?-{$nM(ae<<%TiID(?>X*!Es+liazh?t-%y9}$JmwaIuNEGVFb|Z!+$>qTEnZrbf(|hBSh`z*q*RHVSI8e09})Zr3NJ}q
zFnE>T57y>*oXX4eFdFrN$C1)YrOqBlnZQAP*oZy
zg|`tmeFYP*(uG8Vs6GO=sC
zI6o8z3KM2ftFA~^9jf#5D+-%8FHaGrwQaTTdL{ugZ*WdnnDEDyBDEU0B3n<0N;NFS
zN)Hi$G{lrKT+3E!5p`8CgP%}3C4m$%no%afUo$L%19f2PbKG#X?i=}b%W!Y;E9LbA
zD2It+(RIY#3y#DEqXf|$_+`<%^A#2Qi0J*yu%>-nGN&G*RbY&=%o?(Dm{9>NHf}08
zY?i%TUr_-Q6ta}Co~27fP{pk2F-Xi2Em<#bA;wS!yjpM+Wq0AhG{hBO$`
zx4vbOwxWl$KITk`iNWBQRM<3(PK?eEZ&4{1puKs5YA_r-SaJ=S2Rz)f8fe$zFAsy2
zzB*1K?RJ>kO6nF)Y)1ZY%UMd#;pH$AZO62L1yJjV2iuLNUSpC@=y>%HtO&Q5F2%a!
zxpvr9*x&URvyzi17=EFuT8ii9RcT&RpPpf)*fMta0C+hUwTgZ13E+tuY7%Xju
zaj|mMS1=8aOukkt)o478`(^vWwm7J^a_;5s?ySsU-p9G7kH$RRcXRBO6En2mLVI|gTzfjjVTir`>LTp`1a5kFGW>T*?J0ns>2VW3GvsJ<-(upS=_s>B`Wju#5fqYKzoJ(NTv;DS&iWthm#Cn
zh{wpLB&=4kquMg2%@nL<{{SEP3lhW_2*4QZJ6-|Be2g#@n@tf)_HC%B$Kinu+l?Xb$vmM5F{^!QC0xEabNaoIZtF=KOeh`pbOATOks?*
z@hqrO^k2!iP)#3wpUqZjSR1hVO`7T?dQv
zC@h-xBB5?upNHyPTS(>Id_|ib)<_JN+QoFxs|LdXmDF~L!SNmZ!Y_JuVY~OZRBVPj
zh*WdP>C{GGXWR`YU|VXt>LAs*57e~F;2CWwPGf0NP0=;EOISDkEvjUvVE5
z1FPc%v^~SRcsaREJD05YT9w08Rgd=^v01}`fS}2>{w0_-Rbt;Vjbbgs+-JmXa>F91
zqWFy|DphC4a}q;IaPHC6s>$I8zEphrfTGoA$GKp#QeS18DH4S?-#
z$xO9!ohP`n0@y8hAh-t%aLD{Yqk`(ZvX0n2bp`w)7nadrAk~d=4Ok>ANaQouaW0~x
z1@>#^52!S7kOgQS*h9p66}o<}?j{QI11oK9OUj>7lYc_MJA_=ixPRM;BA||)rb>=(
z9OJ~N&I{&e^HWZ~A<;@;DL@5Pi`EGD?xD6s0axYQIS~jHV;?{2Cc4^47HS#60U<$%
zTQTg^pfB+e@be2ox?@)cJByfB!nM>nOCUeLiG@a@hH3K~@S{*1gnPsp2Y_W*!tXKS
z-q`6Zw~BWw(;Icq@
zh2~(m3*VZC4905eH^>yt9J0@0f#_!j|1$
zEP^@U01=Y!XX(epxbJ8MmEY7|=S0YKGS%2!56>_bMK~|?_XD{*
zbN;{FX6aJzSLBQeEQ4>_kT*nFIVHUPM=Y^eOq=rru9h`%fNAHLANRPc9+}KuTgF%-
z;Y#fi&M!Sdys)-be!0}UEAvob+X%m?cmDtqy&?>!p8o)emh(mdAA&SM)%fGYLkA7q
zOGmTBvZxpm+@8cxwcai&KAgpCR-HSRCwg4Ih8ame9X|lNW%-^VG8TiZrB~w&!G^hjR>;YWr@Dy1XZW^Dl@H95W@!p
za@JH=Eky=8OmHU<#caiY_UEmrIOOkGL?-D35)
zP^*z~%)F4(PwqMvz=!c<#K2=YF5348P+H-?sYEcQ+%oLsy6#-h)EorSM?Jox(iLEN
zm0N*d%&@Evn6B+i7fO`Fu4A81J3{Dtaq|b9)d6YVGKpRwSC)-qs))`mYriA8e3*@r
zg=tJeGy|HBa36yn_KuOcdrP_%V!y%1HY
zd04o#D7xq7F2#=qBKrZYa}yc?&A@fhyO+Y-ikk<5JQ$`fAa#;IZ}Ak0jB0qekWpfw
zm*gwd#8y4TFe+V&S&wwBed;*flD%Kam0kenq9yyaXI7^hfl?R+`%LFxSNeZYE}CRL
zN)}FE;9?Pz2`Sg0^gQ|bmi??FKE9(XO6J*7Y9n{dphn#DDMwDbfwz~0a~zL2M%8Nj
zmnsE2SD(yGV9MQ@e7r-+OyXB10d4Fh&Qj?}0{KJ&9BU70-|8O~Jdclq%tCRddjT^G
z7C7Z}`Ik&5tZgx$s|tyPFFi0uuP=@SMQJshE2!;FokO~`@snc$5*(N6AHB4zcRwtjmKE?av8e~MTcISKk+E-y|((xGh1yY!>l!o
z%G$>$l%M-6^lE`aK{Y26ipKg$%%z27%FU1$QXTN=7uDO@m(oIF<^Vir
ze&Helj%VXO2!+U@FHmS-nIRNb~+qT!dT
z%%z}Q8o3=84+sh%1*4fbM=OtTd$=o5@->-3xqslWLNMS;kRdU>B}c)P(w_uX*(r2QOlaFdDrn45rW(vSKQST
zT@yEf5xMHlI10vn!chPQ?ijSEE(ml0yzC%}(=k!B
zNI5*gkok(Fu(5UUYw8l4&|)4&JzRiL5C(cO%)!HZ7V0EK7sEC%oN|e!
zL*lA0lFLOvfz${0HtEz$B{H9^O3Mp$y-FPK$qHC2S!xv_|w=tc0zWBksUz}bEc-i@_#TZ##Cqk=n-~(q7rdl4Px|Xas7F8CSmEa`STpU;wrKz$?swi_YN{b+ZWZ
zjlhSTBwsoH;k|m*{N`8|7s^5vhS7e{0FmXgI<-K|Rz%lqo
zPft+>3g-9f8d#=lr`&7_(FLuFc!Ek{D$8QnUP*Wy3pT@b1jC@kd%DpeJm-mWDA{Vt
z>0dvHnX#xGyoe&2Dp+Oz05cT`=GRUp5M7X!V^;GhR!{}J*5N41GHf3nCF;T${WF~`
z0Z*PI;LcuC&u_$`82;V&C~m77hgAkvB?xAA&3cxZrqtZZ4+l?(DUqIs`G8(@CMK)*^*fUg+W}2SmE!2^WvbT4^iXr5&_U+zh>@
z1BMkGp{Rx{uMi6^U(_s@b!-)nA1^TyHk-1d)#H{J6#0OZ38BmtTg&#rj7Mk73XY{%
zr)I=CnJl*YWtSmCQlQm6OC82*iP&q5TvY{ZFEzh<`GEjq13)9lRl)|-+Y&mo+Pwr|
zie6kc70-w@YvZGj<^?NMbJV4!q}uks#A}GEyt1E?W(K|qM$*K)gib3{E}5mZd4VYN
zp6Va~a3SM-e$`O~J*n1@|fq
zaDb73#TuKCp^MB^%Lst65)!vC)hO;RL@`4(U9!RzahYo~xR7SXF6J6}joE?Fe8b06
z2Vc-jI1?ByLqfGw6z)_ahQXooeGxni#Ig_8If4%^U)xav(yoAPc|TJ+Y79Fu^pwF_
zlYeoVi<>p-A80XIU#5Ci1Vj%3ZPC#2Fh=q_T|xRJ{>Pa~KpU3+7~PGwPcRw}gc8}^
z8lJ&=%qnjWa?rM*3d(oq5HNsraUCwfOt1>?MybaV+BIRc=!=U+?4l{TTEEOh2&(`$
z)O>!0Ie+gld#ARCUCWtX`+&e>_P+rz64s=3G|gFzxo&wc%&U>uZN3R>WYgEowZ0=U
zahgguID&RAFQ@|2+-$*)%fy+qdx2P5r(XvXn+0r|q46la0=s>s8>DhE-V*m@u>);y
z%%=e~T&qokzaAwF8s;L3Ed5Ir2z|$Nw*!&&00@?O?fp#dK&`$2IhKk`rH^3bhCpg3
znXa&EP^zyiEwKfIPnV)8lPG=6FqIgC(a=^3<|w&jgKz~bdc2DoN5sez^$_Fb{Z4dz
zoTWhQ$yETTKv%z~Hy7rMQmrvqeXq@wvu;~bh_EJ?Xz_Oi33gQcL~CAhUj#97%elmR
z%qh7@T|1jYnB~pf7XSr=c~wpJ#ZlAymUn}wDyit>)CA$fsYP>bU$4}BLxd^-FI=!=
z2rnJNxC#f0i~t+pj#`x}HTjRI0h+mTc2M1a2mOEXp(}u4dL_x0Q|Gb^icnP}D(SE}O5j@iI$lzL4;K7@~gxj^-*>3>WT#j^PD>Y&IGF#9kxrJTQS|
zO|
zap%9oFvKp)ezSivlGB=jS;m+msmL|&!^C#1LLhaTuef={;N(|5!W5TP1WroZ#Y?w#
zuoU}wWm-YS15`f@M6Cw8wwnAiNdmP*9hV(zMpyl~&`W09W!H0QLmm)(#4(p1V)nz?
z%)|=dg>Ay~UzmXnnlJ;z?xW)5WeV29mza#P3ObmnQ$WlZ5ThfA1uP#sejt;rSY1L8
zT-CSKp+h|6xRG`obqUY~xAiGBJ&|LfBflXAl8vg3f22}nyB&Jym$g;fZEgHY8J^$Z
zm;<--;PV$fSr3JK^(%qkgs*5MGM=I>Y*Mch%YstzD&`e>s03^*`GB;FXOz{S2^|fV_V*1WczN+`<$Qe7i}W2LjpEa{yo0L6_8D3Lc@&
z-BpGC)M#yaA>y0Fa||nt>xofAIzL;SP0M&aKy6WqAOf{4gzA+{sEyC^OoDSXX!)4(
zJVjY9uHuC6Mvtj(^jUw1KuwMPK(OP?q5k07Q``U(F*ngAi=l{PLo5MaBaMae96|RB
zU@zhm8~Pd{p8nji8$$UBwtZ2!5z8K@6}$GoqwW=;MjOYuR?=v0oM(;sk2-|tDx7xt
z^8^WH=3=p0tJUvOJEAHI^OBMZe0v|Csksyj3c4*rhyX^<0WF|X4g%}l8DU7JwhRdh
z6K1Fv69TPl<4_2qoZKN^;*7sgB2_`xu^=dSFWV1V9%8HJC1(7I)UhE?IDm|vc!F-O
zD{|bmJ;4IoR$unxJIZz9FCJ9S=2HRmwe^xyN?POaGQ#9^2z)T5J$a;YwfC5aaL4-;
z;cpq9;3AwPo~VT=KwGC5&6Y+6um=j+Q0iC7+ZwFC0_s{^S-(1jheE||#iny97f+Wx
zLAwoZqAJel^dtO6s#00Fv5c2oLK!SY)KIoi?l!eh2<39OO=O9bXxl8X4?bovGoK$G
z;-ZHbw;%h&WVK}c^#JaLFezxhwnuP~g1DU*(`ltnV3tE9ha_^`hWPbJSfR>eI2yVWE;g)@cKF3)y|TRj0N$g-DgxR60DG3$4QM$0
zN~Wkab@AK~gDh6QqGu(k==+3GO=(RUM-MQHSDr+&u61vGB|CL;ZkVMmqh>5#!c*+x
z;713GgYa9=nPZj<+y#`kq7$I_O7OcE6FjyhW?0p{G_eYuTQGoD+gsg0ZxcukNn?t`
z=!DuJg+r?I8*f&xZltVUXL{#nU@E)OQ)w0IcDY{VR6sufELQz#H
z-WjeB6AS>?xJrTTV3Osxul9l$Q)xjcg%_Xc5y1tABsDOR1BL#geNwKX1XTeI8VaUq
z>DvHs4M8o8-!mstZ0c8*{v#_jSRLH7t4`DhQKGy6j06bRZ_*Zu9HzoLHA1Eibcoy~
zFY_*T;#3!?&oU;anM|fHa^WGb+#h*6$MY27s+2pW;t@yCR3`MW9N&HjtX_if9w7>l
zI`=cNP+5A#x$`K27_>)HGb+Ymkar&;j^z!p
z481{Sv(#kfaYcu?TLkZqekIfZVxPD>2NC9OH($A1Dlz*s;-zBNQc|7VD{G=`ReHE3
zSP?--a3NXl3sW_TzcR~1AUhx-AMpc0M+{pmLz&rl{$!ol8
zAH)I|vYYxknFR!E{{Ud4L9GnSN6Z?7qR;!{
zTi7cuV+*}x1bmeU6&J>(liEh(-(FO7hDvCAGR~9+TcXBQhQ@)R{{Rz2fN`9$@KkQj
zNX>}J?AA$;XtQFD|H&(P-7)n+S@9D_W4+}1qVIL-(PGM
zaE)*y@FTl$!q~JimJ|cc^8wu1U9%(*QPB!kSGFdG;0v02f=R-sJ+nc7aBe7Wv|#)|
zZZ(b>Xh72wkf=6z=$B+UMVMWM{{TAZh|rTa)DTibcU%<+D`O4A$sY_0WPnZIsi4v2
z1W=+qPz}NYJ!HZ?ZE*ZXqn&GP9Wp?`?Rcr8Ppqzvh#%o%@h)5re-fi7!wA@D(W^U&
zo)mM;%<8$95fBI^TnF5vGwxWL)rysn(UFfdFk*0k*P4dafyluRJpeLS54iAZeck?J
zBf+)i`jv)?fz8wyD$K7!QJ!K^$ASqFcuXgU5W_NUfUK{nMy;oEu2R@#S#=B;^i}wR
zIYdFjZ^nq0g5d5r456Z|{{T@*%m#IiW2gXAL%t%G!!HXn?4?*;Z}^vhv6J*ZARSb9
z>JwNSwlDofE=w{X+6FcfDCt2)1kev~UD~~pscIY0R>|bzI3%TT(1noX%k@7TpWwIimP;t*d{7oveZTGZGIuAoXo^H$IK;wv6+F}u{Bmj
z@~)_e?-_Ysh?Fw&WO-nKG0M7PE@dlu3ZwH8RdAVPqNz&dgrTbNaqd*O_?3;V7^zs3
zk0w4LMPR`aX!Ho0WN0+MxF;&kcz~5v(B@YK58@jgHNB8VRyKqvIP!57v^?P-?hZFX
z7(8@EYm(b9XDk|{?gi?nlbB?<4Ye(?%M}c|*sH3B%l89zx+SKr{`wuxcMEX(?h)6m
z>!Z;cx)$icA7u_k2F-weJi_-@N_c~(9^)2AkW4%+8&*C-oHc$3JBiVPJ*wO0BbLxN
z7y|ATue_tpJRg~I;?;E*JD*>11}ib9<9-h^iOusIV7$hLNa6$Fq22YVS?`&X7GXJu
z%mxRBWGQ&cz)`S6F!HZQY;i-CnXfPKMpcal%&JN`d-<0t+BX%j32T5q>J@`YX7WP8
zv2`-$&WUw1g)p?RV|zJfSOePXJ)|A6gd04{dJ+`w%7go5{$M--j}tvmEo%FW(fw`(
zEq8};z!>DVSY@Ez@Uq$z=nk%Kp}Z-da=>6mCvXu=yKA^TgaL^X2e6j1$iU)P05kB@
z1W{tLHoNS<#2I$YBPag=B~awjR(Ssa$!yhQ&l4+GK(h90FsfKGKTx(|V5YFSTChs1BCs`#WZ`OM5)*c{X7ddtpG#?2(GzUmVS
znhAK&0`h}sJW3DyxLON42mP6X)W8}K`(iilnzS|
znB~U`1~tyjtTrp!H0>b^3~gnFs{SECwqMl7x1$$$_>SArEG`rw3%iX)5{aV8oU>*|
zS1uI2OkilkK;OX>U<1vc}kl$l`M11deW7at29)UZ-0u44$fbyrMk
zW{(l3jNbDbI_cseRNIQ4V!vWFX6tdKGmzJH9jYZZe^7y_+lBjrg_kO#9H2J0H#hHM
z^#+762DD?m=pgGMkU^mLQ^)0H&
zlHCwWQEabqrKO{r)S!#ZUXEdbjsF0s!q_!zBCooxmobeDpd0YwGvY$MK;=CHwgwQj
zH~hs?v{xm@$X;{IxT^`6Y)rfXbD|xs4FK5yI@`lu5A`bx(APoo{{T?kTan-QD$&TK
zv9F!nRCc+&LxNFGZnZHfY-ZvNAa+CuY#t?Kuf;hbGJ!@HXS;!}Qq?(z6;^M$lnxTW
zznJ8~3^%{T7AW|aS`)$fVWSEFL(0urF;`DoqE?u+}g8m=|T#AKL`TL1%OqJ_f!feJvngoxSxd&qF}*
zTf3M^O$6fO%InD~SD`5OUTZaY%xG6ZQ+z_h1p2k&0!N5u)LxWJj-R#UN<$a&(
zU;+%WpP7^!GI;cH6BLQc>FPBZ7d%A1oyv?^_AnzYAq)l+3@C@iZxZg7yM^bWFR&h2
zbc(ySUmcD0D;;?H%wCGXDcUiuQ@w%uN5;WmBKsIcf$A_z10Fp}3^WTZg>Fl7qXOxK
zHo8m^`1Z+&yW9)}riJD($OoVw%nz5YSQqeg2KJGdV*+?_1mGVK)R#+!BN$aNLN1Xr
z0)o?($8S>0W@L&UqQc;p;0_2Hm9S&~0C|`{IidKJBbVbeG1CNeo
zs4xKHzUp2(;*Ss&aE5$9e>5Fo=y1wV+M}YJQ0fCV6r$Us#2eX%H{ZE^n(umntocQY
zu-P(Mx|lMC)!Jq(Np%TSEA8c%#!slmT#j^)+z`WUOQtQ(Vv|z6;ndEbNt-feHZErj
z_dK(r9x?Pw%IxO-I<9Ur(Nfq}ynIEu5i}GiVbR5ILVp-Yne@!ADFGu+0z$_J!TO!
z=2LV~K&`oHf;1P=E>r}*#rQKRh~~d9+*^Kjsv42;J5xKRG%J2A*UBB~+Fv#WGKJihY(058m_
z&n9$_z+jJaA2wNmrz0{Y%{;&bM!m+oZK0t7dqJq2AhN~86BU1sSUEgQum{9^H*AUx
zyb=wv%oc1-iXBIM3gs#ZWHpYZn%hIFl)RMgTa^a|ZImVhkl}{|EGHjK#F;eyK4VTI
z43SG%Dt3$~k#H>WQ?T^}!+L3V<|??nnR3;r%tqupB|=Sf%=Td>qM$(6CqWRsvW*R2
z`i_H5*iiZCjH#V9`Ee1k^q)Uh80MP@qA)hs8q{klS`+gCrdXw>0w_M1gb?!Nit^@O
zt(L(#86si_&oLziGl7Y-jN}}Z)*S9C!)a%$T@W7x3c;$<3LwO`B}@TLoAvP+Lq*2m
zimM8AVljtj(}=I>Oz1g^7XwQ3E9K@9LR*NYyOuqm4S-AJk9Af50KCgG+p&Z(qKkSQ}WaUS^hARs*$~RJ5-W6yY{@(>Boz9`R#ZKTsG-
z4j0e*fM42V?)Vs3$!aXq?hY0SE~Ynmh}x($MorLmnOdc!?mj0kZsGt>JQ$ik9QJE>-~rdy5;{Ur`(5c+^D%>}9(SNfQ@ichsb9vxlj2G-#Rp
z)}l~UH3Jtg+D(^bcq5d4FB$BYz%({Uy^@sM^5u#iQ(bx{6lWDPl@>1J+gs;QMfYXu
zbK-4TNV0wtlQ&V8R)XrL=`wz!5WH2Im{Ef+Vs?R1)JyINNHVcd`)r#4orJ(u3(@o%=ZHS(Aly{=8qk85
z+3A4grXBG75DtqS@QClAyE>M_8eAl9mw3T|=<)s}gv>B(R}o#_CCM+MAwNeZDMcO)
zM@Dm)n!vW$gXYk58eigiVxXBy5LY&J;tbNnu8UUs%)Azj6-pXBaZpmSlCOJ`<7mbm
z8YLPCst^0jvGKJX4S{Q~%zCmEc^90-!jM&=NA6^*8=6-~IA{{T}h
z;Pn7$#GQ8HH}48UMeD~
zVBRk?eU4}|{-9J`EPGx!m&9|xx>&w){l;$%I1X>k+_6O`UJVrjKbpUkDw8tm{{VSH
zmK0)TVF9XRd6S6sHfy7}AEucqI>@*rd2V-25i*+fEo)cP0K7OcF~^KnB_SkG^@wok
zRhs*PS<5NAn&oVruyweQ3vge*;$GPe)ktuSazBWCFgb?Pm8jH}D9yY9ScI^^s&CJj
z3#DRH%hDvtZ*U@4EYWIE$WXdB10|hLLg3aHp%_Vam9v&FxM`>mj_X!Jv@C6Pc=`B@
z-PN)82HToDGVmHbu*RhyMYbRcI=4^`?)g9JDtNaMwqm1jGhujw5Dg7L8ZG7gf9!UO
z!&->@n&>^SaJ=r%SEvq_(p&UJ=2;u?pUi9Q8T*-FnvzL1yob!S1bQkZ72ah6oaMN&
z(v!I2Z)ekq>F^VKJ4#qBssM0fYwI4My3nZH8lTlEQT}X1kd=J9(G`
z--)YxxFbcQ%eWv_Q2=Cusy-=%QJUjd=lX!5REp{X4787^
zJuX=?uQ`-(whSx+^GgoK-EB#&nMF&~LV_u4^nt7~ZEWrw(G}oVP$uf-P**YgATWO7
zD+_T+Xho=F!6-P_oy>8-m$sgg(Zhj)Ql11CC5I1+YD$@~H+5#`S^Bb<>q}%mxav8)Pa3mOjnPF9$Sv%US*!`6i&nmg*4!7!tO_FSOt1yQItAZoHXG&o<_q7
z04=TVEs(1*SLBpE%;+mNFj6AmDwoBYwGN?*jTkei_#FYvb_Z84a+?cYST?mpBU`%M
z1Pr@novgRR#2j6>_=d3Ht+2x4^*7M@hhqIo#c1j^h(I-NmDIJ0x&moxu;m`Oi6&*a
z6U)b_0}V?Hjn=53d|b!V@UYu_Jbc13!gdAZ<+{|N+c$+6!ZBqvcAL4!{fI!dQy@uA
zVE05|rf)aY_)TVHz&U956opGOXA$CA-V~>7n2{kN;;?u
zUaDa7V93Iuv+{-^mJ0^kzZ;ei5ie!}kPEN`t50tH$B7>5M)2=76vZ``9llNbguixL
z$n-oxLZUo|S&@4XAi{Zbn8<+q$E+TVN|iV%YVI{g;6n5+Gge#^R(s-EY;lv8GgPyi
zo`SHkSf=wX-P6RlDvuu0CID7B8qZzA#?(=!7+?ixwxAUxX&zu2nn*mfR(?;!5GYY|
z!CYA?HZhhi_vWDcHlmx=gQfta8Zy@_-F~H#`9=Y(9J4Q#ov*p3NG)Krd_frtko6r5
z7AhT*05~GLD2VHXafZXpW3WpafDBOqY@s_B2*Jls3KqW=ErtzH1+RBi<}TwQG~{9Q
zT3?txSAHR@T6TmNHMd3R48rXUsLFaZI089g3Qeh3F5%#DwF2c2vq;fG?*b-ZYH3UX
zQQ|TlD9pz)WeN|NCQgs$IF}V+j+7B^E@j=THEXt
z(Kx1jE|@OKHa
z#t}J9;>xp(%G|e+GR9nUC`>c|0DT^Nj#IF<>YTo~fe-+&Kvg&AnUBP19++RK`ht+l
z{-Cz`(026?2}Or5zM{Fh@lxp7@@8Q`%Zr*9S>_BIdEcKICJ6-
z+vk|KcKt=AW$sf<)?iiawm6lFP^8>I3y2sU*h2>IgUqXGz1+#xV}bK5LZA_lo77%N
zxJznY0svFUe2&jBIC>YC~Y{M;@Yd)&%(!!bxx
zequVox|wj1^dHH9a51oUR79eUG}jM_U=O>6guK;51IcK5y@MI6`1Zr6znI236hICO
zy2buw(K7Eu+p^#`x}i!B;6FTIF~o899#
zAR4MqiMn3X46x-qt8*6)V;3##XxQMcVm5iQ3zGr!4|d?7bLR69ny@xYcW_w;-FzFTh9cl$Na5iCcEkB5w}H
zRfD*(GePh>nqfEGx4$yY7ThgfX#QsP0hQdxnxHW%Ac`6S**ZR^R`;1+qTtv^qZB|Y
zQ`Ddh@D&2J1_How#3XD25XTT)CIH_z2+4c6^dHPx>N14|^2M0kBjdQ41EBkJE6^|;
ztk=xiKG%pi19aiLqmTWKl-F9_T2@Q=nWnrAT$FKG;JEqZ?D=cl&CTUKw@bX-TP&kC
z?6*NJbV3Ry@ZaJjX|9h&#(=0m;V<}$F={qAyh=~7t(tydi_Ea{g5C=+R~oz?VNTb!
zVxz93m7@G+ID_Wmyu{W=#BVgVmzG89xYEL{vx2sQi*Z&>;)9-{;0z2$AiRcm>M-K
z6&Zn}xC_PVVgN5k^9leNUh@zuFXb|<-;Iz3t35(W>g}#z8y0U4q3S@rs0Wqc-er;Y
z!Yt^@3(O9ZVDLJCi;0Wlm*sdMam6Y%yJ{A%mDsvZ<_C?Yfv#p8sJ#AVRN1V+@V!eK
z0#&~l+}g}v1F33&HH#uGqV&SzWr1OsNnV8sK*H%lTbFSCj@&5|RhBkVKIMKh%mB$<
zNGl%O2St8iw>A4lT=>VLED|ZknNqC|T7Cn}A6zpck
z$H5;!bfN~z`P9o(>OExmAazU5B6I#HiEEsTs2~caE>d3L^)jLUFka9rs3Ma6My`zW
z4$7|!Q(#AEj47A-i;y-733O@0UZq%-)jxoRRt;!>xupt^Pl-+dss6xS`hz03B}ET2v1)D0*bWbu8s0vmPUF2bO7u%@HOF=r5RZep?YXD~}>CsL$;
zi9vATcLydndiIK22S~&74yq5R!=gOQlt85}<$bTA16n7Ey#^Uyxxm|^TE4{#ekLAu
zKPsqOghlU?LQ}Pk?CuSP)BgY<8pJ#gf&%Z`2M(s}_fyDIN5vcmxF@!76$lj?0l&Cx
zp_GQ(ssIh^h61+t%oN&Sd18}^&wPAENTuAMd_!`P#3nGmT8b+|)CR^hLd$-m0F7DM
z8CwY8=&4Oo6>vO2Hj?Km@9}b#fMHWEC6vp|Uh2x&j$K2x9fW>Yqx+YZg?S!gb}8T+
z`GxR@NUDztEj>)!+WbH6qc|gW3)BL+rx8F;5na&a%(4q*lx8|fLRnPGwVdiD`Y^5Y
z3IeDDSKJQO>>g!7MBbn%d$W+)RWLwaI))zjRTM10I|Ixrir&ImR*8$eOo~OObj?E1
zxL9)mxI3RW;^!KqOh
zn!L+4D{AM6QJqUg(gY!`Td#?PNY!F!f!=0i3Uw}2UR*;-XM9H>gwkLoSQ=9zw*UxU
zi)l9r1VU9XGNIJ2;P5ylWdV<<1=e2Ti!R^-%$F=j%BJg>Xeb&iyx*B>)mrf??ER62
zm4POR4CSbncJSO#YlFfT#F}p;Mk*VoA2IOxQNvxp{{X?YhCZw^sP7OqIZVngna3}p
zBm%8&;W07EABCxSww7SRV;Y`dS!+_@?cxfR_mr64-rJUCj<}SK7xI=ZUROMk#Mu-Y
zOw7rXwLz*l3dhU2z%3fcxV>J85z)e*f>a83om{>nq<9E{1XMWqqFfV0q2s8E!0MTq
zYvvlHtAIkTSuJ-2W;l*jinpP}VV4IB80cBd#g!?o)G!JF2(c^Q5Ya6z)!>;mu|Q9910^YIZHF{8P&IMfszu<;=F}d>*7>FWq&3WgHgp*
zj}N|KlngG&3xfF|30>wiu#Z@ma73A)`;^;D0e{k7z_>WXTMRbSwD
zh`K+CZYkIimAt5bq}4#Z;$tfp7lJ=Tvu?X1NDUNmE80zXAWFIf2h8Yip5+uda$hpbC8maF3S7VzYhjp&OA6vUv4WX`N6wjqV#c|p
zjmHKVNcM;743F#ts!W+J>0taBtOU_m;5m8&p5Za3gjKyxT$WXP
zi73@a_L&jxQyd})Y8b|7{J>nM9K{eO-Hhf_RJ&N*7h%m3s_N_UCcTbuG*@L~APO5V
zf{Hy)I|gmnuzukR28+x?^*|eeu}gu!u~S!kd`dtHIEA4o
zIXuPhm`oM})oLuN)`O_F^k0@)7Y9rhRI`7~QXJ#Cfa5qjmLTBA0HfA15N>R96j7`;
zIO+?A0?;@`n9n@HtLEEnTC~PiMYfAS#J2Z@4Gvfs<;s^VFSv?;fq0cuekB^ItBqWo
zi-?&@ASqgALKrKlk5Lztwg3x_uuDDCm$&oRGS>dn3l-dGwHW>ps3almgas82rUKRA
zhn{dLW~wuuBGNN$ZYEMXHbikHbGasgM3T{VyInAV)}%(&P(P7
zlxVrB(3);^Jj{wZ<7)E2;73Bi9+^rASO7ujCr)8ziJvA68o5{)4l*0m4%bW+I75VS
zImsrLX?u?461i9D)J^Zy331BZ5NW-~-^lS616d^oE3PS5;9?V1MBfCo;d5laC3OHU
zGB!OcBfQ3h0E6m0;sGQojv^INSMFcy5~>Q=7hmZy7)MsR?kL(9L8-D-cQlRojfhte
z_s!~ZQAT150K;(T_>L{(Xn7CnG-tUK-*mxLwG(E2$iBGX~XN2#TG#JDFBm
zaV+<()S*<~4EH;*?1AL$c!xNm85Ji)0HUTalF4wQq$0Gh5yC6EV}ps7BAcNi-3mt;
zEO@
zT{7k}K*uw@=wH-hy3$W4zkJKyDW;6Joni&<5TC97;Wll7RO($`0P(~z^n$9$a3HAm
zmm89yH9k0$;2pFfRRj=bP({7K^g(HD#rFgejROqmBNe7d2{Oz?EkV$zSADsQR@huz
z24)T|j)`~HjFQ9!o%X?;`tsXPs6#;Sq1yE;l$&_~Tu>;2GdLY8vE4L7H+Xv`QS(XI
zrwrs}SAiJYZs7jg56k_+N_+%;L3X%I-iMYK52b6g0{AC+#>B#alw8f
z<=9{Y0^Z&{Jdnaz_9GSJDCH7XmA-3CZGeS}yLJ1E<82#hQ+4X)QHNSFpD-X2n|~jC
zLY$_CfH~-j3l-p>Z}A%j9jz8L4{`g90}jY_=3O}O^XB;>Mo^b+a!TRYHNC;4v3Qg`
z)JIqp;uQ)j2G|0Deo~pV=Tmijv0NJ53RrNA$g#r{dnpVLQFrygn~^xi7$HiSwHErB
zl7o47HDsAuA2CWsD8Q17IeBeSW~&ZJf?5Y9%A5L$?MgDMTk!-bwNS^*VzSxfgG>tvC_jWMYPyD?6h~D^vxX_F*>LPU
z9O?=*TYInS25E|2`Q2P%jViOugrI%R&M4=s_rm=E$5hpDy+a2IWDRV7C0ibmRU6k
zZ&pp~>#0tGg4j3mgn}UFb$mS}LKv@2N>+Y3a|Koe{{XiIKyzzIkcj0&gUoua%3vAr
zoaxZUAQ1~@Z{`ajTOoGy)B>Jh7o;J)E{5O^SLP!|6rzs|V5>}^v_qxAA3-ci*CeoK
z$yOSJ+Y)SiKn&EeyO(g=)B=n*MA$*@EdlB|3XDQx*j?j|;(0QZ^EOSEcJUqAn%dh>285evX>H}m-Qn2T8roiM_8|J
z(#1|MW8z|~&UjV3{7ha&7tsD=OUZg%V~DF(11L<_sRiF>{{Y3rb#NmtL}fVh0IR>^
zQc~tcm*O~^l%a%J%(jM@wNaFi)l5}G;w)?(#0QyJ*hm=Z3IL0EJYrE7l`eHq7+>Z5
z#}wGqvQt`Mc(CKlvfX0U(476n#bL~}G6ucDWE+E^@%xpER|{xe`kFMvRfg5-E~Fa9
zv-*gmi+2Fo?hq7T6*eNc8}yevKp%(HtOSve3gFN80jGm;=0yHw>=XmV%rfT?Zli)*
zSj~40+)6M@b%?YS0BvHci~~Oslz<`^+Qc|Tfh*;#O~Ycw&M=!o%DcN^`GHYMRg_pH
zwDYV+L>#E(?a}uuSY4oj(i+EZ;(XYYt$7Sf9&&z1Z%_!|&@_Km_Z)n)N2sPFr0cb)
zLR76FnzH%KDuG_EJ0@J~KA>8q87z<2RIsuK^KJ>Gokf>TbsrUd%LyX({RxLB#2HW{
zJg9>3`I)9s6S6TAR}2#mE6-iZ?%-@MJVH)Lz`E&i0-cgV50tq2i%RD^1K62}iWaa+
z!zVN>SDUEI2%>9mF{@uLKBi)kE2O|i@eqv7R9M~UTB_nT>qiHuy2`9BUv45dG#C};
zIp;8PBWNxiqw57>&>4pm92jrg%p1dHpM$mpB52#{N=Zdp;EfcG!DPUB4Z&rUFkkLD
zR4ZIMg^X~IiMz}V`TLSp5k>Qugje6&M-%C6F~7W2%|R=n=;xmkt*pq=$X$xp&6E?
zbl_fdP^c=&CQWW%lqz$+Sy_m&UK=^WBd=4C)|TQe+SKltWiFJU6r54SDG$ktkGH;!
zHvvvX6JhEtuNs5i=FrHEhTw2KO42r$kfDIa)Ro%_AYzRrxo##)yBd&fjk7B{j1KXC
zP^6sPG|enQiujslxU_tCD4U@>bUXQnfr^CkF{_uBhHuJyhO3aOJ$U~BQkf*A{{U33
zfS^U{p;UT>4V8>c25tZumKp;MVTu*K-!kS)4&v4zO1J&)UIF`J;D*fVUfmL_8EA(H
zc$hIDj4mq}90PAP7LO>ng2qd{VzGlnM*#{`Ny;n-Yb$}PfHPR2802H8h8`*R8O|uX
z`GX*`maOCESp(f=&Gj+urK%?+EnZjy1g2yF6gxSVu$k&D=-hDd`HMP^{-#UCS(n$S
z0a|+X1tlZ~e4|yXYny@Kz$*6z2kjaY>M=sBxu~oP;2I8yMB4;@rV_<$@DFPdh$|MK
z5v-MY&B0L3>(LO=SAZv~czK(Y%ads@U?yMG2&Kd*sk&zNzRutxlWQa41skfA$^mIX
zsOgK;z+m>2=b9^4@s4KmfFTX$>LKOEpP0*Rn5D0nm^i#d)ppJvCUtN&%t+k`4qKEL
zf-9FKq!7x{h*GvnoyDavCm72QG2f>@=uBr={EV)Q`ZRVyi7n@Y8e0z|8%f{+PG3G{
z**pTayS~lG>|2XdW82gs#)iwLG(j}P3OBZ0KuqbmbjoDL$b@eZ4x&{>0ervt72Oa-
ztINmwnea4y@}6$u3S^Mek@}VfOPPG)aLlR9wIvQ_!25*hGf>5VfUh#rfb^7+CjfVt
zN`;0|;TwZ6E?!F>DKibHg$odJ51-~VVC-IB9%ACooS$|70BU34TTYK2BO<79w&5Dc
zGv?wH!qfp3SF>%(<3jVx+$1ey5fmnx?KYZ}okL5NF+TpKvDDx|^o@f^jHoNg1fbk8
zO?Mr;TZot*XlMa&1}0%ZN~dvM3c1wYC7{$c?0P1YQypq|I)o|yM5hePc36=>qbrtD
z#sQ@B`EOgMP)L9S
zqbxMsq!hx~O{lBo)OaiWu(h${Jw@q!m@k@@pyp8KrG#y*^%W;I<_H4Yn41!~Jh7^%
z1~#(S<559uvtCU=rnMP3vRyP1n(yXTnije;ivkE`lf(dpyG02OdytV43~cjjQW3;zJWY(%x5
zG{7Ta*GxG%3eKTb67<(H@_J$@Ie=`eV{s9EmM+7pjE5F!hZ(6+tIRthX5*Bg&(d0r
zG4KfT+_uglfD9v>hFQtPO$H?m2rBFJ#LEYkP?Qd(2Knkz@o#dYkVuX(6103w8jZ^c
z)UdCBzflVX?UZ7cE%6vj`nu~<-mOL`?)M#A+i~5R@%c_Hct+|dIxP!T3)`nzNi@TUWXPcXW
zbsG*jmimuE$H>*~m7pA}9K7wUO2KBEimmyg0GLf%h+8~7_Y656Vkt!z+(C6W;S)?R
z%NJb43@~Ms)t{+Z1Z9SCjgpWW#UxzRQ%B*CSI<))06ryp;Y(9=ntfsu`9-k5$L0$C
zF~6uYtoV-tHYhw%seW=0a+OC-Jo|%uuCu-a-R9%r{nb_Bu44rWWdO5x;$`zcZ<$td
zcf@MJJ!@KsyDsF;y=Zxc0?8PF=PsJ4flMr)V~AkWYOcpwTMLZl%Po}kE!SKHQB7fi
z$he!br^_TYmg-hwDka9)0u5AERJq#;EyFg(-ECY5wfg#wjmCxQRC1%{ZcJwGxR(eZ
zx%Df%Clv`=n!!-5!BX$I&JBm+ZMp{@{{W~2RI07Qn)x>Pn*RX3K$qzvN&xCshs0p5
zV3oS@6@p=(ea=9~Q*K;Eg_t!0`9*D|g$W**?xYyRilhOQ9;2MxO$o+jEU@j8Pzrz+
z{1JC<npWn4
zV7dm`ctxZZ=y;VDgAPnw-7$|NXiMETpt--AZJCr+yNFxKFyJk?q9F;UKyX}{#bBmlM;$fJryi7p|YaGSzM(dIa(vOs&R;4dd
zET}xOmOkA;eSUj@tZ*h-c1w949!X>#Or1n<+!;dDTBaqmdkC#&R%o)KsD**d%NoBj
z{QhvyH|>}46%I>|%Hc6xr5s>8n_NIKSGN+7miI1npUh@&^oN+N*vt4jg}L4Bd5?v3
zGYL_lI`W{!Y$^)5l6ihu2V>xcnhLp8)etLG?^6=N-x9f8ItXDjHHl``3ezv)w6-5U0nZB`cg5Ym|h@iB=
z3w6R9v{xo9%uKpM)B`?0S?w?6uwX6w^Ias@ukvM-5krm>Ap^
z=}6!EmMyin;u-^05-%)Z^aNjUtBz$)mNf`$HT*_q<==qcr^Iz#Ef@mV6%J?^KSEdT
zf#;~qAzoqxE5t`wBHm*8MQNEpi7HdXS8|rg9I22T7>m`>9Xv)W4ZKVjwF#yF0K10M
zrc58(FT@lfN>6y~@JqlvC!dJT3fio&YEqQI4^(HlVWN#SFnE~Xf*U_XTZcYi`CGV?
zRb|H&C^QTP4uLq9jYJG40r@l26l(9xxn$&Z5GuBJ!z}>IQ;Vm#@&<=6t$_J}f^cs!
zRdBU&jzj3q(+~OO0igv>9h@K?}amduW4MtrR=57zDB8V!)!`ty3
zFEMDuAr!I*plC}ksAUR^278rpa5*E%2&>>wt(Dq*#SjIyhmCfaXKtdasc=Mr(ci);y9`p|ucxRr;1pKRs2&nDw3T44Eap@&
zIO-krgz9Q3i>Q`Qscz_rcbD!u&*Cpg%NC-c_E=&XXQLSpGX=6@aZrc1rv-av46F>A
zb*QMf%reT4*(?Ks`;Eeyc;*j77eu%X5eu%PM3KSFEUQf&u_n00_0Zcw`Xi1h6pJi>
zP&dVX5}Ti7QtZU&_2NI2-pxS8HlbK>$16-^f!i1o$H7vo1S^V#JjBGMI-1re4j7d|
zHXN`+{061R8EVS;OmgWgJsidee5@a>J_$|g-8uOFQ4{UADvQvA2N4(MDRg3MYW40g
z!GMTw%nB+kIS*(fHv!}ya3EPyt=zyD$^Pp292#b6?K#bBeo?yv1RzUnH1ny#_lO*@#0NP%!r(wsq6LZae9nd@g=*cCQ&rpMHwI4(T3#Dja^}S)
zXlNIjy-F5tqZa@M@l0p|7NK8Ab%JM@shgLoqAas`)EUKQCy8|&DT{Q(t{}Mq%}ix#
zr{rK18go+#e-W)qD6go1G*idwB17fkWTQv;H4RE@1;rzy5s6wcbqu4i&BbWFCX&9`
zBwJ_Z0YE)AdU0zyATj!|t5MQhs
zD;bG~G5MB`c17hciABS{p;uv}y-nl|N>3k6gnRt0co2*ts1(9?&AIGUkOqe^*7ZK;re
zvLUPvUx;Bw;2-rtpf!(}!qeArO4}CkEOm+gqLG6Wqq9bLu!_2fY_@LBXzzMkD$~v797&SAi5!(*fskM=G=eP@F-t)MNmf@EJ
z358#so=9n}0Y>kcOr9K)od|ElLILUKhJyo#b9N48x;c+txQ*P_u4N0yEIXL|kc0k0
zo?@&v&Diibi@Iw8c=r*(U=I6cjK$B~?Pi1K!a)DJvA|y@aiC7OEld5<&fDcpT
z?p0#PXEEM%?>Q#-Z$T?55l{R>uzB%T6y`JHcrj$;Az)THOYKj
z8W(Z1CU`qjN9r2H3eP4WbN(C&Tl$C#QNu*I#Ar1hfnAu;h-B_~AlseAcGrj=Ac2CW
z*{n;zTsw_n(xpm~R4*fNGT|9*rX%5lo0*7Bd6pdCse>HzGo$JRwTwN(vk_0}l&b4<
zlQwYKL4$Y*%hd%c;H8D%WJk!V;G+vyb}lct`9*Nyg@+K0J~72bH8_-9FG-5FU8cWs
zwL0hMHSQEr%P?2I{6@AKDNf#?6>o6$3I=%0s9X;yzCuLMb8s-#%BJI0ae0ZnVo|o`
z>X>6w4ae?94Mb5;Me#Cn)I#wr*@%3LN~luowL6N}QO2VsggW|MZ+;=`r7VTzPBCYO
zQ-UQ+U6geKf^8kTc)3zzAUP$^uvXg<4e(J^Jj{~04k3bXV`ut^2GmRNzM@*#^BgMd
zWIUK?t$<`8o1!heIK-eLp5k?g_{8Ea;>36HE=?cYz^kfUEb?;&2%T`Aqcp(W92uq6
ze$7P5<*Sux;YzRObzp`^<`8w`F-M!kNDa4nPNgzXeI48|jevd;?9|DF_=m%KiC_Cm
zcQwlL#8SASg2Qqm-kQWwtvyB_-x9Gc%*x?kT8R`GSxv;yZdq0~#X!hWL|r|GAj)8C
zhn(|Ulw2dihOP-=`G=WA&ru$){2Wv+QGYe;%bb+R|U%$iE2d^TR9>mFjQro(~e^2pe2-!312Ng
zaTYA{(*XG3NBNF}Vamw_0G<;BeSmL<7R)M=cB8{D&XmR<4z0w)Y!i3Olh!3P%mZuS
zZ{DNn48Y!nKh#x?uX|x!t%3gloV_Wx&CW8(xJW9*#Ie!}m%BKK2C_vepqtH0wqmWX
z977&*tXys`2~#Gym?h>?zqnNEm{>$9i9(yG05uWlWwIbTF0(A4FG}FaW|YiPAzJ0}
z1Sl5EU}kUzi$5uT6uqCA!Xq$`TZoPF+#WcGz?oW&K#|w%!y@+77D3hrm=JLbG(qe1
zjtDxV5nir2xaK8hJzRP*55{7oz>STY?o_O*9MNBlLg+v@SHFqx8EU7HH!%D-mTDWe
z8G>yi1xIp{ZSGOy^<~0jcYs_IOnHwAI+e?V9aq}tW&LIgnryv6;pSFn+;*>FEceW?
zBi-zR3%)p6trVNwceJ5kIX&l9+^VI?d+2~*6$$Y!s)#cxGb_*dSZHr*K#CwHCsyZa
zM&JcGh7}pWOcg^YIn+v+aSte*qjrVx~*W(6Di`sev!jw
z69pajFlP`umxzqho)YXBBD4;!;wBnR!`>NgJ|evL9C?^U+!3B&+rD95V3FaK8O%F7
zKp)h)0x7zUO`)L6Xh1fBUf>xGbu6wT1TYO^U&N|eQEoFVgnlKim~na%h|a)v1eBhg
zDsq3(sa13T0EmLw&SsT(iE7ajvQh!WM~i~2OM^$2FB0E!(cW%bsl7}xdV_zd+o;Yz
ziLNCSIbm_D^#St00n{uLxsohSqkxQaMZgV|m>T~817?;kYSgbvCJnOP&&C)idxX1U
zVpM!b4EF}3bIdLY({F2*Hlab4jDW9;D@>hNB;;-^CqswaN-^vjfNvO>?2T5*?fIA9
zUkPVw!4NsrF|&xlbt`TQSM-4)6^14ws5`Q}LSLDQr_4Awfma%OnfZp9%rk$*8KNhB%x$pKe9m~p+jf~2r
z?*S`jr=mANd`fOg?sOAx^#sNT58uS23}Gyv8~+
zsQPvuUCy!8ng=}G5IhtyXi?N`rB?5ll_U(XGlmp_Zst7>gccWn#H~Xqr5wl3SvO94yg^A@8
z!cyx@E4{{#@?_=>$0=&Ap_NQGGG-?ZBD~xNxr5cjRYI@u$-GRJxQj*3BJaddi_0!^
z7S-HX_>08K9mlAvQG*F)i>@V$xBM#}WjBeNB88~gy_p$&j#6Q;vtuwgjcYL~?|AgZ
zAyQr)zy-^6v3sk$KM<+0ce#KyShW%4>&_xzh1RnLnPuEjfz->WRV=g>BRPaz;}Jo`
zw&lyrA*&UXUZ58Q+6*v|P7uhb8q8~S;u6X(CXCFoiLf+4U)21`v;G(&YA0}2-}o)E
zcNg^sA8?DfFpH^rj-$+VI6;s29N32AvN~a??Th>$_~AH3{maC67;!K64YIJqVr)Sq
z5}}MHh=m{HjuVvOInGJ5?r=kK!TvkK0_G9h9nR)*02S2X2n9|d76Axx2QCSwD5RTU
zHA2*C7f>}5PDuKYaSDRB3r5_*@F-?Gh*rtF1OVb|Gk~$C

literal 0
HcmV?d00001

diff --git a/doc/pub/How2ReadData/html/reveal.js/css/theme/template/mixins.scss b/doc/pub/How2ReadData/html/reveal.js/css/theme/template/mixins.scss
new file mode 100644
index 000000000..e0c560692
--- /dev/null
+++ b/doc/pub/How2ReadData/html/reveal.js/css/theme/template/mixins.scss
@@ -0,0 +1,29 @@
+@mixin vertical-gradient( $top, $bottom ) {
+	background: $top;
+	background: -moz-linear-gradient( top, $top 0%, $bottom 100% );
+	background: -webkit-gradient( linear, left top, left bottom, color-stop(0%,$top), color-stop(100%,$bottom) );
+	background: -webkit-linear-gradient( top, $top 0%, $bottom 100% );
+	background: -o-linear-gradient( top, $top 0%, $bottom 100% );
+	background: -ms-linear-gradient( top, $top 0%, $bottom 100% );
+	background: linear-gradient( top, $top 0%, $bottom 100% );
+}
+
+@mixin horizontal-gradient( $top, $bottom ) {
+	background: $top;
+	background: -moz-linear-gradient( left, $top 0%, $bottom 100% );
+	background: -webkit-gradient( linear, left top, right top, color-stop(0%,$top), color-stop(100%,$bottom) );
+	background: -webkit-linear-gradient( left, $top 0%, $bottom 100% );
+	background: -o-linear-gradient( left, $top 0%, $bottom 100% );
+	background: -ms-linear-gradient( left, $top 0%, $bottom 100% );
+	background: linear-gradient( left, $top 0%, $bottom 100% );
+}
+
+@mixin radial-gradient( $outer, $inner, $type: circle ) {
+	background: $outer;
+	background: -moz-radial-gradient( center, $type cover,  $inner 0%, $outer 100% );
+	background: -webkit-gradient( radial, center center, 0px, center center, 100%, color-stop(0%,$inner), color-stop(100%,$outer) );
+	background: -webkit-radial-gradient( center, $type cover,  $inner 0%, $outer 100% );
+	background: -o-radial-gradient( center, $type cover,  $inner 0%, $outer 100% );
+	background: -ms-radial-gradient( center, $type cover,  $inner 0%, $outer 100% );
+	background: radial-gradient( center, $type cover,  $inner 0%, $outer 100% );
+}
\ No newline at end of file
diff --git a/doc/pub/How2ReadData/html/reveal.js/css/theme/template/settings.scss b/doc/pub/How2ReadData/html/reveal.js/css/theme/template/settings.scss
new file mode 100644
index 000000000..e706e19c5
--- /dev/null
+++ b/doc/pub/How2ReadData/html/reveal.js/css/theme/template/settings.scss
@@ -0,0 +1,34 @@
+// Base settings for all themes that can optionally be
+// overridden by the super-theme
+
+// Background of the presentation
+$backgroundColor: #2b2b2b;
+
+// Primary/body text
+$mainFont: 'Lato', sans-serif;
+$mainFontSize: 30px;              /* changed (by hpl) from 36px; */
+$mainColor: #eee;
+
+// Headings
+$headingMargin: 0 0 20px 0;
+$headingFont: 'League Gothic', Impact, sans-serif;
+$headingColor: #eee;
+$headingLineHeight: 0.9em;
+$headingLetterSpacing: 0.02em;
+$headingTextTransform: none;      /* changed (by hpl) from uppercase; */
+$headingTextShadow: 0px 0px 6px rgba(0,0,0,0.2);
+$heading1TextShadow: $headingTextShadow;
+
+// Links and actions
+$linkColor: #13DAEC;
+$linkColorHover: lighten( $linkColor, 20% );
+
+// Text selection
+$selectionBackgroundColor: #FF5E99;
+$selectionColor: #fff;
+
+// Generates the presentation background, can be overridden
+// to return a background image or gradient
+@mixin bodyBackground() {
+	background: $backgroundColor;
+}
\ No newline at end of file
diff --git a/doc/pub/How2ReadData/html/reveal.js/css/theme/template/theme.scss b/doc/pub/How2ReadData/html/reveal.js/css/theme/template/theme.scss
new file mode 100644
index 000000000..b1dbc2736
--- /dev/null
+++ b/doc/pub/How2ReadData/html/reveal.js/css/theme/template/theme.scss
@@ -0,0 +1,171 @@
+// Base theme template for reveal.js
+
+/*********************************************
+ * GLOBAL STYLES
+ *********************************************/
+
+body {
+	@include bodyBackground();
+	background-color: $backgroundColor;
+}
+
+.reveal {
+	font-family: $mainFont;
+	font-size: $mainFontSize;
+	font-weight: normal;
+	letter-spacing: -0.02em;
+	color: $mainColor;
+}
+
+::selection {
+	color: $selectionColor;
+	background: $selectionBackgroundColor;
+	text-shadow: none;
+}
+
+/*********************************************
+ * HEADERS
+ *********************************************/
+
+.reveal h1,
+.reveal h2,
+.reveal h3,
+.reveal h4,
+.reveal h5,
+.reveal h6 {
+	margin: $headingMargin;
+	color: $headingColor;
+
+	font-family: $headingFont;
+	line-height: $headingLineHeight;
+	letter-spacing: $headingLetterSpacing;
+
+	text-transform: $headingTextTransform;
+	text-shadow: $headingTextShadow;
+}
+
+.reveal h1 {
+        line-height: 1.2em;    /* added by hpl */
+	text-shadow: $heading1TextShadow;
+}
+
+
+/*********************************************
+ * LINKS
+ *********************************************/
+
+.reveal a:not(.image) {
+	color: $linkColor;
+	text-decoration: none;
+
+	-webkit-transition: color .15s ease;
+	   -moz-transition: color .15s ease;
+	    -ms-transition: color .15s ease;
+	     -o-transition: color .15s ease;
+	        transition: color .15s ease;
+}
+	.reveal a:not(.image):hover {
+		color: $linkColorHover;
+
+		text-shadow: none;
+		border: none;
+	}
+
+.reveal .roll span:after {
+	color: #fff;
+	background: darken( $linkColor, 15% );
+}
+
+
+/*********************************************
+ * IMAGES
+ *********************************************/
+
+.reveal section img {
+	margin: 15px 0px;
+	background: rgba(255,255,255,0.12);
+	border: 4px solid $mainColor;
+
+	box-shadow: 0 0 10px rgba(0, 0, 0, 0.15);
+
+	-webkit-transition: all .2s linear;
+	   -moz-transition: all .2s linear;
+	    -ms-transition: all .2s linear;
+	     -o-transition: all .2s linear;
+	        transition: all .2s linear;
+}
+
+	.reveal a:hover img {
+		background: rgba(255,255,255,0.2);
+		border-color: $linkColor;
+
+		box-shadow: 0 0 20px rgba(0, 0, 0, 0.55);
+	}
+
+
+/*********************************************
+ * NAVIGATION CONTROLS
+ *********************************************/
+
+.reveal .controls div.navigate-left,
+.reveal .controls div.navigate-left.enabled {
+	border-right-color: $linkColor;
+}
+
+.reveal .controls div.navigate-right,
+.reveal .controls div.navigate-right.enabled {
+	border-left-color: $linkColor;
+}
+
+.reveal .controls div.navigate-up,
+.reveal .controls div.navigate-up.enabled {
+	border-bottom-color: $linkColor;
+}
+
+.reveal .controls div.navigate-down,
+.reveal .controls div.navigate-down.enabled {
+	border-top-color: $linkColor;
+}
+
+.reveal .controls div.navigate-left.enabled:hover {
+	border-right-color: $linkColorHover;
+}
+
+.reveal .controls div.navigate-right.enabled:hover {
+	border-left-color: $linkColorHover;
+}
+
+.reveal .controls div.navigate-up.enabled:hover {
+	border-bottom-color: $linkColorHover;
+}
+
+.reveal .controls div.navigate-down.enabled:hover {
+	border-top-color: $linkColorHover;
+}
+
+
+/*********************************************
+ * PROGRESS BAR
+ *********************************************/
+
+.reveal .progress {
+	background: rgba(0,0,0,0.2);
+}
+	.reveal .progress span {
+		background: $linkColor;
+
+		-webkit-transition: width 800ms cubic-bezier(0.260, 0.860, 0.440, 0.985);
+		   -moz-transition: width 800ms cubic-bezier(0.260, 0.860, 0.440, 0.985);
+		    -ms-transition: width 800ms cubic-bezier(0.260, 0.860, 0.440, 0.985);
+		     -o-transition: width 800ms cubic-bezier(0.260, 0.860, 0.440, 0.985);
+		        transition: width 800ms cubic-bezier(0.260, 0.860, 0.440, 0.985);
+	}
+
+/*********************************************
+ * SLIDE NUMBER
+ *********************************************/
+.reveal .slide-number {
+  color: $linkColor;
+}
+
+
diff --git a/doc/pub/How2ReadData/html/reveal.js/demo.html b/doc/pub/How2ReadData/html/reveal.js/demo.html
new file mode 100644
index 000000000..505bb1882
--- /dev/null
+++ b/doc/pub/How2ReadData/html/reveal.js/demo.html
@@ -0,0 +1,410 @@
+
+
+
+	
+		
+
+		reveal.js – The HTML Presentation Framework
+
+		
+		
+
+		
+		
+
+		
+
+		
+		
+
+		
+		
+
+		
+		
+
+		
+	
+
+	
+
+		

+
+			
+			

+				

+					
Reveal.js

+					
The HTML Presentation Framework

+					

+						Created by Hakim El Hattab and contributors
+					

+				

+
+				

+					
Hello There

+					

+						reveal.js enables you to create beautiful interactive slide decks using HTML. This presentation will show you examples of what it can do.
+					

+				

+
+				
+				

+					

+						
Vertical Slides

+						
Slides can be nested inside of each other.

+						
Use the Space key to navigate through all slides.

+						

+						
+							Down arrow
+						
+					

+					

+						
Basement Level 1

+						
Nested slides are useful for adding additional detail underneath a high level horizontal slide.

+					

+					

+						
Basement Level 2

+						
That's it, time to go back up.

+						

+						
+							Up arrow
+						
+					

+				

+
+				

+					
Slides

+					

+						Not a coder? Not a problem. There's a fully-featured visual editor for authoring these, try it out at https://slides.com.
+					

+				

+
+				

+					
Point of View

+					

+						Press ESC to enter the slide overview.
+					

+					

+						Hold down alt and click on any element to zoom in on it using zoom.js. Alt + click anywhere to zoom back out.
+					

+				

+
+				

+					
Touch Optimized

+					

+						Presentations look great on touch devices, like mobile phones and tablets. Simply swipe through your slides.
+					

+				

+
+				

+					
+				

+
+				

+					

+						
Fragments

+						
Hit the next arrow...

+						
... to step through ...

+						
... a fragmented slide.

+
+						
+					

+					

+						
Fragment Styles

+						
There's different types of fragments, like:

+						
grow

+						
shrink

+						
fade-out

+						
fade-up (also down, left and right!)

+						
current-visible

+						
Highlight red blue green

+					

+				

+
+				

+					
Transition Styles

+					

+						You can select from different transitions, like: 

+						None -
+						Fade -
+						Slide -
+						Convex -
+						Concave -
+						Zoom
+					

+				

+
+				

+					
Themes

+					

+						reveal.js comes with a few themes built in: 

+						
+						Black (default) -
+						White -
+						League -
+						Sky -
+						Beige -
+						Simple 

+						Serif -
+						Blood -
+						Night -
+						Moon -
+						Solarized
+					

+				

+
+				

+					

+						
Slide Backgrounds

+						

+							Set data-background="#dddddd" on a slide to change the background color. All CSS color formats are supported.
+						

+						
+							Down arrow
+						
+					

+					

+						
Image Backgrounds

+						
<section data-background="image.png">

+					

+					

+						
Tiled Backgrounds

+						
<section data-background="image.png" data-background-repeat="repeat" data-background-size="100px">

+					

+					

+						

+							
Video Backgrounds

+							
<section data-background-video="video.mp4,video.webm">

+						

+					

+					

+						
... and GIFs!

+					

+				

+
+				

+					
Background Transitions

+					

+						Different background transitions are available via the backgroundTransition option. This one's called "zoom".
+					

+					
Reveal.configure({ backgroundTransition: 'zoom' })

+				

+
+				

+					
Background Transitions

+					

+						You can override background transitions per-slide.
+					

+					
<section data-background-transition="zoom">

+				

+
+				

+					
Pretty Code

+					

+function linkify( selector ) {
+  if( supports3DTransforms ) {
+
+    var nodes = document.querySelectorAll( selector );
+
+    for( var i = 0, len = nodes.length; i < len; i++ ) {
+      var node = nodes[i];
+
+      if( !node.className ) {
+        node.className += ' roll';
+      }
+    }
+  }
+}
+					

+					
Code syntax highlighting courtesy of highlight.js.

+				

+
+				

+					
Marvelous List

+					
    
+						
  • No order here
    • 
+						
  • Or here
    • 
+						
  • Or here
    • 
+						
  • Or here
    • 
+					

+				

+
+				

+					
Fantastic Ordered List

+					
    
+						
  1. One is smaller than...
    2. 
+						
  2. Two is smaller than...
    3. 
+						
  3. Three!
    4. 
+					

+				

+
+				

+					
Tabular Tables

+					
+						
+							
+								
+								
+								
+							
+						
+						
+							
+								
+								
+								
+							
+							
+								
+								
+								
+							
+							
+								
+								
+								
+							
+						
+					
ItemValueQuantity
Apples$17
Lemonade$218
Bread$32

+				

+
+				

+					
Clever Quotes

+					

+						These guys come in two forms, inline: The nice thing about standards is that there are so many to choose from and block:
+					

+					

+						“For years there has been a theory that millions of monkeys typing at random on millions of typewriters would
+						reproduce the entire works of Shakespeare. The Internet has proven this theory to be untrue.”
+					

+				

+
+				

+					
Intergalactic Interconnections

+					

+						You can link between slides internally,
+						like this.
+					

+				

+
+				

+					
Speaker View

+					
There's a speaker view. It includes a timer, preview of the upcoming slide as well as your speaker notes.

+					
Press the S key to try it out.

+
+					
+				

+
+				

+					
Export to PDF

+					
Presentations can be exported to PDF, here's an example:

+					
+				

+
+				

+					
Global State

+					

+						Set data-state="something" on a slide and "something"
+						will be added as a class to the document element when the slide is open. This lets you
+						apply broader style changes, like switching the page background.
+					

+				

+
+				

+					
State Events

+					

+						Additionally custom events can be triggered on a per slide basis by binding to the data-state name.
+					

+					

+Reveal.addEventListener( 'customevent', function() {
+	console.log( '"customevent" has fired' );
+} );
+					

+				

+
+				

+					
Take a Moment

+					

+						Press B or . on your keyboard to pause the presentation. This is helpful when you're on stage and want to take distracting slides off the screen.
+					

+				

+
+				

+					
Much more

+					
+				

+
+				

+					
THE END

+					

+						- Try the online editor 

+						- Source code & documentation
+					

+				

+
+			

+
+		

+
+		
+		
+
+		
+
+	
+
diff --git a/doc/pub/How2ReadData/html/reveal.js/plugin/multiplex/package.json b/doc/pub/How2ReadData/html/reveal.js/plugin/multiplex/package.json
new file mode 100644
index 000000000..bbed77a67
--- /dev/null
+++ b/doc/pub/How2ReadData/html/reveal.js/plugin/multiplex/package.json
@@ -0,0 +1,19 @@
+{
+  "name": "reveal-js-multiplex",
+  "version": "1.0.0",
+  "description": "reveal.js multiplex server",
+  "homepage": "http://revealjs.com",
+  "scripts": {
+    "start": "node index.js"
+  },
+  "engines": {
+    "node": "~4.1.1"
+  },
+  "dependencies": {
+    "express": "~4.13.3",
+    "grunt-cli": "~0.1.13",
+    "mustache": "~2.2.1",
+    "socket.io": "~1.3.7"
+  },
+  "license": "MIT"
+}
diff --git a/doc/pub/How2ReadData/html/reveal.js/test/simple.md b/doc/pub/How2ReadData/html/reveal.js/test/simple.md
new file mode 100644
index 000000000..c72a44079
--- /dev/null
+++ b/doc/pub/How2ReadData/html/reveal.js/test/simple.md
@@ -0,0 +1,12 @@
+## Slide 1.1
+
+```js
+var a = 1;
+```
+
+
+## Slide 1.2
+
+
+
+## Slide 2
diff --git a/doc/pub/How2ReadData/html/reveal.js/test/test-markdown-external.html b/doc/pub/How2ReadData/html/reveal.js/test/test-markdown-external.html
new file mode 100644
index 000000000..859d0a199
--- /dev/null
+++ b/doc/pub/How2ReadData/html/reveal.js/test/test-markdown-external.html
@@ -0,0 +1,36 @@
+
+
+
+	
+		
+
+		reveal.js - Test Markdown
+
+		
+		
+	
+
+	
+
+		

+  		

+
+		
+
+		
+		
+		
+		
+		
+		
+
+		
+
+	
+
diff --git a/doc/pub/How2ReadData/html/reveal.js/test/test-markdown-external.js b/doc/pub/How2ReadData/html/reveal.js/test/test-markdown-external.js
new file mode 100644
index 000000000..cab85c6f6
--- /dev/null
+++ b/doc/pub/How2ReadData/html/reveal.js/test/test-markdown-external.js
@@ -0,0 +1,24 @@
+
+
+Reveal.addEventListener( 'ready', function() {
+
+	QUnit.module( 'Markdown' );
+
+	test( 'Vertical separator', function() {
+		strictEqual( document.querySelectorAll( '.reveal .slides>section>section' ).length, 2, 'found two slides' );
+	});
+
+	test( 'Horizontal separator', function() {
+		strictEqual( document.querySelectorAll( '.reveal .slides>section' ).length, 2, 'found two slides' );
+	});
+
+	test( 'Language highlighter', function() {
+		strictEqual( document.querySelectorAll( '.hljs-keyword' ).length, 1, 'got rendered highlight tag.' );
+		strictEqual( document.querySelector( '.hljs-keyword' ).innerHTML, 'var', 'the same keyword: var.' );
+	});
+
+
+} );
+
+Reveal.initialize();
+
diff --git a/doc/pub/How2ReadData/html/reveal.js/test/test-markdown-options.html b/doc/pub/How2ReadData/html/reveal.js/test/test-markdown-options.html
new file mode 100644
index 000000000..5b3be9758
--- /dev/null
+++ b/doc/pub/How2ReadData/html/reveal.js/test/test-markdown-options.html
@@ -0,0 +1,41 @@
+
+
+
+	
+		
+
+		reveal.js - Test Markdown Options
+
+		
+		
+	
+
+	
+
+		

+		

+
+		
+
+		
+		
+		
+
+		
+
+	
+
diff --git a/doc/pub/How2ReadData/html/reveal.js/test/test-markdown-options.js b/doc/pub/How2ReadData/html/reveal.js/test/test-markdown-options.js
new file mode 100644
index 000000000..3ae13503a
--- /dev/null
+++ b/doc/pub/How2ReadData/html/reveal.js/test/test-markdown-options.js
@@ -0,0 +1,26 @@
+Reveal.addEventListener( 'ready', function() {
+
+	QUnit.module( 'Markdown' );
+
+	test( 'Options are set', function() {
+		strictEqual( marked.defaults.smartypants, true );
+	});
+
+	test( 'Smart quotes are activated', function() {
+		var text = document.querySelector( '.reveal .slides>section>p' ).textContent;
+
+		strictEqual( /['"]/.test( text ), false );
+		strictEqual( /[“”‘’]/.test( text ), true );
+	});
+
+} );
+
+Reveal.initialize({
+	dependencies: [
+		{ src: '../plugin/markdown/marked.js' },
+		{ src: '../plugin/markdown/markdown.js' },
+	],
+	markdown: {
+		smartypants: true
+	}
+});
diff --git a/doc/pub/How2ReadData/html/src/Hudson_Bay.py~ b/doc/pub/How2ReadData/html/src/Hudson_Bay.py~
new file mode 100644
index 000000000..acd44518e
--- /dev/null
+++ b/doc/pub/How2ReadData/html/src/Hudson_Bay.py~
@@ -0,0 +1,43 @@
+import numpy as np
+import matplotlib.pyplot as plt
+
+def solver(m, H0, L0, dt, a, b, c, d, t0):
+    """Solve the difference equations for H and L over m years
+    with time step dt (measured in years."""
+
+    num_intervals = int(m/float(dt))
+    t = np.linspace(t0, t0 + m, num_intervals+1)
+    H = np.zeros(t.size)
+    L = np.zeros(t.size)
+
+    print 'Init:', H0, L0, dt
+    H[0] = H0
+    L[0] = L0
+
+    for n in range(0, len(t)-1):
+        H[n+1] = H[n] + a*dt*H[n] - b*dt*H[n]*L[n]
+        L[n+1] = L[n] + d*dt*H[n]*L[n] - c*dt*L[n]
+    return H, L, t
+
+# Load in data file
+data = np.loadtxt('Hudson_Bay.csv', delimiter=',', skiprows=1)
+# Make arrays containing x-axis and hares and lynx populations
+t_e = data[:,0]
+H_e = data[:,1]
+L_e = data[:,2]
+
+# Simulate using the model
+H, L, t = solver(m=20, H0=34.91, L0=3.857, dt=0.1,
+                 a=0.4807, b=0.02482, c=0.9272, d=0.02756,
+                 t0=1900)
+
+# Visualize simulations and data
+plt.plot(t_e, H_e, 'b-+', t_e, L_e, 'r-o', t, H, 'm--', t, L, 'k--')
+plt.xlabel('Year')
+plt.ylabel('Numbers of hares and lynx')
+plt.axis([1900, 1920, 0, 140])
+plt.title(r'Population of hares and lynx 1900-1920 (x1000)')
+plt.legend(('H_e', 'L_e', 'H', 'L'), loc='upper left')
+plt.savefig('Hudson_Bay_sim.pdf')
+plt.savefig('Hudson_Bay_sim.png')
+plt.show()
diff --git a/doc/pub/How2ReadData/html/src/plot_Hudson.py~ b/doc/pub/How2ReadData/html/src/plot_Hudson.py~
new file mode 100644
index 000000000..3b57c3277
--- /dev/null
+++ b/doc/pub/How2ReadData/html/src/plot_Hudson.py~
@@ -0,0 +1,19 @@
+import numpy as np
+from  matplotlib import pyplot as plt
+
+# Load in data file
+data = np.loadtxt('src/Hudson_Bay.dat', delimiter=',', skiprows=1)
+# Make arrays containing x-axis and hares and lynx populations
+year = data[:,0]
+hares = data[:,1]
+lynx = data[:,2]
+
+plt.plot(year, hares ,'b-+', year, lynx, 'r-o')
+plt.axis([1900,1920,0, 100.0])
+plt.xlabel(r'Year')
+plt.ylabel(r'Numbers of hares and lynx ')
+plt.legend(('Hares','Lynx'), loc='upper right')
+plt.title(r'Population of hares and lynx from 1900-1920 (x1000)}')
+plt.savefig('Hudson_Bay_data.pdf')
+plt.savefig('Hudson_Bay_data.png')
+plt.show()
diff --git a/doc/pub/How2ReadData/ipynb/.DS_Store b/doc/pub/How2ReadData/ipynb/.DS_Store
new file mode 100644
index 0000000000000000000000000000000000000000..5008ddfcf53c02e82d7eee2e57c38e5672ef89f6
GIT binary patch
literal 6148
zcmeH~Jr2S!425mzP>H1@V-^m;4Wg<&0T*E43hX&L&p$$qDprKhvt+--jT7}7np#A3
zem<@ulZcFPQ@L2!n>{z**++&mCkOWA81W14cNZlEfg7;MkzE(HCqgga^y>{tEnwC%0;vJ&^%eQ
zLs35+`xjp>T0  \n",
     "**Morten Hjorth-Jensen**, Department of Physics, University of Oslo and Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University\n",
     "\n",
-    "Date: **Jan 14, 2019**\n",
+    "Date: **Aug 13, 2019**\n",
     "\n",
     "Copyright 1999-2019, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license\n",
     "\n",
@@ -18,6 +18,7 @@
     "\n",
     "\n",
     "\n",
+    "\n",
     "## Introduction\n",
     "\n",
     "Our emphasis throughout this series of lectures  \n",
@@ -32,47 +33,147 @@
     "concepts and machine learning algorithms before we delve into the\n",
     "algorithms themselves. The examples we will use in the beginning, start with simple\n",
     "polynomials with random noise added. We will use the Python\n",
-    "software package [Scikit-learn](http://scikit-learn.org/stable/) and\n",
+    "software package [Scikit-Learn](http://scikit-learn.org/stable/) and\n",
     "introduce various machine learning algorithms to make fits of\n",
     "the data and predictions. We move thereafter to more interesting\n",
-    "cases such as the simulation of financial transactions or disease\n",
-    "models. These are examples where we can easily set up the data and\n",
+    "cases such as nuclear binding energies.\n",
+    "These are examples where we can easily set up the data and\n",
     "then use machine learning algorithms included in for example\n",
-    "**scikit-learn**. \n",
+    "**Scikit-Learn**. \n",
     "\n",
     "These examples will serve us the purpose of getting\n",
     "started. Furthermore, they allow us to catch more than two birds with\n",
     "a stone. They will allow us to bring in some programming specific\n",
-    "topics and tools as well as showing the power of various Python (and\n",
-    "R) packages for machine learning and statistical data analysis. In the\n",
-    "lectures on linear algebra we cover in more detail various programming\n",
-    "features of languages like Python and C++ (and other), we will also\n",
-    "look into more specific linear functions which are relevant for the\n",
-    "various algorithms we will discuss. Here, we will mainly focus on two\n",
-    "specific Python packages for Machine Learning, scikit-learn and\n",
-    "tensorflow (see below for links etc).  Moreover, the examples we\n",
+    "topics and tools as well as showing the power of various Python \n",
+    "packages for machine learning and statistical data analysis.  \n",
+    "\n",
+    "Here, we will mainly focus on two\n",
+    "specific Python packages for Machine Learning, Scikit-Learn and\n",
+    "Tensorflow (see below for links etc).  Moreover, the examples we\n",
     "introduce will serve as inputs to many of our discussions later, as\n",
     "well as allowing you to set up models and produce your own data and\n",
     "get started with programming.\n",
     "\n",
     "\n",
     "\n",
+    "## What is Machine Learning?\n",
+    "\n",
+    "Statistics, data science and machine learning form important fields of\n",
+    "research in modern science.  They describe how to learn and make\n",
+    "predictions from data, as well as allowing us to extract important\n",
+    "correlations about physical process and the underlying laws of motion\n",
+    "in large data sets. The latter, big data sets, appear frequently in\n",
+    "essentially all disciplines, from the traditional Science, Technology,\n",
+    "Mathematics and Engineering fields to Life Science, Law, education\n",
+    "research, the Humanities and the Social Sciences. \n",
+    "\n",
+    "It has become more\n",
+    "and more common to see research projects on big data in for example\n",
+    "the Social Sciences where extracting patterns from complicated survey\n",
+    "data is one of many research directions.  Having a solid grasp of data\n",
+    "analysis and machine learning is thus becoming central to scientific\n",
+    "computing in many fields, and competences and skills within the fields\n",
+    "of machine learning and scientific computing are nowadays strongly\n",
+    "requested by many potential employers. The latter cannot be\n",
+    "overstated, familiarity with machine learning has almost become a\n",
+    "prerequisite for many of the most exciting employment opportunities,\n",
+    "whether they are in bioinformatics, life science, physics or finance,\n",
+    "in the private or the public sector. This author has had several\n",
+    "students or met students who have been hired recently based on their\n",
+    "skills and competences in scientific computing and data science, often\n",
+    "with marginal knowledge of machine learning.\n",
+    "\n",
+    "Machine learning is a subfield of computer science, and is closely\n",
+    "related to computational statistics.  It evolved from the study of\n",
+    "pattern recognition in artificial intelligence (AI) research, and has\n",
+    "made contributions to AI tasks like computer vision, natural language\n",
+    "processing and speech recognition. Many of the methods we will study are also \n",
+    "strongly rooted in basic mathematics and physics research. \n",
+    "\n",
+    "Ideally, machine learning represents the science of giving computers\n",
+    "the ability to learn without being explicitly programmed.  The idea is\n",
+    "that there exist generic algorithms which can be used to find patterns\n",
+    "in a broad class of data sets without having to write code\n",
+    "specifically for each problem. The algorithm will build its own logic\n",
+    "based on the data.  You should however always keep in mind that\n",
+    "machines and algorithms are to a large extent developed by humans. The\n",
+    "insights and knowledge we have about a specific system, play a central\n",
+    "role when we develop a specific machine learning algorithm. \n",
+    "\n",
+    "Machine learning is an extremely rich field, in spite of its young\n",
+    "age. The increases we have seen during the last three decades in\n",
+    "computational capabilities have been followed by developments of\n",
+    "methods and techniques for analyzing and handling large date sets,\n",
+    "relying heavily on statistics, computer science and mathematics.  The\n",
+    "field is rather new and developing rapidly. Popular software packages\n",
+    "written in Python for machine learning like\n",
+    "[Scikit-learn](http://scikit-learn.org/stable/),\n",
+    "[Tensorflow](https://www.tensorflow.org/),\n",
+    "[PyTorch](http://pytorch.org/) and [Keras](https://keras.io/), all\n",
+    "freely available at their respective GitHub sites, encompass\n",
+    "communities of developers in the thousands or more. And the number of\n",
+    "code developers and contributors keeps increasing. Not all the\n",
+    "algorithms and methods can be given a rigorous mathematical\n",
+    "justification, opening up thereby large rooms for experimenting and\n",
+    "trial and error and thereby exciting new developments.  However, a\n",
+    "solid command of linear algebra, multivariate theory, probability\n",
+    "theory, statistical data analysis, understanding errors and Monte\n",
+    "Carlo methods are central elements in a proper understanding of many\n",
+    "of algorithms and methods we will discuss.\n",
+    "\n",
+    "\n",
+    "\n",
+    "## Types of Machine Learning\n",
+    "\n",
+    "\n",
+    "The approaches to machine learning are many, but are often split into\n",
+    "two main categories.  In *supervised learning* we know the answer to a\n",
+    "problem, and let the computer deduce the logic behind it. On the other\n",
+    "hand, *unsupervised learning* is a method for finding patterns and\n",
+    "relationship in data sets without any prior knowledge of the system.\n",
+    "Some authours also operate with a third category, namely\n",
+    "*reinforcement learning*. This is a paradigm of learning inspired by\n",
+    "behavioral psychology, where learning is achieved by trial-and-error,\n",
+    "solely from rewards and punishment.\n",
+    "\n",
+    "Another way to categorize machine learning tasks is to consider the\n",
+    "desired output of a system.  Some of the most common tasks are:\n",
+    "\n",
+    "  * Classification: Outputs are divided into two or more classes. The goal is to   produce a model that assigns inputs into one of these classes. An example is to identify  digits based on pictures of hand-written ones. Classification is typically supervised learning.\n",
+    "\n",
+    "  * Regression: Finding a functional relationship between an input data set and a reference data set.   The goal is to construct a function that maps input data to continuous output values.\n",
+    "\n",
+    "  * Clustering: Data are divided into groups with certain common traits, without knowing the different groups beforehand.  It is thus a form of unsupervised learning.\n",
+    "\n",
+    "The methods we cover have three main topics in common, irrespective of\n",
+    "whether we deal with supervised or unsupervised learning. The first\n",
+    "ingredient is normally our data set (which can be subdivided into\n",
+    "training and test data), the second item is a model which is normally a\n",
+    "function of some parameters.  The model reflects our knowledge of the system (or lack thereof). As an example, if we know that our data show a behavior similar to what would be predicted by a polynomial, fitting our data to a polynomial of some degree would then determin our model. \n",
+    "\n",
+    "The last ingredient is a so-called **cost**\n",
+    "function which allows us to present an estimate on how good our model\n",
+    "is in reproducing the data it is supposed to train.  \n",
+    "At the heart of basically all ML algorithms there are so-called minimization algorithms, often we end up with various variants of **gradient** methods.\n",
+    "\n",
+    "\n",
+    "\n",
+    "\n",
+    "\n",
     "\n",
     "\n",
     "## Software and needed installations\n",
     "\n",
     "We will make extensive use of Python as programming language and its\n",
     "myriad of available libraries.  You will find\n",
-    "IPython/Jupyter notebooks invaluable in your work.  You can run **R**\n",
+    "Jupyter notebooks invaluable in your work.  You can run **R**\n",
     "codes in the Jupyter/IPython notebooks, with the immediate benefit of\n",
     "visualizing your data. You can also use compiled languages like C++,\n",
-    "Rust, Fortran etc if you prefer. The focus in these lectures will be\n",
-    "on Python, but we will provide many code examples for those of you who\n",
-    "prefer R or compiled languages. You can integrate C++ codes and R in for example\n",
-    "a Jupyter notebook. \n",
+    "Rust, Julia, Fortran etc if you prefer. The focus in these lectures will be\n",
+    "on Python.\n",
     "\n",
     "\n",
-    "If you have Python installed (we recommend Python3) and you feel\n",
+    "If you have Python installed (we strongly recommend Python3) and you feel\n",
     "pretty familiar with installing different packages, we recommend that\n",
     "you install the following Python packages via **pip** as \n",
     "\n",
@@ -94,6 +195,7 @@
     "etc etc. \n",
     "\n",
     "\n",
+    "\n",
     "## Python installers\n",
     "\n",
     "If you don't want to perform these operations separately and venture\n",
@@ -116,22 +218,46 @@
     "analysis environment, available for free and under a commercial\n",
     "license.\n",
     "\n",
-    "## Useful Python packages\n",
-    "Here we list several useful Python packages.\n",
+    "Furthermore, [Google's Colab](https://colab.research.google.com/notebooks/welcome.ipynb) is a free Jupyter notebook environment that requires \n",
+    "no setup and runs entirely in the cloud. Try it out!\n",
     "\n",
+    "## Useful Python libraries\n",
+    "Here we list several useful Python libraries we strongly recommend (if you use anaconda many of these are already there)\n",
+    "\n",
+    "* [NumPy](https://www.numpy.org/) is a highly popular library for large, multi-dimensional arrays and matrices, along with a large collection of high-level mathematical functions to operate on these arrays\n",
+    "\n",
+    "* [The pandas](https://pandas.pydata.org/) library provides high-performance, easy-to-use data structures and data analysis tools \n",
+    "\n",
+    "* [Xarray](http://xarray.pydata.org/en/stable/) is a Python package that makes working with labelled multi-dimensional arrays simple, efficient, and fun!\n",
+    "\n",
+    "* [Scipy](https://www.scipy.org/) (pronounced “Sigh Pie”) is a Python-based ecosystem of open-source software for mathematics, science, and engineering. \n",
+    "\n",
+    "* [Matplotlib](https://matplotlib.org/) is a Python 2D plotting library which produces publication quality figures in a variety of hardcopy formats and interactive environments across platforms.\n",
+    "\n",
+    "* [Autograd](https://github.com/HIPS/autograd) can automatically differentiate native Python and Numpy code. It can handle a large subset of Python's features, including loops, ifs, recursion and closures, and it can even take derivatives of derivatives of derivatives\n",
+    "\n",
+    "* [SymPy](https://www.sympy.org/en/index.html) is a Python library for symbolic mathematics. \n",
+    "\n",
+    "* [scikit-learn](https://scikit-learn.org/stable/) has simple and efficient tools for machine learning, data mining and data analysis\n",
+    "\n",
+    "* [TensorFlow](https://www.tensorflow.org/) is a Python library for fast numerical computing created and released by Google\n",
+    "\n",
+    "* [Keras](https://keras.io/) is a high-level neural networks API, written in Python and capable of running on top of TensorFlow, CNTK, or Theano\n",
+    "\n",
+    "* And many more such as [pytorch](https://pytorch.org/),  [Theano](https://pypi.org/project/Theano/) etc \n",
     "\n",
     "## Installing R, C++, cython or Julia\n",
     "\n",
-    "You will also find it convenient to utilize R. Although we will mainly\n",
-    "use Python during lectures and in various projects and exercises, we\n",
-    "provide a full R set of codes for the same examples. Those of you\n",
-    "already familiar with R should feel free to continue using R, keeping\n",
+    "You will also find it convenient to utilize **R**. We will mainly\n",
+    "use Python during lectures and in various projects and exercises.\n",
+    "Those of you\n",
+    "already familiar with **R** should feel free to continue using **R**, keeping\n",
     "however an eye on the parallel Python set ups. Similarly, if you are a\n",
-    "Python afecionado, feel free to explore R as well.  Jupyter/Ipython\n",
+    "Python afecionado, feel free to explore **R** as well.  Jupyter/Ipython\n",
     "notebook allows you to run **R** codes interactively in your\n",
     "browser. The software library **R** is tuned to statistically analysis\n",
     "and allows for an easy usage of the tools we will discuss in these\n",
-    "texts.\n",
+    "lectures.\n",
     "\n",
     "To install **R** with Jupyter notebook \n",
     "[follow the link here](https://mpacer.org/maths/r-kernel-for-ipython-notebook)\n",
@@ -149,13 +275,13 @@
     "languages.\n",
     "\n",
     "To add more entropy, **cython** can also be used when running your\n",
-    "notebooks. It means that Python with the Jupyter/IPython notebook\n",
+    "notebooks. It means that Python with the jupyter notebook\n",
     "setup allows you to integrate widely popular softwares and tools for\n",
     "scientific computing. Similarly, the \n",
     "[Numba Python package](https://numba.pydata.org/) delivers increased performance\n",
     "capabilities with minimal rewrites of your codes.  With its\n",
     "versatility, including symbolic operations, Python offers a unique\n",
-    "computational environment. Your Jupyter/IPython notebook can easily be\n",
+    "computational environment. Your jupyter notebook can easily be\n",
     "converted into a nicely rendered **PDF** file or a Latex file for\n",
     "further processing. For example, convert to latex as"
    ]
@@ -175,13 +301,805 @@
     "\n",
     "Finally, if you wish to use the light mark-up language \n",
     "[doconce](https://github.com/hplgit/doconce) you can convert a standard ascii text file into various HTML \n",
-    "formats, ipython notebooks, latex files, pdf files etc with minimal edits.\n",
+    "formats, ipython notebooks, latex files, pdf files etc with minimal edits. These lectures were generated using **doconce**.\n",
     "\n",
     "\n",
-    "## Simple linear regression model using **scikit-learn**\n",
     "\n",
-    "We start with perhaps our simplest possible example, using **scikit-learn** to perform linear regression analysis on a data set produced by us. \n",
-    "What follows is a simple Python code where we have defined  function $y$ in terms of the variable $x$. Both are defined as vectors of dimension $1\\times 100$. The entries to the vector $\\hat{x}$  are given by random numbers generated with a uniform distribution with entries $x_i \\in [0,1]$ (more about probability distribution functions later). These values are then used to define a function $y(x)$ (tabulated again as a vector) with a linear dependence on $x$ plus a random noise added via the normal distribution.\n",
+    "## Numpy examples and Important Matrix and vector handling packages\n",
+    "\n",
+    "There are several central software packages for linear algebra and eigenvalue problems. Several of the more\n",
+    "popular ones have been wrapped into ofter software packages like those from the widely used text **Numerical Recipes**. The original source codes in many of the available packages are often taken from the widely used\n",
+    "software package LAPACK, which follows two other popular packages\n",
+    "developed in the 1970s, namely EISPACK and LINPACK.  We describe them shortly here.\n",
+    "\n",
+    "  * LINPACK: package for linear equations and least square problems.\n",
+    "\n",
+    "  * LAPACK:package for solving symmetric, unsymmetric and generalized eigenvalue problems. From LAPACK's website  it is possible to download for free all source codes from this library. Both C/C++ and Fortran versions are available.\n",
+    "\n",
+    "  * BLAS (I, II and III): (Basic Linear Algebra Subprograms) are routines that provide standard building blocks for performing basic vector and matrix operations. Blas I is vector operations, II vector-matrix operations and III matrix-matrix operations. Highly parallelized and efficient codes, all available for download from .\n",
+    "\n",
+    "## Basic Matrix Features\n",
+    "\n",
+    "**Matrix properties reminder.**"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "$$\n",
+    "\\mathbf{A} =\n",
+    "      \\begin{bmatrix} a_{11} & a_{12} & a_{13} & a_{14} \\\\\n",
+    "                                 a_{21} & a_{22} & a_{23} & a_{24} \\\\\n",
+    "                                   a_{31} & a_{32} & a_{33} & a_{34} \\\\\n",
+    "                                  a_{41} & a_{42} & a_{43} & a_{44}\n",
+    "             \\end{bmatrix}\\qquad\n",
+    "\\mathbf{I} =\n",
+    "      \\begin{bmatrix} 1 & 0 & 0 & 0 \\\\\n",
+    "                                 0 & 1 & 0 & 0 \\\\\n",
+    "                                 0 & 0 & 1 & 0 \\\\\n",
+    "                                 0 & 0 & 0 & 1\n",
+    "             \\end{bmatrix}\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Basic Matrix Features\n",
+    "\n",
+    "The inverse of a matrix is defined by"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "$$\n",
+    "\\mathbf{A}^{-1} \\cdot \\mathbf{A} = I\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Basic Matrix Features\n",
+    "\n",
+    "**Matrix Properties Reminder.**\n",
+    "\n",
+    "\n",
+    "\n",
+    "\n",
+    "\n",
+    "\n",
+    "\n",
+    "\n",
+    "\n",
+    "\n",
+    "\n",
+    "\n",
+    "\n",
+    "
              Relations                      Name                                  matrix elements                             
   $A = A^{T}$                                   symmetric              $a_{ij} = a_{ji}$                                                           
   $A = \\left (A^{T} \\right )^{-1}$              real orthogonal        $\\sum_k a_{ik} a_{jk} = \\sum_k a_{ki} a_{kj} = \\delta_{ij}$                 
   $A = A^{ * }$                                 real matrix            $a_{ij} = a_{ij}^{ * }$                                                     
   $A = A^{\\dagger}$                             hermitian              $a_{ij} = a_{ji}^{ * }$                                                     
   $A = \\left (A^{\\dagger} \\right )^{-1}$        unitary                $\\sum_k a_{ik} a_{jk}^{ * } = \\sum_k a_{ki}^{ * } a_{kj} = \\delta_{ij}$     
\n",
+    "\n",
+    "\n",
+    "\n",
+    "\n",
+    "## Some famous Matrices\n",
+    "\n",
+    "  * Diagonal if $a_{ij}=0$ for $i\\ne j$\n",
+    "\n",
+    "  * Upper triangular if $a_{ij}=0$ for $i > j$\n",
+    "\n",
+    "  * Lower triangular if $a_{ij}=0$ for $i < j$\n",
+    "\n",
+    "  * Upper Hessenberg if $a_{ij}=0$ for $i > j+1$\n",
+    "\n",
+    "  * Lower Hessenberg if $a_{ij}=0$ for $i < j+1$\n",
+    "\n",
+    "  * Tridiagonal if $a_{ij}=0$ for $|i -j| > 1$\n",
+    "\n",
+    "  * Lower banded with bandwidth $p$: $a_{ij}=0$ for $i > j+p$\n",
+    "\n",
+    "  * Upper banded with bandwidth $p$: $a_{ij}=0$ for $i < j+p$\n",
+    "\n",
+    "  * Banded, block upper triangular, block lower triangular....\n",
+    "\n",
+    "## Basic Matrix Features\n",
+    "\n",
+    "**Some Equivalent Statements.**\n",
+    "\n",
+    "For an $N\\times N$ matrix  $\\mathbf{A}$ the following properties are all equivalent\n",
+    "\n",
+    "  * If the inverse of $\\mathbf{A}$ exists, $\\mathbf{A}$ is nonsingular.\n",
+    "\n",
+    "  * The equation $\\mathbf{Ax}=0$ implies $\\mathbf{x}=0$.\n",
+    "\n",
+    "  * The rows of $\\mathbf{A}$ form a basis of $R^N$.\n",
+    "\n",
+    "  * The columns of $\\mathbf{A}$ form a basis of $R^N$.\n",
+    "\n",
+    "  * $\\mathbf{A}$ is a product of elementary matrices.\n",
+    "\n",
+    "  * $0$ is not eigenvalue of $\\mathbf{A}$.\n",
+    "\n",
+    "\n",
+    "\n",
+    "\n",
+    "## Numpy and arrays\n",
+    "[Numpy](http://www.numpy.org/) provides an easy way to handle arrays in Python. The standard way to import this library is as"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "metadata": {
+    "collapsed": false
+   },
+   "outputs": [],
+   "source": [
+    "import numpy as np"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Here follows a simple example where we set up an array of ten elements, all determined by random numbers drawn according to the normal distribution,"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "metadata": {
+    "collapsed": false
+   },
+   "outputs": [],
+   "source": [
+    "n = 10\n",
+    "x = np.random.normal(size=n)\n",
+    "print(x)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "We defined a vector $x$ with $n=10$ elements with its values given by the Normal distribution $N(0,1)$.\n",
+    "Another alternative is to declare a vector as follows"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "metadata": {
+    "collapsed": false
+   },
+   "outputs": [],
+   "source": [
+    "import numpy as np\n",
+    "x = np.array([1, 2, 3])\n",
+    "print(x)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Here we have defined a vector with three elements, with $x_0=1$, $x_1=2$ and $x_2=3$. Note that both Python and C++\n",
+    "start numbering array elements from $0$ and on. This means that a vector with $n$ elements has a sequence of entities $x_0, x_1, x_2, \\dots, x_{n-1}$. We could also let (recommended) Numpy to compute the logarithms of a specific array as"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 4,
+   "metadata": {
+    "collapsed": false
+   },
+   "outputs": [],
+   "source": [
+    "import numpy as np\n",
+    "x = np.log(np.array([4, 7, 8]))\n",
+    "print(x)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "In the last example we used Numpy's unary function $np.log$. This function is\n",
+    "highly tuned to compute array elements since the code is vectorized\n",
+    "and does not require looping. We normaly recommend that you use the\n",
+    "Numpy intrinsic functions instead of the corresponding **log** function\n",
+    "from Python's **math** module. The looping is done explicitely by the\n",
+    "**np.log** function. The alternative, and slower way to compute the\n",
+    "logarithms of a vector would be to write"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 5,
+   "metadata": {
+    "collapsed": false
+   },
+   "outputs": [],
+   "source": [
+    "import numpy as np\n",
+    "from math import log\n",
+    "x = np.array([4, 7, 8])\n",
+    "for i in range(0, len(x)):\n",
+    "    x[i] = log(x[i])\n",
+    "print(x)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "We note that our code is much longer already and we need to import the **log** function from the **math** module. \n",
+    "The attentive reader will also notice that the output is $[1, 1, 2]$. Python interprets automagically our numbers as integers (like the **automatic** keyword in C++). To change this we could define our array elements to be double precision numbers as"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 6,
+   "metadata": {
+    "collapsed": false
+   },
+   "outputs": [],
+   "source": [
+    "import numpy as np\n",
+    "x = np.log(np.array([4, 7, 8], dtype = np.float64))\n",
+    "print(x)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "or simply write them as double precision numbers (Python uses 64 bits as default for floating point type variables), that is"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 7,
+   "metadata": {
+    "collapsed": false
+   },
+   "outputs": [],
+   "source": [
+    "import numpy as np\n",
+    "x = np.log(np.array([4.0, 7.0, 8.0])\n",
+    "print(x)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "To check the number of bytes (remember that one byte contains eight bits for double precision variables), you can use simple use the **itemsize** functionality (the array $x$ is actually an object which inherits the functionalities defined in Numpy) as"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 8,
+   "metadata": {
+    "collapsed": false
+   },
+   "outputs": [],
+   "source": [
+    "import numpy as np\n",
+    "x = np.log(np.array([4.0, 7.0, 8.0])\n",
+    "print(x.itemsize)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Matrices in Python\n",
+    "\n",
+    "Having defined vectors, we are now ready to try out matrices. We can\n",
+    "define a $3 \\times 3 $ real matrix $\\hat{A}$ as (recall that we user\n",
+    "lowercase letters for vectors and uppercase letters for matrices)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 9,
+   "metadata": {
+    "collapsed": false
+   },
+   "outputs": [],
+   "source": [
+    "import numpy as np\n",
+    "A = np.log(np.array([ [4.0, 7.0, 8.0], [3.0, 10.0, 11.0], [4.0, 5.0, 7.0] ]))\n",
+    "print(A)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "If we use the **shape** function we would get $(3, 3)$ as output, that is verifying that our matrix is a $3\\times 3$ matrix. We can slice the matrix and print for example the first column (Python organized matrix elements in a row-major order, see below) as"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 10,
+   "metadata": {
+    "collapsed": false
+   },
+   "outputs": [],
+   "source": [
+    "import numpy as np\n",
+    "A = np.log(np.array([ [4.0, 7.0, 8.0], [3.0, 10.0, 11.0], [4.0, 5.0, 7.0] ]))\n",
+    "# print the first column, row-major order and elements start with 0\n",
+    "print(A[:,0])"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "We can continue this was by printing out other columns or rows. The example here prints out the second column"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 11,
+   "metadata": {
+    "collapsed": false
+   },
+   "outputs": [],
+   "source": [
+    "import numpy as np\n",
+    "A = np.log(np.array([ [4.0, 7.0, 8.0], [3.0, 10.0, 11.0], [4.0, 5.0, 7.0] ]))\n",
+    "# print the first column, row-major order and elements start with 0\n",
+    "print(A[1,:])"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Numpy contains many other functionalities that allow us to slice, subdivide etc etc arrays. We strongly recommend that you look up the [Numpy website for more details](http://www.numpy.org/). Useful functions when defining a matrix are the **np.zeros** function which declares a matrix of a given dimension and sets all elements to zero"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 12,
+   "metadata": {
+    "collapsed": false
+   },
+   "outputs": [],
+   "source": [
+    "import numpy as np\n",
+    "n = 10\n",
+    "# define a matrix of dimension 10 x 10 and set all elements to zero\n",
+    "A = np.zeros( (n, n) )\n",
+    "print(A)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "or initializing all elements to"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 13,
+   "metadata": {
+    "collapsed": false
+   },
+   "outputs": [],
+   "source": [
+    "import numpy as np\n",
+    "n = 10\n",
+    "# define a matrix of dimension 10 x 10 and set all elements to one\n",
+    "A = np.ones( (n, n) )\n",
+    "print(A)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "or as unitarily distributed random numbers (see the material on random number generators in the statistics part)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 14,
+   "metadata": {
+    "collapsed": false
+   },
+   "outputs": [],
+   "source": [
+    "import numpy as np\n",
+    "n = 10\n",
+    "# define a matrix of dimension 10 x 10 and set all elements to random numbers with x \\in [0, 1]\n",
+    "A = np.random.rand(n, n)\n",
+    "print(A)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "As we will see throughout these lectures, there are several extremely useful functionalities in Numpy.\n",
+    "As an example, consider the discussion of the covariance matrix. Suppose we have defined three vectors\n",
+    "$\\hat{x}, \\hat{y}, \\hat{z}$ with $n$ elements each. The covariance matrix is defined as"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "$$\n",
+    "\\hat{\\Sigma} = \\begin{bmatrix} \\sigma_{xx} & \\sigma_{xy} & \\sigma_{xz} \\\\\n",
+    "                              \\sigma_{yx} & \\sigma_{yy} & \\sigma_{yz} \\\\\n",
+    "                              \\sigma_{zx} & \\sigma_{zy} & \\sigma_{zz} \n",
+    "             \\end{bmatrix},\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "where for example"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "$$\n",
+    "\\sigma_{xy} =\\frac{1}{n} \\sum_{i=0}^{n-1}(x_i- \\overline{x})(y_i- \\overline{y}).\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "The Numpy function **np.cov** calculates the covariance elements using the factor $1/(n-1)$ instead of $1/n$ since it assumes we do not have the exact mean values. \n",
+    "The following simple function uses the **np.vstack** function which takes each vector of dimension $1\\times n$ and produces a $3\\times n$ matrix $\\hat{W}$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "$$\n",
+    "\\hat{W} = \\begin{bmatrix} x_0 & y_0 & z_0 \\\\\n",
+    "                          x_1 & y_1 & z_1 \\\\\n",
+    "                          x_2 & y_2 & z_2 \\\\\n",
+    "                          \\dots & \\dots & \\dots \\\\\n",
+    "                          x_{n-2} & y_{n-2} & z_{n-2} \\\\\n",
+    "                          x_{n-1} & y_{n-1} & z_{n-1}\n",
+    "             \\end{bmatrix},\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "which in turn is converted into into the $3\\times 3$ covariance matrix\n",
+    "$\\hat{\\Sigma}$ via the Numpy function **np.cov()**. We note that we can also calculate\n",
+    "the mean value of each set of samples $\\hat{x}$ etc using the Numpy\n",
+    "function **np.mean(x)**. We can also extract the eigenvalues of the\n",
+    "covariance matrix through the **np.linalg.eig()** function."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 15,
+   "metadata": {
+    "collapsed": false
+   },
+   "outputs": [],
+   "source": [
+    "# Importing various packages\n",
+    "import numpy as np\n",
+    "\n",
+    "n = 100\n",
+    "x = np.random.normal(size=n)\n",
+    "print(np.mean(x))\n",
+    "y = 4+3*x+np.random.normal(size=n)\n",
+    "print(np.mean(y))\n",
+    "z = x**3+np.random.normal(size=n)\n",
+    "print(np.mean(z))\n",
+    "W = np.vstack((x, y, z))\n",
+    "Sigma = np.cov(W)\n",
+    "print(Sigma)\n",
+    "Eigvals, Eigvecs = np.linalg.eig(Sigma)\n",
+    "print(Eigvals)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 16,
+   "metadata": {
+    "collapsed": false
+   },
+   "outputs": [],
+   "source": [
+    "%matplotlib inline\n",
+    "\n",
+    "import numpy as np\n",
+    "import matplotlib.pyplot as plt\n",
+    "from scipy import sparse\n",
+    "eye = np.eye(4)\n",
+    "print(eye)\n",
+    "sparse_mtx = sparse.csr_matrix(eye)\n",
+    "print(sparse_mtx)\n",
+    "x = np.linspace(-10,10,100)\n",
+    "y = np.sin(x)\n",
+    "plt.plot(x,y,marker='x')\n",
+    "plt.show()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Meet the Pandas\n",
+    "\n",
+    "\n",
+    "\n",
+    "\n",
+    "\n",
+    "
\n",
+    "\n",
+    "\n",
+    "\n",
+    "\n",
+    "\n",
+    "Another useful Python package is\n",
+    "[pandas](https://pandas.pydata.org/), which is an open source library\n",
+    "providing high-performance, easy-to-use data structures and data\n",
+    "analysis tools for Python. **pandas** stands for panel data, a term borrowed from econometrics and is an efficient library for data analysis with an emphasis on tabular data.\n",
+    "**pandas** has two major classes, the **DataFrame** class with two-dimensional data objects and tabular data organized in columns and the class **Series** with a focus on one-dimensional data objects. Both classes allow you to index data easily as we will see in the examples below. \n",
+    "**pandas** allows you also to perform mathematical operations on the data, spanning from simple reshapings of vectors and matrices to statistical operations. \n",
+    "\n",
+    "The following simple example shows how we can, in an easy way make tables of our data. Here we define a data set which includes names, place of birth and date of birth, and displays the data in an easy to read way. We will see repeated use of **pandas**, in particular in connection with classification of data."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 17,
+   "metadata": {
+    "collapsed": false
+   },
+   "outputs": [],
+   "source": [
+    "import pandas as pd\n",
+    "from IPython.display import display\n",
+    "data = {'First Name': [\"Frodo\", \"Bilbo\", \"Aragorn II\", \"Samwise\"],\n",
+    "        'Last Name': [\"Baggins\", \"Baggins\",\"Elessar\",\"Gamgee\"],\n",
+    "        'Place of birth': [\"Shire\", \"Shire\", \"Eriador\", \"Shire\"],\n",
+    "        'Date of Birth T.A.': [2968, 2890, 2931, 2980]\n",
+    "        }\n",
+    "data_pandas = pd.DataFrame(data)\n",
+    "display(data_pandas)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "In the above we have imported **pandas** with the shorthand **pd**, the latter has become the standard way we import **pandas**. We make then a list of various variables\n",
+    "and reorganize the aboves lists into a **DataFrame** and then print out  a neat table with specific column labels as *Name*, *place of birth* and *date of birth*.\n",
+    "Displaying these results, we see that the indices are given by the default numbers from zero to three.\n",
+    "**pandas** is extremely flexible and we can easily change the above indices by defining a new type of indexing as"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 18,
+   "metadata": {
+    "collapsed": false
+   },
+   "outputs": [],
+   "source": [
+    "data_pandas = pd.DataFrame(data,index=['Frodo','Bilbo','Aragorn','Sam'])\n",
+    "display(data_pandas)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Thereafter we display the content of the row which begins with the index **Aragorn**"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 19,
+   "metadata": {
+    "collapsed": false
+   },
+   "outputs": [],
+   "source": [
+    "display(data_pandas.loc['Aragorn'])"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "We can easily append data to this, for example"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 20,
+   "metadata": {
+    "collapsed": false
+   },
+   "outputs": [],
+   "source": [
+    "new_hobbit = {'First Name': [\"Peregrin\"],\n",
+    "              'Last Name': [\"Took\"],\n",
+    "              'Place of birth': [\"Shire\"],\n",
+    "              'Date of Birth T.A.': [2990]\n",
+    "              }\n",
+    "data_pandas=data_pandas.append(pd.DataFrame(new_hobbit, index=['Pippin']))\n",
+    "display(data_pandas)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Here are other examples where we use the **DataFrame** functionality to handle arrays, now with more interesting features for us, namely numbers. We set up a matrix \n",
+    "of dimensionality $10\\times 5$ and compute the mean value and standard deviation of each column. Similarly, we can perform mathematial operations like squaring the matrix elements and many other operations."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 21,
+   "metadata": {
+    "collapsed": false
+   },
+   "outputs": [],
+   "source": [
+    "import numpy as np\n",
+    "import pandas as pd\n",
+    "from IPython.display import display\n",
+    "np.random.seed(100)\n",
+    "# setting up a 10 x 5 matrix\n",
+    "rows = 10\n",
+    "cols = 5\n",
+    "a = np.random.randn(rows,cols)\n",
+    "df = pd.DataFrame(a)\n",
+    "display(df)\n",
+    "print(df.mean())\n",
+    "print(df.std())\n",
+    "display(df**2)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Thereafter we can select specific columns only and plot final results"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 22,
+   "metadata": {
+    "collapsed": false
+   },
+   "outputs": [],
+   "source": [
+    "df.columns = ['First', 'Second', 'Third', 'Fourth', 'Fifth']\n",
+    "df.index = np.arange(10)\n",
+    "\n",
+    "display(df)\n",
+    "print(df['Second'].mean() )\n",
+    "print(df.info())\n",
+    "print(df.describe())\n",
+    "\n",
+    "from pylab import plt, mpl\n",
+    "plt.style.use('seaborn')\n",
+    "mpl.rcParams['font.family'] = 'serif'\n",
+    "\n",
+    "df.cumsum().plot(lw=2.0, figsize=(10,6))\n",
+    "plt.show()\n",
+    "\n",
+    "\n",
+    "df.plot.bar(figsize=(10,6), rot=15)\n",
+    "plt.show()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "We can produce a $4\\times 4$ matrix"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 23,
+   "metadata": {
+    "collapsed": false
+   },
+   "outputs": [],
+   "source": [
+    "b = np.arange(16).reshape((4,4))\n",
+    "print(b)\n",
+    "df1 = pd.DataFrame(b)\n",
+    "print(df1)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "and many other operations. \n",
+    "\n",
+    "The **Series** class is another important class included in\n",
+    "**pandas**. You can view it as a specialization of **DataFrame** but where\n",
+    "we have just a single column of data. It shares many of the same features as _DataFrame. As with **DataFrame**,\n",
+    "most operations are vectorized, achieving thereby a high performance when dealing with computations of arrays, in particular labeled arrays.\n",
+    "As we will see below it leads also to a very concice code close to the mathematical operations we may be interested in.\n",
+    "For multidimensional arrays, we recommend strongly [xarray](http://xarray.pydata.org/en/stable/). **xarray** has much of the same flexibility as **pandas**, but allows for the extension to higher dimensions than two. We will see examples later of the usage of both **pandas** and **xarray**. \n",
+    "\n",
+    "\n",
+    "\n",
+    "## Reading Data and fitting\n",
+    "\n",
+    "In order to study various Machine Learning algorithms, we need to\n",
+    "access data. Acccessing data is an essential step in all machine\n",
+    "learning algorithms. In particular, setting up the so-called **design\n",
+    "matrix** (to be defined below) is often the first element we need in\n",
+    "order to perform our calculations. To set up the design matrix means\n",
+    "reading (and later, when the calculations are done, writing) data\n",
+    "in various formats, The formats span from reading files from disk,\n",
+    "loading data from databases and interacting with online sources\n",
+    "like web application programming interfaces (APIs).\n",
+    "\n",
+    "In handling various input formats, as discussed above, we will mainly stay with **pandas**,\n",
+    "a Python package which allows us, in a seamless and painless way, to\n",
+    "deal with a multitude of formats, from standard **csv** (comma separated\n",
+    "values) files, via **excel**, **html** to **hdf5** formats.  With **pandas**\n",
+    "and the **DataFrame**  and **Series** functionalities we are able to convert text data\n",
+    "into the calculational formats we need for a specific algorithm. And our code is going to be \n",
+    "pretty close the basic mathematical expressions.\n",
+    "\n",
+    "Our first data set is going to be a classic from nuclear physics, namely all\n",
+    "available data on binding energies. Don't be intimidated if you are not familiar with nuclear physics. It serves simply as an example here of a data set. \n",
+    "\n",
+    "We will show some of the\n",
+    "strengths of packages like **Scikit-Learn** in fitting nuclear binding energies to\n",
+    "specific functions using linear regression first. Then, as a teaser, we will show you how \n",
+    "you can easily implement other algorithms like decision trees and random forests and neural networks.\n",
+    "\n",
+    "But before we really start with nuclear physics data, let's just look at some simpler polynomial fitting cases, such as,\n",
+    "(don't be offended) fitting straight lines!\n",
+    "\n",
+    "\n",
+    "### Simple linear regression model using **scikit-learn**\n",
+    "\n",
+    "We start with perhaps our simplest possible example, using **Scikit-Learn** to perform linear regression analysis on a data set produced by us. \n",
+    "\n",
+    "What follows is a simple Python code where we have defined a function\n",
+    "$y$ in terms of the variable $x$. Both are defined as vectors with  $100$ entries. \n",
+    "The numbers in the vector $\\hat{x}$ are given\n",
+    "by random numbers generated with a uniform distribution with entries\n",
+    "$x_i \\in [0,1]$ (more about probability distribution functions\n",
+    "later). These values are then used to define a function $y(x)$\n",
+    "(tabulated again as a vector) with a linear dependence on $x$ plus a\n",
+    "random noise added via the normal distribution.\n",
     "\n",
     "\n",
     "The Numpy functions are imported used the **import numpy as np**\n",
@@ -209,7 +1127,7 @@
    "metadata": {},
    "source": [
     "where $N(0,1)$ represents random numbers generated by the normal\n",
-    "distribution.  From **scikit-learn** we import then the\n",
+    "distribution.  From **Scikit-Learn** we import then the\n",
     "**LinearRegression** functionality and make a prediction $\\tilde{y} =\n",
     "\\alpha + \\beta x$ using the function **fit(x,y)**. We call the set of\n",
     "data $(\\hat{x},\\hat{y})$ for our training data. The Python package\n",
@@ -230,14 +1148,12 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 1,
+   "execution_count": 24,
    "metadata": {
     "collapsed": false
    },
    "outputs": [],
    "source": [
-    "%matplotlib inline\n",
-    "\n",
     "# Importing various packages\n",
     "import numpy as np\n",
     "import matplotlib.pyplot as plt\n",
@@ -287,7 +1203,7 @@
    "metadata": {},
    "source": [
     "where $x$ is defined as before.  Does the fit look better? Indeed, by\n",
-    "reducing the role of the normal distribution we see immediately that\n",
+    "reducing the role of the noise given by the normal distribution we see immediately that\n",
     "our linear prediction seemingly reproduces better the training\n",
     "set. However, this testing 'by the eye' is obviouly not satisfactory in the\n",
     "long run. Here we have only defined the training data and our model, and \n",
@@ -297,7 +1213,7 @@
     "not in modeling our training data.  You will be surprised to see that\n",
     "many scientists seldomly venture beyond this 'by the eye' approach. A\n",
     "standard approach for the *cost* function is the so-called $\\chi^2$\n",
-    "function"
+    "function (a variant of the mean-squared error (MSE))"
    ]
   },
   {
@@ -335,7 +1251,7 @@
     "the  $\\chi^2$ function becomes smaller. \n",
     "\n",
     "There are many ways to define the cost function. A simpler approach is to look at the relative difference between the training data and the predicted data, that is we define \n",
-    "the relative error as"
+    "the relative error (why would we prefer the MSE instead of the relative error?) as"
    ]
   },
   {
@@ -351,12 +1267,12 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "We can modify easily the above Python code and plot the relative instead"
+    "We can modify easily the above Python code and plot the relative error instead"
    ]
   },
   {
    "cell_type": "code",
-   "execution_count": 2,
+   "execution_count": 25,
    "metadata": {
     "collapsed": false
    },
@@ -389,18 +1305,18 @@
     "different training data sets and study (graphically) the value of the\n",
     "relative error.\n",
     "\n",
-    "As mentioned above, **scikit-learn** has an impressive functionality.\n",
+    "As mentioned above, **Scikit-Learn** has an impressive functionality.\n",
     "We can for example extract the values of $\\alpha$ and $\\beta$ and\n",
     "their error estimates, or the variance and standard deviation and many\n",
     "other properties from the statistical data analysis. \n",
     "\n",
     "Here we show an\n",
-    "example of the functionality of scikit-learn."
+    "example of the functionality of **Scikit-Learn**."
    ]
   },
   {
    "cell_type": "code",
-   "execution_count": 3,
+   "execution_count": 26,
    "metadata": {
     "collapsed": false
    },
@@ -500,8 +1416,8 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "Another quantity will meet again in our discussions of regression analysis is \n",
-    " mean absolute error (MAE), a risk metric corresponding to the expected value of the absolute error loss or what we call the $l1$-norm loss. In our discussion above we presented the relative error.\n",
+    "Another quantity taht we will meet again in our discussions of regression analysis is \n",
+    " the mean absolute error (MAE), a risk metric corresponding to the expected value of the absolute error loss or what we call the $l1$-norm loss. In our discussion above we presented the relative error.\n",
     "The MAE is defined as follows"
    ]
   },
@@ -547,7 +1463,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 4,
+   "execution_count": 27,
    "metadata": {
     "collapsed": false
    },
@@ -589,248 +1505,422 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "Similarly, using **R**, we can perform similar studies. The following **R** code illustrates this.\n",
-    "(more details on **R** will be inserted later).\n",
+    "### To our real data: nuclear binding energies. Brief reminder on masses and binding energies\n",
     "\n",
-    "## Non-Linear Least squares in R"
+    "Let us now dive into  nuclear physics and remind ourselves briefly about some basic features about binding\n",
+    "energies.  A basic quantity which can be measured for the ground\n",
+    "states of nuclei is the atomic mass $M(N, Z)$ of the neutral atom with\n",
+    "atomic mass number $A$ and charge $Z$. The number of neutrons is $N$. There are indeed several sophisticated experiments worldwide which allow us to measure this quantity to high precision (parts per million even). \n",
+    "\n",
+    "Atomic masses are usually tabulated in terms of the mass excess defined by"
    ]
   },
   {
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "        set.seed(1485)\n",
-    "        len = 24\n",
-    "        x = runif(len)\n",
-    "        y = x^3+rnorm(len, 0,0.06)\n",
-    "        ds = data.frame(x = x, y = y)\n",
-    "        str(ds)\n",
-    "        plot( y ~ x, main =\"Known cubic with noise\")\n",
-    "        s  = seq(0,1,length =100)\n",
-    "        lines(s, s^3, lty =2, col =\"green\")\n",
-    "        m = nls(y ~ I(x^power), data = ds, start = list(power=1), trace = T)\n",
-    "        class(m)\n",
-    "        summary(m)\n",
-    "        power = round(summary(m)$coefficients[1], 3)\n",
-    "        power.se = round(summary(m)$coefficients[2], 3)\n",
-    "        plot(y ~ x, main = \"Fitted power model\", sub = \"Blue: fit; green: known\")\n",
-    "        s = seq(0, 1, length = 100)\n",
-    "        lines(s, s^3, lty = 2, col = \"green\")\n",
-    "        lines(s, predict(m, list(x = s)), lty = 1, col = \"blue\")\n",
-    "        text(0, 0.5, paste(\"y =x^ (\", power, \" +/- \", power.se, \")\", sep = \"\"), pos = 4)\n"
+    "$$\n",
+    "\\Delta M(N, Z) =  M(N, Z) - uA,\n",
+    "$$"
    ]
   },
   {
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "In our lectures on regression analysis (and other ones as well), we will discuss in more details various **R** functionalities. \n",
+    "where $u$ is the Atomic Mass Unit"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "$$\n",
+    "u = M(^{12}\\mathrm{C})/12 = 931.4940954(57) \\hspace{0.1cm} \\mathrm{MeV}/c^2.\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "The nucleon masses are"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "$$\n",
+    "m_p =  1.00727646693(9)u,\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "and"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "$$\n",
+    "m_n = 939.56536(8)\\hspace{0.1cm} \\mathrm{MeV}/c^2 = 1.0086649156(6)u.\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "In the [2016 mass evaluation of by W.J.Huang, G.Audi, M.Wang, F.G.Kondev, S.Naimi and X.Xu](http://nuclearmasses.org/resources_folder/Wang_2017_Chinese_Phys_C_41_030003.pdf)\n",
+    "there are data on masses and decays of 3437 nuclei.\n",
+    "\n",
+    "The nuclear binding energy is defined as the energy required to break\n",
+    "up a given nucleus into its constituent parts of $N$ neutrons and $Z$\n",
+    "protons. In terms of the atomic masses $M(N, Z)$ the binding energy is\n",
+    "defined by"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "$$\n",
+    "BE(N, Z) = ZM_H c^2 + Nm_n c^2 - M(N, Z)c^2 ,\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "where $M_H$ is the mass of the hydrogen atom and $m_n$ is the mass of the neutron.\n",
+    "In terms of the mass excess the binding energy is given by"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "$$\n",
+    "BE(N, Z) = Z\\Delta_H c^2 + N\\Delta_n c^2 -\\Delta(N, Z)c^2 ,\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "where $\\Delta_H c^2 = 7.2890$ MeV and $\\Delta_n c^2 = 8.0713$ MeV.\n",
     "\n",
     "\n",
-    "Another useful Python package is\n",
-    "[pandas](https://pandas.pydata.org/), which is an open source library\n",
-    "providing high-performance, easy-to-use data structures and data\n",
-    "analysis tools for Python. The following simple example shows how we can, in an easy way make tables of our data. Here we define a data set which includes names, city of residence and age, and displays the data in an easy to read way. We will see repeated use of **pandas**, in particular in connection with classification of data."
+    "A popular and physically intuitive model which can be used to parametrize \n",
+    "the experimental binding energies as function of $A$, is the so-called \n",
+    "**liquid drop model**. The ansatz is based on the following expression"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "$$\n",
+    "BE(N,Z) = a_1A-a_2A^{2/3}-a_3\\frac{Z^2}{A^{1/3}}-a_4\\frac{(N-Z)^2}{A},\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "where $A$ stands for the number of nucleons and the $a_i$s are parameters which are determined by a fit \n",
+    "to the experimental data.  \n",
+    "\n",
+    "\n",
+    "\n",
+    "\n",
+    "To arrive at the above expression we have assumed that we can make the following assumptions:\n",
+    "\n",
+    " * There is a volume term $a_1A$ proportional with the number of nucleons (the energy is also an extensive quantity). When an assembly of nucleons of the same size is packed together into the smallest volume, each interior nucleon has a certain number of other nucleons in contact with it. This contribution is proportional to the volume.\n",
+    "\n",
+    " * There is a surface energy term $a_2A^{2/3}$. The assumption here is that a nucleon at the surface of a nucleus interacts with fewer other nucleons than one in the interior of the nucleus and hence its binding energy is less. This surface energy term takes that into account and is therefore negative and is proportional to the surface area.\n",
+    "\n",
+    " * There is a Coulomb energy term $a_3\\frac{Z^2}{A^{1/3}}$. The electric repulsion between each pair of protons in a nucleus yields less binding. \n",
+    "\n",
+    " * There is an asymmetry term $a_4\\frac{(N-Z)^2}{A}$. This term is associated with the Pauli exclusion principle and reflects the fact that the proton-neutron interaction is more attractive on the average than the neutron-neutron and proton-proton interactions.\n",
+    "\n",
+    "We could also add a so-called pairing term, which is a correction term that\n",
+    "arises from the tendency of proton pairs and neutron pairs to\n",
+    "occur. An even number of particles is more stable than an odd number. \n",
+    "\n",
+    "\n",
+    "### Organizing our data\n",
+    "\n",
+    "Let us start with reading and organizing our data. \n",
+    "We start with the compilation of masses and binding energies from 2016.\n",
+    "After having downloaded this file to our own computer, we are now ready to read the file and start structuring our data.\n",
+    "\n",
+    "\n",
+    "We start with preparing folders for storing our calculations and the data file over masses and binding energies. We import also various modules that we will find useful in order to present various Machine Learning methods. Here we focus mainly on the functionality of **scikit-learn**."
    ]
   },
   {
    "cell_type": "code",
-   "execution_count": 5,
+   "execution_count": 28,
    "metadata": {
     "collapsed": false
    },
    "outputs": [],
    "source": [
+    "# Common imports\n",
+    "import numpy as np\n",
     "import pandas as pd\n",
-    "from IPython.display import display\n",
-    "data = {'Name': [\"John\", \"Anna\", \"Peter\", \"Linda\"], 'Location': [\"Nairobi\", \"Napoli\", \"London\", \"Buenos Aires\"], 'Age':[51, 21, 34, 45]}\n",
-    "data_pandas = pd.DataFrame(data)\n",
-    "display(data_pandas)"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "## Examples\n",
-    "\n",
-    "We present here several examples, with pertinent Python codes that we\n",
-    "will use to illustrate various machine learning methods and ways to\n",
-    "analyze, from simple to complex, various data sets. Many of these\n",
-    "examples allow us to generate the data we want to analyze, following\n",
-    "much of the same philosophy we discussed above when\n",
-    "fitting various polynomials.\n",
-    "\n",
-    "We start with a simple exponential growth model that is meant to mimick an ecoli lab experiment.\n",
-    "We can easily model this system and then produce the data used to train various machine learning algorithms.\n",
-    "Another model from the life sciences is the so-called predator-prey model from ecology. Thereafter we present \n",
-    "a simple model for financial transactions before moving to a random walk model and ending with \n",
-    "the simulation of velocities of a non-interacting atom or molecule confined to move in  a one-dimensional region.\n",
-    "\n",
-    "\n",
-    "### Ecoli lab experiment\n",
-    "\n",
-    "A typical pattern seen in population models is that the population grows faster and faster. [Why? Is there an underlying (general) mechanism](http://www.zo.utexas.edu/courses/Thoc/PopGrowth.html)?\n",
-    "Here we will construct a model for cell growth based on a simple difference equation for the growth. We make the following assumptions\n",
-    "1. Cells divide after $T$ seconds on average (one generation)\n",
-    "\n",
-    "2. $2N$ celles divide into twice as many new cells $\\Delta N$ in a time\n",
-    "   interval $\\Delta t$ as $N$ cells would: $\\Delta N \\propto N$\n",
-    "\n",
-    "3. $N$ cells result in twice as many new individuals $\\Delta N$ in\n",
-    "   time $2\\Delta t$ as in time $\\Delta t$: $\\Delta N \\propto\\Delta t$\n",
-    "\n",
-    "4. Same proportionality with respect to  death \n",
-    "\n",
-    "5. Proposed model: $\\Delta N = b\\Delta t N - d\\Delta tN$ for some unknown\n",
-    "   constants $b$ (births) and $d$ (deaths)\n",
-    "\n",
-    "6. Describe evolution in discrete time: $t_n=n\\Delta t$\n",
-    "\n",
-    "7. Program-friendly notation: $N$ at $t_n$ is $N^n$\n",
-    "\n",
-    "8. Math model: $N^{n+1} = N^n + r\\Delta t\\, N$ (with $\\ r=b-d$)\n",
-    "\n",
-    "9. Program model: `N[n+1] = N[n] + r*dt*N[n]`\n",
-    "\n",
-    "\n",
-    "\n",
-    "The difference equation can be programmed in a simple way, and in order to get started we\n",
-    "set $r=1.5$, $N^0=1$, $\\Delta t=0.5$. The program reads"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 6,
-   "metadata": {
-    "collapsed": false
-   },
-   "outputs": [],
-   "source": [
-    "import numpy as np\n",
-    "\n",
-    "t = np.linspace(0, 10, 21)  # 20 intervals in [0, 10]\n",
-    "dt = t[1] - t[0]\n",
-    "N = np.zeros(t.size)\n",
-    "N[0] = 1\n",
-    "r = 0.5\n",
-    "\n",
-    "for n in range(0, N.size-1, 1):\n",
-    "    N[n+1] = N[n] + r*dt*N[n]\n",
-    "    print('N[%d]=%.1f' % (n+1, N[n+1]))"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "and it generates the following output"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "        N[1]=1.2\n",
-    "        N[2]=1.6\n",
-    "        N[3]=2.0\n",
-    "        N[4]=2.4\n",
-    "        N[5]=3.1\n",
-    "        N[6]=3.8\n",
-    "        N[7]=4.8\n",
-    "        N[8]=6.0\n",
-    "        N[9]=7.5\n",
-    "        N[10]=9.3\n",
-    "        N[11]=11.6\n",
-    "        N[12]=14.6\n",
-    "        N[13]=18.2\n",
-    "        N[14]=22.7\n",
-    "        N[15]=28.4\n",
-    "        N[16]=35.5\n",
-    "        N[17]=44.4\n",
-    "        N[18]=55.5\n",
-    "        N[19]=69.4\n",
-    "        N[20]=86.7\n"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "This forms our data which later will define our training set. \n",
-    "In this case we defined the value of the parameter $r$. We could alternatively assume that we just received the \n",
-    "above data file and where asked to find $r$. How can we estimate $r$ from data? This will be one of our tasks later.\n",
-    "\n",
-    "We can use the difference equation with the experimental data"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "$$\n",
-    "N^{n+1} = N^n + r\\Delta t N^n\n",
-    "$$"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "Suppose now that  $N^{n+1}$ and $N^n$ are known from data. Then we could solve with respect to  $r$ as follows"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "$$\n",
-    "r = \\frac{N^{n+1}-N^n}{N^n\\Delta t}\n",
-    "$$"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "Suppose we set $t_1=600$, $t_2=1200$,\n",
-    "$N^1=140$ and $N^2=250$. \n",
-    "The following code plots the data"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 7,
-   "metadata": {
-    "collapsed": false
-   },
-   "outputs": [],
-   "source": [
-    "import numpy as np\n",
     "import matplotlib.pyplot as plt\n",
+    "import sklearn.linear_model as skl\n",
+    "from sklearn.model_selection import train_test_split\n",
+    "from sklearn.metrics import mean_squared_error, r2_score, mean_absolute_error\n",
+    "import os\n",
     "\n",
-    "# Estimate r\n",
-    "data = np.loadtxt('ecoli.csv', delimiter=',')\n",
-    "t_e = data[:,0]\n",
-    "N_e = data[:,1]\n",
-    "i = 2  # Data point (i,i+1) used to estimate r\n",
-    "r = (N_e[i+1] - N_e[i])/(N_e[i]*(t_e[i+1] - t_e[i]))\n",
-    "print('Estimated r=%.5f' % r)\n",
-    "# Can experiment with r values and see if the model can\n",
-    "# match the data better\n",
-    "T = 1200     # cell can divide after T sec\n",
-    "t_max = 5*T  # 5 generations in experiment\n",
-    "t = np.linspace(0, t_max, 1000)\n",
-    "dt = t[1] - t[0]\n",
-    "N = np.zeros(t.size)\n",
+    "# Where to save the figures and data files\n",
+    "PROJECT_ROOT_DIR = \"Results\"\n",
+    "FIGURE_ID = \"Results/FigureFiles\"\n",
+    "DATA_ID = \"DataFiles/\"\n",
     "\n",
-    "N[0] = 100\n",
-    "for n in range(0, len(t)-1, 1):\n",
-    "    N[n+1] = N[n] + r*dt*N[n]\n",
+    "if not os.path.exists(PROJECT_ROOT_DIR):\n",
+    "    os.mkdir(PROJECT_ROOT_DIR)\n",
     "\n",
-    "plt.plot(t, N, 'r-', t_e, N_e, 'bo')\n",
-    "plt.xlabel('time [s]');  plt.ylabel('N')\n",
-    "plt.legend(['model', 'experiment'], loc='upper left')\n",
+    "if not os.path.exists(FIGURE_ID):\n",
+    "    os.makedirs(FIGURE_ID)\n",
+    "\n",
+    "if not os.path.exists(DATA_ID):\n",
+    "    os.makedirs(DATA_ID)\n",
+    "\n",
+    "def image_path(fig_id):\n",
+    "    return os.path.join(FIGURE_ID, fig_id)\n",
+    "\n",
+    "def data_path(dat_id):\n",
+    "    return os.path.join(DATA_ID, dat_id)\n",
+    "\n",
+    "def save_fig(fig_id):\n",
+    "    plt.savefig(image_path(fig_id) + \".png\", format='png')\n",
+    "\n",
+    "infile = open(data_path(\"MassEval2016.dat\"),'r')"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Before we proceed, we define also a function for making our plots. You can obviously avoid this and simply set up various **matplotlib** commands every time you need them. You may however find it convenient to collect all such commands in one function and simply call this function."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 29,
+   "metadata": {
+    "collapsed": false
+   },
+   "outputs": [],
+   "source": [
+    "from pylab import plt, mpl\n",
+    "plt.style.use('seaborn')\n",
+    "mpl.rcParams['font.family'] = 'serif'\n",
+    "\n",
+    "def MakePlot(x,y, styles, labels, axlabels):\n",
+    "    plt.figure(figsize=(10,6))\n",
+    "    for i in range(len(x)):\n",
+    "        plt.plot(x[i], y[i], styles[i], label = labels[i])\n",
+    "        plt.xlabel(axlabels[0])\n",
+    "        plt.ylabel(axlabels[1])\n",
+    "    plt.legend(loc=0)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Our next step is to read the data on experimental binding energies and\n",
+    "reorganize them as functions of the mass number $A$, the number of\n",
+    "protons $Z$ and neutrons $N$ using **pandas**.  Before we do this it is\n",
+    "always useful (unless you have a binary file or other types of compressed\n",
+    "data) to actually open the file and simply take a look at it!\n",
+    "\n",
+    "\n",
+    "In particular, the program that outputs the final nuclear masses is written in Fortran with a specific format. It means that we need to figure out the format and which columns contain the data we are interested in. Pandas comes with a function that reads formatted output. After having admired the file, we are now ready to start massaging it with **pandas**. The file begins with some basic format information."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 30,
+   "metadata": {
+    "collapsed": false
+   },
+   "outputs": [],
+   "source": [
+    "\"\"\"                                                                                                                         \n",
+    "This is taken from the data file of the mass 2016 evaluation.                                                               \n",
+    "All files are 3436 lines long with 124 character per line.                                                                  \n",
+    "       Headers are 39 lines long.                                                                                           \n",
+    "   col 1     :  Fortran character control: 1 = page feed  0 = line feed                                                     \n",
+    "   format    :  a1,i3,i5,i5,i5,1x,a3,a4,1x,f13.5,f11.5,f11.3,f9.3,1x,a2,f11.3,f9.3,1x,i3,1x,f12.5,f11.5                     \n",
+    "   These formats are reflected in the pandas widths variable below, see the statement                                       \n",
+    "   widths=(1,3,5,5,5,1,3,4,1,13,11,11,9,1,2,11,9,1,3,1,12,11,1),                                                            \n",
+    "   Pandas has also a variable header, with length 39 in this case.                                                          \n",
+    "\"\"\""
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "The data we are interested in are in columns 2, 3, 4 and 11, giving us\n",
+    "the number of neutrons, protons, mass numbers and binding energies,\n",
+    "respectively. We add also for the sake of completeness the element name. The data are in fixed-width formatted lines and we will\n",
+    "covert them into the **pandas** DataFrame structure."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 31,
+   "metadata": {
+    "collapsed": false
+   },
+   "outputs": [],
+   "source": [
+    "# Read the experimental data with Pandas\n",
+    "Masses = pd.read_fwf(infile, usecols=(2,3,4,6,11),\n",
+    "              names=('N', 'Z', 'A', 'Element', 'Ebinding'),\n",
+    "              widths=(1,3,5,5,5,1,3,4,1,13,11,11,9,1,2,11,9,1,3,1,12,11,1),\n",
+    "              header=39,\n",
+    "              index_col=False)\n",
+    "\n",
+    "# Extrapolated values are indicated by '#' in place of the decimal place, so\n",
+    "# the Ebinding column won't be numeric. Coerce to float and drop these entries.\n",
+    "Masses['Ebinding'] = pd.to_numeric(Masses['Ebinding'], errors='coerce')\n",
+    "Masses = Masses.dropna()\n",
+    "# Convert from keV to MeV.\n",
+    "Masses['Ebinding'] /= 1000\n",
+    "\n",
+    "# Group the DataFrame by nucleon number, A.\n",
+    "Masses = Masses.groupby('A')\n",
+    "# Find the rows of the grouped DataFrame with the maximum binding energy.\n",
+    "Masses = Masses.apply(lambda t: t[t.Ebinding==t.Ebinding.max()])"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "We have now read in the data, grouped them according to the variables we are interested in. \n",
+    "We see how easy it is to reorganize the data using **pandas**. If we\n",
+    "were to do these operations in C/C++ or Fortran, we would have had to\n",
+    "write various functions/subroutines which perform the above\n",
+    "reorganizations for us.  Having reorganized the data, we can now start\n",
+    "to make some simple fits using both the functionalities in **numpy** and\n",
+    "**Scikit-Learn** afterwards. \n",
+    "\n",
+    "Now we define five variables which contain\n",
+    "the number of nucleons $A$, the number of protons $Z$ and the number of neutrons $N$, the element name and finally the energies themselves."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 32,
+   "metadata": {
+    "collapsed": false
+   },
+   "outputs": [],
+   "source": [
+    "A = Masses['A']\n",
+    "Z = Masses['Z']\n",
+    "N = Masses['N']\n",
+    "Element = Masses['Element']\n",
+    "Energies = Masses['Ebinding']\n",
+    "print(Masses)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "The next step, and we will define this mathematically later, is to set up the so-called **design matrix**. We will throughout call this matrix $\\boldsymbol{X}$.\n",
+    "It has dimensionality $p\\times n$, where $n$ is the number of data points and $p$ are the so-called predictors. In our case here they are given by the number of polynomials in $A$ we wish to include in the fit."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 33,
+   "metadata": {
+    "collapsed": false
+   },
+   "outputs": [],
+   "source": [
+    "# Now we set up the design matrix X\n",
+    "X = np.zeros((len(A),5))\n",
+    "X[:,0] = 1\n",
+    "X[:,1] = A\n",
+    "X[:,2] = A**(2.0/3.0)\n",
+    "X[:,3] = A**(-1.0/3.0)\n",
+    "X[:,4] = A**(-1.0)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "With **scikitlearn** we are now ready to use linear regression and fit our data."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 34,
+   "metadata": {
+    "collapsed": false
+   },
+   "outputs": [],
+   "source": [
+    "clf = skl.LinearRegression().fit(X, Energies)\n",
+    "fity = clf.predict(X)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Pretty simple!  \n",
+    "Now we can print measures of how our fit is doing, the coefficients from the fits and plot the final fit together with our data."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 35,
+   "metadata": {
+    "collapsed": false
+   },
+   "outputs": [],
+   "source": [
+    "# The mean squared error                               \n",
+    "print(\"Mean squared error: %.2f\" % mean_squared_error(Energies, fity))\n",
+    "# Explained variance score: 1 is perfect prediction                                 \n",
+    "print('Variance score: %.2f' % r2_score(Energies, fity))\n",
+    "# Mean absolute error                                                           \n",
+    "print('Mean absolute error: %.2f' % mean_absolute_error(Energies, fity))\n",
+    "print(clf.coef_, clf.intercept_)\n",
+    "\n",
+    "Masses['Eapprox']  = fity\n",
+    "# Generate a plot comparing the experimental with the fitted values values.\n",
+    "fig, ax = plt.subplots()\n",
+    "ax.set_xlabel(r'$A = N + Z$')\n",
+    "ax.set_ylabel(r'$E_\\mathrm{bind}\\,/\\mathrm{MeV}$')\n",
+    "ax.plot(Masses['A'], Masses['Ebinding'], alpha=0.7, lw=2,\n",
+    "            label='Ame2016')\n",
+    "ax.plot(Masses['A'], Masses['Eapprox'], alpha=0.7, lw=2, c='m',\n",
+    "            label='Fit')\n",
+    "ax.legend()\n",
+    "save_fig(\"Masses2016\")\n",
     "plt.show()"
    ]
   },
@@ -838,853 +1928,116 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "We can then change the parameter $r$ in the program and play around to make a better fit. By now we know that this\n",
-    "'search bythe eye' approach is not the most optimal one. \n",
+    "### Seeing the wood for the trees\n",
     "\n",
-    "\n",
-    "### Predator-Prey model from ecology\n",
-    "\n",
-    "The population dynamics of a simple predator-prey system is a\n",
-    "classical example shown in many biology textbooks when ecological\n",
-    "systems are discussed. The system contains all elements of the\n",
-    "scientific method:\n",
-    "\n",
-    " * The set up of a specific hypothesis combined with\n",
-    "\n",
-    " * the experimental methods needed (one can study existing data or perform experiments)\n",
-    "\n",
-    " * analyzing and interpreting the data and performing further experiments if needed\n",
-    "\n",
-    " * trying to extract general behaviors and extract eventual laws or patterns\n",
-    "\n",
-    " * develop mathematical relations for the uncovered regularities/laws and test these by per forming new experiments\n",
-    "\n",
-    "Lots of data about populations of hares and lynx collected from furs in Hudson Bay, Canada, are available. It is known that the populations oscillate. Why?\n",
-    "Here we start by\n",
-    "\n",
-    "1. plotting the data\n",
-    "\n",
-    "2. derive a simple model for the population dynamics\n",
-    "\n",
-    "3. (fitting parameters in the model to the data)\n",
-    "\n",
-    "4. using the model predict the evolution other predator-pray systems\n",
-    "\n",
-    "Most mammalian predators rely on a variety of prey, which complicates mathematical modeling; however, a few predators have become highly specialized and seek almost exclusively a single prey species. An example of this simplified predator-prey interaction is seen in Canadian northern forests, where the populations of the lynx and the snowshoe hare are intertwined in a life and death struggle.\n",
-    "\n",
-    "One reason that this particular system has been so extensively studied is that the Hudson Bay company kept careful records of all furs from the early 1800s into the 1900s. The records for the furs collected by the Hudson Bay company showed distinct oscillations (approximately 12 year periods), suggesting that these species caused almost periodic fluctuations of each other's populations. The table here shows data from 1900 to 1920.\n",
-    "\n",
-    "\n",
-    "\n",
-    "\n",
-    "\n",
-    "\n",
-    "\n",
-    "\n",
-    "\n",
-    "\n",
-    "\n",
-    "\n",
-    "\n",
-    "\n",
-    "\n",
-    "\n",
-    "\n",
-    "\n",
-    "\n",
-    "\n",
-    "\n",
-    "\n",
-    "\n",
-    "\n",
-    "\n",
-    "\n",
-    "\n",
-    "\n",
-    "\n",
-    "
Year Hares (x1000) Lynx (x1000) 
   1900        30.0                 4.0              
   1901        47.2                 6.1              
   1902        70.2                 9.8              
   1903        77.4                 35.2             
   1904        36.3                 59.4             
   1905        20.6                 41.7             
   1906        18.1                 19.0             
   1907        21.4                 13.0             
   1908        22.0                 8.3              
   1909        25.4                 9.1              
   1910        27.1                 7.4              
   1911        40.3                 8.0              
   1912        57                   12.3             
   1913        76.6                 19.5             
   1914        52.3                 45.7             
   1915        19.5                 51.1             
   1916        11.2                 29.7             
   1917        7.6                  15.8             
   1918        14.6                 9.7              
   1919        16.2                 10.1             
   1920        24.7                 8.6              
"
+    "As a teaser, let us now see how we can do this with decision trees using **scikit-learn**. Later we will switch to so-called **random forests**!"
    ]
   },
   {
    "cell_type": "code",
-   "execution_count": 8,
+   "execution_count": 36,
    "metadata": {
     "collapsed": false
    },
    "outputs": [],
    "source": [
-    "import numpy as np\n",
-    "from  matplotlib import pyplot as plt\n",
-    "\n",
-    "# Load in data file\n",
-    "data = np.loadtxt('src/Hudson_Bay.csv', delimiter=',', skiprows=1)\n",
-    "# Make arrays containing x-axis and hares and lynx populations\n",
-    "year = data[:,0]\n",
-    "hares = data[:,1]\n",
-    "lynx = data[:,2]\n",
-    "\n",
-    "plt.plot(year, hares ,'b-+', year, lynx, 'r-o')\n",
-    "plt.axis([1900,1920,0, 100.0])\n",
-    "plt.xlabel(r'Year')\n",
-    "plt.ylabel(r'Numbers of hares and lynx ')\n",
-    "plt.legend(('Hares','Lynx'), loc='upper right')\n",
-    "plt.title(r'Population of hares and lynx from 1900-1920 (x1000)}')\n",
-    "plt.savefig('Hudson_Bay_data.pdf')\n",
-    "plt.savefig('Hudson_Bay_data.png')\n",
-    "plt.show()"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "\n",
-    "\n",
-    "\n",
-    "
\n",
-    "\n",
-    "\n",
-    "\n",
-    "\n",
-    "\n",
-    "\n",
-    "We see from the plot that there are indeed fluctuations.\n",
-    "We would like to create a mathematical model that explains these\n",
-    "population fluctuations. Ecologists have predicted that in a simple\n",
-    "predator-prey system that a rise in prey population is followed (with\n",
-    "a lag) by a rise in the predator population. When the predator\n",
-    "population is sufficiently high, then the prey population begins\n",
-    "dropping. After the prey population falls, then the predator\n",
-    "population falls, which allows the prey population to recover and\n",
-    "complete one cycle of this interaction. Thus, we see that\n",
-    "qualitatively oscillations occur. Can a mathematical model predict\n",
-    "this? What causes cycles to slow or speed up? What affects the\n",
-    "amplitude of the oscillation or do you expect to see the oscillations\n",
-    "damp to a stable equilibrium? The models tend to ignore factors like\n",
-    "climate and other complicating factors. How significant are these?\n",
-    "\n",
-    " * We see oscillations in the data\n",
-    "\n",
-    " * What causes cycles to slow or speed up?\n",
-    "\n",
-    " * What affects the amplitude of the oscillation or do you expect to see the oscillations damp to a stable equilibrium?\n",
-    "\n",
-    " * With a model we can better *understand the data*\n",
-    "\n",
-    " * More important: Can we understand the ecology dynamics of predator-pray populations?\n",
-    "\n",
-    "The classical way (in all books) is to present the Lotka-Volterra equations:"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "$$\n",
-    "\\begin{align*}\n",
-    "\\frac{dH}{dt} &= H(a - b L)\\\\\n",
-    "\\frac{dL}{dt} &= - L(d - c  H)\n",
-    "\\end{align*}\n",
-    "$$"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "Here,\n",
-    "\n",
-    " * $H$ is the number of preys\n",
-    "\n",
-    " * $L$ the number of predators\n",
-    "\n",
-    " * $a$, $b$, $d$, $c$ are parameters\n",
-    "\n",
-    "The population of hares evolves due to births and deaths exactly as a bacteria population:"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "$$\n",
-    "\\Delta H = a \\Delta t H^n\n",
-    "$$"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "However, hares have an additional loss in the population because\n",
-    "they are eaten by lynx.\n",
-    "All the hares and lynx can form\n",
-    "$H\\cdot L$ pairs in total. When such pairs meet during a time\n",
-    "interval $\\Delta t$, there is some\n",
-    "small probablity that the lynx will eat the hare.\n",
-    "So in fraction $b\\Delta t HL$, the lynx eat hares. This\n",
-    "loss of hares must be accounted for. Subtracted in the equation for hares:"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "$$\n",
-    "\\Delta H = a\\Delta t H^n - b \\Delta t H^nL^n\n",
-    "$$"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "We assume that the primary growth for the lynx population depends on sufficient food for raising lynx kittens, which implies an adequate source of nutrients from predation on hares. Thus, the growth of the lynx population does not only depend of how many lynx there are, but on how many hares they can eat.\n",
-    "In a time interval $\\Delta t HL$ hares and lynx can meet, and in a\n",
-    "fraction $b\\Delta t HL$ the lynx eats the hare. All of this does not\n",
-    "contribute to the growth of lynx, again just a fraction of\n",
-    "$b\\Delta t HL$ that we write as\n",
-    "$d\\Delta t HL$. In addition, lynx die just as in the population\n",
-    "dynamics with one isolated animal population, leading to a loss\n",
-    "$-c\\Delta t L$.\n",
-    "The accounting of lynx then looks like"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "$$\n",
-    "\\Delta L = d\\Delta t H^nL^n - c\\Delta t L^n\n",
-    "$$"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "By writing up the definition of $\\Delta H$ and $\\Delta L$, and putting\n",
-    "all assumed known terms $H^n$ and $L^n$ on the right-hand side, we have"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "$$\n",
-    "H^{n+1} = H^n + a\\Delta t H^n - b\\Delta t H^n L^n\n",
-    "$$"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "$$\n",
-    "L^{n+1} = L^n + d\\Delta t H^nL^n - c\\Delta t L^n\n",
-    "$$"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "Note:\n",
-    "\n",
-    " * These equations are ready to be implemented!\n",
-    "\n",
-    " * But to start, we need $H^0$ and $L^0$ (which we can get from the data)\n",
-    "\n",
-    " * We also need values for $a$, $b$, $d$, $c$\n",
-    "\n",
-    " * As always, models tend to be general - as here, applicable\n",
-    "   to \"all\" predator-pray systems\n",
-    "\n",
-    " * The critical issue is whether the *interaction* between hares and lynx\n",
-    "   is sufficiently well modeled by $\\hbox{const}HL$\n",
-    "\n",
-    " * The parameters $a$, $b$, $d$, and $c$ must be\n",
-    "   estimated from data"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 9,
-   "metadata": {
-    "collapsed": false
-   },
-   "outputs": [],
-   "source": [
-    "import numpy as np\n",
-    "import matplotlib.pyplot as plt\n",
-    "\n",
-    "def solver(m, H0, L0, dt, a, b, c, d, t0):\n",
-    "    \"\"\"Solve the difference equations for H and L over m years\n",
-    "    with time step dt (measured in years.\"\"\"\n",
-    "\n",
-    "    num_intervals = int(m/float(dt))\n",
-    "    t = np.linspace(t0, t0 + m, num_intervals+1)\n",
-    "    H = np.zeros(t.size)\n",
-    "    L = np.zeros(t.size)\n",
-    "\n",
-    "    print('Init:', H0, L0, dt)\n",
-    "    H[0] = H0\n",
-    "    L[0] = L0\n",
-    "\n",
-    "    for n in range(0, len(t)-1):\n",
-    "        H[n+1] = H[n] + a*dt*H[n] - b*dt*H[n]*L[n]\n",
-    "        L[n+1] = L[n] + d*dt*H[n]*L[n] - c*dt*L[n]\n",
-    "    return H, L, t\n",
-    "\n",
-    "# Load in data file\n",
-    "data = np.loadtxt('src/Hudson_Bay.csv', delimiter=',', skiprows=1)\n",
-    "# Make arrays containing x-axis and hares and lynx populations\n",
-    "t_e = data[:,0]\n",
-    "H_e = data[:,1]\n",
-    "L_e = data[:,2]\n",
-    "\n",
-    "# Simulate using the model\n",
-    "H, L, t = solver(m=20, H0=34.91, L0=3.857, dt=0.1,\n",
-    "                 a=0.4807, b=0.02482, c=0.9272, d=0.02756,\n",
-    "                 t0=1900)\n",
-    "\n",
-    "# Visualize simulations and data\n",
-    "plt.plot(t_e, H_e, 'b-+', t_e, L_e, 'r-o', t, H, 'm--', t, L, 'k--')\n",
-    "plt.xlabel('Year')\n",
-    "plt.ylabel('Numbers of hares and lynx')\n",
-    "plt.axis([1900, 1920, 0, 140])\n",
-    "plt.title(r'Population of hares and lynx 1900-1920 (x1000)')\n",
-    "plt.legend(('H_e', 'L_e', 'H', 'L'), loc='upper left')\n",
-    "plt.savefig('Hudson_Bay_sim.pdf')\n",
-    "plt.savefig('Hudson_Bay_sim.png')\n",
-    "plt.show()"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "\n",
-    "\n",
-    "\n",
-    "
\n",
-    "\n",
-    "\n",
-    "\n",
-    "\n",
-    "\n",
-    "We will later perform a least-square fitting. Then we can find optimal\n",
-    "values for the parameters $a$, $b$, $d$, $c$. In our calculations here\n",
-    "we set $a=0.4807$, $b=0.02482$, $d=0.9272$ and $c=0.02756$. These\n",
-    "parameters result in a slightly modified initial conditions, namely\n",
-    "$H(0) = 34.91$ and $L(0)=3.857$. \n",
-    "\n",
-    "\n",
-    "The following Python code demonstrates how we can use linear regression to fit for example the population of lynx.\n",
-    "Similarly, we have also used a decision tree algorithm to fit the lynx population data. As expected, the linear regression is not exactly impressive"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 10,
-   "metadata": {
-    "collapsed": false
-   },
-   "outputs": [],
-   "source": [
-    "import numpy as np\n",
-    "import matplotlib.pyplot as plt\n",
-    "from IPython.display import display\n",
-    "import sklearn\n",
-    "from sklearn.linear_model import LinearRegression\n",
-    "from sklearn.tree import DecisionTreeRegressor\n",
-    "\n",
-    "\n",
-    "data = np.loadtxt('src/Hudson_Bay.csv', delimiter=',', skiprows=1)\n",
-    "x = data[:,0]\n",
-    "y = data[:,1]\n",
-    "line = np.linspace(1900,1920,1000,endpoint=False).reshape(-1,1)\n",
-    "reg = DecisionTreeRegressor(min_samples_split=3).fit(x.reshape(-1,1),y.reshape(-1,1))\n",
-    "plt.plot(line, reg.predict(line), label=\"decision tree\")\n",
-    "regline = LinearRegression().fit(x.reshape(-1,1),y.reshape(-1,1))\n",
-    "plt.plot(line, regline.predict(line), label= \"Linear Regression\")\n",
-    "plt.plot(x, y, label= \"Linear Regression\")\n",
-    "plt.show()"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "The similar code for linear regression in **R** reads (more details to come)"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "        HudsonBay = read.csv(\"src/Hudson_Bay.csv\",header=T)\n",
-    "        fix(HudsonBay)\n",
-    "        dim(HudsonBay)\n",
-    "        names(HudsonBay)\n",
-    "        plot(HudsonBay$Year, HudsonBay$Hares..x1000.)\n",
-    "        attach(HudsonBay)\n",
-    "        plot(Year, Hares..x1000.)\n",
-    "        plot(Year, Hares..x1000., col=\"red\", varwidth=T, xlab=\"Years\", ylab=\"Haresx 1000\")\n",
-    "        summary(HudsonBay)\n",
-    "        summary(Hares..x1000.)\n",
-    "        library(MASS)\n",
-    "        library(ISLR)\n",
-    "        scatter.smooth(x=Year, y = Hares..x1000.)\n",
-    "        linearMod = lm(Hares..x1000. ~ Year)\n",
-    "        print(linearMod)\n",
-    "        summary(linearMod)\n",
-    "        plot(linearMod)\n",
-    "        confint(linearMod)\n",
-    "        predict(linearMod,data.frame(Year=c(1910,1914,1920)),interval=\"confidence\")\n"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "### Simulating financial transactions\n",
-    "\n",
-    "The aim here is to simulate financial transactions among financial agents\n",
-    "using Monte Carlo methods. The final goal is to extract a distribution of income  as function\n",
-    "of the income $m$.   From Pareto's work ([V. Pareto, 1897](http://www.institutcoppet.org/2012/05/08/cours-deconomie-politique-1896-de-vilfredo-pareto)) it is known from empirical studies\n",
-    "that the higher end of the distribution of money follows a distribution"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "$$\n",
-    "w_m\\propto m^{-1-\\alpha},\n",
-    "$$"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "with $\\alpha\\in [1,2]$. We will here follow the analysis made by [Patriarca and collaborators](http://www.sciencedirect.com/science/article/pii/S0378437104004327). \n",
-    "\n",
-    "Here we will study numerically the relation between the micro-dynamic relations among financial \n",
-    "agents and the  resulting macroscopic money distribution.\n",
-    "\n",
-    "We assume we have $N$ agents that exchange money in pairs $(i,j)$. We assume also that all agents\n",
-    "start with the same amount of money $m_0 > 0$. At a given 'time step', we choose randomly a pair\n",
-    "of agents $(i,j)$ and let a transaction take place. This means that agent $i$'s money $m_i$ changes\n",
-    "to $m_i'$ and similarly we have $m_j\\rightarrow m_j'$. \n",
-    "Money is conserved during a transaction, meaning that"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "\n",
-    "
\n",
-    "\n",
-    "$$\n",
-    "\\begin{equation}\n",
-    "  m_i+m_j=m_i'+m_j'.\n",
-    "\\label{eq:conserve} \\tag{1}\n",
-    "\\end{equation}\n",
-    "$$"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "The change is done via a random reassignement (a random number) $\\epsilon$, meaning that"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "$$\n",
-    "m_i' = \\epsilon(m_i+m_j),\n",
-    "$$"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "leading to"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "$$\n",
-    "m_j'= (1-\\epsilon)(m_i+m_j).\n",
-    "$$"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "The number $\\epsilon$ is extracted from a uniform distribution.\n",
-    "In this simple model, no agents are left with a debt, that is $m\\ge 0$.\n",
-    "Due to the conservation law above, one can show that the system relaxes toward an equilibrium\n",
-    "state given by a Gibbs distribution"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "$$\n",
-    "w_m=\\beta \\exp{(-\\beta m)},\n",
-    "$$"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "with"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "$$\n",
-    "\\beta = \\frac{1}{\\langle m\\rangle},\n",
-    "$$"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "and $\\langle m\\rangle=\\sum_i m_i/N=m_0$, the average money.\n",
-    "It means that after equilibrium has been reached that the majority of agents is left with a small\n",
-    "number of money, while the number of richest agents, those with $m$ larger than a specific value $m'$,\n",
-    "exponentially decreases with $m'$.\n",
-    "\n",
-    "We assume that we have $N=500$ agents.   In each simulation, we need a sufficiently large number of transactions, say $10^7$. Our aim is find the final equilibrium distribution $w_m$. In order to do that we would need\n",
-    "several runs of the above simulations, at least $10^3-10^4$ runs (experiments).\n",
-    "\n",
-    "Our task is to first set up an algorithm which simulates the above transactions with an initial\n",
-    "  amount $m_0$.\n",
-    "  The challenge here is to figure out a Monte Carlo  simulation  based on the\n",
-    "  above equations.\n",
-    "  You will in particular need to make an algorithm which sets up a histogram as function of $m$.\n",
-    "  This histogram contains the number of times a value $m$ is registered and represents\n",
-    "  $w_m\\Delta m$. You will need to set up a value for the interval $\\Delta m$  (typically $0.01-0.05$).\n",
-    "  That means you need to account for the number of times you register an income in the interval\n",
-    "  $m,m+\\Delta m$. The number of times you register this income, represents the value that enters the histogram."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 11,
-   "metadata": {
-    "collapsed": false
-   },
-   "outputs": [],
-   "source": [
-    "#!/usr/bin/env python\n",
-    "import numpy as np\n",
-    "import matplotlib.mlab as mlab\n",
-    "import matplotlib.pyplot as plt\n",
-    "import random\n",
-    "\n",
-    "# initialize the rng with a seed\n",
-    "random.seed()\n",
-    "# Hard coding of input parameters\n",
-    "Agents  = 500\n",
-    "MCcounts = 1000\n",
-    "Transactions = 100000\n",
-    "startMoney = 1.0\n",
-    "Lambda = 0.0\n",
-    "FinancialAgents = startMoney*np.ones(Agents)\n",
-    "for i in range (1, MCcounts, 1):\n",
-    "    for j in range (1, Transactions, 1):\n",
-    "        agent_i = int(Agents*random.random())\n",
-    "        agent_j = int(Agents*random.random())\n",
-    "        epsilon = random.random()\n",
-    "        if agent_i != agent_j:\n",
-    "           m1 = Lambda*FinancialAgents[agent_i] + (1-Lambda)*epsilon*(FinancialAgents[agent_i] + FinancialAgents[agent_j])\n",
-    "           m2 = Lambda*FinancialAgents[agent_j] + (1-Lambda)*(1-epsilon)*(FinancialAgents[agent_i] + FinancialAgents[agent_j])\n",
-    "           FinancialAgents[agent_i] = m1\n",
-    "           FinancialAgents[agent_j] = m2\n",
-    "\n",
-    "# the histogram of the data\n",
-    "n, bins, patches = plt.hist(FinancialAgents, 50, facecolor='green')\n",
-    "\n",
-    "plt.xlabel('$x$')\n",
-    "plt.ylabel('Distribution of wealth')\n",
-    "plt.title(r'Money')\n",
-    "plt.axis([0, 10, 0, 500])\n",
-    "plt.grid(True)\n",
-    "plt.show()"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "We can then change our model to allow for a saving criterion, meaning that the agents save\n",
-    "  a fraction $\\lambda$ of the money they have before the transaction is made. The final distribution will then no longer be given by Gibbs distribution. It could also include a taxation on financial transactions.\n",
-    "\n",
-    "  The conservation law of Eq. ([1](#eq:conserve)) holds, but the money to be shared in a transaction between\n",
-    "  agent $i$ and agent $j$ is now $(1-\\lambda)(m_i+m_j)$. This means that we have"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "$$\n",
-    "m_i' = \\lambda m_i+\\epsilon(1-\\lambda)(m_i+m_j),\n",
-    "$$"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "and"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "$$\n",
-    "m_j' = \\lambda m_j+(1-\\epsilon)(1-\\lambda)(m_i+m_j),\n",
-    "$$"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "which can be written as"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "$$\n",
-    "m_i'=m_i+\\delta m\n",
-    "$$"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "and"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "$$\n",
-    "m_j'=m_j-\\delta m,\n",
-    "$$"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "with"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "$$\n",
-    "\\delta m=(1-\\lambda)(\\epsilon m_j-(1-\\epsilon)m_i),\n",
-    "$$"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "showing how money is conserved during a transaction.\n",
-    "  Select values of $\\lambda =0.25,0.5$ and $\\lambda=0.9$ and try to extract the corresponding\n",
-    "  equilibrium distributions and compare these with the Gibbs distribution. We will use this model to \n",
-    "extract a parametrization of the above curves, see for example [Patriarca and collaborators](http://www.sciencedirect.com/science/article/pii/S0378437104004327).\n",
-    "\n",
-    "\n",
-    "### Particle in one dimension and velocity distribution"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 12,
-   "metadata": {
-    "collapsed": false
-   },
-   "outputs": [],
-   "source": [
-    "# Program to test the Metropolis algorithm with one particle at given temp in one dimension\n",
-    "import numpy as np\n",
-    "import matplotlib.mlab as mlab\n",
-    "import matplotlib.pyplot as plt\n",
-    "import random\n",
-    "from math import sqrt, exp, log\n",
-    "# initialize the rng with a seed\n",
-    "random.seed()\n",
-    "# Hard coding of input parameters\n",
-    "MCcycles = 100000\n",
-    "Temperature = 2.0\n",
-    "beta = 1./Temperature\n",
-    "InitialVelocity = -2.0\n",
-    "CurrentVelocity = InitialVelocity\n",
-    "Energy = 0.5*InitialVelocity*InitialVelocity\n",
-    "VelocityRange = 10*sqrt(Temperature)\n",
-    "VelocityStep = 2*VelocityRange/10.\n",
-    "AverageEnergy = Energy\n",
-    "AverageEnergy2 = Energy*Energy\n",
-    "VelocityValues = np.zeros(MCcycles)\n",
-    "# The Monte Carlo sampling with Metropolis starts here\n",
-    "for i in range (1, MCcycles, 1):\n",
-    "    TrialVelocity = CurrentVelocity + (2.0*random.random() - 1.0)*VelocityStep\n",
-    "    EnergyChange = 0.5*(TrialVelocity*TrialVelocity -CurrentVelocity*CurrentVelocity);\n",
-    "    if random.random() <= exp(-beta*EnergyChange):\n",
-    "        CurrentVelocity = TrialVelocity\n",
-    "        Energy += EnergyChange\n",
-    "        VelocityValues[i] = CurrentVelocity\n",
-    "    AverageEnergy += Energy\n",
-    "    AverageEnergy2 += Energy*Energy\n",
-    "#Final averages\n",
-    "AverageEnergy = AverageEnergy/MCcycles\n",
-    "AverageEnergy2 = AverageEnergy2/MCcycles\n",
-    "Variance = AverageEnergy2 - AverageEnergy*AverageEnergy\n",
-    "print(AverageEnergy, Variance)\n",
-    "n, bins, patches = plt.hist(VelocityValues, 400, facecolor='green')\n",
-    "\n",
-    "plt.xlabel('$v$')\n",
-    "plt.ylabel('Velocity distribution P(v)')\n",
-    "plt.title(r'Velocity histogram at $k_BT=2$')\n",
-    "plt.axis([-5, 5, 0, 600])\n",
-    "plt.grid(True)\n",
-    "plt.show()"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "### Random walk model"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 13,
-   "metadata": {
-    "collapsed": false
-   },
-   "outputs": [],
-   "source": [
-    "import numpy as np\n",
-    "import matplotlib.pyplot as plt\n",
-    "from sklearn.preprocessing import PolynomialFeatures\n",
-    "from sklearn.linear_model import LinearRegression\n",
-    "\n",
-    "steps=250\n",
-    "\n",
-    "distance=0\n",
-    "x=0\n",
-    "distance_list=[]\n",
-    "steps_list=[]\n",
-    "while x_W@ct=@PwI}>4Z5AGxG^EGeg77Ibm*?n}!-{H_yBK&F2!vE_OanHbEIT
zTP89NE;4x!hyUqL#?B$Y#VNqeMkcGIA;|Xc8p=k+#@XE&Xyfcc#>v74!6zZ71Oo%{
z2~Y||P_PgPT>R|ZY}~x&L{Rt;FbbN=vi~7aGMfJtuqxupQsVz%kevTR|Ky10e+cRt
za*B%oVQ~M0ssD$xwCZU^#3!@|80x^hkcJvS-foBZJa&a$y_Ym9BqL>TW2R0A}DW&s{g-rfQqE_
zAN!vo`;bYxS$--`fb4%PfXvK??EgU>WOV<_5a>~n{y#MF|JCq+s0I}Y6^RJy(~=-y
zApUJZNJ7BDz{0`8z`?=7!NbELAfY26AtEASe?db*|B8)=_Z1r#mw=d>lz@BL=>C6(e?t%$2vD5RXV8!o
z5KtJ9&=`>aMj(hj*@XV@@cn1{{{zr4ke~Js2aoVc#fOCWkNCex|CNwX&@iz7HX%@;
zAt9j9q0v8E9)Kl^OQr!Wz|Skf+s*-$R44En~$C{P_@4Ni|=kMBQs}PbN$|;gxQ)v>wuoK
zhf?pEiqNfO(Sh{#a{E4x6DV-HXgzjjGg;rcETx9uld!$XC{SJ2?I-sEtrwNM8W~-oB3k=?c
z|Ngd%WX$!NftgB@@zsg{CnBUS{j$_?Tbsp*a|5vk!!C`z=?V_|7%GZ7)oSx%DbchC
z0p?F{%!Ru*9e(8kA8UPPm5^AW$bf`})vd(3BCxG1Rn
zf>AZ_uRZm9#eoMAz{<&lo;uOTj_ru-gv@W_b$sk3L
z4Sa2#@=y>F8!Uy{xfb)On
z(EXtqO6mynP)t}1(*S>TcpOdoM#T17i=ggsJ(U?pce8KGjCrC(ZnZ9^0f8RqlUTr4
z#?=d3f^Syo3uL+8BR>-|veNFkCz(>$j$Usa)Ni~T;HLJGh*2FcxHH!LJ^D&Sa7BH;
zr+f538RV5-vgI}y;$qgSrntvudjBBUmE@EB+Oe@lnXa$FNx#uCnW?E)YtF6pUIyz@
zoPCI;k2Qi5yHw1nHHaM|Oeg82K}3FRW0B25y92y8dK8}BxomP5+UO`UhpZ3GgU}l^
z%p&H+3u?Up}$JJNbinW2Ugt3PFpqXop7;3e>v@cncl4{DXHJHri-H2
zF{S{lc*h0Rx80)^`rKAfM<0pg9Su*s7+y^Y!#DjViQ#zUW&*HXjI(>Yi4&ZmswL?~
zHBkAjsAO4qMA|Z<)3RXDLz8mFQUx;JlgQm`=rJ9O6
zNAn>q>8S6mK*yvb)Pv(Q?)%tPg@0tZK_GU2<_vuE9l;BPyO33Afg7>|ZR&A=b`F-hd?%cUXV6SrCdKxZZ51CK(CW4211(W*8GyT-#Y%^hGgu4Kky_{-9*K7!vK((VV>J^zh@
zumnSDXlQB502Pn@%J=g5`t<7NHu0GfIk7q^^l1_qZ3c#nmC3a>A0kWb17On2P8`YK
z-o4WI`D>yN-d-`cU28;=ldiYPKSVzO$5I`=FKe|#K0i$F-zGDf8WQreZy6Qc9|Ili
zZINwH#y0%Qgz}#;0>5`0$9J6yIG>NbWcHRQcS+;MJbu$S!^-%zb$5uSM7XC@Y03Hl
zI1(8;+?unc%#?}Vt#l8@GOI6P)|-}LLnXJ+l(bMJFXDeh?e&;CFaC1Rz~Wv@1;+xg
ztVj6*%A3NMpP|@vRI;+^Q-wj2(qvyZlc674MYepI3&F@HF(*4?TPvfLLDu8WJjup9
zgsV?B%R5juE6KG%p1*#1xk_d9Ia}QLMW`8B%~s5hr&80tnuMD7HPQkgi%d&tAkrb5
z^yW1j*fqM`6y6g?%JQg}6Mc44c8dO7b_YOSVVM)V_b08s%lYZK;z=WA+2ZHBksmjH
z>b=`DFB0x-zmsb{zs7wbp0jFCv$^iO`ZqG}!f3D8G5q5Gm)%CFEs?midQ)a$KCRQs
zb8C>AGk>o)BKZlrJE5T}%h+V9mq1IlZXaaDtPK-7cagY_pjeN=C2@_Oo4B>M#t(Ea
zHcKCMEv+SXi%#iuor(&}e7vgapA8(Ma!EC)b(Hke*IW*o!c<_b)$XLN>J>(C@tv}1
zEmFugsG>tE-&k;hr4OQ=k9;eb+PIjUBpA*3s4wQ!tytiQ><&igSeMlkTi>pvtTx@~%_tShUBh)v8Sv3Sb#%u%9!oB;=ay}oy
zKXboNyB^Zg7@<+XP)nK7%EriLZ{`kx{E0M*kp~=i|TXNrU9J8iFcL
zA6gE)GP)yB8f`A&hBb=~qr!`3p(3!79xzX{>Z^dLU!2*>`0>fs425p^7!*|qNZsRc
zv9*pWR9D;d=6ZLIyzGE`C>!UcaFn1`yF3@TDa1u`47L!D*A--T+)zsJ|
zSSuex7}g(EVBe+GL8U80*n(2LN}p?19qICReC0fK=d`0@_&%>f$(Sx$WU5|RgcYjE
zN=z^Z=4yHof~f3gy?6GGY*d98mMG@pDcFxxV!GJm;m#T!PCRsfoyy=keFR-#P|#Vc
zeP6mXolbu3vS2$_QF)k^hZ%BUu(a+-P)%!;D$A>-wn;f`%B+`Xg_hc$J(5XK%c`7U
zQ3w+TirQ1Ebwl&7(!^`gD_5EReA=Tt-d&A*?6!Wzu_wawXK(lBzV-8Yvx!vah)3qN
zIi}7@F17g*;LfNIV-)>^=|p$)*g%j~i8v;Sid!gR!aUOC{06P~#@}mu`jJbFo-HZ~
z;}u}l7PEAx
znNqZRNKLJrMdjHtG`bjp?z|e4hvle=QnN!-dTk1H>A&}&y##mG?!%Rm&VlxVv)YP0KzF~R^w{Pqx(;>2H#W9~oR{+Y=xs_EGE>qb#rnICjM&x}_7E`&8
zR=%4p7DvT=DXw!_lNOf0Y%Ql{sQQ^PJ#T+${_(BBcQ&jb>Q|7Ce~{2!f!a(dh0{Lp
zJU8;(|A|$h*-x=QF}W2@A#pZ!aeo9BYmHxJ@U%*(qwM~1VHS_bEDN8$#?zYLj@n*U
zcw3ddSD1v*%y{7Bhp>zO(nyuCp*fEtY+bzw@eK=nfB3zyk>AS1M1L|i^R@E#+BvHn
zgQ28>jm8E9j8OsmIx5&?l$k!hX@Q~=~l{)({zSF};Cf=4pKW+W+mC`)+0%urn8(biv(X_++q@bi)3R$F?K4XS
zd}PX?C2iyCKhMkk9FF}@<*p(%%RUA9I1j9X_YHO=JBc`#)XT8{AVf}QB9gWf76unn
zp@`YmddB9Q*mq|h`l-%dt(*SvE{3j}j=2SDI{O@Lp;eA6&I$?K2w8z^PN>V0>1_+y
z#E7({kIrGzii@KRKspwVXY_pC3;_@mIeL#Mk9m5VF=udNT<@ct4USERU$l$w^0SpI
z&H;DxZkOP*pBMyZ#aT6{T+Py_e~1-4qq;3T7Xflx<`8ARqy7hkG@eOaJk_(uwQuYr
zPjo80(g+Gia89PvHk)t;m`b7%=jv1SDLIhc_e%LW6gx<5<^Ipq3m4n78o8GSr(=CzBjq$v-W!-r43=IT2S
zxIU!CZ&2k`4HNVCffJDT4y4uZ-qEm7H@*zBc|l(_`gVg-Dm$8?OIKUC!^e3lxBGAG
zCsly6kcDDdsQWPD8*xUCD%D?(ju&p~6%4kzxtw5%R~*A5}pEG
z%IR1nQ^as$sA?3V6GmB9uI=P$f{5FqJR-q_&^kF2TxFp=mpp46g#sFAnpo`T6iv`8
z$dX<}D>#9X_l|2FKj3tXrGHP6{m6RUE=U3T5w{dYCmf{!TA@TLx`=mBZqB4NZRGwK
z=LaC4))p$lbzb`oyyOYb>AbfE5Wt)&_vp@PwF@ntG4*JUkHLmBm0EPBofMVMP_mkE
z7|%DdbVtrNt>d+n#9>I{pa$1QM6Flk`@kCxL`&zQ#&&^x>O|?!*m$~A^2_o)xhbsi
z99IhOUgnyFcS-YBPHQjQf8SpCe<@q4e08le+aDX#I^{lGD62jDiHlXJ>Z@lbQbO_G
zf-tni7?&`?Xo#L<7;$c234;(Rt=}OuHwCNC8_R*LhOA1FKtAi>D(K|UXuYmcq7Im3
za`(&ij$+&~GZC9>MNJt99A^HQ+s{h)>S5wpEB5U*YU0oHj*xhn27kRr
zj-Saf%aKDLU?NeHw#e@4Wp-#*mva-kq??!);VDO|m_}-c|F8NZBQTJ)8fPXd{THd+
zwf0!6@N?GnIAX@wez)1e*aqhxKKl^R*8mo_F0sPz<4YQ_CId#jqsk(@)pC2FYlp|`U+*~iEr=`K(p|e*YyH}am&iJc7nU2Tr`ZbZ!+n=j@D{h46nm-cpo03?5V^bcHu{X)%$e7oOI
z4|kBs6czv@`3@I*B4r9KylRF{33AO_=CVdXwvytrW9Ew+|J6P`U3&HVo>0o=aVe#H
znh@8gRAIgFPJfsSNoJfv+>%Pn*5E!{NMA3={*rs=mQ(#Cgeq=Xl{R~^UL*Co{go|FQR)LYVw@TK5^2dudazbyLzTL1bd
z&}|bq`vIOn1}^!8I3|>m@HVY)Vhk7X1D5G_vUIU|;FhgqvPuQ!g$HCb
zO`FNGyee%+B>Y!l0C#f+CP~k9?&&MWLCvN#d3<+lWe0MY4N(Z03vSGSq88MmuC-oD
zbdHO|hyC$C2+5eWcE_BhuzmdMB6!9E8(AHlu2B`m`=+l%IOk2jR_lSGQsAl$DZB4ksH?e)sLgB}0n8
zwZ6XvaG%Y`xvUhzr)8u@V^k!k_h%TL7XI$iaRnO}fi$VLIrChV^S`f`3++Ic;M1(t
zZgxmLR)Xs_b1Eu-`#lTb*HP0L_x&ch>FPsETOI>FOlIlUwgxCGUd9X`_lC5l(XZVz
zsb)`HLY`-Tty`Xsi`JLqh~{4a*V%7o$BBZLtQfrJ@&uA4NW2vITe}jfrVh4BieLGI
zECG%a5#8;0L%9gkR;?By*2xb)a!#}VsOB6g=2wP7T#-eWsTNdToami-fUM7#_A_G2
z=VB$oxqjx!8okuFwZ2Cd;>fJXsp>P$r7gRXR@*73
zKatc6)i6j<6!f^Ob={U-Z*Y;<3qqOp>yo8370@b&!VGt7B~~4F`_j=mi&l|sTN+zw
z#-}LY4n!@Vwd_*+!5*bi3RujUuxEDhL$CRwFE;1CF`Q8V$*@$5lB&yKyV}6h!y&}O
zv!7R~=))I%zyqtJv4yy)mFlfcm~VR3jdSw43OnxT+wphh_g)o^_=HTwyeqeK?5)sz
zCGH7Upeg@7x63Te3QP^Fo?7UZ+r73pPn?aNK_0KBj}|dfo?@EKEO!pRVSNDg$Iv}^
zv#@tkm>c%qAkp-`gjkz>=uk4G)~Y`V8)5DtDi$gL(|qm}n#}YNjUhobrlCZ4De+Vv
z)gv?|6@b8`X9E4L1jIxEQJ)24{=9{8$x9M9{
z#gBfZk%NCtD;vg*+?0gLK~Tn{YiES+p5jDfm%t{EQc^J(>&PzHqDz_eIQ=pm7tGNh$VzXnc6oQxQc
z3(Xo{qKlTINHrxw8r$C4I>JO^XWf)lKHoY?*(Lt_iQ)8!Z#hWiOvtN#>7^DK3p0T;
zx#|(0ZfBM$T6IB@{g>=VKY=mx5uR7&lM+^QjtrL>B)7mLl2xuK!8|KALVN`6<>^3!
zH4~DntjuaQrtdA8#k*mv8-;h?M%-wPfBxq05k87$HLtC^m}*%4{7^WZ%fIh_q?%(6
zF5D=l#%j?vrg#ztu?pbADzX*X+RZ@XG0-6@2)mvZ_t-z&X89$(vRjZDGoSV2!abzf
z7(LHjzd!Pix;tkdY*^11sUfos<${+!Y|V9kIyovJV0fF*jl%tCpTB0MgOegn|J*mGotXO|@AD1K(>ebPS)J6$p6trf2DA~H(_z01QdHzp#;0NuEMuKd
zl!4~D)-;k9+TyT~H9Rb6^eovGF$XQjQuEmtku=}ggtTRylz!9AY0Bo%Jw
z2;o1M+sbx+ys7mM0t$#WqGo1zeCdA#??ov^toDiVtD&O&J?*9*Br_dPtO-n*V;qjXV>`&o_
zne0l--Fg=C2cvHfd+0*Ccr3~igC-hTSgNeNQGKx3QD~RF=CiM8)MuA~DP(8%xxlFG
zcwfi{0iUau^5;wSN@{!Zy7S(Iqn|BW{P|vrTCO&oxVOwf(qZ5=zVKi@#<%fy
zPc0F%jkE)6W4WMXt%b)wx|bmXts&odMy*|u`7T!JiQda39bpN;E|f;Gb0~|^>bmm{y8mlJ^#r@L0XbR-IpJJ
zs-ooAc#Y=X{~(GP5)<%e%qdX5`=64K$Oo1!Hw^%N$S;hV{Dj&$F8Y;bx*Q8)6Vu&lbY19q!B2jdFL-o
z5rjk#BwN!hy$1bKL0uqlBqe_UeQk(KfxNMgk#WKi+w|_0ZImPaI=!ho!$&wpVzv5}
zV%Z2u`U1PMb_+vJ>X8Y{C^%*}uFVPqoYUc~d8&Ax7BMJfag{5vKt|h=CZT?=sjY>M
zSb44dGpG1eWkN_%zV_F+-J`Ct@o9kE^8JXR<^{uMoXz(Yw2D_n;xPWZ#Ku1d@3*Tj
zw<@Qj0l*VuIYm+02RsvxFD?1s7tQ5U>ZIJmvkkkl^Ro??Ct`Mke-Kt3+rX^8J_}Wu
zqHk{ppqxh|w(9VKJ9Fxc)C;UkLo(LvS{A7%739edQ|rsE8J~19RSwWJWBAb9GwGCy){Hn(E`I4xq^_pRAcZ
z#rFNW@RE1%L8vfcD|dE?uLb@R7#u*!J4Gcb3Qzzsfj;8kgptxH>nRahc@#5EqD|&<
zu%KgfMWTam5{Ps4hA%KPDPrMPikmr7I%szEk*uiHeug9TouM#I3}D%nDj+Q>N=qiR
zIMnxBSeI-^(_AOlZL$gnDVH^HopwrRg)Nx^m>rsl$k}DRzjE0{>>GWBSF#j|3a0k0
zWR@i#;?qw^aPm$0+F3@Uog2)>8ME6@XhjlKG*B&k;|WCS@|>fx7E}Z^ZGeC08VwPR
zQwSFEokwUfLsL9r73w^Cfv$H7ji_n12;7C
zeO}}&C+rYD1K0x>OvwR#=%ji0FuMf2Z$z+{dL4naUp|zfq*Pd4CUD`&?4&VvbSeTX
zV2|8lJ@}aIzF~z=^f|iV=-_D-CvQ?fJ;6W=^%gEMB5l*-n`Ow#YT{lDjLSdvxDP1R
zSjZOMzoB9ndJrA9tSYhJ9HAI4VDiv$`!3|sjYE_#Oo3>mPx_E{bdVp(SivX8k%JXG
zYhki3Vs8#6vi55dzO&%)79NXBxa9tvhLZlCidli3bGOZx9mp
zZ*DC8I+=GS-Mx6hH$?Mpf<50ExA(k&ls%z6UAPJRxn8h{+eA+1Y9jkYUc8Ij7{s&B
zUKG}yf<3_*H|aVzM6xc-JK$;J$?d#Jjey}t?wd{kXD8mu4JoL>&9cEg{pRS)$4t&M
zjD8n}@F4p(vC-#}L|!gIpII*K=nzm)P>?W?&@i7R6!d>dE)dW#7?`l=Smf;3WE7Na
z9O7`CI8Xj~u03|1cLo$3SU3(3EIxH)rTMBV0oU{aVsBeIE==4Vp367~vW
zRc0x&*RbegC=r5AOLvz^oJl?tWO@(#EtgFD|E;F*!8+=vBpBgd
z7oGtgOxRGobtD#kC%`E&E9Lvn!wD$&W{<;59oGX?ONCY{)~;~JLW07mB+=}C_Aq_D
z@qQ9A8I~AKWF`o_A&+daL-pE*1IF#VcrJnz@9;FDE4$;zY?L`w`?Y`CcT_nkaVsX&
z)^G@8JRyQf8w7jsw4X
zPkDuhnOnxfyExwXq6d9QHiMO`j2Rk*Ev+?w_WD2g@zVefm9V*=G{wp|jMF^tcEIPS
z#^3rq{6*~PSDN@;bYps(gYYlg%i6W|KvV8+JD)F`Vogfy(_5qSaIS!|a9mNd}n54vEj>g~X
z(^lfXEck-Ko(NrevToCIe8ep8%b5#|uX?^;^z*Zq|Hf3MFE{;0dza>J_=uxV^ffB!
zem}Q96qbO9@xw1dxcLVm>it5T;~_}?XSgP}&PC1R(DtY8m>wvps)Vun@P&P5pkp*3
z_PDC^zzj64bC_?x%6LF`=*eS=7M>iRl8ybyQgEwSL2wr!CeNzph7%fLDHo(#Mm9Ck
z&#LT)s-v+651Vj@o$AS8vPsosPPj
znM(9qf2Y1639Z^UZBjDhm=$6#Q#7kTMw_9)?IWuh^QLiEd#oso^8PVSunwVkf$91v
zaC01?Ya`yrT!052yd8t#$F;3`J<{Hl8n>bw>U*>{X5YS#wf+2r))d*8w8jS@IuN4q
zM)`$5WXHEK>Eu;;wC;V(sAlz9qelL21V!SPOu2fI}f
z9!;q|QD{DBrUvoy!}dZGfcZKCBMrBdSOhGH%ve!m97}~Q7aiR)pB?*+lvtZJwJGb7
zuzs}6rlLpYkwGv*s8KPT@ACmJ3lU`(^SQS?Hbqo
zkw8gX)w{cI7Bx}2*Aux4C)SNHa`L=0P%=3yjd`uFf)sPQoCVeWX&Wox`*&f4-7OPk
zb@0NKguqI54G8jkXZ(oZG`0{|FsF?qQz-($NWO);aIMV`k(0M8v1Jh1Y<3Gz;Hu+S
z$3boplN6*ZYhh&$q;vZ26;^|y<<*B1hv^CXF
zG5=Nv*8~wLr#lVi^}F&pG;=Oe_dwzEK$8h-iP11&5~Hq^iVg_@io
zHy2dMj_>g&`_EuyujeZ$04I+6u~w6?2dTzNiG+~)EneS2-z{ufr@drW8J!8st){m<
zpgW;24xN{3w37r=gTO$a^rC`VV=7VbBPO2k!avs
zM`d3ZF+5w>W>eUFKq5Vu&}(0FZh#xE=Zm08Fg)0vmw@E@t{{xWy#^_+VVWq2Nl?h^
zetRQ_&ADAOU*O6yX9PZGQF3>E{Zir|gcZx<3jmn>eS!(Y=qnBGBVz7H-oQdroMP7`
z*QzN=i#_NxE2Z7(3o{|HBHxkp@2WZ@90GejPWCAioUgZRZy?&JVO{j-tnCI)SfOn0
z;Hu5bjI!Cw%9OuVsX*<5r1-GEy0(lQIaf2WuaYc6=Z>F`FF}_Vc6Rpd75G*4D{CzW
zm={qXYdqN^?yE1`eXIP0k&r>hP6XZ;B>Qi1Cg#w>^FY<6@ud1O!$8KrXbP6ysd|;U
z1$$~O&cCO)*_&qPl@!#*IMYxsr-d53azl^MWWlHWUVDynssW?5wlxHde9xV)#z#Le
zF16r%AS_#@n~`a=q9y)vtS!uR;tWq)VsG}2uIN5Nc2yN|(PC%VD2>}D?Ung_7`~qW
zc!}M7Lt@pMX@29Jn_V1$z$
zRf;SE_nzJ?yZbbArT~SmEh|#>>hl!=J7YrJ$L6>o$kBBYfOe}EnwR4Is}~BwIwk
zWko@k6Y8JQFaPHau3?al3)*6?MI$=$)#Igj>=9+#b_}TlPUzK}9=wQ^oi8`#mJ6!j
zpBXmI|CZtRTf4@=utwQ1`qA_V9FN#F1=O9i)}B4u!>TysQ3nxu))2yz68E$y5Lm2c
zBCRws>3gB(qXW&Zkyjyqv74-NvBQHpZ{DYVLiuy?y
z0Z<8BZf@?C;t|=QDBhhEVLW63h9Cb!#cy>Unvv}jh`HGh;Ta)ddce1~3Ug#vCd+m^
z5hXv7LzczF;JP=QJiU_m)2
zFggG$$k4xE<<0XFch!IvC-n;@SFNBBPNp
z@9WFV^GAE*_#KMPv%yJA?&{4xi63M6G_60DETyJGnX;`U(kI+zEaBsI*5a+`^#i7>
z71J41t2$S#c;t^Y=VjA&qbCnZ>12Mm^0o({Z2h77lU6Wx`*^5l=kKTW8jKYMl8FL)
zJ|BT%GaiTr{8)-y`A3`KwvWhcYEk++%urFsMV(Gg-hA_m@jL5zn|PXH=E>p%zXz5T
zFVM1!;z7g|yM`XUyXKdVwbSfxi&XGWxrMrhx#H@4Lo??yNIHaO9do=Oz}cGg!GIoi
z_v@0^7ziAF=#$*81a`L_8qj!%l+%a21}o>Ba^A5ME8ur!_=CU8x+0c()KeKE!1sv&tqRP5y`q?A{!RgU~IknMV
zgDWdr`WiM}=iJtVW?VP969Fo%R-$+H4iPgr9bws0v%zBwottnYJ-5aB5F=)_Sb$9P=JLy6VZRzivg=TJ@y
z@TOf;P7V@M>$${XY^a0Sh+{Y)b1f9-Aq)K@G6`}l=>%>>F&49?T<~AT{9wWyPT3Yl
z{1<#Tg8K8!mnK@x5q`$#B~3y`%3;wXAs|bB*3TLjwl%0Nf;EoQY&72XPx%=CL68If
zK_ryBx37RUJ;*8;$+?E?T(=>rN@(K(5H{PG)Z&E)&7+T+F#)b3*3U6kvTq}m=QU39
zFpS*7W9Ix>Hw!w;6H>p~KOc~XVao|kn<+yW_ZKm{(tqBq4NGTm)~?450kJ?|tS0kM
zHi0$@Qs!T`s|d|A#_^2$P8$yl$!OzbRLe~$iN|8MRH8rc%5XVsH(y-bVtKLd{UaE8
zIhjwb>#y&Bxo%BsU<8Z{S$e(R5Buax@;hn!pj-?#U;IvpOg-r2t(!Ko;@RNaP;lRt
zBqfL`b9{Yj?K^SI5r^6MP5T#Cl$i6QofpR=tyYJFQ)+i2
zURV+$1m-6VKL_TmVLHqtsb(rY)!6=oHjHRd;0=$jGr((ey{
zk{90bNTrwr<{C0xq*1C?p7;f3sCS(f9PIi
z;h;J|8q-Y!)c~}tf|{vD{z1sU=I`f=(b7+EWor}jj>Ufn^oAq6m&x=dkK
z4RQPB^IKZ`f?&0%JksH
zYq0cN7&$UDar47QtYj7ZdB9&5-AUT3`NaZm6o(;xM2PY1n$7C|ztJ2fRnN9CrrMH_pgZagJVlU`^`pWh+|ARnYe>I^lMj(4rJlh~mx-yN&
z8|N6y`0HsE#78Q$DLTsv_&CuBi#69
zS5@v-uukG#rft&%nesB0+K!bpfo9aNQQ+J_2S?=PuM273X@P~Q7$WZTEk3i>Njmq1{UNraOV|KpEIv|>Np+}w~E
z9z@IUgtVfjGxzu>`8H15r~LGxVe4UHFo%4@2v;?imA+;sM1o3lZZ1kO?PR!`aB6N?z_bqB9LvD
z!d`K?v%w3|*OfmtZ*~J~RToZ9zPJ5PSp`@wty|(2$B<#O^V!y;=G$OKw5m>rhzWh5
zKFFH1@o6KvS
zHi7zV=KlDjK}W&EFUQO>)Ui&Zh1gKK{V=CaA);Kt2~6PWBrgZ=msoW*y3s(of@f`?
z|D&{Vn0+nPUy><)^T#oo`S0C>$0w&RCyY4U)`7fMg5nuNxJ+EXhZa`;FpVi?w5j;K
zw`<3i*$r5zDVgOAXEb&mB5r6c9Cyt`5eAu>P^^cg79B}4y#Nkj{WHHCT3HUhK?={i
z!eERIz1j$px~95~X(GCHteX^UDI4}%%JI$7Kt6eIVnYRT)U4)IkJ}sA;_M32Fj#sp
zvYL&|c$5oM}^QXKe>#mJLsD27wph0w2ydx6EfI(l=u
zWPMBxn8We`uJvX|C^4~$egvKN=6E={I@Nk`x}CWZL039-)uWAAS2J;yqDfj=UNlkL
z3-P#iye{tcuJypA8xQ^=xkSdgS_`TA_72tSqm@gN`l2+q`e46*5I8kpPw)>qs$ZJP
z#(J;vnNU}l&ljKnj@Mqb-<%%}XTAIRtymA8Su39K0Ah&;W?>~K6j1o&808#y^~1zP
z!e?qr)v7eY*5uuV#|gBL_e>=F*B{R90xu2q-D$enhZ(0%!u5B7mc~O){>nnTVbxiy
zoN2BiwRF6=LPhKtSMF#1I=(d6{~+S()X!?a4K$bq_!!~|3Nkj2HRS+pX4}rSHQ3yH
zIvb12>LQ4rqug
zKp6&J5(()3t?#6&>S^29+so%^3T{oVkt*~4dfE}{*ZrDvvY-G5Tkf^LL5_7*Efr?
z%P?3KBPYHr+I8PHuBeN<_)xuW@9jxC@W>12-pw?VA>ch+aj<)r@?q=&H_-O9>>t_(q=
zbr9J)?@{X)#ME@{aa+D}9bYRR1z65TiG2;Tb&0(&*c%!bT26#>D1t8v%S!Ij9cxrtYnODmS<{~!`sV=Et?2V3ANrbsdO
zDoZ$t9q_dX66ehV_wKG*8Ns$_PR>*$`Y>bt9m2f0wY70I(s+LqIlDWIhbEhE?`ric
z6*sYH3NM#;_qd;r)TU!MV|)st`X@=+;OEN2Lhd<43M518$o~Jj5hwPp~TkJ
zgoJeB4Mzm>Xu5-oLS9`_C}dfX;|qWlVnaXPCf&TPsa76N`n{y#C-!J`t2ATC=9;+5%H<%^VoV$nzZ2lFd);J!O$0!
zEn9*WmSktApPucYnafWij$(Pjj?$*
z)dL?fSzS*fx5N-9Hm7Q#_4ShP=jK_%;=2Lru$;CggF?}#YkgEG+g1W+lcqBp|2<7H
zuFEn!`o-pH)LQq5qrs8?LTBLcr;AS4dg%@zXQl}t@7mb9H7;1J4@f+Qfu(KrM*WmuSi$0+}u|_
z-B*uCD88Zh2X&v{uFvy6J2zo5tCw!`T{D0qhrVgPABzGbF2*VFPGbqS2putcliO>R2=
zIKRpW$hN0K!x;7T8$4J>e~)D%V_jd9sb^&`O2W(QIak8W#|E#Yr=?{Jt|!&4GG9Y`
zJBPB?%?>5vA4Xv|jf=_hvPBpq%5F8;h1FK|d!skJq^l;oG5EC%5@uxqHrW=BK=x6&
z6C-eZ$iYP>q<
zQZJ^PUYjtwm~b*do_N;BvRd-PGMn-^^t=m^j#n@-^48F=DOze*z)OC5$NoMG?vohm
zJ(?dU`#ab(l-wUHu0#V7IXhgHo9*Ikb73=cefJN*)T>%E$i=K}#%Wc2r!loYw>K_2
zuMmD+m)lajD(W@)Fj)vV__to&FB-f#nkmU!FmRS!n6(Vd?(0uA1u_mf}%t$mkfhN9a#XxHSy+7bmwpN_vfz#Of88*=R%L)+M>In7d=tD^9{G
zgZQ*>g1U%CnYA~#%N0abg=1(y`;~~0x4vGP%<>Yf+lWU89d*^q_u7PS$pmQ@JU0gP0Rvc~)sOs9TAXArTD`&7GE{F74|ytroTW
z7|hGXG-O)WT1?+!KI~OqTi;{Z0iObV%Od)PhLc7e?OG%<2jMNg$aTGEedAZpU2qk_jGS0?y
z*W;Oy{>weqD}g5lp4JBRymXd}jbF~g{#Ms*_C%p
z>Gj~%_nH1mzoQH@0FNTmMn!jYl!vD3HVUkCP-(|vZmD(i6#J>UAa2ZiEVJC@=*rAv
z?!rVw+;SFgWoGB}PhfoyaYzyKYe;jLnsPaVWIrd={+u(U@o%i+wx+=r+Wh(MZVAJ3lp@U0`iDY(5Zpu-
zshJyKk(eS9T5{BJ%2KchwH{n2xnmI#JF%OG*{{@XlD}Y|#G?k5S)+vh8vLThveM$~
zdDYp6AP|Z9ebu3{x-U%_mHyG6>V)r$Vw-N3*?cLe^lVj*eAL?D4San`Ws^zFTPTev3ucHFtY6@m}I5*}S&jMjf3
zX5$z?k(>*;$P172T{pyme&RT`NT`RfSb4ZbMUZl;7XEzqFp+KN#I^Jz|=>lQ%cmpL26?n8pgr
zMy3xLL{=qS$0O@V#N>u_H6Jf$^4~1I!zT#)S_iEn8Wg5x-(ysY()1I
z31uOp)y6vUn}E>KvDKrgL;nEv+}sv!8&zRDpWpURV=MO6Kk24HN&43gHKe(~YY!%i
zDbH4wWlyNxl;nEmjSY3Qd|8q=b(Q(wAO5|V{{T)`KKk+K`nTrStH(l)M;dwVD)x8v
zu2B)}MqwX8Uip=KF4L
zZ|taTzsIrl_83ocdlLzkWY&_*$K8K{`pVvA<&F(ym&);iLg6LZ0yTxARk-C=Y`Fj=
zvH2%taZ!VS)jJ=e8&(rqmaRSULIa6gRjgi$x
z@iVzUQOffvim`6|SeIp8LkmS+#Sr=q<)BH%V|s6a&s9{uOLK7ic5rR@eBE
z>QTk{*I?Ypvd!*pZ|&b>H#avo_6dkK{Wx&PYD~cl$9LR-P&GOA@?A=~70
zyM95fkEyLv>*d$`N}ke`rBmwooi(v*N~M-hn7!Y%9E}>>$vbM!pB$=NZv%fJkX6kW
z<-Lt(nU)*a&U4KYzrwWREFM8>uJs*%E*?3309%ilU2T#n$fs_DlpABgfC6!c$yI~YW|e?pqtMiQeQ1~lp@hQGPrqzd
zjRw}W8^$p(`ilzgOnmD6RnY8O$>nv8bF>qEw>S14$Myv2zwNE3I2s0=p=e$;qAHdH
zDQlT|yn~jqGOO5Ia>w^&g&+@Us~u83%7N@_O5+_hfg>?u1k?lHXZ|w?fr*Z)mF4J_
z9GcvJrd06>kU2r_awNaxTjX(zU0Sd5eM<+YACvra{9mle%{s}V-WIsx*k_Hq(^#OQ
zuIG+ZMM$-dF;OF9qsGRrEeIg1V?wpI*|#99+}zwI7^BxAKgVpY>^)1WRvkyCErIJ`
z;WalmxT^q*XBlH{l_r3UVKd~F_7+lvvsRTTU7rb#!GNoqEqLeQnPdaWIVNtl!#hq@J6Ok~30G|<_aV9EFMrcg=ILNW-Ce_Ar6^M)N
zykf9_0Zjtd$hXFi$DS5arB`(adoma>s9DR#49&({THIS}HderX-lmMy6CL!`-RT~A
zCyl~@IX+6)FUp^*kocWF5v;Pc8Mn)>mYzzg$Q$FAHA?)hmE+^3okQKW7hylX``3F<
zZlB46AgS^XIgO3x4`{^4#_DzH$TsR%###QWp1oGf2$cv)+g$5BuE73L*2M?68T?^*
zeZ{D1ik+U+6)Ln-6zqG7S_U%JZXUsL#PS;7ew~QfnHbcq70p8$G1O1aTT-(19+}YdzRyH>{=I$-IZk(q+9Aykz-jk7L!__0D1nF|$8Sap1ZP&4dX%5lQL*D$;#p#7jcip>U#F@JHG>S#
z9V;w2;5X$fZUyWu0z!sxGl>K&3oFSUB*+0@EAn=HsZKR@#<&4yR)yJ9^R%S^0?L7}
zW{Fc6eMY-unVQ@JYHPCyJ|>LeJS{wNfD_z6z^D@`!qzJ-844VGlAvznMXL>>7Tjs5
zyN~f!O_jlxh8w9f*8R&zu%kkdV&ekePfA3zR>sD8^o+&F1*;`#7F=#MqLJYfi3Zc4
zhf8Dfj;Csyo9)r4^JgcaH!`y8$4t(>V;}h}qTMeovazW;9b2h=BGJ@7ro6ySwSOCq
z+QKeBLmX@s0yoAwmb1mPRXFmaijwTA=OQ7q
zaIt~eN@W>AYi9}&-pbL&p?_-Kg#Q2;*PM2=@lK_X_ubJG5LKU>Mquf<#}^XYg<`yF
zI)$|l9%bLjp@h$vsVE^3{Oi<5DJ#Y{uO(HrefGNE$~hjBh%vFdh-X{MTxEQl7}3|ID=fX9r)Ox_J=pJwO?TY+iC*9lE=tnNc^{ct0^d%8aa9I3y9Nm;D%IG9
zoce@rYJ|pN?Pk3y13Y6^)9j@9?_bOy5NS!ota!6gV?*+wn5noaUB8+YuoY)BfzwcD
z9e4i#8f|5RIYuCi{*h;rj+1FUKc>@$EaN(b0jo}QH3RkU5!x}_192*WXT;X!6{=R`
zGk^;oE
z=#4XiOKr~s1#r|S8RHX#B71<%`gK=^y~4UYwF{FqH#x+tv`lYFNE5~caG+UjQ_Ec1
zH)8SV{b@K;n6QtH8`Hc`IOS`-%PcymbISDA|FL{025H4@6oOmW~)uceN4>drFt%H
zciq=~je~>MUX{D4jEMMuQB8Dsjm|g|xm8VJ@dWSlD)i0dy2G_0j{J^bi8nHzmA#D)
z*todwk$y68su`J|jf~Y=uDxV2r`Zmiu=+kH)LQfkUmn&qWL;Ee^`1Vl7D-}I6EM#Nz^_7xm
zR>gJ16&XiWbW>J4+f6}3KNZNx{M?o*o0V->Q;gICRwG;x9qd40Fh_1EetUn6fY`#M
zS@GMgbrFYM7`YkE{yNOAv`(jwKpDXe7|hiHWx5tWkeRr!kIS<|1pd8(%v-wB);CN2
zO1hpeg5t1=sjqdRbC?yDoZrI_9r}fX#f@G^Qyc#Pm?Bfu>b764WDhE1CY(ERZ_=%d
z_=9^VzbLw9b6h9cO?LMmjtk8bkQxT>o{2rjcQ~IYR6sXXVdjKzF!+yeALlO*B;P<
z{GMHxqT(8iG1HF`*-&5CW
z@vO;fIzg$|8oFUM2&0_UqIoP*mj{pTG||`)jl?BmzUR0}if1S-ocA6Q)AF2Wh{at8
z8YucI+Z?AjZ=P9i78YaH(LC7YGyNx1Z9aKQpSe{ib8L8$dki+Lg^p;adfksXehUtjRw`iH4b3tV3O?U+Y_giH56oUN$(ttCyB#
zSkbIIEjOsPu!&%3phA<3Z3xG3dW6l*L+~{WOnJ$5qQBns7|rGcC{jk)|`
zZT4jQ2W(r#H+kPFtx=C(Ex`cnPNKKw&Rm7uT@!1TIXe&BdOZ&_jO7`oP7OITAEsl_
z*)O4M9~Jw#loIEcDs{gddV}q=;;zO%yO?7S6Z(g+8-lR~U``?rQ*&{QIRL|WS!MqKi(ASw
z!b$AY@+0ODHvltdVzy(4ZIoEn7a|VDyMIY8{0m2!%lci9WMYS>XA9Gm#K6zE_qSA!#Q*(23asCkdw(?)Ges?)W(Wi3n)10HfuAIq-)AF(z+7^!Up3dSO
zM`&7O;9T|dA-~3znvW2c<0FjTmp>$J#_S#gxYLv*_Vqb#C)i<|g3|??oafJGY4Z_F
zPp90of{jH&#cNzu@P2kNX{aK>_8CdjXL`tU>6>t`o2UJs)o#@`Cs7VOpvJajHz-RA
z==t)26ZJ3Gt@=U(n4zJg3N1w<@|To>k_a
zGXet^uqBFoZahgwPX>J_WTVq@Q#Ca>okeXoP_xTs;cv~YC3mt+V;I2!pVJ-Ai-G5|
zs+AS`ek<6Ot!oT*-0F4Rk7`@8-GC5AFh-i1F=b|Ma-63qT63Ir-{KRwsZCtlNm#bV
zXaqzoRILCsU*yBtfJK66>)&csF0RVWEs<|Iy-qCC9QfNIcwN7_BLoK4a
z6R7lISAD=?>`di3!tp-V?0v%vPnZO*u2aYs3CCx~wGEgA1Gf8UPC{~m#-2k1EY!Nz
zvQ|R9$`s--66ye4zi00i?4Wzo12Z;_Sbfc^W!O2ZKbi&
zkEo)IYb%~3mTja~s#2*!mZ`T`OA{umssYSe*{1|pW-%%ve0B~GW6Bij-W7i_?+4oXiqsu>0RgjJmuwQA6=8st$Eqg6NO9LtQPv*
zY8xWkdM#8@jQ%{9KdaR;{+f$#YZ;i?i}!&5`;29uGPEp3e#Vj=u|?!oq@`}VpV}gP
z!#DSaI^_Van}xOejwv*B~RP90W+f0T2poHacc=oVfuZy
z_BYSi4}Hze!5grM2`^h9J*%>YqaYjaxK<85o;SOVsb8A5Dr_J_+f$4`vrVR^ve5O~
zio<+Mt!5Fazj$w{RncvN*?vxgt+@%%GrS|M@OUPN9#*A=aXb;(s
z%aw>6E)KB9KaKCRlYOe3rxZxu$?v|vO?}(gGQVY-6WkfT#LdI*R^<;W?2O!7lQg!j
za_nYUX{vo1mHKc9$E$^XJy=tYaP7*Wxzv?cb)Z|3kixpA=H}+$hv)6zW8ZUq!;u>2
zVBN9r@z-Sm@h0XAZo4r#Eddensq)!kCqEuMp%%u-1uU&d4tX=t#~1J3r#SxMQXwiq
zNk}nSk}dG$N`c6>gsmvetLg0e+dOGnEWof*nYq5QZt_{!8M(gLLe;Yy4~9M2FXgu~
z`zulR0!L%q_aAzg%LdkQ3<^zlA8Ig{C%MXU6Py;XhMMhPr6AWz-nQcSrFJ+AXXlZV
zWtK%w6wUV&``q6CoN|VfO#R*DQT(A9Ft#fOO
z*Tml}#lK{p+k^u&L_&lEs7&%oGCXX(Pu5OQwsR0z1Z<5W@!Wl<;PyR-W7>~&G_&B$Z}vCZ%`XAmW=L&Q!RCK>vOd^Zi_ywPhH#Y60+iYFU9VG9?FWtWRL7n;O-R7Ke?ncUClPX7Rj3D-=T7xY=R
zpT!&gkfzs@)fT7Oxn{=;?DW|e*L+?q?=J}OcZf&j@|0}aS0$Q7WfaF^v|iR~c8xi7
zYIWHz{{WIM#WuPl{SKFG?ng*+5;l!tIuwn@=2|TlgxYwDa&(9I@8OculzgmRV)13>
z3g59ex4^Xq?
zHyg`C6KlZ-x+$NHPR)mCp4og32=rw94m)u|Lsg9~E3xBMb(#_=Rq`~7%4hmEvtwtZ
z&5YYwvJ$jvDtC=F*(XG$e0*Z8*v+g(#*VP@y1WYG+99hk&E+%2b#6CPAW1X9C8<3(XEjW{{T
zO1fD5hU*=R6l!UI+5}yB;eCyaUP%0!pRoyWs>aj7V`9X-n)}HQ#Ke0>b-kD{gNW065yTm%?#>kSo
z=$4+(IqNCqVv6Tw^N@sTc^uKbzXg4iQ5jJ^ILL5u#E$|ZMPEdeQ~o02;>0J#DW#yIB#@6#=q2;L`Eofnu&E6Cer8on@(B6yCT;VK@CWJ~ed
z@M%wHLi@zyxN+L+G10YTG@}xzo|>QgAswo*s_tBx#9_OnM@Ef^qJ_mu)p4Y4BMWqm
zk%H{9u)G~HjS4)qA9(Ot7&&&0Ju2>5N5L&L-SC`uBTu>ZOHMKpu7bItbAiY>pk;&YVV8alWqGND*N|XyB!HE*YWn2Wh46^czDuv
zn;fs)vQSbQtUNId5RTPcYl0bN;pzOLLATkd1q`p#+-}Ol7E)w=*739CiL2+05khyt
zB8ITBDIRfW`#g9xBe`~ZdwY0-eS3DDj9&PfVx>}8>MH3M11P*zlhYz1E7$l@3_D6D-=u!ME8s>?}vBdZ(!(*q_;Dk^3xoqQvO%
z@wYZ_L)14L*#7gm;g<1u-Nwt}#5OiwB{94uh(~c4DCNa6T0cjpG&a|g&pa$r#bpW>
zZLSj3_(PPWDv7w>S$KKA8?k&Z*!d)Q9JpEJ(~i~0F*5HmW3DR^)A?eGTx!YbeI7lp
zP;(m@zv$jAPqJFS1LL9Ri#0VipSq*JbdW)O?ehkgXy6Y-QdUmpwUlFK1Cx%@xJ5%bjtG6R!eynMFxc92`9HzhOGz
zw4$?FMx39|6@}olYJCkgm%_&Hm6es4vc$yiEYz*v#IItw(a^DF@{2bLSIHyfeUgn~
zi;I5B;T9${a~z~Ob=aL^v0m?A1q+&Gj!aT4-A9vfvnbLRd|qt7X-M2|EOP$sKP
zD5HXImLRm3o(uXtHDq|>6XBLoq^40=Omlg%iYp5qpyXYjlFCQpiyp|tQA5O)^RXha
zW}@?WvTn~DZP{3SWpH24e4P>H;T++kjk-=zW6hLGv1d3|HJhWck&fo9Q8etVC7MMI
zBz8qt8Eb-sKJbx5qhuDDbB$C}eUVt9qoT@8PK{V=4-Qqx{Qm%GQ%@FIdA!yZ=IBz{
zWtXzgBjE8z=XDKP^PZ37zoIb8EYxs3Yb?}QhE7d0&uWc-XW-8li%Pc2LzH6a=7fsE
zf0b;llFb!_v0FU6w2-4piYTG1EJ;wYHa>VG4I(JgQ1FpO2_u2x10N^$dTK;`5s5_-
zikD>kUYp=vNv_4^Um|zWLPzcLEWNYn>uqN9{{VWisMKD|8nG%U^7cYBuPCy_p5}*+
zmMM{LSghG4sZ%`{7IoRBd=d37kdsT2PCi}}Aw6C8KfTRbBg(~=a6PtNj71F#be@z(
z4ZC?_8WaT@Lt7ph5^ol1(c`7H3`)sDEPe&>cwwZHwMV4j>~Ejsg!ORyD|AP?@kJFw
zBZrF}WgoeVELJSbGUF665_abZH+2?0hjdxvbp@4oP+?*?!gG<(G;p6lgRO;~p|YWLG$qE7bDv3Vn^x3?JfYQ9nx;qr7|7T&ur2v~SzkrKxtY-tJz
zLsmZMjBByFKfI#PBsCs4@HqVjfkm&AQQ4(4N+#puS0;MWoR*%8;8V6plx>bO
zgCWqZbZQeXeeJX7+p%ScL(Jx&xh3j(Iea70e`lpN(-)F)?2OajR8Ylb{T^RFmr+*5
z;+K)?@)1?h>DOl$hKWjYvg3RUZInHW4;AgFxoo=SkC&;I*bIx6mt3;BrG*f#fdr=7nL#j
zIJADAZg+UHZ}V7jvN7Y>OBI#bTWbybGxMnQB{}mW)KUCJe$8ywQh2Ti?f#ixXz=+x
zHh+<;t_JLs<6V)HoVhBF&aqgbAuP9E)SU@$f2of{l)Rh@F@aK<>7|A_4VE@1wPKi-
z{Y3u&Nf*%k5{f-JeG%}r%@JNk>2!J$bEz&5N;hj|ixhNrGO2J?_3b`h>3Jz|O5}Pe
zJ7TxO6cl-ozl`=fFPh55?7h@BPtPmh^(FR3(xe
z%PzlY;=kzSmsmu^xv~B2ye*bFn#bpgJxe79*%eD;!Wn0Hq3~ZAZi~(PkINr{%Fc*+
zH9kbX&X4SJa^eJGr?I@Dlrfekv25cEyehKy&dU|iMPV#Uy^UtX_jzR{`%Cgmfpy7$
zLWTAv718~j;`Xdw%ECsRC&8&2udT?vh+-cGkw;^Ae#PR>PH@oEV)6Zz^d$cP=ytj&
zy}e?~y_b$QS*W6gheJXue2Uwoy7nF~7q*JTq$K2UjtVOg%GspoVwTD}H4T&`JgIvX
zip_0!SVeACTPpi5`aLs7J_NQ|Wr>pVMklMQ;OFQ|HHtKpP}&;2Q4iFnxfhl{W1rU5
zweLb7ShDeWMTy@7R6peCNTP?f@j{Q;%=+kFB(}KWExZ?>@u-i4QAky>ZGPPpQCX~f
zRvHr7L&M5P?|;VX#bMzUiYTmB8nI%9vhe+h7nhpK_MXoguX2hW7TVtNW08B=MHEq9
z`&(AWhAppU;{BJFzDpOqU+Ax4YwHcu3GC%3doCkD40Idd)o2dCVR2knZ@KImN
zB^Mm0hgU=gB929el5(PepjlxBta4r%UMmH8<)Hex6a%g@8CyNXDWdpTU(&QK_>i|f
zZ4N5LB6O8*3#;W(k!y;iD`a{fl%G0XWLt%9s-{%>F#44rr#}>PJk+_Np6gDjJn|*%
z+gF&=$}d<-9C2+d3+Ry@5^^CY2*_QHaPYipq1cx0hcl
zna{&vweo8RGJ_fAf2hLc>K#*YFzj-#k!5ZhaamQX(w$1>Cc`VLaKn?C^qV96xgyfE
z8x%)1Ci0~<%l$?V-3J4d#d3!Qvdcg;P!yj^(#n;%{g+1dR+HDx(}>VQ(80A^ILfMv
za|+l<_f{$#EAwT@l53H~a1-V`1o|IVt167I!(~vZ!Bu>(#u_f60Ts^b>NCE6BRxnS
zVKk%sQALxMtSfUII$EfJULFdD)U8;RDx6hR@{U)^oE8efS_;#Apd7D0>w7;7Y2tpL
z+Z3Mub67KIH{%&q7T}FXs`!lrc0(5m?c5U{IggOP9ntKkrdL+^7=<*GO;sB(Zx-T=zBM-LD7hpFHA2muWi9&J;yWX;e5*~^FlMhMm2
zddT))xZ~KSKd6;3Q2jOsxhdGVcd!fa>c_xRk%4WUPX9$=VfH+3btE#Xs)E?La)WF
zTx8e%1qTaeN-9m)Wwd8yWo+%m$$cxAn@$~gE{F6y*8C$KMOAX9)ZJ%Gf{zqWMYd0{
z%TT%v#@yAf4hUDLLbo;*50ch`os%?@45pl_i!r^E*1%{>a47-$7O82YK2l2RSK?NrF7+}^WnO_SI(-OL>n6;fsQtt
z72)Dn2-v9^IOW+uZ<}RhWo2(RPZXlYF{k{YOe;Z5X$8$Dp=`rwUXZmto4YN#g7|I@
zRR%e%pqxa72*n1byn<9?n7A^z5bhV4jV{-Tj)^+22F
ziQRvv$MpR!^0(z>ZVGV1jvMe@T~w+n@YPjm#BQdowP!`?MtS*nS65fg%G`AsZSxKe
z4Y`$7CXDK|I|-yMSXf{lyithRSJi@ltfdlFGNCG}@UJKisA?(LN1c&tvhevicHku+
z)L{BeALA%lzdIn=AGsvN1Eg0h8me5;j575w@Y3LI6`
zuE7r9CsL+W{Q9Hem>pE0(dp)cS|D&vez2q!RsI~iWm>8_J=axSsHhLRZBeb$qTzBF
zfkUno${1fz3cMxF*zU@>i$)mE1%eqtSMu-Y77s|eWGFzNQ|rVNS%>O*gmDfl3x&cAf1zq=X$XRz4Ia@x`Ow0w
z6=DjmL)}$sX}a+giedy>;RR}EB_q}p9H_TtMJF!jG7}JcZWCTP@xThEy%Y@V!+fe$
zK1%0;)>C^XgwfkF%qqQj#*F^}Dk&le
zFmf6lrQpj-#MdM~sQBHEZ`Sw=e3<}(L&b&18B(J0Hng*J4
zMoV5#CGlrZWu=xqe5gi=zUXD$%UsD#!+bXj`7JH5sU=esCgI^rk5mHmH}I*Gz49OQ
z1&RLvSj!Y*G!?Pr4(Zr-jP{j0)U9l>{*+>VR5@)c;wfoYS6iI61-R!ZCwt`H^ve81
zMOAF5tHFK?D|uAova+#Ki`S`Ks><8)Hp;8F0BdegfxE7CD!>Fm;pz?Z#h#i4PxP6Dc_{6fVa<(4P5MZ=xc@KC67r=YD5`xT_;
z>MPW*#1pavA;)TYHC9$szIRv3{!NOq^o9FN_S7}7N&>l5t1D$*D=RB!T)^tQDiAs4
z0>gw5gNAkEg0(QyqV`7t_B;Uq@Ekj-@Yco&P4~_@oa@nJ`{e%sWh!0IjyYxiV}{v!
zN~p?sS?$ApM=cOM$|&H!mqC@)(Ek7&>+(hVt;q|h7bHRaH?_$qr=*P-y;~dPT|s
zIPt?pD^n82p)
z;eMhRN-b#B?59Ky>M&&@WivnE?Yvv7)if24PbTw&1^%C~e9YE5BMC26wpas6C^
zH9C4Kl-6?QgBTzTor!?$-3o37Ea9H$IoQ~N#@x^ishTO%tEgI-(WPx9DF&zj2DPj=
zYN1k9I2;xrJkzU1r#VEU$1eDIT~9R9+0bo1ZqQtNDa9iUce;Tmx`_c?l_5vdlnjHc
zw>XoOH3N+AvWzjLa7DtA!o!4TNkdsJ39Zy!ht*O40F~k5ZX=qs1*{0=lbiq|)yr_p
zU@F`@j4g(9A*-_4yPPJn54l$0zn4LrHCD0BQnQi9#`^-N>CNws*~zkNo&n|x!)87a
z#&^>t4}#nrL1!p<9_XK>tLjuhe-&Vf{XM!Z!!C-sLUZP6997Ev6cS3+i4S*9L2G}Y
ze{0)NbHDdYe!(IUv+diAY=#C(Q0wl<)}6S_=Y`~!fbadaVId=HcHj{
zhsXG}dR^};P7{b!A`IQlib^
zKKVdc`aN)f$Hig8JTy}Mt?snpw7q>S2dH1
zF5zK>q-&rf-+qR3|jtWe0+IspL^;Wg3S74HYOm
z8*{oYLpNb_lnyQ_)_DbaVX;89%Tz&Bqm&!@1%{Y|pH;24ij>FU;sIN8Q{qoLsKq*c
zxuJ{U{5(5PH2g#KgdO^$JmGvCF#IxP5Gp$?ZyJp;3Ctuyal-v5bN>LCDcFu_)b7cp
z-l3mm997|rL0+m`Y~&TcDMdiHITZ6MO?RMI*3?~mertDL#UuS$MT$6QEBVJY`bt%$&c-6V$?=*LZbvB
zisr+e{Vi_$1YGxhkY+K}h5rBwMT%}V0QijM)nQ@=IgxYGHwle)+n7@74yZAf%UJaf
zA+K*FAFopiYhoZ|3$9iiC*e$L7g8}=JE8ug^$x`CWr>A@h#F%}IW0S$4>8SrLsQEq
zR#Uc#GEM`>*`3+$Y^wA7#2iqWEt6EjXJVsEb}DrGG!uIQ|yA
zH%L%(hOPI@SR2}jD^DYq>vdgMLZm3wk*c}@qEBV79vtqRg4(N`HBrS?PHh)4QnhJ1
z7AY`BnGKZja{xGBdz|B
zkMQWaE{HCw*;W*?-4r>sR#Y7Td}o$v#*mrKnp`XBPuF6Qu$a)zXj=*rm08y3n^iO
z6h4m1{9g|fN$#Xc_FoV|{3;bds;s6VVQI9K{4>g69QC-Mv-(rvxcMxoRgW
zDKu&VA&Qxv025j{EK%86Mi@ux#OCGU;?yZRk7-)8HJW*qrw(JPanC@`Fq(hrK;xeg
z-_qE|x7cOB38(C~>SwYi1>xT7P9tH;UBM3A6-+POUb#@kT9x0*rgm1%(e0HKggPq9
z=y>}sUzITch5M}gIa7DBM+X*%
zl=WDSJs}t>sk}j3C<)rGPznWtrc$rLOPv1DuQ|>oFvs?$5X=ndnLTr#4MrFZ9hTCF
z$CopdQ9}T4adod0mpL?8T+c-ONPumXE~KaeN^U3XDgFpGjRB*l6Q1r!y#R|qbiemL
zRJl{+4yz~@Rn95wtA8i5s)P?t8>*fSqH?bmhU
zd022mRJ%Q+ZeXu`<@~O#4HgQ@naMyk2Vy>4c?CoY96B*avL&+MziIyfg;w}X=hUFa
zSx42;7dZxDw^ND_#D`;b>Et!NPX!f6k>5M{GoGC3mwd0CZnoDtWe7WiV@Y0v!>N{1
z(*;SosvJydu}7pfv%!Cd;eOSq)a
zcTmf9S{X)g45+EdK@qQuuBBs+_|ZB$W_6L+ULF_(TbT&kqWmL?xTR@R<2D`$y!S3
zms6BzYWPwW1p6p(gOj4%Ej{vrsG=ypifmgT!U5`{!=`Tv>SBuEDhckO1MkN{E>$zd
zOsJoXY$!d^{Gc1?B?r>Bbw^FzBiNmf3E
z$f{8|I>w-?Da^&tIG@rFc_Mz5A)u`~cB)w0A*58Pax_MHWLCF+6#oFi!jGv~X%qvh
zZ~C{Ae~9FUgDoOQ4yRG78)QpbNEc7k*^xujikSvYCQ+r<-!6;C#1wC*&4XnHfJa5G
zmr%;Dh*ULqTc|QHikBw@Jc=nf3RxHFiR8O>Q_qZWhV(;#Wo1U!s;s=C9`>+w3W3f&
z*G#t+RZyjEl@6+mM*dOonguQu%r6o22xXY`5}A_^MAsWM
z7bcfhl2Glzsr5XXzwi`ktooC%@Q#rG09SQ}@Z&QM2+Vsb+-22UE
zAd*#0dIh-sg4#5(cg~A(P}RGd95hkKQw#SM!BSO3s88MsyHgCQ{+jYD!=`C
zs>Ty~3$LZ#<$vi;cUMiDWw0pU>H3|v?nJaG)
zC@D^!O13mamx4QIMdCCP*{hW5-TYA3WMSHmMK3q9fn@&x5DIftx%n!ij(vxjQ}{vb
zg2{!i8x#aXIVsg0VMY*v)k~^-zN4xyYh(quN0nLyO|1)H*8c$MPNEf01qM`d8ihjIs#R1q#3N8?%?T)sHp+<7
zmya}sXiNIlNIRxPFL-`P@
zfoONxZNn$rpvQpG&?H?Rla00goQVP?^FS7ScEGJhr2KfL5R1p!C)%Xb(>T-ly8Xnk^2~*#akngd|
z#ak+H_EZ_|3*tUcK)xOj_j34;G1AM_w>eo;h7}QUKy-3iT+-9sNO%G&wIKE%cJx$1pK_)ouXP$i^shivU|l_x6%`m(^BhuD6zBo%
zp~|fV7hw%9MXnI;C|h``mdmZ%u{w0hf>s+NNNS8QXQB;zjvg#qkSVXlaC5qfNKE>g
z$_~ntTEItZzQ~L+1PO;4sBj?VUyyGW+O5vgj+J$(q8r^4`hzY*!?lVBn^v}c(HyK1
zW2`cUtyqMG4Xc!L@`HZMMC`KDi0wb*j*_rgM%b#Qrm6?0qVzLgC9TAh&4C*?qVA{=
zjE11Kw{v?hNy=(77|z5FB*va((4e@-6x$D(
zO&_N9H~U9ZbbFzO+MZ{CVlY#m6?kASMD*&P@)w4DX+*508c(kc~H=eSHlQ9^%X53B|U{bcZjEyY8WkjO&yG
z3v-?MS1R%|y7eoZq{;SLK!AXmJqI1Aj){`DQB#00dU=!>!2od6A_x$UCJ@`xBIk+(
z*60qzcPO%(JEt7Hs9=(}8gx`%CM&e`9P+B5$l_8aGWX`8zL8}DsvQ8wGMxir(m*5y
zx|wY=odUAlKvi<*ot3yX>e9i@Y;%(}nW`fy2pvkCXGvQ!AyQPmY5Of9ONADuFMl(-
zI*s`R=Cngxnl;B&CC+f{f$3m5D4()23y(!6oZ$yL1tN;bAFnIr(5j9aaMpnc!Y8|w
zVq!JF;>Apw=ZI?UoW!m5?6$N8Cj0;dKbx=QltN;jXp(Ki0cd4zk%%F0#V6Tp1eIlFYG-}{hsZ99^+3jx%}J?LSI%8i
zlr>AJrz&|=3aa69T#($GzUjUjiKS84;#Fv=S9e68Bl!-6R}FItj$D=3sZs4)o<7TS
zm%?cGE8=1jwSUTuO_f$^e^oCzQ#$8d+0a*mRwJMyFdL+GDtu=maXoD?iUE*bQPoF<
zApmaEMW|FNTRJWO0EMNr9kK!?PCLKtQHCHb>MY)w(Oqf6s`!Ij)fnHx_?Td=T2JL|
z5UEg(qYBd2)YW5usu1s>CO1^Eo|y|vn8yj5Elk3A9Jz(yvACLX1!tNi->1X?b1IGU
z=M&Hr#d4~;0CVQI-BD3X?Ug*LsB2W?80?{qRa9r?dO`*V;R5Xg`ao!5M@EqDkc_&X
z>Ww4civC41x}j8C-WU~eNniU!VK#mED5|6d@m((Hwul^0_RP5QPA)%XiH-WGac?B-
z91qgALE849*NyI}MLF2m#khv8Y;r}dG6ar2zP_rcWwz^HA=1^O39vfwLvjPE5(kKH
z#Ze@p<`O$2=`+=8AW2Z$uZgTbqH8DNw!s6em_>s~5xY@Z+~!a5P+($pmA0opM6cQk
z(VzJ!H&;}qRa8*xj%vAR^COamMCTb$a2aTH^+l`^H-RbIU8n_FjHR>U{{UQ6`#s?#)65#`&4nOiD0RN}l2
z$}rm}sa%!#ZJP<;HBQz0+^3>G5r{SRS~0h=X~74%C=hhwHEttPaLO7KT<}^~QlRC2
z1DVBZXfB$-4&s>3O%a})Tu66CuMOIoQFA?QvKsCLDd%2c^*n0IBJGkol7pIHqlfyTt-Q)U9wCEffVPMV0*2?LQ--7k
zrvyiSXNd4(hd~myv=40ImBn?tEKEVlu5*gUz`r^vb=QdeCY1~j7RL~^YIBr0{k*y=
zna^PRql5aQ4@psmei`M%NTMDay*69M<*yc)K>*Mi(6*oauS$mi-*qi?EjWYO$}ub#
z!`KfdsOCqIB*0r`Ivcd97S7>Fud;9s(%)CQCD%DwL%lwIN~aZ7IAahVl(
zFtDqu)e9Bf8Upyvsjx!}39#8kZuE+kv_nOdoB8g;f7N6Tqu~>Nh`3^d?sJS>PYu7y
zB>>!F_H%G*_jA<1-S2A3VHv0C0kW0_4l!Kx9{jsntdptQZ`lBNIw207YU
zKZOT6C~HfUKOLUJrBjzOw|*4(McpZ7`!4(~%_Fg0{E8~@(D`t3s8pi(WbH9?+i&N1++TMdY<-J9RMK)?A4ZTnt40z<4;m;>GvT$%vo2rj}
zN**lBd4;Gs&wn7V5}XG$Un0IhDhUVvrTcJuBh!Jw&W6GmW=6Y0GJxW*C58Zsesa>m
zsPt9hl_dktQDAJa{YLBg6VB+o>7JQVjq0^y?>*Lx5#ckkhA)qap`DhjIDc}gMDz+U
zu;Fe$jIOWd*UFL=36&I8RXIS)c!6-#-eCaSEBsC%O_Y@;%3nq|l9Tx2@LYYfmr(Uu
zqf@)(t)RM-3SU=~C$ea5)IQ-s61lhELR)c?4ebsoa#f*FBCQ
zR_ZgAwS7MwZVan&~zeL~6DPpc#GLF>6&G^
zpsiR`T44EADxO7^LZYrz%7RyQrIqiL0IOV6{VjNhaXys<)5!;+MD?nioH@6M78MVn
z;H%&*XfX3oXxpjfkS5BWlTq>#6J;PQ>MQCCVKBFS
zveasZ(F^_uO!WiySdJz%jvDPMl~o)RuNY`Sqly-nv=EE`08eh8FD8xZwBb{}R8xsp
ze;+FOR4HLYUnRIm=C+v#f{7l5Jp76(sWws#CCg(yQ27Tz`bGK;b_DfaX;lZD3FO3KPm}b2TY{UD8d!$tH)j{WxAMy
z`^Xn5!f^H~bm^b($?rG#xHI1Oll&udZ
zXpkc1VB%oWxPoJ%cgWR4Xfx>ErA8H9)k}{U^Mu(E>eN393hm`-tg5q0*UDq&qZ1Ru
zZ&d9rE^NnS!dgR%XCUW#6lyWi8WZvWfBX}_PV=Zm26tZ|WLq!XdG6FgK^srjnkE!#?g{VI@Gb-A=(QkK6VwCd`rwJ_@JhK+&WN{n|%
z?1;e|Q*fGn{#uG0WM=ldt-N<$J{!p-?q^Kn7tu6YnxErOrW9M~y+bTFM!!X`UKoo@hC;EPpybMo_&b-#fLGFsbk?^$*RHD_53BETE8)MUoH+-&v2Er1;1~yp^@EC3GAkYVd<5m-(x!t-fW|@}XB%JUU>1v8uVN
zN%uqyCjHSZx%oMc8f8%-SzS~8r6bF#r^<$k1yAc)9RpDM2{{ZAUqy0J7i2YjAi}UKg3aE3?J*2a55D_X&Nb12bkLkp@+SCd5}}Vq-z^
z6+(oPyPNNlakfirGv$lmqi@n$+&+^dz(!S@v2o6zQaN}&kv`%C5*`*rOn6fDGhmrO
zuKK$9A;ZvX>S+H0K!~#m5PN3sW$4Vs|sMCU6>N$oW++uE7qHt8lc#1(o!71V?%3Rbj;#l!2Qc)UiRM0=U
zOtafBL_6`aS*-C5F>5M
z&PBtpOjbcW#hA+v18H|rzi=@+Qa2!jJkCt8Sli@M12I%OW0tV$V2DYxkrmUaLk(V7
zAM9g?OLKsVjwf!=oBCzu7eXpwgzcy(CDzTDR`_NbhFqc*DRTAkCw6sH0?)YM-?$gy
z5yiOf<(Nm*M=)7{Ai7}ai}5(i1Et^=@Lc8)P0V08nVURIrXwTRcsyAK4q!DcU&H~$
zhH9X;c;wXeB8?@Lp*uktQ?$E|W=%mnMXg3@6T$S4mAmSGPiZO{{UF}
zg7b}T^K`^@D(kVr<_!yW?E8q;-@1!`t*yeSw>JT`$D)0pHlrA+M$-hNxso3dD{)Ac
zki^8SM&&bGmP%~_Sc&0*8{EVQfs7Hb1j`U+trFQTY=FUpmy{ni1UdMfY~hu+34;aW
z)l~5tYSl5G3_#)uk`fm?nAoXFG6N3d)rqVp`I#8~xvPa(S0oa|JEYibzS_C&ELX^E9j**2jGG7PCbog7R%Hf7Gytx6u^sTaqA&<57en$v)LcA
z`eJm0>ND`fp&IyRJ6@=x40kx0A*Jl8Ej1ji3-_#$Q$?mUFGQB^sB21t%P6oW1U
z3S7j8?pwK#xYJN>D`6<*j@emMwxB8|ctCbC!y1l-77WL?mQoa%tFI7#Qqw3-6@AC<
zmgF%XfN?%=pmMN)29jE&X>+kS!yL2d%YDQHw}~^7d{H-rB%KoJ1$krcW7Hu|ptN`aN4jL(SHS?~
zgNml{h%b_0MTO2Fqd8SnMdeFurUo2+d_TeP(U?Bv5(Fb~6?IH>zfj853<6kIBQP<~
zWWl(J@$ef$qVZ-1skzj%p3McBmJwUrdc~b{E^V2V3YEMQ9a7_3+X})GR$G-Csht?E
z;O#pyvRuFcz08j(e80HW-;(ID6a>P2YB_lY7rPRr?JTOfU9vKUb*;KeyVJIBX
zzR;obsKWmMRK#HTpSQ$S@mS$ABrW$dD!PkV2fUzoSE-!0a06_@FP1lPB2g&XPnoj$GY>4J_*Wo2u3ps_r`jO1A{dg)`Oi@G?
zl9OZ^$4*Hva`hK6f(=5s9m~!%O8SY4B3xsb03@+!W$JAco5c@L~
z-#A;#Qq_+E%M3lD9n86ia|AUxS#=l9NF+iYHp-Ql_#w}gsR@A&wq^X)3Y&rtyPWCi
z?ki<98H_4=a~t{yEwdXQ1EA_Y!TuSbs(R}E?|RL68ur|Hqx0@N4dxA)}Z$nBB_5VqVe#mQ`EGyX~a9kZiri_
zx-pGHy#AwxJQzq?)jrvyG3HTU5ZW+b@hP*eW^ND{z#
ziNsxz*#SWE1RjXS;}6s}Vtu)rW19+`B@9?pQz+HRGlRI7Uloych!=wSW)2zZRfeT)
zw=
ztwi@T-;NSY9KH+9lC7aNWiQh(WSZ<=C*-5a8jaR!+{^&=;UXvE4gfR|CakAfCnUi&
z{&{dffK)ov;Mj(cZq#P0>J%IVZe*bCOO$Q$22`z
zhb#|%7b?aB*wo4{$$YIt-4V(qMlPB
zGCiV^=!omWo+8u~A-p~(%6C(m)A2Fw_=z~{+(U1|bR+6Gcx9ZS8e|`cTZ{q{<6W}b
zjY@!>XhV6JsIPB3QxzCSxMlo7Foy|KID@34)b%5bg)n`Jbw8|+WZ74G;Y-#?WR&KSbml4Ot
zL0Oi|DcYuAMd-|!8%4x0^h%ZolbJ=lSz`<;xRJZ&o54oMPl
zl)IW34PX<6F2Qot5v29YIh^aC7AT%shdP61nQL$!XDyg;C%Sg0w-C)nDLb`<5lWBT
z?w*VsjO`HWHgLm!MCx2K`)Y7d`;=TDTWrC`Sxr<{T)}r0zM&L#D(>eFE7D^DqqqkV
zLz%TytWk?HhS-CcHlaCSA8a65Q-sW9dnKj!E6QQd2X_j1lwv&c$@*W^M4X-=!%Kak
zLJm_un5*p*;FmXe5|DSv5+aNUrtTWk7$DyfE9Fp4mnmX;nDHOfPd}tO*X=!kji?1~
z4X7eFWWx~{Dj??UKkRHTSaR`kD;c({_iwq=TMpeYwMlb4)&?Cy+7}5_ZGXB+U5f!
zRBA6-h~`>~nSdjSkqr2TTtb+KGb{_O9Y*?qaLRm5sQmajc-+j)qd0Xv)7>lC78OA;
zuGsd2&5K;uF&t5s>Iz`;r}imH0XH#E<&+f>oMM@`TfjG)AzmPu7JWg{Pg6iw)U?G%
zN(Q&o36>(f
zvo3J}>6GOxWvZ@Wk49U&mBlK!_&`?k4Vc7Vn7Z-9nElPS6{kmuLAgr|b6JvOh}_&z
z+&dxEN1D)Jzmza*7Wzt+%=1wK{7XRh7&@6KfWN4eD|G-wZV;rGiD);9aU2M-FDy~X
z@L(Ua7kv{Q7>Djs+MQu=l_SHyOfy=SaCdELE5J7@#6rnT5=sKe?gXPYHkv^g;|yf%
zz_~sHn&S9|a`P>x%tW0^-_B*0Uvi4*cy~hr29t{%DZvYcT|iUu981_$D+RIh{1DQ$
zh!vSbaV9cjF^oY1jeH|QD-wNpxI^s|+!9?Q=?b`jI-G+WaRV0Ibz|cK@C|+UXsH)JxxEhWKcSg
zXC{{{+)c_&F)K>IW{$UA4;WEY-
z3ic~N_XViuF}R`W3|7`}hGY#o@ixq!p^Es1fV_n0+}W5hR7SEP?YUkS3q)ze&U(yZ
zdH&(b9-*uhx72Q&`-(EbFeI|1c*$lnj54CRbiB%)2<-Zn9uvVa{lx6UqJYJ3P>dT*
zvZI)Z^R8Iea2yPw#i`#H?1n<+m<)z2IWpT_#}Zz`Ma8XyAg#=LiuD&SJ-ii_;U96&
zEUmkT&SRl6Ga0p0u3he67THoo9Wy>G`%E;;@|fmmxWj%q=ueg%(v0Me=y(i*F&3j&
zGUNq!714SnMYg+EKM=Lxt6VIpK%k$wnLtMNOD@utgWLd^z97tZ3h%;^M@(@G+>cP6
z=7u|p(aGU}6vPsUR#|W%6P+*p>R&qdx44>p;-+54rXX3}WQ*Gdh!ArX{HVG;N=%9P
zB=Er~(Y_!G=mOyRYCKFqgVf^BQ3qaL;n{6*8Ham=YIfHXOYy_puvAM#&*BaTFyO*H
z7~CHjBZX6_M^I$9DFTi@a^|#TlxfFsgjn2kT;)^h52=?BfVY02nfwSipS
zDx;0UBG?WQZ&77?cP!q#Zrrh`H>euS$qHv-Dk9j8vZaf*ir9k2gAHjgvNZECaa92IZ0U8&QW)=P|meUWw|B3vh&Ea-%#=o;b@Y73(u3
zt2csWD!I1d3DKQKQ-Fb{n1i1J5o>TN>_4eOE3*tlChbGLwyB2a#kvwvG7kO9l^q8J
z(!^mbKg7B-+5JmGX6n`fs@v7nr@?*3%JQ(t<=jKnFQZwcTAYcRRtu{;lmk#gIHjSb
zjybaa4kmK8O16@-3L8ktSS6QhGhqg0K@u_{m1VeUzV1F|TL_G{c1%$&;_Gt9GL9fJ
zy%g{Uv_+pCJ%;CrSp4Qa
zq18(;e$t|{H!@i?MOO}`T&L7c5e_5t2!|1AtTf8rF&?no!*Lfu1)?g{Q^=O^%AtZt
zP3~*77+gr^J6yW|0Btro#C7vD{{WeWnIO0lW;o6zH<-k!UI}*-kV7m~>=A9Vv6fHe
zC$N(W+FZUR{Y@9c1DA~i^x=k@oePZMA56%TwES9BGXpONOhFBE^*O$iDmW7ve}yfV
z6C0a@4t`0dyy)f=ij4+(%qgVFGr#)5N|{fzNTpL;O?Tmjpndq(tVsf&vTy1OFv|Iw
zOfqU2n)#QODCSbwLq`#}RS}vN?z1exP9-~k5%*)>;9bgKK&BY5#$m3^Wvs{b4Q>%w
zvzVwF>M3qoDM-r9M;n4N3~WMh4m>gowd*8#;q*K9CJ3Bn`S+A&$+qwGh@Whx@?7V>xk;^T+U|`4r4h%PyK>HRZs5!
z0OWQk4PSruV8{dNQP&y2b1zmj{?V^8@q*8AtZba5v2!#1HwK$vg03Q)(ZEY8_tZ6!
z4G*Ni^%gvGkIYGlyTXV<>sn{kn@aHolB09P_rL(FO%Tik!oG>bKX8nYH59
zC9L6=wy_5lxm-&&Iw~$1_RTF=@6izP^V%{lGR_qhxTl5>mw|HJRorVWMs)zz;)dMD
zpahl@ic^N>bu0Oohj9{fznHBf=5q*b5Hz@zfZl$kC>6;ThM~ro4jEH8tgT%~Rq~gb
zoP!VvCgfpRe$aS$XyRMJ$eW!J+=V!0r9*JpZaqW`qdD#i2&Sgp4xe(x@tA_B%-hs)
zV9jsPCB4oF&@}MfZXPP(J;=*fE~49(Iv^TpPx?-Rt~{X`4q%)ikC}P6A91LM`hhir
z+Ce*F(4UabZVl}T*rXezD=eAM^
zf;u3(ge_(U>4j6ce>2db*jG(i=JG^@iL)4({0
z%)&#L8-TAnr9>9fL;i@;jLgFPFyc@wD6ZP)GbKQz7?uMqvPP$vU-^f;f^tvXP9}6<
zr7;cxNdutzCAGaFO@Z!cu46WXz)nQP%nPOhF22ym7W~Rw&mI@j1|Xhex3*RC`GdH5+-o#b
zDXeeOBX~x{;%zW=#*AUatw_gdM3~IP7#8B8UEJ;0scM;8%-htkK%Rcc!wgDj
z>KPY)VR`A)II&()1E@x;5Jm|`5)evou3-#?c=Aj#jhI|^HnM;)V1To!QPIp`l<0us
zSO<%Y+DUsN`JxetYd%u$R7|(ISVM6RV5S+L;xF7(w7(xRkQY5XOKE2bMqUxD--z{L
zUL`%wMp(q?2tJTpz!~Vxxs|H^%mwOQ!Z1RmLS={=#I;KrBg*)eBmO&$gOX9Bpd~lk
z7vf?)WT>yFNL*bHS+u@Rl~^yZnRo6LrCpnW%Sc0E+x?R6o!t6NipXekcNmsxRG$gy
zf`G>JC>IkD2kKtM_$HD|qw3&wTJQ=o2(2}7rm-2k3^6irdzu@R*DxuBea5xXs+DeF
zhjSnz8K=QqxNZ$@QxdU1aK&Q;&>y)=7npYDa9sCCa^G&HOKr>rO*85Qy9j0G9$9-r
zu(1prpKyF7wiPzuvZBI&o&|S#3ywmoN=(!2aekR;_bQ4q&JU%Tdfx>R_rij1Y2n^o2k?d5;b)Wy|26
zm;sK|mC0>VDWX68`-layw-F)4|Q%tdQjfv_I~xqYC*($-f;
zgw%$h#J=DouoJoD3=dUTfYSye#Ec>RKqw8E|0P46w_vX)bPFE#Gi}angU+
z4AiG6a+G6J7%Ez1OJWO`LzKb_4cs%6=37~q)+icSQu-Vdf*GSdL?y{|mHz;gVJ*De
z#w+bx?qr9-4HCy*V#&Ch7$2EINzHKrI7hql1M2B=sGQmbCH*8uPYp2X_(n5Lx
zVauh~Ov70FC7Foh%(cTAlkh=84S&N7V$PC~9MI-Dp8*!8UgfbfDwtwU#j`T8?887F
zt}&G&MOmG>m<6@0=>g@aKO_q30W3Sd0t(T*XERayqYo%mt14;r0iMcb?rag~6Fk2p
zxEsDRXl4&WU?8dD?(xLsDSiEOmA
z#ljK1r|vcbJ_$fS2O0_2SWP9x%xEv9mZ+Bv!ok^`a8F@JptYOpm^8(ykeADlviLr1!q42$UGc#K2@MR4T}P)=@*z2w?H5Wj;E1?~
zKcajK#2-kiSzkCM0Oic9VRH69=Q8+evI`u(qOSl|EtbVRGAc913;0cfIyn%gQL?Xc
ztRZ95Hk3`HPjfL&n3b^bTe6p!Pg39v!Hq>gnnJTra){99gpa^A2?`5P$*DsFsZC}T
z_@Y_YztHM;v16u}1}w-vQiCJ@E?B2>N^kO|pcrEOHsIlBGcCu~CO#IniD^LVv}(9)
ztl1ZO6PLKR3?M~YvRZKrn*RW4x7?|y-7nZ|*TR
zN7Q*@%Tlui%apuVJw$aI%2qTE^ozYiM787f0WaD(5;p-bYIY+KbU+XRa_`^l%(*pn
zb8vc@nXSj+G1&J2iW8Co!Mh?wM?(sVQyEEuDfbXHZWBqA{-H1kQcr)1AFoqkBNmTR
zzz?lHPGD-|n=>(cD&Oh`Ypv6mX#o4tsB;(O98BFdtEpZx-r1--`Gz+djm?DTAI>2b
z7gUu~eI$JassbB+BeGfWmV~#6DTJuOijSY-Tbbd*
zQJ0
zukg5mS8Jn4TG#Z&yv}24$7~9NL>MJ|%NubG=^QXcjt_%qbi~LI!)}SPismS8Tee*Q
z%q;Q2$ILsk!JI4UotEH=SXB3m{@yo(@+bIF{{S$82P5IL{ltox9dwZB0`V$JMj@os
zX$AWx^#QOhMYA^Ee-mM@cN0$WRcG}se-y3{{WP+
zP~W&{e}IiOM)1ov!{R)D!YF*t_(WnJoU*9j$V0~r>RvfbE}{<)Qxd!%r4l!BML6_L
z!03{*D6!d(jXHS}tCG8kck=-^8;OB|1G$e4fQ;?{?4Hlu9@$Er%kb1IFv^n#-h_U-
zJ?-}sxYfcvGe9K$?m3h!Ru2fLE3U>pW1)bJT`_K1Amflo?c-;na>o|zR;!Br#i}w&
zTOA4SVrs!LLP0;gkFEGoY*BoZ{H=CMUmlNB8C(V&)j|td!_=VRWZc~l9ZPHs9Hrxg
zxs7GZ_>BO@9}w}$dsNFRmFrUuV0|eR`OC>N)8OWb0%xVcZV0Ahof-JJ}F3n4>
zB{Ll^(HCH0!55WZL>9+hMsSl@ho`BOM9pIfd#aT=OI1fJPt>I!lS~rS3i|^v1<*iY
zCyPG$?t?QibzLMl&zoP0qSLuUDtyoPGe|v0JuU7ntn-L6)(MQt?BhTAEo!{#=5U`t
zKH{R{?1Gy)m!DG-yi*ZorAc_(;yo1}D6eqNJYRv~#fZf+3`-p`3DC;Gn1PGTNiXmu
zMhpky2{QgPRe4A?JC(SzJc9a(0IwL)2lOwY)OT-^I7-QV!|xjMJGi$O#pY+&zyZ~B@T&Mw|FTf^c}Lj?ACD)9QI*FxeMwQaIdouPodluqY~Rb9ByK5EQ_``
z5ST2&9tu^#?pnU5+a77J4t`;^3PM@m$JS4Q{JbEf?*~1Cc-MLBWBBU=3e+
zsn4lN@(F%F@)A6VV3n@85NazKTa1tIAl&g*0(+rUwuG<~2M%Fi2#qSI{HO&E5~k41
zY35xtCZf<$tB#(BZ5X;1wTWd*qQaMmxHt&oz!l()2H}4ag+g(1OK6K+CPpoQ=2WsK
zJ4%7&uM31)Gc3G*V!De}LA4>DPf-TnO_Sj#gMgYv`hoFiGDGxC;aKpm_M_poMKue_
zl_jq381-EBFy~JcyU+WZ3h4K>g+CvRxuku|C5K%lNux)<0=Z*xEQ@Yt-W7Xd^q->P
z7l-cyQx8nCC1uRP&gqq6RL2v(S?ydNp`7y(=3ZIO3*3w4099>MGGGb_daigS3#aul
z4i5`wDp$p9)K;Aln(j!OO-X+7UlO`8P1?+CU$$)4xPes8v-c=5Tf2h*=)p1qpbi15
zjb_#BxLc|lhPJnfTeX6R4q~k}1HnLQ@ehD1IsVd=uf)5W?}Bfl89$;g8uNy#R24;9w*nh06F-h&
zK4+dU>L49UN^=~(&&(yj%MUCB0m0d*ry1}{bWhyeSvPl_OxLU#Y#mH|s>c^_OA0=r
zLwviHIUeQ!KQRkoeUIBL&c*?~_ZqJZ?C@ul1Q!fOZDlCB%(HT?kh(zVfUTkxw9XYO
zTCB>P3|kK&vkNS%w#FIUCJ5K5*vz1H6O7)jJR?=lGn1K4nPM{)8c=2piAO2LWtGbF
zFKE!T1#2zldu7qLsP-^8A8>ELGYEc{!#Iz?3Hp^ZI)E^X@62VKGqctZ}9T49yL!MTkLQG&9^4#IgsPXlx=Ixp|Gk!$Jc#
z$Ahz(_7+nGNaCXU5G6mV_KM600mL;T1t&2py~ZnRXehd)qy>Z#j%;U(#0~0^6of7|
zTO5LAKh8z=%Os9jVbEvjf>&ixX}%$-v(!r^vgDJ3I7rxe`I!44e**Y)Y$7RtqEX`R
z<_MkmWx{4KtqcIjWw#a~ZiXUTTQ1zkFH!LgGYJSD=0+h6Einm+NErge?Q@Mi%b1q+
z+^I)5Gh2p4M`gL0Yn|#*uTs=->UYH2WUeA<#HH4r<(O;gTYY~?iOlX%eSfKiGXpMM
zvCBGCB9dFo$1Gm87p3@0!cz#-m_Ox+E?ez}(DhQGQl82+d4=%YhT|L7{^9fuZd%82
z{*&&7ehUShZh4qlyhXiFW)5eN%d>3`!<@bCu_sdz*P2v*BU^SFf7)EPzi{$+!r%O>#LMjRvsLaofG
z6=f_vx`0QinC`y>vx)m$m_Q5xFHET0%Mx-Z2t+S7{-4P_4iCCM8o^sQz6Z=pWsqi6
z3gKmwbi>zjl)zlQ7=?`D#8d_uW~+$~?g$A0ScSq#gmozHVjXRl1@Qw5ilw6IDZXG8
zth1Svy?U1^i&r`3J>T3A+tO?T{mOB*adUcCF49;zIF#N|
zp=fAoFxt54UZ0|B{{UdzGz|9>5Cik-4Q?yeV&)vp)Uf!1zDQ597WEWz;uYcmR8TNS
zPjdV926nGh2U6o#(w_k-z^zP@eOwM;IE(R)(@;yGh5b(ETXobo+M(OxQ`4Ein2yAu
z1xzI<*CwQ^W{6%D9^iqaduD937_>e{A;Hypg2EgPuyclqvFlR(rQFLRwj9nTg1{Rg
zNl&Pnoc6_SYZOW<`hx&N=l*Lmj__hPsd;E+00ljmmrxxfYk!0a-IkoYmaaLBL4s}=
z>pYQxUsEl~xQt@+NE8H5P`jd~beR*t{^C1AD)B2*{mkSZ?pWRzZN>8Hr2{5KM#^uv
zTD7TbojGO&)b|w`%ACZqu4^sjIa@3m!@l66s{4Yiv#6=gH2XoxhH@F|SsTN`35~j!
zkMW;zg$po-y~)J)pQJ9^AKd+^D434WK@;PsMPikNS(V9C^iOso+{OAZumu8M!OBzz
z+yqA>P?RpjHCmFm9e!R|4LIuk)(aaFpOl<|KP9@2rP+lVf?l=Ueeh>KP|
ziAG$o>ZSw2#jRl|d&fj@0RmJAEsi~^dd3w`GuD3PP-$;vgUAB{RdhpVxQV#kL!oL}
zRFpIMC0{G@!|Gx$QD(xNLa3MH{1~{GnQ^&ssDBX6<_jE4QE)&@d#G-5GC^32aUPT1
zBMrEYElf&w4%w+zAxIo?FL&Bp{lhH6TAA);>1cPDHHoy;&?jihVVNO!D-AGLW&V;=
zxD`r8_RImk;^Ol$Q>4T;>K+49fheGL++qvqG>zmT+L)MpPBOx#7|pEN1}cNMiBo^N
z+y&Rk61n#U68A(Fn2_T@BDgt6$&TmDK{)0{aeM>x80NY45df+{RlmQ)8q5d-OSI)=
zG{11!xdK?&<1sBW+X~d-Vr3|<`j=UwWy|#}ZIoY9fE*+t2J=GZSSme1XnHTWt*bf?
zRQ`}PHpj$tW5n5l*{NSFW^jJY+TB5@MShE_-N6+&hCCqQExRLa5(X)1Q*aKQI*6MO
zh^K4$n~PmVNAmG^6bP_Y?H!Z+hrc%#E+@Bxkr&+FVj_=;LeQ=l7lJf^v4Xu+wMG4C
zg$f@L%5I}9yXZ#k`sN;2a>%byOj|2ZO}UkU7GGqq8lBWG%nLU#8M(WGR3KanYY?zP
zt0AECQxlB;00iELTZGg})ysaO#ol}!^#pm25sl6Y38l^@W|{H^HloFB8cwE=Le<^~
z7_Vkx%XK{wTEO;Cw`o%%kA&NBJ1Szz;r<~N-|B6F^~A%W_bPt65MQSgDIO6ltl;K3
zR=v>T18fqlN}F$i+bl;F7*oR@QCiZ}F-vU;YV;maMCuEIIK^u9D;Y5Xq*Akx9_0kd
zeXu;Dli#?iUKDzrb8cmj2d*Y%qd|gRuXn?#)v+Loa_x7B3dm?N-;pueA2Np#^-EUn
zI>pS^Wy{RHPt0)4{5&ygEc_Dg2)dsH8|Z7pE$6`65!ZmRg|Q_VfoTC0Gma%_I%HuB
z&&Fm6k#^_10o?6kVAi{3iUAQ>;K#U>eUkAAn}+K(ObHozRJ$s#aR{0mFSuW=l~{JC^A}gB
zwV0_JIGk!fmygmtrIOxTi{hpS&`~eAv0{l*;-n2l{LgQS*tfSThjRG`9_}`nRstqX
z5aoR_2Nk~D%l`Pythv%ZR%5$n5+1qS0~_v8YgbbF$cuC5#;Ug($WxhKW$ed651QnQ
zrqN#`Q)PsvlUQ73O(HO30n!P!A;Na1$LL(K7t(%XTGfJ)Mq#W{ljRqdIGxkEdxi;6
z0*OtesvSR!dw7lr?*9Opix@p%n_-(~w4UY6#RqJpX)_zmUgKUeZ@GHes60vUAT7!v
zI0M_bvzD*rg(!7qqHY_bOx%#Z(=Kz|zV4EMfqFm_0&$y>wyMS$C5py{C&so_9&j=dh+^g7*lwh|UN|kFq)kpaT
z?8D0=m8r{z4NC9|;Fv1rB+DEpu<=HJM4d&{aAhyZ24{(mOGCDaZ3?_20O(Zo0D6Tn
ziwg+08X;i#8Z9F>PDyd9%EZ^u606PpM$;^HA2L*6_Z}g{E8#s8m@M@Z-{PL;Jw>dV
zKa@NPr@2U9gaO&xEqCDIY+M8#LNF%X(S*36_Y514QQHsPpkNc)H~N>H%c!)nr%)E;
zdb|X(;tD21a^~0&8%K20@aP7^rHM&9H8zRYBz(STCuz;^TOW8;U*0v*5A0ean{z
zxJTT>=4v~hjwXq6GJ|IBCuQ{m5pWTejp`^;#IY_^FQG8r@%REF!lk&z3z+K*GKO{)
z;v)feIwt7Nw97rOLp2cq+n&f`w|7r74-2kE{-Y{6>O1yUU-1c%w%9|CO%-g(8?REb
z!SF!fl6xNz>CuW1P)nVbFIkxsQ^A%PVm2x`OcCc0NmutEFyg@9xP;av0FQ`NHrEWU
zA-nKs5o;4Xu2<>Dj&?wjzX6Q@01*30oMr;bI=(1k96{|5gvQsezF5~+2rZ~1PnQh|
zI-C0@NQP4fc@YLFnytX9|^DqV;7euB&4j2H&)otz)_`6Cx2L~F8
zEv$nV1?lY2WzeSX6f0{t+`l3N0k<pj
zZpw!UL5sr?GFiHrWUFjdUcs5a`BQnM_&{|LXd&E=Lx`xL=WJu(1_%lyCH`XKCVxk=p)E&Zv#YD{ukzS6%n&F24Zb62n
zwsQkrmLnl;7FvrhnP$N4U-viit3Q$@Z7&Alaexj^f>z@Vi736D@=BYBW^QbJOO+)Z
zT@0}~THaG8kDhl0}c%Hc8VS(#amBGH4PF6fn_
z$g6VC5~xrydyRk>8+r;MFO#77GaH*S5
zXy3R+7SnnqY9E=0GTf$RXaP6b09043r&+n&C3hatyMlu4x7@JINBUN
z122EHoEU7W+Ho@qjgZ6S5#Iot;*!pR7VmHkT~6pQ8J1d=HZKGLZ)Dtvt!2omVy5TM
zxDkuxm8@bR7ITpWsH)h7(Cq?qWv>!T^J+5AqXd=&VaC?PIhh%J@o>YcGIq?rdjo^+_a1Ld?;#o{CTNfMx&Y~A(
z%8DgBMPM!i$1s>9_oYyKmIB5j^8vYK#pv9`cs3BaV9|vgPk{o(7t~)kLZ+@*aUSMr
z$HY-R^;0OTxQh8c48Zb9Vvhd+sp-Jsh@z}ya(zTfFhzkZSnRq#nPUj{A;&KYW%DiZ
zsnZz27GbaI7jBqzGUWG^kHm18p^(x{rq1I_g1?qkS-Xp1V3qQ(xo!|}(5ldA&$R6f
zQCF#SOrT9>Gl;n`ygUA;njl=jxt1?-@YGuA04aS!f6FK4iHv54sY}1*FccpIMJzjq
zP>f=s6s62-3l#7j+;WI=-rk_Q`>ZIqh(*CLhcczyS}~-Gqb*B=5|An-#WmQ?!2C>V
zA5i61A)lH}uYe%-<@em)6V8!BvgV}?ekJLoF;;{KuW<^ln?DnAF2^>7O1>A|Df!ks
z9t5R=
z2r*1y)3g!o4%mCJEwRMzHE>9$?_#gaQFgH@)HcacjCk=1=wSpY&90?P2WoQ?tk%mY
z`oWt}Ytx9X7uhWqX%qi-LNzR1<@N0{X|GI*Z9)`b8-5D)l4R^_lfE*S^7$b
zS=2Eg3@mkAuK@?7-XnwqNfa@oa(qQ}1?u04rvk}j;mqFNGD8S(^)Pd(
zouQ*pS|lTeIk{Ahd~!{akYfJ;@=>e^9(UoEq0B_+@RX?t%NtJ2M#88Fp_}awsNDs-
zSe$WshT-6?CNsevhZ7JRJQ4)~!miiq8KKJ{wqJD{LaT-$zGpg(gXNe*-e8)Rqp973
zbgBZq$NNYbvG%YUt>Ee!`IWkXKyfnZa)FP;0gTEja>FXv;q@v2rGpOj+IaOfxI$%y
zSi@um;}&|zw?w;m`FkgC$fJ$Q#Bpd$KBEfyI*YXrGQp^IpK(Kn_=j+XcXK8Q4dw%x
zU1k(=#(1LAt*Jtp`s|k|4+AZ5TUx(}ZLl#*%X6w>bWYA?O+`3|^KZn$n@q$QB|WQ|
zn^*WQTx1}{H~Wa0i!&iCi;cMZQaENOlAsaCVJKL&6^GJrsH@-(Q9@O0m%t2#H|`i2
zHM}YHECMlxzG7nAbuLiQ%|)49Ad~1rGQ2QTQl1nVh3z>-co4;`+su>4u)iuQa5pPV
zrf5Af!vsNUyNp>d7#oWfsDo0PsQ&umwh;N_(ylpSn3X7IpmjB
zb?87o*dEd@yCQ(8p8w@GM;DPskaRu`2g5)3Z3~
z6v?R2xy&pkr7e*lsd8pHj<>!HzEPtt>Ln}tpnxd#NntYqPU+O5zqGG|+_z#*WHgKk
zK@!=5<)nK|2uD&4loT%2z`%j+KN*uzQQ)}T=L~Bqy|U{N+{<$Q^B9b6;{se`&yqmK
zDw0}!0n9;Zb8^8H3(bWd?=X<0SEn`ZSi!jh4#Yi+e17{x(J1lt{R99--j&fKC|hF7aG!UB7VS|9nC)(h_>uot7atzfnD
zEk&z!MM2HQWsV@5ZYmuT%Z9uZIh8EN=8^fc73v-SW9l!No=NUNAXhFZmpfQE!YX%?
z2z*Ovl*yitQ6P=WBZRKd(TF+Bt-0?bE~6Qrm=|QKr!i1K+ce7zQ)EcGGHxz*OuK~V
zL&ZvcNabqwl!PI@?K_1ql3Y#>AW(6?nG%arG~FM*t&jVGWB1(MLMpfAm)xZdA(4p9
ztC+V!nYR9A+w4sciLzLpn9OsyimG@y%W~>#j^fIe%qCh{?k$sS!h1sHq8ATD*a@gb1+!%8B3I^i!s|Ow;JU(#&6e#z1=eR
z011#Ft5m^~KWEhJgDyk}P(o^0{{Wcn
z*e}B2a)|i~SXL`tTzVTn7gFy|s10Fq>NtWs8GNlY{$LXbcX1eL3c{HoZOUp>l?*8d
zW8@VT+{wvL5DJf6qsEYeBl0D1%k-H1O`P(}pHOkqS$s+jmvtzQL|J?x?{gDOVBr^6
z1ySXxfMwT+wCVyc_M7lZ$AKR(YttLB-N$BcNY1>uK9eyFfHYtWxXEs!lLPlTnfgv&
z1f|FLH47I#yu(SH$}NUX`Fmr#quDG0?99dNFKot(c`6muSSx
z1Tf4sw5l@H5f|LC@#fis5vEijq8VTaVl1*agz8Hqh7O|+3Cwh)s{RAOwK)gSO=_+>
zidMOqcu!DrVbEC3M1Nca-(AL_rFG_L&s+#V_Di{K+}zlmWacs~gv?9nO@Fo5!#yN0
zIgaUvI`s()zzLER)Z(Uy54h%M{3e-^6EdYrhLky!N>pjVGjf@0il@wv%(}xIn1lo7
zu*`^$!yG^24hUa^;#I)4grTYf7oTh`h!IS0UUrAY2
zM^PxusZb_OK*6{wI5D!|P_wz!)`XY=~h=if7aoP7!UV3H3f3
zWxO~D5{;b^i4il=hmoYLV-!IcBW^JJO~NUCOVr)ABoQ^MAknHRWo@WjPk|X;WhSCwyFn(;Qn!y4r3u$JWu?8xsh1qUsq%;;
zoK#rhDGnt@g~REB6S!s#*XDj>{{TdIM1)_0eLfL|a`NJ6$(geSn28Gn>N2@XmjId$
zcd2=eFtuZt**fJ3vK@S;4AU|*_=3lTisPt@I-cT?x6)DFA~IQm*~ebxI4NO=LscHw
z#<#c87MsMN9YWB_*mj4^38`>aC^0GVQj(rySC13;4Qj?4SpS8~E}+mdVsA|L&d*wo{=iliyQ7Tg$#xrd}0!E+Zf^)4vYTtz}G
zZblI&)T}P9E+dtQqeSa2@C_aY1~mzLwidcG58>A`_WO*Gw`8$hmBhR^vNFeyXf*gv
z(w&msMAggp<(93+`-(7UUglN47Zz2u5TQZQNpaz0!D0dKV=tM3G
zCMJ?TV--cgrpbgKa4$bpsK66aQy%`ig`SsJ7&(HuxCR>M6Hqgl?h@{FZf6$tG)EHx
zxC0zp(It`fB~iQ;s6N6|EnUxU;ch+GhFpAX+;aB`K#u8Bel1
z6#V8XTDRYUi)q;z?C=j}Hf;(Zom>gTvR%aiZ*sgzXbQMZLZwfALIrYWO3GSZp>HqL
z9&-;|&1yME@twJzv&28eOP9i%mk}PwFPkhol?ABL#8SwJdPfOpC>7?i{4oXqAZAuP
zsXzqqVt9As)!K2y1$PldsHfoH%mWtYVVt1l#4-W3ZJ81KGNIk%zUDR1WrN}of|v}m
zD%JB-C;cf!GVoI)503Tb-cSukY9^EODtigHMzJY@duj+2;^wC2O6KJ`%AuShm5773
z@qR4Ms$g2Z1QkGD$zvMa5%<9N9Ef*un6|5&Uop6mJXEQB2=R|K1WqPYpbKIZQuFF8
z=iZ9KtWns@hF@>`ge$JjaZ?@ryvz)}$Fyx$UM1$qK`CZBhK>B8AVY)*=wV=>AdC0{
zbAPokh4e)8u$m?K6Fp0p8SyO@Sg(j~6?3ViUo|leGV;x)(?+F<;#3|YH)C1CQbcLK
zBbBttsSDTm`Gb00ai~~FPgsYO^6Qy%(%e>dO0zqdZY-$ysuH49JltZ;R;^_l)8KEw
z%kw~sh&M|$?had{tBbAr6M8i+EV+1#n4sDkAo&iZ%fufBkhEMx_zjoik{ZH{vgM7M
zircusXty6zVBF1KSMSuNtz=BN2N+amgD@hl
z3unW>7ntAb0%8}GTZMu)JMdrOSA)M2)Cib|IA4$TVn35n@07tvwp4J&WuSv2Z@7baBo0xkqGSZ4eFVR662Cwu
zF9feQDnFv-?r=qAaVj-a)5s6xQR;6EF9i3?VEJDpS25HKmq_9pFX@)tdSEyS?j2!W
ztCAtW52?Q1tC-d+h?>s%*k)M^IGNG7_9f&awc@4@51dXy)ytzMSuevmw9-2Bu$zNX
zi-OCSFUfZXq85?$EoY)H7n;J4Oa-Ry#Mgn|gQ9q1Ufsjmi(%ZjDFScGxLaHNO($^b
zI*PQRi;szd5!BlQ6S5gUi{MKTcWL;PZU{$`enTTBH7Qk|t_)lq&vd~VigT5fdOqhW
z4a8~-6@(;w;7T)?TehoFB4JQ9|lQSl+8{jF+;%2sQEJz
zrAjd2l$L8m0Vg2AYwOKGNdh~V0@~_)doPIPBX!MdOc%ItIK(osPtHLO{Q`5Q6R{Kwy^|%vA~cwX$)!Lg<iRaUL$M3crIJqSGU5YyN$$Dd|@>yWtv_cr5N=^^W$;xau2~0DO7Im^C%uJT)AY&o0TiYK`I_&
z3h9*ll39%*mIsHGd=!C(+jDN>X)avr;6=-qE?l{C<;#Omzmf77zu|Ka0W}=OUJEYe
z%a`Ki%a;^jmjYBtaYYSyxIO}D;KIKQxpIQzWtaQ`@JC(g^R+---vI?w#N^}VVDvyh
zA5xB21wIL0p{7$;Q1vLw!~AYsUo3nb_;i{cMvop-9|qqpd@%59{{W%mpBz3P{vq(r
zfB2j5H93cYuMBnPJU_(wzr$yN{GOafv0T}QC}6TVBAO4Vl^7%dL&Zuw2!pa=jCu1P
z2mb&M|Jncy0|5X600RI301&5%*hUOv&g&3x&Gwm8h2ch;*mmv*l4$2DMV+pp@oaGm
zWJHM0IdL)F>3!fW%?GUeHInus+AT`s0)0kRua=B4n9%4iv9B`LDR18K>H*-TA%L@7sbF
z{>qboO}8-d@-6YEAR>cZj}?OIi~J;dA`F;592K7uDi8;cCJItpE0vi!9^6>0^e
zOT(ZUNjDjFWuZw3Lm5!)Utn6%a9tFLbvo$deIx7{4cMZ
zeR}X(y@PJ`$Rl>hm@Dw5$HR({>dK(>;CK*>Xtpt5H^RohqtvJ;RgW9wLq~nBig?RY
zF5G0STuE)BJQfo|p#9uY1+mNTbSFl7jjTrR@^Djhgr!j36OM{oCw4fVk*H(WOCrAI#=0B?^vN#XjZFy
zg7ygpdK0mYGkpNLwp0vV!2I6+X(U{YH=u|XH%C$rcg7m&;iGEMob}&RpJ02Px&gw(
z^3$wP<#+iqb>BIt?7t1CR%oe8V+bHcC7!WgHexlol)@#Z*(jvraavn$`XGe+TL=sf
zrF>7=I2f!H;wYgRr_zC4mN*)-Q1X^0$z6(wsxD`)3Zi
zFDXC`e@0s_NxBevn^bhbM<#KPzo?jjs%+jLR5i8m{jEh~Cjy6TW?WOx80@`W28e2c
zRM0Se=8(HZ+N-;KIeRtq?A#>vlu4oa6mZ
z-Mx3p!$+1Zrgw{SO1(Lfg#arE5KJxVfOYa!JWkUZaV0$e03wD5QRpaTa;C#l`rs$y
z{7f+tev+&%eKGM9lsB}g?fojN2j|oh)ENw>KIi@b7ladHH?PiPrg3C3zSL$JT`Ogn
zCzT+HrO+;bp{O&`>&1om_)kGDbTNkauVf=3m56_cvK9
zz|`<+-*7*ZvqO8Kh$n#EdE@xnUMGqm>{)xE`O1zZJithKNTCtV=RZ;2yLIeUz
zz$bTat&D%nCDDe&yTG+NiBYd|jJ<8e8Y;r6+tQ$2;on*JY-O_tml&U_!wuux^qNiX
zJM2l{c=*O&ikC6DI%g!V%66VS3gw80@6nwxT~)}s=X(xR!b3maFMGs;zg(l`C0`?&
zr1nF^f$qQ^sPobdNbv_rbW83bQ&)2Lq71L2rvwJ4w9zC|WfA4#1`4;X?~IH6WT_&C
zrJGw%>U>Vk^a>@U`C`-9vsQ@8NSLIx!v6re6v2&TgQ{+wzrlCaOAz)`kz?CE@vdza
z)N?MeaC>ftK`4f(43z%>2OF;lcu+jb@RY+D=`(iUt-*s#6|kYSnF_{U;@jynaD2(2
zV2O2M-XgZ0{{Yu1(}YOjtd9~nlMDkBj5Z%72r!FtU0uzZCAM&DJr28l9Xk+}YRD6E
zmPE8cy{g#NHPoqVj4&5N^WPx!0D=4fqI=-NcmN{+FTQcdzInhy0pCFUCm(+k*K@QMAxNc*2rx5J!rJZ?t0nA6q3w|coQWA$I${Dfwl^(0h-B|)i3;SG{4P!lU7JMFxlYDCM*5!^KNr{J
z^D^0oWNUJ5voR+e<%86Fr-%|$rT6=*sNHFl}Y8B@p?UyiuIx^eT>x((Qkk`O8@$KvVNiNQ^
zbUk@ojj?42!Qf_0BKH7j#w}Z|(_%zo086psBzT6>c*COxk`u%AV*GyXyXqY@yYcQ5
z+>ciAWgjg23xw>m1Zdj&BxGhwkgR+H!RkL2dXRXF;%7UP
z>~_t1um)n=wgr%QDn`{Wt2O9=`U{H*9xZnX;bScDH+nuM;w|`ZoqDhs5R&<{
zPu06`z_MNLWSh0K)%Lc|HV&5jLZ>mq6Uha4CzF<;*O@tl_v2+gA$o@aXq?ZEEMA~`
zzGn2G?z8tP0f5#C&inDzvaC5R%iXQG5kHIz)Ixh)
z;d_^g@d2g8yPwJIDO$3$Ip0;IvcX
z0^Lp)mS9PoT>F#bO1W_znqW41B)ft2bGhR+;_Z#Xb+~^20Md3|ABZe$9xbOgaW&H6
znL-#tUms|Y0SzC<6s|XoL(Y1&Sbln
z9I}QIGyMoi9EJRrgI;V$CwCi>dbcms8*p&XvN6<|9Kr+F{HJ#Nnc-s%F7~n=L1nq_5>PFp
zjniikk_qHxw`Q_*iBe?V?l0U@KjR^DG7=8}IOYS@yyPc4Oc8^&S;d5mCVY4hc!g=*
z`n&Ulami@&eZXvko?>`I`;mu-_dN36^UKn}_1)`V1hV0>>u<%I+_m2yFR39TyJY&m
zcIQ^_J_lYhSP}V)iIA=C>9Y@GY{pp|gL1fo8yA8mM9Vw3;ual@L*q+kJWt76@?Qc7
zT$smM%z>{{9mTiqQ*^mL9PM8GI7}Qx$g=DAzfkRt9L{>Vv$JnGJ=t%L%#$nQXP)Ja
zZ9;f_CYs$7sVlJQdXmfP&m*wLkg+9Bt9Qn>=K>Jl%0F@n&~!m{>cP
zUM`_J?hbX&h~3CzCh`|T+int!@t)as&%uc&4(vqpBFDcctE3%;Y#EkWV)ZP)cXz3A
zER#26-({K>Kzp}jW2t^@-w7RfV~(KS?xUG=iB@$Zcf$1tXOk?wNh!>4koM0%mh2Wh
zTWL8>T#pt>EcGD+)rTwqPj-070$U%u?oi>k!3!U|B#)#%$qS{E*ngCm#&B@PxJ=t*
zoX=P844;In3v+_V>nDhp54b;8KBJ3#?E03xerFq!UCu*jbb^>>cj#IRej$8*VZkleCF0^*=CVJnl~{nm}0u5Y|_ge||y=
z_!|+)Hd~R+mjpd*mDWhL@B^inr!kzk!>1pH+F9^J5J%`19^u0g;PO~Bj(CtJjcyVx
zmx;l#o`?`V*g-$S2O)6@)?0L$;%9wY?=TLJ1QBzV9s_1Jd=SsVOQ`r;+An|t!wPUV
zAS7rYOFCm5H_}_8bJMxw;4P(RsJx<`Fu#!PFATZH4ZLxy_aCZuqkAi{zc*{M<3iFQdm`
zT>9Z$WY%`Gyjp+ST4q8vbGS42byS6x-V%{Yt{ZEW{XFT1%b~*3Ie8FU}+tChDPFmqXtU7WaN#5
zj^K*xz)x=(0rMSM4v8FQN3wm1ClXQF<9`6M;!X3H0|(WK-QR`-b>JSdb1w4QwsEXC
zxQ@OL9`<9C8>!}Z2dFPryfS_~Ws{uwFKK)2F)j^;dAY1PmtPwUo`}x9+;h}eSWkp@
zA@973#t#f!d$_mC&5UMGh+8k;!%togqo@|l
z@WY`ckuZ43A+atAeX^~YGHH-_XMqD7eaY@_eP9}R+Z@~nFZQ$^rMBM;!V8bbvRWJv
z=Tp6JGF}+R5~Ro{+r|tzv}Z{TQ}A45xLS(bkm`2d3H&wSWU)5byIQg~nK`o{9BxM0
zB7ZRSdGWF{nQRUV3&o#?N#&49bJfg)v2ynuTE5Wj>dym?mYJL|&rp3O$Za8C)EVyf
zUq}NPbMEhme^MU2Z;r2m4YF6m;Nn(T%0&E@!Y!u@adhqv+i1;faF`aH&YhgT8F2gp
zJY|Xkks#tS@V$2Qqx>KiPZH^Dp#2tC>HWtLgsJ-$^W
zBbPRgo9wrHm(};O?`;D)usE~ES>s`p5sw)YK5VkfEQ2hvW?2Q6StXtUWtLgtxonUY
zcx9GB<2+
z%jMu~Ks`WvWC=smFH*Mr+Zdaer`ZZOc!n&$&-bAd?)tKht~|WI
z`-^=`IELHCTU+4>aXPT1sj-gjBPL6FAZK>wSCq&etUM%P(Q3q=lHIdp9x^#|Zx~PB
z)uzbXR%~>+AM)iZ=&d*I5?3w@W{gMMT8k4
z8EGcmb~W+a00oIVYmDa7bKp8d&Rkv|gzkDu`<}mWo!~-sIK0N53xkNx%x9_UIbsM*
zB0#(rCtS!(VLjcI@?mjf978)~_FH}2!=9xl69feYE}JaAH^OwjLQ!t*cZor1g$eF*
z+=plT-sD>R%cnN6lktFQ4r;;XdXx)p^X?nRW)qjW%!CWJ#v~rDPX~tNaLk5XAmz!K
zxn#mx4x>B?J4yKv`j$8?PnI5?@vPYyXKXy*+W!FT@I!>V!5=C5ha;I8%y9oBA{{YK_WOxfB&U{DRvCK~;#z=rmmr?5-(s!E~4}rvqq7a)?*XWmq9YH}3
zV%P(-!yQj0y!7)J;JI8`c|4sWVB9Cc@a?vIp5@iD%ugmUctSpNr~d$YljNV-`?AYC
zceumg9}w8drTa%$2x||B?{|5Iox^hV8v`sb5x5C@xm+Iuq=FU-2Z93a{Er)bqywuP
z2Ur_of7uVrm*>D|z~SIW5!5a4dX9PUgTvIp!`D%QAaJzxbMqvgID4C)Y>p?nERO^o
zk_@^`J{yI;i?qHn~
z5bAiw%Nfk_xdTjLH(Uk)5ky2R?TXiw4@
z349^ngP1v&S;fn6Bd9gLH^LZE^KLyezU;lWK4SJ`n_kO_q+2jH1?op7Vd-JUH!>rJ
zGfWxBeRw@?iFvtuiPwX-e*)tPyKtI$&j&mgd&p*4obdykwqGX7EG})oJ{Wo6?q75r
z)tz}Ik5?;xu2N%st_>m0!sN*hBzNVI+XKZ)43N=Kr%o$
zb18E((#BePnC1xsW&q$rpGiMS=6q=&4~wlimTj;b0g*7+<;BZ7U#K4L%=IIzk3z)a
z6yn%G-tMfOPPv#Px$DE!P)tG=PY!MR;>_i_U2U@rw+IKT?2hh%I$a-fH;x$*yJ2v$
z_c+W(F`MV*Kjafv0kYmMBzv$4gtttG;T!DoIqv7ic;hvY!>15uHZb#NdAHN31@WF3
zU9&lkWdbp026!$*axCY9g*LckW%JT)Evd@8Sbd^$`0d
zffPHq8vxCe^F7CK6T^YVGbgz0j}0LMAe7eM2M-p$;8+Jf19_J&Ui@nuws=GW>J8`d
z9%Gl7lpS*!hCB{Bf!}t9{YS)>MX(MdWF{QuP$N=hK0-KY%RA=8a}Lh~mKXOUT*pqQ
zczd&vEI)APv5aEkPng#-bpdj?MjRIG&l~Y?jQnx$y91v8042j%>N)=a
zR~qUkUReDoH&KTci3Z`07Y7!EJ7k^*JLA=01(sNAvC#jNu
zmlgpP?s9N5zU~g7pMnkcl9%0|;Hgexr
zLz6Eb@&5qr)BaAl~g&co}&}*R{dOU#IwVeFY0oA
z_|>hJF<6Slpvli9^@U^dGPSh#BfhB$8z*WIccPDEp
zlW~jz$B=wOo%vjx!>!r_mBpM~f7*LPY|aZTCq3I7Sll|>lLd#ldRPoR8v>Ev7g2f{
zek{4$PjS_-@jXS@F&57Wb7h$ulJR%r^P#h}vz#yY3aaCnkJZz1YZc%t*qW+@?zB#C6Y}=ZH7EImvoF
zHru8bHZcQEU{)B_#OZ`w(r9{E=ELM
zrOr#0$)Wp#n|{1xn{l2n*s)^9cx9e42y*Vv0$Bs++2diJxVm^;A{>ps9s}kMEp71M
z8yn!KbEq=kDYj>J+v2_(G1>lXCbGlo^{3#xdB`oYe6k7RIkP3Y@}4`g%O%4l#}@3G
zU3qzgE|z&4Wu8{!*4uvs-y=^9vC|dkcOlAgY1x+#P#c15vdN6WWVyY>-A6Mf+mR`L
zXNTj5a(vF`7JSL^1n$W8j|}P6zdwk#`bY3i8~x?HX)XpF?)qB>!Jc+apD6l=!b=k}
zb-6C=@t#KV**-JQS>-2#&QFZ+c|Ql6ENt_ZS>!(#z*%LUa^D6l@X0K)_mE+h%MSi7
zJZ;=g{AHhm{Q^8c(0>ANe~+&Z;PvJF1?$7z>N@aVuHS>#;CTPU045Lt00II60s;a8
z0|5X40000101+WEK~Z6GfsvuH!O;-m@bUlJ00;pA00BP`tzi{3XK|K$L5Nio5BikF
zj6uZog3>Wv5pHy0mUSD-brf({?s}eLjAjZ8x)*+;8)gfOLU7=yY}5(}_?hOSZ#jU?
z2iy>_!L*6za=-Hv!Su>qf$k|xOZ6{$+-_#0t<>#c9LiQpi8z6*K<^Q`sMAd6AO|A*
zg!r1MYNq}o*G?hhaCNBo<{b#a)e?YD<_IH~QHB{V0#O%@!4xCrC~nzg*BneXs}g`W
zS!EdkPF0OH6yvyT2qEq{c~;ysI}8y=-#KE`zGNY2<5AF*gl3ATF*T)JUR6@$a@trV
z%aO&k-#{xQw=G?q?>OY8b+C*hM@Uz~Sa!bh6^%tiqtA2A(*S
z0Du@<9Bgn4ADGx?D?whR1+HJjbJVvwh1PIH<$rNN)8ak?@t}v`*gpg_>{0`wlv_m
zQ#B32ETD{1SAL*aMIFEp3rn~InZhsPbz15@iN2t-n!S@>Gt)3^%D$$YY^xDUks5E0
z?q#aW4HclwP1ZpKonGN?fOo62vITgl!9xHI7wQ7(lw}N%V;`8;GMY$u>LDKIQxRL<
zBOS+x@GM;64q%0FJwR5jV(TCvQKldVGQhkh8KfpUX>0%ugL_zJ;-3*4M(=fqy#e^B
z_{#AIJl@oYQ2xn$jQR5jL6UAiuEnvJh%1+VVUvSMWLhap%#qu&kIZDP
zNXmntMqPrunVUCi_XR-;=6sT&cFiY}4pJx7)LKH1aaetvOf|4CWTDbrEYA6qy`J2Gtzx+BzYtWz?AsPgZ&9I9eIk$AR1;u(xqE5j7v^?OQ5b@#@ghnky~BA0G{nQk
zNmPYe#@89_FcT}OlOcgXWS9;j;v>k^R0GJ$EK$vWu2Pb$V64B%GY7*L;2px@4VU5w
zb$%t=iB-hR2H*>37{V$S=4?^yiQxAoOI3nZ`4apBxrQD{3kmTv1wBipxYTh1z#gM?
zUS*0#@fTFY?mHoitCe*RO~UUo&VO>1+u|eJ+$n5%h-D225ZLBb%0kBsJkSE9BeoMk
zyLybLQ3tQYqLcAo%wQoLm}}e-YD7`4_UZIYsPNi;qQ#_BB5vXhvtT&V`{DqxO5u+(
zBDFKnKB36{LD~6_lNoZ{JAd_t-M(v|h$aYz8(&u)u!0RjUSO1%62q9jpK!D$E^eFY
z=4j~oi&z@)!qG-G6<%0{=Hjl1YZe^fAV@EnY5RaJxa8ho
z8Y43+!NegY3mExBU!+|_n6$MXBktoRmKD0CVc_ya@Yar4t?A>-7T#StjH=zKXxV@2Ezs2YoSM+$0TH${w9m$Z2mHgdAM433nA*aPq;u7KLk#wx21K#%`{eGW**Bsf7h-6^R2@sknlQIUn%jP`=>lZ_GzXlmZd#gcL5xuV2)}V)ViH
zmh3v)^Ft8H^&_6B%%Uome-J5)x|TS>GgX;?ij7-bMny`sYBV%_%TA@%M^qB>*-=J1
zlm)!Z^#oCuP?{ECa1#b9s
zh0231Em{fV6=Ijc6F$7I8)r+Km9D69P+B49%g>fYa`hcVhF0l6;KkYwD7`$B&%(MY60*v?AMZ|2`s_Fq1eGp
zA=GLhU#WK{Qmfpjgb#NJ4M9&Z?8>-ZnZy^xj28)TKayNy^%B5U@<5y*BM>}Pr#Q$F
zawl+iCAFBK0jQKV1XMmkH*|+_b)^$Fgk}?zroCLbLDe}W9L6l>pWtOsyyM(n*F3W<
zHRB7F3iWYY1u3mciXg`vbtfJm4?uchD|a6rU|W{&RVpD(e4gVQBdu1QAq(pgQ8wMG
zKB)N8JfV1SLYYAE%Ea>`3{yBLh0N@ah$^QVP1S5cQumSFV;I;e2Z5#5Y
z2+sSEj+yuqls{315t@b;2=l`#uHXsW21~O!hZ1=L7*d-fDo&9mu(3->REiiQmHFhJ<321#%0f|e79-uIpgFp`wlsAcVbyrihL)ht
zipv6%4`k%6*@&~3xB`}UFz*D=j8qgen5?Y<4*^x6BWNkK$Ax3d0{|586*MtwsyJCa
zMsa7TR40}-dncqtv1ZFDvGAe_WNeJhf~W>db;d;6C;>;f1u}DnJ8INCz;3jOEy~dL
zKl-Q=E)KSouUbKEfv@rBV>LjE*-5hd2_*Vic{h
zGa@X&n40{eVx!XGg{KuWZ@QTn(LLduc5am|s8g~g?B+Qo+N$*}zU6kLpbL1+7$?BO
zDVi=&#}TLsr)B`#PUG7bk~x8KW)H+ol@H8*%X^9pgNfk1F{cCSBIHi(_<-#Qc!r><
zTE^gkfZPtH!{EN)(v=ytVCocmeBTbEUh#Ox^#Y^izI!A^9JtgDFVVZ`#Fyd#;N=08S%1R`Gtse?P&@XvU
zl@vzgAo&0Vj43K(BK(joqgacw5bQ*^E%+guT8yG!QiXjkYXa30l_oUE+#oKzMKHZ)
zEwx{Xu%-79+OTt&2TU1ZHl`Ce91-rYO%_Uq>IP-a;sg`8<-rYJQQhuhJqxKtP|hjp
zv0vbXz?piKJ0K0A+y^nk{XqsRxDFMTL&=`)4$qioJ9&f+@eBnA;wvikD4?G)vXl<3
zi#2(Z0tjL^J$+qNF=b^7DhZ(ZfuI~-8;z`bOWf*V0#+8JTIW0n>|
z`J!c`Ogsx*q%$O46nY~Gw!)xf4X-mK4`K~T395${GfBbbS)XBr0kOvfs4=|e1bw8S
zSx#0YS_5HA2uF?TJZ4!wJNHu3IX*1ZOR6nJ#W*7)6`Nj3|a%tQp5Vc-MeQ-aSEXFMXZG-QU19w?P5
zNw;yVj=o_$Cx#hVW!y!TS(hV2eZY&(%8kb3Ar3Y7am`9QV5>kml?qGd6~Vp|7{yBn
z2gr$OK9Zl(wzz7do2H>%Kw+s?U`VUI3=KSX%(V&d-{*55$*?}R`le!@VtcoZKm*9Dh
zRrxA)C?j?hh#lyw`X(MS+!4m6L?m|xb#5=
z>A6Y=_$LBCaobTY!ihkmb53893jx0nz8@D9zd&W^DNxb}BXG)djKGaYf8huOJ7uLk
z63U_KR93piV|D|~VXDh=W67Ceyp;fO76e>TSYyi-aD3q~teT6OY`EwcA)EuJfC3pE$^iGqMEcNem~Fc%+*V;~mSQC47Cc!NiS
zJb&z?gJEVmvrvzqfqX(xhn#MZN~kie$>Ww^p#arKxCYfg=W&Pu4wBqerL!3{(MOrV
z{lZ=gv9ZOA8sGTEEf|MH-n{D-c7Emywe>RdJ`fuJ01>-PP}3M^!*Dr=f&)y}2y%+v
zZxL)-y+I`&_<$&_cNPE_s)@A;#0{#hVpQ430~=Lm=z+{V5U>=)QRWZ@OI(vGmEt)A
zy<$Rx(Fhp&ggHmlQ^d)Ojhoa1d5AP`Qrd)rRs_@BIFdsA+@N9Zh#e?YQy-8cKxw3e
zoWPGH)k~^{Z58SuI1BMBpdw#jtV5LpETAit5Mj9>TmZ^*?krd82nJr|5Fc{cNa|p}
z@fVd(aoEZRobxJpiPu+CRi-yGGT#1Ww>lVR*4qeT;=Jlv0eFN|`$K4gZ%hS&
z1d5{UW&Lv~P~1y_07nxMVNFUTuF8dhk{=B~0BelE;cJVc8mT}S6w4{O%QQ?&5wA5J
z+e4T8jsPKJoZ}NlZU?aMJ~K3-RC>;Qw*m}Y+kG)O1gmDyM|B?K8t!NX!;ra@P3eU;
zlhmdj!4LE@tjDXEATgIYiFE>26%@YB%1pK)A4l_usk1}%Ew3;JK)6QWq2R9G*yI7B
zwp`A`t&2kS)TJB^kxR3ei~54CR-)SOVnOmp50ZeXx_IpJZUalF09$Ba@mMZbq|Y5k
zi9_<13$U?9z0C$#t<78zYv~8cUr@f^q`QxCWqYrt8K6E~)ER_dQTl^gA=&YZy?V?P
zB%Ke;pm716d1i9I8HsE!h|L!L62uCtfdsSh@c>tdxk`|AQI@h^qa5)R%?_K6bX!Rb
z!3NB9AIc)zVk;mxhTY+X{e-A4UKXG+xha)&RcP#js38jlmNl3BK}%nR4OOz8#Yayy
z$r=W|Jdl{#q-)d_wAoA_A83}uK4J1uiZEZ5veFNTH2Gm8nr_>Nub!7P6EF
z%(G~&mMC!^GXUFT)XYf1Y>Sk2_=TUagtdH#B(eB}We1W|pxu^1M&HU?LeAN6{)DQ_
zXNZ>a8QT`B&>XNrY2}*7?GdF9Q3ojBmg*^NG^m47E&X{Dxr~
zn^Yq&YCJiO$|_~cmEQj{?w4W5q^byMF8D4&f1Y
zuAZlZuF}7AAJiIsZW>;`)WBg+n4vQ}cziOkS6*Ni-X_5%_!T8WUM}mYOGc<1D5+OD
zw>6875)74jhWVz$Wx~8a?gKy(zNM5;MV^c$0ocKW04hj@f0%n-W#yQvgI4sl4h;<9
z-^>LBSHkg81+IZZ<}KK_Dty6Xe4gKIz>JSRFU7#XOhaN;klb2_R2V~L_x#C
zOE|fd75jt0FI$`K{l?m#id3x~kz}j8oDfhEL`pl~;soWsQQ2L9*Y^xdUs=pWMyFwL
zbt3kflrC>YhBl^$M823-ASM~UBfg|Pf!hF_J)WYgt*uaRWB7=`0i
zQ1W04Fhv63Fa%pHv#1CSM--7r18bl>u;?qcACs66f}5(V2u;|C(yKj7UYK@w1fkq0
z@o|C~cZ|B!wc9>xa-xyWQ!K_cJ{>^=r)lBle6hj*0JAl5i-BzoUx@hc81}P9<(O07
z_XH2XzdD30DcIsEsN8fE<|`r}bsYHx(+Ye%4eQYzS%=B)0J#BpW&<{n(k=rglr_M0
z#Hmec1WqijKfi+^*8mLWCWw$k076v%VZdZ0J{K)PPB0OFf(!-J4+Vars!lfZ1!+jVhtUV!J8Cpi@05=2_^?rfxtc`8
z=#?k|Q>|QF71imLp-8TlFZl^{xt`%vO|g>_E`s@qg@@dFS?FSUpuDk>HI2hS!Gfdt
zOQolx2nZM4GB77|9sWaPV~Nm=h%=IlDKYdIK~&K|CSg!vXeBS>+ZRBE#Y{5onG||h
zsIYGhcN21ZitfqSgm*syEV9t?C@4l~F{cpkJ|NX_6BPI%{{WDJDvI$eN+*Lc4TIFx
zC&ceyk(tahX!jqn=B160F@#}e#V}cnANFzeeQ4Ztf0*)>RV(y`I#RI*VL|p_q4UIZpYaZ_)Q`#v?lA%@hlZ)uK{=_
zM{)+4eT_)|=1>82_Q&uD8X60KgSJvblFUtkx4EBn4F?ba7O>RF4((gF9KR5bY^?*`
zoB4zoz`KYlTZpFPD`C4nOt^N}s7No&xWzSlo5kK?uHH?n{{UsQ_mKYpnur9$%(c1_
zS+KjN5V>DuKs0y)24zajG0ADu#5@%*#S;9~YPYy+g0UQZ%$P=$@<-kr
zyLIXfw;PW4%qthXzou7kR*$&gT3oRuJgR2$BC~7GW=Iv4{{ZS;stqfC63C-$@z=Qe
zre!yb{@^QHP!^S6+$-?K>m4yt)}1kh2*7C>RUN->&i5L=V6GkYDu-`!pePhvwt|eE
z#l*RAC*k!mWnP7VavEE=$z@f+Qq;XmhA7**cL75;LO}9QOh$kJFFc3LDuyHWJ&=cw
zaK9Eo;9#bMA9FEcac~6D#vuy%(Eus7K^+zmw>KYxEuo=>2BkGKYgYZiYK6SNmL=2C
z`i^QX@pC9w$q5ZFxH#RtOEqi~+hN;bcf3bZ+kiAeO$Gs_P5@M>AepRl+-S<&-c>=p
zGmcd(CBdA=L_p>^KMAUl!bQ#pGNc@NxoR7EbC{{Cw{hl`;wWpo)Zn+ts7#`jM-f44
zd5j;XYh8R1G2gaW7Ed8E5eJ+=9hD5_V7C>##pry_IGko&k#b(37@M|5f&fwj1_rdL
zMPqXFHbvyH#i~X1Fb-LNxk8FPOhqr;b##=dn5{BRB6P>AgUD_Qv}>5H2gFAU9I(r?
z42LWFKP)$Zw~W>ya#HBKe1C8$@Xlm7JwUh=Kx@>(57ZS~ZAUn&0WOvR-M%2iteTLu
zCo?;sa`<>47zI+UeHffNLSrPYSQY@g95E{dx?zTmTq33249%y!bLz;HZ0i#BaZcWI
zcJ5z8ZxL9nlD$As5{3MqGO$1arp^BV61cmT^
zT204Nshb;u7VR#iM(Yf8XZHcDk<<3Bp-?-PrdHFUMQ3*iA%-a9+#m+*d_sps>~mlH
z7*GMt_I%1E2}{$-6=`eT#TiiQ=2&C67C3}(d_cORx-Dm<<KQUn!=@irP
z0T*%et@Rbl@`-LlV&798i$xDL4p(kuwXx64TdW$i(x|TU4X8n|rQnA`F>=>gc$D0r
zSx%)*gpa2|2X*O-SU7{R#p#rk!5{H`z`g>*oc{pT6d4K=Rpp3kQWc{++#;%o`>CV=
zuAtUgmC56{HlS^10Ss~vUAqwQO%-bs2&QBQ3?x+o#5#0tV#?0VJw(Unfz+ZPqr^fo
zZ1CbUuLuanA0qB8O^EUne&K8q-$>k5a({B61W>SH3LZxVK>!XFr5ah{7``C7(Ureb
zA1`^8j4PwLUuChs?8*T5;5?Gq3$j2>_b9XO`JWUz{Om1Xqu+MiWELAygnQ8M_$6fyb-en57^GsE=k<$M0&igK=
zMXu16hhXHs5?W*ljUY%U*o0qDD-b+6gDJr~lvL{bm5>~-bxi-BZb0RTYP*>4b3p7?u-|LBT_9wOo$aU+;JaK5JFYIv_P@}S#brt
z%3)sBdzfA~jGE>c`9CF`L#ttN>I<#Uj?oItDr78UcMj7wkS32u)HYEqWmZ!x4LM==
zQ3pk|mG3aOmI6c8o|71Nj6@a9l`$kthqw!p@J~=Ucj$<9m58I%x4XpWCr-!A0peLa
z5BGA`Qnq9bf8OQ@sDgm^d`88An`%6D(c&CXyDcOu6$j=59D2I;JY3yD@Q)Bn0m*y0
zN01nH{{Xl32B^z__jCTL*x^tDn87#2z&(yH6%zq$;6Zi?Z{lNqVRAhkM1@|jpK%za
zwSB^imv?%Z$r!p!KXn3|8tg!~ZE;*(A5uB$9GVY;IHB^i)qmX49m)O>OLs^OFor&PN3%C&IYPfL5PgQ
z45$*age+>M8Xp%qsQH39eaEFI%>LnskzmE2MpO!4xn@FqM1~X0O4Gikn9or9(zQwn
zc)3Yu;th&EmH}5+6&CSpyiAq$;ytheUKMEWYt``sXg?C|C#sj+{lpGR@I!-NF{#t(
zi^3UEWV~Sz4Kp+L6QNI0jy{N@p=qo{jy)KhkTR>pS{)4&aTr7cIQ&7@m|0vx003?m
zx&a+70LtKjLEs{q4q^qVnQBN0S>ka+Dr-LA7O@4YmcT6>u06~Y^BGuv1*%o&_?VM@
zbjSFa^AzbZh83az0F6MXDvo#pqEn`}MDLYsW)-N=wks65Kp&vIMXDf2f&~|2=?oqq
zQjSjMX=nw^rt)v*RR9B|du}h()vu);yXsN={=+cUH_0rl2C*I?^M?>IcpO;?DW?$u
zS$A+T3B#r}4J;$4xrVnWt>Q5R^N$7jl;L+A`G#`ySba;N%+UIY^i`QsDZQlmv;QRzO4NggZ<^WXl0m@ck^Bv0CQCQ5**>o@!2K>Pfwg}y?a;ZVgIk?G;
zBr|r@dg1phmU?4)Q~|PF#XDon)=+tcdppD}g|BlW6|tIRGaE;!w5q66PVgQZnX5BL=cCTPsifrppN4a&geUuQQu?6WaLyhma@pX$3YR=OKKqdhRSGa$#LKuMj0;#2^b6s6$(&3>R}-n
z<{WaBG&OzZQHZeJY$mnlGk~`15yn5>UHXh|fdkiq+$t*IVxT`=InK$FQ+ptHa;M=754_0IncM-B#z8W69>eoQgS$flq%}66AVt5+9k^p)>)mvYUP9i;R6Q{u8mB1e38o)_>G#H
z1}nr<`=F}c5S
zXqk#wm6OCarD~vzD5HM^iJXfd~b!^AL6xx{CN8FKl~(cy47Ic@Z|&(fmuzbt#iHe{$iL
zn;%R}aM(~$o0lP%;nOp7*dn~N;t^(6RHCwx%nfeMU0eh#y;NsHC)~P{m;SiQAqovK
zGHT?23$&t2C2h;KcQOw+FjRb5#hgF@9+Mlys__8>HgPzZEfEFfIqqcHT%o~v(_Upn
zjgDo@7u6Ug*r55SfB-$cN{X5$GB?2}?k$hSSX3(5}@RBp#8X_%niakD_(SyGEw$Imm_&(8RJ+&C2%5
z8yP?dF;mFL0Yf+6E9Yd3*YFC
zMI}k+mNXHY$&G^3l2m&UH=Xwq*>>L*+(99B*95_R@HVejM9yWJD(N;kmp~HAWu?;+
z-xc9*U^4=WxR!#zyHNq{_Yf^rZYrD6t{D59w3Z*I^Df@P(5rgE6cbG%x$;}7p23CV
z*ZPLFXNa4(Eqj`J;dPl_#c5Q2Na?T5AeIHSh2|IlH)_O&)y<)JgH%ibdrV1yL!#Ed
ze2`{qS%2!5s?1^Dp?L(sjbi>_noQj>4yYJ09z78DdU-hv^XUJpt9s7web}p4Tk=sIsvMMk>c4jQ+>x0d2TN4
zCwI(2R(;EC@7!IB9tl8!aQ&oi?^4}{WH6PBTTWxK2LY$*TB}wZ3WH0o
z%f1WMyhX*A^Di_l;Y>gT2FBa|TlV|u|QoAl<`{fGQ$gMw)tQf0=wcRQbMZ&GJ%=Z
zaRIA${E#?*pvU@!ODSI!EBZ?Jh__YL>{XbEtBGcd;AKIE++rjRiRKtBq+$TEO!|d(
zYW5;kmKDd;7S1@9l)8au4aJX}xMhaCOsvANY3?1Uqn2f$Xgm$>Fh$5S&ZQ}NtTM1I
zrVlchfclDJ4=-$ELY<@;H+muhrxt1@?*uq{NM*K)Hxh`Z`pm0RfF5~ZjqU??e-oXv
z1|k$j$g>%FBh_1(>l~yqR#Q2S-IvZ~kPVfhUgn=sD+lERf<4MjReFxGaEdb(S5%vm
z?j2=#g?E$8rGKdOaohwsGpGlnsfq;boa~k>2rOu?6ND?yB{y5d3fy^ss=*b2MJ2|w
zR~5qiK&5`7wio6l0uC_8alpP92O|2oK7Pf78$YF1xmk}lU4N=
zIx~<3$_QH4A%M^>%9bPLps&&(%~M$
z0b4uqi$94-4UaW1fHr2~Tc}6?TCTbu;}P^7*eY0`p@8|7+y>SBMCmg?vkj+ce&*A|
z-fmn*X^bBc0I4mq^Ai+RZ@AR3kT`+`Ist|cnT6XMYS+AxxXeqziIQcPV(~U6#Teq@
z@NH{%#jL%=N(_%vKSf)ZudKlUDoob|cw^T9>&)j?0~{~8kQmL=#B!0t*Ze>x#2D4b
zL10c67O4ZvCobi~FUq1-EfmKw56K&aZJAy(xpWl@mh(m>)Er&c#IeBM-k|{%e+w=)
zK4tM;nGDWaY;7O{bu2FC(Wj;fX@Qwk{6e**SJbwoREGqlwc!QaE{Lqyb2$`V5!m^0
z95aF-fXa39ZYcgo5{*h%PK%UElo~N`L%>nN9P1c8czcejEW48!UAu`)pJaZ_cq3gX
z^%2TOB+TjV2}?t`1h`K|F_LG61~%@`DlJyAv%E5nv)oeHdgcMS8X}ehFnL5U
zaR&l?#7M*4##{<}f!Xs9D`U(y6gVyrc6*D!KFNK6jWpUUS%eEKxrJHi%N%x(aRD>P
z%*z2f983+Fy1JMZK4o0cXuV77x?-{$nM(ae<<%TiID(?>X*!Es+liazh?t-%y9}$JmwaIuNEGVFb|Z!+$>qTEnZrbf(|hBSh`z*q*RHVSI8e09})Zr3NJ}q
zFnE>T57y>*oXX4eFdFrN$C1)YrOqBlnZQAP*oZy
zg|`tmeFYP*(uG8Vs6GO=sC
zI6o8z3KM2ftFA~^9jf#5D+-%8FHaGrwQaTTdL{ugZ*WdnnDEDyBDEU0B3n<0N;NFS
zN)Hi$G{lrKT+3E!5p`8CgP%}3C4m$%no%afUo$L%19f2PbKG#X?i=}b%W!Y;E9LbA
zD2It+(RIY#3y#DEqXf|$_+`<%^A#2Qi0J*yu%>-nGN&G*RbY&=%o?(Dm{9>NHf}08
zY?i%TUr_-Q6ta}Co~27fP{pk2F-Xi2Em<#bA;wS!yjpM+Wq0AhG{hBO$`
zx4vbOwxWl$KITk`iNWBQRM<3(PK?eEZ&4{1puKs5YA_r-SaJ=S2Rz)f8fe$zFAsy2
zzB*1K?RJ>kO6nF)Y)1ZY%UMd#;pH$AZO62L1yJjV2iuLNUSpC@=y>%HtO&Q5F2%a!
zxpvr9*x&URvyzi17=EFuT8ii9RcT&RpPpf)*fMta0C+hUwTgZ13E+tuY7%Xju
zaj|mMS1=8aOukkt)o478`(^vWwm7J^a_;5s?ySsU-p9G7kH$RRcXRBO6En2mLVI|gTzfjjVTir`>LTp`1a5kFGW>T*?J0ns>2VW3GvsJ<-(upS=_s>B`Wju#5fqYKzoJ(NTv;DS&iWthm#Cn
zh{wpLB&=4kquMg2%@nL<{{SEP3lhW_2*4QZJ6-|Be2g#@n@tf)_HC%B$Kinu+l?Xb$vmM5F{^!QC0xEabNaoIZtF=KOeh`pbOATOks?*
z@hqrO^k2!iP)#3wpUqZjSR1hVO`7T?dQv
zC@h-xBB5?upNHyPTS(>Id_|ib)<_JN+QoFxs|LdXmDF~L!SNmZ!Y_JuVY~OZRBVPj
zh*WdP>C{GGXWR`YU|VXt>LAs*57e~F;2CWwPGf0NP0=;EOISDkEvjUvVE5
z1FPc%v^~SRcsaREJD05YT9w08Rgd=^v01}`fS}2>{w0_-Rbt;Vjbbgs+-JmXa>F91
zqWFy|DphC4a}q;IaPHC6s>$I8zEphrfTGoA$GKp#QeS18DH4S?-#
z$xO9!ohP`n0@y8hAh-t%aLD{Yqk`(ZvX0n2bp`w)7nadrAk~d=4Ok>ANaQouaW0~x
z1@>#^52!S7kOgQS*h9p66}o<}?j{QI11oK9OUj>7lYc_MJA_=ixPRM;BA||)rb>=(
z9OJ~N&I{&e^HWZ~A<;@;DL@5Pi`EGD?xD6s0axYQIS~jHV;?{2Cc4^47HS#60U<$%
zTQTg^pfB+e@be2ox?@)cJByfB!nM>nOCUeLiG@a@hH3K~@S{*1gnPsp2Y_W*!tXKS
z-q`6Zw~BWw(;Icq@
zh2~(m3*VZC4905eH^>yt9J0@0f#_!j|1$
zEP^@U01=Y!XX(epxbJ8MmEY7|=S0YKGS%2!56>_bMK~|?_XD{*
zbN;{FX6aJzSLBQeEQ4>_kT*nFIVHUPM=Y^eOq=rru9h`%fNAHLANRPc9+}KuTgF%-
z;Y#fi&M!Sdys)-be!0}UEAvob+X%m?cmDtqy&?>!p8o)emh(mdAA&SM)%fGYLkA7q
zOGmTBvZxpm+@8cxwcai&KAgpCR-HSRCwg4Ih8ame9X|lNW%-^VG8TiZrB~w&!G^hjR>;YWr@Dy1XZW^Dl@H95W@!p
za@JH=Eky=8OmHU<#caiY_UEmrIOOkGL?-D35)
zP^*z~%)F4(PwqMvz=!c<#K2=YF5348P+H-?sYEcQ+%oLsy6#-h)EorSM?Jox(iLEN
zm0N*d%&@Evn6B+i7fO`Fu4A81J3{Dtaq|b9)d6YVGKpRwSC)-qs))`mYriA8e3*@r
zg=tJeGy|HBa36yn_KuOcdrP_%V!y%1HY
zd04o#D7xq7F2#=qBKrZYa}yc?&A@fhyO+Y-ikk<5JQ$`fAa#;IZ}Ak0jB0qekWpfw
zm*gwd#8y4TFe+V&S&wwBed;*flD%Kam0kenq9yyaXI7^hfl?R+`%LFxSNeZYE}CRL
zN)}FE;9?Pz2`Sg0^gQ|bmi??FKE9(XO6J*7Y9n{dphn#DDMwDbfwz~0a~zL2M%8Nj
zmnsE2SD(yGV9MQ@e7r-+OyXB10d4Fh&Qj?}0{KJ&9BU70-|8O~Jdclq%tCRddjT^G
z7C7Z}`Ik&5tZgx$s|tyPFFi0uuP=@SMQJshE2!;FokO~`@snc$5*(N6AHB4zcRwtjmKE?av8e~MTcISKk+E-y|((xGh1yY!>l!o
z%G$>$l%M-6^lE`aK{Y26ipKg$%%z27%FU1$QXTN=7uDO@m(oIF<^Vir
ze&Helj%VXO2!+U@FHmS-nIRNb~+qT!dT
z%%z}Q8o3=84+sh%1*4fbM=OtTd$=o5@->-3xqslWLNMS;kRdU>B}c)P(w_uX*(r2QOlaFdDrn45rW(vSKQST
zT@yEf5xMHlI10vn!chPQ?ijSEE(ml0yzC%}(=k!B
zNI5*gkok(Fu(5UUYw8l4&|)4&JzRiL5C(cO%)!HZ7V0EK7sEC%oN|e!
zL*lA0lFLOvfz${0HtEz$B{H9^O3Mp$y-FPK$qHC2S!xv_|w=tc0zWBksUz}bEc-i@_#TZ##Cqk=n-~(q7rdl4Px|Xas7F8CSmEa`STpU;wrKz$?swi_YN{b+ZWZ
zjlhSTBwsoH;k|m*{N`8|7s^5vhS7e{0FmXgI<-K|Rz%lqo
zPft+>3g-9f8d#=lr`&7_(FLuFc!Ek{D$8QnUP*Wy3pT@b1jC@kd%DpeJm-mWDA{Vt
z>0dvHnX#xGyoe&2Dp+Oz05cT`=GRUp5M7X!V^;GhR!{}J*5N41GHf3nCF;T${WF~`
z0Z*PI;LcuC&u_$`82;V&C~m77hgAkvB?xAA&3cxZrqtZZ4+l?(DUqIs`G8(@CMK)*^*fUg+W}2SmE!2^WvbT4^iXr5&_U+zh>@
z1BMkGp{Rx{uMi6^U(_s@b!-)nA1^TyHk-1d)#H{J6#0OZ38BmtTg&#rj7Mk73XY{%
zr)I=CnJl*YWtSmCQlQm6OC82*iP&q5TvY{ZFEzh<`GEjq13)9lRl)|-+Y&mo+Pwr|
zie6kc70-w@YvZGj<^?NMbJV4!q}uks#A}GEyt1E?W(K|qM$*K)gib3{E}5mZd4VYN
zp6Va~a3SM-e$`O~J*n1@|fq
zaDb73#TuKCp^MB^%Lst65)!vC)hO;RL@`4(U9!RzahYo~xR7SXF6J6}joE?Fe8b06
z2Vc-jI1?ByLqfGw6z)_ahQXooeGxni#Ig_8If4%^U)xav(yoAPc|TJ+Y79Fu^pwF_
zlYeoVi<>p-A80XIU#5Ci1Vj%3ZPC#2Fh=q_T|xRJ{>Pa~KpU3+7~PGwPcRw}gc8}^
z8lJ&=%qnjWa?rM*3d(oq5HNsraUCwfOt1>?MybaV+BIRc=!=U+?4l{TTEEOh2&(`$
z)O>!0Ie+gld#ARCUCWtX`+&e>_P+rz64s=3G|gFzxo&wc%&U>uZN3R>WYgEowZ0=U
zahgguID&RAFQ@|2+-$*)%fy+qdx2P5r(XvXn+0r|q46la0=s>s8>DhE-V*m@u>);y
z%%=e~T&qokzaAwF8s;L3Ed5Ir2z|$Nw*!&&00@?O?fp#dK&`$2IhKk`rH^3bhCpg3
znXa&EP^zyiEwKfIPnV)8lPG=6FqIgC(a=^3<|w&jgKz~bdc2DoN5sez^$_Fb{Z4dz
zoTWhQ$yETTKv%z~Hy7rMQmrvqeXq@wvu;~bh_EJ?Xz_Oi33gQcL~CAhUj#97%elmR
z%qh7@T|1jYnB~pf7XSr=c~wpJ#ZlAymUn}wDyit>)CA$fsYP>bU$4}BLxd^-FI=!=
z2rnJNxC#f0i~t+pj#`x}HTjRI0h+mTc2M1a2mOEXp(}u4dL_x0Q|Gb^icnP}D(SE}O5j@iI$lzL4;K7@~gxj^-*>3>WT#j^PD>Y&IGF#9kxrJTQS|
zO|
zap%9oFvKp)ezSivlGB=jS;m+msmL|&!^C#1LLhaTuef={;N(|5!W5TP1WroZ#Y?w#
zuoU}wWm-YS15`f@M6Cw8wwnAiNdmP*9hV(zMpyl~&`W09W!H0QLmm)(#4(p1V)nz?
z%)|=dg>Ay~UzmXnnlJ;z?xW)5WeV29mza#P3ObmnQ$WlZ5ThfA1uP#sejt;rSY1L8
zT-CSKp+h|6xRG`obqUY~xAiGBJ&|LfBflXAl8vg3f22}nyB&Jym$g;fZEgHY8J^$Z
zm;<--;PV$fSr3JK^(%qkgs*5MGM=I>Y*Mch%YstzD&`e>s03^*`GB;FXOz{S2^|fV_V*1WczN+`<$Qe7i}W2LjpEa{yo0L6_8D3Lc@&
z-BpGC)M#yaA>y0Fa||nt>xofAIzL;SP0M&aKy6WqAOf{4gzA+{sEyC^OoDSXX!)4(
zJVjY9uHuC6Mvtj(^jUw1KuwMPK(OP?q5k07Q``U(F*ngAi=l{PLo5MaBaMae96|RB
zU@zhm8~Pd{p8nji8$$UBwtZ2!5z8K@6}$GoqwW=;MjOYuR?=v0oM(;sk2-|tDx7xt
z^8^WH=3=p0tJUvOJEAHI^OBMZe0v|Csksyj3c4*rhyX^<0WF|X4g%}l8DU7JwhRdh
z6K1Fv69TPl<4_2qoZKN^;*7sgB2_`xu^=dSFWV1V9%8HJC1(7I)UhE?IDm|vc!F-O
zD{|bmJ;4IoR$unxJIZz9FCJ9S=2HRmwe^xyN?POaGQ#9^2z)T5J$a;YwfC5aaL4-;
z;cpq9;3AwPo~VT=KwGC5&6Y+6um=j+Q0iC7+ZwFC0_s{^S-(1jheE||#iny97f+Wx
zLAwoZqAJel^dtO6s#00Fv5c2oLK!SY)KIoi?l!eh2<39OO=O9bXxl8X4?bovGoK$G
z;-ZHbw;%h&WVK}c^#JaLFezxhwnuP~g1DU*(`ltnV3tE9ha_^`hWPbJSfR>eI2yVWE;g)@cKF3)y|TRj0N$g-DgxR60DG3$4QM$0
zN~Wkab@AK~gDh6QqGu(k==+3GO=(RUM-MQHSDr+&u61vGB|CL;ZkVMmqh>5#!c*+x
z;713GgYa9=nPZj<+y#`kq7$I_O7OcE6FjyhW?0p{G_eYuTQGoD+gsg0ZxcukNn?t`
z=!DuJg+r?I8*f&xZltVUXL{#nU@E)OQ)w0IcDY{VR6sufELQz#H
z-WjeB6AS>?xJrTTV3Osxul9l$Q)xjcg%_Xc5y1tABsDOR1BL#geNwKX1XTeI8VaUq
z>DvHs4M8o8-!mstZ0c8*{v#_jSRLH7t4`DhQKGy6j06bRZ_*Zu9HzoLHA1Eibcoy~
zFY_*T;#3!?&oU;anM|fHa^WGb+#h*6$MY27s+2pW;t@yCR3`MW9N&HjtX_if9w7>l
zI`=cNP+5A#x$`K27_>)HGb+Ymkar&;j^z!p
z481{Sv(#kfaYcu?TLkZqekIfZVxPD>2NC9OH($A1Dlz*s;-zBNQc|7VD{G=`ReHE3
zSP?--a3NXl3sW_TzcR~1AUhx-AMpc0M+{pmLz&rl{$!ol8
zAH)I|vYYxknFR!E{{Ud4L9GnSN6Z?7qR;!{
zTi7cuV+*}x1bmeU6&J>(liEh(-(FO7hDvCAGR~9+TcXBQhQ@)R{{Rz2fN`9$@KkQj
zNX>}J?AA$;XtQFD|H&(P-7)n+S@9D_W4+}1qVIL-(PGM
zaE)*y@FTl$!q~JimJ|cc^8wu1U9%(*QPB!kSGFdG;0v02f=R-sJ+nc7aBe7Wv|#)|
zZZ(b>Xh72wkf=6z=$B+UMVMWM{{TAZh|rTa)DTibcU%<+D`O4A$sY_0WPnZIsi4v2
z1W=+qPz}NYJ!HZ?ZE*ZXqn&GP9Wp?`?Rcr8Ppqzvh#%o%@h)5re-fi7!wA@D(W^U&
zo)mM;%<8$95fBI^TnF5vGwxWL)rysn(UFfdFk*0k*P4dafyluRJpeLS54iAZeck?J
zBf+)i`jv)?fz8wyD$K7!QJ!K^$ASqFcuXgU5W_NUfUK{nMy;oEu2R@#S#=B;^i}wR
zIYdFjZ^nq0g5d5r456Z|{{T@*%m#IiW2gXAL%t%G!!HXn?4?*;Z}^vhv6J*ZARSb9
z>JwNSwlDofE=w{X+6FcfDCt2)1kev~UD~~pscIY0R>|bzI3%TT(1noX%k@7TpWwIimP;t*d{7oveZTGZGIuAoXo^H$IK;wv6+F}u{Bmj
z@~)_e?-_Ysh?Fw&WO-nKG0M7PE@dlu3ZwH8RdAVPqNz&dgrTbNaqd*O_?3;V7^zs3
zk0w4LMPR`aX!Ho0WN0+MxF;&kcz~5v(B@YK58@jgHNB8VRyKqvIP!57v^?P-?hZFX
z7(8@EYm(b9XDk|{?gi?nlbB?<4Ye(?%M}c|*sH3B%l89zx+SKr{`wuxcMEX(?h)6m
z>!Z;cx)$icA7u_k2F-weJi_-@N_c~(9^)2AkW4%+8&*C-oHc$3JBiVPJ*wO0BbLxN
z7y|ATue_tpJRg~I;?;E*JD*>11}ib9<9-h^iOusIV7$hLNa6$Fq22YVS?`&X7GXJu
z%mxRBWGQ&cz)`S6F!HZQY;i-CnXfPKMpcal%&JN`d-<0t+BX%j32T5q>J@`YX7WP8
zv2`-$&WUw1g)p?RV|zJfSOePXJ)|A6gd04{dJ+`w%7go5{$M--j}tvmEo%FW(fw`(
zEq8};z!>DVSY@Ez@Uq$z=nk%Kp}Z-da=>6mCvXu=yKA^TgaL^X2e6j1$iU)P05kB@
z1W{tLHoNS<#2I$YBPag=B~awjR(Ssa$!yhQ&l4+GK(h90FsfKGKTx(|V5YFSTChs1BCs`#WZ`OM5)*c{X7ddtpG#?2(GzUmVS
znhAK&0`h}sJW3DyxLON42mP6X)W8}K`(iilnzS|
znB~U`1~tyjtTrp!H0>b^3~gnFs{SECwqMl7x1$$$_>SArEG`rw3%iX)5{aV8oU>*|
zS1uI2OkilkK;OX>U<1vc}kl$l`M11deW7at29)UZ-0u44$fbyrMk
zW{(l3jNbDbI_cseRNIQ4V!vWFX6tdKGmzJH9jYZZe^7y_+lBjrg_kO#9H2J0H#hHM
z^#+762DD?m=pgGMkU^mLQ^)0H&
zlHCwWQEabqrKO{r)S!#ZUXEdbjsF0s!q_!zBCooxmobeDpd0YwGvY$MK;=CHwgwQj
zH~hs?v{xm@$X;{IxT^`6Y)rfXbD|xs4FK5yI@`lu5A`bx(APoo{{T?kTan-QD$&TK
zv9F!nRCc+&LxNFGZnZHfY-ZvNAa+CuY#t?Kuf;hbGJ!@HXS;!}Qq?(z6;^M$lnxTW
zznJ8~3^%{T7AW|aS`)$fVWSEFL(0urF;`DoqE?u+}g8m=|T#AKL`TL1%OqJ_f!feJvngoxSxd&qF}*
zTf3M^O$6fO%InD~SD`5OUTZaY%xG6ZQ+z_h1p2k&0!N5u)LxWJj-R#UN<$a&(
zU;+%WpP7^!GI;cH6BLQc>FPBZ7d%A1oyv?^_AnzYAq)l+3@C@iZxZg7yM^bWFR&h2
zbc(ySUmcD0D;;?H%wCGXDcUiuQ@w%uN5;WmBKsIcf$A_z10Fp}3^WTZg>Fl7qXOxK
zHo8m^`1Z+&yW9)}riJD($OoVw%nz5YSQqeg2KJGdV*+?_1mGVK)R#+!BN$aNLN1Xr
z0)o?($8S>0W@L&UqQc;p;0_2Hm9S&~0C|`{IidKJBbVbeG1CNeo
zs4xKHzUp2(;*Ss&aE5$9e>5Fo=y1wV+M}YJQ0fCV6r$Us#2eX%H{ZE^n(umntocQY
zu-P(Mx|lMC)!Jq(Np%TSEA8c%#!slmT#j^)+z`WUOQtQ(Vv|z6;ndEbNt-feHZErj
z_dK(r9x?Pw%IxO-I<9Ur(Nfq}ynIEu5i}GiVbR5ILVp-Yne@!ADFGu+0z$_J!TO!
z=2LV~K&`oHf;1P=E>r}*#rQKRh~~d9+*^Kjsv42;J5xKRG%J2A*UBB~+Fv#WGKJihY(058m_
z&n9$_z+jJaA2wNmrz0{Y%{;&bM!m+oZK0t7dqJq2AhN~86BU1sSUEgQum{9^H*AUx
zyb=wv%oc1-iXBIM3gs#ZWHpYZn%hIFl)RMgTa^a|ZImVhkl}{|EGHjK#F;eyK4VTI
z43SG%Dt3$~k#H>WQ?T^}!+L3V<|??nnR3;r%tqupB|=Sf%=Td>qM$(6CqWRsvW*R2
z`i_H5*iiZCjH#V9`Ee1k^q)Uh80MP@qA)hs8q{klS`+gCrdXw>0w_M1gb?!Nit^@O
zt(L(#86si_&oLziGl7Y-jN}}Z)*S9C!)a%$T@W7x3c;$<3LwO`B}@TLoAvP+Lq*2m
zimM8AVljtj(}=I>Oz1g^7XwQ3E9K@9LR*NYyOuqm4S-AJk9Af50KCgG+p&Z(qKkSQ}WaUS^hARs*$~RJ5-W6yY{@(>Boz9`R#ZKTsG-
z4j0e*fM42V?)Vs3$!aXq?hY0SE~Ynmh}x($MorLmnOdc!?mj0kZsGt>JQ$ik9QJE>-~rdy5;{Ur`(5c+^D%>}9(SNfQ@ichsb9vxlj2G-#Rp
z)}l~UH3Jtg+D(^bcq5d4FB$BYz%({Uy^@sM^5u#iQ(bx{6lWDPl@>1J+gs;QMfYXu
zbK-4TNV0wtlQ&V8R)XrL=`wz!5WH2Im{Ef+Vs?R1)JyINNHVcd`)r#4orJ(u3(@o%=ZHS(Aly{=8qk85
z+3A4grXBG75DtqS@QClAyE>M_8eAl9mw3T|=<)s}gv>B(R}o#_CCM+MAwNeZDMcO)
zM@Dm)n!vW$gXYk58eigiVxXBy5LY&J;tbNnu8UUs%)Azj6-pXBaZpmSlCOJ`<7mbm
z8YLPCst^0jvGKJX4S{Q~%zCmEc^90-!jM&=NA6^*8=6-~IA{{T}h
z;Pn7$#GQ8HH}48UMeD~
zVBRk?eU4}|{-9J`EPGx!m&9|xx>&w){l;$%I1X>k+_6O`UJVrjKbpUkDw8tm{{VSH
zmK0)TVF9XRd6S6sHfy7}AEucqI>@*rd2V-25i*+fEo)cP0K7OcF~^KnB_SkG^@wok
zRhs*PS<5NAn&oVruyweQ3vge*;$GPe)ktuSazBWCFgb?Pm8jH}D9yY9ScI^^s&CJj
z3#DRH%hDvtZ*U@4EYWIE$WXdB10|hLLg3aHp%_Vam9v&FxM`>mj_X!Jv@C6Pc=`B@
z-PN)82HToDGVmHbu*RhyMYbRcI=4^`?)g9JDtNaMwqm1jGhujw5Dg7L8ZG7gf9!UO
z!&->@n&>^SaJ=r%SEvq_(p&UJ=2;u?pUi9Q8T*-FnvzL1yob!S1bQkZ72ah6oaMN&
z(v!I2Z)ekq>F^VKJ4#qBssM0fYwI4My3nZH8lTlEQT}X1kd=J9(G`
z--)YxxFbcQ%eWv_Q2=Cusy-=%QJUjd=lX!5REp{X4787^
zJuX=?uQ`-(whSx+^GgoK-EB#&nMF&~LV_u4^nt7~ZEWrw(G}oVP$uf-P**YgATWO7
zD+_T+Xho=F!6-P_oy>8-m$sgg(Zhj)Ql11CC5I1+YD$@~H+5#`S^Bb<>q}%mxav8)Pa3mOjnPF9$Sv%US*!`6i&nmg*4!7!tO_FSOt1yQItAZoHXG&o<_q7
z04=TVEs(1*SLBpE%;+mNFj6AmDwoBYwGN?*jTkei_#FYvb_Z84a+?cYST?mpBU`%M
z1Pr@novgRR#2j6>_=d3Ht+2x4^*7M@hhqIo#c1j^h(I-NmDIJ0x&moxu;m`Oi6&*a
z6U)b_0}V?Hjn=53d|b!V@UYu_Jbc13!gdAZ<+{|N+c$+6!ZBqvcAL4!{fI!dQy@uA
zVE05|rf)aY_)TVHz&U956opGOXA$CA-V~>7n2{kN;;?u
zUaDa7V93Iuv+{-^mJ0^kzZ;ei5ie!}kPEN`t50tH$B7>5M)2=76vZ``9llNbguixL
z$n-oxLZUo|S&@4XAi{Zbn8<+q$E+TVN|iV%YVI{g;6n5+Gge#^R(s-EY;lv8GgPyi
zo`SHkSf=wX-P6RlDvuu0CID7B8qZzA#?(=!7+?ixwxAUxX&zu2nn*mfR(?;!5GYY|
z!CYA?HZhhi_vWDcHlmx=gQfta8Zy@_-F~H#`9=Y(9J4Q#ov*p3NG)Krd_frtko6r5
z7AhT*05~GLD2VHXafZXpW3WpafDBOqY@s_B2*Jls3KqW=ErtzH1+RBi<}TwQG~{9Q
zT3?txSAHR@T6TmNHMd3R48rXUsLFaZI089g3Qeh3F5%#DwF2c2vq;fG?*b-ZYH3UX
zQQ|TlD9pz)WeN|NCQgs$IF}V+j+7B^E@j=THEXt
z(Kx1jE|@OKHa
z#t}J9;>xp(%G|e+GR9nUC`>c|0DT^Nj#IF<>YTo~fe-+&Kvg&AnUBP19++RK`ht+l
z{-Cz`(026?2}Or5zM{Fh@lxp7@@8Q`%Zr*9S>_BIdEcKICJ6-
z+vk|KcKt=AW$sf<)?iiawm6lFP^8>I3y2sU*h2>IgUqXGz1+#xV}bK5LZA_lo77%N
zxJznY0svFUe2&jBIC>YC~Y{M;@Yd)&%(!!bxx
zequVox|wj1^dHH9a51oUR79eUG}jM_U=O>6guK;51IcK5y@MI6`1Zr6znI236hICO
zy2buw(K7Eu+p^#`x}i!B;6FTIF~o899#
zAR4MqiMn3X46x-qt8*6)V;3##XxQMcVm5iQ3zGr!4|d?7bLR69ny@xYcW_w;-FzFTh9cl$Na5iCcEkB5w}H
zRfD*(GePh>nqfEGx4$yY7ThgfX#QsP0hQdxnxHW%Ac`6S**ZR^R`;1+qTtv^qZB|Y
zQ`Ddh@D&2J1_How#3XD25XTT)CIH_z2+4c6^dHPx>N14|^2M0kBjdQ41EBkJE6^|;
ztk=xiKG%pi19aiLqmTWKl-F9_T2@Q=nWnrAT$FKG;JEqZ?D=cl&CTUKw@bX-TP&kC
z?6*NJbV3Ry@ZaJjX|9h&#(=0m;V<}$F={qAyh=~7t(tydi_Ea{g5C=+R~oz?VNTb!
zVxz93m7@G+ID_Wmyu{W=#BVgVmzG89xYEL{vx2sQi*Z&>;)9-{;0z2$AiRcm>M-K
z6&Zn}xC_PVVgN5k^9leNUh@zuFXb|<-;Iz3t35(W>g}#z8y0U4q3S@rs0Wqc-er;Y
z!Yt^@3(O9ZVDLJCi;0Wlm*sdMam6Y%yJ{A%mDsvZ<_C?Yfv#p8sJ#AVRN1V+@V!eK
z0#&~l+}g}v1F33&HH#uGqV&SzWr1OsNnV8sK*H%lTbFSCj@&5|RhBkVKIMKh%mB$<
zNGl%O2St8iw>A4lT=>VLED|ZknNqC|T7Cn}A6zpck
z$H5;!bfN~z`P9o(>OExmAazU5B6I#HiEEsTs2~caE>d3L^)jLUFka9rs3Ma6My`zW
z4$7|!Q(#AEj47A-i;y-733O@0UZq%-)jxoRRt;!>xupt^Pl-+dss6xS`hz03B}ET2v1)D0*bWbu8s0vmPUF2bO7u%@HOF=r5RZep?YXD~}>CsL$;
zi9vATcLydndiIK22S~&74yq5R!=gOQlt85}<$bTA16n7Ey#^Uyxxm|^TE4{#ekLAu
zKPsqOghlU?LQ}Pk?CuSP)BgY<8pJ#gf&%Z`2M(s}_fyDIN5vcmxF@!76$lj?0l&Cx
zp_GQ(ssIh^h61+t%oN&Sd18}^&wPAENTuAMd_!`P#3nGmT8b+|)CR^hLd$-m0F7DM
z8CwY8=&4Oo6>vO2Hj?Km@9}b#fMHWEC6vp|Uh2x&j$K2x9fW>Yqx+YZg?S!gb}8T+
z`GxR@NUDztEj>)!+WbH6qc|gW3)BL+rx8F;5na&a%(4q*lx8|fLRnPGwVdiD`Y^5Y
z3IeDDSKJQO>>g!7MBbn%d$W+)RWLwaI))zjRTM10I|Ixrir&ImR*8$eOo~OObj?E1
zxL9)mxI3RW;^!KqOh
zn!L+4D{AM6QJqUg(gY!`Td#?PNY!F!f!=0i3Uw}2UR*;-XM9H>gwkLoSQ=9zw*UxU
zi)l9r1VU9XGNIJ2;P5ylWdV<<1=e2Ti!R^-%$F=j%BJg>Xeb&iyx*B>)mrf??ER62
zm4POR4CSbncJSO#YlFfT#F}p;Mk*VoA2IOxQNvxp{{X?YhCZw^sP7OqIZVngna3}p
zBm%8&;W07EABCxSww7SRV;Y`dS!+_@?cxfR_mr64-rJUCj<}SK7xI=ZUROMk#Mu-Y
zOw7rXwLz*l3dhU2z%3fcxV>J85z)e*f>a83om{>nq<9E{1XMWqqFfV0q2s8E!0MTq
zYvvlHtAIkTSuJ-2W;l*jinpP}VV4IB80cBd#g!?o)G!JF2(c^Q5Ya6z)!>;mu|Q9910^YIZHF{8P&IMfszu<;=F}d>*7>FWq&3WgHgp*
zj}N|KlngG&3xfF|30>wiu#Z@ma73A)`;^;D0e{k7z_>WXTMRbSwD
zh`K+CZYkIimAt5bq}4#Z;$tfp7lJ=Tvu?X1NDUNmE80zXAWFIf2h8Yip5+uda$hpbC8maF3S7VzYhjp&OA6vUv4WX`N6wjqV#c|p
zjmHKVNcM;743F#ts!W+J>0taBtOU_m;5m8&p5Za3gjKyxT$WXP
zi73@a_L&jxQyd})Y8b|7{J>nM9K{eO-Hhf_RJ&N*7h%m3s_N_UCcTbuG*@L~APO5V
zf{Hy)I|gmnuzukR28+x?^*|eeu}gu!u~S!kd`dtHIEA4o
zIXuPhm`oM})oLuN)`O_F^k0@)7Y9rhRI`7~QXJ#Cfa5qjmLTBA0HfA15N>R96j7`;
zIO+?A0?;@`n9n@HtLEEnTC~PiMYfAS#J2Z@4Gvfs<;s^VFSv?;fq0cuekB^ItBqWo
zi-?&@ASqgALKrKlk5Lztwg3x_uuDDCm$&oRGS>dn3l-dGwHW>ps3almgas82rUKRA
zhn{dLW~wuuBGNN$ZYEMXHbikHbGasgM3T{VyInAV)}%(&P(P7
zlxVrB(3);^Jj{wZ<7)E2;73Bi9+^rASO7ujCr)8ziJvA68o5{)4l*0m4%bW+I75VS
zImsrLX?u?461i9D)J^Zy331BZ5NW-~-^lS616d^oE3PS5;9?V1MBfCo;d5laC3OHU
zGB!OcBfQ3h0E6m0;sGQojv^INSMFcy5~>Q=7hmZy7)MsR?kL(9L8-D-cQlRojfhte
z_s!~ZQAT150K;(T_>L{(Xn7CnG-tUK-*mxLwG(E2$iBGX~XN2#TG#JDFBm
zaV+<()S*<~4EH;*?1AL$c!xNm85Ji)0HUTalF4wQq$0Gh5yC6EV}ps7BAcNi-3mt;
zEO@
zT{7k}K*uw@=wH-hy3$W4zkJKyDW;6Joni&<5TC97;Wll7RO($`0P(~z^n$9$a3HAm
zmm89yH9k0$;2pFfRRj=bP({7K^g(HD#rFgejROqmBNe7d2{Oz?EkV$zSADsQR@huz
z24)T|j)`~HjFQ9!o%X?;`tsXPs6#;Sq1yE;l$&_~Tu>;2GdLY8vE4L7H+Xv`QS(XI
zrwrs}SAiJYZs7jg56k_+N_+%;L3X%I-iMYK52b6g0{AC+#>B#alw8f
z<=9{Y0^Z&{Jdnaz_9GSJDCH7XmA-3CZGeS}yLJ1E<82#hQ+4X)QHNSFpD-X2n|~jC
zLY$_CfH~-j3l-p>Z}A%j9jz8L4{`g90}jY_=3O}O^XB;>Mo^b+a!TRYHNC;4v3Qg`
z)JIqp;uQ)j2G|0Deo~pV=Tmijv0NJ53RrNA$g#r{dnpVLQFrygn~^xi7$HiSwHErB
zl7o47HDsAuA2CWsD8Q17IeBeSW~&ZJf?5Y9%A5L$?MgDMTk!-bwNS^*VzSxfgG>tvC_jWMYPyD?6h~D^vxX_F*>LPU
z9O?=*TYInS25E|2`Q2P%jViOugrI%R&M4=s_rm=E$5hpDy+a2IWDRV7C0ibmRU6k
zZ&pp~>#0tGg4j3mgn}UFb$mS}LKv@2N>+Y3a|Koe{{XiIKyzzIkcj0&gUoua%3vAr
zoaxZUAQ1~@Z{`ajTOoGy)B>Jh7o;J)E{5O^SLP!|6rzs|V5>}^v_qxAA3-ci*CeoK
z$yOSJ+Y)SiKn&EeyO(g=)B=n*MA$*@EdlB|3XDQx*j?j|;(0QZ^EOSEcJUqAn%dh>285evX>H}m-Qn2T8roiM_8|J
z(#1|MW8z|~&UjV3{7ha&7tsD=OUZg%V~DF(11L<_sRiF>{{Y3rb#NmtL}fVh0IR>^
zQc~tcm*O~^l%a%J%(jM@wNaFi)l5}G;w)?(#0QyJ*hm=Z3IL0EJYrE7l`eHq7+>Z5
z#}wGqvQt`Mc(CKlvfX0U(476n#bL~}G6ucDWE+E^@%xpER|{xe`kFMvRfg5-E~Fa9
zv-*gmi+2Fo?hq7T6*eNc8}yevKp%(HtOSve3gFN80jGm;=0yHw>=XmV%rfT?Zli)*
zSj~40+)6M@b%?YS0BvHci~~Oslz<`^+Qc|Tfh*;#O~Ycw&M=!o%DcN^`GHYMRg_pH
zwDYV+L>#E(?a}uuSY4oj(i+EZ;(XYYt$7Sf9&&z1Z%_!|&@_Km_Z)n)N2sPFr0cb)
zLR76FnzH%KDuG_EJ0@J~KA>8q87z<2RIsuK^KJ>Gokf>TbsrUd%LyX({RxLB#2HW{
zJg9>3`I)9s6S6TAR}2#mE6-iZ?%-@MJVH)Lz`E&i0-cgV50tq2i%RD^1K62}iWaa+
z!zVN>SDUEI2%>9mF{@uLKBi)kE2O|i@eqv7R9M~UTB_nT>qiHuy2`9BUv45dG#C};
zIp;8PBWNxiqw57>&>4pm92jrg%p1dHpM$mpB52#{N=Zdp;EfcG!DPUB4Z&rUFkkLD
zR4ZIMg^X~IiMz}V`TLSp5k>Qugje6&M-%C6F~7W2%|R=n=;xmkt*pq=$X$xp&6E?
zbl_fdP^c=&CQWW%lqz$+Sy_m&UK=^WBd=4C)|TQe+SKltWiFJU6r54SDG$ktkGH;!
zHvvvX6JhEtuNs5i=FrHEhTw2KO42r$kfDIa)Ro%_AYzRrxo##)yBd&fjk7B{j1KXC
zP^6sPG|enQiujslxU_tCD4U@>bUXQnfr^CkF{_uBhHuJyhO3aOJ$U~BQkf*A{{U33
zfS^U{p;UT>4V8>c25tZumKp;MVTu*K-!kS)4&v4zO1J&)UIF`J;D*fVUfmL_8EA(H
zc$hIDj4mq}90PAP7LO>ng2qd{VzGlnM*#{`Ny;n-Yb$}PfHPR2802H8h8`*R8O|uX
z`GX*`maOCESp(f=&Gj+urK%?+EnZjy1g2yF6gxSVu$k&D=-hDd`HMP^{-#UCS(n$S
z0a|+X1tlZ~e4|yXYny@Kz$*6z2kjaY>M=sBxu~oP;2I8yMB4;@rV_<$@DFPdh$|MK
z5v-MY&B0L3>(LO=SAZv~czK(Y%ads@U?yMG2&Kd*sk&zNzRutxlWQa41skfA$^mIX
zsOgK;z+m>2=b9^4@s4KmfFTX$>LKOEpP0*Rn5D0nm^i#d)ppJvCUtN&%t+k`4qKEL
zf-9FKq!7x{h*GvnoyDavCm72QG2f>@=uBr={EV)Q`ZRVyi7n@Y8e0z|8%f{+PG3G{
z**pTayS~lG>|2XdW82gs#)iwLG(j}P3OBZ0KuqbmbjoDL$b@eZ4x&{>0ervt72Oa-
ztINmwnea4y@}6$u3S^Mek@}VfOPPG)aLlR9wIvQ_!25*hGf>5VfUh#rfb^7+CjfVt
zN`;0|;TwZ6E?!F>DKibHg$odJ51-~VVC-IB9%ACooS$|70BU34TTYK2BO<79w&5Dc
zGv?wH!qfp3SF>%(<3jVx+$1ey5fmnx?KYZ}okL5NF+TpKvDDx|^o@f^jHoNg1fbk8
zO?Mr;TZot*XlMa&1}0%ZN~dvM3c1wYC7{$c?0P1YQypq|I)o|yM5hePc36=>qbrtD
z#sQ@B`EOgMP)L9S
zqbxMsq!hx~O{lBo)OaiWu(h${Jw@q!m@k@@pyp8KrG#y*^%W;I<_H4Yn41!~Jh7^%
z1~#(S<559uvtCU=rnMP3vRyP1n(yXTnije;ivkE`lf(dpyG02OdytV43~cjjQW3;zJWY(%x5
zG{7Ta*GxG%3eKTb67<(H@_J$@Ie=`eV{s9EmM+7pjE5F!hZ(6+tIRthX5*Bg&(d0r
zG4KfT+_uglfD9v>hFQtPO$H?m2rBFJ#LEYkP?Qd(2Knkz@o#dYkVuX(6103w8jZ^c
z)UdCBzflVX?UZ7cE%6vj`nu~<-mOL`?)M#A+i~5R@%c_Hct+|dIxP!T3)`nzNi@TUWXPcXW
zbsG*jmimuE$H>*~m7pA}9K7wUO2KBEimmyg0GLf%h+8~7_Y656Vkt!z+(C6W;S)?R
z%NJb43@~Ms)t{+Z1Z9SCjgpWW#UxzRQ%B*CSI<))06ryp;Y(9=ntfsu`9-k5$L0$C
zF~6uYtoV-tHYhw%seW=0a+OC-Jo|%uuCu-a-R9%r{nb_Bu44rWWdO5x;$`zcZ<$td
zcf@MJJ!@KsyDsF;y=Zxc0?8PF=PsJ4flMr)V~AkWYOcpwTMLZl%Po}kE!SKHQB7fi
z$he!br^_TYmg-hwDka9)0u5AERJq#;EyFg(-ECY5wfg#wjmCxQRC1%{ZcJwGxR(eZ
zx%Df%Clv`=n!!-5!BX$I&JBm+ZMp{@{{W~2RI07Qn)x>Pn*RX3K$qzvN&xCshs0p5
zV3oS@6@p=(ea=9~Q*K;Eg_t!0`9*D|g$W**?xYyRilhOQ9;2MxO$o+jEU@j8Pzrz+
z{1JC<npWn4
zV7dm`ctxZZ=y;VDgAPnw-7$|NXiMETpt--AZJCr+yNFxKFyJk?q9F;UKyX}{#bBmlM;$fJryi7p|YaGSzM(dIa(vOs&R;4dd
zET}xOmOkA;eSUj@tZ*h-c1w949!X>#Or1n<+!;dDTBaqmdkC#&R%o)KsD**d%NoBj
z{QhvyH|>}46%I>|%Hc6xr5s>8n_NIKSGN+7miI1npUh@&^oN+N*vt4jg}L4Bd5?v3
zGYL_lI`W{!Y$^)5l6ihu2V>xcnhLp8)etLG?^6=N-x9f8ItXDjHHl``3ezv)w6-5U0nZB`cg5Ym|h@iB=
z3w6R9v{xo9%uKpM)B`?0S?w?6uwX6w^Ias@ukvM-5krm>Ap^
z=}6!EmMyin;u-^05-%)Z^aNjUtBz$)mNf`$HT*_q<==qcr^Iz#Ef@mV6%J?^KSEdT
zf#;~qAzoqxE5t`wBHm*8MQNEpi7HdXS8|rg9I22T7>m`>9Xv)W4ZKVjwF#yF0K10M
zrc58(FT@lfN>6y~@JqlvC!dJT3fio&YEqQI4^(HlVWN#SFnE~Xf*U_XTZcYi`CGV?
zRb|H&C^QTP4uLq9jYJG40r@l26l(9xxn$&Z5GuBJ!z}>IQ;Vm#@&<=6t$_J}f^cs!
zRdBU&jzj3q(+~OO0igv>9h@K?}amduW4MtrR=57zDB8V!)!`ty3
zFEMDuAr!I*plC}ksAUR^278rpa5*E%2&>>wt(Dq*#SjIyhmCfaXKtdasc=Mr(ci);y9`p|ucxRr;1pKRs2&nDw3T44Eap@&
zIO-krgz9Q3i>Q`Qscz_rcbD!u&*Cpg%NC-c_E=&XXQLSpGX=6@aZrc1rv-av46F>A
zb*QMf%reT4*(?Ks`;Eeyc;*j77eu%X5eu%PM3KSFEUQf&u_n00_0Zcw`Xi1h6pJi>
zP&dVX5}Ti7QtZU&_2NI2-pxS8HlbK>$16-^f!i1o$H7vo1S^V#JjBGMI-1re4j7d|
zHXN`+{061R8EVS;OmgWgJsidee5@a>J_$|g-8uOFQ4{UADvQvA2N4(MDRg3MYW40g
z!GMTw%nB+kIS*(fHv!}ya3EPyt=zyD$^Pp292#b6?K#bBeo?yv1RzUnH1ny#_lO*@#0NP%!r(wsq6LZae9nd@g=*cCQ&rpMHwI4(T3#Dja^}S)
zXlNIjy-F5tqZa@M@l0p|7NK8Ab%JM@shgLoqAas`)EUKQCy8|&DT{Q(t{}Mq%}ix#
zr{rK18go+#e-W)qD6go1G*idwB17fkWTQv;H4RE@1;rzy5s6wcbqu4i&BbWFCX&9`
zBwJ_Z0YE)AdU0zyATj!|t5MQhs
zD;bG~G5MB`c17hciABS{p;uv}y-nl|N>3k6gnRt0co2*ts1(9?&AIGUkOqe^*7ZK;re
zvLUPvUx;Bw;2-rtpf!(}!qeArO4}CkEOm+gqLG6Wqq9bLu!_2fY_@LBXzzMkD$~v797&SAi5!(*fskM=G=eP@F-t)MNmf@EJ
z358#so=9n}0Y>kcOr9K)od|ElLILUKhJyo#b9N48x;c+txQ*P_u4N0yEIXL|kc0k0
zo?@&v&Diibi@Iw8c=r*(U=I6cjK$B~?Pi1K!a)DJvA|y@aiC7OEld5<&fDcpT
z?p0#PXEEM%?>Q#-Z$T?55l{R>uzB%T6y`JHcrj$;Az)THOYKj
z8W(Z1CU`qjN9r2H3eP4WbN(C&Tl$C#QNu*I#Ar1hfnAu;h-B_~AlseAcGrj=Ac2CW
z*{n;zTsw_n(xpm~R4*fNGT|9*rX%5lo0*7Bd6pdCse>HzGo$JRwTwN(vk_0}l&b4<
zlQwYKL4$Y*%hd%c;H8D%WJk!V;G+vyb}lct`9*Nyg@+K0J~72bH8_-9FG-5FU8cWs
zwL0hMHSQEr%P?2I{6@AKDNf#?6>o6$3I=%0s9X;yzCuLMb8s-#%BJI0ae0ZnVo|o`
z>X>6w4ae?94Mb5;Me#Cn)I#wr*@%3LN~luowL6N}QO2VsggW|MZ+;=`r7VTzPBCYO
zQ-UQ+U6geKf^8kTc)3zzAUP$^uvXg<4e(J^Jj{~04k3bXV`ut^2GmRNzM@*#^BgMd
zWIUK?t$<`8o1!heIK-eLp5k?g_{8Ea;>36HE=?cYz^kfUEb?;&2%T`Aqcp(W92uq6
ze$7P5<*Sux;YzRObzp`^<`8w`F-M!kNDa4nPNgzXeI48|jevd;?9|DF_=m%KiC_Cm
zcQwlL#8SASg2Qqm-kQWwtvyB_-x9Gc%*x?kT8R`GSxv;yZdq0~#X!hWL|r|GAj)8C
zhn(|Ulw2dihOP-=`G=WA&ru$){2Wv+QGYe;%bb+R|U%$iE2d^TR9>mFjQro(~e^2pe2-!312Ng
zaTYA{(*XG3NBNF}Vamw_0G<;BeSmL<7R)M=cB8{D&XmR<4z0w)Y!i3Olh!3P%mZuS
zZ{DNn48Y!nKh#x?uX|x!t%3gloV_Wx&CW8(xJW9*#Ie!}m%BKK2C_vepqtH0wqmWX
z977&*tXys`2~#Gym?h>?zqnNEm{>$9i9(yG05uWlWwIbTF0(A4FG}FaW|YiPAzJ0}
z1Sl5EU}kUzi$5uT6uqCA!Xq$`TZoPF+#WcGz?oW&K#|w%!y@+77D3hrm=JLbG(qe1
zjtDxV5nir2xaK8hJzRP*55{7oz>STY?o_O*9MNBlLg+v@SHFqx8EU7HH!%D-mTDWe
z8G>yi1xIp{ZSGOy^<~0jcYs_IOnHwAI+e?V9aq}tW&LIgnryv6;pSFn+;*>FEceW?
zBi-zR3%)p6trVNwceJ5kIX&l9+^VI?d+2~*6$$Y!s)#cxGb_*dSZHr*K#CwHCsyZa
zM&JcGh7}pWOcg^YIn+v+aSte*qjrVx~*W(6Di`sev!jw
z69pajFlP`umxzqho)YXBBD4;!;wBnR!`>NgJ|evL9C?^U+!3B&+rD95V3FaK8O%F7
zKp)h)0x7zUO`)L6Xh1fBUf>xGbu6wT1TYO^U&N|eQEoFVgnlKim~na%h|a)v1eBhg
zDsq3(sa13T0EmLw&SsT(iE7ajvQh!WM~i~2OM^$2FB0E!(cW%bsl7}xdV_zd+o;Yz
ziLNCSIbm_D^#St00n{uLxsohSqkxQaMZgV|m>T~817?;kYSgbvCJnOP&&C)idxX1U
zVpM!b4EF}3bIdLY({F2*Hlab4jDW9;D@>hNB;;-^CqswaN-^vjfNvO>?2T5*?fIA9
zUkPVw!4NsrF|&xlbt`TQSM-4)6^14ws5`Q}LSLDQr_4Awfma%OnfZp9%rk$*8KNhB%x$pKe9m~p+jf~2r
z?*S`jr=mANd`fOg?sOAx^#sNT58uS23}Gyv8~+
zsQPvuUCy!8ng=}G5IhtyXi?N`rB?5ll_U(XGlmp_Zst7>gccWn#H~Xqr5wl3SvO94yg^A@8
z!cyx@E4{{#@?_=>$0=&Ap_NQGGG-?ZBD~xNxr5cjRYI@u$-GRJxQj*3BJaddi_0!^
z7S-HX_>08K9mlAvQG*F)i>@V$xBM#}WjBeNB88~gy_p$&j#6Q;vtuwgjcYL~?|AgZ
zAyQr)zy-^6v3sk$KM<+0ce#KyShW%4>&_xzh1RnLnPuEjfz->WRV=g>BRPaz;}Jo`
zw&lyrA*&UXUZ58Q+6*v|P7uhb8q8~S;u6X(CXCFoiLf+4U)21`v;G(&YA0}2-}o)E
zcNg^sA8?DfFpH^rj-$+VI6;s29N32AvN~a??Th>$_~AH3{maC67;!K64YIJqVr)Sq
z5}}MHh=m{HjuVvOInGJ5?r=kK!TvkK0_G9h9nR)*02S2X2n9|d76Axx2QCSwD5RTU
zHA2*C7f>}5PDuKYaSDRB3r5_*@F-?Gh*rtF1OVb|Gk~$C

literal 0
HcmV?d00001

diff --git a/doc/pub/How2ReadData/ipynb/ipynb-How2ReadData-src.tar.gz b/doc/pub/How2ReadData/ipynb/ipynb-How2ReadData-src.tar.gz
index 8d8cf114d544c678909c754268274691554f1b88..4e4f7e1ffa0f085b2fa106bc135b7891babca7ef 100644
GIT binary patch
literal 103580
zcmV(xKCsSPj1MIp5aGbHSE;we6*)cOSGh@um3}eO_GdpH>%rItVW{8=Y
z8Di!*wv!l=taHx2@7_JT``+8Fx?8ndudC))OX_ZYEvZ{l)mYovu(6uDnmb#WgIVoe
zZ2wVfQ7509oU_ny#Vo_fS{oM4H?wGAVvCj$s#CdQw&2}
zcXwB?ARC*jlclFQ$PVc4!)k5k2n4fwxqz%(K|nB=^=&*hPIeAnHg*m+5YXEiXzuRu
z)?vymz`?`L!)N(7%*4Dz-nKnEt*rAyK?ED-;JeC$#mV7)`LQ=MG(*P6_kdTpP7m$`=)*VBzNB;$q?F
zkdb2Hkm2L`pIg-bMgPAf>$m*>Uz`2^2l=0i<1H#U{x$z|vvdBh{Qv(3
z{aOFB55Q26m6v@>E>KX+P|R;_zX6s~F0MWxI~!Yfat)w2x$M8?Fmfe3Cp&kb75Q6!
z;}PO|JG!|!_}GPbd3gAQIQcmEggCgl`1rUv__^4HWI=Y!b~VrV!3;=d)v;NkTGr
z2exyzVI`NbvU7I<*_k_%znxX&s?Lr+tmJP}k}fU|;Q5LAM*c>8|3u=J{JsV$p11E
z|A&Tun;JA^G-TqxR~7&);LjF73IGoa2M-4e4-W^AfB=t(jDd`dgoKRq4jmQaJq|wp
zdmKDGLJ}G>LLw?+JUnt{aw=Ln21W(~G8T3gdUhIm2Kv87pb!uckP(rwk&&_K3GoQ&
z|8LWuK>#KqG#AVn%-dxK8WRcz6Y9?}fcOn2%-`VqZ$SwI3-z}D;SmtudI_KaF#oyl
zji8`mVB!930#IR~0MHmP7;h~P;9{jl9sMu{bPWyvJ?Nwv4Fux&YB`GNl}U1}kq|rj
z1>0Bzw4@lD7?_+%xRImKvGlczhF~{Wf@ZB&C~si_1Uzw0I{?}ux}dQ|o*h}M@VPIR
z0yp!n_+z@bOjS>b)f&cFIq#iQkz{je6>_D}30NPhTX_EFS9d9-%246ScQ=-ag{!rx
zZf`--Vok86-@wH~x#vtx_*Sa$KxX@KY5ktGojADLJbFQK+4D3xQ7`p8WQ(zC_LN1E
zqjGTWq0@>=k&-c@w{xUoZnQm07nvNG4O3!+vY|
z5#Pn{v3}seM?vT1k7z7^-P62R>VFVbT{#&y&?Nq~W4EK9l)^vUTc+=jqv;GY~)+Q^TQz|r7(`ca+!-|$7Ut}~^
zHjS4NJ;E7YdLYG@Rh6U+g){bx4CXvaYen~b!mR%NhUAY{teNJ9+>YO+2bw~gs`m}=
z37y3xBEMHJ#6Gk_@naAehk1Gq7Jl*A{pKKMiLvX=0Cy5IhPd1U83ydQ$rGHkP
z4S@0sWINAe1d0{E)?c#t)JvVZUy6yH)f(uNdhKSDWx?0yow!!@#X6x}fnST}7R}{q
z%pG`ABN4
z#Nr8q9lOxJv&IXaCF-Xoax`Ul8U9A!;e2V3wQrlKP@Kjab!2cDA$g=%LC&g#Jk>h*
zIiqUGWN4{+?`2zL!2O{KTKWj&9DWw=VRz~GKmyR5X4mAY!&Gwy1S@pRhv
zG__k>TwJ$jLm$PUXG*EM;vE-M*Lsg$;B#9}6MZC_do(ouWPCL#g3$PlG=}q$hgp^V
zVvNHZBtdwFrjcY2RZs1=qLyjp5oyPSLC1>807J$dllQdNeovrNS4ptR0R9v`nu|PJ
zJxIgzT~n4m+bCr9p;ABV;3vR0pCeJXef#Nlz8Xp@t>@FX-^ByKrbHA~7ME)thd>GH$J_^i^xq#JW{kXCkk$H2@+OU*I`cs_J7SfdL8=WKLAFSSplO*HEo
zla32v^xSjgk@z{#2)+DgN&)-FM3-`+oP9qFW|ZHeJk*g?eVH8f
z?G15ic+l3{1Zt?cb2c5)k&XD?3bs!;K|eVCzi{
z=7Ufi=~9jZbfX}@Yl7fzb$NAj2^Dj1BhtiX;hWrrBimmtg;yj@T-+yx!i}1wz;Cd`Vv_g-mJ@tKOTGRg
z_{g+~;WdsVQ^STq;
zVlb8F9}&!1fB#MDF1JoK;{us(&7j=Ava99ha<1e(&cUE2hNaY?JFxqdQ{%d9$w<-_
zN)Uw&;
z)rT0UdjL*)+KD6m-LqHnI(JR{i?2ralct&(m5BvCn7o`uis{pe7QautKJlvYKqsowt-mP#C#N
z`##c2RSt!Y%1E?bF6qT*yJ
zp?1U)GMbGU1$lJNPd{3MG+YFFypbqQFx-iZ)mcX;QoID4v-EqR%4ckuF?b3kY=y+T
zjV?*54L}k$x>}zxyx4(0nmRg*99A7NX?o@5z&!lQsxS4NV)9AVXth)fQ`g*%+9K4;
zI;&ktTU9Ge%SCr8<~7J6AD{~lseNOY6M#NQ_C5+N%QU7%6r{oErboRor=TLiBl34N
zswCLQ%AhpmKFu(crvMkZ4jWR*xvP{TzNbgN)b&BdfOl@fsd2ISx*t#TtWD5fjAk?t
z4Vbcw$O`uEH_P~agnrI`JMDZ(O=W^XU4~xFh*mL0DSfeY3=~K#Jfk{g4FEX%W10K`
zu*}83H;{qIZ`B8tpFT7nc%^qmpf=cE!VhT|8ApW|&Ok?CCp}=DX4X}L(cZbRmkJP&
zuNe#9@G~l@5t6yb&>tM<9B@RLXj6qxKG9?vT%
z#A&e3DkV2^(yJ-4NpRLa#;|OkE0=v2QwEf;jN$T2@GE?-+4N+}+6Yu~HCAar
z|J%6~zQaez6&4kPt;+YQQ``CE>nm-vzic8N!-Ge`Y?>-#j)FVpArGNuuTvju^L$G&{dR
zFS_yf+Mask7H43OO2Yg~$8l=3`iY?RUc_vIsi|Yqa3V`q7QdHxia;NRdaa%w2Tv27
zLwzJtowrq!Azg<>ro%!xS~H}&M&7F8YzYQKoKSyGgV_T(Vy4{W*qBz63{%p7smuC_
zRJI)W)nSQHuE9}m-&_1@Fjtmu_CUm0i{LQF{^<|E^w6Q0s*RU-Zzg52LpAIAs)X|H
zm%HwG>+i}Vl$nh|$b;hzRY+x!KpjYE=<1Y`Saz*Q#abN=Iun
zDu;2py*;bQnR>b)15^5_OMebXU9CH<<=(F1^Xf5k>jtuEpeC$ve&0|aLiw}Hb=c&X
zQ9VyHnYUb`)?)?Yx8zfAaUhgm&WF38-E*weYrOz&MiU|F_sFvTn#O3#ee2za4f5q(
z{q|h^@cv$imQeY)1$rXDKr%MHfLTMsCE^2?mq^>j{wI1QcAYqu)97;5(lj0wwfV~w
z4$es(8LMFhU(|&Zo}-nI77ImDG4G0MUDsqp6fRrJ=ol-%q)*K`Tw4D6Q13evmLK&s
zNY6h=_%2^#x`fhsAAFt@dG7z6O|i*OsW0(U3%X+BOv=LkFdX)pfZD)mrEq)c{p0)$
zKCwk60YkN?jetFkgPh2=I!BKPDUpR~|H)?&SHs2ON)cmAUM0BNI#H4vR)oIrdl3`A
zmGSYuPdF^sDj#cRt+R~=llnIrra|e78R|SJM%hDolPv=dORpVQkBqN>0BaQ@A}XNM
z;je)|m`$_~Mm+I)c+@Xn&9(ywe>*JC(fXA0C!JNHeB=qawQ+OeI9~l-m3umasf&S|
z(&zU2G44k7{W)(#i!RLM4w!Qtwn5!j-9jmb6bYLots|#G;bch}Yms#{FCfndt4(&5
z@67akeP&IzOLg8~J}sO_>F7_U0>y_oSz&K8ukwPSlsy$3GD4
zrA3PoW_Kc$Se{>;xKE?5D3|Z(TocBYqP%6L;={GUhf7)4lMpevr)Jt9jk}j!
zkU;9n{d8Y_z||+GwMkOG3R346GHO+?iCaGk|I@jb&-pEKr}LRHj~SC^BZE|Sxz)(#
zrPq2Yv({0LRYGY?gr8LGo?a77^0nq-`&@Ps2U7#@4Nq#ep2q`SF1$)##g^~6c@3T-
zdsX%H6(isgT8f2NZ;LzpErs@cXf-1@NH^G>et=Fvl@t>*Iu5q{!{%qJe*mmSqKy8&
z1WarTX~`xzl$-6@)zKTpJbAP3WD*QOA9oL@5)u;o9Xw}i=7iEUdx|n*iySQp67BiX
zJe9?fk8X6##%|@hXFx>)6sn*_UDK+cKbHD9o%+6)xrtUU`Q+!}KClVh*V~isB;sDu
zEW!N&h@MVIByA_m4=kiWld!LKkIp)C>`p)QQJ+2AH2&mU2wgQF1qEum_#ADaSBxpm
z2n*f_TQ66i(3E~+uq$8}C)SZUI)_ayDvC0K=vg_PG4OXW1^~vh4IWV+a}73Q&frJ6
zUq`s=of;3{=@#APWvP^(tKQ9lE|<@~U=p4cWmcbZH_4p-BvJH?>ay}&P?g`Z1eE%Y
z_#Y6_dM0)9R?QsOyl@PEr&r^XK~y|~cQ&80-Gn#7QWlFi*PN_N&W7r`SI*0(+(B+F
z^Z!A^*e_1vdd^>0_omnc8v|R?v&BbT!M70T^PoGQF&=oE`W&cw`zXBc=wqam-pdPb
zS+np=mi&MH|V`q?`}|X
zMSBxW$!arC_!w`+cHfP|gc^7Tsz5vwZ68)*BhJK0t?J#;@%&AlqS00tw=-_GC5z+6P**EPCycPJT-z(q29dNzc|mHmH9SF{yFn;J3krhN77svop6+_
z+5#xx}g?^`x+5nu^Vg({!$ZwJUP2aUH+8I1W<^7cICZB5J)n&j-P{KUyXSEw&Tl
zQ!B=B#?IT7oL8FX$wO&_@3c~I_cYrmvP+h`a$0lQ_U-n<|6S=~#j{(j#s281&MD8~
zd}+)>sYs5c1M~(md
zu_LS~>PZ@-;>+bAuO(3Dk?m)8%zEV5t2&-2MOSEl^)$29Ul_%i%p&fKAi*wkL#dWQ
zaMz}-eX0sDW?isT`O0AK*rBf|SBNI4~tqONKD(x$o{I%|Ai^z}6>oKJC(fuxq`OyuopZpFXkoN(s?49BTA1Mrs
zrl@B?t
zM!XtvRV{PhHDA-3XU;21LB^E=SJ;Z5EVxykrSR`fg9#jbiSt}1f8>pl^&4+OexV$x
z7vxD1h9C_U2E-}1=Uy4FWMnUy7uIx2myIoI!I|}&^o*#8@>R+2L9QJitG>SC<~1X&
zbjft?X0G+=DqW)JEnH|@nkAWy_K{VjNITeJkqPxC2zRzRIe>_pHa4FujruZ{O`qD@
z2m_G)9WZ|35$+dgKjhhe`~GkT^@-9-)kLA))qz-rQU||^u|twV`Lfq7hj)2^S(QjYH3Vb`JOh!?R$!dL3oEh?1dByZUJ6#1y)ONuN{=H7gS&Iy-V|{
zVG?2`kDOYogLn@zhCw_>dwy<4nn3NBLxvytyNYtug+^0LeD|E0^D5WHRf3*Wf>1PD
zGE4|1%l!}7v*~`B4gqxjb>AVkjmue|S={tO(g8lh?j8c4XdD=gPK0&K^jfhz+YNKy
zwO7bjQ!Ba21jK1hUs9Lq+eVDZ@ns!(|0-Pc3427?XqaW>R!E3rMlB9+)%hUKc!AIl
zOtY7xkIh|f-ugsdp~y1-fP$`VJ5icjsq2JH@GPRr)0B=y+C7zX`iyx{y(vQx-xXWY
zjuK`|9767j7t^n#1HGVcV~`x3?dtf;;rI_gDrT+CDSI(&pP;G`fhpfsPEW6ML`~_w
z@jWr_dE?jBI&i4;a^;4#vZsyU7P84nE3;96qZXL+?GuX
zA=)&$T)A#4c^}uygm++y324`9Hru5iE0*iDv&$>K`TY6(017dT&xSt+VHXAD$&iy4<&g7}CwdFOk09STJUfmE-^8Nv2nw76j
zadLb9zGJYN4?p;;T5DDXovw3sUm}?onsJbj802wR=ejkk&gdey2ZB1~*C|J3E~rxm
zjTH`RAyFR!y=!loK`+m;D~YYJ;8zlK2cwnE06R54b3|#Cs4irWJFvL=VN|~}6rXk9
z7)sBFVqC02P0?qxTdn8q<`m}T-OsI1^5Ks@;DyuE+CtjYN%7Vt$}_*}!aaFjg&TA7
zZU4RUZLgA6VqCU-&W%SV_EvbV0`CMn&|Ki2$90Bw1-6>aKqGVubgwJH8)s{8l*^~>
zqeFt6tCVUx!;?*5To*v|Yw&(~v!G`}ga_{4DAD}Bm_(Oj@K7qG#=0*F2XXcxDi%5b
z%X0P%?S)VHNaq4L*E|}Ai1$q9(hI;-N^l$aTTr@ZtVN#p?thybOr3*3q=fzq?
zm8h?szZ=b6E}m}j#dPn@OmEX5kN^Vi$
zetak`;zKra88gbNUs{P(`uud@bdF}k+qE;(9K9;P(BWOyqo3fYPEqxw+Rn=!_VJ&dqj_7
znN4f!uI5_SUw$c`&gR{BJyOrI1Q%=+(O@_0no@ojfv^eU!6~s9+SyM-;WN@BDT=tA
z7Iiy3+-CYEJ#$!*o3fns;lV$o+M4{By?%WZ7;$&WI@qw8E7U+?AIw=^{AFjU_r=*s
z5fRhdjD7_EbKBfC8y(ET;fJK$MtvpQsuo9@vG^aoqq>PX4+=gX&^=x9&QLVToE^xo
zfVPm0(Cl`HC5Vy|w+aC@vrsA9T%s%t_q8_i5MTMpLxW@>C=?CyCi{wx3E|{(3{Exe-P$rXmc$uR7nWXc2j8o
zVd?(|uv1`)^P(dus`&W93hs+Y{d86ZaJPLiK_$82C!R=h7%y`axkbJ>qe%?cU^irVv@grhIbIs$oKN;+<~
z9cArZ8hE!XK{8>>Yy9EC223wwZJs)!78|JtHm34H$2#+mKlLv|`ddOi@Q&EHq3~a<
zG7!I(N;$>sic4cI7_3Y9j>swws4;R&a}1pcRZcs(aQeUez?9ZSWuw^RR6{db$|p>v
zw-7?`m0p}$cc&%__B%EE*pyce=D%#MjOb;#=|TQ^U}}TiKtdCqv+|lNE0Dz3?$c8
zUQf?t)ia#@&WC$l>vcmoUVn5RU``0ZGM>|-p+y#aY-#2!z!pM0D+?a3UOF@J
zaa=zaq3@g3!7%WjXb_?!Ezp1Z?58e9agE<#>HP;##F&^sFl|YR`qBTCJeDyn``yES
zC?}-7PFppZCZi&Y@T-ukV4jPf*Jlqng?YITc#{@URoCryXgoAUs+dza$ykloXc)1%
zII-JAI=1|?j#P#k@u%%c*gh-EcEnJKy`whZW5VL8d9JdO7B|7Vc1SBBqV2>x7ClN@
zX@+xpylC464r(PW*B%+%gj#GaD{u^w2oFrD-S&NmNuGQ}8u*>8oxO?D%iOb=aGv*m
zLd6b%4)n;G#y)r)Hl#oHp8l#mJz>>;Ew$}nDXe3lxLrs5s@9cntA}hUNxn>qH}W1f
zC&YF}#O&8{dS$OQ4
z#(Vc9`!n4vKMm;#Eg-F60qv_mL9!4Oq7eC-e#tfDs~Xxop%WR!1LS>uTr$*+LyW95
zuK1>Rk6eR1$@{5I{b_!pNmA?8_moQ}$TAl=6*XI!^3spY*e1a-yK$}7n9JGiF50I`
z=cy3`!d6!~lJn$r&8d=_=i0hD7)TY@DqpgTPSwVRl@w~ej@duzo0^^m$S>Uw8*5)M
zZpPVuTtP2?Rw4-#xJzvKdGLC>`gE&yIuZarF_l*mqkF(N^LW>s_i@2eA-PuCJv_^}
zGb=C4cxgOlH~2GA<*_a7>hllbN^^|u?EuvCXrvZB{^gGBS`*EDYx9uwHT&iTnh8Y(
z^23z6G8?Av`naJkBB#B-{pxX9m<-&PVtPZ#P3lpCEVh60>o`$&faLl8@oLn}dK9Za
zl*n`)Nk3?@sP~`9F*h7p(v0?3(yF6%hxYAov^7N*bUc>yQIf_Ak!-NE9-{kilrL?*^!Q5OiyM08~q(O!K
zRU$W@VB}8ES!x?0B}n7O^0yq5LEXQk(c
zcQu<2wnrhg@>5~46ijN!LtsoO5e%W>du@6FUO@`hgrt(hNWZ1f2!r4=W=m5W`&>
zvQFN3+GR~}dncLa*{;AgS_fE_;G>*R{7hp$!iX_FHAJ1mTF
zPr)J+@-_p3MY^1vHr_@5n8IVXd%tqEm0ZF73mT@e2k~L^sxrsT5vuV#7B4-I?|d%(
z7@%x^5<)9;(u=&Khw@0ywtQk5IZ(c{7AEH^{t^**V-~xOoYN6NGKl*UA^78+WhXYt
zbiEgpO$S}~C+uw!l7WS75|M$`8^nbDn;W2C2g}ZcyBFW`4e=aEsQV++_MR7*synp1
z6E9&u#|sW=o7nkWLv)|mi*I2YlVs+H7o|;yP06B
z&rOFaR|o#e4H={!1gv*YyE%IIYdZS}tYIgn$N)TpOf8
z-0Z&uu^||BgIJd%jrEliRG3{?{s)b^WoKX4ZQCTcLgmxs_rI8R2{fnNd4}`
z%bpb{?y~e-CW8$$As=69`auRN<|rpoXOSj<4vRj97L`44@yp~5ZqsR#tql|Hn>$3x
zi6`W9lI*&DLuo@$-|$@|Y@61hBOXWDYkAFj{^P|&J~mY)jSV_yPky_mrkW%yKovia
z;SfTH{f04B!en2fTz}M$-5-M-pepMU$1A>6sXsASDdWGv@TJ7w5s}ox>X#$%Bpzeh
z>zg`PlOivP?AE(a9K$oTJ#|!6gu^`RBGcf5aa+ol_QZmZgt)~PCHxVrd=?@8o>pj~?(L-3(T#G-YfM0orJP
z>G6N?6QET+RL0@?t}R~5X`1SJx3heHYWkti!(Y^aVWm;PRX?VuDTv^*t+Y+o05a(g
z+9@=eLLBTZ{aLR?iOkI5npq!)bf|xbS-s3?y^di3HWw~3-!0VEF068!pHSiTayuKn
zx)Z_%Y#$&?mf>=#*;C%&E|V5J&eHmOz3r8_ck{l>%btjxxpJT>d43Ys*QJbkre_1+
zcZPXcOTS|()0P^)p})%T)c=ZONc1%+?s`49ITVou#P|^uBHsJ~hH);S$sw`%zI(*`o?r$Fnh&`_CIIw_B=^f@dtTG+YAA0fv(ZfH*CuiY2
zvgY3^l@s0th%2xefN(=2fbv1=rR0<2eQYXzXnI;}2yhw4+Ipv-KD~#sS%Sh#^vAP0
zph=!a(c$H(Lgw8~kE1c2@HN;neWj=AVxboM(AQxoL`tXeL6?l&G-idw%N*VMr^#k0
zc>BnD+Ol!X%>g?Kv#f8FYgvy-BHw&{1iU$h*trq!V=2gsu)H0E>BqgTem&gQnG(06
zAL@IwHtNu}kG=ilJ9=YeL(&?*D)E6ZtvBjdf_z{u^z0Ij#9DCB&`N*WjcsW-Zf*5
z0g;2xF=yl5c!mM#?JDQLrG9F)d*JF@1MA$req&4BU(!NujyvR^ktKaFgLl=#rFBuM
zd+f%W_By|w!cLx}R59!4V;dOjuzIjx73I~I-V=l2hheUlC_8K`FjKW$M`WVqkrt1D
zgOHmliH>2bvFBi50P{F-UdTvv*-{!aABpNlN^Q%#Wgi)ZB7_^1!ufw3cJ$>FN%A0n
zKj!-mdQf?>$1JL&;9AJWuxCP?tDKr=Poh(6)d>{@3SVQs|HPw;lxT=lEUb(sWaJx|
zw_6&=$}@7HC4un=Kg&fJK*C-5yagsGZoF{`724mm8xn8)W?qaVWBA;VeViqTPfxe
zM8e@bD|e9^+s~pWFIVDA5b~LQ3~PfU3KPaqK~&W)=jc}(_F3&B2-Cp9>~KhUFd{Dcf87Zn
zwY4%S5zR}yp`#%vY)Y@Kct!<-85>mH(-zQ`&>M%@itb$NJRykk#P^|MC2~9Swi1{m
zcJV}$5uTN6MWNDg+(eY?uE$dB%3CNF_|jg{+ew1J-nrQrHW!e{uuSB&uRYt(1K<5l
z$SfFP*@2Ic^!hG8jMTjvIj(++IEYzD*z10KBb(i&O*>EU$|-voA!b2pcYXa*@(;k8
z_3=p+{ORL3Gp5OVTD(W3oL{;9^Nn#zofF)v=A_LIkkicMHs^ONL?lZ5M>5|kYfW$o
z9r(F8Ce3i)-?G0z=%R-7F`_fK>$%{Bvv`6lH!ssmXD%y}e^;h}b@P+r!+z`AF>z*J
zO~*bkY!ida))f>l>>PC$NnSP@y0=rTSDsuAoG@4z$P4aLw&dez*YK(HFqFqi2H+1HN
z9-+%Ep9*;GInAmEjMUgw6EgAt=y*0g`iyz01MdR>w#YQ0&}BwT{^neppYFgNngZf%
z_Kd9Pe~0RpX7zn6I-eOqjUvn0xCrtnsrdRzAyb3anyb
zYvl4M*(x5jm2*kAX7%Z9T4$Gb&Ejl_yAjXPrd~Y=?47eo58RlG!G*ef!*aM}GDU!t
zP1D^C*SJ}mj{!?l+mBgLNI5;e#HgEo0R^_2t%M_6XXm6#YpD4zLcY3Pg8U`rdmf;c~)&w5IVhMqkPP5E-1$Z?vHPVwP7nyq<#%
zwhmOo=CvMPe{pVzR!A4a@Px)>CL~zXIhOW2RS7uNmEoCX;uJLDl@gO0TUC{7q$O3}
z5q+*#Tu4J@B5sOlV%5u{UxIu?Gz@?QBcAN2Q)UvnclTu4-=|tK2Pk%KS(9m2ov#Sm
zn-bwYHpK-&jjWTZ>bB^hdnwI5d!Zt(L&A>eCDXGG8Gdo2Ag*uukga|t
z_0CZs8@X)8c1_k1duN{foNGsTQ46mu*SNH`H6?v^sDFB&!p|4@`T=@w7^~T4t?0;S
zkEf#1N7QY*QRH@b;b(7#ORN?#$61cKIQ%CWP3tnh1f6q`O&>&}uaUd8L8b&c(##Dh3$1X{
zq_Fs7yr8IQ!mKakkh`-C@s~;KQp`p?q%~GB;~*mOQ)*Is%}a;2l-)P@i?gSD3*Xp^
zmaT@=ENBr>A+FgxgW3LL>O||&GBXA{n4%C%5
ziqjNt@U8!Gjw1X-CgO$O#0v_Lckq-~UN-?N2q|XI$;r7=IwC(5!@rXziigU_^b>d}
z|E9@HJG^}Yn4S40GA#^F3;57lZi(W?3~aL(RrV7-WL-!Mu6@DX!y4wYM$p@))vh*x
zEKGL$HxDA~sWdU%S3BtwX_i^DtbALx#8)yM?O)<6TX6gXh+kr4p%-Pg%kfHIei--C
ze|0?1GO=37JMFvynTr-4a^cquS0;E$IW%k&W!vI0UbdY+f)!3Hz7E<@@!E9WUTeo*
zX!MMIjo330M+@U#oW2C$Kb+guhceBA^UJuv(UD*`C0)IM?_ANr$};h!Y?YrK7Yr!2
zn~-`(veTnY+o1nm){}l~E+c+XI$QruY%)CReSMj6{^(#DzeBlsHZTF?soLz7{5+aR
z+wyr4C_Nd@zB8D-%sZ`7&{6g8@25D;|LO){y^OC$6DwnFwz9SeMD|sgWB6}
zfrd6F=6rJU;+t1Q&{4U2SfyEWm12{YvG*Qub@A2{;RE45t#?rJ?WqV*6hV+eT;
zR>?l)Y^!jn_~Jvee6oVqL?>?uHS-f(4x_&8-1P@=!Nh&H3i8DV&okEcuT1^
zkftfSqV&mvM>)|??a?R;_dOHd0+MdLhU$oUlTesp{`l8vA8?>{)?#gN){}}duaak8
zvA)P$ChS6pB|Qc+xx@CM0H#Q?4#hg>kbsJ2>M@M+T-Mw3DXPdV8)E)J!}~F$0J$igrv%m$B(rM?oiX;fqik1_Wi3Ir@ugzdc-+T@K90jy~YIx&F7RP<8~vq<}c)1m{jYofgdR#vtQwQM`jd29wOxQ}dT!&4Qx0C)!9
zNvH0jS^fa*XCbd3@nA&p><1hvBL+khdZ6sms!31#kWE09eO@^U5$?m>IEZ9ru)U0eX-
zW-GHsyvTrM^idO*s+*|Ij~HvYm*I-@YG(ylCLWPdOM%RrdA+4^>2Dl)QiMZrWklvp
zR3S|J3s{|LU+&h1WYW26*5d}j*pPSD6L}|_U|U6L%lF%rM3(7e_$Ixl4F|^LbaArk
zWoA?)qp@3R(Qj{M_-yu@cdnpVKJ0t{2qr!*mQ$O$>-(>6TT@z?0b{~Iujl(ApFAl6
zXI&rEi-D$#ZwZkp2OWI1Qzq8D8~huJ?%PsigfXQ~&)-{mPn@zPU^l+e{e}}G;ri9a
zhwG94LK11wfE8lEf4^3@nH_=@E6>5cQ~v6a{S1#ygpgiUD1RKQZPcUs8HurK_(g2;|X3grYi$5@(-?LPh%K5Ow0U@Lub
zc)f|Qh&;5dP+$Ga2GX@As7BUKRnar;p=*_ulllO8R6i9`t*T=k)I>e}2cYnrx1T3Y
z$1t^(rAxv$8Yfz;qLw?Nhg(XI9KclKI*C;|$m5qM0JQPP7bgtV{1%>#$BzFY?ZTn0
z;DhF>2)jCI9_O5TFHXZHczXj9cPxO9bmhgL!mq-$=NFatnEiV56H*x2a4?3%FR->#
zArb6(?uqA+8MP^^d`rX!u}%OQ;)$imcH_ruvG!S+I59Tz2*5?GWaj^TAXpOHN!qLa
z%DUVj0Za0T7~|Q?2dAg2LA|_8iAgBjX8iNm+CSQ8jIe|qI~AchV#R(~Y{txCd6}L<
zg0QkbD#ltwI8&#{NC&5~`iQDyjrGJH@k4kyg%~zL5Vfj=`J-@{s*qgE?Q;W8habRU~cRdR#t@fEA<=?jHjC>tKF-0`R0_R<4&xnTceP85O4glDl2x&*(UHWQ@3e@%=wr~>_&?lA=8@I
zsPG`j!4ZWUK^PiBhxzr5J?6UedFEzyKs2Q6ySw>%+UjU!V#ej5xLQw#y<1GPrkWqW
z{9)Z@z9XSMXCADS?8#>3qpqxl1&s!Ektpb@}kIpitGZhoZx`I~K9H=EMDj
ztrt{T`@Qm0T1ik&cyWtoI`>rfB6IK4ZZodu3#6z=m$Of0Apav1$2cjM(O!TGwD-OS
z`Svl4qT!14B^VrJ7UAqH@c4NGy~qzQCnsc@7YX>0h)&FW`kvq<&(?YSRDdBgY&}dI
z_K<%F@v8c=!q>v=XxKtDJv;2sRy`5e_#Ck9hRZZ$pP1a2mO6w9a#fCLBgfN**8pLI
zQ`_;ffr0DoWh&~`%cQZi$2@2J2!vffjK_7}jZ)Y3&XUWk?NynvO8#ekV-Yic-fcY0s
zH9yy}dR#3w!8DG>Cypr;PPe<6T;;eCN67clvvIT^7VeLq>-7{p{IV^8p-#10%_PP$
zZHL*liVM1C
z?t}@K$0m@^T1X;&5RaMr+u;1lPv%kO^j0;W*EZeQQu}@@4P}e$q4b81L!=Fz`Qy&%
zD54;9Gs^X_l)@t^<|oxdIRA`~#@4`r7buZAH(1Qk!Dm|`GPe}as5TO)ecdd7OU1Yk
zD9=Aj3-#T569+nwvwAhVYRtjN4tH0GmJ#T|#AY!(n-c}*sM*>!?zl+n!Flix28II_o
zZ=9v4>M>(sZI-+DJ{M@@%|&PbneQe_y6|6^%hI&Q(>GD;zbUSSYU*1*R9o7bm0yI*
z535k;m*66_FGQY9K{KA(&4<2U*b6M$)iapYC+}sh#~M-yaI3R8LXC-4@+0hUu*ApB
z(W^3m*YC)Q2)fdvuNrBui`
zwY95XAFW)H))l6L>Vp0L0C209J(oYzQ-9U|WNPrNkO6&#_2WX2hkfl?_r>MWc*eU=
zz?$vQg{}NMzN&cQfkjyH2_-ZE1!fuNUEL5#p~#uWVwF0rhz&(o!EplJ<2^I!{`D`H
zHo>R*x~^3Hti$wEXOX(QK%nWMv%iY)Zdg_3Dp#tTXbn9do^T-t=9T+dpPnx*&L2Qr
zt>#(HhyHqt03Ty~At9!w(Z+1B?M&;rt`@s{cSl1}sRI#1Myrnwjk+P240(&veD!0G
zFU`3|ZEXZeW^|UMZE?zTtXh4n^zH7x{$TcKFWQW>9vZ)#6CtPNp(H%0zZ4!1U7K7B
zKQ2S>CcWnn1u4!lyW1nucDo#OiS-9)IZCYL%ckxU8x~-=>
z>A)j5oM$(~LY9#4aK+L7RoauI?7@}m-JOF{Rz_(P%vhUtWB_`Zx2BLFgj&QODL%jl
z_YWYco0UxU*|8=;H8SN>pG_b7p{~F$_R!_;vSmdx)8OZ}{ON{}Q+8lb3X!|R*eN4!
zRI0(!Sv$H?Vd#SqFP=N9P8Qi=EN7RBijW&)&`2#HOYb#m{epyszAbLcSHAsy`J*7~
z*$9cRah5)b7bZu&;Y{WXL)YZ2$iRBAc_8%Y_C7v@+s*(fK-RyRYYk0@y(t|Ymuq->
z45_l_2$bN(Kg(rl+l`i-Q$kMpyj5L
zYOvpG+;wzt1qxhF;uUal1!(yPkjNHW@$h4y8G&+=3~R5Vn2W@bK!-4K&LVK{?y7}p
z*$&;=g__h5cC@cugb%N#Cazir|ECgHSG(!pMAPkEjbVk-CN^!s<UN2Ssx!>$qhvG!?I?wH
zb!cft)?{*ggF#(&ZSf$V0x(}z+RDW+dUn|BmK)4l*uS>*9Q)Ep(rZYkJArQ?p-?i1
zCMCzm$2LCM$~={}m>a1{?K`dJS8B3o6r=|ivwB^oi%8M8g_VKIMhhJYan08$_5e$+
zwhxKV<|IXi$4GCf)}+fQ|J8Q?^~jKKcpP*#H8KmZDP~{%`&n|^npukK2QulQty^~f
zwa-ACTI~c&d(ZuKw3o)s@Er;iPy6xNnxChZ+lAL)KV>uEHD!5CVw314jk?w#_OGe(7ab*(wzoCyE;~xP6?p4D>C|s*=tytsBq*@<6x_Y`
z$lI=+{w*ybV#-&Q=J?x4;45Qfk4+D4ED9s{a0a%R9m$}JvD_lTj@eeM8CR?OM71Hj
z;g`KTp9_di8+{C%Oy9zJ!++qDogmcEpd5e@kBg%t&Dd?xEUc{QwlmLt$y74O#=_;;b?&P~~eoJ5pMiyQrNqB#&(Y|-BqxJE_x%Xjw
zhVp%Q+}%ICzGn8^s1T944(RJqmZ8;1*f+iRKxcyJRc@IlgKWFRe`@lSW*KMkcBDJ=
zf5`P-8fnl+4De?mM6bKvB*yFT@*$RH>e%#cb-pVF`i{Ivzd27PJCVwyWqFiX5lGpf
z<*5Dm3SUgZFg5FJ*WS%kz%Aeo_xnPXzQE{EwZpIt(7%kYS-Mq(Tf7c0&?s$Pvy1&i
z=ntRW4%amECKpn5%S-tFNf+Y#LsqLW57shR*=u)ErNTgNrW?475L
zA)?S=9p5ngM66sEC~JiWi+|Ma)OpIx+d&-_pt5aaqP4Yt*;?zek7Y19}=|CU)ERN
zHj_RD8XIhzuOjE6_>VI@ghwRHQ=jR-<3B*dv*p~bHC{b@ZqF{84$7UV4}4oU=5m_z
z$*roFZQt77{1!=>E<{XiN;5Q;HQ`M?)NQ7}%x_ydGBhJT8fCf6WwyJd=15EB
z$Ge5Y;CG_hqcW5lcWNKi4lqSk-24f0!)w_~}?|MXQ9kB6wl7931ws~z~_#SVAcb`{ji$I@DL12+Zbf9l_t6=)6?ymW2s+aQd&5y;qj2Fe;
zqPGG_2@Lb{05A8H$kG#GtZH|4n-Md|3Xk7Jj}`57juU>IjT;9?@Cozt{*NCwTuGe3
zmOro6z69X9e8T^2_sK0i(^V|gLWX7GB^P}^>apT4QqrSzdCmF8_U@il+`={_N=wq&
z9GZ9az;Xp88zx*Q{forHvsiY|>7=1nHzLPuNuzter;eeTAL!1Nq3^D9(?g!Rb$ks(U5L?j$?~
zXrMh?sbAX!1N(9mXir>>^gA&l%sEF!*`$l{!Rxss4c;7|J$|UM<-S@cZ~Ar4o0&U^
zW9kIVXsbr$l1W_4Z%aZ&cfD(~x@p4+ZEp%&aC$F6U;DctsoG2Ad^D+LUIsDn7s(-P
zZF`u&s{KfXkh4EOOWd33%Bgy{rL>P^?di|CBj$i>3d6+gL!@jV-o{6giVVsxiRlak
z`mPHuaLsy*W*S45c5{4u30PL-YTbfEQty|%!6J01UQyT!#Z%#M!`$vKpT*TkI=tvV
z0gDB&4jvV<)R)=3eKPn$pFw-yfHcdHCJqZsbE(ezv%|qUNigR~IX57fA#WJE81u5W
z`Te`B7wSus8NJQON4j8Pm!}dvByu}}@gV*nq|>ZXKusoTRtOVnMYLAOlE(YH+tVq5
zwBw3Bg1N*RGas*w?e&_xtMgou+jH`sp9{<)?TPbOXZqp+MEX;VzEBIav;#k4DVUA^
z$O0N9QS`SAzD&w1=Orsb_HuIY2cY-2p8QH{RYM#=$U2IH>o+o60fo~Ir`zauD+D21
zcWBvbSl;$TVLsUXib_U8q%WM^#1N_uV%Gj?Fb+gXHARbO#NsB|RmX1HRML}A)k>+u
zxlO;%VBK<{`t=4BsH39DiXU|tn#fl1WEq`uX5Zcf$gN}?Cc`MN6Z7>vMXkUm-mI4H
z@~~Jn;s9^~!+~s$a$0UF+uQBEO}U#gWSfpsW|^JIA|fk!Zf?%$!&u|;HSLWxj6Cn;
z4tlg?-!%;BdCKpY4dEf(+F!DdJ(b9as}QR!6oUEE2M}L61Wc0cKi&~xg6WGEhsCe5
zzRt|T7K+ro0!=0Iwr|zUIc^taj!Qp$eptp4DsKO7hszwxHVsMztCer2
z?E4-D}0d<%RKj}1~J5?HG}#@F@oTh9GQ@Pv!L%`BfLCF-}vA0)DSy~P^1jd*wahph&(
z;=ia#^H|LkecB8Q3FYW-e$;U*u9MbDB<0ZhwV-ff@QPkc3(N*dT$H~OXqVA||7MGU
zIX>y8fsN$e0E~w49&irwKM$MxIt5PywR@D@oS(c;{7w0Nq?JI+_lYMOLw}aPY>YNc
zo;y07ZJUeyi(AYn;b&3jk%NC{%&hG0e`#(s5080zNFqN=+COx4xj
zt8okvy&D@F|4ZwU;R|+;gpJsgR?5M@w8D$l`76OD^y~X>oq-=}5-Msme;q78D(23m
zlxUsY=Lo^<6ic9L>D}ASR1k`dF$vqLJgQ0qr&mGr-++^(2i*{>3@GKwl-ndhZ-3Js
zC(Q@j+8gR1ZW^%wB!h$uBZnlm(K5%-6iV~HuP82&dk?4O!=`&ep(FY#$`Ro$mjCXO
z|B$@Rwbw|+e4}sf7%+6fGWDxz(aeHhfOR^0L0JE+TCY6_g1*#DIn2lV5Q}xL?0@nE
zoH^AWJMe(GH4z51irmtS+@c=$p4F6zQlMuSSQ`r!vO{07*=6{2g+d$nt5ytLNAAGk
zbhtKE)S6Q8;8Q!rJ!+v~Mm{stg;(2FF9I0B(eah32S=$?nHxc++E*g2M{*2cGd}@y8(_l0&8Xm6Y
z!l0`%mu+rYZKbRJS9NBxqGfuS5&POxDWysfwL&RZ@UJgBLLwqKNi@S6L8+`pd6@BiU#qgXyb)U|uE>M7XMuSiGDMrx>ac6#M4K2DdJg|0*;rye
z%|5>lE@RsdzZ2zy*lIgY@@Km8gQ>+hGP)$82B#V0s;^;Y`F^ofj=7GcTx3-qLqG7m
zP-vF?9Zkj94mQ9bKPef-?lozlAT{vBMAg&bLV;Ub8NdH$cX~kt8cwW`F`y46!
z>V{_W*6~_su43gVayj1C`=$A-BC-8xLV5Ag<$XjO&4Wh;_xcryMlrech7jv=hys0&AI@M
zhBacpFk4;ZTu~i`zeoo23R)|
zPAlWR)uT3ZQp$F|IG?xx@{Gdx7YXd_9kZ5oQ%hz-5tIpkH*?*nFJy64>tn#f;=-X*(4o=Y@A_p9fXs28#vJuV#
z(WsnnIL2t+jUt}3E6L$!s(dY9cmCOlYJHOOI_gGPL@3`BHT|
zuHrG^l+;i_Gp?jEInKRD(LYa1jWEG2){>3kAY?>|3)cBKBxj{m3O18c7#`f)qqH9`
z46hq^OkI!ahxL^l8Mt)Ws`%cBPLpR_mUh|(c{-L3$_2z~((}-qDe>H9*V^%QaWAo}
z2?&bTyr$UOS5@g&c`spiT4M&9up4+oUZuV-I0HRrq5IUZTeFAjb2&$=N9E&vCwhV`
zBpQlki}K~(^R)g}j>?`GMPyhU+Bnn*6h#!pB%N0NWT0{Le-_#7LT#nAshd&dJ`)FLcW#-L^|I8;
zMf6@-xQwbdCr?k`f*Qa4tfUq7cnbfrwZ_j`dx$FF$9tBPcb#mtSXrS;(p4TWDrj0G
zqADt08-JpWf8cW~&W-!fX6U1AtA@A$(SEd1zt&-xC
zZKPyM=OZ}x#KV!RRNaMM-CW?W@ol_vVGp(%Tsk1G1nmoaeIry-#>a^~pi`0LZ}$w(
z8@#@ms61DA`)vwf#TDNjzp9rZ7=U9bdl@j5Zu^_cnQ-6;A`qhFF#QV6d4Yv|k1&kE
z@9X4)hfYLye}`oAn`9vovBBylWv9=d2!~Of^Yt9G#tqj*%#l`sdC2OCSFamTUq>tK
zb4oQ5Eo%rIY95w>e2-vD%R&tijaQyOQMW7Ow9bb%Oey{`cSF67ZtERVL3Xzt*j
zl)3Y5<>oqQbM6EKEx-ebP(A-*wwBnkN@X}edE%hNm)pT>bJ-sf8oFeh+F217bswam
z;_eNZ5+!U0s+eA(Ui@aip4{&kX#O|C#eazw;m=F$hTyEUwOu3z@;g_OXXHWR7WHep
zE%?e-akA?lq*)g9udz&MmV_*fw5|5nH?&HbZSh}^w!E>006c3_8_4?2gUv1Wq%4QP
znGx-S-x+ug-Xnjs)E`MYEv+g-gqE7qIp)P!Dk$kb9XlmQ%WZ5c*|sM9?)Y9sZg!!e
z8k&eq)sStVP|-8IG%vP9w%;niv?WKfZ7l|ndtJMVg1@RsD__)Zhwb#So$u33Dri_r
zySnku(jfZQNJWF(yElzS`k~$Tx9Y~o3r?c|d$;uEz^
zs$~i9$@{R$CKi#w9V9Z}GTQ1B#vA2OU%Epk3_9kPM3kNtJ92qKcI%@n>O2bE(utwj
z+XJ}N58)lwzXT$8Tp)#|m)B3yU04T)rs1rchGmG9_bvK?M}rub^50?>l`q1E7Kc8BLQ
z)Ot(zS+0s-qI%d6TZ=cgBSwc-^EaYBCX~AV3$u#LG&3PjxkhORL>Nffmd)(tt6U%7
zONDft_Qa?gqkqg2pzo!blo{q1vene^}^fT}*QtAgFcx
zgDT`_N7*3v`*{G@>BQESIESt-Spo~rGdet%h>^k|wG5OMEz1h700t^)o?KNLt+i}>RXt`W{-I)-qQg`OqbPf^
z(1%kq?RCPr;Oq(>WRS_c=V#mJ7%Nt;>^mBQc|^8V)R56@3f2eqb*eH^r(8WJ5)$*J
zS;>_HfoB#i2WcOlf7flooP6~WXQgN~WmiDfPHsq3T23TTK%Eu;bOUfe9YmFvJIQ+{
zxS3$xQpEFA+0$s$DSosVz)h+?RK;znkq{CHR*1Put<5sXwk*hf#3x<1<(K`Z{zN#$
z-Qmne2}Zv>&|Bhqn)xWhYV7PIhwbeIxVD`xdYC#Ki@yh-XmfTwmzuXyzLz{+OPy5Q
zF+hV>NWplk$7BG}5gB^baz+&-?N6o-w-%4D?G1)Re74zYG!jg>
z`jAdl3tJDT48u2!2<@O^61W?_Qn&(f2jr!y*kS^-rRFoMJOy^N#E>gfB2)$4s?5H4Vt9f}1fTa5kFtmmH0V`Q=IPh~<1SDEFHsLMSJ
zc1GpZo93_OOm?4TsP-*P5{-#PESjNwOz}m6wLfcLO_5w1lV%$LwLJ27EHFk#?V6(Z
zSGM-j=?EVPA0q21LttoMR2gu!++5ue?E^e5oC~|q
za&vQYdBJ-;cq1nwvN2>k>0jiFR
z$KZIkAWUF^USjgIWv&Wp2m~NIWM>Gx4D{r)
zDQD=rCI38c=(s4%qJblsKipy=!m5OI^al}4c*CuYjzo{zFW4(_AX(2g^J<&XWf=zYt)aN+BHamCi?@cUjZT3
zzqW0G+_msYu0$uXZdJ~Cs{3_%Z=>GEfGW5o?rR#$Po$taZeVo)5
z%WC=BY2bz(z|#V2jVTaRQ++(1kz$m_k{YC0?G-h9A34OSu9U7}v?@6+cH#Fr1vtsz
zDW6%hr$L6llCWrgA8|J&AnT`d+B96g>4_s+ntLcH$5q%#SblA)6Y$;iRQW)x|4Mr4
zG+^M3S~_ZNt=1?B=7`mBGj=-atCmM=Raw`N;as^_oC_UG4ItTWIMFUI!3mOKuhj+d
zX$jItbq0mWa&tm#n5>QD@-)U1$&
zR(2MN>EH66ZN<>j$Exe#KkepNll#XLQR}
z1AA*NBzW}3wdOnW$0ZhvC#sV57B5FH+S_|}c~(qMN7Qq^)9LNeNswelf2Lb&hFo-8
z^3>Bis9f~?a9qq0Dq6Yf7LR^#tlH1eRe(f~oNa6zu#Vzs?PpNU!MhVs7z*yz-TrHY
zIFo-5k@bWfdoRr?sL~v<4t`Ne1ZCI~eOa7(s*WXw)gsIcF~>-}*E3zgup(cH?D%|M
zulwvZQfoCag=3?i&yoiRR(wL=8@7}T>%Lst8gVa@P?@Sl{+G;H
zCp+@=)!4gWq{ldL9)A-jAaFQ|8V1-06_+@6=B+XK4~X1Fh#%mTfT{hdtQK?!M2IkL
zU8Gt?Zp{vaT$d2t=77IQqLqQaX)t@&1xS8>9&M;1ckl?={%l2TEYJtp2`XhT{kCA_
zp*d7#tPF-g-ERc~A{%I1o&D$YpOvU>2b$XjdDVWE)^S`V?AC})b%x%&nXi2?oNAj+
zh0KY2B)H|!G^ZIDQc?3*3x96b-iktw+&+m>oYepKGrHh=y7RJka2g-d950hjd9E7}y-
z=gWhYR9nW#nMqzpu+f;CH*{flWZAUh`FdJD@|rVv$7can-gG^c$>Uz5!Ed8u7$@FS
z$2K%o^irsn?txneV}<>yAy0dlXWd>SK}Px-j%zOBwyhPnhx!^Z6ZcX6b{yAbJ)pU*
zxI$}G&t)0ME|6}|fk|z>4OyUYUSG@O%*75<+4Lcx^;Tqh1vg-FvX8Pp3X{ixPO7RQ
zqKJ<2q78&2l`GN-sW8cGVYjD~Ti1${X;djKZJ4=Piy81&sM{TmVsO}CI?hzml1v1
z8oTKEt`e}m!ru$dB-|kH8JrwA2OZ$DpT=z%Bn!6Gx}I9<+EfNmq&_%fj8ugONaEq~2E^o0^
zJZWut(^%T!u7uen^%0q8CM9&sid!wa_a&a9Z>)YMCWD~>Rieo|!pDuRm%Yv2FHcIk
z5*pvG2a@Xzu2$2#W#KwgP{rg4hapTwkYU}kevO-5?DF6?=TNUj?t0iO!H=?GugaAX
ziu{uH-D*p!>Md?~x~TRLC8JzBg`S@>s!vub-9s5A#JIgHw0L(?%}CGjEbpT=B1oiK
z@3?YP_e8jd(Lb;?W0j#h+L!UX6H{I%Zm3n#9t`B$%)9kkx!ZP_l$D;Gd=5Q?fl|zl
zdvZz3kt7&X{5lfbUQ4h9wT+xDH_@caG8a60tVC(G9PVa(;p81g2;A4o`N;&vHwy{j
zdJt-vSax2_<7n2fTA|aft!Bp^nhIEa2nZ31-?j-FxKrl!JlJ>+p?zr=e+;V$BEgFv
zR8a{SX(i^>(%#XGInz$feSPaQWl1WPV7fuiV6Sq^7idQrj=F>Ss^v%4bI#=C~Vii%-pfrJ2+_oy_TuX?wn
zk{ie!@?u3+U_r@zsD`uI89k3#S0LO+?SchIFl_ItGio_z^jdb
zvLClnk>+a&?t+a!CY)|Xk@TLj?t|~|q}KNyp6mV&BC+Em{ci&o_5b#7|IcV1AMX_b
z9X%f*?*|Q3eg;z_m-n*j|J%KNg^VnNydJ|LbXH;&k5THtx~8F97$2Flf-n$%YIvaC
zV)=>X&ZX^}C{pCh)SehC9F47~8GS-iNa`-^TS`x!TDQWtKH1?mU79q_X
z^wS2ca%XHzAF2LDipE@4&Lq9oYt9>@eEC+ooV-X=s~m^B$M*?L06+-*Hpx&I<1aSsW1Fh1j?*`F(XawQOkn2HlYh7a|*
zNW#{0RVFPaY1NohAc>^+_
z&-z?@;|%3Nt8C90uupMETM;v1P9l`!a)fq$uRGZCAgYs**14Uv{-+J&gB_e*D`f#RqGd)giU9wj-XvdNk5$%Z{5FL5JwZ?UUTcB4$jDK0
zRHg`)95`0J2NQD0s@)Y=y4sVx8_94X5y}1bQ;|*C&+Mql0=t0BhSt<2rx=8t{E}VH
zjiH`G$l%qEAMwCdksa)~f~&MNrfzSEf&hM;=C4ODJ%mtcdSHx^w2T=H6SCG*i-~Ib
z)g^;MNTUrv)f|x91DJ9{M8!rGJ9M8Seo@B~+1doHp&I$@1VG}9KRspU!7!aGhv)mu
z%?7*n`UvW;h~PnDay$|%S;SdnBU`!+5tFQE@g9sFf*>bC_v)LAThf3*7!7Zcnebw=
z$>>KDLsH_%Lqr|``kVSA{9JBtT0OgeM8Kc*oE{P{*vQzu8%$z5i+cOh-B`5}2OMKaN%
z4uTh&7y9I`C}HH%Vl0V&njjm$LvOmv9bM#Jw4ew(PMO8(8BTh3?);IsLlBiiny$z(
zx(=r3;Z0!hj?dj;r-`#rx>~!mQL~bL9NcSjS^LE|+iW3NL{OfanfFLfT?wl^H_RL~
z?JQQ1heQ>I4=FCX6gx1!afc*^oHo>VQS(8tB%5iyFZIa8JRwr-=Xwf#T!NdI*O?Q44xdz9?(NWuPi
zr`t)5$+>)W3k5Si(t3N(d1O)5Z8WHBQdCRwVdKRw8rp@BgLjUf3O=;1R-nDDo8NGR
z50GeD1T0-O?nhVsgy741^)W=$!`NtCtR5x3g&5mCef|8rLxSEx7f)9MTRjtKf{3ThMsrBv?K}HGAaG2+3BT$G4_O3fr_-J
zp0{g}n|IQ$0_m{RfM}+D4H#)w;R}BC3l`%%*~(=ePWMvGcq)!=Cn4wBXkY89=qaZ{
zrEggU!%E&NIPsCwyKwO6@~TzXNp3-ea^>ZB`wsNVdvkbvcTiZ^n2E28DN4bGjP+oM
zUtdO{Oi*|^Xsk>2<_a1F=sWs^LzS=AJ87m|tTwpMgd*y%9+5Zb>7xb?L
zr8_w;K9Kd+-)nnv|F)kClejB=HbzJjW%5
zunxs51Mpdkic0dT*HnC`sjyK`K09xkIQ&PWZ2-1VTL17xYHDhF!{Fcr=R(c)V4Ydp
z>#up}NZ!*F0%9@;;3dzomtaGIP
z#`e87pEevhUH>A58Y93^NAFk>%2~FxTjPI()M+%VO7-K(*?DyOi)8l?wH*IHYLQW2
z{l8zk{D<2ANc-wP)H0y*o4OGFL#^ijSc~!>Yi9x(qjdrFu_Emp;;XC0b8tZju`Fv*
zh$ED)DJ}QSgOrek_a5O6RHL|5Ls!|&ry+*p4Nd;OQC<>UVK0_ubWlTXF!eW#0cKhF
z+q=PWn#y5S*B|U`GUcCxf3QQfQv~Ty)i9q-QG{8Ef`qb)V2SaE)f$4Q)&J0n{mv7h
zLv9xmCViRsvB-iX!wsUX6_BV~P*ez4R)y%=;o<;CpZ+3QYpIdw`E)Ij93E+|g3}tN
z8Gl?U{Pe)q`<wBuaRjZjq)n6`xkS^bzMja6)c%l#8~hhZ`OL8X(TCOvd2dSF|j*zP2Cz@uL9*F=-Z0
z##uj%Ki3W&m@s(F$|{^8IJ+BsT8Ga}Oiqp=^nc
zy49;p67|TrWx6Nz8R0`vXQ>vWH~{t=_uLeD>`osu
z3L-QLIRhGU0w)ri*}anHw~Ax3oPlVU-loM#%?iaoPMz}h!)}}(IouUGt^mYxjFGX_
zVA}RrP7vof{~i$^fhXdRwvaOTFpB1*B=;2$fn{EVyw1Q{)XUGvmijQhSX|4b$bN`aciRAc4q;)g(85s1n-(mJEtjfgX#t2yObC+K*F0pBbHF17J%rr@8}J#
z_RYsr>y{5E?l!b&r~Z5*?bVx@IG;V)Rs$Dz4$@*ch^z$(_GtL!EObOGPkRgPKrFjG
z>Ye-6O*Zqx{&on;Le<-8m=9xRYb)8UZI34L4#M4-WUG@0L)Y6Pkzj=0-UbUp4kUdP
zX9w$$=_;IlCqyH1Df-M{c&qrflCaRVyHx?|(16)~DGzL&+{=C-jpqRG!u)uMKqF%Y
ztaD7we4fI#CTLx{w|^wvZy57}0d>h5t0e-8%zJw}%)WEk(A?M`S}mVa9OC&y>1S%(
zZ3%e-&KiHN!@yB;6NmZjOea2K{*1FhRwY97q)9HPar4{S>O4baeTTH*-;
zA2@rwY(*Lot?1gY=l{kbMJg7T_^hnmzT+Svtpd9GPm7p5eM9KHHAu-kGM3Y%+I>;u
zOmdJ^sKtCX8?M%nRYc*iatel+mxQ@xl6A*$tRXz)_6ryqS(&{{t~wW#8?H*E$Hdc1
zloCf9VVv)bsGfEtg?U<}0i(CoI*s6j(K^?mq#+{L2da0RLaIa2;v}T0e1Aln0&^X`
zyc?W;>$)`X?X4&gr>+{?0+5I&Cwt340pvfEO?zP_S
zJvvC!{W#bTdp_&pf{JC1{EU%OVnSafRx(FLMhl|EJ!x|mWg=w
zIcN8_+8Ew1pUu)&K~YyT+8t#^i=L=p?j1O^JEiJ8GjDeFoO=E4FVc}`2Wxk?*?4za
zys%z*VR)pREQf#0dU3}s4G(Ec%wX}medl;A4t)4(@sSx6E-r+f@=mhCk2q%1!CM#L
z2sB`S(A>>)As($=DQWA<0Oi^E>=+eMlC~YT&%9!UZEjC}s(&{!`
zc>|)iUunziAAKr}Y`Kz6Se2v@P^ChXwtMy_-*bqMV)I54awTpm3gAyDu77C_|X6~62Vmm
z-hEDk_9&5dn;g}d+WRHlT7dAmv6bUs235AVxjeQ8AP$dHwfnnl`xz$P-fO!xlZV!Z
z?q4f^kr1@?&#h;km}^MitXz`%&gK8}XhN4k`HN&~L=Y{e#x!e$m-9^p1OjQ$JLrSLTo=}f
z48GI#0cn(xDywVja$H>#<%TwO;$ya@kaykIHjNfpcgDv$+J3i#?aOPR5#{PdA->h}
zzekDYq0mh3Ra+6))N_0r^{l@3DhKFct!;v~8*r`enWM4>Vh|F+aWzx3^Qjz!wfY|6
zgdm3Js@^f~G?V8a@yi^KXTNs;fK-G2Evx*Gur~QN$EAh|oQagGEK=v=Yle70WRh#)
z*L*!wv}#vla$^oOpM(5|cA{6c{Nx5wgKUNOx+Zf?T=(Bvr5teB2N~Q`9p6+PXe8Rj
z6mwYk(x&8z8(A2HJ64uzKkz=J(w4~(xk)ax7v%m@t&l`8oo@ItP%t^%ryJoiaXz^t
zY!Ifw3TTa7axgjkB03Ff?PsNhEXj@fEkeKWU4gk^V`khPFMXq5s&+X@Ox2b
z@fzLx*m>z+NnN?Y_Kp!Z>@KQ}n+|re)$DNSre=XJP_{_160;Va$x+b2f~{}RV6~ie
z$kfKbm{bEniZ~nKE)15ppn9k`LT`FcqW}q3xIQVJrYZPMIyu7q#~`F?5ZsUS$oQlx
zSFSSt))ftZQUdJ`qj^cc-bsu_|Qii0jrfpu~A@a|w$66RD!XN{;)_87QZLu;A|N5=GkX*0s^HGe4iK~Q=B
z!3njN3-76GT_V_#(nSwgHJ77!z&yZF={%5}ah=8Z)Xp&1L`kF|#Ed8r8h@6@-F*Ej
zWym!WtzrDVpgK{W;v77+?Kw^U29&%!H6Jx2JJZ)}APTRHb;DZubO&Om;`j|7O|nZI
zbS6BMKz*PERk6p_4S>}v`hfm*!xY>F$i!18Cjvczj8dOyOa+UcGyuuaY7RFlasBeI
z#xz(F3pKi_?AbtVM7$0>n*^&*A8Xz}pQ=v!m!HfVo87>?b)wuCH(dd1kv(!C$q$G^
z$h|9L8T~r
zx$9TOEhKo00xs@tWX1M91~N7Yje}N$#pDdi>Ub2gJjUXt@93>b+ulm4OnEA=`m^xs
zcD9u^@6)0A05P!=k|0urydZ|7KRf>>dHxVDv_OobB5pY4(1(E_?hKQVu6GnPczgb%
z3r(i@Y9+4&%Sqb|@*lZsX#l1UlVhs_5)5yTfgc`?kJ|Nl>j#|s+VDS8^g$UfksKMr
zXnv80cg*=ELCc6?#OG&r6)#e0MNB`Vbe+}B!Nf`(4tm9uHJz?ys=n^gP~I<+bT?q
zo3s>?lPs9@wlupkHr*U%Tp!V;^N3P}R-aQU7z*yiz0BMY-YDFVPw?RKLAJ5wCc&&d
zsjjXW0o!-mhHDZ7v1*M=+z-ls?{xI&OjTxoq@-jFju*}%-r3c`Sg9ZvJy?D3lm-45
zPQq$Lg~NPhPX)>xACzs-CYcND2ds_1HA`i6a&?5-WKAF7~CR*OS&-xj?8B{iM{mhMMJpemmIrqm?
zUK$x`SF&F&0&NBzOKX~~0LSt^f4p>bbf(}N9^?+cad5EPQ-REJCruKV#az0&Uue2C
zqE>YlLxQSH2!Q+r?3I^wtT^SiY%2;miFOT_f5be0Fwri2c~IF)y6+oK2|Gi6n1ohq
z@fl7VhIY`0#@>o8R8>_{Rpf*;gnRf`r793VJrHA20T*ifaJE@AOMy=hMLX%#+WE7`
zhoSm(&Z>d+#l*lDs!p(erMT2G(6GE5x5GiBX1;L8l`L4UmIzK}qK?(YG`VsPZ;O>D
zn?egA*UAA!Qtv(PbcCi!9=VpJN*h#m@u(Y@C=gF`akXn|B@iznYIP77KS^*uby%>z
z&(r!fNT?Ti;uGeA!|^7(|7}o#t&I&vNG>+btn{Cm{%T?~tXjEEpsh{E6;X^Cs{Rg|
ztrZjF5)oB-D;U7S{ga;%&(D$2SnTRMR%tiU+g}6zl|X65pP^c*J-XjA_o9NXOD-Zg
zKg-~LW6e?V`Vtw4p*66i<;NU0yUsts-t9dc_ZNwsjRAw#m4dqD$GGC)@;m8$4o2zh
zoO|`G(PPyi?s9i4#yK5}*~l-gta%oPS>aqAN0&|Qwx;i=PJjFaR16XJm6a;sa#S5C
z*1&lO-Jmh}t1CLsM9*}SS29*(y0TIOmq4Zb%jKhc
zj=-DahGQxN{vvPwaBc4juV*)62ta!j#oV6
zZc`-}YV2VzDQ1D%Vh$X;Y&JSN2H7oWZ9iFA{Pwr5;ZGj{K>PBlzQ8veKH7T?4KU5v
z%F+TD$XU2lFLy1=ufcsJF8+DKu~0Wi*nmnuV(TLW0%`Yn
z>A+u9Vj8Z)AC+zwgz6Ph+O?}Vz{VUL99D+kPrBIg#12pf!tg>PJoZAGdFKmn6_9Tw
zk&!N`Bq}bKhA~y{ro^&;&6+>%|3w0`8%@WHfnUrcR(%!A;of>@0|5$Lc#?3|AH8WG
zQWt0bB2oQ903K?_-jtKFd!ws5rFJgLq=*nSqkvj+##LF)aW6sy$fr3E0?Sw)=}_(v
z&8}00Bzx?TAMALylU;b*Qyz$@*|wF!VmB?UAw4~fE)c@14voJz7oF-@T?PM4m1_=H
z+la1#n^-cFVVMH?eGAr&`h6J9WG#uli*T=0*K)!5oQ{VF_I2ffX4G7-ri8_Xg*tn#
zYUu5U=_jrl1eR5c15lCxji@|Vf`h&Gd-AYGxu~NUJI(SsxPhNr^q
zsmcYi9wK#L4b*~(DO>Pf?KdVK1=~e1lOGl=NG*zTekIihkXAUtnPXB))NfNiBO3vS
zG_=2~mX?+Qgj^(ynhJKv-GM+2tuY)hhPw4jwl4sVEPB;1MFQ1MEzSsfa+yD@+V_N4
zZmFPY$*a#YsnR==or9Ko$^IfK=F*Uzc@W&;f*Asjb^;5@6uGPyPs={yobXXr;il_n
zf&Qt8H4ArkD#2Xz_L}MgRIAr4s4v5KhrttH+MIT)R&_elUtf>;=Ed
zDt2}vqhLCWa2_f`JEk^d^xj_^OJToGSEqLV8Pln
z02z`S>@O^c*2epsnxAVJgM*H&IfuI`7g5S)I_jzYE+krSyY)z*G7yrSJt2jHpd+H2&F_Ns$ibGI5!=w8n1NoIatPnPdZmNad
z@}oCpubi4SOTi?$M>k&hc%ksgR>f44$)@4T<=HSqNZgk_Wy$q0l#K3ePW0EaO)N)r
z&Yj#Nv6YqJ;f)#WicGfos&f7G*%F6$Z0O}k=*`t4wgs9%Q7RHT$@wZknb^mo5$=0$
zBQ?Rf)foqA#~DLEwjsv1ULCC;_N0hST$&2{au?D+baD;LwEQ=_bq`)IHe
zvUJTZDaAJ{q6_i-QAcSA$Jf@ApWK57f0(Gea6Q@zjkLqmywK}W9q1Q#9#OS5h`GAc
z57Lx5$1=dNg$5FXc8trvT1A9=dne3q+PHY;HwltNAjZBo>Cnw^cNqrhHTtp*3GL4W
z2OV!LQ4;YvD~GwK*XTUx@D=N|RTl=nnrI^OW~swo^hPa`6x4jKId9r&>JH$Ue__p|Kjr=e<+_kWS9OP$izwOIr=z^_BREA}<4!4#25*u<<7lx`i|aw4W~DLCA0OFn2mrm*+T
zAvMBzOF&
zc%S$!@Yp3dohTrDm=o`0UHYF{-Toq_v#rG2ES!&(Y!&2zsen6iAB-yv?DvUxrK;Rg
z1S`9yJDx{yJ62aJ{NVn+;_Ipgl?-7kZPs^j
zHpW#ajSbr!L1;Y0kmT!^INYG=i@mP}8uni2v$>?$08l`$zv5SREO@rQXbXMo2nQIp
zT8T&L#zPnLfpX>f;v*efFKsT4M=kHf9U*uChg_^f8A6InP{XDpMT|=#7ng6+UruYB<-ma5Kcd=14N!DlovSs5
z?H#*x@L2a$x52WhOAGFhExdPpVxfq2mi!ShFc^*DK;X+59c6nObut_A=
z4ASP`ijn2=_@JQn{{b{W%fF!i0Be`GMM6TUm9%k5{Dry*>WBtHu3Jf8R!+tJN!knn
z)imJ7hguc$s_np31ym_zq_PiXi8JZ4FLBe^SIN{3SH=6O!)BjytDdAYDCectPsF)*
zKFbx|vMeSDDV`Q4v&aO^Et6oQR*~6ONkqT*9sMj$nUbf2Ye`kYf8kVQDPe*XK90)#Uk?*W
z?xacfUl2k3DiuJgtfnDhX|$C5Gs>faa;FIClpq-0y;c0D1SvteY9}fwG-?7NikY4O
z6IwYeQQ28W7)R>F=H=ny)G0cTX3nt7F{4r8iu&p^&Fnt$p*+;oj>`BVo#2!4BLNOfTGCxlqMgmEX#yc2>>N?UfXSIx5QOc>6A2l`#P1
zNU_TFRaegWd`5m2*Osl`8`T)`?B(jk2e?$nZ~ef9`>gvpQ+KgP2Ns8v^;nKQAs8yD
zyg^$i3EHku3I&3uQm?^Foc_?SInE_8$M&WW%naz6J#(K8Mi>npmePpFmot=6LjZ4a
zb*~hcIW$;Y&qVx4fNhm7q^JT)ZYS$0{s=XV0i&lAp6*D!_dZm)Q{)b-C>B-DDeS9%
zC$g%94^A7Zo(-aMy7e5bl^nD^(6pGcG(hnQ%rtt9){*ME@iBQ=a6?qPJ)>@5uYBeF
zuB{Cg3d)(uKs5(qK3sVPL<$@_F-Ni`vf#gI{{V$n_)O>2pvGB8)zKF@24S~TiVwtx
zV|D4|HN8&-6-SZZJNYx7oavW*ubpnT*E(egJA-3MUW3D_mQvFNNxG^WOlh%B;kHH2
z8$vBC0+fU$JEW~kc73z1S5Tua>X%zRT;0(`&~Si+RR-aRJj$t(wPGK(T2GPcR8r(H
z@Pv|~wZ!J{Ck@>X1y=TibUcOYRdS3mIe(W0mAt8OfVR;RRIhyLooJ!8_FjrB6{a$$
z0UZ%ggGUr88P66Ux^gPn%e^aWKnad*B-IzHB3;2;`ngNEq|kRz%XL~AMsN(Msp>(k
zA=rlJqAuhS#l-ZB4yQQ8w^i!Oq%TtVZTza7!x$u$5IjTbK2z9>ULcI*D0Wm^mhZxb
z&@HQ1JiYiU$5pfc0J1S`>&4NEnM{!iw-@fCfaF%LO;gEQO6iwVlxS-BQWXUIC~$+5
zqTDS#@`9+MD8GtqTOh&#>Y~G@Zwl&Sir^{u2Rv(zfM9K$@s?S4GIGQ8+rrpsFd%#n3pP
z(hqqeew87htvYt9Slc0_RHPl-DLD#R7wL)QyLMC0jBtkZLx5#vM%SvWyrLfVuyhK6&OO&mw-r@TrEQfCs*Fbd
zQSh1tE)~o#5%dUY{4K2?blqZ?DCQ@UzE#QNs-hB^lMh7K8#EUtmsXNc?ZK(_Jet4o
z6ltvbld$lPkpBQzb%yZcGY$yMdn(*z)m!8m!FLqaHp9-mO^P6rRZV&Yxc!3KG_iNi
zi*ZoZyP6y{QO8pY_Z7iXRYa<_!`XZn>g=jGDydWCNuVmf{dua!6M74;rQYR#=}vc7
zO`B!7Eu~R4y68W(Ru?dG3En0d)9it0PQ~e9iFY*$3ImxdZxARcPM%7(G(?wzJ7-1W
zG!ogXla`LnTLb9w|F;D_S2$kam&03t6GV`VQV=A;P6BF5UPP_ciC;jC)}XNfYHz@
ze47VvD{)>B!>fkVDZ#>{)U^Kq@VvlUU;hAavWzB^-A+5C^jmY+$S&#vm#T0Aw+K=S
z)5tHf_e?A&S2YIt00mSL5tY^W2^Q*dgjyON*pdlT-;$8;vC73;Dsc8x8SM+=K2Jct
z9uW6(_>VEt%hb0ySyP4;5pm&XX;w4;bb2UjZQ9{zS6R;R7hAH{6U7q;R;_;{FZ
z8bQDGgH8in&*jV~pVE=238;j1ndu73icPYhR^Xsg=R;Sfa?;gaJ*_2AhVR0w2;<(R
zs;V30P9cR26w3^=_d}{yAjigZn^7EDIBag-?kVkg+|$e^;NU@V6Dlm1Ho>w2P;-yI!53rn^9Na#~!{)7?mT
z0xGp2_8)fiR6(C|rX;U*8bb81KvZB|J(U#|7*_KfQdJb_0qvp6tpyih4K78l5bh{j
zc&V1lt=q9WbjpHO8zV?+j4)@S4SS9rEL)H%uf=e4x{63l`kKlP%9C2aM{B;wj4}iX
zhZ?AGAmv|>Zx-6E&eD#Rb*Z8o-4ptQEOlydTe
ze#=DcveSs|Kje;*uvkXes->o?2dAR+GhZdG#FNc|8#kivs1S^XptZMidoM}KYBLzl
z#116Jo@#(91*;Q#NQX2phQQSh{HPU$--~&b@R-n`xW^RR51CCLru8@bM^kirp@!O?
zXMkcbQ=b)hU@k=UP|;;3jU38`nkp&8^>oM`h{Oi{I&M*l_bU@q+;p~TP|-%CrZsg?
z=PlIsQzKBK6#OZ#$PNHcK1+E}(2ZBa2s`x^L&J!&^<*49l-^dRspb%q5`(=(Qr>3O
zsGUsh%}5ZO=bbGS9m>ONTP7hrj~5BWw##&fFO9qIp^S{{lmiQMo%vTP@-w>iE1aas
z_F6!IfSEl99jK0plDAP)fG~P_lo-JPaML0P5RN7g+tVWFiUii^4#amTvYR`n9K5Jt
zlD8UkR9+@4wDcVEs-Vc?QYJF@=ApikWdf=l0LL<&17gxZBn7&eZ8Mz$vfMyba_60u
zxHan1!Od)QlQo&DBPs|TN}Oj&TQVV1RK02YEg?&V7N#$MGrBsB`2^;)LtL6Q$5kcH
zaO{EUU^ytCvN8*gMJAl#2Ra2JipU?YE9KCtjv8>*fe6AUyOd&LHNWD;Oq%D2YVDlF
zt@Z4-v;`*o00cjqujG_MVxDM{ZNmX*WqO>g3P7UTH+4nE-A@Iwx`qp62;jWr3$-?R
zkiU2@-C~i5A#cSe*=z)rWo2q-egTKbE{gR)#*@uSsZ>|aT~m}bOQ@$Rc~lCj;c{G%
z+?&4Xz8i_9QP|>DXsK6sM4u!14uw|@a|(`JmDj0J?OUEc%X62)X!k4PViL7~%8gBx
zR%(A$FF8{>=Umy)SA$j~pdv6Eq;)EMXCiSuZ7_-fkX}*MM};8(Zqr4mR4Q9KE&l+8
zrL-Ng0wqp6zwJ?mAT8=F-kH%|X~L@bgIm=Y-@^EqV69qD!Pt($>_~V}Gg;
z@1Q0(RI#3!3rm>C37aiU!gw6Hh2XKcnsEhZnkC<-!~kLc&=kdTs=5Gk=C|EZ
zQA_QWJgTT`RO1-zp^a5kXXScA1_$8+?F0HiXkkZ2knWI-x}NHdBj1YtMKZdfR9oH{
z6>&*l`$Ss}$!)uRcpI`Bhs1F8}Sh;PMFB%X*8BlN;Xms^OtPwYXDcW7A1zC)xv*P~%TvX$(TYQ=?Tze_zs#Qv&u5NEu
z=LHvFRlOI*A(FLS{YtWQQ8+e^w5F-yrcxHXf3QS~WmR>45csG-;yYF!FoTCNJU^$y
zZ`8C-rmA{PvX^?7itkysWjuluRzd==k-E!kfm0jS(_V#FhMY1;S5=?F+IV41Q-VhZ
z14}^LGa1WHm8`c6YUUMNuAGr}lcTEBG*c1f+lHB2DmGN&ybj7R+b5}9mH2I&3E(wO
z)%)D1qCOFbHTGIDx3Ou#2e~K^bmBE`BT{h68WddcT31q_<$eR1#cOCTn!pa?n9fZR
zo}64rcSWxa+L}>wJ#DfY?gT05QFFxMwQIq{j+B-0n}&YKX%pKq!)3n>YQQz(P2x9z}Nu3D%{cJOfD>KzlA5CK$MyxOk$SNmI?C$if>wvr`RJE|3a2*ha_
z0uZkD-zaU<9_x~WnqZ@c`k}47%03<;gJyuXhzbIR=cH4Hqy?u0M}B9B@M4ES61KDt
zY~hu~b-OG~LCUUkipIddIw^J6i2Np%3=kH_5VdM^lsNsox+QHFjQ
z<-GOTpZu>%hXCJoEp#n7gW1Y4EEmJr4<@MQN021ITV*;M
zw5b-(;YhEta1PSnSGpzFIax!!K7C536;(K65J*#l_*;V1EzL_xZo4O8i}G4=@VAe#
zS}^cN9v|{5nL~1~1yurpxylY!R#RZMwog;U{4>uo?uq!1Jkv0+tE<%u72O&F_|B=Y
zLkkJ8*+p*jij}lOMUaf
z2Q^oyFxO6;;$u!0E&Kc(!r?oRpON;1I|%kY_R=C
z>-iJT=)CEknNp4FwPWu+){GJ1GqQ#+kBOn3maI5`a;il13NWzYZa<8!ujbdvk`)P+
z6jfC@K+1T5aMa#m0NX44P9RN`l_ttxMmLg^_~P(feY2NP^;)A-yXCE*x|0fDSCc2Q
zXl>L!;Xx9)x8LL=5J+0TPl@(gaj5i2rc^@BAVO?%H|Ut=nV;=NxGR&92arQXAKA&6
zjU+kqZzwUh?wy7Rq2+uJvq+xA0mm^gg^pv#MSAVWx1fOSX5eI`Bf^OMU_IL
zu2jl`S9PV8@00+mTvPomc!zO5l?2nt2cbmus+^oTw}=)M524_z;4Nq{^H6BpspXI+
z%AS)^@)8wY;bL`KUj%gunPChJGvf=Ms}3eEMAB23aCIm;TIHq{AzLEuP36^U0t>&T
z55pLs-%=sBKh{#>8au6(;GCs56h`4}#F{Wb@5BFX?--GbdqOBw*}!0Ggpf0nNW
ze*r~@OrR~Bu{!ib-Q855C&P6dEKzHajzTh#We3vhrw%GE%bMo9A<;GNtS1kx*Whe&
z5%OOWtws1W9S|JKaMCw(SY#T3b>rcN_^j#_I-2ow|%nIYKGAZ{s&C;1NK;s
zCNz#3?JAX392Bn@XhEZj7MHXTi~j&mZl5nEjq0@FQ@&JFiC2FgD*03?VMAXfxJc%<
znFxZ39)&#oiYlo#QWu8C>Uk9<6U_X^zUhrHdCF^;H#Ud5bHl?xEYbAxmLLZ7&)Q-n|O+<-;2a{dyKoH
z_;A3zVcy|r#2RrwGR;W}((5t}K^+Px7L5#Ut@v$CV(W$2i(gC4OMDyf}
zi-{?&HytkX3vo{sxZ&UxEHpz88*@dLrdatpN!qnLbB{GeJk*i$-CNND
z#^@{5q3w|r3JU|2zYA*CNo{OUW*hV{e`~{Lrw34yv2gIiT*puq@p1QaYgHIs?xPK@
zYN_*EtWh@?3ZYL3Lq8h?ZxV9~gOW1@G+
z)kA1A=-#D96W9Fk16T@#*?Jh2C$7I4RpTP|za{f*P?rhhjCm
zTXfT5j675Hfi7Ghf+2#KY9~>LeHgvT@eeVw#2vrg*Gzk!7S4p|%q|
zR8^_@s^2Z!N`d_Z-Lc>gntlFSiX3ES
z_PVXScV0dl$t3P)Oyd{PG+LUU<4>j(Tirn*FWg0^>``Y#UR$Ush(i(7bopI!t83dT
zI4#34-b+9fg#0J^ev-TlPmIpI*2Y2ZiocQYwGC9F)r|}Tt=x{9QYf^SpSM0T2gykC
zcPR?Rb5~bgLev7H1>dG?8*NC8H?V=osKjUi10k*IQ@3TQ$4|}35QuGxt{YTFT2AJo
zP0G=R?Zsjoa}OSo)ocmU?dOs%b=H#F_EfSGZRc)c+J41!%5tiuJeG#k<);jt(6zf(
zuc2DMF?C-q4pz@_m9z!FZ?d70-B1bariEeYnHS2{RWtK-TgjDI6+-2;pL3XSxIJL#(Dk+mzivRr#_0*%QM+iXSQe!~iD{0RRF50s;a8
z0s{d7000000RRypF+oufVR3-REJNZS(W@wu4zh_NlB@^}EU
z-^6QhcvM~&nZGNu#q|e@@rm~deWn95W+RaXRfHzQS7u^kLGcwrgp#|P?~-x0OKdac
zi{PVg(pua;lOw=JRhzMK&Y)5`cs`Ln;sg>N7DP;VQuQ-nnLw`qhcP@VSCBi1uCeYm
zp>cAf$&AD!T_LI6DAZ30;0Z_xkA!hL+_tGpX{10d)o!3E+sr
zl}64-g<(t)OdE=j;0&^?47HeXJO&}{F{TBOY?RTm2z^JmwxQ}BTc+VtpdeYWSO>%c
zQzIxXraB#5s@bLtK~dW&Qp2);j5A)I$ZnJUq6=feFQ3#lf67_xZhx@C)nv6ayY)K_
z`=0o{BwbDHmdBmLcs;?J7%@r7{wz`LaTgvhci{n&xpPD(SK@IHz043JZOYC?!>~+NK|IBn%MSx-
zcTvA^F*;H=AcH*4Ot4tnc;Gh3EQZ2?$`
z;ei|6#0Y_m5wHZy5N53s*)D8=!GxETA2tLz_?>LwmA46l1>@CJ@f&K@F`f)S;t7%x
z7dx2PsYx;e4&&8{tS9-I82!1cg;-Z462v{sC7)9Z?h?!u;G8)e!#XQ7BLH<>{E^My
z!iX#Ar8WoPf$~m0jb;|h)EyRNKY<#AF}#lbc53|6uZPXiwg
zt_vO`bxRC#>10QYkP#IU=Um1mzEo|JvL$mEN0L$&u6!a{Rx!fb@gAd>Wp3q4+2Sf(
zoV$dKgK7cP#vcfLXA&H3$~2er9xiI(i62p!J6E}4N3vDSJIE&w+)kV1lH1(AbQ49GO~itzZ~?@xa;7?#>R94d%w)D%W0G(phF4eo
zlN_=Zi-Vv6`TjiDEkOjw4{Z1%W)$u`gtbvsIe-R8jyx2DE&~c&#E0%%xsbTiP;M(>
zDCLgXSyZ;5DkgY9b~3{nj)fKs$G4VJ6q&295PnkAC{7i9$L^NoF&}_&K5w9Muz&`V
zTBK=nu{gsVv*^ox!~?g9Gm_+rxumCJEri6jHGY`9CI0|1U*16B+k2S)7@}L=p*$Ys
zqkdXl6iu}rH3h_dlJSTwtCW0EI0S7Q?p3Oq;E&S1PB66)c8Ovr;h9EfQ3J**$@_pR
zy^&Yrgxnz%5A0$XzNRFd66pnbWA0X
zRa8agOKheF9DRI0!SK~b+fb|H42ql)2;#5e}y)3Z85YYRjhWpq{$cC`Rd
z1JoGi5IEv{DkyTqIhk?gW}n`+#hS=5yEKC0PqOg~0le%&`4gWEsa!NiTBs7cqhj
zLb)Bw&NNE;iHagzW0(LWvSh&ta~r%ze%gqT<#xx^Uf~e?GZo)BTgy__j{(aJJ)#}V
zxrlQFH91*z7tKf{LLN5Cm6-S;&y}eOfeyB1{M8DZf)Bf#>FMq(Wi%O#DtdDp`Uowv
z8y@9BOb5(3o+Pj4YeX+7x0tNlLIs6V(Rd)nZU7AorjrMYiF@f~&4+9e;t_QqpU5s?
zgIE&$QSmm?nN>%*$LrRh_ZK3me<`Bz@TybPw6tl&JH&2?Tc^4)jY7QsqlP>fNLtlC
z*`qP+fmQxtItDfTtf|Q72qsbbL)@t0$0QKP_C*lqOG!Q1Nr&%W?!8QJQa6o`mI@I9UhLCR5
zW~=HHE;)BqG2LPId=ma8#wNXEAzVicv2$xT0l1EL`I)YY`;ARO^q*4N49!M;5l*7L
zsTT`j%1hj#X0E59TV^15>IaGHk9ov9r)puqi_9iD**3>CJz0k=4}KRa#sk>Y$}Y)#
ztwY@r$|dA>Ic{cbOX@%(f)$y{i1S6bAyD*6IE;w8ohZ7Q?S*_m^$Cl2dDI`6GXw{z
zTb*WpDc*^`JQ3WvxPm9Q$?Y*epG0O_Jt;+U2ztnzalHSc4ljzi9?vi*bi|rLdIZ_&f_hgHqK_0MRowhBOROYATk+|3+R>D
z2+rBy5pheEHo;;T2!!Mw)Jg$EaK_)XQ|OAEWAC`{WCiLjKL|2CqLJu`>%yKQ)D$7S
zJ}1g|Q<~H9G41$?IP2U)Z^CpV>Nt31oS_J
z7)H2d{6R2>2~#+Oq@&dJBaDSGeTsEItdC|uz?kJ3ZohKN<_$x0G`u$mv59t+iOfr^
z%Ml1nYEj(8B5^QcqZ^wN`3S^xp=1U;1;^9(1G%t1G{05DY6F%spm
z64Y}M2lzfB)(OVHWE!Z$DU~~{$NvBVIjFh9ZR_&|Qw|(62rnH;O~G?RL@*O>p;{ND
zpS2Ca_v(c8F=n8c)JFC00==tjOFeBUGb(Ip{-9QFwW^m9$HqZfmdh#Hre8(q%$OTR
z#4z+qmIsrWMZ8&K3@W&nuAFR!e+Q-z@`Vo>_-FgJzj)a2{tZm~bb$cBi)x%|M>a8Ub{Tp(L)!NyrlR90NUcNM;&6m=@@=MF2(qF;R04TmmW-Wbv@JFE7=wmK{2k__JhrfT-PxiQJ3lp
zVDhK-DMhBD}LMaRBL*al4FS6+)&&*A=F2j&|trmFl!e2
zN|ns>Q3Cu+K=&9rnJ9q2sFW*p07PyOq?d_kH;Qo_2(d3LQOWRNAF~&I6CD_b?o!&F
zVQ`fr!@o>3T98(zS>cnL}|VGGj4}K?03@
zBSI?@eR#M-?GxM*T_foVxPUsGgBxmxF3s-u_@{vl`Q^Y2}R*D#up0qD?s-JsOK@bq3H}(
z)^CPn4Lb2Q%$}i&_=bSIgy`Jam@!mFvLWrcUKR^PX~fQY%wl=|;mRJNtQ5D@Zk+py
zGQlt;vZQ#)W;2X3qPcXu%AE-8`j#FO!7=^B?8Bmf#cxoI8%?sKn2GbQSl4hI457uT
z-xutLLgknYhAcTU+g-;JUcyDit%D$~%zBFT7cV`$6_w#1anCHRyNAwWp)xZWwNtKL
z?qC+#QbipzJ}mo8G|Tds=4iOXemUq*mL1ZJ1+yH-CC
zwcx80rd+Xjdba~1rkx;;uviTEV&!6?zbAPVRL;Q4Ai
zOhJRx;?GeBUS8qZZE+cgdxL6r*Aq+e!`!e`OGMA&4hJyc!aW$=9~mQsQ>aH!WVa~-
zjy`hcv}BZN$8dyL+;m*!Q|b??mk@xrexRBB2soi!kPhW}a!M+rjlv?>4iRrrWqWrl
z-n?$yv8Xqw8qCQGXJINL*p0HOpwx6rad3A|+CTXc+K3S{fyMZ^WetKH!KjZnI%SQM
zHg+rnxXT9Rk@XuWvFYY^{mPXc2L#f@VJttyx-;4ROG0Mq
z)&Z*9)zqiKea6c2u*l`yL)9;%S)^K=iJDdmt2>kfP(nDRp{0&Fvi=Sxa<)pglCugM
zNXb|wmuoX&24z7KG9i^^xN5%cK4x18jJ9@6Q7+=^a>p``ATqra@CLL+pB+8RS)7VT
zT|zL907AyXP~3;=bP;z*4r9X)R&XV#P~4%6d2H*$?m<}m<~^a+OE7-YqOvzKSu{mg
z4y9bD)J+i%BlHM|5oxS6%HA;^u-wCO7eNK0D%4ZRmhj4Pqav-Q(R4V;fA1n_|~jR0-v&P>I*Q+`I<~JY8jgOmzF5zQrJUB
z5w}$lnilS}EWu7CJAV=PW8UCh%3wgI7_i1+uFPeu$Mp?v5m>XBs2b`iZdxfw%FIU_
zf-($jLU0Z|G7GmRUL=$cC{>fS9H-8GCH+N6m=plYdWBFpLn-o%{N<;rtF*?gEXW*l
zHkzAeJ$28yx%M+-#Lv2Hg>vhN>h4_5XA=%%IYCeTftSJaUiBNr}6{
zh(haHXVsfZ@dc8jbHw++0IN+9%ml)|iNrr}jF2@Hz`L<^%i?h6Q}rri#oa>enOf~d^f)Nx?VZ_p*Z&Ir&n@ZD}6
zD&alI%U3R<+m<>Y8fj1ZPJ*sHp&1TfoFR{ydAJ{OsE7K2HG|qgJ7UnEkj~{95YpFt>|MS7O#nD9h-BcB8zbBXaHY6vb;Ja9BY6^Jye!yMDVIEc)`Lzf$XuREng7Slui
zh|-MA!u&AeP%J2}+UGMRK%^L!11+*frbYP`14gpC6p!y}Xy&+A3
z?r7y(SyVB6q9t0xr>VZe@fQZ5uRgCxqywa=@+HTDxU-ls$avp4gQmUAn2!)*@K~43
zEX3Tw97S7LPBOqg2+)fWPx6|#FLx1Z-x9o9#mh7-f$tz&oEVfqLU9uWc);Zog$wtK
zM!_>QN8>2!VlgZ(w*goC`x}UQq+6
zMyn7;2}TkSN^q`W425{|Ofrp_Ty{3HfG}Wyv#C+h%wUx0fZ|vOi;UVydm{Ow5s7O)
zQtnhtx4BqDaSmXn8K2@W+*Gu`A2N^^Jv>WkX9-4L5v<>c^RrMxLZw1wh#JJTOBy4}_?9F7JB@>qQKO(GH`^EDVm)N2uct^{T@P8b
zzD|``FR+<+?iHn7n}N$nLt)$flJ1?{`b>(*XmWQLmTFX=3F(4>#`7o_6A%aLUd8w(
zl1roN;B{K?3Ni?-HF2h~8N3WJGH`pE8IAz8W#%4PdqS|W3>=?ud?mIOHsG?N
z!hfCxcX{O)$!pAJ)8#B`Qc?_AERGRi#ksDf#u$khKGQC6z~2xZ&wQ-6Nr!2%0v0%w
zZBaZW&Lc#KrJ(N`I)RB$Q<-OM&vK(|8xv0pV0!VF>%?}R9#k4VcL8jG&MGl#{-2jH
z4Q{~x<}y~Tb4?Cls0YhY%u(uKsy2)ea(DEFKsXDvKr1QdB*WJeLX^p$Ax;gy$%Z*@7^r%R0dpScC*x3SwsF
zQ2vvYX1FmaiHpodYg&P@F$Lw+yi1}OMNoRoS(K$RWMO}a5wDZXPM6D=rpPmCU)vX+
zI=DsEcHb~eqoO0L&e@1_a1rQMim`plL65lf@Lk}z5IqUVkH-`9;^uY5
zsJdJWfH`1_E$Zf(R7#zWR#;cT2gw*03I1bz#9Fz1pu*DDS4V`@hM>g0;3KdTx#SEF
zRab!11|!6bA}dnil|T%v^?V65O+zjso7CM2kGLc$tv-lJzm@rzr*}saC_e*n3k%Be
zDv(YCi2bC_Z2p8LV6AQfW~s{Cm$n?EQ>+Xr_dNrrGcN(FxsF`EwJ`P_FkUmAFpSYu
z>5lJr#-+h+nx68)Mp2X|ywFT1F^0MBD&pZGp~(k=j$yuhMY>PHmC5rGalh^bc(CI%
zTG0v{@?RNDLzK+eGS%u@D9jmfVA%|?%dlxKZeA_laDZ{rf7cAurzmoiV^bI^T4YON
z3ztKb!U_%CGn3|9S(w%+8dy^L920^Wqdi0=$#j+f0F+@Zyxhhs?OX0-hrtaJ$6jK|
zxSJRsnL$a-aRN9;yYmC;>2j*GFqC@>mj#uHhj1o_k#KaD8t|jcc^(JZE*+8jN*qsAzM=Cg-LUnh@$jG@
zFHoi_-MD*|f!ik&Fb*DI+#fP6ZW<-dWGm+GkAM=@Fx<(?QQ7drwXi53aJ6}fvi{?~
z!$(Zzt)#NUvC9X@i-xBM<^!5=kjYV-#Y@Meni6dl97iyAs&cO{5m+dq?U{0fO1;57
zx#}6rbh6bf;x1vhwLXwRhYnBsT)NM5`Vf#R)sruhJUNa9Ip$YwClb!QdTtc?my0HA
z0GBO_E?{ye#9T(zeW-?M;eS&QP46Jwtia9vBYb9Y7jWBnP-%R
z3D;OnCB@8WFQk^Jmkq+f*_?1sVMm~~o9mb~#i@{&%aF48MI0tR;4V73BGey>j94)Z
zK(dI}+_J5luv@~<+|XU|!2<~m42)ezr&IDFIhXAhN;BYyxQ9QYd<(=MNUB+1I3)n(
z%&TE?_CDt__-e8X9KNEj097rP#XK@9GsX+}O@cZ(5T;SGuX3y*W79U2O`}h9F;1A3
zu<%>5mzYmd;0(cyML?QDvrlq}(C37Yz%>a93sA|aLj$QzW)=9NS=YbN>UXhYrk4gR
z$UahoBmOQ}r*cYf@}-~{V*EDX;b$`~$JQo37Pg6LKxNNN17kU$yxVH=-MO(63
zaSWRO0BN_}si@sA*kik!9g@N>{X``w5!5NIQ4A)0k-h`_ZU#7vvmF-8QSKUNYXf(sKJVlpW<7Y?BZW*H!5Xd(j>Axr-vJVxVw)&zoVH-hswGRs&8khL1PTF
zV9j%qR{)tQA>i<}%iOU&rSSxD1EyeiFck;GNphcOs1b#81@xjN;JF~%@E9sq<
z;EGsO_ly4CH-qvg_)-4=FoFjo;j{h3ikKaAkmv&ODoREnq||8z`zG}Pz;=kc9Haha
z>>SwN$tj711WubH#Oa2dHA^iG`it=TiOOV=&p4@qx*Y!il(A6XxM+WXjWtH_%QnN}
zJb%I{e9!nqVji5bsNcv##|-LTIZZC24-ZokydR|!H*rNc^i9C%lCvnW*^iAnc@nFV
zyNP%60XG|ofq?_Lj}3r~?f~qb&)go_N}bE_)GIK`lLp>|e!4yF_Y=6)!aXxUB>nC=
zlq^;c2&XHq#yw-9fQ?--ZdoAXkVx(0XQFb)7VK85iv7i^GD=$=3GZTR!7)NXKfI5v
z_)%<8e3SgGc1m9!k5d_31{~Ew3t7X|py6cP-4Gp1Yz!Qw@yUBs
z%PN)YQx0Px#sC8Z+8IkFFisiBpDE002{>SNIwFEBvhgm>ORgm|9WK!qU}3=*m0v^_
z$6iKolURqRsgy*`V+nhzl{rgQM=MX%r5}?_64VO&126^9Kw&3~KKbs0LfB-{fW*}F
zfLAi$^&P*HE^<1<5Ny7Zm-CQxd#W
z5oV=Hc-!JV6&@(BaLznmf#Jo7#WD;_9We>e%D|X`i_A$c@FYeI2jU4b{xns2NHsf^
zxU)Qh`iTIq7|{pxFQL?TZ<07l$$i7`8uB~1w-?3cXV}1fOIYDVCApX$S?(nciCMRJ
zDJAqBvc2rqaSt5!VMVzM>J@OWvky<9+!dn|+ddp_Vr?vowl@%%EW#cNRl)9BzNgzB
zX|E1`Vf91mYO>%2*9T_s7QN52BOV{t5tZf4#Udt&sTqTv^Z?*mg0OtK|q%)!p-
zm10!K6TVsPTpppE^AYA=Sc7F
zizWm0F{|HD6KF7HC9QatZcvVDQ1?jh<_n=^(d9EP#AH1iWx=8OVWH9(cs&kctu+I|
zKx*+1fGRou(v+{nyPEHUZ=xALqAwcrhN@H*MOwE48!Hn(j$uA$o-gVk9ZO1c9KO%Y
zCBVxMECd0;*{G)(@Je(~+}v3=cbrVutQl+_Onj=x7jR1oKA}T=yOlW}<^Vr23t@ea
z+bqt;0loJcuMF()XOsjN3`T8bD7wtEa;}iNKa4E#`Y=(YL7f
zFgPD@Z@@DMewV{IkH87~l{7klFpKg-HjlIY#b(^GPzIclp@RGa=Mm#F`$|B4#8mZe
zB_m-mQ0=U7rUMMk4Xj0oL0~j9+!e&K2bySXA{)7Rjl#o112)Hlvzhi5Qw2!kqWTae
zKdSbM%m)F)H6jHkF)O{sD{E*dx}&58gb|KxXN$xQ>X8(LE;d^nf@MF>MfS@ij#**Q
zXXt`gWl?FqA*i#|OC_@8lY%%%*m?Pw`yhV;_;hR{DSx6-;_l`Mo%m(KW-zS`0LW#x
z79nniB3oN7+{Z6b@eMNx2p;A}Aq_1t35iG<0>tfejXlelmi64JM>jKDhD1kYxtVL7
z>QS#!)Ntx|#M)%8B5A~>)}G~NkAwCW4%evugI-Nk=ly#hj(UF^IKmqH&*Rm;k%Efsf3uM9K{zsvQvs3s~uY
z<5BSN2N9`wLgB{Ws`Kn*K(A=T)h~D
zjN-&p1{r3ni4N`v2>@7y!bya6DDGk%ZI=b{0}G0!qUkBVU=^&hnUuYHmnn-^Ip#gz
z+z{K+Yy$ntakX)CdRH#eSUEV9?zF{aK}hO4hTp(Po&|cGr#_)*XlgLpxawY?qH6yD
zVB9ne_Y)8U^Xd(5E7oG>9009AQoqgAu=s+$NKdjB^%QdA72*I?P%uYNa{KiLcCSMq_OeHASCZwxo
zh+Y*Q;DMuiW^A+=v_3{5!PR<#!W<2-bB2kr>r(xt+{+@i9L^?!z#Ab+PpF!l_Qh>$
z6iO=kg8)P4{%bRi@M1Tqd1z$-1wELTP#q*|e}oF%mYlnmt~rcBf^Hb=JduH4Q!UB3
zjAHXh6a-IDyP~CZnG?YN;yXer@hemP%;X;KSl$^A`Y_%PxunXgvW$1uB4&J>=4PcZ2R6wi03g!l%=$y)*hEO7tG3#u%vp&dJ
z7q6E#5RH6t`(fnWu}j9MxrvPzYWar1aDI^DJd6fc3pjHCmFm9e!R|4LIuk)(aaFpOl<|KP9@2rP+lVf?l=Ueeh>KP|iAG$o
z>ZSw2#jRl|d&fj@0RmJAEsi~^dd3w`GuD3PP-$;vgUAB{RdhpVxQV#kL!oL}RFpIM
zC0{G@!|Gx$QD(xNLa3MH{1~{GnQ^&ssDBX6<_jE4QE)&@d#G-5GC^32aUPT1BMrEY
zElf&w4%w+zAxIo?FL&Bp{lhH6TAA);>1cPDHHoy;&?jihVVNO!D-AGLW&V;=xD`r8
z_RImk;^Ol$Q>4T;>K+49fheGL++qvqG>zmT+L)MpPBOx#7|pEN1}cNMiBo^N+y&Rk
z61n#U68A(Fn2_T@BDgt6$&TmDK{)0{aeM>x80NY45x>M5%m@QZwB=+pzi`>P0$AAN
zF)cIO3e@3ZWhk!tmsz7_%k?a6lwVSS93&wI^FroWDm_7HdM~)Gt2z!;{*W~`$Ha7F
z#My$`sb4H+aDL3%-9e~Dev7Kz!4)`$JRspMyCZE91}SP&a1Nb1h?@_Hr)&9}i(N%W
z^6_^R2(VS{9h3Zrzc&^xC%1!<7u?=rB9Dne(5@I4f;51!g1uC=Mg3@n3Lg>5Zlf%_
z=tk}O<{npa$gfdMTPsjaxs`zyUu3TuozyPO3pX$sxx0Z>AY2P;5U@h4A)xb96O8`=
z1m1^Rgw#pZ%YLH8-h3VP1bL1Tjm`=QrOqX0neqlUqQz_)PNtAT)!qmguV!M)bv+SU
z!1hnKX;UJPgxhdCDq_mv{vj3L>TQAb#KWQYDt@{UU#Al(9uX|8;O042z0l$VY!aJEJqa>Q^OunTGG@pOKk~i^d3<}>I;H6#cK5{88HE*QnQdA#%2kTcIUeR
z-0fmu*1Khj0TEf?$GDVzlJN+ehU+y<2^o1*yDG18vVdUDBD8~nN!T*pPhvTO{LDJJ
z)Tqk2N3k{=+PRPqb{R}zJD2)QBhDX7+_+aysdC<9b@djhHcO5N(YAdUc9WEPIZ)3(
zY<$MT4c*foOY6Wv11pN$!3a9-2-wSTCKue{NV~{+B48O&X5GQ&5(Sp7oe!9>S?X-#
z_82<%dzn@)H7;@PSzL+0Co{Qqsid$@>kvk!l+!3dketh$miI8Dvbu2^m{m)_Z7T;I
z@IY7r!+lF;OHoccmTHtn2f#Hb>KOt%FS~T+CDArgvnissGVXKp7gwmYn5i2$oN7Op
zkJ3G*lHOa3;-&}CQ7^c$Vu@1Xqzy&<&u@y@x3?;Xa`^`y?lzcK0wzrm<$W;+6~5fd
z{`k$TxzayYW4mS&9=Y5D8}3kRS5o=Ni*x73s<#@*Q<+|6?8iY5n&gY7(O)A|WrU`a
zSX^aIA~0hC(h0U9!gi*|=v=WE(tcxF)q;^mVXRV<Op5@HN2W+HiGaJob<6bgvxq8{CJW1~$Ey^J{1KYT>mapZ7
zD0OC{ZX2Ua+>pN0E_2+z?vj9kdO#Bbahs2nh)UUs)aXfeT(hST^P>uN%?Bu#HBo&-
z%A(#PQB%M8+Yj8JU=!Lm`j?!`sI;=DP!{BRyacl13MNBx
z=GYJ$M|B*GxuRnkivx3^HSysq2Z>?8<{_pGdshz;WpUh9*j?qbItv
zN2z)+TAQjcc(CG0+J_qJnzjQjv;w<|YQDzb;=*rI``|8~M`v`Z
zPZNar<$c_-fQXS_fBygwh7W=^Pe|ew7MJAQu?$jUQsVv2h=bevOc@n_Qa800M5Q$i
zoco`@Hw6Qj)SEyG+RH9PJxWw5+uSjikBIJ9#}0;!T-^3(g@2iin8n=EEMk>rCHq8w
z089oG%n((3BJjq0FA)F^37ard4;zc^4BI-DZA2_8Ae($eMax-MrW;nosODpV>zK52
z%eD$oL}qWq#<*Rycwr^XJYY-MdX$493SUGX(6w*C3prr|(%;
z4_3bB4QI6F8Llj=m4v`T@MEcPSHeejCZ?QXVlG}M&qTA_YeW%EXC!l+(Q$>Yk8D;v
z8PhT{gJ$k0W%UCQa1oV_>L^mgu`X0Ep)lU@_yQurrMSimnClBNhISR=BLQ|gCg{zy
z%RR3{H4y;Yp2%XicTY193$8`}qbfP-JN8yz@d=T(*h7v@6>P~HuTrwX@Ic^_dmj+#
z(TWgIOP!W4S(y}5!Il_eHYzww5$6y|SN9+=;=tdygw`bhkBC$@*9@*9yYOfcYZE-K
zSLw%&c0iK90gV3u5c^4-W&+AOz9?cGLG2HO#@DaDSl3qwEvO?;mkkLzoD)0%wQK(X
zaFC&QPGY?UQSTaN(97a74RFo0bBc+JR|>xuj%Gw+?mx__w5hM364^#!QZTrMdXKa+
z5HgB#^D>P%i7reJDCIfBH#9~mWt`uVUhGl$OVryw$bG=vrq*U!eM+o)m$Yxxc?=^A
zsdA%#Go-&!&^^ZR)Y_1|wj~eqFa{nMM5aIv7y!oAZSE5IyGlF<2O5ektb-Q?>Fm&D
z(5CJbD{D90zaj$xw=+5;TSv2qq-THn!(a8p+cz7qvNrF*TV{>@;@lc;%7+L+i^CBz
zS-P2It87(X!I{7LQ+cHLKy?vlA>58bh^V0FY-8XC2nr-6{$k=N2^6^vz(Gf*U&Y;A
zqb}~oRx2ginXuA^qN*jn&kzD)q8$;`9m0ggM9mA4UXH?=;fDZjL58Kaa|2zLBOz@T
zT8l54X29)V_c!vZKawSFF9zXpfDTTAR^ttcD7~HXN}GpfZftx@l_ecr46!b;fho_g
zH!fdt%EY;FnTXGag3|Njkg1p~D;yzoe>bg3}Qkaa)8k&M;zoHy1S)XrB-#k&BgtE-wRe
zTZ4(#syEa@^RSF!7@nYjXNrr%q?w$ClTiV#PKe2++Rju?8H2PgCMJ19Fb7HZ7$cRJ
zGM?a`ADAAGf_n(Dl$I`m45y@kF3$v{)b5!AUj|y2F}Q@-imiMYEE{`{(1NMVTum0A
zstw^vh=7$N=ciB^@f#@_vRA=xq_dbE2zz>%HiLje%apt~3a@gR*)5r4yh1nu#Jm#k
z1X%hFL9W*?Ll0st_s=r4DSjpaE8<&BMwr*hoA{I^4WVqtqF_$uR6}?kV(+5=0J38#
zZ%^V0`Q%_pkE)6=XR>D>=@oj4xrRxoa*x_+b%N<
zJ&;F3x!0=_<+zT2fRtRyxR>Ax>K>OLGZALy;a{lEU#N!;^nqHq_=7lSIP*#otdIIb
zAh$5e;W6u3nOTk^*S27b_?LT_Re5y>hw2N(CDVF@EID&^3f)rT-?F>*+E}MwK8Hjgh_D16SnuuY$kCF!{7U=OV=#`?#t8&j0s8BI`
zl8K2s!I{awacwREv>kHFm6g+}swTOIX#Aq5;o}8q{SgJL<&
zL1Ns@BZNKTWGyk4c3v-rR!hKj6O*Ye)XgmFT@DA^#OB;g`;^^qshdw|-?&8<(|RRp
zADM?T+@@t{0XNwIR9CF0S-IRLcOKEZf`aY0+_22#D*etm0OU35GpD8lFMqV07;LH9
zaWe{yki+B=-vFE9lFoq^?{E!WPUtWhmRglIF9ZQ^WZa0YWyq;wrsvPN5sT%OtYRS+
zbC_rpc))JO^*_Q9d)i98GGe_?eF3vZW>#p#8461GE-0G(4d{bhmjp1V0uB|IxCz|g
zuf@fbqrzG4;teo_r=^rbZNW4-7(AtAjFP4pJmdE*!#kr)F)_;%L!T2k-wxi
z>c5DQfKTpJetZ`cex1gm{WEG^7$dyt|v1?^gMc_(ZprNTi
zdr(tRjUsb!dxcAfWNe?FRE9x0@ee|c;Eqyo4rRUKSxhcl7aRi4q8DY#iX}TmU@im4
zFqk9vrBHj80>&fr0l8(x=-kA3HW0dC(S;pPfda)B)L%G4rmk3V9_DGs#8ExA>NLqO4?ceMCwyMS(0>?7BahV+i&k$1e(H^DXhI(-^@PVXx{J
zZkTj35yNh68mGZB-ZV+(Ls?cc9wCxO0SE+PNpiO2o
zh`BJlJN~AcAY8z?mM?Pf)LQ8PDSbkJ%O~cEjAnp!sye3lU2q$I4N}QhvF-&38v=QwN
z*n6-ovBd5*a7d@`Vz10mcCjhcHpx+pc<~D8VFW47uBA)|YI72-*2^gR!JANQcQ3@y
zVxpWXTd3<%QzH6~0ymwt+@R$bTVtt9Z*118kG63C0F=8FlJzKsGqP1xq)AqEWyDPw
zjFcs2^zJL82W)EHurvb&(Hjo^L`X8%_|pM%ar%PQOeV1RiRvy{`bvgb)G;6o1U2g_
z52JAGF~=mY0SBbsBZLD<6fvW6d_{Bx>fec{0?B0I%--HILkMv7FmtJ$p`%b*BqN47
zxm1pPa!rzuV*dd0QLG3acj1V49YrsojEfssg>o`$!qF
z_OKeQ;OZLrmAZjIaWd(0fse!ijLIrGurW)^bE;uhthDUtKbe%LRD;+zzl^q?id+0yeai80x^cZ
zVq)8ME>O_TMVVY6ljuV-yf9Nzo)j8|?Kwqw5XG$9%#+8kzbYzlH!Do0XgxB+1VL)M
zj9D-k8;cdFgHoEP{{U#q%tGjKD8CRHVzLUCeK>UZM|aRz>JDHz5ey>voEH;%Y$a3jT44%&FlDmP$5Z(5EL`YM$QhKeD)%wdvpDG#$*9n|%q%9Q
zEs-Fpa%MS>x4sO%QKK&EB`f=&fGG7zVKV_v>C~dXw6BBQw_;9YG>i#B64`>~qQ8Iw^_;JDo93~MUAvg;7s%X0qn7>sS>0$gLyl0e2Pl3IKL%t2{$
za={b}&4nKCFp#8Ir#0?a!MOqtAZ5w0Yi#t6BPebZcU<`G;&;)^ru~D_K3mQgCEQiS
z9;QgFtxJp4O6zRb%+mpcWK6%usu)zcfx^T4N^(n4gYGRHFtwjhnj8MJt*vTOwK^pa
z<5^Y>)$=pJhP=(AE~9A!h8Xa{nnoOilVyCRMj+tdNCHgTEJ0pqTHtjs;}Y2>HjY$l
zZM7X3#X(5~+Z}TpTR~>K*=L>Mxm|N$x-(S1u@*J6JfvDtD3yd`oGR$)1l<
zAdSl-gs#xhh&jxyx$h$`qZyx=7i6lZF;GC;G|LQAWJtO)ZZ39AyM*UM#Y%lh`f4fvRIy&%yYPks(3lea_Vc2;>woHCR$nUEt71*dqU-+7Y{_(39_eJ+{4&(8CWe!
zZFbB<12Ti?m@Y~^dx&dkwTskqFj(&yOO&dMG21G)8s#>|Z`X#s-7@z836LPGRKb!z
zXVmP2E<^}WdJm}h`(x9#XRBu!h}evNAe_a4A6K9zvxI7~KVf`EusgmZ%S$HWL(&@~
zy@8E$Hm~qv5L(4-V{(PrMX3DCSXMD|)E6W2i4d+nB`Q}*X_a3Ss9yLy0r8=2!JTDw
z=jL6{aMBRq{vzU!qG5~(<4PgCKnZ&hNT#b1ZG@$S6T6z`Flrp&L+pUX#@k@uaO;@N
zMTLFMBx@Bi=#8PmpP1$s^(>!DH<|GR%y!2un%h?}m3+r77(K<@3`$B(6>~(6*sWf~
zGN1YG7y1HREz~Um>LMYkuH;tJoA64dgHoIc`-)Wf!X8&PtXXvE6|Fw8Zysxs6O
z7u>P&=GlW0rc@!K8DI!vEV4L+>PsYs4x?W#TnIq+OSx^_+}NFD<}xgV%uDG_f3?@cJtQzWj_HRw^$82W
z36d1l;--iXxaMd4CYg~FGNnp}lsS}2RB6F8a+zz2r_7Jcy2BipgahWV%!rS}96#a?
z2w#KZRlv1`p{fHHpLPtX>`zEyT;LKQUW`oEY{YEYb8B+JljY)HNm*4#Q7Fu*P$o@4
z!MG|oF|y!LdX%GmDv7EvZGTCU3pMb}jag(sF{7Ms!@)Bkr{whnJ8oJwSfSjusemZd
zsK#b&Z`5BNCPVC4Kz{jNW8yjvpe!Q%1(|6Qp=MgfX3?rTL!lLKc*)
z9LlOvZl+yZ)M0xNxF*7CSS)G{qwIw`KFlh#tnMKItU;?2xr3L)-xSb`qu-d7V41AT
z;W7Y+*01FrdPOnKQZV-n{7t%+CJD*F7&T~Yh+#>JXVex>5pAal^*$SAyf_FFjhzvR
z5i`+;k)*6+6hRmxZZP{z!YO@A)ZMlu7E>VvAm1^72Ug1$(WwMF*$w7e`OZTQWv!Wd
z@O@1xH|k*&ot!}-p02K3Q8$r&%<*gr1SMcrDJ2gqCGx1boJyLCHkwYFGCuPjP}%ne
zHgCA1fQ|M8Yt&S@PedB$GO}!j1Oi6COxrJnUrae=t=~y1<1+^b&_OLuNdEwcSil_3
zaWZC~A{nv@c3@uE12C$kDN_AHTwddf&L{qB9tEC}ReluXCQ_|pz>a^o@J>+k1emn4
z>J4H8x4ZuUrX_$lH8k(|238}ffLt}S64v%$noct+VU+=S;|uCt-LUlr^7xF9ND)ZX
z!a%Lc%yiNkiMrJ;^L>6CW9TN+&6>x+tha{kCeGG5m4;91-cR$0^5}kPt;6|1N
zZdrl&3`Dr(lJ6ohOlez)&f?$G5dqNQHmzUkUMurelpB
zHhv<H^0MVbxqgD=NPEj&I_0dH07h`j^9U
z#BdoP(W)tBZKzyNff-(9CZb`xK_<^qw~rO23D-AerM<_gmmI*U@`xjxR9N9D4kbo~
z!|8$(xMmI4=6+-U07Q61gkOSvJ`sg-^5SU8nX?6$i3(F)8s8-jl57SdAN`Wp)Z@5{q$$A`+!%?uhol<8a~Cr8E-2MpMM5lYMiD2}tS+uD
zBbA7wMC&f_4ITytH3@sR7P>PJ;nyG=E#gswGi||?V=`nP
zseAgOUbGCW<1d#u_)K?f%Ee=|vMctF#Kl@+gujeI#ldMe15>MxcM`Qx(|5QXpfzmz
zj@z76BGV`lg9UNn(iJXVWy{Pi7_ltwQ7hDOmR}cyvjV)G8ANbs$CPyE@j5kbB7p-o
z&)hTacyR&Bx4g#?F@XUP*8Px_DAXrF(~%ce#Il~C27+oRcMxyrL@o*@CXzm56-B|O
z$%G$pFF#bMz!Oqa9{#(9o|jh`IfA*k1{&uRP&1hB67F`G{0l5@a}FEJD2^T_DW^giPqI4{{N^cIx8H$_
zY1tX<@DFA-Z3-ZrTnWUoUBv-!a=b}s3b;)|rB8iA1#)Ie%35BbZ!gpya}Qk2YB@*o
zow=T~#6QJLm%^Ku5gy1dn=Ct(1*p-)QpkvUM+s;s73Q-1F$Mr2W>!3@Km_n&cz5H~
z+Hu4McM(LWr{Ler0~Y3CoS@~zG6A)1nGyRkq21)Z<~7h|gW?f_m<+Qj)$>y){V7E<
z@KYlXj`il=Pz^_FCX@3jdkMEju_=LjY6ukK=BDOK=H)rcp`0R>h=aEAek{(aU|PKd
z6+m9eV;bBM_rUiYh<9px&S^_I^pFhAw9RyzQ-X+*WoEU5UZ5~5T*++xh+V_UPf7gRw9wz;8r9_+t`
zkMIwIr1zf`N~?i5pBOHQPjIL3X(=qMjfH%o0Mi9>oCM-ltz{e2;BUao^FWJ;H%m3{
z4qKzEi>>+-dNnRAxp<41pxPQB`3|Ma#2*Hbv|L2^4VUAR8p4dS<&Bz(+ql7Kw;xkr
z+|6EB@6@HOWK6gS<~B8`3#r`{L~A}D8fC?o=aK3=kRE$XU%dzZqo8DU?phYj%bmyC
z3*5rbiphsEw|f(q@#oE8eL&m>3C#}3#5@5U%Pps$?oz$#Ix66UFe0uCXT!f2nBVFG
zVi%NKg@QIa@L%CqgTE5g3&PEfU3nfv!BjgignC|~Rl(xOID(iERJn6ltRZI=_+FrP
znE{PMtwG{lS$>$m5fRMXxo~Dgw_2N`=C)V!Q=TJ0)Y8252++SYxqcTNOTQN$E?UuDN|y5u%?kAijCoi_&(0O$7p%maN1K(h3yy+R+}!q*I404T
z2OrAHTeA%1oz8e`ClYgG1Lr~Na7nW>6YAjU^ofx9bsLok|DqkslMK;
znAR(Zn$G#yW?2h3nbElRCFCQu;-(G{oK8a3%cCY)FT**s(mM07n}bn{g3FgL$#({#
z7LoNWXQD3`n!=Av1*Y!A*MZ)HqIhCn-NV|8VcfVW0&mK=TU-21CvfUIinO7NkBNd2
z)Y}6SvKc>%;7bs9Y50_G2uG5BLn9_NDOH}X3|t-0bio;lbCs2PKIbY8#A*u_gd}|8
zw3h>xJ%SkXfrE%X0y321`GH&t(Z@2_T5<>CmzEn$8HG~@H{g8KGXfd#V!*-;M}7!-
zC3tl$$r5-HiFlcUG4WB&-w(0y~%j+Uk6JFNozMbi)KA2ZUj=W`cVvJLMDbAj-JCLKEyR~Tp@^)xX?8GD#;Ao#k0BYcUlH5F1*5!5OU
z_XRypn!aK#xf@xp;d1;ED1szONl=OjUJu|g)15(|a5QRSGdf%%;>+>O->L>I7+MKc
zvQo7e(v$&un>`Lhb4h(BI=X5?HLHZg5&*s{>LY@pH3RL$$}~PN#mkp2m_YiKe509V
zLn-aze3Xsl%EV&~7DQbA0&}Jlu@nrwlO@6;G?<6Ur9dU+)J2<SPPZ&*L6sMVQo6Ztmo8kna^=9Zv)-9@mb+>Q$S{h|VLyX&`)io8;>+?=
z5b!$jaElN`a~)rWl`3sG@;op25FrLT@c#hxsTSYi#N7T9tInr!EBKg@>LWI<)F{R7
z!d;C@yqK7l_FTDt!yYa#fG5Qf6?XX>d{CRBrC`C>}0cxn#$il`F+TDjs7B>6H7DS&bo<
z2Zxn>6oH1@b8g~kE?n#2Ma!2iT)A@P%Y#tAk@6V7;d2iGH5|oW3ohl$m*VBimlR-^
z0#r$HMGbhkJ_2dr!oLi;a)RS!m;3?nM_uXjwLo3p0R>dVt0HNcb96lfZA@I(B_?z%GIfsF-
z40Y!`Kg9XJ!)Jl~o}5OpT-k>xV6r(Pnh&Uz7$g8g#Y#K~gR)_adGj9!{{RpF+5ij#
z0RRF30{{R35T}XQMhs)l>kx3w_L)?L;YOO+cJ2p~Xy+JW#`XJ%ZiAy)AMD)!077?F%{c7S
z3xS)AzOMYT23rbYeti=Wn~AI;QEbD|@fe-510`DE9?in9VVIJ#szFSrp(q
zmuU=_I01{mI8jp3y*>IEFH#xg!gS0WV0WGr!A_nK!w+rnt7?AM?c(sjv6*&U0(cvx
zYU{G;ec&z42dw)wlJ+9nElT48v5>q~8pT2G2d{q&W#`mzLtG7V7;H8Rs}>^kV6a^4
z*d{QqmW(l&(CAEd=)TC>2zVCP0!;4e{d`dQuehEWr`oCA`MuQd+kzJU%9DRhw=nVY
zE%D(^U%~cpwyWwd7D>8@=fa?|
zU4pPZ-ZvfU!r~b1T2;0^#M~S)PYEMFf+s{9iww0o3SC~Z{Ixz6Y6YW9!=M>SHyL&0
zc!*QG8kJ&&Kw&x|x>o+`jP=Y4AEX=9vVT;|kQ$gmxAakZ(Ho!qFRz?^dhlAkgKqW6
zBX-D`EAXYq!-|mV%AoY%co2+ewlQBf!p6U&)Tk#_j~nDeM}4h|c*|2R++?g=No}G$
z7863C{oGOH<2Y#!+m2`Tz3|n_T@~{s*c7g#Kkv75#4Ri=mToTBjC>t-V>!y{YzAdp
znA@}$a!^}V+$qq9S;;L4kaU=w2*ed)D#1!*a^R;rSJ_JMSfN#DR;zr1_6Y`h6S0jm
zeE_+(R196f{NDa)BwUR*pokVXM^X=W#v1A2qiWEc_1{yUV0)dq0m8)c)2vYCclk1P
z-#Mu4zYV8WXsJqL2p~lzp0QsxVl}yx!X>8JD5T?YT3c@VAcXr{2n-LUd{5Xo7_1cH
zEbMPqU0yu$R)$Nb(t%x;I2yB1@|281NyX)^a!d84Y@5g{_^_)`o=-ll22C-n_C}o>
z-)2s#I~E)W!8wDVH9ZKnK}ioXTBxUL$sIdD44VMDz_>2l!Y3sV%HDkm=ZV2J=&2tD
z5?;&jll_mR=N{&L$gc0Tepj{ZHQ|mGC2K${BH*GaBwu^Mq4gP
zx)6GsRCK^cCUK9ysF;DOY~CMKHMQ{ltwm!e0*7p7TvN{&?7dwEh-!jV&@g@Gkh?|N
ztLsMd*2LqxHGYCo(xPzMD~>-d174oY&?``G)g`^v>a{)SbR*=ZZ5uJ|z>jrr1h(t@
z=p82=s_H_rV=tTUg);>I0MQR!f&#%K4p3V(&^ZO9zS|OIii0R
zT_HL2hvsN|@M9c%xS^>m#=|SRO!1P(R}vQB!o{(z5H6_ec0;qA*&2;|Q+OxM;==Wz+{yL|DD^-&`C8v_{Lv~
zmod3IXC$u5cAh*6<%oyx(Va0}Rmi&Mdk$2>LqFaxd&GmkT%+YBUn82N_Cv&h?!XYz+8?Oj>P&~@;l*1Y6Gj`vt!GlZ{u%Wb>3dUaI+vziKe953-iFIM#BDS6X
z0M{wggh=76j}kbO3$P;pwM6^M@s@T;v
z)TwKXFc(Af-yrk=f&2iXd*H%&03!e|zH!ICdB8&f-$48)AAlI*bN>K5b|?diFQIk#
zg8#$-DG>ny0RRF50s;a61_J;9000315g{=UK~Z5Kae<+cvBA;s@i0K)|Jncu0RsU6
zKM;r$j%`^5i3rOJgQC{f8(`TpxjLIBvEozCm`ObAEjt
zUgteZTpGyq*k&xpUMJIYa$E#2BVgfa`H!V7dWbbE*3gnmkE=~(wn?%vMPR2b(`kxp-7uV$TGTDb@YjSO~
zF((}5gVcMcnp~S>Ev{}(EIA>K+ip)blerIeP%jTw*#X7I@neD9CL-@bSBS3m7O$zz
z!t@C%Bw?l`XQJzwIp$9~xhOjq74Civdg7UlmS#^kp)$wDvt;>`lFJ?!$b8GTxmr0O
zUT$T+4Rp!VXt?vWql=Y^>N|sZyJD{q=}vqSd<3JrEUc%ylDNlAcW9r8KSEdaCo!aV
zBna8%zR2}um$;{M>suD@A27J7q$vp}HlD40P3Zu^upHm*eaS1qCs!^f$neR?@QuMP
zC%Zb|QwIB5dx11aF!1SW73U%CmoS1lGTYSai#fiK*T6LK?d$zXF3z%aJ$YP>v1JIs
z;ATuB_W)?dEnBVAVnkv9OR?i5c!tt=!=nb06T|gl{C@4b>K!z@@$M7ck5=(!K@2k2
ztJ*gCC!j!NZOLmYK+B&p@0@&ry<4`-?LIi}Ifw*5i=S~5EG}7rTVWQ}(Klen_Wr`-2n
z7s2|gYgzU2fXxjQDWM)f{tb78&>OU5Gka&yYXFHSZcFlUQ24dT`
z1(0|uM%6E?HRyo)3yTRJEq4jwV=VADdOjxNE%=qaT^?h}#r?n9>@8UEgBolb9_)li
zqD0HTg^ZVNICjm|xH2}2nO#A#A5v*{mpo373y#4UNRMriHI|tprHzns99f<^WC4e-
z?#&+Fd^0FE;PWq|#Ola`^^nK^{_<7GY}dWQjMoX?IdUZ8ruX7r%$v-c?h
zfZ2M(&j#om{M_#%al_%m+@KwJnewctoWJ~w>`McK-LBS5=s`0lto)fkLG6P1X9vna
zj7}{F+}g?EbGhx@Cehp+%gwV5uc-UZ`|;JXtT`>q-L1G0Ka301LVH}{dzXsw0j0#f
zTzh8jFq{|z#jUk!$u03ZmRt8=(%{=4iasLvKe+Qd$NcIGuU16hv{T{&-A)#kU`d=@
z`;+5Jxp5tuU^aRryMgs{x#KnB?Tx~9xPJct(so`Sh%9U#EvGkeHPYdkLKs6|A85B@
z)Sj|6lULrZpWV@FfvW5~f{Rl`N
zuT_`L&+g@}o0L`qj}wL6Iga>cJ@;~%%nn)oT+9=@?nO6l**rS19^@>U!5BN~;4(Oa
zUTjAvcN>v8@_lPlwAp5={gLU?>8n%xtr
zE3oN$lFRDPBe43931p42$%eS_qb-$G<15q#cE98J1aM^(?=4cd2nKlQ(4FWttX1
zd$(j`seWwV2_1N2j-cM|qnUGwR&^tH!u1DdlPtYSDa>z>_Rl|->=ry*X*o??j}}QR
z^&tb*hb#b3c6i7FTOYgbP~o@13m>~AkEA}y3#F6Tf0UTUaB#-BOxt9f&sXjYpMPAwYGGPcs=AB0AaBm*me%$WcAa=0(qZ^
zFN~WT>%&Iv(ssmM^DsX4SjTthf=a~ocaGb{uv@XmbIEg%;9l-Sa3|={zC7|10E{~S
z0K&8Job>{isP=lekyZuZw|KKBU1P^NvT|exi6r@Yo#!)?yBHD0@LrY#vg<6CTaui%
zUZ9;GN!KjnnQhyvyQ{l`GDkGB9C)@Sx824(?Y8*eE5Z18Es^L;UCvMJl>4+{ZLQ*d
zu^ri$sOvc;*Zj*OLVVBe0v6-y(+rCBKQLrG?oTY5Kv@J3)>oE)enJZP8xhGiTanF|
z1U+n()=0JR1ErUzF`T%=ryqveS@1&;N9YzF;lmN&@>n#Ec#tNIZW1k*iNUd+h!8#4
zK|jI=A#n-TTXdP?XMI}lFbTrTF&TQ^#Fsy2`fVaUpoua4dvauTkc=ntK1i;2a*-B
zmhs%{f>Q#~dAH)`5jwZQ)Dp>Uh3<&C&6VzM__FVdeyqIoSzmhJxFiWj;7?r23aU~Z68j5f5rP#c}D$6@GeGF^#Qlz#@Nh~;AC?S
zT2I0<&mOP4wm6((-X$jePmFhGJl(%`Iq%1O!DO)8(GF4MoR-{{#&;rfkQiJx*SHcF
zEZYV1KBd1&8O7MWOM}(_04tHjmO~?n)zM@66EHZ*_`#o!FO5A!&9m%7>R(bj8+=zj
ztxhH3;jDb)R~uxW4~h7_Y--KGA9n{{7sI4J13%RjY_d_K21>nTeUXe*m)LP4kxn2i1w)--ZKq;2yGbF7n#8ajZAEj=m2b_G6P9
zspfYFs4rH$GJZT|lbrc4X?yK4E)9lxxvV*tUmFabh|ay-bJSQ^PlR?M@4Sk}4-8y;
zxVOsWZ9AGB!JHvnn(!}|gv&Xe(pb-v$>Cgy+|QOYc%9WB4YjxMT82Eyqz-1jB~JO+
zyCu&f1{Xg1w_n4*18jWG;KLcqAhF{tv5A;VytW4tna3rA#H$C)g7G22?mp$PdVrdK
zE_bMaUk;2etdA#+w=b-3=$jAl=WTQA?kPhJkAs20rd!=WXSFnGuz
zu`UUHvaOjiX^?nlfddbPa9NY&l_Ou?Qw%-iG3y;ULS{xAPQ@w98UKqy`
zq{t`R#tb>MXGslH@LXiLT8i9|>UQ4={59ZYu{PMdTCz5oIkO-fZbsQ6e=ziU@v<|S
zYz_+x#h->r<&a5p)y#vja`zlszR>OJ&jXH@nVc}sP<JQot(ZIaQp&1WtUcL{Sn79
z@Oxa13mXX!;7E4Cdm$VN>$C0`&P2z8RWX9b3p{4WHrZq98!VeQ!8{5GJ=yVPmRaCEzEvb6mo|=@?6-TD
z)%UXRZ38*5IJ3rC<6)E$j~Np_Y_iKNgDkRUSp}9^C7uCgmRaGsY>*asWtKtXJY+bs
zIU{^^3Uw;oxp>dWnDF&INB)82{0Z>*^&f%c@5ia1Q`_Y7_JBSj$b5Ks
zVrO_^uHnc}yMea5)E@zs99>?abcI^lEP+mMqUfJEIFUH;$inVLgc%_jX(rruHSya3
z1&KRrjONmF;5tLjTwWf8?s`i5p1*LN;6imcyvCjjgNV+|XQ}ErVhBtkK)e8Y^VR2&|Lpx;lTYcQao~0)f1O)~zn=HOJ!gRhuQEu&bi9umL8q)tl1f7Y&_rE{{ZapLxj7*A1V5WBbgb@aR&G|H@JdJwUY(*HIN}&
zEqDl6TPKDrrLYC3hzH%uB;6d2Bqv+32-AI$E|3u?fY3
zS;|-f%3a;v5G0a)NOjj*KI%YE8WjL!E5PyYa8IA^i{
z0Lz19cnc%Wd`I1}%ugl8NPtV1QR^Mjcbgdxfy9ZT5Svri=$D2aK|u~;*aNe}9Zw~^
z^z#_txm;O!Je?w7+$X{C?Y4ZL<<+vxPbM*VLOye+{{VWETpt6Zf))w~f&%UQj~jiY1FIVcSQ}!0*$>Q@
z=fG#c;owIR)GhFOj(PBd!_>jU*HMBXaJ2Pv^CX@)dz+tZjwiV+j|3f(47yA{8->1$
zmPG?RHsxZu@ywmgxxO9UxQ@7hJb~OgL)_$r?n!jINqAoVZ$WTjaEyW}hZEvOgYbD1
z+-H({_!}@XCtUhS>8O-v!
zLUwaIk~|XFj@WfR$-zlpB)pYAGA_1`?c>H<;|@z-4jfOq#OktWPtq6(d?DY1m^qhO
z#mjIbs5QPf!WdEWZap);?7g-=V)kR3UdxH3TQD{S>PIDE>0!n*19ZLk{wkucch#mhQhs2=Xj^&_m0Ld4<};@Cjm?yQ|o
zxtJrl>%-JgOhOh<4sH74%;mXVZL}
z
zcrHV7Ea!rSJZqe}4}hM+oUMPGi@mQ5KAAG*wZ7%kn_O?Tl+2KE=IfCFO+d20f0LbU
zp||VNbS4JJT1hqHJrmTZOd+tp0ByO&mx2=AHXoa^!hA%tsO6k-JaHa$@zZ4KZrXa2
zSRnBD`k%iD9RVl%S@A5^JOidM_Yrmy;!io_l2->LckW%JT)Evd@8Sbd^$`0dffPHq
z8vxCe^F7CK6T^YVGbgz0j}0LMAe7eM2M-p$;8+Jf19_J&Ui@nuws=GW>J8`d9%Gl7
zlpS*!hCB{Bf!}t9{YS)>MX(MdWF{QuP$N=hK0-KY%RA=8a}Lh~mKXOUT*pqQczd&v
zEI)APv5aEkpe#`sD4V2tJ>4o_#Ok&2@PXXeVN{H<6RFa)8+NOSRu;W{Yi5&CTmyXGH&
zbI$r)_8-8C{+!#<`VWt^Njgm??ol2s9!-k8dkvoa$+j~H=EDfZjo=v9wj`BFiGa+t
z#ockY6~&j)0Ku0b!Qzvbq;<6^Js1ZzVPbM~r6xx4*`wZ}j@tOm^S{LPtD%g?vSYo2
z4tV+af`{%dTj~D3OQa{^X{)KPSxEL=)HB|O*T~mG5+oCo7g4Rk_u4K)HqcA)Hai*J
z3El7#go@!&bGi>^ams9`r4KuD=*xGqxxFj#WY}!&yYuB&M)w1<^ll2cOS0OZ1)kLe
zzE=Cjw!mF}pMJhkE;dqLt$I$t#&hMBKuV@O`|yq?lDPWmyU}wAG(!QqAGGQVKc70Y
z%fV-~0
z_RqnD&qHGGx4Tfu=h%fqwT?8z*OO$f6liUpG;Y(+h4$H^FP5wj3=OZwo(Q;IqN)+)
zIbU7WktzmUniR~G!U}>5ICT3zuB+7}f@_4E7w%kjUYcYkUTOs|?Y^oBIb!`e(@N=r
z`UDR?evmSXQ$$Huk?%kX#ty%9AkNu0i6#mH$}At>>rs@E4NMU|8$k3FdST-nN7FHN
zAK1m(9g3Y?toLXK2CR~x$HI4Hq4-5`u{#0JSr{D1da-1Z{l<$BnQGGx&o}n&
zoOJImhn|>{9A(U-_C<5{P#sR*Ms4ER7x#e3y{FQ(%-#4>fbg}M;9LwCqi8)*dP~~M?J#(J%VopCkey(@N8fI_jTgjrN9xgK82h5FKf=9TyVr(XMm!K$}*%#;*-A~A{S2dq9q{n2J
zqZOp~T4<$2u`>tM`dx6JDyWx1_efnHdc0I!7#Cmc7+*3~x$Ft_dNSTK17wH3%_#_J
zX*e{m>y<9o|4yok8+WgQ)`;x(x-LX|)QI7ST)x7x!T$8_#H9u?8fnkYm^;OcjUg2b
zxH@29IBX4tiyiO|d4=yCPcn$U`|HouH+O#9dN>}nB-j_zM{q3L&pGA<4gQIcLUwWCFRbEzY%9
z$Q$RCev%#%v;K)DX89WDa_MJ2@!9s2a>np4?JGluKd0k9Lb!~VzwB^lYQha_d48qO
z>QRY>jR;*_*JyozrzE~^bIP5(Zj^HsrHof4;C}f(^kr`UT<*%T`yl!4VdQ^M!S(;ubb$Y-
zru$zrUGK~$`O($&|C#B?yIjB$UQpO`)eH+NvR@_hIjYDXNv}m6C<33kz`__-!l;9-
zE~9H)b36%&Mrq?37(9ux)q)S<{#%6j_Z66iAlZa9JT!UF%s}R+kf|H0nI9_-iI=?k
zdbh&S=s?fLuC1@mj4PIUnKFr;D&HtZ;BWLlvEs4+PPwmy`?maLe{0Bg5*!A<$1vf~
zZc`>0VemQ1Ow#CtuuN!kXQ(+aJb~}r-!ke;bx4
zd14Cf16kLtNq08y6O)aS6wjy`m6Q
zN8}9oTzYX;`9fx)yP67g(?=MmXMM*ttkg(kDcqW@2WZr(t<*4VryUU27m*dME>-
z)Sd5PKj%-G>!f&KPp1V^^p07Nm)^ABm-KiINgR%tNs`P6)a963#noABGEI{c=^?Ny
zW`&Q#Y|j-lC+qqvCyJfK--!6HOtw_}cTngS(g)>VdbF>Sfa_2OJE}M@9FGDMB0Bo}
zK#^@T>HrejbF8xx58?thmj8OP@;??}WBz1DVfTT9X))tU!2XETIW<^O#j@M=3oENh
z3oD-3^f(t?6PKKiKY%3pxwYqUEd_(tx3G)};640);a*=p%DqcxCAm(^A!)p#O4|1hq)Ld1DXI@&
z<1QjK<=)z|Rm+E3$5!i0_6WPzzqS`xe<+!7<^x~oHXL@jA}*%L&F3!mX!L?i>P=$V
z^WU3OIX@qH?sGzm2{$x>B*CxQbvY;IqCgVJxrL|ZskRGm=`KX5_}hxv5>?#GN!gD%
zmbT2XmsSK!*5bQOi#nY=4cO@W{*h9j`73dIQKFwkSN?kB2UOHA1%h;M?X5o;ukS%-+=S{$UP*NoVLVoftQK6CWQ(z>QKM_$UaWp|W@kr~W>6I+iD
zDcw8f*hBOBM`d`B;F0fQELPMOi4``mnmC52RL7!oT21g?A{&aYolwqfjOBi43SSMA
z?w@Br^x;?U;~M6h%-92uN&1~-UjIG@^y(hvi?I)iZz$C-{ZW1QkY7v}f|ifJP1~RJ
ztIgoTPh)VhV(LjEQFC-IDW{w(pR5fY;1UX~I>UH>e#o|_znb0Dk_0zoPP$%#z_QuP%ln_UjSYh3N*S$CbCtkQgNkjrV`|G-7%&%o|&DtAr~4EzSy>coo`bw)oSv0h<*
zVZP)OM4!TF-%AhywhjJ1GeMK;mpjyo4GF3A;$uTgODqI-SBU-I1Gc4tyuq2+mTicn
zELoPT6XVIlvmlhVP6-ypMGCdh@=K>Jr0b8FJJ^NX-i@-0UV9M!-FiznyJ9)-!w
zh+9;|XXV=2Yia5Z`btQ|tQt1&Xq!`he`lg4R{|PG-Qt*(8CTLc2UFqlogkP?f2yq5
zAF`uHl6nx6BwFowm#j1rnF&s_JnQBz56Pc7Q&*A&1#JmVz^-L8R7&IB3_kvQF{G}!
z4?*6_y})^`(r!4^INHWV^B(}EZSKqLNAval@5)}~S>tm-Aq$ivvuE6fAICc?SNwJ)
z*qE|?;A%GMNk>4kFYrq2;%!kgfxygc{qBrD+m>PtcBU75W7vPl!qhZ=rS|-Y(rN)f1J~o4xniB?&F3T5eL|-UPr1smD;PLY{IDpamlkT>kaW->e(k%@(I
zb&3A~hvfOo{{T%m90?&kz{|M9!QaRLhVL!->%R-~D(DSocnm%qy)l&PwJ
z8R!teOuaPbI+2QG6*9^a>C9(gHYrCO&*;|TMTrt_GRS`D3KVWbDLcg}dx-q@2nEDN
z*Mk|jf~20O(DKd=DXM{*uU7FiJ+a=T!RpqtRG%-K5#;xysvn)j;U(pR<+(
z$JGm4(0I=S)JA~BLeu1Zw-lxW2Qmd$Hh!vnZr|i()nV&~R1beHeggW?tSD$pEV3~Z
zu+`1x4JBf*R1_gLfg(o*BVC$+=9-?&?AI@7le2xz_G^48w?!+Id>$(3zZ
zoug*`pdqU?%gxzrN4
zEER;C^>JNbWI5lr-wIBihBws_MWP_w)3dylLdJ^}aKctkz@FjB9&M3qBGi)%X;4ar
zid%i2MVy>@I3e1Kea`;T@auiTSNzDdx;EQ|!^8wMY4hCSgO>Bq>YZ(
zk&CxjO<3ZZH)|?lMC$xgR}@<&5|<#%>)Mdt=R_W^SSkH2e7wmQMir*2IcFaYkM&m3
zx;SpsJojZ<2|R2>BQ(Nl{TxYz->FgzbQ5WhDfX4cKH1)6e9Q>YlJ8^73shQK7Y&(b
zz4u^XWu38oatIl!jx6)yK&`76W;?TgE#-hgVzo5(Le9!2LU{qkjbY2aKbnwok9C+;{l0qI+pfA{y@6bfoN1dOhd~Pqq
z$wPsmi`lPnF;+mZCnxbKcRjh(-^*xTfHi(a10|sPws@>+4XZMj`XbcP9
zvsKyVG&q^2#tQjiZBv$VNi3mRg~l~SCOicPoiclx@5tZxgqGv!)452sP{}Y-T=p2y
z=%$_HdaNDeP$5odNE}V}FI^Rh+BA*VdCz;D4pXKpqD$miN$SyfqH}*#arqIXV3_)H
z%~#S_<(~3UDJ0j#AiQIKzFVHUTG74|(ynaDsA4R9S82sq2Xj#m4BFnhbWF1}!efaxhQr@S-(9XO`7X|J#I%^NKW~ogXhkERE_WDZh
z!U-Fx6i}qq0AYR(+}!ft^&LPAg%bCXQxC37(eD
zzB{bR$P1;=7}V04_**>62I|b>AuKzL%CmUj3*+GkBX10GhW(b5t#6|zpRf7{z}YM(
z3V5C_m53F0g!u7{gVioL)?}tT*gZQ!8ADb
zCf07N;NX}ek2wOGwzCMM!VY%3D@b|3dJ?PueKU(L{sGN>7z}rdHTi|44_$mSLZ+?|{~bO_KkJMZSyV>DQb^%`JOW%1R$bgf}1k1iw){
zj+D!714;II`f*l$ja!-Jk_%2BdU)*2L1;CNfz&St0(SG167wR{3|
zst0UqBXGA>K0z~pq(x|p=6*VfP3?-AKeLWRy}k3<^auCzf{=8L5943J;c*zE!CSE#
zC35tNFV{
zYz7Oj&xt5S@*ZlW35Fb`XJ5=$dMt}Q;i{Iri4XaeBCEY+hFs3ht=htl%8&>$eV>wS
ziSq_6Q>6BYu5-D}VB#@1V=3TJ@o7q_p3Z-=t=FC!06u4xA-d+TV|Q}yR4>IAsN*jo
zcNt-|2dso%3Rs=v<2s&_vxL0UId`7ArY$vs?ylfg-|$6dzO^#5Xpt5Z`|#IeSPtU3
zvxo0`Y9qyeJbrEij7%~At?r!^qz_8O$!fUp_!G)wjMhs8Xj4(R80bFfUP(G0gp4r4x{zw`#p6^xV^oTW+?Z?!Pbko^$rT(JO!eRgE1LT
zdhb&_@QTdp_F<}82P<}J>+gd9rnIkmWx&4E?a{PvF!reVoK`M=rD!HI!Q(k7Cz@sU
zbPYE_i|f-@w88vtGG8AM59)hArn%5pnR5vHD#+%p@OiA3<8#zhc2L@5fl8>ZN1o0c
z{^e856709h$J)?$o7Zw4i~E8hJ)#Kju4zr~emxb+vPB;oNbfkgHSJ*ZFfoq2I9?YrcDD;x)aqgg1f_fF*9y4H3$CFIr&P$
z^V6T(>~C7UidI^WMygnUuE@O9P@S9AlNl)i|4yD3wyXO3XW@NqmU$Obp?k+zHMuc`
zhOYdI-EXLM_DPsuN&p>W+yU_nxeNfo=*=Li$i
zJM$0FFA2>~3?OmJxc>(E*{^k4Zgy~oLpD7nmmbvSI3hQkB+u#IluV?YvA>A{szS6#|
zefgQ^uPFa$rKX+4eZJCA33y|ZVk)>4GH)J@EvTnNE2wHHLJ^-t4B8?C#Wl1WO5K2b
zbSBa_@%H6?B}~7SY9K$C@&L`c?CrtZXQlNz-7w(?f@dQ&N9G2-#gsxx
z51Fay=xCeIu2FU4y@iswb1E}QjSqUwtqu72v}iFA?og(`>h5lt8#x)`McKTfKOm~+
zn<`u`#xCBR!v8INmDG8Cv9A@>(bn*uAd(2Gkmp&K%eyHN%0Pj>jcXnrEpO3Q@^4D4
z+rd=S_iXfui|4$EO%5B%cwB2r
zRF{iYQlAyXWZn_>z-E5f`1K5YOFUv&mr@V{ms42h&k_C>gfbMSF_xosqU=Pehus{z
z@Bvn~Lo}YEOpE6bKfagleeKdR!$s}RCl~~*JH}oHU3b>4@t)hcD?C)N5Wd~&PP!Wy
zd>Od-j_TfkU0{Ig%d2Yvu;)u)B$svwZbfXa&R>i%@Y&DBD|~u-qWI(?Q}rU0LE`X8
zZ5Rn2Ie?~TYl-x$-7w=6d7gh4_I}kv3S@dLgU>Fb<%Hj5Oi3RR{uZtyj6%#yKChbp
z;L`oUpyMCFW8kjfAD}V=XAU(uN@>$G)Y0RrO?S=soczk>!c%|{t
zcq3U(2z&&yl;)Q1xr9Xq*Miufsadt^!WU`Iv<0={3;q?ih6Z(X=@Eo9&gvMQsTeH|8^;)P+5{&81Q_5@;fIxnR
zGTDgp{C(yX6Kc^(ga{#;h7}CB<^A}Ge62dlM%d?IH$0-HS;YWB0;-RBE~Z!ekxzB?
zMOdG2YVFeEV-`G+WI}V%W$3zwvXcfN&VJt(A(huVH9Dd)vJawM4h=vLIjF*PBC-fno=!(zsCYeFxLXKyqDs4lIh
z^4&;ZD*+&^kdER@*=^D(VUi#@Sq8BYfBd@D8am(NGLavm^=D
z*Qeq(ON#qh{+aPj9iM`6zS;bh=PP&}%44-mKq8Eg*`Lsp^rzf&*>i~)*EZctkWe$g
z1vVQ3%;BsRouO#&74!!S)d}%D3jxlV>IDrIjEpitd*Q6+I-Xb)W7LQ*LsV#aTgNaI
zQ+mwfH@0;+ybRZGDAlG=^^MA5Y4LN2p%6V7vZgU&6QjVZR$(e7K6M@jkI#ROS(mBy
zv@Z;Ctb`a@
z5F0&bk9}($u~mQ8Qs_boYz-Do+o~9nXub+HX?-Dn=5+_`EPV5`yWUtH)?)dlLDMS&
zpDm5&1kH+j5u=u_c@gUL8)6Jx`x&#sNuc*;8SLY?KTl_Tj=agfR-ALbiD>zh)GD%qKp1_=zfVhodJ^@QSr7
z>XJq$d{ji+aO_xo(}ET`iLigK#jA~;NsV10>c1l-lspER$8j#q>Qcd)S2=%`yqB!0
zUs$^8xf5Jb7ImRLqkeM@q=nve-GW!YX9jAx9+)v@Ym!NlS_Qff_P#>dbRIu?J4kH5geeko{>@trE}$e;2+2W=4&A5xEF+~Y#XtPN5H
zkb;}(ZH^(6tfQNyQ)Vo}QKdQjjU!GTy;Bz}fV)G5qIX1tJ~@e0Oe2-S?M;+hgBb%+
zv;+mx2+9P$i#+nH1ojpXhWiWPNFX#qI9tJh9$P=q-M8me1do7)Y*2;IjrF3#KfnhM
zLd1=}#X#Zfzt`)(g<3SbH(===`imW>f6_qOW&A^StgWXw;lAe+oP
zBVS^E{<}1PlW-tCyTa9y;}Y9wS#V4R{9D_GL3WwEGL%zuLsKLEAefwYsE9Z!tmB(fm(%8+dK|&^Dv$}6l43W{PinIQey5t
z;!NkSVf4*@rTmLnjH=msiPv=YKZ`!-f*my>Qz3u9&q{jh;{)a{2yTW78kx6e;4|M4~*
z8rESB*Kym;RWowth~bodDFVLD^enWp@P{PX*u`dU)Ze3wLg)EfEptMr5l^)kjWjBh
z?wiS$@>j;cUJvRLM2!o2S$
z%%kc~xsAOV01!w!3S<%jtQ;n)9;=*B8vKk
z&N>d%Zt>~9yg2sR;(f&p^_gfIxc(m?R|U?O7m3wi)-t5=t<>KW4Br^yryfyZsTZ3uWuMVLDjV6f#odb!Z7+$C0*g6JZN5^ok~##YpmT&acJKjKlZ(_y{ccq!ps>oVQM
zWOpzb4tOYn`e6RCjg6fi#7-L+0V~X;QlHBs1yf&!>P?Bih+P@1k)}+c
zd$nl(IE`KDT!zCj^cJ6vSa8bZmID}Xd_EocP;#E0q$UgC#y&P4yrsmjjhc#KG&pek
zMC|yD{+!C5PiXrlGj0r{LJAecCOrqWS9w@P4bnw(@=@23yTJK?)(P1awe9~YtGF8>gm8@*Y_
zPF}o1MSk5I@VHVQgomNff|-^LSNIESd?`=Na`u=AK#}tdFeSogPoGl|#FP?H3*(_f
zJ;;XWPyGB^u0v`Qa)Z~IRaxOJl+JB0k9iV_mcho|L00q@LkC-uI}Ph2a2=mdXIoew
z{sD{;O{V()9ZSGyDzj@Vw^OcKzjieNtZ!*#QI+Doo-ga~v=wjPW#}QJ7M(CI0fd+U
zOlYih?!rcgVGJpikc`EcWc2L&a}8p|6NbJ}EGVO&u4M8{F`sHYCl)zb77(jPbf`co
zWKKkcUm+HRCT8N<-SO{XY*?otGH9SvF%PG3tGJ+)66+xN4!2BpV;Jqd3Q5fm$r{E1
zw7Yr4K^jx_V;Z02q4cQmkZ{uwnZy#l^Ij@q!?O3BtCTx7FQ$D@Q?QP(@yNS#^&?{n
zSHbtWm6f>sNHg!Ab+2bH$=OyE^>1r~!zt_wDmY!XkTSl|J*Ih6xKS#?
zvEx%_<_grEu5o51UhIceCTg3B!xQmAFK@`hX-2c|S%m$cCq@Q(Fho&E?QA~kt%dOx
zOT>p9!KzsF^@DefO{&V+%8uTMhwieE4;)i?TADZX6{1K2*BtC|1SZVA)VR^3MP+{x
zD{O=$qGG-u6Nm;HbU6c72P4GDWJJtkZ04&~0L;Y<$o6X>wiHyRT#y2EDtqbv&c1#o
z>W!m^MsA+RD(yCebGPmRo&Zps(9NpkI{7qlEDsv7=J|NT4;+ZrnQf)=636;72X!$2
z&YtseW=^!IA?7ap
z+G=J}Pu~7Y8}!Pk8WwzB8GRI1;W+PO9;b8ma*Y)R8cfAN0Mfq5wC%O}$YgVU;B)<_
z3F-|yQ!xdeem3#vuy&Nohx#a-nOxVu&XxN6qNH$0bjEz2@yY`$(Y)cu{x%;+uu!3^
zi6tSF@^gK@cxH*_N5cEG49K(-jEvik)dzrWnhr$Z857s7PM*H4Mpi6iNwpiy`;(PSH)
z{~Gm#l{|R#c`5zl+(w1uR{e19%txeReN10c=Pk3dZqA1W-#s)v&nvjGg%W2~uz%5&
z{Z$^8^-RVkL4|wLNHi?OH0*_@Gp`f%ibUt$7qijEy&*@C0-44Kfn^W_gzxlE9m6eF
zrdw@=i_T6cp4q
zrwxA-0j*9FZI5ftm)!_U2p;LH;NULo{k8j@9;8kmmiLF~hL=wu=pR7G67oUn5UH)-
zL*h0*j~^AtNAEo~M67YOc`TX0Wnls446{|Ya_CiYATPPL{Vp~~48a;Z<}_97
zz!FQW*oFc-A|4B5;_DWFrRwoIf%$*w-enA1Y+YECXLMwGW*>T}rlDP*kYr*Yf=#Y~
zUj1LG;l4lFEnwiqEt+*1SjlLj7s-5n6~ZHJT)%U+$LEp>-U#!k^Hw0a6wzD~UvK7(
z>J;k{Y#|dXPxzJPN+#^@ruL$VPCA+$tChx=^Bay#w2sG6mNaEX%P&-iX##VUbM7T}
zL_F1teq45Ht)+gtD2{EriZ6^!VwPsIdHZGS9t9+&4UU*v6Mno?>?Fk%1B>Xq-ry{8
zu$6gL@g(b4uGG3DHpr?K9v6
zP0odWEAZ^2N^@Mp5G{rii&SJ0Iz0)&WB-LviknaQ__*bh6e)Ze_tBC3tn&&D(w(X(
zwz#E{6E6U*;M?x7C2R<-R5CwXGo1>OjHj6Wim*VEdRMgdN7}dxX}y@wbSiqGF1LO8
z`iOZJXeWQ2t(>Vtk}oNoAmw>v2iF1Ec8{{0Q2b!p<+HA>Pk1Or<8D7{oQbRSleBur
zVcyJr2(;Jo$zh=YvXRilqwq)pWbF+yq;T%-qY4)i
zvc0J4Y)C`XhP}!ybUG)z(qJJ-y(mhy%#a#T;V3@x3`_{S(c;`wh;IvR+vAk4K`bC4KIfW!k(OwYS1PX525TJ5~*U18lq>1VT(j*
zh(PhMVvss>`wRAj&lBI>kny{rq^M$3TBqBL<}yhQUw!1V?e{XPFd5}_qw)I&
ztg5De~hK1r0pHTRz
zy+~x~2eKVZWcAAG4-xv?#KpqbdaY%Xe_f5v=Wt|(-+P-7mM=(@K^LR4b(x6%vWgzV
z{{i~xw*|ije+Jm~%;awk@xM~v5-YfcS+gf|s{X}XlfJ)sTuBF#_ChkbzLpUAn;Bcs
zbEghd7&m?@_fg@ph*A5wZj`ZIjF;z;x-|cUGcO>GK!}L3pf3@me-*#AHe1Z}H8WUh
z`jjC)I59wAewv>`TWW-jHt`=o959p{U2fxlP2WMp>~~s;j6(x*Zilq
z*gWZ(uA5=VeWE-hs1awA@!n%|>;??}o$>h(s;4E^UaXb5!NNJG4@KV>T5bXqkP76J
z^@mN^JDC8fsQXHKYMS*ljF%`snGE2T)H#ghdOwA4pFY!V989+RTAN}^1aOTEPm_SW
z9-j0l9BN1(P{QbEX3iXzo9{~G80M^#1)1>D)=fku`HC>gTZfQT;3=T{>Al|`3zH?6
zPRbdwe+@6+(>gZI2GGK8a0c89o18*<4#nG+E@N?uXSttH*xzPOR
z$C2<|i`|1;+Ww+qZb)2xyKUAdjD?S*S#6C=)2>+f9UxMcb}%dUD={|ONr=-X3?dMa
z2JH>mwh|%9*PpcH-}Y9>{q6C4c1Q+36mofbdD)3_{;-Hzexa(_ZbQd0nx&7F(_E!L
zwmE>HrIqv8ygKYv1#JhDTsU`pr;Oi9+wJQ?)%+cFgQY6hJxyQA%5#`%;yICP^xO4L
zWoo|#-|(s>O6)di^O|Nu%lq(-h3bzgc6YrsX@GO5z?D|7(F)zksaOQH
zw|OEqk=kHT<~(fUk4A+`GalkD_bBcCMMWx-o>P}(1D8)AGqd0yV3-Cbcj{>M2iYqo
zPJXm6ELJ6C{;=M%88hw6K3tiH+=PB0=kLjsn2{nC!qqhBOGKR_d6TLN}YXKT-G&|}@4d^bOkGLT`F=V?Hl77|aJ+#|a5Vq&L^
z#if=fotC|g@5Nfys5?FW0azZ0D3K8SX~>1VBNn#hp4kvsP84wef$K+J8Ua?aR*kkk
zWJ3pU8BTItbuJ)j4^>n~^H`PQ%T4q%E%a*lW+}PdCGmf-ei@l)#2RDfgR!B>aqTvu
z*z!9iqmfBSiJ?07qr~*^>l`m4<2cW*BEFbd0>+?DKYeA5+#Qrz#0#B%h|y#Sv^+esmxTZhBTiWhO0(n#+2tsI=loA>)CIj#~*gj
z1ctLB_}?&ueVolkvK1%J&7i2fockT4azl4NSX`sD5g375DFv*y)Tf~u$JW<|D!F?Ypq{(gVPO@geLfWDrdPvbTC(-9PJjY*^hL;
zQmzXts*d*zS;mcQjq%?YJMZC7!6W_wC@0v7TWkISGXL8mK$av4mF@czq&gl6T2H$l
z*$0PEqHB=GgCB(Wl&a_bEMLgOc<`RIWhd2ePPgCwnbSywEQ$>%P%KvMVYuj3L=?98
zGV}eI=t}@gQHVDk0a%vO`Q>x~X1#I~Lt<5dKhll_hC)Qa&z+E*M*ISkLBVp7hvPS{f_Kew>HHex0+fhmL8D~?43u{9Awb<8x%3x>lIu*TT)#=BNyh@<~uZ}mtD?-
zIy{qPOCggma@BXS!t6MVH?N)0fulD&l^aihb8|d>vEYp$I3YP2Ep0O_EH^ng$Q=VIt>Z7kF
zfnAvw%#zIfk(v0QO*=9&L$ufa`7OXrAh@gi4F%6Hp%VbsOLo3I1?$Z8~gotZa5Ks|qhk0n#GG8djtnbWNR=;~R-6FmbIb`^dkA
zv{Xcj@Hu4JrGp@d@e{97G47lK2H$BtAdeFdP^ds{*IA;MX9RJ8SHmhapyA|nE+-2;Z5r@AJiNHLcwcWED%EStw><%Xa$Vl`L
zP)fPMLXfm*g(#sS96(Qr&A!cDw7vHQTV>+YdQ=hA{DFqy$o`6oyM2ggim;K>i%
zaKWim;^|V{#LXa31kakTK_Cs$%V!1SR3v6v-4CU?95Gwurd{Ik4#$Wj+p&=*`i0|r
zV$;!y1F-yI|}ua*TLc=(-=>rw*l
z9;ykKE-)}-$FeY41Z#&-Yxd{CDqo*35&kE)xMs(z`fDid_TIa)*J`d9fPz=t5j$fg
z#hHA<=PujErGh)VrybXP3R=Z+ko@V!O^kyVhmv#YF1`M$7&Yfs_%9?PaX4-{m9EDPyu)*%|frnua0?&1{9*0)TcsW(0p4e;CG=a4u9nY^FL{8p)Dn+>-ST+q2JyF!ydYeo8CzgaW$YS@75!sB
zE~dbO=FRtw-JU|KiCu`38BvU!ugefF#Lk)ka4#VPtwQ+E0OJ>|nQuf*26SW;0Qu
zg>bd-v_d9T($JQc>F@C({0rvZfX@`+u?>_^M-tZ`UeG@|BPu^+opIm35)S#KFXUfl
zLsE+}?Hq1?DcbnasSPUnTCb2k=^)r*kABpjDU@fm^e28{tDc3@i$+FNm3qcEA
z8mi19ns?Z$65;LFN8qr2{OjE5
z^y(*0vVV^zqPHJ0)-<#$YT&fZxL2p7Q=)dvEXT%kG4>RhPCX(}4S~~%s6W%9_u58A
zCb)!{yK{Q$7Qa@sKfxxbD=RWd*M)D@b=UD}BqNd3To}$=-Td>>O;&|ZnISyyd{@q#
zlrt)N;f&)M?3H}h|DlXH7$5W2kc-@*pw!LshR4zvG`@4hvm)lt!pwO-3;84dLI7w~
z!*RIoX!a@ySr7TUf5#KpL7#4cYsrc48NL<`t7zFmulx@H96;m04+l?(DUqIs`G8(<
zTQ5`d1BLB4&<{2FK@_h#z5!yCXAKsV>@CMK)*^*fUg+W}2SmE!2^WvbT4^iXr5&_U
z+zh>@1BMkGp{Rx{uMi6^U(_s@b!-)nA1^TyHk-1d)#H{J6#0OZ38BmtTg&#rj7Mk7
z3XY{%r)I=CnJl*YWtSmCQlQm6OC82*iP&q5TvY{ZFEzh<`GEjq13)9lRl)|-+Y&mo
z+Pwr|ie6kc70-w@YvZGj<^?NMbJV4!q}uks#A}GEyt1E?W(K|qM$*K)gib3{E}5mZ
zd4VYNp6Va~a3SM-e$`O~J*n
z1@|fqaDb73#TuKCp^MB^%Lst65)!vC)hO;RL@`4(U9!RzahYo~xR7SXF6J6}joE?F
ze8b062Vc-jI1?ByLqfGw6z)_ahQXooeGxni#Ig_8If4%^U)xav(yoAPc|TJ+Y79Fu
z^pwF_lYeoVi<>p-A80XIU#5Ci1Vj%3ZPC#2Fh=q_T|xRJ{>Pa~KpU3+7~PGwPcRw}
zgc8}^8lJ&=%qnjWa?rM*3d(oq5HNsraUCwfOt1>?MybaV+BIRc=!=U+?4l{TTEEOh
z2&(`$)O>!0Ie+gld#ARCUCWtX`+&e>_P+rz64s=3G|gFzxo&wc%&U>uZN3R>WYgEo
zwZ0=UahgguID&RAFQ@|2+-$*)%fy+qdx2P5r(XvXn+0r|q46la0=s>s8>DhE-V*m@
zu>);y%%=e~T&qokzaAwF8s;L3Ed5Ir2z|$Nw*!&&00@?O?fp#dK&`$2IhKk`rH^3b
zhCpg3nXa&EP^zyiEwKfIPnV)8lPG=6FqIgC(a=^3<|w&jgKz~bdc2DoN5sez^$_Fb
z{Z4dzoTWhQ$yKK}7v_sntua`Aug#RRZd+1_uqKyi@plCYc2xXCYhH3+1Tk{Uxx{1mVM}MRRRmuhe`)gem|pT(DyZ
zFCD_T3I~gf02|xaDRWbn*1|K0<}aPmmO_;*K=t@9uR!QF_#`<_QToC
z#0ubrZNl?kn1KzNFayNyqvGUc3f97xn2fOsI+&?bK+G5rqa%m~EFU|5Ad{_FT|y9C
z)wk54LpQA+<`sIV1Z*t%fVBgqTTukbiMqyk
zjaDyQy~U>U$PX;Uh^QF-1iVVBiFQ5z0CLCx8V)mD!D0nvql((mMpap>YYqdCID$fL
zo6eysuwQXAb0AJTZm(*Vh9-+C3@eQ5iBUs3KUXl5WjnDE-f^#%z`Iz!N
zMOiMc;)L%;kEw3-S$~K?O^y9Pu;a|3{@~hE+yE0XH_;`Fp@?HcECF64jfL?XLH7$_
zFX9s$`Whjg{@k$}Liq`{eNng(%O0l{yY|1M?iHX$8^^g;(r9j+XN~!fI)vydoOb#1
z1PNv4VzFAQ)$dU|qACjWl9CF1dmo>vxfBZux-CP907lONEuc~k0_)uwVMwL63<(Mo
zW~dhv0U-siW%5~x|9#qffQvvj~^^#LcTI29C!sK)ad@!Xwd8Bc*_n3%q$NLoF
zZyBE8BAg_isD&s%Tc;PzmPQ7!2MXCx>Q~9z8mzto>RMb`zdD46Ld9*xrgJG5PnSJG
zyA5ulD$eNiBm73HQdzjMjF((O87xKAP_|I+HnmU)<#M-8WQmk$+bpmTK4vg8pC2CL
zqK6o_AN#~)yC_u3!3(c;%h0@`LI`NxLvSB(R-BHvb-_mI
z3_cG&V}dx|2LutNKst9XClOlJ?IYoXmZl5&h*v9u1G=k&dx8<+ijm06-?$NPS2@b+
zAAQEey0S8Uw&}&1g$GyznY@(xs0DTlT7g7z@2DVTe)a8umX&1)ZoY%zmVJdl>6|xj
zzf${xt9GjhIOq#i;3G|Sxg1}QP){Wl0V-5TL=r67Xd7M#@lw3d=ztJkVJ&FixUUu*
z6%!#>hyjCMqhnWf6|}DAR@GGC8iqAxs@QciWlxCnI02}3C)9W~@idauv%pB<8iRfz
z3WLKkz*$2Twjw~PBX4NK1K7U|G;U&&nazLMhXf7_Uq3Ry08AZMx6i0tTIDL`s5ou6
zSHB)%4G3`sUp42=<8bX4j4WW=*B_{sP1o+V_@!SxDELOgvXCXo)b#HtnJ9Tnyn58bGW-MO9Q|#j4
zM+b|8@LSKBW0nit1(dg<6QKA?@VgfiJhmlfSk=5Vu?n4AFo0IuTirl!6G#q8V~WG*
zgxVm5L#p!|Z&t5vrmdbjp@ztn5TH7JM%~p))}swgDv56SQ4N5%8(ar#C|EXN%(4km
zlh0p>8I6OEJVl|ExDH_9WOELOSNu!sVY$RaRPF5W9-(Q~vc}dax=`j6t6WDyQB^43
z8Lkf#3;@@-N`dWQlI6Fr_JS8vX+bE37oX`7!3BpTH87C_h5n*_Qm&!|RRIke3Z`l4
z+W>G4K`o5mGbd7P>Q|QjBP%sn9o)35PSgicqPzf%1PIq}(iVywrouWkLZ%LMh}(Or|e#;UTZwA9*{+^AzE#lsl#35l7HeCiJiz-+l
z_cO6jS$f5}^C*EBv`13o%(ShsE_bU;1^mHn(mhKb9L@vWdJj4?D#l@ucOM~+BD&->zA
z*efn$3%z6ne3b|l7sjQN+D7BwUQ~33N@#mB&XfjQqQ+H*#(|;#024%jah$R6RBp~l
z%|gRp#2YRb6;bjcE%F&yqQw~qAQ*;>T*^V=E$x
zxsyD`ioz4@BHUmff(mu4719$oR&czgBOU@ZItl%ltMlB`B=0C2R+N*Uu+d{
zjc_CIBfD_I*t9X06a&rk0o>VLvm_8v(F#^qwkC$)3z~a^Ny4W+vq67wZYXZFVEjOC
zHI5l*K+_YDs5W@$mt;9bm|cbc06OT1(33aR5K=>TToni_V-3T}9}Ej*fKA`2pwZ<7
zP@+9h4Z;FFWWqgdaQsH2ooj3zGC;uXc&VaKtgeoTAK_y0E?f?O5~C->2-s-Rt2>FF
z6m!hX>baK@5C|n)2i&7G?pT`Dij|Pjk&iSmVsL=hnugVZ$iWXi05Vq(xbSLy-Tq@E
z!L{c4m4=Fe&D0nw%&$UGo?=nQf(a3LOecpB!!m7vtgop?t*3IXQrKl#bqpBvRrrEA
zL_x!E#)y`J;O;mKp`xt+08vTI26c{Or~p$#z9N>xFAFp5rC42W_?LjOlk`3y9aMMf
z6IdIzFa1R>OEMtZ1~w8X=|M&W&<}861{-I|;fBnhtgF)j1!FZXoMBhYp%oUy;qC}K
z46Yt$6^nY}DsxwGMYo~mY~adrDs%DQ4MWh;3Kqw^6}aG7MHsY>O9p{nq4?o_z=m5r?!saTVb
zCO#oWV8Ie-^az?{Xf(gLCo0Z(fR$Cy=2ry|;u{?`y^uy$HiRfR@^KZkJmDYi4mUy=
zJak2ClG`t5EE=Qk1?s1hm}IyOwJovB6%4xAtEz^}_XBpiC8nnXD>0_yeh)H<&GQ^!yvBw|;sfBJ-Sw$i@0pVpVL6A)
z1_y>@DR|1jQLsZW@~=m1aYL1vuP^XMRgDJBs!BO~`Ijo%Hx;l6Yk)uM6@y7;@e&&Y}I4W6DwChvi55*s#r2VP_|)Urm(qO=zyv%
zm9JBNY%Ia?FE|a#*bX^86EI%4M5;^ZgcM*HFliGn^grBCrtP1mRyW-wm#9rqGtd!;
zD6s)iYiwH>&Z5_c#BZgl_@prT%*Jti@
z33$)~@`GqRN)P+ES_?Y|{h5N)z#0$xm~32FR*W|kH|SkVKya)8%k~PGA0kqe4oeT1
z<;MyJHO|efHY?gR?I8;cZDob3{vkoOU)098qZfGij@!{JE)*dPyNyK>iK5Azvt~wD
zE)=~?U}(cY-@z1M1Jd*RgV@5cUA^4vbxzZinPA)lDm}Fq9}6ASuu^eX1xiE0F&Y&=
z4Uq?f*{Gr89ZiVLrp(KMvjDua8hMqok8dz#l`lT|iki7cMlWnsvw735V+gr*S4?VV
zj}fMf-t!we>Ea<&+lroIzhX6J>v5$skk@q`swFmmP=To1h5Leqmnxzhpfp(0t@DJuMF?54qbLQgZ0ZMI&j4CjW4K{fbr*#r-$zkmL1uj)h@+PlmtwD2WkFE`;lxZR^#ClOIn6O(H_Es5Evm_q
z-4IGqY_D;prK6kFpo`32j$wg~{{X1M*fne-uez?6F^vqM8}Q;Y;zGSZCUUSU2s|lEFOuPYeq8+Ub0NDUK+rwTD^(za|*Fp0C08rgqk>B?!(a5B+
zubtdfcDcPnf>BOxwJ|AdX5tJWc0>ql9wlV2#W^7|fkqc+yMeA!)j5V0R&TnL4idn>
znB>6>H^0OdDEOCJ6T$gmqY3~+%FS6ZS5I-4)=Y=oaI?-#L}irm0}2KOy!BncRI$Z6
zad0VJt=Djo8#}IGXD5#9H7cvH)-U^*7iH8R+XTo#XLJ`n2EHbJEg)Z=z4a^4LqPFc
zyO=~Lb<^uPnJ1Q4V&so;acpgt1jKo~2QYM7-2TXNF6@EZQBGG4$opc+Q9(zV-9R#`
zhvqkZ^DipPyEvz&JC|)2F}9IIl_DBkCB#)s@U_EL307+nwL#UT+sACIDuL!+MyrEy
zuIW4_?vpXGyjNVz0jqHe_iySRD@|?>u&_*W`BV?MwReh*D|IM6#x0Kwqfu%Bq5(!n
z#AL6OQGW=>UGpe8&I+_EVEjN^hXw<NOb`JVd^o%8Xg|Fe5D?3M%RvnIj}R4bhI~MOG#z2+aLQ2HqoSNp>H{_uqT8dy8`+09-?@F7?|Ol(`9+Jc
z*)mzWm@?5BoC=R!m+h4F(>M=3grFuA(&M*K);*4Qg08$(jIbnN5}-n61*;(-(?8W)U>z
zQ*=;3t+{D}G#Ak>R0O`o_%kVp=D#o8TaWu94wV*~{OE)^{^Rzei-N7Oc(zeYCb7h>
z*Dr}x&=0w#IL&4#qwy^kUTUxE8&Zo5jK=!7Q2ygPF~~k5sv*m>t8_p-zViP7FU+XV
zCUlR$V2^VjHd%qEBQhn;JirA;y~ez4p`ihLL8zP{vc<#`6@QLcIXq0T2gH0gY>ExM
z5)HD<7Hmz59Y=f$tjFgDj3)N3kQ6Y~J3Sf!={C_b2k5c1@T^5$Nx
zmccn0B4P*6F(n2wfr+z>bH~70g~jTZpE+mOY>ifJ@|$byfcWyvs7%vQPx7
zZq%(Go0Um-B}XG&5Q<7~U%sGN8(6JgW|mo21GSn|w678r;Wl>DHqi_o@nc#)P#8)M
z7ti{DU)p5u_!wBpYAn<44i*V6rZ;(r+Nd-}P0)6kTBW4!J|{13;sz=1AmNzAb+VoaDV`(eCGFw}vaGWS
zRZcalfA27@;c}CTmgBfCRsjWjiyPTrQ5)lU)I|jBWxEbZ6BlE5)TC{*hpBQjXqo)h
zqEJ*d0~avbO_ybOBb0tG8SIz9G&V@Rl9b!><%%9tU3w-IXB9J*7B1u4Tjx+k_hssH
z;%!+-vVIejH&K>Wg6gK}GJc~Fyj7Z*QG+gGc7ajUOYR6rGOLF$o|sEw54I
zQ3#anqxO{MWzxX+ZhliClf4qa$a4Xp6=K%;=($i4E)of1{WKA>0O5lYo)w=_3z>nC
zaxmD2+0;bH?&ZqM-6H&%gbl*+Z??5H@|YRqyk8z5FiJqu*44>?9q{}R4vQV|i0`1gI+ns3TqJInc)@_^@%|))%rIqR-89(Vyr7aa{|zLBAfd$Hf#HdST({$Eet_-z8P%nJ!)VTO}Lr=08=gC
z^#EzaqP3Z914p?_>KquiEG31+ORax0gA}{mMPR|iq;givg+~UCV_Uj4jYU;nDk7<1
z-Y+wKj%YLfpj2EedtNw~#B;#9SiW-o#%~Qc4sXreu|+3d4HW`En!l7PlQQc60C_@|
z6k=sz0jgtplZf>;YooXyrkN@_$hahVZg)))GMe=*Ygf|%yf`s2$Bb4bAtX@sh;Zpu
zn)`xT%PG5>b-0iVa9_XTUfB)RNN|mEKZtxVIfm1fsMM7x&Ab6vgs{M>Z_k(u
zrD9Xd(j>`ma3WVM(P~i0P`Wn*C7n+~;MNzR7)f@Ovz9NoX{Zp6>sCUvENyjo`S^_8
z)v@;m+nPHv@ESd^#-$%cwjc^Rw@?o5`9JC^c()O@Vxw>~VR(WN4GlmVE#>@w>~@O7
zT8R6a=smD-yzb9es1BCWTl7WdSsU=5%xml!`(m9y
z5f>gdFA*DfdYD?>101ra(4UA9H+Q%y(-BwN0x!Dr8?LpOHOe}9i4C%xtc2F)C=Ki!vZwjqxeX4N~jUDXidyJ7u!&
z9yEk5jnh>*l^DgRA22*xvMSv+Z*k*LSul@2AsdOoFr$L@T%hd*QK&fMj3+VTry{EU
zCQmL1n2u{_xP17mMZzYlhvHaO6?$B?s3!*zL2~DynWDDsbtZnB$DK7Li8>(R$LQSd*WGaag&xaRI{6&
zg0Qhzrt>b{)5N$cj~>z{09H8~&t1aC)KR7wU@C{c64
zTv;kMF_td(=AipFqMOx&rU0ZGGS@5Jex;K6MggoGvoDpMueqj3Enu{KK^Y5>^&Jcr
zDjkvlI3l_zi0g!LhQrKbuuB?%3{e4Wp*t4{!N*Su7QYoOh7C{!uXk1EF5@9IKrKzK4T9$POnwC}^g>xy>4*vkt
zD>SBx)JMvKAj&ogpQyogn0{i6);c3~9v8N6IeB0K!RB73)_f(IKwDEoFV0Vx3XHOn
zZI9Qs~+8W??|fi<%c%<{;gx5}sldDkzs!L9`Y*aAI9JbK(x$
z=a{#4{Y9l^?o&+GU{&n4IF*V}q})IYh!`E%Lk91I%&Td=+{xBsf%7avpb?Op)Luxq
zOKM&M08_|(j?XYSdKc8Tg9NBWGw!3fvxUM=E>SX!xw6%gkxR3!1`NW>S+FSl%JPK4
zGvK^I)fXtMA{9ky{!!r<*KtHGt+{$&sF-TK>
zVmiUPnQ)QxAIX4lF|c-2M52u}*AIzc54(kgywyVk$!L1LgBh#%_QR*Yn8rC2Kn@GK
z#r|c{GVes&vfwtlKu&BT>NS>6JB@u0)X0%4asmF}aq#j%Kn-t;EgCZ~y@-jM-QzbP
z8mdo;x?a-^u;o0fa~BR{7cJ~)*x;^WHhHoOlL7M&cHp3M=JODmur^D0>OL+|Yugyx
ztH3|^5On9EnoDN1yO?89&Q_UN->68+;5&lEkSHg&Ae+fAz(?+smaNx_TXxbSZw|&)
zgSfFXLGU}8VK>~jzcS4h+$~;c{$}+7mE6agpfM^SiW&mhIzFaW_nBUz;Mhl_6hJCd
z)SwOU6#}&e0>E*^By0i@#}Hg50N*zV$$Pl;AIw_nGKB^5#hBb9FZhcwYBo5$N>8w@ntou5%&_u;-U}~R8oVB1PS>_#
zqpqWsqWoq!gXZGA#MVc|Z$A>#M*5WUC#hz#xYwqoE;kU$+_pHqff%Kgt-%cJyun~(
zgn&!SgUi(9p`qNg78ES6nL!=-M=+-ky@4w+8wqE@$KA$QGfKm=Uq5h#rl7huKT$$<
zTN1J;E~ftDlm!nLGho-#RhwmEJeBB04qiE;Go1r+Q4sKV^WgT!l?8Z|5x
z8G)m?3&rYU053=L3IG{i^AIa9Oj4y2bJL7Ws&y6
zEa=J$%np)Z@H&8tiHqZx<#-@*#VR$sY8J1R*t$>V2aTqIu4Wvly#8fW*{s0uy-OMb
zRlgYA+RR@AscL{Viy|$e^upq0fnk_QUWExj!s$X=mvH@#+$j@PmNrs8<$g2F0LfiQ
zD<0biMSfwoHTyRbG&{6Rx!Pt9UssQ1jhvUd1`Xx9@XWx(Xw<^W&exR)azPJgguQr*m!|}52JTXL@h^6{oT>8v
z08+L?XYlj$2`J-;9-$DWiYD@s58>n7MAhYL;Gj}JNt
z!5=_$q6W(O)XP)qJ!JSGbxY17bN(lZYn+RyAPS}~QeNQoGNJx3UeGJ3B9i?^u8i{z
z%C8GkU`J?-DVO<+kTwblbZNt0rC62KKY)c+4QPM4r3#NviB15j1|hDEoIwavcKMl!
zdtnhRJ$QzY4YmPt{X@d-uQ943f@w*G1xhgZnqn7?WvNQrt5}1ZU9)u#Y0<=VA#c0H
z8b$e=!(Yq{e+HoN%fS$o0p;ApAyc^2*r)~`qXu-mT5hl6V+uxeu`*XTD*57G4VNJM
zh6xoE*Dbug%l6VNpH1h~4JaFA@qM=f8+G6=!k~4qrlZF(XAp93VzKmRFiU7BQlx*0
zL2%)B2PQXq_KI5vNW=3Est>5cqCCu$K&39_eXpSdS|^FU1{q(uz}uo)zQqfECLVP^
zDyUn8MemYAQ?-rk?hS?0{{SEw#5@mz0`J=g4yNq)Q^-?C#T*B?C$?}E2o)Luzqo9n
zl!n`?01fMg0=DdM%TT|>4Vgnn0}`OZvyj@-
zX*UT3LRBy_q13M6@Hi!90gtE!)?VU^F5m*pmn=uhrt6q!C>kuh-g{gS9mSDnT8lGTTYf|9t;tG}bl$hS$+m>aHxRi|-@|G=LS3Hr#*%TT~
z%*m6rL8>?k$IH3EEgHzUyY15q
z<{G4{fI_ZWEq4TFIF40{x1q#gmj??N=vmCgl_{;%FbV(&u`AyY(Je03;F(S!R{Ykb
ze%o+R`hn7=)D**9JNlPvp@26mCOaNsPHS`?$$KT`j7m0hDg$Nr7~w5N)6}JTB=W2w
z2qM)TNA8^gmSox)U1NK%jU@j@JQ56C(Ig2F7Bqk~T}t
z_IPs>Ry!qakb9MK&PHyRZnyOu3L`E#OEc~n)wgefk`m40yoFcm;#5Ipek{
z558fP3@*qEg83i`UFI~fk64#*M46!bl-o-If6`vSxH!aH3^xFU=sS|&h?Jv%vY%;;~PPft2zO-%<~2wD#ry)N!O?&XU{UW
z-vkrOXPC|2-0CyFHOzqeg-RpTMQnK@YU&n2xKfl#n5zB48@{dr2Vl)h&Q>iESc!S$
z#wsng(|)y^e4+S7l=$3L7ti
ziak#|25r}{e&GrRi_AmyKpTOvONUe9+|a-E0OwlD?l7n*mqDwA#w&sb#@NVf?p*}!
zBcdaUIbLO3_gj~jq;YXEUykA8a^LrQNSzvK!FPh#u?jxoHOFyoH7A30dZj5bNXsdV+hfvY83rd@j2k!i7i3DTs(6fZI=)H2%i9nKcSKkD6;TI;Sf&^Sey&pg4a=G$#rw8mCNwu?W+w)cb$4pH$)Z>^$<5=9Lu&VvO9A5jzD8F_38*kR}kn_znBMYLu3LU;z}NFP{&&z_9E4y
z&fbBzNukrT!NKtfyQ36Bj22U_rRFinZ0hxAP>Yo|L07zkd`tyms9j8A`h>R5OXdcY
zXt}A-nr?JF%!)eWYVyF~M?%3KnMw#)072*{PGM$=pC$|%xmXwuG8@zm*Gv>RLxgcT
z$tIR*dyeH2xmW4bP4CnRamw8gX}!na$nh2fStSN5t|?dGViQ$F-vqSbb7a3IbpS3h
zHa#mNyvBq8gX%ou0VFDpA{9|r?qBN?stVW_U+FOzM^?J-DB2f6sj^geG>!O;h*uEz
z&FXSdMq	!*J;MjxFP8c@OF|XSo#Lbiq{QFX=}yxN5+q@I23&JmOa?9xG9RWaa}3
z-ObFG3V>w3AToy7tq|&0&_i0fgQ&xnp{=YYzv5C>$2gZOsT^-J2Gv~%ik-PTnO0hH
zEcdO{p;X=s_dBrcf#mFXhd81c6(>XhqNXsC$#9~iBDAj&!YjFBgNc?Ro1r4z3P%|%
zcsZ0_c(bdicIMvRPjvB9Yd(scuAi7JRxMAij7_eEx$YqhGx=Y{pz<3ac-vz+Qsv>R
zQDR|?md9)sPGuWvXCb(05}z}#5t|_AFixKtkM`FBDma@~&ZaU({o|(oZM9e9PV`ri`|oVg>FHpRNAkHf@1a>Rny{@x(Fof~v@HAgK12
z8;HKO+xQ1)$J2X7t%6l3*pNVNcX?-?=wzhHHBJEOf!G0p;
z*kA(!-rhVskiuB@BNgK)?b^D9sZ5wG*b?W6&hgvb8Fd!0}e;<58
zoTi3=Ip~TD72uz5@f!vmtrj#7ar=w|4#;)pT{!Ub=J_E;P?v3TO5xZwy}_ihc$7TU
zM_3f%6$&c`*aCrmQkk^pQ+0f?TpHX8Sa6KUvBML4DGU!$clE%VkvPT}AxfCF7W$cz
zgL!u~WSLqYF-k@#z>ncxmn=S@+i+!%p$Y3T+BqgFT}z~G;!+SM&MQzdO0EJwd~p)w4VB;KZ{x!zr+T<$mTuFWgpyA$q8>A-p(>qVP<=sitGVFBHea31jYAAy~%)KvxbJ$*{R;pxVU|
zcK9)Z7?xQ-V>y@RUx|o0c;uKZGi3x`v<>M^#9(hAFGraO^xB
z>IyVld#~ySX^LIrk`Aa!v48;7a)z!EW}TujVgB456ItRoXaa#vR
z+%eN5PdY#BOK7N;D~oo^=Grx{%y*(zZstjWe7
z>Cna?5esH-<_jTPA$IfB0-j(Oq#?X6hTslY<|9TFqK^z-B7F|PY7qXl$
z@f#~^U9sA&zg}PpZ11>eO>@MsxejNj<&Px}`U$gFGwOmJER3)cznD-`VZy+mWk(I9
z;`jdmfe`Ssc#oX7P;FYwEkR4mul1>a}?0L8?0a3d~6WjONytH0t>
zQszaM;y9a>p@dk>wuYFsQIwF?OjSeTENmUb2boydNEqn~0E>7$Vo?{BE_F~CU*-JA
z6xh_VQ(9nnu;a|K-D1_yoc+bcVa&8L2ED;#8-t+n`<04U3us;Xnl!{!hSlmWq#DMv
z`iP^8cL3S$5ENe(HX^ti^p`w9ABWVe1d)&m;LrB~r-O0kME+&$6a&S~GUpI(qk>yl
z&36slN-#@xh_n;{ZDOm813wa!fFc*##5hHPE9I$Rvt
zRIMPIviZy^fnKgVCS2=2pjxFFERWYzu(Aj9ZV9BFMVC!=9~FJe2_p9W35O@d8BilU
zsDkkMnWj+_vN01^3=jh!Z8HW@c7;oFm8^dLvgSG@BXxr*aNkv-VjTDW+WWae1!DW;%U+y_n
zD_lB-jBt;MyUY#w`;t`=Me~@2T}u-D5bqjp>|p!^zxMZYT#3nvsvMv%OVD=^C?Itg
z%Tsd!Xad4}>H{q8N#Qo95i=HXYF=6*Ju>a!#b`GaCC3uU#x)ATtVD_j)U%aO5F$}>
zrZw(d$HFMI2KOPBz!1oslnZA2tx6KkFh_6`K=CjLqc;+pGTJhcD(N}sna@6<8J4AV
z;9hf3s4B@OO>SS5Ds#SBS%|S-8#%%wuTzlLmf|hi)b5yNE|j1YoKeIn56Oy;x4w-x
z0Zv5|Vd^cf8iU^E(8!I3;BY)j(l(cnp@7EJmD>p*VvQxaZYE2+8jx*`vnx7`4)K3b
zq@3I|%`8ER_?l+8w0w6co1r^&JNbu!iiGkptCyCBZ_0aytB|Tac>e%WnIxqD093Ak
zphfDTRC*Db0ck;~
z>5J6BVD^;fnk!cEj%M?KAr0p0A?3!On9FRKrLUNnIJ`vFcFrCqb#OM!NZkkyTa*`q
zE0-js5X#YrQnpH+#icMO7|Rba-={z5OlMg9jINCOGa;+teb)hRdcjK{Uh)H@00sOzF9F%4EjKgl`cJqE$r!e82e>-4H~p
z%g6ee@HBn$o^Ij_WRTO5`j!SunSA1K%&E+^B@Sl5`-JH;P{n|NuQJkr^pue&0C$*5
zg@#e#8-p+|UP~S+GYzMO3lMS-pXM}R>|S3UV&cx6pLPELYGdGAPLCcVBB*e-;Tp&@
z=He8>)BzP&vu(=bLi5YqBrRhR6egPOHky>3Lrax0KK`Y#)Zjq$je|&xs4K|?pxiM{
zcOAQ1h?pK|XaR5rCSgEIr*T~hxzyezpwu?(dM1=p9cp(vgem?+rwq$>Sdl=ZE0$8m
z0i^T$m$ido=eQ++DK4Kf#pX&pN*gm6DSlkdAaKqwmA0M(R%ZzqW5v*$(kUBFnNMDKm8`>QsK>#_hv}v
zk(kZgdr*urvc3&q2u>C7LN_nO6}V+W?UrCnOs%8J+`5fba6pU(&ropz8gnpiF&3p7
zmN2}r?Wnq>DJbd?!IU6qfYm^dRn=AeOAIqP#9gAc;Mg&e&tf;pDvv~2c+fp$_m9L;
z0-V)u!~IGLYAX_vHylVlGAYEds{BCQ%q)SJEp!!6>wp!LvrzV=3ur9{{X*iM75qY
zz$0PTOgT9U&Y@Kj^w%=-dSWO!fNZQ|aS?r%F2kyfhZbpv8L3gL%sV1xfZ*rrMNRBZQw0uk&jmrqs
zu&;o>Q40m-97dmp%c*NElv25-4a3X93x(~PnbP!2x?FPYP0=eUa2}8Fp-f;7Wn*TW
zE#H}35$ZCZgEE!0j=qpZDl8@;cUbN^%N82_OBy+gUR^+}Amrlw%G9!Y*#$X4%;7OC
z2)qVfpjlwbd_r-Wi-hr(JTmdrXk7t|`ihaus)K>8xsDvTgSleXhGbQXyO=;{o11}k
z8xA>^`j0}#$kpwYpd70lyzQ(?!DgF^t@)w=m`z)VTRc4X3^^WRDMc6DL3KCb6HG75
z7hJ>)FlCh0pQ%{{WrlH$l8_q3BwW-}N8yiG&r=@&J|%kLOH*{3ePR>&MXTfMC_GWAesU0Ul}AlH`-6P0v%Uk}=Huc0)m7oHV+9Fi0JC@EW%EC8nO1Ul
z#A?AkYg&lAF67R=XnBSL$rym=E}E!;Oe~*ch+xxduE$wh3ykK=EtK^w*IWfrO<{q^
zxSO)4%Oo|H>Q-VZCC1nS4OCTBx!VaX!#2jp2_Bg4q!`4Cqydy3qnzAL3C3kCuJ41N~bw4{F4ioj`ACr?1T`vBSd5{
zwcR6ciNj=0tBFje-dw@}tr=7Jg|y(`iRUf5iMjAU?{GmGGw{Gj`~%z$ar-5jR_1|V
zx(3;JMWhz!c$F4|4oqC#F^?l?OWii0xxboinUqz#h+E0zGx?TF`4MG-dVNe|L&R&y
zL`@Tce-SH};yD)nWrkb03p{mGBRNM?N?T=S+&~n>s?FD!2GOjdS70SLbH#HG34(Gy
zdWgjwJm(LZh)|_s-u|iwCE}h!_bl>g;VYJ-d+k7C%R_*n1s3bXSzg~VhWc(PT@%PS`i+xITCR!0g;|FXF#!-d?phxLm|{w26y^_I
zQ82tao^TPECQ#l{7B9?U$%?i<~9mVVJGFOhE{19L4WO>yir6kCdQRr7uw|
zs64TjKHWfletUqda3)!HOL-n1Nn{>OokVcl8A8-rrX{p{2(4yTXtJWHg@Me=8ox69
z{&3GX?U(Tt4oi;8;W1sM9AG<}TtG2bw-S(+_bzmw%w})&hnTF`%lJBl0iY6J?YZ6U
zd5?v3GYL_lI`W{!Y$^)5l6ihu2V>xcnhLp8)etLG?^6=N-x9f8ItXDjHHl``3ez&PrA5t-*zx^=H(%stf7qiPl9Cl;mR=rFjXJ
z0e*RBq3;#XsF<9^#GAO9?jiBi;bYzY+7PCDU#b&-+FvjWanUO&Mqk&Ug+A-oN>8mn
z{;(@IW9!pAH%4@k^v9Ew*b1Lr7k&~?K7PVcIqEB81VMcl2uo@yjyg{?6}fDBcqV#f
zA#0(Lo*Hse>Bw7oK&;Yf2mg)iPZTCNG(y0D?h>FeA93J(slpDg?@jklCQg8wZiAEx
zU%yhdIb$H%@2>b*wEhp@)4Z7mceObqx*QmBK7D=5ZO}&NZ`=#9_pP01uhKy&pP1h8
zI)W)QowSg1?6gx*{wdT!5bd6C)VRE^iG&a!+k!}NlF;TqfX~RGH<`VWA5*3Ed#;Y0
zf9s9=vuqr`H_T><3roC#v8c;zoA=K#-6Cwxv1
z_T&@NNHIwb`~}h>tz{2Bq!{xTO`-PG2AZ4VpR}7Agybeur<;>phk5;&e1p
zOc&++rYXl|OuEZZ?GUaH)M8Eb`>i#!ULl2ezA0f9f3#_OPIrcwkd||*
zRBal4_vM_!56xmdxkAWIPxH|mGL?Tvdact}cR8m^=U>>3n$-p*jT^W}X3r_fj1PIW
z2pAOP+R*9sR`=iN0F=t?%|M-wheGw^%IL`h_vFtIThN$DW~*A;DbaIlA1wEOBW1rn
z(Jy?76$UDD^{zz3p?5c+Ls_^)rkMUoOQ~2*bN-KNN$MB8+Sr5M9yRtE+*&y@ZDT%x
zq~j?m+~ZUBdG5VtZMYbvcsS(OC`u!3>`{Z(5!Fu_dE%>kM>zBZ9mQs%e0v3eWHEC8UT`erJD;NZQg|rSP%*1(_cP!HkDJnUe^l
zFW@5pWV_`TXuSiN^=kRAmks()EGkCrVpYM)jiw-_+Cp@**G-!EU#T1bg)H?-=#
zCi0#A+`8bG^U=$k#d;&F0yA5(pFN4+{DPl#amEm$ei{ksHjk{ex0V45ZGTg34PJSw
zNK;m{7o#UDd
zotFpi
z>Wk6X0qwib-kk`F;U4L<-~3)~_JaG5{;dbk&lZiBcFE$@nUgi9D4cyAri8(Y6826{M
zU5}*7t2_8ZxD^nF3qP}{wrgKwT0$D0d5b~JDSU&Z;(veq)P$y4mt+zhIhS9)9NJZt
zCDzOBh{$@L;>6p3&AnNt8TDj@*=@XE!wMM`r7cWaRRNVu;Vr;9z`Lf$cdmS*iC>)G
zp%oIB%8!cXhE-L&UTWh%vsvL@4Z2$;l1O*p|IE$99?efl14Q*%mW0v>>^JRE;^Po&
z(&26FZ-?rLRcV*0jun9o*Zy*3>SkD%K=1b4V;j
zo}=u9E3Q3p2!^5q7Ag`{N;_!^#|nNqoLIzqooH=k!`m*Wz}x7-o6_(WfyWOMg)u6b
zNn5I|K)65$MVHSm5EWx#GcJW}3c`)RpSbFJp#C1Ry+w8(eQd8ihlMAjmt9VDmQsDUB9}r+ihrZt6%{K!mswq^F|oB7YPLMm@{x^nJeo}OD+Y%mo;E7q
z^3B(x2gP%_jBVBp0LT%av8agSZWOl!1?~T3cQ5YA`KU|k$=SZn8Yj-J-~*&o?=`2J
zWl4E|-0+PUkAcl!*|gN&OC%mS3COf-a9YY)JJu|DLTdOa;Y&g|G%(}fzM031uZx%7
zdYVpT2CFDy9%-jrd%LWu(8Z>;)l+_hM1y|d#E~OTGHN(&ndR^eGKp
z@qoqTdFN=kg_Ncq$%$9(OYil%sBWn>-RU<6l-xr4*f8(*0>C+m7h`_lgjjA{&*}o-
z?SS;xBW*~rZCR1(m31fkE$B5G)tpq^KAmI8pEt%ui&mR6lB_X*pN?#^J#&0PhI*1YxfB+Nms7fMj{Y*?f}h!_-9JdD5ZgTU
zNmN|^gPow#V9nvYcG2*I{0gUOQssK5|+1D&yW8EfeO^
zxW&YI@@I*``P|biUqZ+A@Zh%WKYR}c^q#z!GM=Eh>eK1ef5pn5THu7bob_ueU%2%|
zv~_?Sk0-YVCm4!)Yxk10xL(g`K3_-yPI`EUN(!GTa5bt+-OBJ0q{axbI1ydSM1liX
zz6t0mFi_D+LWQUGZpaP%O&ZpOmIUDl+w9{O`GXaK1iL$L*GDw|V=36b5hh7v#7*qK
zd@mNUA_GI)Eooe>XxrdxFsZSKhl2z{<%UmdGw4-Vk;AH;x&L$grZT$mwmJ0Bf`~rW
zqns13Dd>`XH+}=Tyc;zfc{M9o);*&{%&LrmRrP=WDe70W`2EWR_Dfv97@A8s(c9}t^}wkuHA*75sv
z>CB>_WuCn9pu4=It|bdW!briBGH}!-7nN+&%AS?n^D?lq_T^UasI&p5bqpM>$eN7k
z@5S1;ZW{08yd1+X-&rVUHrFl|$Qd=$ROhWU#a9QB6*n%udmu~fYeP-T>T!l{+!=|M
z6;$1W#zDz%
z#_Btw*GD*4Tf>O8Gs_NLRF+JIv}*`JB_qP<%@>dfaLBD2TJ0pvH5uI0V46T_b(Mbd
z%~wsnDS!BIIkQvIW$AeIE151Ye!5FxPOBT%g=i$Rko;;sA*gA<-S`^zX44#H>NLJx
zOU^PJ)3VFIJ;=|3$oAfac#yM-Qk(}ROGzq!_>M*s4biL)gtlbYhv_jT-RrlWMnw2z
zx=8E^kyDt4qBDtcM6+7^vZAbxLId|I{}Ue*AUcluYy@mwB`Z;hv`+Aw_*rgwQ{l@7n0B4pj2(T!5SYDQdvl`iCS2gXA)wwm
zP9VIV&}v%;QwgK|$Fx4}xynJpfc^*AcW)a+b9au6b9?<Q=9_FYmg=4uhPcZ>W41xImT2W_GmuEs?=2F!@~YJ(4J^nR>?}w
z+cTndt>(s6*BSSs-WqQl)1Ko*ZB!b;(lv>!Mi>>zup+JR&j?)HJ+=#%@vR?1wB%9X{Z)BG?4BdI{V_^FSAG|*Yde@x;M1ruIHJixM!XjXL{3Y8B352
zujZhAk?p@-kB_{9H}U(6_2zQ;lsJ(r6Cp0uphLMR`Yqig^w7=odJADqD3tU1!W#5L
zjeMTACdIm|-yy65-^jGE$o5ycR&kJ5!Gzx&Rj$+dih
zR!YmBU#@%S|LIWe;SBw0d5O=Mr5&{?&Rl*_HyYtY?_&-4jsY+s)ZmSf9dY$s%Q6)#
zCHPD>skzsE9rs?}8wc+4ng|ykj-Z?Rj`!-mK}7>B3ib?j-JWlCnyC=AHP=MS>X4YlXnS4ll7bY2GNam=60Eg
z1jNP_*~DFeF}Pkv=~DhwNH4MO3iDWkSKv->tms-Cl}YhGB}rYj(hXA$?MHoYI2C
zibfZ2(=)@!SpxNzmI~wLRAL^8CR&+Tl!B+
zH&L(lW|;;&27df9rGd(Q^GL8FijB->F(|%}Zh1}1TfxgfdwS0bg9&btFAc3x6nHNI
zz3n0W4@HF;cZDM}X{cSq?80}N*oCnUCm#J)(m?AZ`7Z3A?7FJBE3B4ONP`wOT`MLt
zQ!VUE_CfFRmbqm9zMba&Z$mullkOXOCo|EU|eg>*4z2HbyMx
ziTD}z(3WnrHO_D1sQ@1*4xUAASguwdzuQK-lLAj+JNUReF;wX={hNPg`FMDEczAet
kczAetczAetczAetczAetczAetc>X{94}WLYNC3D20NiVA`2YX_

literal 113365
zcmV)AK*YZviwFQvH9cGa1LV8~R99WtFZicJIt1yG?v|EDM5H?vQ5r!)S}93EKmid&
zQIPKLln^8&B?VDZx`g{2-}n1w)}3|loi+2#S~F|@Yq_X%_Sx|~zj*dO-nnNlp#I>t
zhpUT~qK&WBZ5vM;em56;)PLniP*6}@RFnz%Dj_cR_p6}rKR*i!3W*7d2@5j`i3ke`
ziik>xiU~3a3W*DeiKCbV|4V!EKlnWG@U*!PUAgOE=i*`K@;|%|F1vH*f6~t1!3Z+_
z=hy#_e4gCAp+<;LgAd;lYN#vgqEHx2C=|LeE;jro&FhgHe8Kck)X>L;k3a71Ncb7g
zP2I!;g(9{={-Nc_f4GN2F`+b+uj>1}T>IwhPjGOKvo+?lp-DuBR};yqb1k&`GOj`U
z7s<3Cl5DRRFJ2u?$zfGdo;KR@7)_8|jABz%R+ej$->)-slsLE}
zF~8?8vpx8^&wg39tgOpV?m1g12@?zQ8A3~K#Q6Jd)))>s$v?ll#LlFQeBOy%SBOMD
z>h>>!Nl4+7@L3Rz`R_#;Ht33fFOq7*4E=kNXE+`%at-NEg8$z)^v1~Y{(a%w!j}}4
zlzMk16A}`he!6X3VylwM@F8E;Z=YcJy_Qmj)B|f}F@
zW%h03XplumM=$nT
zHw}m|x}C_D{L$Y=exFT4_srn7$B)k{K0cBkjf&z17nb5<-W2DH6aW?RT@0F}yifd|_{XmxMY@?f-TX1UXB^g;+QU->&wY4?>
z-uCu(&iGq)c45o?>5Emffjs-ZcW&Rd{+e&HGTS8oYcvl>NlB^uw(#0?O=3>evT4P9k)BRfSFh;e!oNI_89z99Q`BJux3jYoS-o1%dvENF@{Q4~
ztgK)yS~@$i*C#95Quxfazny)0#({g`g3`Cz2Y_$k@VqRmra;M{Gyg_Am+9(|md7}6
zO)DG(#l%!%X~YEY^ijY2@FCAx+O{2cd}6|;H5Bh?!}ZrlF16@gl8xz_$L@=rHmhF?
z!d`J0+p2&u@3-R7zpBccOChp7)SEHe!(abhM#850RFSoY0EAMX3S|OL4ynnFz
zb>Zjm2M4>0{N|i+>Hhxy!ZU3-(pz0|l>Ges$k1TsY%_j5A?HY%2YkAQ&U>({Rfu!tSp2c_lZqDx{P_wSRzGoc$L
zodjWBcOk2x=bZ@!^m3>28E@%RAY*xKBzVBYZ1fH;9Nwvznm
z)|MNeilN+Sj*ikuj?PHY-!E-c`jUnOLR$nePY`uoO%3IX
zU^;29>(FCJIAvYk#GYE4ws1U+#P*k0JoHn3EHC2=2nfIgw?hgh_wHs%BRy|Xr60;~J`cNb$`
zN6O2Wytjo@sii#chWrSaZ^eQYqZYoc9x|oV(SaF?pKOKB5=vreX^8|!;Gc&}rL1ak
zk&qO4etv$)luS=gw-?*AwHKIGjr8y|1{a#wu-qAV#rNxL0nc?Ut+2APGHQ`KF+PV|
zZ#sPd8pS%_j?YG0r5s&mqT)9vA!ol%A}1%8buF2@$2fwB{^P#BR*98Xkl+3qsp9Fu
zBEPV(E?;4y`LpleZz#R^kZ-?np+`D%ViJUC2^~qKUt?|jp
zs2dqlPsYc`Su)<%)%m^Yy2-{=Q&Y3BTGVv0!eNw*ot?e4qeCTzQQmTKxJ~}-Pzg!Z
zkwObIV`F20whLdX-ERrG&d9{d*IU*bA!zVxtVp1*uTOs?44xq5zMwq+^CuVlBDA8S
zVrB5n5|DWOg0G#BS~rL
zXqcoYYVi#Bctrq9VWGy0Z64oGU6A!TKMN4@T)8>Fvch@&`t^2N*ZMd6ZJZn&YNGds
zlp!3%ohRQ_oSw=APlwjk)pd7wle|@pMe@J3yIXzfO-mfTwDN}=
z8KDIQ1%g6Cq!o^17h#nK4;#04cE}-|><3;wiHyY6DZ8VEtOOuA>qP;}{OOGd#=x61
z@t;3`X5r<%zTRc(PLUFPJylo*Iv{s`>QV6^iep*K@vFYc8PF{*gm9hL`bdv?&3)tw
zh$B%3wc$S%P7}fixZ~k*b*AQW^P=grudS};xd*g`69VG#+!wBWxaSb7p{Y@*bro0w
zV8bC(deTRL;xgJLL2HFZVw(3iG7vo0`*6d6zqhxS9ENse*<0y61?bRTXi=Bc%WqMO
zz#;(c^s5e!y2149cvWEc=P(X7HulQ5
zItd_q_wU`Lyq5LUL4WqB$jHcUzEvfxxc^oyzH_F*?@N)DJOrNgRHgF^30H>wjc?JR
zp_staIZjqLY4@`#U1zJ^SyOmT9__DBy#-zn6c_Kz)%#HF`BT?vvK+4~j$Uc+*C-mQ
z;o;sRHVu$bSdcXHo0~<`@es-(T>?MFF3AVVc&&|xY<#PG3mRs;!jUTQ&z{lg$!1ez
zbTrS+%)5ILZp<9da&$!Bke&*pSAFpXSE=;E0t6K;?LzJU#daYHI
zlyD7|gp%MA5veLFqCf@rhpP2qbX>nFZ3(+~~8k-W~Km4=CdfdNy6KALBw_VA(9
z+0hTim`ftq5yXt0AO%Dj>;aS2)zuZMT)qt!m`PAmQ=?FYmJRqpK|z4wlt9BPlNCI|
zA|j;Z8jQ$bgM)(=2Wlh_
z2BYH`WTR3=?6Y8mhg)sL?t8xsoW_gM=%qZHfj;$*r>C2O7=iHs=E)|@?>+unV73VV
zsKuR(6Kr`SLqgExPPdeRs+hUCuPq&;w)b~=j0!`aJtKd(yYi-#UsjgBz_jAglP7q9
zqR}6;Gi^cXKo4<2;$johV4?(tg(~q}mN<-Z
zV`5`x0iTYS-y;V?#RdiH56jaLq!Tp<6O9UnVE9;2R#FlY8%vn@>{-Igm-(mulMQ|n
zsMioj4t5SBIU%4X7KgGmtM4yG-EYE5*4eG&`em=9LE-r-Y6(!G6@o7L!0alJz5ba2
z=cfuTbtflakV~fJci(vVJJ(a;m-!vIs3#_EuI`T-S>@KdJQuL)+*=(3z{Hb~kchQI
zast%qTZ=lco?9pdXmq5qs=Qn|&#-`mmNr@`g6Iv*0Qag{rPGtgIK;N+XMY@4SH!Y5
z2_k!%P8}3rUb2TUc|=6?Ku#jMsKVjMwed(5Kt#%GEvMA{H)h|=jOEf3cT3Axhb$g{n2vTO2um{Mmt?h%$2KO%-!3YYOKi0h<~8J
z-$3rk&C;Smy=Wj6FC{Uy}g~Jy-CC%lT&)b-Qs0NhL~LT4!|jZ3U`IOB5uN~
zSNsTR0HY-JQx6RfgY?L{H~gL^Q{T~{E?+}q6ELyewdv3MQWwYFN}bNR9gbS@ecV2*PWpbFfGsOX;i9ajkGrSzAMGe8bg)6gJ-ry>Ao
z`vy!Lka(|kbfrGG!urS!LhT^I!fns8{
ztN3|nM+?yc_yuSQpKjX%PWCKCy;r7{a2cr{0C0wx*6-2J)m1?h_go7`tQ{{eZzur2
zyL?0|EdJU!Gaxs#JY3xf5)PS(dqeM7=!+nN5mmD@AfLx8@TuaHwwG5W@Yk0^hMS!I
zes79to&t4(a5^9y`VMG_=-ZW*9P^9h)t)C0{j@+uT<`-3$IF;E(6x$!z_j=~Rq=`RfnDV&v?3MFj;zFd&?x^aWZ
zu+SV29I^kOy)o`pAm_(AZ&l~PG9$K6^S%R3|H~_SOL(O1hkdC>qB;
z5Y+h62B^zT5eV4AbXnO=%(XnWv9^A_Jt+tFAM#oeqJe~gp|G?eKAspJYxPd@!otc*
zBsjAL7>}g;LTFFzWz7^$M-9t6HZ~zO9zQ;s%R_+L#ZE1D#T!&^4u2p<{0ti#n>9$i
zm8q(tZH5Yns6Ag!^~6x5A^>XI?g+GXLV9|10;?L5gxsrl`d*-EYHH@2R^VT`a>e1~
zbfa$bX0Gn#rSq{I9SneZ#XEP{8&9`2C9hoJ8Z-E)j|W)a_UGileP>y#AY&{V!CKVe
z(sL^}pqZun%$TsS`KOyyYAik7-6*CY`MX)hmWjeq>*WOod4|O$B`x!95u@SgRzS8O
zLa`Jqhs|E7L|?Qy*?Mn=hdz_)*a9nId}9^qeNMmhZ>wAXwV<*nD`rS~MEynUch&(Ghh;
zOpLPZPJioc(0MF0m=l|vIU8Sfr#~&1PuguRc$-?_{3cGY-@yi2@O?=Qh)Q56+nwe9
zt^IunC=$~u7eXtmSzF>%lSDuD;RnK^qRQZ_3<`&jc7FW+{)t@Fd6L4=(9pr|^z^h`
z_(Rv$d>bh}eJn(Z_3woaR4|xqx-0iFEowb=d?jDs{BVOFQZt9o+udE~>@dLhPrdI>
zanN~SU#bWWO3-)vc7a3-*rbkh*T&q+`qu|p1qFQd4GksUo0dp9hmclWzwj9w9{3Rl
zI|_Nh3#Q(SorJMth2^F|MZ|K4C~&pN%}-vxW&)RC3l>o_;8;{fR@VLO)Hh?Ge+R5N
zi;$3xU20WTmFZ4Hmav3G*XhxY(A`0%`kl|xD>Dt!;2gttehl0KX?1rbhm?$*{DSNN
zJlYy^&&|`bJ)Dp>sb9SJ(L&k#;t1+mAAuyJWF8~zt=Wp9_Q63Ca*iAABe{BrIjGW4hR+!!IWH!KNk8cPw$>`@#nadB}pV?3|O93Gk)RyY{M+R@R|yZT4caqL&H7!?&xPY>HHL;tGrl
z`#(HS3uJSQx&rb@D_vaW{(WIYTV8#P-SYV}QFwT`x~66u4D~l?W%R{B9dHIH-vW(5
zF^)vGT$yC6IxoJ)lMU&7%LW<{=2%SUzbC6)t@>Wc(n8j8?fPpzBwjig83aMKB*kT!
z{%D4Aa_+|u9KZl&6&0J$FRy_4ZBD$bi9#_3otAV8V`X5xK4K6P61w`F-@@&9_a617
zOE!mF^QKSERfvd)Xr5E;HNcNb6%Vk!jbSxtUr6)FDi1tbqB=Z2UU+U5*a8ZUNrMbz
zONgxBzBOQ%WYAfCPX%XwrDDlx2hDZBIvJlW)TK+8iYqGmt72?tLGM4mrpT*7hD}6g
zynu;`nh81&sIIP_tJ`c;GB>9?J3Db5$ulD6G0bnykoJBGs(;gw2(>y^q~_)(2-9EU
zIL03vAbIV+^Y+^uQVK%PD>o
zTeqP5Z-3%YiE0@cp|G*B0V~`VODjRb$QTdm{}jZk`i&d>GKr%KELtq~e7cjTBmhZIjmJkmQ?}eD-#kKWy_stnu#M`ct$1WYbRdwfsVazQp
zWmj5eZbl^1NO^K1Zocvqq6Y-6!eVD^axVo2tvSf8#YTS1da#Q~dM}m2vLT8Pyv-w1
z@}Z<8iraQ}Egcieb+eHUh35huoUp~og)r=`jiBv0P0yR#jNZOA+lKdd4_~w
zIv+(uM0BqIT?QXF5E+lrt9bGb9)*QrLEponO~i@w%^UV9o-NdHem)0y8?7FA)I;If
zZ^{4mb+veyn1tVt3~#s7Lw8<_S_<%L+}pvN`a;FTxc6cgdT$6xbw=O!@IZh?J@MUk
zcFyacT{DoyZjgV~b#-|ofdC&1khcy~@66j|U5s{ZNzPgKg+$z`Y+}B#oYd!H$Bgc-YW?<5&2KkP=A`m&3
zqq@M5Y&IM)fZHS@CMIt0>Y_^POZm>g%@6qm#)=s*YVnz`bDV-GI4!$=HGO>w2HA(9
z%u3;RoxkpmjOb(H;Iu&y)Ol~J!fclWoOtztZs0i)h*^*u^XvzOiZBfHC-#$u3KZyxxb@#l?j?piIhh<*DnXj(EEOG1_a|+FfAn=I7>u
zljRJ~Vn<(R${f@yo0Rj%<$;Qbk2BhKlIxUkXDrf@5tejva0Yj>;
ztzBphyMV-_?j;qy&hc^Mipf`?0oJCfh%a8e=R@H^@`Te<#14$Gn^MN
zunB?_5=bxdn@9FO7s$&$+3pocNKMV1eqYa-b4mNoo%fTaFml`Or_3OzTOdUZzH+5a
zbaZvy0xs`a^7ZxYv9YppaiIcNK~7H2Yg(?JUFK(MN`uHRjl^x|EF(Isj2~~+;zv^K
z?CfmjnnSv~yPM1J4Ubrs0kRO#Ns?@AY(&P#8_{zAvAbL&ek3F=9$r)=z;98Tw5b$R
z8YsYN`v;VVfi0|fC$an~N>5L(V$(n+bwK{yd};3{0p)oot!vKiXue4qyVQeUq3_K1J52~9bsnTZ
z0cz9S+#IZxP!(i|6NRUC&XeU=->SwA7Fo%I1H_xEa_!31`!I-zrIOvQSlWp7
zsVW#E!^&6@qm#392?!514vx0j*(P=EFc85}i1c^-n*X;E5b;372-GhxFZX1YiJ-WJ6PdpRPiO`8d9t{Jo=HD
zkeK)gz=lFHQQN&QU4mFhNC?sMrJtT=XJ7tWXc2vp&y-10lA7bjOU~hCZGaD?lBOC%
z9nu*~`_^@~F-JEWe}8S_@nam~*52L=vjGPgrZo%GW0#ZD(xL|BPOcxFA8gJ>kf6f_
zhF=x5(nRqdtbR3c5%JlwHm~y{oSB&^E-O>g)g=>mo(%mlkhuu1MeWb=ofcvX`ci5qiAKY(7
zJ_6Vxf?nV-xrSq+KqKqMb&lm32_(z0CL|=h5V0qktZ>9pQ&W5Q{yn0Is{>BNRpS{7
z#1*GDzSaG1!C_RuMGFcF0(#Ns!!|QB^FRLiA-h%_>*q)=IT;yQvFof{1$)e8UjhG<
z^{O5;QHo41SYo}ehn4q+u^IihLhhfyJX)`fmmHp*Y$iXZ`2PL7_4;ImI$$pV=V)VD
zXB>U>ix<@SR!t0GrL!2-n(G5(8@~#>1+YJ$-!?VA`nYrM)OX+v$8Z%>M)c
ztQCT4~_qE7T^(9mdm$VWb14sM@LCnSvk^v;8pTc
z?p=$(o#5|3I!u~kV`Ib8)6=Qt{HrHrNEA;ci2QYR6Ja{zwsv>*Q}JX)MD!Mg#9e22
z(?svFT-VZ4N#-_Cc5^GP_F7M)7QOovOtnfH4jT`T=Ji)HCL=>UBGI&(!Ph-S;OJ%*nNWyuLY|S`_wl!1v7yyl?I8
zY_q9@l+)k7MavSa2aajtGPAIK_kD+dLsox$_4#atQD3Nwt1Gfg5en|ZxWqOLsRn=j^K{nnH;b$5rs5A7&b$1t
zo?Cr@{6uhietw>rLC!y?@(&2cP&SP$(2gon5sx+=XHC}~^@pA<$S~p~>bV2ai`x@7
zsXOeEjrFsoX
zDJib1qOz#->OVL`3Xe4@X=mGYY9Qc5YXbk&3Ie@tHk~*2Y$rBSo&NbVx)DuPWD-~w
zrlzVI^ozz=ybROxK_fh>42oA02~0
zi-uK+1X~9?m}osnLsc$w{Ee*^ODJf?j7sOdvClf
zE-fVk{;*yBT3`zjHY@PYan{4VUkfu0e%wCe5$PRgPF7*Ie!V)bU{$9j#@dn+A?9)LO
z#Idki++uus?(Xfa288RiIb&f>y9+4Z+Sf;bAnSCE2=pL~&#ZC|dhtTmukwB!g!%PM
zIeNyxL+1C|nQ@iQQ?}F9?ugniYC4PZ`0@GYLV09l)LH~%U^B#E?`EM$TkA9Ndd9e=^RQ@9dzD!UoPu82Vu4m4mk;-vYd0EhldBk72uYW_vaxx
zkgE6D;Ue);sq<7NQU-URZ%|ZHVh(N23_3FrZy_h8Yy*A~3OM#8pcN;8_;mB~x`VE$
ze~pkvR6$)GceyY1t@x*3tG~J!gOndWlxqC5KjmK~Bq$jC_%Q}z?ht_w`e)20^quC~
zH$4I{w5ncSm7OMDw4!%|K_mhSHjj@}pC1IBe=t8|0RYy|)y1;^{PK%N-%+0%y1Y4Dr
zkGHqm%Xfd-lJow1*5+$4#baBRS{!1AoC3eCIW#tnM+F51_oqMvvPetQg3oc70^ZVJ
z6w=kzeVUkpM_LTYg^B;bYD_l6Oaq-DSHGLl34l{MtDo15EB4F%9$LUuhA
z{zpsC5sL4MqenLD;0|k$=RgsC5=7vDL|vwj%_1e2NkZw=7vdQojhU
zw?5pOuLeJdI@&7!^5sipTwFMthWfQ@kMi<(etj*7uCA86{z4E9)Is>0H_X5h!4qZn
zJSUNy{=Yk@ApcuIoFcz!`}7G9)Isa_PtW$;AO1Z)^{@-`JC=rqhRS52`(kHgY%JeU
zC97Iodv^k>Ciz9=IMo-P$8B3=|9L{|p%xYn&Z;!#LoMi?#e9RRzR=&_{*F%b3_GCt
ziN7h96&3QI$F-#2ki*x8xrL7C9_FK;d$x9VVZals(lJI1e~)ryo!9vMW`W6iyD6w1
zfA;6Vik;am=I{Mq&N&_AkkT3`^YkmULx
zEiI9hjg2KlM@J{2rOm5sLU#86`nUG?!!t7I`}+ImfBixzdl3MMYuMvMmdJ=V3h7MI
zqa02q$-_}uOC6*sY;5eiU$U@}qhml5r#l1k)FO8Xh#2K!K=!^6X3{O2c|O*+V4bU;{W
zsP*Va14O%l-hXe%FC;|z9*#v6P#eixixYN}PGkeJtR4?p4>B`ds&*w8TUV93m8IXlMx8iKsjS
zHbezO^kAU!^YcllsU!Q-CEBK@Xs=$q3PN8DG>MI+p$Sbg?mwwR=JOjT$S_6b(hF%L
z?=D6W?D5))H3x+#p8s_{R4>+!o}PX*CHwt*bd>F_TVh_4INzTP0kbBbH+ij_`X7Ar
zHg0$*_Wni&J_aJLBq89pQ%Xd>t7Nhjip`Pc=7e?62}oL*ed7`
z<)=@daEXbH&JK+v-R4ls1DTnwCR$qW^UsYc%KlH)^HjmiTpfE?5vdNjs|=Ia|d^S}1AG5?jGcI4|%J+6OXEXy{5O`~8o
z{Mj>dWaIh?IR^_fGa8uxi099#0*`-kc&-dyy@j52PhQ}oWrI{4y>xik1uEONa6)8B
zU&#j%ad2=TRpHFQW3FFgMZ^l07y#2Dz$5J)9rIw~K|Tv3%z~V0uxbjNY{1pGP3(3J
zzcf?>A<_mIha3vIWwCF|3jINy@&{&6?p6Gd)qS{z@&ATHzzK=h*&ko;%#76c^6~!L
zFuq3OEwotk?l*o18#exTIcpn#W7m!qL>Ic&hx~r?rW$+)k9+-d*`PBzet!Pq
z%F0d{xR}chSTY6nXEvaud+dZ$2>{HM92~fj(;Jdkt{^cDvY@Nfu2;WD-fy1(vJkvl
z8=(K_J|h_!a@KgW@#IDwhnm!_oJ|zcW$vsx@6Cm8DSSeU+jf1JOlaL0Zij0oyHj;k
zj>L4q4GjTdvai?y+>wJNq!%xS=j3oAxNH|-Xm}A}M_9|}y7mnMiN!f&PO(ftff;jHcBmEa-)tIoaw`J%zD-hsh4N;c5sWzCeF
z0;HS`#w`+5Y8!yI-M3HAIKEUmo3XL8+n${s8Q1#=fwIHFz`#I?wsuoAVh(f2md9fp
zV&rHrvw#3GkWL#%raw9e;M22{O}mG2`jy6i!$}|Ao|v(-&;P3!hG4(|!`z|!I=$<_
zt4HzPhs9sx3`pRP-L(mF;15an10UX!f+%f6{$VOYzbGxCO7zV-HAKu+K&
zDxx#U2NHn)TB?-`IJO=uv@kBcjmLidMKCx3W)2P<9-~6uoiqk+vwwo>S~HUb)ln?v
zWen@5NtlO+cQ<_$TDp4sM<+}0Mf?xlkTWlz5-%eoL@@!Ysm0MD`#*CF3t5H$pHkd-
z;g6!DFNBh)Yik>s$%D{bn{8tBSnfkZf#1KhK2;^~ScJvG_8%rgA0Q-Tetm_j8)K_p
zmC*Xn!*%*1Z`a6o?@r_GbLui5fRzK?s;I9|0mcIZMq&3XA35)E>yo3RV{9+sIn@CW
z$mrM97cX9GKIcdF0>PdNLO=O3(0-EtOB^{rR-={d2eN9yA4H%=c6KmmB)-o4K0S5g
ztTmYr*=Iz~O=GtJ86~J%RNY`tXGy41Y
z@2g&3UX4@nbXUkg-}By&(LXCD{)?x{-H1Ktif8ojG3UPc^)Ei^B*^+7l3cjR+YAb%
z#A!lUQd(LB?Tzifpsccw$kz-h$~$-N+?}bva*^L07mRg4gbL+9(u9Ru*xUQSc8{F-
z;iZ%RYtT;ArU{qqCL-2f8godfars~UwGss42!+DHCK&W!dnXX(7u-Yiua(HPs1rvl
z1bDPx(iB4&JUl$e3EEkk`t5&kzady&;YrWMe_YA`)9=5&sl!Mfkbl#4~q2_EmQB3m?-
zP7{xiYCZDiikn;U{W?hVw%J+6nwlDr#+^VpTboViS-?0**@Iyq_-_xJ2=?d|R7&sk
z(;&6hKgWAHCS~lXX1I@wi|cRY8+0rn$G?jSu^a*+jA{mkuW%geT$?ES4FUUWNA&N2
zcf-ZNbvroVqks?%OwjiE*g1LlqXmnJ4Ad_bR#_DI|vL;;K{-nbFR
zkj3$L0p!J7xUm6)K%^kFy|uM)GS8v
z7D`)R9|3Bx_JJc(m;uI8RacKJy|V~6J1jBE_@MqCf5{uJMv5MhJr~%SIuLOO3QIfH
zpA$LnT+;|1Oi)Cm4Rqf~Po`V|4a{zMNr`(%*Cj9}PEJnxDd4N~2=W}2=`J5gK@$&K}^%B!bq7-J-+MAnEAt52-)$Xi!KEKSR
z!&?OlJKo?YA$PJ?rWh{*eDk-qaN`luxhUmfjJba#oSHUi1C1oUp%8LZ_F!{X4I~|M
zlEtFV%ce8t(&1X!z*Z-%>l~~?5d22yc+GbQFSOSCO6{&&e?DPCkiBJeLi?XJbQwmps@wFCZpLIVUAG
z(AUv9rDM(7j>f}n@Otz(#N`_*D9Cr)#)Zgf^4(@mWMp{+JR#ZM>5)oN#!EUTPTXow
zJnAdDuR?FzuS#?OeX|dPoaAA__a(H)I2eBWsGvVsyX!nwfxoi?7oqbhz0a434IyrL
z#9Ok)dfJ~f{~B$ctk`BZ&duFk`tcc!kY+_p6i8cd*BK+Y!7pUjbKm*j4v8;+#=6sw
zevub-yqh49XMQP17fUk8vZ(}MJ)EUQ{TjN%*CYY68nPUHjV53s9h=TE1iauOpu({9gH#RXivg!6!%*yD#V!@D1i
zn;38jF#Gy~gU-Hqiz+Lp38B3)GPPwHEaMn#l-*h_3TDf7U0k}Xt+2iA$;e2WP-$L=
zO((_XYCD;K2^NdH308!M=gK9Y6X(}17DswlSy}t1d>^7~ze0@_t%}O9uoO%dqX~+q
z40!B*QEN^WQMeY>_?Cr5w+l1WQsZC~!?gj+*cfG2iAqU1a$Wt&b^g4+I5LODtjY4M
z#wa(j{wo#AYh%RqI`j>B3-I$|*EUD+&!5fv>%oteTBho1(Ae2wyg3WC?ik}u=S`$%
zbacVtgcAhOkv$wR=4+q&v5xmJ&Nk13o6fOiWbg%USu?r5?*LZd{uoQU)PEfVraP@@;f{T2fNTPwB8i7-sc#AyhXeb3zxH^xG!U=s~;8}B`ck!!di8oz?@h*dc-sxc_lcF+a*IB_rk1pmw
z#9V5Qmq(AL!uVF3+%H5&hoYs04A(#Vvq$yxISxfdg}VaE?N^fjxFriqK;9r;H}Dmr
zfWI#3>Dg4qwojM)0=!1if0~IB7;CclgTP*Uo0mUrO#y
zd9N^?dyf^CMn8$n$$@DkBPYjjb#yODyDpX
zHYlXf0?pOcde7nLC1dT}Pdu52nA5e{>5CA1MxLUB9}Rv-FcafG)n4dHIEz%Dg$9M)
z7ux;Du3Wid16EfmCcK>MGM3LO-)jehqi1p=BH!rKS#3yld{Yw8*2Z<}myPh9r+VyF
z?L~N5hN@smDLNI5;`=o6800W(JrlIkDx7Jzbg_(5^GwJEg%SI>G`!ruJ<#>~^}_Rt
z(nMmL&kzi|1&{~u$irWGMRQ9_SRm88Zwn{Ovold@>Qij(=GJR@=qAb>xw(G{0KcTL
zKDicoLwQLLMvxWeZ`+cHi~BS_bnmpr3%H7#NGSm4rmmI$Z^TqbSIq{}%KGhxK6&z_
z=b;9bY>I}XVNr@i)iw2}5&}6^RzW3hoR%!y@zCK)*XIM#`noW`K9X*yl7r>BUljp(
zQeas5W>rL1fk&5H+uHW%@$a=vlx?RGcZ}~JfBy-f@@AGgGSafrF1r4GHQOI~3jY8!
zO-)}BUWnFr0+f|_hNh-8r>Cd8^w{@WAfk;5yCWv~H0tg+?`)&H%wP(O?9t|^h
zjgwRU=F=SNM~|0+o(o`9IDAS+|0@{2Be?2!CX>zyK)rP^tKe0+4_QiQJuCP6R@cyG
zd}K~F0ya51tTKN41OdmpA#56mdmuqiWaByyINW;E*!E7*X_lYS_v1Cn0JUvK9-h|^
zAACiCpA<0W5GCf26lk^g3oR~g`;IXHsX8N)cKYh}XmO7Wgq)_Z%U$@i6?@3zLV;j9
zfFfz03N(#;K4a6cywDnmOzhH{z10^LRn=e8nB3VPw3imYXs4oA*Zh>GF|f33{+w|f
z_b5{R>@_=&rgVQ11v+}LL0<65L9Y>^C}{%&w-d+iTebMV_XeDN*WQ=%PC3Bg(w318
z(R}`0govEB4^j~pF-qR
zDY=Z_oK48t*-heL)X+GN3yrk22gx8Lq>w2$`OS6j*LIpE&_(<4d?q4|*Hx6HHa2xi(yOjW?h98#p(mJ3OO_iRR`0N!P&axgR*x)Naj$EJF?WnZcaM>-FG9Iy*Z_6z2ei
zspb47k=io`4$+FV%)?KW&cQ_jH)~28SwAs?s9*`0EWzmR<4s>SLRmZWozvYx(4+!d
zdMV>ehU~T@TdGgUIa
zzSJ==n}h`Q?maq(cFMXt;DEjj`O||LKVlOT6XQnzD}`1~arcJbw|x4f;Vdm7AwfVb
zs6b4r8(8U#;p80hUTcqDqlH47eu6kO(sMmCC^#ktX>zdsinYsUpx<|9Kpqoi_QXhq
zJOl%$b$ncMc3nnBM$+-&l``k_`kk6LZyH&-UPMIHKW0aQq_Dt>c%7pX*1qvD
zK8TJ~r|LNpxof~;YkqrLd_=~LmD@M|pMhBLW!^#|uS=hUHtj&*054JSnLANv?2
z&oV;el2C7Q%WsmAger6Cz6&K|%6}B4Lo#S_a(v4`p~FrEODM77+uzioMiP$Y}iG<+i&Hiaq&V1D_-6X>a->uvP*JcPm+_-wQ
z{r;`awWHddA2u!MJ}bim6%D=d2W$+|tyoTfqx*Eqxt4M``f
zBt=Qx=>Dj{6grK40UWz
z(TM(dMQoXZPK>b#KhLo4?(xrHQPMTx7>V|
zC=ZY9H`m(>lehz&9t5yTOP{nluYN5UDYD9RU+gUL_;KS#rd+>3(`jq3K-2Zd*o5OB
zE!t8@X)k0*O?Y#-%gf&#Ql|Zqto!QsH}B%0rU_`)%q2wFuAD%VYssYYngaiR(D^~o
zwD)X)%*k5WNP*dNVY{B2MV7L!{5Gy&6VkMkYs=V3DPz67!YX|-S#jEy>TW#G;KS<1x~DgsK%wO(c=|hHI!U+3;%k0Lz{%WaX)U$w4_j`M0W^*0GygO*h50V%
z{0xvX1qSw}xs&refe|&*H_MCy|oQ
z3!;+osRlE4wTXsvjN9ZY5lZAF*-0mo8HvHWo5W{l@82Fh%VhLieUAsB>)|98eqc+<
zV|e>$dx_VqGA1+>lZajlc`54WWCf2baN1xXbB{<0-!tM-mv7hAbUp2Q6#g6|X*B&K
zg3)?_!7=$SuCT^8m)yFUoA~oC#CK;U9&T{k{A;5k%iZlT%{4hral`ku`h`wXWCw$)
zp4my#5Sk$&!Vh{JTHBa;G}FGE?1|9F>h_F{OQE@IXlwuIOBH!8Yx&^@J;oQ#;GG|K
zM}Z@p!wJ>eZi2kV)>8P{Ndj(CHXG)v@bHl$VnUkklWKORYh2!C&gl0~58co9ih_{y
zZl7Pt`c1e;v|TkcObO|xP|ziNDLGI3hmpUfrP$*`3?;u_r>E#OE@@4T5R1BelfWf1
zhQa`7XQ0>_UD*H7?M{DM2L`c>!p?F(mql&)&!;0Qu>R$*4S4=`rr5k8qCjaO8^jo%hQEI}XDqvFK7yJC|O
zTQSYdro2(*GK#6pW#Ch|oj2tgL3Z(sVq-|<{W+f%(ZM%+DH<4V$@*AZnf<^}mdb@;
zn;dgGNtFDb?`BIzOh!C;2G72_{fIk(}
z2P-z>MB2DSUp0)c|}m$lS99?Z`Jcod#;XHWOGj(#xmY9o|D
z3EH;fs>V>ePq=GeZ<(xeWh9n6WFnS772Iq**$C@${i8`W4>-wbR>>PpC9s_4Sct~N
z%NwH_OY?iuu?YFEiK;s7E}NK?qRqDvr^mI6C@NYnU+@0BUvLIozp_1~+$>KP~5ft4)Q6^rVFC
zuU|&ttU;_QbS3Jw++G^_lqi_y(|#
zmiyDmsi?YOb~z0`CXFo~gpV8N=;KXRiqSu@2cG?t!iOXr?bT7qD-SB)y&~mod}ehk+c_L4DNEy+1Kv
zl&g#NVfa%5t6Duxl|)BpPegLEn&w57Z?(E{uOXLmA5GSq@LNv1NBmUBXO}S-Ptm#|
zPX9!=$7uZ6(e}TC1>Qy5L
ztJ>z7n3$Ri-2WS2ZvhnL7ygd|q6i2mNJ)2hh_r~rQnGY|(%m6QNQ?B+C9xn4OLx~I
z-QCi)ARYJJ@9+1YxpVK_I|DQ5z@GP<^E{t8?{m)4`RLQkZ}GS_v*98o^&`M5?FoC$7I=ctHV6;4$>!KW(w0KI`P{`*aFOn*B~Va$hsW|6dYEt}UIYe(I=7x<3{n@*yw(
zqzXjnRGAwfX2)`6aXxN*NEe){F!`{3zB`ezJ(A@iOpd!9E%D&FHv1)7>@T^HBrtkGFcxVQ36Iqh{QeXhxlae^)^=e*rn9%cW`z~O!NPXah6jKtL2ki&ZK8I{%K1u^
zee;{-TA=I91{WG$aPR!QYF7xX@i1T=4XVy#3J3z-8f)Y(Vqkttb!%qpoi>8M$VQRV
z@bOL1ZD;UV-~tq}4Paru$MG^UGDNR8T!00^|oiKo~$OW_)
zaqrbl&aC#e@!{;4D}ClnUicz$K3=y3UwpIs@6*9|We^0%LS5I*zl$_qH=fWB|0CiX
z0`0sdr9OO^tF|JzzxkJ|k|o}JKCYl%q8$sUic?^E#-~4!sX|!PLbMFMf`RX?qEoNW
zguO+dzho;AGvj_}2lS#-bFjQ53i2^KX9+W`^BSCiQL0iA}4s!KH@_mE{jsKaj1N*jTifI;Ygz;+P~9ue*W&
zxqMm(;XO(+qGsq>28@A#wPXa5ZU_{@Dhx23-`4iFYy=Sp@`X4<_0OJSU=V!2S%KL(
zzX#HADB+iXfBxcy=|z#oA1rZfu
zp){reS-@Y>IYpm1&NdN))s$p&ATT~9N%h!+SsJ6x2(@d%yLUh8xW7?%_kDnZUJ^0-
zQHqlIUS(S@6th9uEiUb^3l6b?-em~CABg*}cW;n?&928tC1cQ-9M#=aq!kKJz)s=o
z?it|vqro~`S58mQXI?KL1BK}1$_|>gi+@v;T^HXP?Bb}i1jP3Ld;q}Np}3#2zwgR5
z53Vtr(%lTi$3W1RnjWvUQ+T-@seNu(Of9zWt3WJzIxuO?&~JzLJfh_I_>y%TU=U#<
zBX!XzFRyXWk993z+VR0*=$mN{n`$DWcjJ8+cp+lmoAqG$h3`FK6!~fCe}+WJ{l#>d
z0Fp<_Y}5P?7V4r)iNDxrDe2r)O2hPhp;@OV-wdb51A;4a6ntvMP(V~Pp%iL
zfB0%Sxij9kP?x0o894+hS9dNtERjF?DZ+1;TnW@mewzVWrxTEph|v#tr^6!0`d4Rw
zsrgeiB0dN+8@%e1c<6X>PrL0@S^24$h~EKFDtzAq*SayU4(rhFKKV35jPBh`le=h&
zsa>`~ngtM8hECNlBZ-Dc3N~V1C=c6B*A0KZ2erIF+v2nE&(ZFc3)%0Tja%F8hi;JtxnN(DpH#Hc02|o)A
zZ_3Hp%sRe0bv*xF@|s=$YMX-eSwN%pgH;4@xS_7%UWM6RUUG6NThs5d)C`_H-4=_8e4ZHx&i0`&;p9TMzQ9rmoI@N;y`f}B2Q=AA(|?ZOyqbX
z(bmlMmly-+JDY4PK}@}n<=8rW1@sIr(a3&y#SRK-F*WTM2)aFrzmzkY%EQ$6E=>fC
z9#&tiLo{IE_{PS@kX#A;7*8qoRbQwUu%6a4Wp>=cD|NP{)%G|V#i>!b-KqCx%}e?ade#ud0?6BBz^9xmSE
z0_?dnmdi!1S7D4=s#hmSlHoA;2NH4J<90LEgJ>b{ueAdd(p1t}DsI-+pn{
z*tM18oz5F}0n}J7(%TF9<150qWDya8VCb(eeZ_4%-!3~2>$>ulMoUr_K5KxyrZdG(
z{>^ZE!GL_r)qC4uR4GN|kN{?Fvu#2{`_$^=)!Es}DKCzKJeChL312=#X;bFH0TB&oRD2*Jt*u#$HA@>ek;ngO
zmv#+^b-F?KbF#nmENhUBZER$Nq2H%UYOLt^d}~MzSoD&4KZ1A{klNW}>1%OGhz)5X
zxQH469d@JEbU^$&9@fklx|e8{>LLrApXDIFRaQD~M1pUNrRC(r@ngyiur;gxI`(H$
zXZe-s;jo!Gh2nL6e+s)$VAnXC4O$13%GEUj_fv?e
z#oz;#uCH6+s_|^Afvy;*%Q|2o2<`iN-v`oPU#&;b_Wgz+(jV_t=4+ne0m7PeZ4LZA
zpTh=Jcn3SII|q~v;02QQD}i)$0?d-B=l^doy@NT7bC)J`;vjZJ+U9j+-o?Z+2i%ZI;$8uOq*KJV7xv
zHRbj=(qHyI7Nh*SL(}!@CUaIh<;Z!EBcHA^an-nCq~-;iIOX1CKnM=@<$55-ppB(`
zHaHOyM@ww+L+|XmXTCaiBt*0flNCB?28^vJqMz;sDQ}S9{l3xUZ`CI;f6oG*VDMUw
zQitGDSPM#0y(VKLQ%PVo6UMhGG|=1o<)S3HzP^btHo@*XZYBUF2Bq*tyv_x~`SKNG
z~DXHmOyC4FoCU+||N?qO-yXCuT9J#1Qu^HHRHtGfjeuM5ENmf`eRn|OA1v@4}_Nq?Maw6YBdpz5^h
zA2qpi%tzo+vdj11{Nw$u$YO?><;O{Ean!crO8+-F^n=VZpIf^%b9-j!4`RJU;{-OX
zs49yQFp%gI*{{heD`EQ{_cy>$=dT*i{XB_V^50meH(|E6ur4?JRGM1yof*ph1^#+x
z{OKEhZU6*^Qr}0uL1s6cN{^8m#h}sEq+uVc3c*~UhX2ywzF*Ign_P@ltrO)M+S5mD
ze=}v2z>cY^%Dc}uqtgaraZpeQOkj)f0H%!`SLZq`IuuD~xeypqDUC=xiUP*p6cEqI
zib`@}SLCUvM$T$Sm2PYYbGaKHfS`~*;)0miEwRQIFKR?^dvf1B+-8{>fLL#!F~B&p#219LZWtIc;qcz&#*O6Rox!``+FzS)@_yzD7hsVks=0RcT8K
z;p9+LRzLdwJrDDr6dFD;(?+MTyXy$1eWQG$4sw}sgO`tn@!3&}7FikBbZHvyVy7chbiX{w@I7)-Q46_L7ffu(O(9k3`1b`pW9QW=FgvZ-@fE{a
zN$V#TnNG);&v3qv<(fSBknnc5l$S!_bu4PX%oLvE06-(~C-tAjN?!^e7`9NplGT@+
zvzZS8a{|;2@c=$-u4?`ey
zvng`q-$xlA+ng)Y@c1tQ?By&By({wbd;Xl8B2u8nlID_yWm#Sl_y5=C5}7WrXy?jC
z=PRc09?aKZ`yaPGW=>A3XMWvddGpr(Po|jnpJzKy+>e6n7OL-9+fw-hY=qu7;H-Gt
zv#tIhT&rC$zBz9@_zECwbsqih+4$eH<^&eHp&g*@cPrhhxg=SMGJ5paDA%7Sh)9wW
zWzPF!mocSTG^l~8O5`#cj#m+vKb)^YJ-NIrFzE_WOc#nhIdRPNy|Leg-*%@V-t`z)z8ss$=PQdLg_=4=2Z|(i&dDQ5qUDS6T!)MzFVD`$f
zOxn^nfRL#?T(GRMo)U82e2yC$2uPVqz<6T!MUlUg%5U9$J<#A%@i{orh;WjlU_-}1
zc(F=}j{(#!cG>FEy4Gy#_Qe<@>tFWgl);d28)ea
zHkSx5*cwC)8#?GcTA~!!8cYPF8GkMtRMdKqsDU|mX!}~NwOR4$xi*^fL$j?QqMP8S
z&*rO4n^Avy)1JWeTY6*HPcP|mbH!cuXoMy
zB|g3srhz+0+^lxPd_m3d)0b2u4f!0`UtFm5>ZAgnQHmyY(G2Z8UgMFA~YL
zqL+h-w)l*=?(go9`wWND-p*|1hFu(oy6sor6-lNmhehO?N(*hl2YxF@H*Hh)_xq1X
zTy_pXSTlz+;)8oT0<+awCdOuEEtyMwI%JxUouT_+!=}}*I;Wb$^(3w&
z!kAF1;l~lpa?Tdo2%_Cy4v-GSPD`?^;qM#P-A$@Gy;?<1?i#Bv!uVwJ$|3PgP*D_l
zNt9ZHWev5TK@OM944W%-b
zZ>1gxRWQx-`IRB2cajB|n{+^`XMYIgb@{6}VRDr|t8FDd^ZxVGm&6#Ts0b@;ertN{
zSBCpfcBgTyr^dzJL!5KFrLVS#Ioeo%RB#KqfkL#l^rXJ)qXI1-w}k@o+XK-r_wE{=
z5Pa(eg~-nB+5Ek`KI6JO{yUV$XEV)COGmdmQ%(g4kR!nve9Fd-?`H
z7W`+r>H4AvH5xruS~g9}1#<#zL09uOupnp?ltqJHHX42*{H4!YPeWC`jGYXYpVu!C
zCu(NaF`?3x=FmwO-LB{Gejx6nNJAIRaWFPf|83Pns5%^Mo8VufUD@gST>S9(Fpi39hro2-#SJIuIf*XK5>aVK(CAzqNk&8WVXz#FUEHwo;t|$NMb1uDuQkgw
zaevex%egR|vELKVx}`T=8XrevC+-nl!L73?r*JzMpi!JRCrbCB%(p@08f7Y#
z(y0c1jNL*W=~D4g6Q27)Os_Ijz4Z=(U80>mv~7R6w(hB?^dpC29GlK^wE
z`Q_UAI7^Quflcq*7Q0p%d4X~Un(^h)5@1wGqWT{+OZ;2Bq2zB_;ja9G_*8gMg%5pE
zUg8Y&=*n4%E9F{lN9<`0mVAxiYeBanMcW~>HoXMA-QC@vB!WN1hf>GNz7L@0m!Lwu^jSrmKD|O
z-;`FvepwUAJ7;2=FE@jNffTe@|E>B{MIra_WArtuU4Rpl^2_^UKLR4b
zf8(mn%S5D@4$U-Y{tk@;#?&B|3LQ+%I{T%ojcAFhe=|m?$r66z02?Y~zI8Km`P<&$
z;#{du*q$ojh{!u=aC-Xiz-#blKfVGbRWMLuvb$57KfJ9f@d
zp&@V&zb{oh^L3&AD9AMnTH6{lv{gr;`iprR~kFztaHO|C7><7Uc-xwbvElkdtd
z0oPN0Y3}+Cln4=fPsABi`2`347o;({R8x}kq7$1-UrrpXrX~*peZIXo49SH>PpV%X
ziI52Xaitf0P0W;rcXMm>uTbU!%_;mx+$(*Tza&_aEcXaInGWFR`QIs7NRsFSn4swI^>;vh_55GMLoW`VDp&%KEEb=(iT$8~*;8i8TJ$+fu!3
zy9G9h89E^Uhj7I2)%PxnJ^-o+<1aBxGGRf@0|~fJUp`
zY|Le42}nquOIkAfAi~#->F7)*nMvuLhXg_bRg11*hgkiMZu_3+W3pWG;wX1_cgP=g
z>g+SJ`%AbOWm0n;TIZ0{OvSaOb$y`8p{3k=`bH0xMUUvHSq*C`GBQgQ{#7|MXhb|-
zyxynUrLg$<-3A_`x#`K&<3nGk-NKdi1S6wz##fqa6dlyLHY599X&LxcU%gP^duAx$
zh^}sLU5*wT^rJH*{2pN4(&4$XCLL%x_oLxlPpciWZ1+^#2ca@#WAEV=fRlA-6Ceo!
zPxE?krVB#@g%%lkRBzbq8yH3kv~ymGG0T;;TahM$W#%sQZie2jD7w1xBWo5mPpd3O
z5c4+WV={IBh7*qNRnbpR<7(Uvb$?M|<0dxuBu7!0k;3kT&nF4@Cnyz#!W`Djn
zylubk+6hl^+%MJgNo9D0?DO~p&AT~AY7n6jV1=o$&v5Y?=M%jueY1m)Th9s$9aq~m2wtvsM(xPd*s&4EA(xWA0P}*f_;tUOo`J#X?#k)nV6Ln3OW)PS
zg2V1)Vf^kWyVX4Z_2^WK1eVE}`P^vkOU$rS)n@9AdfmEdUUrRx3<<8|;J|0QB+hm6
zijah9AN?zA?D%ge6B8{jwG8j>vppP?Bq6;0)0klj*NCl#qsDz^qt@o#PwZ@L3a#FI
z%T&RDeE@hX6!+=F*xr=cUGo?^x+$UQF1Rb}4wM4Gz2nKRM56|c&pGUN>Ls321-dz}
zC^kWhBNBuDxKPN?|r+K;pEF#G~UBC>boGv(2n1+}6)=zP@^-q=hFjC`&e!P)nRrepx5I|Lh
zzf_X^fx2{?8`k=0DENX|3T=HJyZg$}|8faK;?6k0R*vO3O;CyRtL(>=sP0f8{4f;y
zqOKoaDbQ0>PgLL&m`R3$;l=!@EYHkG5+kk}FJ)%XJ
zBTsw*V6>>CZd?_CD({a%zv9+N_Y{O6p>D#F#IH$;r3#G!cJ
z++}xRKOuO1W`i#3R-j(lV8d15@g7h>bDKJW9`BA7)yU-_lN
zTqzb_F8PK_A(3D2n%`-Xxwim1M&zim{vS76SxHBeTX70OMQ!1)pJTy2ioe674VnN!
zg!Tv~6Ntl^h|I+uQH8>gSw4+(*kTa$IUTK|VXFijsm%OcMaP0QAhZ6%j=d=ykUGa9e{n
zT#8Sy)rxXYMXeW7ZEcn<9@7_B`z&
zzW$?-E5oFc`7zFi`aB@kSx0@%E~#>iUbtK@EJ%VqK+O(5QN^IzTnP0NcRC7~=SiGpG?Q?|7^5Px^*;P+(pZ<>QSpdTrtO;%dJiQb=2OqLmt
z^%-#%+buOF3}=YMr=$!lHn?1G#;eFFDM7#VMFD1;te4P-n
zi;i3qMZD=u$Z*kTG2%DFl9q0-nE}jq-)rN?=r%0uz{3R&Zh!a)-p$37_)BQ>pIS^g
z0MO=Y?Z~L_sonOcw;K1$o~(9<6F}H?#y5D7We={quMr
zEH`YazrR?x-YYJbQ&HKjnK3-`w51-}9!eb`Yhev)@n-nK{Q19P=jP3D0(-9pG@JQG
z0ULx+E#r1QxYfMt+105mV5FK_yv^`pq!_8oWX|387fIAiKFV~~+Qq$Mfi`ynj|a5l
zdDO7&xmQ?#tO0B|tX<$FUjQB$c!zV)6HHk?gz6S
z4KCZFL{$CDR4FNr$1Rj%kq!|N@yY|%Q;P^^rPThGLf(@0c1d$41|GkiPqb-*P@~8<
z(r9P7ImsCiX=CGGDeVwM%+D9vl~NZg=bbs8r*X;5S{g-KT$9naw;u)*iFTi`7#YSP
ztJ>iDTkj@+5VA4r)t2-P-_L%D9;>!eqx8L6(~kzs>vMlrjUs*b
z5>NuGP=+#cRW|aCfGANez~j=&1^mC7Dg*MRo)*4;JBNW!Lek$|sJ4aI_JHNK*M3+R
z$Vdgp_;wn6z*n%PMH!RKd`n08-tsp)zn+vSz34?U1aI>^i4`i^m&itKF`S;Qn8J%Z
z?yiBdT)hw<3Q9oqH*623^#L=Q!fzEPYq&yDYa3h2R(;+4T+r_O6O6@a|08^ITX()r
z1%1C3eoPJnYhiq-=z1XDW>cOG-KQ-
zfQjVtj_eA>+i(=(R6IchSxx)`5jOnF#bV)9w*J0Q=dnK+5|nI!+5qIpOENY#anQBD
zPL=6O?#1C2?R_jA!eYh_>DX`l9a>{=X&I@rvk}u(zAa>*S&MAS6Mo9wNUN#73jJvUuegXMn<41)#qfa3BrJFs$d#S8EsSzX0kD!l9RlLnf$ykse9*cKo+$
zE?e2?3o7w+cmkUdLR($^wW_MBRwNIf+U0{|1VASGgF{fCj*gQtS;Wc!zM2Ca?drc7
zGxgEPx2aJwg|rbr;`E{^VIMy(HR{eEyN8F5);q=8_@oIYj1<<`EcuoeIMULswLK;@
zirKvZzeCnBm$d0O2z@>Ooo(q8O4y1<2m%_#6Forz=G3I!A9b=wV;|NcBxT_8u$TuV
zD?L5EzB6UP#6S=xft;cuy`W$s^6rM?V--_VdH|^b7P#r8N8Yy@6BE;}rXuFnbJl$Q
zH!V0yj8j>&uja1k`EW)&-mKr_!XXz0IeV6!hzERlr`K(V8YfvH$r>%kr#DpqVJPG+
z=yOxJYw0rzPxxqfk8&Ngk`oMM?tAbF4VG@R8#MCQ{YB)9xb3Om-QVxJjtO-FqRjbh=q<9)FVpKU
z>2UhHu4JAszYSY%GzemH0L?;Xf92BK$QmSIvpUIAcj?!jRkNd4r%R8@V(_NXEgpR{
z#RBcZ^|a4SVI7nJJcW8}i2=6JH-_-gwl>HYmbcDokD&_M
zB#No%1u7pJob9~^I$eY|BfZ9GJvVUfsWsq%fJ}W+^C@y{vQRB$?~UEpSOECe*Vmhg
z`Yf^rUg9CO*7}B-M*GUjA=jZ}KWo_5o3Sd`=HsP5YJRS2k({9~=^MeIfG1lcss8mk
zpK19mQDzLb6O*uUf)8wG>tD@RI}IujDVfsyTxIJsSkF(o@`J9~^^`kqitQF^k;jWT
zkKFkz()@=7ixh}oSea5=1*!N?8(H4Ub-BWR9XFJmj@nTm9SWE6t
zBg+Ud<^)k$w93f)q9_Lc%{sK*%ScN4C(-dTG0M2#*-gO#drVBl^`%Hu^sz;qMP7!@Jri*&T
z6=eCo@ju<@tNM29p2%qcI{Q0>BfCSv!!z1c2ch~+3q+HUqhv4F*Y|SI50*Ka9(8|2
zZD@eTF>yE!NW)C{<_AYG$q-b
zNsW)PAA+$?c5?ISPM!Lq6}~|*;N|)N{4KtVX!$2C)0rs-y9@`zOKmg!>-lYGV=fLC
zCi3J7CbB&fY>uN_zXAU7$@==Q4;bZyu}u#}!)e@k}l5@H-Z@XN|67=+ZjN
z-QkFte_4T%GHB=bIay)`%puYkT8Q%DNQ6hkgb;UR69{Sxx@y`E~TiF&|@
zBX3ozv6*3HWDEgt;dDYpA_aFWH<4qaMPl*q?J<L_#Dsxq5TgtXw?kqzpa}M+ijk23;2ZLk9^~PD
z3v~`m+}tt9TB!UF=f9IlQZkld&|NtnN&WOmnAcj;JDWsBkAoSNR_XhPCZ81#F+rQe
zz7O?{Vb6Ebr<|JMIm@0OE+R?97hUOl+6&cAHlkO9m*JzIwA&0duQ+l&98S&x%UgK?
z7#2t7n6oo4SMv#)+umfu=4qqb{(mbuB|2Z2c}GNCIgK$*sGg|C`e4>=$$Ymq=!#)XS1kqi7dwX&*ys*f$_E2(F!ulr_m8La7puyW!ZHWkJdr&9FtoN?Lwv(@GiF
z81&6Qq#Ozw@JM9EDD~fwEWb6s(BR+P(GMmHPeimXl*BeG@}m=AJM&lr*8<_Y7x
z%(Jr*aFq90xfJ2(-%Zn;Gok5cp6}kx*;I-m!p__LP6Fr?62}^m;vJfp{eWa~n)`yh$VuYi
zYQroVk5V+*`>3Ih%*1P|+6o+3bQ}=8os}YQID@!zj`AK9yX?2P3&iUzGl0>4#xxrQ
z4SCp>OgAzs2>Fsf+!|fifr}G&Eq{a`wp(pG7p!E%zh{4yoVz5VJxXolkp(ag}(i)`1w{rGW!a4A|MmFM~n13vt
z8J6;Ydyjh-Z;uC!K5}Ww6?IGRp0}+%E6?ENSAP&G$cJ|Znn}xmo~6gp4)TKvX)86*
z#^PKo|_C$Zz9qP1U_WvZIy>Y@*JaJE-K!jQ$D$P`>xu
zc`W}giL;-(azA3LETHIv5nEa)sHmUXTAz}QDI`K7c!i_1Sn4XKu=X8j!WhkyXn1m^b>)1@gC
z6iUAWPK9Y`#xL(5ZXGX|3?;rDk>g&s8^EOZGZO&$(|^1C?NaBrY>|+ha$=Mp7ZbJd
z7!b7LG2TZ3E#7V{%7wJZI`*6Wa^!u{XiuJCVqxXLjokB`uV#~=I#rt@zwj=m$X+GK
zc61Xk86WBQo7D$kMvJ07!GP!SaV6>-=}4nDUuM2GeQWhu_>IO3rOX$*yMTVmr(-e4
zCxCA5^pdss?u~v*=I;0$xr5U3HK0v`-H?578dNBSEZ0A3I_9;VGrO*v0
z9vXod)D6HteX-p4IHwAZuss)@NZ*Nc((
z_`kedQmkxWK|viWkGK;BT?6s?xdW=L%@AWtsxV$^#%I2_I5N?K0NuD8k^5X`#3I=0
zt6wcS)6$t-JE#2o1!3g}gu&r_4f0%?{H_zMnPsZP`QO7wz(*?(PpInu?ZDV_IrMwv
zqZs%|OKbXebD{v?<{zZITEnnv<$?I4KkNKAqbGzq(C=R_1gKFn
z8vsg|ea@7SS281Spd8K+u@EMYb^`2&WlV5y)%;SKfpe8Hn)CfX#Pvl7#`xJ6%vY}v
z0Zq;T)u{9EetCAVTv|*eL(eD`j5S)6Q2CO5A#C|~tZ8=y5prvR9xwyr^-fy!BhVk`}pax4BH*-3#~B+p^SZjw^L+5iH3R3x<@{u|Pg_W=1r-)o
z0RT<$Tr
znTWV++pDq9oXvJ?~FastrG-rZfoz6S!djp
ze%ekNgQaV}UiDcjGZgDjEoML9tAVd)4z1UWBJKysd#{p~4s5%gH6U^UfteUXono0?DJ~gMD^2OXj%3E=
z?XvziD(IWNWDceawRXrYas2j+$<|Xv2F}>k*RYOiD;OdhV2RiXtGRm#=ZXBt7t~j)
zPbOkF!@~TlEVi%;!S;V$;d`=4oTJ%+0RI%a?VDmdV>3|i?t%qu-Vw4%Z4U!bpdnXl
zCq>0){)aGn!3#^%oYCi|@ZjDgK*MpR(v6K=$T$tXdo3om89*)X3&{-OkSAD}vDAs2
z3Hh$j#hv2v)q*TjnIK+zo>y>4K$?Rf1*W4F-HqE!N+g3;Z0yOTT3q#B=vnQq0A`~N*G>hl
z0g0jE3?*5$pMb?B$OI(;EuhR8vL&Ffu#A>x?OHm&BE?h@MIB7+$nZ#wl3~%Sw8Ko~
z(yNxMEwmi#bfp((qU9%JfTTB~vqP=vv7M^|md%(o7qA!&B|D+JPC;(hQqrWoK)MA@}6*;s9
zyj|v|2#y{YPAvAu6EhW#L-W2e0hFHAXv;olSBL99qg#MNQsp{;G`e)d06YQg-e?(H=1Y>0%@YRm-7n1Q&d*&LRWkqJyj^VPBpdmAG&@GyWON%BgnsR4
zH#6)?AB@<^Mcf;onNP$5_MJ_$Uz#2IV+Wd)w^^-11*DuyczB!-gu@2eR0zC<;VnJ-
z`2LKAbPDfz!=d1ffR$91`49<@Ey?cgZvF=~_&A{SH}iqmUG#grm;U(iw3|RB8Mk1R
zmQG1_HBhr8X0z)vRoS)M?ct}~5L-n{ed(Zsr--4q`E;s+bzf`r$pD}-xaVdS(R&cn
zE4jn^WpD8Od4)Ne^U3P_XSEI^K)mbt?7YNQQ`4#T;(PM>Gs$dO1|to~AV7tInzK}|
z&iop&mEXn8PljU!3oT)9XtanwZE9fAz@qe&ya;{gHSWkglse$~CL`p3`WbpzyPzlw
z{$scNA{1B089gO|Y>X-DP0v*||2$gR-5j1J_D=-`y04pZE*&gpPU0#4wzCxhHn#k`
zJLQ?ax;OJT{{oR)1U8;bmnZ!Nkug!D;E6L2gr2q$;D(lbXE(4GdTC}cFu;4icfH*)
z!Xy!HOV6VkqM%(ZExpnl|;k*JSRs{HfOex5SOyr^hu*a;4?(x
zmm2gutNrPcx#pC`rLbRzq}-PeEr`}$<}Wde*{&%n=mx!J2#
zQz%`T2{$wlhfL6uhSkgp6B8z4HxLJmQ(C)Z{h9I{!R7!Hj`Ba)x
zUa%=Fxv3(JLNFARSy!Y{3~qF_611D|9+7yo71lSKzdmzzYeg3F*DvKpOX?jp
zaoZz#A+56%MZxWo|C97GAi%YihDSB=lZY!f#ejx@HT_hz&5Z6{D!9QJhQ{kFdbcg{
zsiko@3<9IqEA`(r8%}@6Z#NI=qw-59b7+pOv6(^a7G$O3Dx}nY4*scZg#?qNs5>ZgKwT
zI8GndOAPxugu%j5D_(f96DwzI2CDKx)?mhMUG?JyIdE?CuLwg70u%Z@k=*OMH0@EI49qG?Glg{o3mcns9E&@5_dKQ9FJ`E3F>$fQf
zTK}#81SeH$*n#31mHPYpkw@zMB!3t8gOCm8w9zY?MYZ>}4&0+6_5F`^^Qp{4tmh=o
z+y2W?Jn@*ZW|?YROaU7ZX<{^!5366gvX!t=t3Ie41c`xpn$fUn!RfKthN@eBtQxbd
z={0E<*>E^?KflJ{wjp{!3ivN5dygijoXl|4U}a@xcW2PdZ6+7M<^|W)JVuGW?t8wPZKoEB3DNu
zg$*2xTar+PLCcBDm87vVnZ1K1kECMn%1Yk*kJsrxUkvOUV9Gl4nwq^|YUGXdI=7t5
z%0_O6E-D5oLcH}Rc??+D6*+6uh0WqKb8sn4H}bWN0LxJrvGes8LJYtu`q7egxsMu{%RVp>51fKv^?3^6hGAIixrqz$|buecv^`4s+{6&p@gZftu{+WJ;9{#H8>PG9gAOArA
zh`Dv+^lsmaD@oa91rHC8bJQda71b|C6P>U!vcrWzw3b4(ny8?DbD?3vH}67c=7-`n7&fL1}*RzYJrNvR8ca?YUk5J
zE^gc8*P`C(&6${CIPbpc@+wi3J)-IL?If~z!#w+84|KidwMno7Z`{I|H;(U
zF}udFbRn9|*Yk$=v%^A%J2?u@;H8LrQAV)Ms@Fi${0~KrZ=|Q9
zx|@8Q`uXmBGFfY=sB^@H8a|p{zIGH*)54+%PMs`!k}3=@s|E*lOoUD9NCX`r&#aXk
zO-f>hk#8P+^EK#5?uC4bgq$JQ*B1H4qKs4?T#Rz0w%cAku!r;$oY&sXB(6)Ze(g@6
z=`N6+{E*z=!vjZiYtA>g-B+lh(k$(Nt7Jtx;`kW}WoUZ8Op6prl5shJO$>i9C2=no
zgV=KMZG|pxc(J=3C%LDBVe@W4*yNb{sH^umO_S>5gW${dFP675*l}p6bMeaQmR1D_
zYF)f6dTKNKwIBo~5%v*h_(Ytfy?
zLAw9DSDGl_nKW&h1*y`QZAuHI2GT~?9c}cO-ONuBkvg=6rhz;Wb;b~Q_AfrloQ14!zg!ENX3Xn+E1Eb*xV4e&VxyYTvYU#G;K8O9lXRHv;hs{{%=29arU9<5pJrQFqzX(vYQ1$C7JZgkI`@AkCtnRb$;6=|&+&a2LHyS9
zL4vEeaGxITB-pc*O3tn3H%@}0SW{Isa1nG#u^Ls+Jpne4sB=387HKcrb0(`oyepa1^xNq!6
z#te9~8zQ)p@0?Bi@G?Yn9CH&@MFT0fmyB&IRdOrc`7VROEJtt{3f=F*x$g~zoR3M&
znr%7E_Zv~Ct#Kr<=~;g2@^L3Q@H!G}vH6?#2$7AprJ&e(Pf_*AJ#irAmW)SNq`g?8
z3YC}g?(c{ATp>Fyr99aRN4KF!^f!Z-`kS7EN&dVaksjs#R(UysPQJ2ztB^#`RyT!sUhx105=@X{X(nGgC%^D>TGoqhTlmzvM3HVvVh|{?U0{>GD7EHrN16w
z5eOed2DB@7rZyhwCM@PUjZ1Do2YkWLxZD_eqno>q<|OV0
zb3psIpJfu`<4#IwJ>m>bWIHJt#xW=ZPi^HUk8bC@OJ!kY-CIsgG97arQ)FP{;>w)r
zv_yMZktV>CM8RA2__>}^@q0AF(vpZN%AY^9R;g)R=~?rZ_>+_LH_k5bL|kd_?JP}H
zBo&rNbKtP4Uc)Cv*iIKQr%n?Wii!?pT>QPzU8+53m4KUT(W(Z>fb^a)nYGMrR>=v6
zw!Xzpk#%xz?wk#OkxK@psLErmZjw06{_FJ#uj|`^CpISPn|;7VBMlFsvMpzPW{~5v
zvD44xD`uHgZx3@)J?+~rk5HtfiVSZyhPt!(Pi1G@LA)m#&(}(CUeDRUrui;gNqtr%
z3EtTqX~|oDb1+w|!krJ>X;FlwQPk$i@!c6z-l7WtT_;JbCT6D;QWagTO;67x?f2b6
zY7nY<0@`s`JvJ}XiU^!nL)eq}tDaSvQVIEJ|M}IdE+@Hf{~B{HDdTCrCW8&FMa12m
zrGNG`+kNIj5S-?+`wV*(-P<>Tj~uFg(o#jZs0P@E`iY<;cGVNqoDapJm*1jz2TfZW
zQay~SbJ(5ro7K?E7J@!8XD%4x&Q=e^4&G92jaoh4mTAo$4bN1u?FAEar6CsQ*Wzvr
zr~!1#C}IBr+ZA#ChbA>(M~Xh9S{O03Z_{a32Ji6#m{ykM8fc&fMI-O-4}*$4Z@s&Ga_?EQwy+b6-3-_Qr@<=|i12
zhKBOS#MuIux~im*ZaQjAsz$cxP~0}&8)aS{05(FjcE}kjdV2|1(^>A$hs0-zdc0Gr
z`QW(yva0Z{5S?7VMEb!(aR9mj5q^T-e7pCv&YMj#Kwn{NYA_qO^1+r9g>9A|5D;Ks
z(&n@*9u^T{F_tUl{PyjO3f-#X33;lI&_-KQStsilxtl4qErD#qn^VbjPF}3NMb9cP
zX+EK+8QgG@(FpMHl?_ul4$GXvdkHji$rVVKWRR5Ey~LqRX)jjeu8*B$kyFQh2YeMMZj6qTr7YD
z9&WgS(iYb(T47-l>35jHH8osG$;q%SZTK92KP3~emmF%F&Va2|N5lc`9vi~}VuZr+
z_ZN`!KR$Y&ZT{*Clo+h5(C=}WqdhaYZu7Ieza47qPvel(OiC?zyuKYG{QLJJi=J9T
zgBRkuj_Ul-ZhyN)wUek>6iF^s(~%}t%WSC2YWZYGS0#r0=h6~X-dqC|EU~y4Oi_i?
z-d-!7zXh_AdE!v{HQa}5>40hDHl-!`nVy8a#C7k&l@
zdyK=#*tpi`!j^!`gd8%S{{rD>r2#%{d5jY0`IkK;v66jpQ2CE+BiYLJZDSJYH`Tg|
zk}4ZsvsDPRw3VNGegFKi9Gn~dAo)RrxBk_GpV>5P#k<6)J+c1sCdX7>agWxA-fv?#
zUwm{p>D`wlNJ?YfsbJS6VX6A%$UC)?-;
zvC9ehJ-t4W2l?-N)XAoUp3aecA!8|~B00P=S*-l4*@?=}-yaSV6&3Zx`;2jSrYifO
z(S7%44^`k@@Itebp{NRs-c9pGyKOC0)WFL|4_+B6y-)QZC7@@>V>gPsT6tlxrP^ViS`AwII%M)%<)?fNN6~B7e0Un`l4{fFQ}>sc
zkf9L1!v{{Myk7)WT|M8$${?%uO(K$~NoV`-5~Jo(TBpGE$QaI_RcypGrhy!=dxqyQ
zoZb?IkV~n=q?}H;JIu>l3t~*D%EctmyYyiL$0(zrb%O%pejRWR%LX1emi$_aKJ$}5
z->jzsFM#d1q@y^yNu@je>&d1n_Fvf|I(rKHqm6cTEA3Ezo3^j+RP$!=nh-oZ
zybL~@CyeU3F>naTmX23n3?)VQ`S};dib_hpnsMjz9sQQ*lz2$Mc2bcm
zX-SuxN6;iKAjwbg@kl7+Y|ggq2tI6N5*3)Xe{vvyO-PC!Lqf;Y>eM+QLfUpbtygzx
zF+s~Bbje&$P=HHFm}<2k+a@y3SHQe1e!YTV#r^c>m$z#!{;KhmV%j&S8`G(5dc4ga
zgx*RKavJG(TrK$uKdAWmwZrHCSz8;StT*17(_#q>tSTJu_#S;0AWrk5>{V~d>A5!g
z%ko#!#huquWzob`PJxPwinjXG<>gfs6}F}oX|{W7h3~?{%ZiGMA|CcLpDA_RR%^<>
zXVxi+(WDCa^wa;bA(Gt0kcujmM$oHXY!YsIrcZtkm*X&sDWzOPdXIT{e=R=M<*ml<
zH#AT%JN$3gMc1pE(H~6OP1uCgdIn2(rb;dDu1`?W(WMre9BO^94%u;%BO_5^+YcPf
zH}?L$*sFEfPznnR%QtFqc`D-e>0O@Wmdc(4v~g18$)fMCAn3t-o!t7mqS=j$!Dn$Z
z$jjP-%C=uk2NM_#mTEY{Z{aJPes`A_%AZ(BeU&bapl3qpjX;>LPrd?V
zC#;Sl&vk{8x?~R?x?OquTC!G8d<^BIy<5Y{Jg}L+?2fg&2h=@Z2F>gTCKyMfa(dBn
zL`@A}786;%J93dA2ejd(nwr|68Mmdckg%|ow*&CK&&hY<)a>kR7f(;U52BvBeeqOd
z7af@z5+PxTEfvkLLT3_%ijtX9vxGFtcBeN2rAM-ON#Fi;TaplDXIJ^Y>Eh`@oDaNj
zlZZ8?&})5OuYh%#kekGkc~uZrOjQk9gaL`a{Jit3h`
ze@_t!G}%*dW1*8S8~u>#;;F~VGx7XrV6QP`4OKy$Bv!7dw3H4U3v~CvyW)T}z;>8)
zzSz|%Z}ocJ5xO9TA(2R2&7Abymt
zi>bNKZ;G^g=i3cN={zjPsY#YH#SDSHJAfnrt#or{#rMG;0q||&LXB!4gH|amH8st!
z{ROAssnTGBdduMJt_KEUI68TF`S>N@``(g>r;eqsPAwImm#8Gl9`(g5BMsJBRh>jd
zy0LR$t!|CJO6dvGhmd#*T0%4Z-6x4O(2-Ru%FjyKEFSU6Ap%F;x|QyyXEx2I!|IN2?oSGOgBzO01}OlzcPrt9a+;Z}{?qx}
z>wM`d=$oUpP{LQQBI9DiOG`^<5}~qhKAj)m&UV4>-7tCK@oM`eHNTGI+7z9lB3!E{
z3aLN}nW3ej{=4+^wUrmbh1_tp1{b>ns;&JBW*bT@TVzDY+Ri8P>`%t&Bl703R-($s
z%%U|!8t^s5W2m@BxH@l;a~InzpjU@b`63prx3S?E!CScqt!|0p@~vFc#vQFjs)6T;
ze9s4~rdZ(;Zc}SISrOzWCU{-5?J?vf6(nNhc6f<#^WEvsajUC&vueHTe+Ogwi$3Yd
z>G+IqP#Uq1$UQ$tvRr&1mnwScQjyfJB>rebq#!FdI89{b;+ekdFCRaMdQhmlUvIOz
z+HD%Bh$1#PmBIp4Bd_Z908?Y!I~SOw>wS~5-a8FS5i?b}9vJqyOssCLSBL%6c4NPl
zSm-CMVU*BZRO#=Gz8dRl8u^_=YkIu??EmJ>jn}8hZ-%lXLZwW4Wg~=6b2)`?w&sBTNI4I87|!&1(izlkuus
z-GjR&&3cim=xFOyBe3BP77$dkAj^Jq3_Hx~RDqb+Hv>KiXIqVNB|*@plKbhz1~
z%GA;Gsk2m4U$9vl63x@$oukQ%qzMBp%WOONBnaAMq$n4hRJWLFk0Q
zKa}r)_yG-%pxlJp5v!!y*1dI4jni0c8xs1PZVKh+q|J6;g~uglZ+4%pN)G!#Ift(f
zQBG4pm_Av9nJnTCZ#%Qj-0weAg}|tq^uPtXih?DE&PZ)}J2|=Dgj{Mm7i)x%E4j-M
za2Ct{`TnZwhC$hF?Hyl}kgskDlWuvDyG-lP@>HWDwJgND>%y4{f}r|(0!sZ_TDv|y
z6aqp9gwUE!^acmEQ0XEZ70Ld~?|le*)!cAz>DAOGP5gfg*;I&|BUW@KSSYEZn0O;y
z-&orm92``EW?u`cqWfN@HM&yy7rE1CK^qj1hocGCl9)SQF_G2z%h9cDGGZL{zkP}+
zDHGNJrQ(%og#?};=@BKZy;K?$tb!u>J&KKgshz>w$?Y*?Qbms@l0Dt4p>sL4qz6%E
z9SXO@$$5NoCGTZf)a%yyc<)I9H`j!9d^|2fI*0>vL5y6!w`&uxyLKoFg4H0z{}w)-
z6YlYrY8F1xBI7wp#xwaDsYbg}cRr8aMdR08{I-baM>y>I`CUpsK+WE6@#0}*3-7n+
zGYjJLj8#^Uj6Ybwqad@JRqP)~)+}AWehU=X=BCfL9@aXCW+&wlU$rh5`wm}a!!@0X
zBB`0_3aGq{rNP-2g2fLAl^>-2p><~B*9Q@I`kLEKGP6UYs=|RT7?nBA3oPJ)XdSL2
z=JNz8Mny$p;>>aiVO-xjwOsuy2v>>pI#o2=e(CLa_8JHjZy8w|QpODgdCC5!gGCIZ
zu2IA=%RiT`?x!61Pf;rkKmn4ud_6+C5-S+gyp%*A#{MT&FX)%P4dc-k%Jr=s#YCZU
zWVEy;=NLacHxK7%)@sHgzzg}ajHljrH>LJ9TJp%fh7FM1K+5e|^+V;v2W~i;h1Y!W
z49f5qdv+9Pb0KLQ8yTqRNxIEsf=Vdvw7#|0@oqv+%aqsq&+bRCCNoK%6^$y+R(E&$
zOU~6n<l-71FYPDU=URz-^%z(;d})WvjWp#kA2gTQl2N
zBewV*3B8{OZQ#podzB_SApO1raw;*EJKRr$d@hQFx-qyEzV^Iuy#3BkZQ16ynDm4p
zCqNdb+?ecFi&iy37C%D3t?FK6B>LtsNxmvszMprd)}P&SptRQ>7{k|o*ieiVo}66u
zdKE@j*D7^#sAnrDtw3^)Dt
zr37<)U~Fw3N3iRJhZ-_CI*MRdO2@z8Y135!EJ#*;Gbeq{~j5!?MS9nkxWTC{gM}meZJ4)|12*6>NCX1
zI5?lpZ%4oSt4$A@**j*hwK%Op8E->;zd!%&Mom1ZO!NYm4I_bx@_(2L~Z}lmw6XqVQxeY
z=9Bj}NM^PiX-uOqsDY1V&3;wbN(l}HXDM5=(Sa->h5lJ-o-Z~kMzXJrYBoHBl0F~7
zBp%EMP4!ixCeL6g5#7K54^
z!ff(;zl}k(hfTCjb+%@xin;nRUtXg!BPPIYj}iZU*Iy+t*_4jg&eS5%q|L3oSrO8Z
z-(&inD>J=lJ{o{WV=7V>fWB{4uXTqOAC8#5Soos+h~4aYIf1`
z!6l&A(!M%OeHDd4jDD9kF;SO)x9B;?luSv5k6WiISExd3h0i(26$+QZ{o;^Q{R8V6
z^W+o=2!*?I{PEKG=Zq>aJncQ-sf%p2)1wwP*n5KxVAVj&dq2akCRg<8ms8^_gyY?c
zr2LVLo|-p4emNnsO?$Xv3>`D&LQdiyQM)XurukQg
zA@*{*3DBy`z8A0b(bFkWz(~YIJrr&v!6b*EV@N*<%Ju(i#v7!du(g5-d5ea&UF$De
zU}2FNmast~!Pz11=eJ%y1FsX+f2X2~j(||CxLTwLmDh%Jc20`!d?&Wjwwz+je?21q
zcOn!exXFQm#P|5v3SKHc6inge(^$!H^q}TuFdY;bMEIct5K`$R97#35SU43W6&ZTs
z^5kL$2E}w&PtP*F-OUnrnG~EVC>&J)vnqPv^$Xy?{vqY01oQ9oXg3GV5fA-&frFpq
zQ(5bdf2S$qC&RApFIeymZljbkC6Lg6`~Fb_Ky~Zw^TZZM3t1E=r?8*Z
zv8&?Z3Y~rijN`5is_E}=P(<957{deaKDzTEoqxmUrKm8M-Dyni6F&YetSJ7TFq!2|
z{8UvAGZQG~O-8MV-};6)m}s9lt0_x6mQ_wIZbc2#u&|(s!NZSR^}tuEDf|<8YpcE6
zNl77;;TUw8v3PDnW{}M>6aZekO)yfg0hh|gBzS8(iwtc}Bih9I;->iijd2^JvK*`S
z=YR}L*d=U-xlZF&1lN0%OvTn{3>UU<*>FJ@wf#mVu}^~sqN0S=uSC6*rtaNF7b)~>
zrQ{^t7K_G-Wa4+8*f6s_b6}GE^%Jv$1#5y-cs@};sB!1xkG?^f{vZd
zw$xVcyz@=OKA0*dOdrf6{j5OAizYtHJ={8S8u`>8f=-h^+BDCO3huW
zetFZ()J*8^K@c&6y9b-7HKEEJUaASmi&RIQ3R?;^VcNjJm2}e}j|0m%+2zh|7#Wgn
z+W+eghzyA^n=$F?Leqx$;@E$SMy1z!Ep>b>e5Ap(7J%8BZH^+z#Us1L0SYf^fo!K<&%1-ICJOR}!x?5sGcc*!sjX~ZZXf^rD8r>DIl=Q}{
zApu836K~l+Il^bbTE9UgJ(;Yba^67y>`e3WQ^M=oB>zp>3|9}ZzWsnl
zK!KNj2RT?+Hgr&mv)NKJpyRW~#w9`9|CPvaL*4g;l=Wt-T&G*&vA@90)6>QpRQiL!
z4+l#ei;Jdyc>13u(|WD&iI8RF5r)zfQGR3#R*^Hzv4OxR18n)VYD=t)CEI%fxYxr8
z=GVTLkTG!#`=v^EZA*vQ8idCiobh(yl~uU?WgJYhA1o{>17;9bT|{gOc<~2)Qlgnp
zG<#oajHg6e_rX_EQkL_->a1``er?Ya!X7n
zO6Wx4H4hD(F%i3Dd=SJ$Vmpd98;^>r2J-3p+N(;s_KWM*Xjz(x2`kY{5={%~Vg&Gp
zqoZA0Q!{CX=808$ZuQRyD%OqtO6@{9B?^gZcNt5T*=iPkug{vup_*`TN)+Oi;JY;<
zuF*~o9*wfY#Z>r|ay|7j)$B^D+56ECTIH`*i_7)&cF%YEyh1yCr&~0OwTHfj^WQvB
z5)qY{A03|c^l?gJVPMq%>O9&~n6*nYfBF7m+Wvc||1Io)ihS$q>m3_Y-*aZS1;qh2
zF4C=IBlaM|SFeov6`Sm?95W%gS?I&rDCgTBfhlUBREAe1Gi?xEuJq(M$|5}>VWkYx?Zt_Iy!St2bUrb^>?>BrS)_Ei
zWY=X2&lwE1guFT02HYA!a>mb;x_K@@7Ll_S6!5u9hXOM
zB0l56@z!AuCr66m;kGcLLLBT@Agyi}HRrz=yR
z=+o}SWCdsze=p6T1gPIVb0w-XxilBMHWVa~Ot!^(MSK-CP~#y8>)5Z1q~Lw*m*W5}
zW~8}V=rqn!&wY*t06N8Z^Ak#DVh;pR&8Y7ExjM=F1*hPvrQ{a^dU`|qf*h)c$@`0J
zd;NPe9KUXbON)Ws{x@|$=>-_@bjtg0PJ7n>JkS?vv*df7M8p{t$;(SKFl~vzCz%wZ
zkGD1F#P`LgXQRtT!()*t^^TXQ1~aM#O%`KZD(Aynz)Q_E+u%<=K9HgJ-_!s3DxK`s
zZI)0bqMFq&=vOGm+x+LCng57{*Wnu~7|1ZM>ev6@74%V?xFDna7*Ym0eK-C**((#N$x4#rpITXY^%Phj
zS}w>%8Jh#(5mwg@
z0YfE^(^V_OFCo>-OW~|546)_K<8Fluj>Uq&S$mge_wnkag!8Ht%ceCpUk-py33>`=BG7Xi7)0>U>Jj)G22sq^L?d)a{vf
zZ}xdt_WlQ_ZVzq8?TLKx^XDn$`Wj^;**X9{NI`PX|GoZn*=SqVBra~Nudm+|e-eXP
zoRpOEK3dCmN3*y@Rm*lovRFzg<@7@50P7t&+xHC0g9>ep-BF4&ztya~fcx4U?DYm~}K16x!>mT=moZaYnJ;JN*$Ec{Nm=zG1eUX!uM7y=0NSA6m
zqf$mIyPm$eS)yH`s}xIC`tu!@l5uNFvCr06kiSWd0bYgK9Bx}hc|tC>-vuGJN2Qy&
z&yg4lf{SMX8*zWqqKH4~_B4i~9Gy=ZYxn>$R{`FkVFa96O1j?C9(M|SQc}^IuAR4k
zWM&>a1MZjhqKVUO5Y@$yw5#4HSPgyt`@7|L#dnClRvrC#L3o3d<05qM^w%w4n8Si)
zRbQDF@7Bf#&o48f_HuIX_$UL9#DryTuhLndRzsUd{+lDoq$F#7LsFiu^$SNwhRs3O
z*S=Ft9*2mUnh+)?OixewbPVG0r`_TZEXZw($rJ=1m&{xbtv`*UJAr1hy~@)m)6<|V
zX+{IjK;1{=-R*7RP^W)Odd2<{#S}8ms94Vz@3SSG6n1Z}o-bSz{dY%`#r51Ei|XkfzNKe6(0=szpeOPTZ!SMu
z&`Mx@ec*J==lqQURJUQfH_&5_H&YdM)5uQgWuSUbK?-~vIh@C#O#k>tegB;&m}c)3Y#$z0sZ9S`)*iRyF85wnNs?wa4q+}XPM^Byr+u`
zuMZ-knWi}{=?kJeE6L3_@d3`~8}s}v3(eloKHDU-|JGLG99J!*<1pMUa2dXz?y8Fw
zA;FA{iwxTOqyaMCSV0Ytt{|??gj+bR5(Sq;L(P*Z^-1d@JH4{cR-;GTQX7+6iO@7`>Rxd)n`tx0rFtXXCKUl26$f{VW`|m}PWeU5^jK8ms53Ps*
zQR3=q#&tJe6c9FwUs@D}R|Oo_k62iOCYT5Mf5dj@2%kE&yg*QtGx5`q!#n*%Rz6J-
z`@ABMCA@#Fk0*8jWhbnZ>3GhX_b(nLr)dRMAm(re&DFI^K>>t|D_@K0W~;>vE*Zyl
zsNyg21`xNb8Ru>+4uU>>&KvLVgzcZctjw56#Irg4xWTHNniOYA^k--1#;e#v?D;xT
zdD-Mp6h5y=op-&_oXQ8*Mwx8B7D1`Wd^qQ3zb0AVgx_D_azvTp>#gg>M?Xk^{d-~W
zqU6!HU^uu<8hc8(&`@@FmcuLiB%Z4aG2oui>(U@I@KeN{jZ-lyzVd=RMo3u#c000Of55rz+TL*=C^wP+l$#iv&5#+G1kVg
zX}tzz;Dadd?0TmRZRt9^^_Y_9#;vfhBHIo+VGuguiqR)rOhL!>Yo1^0Dd*Kz$~LvN
zZxS17%8bdp_Bf~oh(2jZ&;L4p$g>KQmn-PIJjh7Nh?Ih&SfG4C{`PMBH5QPcN^eg#
zhKk?Z3};GOEa6l8yzHELwLSeoR8uf;s$>}HfX4ICjNgUo#^&fSjvtF`NtshqlfJek
zFCv1TaAU}tkvKE{w%Oau(qQal++9#y90-hr8*zz5IZ(@Y^g$q9R9f`u?|q+C(Neas
zF#miuCgJetGxF{&Vi@IODw}4K#ShHPjK2rXl5d!qvYu?vn(lF*m%pUhp{B%5FmPFz
zTOfO(ukebV5)f3>b)fVBlAxuuN2QSI!j?bK|t$;HKwY99vu50o2i
zyC;i@=I-yO=v~O&-w!*4qb3GC&oFb_hEzg}ji1Zhd^em;X+Bsej&bY?kgU+v#1W+p
z89>3?7=+b47w^%@LfjIKQb4@kpyMcIK_Ix5uzLZH)#!CtP{)CU0ILZz8*33I?(C9N{Vq>t!
zbyX@~kE{BOxvF0{if|Bu^u4I(+T$$?Ij6a)EzQtpD>7BxKek%_J+bv8;;icybq$J8zgVG6FeasJcl9d
zZdA8DWcvl=oabnHf%nSwnwc`-SkLf^Uda93j&^sR9OG6AMKbyV(XPQh-$8KOp}l{O
zdG$lIfiqPRc4}M%TI7;(7j~r~lVacQbayIS_mnyU0*cb)$$q^-QrvB=HX|4To&gy?
zu2`!diGICYQe2~5Fqt6z#8YvjBy6fY`2e=GL3&zn*qOtdhc4~R6T)$|2OZza({LIk
z*yw;^kqJPWorY4%Pu3WAK!z;jN5O_f2?y!~9|+IH%dNcaT6eYr6s(XfFupVn&5?3N
zd?3v8U9IzNb*iKpV9gH_Gg1pM*OT3t2GY3v;#RSndy*lHzx@;KleXgCapb=yfth48
zsNLV7jh5&C{ML{C87C6U6@I1C9TzrWnb+SCzrCv^6NZgyjtsOIv)B(ar+?}AvK4C(
zf{Won#{}!{wigeW_XWydiHafZX3yT1h!2}f<5kp1tgJ?RlDoUR2JHXpV_yhf>R;o!
zy@+A|45@sFhKl#kaaj1cu-0hnNExaBony(bBFVnJJ6-YgxfeI9UNxqkp5D(IQ>uxH
z321~Y?2ICkwAIG^(GQ|)u+1iZmT75J=yI?_8&E%f{Mg^$FZt%prGUSaAIz!rRRvq$gsUx!7s)|WL%szG&GLnPDpiNwb4MyGM}IA=I`h>cwlchrG*O%zt~MbQ1*gE{
zVBXnf@Go1nRw*`QW&EQwH0-}jW?`fzN=ix!f!&I<;5eU_lw^L8@Xr-|egDHJI-l9U
z_4d;BX~g~f`Ez*7_3ZYj%aEOuHaVJ@%Y3=h9~$ujJLtdct3x&^B_zkk#V9ju<}4^E
zNJvafw4&&D>bggUcx`4{*WIgpe^01Urpa)=J3VCkHp#+nJU@7bCLwgCR_>qOA2o}W
zKIzvM0CY`z!BX6S{gtxHw2RVVq3L@PgYtNX@73_Zix)41+_rHa9&VkUV!pGN@Z20G
z$q{z_k@fKj@UDh9Y}ZXR1vqEKZ&D#4A-GSUMn2r%uAOcS9iN`U_8$>&{2eJ9O|+5W
z2YLQ4sQWwLB7O_0Hg0=2Q7CV*Gg+*lswxZS=VZzE$chAy1{Mh}^eU479MW&A|1SMM
z;i3R%+~GsGy1MGMQ=GoCvxCOO#01Ob>v?tY8$@-9d%pKX9PgD!uZxdH?D5c%=8et+=Dm(dj
z%I^s@(#QMtV+Gn}n#ae-lJ&1gKKtFAk}oH3HCAQ>5scrQZJlhSoBjbo3)>bLvi_yp
zZ+raXQxP|25%=9Ne!Cg7_5MWtb}t@y1cZ9yebp>JG!Rd6v(?76Z`b57pK?Qznbd!D
z2P4$CAI?-6zFBPLkCCH$J(3l!Q=$72^o5F!{EB0jK<5rc@m`*5+X{`p`E
zi^T(uau#ebo`*(v08n5{l&-5knUEu1IbLO_4V~WSBswY_VMrE
zYa)lffUW3ZuvGYFqF}eGd1>%;9-PDmCM-1}A)&?La(`~njQi7Ko2Rv-va+&PXJ==*
zMxos1-5X1bO8?4{JPA`+>r_-u!RpdY!{mMQp??1UI=~=Tzql>_KC1(ZSlCOP1&6S(
zaNiX+8kz=}&E3m`h0Fynlc6{|>eods24h{mSB{pJmiZv04Vogt!XT1qtF0G%vunVC
z+aQzEW-oW9P?T8w9`CO*7j&zQqF_r}#gg(5{@&c*k1^@+q2c6=yO-I|XpC=+I7r`5
z7mjPZTk#%wYjCUY`zv4|AahNh&9KQn?r8PsD1L9YhK`?~-<534ZjJ}8He{jM2?6q+
zfF;Sw_oGxe7E01^f!hRu$P_P5XO_wQ>M`pn8K
zMv9EvJe=RY4UULFfh|m7F&wee`r5APo6dSyt2H
zBh&>GG|8%48OdYuV`Zj&@;YO5n=a;etEkFTX4|TA8`^bceWo2Mrp^=U)$ukD1u2Y
zguB@4-kbj-wDM`A_zBrpr8Ex1cO|OX1-@5@%vvRw&_>&E04xPy+CCYzxNPn{fBxKg
zZ-(P{yZ5sNU)T+d09uFaj`A$L9`{tnlGkd#)uttv>r@ojOqKlV@NJ%xK-lRmq_n%~
zIV0Wp_M{K0=4DdYCp;ry|0@XOZJ}zmfPTXpjixQ{e<7v|D0>u0+?BhVGa=tg_Hx~-
z_XV=if9l7?rLPVbpP5=`(RrjOt9GXJEGFB`YiK=jvc^
z|FS>^Z8-B8%50747_T-l0YMlr#T?+7L4YIpFZWZ-?O(MPn@wsTL{^Vidn2QxhkmD}
zrM(07f(T5pTsS%*`-6ACN9LLS(7?%BU#U%O_Ac7?al)^4JIQY$UIU#@yjdriM
zu+1$aV`9Rhqm`T9DO*wvdso7Bk4ZenbUpmVfH>crT0#2#Bmx5i0dyll01PTr{6iOu
zQ45E@I11o4I#_C4E&1ZyULFk9ahaN$rZ8*3gHwekD-oh7ym!k=Q+mu^0@wp+WOZ|d
z&!om&oRTHzlzVz{p#y-a9B_!;aAK2}6}nY}7h={US;g%g9Xg4$vOl%d#fm}6(}qH!
zO|u_FJ=Hoc=j{f2d!<`iTgw?$vp$Vv3mWd%Cp9&Q%)G-SDS+YNKoWxv02FPI`8r2S
z9n-ZIee8d|6-772XTE;@TGt-2BlaOD=jYdOjQaL8Fp`4l+8<|GYdl-Emk+l`*1mbS
z%Uq0%j6-`MjtBDzM2}${952z`8SDRylKpo@&ItOiv!}V?{uOK5idX*^+-^{(&Ew!@
ztD8-Zh&$_oYgt(tY2bn@DJm^p1GRu?5e3+Q
z^Unq?fUo2A*6Ns4;^a(fIo+ThPjm$aHrP!4TlXd4;l|o+dmN6Ek`iEKcwQb==j~w|
z`_bEnJ8xL+1r-Xm)AwLHS3JN*3lJ9uCRekjfjt8QXaMMXzeV9|R6+R(IG_Xo$s7ML
zG1Ag1yG)2cfD`@et;gh4R51Tu9z+AYr{&}coTH6M3j<}fZmeKSe}9iA?#mVPU0HW?df6cz
zfgKqi-v^4bs}WUFZZ38)K}cw*MxCW1yJ;tRP)v|^jR^(6?a%I`u19QZYwJBfAy`zX
z_0ZB5$Uh+`j{$Q~s9Oc~0_Fask1BAbKama<3rki`4ta8Nl9#Hdr$@+j6ElunIO*oh
z)b9idXo8j9-S<-A=pUU|zryx$#-W5
z-GYjZE%)*zF37`|fO>S7G$JDv5MY_Mok)WDj(w^S+VCd$%NHS)|007w!ZCop{z#)
z|5V$K^v6FE5O65Us>xJKnT0_9Uv<2;vF1{6w{&%KG?jYgZR%>{!0GI0{lB&kZf;FrRsF!N;Sm>na;4PM-f{Z2vf-k5r$gs2%AllGGf#A;nKv{H
zS*yaJwhoGw8ZrtpQfX(mz0b~`K1){RwMZJfy*Ra9bUhSX_T6IK^c@r1LS-pN{FjW0
zXw1VDkbiGeDgKkrmj6lTf&Zj)@_*8q@jvOT@}G3h#Qg7c<|X+LF7W<;4fP{F!N>pj
zk{Oqj6k2c~BPS>I@ue9cv<&lVibI2gUt>s~y=?PvU`|zHCP4qU&!<1&3HdLN^|Hzi
zmb<9X!{I^WNad!`Lk(8ry1F{p*+8I={QlU^?Q^~jM@2FY?O28U^UBhpH7%uhk
ztZ#mvw7R+)cIiRDVczoa7+nSR#OQYG%1RyeoeE0wcFb=fW^DFPWYo5)K?1*<*WM7w-D6z4z
zVQWtp6cn8P<`NMheevRjv$L}#D6bRk-mZQZ(|U4yTkGp)69uww#`EF!_xD$T(mq^l
zqZ1Hl8sGB1+&56k6Q}6#xd@4fh`79R0`xie*8ghJb3Wr>fB$7yfdB9ZQDOrF10VMq
z)2=umi9XuRRv(|9hA0(>L`P!){&E7_L1|9yRgJ32!~Id<8Xzw}fHsP5==#xUH_PpN
zX%BnY2+)>!I}ac_o?i|1_jjM~Olg#=QGA-V)vh)osp)zU0Zbv?+7R~qYFnGAHY*_z
zF1SQQ?||AQS=<5|3bq!0PEL-QwKZ(;=75zWOH9EYY-;zy%k#f9=;4BByilaYpbkqw
z50{dYk?9A26n5X$o-0+SvT<c?w>~rsG-~ZKsuVKfQeM+RBR1E6?L#UNKxsBSa&S=4LT@?CTujTpkC$ntY5z#g#C^ttkR&qgq+_tgNGKyOdZ&qE)MdvU@=#$=<3Ko1SIwTbR~GpV+XP*wu;%;aPXw>-e4ltAAv
zKZampVWm<9Tz|Bmsmug@16{gvucDs8{leE6=p0%m^<4I4uQjYP&9Wj~sdwSwghP!3
z%CBFSxVgKhegHz^XTUqyWgB2>Rb{1R6|}Lau&_{@nc(!#_h|2?<#HW1maqh%1YQ1o
zi#m-JTuKlS5SUzAGI;m`@HS)hv2Zw{762$=O*wMkc1cgL;Pr
z0UIkThM{yWt_W-j*1mWuE`NW2;Nf>+VNYYKv`W<`r>Awqea=5^6NmRh7^yAs_|W?6
zfo5eRL=SJMt4m7fG8K@vMAG_=rr_n}#Up!ZetLSEWMX0xJ|BS=>7DW&kzPPRH&@tI
zZgX>U@Y}a(;iexWgYnW^Cx`X7LYs}iFERo
zz==9UGj7N)Y*2lkN6%`c|0r5`|7Z=wo)>M?2mnR`7
zXTRE?)5>=}KR+LC#!`h(`tX6ndMua2Zn~TyRjIhrq@$Hw&@rcn$!j_Av4dR5Isg3R
zB=Pd{5*Eb*Hj^@(Chfmg(kn}kQbE2Az&dV(R=IM~GcgUp(hR7CGTn;e(9qERU$uq5
zJA9j$PI#X^+xQxeVO1y>`!8K8rt5qzc286FOeWd{eh_R@CTR*r-LJ(IDlnjDY=1yw$XOmDjzw|P{7
zKfDM=VZ2zc8p
zcXnvArt*r38C%~1Z3Y{yn~Te&Z}-JY55msg9$Qsam6eK0%!I{OdeHjjrV6@5!NIC8jv_XfPD4XK`<9Tq;Ll|*js!J5J)*9z?)E}6*YoGkafyi|#v~re
zCGL(9%+1YLHa3czmw(k-yxktBX@B^;2fKU%Mi>sH{Mz_O>By+4V9m0hWqGrr`$+y>
z{(Ju$ac><}RrmFe9=baOq*0_5kdP3h1qDG+=`I23Zb3j&N|cla>FyFqk(BO|?gqiT
z7JlD%yyJK8xZ`*478{lslZNJ8RAE9&+|Hi8y>92o&Hg%UhT
zwZ|C`fTTdEC^$Ls1_lP;BfzEoRLQ{C5xi#o^itB&Ewi(9@YuUwulWjrhyYkdR_DAU
zldYJvFS_z=N>V8#BBG<-^#C5909T9vlD9k0dIrK3&j_)reoA|L5YXk8CbO{6m1P<4S6|y&H8rEV4#gs278$LfH~_&Sj_C_#csGP_Ls;Dk%Nc6Dp0!r~>O|BSMyr$@@t
zl4Emg3ol%DZK8s`@#@@SbGE7JJ9F#lN{Tn4dCv7k{wpFf_7_qH28=+W9(kEk8WWI;oIow4S>b>Tpj`*{RZ1a8$I0-e}(_f&FM9q!}3Mmi-KYxFaH2)UhPly
z>wOqds18yl^+nzupfm<{c7+wU$;ru9Fl*f0+?_o={_>e@`!&RC_ngtCR0-SN#9@pk
zy(}Ia9E4m3l*hov+rLZEN`QNuwx`rh>wf1}=H=z}k}OV5X+ix3h=td*_g6?Z2wrqU
z&L;$&M;oKd04t8dOYBz^0rO{$h9%DqnMf8@iS+U99UjKKEYcZ%)7RTO@-UM3P^Xlk
zH2@C?wX+%boi$4_@2h&KYRW4rhTXLRyi!TKp($W8pdcd~$;$^jxx2a5=Bwp>gt7pT
zfzX9kt;6FT!J3+y{htD}G2mywgRaGx7b*Zo!hKj^Ia-(mK0CM;7Om0^IIRo(;^PVp
zPV1Ta+-YY24?pAihTIz@FodHJZy%1R!3dhwzw5HDbOfbdcS!$S=uY?l3HZ{hi=hP^_H-VBy$@}FQOL>tH;Qx@2
z5GcVw@}WnVTU*}+La3*+^Mi^!phj<~Yx=+Vwwyy{g2|8|V6F3MXed&bpmV7!`d|x5
znT6DDv{3u3((&(=r+ST@F_d3@Ny6FBUyNBq$6a2W+^k?t1WB=)xOZ00?FDtAmukbKSA*ZXJLRPLvzm9JglQ5yJOHgM=O_)YfEN
zS=!z%JaHg!*^lY&>mvYwm-EB5^yk0l=6rzL14?iKs4o=6R!ob0=Ad2#(CE*-va!Fc
zo;*w7x7zjod*UrL3Dh_;J3D()A3r-g4qT3BQ_X?jPC5sUqOI8%1&7pB=c{V&TM&R`
zuoO<`{U`N05UuWZl>xvhF&kh^+E`gx$=pbI%0do1%J$+;VX4wt0
zK(wV?h=i^Uq+=lTz%+s^|Mf;mNvR|8fql8Rx7%^r#mNqei`@=I+IV5^%l>oMpC1nE
zYn^xS5w9vl7gt3dmqv&baLhOGgdswX>#H$0jwhC+d4*%^
z>gooBaBgr(fdt+w^1j3Y62HTJry&r`p5f*B(HX6nnDp6HaBT!3W1nZJCx1NcFJKr&
z`rhT-JH;kfC;fR?CV?p^B$Nh}k;}za6~I7igwOGhkgU9O^O2Dev&l+M2(Q#)UevU-
zv_i{Z_J$@qe*C~0{;Jlx{39+hK0X+mjE(e!RUce0C^Ymt5NtL(GonjNOJ+ZkJ-6~$
zKwO}rqONYGKbBKbQQ5b;0CE%ziBy@%Pu#24(Go-1o@7zY9R*WmECy^cdpkRx-+d_&
zKp|npacO^R4J2^v2KM+(aS|Zt$JPlZ@k22&qz!l15jGdcR)~JZJoQEo*ZTc#_S3x%
zI><4=|6%|RmeJGuGLuJt#@a1vv?`>oshMuK+$}V^HdSK}J8Qt@pMftncmj@n$zkCZ
zO8tOX8AutRH(+-SqIqwzg9yl%H`BFiBGmmk+I9RO0*)#Hc4}W3uaECT%>qQ&>EVzP
z65@H*lYz!86_pw9zn*}VVPP`oWf5(y(-t=CZKusKBoKFkf`asXe8Wo{V?av0+QBj1
znQ0gRQk;>d2zY-u{3RwosqUNgBRtO!ebdvK@M(nnfjMgdth3Ua^y5UN_1m{%;~(VZ
zZ>FihtaGSvpk#;2PD>cjkQRo&McbfS3FF3FVD^1)VC99Ihq~7fcFlcn%p#^{(
z0DMIK;h+L+eS&KXR#stP-Z<1tQ6p$YyLq>>Z{N9t;xcGqkiQ~@nR=k=HI()AbhPQ}
z9jxPm|3*RNsInbyj1mAzQ{m0NlnK?bJAsd_VtG<-4a{rsp?D_
zUkq=)$9{{28Os*0+aq8-bN^LxOiae}9rM3`4hX=v4=XLgk2W8+zX;@9Wygo0F_@_U
zJ32rfZHVUb@gr)BK|;Q=n>>TB69(dlh?chSa=(b^=FJfBJQUNT{bJwa?O)Y8umYl1
z`$D10Mh3;-+EDf)$P_`BU3x|)roG{px(opHW_`=Essrz@WK!z=;V?|GQVylyMmB5@
zZK;~|3PD0aha&e8&;jK|$E1O9xKqUF3JVLXJumnH7{5&tb|#ew7FJJOM^
zcUU7H92z1bB>X}%hl_xX6!Wctuyon*@=kPA)b8%Qq$)^2VV1GUNzJ-BZAHcK-V(%I
zQPUOjZpNrwdWKFeCBG#FlmJzizjB{L{S62fcUu4?2CM|q2swy5IFt_{^}TE0cOAHO
z24?qX*$kK=FakI)U%vbU2p-m+n4iYS?Cc$+f8Lx2J`?>1p68o*Pv8@D=RDVc<@%+|
zMl1jzGn=X|QM}yTw7hleR&#rM`y48Ayv6gVB3700R&ILLuVz>J?oKk4b6t1GB?Ivv
zVe0SFiA(Ap13~K#_2kgd^X!w>fdL|@$nsL6qy2zxyRrIs0N6PCOeq_i;+Y#AAOR`0
z9=trrOrvigpz5=KnjBYDbQk2KVxpk_ZdU?xR+O2r7~pCF#A?_m!Us5d(sLIeX%Gl%
z5tm}QJ=MGyc-HlM=&&^G;_ki(2=t7(Ou8lz`v+j}*A-M95p=SzUw_qcePnu*EvRb7
zO=0q&M86eX#AO!&Zkx06PEl7^7sWk21c(M8$^(5-Fcx6LA}|)g%cs(*>NT1NRJk(1
z4^J>ii#OrnS+(SI0;NjYU=zlZQinhvJj2srKQeM
z9rvGe9IvKp9f4MEsG#}Q+FDWaTL5R|XMzCGYmETwgvz=zGBdlt>{5uV3}q`RF!;k6
zXgLtL-`anmMIsp@Ozt#|@l)#xgwM$M)yb^40tN;~t}5G+NetUM$E)^jBFRzX?Wr0d
z^Z@&PeSQK)rBvb_Ho7V2kBwTnGoxNFz$EDwoQWc?D#DY4z^pEJMc+AE&&bN^W>fnr
zck|}WEd!Hw(q-E2odaHnExwfbvOJZJwzkI>78@(cXdC7`^#|d=0y@y;3Oj~ENrGv<`J0@IR|S>&-!6LaMN&A$kW5aft=+R
z^~+3^$EK#HNhuLP1j<`mTjxGwIW(9l*7qY_=z1Hd)R*J9)JbG+VZqM+vM_N+H1@_%
zj#uO3<1!l*8Un1`+{$k5?(p}Nl7up!e@sYVhYOVN2up!cK+~|TV;OD#)YJ0>uCpw*
zhv9vkZE%0-=Hen(p;2(@Dr&pnubQh`3UEq@cbhWpGup;Z(_elKS>Buaa&kevp>N{(
z4Aw`BimLj6)8u$ZXg6MJEHnv76t{OhHa`BVHp=?iI~radp7OGCtBLY1J&&jH_aoV2
z&O`x7|ccsVG9DRaKn`c_v209goK3ShJDit^C9#wD!y+(
z&V~amGzSv{%n1N7_)5u*Uze-t5xc4Tu
zhZ`MytoXplx1~x37JHo8eQ#@HU}3>@-kyTSI}+p{&}ugm|2DE1J>0Psv+Doy2FetlKL@xDB|0d$J*L(g+PaDjJk{sJ2(4Q6Bac+whnw_@VrO0@-(*=|knf!HI?
zR!V7y9l8K9VRH)@INzowu|JoW&@X^sWE2vjf}A81K^q2i*YdcI@^)(4Ctg#0*a`)-
z6Vtkq`v5B8c53GOC}oOcI#zJ=sDRr|0Ks
z`mF&u&$ICXf&u0YXmCII_VXv6*q>cLr>XFO?Cd*Gb#fcF)6>%V0iKf{adkmo5
zDS4zO+4;-|ZzZXJg^UNX;EU~IyN#V4h$ZAo>)G(cL~0KZ6gj$0jfyvUO?sMvcKY%n
z4+ulaB=wg2R041E%g~=F7lv`mXA%Qh+YYc7SMffNQ755RgBw|j=P@N1btD8H)&1t|
z6t7_AbQ%554-8|#-1a_%f;n&KJokyF#4=M(o>o%3CYycau7hCmg9!3&?s^8$g?@l`Fazsgz{6%8%b
z`|p*FgTr?~3bym#&|o!6x5>+G3&7w!@WPnb*lZskD+iVLBqAE_4Gs=2RogBa0Wk~x
zwNvI?{nGE>-CzXe08sM1j%g86h~tU{tRE?#sp979+7%Rp`8hULjX3++SF&;psl_$5
zCb;ehvTdm|lIdu~_rt!ok^%`a@e+7+HcwAa#}^U;$KQ85=)7Z)kdWXC^^F@h9_TeA
zT$j7a9=PnzAs_k3u`Yx3TL%8>y4WJ083nA3-LD*Fo(B)q(b3Q-MOLb-1;+s$!BPP*
z)v7F{LhPyRq}YB+aX9u*ag+RH^xefXJW7ARGr8@MsbcPM+W}8WsZZj>hoVkL<>BY5
z*?z|d_P;}csL9R;!mqrlDh}L^-OqSF9&YY|xs}*a#{oG7g=nBPBZ2pf1rZ(nxVRaSMumLc#6F18rBF&LLXo+*B|lSKNqPk9r|ED#2QYR$
z2f%Y&T-;xpP0Qv0(Cr4&qz7kbb-xb3URzqa1=ygl3jNt5GcRjVPo2Kxm9o6TBF!qr
z%Gbm4LZ_Rm6aUoC{Zf~IteUve3
z-&^$Ktk*@sfFUv2yAP#=5x5E^z|O#|2ch>iB7)#Wk&cy(&HB*CySm{u_A3kRVK*uG
zUXy-=oc+-K*z!pNKOWGX5;ivX!l@s;&3&G|P*63)HgKm`RnQNEUVo0GZmxumV=188v24l897
z5}KG|c*fd}n(^U2uO-K;;^x)v=H}>^#WC`kGd;}B(R{b==P^g0!Pb5^-VwscH5WIx
z*Dp1h8Pf(MiuA|^;X(_TQNz{#)Zh5;u3I4%iZL<>D=o+0rg&db0_Wq#PAcYhhbzquJI8$rv!%lq6~dKAQe8W;lfNjZ6WKyfPXli(Y1h(zfG1O%#UYCa3&
zhP0t(5GtlT3IHR;TL(;F`p*yq5%s
z{tAniz*GY%ync&5GBT1-Vh;GYVPP-8!LV2JXk)asudm2RDWJRlKd1cu#`o(+%vcz)
z)sJ?P_9lyl2+VqrJbwK6iFj}jR_Al4?Wy;NhYkSkkz58BAu}SxPqwE42jKxcZJldW
z%X=X`S!q?_?F>i)__4jc4!YK^E-c{uWVE!X!2{d_AqD7}Dhynj0LTc|c3!5zmn8<$
zGp0tw;Hs)FMMUc8%=Ai>5A1Gn={Dldb^Xd$R#pat^_CDB3k&O=&TaSor7jlTMiJOA
zh=xNBPm%`7_YEMtJfjC}TS>b?OY$Kbcr;S;ruwW1AmHu;VWR!(-?k-rGR0Pf|L$tZ
z38lKH=Tjs3h3D-F)o1J-KE@x-@UIJEzpjVjzi#M1y)xAPZ&I($H|}mZv>Y0+gPQ=d
zxrYWFO!fedFIEs5@F7jGuf;y_(K{PJk<1M6dwR$Qk@rREw74(Zveh|b8`R(*mF^*x
z=xK2$UGEjd!$>9hf7irg6iJGWMjBjit*fM@#6pS<#A^7-hja==NPmtp
z)1QmkzcRoS!9`uT3I|sRU=0b(aJ;Y-yiACp)ixq1SYG};pmRA^pyKPCi#j`cdwXR7
zP=APw^abWH6{PV*DIm=+AZ;jkj4)wqkt$or6h6Ff^;XmjT-Q^=YHvD0ryd8ZnWn!P
z+QM~F54^zLCofCW_%|AM;A_=7V8I&c<9~;V11gk!^@;-O%{SAvWVv`S5K4ov$AvBx#RnG0Og)Jy3a`qPqCHW64z69pd(h>#d$>HJm;k=h24x3|RuMpxu
zuK~+<)cqGsHli7T{=eoZGGvkuD%)#!{}9kUe2h-c$)kOtuixC8bbAy>66hcKfmSMbcng#S*`@&=YTb&H3<&@pUXJ}
zep3p)+-v|DVY55;h*uWQziRD0FayCBf7(JObZpm#^KJmU46#3bXsf-woso&D^?T%X
z4=R7-TebF>3e@joqzz{!m6dQa%-;TwyXnAP!Kx<}za^%yurQR7wzjr%a&pbU%BH@^
z3wiJw4%J`?7G|u5L=dA=D-Po7>Ppad0hOFfD`<7q>|d`}^YkK4@lM9&-@kur-eF*vubH%UbZonm;_+7L;n`A6#Pt?JYLx#3^7sD^t-u4H#M~OyRt$6)^A%_2-
zcE5#WJY)ItZ1)&eQ%djuL6;?0z|`L+gdfcL@IOAtit=W*G0ZkKbycT`)@>x$Z~JIn
zU(1XI{8gK6f?W~zzvn2Y!^Z$nTwPraUndq65h37g2Cts^wxR~*OO)fr2T*rE<``~Zc!qVJyw4r^)$K=9kfSOtsVy}I63jd@@;4+
zF13&Y=D@(fCjo0(_>$tp2lnV(TwE|GX+_Jt&$kI4OkB|*Q
z&?*0PLfD3W=Qy02wX!mJ2-qx}wZYq=bYirenw7s;?d|PbfwYA6Z=h_`G^^g21V_Pv
z-UpDfoL^d|Z&}EKhRE|r{bD|}U_sDX*RZ*1V38--*4~bcASNazq@^uboBuk^JTmI0tS6S)=sn!i_2)1<0>9oM_V|bKj_tqZrnKdsGz8bj*ya;rYvCc
zhngo>0?s&FtWJ-$pQ`0LI{Ly1uWA33`1lTDz0v~XzB%}Qg+<^j32|uE4T@KmR%CZD
zFfb6!fLDRwA>fsjglXtzOX$Wb;58%ix5(ni
zIW-Und|DB+-*VT9FThNnnV6Ty&fnHV5a|$dePFZTmWA!vrU)v&*D%gL3EHL|^Cf44>A;2h&%l6jYI>U6WmoS>vIyk}
zGd0fq`g-u0u9y1yM6`;*?pzD{4}Zeb&9Xk=NO2rEEG#T60g~Qr`omM}yb}(5B6QWo
zWo4}ZCt510$lP4TpPd
zr=4e|foiuoTvhIg=feSy!?dutn5vRnegx!dnE6na{$R#qOnm(ING92Pj_az>lLI4t
z|A_UY5ODZM!!O+?GhzrO^tz&Mzmq2icGz~|I~IplH6aLNxM4~n)u*mbIG*1s_%pBR
zxAF0sY9+XvJLic$^1g0zaU0m3SuuUf%bsLsZ+}pviVCvybSYYC
zx9d(yOy?fVnQozQByMqebDKJe)c(g0?*pTcj;MCpig%g>3B0>Hz~mngu>K1V5AR(J
zF%V7c?LpTY9ym})J{au&xZ8w4tPkg9PagwAfb6ob(Kqx}H5VBWbmBwL2?r#&jhU!{
z{uFz}zSiqfh*>c)$mD1IT0UK5Vj?ausW!N8fzxrC7iN19@Z!EehV#yh*;uiD7dnh)
zDa0oqkvOPYL0W>59Ljq78$U3a+pryX>x+Y-KOx*e@((IW^pRDhSXo(Dr)qekqN3o9
zHN$o6sqt~j2wG7I6O*joo>$G$-=%rq_{%Fk##4oR?9Lk83m^m;Sy)U17ec*8Hm+7~n
zTaFb!GM$7QyGfs}wLqcYB@3eYcyrwFa5#xeyH-3?g@K1hVz^sjVE>AgpaIh6F`fg6L!J{9LD!tDiaKI_U2R#$=cc)@a8B8FsyKu1PS4=Ged*e
z1;n{{y0-|IqG-juLV-VgTIUVo7D72f2|ZpZNr;Tj`ww?wVj^6@gG+FXjL5A4cz#7i
zMeqB!b19tWxvr=R8%ze8H>$wHyMfaHFKWs*P)zq|`&JO}Tb??!w>p-K}
z{Qte7zmt)b6q|*#-t(Hh@tRs9fe-j3jC#$4xBhpoSie
z!SMg?p)@ivq2hG5xA#j((2O*^^&jylM_FdKEDwH`g0o1oiVJw%q!n%n2?-nUjr8TA
zY$aI$t;&+uA6E$lA3x|7-uVI`U*?Azn_6QBaCqHS7}rnNNeC446We;)*jaXs>sxKN
zAr$&wAGh(XG`WKOdq9mhIY=bt%K9GqX%z5VJ%gW*xdV(*2_UoeyrRR($E*zgaHHkF
zr)Z(TUs%gO;sVptna&T_`~dO#ImU5nqU-AF0$mB;=LA%NYWGO@8?TZ|UAy~DZDpY`9PiEYu#cZ#~0>@;4JvA^OAWb=420mYVD5W2`{N_xKBM?@TmC`R>
zkY_0-1p`m|t*@_$CHv|1%#?5lA=TI|W6Ok6_`+otvwlQJI87_uVe5H*xFS>-GdOn)
z{y|Ds))0uDi6s<-h=>Sm>Vpjw6R1W&LAxBT5=lr(I))QbQ3bO;&(Z_T+?V8&DC&M(
zbDSO3(FaFYF0PH49*7N>je;sYYwHNOv+~|OLJ;E}z#I{gkoZ|Qo}tIGsj-dl-_@xP
z>gjm`ZbrPQXLc5rd0K#L)wi$+5&Qe+dEKSC+E-Oy*zr*2vYZKUAK@b~>rvL(K|eI;
zVRCJXlEBp(+>RvTxU_eVMs?wW3gVu{2uHcaC_dZ+48_(>fj6WU2|yPjzF6&;D=I2}
zP+`1%8x1HM89O_k82Q`3URUU7%gV}bl>~q5=)kzar6sPRp%EGJDC>EnhwJI-=_g^_
z>!p*o;A&;2eeK`hlN?n#Z%QL=WJb4
zwYIs5i>P)#DbyLCu5)Ja_W|0YavONt51&6{#mZ;CtO7n?W|C6aDSt-%90>(I<~$=9
z>UrtX|CQeLOR(g;^7He9FFyfQ{48%6h=mND@!aRxbBl{8nwpx3{&129{}B*JZ~h60
zUq!mJX2Kyi?#e_nKMj81cAb;Bk_0{xG{W$xg?QGq!4`v2B)I9vq&E>uOiXNSeEd=B
zv(-PW?@_-`Oi&fo?_*cKnGVJvpf#(~t+7M9INoZtC~C->9?4X=@t<`xYPeff0<->n
zJ+Fy765el0{PuC
zI(qV;H~AVSd9cha2SRQn1_ollYSZrSu7Pk?T-=T8b+gCF={5HyFkz=cYb#1~bYT@1
z2{YUPJTfu@_d-E!arf@s21Bm@VGpLJ$~Sx{)>8)@)@VfCVv6*x^@h)>=hm6JB>|uU
zY}h}Q-|<9Fj^$Yc5LbGA$)adqh5u=gbJuVG(2$SA30DqewauSXTl>IA*TDHsuLP^J
z)c6NP8;_G|XO};xB{i+rY?FpuJod5}hH-vAhrhqSI3|+2ySp4~g`t2yq1@A_e9oV)
zf8vl7#2vuDXj4~Zla}jCr9~NopDo+4ApAEQeFhyq-#a=I{!ty^x_`I~_i*LwHqioP
z21555+pM^QDzufHyQURhhFe1CKpI1=uRUNTw
z%0o}M7;hgadwo;ruPl+%h$tw6;GrVGbphdo3JlFa?(;kF@TU~-s|P&1ygz9euZiF?
zuI&yzm3$`g>k6vM%1V%>qM|fRvf)w%8Wn^$Z(3~sJMIcIlOE}T?E-250Uf@xvoj14
zUtiy#urO%}31sL3Uk`tsPja5&{?Evu;^<%)Ew+b2(dcz~^s>6PcEOL}VRH}>)7Z6X>tDHH2kT*zaL!Wra7Ar<{&H<8knmME2hU
z27m{&n`=fYHy>i2bptu6#dp1Cxr(M=VknLs(J=YJ}r;S~oPQjNn}RE;G4^(R*Jl(B-cv$L=NpUvJV{QJ)$YsN(h
z^m-H5M=aP^LidRV$~c^^B|vPw>?qy5R5ppA*pkQ_tLk@&fe(J;zYpR_ls9q+n_Iz}Z|z>Pn*8QD0l
z={oLr1(X1Tk-N6YmRgo`7qQm9Q4xuu)LQX&SOj-P
z?mu#nGFltEmJmYg`Ct7y1n63`(w_RB)Ku(|t9UC&xBk7)@hh!GO7-}aDl+hG$~n5P
z0mZSmn88JYJM8Z4ISs>TMMafw`>^Q}*WIxHdK?gZtyj-PO=n_MFtx>!)4!HwJa{Wy
zy*94HXfI84A1(XFj^jv3$*BLL_%Jo4wUuJO(%bUm2M#>50mKOHLysVZI4=E>e0AvM
z0I8d9+W;}xO?q-NwYNJZnQ`^;_4YF`i6+XT^h)l40J+Wj>4DRDT=#Qmhy%2+Xu=4aQG&C
z&`$YE_*MGMZ?`r(k&0|8q-rfl#xvmWEm2I0kzg)?Je-BpTOT3
z8AJBxzN6f-Rtne=(#2$dSkA;aRI-XH1&{uKXSz*p&(G@zz~IpFh+EML=vH_qkHYpD
z{+uEBxyMhRO22wV0h=I!uxPMjK((_IM56~OST@*~x#;xE>X(lCP3)T?Nrcfs95LXz
z4Licy0`P9a-VV4GK|XVE3YtZAe~Xa9mdaqxuee_gKrH0omX4vq()u^JfD6P`{J*5eD{cI?~5CqZ&Q3@)^Jd3gr_v|MAnk*!F(`K4k*GMv?-3V%He0#LlL6@$S
z@(|9m+wmrrsi|p$5a2(PpiuA)gt&fzk0F@)Y0n$ULn1$E@QE7e!D!^_Ha$e>O;y`c
z2)s$ud0+}@fk&*PH~iG%mu{!x&j3fXkjsfjQf-|E-Xu>F_&NLR3jGzq-FhjuD2n6T
zSkH=9AKGy{3rZa#L(MEAGBw3O2xsya-|#cnJR9c|ZGa7^&2O(OG?B(O>zn>De{JZ6
zh(z^EMClP9l9FKcW8q(;aX)+C4j(71JkXmzC4OUF6Lu~(68LT6u#c8zLU68qMc_kh
z6xvPMo6%KDQg_J#J4HoD57atZB#L>@nEk8JG>63s=soXh=jg&f2%b2|{P5P`IiqJp
zkpICRiD7;IMJ4ycr%#xO)2*smSYm=g9`-92YSkFbAV7}mAk4ip7&y7`wp*uvlKVM6
zuo!z$%BOiDb!bpBhH3sr{ybX|){c9qL2N{x-7b)S
zTTsWDSZiy=!RW>fIbdso&&AE{TX#1$_B-UYwY7=TmpZO!K+tj?LVXQ8cgH3s3I4J12J{1*wA34+weq(~M>RB21
zDJvJE_(Ul>tK)9?&@LJd1{aN7z6MvA5U(*BF5dAa87Jygm5F!$ZXFXdMku
z^VsAhEM>wyIhuh>V{>l1SV&UT8x=%`fV+p<@LpXQ?aBD
z1`?0zzE4w=Sbl-R@1Lb$4pJRAMylP8*gkS;2h#{UeewQ#1s?z%m*PtSkmCS;<^*Sd`F0E=GeyfQ&aLc!eZ^Fx1CR%-Y)+qS**PR{t_JQW~&Y!(}&MZKp8S3uAk{`klVc>zJs!h#7*{^>%M3Zfox1a?H?u7RL*+$B6lEuOSf*2u=h7A0J<6l7^aE&)66Z
zz&*5*LPAu)gAFj#A?D`ieF4zHl7)bI4~Uz)!FEr(k+A8v_-FF%=Dm21?(r4u5B>`jCKRF8^4YtG5ig{TEtin0{QJcuO
zxiQXkB=`0D7Z=!Hy?a)m!M8cvL>|#w_BA;Aco!*iU&UfFLZ@xHl*`@D&aN;pz_RTY;Z{@)2Wub?A$2i=o!Sfle2Ljty
zqS!Bi$sonLFf#GfMBJZc#RxqE6S&|tp!kQssu?bK$68L;#scfZ<9+1-azKs00biqC
z&`JWnA>R;Oz!v3nFD$W#xoiYD>*T=Goa4VD6CMGA6J2;H|0Y
zY4t{r2OtJSE)R!HXbyJgTVW%Kz*#k{`qr#WR#*gt0lSs)X1W%{-IQbE
zd~4w1{CwofD>E~*WR)iHq~E=hQaqD~hLnc*8#ya`D>VY`WedZ!k>tc;xUhm)jzE%h
z4`wPHP%9`Zj*WY&U*iS`s&rI?NR&RJ^H7+^RR0MmqsY&ZsX~oM_;MTs0A&z3*%t*GZNGA#+q3JpwY8N1!3(?M=LDf}_?aLeV6h4A
zshdB#OX|;pOZ|mw+{6+}bMMdaTHcL?5s4@Q<@4z^q}@i(3$uZ=cQQn=c(v0gQJ+4^
z0a=6)v|GYl=}WmIB0}vWOXnGrlfzOL@xb6aCXfYEz}iFU4qUv^{!&*c6<;v8@IZsR
zGlE$$v2!@@B`o|l{n^LKds&nPW*Yc)QZ^(aa5uqhj!JBl+?1-WUk`6Q*mLh97!4>y
z(`E5ziFQcvsjjOdp`#;!o4;~Yb5-7~Y5KM3MXFSZx*h5r{Qe34nb+gg#_eSLXPR^f
zJX{m*!GD8s48RTUirw7WTAOVW^TQxuzIX3Kfo9dh+M4-swLhUm&}rMA;Hl6Y(Sq`B
zn*?Sm!!%b5AQCcI9cgLtRT8_538ZynIhlG-a%cDGsG_qPo|H9R>xiZ^3cQ%t#g;(G
zcY5$cQ{A78Z_Q7l0uA!EW#o&apaiDv%i1I)*PVtF;CGhi1L>F<7<_>m`JJrU;Km1T
zV&tAs>Ycyin#}%$wf1LKKa8|{2HW=88T|XVaOHrh7Ia)E1@H*w5yRQ}WXirZ0FUY3
zy=Sef{>dEL*eZHYFjGzULKbb!mn7b~D_EatGx%3Os=~tb$;;cI$s>f@T>Z?X%=Npu
zZO=}A+~QDbjxqe!7$xeI?BsH^$2MHC_l_{Fn)<8I+-NU>-y_Z@wmyOvPnbzU$T=C=
z*n$}(0|*HTzc|WC1pV#bBIfY%_eX_(>h0UNIhs{mfdq8ub9qIL3m`v71cg%HmFXrL
zn>H)rI|z$+)3F+TH~LWi@ICKPUvbsA&O%n(vd+z$I6o6wuP&+3Is6Gto~=X+oi;4R
zT{yaGYk+xDLnokp0`wqYpbW(%c--W@lS^O1GzTymonZ9grmmNTTB+tkS>M~+A&DdK
zsRf&X9Q!2Z-Po>-ZINfAi^g`6|D>X2*v4Y(p38BV(R#qTFfgEhIP7ZM5);TpdW^HZFE5+r{!aB`N4?dKok}1DAT^Glu_H}8EYo^iDea7=d8!w$i!`6Bu
z++FizGJJAu@|mZL;}d4VJ%-I0yy?1D5~b!Zk9N;i=5~r6H3Nh)$ySn-2^di(<+v0V
z>Ik4|&5O7Z?iGH!aTW-gT(vy7m|tUjrt7*)hK@(|y*=#a?K8qO(g7=Exa`z;x`6ZN
z&!5i5J&FrmfVYc4!cj>x%jf-4#Uuz+%bQDb*87H9?eI8hvP6=Ykip-JQ`^mceE9dz
z^%ZUNlyOI^G%BJjnkT#p$b*I2!V?u1&Xcx45`bsdT)4nt=#j0$k~1+eX?%J)I5bpJ
zGH`TEu5Th6@r@~>`7>{^TgACGf{xC{);CFb*fNMvVs#)4)o@zqbWhgNvYXUw?Ac`X
zc6-+>*3FkU2MMwk$DhU@Vk;-F9&5Yl>FG6wU4f*Q0iM5rI+CTp>l{m?0y$lov!OQ9
zW2!&@v%EM+uAMB!cjTPSGqmJfzl7rJc*iJgtTyG(TO1&3y!!lx@o*b2Nho(I-Rp9)$bA88
z>~CCT&l4O}c7MVbrS4T4UPs51b1N7jHzXCqN0g_v-!YTPiQU2qB3upw7fY24Bqk?^
z9p5=B(vDtMvyGw=VpoDnqDQ3j^YatvEF6q0uki!v#Cl3zhU@cu-f!T#jdp&lhSizz
z`1P!8Td;FurSWUzva*xnoj#ST1Px4vD`TE`LH<7!=7EED5l<7)9XGzU614L6HP_hh
zUhGhzt78JXcq|qb&OMpq$RdHsKDb3o=-1M`IbP=a`?D#0wKP0P@W+pknFjZvN-Isk
zu;j$Ni~&81pUcN48D~qwA|qdaR`aOWz?m*vncf*MF1nd(ZEClkE;#Y<*$}ca4uijI
z=AX!K@3-4@0$QHG
zQXHZ=J7$I6wCB7d{PrhwG*nB=E0PT*Pp_Dpkd6R{yxF42cZjo3DcKFXP&aV-2Omy_
zI7nC2HdN)3rw=WRb_@Vis}^X)mYMdEpH*m9TFy6|%+zhD=rH&%uks)vL2Z4L-?Co_
zS09Zz`hEIc>bBnL@a7^EYfU%Dv(gE(s}_vi`M_cWwCWl%tI!det)P47(Ysd2pG}Pf
zY6;U0<+bAEj+ZUJiew&(d0o|fbvs%JaV=M;D7Dn-c^IS~0d_Ap8+Z!{
zm)mwh3U)idLyL8EYUpNWW;6m>dTt%4GCx#0pBrbgPX9{AuDUR{kaq0uu<_y#w})p-
zUM-F7QJ&qK;|HOm8H^!dY`Wxo8TUHAPmM>^*fVhA1rRhH97Twvu^~QIsDA*6O-w=}
z89^%=eB=i5r&z!B6<}lFZcM@z!O&>Gk-(r3^ZKLRcVp#!5KGdZ+5NsUMS(lA!!;>F
zYDc@Dgm3he(69(njC?=kX?#RAx>&TRD#@RY!yq50ziSYP|!2uu8&
zh0GtCyIdaCH8FEM5zQGdmF6?%#DQDrBqSu{RYThWLWI)@m)b4MKNj~dF6G(hkcz?h
z`v-+n^IM~5v6pBlm2u+|iYju{XDkf-Pdmv_KlfH7<{m%cd&10zGnI~U-`wtGwA=!t
z!Xhro_*HIhuH|^?yJR#zS2lmI%X4^G`nUj_f9#qQIx6tQ3BU#tk&#Ku$h2Fn;Amvs
zJ~ZqicG-TA^OAty>a7prAY!2YfuLXx-WI23{NF2ZIj)V-))v37$S)q@qVwVG56IrL
zXI9P8J39?-HR2(AeRlYfQ?s+Xo9*~2Li1>6HU-GF$Lih;`8|4r=0uy{ze`P2St~g>
zI5dt@gq=^w)mzU#)F?M=(e=KR22!yCKU=l4H-Ovlm(WUfvWm8sHpdu;01M{&FWwpNi}&oeD@Yd&)@*rbfmBBJ>vwk@_c3A(fB9Kj7&`C
zW5uatj3f(A`(6c04t1y&n`0${4<6X=FFi&$ZBKO#q{~wBy(abb_5I$~W`6SvOG(I$
z`P4wYcHAQ(BK>hV(xrxh#vMS3MgdU#i5fxq{M0xH35k$c#Vt;MT&;ZZ_0q6}w#7Y)
zTW5q4D*hOxN!2wo1EeyHv|-DJGT5y}G@i)%Q8nkn&S#
ztyTL(|?w26}qfXm4q0DW{~=Cprq;1ysF2;Ye3FI0dHdj7g`J)@
z>w4#AJB<*NW<1=PFmB)m`m)%-KyO)I?W}vG(yVA}R4JAtBB=O@Q_DZ%x_WzoP*zq|i2JrInhW7+Na0F^yFb8c
za=*yRnw9Uh`0&@)lpnjy7M8w8-zrone*;XPZHQ*oYsD1&nVEZL&jxm$r@ob!U}XGK
zzIEl0n>{jWA~mb?Oa@c=c$FW?e5U>Z6tl2~59_}i8s$O2;O&iQS`#@eDp&Tuvl!KL
zts+Tm`AnOlw(R{EArab?
z=|TtsSTI-yx;k5-JKmX9GBUb#v@tp=C}At05GRrHaGfb5{r#iS2>X@-WV)TBETxnL
zcZqUB(u{6RS|QY#%LZ<3-pJE*j#grcSGo&I@0YZ?TDBtDdg9R}1Dc!qW`f(=WjJRw
zDsi{_A1XZOI?}rTR2(F*Odvi@zqvCqgM&1%y58esWBujk&*Sf#zfDRiykP%`lbX8Q
zN^sAKqAG5hYx?ipZ<(9ep&f8O1ihPMV6k3t~%PO#|)e~AFi>-D6
zO)a^n@8(+-y!syb&2^KVUR^}Z{z!ZsTKMSad%gqFCcPG9A?Iz&5?Wm{_7_GM$6Hd;
z(&|fB5$}>y4_?wUE5z-
zpb?O{+?xFN>sZ!;cAxc3{pTNXcMS~LyMI(r8c{wpI~2LH54=jXB!
zw3*I3Gt|OP9J3a;kXWZxp3hGjqf?1!Mlc0F70Fy~kNAs;{TA=$FK_qb!&^>999;>~
zd+W=YR(cXac8d$aA)lTwUpeeDaN*Cmw0fN269nGS^)EAR>Fe`2serFIfc3qQ>Djin
zcOO0wjudEi09DJuA5JLIp-`;jwssr%tM>inAMZm$mrnNi4?TO3Q_@pZ6>?D7TNlbjAsv6VAuI02b62_@nxwOW~?0-)Tq(q(b&hPYyhiTq_uYTpdjPUXC
znS3a<#_!DKu#x5F8Ixs<_#(ge_G^IgJ&K$MHOw=Y^%Z#Xn96UoBAOJ}w^h0B7k=|a
zxn~mUy0f#5519X>{+fQ@(e`v5lG#vJm~_Y5OoQ-fk*+CV19BSKpnXMPZ{A^(**H6o
z3gIe0!NAWEIO!8Ba=hjIqK@Ci?Q4KPLM=H*B2)1;izt&nq4eCdHs86j7LWR&kdcD<
ztBapvGA@ErGT&wzem7OsY)w?)5D*Zoj}%x8*dn8#U}0j;#p!yLtO`oRIM+TE|JK^7
zUTXAyL_f$No)_b*s&BGrMfg%LE_wy7#foA96OlyB%L}CkyOS?8Wg^-c8zP0A(RF7B
zM@g0xowg>V6Zow-Dv>VH{aWCb74@n&NzxreY4tPf`TFOs6oau*j$_B&M$F@54cX|XIlWK(c%d?C!n(i)n@`-&c`nNtTZ*cw
z>9WpxByM)@VB=8UBY4}G=kr*F#SI{7%eq~P8qeAE_4Pqi;b!It1``^;FC_kOb$K!C
zw7Jw3{pZg?Ulol21@6+)(kmtoWhwTT5_UWy7x^QRQ`*l6Gcv?gN$*?cINp~C+P}pt
zbD95V0VkH_mHaD;CrQ|1m#CNL>TeRib&JG4{#J_mN9U+@Ik1$~JjvdUn)>fUQDJH3iTq2df&ip94ImHT$+PB(=XQ=h{szc%m
zug9KU=4)0L$K0Z~{Y*_~Pu-`JT+Q!*);u!u@lLXf#A2sE;qQz*)LXZ1No^2HVbPrP
zziX6DubG-A-yIN2to!*91`*)qADLf5@6TIV({ndNzDDIjYso3
zX%Ad|%bp$2QJea!klvz!q|B3mkyK0u4mn~%eZKeJS}iM5#CzI*Y@G%=+8e*n3wlr#
zYANfQCimVtKQk-TuFisa&T~ijc0!2z(B}uwV_Ax{#)TH$+$45pGSjI(w&*q2gqPi-
zmx!^A^1;kIvp8j4u*sT?8EiOQf6>zN_3>!3=(>eplJ9|iUvw#-^klO?p|RqhC>Q*S
zaTHedPzwF&lGaxJzg|bFd@B>ZM&IQDeW){I1r-OE`V5Z!rtTTz^>%q>F4w}utvrb?Dp{lF
z{pgLei*9*VIa*R!RqyND8s{YYGUB1qa&ge(4~f*M%D=-Qetg``5c9p@4ou^Ad-3f*YcR5Pamc?k&oAwC|D
zxDK&@6!2tH7gIDiT%j0{)U8HSE#gCoa(x80Ja#9CPt#wIi<9Ka0)42H{mCu#Ms5ol$`e`(8;tvyYb$Dg#+~k6A)N$
zmInm|ff%5oV&PR7+Cly*6qNqc&Hh}eLg4fL>1?$Q@5y{cXm__DooeM24;#4EM&+*y
z{)eoE_$2M!L5qXk1rv1p{+RY2^yl
z=&db%Yls%mAdzn&NWJ6ZGN6A~7O4s+z+2igA2qZ6|
zSYm#*sNb(`cl}f`NL+nn?WTO3VlU;42cIyZZVxL$f=u>3KZ%%AFvw7mP73PrjIO_{
z{Y|m>80rh2YOH_($2C~n+=X+a6bkGOMoTGP#f$TOe+o6|4_W7+vM@60|84LeuSd3s
zN-x%Q=>eH=imE2wcoWZL>y8^*;C0NG!IT&V(nH43UPCyk!(C93%${k
z`P@GXuz^+JfPjeF#akfJhBL4S_=J_Ubt|B|i6%`C^@R-0A5N
z+VK5&<6AsiZH@(f4?Z{$FOo_=TC5?VkPa>jaX_ekbv{w9
z*6qwVbTc)j2K5tI+jMBL)l>aQLHqPc1URF5gE{)?LY2{2+E@4L@Q4UEP_-q?tCyZpmHG|lNZ
zX8DPeh9;ucrIjS}(PAzhl5$b<;^Q+_6nO%l4}GW!mwJ`;el`r_bq=|
zP-(&2%R>pEEwgxCV}P9sB#B`FAjbs`mg~*%Sd8JAx*o|{%_he9H-KHmw-`+#NNi?i
z=JR|^L#td0CKQRy0Q&F&x;s!8gQr!I@X9+|0&aB{`2W@d__MJ;l?3odXAa@+cJq(%>&f?^XQDwbmH=vXV#qvR5(TRu#-4D3{5#r*&q48bgKXIzi<`W4r
zN&*2E5SU?+krY%^!GiCPfiIWco8-IJ>up$aa&j?*+!8+jm2zc+qp7^Y;$jFi
z0)ii=rUyipshmG&g3{8`ydJ8LGzPDcKBZ9EwOKn?>(-O;X`hDCQ+$@J+I)Ypy>ILc
zy*swGEb`clJN-LA=Pk%*-y#iPwt
ztMGvU%s09{066`@3w~g455TzHFIo@X_LJ4PoQ~wbcOjDn!Mr~s^0{OU9a)5`=`mIH
ze`&a~<8nr5vZ|eM!^6V^i20Vbma`c>a98K&=NcVuUn^9rW
zwp1I1#~K2xPR^p9Z#JsnYaFmz{p2sQcYO=g7AbEJ87cSlK09a8;qJUUmU#R&!)nd$Lxfh?q`IAdKSCneRV
z&n#c2SkxW^`c%xv$9F@Bh>T3d#@0Z&t+kfLEnkJ7bpIYJEd1MQdAx=WMg@wZ@Zx)s
zVoCeQx-0=}$VhE%19*U+&=zYC-XkeTN*NGA*6*)gdY3ypeSkv#xm-sDdee^BKr2TP
z^f_TkovV`t9(C5)D`$S_rh)(o2A2&^;8~JguEWH$K#Y*vA;`}U3<(BO08$}E$?OQNs2ZhD`sG}3L}!UX!SG+OSS
zos9u>!G_oOYD*YUoRNWn{W?QWce8RoH#*$IcN>c<)V4Eiw~A7Gs-r&iurQ%We38s1
z%UP1Ek#%aT%>O16sUb)U#hh)7D{pb?b6(NWK7pA-c658X`tNd@Px&!*zb5?t&HVgu
zDOaQ0DfJH-o{at?QEtoN1+itW#!rit6Ns=n;XwUYH}yt@NY1qAsHj5WK$sMw!ZZ;P
z5$1GhtJS7xfZ(g2+67^){r^P;bx;*>0K-5%Aub=PvOi~30mqVRoa*osL`2Zg|B@EQSK{m+k$P0P!;ek(mt
zTMJTuI6i(cMFHTK0ZZ)W2Rt{mazl89QJs!bo}5+WvhQ2;bc=QD7{0zob0atW!qU`A
zWwJs5Jix)h#Q}UKNgmHv)mk^3*R@!!MiU-N$QMYVlEC&k
zny&=ZJSF79oTUBjeYQ_YjVIZQ)9PrJtSmynlN
zx)f?}{UeL@-Ls1shyy+Edn{R5SzM&xVXq*x43L_mLd`jf2#rxY4+>sii@UJR!=}bz
z)FuV6GX$AH(-e&Lf5@R`C|OYTu?vhbfIOAzwd4<{kB@F`u9vd&}j@080jtWqfIMb++5r7xXD!Utiw_Z?le?7NNFcVK}l%YJMFMD?;_yWSOcVX=&@r
z7fT`79ge2}D{_H>QU=vt`}1DhV~SwL0Gt6>k<|WB)I%SBk2~`w+f7+JJ7zkyYU7i+
zGHkrXk&$o^<|viO+P^(J%Vu%)g36paJH3CMW^me0fRdf2CjQ9CNTqgbLI%5~CM7ik
zZhbVRt=yn7%U4%M&|4doeK9-cv-Jv(0L(u}WlHYtsthim0%gfYA=jx%
z0Uv*?dw1w1cgv)AB@ePW<^VMfN!f6(
zcE3MIHZ(M>MPOEGHtx)oDQSkN6{ky6?-UOx9Op^B`iv$2NZo2ZY={|c%V1aH7mlb$
z`s;abQM~TSBses9^UaA(@N#XWAquI(_p$fYAAyXG)i7v<2}o(05{3dqyyeEDiu}T$
zT9AW}63nL}_EgDdl@wlhg_I)>@ayYqG9Dg$`4{2Wx3|{A&3~(9Wo69_;hzo??|ytb
zewi)1Hbp)dQ;;c_y7m!{v~vhJ7>|tiqonRb6)erp;$$-ilM{d^@yab$2|{Z#V-XgG
zaymzN*r;%3J1{bWloPD-ATu#e`*@$PP%AO{o8lgmo$Z|BsF%a~p)UeA4qU(hoS=kWV
zZk0>vUtTPiBd(jlk8Y38mbWlpf~E#+y8+rlEiUM*b1FZ^NRL#l?ayvm$@4$r8{GD*jQ|Dd>mQGPA*Z+n6x}@UWhs?ry(t#mFJKGYfc&bx
zZpS(QcyyxW8>cV^3wY1usFD(z>izracJZ^$vuocWk{o#wAf_{aXL2s`(4ONW
zHFHurS?-Gi`XFl2mKv=4Jx3)=U3=hkik
zyS$U4-c^U$m`phrch2Swtf_JL2rxD=xgMpfu{#<>o&B)0JXfXV%yT`N3zx~{_z7Z<
zz|JiM_Cyx9V@NO}&P<6M2_O;0K(np|fgg&T!%PyU;T#Xk@4GK9G$jf;u^D<6{vEK;
zYnf?j6SN;(KrXk~Qvf7kKfd%c`-=fHrq`^+y#5k%vL7W@=hMP&tY8eV>i~to2zru#
z5|auemO8H>aDDhfe|*DvmifFYNNH`YA{STmBd_q!Lkx-F1b`p__1__y0i)H@vY3d|R
zi;lDUOp%pTB8D(eH<@;>+YKECh5j{qD9XR4E8uumC9sd&&fcDko<8d1?b25|i%S84
z1SkY4-R(miRtSrE8nyE^(6)t7PQdY^Zhtxx-)QL
zV85MDRRNM%W6dmb6b5AXtAK?W#K<&WHTH`X4mwU~SRj9LcDq;}4?ejcj*JX%M{{m$
zHDb{*H&~TPnPR2HqITRMjHQA1hXyvg#v9@TyaY5!X73vuU09^j-aQawpVxEx2&^CoX
zDoj2n$49fkPpT}J^xpl!=6-o%9VL|ZV}g%f8!nkv_;N$6ySAhx<1LN-$GH`jND>b$
zN@-5?WfvkAtvOLN$1gV#emaf1!pDcOhzQUG?gPZp2!MfeIqZd3SPap|TnGMM;XV1g-CN!wbV=k#SQ-gP(t%HDyP&pCw(om6g;3
z^V27hP!yt@bs?spmTVv>K#UGJzt7_~?5~eWxpx?3d{9J3lhYA6usDWg;)hP|3UIk`
zUmfP4;eF{8yx0@})~Cgr4|(-+X?LN0ouaqCwVk)%>~)Wo#jTXU9E@h#^fOV@gUy{~
zY1NV_x~Z>kE5}#=<2}Fo-%QaD?Jh5kW>!*wAAyR`0Y?6lC<3i5Zxz%}o-j2v#rvy5
zyOkw8JX|%V)T)*F17eX$d+5X3S7_X4t`Z)VFV|kbNLe23Y^7)v^H8%z{(Q=6xz3o=
z_nckD#`0z_VO;{wfP*@bR&}wBeR#v@XhCLvRJ}z$d7mNT?#{{M{zMEA3iKFRX!&!~
z)(|^8J0em7+4+;%yb?uH$(4i(%LW}blr|9=Z+2yeB2ozSTE(VcK^2=`Jx(iA3sdJc->v$O
zPQPzu#+!N_RJ9({vQ&OuwF=!mweb+j@Yooo>4CKLVs+W(@WIv1UkmO9&r7V+{ZuBs
z1Assrgo}kaFw!Az_z-y9`Na@zzXsf9Ww|My{$>NPf)-%WbV1Se$Uk9cMrFc{$VOQ}
zWoJXNW3CNir%bR?ge>!3PGhkBb2P1qB5qX~IP|
zX8h0Ue$s9flNnEDz#C1%!?F$b#bA{KnDx%Rz1a1^h^?k~Iw&ixKNBVfX*N06J2tA?7Dw--+%G
zETT)V(`qH@T}o^Oj?)Ky|gjg+r*xZ3J_HOLB6AUn`
z#Dv_R4tXY)+?Uch!R?)b?01zl>b`{{ys>C$X~Ccng8ImZM@L!o_(YN@5~<59c9O=$1|=UHhnFL@-N2IA+veE3HdF=
z9|cAuyDiffboKg|^LywsN>vEL=7f#tBFe_T;05T7nn
zDiPUSeGt=AA>S|#NTwU%c29Vfvc&Z#q^lu8frkfVZHD9Fgod*=*`0VZ{P4wkdpffL
z1iQuDu;be1CTO;1o#i4UD4!inKr=D9JsJS95PR!*=m3o&QLi^#m2|o=8hvV3<9Pnc
zev9(&ZdSgzT1qrRe=1kFu;@FHH(7+?gH0pledmKR102{Uj@QrD8Y&b4loc2?gr}Ep
zogTqjAy`C2I)!hU7w2o&mG0L$fpdd+&5k-s~O;FMC0Tb98w@UYYt=dv8*>CM!
zBXq==9C%32!Bt18DKJKZLbJu?=zn7cSv)ocHmY?piAwQM-`_eQakbhx3QA-dmN^wg
zRmzp~gacvfkbZDHQ9wU`Yfr#(zyGB_!Gu<>meJ0
zRIH(){whaSC{nR5Kd;M$awt&+ui|NeX3L!8*N-$gK5%6GQqjUV%4@WI^M$HF1})`T
zVJ)qyg6Y^$;sB+RxsCK!N=nMOtSs#-q_B}f1lze9)EsY^%MQp;)NeJqZl0d?08pt0
zht9See0zq1nOR*nhpAZs&24Z&zPE$
zFITM^eY!o=Xmk1k4-e1xZ^;N>yc~7ViXxPRH%iNq6Q9vK&C_dw(0sE?(wbUSG{_?K
zeV793P09wskQul1Yun(uo|bk~lLf`|ZfpfIul1^?DIwWtSr*}aPp6T@(e$@d^E)a^
zQ5B?@do?(uj$8KW>1i<69B^(2C|!3k>(B0c=#SkW?NuzGV(Qt0$!>T3V@0={?=5Z$
z$$NEmplZ?mR6U2Az3&i(5s_%Fg%wbRz8~++M7-Y5Y@QDno#&5$R2xr}6ZjeZ+lg*F
z6N)9qbn0p~b7bgx_qPEgy>_XAzK0Q8Ih1h^n4LWt7cMrFs<*N>L#IfAuWuvacDC!B
zUKU!P6>xaI+VvbFosS-0P3@l4^6fd3;NgRgmKFhkK#Rp{@0XD|<@rnVL8N56N-ed=
zn3lKavIy{c(=t@L)44JXU0q!OfF%JsFW{8yXGJgl9a-V?AUy>QP4>X40@GSX>pt%s9jnDPi^nN_f4yi)y7KOjw&)0qiLqRE*)g6qhh*dIZ?I~2DnO;2|+FfLMe9%T5
z=7lQH4hnvB+}6jJmX>z>twAAHo~6=iF4u^X6Hs6{j5HcXd)$I#&;B^6_|q-{FL^^Y
zus+fq=P>LxK3w)7yUpK|i6BF}N9=nm!{w{%;fwiQGvj0k9XsV^aC1VI|7)iL{NEiU
znE8D7RgT9uS2|=`(PihSIw`&oB!&5E@apXD!PeG8W?j~ew?E{Ioeu%Aa_{?Ypn*;$
zIX(zqzA$@nv01`gUt8=ByKe4hdKv#3**jA+`)azrf7!RFM)eg~L6bU>FhwySZ32(g
z?W}eP^mZT6X5()y({?gz%TKL`tuFuY!7(6i2=zK}HJgH7E@r72b)GMHS!QNO-Y0zu
za!J`C4I5nu_&tO&IV{E{$zb_?-}!R9Zs5hA_UT250i~*;;f^?#oo*kp6koe%yl0M+
zHnFr+su8E>+j;wBp;Dpv5-X@Y8`tWnvf#P}=5*|^_+lz*W@mNul|{~EgWoSzFQh&=Yl{wyPo4^D^hu%7HKyP;TIhE_31z9f-z*+X#Hn*R8kGFPJr&D);&krfaycbLtq
zqPBIGWW`9|=I5$sOZXP?JH75RkE?3;)KIJ%D>?KT^v}g=qWAZAdfkq+m#R7qv9KB2
zSGPtp_qo!{p$Ttq6rFeYd5|UDt(*prCrI_$EHABgb
z1glmzn=|O;X4HksFINr(1Oy?+Tcp;*{v-<)aO8a(dXW4v-c
z9cT$XM1H={{QzS~MC*Px4069!R*Hj0)IU*vJj@}dApwKYf&_RCqO6dR2|qD0?KZkd
z5c*Lk@6-lv?VFc
zZ3r)j;6(`RjBa&T`EU3fe_2X29i`}-2#H{qJOL9d3ZbNf1B)eMtJRONy0fb0xxhdz
zMEe1zn8WAb(>b?(5oEEyVUajYH-lTfcUE&@0{17=^$HXlxO{2AQI^LI?~Ye%UVxXf
zQ+*<;Do#6#?9Li*l_kSK!@*&{_;`m^tHJi{*clxi-MKrSRV7gin~?>`4Y12cqwrYO
z>rIgtYxEX*2-l^ggMQBHFEQJ>Eb!nHEq4*gX6ajwXDlr*Pya*F84e8GzUtzVRkIH|-n@b`3EE!>$*rAIQ
zx-D~wD|=7h|L@zeoJ7o3|8&8t+aBB-HvQH$k274fxxbMre(lF9(xz7dt#Yz_9tIkq
zx#nk>kc;p8`}^^j_5J3Du<{#btwD8Kk|na)pqYz|jEu^a>gb%D&3rjxy#mjY44odQ
z3;;Jvhepg*WoXLc{}GAJVAuKW?`(_nE{1d$@&8hm5l_;CWnII{E0|UDk
zbz}rv+lMX)<@mq;MxY@y59_WZkT6JF(-I^_pqh@Hm;2Ka=dhxUR>$(?h2eOM$pHFR
z+QK_0^;(G8Tw$yACWcMqHJQwS#MyBZbsL6tW?gm%_EChtVAQff!;+8xRw02wycQ+cP;&WE|1C4zZfO7Jg*Dix#*(HPl0_BMS%s~owTTU)u$
z<$KhmS0zs9={Nxa|&qZC$|QAh;F0IB3&0i|@A+^$T0
zBr3nH2dX2Y<#sHlowTF^&<<3Cet)|7H&B7hCT|rluCkJz*yD(-s{ix7;)m>hUp``5
zlzU0Gk&((~3FoFV6^$4=y*d~f+1%Nb|I(5=&)b8}Uxt7H8`ZwL3iHq({5Yp)kl-rl
zr%PWy54^_!&u+t>BofVJeXSJ@1M-Hx;?d#kY}nQy@p|)Cr@j2mPOC{sSsByC#bvHs
z<*$eE%2Lk{rm{|ZK#16C&(RM*-=BWfaDM_~5!448O6(Dq%}LA<$;*;1JrEN?by2|e
z{<6Bg?mAyrk*c>q&9TnojKE+
zuJ?m*w+}6Pya=+$8a0EA%qM}j22GYP5hVtv3umK;Yt_D!ixsf*6&2M-@$vDX9`%<`
z%DuEHS9W$zLCXMtK!CsXRXkRU)jHvkkrX^UE9JWg3K>8c;|cok0UK=xpq5nB)FFk1
zg{uOi?S=Q>DQ%^S7eX7Emo~CU1>Mk6=oLD13vX1=88izQs<0er-^9Q5>3qDqRP$GH
zGzv8AD0RbQD9^C@cHIwRiqI6xsr`xH@RU21ytBTWE9=l{=irrRC4PNzhh=VZu@;xh
zj_3u`Y43Z!rv;xj9w}+(MZ$;D^R=E2WdXHa^!9>ANN8D0gJP4K^kKu331HJvfZ_r4
zeMd*fjhCBe>ZNl#bwT*9QOxef1`+U(+$W+5O7QyxUMj_VkCR6V+mh}tQLzx3L;EM?
z`0v|mk^Y|GbEFudl9$EU;TbFk5NSZEbB+#?$+0b6unH@8FRNUHd~S
z5m3#RV~AS2V@W(D=LfmmuHZnXAk<5#x&eqdbGia__Rh{l-$GFc`P`-~j1Ux7j$JUG
zAFp>WJvQ`*ZGUJ#p>B^XJ>tDM;TC-TP}WVzS#tz4seN7(E`GY93|
zKoI=YZ?ds;3d@V2$3@Re|Ej7LKSrsEmqopF)z-QdSC@kLQD3JEB$#gq*Vl&)bRN^C
z@#P%Mj?{eDLGvLx^`_%Qe;W}Mbb<8*)NjB7S&}AqI(HVXD@09~ZVT+@nt=^VnO>7j
zhUA|LLyg~HuWw{I
z?ysZ2=s+DryWp`7A7~Yl($p$9?Y3$+-sok|-qN?RaSNsD{t?O4){)?DeB@
z30`pVsWz$l;jh_GcSwtqx+MpD8SXJ3@4QwUl|IKS!<$~y+MNi0Q@^RJ$A4S}1d_30
z{JU(Hw+jC1!$(Qkld5Nd8-!NWzf>#VNZavzGHyhj7{0V<6i!4`p`-87Xo&+*{!+YP
zm-DRO%_KtEJooe@YCg|*{+<^K^`s(jGDj{j#ybPy&;oh1=IN>$#$y}~1{=^`dN+R>
z-^XbfIZXuv0#!WW6ht?DQPrwTk)y=sD$_HbmRgnWp{GU9SIQyAlD=H^A=;&u*eBA+
zQSqY6CscwEC9P7K=2TZ*vX~H~o#4E9Fb!XsK)dRSsi8g)NRd1pWlFuX^;?T@WMfi`@RGu#
zMi7QZX5S*aHP=*KT<kasv%1jw2>yw?9s3SV|Q8)l{hjm
z0#8#m)zN1y#w!Z$;GitdsqeRu8?pdQ>vsCL(wIq*Dhc6~0
zO)^W>l&A7pHktytI$CA!lDvGJuIuE(P?)&J%EfwT``;)2NdZdF
zOMWx#RL{w*boUY{9w*YT&+al6{`U!sl($t&9C?W8tlQS3Td#j&$TpJdG5zcw1xyZs0A
zAA{}h-@g|*2p}BI9qUlhYZN23PO`6M7Pweb#^42VM`9c3&lcm(wS0x);Wn5U2$z(W
z_HtZF`)U-IloVt%6!UXI07YV)kXQU?DV=IZyDDRt&zy#KsD$9ClpQk~iQY+o&9`s4
zuC@ugSP-)z#lvlxD*q7A&qoe6eF{P;L;I&c^PKv9?wgoENu-tuNKM5Nc(Zk|36tX
ze>FTVHdUp?<#vdIA|}yqbgJs+a&k8aZ|~qB@;8N{*89aJdSz(46eB$f!gsw591^z6
zJ4ZrLnUWc;RH;OI8%a$q0u<2P-Z16orak=ouIgi6Xj=+Y5eegY1?p+;wiJK+fo}=)yc3(Xd6+bz!FTut4PmABpfVcX*@z6^x
zf0o7i#pfOXP_+WhMvGu#?Pix3&j%YTAk?0=-)Jc*p}uHK;7#NVAo}O8}uCf%j_FO9w6WDpSt4&o?
z`GUYHK$AWW#xp_Q$c?Ma)+=lv$1*HB)q!99o*Udv=F{Y985!iCKOaT2&NW$Uu~{vl
z3BF!Is#dD!xxBEjNIJN#ah-e{QV6KhDob{s`~~KBdpKF3TBQlZifnpFyyS+pr1|p3
zCw(CgQ6AcVrJpOu#q{HHV)OrfE~<&ekuMNO2ACn_YO`(VgXLjl|Fp_NHS{k&g^n%j
zwt_D=iMQ@|-|!$H{i$XAuo*_Z$|~sW9+{!jRi5TE*^4zhDpxg_E#;rvIyj>U;m0z>
z^b&G>RX5@k6>D_)m1}f~LQzRVDk>^g4tMPuu(G55K#rsv8ylA$#j@GYgHP&>7Rj22
zn_b?#pz=}B49%!cI_~70oY>r4QXoP*Fv3B0EFS;P!rxvV5KTkrf2f+jL-@R>bCkGE
zic^h@=EI&nUfX*3LczioNZ=8mV{e@y1qCad_-kOWN{$!e%k(GU;de2`{DXNq-Tg2D
zwTn9*ApH{XE_YewB6O~`r}%A!7f#HsI_1$`NhSnlh*VoOHz4s?>siS{j&uK*MrU@%u}yD0#0RIHXC#Nazq4q
zt8wH4w&a_dBRh5s5zm$)`VeEMz`GvE&U$J}Ub#vW#@ic3;_u_TXL&GPX!+U6{xG`c
zZ%>bu=Y+g(p*s~dT@XSMQR3p|B?jLMKe260;WpVcQRQf3DwQbKFAIJskEbcJ*lh`|
zwY#DMJ8XK`i!g>GzaB+08=$tG<;rFG@yyRdB)&)z5)#sx95UP2sZ4qhU{=(&!ct5k^Ptc*jG~7k+zAJ3V}ucv4kW1^E)o
z$!s6d#}dasuT^Ty(Zo`a`uopqNh+v+2X}XOZ*e?CA=EpB)KUl2X@jE}I%Qq~Kaiyr
zt+iNy3PH*-%8%|M1m1;S&YMG(7_!YrIHy-A!|>J
zgU?`>hh?Us@FO}q`CURn40D<}*NBL|mKuC(`BAjA*uf1P?AzGTrEsz1%GmkKRCYOV
zwrr@CZ&p|YE-aknqatN!Xcz}ytU=tmXwr$THHZ{=(o4FuBcNRNwaCGsGvhaLVnj%
zuqOlRK*WfWib|L~zGrYyoJ#B0>kSlMJf1nh0`{jsS6`aHDd2;HuXz=pR-0#;?Dr&v
zL!DiGUcBm0Z9^x*vkkPrJFX9&pP$z{{7q@L^?Lr|{I$9?UBnGVVjT)<>U5cPYt#3*
z^TQ&+%{}?&`k2CiKO`lCYRJp(cxDpqs~+sYcy!jll)TW$49_+INOleoK2Oj`^FPjK
zkqNob02(Y}VL@A2Sve;_5?Ww=lmw~x`!^FpxM;xLol{U~Xobl;4wEh^85uYT9(^9S
zLq&$Ht*n@hhtWU|^2No)2t>KV!;&BX9UK&6;_>$OR=Dnj&I8hUhpr%dJ&DuBde_&9K<3YV{r%Iojicleyzp>vLPA14yq8#3
zLXOIQ$M8TvTb-{ojsZ&QdyyiC=@@SBA?TID;Y7C4;be~QEsyI3;pt)x2`f9h32?wn
zK|vwv>mVOLe|lgbS0Ztg%NHIf^@TRm+lJY=X
zGgW)Ni|1&kPtUItjHfd(2ow)npJDi
zk6T}FL@)5`tLsDa5X+@W-BIl{J6R3$a^+mq=6SVBTVeSLkui}m*IFTb_wjNw^WS+9FxNw$`1jh4E7
z1z}JK!$4Mw3zZsDt5-KmKRT9PAntuDQNNwKLaHk$Q0~;)kB^jTp=6W~-oUk|4
zW(w?gy{8szKGx&o6!zZRi=kHEc4<_%;BQ-!tC!2U*W2y^#xSv)W#em#*tW5aPCsoF
zBof`?zvL7Y(;e=&BA4y0;OOW=hWYX^<$l4?%=YBYSL3R6wFY=wWolVBQL?&)!Ds@I&loccOW)OYKgha=&0-F&x3`z7
zH~}kEMS^ZlPlqH~LWR-EYFi`%UoJiW?kX#bP>Pg)&*uHp6x2U$>GWT!_WW=etJ~=TB9OpShXq-T
z2zWn-8BPK#7nh;glJgDZ@P4!Ry|AQYcfLXmRN{ZKP*pLbEzTicdaP9+*MEE2-Shs|
zlR%}+JFbxMEwnUz^mK#dq-Png#VMQvy%BCT;X6Cuk@ib^6d07(DoVi{16(H6+*xJs)T
zv{HT8TaJ@z7C1iD3VJPkZ3bXFEUv|j;{$C<`vt>tTz
z?Iwzvni^9&CLzw2P!%MBIVC7x_C*_#1S@wlu~B#>DYhgW$|a7k`QBk5!DO_2LmD=BZfB1gLkc)o=e+|i!
z;4Wn^0j{U4j_YPZkelSxR1Wl~BSSJj=d5%L-R}E39vtu(UD#I}wX+=AuPI
z7ryot>~`Jg^0=p+nVFgD^mG+2Ih_k%c*^92#Gq02Q=p=6r(LA?cV!GY-liLa8#i0CLK^xEy>dh^_levCQ@B&m{==51AF%+BEZ**!u%>z%(-
zY0zwJdT{i>($fng6;vrkv<`<&p;$t4J(l7(M7pw{gd`+e$4Xnn=)s{{4d(s}Ra!>|eSX|Ik@_h1{iu4yHI*5Q!()hpgpgOeqmlE2
zgbgB40`K&=4+sy32i1%(xN{L&2k??=O4%4PUQDzgLb5^zXWX~8_AKft4aU{i!@-E5*XdUW66_3dFZphOScPg&}x`s6PK%7L~_cH?6f
zV{M$P?J{~`H%=Gy^oJs#`reGtaCtwo6)O6?Y=!j)BlbLOLPp{wPL3&1M8!k0IDN*l
zOo|7@k^aB4WvX8^@g2##D%g>#*vu}>;yxZ-H-R0KQFuaj%yw^Wc1E=Nk5}6*cH0oY
z_}ugV^Mw6tNE_nzSVtt+lEXr}*@Eo%`*<(Y(|b5KathrUZn`k9r;UI9QY
zJUz|zc=rVb{EEvC>{m8Vr@e@Sg9EeUfiw~^pZf1sNQlx9$0^bi=$jL)scItTYhJp&
zA5>iR)MCF|lT7D6OZ72|+~t*KP@+g+7#qi(sDz^QthE|yDWSA@BxhiMN%wI+Tl&gJ
zZ^*jqy|(68`g!ZxsHj+^NH}>)ppGAj0gB$1h$o41VWAq!Mj9FkpAAl2TpY-SHJXRp
zb`8)Me&%HMe|Pmcmr#JLuIl!>sOLW*DkZ#?l%0HgVBju|lm(7SUsME^1x_Cs?!S4e
z5?xmI1)D*WvUIO973WJT5uapxi%L1zWDex2>vx02KM41%!9QI}KVf6qQ)$x7V2WmBbSahZK&Yd3QxFxWnp}~1Z
z=tDeR0@Yu#vWCo)H0?(XjH?(Py?f;%)$a19dNA-EIV3GNUe!CiuD
zfYbe*z3(04j+=k|BZGRYR;~5SvUzt4b;?{9o>Q>8w6;02yC=1gOmnxf^O^hS&$Jed
ziz7I02kj@{mjus!(b?s0ix)+Sm*p4?#VT=7pdz!G^)k0YSvo4w*i|ouRz@;t0lS4$
z9rUP%qS55!<>7bt_Lc?FqDxRd!$~7m2~^N3S2NaC8@D2k&(Qk0C%WVBAC**6Q=Xjd
ziKNm@?s
zyE7`PoDpbp;ac*EyRpR_&4w<24^vzq}tqwOR8ylNS
zqjs07dN(ha)YM9zG+hoWSahtCmCA;M1eAQypx8f(rCo0PKVW^tTat@o1N&Z4+1cj?
zBdbBBw+M4xu+z$YMRJo%a)R@Eu
zPlmFSrRzuvJ
zoFT8pwf+u0jZQ79qQW-2-ovBhCT&T(tU^~!YgMB}9^<@yD}wCX5Q8l!w&6Ow;&985
zaFZn&<-E?5hzBNec-;7Tap7FIMt3UdYc`u;9v&V#LOw*nramCfc3di-$qSt?34OlR
z3AjR3?DAeZsg1Gpl#-HC&J)CWe}DJAS45(Y5bz&qbEYD02
zAP7tUno|U$VkJOK5+y_~*gqI4o{>O6^V2jjr<4sV~nDhR{
znV?pS|0SZWFg&dCISdl)kk#ES{O@nb$6|vUaS$ylYL-CS0#-D#Mxg3oBD8!wja|c!
zP@9BBAJyE#d_2Zdvnt6RfAMhGiPcbi0ep(Rs2t)5`^970i@){x*8;ijkf0fmd-d6T
zmel;(vR}H9ldYnLSPZp0P+|H38*CW?_RIs#CqmwMM5jKYl=vkw%NDw6-#fdP8|Poe
zSD!N&b=Wh78hL`iE+GcWt5=ff67Efid3Ri9Kn}Ntao|r4%#aZO
zhBiBmWA)%*(V09Ux@zRzWYu5z6d@5X1&3d3hJ^Hu7FF#yMvlpX?l8b!W*{B!xW=C-
zdUIBv9^Cv1X}K3CT7jpRi;5mp07Id$DRe2`6m^VELf7;)ds&zcr>7`pQ>o5^zg|T_
zXvYWd!J-q_2^ibO{unH(w(MEr2McfZe3t!78od>}yt0(R3`2o6^A{5&t{zvq9HHY%}|nxJS6*eSwqpGcb=(RI4SueC!H0_*u&k~0)R?3lB403vX1~6
ze_q{}St`$;%!N|Q%3zl$QDZ6>h?2ZM{*?<1?2by;l<1jB@c1P;q#`17^K@QOC~J4|r1
zSg_!#ccZ{M9P9{=ePijN&D1<6|hp@Qm@fmh`=
ze;Nm&!i(oOWfPo8eg6U?Y${e(R6JH=b?{i}UzOG3Qf6vh$R&}c!%bqu*vg~4@5x}+?%G0yS&_&RrUpQBQ8RT7H#FFFbW{en#K_3VYCM^Oeg8R(n5M%W
zM!yOAX&1>sa2g}D?;Xp58=Gk8k7oY7QTt+^83iAo~;&*F%AR;|99{x=u`o#bsBe4!-V2*R!6G+{Q!JXEEHM-rc;of2~V@B8olOOA3vEHh*+!
zMP|!TPJL=Om)I`Vud`zDhyE(wz@%z>1
zEDirAfK`B1Ry%jJurL7>@2u5+p{|d0KMKU`Nhsy$a`3L6xUs{;c-?U&V
z5)D!f-<#%LQ*L4i-|H}GT5aXN)X4*|Kg)WHeMysoCc|8NeQM;f%CyDufFh`oWeQk@
zYGkdrC2Cph5DW|qPiOUgOT2K=RIs4bAW13)+xw>L#B>+mm4%2iyGed1rgA>U>97IkH{j>w8`Re$m`^wEPk(h7nPlUac*3yfn+Xb#1
z;Nh~2NEriTcL9@#C=N7>TU>q{qBk3xRFSR3;e{d7@qgJHwU{YW!q=}Sh2?RH4^>so
z;&P+J#KM9h=#vPCJzl9LYQs)>nb_9*(=Qp4WK5m
zLPM$9j5?bXhm1SjVF1ux;`oa*g^rSvkiwbFp>V*$*GIRj84}{y#Hg{A+N<*=1$MgO
z)m6MaA)iABec~Rl&>C;Cw=^oWWruazLiP0pl$To?m0*X%Ev|3h&W0K{E^dz{(uBU(
z8p-fRl$=fFs|25~)N1G7i`LKd4NCPzS2mZTG8-8=taslGEmkb~IxNv=M90R&h~J+s
zYZ%DLl>;4?>5r@9iR*tD2)L4d9UC1a5SQpI#|Cl%KXGteeG}44_KQLxiW-T-lNyYZ
z6!3=za~D|%G%QN7o;YxAlDX$CoG(izp(Nf7gXTA;cEVe-*}?FDce&43JsMnbfLZny51YreLspR!&GoMn@xQ
zeq5VvZdRg7Z$JX(Ei8`phZ_^?lRWr|NB#@*=BRZRV&O_2_wyeKePZgZjN7}EZ8VJY
z-2yUWUzsU(a5S#3AuO#FO8ek@UoVBgj@DIGRp3@2QPY4|){rKvw2;VbIirr}=gs4b
z?8UX@k)LMj4H1x`Umoeogm98458#oI!khaAC+h03OTWZodg^CzjL*txY$=1Vn;2D&
zV-sTMnD6;_RBv_A86Zh^`3*$GX<iRCM4I+cjw}&cVBM`vQA4>2zi~SmpZp9;iGx!jBc~IE
z11!hZnW_}?1U$ytW!18f`;b5yaB06b3UE$XT3`y?tjZIL~xH=c(uC
z`|r=fKr}|g$44$i^z{0-$;9^83MK&`C;8-81edh`&psFks`(ruf<{NzewCw~UtGWx
zqz%pE{ntWNVsbj`yJJcs3wAdAqcq5omKd-S;pp5G#@7X
zi5i`(5WdbPV`LZMcAfvXiG@p?7&|zy(eC#2;9RrW0P*np03jg}Hv4#Upw!XX2{!e-
zY7`(=86-hZeBb@6HnMPkisLuYXf>u@Zz{sAUJ6xBga3-B}r9#3l#a~^_VS@EVz~d!O&7$Cc#cb!8uFHE~%6xyE
zQmxbbf{;|iKLYgk9Ou{%8i?%R{Gy2K!_0ba-v=X0(YZo;<)a|_pRVt^-nTwtg-52R
zQF?oOlNFy?Dul_?AlhBiARmUm^iO6kU2IOA{=U62J*4Qi;@|B}z#5rxR}lx8yQ0C@
z=$EO!Jd_zt|4jSTHPcjEWzlu3$LZ1+ZIf@@Mbq~tUB}&IA=Hg!65)0iOtWcJ4%XN=W(PwSHQ^m(
z#oZK{X+Hdx(B$Ivh@<$asi}&h8ZzhSK<|r*{HRaA{$LWUa*$StkjxLKC`T%nl9JFqAEBODyWf3{fM$ezxTSK2lauRT!2IkDEBcNP
zj{rf)UD(pERYA|h#uRrdib@*zdUFGL8RU%y-l%A6Cs01iiG7=&z$+^1FOdr?GZXQ>
z0(g6SgHIrl@8Lv$XOa=0aIgOkf?#4=e(|s67I4yxFuX3Mxs5V$&?NqcQ`s?baYLXW
zfFS{;CG2h$I=K%JVRK)=p)fwQ%!Xpth%?Y}`8!lLfgZ6K3Zv{sUR?Ynz;*mUTp4-U#kfcD08+1BlMD^e&;T%OD?QJf
z6@xQuh0M*RCu!goHu=C-TNRZlxld4Ul#@9SMVIq~gai`u09j3-3F-<{9KY+U
zgCOVABrpa(HYy$VmiExDsqu?H3RA3X0+2?DUd?-7UcB%y0_-vl;@<|FX+mToUgczt
z|H%-+1NdcTC`9G=39S>cDrQ95d73zw;e#mH#>N0NH1j{g=Huvw7I=_FK0U9PPwW}
za@2SZ#L(pgZf-+=cPUgU6z0^#;jtB43`I38_HXqxbIJTx&86>wq7*C$QSFRLqfI%Y
zR$mGNUsD(H=LTE98n!whaMDJXFn|*;8zm*ss2#}TINT=v8JQ8k@R}?kPPo^dX0C-{
zStR&20TzYHd=T1Q{QXf(e5_dMAQF%5j~92HX_**acKk}QN=|Y(hc*10(=W_QV}pA?
zB;taaHD+m)q%bcu$0aipv-?niPe+>q6J=QHZ;agB*fTRTVF`M~va;BILczH%e1f4x
z6!seJj&jdm;_#3-&$kD{udjBb`1!90851uD9L4!xBKyQ@v-}@%H!u^l+F98haj%BZ
z#B=eBB{LW`_N&rIj=`5HfEouxow*JiqX{M5egOfRm@hAmj2WkYNS-eE7%A11po)u+
z)$C@71=Zk>pnT7O`=XPP_9j~V7w%z8$zs9M6y$(PGcvmk-Q*ojO--<&_T}#`Bu-QN
z&t5RB#sEz%9^SIk7AJ(jYZw9Yl$0=Jdpo=FJha4_!2$sBn*?pis9}jLLh*TV+*ASD
z+8R8oh?#$2pq-Kkb?Mq-B8gDcR~Dn)n}bP0&YD2CzohJ*^q(jmE|AW*(5!72bDC|{
z)+OHevD`%i1L;|QPbt578~C)zerw=mZ{pGXu+AOa-&da{Wn)wPh7C+kMjtJJ@6i*LKLQE(Jz)7g
zEAL=VB&jg$b|y8#_Qpc>>Y-Ko1VObLj8QN5Jeit$eWE=RjAKlKw6u3}{?D8Y(VsnW
zN&EFkzzGX3B*gGvn=4xzO);kn@f%`b5CeUw24d{I&tKTs6D06{ewk9{?pmW2JOhQ{
zAOm4%E^+MjUMQnZSn7*yYfs!Uhl^elup#;(n@e3LS{q@?8b?ryQ$*uruK
z;=EU)YMwmT+xL2U?Vs}mkdX4m&02JbmTubNLgd76lsWy`UH16<-&~Xf)Lz*oaksc!#1ZcFi5z-YuKnyl)PV0`A?Ti+$-Yn
zLSc}A(P;L1Ym?vaQaf_pC1qhhZv(fB>K7~2hGu8g<~WqXzY*1LSy?HCBrq#Q;YI(a
zWM7b(P_}}Rv}+Yra_^oDN?m>ZdT=<;)-;?PJ;}g56kbR^!NoxFb_(8k9*jh~7ey)6
zqI=O{)(4c9+WFp0WGKj1MOb=PX~!N+W>WF<<4Z_L5OS;YDX77rT-Jd~y*IQ)0
z68ivvN+zXx0-=lL)Y>e+njG6$W(kn>X$R&Nnf8+N)$6r`j*k-swEyTyyCm?rIqs({q)7=-~aY#Ph
zQxk*m`Rg#p548Szy3EvwZ$g>UKRb)Y=lW;(aQYjMb?$@5hbWJl%N)?#UrO?L>t+Bd
zb&-TimaAH4BDq5hW1gz`+WG>kQDp}^ITeyvfYR=|lXv?RF;^62{~+U?@Qigu{#kC(
zVuune-BGi7Yqb{+Xuy|VL4&$CS0n|AKgO`#^%s~VR{o$%#0!9hwKOUv`|lIZ|EL&^
z(D365`CLlJ;j+w?Dqw+1z9=|p-yBc)KJ?ipAK~?E@C5Ld5zgkiaayzO;Jo(_{`fYT)58L7uHZM58UX
z{fPCv+$8w@!)pkX>ikUd0PM0vVrIb#M3Uci-6YT`c2MEv9R>y8ubjpir+(B+Fr3;pl3Cjx|5sqwpYq@N9WxHNdNTdCdDpzO!$IOcT
zzw(%F$zuWe?DFwSR;G_LetJ46^;c#N30!)9n+tLO{*%r}xI_yHRdr^4e`;w&N$sDU3M
z4K^qZb}25K(OwAe5!=VSZf^ZsA(>ghUi;%1*4Ea?G9XIEtz|L)Ge~^>Z7)v5br{}o
z_~pf+<54Gr@$NIrC)>Ll@lYsCJh;GHlrsJMV+k)(k3;p-<&#C~GA9?->Z(t>2lJUR
zH!)!#`uU9fIbtbB%&;YDSbBSjp+~6?k{+jAt)K~~Sg+Cm!bW8}*|Wwj`*XtW?d+Y$
zp>js_>0$-i5ATaunK;~Q`k2)H1C5@sD3lfi>V7dKIf#D@1M|+1kapif#*;18Hab1%
z)?4huVNverSfnXa3QtR`ef_Rz+_B=mW~9s8!5tl{oFWxz1O@_$_~hwymC(r-i>5%O
z$1HIysp_Fcm1N!E>`z{9Qoy5Kbge4M^hPBQ!Oi
zsC{~Lc{$4BR&a{D!+S5Kr>amZ7>z){ngeOatZ&~+WD^L%Z-r1&QX=LNfoim9&`bZq
z0wymn@72{+|J&=MOoL(4^<#yJx6-7gnunWPXnlQsRQv*Q#+WuNK=?7l^xV&ReJ;Us
zzPL%GQm0&v%upv&J-$vM-AlZw`R5lYC_ceyC
zh?#6=qt|<5^B|Z2^w%1-gVG6Hi~S<38Hib$nwlEy<|(P^==M&Rzk~f=iyEmYDd*ap
zjUu9>OH{t`=Dgee7`s0cpZwZ_NzAYA0tdY_-q_y~u9iYl8#DuRxQ8OCbz1KzsuJvANr?l0Qr4;iIcm*25s1@)`i
z(L&^(s3|BQGMMy&k9y+ck-6>W0Mj`eK
zfe*iI^&2f^iW;Bse#B4ZDWR=Ys7Uj
zl2#g|X(Efh>qBAnB3X(1^2;zmLekT`2Pvz>c~dI->r|@y8^NM1%>B(Bf&hYaAwt2?
zEQF)uKop5+AV2s(0}ci{;OUFG@m-J
zK!i#ne1W;|$zR0_d4yMd)pzR|P>JOP-gSBM;wt+0FAy3AW?*CEHQ^Ql(CV-(Ih!xi
z81fMui3(qVJbuuULoCo_EJOw{ztnzlI8YNSJKC@<$Z+1N!PNP_FcI+Pzg?1~*edD!
za83Yr^8VZRVQSF=5>0{f1pa;RilWtD5Xz)!Xdz90s7nuy$J`-}Ch0jKqRMq4AeSZ=
z0B7V}Tv%0e`9nKBju3gvt2;V1v~Lep`sYDxqisEb?cb)74Jj>KT`D_}mz3&v1>}ru
zZjw)Fu-9s2VHCT{>I4B!DLEx2q^hba0GO49Z*6N^0?KBQGtPgy1wa%Yy}4N&7bQ?e
z&Y!h_q^RDv+a31>!8tZxSfMMDGdqg*;kqc=USGdTHB+$+d;ruYvoTb`kZ@V=?o_6X
zZYvO~qeH!5;?9$tHcGQlQWD5zYu6qpL`K|0O&e9+45QnJh=3qrXh;a_pt0`wc=$Zh^-->p7xSbGBP4Tj0gaj_SW6B
zwzjf(*45Us;9JSF76}klK>vFS{oxg=nggCC-CW9Yv99Qxd!7SV|a&
z5?8ZA?+Z=561_TOg5cobwKiv>nS2qW^Umkz{zj|)I6;2(#vW-%r$5TsbnoxteZjyY
z+~EMP)<5AtvmfYT-D0UZI50eDRfomt=#Wp>Fra8KrvyCVzr^04X6#?>3{~pZ5`qLD
z&01EqrGujC_!q3*%1s2-aM+fBCYuCZ2KZAf3=HifjYdj_=ha%KHkkvoOJLPAfcPm-Y{>Z7BT_?!qxc_0;y3uCA}aaoLTQM#prfpkdUyD
z7iI;&{J{U71(-q*2z+9PMaEaJ*^b2T=!7_$U5pAh-9AHVafAa&B9UAEFbqt{yK@EA
zvFF=#nyGnV!v?y_E@p!!=*h1vU|PAJmm}-(+nSj{1th0ycg$;OYGP7jA~-oY#S*Ws
zuHxZrX(8R*+=OMa)|oC1q)<_7jW5=xkItbtw9-7U(J0A7>@P963wvJ_2RdM3t?PJt
zlw_u*IlPu>K+Dvi?B
z9Rmyd`@ghY+JJx?sj-BT!GVRfv5$B~H5W@(7vvuB4XsMyrq6Fgasm4PZA%RIKa2fU
zfZ&JH$Hju`>+}FGYi1nABqWSnY_zvFMa?hcWQiCJaqgWC&m~*;thd2@Wr6z2inBr4
zv;E{|eypykBIMiP`Ar)F0s`Kbt>r;crpg)>Fsczd5{Jk2MSAcntBK*UzNslG7Z;Zm
zNae73PEAiI@VIR1?__~0H!=VT4GnM8v}bTAym6&APUPTZv7*LtwT_vSQ&URzg!J+<
zb}Ms|ATHMOe2k31!^5gZ5@r-ITCHO>wehVH2pBc(rr$80EfBL6Ia{?Y0;FjH!hZsA
z5gYRF&zd75A}H(g_*^Nt*x4=f1Z6m@(mAlOMr9MNndxI1H#eJp4KRqD@GThF<*DOp
z3Pkk$xKecT;HP2lpTC~oT6tw7=!fMOtQL+oX0-)1mRUr^1P~V~!)jaOpfh^9!m{CI
zkkq7qdZ2xePXlchGM!aKVYOO^z?o~zif2u1Vm(Vb8Xu4Ux(zkT$)!US_kZh)VuR<2
zCT)uR8mLIe3Xd%=rU~%#H?*|0@ckVb8R4+9vf>SS4~$&BV)JB>SGAd1%WBm59xHNj
zR^Mj>`o}CFU?3G`k0vnduJ7;p6Dfj3NRiLfhSrKsm7UiS*J?UvK1Y_)q9!O{6qnhY
znZLtY&b+Oi4~fOn%@e}Bxu<7KKuoNlF^!IZh%b?YtJw*bp93k5_xy3o;}=seuC5SU
zVz2qY*8%1lKm1AmWSTaCT+Qu2l_`UL^9vO*LC<8|WVo#~&
ziO^A0baXQaOW8dcx(4z
zY=iwpgkwu*zhA{_toQ;vyrc`18ah?vOJ
z*-9BvUkZj*qRoh$IAI~i?Ung?eINZEkc!l>2WyI9YNa+lJF6i=w^o04z4ce*YKRt0
zh!LKmhHaS|_EMYkro2RW3KhZR4C{Li5tU0r$i~0#!F$>iRN;d{zq_<5xz|*Sgmq|go&*xIGU&d+LM9&*7dJ{NlXYiNO>DW)
zVqNaH>nSA_C)Ag&$i)vf6kXl^WQzVDUQ2KnGbEZzoNX7tIm)_Dc=G{sIE=>z1>%SJ
z_3(l4lY%TN((-%d5#CTO!kZg;p^Xh9d=bKjqN26uzkO>*7bre*a>x!#@Rx~+pkE1j
zMMa1ihj%HE8C_^%)U6Lta6mNCLfKLo3$X>#6O5Db?#wCzs*HiD_#A#1IHtH!Y!ipC3=
zSZlO00EumbUiB~ef>W*Ut7{{pc-hQq&%Zf~p57jzSB7guk^EeJc%jV`%e~ezbfN6N
zI}F2*=8NHN7=r-L?;*a?#hRB0~ExN*hl*>T&W>DgwuNm9Ed
z-mbk|M*agWP$J|q04YfD-JWdaNV{0x)HEp#wyN|-=Zz&kG_(3D)Fj|%&+ucr
zQ(-;jze(rkPPzPUkX$xXz|hc8*n~yZ(_k$;gZAdDe<8CYyf9i7>r_t?^zZ&XLK`c&
z-nJh%52KwO>@BhI#k7&OGwMSrCSBtyJD>Ugg68CXR7U_Fe*kWlsL91fmqC~}%`!79
zYk#Lle>mLI^UKcA;1PZnL1AhD4fVO=T!WY5_7R-E5!7NiLr@Yo56=i_HiFf|J)^W%
zOTv_LZ2!kSo=o3p65a--svCHfh3M%OikElk<_A3ZG`Wz^W&hZiTzHf{+gE26i@)|s
zYtN0zpT>uI;T|3y^gBNxp~}e=`PXmo6XQf?7`3T8#7IN>`lpth>`H!!SVCWQ`yPGK
zAo*q^3d~QwQA1pyTo~pJFG2pstKlH=&7nZaq{SZU
z-z{2lo=RWSH$J2SEkET?St~q|%u4?PJ~iqaX(gV`moT;l`G8d7LYWds08$saNhBwa()B68Qck2oToWoWG|$FUbr=?MG>q)V{(i9f8SM7E
zMJFb%5w*Gp-VvR|9OH+|F_9Hr@_Ef$8(}r7?52OMPz%i$37!5Hib4e33So1jl9h*q
zg4#Jegf%iUN|>%Qw@ek$juI`bm&j}3to}N=o=kXS)TjD+
z2XZ+;*jM~iE{)F7^JQ&K-?-y5M1FogVV+*z+Is5Ve2)m|iWK?nlanvq-NGpe%cMV8LI>S?6
zB<=~8=XtwXn2n5LopCF;*=18u@$d>IpsF8_*EIHQoYop3R#sNr_kSuJg3Z@Q`E
zAWLL56e8;y-Tw$PtH}eofRG4~3;`y1lv_90Rh|J>&FqEPw0Nbw-N0hmC&0h
zyu5TYjaxTyD*S?jPUozB-~nmOo`AoC1Su)vPRt~kiVEC?GD!Waj~AGQF24sqc#XrQ
zeRkAn6CYA@6y0t{;&4cVpROouW^%ung}mV`f7e^BaTZKumMRyT(En>?3+J?@I-hL6
zD9_+HO{n}$5NCdVUcb$WU^IcKBD1UQlY|3xL*DXJ9t=nPsx3NwPKX}`^y;{vtbK-
z$j2LBO>M1`mx=}|ycPAreQyZ{4$S`koB%57z8x}wq^fKjuJ~d(u&PS)(8Ck{68O0#
z^X-*^#Ny5_G8&IrA2>Isu5VyqzsCNX_RUbR?x8pKP-5ER!uSVVye_9NI}LQm1A#)?
z<$5Gp$Scb2(Jb5J44hoFIr7uJ-uCURf}PcoYHA5?HV%_6d4+j~K5CX=#slwYJVa_*
zXo2BYbaps~!&D7Ln?0i2`5Z)Mamh~2_8JW!(ao*DNQ(NG5o`{hYhp(SzjD4X0r>K3
zX?mivEUW94
zvHVs{a(6MQpG1m?X3`s&4Pkz^TEAMO;tyn28y(CW`W2ajr|%uuwsr_VR3sIz^L;c6
z1k&GlIcPKEoKw~nP}Dv#e3ZvH!^+`sauE=T<#PZVXGN`;+0>lkCPXS!IczKj)_hRg
z4H_=CnAvoMK2$ID5wB(`!bv%#L_*#QSJd5en+Mk~I-^U=LDZ$S_Z)fTWn_YQNT|Qf*m8?c4QP*z{f>BnKxj@#HX)es?jHU5$}JNad%K^Y#Ql
z+0ForJRB*645GeG7Q{HGN`emsadBh6cK7UD9>+)|5DxsF#{m6myxj`12-$^R#<XKLUBzp*nH~h`WF!(ac!04uj$&y>I0WkDO7UO?8+hX2uc=5Ou4oi
zR>fJy1(dMSXk5BIf>vL@#=l*|S-A@+$;*Q!;4grj+~J=}45IDhahreukWPkftYzVO
zK6z$-IMet3b1sb3h-Q(rDq_x?IN=Z4~7F7cL#z$ae?)Hr&
z^?lkU%F!0PR9iT3JnK7CBwe|(1v5{pcILtUe#1yPDGcNmYR>o6MXjZk?RY7ljc+vYjALlq}|9N
z9}t6Zo7^4dmE(s+F0>nu!2l+qVfsHr*uFkot^LrUBl7;~vG04cfo1IAiGbD@5OZ3f
zEjQLhq7h{1^+#QO)jXb(B^WVXw_}L1G5j4cpX#1lp90wwPHdJq5MMQ%_I(4ZA
zfmjH(#!Rg!?r@r6SPOD=bQFC57#SHE{8E7Rr%$0AsXZ6JzopMnQ36>_xUUHK{SiX^
z-CB}_e2xcd$lp%m*m}hc4GCB3OpH!<`Bmq&Neu|RdaJ_WkjS>@9Yhojs`XP|=!r#Q
zGd|wq4&2--;R_3+f7EfGTXIHC*0Swg()nmK+RAu#EJNJuEkzgcs4dS6aB`jGgXBF<
zE}4kpf)a6UnM*X{=PZ33DR;7>kfeVLH5;v1U!bq9i>j}Fc>N5|;s|FPiz@St$MJ_Jxe~KMhKjbt<+-uWPg*POews)`C<93h!+_@Tm)DEp$u&fy-lZ%
zExjCNl2h1J@swJVyO5Hc4AjUIvhq_#{wJPTM=Y)nA{`Wz#@i)^wZ~(Mri@qwL_|q_
zef(Zf&EVuITJ&LDsstL|Iuw08>vhbBS`5P@1b>iZamqMNrb{!zh(1SDkM#6ZHGZ#W
zYjDaQJS5s^hY}UlnQx##Cq?D9Q}>kzwK+Ys!rZMTPAL#}#-|gCP;H;3r#!q4&|Az9=
zV3yvic?}DkVLnJOl?zmVDQ^YIy?zo@3Wx1~zSUm3-f-29^E_GjHY6a>FGmKkH>x3E
zVpnN-6h5SZ`(HPPgcKU`h%5OZ2>gh)ibkdPtx<(&K_Pm4>gS+^)$hiaS6Qw!
zIY~L`;JLKkQPMxt5|5YKvhBfd9~)h_9hDPrXxjzZ
z7G~1;;{7=4>gpa_8FSj3e8yZMhGJZq0n3`oixbA09jgTfhK7H-+tVj_7K~XV%Mn;(
zh+$f7BIcv#%9r^UhhH`;r}g_Cvcy;PT`+4mbbTa
zE8PA}V`K_v)QEKaYdm!QGUjIgj|RR~JkFVA?pfBBJ9iQWr!H7gT3#Bue*B~ZqJ+#BV$&2*3`SZyX-+Po`u0d;A8lcnPu30
zAs<59XH1_k>-M$I{-p=z-iSr+htLff;$<))&mLF?vZyjsnL`b8>R>bTxzqwnT1m`Mth2Qsq6G0QGPQQ(RnXQ&Q;9n>0QS
zh(b>B_m&M>L(H$DBR6V-*6+kT-X!1;>0R18oXLv=KUgu5!34|Ncx|UmxQWmc{ZT(6r~)}BcYHdLQ&3b7?qZn6Lwzxu$j)G0Y7!2si_%n
zlJ39)`X6+36jRfYzOT*XQIr0ZveIIPDqoauPO=X@IpayFce!~!@p^<5^
zo0sh9>WYiQV$FU9#zXoU#W0d0u7M~D5VQTHMuNYSl>L|
zF)c@tDcn60Oct7gp3Bg`ewFk*CLgXf`gES2NjE$gDLHWYB^@7zc(wk|Fa{1WP!62R2H^1J>73JM)d^Bbe5L4rIqIWsom
ze4DdTN?MxLxuqrpXYe#Solya{MhvFyYaR}WszzB;-${gMrNx1sw)16i>uBn5NAWL;~>o~AgPg;qovl}yw
z;pR3qQcSu-AyxW%fAtxps|#Q$Jr2`5T=}YVi7CpJQzrPF*F9hr=v3k7yF0NPBcfuL
zN=u8yyZYMOD;=F#n5U*?q@^QFCq-RvtTEn>M25`F#vnLAKYD_Li`G5+%Grp4%CZ{k
z{?FNfKTzs=B@#zA)8;0Mn$(=4W8!k2F=v3ji(qyS?r;N>#c%pXIPUK5$?56PfVq~x
zVN+<{dxzgs_=AFRi^mu~AJj@$evFH!m8ldq9928>%v@|}4z`UZl(p(rs6e^=PCZ2Q
z^CRf;)RAP(9NOI6TpOwU>J;caxBCrOej_EyvAG^++MuN+{V?j^%XjXw^8~(9>)`iy
z@!9--zAZp!=P1+T{n>&;*Urwu%@4Cc4ay_e6G1P?nOq3)Yf$p=qrCQE8b)pAu9A?b
z#(jxKot<_D>cE|$s2#J97rkX1eR%t*5C_K3x3rkSuPUxeCP);8!Pfpv%gfq4w=<#2
zrh`Az;8`BF&>-$$l99;~#m7hDWEt>8$g+)LSIzZ0U{^Ppe=B`u{_jk@^G4gisSXo%
zDR%$VJf`ZgU@`u8FZu5hyo
zRd@uH$hf$;#yHy?Osuj`+3hYF?o`ai@KZT3`~n1B*4CEBvOoHdkjRYvZV{(UrXaZI
z{kF#7;8DOEa>4&Fo^F)P7pUzM;uO+KolaSd2{5=L%#UAHl`_3*5f85-&?NWxEMbO;jC~gHeCiTz$M$
zaHM{HR?sN;e#W(mp)00K<>knDS=$GPgof1ZSy=qRen*K9OxB=BM2sa&VFpM5T%-_1
z|5@am9cYcD&>$&+>E5+Od$hM*7X*-s~S)6yxZcRsV67AxHVugEIv#
z8!i-1Y;h-<55znKdPUqc48ie{?Rlyujy$RwE-0nQ)g;Dr^z*v~?Rcf&rqJG89w5}N
zG%(rQ-|$F!8PI?ii7y&9DBT>2`F}sR@K*~na>JCdc^B$_2eCa*oSZ7g*r5{F+CI-l$5qh)@0ReT1oyNir@o>TjU;cOJfE(!;5}}CWRkdn=R8+~ST!!IJ4;`bnbZq;uZe!NrYMl(j
zsrqqrvI5PGjKS69D3mf7VN9&ZjsQKw{-W4JLJ_?#U3GF6nKq9Kbh04o;SM*+78jO?
z$Vl}e%}u#veyfM<7|C>g2S0J+zqfhgZx4Nvn_YUADJgx7#@5AM6}HoTW{X_e=*n&8
zj5E0c;SU!ZRJ^>n7wfHD-fq>2x=6&r^a*)?+R++XM*`8%;(r8oRwC(b+qXM#15MKC
zsLs#o6E1%Swf*l|04>vMPD%KE>V5J(b68T6fQXu7=6yP)FSS#rIYp|Ef_6QRBe!Vq
zwZJ*V@#%^DnXf^#L=FfhpU
zp&7QL25@mr25RyTuj0L8Vlh2FFK3|l&x$D}MI@2I8MdaTs>8xT(Pl9xEB~MNt~02q
zu3HB}moCzzg(gL+f^;x61r_PNgEWyQ#n6Iu5di^_-lZ9e5N*XF>Ak+k{eSYcO^1?j^Z{b~20I
zM@9Oal7?yx+l#5qV-*9z7j#!HbLg%-$K|!O(&egUo!^XUASSl076=F!qbtjlVQ_G8
zpr6F@>jBX0jSVw(KrDzk}Q%`n9>{wXM9uhkb_6-R}j1np*Wju~F?(`9mJV2t&v
ztMj(JDJb&ZG1D`n4lDD8J{$GgW!t^`s)tKMa+j#Yew2=2K@;G98glt?=vBOoJx
zpfu-wHR+k(2FB!vsV|*)qj>ptMG&@3?E0Fqp2XTmwfq(kh?EDMp`g4tr_oXYCsl_5
zcY7W!4|n0-E)srLS)?7!RRVkqO~ZYYT%3VH&*zZ?xG!@u1HX+uc|=fCYh{m;V(Z@1vy=RyJFr;BJB(N=hCSFbaFClC6QcEyFNBlc+wH7f-t=bf@69|t)D
zvy94^VECf3frX9{V=YVqo+V3<)**nX2?s7E0&U%qH}#5H8WxH~z1$r!9*)~U
z=JAgTCTV-0&`2Nb!Ak8L5*4-
zPq*Q_p)Cp#aJsFEhliK-#Fjfd;WklFS}W67RiRR~rW<24&D&FkSW=?~MqgIh=cJ?x
z#ki1=Fz{H_n7n=brXwWE{rmf5ny2?NdWNkFM^KL_Ei8llO=}ZaO@>Mik#oX)dkI=Z
zD}qbwhCfHKP6H>4ivz+TRnD0NTtW?a5)ui2{1_~
zZu_gcP=V@bMln&ZxWeZ{jt-SG?a|HWXIk3Amd|NyDqSv~Hf1a3TUZbAIN+~*OU<_n
zJlFmUjaExiw8#7Kzy{&(A~oQZk2rJ}lrmc8_n@}-kzT5}LNI}pbv{29SN}=r&CGYWWg-0fYNpwQu{(Izo7{3+F5V1lnwhac
ziO!ek1ldGIUw;lKrbO!aEEgmv)0mJlll=I^flr%yOZ`jb1U4QwZhJ2C;Pp*=Seg`<
zr_pMBRFryXxZ!Y-Hetbo^p?E4@s#xR5l|zYg|5Ua8;RQTS{wkT2M*tHoGm>q92^$*
z77uqXr@lXWGt~3O+1`#rPtViE60HScSoVwVZ;5YM&rUsN=I^e&jVwhp)MsgWy_}qSge$-lcn38L#&VPOXi<
z3dG-X-%nxV<0F9@Gtmkdg`AxnLS6iphf791n1w2FulZ?YvE{?6FwV0g-{4)J)rMAd
z2t`8!6yZ$|(p}jR*+Wyh#S&A*1G{y8hO2~AACzk)m#10BKB}UujJl$AkVstUPR^Hb
z3aE#vK?6!Gn4)-F4`UPx6@3BUq?r3gp!>8%3TNfg{YP+ykd?=X)rVAOI{1Ttu*+K3m!FKwvhK^L`emiS8$bU`3C-{PoJPl
z80RO`7rzQJ1E4Lc(Waco@yQJg?$jQXlY+Ee*=je<2IxJdhKi1!_2l*5g)RRyJ|0xR
zm*9TTa9mvJf9kVh`+|l9e=6X_8LB4%^%TX$#oanyE91C%lSVE7!w+j)cLpQBL0rh0
zIPS^t{wR5{ZMMLdp{Fg)V7qP3mngD>rYUuh{obcRy1h*WKRa!${~E6h0`f}5fJVy6
z>P!7~)s-VvA3ste5J;0cy+z3zH*U10NqU4JvnlVVieHI~izBFIWS{Hq>7ii_ILdZ)
zbp;XWo1Dhr;G^xnQ}NJePyUAUCo9v2WV<~wINwrMet&fGJ&*TT>LsS5qpNV4mb0b{
zRm|2LNA())37-1x9$C{b!N7NM)703Wo!Y4v4-47rj;L#55pI(}{!
z%3VhCI&qnJ7p=qm0C@`$mJwpM;V2S)b?!tncZQmUoRbSj7Ii1SD>yj6veb60(O()`
z;9jMl2t>+q)z$nw(s{$t^yjfyQM>{ei7J#jGOud#B_OM8&
z9CHtsctARv%4cg1{$tvQH91@~+x21AejjG`J7ztUx$DML9!e_Bw$LH?@5=%otUkmt
zCq;L-#l$hEMyG`^%}2_N@&2Cq{tZX&sy|`0lVXrQV$GUuYKA%)^a#?^vTffxP|5y%WU<{f!&
z7+fgr?Yi+p^62E`YMl>Ccgn!QpF+0e^(UcWreKP}YucSNq67GX9Hz0p9CTmJudUfG
zaBFMF8lpc6Wrp@>5!8)%xh`*@!*&^4vQy#jQw{ZfUd!p4uj7ay!d?|-{z>@o%wxUo
z>D|o*wqmjmA3s7>LhhwWRI~?v{XqA;a5ew;-QauN!O=}*?`1Kb+@^GkI3k?;9AZ09
z7FO!7Up35`{SmgnQt(3m^F#-uyT`LnCs(eSS>TKKTT$Fus36P}r$QQaY#vSrS$AaB
zcwtVtkQXVWWba0aD0S%$4i0>NV@F?_k_KA{;b{)n?QS#MWF|by?8eNOJ6hp)6TB&Z
zY!TVJFR-=j&Bau!??;qGi95k0Ff{IHP-Bo#momiabgr;7`qjt%ka&n9vy|uB$Ar(t
zkyjGnU71ScGodIHgea}8O4kj_VWzdN%*82aoICg5_1|$iVwL8`XKuO>6iC8>H8y@9
z8&jBHlJ~&UWAvilzNLRW-N+1VK*DpC=6#o8Xjp{VJufeDXcH^(Wp165dNPz@P*iME
zQaDuq+McZV4
zRWXT?P`J}Qj^xK@{VwyfVE0>Z`4cTK7P9W+H?BmkXj{0tfmj6}%CN>rfqQiN%U1wa
zmWLZZmnX3NP<|QyC0dJSG>#veQM3rG=q`&q!Gkm(Mpq?O^~s)Z`J~E#bgcdHUL&Zf
zX?E$NN#o(L-m-+f+1Pk1>A?+A52*I}lP6Dz$jQUh@~vRVJVLwPw9l%n{vpayvieoe*kRu{p&|5{lH&*HD7RCMJf0P-ZUGrSUUBQ4$zb
z>jm=g&`r&bj*d<)PDN_u%+~A|pTnQtZaedL@$m6yGcQiBot~b)G`-25ONpFW?$3qh
zd$2i8AuTNpapsJS3<7@m5?Y4^qGkr!8aNN(20DHp-sxT9y1l7O|KD@b
z$;PRKQ<7Qe*Le|nc)(w=60*qQ(U4?0PZr>)9&VuPd?P)Ih8=%2`d=L_H646Lj41cs
zG*dKZq|()T@PPbveEjUfg6(=vERM~iM^LUc2*70*`!Zv02NP^tQpQDEz_gx$WdhFf
zh)0k-#`jQVqtc<%ah$retRC0*R<&r%_}~5qhfN$}Vpk^Q^(&xS?qGFN#GOgZ&CN?~
zly%6)$HruC96@YmIBX5GdS0t&l#m01g`m%bnqW<*xC$h~&x3=zWkxzW2nP!^
z!c$X?HNQidFTFq6itz!yAwrrj&ZaMBgDK?D0M{}z%{7I02GoVJtvg=%!7On_MMY<3
zW*$D7?3oEJ+|km~+FI`CsNZhC`c}%54@y5fKRd|{yf}vhGv6m^lU__4Zjl_@BG}*!
z^HF?3F2B3g+!Ab7m9R)HEoydlc2_{JlvGsMrKPX-_V&^X
zTfhL%*2-20Q7q?+k!9ny$Ia-)Ss5*3ZI8ZcH&0Ea>B*kd-CzIAfy?sPfy$2rV!ms4*tA`bp*ihA~_i27oCe
zrKXphoSZ71Cuv>kw+SGm1L(c=D7>??=i4_Z+iP!qd^h=o1X2EYTogFOT*YwmVY$2S
zi!6X9ES@G4&Y##V;2_wGtq94PU@#fKBg$JYQ`b>HYuw=ANr*}=@8Qv0N6g%yhJ0vA
z34dBznwr0u^Tg1+ug<{WAQb*^a&aMed5I4q
zx(FE6}jmjEq+RVR8-y1O(!Mif%~CogZ1Zc65AcZzm8J7Z(OL^XZdhPfw4<
zqeo%jBvzfV2wRp+;BLg6Mig@U6)<{qXFY&@1Eba9yMja_zZdJ4i1uxNp^%66B*5|t
z@&UrTdV7b5hZCaqmbXoAv60F}6wBvp8SZT%#KmbVDk^|BihETQ%M!4m_Kvo#p}J#}
z)h?W_%n%hAY`TLuQzA9x87o0=N+@+GN_jSZW$^ht69>-hM1
zjqjl#n1~Kv;seTTy%Lc}J#XFNlvtx^tkY-<5Fbfs&T>I$DE?z-XG?4AU?6$-Zm8xs
zRPY{5^dVm&len@@_ouo#>#dp3^fEre&=s;^LJG#ZJHw1{N?Arpcg_cya?t?)z5v?>
z@Yi;5N#NCXy*T$zlW=Rf!^QD>D=LLF)H51-6rwC(IoIff
z%&vQWYHPV019YaHhZdkXRh*^>tUrEl@2P~OWa5}vjexA|&+p3Y&=8d(U{ivIjj##6
zx<-PUZBYXg0|SHdX41kyzB-DZPBS|k(AgVtjGDR@rW}fv;4$NGv;duP1Y^u0Bg15b
z(X(Ue4824Lg9#W{U3a@@`MM#MK_-A#vNx3@AFmzanT0@IN2a45xltk%;hIT>x3{r3~x))WyeMLK--hf}^s
zv*E0l#+qtl#We$$g4Zzr#92YtPhtvio
zovy49e=>R1+}tcGpZVM9w`EUlyBEf*XtmaBq3cRdx-^|o!yYxT^h*FLS!7WbOTFp&
z8S%eU1PRy@_hvrB{r)T@@$m38M^JIVvWAEkH|uxFr=Be_L$2)P;P;oh%|_^EOt#B(
zBiEAyz4rz|k{*j}5)$;`12%?$NAA5Mg2|}zpI-YuLpv>
z$_X1KvSx$;e-(8{u>eMAmofv}0n>q!;T}&`kwJN+(7ym^(>*?z@`q;e?Sh|!i2vFO
z)X}qQy1`eUhlWJN_uz5*@7dW|BzJtIP}8Z_5)*KIzcAHP1LNKcU5QA+q^OXPkR0c=
z(K0%ysbp&_jk9*iaLe?x`BT9B_=KYGz06*l0IMNjBbp4n&+wPwSl*HswFmm==jZIg
z!sSDUW2M7pVWm!wM;*(G3!MOy%Fpoqo^`urWJGU%X^GC+x=xSUSD`qT$678Bu%t>O
z0iec(AN|2Gu&hB-@0n&CnP)qAzzz0~dv_)9QX(w?b5foJ_8h(VEj1Setjd!h-sXOg
z77>3U$e8AkN!<
z7TQ5Oo#aU9Ka52JpbJY*W=K%uQi*14lhTXQ%`4T#n-n|dhu(cWTKG_R2wPalQ&U^p
zGCYj6zdDJ1O^>EWH#Nz_pmrcqlQFFFB>O)n
z+<*VdD<;Y3ZBcY$}8Vc88N`z(aub!31Q;5@iFNLmLvU875~h`0TNQW_0Mz|*
zu4+vCd&7^U2CJ)))06}sfL21h9>7vieEen9;YKuI`cTUE{@SR}ghr9hudngfI=j31
z-ChCD#FMC&?jIE+a=p=f_C)7oIkzdOrAa
zA+1qz7MX}R4%HkmUkJG(564RrTUjXKZfd
z&2N1Pl`Y^l0DaZ&KbW7l$tx(xwe5bRJP9?$+Fj~PI(pwyI``{D>vwictzL$mfZJWM
z(cKa33yV6H{C-taeu_!#L`jb4@0riR^m2X@xNvxd$I^qh$%*yYM}IY&vzHL%SK!+C(#2?ESWF<%P-$SArpGCg9`HlkjIC0c6H&Rb9OyfhlryY#c-RwFyf+mrT{QbbZkwf
zz1Hc&C|Jq8y}hrS)KCJ3*Z=2wU|`_bp=bPU8;*+
z`3sH&wS2&eX@AywUT!+wiJB@+eA#W_=?tI{f<&N0s&4_iI^JK4g35rD=$EbTqrUnS
z*OuPTkTG0h3SCO*UidYAabDfq2=I{38nRHVhVBTKLjJV8_1TNeP2jbpPCl9ra$SDT
z;1dCjPGJeckiXAJ!*GZoB0|ML`oEs`e|ZO~!oQBTphr{wWhvjPA|i@8pp)yo`<8m^3*dl&)T=}uKKhn|
zpQsjkak?ZAgsQ^BOhu)u3VnCsz5Uf;5Z+yNUH*;$gsjN^q|~%->!m;F2Lv>l$pb1!8vb_*PJltu-+ICX1*w3%&V3Fhd@JYww{H)KQ)U4vp_h8f1Az5C
z2Gd_^D*br6(R7K4f<>koc-NWDO$xy1hD!7){EzoO1Lybjpw4rRcHUjGrKP3Z?~l6=
z3f0opMFL<_eCEn!syMl1hB_FK)-iM?2n6Xf`=jRJA3xpzh6X7+fM`5?{A|D;#(>vl
zVq&6XVS&0dvjh4c&wU@4?He5qnoIpnGoW95i6nJizgvLKyfhE^gO7uQBb8bGIRP%9UWCRLW0J{(Pvp@&1=grakY{hWVBR2NpY)F~*^r0UGC*z+1*w3@ci-?HIKRkWO+>yZGyUK;rvgb1
z%mJc~GBPp{V^vQQLlmu8A4rFDh
zCxH0a`3yk|4d{hc+lwnuJ9g0le`+j%u4enZJP?M@r)mgv2
z_o?SooYP8zIC5|{4&KCg7kVJ^parq#x^_-QkF!8Rb@rfX%XHPr%Pv*0hQOAX1Fcj?
zOXdQt*om)q3k0;nfPI{D8QLF(nB-;`e95XQcHv~P^8pa({p0hzEJaDK1xI<$;O!G*
zsu9+(pb~#f_EY^~O`>-Mut_mbJz}1MS(tVu$ULF|;bY
zHWFHV7t!$2%bU-G*s<98{~gZI?x12W7ci;$Bhfa^sY=Lb46+}KG5q51k4cW{eZ@vV
z8gDgs&I>>^@$F>(T}Qr(c&Sr(x|bIzCi
z$+2(PapdCfGV#zxpoW6Nq>9c~wkj}0@V0c*{T+^S+AdW2@#0lAmVAA-mTb21+e2e&8v*{S_x5U#9h-3=;SJ#_IfCiDgYhfu>{m*)JAdS^d7M;!
z09rw47N`u)Wo3I@8rjS}VozX~*ot?9+ms*izxi6N
zTR_L^c<+m#CL8b-&X*#R#&ryC`q3t82E>}jO~DFfcbw+H%c4
z9kLlml#p6hJ&EY
nQsGw@FacGPavFMn>_>ZP|*Qt#%M%
zn$o|7PfoFFDXUqNB!W=pit?s-!f)?d;@zH0_^|qwkljQX(j?zlr)6s-%bYZQ3(Y*ZWAoWkbQ1ux>BKF_QQE6-M5?$eg18oW8^(EUD{o%4b
z&sskT`Sho&qQ(D&hE*<^ZY4!7kjl;OZ1f(!!;2$ZLXgUrndK0z!7JC%$mjCru)|Q8
zQg01N8(UBi(>WXW@mRBGj48KuTT45suWcWB(T2l)v*juAZh^Re%FdDrb0B>dX0sne
z`%$M4UO8w--*SfS0g8!+;_EkDF8;eN*k$1)xAb7)Z8j2zw{)h7Ow?W^uO!&wInJU$
zpVFx;X^1-QU*8%eTBSS@bL3Df#|q|S@{x;zH8A#|SrNOC>@n31L8hUd1@J=XXP@jx
zglun94QD^5n)wcYuO%!Z*)|o8Pjaw*YmqK&J#k}-to7V!
zyVVRw4KL%qP9APaAAnh*&ezO>CFzi$qykWj87(Y+A39Q~X1!iJHxwR*OFMZw5K1Q1
z#2+>{a2WMD5}rglW#s(Wlw+Rh$K0uR#A;mE07Ei#+sO
zsLBEy4PumitYjl22`11d|3^lJewnF(1vY2CI&VT~JNl(;9gK~8B<-3#*s$R7cVV*e
zZ>}X#AHXJm5y>e7mb|Yk$C0jU^`YKhf2p;!Ie68v=&8JA%1JH2I=$GETW63g$+E|+
zf{Oc^U5G7Xqv8Eye$zssM4wb}&Sa_X2(wo+xd$ry7GgrtQs3@das&feO_RPj-Lcne
z(=MVTRideFGAom@KxP}WMsi2bWvw0myH}*y{^mG;REXHr?!=!Pd0WNkgRrdzv5fw
z(xdQl+=pplHP8O8)3kJydOUAzv^&1Y!RP#~Mz*_G#jq;8KbIMg4gt6XDZWBXpT(q-
z+hCM`B!#0zKeFuxy%>g>Q*GdMP(_7JriXNKAB}Fj62akL6MKg9^-EZaez*QUu=;pZ
z^zN;k{RM(&`;bJ>n>~Y=({rcXhI$Xwi3z|C8$vC~2WnGIy1I<@|KsMu8oE*jD83)T
zzG+oUnyuD}wToBxF|^=(>yBC3o!3Rl%`9|(K7Jj6<-b_gA6?W%3!FL0Uyf6vfx?Gl
z(lPQJw0|+$V-mPw%w@F?V@1H+kj%i{?k0%IK(hcH7hmh8J9GF
zUV!t;Ay)sf=;C>dI2r1U^A8pUUs3JHh!PC-JuSrc3z8_HSE@9_W$S*(hMN=AvueWY
znRLo*CaivZ_sehwtNqk7X^>KkR=DMGh_rv}g5Snl1mPl@q*#P7hzYu=m)n`K9xI@bCc9=3X|h$mw4<$hYs`RlQ3^HHSoYIG}j^ty8vmQ0M2
z@Cnd6Q@8`*__2_*(P4Ow3d*{*$4qH(5t`=rAG%j&v<6Le*zftB)TG_&oLt%(}FeR-cxAY*1&jl-9d)CJy+dr8_TbK6}AKe!tkYq}RvplHS@k?pXC9^#%O~lj9f}M
z&=t#p@18B;U1n%;l!;R!xPH_LT>|th`c+&f%^1aTLXW3mF__mmS)^vu;(nOT;tG4<
zj92!vh0fEjs;Q8;r`dywh#&-0U6YT5cM)&4*C}1n)~>PCOU1d1X`d5+_eKUJ=O;x^
zzLqB&S3=q72=!$#mr=5%fu{PVj<7A)Y{vS>noQ>I45sYt;07;{}L~2OcbW!-BbFH49fLSRsY9_n{*KR)Je;Cd$ryr)!b8gLCdc32y_n(n?;M}*hRK#
z*n?;o8LyyKV=RL*%Vw}XEw`T#6MsH0+n6pj|EO(L8!cbKd`~HxB0<+N;w>6GQ-S6A
zQj9+ZYJ-}!l^Z_1Rr3ysKSj_fNY_Mr7Nsyi1c++K+`k^6XQmf_!@lb8PiFFvF-CPx
z>}~Vthy{2|yx-&_^1Nyj=J*up)BNa{FF;n+UQrTS0;+Z_8gbPiO+W0IK!q#eX^|q@
zs}f7^R4EAvh=&JNy9-Hi@yZ^DI(#u-d5~?pJWkp_bQPrHCgec4fbV5%rWo8}z{3t;
z2zN0#`u;}ZON@0bg9Ye77jL?#YbG0J70r)
zg9bQgIkrPvc$^PL#Y`b*E$o(ffWcv|i&7lP|5~SM`H`;PJJDrW?PUH+abUYDF4K
z2R;wWb$Q}LZio9J`+c&i@ONhtUD2Id%lW5hHSTZ<$FGcp#q7Wi;HN2DQ)IOdF}i%M
zUN>d&Y*v8RFL$u~V-Z{ZB_;H&{vm$r7klOQ6#;^gGDeG(@3l~e7BtO^6yX9;V~Lg}
zr}wkS+8Tg=VQ2akz?c5gS^dn=mR;SA!Q2W5wO7%F)g-TDj)K9uWnX+go^mFA18pf{
zDIzeh!7o|f_WMsHb4~}kWuYY*4tM=@zBy{4(H^Yy&mFay?p7j2a#e8({uQG&wl8g}
zUhYS9|A32Q8V4INYc`u(ZW!~vDrdAM6=e_nWuD`IXLmp*QsooqDAL_ggMW0ur?eKE
z?j5mnU}p&R2Ap=Yb=$M4iOU$_PU&>T%kj)Ser*QvO@S5HqRxJ3iuA>{QE^F!?OWOR
zt(9VlFO{X(?z1cGc=fP;8-mM~#&{%&I>Ph%Sm)C!CQe-^YvBtYj>!o){M@*<&p$mR
z&mC2NJBlyr<(cStw1sHWM_l}H?tDQ}j#XmRO&c>0iZ`GxF!JKyz4K-mn)X>7zQB`;
z?QW8%6#=`*)Qcj;_}zaW2OpQ@BXwB}FUBT7$<(Bv&BH`|t@aH|Louk6m352d^Ef|H
z)zx|&9;*2D0o@V*%>u#$b(9?rV$vy*i9lk1fUgBFhERkips<6(Szr4ifPC3sc0
z6mIQ=E2J3*-MH4C7U^Aw*1L>07p|@ceQtk8l9+==SmWrt;^I`rC05-!B@F+n-FIYv
zEY0Gys3&A>bh#xpzE-yk+X%h;3Ux|NWaQkM@r@N!P#;1%em_QLddy}{x}QBMO)7k1
zh5PN&$9=tVsxN{}w!9w1Hm8{01^pH2nyTB;rXppTAl0Oyhq3zi!U?OYGQ6lkl8Loy
zG9RS`M(h*{(c2}<(n&jA7Ls!_pMes8=vZtlL8EUxueI@OMaQuMshGAHfp{E;vz#c4
zikEicTOTEt!iZd2P0WgW2-TP-uW&T1hD~*e4UyXbO&KQ9=nmMIb1+X?v%9x?6Y0k)
z9EEaHO*Joh4Tw)e{?>b%bh!arC|tj(h?2hxm)67@3Mw4q6Uns^aoZyb^-bqz>@vG
z(eeUMwmjvDmn&I_q6M*ZOu6UQxBP4QlHMy?l_U?5;vng?aUM_?DdF6y-UrVkOkI$$9q0smFMccKJWSbm+5q_
zl6{bNsmNV^sI@bm`NSY+Hcv4Z<}x;ZQ7hfKIx~1l5@{~nim+%=zSm+)LAWd
zw99LiNO?{j#hJp`y&(l={1*P(uG3?}AC_&fI@hU_Z=3g63<7@1*5}eOitAj=@$NSn
zhw#THA_W3@Mc*fN{CImIY~*Oqy7fHtw9sG=mZ>a!kWIdSY%}G0AQu0k`O2YeLOd{f
zT-szBK{@PZlV)PSM7G4%SbKg@o_TL4Fp3D(S|LGvx$679rAajFx&!R{6w>xPl0PvQS=X;YLzE=$jxQtA9N?sG4b|?>BXVlFt!4od7%dMvtVOxMXNPaqnFkq9e!6^r%BN+ffmF>*=JaXq
zxCENW3tvq3yPcFV5(yr?Dd|Se>XtOjj;k75gsJ=$+gG3Av;X6d#Iu#gXMyGSt|vG8
zK(ZuW&pVvbeh0hUh5Y(ozv$(Uq3qOZcyCW=au;V2}?_OSYzV$O*%9(
zf@M~f!670MSmk@;P}gkb#_knw)I8x>I8!j4jwpvpRQz6s75zt}y#zFN)eOv45-K?<
z#Mo5;pe}3n&C=rDuO*<-?Z;N@TZD&v#c17sd{~ALcSKm`ro^VoDQ+x}bM5FuW1x3#
zS`-6#_)=u!E>D7<_>JOd6kX@R;|r@eBoqoKsM5{=rLe)XX=C{=V$=W&~sRA17R9{ezJ$$~~Rcd^%je~WjnO@6uh-v75tKfdnTCMr8R9-Z@0o|Ah63_p5M@Vj1ZH1V#
zlI}~-ucI!8x53Rdc-e74zz&^;$*{BXgT=GegS1nhv4K&#(dTpZRlz$mXL{dpnMTDa
z!`lNi5fYxRgXxHA9Lqz0CAhhLDb90;KlMu#^#l?wcXfhr_)l(qka@dCVx<;!S$O|KJn!qw%96`PAR35-}nhz~yW%O*)
zRJDG9+AYhbTl9xhae0SB*v8vf6d%MD8aUhhjNsNGvV07VRoC2e*)h9+GmSu}JU9aG
zS|;=Muo>j2gOHVvPb3xvsK%a}d#p_@qRiKFMm6sF>u(%4yl9WV8HOl^Q_9(;W{&l&
zAsobs9=mRN(@QC)-6r&6vu4c5!qSzlg*40aEVNXz6C1rvvW(fLL;-Q}BdNeN@gDQk
z5;Mzc$WdNecH+g2&M@kKZ@vo&zaqm_k6=D^rvp-ZKVgbTGIPQ{X=@^I>>^3ON1Kg(
z9H!ilNQt^0J>)9}TBbCf>TWkhJ)bmQ6VVKmPedor!rg^=`^`@EIRMdNlD1ast$H_q
zrDe=g%wps+J*!W?GJ9Xy2SRYE!v8-uzG+7kKufl5+qP|g+vaZDwr$(CZQHhO+ud_+
zGAFqY^A9V9tg2#M;k|j0|2K>f>{9ijHAO*7sTh2qH^^M#kj68}=Bgyk?6A}t4>4F5
zT)*}K4nclCk*!l`G@`eA&s&bWQ$u4LYci^j#y5#>%j(tl9Y3I{V=6rV;q0ypN|}=4
z;P;`iVUSdR7(u9?UXsa^jlE#oil6V??m}O&icj~3#q2!x9#Z-!l3M~`yYDznQM%Yq
z$D%3(Q}LV=h?3I68g`+hF-c}UXir#=SYCG@0e6`i>1)go@B3ld7^-J
zPeIqeI7TquRQ3xk>z(p^M_^q*CF$Y~rDY+rTAj7yPwErjC%|xv(eqB#OqX23!BFLW
zAsBjp)Q7N1eb7=X`=jqGu{D3ZKBK*}kOEUEj&4M;<%X{>zG|&39&1m!Of5smSkao%
z9NO^6l#Y@g%VY{5BxbWtUPI8)37NEKC?+1vZ^tAT62^%^baEkMLg*ca7#T5o?*_N9
z9mPS)&Y2`PPq(c(f;2zC)H&Z(u57+7>&Ul%SsU6MrA6edvs|MxX=P%#o2JZrD{Lgx
zmA5q$KJ#($E=&)ZsDCT^{Btg^6yP@7g*`Sufu~$ZQDnK`P=cV?-nF7#Yg%u{B3$%j
zk8QuG?|n$cy+f#Cc^9bPi48TGn%BXQRbB4cRbERDkthM%&+weOqUklrW6pr&!c9Vd
zLA6{2N$Q6KwCTo^ve{A?GS3H?du?{fdIaaFz@%Ivupr75h*%Hk&UQ~nhSJq-B|6oy
zx-hizz-P4BltlyEROotF3k^8cdtA&7!jEL7#JZ2fywm7Ky8c5u?R$^q6e;-JF%MD=V0fKdp4VJL8EB6)faD=jYQkcjbY;0~zZ=q(WX>>BH2
z1KfSOlfS2HUpX6cTj|tH1|mLo
zbJtld8&%}Fz@Ky;ckh2Y0<0v2^CFL#{%3^6K?zoKzv*oCVi4@c(+=_pw~z;az^6@z
ziQKZrb}>EyE>lN6SXi8a6Z-gvM)|T_18>J);EaMomcG)>KWe@LK5NVPUN;NO`d73$
zdlRv(q>=czBKXf{1-1hF2p1v2V20pT$&UdSCDW4l0Bd{lEXlel5!u~na)ZxX?Wai&
zHhDj(r!QnzdRQ9Vxk({R!0EXM#cK+V5uB%*c_`bYKM$(l;V#1ry9G=Qafdd2kqR)VIf^W*xY#_u8s9?g
zVS1G5Oni4qagv(?`m_Fp*nGF(3)>Vvd5uD2K2sPj@M?{jc%V=UN%rG`#uqEz&mU8|
z!nWh1V7QU2{7(7mN#khdN?6W+BGFU-9SCH_fH;tp8P4Cfv98@W0SCe{9eF
zbr_Q|rO7k#iE`;z;AiJ38*!r8i5G4McIHwwXN&$(@|{qtGTq=hf<5<(xyxgCO}9bN7KTxyRCZ1xAqM<-n-38fNK3L@@Ye@3vvQo`M3P*}!<
z0hdQT$}@Ykm4Fz)awV!DsEj)0w!4ya7K;Ww4{20t3zyD|C^u`uDndBP<4ookzVW_m
zw9XI_OmVNG9(uy1Y$e#VwQ+&~M%|uiAy2%wGX?K$^~%G}L38f3|>4=!yr^3~ezQ61OwHrD#@1|8{&pG!pj4okJq+
z_sQxWzcwGYuRXj0vO(@(nXR@g6yG;N0xIjeB0w_d@n&J+6xie_{@vb7d%iqH5oS)O
z!;tLtK4KjWVDL|N(aV{-otv#mRBf?2KZ3zAxErH2J-ZBNu1D+GM(s$ZQo?kyp&Uy^C
zah1Y>azG3!0fPxxwzUCd7-yjvKabHiUdv3TqR{P^gdMYUZ?ko;g_+g7$4fNr`l!7V
zWi-w_@wV)He_05eZfS>{=SO-z>Kf3O@g7?QEl`yVbTGMxcC^Jm%b#GA1J(ufAXJIS
zb-iRh-Nav3VLRWpbwCtTsE-lZ;CQ_(`AR2|)$8yZEf$2(FjIh%;NG}L-__@l%?qb%
zXR{g@xMo;jo+ml++1)1S-tHg)X1xNwiswClxa;&of40DBD+vuAb^9~WIK&j!PR$6?P>*)oy_!?x{atHN&n5Bs)+Nw{PFb&X?FHdPBN5|~
zrm$D1ADCLZI{c-3b&+DjQm?khmo<5d8jM(g)qXjp^6&cE3HS@o@)Ud#Kr51aY*~Y~fI0jiJBupT29EDS
z!IK#-zRy8sDvyxwz;2Q6QWecRaQ%w%L{A8)ar&Wwsxaa-hy|#lD`y~F!67ToOn7XM
zjD{{xT$m~xf7&NhVxwL0)
z&{@srVQFIbjD9uc94;0Q$HD^PaW{273h&j5q{{dbS`bsNHwfey{n~K66_~hk;*)ttMPAn57AUo;PI+TQ~
zhss)?kZ_t$Ygd&Uu)<_gAk#`%
zn#y_AEifV0DQ;?^CcNE#qF+VYT;w%5f7)#DxZC3`B`tdE`;5}VW-JyH!`Xz^LOYSU
zv@*2hM~wTT?W?9?vAnfp;U?LCLm1Yjf2GrN@k4mP?65nuLCTqFDXtWt1x6zB>X_L9>xKwL
zNmb#>i9E^NXHeX5N>i%zbhTWB9O@*ymcAtHP3?=%J-rOVcU@
zt8`Zt0>8-cJg5{ax1(WHg&~};!Qai5I+itc93>TA&!_}KZMVh|mcOqW%zE|(RPEk6
zp8843z#y{HXK3BlUyV|le~)fy>c+0|O<5RSnTo+)PVX8#k>Rg8+Hit+Gz@mT!__yp
zQmUH$;6q<%_mxNRiRFVAOnrKCD|`1Ckx(Gk$1R8EcL17gCQ7E6e<3VqOvs6%hy>r2
z&*Z#95I`wkh@_uZ({5CjCJvKTv986kJM}O7y5@iHDZ-%4z_x@G9yW!?A19xbpkHL}
z@_FP%Pmg3GNhh*xr1fw37Zv>-nJ|GT*0q!H$}5a;fP
zNY%+Q@#Sj|R*OHve?!okYOF~uB4`WYs(Q7E2=+E-H8X<}1b4<^>r{_v{|Y7-0hJnz
z!{X2n-Ce@Z7D-tZEnPli%f0w?JKy1KH#|V!Y=`v}yEj?%-s-(Q=yp#$PCE$XhOIq?F#`QygaGf3?8aLX@I8gz86X@6J6f
zQI1{1$-oFT{08ho|A>)XeAE4Wq-LliQe@fh)aRoZeYX9FO4R(Ap!f>X#Ks4PIruZ}
zXk!&VEyj~5M-^GSmC0Hx-EefJGpRE#)rqR)|C?6gp&t^{iK3%_L=t?8lr%*Z3K`Bt
z5bJxW0|525eM6yesC=Dh}FeHMV{
zr4c&n&*0QJ&*Hx*<%1RnUMj#~xo#{FVzhsFt6fpke>1QAcR(PLtk)i#b~{2CK)B3K
znft)(JB_p3Nh;-Oj11(|Erc~W;x98jo7|1-5FOGus;IN^1QW#WQ->MUs;vP1$l1=m
zlmDs5#loqUA2ZXhO_Ac>$M9}|BNObz$LVOg8G-0a>L;i$U$qqvbstoIQme^ZY5
zPP;zq^F)}U*0fW_)N_^o)5z$W^={y!7!YuJi1UN5oor4^!@Hhfn8qn@F{Jc-iI`}g__0JAaIFmy
z(<;0qNm}QWYzwN1rvgv9=9xAc{;UDnLX=MvW1kH=tN$5CqzWh#Q&ez`L(wKc8GboD
zRLOAPh}+t334a5HuJ#z-}qu<>lBOlsE1OX@5lue@^^}Yf0Vldo5kCvCijjViPk{LJH@FREDF^?||h(A}uTEG{?OOymKwdC-yZ((G_kCOY9
zhNE{|ggJGwBmi_ucfmrae@M(P2$GTbZNm+=_?ARDguycB@Voa0Av#-!%!?E`u*Df>(H2w|~*w
zF13Mh!chnG13!Xqe-2Skj6Gq>`Dhrh4mN`NIYKY7iMNG0B#%LplS;3d9_2dO>#b>g#=~dd}ffFtRVYem~O9j
zzIzcIWHEn#tLY=$UC@1gS%&8z5rdsrG)0bjMyG!?Z~9d7f6Nl5@A=Oo4v-^f-2vE#
z_$nR_rRk?R40hB4hcPpKS4pn%8QIE>4r|Uo!9|MeZ?0N&|)TAA?Zi6eZJ
z(Ix4P#eYbpupz;&
z=BU(h0r&UV=2ztU1;Ma$$f`w%W0BK?`PiZqH&pfvf8GU|JC|c+yVc5!5n+_xa&K_b
z9kNSelBO%9CN{Z97@V;G`{r!%;#r1I#_z7$l69Qr6BOZCi>Jvp*E6|G;knADqpdmq
zb(~g^fcIY|fML$S#t-9^`>slBON(s)EFt*!#D~Me$x}k!mI%S~wY-fl0d
zH%m
z$qkbjq=d&iGhM$vbraYf@VbRiD1LjSY(G!9Cn=snj2~98UB}QxHo0%MS2UFU_^Y~o
zH*#;SJpcy`=wev0pG6Hxzum#@2j7D2JPa
ze@P(<%RU4?cPfs~Jv!Hg5m2h3GA4Z$-k7xRepkzwA{o9h$a%IGpFv=JUwP=^)@60c
z`Lr=ge~V}`YZDr-RVcGkTH&C5RR)*W(|2osclMSOi6C`O!7K~5SFAIskDdLs8?Wu(
zvQH}K0Hwj>r8n$7LZxQPI05Lk7r6-C11ii4_sF!)dBlLGiOsc+j_J*N
zKKS@Z#=YV@27cjYY{OuU=Fxd*w=x>VOx|V~MgVKm>MT4(gf7mItp&Xp$-1n4ae~Pu<
zc`)>}*&Z&Lsp{e1nQu27;PAu13i!;9^kWQFbNt_Ptty#bMnQD6(S#R8v3i(p4s+^O
z`-~KalQG4owy@dbs!8rIL7cpLQR1RgT(D`yhWBie^O{Wov8R0Z#2*G|i`hXMNy8Ge
zZ|F?TAGf0ul^g^Oz}Fggzm5N=e%p@s#&&9?pB;`$w-?9ww^8WurnsrCPX>22qWSm;~vB3kNSBEr8tB8Hb2v-u^HivfBOKzC8r&`
z;X>Fo>re`Y_nfsXa!NjoUi1m7o)JZ1?YSK3kE(RoLV0h%i^eO~kt-`v?b_uMzZ_a-
z68tLRUvFM0Gh3Tz(pwO&Pu0)IJMHXos*2f0JShG~*+umO*L0ChNA9cgsezQwB!Ri&aBH=bob~PP8NG
z+&Z1RG39EFZvz;z$o9uhrb3kM5++(|zK)ytQu3ggV3iafC3h(-Ly*DtzTC~#onN&%
zdIT0NDi04fYrJagjK|}+W`jtPb^;mNmeA7!R-`LC^*hCgBO{T|e`wR|pUIo+Rv7u&
zdYa$oua~QZXsvU5-YNUlcttIe3*pf6NpFC3GToh+xk^9MYuG)j$k&mAW}M*uUI+ii
zqrKyQt_#rK5aG|i8{qn47jXw|qv0YLgQsjL#sB?%+xjX4
zjH48{6Ulr`xMW)uA|vrL_kn+tnLrhCqCHQIf_+eI;Jt0J(u<|FwVgw-qN3INMw-T&
zFQAt~p|C=hxuO@k8PyVnY*?9*L?PG0CpjXCN5NV3tqI4IfBsXD-mM>!(WxFL3%~en
zmpIq!ETJQ*jEVi9+Rxj`28IOWX&F!dfXiQ;2U_d0xGNVmi&
zF$6K@_OGhNX`Q6OZ_C|Ez6Sav`ZAWG!o|&-9v8D0=XIj58&-NLUz)iAF&vuHk{_qn
zQ{-#9FCh6lf6z@NTV|F`0%a@y*A|g_l{{t(j^aPv;dY}F$KFAtsG+f%RxBa%IJB5k
z>i{|?gCTyr`9g5P!*7!t()&=Rkv&AC(c!5MotLD*mb>^BD$``77<^fb9gkuM6IYCWXZ*Yt6a
zTJciI#dDsIiVOwcj=symnr*a21P>E4R(!|Me_LxYdQ2@meQ>2WH06)N9s|fpEHU&l!_;@}(17fIoze^L+?Lx0NLtyk4t^
zjO>UQZ75s9;LeAL?hk$ZF?N)56`6Pofd-K0c-c1?h-vAp({vb5U~iH^{myhS3^VK=
zf3#%MMq{D7{5PyJ7R$tlXCd$}h;=!VEHCqi+0gSHiSoC8Z{8F)Z}I&!J(NRs5zu;=
zNv*&Zonp9carHnMS3~Jm_}*v|b`C0XCK!!B5qS22TFcr0IaZu4#f$Y0L^3&(d$}?E
zIv2DNE-o2NorDEH-edHKzTR*Cmhbooe+0*IcdY-)g0J}*mq~W*DHDxU$J?+tdltG0
z>z%b5{%qf)1O+ke&>Ut6;1bt^F^9RmC?%Wgz-!63q>KXZz&H#&N!0PfwSIOj*_lHT
z!-THMyA256--F1oc=6%QmyA;1*a;67%xwm3e&cH`RPEAmpC-I0uxY|W>0fbtLo6>_
zPrXrmaj;Kp9~z1}8IpC{dE%+rj;h80Z(U3BcWc7JME+nIZ!kBX%>b2FBp~*h
z-Md7FaanNO=+j)Ww4aJOeG8Q)Z2s5$bSD>RqZV9xYFsDv4_~$a+v_Tc>}&M#gJ@3-
z8Nz*-ae1%oj7YT3FHxz2VGoWrkHf>q=D)KKi)B7PDJiA;Mf7ccn0y1re_LI=9qo|~
zgJbrDmk;*`E6BdU976M%K`(G`EIwmkSb-yf6*ztEC8@i
zG+?R~1;7F}(bFFe&Q)`AeLxf}&kEhE&un9y&<=sLv|>^S1r0LHpMN&5p+AWg2-rz(
z1D1j{Ig~3^K7%};`6bq6e||P29o(CYGJ;Qh7?Om8PY3Wl8&&&1k{2jR75|_cjN=^D
z)(6Qwq9Fk;p(Hi`#6S|kco1a*Fk-6w&p1Ie%ZE3!)$-4NUaEACx&W!9TTo-ee@b@<
zU>RRK!>FdND&+A`5}_~(57aR_XsEP|(qa|HWQ31k*vPBSy6R~fe|;Q2enn*@wyRht
z@kftkF}G!r7M3fED8@uZ_>&rjaC!?eYvsR#Dwww{TjvE)DK#8CjOt3hnY$WkR;_fu
z>5?tCI#}K1GTSmhkd+BPaJSu|euW0Z`o(Pf+eYI|M0Ofr39p0z)OODff#4QCJ*Bv-
zXmjA-z}Y*yW05JUe-N9QW_Gck?zfI7?oC3SpY_Q>+gUE^Iq{IJQ~E^{J<&7&^-|k%
zo^WgKnP(+HZ$iN`M#%8Uc4=9l4m*jw-G^Dw;S@rd!^1x9wo`wFQr7!i<;R@K&`E1C
zQf{3R@eV{+Ru$;af)73AhduWaR=|>3`bnu=C`RA2xNrpFe={oT!Dr(pbwG?k<2*|{
zP4^)3wMujHmD5V({A;V1y}^G^M@PLcI{X;IXGpShqhR|KtJLn{U6)41=ljlu_kGY5
z^*|dfZ_u{mIrDi6j*rk;U6kU&rkU%t?cFXKnab2XVI07+DzcJGm$O1Np9@PJ!k=s%
z38NujYm;rIe;Ti-C^5ahGMJ-&1=dQxjDOKI)E4%UY6LA!t5cf);b-Lp0bQ+FunrBJ
zU3R^J4eI}IM9bG>VjHAY3kZ^!x^$8GsrrV)nNUvOrBtJ7+6-2|em62rPRE^`z_XPg6V_f0
zOZ*?DF*ve-?y$~3W3f*=emK}oQ?aUqIu!Cr=GFvY_cp&Z7lB6Z>J=bD6lgZn(>
zf6AOE7q5#ouva3tEE`ld&yq5=n`u9f9#!KOMrsN(ZB~E6oHxd9JZ3#bB&FG2DH4lC4cMvaau~4oWy6O#Q9c>Y8k=e11{L=C`*pA(>o;IVK
zsQXMReqMbfReUwlAO8|&M=cd9e^~OG8U{;KlAtg{rHUvU4zUJDjM`I-<|B#o;}YF4
z$@iaBy2PO~u0%&i+)uPl3nR~KP~KsN=*^e}>Kck2;1T)lE8^;q+7~~19No>eWk%!=
z28NmGL0{@n&Y%@_`wxV$@*e;@DP+MjeZbb&4&h)(KmHkLM2_W0YFGtWf99?CS2JRP
zFinasy6x4z1?0MXuKgxPTADxJgC^||jW7=Nj
z{0o@sr?2Qy1Mzf
z%x&Ry`Sf;eng939#5ia;f3|b9$!JWr?liTdsq0>{+wK02LsRBvy(UwNT%=n|t*9WF
zDG}Df?!;qc@_9*g85OecbHrvvSG$+9>HbKL@|ANA%fY;R?Jlv_-x+3xawfbVeAZWg1QwO;0}?Mz!+#`%6Tnr3
zmc@bW2V_FwGTZ0pwqxqHPSEsT>%$2j7G&Nh81-$c=ope$!P={4^9101t%vq7d-Dq<
zXfI`oM4-AgJX?`$e-17RAyTPR*xE^|b{B0NnEu0th|Tuzp{3&L?^*Yj_E37FRJ88j
z&(WeTR99{!?LY<<)cXq7_BqP!l3SmAV7Bb}ZI#1`2L7}
zWH2+Q3pj0`9h$bwRUDL2Lj4ICDDqpbwU>pm3+V0B#}I_-e<(BUjz*z>UWT~Ai&ve|
zfg$~+c17CDkhLiO%$wKxkRx(NiNWAm>`s92SHxr%4VQ74&>pVN2GXQ1oV!137EuB%`l3c@fL}`DmEackGMU>(J+$Y6ZU-DvS~ls4(@lVv
zSaa?Zyr%UA)R2mU_0Ace2zIrDT$n`((!BP^+KTbae=ZQnRvsUJFN75BbtR^84dce*
zv!mwWIO(Ea$4|`K!UH3{RFezENWexH!O-WL-Z^#@O(;??&f3QD@fH+y1wd=1@Rlbh
z#g!N4QX!XoygZGJio6Kcx|EW{m*H#NkA2u8n1|hK%b^VQ5~>Wv?*4uM=lD~?_h95o
zihm*)e}qZlhgg{x&LyG7o~3}t2DO6k&dm)hp%emoMKhpJrYW3Z?Z9ToWh0ser#BNS
zWQIW#IMWHx1TCxf$5!I=Zw~f=1fmC0@)2ePcun;rEmU+9p
zIHZ<3geWMKYW^UOg9LyKaUU#?C=B*x7_aMI9NVrwFl`EMZ!hlosO$@riDpc^G&E=t
zoYR;JDV7EdQbTNYa8Ya5WlBR*duXj!06GYLhWrL5+4NAfPbBFig@W!02Z;k6e`4j^
z!`_fQEG;EvpFQ#vE726PlNn6)JVx?`^*L0@1puUuqth&J)QNGl_JsQ}cXP~koZ@TR
z(RR2Ilr&3cFCCRf5W>V`Q@+Q=9Yf8Gs?AE$<2JetHA!6@h#4M4rdX1{{OGGqj&%M_omZ<#GAi~
zC_IU^mc~g)&KehMp+7u!0Q|+bV;fMV{wzY1#vQp$*vTN(>Mi|ecgE^0f0kIGIo512
zH}%dPKfn8kpQz3Lp%%(2nq*G&6_pV9EaU_M4$_yO8wPehAwP7e_<>0hq5^tPgq*~W
zqgY2%R+M>;&@(UW;kym+x|cAKPpNSU_@W1};s$Mgt-}2wNUitrOv2c4Kc
zC^~pMNQw*SfdDype>rjiDNebhA6g0`Zdc6mc90>1I^Ue-tG2KcEVBoCHXQ92qGd=>u>c9orpbd*EMbAQ7
z3Gg#b?{o{o1@!w4KP#aAP0XZNFTakOAjYIlW8>w>r^^QK59{CkP)sw{-gl^%*TcoZ
zsF`r4W!DQ)G&5twC@wzufX|HdR1rs!1`z2iq|*XO?0yq%e~c9sUU?K=1rn>Vgj?DJ
zap}UpL_S-^*7Z{f2F8!rr(;XOk@Yy39Gf=jKz%tFJ)LeTFI+&LDT6LfDnq#1$h0@?
z69zr9t+Qy!Ohro?nNgoxk2b3fJm#q^GFM{x8RQJNB|Td1XY*vg@AHJcH>FE0a9%yE
z{#_ODJN#A#e_D~sBA@DNv=liqdkaMfk{4?Sv*i8!5%FEsWnO0gz~s6JJKDfJdIR5Dpak5Ex3prdhR
z(~`s>e<&3*qc-$u<8VH^AQ7Z2=IUH09r{j3e2E$W3*OuxIzWV0furc=vYY;ysoBVV
zFe+QOC%)zg8<11%1a50VKkhN(uGyY$m<;4dg_Hgl54R1
z@Tq4%rWxG|Z>qwZK4tUtCxSL?KudQhitk~Ie{QS{psr>F(hDx4bb&Wj4y-v-oGrZ(
zt+v_z`+zVnMmN*lLP%h=1S>T*sv9o9mzh4&53wPn#z+)#8>y@vsyQer#NKNvLb)5v
z{^Se<4SB!gEa3M{uHa;(kJ2(>+?TrWSRO%nCnnsHKgBl#%K@+{*~-f%gcG*5k-dTs
ze;9oEpNN5=ZpN8!*e{)El(!iB(+-e0P62RN2aKL$TqC!+ZwVS8ly4+NMjcoah;oH2
z4YY%3JVEIhMA9(Fn~BWczY(K@YY|DdTtZ}*hV549W!VP)F+uhzlMd$suM=AvvWue>$e+#1_vGW3Zf>!20`vUf{cD1i)pPfQ{d-bTt
z3zJxXvP!b}JIonpbn2Lc+Y#TPrI*(Dv0)5!^;CrJ!c-F{Eo>Q2D-L7R>9wr)
znm4uy=a!fFPHc2CKABmHNhRGBXQG1JZ74yt(lT#+K%E^^6h@3JFDh>WKHBl^}NiP^6
z9nK+k<16fld!kP4h*<+9S)
zha!n3N;v>%zN`Mv^z;|iY7C|Ff7-TuNF*_;)^7T|7AC@b$+?E|Q8+bXnI8id0EAhb
z|M1nu>R9B;;@)*?dvmnvs!w3ra;k}cE_y`;;JMkgBtZcG3YKNwyu{5Z3u;arrTXzOV%X_JU1&8I-Ts2La%o#R0|=Jv
zQ~;tgs$>EczJ#J*wLZ6C`vwPEb`Y8W@1E9F?Pc5}bcDsmOf4k0bFs<;)<+eYifBV)
zOd{jB`*G3KD3EwVbhK?ICe>;O#=1k8_@QrwlxxrFp5SW$eO69Dq!o`UV%(0L$&Byb&
zC&gh7-q(w(H)r<4{}FI#-cb5KHqp&8yD;qImZDs_v)CppLuz;vn*<_S;`xR2IIKnl!@411kI_W*5&Y^(%?)s<4cc*?6Gfk8K+(pZhei5
zFkJj;U#I(w6{-OVe?-aBWDQ^fx?x^}#YnfV+NYVIuR6c$NcIrIR^XYMTD~5W5PkS~
z34dW%VNhe6d>Y9J6}?|%VM@2Aw!t$GkjO-8!7;kFEr()7
z6W#8+zLxMZda&LutK0cJnfjQlN7EP3K};2ko4i3>ggIw5fB4pE*ZLoGSOcf#OE|LK
zDM+SS*LGJ%{Hfk_X%?t;hrPV#%5kiRG@2O(Zl$m+Jy7>-1BonJ7^Cc$(#cC64JB(I
zih6BElE36H7&NU@d?OX+9JaPZINI6dtQjW?PQ>e=yHW4%)aEzWt=_bI&7lMrwb5
zb7JPKi1YnGn14-v%U*mjxxvD6Bc1B?G?}p;#w%$ZS;?SW4N^*>bOWC|{}zAnCk-ya
z+(M3He&GDkPf*xpPKA}$lpVyaC2J6{Q0b?19<`ZdA=egr+Gka~B9Dw`54@W${#c6T
zlgVpHf7a=M)qjkSdDWY^AnudCUvol{+5#$^z`Uu3`>3sq`LZW;BA1TAYvk#6-bqef
zdQ~AAx_uQ?-oIfo7m(z+A5eJpM5Dde*Os=p00#(0#{n$()bu#kwVF&FQI@$$ZZ)1S
zof0_2HMg#apTHWQ^>2$&ywLQ+qA}V3(ySOse<|rxNeNhkl^SG$Nk9Uv`INqcg%>1r
zvhq_p7C3FEQ?k^(V91=tf)d8e1dzO6EWcJwNsvdOSuW^g*bKy?U_}y9Mri0>wnu;n
z{YF6hFRU_G#h-2*D|TWpW3h02&rnDi-dh`78VUeWTH^hVlZ6JNKqG}(lxpwze;CGY
zr@NQ_=L)(bfVx8ZJ9(+zjk8Zs=>5Nx`t>~xPj28$8E}Pg)nH~6_E0HbgwX@H$Y=Z#
zxH&pMSW~j0R|D2QiK=*Hxa$fI&*w7JVXShnALOy71RgCNE1JQWM$jf`f(J|hy?1Lm
zx>Z~;6wCJBWq+VzD}cBbwPda^f0Vd#AKIwM=YJ#a%G>nH)Fr&R+BgLGDr?s!pNn;j
ze@blU+ryE}5TQ~%ZA<6<_v4;8
zh!K+Y!V6r)z=`XzA$BURBMu+Z8X)`%B2SkC6Wy%;B9d1kj{LGPrzg|cf0zn%=^ay@
zdhWI6f!Z&^=(4ePUEdKOlf&YNX@1=H=sCBT($vqOJAC*<}ja_rxhM>Yyb
zzb)U;#*Ke8iG_H_f0T<;NNJ=JfnZy)w~TNQCmL069RF%wj%q
zDoAF>tr`)~p{(KlIKvacf9Nx=AlTxQ!5Wt*S2F7&#oM?XY|uRFaLD#!?9fP{84-|Q
z`LTIxx>oD7f3E(6atTd35ktfbQz0=GGvI(?6mW2eM6
znk8FlN@m$BXQ^JNoa8oH)Ia>i>HrTy6>Dfc4qgg*eIXQRkx=aLf8@0?N#WD(*>LeD
zl`&5`jOUd-ql%ulXKsaW59zhdwJ%TAu)$>^Ye{4u7?W99tG7XIc?O)@_4=h|f5OwK
z<5GhW(|JKpBTB;Trrm`Y&+yK~HTO=LOaoErwXTylqYE!IbMqHRAJb&kXmi+gzo_X*
z-FNYIcM9V&v)ONTf0)I&$@3<)is1M&0&#=M{Y*BD_<4)Z_P`mNv$*a>B;mrISHO-n
zHE;L=sQSD*VffanxPmKWN1tM2cVXWVKy!c=I4u}J!ByZh>X};&lU>r4_w_Ih`D5%8
zkVa(#6`=0`8DY5d`Y}xNTDQM*5jpe`PVyKYr)k2jFj-TqEAH
zk}Vr(q=<{%IBBp=12^6dZY>L~HGr-7-EAG>9NY2H?(Xx9;a?T^-d@ujmLA%sRWw^G
z9vhaQMOiCs7k0CQk?c9HY;M}GPfmo;E0h*B?Lwy9znWeE((Kmcs7MU8bT@J>xTo~a
zSCA+VK5R8cf7a%p*2_|NG95A%K7V-qgH@g?%bKh=3#bydJ2Vhs_M6EH+g=H}kql3C
zZze0o+229}L{4KFnqN%ku?6PEvUn!3-DtGBoWwK4TP)Qvb!_Z0_usi^59#fu5!)<9
zT%A$zXz~0RwaDsyv8eMaXlKkVDd9t0ZSv$%)$k2Re=h_%_;MUiI5Xllz)AvlQ;{@AD8i@e2N9L(SB=JwuK10^lrl+icI
zk2_xN#_c1*IUG+=OA+T6z>}d&Ubu3YVzC3uX?Ybd#UiqEzyo5yop!vYzWV$^R
zRJZ1*e`R3JL#^uCm=x(;&Bpbmq%mJamf
zuC5LRMH}kEiAPBDG1^%HB{`m=2j)y#8w6@X?OGC_kZ^~ITF}U&=??=&e|#GDfcsTE!L}q;L~8(dWxo2t-sv>F
zXNj7xF9vo@e#TcOUp4u<1CBmpR*EmeZ*zRhP+*q!rvEgFYQ&3!zGe8^m*gtH-udbw`PxxM!TGLj2AQ4JuKf0j2(>>q>{-~Y4vi+POPhJ$K{CLApFt8IF-`PXvNj8
zfA;X#x)y0%pkaF634=`=!Nne=rsiIcJ`f5_r@>laSwU7J(n8Buy6mW=*iGg*FBAW~
zyQ6|%_eyAvAarOVp5DmMgX}$Anpt?;cLotlVBi
zun``=8=EZ!*E_bXO2KHr(bV3Rz>pjSOp7v+BT`}SbmiQ_C=;N7j$#vVgKBH-3O2#BK^-LO&l
zlY>KMZ2p??!zT;L8X5IGqossf8xcvKdRR<
z<;h>g4`Ryzr=?1s
zIC>)K#Xb-}veL(3{(BIce~aH!GVs9}FrjsJ9sR5j4)Eab{f`!(x>Ze$72@g}otB|G
zRb4wn=O0AdNvN!!ZT#VebgZywCPem{Eb2L@Z9~*Vs676k0744A^|JsLx4ZFH$FQ02
z8|b9h(@#=8JuNqY$8{8old=QOn|*Q<|7gqgkP8FkUAc-q9Y3tGe@6=O-%n1yk?{4=
z_v(Jq;gL@a$VlRQ&bK`ZEkHs}`}WQQe-u(J61cp6`17#Q@tk(umS1$f-?LvZn|8Na
zLoCpF_6}#B)QPXwTT%xAwMB#sri~D8-@(*n*eN2eMy=zwyqn#vxm&`{Mu_{z#stop
z`F|x%eeR&%wvvO2e?y!(7cw~Z&{yjvviJFt7(l+Y{Y)5M$XRJaqf0-s5iRxfO0z~5
z#wT}|<4v$Gu<=OKXi~xUH(MO0Ds(gqfqa*w|#5evV>brR;iFo~`b2p{V6H5|gHcT-ds@
zV&t#c;#98aEaXr;4mH&D`nVFnQ_9IKM5q75ZY_^&;Fo;y_m`eU9#(odXm5=FE1gpi
z?>`pd*;U>w*Qy;47LUL`Wy0Ey)*O*F1JfYAusTR`e_>PvGGF(ptv27wru>>Ujj#GQ
z=^kdvp@3H>Hy=GV*<-(kXUG%>CAZi~ofugE_`
z84Lg)d|@%CXkk%KXM?<}&~`^Xpq*Z}*+*=PcZWy-o44he)6yQXU&WJ0&hRd;c}ME1
zwOd6JNN>(gV>O>NUSIxBw+ODQvZG&5vD+voe~tP7e!tjJ(XjS3{J{aaYdEQL25}jE
z5B^Bm+AyBNX3&i6e{P*zBli@x}ltkl^6TTswP$ir*R&EddXN4{^61EjG67*-&e
zbKn#!42EN`GH%Tbvv!*YzQU8yc#7Bto|u)|qDr}ssHm6$1@QAc#c4I~ye!nAQBo90
z`@OJZF32Z1^N8GyEkdk$uSP8U4W;DM81)0emZIbD5M2Yk?m{`&5CYnJE1o4
z;NYwB$aN&W8ePG=(>tEx3o6T0^?At4oH@
zLk&uxqAvba(v)_M;D!9NlqjeL_jn9RnAOv?_}LrxdEF$ga;A9kIWl(G%%{N{&lPInA&4)M11ur4Q(Mg4A!<9BN8
zbapjwktB+>pP&6&Y!M0*&+3c~?|+qwYpbDonabTV2RIng;KH93BVG>MhR2Wau8v?o
zAZw$X;fRyzEX)>tvu)>{WmiZ(*7onaMo_{uP0H&i4{=kpLNj#!dg5$pyzBpidb@~!iW=v-}FGR3Ddo&xHNX?tJSBmiCYLn0Vsj%z6Ba^JKi76s%m9?^X=zRE=G>v1^
zTV~JfG2OrgmfhI@$;hDZ62upEN{TFrVM&-%
z2d{JXJrqrtbnO>gj3Wa_IDf4-R{lhm
zE?D4lsWfP=tH273H-Oai&<&_-+m&Q^sI^`y{p6baI#m2sBuMVkml3vA@4?X3egjYr
z>@uRG16+n`x-7~rc>w$^Fkbif1=(SjAiwl?Y
z#+m(9eSWHO^$Zo6<2#4Nxoy}+EjieW^C9bD#j&Zcsmb0%DBjy}9XS3L$*#j`4--F{
zsC^p5gBtYAbWKpoK;Z(EG$@wDya;X`VYC8R$Rv}lPDqg#SyGoOGYaX@*n}}EH8Fc_
zcr`50h@ry~PM3+{_S1MMM{!s0_&)$gj(^`j^m}Y++<*PhEG-q?
znOm|FrT53gBM8Fu!z#{<-_zbO)X0m&AXQhZgB(|Dl`mpH(dHplTf9HONI#eWPvADa
zniIvy&AN4kcM!f{X0_Ucs}7c59h#W6a1(=4=0DA&)wik9UeX*)!63n-&8RqFp0P(g
zj8z*veZELX<~~OlBQQ_s94uPHTRdL6Nn|BQ4N9
z1JK|A5{1l0N4({{u%yWbg!)4nU~HS_TnOm^bnetv5d4w@cIcg=we6Z)=UTGlrW!2}
zGQCceSgg_!r!I!#rw)z7yYe{T>9`Y-qxw80LM|ql8h;SY)9eQYY8A{7(;ypAj!E%75!jjdMILMcM02^8%X>)2fzJX>;9+msnw^FOwni6qg~!
z1r!r8Ha9Q|FHB`_XLM*XATc&MG&7fBP6ZSNF*qQZD+#P}h3-0djP9P8n8r)riJHg?Qf9-wtUhCY~+h{Nu{j)y%=wq~cC@L})
zbtX{@kQq=4~nOP*79@IhZtfSjqh9mvDQ&IVxWXaSI8R%8Y!
zfjr(p8vrfH5nu+iHnq0|fGhzTKplXlx}=&qKw3>%Q$>HBj`_W@x|@>|2>f4M#ML!4
zr5OPdqDmT)0H78lKw48>#g!QZwqka&|A(K3
zwT%nF0|bAz1H3=MKzran!niqFyvylo4g6PwzncP3urUWZx&Z$cNrCe1b)Ba!S02d(e-^N&*y8KhCprWDxa4@xTbOkz^I-0*Hx|+JWxd2T5!QOv?
z7Bv4N2n2|`fx&-gDE@Z{{C~&%H+3=4`{(nL4_;c>gbL{{6MhL5?mq
zF0L;B>Iei_+SmjCvUmBrXEu)iV2YwjvQm=j8cYiB#&cv+1ih={$n5Ir`cL-Xexecz
z`~Y57ZU7q}CxG?cq>_#n;vffych)Wlf8~>~dDqDm1omS2e|&A{2=Z|B`G0Xs8%GPv
zzsi4GxH++CI@&n90c9orEBhTn_#ey)=n7y30G$CqPjhRQzvKRK%U_u7FZ?cppN|vB
z31Dez?*jC*u>`(95PV!r-GKmCup7|N=YI$OeTBfr1F*0$cYQbd`(Z)&r@O49B?!Rx
zFZi9we~13d0<`}eRJ!+5Y5{Vz_X1b|EfIfMlt8ZUhM@ib=bZh=EGaj8dnHo`Ankuj
z`k#5G4mS2)|Cae58QQ?VTxpd+U?7whXSJQXzi#l4_
z1K&60A4v1>ak78+?fczh^Y`ikF!6BwN9f&==5~%i7Z(6K?>`~ny9)oo_g(tGYyp2P
z>S`(y5-JS;k7fQzlXNr(S=cyQ0ob{?0j6NEsTTt4yARm8xBxzE@0PXzdj4Y+01LAt
z$o0Jh;N<4&2e1Tz5&mu_Cm(w`&ouSI#1Nx8m
zSpSFef8AXFtkqq?AUmM8jm7(|{l9-w6ir>hHlF&d@5W(!$KQYc`#-~f8$k6xp8sFf
zVqzdqA0{qtZU7SpD+hp$hld@&#>UR=_iwf4|GGr}G3~py|GoZuf&oCFC(s;WX&z)Q
z5Ney+5?<~vS@^30nu?G4qzXq!M=k+sDdSfcA)Z9uHaSqZD5xzkle!3`Aj5xe=${IZe~HuNgdE{Fqfae9@^M`I>WXNr1gA=#tn?qG
z+EMbSJ4k!0WrkaI3b%TGLi7+wrk4c7XXYQvH9RWvj%PC7y4`QSG#r2EIT&%DUK^{)`j;3&W5_75QC#sNE#iM$=N?JSMGw8HNM-HVm~wMDjX`M
z#8U}p9Dvs!;7O>;7hYUcS*oRy2xB~`Xm`pG5-%A`+1Ig5MvbSI@9`gVh$j1BDtOsv
z++1*w5;Dv=Xy)WRMNogGA-r1Z*>iF~SM$oIcbb`oreF2juU>1L(+CTI$Q^SuZk#{?
z7R#Ag`Q+#!SeTfS{#$*@Vxh(qMz*Hf_W{#&Fd4$H*6>om>JcUoJeshzE-T_L<=#G%
zUA`eXFKS3uM=#Ep@!BzrkG
zTdsMNM1;-AHxd}1iljEePg@T4!k3%XVP8M}Sd}N9*dkZ!K?*1%w#AF>uTYLEnvIR)Fvn%*K
zbIpC3xjZy5MpaTN6%utrlT|=GsxgOo=TU_9%pLtzE5V$b*7Y#yer|)my%N#-oNTu~
z-!1kT%P7ysgcSC~<~O^JYY6WS2WYhDs~$bl{`!=R?+Jf&JLU%v=J_X}EAH_ZT^@{j
z%E4;RJ^Fs~{^|e6fi0i){@j+|%(Y#>)K?&wf)K$61cPm$_o?%4D?gGoH?}XcQT$lL
ze!mdc2hzuv0vt8PlN~9SZg&nfm(kWf
zawOv&EBt?Eqqya*Kl)xX1Zw58>{ygBj)D8Q&Jf{o?6k;AbziA*gb=q439!)M@%}(h
z?m3jTmm9G-A6jhelFt(?aMzIFaN~-T_?eg1qImuAEdf4XQx&ZpbFa*_C25XizOo#g
zQUmxIxC8~$xNeH|C`n@mSGL~J2oPlHOuJe;>|TFO7n8smuD%bUkssDfV9Y8ol$mI!7_e;)vl9X#BWfnfwv#zAZ>oSfx>nFE
z-|*IN--EmIPm|jifX~4}E4!@fLVnd+zfR(>3J6Ng83|?+qv88D7%jIJ-m7uc9Kq;i
zt*}(+t8x7?)iMLcxbcqED
zk5l!0WPic5gS)A0V4fgl0A@qsx=b89TcAR#4tCWZQEe$_>^{|E$2sP4>fA7vxxrc{)o0G1!@^1uWt0hK&(t#-o2i4rZr;A@O*$5BVyyqUXgB}P%d}D6bGjwa6LVk
zQan{ZfO;d0R48T?^F8XH4xhOACadC-BT?>COvjiwU9}PRt+O%b`
zJ%1^_Da8G4AtBWy-}ZrFtTsmX;+3b?{VdIKOy4qr9r9~J-YJUDIT}Kz2!Tm`fnQ#C
zk6zEL90)Pt6+e7!4zO56_qtc?lO~Q_w1+
z6HHE47eCfyj}XV8n>o$OEA}072%`?V)A}Jnu;*o&E=8`PM=#`QhkLPqirolO$iB3W
z7!eEzsD;~BFhl?ph9w}~jwQ#pPPah$*<$I&u@c()r*eM_jkb>Ox^__OV=9fUcEFITxLrXy!G`~T(!TIJ@oMSd<iQ2Ij>MY+coflMk?mW73wuV&}sM1P@p!&L@
z`iV_r(}TBQC)@U?SS|rq_Q3(#@`r?lF9UzNNFX^m$1qDO>M)$(qQRfmuGEk>FLiY-
zvr-MYa?nt#%tr?K5LPS6G$4IAz!u+}p|!n#$`1QW#GFCDzmD#a7Bd-vQ};AR=j|{@
zKiH%820x_!QI}`K^l+@O_fWjw00R1l>Moyo)xz)amoVk*CgoYXPp@92q44(805u8M!lR#Mr12tgFEc!2)fcPM3Ls2A|dD=QsRo&K4>PT
zlB4d2x}}O$1w5n(Yl~fOw8Wkd>&SmvS>2s6BtHqhJz9cnMNRj0g>x=rc>VSJiqm-5
z^G#r_EpA{RGOVk;YeMlKN?Q12Bh{NJ;urUl+JE;E!Rv|GXCrp0j^7#XHK65<%+Ms`0PY`b|
z9pa2T75^nU>eqR2RM6*Fx}ttcYSXrZY;KQ+
zL7=F{8s{g}v9d38iKDNUT6w1`jdZ8rjW;aJeUtpk(e{X*MUZ3RCOdzQlMWlnuWFK+
z`mi}%U+glCC94B`<6(+SH%bAK2kmZ6c*4ZC4^>7-@PSD*
z(i4rb2G{jk0^t^MW6(E(EGP>t))C74(eu@yiB}T;8?+DXK{+kB{&rZbbIK)W_@aeh
zM{sv_ceoonS@*CqGTz$l5H3Pw(tCwbvmtUVgvs9((PriV!`*|~oP<%L3@
z@Gl9MkABL#Qnjh~r5Q7S{p`%5DR1`nc^`+SvUrc8&l^DU?m2G5#Wu~|?vg@zJVk=Y
zZc|0SHx)*aXeBDPUo_J9J(iAkFgoCtpD6=M&Md*DL{LZrHduet*{b4Uk)MU-`SWZL
zM3bCTpyoC2IT9yA{h=C{hjww{6${nA{-Hwk((MS?h8E6i+AP#iKO2NDQIh*vH)h`g|F)y7Qbu?NrnL9jsKD9-(G!1AuM6C{0|dYj1ht8$^>bXQ2)C&Deb
zUScn=t3sv-3_*V=8a7AG-(LidwydD`cJr(5G9W?fmPc~zG2)l(fpM8qwwpExX
z_~8~{wJ6vVcfu_|8^@pONOjjaBn(
zzn#D8E~dn;|G5g2Nu#(NuKB8vg*OFN#5Yc_cfOcBwI_eV`$a~;n^|&hv{yIiV@?4Av%0iHGMG
zP*58?uD5>!sJ8+bv&Udi5HH4Nj@RqfVeQF;7DM`)emn$gwK$qYoz}NFu}d7r
zXU!JnNI|R&y=H*@@Q%`Rro9AIB=VG?Xfi4^h^gIB!XDwdDR;0we9Px+iNWWFPDbwu
z*(QH;d*N6Q0EZZrQ$tk1M@xJwBA+(?
zH3H%Ht%uSqF5@!?3gY@_H==c!Vq+y}$eDixU@HGPK5=1a<@hpg(te7iZ*{S
z$IfC;5=M1&SS8EmY{l$FRhB
z5S^9wb_BcsO|P^A0ZDP1fwzUZfTe0{%4k+@-T00*CDBIq<(JF
zOh~0jNX#IXd@3gqUW0f`^%9l1;N-9_$3;9j2$;jZ$O_sp0D11x-oxO!9;Sbr(g?Nh
zPSkdt1#81vxuECwut46(}Si_EpDLxEFxQLRl6#e
zMq4Kj4*lukJ{f1(m|ZxgE(puLV5tC)evbSL^G)=uJmH*cubgerIMoW+mlhiZ{m?^y)0
zh4J%JL%pQcVQpw1!hP01=guNt`Eguvib*;RhS0zZ5%!JP
zWEmm5xV0PgDY%?CJZC+fx*RVcZ*hv>k82&ws3P+-T}ieJ>p>s!W>?UVRXN(dL0}(o
z`lDuHKIT7_$=|#ZkWW5*n5f2qs1XrcV^@?WQ1rHtT~Pge{Rd+1)k)j22!KF7Q@#ri
zr75!wOkF{iZIj>PW}<()r2s0ckrnFNB>9U-Qu70UoC@LNiL5HSOkb|5
z=a_L_c4J1Yy$BnRzU!gpo`1ofmP9M5lw*k5_v=0OC$3$2hGZPA`vh;X@rCNHd;E8>
z{K{ke0G?RvQNVwJ3R9G6GpbZ`I|j=QyV4mCLV51#GzVhAb3UKPo%w}BIy?e7hD
zC*k&L8kbc1K)1VcE%wTfZIS6Gw8vXeVy2#SgW)uP7(W3+t@nnh+2$DX`V*X+P#6aG
zteRHM3o_xt!_CS(X^>fZcDxH%Igo?S`z$x+I5^Ta!SR1%HXN=hPF^-Ku%+!~WHr9E
zVMxC+MmnL{>px5b9815#m)`r32C6mvGzhvZF?n
z=BzZJT4{fzb{jMVGh8dpSFnAaOOy8@;~GKPP4zc9RUbn@ZTsRkig3w(_1~CV72Q*l
z{?N;rN2R|J6_5-X1N|R+S0C=VrzEfh85+~FiB-4RDXGy#C@HhYGGYxL>n={w{PlVU?Z8kZzk
zl{wfPivFxNr7AY#EVs9v5!M>!?u-r%&UAnClQiEbd+9H6l>BKQ_twQ0du>Hp=93Xr
zNSQkhEppnOl;WN>zrmLNeNc8{fHMI(>Zky=r?sqshRmDFT^DsQ5d7Lqpd+4Wnkf2qS0P4?fiYT#>aX6saC>)4*s_B32
zD|Xh0n3B~`A1$}-NRU1`r<>25K`dGtY@#}eF49Gw(KKL|Z~2h=)W9wu%*1nJS!^Si
zoWf)~uodyl3t+wW1{|L8(-PG-Z@lf(Xp-~T$8hLJcOdV4ijh(`8;MP59MVnVN$pn{
zW^msvl)P?hlrn$NM9;CKb8DulzrIM1Y&0#ydB4D~nY%%8;qtIL{c!!@xx^!^7
zXI1h6*$=j?yZudiIj2e}F4dPD8zp;SHldzCI|7rn^AEvAMKM-=QA(}NW`BRN9mU(H
z4oZ?=;$7w8;h4T|yIBdv98K_>zrxKD0ZkNPANTo_cVQG}S9KvD32(Nx21B!N_k`ID_jJp9;=OB)D)_Zlsbvbcm-2pTG;FX=O(14A?|lA{--a=(EbF`<)Xk
zqodB`S16kowrBFlkNGx_(c`%Ukv&2Z&UKPL$@askirwOC@9!x2Pk1nxg*=P+{v|YU
z+0I8rTad%Oko%xdq2GVfEX0L(SOH8HmjT7u2tAMJ6KTE^yn1dI1TF6S5W&JHwBM1*
z;mWK}kRgJXn+0G#_?C#g5tX7*_1*Gdk~<*iPJbJIl)dHg#|d;?9&15G4h(xteYLmV
zJ?z}ALRd2D*b|~XZLo?F;%vg~e7P(g>}b45wHZYk(IF)r~JUiNjT|f2#T;S%yO0ZtcCBoa5@D1+0DcH!*77L
zzVdh{tz-dD*IpbfZz^7dU#V{PX6>KKWZb$>_YN{wSbu
z5O?q}8Y~R|MY_6@1F?>0!MH9x1hj*=?k2%Nfi2ZwAr^tCBVJOHLfZt(U0?B3U9b+~
z8`n)>6^C$<%aQuFB+rNbR>ak$g`Ls(rtbI^>EE>zUL`MV
zT9fRI>KBS9#~`%VCSlKzQGA`I)}wBfBlj2sF1t+0cBE)bf%hKDAB!gssk>zo(hRL9
z7pgtvce8);hm?QPbyW2&(dY{b4wQnj4aBi>BSghQp1t?|V0&k){d^@|FD_KIarlK9
z>HvcULXgKtB_fj)i`hghsE;i&icAeS!oJmAB8np3rVjSdxGS}z+ISzI2mO6N?4ixU
zo{Ztcu~S>Z(-Mb{hM23{TV0@l7WNVIz?^zBkKzNbTJG>`L
z-r5fx&kAsXoLNMhO^}az$WIfLXopzVm8O3S>f>1$F*+@JWYxtc_C^iREFQHH=XRmu
zw2{M;tWyn)M?2@>}Zz{Z;(ed);EAeLQ&^A5ag!*Hr
zq*C>TPOI^Z)Q4<_@ME%Ph99?KEo9GpH`TjH+lJMMoKzo6kdR()y}$lJ;r&DpDNBDi
zwNM$_zcy)D?%Dto`ng8B{K9KN)SnxAeLP!w*D{j|X+-Q^ES6?Pp`!Y_8A)u)wT7Cw
zhk0mIvcM*C7x3+yGiizNuGXKE-DxAt{8uB4&<)Ez!pBJEVwB6C4wYmmy$&xmNq@6l
zIATL>w$^fI8P2lMqGTU*>#409E3to7x-S1vI+RJQQ_MsW1|I5QrXRsSK72sjPxV`zn;uzbj|PjW7`^pM~HVY
z&)xBDExQ?O8`5clP|JSyIedpC0qvRDh4F^HdMvcQX$WENH!p^k?G%-y({{hhbd79{
zF#0kwf0>55j)wBmfNfIgV3~hOvNVT*kQ3!A7g!xVY5Bej9!S_O`{Lgd(#mFYN9pBb
znr3uldi@+aHc<^lzu7W#FH2=m15e~|V=kgJpOl!?0(t`%?N&`zsAFp+R9a2~p~vk7^`s6|)$P|t^zHFFSX*n2bC;7fcsk)PZW4iY
zNv-*BY_YnF>EDdR1mSXShE&A8Q%qveJK;Z$<=B7y!)(ME#WN~_WY^iDqM2apM1PJb
zkksvv+8068}NfC|`zW&+MRYZP$`UCjiQ5SUSGR+uV#dPkVLygUahFd*B?Ov--
zAKjgNuH8UC{Q$oql}Kluu5DOfC47(UI+kkHF@y4EWuu(xGY>;3ocdKVn`0&&cSmJ^
z$i=kn8{WH6!;lumhj`3Ihr1mU)@Cd20b+PnIClKL3Xnb!us@E~4t0ODF{EGf<<`k0
zX-n{paPaj+QCxHR_Jf31`Gl?@R($Ps5|f0cjHQ3Ct#ij5d++;5WiV`I3N82kxOn4n
z59-BUvJpW<`_Je!Te(2gHC~;$Yro(wIse(&*fwexcO_WIvmjdIPQmf;l|d?ECSDny
zIWdzQITU(p3&-S;miAJEotMV%MhsXwrq<&6myUS|=Pd&W-i1UB0
z3(%sifbcw{{7iyzsWc$Uj+jz5R+l5drck1TkHDo^h0I4PCWvE$OsX;$!(f_}Vf=w5
z->A;6icZ_vujV?-7Nyq)fGT3gX*i9R;QTu<=XFK*GXyYqWnO*cwMpg9M+O3&^X(|)f=^DM4
zF1cX^pSMrbFugKK$&fV*QCZf>%-B@U)ER|BPIQDaOAdw1$Wj~san|yvGG>eVlp+21vAj$8=08k`HQqNigEq;)t%xZO8|kW~03~PIi`{5T{3O)LzAe82g4C
zO(+G}aFi&C@&dv4$AG%#CSHFUHX3HQ-1^G1!)FBIC?1g_zy!?1GI_7{Xe`DDuEKBb
zg4$5TsD=g|`;|%sqKhH-Kq~6Bc;YUTU6OHYsOY}^Y2+G|hA55kO@Z#MyWq(hA5(Fy
zk)HhED8N5e{4E~si>$@ai;;F~`z~=&R|%_JjK#DEsnv)qqJsPPveAE&g6Iy?XW9xx
zYLfx+X>kAqAM1E`nK@tXv?!i-eH5i6}klvt&B;|u262@y_bt4OYrq)yTRk8{ZMGZ}WRx|n*t*Cf6
zqrOmj^|(v>tA0%qSNDH%2(K_=`r;JLRK#ZB#A80+s3*f+P~qH|>3ugOai7%9w_5Ym
z3C$f!G8zUG#cep<{R1kcMH_p~JYV@YVz2*^D%Rg`AjVn!&}>`M%e6?BYDSlp^ogke
z$q33JTboJVIEa&{-+NIor#-xNu{&fVx~wUSz(@=`Me}XgpLUMV(q!Mv!xR
z%M8)DXiL+zFA(oTGhEtd(_Ht*ot;oC-B3hZy;3{1zbA5#{hZM_{bNbs-IBTsO|kxB
zrV|&p9LfB-fzqD1-mZL{Rpq&p3CdmG4I855y3+d#UkO8rlXs-!pkdkpGbgT<@0pvS
z=ns4b9EvNqoLhfzpaguIXdUM$MQ{{;%~veGYr5&~x!=*W924AQum(5IY4$0>0mI*N=rG_x+5Xmy=B}
z<*t}+mlI#AUl^>5SS{(}t+_QztiKX`tlz%mur@_Iyt{vq8i3D(w^HPDxykt`o%P)H
zhj9;$lN-X5=-J&J36>u>+11cL4#YIa%?b7?6lu_i$P6?+H=j0{
z;c4$I4AOtdCG>S*)J1|(%e%9>NZIN6&PQ&;(0t~K`#u-lk?Aua68Ktue=TBeRwh3z
zWv%=oquCqd#Ro4M7$h*g9c3CmvFNY4cYZ;xg`b7H^plzviQAD~Y#I_#8>H`<>xynV
zs%d?J)X(^xBl-(-OmFjn&a<=UtEz-CPmUwiAnkv!QoAf~=WFb(wX6-A=#WCqNN+e!
zb@EHjOg<bmJGyvFKEh@4>LOxdHsrnaKRMI$mnqFLJq`Fb6+=LFK9=g+mKnVlG7J
z7biX&vu314l&`Re22+221JC-F`9W59=HL-W-1Bc=X&sa48hPo10~VTJe>G=~S;dSQ1DigqkZd#yr#?;_4$)M2uu#fOPIWz9Vub-qg1FW
z?&KT!V|A0VcsI|U)BG_;XUAdq!dJ)p$Zd!=*p4VqM{%}#WQ*51(MvrYB0=2@>VH
zsmdvivZYyhnAu0*aGR+4Zj*qq;yFT>%Y1C+@MnoW4{&q-h==hiuk!d|Vs&5RQK7Gx
z=G8xMDiZ3m6l$f*f(X!EbgjElV5EQJ7-TY0GN~DDKK@>|YDF>(CAf1$r+-Rv0oMDi
z5QKFp2B3YOMXMP9%~+9r7b8me
z?n;~in`qp#=Yur#!~-X4@WrOr!8Wamhd8nA{~mh&hEpZ80r
zB20-zgpi4RryGG=S?=en#kUYU_#dt!3dHe{MjM7_ppWBIp7`jSz7cLIJU0h}{$SSB?-T1YNmx
zlx*EMo`IYFoipYEF?|GC=?OtOp~G-f3)xqAIpcWIef%BYi)NZBWr$cz#ird4G)>A%smBzMTkiU;8Rd2Qy8y?Xs(p?`hrn=dG>
ziO9F{sSA@mK7Lh~OSvN}#R=-r0*HMacdoheU{Ty(kwVdCB}YX#-=Ap2JGwtsmI{_T
zcz`>X(d7GHDcXMs`tZXUag5cXG18H?#wBvvWD;uSidM;g%Nhim3qCtTLLt}Baj~~@
zer2N2y;3*MpDKoXQla?e$H%g^PX8S>f5?p+HAs3EsZ++VOGz`{W)2ZQOoeE=_f3;u
zJr}xn217qX?W^2?BXx`qahgq==M=MI`BpZVFMOd4@JoOAsgsdmUDa=O-Vq_-q|*K1
z4h%$PQFnlu2|kw*?^2Rmd#a5Yfi8;Y!#}py;?Zq?9_WWWtWGzhe72>CkA7YhL>Yv>TdQs?~p|8La{#g9d*SE96rzt4!~TZvBZ8
zb>!JrZG$WC`=Xv}R?j`#lh}^KOOSFW)gKSP>sV1%q&!=9Z=kk-Z5iDK+0Jq!`s_O7
zA#9CKv|I2Uv^3_eSm!tG^yL|C%*^upVn8))j_M#Ss%jSn#GtJ$sMHA=WRHm_T3314
z>n49cW#y9s7_($6wDEGLN{7>V_*(AaY;xvUxPS1(7T>zntH3a`8WrJ|&1zhWR~1`x
zoZhI&?4q
zk*MXB5f)FL^C=_L0lM;!;bIg*DeQqoQoMidXH7~x++wF$7y5HmRmSt=e0?G4iD(U>
z{MVGN6b!8FIKir#NZkUH@tp$uX)>w7u=}i#$x7(TAGX@B*!*%47CxaZM{UU+ro3)K
zSNzGd1c5f~!~qW_k5&OB8&6I<-qb(7Uu}J`5lk}ws$Fi%V*IElt@Nw1-Sm%HW^R8?
zKdoMU@(i{$(lrJ|Bf*x`9yf6-0JTn49CP3hDzB5L$=2Us64;+A
z0`f3FD*5Kz4A+hc2YxBlBYjI_!15zc9t0t948GqXrzB?Erj^RWtf;ejkeLnj;<-cc&0?+M)#7kLeC-7KND>ysJ$j!n@_E!qHTK$X8hiy=03
zpq@(3jp34B8MYq%L9XOg#*fl~zGE$>tHw`ZZi#m*aB%`)y`#nQKE
z-{GAtaZu*1h<}jiKtwde3|v~XF
zRiVM<*zn-M%6y+7XxD<8vqW?s$!+nwe{4b9)%V8{YhqhSqP0qsw%-n|bmN3Rh)^Rl
zQ547gi+g=Eje0^HtXRe$`=A*zV6VWz5asKUNB@MW;nIZ3y{
z$$l&KIW0ChsSzFzxFO6pLf((%Xf7mUSJ`dK#<{yj$6V-D)LL!+rEuYPhY;Pae=8JS
z_$o7wAw*fC&n7K@Z|5XtJ5skmF?Mal^DHz1i*;t+xfB;_Df=~^v>_!+HYIO{de*bm
z*Ed5i00@@#w$|;8mTGRTUKlgtzF?XeEmW1)E$Od!$k_j{};v!}it
zQyW3`e`%H*9M1^Id_(o)k&SX>iR#~-2)Gt@u&fQCMD$sIRoTCFAJ@UgJyy!s-BOJZ
z_%Z)c%A+u==kCLTAM|sJGBv>>qiowA^UFndt6QXjwA9IcfJgylGKXT
zHHa(QJd;a_>~>$mIuNZc=d!Z0`A9jQ?cP+vsBKsBd_DGqVpoi3TQ-KjVGCn;!l~IW
zhm2q1C8=P4qF_Ru9(AiQS1K;eiXU0Z#>-?=+nq}nsh@`<0v@LDznPt6w-T>3L29qu
zO(gQd!2GnD9;$h;cD^n28Ywp-LYb;2tiF0|6Owix&%D
zzSZ`P;_kQAtP_RQr(eF*Z@4x^!v(4_uP3U73H!6TRt)2c_v-s&q#Qryh-%*yW
zzgnSxCw%7R4AhLyrA`RB9CnqidzJW~*#o|vF}$T2mb?ui1J9>fLl}9>C*M4T@kl7A
z>fN98(NJmxk+9D7l^4n596f8mO!KVwlaL)2H=hpCLmq8HODNeqPYL%iu>
zqsn^hkOU7y)r5TE=kkws)=&^;ltAJRqz2Z1O6o{LBq^`D6ycGV&|VTZjMS=+7|4eNjI>6
z*RwFX;g*eg7MjC$z*v{L2;vo3oA?+&0*^k0t~-^OT&UWS*%27bOoftm!~2nYi{K`7
z`^Bp7{PuoijgU?-UgOnnrfDBTv>+1tVL3Y7FmGazov{viE$|fT{H8)*4y<)hfnUqt
z=#crNb0;L8$CJk*-LR3fYAkGp|A)4J_m4pAzH`JE+#
zaB^D#+dj}Kku1ZYihe{;Ye5|*f4?Tp}GMi8sRTq
ze@q6}NKhO!GiRH$odj-n)59r-#9O6)9T~dGlx{0d!>(uA;grt(9+yIY?t_C6xt+On
zvkO=I#ViSr-tJ02dKaOS)tyF%I#iIJG{93qb%Dlca3iGC3K
ztSS~(JM5-_d@gWfL)w#ntUQ4WXB3h49<@_exY>RBt4wCQkHZxS5A1lq3;iC
zW9D3#r$Ub?%5TV7
zySPLHc2snNp}TM<$oHDH0@)br5HJf+dKq#=6_D#C@b-_D+wcZb+&};H8NVc}upXRe
zGgV;uR;beCA>yFS>ni#$-IIeriQvexoN$h(r4~r@=nXVWUwzpXt8s|`yyK`ytjR{g
z?aezc7Jkv?GX#m3m79)RxwY3ss^?|!huoZQXU9ly1svuz~G*3pTGI!AdwW(EgbOKzB5)b3-w7jZ(ubl4;aXN>1$itGLEG9M-(LzIP5A%;Dd-p0h4ekvY3cUYkAFHo%A`8EC-c)Q7(&Bh#$5XMEhXt#{a2t1m0#Go-*l>C*M_2x
zl_EP0z@~ITKyL2b_u=8AZi2kA4qj2QF{jRsP$Tz$Z+|1=97&qYds}K6H%Y-XD)a!D
zPFW^oaq(s}aPkKsY|(t%tXGcn(wZZj)r8bt~zSCMSTy9=f?X8SHWxXyp<{sCn-r7zK@2yOh)
zg7)vXlc^W8wge|1w`BUbardfPU(2p+Za5_oB{*8~WO`0PY1k#Iz=moBz+Hbve4LNB
z%S*5YyM}RdYqw>5@}~hHP+RxdMVEmwL&gn%JsMG6Qp<-jzS)6p)cl4;>8PuAtM_FKr|IQJu9vSo<}&8+Te(SG)J(P5OVV0AJD8wyo35!H!P*(ije
zaOp?6R`7RhBAnJEDDQEO&A(Uu%A89By?+QKT_H4F^jl*b_XidpSckia9>GOY%5ORjZ`M~*I%c~H{k`p+xZBRso@<--?6kS5iCVur3%
zd(bLkAL0mg;XCPSRkg^Gdh)-HcIOPIKUPLv7=+q$Qg@DanIGZ-dN%`Syr=>pBNP${
z9$1!gFh6(#G#&%w8z?Cri&fe(=2aJBMD)To`mBoNNQ5XvOENw*Gn47WtOsSG>OWN3
zH(F4L+aLb=P7))C6F6)s>m1!3@2HE6f1bE>e3{!H{ARoog}VGaTm5;!+93NK7$ZfA68G9WTAIW;nuVNL}U1u`-*
zGBTH;r2!~^du3Ex>$WzJHg#4R-hDjcZwD*E`{Rm?p9n2AKm+$
zd*t4~-^fT-<}-CY@0{<-8f%1-R7IUp)ZEb&B<1J;W@Kez;R7frsyW&N9k?0QK$dQH
zKoxp-e+ep%u3$z}pzE6&KcDOH9!t;^;a8-sze;;^wQzB?|0@B2#tIB}
z;$vp^@bF-=baMqWIl5RfIobVHpN5sSE5O5l(Zv?iKS1p1FORzTOkd=*qw6ae->
zYX>mM0q9`%W(Wp?-CO}Ce_?NLkU90=1%d$LZZ0l=awz^&
z0X_bE#efcOuHOH~&3~S@nWKZNwJX^5?}#9Pg|!{%PxY>U*38=BFHBKXNmfcyU4v2K
zt??Wf6&>H$abN;_g8#Dq6Hin^fe*mL!UbUE)lTvBd!c1l2d5Y2y6`rmdyduuzd|FHZo3vJM!T4|IVUF?B&|E05bm9q8(
znX6cX&8+?=_HSGk41DW-Q3p#q(A%Q?1!?{nC%d=4ecOAi|7=|VMphQK|I)p+q?xS)
z$ki3V&heKD^hU#f75v8ipK<|z%&LkK(&{?&|Ep*IGLv*Lb2PVhumrGiashxYE=v
z|A<_FBxg6E-Txq*Yyf5pM>m&$^WX$9TmPHL17QA_IR`I*+0o%&0XbNI0L)InxAC?E
zS%CkAvi<}8yIKAx7ApsU`QI|JasinCrR98c0$aI&{uP|_EjZZ2@!uTYYH|NpmN&^8
z|vumc#`c-a7~oLp}otZW>9|KV%)_jdT#_`NmBKjlAr
z5dZ}81eqZ&&N-U#huWkxg_rqD7EF}GQ1UXJRN@Hf$R$88rcbmJ;z{Igk%5E@gPH>~
zs0tkwWcUpIzXUlH>rjSb+r6x{Bvp-Ho2!a!1N{~K2~i|PM{=}(m^6bk6c+-^zmw4(
z%H`zimT*jEv?jC?12jjD#Ch90A7a_YUePa;$qdR{SMp#yie^~9VY%30crKt3bPz5~
zbi4zhvSUM!8&z!V(w+P9tdqY`j4p`HgjYVxJz6f3O|^t;;frtE7Zu6j#&4{C
zu1AmHZQheU0p5IsTJ#@kqU{B;c6(bqh2{b^97{Sfx(Hfz8#T&fmxa0Sm7PsnYoAmwD*XMpuDfuQAeo)g0Y~x%B#*JOa
zQOKH-z5#rH2PkvU6Qjt(gBb>8427wd*92_ZGm4e9!)3aMGS$0%
zRs8rU*81QC)v5Xi*!aR|eSt6dFrlEbdAdpnNR#6IB{|x{o0R-_Ie76J@AdOb(;Zxn
z7*XaL%ZrV#SL-?(p1?&*)STGJx>bjn61Mv@wNmnPbuAf~$Y+l$Z{$r+pR
zNGfM(WSC|iC;j`C&In)|XLPQ~;l@cA(@zxXAV^R|O8!3`k
za3Uz#A$|RNsa(mLV;(IyAG2`n0bUb$ZZ|ijg^X&4f1*xKZjiPOb!wVsyw+M##AIJC
z{au=z6=34AsExB*86kKc{o%@yOj;lclZ_wa>>(z`wLq{O5f@XpnLAD>y)N)N`z3~d
zG`cm|9zJOA=}Xo~fxNN)k)fou;=CL|j?)?KSL?Fp#sy#e4T=qdpQPnFrrIO&?*iIR
zjXAUX+eNCi8-8%5^jhs~q@$3E1$TsoG)m#KuVmBXL8wl5`>0>Y6onfe8n5uwpmD^H2r?(tPDdZTv
z%�i@aS!{Ys}9{$X;jgKJKS*RYK11Ao$|nhjd~z4|7EVlr+{NW~gLhqw2BH9IIfcLwS36KrG(W3-l%g8(#tt=W=C
zR)QDmToc~C4R;b71+ti@3Ke-(>qrtrY9VG(L{jLYOyh}F>a6`s?l4$a2=*8!mk2J6
zacHILpL~{2@8Sn0^7jsytVDJRdHS?-TGYA4Sj{W9z~wxv-7fX#qI3En*t!eDmf>4x
zipFsnwt!Z1gQTztB^Cl_q*oVzqEy520Aj=?so$VqUGya;c4mzF^9KYUXWWBFNCp>S
z$X`i-CC#5-)Lft_i@%2ja&g-!>sFWI!lOV-@how&DpyuR*yh7Zk;gMc^?98gHz6E5
z-pL=*e(ViCc-?wkU{RXfuT9$b-2%?#68J{nuv(6n@
z8td|}!Hz~SRh3Hy*mij~8x?p7!n+&2dcXMf26`q;@K%pi3acO@XlDbVpOY^?mw@|e
zmOw)vy}Bg@?Mh=*WDj3|J2G7%rhBMRli7o8#)H_JM}$7Yp2cS!^h2ZWv1~s6ZpnJ3
z<%r7OC>#~-MqpOHoDe*UnvHCq8nL&s`?T#Got{&EM}4_K1N1nfZk-SrumobeA)g{e
zlupwyxD0S`c!E9j9qGk_i%nSOY5P_#4>wNfq9_U&swPmHzrZGcruInoob^zivEypO
zK&DB;EDh>rsTXkh6!gS4;qmQ#0-58+cCkpblsSI=G;xJ4huKp)@?ACkllFiE4Ve$e
z#JjTSWIWTxRvDtaon`rGX7k1~gMrg!&GDJh2o;xCuj`7{Pha?@328<|Onzl3+w>?w
z?vvv1){;cE8$uF)GAkD1*8NNsIwj6U-O(a=x4g!B<9;jQw#VGb0w?c8$ZKINe~p<{
zZ}6cIDG4`knefN>dY)d&&PeHLhjorV@YXC~kKW=r24+B(In
zQWp~_>ra^@#WA>UCI6FBS9Qa1Jmdn)hai|1?+5u5f=tTTy$f&zkVp4ko
z@OdVikq%(`$9b?FL&YIO-7Yz#uFoqf#{9a$7QVA3HXN(dcY168YPZi6aaA9@S9NKQ
z1W)?MqkEtM_RDLB@iKc0)8tX#YAyOkRdRwg(Pf2ytOiYo`0R@X?T$=8d^+c0UjY3W
zI}>76f39Fr8FnjWN(wpA7pxnuCzFl9bMDaG#lCyaQQX&O=zT%hKP}y!Z8b4tc{Oel
ztGjk;P1Im2>U1?=Wj;zL7^KOT7jIFYI?;+Soppu?De+ket}3ZDR32$M`!#v6sPfuq
z|3i0wE%;E%HWmp2-9>0&?<yWmoso7XQbnkebJAVi@o4{rA_(}Is13#
zWsn9rV6R)nLlil|uQ`6Qk>4+y9N+g(rz|sn&LfkR#_B?O!*r&KbJsqRF
ze|}=*;5_F-h;{{5S-{B^$yUodJ#Q4g+-oupmjwED)4XRmN2}dXNytplr0)aXSdNZ=
zbc+6>4sz@IMHG^7+_WEOo5^5=!HKTxgI-Jog-(;OmE%M_I#N?u4;FMPtvRFz@*ow1
z^I2#14s%kEMf`AgEhf2l<4>q(3dlKDXe`RypFN8)k8w<8s^0yt1ljCWYF@@5)|n$M
z*&sPw^%IBhN%)f%81+YkHfQTKGLAZbFPCbejg7!4OVeaevXOgj+xX(XrMbDGGq@Tn
z=0xLn6Log^5xYKNv-bILKhlOz6eRR(zc#9EPn0D4P_Q?3c$mie{U9S1kNbcv^)5);
zQP%%!dNNBc>?;EN5$=~hjMuEIVjLqNuFHG5Xd=TX^-{}4Ff-(rn9#|6X@o(4p49Bm
z>v%!q+bBeS@q77?$D+|U&j_GSA8P5~p*eW#DCozeh0$k=$bhSUGdwNRvn|Y%aA#69
zlrW-su4-Z&E?n`n#Y0@h8Q6PH?q7-_lf|V4w?gOIR6oa%f4fZNh>%7Wqt%4%T#cOp
zJCYG}NEd4lv^*BmTz#G&&J<67@kH;SxQ&8nW14x_OO2|}$-3FOI*tAO$G#ub&iIg|1?oqDi!@EOi$7|y%04Dk2SZ6
z^wf>5i9UIs#6U&l8s)NR4-tkI;WSi<-`Eblu82`e{4M7Q4n?Gp07DgjWAiJlE3^e<
zQUStAoZwuH!6%B!OVa|IV3>iZ+~&lsjwg?XAR$#Zv5G^rN!@ru*^l&2I&?ulx({Rm^w==1tg(7202>a$}ROp9>QOIyPw&X7Da35;MymoY1uE
z6_eVcyYl@|=UQlMS#@uJe=ViPIR3KrI}iO$HSHy4QBjU_RYqE61d{B-;&ANvd7DFv
z=fjn{G7b4fW~P1TXT=1n4QosAd+!=weEKOJDC-91XX1
ze2y2ZpjIyku%pGDIv5FhES)fQdD~Qpum*1OUZf7fB1s_6_R)5K-VtN8_!W=~BA=r?
z86YRrZA>-GFq>)+)59mg8-y^<;Y3@wzLP4)^Y}IP=JY{VLDYY820yX*3OsD;glex2jKz#7u*Yi4_s}%wm6!;Js6x4J&7dJw8w63+F}H)W;1|+}Pt>*&ZWFw1
zwK1cJ8q(Bh&Pd=~nxxDus%S3#fajkj$5U4dJPB|u^D2hJ?Tsd%AJ*s)2c}ESzWkn<
zA}2y!Puv=Rj|?oEtd$6~EOK
z&y8@}1C+Wbp^(EPwW~$tHze#bTR&LALj$A|ty)KasATC#kjYd|m86iH4{^O$65qm)
zWcam0CcIt?k|*8%s2`=`VSlPp5p`$z>)YpArdsg)Y_FE)bTa=QI(^>vNpAoNGEcsSCgCMgGcW^oqj+(eXCqQ
z1Cd^T2iK=M_te*qaRa}H!$~TQ@6vxyQc4C^Ry}E1~h~U|QH29I9DL$zi_1wPc{Oi$mBw
zZ6fosg<4!Qw%)aN_7R`sre}XCsVhO@OK|E7X0??3?0tO6)i8G*rUVtAE+TBn+6o
z2k-AY6;{8^rxPhZ_!NG-0(bWoI}^XY6kS)vPsD8(i2huB58Yn;g@avU2jo*%-_+*a
zE}6-#nw%ATL+UROB0;{mIP`gC$xc$r{i*N3*6!82mqh0Nlv29~PSmuMScS`yiGxP*}spTc78sF@AfYR!JS!d3@rVwP4(+$D@MCUr%p|M~lP~=KlfIP`)
z9rBpl8%s^=v*EI#(+D4H!es
zf382*)9_|V^0bMDf=#qN5&ZU|dLvy;&3XE8)=!G}GOWO6pB2rLOSA?8dq3)1XG9zO
zB8R_V*yh+FdlEI}K4a(yi0U_=YM;P^w`>cA&LoMfqt^vDb>eP+n8z-Li@m#lShqUp
zgqr4f%5-FIEh+6`GuQYNeS3GmExE7!vU5r9r@#97_AazAePa;S`AGCD6VW>3f!X_K
zV`j|kxcBp?F!8U$i%Vi`Vfl8@$;eM@P?2eAa6{F$8y_B#7ax#}o6U&MTLQ1%wW)8G
z7ClX2E2OWW<^|Dz?JdihVp$M8B()PoSQNheMrU_OG6yp^BBWw%Jl>NjuzcOGlFE(T
z(h~93R9cJqniua~h;y3YHv1zXy;#)|#iQR}OMokCj&`;Bw^wB1LAX(;n)~b*8FR2@
zjzjBwG31aH*tIJ_${AppG(ZH4uKX&
zDDKw8OA>EM9Ilc_VD)k!9sd-<@oPHFIDYIsqcU^tXZTMk6bt!9#X|j`2ciOS5hi-M
zjsxC#H=AsKvYw_NR;z|@k)5mev)Awg^fZruE)_sRk9g_saP#`56!0^{iJ=R_S2N{*
zvd|uM^k`Bv(fR_^RcX)zYK{l|5c^IF(+qqZ=vCh@^hPHaPuKxZ>dN%7iHVX?xx#tO
zfrh7ivg;t<$Hyfc5WfNm=Qg*n~qJY
zrSS;Ofp^uHHqM9zkykF8Xi7Ubxy+)UdPsA7xaWO~sK{=N6m=f;j0tjs%ryPwp&ZOj
z*wEL1)rW#)K_qnGXL<6X{vkiOPpi$HsII2t`SNvFGv|K1oAC_lFaqQ)B;lx}t+w%D
z6u0IXx)I4f97?lMO#3eQ3FVOxO;0FYul2;1L8L^hwFyv}DO->K>wXV>^GpU&^&1Ob
z{SrUQCrJlW4IJmxFc!N<`Wr^+qgtA$@hv`o%I(M15!0ZCL?7Ij`y%c{;l8Pwm9?nW
z{9Z1RKy-~O`5fqnOJn~)F`3YsD4BZ*&(N5~MsKcWmhidi87XAv&N^*32E8*e@pRW9
z0qrM6R(Tsb;`zHZ5T#o0kBgnt8j1&2Ui76|>Fh=nWJPH}z&=l$v5uNLJk%Tom`(Pc#xF
z1*#B?%mlj82Je&Y2TuoS-pGXBY=q5!DtVzagiF9-CYMigA9!7Wk&<`w!wsB~W1#%O
zFPV`FR%jpWFxuTz=&|U^NqNqsNP1b~X6ND&+tCE4cS4pG*AV?!ni(2m+@RHYmA?Eu=
zI)X>e2k59DD|i|yKxZ4-7=1Fdq)JAF?Ix;uzjn`$N(ZMF*7j?x9d268`^^y}P$8#+
z7tyFwk(|1(Y1c6Anz|L`eEXtkY1zoU6pT7FK*%AAd2WR_ij_ay*^%2?0W4^30)5Aw-rr?tCtTpV_xP+!eI)Y#}!C#ry^W`!I(~>9HcQv
znkqf-uyt@xB!#nne2i9pZl9ZSyUnhCD34M1VvF*4?Zi$@XiQh@Y>UfgMS)*PX2Vy5
zgA&!6?pcM&a?L~PSoR$nHownPJ2Lp7?3Uevd@qd;CT{P~9GzA8t#YVEw8!^KudOSr
zi$~+qD;<7`{`<64rkLMPUY9yS@DvO-U5iNlBNgr6YVx$k7H;f=Q8GGTz_o
z26M4D`Q`8#e&H49-B0rhAmhlhQkL&1
zrVRCCA<|oUX@}SP2-(Twu`snZs&j_na>eokHm#7MUzOsNKgvIzJVM&uA(4S>gl~u#
zd(1E&{S;vq4$H3hwV-#Zh|shSW-V*>9h2pAoPDMu$`l@c%KI>A=FnTJp&gVa)D_@c
zSt~$)w-)J*+dn=s)bhI17fwOjk+1x+CL^G!Pxz=;^|3Y34E05*bslx$l7(L4k@Dl(
z_)OP0eOylOD?jM4A#gW~WR0$SC-5b|D2Il6z|+6E1p(sAmNsm%1s~QWoJsl)_`htNf<27-(8I6)CZc;xFMM)Ib_UQHoAyq7MZ|S<5m`zwa
z(@h})HaXe-<=S*{%a_>n&TNLsMDV6srM`#qWUK&Z^q`oL+xu$FORY(xhyc^9D*U&_
zr<5+^$R(AuNd{Vx0=%LUOH3+%a{JK8F{Qp?5p}Do8i+QAFRiczA+T-37HrH@uuM1!
zEuDNb0@4g!QcM&T*X#P<{H8T@nzlYCLh4erC=2b0geZ7GdpR@F=>b1oUGOCNe^C=*
z;FRkf$M?K{z9(eUr`why`?3DHL%%lGNUT5J6!VK)IMZ@!io}LP_v8Y9hiVxY!T<4K
ziw+;#EsYw+^EhNyXMzM_?O5L5o`G)l*@~%hUjxdURI}|8pVfgXk
zyv*0Q6c@5fhNl^imX(fRxHUg@5Kk0@C4$o66TbB%9O2PQ)wN=RI^2dBG2K6+US98R
zt_$qdY@hwOP1EBp$b&(Dvhe^^^W~oa&A-2Df^h4x^A(FwYHA%vS4g2xivL-oP+D@$
z0d2R3ZYb%Np)`Z@F?&h43Mzw-J1|p}x$lB#-ub-tR5hDStrQ(b`nQL=D)NgBeYZ2t
zQ`Sk-lN9#{XniWjJKWZ!K7qPA;v27+Xt-B}#1nc{uF#)Mf~zxs;`|5oDpqLn&;;`?
zR5%SDKVOt(Y+V+Wf$P`EPb;vRbD0b6>(7Xh{cTf)=X>>e3B@QSJF%7{yi<_u*MJ`*%VzW4{
zCu5;}k~O;_kUNim)jDo5z1>beBGippp_fb#s!c~g%r}aK;Igpoo+TL9t7`lZg3Rt#
z97>pd!oicrke;O2*aMrE05=zKHyq#2tN(=MpwDB@5>!T%w`es_
zver5=V7&%-4dz4JIsQz!1P?b+AGdyg*tS!NxM)JgI<)(?`ZTQCBFF9alGhRfILU&sXqd
zTtQ&b-Mq*eq_JHKmHJqH8?^y(wXayrN)y^
zaM3?2lczy5smkh1^1yjXiVw7-E
zLyam7QtNW)l|_nVgc4-3Y*R)kb$iO+(Rqum*s0Q-`G`=cdK1au*`D6rjb9j{B55|(
z86V>gP8qlw_SQA7xr{3CzbuHSqGwj
zYN7zXznSk?)O;BOr2wP-qKpXK?RV9pAGbmi66<@EF=9&C4bj6a#bx;#wKL}
z>$8x1fF$FgAxwk&sT^5(a=0b;Bx&A%0{q?t6Y|Lgx-BU8u{UGNc!pF11@)HJZ#$Du#Y7pAev;3AA*^>2z$(-8Di2>
zvj5;t$s;J{1Xd$Ev1Gh(_#BC$H$R|Npw*8Y9Ln0gw)Jvi0aBj;pyIi$@V7A_!a+{<3
ztzoiCsJ;w1_x1Q@y}_Qmg7(
zV~GzdpBW7P7R5Oh`yG)MsR}-9Vo?3)s3`F|4K1LBiEP5jdES;|xy4aW+2Mlm(g7Wq
zdT(#cS7d?<|C|L$xI(~xa?5xH>FW%-PD)UN=4gbRck{jc99ndGcW}k4eI%*O(}>{U!gwD!Z!rc=Vns`2UaZG)ZXJ+}
zzZc^R@Aqvzc(`YO0-d&x;7VnRJ(Vrjh`g&UG2;H!eBybgdk#SzBP)Oh(#{Pg{C&CP
zcg?!?443?o>dB*c74`);Qmv}J;;X`f-MZ=Y
zAFpnF(WlwNgStMKyUD@dpm~5}Nf5sf-pRYm>XmhW6%e&^Qz(gyk$Zi1{U+^VOm?{)
z!0zpn7p-ly;Ndz@Us^F#Q7iP5pCSj(v-Hj~Nt8CJD3><+Tnk<8p7~Z`nl=`lg4(C#
zmyG~d;b7vV)e7+I#`}ah7;`n{Y1!15%Q)HnwgQGnj3kC5ho|;UM#7U>b?h-wFK76W
zdrzZ(qy2J$T8DtFn3lD9gxGfBRNa~M)=n0z3^o|1E9u0aT!Ub<&BbXjZU&pna*hnY
zl2FKxv=afs5bRE~-qwuJ>{Vma_x_Uj;(4lsfunreLGJ|%I0jq4?vi0k0SR;*gVMBf
zq8rsvL-7&{1&F*pZ)BW88y+Lj7vWJekmD;W)g`L|WR8%p`Ats+}o|
zcND5L+{(G$yHE;Z%QMkD8)9X1fqMtl@t)v)k14q-&k<+U1n!x?wtP6H`^}N8vFc`h
zsY(11blK9T?5zwZc_i^V;;DlvuST>?ca*PG?4Co3de5XG!vBna);{$6erm
zIzw~j&+bMsCO?KSOwN9}J`sl^G^!AYadmWf{*OcUK{eDJBTMbtnrBAac$&l=8K2~G
zdDgvj0?i%`Fg#=K7$3uD`t+C0NRM)oB1i=|Ub9wx5yPl~@#LMN5Cs9B@Iu4ufAJEL
z12`aRXd@CCrVz*!XH)&VbM(&-4+Ua>??0?tICGd~W#nN&akXP{vQa4KvjK-Tz|bS@
zx&h3YUoxzd!oiQ2!XBTEDGs6x?~gi9-7<~zYPbB?I~=?!~o*#RF)-E+t*Z@
zk*0zbS4+nYoZz&Do%1!+hC3juM5*p5Qfxeb__SD6=Kw~tLl#9
z*)Oyd^|sA*>Nk2BKDK&M`kE(y-y~=z*W=67&&VFyWJ9^aR8^hzyN)g{ug(Iik~|AW
zItXle9dkB@+Dm?D~W>TG?NHA5(=woM-^
zp@tFo!<{CWx-2&E92(9(HGApaz}+d46F+y%)HHrM=>2stykkpd&w|E(X?=LUT2_UX
z3u2oCo%@<4^pXPWI+YnZc&uq;XP~c0j(RYkj$ls?T-&a_*^mm^4Hd#&_D0?ZrF;3l
zdntNs$4b)^zkrA@jb>5NgRDCZ#XQ`
z^OnJd`%u+vZcAE6zx)}J-FGpTg#4t7k>B}~=sx~W|J&HqIxKvn(Y#*Rhxgs%L
z*t)_)ZS!jDI=5{ew$Z0QG)z+TSInR|eCFBI%-DT=&>ebhYp=|Sm{n7DLNK$+zT!KX{*
z653=20T&(np-a;~>$Ig9RG2NE*i!d`K6Haa15`G{k6MIOrOZuqC7l?nFlul$Sw^)7yd=h;3_SPK)6r1
zW0(tUaC^Ws&+(X-7U@_)sCzu#EP1Whewxyb;ZRNfax}qG8%s}^7|YN+d|RLXQ?wQ4
zy;Kga8+pKzpCcA_e@WMYYtr4Zn_I8pRscZEG8&H56hk3M^A7WV)K_A>XW{VMK)!%c
zgD}*I|=7_g>u#YfbnR6s00rXIBe|s%B{grGl7og2^TuEPFMRbO$uNU?y2
z^>f_k#I-wJ#RKYag~^FVsP#40CCG;Z1Sd$;7=^5)lNJq{Bj|?mC7In*KKerbGFV03
zZs+QVHnJU$f4TfDHlDA|YoZB88tazZ&g4}Hb2FV5au6Okc_m#;+Zfv~k{ivI;ZO>i
zaIzlGnfux?*~N##Ls9^AF1yOTS@R(xBD%t&2D1-U2997Hb3*CYCj9_&C=2V4DXtfI
zg-$_I1F|!W9#&7~I7a=wGlJPEz5
zJeA^C?S5Ww2;b9r0{o!ZMYkOB@tWnUumWIqh
zPV03lfAJW2OT?ZpH~1;`W`Mib%*D~V-|Z~1z-4LQe!|5zl@qh+`cUaA&d8Mzyocf;8;X6h6|3u6LpRZC+B
zVi3T6*Ze!$GyQx|;(>Mp2Ew1sc%W%J*WrsNfAuEA$O|h6El||!ofiyB3Zh5abOUz4
z0Zm4|dYD1xz2JkwHyXd0e%5$t2=VYX$naH{<
zB1_k1wU791AY1rujoWFf9+ecA%}KLRT>l-)Q^d6v@rsLyez{Th5}#V_kPE>=M8j2`
zf1c!E(MMiO^-SiBAdc`Vx@X;#kGY!K;-e5CgwW*QbFh4Lpg2D|U!}
z7$MwKswaTTclhN?OT$Q}WCEu!)2`0%GQz`+SZz5FBWLZ&s^0ZMYZd+A)%HV8f8K1C
z7~j_XI_&pvjwH+RG6|~Ps1Q6d>%(kDIo7jfvUSeLa6MIM*b(QUOi+xsJMWTM_X;)_
zCr$Gr+Tha>9MTGfRQX!F8Ege=KM=<#9@pRrk}Diw^AUcOKYm`?4Dd_mw71hPFIj
z0MN@(@k8e&a4z~~Y1t({XHHHda-L<;vm@h&X&yWwwja3dDby6-cb-4neoM8Nl;T%!
zECkycVNVx3Kf9k@z|@eRzTHEN{mm0pNIZRam48e%aYo|}`Rb0ILqEw}e{X;BwOirW
z=iPA|D~~V+lgkO{=(EO+q-1cfw3Ro)waV&5+w_n0CC*vLMyyi0cg7nM+e+zlb6h&F
zk5m_kYLX+)l=k2H)4S?s?2snQ=MUE@yw4G(MciUZ8Oc~U3BQlV`PT7#l4+1|p9OCy%!BO;-
zbVDUIJkq$dc?9_plj$7C$>AvE-LQxufpAOsDD_%AW*(}i5jj|)mIz@TASe_32$%h{
zxvM_wu(yb^|29NJDE<00Hlu-6MHW^u=Euq0v3E=ie-kT;!4blte6ZBs){MjR8JnMDJc-)Ee@c>|4bOufIx_c^a?M)-uc5Rak24
zQ1#_Zh>L1lC#(StzpoGf#79B)7%6qr;kbnQLER+olpcj4R`)TcLEuM(ZRKEb+yi&(
zv`PQUXc|mL`Vqrpe=+CMDK=Q3yZSkrg?4QFJ6dT3(-h^|1YvQpiLYuQd;7IaE*Eme
z{1_hLq%ytnp=E>EI6CxWl*f178w=uC61TF&pWsDG_@|bkQ~ewB7b7aTm(U25oolIR
zy6$x=KS46=
z0^)@I1GYf)e|J~SK$`!OX&m8R#ad4P!50p727p=+-&4zInwfHCE70&GlJ3Y0ny2NB!K&7tcT|sGXe`JD-S3E4g_D_
z&-6I?=sMJ_aty~Dvq`==;mEd(O=CC*A{{UuqdNJ|`hK{G<>cwz;?OH?A4o2k^j~Hv
zS+%xHe{3bkJ_n!m4{Caa3$E9>L=_Pylcyv5u1%)j6xw3_g*&nj+T*Yrm16+cOV-HAk%~xE6K~he=;q5gdnue16+VHoHsGf0vNW%mK1(j
zQP?JeiC{B1pz%z(A5aTNpop&+XnQ^Wb@v`+m3}VgL$vd^>1AKF{iXHEeO>7GM9S9*ji)Y^74KD1kuvm)*F*9#l-QZ}Pqw8JUJ@p2@=j-5r=h9T5
zJ)K_DW&QcNAuOG=rs{0DMkft0W-^6Wn&vRN0_2g7roGUK8?{syziw72Xlnn2hZC_NJ|O9ZjVZ%csJ(?5$^Y>C<#BDI@ApoCFn@N}MSpxbj_GWDO-CHkoI`<*X}c1k
z3S}9!K8wux8C&nV{Jz02X~rC-|4zc@w&A^s2{Cem9=
zX&Up{_!-|E%1@_dTKzc7(UJ>Q1w3ck5_0
zsMax%f81}w6ujNHpcnH+kOKm`9dst0=>YOpj06+y#uqzD+8n63Z5e>J@D37tRi3LF
zZylN$;W5uOiEnD8<2j_Ke^VvH9FKY~3MQZstC_cHYxQ>3CX>%zFBkWk$6_9?Xe_FR
z!ybQd3kIbLlSzJZKTR}^yVZQ(^zF87sbsl)4W8T?~%ah-JLXojvcms2l%Jm{6vf(8Dx!PWpaCOXK(iWW-m(kBcmb;i=p~Jh6
zJMj3mx-7(n26v>D@G0YSl}MUZf>Ulx*sDy=7KA-@9$hQG>Q6vj(+0i{&SabntQK!g
zte2RR)1HgoIdPKYe`(r;>?NHL=oi<(M9gg7RVXq^mZ?~-98UT(p_Epy1t(tOVIBFs
zN?W;n&?91s#UOCsFz$HOZ^*K6#J0F$Td6Gjp5I}9I&(aMh*~5UV3gtAlr*JK
zI#5!OyCt^gPp0NG>5DljZaL$bzgogx7W_NrQ}v}Z2)ht7e_|K`R|IzG62q{5Y46Ap
z<*SCEzN@TAHxpD1w5McD~7ZrhXd>VjS34D?DLEByGIAk33`SmU^=~i=nB1f4^q8
zkn|D4+Yrj$1+2Vhhr3>+W&aHN&70JcpVc-eNrbmQ(VzpLhUiF;1Ncc%*n6v&VVeaL
ze|}NvkpgLgQ7YI%2J9Z#UC^fUO?ksK_xkO5?%i+GwqZSmU0xT~0u3kJFx{W_a@`Np
zTBoAQOI2|KL7hMR?kO-6vh5k;eDJj=(N6%56HlOVALI+jsUhclUfpArBb$62#Zsp1
zDvEaKdgIU%RV6Szqs?)tP;jaKL#@ive`-tyaS4CWw5Y0!+xmjNnhG9Z{9k`Q`W?Qs
z-TMat8^0k0UBH}`$*xOI_cf(KO_*PAYAvvm!oIC8#rLnfRT+$20w7MQ+ZEGP{eoVQ
zQ$vOU=rXK|Jrr&~uHC)bs_Lw_FF)WW;Mi_Fb{mW2pzQO3;z5CLs9vW1=7w@re@br~
zaR@t#W|5orS_pjHF}f!CO){LzE`B91OLp>xNOy^$Xbs2i$X395ndc3Gq>-!ZKo}Wr
z1cL7Qi7-Mn4uK`6{2az79=EarNz}!{yj}p+SF_y$!iVMYI>Ze#QJt}?{-mywwSe$1
z@<~kpnC9)80o+8t6shLN&KCe}e<4uz(|Gz!7(~F02ILH1;@a;Ik}4WAWs`9>(Tcr5
z=IrNq&MZA0p)YvM^S;jg$^i#l2ey32_Q`>wVToY{6AI#4Bj@XP!2R_oXMn?YJ(wY9
zJ@8vR$K|Vh1Stqk&(6Jfes8t`LB1AVZ-7ha@JP$6vGIHpRB{6W2+!@We}0W04Pjip
ze?BlSL@?N|d6+#>aV!T+U;bxz*5h=v55leF2#CepfQx2)5+CW^-e6M-%l%XCBP1(K
z$59l(6%m>TlvW8kK*0=^|No!rqz57ozx{8r_tle}$+Aq`2tD#Al>Aw#UL+?~YLuJP
zgRJjpdRX=t0Q*MVV$XIbf9*QdMQ%E0zgx7KAol)dAl8~vf?!i^%uk+)3>fI-s!Kpj
zu#0`Oy(k^|VfN;})S+cw+%dv6(hDGCCdcE<#}z7en|9#9GMo>qRMeD?W=JY&C{dtsCaoX_8M?sG`3oO7r4HYe92V9%@5rGG~auF-cqD>t}It$~x1<
zNE5(%w8yJI);3Iif2)ZtvjI}pA3vf*Lp1wTA^17B{$Zo#jK~Gv;dv&d`&&tWMl!MC
zf^7Z-XO_5`61>#j__jpF1;ZpkJjtqp%L-BUs?epd9ptU$^PYW`+MVeBQE4OC%z+Lt
zYF%9Vko3b46eeE%k_O$~)Lql{Xd}OZF^mUis3=81I!(-pe=R5pb{)18L@QXy1Qqr-
z^0p<@q{VT6daDk`-nn6LyhVMKf!yRH?Pu5re#AnapKs%xY{c7@FbQo4Jk2LAid)70
z)ZfK08&ylkA@$pF43@{a%4B{-_s9sb1J+kHKDB7vcP;1+qQOR>K>J52q;)2yM07h-
z&kHW!vRNm&e*hk8+N*}TYb#DD^Tt(X;B~vm$y^w%J7J++V|?J)VSd|_8EMwx1i>1S
ztgwevFO+o+#WFJHD$G1)4Yy;}bMKdYOmhz^H<*1KmjvksHriwbW~jr%uw0tcz@aix
zn}I#aP`L?E0aYua9&eoUDYEo63l3Kta0dUwGN1|9f5jCOVGp)&q9ER7c5fav*c&zw
zcuPnXxX*Jj7)O!I`(Ik|*vadiPPq2rE^pPQggw-O1;^>0Eh~8>qYOw+?2qpzZr_OW
z
zW;ofJe~KKH4UHgI7A~I!o@X7N_M%%bY$nYQB}~|%9@l*Z{I*Xf8rUpJu?O;TCO-M^
zIBx4gfC`YAV~eCI@JBSZnc8ex(t=nd@ePnzWij&pIFCshpaVec>Ca+wn0VjpY{6fE8Mb
zZ+=3M0}h!K4LLI8^>Sn@*aC2?wqWE7ZU-G_5dDFr#hnk5B-#|I4-#2x?hB2~KUN`}
z_L%?Eyk3dG{at%~l`i4cqy)IMTP?3+e{8`49H4wm5x5Ytikx{=wGu`;hOuHh)k8mJ
zQQj)27Scj7q9-2%AUX6B_Z__T?pbQpw__%0xQ9V0g6)_l4Y)%$i7F#1Iv!|?gY{B%
zAbl#LM`;r7cMv!D)f<2@wYo`+aolvx@6+Yi$o^ExrtCK&APzz)QWPR?#{Y#Ze}$Q7
zbIH$&9&9PxLuK3;DiSbJm#s*tpROpFGzO`|wVfXcV(3uFfDJpdq(#J56SRulDg6wo>&ub%-h)k6@SJ}K6d(=s85fS75S0h+gUK%T)U>{aIWfk
z@v!GV)526uFv2u9&zwK>)u`zbfAkbEGGCn%fJ_VPe3#6Cl;=2b!v!6r)mr>i!?g11Pz~CN~4UZvhVQTW48M
ztRSwK>ePzuxi+Gblf@P^eq9-^IAm6>j9qPbkQK7bHE%#cmvL={>vLEMsq4ujIg!Dj
z&W;}@qE{!VQ__CW?mpz>e*_$IBI|0QwGJi+lw9feNY&!pAy~W^n7gKS%qN)#jo2Dmx=9ymX6{+a$KwTJ}Z^|elB!prBe3<>+1RWuWDqS(D&ug6$K<+k6V#1
z%`o7jZ69(0nh7~Sf7K?rqR=4`8+KR=#HV)MhO})`YY&L^*=?8n5?=s;UCOCV|DfgS
zjEzd#$u@D6gjR16BS%e>z+4BoXrIK0R+;3xc4p>#l|;L7t(JUCgf~YHO#%|HS+gM!%?ciD-J%k81AlbFfw%{;_e}1#a)(SU~Fny~TjCmMv
z*B)Bm8BlI&m}8U`Z3IBAV!n%6ZfBM{@vc{k(?9}gRGclso9{NF
zM>i(4JNO6*$Id&t$}MxBymILa!03k5y7OMOW
z1LmN~r8%eaoyH7bG|yLoG%gR+?}?0p93zhKP4~TXe?zx&MrpJ=N4v1!)&E>nwvvU(>!?!?k1Wi8j>%qxhnf{jW+ca}69;UM*FthG^#vbwq
z-GlyqPj+tjbYOpFczO9IouFB&7TfHH=zbPDLFVVWhZaZUl8>V(<|;71w?6yMZb9b6
zuJX=ue--)rh|r(p(fq)%kM6?eJ2b2%pMwGQo$!W9X
z+F^gbv4({^uUWlmu}CwdfDM@)XWJX%#J$f5>`5sT7X6)q4)`t9uTHnBTx}Z-tgllC-nucehm_^|qNhCc+9K
z6ym=w95?L`LHDrGmF2UnaPh$$>zd-Q!_fpB`Z}Q%W&h_B4YHE&@;M@Fm{=WZlqbpzZT?qbyC(7r!f2;9?WW
zNf}hps8E$v?bh?Jr*rRjSYQ)aphK^&8ZiLYcyFtbF&d*2DvB&BRm_tvlIXGFVkHE9)e>x1V1qqoZ38VbXcJ=K{1drx4Fud-3w=|lM
zYm|ZYJhSB|ls9Q#N)caXoz?@N<#TQsn4pT@vb7)Ce3XAYuSyrn4~yhUoYq^66v&}2
zqVIM$wmy+BnolplMBRoz9RvPWWlw%q~@TPUUVe};b@
z2j>KU&B3Hq=iCVr>d19U;-0Qr$aWUM!e~{)ORT?fWC!W?ICCR?6(dK}sw`Js91X0|
z7SvvqU=T^LKl;)*a%$@OAUe;MsK+2CUl*TpCANnfSN+KH^q1@5s5J7uA~WK03ViQ~
z2K1E5OT)gIzE%eekV{JGf6*sVe|S4}q1$%mICi%Cnk$RKlc&PR7c+v~?2%YHELrt1
z+*S*yR3k<)?FHN_QY=Bd!C?CxdBHFtY%LeUccl|pv=3GahF)?_KfMsUyN$Q8E3p9{
z>?M+b6d|(wxv61P-F*SLIe2NCh#&lZKMc{k11#{}sejkQQW%nqNK{dHf5=!ORN}X4
zC@^lZ!~h>SZtsNJq~)1#T**wx@@LK3HbWc8J(zGr1j6Ti<#`2vi$R&d;uc|>&IWwJ
zy5Y#+byA(btWc(bRfcz`t^djc%=d+4D2g~3->t!AZFfzLCtF>9T9<*cbDct$KczjT
z3aaMRl2Il!o#1W|7-+uOe=D^fkr-Z2=Umb`Y44smd$Yb`qfx<4tV`;Dy>UIXn24G9
zo|im3Af2xCoIDYZC_KsXMym&cUW8M+@(;ADkx`)qKbF0L)niG6JY+lF2lgF#_lGTH
z3z2o1B?LN`N|AfB{$3{t3;#UGbdHSx_@u0jse@dD)(=g4dw;J^e~Ytt@Wcr0EDa39
z+QXz{6--ma5I@i$qHA)WH2g(NnN~-u9>O~`gFDl)gNv!xcDNYw$uJ}w
z1k*jTqvRgfrb`6Ps_=wbhXz#S+FFDN+LS4#F6f8;6{-9x;gpl6jMI!|5pk>U2NCV9W#HdL6U#t=9%3_Dc
zg$RFP5r34_w!$eBm)Rn`tgLG&M(+B<4^ayW)e)_0OX6>1lam|H#xu0
z1H@?X8-Khzh&{M8r&P=8{UeSXY~F)()p=fCl3ptNYN>Z1e~Qy_Qn{MHbqoKfKKWkl
zk$Zz>y9dvJ7zd)ltrnm2LS|uC!;uk*gvh2UmFznE)%up^*G~W!zpPl^q$efHxalP3{vjXPkG=cR0pil3
zcZ}(Iu5iEsf8R3fBc5l~Gq9!_kP>^qCZCA<&Mgds+2PwvVlOcwxa`g>31c*JF+=8~
z+Ilmtk{8c)-SjRFyp2Y7NPep;SZcEpDN
zbwH@I;rN*3W&#)`Z-<^ilJ~TY(zG^6Ms#&NPORo8f79+5Qbr8AbY#!Tdp?>%t*MOg
zp&OzJons`F&q^cB$G%zD69NrJN8vVtDGnx1tm|1jz4V+`%FYROl#B*ozFYicIz#NH
zsa2&3+3lQJsH)P|)u8fOA@=s_1FaD7>_hV9vqi*FM@19yUU(|b6-RZ7L{$Eza7)uK
z4^r`Ue>T-E$IETrUqZrHeBkOjDt^E69^~}dBCUq^Zx?gA>u<=-nW)gv!X=O3FSh|&
zS|Y|xF4zu|UO3`Hq{SC6cEA*PE11{8mDmT;tfhLjBJfx{wQLjxglVi+8^v3|#kj*e
zm7R(rCI@8)A2DA5;70|I7$zYXQ%K33ZW37se|8-(kEhTPKUPaSXP!8pWU@vx^`ZD*
zD}n>N=nP2%$)uZYhcI;-y6dsfRK6aahP|e&^*X}_W`p@?gHFRF{R2j#JGnU2n-;AC
zkw_90E-P+5^N0vT_W=AKU}am10QpHN4%Xl9dhT6~?)YMOjrT?_&6JaF*MCkFkcp8a
zf0?YByRZl+2;U7^*pa)7JqFPayZqIla;MYV^H7L8I^GuC+?W&RQAJOVUuIl_KTC@p
zvqulu>;$-AYo_-I1dVX7PQKXTe42aW9lCCmI99T!yz4rWf0AqtWo$Z7>&O%k8`JTp))8_K#>Kzocq(J2~S^MXsPn$txlL7t|m0kf1mt9f<3AdOl59obZ9alF*GnVFqdIY
z1r!A_H8M6Zm!YKrCx5lNbyQp1x;~7%Ln%&hcXxO9;u;)+1b0etE$$R|EA9@(okF2_
z@#1bjy7xWz>~rq-?>91%mFJcDyz`a0vc?D%nW_epm<7lTC=GH1GqEwV@&go=)IkoW
zj%=(0VGb<}65*3w%3(yp719Fr!1q1m3JYZ{pihnuy&D90K&dSP%L^LKzifP|Zi%O4JFW`AJs<7eve-xV`;baVCoM{fRe+vXrgR~uKb>)$H^0hTuQz(3Ku{;8Rb<6kl*
zF=aVvDGf~~#kad%FP8t)*LIE|4@aN>E4Q?9w12StlX(j_Cl)P78)rA5oaFylzlo6k
zEwciG0jvO^GXUsmZq4$C?yqk7BWC*}eoKR&j}yoVU}_^pNjuPNNhX+3mbFrTcf`X3({Y!%Q;$t0DOOo-y->!>VH~*_OC&udz(@VkfXg9
zz<&a0iNvA|0>3o`?f*Y>_Fr7mZua)drVc>b|4!+D+nGAp*n9n_<$syz0RIF_s|<2+
zFtz_LosFxsjVI7T)dp;C{dZ>nmjBHq=4fRP1Te92GP81Uv;VEp`ZHGcZ~gmr2HE^M
z!v5&F|4aAQn&x(nKv!1)C+lA-;9ENWD}V5}8vF?xz@j4|uP&#~@PB&eFEc4ebC89N
zqZNRiiyL6-;$rHB#QN3{>|9&`AGWvNwg7tm)iD4TW=9bC%?04(2KED3f?SaPRF#Jp
zz@q#|^cUje0kD|<8@=hw{*CwmEav}<*jQNsEEfNOYycMEKOj4R#quAJ1HfYS4}ZuB
zV6pxO)_)i3e-7@yum;!#WCzr-v3R>_{li4b6zpQ-sn7a0
zH*9b6x7WYE8U6sVfUL(QQ+IC4WLRDY3C!ZGUF1;7q0O
zf#qG~bO-Xe`MM>XvzcuPZ6pA#u|o;Ij_&&y_Q_YwixhH$^0w74upUM8Y|S_>_E?_Z
zF$g*dzfX5Uf?w}(;-XFa#FpvjO!7l<$K>Ihdb_cAAft;!o}y@XGFkT_=y8ziS=W=G
zar;YS+FmZ?9!wkc6|@J_=6|Ujv2JV6;BZ8=t}KJ{_CT9>{2l<@8gp9)R?bZZto9>8
za=*#;k7>3L%3Tpm37mXqn~>&3I-{044P3j_nERq)bi4ut);h-ua@3DQ`Fk?K_gn>!
z=s#e`@SdZl;s|OgTzWV}54G1_e&}8hV(_mH`h+oi_jsi!o%Bq%!GAIHAr3DchQJRl
z1Jn*v+^f-pA{>Q8KTXMq8TAHfRK(l84mg}V5v?*$6ij%>%TrFt5W!|DD`?QI
zy;I!^BC!3K7R-AlMP-1oD1jtv`b=5n2_TcK=5
zN-}!FV3*9pMN!b35OOX}w(W6VHI*d^BMaU~)Gn5z9$b|>*?)BwME^x@ZY}_-FAxG?
zCB7pho_dxlqa`Vv+?wLYB5d$->rc4%%JCe~(t-no_3NXHn5(=ZsDI9TX)FKPm*)!i
zxsF=BuRYf>#e|T+Gjl`C_|sS(&PwVFn-_Xyk#;>(^z(=I!HB=QwqgK?W*ySx$49e`RxRkji8Mj*m`6H$eT61BsKBra%
zy(G(cp|)W+bhUo!e;!{em{zRGuXL`Bn)^hG^2?EIx&Q)4#4qByB;Mg>Y5e=VrNVWm
z-9u#H(tkw2CU%Eu41>mIXq8_7yAHHo1BH(wvHP-Bbov47gQM^`JMt!@s+ztkG(byr
z961LOiS|!#Q!3x?6HrcXjVT>J+lWU8|(&}yr-K~=1
zc{mPni3Ns{`J_NylITjzQl%FWL~{Z36*vhpU4QKZ`jYL%lvY3c7y|!X#rnzc;P9Mi
z+Q*=YQpTwB4=E*eAEhTq{R2-}=Vx)aKrB)}s#~?;F!npbk}NHBClQ2C>?cO~Q@
zZKX)}RGZqV`eNKg;b>CLrE;O~`h94U!+);X7qe=ESCq9vcmA3S8#3yKhjqhmWUTs9
zbuF(y=@F;QA+J7-G_NvDNy11|HHHe*<}xwM9kYp46B{GlWO`_LkSdsDe$+Ed^khg|
zcQC~QK#SKSlccj|bPkZ!aW7kw^u%}LC@K&i(t-|9m_OR~!gqx8bEvwRy!0P#c7ISM
zl=`9URG3U71x3Mca1B}O;Ww$oV6SSCfs9*`b5$-?D
zQ`g2zjE$a#_+Stsa#{TvKooa&{M*VFycV>-A2QQ}@s9ts?e(YMB+=saMvhpN^#Dz6
zm3Q^vc8Z!b=Bw}raTbr4b>&B3r+?)&$p?JKD7%6Wnv;fw{uqdjkOCb6lttPL^qfwS$S$5mfZM!L!>>^vf=?&$3cpd
zxkK_ZjcyS%6HV;??xo*Hq!j%#Is>5*32E&HdS13~?rZzw%yYcWfa;dx_J7YOeQE2=
z!%mL59wT`Q>RE#iOf9#|q6^nud_bQJ>S*3l_{tZGl9b
zgF9CW=NR??PuF^fXue$(Qx2bfb&CT1sFNC+w&bw1f6S-SRz9?hF@
znWfGK7gmg|ZLg&r)+=#L(AF1oG6&2G6TaYteW#AV%xuwinP(DoZh!nGHNyP)Oy|h>
zQ|51qm9h^w%tvapUXIdZNLP(KX}-)0>RF+U9A(lmo2`M4MB=UHXDT$s%IbyB4D3q%^5{ak-(!l6qr&w@8*xusBw!Vs
z<@xU@rOiy%qjqv=
z=?KsrjfzYfdPugi8%emdMZ&jr6(!ENY>gw3ywn1*s9&B2C}Nz|{Jxk=>y{`2>^42w
zQ6U@FaSR;>-;G3%@|656ir+m#pX21bjOZ<)crxXkx*CCPCtJBcI>)+1p>9mwi*Vnk
zOT#12x$5~)pMM9yJAp1$YO}2kS2V;=CSEmekMhOaDrwYn4t9cHP&N??C&POH^+H#~
z^Pj|jxs~H-jZg_999o)AIyfwjIO@+Skeav?tepMsglB?P&z@{Ayaw9^9@qI*l~`P=
znep=wd90S+4elb4-X<9vkk?aOi;sgs6{pE8#fny;Zg
zT-Qo^yi1l_W1G-3aH}wRR7xO;K}xCBJ;XnS4z-nbC33xW0J6RX5M$yv|oX_bcl)
zR(}9x&zce)E&AX!F-L*G9B3)dU@{wN9X9z}$3YD9$Me*|SE)D}8UzLxZr!f5-xiz#
z?X^v2S4wzxYyE)td>GJcQziLUu@-#x$9t_d2wPzZEBGmHaQVb90Z>=yV?Xtlw
z9I)t!@Y_v<*|X|TQSTZZynh>yJC{)uynk+rS}|Hkxj8CJj7AoJ<2;d_EyF2N9YBdt
zQ|&Yu_^C@X#j}vq${-~Ki=x(X#zDJP1EvJv{Ny#sXss*9{vcwdpC?uNJ!3G+1Av4$
zz-3!qy>~D7>006Tk}4m$4R#1`JB!}=wpDOxnh#;O`ElvUXnzVT7p{Sl
z3O6t~9SE7MLat6RtcIi#U0&D!nX6(5HE*}8uVlr}Jwf4Npu|C@eBkj;j^^H-cJZB-
zqhWA+4lEz)?_rxEKC9bA-I@A?$B*dJb}5HB+^-Leyl%r0Q^l%>?#?#Mrrr)ivz8L+
z;C#;034KSjWUVlZ7epkfmVX)FW`&i)?wgfH-nYjDmrLI|(+^hDhn&(^5AT8RaR}WW
z)NZmhWg-u@b4)|626GD2n-K^QB%4+-vTVT+Ld5p1=Jkd7C&l|X;SAiMcTCHs14`I!RSy`HQ9L8GXT(q!x
zPh$C2No-Ul4xXWfzkhabz&F(+u0*{@{%VH)vvmbXLWGK+Xn|Mn51x-GFTSHSj%P6m
zEGp8u#8rrOv0bC7hsDR0o8Oip_1ab8&J#_I<)-af&=UzLa(_8&^cPA>A<&?9#lKP0
z=RVEB;&!vl6m!eI0QGPm8D3an3sgCPB6Gxv8Hxhb48*}{Xnz4#dOm(iCt9Sx(a><{
z0!P$rkg99sNy;oT>%%Ic^XR`o+k2HTJwrP|)NH`&dC9IVJ}V&E^?>UtLCu=iRC#BtRn!dyq5Tn|Jb0x!k0y4F~ny73*M
zggTzz@hO2jE=|Yf(|gfezAs$Bl{2>ARNoV=9Co1$Z-45VZzc3qqtTlLK#tm8vl0NQ
zj?O36uv}#4LXz2-Ug#`4WJoS*pe1RiEzLumy^2VEl1a_apc{kCuf3$PIDc@{b)+6rW1@W4@&FWix_ya}OJY2Np9z%v+OfE;HKrS2JMpsm#v}X7~XP$1$y13dB
z1L+VpSH}kb%&+v&psb(Lt3{>yuw)*mWK1Kk0Z>6u?8k@0E9;fpWNh9`c0-nKF7XQ2
zJE}&m0$gf-_>gpXgiJS$t*zf9@G*%D{eJ_qxk-3|AI?nKjfcHjSE=XFppC4_(oRXY
z5T5Zdf-Qm9B^@I)L1B;uO962@tzQ;<6D}#yf>7O#xmg~_P3xISUi(8$0Somg+;$kD
zt5|5<`*%Z|pF&HfHxkBiEX;Bc+h!<}N3Q86>POs_6z`_WxgR3Fmp7TXNX$#l(|;&6
z;5_Kj&#Q#V@%KP5V^{Z3ws#G(+2ve`ZOC=l!sM#CgM?zfbbs5X^3L|!M-2HPxl!JW(xDravSu=YuDwnob$Q
zgz+NdFCzr&HyVcZoJAxB5WMJBBYy~o@`p>F83iE#PflHn(@G7&=qo)?;vd=ce`^Z3
z0)Dc{eHVR-7>g`g=;5>iRwzP?^FD(8o3^+MEWg88cuXsIX?mP|TM}V(55rh75?!YV
zH`_6?$sk8!E5|2)YAtR{I>rrTYB`C1mx3(ibq-no#cgSyiB`JdeqXDjv41#HHH*P-
zI(J!iu12|v2Kw5!=T3c*O9id0K{TvodaL~)q%PoqRRmi+(t7br^5^i^CZ6MvYPkkj
zakv^W>U@#b#Y7X&+j|-k8{X={eN9IV)TPfM!l(f%yvO}%vvzJ|u3ch+&hADZYzZVS
zd9jz3p^6>u`{=U6Xj`mXpMUgfDZX+YF6m}^84QNbp7V3O$i2*)e1WQ`dR6NmU7#uL
zR-Jk|1A;Nbmf+zE&PPz?aQ2N2;*C`h7g>!=hk~YoBv+!7x5unK8I<~
z8eDMnqgMjAKI#UkfcPavl;GXm+`)T>AF7Dk|grWJ*a>oVZI2G{XZ6qh&SLBRnlG(8_1PEDB8|oV!DtbE)cqE<@
z3-y~_y;?;blR+OJ6@j3bIsyK->R
z$_Z)&J4_x3@~^uE5b>_@?{v%)DmqjjC-EFNENn6D(B18wac~$;eMMuwEsy;2H>D+D
zxHKKq$r;>0(TipI7Vyo;|KUTDR&$TrIKv2I4i*`4M+43X^?y`DGA*)!!})KhQYz;2
z5R@$UyDyJI{Ivy^y$EgC{U}ShiY|l2XG4cU@RO;j6-XiQ;Z*MKL9@@eF%lc#DAu5$
z&ER>KcV4kVqzb+J`06CsZMCsH35_40&UdbC%WiV3eQ=0!1#dDXk_&{6zF9P*`-HCu
zLAM`JxrmKvEq_{zc0#&^pf
zN#RY_0C_-$zxRuI14jIq4dP1ndYnTDAxZqkDN%C*QNNC)@kM{e733t0d7AQm(q)`9
zc~(+#<92z_Qrl$l`k<4@%ug=ls?BL=DaC%0ivomQ_r3_VH%(
zLvDHc>~}d@TfT7SXRB+uF|%`&`u;>$0zs!#m6aOPj9e4)Uc>Z-_tVDGE+{
ztZ{)sk$2m%kXe5%W8St)u!%OFdx=#UAwJPzEE|RbP$@2ApM}$S8Z$qs<3i#BF!_A7
zj{b3h{o7!y8xr0Dhc
zGUpmBJ7dn*;_>+z#sDS_SKGB>?leR2H|Fz3H8>GeTi1Vh0rMlGG|)K46?7W@nSmQF
zbc+&E+yVh7f6J+E0QYtN=iO&c!Dq-je{@9yV?dbG
zA&{1ZaJ3h}5~c0h+faL@K#GuvG%j5v@261=b_unTxmMF~)^vR^!o#C2?4@^J*{auZ
zojg`7R=NYv+-
z+V?^F0Q8KU!1~gtlNU!W>>XUrL`7y!qrCO=iR2okI)@Odqgrp$>JI~8zufq<^6`Ke{iMsg
zJj;I&0e9-H8D?V0@{7K)yX3`ITe@BYR_^(;5$!8^_m
zR9ro}wg(oAWQAC1nN4At4s$a~P9D~luz>0`Zx8Ap=LGYmp3w7x#Cfn)kjD33@>6sy
z%1{Ddj}xltL#|{eR|oIV@OuK3E7AzPO09o?LcjA2UuDRV${aP>Qyyyb&%F!?^qemd
zW4#E-0u~BSgXyL*R5Dw+U1X^)O1E~E7l%d*XuEZ;o)sYR{oKSmF!t`JSk_)X*229I
zIAD)|*D>3s4U1Gy!;UGd99!oV!jhjZcq%($J`+h=B_paIOUienEbLRW4Km~FD#(9R
za2;b-?zn{v8Qfg{w!Q9v)O;mRlg>OqPfxxe!56sW;Hj}Xag4fU+(Hu-vdnP*&5Hu{
zcA_*?kif6`De{xpZ&e5gB@EO|p+~d(9+QwjG0_V;WG_`&;Rk=RzGEJ&Czn$Yo0sS%
z8ZiiWk+y^(rm9@n?Dg7)x@Z)>NF{%uD7ksq6v9N!?e5Dm1PM)Kd=+D$vom{Aw-^E$
z242;BPhMYXD&Aw>K83rN^^)<{JYHi0^}vF$vgLK6m!5;tf_mzUvy|2Z*WIjx1Lh7c9L6cD0t3nE-LwkGp$~|kVlUZd%(Uw4+_v)u4dyn{Xze+n=
zJ;hpB_lP~F&7}nobR;OBjSVEIsI>d|hUs_j1jm{2G-L$Y%!)KE2n4KgbynKd+fgrn
zw1gR7kw7X&X|IYyWZ%C%T{VWhN<=lMu+!-}T(=z)BK%*$4oQsYIS
zYtXP|LW?9*3sL^KFP|-5B*$=oJV3zCb0X-C)r5)HRv_^3Vs^W3l(2t0RF&{xA3t47
zIaIJo=uj0#KAf?rM}+yl2OTp=d~{eG7v7#7k%sAB_FUwUAc;mHy`0_1Or~<_zL+>*
ze#LtS-vdL>z&P?~YU5ni&j+NG9WE}tu%
zD8|2$#Mn=O11=%oAq{_`ju+Hf#qyGw8^#tFiT_s2+a!q|goCqN-*LgWO+@!s`|7ZA
z-B3v4c#9Jzr_onqK`9RMEiGGEsSL%IpW;(W1g%#;T2H~lj7nVC}N>vb^xEH@CZjGiqAK-hWqZB4jSFQXxDYg;aVN@LMs
z9L(!fwHb>%fEH~kMUlOsNeHzNvsPIdZ!@2d!bX7H-;s8QeT?!}lh`A)ZgnH80$^6y0i
z
zU-HM9StRHP_p;_kKa7ex&XOKWez{r~+dM%0^;@0aYn0ql_DD>(1+*2y6gbx*!{Ink
z*roQ&Bwc_e2{fotFP9`VLG+M5{GeuIM!n6rFFM-;c#888T`SX;`
zgeBMKZY+PCEm4ys4yg*C72?5=ZV^icVIpFxJTof>tli2sMM!xjnU!lx*jv2ju!87W
z`Ni;g=d7R(P4;>H1t|gS3fD-WnT5yD>k!-)94$|FP1Vq){KqY7UyG39d1{T~prM$l
zV~#c7?HN7%oof4lS*)|9Ld_r4lSRYCrdHi*hMJh!-$n#T|zOoKl9pT
zCmO!*1+%aXCz{ILZ!`v=Fb@V8X%D+Tza}$lnPQVvSjYUF<)mp+m+!4lAG!7S=yHD)
z1qXjxBpKztQ*-$$BzA-R81S_RC8fo>n+9<5jqig)C`X8Kj->p>JD
zZDlsT_D*PS@YffQLV1GRvta*kYh;BRA5ZI*5v@zK@?{Lh^vobuIA#Rezo&0v3bh<;
zB6rc3UqS|t0E8-q%avl=ske{?ULVg;Ct;=eTRBYd!Q@N6&>wlTb*G5f;(o9mR|U%k^d+~@f0M0xpETUe4;}oX15^rIo!oexB3V%3wH}4
z26;@;%1)!^oFnP0S%R<#kLL%_$(^b_9ze5%acugSU#}T`%pd@rFR3s;R|ODF7V&L@Ax&!5{-c#*9$yZa^=(K>f^KF6s9xTCHuj;Y9J
zzSghUIg8@fVAe6c!bh+(Q`3L^&M}Cc>P_rfO=gC1GMyK}uQ!Rv56oexN#9`-I_{#N
zbT?^xZ?P=VA%*`17g>fRg>@G~g
z1?U5Ue83*dREyeXA2WX@sY7eY-SUNKiw1oHS@5Lh-r2XYL4|<{X`uJ^>tj94&_Xur
z$JD1gPOxm3qlQzK8e~qx%e1*UVQutHy>?Kqvn4#J9`o@?%Zk%c)AlL
zNaVKJvKR+fjHnnY;<)i;WQNsfPQ_i-8N05-foph!eG*{oV;4vxm%cT3YQud<9?wQal^F;
z#+#h;&g*ZM;X!}8&#_kg&RMIhEcfL0o8X4z{)Z!Q$tz9Giqs=XCy|*YY?sH4%aS7A
z$ba|db4HvTxno?}+(+wAJbOI!hh4_fGhX+p-sK#>%7ebesY0DG3Hz|e{;(G|_Z|5F
zk(NTVczX0L-E~1nZBB1_usJ7n`ofJ2LYA;IT`4@Uq0E1!zdMh!9Ak#Dr(geLItLV2
zH#Wdl=*xC|?bonlf+T28Yi_Svpry!-TI|hLpp##hmi_d=sFNtX2X@rn8fngZo~l%1
zfygzUOJa8Y$Ds31BS67!TTVp_Dko#7R?1#ocv5=oo{Z8#2OT>MR(`n0%&J1}od(>m
z$}RzkSdf2f@_v(P0E^f}afbFaftTl{kJ(ak&_nRQ@V@RI(AE9q7jo3_o@)>U@yBbW
z>L5kiAWWUi{NwNVm%_0I4u)9^`(-BUeu{~k*e?b*{z{Y1jkut
zO*ZuJ#vp-{F+{3m5}Uv)qapOi(aLVYas1L7*QQWd^I{!j$<8S)0b{Os!>v-ikuFR
zE43bY9rz&qDIzhB@GM}GzdmrAs}YZemytpxgtuV$pof1mjBn5~d0H4St&_O`r-7J|mil^Xb8S(ph}XDNIid
zGGKG~X-TI?k(xihFfYa*0oF+WGjqG?2RouQ7;|cO_fUHjMa%(8mnPCcdh)!|8rNSA
zu*Gos>A-ImaXwr#JL@M!=cQzshmEx|C+L1uPVv!RAmprWP?2NVR3Zv|$G-(Aa8pMJpCc(@PdTi&5gI
zw&*AkeyDu$sJ39l1#}A!+Uv+igYCP0KITJNXvt2~_`BtWt*EUyk}ujSBtvIi@QPtW`*UrH%lfCFP!pj5cRN)DQ*f7+GX6+pmZ#0_}jXkb@`2z-7Bz?6*QRWX9R`PE?#I?FY1GpGDYO
zUb5VJu3@{(fQ-WJY;f0#N68{6xXb60S1#ZI8}vNe@ari2T<33JO>NQLTwCa^iLsRS
zcF!4$SF>p`!~>i%Q%Nn#O#^>-qIsSsC4ZpH)PQ+u$bLTuQu5E)te#gv5d1Lptn9v*
zw-~q%h|WnXyD&E@VlfzF#PSFr`+ka4$VrVy>oNnT814EL(nWf0YXhB|4B?4;1I*uk
zzFg9fc8Kp?AqHd@3pf8dqJAzM*A+N4|61mSpfUmwY5?R^yS6*6_sCU$I_43$fHHJjj%O`)3E&+;-yyHaE@y*@<+;NE(XRD_yOS~8){ZbH|J&)99)$b+|KZ3**
z9mQxWGBh)YsNJzhUwDQUcaWD@V%f05qexDRQa8=G=iIT0^2w$!&o|UcLD_wn$>?<`q4f5Y_URQb{Wi*-?s$O
zyg)3J)@f`W3I-KxB^u*XjLng?gx%jgbCog)3>s|)T);j`m{1yPe88!Vc%aTtfaLoy
z$VZJF}rj?9e+C)s}^!D;t#b?h(lv|}Mq~4=1
z54V@00Y>GX`QV!87(M+qY!GUhP$o@9iv}
z^Z|b(p|a8a-+qW5?v8qwvwduH7zrybPgd(zzii|X8*SrG%kFVF7I+7TqQ%8s^QA%y
zUi6lQ70T;IQ%t=!Y_Z}a;u%4!e~)g3=X8l&6?xE6U=CTZw63l6q7G8#(wv3-1b13a
zf7~Ju%OcL#VjmneT@rP0g5S?Iwj0JJ?ka!pUz#QP-#;J9yo*oES&Db-`Otw4DV&tV
zxO%sRdTcOnW2HN>Yw!vzms!aCnYMn*Xl21#kzsNXheACO7fTP_VG=2lFLN)|qFdi$p(+G})LiP}aw#NEx7VUsivs
z=VjLKB5?DIL{J#xCsj2*!ksC?3pR9!_$N1pMO@zMN&o7vw(!XC=={*dDLyGm;}?cx
z@v3^zCR~g|Ojo~~_FF6hAL3nhc0&PEk0V37SgIEcA45w{FXh>^=JD_=`xD}cA=?39
zDa>tW^QXl2x^{VXh*lQ#mn)-56y$$(HXc$Su~0u0IUF1blxl4T>=b^?9g_-ksMVK!
zoKj7?ZuiZS>Bx>kN&fD0G%(A|wz>0m?>u$R?OSg#nzsICnwdurA%>1A_*CaV0kcP}
zky3}dbIa^j&E<8;atsx68KwEU>7s%Ix5)+>uQcVc;LN!N|15BHddBB!ZMc6N0ByI3
zs?1drJ@}#Z;s_Qd{|}$Ri``
zQb47@n2Ld~yv9kYC2u$N+^w3O9O<{OTTxq|a*jlmcAK4kfsQbP>s$(J|6V*y6DesOLnD!&RLs9Sdf=(WmZD)98jp
zvIxjCo068@oN%)>4!grwb_Z7rkhV3X5o#)Qth*_l_Vsdk#^hP6j6NdLiPNlh+^Sf3
zNv*Si1LrnyK#9cT)+KRxGp&WStXDGhwgW5DbKcBKnETTbIm`4ESYv-}_v~L$dJ>4Z
zS4P~^kIk?@U_N>6dtXmZXWhPEh@~frGV-S1*@~eerBQm^sK(Vsel>{j#_^COgm%Fj
z=rqp440vHGZ0s_4)$y|jSA&!kykj~}3~p}EuzdAC7b$1Qs`lu;n*N4CeR?OQW}md#
zV7WwQcEr%ZoP+hAZU=vQH$7E4!O~=4BKMb5F=qnuXcRr(4#hH8Soh%>TCK|w_1?Ae&|*@>z9>oRZ}X77o+2H
zQXX0n#0n2YGm!eYIKCXD6K+BrQvGU&JyY%HzOE|f!P}a4E&P9I11NxU+0TYGs6{{i
zAa*PCp-U08S55(&TW_8Z4|-qf6UY#~?1}uBWkAb{#d-QI{YB3@v9AML>&^Dn=k{~Q
zgfM!Yfn{Q+btBG4j}qU-)91Kd?cUXDC1BC-661nf(XH#UM%-*?(tPDZJo-%BMoRI2
zy`xre%^1(ZSpX6cr_RvrWo6JVFD*+$^uX-mzVP_eRPxb39l+UT>ZPYv5j7Gp#-&OUw`!5
z5fpfCmc)Mz55l*7)zYU{MJ2dH@uK*)enf<)l-I*&_q94idgeg^A}wNdpL`)o_H^dc
zklL2~1Tj*;V^I8W?vKrS;dAZv9&yXBmsaGCuA`aJBHqd{?d{U;8Zt?f6}-qK(nOl<
zHJoLGj?z(w2U(_E?ewS}StO5dcM(QAQ$vitZs~t{<&lqex&8c3gXrDWH>fh%78o!d
zyrFsWD>vtm9|=3O6I1Zz^#jwn8hJ|4llU6<+wQlHpw9vj&pp1(W++?j6{*>L9qrX)
zmTQPUlim|G9RV})i}VE7u79q3Z;YKaVZ(@3bFQsFUJ+g^c=}t<01qJ`2;MS>{e9=?
z!Crr6My>n(-F2G?dJ7F(Gl<>c;FJi>n0t3|pCUWr-TQ9xf*FnwA5m}kPwqgqn1
zigK%-z#=3jj3i^QRSlJpr8w#hbfWCq8mRSH;G0llfT#WOX>4l_j?nwtv$@z)1QG2w
z2@M7>7SZFxS*c=Yff-_((u}6>bBK+oU%Y=|^9aG+F*o+#w8PYS54O`|#NS5HFvp0X
zvHsFmF7rlk>vnU@3a;B0E%z0OCVsCdzj8>7oqdp?U>2po
zW#ykx6=ii2+J%oWW0<6%Et;shFxggd`x)|`BQXrES)c10ZmT;?!Us{8U%P6ZL>_;&
z{i$@W2msldhMQ6ZM#BV2<}c$Zpz;rMC4$0~>-M&>_wc~@THaS*mQ1fxOV%C&jqkqY
z*)TnlZLNDZqU=T3?ss+*KP6^4__!aqzt*yxwNbnVeGHQAUgMP(R*Wj2$yMen5$c0#
z_eX7y^=$i;F+YA#+E*FK;E0iiWiNk-DD!mI*@iyk#rP?Qq3tmQnpl}OF4Mc`_S7bd
z?CAZ)D2gA~MJ$!DW6h_^Ylg2y=r&V3<{RVC6@3U{_wtHz)RYFN!rge2gwn}siAc3z;ALNX1(HE~iooIlpKlvO$1;E`ym
zHt|+UG>_OCvhjz}n_nSwFdLA*-eS_am|__zKW#vt!ciL>O(5jAy*$brApD9k+Ti$h
zk_!^+FS(S)Gq&;r%hj3vFQ0!JcFsJTO%PkqhsKx|WJZWwG3KLimtrpH`vbrAycZv_(pTbJju;DGhEk38q^1y0SV#LNNGj)c#HjeW+XFb;
z=EbQ?sU{_ADrB6$6MWd6D7Wy_f%|mt#HJCqWG57@S!j&4%qU)OO)c5&xz@K~P76u#
zvp~kV6-cw3-k+ZUMFM}zOHw=5@uQ6Si10G$IDJV3hC8}Zp32GTI%Q+vJWo(~rZ5t`
z6(xGr+IY3s@RpVgaOUG78$EP8>M3QWz9gnkANLKZFgAoK;sbbw78Kmm5#WW_UVSln{OL(MEl=BihvUO;>@t;A5QjKo^
z52v20yOSaE69O_ZmthV96cIQwH3~0GWo~D5Xfhx&GB!1rVNL}V12ix)laX^Mf3;S5
zJe2t#UxZRA9agR$a>mRU_Yrc8`^t;WuK_#YjaZT0-ThwRCKU
zt0>jVR!T*bBeET`q8w>|&#-MLmNjz!-2%ua(
zC{FsP;OmnT(FUMHG!$!lY;=@ny?et91hAWfu}nUT{%Qy#b38Ut#@N`>5ttzHrtX_q9()!nfXW7q{)O3Z)v0WVCHh|m-zbKHf6~m20uU~n%KAnI@jM_Q
zNDqQgTFe~HbGSE3#pc2VX0kv4N3_J7Sy-FTNkXI($HH10TThU5!Aa%T-^j48rNzM@
zj|W(q&5A&bn{U#`3L;G&AZ!i^_9AUI`L}+W)h59-gbu+>z}(6jpmMoXk&+qKB<5CD
zKoSw_emW?ef9)cGfQJzjGXXe!R17c>u9CFUL~~1k5REy*&^f@;4g)a4v0nR(nVZ=F
z1Z=TLXW@&YjRinpL)c_~(IZ-65TAuY9PBb*P$EEpLFVkz=dq}~m@htTF&`X|3n6sr
z1_HU*z)EFb6A~od8M6am_JyIpmg(%7jB=4UFchL=e?KMvP{CN^Lc(w}tb2(Vj=esA
z+wxTa{V#*}4{KLfM3{s#w*zo?))=EiqBUS+V)YX9pH_7W^VvVIBkyjlhMN1aK3&lHci#L^8
zw&h>me_0y{X?f{pSKBa|Y5v=vDx-&VBPuR;70C-uJRn|J!DTHIK3=L`r}6l1oecWt
zgPLG7c6qF+7}#r
zO}+Ad3gwBt|p@LeB+hH
z2Oi}tx_)F!PD8oEu3Oxa=!#k$?d0EvlAGDvDz2|+4OKB}_ASwHWNY{3g)IoW%6#}f
zbD5Uj@2*Yy-P`+beIMHye$LopO9p*7fAeIwAq&4mecZTDoE<%k)FYPtZ+Q!QZtY;N
z!O8xaQJ-|wPR=4mhT*$f!Rt_0#qkP!VSa4)B1?RazF-qtTm)sRu5!|`_seSOVm=1h!|khZRN#Y63rr87GT9p_zqS<1Qb=`T$cc!eFS?Ai?VfaDdv?8JV#f8{Fi
ziB(GNj_dLhw6e*R)+0TaL_P(Iwa8QnI0+kujXBg
zz3#YSL?U!pJ0O4YWVTgbWNxU#RI`sP1?RARb8XCUpwbHKZm-{ImRZQgAO;fmC-?r2
z6VpuhAC^~{AJOpUTo)aPe|TGtd)i)aF+60rb$I&^*~Y!v0o8rI4CsueqxByj%gMnO
zRW-Q;{^@tb1rwEtPrWI3t9w&``DY9E&BWfvO+OgAnOAs1VDjkNZ>2XU3{CtjXAI=i
z%lo&o2Of`BU&yE?jzs62BY(#U7sg$=3))#Uhdxs6dP=Q4&8{hoe>7-X@H+P7^STK}
z%Gh^)hEIS`li>@WGtXZbvp%<_OwQh!A6c_wrEA^DG3bcN)UTk{ib57fDn8
zms)?leJ{l~$GtJ_If0iDpa>7A8b#KuYu)+ey1o|9*2{|dQ_3CP!Bz8atqp9AnYeD7
zf8$*DnJ9~qb%J#D)Wxkke`G9fvIQ@^Xbb~ddQLmm1#V7yf70o8Hr>__*RaV%St(R2
z^+55iaYY}A3_iAf-HuYLR5&$Ma8I&9ePF-yeTkv`1xbe@DKT>X)(=ZAcDB86Ub#nj
zswqsK+2)x5nU3C&hBSFWFLZ#ty`Nnb<+#%k9a%y^qu;m9tycwo}=3`N58Y>_57A
zt4e7vi_o>Pjj8V#@%Ck7R^p=doz^M)cHf|CL@Cw|e|y)ZT&9=P;Av|%u
zj#p5-1ddv`JiYb((wlFfoSFq=t?jo~S4`D~IhDNLTJZd6NmHZk(HE(1>$=VRvqGzu
z9bMFaKj%JQ^Zqj}+h?$oj_2#WKZYK5(b|~Wf1cy?&T&t5Fl}DM0XKQ0J$3=fKb3(i
zi;ORrkxL9Xp3dCUWk>g?OlV&+y_{k|ReY61oX{MPiGU{CnCA`03k@$B_cn#v4*Eaw
z{dNAxfy~2=rNW#gKcz}a21Q9T^}osreo>BAbZW6#g=@BtYlu)CO3p2vd9mC55o07l
ze=e$Xqe6Ja*{S2Mxizw{^i_jf?%E7#)&60dsOglF1m;$;v&?IsR&HkcB-uxPEZdgp
zZ|bE>8?s#X-t>u_rm>JNTKci6ljuG8Nq522UTtwqzNkl)Y(p}|^H*{>$Esd0jpWon
z>bw{ywDQXu{WxAdvu>AiccbpO%;3$if0tI-T{0byd&*AqZ}>E(cJdulgv>KN(NQpS
zd&g06kPu&~<3B*Wp0dw=S;gyX4i0_QBl8L!fI*jQ(+@6wy0@HpGJ9f8YH^pgOk48w
zkyoABc}{I-6R)2f%6S?%S|?DhNgR~y4qvhYZmgeI^P%YDR`vWE`{k8g!*@KRVus}Q
z>M6Li8#MV|&S+Ms6=mIc^jyFAiLpneaQLOwYHD0y!sJ+@!Jct;(y1cnH@Pg@J6oNB
z5p}{vVV#gZi}y3$tLrj1gp
z-z<_axl^}#^z!;J({4g<2j)24j45Lv93Y!oiE|@SughWN-(j1m#vNV%4+L;2p
zAZ`p6Cyqe?gz|tFg+T#6SR7$QfH8)G2sj$zM*@E=60l$nU^CqUn7|{zm+i+ip%96g
z5}^d7J9hDjv7e|Sbh;a%gilPYlGJdRR
zL=1o8Fi{L3z9N_p%Nip=c|<$$Xm}=p^mKqX!i7;WEDdtUW
zSDBC0oiSo(A^gN7DDLwUgHVZh(?5Fi>9&8l7%E1@xOgTb3?dOeEM+g2o*9D5Yz!wK
zPj@#ai|kEk4<$1&qB|%BkH=-+(tJ*?-gbb#${Nrtt$|dhZYa+M69@>_Vv@9dt_aa5
z91|r_|3lz0C>Dn%{udgFpu9-wDexphDjP*&C9tRKpD~1q^tUYv#(_D2V*wn`ji!G}
zOUslaHPfVKq9sX*LQDuEA-)(+LLy<}LrN4w8)1NpBydvVKY>qIB$^H2AzYjgJux^W
zS$0n}5(Bm~W`fB_=ubHqTTqC3G9iYRhoSrgz=I=6R38i{#9{pZnYqtuxl8zbA4mWj
z|9fLU_l5)rKjB~eKl2NMrM!)OFp+-%;(vxAVs|7S=J_EwH+rVYGj>lLB4)w~jpD8G{-B2#ZL(nK-VPy>cVyZo+Wm1Y9eM!6@i3Qk_a}^NzjZ_Y*=<$HZe^Xv
z<`~#gUQ}y1a=fzT?-q8R*IMd&=%F6GraYx#%K^i3%=;_5(Byq7Xa&b$t2+OW!E?Da
zx8Lym*FJ)h8Od6U+?+ZN22$9m2N-=@j$P0*`Niwt;q{gDt^8oZj8g@wr@=f16PJdw`BKc{
zq^xZkOwF`}fd@j}%}c*p50MXUomlip^S6ZkDplDYM{k<0)Usm}l?>$UOPLi@Tg!OR^8RYCmBtlQ{=0wD=Y8w_y~BlT?ce4+
z>HVSo-JPoFvFoSHPj4x&VA!O$J}ncXG}V&o@XKfA)2?t%iTp+uRS(L0i(D;+%NpL-
zZ_n*eLVjwW`z*lW?kV*zveI5?A9K`g>N}J^Z#YMNRn~sb?sX}3_JekI?6y!uLg!4@
z&0imrt35oGFgSluz9wRKghEF~@5Q0~4of7h0P@w^CyvePr+Z
zb^H)|^0Q7El9XK_r{}p0z>C)-7~6^{0StHKH1RvfI_3xUohX=
z)u>zTkbPxjS(utvp+Z-cC|G{;a(NYu$8{WVdS!P^ai8Y=T2+0c$-)zsdV1-GwGQ(P
zG*r#if61v-&=30kjpI?y&8zwHhAW1)%&vb5Ew$01JPy2pkl)tpoS3SpZ*c
z<4=;EzpLl?k*(x(zDY&1W$@3g~0c_UqTEIc&Y5Og2RwdaE55^sY#|x96czPI%rZ{Jx&u
zf2P+!g^ksFaM&wqW?foWnwR1=Y*;K(UXE5|W+vGM-!*@EATBQCi4D_os^j)YMyAen9;t6DKQ@*&
zq_cOq+TsDpug;>DB$w}S^Gy9L&{W%NnZ5^_71K0JNpZ78#iVO0?P<-~(#(JH>p8#h
zPo_ey?_A$$*Njk$;&9S!80wv8m5Z*
zhetzt*Uar49Iu_@iU`!je9NNX53O>Arz$3YZNd)+!+9U%LUpeVnVLtuUdk~$k{Ebs
zPvEx&%^PBsQs;hkvne^e&7gmsG;;8$#`U?}o>)%7&VvPuvW#w>+JE-6kVjb)6z087
zX*lr5O7-XTDM;_#LHNGNxxHUKrfor2%Yrb+
zf!LJcC8H_pSO*^JJ%SYmQD!}V=Y
z{vt(kt`fCOwP}JCMzN;3~_4umC
zBh~{C1!wp_yn1{{|6PMY6U*c6Q?f?JY)8ja1>-yG930P&)eFI8`}RP6{eX*3$FoN3
zxC@$B=}A`JdkU#P6pYe2)Sg=YT7|Ny4bLxc?#w-V=Y8Gw_2V{149(rUI}HxA{p;F`
zb^Zt7d_)ThWp0xs@(`DB3IPq5T~Y!FxA2VwX9)r_GM8bc0TdH4I5jg0FHB`_XLM*X
zATls8Fff;4P6ZSNGBGhSIhUcO0VsdFWmH^Sx-|@e;On=HyB0n_}p{Nwb-ynk$#a=q!BQ(F#w9%SUb`%(9&}Pq@<;7
ztZf+RX%v7aPL}%i07hDRdS)0>QXzYwzN49qwXnV;kQ2b>XbO-ubo?m&$e@3x=YSyv
zhyksE_8&nbfPou88tAC6N;_GTug
zj(>VE(a`+q^ru>o79go_Xkp{xU||N(w>APu(n`|;WNchMoMr$D8*6|8&{W^j7+_-z
zPy(s}lodr36aiuivdVIbUul0o8Y?>4+S=Iv4;CRsC1o*cfUtm!k_Z5(LJbg8R#f`)
zT?uIYf!~B0AfxmV|1;+!@lUz5h?0Pkx||3D-Jfd!FaVr^_6}x$#{L5v*#|Pff1rJo
z8r$1g{WSnUVe05;%SlJ);^IPU;^g2+Yh!OhYis!zJ|$B#2Y`!>y#;^ZjM^v$dtf!6xgh98NJ`i@Qx0KLC_A1|O0`M&@H0YXmp_J3wb|2t&=
zzhnLlUC`#^%Csy!y!3xv{<&iM)=m!Yf8*xAFWb<@+QH1h(cxbmfdFGOOW+^$4u96n
z%=#~%w1A9+sEDExjns$mtZAfeKG?CQb#!(7EBjAB0b!{RDYCHx7#SG>^dBM>u{IL2
zv9kI=?Ev$KK4G&Do*ZrL-RS-wU0Yb&xLAAqzkXvgYa`=7%o~3>+0rRno7p)5C4~Pc
z`@;qEAD;=(5kLIU?hhKgJZx=j0mk~44nQw6W8lXdjE951
zGZ5ft?*#Pn_
zjclwf-2g^FV;Fxr85_qBK~Vhvv(Nr9OVr8IQbykjNbyff|Cy(6WoGI2Uzz{NPzC4YfkC>T*sF^F!NY2dB(DW}B|K*l&)c=safVGJw@MBH>aw-4WO_m?B{WyBe
z{+wI@8U|Ljf5bi%X=q^$bZ`K$a{Ltnez5QlydTv6K?{GNlTrF6E-OU!|48PqG!bh<
z8zVDo696L%D?s1gUf&Id{zC_hEGz&Ih7U;_0bTzR3V@E*+Q#vt1Yqmr=mjvgv4{Dy
zm>lc?I)gv1zbuvy%h1O11M|N<4D<{DI^f?-CIFqu-%Mrzo$239766^u-^`D8mj5vs
z=mB(p>&$=f(b?v2CL@5(_J2)g)(;D4Z)Rikw~UWr_W#LX1JF5G>O1^xJi|wkmLje1Xm94KN&g{uh7bS8
z*S~+#{;vU~|Jkd5s}>ZrarK~KW?=x(Fns`JV`6{!LoTb=f0Y{k>+Jh$uYE}3-}aw_
z2LJ@R0u5o7=4}kQ0?d;df=awa@+M0^lXB3Wlw%S+iF0auC-_w8Ok<06O%jAW_MD*L5LFZz_W6Ma3D
z%*uaNFJhifYl&?k04R?f332@Dc#LEme@DJbB+@EvSW~{|W@(W;sVB96Sg8v}H{?-Ds2E>Qwk%(-fUozU
zNPwUr^j!z!u)&`X>Qu}8ixqWAmyh|81(Vh!ckl=cvNlDPLPvt`cjnZgZuzW^j39H*
z&N5bJq4#`$oUAGVEWCb4wrK{1c8TKr&67EUPsCi|#8Vg7_AD9-XbKt8a<`|idN;l0lor$xsiQIRSedsxiTIw+Y
zJJnjO>-0yf(hNJUV*?fBpWu6nY9Xvc
zH5$or@Q|`?XG9*tlp64$mv7>2Kdm;8^>NgRUumlwUO;)pIrjjKvE&x45YzIk3X^Xe
z;0W98Jo%J_G-cP^Cvv!Sma~zvUZE^1!Mr1Gf@~nr2H(!U5df};5N%)db;{sHEhZ%_rNe}19q}dPsl~Op
z8>R$R>WViumkyE^v1`20an6{HSvux-Xm-nno%_r*MB{fsNb56S3f)$2>1nG@qDq_3
zDgu3TpOs?t6)8%qiIE*<7AbjuAXtXNRO?@-yrM8@>xu++%ooz=Q~goWVtRSj;=i~~
zE0~&b?$a4tnc~P6*NVCRz|^IOI=fXk6jitDuZeA%Id%GK`bbW>WIs9;Qw7`g19vMs;@a4SyROy|uhW<(4(r_$pz)
z{M&8$>}&QNnIC0w>FT6^sc>dlWx5PSUr$Rx)y)GKr5rx9o_aDt2UXjY0!H^U9Vc@I
z$dO;L899UcEDA43VQcS*!iq@lL93rEEf`W?W{{*~uzNkgTOkPi=>Rpx_0l1F#kD=t
z$2wNdCJx-=XyLW;m8}ZH*-BP3xF`Gmbn29WNNqq483qgx<~vG%x1|)sT^kzC>6=9}
zBEp0M$?cs(bQ@K~PkUEXJ@B=MFf+o~K}-FJefwAWs7+wuj#7Wikz#qbVu~wrIJAy=
z*oeUz^JexQS%NNU8;^kRpfuR}wq=$Bc_IavFnVB?~O(K
zbr6b2TRjiyUVwOiiPM-$J50>$Nh6VL-~JApr!}FS*SxMdxkUWL{tNc{c|C3C0c^CS
zF=moVzi||*<5T7l6lJ|M|x}xlRbX85!M0ATeCHL
zzeI55xBXQc{Di;gI{8!bG#;V_f%YX|!aX@UijU%9jpS#4?JU!9LRokshJ`A=W{s4R
zF9h$*>?Fw;2;WrdBe^?Y$tBn*{BsS^;ABkDZWk;XoS>QCXOQ7KqFux&cF3bnlq;(P
z$G{Q3Riwu0t!am9Y?GURTuO`^civTOK{KRYe0?LSoj?H!15*4}hHjgkl;n(7j+Wfm
z=XwYJJ^)02#Q4{?-*k?{^z%~{^~%RYZH7`6dac$RT_u;r%Nqv=Hh6RD>^nmctD=GX
zYF{FrKx1F5f3?I2K1nVqBJ7tGZQ`dbRr1y63??TV6M^*0aBQvn&;Ni3MDdE9um_>E
zC^uOadFrfT$~4T~fKn=3UIb07#_h^0;jlh{A&q{2;((7Ec@kFm%++BCc{FVgyJR9m
zc?!L2*+0{fkcx9g;ZqH4eun5rrO=0e0yVAAS^XEzeE
zZ0d3R-K)8aS9Js(a6}X;D)Ka}?TO%KPjfbfi4Su+};(aV)^FD5msMo}mT^nkHEj0eq{uX3$DI%{d6upB4U(C5mT%xw>`aNiKq
zW)Oz0aqy}@`gK2m!A#QPNfV|8QG&;A4QU+3T1Lq=CyjRQc}lFh+)o|>o?OzizMLHv
z7XVQkK4(JO9V`qAsd&AJD8Z^N2jQ1>YB-XAU{{O%Gw?iDtQIvl_+bX8w;VGhe?N*<
zSG5r`bRkX-tktmBAn{F+nw{=+0=!Sp3p(u;=v;m^b`5cMxNnG1HZ-|JYF2uO#l(-Q
zC`u7iWml~kPEt@$H2iPJ6bHxA@ufYf(G4M>zp2c
z)`oyjNukB;+UxOfarz?J5pV|vAL8C$C}T8y4Zf#xr`{hQfd$2JeV^@f6;VxrUtHE(!dV3*
zhje~TkcGe5lJ|gVsDs=qjR_eb$x9DnTuTElYa#Rr6?D?yiHUg1-kPXWJi*|9JBX@0
z5z+aC2z|$(I36oH(NcSVb`eCr^u9x<)DemU3pwq<=>r$rdL!J|7f_yX2Z){3{N>
zkpKt$TFaoamUec$Q<85?^3B-zn%`am^+UqTvk&k^uU~40Odj`^Y_G|_=hR;DRgoL7
zhdw#WGDvn=)HuT7rZOu^qRUW!fy35~Bt;ih%CjbBp{3UZZMl$d^`sIl+q(r$Ok%
zkMpuM)XSL}2?sX@6_$$Agkr~ry$9pPwY)lJxh@WRXrKAVY!fqO#H(z7#i9*}#R20h
zGO^Jvz60a@Naa-X+k`YUU5gUwM$uThjW`v4$OB-n!NHCloF!7MYRzrbg7R5O`wJ0V
z^1QSk0b#|^B-*<#KSFBPiFN|C41F@DBD<@n(z+zTkkO$TDrY3zTZ)9hW*#`WP!_Wy
zfNBObCO&fA+Co^51|SrFPWx?3>JU8yOy#tMnZ?lZJ8>|Dfx(%`xr?sPqyqK&m0eXo
z+ps}B?&U@yyjJY+F^z=<`4|-sAjF;}`wrV>c?%`X#ZA!*(cydy>!&9>d7p_lNOEp(
zs{A$SwO@TwDh(3GjTByz#WuI*Fsgk(9ij<(Utj~Gz_RRFdhdvG|ra&6Rm
zt#K)r_O|52V^#YltcrNx`RTZ;2G$%bs(XVINgN=ExG#7)+9b@p=$n2WG&AGVl-l3F
zZmVp#^UFUbYe>1Y!5o*zv~pbx{m2KF1cIz|9IIS6_Z3j-#8#7{B4N2lXQ79fu4m
zgPm)ogB-%*`il(irweYl1y@{Q{ZHH_jTApKzcHWUZeavMcuxmGBxXE+vn7vrCp8fM2(~#EvNYGJ
zNw&WTeGoMf2v-08r6;2yci7WsR{-|iJKG^AG<-DOpR6UV#1eMB@Hn=i>3n#maFH!6
zI}6dJSzMc7+ETnt2HFM>tfx}vVXE65T?{CIO*+Y=mP<26d9DPpA*zol7zd_J41OXT
z<8NnK6l39kBW_VY5EBKtU{3L{&o~XWD=vJJrca`F-Zvo-Q)bqSRS=L~7ZS!o4^m7q
z<>KaAK?D~h(MQAKl2LkH;oQ>aGdmYshWalS{T{;pj2`50kT!k^soAW@f(RZ@D9s@SK0s9;R};SK}=+qL9OAEtTkimpmf
z(4!Cv{VKuv__MZ}Ko5u;IhX;t2NT@-*0q#Zz-lNXA(e}~xjd!uJ
zxl>Jl+9%QMDOf?t&@B?6vmbk*`2ykd?T|mkSNC{!6E_m3o(=3Xk17?@ytEPpyz#mG
zX?`4kpK)Tbk+yT3V{=~Idz+Yk#6C(OCzh@djvW=wX4?E2zcud|6%_tV|6*?1JT==G
zzU9=$CC7;KI|zberCHvZ;1->l{AAD&I+87hqAU3j>#bwh_E_aX79I;NS9H8|9<
zEswwZIxQsWn!Z)kvfsKz(ZiE7`-sMNU9zEnnwIx0%H
zTjiLU%+;-dA23dSjKK_FSJU*Bry$S^n>v=^^|8oaQg>CVWxnxPkX&lTk@9daFR!ML~72mou$y9T7Re&rito*7pqpy9`Yf%1USp@bk{
z_tn$xB>x?olSiia1Rq9nH5be|B>V7x+XOao)#&Rznf!|IFT=iFxQw&;;NLdH*^_4?
zZ~DI`5PnLv#=3u+*Nza*NW~jhm;z0R>QFv`mgBmtT8twvaBPdTG4StjwI9ruzSnO=
zc~xwg1ejNOW{@sritWg>quCrcUz}+bp=>Zc=XoKdBygI+eNHoFDaOBNuAxt&DrV
znYa*-Bnsh_dB2w6VCmGHC#HgRkR%7Z!Q8E&(PhL-_}8Rr34Uhu+k8dzgJp0}F?DVE
z>=y9rLZxITob$W}iXNP?6K#Kgltds@ovvAU7EC1ih7S@u`|$A)R$O*)aGve%t`pT7
zfe^7euYUR}FxfgNOtFryk08Jg@yLW~MYAM#1iH8yTG~mx4Mz5OOlA@b`u9X#_(m2A
zFbWr9wLp9pr!Pb<7GIjvw4*n@D_1Odc_qfm1Kfgp`Q)(S2a=X;(C$HhcTLL9&a$F1
z(O6n$+qR_BAm=4X+>md;!51Rp@s!MjFAvA9+w#)~o=3tKwFzZQN3OnibX-
z6~S==m@R}lMFXM>gdZ#B*CRnJ+JK%Rr@vDe-1*;g2NOpcKI+l|UqD;CNPu(&=7Nd{U%Ac&kmg|2+4e#!a*#F^h(40gedjn$6MH!#`4+yZ%8QgEyQi`d2X)%l5%$1XDh1KlZeECF$sYz)?0L^S07Ya&#CG_lI0lJt?R>vIOSy
zymO<8v)YM1)nI*pMkmJ*hl37iK`j>IT$!guw;%bGYE^?c?x7y1!Kd4}!t!$=UK)@o
zwJ`6^q%}1=LWa!)aj*EowUm|TGT$=Wo|8auhA^Q6yX(|#V0+A65tQm>%|M6A*`>ni
zf%APFPM+UrIo6MKc@1@GGea;R``P^T)j2wop6ck)NT^_c+&2)F+Rd{2P7!K46LjDQ
zXCgdQg4`lI1xICsYpNl~{ft%BrlFsB;wrV{uVftSVl7l%JrXOilcCRc;4-#fPHYw6
zkUw}J86m6-6OM6Za20f#T=fQDIL89
z1gknEqNCSPiG;`B?{oEkipqkiK!(j)oiP`>(-M_69U!
zcCtfvzp~EWHU)%$-xTeb5s*xB3$Zv}h6~})5s0NZwHOOe0LXFgi|qpeH>fmQ)>yNM
zzVr!yea#rEAQSQYT*S4FAzrCJd8%pZ)MU>MnAmKc%kD5L(Fd;{dG%wylUBqc|6=+E
zhjEptZ9ddh<5|!ulN3o5VJfU3&)v~STgYvQyHR
z8OS6|pl%(SDZIU&m!66DtUKywQS8hj{cCoA)YxWa?wGp!=lBeOX5aCT2IVDY(>~Ff
znTaFVXHiAv3r32YFUog=U4tr?usX1+Q{Rg~U6clD#?Z($i6?vEOdDdii@Vn3;VQaV
zN)}e7<1u$^!nTrI)j-nBgD=Asrc|W(xB7?J2aECny@t(y
zkKp{V5AZ@dsK%#lx@d$qyD}+VD}Lo9n)1y1c$k>pOd5F8Z>~RcAMI@?D2q?32GUwr
zjK}eC1e;x>>-WMFFnWiwwC{b|D>xexL)Tn>A${eez6Mikv;pt$Jb>q{0(DGXeV;jj
zr3v3#oz3(x;4x!EG+tj1W|{YdPz^kPr*cHO;WHpTU!)1?T7(5km?}LxGjBn@avh&v
za5$K1FMS8In1`2Pekw81Qs7h6l_>OfqqpXS*1)N{U5HZiKhCNB_#@)1yPJy+MP4
z@%rueJr9pZwf)xag*Be&c?w5A9-5S!_jZ0jFLOp!{iCio;mwbLZ
zo#g|J8Er6GoPx1&mQk-MlVWuJy(t37YuWL=VwL>WaiXN8{fh=cu%NnTWRTjl=~Ys`
z{?1co+rX>bBFP-+Q)&IE)fF~O1x>aYo`KobJ*qx5nEIG5W%5gL9PD9#u;Z=Ev)xo4
zYrbwSVCA&HzQ!z0rouMgl&EdL4J3TT{l1=u=p9WjyG)7|ByI5XIgZTbo5O54@5~t+
zU*FS+i)T(>4|V5L*iRHqn2}$MGk$GWMH=3a7bu@zntFYR_?{iws8gIdf0pS=-_$&=
z2(Tiyk}ns0-Fj&5j(qcf6KLZuuuH2`nJV*X7=%-XTklNU&>d0V%&2g!5FQxHSi-j(
zrQ}7p=SM*(7bQZUj;y-)Y4+@mmP7Y*cHihHnM+++Fb0}%6QsSTNCX904iEc7Ohjt}
z4KgjxZbAJ?u!-g;!ovkLfo%n3g4+C$ZYqYUTlP=>wsNrI2;ZfD&SwbEtw_+5&0ABs
zu8o2IQan{;yF~?#)aT4gndRYpl4N}1WZdLcI`E%7L?0}wg988b!2bt<%is_lv|eX2}!u6}!TcJ5{6A)5YGfK#mDJ`NB~{vSZ4wLeO2T
zE6IVl#VxkLe%q*j6SAU_Wo}?m{8{+9Oees?@KeoP6iwMrCxML+@N`041lOD6D{KnK
z-w2|96EHh+-_uejq~zB>g;9xgwN4h!T)s%_9gD*cG+VCU9b9H(g0wORvX^Fh=j5MF
z+j8{6XCepDXR#iLZUy$DdK+fL@FgJ$olj!FQiwXNzbXlT9#1KIz_x9AuOM{q9#}s#obMBauWgD+uXsB%=XU-WqAHi@XaUKA8GSQW$v&0)
z`SsL(;~fokPcNrzq&WcnOqAukOIzEcKvrV6+Y(1P)K=KVFmE62zB`||&|l6@
zRF+=#*@rTJYCECJKvOyU@OfM-p^*g975tgWE{R3Jgqzk;aK+;rqh(Fm=sc#2=mQ@tPbgT@2hSATM>Q>N)y@7J~6{^BUU0JZ`yUHNRFScKep{O{06s9Ru
zX(ynsG7h7v3xrK50}1Q`rpCI6K#>SRq<_*BXf#-VxxoO1=zeeDZ+I35n(N0{G!oJ5
z&;f7a$sQK7qd4tk(2LEA^k$_sS=3nlxu%{xcPGfR0utY^-n6x+dT3sJ_#>Ubzz<*8
z&NPxoAi@)!$k*B$6RZx-2g4P$;?Mpd8p33Ox0eaYS2nYDxFj1(eFE!TNL1IY`YrvtD=xw#FD$#
zXd7oA4WQajG2Pm=$78Tq;&S`9-hWozq_s<=n#O?S_hu&$Lf?)R>#q4x8Q8!|U)H@5
z(MLzs$Ud_eLYlo;G^LLx@7XU`N%+E|4s4r4C^lx{Kz;?I
z+TS%2;|nBo69Mn=bZ{PZUGEZ@vQJ`kSvu%*A3lF6H0!-_d}GAl@g~Y*9}>jdR!vkfSVuPO?hNpN%h6xgvXTr-H=J0fq%%cxP~`=0>R~c?|dwO
zE+qz07`h+-^WSYS0H=+H2z(YGaVAeJ5ko2c>(Ib|u}ScTi^re|`4Texl;WRA`GOsy_L?FV{7OeSS7)JH?lR6Yp~`?inZi}>xw*_O_me;98|6V#
zoq86!wwZk+eehWMwy3<4EAg@u*c^g?CvCiZ=L(S%?_3nKOzh_+0XpX7IbW_C{Q7W&
z5Sr`HW{Z}5IlP&fBFyDt*`-mffViO*b5nWOzDS;RK;8&s3g}
zryHX{%|0y(TMcBPafI%kr*hOs^ee3y_w*hlE*`XD3Odc>la|?PJV8Vm)LqwqmmB!@
zV_ZK)pyVo>KfLMs7aX&S5WZ0IkaAn<`D1zZ4m^ul|JNXo?jJ$WX9}I
zzK@hI6LoWxpT4=AB)u8y^DiBLh4qIsv)MyQi!^`pVyH(AxPosdH9vWCs=t$k
zsq0Uk8aY!=jKXKO7@>RGfp_*W=3$xM$KoozVn^-!6p6@EO$db)utb}G2h%Nte>Rex
zqM8Zw3rf^#J~up}5IG4%_+tM3IwkShkotH&w(BbORbd3HjWledy6)FjLxa^|Taz+X
zHqIIQCO{L?laAWR3L#Glu8bV7`=q%|b+khu|%If4Xv-XxLOP
zp8BoK3wy{HB8k~r?TFL8<8nQIs~T!6OTU?}u$sdj@!dKMcZ_)52qTLs`v;WrYN{=$
zD1aXKr>wDoa78(PtvdV;*nJ8zaCNwvM`IiUlXi{7t*zyL?f-aik+?HTMQ{RX>x4cO
z)z*t(!gkg6=K$r(Rb~m`0vRgOc*?&kkIF_h$P?TQgC)Uj=INs`C()
zjtwg2V!P*!cb&hRE%8^c*fj-cA4ibU{Q{-h-2wkL{4*!}U@yoivy%N2-r8rMgPzgh
z_CfqeBRWCZ!I+-%=*OhpL{`qG3$g-(Eo~AgJXlS~@3AK9C*=)H3}jZKdkgnRDmcye
z$6)k?3nGVq6t;o6Wp=Jy82KAT9t4hM_Znr&$(_&_9NS}~7Krh__3J@Pv3n$C_tUAj#IqB!;O)%93w90>g{d0t*e%MVabms1adKFrhTd=W-O%)JGdbNsZBIxs}1s
zI^4EimV|Y6+Bmz$qolj
z{XRs0%QOrTA#%0ND;#D9PbYdoLEGSuj
zh%85n$a#+k$y=u$qqIkodfG0QG^}d5b7%m(w;|slTld
zt+s3u%%ncY-&}#!M(3ioO~yX)<+o^fVhEh3zB-{QBSmR&=$%!YJSDRd-dn-VQ=F?*
zqKNzPyChx;x(>`PG`D(E7rvNK({+Xgm0!0$HqBr;J}p$K#uX17tV8_xO6q>NU=s*K}D{p@df#c@5%Z
zV+j{uQqEs)l=5M^GrmIBoj{&a01}$vMUIFJ;OL4L6!5*i@pX2URq1E+@o7-Pg|=Rn
zsr9TJrt-~-3Ig`5mdc)Fn7@^OvW)yzFzjVFpOFH&L&;0@DoL)Y2$WUYeU)B@IkmSE
zl%E;5&)8%(6xhC!_f<8kr4OLw(H{sG7;1q+l{0eHu2qG+0gw(>yD{X%rP(rxVG)xj
z>Lw}l%VhAw)~j@{$-mJBdRfRT>jb(?5+^1_|N2#RsFRA&5tw%zB%)Mthm
zKyIcy_ho*}(j^c5mN)^X!6kBPuB>cI!AfHR-%n#acgiG96h*Z{Y;b~^j>2n*#U_9t
z6_WssLH+y`1bfVNgE6$V~{U=Uoko)F)_#W
z%Yt331k-SbJ|ALM-WHgDzmeY&C>KkLf8~QAf-$?z_uuJ?=Z@pQvYPmxsDI`_4r28r
z-yxt^8fk`K=SsVhFd@n*)bR=yFSY!_<5~7lYPSmWb;Wt3$_%DiN;Agl9uw>INAOAq
zRQQ~!76ae9@~2dj{2F$YS9{Wm%_XFbGCaec7ppxPIKe^4#J{P3)(y{?>%Q{4l9`($
zGcBUyfWRkg@*Y>neo>(-KW_xdsD=K#1X0nqy%Xx#RFsd
zlcweLn--*)kJiMG;ICh#OyAFkhmuYxZ0IXKiQGgLp;(xv7&Up15+CqkvtXb+!xGpo
z8m;{HMd2S|bl0ALm28NY0DyZnp1b{`v8#gBYZjw#-+4L0N`{AL2$kZN?#Sqa7?trX
zRU`5fTghz{zzTO@q7>^c*cALe(RNna{~hffR|K}tASdDQa!l+9zg6!yzSPb!2Z_g>a&S%@^h!O-8P^|w9d#rO);Re;gGKYpOR7!z
zU;$uwGw|51Ad`k9xY$svb0x#c?Gjz$qv+3HzWIryf3o02A=~fxq8w>kFMk|B83!LR
zbYlDiL7C~D(}Ci`L=ol4SkP_DQIQnQ;3pAsWCfER4Vn#|Vyu0d{Fp6Q^uF(L7bds2!=lZ+
zis3Q>6lmB4^F&&~LJ1$cMU0?P38mv}ZEfb5tUcH-3aR|5>{9aEA!daj>Pd!CXhcb22ElSqw>h9~W#r-tq
zn^3zkaAtsJ$|(GiES@--A2BHWS+#6mSGq%iF`)za6wd&9>PjFS&F5?^Ug&owt^;!-4xmtq=C6g_UE*Rdn{9H+OH3W|j=}AX_
zFBv5U!U?8ckeusut5Y%_!rdd>6m3E9kd{8L2@~3JRQ^)e)$uFU0*|F#Hj!>CsJg8-
zVcuBS41YQd!z1BijpQG_s!@*FIRG;}1yUcOO^(r3MND=sC}iOwX$m~jq3CcO!y|*G
zj6_wOz~GOG!@hia5xJ@%f%_`JbZ_8)e%{GWmYY8Swf3qen9ZDFM{x865ZtS_EiBlr
zjFo_6bX8MRjBa-0Mby6&Kc|ElU&R+(??MDJA$?uXJ(RpkjhG!sVhpFg)T3u|5
z6&_9J*yvV=8p4$C2ID?aDmxpRB>GMqBhEfO8VKu0O7kg%?Q{{m4b~YqYHA-#JSuU2
zB`XJcET9i{E1%}wTF&9=lJ8+GRdWhfms-7}ScX3JD~!UY2>b$xu^{LJ%wk>L4Qv+G(%Zc2mQMm2hb
zUYB@kUw5!ja`zAStD3Wd5XLpn+peuxa1(9;!emZru-ruNfdeNgjU9YO*<~g&HD>R1
z{)R>#l)uVmw
z=JC|@$bxX@8ZJ1hv*PQ2og$AbfcNz<>t6gfUmvaD#|6&1j=mlV-)b#2zg(|non(lA
zPJsr4{Bkg=%_!^&`I6{Qc~`EG)zqSE`Zh~5|Ea`CDloL+eqlE_rnsLho$JMfSLCl{_d9%}j$MXbsItqGkW&t?k*PR}fj
zZ-+3?S*D0P31Xmj(J(t422VvRJTW$F=oR@Gz-YnJQv8?Rp?5r+i3&!*^z_1SI_ZSP
zZNv?Kj#I-4=8@#s>-JzWNNyd7IDry~i*xi4t7rWkb)#K>Zj4T~F!iFVOE;3!C*xoH
zR{axvk`KU_=R@mW2hWGJSzq9ecE4Rc@qr_4_7&@ZD79XgE=Jlf!nLN=wedK@;k<%^
zf=+$L?MJ#)ug*#Uy6~zy_pJ=hyLeRV*V-R3>hdgF(vq^k4jh-H(O{RZm1s@)k>~WV
zgk+7+c6;uB3O#W=v2KSP2fK(KJIGO^)V+viIERiQEb`;+QBx3}74b)4!LGbu`{p-}
z_uOO5l9a;3ST%5K8K;aTR@aPuJLGUFL&v&5(Tp;2#ZO0g$KB(jox-aMQ=iW}oTqJw?-p!im0^fk+=VGBGNhdh@8WhDdSeN1v=A`Wgl!erTZ)AowQz}
zZ%){M`ut>2Qxr!JwN-_Cbd*!olEzJG4Yur{`}hCk`oSU9`B0a=dIQ1PDSXT{X!tc*
zvO^4HT{*3>)=05j2+LH3B=S42p}m5~2Is1SU}=5J^mZOH<85*R2~zgEeWA19@Jvgu
z%`+*b9l5EVPENc9<
z-&CYUE1Db;X|yyUrz*F?KE%oDkU
zOI$!~rKL8}7yU>7%7k6PGqN=1o)P=iEknJBzJ^!rucA>X!z9YaWo#vC967#wyJ{?d
z%e=mW0cJiYTBwtyTvqZq%wUYT7L1X|y`Rat`Nq@ca8vgXNx_Z%Vjj0edqRSPFYTl{
z=a^e02b0#D%zw)D+8ew(N3rB-L?LauTHe5hX@!a9pzQF``a(VePN>lT
z@*1t;o%xC6g{`g@%ab^~@$=Qs*Q}60f~NOK(ef0}sCn0w_VFQ20^S;{`Hdcb;8C50
z{re2tJk1fp$#Q4;U{|f$u3Odr@h1y^W!8sQpoNp2P509O!qUW+A3XVLGEr(`LR_ymysDxavggZ~Bc@aSIR>
zE^@q@S)a1@RA8ppvJ|$6U!jA4^nMCAnt$JZ@vo1rPuN}-rKx1%$#bLKdn8Y1e}UrO
z5n<~gd>}9ytAaRNXvfR*^BVW<^Yzdv(W@_lA4wcN@Li#MnhH~8f?RDdhXCfO6D#3K
z3?AMXwCREGlXUQ%yhQ+ws*MV$E*)+)!)hM~!}M3V^%S*zUf#-$+Y6U}C4$W>D$APA
z^!e+3?talbYnwQ=n)WAwPqt`V?fue0qTl}+MpHneIek%E!qH3O+<<&}O9>8oRb#E;
z!}#*M9{997rr^tQ9o~f-L7V7Tsu150Wnw8b|LM@isQpA
zYT_Ww?ssS7Ro60>GE&+UcL$wBaUCm`rkyp}GVsxpa)J_$kakj;Ahl#esYXjG?e#Ou
zSLdYnbo1oSM;Q|2PkcmUG;4VJsd3qoLX4GOV4sb$&mGHt)X^A!{}^rJzQeZ-&Z+W-
zFs#C(yHM<6%My0aTAz_8UnZOAsl2~^=?tC9;q0sZBC+PsMZ
zYV?=0&cAPeK$r)pi?f(~xQ{0nL!t*p#-pQ-X2iInbx25zFb8ubWzImOFu345BU2(i
z;q8gmuH05uNWoGgW}-#KuhA#$ZQ9DGmbRwn)vQu4N-Kh~r|M(5v09zzV7QA|Mg{e>}y4mn(jM#M~XpOwM6%(54o8dN(3v}
z%!wo}i#sUsq~7{9jZvrZ9c8g-Yt~1R^&Wrr%h6~yYx?|L{5joMcaZv|o!XVXX;obJe$0R2|#>VqaCSC9X7R~1VeZ9)XxKx=_
zb#~T&)p#j@EJYB7?P7Z1FxrSqGu6NKw&91Ose)6Z?Dq68;s=-2oalvDenyq@d%@bP`59n+`Z)ux
zIH>EY;$kA8#^s&_!*}K=dk|NJz)Rok8P0SU6at?f6`8G!h*I9uP;q5WK2T(Th0xbh
z-IQM#fmIF5F$eR(?LE<;HdEd_hiqRk_w05K_cvu9lyWnA2>;{v5~;4E0B0{6S0&EU
ztlzB%R!Lxu$JUk|kM4!yGC2*eIlbET0i3eS+$Il1clMFHk&q5WR$E5^eLtxJYw{$-;W9>==
zHe+76H^uPp4U~j3v)068{12hb6m^ediZ3t{7{EJ7h-#V|yAlqn-U62zNkyhXB}s|n
z(@aq}_#Fh+Z)cbjJ9gx0GR?p(?=CP?8^OuTa|e1I-CAqf(38d*7NkvoFX313xX#rc
z+>gQg!lX;eZs4b`t@OV39Vt6hv*X@#AEnIxwl1fRa)F`A)v=neVozfpB)Pg>
zBpS@ceP
zZ13KqO)way5pG}Dkr^6$hY-M5^w}+xIw8<*Ggx_0E48qer;eSFBSm9!!*GV*Icnb8
zmJT3d7Tk7A@~d%wx!DlJFUqf?pa@%*(VOfR&&m
z7>a#+xlU-Vv-1e=xfz^!6GoNqZmG1rf#UEk=yBHk9GffV`v!
z3tZ1Bt5v*U30GZXgUI>jdk}G5a$nA@qw-kd>`)8ZR=cNv_{6ZXya~2I^;)>rmOfr;
zmzsf6oDeta+*7sASme3)GeS7*^ZC!e8O>ptdjDAYU3d0A$YvvY&mr
zr-w>FC(*LJ_si1nd7QaK!*I+WiyOUBeoo(A$>JrCBJ@{%KP7wX%hdz5CVWXfYUPbg
zCV!E1f^nXIS=fuHCd;&{h)g-jX*}cfjfXRUvd5&@?a^%!1i=U2V>XBrFDOj;XEUJD
zAeIF0JYh4k*P46vw=&`rk8WB@%+*+nzNZ?VH5$){k(m-Y|ax?UpcX}t+dw%;X^%SOwldKBk&hxL5j{1)WRu*2iS+N;D?dC3cgoV&sHn
zY|mLB5&ioEeRYr~$tc;OFx64CbXZ8=7z(6ULpvz*hhZv^PwrJLxfBag8+1B!!2&(+H|=N92W%J>`zl(
z^s}1btsTj-(rrW}MdzNxZ}%GY?apjA0@5&lE-hP;-j11|glv)&k>BdU)Od9w&@F(J
zDK~|}Q?9R{`-B?SZe+>SLls^aYRQG*Zsk;~XWTUA_;AKH4O#U43k98*K+XcVCcTKD
zOFi&Wr(&ME#q!h5!Y*61wf^lSs&zU_b5^8q$~3>~wfVU2}VyNFt9
z)dUMBvm4#ki(l?Ir=pyqpe|uQVC7r$1S&pO
z$U~e4l;a6F#jBB2P`X#(zOil29iv=!Mbj%Ws+(zx}(h@(b?=#kaD|BG4
zY4JpZBx%kNh}u(Kz+drR=NR&oT;SmO;F_)nl9fsyEDEZEh~Dq|8Ezo47GE9u60TdB
z$_YT(F*m4Aq1~Y0W+>gj{=*&oRMedItw`d~Xt~UqjzK!DgD2ZX&eWlYBl!WpUu$oO
z8)#`dfJO54#QvoU?o}~s>)mL7IeLa^l1cu;N@3-s_sph^9q^-`y{o14t+soMSs5;fz(gk8L
zoQP!nZrD-F4^=IK3x7x@Ry5xZ6HmWcAuk_Tm9hv)_SiFRzUBFR(Z7{cGU$8$4}-G2egWi{>C%(*HiY)
z4+9r@7F8O?Jb3j2PqLav^gRFKmcWz-wcD>
zkvj*uk+46V6b$Bk{`
z_6BkpcuQL|&Ce@;6d9o?$s9KB_J?QOQj#Rlyhd+8@o36S;DU-gL{?_)SOoUR=TC
z$@u6D_KaD+UeBs>z*~aFg$c|qVor09qQDV)_LJr4
z-P~epg9@9lY(koyfNU)IuV})T*Du|nbi?5JEhdd)
zwOOPD+~;7R1KDtPGfF`fY6l1z4U+}6ZlBQ!-g6+3^b$nle>p*jI|R%6v@7*OxcII-
zc{z$10tM}6fWq|RERl-#vx-WiKn>9bX-+B_qB7HpqwURA&cuttoEKlc0ly=Qbv-_L
zyfp4OGY{&!#@1`Rvm*QsUZ;$u;li%&84>7>GuSS^tgOcv^^jhMP1Q`GV(K6C0oAQe
zi%4U-MO%!)f9ptrsq8v^Up1BTnv|*aqB=b2pcNAe54acJI5nJO+aeDFy{n{ZR@()f
zKjn`Or*4CvTlk9vEcbHUj;4!c=8MOwg@iYnZ+;=y7rN{UBS%ZS(dxh}lla<$hI(?P=*=
zMi!QDw|~O!=-Tlh@0&wdomG)xOW)xD$;hHZril#bR|k1O>R!^ThoX0~Tt*(|*Cz)a
z6;+OXfMCNL%a-~rcxFgH@T{&;Rx@r@--sLTe_a73Q>-C{|C$Fb1DN+f*t^uFL^)af
z;;Eu+xq@tY5={V#%~wB`x{R%`D(MBq80Iwgc0KO(pRMO4iy<4Oc^t;2O2z7gIBfkG
zdwfT{K+aJZ<^7_jCVeQN!F5J0zZ{}cDA%@VJ8M?Lg)xSe5#41bR;uYtem_oiIR>)`
ze~H9uOH_~B|KsMc+_f*8q@fF-(_UO`l*ylMYwoH4IK^Mh{Q19!RR^GLadwR`+Iq#k
z4^TZizGK7GQw}vgQgGG&ulc8`){&(<(?6Q+HDScyfWxBB9>jHU5^-0Eu9&kvMc_x;
zdtKSlF3NgR)vj*;RFLJwT`n*4fl`X&f8sqdG;;TEVOIjI!}U}b4I3iPco^%MsOSkJ
z?a&7|`Q?XF_0GM#$1r)kuy|dq8YvS9*e53>g^E%uo=4QR?P$H){0@hRX%*Gs6qgkj
zibf_r|NG($@I}$_D9S_g_vAPd)n<}^DXS#Fhv4|0tROGQse||fp?>M$H3Y8`e`vZO
zViRcHd4TPN%Zt9ae1u~Ri@RNV^b_py&_7p6g5W)*QvQaQ&yt*LN2`bfZ`U38S+7Cw
zHoQZrV#cpa_2TBY`XKRm8>G!TFPRClI@a7w@CDP8XDaqz4FOz+>2=y?(9WLQU0aHx
z)oL*jAeUnlz${IK%4B|}*@IPpe}}av(e;t7*#b!MM
zqX3PSLhuVSLO@R8?Wf3lE@)6~K7&4l8B%@UhmRU1{q@jIfML-or0fA=iP8t~x*kTDRT_nysfH$m<-9Dy=gGP*I6Q%EhH_DwC
zb0PWmu}bn1cChlgBc(*d37`pQO*wP)o&ral{Jh5V^hAI`Adf`NonKKW*`J~lE^I+!
zio+EToyMsAijzCwybuLF{?BM89mD{MdwnkCOa7qXY|N`I-zEu#f1}_L!&<}%d`h>r$zPebG5u*~
z0}yyS?j6(6PPKsGJsdRo=U&6it>3|n17#POeK9!qX29+X%_P?e3-plbmvA;l8)oXo
zyO(Ti+Y4_+YnM~Pe}HVn!s+G}yRBbn7|sf%4~C~v;#(r^XSkICB!P~ySXGTx+IyBu
zFK_JM(_scW`KHQmm|%OsPTDzWc8*_a{Ep`0?wEEtGuRZ6AKtUZu+&+=7il@=J6Ydo
zo00goIe`O*qI+{z|Bc!hD<+8Ro|!I_&xjV_c2&V4j8%6$f1}3Lx1Ms;7Ja#ESxbL{
zmi*&{UfBBap7Ox|?o;rby}(4B%z_Mfq@N6(r_ED-8UzRkO2VYj{T1)};|lb4A(V$B
zvysstim8e{^wCroW^5d1dF1lR%_-hN$m0Qd?e+OJ__E5b7qksnI?FnifES-g?
zdF4aDjBM1Rc$49OF`_$A|4ehPf~uPlNKKt@i)PVKSsnoVqHP~1&O!&?!!oQEekdea
zy{~UfR@h?3uZH!GoE}$CLG5~0Xb}uO^Ozxk$22>Z>V2qU+&w;ICe$K%v4uS#qa0x^
zdfvLwf4NJs>Et=mDuZNvr;aJ8&2Fw>y
zfcxPzQbV7kS8`Q)Z?n@qSxK-2#C5bv^@PoowgYB3>*GAJ_I~CBKQsL|cmhg1x}C(3
z{de&5s*ozsNtFVkcC1rOa;Tk2zW%rL=YXI;U~q
zaQ-xJXRXS;0i(OF9=2&0D?pK-~vVQLTcP9Pk=#GH`7`5^3UaGY|&`-_5ftH}614rt}!BC_$iie!oMr$4x_2!-7%gXg(U;b!0=2nuq)^;$8AzGviV!
z4wL|e3Ozu_fI;kV6q$pV{TADba^N{D5Rr3Kx
zv#6YXpGjYLPKy{M8mpLPAW0AH-Ok*6jGU8U|K9bmD~mvi4uBAAr9jK}L|M(ilme^4{T1rFLk
zz(5j}mi{P($@ES&%*e0#qcDO?ThA#_+K>K#CJN?~a2C&l
zRHq!=u7Ftxm3t1ekPgo0{@r@Omtc{e2H$t|UG
zk{deGnw;+D&`PS5B7mbNu*9V*iJL38WOCfOTNgd0Ei*sav1efr`N*1O{qm*kUWhn>
zV#4{C^kFFZ#Nta-#)jjx|4Tpf3D@cSCP%MFSA-gkbW9>NK*ym9i^iCNJfkn4LO7l*
z6@zdV8O;^7JIAv~EaV3oy=likpp*^mC
z8I2h@Hj^Rp69F=pA;tw16EQh6GYT(EWo~D5Xfhx%IW#vomtjr?6a_LcH!?Ywp``&R
zf45~+oLkm4iUbcD90G;AyL)hV*TSKY!rk3ng1fr~3-0c2A!vdG3-*!I-EW^h_xBwP
zDCW}ntUcGRDhd)MRR$3=kO}Y;$R5nV!pO`Ekdv1O*@Ji(RDc#Pw#H5XRz_xKc0>va
zF(;rg*a~DXZVU$U0=U4I07X;qTjpB?e={==A_YJaXb*IHbD9B6JOJ`Qu(6tl1CRwk
zZTuHd0y%>jOpKl1+(3H^D|;Z#TZ$OS!NbYQ!V>(a1{(vzpGtqSMHvAy#-=tPH)k6w
zfU&(9K!#DC5ugBadox-As6qAs6QHHBtvLW>4p0MX0n}9`R8#?yDvIh#sx*vme}z?D
z92`JS|K=j5s-`YU4-gkoP?G=vHRu77>Z)pg{;C1(-}qb50~FNW{D1nq1^!8wmrxT?
z(^isTVfyn702Y8N(8<~APuqWRqkLlq_%F4$RC6bg-Cqp=)Rthd11}Sko0}V>g^M$o
z5#(gS=wSO7KQ&7$XMh{X$p-NDf9eFZ1^zV{7kjg}ae^&@|0?k3OaO9Lra*gV;Gd*V
zpno~--bVSB^kxVDAF;PVfdACA{Z}}^83_EJHkQWDf91+4DairsjIHd!Kzn0*)3-pd
zG1$c!VDy*m?H6c9^)G@zfS8Mu)1Myl|8Y6}Uz`6>7X`gNnVzkekFnc7e^1QV-o@GT
zZ?pN|$2JAoJ6kz}o&Qx42r##@1^!|0{O8Q9?EkXKizrBcl2BD+kb4uJJ%c>xZ94Xh
zV0Z9e(SPcRh|9f6k&7F^%E}60eiNyLy_p!u&hCx1Gvc4|iCewR2@G=bVEU)5ZR|mA
z_Fn%Vv$>VMnfaf&o4GhJf2rGBIl2I)#s3}rWoDpFav;&0HC|6CDWf~e`)29
zndOi9Z3sSI4j>1Bxv{M?(8tOg`1XS6c>ZO6WAY!@zXd@3*P7D2?Wh^Z-qr(P1~f-xQUHP9f8;>@|Ie2Fqt_=F
zTU!NVJ0SHxCi>4XV>>HbkG}{0BR~`QhbgrJ$jQ#w_8&eg=TBDdKrC;>~9DH^>QKvi=)d)sxVi)~YY8p)l_;F4fT7@FJgdSM)RIYfx0*TANq{S!w?_&TDhg=x&!Q{>
z$w~3*`=$igmuOK0W7)oJwI^3k|1wh+-Z%D@_a#7)e-IhZ)nHT)%#vU6ujnSFIhM&S
z&@N@4%j!tzAOfh5pNR2v_B_O}PQ9XErIPAZbZq3qxD_w3v|>8hVz{rM;dc?N%ydD5
zUmq~zqK*4RR%z#r3PQ2RWndk?_Mqv|r4^`Yo%DL8w_R3_5-mh*>l?=2ZfY8Cjy_QP
zzJ7`ef8B9y$-dmx{A?~EeECKB6x#GlEmnb&!iiaNn4>AYZL@YyFhOakez#)gxBSDH
z8k?nsuSoBlq5!rdzl-htxq(qgl=~jOCESh@$^3$e?aQSBrR0vFzJ6>&wY2f~Le!(2A_`
zf2t^Na^#$d>+HQpMxBdAOYJJ;`Luvzk^@I|8dF;ci|+5mjX)A8TUNDnDjq4Zif-dH
z|C)~j5&lnZ8TFC15n>oW*bLIdJhBrPR(%YbUsFSRky&#^0)Mj!&PsZB^z9_?<4ZB#
ztcMn=HGgqF?*39aWyb
z%vdFD)7a$F?cZgpb@7U6lgVRoEOXVy7|h#PkGuppnJ%f#iklVjd3@%nFJ-DYe<+l|
zIGN0j4g2wJMmn2I)*Im}Hd*zy`s-$ot}SQK?zhmB%DdtZh_c?>iW){AO?LXWsU%h%
zLY8o{AxkaeZ#x3#Zq}*c17=-5ULTns(Z+_R?H#{V<6N$Udlfn-O2RkwQR_jb;#s1F
z$)oWIMOIYdav)_{^m@l^ARzIee?JNr3LQWo?uGDeUm3vg;XP?-5Pcx58iQ35`9a^fu0cuCd1*}A+oD=1t9%$f0
zZn>QlMhkmm*rjQ2NQWWV2ldH`mqhpN49st2(kjx|$$=aE+gj71Yf)d2fBBG{W)g1H
z46glCV(O%Ou6HQBp%O>(5tF&9HR`h}LKL<OjBz5&K>^PgD12e?wdD@7KYmb=3M1D5}*Kwr4$6Yf6Ibp<*NFUpk>z3Su(7
zu6yO`Sy^E!r?HxYs3-$+DhZ13P>M6ZlvZ3mPf+g0OdpQFyZ!X&_HKLIolm_akl5Y0
zp4#oJPq0!7^~adzLyROH!8X3U$~a*4Xp?T}rHc4b@_VMNgRV73e=*Wtl@N5NlIfP9
z_dE(FqP)nYi$quQ5w>J((0{dBgf8bg03!%&*^1f_Q=D6^!Ra+IWX!SC#D?R=635Ou^(xTK!8|r~~
za{YCWTg4%qCzloT>{0j&ZVXrJPZm4Rt|wU3K2^byJ+{f~?f~8#3Uv&+PkC&~M=2;`
zxh;2f*ow8#Rg<4fgLp}Va^bi85>}9pQWQh9#l2VOX-LCUAY(0&*dOS|A
z4Jy>E5e{qumcs}r5;6}5s)?6PV@^ZTwv^1JUD;VcDG56ORf1XbgJd-4{?}e9**Y}OWZ0;Pg
z2+V>*Db1T`zh;Ia4x}|;SgETEkSG=oUbySh3FF-=t6&Xy#U#KUDw^AAW|Xz-wlsLK
zRu6=BWq)N}mFB$)Fhiv_pF!@sg
zUzTBQf2dUX-q2nTvP*6-tubXg4qLzH(zCs+IV>@aMhkmj&nvRm(gSlyYux+yau{Bh
z9p}lB+&dprV^TEK>81|B*@~22F+~w;`0l*djr(#~ZgU8prIXbXrcCYDk2$gW4KfZc
z3_+h4if&5?;0Ow8S#)DR{=|JLw7g}HxK9<3e<ObaE(5LIyZ6(~p7tH>M-+;6^@1Pa_i
ze;mB0iSN>v$=g@A;I;OY2_9Lg$lg1Fm>rIIN&1xS;1l4#aN*wO!CbP~dx+eYewc9x
zD?d&CV^j0Joe3UF`+S(xkfuajYqb5inNdMN8LAQmStk??4R-VEQ>bt}*&1~6{zN_p
z)C?a6Js@wZ!|}&Tm+!PYO(U9UQc$zBf8TYse=efnnr?T`2g9x{;S3f9c9q$Fxbdwl
zU6}+twz9k!q~!JdAa%knO$|%1xBV5;I{&N$KSGfU#56eTSEdTXqKIx7Uf8Wb)>u&*
zX=(Kx%`bgZ-#Hg9!QzNUGONB{FM_};uWt4BM)6ug#ete7FUT!T7qI@g>)}
z{Zn=OEFBoQF|{&T4Rs;7v)+QvvkuO7?+Ye}318=0EJ0D`p}eT3$M;*M3!gaA7$hs>
z+i5#J<`VboYM00lCH63?O8|M^e+9lo8T2qJ_m8`N@?nlWLufg64jIZ;3QO1CB%E$0rATPBHrq9oTf5VcDn?KuZSE44pViR{YL_-(C&fs>h6ODtK;~KghKM7a;JenA?0!
z6-?(1cfLIV)E(5LcF;kWf4)!OF6J386{;GX5fR;@@fFQaR2I$%(vIseJjC7?Ecm5n
zqI*z33$E%25)p&Z@tmyH!_$=)#;ie)gKHfa)+0DAy0b=H++Hs^OQBWdN{%w6|sOlY!RzWbip-E#ED)nxS>V;;2tzCa85suxJ%j7t=u2|fH
zHUx_$tkf(kDZDkIVlcJfx1TB_pYxll`2CZH*Dcl1{DHcCo2CVDD;@;B$;vdQ?sV(J
z9Qz~IcCr&j;lDSMe@h+7_Y%gbde9Dj4&Xf3V`UjPYBRBFBwWq?Vgh~itJo1N=5GVX
zy*tx*#OD7{qD-A34<4H%rYnqE6oJ@M><#@wQt6}|FQ)!HSm2kWSQU)3u}Jpfh8vZkN(9FF%o56+HKR+JD}lR|1ePJ{x>W7$Z${oz$?_sTz1^RS
zcI6j2gXHQ(5Davxg40A-*dZSvq8cZsV!y*d`n(9^g?=L3hL_cDcDK|vClx(-jKDS6
zDa6X8dY3jJfB#;e{Ylt1%=1JH6aPBwokVn|c*-0sL;lLGr96UlzN9w7_`1Ls0qQ0i
zi8YyIbu|8db}$dD(N`%%X#Sy}9lE<0V%O0`eEbbKEGi@dcacX*nFoj8@Bvh3@Kt)3
zkYM)s6hS3W{*PQp6uG3{xXHW_pl{-8Y!=&YLA3WFe_3POF_mQcHDj}cj!|T4x|Eh@
zmpL*${On7VN{d)L>%wg3B9jE7e_Rl>zW6hZi9>QQs?+qr@J;YMv_sCu^al2q2CM@|
zxO}i$Q4^U2XHUL`UoKWg-Swt<)V6E!FL0aElo`aJ_+%>t@(HSFBfRFB-rS?YZ=G1F
z%CHXmf1G5R;+cbaBNEC1WQ}VZFDiq(JI9s{>a%JiK9ROnZtUzJO8=K_-SL`}xGN|{
zJP)!%E!FJ=o8K>NBGX0d!~|ZWk4G;xLYLPY<{+v+l_81SXDSSCe4&f`s
ze+QRJO`=tDkpRnuyOO%91gWUdmON$lS$QM{pAFG{=$sFmIo-9N^@2tf9PmghCc*>N
zJcsaFv3Mu;5;JAa1iug|JOOJv_etll6lP6o1JmytyI|?49#J{QB)8?{Z1*8bO_@jd
zG9EQctT1Rk^XmhmmnO{9$`D)($S<5fe@Jd}tm1!M0CGc9zdvZ;=iagBG6@`&dG_qY
z4@x*hsS)kp4?v5ky*`AWBd@6YIJW=xRj_|U3Cb4T*=>w;bs^(*BH4&XH%!WCfq4GHF@&$U@-gJcU%fBnbd
z4PrZfBI_-^bs=XI#=Q3VhH=@1P-lmQ9BJl{P^-#5&7=NQkq2J1SeE?u4-c<%X|`!b?$1;UR{8aw}LwCSG3S88I&psH!{^=qIyPKXl^E5tG0GE=I)#3b$zVO7HYqeyc+f6CPNmrc*h
z>WG4YO_DG5VpbO0olL@hlp5yCnpsmNGpa9Izv)J~HSP#$Na}T#qO0FNYc|6MeK^X>
zC&+~KLwu=92q+m~S7x$SlbgNoGzv#u&(z|$gHd!IP$tcX#earcTXAl}co}c4Ls=33(`l*TQ)HQ*sfL$@qWv0k
za%QHFbEnlnDY0ow!%)jpJ2%uagUj>y0sQCUC3eTt8|T!Bc3_-Ke-ju47v99U#e~u|
zK17#SL5YrUSam(r7Bz+m#@&ednZaTz#>-{%ux0A0n@aS^8#6gAT0HorrrZ{vsGasO
zfI9g@sP4rFeLBCAoRZux!lsZrYqDf8#~mcZyU48xd$~#d-iCcjj1pP6=Xleel}|nr
zspo;pB(0^(<<3_>e=V*&4D42>)!PGkTY}E^==-N`qQjj+HLe$#s*O=CXX#_Ltr>i!6A@n6#pF)lnAstTx@dlynM~(h#Zu{4
zvbz0-mI0iVXxv_c*p(8Mxb-e+1o{_4X2~+E@v`@zVQH
z$tE0iG-R}C48YID*k>b9)n>%rQA&alI;%|fbRr8cv7oHivQ?pAo!hJBdNC&N>}l)c
zrk8dFckvmv
zht?u9$LR|pf05ij<6T0v*PYq~BFt18bJpC@v5lv*`y4EZq8WaX{3>%_q`Em62n*|M
zq^HK+qAvnFT{-ZP;Kz}MX8m2$N~Q*%pjUSzQul}3G$!;`C`1O%W9WP3VuhE4
zrO|Y+*$<6G%LAv@1r(gHNTkQ%ef
z(%=@i`g(0aYVSOjgPv&=m)o;VvE-KFh$gieJ;j#YP=ev}{0!gC9vpW9fi*$hk;GA)
zgl=Y>BBg**jmN7#>t=rFf3+at5=6d0Da-tvfB4+-n`znNrzXAL^wn3g!`xzoHR+P6
z?30ZFt5-f+QKpY+*BE~4gd92-Xb#u&*6)REp1C4?LPQIFvpJZ#TQS`nbs#oTZ%{pz
zVjmh5C`A%)RZ~d=DB4wBUd}^t5v82_5mM~us=l2S_YCtR%$tBmiP5D;;>p(p1)=4{
zf4qMWyhhi$L=Vwg04oV+uFvNO)qgeSmwcpTHgf|^p2dcb!*$;sjrUgWdx|?%eD5F}
zTSe5YOrW#NCc(j(YEMCnvL>1rz{`Iu_jvWHgsrb6se-`*5CBz<$p{AxD+cV>GJ?ew
zhn14R-&ryczI`xXa1ByQ$gxEp6RjQIfBErz+>I1wZTtf!kyBp=awFB;mpT<8W>_VD
zULdj5TD<|aYvdwIO0%y$N2MCMs;%`(Z&@*6lq~z#S6dY}6`Icl1%T|LmHL!aNNTPz
z;q#6P+DSTbO>zG`$G-7ruh>s^Ku>>#pV)Zuq!fA1bYe&(Knkn4@N;obY39Spf1HM`
zv0Y+b^Dn#FzZDINUQJzjSp?WjqhSp$G+^JuojX%I(4P5yrLC1QKEEd8UJmfoMxNje~u>
zMj=`*mi%y@etQ&H;;
zq_>`@=`nM^SM`S&PK5b*fB8X8Fc98^Z>p94Is`)^oG{b^NrP^3Bu5xjXm=*THi@Wo
z5}Ygel;PBtg3GYW7U*JIl0dtN`!(f4`d+n}}40+oT_}K0Go2lVThlka(x#$SC;1)l;UF?6UpE_#}y+p@(KCCt8_b;=bz2n_aML0MTd4UI&mR)p}$!?g$VBc3Oq
zmu*>2)cDuE-uo)ce{|loxvLhYQ4y^(f_^^RIWwi#OW$1lrWd}ZcaL4_3k+9INF=gvNEa<`3b_F}#PW_{d(zZ4I^&-6e>%2J6US5sC~RVHqM&oj(Pg0bwzBbl
zt#nBYCTvxwusy3u2>!YsLOexRUS|FAWS7riRm$N>XD3!ef76acO1qn5P$r_qr;3ZN
z8Q<~Lqx!?+cvJ;B-a^aoZEOxZsmlJFJ;9YHjt;gP%kWspG6)}(hz-@ZNfd{M-^hq3dIvSp>IOv4*DjFL;{
z+p#xuvW6NQB^H?^aArni(D-dmrE+I3T@e|s?CT8As~2m`$R;@z3w7Y=^7#O^7x&=@
z6r!Om6pr0ZdzpBzjLF8xmx+eer*|h8HJr^#Z234)f0x~;&P?hXq*eWR2JynJTfIkZ
zBF|%ro7&K>6v834DFHu>T(6~=95>9S`SP&K8}L7YTiMh5ON%
z_yC`rI%K$4H3iFL`ZPLtn&+Iwq>#Wx=3z4|B*JL*J&-4!*(OviMa{B
z2KJ|zj$L!TBj?T+eFVga;A4O07aNk#Z+%H7T!gkDrq*DK_ZI0MXdNx#X*1Mzq1{)omXoyGc`_d!05mf8Bj2ikD3p&u9_-cO8>5B(7HdNNRl9
z<;0gn+G~KaGXqpnQsSAjVl-669}JiGJPpOG5N9zWpO;ZN*$^ee+Vwqo_PeC)tAeCp
z%8}V=jQ!hsoU&xW7E-z5b4bU-)n0rh=KiB>*?tV#u#;&&{LgVT+;^^2wXob
zUgyXBq>ytKyMZr_EkWTj-1tlOe9F$Je;&e=Rt`r<{<{q87#3#Zj|joOS&gh?a&WW;|;~(--(!`8zI`2}>NYSVtoJb}!)?&vJ!IiU>+guzs5JTvFmG7b
zImDJjI*z7~PY3U3@7Ie^TWco|f5EG#WeHO|4f;9NxR!XFhoqvdeAA=<5c%*#
z#oc8yOfGK4c9iJHTXiUVXu{v8C0Z(v8L_MOBXSUD@h@TC6I&o2gMM+
zWO{iF9OIB*W1RfJ?AN?rf7G&Ta~FbzMnchZiSwjNK-*mKAR=^&uH?CoC<*?B&}Bg{
zXieF4?d&T}HyF(ciWhS9#?Rz(jCWWkVn>vgc~^VzxRW);DqzrhpRaGEcpT&
z_SCe>4|TOoM!>DYcwh=3^yan~t-r7M~!ahf#Kw4C(dF^T7|jgRgk`
z16v^~KFaWc>=nUeL1;gp$NF6CQ@-XDmB@?ho(=N~w*ei1Iz}~?(uf0jc9V_y*JOS!
zq`#nwJ?BGKf7TH|DY{1KhL`hlKk(74jzz1jcb0HX%vhFLe>fLDf@ko3U>DOEU(0p{
zl@j-?42okGJr#cAXBp920v-3Y%q=kUk-e2i!m?3m#i-i-`8KpGLPznZfE)>
z@&q1s6S71nJSggJ?me$B27mFOSS}ZVltt52nck9se_>wh_v5(AzW)7VtV@
zMoO%hIm&ExDx@0JFD`%=6M8sF_WV7r_wR!2TRiW=pI^DEwn`k)rV!{^Qlrw2b)ker
zEYC}+rjeg`yPXVCcZfP}=fnfkp6=PfQr-5de-7EokU32+Gp45aLEmr0d-@-+pq``e
zj?fgooSh<>BHKa!9-2uQ)ZWvcqozxZ-Yh#eg$Aa+OxTpC($8!R)IB=_E_!$8M#*W5
zJWT1q#zD_5kA{Mx=B@KoER2tC2-uI6dtonTG<=3Vh+ILi64_8pB2-s<07qHG7edUm
ze-J`1bmsR0R#Ho^vK)tjBaQ-04kkgvLx>-9e2JHm!l{W7?_3uTibzHCGL~|fuSCrm
zwwk5XD|G@X&Ca9KJ4R}*7J{6Vzx2TBQS*6w@y*1Y;9FtP*?wbqx@W@QEU)Cg4aZis
za4tV`7M$h5>u}7*TtUq0R@<0>|A3cFe<;ivvW1CVhqIuL70;t@KCURnzS2AjB08W&
zC7Zx8nJ7tfYa-o?{LP-x*mZ<9_>L2GoXm#pM7M;DkH$H_s6+wlfnn?erV{__)3=EW
zkvwX*RwJc4H_4yu0(nFqf;#LV5w&f*rgc((vtq6Pm~`=FX^9%E3%J-`&fJDme?t;L
zj4;VO(y{I(Ia28_yKnGX{Lx0;&;bouo4)OJO!gWvPp-BEB4}m|qAn}ZPPa#{KpIJM
zo%I{q9glbgTzh^tT{r4_*LD)Ec}-fKr$qrFlk>@4S8~sGsR6l9yB+jEUx~Asc1TK!PM~MHxZS|AEY+7A@LNYS1_0R?w^cEe;H6-RhRTCJL@l0>twn|$o&&6PUt?ejdkB_fSuUHP63RS*1!%k}NgijJ)
z#Scd=J%khYKnCxt&h2k$W{LDPZ%2F|iv{8*x)N5JGEqwhV
zMVR@Dsh-C*y%9p;+fpzrIqmo%EbePvvQz2@_h3G@V=os2e+8&JH+XK-%z~{&|@d)2T-l>kHK+KQzp@r<2cHPw-02aZCbW%zQpgFMCfm(#^ZRW
zYd&+1`@%kK6}&PQby*gLH=*)T(M^y12-PvAZ7M+@TcJj1rNV=>RMBaK&rVp%XBYXH
z99i&yvXn=Ze@!BEdg_qX>^Q!(NC?p=O&PP}c44)3=S9@?KGIzAFzRJJeOH0mx$-*a
zf_IS$U@<7c?sao*vH<-+kpHbGJ#9~uQ_koH_lHeVcIR09bC7S5U*UL9jzcnJ=7y+~
z66;(rB!QP75GJ4Klk}B*)By!Yy*AQ-rZeTt4nve#
z5Vy<+l)iIu4loT@BU(+^&<@Qg;K*-oao
zc6#B;e-gwuhiZ1{puq-ydf}fvdPxsp?K3UJ#w41#(h*mmGAytCz4~nd_pu1!n!$tlH)I#1IQ&cFgNP()PS%`ewE&UfJb{m0^+d9JErg)P-RBp-thDP%nhol^>CF
zivxBsjC}6TQbz)pq1v<8VB>^eL(R?sI=tn+JHNwVoOt!6XQY1tC_&J&PCxh@JRcN!
ze=+vnFDOcBW4z)>2!D#K|2p`^lbdRD_j^aI8u`dfS;j5NxHO_ASx+FF^;Amkr|Z|V
zj37N8if%-UgIIs@wB)kFT@Yrl|Nc%9rBNCOKJ>H)xze`~ciSejnZ?4vG47!+jgw@Q7BjZ`ivZ23~Q
z!qvb9%srvG%6?p@Iz1dh$IzP&3O~UEbOR%2V}f1onyxkRquGkg@dY8S;GmTwY0_c_dGm|9EbY4k>hG*)@mAgG+b%crwhqFx=Ckx)fa
z`OH7TR((B*LAtu(`VbGy6nA=6>xGTEKN$7*FzE>k7tSdn^2|*V?VuOE`v!U!N3=^x
z{&l!;vkT~pdA^K__QW>QDY2_zesDgaGPut_o$na75*N
z7ac<{^~KBH
z<>md@M}vpktS&oUT@g_Xf41+bvH$W7@27x@|8oK|$lA#*
z8Gj~znK2*vB?-F3k5zF)Ex2b!Skc0wOZzac-X{d;LgYd9a0p)&f8tsD3l+6>d7a_L
zN+KY9^R7bSw7}xVtX`BAttSkVJ(3?fq_bI97{THBkV}hQ_rW0m->Kt32s>ubto$A+
z+C_SfiMdGXvY@nvH1s6NBp<)LaaI6!)NV^iE?eVp^s;gGc}S!lMqD2bvpmv!8cS32
zUb}0^{+ZjLZ6W(Ce>b5(en#(r07);5$WYY?tHhBI-HbIxz^RTm#tN!o17ofK#6_zT
znPoMXA3#s#mHqSCu7v0~t^6Z=aO}*S^Qo-?c%(i6WdVDznPGB6%jgi}85?}9s$VIM
z5(h&dX%wA!Z1`MT-f7`hBK6#!!3{FD96+753@MJde1tf8f9>4edJoUt$W$M0b7R)E
z5`LW$F<7^R(7$k0mQz(VJdG?T1d|1UyD2;n=Ab-oo?EkAueNt*$P|VtwyAQ+X|oxw
z>RO+y>>CUlD}6IVgSlpQxvxi6n2PZBT}eAH?uS*Bp3#&+=tw{2$%UiX^rY6s!Swx%
zyF3!hiWb1ne_u{S#+yjC!`cU3qHOi$ZieHUw#8iD*cF#?rT%b3w#d`ong|>GI|%m5
z$KG3Y4fQ7E(Rtph5u-Z-L*kSfytA;l2+f-09lbt>JuL+x{$Nd*i*1hu+RK+Solw=h
zQy++x{&fj06Dm;>BW5f4y*UnP=_+kw-YrY2fG_WTe*yY}rL0rAa0rCtCJ}^Xs`zn*
zU1N+cFqgh#+r}N+wr$(C{SNNf<{jI%ZQHi(+52TTo9wUs)3hgTnlw4>>A~~7&EsP^
z$C52{*|Qc0;<)%s$4rJks{OI4({0K0^qv_Auc0>%V|l95X0jD(pW@++EE?=zy*
z*1q?-epz|t3STmB*{Q*+G~J6=Pvj9aVhiwjm8nR}T;k7!81Wc-9^yBL|ENQ5@vA9|
z16Z`yNoPscwUL!Q%TJJm?-|qX`7{zLt^TPgZ)jOUGw!v?KO-P7foM`@Ks8P=i&Enq
zNlk|!;)^B#hAHmqkd6gOj8{`$?vR-yry9AYDmt-Nv5(s1lY#+@&Z4Pm-CUthJtddN
zFWQWr1pmY~+26tOMLYvhUbdSx)DFLk1iU}{BsXvja8N7O3v5XUWih=`xpb=%8DXlC
z|9H^(j|AUX88NLW3HLNg$rbYy&X{P4i=R8&CAWE|Mao>|=*qR`Ho->@*d`kaR$u-rZIJP7{x9g&jJhEO}$d1>C6X
z2o%c3Si}`@hKEN|rI^6PQ}zENZ;_-$hd^T9UPM=Xsot^HA44!+Vp-6O)B`7={+idm
z*Z3Tor|>9GJn2SA-gq*#1=>)SMGnkc{Zk^J)Vg>_;09u^n$Y!2&B1fS=N-H@w}O7B
zhZ*+VP!Or**tD-t!uFcVRobe{9bjlY&tmr?%)3zQAW^DEO6tu4P+Y{saJA?mL&yFq-_H0nOFn*Axyf%
za$B3?gUQMeJnc+WpKO(#5!%6Kz{$TJFh%=`=RHgKTtPU=kG1VbWd`~44w&&?Q9M0U
z)gI}w3kAV
zi0&tkfu(^i0eM{4LAqSyfXJw`y9iBV~c`^Uq;vUe}Js*F9x^i
z&cdf)(38a0cb_RZ&0FgLW3hw^h6@v>t;E%X$HpWA-ZAu0%lS>A0H6RPnzG4DnaF(d
zoY7WPUed=O)}>-F^74gl-(yQCUx2dj@<=5M?I?TMSBvWmUQ@U91Vs{`KGKx?%bG16
zfxLaNG+~ZFZU(|)WpWEMvbh>^CmuBOq8st~W=bN6F9|J>H|c44L)X}BMl>Orb+4#d
zgpuG;a`l#&wo%u_4cOV2ORyvB*eVEym`XQNujN%VMax?#0dGKYS(%O;-
ziWTHT&XidVSdPLG-VdcWucTzIQWI((*b&gvNcT+KylWnNV1|@ywm}l89bcVnCM&4IN$-odlLK{bC&)gH;jmO348|OlS{+L3phus3ThLi>cR`c6zY79B|kfSX~WUU45r3&W}!A|
zMTgc>d6>`F5>mV6MGad2Hi0aaVkg4dbwhC~V^0;J<#hh-l57XF-&hB=>2^HWWpHQd
zo!8GB4qa~?K`ukHWoh@wNA4`3Iw!3hfY$_G1HjL9vi9?D^zCz{fj>Hlm1|afP5Lu1
zwm;IM1$yod@$;(FiJV=U$#mSrT%ph66&WqEeR3K@;AHW+5vMMNfqr4+Hw3uMwQ4;o
z?W5S~v0S?LEiPH7z@Eu|fH{XvaDlhnc-1j!ftKaFCo%YDI)z$kpDqA_~@S~OwLGOpAAz(K%>tTP`Q<{FM3%eZ*hGO)|
z>2y?Hee;s
zhQ~lzwX@gZ2T+DUnM3BF_UE-1Bd5Wgs}x4o#8~t)>(15^J0nVUfKX(*>7IJga#L&`
zi#rOZV`>;acKI}Hx}N8NHT7%B*EcL(ufZ;fXO|dJ>jxV@;#k*u*{DnaKIFL*0nzko
z*V?AD3NA9yRG*XGfUMrb(@xlBh+uZBYzizi@E{E_o>
zoU|{b^Vg@zOy?rTrc7YUj>WBpOmSVsv|g`8?Z0NS!>~uXbarChQSUc2_-Qx7I~5jy
zp=5whR-Tl44)kPI?vfl@)GqWa0f(WT2wakLvIv4NzM)_Viwp_81<9-^&83on~C9+2v7!kSO}e&hFYlLX(r=WvLFL`8&iCFYRf>_D}R`k=U<409Hi6YaLg3Y+WJbUi&?D>yet84ku8zw4maL;uVu_b`hox`ZX2Rdm@+9c|
zJ@=wdZt)nl28_`nF_8
zyrU9QWmmF@dCOlOsY55osp})YeMidH(WRosa=p|95)?qD9TvpNFCou2#&*1uv64w0
zmqu>SCVGfx9G3xnZgjm8$?^gw!BEnXi-73%YNfOMoWHia(tzR=pKW#hlM_jxBeGHV
zf&~att#dd6MdY5EscB)sKm9mvK1X!(aar`|c?}{i$yLz^ZTMzBdIp_Lr#5ij?zwX1
z;vlOxpLbY>GBc+U@YhlyPRtIod)FY!Yw{)j?!c4UGmj|;(1fV>9w*dzk!P{jfdfzN
zMw7vQ1^V8j#Q+kyliaYp`|VJ$aZ(}!al-dtxFK5)NI+_0lJ)cait1B_ceH;?Iu_Z*X1d?4JOn~Z1Dxja1*EKbsPhj=Ix(hWy
zqEm1PU_Mui^=KX2<aAlY?%SiuxjAqMI
zT)yF)yVucvmSE#+b$H^&A->6@2-B6{zJsrzYm5gh8<-=)N}xLS$eZf|_!ePxq1aug
zX5ESl0|-rP$LWPf$7RIpF%r^M>aRLZOJ)!CDeMV^&`1;m2fGI!nCbT$X
z&qW>I)Ehqtgfro76l{{IEhhwR>k1v1R(O^N9ia`;;XfL+to){ihdwNzV>s@XG4@$-8OMSNqRK3x>Fn`-4P6Ht^zOpdA7YRc87;Yc&R$5tV0B6*N_h
zE4T`ORErH)lEu;gC@Hm*ro`L^z;vl(Ew<9VF3mc(6A(v2^xqOL7*lb6bc$j`sulfQ
zm)|q)NJ9?EMO#GQ!9ws$QS@`SFVvmL0kj{%TkK2?oS?zsCla89L)
ziGt|TZApnaB6)1`!#U{GnWJFHy!R8so*t3AX7bm!7Dwx?4(YKx;Hi%$D>559Vd^4F
zul;pMMu_v<(rRYc*d*d`lLVai>AY)Lg$vu;V!L6r$34kiWefw2k7xTV41PkVXl6ajLk!ncE#z4km!TVba)&K{4ZEbCcsiunj^J3=P7xgUWp
zg6!Mj(@z;NPq}pPC#h2KU)92{8Mo6J-`O7=*;
zn6OUS^go_Kltcw_In)6g7yOn
zcZ%o}{CWgqKVP7Wk@Q#;zve?Wz&&joFm$Swqgyk^
zgT|n}2KN@!%R#}Yl&RM9L2ZIGqbzhbc!)s6DUnr56?1(Mse2nZEEI3xPOX@@jE!f&
z7e%s;$ZZG)Z#phw05HRg+~i}SX~yT>XiRvZy@_o^nN+Y-5ViO6Nxfx#T%Iouh#_;o
zKqF`_71N!_r`{!h)==s-l|FJ3w2^$j7n+B&F>+l%aAF2~Ci`srofmWQB$yz$J#_@1
zWAxMBpok_9+{xf?t(YecCwD7=P#p6f#N8wmfH?-?*L&r`0g83KJ;kCKSIr}37UDN%
zZCGYN@k3|_|8nKgz+hem!eSG4(Z+S-8;swwMQ2!&*yL^{VyhHJpSHV?4qn@Tz^(}TpD4z|2v@sm)BC1DAt^tu^Ruguuz*5{c#3RJD%s8_S**+DrzV5t
zoO@n{OYLj=HZ76+`u(k>G3ITji%X~@AjLkjaH@PkWsv!bW&A2D!FL_YV!cYYD<7f9
z*glbKcHaP)k~m8Ll(I)6UWTTieZp6mB6Vy#K?=@1zG^VZRmJq>q_1>M7R%~hv^AXC^i@|v7
zJ8;htfVUG|w#p6K#a#E(-fFsSM&;7$HQ6#LYhvm!0fgL^uBwErTtAJjZVz8j$*e-@^pxUOp9{U?iLKM
zIZhNm{B*+KtTs9-w$~~^M`8JzU5M1>-!;K9033;0HZUq)Cm%3`YkM+TADTlyw*V&5
zW%^UIO5MxY=V4&K`fJ@90cNCI&S9chS&G2-WD!8%RcUh93xmid_^>rc-<24P8$3Q!YeN|fY2K_#8v
zf(aX4+iBV+-asAEO6~d+LOg(t2>vjTUKZ-J{92rSYTfWk)}h|41FFHO_hG6$_F!1u~Jd0c9I2C)jG{0z}DW*%%~>?<;0{gwt7L20N7
zo&~R}Q6OUvBFT2ea{jh-$VO|#39Y}5psWIN50Pud9$%xUlBLFf1C^kMnCaeJ0J_Hi
zNMhGenJecR0xa(S;K8CPrZrH0i1CJ-qCa&4y%3m%M6k>VclRy#!$N6mp_uG8Rzi+;
zKv-#22Q(oPia-hqttW-#bYWc7Mt!QR@N`4DJin0jRN(2}
zc3Ti79>Y-UUq!yEAS<*b9AtBjo@layy3jKAHviocY!7qt@()QN+7f(6u8`n&itCfG
z3}^a!Q)La0U8M?LQUxl9HxT-j=Y6K>uxoTv*>R?B9$%t7pRNxK4Kwzd49Jq|di?ew
z(a9ZK8ZmnqtfG`vx=Vi>fs(#GUqz#Sk+6t!D-}uT$&3a9R!tKBrRMsNV(Zq`{q-TQ
zc{dUQ31f3*ZT|mS%#dooH{dhsR5JZhSGYLz4XAGR*M~|E1mv14+Jwy=a)0SxT9E*|
zI2Ys=OnLjv#OysY{#jalJePYK*+NtHI5^GUVpBEqI~88lKTps(+={y?>sgxAnTX=6m*$7CHJ#9coZ)egVc(5TxG006XZs
z!E(<7#YsJbLys@WX%_6p&V~|q{;^BeXbC(O(6PK@&mvcKM`w)MQlAOKpI-+n@l>Hr
z#ru&p`xtV)Ip>&kf6DZ<-{M)T)^d}u^*E^;(*EKp?lX~)AqkM&wlV#SB+YpZytAB5
ztr6C~><`4E#Ox-}cL5ewI=Ya+b7Tsn*Bi%84D6qTwS1E9EZ$+YOpCd^y_{6AkYgH|
zFXJKjBOliH)=*s}q8Mp!cFHc>SAj6R;osG%dgPAp|Na^veKYa((KOdPf0buewJy6s
z8zZqSF*7Ac4mE6!wS}`oYWOA#bz<=Cv)IM=lxTc|b5E@aVFw5@?4FDi0kF-RNesqy@Z;zv5yo9-$CNXI
zjmn{58Lo6t;Qn}cx|;9hG`P`ri_g}!AwcCk}k
zjS(mHA=nG*asr&kUuMA-NPsiP%^Q|zM>_vtj&SJ;mJyw(7$3xWRG|Ni5o-vp%{N5%zI?d&y8zc*N~9YGwPRd=$2
zV46Q12&6dANOw}~?$s?PblU46Afh%kF7(RG>6MCO}J#hz(~a8PEBH^&Dg%ujFaw<4PQ
zZr$t*MwXx#I=wz{|D5goifoX9W~(WGj$%(YJF0vWkOQ5XvNLccKPe6EDGQFP;unjN
zRxF>6@dJp9PF>C!rIX-8qODQt$_gJ+$cBkVQJ3%U$3yUQa38{~rwF_LH0i{R1~?*F
za)6h}hSHWCx*$Y$#EldPalK`rx+VYNUz>@xzTnP?-C$;JPHOgDZC`PN
zM6o>@#yze}=8A)Z6d&TJpcxGpvrRhKyKAuWiUx$9$)3ppV$(j}JWUFom445Ic?A{?
z@?)vzONYxzZmaA8N%};3ci36#s060fG0S&GJb{o1vI(68q2fs;J|Tyt(b0o?hiPk(
zd9-y75TU29R)fe@67K1VhB8(X^9OjA40hZ>eYbmo^x$%W?zkD8ZEfm0U4$Vsx9Z@|
zoB^zI{2}Dx9F>Rpf^TEVpwP%EH(PliUP{MB%v})DFo;JW60TuGAXT-!d2KI%H^=oAySrpU=8P55T^C!%Yao~Eas%VH|a7WErtsr*t}dv%OnEP9YZ(9c8xZcK|9G5vL{+c=Fc{#+H6SzqLPPwS)h0w6e1%zjR30
zlp$N8$o!ci2dO6FW|`PitilP|z#|>_gDa7Zi;Y*4g~^^QVU!DA*c6ITJfa~<33v*$
zVlr{3Zoi$spWyR3`LvClxPXV83Mh!WMJp+EbcY+(l
zK7mJosJ4f?d)OTH^Msi)TEX&drSF-!lKzbGFzQ7%{+_9=m+&CFx_Fuanr>~h0*zQ}
zi`bF}trDf|Mt?iG6DL?*U#ytBHnQ8q5DRw6KhvyDn2Vzfkcphe+71R&02#-6
zlXDxzAJ9{kQlY?;?ybINTLz>R6C7bLR5?@cQ}A+foms{!!^hdq=e@INEUYwD>ey6l
zrehM725oKze-L&DAhqhYpycxtTfv`vhfMeWy)n%4p2*r)l`&Zc>q@{0KSPx95D9I}
z$%?n35FonI7zVlKYT$f?z4P;}pcdP!bKoqIZ`C0Z`u2>B-ucbVfewIupFy*wzyMNA
z7RY0>&K(wE(!5iRktkeCJC0{#h<`1=o_oacs9^yh64|cM`ikDs!gZu<(nIz(Da#mN
zC`qPrI%9~3{9fYjAP#WajVlGw0xd3nztdbR!V%|h&#Qr=-z;KP6h~q|r4LGz$j)vE
zA+NUNd=2Q~dlJMINddsM)zhuI-5wc7GEehvYX8ydBwX>x{3wkeyimtKUm3I?deg6Z
z#?yc(m!@_&5!RzcA#QzsK``$-*|nA7SXG@=u3-Fvnz|*82Ys^GE|>%9=SIED>O6X5
z(k#Z#*!+MWt!{BR%vW)t_E4@9L>MTo#s~?o*6CjGHuAX}kpz6lRam8TmMJVx@LMZAi0}bwbQXcNrmtrCE@F;-
z($vZw#8;{Pi^3=IS&k^{ME)P1ij~^0&lOvqrMJ=qn#6L#_{Icybb5zVE61OERBbD{
zdU1p$Mv0D5PSrJB_{PCv^T_cR!6^&b~#7d&#JY;VH1GVYJ{#iHKX!
z2%L%m0yj7Qc&erKG%(Qwzb<%OVCAA_feH&#SHoLjU3lDN
z@&fy{t7nII;k##%93S!+rq}VX#`{Imh=OP;16d7b`q%yObU8YO*f@S0s@2XaLS8~+
z;O?UP?7`KFg($otI;zq50B8)4G|&Y_)8GsGerxX$Qx$Sx?`jUhPlA@miD*i
zO6wDMF=(drV&$TGi5Vi8{aX2qp)v%w*@q@I3}Nb3{?|IyHVC<>CiSA6Nx=4!B?Ytj
z#|Qyg-2ll^qOw+3Dy(9wsX`CgR&6Ve`W*z=6iFVIqK;}-Nbdpz!tpSHg#;)eqH%{S
zkPCksy2deA%jdTsUmJyz?Uu%l0+I+%F^vvM=fjoJMiq@A*6=N;p`zsp6KQ83wvQV`OzY&g;5C}j>
zhT%gA&kgSWvTck5U<#&4NzY3l28qQ7IOK+6m!8r=FYXd!2b%}b72%zi4B}HchONR-
zh$*lYDJqP={S}XO8_c{}Q6;o&kF(2GVorSxmqR#h-)U
zzdN&Gzz;XRdp_m-{&L~W)cvVJEXZ6iT*}dTDp{}stmDYj4VuVZr9Rvt)*{_}6hzYJ
zK79YnvocVTt)Q==>yhud^kux4F6*m1+kRSm1IayskQDL-9cZ9?#I*=aXf7d?&?N8{
zYwrP*G{G?ZjngjQpkV|}>pUh)#2&Bpm3K4tL`wKv;LZqpfet&6LEtL8Y0jeBq`<6-J<*;Kg>bn4#OBH(NreSR&laX&h?u^%97_BQBX#sQ{Ydx)d7
z1GO9cfwba`Q>
zd!oeN0U`3eUw`Vw^K;r=U_7eS#)MIPB9xv7)Rn+oZy4BL+0RRr3G>=+jN#D!LQQ4~hE$M!U_@W2CUhOx`zc6fgd62N!4Qt8^*q@*6}H<|@JNVb
z&${VaepyB=U1jP1Y!iW!cW%3EchXQBWJWd-LdGKh}sQp=&h4tX3`qP5Bg
zNJ8>=AVEi(*C>VC#kTq-qTqy+ZjkvbgblY}#u9b-xIJ%%+xUyX&YF2nAWyNS()Vqv
z5&r4TZc)Ou7KtGoy^cxocMTJ+lb_i~{_y}dJ`ZR80B~J5L&!2kJf4`h#)imYkt2l>
z9I`)aC&<5x2Ka;9AQc2NF_#vVtQNTqK#30`p-jAtw04yw`lcb7pwd`h&_Uvf<`17$
z69DoH4pq@q!42m~k!4e>b^Dk;=i&k_1AkGU!nm}r&pjVs5`+Wd$Px85`dlypfzFan
z0+s`c2h>Lp8=Q3Do8%q22^<+mBh-=!lSFFXJTNh%)lwo9?F__;4Lgy;i1NY(;DTXw
z<&T}#0;f)BE9}{i8y7A##Fn83nkgt(kR>Z16a69?qtJ1{ONoNJD35dnMJ=G7UzP6v
zxPfdL;)HtyeIMi~feRubl|RWdmO#j>$qhy|80w&$J&Y)d&_7dYnP2+
zC1aTOjCLY=L*iYG*7IdiOVqs$Ftt3Xo#Orgal`~o=v(`<%a(IAdP3WrC$fpIJ!X~4
z7npRYTpAom@RKDRQ)ve>=9|dYSJHIBYawn2a;GHKE@4AHTufA&b!xn%Dv3gDjZE8D
zL1#_cjumN`Lbvv)Gk6xo042C5i}JQRma$Om;6QWgrBqsk{Rqnm54
zE7TZy6PF}_M3$hz6f!x72&Hqm=KB>mIEW)X#x|v)a3Lw#6ou>B&@J2LVtc|D9?kj`
zpflP;34Z{KFZ;m3mCC0pe!pU=2N(GgE#4kogx+v|iSG>M)%#EXwVrl*ngBmXiwI6r
zHDjkfZsx9;nP2D&Y#K=!8{#J)u1=eD8u(52V5+Z^{)DNA&>8_pM2ST8h-smk&tHw1
zv1!3v7o&j0r(xM(n8z+CMBOb=?QcHB_*1~$cU%@nE3V7KWn9-Bm}JR>X`PTrbe@$P?~fHpPp
z%LG(l%!GB!GKons+`Xy^&JN`nHs%FL<8!HZ>*`lK=HZp8(!gZGrUI|zI-^z_ULNSc
z{uqlz(7XkkT;&u@pWK4{qE`R-sE?HM%#OxR_!dFVa;pr$a*Bqh!;ZK#qpG6r73F@y
zH)W-d&LZW%54n=Es0kbr!Z>i0TZSCa>7~EH{CK0#x$xm|qIac?M{1+};Xnk1Hb~5P
zTNYN?%>3^(^orKQJsGYG>-iO2GRDjiT_{^4B>^Q&vQTqOkRysw6)~$P9}Joro}$3V
z)J)GS?eQc4Vv|x?a=1%ktz}K7$;A)pjjl^2;GwgHh)}liXovGg^Z~&_%0mkLLn$ZS
zB{gMbZ&_SQw_*9V4?XrZ!G{{J5t9dapyVLtnp4VAc&ra%uQAnSk;sw_f1HAgP4U6i
zL$AAlG!*1v{v1}EcCpq)FAk1Vi^p8}ynz9#c!&g`vFZ?2h<`kwm2qijL>~hLOr{%(
zT?L+Slc_YPiTfv1g5329kIesLao3Re|a;b>LjD3WZMQ9=^RzhD)DX%OzWDuanwQDIVV+}%7;#t0_8w7rFI=(3c#
zST-=g)(?5y_TzNzmN?;637LR|3_UH>;}B|eEOF{RiOD}hIXFabz}Pg9Xf4mXNQnTF
z?#v(&H`KHuuOA)()X;MZ!U~fotA(bXE9x$sk(ay3|IeB*7fr#k(^<$)ydvsNP+p?5
z?oNkv=JKT0=2!w|+t7uC%X5}@Yn#L
zUT1zEf$`O(wxqB>
zeFIftei0Op$Miz}qUQHu5}PR~B^U)55-;3J;zsNs;@Z*NX-8}xEraPB+v3>Os->-N
zGNV$U=FHELuc(PBqsx|;TG8EFEEYOOt61p4O}s>#S`T3r2jqKC5?w2QaDH_E;=9?d
zDK1Iud$Qr#JK3TCPP+6z7S(*LP*_jvVDPjN6`?MXuq5p+BA20GMV)-S*+TK!7NT6~`EW7BkrOP}A)i
zfNcBCS7;fx-ba=C1;xkvBkAMO+d%+p5OWA|b5+AnLw@pa`_j53`H}_i3pKvs=
zb$5H|RxaZpfbMsM77d9wkEIKc7#(&=hEp}8RE-^~Ud50f1oB-tGrtqUjY!>BzmvlK
zXS#_!4ux}|l^pDOfL<-B>Su*ZWko&!ef~gq<+UQF?%#hrnX)2-+yVC88Bx`ZNQU>d
zbPifLLzdp6yio=Pbkr03US&RszFW(RAEQgd(<0K-bmhvWCckkbaqa-BpOoFvW{$yESU+W@j#dse2_@g`6DqdcI-tkjIW;j{cbMSrULM7=NsjzX8M
z)`fGz<;BDyzKmW^)?R>k^r)fl(QEZ*8@89Lfvw_ztN`5dByY@sdR9m53G_I@3|zkO{hf|#Baz8hOcMq
z%k$=J0X^qRiY+yKY<NtJu$C0iN^yM@2J40?^;7dl?h;)A)eW8my+<01%TzZixW
z5>)|gK8EJMyK!CW8*=ph1o5Qh#M+a!h(A_*pMk3V0RVy&!Nj;i++6>h_%Hq`^{I`3#xCDX4Z7~Z!h$1UI_RmEnTcF3Ma7v?+38yL0!zz
zf;D7y!ho#J@ys5+O@NmUO3Uz4dTu;Nus$?7PzE>3N$Lg{7+*anQzeiF^4gTG8nfk5
zuRg|-mK_=x{La_?#Kjh;dOvoxMry0kK5zJD2O7Qu0?+mgWoX(-yK72hmK7)Aw4-y+
zLLo%MAv}6plWOnl)$Uq_y12955bGwBGhY8DI-nKpda_ER`f%9~9V>bn!
z^P%y0VN8$y->A@qreJv*e5I|Rp0)?R10WMM{uG(?qjAUioUlY3$TGx@&?Ph)(rWPE
zVSo;G_JOrsuUFluau_OEj)eN7-IS$5F6%lm+ankLOFW;=i&`_?6evT|eS4#1RgZdC
zf-KEFuaY!H89-?NrBc=tJzef5;c{d6_IL^3_tjKGEd9Dj&0S@hme@O*Q!fO|9bcH9
z9dg<)_=<(Qr9i({;Z+UKA&vN;;gWG;Buu>nq1%jYwU3t32Nw)3#$T$=IX|j
zMGaAfr+96u+K5woD5@*rDgJhI>;0M;oVrFG$6}RLry)aC=QX~g^DL7lgs%BEHKg(9
zHx;D*eyC+LPzg@{(=sz(3BO}=Zorud#3M{U{5kmFHB_yAK?TDBomLtLt_wrX660JF
zYKH)t%6{r3^j2w9RoY@=k2`!O(mO{G#=WwaSeOW73vLZIB31<2xXZ>)xK2^w9zEffVhrP2#~VRPz_&%
z{p&Mi^%p;V`|bP6G8kN7&*qgp^*iO!zHwp7LgrUc;XrC7R~B#jK|yz-8RCs
zS*o&E!$=j9$Hc;an9oOQjnfc9jnmzUdGpe>4*^@xr*%=(|?X@Vk&>
zY=5Pd!0aJiAS`A62Bd%V-h?iYtvvgh4!73&YrGLm%W0LGa-Dqn<~Dy4sj-7-L}9jd
zt?O(Y`WiHU&M9LsH(@u(WbNqQ4-;iRZnk(ueTl_H>8-K0YDP%wm?O8G2>z0x*dczf4QGNxzo=b=6)q
z)t30XO_Z`d%Le$p^w}hFX3JHvJgk$Bp$x?@M}BM%S$?uVh9~gV$j|5N!E+&Rh5+ps
zq4D|Xqo(_G7!9vW61%A;|@+I
z-uDa_)gf;E5bzsYT!_yIf0mn}0qMhE&pADs1NTmM@m$xslr-!O;_-aKOt4q_1m~C9
zy4i|<;#dvP?~?}W-cY=2Vy>00tvYUSWSBf|
z$T*_y2%P+s0ovV_O;GS5FjRl(IX8#^)j4;F0DVUwMu35q4k3`Jaywl+t%%kvwF8+L
zU2fneO-F0)r>Q&alfK)Ta^WIOZRzSby*M|Ig)qIicAjV{80Z*PJFNrLvY{a`Uz7p`L?)`}+nRWUa>$4qn^K1h^qDs1nH>k>%qlOtt4tqw~053m*C
z2|To^T_VXnA}|YkDlZLqFf0cnGZQldCo?BC15@fW4LHXCmE6&Qhl4P&FsAy_g5zMY
zLDP#_Iyt)#vatTIQ=5sABejkeoDvYNI04Rk
zPHf*;&*d`fNYfk@`y{#XV%M}8i?@w&NZGg<+tB#4@ajZE%9$5uV
z=tt!OEc`6+vwfo%@0W?zSDD*0sR9`?-rB~b)biS?vo#I_10OYW)CU%c`jH;z
z{07oZ(~)RY^SWNTwuJ9m4SUegF#e$juea#V+U{ZYA9f~S7LHUi25>?E6APyJ`~QqY2(@`QMA(GbSlESF
zMVJ{_7#W$F**TdwL`B&d1V!0dg;~V-2>;(H@O=L^`k!zD8#CK~pFHWJtUQV$>hPVn
z*|A)e7ByivaUirsfRcqLC1Dsel|oWPsYcakf9oy?rfd|9F80hn$Sk$tE;|?V%~!A}
z?q-r`I|-KcAH#=V8+cwbqts
zf$!Mx{gfPVc(YKt4&c*5;O%(`T0^FT|JZ)I;)4A~
zg@M2gA;pNu>3-Jia6*tUcN>mNBUBnxaO((A*xx46V+T0X3;<(r8iAe^jEqnxZ90V+
zfiMp7b=)e;_}+7kt?3cG>0+K4C!D(W^z`B0ou&r9yb@X04V4~sz0kV
z9ru^xiEsdn6w&YRTFr?7TVt^6K3|VtXi!Q7Ho9ob2C9K6>TJ((^1E!L;aUjRdnhPU
z8mw_19b0s?UH}u>-%!P7wlqMDwYoaXhQr+oHmotW^ZfmpB-Gw}<
zOV%a8RsG^raLQ%Tg9@q+e9FRI%p$?&YHbngu>Q=nI?Np=C_5&N->#&$)N%g-a!8Nm
Yg-)Ge0mp@9V`gFFgeD~wl^28lFM}znegFUf

delta 335236
zcmZsiLv$wK)}>?Hso1F4))(8T*s9q4V%xTD+pO5OZTG$1qhA02j?UvFo92qz(3rlK$4mdTiT2m%*qYbs^L*rI*DW+{Gu=enftG>JD={m(tnY+pxUX5|zrbbd;efrelf1W!n?$5+U*DR%yJF!K=@1#I_uIPj__RU`jn%F)ps
znKYePz^0UA8op%e+!^^D_VZ`v$;>)}&Ua&F>tIoyukwQcJ@*1=W~}OM;WY{`*y(w>
z;rmlMEodhwCQap0QBci*rZS#Rw8{o}g0A
zNQG<^%loo2;dbh2+WtD+*!4iPl#dmTuAq>%+%Y>**f9~OxUjmK$;@#a#L_EEGvoE3
zmx=qYV7u=C8qyH(n{Ohqf~lC3I;mA%f^vM&mE!MG#I&Ds+YamJTdqmw$t0w#0LP>b-{&^)7
zY46Fpbvv>1@$A2B(C_+8#OU_P<1!ctc1){eN@z0)!PE&{Jp3UgVq~JNj@gla;CdW%
zD3rlc3bmvBF3#BiH8?)nnJJxKwq8W|y559gxjWi2UO;q4u={BLh8kfZ9PuO^8I?;P
z=A3$Mw>ixLbk=%tJJ*=M6bwsdrRKfhM^tDED}hS~lT7xp1sg7S->=}J|CUL>!MjjZ
z10xP_)XJ3cOVBGL{2Ug4WwWQ;=_SPbVZ>|NvVM?
zN#H+L#{LP!;mtu}KY^J#9^}e;3EkrheG?zvJ2)I!hQ0ap71fC)HP#lN
zr{Q^u>eWg#;wpe2{KnT#pkgya1`iRSoOU6(dJ)K-J!h%F&AuSxsIt<4t(AbKg`(GY
z+=!pGBb6fasrOoeuwj#qF{TgYfe^U!<&ZbM!%s{U=3EHv@C!(FBBYg*jdQ-R=zC!j
z&It?Lay1c9fsFkby~9u!fhtK%tLLAPBKuq(Bkk-AxFTPkZz~xj;?N>op--_CX2zoid2aS3P4FT^fsK!oMZO6z3%(;Pr}HT
zPhzVJDcWH{y?
z7li66^9;Mje{a0
zs7b9BkN2830qP$urLyoF!~jig$RHppJXw^hT-dF={JtoUb1Ze2&6Zr(`w;H{BK3IA
zb*QT+dy`8snhI3mjjJs*`r(=i--arr+PtbNXg{sr>y?9m;wM_SG$Ld%lpeps)b@%O
z)e}zdkobWB++ixRU4bCq!;PHR&Ohzgp6=`k?70=*)JIb*w7yX;uPky(J_z%gVRN7~kPf
zh~2DeAzOto#2LBR8%Kf;!HklZ0q|3~UM1d_I88uMyLpW=_SGr19b;hGh9!^4#lNco
zuvj-8MQ26L^|`0c^N!v*@x5U3lH3}js|xJkR9VYGV``6MxigQZiUdCN!cE6~5}~Rt
zgNfp3nld~eTD!`9&$!?PkEBa1c_WMDJ715f3{pO`@}Lv7&zy!Ok0KP}s+}&l?k35I
z*{p!f&m0xq)3%M>@>C$r3cf%Z=Tlb`L7Ft1e+Dw4*bwcs9x8H|5P=%+^&$Bgy5o-R
z9Md%5Mw{xQGJ1?e-BXElPEwbXqhcaqAmWRAcj0w|Iz$?mzU*ef-nlCv#z=5^^kAUj
z@n|t*BLsUc@Xi?73v!yq4f&M0^VV?Rg&|V!b?MyMfZ&zXQx!;DNZ=HbuL?GGRAAeT?CW|iLun<5;^Ngv*%6QsC)F{d8bP&%@
zS5QZ#j*}*V90F-LRZecsnhC)BYB0wuis%CM8S=nmx7&HfBGcWFM{2HL%$Zr`Ha>{S
z%_1wY4E)_+29af}vO-Vb*B~nrknOOFq=pMyQJc*okyTyjJLnUnvGWVFsJVzGMuSb-
zN-c4v3%oIS4UarjXSa7@XWdS6cJ4N%P}GsHlz9R0qQD?Nq$(gj#0Wi1WqhUnIe3$CZy9gwN=*)jvn*
z&KNze`~^I{W)0jdf`UV>`*|#1bY$JpQqKzeT*dC?EpqGElXNTG;*w&@c9_r6c0eD9Qh!04w-|C6El
zQ;O96B$3nOHF~gtc=Y7)w>KMhYC}Kdd&{1$gMF^mJ_l032)W>A&YkY>+YbX)g*9Hs
z!vG(!)8|3ok@7~h*Z)JPXiX`*-{Xiqz54mu
ze{d={b!8YwW#c&A10itT@l66exRZ(#mh2ft7@w(o1x(_XBI)qjC7U(kKVRc7-5%K%xy2oBw3d34wK5*H)Sy*f+7`J}S9L!{DV!SmQtiz-y
z>V!7z#QNyBL<6g$XnE`SLKsruAVJbkT@obrdMmn+wkM8(>EaBlHWKHv59BbcipG2k
zz~~0-xqh|1yknv0hbpXAp87()$QyU-E}fey$e&xYlnErPrM%a_#f9cS2smfZ8;I9J
z*{zOTO7Pdd*MD^bnZOv^erWe@MKMMsAtbxj#$`=BT0Hpr5PMrj4J*+U5kFUvax-yz
z5PwBGL62Sm8~lI&o~_hq5cGuPZ&Iv}gA$j^2^Vb)|8AaUHzY^I@mIIe2NO*K?L(|N
zn}lVrGq`LQ4wEd@XBaHEDU*wnE@Tz5!pa(<-QbITPIqerTPVYKSGby6PJR{&L83(X{+Q
zvjCDwT?3&+`_r*O=;YY0c5^_4wHD`ZWE9X(UH0y+vKSdNiA4&qS<>28(=h1rS0S)B
z|1mv&BjjD`r6C++4{mOAES`>p7?P*54Yo?si-Ee;*m|^q#vCmaF_X$qdoJXN@l%M)8OV`T$3jn-0BtNZRE(qlG+85R;|W#b2OZ{5x%S7fxCpsfnyX
zA7^*x?@tvV9R>vIN0%~UkEukyZH3F=F)?4jzOCAC{;k=!(moIfNUJ1?_gq*Mo|AtB
zx}gm?nEW|ur=Uy{M%=r)+tZ`|4PL&0qvyd@ZatK)9+-$?`3;D>2mq_qvM7Aol|?kr(uS
z6pWtXkSN|b1cJosCcQ!
z*;vqh@dHqXyKznzhwLJYv%y`KYq;rvZnRVG7JVVNVQH?IF8>IgL;6+j$t7G(
zrL!>yBvg|QEG`;ot5?i>3&>3Jc@HF69Q_XGi`Uxb
z{Ef)ty^wKPqXM68Hh@v}VcaS2)fdy4!`@9>sxfzIZD|JyW*Wreo~S_e@8}H+_=es0
zKwDLxGhKr3+9*RDX3;k!PYGsQpcbxK`==vuTr49!dYOhCH{-BB)2%|Y0*q(-Y&2v4
zH{#QW<68BVea$-P0OxaK7?l=!(|0kxYjPt&?BYzMJg|J6V3`(KLG=%{J+UWg-b@2W
z|B=?0?~c5cl1?Rv`yW&}*%2i-;Nrz*dxMwTAaO93*5}VEAe5c3_69itO{fqXx*U(H
z-${jjkd!&N{BZrsxJh5cbN3Y>d`Q&$Esi+$(2-xpUDJ+Jjde8Qb7uIgYgs|b
zm7zTTRe&));F~xKY!55)@r(zZ%RT;^jgAGjQ3r`0e
z%FC@qU`LkJ&b0^yC@mw>G2(u$BPspaedYX_B$sYo&lH1!Kd#VnLqjm#wmN`75a2u-
zu8JkRPtW}Hl5GzP%o~3NM=a^16L+$o?kNi5f{zVp+CnN7n@H9#)_F~i7=|wjhkQd^
zpI2NM2_D113-JJzegNx!5~9D^J(%KPdUU1rxpymIkn+m}DhANSz)UgCwofRn39g|j
zIGfiL-j*juF#!zz?U!4097Rd_*Bwr@*0VdU9sYgMS8Es|E292W+y6traZH9{OU-B>0t)*duAKbVfKIs7B@&S68Ra}~4_TUCi`jR(bz3Nn
z!x73H;DPdMf6kv3(jLgNs_yGW!Z9eHm+g`Q;I2FKKjR>rQvBgx}*<%J@Ip{Cma-sF$>8O>N&eJ
z?7WRhsmE5DJUxUSUcM;25D9FZI;#z<)!Z+AnI&hiFeo=qG?J?GXm8bO^h)d`G_}2-
zeAPBS-7<1i5_WEOec-l810&beSLd0ewLcg2%2|H3ZauaYFE2GOQ&!Z@kF{)lOJmL%
z-i53EqLLEZD6LezXd9bUt7+6MJv#S=)_wN$RMY26qUpr`XP8NZlzpgBc~0OpieI#)RjpU<~|
z|E?yJcz(V9sM&Jx`W&OIXR}W}ihqjsvA}1H15dM9;6)7|;mKHUz7V5yy>jJ6*;BF^
zEErj-8D8kbN&a#!6q;kJX^_otO;~EC-EW{<(+A>hnhhk2xi-)>y|nZMpbM@^aNcsj
z=MFMh;$AfD>-su$15e+hN*#q04NcB=dlI|MDfY_th&0LE81wyYnGO17o>D3A-$f7G$<~a`w1-x`9GkZ
z6e7ns0HZ8XEccjAUt@*r-a2ZixxK}HRiwChrUXHn2R~PuKzHoMTPHHDpuDf`3Nr;JwM|L+767YtA01>@|d?+lZS7X@MbjZ`h=mMlCN}9*hxh?LCbz@`$FG(RXNCR+j-s7@z3M@H9HJA5-Z&CMiB=iV!7J9oZI524A#m3-sO
z?X@Ri*H+%!%wz?=4~NNnUMz&&9<#5M?-CXV(;;NM6KZJm1~4iCXKI1nR3n(ULG%nc
z79K>2rT{_-Uo`&sasq@)ir)zqGFaGDML3bOryf%YKw#wP~
zgb13#1<{qD8CM;N1gjBTR~BNId6*78UTd7Xj03!Dfql_v`ZNCw=96i5brf07V6uGV
z0{s&#fqk+3a7cw0f-bkH9zTu#R%aE1}S88ph7E!=2fIRdnC$o_j!W9
zs`%VI)rY0xAD{_6xvMV5Ez)#Vu_i1|Imn{-fJ7USO?SxMSI9S6dPiDS4OVa2t@1=k
zDq_OhS+#rX2ybl7w)E5T&UjZaYLhF9W%FUOAoCtli~w$iRKj3^f@um*16kBkTEs`8
zFnWvdH(6W$P~K}dicJjub+R2>m?#{hv@5g`rx9&KL^U3GFDJ^w
zKp!|A0dMfI)Y{COL!vFhOKPN+jJv-H{r1Z36b
zs}3-cj2c7#Zs|l{OdQXrMyy%$){(aKu-T|XjtDo|yZ(r$g8s{0SaujM3xLFm<$9$v
zU9HkI%G{@8-e*SfmT4$T4CfU6`%&L&-q^Wd<}P@TxJU6APz$w6X&O)Piv7A}Z0E+?
zv_ioK5rRS0uKClZ@Okd+D7;hwj95^WcTB(bYfP?{>nH+9y^supV7WtwEQR5l^QW^uRL(PLNH2j0aM%
z_R%na#t5gON@%RLf5R@mh~m_nk6aKZiO<8<@>~1+@vT3MzNi2m@yJiAj11~QRD9LN
zAM9sBfh5<}%-6H*gRvcdU{+6+C-t?KCndd&)6bT4
z+6%kw9Yw;{Oo6E5G0OS^cOjh#%Ju?8^2N0kDeU?2u@oy^>8h``1>A#ZgxIMfXLDW#svo@2_y})G>VyhlsxWqg
zWW5-7Pkv@CF09qKLHWG+tG4^9p2NbB1L0Rs+VlFz3I4SQ=_r5*xq0Y5Wvt
zO*GYZRS;pzo8EwXU+|syq*$#&*1MumdBfJo1X=_;NK9vz?53K{Q+2Iu{~|O~jlL#V
z;4$ICM2yb3R@
zGG7cA$t|-q%qy9I(0FrPT}R={y`UIf_9|w&6^s)m^-3a7oc{VYC3;rr%lFQp5W|Q7
zNr%Qm<9nXlcrU62!@to~M)I4iWu}KbuvGF1YOoUjrqAl9U{*GX&XHtu%jw>Y*I9{1
zJ-?;un4}+%=y0(UD#D!?c8S+0rL7S}rXno((S5$aT&o*Rv{M3R?-#@@uY?VL
z({*8IbDpd#+z>OGxs5$4!lQT1sTu)^-Fpp~09##i+)X|B5a#Z*`-QL29}t+i$;kf;
zisb*FRAc{NNRgG5DK%aIiUv5Xy%oDLf(m@mCn*9Qu$IsKI)o1Bg0Kj{5K}B-_%z^!
zRM_7_O_5A4x>O;0Z*?v;HbL$793Ag6`taR3~Vb6CN!(OypW
z+Fri>UEoshF!S+jMN`nw5jeUidR6hM=sJlwtD$kD|MS$DE7~xoi8Z;}V5SqG=}l6F
zam%7kBg5{!8GvF}qm1j1l3}j2aU^#Vk~?v|9qpUX%>Az|gbJAMze(MgtT<60L{3OIKFQi97XuWURBXYr<|cUH5f%R
z>ZKwzh2{7qdD+r{FL#AzBM;Qpsf&rJtR!!aNfUF3-kccC7^
zvg%0n(Kvc901bE~pD9&F=dOSjC_uuXuDd1MQpeB=N!?@x?NZ?(oTMsS=r|GpBT+b%
z1&d)Wejc|=aB>euG>toS6@+|pvQ!jh$T!Dol`v;*S5hFt?kuMjht-kcdlRXrP`1F?n4)Tugw=2~kj8>nO^2II7f>Im5X>G02~Z!=dl%YH42dGi$8g
zgmMzB;0DNe8;)M5i53_3f?btOJ|%17HHx^iyu6QZKdV){?RJg3`w6voSF)Lu%_-PL
z1t6hK$lpDpCn5in^I!q4ZfD@(kD3oDE~?$8fYiYuX`oiHR*qZ9E4Q5Y;E)d#ofDeSFLPvy(BSE!#z$ao7Drty2-a{)?
zT?@-$L3>~y5K47Ok#`FQbA6lk1(1YWbxg|4mjP_|*rDf;k7nTt>swNs^=XKq5mi79
zr#aaPgo^TtS{TZn%a*aeHJ0UhGT0Ge=v_RKM*2bxHP{SW*II>b=D{(T%FgFK34L{I
zeSo_6^+n|nLtM4O7?YgWbA|CPgrI%bUkf)%fh#b#Ac)PBRFLS`52q_%uA%R|L3*nR
z$G23oXiYC%=1+V1W;%2!0
zOnKl}LI>J^r*zXcly@f^2*mzHttFYs=L0HYsL(nez7^Ckh3UMAZj0^{;)7h*Se~O>b@Yk@u7)i&Z90-$qZ{URFW>fRhJ+5Yc*+xyDAJI
z+!{;K7zSOnjjA}c-mE-FQh^F2@X|%?HNlYl`_;Q|y72PCt;GQ|zX8-){DGyhRK_16
z6h6QjGA=_7VrIj&w|F6=5O^<^iJ<(KgU23+C{-xjuidcBs;npej8($3)EoGZ?n#ePR*nkk-(ce!p%`1haasaa*4N~^IC`GrG
ztGCYMEF=%^v0}#Z-srD7j^`g*C1QVNa@gD*Jg{FsVXgYL#}LJTog>wRD3XYtW+GB&M+VYD51
zEd)24T1>20_Xt$LmlU??QwUtuG-~pnWzl&KZA?d^1oMT00N#KoF){~qU_Il_`*^i@
zK!f>AN$S%_XIv?QyIO*!lOvV;WP)1Vc<;Q1HONrliwY))o+0%g@4ef1+@Bu_3kFES
zrsM@?aB9qwC8!w8u%V)`@)OJwC4bODG8PX?xvk5zLNwZA;hPyUvrxWB(2z-{#N7`{
zQ6UR?0lm@!fy(__6CV)wAB|ZYaYTNCk+Fw@pd;Y8LV_lxep4>3?kf$Y!YhZSV!r!L
zmRBa9N`Wmagx^+XskKZcf`@MApfpR0byT1y@Iu^JI>vDU93zJ>J;&pmSN7@}J1C0ms$(oDvQYd$@2+XQkUpR5i
zS)@crpsBf~D@tN1km5}#Gb@)*&NWU;f)gaF9x2pWf<=6!&D^zTduVtUk~4UHvn&55
z`$xpGpb{?fY&@+YjTXcEn5uACIO?~&d`r);)Gs?~Scy|b
zS>;;;^Ad^zTL}il6%y@S;kk(?NNGS~KKtqbfU(<}or8QEuKk(&prW>)GI$ivHX5q*
zx+hsXT=Knxi4Dh0TkH&$N?SD+H=W&@y{b6}r|+&EO0d~6_ggiE5p_&3H>%DKo^UDa%S@KQ+FtJM9n
zKuW5=>-BmZP1ZllZ>}T#N0ct>m$=ap>1gOs(4^V%HsxGt@UHdFz(wXqr^c-eW}nvQ
z(|yKW^Rz|BMUDvT03tlS@<5i2>q9
zS4kOWWxjFZ6!H;*!qGHxkCU#@-Jx3y0=`q34I^I2-hM-|Wy?^)ksjl$2SWnxfXk&Y
zeto(t@U8GaDNd0Cf}}c&y$(VFB5EI3*S99&VO(~()2=VhIk`?7fPI5b@R*)1Ae?LF
z^%@UxX=kz72lwR$a!ZWKb={@@#;>2Ei*)I*oJ%uLob-!ok~|AE!Wz?A
z0Yq~{;^-FcxbOOT2s>-Q6Fd^fK;YN27}D+c7Y!edS}vpdvUwd{xN$W+2fhry
zJf@Gd4+9=G1YfMRMw5r!tE3D@74a!ir~R#&;;_c_eWeU&s<&Bv_tfjXorqn}wq8qo>6zM%Kl
zH1_@dVCv@b-UTBjqVAea1Y_gxA0S`I)a3}k_ggvB_U1z2;E%y3M^r3flB8A!rTd|w
zT{gWFf#T=UQ{@lnz{71n&HohZ98CXHth2Im{#UHi0{>{m9*m&=$E)YJ7RM=N`rJv@
z!El6Sa|s1N;P%0GA@#*`u{L>(kvBKq1AM(IIw~@YQsx=Z4lOgP&kL_>E2y^l?DnUL
zu8qDwIzu~ad*aV1u{{bBrUw@&u{@vzhw<9uDbdSSGlxnPDUX30Y`Jzk_^rHrW`al?M1ZQaYHat(4uRf@Q}JzfJY4$jmg0peoXizoh?
z1GV6YR$PsxhANkLv#AyIZ>U#aksT@uD(pHnyLTU&neibp^m*32
zf^+yEjySaQh_{&$ziEs-#*-36
zfTcqV2jkfk4!u1^hI`O*do4T*8g9QXDlVblJ$=fbUk@gSe;vz;`*>WQeTOcni+|h;
zJ;Z7!%WFTgr40qc=Xz=SDWCm8RZW~~0~wXv*TEUjc(ct+AJZj{ecmb;n60CrpBt@b
z0AuCeY~^j1eTEwmDU8>F6$gO#fB?`XfPEuQ`fOeQ7)g&KpVMP~&OLw0%Aa!3)q40(
zs~Dnn{HW9uS@Pr1tmCA;sO-d(TK>X^Oad`1Gd!2}^yfI190fqiX
zYG>QA8YRe@80l0_d^7-79(jsMfb*T><0OtaP3lrSOMy;B|eIKDBQqYp%w^<8^fePw+&jxuHF2}WBx-PZn8pb2b3-Bsq{Eh
zW`fGVj_P1Tib3w3A;>k;)BQ&^k)Ne5_l8OOzIJo&NS1RD(4BE4jstk+RVAUT($klT
zAEEonr^7SAYbC|{theJg~V;O_QxEpk^hWHLvP#J}<+83UWuOt;G|0CHpYHBzl
zB@_l0i&+ZU3gC`1e9Dy20Xp??74dZ*+1<(%^y?hR9FsOzrCg(8`;-y8LuZp(F?K$!=C;4Bgt=JK60FLm7hoj;9Dnll~oo_ifG?nl-Ot}CPZ=*L5ld}6ZD)V5${mtIK=hqFr&C;HSi1cn3|kgpRPg)J~8
z$?Ci!62uD4YY2gk)jH*2Qbv%;`uPz5ktL94&MKC@L}!ISXvShnV@%YJ$6cQ5!5c|V
z%WsNf8Dh*m9HO#1=`&@aZ}>YvpJE>wzMg=RaK~qf$Ojo>;EP<=a#Q
zz>SBze#9!5=g$Q}QV1*n=|Hb`|Ib;?x4Ri;5Xyhm*Gk^Oc=P5k$ixQmzvmpN5wB`c
zkWj-I06+%MD0s1))Bh0Zp95B2SH&8v=DbH#W`tt`8z1if0n=+D9T`~rPNxfd1MZQ6
zGP4}e5#VqhBMwYb7`}_yjV6_M&Ia8_`?KY(%>v2&p
z+Hv2Qfl%Hi)5XGx4ocq(0^$C!E}i_urY!l%2>4W_=?%P^RuigzyTX?|fC`8Rtww*k
zV0Xf^#ffGn;J{y3)&e|tG##D8K7LWV_@n2>P-`Jt<@4b^Y!G?j;igQbZVGpz)h+`KhB@!i
zgt^2^Z1Qi=a0=_CrC0MyxKA_Cda>+eXNF*)9dW+aQ+;KuHJg>i4O7Z
zRHfM!m_jy?LUGYqMA)@@bXeiACEqvfjSvx+y`6LW|#K%;CZ=d@DJOO@Yb&XlRou(qWdh`(Fx*$a>6FOKIyJS^dDp~Yd27DVvjTjNA1
z%j!iZGSz}Q#)f$&&vNwDMTNp3ujsM1%&u8-bWLr~P&8_@0sC@|M~Kv2k-mL-(=y2+SN4ffsh|Fx1J395=9*C_yUj!NZCSudBdMvhtkP9Y8lvPn
z^LICDhjH{If0zpu$vwdbE@>%U&x93HE#5iH;#>U$&~!6`WZ%S3ckJX_O8^zRZwf%)
z{DPy<#}~Ps%sGqO)EPv?$PhSfddt_rfol1%s(}YWflwZ_7ExO;$$Cuwg1f=D(6ky3
zYC_kKK6LK|G!iQ(!#BU%*^6Qqg^U!iR=kL0d^#;l^I@d>MNTReKUxDKz*bgLPts=C
zph-}V{5*~;X>vVUeDP6_v*Z88w2cp0cvhR2i!5cxFT_<;CO0eA)${e9HWVfD?%5
zuhOt3P&r_=m0hFp`G@gyL)wqYs-h839DBdQ7Rmz&3M2+N*5lk=_79F6vm-f@bKPaxO(bO^-tW^ha?-
zz=3)i+6#VYnbV2)U@!q~>|GdW#NDW(Q|M<{@fH6x=E{F^Kgyxcfrs2L)kP2djl16c=duxN$-#cO*{ue9FUfIpOj!Uqvf8^p!E!QXBlXeQr}dKht3
zCL2M{+340;J|O3_JM?QwD-bzr73*-m%|M!n%O?=ECPG&z=h>-iL@OUp(Tj@j(!W};
zjg2S>dRsAuv8vH98WH@AX9)|
zf?41GOMQZh)9AnVBMvs7oA%1+>76p(ob!TGvxIP@Qi+rge+7cm8$93Xo{6Lp9{GVR+-hmCgP;j}j16^YF&eJHH!9gij((|o(
z<(mO(3=<*Vo|5(p-}=~QGv|A3@z*0D$M_QlOK@Ua^StC*FB9G
zpMp$xK2-G)tdGNgumYMZCxVvRTb0|*ip7UQDs`?=-yfQyCYq_Xb%s1y3xS=_6Ti=h
z{6!A#MwbLofudkHqCfBRA&dK27%h_OoPvY?f&7*NgnUafa^<}@VyPoXyNi!ZyJw$@=A3iE>~04n
za<@f=xs7?#CA3x%)~&Z^!)#sqzQ@8`6HBMYlczBnTO077u($pOX?kpXhXM@&)3V?e2CW_do
z%>7?t*MF@HA)a&M9ySTxG||3r
z@pk*rNo2e2^T8Qaj$SbKlJ3On#8f&qx_al~2XpjqdhzX)a*~u0crjO+34ELz1b3U;
zAQZas#&;Un`rAym4CwRnVC{2oVO!g`&&Xj_uiy$@oor*6?~+gSQmrz*xTmF_e^(`#
zn(+3Zdpl269Wo7>Cl7qtV#$qo)TBUI1=GO}Y0^z4n}GRL$@)=I`XW9qE~_A)A+KU>
z2^`@PLM`?@O&o~?rjc6{MP$m)gk;|EMm^$Ri&^L_0`^iIIGQ^IK!>YGy8&^2NFNs4
zL=ZcnP*AG+Gx5u%XRjbaz7TmICp|c!uR$Nv%;f!pcUqcX#$0;Op$}a`UW7rp%2P8e
z!yn9I5nY2j$R!=sOIpe@U{)U6V7^9BqhKOd-hs`Rf(RsgQ
zRclk{m@>Q_4YUo^4ChfFo#=GEh`kLmb-?(Y-+@;|Tecu5B+OqzIHgka16U1H-=_gf
z=Zd}kQql7F0}@U7D3nt+_Zv3%VnZfs9N?k-$Pwc_I&HD(XjI
zynLvtQi_KM&4qg8XMK0nK#hPsX`h03bGby$=G@@gKtG)*C?(Ph4jjMYJzC!nZNyv}
zM9o-<6bDhX<(SUu#b9RVlgMcdDZ@c=nOONpqO2QmaE%fV^sh~y8P?w*-TVzLJ%$6-
zpIk~cUCNirUs%4H-WC|`v1pqpS|j(y)oMWn)J=a?BESTlHANN2!p*!u0|76>0_0q#
zwpIF=afy%AE)U9X#SYxqB{rLN{)OwB7HnoyZg%2*679=?3ms%?<-xS~h)M5cxu>vE
z2vId)wIvwoj%QMYvI&}pjeU|sf>gJ3zP*0lEWps@k}3K^Q>9E)7dM`z&&opW$;wJI
zZaF8_pw$F2$Gu_dAnz=GPXUUn*0r=ZRrvWIEMp>lCBqfT^Is8
zfcuFsi-0c%oGh+q#R4u7e75z{p=Xr>@?Rw;!SYoxMZITwN=vYGXaxajeHMp**rhSo
zH2eNUoah}gjl@et_X`ILOihyiqnt45DWS$wL!547(^t8R8=BDHCJ}>TkmhtCWO$d1ywmSIb{IlGP%wVZ~5CSE6QKgF+E_$pxH^4Ya
zaTt>H1}ib*^De|mD56(d!zFMxN=5$=hIx~2Ze;`gOozip)zK|MY|q3G_3S!CZ>vV|
z*oZ@hGJGsb_$HVyWPJHdJ@~<8Z0#+2g27-Bmb%17VW9e{t$+TEc|I|Q|Ei~&UFoL?
z_ist!=Hpq)8di&PG4zeT+fW$%+x$}R^H{@1`fCd79RUeg+e7`)wKY)GjEeU^0+=FR
z%){l<_WUA$*yuwxuVx`Cw1yKNRA6orM*c~|z{fUDY)aH<`e){{35H=1w7|_#7
zhSQh-@(^vm!iaW`kPnPO5iM-|!0{o##*??YbmZD{D%f~qg7si(sO_Qb
ziotAdJo~2mtjZ+vVyGC$C=B(UnDO-AX2fhriY-gP)z>}c@s2eA@&v{Sro0u~-o*Ys
z7w)ywLf&Q==WeT_T0P!d_`T%cmHd>&u>5Xhq+4etQF`m_X93iGD7(=zih)e6Q1>i}
z83iceG}k2Ks;{Wwgubi4jcki;Xm9hYVRHr8Xy7=b%9^ypI}Bgq?n8V~gESHBw$s!w
zX&hEoI0+Lr!G%31<3sqRxuM=rb6c>A8rp3eOXypE*!Iw}e`&`y{CVMLNKn00LW;)G
z40l1LT!V2oMh6CbyctH7O)En8^PrwJt&{P?8wj4-$tdI&V!e(bd)kj-$v;|`F%`mm
zH0i!$aun;64^%uxiWZgcC0t46jPpdGhMQ<5;wPsmb%0czrC6=>iMXwvGp=2006jF9
z*+f&ubW#`>JHD9q{j0;3EDq_Ki-~>p$s#gf58eF9$y;1X&lGvwm9k*@$$M_DMOh~$@t@@A=aiTQ_rM;adO}I
znV^70_>Q^?Nm>hyNe31BQJ3bda{9e$EADO8Ne6pK60NCO2BKRV{==DEwhx30a<^1R08#nQPU+PE>
zI2rKh&z9X`%YUmm`4YI@rbBkLVuWbB$~to(HZwx-hv0E;rrP1^j^tNwS)rd-1dfu`
zWlO#~mk^Nmy7y(D`!X}<_l=n|A|DbFBOqgsK&a@CEW&KYJkun9Y!h+lv7q=92*Y_Hs#$!*;v+57qmAM(j5UrLsC;D
zX4=@}zzhs-gv3PYNmDsv&;$p*gDprZGf-#>27Dw^V!kP23o|(Pfy|L566v8(Yb5+<
zJai3VWuZx28Z(hRL8E|@5~IQEkwYyeP%LS2LFSJkgrh@7jHM?HIZuuH&52fj0FMD~?UV<~XV-3BY`vpR`<8XP`*%CuEf>OptQW!t)cn$$
z(clX~vq8)t3Y0(-$MMvkra#&UsDOE`Y5yp>)tJhR)mx-u^Z|@V2`dp(i%_IG8|%aM
zy~2H|cpLfMu52BnE*exWo_i|%papq&RdTNpxNkC_6k4Xyf)EY#C$a(IQ#V6&xx|0c
z_bv5pt2%CJ^CiKi4BxaAAdDsDp9oNY>5YKR2(_c7Bxs4@6h+MJ>#U(}i^*i<(T}-V
zmxPg!{aOO=M$LpuxOqarHc_^P*JZ`R%Vt$fi+ysSNh?dq4q(Flv0*v;1p`Jk-?7bq
zEf_7*?!g=y;3UFapJ)Nd2QWaXf_lt33+FhD1&BxfsSd_{OCL>
zFZ~5_A-{H03ihz6CTEBV*#Xa;(t|6Gy6^0Ih!P)m7@a?9H(M#UUZj8cMeLCMj{hz*
z!9&mO>oD|F<1D(Z+WNC|6Q0yl--htQu}k5idh^+1q+cOJbvc!sar*Q0I?nRWjn4>t$YhQ?<$MpQ}uzF1pD^eW}ui
zgDAi3zsts*Y&Ov*m37~fUKr*WmPx6J+1Zo5f0#S|Rd47D2?QKQ-scL2gb~hybZyJ#
zBBst&APi1owVDQamUNuNJWE?^w#)MK*Yr2&%M6EnB_x2^Ra&ykABgB3;!WsnT>$4qqkTYkeZTYCukDT@j{YF#qjjC=QUIHh1^`S6?b^*Bv!K4
zo<6ke>+NiGn2=rk1HU2)iT?9uvj1=g#F=(7JekzJ7n>G&2GPaoItV3`hwYtg6hZZY
zPyc)#0t`8TiF;>{Fk6qze2+bHodmfx!5P+jl!SjT3y(ct0#6jmYD*ru4paQ38qM@p=#})sweQ!pqy9#lDucG7)ze|6h;C8zXV*wRfYeEI9
zzhSU8QDgGA&YY!2MsRnqDb%pO0C(K67FVuC;{yE(7q(m0q9I4T{{vUVy*}|dMw;mM
zL0ojlFu7veFQW6-(h3T3=XBHqgMN^i&g@BLE@E&}XJqpbl9og4TZStQq9$5_B9C*D
z;iNi1k;f&;@Z`70u=kV8tUI=HU9i|B4tY&^Ggt2Dd@zh*yZPH4?@Xj)xWofqLEig*
zF2Tee7Jm$KxI;t|QY-@jebEPQ<8NL!lOg1nEye*do{Rh_WrPl5r#umYpY3O?NS#Qx
z5PQkN8hTw0SvF__mDKMr^-$73%ltGEIunZk%0|t8Lg(&yQYT1w^@|ny5f%Cm@9cj@
z%lxL{V2Xc!MMn8NV`pb6JN$Xe=%RcL=Kc9Axv%WyN3V@mJ#d!Jq}=;M?X6T_zh7>{
zbA=N&0+p={=95sxF9sTG2G87s)sav?G$`iP&Yn7=wP!ka$+LrT?^`Wz*+L=Jd97b$4G
zzNIK93p~|=yo^qVBUGgXxreo@9z7d{z*!P>4t}&iJa^zjdOvl8=;vdjh
z$0?;_L#fm6?JjVfgY}YF{COrw;o6MpObGh8W4d0I6!owIZody@#FD5vF{u(90J*)F
zO7k%$KaUs>C+X1{xHOlBDWl$egMl$;kIS19rMAnA`w3S#(iH^S2(
zwIR*-<0dgfrvAp^IG><|`UhM&13BEs-(OThr@1$Ha|NV$-kZF;71-TG)#4DedMt(C
zF@f%RJmv~{?6px86MdpMJ=5=h0R%7t29G0#@*p_Vjb`eMk$_BOyN1Z&NrB1AdA}Ed
zM(07yFaP%84Aui!GUJQGB#L_%VR
z@?dpW_P5Yb^~l257kV!ufEGHPXi#|m1?`T!vS&m2+QQA9on6^biLmE=;TwH){L#pCIS9L)I&bo%~z4m0ZJy2KqZ&;FvxGR#@!m*VNiBVKf%B0dekxZ@xx_T
z`y@LDfj>B+?qHS;dFx*LjZJrU(*?gYPK33@M%lz0Gbzj;33wXv%=F>
z09=A2*4erudXByrSM|8ZJU@qmH>PFR?{8<@VprF66nUjtLl~!=s5m(jD2Q)U(Z61ftXUsYy5*r027{f9-
zKq&g{l7ORD+jPecz;@y_21SN2%C2t#RU-=XC>=fE{Jy9Y{SkP-vy||%ifR?@3O%@4
z71W5|&>Z9ww^C#3H!=?{CIyzns^5lX3-wn~5Cm8+b=H&7wa^+O=f#Ho8LRFbV&^mq
zM%K!Tz?(|&EXAc=PA`mD9$H0SmRl6N@%UB~lErWuzJeg0a9}C87lh+8*%Rn@`g=tx
zVX&eG%NVpho#NM~ml@Gl$9n~i8e4dm<2fULnmi|@Ybz56UiW@PNvvdF(I?@aN=r*p
zEejOIWV5)^*0ety`-eP2j0G&7+2Xa<#nsc=Eid
zzPr`!1|W}mmddQ*-|^igLuB|R-_r&9
z8Sq%sqQ4RqybNh=&tu0$$+Mc%)sH;-8%HiPpTLwpYJ
zKFPXhxoFBdr3%$PAIp*IJZhQ7Xg(efRmmjTW|Kv}A`$^+#eP|Gy2`AJG`5=Ey)QO1
z0DMe;81dlO#VUHZLAJj%g>$1dFP1kOub;qy!0nP-k~w(2-(GteZAG62@G|{rpzd&s
zl>3z>F+W{gV{DDP1^Qw$EENv`=&HDv=fds?q-BzBf3=lNSMK`d(tD@N9Vjspf%oW1
zZKhW{lxw!y6eZV27|ZdHy&SE(=#bs*0T^TJ{Su?_4gU8M~>egtd;FELwQzvLMDro)ct!{A6$qAJ#mmb}H0@
zEQv>*th+#klawlfoE5%W?3AbuBH^D%?w*r}f-#a*JvgmhV0AV);J60K_qyVDq%mt3g2tC&u_lviQYH7ckQG-UNNppY;+iz(Lua0S><}6egL1%XjC&cC+?&nZQmt?YE~M5gf20*6Vw+Ks)@_X@DSuT&;f`ot
za!(u0+wk6c9mF@F4xi)@!OvX+;Y?4kay~30ow_~
zK?n%`Segw^4&>IH`v}lWK<(-F51VCN4ChQUm#Uo*(VT3?1?@mGtS;QG&TNZU_94|p
z5v-5M+ulTv8Y_wXeIa})=-DOmZzjqLG>n~kjjvw(D{fBzZQ@|fJ)u^Fi)^!S<-PCh
z2Mq!&eFfKC;(1WKvJcj#mFB97V=S%_wcnr^na}O)bBnY&0jzQB0Qr1~K-cWXNKJN}
zrGgKa28eBSxvR1UJ18KcRS|Ly3V8&$wv&0U$ug}1++;mV?)Ou*?!|+``@N%-+}03b
zs4FFu`KD`2@3GtJO79utw$Xcg46E6i$`(k$z05BWjWk=u9nIr%{yxdjKko8f;q~Kp
zrTSvuBH4IGx}*GU0A&PPZx!`M$KAFWLX#Y
zV_^RtK_f1FdKYqD+b^RD+EEZ6*`WPJ+N6NS;Usj-fU(Q#8=Vo%;ZO4G3+s7a(7kxX
znn3NsRjgiD9g`2gvxwcCSqcaG=MjpeP^O94Pa*t>mOwFiKo~nVNdNVjK_BZ3%>z$)R&LUr@J|vP1Ugu6JCPW}-cUR1*-&P(w
z^OE=!W(#t(klL(mO`%`bQ<9E{>mc*Gow7bBvZ5jraZiEgGF#o`4a4XjhoSs#g|9p9
zM^#5+nVj!SR5^%#@kG{W!9`MozGDLA(XejF)bIo!V0dIbu`Y4$yPiDHs_jpsY2N*u
zP|0pYA&vfUgtRrj9v|2qdn-q|7h}+yw$JCAb{l3J_@{M!hPYa
zE^BC*2PP>Vo`kmd01J1$&4TvJMGKmAgT8>T5@w*FdTlD#mlu#{iA2k5Rog|1A0U()
zFDc9(Kx9++;3M>s*0ff|g8s$7kLj%6Icbe!ta7IvGXjOLt|e97bMH~FYhNniqO5`f7=o{
zS^pQtDkC%de{6|FFbv|BHZG=4X$Ep2IDi_hi@2>;XU_Eg(DLxRaki|7i
zkX6PF(EEO9m>v2XwR*J;X|23EvG11|hfpo4GVkG+VVOIoYz_d+iHB=Tq`E+F_}k0q
z1LkgT(t0J5SY~n6!(|br6ZdW%znJ}b>2CY8O-&*%H^A-s
zUSt1PIDhW=Y3VK-n=irTG3|Ka?DNF@P9x99;pZx*BsYfFvR=)=EulcKhVQ*MLhH^(
zeH>@-4}Ij}3*En=mJnWk8!M07crz>Y=dXVd)+}SMM&`{`^_p9|g%%vl6EKCuOGHzT
zt5k3kev=&(*v|Oxr8#M@s5|JC%mZq>Hfrtyd7Z9I7q4x*9qTNn*RRdCIjj{RNmhdid<#jW4+0A?_@YT1^;DtFmf;!}}iISkTl3M=*gD-BzKnLpTJU2)^k(9N5&b
zVCUT6P3y6%F6U3wYR2=PPka>|n7%CM@+;``Y;4WL0p{DS@g7!UY7|{^L;K$XF(AB3A{~9pQhDwVvxP
zkOOKy_|!U%(I?CNfhzF%xmO>UZd8Z&RG&(@?auCLE3bB3ccvVhslB`Q%?vK+L5D-;
z65+v88QN-}^(*{W%(7<|+HPTytSj@fH&;-d>)Hkre;H4O`f8kZcBgHw
zsnA|Yng?8I$W&8fwC`86KS|6^I;v@KTxSw^Sqx$fki
z>o*=Hjk@ZEur`;L+ukVI<^o8`W}?wjV|xOO1J$fK0l&dMrCTNf_Eks+FLVYuwQI_xLEUK4Xvh
zxO7|R1)t(Rm^S@6h_Gvz8%7D9_QwRWl2pX<_`Wkt2^nkA0*=M>
z%tei83sy>t2L%L5vk22NRz<0lXDgB$;EUv=OT#+omcs+}KC%Mu!2prhxm`zv#nseU
z41{Qt6DRukV+uxI`1e#(q6viFLmGTR1o!|8qCvSdZ{64k_@D9K4r9T^5~X*5(pU@P
zX_1qqNO(c1R-HqEL*z5P|8X({x@j?WHaYDYMS%mO0|g+}X4_iRfIF-Mcj_h65fahj
z)Uygn=)t7DI~@tcGBp4`p?@mt_J3kJ<5e_?p3>q%@3{t>_w*Dp7zpq+2+$PGapw>B
zh>Y2rJi2Q`64n0V_umFe75QRYi-n=p2R%|5%TM(bU~xRM8s#RV0bZSSp%;PKBQ+#t
zbnX#6m;i*2^k5C(gvw-u`bl^N)p;616!G~Hj2p0ur2HH)1%l%a=fJ^tGhKF>Hk1Hg
zq2njI?C|THp%At~;XrtTZ^x!zj&d{M^TOh%j%j3@CE-`aQ(+C;0#33*(JNDwq$IJ9
zavwroew!rc`6y0r+N!HvZh6Q4R1ZN@a$-KS>)5#xdFqCrtVLLl{Yk)Kr_otuw9JgG
zNeiHit${x-qKoNLyKQUfc?~39+s(d2MD;#@xs<-e&a*eLOSa#eR8yCOb=&?Bq(4yn
z0X-Wh2fkZSdrQxHvvP`6FIFzY#lP2azYw=D<_(B&b(cUjv2)(oWV?VjfnXr!532#~
z&vJtg!sTXu!e{x-eHgQ)6H=)4BM2X}yc56x{F}ZbD)mQ7&;w@mq4FPw&rVT841&ag
z58hT2@$SqDO0;?M*0x`7hu-+DI|^8RU~3iE)eJH1X(Gti_4;H*s8YK}h4J*9qzG-G
zF<0YYJqqeC3vhLc#zUv=Oq6uxQ|k&EMxTM
zl7>w|K5{$X_*Rv1Z!)SuL?eyKZ`?z1eSd`ss#KQg%z)p%up6uC}g&acltb>#yRY@GcS+r4o`vAAtzs`V8FcY7ag
zIC+BTAm&S8Y6Tx>6<}Z-`G|l`*U_(l`%TfP+e9hXgkDO@50J+5
zD9Y1aeY!Fp5BFpbVoCu3puQ*+CTnkUD>Xv!j|`pSy@-g3;0ey9L~wu*WKfwjR+1c|
z&#xjVSaW93MK3Q1{JA?8Rtv2UQ4}CdetwfZ_+GizBDoLb6$T)6oyEry+_>~!5d+gc
zaQ`r9hThSJE(tRq)Pr9hiBqpAR^{jv=1}Cr`_xyHUy`RF`V<4R%U{eEMWR|JKNE$@
zP@FzxumAXAyTI&;&RzqcyWdT3psv9XQ&QiyGXwp>Jg^UpoME{0wla-dz6^yR#bJ}+
z@7%(M+7pbN+##mw?19f`dpA4xpwDlP5biz)Rwl2w6m#ClS}d;zd9=
zor9b@z$sG(TnfM>ya6atf-OLybwrD9?gU*d>RUBg*hI%d^Y5)j%ox}+@jdJZ
zcrR!aTXOR~y!Xh|N0^&I(i8jw%N>;^|A|V$VdPe*{|HbZU0RzyJc6qWlQ8lYsFN-g
zwt67bLw5@RdSP(#UIkPuYDE^$EsB7p_A7Rb@Nh)ew^jVcmEaxR#2d
zT!84B
zq^@Bh@B!!ni^FiYNf+$4-04cku!4
z0G(Ao;{N-Rz{UB`B_RO~3HU#uY}9~ZoXyz7)+gT{VZny*E*2vXO7uP|h5R#ci+Kvr
zN5cci;gzZK26Bl+_l~!B0zEPrX4|+c+n3=~$=n~aA-octUJ(wDWjwrZ--CzSZM&%G
zB~&RY;&HRc`yeGcqDoYv8Q}!#^UBq8Hg~V91TdP&Jmly{`e`*jY${@=_&n1f@gglal_LM6qyBC
z5iex@nQQmWCFO|6m%`V4{nI_qK_a&(p=w6|XQ3MN-2VLxypc80pa{H1t4pClf
z{OP!dB$SI=xQA1O1D~|rR3i8y^R{8cA%2h@aN|RT&RUwI$4e3w4hsTEnAwMdnM0Hc
z`t8_IwzNIGXo&C{G=wE)%8-)0Q9WsH#*I>;Xaq5z5ko)wx<9G}YB%EC9u=UXl@1Fz
zNs#*b>WBj+W&1b!*4X{O@7OqWO71FOp;=N_J=>c}xn}^|>mEnPX&Hdmb*GzxR{46l
z9;cfS{p9xE(lwh^zcavQbf%;K7w;m8-!1GUw25ln{RY#Vns|P=%O^`{@=}5%^guHA
zAN~6k%NEO~4p$pS5g9HFLlVRg>6m_d)I6g27$>1lU;*>HHIs|Fa3>QU_&>@_U2g&L%pbli^n&u`%~YOy=%sAwOq1&2Czc^2KiL!_?=r)@$#2N45hze8W$N}x<9JsvHcO2375*T*
zR@LoTqo7eR0h8!X+_(N=#M@h^wsk)Q;5?HlQ1j_!leR@4_(gA_24wV-XoObGIp?72
zkA_nTQt!uyAO!$d{u@2(9qY!(MWIdWIx`3MDT=5^r
zziy*Dq?WeMgs2k_D(6`69dcEp(CMEI`g?6X^_aW3X9@tT+@1DLhB;;FIRagq^|cqB
z?m?6B!r^q~x4n%&s6O6SnPwU6$Q0AJ&N5{;=1g_wvxEtNp^4H1%;F`(kQId}YoMe#
zj#11r2Fti-k3jYVRU{{c!IOw$2WMlO>)q_37fi@0#1T)tb+So`s%Y&nslBfcSv>}H
zN!c?(;RT>CT0V>!5xEWiSML{M(X?0JPe+<5G3;9;AEIGZCWHF4O&q1qaiTqV0|(2O
zVsvLcdjRd(eE8v47X8%Arah)6@Gw#}F0=zLf(W4+aWLgudL(P4MAX|fTeoh<<+n;rqalIurp#zUxVR5FnC-N_!))OkAh5$3^R%MrgjHb5}-h
zMlyRMen~JAIPWrZ}qINg_3f`WM@j^3Z^@
zIDinf;w4F(^mC2u6s(o8TH2@?r0x5b>@?(@l$%tX!AZesqNyk8X+3_7qy0xnPr*=!#_8^V5+_qfP>s5
zmPU{%Xt@5fTJt|(xS+6~ECmNP-4Auk4<>$D%YqD>za6~cmm_%PDydN-<1AOo
zV2hWM_1dup*MyOyzD=!Cb3;s9w6$B|i~7VfT81f6$(%ON1JE@WFWEguJy$v?wE@6S
zMc+GYHC1-P^l8ds(2}I3z(yG}Y3&mBhZtvts22T8Myfj4HJ#{oe~+HL*!Xpw%Ozm|
z@0OK0c#hsR-HUkjV?ymQCs*M?S*N+gY(`x63q4`8PvI)2cvYsD?@N#x5!X3-qjQaU>x*6JGbQAy)34tCtSCzRDmMm
zRGnewY5du^{erL)PxT{^*1ub`LHD6e>ls$|9~zrHR$miZe<(kzdaViSm!A{~cN;cZ
z$2@7$e$TGlX=CD!x!x|BU36#nW5`4@799?GcIkqru@BhWqdpbHIgV$)X!09&?g^cWgR38kAhUGLM~YgjHM
z8Ytkkyh3lLStX;z1$YS99>Ux_*dFCzNcH1Y?R*na$KeC1v7Qy(!hky*y6q}N^o4e$
zV8;CoXupmdL8*-sSS-8BCqS8REZL#W&tz*VtqyOF+J}+On}wGS9b=mf(m;n)c^<8#
zR(^T!sF}<&XpW^rOzJ`;F;tjLsbxAo?E|kk$vc@|OpiFvn2z2(B^o)2m|9dm1FvCB
zIDk*r$CDjkNzyiyRG0)r$fmvBTWik(Jn0x~XwT=)EX|Gtcon!)0&sCU+6m~0lNE;(
zY8H%lCK8kUMSkuatyrAGPjCu$aGMLKraM>4gA2fd3UX!q1Jyped5dS)|NK4(f{bJr
zTn~TQ_2wZK$FB!nniwF08u5$Bc*=P%;y}XAjr+j`2(#L4wPPTOUz
zznOH|G5&Gd8YqhHL~s>Soij%n1)M!MMh)jv!95bg!-$m{8=6SPpOWS3?}`vCq@s}<
zukWls50g;e-DclxWq8bVA94xw++8Ml_q-iNXK;gp{k3Rlu8eWsNw4t#-puLiH^*g~$6=_BLDhF-fFqbzs7ltYYxT
zGZ=Z70irTkdK`3qMsLftCoRj3_$RJo`asIZ3lsBw9bDPg;CjI0B(|%@)nj*;U9xqT
z*&aV>w%AG=#R);lX)N89VhRMY5OpH}PL!a{F9UoCEJ-)#}J-xK3C#Aj^RnPFVz~F0d
z;5?xf;?X^qxe+`>p?xfZo@gxj)Zv@iMc(Xm_shLuo*IaYpw#nc%x%sWzE{gTD7}03
zY)b?8!*w^iYXW}t0(hAWDz%kuyD6{%UAcdqDfsHspXb9<^t766nbNPd9o)==c~Jw(
z2|E!+PEF7?rPR6h8=H!H-%LTNk>2Jkzj$K$<3@M%awqkP^S5+q{@774m3#MAr5SD9
zl{%GI?&&J%=y*Z7_~hV^-us&=@#8b_=q#&uQR9(?;%S%<^=Q>>D6C*!DeV}s*-
zMD=R3I1gavcX@6koOABp7csB~&AaR!;$a2jy?5BTju(C~9H93)Z5%+>(&Rz93C-y^G1RiEWpt)+wK#}b)Cl@d_gP&r`v#sXS)^@L1*N}nEJH^iKx$`TeJ
z91td`gnx5xjnAgfwhSS0{*!0<%P!SHro$YFK+0&4rU-gJn}^mU7#z=2*MiLAbR|C;
z3u_0XV{l~ns8*K4bLF5~YoBFUBFp~VsLml14tYNrh{i+KR;*HyjI(mPMakx!5qEGb
zvKe6eAPZU+eNMehx!=lG+0Du`?F>3&LaSoQC83~TdLUI;q5Aq!;afTYv}dSb%~CWV
zaaakw2P>_-1Se1*oZZGLg&WYJcWm4GcL`B}zzc@Z^k~3D>zM)499GIqa|0;unHM4<
z**+}%1P)EBJi3lbV0Ud%y
zvbMeqvf{Y0>jo(meaT%2i5t@TX}f!xYZaaIWu}dF*wmsQ@)&y%nN&>!>6sX8X+@Q2
z97OgY`-&xUHL^jX{`B$7CYxY;HjEsdi*y`~r&-0VNuQy#~KH{0cr^#TB9T!0*Y7iMuYp{
ztFi^%%*0eRc}%Dq34o=FNYE~7fCU6CIisy#!r@F6%ZAhbR2okP@7S)h76mcW(6(Wy~RahQBwxOYqH8D3K^fDzKeA&8O|
z3#U>7{M3MbWpZST+!jZBc2k#A&pf1J9{~=c>r*x3{zKXz&Xc+0g%}~zp8=TEX}+n~
z^JGm%voK~CB(|hX6{5CTQ7e$tU|VYpyYfXghg1%hq9&+O6Q4pdc4wB2XN``)6!Ulo
z{AgYs6P>Xjgd^b)GcIpMISEqXTWbrAHc5?c2)1kZk>>zi86s?g0;*zi0mZh#ZicxM
zoba*ewyt(01CD=G&?B|51pqJ}8D3V2rwDmJ;|KejT2Z8vf?k)Fb0b(XKR9@6t^XObS)xR9S(7P9=Iy+{<@IGCLKNyVn~@mo#aN%ko;ygW
z`(`4J1r=!`9O~8YdIOEHI%fHABM%c3^M4;^m^hjKt*7+2BxT$tJ5tY09nzeL36Ytk
zbTJtk7poW{6%Lx*ixQbw|Jc1&I6*Cy`RdmbzEkLemNmN6MwA#rShjsGUZ#XQa-83<
z{iwf0^DNBWC|Vkgf|6Lfmjp@lA4=$!DNGsVr*4?tG31+{jvtH7KfF3svqw9v$_4C9
z2-LG|6UhJ#j(M!DuHH7`OV3$!jF*)VCm()&{T@WIMdqh*Wh2q9yTp#}*RtAa&q_3L
zt%U}f#K*Fh
z!qRoZ-0RR2$_XavhYDBcPF;J_hZ9+C_VP`ZUK#+rV?MLXuq#K#*c;hrfz5f#1+NFO
zKu0|wJc0;GGU!Pcf(l`lT3{F0eEpY|$sMxu?z5(V+*i?{*?0fXZ
zT`$ZuKytw%zV5ef0S$Z$R`6<{og1FyoIi+oF|mRb+n%Dro{BN+p|Kv*_{6g#E1U69dDmY&mD!2TkKS
zMmk_JSrp?*3B-&9futaSh|XlO13987iB-|5g3q-VGw{XqFxgQ8I(I#1x^zuwVL1CKD?4I2TtNamey
zUo}4#`(1ueCm(akxI%SSGzOAq&%xD>Z48uLFfgS*wN__wE&3nSnmO|#_??Kr593B*
z(?Un=K0we#vCtTa$fay|z^cRZojUdkL$9a
z<0mp^RFY{Jt7!`xyFEW3{HENPf8qQtXWBJe(lpnu?`4VW|ufhjo
z%~WaR4ue@Zrt)!08ShK5wZ!^iN+`^Prl?&uhN?3(3{!`)?;_d_-oTr3)06tL8_+^)
zsVHHdbp^QL^Vwv=wb23fcie3xc2HPyTL$wsA=;>IV;Mwt7U`?Bi6KqH!P;b1Tb_HV
zNe#|(dmU1debt}y7QN`kQG4PUo=dS@+AQJeTVvgCNFy+2>L)HFrqzx4c{+l*o|Fvd
z;|m5nEd%&n<-dQ6NmgHWaEaa)3dH-+t6ed9D!%SHZ#8N>v;zT_S>1n`uUR2ivN$^w
zO7d~Mn_m6i6D^>`3#=uy3yn01lXTG7*xL^m;1H*?#Hy_i^PtzuYn0?<;kvJ!MJ30
z2V)eekP);s{zO=KTuF0D*#&A;1R{wUF;J1I@pD`L*y|rz1cAW*fzn~uyWv6lxs`<<
zyR|sv+hFTZ*(reh|QnnjTq#Q=^=@E~aybj5b_h}8dr)uOsOZRvimD%!_J{&yF
z1mUI(qNGKAJe+OM-{YFSROsmwf16c5o|68!TfaX&F)6I9*u=?o&kK4uB9Rw>UwUAJ
zdlrHJdhGsUK;pMd(f~>ShW!XMD2|`U6nfq8DR(dl?a!u*o$SGt+kWHUohEEOE+}RD
zR&xsiZH`cs!QnsLKIME_RoC7UvriM81dfpbn^jIdDQHW=)ZYK|3dx6$VEVsotIYpP
zPCzEs|4Zy1R@;pIyRG`ZsGr)Jx=e=`)`AD-TP~7`DK>`$ATIlqH;1CB#7MKPG#)J2ejRT+&qyv?4~`8H|2REd02S3xK`U?9xzpd4G5{_n9QEvcd7-0?H*%-favKx}SyWal1!_f+G
z4%`Xk19P*;WJ1-NR(049#|wlQw$$A8p!eS@2J)J=BMsG8ygLi7Gg&BS{P9JCOs8E{
zo=h1q$|BuY0fKS|V0`omuc=3^yQ`1BA~+)wu<=2z(sx*yAA{j~JU|9Mmn4qyZ2yh0
zG%o=t6M&WV4ZUhg2$@I6GDI`FmO!{Q+Nhuee;-
zo2%OmofdAnuFgCfze{oT1RR$?sz+9Foc1jnuYa;e>-Xf%Sm
zQ_ncwynVyW&aqaOd}(!Sy`)L^>%F9O$U*@A*ku+ZcC$ZmN&T{!4%O0NH*bAhr%&t&
zVh)xX{BBRy%lWKM^>XzqL@+f!k7GD`Cf&KIrcO)a*3*x1>h4INuy(#U2k!3Ho2JBr
z0F4{`U^DQ<^EOOvB#Y@bVtHdVY>*6En_08oC~(YS;q~s^-}z~F5BWJ#upZ8}30eXu
zb3Q-XQ%hef^fP9weY6KBcM8WKX3~eWlqM0%BQLS<03x
zRs@D8(hHszJoN<}cr|*bpV#_oO%i9oVMt4q7B>634n{d6<=SgaI=DTKTgK&`#S=kX
zp-Z)d-b)4L1WDNjSc*qTsckWs40
zc6Q~EGzXJ-1sf>7J@0N=IUzf`riIxT3He)1L`~?Y28==V?+F4NYwbOkwg9rEqVPR7
zO-4r1e$Nj^;_Id`oxmjIdwb
zE&ah5j~Ff^#BpMN2_J)#QUXdqYLO2LM6O6X3Uq>=hP+jYHJVPa-+E^0{1Z&9IvVtM
zm}y)ZOKiLr37!b$kR|=$t&*Cmd$+do;?mc0(TdSe>)XTl7w9WmL(h?j=U)Lg7%HWU*YR+>Q4Ai*g_l=Ryr>fjHMH2Y#V~4K$lHx3RFpG+D+2DhF@^Ae1}B
z0}`kz=evBC;#eVyl6v(wUq`8r$x=m^CqTFYKCvH9tWB6S)PbK?a;$U&P1$!;2`qeZ
zOiH|OBjkIhi5atM=g`~F*;&;53$e7?X5$)5k)_mesC;F4kZC1zKEu$NWEXYCRBCUm
z5M1XbG5rxOZ)Xz4jETbl2!sNyF$3*x_NSYMSk4aEUh%{=O-ci6*1#Vk#sMNA1b$AO!(9)#w5C#|g!X861eb%l1K-i<
zv^Q%)R3S-nsRV}g2MO|}@{^eJIyzDXuo#IVa5T0c@ar;QvZEIOYWU$h7NXP;voO(q
zziq(VQXvyre|+oKSJEdO}?i)r8gq&kQejY68QwB
z@w442sK(3iE#+*>6Cv4|T_{}kGI=1`S^7a1vx+j&kg3X?@Z2@GlqxeI=&ON@i?(y1
zQtJpmsvRB}Aaph3{sA9P+4x^#?3tK2lI|=?0pj*fwtrc$f1fIz4yHs*FboPRVjA>+
zX|bmAhPI|eRP+pHmgWqSt|rd*cKX7Gp86(+E{60DCT7(CLKO_nP32u}jZB^X-`7%h
zX7-|%#x6um%>U4Dm0%czh3!3vwEx*D*f`jUSXr2LVHhNw>|Fs4e=+~u%l=}NOr7mr
zos3PLiT)|urt#Cr%JlCL|McaU{-W6a2PMb&xBrh>S_y`qpFzRUhOPVBG2z07ey$@Kx<)gmRD)sE@cJ=RkuY?lb8}-y+($<+dm2hgzL|Hvp6EAr
zV|OF=4~mMYs(bU~&3y7Xc~0grfTC-yG;)qFERkR;i>1Jmd^}EQ*u27L!-58b_Q4=!
zb1%!6P>3{;MC+A3h{{HopDy)XVm+lu4077>`4NahsVt4-!yg`Tk@R@%l59R{ZeKH1
zR7Zvwi@4A@xDNom`=CjiBwER)&Oyy8ZssVsXSfoNVpw_#l&jKfr5)m7XW{9oVrI9t
z{P=MOs2!GRM&O&UO&=jg=@Z&wlP5PZNz_dM`bP``NaR)IW3NP_J-0&=1Zf;I3itEv
zjH9SW{!FDN>P&w(D!kPpO8g?!QTjv_!iT3@RWTx(uquF^i~*XpePDhwY{38_Jcaq-
zNB!=Od=!}FSSCnxE@VH2J2pAyEHK!UG)R~X-$tPk9<@=@kE5JsD#QZbf+o2^QpG_O
zI0+x)jbR*($!E7}O6!_~r0g0Lvx5#`Q)DRhC7YC6N!_qvu2kuu?LCJ{YEs5W>y3As?DyI
zo0JJ%Kp~!Yl*YgfyOn2K@};EPG_k4;CvEr)vvYOxH8O_#Izjgna_7us0@6tf6B2?#{u^BN{!Sz+Eln?nk~C8QhVfop!;{Mw;xd<9aV*KZWtU$e{7aBxK)jXgARTD3{L8*{wAX8F@7d;h?TyS)}!nO5p;
z(@W~wmSMM7^(nhSWA_N;y+LY%h}Y7Q#Wx$kYPG9JlU_IG^ZT##M#W#-t@IpwS1i8S
zuCli*dDGmVJd{gh{Q1%SESmkjy(+fe*|7C<6f7>|U394KW(eq-U24lF+$!!GOpMB#
z7emoS>Kgp>z!MXKIwZ+Af}8&2M!=)j&$DAE3Y+S|kn#D&tdQ4=5UeFU^-}`tHyN=Z7z+Kp~)Xg&$?hSa~FCBZ1%Co(~TCzv}_kH=P*ta%=>j
zTR2^0aCgq?=|iJEo`2@-!+knV5RF$J$PN_`eLpIhm{D<>IfN|$G>*HkSa~Rm_mPqR
zv)puy*zIES%7hVZx{7rj_m+uD8{q5b1l(%b3Sse$)Q|+A6}at{daX1HihwCYwm@xv
z#26p2F&>acLf*AT8NPMrMKXmmWcZ1!uB`z{}rhIj&X#v%yjRu>VIDUeEY|X?jJb+*XpWUrndqo
z0w@vO=?e^?XJjbTddw?;JUsEOxmOlLdliikej*RMmz{y{foKqb=x|)mmTjS+^rhg$
zDjrrz&ic@J`xC@IsU}N3hV{KOk;Y-_o=(#RdfrIO>1$`El${XfVNag`4~yuD1}C<_ady7A)PPPQdS`b$>4Xg8zAl>K{P~
z~hTM8`FC`$V|0ke>sW0qe?cM
z{7BPJ{j!cPYVp3_KOeZSMsC9Mv;QQc>JP*w0r8!xBj
zK5uKcuOOaG3`Tzs=zovrpVai9$IT8M$g%A
z-$S|L?g5fY-o?e9cXAy2_IHE(pAsyvKaS)5^=WbbDv4~AAW9D*6zD|ltnXve7BZBWXS^8N?>K?pk8lFV1AAW9fw1)E9Yab+xn0
zJNkAwumKJ(V93Iw`p_0_7_u4-
zJUOv)z&md~{pYlMdGR-6*|<-*nP->W^)}lS7B0lI4(Ln^^(@0!P^hB#Jno{!c02W7
zT^CtU7EEE~l7Xw__#+X3^oc)d`kq#aa7xcb7v8TN{5UgN+dC_u61&
zVgJQgc;#E+K^fsrc)>9_dOT>fZ**dh-*dNPVoFfN0#|V+;=?dHh9C*ioGA3>{Ey~1
zei{(0;w4dPOR)usW9={@OevvWG=cz{Kwxhr-faFub)x`G@
z;1)rr(yWr~*H~23waBUg!K<6nk3|UA`iBF2k8zxcb(8;bAx*4k;
zKm3`D92;qayjHfMXLmiRAuugf2z+0|chRZ7?FXKW#%@jE6_o+TVt{NmYgwRaL$r}`
zzuQBfOjuBGKLAA`!#fGOJSAp+9h#(AT-N2RuTGoof=Hy`iJ2y`claYIaEP1{Da>4r
zFR~wJXH9tLJ}f6jPpC;&It`q)8CknWbWqPsW_^*t&{Fo&%`2p+wr9cb6wmn|FZ{a{
z^F7d7rC1|F!T|n$oJ=enzt|wHB4<3PAk_E)x`&YC{6+0}+{4=u%ouP$urMJPt-ZpK
zJW?|C=Jkkb!}diQRhV4*X6?7d%stn?hEa0PGV`ej^xm
z4|_M6p_MadY*sMOB};iz1ttJBrv2yuxU%!|NLs&qaU|@S&NTc;EAm$ptip_>p~yfI
zSt2Egykqh*X2Dpnf(pfgD7-Os7HMPrMt>~-h^l-9a
ztwA>rP>0kI1q`TuZG+$V1?ou|0Ood=%CVMTv<$-#cxW&*>ZvEb;83cJsCsd;wL*eB
z*w@dL;H9HAulrw`gc^%m)K-t1P+qs+L1P7??}y5xg#{h>A(#?hQGgnH-me&{tJ9Yw
zV}F$lfm8yUv$8>9U=gi54!~VjA7c0ePrWEqnC8o1J6T>0bsr&Os|a8
z#tW08U3)k7gpyeKBIn7Uksz56vWPyMrz!D~tA<|yi4oy>m#Z^%V8*Rj1W&$U>#k72
zK&WaGcq{Kd(e440Yn*XYfw
zBy9u!jvmvLEdoWCvJTm0o97zc!8J{}`-5tg2POJB%
zGSN95Tk)fYgI-(kvcqh;VXZvgDE&x-ibkH=MuG5E&_`56AlfZI@a-%PM+J&Zjf&WavY)-K
zHkg!dX5TM6H?lD%{CQf4LZ5VB5|DaeN3xy0CJ4)TYhRr#Z}pH$sTOw$&OHVU+WE2>
zA2sc{MH8vBb}B=0Icr{mhx(-p-|!`X-%2_|XP7!f6s@%5Bb3jQAgaBW#^8f-frx7Z
z@K+MlMKdxaIF!DrGlpxoA@`L{^oIRgk@r#}{Do*_-j8*I-!su`N`6*O))L`mBa(aC
zHpC=OOq2$qFSvM=HccCSSa8jL{_zu-4k!;hrHtj@q=@}rF@v2|i9w;J0SLvkm>gzb
zEE$F}SVw^Goo1PsnIo8f_thjHf)Ge9E2Bh#C5D}cHR#>bZ>SC9TmS=uC53EagEZAf
zP{#qv-oj1)o#`B|s=pTG=92oU}w5ad78T!$z5EE}?
zZ@)LY&Z$QB0P^;rQ1(VhF}9px#>v#%-fRy+4FU6$nN)T7;k4idWZE%idJ5tb#J{K~
zBipa{5{=Bj2i1>5TAPxkde(O3G(|HL+@@OsF1cFW%=Gp(h^;n<#Q^PHmv`hUoKbC
z577BHnUYTa^;_QGy8b`gVt<8Qrhm~TS;cm1k$1ZE4NYeY=)}@J8&*m8-d1=NDz=X>
zkDjuG4C8)C8g3g>8?kFIPE{Cy&(CGQC&)@Owe>)yC%liyx1{~Qmc<5gSuMcAS-3}W!l`_>=vlwDu
zfiV>!x3a9v|K05pO@IocWG1&*L?A58{%|}LKMb4SN%0?|_m|`SgVhWyzjy&}Ja>FQ
z!@Ff>`D2bmJ|^tl7*ixYyq@tfjjH)dh5>z~cTTG-<{>mEhkl#g-Ztd^?z)^kInmQ7
zLXSF5Ih{v6urB&kmloAl$2C~q&yb~kTbdaWESr#iKqZac44d8G7!H(F3$1CB%
zO8bWD6#zJHTN?Tmnvui_SSGO&*=!EC5ud2^Tlb^`w-;cm8%ORG4#9szD${#r;D2X5
zL=`0+z=+`5AE`7x)Wj{UnWB4C1Dx`
zsL?nE6jFXs`rh?X+n4S*sY#AIDn6G4ZQC}HWcvtR%0<=+r(%ICHxFvOi7u2YjWplI
zz~162nTR;Y*@$)cF+z^mWq9|slHo*=G=AY3?I>f&*KGy$6)|goSg^CL8QNsT-$%4{<
zq4CUil}K9a=9Ln&3nJ3ukJmy;4egPJt4*}s5f>BWh3&=++F>4;#tagi3M>@Jyu(IH
za|>P-vpICJo9gIij^qTv9MrNao=>~iS>DTpq(GXoPP~c~Y6D2BpxdkD*vnj%*G)$)
z`4u%<)z8PB5=8|{OOlbN_$X7oHBf3(9j=O^)=2F6CCIXD)9?$&8&2dVDn2kYM&H`k
z#fdx5jILyI(x$DPmz>FXA!?APDSoHSI$+|XD7(>-^FFWtFuRvur23rsvKg0rP29Ma
zw|JyZrkEDBSVA4!!t)lHbM*=uovX?BJB9tju>J+JOut$dt2}qzdl)(445Ew0jPGb^DmAk8zYF){~pmVe1fulYk_aFojjyo0kpAf=iq0N
zzHz|}P&|fkCk$jA9?#pVh0=Cx(NAR9qC8L*u{`g$l$ANUSq}QS#j>{?x)5k;7(lI4xE1LwSe9TL~
zd%XW)LH|k%vHd#VV`gercqc(Dz$n2O?_A6j;!iHtRTkDNM29H>H!ucV59ixJmgyg=
zEyN<2=&ylPWJ+#|B*_6voB{zf+JOH3cQ^ls4gJmDQ_PVe5&lsk+pkI#jcs)XV}uiS
zRP{q~^W{3m<`=JjDStV5GW1wlDxxn#J5=OhH58HZ1UETGs$f8gly>hOs)Rx4O)7@^*f@%
z`d_nLtp9?XIO%v$LMV}kY=08ppL;xM>8cKIXDq2QY%VFKFM(1d#V9lfXzmOzBiED0
zO-GG*6%4%DK=viVwzq
zl%M}>p&0{ok%TaTTYcq3V6*$)WDtp3FsbGm@+=Ztl2BH#;}UE27?3d3HMWS}BFFZ8
zOoj#N?~?~Db1j?e94xTAPwC+ReSc-&ej`xR+yPx^1+_PV6p0|%E>{JxKl!=&#?_y7
zunMOh?Rv#6W7)n-&o~<1>MMGVws6CX`Q-%e!^P-(nTW%@xA1{Z&G?33@$^rElLthF
zaMREf3faG+*MC^?zcBf?MwpDGMu#aw133XQn`{R|oDBh#;||qAx$~$Zy?O|gvxn7th#f;MjmYJ#?_rtt2Te=+;YFqAlKl2ZGYbQ{(5CDFy
zHsMU6{!$a6!!RNiTyGj`1TmbHEX3OvhXd
z;gX5voRzNe^RyDUX`!f#(&hEG64
z!j*oL6NY1C{SOoSm-92S{kh8ipS2RvSZS1g!uJe*rjYohID}WWQ=8O9_ZK4Bdf$>-
z0SH$Lu(C#Q)LC$PYw0#vK5gi!A-6-e_p0$dFlS%2!d@1v@XpJ9_~ybL0yA-P8UjtM#wA^}Z?Qe={Ok1!)@*BDgij
zaGfnlQX=3f;YHPh*R|Syl{2bfEPw>!QJpIOiL#+{*y(kVY|0puJ9_DN`L#DiI1rn+
znGwN;=Ey9=eUz_*eWX0!Sx9gxmS_lKxaDz8gN6p8cC+ZP^>Z+QS+xO4lYX4mT_oFrAj!ON}2dG;LoMaX<;V%HQccb3FVbNeoK8w)o
zF%Iwy_+P9&V&gwmPK?$=4$-eXV5^;$L5~X!MKCw?Qvss;sYYt`Pl(Gs-INRa%Se}Z
zBi{H19Wnhex=74==${TMNZh&#MOS%E*BZ1Dlg-7G(2@hr%%{fpq)%wnhj*U{-_sJ@
zLKf=2`3$Z@?sU;i)+QpB2A=~8hX(KWmt%^+bPtD23OPWY(y01CrP3#?@(Eu@j)+Q_
zkcz90q>L$=713RfM|J93JF+C1v>!ynC())(`qJ)PQ<*9zx!sddi{wNfu|1o>;Inp*
zS1i+l#F{yKsmZ+9Ts$1EJJU50N6RE6(C6D=c(o$O9;%u5UZ$3UY2)Hp*XHMYl
zV?Cz0ls$aR;b~ARJJimthE4PqXynL}RABV3HuNjVbO8`&Vr}tZi>FbN*c;lwoIYR!
znRcnS4DEt%0JFRQ&W!#m4`TlnN@QbY5rl}40v2fjSdi?EI|b&*cq
ztwM>N!I6llf*Du00v%LZWSF9?Y8JVp#(NR*>Ff!(_ntfD$`Aw%mhHdsqxa>{UtmNu
zW*q)~VkLZ`vW;?$%~uw(n7#a3eZr6~DD)^u%zecGRe5vGHZyoRo8Qm=%C=Zxlte?&
z=Aop9coZAbagDBE&Y(CDq$%@3*kCGkY$Tbf8F#x$HlWffN##p`3l9WhQ;A6hln+Ol
zwZSpX6iVu
z>f$XqiobOpl%2fIt8VJ)eWQ!~-5~xOf8zMnV_0Qd|Hzr0r1>LH6fB=6NPj>5265UV
z$mLC#h-sq2)c-9DT?DwRV7}L#Xsf&MPlof873dztWJ7j-hWP
zF(#c{S|mf8ppv93?c?Nh&h*62QY|)JbGJVsyf0bH!PGp)3%8f1T{&ZEyHEiOR4Z}5
zfU-t5lq@|MZyZv2hU98Ivpq<)0+@JQ?L%O3H}txa49$8-4$j+#G;OE
z-P|LHh*1j4%K(eoO1!&aiOL)02Af-nnAROi$G)HwVsTOH%qr~9;CCco(YJ^ZBTxrP_}@h;rBxUKSTCgb*3G-=^-!=Ek@;QgaBz1RTs;|^4ugwX
zX2j}BgU&~muW)_+_5N|b<)WRSHOi>h|Q|DLBi+!
z&i<9xBNs{;B~`Nfhh`r4AFtEoY4t3N3~_Xi{-2Z06KGkQC$zfU*5#IwB(I2)>a-Af
zT>z5t3|-z9#|C;z%`a|D{E86pPLH(OKk@M%&hWVx)rXs9w}`ny)Z!ej^tYg(_=va{
zX-4k)pq=938$QnZ5AdT^1jCj5@YZ9$E3bH1yLfBYe$5+1R45v|xvYV0a|3t^J?9+)->@zr&Mc1F
z5SK?Cnm2Dt)j2kOG+oAYMwB;%Ns;BUB#9{pgdyrkTw%2_&+1hId|RJx=pA66G&F~}
zqVQ1~y~<|x?$=~Gsnnm)4YYzA(>{PBjGr8+Lr-z0SFihehlih2U}^#vXjHOmSstGp
z2#@y?+ak;GKtSr1w(GhF+$1!IoEdJPM>sj9Z?4EVp2X$iCj9TsW&L1H5qlX-Vm=qv
zGJ8>1YVrW1QVB)H-
z$&tAkl254?ZlDeg2q31?C4Z$;mcP}
zWEkQPy-_g*37of>0U$LLg2zh$!#?*q0z>1H$dX;|tawY_*5Gwx(f5W{Rj#M%5)jaq
zcaRcv#H8&43^-4^12rttXJ)+>BhbB?UFU&JQlQP%{01L5d)?z^b;6_fUR9h1oBSo_
zjB+#cg`unc=nH|(Vl^X^(17OWbHf5kD_B-~u6Wrh=HzF9(HgOG7*h@J;TxMHsE>xbdR`sy`B
zV2z^9fnWJyC6YUxyeGf-exqF@Z!^y
zCE|g=afp7GQ`Cnqi0XYDi&3W=^vBkMYe#Sr78lm%OgfmkN*q@%7GO*x1Zu(Y%aBX&
zKc@z@D!<;V4qbD4?rHF9HwU-N}3
z#0b35__=D++re69NZ+iq86BbLd#MwxP)az$>QSb2|am*1=z&hheT+8|SDQ>NO}r?+j*c$^&X>-)&e0w<$NQ
z7by)Dj#rqAd-bBs?F;8VJ^KC$`EuEn%W*@>)t+~RaH(d3*J+Px+pv?MP?{kdf2%S2
zk_g|qJ+S)ZgU(_gsI_4iq}S5ib$u{ilNL~M`gHSVmIP4;K*lpM7SQvFi87FBK!)6n
z$o!+PW;a5IbPO+I{{0J56kVUOj3shm0}uJtpvT5JV?=*^v{QC(cR7uFw;p^hOp7tp
zU3NnFB4Ziu!w%Q25Kq_;oj{EaWw`EJOt18$|I&MXWYH%Yii%GOPy;lc!Ao=slig9r
zE1X+eTj!LE0K~X@ujt$B!uh#usIQVHYD7eQr3x$UjeWSE5w0pO9yX=FYuX9BhdMFb
zE-bF)b8j7%R5>4>tzg#ZqtM#7WoF~R8NfS+!y2?zmJS@`>BdmcLQaMv$itw-tSjkO
ze*;JqeOgMAH}jPv!vi;>9OS4h4HPS1H&$QhjA5z@03^0PP2VxNf&Ach0-Db7IGbAE
z8e3Im)pc`?(EaEdAbLb#5Y*YV;c96SP{D!utn*4Qxu3cOr+fAjnhSK8KjdM(QQOME
zLb)G=o&gK`+pHcX_$$mttfM2(!o7afQZ(6_9o4XdyNH8b9`1({Vcr-VJ!2~Ib%7TPic0|Wa
z`o3;MOrev^#A+$u!A9m%oHmImk~EXAm&Pe~v^Yd8tC&}D#&$_FCrE;P)VQZXF@U5;
z3Fik2fr2};o}F3GKF)qnT^Jk&iE@sps;KC#I}*!zl(S9iG)(Q;b(CM-NKNl%_)M#b
zBm%f969DT?c;l$?@3yeh7xC^2sXD1-#34`2{_tV`dm>(*4p;AI&#IMv#{#nYZC)nmPqI8td?tt!h
zYcJ4s{?_M#=(w@X?TaCM+rf=zU7qg*%!3otqpn0r@sl?1%^ZKQ8!V{iQ_(#w*|9Sk
zlF-;saxBl{d&-!ibH=Lq?kveJU&-r24cvtzCQ^hrd&I6`T6ff$UBV+W^Y-InXY1BwMFpTq;+
zmu^qH0U9~n5QNme)b^0-^|bsvn?EttDG(KYL;u)QYT-dp!I&7>|2l_eLrXegjSZ#i
zS>3KPMB;=f60$d>!b?M)j9YT@+@PSGH&T}_8Y)VoSRy#kwDRW<7AOT`&Y6I@f%+-Y
zLc*HolXJukc!lLw*P5GZ^<^FYiBFv`VPJBB#5q5vthV*hf-X^ALWMh!zyrDX5RpvQ
z)4pV`jfPAvm_*9rSEv*No;rUBQJg3tjup*-hCPO+rSvBfFY%*C9F?rl+XC_@sv?8o
zwqx&-ya!TKhx?|C3cbmRL*q-|CK`S8a}?p22i%tdghg{^=yd~u*w
zWIEBta$2<+AaEYvVbjs6GBmHKeBU4ou|PYp&^UQ7UdrZW2)JvW;j@X=r?!IzD8|bq
zMOP!mg91Xl>Mun(2ILtkM=uUi0bx!l+96-fiL2ngB&`LyJUU2`*QP(`LeiFU!qi-}
zBof&TBFnpnB7NkmFZUK`X!q>oK;e
zk;v-$rD~ZNB6Z)^yy5d;V5q^|RNQ;dNMim-npkX3C~UMHqtXkSQeF|zH?W0Uj^mmA
z!^M@
zI!lT^Ut3XHB%BP|o@h7|HkQRENuHvUx*${FzQ|(yIA||h+xKCUR5u}4BUVq-3*WO1
znQ7DNhGm@y0qC&fUxzfZsFu&fT41YRONtsM=s;;-MS2Y!C1{;#Te2>_XL>ahu1Iuw
zo*tFX)}-#ROb8f`Zz2V(Wy~;n_4JpBLxvSv1ma$(Mha3
z`j~g_y3I$lbnMkIv$j}V5tXO5LmC+dw9%4SsT~F&07dmB%1Kk!Upwj~F7u2KAWbSM
z^3;c-r6ek`B;48ppA3fkef%BxZ1S8QNO%H+)44zG5uJj!0bVE`Q^jUJe4Z2vTO2M{
zvxKZ4Emz?0vjM~K`YN+mldif}f*%)zze4B5>PGLNIGFZ2$XXW_T~wORhup8AX?n!d
zH7=r_41nBMfTQKq3B^!U1JTe_vFS#&m4&9Plqr{`q>kETDaz1-*+|bxbdw`x5|lN(
z@Vm@2&&`HlwEU`sB}6kSa(1;phHXI@1TS`@4b<#$Xiuqsrs|>aMM##=jHeO?8$kie
z*Ar$QuZd7On&PTsg_iHTzPf)5@|}d9e>lM&G@z_5NDiBLDKNeU1^E(pFdWYprVfw6I^BqoA!#V)3+gH*f=
zOh2~-uMpqv8rt74?vyNS4V%21(m6N-b4=1`)GKZl2c{txnGgqUWH-bBIukf!-nu(k
z>ZWQCyPsPpJu(DN;>gLk$YRs;R^3Q$>epAh>tYnwgE8sa5ACuN(SZ-}!t;n&!~?MK
z7CV^R6;p%4!MLq(Dd^#{N3?uW@AH+qL4X3oKumC>Po(8urCsv4Yc0!pFFM63@eGGQ
z6A$QMuPZosc2oy5Ynb~C<~zE^oG6c>iL9+kp)^}{_7gHLE``%Hc3O@Z5nj>M{8P%6
z&c_B6TlH}bGh3^fRcB8+TnHt3uTX#n$gCq^W+l3-;TeFbs>t5eEg_TjN_Ss#x=(Mh
zujxHOD4Vp%U;=l8V-i|F#mG2Jk%E-^)<5k|_3ed$WmkZ64Purxy*;m^>%%TDkIM6r
zW~yD3HR&@zz$G9j0&`!-n4Vf?x9)7%u|Ty`Rw3`(9e*cs#8)Wy8WL8+IWK_IcXyES
z0DNyxAhp4#&KmTh2q9sy)eYqqyr
z8c}ZiT|WR^l;oahOpZ~IW&O@Mcim2zFD}WJWb%v6k^7C#t;{QmUuWvKn-nME;c&<3
z`1GBX@6Kd@oml~%opQ&-Z$F7PbKPHRs>@_WlB<+f
z@b@xh%~d_+0w*}*M@qjq
z!j7f%0WO7Y`uq;#TnHFrl}e32e}l
z{SDA&v!54wstCA&Uno$Hg?ug#{5iiUT(n&8+Snap9GF-7G;JK%H`CET-tjOJQ?CvH
za;)7(`yHA1O)iO%oiU{+8#=IhL8i0Du-
zgIc0Iwcx2sE=s5@5$(|cI5;{cCa_|=_e!lSZq@1-Oolzu>-?ZA1y~aOjxMZEPgcTO
zyu^BV0Xp(yQ6pJHAqD#hqjU=q{fgaR1kaetBeZXM?Ayn8sFRLHKI9ew?qkAFvZ(l!
zEj9>Bb}q($N-b8=j9EUnhFY>#CZ2Y+a3|4T>8_ed^AY&hRrO{92!Qk}N#prG>|rlJ
zb@e~~2vBO5U6c8c&UUv2woJdomz-qT|6H`T90ax-78c>YP`)V`Gqg%jA~hm2sTyZC
z3yF!v@>nnXR0F7Pcf&}0n9dv3%4w&|PLfYAzrQ)GD_1$&D?7PY;hY&@Kjz9J$OalO
zq>LC>zj`5?AS#stRLMJA@TLq6{xs(Ml-OJ~cB=nq%0+xhb{{IvF)^R%se*zQJeTpk
z03aS(59L^IWMbZR{D`+b7?SBxJw8itx#Z1y#6Urvd;bH?9c3avFB=$oA}tHWPXIYG
zkEQTq-=#Hz)*COzLWm+WR2U$`H0_fC0auosa%bfbD$Y|mpo{Kg-6moKAD#WI@W~bW
z!;DgwK1PGJ*2JtEvBsV}rqNDCpI;1Xs?#rDoYGAD)tPHc-E@j+pdNO
z+G!L=t;M%UPL_46GEaJ=0m!q{eBF=jr8`_xsuc&I0$x|i;i}afRyZOOP;RtYx!)U$YE
zj{IK2z*tg2ffvM}DD_q+A67f67|irc*tyU143x0;E1;(5TMfNazBDWK-|&y
zfBIp)fKbdq3U$PMwhaTK#>k1(G>#4i6ByVreS7(86hR*f)pKQ#V_!i}Dkg(DN+nz;
z%}a@t^iy9?p)a?6rr%3Y63T?;S!Sa>)tVLKP#wmbjW^E5R@x6sBK^I(6b_kye4eih
z2@M$^4{!(zRbA+mig`XbgYlUGWU8lf+6)K5Pg#<+El8o&LoD?(*7pIewdrdIg)N=NPn
zzTP%kTqZeUj9HuMK^#t^@BswSuSbZVxpgh>bOEd0HN>~O!8
zWif%Ia@SQZnA^h|#gKV-*kYO(EHP(6qTs;QmS8-fCJ9fh;Xu$3vL&|wK|zOPxVJ1?
zHUrk+q;SkuIQ%rVN?V#vv?hs+G8z)TCU0Dd^clDq!VEX5Zbq7OwFcp4+@^V89;gDD@?t7leuAuu$0ilBA-56+UO`*iqL33Gz~)eENiZ=
zueoEUW|nVip5X5&u*HBeg|jhevS&E^RSFP*0+D-DC$}bQL$EgI<8p5?*Js_R?$lW>
z6R2hlmhaPOu`0UtwyF`|Rv{x>p;>`qO%g|Kr>mmhiIU{(?JuJAU|!&{+OkTojVlP%
z4@a}aJH%7sZc<}&x^7fWf#ph{Qrn(=`|;eFS=${M?v4TkLLEZl?i>@%m+r*HE(|b;
z5OABvJ+`abrm?^H!nOa!gLA)yQ_IS^W#YB-*4*^KzCthH^AURa`Q7m?om)#W{x+!V
zd}r18PmL=AD^Fp`RTWJyCzwj;{>vrUJ}8vOy!Hn$d_~X%1lvVa`SIX=+;zW`oifU&
zL^}@jSZje4_q5Ve@u5IRW
z3}KYA)W(pk
z&O8p|V|QmaKXFC-H0Q>AV(V7gRQXusMC2pACi(-vjs4pT{yo7%3!n@9_0GP!^Xehs
zg_iTGy5{xO;Mw%~6qDx)R5r(!ch-lL7iC!S=Wxg!NsK+5kNrI0NuKw$ACy-+j-6laoI(~TezLiI=b9Cz#4RLIYz~k#{3cxocW!wSmR*U=ms{5(a=hq
z+H*%f&3&W#o+o;xDRDeb>)})Qsy5HumIK#2R6aDs$mqbKShU?|3-Ow+it!JHE@dd=
zh*LdViA%71m*4=Rr6^uX4c49Juq~|=y@TA~87Do=^0%QJ-6+!bCnl)R$Ua!_Szd5X
z8zsv6O|6|fN{d^08j6{34rGgS!Z4fM;;lBmp{z1CVaH@XJ4=yGk`#AB4$W*P;lwDp
zs982;q@`pd&K1h<1K4a?EGdWN%&sD>KQWO6I}G@2WSjx?Bs%oIY`kV$@f^SI-cvBm
zXKbFo0({o_pNDJSCS6lJm6X_OM2e;y!|*p$*JhvRyRd5Y0=%4-+OafJJXWeg9^!mD
zKV9E$Tc2^D?x3P#iPM~|ga$gx&JOc_Ps^#I%6MdPsQVKAqk%e{mbW{iW@z`QyI}vo
ztqtfxj8GkLP~W>*)FZ{)4G)ETCi7jM6kh+@E-G4czdn*Uv;K!3-0k4~;Omla$;0uM
z<_)r(&KTxtrLS5ia5{L_mjtvUm^4P%*!Nk!;64z*!3RFAa3A@mLr`#d0y`RO4jmT5
z)P=!zh&^@X&|rq@f542T0TPD_lVbS|K3tq!z-q0lAPg-YQU
zhGbi8%`v)n5GUPAxL2%N(G1rR>j2cf4N--uRM%F!fD6Q0xQ@e`W!tkFSBET_UhOyM
zdJiULw1cVtwTh|nC(6(B^?@IE)PB5pP{C95f=^Sr7ZN9IIS5a3
z?eThm{MpVmLr%KGhcH1S9xaLKWSFUw@1mlcSv|O^9czo0DqXJ199mvnTOa(}n#I4H
zy|`A9i#W{-QF2rvGYM!u0?s
zS*pfrP>0Xumj)=EwLYNJVXYp)HbiRLEfom@jFJ>kPM|(u*MbFRx@aGAmFqi4+hq^ak<`ix6~XmhRFoqHJ*ZkO1!!Pvn51t-7#ahtTrvr$#Js4M
zvsgwr%#!0$iwT-CEhSpd4SD83f@eZs%m-#H*fgh`Zdod+vnQOR}+3@*T=c_%Vvp+JEXIJ81q_r)9U^jmQ*1RA;nEgO{q>3w&Fs}LC7
z#7f=W$BLVqyiKC;6|aWC1VCe
zymW3(@gjM^C^D*`AQzqEOf0;ISIUnO9)
zxebHA7&bA$ae#4YJuE1VH*poza|i@|=N$Kow_-}F0VRr8eaU}nqRR#+SaBLREyv@x
zZi5fBDp;7I#SjG&DCR5tGyqY_BOKl1IDw~*VN|~jMW4OMAIUH^>p)CDE)gmb!}pY<
z&5Dm(o>nDN2c}7VeM#ga+Ipl89bsCT`icoUcZN>VH}D6p)4tqPQCS?beNGE=x&TGr
z9W?#q*HMwSc6>4`f5<{m=hP5>peEUJ?ay$PIQ3Oz%b>XijEs1yY=AT!`zSasIYG_&
zDI(Pow6%;(NlFLFOGV9sbirKr#jSE1wMD8$m{ey)1RE-WaIOR7ct0`@!SB$btp+YV
zh_THS60noXl5l}ja1|5dt;74H;-yh2wWf8+G*Tmm`7o-({A`d#C710~dQ2UIXjZHe
zB6il&*T_i+p_5F$o`8wG*#g3k;C+qRBO=QouvP}eND82}t3ftaS%-|y7^ny^=ZyMX
zALi_Up@ak`IA1_yhFKQTYw$;Y7>piQ8P6}1Y8dSjqFYj=mex|1OJ3Kwk9y+xixdyn
z<>r38Ha%e^6Zpynt}NT{n*}FS3kM2(ojEtwuO1uNNYY4rk`G9noR5@?Et-c$n}F)c
z2lfg@3BTe?tmestKg~=jAg4G3fjx@nt}ySQ4+1ttK@2k^$(W!QLImt$RYZsD07HP%I@VEC?O)pO}68RD%b@L0nwpa+uL+
zHb)}_dp~+n9S@v=@NI^oP(kxWz`@_U6kP)=Kph0G8v==Let^#K#Y5GIH}kC**Pqg0d5WWjBLjmAzPS;q+l(gDY5Nb
zh!sw8>+f|-B!$_gL5){Nmep|pQAtT!h5T`*n|u~eFZ}86u9G4Qy4PpLZU_+S&2jUl
zZYf7mIleHjm_R@dV_ZHTgBdpzMBoC#NWU;!EX|V#tu+jmcI(b0z*Fs!Ry1{Lf=os`
zX9pNcpsKj=)XDc~YR_S?md$@p)F~)&0GgN$TwHMyT`ebu-dszpunT{nAAuMoLdsgG
z7l{YC!>9=E!l8NdzoWkh?*q`cwKXxZkowXR1fR+CFCT=A?0o};q{@Ha4QB5YqfqfK
z^IXRy;&D-EKrz#Q7>rSjmS(Y|Xm>FQj>owz<9;wS2z|w45f0{2)q0~CS+DVvpa29T
zfkR&pE$1TxVHV+gs=t{kS&^^-eS;Yn9IwmO3=OL&8HgX&>IE=*PXMTU*yPlJT?vKG
z`naP}E8(rQ{0G+3&@zqbKH~J~E<7e+h&U^X;H4qS_J565%~Zo`+KmJqwD|ITkB4H`
zz}upOB%x>=RYpyOg2$}bvX{5ir~+pr%fRXAr;VTtqOMoNLP1_DoB
zdafc^tJs&WGy~e3C#XGdnTz=f4jn{n;j1$xhTF
zHt1H4NF=neW+RAo
zu+yR#C%S7Q7=M;Mb~KFI-WW>11hyjt5*~NmIKMOwhW02b+2ht3JMbD1Yhtk^@Pld6
zohC=`2NO>MaWiVe`OzKV)#H!kr_DJZm`KGZW4ZV2qaMC!gPH*+z2Y-&RUCY>XD&L43g--uR4y#%)J(Ea5t33erx7Cf?arRGC)e=rzW
z%_Z1?^7Kx5HD@EYbcA7&D(kpmAy}Aw@IOO*`tGph*&
z5d11Nnyy+k0xQ&nWjI2YW50DjFT8Tb<{UAAG{mf!Um=z+_z&EsNBvi3A)J+UQdi8)
zJ910I(lilsuq3#ZmBe-qeB0EwsW%$vmID|9#Zx7#xts}^QY)CRpNX~55Hg)!R8=uio!*u0qraR
zf_YV-{1k`jGhO;_1p!hQzY1Lu3DQ4dulG3aF~RjFFK%$Ay4*)w_ED%DF~R^ZU*F$I6x%vlrkqhPPH3mU4sa<@xrb
z^4#KR?CSRYo#3(C7r=jPFw{z^yH`zcIP-Cf`~G%M3lu<%<46(c#Q`GFr~9Rfzyj>!
zHhTN_7lB2~=8sE+n+Bi)QriFX7a*-DaBU3`*B{vHJ~wX9AAs}d?FZP_xU|ci+>!tQ
z#yxwWi&l_^P3qkNOkd>*xZU;I=lLl^cQ?+JTy^zXTc}rtc$?4AV}zjE?GZv~N;o#n
z)GJPLkfY)Ii&7H&XZPT3UQ
zCDBO0dl>402QFDX>$;y@
z1*JjrhUoLlFSLWTR{}AuJL-s(g-T-RllySBN6>f-N5F_o$+){L`hI0IPOYbJ1ElBS
zhlP;2?`%Ga7GNOIN0$eqfJf^P|0Ct}u}6T+|66}I%KMn_)qD%V<=q!3l&}5P&{7up
zcOQkmBUPoJ$I}F#d#9IxpWUzKpZ#fDfcO57gBF8xf{(|w?rh)9iS0p)@%Rrs|8D)x
z&*R%DSimx3-g(FOErZ=<>u?uWCnjFc1Nl&~x9P3-$27Uchsj+=FX$CelD!$CS{m^{VV+@PqZw
zIJcsE>SP&@7Cf&txASXunc~M?Wp8`-S}zC5Wvgi`(9aKdgYJqHmuK_Kc2mm<%&(Bu
zGH$l7QU)(Sj*q8?m4ql>b_OAp7!L0(<;5(puW9=L9xSB_KD14{%qSZAd%h67C_7}*MnCU4W-fM?KUJeU*Vb`f`(2H0xu!=aYlY3R
z+m$(Iw76}8R7)ZIygBO{lx{W^J7Ymn)#P#c(p_*a)yK+mRNkH2(;rGWtf!ZYU<%yd
zD3Gg!>{j+=Y-0@S$IxOO21^+|$;
zmoIZRBsmp86;&ED0Rn$=?XoAOE>;WpqNm|n*0a|1$`{JXRET(nz?G`N1mZRU6Qj!VoYe$(vY-ES`j
zmg;8sfscLWKx%E7(}?zk3lVc%c~Z21B;X?Paaj@L(;_(PbdCF7Vgd}=maVfxIh6%6
zmGe^ti^=gIa(kHYNNn*p@Wvs@c?KY!m4qLH)5?#og#CA}hU0FoBdQPaJ*#P5l_=yQ
z8jxMdN2jKD%6Mlt@9o$Gs52-YJ|BVCh=W?rYRei9#cyhcR2_Kr$(zvHns^xinQ>VQ
z^m#R5i^d}}#*9>VktO7rOw~QJ)U@YTY6?&GYWW&}X?Tg0c4&2Gb5q-}nLceZCrm0{
z$zKbM%E$FkrHsMqT(H{ic_aD~`?i$J`_%42r3o6d<$I||U!P~rh3CIVyL5EAv|{XY
z!xr_iQR4*|a&kfGiF&&q`F*?rxcW&;7Eer9Nss7Htn0Bke6rIPcYW|i-PyMLEHVSB
z3H6EQ0-gI%gyMUxRX#2{bCG>cOgj{A&D;0yHz^3q(l7Be?l-?W##$wNZ&!0xR4A;m
z(0?BRSt^MH`2|FTx0D>f{sqhYs>j`;Q#S|yU}e`Qk;Bs%0QrqO151n?AS1K`At#uODh@g1_}Gb#(eY>f6IiR+%#&aNS(-#{AxO2)2m_y2_UW1Zc
zC}-G*lPum48vuXFA*OeyBXDrzIp%&;EAobI4c+kqElq=2QFi;tq$*K|5k&^y(-U472e5DA9$*AHZUZR?m`sU&QRLz0{HBPq%rMJE2cB
zSaYzRMOw8A!6ShKh`xeY$>dolRFU;Mt*P21q$T-yDp&zI(I|BEOIs|>T-c68m)7>1
zlZSqv^B1W2x`=9FSeIV)gxRi%F7sie9rNthy@Z#0Ct-IyUo&dAd4bt1Ok+XzP)+AJ1ED{Z`US?;BC~Vy
zj#0TBZ>h)&&}wKx64$|Yqq%(TYxgN#UR^F0Nw-LR;L!AS?&BGxGvnu&?AA7m9r=>2
z>aC#6l@LZO@-iDMEnZp$EZP@@D)i$rH#w8(6XMdRU9QQ?*FyoyfS;lxghaGxA4Ci8
z!ak3b*SuOWQJKDgHmZK~BonvfI03YSeWhi!CMn=^fay(M6|5Hfqci?JZ#-un#<(M&
z&dj6BR5qqxTAncSbkx>Oo(806+gGZsUvhl?rWyX{WHI66*yiu72Ix`(I0PRS>{cb?
zE>1b^7s7O@Mc;d0S!<-|#$8HsB(?bzH89=P8E^`W3e0w`5;@kAKysa?CAsC{Tz*|x
z;=GhP0d(ce51?f%v+)QpX26cmNDBD6WHnhpgqSi92OZ~^PF6-6jfQTHpN+MXC%`@=
zlr%c2VK;rMk=(9Lj&cd6{TQB*iECpq?%)|L%z8U_Ytr4{;BN!EO@yv|(TP7(#HOA7
zL@h3T`e6NJ$^2gi%QrD<5%EqIhi$Rs!Z(3cold!s$Z
z_F_Fc(CTVEJ9JH&w%DUM812korl5;c5;J}!adrXTW1!8XUj!O{@(cZL;D#d1to|aE6(N^UYyy&8Nb0k-0gNwiZp)jHkVK8KC~Cbe>Mw9nfO;W}tLFG^KPd9-4JD-7IL+Aw*H;
z2HNNS(@u~RX$>CA*`@qw1&r2l-~hNVO2dw_`CBf=UrZ^Ef~
ze~}&s_FLkqc+kv?@2!l0SXd1X(-iP?_?E@q?)!KtNjv((@Arr$rLpoJKLBJQ5;#Qr
z%UsQ^T&on*FXP>@eudvj3cP)#r{*qEMcQrB)=U9#OpKwrNwKg7BHS%_4|e!JeH;~h
zbt3LNs%(%?5eY8nnQB5Q#IqVVwA!yS8W3B8YU$g6JD_nYv^0i;ehGbWhyK{2MznSx
z>rRIR-wXM)xyj$gr$vPMh!XcygaWkij@+nmQJoyLRz%^|7uZ|Pw6GrOVBgWfzpRxC
zeS#jCr?>o_`4-ndlChZn@8rC>{s325{CT$Wzs!06y^Hsc_=Eq`&NtVeyn_E)&NtUT
zVhH{_I^SGu|HYhd$^ZY(_usqo|9~No|GAs}LGAtn0G?)X`$qxgpU(Gxtnr{Nla{=v=tC+Gbi&3Q93{xyUAN=dEV@#}4yHRro)W
z^JZrJO9J~>Rao)Q1JVC3&O6OM_uu@(zXcrsx2fI#cARBq{A-4SNi+TeiS|DsnSc8$
z{ttQHy#FhncUo)dAFt&<+3tV9b~7{nA!14W<<95)=
z{x1}``Ts5o{O>(M{`zX<-(qn8G{XPJbpAal{Rd3`e^VnI{%?%%G(Za2ACfWiKPhl)
z7-puwB#=L9y8k&8IPU)&1)g>|{l`}PKOlbpmX`n0wf-FN|4WG9-2Z;@J8g^{{Ez+N
zzbEX?_77n1zcUsw{UOo_|2fk9`^E1x#Mpn+4*yMIZ)T>yNVTpBmz?f#`21W&cS+H_`v6(EX>M{a?%EX2xXs
z>qGIc$5Ppuxc>E6s*X(R;=$TaDfhZp<`T-A?6^a}vS4j@^d>(a;mOOWED8k!X6f(oqFtBT
z94+X#oYzqf(ab5e+{@H0OZkAhKmau(Tc*kDRz<sp%BcURW3_lL_~m%GzNG$IC{`>>#V**
zLq&{<5(^gYUwh*|@gclr?)D^1T}4HKXU>Cr5!dUz?ro+!n2++O2i5qofj~`>_zS2S;nRE1_)!I%9e+5!>_GSr)cJeEWQXuGO8p4ko${<0kojnSPBc(O{vg>+Tm
zfW$uh?xQ$-2KIBj_VwA{Y5U6N01i(e;TdRPCDL!z!feat9*ZxWh
zh>QxoMkZTL8(g4bznQqOHK(WV76OOYdNm%$+rv=W7%$2Iyh0fQ@e1y~y&x-Um$i%<
z-v8$M&IJf1%z!a??9k+0!@M#M3$)MgATQ!aObm!NvHE-oyhw8!z@vamE`%RUTzN~D
zx4M13-L82N#kqe$$?6-wDlZQO@y`I!rBD<98K{ez1w73mE
z{k5^huUk~@(yWJ9TkMCb8z~RH_rDM8%uhHMlz3!6+HGev&iWN{>+&T_>Y}$0CgsTJ
zB=(c5G1%g`3Rs%~{fIyxF*Fw@RNR8>q=8*rCUVjddTv(1wi7HH1GcfE1mz+F4C5mD
zXkHFceAjqBy
zDp^Ut)HzaVHDO?x9ofCG{?(5E_0WzeQnQcgPE;kL0NoKZ#!;^1MNc?oQaMU1n+Zfa
zG&Z#rzs2sRnh23t2dDuHv7Ao<;ED)G;yHMR9o7ySYGAomldqx?KT*I>cpg9sBhnx5
zC=VhK9KVgk0h3@tP-|T&qO)Jjor{eSTe`JT9J2c}y=h##*~TF4N!(qrlwWn=n{9&zFYduxL}3R9zE#m%fRq0goP+?MRDa)Hp53bben$%@|y^v8Ug
zHC;@ocXG6@b$2HWUkA1Gz_kLIu4J5?rI1$N(%TsZ-5T+M)%S))FjU>I20&
zjd{5I4MTxCccenFPxW;wAS6VI@tDo4?Q48M)%CNp5;e#pIRB>go3&6TgDIJbDjTL#
zgRH4FJDao7NPeRcW173+^Ak56ayD2_SyDrS73VhvA4zb_>GKryqwX4?bR}xFM+w6%
zs`Me4JiBo?IJ96_d%Ik)DT>vtB~o(K%RmlqYH3E~I;6sZA6l4N0R9LN{E86Z0b)pF
z7>}H&%HKf55`_fxVeX>Ny-Oq&Cd?`Qgf_Q)oQ@<2NAN^%`r0q}H-5zAw@vOmPI-xN
zIiy|26Ey_91WVB(jdMnR;1v=-q>9&JgxLnGn((&-p$-J-f~9;i_I{ntI8Xjyb&FOD
zcuT_L@Y;OPbIbxC08T0pGcf56@Q%BpZ3}Z&C<=ElwM$9aU)ThquNxW
z8j)dR36sq?ds42jig9b_3wpAS)8w;Q6=s6s{oHgOJz<5W3G5Z+5%By96HbpdK}}3#
z4cm1cNWS_6fK_yK%A{C8!u2$@(y{3nVTQ1Z@zA3*t2r+Q%pS%Raq73jXy@ZkK9ycqVFLd=X-u7Zb){POmvaO=x}37PzE@WMjKr*m*riNN%*3$dqlx
zhBg^k@O%Q;K|>|hE$Hh!z8@_y(boR(11i=Gcf}hg;AaU4*w4M-i$;;^@r5OJtr4TS
zpCu#KYF=DISsg~CjSA%fslNW`(A};i)D7a0jtz*S*_q%Kw*B8d+Xj9-RjIJQ6+|}6
zq(~^vdG#Wh@+aJ>I>9lbW!W35jKSoKn5f&&=24)|Cnb!lj{kyRncq0*K5Y$d|8R&~
zinIr80Z;;Z@_3%bycNmvT=s&lKcin8r>OhXBS5OMI<4X&im@$Bb&)h#kP5>u24
z!+~pgmTZ|0BWA`u@7IJmoSZW!OweC8B_BAX2Qe3Sky*EuPS-0|s&J+$aHb3U|T?^EIxczfSQjMW}mm7J`BMv7p4
zr(WjY(JveNrEw2+K=0xw3W6{9ZH+P!ROVgA3xEmnB9XrYccNoCjSxuSwLRSH-!MAZ
z2zc*`B@>&5H>zSbX%t{_3k93%<+X2F?+;iPkW232AF8}B5Rkog8+oZelU06WB@#P=
z6O2Z@J&!DNacjx7W&8EjitY_O4h(;JsU2;M_Lc5I#4~
zqJY5T-iy6pBpQ!s1(P3oW&W54lVHOhH+o|9MbNZ8x_vJ69?sw5u~RFMJzI4DdlJ>J`{?O0-DJH`bE5DkTYSh*STYXsda
zx>QDy`TFy{Cn91pRAgd%Y4)fXSY0Q){WDM`13PTj(@SjFApya~rXSflu}MaX+x1oB
zHmwX^^mT|PRB<{wtPO6`7%hIq;Nxh2bE^S$3VEe>2H9HYW-#FKx5L_A?i(kwnZo57
zB9XOxS^{(Koq+1OoRZaE7WtxIBBekg;Lz_(%&5Ozic9_G+iB2eKienk__^25Vc(s2
zED8FhXjz&OoFw5%z+uO7QtW#Zvn&@@jrVVxLQFsH9)b$@IX@*EoPU+k@UZl8tuqe=
zzJxlLE>*csmKK1&#Mo5e34>WbQI2|8JYm%`BMT*#z!s4uD*&hv3m`m=<
z+z+dB;@at#sU~B3oW2a67ODFBbqJ7}$Yb8CmD(pXVhUh&*4VeXLwCSSxtq+sCJQ?z
z=NyRQ74BXj@yomqpTNHS;&c^W?1JhdJrkq9l8SueYmzMdk>
zNjpxPqzf1v_sNvV?S8QBI*DTY!QaHMBtla<{Ea>@ll|`dc(wibu!sRWTZe&le^U1X
z>w-OGq@LEfyU<|8HJ6{>na?ho)P~}enQp7pCR-?r4P8WbmqL3
zQjY}Syc|uC^Q!uNg)l>)5dopWnL6u}YRjz+oTU9$CevuW*v)(XGqRU78SKU*o->9`
zxHZDkj(kFGQ3vEepjEO;+)?W+7afItK37N<5dkSsy$$DO3^do~LHf_8qrUap8T((E
zFX4a+nMebK?U#dUpReO0Xi)c_1eSKv$lMctqQcVy#Cdc(;@ZDt8Ml`Y_Dm$&ywAED
zv~YJ}GxzF0{K)e-$qCEjqLkSV5wfYuNDH+XsMowglTFG|`?GhK-`|xrIV!dss}J=b
zPuuv%rgKl{$~@Uv$33{^p9$18L%>+g_xb=QA|6BJ;{KmDnx;NE9p>2Ub%n%vNgP0s
zQLhK8+k0YNuQxjvX&t?4s2tmeUf;eOW+X(Zxjw$f`P7B>+7o(GkK|syFIW@NekHAWGuMWW(~Cj;ipYwo>K@8$b!}&
zFh271pRH>Bs^j272s&1}66U}4lNF|!6vo#i>C!G;P^dELc~wSeNj(=5XuUpeH5R;t8OPef*5Kg+{YD
z=RB)U&jO2J0Yfh=&M$90=)H&nX(cXdU295l29YCE@C8j0H$n_{rIA8`$t$&hFjJ8f
zu%CB+I$o0ClZHNyfpI3}oc`R~4RjiutsyY9N2>He`u_Os9H4C_uwV>kdqgJONDF2t
zLWe|?N<1mpsm9D$466s{=1AzxB_-kP{LwZb@OJpj@IXxAeN~OO4_JRGQRzilAm0h7^(-_Vt^479(OdOeT-=c*c(sbI}aU;ZU1*+j|{
zdmg1>&moaV-%i!Vwhx4{(SY?xlAiqYNd;TVFa8VS1h70R_~9#d)E4T4kLXK{RvdE<
z2$e|)4Q6~Z$?7L
zSv`JNOfg7?!e_{R8qQDvZH71}X~qzs1+0W2n6jX)fMmIjl17Uo)UZH~9X?f2XPB06
z8@O&ZnQ;28B>~l#6&JCgtDuYO&a_M_ufP>7#5mP#u+q|88@=qMfZs)!ePjh)zxf{t
zHxS#
z!Puc
z-#z}r-2g}g=(AmrCBRHaiNg>?n9FFzYCJNw{f*rlhGp%HBD#KDS9u<3*MYp5S2rw-
z24mn08{arK;)UjeFklgva%aQ;I~J}0$!eOR4YuC;Mj0r6lQxdZ1E(sS0k5gBdXTVQ
za2!4|qcV>6-LptAP>P=odzeup7aV6K{ucW_g3(h{=LVaFJb+1*Fq`DN(l>5VV2H{P
zIecK+s&XzDrF>zMQk*^r2H3a&nD=;SDpB7GrqS*%9vqOkF1314l8)L~N9VWA3>uKM
z!XJF5sQU{u4PyZOgy~ZgVSX7(U6w3xlsH|KZ*L=P*J!-lKmZpjZqBD+^f|W%tJ_*y
z#D-jid+K!XO#tOLJ&@udU}Tp%)-e+(AwB!vX&VfmZ=PQ*A$-dA-v+(qc
zm{cr(rtjkA>L5>?-RHJeKr(#%-jeiqY|dUl^{#30=S->5H)&{J=1*QiYJ;&;u)ow$
zfnGNM?5P&ytqP*O_rm%gu?09Lu=(;4gqH-Z?oG
z`MJMPu7M>9TL#Xozk6x4NYyh#ifZWR&J<6-@m0oOl^&0uD)dz+?^oT9iE-=T(A!ra
z^vVY{^_L#NC6?J$1;(GIbN_YEk~i+0aD1onn|{
z$-E?0*8d9p*dm~EnIK_B(Bhus#M#Tmgf}t*^>4l1r9?74ntqQW)yM7!B$aOd
zi307w{Lway0k1J~wSb|eVU9k(5dOKXv=1KVV+Y2CW
zow@AO;`0Z-PG#r^WQc9m7t?G${Ky{|`|&=oc^L-PF9hsj8}2oVzEpxTFSP8drx@Qw
ziTd?!MqTPSxBLQ*_FX}URuEGqu3$kzXXMI?|Rj-ww-ZpXO_VFuZ3=yx&FK?t^KEE={$p-0@$i`
zQ_XAgNa&lHal2YE49ZpDmv}I6+wo?IdRCfImZXmte0|cW`u*{rQq4VG(V~%ueV6Ma
zd(ae>Qk(G90SF0n5|N#U@flzHoPJ~
zxScHsNYPd5l-<
z3K3W8DfzU#)#H?8Nqv|rRVdhyd+=L#)hjqy2-S!~0iNUJms~wk9as0pP$+YZv}#?_
zp;e_!JdhsikRSI8SI^H3x5ls|-Tcd?517_#KRoNM?V!c7VQhpQ+o7$37T(3sO+UNQ
zJ3*mSGMB9q4CW)Jh!ZeWr6rC5b29V0jU1;FP1Zz
z3yU%$0Q9=oEhL`U^r)BpZ_*9K-O@1~?Nb`m^2aw0#<$EPut68@LE)SL9~PiD2m|oj
z1_k-o>Ls2F9|tHOZjEk@<>?LzEk9nelio0nP_cOQ0^wzW#|zftaEZ_-EJG
zjdy#$$CzaIO+U1AfE+%6P7h+wkVRFs=3u2CfSd(H-bod>UUzM-t3|qDA}}mD;NcVn
zo|vp@2CzqLzUQuEIBl$mQY&Y7)1>v@!l+P(JS~qCyFi#~P$fv8JPoQiJl*GhJPE7n
zI_ml6&a+)>L2&NL>13o?;mOQU|I5iWGxHs^JpV!2>;se7ehAsOn>vZ~)9)yD_NlB6
zz|$1khT3*0JDFIOR3Wc&({Xr-Q1KIbU54%LjT!zEhF+@Br=yT0LUIy`~
z7mjK?rLGJLUH@Y#FK=$b*9Mmq0CUqf)f3^KlI&(@u1iu!pnhDqDcj&z0-1MS`gr_R
zedV~C(e(R<2EGXts8Pyi`I0GXI>y@9Zw6SLx$MqHkXHQD%~1pN$`4AF7K>*g5D%m4
zsgntVbf+%lv$<(B>?AUJ7o8XNc1T06*X}5|VDd
zn{+{rgiC!1?K1KO_%zo9x;v}*9Y&zDGt=J@s2V&RGjDD_#P=c^G|2k9^bd2RuyPeG
zwpWNwUz!9mpXI{V9lf=q<(i1sxzPOt&Lp4;3KBeodUdA0USh_ZaMiA($QZ=P
zVj>CBXdi?}i2-HVfU+7JfLXDyEM}BWhD$WK36yhQ&;vx;5EcGhN>DY^Im(Ll&$T>1
z8vxPI;z~zfu;|ux6(|mR?$Eq&CEWY`pPV3E5%H+3OJIhSn5|53$ma>rgkv}iV$P8Cv~qB4~oNw
z=8rXw=9wD;k@MHn26&om;^qaC2A3?&e54(lzW@H^uqB)rf2XxdU1VixPVzyQ0Gn&RrO99d`p$5uXsNEhaxTJhSS5(#&h*x7B41m3
z)X+F=YeXI;a?pTX?ccM2)p&{EcE8xE;VX3n2vI50VLd``;wOPbnK(|V7a$EHH6Nan
z&3%sIJd64irL7_Nn(3t>#aR}eu#zH^^hGEz8@nkagSmg>1I=)uLJSnDz3|Qr1MgKN
z5}U5RfdZfh3zB1iYNq$g>-o7cRCA>BF1H`9xwW64byALtx+4+DB36!IR0G3dKd{|*$12UC_giwnHfqgob>wW-J-%XVO%l^N0!A!OVB5=*68Yo?V}!!TRCY<}=_{FVtjKMWTRQ`

zrpe|N<=5KrOc1TSe)9L?(qGT3|7A;`iH#{uNe+Yx@Ee-4Ut>r8y`zsb1LV4tzPRDE
z%wHumZ>JeR$a0ei9$?JEu5`j2DZl&iqC{*Zlf0{$e8&3~+U;s{xaY_;Sw#JBePrzd&P1wg69+}+;u_XK#3(dze))=LbyUZARU{cm7fdqLi
zUq4vxr~uSn1Gn>YO5Inj<}Zx7zAf9Ha3IS}p{a;NvC~?9&dhgT?Hz;?xu8itLdbWa
zS*XQfMLUf##C~GllQQqLDVJBJ8mH-^RmXV+vS_8Y3uQYb!(#;7F`>F?0vVDfOCuNzx*)
zoZHSnc*E4a^AE%dXYK|YyFe1-ZsO?!dv#n$fge*7;ORqCm8^0f4ZcFlXh|(BB0T=y
zEgp(h*F2)`5^_tW{;9xTPHZT1BRhsQ`9XA-Y(esr9Qj55*Q%5U>CM)ov)RwYV&C^kn7n}&q`PN8C1#aKxijz!arjHS-GPhI;+x*Sp%Qw
ziC6@3Nmz-*6pzYW$BET($&7)hiC-~h2La?;((|Ld(@5QS8b{)?k;o
zbdW(zR^w=m>xBfystt@Ke}$>wj$?6Gyo$NP`u1On$~dw-myR>$_JjUP*H=v;Z~}PB
zGl(V1UPdxVGJO^>^65^Uh)L?fQbAh`i={$0i{02G%1%4~l(38#g{K;1q0w34yDYYi
z;qHhG(+lxfxo=dbrjQ%BMM)$b?-vmc=g*%m1Mww4ec_HM$IJ}@ixTFAEkrHkhElD(
zj;=S>iN;VYh`cas06T|XfoZJ*0s}l~5?hd}4v4Y=7q<7_r(}fc4fixH0pVJqeX4co
z8m=^xO0r)YxF)R+*y0W<$g|+yI*wDesq}9Gubz&i5ZQPa<)VZaHcDUPP1|GmQ=j&fp9D1Eej!D<7a=T83S*
za7R)xl_5;?7-0b3Tb+ToVXtsz4dS~o^71pib6aJVeR*MGvHr~$Vd+Y6)zls%dcyEB
zMR(E}C#cg^c`Fcz1KNZpyhEJhl{+`u2!T8M5ZFF%mmO*qUblNqpna*M5LL3;Nyj5z5OL
z!Bj%m4=rFIaZw~p0v^;p;mU&AM!s2d!FwjxdpxKXr8l8j@)&$--KQMlnHKjTW`hTc
zY^5#qcDnOECHn1%r4E^fTDliQC6U|`2sKnuamGq!p$uL;6i7q%c_M)2HWW<~MmO9Y
za6Goh;C`NPnS4_Y@;HLnmU>O6oNC_yIyPMzT>yX7hU_et-JDBS(Vs?s~ml~tgtyY@91vUOwj8H*4(-7QvJvykG~f{p&~
zM|&Vftv_Jk{LkCx%5}63P0k6Yia+!1IH+?5-L`?ZEOvH&5<=;AmFqSAOdFl;c1gz}
z{_7w?_&+}W4;+%2?XM@%{yIu9vHy9L*s|Z{MDjbV32#SS+X%_dK}DwC7%yK=k$-hK
zA*NGqiIg=iTZ|W#R|o%iasCP_Wl0$eN;~(XXkqYnFKe1*Pbz?k#s9hqbbq>AJV)Sy
z2q@GB0Wt{3gq1DdB!h!vDE~4&b%R4Z;MxN(o>e3vPJl1w;~3}`0&@*A2Mdsr5>|uP
zd^n!ByQYky6M>xaZtlt>ri3)zBoxhw
z#jHQVZ)p^i2qqzfFsW@GGZq2?n`uOG&m#=T+#$7{-x?g6#dncg#!g+A6XxTS!gTJl
zoxDg5TzX$P)p@rOKrpE&CNGSAICv?k3=JUK&tNomfhU`Q
znW!#%sIs_uB0gAaPmXEKPbu3vpzHorKXPvt-OUR3(mX|m90U1zrv}IyB~TfU1W0oa
zald3P@~2B&A1Z;S@UVVtC;E1bNK3z?xDbPhtu%McF>l;z>xFJ8=?!JB)yjK}DCTTf
zHl~&IE~SQMP@+S^Q8Azzg1tKgX*fjUyG~Xg41E9o8T%_!u9v<4N!|wY61>UD9_ejS
z`eU3%uG;wvt$4=B53Ok@xhOqG|0ln17y?z^wFQEt$@%dX&iJ
zRmA~I(&TEnJ@Npa!2*k1LtC$zUA@m-Qw>OcE?1HG1lobI7%|uNx*v~tyzcXCKE0qt
zU!Na4?ebvO%~)_Q_vKu+&P4#GYNWVea-}jzx)(msfl*kDT!X+#GF#uUqD|=A8?gls8k#=Lgm38~Z>EHS9XI0xw^D9WM6b
zXJ#s?DP?V%oFsR-me@Z|_2WFl@i+j4dr}+G4Sb>h+#@AJOD;MFNGiLIZj?_1HDs7c
z{wIq4PZ0b6Ty@$06Ttny=+!9=S;c=$@I5b@Vh*4YA$*ys1Z~8Gs8qpTLSSks%Ptsy
za}tSHd#Ut!s%f)4Y-u}-YBpBCFJ~fP#!-Yd39WK165M^HVT7qXuptB@b5Stxp^^)P
zg3}!X3gEYh@f!ERiof=ZRQG!szMU;K@slaS;GK74#eqPZ3!uki8ujh=%<`kk0NYa
z$=ZLOWKV5linZZRofCS0mpM)&Pvb*mIp!}6Vk<4k5ppMM|CLuML6)UuraHY
zi$Je3R5^sr|2)yiP7=Cgf$`!-^XNSSZe`4e@fy58k2cs}2?9&jsw;QW-?SU1y}KGH
zfe6Y+Mm-2lqLO95ql0%wpsoH!Rk%DxUV@$$>GDzs)$~4M^akUP+pe{>|D`!se%j{P
zAuJSSYl_uUQk=_n;v;|dd$@?2E}iDpwGEFN{eW;7nv?whY>NNmF7{uJOXD;IM?n3L
zW6Ape5*!OL3kMr#+TRB-EO1T^rnI|KNNQl4nUG8C0_LW^$n7rXA;rn$h;cEm}(HVXJUVt+49Ii~&jA;`2VK9vIW*&yu#{JnQ;p2NbHM#Eb|GDyTHXnsfo;m&%;2A*
zS5u3dW82rYARxH1SXfX3=H~A1?v!wv7Fd+tRB)6~8@;dPx;ot8n^!ruKoG!=f{^a6
zVTj|ejP8_$A=KB_pdDSX_*Rx*f!P7QG2~Z222I|yM_F@On@56!+pv508>Lq}=O7lu
zx4i>exV%4Rw6#@qNOi0&VC^8n!^8`m;#hyMG%ft+3>#WGXQST%fkOW2l}QUX*swna
zPVb88Kvk?};Dp@H^2W+4@C)cZiG6r}`6Oxg&1y
zcjs*3*;oYc9HF&(RVGDsCs958xHKb-ZWA1S^{P12#%H|URAjXSd6;bctMK&$SoVMB
zebGoh$r?dAK(s)FBtd5XEhfIXm-X@-wWdF{zE(r@Y$HA+f;j)twaBkc;l90s>)%*g
zMSym2bo%w~_0xX4109fpXJl@6e!qP9--p?4nmbN>JHP)6;0`Yd$oOURVfBrxvfc{A
z-Uzkk1vcq_3_TeMd}aVl2|fJGM1LvLNo(rrivQsyo&sV$0-K|+s%-gxbZ>xaP5!;*
z^Pfb+8qDm@R#+1aGoMciyqKJGjqRv-R)3=AhWx
z+-=_$SJhbZc0xiH79cA-=6$WWpGZ5c13%_)Mwkg2C=1KLf}cFMH(6;gQ)__vuLh7h
zM<>weoa|_R6s>J9|Cl7S-KqEGadV-#-b{$TNr(n_PgYQ_wvJb@$*UY>KL9xvlE3z6
z>nUIm&M5U2Y2O@_N%lKv6=ZV$hZqs0@lr4xu8-~qX&;h__7m!^%TxIqs@qff3#uMG
zed#;MW9JeW{UZ#Rl>7eQ%})^5c0pDT!n)vk6&=8M^83FdXueByOJDUS7=JaD)xJbD
z0pO$i_6J_5|ZJPqudyfELP(HGQ$4IZK5m*1sY1p9c1?57?)b@>h{ZlwX?06R
z#;KQnMhaxLDtVkhyN`ys_j;%Mb)ekQS519hAQ5O}@TgWSWs-za$$*NGHWtp+Ut5J9
z1(*>zb6pcWW}!cqQqir2A0(+)8?^6cG#P%{8#;td-gEditf@)#p(CicepLPQYE*3#
z7vLH`(T;heEL&B|e%RCLbkU*!&P)bhot_S3(WQX^9632xMv{OWTm)$8e?YRjcT8Sq
zm9@k*EXif?d#bOHukj
z*JdsN0M|O;Pc8#mBJn1XDsE_eB*sBk0e(w}IvP&9pV4oe1_b$uz~Q%n-6K(|xX
z8mFxxW@gejm43T!wB6vro_$j~{D0Vkn#I3-$+~Gg1c~BxsyN|l%}m;3=VY^!{0+s1SB67_|!e%qI8esWNl6ajtm_x|YkAtuRI
zDRxo5+>h?JYd{6An3`lk@g)Z$$@>8neV19097Wj>yoB}1)AWW_*d8O;yHC2MItNRA
zTIaO`H8}>JbY0#bdbV%+Y7;`%Dha3G+SkHCrh{%e!_bj9tPsY81*Oj{T$G`)7*}yI
zS@=ptPR{YJ$H4{ZD}YGP+$Y^1PoGDi&QQL-_qwo76nienlO`ySk5`@+-aqjeqQ`N1
zA^W$nYWN>mR*O&D;~(-Z>GfASZ0Ga@)no|)54pNH3152H%zklJhU`$()L$?SVH%x6
zeTRf9&UenAEZX;Uim{e{v5AgndeTa{&_#$Rivu*iDJ?bcVn7J4n18D4*Ul6Hu-M6d
zX{xNMN_VjY&O|B4+o3G&LW&~}33{<1Hf6t4MDOb~wT>9w$oAeWWpoEsZI=#fvM
zs^U+J={*#S4*;XgJdJgM-S+|H3${PW=Lyd_Hq*6zpPEpjq%MuCA1`&+loSN#xNWm+
zFw{0
zWAeZ8;dmLvXmd!!>zvkYk@c7D*a@ecFJt;s8za{}9)N9eq~(+o9XhWM2sfx`EZrnq
zQVhl@t&lPotXvrN@vO(j@pNXAD
zLSWbmQHe6@QhAGFU)Ftnn;<*PG-|Uf>Ns^M3>Rzy>qp@j3gP`olYil5cF;nfl~Wsn
zZ^^g!5eE`ll^TQqMM{JRF)E`X*XpPK@;{l?5`sOp&mr{l8sf#-G1d<%qf;^p+IRnB
z7|poW+bjn`>rt_%(Xka!P@n{O0T`@6G-{I9^@ou$(6~#rQZ&m}ToV1?^~rh-Q{TfD
z)cv&fv6fSup83mhLA#Jwf+}l=|G?cPxB#E50}I`(QLSHt^gVePH6P*x^)aX<
zMh~ogdX&aceF|1b4FFE>0#?so-_8Q`b`I~|sS+tfb0BY}HQDE@;Pth>Wg$19Wmp7C
zT8d{UC)s=*ow7a`$AKA*&_$q>k}@W|
zr$Fj?up>Xg_t|$*2CdBoJ*;ivu2{#aHoHli<1dE^a!M`E76#Uv_%i
zPH>91Tv^v&f-kj3ms$mNHWkn4&Bj|bcfhz}dBUu=WtQN)Q|)N12%H}1F1W@Exizxj
z^WieK={G$ZHq~bVpt5w}p=Dt4tX23ZTHol3jIN?P7$dvyyhz?H-1)%J;aIZYz2no@
z}@N-sAjnfLFOcNQha`599>k49d^>?gnod5a6xE
zkg98omHD#!p@IAUpg46$v&n2SzJB4v%0m+!?Bzk%k%(+2eI4#h^pZx?QT~DR#kKT^
zf0O}~lKF>c{!QogXo<@$RtwN01=Ks5rpFQaVZ8QhZ9#d@FrgIrY(cWeWdp5F)Ulcz
z4ApyPuqE;T?j}a_a&?^GhYK?3ciL6jdL!bXpVVrULIbo55*UkZe%s@(nSJPY9+br<
zPFSru{Kw}B91pCCzu<=V{)O1dR=qYt0Y968o8reuK0c5#;%~GPX982Si^@=5C9CwP
z32)*e8CuohYWb;z9q*&YWDhK31Iru2#VM;SK@li*ce?h~>7lOC13VGkVv0suT=x4~{5&^rL+83?0
zja#fCjnT~f_KFTfld*VUiUzwW*BZ!8cnWF53uYsN>QOML6!tp}S3xmgDH5PVNV|I+
zto8G}N>?+j_i&;&JOS45@t)Mo!(g1>Z0YPGIm>#tF}a=~(7FV`YT2X@AN)SAluK+L
z_U@d3im^Z@7j(>A#UjI_6*NAqvD8`@hZL-mMq!ka#0XP}h;)?=EU!A9#LKCR;lv>v
zhuAA3?ZXP*ScRzu)1@WVmLlVNWzXx1?@z1__kM4eE4!3&JqJ<~dox;-_QwISJxPKQ
z5?_Lz!Lbox9?J~pTTWE{f{8NEv&|Ba0KA7};WE(iGS}%(oFg3C-|x@#w!bT`bJRKT
zB$?Lt-?3@Lr@_NP(W)t{dT;8gK>x#i$SeOlA@@;_QfS)aK@*CB=^2eupvU25MWq5W
zK08h0JgLU5CI=FsUjT#X5*FA3*J|FK)cH<2$?7HO16_neN&Q`T|>@GGmLVkBO
zSi#Im<{NjdPehIzGoJX~ldaJVH~*oi+-o+T7jEu+j=n0-
#h67Ho&Z3pePP7L>&
z{W;x;4+eX}$>6twOu={#_(mVnP7_XxO4`n56NzT%RRJa{zM0K3iHKTCVqGnK$N1iR
z?1ni+ra2Ri2ODT|!tfdkkB*_if&CDZ6FOsHREq!p&n+yPOe=NR!v%!Orz8y$Q$2B#
zJGU9qIz)2WUtWD1U7oz3Q|le{G~0oKVx;J>Jd^c`fgz&o-4U7z@~$Um6MmVlN_A=>
z6KP!Vs|ZN?5)_kmAos-nIGG)=Xy5vm5ihNLM{CP!i#dn+F3`YG0nt#6otX3YMHh=sPVMTcOD$NR
zMPg86d+37cSyb1@lB@6BShV^hrI1`hnoO{>u56<)ldtp}gz)tHJakH1GvZ
z%L7RH5;lRZIuQ$L3;Q%(g%qZWjF&apiZ8bWjeZ+~nn9Md{YarfaJAtVz*!+*)|SSq
zIM;HI;cO7G&AcF-Yr5sdoh
zH^s3&7jdRCL?D6&VMRbm9Xo(_D+zcPPYYy1@qd=Jy882nOOL(2A3b2e44y1(nKgNZ
z?5ZjrTA`-$XpV<2_r5ljxZ&WzyzZUFoq(kSKZeXb+HF*#4za91GQ6GRa3MTPP8;x(
z49@h{tw&Y=I4YG95sG_HJBTrVpI$Z;)u{^u{-*HO_)F(uS;pE#*5x^nt;N{?{}0ihAQ)O3&=hXtl+T6u+>{OmWEr@JHKTE`k?
zhy^T!Iu+~Y=-|P#DIuq~WeWU=R&h(7ivFgNT_zt?=gPX{=RaVA`^H-HJ;vVoDhCUC(2npYq#<&?sgX5;
zn=e%DQrmA#(GTp&roFIpJRT!A3*$bO>#}-guth{0v%i9eS`tarY7SAvo=Lz0!xgRl
z6u`6Ox|e{x^~;{1p-C^2fuZ`9TdQ*EZ4z33QfnHqw(Iy~1V;rpBUt0Cm;V
zk*(bpkh78b$aNWj0zk-lkT_KU4J8h2r{{>h4T^;;U#)^5@XZ{RB6dvy5IQI
zC_V=e;Cwh75H3)2;DUK|caYqr;L_mpfG#<@AFK0#CN6u3(beB%{-vsXkdH*~iB=S5
zM~=~slFJDgY2ZP{*m-WKU&&qU+Qc{@5&o4!{^Uifb;mg=f;feG#}8a8uw|k9kxMzSbTj%JI*JMHw5<`87j;rE@0F;2#|0Mqb%JC)Yb56Z=c>
zV5C%8dSJW97B|%#RTVfWg6{~M;$*vn=V{Q4+KJ8lIz9+Oa70N-1B?ejg&r
z5B6@_8Exn=L~jUMLW;X`V3mW|`W9u`SWv}T@@=4Kl<#FvHjcfe4Qlb{B-Pmx%`+F9
z_HL!<>>2$@8>U={7J)e=kZQ-G
zZC3wRh~*sGmCg+(qXP9m1d8drGNAD
zjeC9Ap(XAvmX~}YGG!}4nz~x|eUnOK
zsx5%EJ20*ye33pnB~$&~ql7^*z2lH>0?i0VTb-GD{0lYQ;!4likN`v%cvk!`Kx^5h
z^Ftff{dFwrD^=Np5i7AE){{W5OE|I;2}QPr??ILNTn)GleqQOXTb(IK3c3kBP<1>3
zMdxb)XG(fW-|kK`UB=U2Wpd(-MEJm1sbsFS6A^9I<2z+3ti@1%;#Wt%yO;z)x4Om#
zF*$p$%1;HG=m(e`*y3}jCrwG;Kc0($a0iSXP&BRNa`;q5@Lk%S@&0q971o2y`i(okNlYA)sO7bL&R@0GjIh@uA&fr|X7U4*
z+D?l^^A5^cx9yk+4+t{GVfdwC?Z+g(YY&neZfE1?@8$Q?JKGpy?$r%~*V+uC-tWmq
zPI=Q+-E5n8^URIVHXOiLUFpB+VwEsGt{=o7wpUYSfpeIrhKV+Z;)_>F0`Y3JFrOAdv@q;wJ%afXhFQQQ3Y
zwlB}&>d%kZvK=<(1^z?u=f8!zEGo*;n{W~2OovDO#|NINbT%?GtpDYtDb#+$hzALnF-k8
zno8L4-w|<4`y$EbX0pAbaU`vei2hwrda#6#96%|7mwRX>Xh70kg=?c)C3tS{5S%S9
z?=4i|bq@Gvk&Tt(ZFE(`Wo<1S3MRf;>nnv;^)@(5{7K$;bKKp=oe|bV6u+plxf_J{
zc*R~vP9xKl6d$z`6{{RUUsXbNE&!ZS(%6G_mNt~3F_Q92ODIhmiR3tfrzF`Sn}fT{
z&*E86XE;PHvs2QCSmIY83$ew8I}Sp=r^QrXCbMbqY!}cAw(yWWK4_ch;)+`pzZ3cT
zi#30GdaeKY@lQm)`PsSysW5XihG6%82+V6v(F5Bv%h2?qdF6q08u~x(+%a&8aDr*;
z_@bvURYQL9MpORiIHX`~K&XFC?TFCvKTc!OW9;?MtJpg>2WNh(wTxHt19P^T{{g$r
z78HAK@cax-!y1pWeC~Xe#1JOn5F6!ZjhML6{L*ByE1}czhoyB|xK}v~+7=EOBVP#G
zoJILl6^I-9SA0+?K@$8qb`sc3DGMZF{*!cuwgFNNYQ}T$oLR%!SptA_1PH7~+z^&v
ze%w0#)>LN8M%`uQz$0M%eNnq;@K)~&p5ylG-rJl<9?gr9Y!kolW_m&O$FB{a-_Xz+
zoIM>z;FOcOW&%+`~!5c#;}4Qep9vgv2firEf+mFuWcO~wlQrcpJkg@`6dA1q?d
zJ~W)aq4DgiW1GnA5wF$v&wP2lEJfZW9&5IJW&w|b%D`e|0mhyC7M8%2arvEu8slY|!k3v0P#Mqt&3M(lc3Pzs
z{FE0;QzmFZLxc`qU1>wS-_bkjgB9im`aOPsut?1A-3!$y%%t?&>Zw*yCeve>jM^sO
zaLt2oU5vvm`Dv4*(i9OyTjq%!aqQ)r(Eg4(>Y+N|pLNvVi2|v5eILIxAD1br$G9-0
zN@u#9R^3et`DF+yac~QIpm&LSgkW}&3JIybFifrc%#4@Cc|vU!q_KWf-oz}Oe%E)U
zjZtNuVcC=1!=guHs{xcaq&YmfRFvpkJs9m^IK^+?U8u&K5oy`6zAa$dt3_?@A&@gWc5Lp*
z2%U}OY9oChXPA!cgvcxP^i96dINK^A4z`!2uVm2NJQ@HImLTRWKtFZW6!bWb#IESX>nh7xM)2mNvO92L}pC#y*j=&j8^k%=a9SrAZ
z?zIU@BsYsv((TkEdb+osR!Ltp!LyFui29`2O09V7FL&(@AH}0)JB6txm5Y>$ni7~b
zAFtH=tg5-uY8h^IZ0@ttaTrf;`A0M8v(@Ak4)%g$UT0`L@i34Y2Ok>DC**6N467?$
zll1hV1_5aq_M;nYsUpGk2j}GL{KIAr;ZVctT9DMvYT!1Hy0ZPWzD_TNR6bC45M;kd
z93aVFRO0-=XG0MHp?ew)O!Mxx%3_JX_vJ=?qf=2J}Ux=;k3@CrtuZ
zZh$&k83u0xG2!xw8+|k`(E~1G!u6_8Dv2{Zt{9~fhiCy6G2tg*$}U}L3)m$y0kt=<
zjm*GPn|&nf1tK4vX3_Cm6(Zq-Ek{4$9FgQPN*kQ4h#|&X`3mXprTrz=YDMSd9Cf;M
zGi>Ob!A|L0;o>&iHt>|+{!3y1v`TA03V^?Ss-!H1xIZn=Bz>A2wSLz-54?s|L`zhA
z6qVOrSc@(<^)=;^%j5An}+q)p|;QWz*cuvJpS#z-)>2
zuX{n*#-xl^p*`oo0D~Ns!&28g{Q+&
z7%w8O&SAHK1z|cngppN>$-Bk!m>=)&6TQJ}i3+(I6siZ;GMK@zHzg38fr;*Ti44@x
z2d$0_N*kOisf@l+B-M=VVOthzE#_v-@u|pYf7?FA$jF*4_0tmlfHyTy)WU4rGba;$
z{v(hsuCb9%j97c-ANk@C@cDA1le$%Tf6Pf9Q_S!qVE1=}+5sHm)Ub4xwnaSbu|P>-
z-tI^`d)}cJ8!FhlNg%LOvv%`2fwA6O3Lm`|KacseKY_j3kPf^D%E6pn3bzzZ();-_
zV$b*Jc!!Inj(Jf5Z6#?yZmsDgCxHuBUpi+ka28tA!2!(a6?bl5Kh4do6=Tj^qg0h
zzQ+RQx#3b3`Y#X{8Q=@+fn&OwsAHvKOZF}O-NjUGLmb9l)dz^+>ng&-L9sgecC
zOrT1Q^X9uWOkz9Bj0t#?^oSeM{;!Vf=we>ir25rgu0I$ZPb+KKN$7lJ)O`sV?J-lm
z6PyU_zOcyoWR{PJQKFphKk1B-(Mf@cG{D#M;)GxIG6!m}B?Uxv_%&@QPZiGBt*+xe
zS^DswKTjK%{+n9FS6g$QRGB-MAxd=aV8m>8Svtbctq~z*UC9e=h`J4>ZPTTVccx%k
zFtQICP!ag&i`*hqZ|Qj#tXlqyTg9lp7abJEL(Mx+b($@jN1YihyU=OaAYFmV<_xE+
zw2=^>KLJR10yvj)D|w#h5JvTjhTPhCg-%tWJT<=#X&f}J3T@)ITu%RX(a-LJVt1Vt
z(eNYVV6o5q4pxax9L3OtV9%G0JYHUCjrv0ozY3D6m7tp-Iu
zl4y5E2CvK)$JvqUq@5SbE{Vu6;o{KV7ko*z(F-;}y-C=VjXdU!tdyw;X0e+{}j
z;d+RM5=Oq(exaiMrz}@@^co5NRd*jiZ9MqU;RTym?KrxCrHy-^h4l-3DC{qlj+!Z?
z1vzK@)2^vsou0cARGX^Bycujdh&b9PCcEJ;0EVGi6vFl~6N>ostVi9Kk40ZmWUf1K
z5u8PCRv3}g1-FzE0#=8Om%mFbS`|^dEAoTmfib+K?;R!g_s?Fx+8jiybux5pHRD2R
zxWIs6{(mssf5B0{Z)q*T;wFQ#s*vSXfBCgkpu2gKt0yw8LwwIrIU|M|jqHpS|7dxN
z$uvRcsX1Kk>$!{Vq
zg_mZxlkspjN>N2v+WuoQ>LfTc`QCYxbMInz#$}ps?ilTX;@}CZY{Nt-(D)Dq6sE{6
z&VLGu$)R3l)D0;z(N!GlNLW{LN+Yr3tyrq92
z#c$Ow;c8x2NrAG7HbUFk157VFehozZ=Iu{
zKJ+?J;quq~$ei-_;}g#Kzgw>NQd~E!1Li@kC7O;`d`#kN&b{w#9G;dL8zB>Kc*A#F
z0yoeDvGu)*gX5qWz`T1#l$hQ97l1oKiFnmB6a1@4&Qmq}0Y#kvjA9j(scMT9EeaH)
z86)m&gOM9e*1w)SLODsmB&&ew^i9Jv>~#*sDChc8V1gjzQ!Qu_;;UWUL6gWPVKv*%
ziSBA?lF5j*gp%%+hWAFnaG1`@gK$XC{&BcR{^<)HietCfbo+6+tK|vKG0Nf8vw9YRj35
zA$T97=pf+~LNGY;=k=PP7#1{ee(@La`aL39E3f9|6O$T{UWse0@D-F_<%1+Ym%5eA
zSLog^KPd>d-4Yoz+%rwi6#d7+AR-;_iu5~yJHa!eN`k5|w(+MRJk0k%s!A3-50>nJ
z`GOR4s+2ohfaI5FTDMY{>{bl8bcD{GpM<+CFm&bIf+Nn|H%&S~rdA|mGvPm3({Fs*
zD19`{bUa32TXD$Hg|Yubv9mHs@q=%}XkAkNpUr0|4Ly3#`uX|a7Vxqz>4qfXFbQFW
zcAA|9LnIJ^6-p|BL|HJNwIg!k7qW75ScwYzY;d@VJ<7>-Xmsc#ls|z%C};1H*R=p+!o?17v6pcK9)g!lH(Y+qnYA*Ng|*?P9%91!a|HNmYHXYPIZC7*|bfj=FH!al7nB^TI6}&-BnHznkqCJN@x^93xOvlct
z04m6d%>oC?ZkXQ7A|E?ss4-;;tnaF{4qJ=prXJyf!;mKNH7?okz)aR#G7eW?2LJ0!
zxOpi$9{BTTE85^lYaGJI2v}W1&jd4(jHw>5W&xPFARQ642y3N}UV2NUwc=nF^H0;j
zmZAW`cFDIP_n*UG_6|z{5&fZjoT|{1Q$DrFRGzhEIMpP*Wgofhv!HOp$BAP%6#xSM
z7MwlKZ}BQR-iVO^Y;{y;8T3b}2@YXNI3;mHGW`}J@zs}rHS5A6c1T&hcc)xSJt-xi
zam>}BbGUjPmXL@rAp20Y)ZxUg8WA4T9J)j;!l>m{PkmVSLO
z7ct#+GzwR>}L7E5f
z!gg(~%p=;dcgHkTE>84I*I_^f{}ek-YZo^ejXfQOF7SN4GEvzJ8W_XevKRxZj{TQm
z9I&-0Drtx?C+hcRz=EAkMQ0hH6T0U8)UKp2ozY5%yY}srDBgeRe39ZN7tZRowMq(y
z{0B`;#9@y9N{Ku-a@`<$QVA+i`#a;6`pL<(L@zalCj1SXSTvxx_SOU!R;ukDP4p+9
zdXsu#U5s_vRUAj3K;q*+(cw#A?_5>KZQLyDQ>SC$v^C!V0&Ol;>gIlW*t6hEdqYf}
zhFBOu{9?Rgh}w=00}-%-XvpNotVspI{IA$l6Ge#}aUbW`u8!N`BkNWUs$;vKk8)4wLiEd?W#8{a^qRMR$+@UYGEh9RkFi(~WQeV!>`o;n1cmHB&n
z*#yu~JCHdm&bL~stzPLShMFM{rPA}$?1(|+79nDD94kig&1$Vg;_FT3T(sng0XINQ;Ah@4(uD_6tWELa
zE%dPCqo9soA;lj4v04O3#6Af545Xb3WtUh;g8^u{N@?#gN3a@~=@Hr}Zs_hWk+lpT
zwoaY$CyNE$j6Fev?=Bx<84dKr5h-Yy?>L?`V$Nc&R4lP=7AO+Y-1SjYcEll$%6iSc
z2BwcG_HV`fxGqZtyj(7z7v9DZN(|bxicfIoeu54or6x`jj06LBIRp~5$LJ5Rz+eaD
zY8Hbivc=I2FZ9QIw26EP&|qi~etwUr*)YlkDHnD*adpQ80FVP7FE_*UnmWNF#th!%
zM>ibjGF^rr{;8Ki6wz-qV1)kVOh6RQ?wy9A4QNieySQvI5J;rzHsjfhQ>3&+(uRWM
z4KQV%*3ZK7ZgR;8$agUXHC3MSkbUgBU_fyJ8D
zpE{N&pW!W1Td%A(K8FhROK|5^rw-X$A02Gju?Dh?!6P(n#(Ww$F$Crk#)
zM8#A&uvk84m^?`g(ZAm1_1{XJ!iAjfq#@7wG(5`E8D3m3Jt?bD?q=9gM^`c01M8p+
z?cS(W&QCnY>Zz&J&8f5wj9nU4at%soxgfMSb8WX71Uyk#?>g>Kt2=|1zFtX7bvl#t
z5~kwz+XaA-d+m<7?R)Xp9-c*(CM7GzV?j_}7%dJu;WW};e@zU$%OgDikU`(*q
ziPrC*I3%8YY()wUvgKl-jMDTcLsL~vF6~)ZB{0CX3TD2>MqpfX-)u8oSb5Fj9i_?l
z9DZ^?ANYz4nrz5~?Gy*ENB&T!H-#Q6UrL`Tc1|8F3;}#(>PO0e`?SflJIJ79GCA>eb$_IR+PG?j+&$r2yiz9lS^eGA1oa2nx0D70>+BjGyJ<
zq;4Rb+#W}EhP=!DOAu920N?!gtKwpXK?8=*K>_sDAtXs&Dr~RJo|$R)7H4nCxY3!A
z{_*S6=;(Ky*k#~}f{;GZj(paiMWUD^n+Dls;~LIe
zW#46aG*gsIj^!(&Y7v$bK;gs{bJiWP%q(y+AGvfcZ_bf1u?X>kg)t&MAWq#6YeVfM
zq#~+Ug_pp6?3$3MFV9BWwVhfKe825aUO592Z(>|zdrs*&-$X1E(PDItgyo$QtyJ)Y
zJoNL~JQs8{3mPV7tR>ZhtP+(FZ(I5a^OGPr`Z<8t488+|8M<@dM3ci27={VN;ms(u||%|9&L#;ekm(XY$N~hH4-mMC;u`rCY+s6
zXXu@Avk+x#-~$52dx&NRq|7}v4;#=*9QxmcPJ3T9r$c!{+(yg4kYMP>rM?1Qu)p?~
zX;eOy4r#J0e&bLi6iljk-`RE$=;ykpgiHuwlE1YU_o|HYK}g8kiQV1%2HvUP{7;j5
zQ5vJXz#DoqHv{{363qfE+Q}s*D--1%We#a
zi}vwMi)&9_OYJUrJ0#3&lk)xQ)_=Xd_K+p=*TARkTMt{6Al6D^aUo(&EgqK;tcPTcp$s=W)UIvv)&ojOOV
ztY6_d&W4$qK~IxtVmI`Zh!{ZTEttwbuT5OYbd|Ef$Mqbv+vFD6@2meIVX`rxku!j1z>p#0-%%d5v%k
z4t!H}TQ1tzUHx0i#+^z>=oTf6fa%VMsSlFQ@xpt#XIeiP}BDzI9kF<{WGZQ@26m1rz
zka$2UX?P{)=k|5}^a58VHdUq%aMSZlQA&aHOxkZHDOpM54r2ZaBdE4q7d;_|8*BZN
zwZp2kc|K6Qv^j8$hWy<6kg&Z$4f7TkK(??ceZ4tLk30%Va66zQ(~cK9mTSa}H1OFr
zF8f;HoGIMzn3xJubN3K1OglKS*))_g04HgwiT0-ax{xvI%DO2sr%AUsA0Rs!ItO1!
z@LbQ0c~FO5d-SQQ!D7-xV4tzeeaXa@+3=*K_h=Z0ri7?Do(1ZusAg1_Ehm_JE((la
zioye{sInG?nF6Y@TQ`o7>T0qimRg$c=Vo&EbA06H;3ao=ghVU^7oqS7m>b|1;X8NCGqhSWb
zRbJNb(Ku$r2;&q18jAFMn-}JT^*YlhYC#?hiXPKMP72uPA1|a$d)||(-d@|zO+c5)
zD0C6qKQcoTu43U<(aD8zeZ#;4&(iSyPn3SV>`4AZ2^EDYwQwjaFn1y|>FX@}ZH7r4
z)c~N+I;x&Wr(US;JM)zc#kq^H*bN#2FU|gmUz6Hr0h~P@)sKiVy^O`Wo@8*TnZaN*
z<-+NQ`fcb-TmPN9M`7b(@`Bp;Wr!nV4t(=G_aK!LkrdIZ>>4h;OFD;s{qj0%fgUc0E6|emUZ-XcNjGjW@TRoNyNlgdHvTu&Xc($W
z>TegaPwEU4Sb6T6!t)qjjJ!cC*6k?htHwV0RP8p%s9c-V@l=X_kEgW0X7D$6f(xKl
z((c#1GHZC1(y=V$>=cLX9;Q&hXz7XWt;+hihvW3Gltm@umiJSq55(LTmwX%PT2X=q
zKmAI^S_$e&El5qQ*K8Qu&&*6)%^L~IKcF#cTn_B5m|wvA?YW9#<^dr1kPgJy
zqHlpquLUQ+^D^rky!AG3!O2ftyXUhf*rQp*tqrN
z36TkYi)3jn#Wh4IN;Le(yzHC}t#Uo2e8_~gS>J5#DS@t^GxY4ZRW_%3~zf?P_%BdmlOOZ(H9OeBj!u@fARi~Y(JkjKY3e$vH(z2tZcF5
zCgfX36dq?R;*s?W?1TxAnu++CYFE_{b^ndUH^&y84zgWYN!tJ?+f3@0w=G^k1x{Ar
zU?IpL1}{Pet`+h`N3GyI&7n*kZ4qA3+A_*cd^+~Ku@lu@7-Gr)V+<3zFIKX#abFfL
zx+}X!k^$NmQ|U5r*k1XZLw=$9QpL+aue5QgYGxq4B`L0F9B+*Gfty=JekGjg
zOH-Qh${l%bJbD3*nWO~LR<4NjUsAmTghJn>ewY&1cxb`;U}nB0D&xj|hFCZ`Oe}}k
zG5AXW9M;V`KEX<>9ZV6Do_7UqL%|g6S&C-Xg}bFo4qfZXI$SU6X+csWgkgsM_aVqh
z!kMM;8ke=Gq)~W~RM`drd?}CVZqnOZ!%%@xHOp4G9&$i85QNEF-L9q=ce6jZ?GqC#3woe5Hx>fBO?$8AzPq7&Ujxln>%76K%UYWmf|EHGKa
zgHm;8{!ZUf*^{iPAM*IdPplOn;`7Y0)HW-P$P&9D<-2!^M~~Z
zB%VhM7|Tv~&D7{LL*_=6&B?e-J>hh}MuD&H7k)_;v8m124+_P8euIvc!+qH6h=-Oc
z=g1He(o$@5*vb8PSe_+AC8uj*Q3R0SNaCT{0r{lF)ie9-+Pq17lTTo<&YVsIePYX-
zC|NLt1iW^~2Y(zy_yO8K
zY4!?{2J3q;-OtYg$LH(FPo^q9Qc*MjS1Bt0XWOO^E&F6m+eQ#tg6wx}Qn_?`A%?
zNSq_!`c>jZNbL`iMNqL}hFBZ-8Etfp6W<}vtABHhrw~Ni-1Qq2XK1yBo1^%eVr4Z$B|ol&C)a20`6QtaThjsY%%xAIQTtRx24*ovX(So64S
zu%c^HxgjP9$eSYz961`~vsIe9=(DJ-O-~It%ts&eE2-dWbybz})}gRS*jO`+_J6_>
zXMdFD)ZCxVFP?FldBd=@C=v@1gFa5eaI@dUlqZ?ku$TCN8N1)8O-!1A7{+ZY$7WG>
z2>?)6#Ne-yiXt`?QT96-7HikrV^8~#R0!0*4jeX4`Sw0~zpy}hJZ3C`Ux@OSEvJo}
z%g{*t=J!r6+I}zu-NSy{&(t6%Wi)xe$$xS^jZ3Z;+G#aqdC@Llqxk`FU01khN55(~
zG!h5A3+*h7J#};+JYAlCKt-EZ?A9sS70k83^AbYf0s9oTG|P^@(4z$(eT>X7ffZI@
z@%dA$cmB0o_$H@!Y0E9kkz1B^5KS1D(U97=Hq*y5xVhtx{8dksw!o$DCl8{lbNdiyb2Q|JpdG
zhDrd1i)Q=enrz#)J$bTi+fBA@yJ@PEZTn=~HSvBg_j|fOV!!OY*HRX>f$J&!-g>Qq
z;jsYYT{&MbXW9or1{QbACA*?FSf%F8`~6UaME1i?o+-@e|LX9x4-7(rf`4#o&5B^O
z#NH6+k(0OnOGSSkmnrO0xg74mbAjF9k<8}SQ|RorMdh@imiv#vMj`H0G??cz;rEXS
zo6xd~JAg-jAR{eN56O6g0MT{|6V?i?Y5aDy8=+avkME6DX|sHT+NE-8f6uB!UV*Yn
zKHd-?vHN)-y{@Y-l2kMzYJaGL1cQSRdCi{K1}(O7<5bhCbSXt|^XzJ)u(Ykn;Dqp3
zK~YvOL_?GPLih>=kiL%rKB`YkCOtph;OnEzal4W`CL}%QA^qrP#9^O{F4S^z&0dB{
ze3fA*C%v_Z(U&Um0~u|0`kArzr0xiBU&edaUOZ3TG%CS(jZvSD;%7&@XW3zO$*$tBNU{;=XGaHih5SR
z16&Ygt#qv%Pub!@V}Byr$QpwU86?5o(jG@*!RnEmw|H%V8}f~)B2Nuj`34NNWAVzO
z2&7Wi7(PgeBO9%~rnxY2ffyLu*RdT9JPD5E;rCZAK0VE}CI9q=Mt{$hs{NbP*^)=S
zj!p4_C-Js}pLA=vm>DH)Whh8`#0@U7u|L>q57#rVE=^9BlYht;g31!!*&>@t=wmyG
z2{ssVzvxc*AB?D0#uR)l4d;z;174eu#LIPyD41#_y2g0il6w{+0ywN412NUV1(F`a
zEh3s*5<=H&(0t7~14=;jiIhz&fEq)6#ERtFg-KQ<6)0(HeTRzuVp)o3zu=N1RX#{quSzNOk9Rl+qMkb25_#9+t*HLGKHq9C
z&ZzXfp)vhj2r3A2p2EwUI~;k0A!BoB&qF}%XD~MtCPjBqnE3JG8h&}Mq1@0n=wh5}
z++~O_rBi@o71L3Vngai>lc@iuZkU;^A-C&Hp5s7Hw0}`Uun#9Yb3215hYa;wc?f7a
z7IC*qmfi{;fETq+AW;r1<^pec!7lmI56#j1he{UcoI&&cOhm!=vKgPdvBO9DmzqKs
zY$2I2=~8a|29#GUK<~g5Uyh3{NM_}#Jj-f3PWN*nRJNHdSJ(Sx0acU27p;UVsr;{h
zNbJwRJAb%=fA3%loXqe80=tl)b8#>r>#QBi-yq6JGYUWI7dcWuTq^_Y5Rln!?TWX|
z;z87iZWM9IBo3CWtY1^qMjM^u>4CRI=nPvR>w;x?J;wJ(TeR_i9O%gO#0BYrq?TRz1;x)57}PMxXo(VVjpz7
zpi}NGHI{Zl4C?SIN>roG+1s{o+aEQ4aXS`(qJ@7IR+M@LX>
zD?Mf36*D~Xz1{HC8@x4db<;{maRCO;0{OZ@Td^L{ez@3QSoS8zmRU+?7^we37lx*r
zR(#ku%DwIq-x3xt#1rj}dBX)!I_?;@Fkk;SkpHj{^}rzE^&pN=EgaGx_4ED4^{=jNvxhZqUea!4}aBqX>N!y
z97fpt+M_+el%^E7ShMk(c$)mx$VIG?onpP$ahLdKJk1_azQPCsd{0%m5^`8~CD_j+
ze~xdRWB8?q;q;OFSf*XDd}qI}_ud=oDbALs0s+MVc)wX+=7E>!ZrQ^cGT}+dsE>OK
zE{XAQJao=mEK9Q6-XCw%yMJd{W*ENXs+&;I8qu2-3=|xUR&8UyaXrAWqP=PcwV@@2
z!=?-_v+(t|Vz~iBu{umxakWe9V0xL{Mx(iK{Z!hVBqWPfhwxB8W%kW8Hj!mJa4UT~@U-TN7``zL&J@g&}aptz96d4FmP6!1Pkn)DPI
zZ!AKam#odl!>O%1zQ1XOU(z6I%+PM#SSz+Im`9ij$Dj{*TNm)k@{7)#ZggU|(hqjq
zU!5v~oJW%}k=G1Edia4~10{KBxLyADn0`R8e`hC(*MqgjSu$A!j32c{Ns!45)$&_M
z(1~;Jl4ur*Kq_Qq;(vgAMcR%@^mwH#@?&C>7W&7rA(77WKMAbkqmPo2x)6l;rHpt7
zcI2{O+;`cElN}vvbDUM@OCu1hG}2&uR-aB@DBR*d_-eS-^`<9yHJNvwQUoWu9PJti
zQk+XN3@zJkG20SfO%+8rmYR!
zRouq$5hBSCR43Irw%ZkWhk`hLUHB0g`2?)kt5~jewlHE2#Q1;p2*Ni2L>L6kphL!8%nc^7t*oq%H_Xs`uaixF
zenLV>)mnLlm1y`ELh+>i*fcMN7w5T6^e9Z4ux_d?tIjgmDxQXOACZbxu#23s7Q9rU
z$yS{y*z-Tb%HE42Nn0l@DE?9o14gbcseZd(l7P8C=zmZmzXKc4n76X(^8NF3L8m;VIZSfqKs)Vls*T-u<7!UiVQEeJ|4OEq)stmq2jMsAN
z@W6&u0Dp)^C&v~Bri}`6!)reh_myV*CbGo;)u9X{L2lt{RlA0vnyQE-ALpJ)nGz9y
zfg>nLvY3M-wDMhryp6~logF%bVUxH5h{#24p!xn|r<6&kl4`8K*>lJf0cr3CI;(km
zPeW8-Mjs^R>|PP#|I&khpR2RAyZ$zxu4|6>_>
zSp4L0Q_vw<{Bq`*R-|e)C$*9f9*XX>TEkn21eU@dN8Up>Z1_(WLfEP0ZIRHZ&HSl5
zvM?%cMCbf5)QCPAoGn(@BxrP))sQEC5HN=6+tmU#%GENS;`^-roGdexsO=9&Ov)oP
z0)I_wau7wI!cnqroOS!>GqWRU3ioteneZ6iGtnV)EO>D3RL|qR2~6a6WBaHU*d@-k
z6<`W_z>$MLFg>Fqs*O~Vf6-7Lq^jQ%tCBqLw0NTbMAyjD$|mS~QT0VuH`s=fMJito
z!G$IyVV!q~tDGuPjF_Od;vzY$VH4P=(|^hFGjEYot39j4>Ek*UUpOJVY#CjSZkkRq
z0BFJ*XUX|XPt~z&F>qBsdHH5KqNw5=`W2$M=Df)5CU2Wz!~e-A#$!)=Y=pLstQuoM
z3#*TWQ30>q751L9e1kU@=7|W?pD5&AHUX<<{I0P)n2pQ3sfj>x5gioMOq_{?V1K*{
zBTHKP
zrGhp0*Oze+3!PyC(0i$Ym(7?xScq8Y@`VLxdA9a$y*fq=SzMRz){f~XsehHJ)TJpo
z@2;Zzjz2sT!wSxmA{3O=NMu|LqPZ%PH@v%==%myV)vnfGSeEbrN!Xr~5Iakm&4ex>
ztmKHN(
zc2O6p@04cC2SCBw6@2*N1?YiGjTy2X
zi9+RsBc#}0>xXqKd=`^1#Ayk*&OKHNxDtJFFo#Hnd$fA<;s)d|ynouxb?8{JA0+}%
zwVy@AO?Z45OP}?PZt?q%*?pvs4>8wq&un&>2Z3mxP}0d<$N0nv)QF!!jGC3C-fNksXogotucagzu*@=adcWUV+J)IvMSC|>whY;v~i*M-R+}jAqpgh
zFRm(b1gi&{URCdu*;MPUJyhhhaQ#g+b_%f|#)}3~@*cF*
z^0I(fW@i1_5gWaABni+Uv*D!s4hH#+Tu>$%qV~)We@ZoZxKZe=<#>0}r#PyH%Y7_8
zLM%uyh)NMLxqq;$XNX?1rRH#K&ExqLWnz5fb#LNp^k~1?e-`GEGCtayU{Qa@&Va?!
z4`Naxs_8ZTeb6&JK;jBcA6wOsqyXdTCb*08vXs}|KS~$9%S+=$=+%hThF3jiY#xEev`n$Nt1paWzW!=#LVumPKaJZxrNqE81!_&76}t?%
zokk#zLCZbI=MaVfYWZS)uSxzZ1D|ys;Ye^&ZrHRC>V~#(Pc(5#ceu#(KZuLvoUo@y
zJ*(^Bt;y%uIzK6l#K+9@Jx|+nIEDL$`Y9LUC7on5J4k!QavTCN5J^r)(76e~_VF)~
z1F?4d#DAY|avHvBC8V_^6sdu8ayL|+0XkF~`y5oY`^ohG;Iy1xpMbxsPbs{FPwgvL
zWh3I{8>AVG;C2!}HD!uSesiIvO|yC6yGL2~_a(d444(S&9+}j$z~zf!OZ8yxcPD;m
z`q2)(;|w&4Zd2dJqAoA}-GAega%_f&qdT!CeEssOeLv_(Qwly)
zS3V-y`76n%OPfq|UpmCuVb;oB?ZV(lh5Io|8^Sw}1|56w^5i=l#@5f+DF6nH8BOiO
z(13!E`asRRs5Ibl4Mt@pF9h3l08hIf3c9C7&!$$Zrdy#eugNfQCPOw5q{wC>Zq+F`
zYJWgGP!+@^WCc0HcUPRP@5W#&FM!((v1t}LRN9J2_8VEPHG2UBLYEEO5s1~o_Nl_J
zCUtKecBeDYR>}PpdYo(REjr?NirEqRd~@YyN1ATz1BPnM=XmcwyoN#
zo?@WcB~7ocE0+PLeb`>l_Pfr7E6+(um46JjbasJjq^{~&*OfrNjbD
zwBHm3FK2{zFjI9JItIN|A$#g+8X~{Uy>;)>z=nA)SYwLW6CTszf{UwkWtG$9o!k~j
z47$pJ5+-~BRqMumquO{(Z{EwOtC+W`X+?W;>I~K9U!1&AHxW`3qF`P7)DUDO-`Yd?
z)C+WWLN}Kn`WOpM*ZvuF<+)M7=YPi&qADfaP~q^EE$oU$y<5d=Q$g-{^#u1y2F{E|
z7(zWUA`LqV*x3Tw>R*?o)P}O2;(q=!FQ+aohl8(n(H1^l9D)9Z>4!xLpIV#Y
zNMzJa{~RuCzFB5!k2Tt$s%ZM8s`_5i@-_n5!6DfE+@DzPPRg9-+f#KRnt#bWB~1#3
zM%=f!knHVSp(T>I6tNCL$2iVlFk9&)cO-y5+OBd#qlx7ld47&0sUGaVTwr&U%AH4h
zt}MT((fRokI}_hrM4WT)A*35lsmE#nS+Z}z9l4cbIGWT-78x%;*kasg^oXlUQU&pB
zL`hQ73+#Nu$YrAW+vHsRwtsIAFV!xD6L;=#xAAtJ8z<(EBrY9k3cfG3>1R`f0veJS
zA|~yYyLN)xZl36+l#kWbo*cZEWA1j7nSqO|o$-ll1LAT~5zC>tgkRRe>CbM3)YgrB
z#Y82qN(IeVy{#SL}Qc(YAcsjaAlpp(2j2MIOGAd(xl)a}Bs4|B(zjJq{!;7PpMunWXmlJEMS2
z_9i5#O`sWWP4Xfri-Y2a{_c)@luZn<*
zO0>stG`KH~1Jumm(6cM8KdD7B}D`tpbr(o*hACgcv^<|`D
zVOq%lJ~xuTKVF(w+!Rm|WC|wI@be>H!8#DCdfzVrx@qW6NgLdq04
z+MD#JT;i=ivRd4a{B^{6?1>X92rgWTIT|!IU`$bMitkgTPmpeZ{H-2nQ>`;R>#eZ_
zSMEizvQ6D6{N=8I8oa}nDGqUg3nlZzq#klvBN8*TMs~6kRb#!w{Y<|S`$EMy?l={a
zU6uGThYLBiRDY|c7!s$9!T-w+GJzDz9}Mr3+Abq$-&J3c%2`oRFxHP1K=#;z&JPl5
zht|BPhIz5z6urBHBF87WB4oQ|Ja)dXW__n)EA8V76xgWyTjE3(19EG5nPr@z4qa;c
zOQR_BYYB1sqe~V+DyX4vRjt4A)ppxxwNKHHG?=Y*(SHh(Izq}P^{0-psAAA(Vbn}X
z_gah_NtEotQA<1KhW)M)Yg{Spc)_OStCF}C^^u(zk8EYMh``I-ZA4;m8;WbD#_iRv
zVLPipzuT#H&n35nOF8|m?1x$Gb0Nz|iB!b{7kGSF--q+qz$#y+>LaR*Q!^``QZ_Ns
z{~*RKUw>6dXNYs`SV^?E&z53thcj~u`A#kA-v7r!l}RP&B%4>&wfh4ye5zW+1D+l`
z(^%t2TYs{xl6o(!tMPWu{pwL)nV{?mqbCC_!9PTUOM2IAv>finfZrDJR`WO&yczaF
zCL;A(aBBE9g{s{h0!Ms~B=p(tAG&Kj`oAUswA*B_BF`c^D&9eIH4et`rn@g*B
zx9J>jg!3SBhv%?uE68Uqdt*tCu?r%rG;?~H0pxK3tVHNL9g^TU;9lE|bhpCqt+Jw<
z$A1FF{N3OG3CjeT`69B`wfu{hjwWxdr|BHpj)Y$vo8wPhw^Iqcs~~&M!-kkZNYxF^
zsqi+EK$d~8ategD?s=Z++gDP$6MlxE04Ns(61emlZMuq@PBRwqpF2BEdybrOwRT7n(*7jz}Zo#2w~sJ
zgGzM(61v58=FLZ!R*HV`tltQR(JVcg|CMz%=(qr74PjFeB&1RP0ruP>$i7hp$L&)V
zSfPWUtcIl90HCApfCtC+r~uR4ULTsOF{?`IS(#==xabHpUq1FN20O?`o?>ItWqE!nj^Ah2D}HM>
zU;HIoNXLZeNn5R65(8qN)Qlg}aEy@8?0Bb#|8v`!^_Ah$9_SsI-_{C9+K4=^OJp2^
z%^L-?n@%0PO_4S?Bb+h@yKv84<$q_bljPFv?_gV257{H*7b_aHd~0Z8zUI$|X)S1B
zaF)TIYf8Vknz4|5X+%RsUSRiADy0~{DnOVik1ZYkLPV}-nHmq*#sv0N{lZ7avk~yh
zsF&H8y(f$ucfcom&y$Pqkb)~a)1^$Q@EL54leoI%P`Rg%
zT+WBV=Z_LFPs}e|UnDz@Sm_<|H4h_di=H{Z)GX19jKu2)$p0vBQiuSTu(9I}UK;gw
zcOPHGYh#H36(|=q^td+^(hF53=x)g9n#plYqdK+KRcwu)=<%N}C1$lc$|bUWeCXcI
zMv7I@br2wYDg<_KOWmy>pMUTZteCV6XL-}DI3^BaF^oDmIvmc(&Hi5)J=dn$YdLCs
z74)5(&F?#x>h=>a`))U^e2g(gAlYr6GSYVjFB2c0?gp3y!1XGJ?7s>jQa&mtR)7rN
zsY?yXoxe;1%QN=Sx+##>VDf@^cepjwbe#2Q`2+*Pq|ILCqt-Jz^?!Z+@F#LAOO%N`
zn?GV-1S9?NEs#|-zvmX$=7X4-#!TMBi^Nx(v&e+j)YKqG^@-kZcn5x6UND$JqR6=N
z&-DD0B*$t~Chy4__oszO-r#HsYAd}9Ja32lz45&MskLvIh5YLqx(IyZp|>bxjO+}p
z2|T&<51Hq$rX`_ej(?ftBrEYxG@_!8_MmR~%#=V{zYbFx+IrWaJ+)bc+A8UwAMp+M
zlPDE8;9ZAm0*1>q58L>=*S9Hhb0geft!g%zrkI{Z1xI@(Q5|jDz506LK(803lxWq#lDZw=1O(2vUj+H3XAXyXG^s)V@t2iU=VmD%SOvfm`
zCkd_GvAaMm%kuqm{ykR9!8djH`0mI$HkGhO#z-hVOiYhGC4~s$Txy%Y2giTAz#<^LwRJIzSMsktpT(vG{)5;Yf}uZ6pF`l(#XRDn;y^6O
zBfE1D!Ik667_>Uv%|>C)jei+VxK!4#W!1Fz6Bb_%$+v*4K|r9{R*hk}zd9lg=An=u
zk(8*e&DBEJ>>nGOyT>au@+FRheOLL;{`O=J(8!K}*C{Ax8^nBB
zydlw11>-uS@QcEMKu-syOn
zm2g2y1%Ey353J+W^oSm3$TfxVphUi$uJVSU&&>qs@KA$^^xj8Rc0UNiGEBqrmGk3W
zJNz4>fI8G1o<*ywF^YCwn1b!i%)JyEm<`~yKRP+yP!C}jd+<0MTgkjRIKYq);R;T?
zSg6k{8)0MqbEkClCbH}suz49g<_At?s>CSfwtvV?I#52lIO?OaGp0}-HU_G{sXeND
zF?!W&&t)EG^Tdc19`BtiAz6Nf-Db}q@Vh7WUj#U2?d%}vM2o$_Aj05XMG$!grcrEU
z9j31D#eR-iZl?SQ&6kTE(yY0YBr8dyrRRsF`bKXzOlqpJm_7Nv@yPmRJqSUn8Ma#-
zK!2Lo%Ai*um06{jk_M09G3RK|H>@UJFmQst6Y^Bu>{Hd~+#$;g5k&V$Vxm5eFEDIB$%{;7WgrbWh?I@UXkQ-3mGxy
znX5u6nox%pG{X!x>r+yV%D))1_X4Vw5lOiw*j7v8+nl$>ZBu!WlNMz#L4cD;Tf|?T
zbg{^GSOomX7I}=*1`D9n{c=KE#<7GCWz`3vRcrRED^%r$?2&Bdg!;Xq+f2qS?tg72
zv(hwCmvK>z-c`!C9c+4Wzpqu#c|#(AL~0rFU$SE=HLS12z4tR3eWt7Fs>&6?sof-$
zC)beEsmES`yfH}vq!t&3azL061DX9_ZbcQ0crxGhm?&^1bc5^BGo
zg&$|!1ZEGMW1JFO)>!k2Ha$D1^nZT!(;w67hEl~xXuKAzFUgA*k;SZ6wE4ZnP5!@(
zA7Rz|Oy|XNxbOmmYq4?>X%x$dc9efp$Wt_%Hlys~>W^Pq8VCm~u+~A4gQx
z?Tw>&NCH@R3qH4}McY-1Al~$f#b7+(drIMIAGb3hliv8nRziA+IHz{GA}0y><_7qq
z;3s<%a$2#%68k)z5&x8C`hTWVq@rKFUK7Ca$<^b4)RPi-8+ns1F+7G64<^*?IROga
z|JKY;blK5^;CW}2eT0CeNFD`+Ny=wiq+3&FQ8*gULNcB9{kjcYOE2@2FR~~66I){}
z7sak7HxLv2{iWi;2Wy>GH&CLk29z3(aw3SR{fpsyVqf6&mM@YuA%Av3eGI+1T%h=*
z4rm`A-qCm?5VfEEluqjgNsn(6(<_Kc?8v5r7b+Z`gleG@L$tL*x&3|ZX?hU0ob7;K`sy;*0)}vM
zk`eL~f1pMUR62M~4zL^)s!eB#NAw%-z_4vs<@UiPkB6C-2Y)^;fbYI}Z%{aL)+L(8
zZ*m!1XF&h3GyAKd-`?=QpK#<>R{yRf8MqEg74_j>0iZMA^Co8Dw?jm4D+89fAj+Tk
z?#(^|tgnO+REj%1k91D8ul90E0N);|?`tio$7x@Sxu19G+9ZcQ@0B}$Vls^iGsT*5
zpglP4IO}NfV}C=}r8r4KKG*n==}?tJ@Si6xCT@^U#5wrPKwtCeQ(VSl#>fU_OM*Vi6ozX*q-HUkZ=m>1=U{jHU6G`LZk2WNyI
ziLpH~devK|pSneBgjl99(xRc&sQBV1ZZc
z*@oG};(tZIt16BPd;}H>J6vx6rw%+SHB?~a;{c85da&q$Gn~bp*j@|68D4`#w>Mhr
zW~ZsY|I^^m!c^(y90cu3_T<3&1gH-lUhE99*|&n5S6*hR*Z
zt1C`UrrXv>rcJCp1Ld7#7-uN)=a7sMoT_6m+>1RP=Ag4bWIu%x+?MhjX|VTdUiu`5-)EEXYEtN6+>P`
z^gbaF`B6K#7DdP3Z=>ySa+XKt>gTh}w93MP~DNOtK9n$$qZ4+yNpHkoTX#@
z*g?f7zbxC7c;o_dsgi;Q+EiF_Hv!pA@mZci
zx%qxoLw9pASA;0tS|-4oRtXfTigh*T+Sz{U#?Y#XMor-{U+MoraH`8p@=W@)$C-83
zxY3$6W%dE-P?vj~5t|jUgh%x#&mFUr8lklDMkIdC2@B0jNMtaTiaikrdg)k`bASFZ
zh|i}0+t|V-o9<4s%+p8uozXekHl)=HS{Q#NY|}uUK7x*Jl2+(Olo;k^*3_Ug`4Bxi
z0{gbW{Z}-x(CrXRGk-1;Mcd9oaS{@FWqhC}qRV`cJ{HFB-Mi{x;In^gQaBwmW!q6y}R
zf8`q7-~(=<^Yo{4JdB8?Oglf4y*kQ@$7*&>wLO!Os7FbwOPd0OiDdjy
zJG2=M&)@Brn6dD;-Rx8;VrfAbJdQ7;m>yHt;J)b;0a?1hX6(zv+@;cdDX>j{YX_b^
z>U#UqPf4)}>z1VB68=%*e}92aTc~tXrzA3GQn~J9vyY4rd2I5j)ro9cdNfI2z66s+
zXT9J>De1oMm|qn>zPp~itCrM0YlNka>X(%aCHC#aA2ij7R7OJ&I#tq{IuA)}Yq}S0
zKkJfP6LDPP`v%m`z@+_P5Zgi6?lF|KMG#>D3r=62xKVEkhz>m=&3{3@7+q&pB#N3E
zz*;|39M`mqR36zhzNy{S?o^>8<8oNZQ%`{T&_k5l7dctXi0vZj_P)alNle~ZsJ}CU
z<{kPckZySR)et+-uJ&I$q%UXTA+Bkyd)sgR7OrC(5@*%aU5k!Muj?c_9JSz4j{vE&
z%&H=FW-M#T?OY7|zkk+1Z0o%}PQpX%OglPMyswF2$w(2?9Df^g$08RNp;xKX|CE4dU4Me<
z7e1N@rm*XnQKnl4SSBV33$ZMWiartTjx!C9cfp`cv?oHv0DEto38`CiW+g<w8_S`4+tP&o2d1YR#tLO_WOH^8ywGHa9gmm+_|tD1UinRGiDUEx{$Y1@GYQ4#5fT
zPH^|eCAd2TcXxO9;O_43?*7O=_uReD8TaoSJ?Li6WwTbTs&7D(e3ez8;Wq&48GZ*^
z+0!u4(sKeNrR0H@x>gK~Gy*^i0{|l}Jv}ol35lSsp{~6t&`Lcok^n>utqg5Hj0ONb7l4$Zy{@8*wIKt5T=y>^3$(MR(bKj2FdJGKn_3xC
ze1r%BtzB$QjZN(TV{Qv;+GKkR?%d^r9Iml9UwS5%V~W}y4C1^@%V(a_e;^iSFU
z(nk8B4Dh$Kk5D69pygi$0OTh2_ST$qbWToAw8jp0_Ow7-V_IvAzw{}ZnA!oHfVSp<
zkAF{FLkq*d8slJP@X=0t6T^Qc__HMdNmG48D?7tKLEnM@GFpDL@+0VjxBp*ZAC0j8
zlhfi~?f^SO!~ZB_qHFh8tfZ{0B*0SF)XLt_O4myN!_i*X-oXx_^B4E=V`xD3FM);t
zK?hsgKQ*NOZLMu@;Us~+Du!16uaR)pLo@uLKU(1KY7Mjo80lKr
z8M>Pq8Gd}gy4vYF8UpNX9Sq%F|7rM71k1n%Ffi4({}}R*g97_kb}=g>Ab{gv^oNpv
zoBn6}$^SZ}6dxzl0BB|50x&Q%f`6rx2HJlN1Nr}-GxlG#zB^c0Nb6b}lK*Ez|JzH~
z($vD`KmGp8MaA%sRPz5L*wpU3sk5Phtf{@e$-lb#FJ8=E_hab!t&A-UKX&9VO6kuz
zviKO-k4wk&&z%LJVPIqVFWbi~>YG~`+SvhES^qK_e)QnK^nNt{k6Hkokbi)nytFXY
ze~rsuPQq6DKm$`NV*n!yD?r!QR@ViV{$mChSy%wB3?HLxVCekUkO1gtt$_9)ApmO!
zdv|~l&=&U3MzXR1==lFo|3Yj4I>EmY`$y%!5eI-y_s1b;2-c~rH222AF_=8hKwKijZFVx{?KXs
z5BQPZO=Kp{nf-U|5KdM>&4H-U^{(mDC!-rDfKj4Q_
z>%VJ%2(z~MxN85le=s)xfFDwA{{cUA{nPnl7w!K6KV&-m1Agdo`bSefbUFV6e&}-f
z3;ySP%l^68|GKy6{~q}NdE)(r73^(+=7uV!1|LuRf4E5L+S{5s|DgZ4bQnJPkJrC{
z)BKMDB!AzR|A-b40Dn5W(lCFFB@H7x2Y`X)VVOP~;(;Gf@md!a;jdiov5Q9tFObJkg5tMJRKN(2zIRyU{C!
zG#4l-%BkrQ=WUg*O5%rZ@wV0!TQ+`UAjh|@>mlWVgCNX5oPVKAtK^d+wcu6UPDpVm
zo{_Cqz&w@G9My~uP#QiGWb!&#VU
z`(*$Ah!z>9`eyNe%CsD2j++vFswD$Ni$wt$hr~n^ef79NnO+
ztB~%z9#nT-%)NFSLXPWiqi$MZ#%*`>;3jVfdu-~3bMg$;6HYn+#cL&bNEX_!72
zQ0Q^&2}lbsHNF{B!AeiwD32aQDtE@Gf@k4i^o~9
zBw3?^a#zA$tdnig4i<_*EgJk`Uuxo)TS-5guv1-wGB1`<{FqQD(qY!&4(7E%rx^;x
zgJ6=AUrc9c&3%t^b<+NwM;zyzM)*68G{A8AG@${ddV#f_GGl1sLLel$`*vWoXTU{t
zVm$G?%732<8U8iv}NI(80skY
zUl9U{kU5n0Jq8%@B$wdq_t36a0iD4|+$Kh4Hxg*Bj?-815eeV}6&XH6D`~1kkey+O3OIa!LUc?3)?9IVwx}2Roz;6(qbvGU~tB-XSte9`v`>zo42uT)jyu4@S-?|MT
z;p@EjIv!MjxT(FJ
zyI(md>EN{kGD33oYJxb8ITQ5BH-9gdvZ=g=o!Hh$*!JsWv@is>LB1cjU56(N7uh+u
zc3Jd7fP*oROy!5*CJe!7DA&x9`Nl(e)nLS^nV3G^no<@%84143-)h;$hST(_zsO=d
zQ;Sxl`C}2q^!YS!919LDbieW`RL^ohBD2s^75+4Qh)scW`Eq3IzLOcg;eTqw`XiD7
zi9zj7lfEG5OBaOs9s4MtMR@^8c*;}S19U*-)2@gojqu8a84cOEHa95_)1KNmuvwnc
z3ps%{s~EhH9AxI(EUYZ;S}ALA_O;VtN+#)y;QG98LX3zIcK9BXE`hCYXe3@@>WK3*
z)ith-qB1204&LXgJ0qpDNq<_oCw!Q2;bO=_pKC8d&FT1C5E737eP3p1?epigf|>8E
zNHiiPznbWB2?b+$YW6)Wy2DRfXd2s{oMX(bpEhZ$wxYNkEwY=tpOiG3m#&8I8nQ87
zVniSmsr}FeS_zg(m$!q*CS$fju*OoG4PK5#dKUYGG5I&Lh@RUM@P8M1UEDyAd?B3~
z?BzuX8p7XRtJPVB%Q#%7Zi+Gch=*!^_{Vcwo|00YY910gU*piK@Gcm|F9v}K@iLB^
z{^$bBDa{XjuDPLn9Ec+7#{y9mQVlTfULvo^BWgZccReNG(p1yDwNZqJR(fJ{e=_m9
zB?$~S`-bJ!KMA2EK7XPKReFREPi7=z`E2zb7!GTqBWyU);qV%|LG;8rY0%J4^7w$9
zYcvM_draL)dYe3iSjDvAl>X%ji3oOX@BoGx=XO_1^_o1=HgOIR9Zd%XYA4_77LPJM
zZ2hH%L;U{C+L1%cxO+UsrX4bQudfH?!A#7`%?Xq9^Ag<1cYhOZq6Cm}lGHnsrrW_^
zYYg#?68q7&_?W7^GldM!iSg_?&I!eiY4#g6*-5LhHE55`Pqvt))(eXhBtm(@cR>iJ
z%pA$SN@ycZF6`GEC>2sBU3`6GL`dk1+~>W7rX&?!rm7Z0@~TFmugs>SMl8`CC}_LqxX2b#wXR-TbUIGRCgU#SNGmjj
zSM9L@$$8f6lzIMNQG0P;{JVD0F~<0*YmwSO^%%A&Qzz~W<}KMZx-h{Vt6WEOH4Hwh
z@?tCjtbfTrtE#}p>-!dGxb4O`$Sar*nJJ~5BS2i@7WGq-g@OOTBe6Y!UPB|E5`b6w
z
zc{4!1%6k=s>GUt9(RpIP8%&7;>B>tzmwybjR=~hfGoiY@TF|B`I6~Y5n{m23fIr%}
zlJ&+JF|Z8VinDNTOvnR)w<}|8OKPIE((ed&adK8WLG&yaxy>93^oG1u)SCu%KSJL&
zEA8B8PL0r+yAn*NkzHw8w6Fk7q8Qyraai~Jo?EJV>icp`9wdu8pOWpOzfDX5&wuvf
zL95$TC%<<(J@pgJsw@5)h!VK+lWf>yq5707I@9(Oa^VCKWG~6@L}{Zo@b$Rn{)Lw>
zX2c4$(b{iEcZ*ZXMDOA(+i&T}SMV*0rwA^2#T`_GDk8@u&7qjRj_|8dk4Ph}gZvtV
z4Gz06heXkwZ%;4Z#&`q6{Szm6z7~5I|u?t|l_kZT??gIt)=@+LbL?&uZjLz}cqVyA=
z!w!`;X+na_jANqd}fW%yrq$zNtP6#K2x;_AusAg%3M@$+W367
zlb8Zal9d6Ic#{O|=b(~0GpUY;%jAY(H)8TXS+^p45?4;^D9AqGrP${=G
zMhT61e|BCN0&R2KD!)%BD;U8uH_2)V|hKk}ingjxe}n_-ms+xcM!SVfxODrJ+X$TcTtje|RNSbfke!jZ{Z
z86;{xH9bft8w~A45wBms>wMQ(bO_f^y-n?lT^>LFfrx_BWsK5QQW##cGBOK~_H;#z
zfF&)`+@YiH1oXAeQby8;c*uW+bM;ZBIai33yMt?lZM7oY{rG*8=X
z6L?Oi7sqaE_03>tdywU%x2T?d`)zd5v^#fR&{dc&!Tp=rCiPQv&AflBH+{u=wOu6Q
z^r#>GFUPEf4rKy3H*~rb{EE2}@MF&`R2`_ieAW07{3V|+yamoYH`ZQ&_K>+d@d^9U
zbW)ZRLJnUhaU{=m5b96`IrMHFqmdzy8gcN{Ldx<56yq@VdiHpgbqstnsOhRnC~~|N
z4(Nrn-;Hoah8*1jz*B$WR~P-&xgK>i5@cr&CeG-r?LaOaK4)cGbJp+{zReh-4pT-`vIPCo152$DTU7!#;g`
zivxp&i)SqJigsH`HJy*Vd{^qr^esBQ9omFGs8H8NT1UCZi76wh(IYT5(Z*zWZxcrNydr
zXSBNwr=Ld2s`GzvDeR-f=)a1ST=KaxBP(#{*0*9!2kz0~O_o99(RjwQ(JCBp3A45E
zS+RxdgDg6S2tzWawq%gPE*L;@nxPW4>)Me%5A9|0D8=>^n@q74j3cg$-TlPlIA2UX
zyV)*cXe&Se2KAE8pnKX*c&)$D@ynbIVqt;>m!U)wx%Yo_CH)Lv-*&1AYSy<|zxXum
z&1wlAft2rrSb+tf8H{ZAz73?w$zOFycMD|4B8;F?#*$=b>J2bv}23PGD{F0Ww8
z4Sk|PQoBe>`LGcXEezT9d6!_H#;0Cp7ODM~afd~CyZcg}>5=V41_`nuFS>?H)L0WyuW52@KkJqU
zp-0CB!e?ua-}%;MO%b6*56dB*jyI)}8DD>7)Jry?w2|k@1974(nn%yj+u~iOQfO#V
zu^D>yjZew9JzP(A{xkfNGn#$=4mQcfpmr+*%9-mVI~xfE)sL*?X1pI4P@Q9zM3wxZ)E-$M~NJ)`NJmlqSx=^$u(Gm*R7d@n?bwBC|ybS5hr8=PnJV)&4_;w
z@9Xah#kL|`8Sm-uK)w13pyWRVkAk>=^u!wo$o?0nC>(lx@53_x13i~n_S7^9x
z>h^>OXrnJwj*dw_7-Ud?CH7Zz|ID`p!ujh(R!byRX|Ln78$ZMu*N?bb(7SXh!{3M+qi}iMh-QX
z?9amhp!X%#>fA&cp~X52Tu({D0+0v{uJ7ts*K>@X9F5`-0iFoUSG85aTTg#|t({ZZ
z@+^#=vGNYX;@_h=c$KlcIkYWPLUvveJf8%u@j)9IU?jFQ>Fx4kVNWWt=fL{YbGIm8
zdup`MGCV5Pf49E*OjAr3BPPT7%ye#(R$w~4Dp0b}@Cpa(thI$(g`-9RAd_-ToGug%X)V8Ax<|tA=#5E1lG}>{Cko
zLymbL7ZcNiNu9vuOwH-6%711Hp?r@!TLpK!b1Gmo7O$ugUTW&`k=cXef^DPbK}9b!
z2-C1`=8P(FqiY)W>ZX6*X>zT^_a^m%ZKgYMmgzD!g!RIc)_@jxrahfD{P4~m`E)m1
z4pKO)bJ0MZbv$GLB$vmOigepBsFjD27(B5GJ*%`7w-itREBcr=(2K3JJ2(CO0&8gu
zcr)9^_sfrK==m$IUjSNfm0vAw9-90u-;l#*DD#*4BU{STnIeDDhc1NWv$DvjW7Pc?
z?PW?DnnLpWnY4k0hv2Z^Vq5}(c$C{D6oHZJCfZsQY=b{!C*1TgN;>`G&KVN3Nymf&
zc`u#7d!|iRx0>i@#@m}-KcAbOgq7e5y+;RL-Ru!5t0^Iw(LKF&j)o?#z`q$jD|xc~
z5@1Gti)oyFizR==9u2UX{4q|~EIO&MYgH5$lZ(xI0ANY~bK`+ck`tKe}=sFyFG2ougI7JL&Yxg^PbmJRP#Mt`}J@g;%^xy=Toh
zRlPL#@vKSlZpxMuAS25#9)4SS+9n?ZAD7&56
z#_clJ9%FwoHJb#x)s
zXk@h$9JTjY-*~-I5x8I4SXV(npX}aMNjgA*rxbtH<1gg_V+gOVe=AiLg!XLgUIDS~
zF;oa=NZZjdyA*N+73E~cR?BMA%Jv_W$-as``2sq2aEn(G1_7nk7@;F2P5))rNyv6b
zpjWVKtVs|>#9rjOfsS%ra#c<3%Fes#ZV@a39!bla^?JXJtZHF%R&6uUFCZR(BIdG
zZV??hV-qI@gv-YWjn4x;&e(ZU+!iKWGjV@aVXYsq&#e#qZA#rk5qBA6F^;R1=nVJ_
z`9e%DTY#ssqPjc?H>mUsiv~+xF%pkzht@OdZs9z1+9S^S8O%@UCZ5hJ)XRRgXq9T8
z`K?^P*k{vJvdNu0Y3J^g@0|!lq--NzbDopA+~c*?`^2
zRz7Olo;=^^(e>@V-sl=R)41*8)Ne4)PQfHIb;nEtbEH(som@f9x+An91)gEeMc6X4
zyq@auA${h;0rWo6or@wd?2`N?JrsY4WM$-;wlBp>1#)N`>UMv7plr9(*mFZX8~eyb
z3md$@-STs-IW>r~d=D1XGFwwF`--pzadrkFSAXm_)IU(UW=!`zS4v*^TJdZBngA#M
z6%`GvZ;y}uL*Oe@EhO~2gD5x@oALT?_58k02JMe0*VmL@?KJ`)3hGQx4$*%G(!=Q$
zAZJ#(Uq5d-c0+a%WMR}7$7nQ3&;u#4W)VNDe35p(C;RG3Uu4CT_Ub5kX|Y@b+IpB<
zZ-?cdbTs3!na;f@n`C8q*0V#1F>y;cI6kj|31`O?e&&{qQs)!>S8pn~qx2H(ZwGKq
znNw`aP1t$r>krpR0=Iy`mn44-_V-7fiD@6^V%#K<&gjl&Q!7q9P>Vh>v3TnHXc{Zn
zWdfG~$OI+ud=BEL1^}3va6R$L1@5TXJ=SZ0Q(Go}
zlYoj?bN9(Siz~@YB3H$YePCuuR~kL?d(3k;`Y)A7xdkf(NxhB>`WxLnw9-IW@-a{r
z^VVr=VfDmpj@eM>aFKJykt0qraAJxW36`w`gvJaJpM`62&4IC%)6@0KPd5ZQUDE=C
zf`n)MQA3S7C*#$SEp&ed5tm2?iIz(W1)lA+tL$5(!PAM&P}^`3sIK#7((-u)f`NnA
zpH7kVA!fPOSi}HL@I^mVlLp`T(!}s;3Ae_%L{V0|Yw_eQlZsW8{BR1Vf^@n^Y__m;
z&Y)+_X6y-)PPMZVzsB>CV$FD=Le{l;nZ>qSJ(JsUZ>L`WS}1>66R%Pl#>ZOspm|%4
zL~4h8n`I~Cp893KK;;pKZZZFh8&_Zx{je86&o%(vMj;np0$Yb?g%1`h-VVuC+)YY>
zquC9AAits;Q$|OOqfuD*<0`xj!xpZphdMdlJN=kg@7L_?$p$vrjaeTe(Oh8Mfei4u
zmwH2Fsb0Dn|D1p5b#78!O9HK5id`-VGz9cS*?R({P?E!tgG2wheZj0Sb@QudsaXdr
z>nQ*U`7W^7j37&Rfd(a%xHMerSMt|cLgu*?%xstRN7sl$AXZ{#$XazB!J=ENCwGN7
z<`81&CVe0t)~9LzE0YZaG-ysK%t|J)xGv?CpJ-C4mC%2wQFB?${CKe;*^dZwnP0X;
zkVmaLVil_I7`M&PYwX^6%M}zT?(;PrjEl*AQ52$m4W!P54iu=UXHbr(Zyz3nM)kit
zojgJkQMbT*^7M!1l=Uh#8vg>X(R2|1l8>G<#X0J1pc!E;r;__qp_Y!q2`%C9w!&mf
z5gfVzjc|YYZkh(X&%1{PeW7t>?cODD0J;Ny=T|GsYX$ugSt2LNqVn~v@a&6oxt~BC
zx#ipWL?WB0VlgLO^H*eAq}JnS2x~cMnnPP;#^9&5@Pf|!erlGH`O>7U7)mh=0;J_j
z3DwjHn-u?X%V;;XJTG^^CMQhqSh#UC%`soj>bZaYFJSJGrMoGyqlYl4I_np8&3?Wh
zPc_bV$QOZ(4PX@E{=*4+2xpq`?+Y%0)Rw+OeQb%myH+AUN+DlTqId*po#-dmkemQu
zwaDS~Cs}aOeIW-m1K#zEgrI>y&&NruPQ8<~2E653FqL%
zXmx)wD)(4=_LFlr#UrS@*Q(U`uB5+tiDv$ih=AJ8&)T<3VUYGa#&@$
zl2i?l(#3}|h$q-M62(y!L`K}}+r*ewKR_m&WZ;&05eVlMUi(;dvB&H}8eM3S#o@Tr
zd*4VY#SBt19tL{iZ(QmZyZCsDBvax?zq%9%`DS^KB#l>GJq()uVXZD48FWVHB#v$DI+$LO#7Y~
zE5o@w26Zc`&^t(`V_{%r448jnOWUwRc4qvQc?8Wa3NvmugW^sgT})YbjMU*p?fXmT
zQ5qKN3`T@Ie}LEVWGaX%>g9efw&i~aR5Yvuso?yTKx21rhi6|=%v>%BuiSWbl;?Q$
zJ4xnmOvoJWVSD^2dYSA%lgJ0p=3LP_*_H2_oFLrCAepv&O+b;$T`h=Jn4aV+DXi1q
z(;h_v9Evp4_?yC_-Ev{QvuaAmH)*Be>jLfzt*5V==(k+@$#BNwFS<-{s%d`}J5F8a
z&t790ufmf4rH|_J!U-^>k(7#{O13YHcT$v;pwVP2bro1xJ=bRpQiw2lE5tJdt7wHJ
zO}wTF%9mK~p(qHVgTBjhH!non==xA*70Sl)hSJ^!VH>#R<~|Zo{5rN8j`yf^h3B^3
z;NR%iUa3unrGW^<@pK*A^GSc7tqFxv9Dn8%9n)%%{d%avZHNsZWqN?kiireMK>F6m
zJsu@^ot3SsEBCvCWcN5nXj&$0y_M^lw&hcsX?;rV~nD?FR1)@=ma
z*)>9msH@e_YWm~Bpl+X6Xd#XmXwsGXz994kE%A~|z$rPgn%@8MzeQSeB)-F#Z`j|H
zIlvMG31g<2YKupV}mQ6KdktBQZ2P
zE`K%^)_E~FfuBX0vAv%$StkzEaXFMW=ONk$w&<35jT7!+fe%v3jG8?3vZwUdu2u%o
z)Pw5(HscN8Cix*-4Z_w!5u2*XF{%Ud&OfEhavAqE#7lxyE|1S;TolZem-(IOiqk%?
zi4U?!kI`RgE<}F;GO@!9`y#763zV3@c7rM0qMbo#`+}&2(ogk1Oby7bl(e9)vbL5t(uLXDXDSbUk?ILCBQ)>U&Zn5QgLdWU!L72T-(p(O;Q6yDFvuy
zTT>M%Qu8?xpgA;Qb#BB;AyCLjp~e#wQhxUPxzhW^j$wZyfK%A6T^@SH6l%ki5i-b_
zYGl+S(qsFGtJemN3r8l9{VmLMB$m_5}9-9K-@Vqwvr*&p#
zyb~CbM%6Nv!~?E^7(rJ$rMp{PG2R**8zvFUw!(sx>C
zZ=f$5SKyT0w(=5PjwtWh^Lh%=(}))^((u__{E?TYXOflpa5xskA%vt|vX}W2KfK!b
z#;1_WtAoW#lpCbetc0ThQ#9Nr`s?Dush%FnnpJ;MWHPS)rrO#77m&|=`T5NOw^2N{
z%(KZ2leTU0rOtCgD)tfZL!QmNkTB6GoMb>|-;!VR$1a5)2F5d=g`h63b5BT54ZSN0
z>u1Yk$X9=W5{bP7I*f@Ep)%YeV19)Sct_BPzR<&zSNEeM3o;MikbUXs_7wbHW5C`d
z1$uw0Z?8$e3wLI>H<25hDDPRG$y5a)+EzFT;LowfJIGM@jfm&NOM55P;J#iTn#3Dh
zVQ9>cJ+x(bC2?_jRseuO^fCaS#b%sM&i
zgSO8PQ|pQtDRq@5QW@Mh*bliThRTdikEMUURm(7B?cW~Z{3TG1k4~vO;&Y}If7Ou`
z9x)8?vnYxYX9SUS-YMD$h-UV}U_7&}!dIfR>k=gy%hh>|FM+s6M>Ysn#
z(m+Ks9rp*N^mq0}hibrg3mR-jE0nzDU2+=fEu(oCJ*qi&a6b>^U>?R6**HZ`&w_2u
z`gW+hy@%8tQpJX9TC`+j9(fEjlTkY^5Oa|<5nKrVY;I152g7*q5iamje(ort&FtRv
z{B6y+mu#pSHiwM<5)T~T{nFwhaMORPB9G?U?l9q!!pZA5cGjJ6BW{51i0o0i6d3rP
zQaT^9NMtP}X3jFj^iqI=7|9tSfGUGy{+T#j`AVeNw79R6bv37^?aCHQ?q$)Pue9N_
zc3w+gSZZO8PXaA8(8u$UP2kuXL=fN(VVQ*$$*|Q(PIYP*-QP~7Yl@#o$AEu;fG>C|
z2H9%a6%D4805a;ka(%A)CS68OJ&05gm{%;88$1WxzD6RXeiSF!WYWJ69F
zhdyQzG(ytaGElUjVBP`yok8Xq%BWyFJHgTo##x7|5Vf>8_T6Ax<@11{?VCNYpbHt02&tz3kMAXC>G~Yp9-rnM6&;z@8OC
zd(!8Sg+Oj>rN=L{H`-_1PFpAQNJO{Nq%Jd>R^Y}B>fuIgJriEW#4&$VQ38kL2XW&(
z*-B?!{3R#xw&&lbnScA0UwIFg0gItF7mTGHSq_F@g9m=v$&gX5
ztHM2zRc_zI@J-$*t=FY1AbU3HmcgxBhcWLp##Lko)*dfx;ukPrAXT?+7(Y%}Rymt@
z#>E>GesSoM&S8JGs@audZne9Qd+6#-QUXDqj9DuJZJPb~+dA7x^L@?++j!F|xT0Bx
z43kkH4Z)p`BzH6$3)W4vupmM-T+Qk$-A4Y;<=>g3`bRQRP{8%9p?MyLHb#oxhZrpH
zhR5H|7yjH}b6_Qc1rBRLFNwc>Q}?W%8()iyD0vemm$HA?EI<-YaC+sB4{O_3SFcubi9urNUeeIj1VU{n(@|*ojf^YNeQamh9M_hP2-WoY<%ui6kYI~lUBSIjr^>aBj
zNdb~@@Hv0G@2$&~2NoD;-7jSyw3Kp#L_9{OhKrtg1}7|-#{g?P`jeJ~Fz84j{cnr(
z@Q_=T1YpNyeb3Hzv>_-5&D!Fdv%L4Pgs+UoCcEw@krvIfcsSNwGu*!mhp}kYADaX@
zukgeDdJ!GKnX&N4F-)Xn$xnU*)pvR{b;$Cvjo*I{d&!}_8cn`Jsg9th5{dOE(eNdi
z`1O^cmd+N&hpX~sHYqm^2I_kEc5s!AzbpLa`po*=I3Z=T8T$D3#fDd)V`ppJzAZlc
zY3x}c=7gH=GPq_8*d>Bc4K)#T7I82T$OmLZI1rP~&lB5g-xQg-i^{4XjVuuSjY9XO
zWCVYncq9yZAIdV^+U0tD5Rr)s!0}q(L;mjh?qnRST}kJ|Mah^Q_S<}5Ns^S6;M(QLjTOcfw0&h?Q}|;@r%fitxDCX8i;EtQ0RqS@m
z^uC_S)HFkO
zz%$QLl8^DT+_=D5nQOh~IFkW;o49bpz^lB-xDURdvZq1?{jpzuHM?r5H~{x&+8VYOEr%9DO&zatF&jtX~YeiIQ)
z(x=p4jqTm^EEL1$ZvN}*h(RNUw8gzgRQoBn=E1HW!P%rI6i<>^rV%96_s)E>ZKLd0
zVg&8oL5ndS{IAd?Pz_4ygk;zeO<;cwqMxV4yUtZ+m}SwXwMtV?cJ`=7J9PTMoqRXN
zd$cmumbkIZ-QRrO=`XSnFNy;!DXbpS?5}m#&vX6RE$A^#A2&Ah-?8pOD_3NBg1wiG
zKi>{dq+?ofc_5X&atM@S7d0tZHmz5N!~-OYc!AIT5L
zyU0mzE6KYi1}ZlJJo9
zlvZVY4p^j^%fY2c+cI7sRFD?R^OK!m5-fU`mMbf7ymeG#jYH=?Jja~Xa2sUGZ*0Xw
zR%rK#-Qh;q8nA0igU^3>C*G{|G<~3A*XGOXy~X9?S)oW4f_oY<(V7|Eki2%jc&w*`
zzqQ2p=$Dzl4!mvb)lo6@^%Mnec~gbuf`Sq-<y?hb+55Z75H{o={FZ
zbcc4g%)k9WdrPR+u9`QgQqlk|!cA7ao@pItW%qX@>F093?bg0n#XdiuEW)4p#j`1R
z@$R>63>S_E4t$~YgI!@J4xXO8Jf>^5i&8~V8cN}Vip-?O|LiVh9=Z66H+c88Yd(&;
zH>*t}*@1YEpyYoH#XQ1z*G{gXOL#KZPXz0Qx;|AMnGcb?P+0?^_kw5BC7`JW-9hrg
z3XdJe@3~&Ar26Vw2?PcXgbNr{R&4cTPe9}0Ty_ZM=LuY5>ElYuzU)s9CiJ#X1x+?B
zjW+0_z}s1jPsJTwbAhzsjKlllXHl<7ETRK8qt{W?+Zum4D6NcZod)F$>S3?Wfhf~X
zM3|3RCeyphneUSx=TK}l{q4AqoQi_;Y;Q{sCDZN%UE*uL{$%KP!9D%JlDHvQ)J+XG
z2>m0PuGP{HptWH#+*k`n<5GeBA->)ko_;aEm`Tl%A
zCqiQNWki2oQd){o%*nLTc>Om}ILc@Cj31#um%8Ou8qtDnikp72HFAhUOr0k{wQ)G>
z36Uy^dlkwhwVhfQPr61V4SnlXNbRY<`e{fwFxH5cksPI#g;hB^=0lCdxi2lz+Xq%OO8>q4EQiTpDGMoD3va?y)F{Z880~n&NkPOv>-hVwi8?
zI~C5gQ5ng^?oG1rQDKo}GKj){d_B!lx8iv~v4%=PgH=SS42SniYTVn{V;;i^kIk2~
zdggy*g{96Jd}-MndAL{r!LUt*Pc{yiMpZIl(v^}&1RAA?t-WgCamn4wxjBs%frquD
zkliLe|Nh*@``s1TI`jzHVlU4p1tGLGY^AYqWhhRpOqvv`n;&5iM4x`AsK-!3bZ!cn
z0%}V=X7WLho1%Bpk6T&VZ@y-h3-fdlhM0fFDAd|XsQKZ(SrM%|f(CshA+GcXLs6!T
zzKTry(O%wgS;t3v`}97++jHbL$NJUh+i98}WTmbL9&&D{8t61!5u0o!2nfIC;BYka
zFBKg3`B!46J-1Y1Ti`esVt79m22{{@a=vsk%ZRW0SjQEc<#-7;Ou49Sp@s_RTk3zy
z`M3wQpI5;>{BM7(fC#lW8O{Xa*Um-{_~Dt{59Bf?CND#inWQ@nn~YAKJ|f=mEb7o=
z0y`u8{O*Qe@ts<31tT3JPy+epeiNc5%x;RZ>*bn40TXt9r(C3q
z5==+cQAdcA&i$eZqEl&`w(3hs|KaAXb)}fitF;)IQ^#=Q!G;xR0xI^RUyQmH};gs&3$cZ`-3ARDX$B4c47a(BMvmu16Xr0{&JRZ@S`;
zwJ-Wj`yoUrgY?EOWf~@zN_9m4d$pc^t;g%^^N~1yy|(I2V^`IqJBzhS6$ccK|64pu
z=Ix5;uP@&wTp9FqmNx6X*b9HLp(v*o;0aT5=+sSI?U%a7f}4;P-aH=8GfZ?pDN;hy
zmy*7;pi6={t2Xp%^7d=mq*njHC+Icne>R63+luT&P*he_20s8cw!Vn2=hx1k;0#m;
z;Br?ddu+XO7BU9sZA+z&%It972(t-^#%n5X_{b9FnM3KGbe0aG>%xD=Ze%eiEyh!;
zaBwu(ozIpeb&Z6#2>&>sl{vEQ|EBaLJ+T1yLfIRH6;CV2-14ZoAFD1Qk*+bKlB{rU
zSgWRL6Uf-As@UYcOtoe($G~58nz{Erxj?-$;m1Ooa1pKj!)NL^?JLSj&hOoX;rXJL
z;>SYhcU}z)m2(g`!2o|*vF_)i<6-7^^{i7Ywz2{I?AMv7DC?^
zKp5#axexQI@l7=hg`g}+
zcCITeW?4=;kV4c$>+S2E%p!uzw|3tr2sHXsMG{5UDJ>+*>%*!>`sb#UO5dBZZHK5_
z+UP#y3>MT@x&(iRYJ8kXyS*1dej{8H4cQzx%=4e0wzmPmPgUC;G_^z6S=5Ybn1dq^
zQ$(7fI|58QyOLp`N<&V?nJhY?)mIYm@gPct5tY9bB&Rro@GS!#s9}kk#zrVF)-r9R
z&HQW?9l3=h_M^z!gi9*d6MhCyI!ul+m$3QS#f7-oIWvD)3z<|S3A_R5hkK*66wJu1
zAv&5kDd5H$-gcQD)ith~YH^`0AeD0uebo_m=#ecBLy8Cxt_6?J
zqWht)j#J%r?ca`3B+(Q_*m(qgZ>Ii!NCasl%{Wvvb2IAM`IJSetjhLk)24q=?$ep|`Z}5u9RqWefP0?XhqT-V`Yw|YYI1BrlnqVmm7)eh7G@!;32T{~
zZCpjE+$zAqH6zlBV6#6j$;wZsE79#NRz?rr96V`$1cBteh9SshBtG#x%A_IVUXlE>
zN1MHIzJ{-8@XBR(219{7Z18)}t%D90?L;9-KqY@
zsDzgsxfcgPuti-4OZu9NWUia9roN9uiEz9UK83BOhXK=;g!;p_haF9{csx=eW{F?@
z9AEe4%7p@=i#qq{)CvL-OFe2ey3^t#qiB@euTjbaSNyKLleA;6tBVQ&qt(@3X4P->
zHI;ws>Y}*nx(V;bl^GI?LfW6w0!Z9ba`^|SXsz3$Ekh2Pk(jKBo<8P(!fJr
zyAavy>6n}>C42zhp)q+|jZJAhvj8mHiZi#%X06w=*DODy0eO(b4QJa$BY8$IZ_Sb`&q5xEIh
z4e74_Sl8F%ek|;hG-%hmddlm|>j7^^JUGGp+WH;-GJf+7wIC_HHR)!9C0Fo(cQ=1O
z(E8g!=GODxnit$BEw5$j)I-cJJ*I)5INy_I!*M+%Gg<8r;Osl=EIOdg^!
zb5)&y+GxUZv-m_4WJPEx-!y0l`@22*)ABR%VP8hv_H4Jh?U
zCl*8Q72LbY+fI$b#S}d$B5+kgw2yx%S@f&Td^XZ~Nv0P!VT}iY+F3mbW1IRSv-fn9
zK01o|(*)C@ua#5n*Ogl6$z+)$17q$`Ao0;s)e+6%g&!2>4<;XEJ?^@>yh`st7aI2t
z<7i^ITyl-X#6Ip4e`&AEAam#fDm+K~R@VnI_92*h@(}}FqL);v#wN7X<~4s;6637e
z-i<5@%_oNuYVxGkP3&W=E`&zCmuSvL8=t{B=Jrr6$RtMk$_W>Hc|w;UCxJ7aqB)P0
zdDzfhbmD762IQandVb|7!I1a;EX%|tGVx9iLAzY(td;q)zb@mHclC@}&Jv{@<~oCz
zzG*+lM|3Eiz{ql4RxCu5<=KC?g#-0`TfX|Oo5k1BZzR$2L>)cLfl)*3&yb)ZGXwlG
z)bw#xktar63Sp5d^-iT#dBx0T3!@Z!IRc(7v)%qr>--{bT9s;T*
z)+%nQaBo$gqaMOqI*S^5a&afVxF89EBfSx#2o5r3uGoPo?{WjscL#r0cT!pIQD<05
z`^warO}{wr+V=EXlP%jB#oJX|4Y2i|#|SN?6zrUA6r6DroH5#o)cFR9~sbNJw<
zkyKiiQeU5C$jaf_uvJgJGCvW3PuLAU5ySYHl`QKR6#pOJD`tOj!h>A&NBe#g8Q|*G
zycO*WT}%4h&2iA~HI)sIVpeUjNAJduT`HWZ<1)-t)^EifXl+4;kos~1{hgj`5NjE4
z5FGp%@sxw4b|yuHXHah%PLqB@e6p3q8O@V5cw|0~ABdX8?T?if6HEl2Bt`z>^Jqc+=Vwg8<@sV;zj*yP-e=>f8T*{anBcx+iw=KS6tU!L{eJ=XCJ5QS)1#c<
zqqNd2HB#pdhnM%5kT#%TB9t_W?UD_zM>A2Y&=k!$g2_f{s%|uv9F;f^=IP!B@5D@D
z7Td~TbrJOpE2=DAbE;u?y_k(d--DbR22
zAc`Q;Z|Z-lOj$R>N_IwRZ^2r6vCaMr2{{I%G4b(52t7K>H82tw>=^TUz;DI_CM8EX1W#Q9Idtmh6`_K0A80&YZ7~~Fy
zm}jH^xRXw;W=Ii$)JfwP1axpEZ0Pz#+q8p$qq(omxL&8MuXK6#dIo;35uWpO1i^JR
z0zQ9p{W78N3}uy3r6sFY5OKX-(7c~T9q~p8Tt5uIt8Vev(dnfVE|y#vqL9_+y|e5l
zM)>+XreL(aJ^YZayS8OBFr_6ChfV^{rSMi>^YPQgs~jWf)LP^nGEdqwCRydKlCI~RQn7#GY=6(D6z(hI*iuranq|mL25^c&5V^!b
zUSw4*DQnFDR?j=RFI5}kAjXv1TDznF5W$%il`0WR(qz@JZmbu-W;0ku3IzutpFtZ_
zh7S12)1d^5Klf!V`&)zM20AH{%kD5e3vRj{;&eB#5tkbia$h`>V4K2i6=5UJM}U7t
zIkACntcpCbwt*{dHUfM504Wa#OtL*!F~asz<)+_Cac4x+(VbPo21*6P<`dwPB8GTa
zZ55F8$Xw3=t-1l%B?L)4-?LU(IVOP#9*d%Jrah?=dbTbqBH>He)4l|LKr6guxzb^fIzq8lX-4t>HL_GEjn
z*eN7R^1LYsB(V%Pd7;w>lqzsc)9(T`;XzjGBoJ`kA!~k8NmxZXOAatUmhq?BWDv|<
zKZNXDpJ7&1ZSkYFNG8pd@FAJ+{Pa|$3c{T!2eJqj#+ETM|I-l+zmIoEP$PeRlE6Lc
zUF-`u=}CmJ=B%m21|NK-+76TT@ry}ve4B$G;@50eha^O*EL^~PwZ&c!B2zOIGOzb9
zUH(FrsPuADTy=F6+Scm4_W(#wQ~LSi(C{3nLcl_4N}DR;7c!Zx{)PfAcPw?ioYexi
zpJ-VE(&wK=l)CV|e|XKj>GXf~IadrML#_wC0>~w|)hlN?f%w!9i}UL+&mSbL)>D^N
zfEq2H2F_CzP696QPmorE3$5nLc+d`KBJ*_3#0@hc6ZR!o
zd*&;+HuO5RYLlGm2q8VY6om-yzF%`?Y2FY-YA|_8BB*D`@&&G*S3G}#TRVvMng2HP
z@Fa)^WtBDAR%}(My~CKqrT{t-7YFfa>4iMF%#Pfzj?)Bcjc%%Q#GCP4OJlksd5kWO
zhsAV+`rKFjWeJR@7x@+JCiAn&UzxWt{b^0J&`HtcA`Rm*~ZhS8=}+nikVUUYbYXklZ5x#?uMYrpvBDJ
zqETfcKd?+N))Yao*f|DC2`htID+j6eLRnQ(^QI}7PfaOpfk27ViJoN@h|Kw2N%`Xq+@79Nx<4zRH
z=gPP{$aux<{g>ebMRpt&Hu#9c!=B(yQBN)C63+@J^iDdcjpSdCaMlX$m%uvHc2|i)
z!X|i`a{SdcCAz=qCQn+FjD?jic1+YAbi9Z7`!+r($g6)A+{4<}U=oW$D{MdM?lYYG
zt1O0Xjv*;7!2o?qJ#_2a(&>}+DOVHox$9*pc+%JIrTg(Ryl{z5
z-AfcWUt53qUd#c)E%I$|OLZzN{#FU$=-w141u}qw1mekznMHJtTx9OH3Pyi@+q_9^
z)Xwmm4$%24_Se}B5a9I+`gcCi3rlv-2U_J;sCT>z
zRZ$aM2v}b2CP+gdLTYh(U+;>YP9zJ9wj4++AsK%$6g!|DeGZFN@JYj&yN+8XfjnST
zu$^nXL!Ma<^9SSX
zZnVDoU0h0$%*5hMP%-Y}ZG-8R^lYT}SDZNB9UB02eS2nW>eWH~xT)Q8Q5>m_o%~7v
z7x#Y`87&i#j4z0$2BOdRXIL$lA2WYKlQPFC$0qdG?VWeZ>rKI%AcC6|Nng_h?s&d;
zs{?Jlh#;UXXBZz+p%x$9N^-we@zZ{)N%D~W+8ZTE@Mv4^xT>fdsN!?Th|lF@pAUd(
z3m!#1)kf`eWk1FWV*;(8wM2
z?KD-d{&JNk_%rsrh+&-TxscSVnDnFl$Mi#Ez?5#c?#=lbyxxlgZnlR$seL6%{?;EM
zl{nZKP(t`?w_^7r&7x_AgxDWITXJ{Ga70m7Fq!b_@UbQky;c7s<5O*L{*$m_;evl!
zq9O}5fxv#(#@Khb;r&WN{s$lG0FcANf9f3AIr*IeK^R#T`|C=VpZ{+aY0<6h4ge99D379Mou
zvGa~jD~@Y^O`@o9KIA1PV#XN`AXug%5E3IIN5d0lS)Y
z<4~nRT^uMe>$d6Ru#bqzmL7pOJC;TOcScNozG4Mp?=wtQM>nl~S(GDZm-H}?1Ak<3
zH{rmSahf8(_0t@jY!CHsn>~N=mNzmic6?an;HBX-25>op8UCc{{{gZr59K2OYD}BW
z3$k7mOu3Mu`4$*iTDH|Vhfv9JZ=WV!4|f0dV8u$nfibmh_srxYWzmN-Ct4Xeh;1#w
z0)kw#?<#vlc=!ohLVC`aknd5SP1w?Qi1BaESx|I!pR2^>kv+u5VZ(p$=H_A4_nf>U
zgbvhhM++!|CPE07liCO!$Z?``@dlXh&F<}~oOP_zpk_nyM@hr?VJd1C!!jowB&Yct5z6c=>VOv(b-Rq8Mi8Fvs`^MxZ
zFhZ4u-gX0%ygH#Wn6rP(zV`HvO*b3V8QlYymw-JL_rh^~cq>YeiH=;o=drzC`_aqJ
zVA2?9p`2_$UmQFRW)$Zjsc)l|wlSiRm0_iJ+U{Syvcvh=`yIx2A
z)w=r{&y#57j<3*b6FL$3%EQ&-(03PYn=jVzd7?VOxJdHW^N2Q6syP$uGO|
zAI|YpiZ|Qx{q$ezSn-eqKldStN)`4FsakZ(HCkbprKKDe*pHIn2~$K6BX0Ad&IA8W
z7J}E6ZdU?cf0~|i$!i##FkEQjM!?V76}|f|-+gcO?NFO#67c$h6$PL__WL=%c4-M@@L9slUjRuH)+Xto7+GiMLvr
zi+g|h`->|J#AL8Etc*!TA$1@7(1I>{2_3fkd{yZ6(WEUv%==FJbnai8SAiD!g7kyh
zS03(LcgzkQX096A%k0;VZx&@p?-WL3DY%IzGE2k}GKznjhPh+izuzKBk!$BsGe;(I
z#sc5)2)$583jy~B&sP#S9gNs4!EJB`v%BLXqL+sb(hz7tO=ub5D>DE;0R4Py?eecu
zmi)3cFSy-Mj=dmagNa3LPR?y`7FMmqj^miD$l8|o%9FP!izJaa3olQHWN&MvCGHK=
z$_t$T{|J8q?~}ob03du5y>)xgO0)Cj?jR1^IxKi*$&2mf^CoC-8mn~xx1np;t!sBb
z7tfl|1rdFTIPS3q*EcBXmiM0=FmT(WY|)y=;~TyU
znegh$r7kj6rngK1_JU`O;@Eqe80({VhOri#%%p!D(3ApxQx+P1>d_OVJ8>@K3AR<2
z=cU;hy;i}FJAy;8gmRMC)W_<*u;d0ogv<}!ELQ?ZZ!OkM635xQQQig>419WmT?o0V
z1lnmMhVo;lTpw=MCApl*!{9#?IIaJP&))x(nH0pfpHTKzHk~JX;v3rokOtqIr5vs>1`gNw^|tAL1J*PKEMTy!EKwAkzWt92f#+o>vc~eTBmU
zA!4Olh<3Q%V$&7DnC&PAMRf}_dv{dBs~q$XQXe4Z{eq~u%td1i1liE19F*n@Al_JU
zNF!gcmkST*?tbFxuVDnP0HvS_ist@*{t;JwpnouL9>n}%kMkkoCWv4G5_nxAD08qS
zu}T@Jb8qcMaT|*?2vzY{Q&yoUy72!36tJhH0p9AWy?|NZ+r0OPsWOBG5Mn_;&}|vs
zlj)_0Fi>ApFCo~U;lh@-+X|hMmgsx>d$UK`FowNdsVgl%UKH!
zX*;WEhcuAwWc}Y?)NH9k@PEdD4XXX%#6B!g7E8=pwpJYO>jqaXP@w|^xd=JBHRwF+
z3eC#08-c)A2`@}Uz~joB+wviVOgp}LZMZs1gyU*=%j4&pv(}ne3h6p`;_;f!HreA_
z35yN~k$vB!_dc0gkO%!g4!%$0&cF507)N2;PA|^NA3IHTt4z9SR_H)|Z3Nr^TMd9%V+QT(v4QKSG
z#T97oUzKo9tPdIFlZ&DVHuT1SBQrD40XmL(W#puz%z1dLJ2Si^7{*sg5HmimJ+d(&Ph_g#k<>0!;>I#c=l4SfD
z9DymR%8Fyv%|}Bnbe4sU%j23S4OP3r%z7*x*qq2hfPtLoxpy^BR0IEiH2s;o!Ju0S
z;;3XVD6?opkI>oEp4lXGiJJpLO^py*=mre~>zx-m%fczc$Kz}m=>?_KF!r&52$T9p
z)uqKrmHzg{<2;{
zv;A|SQmFZ=n{bAxSkF<2iyz5|8-11-Uda&lz<{Q4of$G57hvx}GHaxjj=_9e{QKyL
z*f@^AL6ksu(u9&_Bgijp^6~Rks*x7a3LcuqfKs}J7NlHlDjdRg^@JZ!kg4X6@8ILg
zas3X+VZ_a$4y?MI+0!v%MuHq!FX3-FC76JKe#Zb)!4rHfdMb$&uZmJ`z5T!ZH)ER&
zh#R&_Y06fs&Ql7jERu7ff6o6}|C#z}WUN*a=ZfxCfjC#rzl%J~t!KDfZo1nz1Dw
ziof9~O^ZnFwo|Kh%&)62I9T*-==y*nKNZl!t1lp_gQt9NuRc^5L?UOsV~nfL24~Y`
zjQs?=+Kp^q!eC4G{3(gBAgEzpHOm^2|KPF1F`#xU{C>^#SXqFN`j^u>b@8tsU{HX)
z~A3u1=ia;nWEuMhh(dHQxLl#f
ztAr(`+j_-iu)N+g*C^FyF0o5O;!xcx?1N8e+LbcTeH*2;pb73F8F307U}@IP%(R^-
zV*Th2OA8he-y7nR
z!ibF?-}q4S;sZ>44BsUiU@n%vPJRat&ZJmRK;-JDUw@679dbVixRrWc8EBvH^5mfs
z<;Gj>ZoQ%|LmY=7qG_rgEqlK#;`nzO8MowXYAY3=DH5%WkPPtiytq`(qrI_=
zK{mrFkr`cztr_yVUeHff?BETPmAu#czT``!?8SrU9~Lsf%o)T7n9FSzt}QQrxM!sh
z$#85Jj{MP$hI67|7BF)*_?0y0byFSjtRXm9p8hgx@JMFzhpd9ePp7G?lf`l~W0>4h0gx?%Kz4b1PWhJL&STw
zoaGk>^$A`e#xSQ%yEB9mSBP1xbK;9I-R#akr0;0!ZtQf`g(JI6NPK8Tx1ybCL2L4v
zXT`O@j8UEG`eV}6MmRIa--ht-wQ(YW-z}xWelDM=ts?O2hfB@C}
z7KBOan)B&knIUPZIVdIVP@~07QfIuhu_xpoXcNCb`hj;IelXG%2
z!z#^_oy0Z-Jwop(sxm=xl0H7n(XxFe?bSrt-v^WkRStnP%rM_F&78fum#P-R0S^~C
z68-(k1@TZKXU?)D=(q7yI*@7)$&H_=ku@xb0MM>f0ejdDL!9L3-f8xL2ijybUz5%q
zWru7qz!oI-A`fzq;Y5f5!!+!^&x{2J2})APyp`OC)An$d@-;rAU~akdz0JM)MOTt*
zjJH&qZL11S2Q}({;jY0QGC&*MSHUIWQ5B#qfJ4C=?nJ(5@2OrshE!TIG;@MpK|%4c
zeo{GZgfOwpHfZ^&Il>2~=sf%Ft7b?GzLRGuf@s+dJRn3JU?T`1xA`Axi8&rLAk0VF<108J!Y4hqqlgHda@S>0ZcCcafN_XAEVsI`_Ik`GCmey(v
zKn0F#lckDJ%v4TwEYqRDFud#{xk7BQ8RP9eX{(N2Kz&v980yhV4sw%E;ZH3Yq?%}6
zYb>;9J{O&KQ(;iswi1x%98K){w0>L)o(er~Y$e~0#))BfX#CdpE~Tagp6~j@%Mu+d
zCK$0d0R|TAf0qkB*PY{iepsgAHkJCo0QQ}p^(AucsHn%H&+ybZkgO06do-dNLnO{6h!hu;=xw$U}b-r0DR-7qp)Je+Zzkyt8gS8`J5E5l+f+`ywNq+j9<_J`3q+vEQqAtppfWe}_
zg}_j$#`D0P{p-kRQ;x=O(AfopG6ThdPAadomt|7=-&7Hfar9OH(O2lf1&FsBv>$m1
zLi)BAvmTcu^qP^>JEOak7*^Ueod#ovyg-$@fdJvrgxuLuC0*2jXDdZarnt|i%bUZSBT4Nbq;G3=0jzq-ga**V
z6r}4#VFy01Zv-N!LHtQFXqhxV^^Sr3CGrU!4`UorayF6l`bZ!&pW?TvowL5rK*MVo
z_jEsX8W>6a^B$+t6>F6kU;SZ!mbquGqFk_PouWmVgh5`kx%N#OmuFvtDd}Iu$xJ4f
zaxe)`qGxHnu{D?`yBukj03!TSH2S+-P$#|1ajZT@GVP5!HQN$8b)NqjyD2gqGMy!(
z1Q6qle^ky|Lxppsx?m7-ES#k7G3QOJnlLr0yS%dM9%c$)vH#-Ts)DRZKm@A=IPtW$
zSg8{#YMw>@-XYGK%
z7Jx@wtR2xG{IHdtWN(D2JF{*S1$<>Mu?R%63iUdcWng%qO1x}uD%vavsEkN(Jim_hANMg(Z
zGu3l+E|VULWku2af~kz)p^90^*L+_Lewy_;EdoP*8|^$$2H{64=&5|PDY$jR9h6Qr
zv(X>guI5*s&1UyJXPTLi_b|hotrpuN-XBYLhEoQ85V+vF1@KY-BuO>U*Rc+IZE}Tr
z3i=l^{)qe8&;LX4IITt@jct)6PTUIoUHtbtqyWjddCSE24Ww4z>s0=p#JmJ={^((C
zsaN42dT5-+2*z%BAd1vXM`mN$A$1fHJIiU0qkzsoF#qq3m*+tZm+L^58&8vzurLluE`8PTo)%1X{6J}
zNSvg|n|{a8N}l93rr(iG9bLrW3Ucy-{T|L|!4<|%e}QLJbsHpCEbdX_k6<@MZ~>5E4QCS;b`&~ml~%d=9)5sB8b7aQdBBoiljTxAH&td%?(%@1PE
zaQWx}3s6gqKc+SSHIz>0?91H+wf$S(y;RxiuRNc&(!XNKkLmEi;DZaPxB1S~lg?+!
zq9TsfKbL1*#Nsi@9+L(pr*tcx^8^9x#bE@DoNyX^Swdj}XQ`Gz?OiF^*M)9R(RsYjmOs~kP5^*Ng2lhy#z(?L2`i{#d!57dtP)7K_6ln-
zj33Yf*UP{9o~{1bV~TgAy0BEuQ)Q&F56COfPO)j!(xnN@EBv|~6>Azx;^$l5tD#ut
zkl;rz3P(^!h`y5#>5N?DVjVH&{DXok)o3Uew9FFLvoxK0b5UMSh{;AZHm4ydo;zP1
z`bM5+_U%Nti;DhtgnLv<$BB7QcbP?4`B^Jnk?bOdDx}UkuR;0HJ#22=tO(I(3SdqG
z{TnZbWLkXjWZ?|YHfi>h1}uh2oc~UMlb9nqEf3`4QL1mqFTxJg$N&n6m-J<(Ha+dks%zA-l=SEwA=@{
zCn7Fncxrihu{UzOlWTT3^`o(<8fL8DjGL>eduy>MSe#R}BT;p9
z=;@lZbjXXs9Wt!{$#s@(4z{MF2bVD5(gsJ<^>kZ~qf1OsqkyeN>~bX(1>jv-4>L5H
zmeuGy$etayj;2}#ask>tT6P10k<77&ycds_7L1%mqB8n$`5hE
z-|d00<={+FlxMi)f<2nGZRcLy=k$MahiN)~gT5eV9B#XumJ%8y>&LkWDY)FfUl?){
z>Vf&mAbE(MQ5GEr0B^~!F^~kK0qspUKa2Kza#fJc0Oc4Y@q}tOb;hb;Lg1wrE7S@F
zxSv56W|Gv&PXLc*UIDjztwkV+|*Iyh*1*8
zq{eI2T-A^+Bb0o*l5~(dUFaET=Q45Tr&@y{lHW{!ieo(Mm9C!axyfcMgXr7nfX7&JPvr>1Hx>F
zDp8m-HVgL<8}k33ktwaA3<4dQgN5~feN0TO99(JfwxDQW%xo-dY5lpNv;YY&D1P8zb9$Clish{k2#K#Dps
z(ULrjCTrTnXwDB8DNrg!l)FR~2rexEDK;E*Ai~8&6$lapGJ+9H1uzaw1c}m)BpMNt
zV1$ImB|t5FH6|1>tQ8YVxa)9_AyXKbE#zi
zoY%&9A7!_%_(VQ4ajdMiq4jkcWz>bhz}J>0tWaWQ10*7YbO?iEy`U|!fY&xf
z_zVVth8RKfz*!=iM?tCLWs8{4by|?;*$bv)AbE5G<9lJ|z&K_I6#PfBf)+)k?+>yZ
zK|UsXlV`0HfqL$-@(UGD|GaU1cVxTV3+qUa
zp?RR}b-@COCu)#Sf?EdP-XJu3SOA$Yick>$J>CP0ArvesX$r_D)RZlH=LOzV*7{c%
zrkO-b28f54G81t##GQWRriEGOpb;
zsTk8Z3Tz%0aeMOqME~uFXuTad;A_q!NJtny=>lfoRq>aElx>!XoqZz*xh2GmfT&f^>3h7gq)YjZSbv$DXiZP|kR
zwjSeuniXxc6Wipf`^(Kzop|~PqvrRG+sC=Q@L{2>
z;u9Z}aO!;f36Z9iO($B?654Q_e{>Avn_%oN{G=yBF&Wy_x~?#KRTlNp+1BXQ?PUvr
z)BRyScF9B0J_^@ufgP*{S}Vlz0>IFSX$j}>xX#-1Rp)Ovw-nM&cuMe;ac^t86FywB
zu$)D_Ttk3-Y|Tb|p^W@J8q3gb`{MSiO+WvZCY!C)R?O&PJagEM7(OvR@I%i@KXk4b
z-rG{57TgC+N@<@pcE<9Vgmvkm)x32+c`puWlf
zJdRchcoLaKIc{)dV^P`Af?royk;Y~@plB5f2gx)yEX5d{PE8M985@M&l&Ff$z_!n}
zvQ{+AYEGtA4Ab3hJi8jyt#Z%M+vcI&^8K|js-m%98y+Yw%3tsjHY~01OF5zT>2Rs~
zJLd9Mms5v2;;o>zhBs|s38*XHvAz9td_0D6&nd$O3wu>txl=#I
zMNB@;)X9z5v`S@QyUOmrzMZ_0k>jUbyRuSg*}6F{;$P;Tv2OW3#rkwhw%I0e()*pN
zDybLtk(okZ+q4?MuJ3g&dNaANq!~G#8RdB_Z9V2w0QWPIiZOm)0=Pv%U8-1{?i;R<
zPXKC?XZws$xk~SS6rz0&b247Kz3p~U3*P?h$I!n@p>i-V;=9)?7OFRTrpT-Tf65`Q
zX6IzR8EwB%`17{YcR5Yk$)81^2YYCqrw5eAFTW);C;Q{|^leejaxsvQ&JSm~@hDnN
zyVGR#z949rU`?Su0$zi0XJTmqwoFw%QM>}Nn_HTIa@}RAKHOMkJtqeI?7dxm$FBV0
zWuuMr&_17~7w7Me6*-%d6FWTzwT)L#5qYpzM_;|~Wy|?PU2p4LS2kWrPY;jNL+kCh
zH+~Tsy3o|$gGGROt=w$cmBHg?OAN(}D=mI(v(>eCJ^#25AXZQMM@XoBT+6ZoXJ1LL
z!kx`F=|iwZq3>>h&cWDS=?RVJ$I53=u$5q^mU|i1p@dj~kNia0i)Pss5Cky44LOT7
zJH<{->&L9BV-VMEV|LR-t>Ejj_0U{9I5uWj@ulg_S!ix8KBI1W!I}mv8gnSHto(U{5tDH-Wo0iKbc1W7XPSf$)8bMe{}P=7EgpRSo4@$
z=7;@y1HV?zD{*kh+#(eet)?tZ+tJWc&^C4+S;@X!Z!@)lCvT(=S2tp;`Q1M);-7wL
zo-|}+uD|N>nu611ed7>ms7~vKcbOR-2B%z2U-C-qR7pS&NTmV6Ruef}tUb_N)*0mYtHz}`r-q$Szp=FsH
zxXLSvGV#1EC>fju}qJ31UOF-;5z331^<
z#z6J~gi1hhgj)#t@J^vY*|b88Km(#+VZix7f&K_U!Ya
zT|=BTKy;CpoGS=7NeepvOK#uwrd)r&e`!ml?0AJi(06_o|tpJ(I
zv&s8&FX0h^fud0-F>FV{-%TK2<87u8bi8#UsqLDRKPfZ1QIzQfk8D0V-5s3aH#ta+hZNTHky5e
z5ke6jLIiD}Dt@NHyxg+>7Sh9uRZk58<~K3Ues*~vg4;X1zSXjRbz0HU4q)6ux_(fR
z!Rmo|0&mv4c5bmSAGAx!OKM3O%4!A+BG34s!m!bl@SrHRKz}wr_;Zu1D=7sPg9-iy
zz3seK;0|d)TpR@(Tt_`0UMGDW{;%-t@Wat9aKyO=_S+wb#E@jsQo$!HI!u+<`#2ykFT{K7rl87B066
zStNxu$VJdT0$VrFfxjgpF4ZZZU`H1)d#67&AVk!U`d7%Cj)rTF96m>
zdtyN>kPs3bpcy*{eSdhg
zQ0KnsKw=>J4)1^6zaiK?<*y*Hf14aYJd(f6H6jXxL&Yx+G|5-<4WwH+r>5CS5@h1+SsQ#nq^dBp0J>ZvaPr{^07Qi2hOlpO+U3%IS-SlmRi2e{3E73M%C8OFME+jehe}8-}UmXGl1>?GqLTsQoWM
zNVNazHxSA$7GT;DCP*Q3+1Ekz-{Di*IG@YsKa{;<*cUi*a`rEP7Cm7a3k+1G=C@Js
zF6dhg56lI8^G6NdPxM%96svf+Ta1YC^RD!%mu$RNbVfp}p&5s${e55~QJUlmyVmHc~W&ir#sM+g6W
zcd~Z~?r;a=uzL%LWfYY3-+UF|t@-J2@k3-E()`5e&3gkv8Ur;6Vc`RA@%m$vr-Q2X
zBu(%BM#3Egb$Uag>8sS5;G5dRpEDk_-SAxg#0f
z4EY#Jv>#nzAXWFTThdeFO-{O)T{v8VmsQVt_+1m8Y%Bp}KkwXu{=PnizCl>|+x@mi
zOR8Lo!QIZ`?bnfNCB?xzbDiD18}qi1GD;A~QGq;|MON&anY?X-OX;xIi&$m4E*38ZRv)siL}nr<62T@ArcPsP0=@>{Z9;(y|U;3ZWNKaqZMoh1JPkCxWA2mpu^^zj*VFA3kiKJ_R=i
zuIE4m(F^`SvS{fcAy}Z
zy#^9(*1zvb6|xK;js|<}?`6rMxuM%G+%@51sI2;FQ1{b^jaywjpC#$&_&L4_GM+48
z{j+CAKK2K^{Ya_yK&m(7&0@~2wX!nD_rt4xUbi--u+f9#H@&#sk5Y2De&&uh05T*K
z&~f}^9j*89PkTWMS=I5IC9!YP^~Ysa9o5oUBKH+-Gx+FnUW%;%4YXQ-SeK4xi;*d^
zU>uc2*Y^mj4om6I!$tPbexG+VkCrfC`hi3pSZC|gjOfAnNQ8V?{XO9Dr1v=?sw}8(
zZ^J#J;Lo(g_&@)(5b$NXK-lr&KPN0#L&upG%~ceq2@%VAW&wOQLO-^#-3
zh$)y06!TUTIh7{8u!pW&Y{~Y@W(%JL+5m~`&;h<_bP=AIB_rD%g;-C_glnxzvQcZ1
zyE6NOXQ>yijGEkMB~(YGNaJWP>&c;o>BVzXy!>Jk$qHh0WdGV{e8yNf7$1svrs~ti
z@KcbST`rMj&*q#;e6IuGSfR1-W1(H-nEtoPEbbp9tL$taUAFLEq{KNpie7kMof3^w
zXX9xOVGG;sNvn&+zG6L~W|{MfSdTy`8d2+1UR+71jV`MYRIWA$*f=E)k-tI;rrVTN
zVmSQt1;VZt`cZ3Xq}&n9UZw>w~>^u^fWz9avxvS9Pcz{S?=
z#3x`>!vZZI1!7yOUWUX
z-xh$w9Oia=I-Te;X&WrwAtM%`PtUq?
zJtD_KW>B&We$35Y_T%0VzMXuMy>3Q)GGgd14Qvdr-a@V)&4U@X=pL31d_Iiql$&eZ
zbNVT~y4HtL8i7LKvA`3h{umdQkR4O0j1!Qk`b43I_}EJ0?5?(M~Y+Mf@
z>Zc{~ve~fLFG6IG*r9q1mRz&w^@>yJ*JK^E4e{IIM7rNOQuuwr7NPR!8kD!Xqm)Ec
z={HW%X08w?X2iM1&h8Y!46JvtjE7lqpN{Is|Gt}9_b)OtT!@jcAv{S?)j1O^I=x|y
zA!^7&dTG8ELgzW2)e2E9ihc%MA5sB0h?gXym16ntyhYyL0zFU0s1tTtoft4&z^OH`
zO_APLzR1JI7TfnewLRmWa$2N_yw08~-fD6~?zy(;evaSCyTFNQZjxO{vDN2|id^`R
zIJFzN;0_T+GLyRwf>M<)+uiXHORLj56TAcf#8~8==GR6re$u{4Pw?ZOWEcSiIF#%4
zOS%$!?Tf`Hf*&sbSdlLvA84h!BXeqFrRO(&l4%t#3zCRyY%J>Nu_myUU0}4Rfdrt$
z+*7Er5%psEqIyLztovrcq>$kpl9do(v{}MOxdI!9P&aU7@R-_NeXythxlCp*+4^vl
zfdyTJj;l(8c#xfd--fhLeSHONt6Q9C)J2u@%*?%~R}<#_X^ife(fbQ~4IhPXf+|Q7
zAyo%^%*rV}Y&K|e%gAq2?}bPHXK-Tr8z4Ds)Gxi{6MAa_CqOy%AZHozX%~^TAn%@3
z$d~hM%OjaP^XIOAyOOyJ>t!j5n{~4X&b88)M3a`CS--bz;Ihd`&7%uY@*8i3@=>JL
z;*ES!b`v*W_jwH4roRc27%H-nkm{f$-!HfBzU`coG$7T0?Or6TmxzZKlizP7(Oc@`
zvt50DX3QHse{?nO@{WIV);Q<9KsF9QE$7yc&`aUcyP^gl>+QNE0;9w`!t?&US9Ly=RsKU>EApF9
zg#CEX`C$5L`P?sm`vFW(i_A>ibtN+=(9zYKkYZ0lKUY702OSljlF17i_f7*bBM}tu
zDi@w4Lt^IMbh=(NtEF0B!Ppj{`b~|aSDS`rJPKONjviy$lzajhbO~N+*zr^nfdJ=8
zO1Ei)aj(G&%x4G%-I|Tlrt5l`*)kJE)8y&uc=kJAegv7A7Boc7+)P;K<#om&zUMDg
z2cx@tWqPz>+c>r8If4YQ1y`=BV78^d_~X1`pA7#xrsX^iyD5-F(ze|#pGZI>uH516
z_nlY>{n4pWahjNvRxKr1@rqO0MB>DkhH*|xhJeXzp)PGUR%$kW`
zT6~D}MAtJllWOeSBkbC!;YP;;5|!3!1`
zzyS@u*y_Q3K5{Z_gNwQq4drs=uJW;yq7;NWGY^VM8yfCkDH$ttk#h6GT-Cc>iFH!&
zJyabVX`~rwbe1-c^12Y$(5+%sw?x}XTYM89b?<M5+2lO6wVB73qDWE?+`uqb}{<mIxNaXj1APuL7`SxP2)olYO{)5I?7wcL(L(|m@J5*|E^gJLfR
zj~PI#F~>N)bpPYe+obWq>=DRhLUGCT#g;jhQg3jMEH>d>RiTu
z;Y#ao{N`UhDkZz-0ujbFeQ|sEOY#Hsgengo$O=eSx49GsG8y#_q
zr_?D7<|%Vpj9`PRp0{^{kJh-7tC-w+kStBrUj|RBi<$+x*=-VPcfmWMu&h+SYI>==
zcN|8nRrYp2HRYe__n=7Nw=_8OZm7M23dgxQk*hZR$i8xzKwb}Vc{uRRJj-c*xi5g!
z&1G4GA{QD*qC&b+8duIJF89@PTVL%-^f0$x`S^b-umxsJj_TH_%sPMnqn>SfsC7-5
z%M;=fh}C6LGlG5NCp^`S_B5Spb}U^kTPbx4drZ}9iZx!Q9@9wl_34lB=Z|Hh%4Stg
zbQn7*zxKANeFAZsoo4{vAHc03tONxdIP)bl8FGBC-U|CPX@*7#=4>IZS_(rJugE&<
zicrq?jtzaZ+n7l%)mHA|MbG!}ai;}FI8g}TVhqFn`Y6-8S;W}nK6s*R|9(At?1ZSc
z`NsQAdyuVPTQcL=*Sh%jJ=FD+=>I@Cy+u!VrGsTFy>JS7=^fUa3+i_^p>zXmRH>)V
zt$~+IOc3omjn04xghBw4hlp|%50nDJA8w%gshuPo4F4@*P?YH7s6~b+czo;!Q118u
z!QRZL7{{e!K-V=rSiHgjaajnNVs(H`@>sZ{rfcEWDN&eNp1=z@*N~<9ZmT}uLsj#C
zyvW!7Xm|B#pRNX+P$X6_Eyn?t2(E{xqKIm{^IdNnu5BV`7p0TJO7}G?K=;*ZUi?7N
z-|G%Ip*P;o?M_om;U7>=Q4vL1vEsmY2zcx(#abk6;}Dr1UE6~#!S~Med<377@Nr2q
zZTuqZ!v4!-l6YQOIMp#=ybkSR+RymSHqkqbbg!14`@HPU3Q){xM6Cn}ePGs&z~JSK
zs1zMH)tr(_)>jRp=CNkECOI4n4RfIM=W!_b$%W5%u)(*2bi1V;fASobAGrJZ4KGAI
z|K~l7Xk^fu^-@@swZi?UTAWZ~TUx#<
zPjJ;l>x)Ye#jruBPHQ+@zo5m3rH@gfpoPpPij5HatMtdnNQtsc^U*^h6vXt`yvYQK
z!0xN8JUP=6LXY`iX!Rg&%}s&j5)8E*A#o>juBx;;r&;#}wlWzY^-wD|{9R1!yo@j(
z@_`1PaiJ4)CBceUlb6h!#nzijMJN75k)Lo1yufyCe7iLpp!B3*^$0tyzd3+|_2lcM
zNS4uOZ5Yx?&T89|E=>E&bHQ2r_Eyhb1l^Z?dA2FDLDe+yWbY-
z%Fif)Wytf=Yq%oeX46QD?pPUL!KCYc*R}kYYu;y*&lk1U!>Fr!ITN8D`iiOo)30wr
zNQK}Med;Ra{_9+k%nP~dOG}Q{Q|Qd06^M;%oz}`=2HF;2wIMG$IMN~k1|hOgOVqAe
zz>KSXZ0Yv8ybOn#GiFTJ&+h1y(vV?HTy5KYS^4Xg|gRcZ(U?j#)`;?t?!u3%0
zUDRdLy-2ZqjCTl&Zul8fLg2D!*EDG{6|V1H?KXM^D=sZ-!kN|AZn+-${IH>^Wa5Fb
zf^x~&I#mat-@fj5tY3l1`0j2C%=36N0O~>%vzuY$cA9;|nMkSq{tso=H8auMC>uuE;4%Wc+AnLg4P*`KpT<1((@vxxB@1bsIfW7d)_2l#e!*;e>T%$x1I*^g_Vrt1|YtUc9aJ
z5Iz@lr2kdra{oP}3{n#zC1%=mhT`{=L139#EPv^qLS;08cV0*EgbveG+QN!o`ej-E)X1W9r
ze_CV@c}CY(;!J5RFcapo;j_^-k+|cQ(l35~e;~VEb5rTxDSkAvb5_1ecb*I8UWt-W85!C5y);7>kDTN$}+(`J$O063J_d^PVj2%S4D8?0M9@lU~w1
z{}jH5N>9Ie+SomDYW^OJf_ORVfSC;l!ksOQl)xux=bb*_oH8x=ZqDlz5S6CKrQC
zHE^R-#lRm==uYPm9{Q4=N@n^J|I1zYBL#oFk#XPr5an0=m$8(hnEma&Ra^*o`4?#u
z=FabEBS4+Fv1RLRP@I%3-8
z#&n9|l{U473Ml);rD5G@YGAmN*xV*&<8jPiTz9ix3Is1S2T-dD81y)~{G*qzxypNl
zZy)cpRzAB(|Eq{_GOkSy@~O0*ejyl>5Q>_i*KBNlZzHNbjyLCWdVB-oECOHC{VXbK
z{tsK{5M5c=rQz6i#kOtRsfumezOilFPQ|uutKw8_-gI>j`tLzc&$rItjL!Pr{p{yO
z!w8q$g#(FvyjiPI$#DJ1-N7vyHdjc^`jd->DB!BUCI?nL<=Df-jqENdx1@8b(8X>s
z&s)lB&;fhZJ`3Z^S~p6b2jdwCYV&N$mv#;`J7`w^6N&7;sg_2`JwdWqe()}1$`yWW
z()f^6Gfiol7)`N)@&Wie-#g<(e72P!WrN!IQm|baH(Ba>*dIk_udQ*HHy
zZ_W#;MtDk!XbozyG#Zw_@Pd8r0Vj?HzY=XK3~0Q@p%J8HXaLmxIcC?2Z1?jLP9$1h
zYigyGSJZ(YyV;&NSMkO+YMPYR3ss^{7WK-S_gda}HiZZ0{H+iO*BI^I>!d38UV~^l
zZvg)jhjwk{FWBb&l@?C**s!@&Igw^t%gfqD-_(zJ^QjB>j!RYLii?aR5MY7K&w(!*
zu?arbwP#!#l4XFR`V|lMt*lr7+YdJ0yRA94;HDGl!`~uJF)4vuMQE;*+^cFr|Gdw%
zFZkKf3>%nN!S53*&`&*mgzx2Oq`S-B8=wL=?}yuZ!z-!D^k<5^o#IT+kJFq!)I`-<
zsr-vy>@((lbD8Vs%8x-Zj
z=nC+dR0RmLA`dMtI_1f>zZV%1s(Qk{tQM~c`Hdx{Zh5mR^`;-ip)u+y#ShaB&Hf)D
z`Hie$ZviI_)G~b=X1Sm#v!@&Tj1g1=8;Yzx5Qpr?uVLmCxD+mHt?63vMwvY(mX^07
z#bpL>w5@=suk87V0oY?LVrD90)E_qNIGwoXdaWijxf^npe{2X2Pc&C?4hfoWO2u3X
z%joBdNwi$7GvD*T9yb+tQzZd0IK}rCmR*OygGRI~Ecl}3yyo;6
z%$k`h!h>^kV92eQRc!m`1cyyO+hqMs=^~>Hfm7{BsUJX`sq!Uj?D^tmP0&-R2GwDH$n9_&L*uP
zQ=$2uU^CFI(H;SuZVtJuKF
zxo^Uz%$JSnU7_JLSI+kg5u=ekl{(JQ;3j~q7{-xgA`{B-X$r%c)j~sm9m44NTu^cD
z!P9SRqNKNVN;izoJc=E6Do9L=r5T!FDA10kwDxMWJ6X32$z>v0*K%JO!Yzo3OGk5&
z4^ILFKD+frXGd;U0-ny@ctu~IkJUm;2oIUu4Z00eL2WzBMu>Q2WQ?LBhD=C1S`2_h
zIeig_ciFrMZ&TkPosyX^*+hyO!ka#Qc$02m3CraWpPFcz78*;J#OUG_X5J1)@y|4Q
zg3*K8J>Tbj_Z
zHKU;6fiEUGU3PU~-8q^86&D6PDht5gB?TAhsAd%Z><1@+vC1K-XY7Ito(TZzS$NyKB!Z!Zvxkfm3Kd6;DsY!
zC5mB&Z2&UN4_iq%PmbsEG*hBKSbZ?U94qne{-gEgoL$V;QTA;yL%7E6G!alRQXYua
z5~{SxV%~{+A&1V?wVv@<+xDK|LniiIo0`N97h+em|JXqYZ7wRzR)hWiH&OtaMA(ds
zTfTbZMjPqgX<#;W)KlLy^=relK?FB7#PreTp>ej&p4R33^h_U|$PpdUKy5x|4oZlX8
zM^Q{A4B6*T-Zlb5^8EK>tf5ev8?lLNbHwmqBp16h9={mT)m1HY90&mi=vQr7=stF1
z=%#Z~L+|c5(5s%hWxqF}8~2pUk8=AHcn&BT{bN
zJF=TW8T@3PnxRz{(4(vGaJxQ@+bD3ht2hnk%NRM#73a1yb??AVgolFI!>F+H-HH@y
z#)~$@CQ+3niA(V1d!PZ}JYXM)tU8AO^8l)Zk`ksRiAqeS5gwCu2Zkw2t*znVn4izk
zRv{`*8ZWTuQ_{nIE;XA(e)x})4E^0bil&anx3&AJrN_(ndp#=h5lpWhF{E*~Sq<0z
zaK{&Dxo&NZY*89}%l#%I#Isn*X|&vVTlvaxiArbgh!UsE%
z9Zb%*1y*s>JJ_kl@8c{Vi2pIv&rVi{u*G8WwcXR
zO;YXFICHj$|J4WRi;^uom6QCIc1f28&*uI)yGA8GGQ=x%1d}&>Co}WzM}hNp?AW$b
zzLR~LH%pgWn{Y+SXnuhI>2%0=R$)QQdiT4-dujcR6`@|&KYuIpGaOe$HK3c`dMk*8
zTM0oKE~kZZV$hhpY7txVC;yEn42
z(4H6YamWGymLdfN-6<~=cQX;)6y-SG{EUEIJ<%TQ=`O2~nv|7QJ<)7(6-TdNj)DGq
z-70J%`GebAj=~pRd6l6w6MD|cd)Z{n#+7e@)Cf;8>)4tN>Rtx?AOI^zbOEzO=lA^P
zdyzHP-hMEc$lHJ%iSZ?@4LO6K28u-3&#Q6jG0TM1E1@gF(L9Bjyz#
zirBY|kV*C(58?`)>!dVyVhn`Rgc7g8#G7+bAY!&;`GzHuTJqDzPuJgMMXywXb=JQ#
z1W`T#Jq;Kov;Sg^)|&O~9oPAwChRXPFSxDVkjR)7-UH1jxu49eCTt>j^bVHi91le@
z3RJ7p+LowFjmfyESbH8LhBD^DVqL;My()i)u(RC=x3YG5oRO5Mh-&BF;V(fLD5BO!
zhp!dG@1{Q}<}*8YqV~M9()cK>PWp`Q5?0~>jcLUECq_<@Q6v_9jb42cjOj|xrDGw~
zRlzJ*J+g-Zr)=n$@>Wq;r&iWUa3@Rz0}A%Ba~WL`WD#TToDuC1d~|v+R_sJMd02>2
zkpqgh!qrnXWG<9x8ISh6wI-Mc{uC79Kr)NF77FX1YxKtX0@)|NI4Og1d?3e(Zl9I_
zQ^pFc+RIbTugK;Q%@*^(cd?R{0}(-}#ew)8&7txrf2{9)Nk+d)SO1No?COf}Soa3t
zD)Z>Y-LA<@V%&K{%}xw=-t1sQg7_a$uzB8=!txzy>`2NX
zzTHbf-Q+$bV$LN>eu-*rXG`%;#N#&J0xv1goo-zF!j$)b!TQPv3WEmZ1Ra%v;R5fV
z+4r+=P-?H@wEu%x{x4z4_TPjh1}Ya9^M3$KRvtFC|5?caXJ=+lPZWfr1*m!Pt6ps>
z#k)z9cDa)4WQ_EZv#d@q8=+{8P8)?eA+eblkS3{b$jNR@E>f1H?!S=
zDR7CED+V0N`7%J2iE)L6(H8>9@FS^z1n?q&0*P}65kU4q`Y|6t0OO3Svy1_?P3W0N
zMS+rNT_NFNgEfGLeLdZYR_%b)(Bf(ikWMSmR#R+^>v}pEl2`B+5Gjq}+1g-k=b^~e
zI;=WA=~Zro7X%Qb2oNn>sJ2U@byZZ;ekDt3OAvS@En1L@C7v8*bOcIPRx|>f*^>ez
zbzj*z+3Swrwd05ctqa2|aEEF%Xcei6yl(fq(M#P$IQ5&|tuqlkS1Vz(`Vh;}3d
ziT5lZfFyR`(SAs2-+h#5BU2AdMsp6AJ1i908_+Duon^J`44k<~ojDQpo@g
z0qV;gJyI+3xiGyziqdPT5Jb1<)vg{Du4zb@8xV1Ap38@Ez+y3}ownjVV+H&o)Ru(%$1_z%kL
zQhx=20tkY`4&jo3Ld>3njttRHkKr-Ot4RhTW)!waqx+R4tAyOIjN19Vyx+hL0uCg-
z0LK|+Dgq6FeV1T|VkQxiz{lzrmiu!PPVV#qh@4-~F9)6)>n7=O({tmvTFfV`4KguG
zQ)$LfGtm!9;ulH%YnOTe)&@+@NxV~RE}{YAqE<#hH+kQsvmL{@`<03CzIBM*-h#BX
z!A&{8C%3miK?bN3qHr(6E6Zp^yxb~}VOK|XFnwu9&njsv83;oVb_YJM|K78zm*?4m
z%*Z8OpvPCI^&`1c?!xlOC})PtMv*^!jIUg}=&?0l>E!=Hi-5f^TE%*S-h{+UG&#U*e{C!5--w2$25_RILsQA@f`-T-WFiAQ9;6y_Hx
z_x@De7X=y?>zV{}Et5uR754opr$7>b#MGFubWhehnMnVb#n-Qpw!SlYk>3kSg3Nzc
zp%l+vb_5Q816pQEir=S-eF4+hgkVpt<>xlT&dIYy8rlbOJnPpYl|BuU5b-Rx#5s?;
z6Q>sOhvlS!iof<(nHUo5kiUS+-#MXt?&LRy$W~uz!+;fdGc64DI7tE}iA@UER+qcEFqo
z&DjS~Ni~
zS3fjW_I&p&W9os(C4B-wE%YL4=>UNlKGC=S$vy7)y4D<3j6+DM6EZ&@WH-b&~ozSUHV2+^{TI9>-r;ECRdRJ
zwb_6=mz1*vb_*p)lhfXiljN?ZjnE3qj)QbY~m)MpDUXwq*K$5kp70H
z+#E0oucXwo!kyyx4xh#Gn-0}Pi!bWWSeq1YPiJFG#ik4R`xCE`gzoa@4*Kj}kxE5-
z5acmPR!wP?k%6;e(0r$&Md)hQ{Y2Qfp;#lgb9ueOxygy?gKJlFY8=OwlK!$V=QF&Z
z#Snx`UYhewu!b4nIw%4EjjUcL%?qzqFV8?#&Cufz-^OpYci)xc#`gH&)Yg9oN^t3)
za|fLe`FO#+P)DmvXchX&!C~p>$F2vxkKYK}1=0t<%1dW6x&W9<7c^1VTNY&sVcC28
z66n8Nu`70CY<7jak6^5amxK1b7MHGxYRJ#D7A~rx%~1njutQ=Md?@(jL-D19+;VT`
zp{R%&3$5`gudSuf{BYTkkK6_Cra)C%`$_kAYe6SB?OtDYHHrn?K4Qtv+yt3)6n_}
z-Iw_OLxQO1eW=a?ayQeg3VrL(f%)bT|C}|`RmMD|Y?mViX9Qq1sko)wp>{LHdK|aM
zPa!uaWEjN3P-FuVIU>i>o6?W;-Iq(&r?Fc*_GtRD15vH2-QS5C~Z}1QB)I2
zC6gM;k@o4?P?PaMGFanxy~p{Mzk02kST&iAkMMd}YqqC^lk(MT(Jsa|&_1kdbo-|{
zo6zSn8#^7vp4%heB$V-N-7_)_Az%B$p)!6t!j@Gd=Ce#^St4W4v&N6^=Qt&ACG85i
zRNe>}-7?KWkzvTw<=ay&(oK_xYjPUw1Sl-y#La(oE$LL9O&r)NdYOyPO38{ic$y5W`Mcj3!**1q6Rw!9O92(t&uM
zb`m9@hjM7#znxbnU30h3nCtIWDwF0g_;~{~f%*urgN
zw|N3~J8>rRJ;*EX;JSXkPu_a>OnnO8vAN$I{?v{~4wp^jN1g$oW$-^Z56+5C9^Gfe
zzqo7wThRPE{}49z2sBjPkn!BNvCTF&qhw2LS85)vWK)Ura*FPl*nL%->i16V)9V7Z
zX*%-C|L6s-2)OFvrEC`cDKmRWX(+2o@wBSiag-ixYMW*tm0U_-`+D5zIrCA#C|aZw
z!VX<{6ohQEoWi?VvuB%eW>J21U8pU;8UM;8F0?tsPI)4ng`TdeT0I{}?{hAh*&rKC
zweLbkk6s(jn9ib*s=g;Oz%>5XU+~Wm?RF9^Ajegy=;O8)sOtg=-WC)NhE{a$Tno9H
zRaN0d5!m1rzMC}>cLJG&)2Qn2>Z_5xwmObWhP&y^^0ru+Ng~8$p6Zxj+ao^8XEWML
z7dHo)vv)K{@WYGE>A+gF>Z5$+!B>UTMc*olq_Ssed2!V|8nehW`6DcTY7ZBnXOLl7
zH*?;Zg>3g#$G9GJ)4WYny#EuE=->)z$EvU4Ph46NtkE4Dje&cs;G9d>o=7OG*Z%O@
zMD$FJ;2-Lpa4ySBYoj!?Kk&K*H#F~%M
znWw{YcVU?MSw`8HHAf6LEd>hd>6EF3=-UjfgnK_XnDSk&-0VGgzCKpm(*=wN9!Jwc
zuN2N#k|lY}_i`R{DR#u6VS1{?=0@orZwX2~5Pfe`^XF$>nKX`U3(TPa@RxG-o<+IZ
z!CIR2!J)i@oV%JD&tzvl)E^{{wm0ynkq(~+FZE4PQZFxJZn(fUfu;4|MXnRCa!T(n
ztAs76RqB4<1zlp@wBKe{S{`C4wKnV{54{m4UMgdn)wK#KW4&LNGKXJNgw`}$U8T2+zI?7Ey3xId
zyavfPnMhhp>lLKRclFECqf>6X`VxcNf6^FF(La@oW-fiBB$I91RFK#NA8c$<5`s5e
zR@m%VyJe-=xsKda+$)dml>x*Cmsi^(e5{m_>txh^jo4j(%TMtD&G%)>1K=;JT3X;z
zTF_N4xl~`<#Ynn~y$qVRg$*hUz_EX$K;ztYJfTj+NNdc~5kc3Tyg3XjJ`9THci-Ex
zLncYtmjkY`aRI`@oO(DbYTtJi^4MO@&M>LJo8{O>D|ojpjo{@;s!V(ocy{3utzUN6
zb%J};jW({oo+2**PmyBTblqaFE$*wOrL{R4q%bw~9Ztu7nSS2lQCfx%Mz!2&CEP20
zaqN2Ywq|Yn+zhzgX<$5kemJgAl;Uz*N_q5c^J%_+t1myr7z)$MUH4GKc1Q$2$_2AV
zeI2-rwi$iLEmFgG)JSuOs-$SN?pFP>NCtZ+XB3okUJK`58Lh`hb}9
zCio!EjbuH&xCVgb{0As(t#@gQ5v1)em@PNIxC*cM~+a_t(N!TCQ`Eq;ieKEax*y(d%F32=%
zs#0h80ZuKQqWU9b=29U5?b)3Xfj(xs8-<(?`dI56c`1vmlgid>yuba(sF^PF#fK
z0)pMW@pl_Kn>*+|Si3GgXjB;^z|f!pl8p@+4N@4*6517vZU%%h55%h=x-qL9JONVJ
z?h{-2w}-nCGmAS^b)Y+zy1OuLLv%$?aCvfSXzA1qjx7R0AKBM4mJ!4_PWI+{7vDdW
zIp%5_gf-?q1_aW_qhTIS*vVpAUPGe+5>Xw(kzP^sZs+(nu
zc~-=VK`Q9G|JBs&4As5`;M)M{&sP*ixvd{<-^>i$IyZ*>Jq>b$mNFW<7nnh>|g*edHL37G2zp&s0Tcr
z?3rb6Zt>{wba8(Zpu#Y>fr0*8jsEnRsksCuO~BJh%SBXIv5W^o&&7%C5^fXlNn#_X
z_tgN#(6N&L?X|JFfN6TJgHG?0V79TLx3slDT3p3EcKI2r?+bRU^Lep;^#$9RSlrm3
zegUYbYcHp#0WW~v>6Jj4o!Q9+OlpQ_p%X*YMWRf|dB}bQ5MHnV!m>2^KG^T9^gZLW
zJ!3~Q-yVWX0+5qYVvBcZdHZVPs^46nu8SYuJb+q7nQHw`T(+gxKrAAkt;oJUd&)Evw
z;fr9z{C4L@@l`LWHp&2ACmS
z18;mm0rJjKi`G@vuHQH7pJr6{7S;~tW)`4^|70IHd^|sVNLhj{0#k#*_h-NeZP^OO
z42HlAjhmZ1K)XQjb@hBfPo5Q}`><1z!TZTSbso9?Krlysis*t+E(k^7^^-qC-+^Qd
ze+Rc`fXbr*P~Zb(?vaflnPt92Y(Xgh*BO(KXoy)R_2CYw(YL-(0~XLV2iUTpae_Uum^1)!`i9wCEy
z3=YHcZwOhmz2EB~%oUZ3%?_9kj&S_Ku!<#pU`THZ%yO
zIQXd~9k+M=T&MUA1LNNK3JtMl^j1!LE2`AuuYA+~+2`+{!O>2y#Zb^L%uoiuzO|Lz)4mgd
zK0E_>`4gaLrvKQs!1aMxN030KexbVov@xN-4|pA5x!=6JK$D@_Cmn0#-fDT_Z&Guk
zQ(*BM*V~QBsly6jLaTq~du*{GfbWZn0Lk%fR#>MRFh?-vfLE|3gcSjf4;bEcpENoL^#^zOwPKCkcm~=)x#h(b{VpH9
z(}mAizTCyfZD%52_GBmtMR>7sN<5p%TM
zDR1}0a{_4+fGh7d;HR0=n7R+dxf%LF-Ca1Us48os-l`FRifmb0W&Hs*86k<=f!4Yo
zr9`^t5AOg&GGrL&Bc2z(s7;=yCBf{&H)DE-@$OUFM|CFaHqymgehW5>T{fI>zlHwDInGPpq6?bJ_On
z{4m>j9icZTzJL(W$k5BFd=xWsh>L4^=J}UW_gArLW@vFug(*
z`&vfrL#gE@`9CX3k8=9YzgW|d$HXd{qA!@Zdajs
zbAxXX(REB}%SNIT!404@0sftsq3P3!Q9Ks=eakzR_*ATLGT#p
zn6TymP={$qsPTHrTs2>JIx^8Mx(*#~&$biO=t2i0ZqD;T~RQ%z3F)^MlSZb#Hc
z`34x={PB`Wuu1PGiwyCF1^?QXUPnEIrxP~x#M9tJ@ZzzT!YHS)Y*<4dmL>jDAU+jy
zH(&|@7V4@PDYJ_`YL-5#Jq44I*MJ(*_ka=u7~{on$fkx?u{0z7pYxQxHcfkFEDARU
zFwLBSqOf)yK}8yWb~htBL59)_AH`sO=TuuNQL5Q{d>XZc4js2ZOq;#
zgDqv)(zjbo@HDzyVAuM_Vw%j$clFE)@CNz9!?ihj)_0#Jy-Jb9NDR*O%o6T6yYmo&
zy|P}V*pcPxP^ZPX-;Bv4hGA;=%mi5qttUz(v*!_Hrf^i)-M#s!$96iqQvc4R98Y-pw>
z!oVHWG<^h4RewRqWI#rZ;tSF7`1+&P;^wuc1-pnp3d2Pzdr$ufhWi3G$m<=w62?W%
zJvd3S^ZrW-R`Q(ZAlYxC?O4=BxP|cPmgne}p9&-!!*Hdlz;$-w*WP9)&;6?~pc~l>
zId;7qkk9;5P-npVcowOCqC!gogoGq^NX*-6<9h6IY>!&Jjm52WB&;jkNPe)UZ;LQS
zmp3{cINja*$;bNBu_q|M?4Pv?=1GU*Dptd9)1#u5%HchH0!g(V+w|^D?S02l=w{1RwAxXz@dOw-h-pxPllY_k!=JDs1x)$-82n_fF;b%K7OA%Q>NR12oU}%Uo0_M2r>o%pI2T7q49eY3K3mRkP0CZU|YCv(Y4^Qr`$f2BzNb4bW?1s+f*mc1UD+(
zbDpNPw+p?Pp0Yw>c@u}W<_CG}%GzLqLEo44wDzmys*G{i$4Oi_CgUxqoP@vcXKYKn
z&C6#abL^!TaB%B!z6Me>2!_b901-#>;LD7{>B}+V++y17YSm53(=iwKa7rOW=70$O
zP{CSdJC8&-zy4~#jbbXvoF1pDUXD6qnc-wh
zkS~UTY9rl`Z>^=d#FjsCjl-Mm15G9YWou*>J`#iuQ@H9bqVkOsZAXjCfb^$Askw&;
zvBFrglEnHyk+z8Sy7q<7W~uI<@k>JXXb0q_>3LF?yN5GcMznFbi58I8jNqi#rD3zS
z%-jHt+G$OKEv)Q+UFrH3ap0472%{z#YkwtNWM2e2^1ng$)@|Bjs*C?1ygd|kGz@NU
z(0)~i*m;RMBDro4N{6fctd4$%NzZ|JC}v*EDZ~>640nNS))nWY-4)vPluq1d_SvdK
zuPY|<212m$M$B>!4ZW%UwP62(*NwqmL{`Sy5!LLdkqDN56@kzusKP2_#K6Be2H%JZ
zlOz9?%@Fk2yQuj2BcBX0=NBo-Clq7h#OH%?o%s^sSWsFe=G5u
zTMhOZ?sHs(VXk0Cm8u-G0CSpK!|||>hZ#DL05%_=DXg2tGc<2qBw)X@UroOz4~COH
z55sqV$5MJEtjIz2*KcH@if6gwlr<*fHLo|n7JJlN&>mruPBo!asX;?+
z(YOV^0Yf91U{7V%^}a7i|L_I=e$hm6v#^h)1ZzCG{|X91W~=*=XGIH82^Cz0p@#@l
z09a2iG!?1MQJ-V%0zPa3s%py6Lg+lIOH>y9OgQM6}(H5R8_kJ2&DA?WOzM!zKI
zbX*JTXgrN&e6UtiI5W}==BN~*P|3or955z`u-Rk?m?@LL70mZ;m+8)@j-w$Nwd4L1
z;!HuU6T3|w<1@PQLo+HEt6Nvh;7+5NnOpMX9uda}UGQ<@X$-V4IW%UJFVCWkW{Ju`
zT|A*B9}KMUt}1;WnQ`Es<)XBlXXiPNjMeSmxFTIZ`H^aL6xoq?
z9_1BB@r}LhPL;avBn{LZw%z3^UM*B=mVNTdK`LTk>8vD4kT40fY9$v<1xF5&gwhFS
z#mVoZiN@I5P`Pro7Tuj=v_B->0SJ2Bll@_qq(O?2ai>f5JG~h1zZR!H@a+w>AvJED
z?fc5$!=Ojc&K)t!F3ocLY~RUrIwu>_&ngxP-Q^vH`d)2BqvP}67Q)Kol5}1@gybq3
zy*dQK*Ta9r5xb=~5Z$|$yXPkx!vFFRnXy>qr6IUe_vWi34KKhv}QaY9Q(sqSY+I&hT)MiS5zat=0
zF#fWk%k3IevRk-|$)l%9DU#@+jqqnei={j7HlO;U4>Idwj!ti&8PI*TlbK+ZRswb-?wf_HjZOjhg4x7pJCD?(t}m%ZVJhqjIP)!F*W-+syak6gd4%t&BH
zY<0%Bh%}M}r03Gg2LPo;dNeB&cg(J{dsZO)fm!4f29GW6gtMJ$qKC|I>6wQzR;=Z6
zW}g;J9E+p%k-v}B_z%Pe;x+w+sSY3%Tfq8n?%sGChXhT%n--YY@^eVdpXZVsm8Wh)@~n4Z+;ky~s~9yFlhJz}
ziCjQuUCWkkVs8_;FVs{8X;G!lo*=uX*bj34&&Q7;y*rG5|8y7KkIV*1{EL*J=uy${
z4@yLP(9b9~VL%#q*k{gz-@1fV0M?pra(ccYT5jySw-@wK`+;M56)VZH+tb8LOUAJag}x?D1Cl8Z6<^Kgi+>qugyli_?Z*Sb%qN{LN#;iUo&x@S>=Dcy@J`
zAt!FY^6;3JOB93op_}dI+*!e2-YO1Ltv^Hg79C=>5aog5uiEmj)sjbG^G8Zq
z{dBZiHKJu-iaClsD(M2YtBruSxl0A8jdZ-Wl0oWN%OmGqXE
zMLMsL2!O~DXQ86tt1pAZ(zawTIGW(-uLxW9J`vhoPcC8jwXQpvjqvl#@ry6kQ-y$_
zvBiO<5r#7xT9?hLj(NxP(!-XNsU#pK2ard$_I-=81^;e4EtxRZMfHI*-i%>71by~K
z^qjmH+jfLE=B-J#z1Hk&KZs4(rpKSGZZ%u+`nLJ*+LFy>2
zcAtEB*$PqoeGfo@e0Ql^J2h6`>P(MQYp8NK>lyd|T#`6`f+g@9Dw
zQ={`mx&gq^tSk?E(YNI1D)XyMcFgJ_Ur^DO0aE*#6JffBs)H$pM@yTz8U9m#IFa+V
z93Vf}X@odbp&ucEORv^Qha#)K4w4oi7}YD`X!{Emy_c730PgrL=TUU8l)>Xk~+
zRp~-q`H_qU(Dd~UaaKpxfr&*9^p)Y;0B{f4Oml{Bo$s<<_d&OtWwX|s)p9{)#)jPC
z(@}0I=(Krt}-u+FI@Q{uhhOaSobCxmvZe^Z&k4+c!gW#*Z^1
zZwU3TbaK4hjk=IS&>F7>dp;!YDd(Aj8V~RkqXXw4NYwb$4z+uc{{T%4Sg=ep3aF9@
zKyQ4^QD^0f2{PKqDv(g=>LFNKB^RRn#hq;)FTkheXS{i^QqH>{+gU}oeJx<
z%k>$f-)Mok0bxAvJDa0qR586uFzg9!p0Mn2^oy^8>w3%xan|9aUExb90F+gz?(>xL
z!98V8;-_oz)9#Ne{$8)VQ8i-4W`K4y9L;*IR4ZGsRGtfLB&^+sdq+Y0?oXAF;om|t
zFQyV49MYo0QwWXG0L~0^Yl(;-hTB)leeE&vcIjgFVEvylcyY`7wO3VK#6Qs+c}-TJ
zQuK35q`xw!kxxs1H_{MDm6lu`jXp(E)jQ{MO5G~hyVqSz(1;?+ke}9QegG^5f;kC>
zL6LX_zJyOBtQ!%g@+3NUV!!E@t2qfH<()$uUIf^+X^(sdL_5BWI-`fiBN1oEuk`h7
zij$pivS*&OrKpaYpmB|vp%%F=p?Uh$$1G7%3KDs(lXjL^iIGA}$(*h-S$vY-&kL-5)6XW+9#qY|;_klYPTJU%!)C3{{x%>e)*CU!H6{VmLU
zSNS6EN01GC1Z5_egyjLryiw4SvFan4U}>CZE^x5g`EGDBy-{$zgM4rev#3|BMPi96
zmSMT|Fv%IMaZ=;79%ZoD-Cr2YIbCiabrA%_>uOH24jv%^eJIA~#W-rvhU1kFmpyZpvF?Tq`8%
zvA1pRXOrKs6?=htFOX-MHCGa?L07$#ST7}=(9TFxyBynB@ErOFgtNy(*ZnCQSR!Gz
zL33W{dtFlpNVuOtmzmG)i>A?2%M_9*6wTKWhAycHEqklor7$XA^hg#bvh&WVahX33
z(z00{SG}k!wgAU#8B1+T2qu(CICJkYaLxR1=m4&fouOw|zK0)i$mVLd#|xH2&m}AUOICGkx+(^mKa5=QtcW7@w$7XjW0
z_>qMxR)D|c<_91FKiFPvhnmhSXEbd}SA+?sqc0EFs9i~6Q9
z?s=UVeak{+N~0+o>z?_x=R)?N^Vm{f0HHmxBmi8-j+^>73SBh%?2yJa8I1eN%07i~
zUSj!DfeUK-Y8r+aiNn3^a~JeBbk&cLDxB+Vd54&Otc~f&T}MS$p~XUhZ)&F?>S8Az
z?rczzt+rhRBY&k^r|tNedtsY%q>qVLprWklT>#3tRIi=g*LF*=0rXUK-y4=QCgis(
z8X#6t9kjY%>BvkTVvEN5uoSTxxM&6r+u9AfdnA5Es9_{fU>yq4k+Qe~d_hH<6SSrd
zPC-^_6Wbysk~Jc#o*R$j^LDMgCRxA+81Z8O$4cmbLjANqX*i22;75^t#wc
zy8eyMNqI81hops@Fs;ODD>-FFgf$`;C!Pxp@ru`Z`+{9Q@R1RR-Bxe@07-W;0pv{n
z`c+|9@`hm%9TF(>^3)P=vHDS3NtvZO%xJX18(9YK*7O|WW^gX@lO?Vz9yV7KNpTS)
z96m`am{59Yp_0zQQ|D;7@+JLZYv=kQ4rgXo7ZiO&^REY#_;WF=`>%sl@WxBrvSE|j
zS|oZGY+HZKC?s;v!yfGhNlo*bcz`+2jEw_Cv;0`7gcUnT%iyHcAo>SB7`qi~@`Nar
z%&oPTQ%$*)!Um^*i6(gMyF^HQ!@;Wgw!0BOm{xJFZc~zG`VL#DSdwdu1d$VD2B|0*
z6&|x-GbPrJ$RF?TX>E?(nDv`11Bs`cQTjfd3#B-$M^=y)Kc@i!(yrGOO~Aak
zDHMl1pkhkxZ)@)RtVP5H=}yFzwD0ad<`e_OZwQ1uC7F+XnzCUH);!-OpjL?gp;ol*
zc~l)_WlLVP4ETsqSbSj@1K_o^hzT{AgE1|9c@B#D>LnX&I6YwK!Sx(0tFuU6hu{~}
zMbMlnma!c-Cu~CJ4r4?h(uG2}?ky>eHnqJ0XaxUDv+0e^8~bESh8avOb1
z;6Uh^DkkWZ^2BP8Z2`9&L+?RGH5&>VNSrH|v+{}dj1nDnc6nYY1C>l4@5O$x@vTxs7Q`r)K(
zF1{rON2{S4q9F`oxoVK>9O*H4S{fCyv`QDTgVH=d=QyO!^#Oc&n4|F;BMn%SYFwpE
zDmCS*Rj|E4RKyHVh{)4lSe#KO8taib`LE7PC6cRZw0>Znf+{c=ehAW6iEWKH6MEf0
zB*m+?B(r8|#O_5PVDT^(y$W#Uw^JMs>^xjcsSPO%dA-ORs{2rN?``_7p%+BqSySY_We|bzV%rG4csVBPIifTDdDac7Vak09oisLFQxU?n_KE}4!hN!Cp-Z0
z>T?tMewZp=vkeG}blxf-Z)*2_(|zQ5jpOo~t5*?Z7{KyOv@Gx9*_uO}w4=@?YfC;#
z9@2;33pZ^ucvP8vCvWOvX_%Pz1)?O2KiXvf-FFZHX^TIXRZP~dvHiB18w(@B)g!^(
zfx!0FA+kt!p%i3N%wIfz@%yfGzew5pE6+6Ks=ETT(y4?OH
zxGWyvI=10WmM8rz^D9P4zmTX7|0y8yjQ%D!-jfbc!eUz4rE0i+t5^LV&R#>-CY8Tt
zvVlV;V6qJSGnyj#`w8Yz6ajr)aV)ZoXj`C|`3oS%Fs=1tvtZcTua9GV_yGd87Ss;$
z58U8+XYI8W*+F+x*tIwWzfUloH3?h7{*5(io_^g^#xu!{GEsZ=of}Gg$m8!teA+sJ
zd^F}FF~OOvOpRjs;hS97W#Hlnf?TT}Po{3oWIPQ~iWLGD$)Z@oc%;JLNI*8~{Nidg
zJ_f)vxiKj;xMykaJTs9SeeEDBA$x}T)v5^o*7w2I#??*lp5nIRD41J=&qVF_f$+$1
zcq0k|XRg7}aTKGahKAOkUj;ls5aGaCq`n~d&`_1U(RN2FwEAUCS~15zFkWouJ%h5N
zN^UvhXVQ3T*vW9FRSGHiz%HwsezdRJ=wd)xG>$J`RivJK6l(^Q=3jzh7c&Nna0jRC
z{{lupxxYIh?eog{TKCCghojlT4A$2T
zkjqHje+70mDT4&Z!#9$--I`Y3UARXr8*N(RdZ^$z?4i6vFtWLG17S1OL`FfV?ZI-B
z(z#aFKL{aSSAFk!RfaCDEq89-adO{b5X~+Vb;F8p#-e)cR&w9)`KFdRgKdd&i3w4U
zzag>1$)-{`TT`Pi6zQ(iA>+Z4&LQ95ZC&!Fe^KL?(YO8-oAT=dV$M?tHK?1V&lSHO
zrqN;y7YQsA@twATk)sF);U^PooK5d)o#DQ(B4TlFp%i!NVzUlRX^@xkH`(_swG<;B
z=1Zk^dY6ik*L-Baiw7MgJj#q
ze~@*y)((%-vi3GBoppVQpQ4BubOU_(T_EJb{%GV#74!HcQgaI?YHXvFXTAhiMB|0=
z(F6=;@*9bBe~4vgYSlsnWb0N7tKzDug}J(MT2dwTc`q%b
z?br){30Mh0fG+necZ&xw5(O!A$o8zm5kS%tg&`HNq(aL5g>sHu!Obw*TIMnK{!4hD
z7%;Fx*E7HS@6<3wDJH=ia%>T!8=2=h-=kbqztgv_HbFR#TpU3=lS?5=ag!d!KU@p9B
zc0ftS_i_&kOyLGAZ|-*saTWl|dQ3*H%z!|Ntsjv9E?s(4Cc8zMwg^$@q9q(H(P5aR|)vuOV^_Xw_#suLvc*2$~1%vm*pt40XIk$mxd-(=yp;KN-(3_rb
zZ&3QKvl;Hu(@g&J)KD<;q*$qeKr{kJs&_31ET&rbPe*#2o_oSvC%M)+f1aStm!Ju!
zLjMA&i)*s?b;Eti8;?5|1scXl`{Jq2+ANNwYOgmviFqS#g&YnBT@f(Z#02QHEx1RQ
zO%djf8+8cypG_n-Li^l5&m8UqH}YNXihm!n&vL-vBFdV|bR<`N^z~4`^-p(tJ;jeU
z>6@e=CV6GyH?`ZKad%j>e;n1ebM{hU-zWOqPaAfbu-W}FvJFP)I!UH=wwk>%)J1e(
zxjU|TienyUl}V-bx%#gU-n8!t|F)-*m;iX(i`?~r<7A@>l&d4vMf71Adz4Kf7YFnH`JpI<51?L
z689m8XQdbxvPDn>zf3-bKrV=LNQdXdyA5SV4Ea}a*g=gdy?znYmsd}-`xZaYbTDhb
zyAdmkPfIZ&OJx;ow~{=8*P5T8GY49srfYE{-P~620dl!O4#fpZ`FPPs(kC1HLj-FxP7BlnbT6zQg{8&NNC
zg-B(P`cnb6mh?V#<3~TaL~DYp;PAM=ay|-D$ED9v$dr$OGUkGHl7n`QTjXDk8m&E?{mGyk2y2B
zpsnWr<_p6_e=R?ls$VEEML_djFf)ZikcCVY4p*_sY8XQVJw1UcU-l>D0wF(ZSVHQj
zZXbqx(}6=Po`!mmIU7!@pOX2*@y;17(i~hOERXKM{)(i3mgg%I>WhgjZOP`aCg2~Dm#%{(>HQ0z7(q{BJx7MHH8|RNgbNSH0elL@NQG_E+U~532~FSBOls9!ZsC)@pAFUZmNqP6y6
ze-UI4O;8pOtg~<0+O(&&?>;(|HL~LhCVTKaa;mxd$v=-1cS8wRi}hlU>A0Al2jvlm
z1huo;GhEey2F@RwDF6y&`Tig@tngK}>%n&T_X_FpxF-bnXZrU#^i1O8P7Jy4V`Tw!
ziIifNJGku{4HC?CrJcpx&!60BVU&m0e{vKIYp^Pv^f$z{0`yZ}5x-BM_(*xDNPkLw
z?D(3zht=yU)Lb}LX|m#+w+G>dBkwJwwyJnc6kCt&sl;_ISy@P`+d%(g4UT4#$8lIY
zz0WIXD>c3`$ALX*yF@I!Oon(p$ksoD)mj)^@5v!wQ8<>USiA&?t6drXFtVMEe;q~e
znQhX198Ss2j^SX3T!dd_K9o)x<
z#zA&30ze{Y$UQ|_0NRF$(wA3O^WeUEoC|%meN$joJ4P~KATIR4tTPOGqF+(Nx6U;O
zFM2nuqtp+(S$aULY2mmt7QqmKe<`F$$u!+sq)~$3l%USMjPZp+jPr&&-I2kxZ5<;B
zeU>gTHW~*mD7mAeWMuY|u{4OW^;^e*!w3|0dF!Sc4WjsWv1!PzZqME2#eDO-AGYpU
zKw%{qLar2;1xM^yLe{Wij_1L)c%k5{PW14u!qxh1p)aOC3ahZ(&faCdf1}fdSo(H*
zlG8YMuP85pIhv9;bqwzwyLm&S^(vIZ#YS7)5;3+yf{_kkugaK6)g_}A6|x7U-Q|tS
z1h7$0N}$})P|<#UWX>m%#YDZvE?0JT|yh%ehh5JH^
zRfN#h-=^V)wpWNt`k6t!(nY$5hy=8y)Fu})1e48Df@S43V9e&l0hx6;}na$i9>v7DUS+Y{pQedmtYFI97&5(1nTa|%jMJ)Wj6I4)^TE`(KJ
z!yMRv+kubon!>$Xe{bG;r^~+MW3MA3wr5~1bimHWiY7;#`IMV%rx5@{W^!qg-G~3`
z4go
zu?{!+L1Ihupwn#@8>?t0iA(mqza+kjkr{HGJ4VUT(lY%rZsqCEsDiI=&H9ZiSknl1
zq+bs|wrcCy{dz56D36Z9Lk%$aw2Sn|@ZTbi)wEg^KkOq_(qtjAxXjR52lajt8qL@_aem<@3$JLq57GCeI&h;f#BA;L
z%#m@<5;MezcUIr0ta{MY+~q$f8$B`2q*_1nu;X*Z*TfSW7(Dz@OkisYlEVtxq#ysF1;
ze=H@qJTh)0J>mowD!-=yR86jzYtPl>ASk38jtuo}Hga#6T$nZY{R#Af^$gsAjs+?i
zqHvcy8v@v`V@i^py7t!dSNK4KMWD~o=~>ijU9Pm8)3GGsnG{Kp0NmM8h{|~UUN$GA
zx?CKqvEI(;aC7Sk5>avK2!)D|os3KYe>C&JFuwxzB2F{oW=myHn{wIl+uMQ%kq8@u
zI{$$m@5gp^*FI!ZDR!S-=Xiz9zk7iqzsiMqI5AiCj1>0NrQi+w&SKW}CCgbF1|cIL
zQ;Z)1hTf8R>%Mh0KY>c+nzZjMyS&Kf_#zJf{?6uCfjOCKyZgh@@Zi!}v|+}We~%fK
z&aGa^5dquaq-N1co`MWNfR-x+@wz~pIADYNJy&yrPcEk=Qp-UV_EXo-2ogT(VIxAW
zLX_C@72ckXQZ@S7Artyg`)Dy6l7v-Dd-
zY{|aNsZgff=2m8ORL3%vGUfmc_XxZ>19-<@R7fH|UokKprgTcRQH%FqpEvn3A6J5U
z$F^5t8}fY4+hi+tmZS~tPrdtjOL&KlRDUG7$POw%&tL46eDSp2H&Pv^e+r95NFK_Z
z`Bh(ma)bsFNx6FsU-zzrk?j#kC2AI;EcigL!h4$G%fVp9sfNn3SO-)azMRZ#_|+Tx
zaT)WykJIEf2)}n`k^v?8Z9e37`rdwTiypDvq#i9V^9edh@Y=_zs%tn$rU-gvP
zdeu5(Tv-8lGe}1qft#*v$w!CY%6<_ym^M*7Oqr&(&5mCu;0;CGf8R4Ej1;Sstwr$1
zSDzuhkt!M0%Twgrgb4Rct9~677B@o5-M*t#60EDSlsQqx3{L!%2-PT}V$TAEL7h?7
zLLe7MH7Njiw88QMtBG$>*0xL?x4LI3b8qoAH|ULQz-Y&AF-GsJ
z?p{%ju+TOVzb)C*)>|JCYtzTea~L~?FJsA!4(py1#%e|36NtMF#coX{4FniQbprmTCQ2-6zbvUNG~GcN^OJ6FS=3k#VDiE6p2O4DS=dxn~s5C
z=C$;^#g@;HAL~unAv;2rdc^!3D=b-D^D&6vvEVZU7YX!p6vOFPuD7IPlTM}(dUFnW
z<(Sa?-hEz*#RP3z8v)Y>jc?gHms$xXvm6ehNM6Mze;f=0V=+_K_&*suTNQjB$qg-y
zm#L{&xf8k*=R-E!zOeupu9Z8;UcGtUNBRWtoE;no#pK1<_%SZw`dnY=RQQT}VvxMc
z+GL1KMxN2l4WcJHw3)te{_z3L@I77P9x_qwL*Hqyml#4t$|#%A_cltcbmC=6ki}jT
zKPbK%fB4ALbf#_d6IF>y=KbWF6PY@xi1K)j!}qv>^r{K0WUnymsBzQQ5H>hn#CY53
zf(Lf38UL8+J>_J#;C0RS-(#q4RP2?5ScKxLui=rK4biTl$T{8!u0p(~c?yf2v9noz
z#zj8b^PhXu8L{>OnL2(NW=BuJ?j~;lmCebwe_AOks-RDX7D8RZVxs1*76QbRPM^+!VX`TNB9YLoMLOJ
zu-qTKkok5jZkX>Qn=XXuuSnZd@izJWf9Nekt(fBvqvn+?A%CF0SUiJwGdoIqrum^t
zfBrdTE8`e%+^5>J^gbz*27K+#x008}3r5Xsv>v5xhBsGq4$@l|F5(oe%v&^}*CE*E
z-bAVTTbFD1Pk5@$B9wZN&;O*V6?{sU`ak~M<+`AkAH8wvSPjp5mv>A=*`eQ5(jm5O
zFmP-&DZVegX^3%g6=RF=>E6)h6|Ljif8Jr)JSFi##QU0CeSwMOH&1My0#p@Nfxtw?k`t;Oyf`}mqEsuGlZpM%2q60CPs8o`8dU9@3lIKO<52H-sCsq7~r
z^YE;?BRY7^s}9+J^sh$=-
z{;?e}Q<;0!#P&mZ{3|>!S$i;9P@+#|8O&HfDo%%}NC}J&{H|r>4&Uf`e=1pxw>Y;o
z-mJ<>(mM$cXLB~E-ej*2TYPvaclR77ApzCMYfg}}z2RaFc(H5T6UNCk(028RFv={1
zjL_J-8z00-dCDT1o`BCervou4bX>JoWaohCWof`qIbyOp>tvxSh?^D-$nmd}o-Lvq
zQTWqV?ZUz5u{a$iKj{i?e?ym~T|RQ-J!5hotn#Zg4%|U10|;+Dgm*f6_y)K3lxM5|TIVV-hgcA9`m{64V&S#ng8(d+EhT$rDAWGkvFs
zhQ$LB#RrpFyh=>UNI(0UzT)pW6gk=zHvIW{d`+>B8SZ=T?2zgAe|%bhfekk9YbI*#
z1G9R~a^C$#Ci%BDh@y;sUj+6prPNX1$P!uCQ#^KiO40yZZ8CZYQNHGB?)qVhdh`ar
z6HE0bUO>1%@OV`3(q9jI)eFq6W@NqWXH*IzC}jKg1C_7Oug~{v@Fb!iVFtzb8>IJX
zQmGj#!3@E|=XdW0e<6X!g0pSiF0+sGguAOveOJ36W13;A@&QX9i^yYEIy*!$^TM-{NzqS0+bGDfAUdp&RJ|;ce{_azucPK
znZ@S0%cs%mNc}P;ozD_5?RA~}L=Vq_5Y>7jy&Ypa**W10SL4*kF5z)IP-__!2h$O*
z1Ph~ZXUL%bgOIY0Eg;aSOi?h8_JpG^1|Y&PN2ewA1yDPzWmz(w;ovPyO(bODJojyq
zff{DJPO$Dye-ZaSc(Uzc6;35odO=QZjr}g|ojDP{XM=b70s8y{Q=&QSW@dD|=U0%i
z&}Euta*jObvIKB+zBY125Siy-_Eaiq$7hPLOcQP`h&}D3NaZrv|JW_s{~<2^R)tZy
z&g7f@qtUy3iY3)2-jj*$b<5e_Mc(Jnh%ZLj5k1l-f5*J%a#PN1s_n3jguAS3Iq+jM
znh^Re@8Z}~kPG8&L=3k>s1VfKoSmk+ibZ1CklybTJ=~Ih$noYe;MJ#judgblGYbe7
zTTQs|&LkBLJUz>yRVk@-U6pE_e33cPNWn6$64D1
z=XTEZe-a3!oIF|0Jcb2H8_UGhwW8yaFJ;B_sVcVL$7Dc%6%$_bbO;Q-Wc?MS|CWL@
z$m<2W!+p{wcs+p;uFFT@8$fdCT?<>PXP_RnOSWogUtpam9-GQGq6SNTiNG;IZM~Pu
z_!|ClB3q+CedgTCne9`~6)n6Tx}R;jX@l4_e?BIlVWD_*Y342Wb1BJ>?n8w@m!5J~
z-)19B%jrS)W1WZYzLxVJkB@o24~z#{etY&Ckg|c^*|}R>^Zh7i1A2HrH|EQaAuI}^
ztk+$TU0i;JUX=PihbJ9Ip_(g~LQx?^j>3MHp^0mMWv^1_Xvp_olW7#$?S-YOGA$tI
ze_PjGFQk;a^gJP1$I^Q=;p+~D$@~G@PFXY}RGys=P;fpEV6fUuARxI{Wo2GxhfIA8
zAS&R~oCjE6gh~XCbfH-t)A@*xN|hI_pi9G(p!pi&7!GceM<{ji9yEg#C2DL>Osr~o
zOVT1yGVpK34U70UQPRo<*uR!v1HDSTf6O#|assA;av+~ZUh)nwbK%aPQ>{N+!w_D7
zb?>(c*QNUaU5Qs`F7zRKI-jNC$eo-Da(H{>4EkmY=;aCu$ldN@F7&reWKC}{Li>;|
z^c~6C_=eQo;L6qKTxh`QwZ*bx?MnS&^lg&?rfW!|X_D8sZ_nrjo*ouy)s|NKe>@#1
zyRQV-I(Mj|!rmmv5>rg2NqMzl4()`c60lLYGM_lMRk+go6|eYQrA;=J@uWZM&V+{$
zo%O+8!AYb>`DGlZsU3V}=#VL;I1{%oR0tlJMAEh~;FuyPn~pH*9d+C(c$sO4b6Zgj
zyT#!Up6qM>pyDIAy#2W+(72Rre{kVpx~eYl>w`c_#3_Lf$8>{~i9_dN>I}-#uN!wA
zHg_sGyPjmOuT|K`^SpRv1tj4QUdr5qc^VHAo=m)!D2
zmS{#5&^l7fH_C!YRi>(WC)
z4wg$iwO?WsWs)H>9;^gWh?KNW`2g8g0>lAY$RL^(I*~nzn7)T)rc>GS!23{XPfy$H
zE>Q3P$JRM*iJ|~Gv~AnAe{I`#_i5XrwdUCRj|s
zgaMvDn-6C9T0AOk4;K!j)Lmd1i?JAQW@Rj6m1_a+0wEhv&m+i^^F&FpGUHOdFoe=+9~Po-C2OZJv@%^`E}V{C
z6|AtOo?=(|T4&S^f4Tb}74FJ<7roeL2c_=D##{*^(Yj-Bn?JtVxG(r4HItyl^_s+AW<>mf!oqVcrd;E!>dv^Q5>2Kl
z`P*CC4WmNjLJuc>gt(*A=|0Fk
zqX4C!0^g&m2CA&k*?)f!KzB(4kpKkV$UQDzOH02Ye|FlBw}uize42xp@MoG{5U)w|*070m?IM!(iNBm`Bt423S0vkG
z#>_NXe=*S$yeB^AmDx6@y@Inj6lc?Y(IT+D(pRnnxVj)^4thyB1@Y24*BGXQvQ9T~gz!Fq$;=)FK$-onBU
z+h_fUnz)gkDkpuf4a-0ECP~J1D=e|g$lDj3e+r<5^?dCpEa`H5HHw
ztv8-WF?oP6q|Jx=r3Sh9rCN<_DT+koKh?n-x}fPjpI5#QKftQ
zhN%3Y9*nXy))v471nGtOx1_19rV36770a(hvwauoB#8+d-`xNV$&x)7w*jCh|Ii&;
ze@n#{|M4vk#w0>FYMhKgNRZ;}LX*aUow~%)(_=FC1fVm-VMm!JI$sOY;G1;@e7s>+RNfkd;N9&;*Fg
z%PsOEvOPKXa0GQxCI?z3&xmR{TW{kHf9I|uiU2%2%FW&D?e{Kw0ze2XH+el$zz=qA
z1b$``h0l~W!m*NTvUgr8HMU5lQL_et`~Y{QbBG1XMkVF3-o$>*p!g~+7&YT`igJ7B^?rIBH;w`P?)MYDT#cYM~GXH1ND1LuQ4ICH0e{{@UvEz`KQSQqrpPA@+;4bh(!eJvMS}yx}YM$cz
zDfuzaFWHM6H~LtjmUIwwsR4=5g0a;&%d6VERIVJ{8gBMdCYO7=t0uTOu;m+6^HCZ2Uf4`@WcdMlo
zq2CKTAU>vVHjm=`kr2kgJVeGtC3F7OHPR3es*kee_U8}AjOJnf^)-WHiH=
zj{~P|Bf&xW9VEF9?A;wFW?!oBwJ90^d}uTHk@cDf=elS;$CU!FfAx>nTrG2z{(*mg
zUxNk-R^ZB!4+<`!MY$rcGNu$;aT;3DIn)MnGwiEw%saGhnY=Pch_L)^Ma-RC}@4O8dSe-n$kE
zinyi-(XHQf+K|K1e*j#G%NEkQ0D>c9?5+yN=OBo6)Po^YUu)W-RyP8oWMam(=
zXltXGVJ7QtZmKh_6v&DQN-dH5Wi}1n+CeC&o8)!{ZfS}XDZ2r_M{|@ZF*n%luN(ey
zvAm1@Z(^xUlXuVv?N*T1KNOfM@(LZjeOSb$6b6RGedCX-e|nAU?*pynjC7N5AG!BR
z&vdVnbC$pMbm8!P3#?|sA*Wm;*XtB754|r{L_#J>;uxiz)ntDt@Lp
zRfc@&cr9fH%F_i_DlkB@qG9_0pwBJru?1#BhpsT2;x82l858a@H%;9I!sohjGn#;P
zWK%UCjjYBNe`f4OBA)^W!6+v#vz;rsNeZ?+_P)|W`>62DwL88u=YPifA7Vt+incj!
zKGr522I^493A>UI6LRd-cg9`TCHF^BQix0|{1VV5s$hblD?)@H@ruM3W0BViu-Qy8N!PB1H0!R6in0S4Dt<{E9Tr1BC{wcw9}dgeOXs=e1+)CrVc|
zi;OKF+galqAF8?dadKnCl*G!H=P&FiYJY*L0+F=c*tYPsOqj_YFecT;-;RhBJivW^7HrlkeiLkktiJ9Q;9j<@*a%%}Z14JY#;0jIj_IY4zA
z<-%;%#|`JIWnzH6`i7L414}xYCIas*C3-%@dn-i)MhJR)x7XaM3d8IcjFdnqodhUq
z`sV2yH5KudrMPca^YjO9lt`EJKDQ61u$o_?RMpx0KoRUF-;pnM$nz94@{bXvfiQ+cE%-p=h8i>YX2H2A+EnCATdCKxW1idU
zV#wpSZn{$|6q3|+cSmdct`@8F@k!uL&A@RALJ3rGEn7N+1Uhg7!6a<_*adl;r@bTQ)w(_^f9*h+sb6oN{*W*10bKZkv5;$ynnN?3iOSC>o|2w%
z&$<0OU`fg7jPCvOXL&l31+JGFu3G_rSEnkfIi{+rv-pJ)NF`Lt@Uv`)h`c%4e!Uqe
ztK=!)3G-2rm6IT%beaz13G7WWs3Bz4Tt#nX=?8PLPVeh>?G&;X%>A;Pf8tlxq`^zb
zbY1YC^&eLcj=2^`uGV?V9HmgV3vc&WQ;Trfja}5L~?56hK8b@e6>YTQd)OoBt!p-m(luO6aEp)
z%pTPgHp$JYpaj*XYr
z)ox3P8n6tqE;TxpYSxXuo8_0bE=y3T(w`XP^NaBd*S6`D7)K%wGsCIz#B?CJ*Q`@O
zHJ&$#mArfytTrh}c5C=%$VId@Tv`@-QP$<97ELe8JTw6L7yre#SCIzsb6RWJnIq
z8uvfP!1SoRK6NtL+15IIrWj3Ub$7`hB_TTOfzqqFaq||8?Oq-ZzS#>tU!It
zC=$%-qHBSzP;heR46q$YGQ}=}t4eUJH62$d?u%(@9a6
z24pP@;r1LSnpA%6RE{BVP~E}wmzp(b;0a)ql{8cN7{BwnpQPp7p<1Jg9C{@-;K{&XWN61D#|9`om
z>MYLN0AST2OR{Xj5fKv_S1hk#VL5#hrlh#jweX7epWiI65vDc;+4^(
z(QUoi;glp>gwe~SkjK9ep?KNTHZ;*cj-r6IG1MWc6hhinB~nk$
ze~uHgXv9#z$DW|LfWnOtc{L3an+doJjyst2PG`8Yya+mT%)TI`NEt5{=0(aY>&Ssg
zaP&s)?L1jlb2vFNFEU1Bz3`R{K|!}q1U1v%yCk>&dY&6I8va0;3C3ptOJ%UNchs{T
zFgQ+s6p4b2d9%uTe?l}kK(TMYEDU$le=QcThyUdldL9e@m7#*s2S4-tco31@ZVuIP
zhIQ+_oqXYHxATYK|011vg{@X(5r}a?*?AVOgf?yzk*F^uuJQ9dW|aO%o8eV5LE2Oo
zA7D{3MPbToahI)DRWAj){fMR=TULxLO+a(e7M5|6MhA15B7A-KC^BTlbR+y$e`Unw
z9L6cbc=@B~$WLYWW)JITkdqvAci!*WXrYW}E%!zCnx(OCNOOapbzc^Fz4-`+igLy`
z*5S>1y&X@^#vGwX*RA7r0IcIwfLgxQVB9{eGWV@WH$>b;gOJl;|QIPnDDO
zpkS$s5Cfi
zv!E?Zb37l+N|$z@kT{6ef0H)_7M1EkzSARKfJ>O5(?5uulxxEKCMCfIV;H@luNHzY
z*VK}sGTiO8FqOGKM?gai29jp;sw2Z~cqy^vl*h~4!{6#EIrQ&fykDK8%y|CN{^X*u
z7%8o$aq$~L@g#AX_)xwJh?t1~qYZ8)O4#_h>D==IKVf&tx)(#)f9soM0)LEkvJ0ef
zTYTdE6cn*@fzabrS-ChR7)iei=X*n|zJix5U~R0QujBh8t%=@l9uA^M#i250=h}mY
zII-1r?N5|f@(fm6ZG)S|xGv*xTUq&<{CZWt;=8tXopzHAjGx(kuM&v!+cX8t$l)L@3eSr
zI*<8Czkh2Y)V)Q{mJ5N)h{brfB8n_wzAn?Yw#qQ=VVBrl&lkq+cp(xGH7`-5-j=%+
zEye0&?G7$5q2lgOI$a$>9p&&d+(+8R$rIbroWi8<3*Xwdf6ITNmpgN16j?7%<>K*&
z@cQ3z+v&Tl@LRL@f2QU18)de362Tma0y%V>UXtWs4|PaduTGAPnXjT&+^A8Yl>kjq
zNIPr$VCtA*J@RbGX9_aRq_sOK1aOBFOqHQoZa~@|m#$GY9trLMlB3fQK_r)e(bBOBzleRx)&#rBQLQ*J
z`rY5@W<>6Gyo=ftT=SS|;i6y-eXhAmsI~7GuBkh_e^n7xSEz$eE1~#7bq8o4UE9A<
z=bmD)w=koM+gJDJ6GE0+_C0|R`j@bMbvqcsLS1)#sJGLu5WyLt$0xlTDR~iY)z%eZ1^EI>(6dsVt2Te+QtmItDw(-}we6GqlKkWqxW^*)!w6
zGP>VNdNuQnFG9V_zZuBgi8g`5a(z_^(szni?
zaePvCTFl7REcrIIx=~%ZIc;bWAmB|oEUi(k*Z>V;i|2321InEU&YkC1j2F8+i2H!}
zAwcrZnd(>!C(;rRtQlbn5_|Nj*H?8Be~dMo#em^U3^~mKVOJ=TuZgIe&kCDOuh0hJ
z3Ze2hjXF6Seo4c5H8ST!S80{cXOJ@Ia7%2IfUZsZEn*f?mPjuo+iAeCAB=mF;kTsq
zZ~G@dtT)oHTn}EQ51i~4R@6@>d{i-Kccx-{q=eRTZU*VMJoK%m8}BzSFlMt7f4Gp6
zWWPq#KwE}lZ21tpQ9D7Ssy{sk%f40KX_~Q$JTY7TQjjK!F*`Dh
z^wkd4@<{FLX4K+GH*HOG0_Qbc^~`8LbZLi*H(p_@dV>C_8$w~BX{tIPYzLS%KKgg0
zw1HNyUTYdy`oEwaEYVcw!H}ab4GY0YMfJf$##(6PuEam}Rt21LfAJw#2BiYD<|-jw
zm=YjIOK1o?wRDW)thQc7P)bYA-nh!zu%?!7=$hK^hFc!f&l?e>atG_*|H2(0HdQgy
z0^8H?nVpKEQ4Jf>INQ#tJSjOjL|}=0Ip+Y7!JJl`&(RfYN-?DAi;dxxJ3?La1RPp&
z4k~I-vaqqI*#u8Ae?spX5vNawDooMf+q##&pMn}ksLIU!d3u^Ns9xB>?pmW|Kv6t@
zuB>19=c{&th^$G6$b^5&jz3;Dxqh9=l+Xqa1bu-9(&`b8^Ss1j6#EsCkGJJV!0L{W
z(xZXRKR#fyMcLMHkAr=2pXK$lk>@)YckIfkR`N)a9yWxVf1hW_CE95J5EW+~IV8)b
z#f6-AZP-Q?Mt_RKygcuNH747ruHgk+KP+)r@~Tz~tsQ(?aHOX_pUAl>bW
zTk;ADgy+2QLVieKng{qCn}F1lCMH(UVeTQd94+Ndkp0shJ0!QlE0n-&Tl`&4VVNYe{Py0#G=A_j6lz2`-r{41qm)n
zQf?Adu*FYMt2Eu&aDzHlM^M-)OiLV_P!c}6Rg7!=Q%K1kNaNQFIVHYFOL7kE(c&mW
zlyq}D7PvP>+fP#1;)M!aGY5uiyR)d$)ua@XKX*+Sfkk9%%A9a3vYlEPi%?7{mK+9c
zml*Ykf7K&676#<4x5b;z4V@$a;3TTxtS!A@a1k`MS|_u0Bim*F?(OX9ELHwlUn9-h
zq#H$p>E5e>VSQ-s2Y5oilwDh}Uyj!rNx)Fmp%*5;3P8uk^y<`Z
zZKFSx=|7m+Hn0;kQ>$Hg`t3o^h(<+b^2?t3e?7Xf$*-SDZsq?q&gB)A4{?rEwqUgf
z6Vv4oKnaw_if8OUMDPM#jTp?7Em0dUbl$qL{@qphW6Igr&#g#wECo5-$-QbRh=s7Q
zauZzBl15YAVO%zoQWj9=h~F)|IX^zaaQb_}u(7Ub`)kXykd2NW&_N(`&h&_n`6TnF
ze~_Fo#D%DFrDxoq0eS7KWx;C#`Jq$Dr9aSi_tdpOC*H@1Y>kt3KoNd1m|mq|9Y6j_
zM6+BY$fZmKp`Pz>w%ec>>n1|zLE&ThS7v@RClNM%H)9abP%Rs}$ri1FauFMYnBt!3
zy3uO}o<&b?|G*ji0SsA)AQt-^#sJ>Iui$L?OO(a#1YDtY5Xf*1*TK|mn{nmb9ng{Chf9!l<
zNAPCC(6GqP`9o;hVO3Y4Upvr9ebGcVkr91@<(obMk^4jO4fPa#(FOxXL;zuvdXY*g
z=U&Wbo#jXF_c-;h{g#+${DdQfGp4m7F0
zV&3#8Nkbc_lmu%bTyk;SdJ3GcfAMRr@>Eiz1skDuBWcV>Wch_6nIM*fxqpmX_*}ft
zv}3FdO(brZDthNJ7)DMKVVJjwX$q+%P-~HkWH;ec5gsVS+4yZ}2ug$P)*N$sv8XD?
ztq4)rb1h>RXYeuaT>UKa7PhMOQD#(#m(i$YBk7G)a
z!SF|UeIQkOkAO^>qd9_ZY8aL41esJ!=g+kZ<>Dc#uDOH5f(q-B`C}WVjkhK|8%7IB
z6dQJg3AN43H};&G2Jz4ie+Y8y)Xs;N<{4>><-y)0TU&e$eJ8;3;qoVct>Sg`gkQ;n
z>%8sQem`Fk3YR6GC_t|{GfQZ#o5)vC*>95z`1&4+|Bci{pu#%_%r)Get
zwiLn471%F^PRn1Mqm@PcbA>ceQu-lMz`;t1C>@#V+h)_`t9lNYHC^ln&`a%%@XUSh=k`Sc=1U=~C+97fSjU(+j)f
zRCr={9zP>9s|U(KJ~pAzRV^H99&7d+YI@&11`*reS~={nZ@i6zPc4809(
zhr9FG8tvv%`D5b?8|z#rNE5UB1EWCkM+TPsKqSmVZv~rUe?i(8bni=hR{!y56XKeY+&<|D{I~X07*c$zpLpn_<*VlkJv1>eG^#Z^p9Md
zCAd!wZPur{KXsGQrhgCwfiKw--xeT;Za(HTO^{Dpj>BdebPycxEGhHwZndP~@M$un
zSzL#V6#j)z7|p5GZu-LLfiRF}qL(JwB#{2$y+reqH4~;ctObG*B-eG*ma!Q-siEA^
zc?K6xMWs2wHIY2Vcya@zP1kQCGwNx?-jn2�WLh0V3+-ILtp8mo4s47&-}o<;&h4yUMB~NWKG;J6)C^sFE>v)vl!xc#15$V
zokb&~M8)85?1O9v7NC*O!p#W!oN1p6l~LC5mf&T-XSzP;j4la|
zVg4BGa+q`p-hZ(x{FqXsv!8fg4~>U=$iuBNSwD(E0E7Gd`3iM}12_fsyp0i-Q4PpP
z{OwU2RL&)wK08zcnq%7miZpXck@+&`(T=6iE()+MYeZonH)RRa1BI-iTzhU
zA(kmeYrp$0L_Cq_;fq#JO`B^WSGFX?I-=|*=wt(?+=$_jtWIzG-x0E}8TEYFVr4tI
zp;>Ql(gRf^C4s{HdA<9kD}!UK+|)Zaj9+WB7Jv6ZfP4uN$$RqO@MZ7I&=)SeD)&Y)
z9cYC7Yr>EraZQ#!fSQ41@<77jq-yjbvj)9T`-|@14Coo2k`l(&vh4`|MaN!d9#L#^ZkuY
z*?+3u>XVF#OM^OTrCv!+_lbFYE9CC>rIyMkagfIv+rdzFcVt`$8c-5fo`XVYXT}b9
zn804;cpRSXrJ#iY*`023JS7;}92vGH8%^DaL>0p|{(HC!vt>Y`deh>Tlr
zOW6`C>D}~QWMgNGSZ>K2)f{21k6UyV8jA#PRID1c5C@|)#2UrLlo%~A<_S)D#$Dbm
zjqOkp2ck&M#h5|qO7^;hN@K&8?ZwzN9|1P=rI{COTAOhGEnnvLShoXuk{=<~8h`#m
z^q#~~GuPO4mq_mAq^vB;NL!JoY)J3cqmD7&{Du6+oYL0P)hTLzVRoL4+lWbah>($0
zdVuuG(<`6nyk{dfL4{O3VH+Ox;;z8Ev-U_6+k2f)koLAEuZ)uZemiDQ(<^0%g<$PA
z(>syh138
z15oCL>z+}$-b(GPx&$CRs7?k^miE#cEnKp{%MfK7o<^5V3ax*I!^;+fvPxDkCxX}gv9)cyuQuHWqD?0VZo2#;uRyl_QxsX5e2tuC%
zKhKuHu4-o9sIOYqtz%Q_$bVk0rkYCxL8i!x7`bD_`kTP&0Gdp9r_wzH5;*5*ER(sA
zdz&xNR)DxV$dd_LM&i%b*PN4k;Z^Vx*t=Vd{0M9al!q|FtgcLEvtlMXX(Z=&t+d!M
zb-)1y5a8y9cuLxInEkbx7BEjqDsq2~jd_S38}pWZYRw$&qi9_0^0OaqlxRUe1oD?QDM|w>^}P-xD;P
z;z}3JsbA#N#T)=kTqJKt3#XsKLq4@?>zn>Db!vr1^7you|qB#)LS)TE2F7F>
z|0ST(zjsbGHdJNW_D61wPJ@dapVNZI>
zm^FPLPI@C$igesSp-99c({gwHI2RgEu0)OcbWS4i|{q&C5k!;0V
z3v(`#Q<%B&p@3^#28i!$2Go4CYDSFXAwdMh=>PDeLTq>CF^VNemuJ}q@Fo&h=v~9I
z8spyrkU=Xzu7D}Jbo*)1>zEl;P`&nh7;&eMyIf0&Ph$6rBl_xQew77f*NUXdq{&0<
z&M(_8mxcpzKYu6jdI{>y_KR~cg$PZw6>ROzPr<69y`uh(x|Oa_x2@KX_7y79kSEdG
zXRxLRxhBWsbzWWz!cnuy;T6z_Zl7yDJV*AkKib?h3`cg7TR9Ss#V~iyz{@%t3u&P^
zBIwii+db|~j8e2~p>TNfQO=`ZqxJe)U>x9i*1&GOQ-A(#zMKgQwsY_QNBA#C<$bNl
z&HJQnM&v=N$~LeA7jhjAhF9Fh`4XB6#EUCw>EQknmGNFl?;iQ?%OPP<$vdbzi5jK!
z0{v0C$aco!gRiu>Y5BKx_uSHa9aN-;8gT5rwv}~Ar@*O9gRcDsy8H%__2a(95tvi8
zv1g#;3XXB5I^C#^Q*4V0f2w?G`Y^pH|0{?n3T~`
zU28l7b&b(xIE}}HwWJU8o7z7Da@(kV;F{tIlx*QM&l;zetvRngRu=rCffpw-&6gRTZo60=KOH_U+v4pmayu1PrJ=$NkyaeFewp8ldMCY?}q$kCZ9*F
zaesVg2{i3+U$%dh|I@nYAnQD#3ZnXK
z6Z7x|z80=EKCjWVe?>kCprNGEpde{gj_W#NHh4Sv6JGulRjkwRhy*g_jwiaHmdFWp
zhB_M_&O0KIwnNB$E(t*P30MWxOZah$Qh)ADh83@>n=Mw4RJaPvF8LNbI(tzOflsgf
zO#9h{HT*2o5W_ai@2>NBK@6{skPY#kF%etk5BONU$fF>FT@9_&8(g6ul}^jZey_n8BN=W
z)kRJO$g(>LJ~*O@pOJo{u$(5yCUGDdIX;iTM-|R{76#)FzLaKPKp4Jb$D
z&Ew5}mXdu6L%S#E5DTyiAG%Ef0J854@csqAv3dix%yt{ltQk=IEIZ*YRY>m#C4uK1
z2FHj4rxwvcZ!aI-tKY^StA7;v4zjyZ$0cZtrb>TV6x;w8bVEOU&xz?Ppif0*(475^PB5e7k?{%Ado#a-3tc<
zcXmfc3)}a$D2~FIRex_49}u&p7{;GHTUHg(UNemZ!s4&jqkALjR;EF?R``05t4YH@
z5qmtKc1B3)!&+09)Dvfc`DvT>5Syy@`7u!Mq8q^%c*T!EQB^PR@Mc%&fAQ=9$7-+F
z$BVki1;4sT!F#3y2)p${&2Xs02T0f{hkx*{m{6V&`_htq`grIgI}UI*M;)>u%!bnq
z%F+M8>@>cW6MwDNvQnB=g3g96*=d^V^{T=^_`Fov<4wP#<&z^q8{Xl5QlOnrQ7Rnv
zfB^9mP_zZ76DI1ff~{*zvwf{0$72@H^fbaPhg&uo>UL2~jJ_JN4NB&YTg-%#&=o
z5DgzJyMIdH$q=I_pG^KRWDgI*oHP=k=VNQSa(kswWZKsnZk#k9?NQymN*E%2$$UaB
z0-om!`&|SsIQXqaB9~eI6NDog$voo9Mz^aBO>F@gHM-bTx&BPfzHZ^;JX+R9-5qR9
zs1dwQ`Ebj?d*Vw7-lwYP72HnLUik_1Hmd7e!GB+vrD^IHrzt07eL=~+KIVE-M_L6A
ziV3{-r)#z@s3zBPP~&CC7*ub&<*5XIUs%h+=PdG-$&bT0FWv$>sgAS|IoJHO^e`=_
zB-|8$zvc?eV+ZTBE7%k>>mzVHg;MvyXDYIhuBehi)wATmG5}ZDsbw#WnM{~pIZB|YU(zG63Z1(J}tsN
z8onz4A0mfMv7s&O`O|>p7BS`Hp=pzZG$Sh|saUMihfcFM-kY>^IB-p3j+M@qwg1o0
zX?4Dz%yhAPoLaWLI#Gghu#bl1*?6Oz;(wHpM4~NOmG$vsFf6ty36
zLE?ARz0$f|9tvooQTcnb7!l!6&b=U(LJ@qaHu}%4e>@WE+}xg!+!^P(^fAQ>m|)Yj
zQY552f{a$^G!M*V)7yLvoTC%V3
zL_DR8tmwwD`i**)usq4Ao6uHhwBf3YXhc8Es)wO~`(P~7waS0o6he%_07-n}{S2wK
zm!u)2LfpQln92b&epb=b*+_HtZv67tvafFgZyB&SuGx+d%TE~oFqIIpmw$0c9V!-~
zypS$pCK`@<$+0+S?u{YlF@(x04{)~7xlpaJ<7a{eR=q0_YQ*;uOYX|mlIkf(Fr%FO
zjc>8huvM_?Zc@PZ^b;rO3lF3gE>$MZ<_{fcWpMT1f$*Sydrwvn=wVJc2V2?T)Ue5l
z$#?@h^d=_1K}V#l8ZTlOOn;tFz-oRc<{7`h2Sz}``~8T9!ApQ%0_%xczm*?+lAVgA
zqm@)(HBUF*9lzXr{}Njd0zXYl`gB(YnZk|j$zJIU!At#Rmwh(J?Bn;Q(ZbP?MApXl
zf9UMQrrwn-7EU@f1P5%G^HMne<_fdU(adpT3UQ`I?cGB91XIFciGSS^VVfy(nTv|r
zCQ+@39&TkP!hw)7q<56$(zKKW^O7!#y2EyOo{AZpXK42Vf6mGk#8t@@&c`Xrh@Sw2
zU#~Du&6rEd^)fn}b#`t%(*6x-5Af>$
z4$<}P=uyg7MOFwf*BeFTe8qTWhwC%`R{f3Dq{bQ|H*L8o;E8Xx@dmT+mJc%PnxOO7
zfK5|HS=(0bff(FfG$sT^iz5j5Up0t2l!LMNe;eC<85q+XJAdBfS&Dw`cxa+SDN;yD
z$5x0lUvUREO*vE_`a4%mBQhX*_i3P{fUMiL-a5n7l%(5vB{~;4)-Ul24-8H!2$@j>
z6nKFAYvpna`~{w>HiXfZrTl?)bxQ%rUL96Zn_|P~lhs=L^KC;ON4MAy{>Jb#R{{z<
zCxG#GIV6}dO@AKOt7wxd|Bw~@bw*l57aKPsR7%;}zv1>^sYEzo;8J1L5vSn#H{*5GaY_YO
z;9*%+hv)#*!m*KbMa|G?)Wcy5
ze4b95tA8VrD4U$icTZ7fb3!%(TRb{~^dV&9-c`h10KDNeLw
z4NLPSTd+>HA4qxtw9gXSIA`7Y56kyyJvCrbdITN_K`v(g(Os~P@uGrOQG*|{!&WXm}uz$KYDv0u@l{o6d>M!y536mebj32w*%FX7|8XDP+dG
ziIdEXc_dDz*|m8;3ZMXy=ME6LWI~O1z08pLm$sMx`;AEL-M6Y`^ZLPuGf5@@-Fxp4
zPJb`f$jbO$$IN$OWj4XSzEeb+wZ662*i4PfB-rn`7jX)B;%9C*-r;v93{Zn9M5S}#@(K>KWA
zAiPlqAFm$tZ8-PycKV3u=Ci}cS3q&bDSzil6O^a*U!%Jmk)=^dd9{`@GiB5
zbEu!{#GP^1W(`s(?B3$>^Q}FMnVF&P;oY)vr|x3i`kS;)tvDpFFD(B$ZH%FIlO}2DcPim<96vvxvpMS?o
zWz5(l)l3ojh9KfsSzj}(4dBGHy5gs3BYQ1A+5zs6!UqLDy&j%-5(P?oXdryxPS3ekb|O^qtAJivFTMvI)y;E
zO011d<@c0V8;6Z_t5;9a*{gR_M+>I6DUXq!D`O(h7y&=Et4EVK_BbH(jRM39vHws2
z;6z<0CmVWFUyqy;5^I|F698_8pBSsR@-%wMD)h(lj@Xf?V3GoSw
zB07$q8LMpUtxg>o{mV<8hw48`4hv2>|0%9SYC<)FiqLxs+&qu)SF8R7)cMJWIn
z3oZM=vvH$@X{YTbdk41ox=T@Q^Ix>dK*^4EetnaV}HfV9{WN*uM!P=
zhW<(6HoQ||--Fp=KK96W&4T5>655S{i+)9Z&s*@*ky1&XqeIV__FNAzm@)n!
zzhshGyATIR^|0c8R#|n2$rtNf>p65`tTiI)BMHOT={3bE5q!%M8%$?6ijWfXxEYBu$Sp@Z?F{-vKkCVQaBmPLupbu@ol
z9BhxcA{_q8A=f7upo2vq=lSP|UbMQCgIT|P6L>hrXUi&a9iu?3BvOv@ENji3fJb#djw^reHxo0ck8#5Y_
z>M#i;co5Q^rN14^JH_4)yXIV?o(+X#?W$dXaMfVlf5zftvS=`$TF6M;#hHZBEU&X*
z2H$lZ?{XX3pK0p4Wmd*BV&qS7M$-#yFao$P)~gls94`_PZM*g7>Q)mirS*EuxdU*e
zh*Kmr=M7s>&3`u9b?)>SDQ2rveRc^dGa;PH969hk_lREV!W$zuU%+CG1ygnjR
zSSKa;Q5YDXZ>)vQ6Uv)>1R*7j=&6Z=Opcc5gGjF77OpuJt;IDRtLfHdCa&j|@2K>B
zh?SE1Bg*qsM+}Zertabag@3gPmzqJhm!?03bD({2jDI8>vEHzqmzH6o4wB*Y=~O`@tnibN7(Dd=gH#mt7vxQG#Ny_4%@}=k
zV{sn-Rd9GG2=NW8T9cqp4Ve#-3uYGGh&zo1!u#LoQ^nCVnVkR49DVIv;wk_1wk{L#
z@hzTNlz+&E?W+H=?*sckZ!|H|H3}X70bZ1Gux*+X=Zd02rmt;I7%ZXIdG*(8M6ja}
zqx+%BL&3CjIkAsqRJ+QX5PY0rkbq$Z5PNe)@*8+WR97mGrE*J_Mt`7fU+2MqQ2s2l
zUSQ!YH#?Nidsk4_=w#AT65kVbqk=V+#@qf4@P8Yan1CLIN#UlvxmgBo{>7+~oXgs+
zH7utMP`+o|KLwFTpK?Z8%r*gHAq#d^qp3+UP>->l{7k5KUcbPQ#E8hou=3$J?tOQz
z*ehy`)_wG+n~>wYK9Jz1MXfSLfRWEGEiJY=K8{8(S&;DxXp=~THcMANNUbSQ*G?5j
zbbpqpe{;vZlmE1~idAnoh_wXQHb7f^W!wAw>Tt_b4-+c4=8{!3QvAshG(y7REYFLd
zM>#r={GrqNU)S{k9ZhPJ^JC7HC;t}b!O1htQ=R!I#^-gP?=PTZBL&GFVqgLp#=fdZ
zn9%vZ6;#3cRy~0Ykbpi>I?xM1p9LF}>ekQXW)O#?9_@v7LDyMhnsenE7
z{3MQHVmhcUbYKUjhlr9XwQRbzo|cuo1*`X-cUYnL=@yBK|yqEY*pB
zh94X1q<~0rfBDJuYayQw<*0fbzwH=}G<-XDIQmzl4umiZ?MYRFS*yQP7k_RYTWo(1
zl{odU73F5Ae;_qE4kL(Aeh%9R=%=>G4gTAr1|rUa9#Nbxc
zt0K_S9eD9ZTx(@~vxEPWTtkB@_1lLDdjU{@{CmX9OcT!%
zx|vlmT!s59+eX+{Rk7vXryAPCo3
za-bL}AGa`#Yuza8NxM{BW|HUjImohn9N)~Sf-I&8tN|k4VY%o9;6rspfH2>Gmt|1U
zVrOn0-)vfYRtaNFzk!t#rL5I0Ht$Mwl?7=^xFWd+3NyRvonR~*XMdHjsy{1Bpl9cp
zOpp;IoQ%EFzd3E0ixLu=i}7VCr){aeJ>7nt8ml}qv4RcUmlOei=_?kk&>}Gh7GRBb
z+tksZ^z@g9#P9MI=1-o8B#yb$%SR$;-jf
z+?vi=rDL?dTYTYrEPu(Uyl(Vk!Gd8?tO07W#w?-x7Vt#@=v4qEfdRF=iuBzrtks-C
zgALQ^`X>LGs;;dsi#Z~u34H|Ctyx`f1wW2y{s2#1=}#AZli8wTsj3lvuyD*jdNXra
zbFF;QKTk3PkH~UfCc~G`o5Mw$u;hc%(_UHR9}p#r{HY_aBY#HZY1A2}2vKqI{mmCW
z8Z-#i_1{7=`JZY0|E^x@!H2L$Orsg6CXlL3sKfW!eZ}cWehpV`6a>t()M(aqf2#<~
zmn$m?r1&eBQ+B&%J1;6fBaep{SgW;Lc?IDi%;F7(Z?JP*eJC-IOd0kXCh!F5C_Ke?
zXn;(xrvUw#pnrbg1MQau7wp~zco<~PLO2iPZh{1!$6ecSQ||0{WXcvuW}Q
zRSnd&<_?{AEr9yBJ}7?tp}4;tMV*9!XTpDWErUP90pL;&=Q8Vru+}Mmr3Pq+dx`bQ
zhE$zEAb)Ho9zJ{$Epj{5SW(%(0c2i$_%iYjJMT2V>}F;v8`4x3kGcmO!P2jPiWH_F
zGn51$Xu2=|Bu5)Sig`AI%0LbCqxa7kxx^URFXNpQk9sTz9-IL$S)UxIND_K@9)RHS2CWFgW7k`5f+ZB
zs(u4>y&!vz`+d3x@ugaciwV8b12Hzu;%gJo#eDw@qKD|-w>-6d>`!LEa@dA#`F~f2
zOy5*|iCtXNFVy;%57dl_RLc*aq6wHhVY{=*g4e@r&Uu+tjfDR&n)T|`Vc`;7>Qht~
zop7LVzgFqVW7Xnpd?1juANbCml4F7mtjP^R!rP>Aj--LnjEC@l1+M>U)`sAU%W%5W
zLDBu34>~ojQ7ItfyzZR|%E~MZ?0?0E;QNBS%di4|9PKUjD)~^7`Hu^(z`NtBWlgpA
zr!>V64<~xq9He8d#;YGaGq_+P!BTbX7a^+4u0ld>VBT!khK&S{qrCN37C(Z?0!IJfG?wF3bm*cP?dgx57^2AT{Rv7yMIx?AY@&1zk^hL>RJ*oYwthTm7Io9O610h5yurwH~6yG
z_H-n42wSa}RN)_ltxKyCJ^WAf$HP)Eis@{M(ZswC5~V;+**eA|wA>6|3+A$*;sR>F
zx2jBb9dYu}sOqsV+*Q*K=u-tkdQB(82A^!$Th2&Zj_!#}TiMN|vz}#?J5*#E!fcuQSeN@&M=cDZxF%WVc-Zz6NWE?hB~9uoClyPj
zxYWf9b$o7l6noAtO@C7eoRXeER%stLpjf2hvK>cMDxvWAT97i9ad2aP_l8Oc^+Eum
z8|v=r!|B0f8!=q(7CT=BIN%awn*XB20PMaH@2l(FCy$*=L%eNxlYrr#-NSBL__{6y
z-2@_iSY1KobiiyfQR8j*07-M3fJvpNa0etGt@kgwoOms+q9`Bf%lk!3II=oTUuQzx
zCC~aTE(&FCWOHBDFWF!lf_fLws;9u6QD05$L*LD}Bfg~1r;{9y*!nOoX{Xg)&3
z>>WIuEX^%k{^np~VECKqZ?q^QKo)3XZSUr6Z3zI{nF3@P6&L}E_HG|aO8~XK9l#i5
z0kkm#*qZ^=LD~QfH3?NUfRw6|hO!!e4dX{*HCG1*d#C?{MNCayLy8_CE~2O|0RU;z
z1Ee(6)c<~}gX})=o6`dn)j#Zi>wGx=4Ofs*7g5(ymSAD}dkz2=z*mryv*q8i|ACG2
z0~z4o&^|)Voa}A?DFC3haB*?qWnyx3b7M4jb#`I2cQR*mu=xj{x`m}Pz|G!&$r|u+
zbOPCc{%MS>o#{tAT`WNVO7M440P>b5AUkK!-yli*e;I8*TKN(5p?CRjU>}We`J2<`
zU+w^B5a@rEu>d;%6DzN*EDx{+TH3jQ?0|MAAC4|S7guM1(LcJ6H^`LgUjRV>F;^$2
zzcm#8%jERmW&R7gsQt&8>DzdJ`2gMibH#vmuFjtSqc{Kiv`y^ooGqPQod1;(1TeF-
z0sTeq{CCYP?f%gzh$zZPN~ozb$bS&ejzPixqdRttF77V>c>m2OA}-Gh;AZ9ou<)=0
zm_JA=VP`64Z)^L3+8N<*`@}6j`s8Bou6{;e|yxd4~}AV&bm-Nb_FZ@GWi@>k9BSN+ijA1?=c2Y?yS#u?;e
zX$JZ@Ab2?gzk&cRPOcyyum5TI-x2}~7r@lg#N~tNAH#z1Pj(qQGkXBfztkT{{>$`#
zC_w$spwfIysj0o4jR(Mg6l8|Lq-gK*K?v&qf9C8zYDv1<*eC*RLDc`*(*N`W+FIIp
z{4dY{@X!MN#Y(Mc?_>+K`45|=v!tav$W+t1{G-zN
zJ5Dwqy#3gFEdOp@00tIj*8i}5P}0QO4&>|%U}OKs1o~*he*pe}=>1=~046aNElGJX
zy8nwa|F}umnb@0J+L;4bIXD48Cnul>0`mtESUETVUMwG+HU+u=!xR7$qn*9WM+m^d
z)x`&3X77aXcQtvq08GHYs(&OFW)=Vw=)XnIzmlUX(B{829IOB)GkaI3{}I6fV6yxl
zA~%5P-|p-@0495XyMHHSX9F-f*nABBfBUnr1DO7u?E^HEGw3VG?teIN0+?L>?eKxf
z#li{n@01+O9~w9N{}J$k;OoCDen_4mr+?Z1cY~GxZZQAM4)eb|{(pAfe`GZmCwps<
zmZj;(Rp)=WC;(lYEZy~(KZwurq5pXO*S9bKs{qBnXZC-8M2m{ryL&OPbF%>$Sb10h
zEF7F4A1tiwKL0D$9L4YfsyG05TDhz1x&!8-{mzU=K;+qs;SFBADjAiq-)|yx~d1I;~
zybbhK@FhThmJk`u(PY#J%urbLFaJqOb10jWuT#Q4lhGE}Mg-6pJrd*T=z5G{op?vP
zN+#7WZ~K!6?N&6$@&nV!2HkxT6~B{UajFyC<^2)!TQsm&WQlgxC_fZ?R2Igerwd~*
zyX`=5%iecF`LKmO3;(P5)uGU%u?jZx7v9Jp=ZDRI<0pQXHI+1&!4i%=eZSSv$qkY|
zyD*fk|0aYAF!SEH=&CySw40H_7IUmy{
zgu{V<{)wTU4WYhCnZQ}mDf~&AiiP}J9$*Lx>t{qP_UDDz*JR
z+QCjXN@-vZjo#2i;3HNx2U(lux%IU;1WVnd_)?Y9b`8?%1b(8^p~w;Bg!H;&SH;p+
z15LA@Z~FTD>x`mzeBozRb^msbKY=(g>)7&tSu;|%fS&**b~++tIXD;kA?Ytdlq+le
zRvj6|O1c>jUkfhd-(X;3)yZ2#dEC}^)u7Q66gVfN7{%&UGqU`**qGE|z#O_eU;xb5QDk
z=}y~@3eA*}W=4{5UGxJWt!#Fu*R&E~mj>&Lyn%;t1(hwc
zRf0h36wjpOC^JtIve$B#r5oJ$gtz8<*c?&9%r)jWD{qgsbyi&d%ht$w(b08_P80L@
zP&RTHHf2?7QbM*eh}4j;%NVU=BI?M0kMK$HY;r?jIrd>BJBgd8^u$^C50i*KcNcWI
ze0>3$d$-6O3Om>hfxs*VQYP-9e
zco{InzDuj=_j>KJgc7JtXGu7PwHurJ=Yc0}qj9R`06&u`6y(NjHWA&2vf>JV(;-Ye
zwtBm|FVkR=@TjGP$Fw{R__i1n1il6eWEC9niZ+O~wO2}&EIFo80t?ZLH*PL#{4X7*
z#xxL-9dOT7$;pjU*1-le7Gc+?KSkcPqmL&Z9nG+mlN1M`Ez@
zp`SfQM>`h?lq29^=(KRf3Z~b8``={0MUzCe1=_*|>^&!CjTXom>K%QNuvA!(#m{j#
zqp7tldudwq#@is@z#k(i*EZG~l>_%{KQ-jY9_SFR)@uC4nbK#mvyqNWA{y8k9MmL<
z$Mz?i4m(oFwUG9`om(w^w^?3r1G}MW&7QK*rfpsmQc<5Q_(i734dFr|=v{3^e4b7i&BMX5`HO?u|zQ%j;jRF~rQ~8R#s&zzhLe(IX
zNJ2?85yr{*Dm9jY<*(3~*YLLJCs*)JO|ht@YTvw8PVc`DPUY_%Fj@%j5^(ox<+Q4C
ziL#hhZn>0mul6`KoQursfne${zqF3rIg&R`O0)X4nd&EoOer#d<2xe0I}xUSne-z<
zSeASRU3Alx7}=OG=q()JN6dW<93>uFf+l+>29~rWys0`tQWXCT@#o~SQPQa{#eqYH
zlH^|IU{R{92D8qGktF+0AKCA5cH9hqY=19zNE6W)c<|J#ek*n`yN_v;4z1-LQ-@Ko
zE!E`VzS1ZD{yD&Z)19+Yx(it-uc5syBRZG+5@WzikKqZ_)ixFlH0CX3bTNXJUnrXr
z7JKN}f_(+BESZcTwt-*|kF;GOAwo8qaSs1#C0fY-_fLcJtkC6u@fP%ndqBRqN}1Z+xwqm^Y2-#as%!Df3YQIgpL
ztR@3kTSf&VV9vg09SlIB>@jaXy|!k()38ToZxoJ;^uRMIT}=rbMg9)&m>IRTu=%#_
z7?qw=eou9^NDXv5qiUNH9yAAHxgwn+hLz4z(>o2av%9;v>DkkX1{NDJFVOV=xjNi9
zrHv#nps$*LLT*WdiBIj7=soMDIAg=nfQCqufLbE35DFW(IVWW!-C97UVhC>81_E#?mn?S$gl#rGdV<
zYK2pM;FluNxE*fr`g_frz|sBOX_(^B!TkV=a9$WRNZ@2(GO8B}V7sz*_+5H$ah5FdN1a*!xN=T^
zpazS{@6K8@fYA+Sc$|J6&jEO-fdII$$4Wu$*xQ?9~
zyIA)xISTuF^nGve`=_PbzuSyVnBNVXMeA=I+Tzt2i@KckS(uK}@rS6h<-}UmW==H2
zjDI_V1(kUH39Krq`=T`3e0DK?u%!Hd-emhrXD#qh(mRzxkCWipye5wP;!%6&!0kO~
zd3DYNmi98Zu(~zF03lJcMCqpIe*6j4WWW=
zn9~usm^0e>&9>;*pUb_#e8o+=AX!`R^D+qi92bu}g+pXnf!Z7&ned-i&Gw&v24+)M
z7#5I7OJj5(J)yf&MY|VJ2qNk^kk8zE<#vOa4|t|r@e3=^MTB?yZe6LG>rn*afAL-e
znW~$}3j7I2@qK@3;mD@-vUIm=|4c{k8CaMaJvh&~6r}kBqr~svjA*UtnVvTeSMD(#
zi$e@`w`tltlB3!2MN!a1z^MO!Gwwu=wp5DVk~UJ?`eh`dQ0%M^N4wEbnEr{5^P_Hb
z7`b+{kcIu!cQnN2kX}rvR2owVH>4p-ddIV_>>Z}WUbF8bJ#`plo=szrFXRyOEKryf
zxnsRc(ND3AWh$Nn@A%nlRjM9_AeOl!O_=~$9JLd>pNV+WX6Ow^Lsn;h>owB$+HY5?
z!A(uTNOR+4chb=ZE$i>a{mb+7!)LHH7EJMm;8XQB`C+?WA-`?&VSlBKoXCso)zvns
zZcmjYdy%s>ce)wJ`1~Rz5sUqdB?%s&WiR7fo1V;^3-b;ScZ8GFkN%!@U5sr2#Bus0
z8%6jfQmxc{$%P3bDLQz6dS42Dh&weqVI4PMavPb@=lfp%)3Hd@?F&4p%Zo}XaCjch
zG7{=3adG^`EZpyUzyw#*_-qT~B-D`v6*+`(fwP(jn-fPYZRrq4VGibjgX=;eXu7zx
z;7;&di*jrN>D6f}N0=nM7_}y3=X&A{*qIEkO|n#Xpy{@h=Ir%<@_43jiYsyt$z>2g
z6WzkIUTRQ%PTFHDUzyj)*~z`0rnmHM)CsOcY&Ak6=JJ|y?urv;u;so~xb}@1s4s(7J@wo>}18w>tk)D#!Inrs#7Ayoc%wf0^uc-rS
zT>-t6=ts^GEV6KaAwIea`erSRGn5%bVgdX~tiXJ<{x|ZIK}O<~E}&-uV{4Rx#lgm^NcARcIy6
z$7hy5I-qJbC?vK=b?5t_%(v2i)UoK?T`Z?Y+h17woCm*B
z&U%QNRg`02myuK%fF$}c+3h>W?s8~wy*N|<%t9nd&vopKRZO8+v9tz$@~rXpb&!tQ
zeuzu!QgYn9dp>ag_NWEDT>J3H{>!$u*YQ#nC(&ZlTyOkZ|(BCP0g-n!-iXN#-f)Of;kUxYGvzzlcNJth(}Pn-47W~8xv
z+=lSE^?BXaPy?fB%VR(o@^W%5a9pkaHrXOYk>@3Unw+DZFnSAxb5G1z?{bW72HzPn
z4Q63*MErz#SM_KR-7N7=VD#sTk$*EjM28Eqh9EPdL43y+T6c4d0Nc4OY*CmoTxeFF
z{d0S%3isUn8m8N}y|=uup~Xq)T}-VF8R=K4bZ$L#zwjV|kf|+6S#NQXC+U{wJ
zVaQ>B_BxFT3G}Q#f(H#KaTi9;&xpH~2OYx;8Ay3SFp(k+^;&Fs9~$Sp6A@w)Rw&i4
z={KfofxxOd=XbCdeS(_s2;2XJS_N)fY|JU31T}Y=GT=LwCMq$BD40q;;`(OEa@Usv
zPyC$AJc?m)`l84dM$|jSfa#LI-(Kft$Out?*5kJ(!~M&q>%{%d3m(st%2gLI_nj?F
zQGh56!TURn)2S)^HFvh9)%ihxPRvwM@zHsB
zG17>BYPr>{C2n;-l1PP*I`1@yRdu1malxPV0wpg?$YpU!Z0b<>z7TYqtRMWrMFpgP
z60TZ?DQ9VmlS)_4l%$ZE4s(7|6x+fJr@#0^Dzsh)k|WuU(2G=dvprR=h&(lOV&761
zoPZ#b*qcWroWJQ86+nF`;R^DXc4Sfb0+w+St)n7J&mR9XE*vz<*w&pM&{-S!q{3cc
zUYFWMs2#rCr@=-1*{yG%R?qJ{U7PHG06n2@C+D~Nuc_}5v4gK8p`-opkh{eY_vx?G
z6cYZGRnMAe_RrjX;g3MEwlHX}u_!@Zo;Dq#R`b|)U2_ny3;5L5!koTIR;YeWej5#E
zXflX@DP(&I`*c0a()(b;iU$X^AhyTh%(s!b^7@hyP}NKcUP6nG2o#=L{GDvC(6s2
zz~i2Ht(D=Lo*A>gaRqJ^iCGGNSpKJ9Jg@i4EIymhCz3vJDSS2sU%$5682R+4=r}8U
zBWk}y@a5!v?D60$9O@Q5Ae*`Vp}OFD#YAS&?5NNe)NlzO4)Vsqrpqf!c92vWNCkJX
z{93(xMQG|vA-Q|tKt(f+sqnk38ue3Y+L)Axj#lzIeXn0q9S<4h)V&AQvZr7yl-2F5v_!POczFbb;qLEFC
za}++8Z4ILtG=tD&mYX7u?o%<&o|6}s>rnm9tm~F+j0W$lU(PI?H_E#yF$&(dR64iTup;8Cxm+W#F{YlgAJfNx3Lms}ryAMsZ$JOgo2MsBL*G#kxz_1_4$6Ix
zsv7Iklg4P@EbKHEW2Gp(iT7>-x%xViQuDCo)rZ|{sd=8)8(`QM*!gCYppW>p3nF=Y
zx1WRjx)bkRpbV~#-4q}LZ`dW_n&V7m^_zTuM@Wo2X)X>a^PuR^E5U4FPor75MM|^=UR1DcV_mjIC5REb5g#B?
z2EZF#zkU9+&TH5%`)L)M;PhuBr+fk5LJGKOHR_MyPmp2OD05c2nv8a)x5MVJ3zP?=
zAktJtUqawcQz!-M4d5nv(NW6*cIw8Y&
zF6;hEbs5!*9jwo=K6uf&V%aNT0ww>Y;apeUlR44dDhd)N-ugt~$D7KnR5caH>Eqb|
z3GUm7JgWe5K#jj`RupqC;Tp)r^GVM#Bg)VlDfA89I>!ddov=Cg1zpciM6cyk>jcha
z%eqkTOoGrdYF%Jce>?V;Y2tFE*t6%CWt*LLuyKyNbZ6$)vf>^VQ;jd-5AcU=iG8KF
zohvdQz14)f`{2U#jUg1rBavE0!gYoNlTR;(Oc>d*pB7G`zrT+xEsL^-$4p&=me11Y&dPFpAF(Eu}^}hyhSKBNtdY-|OPyd6Ge-}Wzw<2qdX@>up*g+U(
zR`~Xc#%7mj>cZ3npNhHh^gt@lT)ST-nH#>PDeS4CxE5WT_uaD)`!vq=_piA0VikL2
zw*g;Ge$L2wn$_x8kMQ_|P=hYjufLO|O)xGr$Fs<1q(B;UKg{Q$5jJ-4RkicCQgRSJ|U%eVk$2$eHuT6)Z
z#EW@gP-3b}fcut0zL;NBEI5!b80m)tKh?*1><8}Ie`2)Ba+-cvtrEILdagFWR>KF-
z)i@qoE`Wd<_0ZYj;_*o-;A4UnMH7OnX3YO)rZr^m)~sNpnFQ2PZqx;8O#1y2{Ye7d
z0(=_mQ`;}}L?aW6+i{uJk?v;|6(OZ`hIN|Q(mzfPv
z4Qp(V^nPj;5#EiKq{^e3GeT;Vo~64wl!d+xe;KY_eJn^8Ktuy3$dMHd4Ew--TW#q=
zaW?JNMz){7$b1%}>@!9Ew8HX8k>c{LVB(CoI{EUGaA$JsbFI1#f)@(QgOp_=Ay
ze|U$7d>64gY8=oQ?}hXBP{b84)IU@6XDzZVzmHScA5HyQE(hxI%Fx$eR64jOQu+bR
zJve%)$&<5%Idr~yP7=wnt6s~MUiVB?EZsSPU+YvDNGjN>bf
zVb-;)k&4}HrOMg@gpkBnm(|{kuLCK7e@RWgeh3MHt~DN11}w>hHPa4ttC4`3YbLAH
z@mol4F030B?YEyY2wwhaLMP*=SHQ=c-F3F#I_!7BNm=DRlSI3i3l1WetQ&|JC03}q
z4aauUpvLi20)(9a5@>H
z7RTaI>+v`Ta6$8m$;~cvumr^;sxZI)+`}&pB{#f?s+U-){t~R4D&zwW54290rNMZa
zX7#@gw(mW|-M&|u!ArqYiCFIIfAf&lBjkIB+q;aO57JUSRd6?vgU&Xx(fg%oNE8hS
zI*e5EE_Tn4N{40^*Y;~H?QUC52TT#dP#|Ukmr$uv5gmGNXx1=nntK#vz563+Xjn--
zpP`;<{IRDrHGQf72*d-Y23j
z0n<05r&9r0WoN|p=DG7e_hPyUFGo#|Lx?9
z8b&!d>AIuVb2frr;Ct;S6Fo7{bM#JfPVtzlNxxHIM>&nU8jgU^ZHTiSakpv&&H<}m
zOLHBB%XjbCbm~NX=kdw9UzD0W9wT#Xtr#L6p2Ks8!L$I%5%%y2Whp6p0Q;NcbdB9K
zYknB-__$a>>^!xme`HPVw8@W)sDh7=9
z64QncALoUf%;lmpjvw|&^NgusY`z~wzbG9!OMEL7(QybGf4xobODFsT<0LHhEmvf;
zN)OKhHSSNFHJoSA8u;=v+kIqyU)|Zbyc(Lmk-MP3t>5V%v9Q(pP2}q*o{-aFE0+CNG^}e=ND|=s|}RL@I(4w
z%z;nn^~8(Ce=D4SCH|+Hoc4^n=edk#U!+8)NE;FAJ1yGBO@`;J;L|#y&|w>)XRr>r
zl6K~V<5;=9z+G&Hp-;c*nFqa8>ulYn*hs@M(qyV6hLvn2;!pR9eZ@q%w?@%YxK4XF
zg~d39>y>1GVzPg+;-%nE_L}Mrj-87TXrD||64M|{e-KO|WW#|{&cD6dWMb2{vSiDE
z=?yW$kxA0!9k}oI@Zby|ZzRE=7TzW0d0p3^kGah+hs*E@tw8(Pva=FRbp#Qp>UWe3
zGRg^>+BIN=l@s9Gj>&0!w4qnm`5ZFqCGLH#-}0L%`JAjEqy`NWM-W(G1icwY{f6V>
z6JFx2f3FO`OifI4>fR((xX2M;7=BjD{1e%jzF{IrYAY}8@J0_lJ9#n&y3R^v{!6HA
zvD~0l8$@ueVysex+|$Vug!MflDacCbmXM*>1mnp^0ebPU>}FpRYNv`2RrBDtdCk6k
zvRsa%*GyQM{NtFM7rjOfow+LNL1|ol0iK1We>_xMk?y4J(-VCi&sVy_87LdFKNpsy
z_|y$?5xP|oZT==GZ-Q+LD2rFjbmC7G5o?ok-IH{&IeqVZpubb=^(4i0K-5nP1%Sg>_vE^Yg|B4ya;+86D&K)nIQqPk#5
zf7=We&PqmaI`)~;uq^_8I7PhMt1zgsvJQK_ShOKY-+(&Tw~0@%yM`M{Fvm5_t*eVY
zw(a)3=9UYT=<^SXCPgC77Q0)jzkd>R8zvU2#I%e{&iG6WmdHbjazDHHHg9njm@Pt?
zx=ktm)I5c;S~_AT5l1UMA>EsWDY4-pe@@Y<{4ozKiTD?j+bq5(Xr56^J~VwiS7OgiL7uKV`69`H>+V8pK9{8
zFYq^>p#1&R;=k|MH7Ak=;7^3C%GfjQRANvu8CjDkuZDQnsxywZ`BgezEkcLuaJVnQ
zpLT%Kq1$y~Mii*gLF@X;&Z2mX>QmVPjI-i^;DsGI8{7j`i^*}ZUXzP#g&Zw5D+V>A-DtyXU6
zr8t==z#cy+X5jL?p72m@RxiTGFsTZCHT#y*Z5Y0+oHk8QBV2%6RAP=ne@SK=96q7g
zKO(GVQB?!hPM_2UQxF8xK4QkoGy}tk9oO2$JI60Y-z~{VUU9Ro_rqsaUAuWJAs#}9
zvQ8h|1SMwPKc^waE()}gSNn>|-qc#^(n_i@s;5nwQ9!<%9)Rj0@UxR@
z=t#Zu={bH-ahP!!`bEr0wL5&pG6sv;VLcfW`J0T%Exzn|e~jjFtMT1-@)3bf^dCBj
z^nki_c!Yd|7%)yV>z?2Ele$$+pM#LtT#JJVvQOB#^XStP6`Fcs(&Av}{qBcTGgTOJ
z&F24@j95G6+k5LhW7_F)n=%KK5#}vfED*1?O$}PE0p0_7QFo3LD3;-1ry63{9}e4h
zDiM|p+I9u0e^2YpqXpR;G1~gREYJ=~NvRct?Yvv3!Cr;K{fTGl@X?>l>KaRK!qRt5
zonJu1oCa>P_Tycq@I0?PQeeHhJNz~rPw+BIiQM=-Av36s4Mj%92mogDBLJ
zM1E(J2MN*TkIv&I_Y!0pPe&Ji)`fg|cZNUk%sQKwSwqy;Yr&Ec)pwB_VAnglrjnB0
z@DCv-pL~-|`8HT`8SS_jR?fuBljsX6Ey{$3MN^9mM(nz}c$>NLx
zrSE8+n{V&Lj>TmnC^YTS`Z6qcdrw
zTi;7tepCM(*7@C=?SkH)qpP3?&!19g`IB|6e*qSi-c^Xppe5k9Jr#1a+jtrB(9^u+
z3C>K>;esESl26arFRL{qi0^XOdlB{Zrrg6#%Jh=!vS^h>3Zw+$q%y2C2FUe$N&}m>guFAx1hnU$tv~Jv
zf05{w;~fV03<@P!W=jb2XKy0YR9>yAN!=5I@0ZT}?;9>QDX*-X1K%hc@}n`3ckrs&I!Xahxwk!B&y
ztsEB(M4itkVG(Fop%;ZwoqNgbixzFLf2yehxPcblV-eF8bmRi`j>|Ge1SrjE$f|&Z
zz_Mc^0RH?fX_?-tde3^_jxhQ|An~L>4-|^Rk~9`Rf&@=(>o*5*qh!+JxBhi?*?6e?3_fQ}@VPupOC*%>%ui?*>Wy*}LdCQ*82`-|+<2MFTYnn6O}8>XedYJ~
zXMcI$1J<`KM{Fi<6@t^gNjFvd+niS;d>~n2OnuxPmFAu^Ld+q)c*A=BFB=M@;Q3|T
zeCK{baU9qzhDrEbEOQ?>uNC2Oe{=MOp)JRy>oS-a*2)zrd#K&_qtRc!Ur4?^e#YSn
z{tlD2H{xZ^0Ei-<=L2J+D#DtuYXO@!ml!y>S9A-AK7mosj_+Q^$t;#xTV=75pn#w8
zm($z08yGy>F|6fQK3Tp3Z?7n1W;KJ_wWs`Q^)p1x6K?M;h6)TTzyA;Tf2h9@19ZQN
zro_21dz)QmVwQ5!vRMV32i*L#hlkA@L%9hk&FowomHJ1rZcm`ixSjfn{;Q<9(0=*?
z+|N>~IPSMCUcXmO$kyu2EryBC7-rU7?{?0XXYX8u@dzk~(wimesk%B_wbwnj)RA^S66zwRF=e|INuB#kJNxUNJ|yetUX
zC$>2vcqwVEdiRqZv;Ap~M)?N&O|`0fjrn^>`P@+8k4TP*n4btVh*fYQQ$uP;M@8{B
zX{dg!jHFWzjtkc8E3NjrN_LkFS9WN?)CXHb-XbF$xR)#lfzBvT0ce~m8YeR{|9XY?UTe`gIaOjSvS=V?^#(sCKeP_cNY2qqD
zN#todXnWi`hP?2Df4_9gn;wImghO8c=#4NBld$%&;cc7Q{uBiac-go^wfsMmCYTeLtLlh!V#+FO-!nf;MWdp
zgmC}QIy%~SEn5u!)0k1>nV0La99sv(lb=L+LkGOu4jvyEe?g}mqd1b8qR(Y3HNxO^
zB?epPA*Zc;|@qx%C%Q|^ExBh#t>hZ?oEoz8oN|z?`dcRAN;3VF+IJ@5pASr
zD}M6|E2E>L)VkNj6Wea3VazmNRx8=I7?7v;rY{R>f1dBbg|pFeVBE)Irjk{tN7J3(
z$Gp7HJD#=5ce}P=4Hjz1oRF=A#0zXdWgx9NwO%4#$DjQ;jy$DPE56!U6%Cm=9020u
z$julPt0j-)R@fHYO17!=iLD9=^ysA1MO@!{qs_8~26QKsyUN1dqPn@n5F;cJ+{=BH
z(JkvPe<1AOB3BfiAoECY{vqXLNP4yH$L8sk7o}ye=;l1wP+BouQ71UYN1lW0UV3ky
zC_s?J<`TNg8C
z1}ik{UbKPred(VtFbA
z{^Pvc0iOg4*oWF`cS*4%f%w|?0clz}QBA5S!MJgS{Dd9}8yTliUykAFig2mub8_m}
ze_{Q9X-yKOupi!JA}()-XOgu-*3FfDw->DZvXyhQcc~b_nrEbOHq6561Pcz?`3e71
zuQ8bl_Yp_c6waBimRu;s*V`i*LzT^jQlsxjP-V-TGI!D(WZ^{X5X-04f``=DkfPP*
zFyAPcx*Qpltjb1I93{wJq`TUPpLT)kfAlSxV?9lxj6U=s7#st#{la!dsFXoqlWJ&i
zd=bO80X0;eqstvy8fOOExEe&A8Q)}cxz|0k{Y{?q(cPo(8J!NRDz6!$|nq
z-?LV|5kjbd-^sc}zzY1n;Rc5`T<{Q*0ocK6Xu{&@XW&T{ey95OZCS_PAhPpgq2)QK~k{?9s%MK1O+L*uN
zKQ_B6{21|@*puIojt0cqD6dGQcC0D4AkG9Ttd>sdJHToQIp(XYj&wp)iBR5?r&ziF
z@@loH&H;>P2QSHJ<_Sv&?&YVjf4o=i*M)(vjc|7N3-db`*q@Eo3;IyG%tRLc0s}X{
zJow|_IfuB=XV7JSC`@gjGeOq#9qC9h5Y;^1x8GdN)G!=zn!BTIx!B+Y&F+LfWoA(h
z?=H5>k>VL59?GnO2jWKls3}6chtD@qVnA>?@?)Fp(rfbg^3>)*;cc3He;cQf+<+(D
zFeh_pl?~|(T~&29;5@#xviciXmFQkD+Li6*>IqRiriOZwFD4D5yCeO&eoFO()y7EP
zNJFg)b9@AB#}L-2S7YtJsu@O3wQi26gdBnA3w4-c>^9rLwQD^4*5aXa3wy6fM)cA-
zSJRYq(06e#vSUqZ%Z$ole|dPmT2_Ub3u2uIoqL=Af>Lk9skl
zk6=y?oZD|aSrH4_zQ~6X|08xKo#J*9DAKUcMwT9ee%tt3FOc`wBflbv)k@Hu`H
z*~c66y^Be$$HX%j&kH1lNL{HrCSKZu>*Y)-vWP`O@}8b}H{keUNdzB5-0be2nDN;V
zPf`*#X`_ioFX&6xf8`2iIEE8zcWAJ6UR{0Hw$vZ3yA?1e}j&GPXH2
z5hFFJcWdUY{`UBPo}2Ry=~POHaLk1g)Gv}4mQ(Vg8W=EWl60xqspY4sr07gf|2@!E
zH;Nq|UDilrf7|T~mFX?RpvS#E`^?hhuPZc>5QgxiLXN`;+7p6aY==f{xxdxO^Vj8-(eN^;WC
z6mwk+9YK5yeapyQL;9FV8}uj192{3NzhxhLOstabe*@>l`(szvzAsyT08#TOSQ2A&
z`2dZ3jE8Y=@yXuB!ykkB{05CekS7vrmJM)wmgJAB?t&Z?ekJ{HO@y^
z(86b3;GnmG=p)96o1;G89VM-no*XsIXF5-uPxyl1S{15415RA+aW@b4;TI}%js(Sl
zZffoCe+8!($+oh7THME#bhf6&3CI_hPZ_h+%XItjB4f@47tNu?g{O_c{s(RH5?IQ%#WTYb?tUj|cD$5Gc{|S&1jD
z>eNS2jpfVIyQjQ#g?wc&3OYTG)nVDwJmERaR%z^=G%^BRq*q3U1qXi
zZrFJx-HhAl+iwyZE#{$+@*1!*ZjPDzTG83XheE@W05ndU%DvyF!-Ryig++}fpR4rk
zU9e3Fq~4qL{7fOuEF)5!FL4VU0wf1z<`~>8p0$XN>6J7J=HowuV_D667{9euPi<8j
zf9GY);OgUj-8*#qfeKNELY^G)fZb8voQhE~(xU`B+GUHFqRApYa$XG!R==^PA5(8d
zX&7}5C~CyBtO(1{wp;8Y{20s@x?kgR*lIu_!C`gKC=}BJM}7{w(Ioog
zWTaPakiE>SS~u*3zZlkdU9T%Ke^eB~W3HCTlrfAERPG9-y{=wghJ}$>7CXpD=fTWI
zm)De6Rk_39VK$0j(CTmRTKkiYmpO`q_~VZ*AOBvCi#4*aU4s?pNkM*nm3ow~bq_H+
z)OstPah=oN<^?s)m^lRb+Rr$d)B>vKg3XoXU@+pYh|Z8^O32iGpj+zo#VpM{dWK#6XPb!gp0xP{=7e(Iu;#=*jc-VO~
zv-qRowrTs-QKWz;;jqPaQIDepdx{PCkok5O-Zaz4tc@^c+(d`_d>TnN?XgVC~<
z?ksBEpEXy}4&JRl*W~@qe-h>0T3Cnq`NN)g<-2s8N)HMcxAgi5t3i(C?=qQsMoyDuBXa3x|%4X
z>6UDDN7rtEc)o9-fB43=u=Ztl(#paugx=_C3M%TXX(KV&rBBMj6aGedb*g>#SNbx?
zZ~G?9Qd)4s4e@QoblQ1N?e{0jO9WMkQAY~f9|P&#^>a3e)8z|?>*Sv22vWkXF(eG6
z%p3$iN4c*}a(VMO_!&jo0}6ucxBh(BBWMWH(-;Y3(l9A=$^
zv1gdEB9j{eezrTr>yv~f?=9(pOl*9jc4~JE@F61AK8}^eme0Fo7DWW%l<-pMwz|zd
zR?i}^vp_Brz*xIe*}bp6CAX3)#f|EDc${1{a&@z
zmZ!M(=PrIank$mZbaP=AKY;-o*|Wr$EsZnhWY-rPQVcK6+oh`A_o_Ay4GLU#
zG*m@H`R(kt5CI5vb@Z#xH$)9>KQ5~v48NPlu?%oW
zf4^#&Yqn@%Ne9J4x)fW1o565e-R&Yn&KnuZrp5$)h8X>kVF_uD7*WYVd^^xC-jIv#A(wt%7e{1q!
zjg?*c-6v)E+uB+?HK}+=Mi%e3XvFIN5ogsH#uTHP_2Qx+mq991dfpxTLUwBVPntlV
z&-UPgVo1%cHA*t`yY#$f)
zk&HG_y_Wq#B`6$q1KL*Qa*e*dfBw3D_WB2yK~B(qkOZyILX{vCmgl_Q1e=KDn4OCR
zPFFvLIO2h1MH%@WF7Dxc0~vyGPRV)HjUtf9U?fBbe%-7;iL63|oQW{bZa#~^ulewP
zRcD2*ur%+dx#+QD2mB=QShjCa5frn;FjPWAMMWzbt{GUR@!R{&y{j}hf9ISXlWu;8
zx0;w%u|7q7hI&L)*$*cP3Mxu{D4)^rFV;}&5yZo=t3wyLOo}d?+KY3%7vPm+5W#0b
z^Q~a3-S7uv-f@IkCEu4D1U-$73_!~1vGX2q?@%|2c48!*UFd)20^kne_m?VtkV5=B
zuVONQAvE{nvANuZX9~ldf7e87XD*mf=pjmSJy!DhKR*R`Ns_BFq1lBe9%y#Ec<0*%
zrh-k#R;AvJjz2FU+TNi4Kb}P^i=cAGKWr1qBht0?67Kj&RguuqM3eXI7<>d;kdHh;0WQg>>K
zaJ_1ouWkz&*teALx;A@LPFZo}&L*DYvoUBani7kQ2hxC@ZztN6a@D>urZnefa+V>u
z3veL6mD@ybf7hCo4CWh~af|~PWWKvOmOQXDRxH1cZsAjZ+i;;BED+7-P
zt6y2qFyi%0t#HNHc*BIHA25z!sU0gj+e3&H|C^*(e{Cks5CQ@5*F#dKZ0F6JH@SZ9
zujv@>TM7@}N6*aduCOlyixR`ZBD|6UL_avXmA5ZwlQ|Ev?~|Nm#S&$mNV7ZK!n$f+
z(_w7nk2X%V2?oj_Jr%HjjyjWc>;HO-NuRcbT0;$^di4e)?it>>yUMME)cD7;Z^
zCu)!af2sUAQU1x|IXPA^BzbfLWqFt`T@p%*E?IZ>B{T(8=jWLr6N)6bHamhy^GEL~
zVs3NW0|MNa)_OuLrdJvhrQ6K?k;POaug8>HHTbQ=!(47Mzv}e=3kwzW>O?h~kkhza
zo)0F+IML{MEoTb-&&%&?){3T0XsgZvMa#-`e^-(6r$G(cyL0e%c$7y?f&XS#AsKAR
z>E21@ynjG&x^pd7&M97`wuY+FpH4ZaJ;LAv5Zi^XjgWxrWzFkMJ$NrGx9{~fND8Ds
z4VD0Pwg&WNoAt5|AzBOM@<=s^L1#Y#7wr(3z+s>5e+)|}#VNV!G{CS@?*^Cf_D)jrfYKVyu5V6D4{@DG
zS^B*!NRU3Vo~&UC+5ifDZJesYoDSF>oJe}KrxoDuk#_|y=|xGWWtA2E*lj6A@{bXa
zzCV4C7qmv)l1UQYa~>-CK3ngNdz2(Zmt2V{KHTIHEw+ttw%N3zPo=~&@1g>Ze*tWJ
zuC4B9v+lKI^tXH!KJrX~eKVx;b-uNdXz=Rh);3k!=zpTQ{akp1lOf}>+FNFq
z{BH~t+PBd>j}>?2ndQgJ(E4+PyG7i45(JdFZ#^MjYz+7Tg`wJ8EwfUp$k{4{8Avtm
zB}y6cMMU^N{t|{5zXZc|=?UQS=HU6Yl!YcuVN+ygG@_fBpZKf~iV%no`=lRT(UBio-3`bDuDffA>d{=s-3xhM9rC
z9!7K|)^ID`;ElZBPDHBv(P>@Mpkv^)kg!b6KohVkvDv6cCYGHaiHl#FedBNQqc3YIuJlsAL~F)N01j$@OPs`JCin)Y+4{80WSoL
zM_{JH<}L@}yA)Q!e?II#dH4?~0ux^w)b07hGP6->wct8!TsEHYq9Y0O2}&tY?gE*U
z3;P{3vRC$HB>|$-wkqJ*xUG@XyeadCMW)`ZB>u(RErae1$g06tc%m~odqURp3zr`*U1)uJgiC&<5^r?elO)H$W=Mg`(-+%xjgl+fX!-jCqGgI%+d(K
z3GQ|8BR%ucQ_&dpv>1nSR?Wqypo~IlImuM_i2IoTCv%>VnWVLgwLUl}^kpZQaLR|X
zWxX_ey65fre}$tOS<%y=5$Xi%_uT1x7c8eyMCS|YHi|Wrei6|z8%5q+ozq(qJAb_-
z-{XETXc*9H3r_O+sIA&TSr!3?DR^^(HiE2%1;p{s_7{U>-vZ7{@(Ha~^45dSA${`E
z4*dUiCfxaHIV{6$sqBpD{;5
z0w{Z)wW9fk3n0}YyITz3#adO(D~#yuptBKxAI+UF$^^>nfPH_LWjqEGnh*z+^+gVR
z*cI=ELa36W#PWl6ClW=1+xfU2v$DrzD*uI23bzMp!bbk6d`QpzI=9Z6K_{S5xelh59
ze*z3@m01C(g|-oNOE##h$PQRg`Nt)F1rW*3h^YCYEk?Nfr495$I}3{KYGb`R8rb>Z
ztQ673${+4V1Er0rM{~Ryr8fhw2cZ48$@|BIp~>J-VM1`7sh1j8{Nt78kI0cSn1Z5X
zm33!JDo4O*rBfP$8Dal_xy{Jvb|{mLe{Bth^MNyvyX};|&?Fu?r*U+Dd8eJ5{u!1<
zS%HFrE+r0_^f&@G;TrUW;{9jA@Adf0jxer+hUxl~dH|CWdU0Q<%h|BZjZd`;jg_4<
zouXk;TtZb|_gVuuU*Bwbpp0;Vly&Lkr8}n0PwMT@f@aqw$KlD6N`%rfI>(%>!|1av1#L2AsA;Zd^
zv-!*+0Xz!*LOCXzRuX}Zp5!JC66pr9l~ng(rSx45Gg^YI6DEwLUNt>mPBVBFPteJiA4k8~ZVG@L91wU9fB2XjwS-~(
z-L!E@W$G5NLM{AP7;`@EfAI<~RHz6lnI*VUsHl9&{f9|eMkG|V)H%dfyH)#vwze}q
zEMKDcN*OhiBQj7Ow3k)4*gyAU*jCFIQGNMcjb2gNl)BD2tdK4s1Rg5UBh`WOv7Z6w
z8{8<+dCm$x4mF2=liuD8d{RA?DpsLaB52k4voW95hNZ&f9>3tA5AU;a_GBz5@u{*5O4LlNt=FTLBncY&d)+<2vP-@+t$SB4-$P_J>woPFtJfgHzfl?|)@QDR
zN|usVm;h!0fG=00fAgq}eegt|fmsI`xv3~lgb)eS$Qlk&g|q#6_g>NV3BOE
zVWC`jonxo|e>f8>ei(aChg{q(2oNA^zFId!Vj;(AG{0w_HA#ur^fPqfS`tFtXX-t!
zhW)fooN`R&h|G+eepaMwhZQQ1k=#~C@Yuq3tFbQEhN(Pbf3Gn4J+ht*W*c(kWr@rP
z>WA*n-!`}zuyY2&=p((-?mZxew~wigLiqCfe_;SFqHvM7mZ>O89jiOIT60TSJ3cOX
zV4SQ@d2jK!nag;|LMNlFYHxVti-D-h5t4Cv;)F-*LJbnl_d)z8f+c?-
zSJBhEcf6=^qCpu&d*#b)HtR|qMrb}!TPKw`b;N1Hbu`l|fw;e|h!P_r3r|2oay*sH
zsf64*E{G?yX#H0FsNfrq>?4HYYnXsGe+XMzrTLMVf1{h`D)-O2vUj+qY=jf7<;g8<
zhPLVrS<%vToeB^ZPyytdw%op@7RiQm@nNTWX)Gj>1n_*Z{&i?2-FifMOzWCHt4P%#
zHq0{y#ipn7k`rgrNsXh@6s{*<;?*=Jzz*N?1BCF7=Z{c_c)i~jbt6sPLS;fD!z;c0
z+o8u~e-}nZt$gw%l`O%)lIj&2Z#QCB7Abd9N2aW0-uuY2@;T#$T2c}u=4pbEybT9J
zqZCPf8$zA3%N?gFF6nn$XLhCJsQR!^jSsuZ5vnz#qlqxF0G?yS*6zx<=xM+m8;GRh
zP0|P6n`;)Q)#uGd4Z#QiK=sna#k~Yk+}hz6f5f!N&R1b_;73%k3eOAl@osw#S6!@n
z10;~1^Y1E5a*N1HJ}0U%>xQyhZ%1akU~Y#q6&*j?3q2B#q;ynnH~ndt7-W6ZRYXvq
zyVzXjRjiI4c}5@-Tbq14rCNJr`<=5vC7vWjF>>XtEd{(?GO~Bnt!fmXXN7Yl$xDCb
ze_?A?u@rSB;}mvCvPDGH(9(}MT<}hDHt$XdOhcvqB#Lk1nrGbf;miEy111V5e}C3Yh0kh(ap>QNn_>w4AP6?}@~N4EW;%$a4ouelhpPJH*;7jvO57e{cZy
zJf*q@ve|5>J-EEn=W2#5Gob!UqsB+S@}*S5Dv@*L2k}31|0po
zG7^cP{;A&v|XNxUh!qQBE03v+n^e5qu?9NV~bYif^HVwoo03;|v)FNGTQSI+Cw
z1rPu@5GV3h6PN2O0K&-R&EWdzdX*~45`J*e3_1kvmurbQ{iZ<6FmaRFe;R#sr){o$
zFF2ogaJjS~4RRM-$|0!M_9b{xUdN6YutoF$)k^>kLk2MYmbC2H<&7$!siy)1;9n;2
zv-318X?!!^5+9F2<%BgBj>m*V5r5}2sabUh0Q4VV!P7#?Ln7`$k;=GW?9Z|WzQ1`6
z?tX-0|M#$1^p+@Z7lM)(EaMJyHbf(jW?7${%d{U
zoz&9z&M7Gji
zXZ*fnCRq)j`_Rm~e>#N@e(8BHbFz|~q(eW`KYg&6UkW@fE+CvZsLi^rs2YNhdrIg6
zN1rlM72AK@by_HB1v<4>J(%VI6qC6;D|`$wI_yxWpLic1754TtsgXQ9T%gYfY>N^6
zS(6{2Ev2EIwswIQIq|D54zJ;XIk@I{UsU{UMkQ_|*hxWqe`qZ?+gGVOA&X5iEke~S
zI5CrRoVG5&wA1RBZ4DJvro3z(o63%OY^R%s&HO{&c#$Q?&M?wHhWvE(3MK6(hFlPf
z38=9humk8YZ9I#O1zr!xtE7jbMEc^>mlL^#-)y6Atuj<
ztsRNv$SKWjIM0*np~ZSTclVbL3OYo0C_vycxgX^N`XL8wtF&fLioAxjZ{;OV9>FL!He+IZxvY*U9dfo8n}>t_9e=;2>BZ0U;kNH~8MUGNG~0UgTx
z+JD>!e|GX&zpmz$O|1F
zR8`Oj4`}3$(Ai-#!e|zb?9t;+
z^`o-iw>xbks=J@WV(9tX!Uyf@r|p)GH)6!iagwuvau!=X6aq;0G|LIepP~)anOb5i
zuV3U%2@gF$QKo(EcqY8e&U)!mz{_cIh@wA0VYez}b&y3@Ds!OH_7J3N4;CReIR!<8
zeITPYtNkHP=Fu
zYR`%>cV5%-{m?XP%XX;a<(Nw#tc<$9e*`LO6*y8GgU^{;I&}sqkQYgIf7Wo~41baG{3Z3<;>WN%_>3NtVORjL}mjsFg2GkUjq~pI50N~FHB`_XLM*XATcsCG%=TuZ37hqGc_?a
zm(ef=D1UinR9ow|Hd35YC@zKIP>Q>|OL3`oQ;0E03Ic{QvNEyo0TdKfA+|s;D+{A66lh~<
z0$^ieVc|ffrWSVq0il)9ul{{T=@f3g3m
z^Wyj?Tv1Y8R9#0|l9l<-Jpfn%&L9Uz%K%Y8uD?HJ{|g(<3o^jJp}mBfIY4axDgdCf
zfI{u~n3-K%T$s$A9HC4Q2XiJno4@d>TUa^*Tp)iA)_|9*1IPyS7cov?(-%6S7NEZq
z{5ceWf~5%v>6~l$8|#
zwm?fT6a)r>O_;*GSz|7JH^as7;pEI)r|D{tDRg#sGR8wbE
zcxgN^qax&m9WWEr75bO=pM0Vc3VZ+_7A^oQF9(3-rAZ~hrs5D=+ZWW1sDJ2_uzcYO
z3UP2_{y%+f4TiXYJ^ruW%o1#B_J?^>Cp&*;4X~xX6G&F#f4pB*sQ=cPgP;Hw0LUHy
zay7AF{!{L+Zuz5T{iA-N!PCPIVh1n-+BkwdEzLkL7gP^Npfd;nb#MZCdi>MypAsr7
zH^9`=1p3nGFT;ZRS9V#j83e%lxB3OizfAwr0(5^3D*elpnnJ)fZU9q|87i|91p0r{
z5On{4=Ip;}NjcfrC;@FjbpK81fBOM#Ep6QX)APSvv_OBr(kVe4Y=JiaWwUgYvUCNR
zDqBKLEdFNpZ~fo3M8W1ZAOIsP2NMfB7u(+!jXz^$^U}XBJIM0S3j1T{`Y+o{YnoVt
zL5_|94wk=6pcgv+3;4?!{DBQ%)>40vRaO#X_&>e#mzyNm1Y&9lHV3eAashx24nQ|l
zmY0TL6~{|5>4b$!bssh&4#d()8u=@edb8Ak@LqRgdMRgIQnn
zFRy=nGx(1J)c+p7e?*IkL0ml;IXHO$jBLCwRI~HEECg0IPS5{{HTip+{WU!=XZtVt
z&nf|cK&~JY)P-4y3BSKpQe%H$iI-&FL@5F_FVkT;o{+X&%&UdeiFRTFiR?`ZkZ`_F
zvv(RzK14x=&%i6e2VAI4?T>5owAvD1F@9;PBJvaHrRYVBAt^eXrOBk>o2EGLUD`=O
zzbBWKt5d`=mDU>5N(#^z-WTU>@4Ag(8+*n&Po&T9+Rkp
zcz&V-7W#aP6BQ2Z6{DTJk_NMw25l2|#(#$7&2eLuH=K{d1GO=P{;n-k!+jby8;W^gk
zmuv;2)EUAU!@+yH0c(F!pfzNsRnPg025VPDl%9v5$U^H-UY6$mYwnJ;?=5HE{hK9(
zVS>l7@hGCIGKX$|O76~y@!C-$fpr9C#+0h49DjjofhP(R5bZlZtnJ_kENnX89^>5`(
zA0jKic#j!>bVQs_wfD?g-kERFlKDl&*ps)CI1-1aD~cK_QDnn|;1loeZ;93`nk@}r
z*v+n#e_cI;b|!x?(?^Dyzxy(Lkm~a#K|Hv6gzq8V8SGsxoMy#FoV-|L?UFBugb2API!Lye>WV(O+lBo7E`>C~bwKv-lF}8+A
zwYMz`oM=Q$#11l#D
z4KI@z8_{*PG*NiDLM=K%PuluhfklH=j=Oyn_TW3a3mxfOn#im9Oqzf<^55Iv*K`G<
zOcyGWh_`>$1U+!quNk3X(OhNFKzOw-y0|%HCKagKP@fkVoJTuJG3y@Som7vYuk?|F
z-r05cGss#Lb-d0H?nX6SY(J&X*QaE~=E?0s?9zjN!uo=`mq}H%Bm3s0@1Bh?
zaeXKzEqbM`9VsL!yS^en{B>D%qILF6*?{!cqC$#EnA7=7b9oBcD
z-gBd0Hn7`)5e#Y@ffc%axa}A{`tp9lk-IV#^m<>`28NJuw&aY4l+{0fqy?F&a*Gy;
zmpp%YipPCbb%T{(b?zlc91^#hpwK9?NcEN1l()=~Q3?9c4m-9odnl#q6w}orA(o9}
z8x@yl5SmK{(jg5m$0}BQ5=JrMf3t`vPOhW5_rB=oY+{S2O$3owmO|}Vuy1f?INiO^
zXz_=zGwsBp8b7H~GB58VmLF3%oDgQorOJO6O+?Jy_Mmt(Q=Ktnp(C4ev*)`v_z8|g
zmVp?&0_kmIF<8Yh%!9By6mxe%!EjW(uxIN$R-HYGCk|;1T_W
z@;!Jod`%KBG38{lDZGQ@TGs;iL1rQ?j}dc|XROn2d&i-koc95%3i7HLYq(Vp*=m1g
z-YVM&dluA=;j)Rt>?+Sk9OMtiR2<4@dav*U<86N#=NMP&KclbYJM&eYTawe%->n&ZCuh-OtB~W
zugXQ1#gACW`?opTCeqogN;WTR~TRc6a?JyD@S3o?D++J;z99uhug~!z}t~t1H|q
z2Yx21NMSt-X^SztJgq6+gF4KvNVSbUKhaQ3F*rTJDicMzdD{;$YZ_#vFue-5-tWCb+HZc7eR$U7WP3pu3P
zEZFEY4|r#?b0qW)1cm2kgM5du%dvMM0989;nPxKW>*g=Z5SlakT7kT2$xVo`y>G{2
z{xri5;NePlALDZ;<+y)s>}Q<-|`o>M!x0*{olIY5qaW!5ssm^qj?F|lPk2*}z
zWI*%FhF5;BB=6TMvX9c$7IH9snGzy?!U>wE3BgKh)^zy6C}7{PE%}Wp_EhV@Fd*%i
zaXqC}Q|ZlFGDF<)A~p?Q
z0Nbu>X5bAqQ{q|1rF0d13NMM@Bl0MWsj5(AeMZ#_iV;aItZtmoO%Q3yHaHr2OfF#U
z{IV%p6jl5otCfGgmU!HadMq`DLV4v(ECZWjpWGWko%x7D!?0l8p$5F8W^n|0dpW+}
zRI(TA4XM>IkbVS4!Nj~VuG1!%1M^o~Vbkl`zmDwmg<(h6j*>rz6*n7rW6tjFWg6$stbloi?Iu{L~z<)Pt^e4X>qPZ@t;W7w9w%Njo9VyY$Hf
z6q%RZ+O^pLf+P51#a658UdL-e8joWgzvB&Yr+i;
zyU>LH8mvItsTrm?*P)TC!Wd$}mucxwS%0_gmM|B!kgnxl+cY?Uve&{tLeJU11rtwA
zb4DFKpUc}^ZugUr=KXa({xvC@b|df^4>0LOdxFsmRC-`_*^;dPMc}(DPA&4;JG)RA{a7U@0r#DR~Bo$w)n`(Kk^#ixdk8AZS
zh+j30Z*!=4DmdV5cCPKTB&ygyQ(WM0BwSIArMB-itj+Fm*8JeMU*j#E-cEmxy>NV#2SC7v-8qn)2El?zC7UZwAFI5+(+S?ul$B8tnhLF~=IS&VV+vBfLRD
z`q@aFF{SzthTCB4erz~mUqYGpydh%#!Bo--tRy}ZTKJvgNM@=8r$D(MJw!#hL%)Aj
zhjyHMCccG1QV;=MrTvthZnFwX1;Bq<LRrLQB>5cV>@K;mxay{wFI>|^_h{CUKJLJGEV0#x3N%}dcqeLrunF;Bu9
zlLpx3mVMUyMSWoc^{6qx%SNXB8Gp~YZk4}BuK@1)tX*C*3mp8VV-znI^3z1YC>@-xBV;&Tm9+k9f5L+bJlAB2xX>~yDcouMur
zy7w~^7-&9_nV-^xOoV?d(YTDMVLf}M9pl#MZ{k^E#|y+(|G=AY!L*2oaj4rl&E9{p%WfUHQ$Cr02j(#lUN-CJ$ARSRsH8bcw(BctCYNkI^ua&d5KjK<^M$
zCfdn*g`pZ09aVp7a#MoZV_kvwgJgU-D|yHCJqf=8SI&NemtcG%ks6I7;kAk$*GcAE
zE+?}zQKyV^NH^Dk!MQm$e}ye1G*j#yL%|mneKBY<#usy44^PD-4YFel3>HZbEK$v?MaGJ$mT0qrPHBO9`b*slD>&)qI3lmDoa8gcTqae
zL!Z8UV**^ZMfg6XR_TNXkDccjSYx+n#`a2iEza6r)T$$YgmIA!PRru;BPY2v^-pp0
zBkI_cE4%4J`Zkbqs`8tFEO#-oafrr8C^gEOY9v0-Ywn4AZi0nI*@i9_EeZ?qHYi2b
zpp6fqTYi657D0F_&U;&?vX)sX2wx-ST~uX}5a)p_=N$=z$W7t2rX>=mW@Jk+riS}C
zI?;Q}q4BVEVkeyQ^Ajg%@s#zLdOptF_7|MNbxqTaxSn$On?`;JSkrAv93a`=@xT(4
zg=U{mIu+3apKgr?%Si(=BkQoDy^FH>C|nzQ!T5hLmlo!y^n3liPgE2?GN*kOAd91<
zNnef`Z!{vxsC)1=^5ke-1q*_~I*&KEJT|83V`xR_TY0&&Wo
z0YGCNY>4kCw%6{X5pWc$U
zGNN`qGK{qe&Hf7HKsBmb^|{thtLq_)#6g^>A#;%&7U8v!1E9aB?~~i6)*DQ7ti);Y
zF6z^=AbWfjRE5G%!z+10u^l+jR{MBw--~~IH}>hCp9ecQW&_E`C{7M(l8-;#YSZ5;
z8MjRMfP2RWaR`+|)KRxQvp)TsT2-UgUbMdY_r^GbRNKiX@*EY*FyE}J50e8vWPw^y
z62Cf|Zj+pR*_uv%xnzFx9W2;%jv#+iNFhriPazZBN?%ZW%oGpfs$B(9r6ki`xgALV<6Q8#^SkpabW@`3k?vN0*H#&H7e0m%=PRkEm&vpn*?
z`1pZnZKi^g7~zMGwqcuxu8o!41y_HBykQ|n=ESAJGbHN2I?9bVbwl&caN(>L)79V*
z9a}0Cy5IILbkMu27;XkjutUOn>Jcwi8WaOF#K;n-70r5|W^H~;)JHdPYZ8FyGq9ZO
zV`^dM`I|QN!cGNV1M|HI#>+-a5RaHhBJfmpmK
z!sBnt2QKDeFR8`qe6+=ntdGCq;j~GUwU3=pl$AcS#HOiGN(RXNkeFlS5?(4(>18fKl1A|eW|2N0EAa%~5^7(&EJJ^^O0edLM
zk%P=-Bkk$ks0|4!oUhmQzGGyr;pF5NX_3`DKSQsi@o2@IxASAPs!fJF4WVnrcadIN
zyp#vY5tN~QJ(-Mpi`s+LN@rA4f8QImJIeos*
z6T8t4F#Svr+6}aD(7}J8yyKmTWTZvtcAN}JPfXUiJp)_xcD5t=Q}hWN_BVtq#nuvW
zg*d9u5sB?`m^Y2Gqn5lsM*2Jt^U*WRaeDB>;MFYDb79}^9^vjrhbHk0*GbDQml8q7
zsJ!S%R@Y)BM-anc^yzMb4$fEW4DO9_^Gi8O3Q?kZd_s30gQ$OOw)BeD%g3{WN!n9p
zH_+QUmc%lTxgbyY%&MbUVCGdQh&ip_^bCxv9yMm$cPwl?
z_!FYJwwN5Vk>!7*CXlM>uY%!c3?(louAS?z-TW1YB=x>8>ZSPJv#wd?#A|&!8U}4VXMEVSqt`R)6|7|`QQG8P{zQ8|=!ylE
zqKywK7CBC?tZf#7e5}&_P=xZDozs*ck!#xO#Pjx;sw;ocVVHK}v;6hJN1}iKb@mEaywV}!550LOL9TAFkkMd1)2C@b!sW=HG%2FkFB!Nm5GYkpLMp`{
zUXuG8%UQ}1msdb|k_Erg7!Tq4G3#EN{j2T=f?d{JL^7-*Zx5-t$_GAY5=CRly1u!@
zI&JOJpDDTZ{O2H{BM0cb7~xOx5xCIfjzjlt>E3^C>IhHs0Ip||SH9N$Y6`E>bFZ`-
zNdQnBIbWaKN8|fX#X@yY+g*FEIlfxP9y;t#do9bhEmJrNGOvZ7=rX|$A650{+C_IO
z@NlX0-;~=WWY?A3pxS41%-bkVrZ%<1xbb$aqwD;saE~u#?i<2223~3
z7REg7^K;nOeDQH$kv98)H{yVvQnc~%WFbf@L5&|GO3!w0gg{2q*JEXK@9)QQjIB*k
zGvo?b&U6SgpG0iFm=+RfMjCz0h04!`>_uhAAiGbugr9A
z@5AYJ6Bb!B5HN4T8{&?Y_j{-@y+@zxpbNt$vPI5&hmzbzZ{1m%oBf77q?smn#PTG(e1^v
zkkxz>A=`+apKFgRvS&Q3(Mburf!oD18M-S)CItpK<6_bGq88NDq&>B&mR5JPt*fP@
zEURv~_3b6DmpX;G4G>;wPfSitUjhKYccDeS`j5}Un0nOI@%c=LFc
zlwHraneb!sNXQ50{9V~UmVyTj5?Wl1jGg$o*hCT*8yUOXIk
z*XCMW)pf0Y4q<7$$yX*Mf3cmyHnE0oV)f92&c6MVFHT?0>~|t<#%Kehs!EFu$7kPH
zcDK2Lj*@vH-2E_-+{4h=Ja~rQtn>%-k2Y&>ryjRjd=8R{st?F9L`Q#>i;tfIr^K5W
zgzVlVZAg+R7G}p=Ak3z=Z)9HA;(%(~U(w?+SX0;x>M0&q?T)NLYaZEN(Rixy6ftMW
zr=lIH>4wAe-yMl|+FjuXRYb~qD|J=bvdp!1AE9iB+??}|Gbzn1ASmO3|iCUX$7db_Cjq?5UK^M&97;LTn
zMS&{?B+|A#1b^q1j-cEMg`N*Gdszsnv@0TLRu`A-ZMd2ti6MW;MImPSkpw6GOeUOT
z=BUh?$}`>8D}8kS@@?^H)
zMAF^q!qdlsISYS&8u94N_^*_Er&B$WcmzTGGvup({vd
z(oW3je-#lVbHwxU0#A6Ygd)=_n}CEFM&sVy?`0`$_AiA9B+zGis5RT(Q-CnN;x+Qu
zdAKFi-(aB&qdzEj9$R@8JM7)~)tWsqeopH+phPOSUFUxeTj0EgY2Ru*hotJ{Dxd%>
z1gQ%}V=;_Z3`1k;Y2C?Wzj{%h4@I;~K0VKUU$cJ`yt3uXWzeW1eRVlGua=C&%M+9o
zW%u>wkT79yE?D}<;x+-$bQV)~%|G|f_bL^yfOOgO?AwN|l)K(@zjs3Y`-6Gb*M%rc
zPqa>7HY0zQNl~NOxdJ;=jNgnx7;gk)`Q*
zJ2K%ixt~7qscc~+S@JjYeF|gJP>5s$>i-&2&BaYCe#fk$FGZ1|e6xfM&_Yfg)8ri!
zG_fP^XaiffR&Sw}Yq#%=5L^U3$#~e}gnoKB?)!gKfj|ukG0z}N_@Z=Gi1=W5`H3Mx
z-ZnO0FEZqP*nH(PO;wyDC++vwBtI#e{5zv`@lnGn<~;)l08A`bQ9Q5T9<|09vTn#w
z3_Dz?&i0k@!j(qR6w4krp*G>ajw#Fckz54_Q&ft=F7_#_jmxKPAEjR{h1wRZYtf!(dRI|@ovS0Eu>($Q#YL=8JLS*-y5wN9K3QZ{au)rN<*%RY$
z7hI!=(wKwjs7g1cr<1%8$(7R)Ujc--jRok0Bd&IFls0(qc#8~hy@iX%BN%C6Wj1En
z>f2tm8~~0eKwCIG_(t((vL03eGmzi3#N>aox@s^-l?Y<~;xXk@RVA%(<{s{s>5A)Z
zxbIP5Z(89zZ*9(2gAyX+aa@z6;i1!cg@|%eH1VlQ%qb
zN}Fwktc4`{eROf^lj5U9nah(-ikdTPK@{wSWq{SQ<@0-v&JJe%Y*J$Uy)uO`^S1T_xnwQx51)BMWV2Jchmb6JJwOik%1=rzhpTO#Kjg39aFy|2nWd~Balg`0hj*`f{*V@neC(PI#J1ki)(-5@|exx)}Q&S3dQ_1xWjW^jre&!FNmJ#^OIoG
zeQpXmxylTzFM3ayg*n)SHfkvDajC;kJo0cG!*2pG_1a=UtxCUeC*4J9roPdD=Pf@`o|wym
z)kgc2IC1;K5X@huN)-hDK~jJ3Uk{z+=Ax+|BgkzS(iE}Cr6x5TxOd+n-%JUII$cjn
z$(?m_oB6bPD5l)>C6_h6aUc08Wap(yPXL_}t&0&s^;u$co_(E4b7ran<}4mZKqc%m
zU~zIV@rgf}hwB=j^XYF0doE}s0Nh)#yOaAUmOrcH2@)9(<(}ly6a#;Ch^lKJy)+S>
z_^PmYdOqrbB8SZN?iuY+;MCRB;;9I~x@R^Y2$Ja`P}TV~QUXn(8-{EwX9<;F5JkXH
z%kO0{!FUxhJU)fn`FGXtx(^{pJ#b0*-ajtGpd+#+v0}4)JJeJRhcn0&gfL1GMR>2a
z<2c}cb{BuAIr!D>3le|r)iMdf+6-qGUg^Ak;k2+g?^@_Yi)Li(^`pZ?mdJbVSepS<
zgq_v}YMuj;x8dbopBU(U#IIXx#E=KS
zF8P*qc$gS%(YNohwAHm;3RYe(y5F4fF`Z{Vb*)CxA$;4yVqSkoYlMebIPR9=#b
z=)S8dY=YBx1-#2M=ecyxKKgE?glhtpL~W-|P*3^Dz1nJ&&R9?9M`9{zp2|1A;W>65-l7b>63NT3NL{)sxCRb9W<~EC}R4D%7a8=Imi<6E|zF&X8c&3u7U`MvRukr2kjmc2k9)H)$
z>TWFad`mX-VYTS)7KlJ4v?fuH@WBzA^jcBtBBSWn5URy2!}Wumx<&0XLlgcng9{7B
zo!b?BICD6GQAUv6AfRc+Nu(vb(mWp{y?5N)c!2vU2
z`jdasofofNCSf?Gr@^laNXfNOo-NrJ#lFSmp$_7h{Y)N+`i-Oz8Te*{tz4?0RB@U;
zu)|4F<$h)~A@n|OlU=~j3;v%e6N5G
zi@y8&4Bq#-St)xSMC=={u#Lg!6nU`9ayEbbQ1F-3zz=ca;kYIUr8f<1rp_T}=h$Xi
z-P9FSW(w)vKSvimmM7SpN7OgIJ7ywed9-_t-3@=R;M_m-hd4$mL^i_U-)0q`ei~e
zSz6bpYu5P8uvYohz3_@hI)#`6KkaTw^;r_BRU^gR-dC3P)fJ48BuP+nniu)O8*{&Z
zwH>il;vM2v`*apf086Wmoc1I2H*$ue-~V&KtRAqNb8nGf~nW0_g6Q
z-MVOWwwjrRu&=1wm(!eDqA}rn`ge&+I2O92L3-kp`=+MlflGZZLQR$FB9VXTzYaq<
zOgRCEwWEIZN`(Ouf}S-lva?%7UBNKT!HoL6+av=97{E!@`HcX-FDq&nHT6)Nc5^#rcs*`yf0-_;TB=pgGNZ4i`q@N
zksk%|_q)<@Yvf?=L?}Kv_tbXcI@eUK-Bz(+s*Es{KLsMrZa%m5On5W>yMx5+mtFY
zwB~{p1M`cwmxn(7zym$YS++PI8;&b}Xu+KpeYzb2-)C&_`c&d^825oS#pKi1NtVb+
zrl{Yo@VMT7TMoV*5$k{O)gWRT=_mA6A`0zO{nlR8_3ytizTRNtP&}{dwe$UMS3W(q
zI%{E5whYbRX*77#x#hWu4YMBD*z{h`#;Ji+DFxg8kwPX%-JOS+CzNfep5%DqQ2Xij
z{p)Hsc3ItokLD21JK5jFg|*~&^^aGtPb0HB`HT8FD5pLt(NuqB>bW(B==Ke7I%4W?
zbB8i5V4o=hdL1T3jMN00kK!bS>e)aseu8+{kGs$q|4Scn-QAKL3_c}>P%;hw15~e=
z{D+qIsR|iOCCAmbeyr#A1}K7d4(4tmFv^GoWn+eKOwNB7Vidxot9qZDmc>+l1bFz4
zr*htu6(=cU62^bM2hewBK;IFFAc;K~I#1P@#yEx=SP{?=N_?09c^GIaQ@xt*!2bVdWnk5
z%O1=D(!fpK>&Vrq9WGlGJG4uDhXTg!61fYN^Y8@%+?{3vlIQyeG#GJV8@1Pj$WDSVuXR3jL57?H2m5>JO8*3WcNaU`~z
zxYVSasPZB&2sykF3hnIsnh@_8=D^tMOtHD8^clnrh0C;yi_8DL^3D59Gn0?*t577*
z?E6?JHJ3?RsaKsqOXGuyF=MBVy!Pwm(qbChw)R!amxW!x_Pp$#LYtU2eIczwX!O&g;(vNn`jKUXIY_$i$B@wrEHYb8rnV05=~%*8g2xZ#o6KB
z=Dz0I-0hK-5@|oB|9FncOVgp~YmE>B!_$9;i9yL$O``zx^q~DF%jf|a5Ddqh@toP5
zd6HDCD1hqj9+QQR#+=|U>E5~C+;`34ay~V^$^!69-Ts7mQ$c|>(j1+c9&t!lc;_uv
z54^YG)AZSO8~&7qyVQL(?LxZUn78}uxjQ3jckA6;zq;ly7nzXaL+tNOI*0M#Pn&<6
zgI0~7J!*~OJo;(Aan54S<-#w(PqWgr=2m{1G~B!^_3Ha|m4f(40Qf
z(;&v=`5^$Cb3m?N#Im}-b7YW3_cY&)^HnV+x3$slNva(?!JUAE*Ia1n
z6AR4}G^ZOU9VcD^h;WTQpV}`rQu%+TQ9kOU3|l^tdYsj6u3`5#Dx^KCz`^vc3umjJ
z(6I|AUwNH~Y(o;j$#Hz)7=mr@U=ryfSz<_Ts@`Ge;=LQho0rS@B_y4%!v71}EwJMp
zm7ZHQ>~p3*UiW%+A(B1!RX^xfD2ecJ1Yq9sJi!&)qE$E`@Nw}r3-_i6Z^?gziTgt-
z#hi4FQLqMUj=FzW`I?AW#gSUrEHj2?BkMOFp&Mr6^W^rh#xKSXKiMuvpY(ng*c=KAzxZtX9$m#Om
ze`aaQqeGhZ+dV3QxevVCDfxe2cj#tak32q+7JYByJLr?Cu^0GNg#17QCKNa+#+#2q
zSF|zcBaZC}GThQEO64);mVEFSbggqX5F?4vetksXhq-ExnIFu%I+ym-2RUqzkVwjA
zO}pux{2<^OcPdC=k`!^JuE;ewBGI~(#@)%YqD}wFPjxaJ2$NXg-V$pv
zkIQ|JwbG}4pZUF$p!yMY?){Fi_pg}Co@ID^F*$t!nHjZ)IZjst(M~NLV-ND6%(&KZ
z$;td=NvBq1-=Ao_hMhUfww@2(tEP|$^6kYf6Ecd}qv46nHzle`Y`zlMMdXB2G76bl
zR9qfUCIWQ1XJXA|7VLi=q{bq3zFQ{Owlp?q1XSVa5^>fxX-xIhsc@;iV>zSy@*+!>yiQNa5|4b&vgOrB8!6=ee?B-qUkl&LcK^s@=Gr8k_v|%)J1NOxm><
zcdppL`XlPk2{-&OK^8X?#|PN)N?2u6K5xnMtNy2gImy(4_;&u{Y;F;(`5rF%L#I&{
zwz$pbj5`KT>&SnPM+FR_hT~zdi0QS`tCTe#OnY0z)o6v{{8082?CVfXv|^6tYm0m>
z-^1*E?|lZht+fQtuy+9{dFHQIW2+O)*B#&N>DDGW$Qqk41qHc+NdQ-4b`V726y!OY
z4#EnMo?C!o4^jMq+sh;Ulo)TLM!BVNWP>@iOX>A8yT
zby`xxD4Jm0U?;8!RN2e@`P~Q_=B)&U{EzCkgYoOA*t9A_=F&Y0H0{Ewu?z{H*^l-%
z)>y}WvTD88#>iEzN&0plJ(zw`To4``(#3s_YUb!!XrE)vLYNaFOgH+GyrL2MB%8R~
zfu>wcXxm3dgQQrWyjjtk?YxxT2!SHF!^k88S%i^!ZQ)&`h~E+$5W(Bv+QcGS8pE;t
zKkJ=Z0t#hrWOHC3fZN)
zmPDa)Q7V!oloneQS6pkv#YBz2fcI)B0{Dmz&q0j5F>3jz#o129g8
zG1&<*Ls?XcKg{)^heB~ZajnhKfGxo<5DI1oQUL;k1lXb-(10Tpj0mZKI>Z2opg(~|
z1|Twk2Y)>QcQ;E{H^9o($=%sa1C2O#<1m>J>l>GyZg_VqEx^Le5pM~AyR`r-cQ-uu
z9}hAR{uC|15s%>9IEW(G+`$rWhWB!|#AtKh0l)wUKo*B
z@&NW!Kajx&xkkI7Wsr`f5-~#L@V~^6M8I61v}JXG4T8Uf@h7mCZ0(($?EyN0%78%z
zf#HWJ!UUMZ27H%f$OV$rmI;EuP7aI3jp6VeV*MNDJ9Tpic{3kccmyH%_lgl19Cp}G
zxqtcfwf!Ilo63gS%N{|1Or?Qb_H1s=sEj3r5Gz#szdAgS0R=O{{}U!t86+~d
z4kQj!+nqrT;(#_5-;5BE@DCXUgaI7@41WT^5I=uyZt$hv;fgU_F_M&sa3;hA$OIZ2
zjG&T1CL90(n8g7j!hZ&SJqcqB0TR^@M%o@39pNSKHViTZ;FiS*lkd>ChEUf<
zBNL^8j4cUb&_V$cNEX(1gkYpS)c=2`?zdRGI5e6gfexzwM`nLlC(x<1(ErQeH-AMB
zkjq@%5n|B^wBIl)dlxkXBso)IKmX-4FUxIU0x}n73!=706mlC~wo4YD20%Cl>3@C_7_!twfHs-|!H5aKP9A)t+H
zD(-OoP&70Ev?<(=1IPk{I0V{H2ttxhMV7!14;Vz5&Y{6nCX%;*6}KXaOn;C?g-G0u
zhWH`<6cZuFWI^0rjuhvc`(IP<%-v2)qo(tt7T?zP(%B7WL4lwLm4tk>eo{f|!=i?G
z>mYN2LFCBq`_tZE0;v2L@t>^C&7qKRlr9cH;jl>UFc>UgXlM}ei>2T4{#=?+q-DOJ
zxyuUx!4S|-xaS$VJA{airz>wi1379Qw^=X%kP|6uzD_Z8dT-7I{NTplQLO
zo6%=g3m|*zZF?g#4lyoxs>DdqzV=t8Uum5rx$GDwL^?#uiCLOG%H552KYZ4qJG!J+
zN#m7mZobz={f@KMDb2970le37Y-HyA3u^k?f`
zAraHEWLMV)+0v)c`1atHsVbiXoL7(f9?Ll`v2wuatgnx#(t(^8_|cSMhT-VKnBZ;m
zy1LDl4*6ejQ(MmL)_?R-&~2KGstXj4NRE|ulRXmpVvxBpO2*oYfXa=T5gV3$ADSgm
zoMKh_7%hYIiQ&KRTn9hxlDq`90@S?vC<0?=7@ZZezq
zmQQJAeVu6P^K_BM(|gnJ6br;Pv(6JsYBniGwv0zM(W6Ql*MB|n5La)qJ1=KUR~$Oy
zDdc>Y^6X2Zgu>=m=J!%d^cvaE8bc9j#CK3Fq(A(g
zy?UVeC|w!Fvlw3+{s)d%&wnM^Zli#8=8KYo5_Im_fIbm@^ngmR7o2~Nnkc=&WYaeL
zGxt7qO}d%DZ-3i8(x*RJLLD!<1T(v4E-u7s_f(qM(ZsTXPQ2Ch*dxMcFy6
z4Ti@RlWb8NoS&gIt(|u8AM+q)dgM|KGSoI(q7KQyXI{z^!63yCJoW(<`@Sk3o1Z?B
z$I3|He39-68W|VQ(wnR;?v}q_0imX3o?gGmdpqRQBc_0_Ku-Uu_^G}S#U@iB^fhDD
zd}~bTvwzJIVSI1niZWa`XWTK{;k!ob{z
zVux~eLqMak%ZG#zqb;NSH!q|a4DHYIFq&<$<$uAWjE?N7@qgzeybjxE{mM^23EJ&U
zh5{E^PmdPPQ7jHimg%0Bvtc%druw`uM)h~q>b)CR+xPCs;WUjwMaS}?K{B;M-WdCF
zzSzxGuk2QqHs|sLMrgV;tltLTRz4UFth$<$yb$mdHP<=*;7o2|u;%lImWvOj)imw(
z7k^awPZSUDqmOofD!&n5j`=`Lzvi}r=^YYS*9PMBnmnFMzvw5FUZ&s5-LHC2=uNc(<8y_TM@xWgd7<_YrYzm8oxYqC$=DNJ4AR
z;ANw}i8*S~Mif?D&YH76Vdasl$)N70rE#hcAB;;UrR!h#OYpE)*Pq2F4eokI#x_{bA
zD)tn>w-|XQ&ZT_?J~lkF3sjBwE#7Kz%d#@j`|Rb_>R^{`p?H?oO+h)87(dhify8d_
zWBcvjLRVQ#N?&W9lH;1^EirBW@mbY^s$rzz*T<^5c9(0Xo8>H|3TN0Bo1~Oml0II7
zdKc16?IRrCc&WEsNs)Az-)Nn5x__;<7Zdw}>8tEw`9#@ParAx>&qw9LI4N^Xog8J*!(8)6ooG
z#-4!FpKviVwb|+e2DYRa*`44+3?x`YMlNM4m17IL}M9
zNls_+dG(7z=}|{kFk6yL$A2w8woB~r77IF=`x1DMYkd6pcm_{Y{M8f31iL4jo~*#{
zGjt*T{mh+{eA0ugqk8mw<6%eI;)4rBs#s&E3a90VzRH(&w4}S(H@~fSlD-iT
zhT1O5X?(S1JXWc#PBlLv=I(B>f;~jdqd=YPo+)D;?;14?;g;E)c{v9ddMhtaR*+(J*E@fuz5xW9yX?_l&6nJ9+BO4-|EIn#Ei!AL@HiQ6q8i
zaE!j+-Jw3~qq_C|J(e|Jt@@P}RKoa69rDT&#GTGM1#99IR!h8nTN?1E&Vyv##~=Bd
zTm3Cwv>fEI?Gn|EZhwiNFSx~*n%EekMwvWlG~ueTCS@iIGLH?tgco7@UeDeb=g*MZ
zS?00k-TduZcYdJ&g+O}0E4wvatMu2UB*sm)&h5*%5*q1rbpJsd+2d8lb@pFs*~R>x
z&Bv=&WJu&|sLT>qYMt)>%yBqTGkJk@B=bHv&dw?{uA|F1(xI<`L%4snx?3fgQwFjh
zU@HTG_og#115QMW%=Z1$B&pu;^tH)##~|yxY<`F9iz0KcahkI3_*QtkR<-qbC}x-W
zs9^G>(BWF^mF?r1*poMOB?;d9Hx_Aot|~P)5udwZ?s8zS(@pG*Rta{pD)FDd4Mhco
zvl-WMm2kY;hMwv#(d>V(7YbYb#eyvg_bh1oi4d;YXO1wv9Y33#KO}b}YUiMrH#=L#
zmAw7W`NPUso|}ZjTF=DOA{=2>~&=%_#P}D$Vw6r)>MkrHB(+hIStoYXxe{o)f_x2l<^t!<-^u#
zqCh}}UQf-eQwCP!z38^xtJRDZEG;Qy|Ak@Y2y$8eDd7!Y`Q_%l#+wa01)F8^u0K3x
zT^;6Q8NSN>4u3`p*;I7!&Z>rTnxNFxwPxDX(sNO-3_s^|>f#v94to;~n%d4?)t#)>
z@ydT6RJN&QNBe&k%+ubwA-S^$(r`l9Qy+V4gG9-$3!^wA{-DpXCnMTjq`gF|3Bvb<
zJ4~Yf{CwE{k5&$)vx;|>c)}-_clSi9eU+2vJW`#<+7*WhyUmaDTwS|OJ0^C0HRJEu
zxQYRb%Lylir+IAV*E9@>`9;D`>)-q{Rcqu{=Qx$z#z2bHn~RSv~r5R(|(LtxOVi
z!lK7*Ga5|d_fMGFUoYZd~2V@f}P%(*wkn~(YvQ@
zay`60Ev0{3dMocp)ls^QfM1>QRqIP*sgh$~&y&v04%8Mrxsf*N9{{S~Ef_vHeK?wv
zZ7>p%W3eC#rS$%B(w#82ZEUlHg}@s1npcnhTC-Xlisw6isz+?ciI=u!ywddnW+Bm(
zp7({GB=68n_JVwhN#?MsnCSF$sq(d|YSWyFm)?JEp829af2Hw?-G7GVNIsJBz)u*V
z(;f&qPBD_E`;SMZQ`066dev)NX!)GDR6+AnIpbDiV(_*!Th8T^%#7VlNx-8C$IheP
zJ18A5+tpg3E@phE^*!V)?=c`k>q%a5d%E%VV~>>=A5pYC3BIziYK6{MWN%_>3NtX5v8M(V5jQY13NK7$ZfA68G9WQFF)^2rZ37eoGB!4sz@7sue`#0~
z*BY)~z#0`?02c%f*2tO-2?$mx5TGoHfg~&!tPIHnh9ol~83>CYivo%$xK|W61Vz-M
z*0PF#O9iV`PyyLoP)e=swXU>CXM$GcX@A~lp3LOD-}avGyx*Z(@C7tC2^I-^VJJ>x
z(U~s5pUZ{tXhEVP5|gW)Xn}B4f1Dgr0T$h!$)r%J9x52Z5e(%(IP3ygj(9W>z)Boe
zM8YaSxHwa&z#B$k6=9cv$V9+}aY&e`gjv7}(k}Rz8mC1|;9(j9M1U2Bf=DBcBV^fe+!hhcz$7>7z~X@LywA6
z<8(|FMOVtT{Djen8fY+;e+&>`6)cCfjfq1gL_6_lcq{>VI=~+h!>Af2gS@aYqk?E9
z5k%nl-^7SU;ABqum^)Cz@JD5$A+rLi!)9+*Nw
zuvSyZ_HhuRPdKJZWc-W3WhkaWlmE>sMNo;9JOxRdk|9Eo*f`jS^EQTnDDSZ-7za!M
z#{xJ(9L*q0Yn4O7e=HIvT9T5i#FRh^$<=TQB87<$C0Px{!vI&s!70ff8a`f8SWZBK
zh;c&n#Nbf0*?mwc2As!Wg2_A6TRB+S(}{VqCWcmmq4Go^fu$4%55oy@Sp9!yZd@&|
zIJuk$DPXIAZtQq(NP)-`|G|HpUocGaw&Gzb1tcG5L)2bKe*!GwBe*zvtjS}z4-OGC
z;f6-ZVL)R!(3y)J?Z+%4a@ypCtcj}$Aul(=&Kz$e6f2gYuv!fkJ7`TX(V21ViQ^!-
z14dvlKUCyt`!8bBI(wpGOoE_Mz@F_0Ae9PAq%aAE*t6LnnMDX+0w-uC1Q>J_!-)`}
zjKfoa6jM>ie`97jEd~q)LW~xPYax3k8!(hAgdjjdA9LfAqpMXS^F7(OcbaxBz*U$G
z4n`!zjsAfPu>lp55XK~A#3FFwdG~MSM+KD
zZml($!`Xjt0qk0wUYS<7s2KDA(q(1p_H?u)gu2#9{-(ET+wtqqBmrOChf=wzCiTsj5x}e$&h+2j`oNMb}=hr
zs9QKta36IV7<+RGe>b(i^lZ&}*^HF@EMvjE%*5`yO7k^y
zz6^zE`_{hJzd!HS#9cFw7I+`|fqC=QtKVD{7qg=Fl-ky@SL}1%%dftDx$!fHm-%=LEJiZFhE|wtS4ZbRf1D}nzR@nM6fJ64mzzbaZ8ZsKGn7Ln4oBzq
zUG%zTo0QtIms>f#--vB(8oYaJQ}QPzwn2lxJ42%nXAWibRoz$OgOaoR(=TbVGo~Kx
z?n`R>MyGzK!K^!h-+S9UKw?&5!Yvh59w7|ot?zNKd6$zxU%zX}W7
z&i0ux8lg8WL^%-qIO}xrGM7rDe;f~FB%~zIKD49X`cwnr2yC(KzCLtTHs!)@^CPn5
zbWjC-mY>({Knto8j^1&6w9d51^Sn93MYsHi-aTplQ^h?m8leO`hr0hEA?bz7Bd_mP
z8}DFhnslFJ4qwBKxyv`j>t0%V<(cD@wmjC%Qj<%L$()uZ&GrKU5x0;Hf6cCNmyA6-
z(&_FKwpm=D(R$OMH?{r8Rd?e>^e2-$XjA6sEJ!w={dJE}WBAK<%ZV`~8z&!|S8Z~6
zer1gG#-bB*A#7yjoih{bqj<{)e%$wbrM~Sw9rNb;pKFI*)A}Tx_+N$#Xkg)tkA@
z@WEBM@y(85Eb>|*`es`gVQrQ!zEVZQG@fwmv*?q24lkq9&nTzu7
ziD&10dumI|4c)p8sXu2H-OX#8P*#%98BkocHq03EwOPc`NxmB1a-p=T^>}dJ6rF&l
z@kz2*^EXum&M4|Rg&Z(Q#No_oym
zz)2CUXZ`R-%Ytj=+oqp*@gT9a)6J|n%xroYg3+JSHWsdE(jUW?FLyjY
zyI}E#%Mo^0g*B@A>36GG$EtrTo44^VTNGz1b4UKP9Z@G4X(iIlh~Lw&s?$K-%{ZQIMURgQdBJ6W~1uJ0s(tOn;(9m;f?HCRQL9M=MK!
zk*z5}hDn|Ypa61tCoKUqAX|Vj(89>t3;;3%r~x$r>Z;-@ssKq9MRg@rTBi5Js(;RQ
zb|8oU(ITpnE%
zX(@@bGXHr704v}t(81C2Puc&{M)j@?@VB)0P%{UR&0hroG!{-yc6`jtE-o%i=FW~z
zOdtnyCOhlD^r=}`Is#lk4pxBoSAPefHSn*-INO>69RN-iz<(w9a~c3SOB0~2Bk)g<
z1n6HzoA*|}2fg!7{}%S%2&X?et^ef?a0CMXql|@-<6p6IN=kA78zW0wC!np7t;xHi
zlaZ6NBf#)4?)?*JO8qZ^K!B*TgTtR1^8agc_;;EAl`aB$e==Qb4=*E^|9?I)BU@)j
z_kZ-}KOfr!Wb0_@=;ZjXj6i^yr8V%6ddEL!W@-BuCoilZEg`O|#whnba<+`}p!e?B
zGC8?A{pI~9pRkx5AAp;O1Hi`01z>p}PjOpQQIL(zyJ|=HKkXBIzX#gN
z7UW{<@&BRCENxBA{`B3{*?*2%-PY3H87M9GKhEzI{NJ28&VEa=qnK5(Y%U)iN?%|HO&f6?zs{@3(B13>fF1*Ltzou(jLYd3%?&EBMzBl2&
z^uBlgk6Hk;tc;j~h<_see-F%GUgEYUAX7_Qa{wDB7r@BD!N?7s<$Vg+I5`0xtncG&
z3UvKzOaRPGwjihX5P+StlNZ1YDr`{sXcCn1TO*?{ipq%G5-hT
z05Dtp19AeGE&l=EwOaiHzKgZ~2YlCM^EYID*Jb+;_^u1|Z^-fP3$p!(4%T;_cK?R#
z?}hAu4u5UHe}{1W#aV*>@vyA#>h1pl-_<+(1HP+w{0Dqb@OL`ScX>`OpnsU(YdZhk
z)OOMg;bP?5Lz&xv9vN?xXuQtVHf
zGBJ>gDHAOOxMEpbWI(}OzXsn_s$7tq6rY}Vf}d@^CS?GY_3K(=eA)Q5sj|?vk+-}z
z0kXL8aOP(wb^lcP1>fQ}GTK9#%p9!(j;Yk9m?mO?`tXq`Z%g~bceb%Nw97;?-QuQ|
zY$%t!8GqJ#Ob2Up*9BDkR)U3zRxqcx2h6Alqb}h^x@p6lAnaioXuHmKRBigC95tHd*v1o
zNNPwj{x!lQ@lcg+Z^5tA5TU@Y^ANNEcU02hKYx`+FJPx#mazFOp5I~~dHn$!rGX1d
z%_6+GP0H<5!;0#-qhZxx~k<@k9(VnD~s
z`Yk;U&Gq_>2^3Y}`f?Gf$#vhUY|jPC1V8obWm>KptL@6zFQiarlFOt0QJN>d3wF9-jh-ZH<%OxPVRs=6#FP+v3B7b6Od%zoyOT|LLk(MQ1*vq|3sa#$6|4^E?tbJz?#B{zCBo`!{dGtKs)eQ
z2{DG4z3ggJwUm4#)iA?zSwKTV(|_dc_Ggm3Ca8|lPEiS{?@;XY;70yy3B!Hg0eefeJWorvWFw2Ct=Gt$r+u2*OD+oh*Y_Hm5hZ3?{sX
z)L0@2UvEr@6~jAVr(PFSbRek5j4=$S$uxYs?`QMZ%L2|1YQ-9Q{vfO5HenQ}AISNL
z&bY`F9ddZm|K+0)-b4iDZhufbNL3*jv-XjU?$7c=_im7_bn4%VY?Qu)2Y4
z2!g@as`8Ad_AwVMMb|5R=jgKPzA2+y{gWu`W9J;@4(&1u23J%oFKEK1La10Gh01!m
zfqOp7ymG(pZxvl4R-9oB^JP59kDU%vlurPcgOFHyF6f{w3OEr{Sbxl%ExXDd?6(y+
z8^l-TuZVK`*a5Rg)uLXKnLR4lVp*2?$HQ2-%&$#IpK|Uo$)}Mf>1TH)x)TKA;}(1*
z#uU=r)!0ponp@Ifnb$m*1Qukil9SJ)SoQrF&82oaBJX3G
zptvctPRGn~;mXiRB?Ttnhg<8D-jw9AnF5Qa2%De>CiGP5`dnBT;9_xyV0^%;O
z)LKCYDqKA>Qb^=%KDxI&+jy3o8UD13yac;m!OEujaW9KUYG)0Gcn7_vB27t_vr!v6Q4Qi?}9_2`jBh;i4~5
zX72j&EE54FLzosdS`b85gxD4oQd9LyLiXe)sQi#cG=J$=QYbP#m(G2fj(MEN6f)rU
zejcJvDWXcn0q^^I0B&jaLL?c?#5l)6tA8M)^hYch}bprM5Pv-j_0p@);+eD8lGAzrM
z-ob{dHY17|p@_H#JH@g>auwlT)olw2oW0>O`tB&6?T7(Ka(bHKWk+lahHcX>3?X<
zg)BJXeV9I3f7Q*L9&0e}&q{~2q%}6ixxH1Ha@cS`s=S;SY#%U08T3AOapwKn9pg-s
zz4L(FCZo9qHtJu|Fxh39LYU$HSs}0DM@RgUIYT-mK>0}@S|BqtSlkHZak>=qj+eb@K0-G+RoAq+NXI-8X&+zi+9;-J
z2{T~dQTIg_qpNVZO*th>qtC{5=*@AvAJt|X9%1`@#=&TK>$hD;6CN+JtcfL1b-OBn
zuTL2yy4U6Vvc?kURg&8M(q5_&rlND-XjsMg*6lb`Gj!&-f6Pz>@AF@7OW%&on_71K5w
zOH2MnZ@l!S=9rN{wx8!m;IN`L8wCDCe;EO9(7Eh&)so01(k8w0mM^`zX~5ubosz@{
z@LYxKB|{27mU|L~m0*H)>wn>M;cs56gtCL@Kif9uJ|Xq&U8?3I(Br!2ld=d>r!tD2
ztzeqcRr?nWBXmDqM1MBv!UnL(Yy1O3
zp1q9(;_0i}DA~Sqyjq542JCE#`zr#~@`MHh$V)LrcjBw@MYpn+u|^v(O+cIws_A$D
zg{H39ebsp8CAjQUxc=NX4XD@pXv)qZ$NSw-JD)17yB*=q)tLT7EO#2l@IDn;jqq`X
z{I0W`(_#(+2Ndo7DSsi&btwT;A4^fdLG#zL){~o?vP0FpuyD)YnJP#iv2357jo014
z4h>}<)E(YhJH620X@=%>x%oM7cHPODVJf_j(08iO{EL|-L67Ximw2N0#+E5
zC#8f3@Y%YTuOBQeaHpq7T@59ifXK4YhH-FV!$6pW)FiCPvVU8b{*bJN-9cOA{cG=A%?`-gvSI(tS&su*c1b=Zw_5bhx(
z)TROQv$_Y@q<;rvteew%ewc-AAX>e+?i|(uteC>-iIFBR&a!Bd5DHCej0+JYMB!Vh
z%)K#_(5zwX)KXID`2PrT-USQ$jfP#7+IrW$ans=*<)g%ah3JJ1m0Qiurf8Cf4pVyM
zwaK?G!TSlyyRdtU3mt8vLoXCeAG-*n&%Z+I}?*T
zItDFxsIvudIH&rgJ7)45NSM3FV#D~|jW-P-oQySipAS<_CsAuFR5@=BTNk=>3mKTS
zzJc#b^M8ok#5BItvn&M-4gsV|iJr}0uep4k)6@pBkfP#)z|F;#3sAR){Mz!Aba+h7Cr+lGwdZ+6OngZFc~Km_b|u~ppgVI6CE9zMCb=WmOJFqjVFZ2^=#;;Vcq?u5?Q@sQ))fGaNLig0!70%z+(#r&={yX#7!umKW$2N8G
z7)4r@ue2*iw(UreZ?HLnZ5MG0u6H5hi==o{$_vt`usy4Q$xvjjn40k7H
zc{kcb+i@zJQSy9qJUd5X7PeLaw;orxKwsXpMa1qy};oNAyyX;{PkEPDg
ztk1zo6M05FYm%^4mGwYx4_ad1kC;r{`=$9`s@%9k3{sKuw&IO!t~u8Ov6i53Xt(Q9
zLmAn)IP{ye63hh}7(&ea_qLBeM@bAHqfl1-^5)Mat}lyL4mPQiMSl=syjb_91%=ce
zZy#s%N8@wR`b}tS=RylX1X$DhmST1#IenSHTd$edeEIgXVaIQR>Y$cot}n@+n>CKK
zeff)umHQFQv7zV{e18N_+FnFEmPUot{ooFWl;1T|sM^u`WW;u02z}cDE%Urz2
zw=?VK0fQ7(4Z6j}$$xysOEnB|Q`VMHTK51tzuPpgK>hebcjfkMp<`sDQ-P=j;RhH$u!)w+$2E6HfAw1!=
zY^C^X9t0;C(xIVm5MVx)*Mt~3-M>lu*+ePr?Tp+XjcX4#see$+ek_XTqZJ%+e6Jp`
z+Nox=DeC5-Ex>gmlD%~>H~0n1f$wy@n@@IS;!UC$_KJ0qjc4A+CM3xckHoP?11Eei0aL#x`5xb_eV9&Z!BF&
z-xRjX)uEOn*nh!NWiJo=0=5R=l&fTlM=xlrngjS)hT_Q)(!N3uC-B3S?rEW_V6oHC
z`nUXid&W$k-3|i(Kp0g{Qqw{VNQzJz61Iwn0vs)(imEW0=iv}z-IOpaQs$n8%4u~0AEPu*r~j|F@id&}_f
z;3m)Y#DLFuf(O52SkAaAcG7A-Fv`hxC}b;_FRke(YuI+Bbh{fjsu@wBf{%#YA{84U
zRL6^ub$^f`l(fC^6$YmjtVwL5E*5}q1$qojs}BzKt|lkS)j%$8K5j;9HRa!9p7QRIFP0Z{E@wA#|M3aBcNgjIo{}c>$F5qwun-8`S+o!%MJh!1JR*#
zhku_`J-CZ~OT8wE6R_z!6H=qV9=jiyM#kWnluBwHJ2!!iYhn?0V75m?k2?*cH?1y)
zyqP4h4`J}7WEjd7y=j@&{REVb;(g=n>Fn$V1<1IEAff|?Pxg`QpPwq9iReYDlTmgG
z#bvD*S4`K?&=(7C>PRAHH2Z@aL2d%TMStRO8Xo4Av*OU78~LLmDRde#!!|V#Oj#W#
z_01h(3l)9FS(Rx!A|I~;vT=+&&qbKM0cm*HB+rkF!$mi6+5F=NOT;Qfc%G_oT`?(<
z$21n82*rG8qTj+kKf1pUn*8eE;tv@l3__RVUs)s|Y$J&*Pt1Ob$CL8tFJv;juYW$Q
zg*6%}QiohM5#hgus1(zx?~7@-Ep*@(r3Z6{pOJDTk=|q``4xM}ctn`e`s9XC6TQ(a
z7n`Y3pX!*}b9flfxWHVl(EItDhEqj25=Kq
znuiD67?jF4!MQGg2fj#x%k1LZ7ZiVsnc{c@Jmc1Y}egKRV!z-v~@XQ>#tSoCq!bgOy+O=YL};r%E{@hp-iR
zov;c{5GR?$PVc2W_tyvVxa=-qu5OTU|+L`oIM4zbn^V;RN$aGQX|nCpC`D
zs?!vpE-{mzeECbQ4I4Ih)tFlF;!^vwZ$Bs;`E-*cbDyxlx__yRd>!6@M2c8XWNVKO
zkk5cD*ph`c-Qn5mDvFE+nNdio$gXtVXBQ^h;m~!5g0Y&9S&Gic$yfa`RJzfFptX9o
zSd;%J41NHmp_mlwZ?LoGGTA1b$Fy-5hFT|5$l|-(Y6p&1w$HAL@xSg&DC#f6
z{}!HUHGn5IB_qjZi`#%`gXu^jFIOp9;HXd
zx8&*<(#coJd8}^8dgbTh^gR=T^7jgr
zqGlU6NPpzW!Ium^7~sC}X@vC^%Mwg#jnG;x%%yDOg5@goKCvq*(&-$_!N-66pyfc1
zKCof;dbD(3>s|*4MQ$b
zE5B;7P3QV%Au6BV!upK&%EVtw7TG&*ese6yhV6>gJ2Aq;`1Uhx*c&R;W*<5+?tH**
zI0%~x!yGfiBMiakMwO>P9zb3#%6?k0wq3&Dm|?TtF4wmK@=+ke&-QVi2HD4V2?$XP?)U{2*wt|_WPE!T;HM;@iYliT
zF;~8f>#l^6G0YLf_#d80A!Pd87@#8;(Y0CJ?zAWf)~Gy8I`8W$BbUF}O9GU>Oq6i#
zQ)Oel27xrWvfyh?4v@&V70xBDA-x4Ej3Z|R63mUT(-RHYz8t_j1j~KLc0bLVet-B<
zhXR>wa1Ta;TtAi#|*b%$K&
zQ@IESFDq)M@VnBhj~}3|AsQvVG2;5m!E~+^E7XuPgEf~srPp!(Lo0ey4rsqiIP%z}
zemf2|qHKv`kPy$zHOvBT%)_R2OMls^E+%atszNWANHkmeabNwbIfZY*5tNtamzzRi
zc(rG`z8f$Iaw`2w!GXjgM}gMN;GCy^3S(ko|9k~JR&mN34!!7IsB$7jD1no5R_vdQ
zUkyIYc(~s>4;4jpRzLZzcnLW8f2Gp;d8*NdIeZ04MZV>G+VA{X>(gM5Lw}m#r|*+`
z0SZ&!{b5ZG*)Ec@a?h;5^2Auc$2Y0gNehG)-G~$y#^>`tqRLc-R8`V{Tx4&};;aH=
z5ivQxyeQYc{nS$j3uAJxp~Pjk*deYO1S#Cb=e)N7tGh7+-J_
z!oa2uBs^;Xf-8S%F2LvE9{YZhb|b=IM>%hTEG>HZc7M^zzi>@o^?%G|Jl|R=Ni}54
zf^FX#GS@q*j%qP3rfcAfRiWGd6Fg&}L0tvIAnr(AwWaE-=*G{d!Xo2DwhMcP$`&Z3
zt*lK?QB)OvmXUl3W5}rq53g$++PDqqFaoD*4Ynnkby#Yrr-Km4>y~EUrE{@X0RgVZ
zmgMBFBRY0LDz-GfV}C4o4#dm8x&ujQGP~K`nR5LdTud7Mwa##A278Q3w5Z7(zmpGB
zh(acqfWE+bC&tO$s^A+ZejtX3&X4J@nkc*Nn+4BfoJ)mqyTjh)FV`j5qT3-7P4xV_
z7}W693AR9W7WCXNF?
ziig#g0IM*r@mVHLN!!&YDhB)fq=QO59$2xAuqPDyl0{y!{$K
zp4l)@6}jC&!hw%%urU*Nsc6d#fs%_#l#BH^(vi==``T$=kp&w*G>wa84kN5IA0;Sh
z8t!R75C(Yt)_=Cg&$LC}FCDMM0Z9*xnfu@2Gy_5(eRr`>1(1Z&Dou{YUO_!cAq*F4
zJuOop{*lhmfunq_t;aJ31r_0CpyNb#!Z)Y!Ekd-J?nCaxQxDvz=NHg{m>XMLrP6lp
zhw<=lCkILuePevQqhw=g%pY~YAr2pLF4H@Z6UJP_g@15|5Z&*I%87*p)u~iZ=rZ7t
zk?ZU4M0J!RQ0n+Wwx8XmL;>P0t*>!GV1E88#o`-Sr)2RuTlncq@H2~K9GrdeOI|}!
z-%EcBG!)-)JEJq0ow&hx()Uc@lh+_fLZ92)NSP5;dZEfmrG&KQkrc%x2O$$uGT@wP
zSb5KBmVdla%1ZtaP6@=n6ToxbD?1X&;TcC$s>j{%$+Z^%8Y98opTD}8owj;zd@=2?
zwcLQIJV362YNEdLnbwdjP|xG?YO-4PqB8G=hv7w;An6;kJX@p8>WxS*O{MEf{|faR
z0pXc}w<2KPe*LAFm^s+5Dh@WbO3vAbREUTh%M{SiKqLn|Y1_;&X`)Ej=DT!_bWZN
zzz2Ei1-F89xgs)~u7`Lp1XuReOt@oOzW2`Om^(otL`-tbJ6ufcLeBTO|4}0*4|xq&
z34g4E1)WHnDqnG1%9z1=DYF!xYWcT&f=V4P9fS0#qN?3tr{JT$wbl&zw!Wkmy$6Rj
z)nn7WD_$f`T9JF2HtW#T9Mq3P@CN9{HWM_hq;XLK)<%4;cF$7=Jxd)3c9;Q96AZwHBY#VnK-(&h7KYyoz6ZnH+a&pg0MM5dx(vd%
z3xKMk2f=Y8arH2%q9UpM3kH&xBt$#BS
zgvV$MW{AjwJ53?KAOAnzSBV}t%Vq-bGNadZ*aN|YxG)c4)`q|WIva~r`XovxF#`*=
zAc7K~8~haVUkZWR6h6l}Q8tEU6B|8(Ee;fcO-eA7@j*)IdmuDs={-K|TLkljaiq9!
zDf|0w^HG6B9Cc?SE|Vq!-X_z7#ec$#=rF8cbgOxdB?ULfOr>he=Iv=Sa?%osEWcBw
z0+P~=F#H#-HqjNl=T(C^4`f_ovENLUpI=V#S3fZ)GqnN2i#EkR=fu$IPddpbY<>2u
z>^grmtvV?Sjc^XyXZaET@hc;sq=)f&Tw!8Ud1b}gkhmAl;FdarIQh#V_!pjPR230u4l(yUF|jQ
zN8Ed19?qx~Ba!3E6fhTui+|T6ANb<96FP>U93WQ=wO%|sv-LW~Svc?s-;O8*k4vp)
zZl#1S>FL#i;k!M8mi&g70~;cL3zsUo2v{z5cjpe?;O*ahdv5g8$9|LG6air0PY#3_
z6qd;QnfH3ye!8pY4u+g;=DR8-ZBxc5YIuP8p#xgs|HkvRQswN&&42k89m4r`xMqZ@
z+|3Q6A`bqbWvcK%7+6ER+TwUtQ1s}XhET$dN3}DY%7n)e6<%BxeOHz>Tg=+J64oPL
zz6w))iSUIOV+vpff8mi>Qu0GT%GAq{G4|RPG6Iqqt4Go{pb%&IzOd)>8lkHPo
ztu)hK%7+}Hl;pu!T7N#I1Yn39CoSZOpaA8mMvz*lSl*yg(v{N&2b4Ox+RPFj|elOU=R*Q*Yx~^Mbb(Nwr|tB>Y<e!RV`^rvYvmRIOXX#!tY!tozx_&hv|R>3
z={{D>!SmSn)M%bd_sO@(DAb&8UFWq)iO4?KWmZha=+lh4Svw9UAvzy5_`v2f_Uli3
zOq&(9Bi{)58Gk~QBVOfWhVM9CsYMdOW#`emG|t?4TiTG`;=`}sPy9mGhq*jX^cf~z
z3H^*Ki^8KVS3@9Z6ao_|eo*WS5ih!y$z|Z5I9{d1pJ!4nPrt3lRB=Rc=;zsFQangD
zKG8{=&m1=+G|f4Z)h$fww?BV&e46~d5eIR9TW7`%?tdE-GUQKEqFf_}CCQT~s}?5?
zm%u-qKfeYAy`rjmHOV(;He7h$YfHUH*`N1(K>1l3{CpykVKXFg2(T_WwfwojI?Is#
zM)Xu{h#EYPThyW(x@1#A*R)bR5d+Smyp~3$;Ol62JPjT|mW!gA{nQWd3VOtVioAX4
z#t_WHMSt%qtwE|EF+uj|C!b&58FeEWV(5B=ll|g;&E(_@HkLZ-k&h|rrZQoyVu6*w
zFELWKUO{&qWG*nSCb_g}pDJ_9vK;hyNJrdxHo;VLl`=s*MZ~a?enGErfvH!w_-$Js
zfr<+nYI*e(2uM4FT^$Q6v9R#F(YBTu?|isYoqy9{k)GRO=tdS|g&uDo>Z)7g;h+2K
zY`h}%Ug+P96=jD=>Px+Wvv~`>&o4DNFcq9C@Vcl`0n=;HHnB*bmOP$D3J+5tbf_kD
zRn*Azih12VY6j>OiSzp{mJeq0@+Yy@+8XYm8rY(shQw;oEXKD8Ln)J?X
zZ)#MEdOdctejeyriPOEIu|?4+1sMpM`H&!J?G~`SWLTCOPDlk!LUW7ElOGd8H-AbU
zoRRholVNFdPlHxyL{D3(2~JpSaj|LB60%>1I`N3lR6iRp(!*RIXGhfxK&l6bZ;1#v
zKHW3@Fe-?(!H0UO5;Y+tG1-
zMj_&qgTEmUX^X#lsQ$HSSQ?zEK{|D)Iz%~)%8NIs|D=GYoGADZ)0
z2e2=)&-}dZM&3BqZe{Qeag&2CRAY7Xs0@#2uVbhltzm14_Fyw#Z&?%YV$s|t&Ma1F
zPCkK$GYG=ukM}L&-SnFSXH*Q@Vw{y6(S4fhS8#4{rdGJc3-Qe5yo0b`E
z{;)m9sx4*gn6tdG3Eu%{?jX59wA0y1Jj?U@ML7b$#Yn!eca2M9vJgTEf)Ou>3Vn%K
z8sZVU9$NpAEpU+tH1iYoS?N6MSz=2|C30bVIDNs4z;
z|Kihwic0m;2qR`a5`X+QaCP^qi)wcL+vC`5c`b2(7|ez+a@x@D)l7wje|@c@Jgi3Y
zj#ll172J~HhhJHHOSa0*imYggR&~dMb27O|#G%eegdZQSwVOWC&l`2*ah|l@>7E8r
zmFIv<%x8%#S3M0D1CJCtg}2G64CSLV?asV#hM$5TEc7L5s9j)_SRIwN7{Qzn}
zmA}Nk#tTxFM%HjM(WujcV1^en%<==Yl_)#a7nY&
z6!(9@U|?Hm8zr;n*uRtf_0qjdFYgv5Jx*>D)JPvC9nD@aPz8nACS9IHxTkMhMQejk
z<-*l}oARv0kV&$GtJi-JEHS*Sy;)kn&BIMxv=4^7(D53Z7JX`*65PR__voc0Y^Y}t
z-tzkstbV%~hlHHu{knWzte<4Y#rGU?cSm*1T2Lfd^owFnRz8(gN&=c+Itl$=v|V!1
zr-|@U{-H9?NXI$YDr-9{M{|2!<@^nn_3iitpX;vT*t$d-8J&N%QI%*TRhr@oax};3
z1-g&WoP%K+wJ-hfi=94LGtmVo>TUSe7`kZW!=yG2ldbibej9?>*ViGiIA3pVKD&;R
zz(ZXU7;y^JS?bDeZrt88FO?b?kfCDkd4FW&X=MK7QmeZaQ`*2ru*=8bMXf-kP7IY}
zwi){KVd9)`eqDdmP`BosaY?9I_nxadC9_hhg~m2Jw|0B8Ihxq!P7pdiK9gW7%owbQ
zo}RmL0==O{7x5R{1~2((;nq`W!*p-NC`|j%#&>_FCb1#u;as1%-vDgacwsE*B#MPb
zDCT-dIbG=Zi3_c-<5xUC_D+2GKdIcBq}y&qAf~-;tKEMov%dDVaUzk0l9M1@c%-lB
zT5HP;ZV_&jUUblpW~=T5&Ez$1U{xXf82rsj3GSpRTDG#9Ob9ePL3
z0bTJEXCAxCKM1;4^??D{5>UhV&FK09&han=q74UU{w3kVUO0ox@{)94|A@ilEz;9W
zW&H*u@oiXbd1^f9fy>b6#5QQ8mVbx3^(!82n!qlj*B_w~Ewt+J3IHlyCOXW|awumt{JH%faB*+u+ynn*A(+LS3NW5d1T
z#OVE`tY1aBizo8dw>-r$^5v&LDLZNjz}Z=PyX9Tq8mzFQ1U@cb-fi$+csA6r!eM{K
z4N0`tKOCz>m_EOGme}Wcn3OE&>
zr?|c1g;hReYi&1A;%-9FEM{q32c0^4AtDjdGC&Pf4eE~GulkHG8U~{o&*&By)DEPL%iIr)$W~K(fGc09Zk3@(L6{8
zjUI_vwW+rYTE6mYa}WcqQ?$!Gs_Zc9C*6=QP=qp&s{$eHi5sKLElvHwm|1_wp@V#1
zaT(1WslRyYr(&aAAhS8G*?ZIrR(IbV_79V~_0WLpCwvOK8{83z%8wrXRqbc?vBENz
zmPz42_(0ybD`rsHkf+W&@>9wx{p>T;bgRo!bOHN%Wb#ub0W-xtWEY-c%FSq`O`9wp
zqpK$h**e0fnc?5EjHCh>f@*)V7De=-xCflBu!ntVI4D0~YGYJjhtjnUw85S^OuNyk
z)jrmwg*P$#Is`Phtn`Of+WTpL|2>?1mQOtE76aVs_uE_SYUA}RBNpqqhh`;8oQ3oU
zvzPqUar#>do~qwMdYX%W^ScZ>#7h38b25rfmdPwff2G9BCovq&lFNU@{vzCzOl%B8
z3*JaLh~#PHxju_@j@QE&dZ0-BodY1FC9E0T&DJh`wm#8m#_IeFh*Bdc&`0DVC
znHUI_!xzaXX{L=9DYLn(+8ehHZQY`4xEB<$X~hqAs2bJs@y+2KIbJAJp5p()!mYxFc%fhA64$cLLQhM;hID
zt~SuiO43%*;8l2sNt%vu&@?(FfhXWrO$QqkD7b^HP
z%)T7;?FV?Kv0EXkmTB<5{?vg=3z4@nY|*8VHVqSja_>O?i8P8Dc<_Qwrt6xpftWHh
zy1P&GS%XldTmNsQq$ngYXrZWz;>a3*dvevN)4Igc4(<%Ya_uAwaeF`K(h0?T!&A-|R*7XmJOa!zPBxbpOpe3Xqe$
zCBx&b%yjQekf(O8=j^bj6yR6ODBY_R1;6xGKuMkb`L2J*4%g?;pQA{JU(kPqDU)Ta
zWk&_62%RsnA9Ico%E5`|>0}FV@r;X<@k#}Ka+OzKHc`T^C
zxnU6hFuJZ&Ivj1z=Lkh|T(Wjd;S#*mUIvJQS~cyBxolvNT}oR`CWB@5Po8I@Qm)?_
zR5$e8th|4z^o6Is`mM43%dl9~47bZ~BWt_5h(%zK+U%30m}L4$+pncxcX#V@Lscm{
zREOAh;sfaXs_GS6=4il3F0ME}!(O!p1Z_`0%U6MY#_}$hdxlYerRyfhS)^1O_v(6|
zr@Unl^{DZWQ838u%@pQ=$0B3yL=z1DB+HRU(wBeVD=tIseV*;b9NB1O9%-`7XHE#?
zQDy}&%{VI;KM2N)XS$p3(>QXM-t5wFP?w+(V<|ArGT0hWTYg@g9h#{n)czoxxAR$i
zQ7GR=2^GR$reW7<{&>C|OBnQYR&mL5_cH*F@`*j0ywln)wwcOB>Ft7y)aHRlqy7K<
zU*&(j)phZtN2@wVD5_Zf;IXCd%i9dcd$hJ)D7m*cyoKBGfQlV~R0D}t19a2y
zP?^Db=&l0;X8qdyVex?u%HlO&)5?KQ&yTnCQ)KI?IeQ(Aq|?&RCC5ftUA(`EM%3fs
zIoy9woVL?;?yb+(I0XrQ(zuUqRObxS$eDj>i3gX5phA#sDFU>V1>c4}izhBA>nOBg
zY{!e0w5@wv)7EA9PC=Pv#3sTxLPL)iIQOjHPCdfR!nO;VU^a3j(gyUu3L%aqsLgsx
z%P+!g?ydZCNXN^6IAxmC-RsdBYw!G7L-rj=HH)lOi=h)yocZY{0
zz*!cfehx9=RHCZ?Q-3IW5^DO4`kC6;1rtpD=$8n>QS@UKtLE|%DYWtcln4d$5pAVI
z0RLltI3L5bIQeh%l(}-rxUum?<#wAX0@p(KABp9%@We8tX_Q*66N
zt0OQ5SZ2W?9N9Z1(hW#V7~l6|(SD7ZMBjtkbI(0z-(Q^nCZ}>TZ4nLmz+)O+nCHSG
zVRG{*n|9685;8~w9RXJV0OfMWy#{qhvo!2pN6E=Q>AI1fezg|V=xnG*W4$U^616aE
zPY{Z0$M=e+Vs5_Guj1D7R4RY5K&B9?vaZTIwVC=w=eExLjue6}YR{Lgo`K`2Ah7KS
z@*G2+8m>wK^|p|;__yLlESxm1$R}$(w$MtIF}#;rZ8RF@v$w0=z}HNNH434UuYe53
z)wg^&)jixdHS?jjIKcTh!qg^BhYSQjS5*2lVT%X4UidX<3n?^dAS!=YT`u@`mKO>@
zh9AkZ0Gx4+D|?!;^V1z^D-fiiGQK7bp?_CmLpX@PXFTk9zy!TYTF~h@+rniVN5Uq(
zXDLIm^g%ZO$5g$^X`skSNHDp1`kBux{l^zR!6!b{F*-xzCz3v}z&sJ=Y6%GfZTy?w
zo*mX#JsS9a<3jZpb+Uh$fp3ZPkP4(;UeTfsS@t71ycne28t|lr6!LVfYr0gg%PYS~
z3r{X?e|`LxL0l|Eo+17n7z1U&x%x6W_P!dzhhX;d{ndlv?liM@2PR6iJMv+>p4sR!_ayC^JR}pO6iEcDbY`A|J2+%rWKSYjXbX3)O
z)MOLn#6BZ64=fc!Yq>spoP05p)Ms$ys)xldBs4>siL+
zM*{HwNdrvzbnUHN2Qze`KtUgE#UJlks;1+v|FM?K?+ejsR$FAYA8n!zi);n)85lCA
z=u|RoN@h_ow`Hf}&b;jWEH;T>$GHDG37Y2|Ot61B{>UpF*E-wQJ-
z?G~YmP$)by3-kNb8pPn+VZpp+=Z|*Q;J>k~sj_(oUH%+EZSV(yaht;`5I5R0@GLr^
zd?Fo;D63XFSU#1oD@?HRkbHgPW3)wU9~w21&H;Z5^;X|mM;*KitW=b-GT@gY&)o#F
zc^E$-i!Bw5e)V*)^|%ZX76};X-h293Wkaif9t>LMiZRBn&V?+d_t1r+JA&T#7Ifv9
zq`aU@YBc!5DM8UqXi@Mh4&HXft3m$Qrhp~(k_7S&_d|MxKX}7Zg54`;$IdS@Pdq
z20w}Ait4`q80uy-q29JV@b<+tVG=qTOcHk1C7F
zv(&SXy?HiIMctlMva!R$p(U46b>b+HxCDCz_N#lID&sskv!pAHxpZ$6wL1H?n@7X8
zLjl>64%U~wJ-2aZu)SAY3>0u6KzSA!Bz0uTQcLgRB`cQFD4oqe
zcvhFGbgwQ*)#QPGHdk+*-opE1LM(FEL>}@7gPvNVf{w$pHB>9%^U#5@--OE1kZGB1paA-3}Ji956xOlLT;}!nD20|iS;)(YTj0+DX7TZy?{?ig
zgh%~kDgxS7g8lRkolJiwFAQhTf0#L!ss2FNPJvju-pA61DUq2kClSdTkii;lWfDS*
z49Da?XJ{7KzZ%X;zrUebuzuCaJ(q0C(gu;Ra^Eew1jA;L?RrO2sA4mv`#QW=uj}QxQ1i{(C~(#&Qca#jRftLCm_|Y_9_vVRNCuIh{pEP$^>gwZ~(62T*mF17K35f-dqMZZU+pI
z{C|Z(I*VYfKURP4%L0#+{5Mgl>(SYK8%jMFFUI2oaXeC8YZA!3LBC`gHPLL+XWmnw
zdkfj1+Bi?oR-C6C1l9jBE$zn4N-of2AEr|g4v}UFB~_i1yAfFyGl}4N
z>vYaWpnO;qh?d1Xh=S;sFi*{Y%q}crbwn${;K1^ZJjP43lRY}@$)D_L!>~a9k(_pz
zN&s@uQdu^2w9dF@BSL0G4j9KUVZ${5NBbb-h<{N_SeO0sC1_fS1!T$3id^gP7Y8UU
zmM@`kB#nP8_0K0_?dN5buO~
zzsJlRy{;{SF}r-xnCX0Vmj}P1Y%aV3c%s~DBsB;lsh}$M?neY+65LV_*aNvVshe);a8x
zpXGn@5eIKf6qw`TwWI0~)bsWkbOc7j(-HR^NUh~AQ(_1+
zN-L4^VhsOvz5QE@V`YWbBtiVpWQ-{`wN>^F#BYHCOLNcAba13?9`z$OU;9BQi)?{&
zth-)c(C{B^IEnrSU@%_YU}ka-z2v(BBNTt|W9yH50XEfv85z`9zY~(9taHkvkPm98
z!58ua%Ym0ZJ}7v9=PT8oSo$AUkZmwPB`F=VJmn2SLs)*4
zO?kB>`GI-gG%$7ZPh-MZst+DQ>BPmp+noj2EBn5+FM)3Tx}5)mtOqQ0A&SKG_2`*b
zUh?d!$7GR{ggt3aY^9SJz!p969kFDP%W81{xkq!8=ZwW=n>=*Hl)3sFq(4QHsf>*;
z|37>Rg!_#TN`H81kdm46#c=0C`>}tYPlOpcV4FHUF6TieNrWp|p5p*V^B0t>n?#=l
zn&=y+XY5nbWv>B7-*n(4fZyt&GtOzgI4T{a-Z{b=gbO+>cgkpI{l9x=!FlR92n^11
z(Fr=QFM3?H@!c2RoWJK!#1E;Az0z8=YW0hAfEO8p(}!~=E>;@q9fQE}jkJG#nE1`3
z_2dFZ-Vom_^xIbXb%l}Y%IQ@B)$B)3eCDr?T$(fVGDN#h!&VzhJ`YlwqQhhiV)O82@{0l?yR6lNJhUSAlB@X?{FE;+qZS;onmh-
zf)O6vN`m@T5&;o{F?CGM&5&3f-4!zJB~M`|u=S}6CE>Lx=xYl`R$>d2dx>-clO=W-
zHU{}J
zrTwSg(`&x!9P%oxp42c|J1FX>MbI2=^zO;9=pKh0Ik{kOtNZ?<3h&QT00YUA<#2U5
zM3Lk`ydSJ*bS?Zyg&cqSdiV>78oH`{=*py8Y`}}0Q~04KpdAs~D
zetLom!G4t5fnc^57O+I+>vky=9+H4%@c>9(LFPF9T-pZ+iEn@Dj0oH^JdrF@SVJFA
z^7`0aSoHD@h!n%b^F#y01jZJ+I8%q~UZ6h|or%7cbae<$ZGGo8EA;4oTg^uQtPmOQ
zzV$$43g2@r4>cmL_-JA2Qomr^n40tTIL{4T5;Wr5nijdmRTme^G?S-U4kxorE-oA5
z?TAdT*1u-upHzQdhEwTRQKBuu@(0SH*OikiC+(Y}vDrnIWd=E;gP(l^JJ-(BY{&PK
z;`$>3S$qEv64&2c;u&#g6$deKY_i@3HOuQ)xQ3RtFPecTq49J$(sb*5+{HD1YTugJ
z!(M7bwt_rPJuU&Ju#PaG@u?h2hX_L#umT#bTN=wQyZ(P)>K`l>foI0LP@C;E;IP^W
zlr-1=lY*3<5~idqiBvI``;Vd?h?LU6ouoFC{qc`rMi`sW0xH;TW*gAKx2os5TVP`G
zrEg`p{2~6t4?tPM>OJn1*P4r|2SSYB=vr6zA(hqzA9B~f^DN5R9E
zQLvq@xz?;^6@w^VX9ks;&`Q%IVGkBg;JvJ@)X59}puEWK@5-NheP3-NYn(3XYnEm#
z`oR=|-0gbvt>_==5$l10y+>y#PGy$)!8jNGH+z3|&8V1Pcx-u=R;&5m)x0^Ic+@2~
zx~P5&zIeJ@Tnnx*H8kSA9ZWeM>8jjqCh^m-u<3btHp&YfT%0E{79&Kjuq9fN|9V9B
zw*|_43n}}Gkd2EP2_B__1o<`V#Y4|e_aTl-D}pepaWc|+f?X>T!r0WI(DaRf*p68T
zhnIiYTB*j|GlI`0G+bsO)kZIQYmj(>nAWYe8vdJe1xn!zyq!7_cy_`1;fTkm9k52k-ig2gEw~(hVVD$OnUG|6
z6QO;_n3?M{j0M{rj#b^OG9)5-(=~9_hkt+NWWM_+`U;>&vVvBCRNp$Xt*<5c%DD??
z7%1I9a0QRJ5ua@$WnSxoWykVzZ-V1pS@6#Ngz5HGU<-(o?kr`ySv!Y6ea#w>D3|GU
z1jC=KV2E@fGoNim8dM7>4V(J5h116rVm0CST`dCHlp08lwD`|1(Ls8+W5QuyEhc~N
z6W5?h7=trH{!^V-{|)VCVag%!vOjzdMbPWG(gc{_G+Y!uxbmWj>To_lfH@p(tYT{Y|3qv}P?u4EFFxw5e{Vul9d(qQ%Q2Nl!(wqjQ^&MykNqbuWC9M@rv`_ORU$jWQE
zlO6cYGft4Am9BK}r{LgnkX(OzfW@O*LD-hY|JeZbm?hU5=zv&DZX6zde9~_UhUTCf
zZ81`D(GM;|X^Jo)?3TE+j8)dqxy`L<4~cfBccof^`02t&%D;O!jgnr}3viHj)0F$i
z<}iCjxy)zxuQRqf$Of_Z2)=s&rf*On8drScFTdcI-nsJI-alXU2yB0@;<*rXanWYC
zq^IBAI4HtPjS<*4CGtI2UAV_(=6UPe2kJU~Sd73Ii3wZZJ4JA17(#4OKV#%=+g(TI(4~in8A9&Q;=(-tlUQznO3BTZQnXB^PS`p@{(AM2TTgz*dp+^8|(px)@J^0
zD^;gQuQ;vbuMt8-wwjXY@W?G!ti^?=`Ea{mKC;Q9)mP4m#dVmk+^~2bwNh!Zg2s=pN3iG?B()3
zab=VsR2{fG*H1ju(FW0tQ1RC>%HjKc&B?i
zYpGN;@1yP;h@pRwW{%S)xF91~r+@!0g9>vV+({&g*9ry3G5H;RcN}M1~2F}-u
z?DI?f^c@1n?2~^9RapH3nIaZV$qF@O&C&%9NZbD<$O))k^wso??`5i&GsVcoaAryrJ%)l7A|~mjC|aBmQE=3gYNkR>uFiB)1#MWP#s4U7}NK@xEMa7
zArltB%CCPCelQ0}_xOx&30V-fR@1E4sel#N-}$co_qsD*9vTFI>EEF`R7ePz(SN=1
zDm?OhtLDef7XWP`Q1;V!`b-!^z>Nmv3|>FeypGGy88v_X!GMr)OJL*7P=F97U8uAISmk7C
zyJ7E)ij}lqTayp-lw|#|(**hK=BW`v*;SM8pVc0iO&ZUrFLba!vl_O7y=fgd!YrzN
z!x%A_`4a%X>sQb6EPT2s)|`G++O>tO5p9qETMyY0Mwq+$gs=im8ioCAFQl0tpXj!h`vC7>9lhyFuXsW)0e8e+*tdzO-gB6QVL5dtzA
zWda{Qn%_BPj07Nsv6kte{tvkYet%-)@jKvIIl`v@psv#w?U6btaBEYI8GC(VwiG?L
zq0I3o>(q}h8wB)bF=Azg{dbZadG&Ss=pKK3Q*{14ZRbL!m<)_nsoisbAZfe+dzl4{
zi;nxL_G`Y*p*J%FdoXguib2-%d?69^uxJ>o-!nI{ITaCZWn=B48fh1xdpK2sW0v#~
z>DX!l>;lI01qzy}TJ8F*7td+xS^2%{k6rD$v)Z_=Y}#J6g%6-ORvS%mOAE?Gh24ME
z?aOS4q-2U3KqFR(l9LAN0nz@YfMH?NAG!Cj9UnRt8!`6>Ar+-ET*pd-Tv~WumLu@g
zg9#mPmA*poh*N#Tia~HxP2a`HiLLbo_T{xMuwNM-&(-{9Fw6SzySR#74z(!IodulV
zTS$#NXr;RRwDG&S3(0Dshib(62(Etumwg#~0cYDtYIky3da}|q(8P9IZp#7EvN%r%
zOU=E4+^DklNp!aua&RU-8s5hSH*nD(|2(7~t
zO&>V(djeAil`L;Q)1qsBi>i*>I6d48#&iP(#15Fvly|JXlF_;(;4?*yHz$AXOO_k(
zO2$v!Sp#+@K!t3Tf$m^-^GGjo+-9A`YF||xt~q|9t$hL*b#V-3O`ex>cI2wlM=wLY
z-GZ(r!KG{Tj3d6;2xKoaV52c!(Tc=Nb_w_lA`LaU;Q4iM(%oL%~U
z@Fq|Qc$YAm=2b$E$O@)Kvh;tZ9%fT~3Cyi{Cj(+9djkP;jsL}gB_#}8n)|RS#Sf8g
zg^WVLp>QO!V)<+UJqo=jD3)EIiELAQL=5_Pp&5VA5G=jOd%8nCBml=MtlF#n1YR8
zp>8Gr=^*#n>px_3Y4r371f2iV=8|*SoT}84{A=@5rGqEKu0xZ-^X}(kWu|;PWP#6r
z00aa0(e<9Zu1?~MlCE5(`Qd|2@Jtnt(O~Pq>Hy4`|ZK&IbcZf&ifKl
z5BWN2_W7z9(nabF+_7bxd>uq$?umY~yP&9aYUGjECDZ^?jZt97>GoFHKC%?y`ULP2
zC4e!8l@Z-#CRVEHOnyI3bvXvJ2!~Nw|1{>UEGc4>zU~;7@n3(1l(9YuSOLY?e6md$
z?ZVT3sY0SEbd+QBy4=m_PGnY;aOK*z0T%k1Yodf(TJCBmy0ihJU9%1%ud0}_XWz&O_`bS?^$C+IR(Y|^WCF*->6=4Zdy?jzBkBDojMegI
zh^OV)M{Vfh9;AOz1k$qfZMdq8F4tUP)M3h^wTXcmtC-7fFGPcv)sKy&&v=G5QLKTA
zCEkKctv0qapRKCqlkt*HI$OvgAl?r|6%kPF(Z*UmT8_GGs^I=t#h%qI`5g)hB1VuJ
zf}4oHHu_@jHW;V-O>BPiJ^*1mM7>lq;cV$$95;}
zI@CpOI%mIIw3#6G{$?Q7l)SVxIB!Z9wZrM!@MGgh2qsJ^$2F5o`!V>mh9A8M(s&jo
zkwgG=Rh)lV&>oI$mP3n}_n_pc)ges>u}SK`QOI8mihFylB-$LcYF9@%HVYbT1KUGQ
zn(S)E4sE4z-JajzsT2G~5c!boji~b0l+MdpqX$Q&pn-UYyt(yU1^b7$V6vQ<_c`Fm
zg9VINRtyM+j}In6O;~zK@>VGAN8R>|ioQHV5BPt?eO>!F=4z+ZE`x93Abvn4&$X@$`n2{hVOIYIpP6fe`_WMnb)uiLg!6$w@dT9%X+loT(!b4v
z9CYO-BrxfCeH^>*#)n7x5Idw0r(KHm)@&ASh_9aMi0KVQ<@Esvj6&vdV0Bo|?jj*g0}OsXvH
zx)jQHPq}@fy4jUqkYV=SX+}EV*BG^4tvh8a4UD|!zA;9Hp|;A)
z^Db4zt;6(rj6U?GZF#C}UcR(lI`|YcM+2br$ppchbXKVBCQ$+Juu`9Ci>tgack$qG
z1-*jvyy*)r3l
zfS?Rf?ffPn`K?soUzibOGGr((%nZ<$7bAaD;ohh?;=DcrbqlGb%2SFptpaI|z&kV1
zl(io)K*-=z!}0D!il>mm`dxqi2$5L>Ji6|y5_DcYxrKvI0*bgW?HiPo7s8c1!K0%U
zFlH?wmi~n(OE}yI32i%XUIJ1)e*~QXYUP>^f&XX`5?=lJRry^5WJLW(Dg1VEAXJeL
zIE!k@0+^x#f;6LJYnk$`(g!K7e2G8VA=5M2a2`qc!^fx6U?J>R#X5hVxZoD;`-S{L
ze;RpxL0^&EpPyZ*BlbU}&ONo;!nSw}pU39%n
zB_l!&RiK#A`UlkrzdvIX39UWRk1JGpE}L(TWmXaT&&Fe}oEHk%pi$)Go07a=!G0b#
zFG1Fj(tsKB)-@%UL{*YLBVEX0n63)stlIy6W%jiL(jnD>(6f^a)~sTTM5_z5#h(ss
zWY^RaGf8h9v@m}~Uxc}Oe`JO?=MEzOut+-%GM2Y#HxlB=M#G9c;?DIl|2Np}ONs1g
z^fHMSd16BD;4;&6`AVW}aQCa2`>hvm!63uY^65^f?6#VKsnHHkA(b@^ljJ3=UTid2
zQ+x0b<$O^HPy3$(%JqZIx2aEUzgX-|1iW!KzKG^-oH`tq%
z<606<<_CgyG;_T&C$3P}3}g{QQ**4+!LHe+9ZiKN$cc9icX~QNq=3Ga&CYh!(v%~>
z6*t>0`cr@Sayeu<=aYK&ei3IqEF-C=o^)2i;w_>LF9qrb7;}42C~HvLMVP$aP`I*LJ7_+pfeBl>q{*F3y#gxyp3XL)YG9v8
z0+X?)@tu!K_)25M!@j+c!;W+^%p`Q`E>b8u(eHmq8qXfNIN&uJ8^QX6Yfs9f7!nud
zz^9Z%;g=qH9}T(l7(d=rA)2u{0r#eP+f58#ZVt|#*37IqfQeCNqry$W^z8bO2mOom
z7m%vit$Z4Q$HAC)*@~}17Q&xVTA`vTupolgyW=CE7$d%uxEPEtx%T$Gkjz4rZ^5>a+`3bp~pvwf@O34eGXr@%sbn@{3Xq&9FHhe@i14>e%Qn2
z%BD^wz`AQWP{>~Pt~fy%5vk-lgGn2oBiMi3lKkR%8lq=;`XTg!6Sg!i)7;=B=vqAE
zU3?{Jlg*d>EuBSuUt&S;5i_+WRn`Yi{o~nAHZX9yGIZsjc5=OUb(;)>)Sn5!hd{T=
z1Vvb}07xgloh+Vlhu=~3AY`&9i!m3p0MxOVDC@HB57v`5=lyx$P
zwB3La177%7iiL(j<>)F)$~)f7-u23coM}uSpqA9ZM*u2+Swnsnjzvd-ADKj*^kcj&
z5~{mS=F>*yN|zfd>8LQTh`QQ2N$vBBUHz8*{?FSzzJtE|5A;0&Db${Rn*6D(9Io^2
zC*Jiqp+4RU@z0-V+0jv1WVF#b%~pRBCIwRVUuJ9Bm!%4?!&C%?f<~PABEM15zxKro
zV=iCKvQL5clE~+!`TT1V*j{XsLJGS(PaJ}Uj(G=l=ue+cgo-A=Q%6$cJA~>N-|~_V
z>?e_1pLNEzOxMgUUl0=BEP$fqQL%kav^fjXfNOsf%YZ&B9i8_sv&c|2rp|vqrRv`%
zuWBF>U!}Wb_lk1{!rh2YVVJH4`Z6A(%3jzY=PY;bTd{PTE`FFCC(bXufK0-c4{)Om
zTfUamC}M`m9JkI-zJshOPRF05q%dR54&{88y*8hEfp6<+>T^(OiN7R2)x~1zs5y~`
zz$7h3#xual&wh=his26#|4n~i0V)_VJaB6v96(xZoMvVlw~gfj>MzpbbKza1sb
zq?sMM8vQ1l{d+6a7mTvhX5`abZ2N*EKJ}D46DfpVRE!|n>@>CeN$Jbe
z5j~7O@{e=+EIPZ~zg4bJd($T_N^toA{+zH3#L}tLhJc`T15D?XE!=-eOWFMMxAjZ4
z4cxgpTS?SC$9JU5A(YSXB21=z0>~;_%f>;Q>=df55NFihMgSqJ;zCY=adw
zCOy1G!ClF5!KAZFFH
zqwhkQB7t}L2-*V@jB%H2Y~uI%ze4I45As9SO3f`PgMUHTN#1yqkzctql1}w5}kh#;{Dq?A|2>Wop
zTqy3$!)G+NUj%>XVxn*0aW;y5y=o523^;>o&ssWpYf5NUu#Au!wr@sf)~KivS6Z(9CQ7i86NeSBi7)-hj_-x0sRiHuqM8>v3cGO>Wa|}PfzzY0h9MN
zB9i8orH_9ku&}MiPxC%?A*Ru7po~7u
zm-YAX3Dj6z^-)5Tg~M#2j)M9=g14h}&F|CY*U0`<6wRTQIWSWrpwA7->APzoMHGiT
z4trT%vkuyy_}J(=k!gO)3rcBQkH2EmVWzCSQh~kn|JeV$s9hT(a!A2lI20WIoG#lz
zTG`<|`U^aEFLqA;HxglS4mxB^*Jdiu{Zrax-n_yDoi(wyuMHPrL)$(P0TT*kZe(+G
za%Ev{3T19&Z(?c+Gch%nF<%1|5;!q53NK7$ZfA68G9WQHIXF3&kZl7L1u`%(F*ld-
zrv@m0cyn}|-P?7N##UoYY+H@ln2j4uoQZ8Ujcwa(Y}>ZcxG|f2>GQllzVE-^nl*FJ
z*%$A9_I2)ALqVdb!XRvHV+fS60XZ_TFf#K3WaZ>+KsGGQ49Y-LCo2Pc04pOiGdny5
zg{VEyz|q_WBxc|UmRlAb=sz%)rV7U}FMM1%3gjsfa7903?+a)D%@{7~d0r
zt2o)(+Svb3FQO`{YLfH-F=2UCaR5-A9w4cvqWb4k6$pCo-;^F8uljEPQ|I0BPq>`8
zs<5i2qBslFpL+nX0GxsL4(5N#{#S35@0|hu)!KWgiM@^WUj+cvW{!@wyi80kE-s9w
zP7aQYHuk2BwpM@jr)p;I0C2Ifw*1YQ0JHek#0mzyg0YMJH
zKS2^Ue;cjeseBK5#~uIK*gFx9e{x#=?GA7N0{^3onSsM!v9gMavH)uXbC4qtWB@XH
zcXTvxbaDWE{R?}41C6QvZV(6%b+WhrQ$y|_Ci{Pu`A6#_Ht%<)W98vx;PT&pJ7xfK
za&Z4QZvOMOjch;;<_?Yye`f>&Ow6r-f4X=0vuEa@zc4vrd1(o86;%e=cj18;??Om0cQSVgum$8-DNPAEy720QFykO7lLY#x@`;H-ItF1fEIW#_?T$5Y+#F=Ip;}
zNjO?(Dg5&0GJp-HjeKh09z+VFMx@SJ^Y`|d>1j3{2%DA6+0_{$>4n!?9Cl4{|WN2
z0GJHz-?s+nXazKJ{1?vlPyFvq{)d5unHj)j{Ldxldpyw2$-wG=Uk)tq4xoQkW6=akRIw1ZtQY
zzduI)%|*`O{SI8UnctPn@{Yg1{_#!sKMGL%YY6`>T13Rg)q{bZg9X6A_TCv68_PQb
zESz5d5o+}Jq4w8*5WVm9Kh}Rv6aWzD3N(UWUa&FZ3$#dT_*UXAo;OtrMZv>(T8{PU
zi%bmUa_UqYK8{%S4k=JD->=CxjWXXxR*F~GJHZcB_=O_yqt)w1b9}|*jj@u@u7S6l
zH$Jks@MxAgqndx3+>&o;2Pw^wOjfRD5&KM9OH2z9KyCDYSd^!&^D%;T;tlO8kyNL&
zWi1EFrC^Sw5!2oZ-E|2SuN{ABsvX?%?GZC7+@MEznRfPT?l-Ja8ED(?PE;+rq+Hc6
z$6X%DO_ycEL~|jVTI}%YZ2SdPLYd0v_yvZkgvHunnRy?<9dbj>tt^wLzRE6D>VIGw
z_9!J*iLa-BTv4i5Cj9!KLWHa(>Z1pC)DR#Dd!}PSVohJt?Q3yt$)+>KA3Vx|rb}I=
z+?k;7!=5_aqmNCG
zPChf>>4&65QwgyFo~J*7!(N81L4)6qVpwZV7)rx`<5nc}O+0paWP>b&qs2_k`%A|0
zXp#@}@TH%05{NtPk3iVhYs&;Raz4yV<**y7v?lo5&Wm5Lu7tOJo6oY>8C;&0QSE0&
z71C?q!9=i;M-EJ>WkmabNMgBFY&@xlbnFNLxyl!ewku5yhrO?+;9ZN7@
z>(&^_SX5-Q^??P$Au2SxSUs~>7$xi@WPO4C>KC=)GOP9Pap|8*rdGBOKmFt@2ns&i
zkoe6$n7?&9fl)dv1CWSL}R=^Deal*ft&5`s#>R>MVEGhZ6oN-pB$tN@djN*zx>+xW3*d;5xH-ro)@NOn{Q2Ah6Xi%{5&*o
z0XJljdCLcrR1luAFfiBA9XBFaG;L`6RzrG-b>WqA8z?aK5p7+~!c|>1t?|xevXbDH
zvCY*NL&18L`$;=%-%#ZQ!EWz+M9+S10ZdiVgwK`>kzI%DP8nVZiCh!Rg~gm+6q1J5
z6}iJu=4zIhe8~!xJAB&;$y0=X^qBgYD|6dtMCH8fqo_fGiirIohOZX2ZHr}b#co>p
zrVf)E`5)}PlJ9xu_3)Z0_b
zm>qU+ainWJ=<+;jYwM$c0_%sVqF~Lrw@~lA{zjKkhMA8J-OGWH@j`e?P~hxXQYo_3
zv$U0<3zm|=CeRiDwX;|&5$^K3e^47|l9rhma3U*iTGU|!=jpMlpvLk*8B8+?*`Um{
zxEizx++Qc5cTM9I{Or;pXxzx*ndiBb&{gc8-n1LEVeqGg
zu-4e$%7501fNeHH8xlV)>NrO23kVg@j?8=QZVKJH(=ZC0Q!m7Pah5Osf1-U0!)ZBt
zCOF3Q!ts$*^~;#GkCdK#Fv4J8Cx10wHbiqkoQ%{jLOPr0xn27oO?qJ6$PM*8ag;+E
z0(9qIW?z0s)?k*7RywCDeZ22n1;yge{?bHdC99ib$7H((1MKuUGozsAtW+>vdH&9=)K4NKQ#5bUDeh!U{=rM5YfAYC0EXh*I$fZ+?o}6NmjybViG{xfL8?~Ya7ZW>7bwKa+2%h}d
zFtUi0uR&IX$CUZXRO#AmHcQ|Qw-oC4Dof5|ImPbRN?k1o8iH&AHVsv2^Bgntui8gE
z2b}I?(mYg3J8bk%<+6P#xNND5dG<34%g$259^jVVSPXxae_5r3L6*4RQ|=hedD&;8
z0r9ofkzqNWPoU>)BfCPa-Efam7K^8wP<<7a1s4($-NOh-e;Bq_$ERCnhN{@lxEsd%
z{*Cw^@e*ujZr)ZW=nNC8;L3wK$5cs(w$`;6E|cL1BY(<;nq0dwv-E5Ltoi~NF`d8m
zt>l{IlT4-4e<${$ZotF=rlipUa^k^P|nhrg1J+sh&t0z9YgK*WpF+O0!3}lwazk`@m
zD!^WU`|UE9{r5w~i5i`^H)^Sd_w2kTlpG%Qx#2JUO)Hg1C~boP{H!iFy4ngovtS9q
z0Qq%Ff7oOC4_jDrFjqHCU#X^tNQiiu!4C0$4J8?w6QcVX4AKnM=Nm{c1rnHhek8-e
zJZ4H-u)9p?&e9?}Jz&TsoEs6jSNSDap?C}L$RM9GY`^}hq1X(D<_QIoF{AhT%+2@$
z&oK@6kku*YE6OS=5@~h5kvzaKvUkpt^%#$ff6dnasv>Mnr6=bbO{XUljw;NUj$i8)
z6#PYONH5*xfifTB5Sys!v5Dy-J%LDU-)!S@ZlNY=Og6q>2_Nf8uHLK@9;Z9g5lniare_K{t
z913MbumWbZ7DO#jvHz{$d+7Ds8~o@wyymJt38z>9I!^`LP+-gq
z&gd&DhU*x$;u7O7)Bp8s4DWQLXe>zOq(kdADv+1W0aNz-rAFq?z%zM#X3p{OjlvLd
zXK_G8vG{80LGXLv>;|-TEYhbHkLR)Cqtp-DlvA1mO!9fF9i{F@)l`WFe|K3pxY*0I
zgC>=$2mUVV6*`KZa&ge>VcLgs)P9pm2KL70K4d!t
z(y3z}+7@_R(%tN|H(<%q=945+Dp3J@c!m4481)}R`%Qd8gr|=m^J*ea=t^?xnwHM)
z>W0Ca#xB8!zX#ly^iXJ3@Tp;OA_K!@If+SpJ{$rq+`aBeiQ
z{A3$_&K%#CP2s$X8R^8bLaolikpt{X|B9O);O4H6`!`SpWq37C$1{g#*?i-9~k>0)d@zjO^-
zP}BV86+shOzH9@P2a>MM$y{gJkfl*l>c|S+wN~-P>&PQKUD#8?vp}lV2W8|G4Vfd?f79nMlUcrke#Da%wg`U^W3AtfjdBgP+cCG#
zE@$fs;>!`!e-J6``u_Raph1q+`?+E&cLp|($_qON!9Z+S&!IY|FKc&0L2t!B_Lw|7dQA=xafCr62`q7;=~HKl-wuqv
zEbC6GHUu07%7-6iB3Cl1TfV2llNJ`us(+uO2(bv$J0#c^u(I1Fmb0}OL=ed?m`Q;iGe;LQCcnawUta%
zj=NRiS|869WUs6j+Z5!8`-O`{sJGuEs5SV~2^yDKUklKCZ)MRzKpH-J=tT{0BX<0)
z(9CvIe}y7-EgPEO*Xvde{aQpR@GuY6urIH-I`opEy&o#m-1Ws5h#;oQ{Hr?L1Z+b)
zuI4sxquyk-zK-3lD~$wdzHMnDT9D
zmJMy8YcUl*yFv%{XT5Wq*(vK-gZ`1UKUN|81gcpLet&*Zg9YW)wS0f4n6F2lbC=S!_KTtL87+%L|jW(6=Bo-M5As|B1-Hu8R
zkd*Mu@~(EHT)~^k8b|8yU$9NCpDnp0Gg7d(M;~Bzg5@4lH)pov{Kuj7epx1%2H8S&~%WIssQYDk-%(%)U(I^rQ6{
z>9y4@9=?p40=Hqc*YI+6A)+#tXY9~#H@FuH4A+f;k#18+jV)xtYB>Ct<@?kg05oY1
zUok6=kw)al!TLj^Z+;hB2;ye^6ul0q~xW
zAH3tsLqrf#G;2t{l(Q4o#IKRa~ZauVv^d(en|6_o6Fn=;g_RUgNLy`$CQR{wT$#Z;hb
zjC!k~tv;I{SfvY5K~4P$e-n&dat`%_W
zKc^{$!Tt!ToW9^dZrU00^LQ}-+ZXb74p7Mw;_SeRP)ah|=SRj=x!}a-jMn1u%M7J3
z{r$x@*T}M&BJnho9mBI6`;xTf--yR=ZQC1&u-reGYQ9_E8n-Wff4`27z_BRM!iI%k
z`3~B=$@*Oza0;2hYxrB67k|baf>8b?S|pzJ>3bci76BPi3L_jaP`rqnRq_L7U*_TB
zHt*VPPKUN>wmOl~5;wLlh~Z8z9*ZS738R>DJWo;hR8We
zs_sQA<`q-_p>HIkf69ki3mm}1+bb?_MHYB3+F^;8dz`&jmfEQCaa{@3OeexPpg)R<
zsu4QRC0nXPSaCF=`*Ek?;j?$cyyr$r+)RZ+g5^ky4~4Dfr_=bar$fs)1pCn+C)wOd
zLZjNM6ea9`_ERDpPmL?BeZ2~k)C=yxt?PMUUQl^jPZsx
zIl?~w?Wd#U@r@RPaEGSL1ljHzOw%~WrV$|LoQMjg?-PqkI_74SZVs{wP34;x|4r(O
z+T|6p<8SlQf6&g`zFw=O9%h;u&Gd2T7iN%&d2wZD*4nSB;fSQ86fT3E#I+ST?{Dfz
z&s(THCO$I$Cw5>Z<5wKnm=CF3ETU&;*ptRMeIrEHf=nqR_0g8j_~Hu;4D~FO1@sd6v1(f1*aQ$LV{gu>B9^#MfoVEhWd@
z>#TEASnMla?GvaT*zVfC8>F8!N6(ce-{d{zEs1K*_j|*1VW0<(h7v52W-h3W);@ds
z0aV>*zLdL+N3|g*8i7h1oDV4;Zc9W<5ZPxM#kofCRn-G&0)mK#DQOgWNFWC6Ub$lj
zt4c*ke^j#gF-E}yQ!**xadTStZ*98a3F!0W-*~;m-|W=iT8K!j@fXI83qQ0H$lB~|kg>~Jl
z#YC&FMMDQoI26x2gwypFj8rldUR=wtsLkQvf0p;gL=%Fy0vyHBcEltH&ka@9VU(P|
z)+3|D)lrMM%3_D(r#uH{94W;dwCAH@oh~^05Hmw2nL!-tw|2hm2h~bSh~6R+BzI?0
zl-#;hl3Gq*M?nL>7N*wn9$4Z~C7Vp(SYYs3rFI`l?4Da0rCvNs7Hqm-LdwV3eXCPT
ze|a{dKUs+BzDoVAJc`p!5xQAj*B{B=rNZS&Ype*(a#hfR^ftg$_;Kgo=`^WMfiV<4o@RPG1h$TF4^#l-8+cqxaC~&~%
z9*}TH(Zacmq3|4%Sr-yJh|x+Meb7eXnBzPgCOi{l00sG?PF(RrH&t-sLA_6af0|Vi
zi=KypZ2LC|Wy1ajPV#%^f=(*vws4GlczC_;=&
zxOPV`_NgyKE}{I;G^>7J`%H^2e?}1tEZVSS>(&wkE_x^$_4G>2f8_{yMxB+Jw|qKh
zPMWVF8vrDmGD@y^)|D;Sp?2I!3f7?;#J;MSS$m}6edn%#VR6=gLQqt+E$Q1xopeiE|KKTA-F~(C|1(Y7b-LZ%>{!=IxMArk_4Xh
z!==zwKT#oHF??{YBk;AzoDWzXhy{c{6Mkk&M)`FsHW97^Yk3Mpe+(`+hn?ybd$t$;
zIBq%ma1bO%i+cq|e_bW?QFlq+4XzF9TU|36SZ&%26Xi6PW49ka*=K8#9oNZm3HTfh
zm(C79O|{DAUU%r2`G7q1pRGfTXZ#;hV5gw5AX2P8+9HJo-0IAaB+K(Sx|mW_;y3PV
z3Jlb=FOx2^g8CM@e`PXK;Zj8Wn-ipTehSRAE(qzxu2Ms0Ky@gsL*zIg+_8&~9<7&O
zNP27wW-kp@^Lr;4bcMS%QIp%N!Z?-;*yaZC53KF!GD;=4E}JpuU@GEwig+t6gP%((
z3^7(kt##uz!Rz&PcB7oYpctPv7y9~z59^(GVd>a074m*6e-^e6uNx)lQmPQ_F^tTNoTmS%%Te~O`|VMj_vTgt3H!PYVTMr9
zA+&xyGPMyQEwyE`Vf8?aeFaDc?{$zcnL5EAzRnBbe?(Wi1L(5iiG*#}=cWZK-#F1&
z2mBr9n*(Yc#F8VV4rqBDE@osdErRGkX=C`Zt8r9H0L_$&41EyHlawb7XS(*>ob@Dy
za+7_~M*1>c5m_;L?j3971qSEfLJ{dVrYE?GsjSlO@4hNJt&!9I@`H>oF6rbD&%!0$
zVc#h}e^@MtZ&FRNC>YttAR!Cxi)W=Rhg~uDPSNyhU}7UkSt@VxhSQm%K2|}+_NZGB
z-sC&+FrLEQWrY|i(9)qM=agoroG;8BlDYD+GvlRY4Mn~(M2(}-`K}$jlLd%5oU!S&WW=r#zdf6R~KR`$d5Q~3%Ml0&R50x85A2=*$$
zx|Ny{r?Y)MAY5um=9W{=U8J#avInL-bk@!c#K-=m8K&Eq+%VUPHp*%;U`}miQ+WZt
z4&rZ)hUJXb)w=E|OX_AFOCmwjnPjWSUD9n3t-CD^GStV4!Nlr>#RfzX%99QIW0hJ-9
z>evA~>xg?#2o;Q40orIHZ|rVx@Bj@z7K6WA8p*cnC!e+J9$t1(xH+E0k<8~-e~@q)
z*inApqKI6hOjyM(*fkazw_a9+3eL}Ae+7aAIF|&K)mwQ`;*k^JoDxKG{@oE7bpxm&
z#WmHB=+9}plI&AF~FPd`GQm}}O|<|Ub^
zsv^uFb-Nx{n^)Ph`IJcOjtor%Zp)ua;j$#Hn+S~*UfaQGQ6lZHfj4bbIeLn#0&28b
zx=zAlU`%@xPMd7kmZI6NW}sr2w6vRpay&3;*kBRZD(hKz>|ld8GVJec40
zEGygK!-$c*WpAos`g`(pdki;QRL0w*(ztJzZ={e?okbuncxK
z{HOF(*mk(zimF505yf=i*X}%h@)U7uM=$4@4{e=fxh1-B*$O{`Va93(uH81r2Zqd4Fe6t?a_NzA8M@~inU
zKH^ZuQs}!UPo%{aiqB<{m)+%0+Rq9nWmC;>+sC)@+zL$ZpK)@9&0Ii;KoAD?Bj5ab
z@{U+3+Dg?4X7HMQm1_|AUSmYXk)hW_4Wc)MTojPzE2&yjIAo|&*$
zZPoTjn`e?x!1+0I2WnoIMtOn`jNoCQl%eb=+Jtq`s@5956K{7Vk=dwK*|aTJA!E6S
zOtm9p;Gf2ia1*M?e;73(R+eAx9*4VU_V=F5Vyi5g{!oiOps-EyE
znI`Tc;dbH;TaY~D<{{GqdO?A!UiNBtv20lq_YU&dtOhl*Rr{p(LPC2pX#F!-=!an7
z_gUrD#?9VmK3M~ETTDY6v}$qt;K5ptjLEj`wZ?1`{L=3Ue+1%E47|KV>AWJAgRerM
zkT;6gTK4haGl0!xkdzrU-n$sey@=pL!44gWS1OWlw@ry
zsOz=C+%@ObIrV_BMcAo#P1;@xrH>k#x$|Zqbr`wb|^~hr86?HGPVx
z-R{WX7fJd2fBYEX^cx(ELOTL9W7Y@(`)b|@bBNkC^p)PTOdt84*H46YojNt2Za;--
z#xjSRBQv4GfvpN)^hLdv6jLmX3Zjt(#%ULj;WHBU3AK2Ny?$dMx|
zVd%C}iTl)zY#Su8JELtL2;>b;(A?ozEAj-_A0vFf2PBDZ&3cmRKGdaOuy-?WdfoqSi^&X
zfzqp9T)+iCJBRG8VD*)Ad-nM%Sju+7^Ge+yF$Bbpk~fn+%CDttsA0D}!mxKML`AUF
z)b?sTm5NDt*5bfd)BZq+4hNT(0SUA->5)rtiBx|lchBayLRM2n_dh-lOE5Bn5@vMU
ze@peZ3MC~&*o}qam&Xk%Qgh)D58H%slpY)0(qr{Afo~l+@a&yt`1K)wD2&;+9PcQM
z%C8mO=GAhC2lepMZ%XQl+do4>vH8+Hx0zNK0B?dizCq3y*HwB0{VKnMxT9Nvz)k^C
zh=}-l>mGN9%q>6*XGnHta_TdO1ekZ2f8C1{{{il%y~>E<5WX0l98FagExLo^>oMF@
zV6S;7Tl2&m?^OkdaH}wPSdD?EjQJ%PA2Dpni8K%#s@4yqzMcr$BBp7P-K*GcpXc=d
zn!*rDU>Jo@SO`FRe3Q=t&y4)ekXZIMs{ox1{7~kiRzyg2$Kdm)e`%da
zX_QOAz-pz_ZmD(zbOBY1AN*#2ec|yw_V^La-TtOGoVVl*$i1QPN*;Uh@H-CC*;Irm
zeNyA3M^!T0u$K~>c;z?m-VZdm#-7Sn2C`okDx5*Lu~|8UB-$oi(Y8=4D7G8+2G9bg
zKw;sxRHO4olZ8-9Gh(M4;By9If9!_wJ_Ab8(gc#qMlH(BC2T@#=vhK}$aU_M9>sD*
zo8=|EMo(q;sWEDww4oc`-sKWnV9I|7q4;JpDC1@|r@%e;tBT)lYcA
z)T}yT0Cmq(2V+KWy^k=hPM3a>W{TX!)uKyC*BY^mrN%I-HJ>&FIb35+Nmt3i@vZ+l
zXZK8u$#{R^mve^oO=$d;^z^|lc=JWuff3|-sV5@I1Tg2As(dG$_B}!)m&q3?dI-!H
zbTk%SBp{@6YTb^tbXJF|e~z!P>N>js%2;s(O%hAdXU20Q%|M*5qq(vIB@KwC(Uxih
zgZk*b{-2hoU*0U0E(+%z_cPkdeQ`Ug*nOZy*7HzslYpO)R+NSjoBg!+@D4k&9Z%OB
zY7$J!L<8NuTANr*5q7xe_rxD=Y1#MRyVc;e?;vO5De^Fi8PK1YyM^w%h
zz9R}aRlD^@E90M2FS4YzZRY3V>+W;@kP`nv%aPW7Y6}@MG7P(Lxg%*Jv(iduy)HL-
z_8eqR+(8J>{d9o~ySFST&4d33k}q<-Op5%*G-fEMea;KI@brB3DRdN$$l(WX2h@FCYBN>9*bAt7DN
z*c<*Cb1U(uawtm#6%Ie4E4XN-53`MZVUf?*iV#+k`9al&f4C+Z1Jxseq-$sc6Q%QH
zzom|ZPN8`^8ZU_%)sHogdc-r2)gv2N^<{F;2RKq
zv^+|i7+8~rC)NOuMrpe6C=`}{I(R)`;NM;9byK4p^hj7R#Wb!
zRksBdLK_6|e+BoDg0T-gy4C;K;426U=N|6nOfO3DLSCI|@eKNQB$C7zx<=B%EYs4E
znlmt=iH327wvf6A}&yFgNXCeFr7uM#}@%?P+s
zM8Ug@j&0kiAXV8^z-o8bcSS9$IPesbm2{Lm_I8K2aW%ar+;jyWz^A6f7gW?SVkNZc`~-b*4Q*JlCxzi
zL5+-3f5+&abd{9$8qhJVdojS=txRF3C5*1ZlgSie8P>;%MH&)ridqF?ZFWn8hYlTL
z537M=_QtWoYBp28_v;*%b0F%>m5CHT-^a)Ra{XmI@52qEd|n@T?K-Y^K>9
zYW302+RY}OdLvXq0c0*t?yt5q`HCSS6dO;pf0AuZjMW$S9bsmnoTUDsA^tvwF?u?i
zFfsDj!{Q8vUjTWa7qZw(!F~E~f6e1eprA{gD;7c39bGn*pcuJKg)_66_0?0hfg0=a
ze!72tv#zxM`8FFbR#HwifIH*lyYMLcewqnll_D3Rc}N=%>4PEp6y&Z}Q}OMG>lh>2
ze`cob8^%!YyPv#X^h|WK&Ssc
zcSm{Xxl8HiEKz)@e{s=iOHj)+kvcRRDK=QaV%C&6UF9+!jBt+b-h
zD3e2f++F|4T&aKc8;>#$e;M;>J-|shiBXJNAg^@oP*5M-uRTzGU!}?}67jCwx`v68
z-#=-1?}7+NJ{(|gh?dEGqDDV%K5>%D7~n61GZO-7@%4`1F5pvPoQ!sRP@V^|fYIqo
z`gdKU0JTL>HU^kkKq^3&-%hiTTWRz5YkaVN4H!RMD$w7=hSn&Pe|W(X0+GH==a!gt
z>p|zq<%zNfX^26zL!z)RoYNgHOB{M_V;>*hT&i?w;^&R6KBIR!25e9g4dTFfX``8s
z_UERZ$l0AXG!2e5i7lZJ6h?(C@r6s(H(s3$3}M=(AmCKux+au27_A`j*jQOwvSY-K
zg7%b8P@yH|smXdne>yu(_fOwEv+;{#g9>|FUiIXr)TH$6U#@DGJ+DU`*>{d;AuNfB
zGt~`L$Eu+4Xb!vM6*miLup}S&pv`ICq%0lM>vZwCh-Fi3jrrTUpW5i3{c0H?=yoh6|bHhE3;
zy6nwd=XS7#e?HhZbSI9}qACk(b-d&m2@`Ik>_6LXt%GAr7)_Vl)_OXlC!EFp08#bB
zv_Qt0u{JJ+R|zv??|Z)#Z`AShKttxT4-LR
z=<&s+=Y80v7P5}+I%l;M6NMvZ7-v-f7M7f-T8Oa#%
zjb#dMEHS~lr$%IEJS7Vb6ntLb20~ZX*HA`CG>z3v1fgM0gK1i(WaO=oOku9nYSXNe
zhZL@igv{cagE48X{aKyXh)4r!rK_DJ6TzZJ&j!COJxBFCnrJ~?$#6N*!;3?BYNwU4DbodOH+zMm`T2~^QphMBZ0-46Ls}+
ze;QJPw8t1Mg!)Rf3N4R9OmAs}bgSv5qfoodZ|194h5EeNZByzJ1bdT>+o(nhj7fi6
z<>KUwet2mXeRC5@x99m>^X~e>bWn_$B3WU_4fL?TC-ESxAZL8DpX_CziwM
zJ#yGM;z|uc*hF;gOBi^p2qnEVy)7AJaVBGiIE!A2gL&6
zsZFR~ZMPgTZ_v78FV`FH7IJvf0Xkrdksy+WS$hEN4TSh2wSOxB${+po9+CXke-B@!
zhtEy#up3${hFwho$9AR*;QmV6$Dy~*d_YTllOr!>kC$vF2EL=y>xuHXnyzTPvuAc2
zZH4ZgAOSYB^Wx==+9C$PWo3JoV0*2>z?EjQbs8Qp=fP|)zVTreXV9l&TpGZ%!B|MFH-J=#7^$(wpsn<;}(TjhP;bH9>uNrJlNWg_sr1}(n&f0puMmS+|^
zED-xyfd3YZDW;b~f0-V#UmW_r312!>8=EVT%%3{$p13(8O2_3^V%!tzS6_obcQ^8f
z=2=-&s?td;_YoPa{n2%rI9T;x2xX!WtyB#6({ljZ$E;IE<6lmI$$X5h2R$O2u^7-o
z=o{^5B6($_znnJw5%mg@e~`nK5K1L%&alXrV{nTvCa8C8@jg!_wv5_)KTGv9fVaaQ
z#c>Al9Ss;l7lNU)+pVW@<#B$uN1AZ@f<(H
zHE4#(#`A4~jO{HT`iF^y$MGGQ;c0uwob`(sg`KCHUukkz2lr8TM*7&JA5cVKEPeAF`=OCgV4T~%+l#?`cYnNKgt;gZ%LT^s_k8D1UI{~^
zbt;SYR7f-~0j$K=e~5<3oY=2(`i{TsfrWZ?Sq$S1N}vk#OkXinI~Oopdn`SLM^%I)x?we|
z>}~vK-E1w6amo?QtBy3$I-8Y^3E_;EuDZRB!(~PPCrDXcQaz-+gP2f8kI(TTug?>w
zTBFFr;DX+HF+;J8Gv8O_r-rahT)r>OOETt{LHwGPMW7ofk8Lz3;hkLJ3$WG=P(M4A
zG7os^15dDef3thjuZaYPqoUkmZ4I+(-f!^T)YiZC4SA_5nB~Sx`VX?|9X*aMM%xSvyoXiGSe+-*}5ABrQF5>zZk>d#<|~=DXll^
ze_-w9>!A=u#Ep`>k<1brfModo3!~np$r0Zy1h1cme=Y5LMTFnKBeK$rK>2_H9(1JG$}HFoD1vx=PjC%PgFHhY&S{Va|WP;>T1%289hJYE@JK
zh!*hCPU?3edq0PFUCYLf&i(wr;x*uNk9L1bD^tBPiE)C}Mw333f3i|vnEv~$mX#Qu
zv!=ZoF{8G}yv3?FBrLMNEMDclvz#yw%l*Fi*$(5U
ze?w5@nb9pZ(elLC0ET|khsCRb2l#%Mr;j}0cYs8TLjlr37An{ng3-!$r2(?W8^+Vw
z`Ol4J44ZP+&8!yM(Blu^hpHUSPN;wDA+|s62oPTbPA;=3^U!TJn`iH7(~x0u@3d3M
z_Ji}w4A>Q_+_}EVgyFF2d>xGq)Gm*fBl*N#%2N
zFDB;>YGzC7!`OU(m}=Ak2A6ny@h#nXLpD`M(YccqMIagP=22b`UP8o9J#g(
z68?q&>#5gr=)gTRVYw4_oRxfD$KTQcL;
z3$VoWB_B}{OJgo{2Njtn
zd>%*(_O;b1N|JIw`BQ_AY7h;3FhfsT=Q3ABWQf1hd2Mqmwv@Hv6E}!7e|&T@a5sIo
zMwoh1)|+o=b6+RYyp5xmC9a`dd{pSgW}ae2uU~oJS-H=kQ@Q|^U_UuW`A5{Jv#%~8
zm~Iq>%eX!*QjulIo%JXevD$^3Ox|O6HC);HGKut;Jq!UX7D~2YmoS1StL^yU(Ah6>
zemZqu5UpbTB!326;y%d{e?5Q30oa!EieP;>?KTu(<=|g5lObT&Y`TaXol{R4%$@s9
z&My1L2l-^pdNIhG{PR?!LwHp?xQ?}YV*bGIa0YZPP`wE*8AJ|-k*(iM2_WD3PI}1h
z%JPdijHb1{*>DrnfzVD=$o~U+R
zgD;6}BH*bbP|xBh^>*INdsj
z8#+>xjBR!!;oU1Gf4{|GZh$ONp#p_IR;d~6q1c>FZ11xAuB($`9WXq3izwmcL_#Nl0U4si)+Bamme>hc|v6O}!C_bqc<5!WnmLV2gy|nN!SJSv;t+?GcxhN14`1*I1V(Gnbz@wo4zPE
z3{8fS9*o6bnjp(x@rQzJ|dnF%o6|ut*d^
z|Grc#v63age-|;HMN#|eiT0JgX|A`b%3f7=Ddu202LDGGzQJ;pdA`>*uWQzQtlH^S
zm9hK=GwKjjX2si7GEenVmc(e=jGUJ`bXH#IkLyVEE|00}-FV~ym;;<2`$shOSYRO#
zHZkac1m-;b0Ze7qUblIb1a2|%^F!n1s*FWv>~vxSfB1xfmiv{FRa^KuCJ0m%tcz@Y
z9@2hZKlxHuxL>E7{<5w1VQ7=O{&HzeQWA^mO5`Hna`YtR;A!2=`?`1Jsc2*R2a#Aa
z|6c$@4ZQL=yF?p|{RTC~dQWy)0ruQ3kU;2NBa=mPVFoq8#e81o7srZW?uj#BE}qav
z1~N<;e~G{5*G&u5TCH4}}6e05ooXvThBhqx=r^Y1eh*U5aUx>+FhaNA4e@!cWs7+xhND^1~fSF#`
zwK*Wu4Tle$rY1ZfTX|#V?2&uM{wE&h!au)Le=6SQswKAXkLJ?c80$;|#=p+P=N=&D
z*@8K5l1P2Tco@kgRkivexZ>9)>)Z!>8;t&X{~2Ke8iAk^&=5wLx-f-JZ4C(ehX3rv+5hAF*C5-hBkwDo7X
z93h4_@AhudNZ#HjiqW$Q+^IN!%Cc=K4B^tEp9&R&g({%RdbB+q;CeFAiPbz1m_nUpgtK@5{fT1UABF4wpD7`eF~(`WS9nWU{>=)b?`lf9V6L
zb2{$$+e4WU{e(MRH16a0UfB-n-UAP&-~{vGW0|~m|mbQULt1vK%ZjPq?UY*Oseab
z{HSer3yD-?Yql%u+i
z`s!TM)Qqh?Qw&@t+ztL^`&EB6kU}&O{*$=X{*!e}W6g&Um#am2(TTNWP5FLP
zt4SE@COTw%UsDltCM}}drhxe*3yfml(pQ;Ig#D95XK=&abTRo=rSo%7X)(Qidwh`C
zIswQGei>fv_R2!|vo05Af9Dflc-CdJr>8@&;lK;GfryhTLV5&>h=8%CU$};%`Ip|G
zQ|syCFmjhJ1^rLk>MV3=eG=~)$BQ)OW@j#Opry4wYnxOa8U%pp-=R8GNC=qGf4%W4
zJo0^0NQq?If(Svj6vRJT&5zwsjVGa`&YM^tt4Xs=E;W8~x6$V!RAZod4|JMv#TP
z!=yi+VX)w_AOTQ7f1hz2vzP3QR3*kAXyfVG71!mBcYRY8Y_-_|1!W&R)XR|R)cTh5
zj$b>GWpB>vRA!S)dnBsR2_xh(vBC_~g6od#DS|UJC*Mn}0eC@jxEf7OlHOdTsyJSGctCh@Nl
zV&Pl&M32+pb{X)!6}D!3h$Xjlv2He-)A+Dr0{@N8X^U1=
zM^Q9>nxj$=e`ya*1+)ojF+4^gnxSqJ?acWKOSz{UtO70GHlfok!9MpPy=8A#QqSH+
z0s96|JW>?IFMVxA)sUqKeY9^k7Cbcmp8$3B60CE1jFxEQj!To!bZYy7wzN=e0EWjf
z*$|WTF0fTg=q951T9|5ue8tCObWx$zX->cYy~S>Be;)VyHO2G?pC?%w7XzO^Y
zJuC%;h{B%XrSuXnu*X=40Ud-y2jD>toXX|%)U)*r6E|eI(*+ynN5M2Xdijqgu50OT
zA#j}t$hu<~-_(_(qg?d{QVnmM9Jkf86_#g9YvC(aspjrqzcYz{0hxUumv!f{H^;W-
zMj(?ee{V%&o4jw}A61Xk5up|A`xUeRlH3h(xWXazWaGI!;fq|)lZPoGOlbe_%E_1P
zC~q*W0!8poY>WvH%X|V_tUPSClNBJXE^*!
zE_ymdIMLB-#THZgqHLOBW36PIGSCm1wFKQ30{hG$EXV|Jh>Rd(;MTQpWmmW{lS;mA
z5j}s*HnB<2X^7nDUBXg#<9=e$oMJ|+qtkPHaxBJvv6C~7XcJJ8Ah43n1@TvgH)unN
zf5Ip$!<<(e)KFLb2?$p(XJZYBY>NO*>*CS&AO;4I`g|9_
zY^@FeZa|U0B8nrwy;X6S2m4&P2}v=_Sl42zSbrM=_g$q(Lzc4ae;5X-6MeK7*QTzfDFxU3kJDT_FWMK4#TyNueh@d;
zUr&696pmByO-!LvW%1&;LSh*(?>$znQ9LcyVIa}NktLFvL3vu!6XRuG`5|jaz6~JtD4uaGmG$ThHI`PE&lJ6)FK&4}G3#}9
z@wFE_(8jPk#%AGJBlRE+I@h6TtlT)IeBRaII0Cz
zZ)wK{WO2*@({MdV+f$4cg%b>m9(^|1VT_p~4Ltqau;}#d3O!eLqhN#e?R5X&@S{U1kCL0hcB?|5glu$0m
zZPMy#(3|e~Jr!x2pR^hqX>h}c0FQG}ls*^;It`iINtr2xyjN>4Ka|zNSbzFL8$wxk
zw?lZ%scbxOG9`U92KOJj`z#)Ws{?QOyllChfw*4@=9a*HRXhK3ja;jg37&^feOUZ|
zqPDXC%B&_DsK6%L8Ro!qj5c{+Gm{_poNoUn?(Z+9UA`Oo(TilyD}S%?WApax>sC+(UYTaf{+}{(WLxL^?(4ICGV3BFt^~_<
zI}$-ngbmqV@{7C3kI^tXy97beto}=8ljhnFwM?|1BUOgfzysVN?LA6>1e;E81;5769;A4}ZM~yfRf|fo)c!9*^87$eT4s&rKNsf>|r@>#}llZ-G(JwG`pV
zRBd53O732xG6U>M68{Mr79Yn@x7!!EnE#4kMq~fzXxn3ShV`Yc76Og(Pk*1C-pbEeA-Wpp(TbEs
zX(~76=pOy4@frGAe};O;PreR*VZ89DnO8cPyFXjP$8ZdYl!xNDt5p-$RSeuuEo!3L
zi0gpf?l3vmRBT<#m5r+XFYpEie8HuXKQ+Rv-x)AZZ?m*lg>SJdb8A!?mAq*;T;k#@
z1L!fz1pwzr#zkD*f)yXf17w(n{%EqhIis8BpJvs+#E5|g=#tyiL$mqKQ_O40Wt$hgB)C$*qB-Q0g6gW
zAO{d1(;J|bo1LjMfSs9@l@pPQO2Qdv>S_aWkTi7#@&kBWtpO_Lf3Dz_;0RV$K13>j
z4A24S40c)o%)9_fKvz=@FGnC7fY$UcpbBzvWim5$0lR??RyGbmI`9$+kfWEgjg__Q
zuN)jqOusVyS}o2DkT*5A1$nsG+5k))ECBM%O3VOdkO$ak1E2*t0L*~argoM9kR?C^
zr~}YcmwKZPka?q`f2pcY#|%!a?&jzSa{gZ}66zY7GK>I8F=Y)Y08on&Afu_S@$07s
z&;eY(6(c}d1ML6B0}lMPTuDkpOhZ>yijC#h9sq0rcc8P2%`e*j)J6@i4Dh=&@KQ@>
zko{i-09tETS4Vyp77q^(W-B)rS7wm26|3&?Z^uj>++4i>(B|LUHU~Ml*tod5{GAa9u(YuQ{;J;PSI=x5{<0~F
zDa%PqscSGPf(OrmNeKj2$AQ_^)Ag_DU-`r&6~Uv(!wX<%X9uu?hg8bJLIPxO53bq;
z@t1s(Hej7xLC#(*{~c>v2atz@&;QG8Y2#pF`Ac^Te>X=KO$Qq%H=vy4|AN6L#NRe6
zpeukC0CWNXJJ9|BI=cb=
zeExL&8zHjs04!|GUBP1xJ}QWRWtVfX1OfQ|HiIkqKiB^TfcCFrN(Vlp79a;ZFMtKm
z5|Kq2f8+`t2ipICPT7CBq}}Z7luhk{wEt1`zhS2KHg;Zr2L2~N8~CeIT4j*4y{X-Q
zd^Rr9Hl9EWRU21x>%TPo+brj53Lbkg2P-=uxFdg=G=H5WJMhSYA3HX`o>~AVHm?8r
zz|&}M>i~3d0dVvDZ_vjx!euPFhrFgt);!Ak&+Zmxa+OOP|-uTFAv0a(O-nf^xH
z02cAzhzGzT@f-01SR{WVJ^+i)9Se*zY&;S
zfAKd0vn%~ZV0Pty5HFZd^)~|Zz4?v6eCodum`~$30yAj+Mqmc*-w4d0^BaNpr~41$
z14o+vM&O*LF6K5i<~GjeZubA!+1R=N=Ww;LvjF~M=Vbq7b_F`yxY+(q0;V(jjldyh
z;L&%nGj*~4BO!P%X3nPO;D@xO>mO#WfB%{PK3V@HVf))`3v~TgJRiqD_WvvZZ_@l9
z#09R%9ApQc*MAD<{AIDX|GjlKRvw-UZkEosLA5yY{1+n~{1sq{%e1cfBU^{wqJ?T(t(B9_X
zqs8?r2I&4rL%6_6Tx>jlmk8d;e~tqinEy8|7gzyTYiHmeQvp`W)dTd$0&otu-{TJE
z{Og+IVh(ctQ*Ll`-T#2#f<68?Y~ZAxe?Ty+*B=nvD(~MlgEM&po&T=jf8S4Y)n6Bx
zzuwZUzaQ`ayW#zX)m@!Iwm@wg3-Imsj}RqO@S*k8X9d4v*}!)2*Z=)y_%8yg-?xK5
zR*Q>+Jbjor!BR1CfVas5?ko6!a{K*jsrlb`t-s#x;MdUq;a?XW01)U2G)MgM8DuUH
z`ZldKyux3qXtolDijVn6e>JYKj(h_2m(1C2B7DjGEpnhpaZr0;7IiU5QI_A(KPAYa
zREH`Q$L@K(Be`bg(n3vi+tgplp9ob->|>r5vu1FX(sE$s2XeXt`Mg5iGR{v~oe7kDGbN`C=ct{-;1utkXERnbJ*vT-hNg!PP!1aVQ3t8y{2uvHhxFz|04Z9eAB%(
z`|O)`j-{07S(n<;EAy^8oI+LQLyM9KCv$ka7Tw-ZqOx$q4=R~E1-r4ewx1XJk)WO5
z0qn-^OB@1uf$xy0f49B-OL?87k_C&VBxlT;4T%C^Lvz-jDf0xWD!XOA5*JKbnA}RE
zoaaUu?oN^851O^=P4FSnC_DT_*sL>*>G43VgS+E{VI{P9QPZtgR%clAQEJ8#wy0aY
zD4@@?q=YjbU6}Fl`%e5kE%jjnU74$S#e;R^1>H2`$X@Y5V%
z;azv%tGx#1FSkRF0h5cNXBfBfz%S2}hBUai?D1E7e`ZVWsTdc5EBjSP&$Bj38+5jL
z4BI!E8a@0Hx)e${TuVH4u}1T@Z^xd4oXtOL%t=}l3wY)5HI%VbelL>3Je0j&Bx8n}^wiA9>byI?csNlPyqGdv2wmGmtBlX2G>@!|AWSMpR
zb!YIWf2&nm_@FsAl8b%IefqfYw5@~Z8oaaRNS`8?L>c(z0a^pdR03=C2qknrk?6{5
zd@iIct3Kb@uLwwd7!RUFBHtkpx55NA&W&IM@F6ku455>P<4M+x+Z;Czbyo3qM{o{*
z)Lp`HK`7asq(C}KM4K&>_A5y~BNcXMg>Ol`e{Z_VC(}19_~hLgBivnDTvqw(Cf6=(uKp_na86u!Y^EqKe
zV5uf|`t^YABXO`A0ha=suhBFk;Ji9ar>Mae3Y|n}qIVElsuqTa??`Yo&8s#b
z+Gs&4_fW%DuXwY-m+${(+aFa>3kPo^i;Sae;eeA6V0A=HYPDbPgg$@eI2~g=;u(BqvhbuLT;03
zc_uOxVDRMq8&D}igN=6s8-t#q&NJa#4yTrI)JtyaRr{dP_89a(+<{1sc{7Ld5&||{
zEPyz3E|(7l_9rARdx}~r&`VaWnl7S^XLQ5ykG2C$^voNu*L-C!o9Ss-e=#(nxP8}1
zo4y?_%j-r+#N<+Je5E*|^w^S6-7z(el6W>>-Y2|&R|W|g09(vyKQk`PCq{h?%6l*#
z9S!9lN*z53Th+k2V%I*CXz+vj?AB?=Gq1K#9tHnx$In{C??z8LW;YY1WGmU!{*r4W
z3}*%Qif+3#D~cs8vPPoxf82@-oAC--AkB
zG~!;QAd-?-*^>G6m7<3dug;_O7n*|mC3PkaXCp!
zMUw^vzmc+wY-iCRSRU2B`ar$rX-oS9L#83b)A0aWbFxScS75u
za5NGW61dIQ^m^iRTN#Zj^HJ;TT15ogu*y*QU6WhXrvtv%4F&g0nc5pY$;FTTu%E?O
zQR-TcS?N4>cWFl(;K&%bMKUKhV9Q;g+*^37ED;&z|ZfLViMJgB;1YfRa
zYb8c_EYQ}l(?ZSQ2c;22OE+dM{f5TpvSlDf)fIdLTCLzDMFL{%9qiiE1FflH92+Ai
zjNT+QRh68-ISbw$5qy{lSxQ+BEGE9F>UyOyFrG4f(T-7Crrs~E4xcJFrki8o`V$Uq
z#}9W3e-lS&|H1<*hzjv-@2T%pvfpdF=lvo^>e!KIk3QVplqEE74n&!V4nuFg?H*Z&
z>JV9&>J(#X(FR72Re3p2yT%q5tVfjL$TsQQx0!p}
ze;zc^Y*aMcwyk3Ue6y;WUX8OaN)-#0Y1l5jWz)R}>6D2HUwmgdlMMJ#E+#qc+xXoD
zlkLRFQ`1OUcg+N!ixWJQfzYp39;M`2c}d~!a6;|78%N@C@pz>vRUmQBWLv@r&4-TdbAoV%LpG+P5u$6nTK^OK1D^w)FM_oH{W*xo1=iK#
zqcocHJ5>9l*E40HL&<0pQ9=QXuSExMM7^$3AfJ-c-*H>@>^XItNvQ>$HMCzdf4-WN
zV`NQ;*Tru1QQ4WRVw&bjcej&g4HJt0paJlu6^AxbYygd)Uwom(ty^jjAMuLPNOVNj
zn)b%)hgEDhZBei#Xryh;>#pBfqwB-AuGtr*V-KCET+Gd0x=50~8e~mA-&o2X+p=3)I8pvR@0mQ?FPB_ds@Xldu(J3wM5(n!!06@fStP$7W-54EmT_XpG%Fe-_>{>Y
zjp&5^O@!R12bDHxj#E=7Y*ab!$>vaBA3sN+H
zkTvt1xw%k1L1LQS!>zr``SgE(3zi2s-Aasq%gQ8>z(B
z+p4fLo~tFYqrFVC*B6c)e-mQNn|e|?)#vw>Z+xVzvNTU4*sOvZ0^G$t_1@25!CDWQ
zsmzu0-#j+Zo)iZcMjJB%(KRq2%k8lE2Uj&KcA}z8hZw7;srGb^TkW-1S@cw!0+zR?
z+3081W2zU0QsV&)9T2pGF7uMHV!Jn%IHVNuz-7NO?F@X5A{irl@qDV@a?+#TN)Cp5&UCS((bf=E_IvEgtKz3sGo2yn_
zZQ&frs<%At-v;hVaEr@Eg-|{dHlE#npXJCZ$hFZ!zuqqLGj3gjK(y35IT6daHjFuX
zydroW&zf!f7$8Ba;J+aV9-MprZ=
z^aQ7pugH%dYms}tj5AA1y|wwlnf7T}f#5o>*NN%@DiPYUR-r!m=eur`mem}BR0CH!
zsR&lNbR4N0_-|;mCHUm!#&2|JWrx_s#mK2$pbt0(pu8j9fB6Ft&)1NrMbD?7lsxss
zq`Bh8b3xk=SXH2oYug`p`ew#i`y|J2K2@*I(6IJHy7bGeWqKffD#Vl9#TyginS#=y
z9PPlZc=yD)7lcEt_4+hQ6G-e5v9^0wmA*%TX6px)u%I01pYiUx6~$9XD_+mgBRO6X
zxxdVM>@R7p!oxS@D0M^UEHb`C`w;MsBnrH&KOJ^$7
zOH4{s5P5G^H;Kg2z<0x3++8int?#KFyRlQkHAGWVedC-+lQF43PHY!oE45do;v*fO
z(2UQ((MwYL*p6ey7%v%i}39MAx3ETO+&l-sPa
z{9b=Zznlo|;)HjIiLD&e*jxcW|8D@Q2PZ8^VC
zEgh4&duZ22@dhRbm_*DrvPQ^kk
zf2dfjWw&wrR#yhW(Fq!BMZ-Q0x(+Px5!Uqe)c#!pE0ae{cy;VfYdLU{=YApO`lrS`
zpR>nDiFNa5FIdU(V=7L*=xnkQCt-9=!wJ-Mdo*Y71q%t$ltfA71?J$ac<|k~EN==d
zpNBkHv-iUq2NzaIvwO`veGEpeUqe5ne+%clzkYzFrrZd8Rbg1bL(2SJYtn}4rj4}r
zE+D)L<3s!R)fk#XsNgpNhp9l5yg)g8CdmLtqBP8VLV)Z|Csbh#g3y9A)ki6RMT)`<
z(iEJs;Vc@?2!Se7Sk2kIw#Zl{RKZa@Qs|y*Ezy0<0`oe_B?2
zi09kb-EkEATUah%J1vCVBG<0-X{|~I8-+0Ssayv0
zJ1JGA6lAO>;B|Qf=q@u?mY$vGl!QDllBSDVpYhzPYg<
zS6axwUDnA~+PUFEaG+u1t-S7wJIAkQ*SOFk)YfvoFnbUXB*La-kWTL-N6)-JeH%UY
zY9-UAHb-3Z>2WJwmro!vPbQOzhrF&}LubBkcVuu-&W6z{0YbEEx&iqbecc&$fu*$tsUnW}Yx^K_%`dUcG4L{Ov&b;Dz{6YP7H5ik
zP0r-w>Yy(TsE(HF-mE?sf~;gkgP-88YLkeSXw5RwGIpEaYg(nK21&nopDfa1Ut!X
zV3bb2KLepWy#T57S!iN1+yC66t)NJKqh`5rtyfYY>bq*m)!|@mf9e!e*~4un{gkv;
z^a<(&3mmcu>HQv${zw0wkqD8;-yX26!bccc~L4mVwN0T6*z5W+7s+CuvP)--0Gvs1YE{k|h>~^jm~C?@DQc
z0Rf{Gkwa<@%gt?~f9!AqUIt>hS&)R?@)$}K$OTnI`}wM(2*pMhJJor67sDm7jOyKD3DmGFnzT5LJB>RP
zHikET7MEDnn9#gsjG&0=kVg&-P^bvDirg%{Ra0H}i0rd+KpyP#h^o~3wHm4b-na%Ni
z-tnJ+tTv;w1<@IN4Z82@36JvEty@lnOgl#=D{6x7ldE!t4&X%xv+*!Dwb@AE5@L{T
zzO3J!-@YNH!=&O=M*Y?-N+fO~2(faEokYezo
zm8HTpxtTx!arx4B-H^1mtNB;O6k%J!0Xz(5`riq~n)ihfYMf|#t(eM&FW%Ls-g8AzFX&}Sws%`j1d{h6FZ6oKb}!{ynAgqfhb&5lGhpdb4!eN^VSa?HL>JQ$VZ#6!igid)MX`Eo)5j-4h63V(?V(m#SD}$%@b+V6T{?+
zVIt&oKbts}M7<{scL$iR!fE*O%zwP7?3zC5f2%TsV62cvJLEZlNTr_!7DufKKa9xL
zs72yFHKLG1|(@Rxs&)e+#%D8AZY?*|2Qewy1)j1c!n`L
zf4n)X67gfEM7Gk-+xsQ8Ug}0Uq37=^)e2CN90^^e6h$xJM6DPtq}&{oB_^Uv$2pG`
zd};6%m@KZh>Q|?tnDYphW|8ktX1#(PJI?b`_F)b&TQs4?)zkX35^
zu2E-3+C#5Ghh)I#FvsOV3Q5=BF7lI&kKV$|yID3<4dt$0dXms=5+Oe_Qxr
zTI3&G&-gUjgN@&JKWhLNh^}PU`%B6l6T{mAo?TfgEVRC{`Mb;`=Jj8#95@hqXPZ#0
z!;r`Ii`F%~w=syd2pN9QLemCYB2sZ{
z_yrae6d^~Ej|a3TE2KdNU|~j!f0a`1U3HxM{Vi0#>$eglM*+Q*@IH}oyO}58fF2v|
z8xNE!KfZ5d?gT4qdT+7;tDAxA7}tB^hR3FxR`93KE&jNEC>mkNJD3@<;`hAhehvK}
z*%NoG0TY+V`cY1EK6coPjrn9aN;C>?{1rA9BTY^ZOIzy{>q2c`s_TM;e?E^%FL|GQ
zw}vd@x>|Up7U|Txe!r}p8q=Pp^{tmxtKW6xv5|W>n^&zhrr)H9pjVcyLpVQEL%VO%
zu_am+{~VH437ThSs1lBATYwDG%cZG)NSm|njo5Ha0k0e8{ujfInmdXc^YV+m-0H=_
zTC)Akoi$u1ohiJtPYcSBfBMFc^m$!eJ#5Yl;w;-q@|j$T?xIw-Y?#P580(Z1ErMoB
zt&A#Rs-*2>c9}#y7vIM3?P1=13pn(;j6>X(^w#XiL~pJOInCqC7!c%1d@KVDI#NSdU)iZa
zF9zoKJon-rb4cd7_KOBIZ=nuWA^Q8i4)KGIH1+%;{KZoN4o-wjP*7=$eY=7oR`0?4
zfRw+JCbXjk_rc7WTZnE<7$vkAH)99jh-{>>@o0XjNO@x+e=09gR~hl?t#*kpXl
z^k;TD;%kgzfm72@QHquFOI`Kn5z?q8`|qM&K5OvU@h;;_$jcstAT&(KZsq%}nE}FU
z16|zsC*Tt|JGmh0_QUJk!*ZC^czza$`hB?`qKJ0tzfgNk?nH1MXTyU4dGfUUpe_XCBh<-m@jLNQlz7QnGGGAQv
zJ=kZgG-_~A)pKAfdQQ`1TCyl5sK-`Q4TwN&CeU(B!s6{-P0cgfPRB2%54y=xx{Z~*_^9^;)RZg%7a1RAdVbf3v%$Vdx}{!
zlzgLae_A1Ps`(vR7L$L_khKw}C<0zJj$_ue{hldQGkkmNN18|k>uyzHIbzNS$~nAa
zFQ(#mKDR4$Gb*)XvTTI=xGn4rhaZ+wlzmd24&DuPkU{Ji*Je{`%Q^8FI|MA9S=FHa
zfBdh4wF&F}!!tBP3OA#^7vj_D6Lh_NlXTXTe{;i)O!7^pN2y1Ad>z^|iU5-HY&`2i
z7){HuzP~EWjUYlJP40~Iw;x_bKZsn1tBgGwhqPoft~RoYb8@J^vOGaPVnKY28ulUW
z)O8BI+SFlWYDAF9RSQyCAa;qpOXPZtzNs&Jgd5m>kWb?{a!)ULYPL3@TdEq)__&2H
zf6+DRt=1Lia~s!f)w`UyZDb>QAe;1U>WL=liwU{#<5${CT2+!QS42-zmMUmZf(4hb
zP~^8A)VZ&pge=%8NMxFv^8)e$&zU`5(KF_{cL%PjHK{H0{+L&qIe_HXqi8iHXS_^z
z4zqM6WB%|cueDIhsIl)pVaG-ix-je1e=lv2vZ+6J-TJBuE+kq}rraW0%R5RF?eb^&
zSzS}X&4f12A|ZW_MGKN_l+=kIAE$q%EyaRZx8m6--ZcsYY9T>JH%!O;xq4qKRwvLu
zA_gXLob!c#xK&r;#}-NEj=&zQNx&0DcgDQRM`mF7I-CD?Hz$Ss@pEi1v|}e?e_(gn
zULf;`Wl~Rf@t%dt?Rl&z;bnS&^Q*P#Bs^%^j~b^dxZMRXvb^3sH_kydw}KpjC8yR-
zlL43I>t4L&6fkXY9a(4`5)gI|-%$A2LoIJQIjae$*qr$N-XFfHC5DVQf;0H6xO5`N
zz!v)2jHIm^qviEu@?P;swZ{}kf3)`!Q#Z4MbK!DI2~D^fPDs*_X&NGT${4Hi#(f|!
zM9aJz6{W8Mem1Or3bhYq;k&Q+!-mY8yQ@<5oM+FbyC&+8{(eoyLdzGzeoIO^zIBFh
z&MR3?bK2TzT&0-|qS`>to1UMfe*_=-00I$z(d!NMu6Jh>Y>0vbUJV$$6R=jYT_Z5a+u08#OqI
zyT4Cu2EuP!T4Z{l7i)a6&-MZkP3JVRR+hNAZ8cilewvxSGU9hAK9*PF(Pt80oohq!
zHhR^3eGk$8?|e?~`ufY)W2=lE-Q{(hwWqTT$53ELh`{Hn)&yOD`+q4n-6
z9RXhp(}xH_R_Vjs7zFLarU_dgKBs2@Z@c^7GgUpiVnbD%)%3}2c-6vdOqyYxe03X(
zLizB~3IZZgNuV~)J0wPN1xIM~dp)O#_eAWZErsZ
zV-a#6<#_#&&j!+$zHx8~L>4^{lVM_>`N2@gFuaCw7ud4krP0UFuqh7Y&@
zH3ROXG}0Q^Ld)D4m&ZHd3m`||A4J(f76%TqbQ19{QU;Q(ZgNbVD)hq^1?4k#5yZ|SLF(1KC^9FiQVVT
zhD#W-F?+kj=V^g9knx}P>DWaCDsGPb__DVj&Rv2MdjhDW8_6RPi&m1I3&~T)d*~TP
ze0*xw;1zqc_FbSvIcw=@LMoc4^{;Z;(K#+JYYTcaZs8Ezf0s1EzRDNIPzVqFgcjt%UEu{Uf&~#eay6+}{
z^(wc)H;ArJB=Qi`&F@V=+QmD`{8aS1uY;$#Fb)}ve$~rR_5q4|{X0vkeLH2@CgDOX
z9vk&C@$sr2e_f+3*#_bhjr*}=#vBPg_M_LeTTJ+gGfe6{I)pHdKC&E$vzi&UUIV*c
zMif%CLF=VdY8-{mhqXQ|3tX$afljkp0ssg=vK>Dtp(;U(&IwKM*H7W?tn9~fR3`&Vnl6&->2{Ae01QOx*?K=_lDcOw
z-*X!Ce^}LiKMz9M$mtN&uuLUv#4LY<(!>vMgqD4~^`ED*YLHo^JZ@sOZrq%m%p6F2
z<<%Z6%!oKu0FV~Bu#(;-BjMdCghUXFkdtS6GVnfaCzYGO{U>#a*_I(CJTa2KYe0f2
z6@bYm*E}EVsKJIduY*J2?a>LzrjcwU1g}MWe|d=qtF-ymy+ZaSYj@<#{3VQj?*8d8
zjLSQ37HBLa*nAq$21(-=d`jPEGtn{nI@7ZG2Rm6=r?4ffh}80U;VV)EqQj4IpDaHD
zX`>`dxNumYaoTGrHV}g+Wl3usE7wt}-6w`Dx0b)OydXfC7BjEE$$G^d-hYu9voff7
ze^7XRLu{~YCP313)xohnI>e~Iv=o81LNna`x`lH6IL7s_1UvIX+
zBS%b<2?44k?9DLWH4;u0Ss=&He5~imf1AupbxqyHLNa42k67|J@prxJ!02~%cVr=K
z%)5ePZntkOK-h`>?pnM~HMC(LN$ls(bHf3Zhh
zvy)-j4FWUWpPSu*C!buykp*r8zp~f`P7hVkkIMv?ny@a>4ShX7*gU3>rYvHtnMG||
z#I^%wCHbTlTuBM|exaikJV3JmFh+u^m41HhM7<^TE;s40D<;MblN~r`Bho)`F58(_
zP`;^qJi?ld%ZuhTxHqu~`zGx~f7Ny*n0u;)4hcg@eL>(rnKPX&l9&aDm&@^uKkR6;KxX*-+8OrV3`+=R>O)@6-}gC$QMu`
zI}U)J)$AN!hf%jVO;|av;cK{;mV9(CHm@!c>
zkeVeexteu-c^_&imlXfHF*RE*HGiIZ!L!Xfz7~_OK)!EjgxwzEajQqpZJg9Fq|E$=
zu@P?-4$fX8;UMRNY}(WC*K;UkA3U%jM(})xWitQ6WwzESYAuX3b@p#fTdW=
z5W^_gZ|=Q2e{9*^S2vo^%4hW8DJ3m2^)n)wc0YtELp!$wh2+=KE?9t={$1j18-0!i-#X;EiYlHVs2nk6^rjijKYwJPfx
z6b0O|zaI-RVZ>QVyyB2s5^%1{k49Q|*=QfaUIBY`JL`HnjPC#c*fq(XhfhJ-^A3LC}QtrtI#y*Y+7h}=vJ5TzN#Kcn4
zRz$C0jekrnX;#Y>Ic8aR|J0=olL_IvVF;G+&yv~ahiue#^W{Zz1-A@tmakb?@wI!}
zu+z^fgcY;-WgZ9&!=K|oFiX+v$}f
zf8|wgKx<7ln$>(!jm(C*lhedcn9LKEPXVTG!?+|)Nfo*)dR?b~d<|3OR0Q7dpHeWfPD`RzWaZ`Cl$
zvc`$fVw{%Kvsds?Rc>;^2lYLDnt?z>f1j0h5gM~Y=9#l@Ie*tGYt#USU+Hqf@a1cL
z%!rFMrwM#_H0Bldqr2MBxF3p;
zR|@SD>TdqmVpykpx))w~3a16fyd7l-Z(eB_K(-W8Rd=#_Yig5y(pimBk^%Swf1Vkd
zE$VVNG#X&=X&Zk}G(K@B@Za*|rW=J`MGnyCo8_bZ8it14%H0Qb;9mC}
zWFOuBPOsxSDBx-#F~sAfsHd8S*6VRkTga3E_!x_fDP=
zV)|S5^PU5lG9FJ6K_hs95ObVme>mlzKi5RcSM@bhDD@#M09QKV30rWOxkB$*!vely
z3*kNGN~23pnJ9TJ+_ISt$P*W`RD@N9zNfRn4yfc7j0t~H-xM0|o1MTO&`oC02;9PPClSwv*fJx?OwY6bjh+wN
z)o<gir^ay+OtVnC|n;^&Wtn^b-LOs
zj_)1NReiB@5N;%_p`OIsc0&B7@A
zL-(F`jL`K`1feGO(WsnC
z2ry8|mBmXv*V+2U@ZM1lVO7=$zlCG&k|#%b8;_LEf4X<8Ig-}@{bPaocy((1YRBME
zKHI2S(?0QA^a5|B!KUj9vmh-c|H=7ylv5q;;J4}lAF_<2-syDS^p2G#!3)sc@@&7N
z%R0Bv4tm3qp*IwCsd!gU*Db8nmMz%&(mo_@G+1bUStPX9)hUuIHS)eOxxQ|En#)T*
zl8p2te~1`l(yYqUim|-KI;8WCd+OCaA}rE457%4m5zuX^dtxy$bd2Y^ouXjCjZ1k$
zZ$6F72H6xH_tmNnXt?VKrZblPxO$thm2d|dAL=m_874Wym+oWx)~Y!DxqS%}=d+-H?CzH2khDO3aFxol&e{$=f12JVks)Okle~{BR*0#<^I(ks6h@j>
zvs8mP2BldzYz7(^k*X0jUQpjl2pg^RRTq7%0E_%3G^v=Hub4s-tWjdd~9b7$kNk$(zB{_F2wAPhP
zeN{iYv)J8HP`Fei8i!rHdHnVvUUh=g-Ab)d$oowQQ`NTpE<8a#g8-)?gVWd#IHd34
z2)fDcF%MR8?i{ySBmh6tH!^#YEEyY`e*$S;LU~kQP)4kyVqf8B8+SA?%23ss$m3o1
zkI7AWh!D?^T&XpXZ@gDA@t76LlSYm*ZeC5hY}9;>F1XI2MlSHc%}aO9N3EWN6~8db
z<*exW-j$P6O8o<${w62nxK~U^us-!&Znbl{^N2g4bL@ap3?o(j+s%a4w6E)$e>ae&
z0u*j9)p&`3>84wolz||Pk2Sr*c?agMT|P1Xf(%F%r%`>kdn=PdgLJVh8szvXc*5t?
z2uEqh!7ysr>3YW#m~Ao~;XYO@Afo_`cU<`Kg|woU@#aKoqnk;B!LP^tQq{v@V(Aa^
zs4Xql0U**h=XGbr2tt_Z7CCf;e`$_xFTy|lf^?LE5{S0scB9Q5?w&Be$c>WY7s`|j
z$#|^Q-fJ~FCg{FndhqDR#95AyuRL{%SnoF&OB*)-s_<1rmNABhpJ<_w1xM{}o7gEo
z?&>9e?UXRY7g~l)r-arJd5EhS=2CC`GoN8Bfx@)Jm$t6%u8I~OK4g7te=hdLCr`u{
zczj~Pg&h{JcDMeij}*|3)IUyX*3!kCgzJI}(;#^|K-o4ZZ+YcxAwn>4*0xsScD6v>
zp26mQ$#(ns))ygA9SpLCCyoL{9oKL>RGK0MS`sFpbTXV+tw@>!Sa$~p2flbWbX(-}
zFdEy*YWf%nGJcHrua&Q!fAXI{9S$E06{0%z3=Z~J@;6vhj)#{v948;;r&qTn%q-c(Dzv5`T^s$o$A@
zXctU(%+XV0_VS{U;=RG{_t2+yI;HMFy+#TPUNYR0q+)(5i(<9$ftLLX!Ozgf+V3?s1fs}24Dd!mT<{)fYb+?#lm+hzeI1WK4M*uj+)IljJ
zkjdsDEg+C=E1y*C4y>x<`Wa-TOciLta!uxUxr+q)iGInGCf$2`IqGX?G5o?sGNoL^
zw;-W~W5%r{#M>cF)+-)O{H?e}Qw}Y?@I}l!k`n)ACbK>?q<o6(|BYl%itX35YD)c
zR5hm;@(a8lO*yBu8)iW+7_B^(H4ly%;gYn6TtUSYc%ZmM?T_3)BGmbgs8&|;zm+=u
zaEn#VHJEb`bzCmsMt^9vB_ivyj6IBwPJ^_8iP3eiO65PR)rM8$BtPBduPJq^FM-~<
zRAIB>SF``d%}BPyk`%J+U$Wv7GyRRO{mmiaNPHi;kfp1O5deQwL#PCfpyjxJ4faz+
zDEho%b-5F<*P72aOnnoFw)0KGt2gXD4cKOb-@DqYum=2KKYw0L=^r}%ryw^p
zX*z*m&&|}+)_+UfI7LQU7`GqWAb_1}OK_D!13ce$Br@8i_0F{id20w4!dN1-?s24p
z$eNef3KG?lS%Z?)m<_*mQG0-q4KZ5M;z;Xq!nG`0X>+1-^Tk4@ZutF|n*}@A@un-0
zpE;o`^~l+xHzPQN;X}CL6QdTs&8M5C+wkFUeon@Wp??%lMOhZXblq`%T3?z9a83t3
z@$+i;pDBK9Tibe0Uw=Kp+C$KO^2$+s9>mpMf^6?|a!t%GT3p%BP#Qx`-pQjuP{zM1
ze^{(lJtI=_WPswgMc{7`k6wtx#ctL3K|jcIw=FW0!{c>0jl)}Lc9^ElD?K|S0MbEt
zglmsUEq}DgQyJ;#OTlL%`#V;~*=*uCdO{N~PD(02Lp-wSRB5Ey&$ZA#aMudatT=QY
zO_<2E5*;UDZ&RtZU}r)#n4k6f#!*Jd5fv%u#FH=AkVPsn`ZeOX^+n>z1YVe0b!+Ah
zGK8t%Yh>f;V<~2VA7LvmIPQp<=+K{y#Qmf+nt#N0^O}uJ(1`Oy>ozvF_WHBzNzsym
zKh*Wshi9B@PF(c&%;yjjE*jQ$Q_h?#&ED}rw64)!plHplv&P6JdT%Ns7a0_3C!VI1
z^2K+5!9UYo@xR;P-s6R(+AV{CYpvmPQB5f#>J>a$ktFE{A(O@-eLfdwzGo$l#t8{h
zpnv(^L5bnd<`TbLdT8@BNHY`yz$5>O*;wXUJG5*sbmlHk*fHF`ASj2oV-Aug4!Bq&
zwcW!AiVh)p3C!3efLKyv(<($`#<-%0E&O^dQrGuM@!n^IC-(8DZH@B20|xi?SIn<`a8<%Sflt%vPE*v&slhdUb%NZk-{ge
z{tIb%=Lan~MV>Zg*l!WMtFt~es1EnV5Zb=^{KE2@)(E2QAgu^;qy3C*m~WY%GKo*P@$i9AsL%gxBNm7omzv3;c|8(tWPj>K~&b@PC6X
zcf=uGJA?iGI#WDn#)J|%5fmy@-A_6?MXLl)nDu%p;;Mt1QJ-z1og`xNK(C|QhVv5kDVy1>5RA<3K)VH1$kUr)c
zD^xCuXu*|svHQ2*9sJ3nd94f3rjwzp5WY4l_q~Lw43Yva!WfF(9W%h=L%SA4
znPEp`80X0o6G#K4Z**pU$BGXiVVrT6)uPNqj&xqsoB)ZXZ$_xsm?N9a&~_|}Y=>dIxw5A~BBt?)ghiu;9O{4Q|-
zf;BDUQaTA1E{ada%}2pDyII6igI605Vm%)YYgu=GR*{+s7)i(au|moI>xEBAWbcd7
zIjP_5^qlI!4xoA7kaRj2`r9x|$qHobItfzkbKf1T-i*x^Wq+GgOk;{HvWlYUc)p~(
zna*5#X&Sw0cf_>0)>MDoQvAhObbb2mCg^9ieYNmc#5jsmUU|*E-J{;DT4ngI3d#C1
z!CDw9mrQ;^ef&=#=*o#(p`WJ8%{xJ!H0S43BItfKhXIB|=%_;Y!4ujzYo}oNV*oq8
z;+wmWll{4wk$+HpadkfC1fVYAkzYXz^X@Qy6`X>kGkc?|cP!tKk*g+5x9YcMk&5v5XjhnOlxQ(?{GDwc-+Nb$jU`_0I5CXV_pEbCM}GZqjE2y$`-KfY
zTrYtaV~oB|_ui@_L`an%Y$yF1n1a;hhzNTAv<+cf^H;`RM)Yx;eNx5y{&Ari!ii!+
z$&En@9e)y``EjQnGG95%Vi!>MqH6pHL9Yyp9R_yc85U{)WF~rxsc$i7sQ>0D&K(#H
zK7ESc(H{HA#qV+t$tOu+#{)ZXreh*L{M;z?C{P{ZbP9MLFRKK28Pb=s3cW`MQ04z}
zv=0{~KHf2zU3Vipd5jn`7M}4x1r^#Z&||mWqJMx=T^I!>02K5thjIvJ4hYum4Wl3P
z-nQ1x=8GX%f%BFfvC*RLTFH`|ZAo1#%A$0#B%m13UbCMAf~SKP?(Htm)|(qZ<``%U
z-TOtfBMEYV5K@4yX%3q0o}!dGiimQPCX);2@rhAp*7`cR
zh`^M$2K!boTz@I{fi0Agr+r^>zGz=*`hwCTW>YVR%#FVH?!8i1X5q07?$C%<9TThj
zb$gCgE6{mW75dZvFx2{Bbj@a)5=w(J{C`1n?Z}27074zSC}(q9`YK8o*Gm|63M6L9
zlhuNI#AVQ0WjCF>3wOvfF#o&(LtTWQmNC(09k92%Kr20$$HkB2|LMiVyi4TrpP6-o
z@UdmBP=9Q|X_xPTA}v6i_Ek;ly9AE7^Sa-}7Qfv10f6pp#~cByYSUAJ45yQdV1G#W
z&~?3yHuK&PD)`)(b;2V1`N=5@r<~F^*(Ysig8k8mE*9=kVb5r~Rl**l7BV5RX#dIBmuqj5Oh
zXNUF!CcS2~4+R&$uBsgXm?fqr!&sZauh=0ls55K<99{}$KDUvdX)n1iF6fh;QxsR;chAgNNxLfo+p5-w
z_w7JgO=D9+qdkjSzN~)9R@v{vuk(kVs$#TgEG>=Um
zX<^5E^2euL;lcaZQPGLiAWB+Z2O{;K$@gS;xN?@74)w_y(|@+_JJR40LOgt!vpkPS
zvQsNajwU@f>e-Tzt9-!RauY(AEp$2oD=eRY>~n<;Tvo;OR$$itSB6P@rGe70o5!mJ
zUhiC`&e$w0Ayk{dg(xmLh
z&bQdC#ZRo?rhhf;<=apF*el^g*OV~R^TR@A`*H-H@-*SW!gmuP_}Kn0hPsmg-L^M!
z*E}|Dsov7p^sbt^eX)9{QE)Xh0^AjRa}zh~0=0{ORQR?X@3f|$QR;SDniZ45e=De#
z_J;7Ay0Q0zQY;Mj+;1fPeSsMC2vFax26RiVb&E%i{(oG%t5G=Ch$00G)KKx&JCl3UH-X
znpS%uSo0Mvi~sY`(LUhg&|7m0=T(bNGr!78fz2hOqjuh`HdGAV-qvC4T$JH8>s_LT+B6`5Ut$H|
z4`g*bo1lsaeJIKQ*1VZ9H^_Q>Hxrw>5K-PO34a5xZDS3_^3$k!``EV64qcSN6);+)
z>k$cQJT*sfqDrSd>Tyw&@QgUmk|Wf^(8I~~5J&B6(^f7gf_47L)pd{=gl3?*Gqtj02l>OzQ#TCp7i2NS`VbK}3+!Jyue+=!
z>=9j*a{yr?y2ZIouFy1i1^0i(x%yd?K7X_+c8L}NA4I2(yb6bg2ta$1&Vc|sqEs0?
zxGcUU&1T(6+Xv`>IlX2Hz1qekJ1m-|B2KIvnxGk-e&N*BtQ!1T^8QNT@buIN8~&&8K_HmA3X(0uCu;t%9w0IDakw
zNe`fP_ZQ~p|M#)K;^~ctC3DDSHuhd&SDTY-0D4;vHUT!eOq|nyvS(?nq7;6Fp5P1X
z)8|_qLw{{IaMkbWG9nzR)h@Lw+6trX7PK)^MB}?f8kw>Y?yq@2e$qkIk6p4CN^vg=
z9y6@{^!QNS(}5AggVozy$H0ote1F+NcK#Ra4^m|-*SiYzneU!SC=`Y!zUha~EFGnh`Pk4Sr8F*P8t*xvK6jUexO
zBx?AFlqkG9xZfSp{4k8eG*0nkuOXpJHuxt~H9OzK`#VfwX>bya`OayEtAD-J?+y6U
zbfveTOIPwo9XFS2`7*Z{tH%?~Nq}M)1p6M1_+ZH3x@}80BQl+%oPB0`ObIYZJK5*^686|V1HU*xZ-jBO_7^_8mpeaGB8d)2Ee)=aBIiv5(8TFZ2QB23_86!
zBU&gMBsaUnDO}B}jpji&B7YN|#%L2Y2!5(Ea=~LkA~1F?
zAK5-#AQk&CfEzi*BaBr9L2Di&y?Ve{yb}RFl$0#o_|E4oXIaYw%ILk=qMAt)
z^g!e5v>V3NDZ&A43WV@Sk^vW=
zMP9V`#BKX&g%=k!tLzg#w&p;C*XmxGU7R;?91uacrJS!fk-{1IF#n%Ir-*>tMZpe{
z<-;LRPJj9iQJxBBpa9iLaDl`NM1mt8YwI*XbTO_WF!}SvlGal}UBn(Q${i3Id
z7=Nt69TStG{JaK82daaAi=`*q!QKC0Z+=Dnz?@>&UeAOBFYyF
zWG6KFpxi6I)Z^GpH{LkV9e-BTuC!;b^F)y75c~a97N={BFB@7!
zB2Qk5%2y#)su}}tOTL?nWte)%Zc7k^-lfC**<7}RPmq{u(PO~hUJJehWL7PdO8ZAKLMCpmsFcUCTfaV1Z9L4V}D*lX~RI9!V~wBCUq`mL)x7;oX&dt&=)c<7zh
zZ=UY*@qH2ayZwa8WvvqUo$04GK7dj*hj(O4YiQ(*g9z}5xf{si2pujF&%3>a025ad
zMY>g3+4?cCoa15j&cyc`Gd8qdGmO1C(Rb2oQU383)=;@)MC_Fg(M-19|$NPkauQbPb(4hiUal``7vR4(Xja-;8x6OMKdLP1wRGsllL
z&;lO&K`a#Ou!Ts#n}wA6C*e)sh;g0U92JjG|J6_XE}Hld>+H_OvP+YtPXnfu_sawi
z=9H^Q`C4A0#XMGa82wm9_9%_IE*HU^l5pohWH=KP&PbInE3OddMt@gLcvIo|E!9sH
zT{FINK~Z*yv#CCj7_%}eTr`PG>p;HfwZ&bZT-)T<=ZN{(VuTIKfQuB6CB;-6zO&}x
z=ha;eaV}wa0+(-(B0P%S%8646$TAzdjob66ooSpOs7`Q80=$LR8roTEQW`-V2B2I~
zF`Ek?Rs+x%P`qe7+J6WWAquMKu9Wvgz$^0OMMS%?of}_(@4IMBiI}%B3O&_(6E(?5
zrK3@M;Xgx7chc&KC0=JNxaW=&mk3N`+rYD$iJwx#
zlvKuk0rrPB@rVAgc3u8*6?bqxRfr~ys)kf}iHu}Yk|$U9AAbp?nlfOz4hcAQQp*^$o1N+t7Bk-=kK5`$tGg8
z@7>=7q}Q~gCx6l?)Ugs9X?J*Y_bHkK3X;yp^YOB8_Ttfbd^q1QmwkXXBUG`OeL4;8
zySS{dO7lLcJQ}A;sDJ0N=_OTd$s2J3dW_KK-1Tm27O7QAvuP5~XZXwC#6}m&>eP`m
z=pb9gL%3R^_h@IAR-KWKkY^P_NUpP@(+6RF{sn*Jh<_L2qW2xP<2%)_dr|!Z?D}WT
z{W{Gd@DqwcE(u;DUl)H=jy=-j~}zt;^1BXCQ2S5KANu|DKI*zh-o7
z+Rp7&|9Va)&1}vQ#$7Xb7oY&`Vqqd>Gk)Snb;6Y^X?{Nq6kRC=3q(i8g(64D^+aj|-~5^M&eaX)oHjwut3Af%kH(;(RK<_A#x
zkD+>&h6~+fi-2;~G*(b&BdGT#z8mwu@m1^3yno_Mx~|A*!dQ&q5~k9t)Jt@V{?+s<
z?74)jpM1V2;bHkORl`Fp@Q@OBsTD&#^CSS
zA_>jgJJ
zPk#$^Wo@{4uv&VD6!976#g-5P{JhN?_bzZZ&~ghXnCH5cPOJaAC=A#p2z=VLK78U(
z7fNOvcS}`le}z5sZK>ZCJ%wF3svgjF9LcZzU}CI}`#GbwnA}0OgOpJ+^7ck5mLQJV3-d29E8<0hd{Wbh+4iM$DVT;I27`uuA4J>Rbcai9(Qu~df22T^l*lqNjsX2RB&3vpI
zB-HSfdP`a^hZ`7ct^-AItg@0&}Bi6WQ<4c{D3(LK;%ul`p!-oPO>ob|@k$kFF
zgFuZs{4;_8f`M>iCMsuB3omo-;od4}wYCslhi0r^D!xZlt~n`~JEq{NcZi>oG$!1w
zXl-VuBBVxwmZuXcJ=ER{l^gpiL+vlv|0JQr0ASfSu!0kx_=Y-MLso{B6@Dl-#?2K{4Q4d~3fr=bw#AGQ&Um^Xm;FEZNUoY_DV
zROb#J({hB})?R`r>N$$A778^akhaX=u6+_uRY&ejTQ8rx_H82m7okK;EftnDdRLtv
zl20!%mPWw{Iz#paV+N@MZ-1h{$cCQ{ISZ4&_6Dsy2jp6JwZS8RvW3Fgggy_TuY*je
z0<{h$!OuQ{q|-T+S2Yi}MG;%5)|f0R|E`;f#g+B9MtF0x{fBg~$1?jiTB97mqSvQD-GlB9CevpEu&Vn>C
z%%OAbuN1^p+@Z6b=0M1PB%ulh^DTQE#?MyDEC@K&>b
z0Cc9_xi>###>&QBf*4%VY^@~Xe6b8Uld&{ff6yB#=O^QTpfUNBGHxy|tBv<>cav|x
zU;)GL7X&TR*Au7T|1Kg1nMOh}X4q#c<&b{Bk6wz~4xfDdSV)dE1Ed!6L-DoJQ6EO`
z!yp(IVt>9wmc$>Zb~(;34AD1I`jYr@1Hv^GLqiQ9fVpDs&4&hiP9=CK3mV+O(zgJu
zP8&#|)~R>)M3$+!1@k0o>-Lj+)KcoEml0H2kB)1TT<+jP2cgO#HQ(m?4+MHo)z&3dOHykynmqzLtzN;*0_8BM(5#|*i;>Ix#aRB
zTi3B8WH=!frIV!RMEQ7rC~B=%QIGU&+^4yu$OPtQW6k;735&;Uq4t~Yb8p|}-^FFm
zh*+&O$V$hU_y@t_As~{V&ev2UPbJ{1RujSKy9P&j$A}R$HMcUo-bIy3S~g!7iE!yJ
zi+^uo0_H>LNop)A@yn*9g*6wz%2Kx$Yrcn7j9c6ZZNFKO1P_?WL$B^(p;hMMml7*Stb!@DV^dbx7!t!ZMz7r`)XO5O`5wtqt+
zcSA+8?>kXLwAa8T67s?ZlWkTPZays}*{`|R6s2jWkK6`<4PJJ`_2B#)2hzy=LVmHh
z_`{T`E{N{QWrzS9KwC*|3W?jenNj5hBBCF6#g34&VVRx;8p`?3RmAM%@-1@DhRzzn
zur5li?$LI)o7ByTaZr)zkwh4e&3`$;dtndAQkn_EnCysoy8@iF*>ai~-fsEBEHh;f*YSLlq
z5PgYl6x{u*I>agKIUQ+h`HO%vB?pY{KwS+K^T}~NiPu3&0Q1SyZ;n(es#dpI#1WT0
zJOfjw;WHuk}&0p=YQ*S4#vB8xT`Ngp8wPcJh3Ysd
z*vy0mG9v07h0td1($w5Hh}dgs9yiEYwy>(FxJJ!CZefSO&XVp`uYYRW6+uKUU`gRJ
z3$607xkftkil`91WhEFxENy%cF!>b6KcbZ>^#BNE41Cx{qbm0NXsqPJA8i%v6W&!jJ5{y)US0vM@DFeU+p-D!FG6AD;mvTF0W+jkKdn*M|O
z&@ieSbqdfl4gI#n*MCbkvBUegvNyZj%r*kYVyjMPhmF_O_B|u<8w4&
z0
z92Km_9SysCLApNr?YV(J&YsDXtED9toIRUVTV+qaz&gACz#p4om*cbKwTQJ>@DwW_|3)G`#+)(`oaR$%8q0Fjmz)3_MZ
z7g{e9|8Ty+Cm{=#aiBnyDT881-V&ax2VMuF5|LI~T>_+|?vQg>riE@jZ#xkP4V|2>
zj(hvU^!d@|g7>I9ya1?s_WHhkM3V5;s?Qty@LY
zE^C-eA_dT@?fD?1OOxYxObeBYFv>?roSGFDXl(nRNS4ur-
z{y3l+bLvJ}ETa-L>WXeQHIDv<|2QF)3BvFvOMk{_wUJpA=^otg}MgN#DOBaIVwG6uCD0OU0>%e)CMKz3ZNJY_s!b~
z!AJpi#Fd3CInD>mjfZ-0(5Fk$dYW!NY4(-HueH~CjUYk0CR{N
zefa&go4KPDI8MUejs1=ZA8d`*pn7MqA}w#VzcjekPlsAD%*KfM9O(_zqDy~s4gc~J
zGqcg`Kbghj&6ePp@cac888vjf!;*Apk$)d+tMlZT3U_o#d(I-H>GLiqfYS`*QtU>g
z=XFgxxfIfE=)^Pn`+w4$y16-yr4Y&lI9eAA!0^9@&ACT-L!OfMJl~VL+xCOw`8Oz>
zGZh66z0jJnK7RIdMcom`Q#w8LS+u6WQS(hJT{H
ze1GXT)EG+XxoKXp{w{A~uxU_?^U#krgv{((YsOzJK8(}^=GO5#evmc6CS+0SMdNLB
z0U`4VB}(q++J+zNi4Ts=R-R>vBcWf4wa%o-;`<4gVpI4B0E)bG=7#QfVc$fcp;Mb~h65m^LPG0l8-FSYR!*<%
zkkGChvbRsL>J0Q-@(U8!@lK>*22EFTdR71t57OXP-DcebwXwguNoZyb?#8WxN5}oe
zqC$Yo`ijsp1sJ-BV7_2iGY^8V?mf>$u}zB(jmTGn_btTa4p<3dZ?Lw~_)_*8?}M+s
zFwgLIc(1+=9QO_PN=OXg-hah8kiaZlO~bcwsqFvOs|$jzB;3H#wR(J0x`C@d!q$9k99l#0H)Y1GJ;^Dn;tTj0-ic)QeJt8{MV%ZOq
za|rUSN=z9~uX!
zCeL4V5iAtIl6GW*k$)o_jAGQn+=5XQZfVk$(=^fMlEFqhPhSAmCd^0DN6g#X5gvlH
zws-zrJX?n4+DyK56lYdQXCZ-0S%kL5HE1_hq)LF*vgPD*(#Hz-VC5F=$=^a&xe1;~
zJ3(7{Af9isE$6hb1>zAN#WHQpqPzEaMg*J=UK0wL;*1yM(0>|ma?|WcH*8QZX=p_S
zjwQB-<|Xw1X!Up5VqQP~HIVvVwQqC{U1Igv!zL9Gj8*_9J$TA6P6m$
zTo3X9n(v`$Egrc%>3aU7{qUPlLlUR!(hcw4EzA%$u>_q-L)e3_0yhw!gLb?pO{C4Bv`H#3
zBKUT7ROs5j94*60tO`#|agW!#vk!dNGeA{J2l=d&P766ny0LT9q(f5WL;nRoj^QWy
zA;Uq$|9^zLEl!P0&YuUmaj0S|CoHHma$)PCmD
z&w9^C1@@VvEtU|jgq6rsM;(o%!uoku6h)@fZGTvUAxJ`Ad}r!B2i4M3WeMLq+0M@&
zIkOkqkS?8xx(S@(9pFt@FR)EW<_m{%V=`j6SiYvIOB=0^nfoxJadGPiqvO9Z<^GNp
z&*_EMxrxjyU6ZJl+9z=RYRU6me+0toafCT$UqvFBs6uv6mCiXE+ZN2uuSMakw3lQ~
z_J6lDJ?k2h?sPBZ2~^#Cn=xSvk&4UaS(%&V%A=*(318@gh(+}EqkLdl%4Z1ss}4j&
zVVscXA1iF-E&TW|8f!PT?hF1g7?phtg@E}?ls)|;Y*bypmV!TIi!T<3RP~HCQs|{{
zs#PDYf}~|ONw>|)*i%R&>Ho(EDgD1xfq$Exk!!6(on{Y`pSEy@y9uLS-|fq1v}xHP9Lc6La7WXBK`QZr_1m-
z8>f0B!((zu%~CGlQVR50HlNy=Tu8z4q3@z!Rn{keEs01wL$sL^gfw!AUO8~OP>5d!
zfU|ix7#HEvX9i(hH}bQE)gD=j;D6nE#TMKRDw@4f=ooelhu9;ZWp`7(?;nC6@6xGO
zI=3Mb;33RU&MJY&+R{1uAe5vBrBm=i)kFnzZ<7L*-!6gF
zHgahrRpn^(5Ie$VUJaYj5!5p~m}xd#0-&{Z;-LsZ{5X+f5C)(`Q58k>yf5KvPgMud
z7T+%WSW0>b%X2Q{_j#C=p?}i>1mA6}Y8p|dU-k*I*@0Z4GxmSZFRdnFY~Vb8rUVvZ
zMKSb4Sq&r{Gj};KEQzk5J;fFL+cNxox@wp11oO7c(n?_rh~PvvbrJ0ch(a
zyc(zvu48PZNv}oMiR?ljFnv_1P3!;j0on(nDI&3A5&jAx`@w$NGlLHkH46=zzMJ!80fl(
z>mgQSi3Dv-n3l|DnI18~T-lyn>+cH5+iG65*?`*{_2)Y?qJL+MLQ`0fL_|bhBIbiM
z`#Ri~!bcan5`dnZ)w9}hM7R-~bE9yT8DfZEju`Cd3tUNfkh+1CC8?v*NAsq%5`R=2Y=G|EC}c-#&}j^{c4r9+V?)_>{E0=l&+hYW%##wB7}PLwndAg^R+PmXd7w!Y*aF
zS?sWYU4NO(1P&$kuQTZ5RQM$O&`~E|>-=v#y-t_6?PSVEFQ)oYg}P(dm_hm{GJk&~O~Y)f8Zok$`guS1CTM#w@C2H(
z`@~zXzY&XTbTXOWt8cTdYQ
z*NkWOOx-mQ1A12s*U;Y%6$)cF*{>XIijEzi=8_{e5cGWwH?H-^wJ}8j)i5D;owGmg
zl7CZeZmfpE&bzMrO;HrH(;inQuiBCa#Po
z;6)G#_a89ZdLgHhHS2#PQ&Qd-Ek!B83&pz&STukX>0@7#`V)f7R)-FvzMTuHZzzw`
zqgSAEvQB;~)i{UQ@f~$a91=9GX+E|)2!Gz>lxBGaB}{yj;_z@3Z~fpfiIE4q9T+z&
z_grHy9wZ*_>UU4h>%)LBN6LCw7(c{c8q4O8U^4I8PhhIf%&7R4VZ%pr>8Pz+^Zz=V
z#rT){!QF@_?{TeFENsE+#urfOp`x`y&BQ)eq;!wLf`=#zxF*3B5hPGr3xV3uYJVGE
z=(Hon_MqI}bUIWx29t{ti-oc@%67F5PMK-$o&JMVrySgyUF8#-~+&zf{sOl_#-ckD-)wKDw4U`+v5``R4gj)i548@Hs)a7tg?jQxQ{aMq|af>Sn
z`NULh*;F@OhYh3cPy$-%mLzR`QY-sw`&P
zDRe~)R7-TM`N}n6l^g>vU76iU@?J6TB7v}ROCoq;?c;p_cew}rw)>xxKg84&f%?&@N%xpRIL6Wq%
z`C21>lHp?f9~tG{E}=^=p@4==%S{snJFnQw5#o$K!q)}VQc2oY>3{xXPA_j^HGT-rjFVO8nj`?=18vV?bdB5BJ&_q!bvvnp3`i-~9Y!|cl9wGO6!4et0t#hr
zWOHRzgEl{{qxVyW%yL)gA?k>UIJve!l?(g38
z?f3q^!GK~eoons2_daJ7DWRMqEx)1lSD=Ws6^NFRj)4mxB`s}jWzEPyD-P1PFf{-$
z(J?TvAdr#@+5z=Jrq)(M`XC?|fE{E4kTn2-J?(!0ObiU12&4c}pcT*#tTY6Cbpc2N
zLHbHAHb6!Ih5lba&e|SC`&Hi_tOi;cn_2-W!7hTv-e@FW5EkFm5
z&^Iu*cCt4&1?XED0wn09=>Rg;PGF%afWq1e@D*sHZ(#(mHUcOC)d0$h!U~E2Q3Y9L
zIYoa;I&fe`2OAq}yZ_@NsHmhYN&^t$mr)W1099xJqRNU&zyB%$t-$<^X#g@xVEykr
zV8h?;(!xsoO6qdLjP$>205Ad^fp+$$ztjGU8yT1x;14yhtC5|xk_|pz(0sN~m4pxTXc7jZRe+T%zCIBf@
z1E7^X@VAqQ_1{WMa4W%1U^(c&#K4UJ{f=qzw>iKb2>hQkCi?b&`AW&jNdYYNO|3vc
zD}5^iupvkvHj-D$z=T2hWOD
z+8W#)D>{%f=r8Nv@%V+Lz@x~{0bpWc0x*DwRM^T;(Av@x%-SB|cl(4)!F>W*+quyH
zd#uf^tevdf{$FAvQ!7KG-+ec9u%UlfwlcMK0E!F!pD|d3@JD701OXTTKwALN*}#PU
zce1}`<+qsew;0?64>ucY8vsi{w7-CnzJ)!|!_)`}{y=cE*LMU0Kz0s554V3R{ud!I
zvI7iF4M5;A2X7UGzoLs<8Ce53{}zLp{73bF13>ZDHl+mbQA2Ah3m1SP&A)+qnpQ5N9Q1s^}AzfUi)o{jxqI`C8)
zm|Fqu?E!3@f2n}rUi^zTcmaNs2G9$C2#ZR|h*1A~ZvL_owlc6bG_^7YFtM@$^zH2Q
zT@V<+!@$JK3UFftPqiV?`LAgK(9>C2gTO8T8wZdFz{uJT;rEJyiO~!G7X6Jl0Q5qC
z5GR0M_`isq0YESE2QdNY#r_~>0KNDh#0H?3`h&n3r2mUJz)|G>AaE3gKL{LuMez?}
z0njV`LExAw|3#eOnEHPZnDy5`h!KqJz~cl2Spbbdf8@;nmH$1M|5bp?H25!K1qU;*
zwg9i%efU7Y42Lz`C{ueTX6M;K!p>O#Q3vi)Ee{z5=j7%MY|ImXa
z#{YoelqP=&u!50`jS0}|9~NMLndv_uxFP2MfZ!r5{sF-i{(}r7nEjtn%-~2?4whek
z?`h+IWMBjrW&MW(IE1y;KNvHDE3)~c28XoK2cO*k({dKZ|C9c2!&$&nznfg{
z_dWO5AJm=RfWLNkIDN
z^7x0hfPl5L8!ZcXtI#rkgPXzuZZCL<**yNo)!^@|_^-nsd^-IH|GozSK%g_w0AXp~
z+TgRlSyE$QiKlShWa%4HPP&tF>@R8(F)&N1lWlnKg|fGZfV}xW-@Mbv@~x%BxU@YJ
ze5?x9Nd2)aUe}uAD<*CX<@vVtJ*7SIP=xtMvQ+4lebb~Dy-PcPh$s&wvU1gnSfO#Q_fbOPTON*VeMWmnK^5u$679
z#MX=OSzy%ag{${rAz7>7*h^`~zH?fHNl!!^^+)6wyBMp-GOOZNp>zo2$6nMI;hV@d
zeJAoRkDK^D_aE6WN9i>iBvGrlsHD9iNqDKLhzk*MF&{L4l2o3I#Xcy#5%i*X3KiEC
z3)|eAWA`BGY5qVOsnzNttG^OPE_>G8WjZw&Gh#}PH9$IH&2;2ee^VAMHzdF6gx#{=}jBx8`|%Rg))G88bky*Mbc{9EZz5d^hY&>
z#2#yYrw|wx>q^qhGUrj6tb11i)hepC4cD+%q=3YK(sl$_QlC|qP6GEzLO-BBYYj?+
zIgnb+uW>E~FykutIT$%)j0g@U$TsPrQK>KQN?%_P~p)Pi)_Eedk&JlAYQCoU549?l)^Dr)N(#WdF
zgsh8ys28z9Un~Rk*9q8*wjm8mF97y(7a(nIq0n{TYOQ%NH+C;7(^ur0&u$HJugRq;rl0|^YQN*Tc9j2xp-)k}(?gv7~uo4TQRIszG7rAjvwEdrDt
zbaIXCrVJYYIC6rU_>{=@rxeN8yyRB@ZcK!Kypo(v6{}33Hhcx9e^8uA8dq$m$z`q8
zS|;{73ayeleZGtgi1!r0+|
z==rsu>fkXoI%U{~f!apvoT84;2eJw?iZpZ+Z4WFScs;AP@p0Qt19=^(b)R04W66&?
z^cAA{>3!|EG_fC#5Qx>1c0IBR?{PW>-e4Epzt2Tqw#H?sha(9y0ye0&TUq$)5WCFQ
z(?;MvS}H-Cv}D<8i?#3W%O(zzgQ`e>(HGfQ6sP*!bIKV^0i>$FthoI|u^;rcWYJZc
zyku797qFhZ+3`5oJ%|)05CVociXj!DuTdL{VI#5Ktk8VJlL$)=R}C>OKV0LcDlN{c
zM|&i(f8USIRIW9Z0CO@E9<65rL)-`gE!4W^kI7M15oqDG#yyC7U
z*w}K}RUqn|R8mzKoYK57VqIzBJ%
z`=lYJ*KK)fBiDgY_#$tg&z~-+N{nl$s>(R>id6}ePKy?{)1F6y(CPW-@2DIpBN<}w
z@2Utea
z+bx-}V-Ze7pZPo|SImm=u|#RP$K;EPn=xONQ;nIt`SkXVi8+OpM5+@mEdO$V`=NcW
z@k)(B*$L&BvA6|9)#Oh75QCecTiV!?!G5&Jj$$O510#V}WVat}_nw~VoHfwJl)F(*
zu~P+>(r!&tHAPc9f$9@~MoFE&o%rnhG8fZl+4*t+e3E2JZ#ElO!SE*O`Mb}cZDqa5
z&b-UBLnEbxh7|k-8?U7onX2t~QEr|IvsP=d_r<7;^@wFypdMiG*HE4E`0FTpEu3|Z
zim69F+|7Zoa7TFv^u=(#1J9>;FRSabN$Yjf#8j3+!T3rnx>FK=lXBB;B>-bo0I|~`
zGo!F;zH67;SF+wD$hlBI)+ILpmF`c2_u-S!c5~O7N6kHZmlk24RN(6KF6%Hnd+Yq#
zI98Tquw>eQ9*|$0!74gr**eJ4q==thP)H}oX}4Kg%)723_fM_2-H#tiQa#cF+kfiP
z>ue<=uvXU7bI;v>Y+}cNC`+w4RbT2vqaCby4W2Hf5UGa3>)M{n
zHt)A(jcGIayQ78eTutBB&uiK(MeJi{N&{T1w;0%T$jEW|;B1+xFR(rOd9hL`+1}y&
zL@&%c3BJresWHtTyPZ?283C4#9^?9NqYQ9upd_8uOY)9?D0)Yt2niB5uhqT~josO_
zI*sSBBrsW5kt=*b)A42yE!9+izxWor56b5$s$Y}5hR=maz{y}DXem>Lz1cA?AhQkA
zxFI914^NNcnZjmg*Da^si6Q<7SFyb7=-EZ*fc_-v$lX{I=aUab`ISUDI%keV*Fry&
zCB*YqyST%D&_`{*oEq}}9jOzMx;OKlTBO`7(LqWibbX7E7wX>#5Pce(H7<~AqJ3OJ
zov%E?$NCz}^LG6mL_DkckqNnkgGYs;uI0AlUtK)L;hYDk|Ki+WLix%LDr@%|zDJ4=?h07P-rGSQZ-
zwywv;nbD|sHmNXw-m55j&7^(Wk>;Dg)BdfKvl|JPTRtf-m#iM@MO-(u&Fa#5+coJ(
zQvABQg6b0dcVk|yM+?R17g(5b!+zqmOz9dfh@J*n&Mj`G1-Iddf$3vSl`{LtBG)x|
z0|GvOxbOO@1+QNZNNSzdn#D3d5Th@9QB!K0ggen*VBhggxseQ)F)ijXawdg3cQz5_7PNn`craBXYM48dV@Ut`^Der%B)}0DRFnI
zn5vQgdyQN_$38`cRhcCH62>k?+>Ok6jEcrQVupqP#vg$cSMFJwaTixe-@EbsopP}x
zy}LEPT%VH=@xf3!1-6xLZ`&>nI_oMznQfN2|9YF6GaWvH?DE)W#L0+PH$E%fPsF5u
zIHgXj91c~nY>{OYy
z>V@b+?E>3pKYagMxvbe;UzDUlvZXI_l0gYuHQZs2ufM~6(MhgMBC*vk(`g&py-|Li
zdbCBSTd3G$8u?~p!v%SPbE|ziMJ028*E#!YkNkR6REg9f=`wEeD{bD#c~ZrZ%(@~`
ziF0?Sjphr8p&zbvQ=?WFI>8xX{TCfk;qQw&3hLg}wB`PCNp_-^^vYiqHqKAUI>Z{>hWt1~Koe1KaK9obOOMe6kg-=$rDyWEgr
zk^RN#)~J(ngx!tN@VEC$MH&NB?%Y>dOHq;sPU53?K6D6D{JSU^4viY9NEp071Xek*
zK1DZ$6QV37Q6u=b3+3_}8*_!z4gji$A9|1yJyDMF-5G?$-%6m}X6uZgX+OB%1r@1s
z6CbUTj8rBw?J0O5{OD39le+eQw(;ZLw*HdoaB7f<4^`%tz_K+h=f}>mcD`9)9*+XV
zt|TAEkyEd=>>0(D;LvT$b7ZC^Z@j8+M(g&fjm`LiUYuxem;mr8Co0yNQAveB$iz&z
zp-*pcF)74t6X5kSZFtYxiY_vG#c>^f-)={q)>}Ot(QA{^6!P(P^hYOuOjFdS1Dvcv
z{ZRO&+}*1kyoXSRYJ4Uq!@cMNWv}`}t~MUA;WNE7gIub0RcS^+u~iM9W;2p{6i?p;
zlBDAYvqz(t+1V3;&6?T$K^{34#5%9vQhWdnMy-
z{mj=h5gi4cr=yZKp_#aUaU!)dA}GLjyN+&v2Fsj|66y33^`LWJB-=yen_ACv|2(T!
z(72}5TQ?8NGaJPjiT-rj*5+Rjm39vTkXh=5?CJXwv$)kxmz5FNY28ulwa02-r0-o(
z59f^Pvt11K-dlIeRyLAd6Cviy+$`&n_%PgmlKByY*J&|w&hH0*N>@!a^ebt*H>q*D
z)WXbW*+8_KWT%P|C1`10`QFD*pV&T4MbXeF9yaWnnpK&HsR2}LWh{4;e`x*esAc!
zuZOvwu4EYtN%o-#@6FX7WkaMBZcJ$~MB@`m*RQ&d#y*4ut2HqYwgl<4uRKpA5F$n`
z{;aR4CtvA0uy;0`j(Y4wvYWf$rpUv43MAg5`4R6-y86Z8Y8gRm6e{H&~}oY;|ngr;IlFjpZaaJ|HRs=7A1u^dxU
z`MA}tnvUfGL%5^GY(!brd6dlKC?>xL^DX6&Gw7#q=5x0rO1g#f9#R(;G2b$fyH#%>
zNb5x1{y`%=h(xWC*Ti}{WH|j~Tm8FduYD7(Hude3M4^X>tsbmJQ`%(K`W3+Z9X3tJLIp
zC%MTpuox*V3@y9w0*_@4s`aLz-EvmR_qxkYbXvkVme3^J}W#H0ZU9`0MVxsu`J4)MNuFO5$pFbXTsgdx>$KgnsCSHd5z
zAvv6Ijc#uvpd=7L;-;pj__qsJB|b%eY0eoI*+4j`g$KzhyK}12#0;dDUR8^hejaHm
zn<0MfY%!Mreu7o=QD1ch$$)THr%WmyPNPFz$G%ubPS;;GqoTi$Qz=l~uZ>Ym3o+ED
z8;Urp%zFmBkRokQj^nU3!6I6!tMK5`S>_jVP<(kA@JNtyzx0wOei~oMMA>bBy#Cy^
z^TDFRt002{aH4<(*;+7q53+B>vKmYAim2cx$FPH{D?Ih~40lP2(enn3syMO|WSHPD
zKD9l*M~wWYpIuFGyu283(YAZqh^t2|Ub=ffrU9|*I{%AE;iy-*`YrOI5P^3MS^^PdZh@B=2mBqHXJy$-L^-NUuTz$YF`hPmPIo;9NXf8
z`(4dZO>|NS2cmIYu{$3>g(cmdW&|?zwPT!+udW3JINz{>3&8(J~}gEf%*cucuo-=3!fZ=-rG9c5U3}
z?YXnIvCDUL9;`iwwhZe5B;E_B@9^Sj1)lR7fw2onYWbG6RdG6edcaRbc?kdq!lbJD
zU761@(!g@Stv9iU*K(?k6n28AkL=ouUucbGiki1eTp7NAn$G6fc@tO0VW$fXMt$2D
z#)pCP7-@P_ip;S34wAKh*WK~v;?ae0gkpyMW-byFmd5F%C6<8;{E6=9UvG7^>k*0J
ztc4>bcPwSNw9~M{DJG|_5PYsWL;Q?giWqlk68!qgJXOLXx^A?f6^W+aLbtR{w2mK2
zk$BumV|(cXPEr?lAFv$n*Blce-&AI_?mm+njb}yVcAq@L%o@Ibw$mw|NwC>X&pt6v
zeHY_JCVCSf)97g*HX(y1HZqfV@xEGpG3Y~^(M{OnU`uYl9vP|Ce%P45#Z@i|rCT~1
zE<|{@QP-7crH9arT;SlK5Q4VIAs$q@Tg@u2YEU
zaYkM$XNjhYyonAe{N_$ukXcB{jj|hK~tdY&o{c;k{(ayiFjZ)h=v(HZYXH^KFT8NbIgbL$Zm8(aN-7x
zgGI%q*o>7>FX_&CxH`td#Qd@YYFF*xqWusP&nAj=DCDqc13ERfs}9;L3KMWl7*V2k
zmm?0U6)5|EhJ}BJlJ~vQ4rWgASGTem`(1DZ=N62%YP?2l8bwJd4dI5iqeu11dIp_b
z!N8*>^wP*e_=Pc~(8fh0q?~z}o7lkIy%hBrY_l@o_S2Ufhq%~GVS?}Qy_>2d=KDyjfA8(-vZo6)OT4J@j6lIXUY@4uuU>*GiB^l2^
znssU`LCSLpaTT%1j_GWN9t^xKSJ~-PLPvN)HIvsy72X(mk~t91~DV%~+c
z&vx|;w_#OqgVMnQI=dF^@2_6*DU12wji0zv-;@`1aT5jU^uDp9L2bo-S$HSam{aU?
zp+iQ>Y0=f-A`Oe7AYnZpi@!)PJJP+(Rwj2FYe28aU-DB3EmD(v^0jU?TW=vkinf`5
zID!Apn`Q)+>7AiVyV&=@fSE|QBHtCPKSYy>;<>rx52NITOBd50#d+
z3#1{w72RHuWv=~SXJz=i%X*4#^se#c>1tf
z(twrW?6Shq!H+Dy3?#+DSB|N;Tr=N2FbcL+MrY&72s7fS&6PJwJ7~w1r|@?xerymH
z4GQzWrJ~=nG*J-4jxV`AVvO27e#OXaOk_)z0grjo)-Ot@;gyk07HPn`P+#GAv4LsX_*^HDjQJuf?
zUdO6G!rs?7pXo39YQ{}hHE_Dm<-P~M?X6_26@IfJbN~!xDsJnK5?!y;eV*+t6`wtf
zj}|n?T2>0&@cp2JVDu<%BQ4Ordg;p^YlStb#G-vEWKW>gMD}Ggu@_FaBR(*CzJBLt
zQy_6`6)wpWgD&Pbd-OeKkeA3kR66)3M9-F7(^{h;oPL5A**tjrY=*18`Y*7H!vYg$
z8GL=(2fvCfWH>w09&hmN))drD!-;6r*<|_0$G%}u#O0(o(3qZoAg3X=oNh(sJg;#2!`U!GXHntbNBs=QM<)Sdxp`6sU!>A;}E$Z;*P4-OAo3
z)`?GdmTS<=>FleTR8kn;c@e(Q4rLl-i~z#S@#MEYJHxqLdj_eyqVvV*zJm#iRE7^EBt$rOr9mUeabJ+q=uR
zT4sbqZ^t$brf_jo`nTj?&;o9>V;D>&oKAQR0!|fWU{yP9pogLy_(aOA!T*UyXg
zny(qx@h+NJH@2Za;2}jf_2i>gMBmP7d_}w@>3(<5AA@4~qCE}6LmV=$p7(5Pnn}c;
zn*J$&vV$3g#PMxKDcxmU#m6X24&oN}UC3`xuT|f$s`bIfK~MAdGD=(AKDGsaQL{_oUl`164+!I2o(K-b&DuXgb6x|h>3;Fc;dZqX`!j^8)`9k(Pn$cVsMiifQ}*JQ
z8o-j3i3DIX0i~*r;&fMa
zxU4wTntRiaiYrt@V_Rld3I)-z05)G%9TmFR`N~jA4#sjDCeSm#{M~V`ouUZgqaD-y
z0>=`~t+|ajp@k6!A2fITelufxJyW{nUc>G9KmbOU7oEb2bcFe|xttdL2U8ZH!?ceS>Oq#FmdlIq~3BGaQUX=G&)neei
z$@xfrJdX0Nd>KV?G>j_xU{{KMb<)*06ahY?<~BQ~H-Xd~O$zUg>UeZ;8Gq7kH%0VX
zq*0k^TNGaCdEOn6s#x<#_0$1FcGa{s=zwN@BGmC7wmUi_a*3X`HqGs01S>
z#5HVa7S28B=idcY%84bJomljPnP|?*Up8kCCy}cgft=&IP|0ohRh(pX|o`#0U
z?6^w}$IBCaoy^r_jI&ICB75hl<{^<^-;?It*pAWKW;EqtF3
zswcUGY-n9w8~SM0dK1I)ejqA&W*m6t1vy#h=7%f?ebmPo7w^b{PBWJam&_&AmcOOk
zAX2oSu%A1=JxyVjP9JanCQx`$j8J4c|FcHcktZxDYgYK?-ScD3!&G6r^FI5q%17$F
z;ANJCTy2^gYZ;e+wI2q#b;0f~2(bJRwsQ42x460Wz28#aAV{8%1rv$No2@Fy@qekt
zAqZssDO;YX(oWulqM4x<8+!Y`8CKt6B;nYdxbwS6{>$`XHk*bV12ZIVr3%??lk|Dt
z4_7S??+PJbil4Zud*f4wg*G=XwDb2(QJ!Z<7q|{tW7L&@TsXL#=Lt@Tg)*PmnaAZ3
z-I+kUiW;Az8LJ2NIuqT0Q)j6nA*!A_zcOQsvm7;{%mmO@dk6PL*U-tmUf+J#=Swu(F}LM;=T?9rmdw){
z;?6ojC&ht(BDZ;$8}@!Fu#=NN9d#XF*Ko43Nz#=ic~Z`O4CZMdR`KIU&+K5_F!un1
zwH#)+Fg=77yJx?#T-PDsvRZ><$`6w4pHT7}6M08p-nh@Ig!r44_*JCgj~L28Y?X^4
z-TGg26`KhvZ%LU!a
zp0+;q(uZOk)<<9Gr{r*vGTrBfXB@CFG~>}V
zztuL~brWY|U;
z6yXeiyFVOPo$#vMPElbsM&EJ&E?hG$w!Z+*
z(t4ZPn9~>X>}<6pb3~GkgmhAbW&3_^uhcAJgXryo#K%xbpZQrk)+cPFAr)PI=J-i}
zh_=gh2L7;qhPESj=tYDw4!IzQK{q|b(fU$qnnEo)Q^KNL2UN!%tJ0bI+!za+7hhkx
zsTXgbwhc0Rl7(Jmjxk!fm=yMG1H8k7%##Bi17|0$XO&$XkIUy#;M5YM$rrwrbb&q?
z$Hzs6Kz;KgFh}{$JIkk3BFEHJwuxbXA#pIEh}qCKG647JV7~C_PCCq!SkyK>C_YkU
z2{F3&x*xcM-@4R6#72oho8ib>mL^4iIMamFDFqv#=biv@Z
zrEVqF*#IU!xHt9N>9<5N=PqK3b_Rcv0Eri_C59?-4|?g`ksi|}WWq)=(52ab_r$li
z8n;v+YDLjl+F|n3iiVFv+V|#a(Bv0|5!acrY;8wI?s`R*v
zK5a@0SAX)kc$eq_K{tRXp2LxUjf!rrorQd7Krm)nnawOABi~O`*A{sZ&)4I}=|xoU
zG}Fq?w2PVjl`Eg~##$u6Wy4}*{Zv)}gW!eVDX53@KobtMPl#<<0SY+#A)LSFNm3*v
z84`P-?<`;O^VyAH$vSAySkUDD)WKG1sltDX6_Hok?dR(+J*_dDt0-uHK!pkTtH
zxyRQIlOuRQsA-m5;oXTbI^AS&$B!PxtSHJOR8W-9`WlVrZQXYP3fSZP>2)P8goYLp
zpOl)WAl3X+E#jD*u{-+--{#7snpP&$M`TgAbyE29Da8_PcxZ`&i`j0Csd+=8$U@QX
z;bgAVkB!-$FsdY%yjR$N&#?ghNAev}of{AB`M81>je;ed;k2-(lJO=Zk|g%e
z*2~TDff!WFy-hIJ!(lSYJ^_w`vjOkq`7R6~kI$(_csyfUZSjtoekln?5eq_ZYTKvp
zFsX<$?X5H7APM+ZRuh}n)qF!ir9kQ+ORZD*#ymjjeoysNeQLvh`mpHJL>KAyY~Ouv
zlp&8U#$%E4ufg=9@;=>B$^AB>`Hq@=TnJ=UiJWZjd>S3z=_9mk*LIzfuLl}YBt7T@
zg|q3*>YpvC$Xrite>RMf*U-Y=H?{a6!9&_+!4ayhcX6WE0%SS6D4ogfq2vh}7tVJr
z^%iX+S>-^H{biJY-eat&$IKr_)rwHhRa@6Y&4@qjl*4d+z$%t(Yh(yMd}tyxb`6gwff){2l~)eE=F+Sdpbq-@ig+Uk|hMgz~Z@fu6G
z#pXHGbB-V#;`CI=j9x5K*=UAM{dfF3V)j}gdCmpk*ob_s2D$rahh-cu=W4R?GgJdu
zz4tD?S$J}PGaF~4{i!vc$^z|v$>C=@$EyS`P%1~$VO4D_sXkPYCW@M5
zbhlC3HY^KsaLTKODc#@|-?3Tqy@>hs7bwmG2^(2|TzVBAWj$cTJ!jLLF0ZDNb%zlw
zm$6YypG#@s4e>3i$|^x&t;o+YZo!akh%KTNcIviw??PTh-AOQ_x%4+Vt0!}A%$B~v
zv3?ku>zSq29q)YpxxzQA=AoZ=3_a5bSwevut
z0M=B4+^ZX|;PBA(YvS!HGu^k^3htnU6{n(`Xt;F7xw8{EXrvp=OjH-bs1-aSK+I<&
zav6-Cl!KZW6=xmCxSa$lf!66Fl<9k?5DP1RX=4(Fe4fDkR%oT3p@i}4Z)k-@iroT=
zZxh8w(+N~-h-8S^$6+w?4U@i*GaO8%1t>JCPJ4_(sX{;?W6=q$;D4Nh7YrACJYaab
zpbz+Rf^wi2c3(!e&jCHi+gFbb`0d-=sc{-S}S7hgIFk8YC5dxyaSB
zb!6q~h#_|^lPHt~&@%wtuU=e~D;=aujG
z$7V;N_DlMXH>B7(v*g}EQ(k=AFxBw+hux7Ww_IG_mV%2CKeosk%nGyZI8jDS8m&4T
zlZT|$Z$hsZt^+p;Z!OenOyg9GL+B5Dl@g!!V;szboN))NSP{0@0l6X4J~i-OMfYnRCMWp2F|(w8HW>WGFu#9ByXL
zU=KN9E9H#^>#nD+LVFSx-8SfUa;xQTctGLfP@wkS*ru_JOXrD&R>1^(CxU#hV`6(#
z+;#wcXd-ny)3R8wJ+e6eiL@Di%7pFBEA$Wl^6YgUI7|HSTjKC$GW9Ln&jxY*W7;+z
zi1QS3$;V&9Atr3w5wdfpRYEkVCdJ|=L1s@rb@b*?gJkj+=lf=Wz9z3*22ybi
zIXvbu1%U==ySFtXL&ujs&_Xt#!>GmmJ0<3)pg1aQn$c`DaiN^@kE%U?n_;7`7o9a?
zo31_9pvcj7F$gH9lQ^NG!%b%+IDt13QLtHl_`ylr$+!#jOq}TWOD#6V0}09@^xCr0
z=S%8)rrqeJHim^a65njo5(axL(sRuVZD!Bd`g26R2_mbmMo@~*AzOD8xOL~su2&*+
z8ApUXa9{KuL<;qmC+u2(Y?X{vtcxwzSH)i{9iG>I@xI+E;)Q}4nIA@}RX9!BsC!?sI~i_^I&lry<-Gebs_JZfO&CghFFH=qx>TeKC3HPHx(n#690*
zIOiayL!W%1L2=Y}btsUOjS(87Ol1
zL6e0ZHiX1v?orrLVy8A1HM*>n_1d~9VvgG$n-e9P(Bmg1s;HS>J50u%3yL?@+q=$@
z`+c`}7kW7OG>aqb9sM04Oq<0DGdydcid8v-GDY2D76(j7zxWZ
zc!wji$&o_(R&(}$uex2hZ@Zxc?D1T3ifNt?ccmvAWmz}s-i__xZw6C=s%QLi-v%=6
zE!honxD6f-{3K3Z3XbaY%F5=p5Er)PH2?Iiyb^Wm%!h`pd#gTxG&9>VC*}X+>kY+=
z+ihUJjZa8X`%&!-m6eD@Wu2VHZ4vBMf^5(V@WwzIkxICK2+wcQM&F_kPOC4tg+C5e*(d9!A@@E-fwA
zak^AVYx}N$6&}t#l96K*V7NYTx)<^*b;bZODk`qxvjM&5bSUUUgV)#gK)&hb8h;sx
zkE&~u$dsj3Z*d05;kFP}ciOL+C#{Tw?ez&SqOb=ezR!lX{JdLz^tKrN5qHgvp|10|
z-?{a)x8{uqspGvulYWMcJ$Klr7Mj!CF5?D0fEtlRHNaDkbDjYz>(E9au%L#k>q0W-E&o
ztC)$*x+e^`hnr@DB4gqgDH%2T=N{i8@~gkca%i*Oy=kM$xUR!&-wSN7GMecBwmheQ
z|Dgs8?O9205xq5dyQqeh^AM7dd1e?|DnXwT4-(1B&ka_W<^S=gAa*x$hZ>E+X!y_LiBO&p@S$&xkHXa*~~zh2PW7
zH)=b1+nm%SK&J0UzQ|}^V~OrOvWY
z43~M&ZjpSmChzUDvc8OS)cS6^Z6NH8JjD$I^=XJ?Fbp^3jcxy1vaN2W4pyCihHb3V&m2vEQ{R}eq&yOQZmVdzYBMHpI249Sq;V;fLi)TtiE(Gh
z!O=i|pc+X#t5wPJwuqZb*Zjp
zM@xcNm36}XEy?c5I#Jz+wly|86x4_hS;xpWWa@dRsvVdk*Dz$nTi&OC0}d+cyb*8Q
z#=(!givDg)V;)*^N%6d>wz?&X-+A;l=3NDeFFj(MXGnaTp*P~L2rv_EpHtv08kUH*
z;CalyQ>b)&c*w?Cx}H`b)|0rv(-j%uTph#{O^QNXnLiz;=twNKpi8#QtN$k$#{s8Pw
ze(B+w1GX6=jQc36se)Iaw{)(5i#bBGvZI@S&e+ChBNs0F_O`#2h16nkdC~@
zIj!n1hYa)Iir>0_?Y)w_svr2P80Yve@|`7S3u5x<7u|8TDRSSL$WA|Bp*?vGs(%(T
zyJ_Ua9yF)Q*f(jeci69PFu6Xiz^d8U3gDLbka-@SXLqu&BwqL!8IdtNxbeEpB$1kP
zB}ZY#f+V&%4?v+p^avh*CzVQ<*Vi=^r~_%^>Q|}_Z%O!nWMfH2`hjXS0)=g_{OzWk
zA^)BP{J6LVGOkMiHFEYqflIwj$u?hMvV6#|FtZ2D!s;_cDYLImdGwarsQ0tSKB9~|
znsZMFQ6GZFW8}}NBwl6n2+wCVFOku@3&sT`6(vAuMC+xd0^ZK69n&yJi
zC$DCI*)${J8%sf=lzSIqy5xsnZA_LKi1y25McW0__diThmiq*oiXy^)0J`UH6^XGD
z!Zs1ETh~oS`B-{qEguZOZyQwarR)`M2?CM|qQ^ro^FT7t7*!-C{5fzw^*zB&*?RdC+D
zJkv4+=_d^tuBA-jH!^-{AbvbI3Fd%hXr0Uh9Fm;WR*(GR|Bu(37r2<GEl0(~n
zSt^o9)=fukZk7545J{$P+McbKZ_Afu_-I?bNHmI6bK$Qmu1GUKpx?zz7*4n?BoZgm
zS2JXqrfw!z%_rd$&`|M|X--STzxb7;pE5kQ!M;|0sN>&al3#bzoExvFmgQogpczyi6F
zfxP^gRju#W13R1(BVV0^ls1qr9MUCBGYHBC){XD|QPV1F4w66ZD12ePvm}{u1=IKvuWe7kLz_ZfRNNKKZWQsq$kTLp_WX;#o0gF;&^=Z)PGo*Bu9h#!-`hjjPZP
zc`dS)M}6r0S`;a`Nu2HEHs9Bc$)C}IGjNG769-v-QXG#yAjXoX`Bf#i)tLywOe$>*
zu~r9DsyRC60sFnWM&l%>r21n!jac-T+RSo&gk=Z>56kq@fvDSttmqRN7f9w()$q`Z
z_tlneSvokg<~2`na)B(_9$X@S5av({@59`Zp&-#AU0at9XvUZ*8~pTQi-ZI*x^3$R
z^Bl)C%V&GPe@#Es3*+tntOjziEis_crWc*$zVQoq$(fy;p)MqtDLkkA0AqwhIMYnD
z5$rBv>~&!FW%e!UfP&sYOQ^#WocEfq
zdT@@~;a!=Dnl0h7l|PW7r#@y}M&J*pD5^b-SMAS-r=RbnOD}=s1%T9-FFb2JHuS35
z`?b+fyK;kipZnXl
zcb03jawvPl`ue8Yk`F@ucr-Yyj)yM|c73*N-SFGXv={Ckt20_@{gcqus>&eITIgQI
zg>Mut+x5z&y?GFuvpH;t?z*P5_?tf8n&^SoY8i~@L~a=81?sPVc++@+D5sLwhm`9S
zk$5$2n#7kyt%C?^S1ttg;`AF7DJ|RS?2^f_O5OY6OeNIgBJQOc&sDYUs
zk49-a^kZ=>Z`mYD4|-=efjH*fSbdT~$5A6yxO7CXkFb)uzZlW_6X+ih(6&F`RnK}s
zZWe>)SYI4}aXF9{F!a`t0LkKOhkY4*FzB^Gu#o{834b;?dGRP>w&_e8Y6}_uhpXvr
z$gDD-jx?lx2z8-x*B7QHjp6?%Hx0=0Z2rN}$elq;>ql*D`efROM47FJnOHL`EA?x6
z{3pu=6g|LATV7Mxs?dbu>)ks_V}Befma}qt>CFaz(sKE5;3GryEPCRUwMIb|(iY;x
z^>4DXCUhGH+YGP+E^L%nvl8}|Ow)dm3`bAE>okEJ+aSkxb;Q@hd`!G%Xoq`1G}b2J
zAWA}S)e@f)H&pj^B?sP%gCH55d(W`Qz{sfI^jP5YrFuP4C89}`a1YwDKm5PV0)?Gb
zP@XZjwsCiNcXziJEfjZmiaW*Ofno)UYjJmXcc-{Z(cJ+
zy>egcZcP0xZ4y9DAT|@G${yA9#3@Ow!qvZJ_wRb^h^&Ez7?VY-L@u&?F30$`n1eT^
z7e1-18)>wz-o4O@GHcY0gI%?zg%SI0w0I>{OeN0*y~il);1AtS89=?GA!y8$;wt3mGt-`*KT|7gx*
zxe>oxQ@u#FFGX^@>6j`~*5pi5M>@-7t_lrAj^u9NV!0!@g)>c9a;}Hym|=nRpj$tJ
zey%DnlqFN9k!AYsI!Z#DwT{$<>B<-tp7PIYRzowu)`^7OJEd)sk(*QSzkcNa=M%jS
z>~DhivjlYGyzyD|+JujBF&{`mT6seYZe-xjJLU|tMc~J|pIe>8TMRH?Cujyk;x4hS
zyV>IdYg4(2^x!zk_NRn*-YNDtPQY#R(O8w*A|V$Ajn@~~MRH5h?wfcAW%|x;z~B(l
zAa44d&rq`;m!W#ppaK~ni9Qf3uq9(K-G}9RPJ!rXpWQo5|&4KUJfYyXooX20s_HOEZE=jEqdK<2vg1m3w`y@ct8OC#FC9ePx~^v3SXWAu_@
zV!cJYAtp=Xeinp(vSM?`xcsgM#Jaiu$UNVrIwgVX
z0y14Wm1}W7WRfL$9OfFoBeym&%d)#5C5LMTF65Q*oheF6^KS3
z=fx3a?9;y)U|+c*AA}Hg(|R3Krk)=ZeeA@czDO!elE278V3WVxBy&x?B#5|--K&lE
zexRd?jmh$ZB?=Su?L(lR7@d(@D(B
zaXTXX&5F)F!HZkyL(R%PDzZH2XP{>nDQl^Uwb;mPe`ZCKxwM_5uvlh=x~51yGE_oe
zH9tj7*W$$|Sx>08aZ<$iQ?emrd9p%#I@h=r${F{wbV)6KYYUv6gkz>cg_U
z)?zwB0OuLzmqFlMDZd0OTzH0Efa?}D!?GT5z#;J%b$ov+3WYVIu=C`*&nAkBhd@nZ
zh>!iMfLyG+iLZ51mn{9_)FsEl3*f}yq*h`o^*)6ZKDo;Lnof-jO5%&Ok#K`NkwR<=
z1o+MKVkA^ZS;)`t@4eZc=wjy-j*PZ|=N>lG5^Sf}v3WD!F570>EsPQ`n0RR=XgJcw
zEv}p3pI$^srNXOuJ_r~Br0{`3N)d1dui#(Xzz$-R(GtF4ws?qmsS2qY0vKNA(kFNJ
zxe)MUzGO9*)=WMsDI7FRR$(kovCr3`g3q6h1|_(oqBSoe!rg@5Ujgl%HI{ds5cq7$
z&z~>*5QjAQ2|0{?Id+pv?-jCw2%L!v%va5-hBQRQtg>%nIfp-VgVbYJ_S9{PkG&@Qb&^_X*C&ynNX#Yo0B!L-MMx?h4y#@|
zwu#NT$zUFARhV^4pcR_RaM@~S(SgXIX64+*HNdcr>7`?+L_rfpvlo@G9%^j6V$UV+
z@907BGZdKZ1tQt@FQ)c#4mex}r;TNuI6{P5P0u_Oh$o9_wJHIqc1-5if?||7<%x)j
zu*{EvyuZ|!iWLVB0GQONK6?*ti7_LMjvOsx?Dwjgc@eNJR_bngj46x5rR%2J<9ry>
znW*$=Jw1+Zlnp#BO8jj|DV#fr!$}n!1A}y15I%k+NE9UkzRgp2-*r$d%Am5qJ$lrO
z+_APb80OT>_9TND!to5D?>;PAMBm1@ic@dbHDIt5&wi!a12F}12#OXvFfm5fXy4UZ
z7>Wb5wuO*{I_*`Svx*hg7$(w;$kzM|l$LA|c}mys0)D;vaR~scaLZ8@$Q_kgSqEL%@T^&(_xHS
zlFX~q@T-w$KqPsle3w7@Oh&1mz91;T^IchZW@a!f
z1^ys1QP69=plYI4>@r%T1wvx(enq{iEa~tzAH?~##yoaYf#1=&mHFoQ(Z2H4dTe8=mv=bi06tCdw4$sta+N}lCJIKy_TbF
zU+y38TIFU6I;y$VewjVSVGFodL-CaqGi9YvlrLV8dhy-NbX;ZMKL{uXnbiK&18H{?
z@mp=oflK58BRH)=)B$2rPYD%8)zpg9e5K*>U!iI-JD8h`gzVdsI4Ynz;7NFu8gN=P
z!M$4n)oJ-(NamCZtz{;3j_qYK)QjG~R<)K2a;?{aR5%M9cH2
zPM9P4yKoE!(e`Sz=X9m{puX3n7hlAV;;Uz;=|B;V0xhRSyOO*F
z448n4-}1>`eYALft++ULcg($(eoFG+Mb|kuNF6!c6M%4=&~`}BVCB2TWWe$G!qXzt
z%wG6LS@=z^*fJsOQ3WInDIY5134B>FBoAN4?p#(vEPKDwxFzZ=`T}Bv|C0zq5BuKQ
z`G^j|o^2Bj$0c=N?rP*|UcKP9%jWO`AP(9+RRv0m(s-KPwVc&Pm=CWEj^ir%eS&@j
zW#suhWZU8mooY&Im)Gl>APab!?3-28-qyKLP4@@ww5}l_;WxgVPh?`iD;;7beEFHi
z1CnAGg;YPR%o+{E`N7@8ubvpxO~w_$VVs=k$;)Fm!96&}J@$(M
zhZ&9N!Pqd!Ebp`i4Dn(@w0sYM=xJ%~_Dra^o9Ali8QCszZ^^K=dPOgr<95*bxz|^R
zVukd^08COrL`&?6eO+8Xg45?bjNJ*J+$&u9)ehmEQ!j$hyV;C@ABRT=$4k_o-|S)%
z>ZYchYI^0QiMJeW69QCa(d+K2`jK$KhH6H5%N<1@ezdt2C{vc
zAMVjJ-fZ{dJ%Bv6H@eqdRKeoLXWYO#l+)B42xC9}PPZD`f9kIMs~Gq=qxCE3WPaB_
zEgSH3@~Ryl-ytNn-HN>d%1~^U@6W2LjcqD%TJ4yX^8VUg%1tS7M$Jr(=N_}U14Wo?
z=9#^VHa|lAIfMH^pQMl(hQfyE>v)yEp9a{(nDboC!iB#tD9Ni`^&qzu6`{+*qKxS0
zxW?UcboT~HlhMe3G;#@8EQY1ON84MGy%+AgWkDUC8M=S}_Tj_;gha#tM&}LSmFI*o
zw~aNqQ&wCN$YfNP;sldBepT-_E+N4>=U_ADPlc_BD`&GLZ%JkgdtS9-7k`$3f-sk4
zW&gPo-pRwI@~ShbNP+A?V39Egxjs4=rk25~Je`e#_d!MF=y`DOmXzlD9@;!5piWte
zy-CuWFQihm=oSzO+)GZAEs9#_Tlk9Cjav`hNA9f$Z|?}tWwE$$AlC?tHwn~d_ZSuR
z#v!&)#*Xb5mpkyjhu9YGF^`5OL#ah7y^KV^5XZ^PfcI-kLJO3$OA2B#9Kb;guwf70
z%~aVE4^k#8B5$*s&|@pEKb@lhjc6xDd9Xb<-!F}~lCfl1$EC6iVAA~N?*$Ick!#Nm
ziP#MsxOubGA)@~$*N>f#KV_2vf(DJBo$LQ`{m8g^_&8H~vmoh!GkrbBHNMrZsp{!H
zBpi0xtJhrZ9P9L^xrRAe4q^Ej#94`j-Fat%!6ybGBEA
z+!*m!L`!ks7g&QUNP*t4Fh=bjI%rn`HN5wwVFlKBTMLJts@iv|qSqdzCW$=1HCJq6|5o#I6mwoFwxuwPjBqXMny89RS&YE)BMz?K(2AdO>EH#8VRu
z)0TA4Rv!ugh54Un>o2y#VIDjarXQ$ajJqcOQP*bJUl3~Gq#!_=bePHK0*TRh?hWEs
z$~DLaeQ*bHIItr=Rv`qZBZ855Npvv;y)dO$=q*~_@j8(WxmMfam$*##0oFlUCa{*!
zR3;W^t8}2mKtF_k!&o?aTt71)oRw!JjoGHLiQ$0cLAihlh!~AuPT>(`Nk29M(xG@x
zt?byzO6e$OWpi-f3t-vnmT70Bb5tOLOfJZ_Ykc96lnIh$R3K`6K^zrKhLqW0P@Okh
zz0s++#S2!t*_l&uF3QvW#Ln;bE{0J~kiNns8cr(LgT+^SSj_`kA5}$W>o`U;=KaT*
zlNLa$HO~>MQ^VJkJ8XT=u2f*KHjq`CyAtE6j3PqrH=TEAjnf@i;Px?N35|durCfN=
z^H*ps>$mY4Ywk?sAvL?Nk{vQ&t*c=+6kAX;^5v9S`obYLx=){s-35Ye1!M9a{Z6sq
zRsqSMnBK|?Q$&Da
zVE6c|a0yEG`59b`K-Rqsr|H<%S~TA~PbtEcn?P3vmjnqeshFN~ym*Jt2keOF%1zFm
z;VMZEcN-CkXi-;)Cl^U{vCM$dgoNS4!Ho?<`_4G6)=ja@%{$TK7i2+GKqw6tl?1>t
zH3Dah+Vy`yaB7G|G%Ato6G?TlO>2{ecZ9K;r|JN+8E7|Bde=bq+&{oAX>f49xN{TY
zE88E0CA+v1F;+I1i2Q~LAGGvVk8laGQ#M1iwFLj(Zv!(cT=MOoO39JMcio4FA+15!
z)7{U6&km^xzT|gB8>u&vXlb+5V@QCXtpJ)B&(8+NHXQ1>oAQ20EmkQ_&mU`Q!1W4_
zrH9K;(#ANFG%-Qgjeh1H%Lc&rMrY2t6}BdJ6KoUu-JiIVL-IyXBH}5A_%kfXOPC~_
z^6F*V_4VV(S0wv28_lTO|7;GkkNFRCvKXMcTO!EQKmMBxBIW09Kd;O_^L;F)Zxh2=b5BuP=_X8$r2P!rJ
zZEmEmi$T70OP~g~ITVv;r*7LM@cbDDsYNz=4#)1@CQt5tvYA~}hL8Zwuc4CW%
z4cbbcDR_zO>pL($zyO_|6QB$3oBGwWhJO#0tG4JByjHDBdJRUkbMnu9MO&NS%6EVtc7v{L?K{jYsN+1r!~pd>B4H}$MHNzwdkDDZW)gVR{bZl@2M7F|f>VQ(%Fof2;?djFg2$Siya^o99d%Nse+j_H=JChwBrJ91
zSt|v>oJ6d8FApoE+ks0z8M0Rr_;iJ;@E7#Vf{%N-uS+*NghtJk#<+o^peGWN7V?x9BOasD
znrfs0tm9mh+u?5P9npL~ApdexJnby{f$oJRFVh9(uJ4m(hC!WqQMqW}6PL_Hl+sGg
zSX!LyhWk2yEecF34MkVity~F=Vc5tqL}>=3jaNXar6u=HKH<#P1!5)1rHOQ>#CZ^#
zoV<@?Gx$m%#!g!NfXy`3LBypJ+h%e=!U%<`b+@xx^%F2wP)EdZB8*a~5876N
zzvSw+;7ek{9oPG%=5&lx<39fMi81HW*^$(t^*~2_lr%tD=%f5
z^GnptWfoFs8sKx@f8zEh(}dwF_bF{k>Z5OA?LSM$r~Il#;6qHc(GHtnONu^lruJ9J
z34;YfFrpO6r0&rF#zp@7kSz=X?H(_G?SYC$qK^+aCWd5{f?26e;EGhJV&)u)UooO5
zymfVz$de4mTjoq;^OlC`0=MU0VqO`zp!XtSkvJN`Xevyys%$yGa>BK<4)1Uxd0{h>
zYe;6vTM)OjQBzXToF|Y|0+Xz1L%EQSu|@?RWI5-MN=dMy#ch){WlvPYH-`_o_*}sG
z@k%GjVeXE^q%d835|+5H*e?=#_T6c7<;kGe5gtZgTuC8WkXkAzlG=2jQ^Qo`Z?Ypl
z#1G67cm)E-hS(bKYFAHq5}(Lb_R4ynB4v`!U}nV_KgLw3%K$NrS(v1d6QtRPL?Hk*
zMMPS)xrxhdCr(#rtodLI$tvI;EIIz1zXjGO*)Fj*d&tp7B{n>tJ(iJvo*~`KQ@<;Z
zItX%i_H;c)j+)tp&(Vd;3q_)gle1I_mMUg*oN^Py{VV0FUH$AY6l)VWIE_IVw2r+}
zSIj1Q|06M}kBVD&OF!Y3KvF7rgCYQC25l80j2CZCFS|jBPnmLzml^smIA9~5LN@|(
zCZAu`mUU*98Bv?-N4Gjo@ImLQ7
z>8;AscE*(j1?5W!#yaYw<-$kH|5C@2#^bO{O!N#kly$W7A6QgDif#t65V?r5wvXUv
z1$&XnN@KxL;MaN-Tx_OgOm^8qE~{AmbH^%EPaXv=RsOyyb^+nM>ep*lJ%uX
zKTcL1yP%)T&ydUAm;EXf;QK_
zDzh_CPvS@?TafKg5lC#iTBRgx&kAVJr=U0p)(R|I9C_%ZEXB=L;bH<)>8;OuqGYeL(=K+NVtRCz8uy2KRz6=7lF&Zb#lHMx~G7H6KdMNdg>&_{7QD
zdi?YVdEv&r3U-x{80PXyWih|A7?->2M%W@PZfxK!q{Ep?eu<_?8=@iZB_92@AkdJ4
zU_|^>0hS6uK~NV)b~38pSLSaCV#o!XL
z%svukJShB3SR%CJ-;LmNfshiR0zMeLNvl|(UOtVEZL9b$j^9&PZAEd~dilk=S#XQu
z-tZ|ss|b^y{shdNHwUTG)AmvMJG6h}zZJ)rjn22>-tqn`DWKrr`G%(_0&(bDy^cxC
z0g3SKU+!M)_E?~83-{rY7FKun!TB4c?<3;XQncrMw$N@hB8!
z2Xke8j@pr5ucUmjwRYB~IxgM_tU|V!feJs&te6K9;x6yHp8ECRVi_Sw|KGd$@6Amnh6WSugfVM&uSavZGN^zop=FuFDz{{IJ2iD-sOqdfYRW%p
z$M|#u_3j4OR`&xXF?bz)W$kE?_H+Lzf5abI?h7D)J!1*6l)o}`T2tX0_eea3jTdh=cIGDK{cKc7%WDCs+fQmfWd`>);#Vg{C@R9EGojW2D={zbA-*G~V&xq)9B
zWAz=rH#^X+>U|XJ0U_Vv5iu{qXEy9>5Mt;W%&O}!7Rg13_x{yTE{%rliv9TlN}ic@
zVa&er|5!O$3PaYtf@zSa)7@^LiEyFRjp6k99cE`=<1TI3ggl)#a{6^C$)+{Q
z(aY!Z^{%}0Pcz%qNiZGNQr=NYl?Q;E_a{!b8B_?<;I+=@AYk9x{ngSWWtq`yBQimq(KCT+#(o{YmHM9!vm4zkGIKT%%scavSFRsky7z)wZ8C?(%>jH
zAI0U*{$mIn`yF;33EaC+&hg-$lc7@RX1pA^{WSRNs+$nL(oEr9n1NU`#|RMSXw?1j
zCW||rZp!Y&KT{^YjXF%u-i<1>lx`ty!coaucFxmW*ND1FD*CdU(>_7S#1d*UvjdXo
z!QAvDUDK~-z~e&xd_!H^ua0Uqtd7ZbJ?>qVcN#fo@6J3LcRlrh$?vkf!~NYofi0MH
z|3a_bB=P&evhaE>3}vd9MHN_wYwH2c8^;ZZPY(~(P8g}_YbiL7CQ*W4)1IN`kzXID
zDK{p;`_+Hk$L!_@uSCu8!@aY-Q*7=Mq87L?TQ&KNm;N;`{{ZQdZ0a8|$fMm#vsZbO5Ws0oAb1kJ)VG
z<1|e4D%X;YzTfZ^eLBnm3h+HeuSE&-8PQ>2Q>*Kt9BJG~e`?YuG$D!`N9C*FBMU<3
zQUuLe#u*w!{AO$z!|ERX9rc#KX{PfGQL@vkX`^SKCDM!1>r^YYUN(d!)A8_HG>lu@
zg>^2G^hU9;DuS&&+W`#*$!U0U`gVNo68@zN+*OyOR%{}bGL`|lZ%1k&j*2DU9w}du
z;hj{x<%Vp>dPNsP7AG`uKOK^C@!68~)9kqYw!H1iqN^;szhl4hU1m9lDb)RjN_P9X
zMPou!`z?xc&$^&9&PSDbcChbA>e`5Jt(HP500&M@DFTaq$~9#_JQH&
zUlc_graxjB2e1HABzQ3esiA9kp9F=B738|~EkcYU#b8akba9XM?<9@+lMU?tCcd=F
z2YgL6b>WZJyu1ZSv9#;)Am0eBL<<4hAfY{7DX9K04BDwRomJ!~p5+Feba{rK4jWf7
zQ!)Zi(`+2eM=4SjPDplE32tpuBQ&E+!HgY*%?7F}OizF-V8wB{7X8oLr)WPDgSsRl
zc+$e+eSrDq@6w$$xqrx%V=Y5;I~T%WGTVH2^y({p@hh(j8S?P#>yp5*LU5%IT%)}9
z@((sw(qMbO62m;^s(>Q>u%^1giLjR(`!{Qy6t2G=>ZR1kqz#P)Ciit@6PvKsCox6I
zvYhF%fD6ZG$vXk_y3JVS>KtttNX5(%RcgJ
z2#o_nt9NcK!E;Vx-{M(QK10>^Y$Rg*dhZ@;<8lCS|1DDgTZEc(s_;Chi*PC=W#Vsy
z|34&64lUI7K9hFgK;F!HmXqpr$?Q?+>A$dEyqPg93C#=v#MF!Uz7OJ?9#S;6$eU${
zqk^gb8mDYi==I|Gh3*YwzquC1K7dx9!z#yDYbflC`W2H#_2?WW2)dfhLs?l`2l9Ic
zzkYE8Zd6Bb@93$Rj-Z2H<2JM{=3u*rBMM)34QMRuB5tLnqSq1qb)$}8f$O8^%zZ82
zQ;0y;?taPIzYJ*)$~=xKU#rE|8bWoNrEd(2_P>ixoY>8l>sd{Hz^ZndasN*>nv;`{
zjE&6E)CNIF2!R#!J>?D+){~W;o$Ir@3>yazGaEN88yhV>0;__fxul7k1sR>B00$d8
zKihwwl9GW28xG0&S&imX*e1f~L12}(c5!thW9Q`j{|Yw`CnwqeQMmn*WF6Po&_XVr
zv4w`93Brl0#L#lSut=p~DflpWqhXMK;!35r8erTKpBC>SsC0iC_9gr6p
zgq}EK7`58XoDkvBL;-MiRZA3oKO@xeFE>h)P}w+(Uc;f2da{1=Rqp-;{?7xu1ab9t
zMtkbY!`Wvo=~35_fXe9aMb?IB)7_>!bQ{afvDunk)twx;b_FAqI;K=Thu2VzX@Ph}
z!A9$A$4154pW8%U$`f8DR~?<%+MR+Oohxy0SXnmZ*a7hw_0O~;Z8y8+Z@7Ee#Za&K
zM#x2~MReQI>>858bkX59gNvpm?B=yt1IZuTT&_cn!=RS)qM4<$B|L38)kBUS>Yj#a
z1${Rhx(q@AY_^M5UUCAEgZ&7ai@PR+@K#$M*`CsxdFs>LW){Rb=`Hemn#CCw3_*){
zwD?F5n_i_gfhhpLRTQ^r)DT~5XysC*GVHc||hPgc#)remMB}&-gw9wV4
z=Mr&1qODW3KRQBkLI?ix8V$p6>?Z_65^fc1aRjfjlH_1Rn6lSvQJ+z&O
z77OoI?f&wsSB^Yn{RB5!jOArP;8NR7iwXCruu`OC$_sEc@2Z*Q`+d>TG-c!TIXy76
zJ)pWuq>cQUJd}LstI_EzRm`A|IzbgO!Tap|)Z_J2dOiwd?vsp49uszpBg)7wAr@-1
zDPXp-VE@+UL;qZkd9gwsJ-5}~q<4Lu^4C@EU5E>H`2Kt6p3l9>rk$vp`&e}02V}QB
zO!j~FKf0@%i-n2(f13Z<_MDvjDNY!$WB?}@n*jo=nzgqD*=G}hRf|lIos5%={c}*&
z(b0{Jlau!|QdALGWgWgbe)j$oH~;BLlIaQYi}OhHbMi}r#6f&~QXJg;AZczkDJd?H
z1c(#F&BrG~_PorVV1&{gN8@WF;+1Dqx!v<)^w
zlX2u18M$$0sfa`l1dhf=6--09X*3B_Q(vbl6Xhal}-ha@6ufZ)2m%Lv4*U=vhHY@^*Q)u)D;dl)b-oesgOz_
z=6R4Xml!2iufA1~7lyadZE=GbSCESu^S0^PQKDF&h7dm4|42;6f=x
zZ3wn=sOMxT>|{&NOeUc=q4qESZvM^C@SD@z@=a5%M|$mrj~P(xyaj7^*>xpb@9b3X
zTml{%rPR$X(r!t#VdL91{`g~<)MUph%(-F+Z>KWU0nf-~?JDI}@YT2~$!X`85k^oj
z9@Po8??Ziz^J793M(A3rdW>>25Io21FWWPpzeu^gWV6#e;&^kjqqj61Pj<8;%=22B8K9NF2;ZMDt<*>yH)vU^Uwn@!SU_S!!9
zcv0effx7$d`S=&FmN~hmHXKx
zEo31n-J94EgaF7~GkYrehb0jI#eQ-45LS*X@zfSp9z41u-hui`09j~xKuop+1Gy54
zqtaY=v?1QJ5{4iE!V*e7K$vmCSt
lD&hPY65To6ci8*j=f#vBY*=C>Hf|m^P6TRdDJ5xy{{q4YO==this.size)return undefined;if(i>=this._buf.length)return undefined;return this._buf[(this.pos-i-1)%this.size]};CircularBuffer.prototype.push=function(o){this._buf[this.pos%this.size]=o;return this.pos++}},{}],3:[function(require,module,exports){var Connection=module.exports=require("./base_connection");Connection.prototype.setupSocket=function(){var connection=this;var socket=new WebSocket(this.getUrl());socket.onopen=function(){connection.handleOpen()};socket.onmessage=function(message){connection.handleData(message.data)};socket.onclose=function(){connection.handleClose()};return socket};Connection.prototype.startHeartbeat=function(){if(!this.protocol.sendHeartbeat||this.heartbeatTimer)return;var connection=this;var propertyName=null;if(typeof document.hidden!=="undefined"){propertyName="hidden"}else if(typeof document.mozHidden!=="undefined"){propertyName="mozHidden"}else if(typeof document.msHidden!=="undefined"){propertyName="msHidden"}else if(typeof document.webkitHidden!=="undefined"){propertyName="webkitHidden"}else{propertyName=undefined}var windowVisible=true;var focusListener=window.addEventListener("focus",function(e){windowVisible=true});var blurListener=window.addEventListener("blur",function(e){windowVisible=false});this.on("disconnect",function(){if(connection.heartbeatTimer){clearTimeout(connection.heartbeatTimer);delete connection.heartbeatTimer}window.removeEventListener(focusListener);window.removeEventListener(blurListener)});this.heartbeatTimer=setInterval(function(){var isVisible=propertyName===undefined?true:document[propertyName]===false;if(isVisible&&windowVisible){connection.sendHeartbeat()}else{connection.setHeartbeatState(false)}},this.opts.heartbeatInterval)}},{"./base_connection":1}],4:[function(require,module,exports){!function(process){var Frame=require("./frame"),CircularBuffer=require("./circular_buffer"),Pipeline=require("./pipeline"),EventEmitter=require("events").EventEmitter,gestureListener=require("./gesture").gestureListener,_=require("underscore");var Controller=module.exports=function(opts){var inNode=typeof process!=="undefined"&&process.title==="node";opts=_.defaults(opts||{},{inNode:inNode});this.inNode=opts.inNode;opts=_.defaults(opts||{},{frameEventName:this.useAnimationLoop()?"animationFrame":"deviceFrame",supressAnimationLoop:false});this.supressAnimationLoop=opts.supressAnimationLoop;this.frameEventName=opts.frameEventName;this.history=new CircularBuffer(200);this.lastFrame=Frame.Invalid;this.lastValidFrame=Frame.Invalid;this.lastConnectionFrame=Frame.Invalid;this.accumulatedGestures=[];if(opts.connectionType===undefined){this.connectionType=this.inBrowser()?require("./connection"):require("./node_connection")}else{this.connectionType=opts.connectionType}this.connection=new this.connectionType(opts);this.setupConnectionEvents()};Controller.prototype.gesture=function(type,cb){var creator=gestureListener(this,type);if(cb!==undefined){creator.stop(cb)}return creator};Controller.prototype.inBrowser=function(){return!this.inNode};Controller.prototype.useAnimationLoop=function(){return this.inBrowser()&&typeof chrome==="undefined"};Controller.prototype.connect=function(){var controller=this;if(this.connection.connect()&&this.inBrowser()&&!controller.supressAnimationLoop){var callback=function(){controller.emit("animationFrame",controller.lastConnectionFrame);window.requestAnimFrame(callback)};window.requestAnimFrame(callback)}};Controller.prototype.disconnect=function(){this.connection.disconnect()};Controller.prototype.frame=function(num){return this.history.get(num)||Frame.Invalid};Controller.prototype.loop=function(callback){switch(callback.length){case 1:this.on(this.frameEventName,callback);break;case 2:var controller=this;var scheduler=null;var immediateRunnerCallback=function(frame){callback(frame,function(){if(controller.lastFrame!=frame){immediateRunnerCallback(controller.lastFrame)}else{controller.once(controller.frameEventName,immediateRunnerCallback)}})};this.once(this.frameEventName,immediateRunnerCallback);break}this.connect()};Controller.prototype.addStep=function(step){if(!this.pipeline)this.pipeline=new Pipeline(this);this.pipeline.addStep(step)};Controller.prototype.processFrame=function(frame){if(frame.gestures){this.accumulatedGestures=this.accumulatedGestures.concat(frame.gestures)}if(this.pipeline){frame=this.pipeline.run(frame);if(!frame)frame=Frame.Invalid}this.lastConnectionFrame=frame;this.emit("deviceFrame",frame)};Controller.prototype.processFinishedFrame=function(frame){this.lastFrame=frame;if(frame.valid){this.lastValidFrame=frame}frame.controller=this;frame.historyIdx=this.history.push(frame);if(frame.gestures){frame.gestures=this.accumulatedGestures;this.accumulatedGestures=[];for(var gestureIdx=0;gestureIdx!=frame.gestures.length;gestureIdx++){this.emit("gesture",frame.gestures[gestureIdx],frame)}}this.emit("frame",frame)};Controller.prototype.setupConnectionEvents=function(){var controller=this;this.connection.on("frame",function(frame){controller.processFrame(frame)});this.on(this.frameEventName,function(frame){controller.processFinishedFrame(frame)});this.connection.on("disconnect",function(){controller.emit("disconnect")});this.connection.on("ready",function(){controller.emit("ready")});this.connection.on("connect",function(){controller.emit("connect")});this.connection.on("focus",function(){controller.emit("focus")});this.connection.on("blur",function(){controller.emit("blur")});this.connection.on("protocol",function(protocol){controller.emit("protocol",protocol)});this.connection.on("deviceConnect",function(evt){controller.emit(evt.state?"deviceConnected":"deviceDisconnected")})};_.extend(Controller.prototype,EventEmitter.prototype)}(require("__browserify_process"))},{"./circular_buffer":2,"./connection":3,"./frame":5,"./gesture":6,"./node_connection":16,"./pipeline":10,__browserify_process:18,events:17,underscore:20}],5:[function(require,module,exports){var Hand=require("./hand"),Pointable=require("./pointable"),createGesture=require("./gesture").createGesture,glMatrix=require("gl-matrix"),mat3=glMatrix.mat3,vec3=glMatrix.vec3,InteractionBox=require("./interaction_box"),_=require("underscore");var Frame=module.exports=function(data){this.valid=true;this.id=data.id;this.timestamp=data.timestamp;this.hands=[];this.handsMap={};this.pointables=[];this.tools=[];this.fingers=[];if(data.interactionBox){this.interactionBox=new InteractionBox(data.interactionBox)}this.gestures=[];this.pointablesMap={};this._translation=data.t;this._rotation=_.flatten(data.r);this._scaleFactor=data.s;this.data=data;this.type="frame";this.currentFrameRate=data.currentFrameRate;var handMap={};for(var handIdx=0,handCount=data.hands.length;handIdx!=handCount;handIdx++){var hand=new Hand(data.hands[handIdx]);hand.frame=this;this.hands.push(hand);this.handsMap[hand.id]=hand;handMap[hand.id]=handIdx}for(var pointableIdx=0,pointableCount=data.pointables.length;pointableIdx!=pointableCount;pointableIdx++){var pointable=new Pointable(data.pointables[pointableIdx]);pointable.frame=this;this.pointables.push(pointable);this.pointablesMap[pointable.id]=pointable;(pointable.tool?this.tools:this.fingers).push(pointable);if(pointable.handId!==undefined&&handMap.hasOwnProperty(pointable.handId)){var hand=this.hands[handMap[pointable.handId]];hand.pointables.push(pointable);(pointable.tool?hand.tools:hand.fingers).push(pointable)}}if(data.gestures){for(var gestureIdx=0,gestureCount=data.gestures.length;gestureIdx!=gestureCount;gestureIdx++){this.gestures.push(createGesture(data.gestures[gestureIdx]))}}};Frame.prototype.tool=function(id){var pointable=this.pointable(id);return pointable.tool?pointable:Pointable.Invalid};Frame.prototype.pointable=function(id){return this.pointablesMap[id]||Pointable.Invalid};Frame.prototype.finger=function(id){var pointable=this.pointable(id);return!pointable.tool?pointable:Pointable.Invalid};Frame.prototype.hand=function(id){return this.handsMap[id]||Hand.Invalid};Frame.prototype.rotationAngle=function(sinceFrame,axis){if(!this.valid||!sinceFrame.valid)return 0;var rot=this.rotationMatrix(sinceFrame);var cs=(rot[0]+rot[4]+rot[8]-1)*.5;var angle=Math.acos(cs);angle=isNaN(angle)?0:angle;if(axis!==undefined){var rotAxis=this.rotationAxis(sinceFrame);angle*=vec3.dot(rotAxis,vec3.normalize(vec3.create(),axis))}return angle};Frame.prototype.rotationAxis=function(sinceFrame){if(!this.valid||!sinceFrame.valid)return vec3.create();return vec3.normalize(vec3.create(),[this._rotation[7]-sinceFrame._rotation[5],this._rotation[2]-sinceFrame._rotation[6],this._rotation[3]-sinceFrame._rotation[1]])};Frame.prototype.rotationMatrix=function(sinceFrame){if(!this.valid||!sinceFrame.valid)return mat3.create();var transpose=mat3.transpose(mat3.create(),this._rotation);return mat3.multiply(mat3.create(),sinceFrame._rotation,transpose)};Frame.prototype.scaleFactor=function(sinceFrame){if(!this.valid||!sinceFrame.valid)return 1;return Math.exp(this._scaleFactor-sinceFrame._scaleFactor)};Frame.prototype.translation=function(sinceFrame){if(!this.valid||!sinceFrame.valid)return vec3.create();return vec3.subtract(vec3.create(),this._translation,sinceFrame._translation)};Frame.prototype.toString=function(){var str="Frame [ id:"+this.id+" | timestamp:"+this.timestamp+" | Hand count:("+this.hands.length+") | Pointable count:("+this.pointables.length+")";if(this.gestures)str+=" | Gesture count:("+this.gestures.length+")";str+=" ]";return str};Frame.prototype.dump=function(){var out="";out+="Frame Info:
";out+=this.toString();out+="

Hands:
";for(var handIdx=0,handCount=this.hands.length;handIdx!=handCount;handIdx++){out+="  "+this.hands[handIdx].toString()+"
"}out+="

Pointables:
";for(var pointableIdx=0,pointableCount=this.pointables.length;pointableIdx!=pointableCount;pointableIdx++){out+="  "+this.pointables[pointableIdx].toString()+"
"}if(this.gestures){out+="

Gestures:
";for(var gestureIdx=0,gestureCount=this.gestures.length;gestureIdx!=gestureCount;gestureIdx++){out+="  "+this.gestures[gestureIdx].toString()+"
"}}out+="

Raw JSON:
";out+=JSON.stringify(this.data);return out};Frame.Invalid={valid:false,hands:[],fingers:[],tools:[],gestures:[],pointables:[],pointable:function(){return Pointable.Invalid},finger:function(){return Pointable.Invalid},hand:function(){return Hand.Invalid},toString:function(){return"invalid frame"},dump:function(){return this.toString()},rotationAngle:function(){return 0},rotationMatrix:function(){return mat3.create()},rotationAxis:function(){return vec3.create()},scaleFactor:function(){return 1},translation:function(){return vec3.create()}}},{"./gesture":6,"./hand":7,"./interaction_box":9,"./pointable":11,"gl-matrix":19,underscore:20}],6:[function(require,module,exports){var glMatrix=require("gl-matrix"),vec3=glMatrix.vec3,EventEmitter=require("events").EventEmitter,_=require("underscore");var createGesture=exports.createGesture=function(data){var gesture;switch(data.type){case"circle":gesture=new CircleGesture(data);break;case"swipe":gesture=new SwipeGesture(data);break;case"screenTap":gesture=new ScreenTapGesture(data);break;case"keyTap":gesture=new KeyTapGesture(data);break;default:throw"unkown gesture type"}gesture.id=data.id;gesture.handIds=data.handIds;gesture.pointableIds=data.pointableIds;gesture.duration=data.duration;gesture.state=data.state;gesture.type=data.type;return gesture};var gestureListener=exports.gestureListener=function(controller,type){var handlers={};var gestureMap={};var gestureCreator=function(){var candidateGesture=gestureMap[gesture.id];if(candidateGesture!==undefined)gesture.update(gesture,frame);if(gesture.state=="start"||gesture.state=="stop"){if(type==gesture.type&&gestureMap[gesture.id]===undefined){gestureMap[gesture.id]=new Gesture(gesture,frame);gesture.update(gesture,frame)}if(gesture.state=="stop"){delete gestureMap[gesture.id]}}};controller.on("gesture",function(gesture,frame){if(gesture.type==type){if(gesture.state=="start"||gesture.state=="stop"){if(gestureMap[gesture.id]===undefined){var gestureTracker=new Gesture(gesture,frame);gestureMap[gesture.id]=gestureTracker;_.each(handlers,function(cb,name){gestureTracker.on(name,cb)})}}gestureMap[gesture.id].update(gesture,frame);if(gesture.state=="stop"){delete gestureMap[gesture.id]}}});var builder={start:function(cb){handlers["start"]=cb;return builder},stop:function(cb){handlers["stop"]=cb;return builder},complete:function(cb){handlers["stop"]=cb;return builder},update:function(cb){handlers["update"]=cb;return builder}};return builder};var Gesture=exports.Gesture=function(gesture,frame){this.gestures=[gesture];this.frames=[frame]};Gesture.prototype.update=function(gesture,frame){this.gestures.push(gesture);this.frames.push(frame);this.emit(gesture.state,this)};_.extend(Gesture.prototype,EventEmitter.prototype);var CircleGesture=function(data){this.center=data.center;this.normal=data.normal;this.progress=data.progress;this.radius=data.radius};CircleGesture.prototype.toString=function(){return"CircleGesture ["+JSON.stringify(this)+"]"};var SwipeGesture=function(data){this.startPosition=data.startPosition;this.position=data.position;this.direction=data.direction;this.speed=data.speed};SwipeGesture.prototype.toString=function(){return"SwipeGesture ["+JSON.stringify(this)+"]"};var ScreenTapGesture=function(data){this.position=data.position;this.direction=data.direction;this.progress=data.progress};ScreenTapGesture.prototype.toString=function(){return"ScreenTapGesture ["+JSON.stringify(this)+"]"};var KeyTapGesture=function(data){this.position=data.position;this.direction=data.direction;this.progress=data.progress};KeyTapGesture.prototype.toString=function(){return"KeyTapGesture ["+JSON.stringify(this)+"]"}},{events:17,"gl-matrix":19,underscore:20}],7:[function(require,module,exports){var Pointable=require("./pointable"),glMatrix=require("gl-matrix"),mat3=glMatrix.mat3,vec3=glMatrix.vec3,_=require("underscore");var Hand=module.exports=function(data){this.id=data.id;this.palmPosition=data.palmPosition;this.direction=data.direction;this.palmVelocity=data.palmVelocity;this.palmNormal=data.palmNormal;this.sphereCenter=data.sphereCenter;this.sphereRadius=data.sphereRadius;this.valid=true;this.pointables=[];this.fingers=[];this.tools=[];this._translation=data.t;this._rotation=_.flatten(data.r);this._scaleFactor=data.s;this.timeVisible=data.timeVisible;this.stabilizedPalmPosition=data.stabilizedPalmPosition};Hand.prototype.finger=function(id){var finger=this.frame.finger(id);return finger&&finger.handId==this.id?finger:Pointable.Invalid};Hand.prototype.rotationAngle=function(sinceFrame,axis){if(!this.valid||!sinceFrame.valid)return 0;var sinceHand=sinceFrame.hand(this.id);if(!sinceHand.valid)return 0;var rot=this.rotationMatrix(sinceFrame);var cs=(rot[0]+rot[4]+rot[8]-1)*.5;var angle=Math.acos(cs);angle=isNaN(angle)?0:angle;if(axis!==undefined){var rotAxis=this.rotationAxis(sinceFrame);angle*=vec3.dot(rotAxis,vec3.normalize(vec3.create(),axis))}return angle};Hand.prototype.rotationAxis=function(sinceFrame){if(!this.valid||!sinceFrame.valid)return vec3.create();var sinceHand=sinceFrame.hand(this.id);if(!sinceHand.valid)return vec3.create();return vec3.normalize(vec3.create(),[this._rotation[7]-sinceHand._rotation[5],this._rotation[2]-sinceHand._rotation[6],this._rotation[3]-sinceHand._rotation[1]])};Hand.prototype.rotationMatrix=function(sinceFrame){if(!this.valid||!sinceFrame.valid)return mat3.create();var sinceHand=sinceFrame.hand(this.id);if(!sinceHand.valid)return mat3.create();var transpose=mat3.transpose(mat3.create(),this._rotation);var m=mat3.multiply(mat3.create(),sinceHand._rotation,transpose);return m};Hand.prototype.scaleFactor=function(sinceFrame){if(!this.valid||!sinceFrame.valid)return 1;var sinceHand=sinceFrame.hand(this.id);if(!sinceHand.valid)return 1;return Math.exp(this._scaleFactor-sinceHand._scaleFactor)};Hand.prototype.translation=function(sinceFrame){if(!this.valid||!sinceFrame.valid)return vec3.create();var sinceHand=sinceFrame.hand(this.id);if(!sinceHand.valid)return vec3.create();return[this._translation[0]-sinceHand._translation[0],this._translation[1]-sinceHand._translation[1],this._translation[2]-sinceHand._translation[2]]};Hand.prototype.toString=function(){return"Hand [ id: "+this.id+" | palm velocity:"+this.palmVelocity+" | sphere center:"+this.sphereCenter+" ] "};Hand.Invalid={valid:false,fingers:[],tools:[],pointables:[],pointable:function(){return Pointable.Invalid},finger:function(){return Pointable.Invalid},toString:function(){return"invalid frame"},dump:function(){return this.toString()},rotationAngle:function(){return 0},rotationMatrix:function(){return mat3.create()},rotationAxis:function(){return vec3.create()},scaleFactor:function(){return 1},translation:function(){return vec3.create()}}},{"./pointable":11,"gl-matrix":19,underscore:20}],8:[function(require,module,exports){!function(){module.exports={Controller:require("./controller"),Frame:require("./frame"),Gesture:require("./gesture"),Hand:require("./hand"),Pointable:require("./pointable"),InteractionBox:require("./interaction_box"),Connection:require("./connection"),CircularBuffer:require("./circular_buffer"),UI:require("./ui"),glMatrix:require("gl-matrix"),mat3:require("gl-matrix").mat3,vec3:require("gl-matrix").vec3,loopController:undefined,loop:function(opts,callback){if(callback===undefined){callback=opts;opts={}}if(!this.loopController)this.loopController=new this.Controller(opts);this.loopController.loop(callback)}}}()},{"./circular_buffer":2,"./connection":3,"./controller":4,"./frame":5,"./gesture":6,"./hand":7,"./interaction_box":9,"./pointable":11,"./ui":13,"gl-matrix":19}],9:[function(require,module,exports){var glMatrix=require("gl-matrix"),vec3=glMatrix.vec3;var InteractionBox=module.exports=function(data){this.valid=true;this.center=data.center;this.size=data.size;this.width=data.size[0];this.height=data.size[1];this.depth=data.size[2]};InteractionBox.prototype.denormalizePoint=function(normalizedPosition){return vec3.fromValues((normalizedPosition[0]-.5)*this.size[0]+this.center[0],(normalizedPosition[1]-.5)*this.size[1]+this.center[1],(normalizedPosition[2]-.5)*this.size[2]+this.center[2])};InteractionBox.prototype.normalizePoint=function(position,clamp){var vec=vec3.fromValues((position[0]-this.center[0])/this.size[0]+.5,(position[1]-this.center[1])/this.size[1]+.5,(position[2]-this.center[2])/this.size[2]+.5);if(clamp){vec[0]=Math.min(Math.max(vec[0],0),1);vec[1]=Math.min(Math.max(vec[1],0),1);vec[2]=Math.min(Math.max(vec[2],0),1)}return vec};InteractionBox.prototype.toString=function(){return"InteractionBox [ width:"+this.width+" | height:"+this.height+" | depth:"+this.depth+" ]"};InteractionBox.Invalid={valid:false}},{"gl-matrix":19}],10:[function(require,module,exports){var Pipeline=module.exports=function(){this.steps=[]};Pipeline.prototype.addStep=function(step){this.steps.push(step)};Pipeline.prototype.run=function(frame){var stepsLength=this.steps.length;for(var i=0;i!=stepsLength;i++){if(!frame)break;frame=this.steps[i](frame)}return frame}},{}],11:[function(require,module,exports){var glMatrix=require("gl-matrix"),vec3=glMatrix.vec3;var Pointable=module.exports=function(data){this.valid=true;this.id=data.id;this.handId=data.handId;this.length=data.length;this.tool=data.tool;this.width=data.width;this.direction=data.direction;this.stabilizedTipPosition=data.stabilizedTipPosition;this.tipPosition=data.tipPosition;this.tipVelocity=data.tipVelocity;this.touchZone=data.touchZone;this.touchDistance=data.touchDistance;this.timeVisible=data.timeVisible};Pointable.prototype.toString=function(){if(this.tool==true){return"Pointable [ id:"+this.id+" "+this.length+"mmx | with:"+this.width+"mm | direction:"+this.direction+" ]"}else{return"Pointable [ id:"+this.id+" "+this.length+"mmx | direction: "+this.direction+" ]"}};Pointable.Invalid={valid:false}},{"gl-matrix":19}],12:[function(require,module,exports){var Frame=require("./frame");var Event=function(data){this.type=data.type;this.state=data.state};var chooseProtocol=exports.chooseProtocol=function(header){var protocol;switch(header.version){case 1:protocol=JSONProtocol(1,function(data){return new Frame(data)});break;case 2:protocol=JSONProtocol(2,function(data){return new Frame(data)});protocol.sendHeartbeat=function(connection){connection.send(protocol.encode({heartbeat:true}))};break;case 3:protocol=JSONProtocol(3,function(data){return data.event?new Event(data.event):new Frame(data)});protocol.sendHeartbeat=function(connection){connection.send(protocol.encode({heartbeat:true}))};break;default:throw"unrecognized version"}return protocol};var JSONProtocol=function(version,cb){var protocol=cb;protocol.encode=function(message){return JSON.stringify(message)};protocol.version=version;protocol.versionLong="Version "+version;protocol.type="protocol";return protocol}},{"./frame":5}],13:[function(require,module,exports){exports.UI={Region:require("./ui/region"),Cursor:require("./ui/cursor")}},{"./ui/cursor":14,"./ui/region":15}],14:[function(require,module,exports){var Cursor=module.exports=function(){return function(frame){var pointable=frame.pointables.sort(function(a,b){return a.z-b.z})[0];if(pointable&&pointable.valid){frame.cursorPosition=pointable.tipPosition}return frame}}},{}],15:[function(require,module,exports){var EventEmitter=require("events").EventEmitter,_=require("underscore");var Region=module.exports=function(start,end){this.start=new Vector(start);this.end=new Vector(end);this.enteredFrame=null};Region.prototype.hasPointables=function(frame){for(var i=0;i!=frame.pointables.length;i++){var position=frame.pointables[i].tipPosition;if(position.x>=this.start.x&&position.x<=this.end.x&&position.y>=this.start.y&&position.y<=this.end.y&&position.z>=this.start.z&&position.z<=this.end.z){return true}}return false};Region.prototype.listener=function(opts){var region=this;if(opts&&opts.nearThreshold)this.setupNearRegion(opts.nearThreshold);return function(frame){return region.updatePosition(frame)}};Region.prototype.clipper=function(){var region=this;return function(frame){region.updatePosition(frame);return region.enteredFrame?frame:null}};Region.prototype.setupNearRegion=function(distance){var nearRegion=this.nearRegion=new Region([this.start.x-distance,this.start.y-distance,this.start.z-distance],[this.end.x+distance,this.end.y+distance,this.end.z+distance]);var region=this;nearRegion.on("enter",function(frame){region.emit("near",frame)});nearRegion.on("exit",function(frame){region.emit("far",frame)});region.on("exit",function(frame){region.emit("near",frame)})};Region.prototype.updatePosition=function(frame){if(this.nearRegion)this.nearRegion.updatePosition(frame);if(this.hasPointables(frame)&&this.enteredFrame==null){this.enteredFrame=frame;this.emit("enter",this.enteredFrame)}else if(!this.hasPointables(frame)&&this.enteredFrame!=null){this.enteredFrame=null;this.emit("exit",this.enteredFrame)}return frame};Region.prototype.normalize=function(position){return new Vector([(position.x-this.start.x)/(this.end.x-this.start.x),(position.y-this.start.y)/(this.end.y-this.start.y),(position.z-this.start.z)/(this.end.z-this.start.z)])};Region.prototype.mapToXY=function(position,width,height){var normalized=this.normalize(position);var x=normalized.x,y=normalized.y;if(x>1)x=1;else if(x<-1)x=-1;if(y>1)y=1;else if(y<-1)y=-1;return[(x+1)/2*width,(1-y)/2*height,normalized.z]};_.extend(Region.prototype,EventEmitter.prototype)},{events:17,underscore:20}],16:[function(require,module,exports){},{}],17:[function(require,module,exports){!function(process){if(!process.EventEmitter)process.EventEmitter=function(){};var EventEmitter=exports.EventEmitter=process.EventEmitter;var isArray=typeof Array.isArray==="function"?Array.isArray:function(xs){return Object.prototype.toString.call(xs)==="[object Array]"};function indexOf(xs,x){if(xs.indexOf)return xs.indexOf(x);for(var i=0;i0&&this._events[type].length>m){this._events[type].warned=true;console.error("(node) warning: possible EventEmitter memory "+"leak detected. %d listeners added. "+"Use emitter.setMaxListeners() to increase limit.",this._events[type].length);console.trace()}}this._events[type].push(listener)}else{this._events[type]=[this._events[type],listener]}return this};EventEmitter.prototype.on=EventEmitter.prototype.addListener;EventEmitter.prototype.once=function(type,listener){var self=this;self.on(type,function g(){self.removeListener(type,g);listener.apply(this,arguments)});return this};EventEmitter.prototype.removeListener=function(type,listener){if("function"!==typeof listener){throw new Error("removeListener only takes instances of Function")}if(!this._events||!this._events[type])return this;var list=this._events[type];if(isArray(list)){var i=indexOf(list,listener);if(i<0)return this;list.splice(i,1);if(list.length==0)delete this._events[type]}else if(this._events[type]===listener){delete this._events[type]}return this};EventEmitter.prototype.removeAllListeners=function(type){if(arguments.length===0){this._events={};return this}if(type&&this._events&&this._events[type])this._events[type]=null;return this};EventEmitter.prototype.listeners=function(type){if(!this._events)this._events={};if(!this._events[type])this._events[type]=[];if(!isArray(this._events[type])){this._events[type]=[this._events[type]]}return this._events[type]}}(require("__browserify_process"))},{__browserify_process:18}],18:[function(require,module,exports){var process=module.exports={};process.nextTick=function(){var canSetImmediate=typeof window!=="undefined"&&window.setImmediate;var canPost=typeof window!=="undefined"&&window.postMessage&&window.addEventListener;if(canSetImmediate){return function(f){return window.setImmediate(f)}}if(canPost){var queue=[];window.addEventListener("message",function(ev){if(ev.source===window&&ev.data==="process-tick"){ev.stopPropagation();if(queue.length>0){var fn=queue.shift();fn()}}},true);return function nextTick(fn){queue.push(fn);window.postMessage("process-tick","*")}}return function nextTick(fn){setTimeout(fn,0)}}();process.title="browser";process.browser=true;process.env={};process.argv=[];process.binding=function(name){throw new Error("process.binding is not supported")};process.cwd=function(){return"/"};process.chdir=function(dir){throw new Error("process.chdir is not supported")}},{}],19:[function(require,module,exports){!function(){!function(){"use strict";var shim={};if(typeof exports==="undefined"){if(typeof define=="function"&&typeof define.amd=="object"&&define.amd){shim.exports={};define(function(){return shim.exports})}else{shim.exports=window}}else{shim.exports=exports}!function(exports){var vec2={};if(!GLMAT_EPSILON){var GLMAT_EPSILON=1e-6}vec2.create=function(){return new Float32Array(2)};vec2.clone=function(a){var out=new Float32Array(2);out[0]=a[0];out[1]=a[1];return out};vec2.fromValues=function(x,y){var out=new Float32Array(2);out[0]=x;out[1]=y;return out};vec2.copy=function(out,a){out[0]=a[0];out[1]=a[1];return out};vec2.set=function(out,x,y){out[0]=x;out[1]=y;return out};vec2.add=function(out,a,b){out[0]=a[0]+b[0];out[1]=a[1]+b[1];return out};vec2.sub=vec2.subtract=function(out,a,b){out[0]=a[0]-b[0];out[1]=a[1]-b[1];return out};vec2.mul=vec2.multiply=function(out,a,b){out[0]=a[0]*b[0];out[1]=a[1]*b[1];return out};vec2.div=vec2.divide=function(out,a,b){out[0]=a[0]/b[0];out[1]=a[1]/b[1];return out};vec2.min=function(out,a,b){out[0]=Math.min(a[0],b[0]);
+out[1]=Math.min(a[1],b[1]);return out};vec2.max=function(out,a,b){out[0]=Math.max(a[0],b[0]);out[1]=Math.max(a[1],b[1]);return out};vec2.scale=function(out,a,b){out[0]=a[0]*b;out[1]=a[1]*b;return out};vec2.dist=vec2.distance=function(a,b){var x=b[0]-a[0],y=b[1]-a[1];return Math.sqrt(x*x+y*y)};vec2.sqrDist=vec2.squaredDistance=function(a,b){var x=b[0]-a[0],y=b[1]-a[1];return x*x+y*y};vec2.len=vec2.length=function(a){var x=a[0],y=a[1];return Math.sqrt(x*x+y*y)};vec2.sqrLen=vec2.squaredLength=function(a){var x=a[0],y=a[1];return x*x+y*y};vec2.negate=function(out,a){out[0]=-a[0];out[1]=-a[1];return out};vec2.normalize=function(out,a){var x=a[0],y=a[1];var len=x*x+y*y;if(len>0){len=1/Math.sqrt(len);out[0]=a[0]*len;out[1]=a[1]*len}return out};vec2.dot=function(a,b){return a[0]*b[0]+a[1]*b[1]};vec2.cross=function(out,a,b){var z=a[0]*b[1]-a[1]*b[0];out[0]=out[1]=0;out[2]=z;return out};vec2.lerp=function(out,a,b,t){var ax=a[0],ay=a[1];out[0]=ax+t*(b[0]-ax);out[1]=ay+t*(b[1]-ay);return out};vec2.transformMat2=function(out,a,m){var x=a[0],y=a[1];out[0]=x*m[0]+y*m[1];out[1]=x*m[2]+y*m[3];return out};vec2.forEach=function(){var vec=new Float32Array(2);return function(a,stride,offset,count,fn,arg){var i,l;if(!stride){stride=2}if(!offset){offset=0}if(count){l=Math.min(count*stride+offset,a.length)}else{l=a.length}for(i=offset;i0){len=1/Math.sqrt(len);out[0]=a[0]*len;out[1]=a[1]*len;out[2]=a[2]*len}return out};vec3.dot=function(a,b){return a[0]*b[0]+a[1]*b[1]+a[2]*b[2]};vec3.cross=function(out,a,b){var ax=a[0],ay=a[1],az=a[2],bx=b[0],by=b[1],bz=b[2];out[0]=ay*bz-az*by;out[1]=az*bx-ax*bz;out[2]=ax*by-ay*bx;return out};vec3.lerp=function(out,a,b,t){var ax=a[0],ay=a[1],az=a[2];out[0]=ax+t*(b[0]-ax);out[1]=ay+t*(b[1]-ay);out[2]=az+t*(b[2]-az);return out};vec3.transformMat4=function(out,a,m){var x=a[0],y=a[1],z=a[2];out[0]=m[0]*x+m[4]*y+m[8]*z+m[12];out[1]=m[1]*x+m[5]*y+m[9]*z+m[13];out[2]=m[2]*x+m[6]*y+m[10]*z+m[14];return out};vec3.transformQuat=function(out,a,q){var x=a[0],y=a[1],z=a[2],qx=q[0],qy=q[1],qz=q[2],qw=q[3],ix=qw*x+qy*z-qz*y,iy=qw*y+qz*x-qx*z,iz=qw*z+qx*y-qy*x,iw=-qx*x-qy*y-qz*z;out[0]=ix*qw+iw*-qx+iy*-qz-iz*-qy;out[1]=iy*qw+iw*-qy+iz*-qx-ix*-qz;out[2]=iz*qw+iw*-qz+ix*-qy-iy*-qx;return out};vec3.forEach=function(){var vec=new Float32Array(3);return function(a,stride,offset,count,fn,arg){var i,l;if(!stride){stride=3}if(!offset){offset=0}if(count){l=Math.min(count*stride+offset,a.length)}else{l=a.length}for(i=offset;i0){len=1/Math.sqrt(len);out[0]=a[0]*len;out[1]=a[1]*len;out[2]=a[2]*len;out[3]=a[3]*len}return out};vec4.dot=function(a,b){return a[0]*b[0]+a[1]*b[1]+a[2]*b[2]+a[3]*b[3]};vec4.lerp=function(out,a,b,t){var ax=a[0],ay=a[1],az=a[2],aw=a[3];out[0]=ax+t*(b[0]-ax);out[1]=ay+t*(b[1]-ay);out[2]=az+t*(b[2]-az);out[3]=aw+t*(b[3]-aw);return out};vec4.transformMat4=function(out,a,m){var x=a[0],y=a[1],z=a[2],w=a[3];out[0]=m[0]*x+m[4]*y+m[8]*z+m[12]*w;out[1]=m[1]*x+m[5]*y+m[9]*z+m[13]*w;out[2]=m[2]*x+m[6]*y+m[10]*z+m[14]*w;out[3]=m[3]*x+m[7]*y+m[11]*z+m[15]*w;return out};vec4.transformQuat=function(out,a,q){var x=a[0],y=a[1],z=a[2],qx=q[0],qy=q[1],qz=q[2],qw=q[3],ix=qw*x+qy*z-qz*y,iy=qw*y+qz*x-qx*z,iz=qw*z+qx*y-qy*x,iw=-qx*x-qy*y-qz*z;out[0]=ix*qw+iw*-qx+iy*-qz-iz*-qy;out[1]=iy*qw+iw*-qy+iz*-qx-ix*-qz;out[2]=iz*qw+iw*-qz+ix*-qy-iy*-qx;return out};vec4.forEach=function(){var vec=new Float32Array(4);return function(a,stride,offset,count,fn,arg){var i,l;if(!stride){stride=4}if(!offset){offset=0}if(count){l=Math.min(count*stride+offset,a.length)}else{l=a.length}for(i=offset;i=1){if(out!==a){out[0]=ax;out[1]=ay;out[2]=az;out[3]=aw}return out}halfTheta=Math.acos(cosHalfTheta);sinHalfTheta=Math.sqrt(1-cosHalfTheta*cosHalfTheta);if(Math.abs(sinHalfTheta)<.001){out[0]=ax*.5+bx*.5;out[1]=ay*.5+by*.5;out[2]=az*.5+bz*.5;out[3]=aw*.5+bw*.5;return out}ratioA=Math.sin((1-t)*halfTheta)/sinHalfTheta;ratioB=Math.sin(t*halfTheta)/sinHalfTheta;out[0]=ax*ratioA+bx*ratioB;out[1]=ay*ratioA+by*ratioB;out[2]=az*ratioA+bz*ratioB;out[3]=aw*ratioA+bw*ratioB;return out};quat.invert=function(out,a){var a0=a[0],a1=a[1],a2=a[2],a3=a[3],dot=a0*a0+a1*a1+a2*a2+a3*a3,invDot=dot?1/dot:0;out[0]=-a0*invDot;out[1]=-a1*invDot;out[2]=-a2*invDot;out[3]=a3*invDot;return out};quat.conjugate=function(out,a){out[0]=-a[0];out[1]=-a[1];out[2]=-a[2];out[3]=a[3];return out};quat.len=quat.length=vec4.length;quat.sqrLen=quat.squaredLength=vec4.squaredLength;quat.normalize=vec4.normalize;quat.str=function(a){return"quat("+a[0]+", "+a[1]+", "+a[2]+", "+a[3]+")"};if(typeof exports!=="undefined"){exports.quat=quat}}(shim.exports)}()}()},{}],20:[function(require,module,exports){!function(){!function(){var root=this;var previousUnderscore=root._;var breaker={};var ArrayProto=Array.prototype,ObjProto=Object.prototype,FuncProto=Function.prototype;var push=ArrayProto.push,slice=ArrayProto.slice,concat=ArrayProto.concat,toString=ObjProto.toString,hasOwnProperty=ObjProto.hasOwnProperty;var nativeForEach=ArrayProto.forEach,nativeMap=ArrayProto.map,nativeReduce=ArrayProto.reduce,nativeReduceRight=ArrayProto.reduceRight,nativeFilter=ArrayProto.filter,nativeEvery=ArrayProto.every,nativeSome=ArrayProto.some,nativeIndexOf=ArrayProto.indexOf,nativeLastIndexOf=ArrayProto.lastIndexOf,nativeIsArray=Array.isArray,nativeKeys=Object.keys,nativeBind=FuncProto.bind;var _=function(obj){if(obj instanceof _)return obj;if(!(this instanceof _))return new _(obj);this._wrapped=obj};if(typeof exports!=="undefined"){if(typeof module!=="undefined"&&module.exports){exports=module.exports=_}exports._=_}else{root._=_}_.VERSION="1.4.4";var each=_.each=_.forEach=function(obj,iterator,context){if(obj==null)return;if(nativeForEach&&obj.forEach===nativeForEach){obj.forEach(iterator,context)}else if(obj.length===+obj.length){for(var i=0,l=obj.length;i2;if(obj==null)obj=[];if(nativeReduce&&obj.reduce===nativeReduce){if(context)iterator=_.bind(iterator,context);return initial?obj.reduce(iterator,memo):obj.reduce(iterator)}each(obj,function(value,index,list){if(!initial){memo=value;initial=true}else{memo=iterator.call(context,memo,value,index,list)}});if(!initial)throw new TypeError(reduceError);return memo};_.reduceRight=_.foldr=function(obj,iterator,memo,context){var initial=arguments.length>2;if(obj==null)obj=[];if(nativeReduceRight&&obj.reduceRight===nativeReduceRight){if(context)iterator=_.bind(iterator,context);return initial?obj.reduceRight(iterator,memo):obj.reduceRight(iterator)}var length=obj.length;if(length!==+length){var keys=_.keys(obj);length=keys.length}each(obj,function(value,index,list){index=keys?keys[--length]:--length;if(!initial){memo=obj[index];initial=true}else{memo=iterator.call(context,memo,obj[index],index,list)}});if(!initial)throw new TypeError(reduceError);return memo};_.find=_.detect=function(obj,iterator,context){var result;any(obj,function(value,index,list){if(iterator.call(context,value,index,list)){result=value;return true}});return result};_.filter=_.select=function(obj,iterator,context){var results=[];if(obj==null)return results;if(nativeFilter&&obj.filter===nativeFilter)return obj.filter(iterator,context);each(obj,function(value,index,list){if(iterator.call(context,value,index,list))results[results.length]=value});return results};_.reject=function(obj,iterator,context){return _.filter(obj,function(value,index,list){return!iterator.call(context,value,index,list)},context)};_.every=_.all=function(obj,iterator,context){iterator||(iterator=_.identity);var result=true;if(obj==null)return result;if(nativeEvery&&obj.every===nativeEvery)return obj.every(iterator,context);each(obj,function(value,index,list){if(!(result=result&&iterator.call(context,value,index,list)))return breaker});return!!result};var any=_.some=_.any=function(obj,iterator,context){iterator||(iterator=_.identity);var result=false;if(obj==null)return result;if(nativeSome&&obj.some===nativeSome)return obj.some(iterator,context);each(obj,function(value,index,list){if(result||(result=iterator.call(context,value,index,list)))return breaker});return!!result};_.contains=_.include=function(obj,target){if(obj==null)return false;if(nativeIndexOf&&obj.indexOf===nativeIndexOf)return obj.indexOf(target)!=-1;return any(obj,function(value){return value===target})};_.invoke=function(obj,method){var args=slice.call(arguments,2);var isFunc=_.isFunction(method);return _.map(obj,function(value){return(isFunc?method:value[method]).apply(value,args)})};_.pluck=function(obj,key){return _.map(obj,function(value){return value[key]})};_.where=function(obj,attrs,first){if(_.isEmpty(attrs))return first?null:[];return _[first?"find":"filter"](obj,function(value){for(var key in attrs){if(attrs[key]!==value[key])return false}return true})};_.findWhere=function(obj,attrs){return _.where(obj,attrs,true)};_.max=function(obj,iterator,context){if(!iterator&&_.isArray(obj)&&obj[0]===+obj[0]&&obj.length<65535){return Math.max.apply(Math,obj)}if(!iterator&&_.isEmpty(obj))return-Infinity;var result={computed:-Infinity,value:-Infinity};each(obj,function(value,index,list){var computed=iterator?iterator.call(context,value,index,list):value;computed>=result.computed&&(result={value:value,computed:computed})});return result.value};_.min=function(obj,iterator,context){if(!iterator&&_.isArray(obj)&&obj[0]===+obj[0]&&obj.length<65535){return Math.min.apply(Math,obj)}if(!iterator&&_.isEmpty(obj))return Infinity;var result={computed:Infinity,value:Infinity};each(obj,function(value,index,list){var computed=iterator?iterator.call(context,value,index,list):value;computedb||a===void 0)return 1;if(a>>1;iterator.call(context,array[mid])=0})})};_.difference=function(array){var rest=concat.apply(ArrayProto,slice.call(arguments,1));return _.filter(array,function(value){return!_.contains(rest,value)})};_.zip=function(){var args=slice.call(arguments);var length=_.max(_.pluck(args,"length"));var results=new Array(length);for(var i=0;i=0;i--){args=[funcs[i].apply(this,args)]}return args[0]}};_.after=function(times,func){if(times<=0)return func();return function(){if(--times<1){return func.apply(this,arguments)}}};_.keys=nativeKeys||function(obj){if(obj!==Object(obj))throw new TypeError("Invalid object");var keys=[];for(var key in obj)if(_.has(obj,key))keys[keys.length]=key;return keys};_.values=function(obj){var values=[];for(var key in obj)if(_.has(obj,key))values.push(obj[key]);return values};_.pairs=function(obj){var pairs=[];for(var key in obj)if(_.has(obj,key))pairs.push([key,obj[key]]);return pairs};_.invert=function(obj){var result={};for(var key in obj)if(_.has(obj,key))result[obj[key]]=key;return result};_.functions=_.methods=function(obj){var names=[];for(var key in obj){if(_.isFunction(obj[key]))names.push(key)}return names.sort()};_.extend=function(obj){each(slice.call(arguments,1),function(source){if(source){for(var prop in source){obj[prop]=source[prop]}}});return obj};_.pick=function(obj){var copy={};var keys=concat.apply(ArrayProto,slice.call(arguments,1));each(keys,function(key){if(key in obj)copy[key]=obj[key]});return copy};_.omit=function(obj){var copy={};var keys=concat.apply(ArrayProto,slice.call(arguments,1));for(var key in obj){if(!_.contains(keys,key))copy[key]=obj[key]}return copy};_.defaults=function(obj){each(slice.call(arguments,1),function(source){if(source){for(var prop in source){if(obj[prop]==null)obj[prop]=source[prop]}}});return obj};_.clone=function(obj){if(!_.isObject(obj))return obj;return _.isArray(obj)?obj.slice():_.extend({},obj)};_.tap=function(obj,interceptor){interceptor(obj);return obj};var eq=function(a,b,aStack,bStack){if(a===b)return a!==0||1/a==1/b;if(a==null||b==null)return a===b;if(a instanceof _)a=a._wrapped;if(b instanceof _)b=b._wrapped;var className=toString.call(a);if(className!=toString.call(b))return false;switch(className){case"[object String]":return a==String(b);case"[object Number]":return a!=+a?b!=+b:a==0?1/a==1/b:a==+b;case"[object Date]":case"[object Boolean]":return+a==+b;case"[object RegExp]":return a.source==b.source&&a.global==b.global&&a.multiline==b.multiline&&a.ignoreCase==b.ignoreCase}if(typeof a!="object"||typeof b!="object")return false;var length=aStack.length;while(length--){if(aStack[length]==a)return bStack[length]==b}aStack.push(a);bStack.push(b);var size=0,result=true;if(className=="[object Array]"){size=a.length;result=size==b.length;if(result){while(size--){if(!(result=eq(a[size],b[size],aStack,bStack)))break}}}else{var aCtor=a.constructor,bCtor=b.constructor;if(aCtor!==bCtor&&!(_.isFunction(aCtor)&&aCtor instanceof aCtor&&_.isFunction(bCtor)&&bCtor instanceof bCtor)){return false}for(var key in a){if(_.has(a,key)){size++;if(!(result=_.has(b,key)&&eq(a[key],b[key],aStack,bStack)))break}}if(result){for(key in b){if(_.has(b,key)&&!size--)break}result=!size}}aStack.pop();bStack.pop();return result};_.isEqual=function(a,b){return eq(a,b,[],[])};_.isEmpty=function(obj){if(obj==null)return true;if(_.isArray(obj)||_.isString(obj))return obj.length===0;for(var key in obj)if(_.has(obj,key))return false;return true};_.isElement=function(obj){return!!(obj&&obj.nodeType===1)};_.isArray=nativeIsArray||function(obj){return toString.call(obj)=="[object Array]"};_.isObject=function(obj){return obj===Object(obj)};each(["Arguments","Function","String","Number","Date","RegExp"],function(name){_["is"+name]=function(obj){return toString.call(obj)=="[object "+name+"]"}});if(!_.isArguments(arguments)){_.isArguments=function(obj){return!!(obj&&_.has(obj,"callee"))}}if(typeof/./!=="function"){_.isFunction=function(obj){return typeof obj==="function"}}_.isFinite=function(obj){return isFinite(obj)&&!isNaN(parseFloat(obj))};_.isNaN=function(obj){return _.isNumber(obj)&&obj!=+obj};_.isBoolean=function(obj){return obj===true||obj===false||toString.call(obj)=="[object Boolean]"};_.isNull=function(obj){return obj===null};_.isUndefined=function(obj){return obj===void 0};_.has=function(obj,key){return hasOwnProperty.call(obj,key)};_.noConflict=function(){root._=previousUnderscore;return this};_.identity=function(value){return value};_.times=function(n,iterator,context){var accum=Array(n);for(var i=0;i":">",'"':""","'":"'","/":"/"}};entityMap.unescape=_.invert(entityMap.escape);var entityRegexes={escape:new RegExp("["+_.keys(entityMap.escape).join("")+"]","g"),unescape:new RegExp("("+_.keys(entityMap.unescape).join("|")+")","g")};_.each(["escape","unescape"],function(method){_[method]=function(string){if(string==null)return"";return(""+string).replace(entityRegexes[method],function(match){return entityMap[method][match]})}});_.result=function(object,property){if(object==null)return null;var value=object[property];return _.isFunction(value)?value.call(object):value};_.mixin=function(obj){each(_.functions(obj),function(name){var func=_[name]=obj[name];_.prototype[name]=function(){var args=[this._wrapped];push.apply(args,arguments);return result.call(this,func.apply(_,args))}})};var idCounter=0;_.uniqueId=function(prefix){var id=++idCounter+"";return prefix?prefix+id:id};_.templateSettings={evaluate:/<%([\s\S]+?)%>/g,interpolate:/<%=([\s\S]+?)%>/g,escape:/<%-([\s\S]+?)%>/g};var noMatch=/(.)^/;var escapes={"'":"'","\\":"\\","\r":"r","\n":"n"," ":"t","\u2028":"u2028","\u2029":"u2029"};var escaper=/\\|'|\r|\n|\t|\u2028|\u2029/g;_.template=function(text,data,settings){var render;settings=_.defaults({},settings,_.templateSettings);var matcher=new RegExp([(settings.escape||noMatch).source,(settings.interpolate||noMatch).source,(settings.evaluate||noMatch).source].join("|")+"|$","g");var index=0;var source="__p+='";text.replace(matcher,function(match,escape,interpolate,evaluate,offset){source+=text.slice(index,offset).replace(escaper,function(match){return"\\"+escapes[match]});if(escape){source+="'+\n((__t=("+escape+"))==null?'':_.escape(__t))+\n'"}if(interpolate){source+="'+\n((__t=("+interpolate+"))==null?'':__t)+\n'"}if(evaluate){source+="';\n"+evaluate+"\n__p+='"}index=offset+match.length;return match});source+="';\n";if(!settings.variable)source="with(obj||{}){\n"+source+"}\n";source="var __t,__p='',__j=Array.prototype.join,"+"print=function(){__p+=__j.call(arguments,'');};\n"+source+"return __p;\n";try{render=new Function(settings.variable||"obj","_",source)}catch(e){e.source=source;throw e}if(data)return render(data,_);var template=function(data){return render.call(this,data,_)};template.source="function("+(settings.variable||"obj")+"){\n"+source+"}";return template};_.chain=function(obj){return _(obj).chain()};var result=function(obj){return this._chain?_(obj).chain():obj};_.mixin(_);each(["pop","push","reverse","shift","sort","splice","unshift"],function(name){var method=ArrayProto[name];_.prototype[name]=function(){var obj=this._wrapped;method.apply(obj,arguments);if((name=="shift"||name=="splice")&&obj.length===0)delete obj[0];return result.call(this,obj)}});each(["concat","join","slice"],function(name){var method=ArrayProto[name];_.prototype[name]=function(){return result.call(this,method.apply(this._wrapped,arguments))}});_.extend(_.prototype,{chain:function(){this._chain=true;return this},value:function(){return this._wrapped}})}.call(this)}()},{}],21:[function(require,module,exports){window.requestAnimFrame=function(){return window.requestAnimationFrame||window.webkitRequestAnimationFrame||window.mozRequestAnimationFrame||window.oRequestAnimationFrame||window.msRequestAnimationFrame||function(callback){window.setTimeout(callback,1e3/60)}}();Leap=require("../lib/index")},{"../lib/index":8}]},{},[21]);
+
+/*
+ * Leap Motion integration for Reveal.js.
+ * James Sun  [sun16]
+ * Rory Hardy [gneatgeek]
+ */
+
+(function () {
+  var body        = document.body,
+      controller  = new Leap.Controller({ enableGestures: true }),
+      lastGesture = 0,
+      leapConfig  = Reveal.getConfig().leap,
+      pointer     = document.createElement( 'div' ),
+      config      = {
+        autoCenter       : true,      // Center pointer around detected position.
+        gestureDelay     : 500,       // How long to delay between gestures.
+        naturalSwipe     : true,      // Swipe as if it were a touch screen.
+        pointerColor     : '#00aaff', // Default color of the pointer.
+        pointerOpacity   : 0.7,       // Default opacity of the pointer.
+        pointerSize      : 15,        // Default minimum height/width of the pointer.
+        pointerTolerance : 120        // Bigger = slower pointer.
+      },
+      entered, enteredPosition, now, size, tipPosition; // Other vars we need later, but don't need to redeclare.
+
+      // Merge user defined settings with defaults
+      if( leapConfig ) {
+        for( key in leapConfig ) {
+          config[key] = leapConfig[key];
+        }
+      }
+
+      pointer.id = 'leap';
+
+      pointer.style.position        = 'absolute';
+      pointer.style.visibility      = 'hidden';
+      pointer.style.zIndex          = 50;
+      pointer.style.opacity         = config.pointerOpacity;
+      pointer.style.backgroundColor = config.pointerColor;
+
+      body.appendChild( pointer );
+
+  // Leap's loop
+  controller.on( 'frame', function ( frame ) {
+    // Timing code to rate limit gesture execution
+    now = new Date().getTime();
+
+    // Pointer: 1 to 2 fingers. Strictly one finger works but may cause innaccuracies.
+    // The innaccuracies were observed on a development model and may not be an issue with consumer models.
+    if( frame.fingers.length > 0 && frame.fingers.length < 3 ) {
+      // Invert direction and multiply by 3 for greater effect.
+      size = -3 * frame.fingers[0].tipPosition[2];
+
+      if( size < config.pointerSize ) {
+        size = config.pointerSize;
+      }
+
+      pointer.style.width        = size     + 'px';
+      pointer.style.height       = size     + 'px';
+      pointer.style.borderRadius = size - 5 + 'px';
+      pointer.style.visibility   = 'visible';
+
+      tipPosition = frame.fingers[0].tipPosition;
+
+      if( config.autoCenter ) {
+        
+
+        // Check whether the finger has entered the z range of the Leap Motion. Used for the autoCenter option.
+        if( !entered ) {
+          entered         = true;
+          enteredPosition = frame.fingers[0].tipPosition;
+        }
+
+        pointer.style.top =
+          (-1 * (( tipPosition[1] - enteredPosition[1] ) * body.offsetHeight / config.pointerTolerance )) +
+            ( body.offsetHeight / 2 ) + 'px';
+
+        pointer.style.left =
+          (( tipPosition[0] - enteredPosition[0] ) * body.offsetWidth / config.pointerTolerance ) +
+            ( body.offsetWidth / 2 ) + 'px';
+      }
+      else {
+        pointer.style.top  = ( 1 - (( tipPosition[1] - 50) / config.pointerTolerance )) *
+          body.offsetHeight + 'px';
+
+        pointer.style.left = ( tipPosition[0] * body.offsetWidth / config.pointerTolerance ) +
+          ( body.offsetWidth / 2 ) + 'px';
+      }
+    }
+    else {
+      // Hide pointer on exit
+      entered                  = false;
+      pointer.style.visibility = 'hidden';
+    }
+
+    // Gestures
+    if( frame.gestures.length > 0 && (now - lastGesture) > config.gestureDelay ) {
+      var gesture = frame.gestures[0];
+
+      // One hand gestures
+      if( frame.hands.length === 1 ) {
+        // Swipe gestures. 3+ fingers.
+        if( frame.fingers.length > 2 && gesture.type === 'swipe' ) {
+          // Define here since some gestures will throw undefined for these.
+          var x = gesture.direction[0],
+              y = gesture.direction[1];
+
+          // Left/right swipe gestures
+          if( Math.abs( x ) > Math.abs( y )) {
+            if( x > 0 ) {
+              config.naturalSwipe ? Reveal.left() : Reveal.right();
+            }
+            else {
+              config.naturalSwipe ? Reveal.right() : Reveal.left();
+            }
+          }
+          // Up/down swipe gestures
+          else {
+            if( y > 0 ) {
+              config.naturalSwipe ? Reveal.down() : Reveal.up();
+            }
+            else {
+              config.naturalSwipe ? Reveal.up() : Reveal.down();
+            }
+          }
+
+          lastGesture = now;
+        }
+      }
+      // Two hand gestures
+      else if( frame.hands.length === 2 ) {
+        // Upward two hand swipe gesture
+        if( gesture.type === 'swipe' && gesture.direction[1] > 0 ) {
+          Reveal.toggleOverview();
+        }
+
+        lastGesture = now;
+      }
+    }
+  });
+
+  controller.connect();
+})();
diff --git a/doc/pub/Intro2Course/html/reveal.js/plugin/remotes/remotes.js b/doc/pub/Intro2Course/html/reveal.js/plugin/remotes/remotes.js
new file mode 100644
index 000000000..ba0dbad7b
--- /dev/null
+++ b/doc/pub/Intro2Course/html/reveal.js/plugin/remotes/remotes.js
@@ -0,0 +1,39 @@
+/**
+ * Touch-based remote controller for your presentation courtesy 
+ * of the folks at http://remotes.io
+ */
+
+(function(window){
+
+    /**
+     * Detects if we are dealing with a touch enabled device (with some false positives)
+     * Borrowed from modernizr: https://github.com/Modernizr/Modernizr/blob/master/feature-detects/touch.js   
+     */
+    var hasTouch  = (function(){
+        return ('ontouchstart' in window) || window.DocumentTouch && document instanceof DocumentTouch;
+    })();
+
+    /**
+     * Detects if notes are enable and the current page is opened inside an /iframe
+     * this prevents loading Remotes.io several times
+     */
+    var isNotesAndIframe = (function(){
+        return window.RevealNotes && !(self == top);
+    })();
+
+    if(!hasTouch && !isNotesAndIframe){
+        head.ready( 'remotes.ne.min.js', function() {
+            new Remotes("preview")
+                .on("swipe-left", function(e){ Reveal.right(); })
+                .on("swipe-right", function(e){ Reveal.left(); })
+                .on("swipe-up", function(e){ Reveal.down(); })
+                .on("swipe-down", function(e){ Reveal.up(); })
+                .on("tap", function(e){ Reveal.next(); })
+                .on("zoom-out", function(e){ Reveal.toggleOverview(true); })
+                .on("zoom-in", function(e){ Reveal.toggleOverview(false); })
+            ;
+        } );
+
+        head.js('https://hakim-static.s3.amazonaws.com/reveal-js/remotes.ne.min.js');
+    }
+})(window);
\ No newline at end of file
diff --git a/doc/pub/Linalg/html/reveal.js/css/theme/template/mixins.scss b/doc/pub/Linalg/html/reveal.js/css/theme/template/mixins.scss
new file mode 100644
index 000000000..e0c560692
--- /dev/null
+++ b/doc/pub/Linalg/html/reveal.js/css/theme/template/mixins.scss
@@ -0,0 +1,29 @@
+@mixin vertical-gradient( $top, $bottom ) {
+	background: $top;
+	background: -moz-linear-gradient( top, $top 0%, $bottom 100% );
+	background: -webkit-gradient( linear, left top, left bottom, color-stop(0%,$top), color-stop(100%,$bottom) );
+	background: -webkit-linear-gradient( top, $top 0%, $bottom 100% );
+	background: -o-linear-gradient( top, $top 0%, $bottom 100% );
+	background: -ms-linear-gradient( top, $top 0%, $bottom 100% );
+	background: linear-gradient( top, $top 0%, $bottom 100% );
+}
+
+@mixin horizontal-gradient( $top, $bottom ) {
+	background: $top;
+	background: -moz-linear-gradient( left, $top 0%, $bottom 100% );
+	background: -webkit-gradient( linear, left top, right top, color-stop(0%,$top), color-stop(100%,$bottom) );
+	background: -webkit-linear-gradient( left, $top 0%, $bottom 100% );
+	background: -o-linear-gradient( left, $top 0%, $bottom 100% );
+	background: -ms-linear-gradient( left, $top 0%, $bottom 100% );
+	background: linear-gradient( left, $top 0%, $bottom 100% );
+}
+
+@mixin radial-gradient( $outer, $inner, $type: circle ) {
+	background: $outer;
+	background: -moz-radial-gradient( center, $type cover,  $inner 0%, $outer 100% );
+	background: -webkit-gradient( radial, center center, 0px, center center, 100%, color-stop(0%,$inner), color-stop(100%,$outer) );
+	background: -webkit-radial-gradient( center, $type cover,  $inner 0%, $outer 100% );
+	background: -o-radial-gradient( center, $type cover,  $inner 0%, $outer 100% );
+	background: -ms-radial-gradient( center, $type cover,  $inner 0%, $outer 100% );
+	background: radial-gradient( center, $type cover,  $inner 0%, $outer 100% );
+}
\ No newline at end of file
diff --git a/doc/pub/Linalg/html/reveal.js/css/theme/template/settings.scss b/doc/pub/Linalg/html/reveal.js/css/theme/template/settings.scss
new file mode 100644
index 000000000..e706e19c5
--- /dev/null
+++ b/doc/pub/Linalg/html/reveal.js/css/theme/template/settings.scss
@@ -0,0 +1,34 @@
+// Base settings for all themes that can optionally be
+// overridden by the super-theme
+
+// Background of the presentation
+$backgroundColor: #2b2b2b;
+
+// Primary/body text
+$mainFont: 'Lato', sans-serif;
+$mainFontSize: 30px;              /* changed (by hpl) from 36px; */
+$mainColor: #eee;
+
+// Headings
+$headingMargin: 0 0 20px 0;
+$headingFont: 'League Gothic', Impact, sans-serif;
+$headingColor: #eee;
+$headingLineHeight: 0.9em;
+$headingLetterSpacing: 0.02em;
+$headingTextTransform: none;      /* changed (by hpl) from uppercase; */
+$headingTextShadow: 0px 0px 6px rgba(0,0,0,0.2);
+$heading1TextShadow: $headingTextShadow;
+
+// Links and actions
+$linkColor: #13DAEC;
+$linkColorHover: lighten( $linkColor, 20% );
+
+// Text selection
+$selectionBackgroundColor: #FF5E99;
+$selectionColor: #fff;
+
+// Generates the presentation background, can be overridden
+// to return a background image or gradient
+@mixin bodyBackground() {
+	background: $backgroundColor;
+}
\ No newline at end of file
diff --git a/doc/pub/Linalg/html/reveal.js/css/theme/template/theme.scss b/doc/pub/Linalg/html/reveal.js/css/theme/template/theme.scss
new file mode 100644
index 000000000..b1dbc2736
--- /dev/null
+++ b/doc/pub/Linalg/html/reveal.js/css/theme/template/theme.scss
@@ -0,0 +1,171 @@
+// Base theme template for reveal.js
+
+/*********************************************
+ * GLOBAL STYLES
+ *********************************************/
+
+body {
+	@include bodyBackground();
+	background-color: $backgroundColor;
+}
+
+.reveal {
+	font-family: $mainFont;
+	font-size: $mainFontSize;
+	font-weight: normal;
+	letter-spacing: -0.02em;
+	color: $mainColor;
+}
+
+::selection {
+	color: $selectionColor;
+	background: $selectionBackgroundColor;
+	text-shadow: none;
+}
+
+/*********************************************
+ * HEADERS
+ *********************************************/
+
+.reveal h1,
+.reveal h2,
+.reveal h3,
+.reveal h4,
+.reveal h5,
+.reveal h6 {
+	margin: $headingMargin;
+	color: $headingColor;
+
+	font-family: $headingFont;
+	line-height: $headingLineHeight;
+	letter-spacing: $headingLetterSpacing;
+
+	text-transform: $headingTextTransform;
+	text-shadow: $headingTextShadow;
+}
+
+.reveal h1 {
+        line-height: 1.2em;    /* added by hpl */
+	text-shadow: $heading1TextShadow;
+}
+
+
+/*********************************************
+ * LINKS
+ *********************************************/
+
+.reveal a:not(.image) {
+	color: $linkColor;
+	text-decoration: none;
+
+	-webkit-transition: color .15s ease;
+	   -moz-transition: color .15s ease;
+	    -ms-transition: color .15s ease;
+	     -o-transition: color .15s ease;
+	        transition: color .15s ease;
+}
+	.reveal a:not(.image):hover {
+		color: $linkColorHover;
+
+		text-shadow: none;
+		border: none;
+	}
+
+.reveal .roll span:after {
+	color: #fff;
+	background: darken( $linkColor, 15% );
+}
+
+
+/*********************************************
+ * IMAGES
+ *********************************************/
+
+.reveal section img {
+	margin: 15px 0px;
+	background: rgba(255,255,255,0.12);
+	border: 4px solid $mainColor;
+
+	box-shadow: 0 0 10px rgba(0, 0, 0, 0.15);
+
+	-webkit-transition: all .2s linear;
+	   -moz-transition: all .2s linear;
+	    -ms-transition: all .2s linear;
+	     -o-transition: all .2s linear;
+	        transition: all .2s linear;
+}
+
+	.reveal a:hover img {
+		background: rgba(255,255,255,0.2);
+		border-color: $linkColor;
+
+		box-shadow: 0 0 20px rgba(0, 0, 0, 0.55);
+	}
+
+
+/*********************************************
+ * NAVIGATION CONTROLS
+ *********************************************/
+
+.reveal .controls div.navigate-left,
+.reveal .controls div.navigate-left.enabled {
+	border-right-color: $linkColor;
+}
+
+.reveal .controls div.navigate-right,
+.reveal .controls div.navigate-right.enabled {
+	border-left-color: $linkColor;
+}
+
+.reveal .controls div.navigate-up,
+.reveal .controls div.navigate-up.enabled {
+	border-bottom-color: $linkColor;
+}
+
+.reveal .controls div.navigate-down,
+.reveal .controls div.navigate-down.enabled {
+	border-top-color: $linkColor;
+}
+
+.reveal .controls div.navigate-left.enabled:hover {
+	border-right-color: $linkColorHover;
+}
+
+.reveal .controls div.navigate-right.enabled:hover {
+	border-left-color: $linkColorHover;
+}
+
+.reveal .controls div.navigate-up.enabled:hover {
+	border-bottom-color: $linkColorHover;
+}
+
+.reveal .controls div.navigate-down.enabled:hover {
+	border-top-color: $linkColorHover;
+}
+
+
+/*********************************************
+ * PROGRESS BAR
+ *********************************************/
+
+.reveal .progress {
+	background: rgba(0,0,0,0.2);
+}
+	.reveal .progress span {
+		background: $linkColor;
+
+		-webkit-transition: width 800ms cubic-bezier(0.260, 0.860, 0.440, 0.985);
+		   -moz-transition: width 800ms cubic-bezier(0.260, 0.860, 0.440, 0.985);
+		    -ms-transition: width 800ms cubic-bezier(0.260, 0.860, 0.440, 0.985);
+		     -o-transition: width 800ms cubic-bezier(0.260, 0.860, 0.440, 0.985);
+		        transition: width 800ms cubic-bezier(0.260, 0.860, 0.440, 0.985);
+	}
+
+/*********************************************
+ * SLIDE NUMBER
+ *********************************************/
+.reveal .slide-number {
+  color: $linkColor;
+}
+
+
diff --git a/doc/pub/Linalg/html/reveal.js/demo.html b/doc/pub/Linalg/html/reveal.js/demo.html
new file mode 100644
index 000000000..505bb1882
--- /dev/null
+++ b/doc/pub/Linalg/html/reveal.js/demo.html
@@ -0,0 +1,410 @@
+
+
+
+	
+		
+
+		reveal.js – The HTML Presentation Framework
+
+		
+		
+
+		
+		
+
+		
+
+		
+		
+
+		
+		
+
+		
+		
+
+		
+	
+
+	
+
+		

+
+			
+			

+				

+					
Reveal.js

+					
The HTML Presentation Framework

+					

+						Created by Hakim El Hattab and contributors
+					

+				

+
+				

+					
Hello There

+					

+						reveal.js enables you to create beautiful interactive slide decks using HTML. This presentation will show you examples of what it can do.
+					

+				

+
+				
+				

+					

+						
Vertical Slides

+						
Slides can be nested inside of each other.

+						
Use the Space key to navigate through all slides.

+						

+						
+							Down arrow
+						
+					

+					

+						
Basement Level 1

+						
Nested slides are useful for adding additional detail underneath a high level horizontal slide.

+					

+					

+						
Basement Level 2

+						
That's it, time to go back up.

+						

+						
+							Up arrow
+						
+					

+				

+
+				

+					
Slides

+					

+						Not a coder? Not a problem. There's a fully-featured visual editor for authoring these, try it out at https://slides.com.
+					

+				

+
+				

+					
Point of View

+					

+						Press ESC to enter the slide overview.
+					

+					

+						Hold down alt and click on any element to zoom in on it using zoom.js. Alt + click anywhere to zoom back out.
+					

+				

+
+				

+					
Touch Optimized

+					

+						Presentations look great on touch devices, like mobile phones and tablets. Simply swipe through your slides.
+					

+				

+
+				

+					
+				

+
+				

+					

+						
Fragments

+						
Hit the next arrow...

+						
... to step through ...

+						
... a fragmented slide.

+
+						
+					

+					

+						
Fragment Styles

+						
There's different types of fragments, like:

+						
grow

+						
shrink

+						
fade-out

+						
fade-up (also down, left and right!)

+						
current-visible

+						
Highlight red blue green

+					

+				

+
+				

+					
Transition Styles

+					

+						You can select from different transitions, like: 

+						None -
+						Fade -
+						Slide -
+						Convex -
+						Concave -
+						Zoom
+					

+				

+
+				

+					
Themes

+					

+						reveal.js comes with a few themes built in: 

+						
+						Black (default) -
+						White -
+						League -
+						Sky -
+						Beige -
+						Simple 

+						Serif -
+						Blood -
+						Night -
+						Moon -
+						Solarized
+					

+				

+
+				

+					

+						
Slide Backgrounds

+						

+							Set data-background="#dddddd" on a slide to change the background color. All CSS color formats are supported.
+						

+						
+							Down arrow
+						
+					

+					

+						
Image Backgrounds

+						
<section data-background="image.png">

+					

+					

+						
Tiled Backgrounds

+						
<section data-background="image.png" data-background-repeat="repeat" data-background-size="100px">

+					

+					

+						

+							
Video Backgrounds

+							
<section data-background-video="video.mp4,video.webm">

+						

+					

+					

+						
... and GIFs!

+					

+				

+
+				

+					
Background Transitions

+					

+						Different background transitions are available via the backgroundTransition option. This one's called "zoom".
+					

+					
Reveal.configure({ backgroundTransition: 'zoom' })

+				

+
+				

+					
Background Transitions

+					

+						You can override background transitions per-slide.
+					

+					
<section data-background-transition="zoom">

+				

+
+				

+					
Pretty Code

+					

+function linkify( selector ) {
+  if( supports3DTransforms ) {
+
+    var nodes = document.querySelectorAll( selector );
+
+    for( var i = 0, len = nodes.length; i < len; i++ ) {
+      var node = nodes[i];
+
+      if( !node.className ) {
+        node.className += ' roll';
+      }
+    }
+  }
+}
+					

+					
Code syntax highlighting courtesy of highlight.js.

+				

+
+				

+					
Marvelous List

+					
    
+						
  • No order here
    • 
+						
  • Or here
    • 
+						
  • Or here
    • 
+						
  • Or here
    • 
+					

+				

+
+				

+					
Fantastic Ordered List

+					
    
+						
  1. One is smaller than...
    2. 
+						
  2. Two is smaller than...
    3. 
+						
  3. Three!
    4. 
+					

+				

+
+				

+					
Tabular Tables

+					
+						
+							
+								
+								
+								
+							
+						
+						
+							
+								
+								
+								
+							
+							
+								
+								
+								
+							
+							
+								
+								
+								
+							
+						
+					
ItemValueQuantity
Apples$17
Lemonade$218
Bread$32

+				

+
+				

+					
Clever Quotes

+					

+						These guys come in two forms, inline: The nice thing about standards is that there are so many to choose from and block:
+					

+					

+						“For years there has been a theory that millions of monkeys typing at random on millions of typewriters would
+						reproduce the entire works of Shakespeare. The Internet has proven this theory to be untrue.”
+					

+				

+
+				

+					
Intergalactic Interconnections

+					

+						You can link between slides internally,
+						like this.
+					

+				

+
+				

+					
Speaker View

+					
There's a speaker view. It includes a timer, preview of the upcoming slide as well as your speaker notes.

+					
Press the S key to try it out.

+
+					
+				

+
+				

+					
Export to PDF

+					
Presentations can be exported to PDF, here's an example:

+					
+				

+
+				

+					
Global State

+					

+						Set data-state="something" on a slide and "something"
+						will be added as a class to the document element when the slide is open. This lets you
+						apply broader style changes, like switching the page background.
+					

+				

+
+				

+					
State Events

+					

+						Additionally custom events can be triggered on a per slide basis by binding to the data-state name.
+					

+					

+Reveal.addEventListener( 'customevent', function() {
+	console.log( '"customevent" has fired' );
+} );
+					

+				

+
+				

+					
Take a Moment

+					

+						Press B or . on your keyboard to pause the presentation. This is helpful when you're on stage and want to take distracting slides off the screen.
+					

+				

+
+				

+					
Much more

+					
+				

+
+				

+					
THE END

+					

+						- Try the online editor 

+						- Source code & documentation
+					

+				

+
+			

+
+		

+
+		
+		
+
+		
+
+	
+
diff --git a/doc/pub/Linalg/html/reveal.js/plugin/multiplex/package.json b/doc/pub/Linalg/html/reveal.js/plugin/multiplex/package.json
new file mode 100644
index 000000000..bbed77a67
--- /dev/null
+++ b/doc/pub/Linalg/html/reveal.js/plugin/multiplex/package.json
@@ -0,0 +1,19 @@
+{
+  "name": "reveal-js-multiplex",
+  "version": "1.0.0",
+  "description": "reveal.js multiplex server",
+  "homepage": "http://revealjs.com",
+  "scripts": {
+    "start": "node index.js"
+  },
+  "engines": {
+    "node": "~4.1.1"
+  },
+  "dependencies": {
+    "express": "~4.13.3",
+    "grunt-cli": "~0.1.13",
+    "mustache": "~2.2.1",
+    "socket.io": "~1.3.7"
+  },
+  "license": "MIT"
+}
diff --git a/doc/pub/Linalg/html/reveal.js/test/simple.md b/doc/pub/Linalg/html/reveal.js/test/simple.md
new file mode 100644
index 000000000..c72a44079
--- /dev/null
+++ b/doc/pub/Linalg/html/reveal.js/test/simple.md
@@ -0,0 +1,12 @@
+## Slide 1.1
+
+```js
+var a = 1;
+```
+
+
+## Slide 1.2
+
+
+
+## Slide 2
diff --git a/doc/pub/Linalg/html/reveal.js/test/test-markdown-external.html b/doc/pub/Linalg/html/reveal.js/test/test-markdown-external.html
new file mode 100644
index 000000000..859d0a199
--- /dev/null
+++ b/doc/pub/Linalg/html/reveal.js/test/test-markdown-external.html
@@ -0,0 +1,36 @@
+
+
+
+	
+		
+
+		reveal.js - Test Markdown
+
+		
+		
+	
+
+	
+
+		

+  		

+
+		
+
+		
+		
+		
+		
+		
+		
+
+		
+
+	
+
diff --git a/doc/pub/Linalg/html/reveal.js/test/test-markdown-external.js b/doc/pub/Linalg/html/reveal.js/test/test-markdown-external.js
new file mode 100644
index 000000000..cab85c6f6
--- /dev/null
+++ b/doc/pub/Linalg/html/reveal.js/test/test-markdown-external.js
@@ -0,0 +1,24 @@
+
+
+Reveal.addEventListener( 'ready', function() {
+
+	QUnit.module( 'Markdown' );
+
+	test( 'Vertical separator', function() {
+		strictEqual( document.querySelectorAll( '.reveal .slides>section>section' ).length, 2, 'found two slides' );
+	});
+
+	test( 'Horizontal separator', function() {
+		strictEqual( document.querySelectorAll( '.reveal .slides>section' ).length, 2, 'found two slides' );
+	});
+
+	test( 'Language highlighter', function() {
+		strictEqual( document.querySelectorAll( '.hljs-keyword' ).length, 1, 'got rendered highlight tag.' );
+		strictEqual( document.querySelector( '.hljs-keyword' ).innerHTML, 'var', 'the same keyword: var.' );
+	});
+
+
+} );
+
+Reveal.initialize();
+
diff --git a/doc/pub/Linalg/html/reveal.js/test/test-markdown-options.html b/doc/pub/Linalg/html/reveal.js/test/test-markdown-options.html
new file mode 100644
index 000000000..5b3be9758
--- /dev/null
+++ b/doc/pub/Linalg/html/reveal.js/test/test-markdown-options.html
@@ -0,0 +1,41 @@
+
+
+
+	
+		
+
+		reveal.js - Test Markdown Options
+
+		
+		
+	
+
+	
+
+		

+		

+
+		
+
+		
+		
+		
+
+		
+
+	
+
diff --git a/doc/pub/Linalg/html/reveal.js/test/test-markdown-options.js b/doc/pub/Linalg/html/reveal.js/test/test-markdown-options.js
new file mode 100644
index 000000000..3ae13503a
--- /dev/null
+++ b/doc/pub/Linalg/html/reveal.js/test/test-markdown-options.js
@@ -0,0 +1,26 @@
+Reveal.addEventListener( 'ready', function() {
+
+	QUnit.module( 'Markdown' );
+
+	test( 'Options are set', function() {
+		strictEqual( marked.defaults.smartypants, true );
+	});
+
+	test( 'Smart quotes are activated', function() {
+		var text = document.querySelector( '.reveal .slides>section>p' ).textContent;
+
+		strictEqual( /['"]/.test( text ), false );
+		strictEqual( /[“”‘’]/.test( text ), true );
+	});
+
+} );
+
+Reveal.initialize({
+	dependencies: [
+		{ src: '../plugin/markdown/marked.js' },
+		{ src: '../plugin/markdown/markdown.js' },
+	],
+	markdown: {
+		smartypants: true
+	}
+});
diff --git a/doc/pub/Linalg/ipynb/.ipynb_checkpoints/Linalg-checkpoint.ipynb b/doc/pub/Linalg/ipynb/.ipynb_checkpoints/Linalg-checkpoint.ipynb
new file mode 100644
index 000000000..3cb1d0d22
--- /dev/null
+++ b/doc/pub/Linalg/ipynb/.ipynb_checkpoints/Linalg-checkpoint.ipynb
@@ -0,0 +1,2029 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "\n",
+    "# Data analysis and Machine Learning  Lectures: Linear Algebra and Handling of Arrays\n",
+    "\n",
+    "  \n",
+    "**Morten Hjorth-Jensen**, Department of Physics, University of Oslo and Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University\n",
+    "\n",
+    "Date: **May 24, 2018**\n",
+    "\n",
+    "Copyright 1999-2018, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license\n",
+    "\n",
+    "\n",
+    "\n",
+    "\n",
+    "## Introduction\n",
+    "The aim of this set of lectures is to review some central linear algebra algorithms that we will need in our \n",
+    "data analysis part and in the construction of Machine Learning algorithms (ML). \n",
+    "This will allow us to introduce some central programming features of high-level languages like Python and \n",
+    "compiled languages like C++ and/or Fortran. \n",
+    "\n",
+    "As discussed in the introductory notes, these series of lectures focuses both on using\n",
+    "central Python packages like **tensorflow** and **scikit-learn** as well\n",
+    "as writing your own codes for some central ML algorithms. The\n",
+    "latter can be written in a language of your choice, be it Python, Julia, R,\n",
+    "Rust, C++, Fortran etc. In order to avoid confusion however, in these lectures we will limit our\n",
+    "attention to Python, C++ and Fortran.  \n",
+    "\n",
+    "\n",
+    "\n",
+    "## Important Matrix and vector handling packages\n",
+    "\n",
+    "There are several central software packages for linear algebra and eigenvalue problems. Several of the more\n",
+    "popular ones have been wrapped into ofter software packages like those from the widely used text **Numerical Recipes**. The original source codes in many of the available packages are often taken from the widely used\n",
+    "software package LAPACK, which follows two other popular packages\n",
+    "developed in the 1970s, namely EISPACK and LINPACK.  We describe them shortly here.\n",
+    "\n",
+    "  * LINPACK: package for linear equations and least square problems.\n",
+    "\n",
+    "  * LAPACK:package for solving symmetric, unsymmetric and generalized eigenvalue problems. From LAPACK's website  it is possible to download for free all source codes from this library. Both C/C++ and Fortran versions are available.\n",
+    "\n",
+    "  * BLAS (I, II and III): (Basic Linear Algebra Subprograms) are routines that provide standard building blocks for performing basic vector and matrix operations. Blas I is vector operations, II vector-matrix operations and III matrix-matrix operations. Highly parallelized and efficient codes, all available for download from .\n",
+    "\n",
+    "When dealing with matrices and vectors a central issue is memory\n",
+    "handling and allocation. If our code is written in Python the way we\n",
+    "declare these objects and the way they are handled, interpreted and\n",
+    "used by say a linear algebra library, requires codes that interface\n",
+    "our Python program with such libraries. For Python programmers,\n",
+    "**Numpy** is by now the standard Python package for numerical arrays in\n",
+    "Python as well as the source of functions which act on these\n",
+    "arrays. These functions span from eigenvalue solvers to functions that\n",
+    "compute the mean value, variance or the covariance matrix. If you are\n",
+    "not familiar with how arrays are handled in say Python or compiled\n",
+    "languages like C++ and Fortran, the sections in this chapter may be\n",
+    "useful. For C++ programmer, **Armadillo** is widely used library for\n",
+    "linear algebra and eigenvalue problems. In addition it offers a\n",
+    "convenient way to handle and organize arrays. We discuss this library\n",
+    "as well.   Before we proceed we believe  it may be convenient to repeat some basic features of \n",
+    " matrices and vectors.\n",
+    "\n",
+    "\n",
+    "## Basic Matrix Features\n",
+    "\n",
+    "**Matrix properties reminder.**"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "$$\n",
+    "\\mathbf{A} =\n",
+    "      \\begin{bmatrix} a_{11} & a_{12} & a_{13} & a_{14} \\\\\n",
+    "                                 a_{21} & a_{22} & a_{23} & a_{24} \\\\\n",
+    "                                   a_{31} & a_{32} & a_{33} & a_{34} \\\\\n",
+    "                                  a_{41} & a_{42} & a_{43} & a_{44}\n",
+    "             \\end{bmatrix}\\qquad\n",
+    "\\mathbf{I} =\n",
+    "      \\begin{bmatrix} 1 & 0 & 0 & 0 \\\\\n",
+    "                                 0 & 1 & 0 & 0 \\\\\n",
+    "                                 0 & 0 & 1 & 0 \\\\\n",
+    "                                 0 & 0 & 0 & 1\n",
+    "             \\end{bmatrix}\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Basic Matrix Features\n",
+    "\n",
+    "The inverse of a matrix is defined by"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "$$\n",
+    "\\mathbf{A}^{-1} \\cdot \\mathbf{A} = I\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Basic Matrix Features\n",
+    "\n",
+    "**Matrix Properties Reminder.**\n",
+    "\n",
+    "\n",
+    "\n",
+    "\n",
+    "\n",
+    "\n",
+    "\n",
+    "\n",
+    "\n",
+    "\n",
+    "\n",
+    "\n",
+    "\n",
+    "
              Relations                      Name                                  matrix elements                             
   $A = A^{T}$                                   symmetric              $a_{ij} = a_{ji}$                                                           
   $A = \\left (A^{T} \\right )^{-1}$              real orthogonal        $\\sum_k a_{ik} a_{jk} = \\sum_k a_{ki} a_{kj} = \\delta_{ij}$                 
   $A = A^{ * }$                                 real matrix            $a_{ij} = a_{ij}^{ * }$                                                     
   $A = A^{\\dagger}$                             hermitian              $a_{ij} = a_{ji}^{ * }$                                                     
   $A = \\left (A^{\\dagger} \\right )^{-1}$        unitary                $\\sum_k a_{ik} a_{jk}^{ * } = \\sum_k a_{ki}^{ * } a_{kj} = \\delta_{ij}$     
\n",
+    "\n",
+    "\n",
+    "\n",
+    "## Some famous Matrices\n",
+    "\n",
+    "  * Diagonal if $a_{ij}=0$ for $i\\ne j$\n",
+    "\n",
+    "  * Upper triangular if $a_{ij}=0$ for $i > j$\n",
+    "\n",
+    "  * Lower triangular if $a_{ij}=0$ for $i < j$\n",
+    "\n",
+    "  * Upper Hessenberg if $a_{ij}=0$ for $i > j+1$\n",
+    "\n",
+    "  * Lower Hessenberg if $a_{ij}=0$ for $i < j+1$\n",
+    "\n",
+    "  * Tridiagonal if $a_{ij}=0$ for $|i -j| > 1$\n",
+    "\n",
+    "  * Lower banded with bandwidth $p$: $a_{ij}=0$ for $i > j+p$\n",
+    "\n",
+    "  * Upper banded with bandwidth $p$: $a_{ij}=0$ for $i < j+p$\n",
+    "\n",
+    "  * Banded, block upper triangular, block lower triangular....\n",
+    "\n",
+    "## Basic Matrix Features\n",
+    "\n",
+    "**Some Equivalent Statements.**\n",
+    "\n",
+    "For an $N\\times N$ matrix  $\\mathbf{A}$ the following properties are all equivalent\n",
+    "\n",
+    "  * If the inverse of $\\mathbf{A}$ exists, $\\mathbf{A}$ is nonsingular.\n",
+    "\n",
+    "  * The equation $\\mathbf{Ax}=0$ implies $\\mathbf{x}=0$.\n",
+    "\n",
+    "  * The rows of $\\mathbf{A}$ form a basis of $R^N$.\n",
+    "\n",
+    "  * The columns of $\\mathbf{A}$ form a basis of $R^N$.\n",
+    "\n",
+    "  * $\\mathbf{A}$ is a product of elementary matrices.\n",
+    "\n",
+    "  * $0$ is not eigenvalue of $\\mathbf{A}$.\n",
+    "\n",
+    "\n",
+    "\n",
+    "## Numpy and arrays\n",
+    "[Numpy](http://www.numpy.org/) provides an easy way to handle arrays in Python. The standard way to import this library is as"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "metadata": {
+    "collapsed": false
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "[ 1.58706198  0.76481088 -0.96520008  0.51539874 -0.47572211  0.25939945\n",
+      " -0.95294024 -0.57405882  0.50879676 -0.24654269]\n"
+     ]
+    }
+   ],
+   "source": [
+    "import numpy as np\n",
+    "n = 10\n",
+    "x = np.random.normal(size=n)\n",
+    "print(x)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Here we have defined a vector $x$ with $n=10$ elements with its values given by the Normal distribution $N(0,1)$.\n",
+    "Another alternative is to declare a vector as follows"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "metadata": {
+    "collapsed": false
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "[1 2 3]\n"
+     ]
+    }
+   ],
+   "source": [
+    "import numpy as np\n",
+    "x = np.array([1, 2, 3])\n",
+    "print(x)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Here we have defined a vector with three elements, with $x_0=1$, $x_1=2$ and $x_2=3$. Note that both Python and C++\n",
+    "start numbering array elements from $0$ and on. This means that a vector with $n$ elements has a sequence of entities $x_0, x_1, x_2, \\dots, x_{n-1}$. We could also let (recommended) Numpy to compute the logarithms of a specific array as"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "metadata": {
+    "collapsed": false
+   },
+   "outputs": [],
+   "source": [
+    "import numpy as np\n",
+    "x = np.log(np.array([4, 7, 8]))\n",
+    "print(x)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Here we have used Numpy's unary function $np.log$. This function is\n",
+    "highly tuned to compute array elements since the code is vectorized\n",
+    "and does not require looping. We normaly recommend that you use the\n",
+    "Numpy intrinsic functions instead of the corresponding **log** function\n",
+    "from Python's **math** module. The looping is done explicitely by the\n",
+    "**np.log** function. The alternative, and slower way to compute the\n",
+    "logarithms of a vector would be to write"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "metadata": {
+    "collapsed": false
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "[1 1 2]\n"
+     ]
+    }
+   ],
+   "source": [
+    "import numpy as np\n",
+    "from math import log\n",
+    "x = np.array([4, 7, 8])\n",
+    "for i in range(0, len(x)):\n",
+    "    x[i] = log(x[i])\n",
+    "print(x)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "We note that our code is much longer already and we need to import the **log** function from the **math** module. \n",
+    "The attentive reader will also notice that the output is $[1, 1, 2]$. Python interprets automacally our numbers as integers (like the **automatic** keyword in C++). To change this we could define our array elements to be double precision numbers as"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 4,
+   "metadata": {
+    "collapsed": false
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "[1.38629436 1.94591015 2.07944154]\n"
+     ]
+    }
+   ],
+   "source": [
+    "import numpy as np\n",
+    "x = np.log(np.array([4, 7, 8], dtype = np.float64))\n",
+    "print(x)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "or simply write them as double precision numbers (Python uses 64 bits as default for floating point type variables), that is"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 6,
+   "metadata": {
+    "collapsed": false
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "[1.38629436 1.94591015 2.07944154]\n"
+     ]
+    }
+   ],
+   "source": [
+    "import numpy as np\n",
+    "x = np.log(np.array([4.0, 7.0, 8.0]))\n",
+    "print(x)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "To check the number of bytes (remember that one byte contains eight bits for double precision variables), you can use simple use the **itemsize** functionality (the array $x$ is actually an object which inherits the functionalities defined in Numpy) as"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 7,
+   "metadata": {
+    "collapsed": false
+   },
+   "outputs": [],
+   "source": [
+    "import numpy as np\n",
+    "x = np.log(np.array([4.0, 7.0, 8.0]))\n",
+    "print(x.itemsize)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Matrices in Python\n",
+    "Having defined vectors, we are now ready to try out matrices. We can define a $3 \\times 3 $ real matrix $\\hat{A}$\n",
+    "as (recall that we user lowercase letters for vectors and uppercase letters for matrices)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 8,
+   "metadata": {
+    "collapsed": false
+   },
+   "outputs": [],
+   "source": [
+    "import numpy as np\n",
+    "A = np.log(np.array([ [4.0, 7.0, 8.0], [3.0, 10.0, 11.0], [4.0, 5.0, 7.0] ]))\n",
+    "print(A)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "If we use the **shape** function we would get $(3, 3)$ as output, that is verifying that our matrix is a $3\\times 3$ matrix. We can slice the matrix and print for example the first column (Python organized matrix elements in a row-major order, see below) as"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 9,
+   "metadata": {
+    "collapsed": false
+   },
+   "outputs": [],
+   "source": [
+    "import numpy as np\n",
+    "A = np.log(np.array([ [4.0, 7.0, 8.0], [3.0, 10.0, 11.0], [4.0, 5.0, 7.0] ]))\n",
+    "# print the first column, row-major order and elements start with 0\n",
+    "print(A[:,0])"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "We can continue this was by printing out other columns or rows. The example here prints out the second column"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 10,
+   "metadata": {
+    "collapsed": false
+   },
+   "outputs": [],
+   "source": [
+    "import numpy as np\n",
+    "A = np.log(np.array([ [4.0, 7.0, 8.0], [3.0, 10.0, 11.0], [4.0, 5.0, 7.0] ]))\n",
+    "# print the first column, row-major order and elements start with 0\n",
+    "print(A[1,:])"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Numpy contains many other functionalities that allow us to slice, subdivide etc etc arrays. We strongly recommend that you look up the [Numpy website for more details](http://www.numpy.org/). Useful functions when defining a matrix are the **np.zeros** function which declares a matrix of a given dimension and sets all elements to zero"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 11,
+   "metadata": {
+    "collapsed": false
+   },
+   "outputs": [],
+   "source": [
+    "import numpy as np\n",
+    "n = 10\n",
+    "# define a matrix of dimension 10 x 10 and set all elements to zero\n",
+    "A = np.zeros( (n, n) )\n",
+    "print(A)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "or initializing all elements to"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 12,
+   "metadata": {
+    "collapsed": false
+   },
+   "outputs": [],
+   "source": [
+    "import numpy as np\n",
+    "n = 10\n",
+    "# define a matrix of dimension 10 x 10 and set all elements to one\n",
+    "A = np.ones( (n, n) )\n",
+    "print(A)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "or as unitarily distributed random numbers (see the material on random number generators in the statistics part)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 13,
+   "metadata": {
+    "collapsed": false
+   },
+   "outputs": [],
+   "source": [
+    "import numpy as np\n",
+    "n = 10\n",
+    "# define a matrix of dimension 10 x 10 and set all elements to random numbers with x \\in [0, 1]\n",
+    "A = np.random.rand(n, n)\n",
+    "print(A)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "As we will see throughout these lectures, there are several extremely useful functionalities in Numpy.\n",
+    "As an example, consider the discussion of the covariance matrix. Suppose we have defined three vectors\n",
+    "$\\hat{x}, \\hat{y}, \\hat{z}$ with $n$ elements each. The covariance matrix is defined as"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "$$\n",
+    "\\hat{\\Sigma} = \\begin{bmatrix} \\sigma_{xx} & \\sigma_{xy} & \\sigma_{xz} \\\\\n",
+    "                              \\sigma_{yx} & \\sigma_{yy} & \\sigma_{yz} \\\\\n",
+    "                              \\sigma_{zx} & \\sigma_{zy} & \\sigma_{zz} \n",
+    "             \\end{bmatrix},\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "where for example"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "$$\n",
+    "\\sigma_{xy} =\\frac{1}{n} \\sum_{i=0}^{n-1}(x_i- \\overline{x})(y_i- \\overline{y}).\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "The Numpy function **np.cov** calculates the covariance elements using the factor $1/(n-1)$ instead of $1/n$ since it assumes we do not have the exact mean values. For a more in-depth discussion of the covariance and covariance matrix and its meaning, we refer you to the lectures on statistics. \n",
+    "The following simple function uses the **np.vstack** function which takes each vector of dimension $1\\times n$ and produces a $ 3\\times n$ matrix $\\hat{W}$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "$$\n",
+    "\\hat{W} = \\begin{bmatrix} x_0 & y_0 & z_0 \\\\\n",
+    "                          x_1 & y_1 & z_1 \\\\\n",
+    "                          x_2 & y_2 & z_2 \\\\\n",
+    "                          \\dots & \\dots & \\dots \\\\\n",
+    "                          x_{n-2} & y_{n-2} & z_{n-2} \\\\\n",
+    "                          x_{n-1} & y_{n-1} & z_{n-1}\n",
+    "             \\end{bmatrix},\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "which in turn is converted into into the $3 times 3$ covariance matrix\n",
+    "$\\hat{\\Sigma}$ via the Numpy function **np.cov()**. In our review of\n",
+    "statistical functions and quantities we will discuss more about the\n",
+    "meaning of the covariance matrix. Here we note that we can calculate\n",
+    "the mean value of each set of samples $\\hat{x}$ etc using the Numpy\n",
+    "function **np.mean(x)**. We can also extract the eigenvalues of the\n",
+    "covariance matrix through the **np.linalg.eig()** function."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 14,
+   "metadata": {
+    "collapsed": false
+   },
+   "outputs": [],
+   "source": [
+    "# Importing various packages\n",
+    "import numpy as np\n",
+    "\n",
+    "n = 100\n",
+    "x = np.random.normal(size=n)\n",
+    "print(np.mean(x))\n",
+    "y = 4+3*x+np.random.normal(size=n)\n",
+    "print(np.mean(y))\n",
+    "z = x**3+np.random.normal(size=n)\n",
+    "print(np.mean(z))\n",
+    "W = np.vstack((x, y, z))\n",
+    "Sigma = np.cov(W)\n",
+    "print(Sigma)\n",
+    "Eigvals, Eigvecs = np.linalg.eig(Sigma)\n",
+    "print(Eigvals)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Matrix Handling in C/C++, Static and Dynamical allocation\n",
+    "\n",
+    "**Static.**\n",
+    "\n",
+    "We have  an $N\\times N$ matrix A  with $N=100$\n",
+    "In C/C++ this would be  defined as"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "           int N = 100;\n",
+    "           double A[100][100];\n",
+    "           //   initialize all elements to zero\n",
+    "           for(i=0 ; i < N ; i++) {\n",
+    "              for(j=0 ; j < N ; j++) {\n",
+    "                 A[i][j] = 0.0;\n",
+    "        \n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Note the way the matrix is organized, row-major order.\n",
+    "\n",
+    "\n",
+    "\n",
+    "## Matrix Handling in C/C++\n",
+    "\n",
+    "**Row Major Order, Addition.**\n",
+    "\n",
+    "We have  $N\\times N$ matrices A, B and C and we wish to\n",
+    "evaluate $A=B+C$."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "$$\n",
+    "\\mathbf{A}= \\mathbf{B}\\pm\\mathbf{C}  \\Longrightarrow a_{ij} = b_{ij}\\pm c_{ij},\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "In C/C++ this would be coded like"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "           for(i=0 ; i < N ; i++) {\n",
+    "              for(j=0 ; j < N ; j++) {\n",
+    "                 a[i][j] = b[i][j]+c[i][j]\n",
+    "        \n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Matrix Handling in C/C++\n",
+    "\n",
+    "**Row Major Order, Multiplication.**\n",
+    "\n",
+    "We have  $N\\times N$ matrices A, B and C and we wish to\n",
+    "evaluate $A=BC$."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "$$\n",
+    "\\mathbf{A}=\\mathbf{BC}   \\Longrightarrow a_{ij} = \\sum_{k=1}^{n} b_{ik}c_{kj},\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "In C/C++ this would be coded like"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "           for(i=0 ; i < N ; i++) {\n",
+    "              for(j=0 ; j < N ; j++) {\n",
+    "                 for(k=0 ; k < N ; k++) {\n",
+    "                    a[i][j]+=b[i][k]*c[k][j];\n",
+    "        \n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Dynamic memory allocation in C/C++\n",
+    "\n",
+    "At least three possibilities in this course\n",
+    "\n",
+    "  * Do it yourself\n",
+    "\n",
+    "  * Use the functions provided in the library package lib.cpp\n",
+    "\n",
+    "  * Use Armadillo  (a C++ linear algebra library, discussion both here and at lab). \n",
+    "\n",
+    "## Matrix Handling in C/C++, Dynamic Allocation\n",
+    "\n",
+    "**Do it yourself.**"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "        int N;\n",
+    "        double **  A;\n",
+    "        A = new double*[N]\n",
+    "        for ( i = 0; i < N; i++)\n",
+    "            A[i] = new double[N];\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Always free space when you don't need an array anymore."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "        for ( i = 0; i < N; i++)\n",
+    "            delete[] A[i];\n",
+    "        delete[] A;\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Armadillo, recommended!!\n",
+    "\n",
+    "  * Armadillo is a C++ linear algebra library (matrix maths) aiming towards a good balance between speed and ease of use. The syntax is deliberately similar to Matlab.\n",
+    "\n",
+    "  * Integer, floating point and complex numbers are supported, as well as a subset of trigonometric and statistics functions. Various matrix decompositions are provided through optional integration with LAPACK, or one of its high performance drop-in replacements (such as the multi-threaded MKL or ACML libraries).\n",
+    "\n",
+    "  * A delayed evaluation approach is employed (at compile-time) to combine several operations into one and reduce (or eliminate) the need for temporaries. This is accomplished through recursive templates and template meta-programming.\n",
+    "\n",
+    "  * Useful for conversion of research code into production environments, or if C++ has been decided as the language of choice, due to speed and/or integration capabilities.\n",
+    "\n",
+    "  * The library is open-source software, and is distributed under a license that is useful in both open-source and commercial/proprietary contexts.\n",
+    "\n",
+    "## Armadillo, simple examples"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "        #include \n",
+    "        #include \n",
+    "        \n",
+    "        using namespace std;\n",
+    "        using namespace arma;\n",
+    "        \n",
+    "        int main(int argc, char** argv)\n",
+    "          {\n",
+    "          mat A = randu(5,5);\n",
+    "          mat B = randu(5,5);\n",
+    "        \n",
+    "          cout << A*B << endl;\n",
+    "        \n",
+    "          return 0;\n",
+    "        \n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Armadillo, how to compile and install\n",
+    "\n",
+    "For people using Ubuntu, Debian, Linux Mint, simply go to the synaptic package manager and install\n",
+    "armadillo from there.\n",
+    "You may have to install Lapack as well.\n",
+    "For Mac and Windows users, follow the instructions from the webpage\n",
+    ".\n",
+    "To compile, use for example (linux/ubuntu)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "        c++ -O2 -o program.x program.cpp  -larmadillo -llapack -lblas\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "where the `-l` option indicates the library you wish to link to.\n",
+    "\n",
+    "For OS X users you may have to declare the paths to the include files and the libraries as"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "        c++ -O2 -o program.x program.cpp  -L/usr/local/lib -I/usr/local/include -larmadillo -llapack -lblas\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Armadillo, simple examples"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "        #include \n",
+    "        #include \"armadillo\"\n",
+    "        using namespace arma;\n",
+    "        using namespace std;\n",
+    "        \n",
+    "        int main(int argc, char** argv)\n",
+    "          {\n",
+    "          // directly specify the matrix size (elements are uninitialised)\n",
+    "          mat A(2,3);\n",
+    "          // .n_rows = number of rows    (read only)\n",
+    "          // .n_cols = number of columns (read only)\n",
+    "          cout << \"A.n_rows = \" << A.n_rows << endl;\n",
+    "          cout << \"A.n_cols = \" << A.n_cols << endl;\n",
+    "          // directly access an element (indexing starts at 0)\n",
+    "          A(1,2) = 456.0;\n",
+    "          A.print(\"A:\");\n",
+    "          // scalars are treated as a 1x1 matrix,\n",
+    "          // hence the code below will set A to have a size of 1x1\n",
+    "          A = 5.0;\n",
+    "          A.print(\"A:\");\n",
+    "          // if you want a matrix with all elements set to a particular value\n",
+    "          // the .fill() member function can be used\n",
+    "          A.set_size(3,3);\n",
+    "          A.fill(5.0);  A.print(\"A:\");\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Armadillo, simple examples"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "          mat B;\n",
+    "        \n",
+    "          // endr indicates \"end of row\"\n",
+    "          B << 0.555950 << 0.274690 << 0.540605 << 0.798938 << endr\n",
+    "            << 0.108929 << 0.830123 << 0.891726 << 0.895283 << endr\n",
+    "            << 0.948014 << 0.973234 << 0.216504 << 0.883152 << endr\n",
+    "            << 0.023787 << 0.675382 << 0.231751 << 0.450332 << endr;\n",
+    "        \n",
+    "          // print to the cout stream\n",
+    "          // with an optional string before the contents of the matrix\n",
+    "          B.print(\"B:\");\n",
+    "        \n",
+    "          // the << operator can also be used to print the matrix\n",
+    "          // to an arbitrary stream (cout in this case)\n",
+    "          cout << \"B:\" << endl << B << endl;\n",
+    "          // save to disk\n",
+    "          B.save(\"B.txt\", raw_ascii);\n",
+    "          // load from disk\n",
+    "          mat C;\n",
+    "          C.load(\"B.txt\");\n",
+    "          C += 2.0 * B;\n",
+    "          C.print(\"C:\");\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Armadillo, simple examples"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "          // submatrix types:\n",
+    "          //\n",
+    "          // .submat(first_row, first_column, last_row, last_column)\n",
+    "          // .row(row_number)\n",
+    "          // .col(column_number)\n",
+    "          // .cols(first_column, last_column)\n",
+    "          // .rows(first_row, last_row)\n",
+    "        \n",
+    "          cout << \"C.submat(0,0,3,1) =\" << endl;\n",
+    "          cout << C.submat(0,0,3,1) << endl;\n",
+    "        \n",
+    "          // generate the identity matrix\n",
+    "          mat D = eye(4,4);\n",
+    "        \n",
+    "          D.submat(0,0,3,1) = C.cols(1,2);\n",
+    "          D.print(\"D:\");\n",
+    "        \n",
+    "          // transpose\n",
+    "          cout << \"trans(B) =\" << endl;\n",
+    "          cout << trans(B) << endl;\n",
+    "        \n",
+    "          // maximum from each column (traverse along rows)\n",
+    "          cout << \"max(B) =\" << endl;\n",
+    "          cout << max(B) << endl;\n",
+    "        \n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Armadillo, simple examples"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "          // maximum from each row (traverse along columns)\n",
+    "          cout << \"max(B,1) =\" << endl;\n",
+    "          cout << max(B,1) << endl;\n",
+    "          // maximum value in B\n",
+    "          cout << \"max(max(B)) = \" << max(max(B)) << endl;\n",
+    "          // sum of each column (traverse along rows)\n",
+    "          cout << \"sum(B) =\" << endl;\n",
+    "          cout << sum(B) << endl;\n",
+    "          // sum of each row (traverse along columns)\n",
+    "          cout << \"sum(B,1) =\" << endl;\n",
+    "          cout << sum(B,1) << endl;\n",
+    "          // sum of all elements\n",
+    "          cout << \"sum(sum(B)) = \" << sum(sum(B)) << endl;\n",
+    "          cout << \"accu(B)     = \" << accu(B) << endl;\n",
+    "          // trace = sum along diagonal\n",
+    "          cout << \"trace(B)    = \" << trace(B) << endl;\n",
+    "          // random matrix -- values are uniformly distributed in the [0,1] interval\n",
+    "          mat E = randu(4,4);\n",
+    "          E.print(\"E:\");\n",
+    "        \n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Armadillo, simple examples"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "          // row vectors are treated like a matrix with one row\n",
+    "          rowvec r;\n",
+    "          r << 0.59499 << 0.88807 << 0.88532 << 0.19968;\n",
+    "          r.print(\"r:\");\n",
+    "        \n",
+    "          // column vectors are treated like a matrix with one column\n",
+    "          colvec q;\n",
+    "          q << 0.81114 << 0.06256 << 0.95989 << 0.73628;\n",
+    "          q.print(\"q:\");\n",
+    "        \n",
+    "          // dot or inner product\n",
+    "          cout << \"as_scalar(r*q) = \" << as_scalar(r*q) << endl;\n",
+    "        \n",
+    "            // outer product\n",
+    "          cout << \"q*r =\" << endl;\n",
+    "          cout << q*r << endl;\n",
+    "        \n",
+    "        \n",
+    "          // sum of three matrices (no temporary matrices are created)\n",
+    "          mat F = B + C + D;\n",
+    "          F.print(\"F:\");\n",
+    "        \n",
+    "            return 0;\n",
+    "        \n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Armadillo, simple examples"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "        #include \n",
+    "        #include \"armadillo\"\n",
+    "        using namespace arma;\n",
+    "        using namespace std;\n",
+    "        \n",
+    "        int main(int argc, char** argv)\n",
+    "          {\n",
+    "          cout << \"Armadillo version: \" << arma_version::as_string() << endl;\n",
+    "        \n",
+    "          mat A;\n",
+    "        \n",
+    "          A << 0.165300 << 0.454037 << 0.995795 << 0.124098 << 0.047084 << endr\n",
+    "            << 0.688782 << 0.036549 << 0.552848 << 0.937664 << 0.866401 << endr\n",
+    "            << 0.348740 << 0.479388 << 0.506228 << 0.145673 << 0.491547 << endr\n",
+    "            << 0.148678 << 0.682258 << 0.571154 << 0.874724 << 0.444632 << endr\n",
+    "            << 0.245726 << 0.595218 << 0.409327 << 0.367827 << 0.385736 << endr;\n",
+    "        \n",
+    "          A.print(\"A =\");\n",
+    "        \n",
+    "          // determinant\n",
+    "          cout << \"det(A) = \" << det(A) << endl;\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Armadillo, simple examples"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "          // inverse\n",
+    "          cout << \"inv(A) = \" << endl << inv(A) << endl;\n",
+    "          double k = 1.23;\n",
+    "        \n",
+    "          mat    B = randu(5,5);\n",
+    "          mat    C = randu(5,5);\n",
+    "        \n",
+    "          rowvec r = randu(5);\n",
+    "          colvec q = randu(5);\n",
+    "        \n",
+    "        \n",
+    "          // examples of some expressions\n",
+    "          // for which optimised implementations exist\n",
+    "          // optimised implementation of a trinary expression\n",
+    "          // that results in a scalar\n",
+    "          cout << \"as_scalar( r*inv(diagmat(B))*q ) = \";\n",
+    "          cout << as_scalar( r*inv(diagmat(B))*q ) << endl;\n",
+    "        \n",
+    "          // example of an expression which is optimised\n",
+    "          // as a call to the dgemm() function in BLAS:\n",
+    "          cout << \"k*trans(B)*C = \" << endl << k*trans(B)*C;\n",
+    "        \n",
+    "            return 0;\n",
+    "        \n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Gaussian Elimination\n",
+    "\n",
+    "We start with the linear set of equations"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "$$\n",
+    "\\mathbf{A}\\mathbf{x} = \\mathbf{w}.\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "We assume also that the matrix $\\mathbf{A}$ is non-singular and that the\n",
+    "matrix elements along the diagonal satisfy $a_{ii} \\ne 0$. Simple $4\\times 4 $ example"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "$$\n",
+    "\\begin{bmatrix}\n",
+    "                           a_{11}& a_{12} &a_{13}& a_{14}\\\\\n",
+    "                           a_{21}& a_{22} &a_{23}& a_{24}\\\\\n",
+    "                           a_{31}& a_{32} &a_{33}& a_{34}\\\\\n",
+    "                           a_{41}& a_{42} &a_{43}& a_{44}\\\\\n",
+    "                      \\end{bmatrix} \\begin{bmatrix}\n",
+    "                           x_1\\\\\n",
+    "                           x_2\\\\\n",
+    "                           x_3 \\\\\n",
+    "                           x_4  \\\\\n",
+    "                      \\end{bmatrix}\n",
+    "  =\\begin{bmatrix}\n",
+    "                           w_1\\\\\n",
+    "                           w_2\\\\\n",
+    "                           w_3 \\\\\n",
+    "                           w_4\\\\\n",
+    "                      \\end{bmatrix}.\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Gaussian Elimination\n",
+    "or"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "$$\n",
+    "a_{11}x_1 +a_{12}x_2 +a_{13}x_3 + a_{14}x_4=w_1 \\nonumber\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "$$\n",
+    "a_{21}x_1 + a_{22}x_2 + a_{23}x_3 + a_{24}x_4=w_2 \\nonumber\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "$$\n",
+    "a_{31}x_1 + a_{32}x_2 + a_{33}x_3 + a_{34}x_4=w_3 \\nonumber\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "$$\n",
+    "a_{41}x_1 + a_{42}x_2 + a_{43}x_3 + a_{44}x_4=w_4. \\nonumber\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Gaussian Elimination\n",
+    "\n",
+    "The basic idea of Gaussian elimination is to use the first equation to eliminate the first unknown $x_1$\n",
+    "from the remaining $n-1$ equations. Then we use the new second equation to eliminate the second unknown\n",
+    "$x_2$ from the remaining $n-2$ equations. With $n-1$ such eliminations\n",
+    "we obtain a so-called upper triangular set of equations of the form"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "$$\n",
+    "b_{11}x_1 +b_{12}x_2 +b_{13}x_3 + b_{14}x_4=y_1 \\nonumber\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "$$\n",
+    "b_{22}x_2 + b_{23}x_3 + b_{24}x_4=y_2 \\nonumber\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "$$\n",
+    "b_{33}x_3 + b_{34}x_4=y_3 \\nonumber\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "\n",
+    "
\n",
+    "\n",
+    "$$\n",
+    "b_{44}x_4=y_4. \\nonumber\n",
+    "\\label{eq:gaussbacksub} \\tag{1}\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "We can solve this system of equations recursively starting from $x_n$ (in our case $x_4$) and proceed with\n",
+    "what is called a backward substitution. \n",
+    "\n",
+    "## Gaussian Elimination\n",
+    "This process can be expressed mathematically as"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "\n",
+    "
\n",
+    "\n",
+    "$$\n",
+    "\\begin{equation}\n",
+    "   x_m = \\frac{1}{b_{mm}}\\left(y_m-\\sum_{k=m+1}^nb_{mk}x_k\\right)\\quad m=n-1,n-2,\\dots,1.\n",
+    "\\label{_auto1} \\tag{2}\n",
+    "\\end{equation}\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "To arrive at such an upper triangular system of equations, we start by eliminating\n",
+    "the unknown $x_1$ for $j=2,n$. We achieve this by multiplying the first equation by $a_{j1}/a_{11}$ and then subtract\n",
+    "the result from the $j$th equation. We assume obviously that $a_{11}\\ne 0$ and that\n",
+    "$\\mathbf{A}$ is not singular.\n",
+    "\n",
+    "## Gaussian Elimination\n",
+    "\n",
+    "Our actual $4\\times 4$ example reads after the first operation"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "$$\n",
+    "\\begin{bmatrix}\n",
+    "                           a_{11}& a_{12} &a_{13}& a_{14}\\\\\n",
+    "                           0& (a_{22}-\\frac{a_{21}a_{12}}{a_{11}}) &(a_{23}-\\frac{a_{21}a_{13}}{a_{11}}) & (a_{24}-\\frac{a_{21}a_{14}}{a_{11}})\\\\\n",
+    "0& (a_{32}-\\frac{a_{31}a_{12}}{a_{11}})& (a_{33}-\\frac{a_{31}a_{13}}{a_{11}})& (a_{34}-\\frac{a_{31}a_{14}}{a_{11}})\\\\\n",
+    "0&(a_{42}-\\frac{a_{41}a_{12}}{a_{11}}) &(a_{43}-\\frac{a_{41}a_{13}}{a_{11}}) & (a_{44}-\\frac{a_{41}a_{14}}{a_{11}}) \\\\\n",
+    "                      \\end{bmatrix} \\begin{bmatrix}\n",
+    "                           x_1\\\\\n",
+    "                           x_2\\\\\n",
+    "                           x_3 \\\\\n",
+    "                           x_4  \\\\\n",
+    "                      \\end{bmatrix} \n",
+    "  =\\begin{bmatrix}\n",
+    "                           y_1\\\\\n",
+    "                           w_2^{(2)}\\\\\n",
+    "                           w_3^{(2)} \\\\\n",
+    "                           w_4^{(2)}\\\\\n",
+    "                      \\end{bmatrix},\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "or"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "$$\n",
+    "b_{11}x_1 +b_{12}x_2 +b_{13}x_3 + b_{14}x_4=y_1 \\nonumber\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "$$\n",
+    "a^{(2)}_{22}x_2 + a^{(2)}_{23}x_3 + a^{(2)}_{24}x_4=w^{(2)}_2 \\nonumber\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "$$\n",
+    "a^{(2)}_{32}x_2 + a^{(2)}_{33}x_3 + a^{(2)}_{34}x_4=w^{(2)}_3 \\nonumber\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "$$\n",
+    "a^{(2)}_{42}x_2 + a^{(2)}_{43}x_3 + a^{(2)}_{44}x_4=w^{(2)}_4, \\nonumber\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "\n",
+    "
\n",
+    "\n",
+    "$$\n",
+    "\\begin{equation} \n",
+    "\\label{_auto2} \\tag{3}\n",
+    "\\end{equation}\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Gaussian Elimination\n",
+    "\n",
+    "The new coefficients are"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "\n",
+    "
\n",
+    "\n",
+    "$$\n",
+    "\\begin{equation}\n",
+    "   b_{1k} = a_{1k}^{(1)} \\quad k=1,\\dots,n,\n",
+    "\\label{_auto3} \\tag{4}\n",
+    "\\end{equation}\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "where each $a_{1k}^{(1)}$ is equal to the original $a_{1k}$ element. The other coefficients are"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "\n",
+    "
\n",
+    "\n",
+    "$$\n",
+    "\\begin{equation}\n",
+    "a_{jk}^{(2)} = a_{jk}^{(1)}-\\frac{a_{j1}^{(1)}a_{1k}^{(1)}}{a_{11}^{(1)}} \\quad j,k=2,\\dots,n,\n",
+    "\\label{_auto4} \\tag{5}\n",
+    "\\end{equation}\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "with a new right-hand side given by"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "\n",
+    "
\n",
+    "\n",
+    "$$\n",
+    "\\begin{equation}\n",
+    "y_{1}=w_1^{(1)}, \\quad w_j^{(2)} =w_j^{(1)}-\\frac{a_{j1}^{(1)}w_1^{(1)}}{a_{11}^{(1)}} \\quad j=2,\\dots,n.\n",
+    "\\label{_auto5} \\tag{6}\n",
+    "\\end{equation}\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "We have also set $w_1^{(1)}=w_1$, the original vector element.\n",
+    "We see that the system of unknowns $x_1,\\dots,x_n$ is transformed into an $(n-1)\\times (n-1)$ problem.\n",
+    "\n",
+    "## Gaussian Elimination\n",
+    "\n",
+    "This step is called forward substitution.\n",
+    "Proceeding with these substitutions, we obtain the\n",
+    "general expressions for the new coefficients"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "\n",
+    "
\n",
+    "\n",
+    "$$\n",
+    "\\begin{equation}\n",
+    "   a_{jk}^{(m+1)} = a_{jk}^{(m)}-\\frac{a_{jm}^{(m)}a_{mk}^{(m)}}{a_{mm}^{(m)}} \\quad j,k=m+1,\\dots,n,\n",
+    "\\label{_auto6} \\tag{7}\n",
+    "\\end{equation}\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "with $m=1,\\dots,n-1$ and a\n",
+    "right-hand side given by"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "\n",
+    "
\n",
+    "\n",
+    "$$\n",
+    "\\begin{equation}\n",
+    "   w_j^{(m+1)} =w_j^{(m)}-\\frac{a_{jm}^{(m)}w_m^{(m)}}{a_{mm}^{(m)}}\\quad j=m+1,\\dots,n.\n",
+    "\\label{_auto7} \\tag{8}\n",
+    "\\end{equation}\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "This set of $n-1$ elimations leads us to an equations which is solved by back substitution.\n",
+    "If the arithmetics is exact and the matrix $\\mathbf{A}$ is not singular, then the computed answer will be exact.\n",
+    "\n",
+    "Even though the matrix elements along the diagonal are not zero,\n",
+    "numerically small numbers may appear and subsequent divisions may lead to large numbers, which, if added\n",
+    "to a small number may yield losses of precision. Suppose for example that our first division in $(a_{22}-a_{21}a_{12}/a_{11})$\n",
+    "results in $-10^{-7}$ and that $a_{22}$ is one.\n",
+    "one. We are then\n",
+    "adding $10^7+1$. With single precision this results in $10^7$.\n",
+    "\n",
+    "\n",
+    "\n",
+    "## Linear Algebra Methods\n",
+    "\n",
+    "  * Gaussian elimination, $O(2/3n^3)$ flops, general matrix\n",
+    "\n",
+    "  * LU decomposition, upper triangular and lower tridiagonal matrices, $O(2/3n^3)$ flops, general matrix. Get easily the inverse, determinant and can solve linear equations with back-substitution only, $O(n^2)$ flops\n",
+    "\n",
+    "  * Cholesky decomposition. Real symmetric or hermitian positive definite matrix, $O(1/3n^3)$ flops.\n",
+    "\n",
+    "  * Tridiagonal linear systems, important for differential equations. Normally positive definite and non-singular. $O(8n)$ flops for symmetric. Special case of banded matrices.\n",
+    "\n",
+    "  * Singular value decomposition\n",
+    "\n",
+    "  * the QR method will be discussed in chapter 7 in connection with eigenvalue systems. $O(4/3n^3)$ flops.\n",
+    "\n",
+    "## LU Decomposition\n",
+    "\n",
+    "The LU decomposition method means that we can rewrite\n",
+    "this matrix as the product of two matrices $\\mathbf{L}$ and $\\mathbf{U}$\n",
+    "where"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "$$\n",
+    "\\begin{bmatrix}\n",
+    "                          a_{11} & a_{12} & a_{13} & a_{14} \\\\\n",
+    "                          a_{21} & a_{22} & a_{23} & a_{24} \\\\\n",
+    "                          a_{31} & a_{32} & a_{33} & a_{34} \\\\\n",
+    "                          a_{41} & a_{42} & a_{43} & a_{44}\n",
+    "                      \\end{bmatrix}\n",
+    "                      = \\begin{bmatrix}\n",
+    "                              1  & 0      & 0      & 0 \\\\\n",
+    "                          l_{21} & 1      & 0      & 0 \\\\\n",
+    "                          l_{31} & l_{32} & 1      & 0 \\\\\n",
+    "                          l_{41} & l_{42} & l_{43} & 1\n",
+    "                      \\end{bmatrix}\n",
+    "                        \\begin{bmatrix}\n",
+    "                          u_{11} & u_{12} & u_{13} & u_{14} \\\\\n",
+    "                               0 & u_{22} & u_{23} & u_{24} \\\\\n",
+    "                               0 & 0      & u_{33} & u_{34} \\\\\n",
+    "                               0 & 0      &  0     & u_{44}\n",
+    "             \\end{bmatrix}.\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## LU Decomposition\n",
+    "\n",
+    "LU decomposition forms the backbone of other algorithms in linear algebra, such as the\n",
+    "solution of linear equations given by"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "$$\n",
+    "a_{11}x_1 +a_{12}x_2 +a_{13}x_3 + a_{14}x_4=w_1 \\nonumber\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "$$\n",
+    "a_{21}x_1 + a_{22}x_2 + a_{23}x_3 + a_{24}x_4=w_2 \\nonumber\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "$$\n",
+    "a_{31}x_1 + a_{32}x_2 + a_{33}x_3 + a_{34}x_4=w_3 \\nonumber\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "$$\n",
+    "a_{41}x_1 + a_{42}x_2 + a_{43}x_3 + a_{44}x_4=w_4.  \\nonumber\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "The above set of equations is conveniently solved by using LU decomposition as an intermediate step.\n",
+    "\n",
+    "The matrix $\\mathbf{A}\\in \\mathbb{R}^{n\\times n}$ has an LU factorization if the determinant\n",
+    "is different from zero. If the LU factorization exists and $\\mathbf{A}$ is non-singular, then the LU factorization\n",
+    "is unique and the determinant is given by"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "$$\n",
+    "det\\{\\mathbf{A}\\}=det\\{\\mathbf{LU}\\}= det\\{\\mathbf{L}\\}det\\{\\mathbf{U}\\}=u_{11}u_{22}\\dots u_{nn}.\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## LU Decomposition, why?\n",
+    "\n",
+    "There are at least three main advantages with LU decomposition compared with standard Gaussian elimination:\n",
+    "\n",
+    "  * It is straightforward to compute the determinant of a matrix\n",
+    "\n",
+    "  * If we have to solve sets of linear equations with the same matrix but with different vectors $\\mathbf{y}$, the number of FLOPS is of the order $n^3$.\n",
+    "\n",
+    "  * The inverse is such an operation \n",
+    "\n",
+    "## LU Decomposition, linear equations\n",
+    "\n",
+    "With the LU decomposition it is rather\n",
+    "simple to solve a system of linear equations"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "$$\n",
+    "a_{11}x_1 +a_{12}x_2 +a_{13}x_3 + a_{14}x_4=w_1 \\nonumber\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "$$\n",
+    "a_{21}x_1 + a_{22}x_2 + a_{23}x_3 + a_{24}x_4=w_2 \\nonumber\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "$$\n",
+    "a_{31}x_1 + a_{32}x_2 + a_{33}x_3 + a_{34}x_4=w_3 \\nonumber\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "$$\n",
+    "a_{41}x_1 + a_{42}x_2 + a_{43}x_3 + a_{44}x_4=w_4. \\nonumber\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "This can be written in matrix form as"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "$$\n",
+    "\\mathbf{Ax}=\\mathbf{w}.\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "where $\\mathbf{A}$ and $\\mathbf{w}$ are known and we have to solve for\n",
+    "$\\mathbf{x}$. Using the LU dcomposition we write"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "$$\n",
+    "\\mathbf{A} \\mathbf{x} \\equiv \\mathbf{L} \\mathbf{U} \\mathbf{x} =\\mathbf{w}.\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## LU Decomposition, linear equations\n",
+    "\n",
+    "The previous equation can be calculated in two steps"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "$$\n",
+    "\\mathbf{L} \\mathbf{y} = \\mathbf{w};\\qquad \\mathbf{Ux}=\\mathbf{y}.\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "To show that this is correct we use to the LU decomposition\n",
+    "to rewrite our system of linear equations as"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "$$\n",
+    "\\mathbf{LUx}=\\mathbf{w},\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "and since the determinant of $\\mathbf{L}$ is equal to 1 (by construction\n",
+    "since the diagonals of $\\mathbf{L}$ equal 1) we can use the inverse of\n",
+    "$\\mathbf{L}$ to obtain"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "$$\n",
+    "\\mathbf{Ux}=\\mathbf{L^{-1}w}=\\mathbf{y},\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "which yields the intermediate step"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "$$\n",
+    "\\mathbf{L^{-1}w}=\\mathbf{y}\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "and as soon as we have $\\mathbf{y}$ we can obtain $\\mathbf{x}$\n",
+    "through $\\mathbf{Ux}=\\mathbf{y}$.\n",
+    "\n",
+    "## LU Decomposition, why?\n",
+    "\n",
+    "For our four-dimentional example this takes the form"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "$$\n",
+    "y_1=w_1 \\nonumber\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "$$\n",
+    "l_{21}y_1 + y_2=w_2\\nonumber\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "$$\n",
+    "l_{31}y_1 + l_{32}y_2 + y_3 =w_3\\nonumber\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "$$\n",
+    "l_{41}y_1 + l_{42}y_2 + l_{43}y_3 + y_4=w_4. \\nonumber\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "and"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "$$\n",
+    "u_{11}x_1 +u_{12}x_2 +u_{13}x_3 + u_{14}x_4=y_1 \\nonumber\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "$$\n",
+    "u_{22}x_2 + u_{23}x_3 + u_{24}x_4=y_2\\nonumber\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "$$\n",
+    "u_{33}x_3 + u_{34}x_4=y_3\\nonumber\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "$$\n",
+    "u_{44}x_4=y_4  \\nonumber\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "This example shows the basis for the algorithm\n",
+    "needed to solve the set of $n$ linear equations.\n",
+    "\n",
+    "## LU Decomposition, linear equations\n",
+    "\n",
+    "The algorithm goes as follows\n",
+    "\n",
+    "  * Set up the matrix $\\bf A$ and the vector $\\bf w$ with their correct dimensions. This determines the dimensionality of the unknown vector $\\bf x$.\n",
+    "\n",
+    "  * Then LU decompose the matrix $\\bf A$ through a call to the function `ludcmp(double a, int n, int indx, double &d)`. This functions returns the LU decomposed matrix $\\bf A$, its determinant and the vector indx which keeps track of the number of interchanges of rows. If the determinant is zero, the solution is malconditioned.\n",
+    "\n",
+    "  * Thereafter you call the function  `lubksb(double a, int n, int indx, double w)` which uses the LU decomposed matrix $\\bf A$ and the vector $\\bf w$ and returns $\\bf x$ in the same place as $\\bf w$. Upon exit the original content in $\\bf w$ is destroyed. If you wish to keep this information, you should make a backup of it in your calling function.\n",
+    "\n",
+    "## LU Decomposition, the inverse of a matrix\n",
+    "\n",
+    "If the inverse exists then"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "$$\n",
+    "\\mathbf{A}^{-1}\\mathbf{A}=\\mathbf{I},\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "the identity matrix. With an LU decomposed matrix we can rewrite the last equation as"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "$$\n",
+    "\\mathbf{LU}\\mathbf{A}^{-1}=\\mathbf{I}.\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## LU Decomposition, the inverse of a matrix\n",
+    "\n",
+    "If we assume that the first column (that is column 1) of the inverse matrix\n",
+    "can be written as a vector with unknown entries"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "$$\n",
+    "\\mathbf{A}_1^{-1}= \\begin{bmatrix}\n",
+    "                              a_{11}^{-1} \\\\\n",
+    "                              a_{21}^{-1} \\\\\n",
+    "                              \\dots \\\\\n",
+    "                              a_{n1}^{-1} \\\\\n",
+    "                    \\end{bmatrix},\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "then we have a linear set of equations"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "$$\n",
+    "\\mathbf{LU}\\begin{bmatrix}\n",
+    "                              a_{11}^{-1} \\\\\n",
+    "                              a_{21}^{-1} \\\\\n",
+    "                              \\dots \\\\\n",
+    "                              a_{n1}^{-1} \\\\\n",
+    "                    \\end{bmatrix} =\\begin{bmatrix}\n",
+    "                               1 \\\\\n",
+    "                              0 \\\\\n",
+    "                              \\dots \\\\\n",
+    "                              0 \\\\\n",
+    "                    \\end{bmatrix}.\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## LU Decomposition, the inverse\n",
+    "\n",
+    "In a similar way we can compute the unknow entries of the second column,"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "$$\n",
+    "\\mathbf{LU}\\begin{bmatrix}\n",
+    "                              a_{12}^{-1} \\\\\n",
+    "                              a_{22}^{-1} \\\\\n",
+    "                              \\dots \\\\\n",
+    "                              a_{n2}^{-1} \\\\\n",
+    "                    \\end{bmatrix}=\\begin{bmatrix}\n",
+    "                                0 \\\\\n",
+    "                              1 \\\\\n",
+    "                              \\dots \\\\\n",
+    "                              0 \\\\\n",
+    "                    \\end{bmatrix},\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "and continue till we have solved all $n$ sets of linear equations.\n",
+    "\n",
+    "\n",
+    "## [Using Armadillo to perform an LU decomposition](https://github.com/CompPhysics/ComputationalPhysicsMSU/blob/master/doc/Programs/CppQtCodesLectures/MatrixTest/main.cpp)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "        #include \n",
+    "        #include \"armadillo\"\n",
+    "        using namespace arma;\n",
+    "        using namespace std;\n",
+    "        \n",
+    "        int main()\n",
+    "          {\n",
+    "           mat A = randu(5,5);\n",
+    "           vec b = randu(5);\n",
+    "        \n",
+    "          A.print(\"A =\");\n",
+    "          b.print(\"b=\");\n",
+    "          // solve Ax = b\n",
+    "          vec x = solve(A,b);\n",
+    "          // print x\n",
+    "          x.print(\"x=\");\n",
+    "          // find LU decomp of A, if needed, P is the permutation matrix\n",
+    "          mat L, U;\n",
+    "          lu(L,U,A);\n",
+    "          // print l\n",
+    "          L.print(\" L= \");\n",
+    "          // print U\n",
+    "          U.print(\" U= \");\n",
+    "          //Check that A = LU\n",
+    "          (A-L*U).print(\"Test of LU decomposition\");\n",
+    "            return 0;\n",
+    "          }\n"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.6.5"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}
diff --git a/doc/pub/LogReg/ipynb/.ipynb_checkpoints/LogReg-checkpoint.ipynb b/doc/pub/LogReg/ipynb/.ipynb_checkpoints/LogReg-checkpoint.ipynb
new file mode 100644
index 000000000..baaa9264e
--- /dev/null
+++ b/doc/pub/LogReg/ipynb/.ipynb_checkpoints/LogReg-checkpoint.ipynb
@@ -0,0 +1,502 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "\n",
+    "# Data Analysis and Machine Learning: Logistic Regression\n",
+    "\n",
+    "  \n",
+    "**Morten Hjorth-Jensen**, Department of Physics, University of Oslo and Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University\n",
+    "\n",
+    "Date: **Sep 26, 2018**\n",
+    "\n",
+    "Copyright 1999-2018, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license\n",
+    "\n",
+    "\n",
+    "\n",
+    "\n",
+    "\n",
+    "## Logistic Regression\n",
+    "\n",
+    "In linear regression our main interest was centered on learning the\n",
+    "coefficients of a functional fit (say a polynomial) in order to be\n",
+    "able to predict the response of a continuous variable on some unseen\n",
+    "data. The fit to the continuous variable $y_i$ is based on some\n",
+    "independent variables $\\hat{x}_i$. Linear regression resulted in\n",
+    "analytical expressions (in terms of matrices to invert) for several\n",
+    "quantities, ranging from the variance and thereby the confidence\n",
+    "intervals of the parameters $\\hat{\\beta}$ to the mean squared\n",
+    "error. If we can invert the product of the design matrices, linear\n",
+    "regression gives then a simple recipe for fitting our data.\n",
+    "\n",
+    "\n",
+    "Classification problems, however, are concerned with outcomes taking\n",
+    "the form of discrete variables (i.e. categories). We may for example,\n",
+    "on the basis of DNA sequencing for a number of patients, like to find\n",
+    "out which mutations are important for a certain disease; or based on\n",
+    "scans of various patients' brains, figure out if there is a tumor or\n",
+    "not; or given a specific physical system, we'd like to identify its\n",
+    "state, say whether it is an ordered or disordered system (typical\n",
+    "situation in solid state physics); or classify the status of a\n",
+    "patient, whether she/he has a stroke or not and many other similar\n",
+    "situations.\n",
+    "\n",
+    "The most common situation we encounter when we apply logistic\n",
+    "regression is that of two possible outcomes, normally denoted as a\n",
+    "binary outcome, true or false, positive or negative, success or\n",
+    "failure etc.\n",
+    "\n",
+    "## Optimization and Deep learning\n",
+    "\n",
+    "Logistic regression will also serve as our stepping stone towards neural\n",
+    "network algorithms and supervised deep learning. For logistic\n",
+    "learning, the minimization of the cost function leads to a non-linear\n",
+    "equation in the parameters $\\hat{\\beta}$. The optmization of the problem calls therefore for minimization algorithms. This forms the bottle neck of all machine learning algorithms, namely how to find reliable minima of a multi-variable function. This leads us to the family of gradient descent methods. The latter are the working horses of basically all modern machine learning algorithms. \n",
+    "\n",
+    "We note also that many of the topics discussed here \n",
+    "regression are also commonly used in modern supervised Deep Learning\n",
+    "models, as we will see later.\n",
+    "\n",
+    "\n",
+    "\n",
+    "## Basics\n",
+    "\n",
+    "We consider the case where the dependent variables, also called the\n",
+    "responses or the outcomes, $y_i$ are discrete and only take values\n",
+    "from $k=0,\\dots,K-1$ (i.e. $K$ classes).\n",
+    "\n",
+    "The goal is to predict the\n",
+    "output classes from the design matrix $\\hat{X}\\in\\mathbb{R}^{n\\times p}$\n",
+    "made of $n$ samples, each of which carries $p$ features or predictors. The\n",
+    "primary goal is to identify the classes to which new unseen samples\n",
+    "belong.\n",
+    "\n",
+    "Let us specialize to the case of two classes only, with outputs $y_i=0$ and $y_i=1$. Our outcomes could represent the status of a credit card user who could default or not on her/his credit card debt. That is"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "$$\n",
+    "y_i = \\begin{bmatrix} 0 & \\mathrm{no}\\\\  1 & \\mathrm{yes} \\end{bmatrix}.\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Linear classifier\n",
+    "\n",
+    "Before moving to the logistic model, let us try to use our linear regression model to classify these two outcomes. We could for example fit a linear model to the default case if $y_i > 0.5$ and the no default case $y_i \\leq 0.5$. \n",
+    "\n",
+    "We would then have our \n",
+    "weighted linear combination, namely"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "\n",
+    "
\n",
+    "\n",
+    "$$\n",
+    "\\begin{equation}\n",
+    "\\hat{y} = \\hat{X}^T\\hat{\\beta} +  \\hat{\\epsilon},\n",
+    "\\label{_auto1} \\tag{1}\n",
+    "\\end{equation}\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "where $\\hat{y}$ is a vector representing the possible outcomes, $\\hat{X}$ is our\n",
+    "$n\\times p$ design matrix and $\\hat{\\beta}$ represents our estimators/predictors.\n",
+    "\n",
+    "## Some selected properties\n",
+    "\n",
+    "The main problem with our  function is that it \n",
+    "takes values on the entire real axis. In the case of\n",
+    "logistic regression, however, the labels $y_i$ are discrete\n",
+    "variables. \n",
+    "\n",
+    "One simple way to get a discrete output is to have sign\n",
+    "functions that map the output of a linear regressor to values $\\{0,1\\}$,\n",
+    "$f(s_i)=sign(s_i)=1$ if $s_i\\ge 0$ and 0 if otherwise. \n",
+    "We will encounter this model in our first demonstration of neural networks. Historically it is called the \"perceptron\" model in the machine learning\n",
+    "literature. This model is extremely simple. However, in many cases it is more\n",
+    "favorable to use a ``soft\" classifier that outputs\n",
+    "the probability of a given category. This leads us to the logistic function.\n",
+    "\n",
+    "The code for plotting the perceptron can be seen here. This si nothing but the standard [Heaviside step function](https://en.wikipedia.org/wiki/Heaviside_step_function)."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## The logistic function\n",
+    "\n",
+    "The perceptron is an example of a ``hard classification\" model. We\n",
+    "will encounter this model when we discuss neural networks as\n",
+    "well. Each datapoint is deterministically assigned to a category (i.e\n",
+    "$y_i=0$ or $y_i=1$). In many cases, it is favorable to have a \"soft\"\n",
+    "classifier that outputs the probability of a given category rather\n",
+    "than a single value. For example, given $x_i$, the classifier\n",
+    "outputs the probability of being in a category $k$.  Logistic regression\n",
+    "is the most common example of a so-called soft classifier. In logistic\n",
+    "regression, the probability that a data point $x_i$\n",
+    "belongs to a category $y_i=\\{0,1\\}$ is given by the so-called logit function (or Sigmoid) which is meant to represent the likelihood for a given event,"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "$$\n",
+    "p(t) = \\frac{1}{1+\\mathrm \\exp{-t}}=\\frac{\\exp{t}}{1+\\mathrm \\exp{t}}.\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Note that $1-p(t)= p(-t)$.\n",
+    "The following code plots the logistic function."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Two parameters\n",
+    "\n",
+    "We assume now that we have two classes with $y_i$ either $0$ or $1$. Furthermore we assume also that we have only two parameters $\\beta$ in our fitting of the Sigmoid function, that is we define probabilities"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "$$\n",
+    "\\begin{align*}\n",
+    "p(y_i=1|x_i,\\hat{\\beta}) &= \\frac{\\exp{(\\beta_0+\\beta_1x_i)}}{1+\\exp{(\\beta_0+\\beta_1x_i)}},\\nonumber\\\\\n",
+    "p(y_i=0|x_i,\\hat{\\beta}) &= 1 - p(y_i=1|x_i,\\hat{\\beta}),\n",
+    "\\end{align*}\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "where $\\hat{\\beta}$ are the weights we wish to extract from data, in our case $\\beta_0$ and $\\beta_1$. \n",
+    "\n",
+    "Note that we used"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "$$\n",
+    "p(y_i=0\\vert x_i, \\hat{\\beta}) = 1-p(y_i=1\\vert x_i, \\hat{\\beta}).\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "\n",
+    "## Maximum likelihood\n",
+    "\n",
+    "In order to define the total likelihood for all possible outcomes from a  \n",
+    "dataset $\\mathcal{D}=\\{(y_i,x_i)\\}$, with the binary labels\n",
+    "$y_i\\in\\{0,1\\}$ and where the data points are drawn independently, we use the so-called [Maximum Likelihood Estimation](https://en.wikipedia.org/wiki/Maximum_likelihood_estimation) (MLE) principle. \n",
+    "We aim thus at maximizing \n",
+    "the probability of seeing the observed data. We can then approximate the \n",
+    "likelihood in terms of the product of the individual probabilities of a specific outcome $y_i$, that is"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "$$\n",
+    "\\begin{align*}\n",
+    "P(\\mathcal{D}|\\hat{\\beta})& = \\prod_{i=1}^n \\left[p(y_i=1|x_i,\\hat{\\beta})\\right]^{y_i}\\left[1-p(y_i=1|x_i,\\hat{\\beta}))\\right]^{1-y_i}\\nonumber \\\\\n",
+    "\\end{align*}\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "from which we obtain the log-likelihood and our **cost/loss** function"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "$$\n",
+    "\\mathcal{C}(\\hat{\\beta}) = \\sum_{i=1}^n \\left( y_i\\log{p(y_i=1|x_i,\\hat{\\beta})} + (1-y_i)\\log\\left[1-p(y_i=1|x_i,\\hat{\\beta}))\\right]\\right).\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## The cost function rewritten\n",
+    "\n",
+    "Reordering the logarithms, we can rewrite the **cost/loss** function as"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "$$\n",
+    "\\mathcal{C}(\\hat{\\beta}) = \\sum_{i=1}^n  \\left(y_i(\\beta_0+\\beta_1x_i) -\\log{(1+\\exp{(\\beta_0+\\beta_1x_i)})}\\right).\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "The maximum likelihood estimator is defined as the set of parameters that maximize the log-likelihood where we maximize with respect to $\\beta$.\n",
+    "Since the cost (error) function is just the negative log-likelihood, for logistic regression we have that"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "$$\n",
+    "\\mathcal{C}(\\hat{\\beta})=-\\sum_{i=1}^n  \\left(y_i(\\beta_0+\\beta_1x_i) -\\log{(1+\\exp{(\\beta_0+\\beta_1x_i)})}\\right).\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "This equation is known in statistics as the **cross entropy**. Finally, we note that just as in linear regression, \n",
+    "in practice we often supplement the cross-entropy with additional regularization terms, usually $L_1$ and $L_2$ regularization as we did for Ridge and Lasso regression.\n",
+    "\n",
+    "## Minimizing the cross entropy\n",
+    "\n",
+    "The cross entropy is a convex function of the weights $\\hat{\\beta}$ and,\n",
+    "therefore, any local minimizer is a global minimizer. \n",
+    "\n",
+    "\n",
+    "Minimizing this\n",
+    "cost function with respect to the two parameters $\\beta_0$ and $\\beta_1$ we obtain"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "$$\n",
+    "\\frac{\\partial \\mathcal{C}(\\hat{\\beta})}{\\partial \\beta_0} = -\\sum_{i=1}^n  \\left(y_i -\\frac{\\exp{(\\beta_0+\\beta_1x_i)}}{1+\\exp{(\\beta_0+\\beta_1x_i)}}\\right),\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "and"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "$$\n",
+    "\\frac{\\partial \\mathcal{C}(\\hat{\\beta})}{\\partial \\beta_1} = -\\sum_{i=1}^n  \\left(y_ix_i -x_i\\frac{\\exp{(\\beta_0+\\beta_1x_i)}}{1+\\exp{(\\beta_0+\\beta_1x_i)}}\\right).\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## A more compact expression\n",
+    "\n",
+    "Let us now define a vector $\\hat{y}$ with $n$ elements $y_i$, an\n",
+    "$n\\times p$ matrix $\\hat{X}$ which contains the $x_i$ values and a\n",
+    "vector $\\hat{p}$ of fitted probabilities $p(y_i\\vert x_i,\\hat{\\beta})$. We can rewrite in a more compact form the first\n",
+    "derivative of cost function as"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "$$\n",
+    "\\frac{\\partial \\mathcal{C}(\\hat{\\beta})}{\\partial \\hat{\\beta}} = -\\hat{X}^T\\left(\\hat{y}-\\hat{p}\\right).\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "If we in addition define a diagonal matrix $\\hat{W}$ with elements \n",
+    "$p(y_i\\vert x_i,\\hat{\\beta})(1-p(y_i\\vert x_i,\\hat{\\beta})$, we can obtain a compact expression of the second derivative as"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "$$\n",
+    "\\frac{\\partial^2 \\mathcal{C}(\\hat{\\beta})}{\\partial \\hat{\\beta}\\partial \\hat{\\beta}^T} = \\hat{X}^T\\hat{W}\\hat{X}.\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Extending to more predictors\n",
+    "\n",
+    "## Including more classes\n",
+    "\n",
+    "## Optimizing the cost function\n",
+    "\n",
+    "Newton's method and gradient descent methods\n",
+    "\n",
+    "\n",
+    "\n",
+    "## A **scikit-learn** example"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "%matplotlib inline\n",
+    "\n",
+    "import numpy as np\n",
+    "import matplotlib.pyplot as plt\n",
+    "from sklearn import datasets\n",
+    "iris = datasets.load_iris()\n",
+    "list(iris.keys())\n",
+    "['data', 'target_names', 'feature_names', 'target', 'DESCR']\n",
+    "X = iris[\"data\"][:, 3:] # petal width\n",
+    "y = (iris[\"target\"] == 2).astype(np.int) # 1 if Iris-Virginica, else 0\n",
+    "\n",
+    "from sklearn.linear_model import LogisticRegression\n",
+    "log_reg = LogisticRegression()\n",
+    "log_reg.fit(X, y)\n",
+    "\n",
+    "X_new = np.linspace(0, 3, 1000).reshape(-1, 1)\n",
+    "y_proba = log_reg.predict_proba(X_new)\n",
+    "plt.plot(X_new, y_proba[:, 1], \"g-\", label=\"Iris-Virginica\")\n",
+    "plt.plot(X_new, y_proba[:, 0], \"b--\", label=\"Not Iris-Virginica\")\n",
+    "plt.show()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## A simple classification problem"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import numpy as np\n",
+    "from sklearn import datasets, linear_model\n",
+    "import matplotlib.pyplot as plt\n",
+    "\n",
+    "\n",
+    "def generate_data():\n",
+    "    np.random.seed(0)\n",
+    "    X, y = datasets.make_moons(200, noise=0.20)\n",
+    "    return X, y\n",
+    "\n",
+    "\n",
+    "def visualize(X, y, clf):\n",
+    "    # plt.scatter(X[:, 0], X[:, 1], s=40, c=y, cmap=plt.cm.Spectral)\n",
+    "    # plt.show()\n",
+    "    plot_decision_boundary(lambda x: clf.predict(x), X, y)\n",
+    "    plt.title(\"Logistic Regression\")\n",
+    "\n",
+    "\n",
+    "def plot_decision_boundary(pred_func, X, y):\n",
+    "    # Set min and max values and give it some padding\n",
+    "    x_min, x_max = X[:, 0].min() - .5, X[:, 0].max() + .5\n",
+    "    y_min, y_max = X[:, 1].min() - .5, X[:, 1].max() + .5\n",
+    "    h = 0.01\n",
+    "    # Generate a grid of points with distance h between them\n",
+    "    xx, yy = np.meshgrid(np.arange(x_min, x_max, h), np.arange(y_min, y_max, h))\n",
+    "    # Predict the function value for the whole gid\n",
+    "    Z = pred_func(np.c_[xx.ravel(), yy.ravel()])\n",
+    "    Z = Z.reshape(xx.shape)\n",
+    "    # Plot the contour and training examples\n",
+    "    plt.contourf(xx, yy, Z, cmap=plt.cm.Spectral)\n",
+    "    plt.scatter(X[:, 0], X[:, 1], c=y, cmap=plt.cm.Spectral)\n",
+    "    plt.show()\n",
+    "\n",
+    "\n",
+    "def classify(X, y):\n",
+    "    clf = linear_model.LogisticRegressionCV()\n",
+    "    clf.fit(X, y)\n",
+    "    return clf\n",
+    "\n",
+    "\n",
+    "def main():\n",
+    "    X, y = generate_data()\n",
+    "    # visualize(X, y)\n",
+    "    clf = classify(X, y)\n",
+    "    visualize(X, y, clf)\n",
+    "\n",
+    "\n",
+    "if __name__ == \"__main__\":\n",
+    "    main()"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.7.0"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}
diff --git a/doc/pub/NeuralNet/html/reveal.js/css/theme/template/mixins.scss b/doc/pub/NeuralNet/html/reveal.js/css/theme/template/mixins.scss
new file mode 100644
index 000000000..e0c560692
--- /dev/null
+++ b/doc/pub/NeuralNet/html/reveal.js/css/theme/template/mixins.scss
@@ -0,0 +1,29 @@
+@mixin vertical-gradient( $top, $bottom ) {
+	background: $top;
+	background: -moz-linear-gradient( top, $top 0%, $bottom 100% );
+	background: -webkit-gradient( linear, left top, left bottom, color-stop(0%,$top), color-stop(100%,$bottom) );
+	background: -webkit-linear-gradient( top, $top 0%, $bottom 100% );
+	background: -o-linear-gradient( top, $top 0%, $bottom 100% );
+	background: -ms-linear-gradient( top, $top 0%, $bottom 100% );
+	background: linear-gradient( top, $top 0%, $bottom 100% );
+}
+
+@mixin horizontal-gradient( $top, $bottom ) {
+	background: $top;
+	background: -moz-linear-gradient( left, $top 0%, $bottom 100% );
+	background: -webkit-gradient( linear, left top, right top, color-stop(0%,$top), color-stop(100%,$bottom) );
+	background: -webkit-linear-gradient( left, $top 0%, $bottom 100% );
+	background: -o-linear-gradient( left, $top 0%, $bottom 100% );
+	background: -ms-linear-gradient( left, $top 0%, $bottom 100% );
+	background: linear-gradient( left, $top 0%, $bottom 100% );
+}
+
+@mixin radial-gradient( $outer, $inner, $type: circle ) {
+	background: $outer;
+	background: -moz-radial-gradient( center, $type cover,  $inner 0%, $outer 100% );
+	background: -webkit-gradient( radial, center center, 0px, center center, 100%, color-stop(0%,$inner), color-stop(100%,$outer) );
+	background: -webkit-radial-gradient( center, $type cover,  $inner 0%, $outer 100% );
+	background: -o-radial-gradient( center, $type cover,  $inner 0%, $outer 100% );
+	background: -ms-radial-gradient( center, $type cover,  $inner 0%, $outer 100% );
+	background: radial-gradient( center, $type cover,  $inner 0%, $outer 100% );
+}
\ No newline at end of file
diff --git a/doc/pub/NeuralNet/html/reveal.js/css/theme/template/settings.scss b/doc/pub/NeuralNet/html/reveal.js/css/theme/template/settings.scss
new file mode 100644
index 000000000..e706e19c5
--- /dev/null
+++ b/doc/pub/NeuralNet/html/reveal.js/css/theme/template/settings.scss
@@ -0,0 +1,34 @@
+// Base settings for all themes that can optionally be
+// overridden by the super-theme
+
+// Background of the presentation
+$backgroundColor: #2b2b2b;
+
+// Primary/body text
+$mainFont: 'Lato', sans-serif;
+$mainFontSize: 30px;              /* changed (by hpl) from 36px; */
+$mainColor: #eee;
+
+// Headings
+$headingMargin: 0 0 20px 0;
+$headingFont: 'League Gothic', Impact, sans-serif;
+$headingColor: #eee;
+$headingLineHeight: 0.9em;
+$headingLetterSpacing: 0.02em;
+$headingTextTransform: none;      /* changed (by hpl) from uppercase; */
+$headingTextShadow: 0px 0px 6px rgba(0,0,0,0.2);
+$heading1TextShadow: $headingTextShadow;
+
+// Links and actions
+$linkColor: #13DAEC;
+$linkColorHover: lighten( $linkColor, 20% );
+
+// Text selection
+$selectionBackgroundColor: #FF5E99;
+$selectionColor: #fff;
+
+// Generates the presentation background, can be overridden
+// to return a background image or gradient
+@mixin bodyBackground() {
+	background: $backgroundColor;
+}
\ No newline at end of file
diff --git a/doc/pub/NeuralNet/html/reveal.js/css/theme/template/theme.scss b/doc/pub/NeuralNet/html/reveal.js/css/theme/template/theme.scss
new file mode 100644
index 000000000..b1dbc2736
--- /dev/null
+++ b/doc/pub/NeuralNet/html/reveal.js/css/theme/template/theme.scss
@@ -0,0 +1,171 @@
+// Base theme template for reveal.js
+
+/*********************************************
+ * GLOBAL STYLES
+ *********************************************/
+
+body {
+	@include bodyBackground();
+	background-color: $backgroundColor;
+}
+
+.reveal {
+	font-family: $mainFont;
+	font-size: $mainFontSize;
+	font-weight: normal;
+	letter-spacing: -0.02em;
+	color: $mainColor;
+}
+
+::selection {
+	color: $selectionColor;
+	background: $selectionBackgroundColor;
+	text-shadow: none;
+}
+
+/*********************************************
+ * HEADERS
+ *********************************************/
+
+.reveal h1,
+.reveal h2,
+.reveal h3,
+.reveal h4,
+.reveal h5,
+.reveal h6 {
+	margin: $headingMargin;
+	color: $headingColor;
+
+	font-family: $headingFont;
+	line-height: $headingLineHeight;
+	letter-spacing: $headingLetterSpacing;
+
+	text-transform: $headingTextTransform;
+	text-shadow: $headingTextShadow;
+}
+
+.reveal h1 {
+        line-height: 1.2em;    /* added by hpl */
+	text-shadow: $heading1TextShadow;
+}
+
+
+/*********************************************
+ * LINKS
+ *********************************************/
+
+.reveal a:not(.image) {
+	color: $linkColor;
+	text-decoration: none;
+
+	-webkit-transition: color .15s ease;
+	   -moz-transition: color .15s ease;
+	    -ms-transition: color .15s ease;
+	     -o-transition: color .15s ease;
+	        transition: color .15s ease;
+}
+	.reveal a:not(.image):hover {
+		color: $linkColorHover;
+
+		text-shadow: none;
+		border: none;
+	}
+
+.reveal .roll span:after {
+	color: #fff;
+	background: darken( $linkColor, 15% );
+}
+
+
+/*********************************************
+ * IMAGES
+ *********************************************/
+
+.reveal section img {
+	margin: 15px 0px;
+	background: rgba(255,255,255,0.12);
+	border: 4px solid $mainColor;
+
+	box-shadow: 0 0 10px rgba(0, 0, 0, 0.15);
+
+	-webkit-transition: all .2s linear;
+	   -moz-transition: all .2s linear;
+	    -ms-transition: all .2s linear;
+	     -o-transition: all .2s linear;
+	        transition: all .2s linear;
+}
+
+	.reveal a:hover img {
+		background: rgba(255,255,255,0.2);
+		border-color: $linkColor;
+
+		box-shadow: 0 0 20px rgba(0, 0, 0, 0.55);
+	}
+
+
+/*********************************************
+ * NAVIGATION CONTROLS
+ *********************************************/
+
+.reveal .controls div.navigate-left,
+.reveal .controls div.navigate-left.enabled {
+	border-right-color: $linkColor;
+}
+
+.reveal .controls div.navigate-right,
+.reveal .controls div.navigate-right.enabled {
+	border-left-color: $linkColor;
+}
+
+.reveal .controls div.navigate-up,
+.reveal .controls div.navigate-up.enabled {
+	border-bottom-color: $linkColor;
+}
+
+.reveal .controls div.navigate-down,
+.reveal .controls div.navigate-down.enabled {
+	border-top-color: $linkColor;
+}
+
+.reveal .controls div.navigate-left.enabled:hover {
+	border-right-color: $linkColorHover;
+}
+
+.reveal .controls div.navigate-right.enabled:hover {
+	border-left-color: $linkColorHover;
+}
+
+.reveal .controls div.navigate-up.enabled:hover {
+	border-bottom-color: $linkColorHover;
+}
+
+.reveal .controls div.navigate-down.enabled:hover {
+	border-top-color: $linkColorHover;
+}
+
+
+/*********************************************
+ * PROGRESS BAR
+ *********************************************/
+
+.reveal .progress {
+	background: rgba(0,0,0,0.2);
+}
+	.reveal .progress span {
+		background: $linkColor;
+
+		-webkit-transition: width 800ms cubic-bezier(0.260, 0.860, 0.440, 0.985);
+		   -moz-transition: width 800ms cubic-bezier(0.260, 0.860, 0.440, 0.985);
+		    -ms-transition: width 800ms cubic-bezier(0.260, 0.860, 0.440, 0.985);
+		     -o-transition: width 800ms cubic-bezier(0.260, 0.860, 0.440, 0.985);
+		        transition: width 800ms cubic-bezier(0.260, 0.860, 0.440, 0.985);
+	}
+
+/*********************************************
+ * SLIDE NUMBER
+ *********************************************/
+.reveal .slide-number {
+  color: $linkColor;
+}
+
+
diff --git a/doc/pub/NeuralNet/html/reveal.js/demo.html b/doc/pub/NeuralNet/html/reveal.js/demo.html
new file mode 100644
index 000000000..505bb1882
--- /dev/null
+++ b/doc/pub/NeuralNet/html/reveal.js/demo.html
@@ -0,0 +1,410 @@
+
+
+
+	
+		
+
+		reveal.js – The HTML Presentation Framework
+
+		
+		
+
+		
+		
+
+		
+
+		
+		
+
+		
+		
+
+		
+		
+
+		
+	
+
+	
+
+		

+
+			
+			

+				

+					
Reveal.js

+					
The HTML Presentation Framework

+					

+						Created by Hakim El Hattab and contributors
+					

+				

+
+				

+					
Hello There

+					

+						reveal.js enables you to create beautiful interactive slide decks using HTML. This presentation will show you examples of what it can do.
+					

+				

+
+				
+				

+					

+						
Vertical Slides

+						
Slides can be nested inside of each other.

+						
Use the Space key to navigate through all slides.

+						

+						
+							Down arrow
+						
+					

+					

+						
Basement Level 1

+						
Nested slides are useful for adding additional detail underneath a high level horizontal slide.

+					

+					

+						
Basement Level 2

+						
That's it, time to go back up.

+						

+						
+							Up arrow
+						
+					

+				

+
+				

+					
Slides

+					

+						Not a coder? Not a problem. There's a fully-featured visual editor for authoring these, try it out at https://slides.com.
+					

+				

+
+				

+					
Point of View

+					

+						Press ESC to enter the slide overview.
+					

+					

+						Hold down alt and click on any element to zoom in on it using zoom.js. Alt + click anywhere to zoom back out.
+					

+				

+
+				

+					
Touch Optimized

+					

+						Presentations look great on touch devices, like mobile phones and tablets. Simply swipe through your slides.
+					

+				

+
+				

+					
+				

+
+				

+					

+						
Fragments

+						
Hit the next arrow...

+						
... to step through ...

+						
... a fragmented slide.

+
+						
+					

+					

+						
Fragment Styles

+						
There's different types of fragments, like:

+						
grow

+						
shrink

+						
fade-out

+						
fade-up (also down, left and right!)

+						
current-visible

+						
Highlight red blue green

+					

+				

+
+				

+					
Transition Styles

+					

+						You can select from different transitions, like: 

+						None -
+						Fade -
+						Slide -
+						Convex -
+						Concave -
+						Zoom
+					

+				

+
+				

+					
Themes

+					

+						reveal.js comes with a few themes built in: 

+						
+						Black (default) -
+						White -
+						League -
+						Sky -
+						Beige -
+						Simple 

+						Serif -
+						Blood -
+						Night -
+						Moon -
+						Solarized
+					

+				

+
+				

+					

+						
Slide Backgrounds

+						

+							Set data-background="#dddddd" on a slide to change the background color. All CSS color formats are supported.
+						

+						
+							Down arrow
+						
+					

+					

+						
Image Backgrounds

+						
<section data-background="image.png">

+					

+					

+						
Tiled Backgrounds

+						
<section data-background="image.png" data-background-repeat="repeat" data-background-size="100px">

+					

+					

+						

+							
Video Backgrounds

+							
<section data-background-video="video.mp4,video.webm">

+						

+					

+					

+						
... and GIFs!

+					

+				

+
+				

+					
Background Transitions

+					

+						Different background transitions are available via the backgroundTransition option. This one's called "zoom".
+					

+					
Reveal.configure({ backgroundTransition: 'zoom' })

+				

+
+				

+					
Background Transitions

+					

+						You can override background transitions per-slide.
+					

+					
<section data-background-transition="zoom">

+				

+
+				

+					
Pretty Code

+					

+function linkify( selector ) {
+  if( supports3DTransforms ) {
+
+    var nodes = document.querySelectorAll( selector );
+
+    for( var i = 0, len = nodes.length; i < len; i++ ) {
+      var node = nodes[i];
+
+      if( !node.className ) {
+        node.className += ' roll';
+      }
+    }
+  }
+}
+					

+					
Code syntax highlighting courtesy of highlight.js.

+				

+
+				

+					
Marvelous List

+					
    
+						
  • No order here
    • 
+						
  • Or here
    • 
+						
  • Or here
    • 
+						
  • Or here
    • 
+					

+				

+
+				

+					
Fantastic Ordered List

+					
    
+						
  1. One is smaller than...
    2. 
+						
  2. Two is smaller than...
    3. 
+						
  3. Three!
    4. 
+					

+				

+
+				

+					
Tabular Tables

+					
+						
+							
+								
+								
+								
+							
+						
+						
+							
+								
+								
+								
+							
+							
+								
+								
+								
+							
+							
+								
+								
+								
+							
+						
+					
ItemValueQuantity
Apples$17
Lemonade$218
Bread$32

+				

+
+				

+					
Clever Quotes

+					

+						These guys come in two forms, inline: The nice thing about standards is that there are so many to choose from and block:
+					

+					

+						“For years there has been a theory that millions of monkeys typing at random on millions of typewriters would
+						reproduce the entire works of Shakespeare. The Internet has proven this theory to be untrue.”
+					

+				

+
+				

+					
Intergalactic Interconnections

+					

+						You can link between slides internally,
+						like this.
+					

+				

+
+				

+					
Speaker View

+					
There's a speaker view. It includes a timer, preview of the upcoming slide as well as your speaker notes.

+					
Press the S key to try it out.

+
+					
+				

+
+				

+					
Export to PDF

+					
Presentations can be exported to PDF, here's an example:

+					
+				

+
+				

+					
Global State

+					

+						Set data-state="something" on a slide and "something"
+						will be added as a class to the document element when the slide is open. This lets you
+						apply broader style changes, like switching the page background.
+					

+				

+
+				

+					
State Events

+					

+						Additionally custom events can be triggered on a per slide basis by binding to the data-state name.
+					

+					

+Reveal.addEventListener( 'customevent', function() {
+	console.log( '"customevent" has fired' );
+} );
+					

+				

+
+				

+					
Take a Moment

+					

+						Press B or . on your keyboard to pause the presentation. This is helpful when you're on stage and want to take distracting slides off the screen.
+					

+				

+
+				

+					
Much more

+					
+				

+
+				

+					
THE END

+					

+						- Try the online editor 

+						- Source code & documentation
+					

+				

+
+			

+
+		

+
+		
+		
+
+		
+
+	
+
diff --git a/doc/pub/NeuralNet/html/reveal.js/plugin/multiplex/package.json b/doc/pub/NeuralNet/html/reveal.js/plugin/multiplex/package.json
new file mode 100644
index 000000000..bbed77a67
--- /dev/null
+++ b/doc/pub/NeuralNet/html/reveal.js/plugin/multiplex/package.json
@@ -0,0 +1,19 @@
+{
+  "name": "reveal-js-multiplex",
+  "version": "1.0.0",
+  "description": "reveal.js multiplex server",
+  "homepage": "http://revealjs.com",
+  "scripts": {
+    "start": "node index.js"
+  },
+  "engines": {
+    "node": "~4.1.1"
+  },
+  "dependencies": {
+    "express": "~4.13.3",
+    "grunt-cli": "~0.1.13",
+    "mustache": "~2.2.1",
+    "socket.io": "~1.3.7"
+  },
+  "license": "MIT"
+}
diff --git a/doc/pub/NeuralNet/html/reveal.js/test/simple.md b/doc/pub/NeuralNet/html/reveal.js/test/simple.md
new file mode 100644
index 000000000..c72a44079
--- /dev/null
+++ b/doc/pub/NeuralNet/html/reveal.js/test/simple.md
@@ -0,0 +1,12 @@
+## Slide 1.1
+
+```js
+var a = 1;
+```
+
+
+## Slide 1.2
+
+
+
+## Slide 2
diff --git a/doc/pub/NeuralNet/html/reveal.js/test/test-markdown-external.html b/doc/pub/NeuralNet/html/reveal.js/test/test-markdown-external.html
new file mode 100644
index 000000000..859d0a199
--- /dev/null
+++ b/doc/pub/NeuralNet/html/reveal.js/test/test-markdown-external.html
@@ -0,0 +1,36 @@
+
+
+
+	
+		
+
+		reveal.js - Test Markdown
+
+		
+		
+	
+
+	
+
+		

+  		

+
+		
+
+		
+		
+		
+		
+		
+		
+
+		
+
+	
+
diff --git a/doc/pub/NeuralNet/html/reveal.js/test/test-markdown-external.js b/doc/pub/NeuralNet/html/reveal.js/test/test-markdown-external.js
new file mode 100644
index 000000000..cab85c6f6
--- /dev/null
+++ b/doc/pub/NeuralNet/html/reveal.js/test/test-markdown-external.js
@@ -0,0 +1,24 @@
+
+
+Reveal.addEventListener( 'ready', function() {
+
+	QUnit.module( 'Markdown' );
+
+	test( 'Vertical separator', function() {
+		strictEqual( document.querySelectorAll( '.reveal .slides>section>section' ).length, 2, 'found two slides' );
+	});
+
+	test( 'Horizontal separator', function() {
+		strictEqual( document.querySelectorAll( '.reveal .slides>section' ).length, 2, 'found two slides' );
+	});
+
+	test( 'Language highlighter', function() {
+		strictEqual( document.querySelectorAll( '.hljs-keyword' ).length, 1, 'got rendered highlight tag.' );
+		strictEqual( document.querySelector( '.hljs-keyword' ).innerHTML, 'var', 'the same keyword: var.' );
+	});
+
+
+} );
+
+Reveal.initialize();
+
diff --git a/doc/pub/NeuralNet/html/reveal.js/test/test-markdown-options.html b/doc/pub/NeuralNet/html/reveal.js/test/test-markdown-options.html
new file mode 100644
index 000000000..5b3be9758
--- /dev/null
+++ b/doc/pub/NeuralNet/html/reveal.js/test/test-markdown-options.html
@@ -0,0 +1,41 @@
+
+
+
+	
+		
+
+		reveal.js - Test Markdown Options
+
+		
+		
+	
+
+	
+
+		

+		

+
+		
+
+		
+		
+		
+
+		
+
+	
+
diff --git a/doc/pub/NeuralNet/html/reveal.js/test/test-markdown-options.js b/doc/pub/NeuralNet/html/reveal.js/test/test-markdown-options.js
new file mode 100644
index 000000000..3ae13503a
--- /dev/null
+++ b/doc/pub/NeuralNet/html/reveal.js/test/test-markdown-options.js
@@ -0,0 +1,26 @@
+Reveal.addEventListener( 'ready', function() {
+
+	QUnit.module( 'Markdown' );
+
+	test( 'Options are set', function() {
+		strictEqual( marked.defaults.smartypants, true );
+	});
+
+	test( 'Smart quotes are activated', function() {
+		var text = document.querySelector( '.reveal .slides>section>p' ).textContent;
+
+		strictEqual( /['"]/.test( text ), false );
+		strictEqual( /[“”‘’]/.test( text ), true );
+	});
+
+} );
+
+Reveal.initialize({
+	dependencies: [
+		{ src: '../plugin/markdown/marked.js' },
+		{ src: '../plugin/markdown/markdown.js' },
+	],
+	markdown: {
+		smartypants: true
+	}
+});
diff --git a/doc/pub/NeuralNet/ipynb/.ipynb_checkpoints/NeuralNet-checkpoint.ipynb b/doc/pub/NeuralNet/ipynb/.ipynb_checkpoints/NeuralNet-checkpoint.ipynb
new file mode 100644
index 000000000..c13949670
--- /dev/null
+++ b/doc/pub/NeuralNet/ipynb/.ipynb_checkpoints/NeuralNet-checkpoint.ipynb
@@ -0,0 +1,1163 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "\n",
+    "# Data Analysis and Machine Learning: Elements of machine learning\n",
+    "\n",
+    "  \n",
+    "**Morten Hjorth-Jensen**, Department of Physics, University of Oslo and Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University\n",
+    "\n",
+    "Date: **Sep 28, 2018**\n",
+    "\n",
+    "Copyright 1999-2018, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license\n",
+    "\n",
+    "\n",
+    "\n",
+    "\n",
+    "\n",
+    "## Neural networks\n",
+    "\n",
+    "Artificial neural networks are computational systems that can learn to\n",
+    "perform tasks by considering examples, generally without being\n",
+    "programmed with any task-specific rules. It is supposed to mimic a\n",
+    "biological system, wherein neurons interact by sending signals in the\n",
+    "form of mathematical functions between layers. All layers can contain\n",
+    "an arbitrary number of neurons, and each connection is represented by\n",
+    "a weight variable.\n",
+    "\n",
+    "\n",
+    "## Artificial neurons\n",
+    "\n",
+    "The field of artificial neural networks has a long history of\n",
+    "development, and is closely connected with the advancement of computer\n",
+    "science and computers in general. A model of artificial neurons was\n",
+    "first developed by McCulloch and Pitts in 1943 to study signal\n",
+    "processing in the brain and has later been refined by others. The\n",
+    "general idea is to mimic neural networks in the human brain, which is\n",
+    "composed of billions of neurons that communicate with each other by\n",
+    "sending electrical signals.  Each neuron accumulates its incoming\n",
+    "signals, which must exceed an activation threshold to yield an\n",
+    "output. If the threshold is not overcome, the neuron remains inactive,\n",
+    "i.e. has zero output.\n",
+    "\n",
+    "This behaviour has inspired a simple mathematical model for an artificial neuron."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "\n",
+    "
\n",
+    "\n",
+    "$$\n",
+    "\\begin{equation}\n",
+    " y = f\\left(\\sum_{i=1}^n w_ix_i\\right) = f(u)\n",
+    "\\label{artificialNeuron} \\tag{1}\n",
+    "\\end{equation}\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Here, the output $y$ of the neuron is the value of its activation function, which have as input\n",
+    "a weighted sum of signals $x_i, \\dots ,x_n$ received by $n$ other neurons.\n",
+    "\n",
+    "Conceptually, it is helpful to divide neural networks into four\n",
+    "categories:\n",
+    "1. general purpose neural networks for supervised learning,\n",
+    "\n",
+    "2. neural networks designed specifically for image processing, the most prominent example of this class being Convolutional Neural Networks (CNNs),\n",
+    "\n",
+    "3. neural networks for sequential data such as Recurrent Neural Networks (RNNs), and\n",
+    "\n",
+    "4. neural networks for unsupervised learning such as Deep Boltzmann Machines.\n",
+    "\n",
+    "In natural science, DNNs and CNNs have already found numerous applications. In\n",
+    "statistical physics, they have been applied to detect phase\n",
+    "transitions in 2D Ising and Potts models, lattice gauge theories, and\n",
+    "different phases of polymers, or solving the Navier-Stokes equation in weather forecasting.\n",
+    "Deep learning has also found interesting applications in quantum\n",
+    "physics. Various quantum phase transitions can be detected and studied\n",
+    "using DNNs and CNNs,\n",
+    "topological phases, and even non-equilibrium many-body\n",
+    "localization. Representing quantum states as DNNs quantum state\n",
+    "tomography are among some of the impressive\n",
+    "achievements to reveal the potential of DNNs to facilitate the study\n",
+    "of quantum systems.\n",
+    "\n",
+    "In quantum information theory, it has been shown that one can perform\n",
+    "gate decompositions with the help of neural. In lattice quantum chromodynamics,\n",
+    "DNNs have been used to learn action parameters in regions of parameter\n",
+    "space where PCA fails. \n",
+    "\n",
+    "The applications are not limited to the natural sciences. There is a plethora of applications in essentially all disciplines, from the humanities to life science and medicine.\n",
+    "\n",
+    "## Neural network types\n",
+    "\n",
+    "An artificial neural network (NN), is a computational model that\n",
+    "consists of layers of connected neurons, or *nodes*.  It is supposed\n",
+    "to mimic a biological nervous system by letting each neuron interact\n",
+    "with other neurons by sending signals in the form of mathematical\n",
+    "functions between layers.  A wide variety of different NNs have been\n",
+    "developed, but most of them consist of an input layer, an output layer\n",
+    "and eventual layers in-between, called *hidden layers*. All layers can\n",
+    "contain an arbitrary number of nodes, and each connection between two\n",
+    "nodes is associated with a weight variable.\n",
+    "\n",
+    "Neural networks (also called neural nets) are neural-inspired\n",
+    "nonlinear models for supervised learning.  As we will see, neural nets\n",
+    "can be viewed as natural, more powerful extensions of supervised\n",
+    "learning methods such as linear and logistic regression and soft-max\n",
+    "methods.\n",
+    "\n",
+    "\n",
+    "## Feed-forward neural networks\n",
+    "\n",
+    "The feed-forward neural network (FFNN) was the first and simplest type of NN devised. In this network, \n",
+    "the information moves in only one direction: forward through the layers.\n",
+    "\n",
+    "Nodes are represented by circles, while the arrows display the connections between the nodes, including the \n",
+    "direction of information flow. Additionally, each arrow corresponds to a weight variable, not displayed here. \n",
+    "We observe that each node in a layer is connected to *all* nodes in the subsequent layer, \n",
+    "making this a so-called *fully-connected* FFNN. \n",
+    "\n",
+    "\n",
+    "\n",
+    "A different variant of FFNNs are *convolutional neural networks* (CNNs), which have a connectivity pattern\n",
+    "inspired by the animal visual cortex. Individual neurons in the visual cortex only respond to stimuli from\n",
+    "small sub-regions of the visual field, called a receptive field. This makes the neurons well-suited to exploit the strong\n",
+    "spatially local correlation present in natural images. The response of each neuron can be approximated mathematically \n",
+    "as a convolution operation. \n",
+    "\n",
+    "CNNs emulate the behaviour of neurons in the visual cortex by enforcing a *local* connectivity pattern\n",
+    "between nodes of adjacent layers: Each node\n",
+    "in a convolutional layer is connected only to a subset of the nodes in the previous layer, \n",
+    "in contrast to the fully-connected FFNN.\n",
+    "Often, CNNs \n",
+    "consist of several convolutional layers that learn local features of the input, with a fully-connected layer at the end, \n",
+    "which gathers all the local data and produces the outputs. They have wide applications in image and video recognition\n",
+    "\n",
+    "## Recurrent neural networks\n",
+    "\n",
+    "So far we have only mentioned NNs where information flows in one direction: forward. *Recurrent neural networks* on\n",
+    "the other hand, have connections between nodes that form directed *cycles*. This creates a form of \n",
+    "internal memory which are able to capture information on what has been calculated before; the output is dependent \n",
+    "on the previous computations. Recurrent NNs make use of sequential information by performing the same task for \n",
+    "every element in a sequence, where each element depends on previous elements. An example of such information is \n",
+    "sentences, making recurrent NNs especially well-suited for handwriting and speech recognition.\n",
+    "\n",
+    "## Other types of networks\n",
+    "\n",
+    "There are many other kinds of NNs that have been developed. One type that is specifically designed for interpolation\n",
+    "in multidimensional space is the radial basis function (RBF) network. RBFs are typically made up of three layers: \n",
+    "an input layer, a hidden layer with non-linear radial symmetric activation functions and a linear output layer (''linear'' here\n",
+    "means that each node in the output layer has a linear activation function). The layers are normally fully-connected and \n",
+    "there are no cycles, thus RBFs can be viewed as a type of fully-connected FFNN. They are however usually treated as\n",
+    "a separate type of NN due the unusual activation functions.\n",
+    "\n",
+    "## Multilayer perceptrons\n",
+    "\n",
+    "One uses often so-called fully-connected feed-forward neural networks\n",
+    "with three or more layers (an input layer, one or more hidden layers\n",
+    "and an output layer) consisting of neurons that have non-linear\n",
+    "activation functions.\n",
+    "\n",
+    "Such networks are often called *multilayer perceptrons* (MLPs)\n",
+    "\n",
+    "## Why multilayer perceptrons?\n",
+    "\n",
+    "According to the *Universal approximation theorem*, a feed-forward neural network with just a single hidden layer containing \n",
+    "a finite number of neurons can approximate a continuous multidimensional function to arbitrary accuracy, \n",
+    "assuming the activation function for the hidden layer is a **non-constant, bounded and monotonically-increasing continuous function**.\n",
+    "\n",
+    "Note that the requirements on the activation function only applies to\n",
+    "the hidden layer, the output nodes are always assumed to be linear, so\n",
+    "as to not restrict the range of output values.\n",
+    "\n",
+    "We note that this theorem is only applicable to an NN with *one* hidden\n",
+    "layer.  Therefore, we can easily construct an NN that employs\n",
+    "activation functions which do not satisfy the above requirements, as\n",
+    "long as we have at least one layer with activation functions that\n",
+    "*do*. Furthermore, although the universal approximation theorem lays\n",
+    "the theoretical foundation for regression with neural networks, it\n",
+    "does not say anything about how things work in practice: A neural\n",
+    "network can still be able to approximate a given function reasonably\n",
+    "well without having the flexibility to fit *all other* functions.\n",
+    "\n",
+    "\n",
+    "\n",
+    "## Mathematical model"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "\n",
+    "
\n",
+    "\n",
+    "$$\n",
+    "\\begin{equation}\n",
+    " y = f\\left(\\sum_{i=1}^n w_ix_i + b_i\\right) = f(u)\n",
+    "\\label{artificialNeuron2} \\tag{2}\n",
+    "\\end{equation}\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "In an FFNN of such neurons, the *inputs* $x_i$ are the *outputs* of\n",
+    "the neurons in the preceding layer. Furthermore, an MLP is\n",
+    "fully-connected, which means that each neuron receives a weighted sum\n",
+    "of the outputs of *all* neurons in the previous layer.\n",
+    "\n",
+    "## Mathematical model\n",
+    "\n",
+    "First, for each node $i$ in the first hidden layer, we calculate a weighted sum $u_i^1$ of the input coordinates $x_j$,"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "\n",
+    "
\n",
+    "\n",
+    "$$\n",
+    "\\begin{equation}\n",
+    " u_i^1 = \\sum_{j=1}^2 w_{ij}^1 x_j  + b_i^1 \n",
+    "\\label{_auto1} \\tag{3}\n",
+    "\\end{equation}\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "This value is the argument to the activation function $f_1$ of each neuron $i$,\n",
+    "producing the output $y_i^1$ of all neurons in layer 1,"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "\n",
+    "
\n",
+    "\n",
+    "$$\n",
+    "\\begin{equation}\n",
+    " y_i^1 = f_1(u_i^1) = f_1\\left(\\sum_{j=1}^2 w_{ij}^1 x_j  + b_i^1\\right)\n",
+    "\\label{outputLayer1} \\tag{4}\n",
+    "\\end{equation}\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "where we assume that all nodes in the same layer have identical\n",
+    "activation functions, hence the notation $f_l$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "\n",
+    "
\n",
+    "\n",
+    "$$\n",
+    "\\begin{equation}\n",
+    " y_i^l = f_l(u_i^l) = f_l\\left(\\sum_{j=1}^{N_{l-1}} w_{ij}^l y_j^{l-1} + b_i^l\\right)\n",
+    "\\label{generalLayer} \\tag{5}\n",
+    "\\end{equation}\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "where $N_l$ is the number of nodes in layer $l$. When the output of\n",
+    "all the nodes in the first hidden layer are computed, the values of\n",
+    "the subsequent layer can be calculated and so forth until the output\n",
+    "is obtained.\n",
+    "\n",
+    "\n",
+    "\n",
+    "## Mathematical model\n",
+    "\n",
+    "The output of neuron $i$ in layer 2 is thus,"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "\n",
+    "
\n",
+    "\n",
+    "$$\n",
+    "\\begin{equation}\n",
+    " y_i^2 = f_2\\left(\\sum_{j=1}^3 w_{ij}^2 y_j^1 + b_i^2\\right) \n",
+    "\\label{_auto2} \\tag{6}\n",
+    "\\end{equation}\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "\n",
+    "
\n",
+    "\n",
+    "$$\n",
+    "\\begin{equation} \n",
+    " = f_2\\left[\\sum_{j=1}^3 w_{ij}^2f_1\\left(\\sum_{k=1}^2 w_{jk}^1 x_k + b_j^1\\right) + b_i^2\\right]\n",
+    "\\label{outputLayer2} \\tag{7}\n",
+    "\\end{equation}\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "where we have substituted $y_m^1$ with. Finally, the NN output yields,"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "\n",
+    "
\n",
+    "\n",
+    "$$\n",
+    "\\begin{equation}\n",
+    " y_1^3 = f_3\\left(\\sum_{j=1}^3 w_{1m}^3 y_j^2 + b_1^3\\right) \n",
+    "\\label{_auto3} \\tag{8}\n",
+    "\\end{equation}\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "\n",
+    "
\n",
+    "\n",
+    "$$\n",
+    "\\begin{equation} \n",
+    " = f_3\\left[\\sum_{j=1}^3 w_{1j}^3 f_2\\left(\\sum_{k=1}^3 w_{jk}^2 f_1\\left(\\sum_{m=1}^2 w_{km}^1 x_m + b_k^1\\right) + b_j^2\\right)\n",
+    "  + b_1^3\\right]\n",
+    "\\label{_auto4} \\tag{9}\n",
+    "\\end{equation}\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Mathematical model\n",
+    "\n",
+    "We can generalize this expression to an MLP with $l$ hidden\n",
+    "layers. The complete functional form is,"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "\n",
+    "
\n",
+    "\n",
+    "$$\n",
+    "\\begin{equation}\n",
+    "y^{l+1}_1\\! = \\!f_{l+1}\\!\\left[\\!\\sum_{j=1}^{N_l}\\! w_{1j}^3 f_l\\!\\left(\\!\\sum_{k=1}^{N_{l-1}}\\! w_{jk}^2 f_{l-1}\\!\\left(\\!\n",
+    " \\dots \\!f_1\\!\\left(\\!\\sum_{n=1}^{N_0} \\!w_{mn}^1 x_n\\! + \\!b_m^1\\!\\right)\n",
+    " \\!\\dots \\!\\right) \\!+ \\!b_k^2\\!\\right)\n",
+    " \\!+ \\!b_1^3\\!\\right] \n",
+    "\\label{completeNN} \\tag{10}\n",
+    "\\end{equation}\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "which illustrates a basic property of MLPs: The only independent\n",
+    "variables are the input values $x_n$.\n",
+    "\n",
+    "## Mathematical model\n",
+    "\n",
+    "This confirms that an MLP, despite its quite convoluted mathematical\n",
+    "form, is nothing more than an analytic function, specifically a\n",
+    "mapping of real-valued vectors $\\vec{x} \\in \\mathbb{R}^n \\rightarrow\n",
+    "\\vec{y} \\in \\mathbb{R}^m$.  In our example, $n=2$ and\n",
+    "$m=1$. Consequentially, the number of input and output values of the\n",
+    "function we want to fit must be equal to the number of inputs and\n",
+    "outputs of our MLP.\n",
+    "\n",
+    "Furthermore, the flexibility and universality of a MLP can be\n",
+    "illustrated by realizing that the expression is essentially a nested\n",
+    "sum of scaled activation functions of the form"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "\n",
+    "
\n",
+    "\n",
+    "$$\n",
+    "\\begin{equation}\n",
+    " h(x) = c_1 f(c_2 x + c_3) + c_4\n",
+    "\\label{_auto5} \\tag{11}\n",
+    "\\end{equation}\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "where the parameters $c_i$ are weights and biases. By adjusting these\n",
+    "parameters, the activation functions can be shifted up and down or\n",
+    "left and right, change slope or be rescaled which is the key to the\n",
+    "flexibility of a neural network.\n",
+    "\n",
+    "### Matrix-vector notation\n",
+    "\n",
+    "We can introduce a more convenient notation for the activations in a NN. \n",
+    "\n",
+    "Additionally, we can represent the biases and activations\n",
+    "as layer-wise column vectors $\\vec{b}_l$ and $\\vec{y}_l$, so that the $i$-th element of each vector \n",
+    "is the bias $b_i^l$ and activation $y_i^l$ of node $i$ in layer $l$ respectively. \n",
+    "\n",
+    "We have that $\\mathrm{W}_l$ is a $N_{l-1} \\times N_l$ matrix, while $\\vec{b}_l$ and $\\vec{y}_l$ are $N_l \\times 1$ column vectors. \n",
+    "With this notation, the sum in  becomes a matrix-vector multiplication, and we can write\n",
+    "the equation for the activations of hidden layer 2 in"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "\n",
+    "
\n",
+    "\n",
+    "$$\n",
+    "\\begin{equation}\n",
+    " \\vec{y}_2 = f_2(\\mathrm{W}_2 \\vec{y}_{1} + \\vec{b}_{2}) = \n",
+    " f_2\\left(\\left[\\begin{array}{ccc}\n",
+    "    w^2_{11} &w^2_{12} &w^2_{13} \\\\\n",
+    "    w^2_{21} &w^2_{22} &w^2_{23} \\\\\n",
+    "    w^2_{31} &w^2_{32} &w^2_{33} \\\\\n",
+    "    \\end{array} \\right] \\cdot\n",
+    "    \\left[\\begin{array}{c}\n",
+    "           y^1_1 \\\\\n",
+    "           y^1_2 \\\\\n",
+    "           y^1_3 \\\\\n",
+    "          \\end{array}\\right] + \n",
+    "    \\left[\\begin{array}{c}\n",
+    "           b^2_1 \\\\\n",
+    "           b^2_2 \\\\\n",
+    "           b^2_3 \\\\\n",
+    "          \\end{array}\\right]\\right).\n",
+    "\\label{_auto6} \\tag{12}\n",
+    "\\end{equation}\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Matrix-vector notation  and activation\n",
+    "\n",
+    "The activation of node $i$ in layer 2 is"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "\n",
+    "
\n",
+    "\n",
+    "$$\n",
+    "\\begin{equation}\n",
+    " y^2_i = f_2\\Bigr(w^2_{i1}y^1_1 + w^2_{i2}y^1_2 + w^2_{i3}y^1_3 + b^2_i\\Bigr) = \n",
+    " f_2\\left(\\sum_{j=1}^3 w^2_{ij} y_j^1 + b^2_i\\right).\n",
+    "\\label{_auto7} \\tag{13}\n",
+    "\\end{equation}\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "This is not just a convenient and compact notation, but also a useful\n",
+    "and intuitive way to think about MLPs: The output is calculated by a\n",
+    "series of matrix-vector multiplications and vector additions that are\n",
+    "used as input to the activation functions. For each operation\n",
+    "$\\mathrm{W}_l \\vec{y}_{l-1}$ we move forward one layer.\n",
+    "\n",
+    "\n",
+    "### Activation functions\n",
+    "\n",
+    "A property that characterizes a neural network, other than its\n",
+    "connectivity, is the choice of activation function(s).  As described\n",
+    "in, the following restrictions are imposed on an activation function\n",
+    "for a FFNN to fulfill the universal approximation theorem\n",
+    "\n",
+    "  * Non-constant\n",
+    "\n",
+    "  * Bounded\n",
+    "\n",
+    "  * Monotonically-increasing\n",
+    "\n",
+    "  * Continuous\n",
+    "\n",
+    "### Activation functions, Logistic and Hyperbolic ones\n",
+    "\n",
+    "The second requirement excludes all linear functions. Furthermore, in\n",
+    "a MLP with only linear activation functions, each layer simply\n",
+    "performs a linear transformation of its inputs.\n",
+    "\n",
+    "Regardless of the number of layers, the output of the NN will be\n",
+    "nothing but a linear function of the inputs. Thus we need to introduce\n",
+    "some kind of non-linearity to the NN to be able to fit non-linear\n",
+    "functions Typical examples are the logistic *Sigmoid*"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "\n",
+    "
\n",
+    "\n",
+    "$$\n",
+    "\\begin{equation}\n",
+    " f(x) = \\frac{1}{1 + e^{-x}},\n",
+    "\\label{sigmoidActivationFunction} \\tag{14}\n",
+    "\\end{equation}\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "and the *hyperbolic tangent* function"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "\n",
+    "
\n",
+    "\n",
+    "$$\n",
+    "\\begin{equation}\n",
+    " f(x) = \\tanh(x)\n",
+    "\\label{tanhActivationFunction} \\tag{15}\n",
+    "\\end{equation}\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Relevance\n",
+    "\n",
+    "The *sigmoid* function are more biologically plausible because the\n",
+    "output of inactive neurons are zero. Such activation function are\n",
+    "called *one-sided*. However, it has been shown that the hyperbolic\n",
+    "tangent performs better than the sigmoid for training MLPs.  has\n",
+    "become the most popular for *deep neural networks*"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAEWCAYAAACdaNcBAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3Xd8leXdx/HPj2wymCGMMBwgsmREXE8tKraotbbWumdtqa1bu6x9tHbYaofaavWhzjpAba2iUrepqypEUGSHPU1IIGSQ/Xv+OMcSMZAAJ7nP+L5fr/PinPtc59y/c3HyzZXrXubuiIhIfOkSdAEiIhJ5CncRkTikcBcRiUMKdxGROKRwFxGJQwp3EZE4pHCXTmFmPzWze6NtvWa2yswm7+K5DDN71swqzOzJjquy1XUvMLNJnblOiS/JQRcgicHdb47B9Z4G5AG93L0xQiV9jpk9CKxz9599uszdR3bU+iQxaOQusmuDgaUdGewiHUXhLhFlZj82s/VmVmlmS8zsuPDyn5vZIy3anW9mq82szMz+t+X0SLjtk2b2SPh95pvZMDO7zsxKzGytmX2pxXv1N7OZZlZuZsVm9p0Wz+283vNarPf63XyOm4AbgDPMrMrMLm7lvYaYmZtZcvhxoZn90szeDtf9kpn1btH+f8zsHTPbGv4MF5rZVOAc4Efh9TwbbtuyP9LM7HYz2xC+3W5maeHnJpnZOjO7Ntw3G83sor39/5P4oXCXiDGzg4DLgEPdPRv4MrCqlXYjgL8QCrV+QDdgwE7NTgYeBnoAc4EXCX1fBwC/AP6vRdsZwDqgP6GplJvN7NhdrPdu4Lxw215Afmufxd1vBG4GHnf3LHe/r80OCDkbuAjoA6QCPwivezDwL+DPQC4wFpjn7tOAR4Fbw+s5uZX3vB44PPyaQ4CJwM9aPN+XHX14MXCXmfVoZ70SpxTuEklNQBowwsxS3H2Vuy9vpd1pwLPu/pa71xMaIe98kqM33f3F8JTIk4QC8bfu3kAozIeYWXczGwgcBfzY3WvdfR5wL3D+Ltb7nLu/4e51wP8Czfv+sT/jAXdf6u7bgScIBTKEQv8Vd5/u7g3uXhautT3OAX7h7iXuXgrcROgX1Kcaws83uPssoAo4KDIfR2KVwl0ixt2LgauAnwMlZjbDzPq30rQ/sLbF62qAsp3afNLi/nZgs7s3tXgMkBV+r3J3r2zRfjWf/0ugtfVWt7LefbWpxf2acI0AA4HWftG1R39Cn+lTq8PLPlW203aBluuVBKVwl4hy98fc/X8IbYx04JZWmm2kxXSImWUQmiLZGxuAnmaW3WLZIGD9LtY7sMV6u+7hequBri0e992D164FDtjFc22dmnUDof781KDwMpFdUrhLxJjZQWZ2bHhjXy2hEXZr0x5/B042syPNLJXQSN/2Zp3uvhZ4B/iNmaWb2RhC886PtNL878BXwhs2UwnN3e/Jz8A84GgzG2Rm3YDr9uC1jwKTzex0M0s2s15m9umUzSfA/rt57XTgZ2aWG95AewOtfz6R/1K4SySlAb8FNhOanuhDKwHo7guAywnNnW8kNEdcAtTt5XrPAoYQGs3+E7jR3V/ZxXovBR4Lr3cLoQ2x7eLuLwOPAx8BRcBze/DaNcCJwLVAOaFfFIeEn76P0HaKrWb2dCsv/xUwJ7ze+cAH4WUiu2S6WIcEzcyygK3AUHdfGXQ9IvFAI3cJhJmdbGZdzSwT+D2hEemqYKsSiR8KdwnKKYSmUTYAQ4EzXX9GikSMpmVEROKQRu4iInEosLNC9u7d24cMGRLU6v+rurqazMzMoMuICuqLEPXDDuqLHaKlL4qKija7e25b7QIL9yFDhjBnzpygVv9fhYWFTJo0KegyooL6IkT9sIP6Yodo6QszW912K03LiIjEJYW7iEgcUriLiMQhhbuISBxSuIuIxCGFu4hIHFK4i4jEIYW7iEgcUriLiMQhhbuISBxSuIuIxCGFu4hIHFK4i4jEoTbD3czuN7MSM/t4F8+bmf3JzIrN7CMzGx/5MkVEZE+0Z+T+IDBlN8+fQOgyaUOBqcDd+16WiIjsizbD3d3fAMp30+QU4G8e8i7Q3cz6RapAERHZc5G4WMcAYG2Lx+vCyzbu3NDMphIa3ZOXl0dhYWEEVr9vqqqqoqKOaKC+CFE/7KC+2CHW+qJTr8Tk7tOAaQAFBQUeDVc1iZarq0QD9UWI+mEH9cUOe9oX7k5lXSPbtjdQWRv6d1ttI1V1DVTVNlJZ10hVbSPVdY1U1TVRU99IdX0T2+sbqa5rorahiZr60PJ7zp3AkQf23qN6IxHu64GBLR7nh5eJiMSNhmZn/dbtbK6so7SyjvLqesqq6ymrqqO8pp6tNQ2UV9eztaaeinCQNzX7bt8zqYuRmZpEVloymWnJdE1LJjM1if7dU8hITSYjpQtdU5PJzU7b43ojEe4zgcvMbAZwGFDh7p+bkhERiVbVdY2s37qd9Vu2s6FiO5sqatlYUcsn2z691VGxvQFeeu1zr01P6UKvzDS6d02hZ2YqA3t2pXtGCt1a3HIykslJTyErPZns9BSy05PJSksmLbkLZtYhn6nNcDez6cAkoLeZrQNuBFIA3P0eYBZwIlAM1AAXdUilIiJ7yd0prapjZWk1KzdXs7q8hjXlNawN37bUNHymfReDPtnp5HVLZ7/emRy+fy+qNm9g4ujh5Gan0TsrjV5ZqfTKTCMjNSmgT7V7bYa7u5/VxvMOXBqxikRE9pK7s7GiliWbKlnySSXFJVUsK6liRUkVlXWN/22X3MXI75HBwJ5dGTW6H/k9MhjQPYP8Hhn0755BblYayUmf3ZmwsHAzkyYO6uyPtNc6dYOqiEikNDU7K0qr+HhDBR+v38bH6ytYtHEb22p3hHif7DSG5mVx6vgB7J+bxZDemezfO5N+3dI/F97xRuEuIjGhvLqeotVbKFq9hQ/XbmX++gqqwqPx9JQuHNwvh68c0p+D+2YzvF8Ow/Ky6ZaREnDVwVG4i0hUKq2s4z8ryvjP8jJmryqnuKQKgJQk4+B+OXx93AAOGdidMfnd2L93ZtyPxPeUwl1EokJtQxPvryznjaWlvLGslKWfhMI8Oy2ZgiE9+Pq4ARw6pCdj8ruRnhKdGzGjicJdRAJTsq2WVxeX8MrCT3h7+WZqG5pJTe7CxCE9+fq4fI48oBcj++doVL4XFO4i0qnWb93Ov+Zv5Pn5G5m7ZisA+T0yOPPQQXzxoFwO369X1O5eGEsU7iLS4cqr63n2ww08M289H4QDfWT/HK49fhjHj8zjoLzsDjuYJ1Ep3EWkQzQ0NfPqohL+XrSWwiWlNDY7w/tm88MvH8RJo/sxpHdm0CXGNYW7iETUmrIaZsxew5NF6yitrCMvJ42L/2c/vj5+AMP75gRdXsJQuIvIPnN33i4u48F3VvLq4hIMOHZ4H86aOIgvDsvVBtEAKNxFZK/VNzbz9Nz1/PXNFSwrqaJXZiqXH3MgZx02iH7dMoIuL6Ep3EVkj1XVNTL9vTXc99ZKNm2rZUS/HP7wzUP4yiH9SEvWni7RQOEuIu1WXdfI3/6zmmlvLGdLTQNH7N+LW08bwxeG9tbeLlFG4S4ibaptaOKRd1dzd+Fyyqrr+eKwXK6cPJTxg3oEXZrsgsJdRHap2Z2nPljHH15ayvqt2/mfA3tz9fHDmDBYoR7tFO4i0qp3V5Tx83dqWVP5IaMG5PC708bs8XU8JTgKdxH5jI0V2/n184t47qON9Eo37jhzLCeP6U+XLppTjyUKdxEBQkeU3vfWSu54ZRnN7lx53FBGdFnPl8cOCLo02QsKdxHho3Vb+fE/5rNo4zYmH5zHjSePYGDPrhQWbgi6NNlLCneRBFbb0MTvX1zC/W+vJDc7jXvOncCUUX2DLksiQOEukqDmrtnCtU9+yIrSas45bBA/PmE4OemJe1m6eKNwF0kw9Y3N3PHqUu4uXE7fnHQe/fZhHKW9YOKOwl0kgazaXM0VM+by0boKvjkhn/89eYRG63FK4S6SIJ6Zt57r//kxXQzuOXc8U0b1C7ok6UAKd5E4V9vQxI3PLODxOWspGNyD288cS36PrkGXJR1M4S4Sx9aW1/C9R4v4eP02Lj3mAK6ePEznVk8QCneROPXvpaVcOWMuTc3OvecXMHlEXtAlSSdSuIvEGXfn3jdXcvO/FnFQXjb3nDtB1ytNQO36+8zMppjZEjMrNrOftPL8IDN73czmmtlHZnZi5EsVkbbUNTbxo79/xK9nLWLKyL489f0jFewJqs2Ru5klAXcBxwPrgNlmNtPdF7Zo9jPgCXe/28xGALOAIR1Qr4jsQnl1Pd99eA6zV23himMP5KrJw3SyrwTWnmmZiUCxu68AMLMZwClAy3B34NPLmncDdEIKkU60uqyaC+5/nw0Vtdxx5lhO0cm+Ep65++4bmJ0GTHH3b4cfnwcc5u6XtWjTD3gJ6AFkApPdvaiV95oKTAXIy8ubMGPGjEh9jr1WVVVFVlZW0GVEBfVFSKz1w4qKJm4rqqXZ4arx6QztEblrmMZaX3SkaOmLY445psjdC9pqF6kNqmcBD7r7H8zsCOBhMxvl7s0tG7n7NGAaQEFBgU+aNClCq997hYWFREMd0UB9ERJL/fDa4k/43atz6Z2dwYMXTeSA3MiGTyz1RUeLtb5oT7ivBwa2eJwfXtbSxcAUAHf/j5mlA72BkkgUKSKf98y89VzzxIcc3C+b+y88lD7Z6UGXJFGkPXvLzAaGmtl+ZpYKnAnM3KnNGuA4ADM7GEgHSiNZqIjs8PC7q7nq8XkUDO7B9O8crmCXz2lz5O7ujWZ2GfAikATc7+4LzOwXwBx3nwlcC/zVzK4mtHH1Qm9rMl9E9srdhcu55YXFHDe8D3edM570lMjNsUv8aNecu7vPIrR7Y8tlN7S4vxA4KrKlicjObn9lKbe/soyvHtKfP5x+CCk6lYDsgo5QFYkB7s5tLy/lT68Vc9qEfG75xhiStA+77IbCXSTKuTu/f2kJd72+nDMKBvKbU0fr4CRpk8JdJMr94aWl3PX6cs6aOIhff22Ugl3aRRN2IlHszteWcefrxZw1caCCXfaIwl0kSt375gp+/9JSTh03gF9/TVMxsmcU7iJR6NH3VvOr5xdx0uh+3HraGAW77DGFu0iUee6jDfzs6Y85bngfbjtjrK6cJHtF3xqRKPLG0lKufnwehw7pyV3njCc1WT+isnf0zRGJEnPXbOG7DxdxYJ9s7r2gQEeeyj5RuItEgeKSKi56cDZ9ctJ46FuHkpOeEnRJEuMU7iIBK6ms5cIH3ie5i/G3b03UScAkInQQk0iAqusaufjBOZRV1fP4dw9ncC9d71QiQyN3kYA0NjVz6WMfsGBDBXedM44x+d2DLkniiEbuIgFwd26YuYDCJaXc/PXRHDs8L+iSJM5o5C4SgPveWslj763hki8ewNmHDQq6HIlDCneRTvbKwk/49axFTBnZlx99+aCgy5E4pXAX6UQLNlRwxYy5jB7QjdvOGKvTCkiHUbiLdJLSyjq+/dAcumWkcO/5BWSk6iAl6TjaoCrSCeoam7jkkSK21jTw5CVH0CdH+7JLx1K4i3Qwd+dn//yYotVbuOvs8Ywa0C3okiQBaFpGpIM98PYqnixaxxXHDeWkMf2CLkcShMJdpAO9tWwzv3p+IV8emcdVxw0NuhxJIAp3kQ6ytryGy6Z/wIF9svjj6dozRjqXwl2kA2yvb+K7DxfR1OxMO6+AzDRt3pLOpW+cSIS5O9c99RGLNm3jvgsKGNJbJwOTzqeRu0iEPfD2Kp6et4FrJg/TOWMkMAp3kQiavaqcm2ct4vgReVx6zIFBlyMJrF3hbmZTzGyJmRWb2U920eZ0M1toZgvM7LHIlikS/Uoqa7n00Q/I75HBH04/RBtQJVBtzrmbWRJwF3A8sA6YbWYz3X1hizZDgeuAo9x9i5n16aiCRaJRY1Mzlz82l221DTz0rYm6TJ4Erj0j94lAsbuvcPd6YAZwyk5tvgPc5e5bANy9JLJlikS3W19cwnsry/nNqaM5uF9O0OWItGtvmQHA2haP1wGH7dRmGICZvQ0kAT939xd2fiMzmwpMBcjLy6OwsHAvSo6sqqqqqKgjGqgvQva0H4o+aWTa3DqOHZRMj4piCguLO664TqbvxA6x1heR2hUyGRgKTALygTfMbLS7b23ZyN2nAdMACgoKfNKkSRFa/d4rLCwkGuqIBuqLkD3phzVlNVxe+CaH5Hfj7qlHkJYcX2d61Hdih1jri/ZMy6wHBrZ4nB9e1tI6YKa7N7j7SmApobAXiVu1DU1879Eiuphx59nj4y7YJba1J9xnA0PNbD8zSwXOBGbu1OZpQqN2zKw3oWmaFRGsUyTq3PTsQhZs2MYfTz+EgT27Bl2OyGe0Ge7u3ghcBrwILAKecPcFZvYLM/tquNmLQJmZLQReB37o7mUdVbRI0J6Zt57p74eugXrcwTpQSaJPu+bc3X0WMGunZTe0uO/ANeGbSFwrLqniuqfmc+iQHvzgS8OCLkekVTpCVWQPbK9v4tJHPyA9JYk/nTWO5CT9CEl00onDRPbATc8uYMknlTx40aH065YRdDkiu6Rhh0g7PT13PTNmr+X7kw5g0kE6CFuim8JdpB2Wl1bx03+G5tmvOV7z7BL9FO4ibahtCM2zpyV30Ty7xAzNuYu04ZfPLWTxpkoeuFDz7BI7NAQR2Y3nPtrAo++t4btH788xwzXPLrFD4S6yC6vLqrnuH/MZN6g7P/jyQUGXI7JHFO4irahvbOby6XMxgz+dOY4UzbNLjNGcu0grbnlhMR+tq+Cec8frvDESkxTuIjuZW9LIfR+s5PwjBjNlVL+gyxHZK/pbU6SFjRXbuXd+HSP65fDTEw8OuhyRvaZwFwlrbGrmiulzaWyGO88eR3qKzs8usUvhLhJ2x6vLmL1qCxeMTGP/3KygyxHZJwp3EeCtZZu58/VivjkhnyP7a1OUxD6FuyS80so6rnp8HgfkZnHTKSODLkckIjREkYTW3Oxc88Q8KmsbeOTbE+maqh8JiQ8auUtCu/vfy3lz2WZuPHkkw/vmBF2OSMQo3CVhzV5Vzh9fXsrJh/TnrIkDgy5HJKIU7pKQyqvrufyxuQzskcHNXx+FmQVdkkhEaYJREk5zs/PDJz+kvLqep75/JNnpKUGXJBJxGrlLwrn3rRW8uriE6086mFEDugVdjkiHULhLQilaXc4tLyzhxNF9Of+IwUGXI9JhFO6SMLaE59kHdM/gt98Yo3l2iWuac5eE0NzsXPvkh2yuqucf3zuSHM2zS5zTyF0Swv+9sYLXFpfws68czOh8zbNL/FO4S9x7b0UZv39pCSeN6cd5h2ueXRJDu8LdzKaY2RIzKzazn+ym3TfMzM2sIHIliuy90so6Lp8+l8E9u3KL5tklgbQZ7maWBNwFnACMAM4ysxGttMsGrgTei3SRInujqdm5csZcttU28Jdzx5OVpk1MkjjaM3KfCBS7+wp3rwdmAKe00u6XwC1AbQTrE9lrt7+ylHeWl/HLU0bpvDGScNozlBkArG3xeB1wWMsGZjYeGOjuz5vZD3f1RmY2FZgKkJeXR2Fh4R4XHGlVVVVRUUc0iKe+mFfSyJ8/qOMLA5LJrVpOYeHydr82nvphX6kvdoi1vtjnv1PNrAvwR+DCttq6+zRgGkBBQYFPmjRpX1e/zwoLC4mGOqJBvPTFmrIarvjzm4zsn8NfLzlyjy+XFy/9EAnqix1irS/aMy2zHmh5yrz88LJPZQOjgEIzWwUcDszURlUJQm1DE5c8UoSZcc+5E3QdVElY7Qn32cBQM9vPzFKBM4GZnz7p7hXu3tvdh7j7EOBd4KvuPqdDKhbZBXfnZ09/zKJN27j9jLEM7Nk16JJEAtNmuLt7I3AZ8CKwCHjC3ReY2S/M7KsdXaBIez3y3hr+XrSOy48dyjHD+wRdjkig2jXn7u6zgFk7LbthF20n7XtZIntm9qpybpq5gGMOyuWq44YGXY5I4HSEqsS8TRW1fO+RD8jvkcHtZ46jSxcdqCSiozokptU1NvG9R4uoqW/kse8cRrcMnRBMBBTuEsPcnRueXsDcNVv5yznjGZaXHXRJIlFD0zISsx56ZxWPz1nL5cceyImj+wVdjkhUUbhLTHpr2WZ++fwijh+Rx9WThwVdjkjUUbhLzFm1uZpLH/uAA3OzuO2MsdqAKtIKhbvElIrtDVz80Gy6GNx7QYHO9CiyC/rJkJjR0NTM9x8tYk15DQ9ffJiOQBXZDYW7xAR354ZnFvB2cRm//+YhHL5/r6BLEolqmpaRmHDfWyuZ/v4avj/pAE6bkB90OSJRT+EuUe+Fjzfy61mLOGFUX37wpYOCLkckJijcJaoVrS7nyhnzGDewu/aMEdkDCneJWstLq7j4oTn0757BvRccqnOzi+wBhbtEpdLKOi584H2SzHjwokPpmZkadEkiMUV7y0jU2VbbwIUPvE9pZR0zph7B4F6ZQZckEnM0cpeoUtvQxHcemsOSTZXcfe4Exg7sHnRJIjFJI3eJGo1NzVwxfS7vrSznjjPHcsxBupqSyN7SyF2iQnOzc91T83lp4Sf8/OQRnDJ2QNAlicQ0hbsEzt25ceYCnixaxxXHDeXCo/YLuiSRmKdwl0C5OzfPWsTD765m6tH7c/VkXf9UJBIU7hKo215eyl/fXMkFRwzmuhOGY6aDlEQiQRtUJRDuzu2vLONPrxVzekE+N548UsEuEkEKd+l07s4fX17Kn18r5psT8vnNqWN0WgGRCFO4S6dyd3734hL+UricMwoG8ptTRyvYRTqAwl06zacbT//65krOmjiIX39tlIJdpIMo3KVTNDU7P31qPo/PWcv5Rwzm5yePVLCLdCCFu3S4+sZmrn58Hs/P38jlxx7INccP08ZTkQ7Wrl0hzWyKmS0xs2Iz+0krz19jZgvN7CMze9XMBke+VIlF1XWNfPtvc3h+/kZ+euJwrv3SQQp2kU7QZribWRJwF3ACMAI4y8xG7NRsLlDg7mOAvwO3RrpQiT0llbWcMe0/vF28mVu+MZqpRx8QdEkiCaM9I/eJQLG7r3D3emAGcErLBu7+urvXhB++C+gilwmuuKSKU//yDstLqrn3/ALOOHRQ0CWJJBRz9903MDsNmOLu3w4/Pg84zN0v20X7O4FN7v6rVp6bCkwFyMvLmzBjxox9LH/fVVVVkZWVFXQZUSFSfbGorIk759WSZHD1hHT26xZbV1DSd2IH9cUO0dIXxxxzTJG7F7TVLqIbVM3sXKAA+GJrz7v7NGAaQEFBgU+aNCmSq98rhYWFREMd0SASfTH9/TX84aWPGdwrkwcunMigXl0jU1wn0ndiB/XFDrHWF+0J9/XAwBaP88PLPsPMJgPXA19097rIlCexorGpmZtnLeb+t1dy9LBc7jx7HDnpKUGXJZKw2hPus4GhZrYfoVA/Ezi7ZQMzGwf8H6Hpm5KIVylRrayqjitnzOOt4s1cdNQQrj/xYJKTdE46kSC1Ge7u3mhmlwEvAknA/e6+wMx+Acxx95nA74As4Mnwbm5r3P2rHVi3RIkP127le48Usbm6nlu/MYbTDx3Y9otEpMO1a87d3WcBs3ZadkOL+5MjXJdEOXfnsffXcNPMheRmp/GPS45kdH63oMsSkTAdoSp7rGJ7Az99aj7Pz9/I0cNyueOMsfTITA26LBFpQeEue+SDNVu4YvpcNlbU8uMpw/nu0fvrHDEiUUjhLu3S0NTMna8Vc+frxfTrls6TlxzB+EE9gi5LRHZB4S5tWvZJJdc88SHz11fwtbH9uemUUXTL0G6OItFM4S671NDUzL1vruS2V5aSlZbM3eeM54TR/YIuS0TaQeEurfpo3VZ+/I/5LNq4jS+PzONXXxtNbnZa0GWJSDsp3OUzttU2cPvLy3jwnZXkZqdxz7kTmDKqb9BlicgeUrgLAM3NzpvrGvjB7wspq67nnMMG8aMpw3UKAZEYpXAXilZv4VfPL2TumnrGD+rOAxdO1AFJIjFO4Z7AVpRWcesLS3hhwSZys9P4zuhUrjvrSO23LhIHFO4JaP3W7dz5WjFPzFlLenIXrjl+GN/+wn68/85bCnaROKFwTyAbK7bzl9eXM2P2GgzjnMMGcfmxQ7UXjEgcUrgngOWlVUz79wqemrsOdzj90IFcesyBDOieEXRpItJBFO5xyt2Zs3oL9725khcXbiI1qQtnHjqIqUfvz8CesXd1JBHZMwr3OFPb0MSzH27gwXdWsWDDNrplpHDZMQdywZFD6J2l6ReRRKFwjxPLPqlk+vtreWruOrbWNDAsL4ubvz6ar43rT9dU/TeLJBr91MewipoGnpu/gac+WE/R6i2kJBlfGtmXcyYO4ogDehG+KpaIJCCFe4ypqW/k9cWlPPvhBl5bXEJ9UzND+2Rx3QnDOW1CPr009SIiKNxjwrbaBv69pJQXPt7Ea4tL2N7QRO+sNM49fDCnjh/AyP45GqWLyGco3KOQu7OqrIZ/LynhlUUlvLuijMZmp3dWKt+YMICTRvdn4n49SdIBRyKyCwr3KLGlup53V5Tx9vLN/HtpKWvLtwNwQG4mF39hP740Io+xA3so0EWkXRTuASnZVsuc1VuYs2oL764oY9GmbbhD19QkjjygF1O/sD9HD8tlcK/MoEsVkRikcO8EtQ1NLNiwjQ/XbmVe+LamvAaA9JQujBvYg2smD+PIA3sxJr87KUldAq5YRGKdwj3CNlfVsXhjJYs3bWPhhm18vKGC5aXVNDU7AP26pXNIfnfOO3wwBUN6MLJ/N1KTFeYiElkK973Q1Oxs2LqdlZurWVFaRXFpFcs+qWJ5aRWbq+r/2y43O43RA7oxZWRfRg7oxtiB3cnLSQ+wchFJFAr3Vrg7W2saWLdlO+u31rBuy3bWltewJnxbW76d+qbm/7bPSU9maF42xw3PY1jfbIb3zeagvtk63F9EApNQ4e7uVNU1UlpZx+aqejZX1fH2qgbe/ddiSipr2VQRum2sqGV7Q9NnXpuVlsygnl0Z2iebySPy2L93JkN6ZbJfbia5WWnaz1xEokpMhru7U1PfRGVtI9tqG9i2vYGKFretNQ0eyiqDAAAF3UlEQVRsralnS00DW2rqKauqp7y6nvKaeuobmz/3finLVpCblUbfbukc3C+HY4f3oW+3dPJ7dCW/RwYDumfQvWuKAlxEYka7wt3MpgB3AEnAve7+252eTwP+BkwAyoAz3H3V7t5zW20Dz364ge0NTdQ2NLG9voma+tD96vpGaurC/9Y3UVXXSHVdI9V1TVTWNlBV10h4++Qu5aQn0zMzle5dU+nbLZ2R/XPomZVKr8xUcrPT6J0VuhXPL+KkyZN0BSIRiStthruZJQF3AccD64DZZjbT3Re2aHYxsMXdDzSzM4FbgDN2976ry2q4fPrczy1PT+lC19RkuqYmkZmaTEZqEtnpyeRlp5OZlkx2euiWlZZMTkYK2enJ5KSnkJORQreMFLqHlyW3c3fCT5aYgl1E4k57Ru4TgWJ3XwFgZjOAU4CW4X4K8PPw/b8Dd5qZufsux9cH5mbxzDVHk5acRHpKEl1Tk8hISVLQiohEQHvCfQCwtsXjdcBhu2rj7o1mVgH0Aja3bGRmU4GpAHl5eaxbWLSXZUdOVVUVhYWFQZcRFdQXIeqHHdQXO8RaX3TqBlV3nwZMAygoKPBJkyZ15upbVVhYSDTUEQ3UFyHqhx3UFzvEWl+0Z2J6PTCwxeP88LJW25hZMtCN0IZVEREJQHvCfTYw1Mz2M7NU4Exg5k5tZgIXhO+fBry2u/l2ERHpWG1Oy4Tn0C8DXiS0K+T97r7AzH4BzHH3mcB9wMNmVgyUE/oFICIiAWnXnLu7zwJm7bTshhb3a4FvRrY0ERHZWzodoYhIHFK4i4jEIYW7iEgcUriLiMQhhbuISBxSuIuIxCGFu4hIHFK4i4jEIYW7iEgcUriLiMQhhbuISBxSuIuIxCEL6sy8ZlYKrA5k5Z/Vm52uGJXA1Bch6ocd1Bc7REtfDHb33LYaBRbu0cLM5rh7QdB1RAP1RYj6YQf1xQ6x1healhERiUMKdxGROKRwD1+wWwD1xafUDzuoL3aIqb5I+Dl3EZF4pJG7iEgcUriLiMQhhXuYmV1rZm5mvYOuJShm9jszW2xmH5nZP82se9A1dTYzm2JmS8ys2Mx+EnQ9QTGzgWb2upktNLMFZnZl0DUFzcySzGyumT0XdC3toXAn9EUGvgSsCbqWgL0MjHL3McBS4LqA6+lUZpYE3AWcAIwAzjKzEcFWFZhG4Fp3HwEcDlyawH3xqSuBRUEX0V4K95DbgB8BCb112d1fcvfG8MN3gfwg6wnARKDY3Ve4ez0wAzgl4JoC4e4b3f2D8P1KQqE2INiqgmNm+cBJwL1B19JeCR/uZnYKsN7dPwy6lijzLeBfQRfRyQYAa1s8XkcCB9qnzGwIMA54L9hKAnU7oQFgc9CFtFdy0AV0BjN7BejbylPXAz8lNCWTEHbXF+7+TLjN9YT+LH+0M2uT6GNmWcA/gKvcfVvQ9QTBzL4ClLh7kZlNCrqe9kqIcHf3ya0tN7PRwH7Ah2YGoWmID8xsortv6sQSO82u+uJTZnYh8BXgOE+8gyDWAwNbPM4PL0tIZpZCKNgfdfengq4nQEcBXzWzE4F0IMfMHnH3cwOua7d0EFMLZrYKKHD3aDjzW6czsynAH4Evuntp0PV0NjNLJrQh+ThCoT4bONvdFwRaWAAsNNp5CCh396uCridahEfuP3D3rwRdS1sSfs5dPuNOIBt42czmmdk9QRfUmcIbky8DXiS0AfGJRAz2sKOA84Bjw9+FeeGRq8QIjdxFROKQRu4iInFI4S4iEocU7iIicUjhLiIShxTuIiJxSOEuIhKHFO4iInFI4S4SZmaXtDhgZ6WZvR50TSJ7SwcxiewkfE6V14Bb3f3ZoOsR2RsauYt83h3Aawp2iWUJcVZIkfYKnxVzMKFzzIjELE3LiISZ2QRCZ0L8grtvCboekX2haRmRHS4DegKvhzeqxswl1UR2ppG7iEgc0shdRCQOKdxFROKQwl1EJA4p3EVE4pDCXUQkDincRUTikMJdRCQO/T/+UeY0C/tmnQAAAABJRU5ErkJggg==\n",
+      "text/plain": [
+       "
"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    },
+    {
+     "data": {
+      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYYAAAEWCAYAAABi5jCmAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAGnhJREFUeJzt3X2QZXV95/H3h+FBEzSAaIvDCBjYrPiEoQNumY0NKhmNC6ldE2E3ChvJbFJiYhI3YtxSi8SEmN0lm8KsThFKfIhjdNVMUhMRhV438YnBIMIkwIBEZsQQAR86Kjj0d/+4p6fvabpneuYe+va9835VdfU9z9/+Vc98+pzfOb+TqkKSpDkHDbsASdLqYjBIkloMBklSi8EgSWoxGCRJLQaDJKnFYJD2QZIfS3Jjku8k+dUVPO5Tk8wkWbNSx9SBy2DQWEny1iTvexQP8VvAdVX1uKr640frIEnuSvKiuemq+mpVHV5VDz9ax5TmGAzSvjkOuGXYRUiPJoNBIynJG5LsbC7p3JrkhUnWA78NvKK57PKlZt0fSfKnSe5ptvnduUsySS5I8rdJLk/yrST/kOSFSxzzWuAM4PJm//8qyXSSC/vWuSDJ3/RNV5JfTnJ7km8meUeS9C3/pSR/3/wc25L8eJL3Ak8F/rI5zm8lOb7Z18HNdk9JsjnJ/Um2J/mlvn2+NcmfJ3lPs99bkkx22f4abwaDRk6SHwMuAn6iqh4H/DRwV1V9HPg94IPNZZfnNJu8G9gFnAg8FzgLuLBvl6cDdwBHA28BPpLkqIXHraozgf8HXNTs/7Zllvwy4CeAZwM/39RLkp8D3gq8Cng8cDZwX1W9Evgq8O+a47x9kX1uAnYATwFeDvxekjP7lp/drHMEsBm4fJm1SgaDRtLDwGHAyUkOqaq7quqOxVZMMgG8FHhdVf1LVd0LXAac27favcAfVdUPquqDwK3Az3RY76VV9c2q+ipwHXBKM/9C4O1VdX31bK+qf9zbzpKsA54PvKGqvl9VNwJX0AuYOX9TVVuaPon3As9ZZFfSogwGjZyq2g68jt5f2/cm2ZTkKUusfhxwCHBPcynnm8C7gCf1rbOz2qNJ/iO9v8S78vW+z98FDm8+r6N3prKvngLcX1Xf6Zv3j8DaPRzzMXOXoaS9MRg0kqrqz6rqJ+n9x1/AH8wtWrDq3cCDwNFVdUTz9fiqekbfOmv7r/vTu77/tWWW8i/AD/VNP3nZP0Svth9dYtmehj3+GnBUksf1zXsqsHMfji0tyWDQyGmeJTgzyWHA94HvAbPN4n8Cjk9yEEBV3QN8AvgfSR6f5KAkP5rkBX27fBLwq0kOaa77Px3YssxybgT+fZIfSnIi8Op9+FGuAF6f5NT0nJjkuL6f42mLbVRVdwOfAX4/yWOSPLs57qN5m64OIAaDRtFhwKXAN+hdMnkS8MZm2Yea7/cl+WLz+VXAocA24AHgw8Axffv7PHBSs7+3AS+vqvuWWctlwEP0/iO/Cnj/cn+IqvpQc7w/A74DfAyY6/T+feC/NZe/Xr/I5ucBx9M7e/go8Jaq+uRyjy3tSXxRjw5kSS4ALmwuS0nCMwZJ0gKdBEOSK5Pcm+TmJZZPNQ8P3dh8vblv2frmAaXtSS7uoh5J0v7r5FJSkp8CZoD3VNUzF1k+Bby+ql62YP4a4DbgxfQe1rkeOK+qtg1clCRpv3RyxlBVnwbu349NTwO2V9WdVfUQvSc1z+miJknS/lnJB17+TTN2zdfonT3cQu+BnLv71tlBb3iCR0iyAdgA8NjHPvbUdevWPcrl7tns7CwHHWQXDdgW/WyLebbFvNXSFrfddts3quqJe1tvpYLhi8BxVTWT5KX0bss7aV92UFUbgY0Ak5OTtXXr1u6r3AfT09NMTU0NtYbVwraYZ1vMsy3mrZa2SLLXIVdghe5KqqpvV9VM83kLcEiSo+k9qdn/p/+x+PSmJA3VigRDkifPDTmQ5LTmuPfR62w+KckJSQ6lN7DZ5pWoSZK0uE4uJSX5ADAFHJ1kB72hiw8BqKp30hsW+FeS7KI3fMG5zaBlu5JcBFwNrAGubPoeJElD0kkwVNV5e1l+OUuMB99cWlruuDSSpEfZ8LvJJUmrisEgSWoxGCRJLQaDJKnFYJAktRgMkqQWg0GS1GIwSJJaDAZJUovBIElqMRgkSS0GgySpxWCQJLUYDJKkFoNBktRiMEiSWgwGSVKLwSBJaukkGJJcmeTeJDcvsfw/JbkpyZeTfCbJc/qW3dXMvzHJ1i7qkSTtv67OGN4NrN/D8q8AL6iqZwG/A2xcsPyMqjqlqiY7qkeStJ8O7mInVfXpJMfvYfln+iY/BxzbxXElSd0bRh/Dq4G/7psu4BNJbkiyYQj1SJL6pKq62VHvjOGvquqZe1jnDOBPgJ+sqvuaeWurameSJwHXAK+tqk8vsu0GYAPAxMTEqZs2beqk7v01MzPD4YcfPtQaVgvbYp5tMc+2mLda2uKMM864YTmX7Du5lLQcSZ4NXAG8ZC4UAKpqZ/P93iQfBU4DHhEMVbWRpm9icnKypqamVqLsJU1PTzPsGlYL22KebTHPtpg3am2xIpeSkjwV+Ajwyqq6rW/+Dyd53Nxn4Cxg0TubJEkro5MzhiQfAKaAo5PsAN4CHAJQVe8E3gw8AfiTJAC7mtOZCeCjzbyDgT+rqo93UZMkaf90dVfSeXtZfiFw4SLz7wSe88gtJEnD4pPPkqQWg0GS1GIwSJJaDAZJUovBIElqMRgkSS0GgySpxWCQJLUYDJKkFoNBktRiMEiSWgwGSVKLwSBJajEYJEktBoMkqcVgkCS1GAySpBaDQZLUYjBIklo6CYYkVya5N8nNSyxPkj9Osj3JTUl+vG/Z+Ulub77O76IeSdL+O7ij/bwbuBx4zxLLXwKc1HydDvxv4PQkRwFvASaBAm5IsrmqHuioLmlFzVbx/R88POwyVoWHHrYt5qyGtjgoWfa6nQRDVX06yfF7WOUc4D1VVcDnkhyR5BhgCrimqu4HSHINsB74QBd1SSvtshse5Bev/viwy1g9rrEtdhtyW5zxY09c9rpdnTHszVrg7r7pHc28peY/QpINwAaAiYkJpqenH5VCl2tmZmboNawWtsW8e2Z28dTHreG0Y9YMu5She+jBhzj0sEOHXcaqsBra4kmP/fay112pYBhYVW0ENgJMTk7W1NTUUOuZnp5m2DWsFrZFn/+7hckTj+G/v+KUYVcydP5ezFstbfGG85a33krdlbQTWNc3fWwzb6n50kiqApZ/KVdalVYqGDYDr2ruTnoe8K2quge4GjgryZFJjgTOauZJI6nYt04+aTXq5FJSkg/Q60g+OskOencaHQJQVe8EtgAvBbYD3wX+c7Ps/iS/A1zf7OqSuY5oaVQdZC5oxHV1V9Ier1w1dyO9ZollVwJXdlGHNGyzBfFakkacTz5LHTvIf1Uacf4KSx2aLbD3WaPOYJA6VJR9DBp5BoPUpfKuJI0+g0Hq0CxgLmjUGQxSxzxj0KgzGKQO9TqfpdFmMEgd84xBo85gkDpUZR+DRp/BIHVoFofE0OgzGKQuebuqxoDBIHVoFnzwWSPPYJC65BmDxoDBIHXI9/RoHBgMUod8UY/GgcEgdcjbVTUODAapQwXEZNCIMxikjvReVOhzDBp9nQRDkvVJbk2yPcnFiyy/LMmNzddtSb7Zt+zhvmWbu6hHGoa5cZJ8tadG3cDvfE6yBngH8GJgB3B9ks1VtW1unar69b71Xws8t28X36uqUwatQxo2zxg0Lro4YzgN2F5Vd1bVQ8Am4Jw9rH8e8IEOjiutKrvPGAwGjbiBzxiAtcDdfdM7gNMXWzHJccAJwLV9sx+TZCuwC7i0qj62xLYbgA0AExMTTE9PD175AGZmZoZew2phW/T8oEmGu77yFaandw65muHz92LeqLVFF8GwL84FPlxVD/fNO66qdiZ5GnBtki9X1R0LN6yqjcBGgMnJyZqamlqRgpcyPT3NsGtYLWyLnu//4GH4xMd52o8+jampE4ddztD5ezFv1Nqii0tJO4F1fdPHNvMWcy4LLiNV1c7m+53ANO3+B2lkzO7uY/BakkZbF8FwPXBSkhOSHErvP/9H3F2U5F8DRwKf7Zt3ZJLDms9HA88Hti3cVhoFTS7Y+ayRN/ClpKraleQi4GpgDXBlVd2S5BJga1XNhcS5wKaau3Wj5+nAu5LM0gupS/vvZpJGydwZg7eratR10sdQVVuALQvmvXnB9FsX2e4zwLO6qEEatrm/eLySpFHnk89SR2q2990hMTTqDAapI4UPuGk8GAxSR+aHxJBGm8EgdWT37aqeMmjEGQxSR2r3kBgGg0abwSB1pHbfriqNNoNB6sjc7ao++axRZzBIHdn9gJu5oBFnMEgdcUgMjQuDQeqIQ2JoXBgMUkfKF/VoTBgMUkfmLyWZDBptBoPUETufNS4MBqkj3q6qcWEwSB3xjEHjwmCQOrL7yWeTQSPOYJA6Uo6uqjFhMEgdmfWuJI2JToIhyfoktybZnuTiRZZfkOSfk9zYfF3Yt+z8JLc3X+d3UY80DL6oR+Ni4Hc+J1kDvAN4MbADuD7J5qratmDVD1bVRQu2PQp4CzBJ76aOG5ptHxi0Lmmlze5+tedw65AG1cUZw2nA9qq6s6oeAjYB5yxz258Grqmq+5swuAZY30FN0oqbO2Ow81mjbuAzBmAtcHff9A7g9EXW+w9Jfgq4Dfj1qrp7iW3XLnaQJBuADQATExNMT08PXvkAZmZmhl7DamFb9Nz1rYcBuOXmmznsn/9hyNUMn78X80atLboIhuX4S+ADVfVgkv8CXAWcuS87qKqNwEaAycnJmpqa6rzIfTE9Pc2wa1gtbIuem3Z8Ez77tzz7Wc9i6uSJYZczdP5ezBu1tujiUtJOYF3f9LHNvN2q6r6qerCZvAI4dbnbSqNi91hJ3uunEdfFr/D1wElJTkhyKHAusLl/hSTH9E2eDfx98/lq4KwkRyY5EjirmSeNHIfd1rgY+FJSVe1KchG9/9DXAFdW1S1JLgG2VtVm4FeTnA3sAu4HLmi2vT/J79ALF4BLqur+QWuShmFurCT7njXqOuljqKotwJYF897c9/mNwBuX2PZK4Mou6pCGySExNC68Gip1xFd7alwYDFJHZnePlWQyaLQZDFJH5jqfPWPQqDMYpI7Mv/PZZNBoMxikjpQv6tGYMBikjvhqT40Lg0HqiK/21LgwGKSOeLuqxoXBIHVk7ozBl3tq1BkMUkc8Y9C4MBikjsy/2tNk0GgzGKSO+GpPjQuDQeqIt6tqXBgMUkfmO5+l0WYwSB2Z73z2jEGjzWCQOuKQGBoXBoPUkVnPGDQmDAapI/O3qw65EGlAnQRDkvVJbk2yPcnFiyz/jSTbktyU5FNJjutb9nCSG5uvzV3UIw3D7hf1GAwacQO/8znJGuAdwIuBHcD1STZX1ba+1f4OmKyq7yb5FeDtwCuaZd+rqlMGrUMaNt/5rHHRxRnDacD2qrqzqh4CNgHn9K9QVddV1Xebyc8Bx3ZwXGlV2f2inuGWIQ1s4DMGYC1wd9/0DuD0Paz/auCv+6Yfk2QrsAu4tKo+tthGSTYAGwAmJiaYnp4epOaBzczMDL2G1cK26Nn2tV0AXP+FL/DVH7b7zt+LeaPWFl0Ew7Il+QVgEnhB3+zjqmpnkqcB1yb5clXdsXDbqtoIbASYnJysqamplSh5SdPT0wy7htXCtui574YdcNOXeN7zTue4J/zwsMsZOn8v5o1aW3TxZ81OYF3f9LHNvJYkLwLeBJxdVQ/Oza+qnc33O4Fp4Lkd1CStuLknn71dVaOui2C4HjgpyQlJDgXOBVp3FyV5LvAueqFwb9/8I5Mc1nw+Gng+0N9pLY2M3W9jMBc04ga+lFRVu5JcBFwNrAGurKpbklwCbK2qzcAfAocDH2ru2PhqVZ0NPB14V5JZeiF16YK7maSR4V1JGhed9DFU1RZgy4J5b+77/KIltvsM8KwuapCGzRf1aFx464TUkd0PuHnDqkacwSB1xCExNC4MBqkjs7t7n4dahjQwg0HqSHm7qsaEwSB1xBf1aFwYDFJH5h5wMxY06gwGqSOeMWhcGAxSR2YdXlVjwmCQOuIDbhoXBoPUkbnnGBwSQ6POYJA6MusZg8aEwSB1xM5njQuDQerI7s5nacQZDFLHPGPQqDMYpI7Mzs51Pg+5EGlABoPUkVn7GDQmDAapIw67rXFhMEgd2f2iHs8YNOI6CYYk65PcmmR7kosXWX5Ykg82yz+f5Pi+ZW9s5t+a5Ke7qEcaiipHw9BYGDgYkqwB3gG8BDgZOC/JyQtWezXwQFWdCFwG/EGz7cnAucAzgPXAnzT7k0bOrHerakwc3ME+TgO2V9WdAEk2AecA2/rWOQd4a/P5w8Dl6Z1vnwNsqqoHga8k2d7s77N7OuC3v/cDPnHL1zsoff/d/E+7eGjINawWtkXPnd+YsX9BY6GLYFgL3N03vQM4fal1qmpXkm8BT2jmf27BtmsXO0iSDcAGgEOffCIb3ntDB6UP6O9WQQ2rhW0BwOGHFNPT08MuY1WYmZmxLRqj1hZdBMOKqKqNwEaAZzznufXB1/7kUOu54YatnHrq5FBrWC1si3l33PxFpqamhl3GqjA9PW1bNEatLboIhp3Aur7pY5t5i62zI8nBwI8A9y1z20d47CFreObaHxmk5oF94/bh17Ba2BbzvnG715I0+rq4K+l64KQkJyQ5lF5n8uYF62wGzm8+vxy4tnpvTt8MnNvctXQCcBLwhQ5qkiTtp4HPGJo+g4uAq4E1wJVVdUuSS4CtVbUZ+FPgvU3n8v30woNmvT+n11G9C3hNVT08aE2SpP3XSR9DVW0BtiyY9+a+z98Hfm6Jbd8GvK2LOiRJg/PJZ0lSi8EgSWoxGCRJLQaDJKnFYJAktRgMkqQWg0GS1GIwSJJaDAZJUovBIElqMRgkSS0GgySpxWCQJLUYDJKkFoNBktRiMEiSWgwGSVKLwSBJahkoGJIcleSaJLc3349cZJ1Tknw2yS1Jbkryir5l707ylSQ3Nl+nDFKPJGlwg54xXAx8qqpOAj7VTC/0XeBVVfUMYD3wR0mO6Fv+X6vqlObrxgHrkSQNaNBgOAe4qvl8FfCzC1eoqtuq6vbm89eAe4EnDnhcSdKjZNBgmKiqe5rPXwcm9rRyktOAQ4E7+ma/rbnEdFmSwwasR5I0oFTVnldIPgk8eZFFbwKuqqoj+tZ9oKoe0c/QLDsGmAbOr6rP9c37Or2w2AjcUVWXLLH9BmADwMTExKmbNm3a80/2KJuZmeHwww8fag2rhW0xz7aYZ1vMWy1tccYZZ9xQVZN7XbGq9vsLuBU4pvl8DHDrEus9Hvgi8PI97GsK+KvlHPfUU0+tYbvuuuuGXcKqYVvMsy3m2RbzVktbAFtrGf/HDnopaTNwfvP5fOAvFq6Q5FDgo8B7qurDC5Yd03wPvf6JmwesR5I0oEGD4VLgxUluB17UTJNkMskVzTo/D/wUcMEit6W+P8mXgS8DRwO/O2A9kqQBHTzIxlV1H/DCReZvBS5sPr8PeN8S2585yPElSd3zyWdJUovBIElqMRgkSS0GgySpxWCQJLUYDJKkFoNBktRiMEiSWgwGSVKLwSBJajEYJEktBoMkqcVgkCS1GAySpBaDQZLUYjBIkloMBklSi8EgSWoxGCRJLQMFQ5KjklyT5Pbm+5FLrPdwkhubr819809I8vkk25N8MMmhg9QjSRrcoGcMFwOfqqqTgE8104v5XlWd0nyd3Tf/D4DLqupE4AHg1QPWI0ka0KDBcA5wVfP5KuBnl7thkgBnAh/en+0lSY+OgwfcfqKq7mk+fx2YWGK9xyTZCuwCLq2qjwFPAL5ZVbuadXYAa5c6UJINwIZmcibJrQPWPqijgW8MuYbVwraYZ1vMsy3mrZa2OG45K+01GJJ8EnjyIove1D9RVZWkliqmqnYmeRpwbZIvA99aToF9+98IbNyXbR5NSbZW1eSw61gNbIt5tsU822LeqLXFXoOhql601LIk/5TkmKq6J8kxwL1L7GNn8/3OJNPAc4H/AxyR5ODmrOFYYOd+/AySpA4N2sewGTi/+Xw+8BcLV0hyZJLDms9HA88HtlVVAdcBL9/T9pKklTVoMFwKvDjJ7cCLmmmSTCa5olnn6cDWJF+iFwSXVtW2ZtkbgN9Isp1en8OfDljPSlo1l7VWAdtinm0xz7aYN1Jtkd4f7pIk9fjksySpxWCQJLUYDB1I8ptJqulcPyAl+cMk/5DkpiQfTXLEsGtaaUnWJ7m1GeJlqVEAxl6SdUmuS7ItyS1Jfm3YNQ1TkjVJ/i7JXw27luUyGAaUZB1wFvDVYdcyZNcAz6yqZwO3AW8ccj0rKska4B3AS4CTgfOSnDzcqoZmF/CbVXUy8DzgNQdwWwD8GvD3wy5iXxgMg7sM+C3ggO7Fr6pP9D3F/jl6z6UcSE4DtlfVnVX1ELCJ3pAxB5yquqeqvth8/g69/xSXHNVgnCU5FvgZ4Iq9rbuaGAwDSHIOsLOqvjTsWlaZXwT+ethFrLC1wN1903sc4uVAkeR4eg+0fn64lQzNH9H7w3F22IXsi0HHShp7exkS5LfpXUY6IOypLarqL5p13kTvUsL7V7I2rT5JDqc3wsHrqurbw65npSV5GXBvVd2QZGrY9ewLg2EvlhoSJMmzgBOAL/UGiuVY4ItJTquqr69giStmT8OjACS5AHgZ8MI68B6Q2Qms65s+oId4SXIIvVB4f1V9ZNj1DMnzgbOTvBR4DPD4JO+rql8Ycl175QNuHUlyFzBZVathBMUVl2Q98D+BF1TVPw+7npWW5GB6ne4vpBcI1wP/sapuGWphQ9AMqX8VcH9VvW7Y9awGzRnD66vqZcOuZTnsY1BXLgceB1zTvKnvncMuaCU1He8XAVfT62z98wMxFBrPB14JnNn35saXDrsoLZ9nDJKkFs8YJEktBoMkqcVgkCS1GAySpBaDQZLUYjBIkloMBklSi8EgdSDJL/c9zPWVJNcNuyZpf/mAm9ShZoyga4G3V9VfDrseaX94xiB1638B1xoKGmWOrip1pBld9jh6YyZJI8tLSVIHkpxKb0TRf1tVDwy7HmkQXkqSunERcBRwXdMBPVKvcpT6ecYgSWrxjEGS1GIwSJJaDAZJUovBIElqMRgkSS0GgySpxWCQJLX8f0kuJFspbuVaAAAAAElFTkSuQmCC\n",
+      "text/plain": [
+       "
"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    },
+    {
+     "data": {
+      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYYAAAEWCAYAAABi5jCmAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3Xd83Hd9+PHXW9vW3h7y0PB2bMdLnrETZ5lAAhRKCCMp0ABtaEspJYEWKC39hZaWQqGFNIQEAhkEEpzpOLZlJ952vLckL8lD27Yka929f3/oZGRH+8b3xvv5eNzDd/dd7691d+/vZ35FVTHGGGO6RDkdgDHGmOBiicEYY8w1LDEYY4y5hiUGY4wx17DEYIwx5hqWGIwxxlzDEoMJKyLydRF53E/7/qKIXBCRRhHJ9Mcxejmu387JmJ6IjWMwpn8iEgtcAhao6l4/Hmc58LSq5vnrGMb0x0oMxgxMLpAAHHQ6EGP8zRKDCUki8jURqRSRyyJyVERWeN7/tog87Xk+XkRURO4XkdMiUiMi3+i2jygReVhEykSkVkSeF5GMHo41ETjqedkgIuu67Tum23olIvI5z/MHROQdEfm+iNSLyAkRWdlt3QwR+YWInPUsf0lEEoHXgVGe6qpGERnV/Zw8294tIgdFpMFzzCndlp0Ukb8TkX0iclFEnhORBF/9v5vIYInBhBwRmQQ8BMxT1WTgDuBkH5ssASYBK4Bvdvsh/RLwQWAZMAqoB35y/caqegyY5nmZpqq3DDDUYjoTShbwb8DPRUQ8y34FDPfsNwf4gao2ASuBs6qa5Hmcve7cJwLPAH8DZAOvAS+LSFy31f4UuBPIB2YADwwwXmMASwwmNLmAeGCqiMSq6klVLetj/X9S1SuetoG9wEzP+18AvqGqFaraCnwb+Ej3UoCXTqnq/6mqC3gKGAnkishIOhPAF1S1XlXbVXXDAPf5MeBVVV2jqu3A94FhwKJu6/xIVc+qah3wMjDLR+djIoQlBhNyVLWUzivmbwNVIvKsiIzqY5Pz3Z43A0me5+OAFz1VMg3AYTqTTq6PQr16XFVt9jxNAsYAdapaP4R9jgJOdduvGzgDjO7puFx7vsYMiCUGE5JU9TequoTOH3cFvjeE3ZwBVqpqWrdHgqpWDmDbJs+/w7u9N2IQx80QkbQelvXXTfAsnecMgKdqagwwkJiNGRBLDCbkiMgkEblFROKBFuAK4B7Crn4KfFdExnn2my0i9wxkQ1WtpvPH+JMiEi0inwEKB7jtOTobmf9HRNJFJFZEbvIsvgBkikhqL5s/D9wlIis8XWi/ArQCmwdybGMGwhKDCUXxwKNADZ3VJjnAI0PYzw+BVcCbInIZ2Epng/FA/TnwVaCWzkbkwfw4fwpoB44AVXRWjaGqR+hsXC73VHFdU0WmqkeBTwL/Tef5fwD4gKq2DeLYxvTJBrgZY4y5hpUYjDHGXMMniUFEnhCRKhE50MtyEZEfiUipZ+DN7G7L7heR457H/b6IxxhjzND5qsTwJJ0DanqzEpjgeTwI/C90jv4EvkVnve584Fsiku6jmIwxxgyBTxKDqm4E6vpY5R7gl9ppK5DmGeRzB7BGVbv6dK+h7wRjjDHGz3w1wrM/o+nsu92lwvNeb++/h4g8SGdpg2HDhs0ZM2ZMjwdyu91ERQW+6cSl0NCqNLbpezqiJ0RDcpyQGCs9bjsQTp2XP9k5BYYCjW3K5Tal7bpOvTFRkBwrJMUJ0X18PIPxvLwVied07NixGlXN7m8/gUoMXlPVx4DHAObOnas7d+7scb2SkhKWL18eyLj473Wl/HhdKWmqfHZ2HssnZZOZFA/AuiNVvH7gHKdqm7ltxkj+9cM3kJIQO+jjBPq8AsHOyf+OXbjM557ayem6ZpaMTOFTC8YxPms4sdFRnKlr5rkdZ9h2oo60pDh+ct9sigt6vs1EsJ2XL0TiOYnIqV4XdhOoxFBJ5+jMLnme9yqB5de9XxKgmLzW7nLzjRf38/zOCj4wcxRfu3MSeenDr1lnfn4GX71jEj/bWMZ/vHmMPWca+Okn5zB9dG/jl4zxjbWHL/BXz+xmeHwMT3+2mMVFmfxxDj+YNz6DD8/O49DZSzz0m3f5xOPb+MZdU3hg0fhr1jORJ1DlqFXApz29kxYAFz2jP1cDt3tGf6YDt3veC3ot7S4e/OVOnt9ZwV+tmMCP7p31nqTQJTpK+IvlRfz2Cwtxu5X7n9jOyZqmHtc1xhde2FXB5365k/zsRFY9tJglE7J6/bGfOiqFlx5azPJJOfzTy4f44drjAY7WBBtfdVd9BtgCTBKRChH5rIh8QUS+4FnlNaAcKAX+D/gLAM/sj/8M7PA8vuN5L6ipKl///X5KjlXz3Q9N529vmzigK6zZY9P51eeKcaty/y+2U9PYGoBoTaTZebKOR36/j0WFmfz284sYmTqs321SEmJ57FNz+JPZefzXW8d5ee/Zfrcx4csnVUmq+vF+livwl70sewJ4whdxBMrT207z+92VfPnWiXyieFz/G3RTmJ3Ezx+Yx33/t5XPPrmD5z6/kITYaD9FaiJNZcMVvvD0LkanDeMn981mWNzAP1tRUcK/fng6Z+qa+bvf7iUvfRg3jrXe45EovJrkA2D36Xq+8/JBbp6UzZduKRrSPmaPTedH997I3oqL/OeaYz6O0ESq1g4Xn//VTlra3Tx+/1zShsf1v9F14mOi+emn5pCTEs+Dv9pFfZNNwRSJLDEMQlNrBw/9Zje5KQn84GOziIoaegPd7dNGcF/xWB5/u5x3Tw9lWn5jrvXTknIOVF7iP/90JkU5yUPeT0ZiHD/75Fzqm9r47muHfRihCRWWGAbhv9eVUtlwhR/eO2tIV2PXe2TlZEamDuOrv91LS7vLBxGaSFVe3chP1pfygZmjuH3aQG8L0bupo1L4/LICXthVwabSGh9EaEKJJYYBKq26zONvl/PROXnMGfee+8UPSXJCLP/vwzdQVt3Ef71lPUHM0Kgq//DSAeJjo/jH90/pf4MB+tItExifOZyvv7ifNpfNwhxJLDEMgKryrVUHGR4XzddWTvbpvm+amM1H5+Tx+NvlnK5t7n8DY67z4u5KNpfV8rU7J5OTnOCz/SbERvOvH76BU7XNrCpr99l+TfCzxDAAr+0/z6bSWv7ujklkeUY0+9Lf3TGJ6CjhP9cc9fm+TXhraXfx6OtHmDUmjfvmj/X5/hcVZvGhG0ez+mQ7VZdafL5/E5wsMfTD5Va+/+ZRJo9IHnTX1IHKTUngzxbn84e9Zzl09pJfjmHC06+3nabqcisPr5zsVWeIvvzNrRNwK/xkfalf9m+CjyWGfryy7ywnapr4m1snEu2nLx7AF5cVkhwfw/fftFKDGZjmtg7+t6SURYWZLOhljiNfGJeZyNLRMfxm+2kq6q26MxJYYuiDy905Qd7kEcncPjXXr8dKHR7LF5cXse5IFdtPBP3gbxMEnt56iprGNr5820S/H+sDhbEIwo/XWakhElhi6MPrB85RWtXIl26Z4LdiencPLBpPVlK8FdlNv5paO/jphnKWTshi3njf9JLrS+awKO4rHstvd1XYPF8RwBJDL9xu5UdrjzMhJ4mV073vFz4Qw+KieWDRODYcq+bYhcsBOaYJTb/Zdpq6psCUFrr8xc2FREcJj79THrBjGmdYYujF2iNVHLvQyEO3FAWktNDlE8XjSIiN4udvnwjYMU1ocbmVJzefZH5+BrMDOJdRTnICH5w1it/tquRis3VfDWeWGHrx1OaTjEpN4K4bRgb0uOmJcXxkTh4v7q6k+rLNvmrea82hC1Q2XOEzi8cH/Nh/tjifK+0unt1xOuDHNoFjiaEHpVWXeae0hk8sGEdMdOD/iz6zOJ92t5tfbR3QzZZMhHly8wlGpw3j1in+7RDRkykjU1hYkMlTm0/S4XL3v4EJSZYYevCrLaeIi47iY/N6vq+0vxVkJ7Fici5Pbz1lcyiZaxw+d4mt5XV8eqEzFy0An1mSz9mLLaw+eMGR4xv/s8RwncbWDn73biXvnzHSL6OcB+qzS/Kpa2rjtf3nHIvBBJ8nN50kIda5ixaAWybnMC5zOE9ssnawcGWJ4TovvltBY2sHn1403tE4FhRkkJ+VyLM7zjgahwkeDc1tvLSnkg/PzvPJ7L5DFR0lfHrheHadqufwORupH44sMXSjqvxyyylm5qUya0yao7GICH86dwzbT9RRXt3oaCwmOPxhz1laO9x8otj3cyIN1odvHE1cdBS/3VnhdCjGD3x1z+c7ReSoiJSKyMM9LP+BiOzxPI6JSEO3Za5uy1b5Ip6h2n2mgeNVjdwXBF88gD+ZM5roKOG5nVZqMPDbXWeYOjKFaaNSnQ6F9MQ4bpuay4u7K2jrsEbocON1YhCRaOAnwEpgKvBxEZnafR1V/bKqzlLVWcB/A7/vtvhK1zJVvdvbeLzxu10VJMRG8b4Ad1HtTU5yAism5/C7XRV0uG0+/Eh2+NwlDlRe4qNz85wO5aqPzs2jvrmdtYetETrc+KLEMB8oVdVyVW0DngXu6WP9jwPP+OC4PtXS7uLlvWe5Y9oIkhNinQ7nqnvnj6GmsY291dY7KZL9dmcFsdHCPbNGOx3KVUsnZDMiJYHnrUQbdnyRGEYD3T8ZFZ733kNExgH5wLpubyeIyE4R2SoiH/RBPEOy9nAVl1o6+JPZwXNFBnCT58u3oaLD6VCMQ9o63Ly0p5Jbp+SSkehco/P1oqOEj8zJY8Oxas5ftHs1hJOYAB/vXuAFVe1++TtOVStFpABYJyL7VbXs+g1F5EHgQYDc3FxKSkp6PEBjY2Ovy/ry2K4W0uOFjsoDlJwN3BQYAzE3y8Vr5R2sWr2elPjgis0bQ/1bBTN/nNOuCx3UNbUxOb7esf+v3s5rTIcbt8L3X9jI+wuDJ2kNhH3++qCqXj2AhcDqbq8fAR7pZd3dwKI+9vUk8JH+jjlnzhztzfr163td1puqSy1a8Mir+v9eOzzobQPh8LmLOu5rr+hTm084HYpPDeVvFez8cU6ffXKHzvuXNdre4fL5vgeqr/P66P9u1hX/UaJutztwAflAJH7+gJ06gN91X1Ql7QAmiEi+iMTRWSp4T+8iEZkMpANbur2XLiLxnudZwGLgkA9iGpQ/7KnE5VY+Mid46m+7mzwihbwkYdWes06HYgLsYnM7G45VcffMUY6NdO7PB2aNorSqkaM2I3DY8PqTpqodwEPAauAw8LyqHhSR74hI915G9wLPerJWlynAThHZC6wHHlXVgCeGl/ee5YbRqRTlJAf60ANWPDKGnafq7Q5aEWb1ofO0u5QPzBzldCi9Wjl9BNFRwst77cIlXPjkEkRVX1PViapaqKrf9bz3TVVd1W2db6vqw9dtt1lVb1DVmZ5/f+6LeAbjTF0zeysu8v4ZwdFFtTfFIzubg17ea1NkRJJX9p1jbMZwZuQ5P3ahN1lJ8SwqzOTlvee49rrPhKrgLJsG0KueuYiCZexCb3KGRzFrTBqr7KosYtQ1tbGptIa7ZoxEJLg7HXxgxihO1zWzv/Ki06EYH7DEsO8cM8ekMSZjuNOh9OueWaM4fO4Sx60uNyK8ceA8LrcGfWkW4I5pI4iNtuqkcBHRieF0becVzvuDvLTQ5a4ZI4kS7MsXIV7ee5aCrESmjkxxOpR+pQ6PZdnEbF7Zdw63jdIPeRGdGLqqkVbeEJh7OnsrJzmBeeMzeOPgeadDMX5WdbmFbSdqeX8IVCN1+cDMUZy72MKu0/VOh2K8FOGJ4SyzxqSRlx781Uhd7pg2gmMXGm3G1TD3xoHzuBXeH8S9ka63YkoucTFRvHHALlxCXcQmhlO1TRyovBQS9bfd3TG9s3Rjd88Kb28evEBBdiITc4O3C/X1kuJjWFKUxeqD5613UoiL2MSw2lMdc+f00KhG6jI6bRg3jE69Gr8JPxeb29laXsvtU0Prswlw+9RcKuqvcPicdZAIZRGbGNYcusDUkSkhVY3U5c7pI9hzpsEmLgtT649W0eFWbp+W63Qog3br1FxE4M1DduESyiIyMdQ2trLrVD23TQ29Lx7AHZ4fDPvyhac1hy6QnRzPrDxn7yI4FFlJ8cwdl25VnSEuIhPD2iNVuJWQTQxFOckUZidadVIYaml3UXK0itum5hIVFRq9ka53x7QRHD53iTN1Nn1LqIrIxLDm0AVGpSYwbVTw9w/vzR3TRrC1vI76pjanQzE+tKWslqY2F7eH6EUL/PGC681DVmoIVRGXGK60uXj7eLWnLjQ0r8gAbp82Apdb2XCs2ulQjA+9eeg8SfExLCzMdDqUIRuXmcjkEclWog1hEZcY3imtoaXdHbLVSF1mjE4lKymOtUeqnA7F+Ijbraw5VMXySdnEx0Q7HY5Xbp+ay86TddRZiTYkRVxiWHPoPMnxMRTnh+4VGUBUlHDzpBw2HK2iw+V2OhzjA3srGqhpbA35ixaAW6bk4lbYaCXakBRRicHtVtYdqWLZpGziYkL/1G+ZnMOllg52nbIpCMLB+qPVRAksm5jtdChe6yrRrrMSbUgK/V/HQdhfeZGaxjZWTMlxOhSfWDIhi9hosS9fmCg5WsXssemkDQ+teyf3JCpKWDYxhw3HqnHZpHohJ6ISQ8nRakTgpgmhf0UGkJwQy/z8DEsMYaDqcgv7Ki5y8+TwuGiBzhLtxSvt7LZJ9UJORCWG9UermJmXRmZSvNOh+Mwtk3M5XtVofcZD3IajnXXxN08Kn8SwdGIW0VFWog1FEZMYahtb2VvRwPJJ4VFa6LLCc4VpX77Qtv5oFbkp8UwZGTqT5vUnJSGWuePS7bMZgnySGETkThE5KiKlIvJwD8sfEJFqEdnjeXyu27L7ReS453G/L+LpydvHa1ANrysygPFZiRRkJVq31RDW7nLz9rEabp6UE9Jja3pyy+Qcjpy/zNmGK06HYgbB68QgItHAT4CVwFTg4yIytYdVn1PVWZ7H455tM4BvAcXAfOBbIpLubUw9WX+0iszEOG4YHbw3VR+q5ZNy2Fpey5U2l9OhmCHYdaqey60dYdW+0OUWzzmVHLVuq6HEFyWG+UCpqparahvwLHDPALe9A1ijqnWqWg+sAe70QUzXcLmVjceqWTYxO2Tnn+nLsknZtHW42Xai1ulQzBCsP1JFbLSwuCjL6VB8rignibz0YVadFGJifLCP0cCZbq8r6CwBXO9PROQm4BjwZVU908u2o3s6iIg8CDwIkJubS0lJSY/BNDY2vmdZaYOL+uZ2ctw1vW4X7Ho6ry5tLiU2Cn6zbjecC52G9b7OKVQN5ZxefbeZCWnCzi3v+CcoH/DmbzUhqZ13jl3grXXriQmiCzP7/PXOF4lhIF4GnlHVVhH5PPAUcMtgdqCqjwGPAcydO1eXL1/e43olJSVcv+zdNceIkuN84Z6bQraPeE/n1d2Ck9sov9jC8uXLAheUl/o7p1A02HO6cKmFijfW8sjSSSxfVui/wLzkzd+qJes865/eRfL4GRQXBM+MA/b5650vqpIqgTHdXud53rtKVWtVtdXz8nFgzkC39YWNx6qZOSYtZJPCQCybmE1pVSOV1sgXUt4+XgPA0jAZW9OTRUWZREfJ1XM1wc8XiWEHMEFE8kUkDrgXWNV9BRHpfmPlu4HDnuergdtFJN3T6Hy75z2fudjczr6KhrAZ1NabmzzTKNjcNKFl47FqspLCq5vq9VISYrlxTBobj9tnM1R4nRhUtQN4iM4f9MPA86p6UES+IyJ3e1b7KxE5KCJ7gb8CHvBsWwf8M53JZQfwHc97PrO5rAa3wtIJ4dew192EnCRGpCRYYgghbrfyTmkNN03ICrtuqte7aWI2+ysv2myrIcIn4xhU9TVVnaiqhar6Xc9731TVVZ7nj6jqNFWdqao3q+qRbts+oapFnscvfBFPdxuP15AcH8PMMaF3m8TBEBGWTczmndIam201RBw6d4m6pjaWTgzvixbovDBT7Zz23gS/sB75rKq8fbyahYWZxEaH9akCnVdll1s62HOmwelQzAB03WRpSVF4V3MCzMhLI3VYrJVoQ0RY/1qeqm2mov5K2FcjdVlSlEWUWDtDqHj7eDVTR6aQnRw6XYyHKjpKWFKUxdvHq1G12VaDXVgnhrc9jV3h3OOju9ThsdyQl8amMhvoFuyaWjvvoxEJ1UhdbpqYxYVLrRy70Oh0KKYfYZ0YNh6vYUzGMMZlDnc6lIBZUpTJnjMNXG5pdzoU04et5bW0uzTse8t113WB9rb1Tgp6YZsY2l1utpbVsqQoO+x7fHS3uCgLl1vZVu7Tzl3Gx94prSE+Joo54/wyNVhQGpU2jIKsRDZZA3TQC9vEsPdMA5dbOyKmfaHL7LHpJMRGsanMvnzBbHNpLfPzM0iIjXY6lIBaVJTJthN1tFvPuaAWtolhU2ktIrCoMHiG4AdCQmw088Zn2FVZEKu63MLRC5dZVBhZFy3Q2UGiuc1lPeeCXPgmhrIapo9KDetpMHqzpCiLYxcaqbrU4nQopgdbPJ0DFhdF1kULwIKCTESwC5cgF5aJobmtg92n6yOutNCla/pmq04KTptKa0hJiGHaqPC7N0h/0oZ33hPFEkNwC8vEsONkPe0uZVEYzm8/EFNHppA+PJZ3jlu31WCjqmwqrWVRYef9kCPRosIsdp9uoKm1w+lQTC/CMjFsLq0hNlqYNz5yenx0FxUlLCrKYlNpjQ0mCjKn65qpbLgSkdVIXZYUZdHhVrafsJ5zwSosE8OmshpuHJvO8LhA3W4i+CwpyuL8pRbKqpucDsV0s6m0sxQXqaVZgLnj04mLibLqpCAWdomhsU05ePYSiyOwx0d3Xe0rW8qtOimYbCqtYURKAgVZiU6H4piE2Gjmjku3CfWCWNglhiN1LlQjs8dHd2MzhjMqNYEt1gAdNNxuZXNZDYuLwn+a7f4sLsriyPnL1Da29r+yCbiwSwyH6lwMj4sO+2m2+yMiLCzMYmt5HW63tTMEgyPnL1Pf3B6xveW6W+C5xedWG6EflMIvMdS6mJ+fERHTbPdnYWEmdU1tHL1w2elQDH+s1lsU4aVZgBl5qSTGRbOl3Eq0wSisfj0vXGrhfJPaFZnHwq52BpttNShsKatlfOZwRqYOczoUx8VGRzEvP8M+m0EqrBLDVs8V2cKCyG547jI6rXNm2c325XOcy61sO1F7NVkbWFiQSVl1k43QD0I+SQwicqeIHBWRUhF5uIflfysih0Rkn4isFZFx3Za5RGSP57HKmzi2lNUyPAamjkrxZjdhZVFhJttO1OKydgZHHTx7kcstHVfr1k23Eq31nAs6XicGEYkGfgKsBKYCHxeRqdetthuYq6ozgBeAf+u27IqqzvI87vYmli3ltUzKiI7YEaU9WVCQyeWWDg6eveh0KBGtq8pkoSWGq6aNSiU5Icaqk4KQL0oM84FSVS1X1TbgWeCe7iuo6npVbfa83Ark+eC41zjbcIVTtc1MzoisaYz70/VDZNVJztpSXkthdiI5KQlOhxI0oqOE4vxMKzEEIV8MDR4NnOn2ugIo7mP9zwKvd3udICI7gQ7gUVV9qaeNRORB4EGA3NxcSkpKrlm+qbLzjmXjElrfsywcNDY2Dvm8RiUKr+w4zmQ90//KAeTNOQWrns6pw61sLW1m0aiYkD1ff/2tsrWdt2rb+N3r68gcFtgmz0j5/A1FQOeMEJFPAnOBZd3eHqeqlSJSAKwTkf2qWnb9tqr6GPAYwNy5c3X58uXXLH/1t3tJHXaBibmxXL8sHJSUlAz5vFY0HOB371aweOlNQdWN15tzClY9ndO7p+tpeXMzf7J0BstnjHQmMC/562+Vc/YSzxx5G3ImsnyOzysS+hQpn7+h8MWvRCUwptvrPM971xCRW4FvAHer6tXhjqpa6fm3HCgBbhxKEFvKaynOzyAqwkeU9mRBQSbNbS72V1o7gxO66tAXFGQ4HEnwmTwimfThsVadFGR8kRh2ABNEJF9E4oB7gWt6F4nIjcDP6EwKVd3eTxeReM/zLGAxcGiwAZypa6ai/op1BexFsecHaat9+RyxtbyWSbnJZCbFOx1K0InytDPYZzO4eJ0YVLUDeAhYDRwGnlfVgyLyHRHp6mX070AS8NvruqVOAXaKyF5gPZ1tDINODFfHL1hi6FFWUjwTcpLYZtMPBFy7y83Ok/VWWujDgoIMKuqvUFHf3P/KJiB80sagqq8Br1333je7Pb+1l+02Azd4e/yt5XWkD49lYk4y5494u7fwtKAgk9+/W0G7yx1U7Qzhbl/FRa60u2z8Qh+KPf8328rryJsz3OFoDITJyOet5bUU52cSZeMXerWgIJOmNhcHrJ0hoLpKs/PzrcTQm0m5yaQNj7XqpCAS8onhjOeOWFZU79sf2xmsOimQtpbXMjE3ydoX+tDZzpDB1hOWGIJFyCeGbZ7bAxZbUb1PXe0MdlUWOO0uN7tO1Vs10gAU52dypu4KlQ1XnA7FEA6JobyWtOGxTMpNdjqUoLegIJOdJ+tod7mdDiUi7K+8SHObi+J8Swz9WXC1ncEuXIJByCeGrSdqmT8+w9oXBsDaGQKrqxdYsVVz9mvyiGRSh1k7Q7AI6cRQ2XCFM3VXrKg+QNbOEFhby2uZkJNElrUv9CsqSpifn3G1atg4K6QTQ1ex067IBiYrKZ6inCS2WSOf33W43Ow8WWefzUFYUJDJqdpmzlo7g+NCPDHUkToslikj7P4LA1Wcn8HOk/V0WDuDXx04e4mmNhu/MBjFni69duHivJBODFtP1DLP2hcGpbggk8bWDg6du+R0KGHNxi8M3pSRKSQnxNgI/SAQsonh3MXO+y/Y+IXBWdB1VWZfPr/aVl5LQXYiOcl2/4WBio4S5o+3doZgELKJYbvnw2NF9cHJSUkgPyvRiut+5HIrO0/WWzfVISguyOBEjd0H2mkhmxi2lteRnBDDlJHWvjBYxfkZbD9RZ/eB9pNDZy9xubXDSrND0JVMt1qpwVEhmxi2edoX7P7Og1dckMGllg6OnLd2Bn/oKo1ZiWHwpo1KISk+xga6OSwkE0OHWymvbrrai8EMztWrMmtn8IttJ+oYlzmcEanWvjBYMdFRzBmXbu0MDgvJxNDU2gFYj48GcN8TAAAgAElEQVShGpU2jDEZw+yqzA/cquw4WWcXLV4oLsigtKqRmsbW/lc2fhGyiWF4XDTTR6c6HUrIKs7PZPvJOtzWzuBTlY1KQ3O7VSN5oev/bruVGhwTkomhsbWDOePS7YYzXijOz6ChuZ1jVZedDiWsHKlzATYa3xsz8lIZFhttJVoHheQva2uH27qpeqnr/8+uynzraJ2L0WnDyEu3O5ENVay1MzjOJ4lBRO4UkaMiUioiD/ewPF5EnvMs3yYi47ste8Tz/lERuWOgx7Q6XO/kpQ9jVGqCDXTzIVXlaL3LSgs+UJyfwZHzl2lobnM6lIjkdWIQkWjgJ8BKYCrwcRGZet1qnwXqVbUI+AHwPc+2U4F7gWnAncD/ePbXzzFhRl6at6FHNJGu2SxrUbV2Bl8orWrkcptdtPhCsZVoHeWLEsN8oFRVy1W1DXgWuOe6de4BnvI8fwFYISLief9ZVW1V1RNAqWd/fUqMiyEuJiRrwYJKcUEmNY1tlFU3OR1KWLh6N0FrePbazDGpxMVEWXWSQ2J8sI/RwJluryuA4t7WUdUOEbkIZHre33rdtqN7OoiIPAg8CJCaO4aSkpIeg2lsbOx1WSjzx3lJU+cMq0+v3sLyMbE+3fdAhNvf6uU9LaTGKSf2b+ekhNfASyf+VvnJ8Na+UyxNqvLL/sPt8we+OydfJIaAUNXHgMcA5s6dq8uXL+9xvZKSEnpbFsr8cV6qyn/sWUt9bCbLl9/o030PRDj9rVSVv9+0limZcPPNNzsdjs858bd6t/0YP153nNkLFpOS4PsLl3D6/HXx1Tn5oj6mEhjT7XWe570e1xGRGCAVqB3gtsZPrrYzlNdZO4OXTtY2U3W5lUnp/TaRmQFakJ+BW2HXyXqnQ4k4vkgMO4AJIpIvInF0Niavum6dVcD9nucfAdZp5y/RKuBeT6+lfGACsN0HMZkBWpCfwflLLZyua3Y6lJDW1ed+UoYlBl+5cWw6sdHCVpsJOOC8rkrytBk8BKwGooEnVPWgiHwH2Kmqq4CfA78SkVKgjs7kgWe954FDQAfwl6rq8jYmM3BdvT865/dJdDia0LX9RB1ZSXGMTAyvtgUnDYuLZkZemnWpdoBP2hhU9TXgteve+2a35y3AR3vZ9rvAd30Rhxm8CTlJZCTGsa28jj+dO6b/DUyPtp2oY35+BiI2ktyXivMz+NnGcppaO0iMD5km0ZBnfT4jnEjXXbOsuD5UZ+qaqWy4Yt1U/aC4IBOXW3n3tLUzDNUvt5zkneM1g9rGEoOhuCCDivorVDZccTqUkHR1/IKNePa5OePSiY4Sq04aonaXm0dfP8Kbh84PajtLDObqla5NWjY028prSRsey8ScZKdDCTtJ8TFMH51qJdoh2l95keY216BLs5YYDJNHJJM6LNauyoZo24k65o3PIMruJugXC/Iz2HvmIlfarF/KYHV9pwd77xpLDIaoKGGetTMMybmLVzhd12yz/fpRcUEGbS43u62dYdC2nailMDuR7OT4QW1nicEAsKAgg5O1zZy/2OJ0KCGl64rMJs7zn7njM4gS2GrzJg1Kh8vNzpP1Q7poscRggG7tDFZqGJSt5bWkJMQwZWSK06GErZSEWKaNSrU2sEE6dO4Sja0dV8cqDYYlBgPA1FEpJMfHsNXaGQala/xCtLUv+FVxfga7zzTQ0m7tDAO11ZNIFwyhNGuJwQAQHSXMHZ9uJYZBuHCphRM1TTZ+IQAWFGTS1uFmz5kGp0MJGdvK68jPSiQnJWHQ21piMFcVF2RSXt1E1WVrZxiIq1dk1vDsd/PyMxDBes4NkMutbD9ZN+S2L0sM5iq7D/TgbDtRR3J8DFNHWfuCv6UOi2XKiBQr0Q7Q4XOXuNzSMeSLFksM5qrpo1JIjIu+eiVs+ratvJa549OtfSFAFhRk8u7pelo7rJ2hP96OxrfEYK6KiY5i7vgMa4AegKrLLZRVN1k1UgAVF2TQ0u5mX8VFp0MJelvLaxmbMZyRqcOGtL0lBnONhYWZlFY1Un251elQgtr2q1dklhgCpfhqO4OVaPvidivbT9Sx0IvPpiUGc40FBTaeYSC2ltd2zuNj7QsBkzY8jskjUthiiaFPh85d4uKVdhYUDn3QpSUGcw1rZxiYLWW1zBufTky0fYUCaUFBBrtOWTtDX3zRW84+1eYaMdFRzMu3doa+WPuCcxYWZNLS7mbvGWtn6M3W8jrGZw69fQEsMZgeLCiwdoa+dCXNhYWWGAJtvqedwUq0PXOrsv1ErdcXLV4lBhHJEJE1InLc8296D+vMEpEtInJQRPaJyMe6LXtSRE6IyB7PY5Y38Rjf6Gq0si9fz7aW13aOX7D5kQIubXgcU0ak2GezF6cvubnkxfiFLt6WGB4G1qrqBGCt5/X1moFPq+o04E7gv0Qkrdvyr6rqLM9jj5fxGB+YNiqFpPgY+/L1YmtZLfPzM6x9wSELCzOtnaEXR+rcgPej8b39ZN8DPOV5/hTwwetXUNVjqnrc8/wsUAVke3lc40cx0VHMG59uiaEHFy61UF5j7QtOWlCQSWuHmz2nbd6k6x2pc5GflciI1MHPj9RdjJdx5KrqOc/z80BuXyuLyHwgDijr9vZ3ReSbeEocqtpjxbaIPAg8CJCbm0tJSUmPx2hsbOx1WSgL9Hll08b66nZeemMdaQn+uTIOxb/VlrMdAMTWn6Ck5PR7lofiOQ1EMJ1XW7siwDPrdnHldNyQ9xNM5+QLblWO1nUwf2Sr9+elqn0+gLeAAz087gEarlu3vo/9jASOAguue0+AeDpLHN/sLx5VZc6cOdqb9evX97oslAX6vPaeqddxX3tFX9pd4bdjhOLf6uHf7dXp33pDO1zuHpeH4jkNRLCd110/2qgf+9lmr/YRbOfkrX1nGvr9zgI7dQC/sf1eCqrqrao6vYfHH4ALIjISwPNvVU/7EJEU4FXgG6q6tdu+z3nibQV+AcwfXFoz/jJtVCrJCTFsKbPqpO62lNVSbPdfcNyC/EzePW33Z+huc1kNgFcjnrt4W0ewCrjf8/x+4A/XryAiccCLwC9V9YXrlnUlFaGzfeKAl/EYH4mOEhYUZNoo027OXbzCyVq7v3MwWFTUeX+Gd0/ZfaC7bC6rZVSiDOn+C9fzNjE8CtwmIseBWz2vEZG5IvK4Z50/BW4CHuihW+qvRWQ/sB/IAv7Fy3iMDy0qzORUbTMV9c1OhxIUNpd2JslFhVkOR2Lmje8stW22Ei0AbR1udpysY0pmtE/251Xjs6rWAit6eH8n8DnP86eBp3vZ/hZvjm/8q+sHcEtZLR+dO9zhaJy3qayGjMQ4Jo9IdjqUiJecEMuMvFRP9ckkp8Nx3L6KBprbXEzJiPfJ/qwjtunVxNwkMhPjrJ2Bzk4aW8pqWViQSZS1LwSFxYVZ7K24yOWWdqdDcdzmslpEYHKGb0oMlhhMr0SEBYWZbC6r7epFFrFO1DRx7mKLTYMRRBYVZuJyKztO2rxem8tqmDoyhaQ431y0WGIwfVpUmMl5z03vI1lXXfbiImtfCBazx6UTFxN1te0nUrW0u3j3VAOLfHjRYonB9KmrnSHSG/k2l9UwMjWB8ZnW1hIsEmKjmTM2nU0R/tncdaqeNpfbp50iLDGYPnVO35sQ0e0MbrenfaEwk86e1SZYLCrM5PC5S9Q1tTkdimM2l9UQHSXMyx/6jXmuZ4nB9ElEWFiYyeayGtzuyGxnOHL+MvXN7Sy2bqpBZ5Gnai+S5/XaXFbLzLxUkuK9neHojywxmH4tKcqivrmdQ+cuOR2KI7pGlC4qsobnYDMjL5XEuGg2ldY4HYojLrW0s/dMg8/H1lhiMP1a4rkqeydCv3ybSmvIz0r06o5Yxj9io6MoLsiM2MSwpawWt8KSCZYYTIDlpCQwMTcpIr98bR1utp2oY7GVFoLWkqIsTtY2c6Yu8kbobyqtYXhcNLPHvuceaV6xxGAGZHFRFttP1EXcpGXvnq6nuc3F0gl2C5FgtXRC5JZo3zleQ3F+BnExvv0pt8RgBmTphCxaO9zsirBJy9453tnjwwa2Ba+inCRyU+IjLjFUNlyhvKbJL2NrLDGYAZmfn0lMlETcl+/t0hpm5qWSkhDrdCimFyLC4qIsNpdGVs+5Tcc7v4v+KM1aYjADkhQfw+yx6RHVznCxuZ39FQ0ssWqkoLd0QmfPuYNnI6fn3NulNWQnxzMxN8nn+7bEYAZscVEW+ysvUh8hg4k2l9Xg1j/WYZvg1VWd8nZptcORBIbbrWwurWFJUZZfBl1aYjADtmRCJqqRMz3G26U1JMXHMGtMmtOhmH7kJCcweUQy7xyPjBLt4fOXqG1qu9qV3NcsMZgBm5mXRnJ8DO9EyFXZO8drWFCQQWy0fU1CwZKiLHaerOdKW/j3nOtKgL4ev9DFPvFmwGKio1hclMWGo9VhPw33qdomTtc1WzfVELJkQhZtLjfbI2Aa7o3Hq5mYm0SuD27j2RNLDGZQlk3K5uzFFkqrGp0Oxa/e9vMVmfG94vxM4mKi2HgsvEu0Ta0d7DhRz/JJOX47hleJQUQyRGSNiBz3/Nvj8DsRcXW73/Oqbu/ni8g2ESkVkedEJM6beIz/3TSx8wp6Q5h/+UqOVpOXPoyCrESnQzEDNCwumuL8DEqOVjkdil9tLa+lzeVm2UT/lWa9LTE8DKxV1QnAWs/rnlxR1Vmex93d3v8e8ANVLQLqgc96GY/xs9Fpw5iQkxTWiaG1w8XmshqWTcy2abZDzPJJOZRVN4X19BgbjlUzLDaaueN9Ow1Gd94mhnuApzzPnwI+ONANpfMbdwvwwlC2N85ZNjGbbSfqwraRb9fJzmkw/FlUN/6xLAJKtBuOVbOoMJP4GN/c37kn4k0joog0qGqa57kA9V2vr1uvA9gDdACPqupLIpIFbPWUFhCRMcDrqjq9l2M9CDwIkJubO+fZZ5/tMabGxkaSknw/4MNpwXReB2pcfH9nC387J54Z2UOfAz6Yzqm7Z4+08dapdn68YjgJMYMrMQTrOXkrVM5LVfnqxiuMSY7ir2f33TAbKufU3YUmN197+wqfmhrHirHvHY3f3zndfPPNu1R1br8HUtU+H8BbwIEeHvcADdetW9/LPkZ7/i0ATgKFQBZQ2m2dMcCB/uJRVebMmaO9Wb9+fa/LQlkwndeVtg6d9A+v6bf+cMCr/QTTOXV323+W6H3/t2VI2wbrOXkrlM7rGy/u06n/+Lq2trv6XC+UzqnLk5tO6LivvaKnapp6XN7fOQE7dQC/sf1WJanqrao6vYfHH4ALIjISwPNvj60+qlrp+bccKAFuBGqBNBHpuuTMAyr7zWTGcQmx0SwoyGTj8fArrp9tuMKxC40sn2jVSKFq2cQcmtpc7DwVft1WNxyrJj8rkbF+vve4t20Mq4D7Pc/vB/5w/Qoiki4i8Z7nWcBi4JAne60HPtLX9iY4LZuYTXl1E6drw6uRr+RoZ7JbPsnGL4SqRYWZxEYLG46G14VLS/sfO0X4m7eJ4VHgNhE5DtzqeY2IzBWRxz3rTAF2isheOhPBo6p6yLPsa8DfikgpkAn83Mt4TIDcMrnzinrdkQsOR+JbJUerGJ02jKKc0Kp7Nn+UGB/DvPEZYdcAve1EHS3t/u2m2sWru0erai2woof3dwKf8zzfDNzQy/blwHxvYjDOGJeZSGF2ImuPVPHA4nynw/GJtg43m8tquXvWKOumGuJunpTDd187TEV9M3np/q12CZR1hy+QEBsVkHuD2MhnM2S3Tslla3ktja0dTofiE9tP1NHY2sHN1k015N0ypatEGx6D3VSVtw5XsaQom4RY/3VT7WKJwQzZLZNzaHcpb4dJkf2twxeIj4ny24yVJnAKs5MoyErkrcPhkRiOXrhMZcMVbp0SmIsWSwxmyOaMSyd1WCxrw+CqTFVZc+gCSydkMSzO/1dkxv9WTMlha1l4lGjXehJcV9uev1liMEMWEx3F8knZrD9ShSvEb6n4xyuyXKdDMT5y65Rc2lzusCjRvnX4AjPzUsnx02yq17PEYLyyYkoutU1t7K1ocDoUr7x1qLN31S0BKqob/+sq0YZ6dVJNYyt7zjSwIoAXLZYYjFeWTcgmOkpYezi0u62uOVzFrDFp5CQH5orM+F9MdBQ3T8pm/dHQLtGuO1KFamfVWKBYYjBeSR0ey9xx6bx1KHSvyqoutbD3TAO3TbVqpHBz69Rc6pra2H263ulQhmzt4QuMTE1g6siUgB3TEoPx2h3TRnD0wmVO1DQ5HcqQdDWeW/tC+LlpYjYxUcKaEC3RXmlzsfFYDSum5AR0bI0lBuO1O6ePAOCNA+cdjmRo1hy6QF76MCbm2mjncJOSEMvCwkxWHzgfkrej3XCsmivtLlZOHxnQ41piMF4blTaMmXmpvHHgnNOhDNqllnbeOV7DndNG2GjnMLVy+khO1jZz5Pxlp0MZtDcOnCN9eCzF+RkBPa4lBuMTd04fyd6Ki1Q2XHE6lEFZe/gCbS43K28I7BWZCZzbp+USJfD6/tC6cGntcLH2cBW3Tc0lJjqwP9WWGIxPdFUnrQ6x6qRX951nZGoCN455z/2lTJjISoqnOD+T10Lss7mptIbLrR0Br0YCSwzGR/KzEpk8Ijmk2hkut7Sz8Xg1d04fQVSUVSOFs/fdMILSqkaOXwid6qTX958nOT6GRUX+nzTvepYYjM/cOX0EO07VUXW5xelQBmTdkSraOtzcZdVIYe+OaSMQgdf2h8aFS7vLzZrDF1gxJcev93bujSUG4zN3Th+BKqw+GBpdA1/bf47clHhmj013OhTjZzkpCcwbl8HrIdJBYlt5HQ3N7dzpQDUSWGIwPjQpN5nC7ERe3nPW6VD61dTaQcnRalZOH2nVSBFi5Q0jOHL+MmXVjU6H0q9X959leFx0QG7K0xNLDMZnRIQPzhrN9pN1Qd87ae2RKlo73Kz0NJqb8Ldy+khE4OW9wX3h0trh4tV957hj2gjHZvq1xGB86p5ZowFYFeSlhpd2VzIyNYG54wPbP9w4Z0RqAgsLMnlpd2VQD3Zbf6SaSy0d3DNrlGMxeJUYRCRDRNaIyHHPv++prBWRm0VkT7dHi4h80LPsSRE50W3ZLG/iMc4bmzmcG8em8Yc9lU6H0qvqy61sOFbNPbNGE23VSBHlgzeO5mRtM7vPBO9swC/triQrKc7RG0Z5W2J4GFirqhOAtZ7X11DV9ao6S1VnAbcAzcCb3Vb5atdyVd3jZTwmCHxw1miOnL/MkfOXnA6lRy/vPYvLrXx49minQzEBtnL6COJjonjx3eC8cLl4pZ11R6r4wMxRAR/U1p23R74HeMrz/Cngg/2s/xHgdVVt9vK4JojdNWMk0VHCS7uDszrpxd2VTB+dwsTcZKdDMQGWnBDLbVNzeWXfWTqCcCru1/efo83l5kM3OnvR4m1iyFXVrv5f54H+pqe8F3jmuve+KyL7ROQHIhLvZTwmCGQlxbN0Qhar9lTiDrIvX2nVZfZXXuRDN+Y5HYpxyIduHE19czv7a1xOh/IeL+6upCA7kRtGpzoah/TXCCMibwE9dd34BvCUqqZ1W7deVXvsFC4iI4F9wChVbe/23nkgDngMKFPV7/Sy/YPAgwC5ublznn322R7jbWxsJCkp/GbJDLXz2nK2g5/ta+Xv5yUwNbPnnhVOnNMLx9p47UQ7P1g+nNR437cvhNrfaaDC6bw63MqX1zdTlKr89dzgOafaK26+suEKHyqK5Z6iuCHto7+/080337xLVef2uyNVHfIDOAqM9DwfCRztY92/Bh7rY/ly4JWBHHfOnDnam/Xr1/e6LJSF2nldaevQGd9erX/56129rhPoc3K53LrwX9/S+5/Y5rdjhNrfaaDC7by++dJ+LXzkFa1rbHU6lKv+482jOv7hV/R0bdOQ99Hf3wnYqQP4jfW2KmkVcL/n+f3AH/pY9+NcV43kKTEgnfMdfxA44GU8JkgkxEbz4dmjWX3wPLWNrU6HA8CG49WcvdjCn8y2aqRI97F5Y+lww+/erXA6FAA6XG6e23GamyZkMyZjuNPheJ0YHgVuE5HjwK2e14jIXBF5vGslERkPjAE2XLf9r0VkP7AfyAL+xct4TBD5+PyxtLs0aL58T285RVZSPHdMs0FtkW7qqBSK0qL4zbbTQTGmYd2RKi5cauW+4rFOhwJ4mRhUtVZVV6jqBFW9VVXrPO/vVNXPdVvvpKqOVlX3ddvfoqo3qOp0Vf2kqgb/WHUzYBNzk5k7Lp1ntp9x/Mt3pq6ZdUeruHfeGOJibFyngZvHxFBe08SWslqnQ+E320+TmxLPisk5TocC2Mhn42cfnz+WEzVNbC2vczSOZ7afRoCPB8kVmXHevBExpA2P5dfbTjsax5m6ZjYcq+Zjc8c4Onahu+CIwoStu2aMJCUhht9sd+7L19rh4vmdZ7hlci6j04Y5FocJLnHRwkdm57H64HlHp4p/bscZBPjY/OC5aLHEYPwqITaaj84dw+v7zzk2sd4bB85T09jGpxaOc+T4JnjdVzyWDrfy/I4zjhy/pd3FszvOsHxSTlBdtFhiMH73mSX5KPDEOycCfmxV5cnNJxmXOZylDs49Y4JTQXYSSydk8dSWU7S0B37A2+/fraSmsZXPLckP+LH7YonB+N3otGHcPXMUz2w/zcXm9oAee0t5LbtPN/C5pQV23wXToy8uK6T6cisv7Aps7zmXW3lsYxkz8lJZWBj423f2xRKDCYg/X1pAc5uLp7edCuhx/2d9GdnJ8Xx0jo1dMD1bWJjJzDFp/GxjGR0ud/8b+MgbB85zsraZLy4rpHMoV/CwxGACYuqoFG6amM0vNp0MWJF9z5kG3imt4c+X5pMQ68wNT0zwExH+YnkhZ+qu8Or+wNz6U1X56YYy8rMSuT0Ix9VYYjAB84WbCqhpDFyR/SfrS0kdFst9xdbobPp225RcJuQk8b8lZQEZc7O5rJb9lRd58KaCoLwniCUGEzALCzOZMy6dH649TlNrh1+PdeT8JdYcusADi8aTFB/j12OZ0BcVJXxhWSFHzl9m9cELfj2WqvL9N4+SmxLv+PTavbHEYAJGRPj6+yZTfbmVx9/2Xw8lVeVfXztCckIMf7Z4vN+OY8LLPbNGMSEniUdfP0xbh//aGl7Zd47dpxv4yu2TgraK0xKDCag54zJYOX0EP9tYRkOrf758649WsfFYNX+9YgJpw4c2fbGJPDHRUXzjrimcrG3ml1tO+uUYLe0uvvfGEaaMTAnqyRwtMZiA+/s7J9PW4ealUt93XW3rcPPPrxymMDuR+xeN9/n+TXhbPimH5ZOy+eHa436ZFfipzSepqL/CP9w1JSjbFrpYYjABl5+VyCeKx7KxooMDlRd9uu8nN5/gRE0T//j+qcQGybwzJrT8w11TaG5z8YO3jvl0v9WXW/nxulJumZzD4iAfbGnfHOOIL982kZQ44cvP7fFZ99Uzdc38aG3nF2/5pOCYpdKEnqKcZD61YBy/2XaabeW+mXlVVfna7/bR6nLz9fdN8ck+/ckSg3FE2vA4Pjs9juNVjfz76qNe76/d5eZLz+xGgH+6e5r3AZqI9nd3TGJcZiJ/89weGprbvN7f09tOs+5IFY+snExRTvDcTrQ3lhiMY27IjuFTC8bx83dOsLmsxqt9fX/1UfacaeB7H5kRFHfAMqEtKT6GH917IzWNrXztd/u8GttQWtXId189xE0Ts7l/4XjfBelHlhiMox5532TysxL5q2f2cKKmaUj7WHfkAj/bWM4nF4zlfTeM9HGEJlLdkJfKV++YxOqDF3hq88kh7eNiczsP/eZdhsVG8/2PzAiZ+bosMRhHDY+L4f8+PRe3Kp98fNugp+Z+53gNf/Hrd5kyMoV/uGuqn6I0kepzSwq4dUoO3375EM8O8p4il1ra+fQT2yivbuJHH7+RnJQEP0Xpe5YYjOOKcpL45Wfmc6mlnU8+vo3zFwd205S1hy/wmad2MD4zkV9+Zn7QDhYyoSsqSvjxfbNZNjGbh3+/n98M8G5vja0dPPDEdg6evcT/fGI2Sydk+zlS3/IqMYjIR0XkoIi4RWRuH+vdKSJHRaRURB7u9n6+iGzzvP+ciNhopAg1fXQqT/7ZPC5cauH2H2zgd7sqeq3XvdLm4sfrjvP5X+1i8ohknvnzBWQnxwc4YhMpEmKj+dmn5nDzpGy+/uJ+vv7ifuqbem+Q3nismvf98G32Vlzkx/fdyK1TcwMYrW94O4nMAeDDwM96W0FEooGfALcBFcAOEVmlqoeA7wE/UNVnReSnwGeB//UyJhOi5ozL4JUvLeHvX9jHV367l5f2VPKBmaNYWJBJZlIc5dVN7DnTwE/Wl3LuYgt3ThvBv310BikJsU6HbsJcQmw0P/3UHL73+lGe2nKS1/af4wvLCpk3PoPJI5K53NLBu6freXX/OV7dd46CrER+/bliFhQE130WBsqrxKCqh4H+5hKfD5Sqarln3WeBe0TkMHALcJ9nvaeAb2OJIaIVZCfx3OcX8tTmk/xPSSlvH39vb6UZean818dmURyiXzoTmuJjovnmB6bysXlj+Paqgzz6+pH3rJMQG8VfrZjAXywvDOmqzUBMOzka6H5D1QqgGMgEGlS1o9v7vU41KCIPAg96XjaKSG+d37MA7/o+BqdwPK8hndMp4OUv+T4YHwnHvxOE53n55Zy+4nk4pL9zGtAc9P0mBhF5C+jpThLfUNU/DOQgvqCqjwGP9beeiOxU1V7bO0JVOJ6XnVPoCMfzsnPqXb+JQVVv9fIYlcCYbq/zPO/VAmkiEuMpNXS9b4wxxkGB6K66A5jg6YEUB9wLrNLOLifrgY941rsfCFgJxBhjTM+87a76IRGpABYCr4rIas/7o0TkNQBPaeAhYDVwGHheVQ96dvE14G9FpJTONoefexOPR7/VTSEqHM/Lzil0hON52Tn1QgJxf1NjjDGhwzbdtQcAAAPXSURBVEY+G2OMuYYlBmOMMdcI28QgIl8SkSOeKTv+zel4fEVEviIiKiLBfQuoARKRf/f8nfaJyIsikuZ0TEPV29QvoUpExojIehE55Pke/bXTMfmKiESLyG4RecXpWHxFRNJE5AXP9+mwiCwc6r7CMjGIyM3APcBMVZ0GfN/hkHxCRMYAtwODm+YxuK0BpqvqDOAY8IjD8QxJt6lfVgJTgY+LSKhP99oBfEVVpwILgL8Mg3Pq8td0doYJJz8E3lDVycBMvDi/sEwMwBeBR1W1FUBVqxyOx1d+APw9EDY9BlT1zW6j37fSOZ4lFF2d+kVV24Bn6bw4CVmqek5V3/U8v0znD02vsxOEChHJA+4CHnc6Fl8RkVTgJjw9O1W1TVUbhrq/cE0ME4GlnplbN4jIPKcD8paI3ANUqupep2Pxo88ArzsdxBD1NPVLyP+IdhGR8cCNwDZnI/GJ/6LzAsvtdCA+lA9UA7/wVJE9LiKJQ91ZIOZK8ou+puqg87wy6Cz+zgOeF5ECDfK+uf2c09fprEYKOQOZVkVEvkFn1cWvAxmb6Z+IJAG/A/5GVS85HY83ROT9QJWq7hKR5U7H40MxwGzgS6q6TUR+CDwM/ONQdxaS+pqqQ0S+CPzekwi2i4ibzsmlqgMV31D0dk4icgOdVwR7PTPZ5gHvish8VT0fwBCHpL9pVUTkAeD9wIpgT9596G3ql5AmIrF0JoVfq+rvnY7HBxYDd4vI+4AEIEVEnlbVTzocl7cqgApV7SrRvUBnYhiScK1Kegm4GUBEJgJxhPDMkKq6X1VzVHW8qo6n80MwOxSSQn9E5E46i/V3q2qz0/F4ocepXxyOySvSeRXyc+Cwqv6n0/H4gqo+oqp5nu/RvcC6MEgKeH4LzojIJM9bK4BDQ91fyJYY+vEE8ISIHADagPtD+Eo03P0YiAfWeEpDW1X1C86GNHiq2iEiXVO/RANPdJv6JVQtBj4F7BeRPZ73vq6qrzkYk+ndl4Bfey5MyoE/G+qObEoMY4wx1wjXqiRjjDFDZInBGGPMNSwxGGOMuYYlBmOMMdewxGCMMeYalhiMMcZcwxKDMcaYa1hiMMYHROQLIrLH8zghIuudjsmYobIBbsb4kGduoXXAv6nqy07HY8xQWInBGN/6IZ3z71hSMCErXOdKMibgPLPEjgMecjgUY7xiVUnG+ICIzAGeApaqar3T8RjjDatKMsY3HqLz5lDrPQ3QYXPbSBN5rMRgjDHmGlZiMMYYcw1LDMYYY65hicEYY8w1LDEYY4y5hiUGY4wx17DEYIwx5hqWGIwxxlzj/wPrOOPwGlxvcwAAAABJRU5ErkJggg==\n",
+      "text/plain": [
+       "
"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    },
+    {
+     "data": {
+      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYQAAAEWCAYAAABmE+CbAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3Xl8VPW9//HXh10IO2HfhbAJVUFwuxVckargVtFWRaVoW27rvW0F61r1urW11avW4lZtrUoBERFlkcR9AwsJOxGQfZcl7Ek+vz/mcH9pmpCQOTNzQt7Px2MeOWfOd873Mycw7znbN+buiIiIVEt1ASIiEg0KBBERARQIIiISUCCIiAigQBARkYACQUREAAWCVGJmlmdmnYPp48zsLTPbaWb/MLMfmNmMCq53hJl9VMqyjmbmZlYjmH/HzK6v+LuIhmPlfUh8aqS6ADm2mNkqoAVQAOQB7wKj3T0vzvVmAX9z9+cOP+fuaUWaXBH029Td84PnXomnz/Jw9wsT3UcyFH0fZjYCGOnuZ6auIkkF7SFIIlwcfFifCJwE3J6EPjsAy4qEwTHr8N6JSNgUCJIw7r4RmE4sGAAws9pm9jszW21mm8zsGTM7rsjyoWY2z8x2mdnXZjbYzP4H+A/gyeAw0ZNBWzezLmb2G+Bu4Kpg+U3FD/uYWXczm2lm281sqZl9v8iypmY2JejzC+D48r5HM8sys5HB9Agz+yh4f9+a2UozK/rNu6GZPW9mG8xsnZk9YGbVg2XHm9lsM9tmZlvN7BUza1TktavMbIyZZQN7iodC8UNZFagty8xGmlkP4BngtGBb7ijvtpDKT4EgCWNmbYELgdwiTz8MZBALiS5AG2If5phZf+Bl4FdAI+C7wCp3vwP4kNihpzR3H120H3e/B3gQeD1Y/nyxOuoBM4G/A82B4cDTZtYzaPIUsB9oBdwYPCpqALAUaAY8CjxvZhYs+wuQH7zvk4DzgZGHywQeAloDPYB2wL3F1n018D2gUQX3hI5UGwDuvhi4Bfg02JaN/n01cqxSIEgiTDaz3cAaYDNwD0Dw4TMK+C933+7uu4l9kA8PXncT8IK7z3T3Qndf5+5LQqjnImLB8qK757v7P4GJwJXBN/TLgbvdfY+7LwBeiqOvb9z9WXcvCNbTCmhhZi2AIcCtQT+bgT8QvHd3zw3e9wF33wI8BpxVbN1PuPsad98XZm0VXJccg3QsUhJhmLvPMrOziH0rbwbsANKBusDcIl9MDageTLcDpiWgng7AgGKHP2oAfw1qqkEsvA77Jo6+Nh6ecPe9wftMA5oANYENRd57tcP9BoHxOLFDY/WDZd8WW/ca4lNabSKAAkESyN3fN7O/AL8DhgFbgX1AL3dfV8JL1lD68ft4huVdA7zv7ucVXxDsIeQTC6PDeyPt4+jrSDUcAJqVcrjnQWLvsbe7bzezYcCTxdocaRvsCX7WBXYF0y0rWKuGQK6idMhIEu2PwHlm9h13LwSeBf5gZs0BzKyNmV0QtH0euMHMzjGzasGy7sGyTUDnCtYwFcgws2vNrGbwOMXMegSHTyYB95pZ3eC8QujX47v7BmAG8HszaxC8v+ODvSiI7RXkATvNrA2x8yhHs/4twDrgh2ZW3cxu5ChOjhezCWhrZrUq+HqppBQIklDBB9XLBCeOgTHETjJ/Zma7gFlAt6DtF8ANxI6t7wTeJ3a4B2KHU64IrpB54ihr2E3sBO5wYD2xQyePALWDJqOJHTrZSOzE74tH+z7L6TqgFrCI2OGgCcSO4wP8BjiZ2Pt+m1hIHa0fEQuSbUAv4JMK1jkbWAhsNLOtFVyHVEKmP5AjIiKgPQQREQnEHQhm1s7MMs1skZktNLOfl9DGzOwJM8s1s2wzOznefkVEJFxhXGWUD/zC3b8ys/rELimc6e6LirS5EOgaPAYAfwp+iohIRMS9h+DuG9z9q2B6N7CY2N2nRQ0FXvaYz4BGZtYKERGJjFDvQzCzjsRuyf+82KI2/OtNNWuD5zaUsI5RxO5mpU6dOn3bt0/EJeHhKSwspFq16J+KUZ3hUp3hUp3hWbZs2VZ3T6/Qi909lAexy/bmApeVsGwqcGaR+feAfmWtMyMjw6MuMzMz1SWUi+oMl+oMl+oMDzDHK/g5HkrUmVlNYmPDvOLuJV0/vY7YnaCHtQ2eExGRiAjjKiMjdofpYnd/rJRmU4DrgquNTgV2euzOTRERiYgwziGcAVwL5JjZvOC5XxOMB+PuzxAbsGwIsTtU9xK7G1VERI5CYaGTdzCfBnVqJmT9cQeCu39EbMTKI7Vx4Kfx9iUiUlXlbs5j7MRs0urU4MURp1DsT1mEItqny0VEqrhDBYU8lZnLkMc/JHdLHhf3aZ2wvjT8tYhIRC1Yt5PbJmSzaMMuvte7Ffde0ov0+rXLfmEFKRBERCJm/6ECHn9vOeM+WEGTerV45od9GXxCRf+8RfkpEEREIuTLVdsZMyGbFVv38P1+bbljSE8a1k3MSeTiFAgiIhGQdyCfR99dwsuffkPbxsfxt5sGcGbXZkmtQYEgIpJimUs3c8ekHDbs2s8NZ3TkVxd0o26t5H88KxBERFLk2z0HuX/qIib9cx1dmqcx4ZbT6duhccrqUSCIiCSZuzMtZyP3TFnAjr2H+NnZXfjp2V2oXaN6SutSIIiIJNHmXfu5c/ICZizaRO82DXn5xgH0bN0g1WUBCgQRkaRwd/4xZy33v72Ig/mF3H5hd246sxM1qkfn/mAFgohIgq3ZvpfbJ+XwUe5W+ndqwsOX9aZzelqqy/o3CgQRkQQpKHRe+mQVv52+lOrVjPuHncAP+renWrXwxyEKgwJBRCQBlm/azZiJ2Xy1egcDu6Xz4KW9ad3ouFSXdUQKBBGREB0qKOSZrK/539m51KtdnT9edSJDT2ydkNFJw6ZAEBEJSc7anfxqwnyWbNzNRX1ig9E1S0vcYHRhUyCIiMRp/6EC/jBrGc9+sIJmabUZd21fzu+V+MHowqZAEBGJw+crtjF2Ug4rt+7hqn7t+PX3etDwuOQMRhc2BYKISAXs3n+IR95dwt8+W037JnV5ZeQAzuiS3MHowhZKIJjZC8BFwGZ3P6GE5QOBN4GVwVOT3P2+MPoWEUm2zCWb+fUbOWzatZ+RZ3biv8/PSMlgdGEL6x38BXgSePkIbT5094tC6k9EJOl2H3Rufe2fTJ63nq7N03j6x6dzUvvUDUYXtlACwd0/MLOOYaxLRCRq3J23sjdwx4d72V+4j5+f05WfDDo+5YPRhS2Z+zinmdl8YD3wS3dfmMS+RUQqZOPO2GB0sxZvolPDavzphjPo3jIag9GFzdw9nBXF9hCmlnIOoQFQ6O55ZjYEeNzdu5aynlHAKID09PS+48ePD6W+RMnLyyMtLXpjkhSnOsOlOsMVxTrdnffX5vP60oMUFMKlXWtxRrMDNKgfrTqLGzRo0Fx371ehF7t7KA+gI7CgnG1XAc3KapeRkeFRl5mZmeoSykV1hkt1hitqda7amufD//ypdxgz1a/68ye+ckueu0evzpIAc7yCn+NJOWRkZi2BTe7uZtYfqAZsS0bfIiLlVVDovPjxSn43Yyk1q1Xjoct6M/yUdpVi2IkwhHXZ6avAQKCZma0F7gFqArj7M8AVwI/NLB/YBwwPkkxEJBKWbtzNbROzmb9mB+f2aM4Dw3rTsmGdVJeVVGFdZXR1GcufJHZZqohIpBzML+TprFyeysylfp2aPHH1SVzcp1WV2SsoqvLfSSEiUkHz1uxgzIRslm7azSXfac09F/ekaSUajC5sCgQRqXL2HSzg9zOW8sLHK2levw7PX9+Pc3q0SHVZKadAEJEq5ZOvtzJ2Yg6rt+/lmgHtGXthdxrUqZyD0YVNgSAiVcKu/Yd4aNpiXv1iDR2b1uXVH53Kacc3TXVZkaJAEJFj3sxFm7hzcg5bdh/g5u925tZzMziu1rE17EQYFAgicszamneAe6csZGr2Brq3rM+z1/WjT9tGqS4rshQIInLMcXfenLee37y1kLwD+fz3eRncctbx1KpRLdWlRZoCQUSOKet37OPOyQuYvWQzJ7ZrxKNX9CGjRf1Ul1UpKBBE5JhQWOi8+uVqHpq2hIJC566LejLi9I5Ur1b1bjCrKAWCiFR6K7fuYezEbD5fuZ0zujTloUv70L5p3VSXVekoEESk0sovKOSFj1fy+xnLqFWjGo9e3ocr+7WtksNOhEGBICKV0uINuxgzMZvstTs5r2cLHhh2Ai0aVK3B6MKmQBCRSuVAfgFPzc7l6ayvaVS3Jk9dczJDerfUXkEIFAgiUml8tfpbxkzIZvnmPC47qQ13XdSTxvVqpbqsY4YCQUQib+/BfH43fRkvfrKSVg3q8OINpzCoW/NUl3XMUSCISKR9nLuVsZOyWbN9H9ee2oHbBnejvgajSwgFgohE0s59h3jw7cW8PmcNnZrV4/VRpzKgswajSyQFgohEzvSFG7lr8gK27TnILWcdz63ndqVOTQ1Gl2gKBBGJjC27Y4PRvZ2zgR6tGvD89afQu23DVJdVZYQSCGb2AnARsNndTyhhuQGPA0OAvcAId/8qjL5FpPJzdyZ9tZb7pi5i74ECfnVBN0Z9tzM1q2swumQKaw/hL8CTwMulLL8Q6Bo8BgB/Cn6KSBW3bsc+Hpt7gJyt8+nboTGPXN6HLs3TUl1WlRRKILj7B2bW8QhNhgIvu7sDn5lZIzNr5e4bwuhfRCqfwkLnlc+/4eF3lpBfUMC9F/fkutM6Uk2D0aWMxT6jQ1hRLBCmlnLIaCrwsLt/FMy/B4xx9zkltB0FjAJIT0/vO378+FDqS5S8vDzS0qL/bUZ1hkt1xmdDXiEvLjzAsm8L6dW0Gt/vVECHZtGrs7iobs+iBg0aNNfd+1XktZE7qezu44BxAN26dfOBAwemtqAyZGVlEfUaQXWGTXVWTH5BIeM+XMEfP1tOnRrV+O0VJ3BF37a8//77kaqzNFHbnmFLViCsA9oVmW8bPCciVcTC9TsZMzGbBet2MbhXS+4b1ovm9TUYXZQkKxCmAKPN7DViJ5N36vyBSNWw/1AB/zt7Oc+8v4LGdWvxpx+czIW9W6W6LClBWJedvgoMBJqZ2VrgHqAmgLs/A0wjdslpLrHLTm8Io18Riba532zntgnZfL1lD5ef3Ja7LupBo7oajC6qwrrK6Ooyljvw0zD6EpHo23Mgn99OX8pLn66idcPjeOnG/pyVkZ7qsqQMkTupLCKV2wfLtnD7pBzW79zHdad24FeDu5NWWx81lYF+SyISip17D3H/24uYMHctndPrMf7m0zilY5NUlyVHQYEgInF7d8EG7npzIdv3HOSng47nP8/WYHSVkQJBRCps8+793PPmQt5ZsJFerRvwlxtOoVdrDUZXWSkQROSouTsTv1rH/VMXse9QAbcN7saP/kOD0VV2CgQROSprtu/l12/k8OHyrfTr0JiHNRjdMUOBICLlUljovPzpKh6dvhQD7hvaix8O6KDB6I4hCgQRKVPu5t2MmZjD3G++5ayMdP7n0hNo27huqsuSkCkQRKRUhwoKGffBCh6ftZy6tavz2Pe/w6UntSH2N6/kWKNAEJESLVi3k9smZLNowy6+16cV917ci/T6tVNdliSQAkFE/sX+QwX8cdZynv1wBU3q1eLP1/blgl4tU12WJIECQUT+zxcrtzN2YjYrtu7hqn7t+PWQHjSsWzPVZUmSKBBEhLwD+TzyzhL++tk3tG18HH+7aQBndm2W6rIkyRQIIlVc5tLN3DEphw279nPjGZ345QUZ1K2lj4aqSL91kSrq2z0HuX/qIib9cx1dmqcx4ZbT6duhcarLkhRSIIhUMe7OtJyN3DNlATv2HuJnZ3fhp2d3oXYNDUZX1SkQRKqQTbv2c9fkBcxYtInebRry15sG0KNVg1SXJRGhQBCpAtyd8XPW8MDbizmYX8jtF3bnpjM7UUOD0UkRofxrMLPBZrbUzHLNbGwJy0eY2RYzmxc8RobRr4iUbfW2vfzw+c8ZMzGHHq0a8O6t3+Xms45XGMi/iXsPwcyqA08B5wFrgS/NbIq7LyrW9HV3Hx1vfyJSPgWFzvRVh5j83gdUr2Y8MOwErunfXoPRSanCOGTUH8h19xUAZvYaMBQoHggikiTLN+3mtonZ/HP1Qc7u3pwHhp1A60bHpbosiThz9/hWYHYFMNjdRwbz1wIDiu4NmNkI4CFgC7AM+C93X1PK+kYBowDS09P7jh8/Pq76Ei0vL4+0tOiPBa86wxXVOvMLnbdXHOKtrw9RpwZc3skZ2Kle5Aeji+r2LK4y1Dlo0KC57t6vQi9297gewBXAc0XmrwWeLNamKVA7mL4ZmF2edWdkZHjUZWZmprqEclGd4YpinfNWf+sX/OF97zBmqo/++1e+dff+SNZZEtUZHmCOV/DzPIxDRuuAdkXm2wbPFQ2dbUVmnwMeDaFfEQH2HSzgj7OW8eyHK0ivX5tnr+vHeT1bpLosqYTCCIQvga5m1olYEAwHrinawMxaufuGYPYSYHEI/YpUeZ+t2MbYidms2raXq/u34/YhPWhQR4PRScXEHQjunm9mo4HpQHXgBXdfaGb3Edt1mQL8zMwuAfKB7cCIePsVqcp27z/Ew+8s4ZXPV9O+SV3+PnIAp3fRYHQSn1BuTHP3acC0Ys/dXWT6duD2MPoSqepmL9nEHW8sYNOu/Yw8sxO/OL8bx9XSsBMSP92pLFJJbN9zkPveWsjkeevJaJHG0z84nZPaazA6CY8CQSTi3J23sjdw75SF7N5/iFvP7cpPBnahVg3daSzhUiCIRNjGnfu5c/ICZi3exHfaNeLRy/vQrWX9VJclxygFgkgEuTuvfbmGB99ezKHCQu78Xg9uOKMT1TXshCSQAkEkYr7ZtoexE3P4dMU2TuvclIcv702HpvVSXZZUAQoEkYgoKHRe/Hglv5uxlJrVqvHgpb25un+7yA87IccOBYJIBCzdGBuMbv6aHZzbozkPDOtNy4Z1Ul2WVDEKBJEUOphfyFOZuTydlUv9OjV54uqTuLhPK+0VSEooEERSZN6aHdw2YT7LNuUx7MTW3H1xL5rUq5XqsqQKUyCIJNneg/k8NmMZL3y8khYN6vDCiH6c3V2D0UnqKRBEkuiT3K2MnZTD6u17uWZAe26/sDv1NRidRIQCQSQJdu47xEPTFvPal2vo2LQur406lVM7N011WSL/QoEgkmAzF23izsk5bNl9gJu/25lbz83QYHQSSQoEkQTZlneAe99axFvz19O9ZX2eva4ffdo2SnVZIqVSIIiEzN2ZMn89905ZyJ4DBfzivAxuPut4DUYnkadAEAnR+h37uHPyAmYv2cxJ7WOD0XVtocHopHJQIIiEoLDQ+fsXq3n4nSUUFDp3XdSTEad31GB0UqkoEETitHLrHsZOzObzlds5o0tTHrq0D+2b1k11WSJHLZRAMLPBwOPE/qbyc+7+cLHltYGXgb7ANuAqd18VRt8iqZJfUMjzH63ksZnLqFWjGo9e3ocr+7XVsBNSacUdCGZWHXgKOA9YC3xpZlPcfVGRZjcB37p7FzMbDjwCXBVv3yKpsnpXAZc+/Qk563ZyXs8WPDDsBFo00GB0UrmFsYfQH8h19xUAZvYaMBQoGghDgXuD6QnAk2Zm7u5HWvHmvc6P/zY3hBITZ8uW/by+Nto1guoM06GCQjKX7KdxvUKeuuZkhvRuqb0COSaEEQhtgDVF5tcCA0pr4+75ZrYTaApsLb4yMxsFjAKo06IT2as2hVBi4hQWFrJhT7RrBNUZtv7NnR/0qkG97Ut5//2lqS6nVHl5eWRlZaW6jDKpzmiI3Elldx8HjAPo1q2bf3znkBRXdGRZWVkMHDgw1WWUSXWGS3WGS3VGQxh3yqwD2hWZbxs8V2IbM6sBNCR2cllERCIijED4EuhqZp3MrBYwHJhSrM0U4Ppg+gpgdlnnD0REJLniPmQUnBMYDUwndtnpC+6+0MzuA+a4+xTgeeCvZpYLbCcWGiIiEiGhnENw92nAtGLP3V1kej9wZRh9iYhIYmi0LRERARQIIiISUCCIiAigQBARkYACQUREAAWCiIgEFAgiIgIoEEREJKBAEBERQIEgIiIBBYKIiAAKBBERCSgQREQEUCCIiEhAgSAiIoACQUREAgoEEREBFAgiIhKIKxDMrImZzTSz5cHPxqW0KzCzecFjSjx9iohIYsS7hzAWeM/duwLvBfMl2efuJwaPS+LsU0REEiDeQBgKvBRMvwQMi3N9IiKSIubuFX+x2Q53bxRMG/Dt4fli7fKBeUA+8LC7Tz7COkcBowDS09P7jh8/vsL1JUNeXh5paWmpLqNMqjNcqjNcqjM8gwYNmuvu/Sr0Ync/4gOYBSwo4TEU2FGs7belrKNN8LMzsAo4vqx+3Z2MjAyPuszMzFSXUC6qM1yqM1yqMzzAHC/H52tJjxrlCIxzS1tmZpvMrJW7bzCzVsDmUtaxLvi5wsyygJOAr8uRVyIikiTxnkOYAlwfTF8PvFm8gZk1NrPawXQz4AxgUZz9iohIyOINhIeB88xsOXBuMI+Z9TOz54I2PYA5ZjYfyCR2DkGBICISMWUeMjoSd98GnFPC83OAkcH0J0DvePoREZHE053KIiICKBBERCSgQBAREUCBICIiAQWCiIgACgQREQkoEEREBFAgiIhIQIEgIiKAAkFERAIKBBERARQIIiISUCCIiAigQBARkYACQUREAAWCiIgEFAgiIgIoEEREJBBXIJjZlWa20MwKzazfEdoNNrOlZpZrZmPj6VNERBIj3j2EBcBlwAelNTCz6sBTwIVAT+BqM+sZZ78iIhKyGvG82N0XA5jZkZr1B3LdfUXQ9jVgKLAonr5FRCRc5u7xr8QsC/ilu88pYdkVwGB3HxnMXwsMcPfRpaxrFDAKID09ve/48ePjri+R8vLySEtLS3UZZVKd4VKd4VKd4Rk0aNBcdy/1EP6RlLmHYGazgJYlLLrD3d+sSKdH4u7jgHEA3bp184EDB4bdRaiysrKIeo2gOsOmOsOlOqOhzEBw93Pj7GMd0K7IfNvgORERiZBkXHb6JdDVzDqZWS1gODAlCf2KiMhRiPey00vNbC1wGvC2mU0Pnm9tZtMA3D0fGA1MBxYD4919YXxli4hI2OK9yugN4I0Snl8PDCkyPw2YFk9fIiKSWLpTWUREAAWCiIgEFAgiIgIoEEREJKBAEBERQIEgIiIBBYKIiAAKBBERCSgQREQEUCCIiEhAgSAiIoACQUREAgoEEREBFAgiIhJQIIiICKBAEBGRgAJBREQABYKIiATi/ZvKV5rZQjMrNLN+R2i3ysxyzGyemc2Jp08REUmMuP6mMrAAuAz4cznaDnL3rXH2JyIiCRJXILj7YgAzC6caERFJmWSdQ3BghpnNNbNRSepTRESOgrn7kRuYzQJalrDoDnd/M2iTBfzS3Us8P2Bmbdx9nZk1B2YC/+nuH5TSdhQwCiA9Pb3v+PHjy/teUiIvL4+0tLRUl1Em1Rku1Rku1RmeQYMGzXX3Us/pHpG7x/0AsoB+5Wx7L7HwKLNtRkaGR11mZmaqSygX1Rku1Rku1RkeYI5X8LM84YeMzKyemdU/PA2cT+xktIiIREi8l51eamZrgdOAt81sevB8azObFjRrAXxkZvOBL4C33f3dePoVEZHwxXuV0RvAGyU8vx4YEkyvAL4TTz8iIpJ4ulNZREQABYKIiAQUCCIiAigQREQkoEAQERFAgSAiIgEFgoiIAAoEEREJKBBERARQIIiISECBICIigAJBREQCCgQREQEUCCIiElAgiIgIoEAQEZGAAkFERAAFgoiIBBQIIiICxBkIZvZbM1tiZtlm9oaZNSql3WAzW2pmuWY2Np4+RUQkMeLdQ5gJnODufYBlwO3FG5hZdeAp4EKgJ3C1mfWMs18REQlZXIHg7jPcPT+Y/QxoW0Kz/kCuu69w94PAa8DQePoVEZHw1QhxXTcCr5fwfBtgTZH5tcCA0lZiZqOAUcHsATNbEFqFidEM2JrqIspBdYZLdYZLdYanW0VfWGYgmNksoGUJi+5w9zeDNncA+cArFS3kMHcfB4wL1jvH3fvFu85Eqgw1guoMm+oMl+oMj5nNqehrywwEdz+3jM5HABcB57i7l9BkHdCuyHzb4DkREYmQeK8yGgzcBlzi7ntLafYl0NXMOplZLWA4MCWefkVEJHzxXmX0JFAfmGlm88zsGQAza21m0wCCk86jgenAYmC8uy8s5/rHxVlfMlSGGkF1hk11hkt1hqfCNVrJR3lERKSq0Z3KIiICKBBERCQQqUCoDENhmNmVZrbQzArNrNTLz8xslZnlBOdWKnwZWEUdRZ0pHVbEzJqY2UwzWx78bFxKu4JgW84zs6RdlFDW9jGz2mb2erD8czPrmKzaitVRVp0jzGxLkW04MgU1vmBmm0u7t8hingjeQ7aZnZzsGoM6yqpzoJntLLIt705Bje3MLNPMFgX/z39eQpuj357uHpkHcD5QI5h+BHikhDbVga+BzkAtYD7QM4k19iB240cW0O8I7VYBzVK4LcusM9XbMqjhUWBsMD22pN95sCwvBduwzO0D/AR4JpgeDrwe0TpHAE8mu7ZiNXwXOBlYUMryIcA7gAGnAp9HtM6BwNQUb8tWwMnBdH1iQwcV/50f9faM1B6CV4KhMNx9sbsvTVZ/FVXOOqMwrMhQ4KVg+iVgWJL7P5LybJ+i9U8AzjEzS2KNEI3fY5nc/QNg+xGaDAVe9pjPgEZm1io51f1/5agz5dx9g7t/FUzvJnYFZ5tizY56e0YqEIq5kVi6FVfSUBjFN0QUODDDzOYGw3FEURS2ZQt33xBMbwRalNKujpnNMbPPzCxZoVGe7fN/bYIvMzuBpkmproQaAqX9Hi8PDh1MMLN2JSxPtSj8eyyv08xsvpm9Y2a9UllIcJjyJODzYouOenuGOZZRuSR7KIyKKE+N5XCmu68zs+bE7tNYEnzzCE1IdSbckeosOuPubmalXQfdIdienYHZZpbj7l+HXesx7C3gVXc/YGY3E9urOTvFNVVWXxH795hnZkOAyUDXVBRiZmnAROBWd98V7/qSHgheCYa7Mtu4AAACZUlEQVTCKKvGcq5jXfBzs5m9QWy3PtRACKHOpAwrcqQ6zWyTmbVy9w3B7uzmUtZxeHuuMLMsYt+IEh0I5dk+h9usNbMaQENgW4LrKq7MOt29aE3PETt3EzWVYpiboh+87j7NzJ42s2buntRB78ysJrEweMXdJ5XQ5Ki3Z6QOGdkxMhSGmdUzs/qHp4mdLI/iqK1R2JZTgOuD6euBf9uzMbPGZlY7mG4GnAEsSkJt5dk+Reu/AphdyheZRCqzzmLHji8hdsw5aqYA1wVXx5wK7CxyODEyzKzl4fNEZtaf2OdoUr8EBP0/Dyx298dKaXb02zOVZ8pLOHOeS+yY17zgcfjqjdbAtGJnz5cR+4Z4R5JrvJTYsbgDwCZgevEaiV3tMT94LEx2jeWtM9XbMui/KfAesByYBTQJnu8HPBdMnw7kBNszB7gpifX92/YB7iP2pQWgDvCP4N/uF0DnZG/Dctb5UPBvcT6QCXRPQY2vAhuAQ8G/zZuAW4BbguVG7I9pfR38nku9ii/FdY4usi0/A05PQY1nEjtPmV3k83JIvNtTQ1eIiAgQsUNGIiKSOgoEEREBFAgiIhJQIIiICKBAEBGRgAJBREQABYKIiAQUCCJHwcxuKTIO/kozy0x1TSJh0Y1pIhUQjCMzG3jU3d9KdT0iYdAegkjFPE5s3CKFgRwzkj7aqUhlF4zI24HYmDYixwwdMhI5CmbWl9jfEvgPd/821fWIhEmHjESOzmigCZAZnFh+LtUFiYRFewgiIgJoD0FERAIKBBERARQIIiISUCCIiAigQBARkYACQUREAAWCiIgE/h8n8ijWtV6/qgAAAABJRU5ErkJggg==\n",
+      "text/plain": [
+       "
"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    }
+   ],
+   "source": [
+    "%matplotlib inline\n",
+    "\n",
+    "\"\"\"The sigmoid function (or the logistic curve) is a \n",
+    "function that takes any real number, z, and outputs a number (0,1).\n",
+    "It is useful in neural networks for assigning weights on a relative scale.\n",
+    "The value z is the weighted sum of parameters involved in the learning algorithm.\"\"\"\n",
+    "\n",
+    "import numpy\n",
+    "import matplotlib.pyplot as plt\n",
+    "import math as mt\n",
+    "\n",
+    "z = numpy.arange(-5, 5, .1)\n",
+    "sigma_fn = numpy.vectorize(lambda z: 1/(1+numpy.exp(-z)))\n",
+    "sigma = sigma_fn(z)\n",
+    "\n",
+    "fig = plt.figure()\n",
+    "ax = fig.add_subplot(111)\n",
+    "ax.plot(z, sigma)\n",
+    "ax.set_ylim([-0.1, 1.1])\n",
+    "ax.set_xlim([-5,5])\n",
+    "ax.grid(True)\n",
+    "ax.set_xlabel('z')\n",
+    "ax.set_title('sigmoid function')\n",
+    "\n",
+    "plt.show()\n",
+    "\n",
+    "\"\"\"Step Function\"\"\"\n",
+    "z = numpy.arange(-5, 5, .02)\n",
+    "step_fn = numpy.vectorize(lambda z: 1.0 if z >= 0.0 else 0.0)\n",
+    "step = step_fn(z)\n",
+    "\n",
+    "fig = plt.figure()\n",
+    "ax = fig.add_subplot(111)\n",
+    "ax.plot(z, step)\n",
+    "ax.set_ylim([-0.5, 1.5])\n",
+    "ax.set_xlim([-5,5])\n",
+    "ax.grid(True)\n",
+    "ax.set_xlabel('z')\n",
+    "ax.set_title('step function')\n",
+    "\n",
+    "plt.show()\n",
+    "\n",
+    "\"\"\"Sine Function\"\"\"\n",
+    "z = numpy.arange(-2*mt.pi, 2*mt.pi, 0.1)\n",
+    "t = numpy.sin(z)\n",
+    "\n",
+    "fig = plt.figure()\n",
+    "ax = fig.add_subplot(111)\n",
+    "ax.plot(z, t)\n",
+    "ax.set_ylim([-1.0, 1.0])\n",
+    "ax.set_xlim([-2*mt.pi,2*mt.pi])\n",
+    "ax.grid(True)\n",
+    "ax.set_xlabel('z')\n",
+    "ax.set_title('sine function')\n",
+    "\n",
+    "plt.show()\n",
+    "\n",
+    "\"\"\"Plots a graph of the squashing function used by a rectified linear\n",
+    "unit\"\"\"\n",
+    "z = numpy.arange(-2, 2, .1)\n",
+    "zero = numpy.zeros(len(z))\n",
+    "y = numpy.max([zero, z], axis=0)\n",
+    "\n",
+    "fig = plt.figure()\n",
+    "ax = fig.add_subplot(111)\n",
+    "ax.plot(z, y)\n",
+    "ax.set_ylim([-2.0, 2.0])\n",
+    "ax.set_xlim([-2.0, 2.0])\n",
+    "ax.grid(True)\n",
+    "ax.set_xlabel('z')\n",
+    "ax.set_title('Rectified linear unit')\n",
+    "\n",
+    "plt.show()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "\n",
+    "## Setting up a Multi-layer perceptron model"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from scipy import optimize\n",
+    "\n",
+    "class Neural_Network(object):\n",
+    "    def __init__(self, Lambda=0):        \n",
+    "        #Define Hyperparameters\n",
+    "        self.inputLayerSize = 2\n",
+    "        self.outputLayerSize = 1\n",
+    "        self.hiddenLayerSize = 3\n",
+    "        \n",
+    "        #Weights (parameters)\n",
+    "        self.W1 = np.random.randn(self.inputLayerSize,self.hiddenLayerSize)\n",
+    "        self.W2 = np.random.randn(self.hiddenLayerSize,self.outputLayerSize)\n",
+    "        \n",
+    "        #Regularization Parameter:\n",
+    "        self.Lambda = Lambda\n",
+    "        \n",
+    "    def forward(self, X):\n",
+    "        #Propogate inputs though network\n",
+    "        self.z2 = np.dot(X, self.W1)\n",
+    "        self.a2 = self.sigmoid(self.z2)\n",
+    "        self.z3 = np.dot(self.a2, self.W2)\n",
+    "        yHat = self.sigmoid(self.z3) \n",
+    "        return yHat\n",
+    "        \n",
+    "    def sigmoid(self, z):\n",
+    "        #Apply sigmoid activation function to scalar, vector, or matrix\n",
+    "        return 1/(1+np.exp(-z))\n",
+    "    \n",
+    "    def sigmoidPrime(self,z):\n",
+    "        #Gradient of sigmoid\n",
+    "        return np.exp(-z)/((1+np.exp(-z))**2)\n",
+    "    \n",
+    "    def costFunction(self, X, y):\n",
+    "        #Compute cost for given X,y, use weights already stored in class.\n",
+    "        self.yHat = self.forward(X)\n",
+    "        J = 0.5*sum((y-self.yHat)**2)/X.shape[0] + (self.Lambda/2)*(np.sum(self.W1**2)+np.sum(self.W2**2))\n",
+    "        return J\n",
+    "        \n",
+    "    def costFunctionPrime(self, X, y):\n",
+    "        #Compute derivative with respect to W and W2 for a given X and y:\n",
+    "        self.yHat = self.forward(X)\n",
+    "        \n",
+    "        delta3 = np.multiply(-(y-self.yHat), self.sigmoidPrime(self.z3))\n",
+    "        #Add gradient of regularization term:\n",
+    "        dJdW2 = np.dot(self.a2.T, delta3)/X.shape[0] + self.Lambda*self.W2\n",
+    "        \n",
+    "        delta2 = np.dot(delta3, self.W2.T)*self.sigmoidPrime(self.z2)\n",
+    "        #Add gradient of regularization term:\n",
+    "        dJdW1 = np.dot(X.T, delta2)/X.shape[0] + self.Lambda*self.W1\n",
+    "        \n",
+    "        return dJdW1, dJdW2\n",
+    "    \n",
+    "    #Helper functions for interacting with other methods/classes\n",
+    "    def getParams(self):\n",
+    "        #Get W1 and W2 Rolled into vector:\n",
+    "        params = np.concatenate((self.W1.ravel(), self.W2.ravel()))\n",
+    "        return params\n",
+    "    \n",
+    "    def setParams(self, params):\n",
+    "        #Set W1 and W2 using single parameter vector:\n",
+    "        W1_start = 0\n",
+    "        W1_end = self.hiddenLayerSize*self.inputLayerSize\n",
+    "        self.W1 = np.reshape(params[W1_start:W1_end], \\\n",
+    "                             (self.inputLayerSize, self.hiddenLayerSize))\n",
+    "        W2_end = W1_end + self.hiddenLayerSize*self.outputLayerSize\n",
+    "        self.W2 = np.reshape(params[W1_end:W2_end], \\\n",
+    "                             (self.hiddenLayerSize, self.outputLayerSize))\n",
+    "        \n",
+    "    def computeGradients(self, X, y):\n",
+    "        dJdW1, dJdW2 = self.costFunctionPrime(X, y)\n",
+    "        return np.concatenate((dJdW1.ravel(), dJdW2.ravel()))\n",
+    "    \n",
+    "        \n",
+    "class trainer(object):\n",
+    "    def __init__(self, N):\n",
+    "        #Make Local reference to network:\n",
+    "        self.N = N\n",
+    "        \n",
+    "    def callbackF(self, params):\n",
+    "        self.N.setParams(params)\n",
+    "        self.J.append(self.N.costFunction(self.X, self.y))\n",
+    "        self.testJ.append(self.N.costFunction(self.testX, self.testY))\n",
+    "        \n",
+    "    def costFunctionWrapper(self, params, X, y):\n",
+    "        self.N.setParams(params)\n",
+    "        cost = self.N.costFunction(X, y)\n",
+    "        grad = self.N.computeGradients(X,y)\n",
+    "        return cost, grad\n",
+    "        \n",
+    "    def train(self, trainX, trainY, testX, testY):\n",
+    "        #Make an internal variable for the callback function:\n",
+    "        self.X = trainX\n",
+    "        self.y = trainY\n",
+    "        \n",
+    "        self.testX = testX\n",
+    "        self.testY = testY\n",
+    "\n",
+    "        #Make empty list to store training costs:\n",
+    "        self.J = []\n",
+    "        self.testJ = []\n",
+    "        \n",
+    "        params0 = self.N.getParams()\n",
+    "\n",
+    "        options = {'maxiter': 200, 'disp' : True}\n",
+    "        _res = optimize.minimize(self.costFunctionWrapper, params0, jac=True, method='BFGS', \\\n",
+    "                                 args=(trainX, trainY), options=options, callback=self.callbackF)\n",
+    "\n",
+    "        self.N.setParams(_res.x)\n",
+    "        self.optimizationResults = _res"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "\n",
+    "## Two-layer Neural Network"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Output after training:  [[6.55109972e-03 9.93684857e-01 9.93925710e-01 6.62304973e-03]\n",
+      " [1.71082162e-03 9.97516440e-01 9.97766376e-01 1.82685927e-03]\n",
+      " [2.05800960e-03 9.98268211e-01 9.97548919e-01 1.77362990e-03]\n",
+      " [5.35659849e-04 9.99320839e-01 9.99100767e-01 4.87503198e-04]]\n"
+     ]
+    }
+   ],
+   "source": [
+    "import numpy as np\n",
+    "\n",
+    "#sigmoid\n",
+    "def nonlin(x, deriv=False):\n",
+    "    if (deriv==True):\n",
+    "        return x*(1-x)\n",
+    "    return 1/(1+np.exp(-x))\n",
+    "\n",
+    "#input data\n",
+    "x=np.array([[0,0,1],[0,1,1],[1,0,1],[1,1,1]])\n",
+    "\n",
+    "#output data\n",
+    "y=np.array([0,1,1,0]).T\n",
+    "\n",
+    "#seed random numbers to make calculation\n",
+    "np.random.seed(1)\n",
+    "\n",
+    "#initialize weights with mean=0\n",
+    "syn0=2*np.random.random((3,4))-1\n",
+    "\n",
+    "for iter in range(10000):\n",
+    "    #forward propogation\n",
+    "    l0=x\n",
+    "    l1=nonlin(np.dot(l0,syn0))\n",
+    "    l1_error=y-l1\n",
+    "    #multiply error by slope of sigmoid at values of l1\n",
+    "    l1_delta=l1_error*nonlin(l1,True)\n",
+    "    #update weights\n",
+    "    syn0+=np.dot(l0.T, l1_delta)\n",
+    "    \n",
+    "print(\"Output after training: \",l1 )"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 4,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import numpy as np\n",
+    "import random\n",
+    "class Network(object):\n",
+    "    \n",
+    "    def _init_(self, sizes):\n",
+    "        self.num_layers=len(sizes)\n",
+    "        self.sizes=sizes\n",
+    "        self.biases=[np.random.randn(y,1) for y in sizes[1:]]\n",
+    "        self.weights=[np.random.randn(y,x) for x,y in zip(sizes[:-1], sizes[1:])]\n",
+    "\n",
+    "#sizes is the number of neurons in each layer\n",
+    "#for example, say n_1st_layer=3, n_2nd_layer=3, n_3rd_layer=1, then net=Network([3,3,1])\n",
+    "\n",
+    "#The biases and weights are initialized randomly, using Gaussian distributions of mean=0, stdev=1\n",
+    "#z is a vector (or a np.array)\n",
+    "\n",
+    "    def feedforward(self,a):\n",
+    "        #returns output w/ 'a' as an input\n",
+    "        for b, w in zip(self.biases, self.weights):\n",
+    "            a=sigmoid(np.dot(w,b)+b)\n",
+    "        return a\n",
+    "    \n",
+    "#Apply a Stochastic Gradient Descent (SGD) method:\n",
+    "    def SGD(self, training_data, epochs, mini_batch_size, eta, test_data=None):\n",
+    "        \"\"\"Trains network using batches incorporating SGD. The network will be evaluated against the\n",
+    "        test data after each epoch, with partial progress being printed out (this is useful for tracking,\n",
+    "        but slows the process.)\"\"\"\n",
+    "        if test_data: n_test=len(test_data)\n",
+    "        n=len(training_data)\n",
+    "        for j in xrange(epochs):\n",
+    "            random.shuffle(training_data)\n",
+    "            mini_batches=[training_data[k:k+mini_batch_size] for k in xrange(o,n,mini_batch_size)]\n",
+    "            for mini_batch in mini_batches:\n",
+    "                self.update_mini_batch(mini_batch, eta)\n",
+    "            if test_data:\n",
+    "                print (\"Epoch {0}: {1}/{2}\".format(j, self.evaluate(test_data), n_test))\n",
+    "            else:\n",
+    "                print (\"Epoch {0} complete\".format(j))\n",
+    "            \n",
+    "        \n",
+    "    def update_mini_batch(self, mini_batch, eta):\n",
+    "        #updates w and b using backpropagation to a single mini batch. eta is the learning rate.\"\n",
+    "        nabla_b=[np.zeros(b.shape) for b in self.biases]\n",
+    "        nabla_w=[np.zeros(w.shape) for w in self.weights]\n",
+    "        for x,y in mini_batch:\n",
+    "            delta_nabla_b, delta_nabla_w=self.backprop(x,y)\n",
+    "            nabla_b=[nb+dnb for nb, dnb in zip(nabla_b, delta_nabla_b)]\n",
+    "            nabla_w=[nw+dnw for nw, dnw in zip(nabla_w, delta_nabla_w)]\n",
+    "        self.weights=[w-(eta/len(mini_batch))*nw for w, nw in zip(self.weights, nabla_w)]\n",
+    "        self.biases=[b-(eta/len(mini_batch))*nb for b, nb in zip(self.biases, nabla_b)]\n",
+    "        \n",
+    "    def backprop(self, x, y):\n",
+    "        \"\"\"Return a tuple ``(nabla_b, nabla_w)`` representing the\n",
+    "        gradient for the cost function C_x.  ``nabla_b`` and\n",
+    "        ``nabla_w`` are layer-by-layer lists of numpy arrays, similar\n",
+    "        to ``self.biases`` and ``self.weights``.\"\"\"\n",
+    "        nabla_b = [np.zeros(b.shape) for b in self.biases]\n",
+    "        nabla_w = [np.zeros(w.shape) for w in self.weights]\n",
+    "        # feedforward\n",
+    "        activation = x\n",
+    "        activations = [x] # list to store all the activations, layer by layer\n",
+    "        zs = [] # list to store all the z vectors, layer by layer\n",
+    "        for b, w in zip(self.biases, self.weights):\n",
+    "            z = np.dot(w, activation)+b\n",
+    "            zs.append(z)\n",
+    "            activation = sigmoid(z)\n",
+    "            activations.append(activation)\n",
+    "        # backward pass\n",
+    "        delta = self.cost_derivative(activations[-1], y) * \\\n",
+    "            sigmoid_prime(zs[-1])\n",
+    "        nabla_b[-1] = delta\n",
+    "        nabla_w[-1] = np.dot(delta, activations[-2].transpose())\n",
+    "        # Note that the variable l in the loop below is used a little\n",
+    "        # differently to the notation in Chapter 2 of the book.  Here,\n",
+    "        # l = 1 means the last layer of neurons, l = 2 is the\n",
+    "        # second-last layer, and so on.  It's a renumbering of the\n",
+    "        # scheme in the book, used here to take advantage of the fact\n",
+    "        # that Python can use negative indices in lists.\n",
+    "        for l in xrange(2, self.num_layers):\n",
+    "            z = zs[-l]\n",
+    "            sp = sigmoid_prime(z)\n",
+    "            delta = np.dot(self.weights[-l+1].transpose(), delta) * sp\n",
+    "            nabla_b[-l] = delta\n",
+    "            nabla_w[-l] = np.dot(delta, activations[-l-1].transpose())\n",
+    "        return (nabla_b, nabla_w)\n",
+    "\n",
+    "    def evaluate(self, test_data):\n",
+    "        \"\"\"Return the number of test inputs for which the neural\n",
+    "        network outputs the correct result. Note that the neural\n",
+    "        network's output is assumed to be the index of whichever\n",
+    "        neuron in the final layer has the highest activation.\"\"\"\n",
+    "        test_results = [(np.argmax(self.feedforward(x)), y)\n",
+    "                        for (x, y) in test_data]\n",
+    "        return sum(int(x == y) for (x, y) in test_results)\n",
+    "\n",
+    "    def cost_derivative(self, output_activations, y):\n",
+    "        \"\"\"Return the vector of partial derivatives \\partial C_x /\n",
+    "        \\partial a for the output activations.\"\"\"\n",
+    "        return (output_activations-y)\n",
+    "    \n",
+    "    \n",
+    "    \n",
+    "#Functions\n",
+    "def sigmoid(z):\n",
+    "        return 1.0/(1.0+np.exp(-z))\n",
+    "\n",
+    "def sigmoid_prime(z):\n",
+    "    return sigmoid(z)*(1-sigmoid(z))\n",
+    "\n",
+    "network=Network()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 5,
+   "metadata": {},
+   "outputs": [
+    {
+     "ename": "NameError",
+     "evalue": "name 'network' is not defined",
+     "output_type": "error",
+     "traceback": [
+      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
+      "\u001b[0;31mNameError\u001b[0m                                 Traceback (most recent call last)",
+      "\u001b[0;32m\u001b[0m in \u001b[0;36m\u001b[0;34m()\u001b[0m\n\u001b[1;32m     86\u001b[0m     \u001b[0;32mreturn\u001b[0m \u001b[0me\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     87\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 88\u001b[0;31m \u001b[0mnet\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mnetwork\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mNetwork\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m784\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;36m30\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;36m30\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     89\u001b[0m \u001b[0mnet\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mSGD\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtraining_data\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;36m30\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;36m10\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;36m3\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0mtest_data\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mtest_data\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
+      "\u001b[0;31mNameError\u001b[0m: name 'network' is not defined"
+     ]
+    }
+   ],
+   "source": [
+    "# %load neural-networks-and-deep-learning/src/mnist_loader.py\n",
+    "\"\"\"\n",
+    "mnist_loader\n",
+    "~~~~~~~~~~~~\n",
+    "\n",
+    "A library to load the MNIST image data.  For details of the data\n",
+    "structures that are returned, see the doc strings for ``load_data``\n",
+    "and ``load_data_wrapper``.  In practice, ``load_data_wrapper`` is the\n",
+    "function usually called by our neural network code.\n",
+    "\"\"\"\n",
+    "\n",
+    "#### Libraries\n",
+    "# Standard library\n",
+    "import pickle\n",
+    "import gzip\n",
+    "\n",
+    "# Third-party libraries\n",
+    "import numpy as np\n",
+    "\n",
+    "def load_data():\n",
+    "    \"\"\"Return the MNIST data as a tuple containing the training data,\n",
+    "    the validation data, and the test data.\n",
+    "\n",
+    "    The ``training_data`` is returned as a tuple with two entries.\n",
+    "    The first entry contains the actual training images.  This is a\n",
+    "    numpy ndarray with 50,000 entries.  Each entry is, in turn, a\n",
+    "    numpy ndarray with 784 values, representing the 28 * 28 = 784\n",
+    "    pixels in a single MNIST image.\n",
+    "\n",
+    "    The second entry in the ``training_data`` tuple is a numpy ndarray\n",
+    "    containing 50,000 entries.  Those entries are just the digit\n",
+    "    values (0...9) for the corresponding images contained in the first\n",
+    "    entry of the tuple.\n",
+    "\n",
+    "    The ``validation_data`` and ``test_data`` are similar, except\n",
+    "    each contains only 10,000 images.\n",
+    "\n",
+    "    This is a nice data format, but for use in neural networks it's\n",
+    "    helpful to modify the format of the ``training_data`` a little.\n",
+    "    That's done in the wrapper function ``load_data_wrapper()``, see\n",
+    "    below.\n",
+    "    \"\"\"\n",
+    "    f = gzip.open('../data/mnist.pkl.gz', 'rb')\n",
+    "    training_data, validation_data, test_data = cPickle.load(f)\n",
+    "    f.close()\n",
+    "    return (training_data, validation_data, test_data)\n",
+    "\n",
+    "def load_data_wrapper():\n",
+    "    \"\"\"Return a tuple containing ``(training_data, validation_data,\n",
+    "    test_data)``. Based on ``load_data``, but the format is more\n",
+    "    convenient for use in our implementation of neural networks.\n",
+    "\n",
+    "    In particular, ``training_data`` is a list containing 50,000\n",
+    "    2-tuples ``(x, y)``.  ``x`` is a 784-dimensional numpy.ndarray\n",
+    "    containing the input image.  ``y`` is a 10-dimensional\n",
+    "    numpy.ndarray representing the unit vector corresponding to the\n",
+    "    correct digit for ``x``.\n",
+    "\n",
+    "    ``validation_data`` and ``test_data`` are lists containing 10,000\n",
+    "    2-tuples ``(x, y)``.  In each case, ``x`` is a 784-dimensional\n",
+    "    numpy.ndarry containing the input image, and ``y`` is the\n",
+    "    corresponding classification, i.e., the digit values (integers)\n",
+    "    corresponding to ``x``.\n",
+    "\n",
+    "    Obviously, this means we're using slightly different formats for\n",
+    "    the training data and the validation / test data.  These formats\n",
+    "    turn out to be the most convenient for use in our neural network\n",
+    "    code.\"\"\"\n",
+    "    tr_d, va_d, te_d = load_data()\n",
+    "    training_inputs = [np.reshape(x, (784, 1)) for x in tr_d[0]]\n",
+    "    training_results = [vectorized_result(y) for y in tr_d[1]]\n",
+    "    training_data = zip(training_inputs, training_results)\n",
+    "    validation_inputs = [np.reshape(x, (784, 1)) for x in va_d[0]]\n",
+    "    validation_data = zip(validation_inputs, va_d[1])\n",
+    "    test_inputs = [np.reshape(x, (784, 1)) for x in te_d[0]]\n",
+    "    test_data = zip(test_inputs, te_d[1])\n",
+    "    return (training_data, validation_data, test_data)\n",
+    "\n",
+    "def vectorized_result(j):\n",
+    "    \"\"\"Return a 10-dimensional unit vector with a 1.0 in the jth\n",
+    "    position and zeroes elsewhere.  This is used to convert a digit\n",
+    "    (0...9) into a corresponding desired output from the neural\n",
+    "    network.\"\"\"\n",
+    "    e = np.zeros((10, 1))\n",
+    "    e[j] = 1.0\n",
+    "    return e\n",
+    "\n",
+    "net=network.Network([784,30,30])\n",
+    "net.SGD(training_data,30,10,3,test_data=test_data)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": []
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.7.0"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}
diff --git a/doc/pub/NeuralNet/ipynb/.ipynb_checkpoints/Untitled-checkpoint.ipynb b/doc/pub/NeuralNet/ipynb/.ipynb_checkpoints/Untitled-checkpoint.ipynb
new file mode 100644
index 000000000..2fd64429b
--- /dev/null
+++ b/doc/pub/NeuralNet/ipynb/.ipynb_checkpoints/Untitled-checkpoint.ipynb
@@ -0,0 +1,6 @@
+{
+ "cells": [],
+ "metadata": {},
+ "nbformat": 4,
+ "nbformat_minor": 2
+}
diff --git a/doc/pub/NeuralNet/ipynb/Untitled.ipynb b/doc/pub/NeuralNet/ipynb/Untitled.ipynb
new file mode 100644
index 000000000..a0665ba71
--- /dev/null
+++ b/doc/pub/NeuralNet/ipynb/Untitled.ipynb
@@ -0,0 +1,48 @@
+{
+ "cells": [
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "/usr/local/opt/python/bin/python3.7\n",
+      "3.7.0 (default, Jun 29 2018, 20:13:13) \n",
+      "[Clang 9.1.0 (clang-902.0.39.2)]\n",
+      "sys.version_info(major=3, minor=7, micro=0, releaselevel='final', serial=0)\n"
+     ]
+    }
+   ],
+   "source": [
+    "import sys\n",
+    "print (sys.executable)\n",
+    "print (sys.version)\n",
+    "print (sys.version_info)"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.7.0"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}
diff --git a/doc/pub/NeuralNet/ipynb/notebook.tex b/doc/pub/NeuralNet/ipynb/notebook.tex
new file mode 100644
index 000000000..1f61c6ecd
--- /dev/null
+++ b/doc/pub/NeuralNet/ipynb/notebook.tex
@@ -0,0 +1,3332 @@
+
+% Default to the notebook output style
+
+    
+
+
+% Inherit from the specified cell style.
+
+
+
+
+    
+\documentclass[11pt]{article}
+
+    
+    
+    \usepackage[T1]{fontenc}
+    % Nicer default font (+ math font) than Computer Modern for most use cases
+    \usepackage{mathpazo}
+
+    % Basic figure setup, for now with no caption control since it's done
+    % automatically by Pandoc (which extracts ![](path) syntax from Markdown).
+    \usepackage{graphicx}
+    % We will generate all images so they have a width \maxwidth. This means
+    % that they will get their normal width if they fit onto the page, but
+    % are scaled down if they would overflow the margins.
+    \makeatletter
+    \def\maxwidth{\ifdim\Gin@nat@width>\linewidth\linewidth
+    \else\Gin@nat@width\fi}
+    \makeatother
+    \let\Oldincludegraphics\includegraphics
+    % Set max figure width to be 80% of text width, for now hardcoded.
+    \renewcommand{\includegraphics}[1]{\Oldincludegraphics[width=.8\maxwidth]{#1}}
+    % Ensure that by default, figures have no caption (until we provide a
+    % proper Figure object with a Caption API and a way to capture that
+    % in the conversion process - todo).
+    \usepackage{caption}
+    \DeclareCaptionLabelFormat{nolabel}{}
+    \captionsetup{labelformat=nolabel}
+
+    \usepackage{adjustbox} % Used to constrain images to a maximum size 
+    \usepackage{xcolor} % Allow colors to be defined
+    \usepackage{enumerate} % Needed for markdown enumerations to work
+    \usepackage{geometry} % Used to adjust the document margins
+    \usepackage{amsmath} % Equations
+    \usepackage{amssymb} % Equations
+    \usepackage{textcomp} % defines textquotesingle
+    % Hack from http://tex.stackexchange.com/a/47451/13684:
+    \AtBeginDocument{%
+        \def\PYZsq{\textquotesingle}% Upright quotes in Pygmentized code
+    }
+    \usepackage{upquote} % Upright quotes for verbatim code
+    \usepackage{eurosym} % defines \euro
+    \usepackage[mathletters]{ucs} % Extended unicode (utf-8) support
+    \usepackage[utf8x]{inputenc} % Allow utf-8 characters in the tex document
+    \usepackage{fancyvrb} % verbatim replacement that allows latex
+    \usepackage{grffile} % extends the file name processing of package graphics 
+                         % to support a larger range 
+    % The hyperref package gives us a pdf with properly built
+    % internal navigation ('pdf bookmarks' for the table of contents,
+    % internal cross-reference links, web links for URLs, etc.)
+    \usepackage{hyperref}
+    \usepackage{longtable} % longtable support required by pandoc >1.10
+    \usepackage{booktabs}  % table support for pandoc > 1.12.2
+    \usepackage[inline]{enumitem} % IRkernel/repr support (it uses the enumerate* environment)
+    \usepackage[normalem]{ulem} % ulem is needed to support strikethroughs (\sout)
+                                % normalem makes italics be italics, not underlines
+    
+
+    
+    
+    % Colors for the hyperref package
+    \definecolor{urlcolor}{rgb}{0,.145,.698}
+    \definecolor{linkcolor}{rgb}{.71,0.21,0.01}
+    \definecolor{citecolor}{rgb}{.12,.54,.11}
+
+    % ANSI colors
+    \definecolor{ansi-black}{HTML}{3E424D}
+    \definecolor{ansi-black-intense}{HTML}{282C36}
+    \definecolor{ansi-red}{HTML}{E75C58}
+    \definecolor{ansi-red-intense}{HTML}{B22B31}
+    \definecolor{ansi-green}{HTML}{00A250}
+    \definecolor{ansi-green-intense}{HTML}{007427}
+    \definecolor{ansi-yellow}{HTML}{DDB62B}
+    \definecolor{ansi-yellow-intense}{HTML}{B27D12}
+    \definecolor{ansi-blue}{HTML}{208FFB}
+    \definecolor{ansi-blue-intense}{HTML}{0065CA}
+    \definecolor{ansi-magenta}{HTML}{D160C4}
+    \definecolor{ansi-magenta-intense}{HTML}{A03196}
+    \definecolor{ansi-cyan}{HTML}{60C6C8}
+    \definecolor{ansi-cyan-intense}{HTML}{258F8F}
+    \definecolor{ansi-white}{HTML}{C5C1B4}
+    \definecolor{ansi-white-intense}{HTML}{A1A6B2}
+
+    % commands and environments needed by pandoc snippets
+    % extracted from the output of `pandoc -s`
+    \providecommand{\tightlist}{%
+      \setlength{\itemsep}{0pt}\setlength{\parskip}{0pt}}
+    \DefineVerbatimEnvironment{Highlighting}{Verbatim}{commandchars=\\\{\}}
+    % Add ',fontsize=\small' for more characters per line
+    \newenvironment{Shaded}{}{}
+    \newcommand{\KeywordTok}[1]{\textcolor[rgb]{0.00,0.44,0.13}{\textbf{{#1}}}}
+    \newcommand{\DataTypeTok}[1]{\textcolor[rgb]{0.56,0.13,0.00}{{#1}}}
+    \newcommand{\DecValTok}[1]{\textcolor[rgb]{0.25,0.63,0.44}{{#1}}}
+    \newcommand{\BaseNTok}[1]{\textcolor[rgb]{0.25,0.63,0.44}{{#1}}}
+    \newcommand{\FloatTok}[1]{\textcolor[rgb]{0.25,0.63,0.44}{{#1}}}
+    \newcommand{\CharTok}[1]{\textcolor[rgb]{0.25,0.44,0.63}{{#1}}}
+    \newcommand{\StringTok}[1]{\textcolor[rgb]{0.25,0.44,0.63}{{#1}}}
+    \newcommand{\CommentTok}[1]{\textcolor[rgb]{0.38,0.63,0.69}{\textit{{#1}}}}
+    \newcommand{\OtherTok}[1]{\textcolor[rgb]{0.00,0.44,0.13}{{#1}}}
+    \newcommand{\AlertTok}[1]{\textcolor[rgb]{1.00,0.00,0.00}{\textbf{{#1}}}}
+    \newcommand{\FunctionTok}[1]{\textcolor[rgb]{0.02,0.16,0.49}{{#1}}}
+    \newcommand{\RegionMarkerTok}[1]{{#1}}
+    \newcommand{\ErrorTok}[1]{\textcolor[rgb]{1.00,0.00,0.00}{\textbf{{#1}}}}
+    \newcommand{\NormalTok}[1]{{#1}}
+    
+    % Additional commands for more recent versions of Pandoc
+    \newcommand{\ConstantTok}[1]{\textcolor[rgb]{0.53,0.00,0.00}{{#1}}}
+    \newcommand{\SpecialCharTok}[1]{\textcolor[rgb]{0.25,0.44,0.63}{{#1}}}
+    \newcommand{\VerbatimStringTok}[1]{\textcolor[rgb]{0.25,0.44,0.63}{{#1}}}
+    \newcommand{\SpecialStringTok}[1]{\textcolor[rgb]{0.73,0.40,0.53}{{#1}}}
+    \newcommand{\ImportTok}[1]{{#1}}
+    \newcommand{\DocumentationTok}[1]{\textcolor[rgb]{0.73,0.13,0.13}{\textit{{#1}}}}
+    \newcommand{\AnnotationTok}[1]{\textcolor[rgb]{0.38,0.63,0.69}{\textbf{\textit{{#1}}}}}
+    \newcommand{\CommentVarTok}[1]{\textcolor[rgb]{0.38,0.63,0.69}{\textbf{\textit{{#1}}}}}
+    \newcommand{\VariableTok}[1]{\textcolor[rgb]{0.10,0.09,0.49}{{#1}}}
+    \newcommand{\ControlFlowTok}[1]{\textcolor[rgb]{0.00,0.44,0.13}{\textbf{{#1}}}}
+    \newcommand{\OperatorTok}[1]{\textcolor[rgb]{0.40,0.40,0.40}{{#1}}}
+    \newcommand{\BuiltInTok}[1]{{#1}}
+    \newcommand{\ExtensionTok}[1]{{#1}}
+    \newcommand{\PreprocessorTok}[1]{\textcolor[rgb]{0.74,0.48,0.00}{{#1}}}
+    \newcommand{\AttributeTok}[1]{\textcolor[rgb]{0.49,0.56,0.16}{{#1}}}
+    \newcommand{\InformationTok}[1]{\textcolor[rgb]{0.38,0.63,0.69}{\textbf{\textit{{#1}}}}}
+    \newcommand{\WarningTok}[1]{\textcolor[rgb]{0.38,0.63,0.69}{\textbf{\textit{{#1}}}}}
+    
+    
+    % Define a nice break command that doesn't care if a line doesn't already
+    % exist.
+    \def\br{\hspace*{\fill} \\* }
+    % Math Jax compatability definitions
+    \def\gt{>}
+    \def\lt{<}
+    % Document parameters
+    \title{NeuralNet}
+    
+    
+    
+
+    % Pygments definitions
+    
+\makeatletter
+\def\PY@reset{\let\PY@it=\relax \let\PY@bf=\relax%
+    \let\PY@ul=\relax \let\PY@tc=\relax%
+    \let\PY@bc=\relax \let\PY@ff=\relax}
+\def\PY@tok#1{\csname PY@tok@#1\endcsname}
+\def\PY@toks#1+{\ifx\relax#1\empty\else%
+    \PY@tok{#1}\expandafter\PY@toks\fi}
+\def\PY@do#1{\PY@bc{\PY@tc{\PY@ul{%
+    \PY@it{\PY@bf{\PY@ff{#1}}}}}}}
+\def\PY#1#2{\PY@reset\PY@toks#1+\relax+\PY@do{#2}}
+
+\expandafter\def\csname PY@tok@w\endcsname{\def\PY@tc##1{\textcolor[rgb]{0.73,0.73,0.73}{##1}}}
+\expandafter\def\csname PY@tok@c\endcsname{\let\PY@it=\textit\def\PY@tc##1{\textcolor[rgb]{0.25,0.50,0.50}{##1}}}
+\expandafter\def\csname PY@tok@cp\endcsname{\def\PY@tc##1{\textcolor[rgb]{0.74,0.48,0.00}{##1}}}
+\expandafter\def\csname PY@tok@k\endcsname{\let\PY@bf=\textbf\def\PY@tc##1{\textcolor[rgb]{0.00,0.50,0.00}{##1}}}
+\expandafter\def\csname PY@tok@kp\endcsname{\def\PY@tc##1{\textcolor[rgb]{0.00,0.50,0.00}{##1}}}
+\expandafter\def\csname PY@tok@kt\endcsname{\def\PY@tc##1{\textcolor[rgb]{0.69,0.00,0.25}{##1}}}
+\expandafter\def\csname PY@tok@o\endcsname{\def\PY@tc##1{\textcolor[rgb]{0.40,0.40,0.40}{##1}}}
+\expandafter\def\csname PY@tok@ow\endcsname{\let\PY@bf=\textbf\def\PY@tc##1{\textcolor[rgb]{0.67,0.13,1.00}{##1}}}
+\expandafter\def\csname PY@tok@nb\endcsname{\def\PY@tc##1{\textcolor[rgb]{0.00,0.50,0.00}{##1}}}
+\expandafter\def\csname PY@tok@nf\endcsname{\def\PY@tc##1{\textcolor[rgb]{0.00,0.00,1.00}{##1}}}
+\expandafter\def\csname PY@tok@nc\endcsname{\let\PY@bf=\textbf\def\PY@tc##1{\textcolor[rgb]{0.00,0.00,1.00}{##1}}}
+\expandafter\def\csname PY@tok@nn\endcsname{\let\PY@bf=\textbf\def\PY@tc##1{\textcolor[rgb]{0.00,0.00,1.00}{##1}}}
+\expandafter\def\csname PY@tok@ne\endcsname{\let\PY@bf=\textbf\def\PY@tc##1{\textcolor[rgb]{0.82,0.25,0.23}{##1}}}
+\expandafter\def\csname PY@tok@nv\endcsname{\def\PY@tc##1{\textcolor[rgb]{0.10,0.09,0.49}{##1}}}
+\expandafter\def\csname PY@tok@no\endcsname{\def\PY@tc##1{\textcolor[rgb]{0.53,0.00,0.00}{##1}}}
+\expandafter\def\csname PY@tok@nl\endcsname{\def\PY@tc##1{\textcolor[rgb]{0.63,0.63,0.00}{##1}}}
+\expandafter\def\csname PY@tok@ni\endcsname{\let\PY@bf=\textbf\def\PY@tc##1{\textcolor[rgb]{0.60,0.60,0.60}{##1}}}
+\expandafter\def\csname PY@tok@na\endcsname{\def\PY@tc##1{\textcolor[rgb]{0.49,0.56,0.16}{##1}}}
+\expandafter\def\csname PY@tok@nt\endcsname{\let\PY@bf=\textbf\def\PY@tc##1{\textcolor[rgb]{0.00,0.50,0.00}{##1}}}
+\expandafter\def\csname PY@tok@nd\endcsname{\def\PY@tc##1{\textcolor[rgb]{0.67,0.13,1.00}{##1}}}
+\expandafter\def\csname PY@tok@s\endcsname{\def\PY@tc##1{\textcolor[rgb]{0.73,0.13,0.13}{##1}}}
+\expandafter\def\csname PY@tok@sd\endcsname{\let\PY@it=\textit\def\PY@tc##1{\textcolor[rgb]{0.73,0.13,0.13}{##1}}}
+\expandafter\def\csname PY@tok@si\endcsname{\let\PY@bf=\textbf\def\PY@tc##1{\textcolor[rgb]{0.73,0.40,0.53}{##1}}}
+\expandafter\def\csname PY@tok@se\endcsname{\let\PY@bf=\textbf\def\PY@tc##1{\textcolor[rgb]{0.73,0.40,0.13}{##1}}}
+\expandafter\def\csname PY@tok@sr\endcsname{\def\PY@tc##1{\textcolor[rgb]{0.73,0.40,0.53}{##1}}}
+\expandafter\def\csname PY@tok@ss\endcsname{\def\PY@tc##1{\textcolor[rgb]{0.10,0.09,0.49}{##1}}}
+\expandafter\def\csname PY@tok@sx\endcsname{\def\PY@tc##1{\textcolor[rgb]{0.00,0.50,0.00}{##1}}}
+\expandafter\def\csname PY@tok@m\endcsname{\def\PY@tc##1{\textcolor[rgb]{0.40,0.40,0.40}{##1}}}
+\expandafter\def\csname PY@tok@gh\endcsname{\let\PY@bf=\textbf\def\PY@tc##1{\textcolor[rgb]{0.00,0.00,0.50}{##1}}}
+\expandafter\def\csname PY@tok@gu\endcsname{\let\PY@bf=\textbf\def\PY@tc##1{\textcolor[rgb]{0.50,0.00,0.50}{##1}}}
+\expandafter\def\csname PY@tok@gd\endcsname{\def\PY@tc##1{\textcolor[rgb]{0.63,0.00,0.00}{##1}}}
+\expandafter\def\csname PY@tok@gi\endcsname{\def\PY@tc##1{\textcolor[rgb]{0.00,0.63,0.00}{##1}}}
+\expandafter\def\csname PY@tok@gr\endcsname{\def\PY@tc##1{\textcolor[rgb]{1.00,0.00,0.00}{##1}}}
+\expandafter\def\csname PY@tok@ge\endcsname{\let\PY@it=\textit}
+\expandafter\def\csname PY@tok@gs\endcsname{\let\PY@bf=\textbf}
+\expandafter\def\csname PY@tok@gp\endcsname{\let\PY@bf=\textbf\def\PY@tc##1{\textcolor[rgb]{0.00,0.00,0.50}{##1}}}
+\expandafter\def\csname PY@tok@go\endcsname{\def\PY@tc##1{\textcolor[rgb]{0.53,0.53,0.53}{##1}}}
+\expandafter\def\csname PY@tok@gt\endcsname{\def\PY@tc##1{\textcolor[rgb]{0.00,0.27,0.87}{##1}}}
+\expandafter\def\csname PY@tok@err\endcsname{\def\PY@bc##1{\setlength{\fboxsep}{0pt}\fcolorbox[rgb]{1.00,0.00,0.00}{1,1,1}{\strut ##1}}}
+\expandafter\def\csname PY@tok@kc\endcsname{\let\PY@bf=\textbf\def\PY@tc##1{\textcolor[rgb]{0.00,0.50,0.00}{##1}}}
+\expandafter\def\csname PY@tok@kd\endcsname{\let\PY@bf=\textbf\def\PY@tc##1{\textcolor[rgb]{0.00,0.50,0.00}{##1}}}
+\expandafter\def\csname PY@tok@kn\endcsname{\let\PY@bf=\textbf\def\PY@tc##1{\textcolor[rgb]{0.00,0.50,0.00}{##1}}}
+\expandafter\def\csname PY@tok@kr\endcsname{\let\PY@bf=\textbf\def\PY@tc##1{\textcolor[rgb]{0.00,0.50,0.00}{##1}}}
+\expandafter\def\csname PY@tok@bp\endcsname{\def\PY@tc##1{\textcolor[rgb]{0.00,0.50,0.00}{##1}}}
+\expandafter\def\csname PY@tok@fm\endcsname{\def\PY@tc##1{\textcolor[rgb]{0.00,0.00,1.00}{##1}}}
+\expandafter\def\csname PY@tok@vc\endcsname{\def\PY@tc##1{\textcolor[rgb]{0.10,0.09,0.49}{##1}}}
+\expandafter\def\csname PY@tok@vg\endcsname{\def\PY@tc##1{\textcolor[rgb]{0.10,0.09,0.49}{##1}}}
+\expandafter\def\csname PY@tok@vi\endcsname{\def\PY@tc##1{\textcolor[rgb]{0.10,0.09,0.49}{##1}}}
+\expandafter\def\csname PY@tok@vm\endcsname{\def\PY@tc##1{\textcolor[rgb]{0.10,0.09,0.49}{##1}}}
+\expandafter\def\csname PY@tok@sa\endcsname{\def\PY@tc##1{\textcolor[rgb]{0.73,0.13,0.13}{##1}}}
+\expandafter\def\csname PY@tok@sb\endcsname{\def\PY@tc##1{\textcolor[rgb]{0.73,0.13,0.13}{##1}}}
+\expandafter\def\csname PY@tok@sc\endcsname{\def\PY@tc##1{\textcolor[rgb]{0.73,0.13,0.13}{##1}}}
+\expandafter\def\csname PY@tok@dl\endcsname{\def\PY@tc##1{\textcolor[rgb]{0.73,0.13,0.13}{##1}}}
+\expandafter\def\csname PY@tok@s2\endcsname{\def\PY@tc##1{\textcolor[rgb]{0.73,0.13,0.13}{##1}}}
+\expandafter\def\csname PY@tok@sh\endcsname{\def\PY@tc##1{\textcolor[rgb]{0.73,0.13,0.13}{##1}}}
+\expandafter\def\csname PY@tok@s1\endcsname{\def\PY@tc##1{\textcolor[rgb]{0.73,0.13,0.13}{##1}}}
+\expandafter\def\csname PY@tok@mb\endcsname{\def\PY@tc##1{\textcolor[rgb]{0.40,0.40,0.40}{##1}}}
+\expandafter\def\csname PY@tok@mf\endcsname{\def\PY@tc##1{\textcolor[rgb]{0.40,0.40,0.40}{##1}}}
+\expandafter\def\csname PY@tok@mh\endcsname{\def\PY@tc##1{\textcolor[rgb]{0.40,0.40,0.40}{##1}}}
+\expandafter\def\csname PY@tok@mi\endcsname{\def\PY@tc##1{\textcolor[rgb]{0.40,0.40,0.40}{##1}}}
+\expandafter\def\csname PY@tok@il\endcsname{\def\PY@tc##1{\textcolor[rgb]{0.40,0.40,0.40}{##1}}}
+\expandafter\def\csname PY@tok@mo\endcsname{\def\PY@tc##1{\textcolor[rgb]{0.40,0.40,0.40}{##1}}}
+\expandafter\def\csname PY@tok@ch\endcsname{\let\PY@it=\textit\def\PY@tc##1{\textcolor[rgb]{0.25,0.50,0.50}{##1}}}
+\expandafter\def\csname PY@tok@cm\endcsname{\let\PY@it=\textit\def\PY@tc##1{\textcolor[rgb]{0.25,0.50,0.50}{##1}}}
+\expandafter\def\csname PY@tok@cpf\endcsname{\let\PY@it=\textit\def\PY@tc##1{\textcolor[rgb]{0.25,0.50,0.50}{##1}}}
+\expandafter\def\csname PY@tok@c1\endcsname{\let\PY@it=\textit\def\PY@tc##1{\textcolor[rgb]{0.25,0.50,0.50}{##1}}}
+\expandafter\def\csname PY@tok@cs\endcsname{\let\PY@it=\textit\def\PY@tc##1{\textcolor[rgb]{0.25,0.50,0.50}{##1}}}
+
+\def\PYZbs{\char`\\}
+\def\PYZus{\char`\_}
+\def\PYZob{\char`\{}
+\def\PYZcb{\char`\}}
+\def\PYZca{\char`\^}
+\def\PYZam{\char`\&}
+\def\PYZlt{\char`\<}
+\def\PYZgt{\char`\>}
+\def\PYZsh{\char`\#}
+\def\PYZpc{\char`\%}
+\def\PYZdl{\char`\$}
+\def\PYZhy{\char`\-}
+\def\PYZsq{\char`\'}
+\def\PYZdq{\char`\"}
+\def\PYZti{\char`\~}
+% for compatibility with earlier versions
+\def\PYZat{@}
+\def\PYZlb{[}
+\def\PYZrb{]}
+\makeatother
+
+
+    % Exact colors from NB
+    \definecolor{incolor}{rgb}{0.0, 0.0, 0.5}
+    \definecolor{outcolor}{rgb}{0.545, 0.0, 0.0}
+
+
+
+    
+    % Prevent overflowing lines due to hard-to-break entities
+    \sloppy 
+    % Setup hyperref package
+    \hypersetup{
+      breaklinks=true,  % so long urls are correctly broken across lines
+      colorlinks=true,
+      urlcolor=urlcolor,
+      linkcolor=linkcolor,
+      citecolor=citecolor,
+      }
+    % Slightly bigger margins than the latex defaults
+    
+    \geometry{verbose,tmargin=1in,bmargin=1in,lmargin=1in,rmargin=1in}
+    
+    
+
+    \begin{document}
+    
+    
+    \maketitle
+    
+    
+
+    
+    \hypertarget{data-analysis-and-machine-learning-neural-networks-from-the-simple-perceptron-to-deep-learning}{%
+\section{Data Analysis and Machine Learning: Neural networks, from the
+simple perceptron to deep
+learning}\label{data-analysis-and-machine-learning-neural-networks-from-the-simple-perceptron-to-deep-learning}}
+
+\textbf{Morten Hjorth-Jensen}, Department of Physics, University of Oslo
+and Department of Physics and Astronomy and National Superconducting
+Cyclotron Laboratory, Michigan State University
+
+Date: \textbf{Oct 4, 2018}
+
+Copyright 1999-2018, Morten Hjorth-Jensen. Released under CC
+Attribution-NonCommercial 4.0 license
+
+\hypertarget{neural-networks}{%
+\subsection{Neural networks}\label{neural-networks}}
+
+Artificial neural networks are computational systems that can learn to
+perform tasks by considering examples, generally without being
+programmed with any task-specific rules. It is supposed to mimic a
+biological system, wherein neurons interact by sending signals in the
+form of mathematical functions between layers. All layers can contain an
+arbitrary number of neurons, and each connection is represented by a
+weight variable.
+
+\hypertarget{artificial-neurons}{%
+\subsection{Artificial neurons}\label{artificial-neurons}}
+
+The field of artificial neural networks has a long history of
+development, and is closely connected with the advancement of computer
+science and computers in general. A model of artificial neurons was
+first developed by McCulloch and Pitts in 1943 to study signal
+processing in the brain and has later been refined by others. The
+general idea is to mimic neural networks in the human brain, which is
+composed of billions of neurons that communicate with each other by
+sending electrical signals. Each neuron accumulates its incoming
+signals, which must exceed an activation threshold to yield an output.
+If the threshold is not overcome, the neuron remains inactive, i.e.~has
+zero output.
+
+This behaviour has inspired a simple mathematical model for an
+artificial neuron.
+
+    \hypertarget{artificialNeuron}{}
+
+\[
+\begin{equation}
+ y = f\left(\sum_{i=1}^n w_ix_i\right) = f(u)
+\label{artificialNeuron} \tag{1}
+\end{equation}
+\]
+
+    Here, the output \(y\) of the neuron is the value of its activation
+function, which have as input a weighted sum of signals
+\(x_i, \dots ,x_n\) received by \(n\) other neurons.
+
+Conceptually, it is helpful to divide neural networks into four
+categories: 1. general purpose neural networks for supervised learning,
+
+\begin{enumerate}
+\def\labelenumi{\arabic{enumi}.}
+\setcounter{enumi}{1}
+\item
+  neural networks designed specifically for image processing, the most
+  prominent example of this class being Convolutional Neural Networks
+  (CNNs),
+\item
+  neural networks for sequential data such as Recurrent Neural Networks
+  (RNNs), and
+\item
+  neural networks for unsupervised learning such as Deep Boltzmann
+  Machines.
+\end{enumerate}
+
+In natural science, DNNs and CNNs have already found numerous
+applications. In statistical physics, they have been applied to detect
+phase transitions in 2D Ising and Potts models, lattice gauge theories,
+and different phases of polymers, or solving the Navier-Stokes equation
+in weather forecasting. Deep learning has also found interesting
+applications in quantum physics. Various quantum phase transitions can
+be detected and studied using DNNs and CNNs, topological phases, and
+even non-equilibrium many-body localization. Representing quantum states
+as DNNs quantum state tomography are among some of the impressive
+achievements to reveal the potential of DNNs to facilitate the study of
+quantum systems.
+
+In quantum information theory, it has been shown that one can perform
+gate decompositions with the help of neural.
+
+The applications are not limited to the natural sciences. There is a
+plethora of applications in essentially all disciplines, from the
+humanities to life science and medicine.
+
+\hypertarget{neural-network-types}{%
+\subsection{Neural network types}\label{neural-network-types}}
+
+An artificial neural network (ANN), is a computational model that
+consists of layers of connected neurons, or nodes or units. We will
+refer to these interchangeably as units or nodes, and sometimes as
+neurons.
+
+It is supposed to mimic a biological nervous system by letting each
+neuron interact with other neurons by sending signals in the form of
+mathematical functions between layers. A wide variety of different ANNs
+have been developed, but most of them consist of an input layer, an
+output layer and eventual layers in-between, called \emph{hidden
+layers}. All layers can contain an arbitrary number of nodes, and each
+connection between two nodes is associated with a weight variable.
+
+Neural networks (also called neural nets) are neural-inspired nonlinear
+models for supervised learning. As we will see, neural nets can be
+viewed as natural, more powerful extensions of supervised learning
+methods such as linear and logistic regression and soft-max methods we
+discussed earlier.
+
+\hypertarget{feed-forward-neural-networks}{%
+\subsection{Feed-forward neural
+networks}\label{feed-forward-neural-networks}}
+
+The feed-forward neural network (FFNN) was the first and simplest type
+of ANNs that were devised. In this network, the information moves in
+only one direction: forward through the layers.
+
+Nodes are represented by circles, while the arrows display the
+connections between the nodes, including the direction of information
+flow. Additionally, each arrow corresponds to a weight variable (figure
+to come). We observe that each node in a layer is connected to
+\emph{all} nodes in the subsequent layer, making this a so-called
+\emph{fully-connected} FFNN.
+
+\hypertarget{convolutional-neural-network}{%
+\subsection{Convolutional Neural
+Network}\label{convolutional-neural-network}}
+
+A different variant of FFNNs are \emph{convolutional neural networks}
+(CNNs), which have a connectivity pattern inspired by the animal visual
+cortex. Individual neurons in the visual cortex only respond to stimuli
+from small sub-regions of the visual field, called a receptive field.
+This makes the neurons well-suited to exploit the strong spatially local
+correlation present in natural images. The response of each neuron can
+be approximated mathematically as a convolution operation. (figure to
+come)
+
+Convolutional neural networks emulate the behaviour of neurons in the
+visual cortex by enforcing a \emph{local} connectivity pattern between
+nodes of adjacent layers: Each node in a convolutional layer is
+connected only to a subset of the nodes in the previous layer, in
+contrast to the fully-connected FFNN. Often, CNNs consist of several
+convolutional layers that learn local features of the input, with a
+fully-connected layer at the end, which gathers all the local data and
+produces the outputs. They have wide applications in image and video
+recognition.
+
+\hypertarget{recurrent-neural-networks}{%
+\subsection{Recurrent neural networks}\label{recurrent-neural-networks}}
+
+So far we have only mentioned ANNs where information flows in one
+direction: forward. \emph{Recurrent neural networks} on the other hand,
+have connections between nodes that form directed \emph{cycles}. This
+creates a form of internal memory which are able to capture information
+on what has been calculated before; the output is dependent on the
+previous computations. Recurrent NNs make use of sequential information
+by performing the same task for every element in a sequence, where each
+element depends on previous elements. An example of such information is
+sentences, making recurrent NNs especially well-suited for handwriting
+and speech recognition.
+
+\hypertarget{other-types-of-networks}{%
+\subsection{Other types of networks}\label{other-types-of-networks}}
+
+There are many other kinds of ANNs that have been developed. One type
+that is specifically designed for interpolation in multidimensional
+space is the radial basis function (RBF) network. RBFs are typically
+made up of three layers: an input layer, a hidden layer with non-linear
+radial symmetric activation functions and a linear output layer
+(`'linear'' here means that each node in the output layer has a linear
+activation function). The layers are normally fully-connected and there
+are no cycles, thus RBFs can be viewed as a type of fully-connected
+FFNN. They are however usually treated as a separate type of NN due the
+unusual activation functions.
+
+\hypertarget{multilayer-perceptrons}{%
+\subsection{Multilayer perceptrons}\label{multilayer-perceptrons}}
+
+One uses often so-called fully-connected feed-forward neural networks
+with three or more layers (an input layer, one or more hidden layers and
+an output layer) consisting of neurons that have non-linear activation
+functions.
+
+Such networks are often called \emph{multilayer perceptrons} (MLPs).
+
+\hypertarget{why-multilayer-perceptrons}{%
+\subsection{Why multilayer
+perceptrons?}\label{why-multilayer-perceptrons}}
+
+According to the \emph{Universal approximation theorem}, a feed-forward
+neural network with just a single hidden layer containing a finite
+number of neurons can approximate a continuous multidimensional function
+to arbitrary accuracy, assuming the activation function for the hidden
+layer is a \textbf{non-constant, bounded and monotonically-increasing
+continuous function}.
+
+Note that the requirements on the activation function only applies to
+the hidden layer, the output nodes are always assumed to be linear, so
+as to not restrict the range of output values.
+
+\hypertarget{mathematical-model}{%
+\subsection{Mathematical model}\label{mathematical-model}}
+
+The output \(y\) is produced via the activation function \(f\)
+
+    \[
+y = f\left(\sum_{i=1}^n w_ix_i + b_i\right) = f(z),
+\]
+
+    This function receives \(x_i\) as inputs. Here the activation
+\(z=(\sum_{i=1}^n w_ix_i+b_i)\). In an FFNN of such neurons, the
+\emph{inputs} \(x_i\) are the \emph{outputs} of the neurons in the
+preceding layer. Furthermore, an MLP is fully-connected, which means
+that each neuron receives a weighted sum of the outputs of \emph{all}
+neurons in the previous layer.
+
+\hypertarget{mathematical-model}{%
+\subsection{Mathematical model}\label{mathematical-model}}
+
+First, for each node \(i\) in the first hidden layer, we calculate a
+weighted sum \(z_i^1\) of the input coordinates \(x_j\),
+
+    \hypertarget{_auto1}{}
+
+\[
+\begin{equation} z_i^1 = \sum_{j=1}^{M} w_{ij}^1 x_j + b_i^1
+\label{_auto1} \tag{2}
+\end{equation}
+\]
+
+    Here \(b_i\) is the so-called bias which is normally needed in case of
+zero activation weights or inputs. How to fix the biases and the weights
+will be discussed below. The value of \(z_i^1\) is the argument to the
+activation function \(f_i\) of each node \(i\), The variable \(M\)
+stands for all possible inputs to a given node \(i\) in the first layer.
+We define the output \(y_i^1\) of all neurons in layer 1 as
+
+    \hypertarget{outputLayer1}{}
+
+\[
+\begin{equation}
+ y_i^1 = f(z_i^1) = f\left(\sum_{j=1}^M w_{ij}^1 x_j  + b_i^1\right)
+\label{outputLayer1} \tag{3}
+\end{equation}
+\]
+
+    where we assume that all nodes in the same layer have identical
+activation functions, hence the notation \(f\). In general, we could
+assume in the more general case that different layers have different
+activation functions. In this case we would identify these functions
+with a superscript \(l\) for the \(l\)-th layer,
+
+    \hypertarget{generalLayer}{}
+
+\[
+\begin{equation}
+ y_i^l = f^l(u_i^l) = f^l\left(\sum_{j=1}^{N_{l-1}} w_{ij}^l y_j^{l-1} + b_i^l\right)
+\label{generalLayer} \tag{4}
+\end{equation}
+\]
+
+    where \(N_l\) is the number of nodes in layer \(l\). When the output of
+all the nodes in the first hidden layer are computed, the values of the
+subsequent layer can be calculated and so forth until the output is
+obtained.
+
+\hypertarget{mathematical-model}{%
+\subsection{Mathematical model}\label{mathematical-model}}
+
+The output of neuron \(i\) in layer 2 is thus,
+
+    \hypertarget{_auto2}{}
+
+\[
+\begin{equation}
+ y_i^2 = f^2\left(\sum_{j=1}^N w_{ij}^2 y_j^1 + b_i^2\right) 
+\label{_auto2} \tag{5}
+\end{equation}
+\]
+
+    \hypertarget{outputLayer2}{}
+
+\[
+\begin{equation} 
+ = f^2\left[\sum_{j=1}^N w_{ij}^2f^1\left(\sum_{k=1}^M w_{jk}^1 x_k + b_j^1\right) + b_i^2\right]
+\label{outputLayer2} \tag{6}
+\end{equation}
+\]
+
+    where we have substituted \(y_k^1\) with the inputs \(x_k\). Finally,
+the ANN output reads
+
+    \hypertarget{_auto3}{}
+
+\[
+\begin{equation}
+ y_i^3 = f^3\left(\sum_{j=1}^N w_{ij}^3 y_j^2 + b_i^3\right) 
+\label{_auto3} \tag{7}
+\end{equation}
+\]
+
+    \hypertarget{_auto4}{}
+
+\[
+\begin{equation} 
+ = f_3\left[\sum_{j} w_{ij}^3 f^2\left(\sum_{k} w_{jk}^2 f^1\left(\sum_{m} w_{km}^1 x_m + b_k^1\right) + b_j^2\right)
+  + b_1^3\right]
+\label{_auto4} \tag{8}
+\end{equation}
+\]
+
+    \hypertarget{mathematical-model}{%
+\subsection{Mathematical model}\label{mathematical-model}}
+
+We can generalize this expression to an MLP with \(l\) hidden layers.
+The complete functional form is,
+
+    \hypertarget{completeNN}{}
+
+\[
+\begin{equation}
+y^{l+1}_i = f^{l+1}\left[\!\sum_{j=1}^{N_l} w_{ij}^3 f^l\left(\sum_{k=1}^{N_{l-1}}w_{jk}^{l-1}\left(\dots f^1\left(\sum_{n=1}^{N_0} w_{mn}^1 x_n+ b_m^1\right)\dots\right)+b_k^2\right)+b_1^3\right] 
+\label{completeNN} \tag{9}
+\end{equation}
+\]
+
+    which illustrates a basic property of MLPs: The only independent
+variables are the input values \(x_n\).
+
+\hypertarget{mathematical-model}{%
+\subsection{Mathematical model}\label{mathematical-model}}
+
+This confirms that an MLP, despite its quite convoluted mathematical
+form, is nothing more than an analytic function, specifically a mapping
+of real-valued vectors
+\(\hat{x} \in \mathbb{R}^n \rightarrow \hat{y} \in \mathbb{R}^m\).
+
+Furthermore, the flexibility and universality of an MLP can be
+illustrated by realizing that the expression is essentially a nested sum
+of scaled activation functions of the form
+
+    \hypertarget{_auto5}{}
+
+\[
+\begin{equation}
+ f(x) = c_1 f(c_2 x + c_3) + c_4
+\label{_auto5} \tag{10}
+\end{equation}
+\]
+
+    where the parameters \(c_i\) are weights and biases. By adjusting these
+parameters, the activation functions can be shifted up and down or left
+and right, change slope or be rescaled which is the key to the
+flexibility of a neural network.
+
+\hypertarget{matrix-vector-notation}{%
+\subsubsection{Matrix-vector notation}\label{matrix-vector-notation}}
+
+We can introduce a more convenient notation for the activations in an A
+NN.
+
+Additionally, we can represent the biases and activations as layer-wise
+column vectors \(\hat{b}_l\) and \(\hat{y}_l\), so that the \(i\)-th
+element of each vector is the bias \(b_i^l\) and activation \(y_i^l\) of
+node \(i\) in layer \(l\) respectively.
+
+We have that \(\mathrm{W}_l\) is an \(N_{l-1} \times N_l\) matrix, while
+\(\hat{b}_l\) and \(\hat{y}_l\) are \(N_l \times 1\) column vectors.
+With this notation, the sum becomes a matrix-vector multiplication, and
+we can write the equation for the activations of hidden layer 2
+(assuming three nodes for simplicity) as
+
+    \hypertarget{_auto6}{}
+
+\[
+\begin{equation}
+ \hat{y}_2 = f_2(\mathrm{W}_2 \hat{y}_{1} + \hat{b}_{2}) = 
+ f_2\left(\left[\begin{array}{ccc}
+    w^2_{11} &w^2_{12} &w^2_{13} \\
+    w^2_{21} &w^2_{22} &w^2_{23} \\
+    w^2_{31} &w^2_{32} &w^2_{33} \\
+    \end{array} \right] \cdot
+    \left[\begin{array}{c}
+           y^1_1 \\
+           y^1_2 \\
+           y^1_3 \\
+          \end{array}\right] + 
+    \left[\begin{array}{c}
+           b^2_1 \\
+           b^2_2 \\
+           b^2_3 \\
+          \end{array}\right]\right).
+\label{_auto6} \tag{11}
+\end{equation}
+\]
+
+    \hypertarget{matrix-vector-notation-and-activation}{%
+\subsubsection{Matrix-vector notation and
+activation}\label{matrix-vector-notation-and-activation}}
+
+The activation of node \(i\) in layer 2 is
+
+    \hypertarget{_auto7}{}
+
+\[
+\begin{equation}
+ y^2_i = f_2\Bigr(w^2_{i1}y^1_1 + w^2_{i2}y^1_2 + w^2_{i3}y^1_3 + b^2_i\Bigr) = 
+ f_2\left(\sum_{j=1}^3 w^2_{ij} y_j^1 + b^2_i\right).
+\label{_auto7} \tag{12}
+\end{equation}
+\]
+
+    This is not just a convenient and compact notation, but also a useful
+and intuitive way to think about MLPs: The output is calculated by a
+series of matrix-vector multiplications and vector additions that are
+used as input to the activation functions. For each operation
+\(\mathrm{W}_l \hat{y}_{l-1}\) we move forward one layer.
+
+\hypertarget{activation-functions}{%
+\subsubsection{Activation functions}\label{activation-functions}}
+
+A property that characterizes a neural network, other than its
+connectivity, is the choice of activation function(s). As described in,
+the following restrictions are imposed on an activation function for a
+FFNN to fulfill the universal approximation theorem
+
+\begin{itemize}
+\item
+  Non-constant
+\item
+  Bounded
+\item
+  Monotonically-increasing
+\item
+  Continuous
+\end{itemize}
+
+\hypertarget{activation-functions-logistic-and-hyperbolic-ones}{%
+\subsubsection{Activation functions, Logistic and Hyperbolic
+ones}\label{activation-functions-logistic-and-hyperbolic-ones}}
+
+The second requirement excludes all linear functions. Furthermore, in a
+MLP with only linear activation functions, each layer simply performs a
+linear transformation of its inputs.
+
+Regardless of the number of layers, the output of the NN will be nothing
+but a linear function of the inputs. Thus we need to introduce some kind
+of non-linearity to the NN to be able to fit non-linear functions
+Typical examples are the logistic \emph{Sigmoid}
+
+    \[
+f(x) = \frac{1}{1 + e^{-x}},
+\]
+
+    and the \emph{hyperbolic tangent} function
+
+    \[
+f(x) = \tanh(x)
+\]
+
+    \hypertarget{relevance}{%
+\subsubsection{Relevance}\label{relevance}}
+
+The \emph{sigmoid} function are more biologically plausible because the
+output of inactive neurons are zero. Such activation function are called
+\emph{one-sided}. However, it has been shown that the hyperbolic tangent
+performs better than the sigmoid for training MLPs. has become the most
+popular for \emph{deep neural networks}
+
+    \begin{Verbatim}[commandchars=\\\{\}]
+{\color{incolor}In [{\color{incolor}1}]:} \PY{o}{\PYZpc{}}\PY{k}{matplotlib} inline
+        
+        \PY{l+s+sd}{\PYZdq{}\PYZdq{}\PYZdq{}The sigmoid function (or the logistic curve) is a }
+        \PY{l+s+sd}{function that takes any real number, z, and outputs a number (0,1).}
+        \PY{l+s+sd}{It is useful in neural networks for assigning weights on a relative scale.}
+        \PY{l+s+sd}{The value z is the weighted sum of parameters involved in the learning algorithm.\PYZdq{}\PYZdq{}\PYZdq{}}
+        
+        \PY{k+kn}{import} \PY{n+nn}{numpy}
+        \PY{k+kn}{import} \PY{n+nn}{matplotlib}\PY{n+nn}{.}\PY{n+nn}{pyplot} \PY{k}{as} \PY{n+nn}{plt}
+        \PY{k+kn}{import} \PY{n+nn}{math} \PY{k}{as} \PY{n+nn}{mt}
+        
+        \PY{n}{z} \PY{o}{=} \PY{n}{numpy}\PY{o}{.}\PY{n}{arange}\PY{p}{(}\PY{o}{\PYZhy{}}\PY{l+m+mi}{5}\PY{p}{,} \PY{l+m+mi}{5}\PY{p}{,} \PY{o}{.}\PY{l+m+mi}{1}\PY{p}{)}
+        \PY{n}{sigma\PYZus{}fn} \PY{o}{=} \PY{n}{numpy}\PY{o}{.}\PY{n}{vectorize}\PY{p}{(}\PY{k}{lambda} \PY{n}{z}\PY{p}{:} \PY{l+m+mi}{1}\PY{o}{/}\PY{p}{(}\PY{l+m+mi}{1}\PY{o}{+}\PY{n}{numpy}\PY{o}{.}\PY{n}{exp}\PY{p}{(}\PY{o}{\PYZhy{}}\PY{n}{z}\PY{p}{)}\PY{p}{)}\PY{p}{)}
+        \PY{n}{sigma} \PY{o}{=} \PY{n}{sigma\PYZus{}fn}\PY{p}{(}\PY{n}{z}\PY{p}{)}
+        
+        \PY{n}{fig} \PY{o}{=} \PY{n}{plt}\PY{o}{.}\PY{n}{figure}\PY{p}{(}\PY{p}{)}
+        \PY{n}{ax} \PY{o}{=} \PY{n}{fig}\PY{o}{.}\PY{n}{add\PYZus{}subplot}\PY{p}{(}\PY{l+m+mi}{111}\PY{p}{)}
+        \PY{n}{ax}\PY{o}{.}\PY{n}{plot}\PY{p}{(}\PY{n}{z}\PY{p}{,} \PY{n}{sigma}\PY{p}{)}
+        \PY{n}{ax}\PY{o}{.}\PY{n}{set\PYZus{}ylim}\PY{p}{(}\PY{p}{[}\PY{o}{\PYZhy{}}\PY{l+m+mf}{0.1}\PY{p}{,} \PY{l+m+mf}{1.1}\PY{p}{]}\PY{p}{)}
+        \PY{n}{ax}\PY{o}{.}\PY{n}{set\PYZus{}xlim}\PY{p}{(}\PY{p}{[}\PY{o}{\PYZhy{}}\PY{l+m+mi}{5}\PY{p}{,}\PY{l+m+mi}{5}\PY{p}{]}\PY{p}{)}
+        \PY{n}{ax}\PY{o}{.}\PY{n}{grid}\PY{p}{(}\PY{k+kc}{True}\PY{p}{)}
+        \PY{n}{ax}\PY{o}{.}\PY{n}{set\PYZus{}xlabel}\PY{p}{(}\PY{l+s+s1}{\PYZsq{}}\PY{l+s+s1}{z}\PY{l+s+s1}{\PYZsq{}}\PY{p}{)}
+        \PY{n}{ax}\PY{o}{.}\PY{n}{set\PYZus{}title}\PY{p}{(}\PY{l+s+s1}{\PYZsq{}}\PY{l+s+s1}{sigmoid function}\PY{l+s+s1}{\PYZsq{}}\PY{p}{)}
+        
+        \PY{n}{plt}\PY{o}{.}\PY{n}{show}\PY{p}{(}\PY{p}{)}
+        
+        \PY{l+s+sd}{\PYZdq{}\PYZdq{}\PYZdq{}Step Function\PYZdq{}\PYZdq{}\PYZdq{}}
+        \PY{n}{z} \PY{o}{=} \PY{n}{numpy}\PY{o}{.}\PY{n}{arange}\PY{p}{(}\PY{o}{\PYZhy{}}\PY{l+m+mi}{5}\PY{p}{,} \PY{l+m+mi}{5}\PY{p}{,} \PY{o}{.}\PY{l+m+mi}{02}\PY{p}{)}
+        \PY{n}{step\PYZus{}fn} \PY{o}{=} \PY{n}{numpy}\PY{o}{.}\PY{n}{vectorize}\PY{p}{(}\PY{k}{lambda} \PY{n}{z}\PY{p}{:} \PY{l+m+mf}{1.0} \PY{k}{if} \PY{n}{z} \PY{o}{\PYZgt{}}\PY{o}{=} \PY{l+m+mf}{0.0} \PY{k}{else} \PY{l+m+mf}{0.0}\PY{p}{)}
+        \PY{n}{step} \PY{o}{=} \PY{n}{step\PYZus{}fn}\PY{p}{(}\PY{n}{z}\PY{p}{)}
+        
+        \PY{n}{fig} \PY{o}{=} \PY{n}{plt}\PY{o}{.}\PY{n}{figure}\PY{p}{(}\PY{p}{)}
+        \PY{n}{ax} \PY{o}{=} \PY{n}{fig}\PY{o}{.}\PY{n}{add\PYZus{}subplot}\PY{p}{(}\PY{l+m+mi}{111}\PY{p}{)}
+        \PY{n}{ax}\PY{o}{.}\PY{n}{plot}\PY{p}{(}\PY{n}{z}\PY{p}{,} \PY{n}{step}\PY{p}{)}
+        \PY{n}{ax}\PY{o}{.}\PY{n}{set\PYZus{}ylim}\PY{p}{(}\PY{p}{[}\PY{o}{\PYZhy{}}\PY{l+m+mf}{0.5}\PY{p}{,} \PY{l+m+mf}{1.5}\PY{p}{]}\PY{p}{)}
+        \PY{n}{ax}\PY{o}{.}\PY{n}{set\PYZus{}xlim}\PY{p}{(}\PY{p}{[}\PY{o}{\PYZhy{}}\PY{l+m+mi}{5}\PY{p}{,}\PY{l+m+mi}{5}\PY{p}{]}\PY{p}{)}
+        \PY{n}{ax}\PY{o}{.}\PY{n}{grid}\PY{p}{(}\PY{k+kc}{True}\PY{p}{)}
+        \PY{n}{ax}\PY{o}{.}\PY{n}{set\PYZus{}xlabel}\PY{p}{(}\PY{l+s+s1}{\PYZsq{}}\PY{l+s+s1}{z}\PY{l+s+s1}{\PYZsq{}}\PY{p}{)}
+        \PY{n}{ax}\PY{o}{.}\PY{n}{set\PYZus{}title}\PY{p}{(}\PY{l+s+s1}{\PYZsq{}}\PY{l+s+s1}{step function}\PY{l+s+s1}{\PYZsq{}}\PY{p}{)}
+        
+        \PY{n}{plt}\PY{o}{.}\PY{n}{show}\PY{p}{(}\PY{p}{)}
+        
+        \PY{l+s+sd}{\PYZdq{}\PYZdq{}\PYZdq{}Sine Function\PYZdq{}\PYZdq{}\PYZdq{}}
+        \PY{n}{z} \PY{o}{=} \PY{n}{numpy}\PY{o}{.}\PY{n}{arange}\PY{p}{(}\PY{o}{\PYZhy{}}\PY{l+m+mi}{2}\PY{o}{*}\PY{n}{mt}\PY{o}{.}\PY{n}{pi}\PY{p}{,} \PY{l+m+mi}{2}\PY{o}{*}\PY{n}{mt}\PY{o}{.}\PY{n}{pi}\PY{p}{,} \PY{l+m+mf}{0.1}\PY{p}{)}
+        \PY{n}{t} \PY{o}{=} \PY{n}{numpy}\PY{o}{.}\PY{n}{sin}\PY{p}{(}\PY{n}{z}\PY{p}{)}
+        
+        \PY{n}{fig} \PY{o}{=} \PY{n}{plt}\PY{o}{.}\PY{n}{figure}\PY{p}{(}\PY{p}{)}
+        \PY{n}{ax} \PY{o}{=} \PY{n}{fig}\PY{o}{.}\PY{n}{add\PYZus{}subplot}\PY{p}{(}\PY{l+m+mi}{111}\PY{p}{)}
+        \PY{n}{ax}\PY{o}{.}\PY{n}{plot}\PY{p}{(}\PY{n}{z}\PY{p}{,} \PY{n}{t}\PY{p}{)}
+        \PY{n}{ax}\PY{o}{.}\PY{n}{set\PYZus{}ylim}\PY{p}{(}\PY{p}{[}\PY{o}{\PYZhy{}}\PY{l+m+mf}{1.0}\PY{p}{,} \PY{l+m+mf}{1.0}\PY{p}{]}\PY{p}{)}
+        \PY{n}{ax}\PY{o}{.}\PY{n}{set\PYZus{}xlim}\PY{p}{(}\PY{p}{[}\PY{o}{\PYZhy{}}\PY{l+m+mi}{2}\PY{o}{*}\PY{n}{mt}\PY{o}{.}\PY{n}{pi}\PY{p}{,}\PY{l+m+mi}{2}\PY{o}{*}\PY{n}{mt}\PY{o}{.}\PY{n}{pi}\PY{p}{]}\PY{p}{)}
+        \PY{n}{ax}\PY{o}{.}\PY{n}{grid}\PY{p}{(}\PY{k+kc}{True}\PY{p}{)}
+        \PY{n}{ax}\PY{o}{.}\PY{n}{set\PYZus{}xlabel}\PY{p}{(}\PY{l+s+s1}{\PYZsq{}}\PY{l+s+s1}{z}\PY{l+s+s1}{\PYZsq{}}\PY{p}{)}
+        \PY{n}{ax}\PY{o}{.}\PY{n}{set\PYZus{}title}\PY{p}{(}\PY{l+s+s1}{\PYZsq{}}\PY{l+s+s1}{sine function}\PY{l+s+s1}{\PYZsq{}}\PY{p}{)}
+        
+        \PY{n}{plt}\PY{o}{.}\PY{n}{show}\PY{p}{(}\PY{p}{)}
+        
+        \PY{l+s+sd}{\PYZdq{}\PYZdq{}\PYZdq{}Plots a graph of the squashing function used by a rectified linear}
+        \PY{l+s+sd}{unit\PYZdq{}\PYZdq{}\PYZdq{}}
+        \PY{n}{z} \PY{o}{=} \PY{n}{numpy}\PY{o}{.}\PY{n}{arange}\PY{p}{(}\PY{o}{\PYZhy{}}\PY{l+m+mi}{2}\PY{p}{,} \PY{l+m+mi}{2}\PY{p}{,} \PY{o}{.}\PY{l+m+mi}{1}\PY{p}{)}
+        \PY{n}{zero} \PY{o}{=} \PY{n}{numpy}\PY{o}{.}\PY{n}{zeros}\PY{p}{(}\PY{n+nb}{len}\PY{p}{(}\PY{n}{z}\PY{p}{)}\PY{p}{)}
+        \PY{n}{y} \PY{o}{=} \PY{n}{numpy}\PY{o}{.}\PY{n}{max}\PY{p}{(}\PY{p}{[}\PY{n}{zero}\PY{p}{,} \PY{n}{z}\PY{p}{]}\PY{p}{,} \PY{n}{axis}\PY{o}{=}\PY{l+m+mi}{0}\PY{p}{)}
+        
+        \PY{n}{fig} \PY{o}{=} \PY{n}{plt}\PY{o}{.}\PY{n}{figure}\PY{p}{(}\PY{p}{)}
+        \PY{n}{ax} \PY{o}{=} \PY{n}{fig}\PY{o}{.}\PY{n}{add\PYZus{}subplot}\PY{p}{(}\PY{l+m+mi}{111}\PY{p}{)}
+        \PY{n}{ax}\PY{o}{.}\PY{n}{plot}\PY{p}{(}\PY{n}{z}\PY{p}{,} \PY{n}{y}\PY{p}{)}
+        \PY{n}{ax}\PY{o}{.}\PY{n}{set\PYZus{}ylim}\PY{p}{(}\PY{p}{[}\PY{o}{\PYZhy{}}\PY{l+m+mf}{2.0}\PY{p}{,} \PY{l+m+mf}{2.0}\PY{p}{]}\PY{p}{)}
+        \PY{n}{ax}\PY{o}{.}\PY{n}{set\PYZus{}xlim}\PY{p}{(}\PY{p}{[}\PY{o}{\PYZhy{}}\PY{l+m+mf}{2.0}\PY{p}{,} \PY{l+m+mf}{2.0}\PY{p}{]}\PY{p}{)}
+        \PY{n}{ax}\PY{o}{.}\PY{n}{grid}\PY{p}{(}\PY{k+kc}{True}\PY{p}{)}
+        \PY{n}{ax}\PY{o}{.}\PY{n}{set\PYZus{}xlabel}\PY{p}{(}\PY{l+s+s1}{\PYZsq{}}\PY{l+s+s1}{z}\PY{l+s+s1}{\PYZsq{}}\PY{p}{)}
+        \PY{n}{ax}\PY{o}{.}\PY{n}{set\PYZus{}title}\PY{p}{(}\PY{l+s+s1}{\PYZsq{}}\PY{l+s+s1}{Rectified linear unit}\PY{l+s+s1}{\PYZsq{}}\PY{p}{)}
+        
+        \PY{n}{plt}\PY{o}{.}\PY{n}{show}\PY{p}{(}\PY{p}{)}
+\end{Verbatim}
+
+
+    \hypertarget{the-multilayer-perceptron-mlp}{%
+\subsection{The multilayer perceptron
+(MLP)}\label{the-multilayer-perceptron-mlp}}
+
+The multilayer perceptron is a very popular, and easy to implement
+approach, to deep learning. It consists of 1. A neural network with one
+or more layers of nodes between the input and the output nodes.
+
+\begin{enumerate}
+\def\labelenumi{\arabic{enumi}.}
+\setcounter{enumi}{1}
+\item
+  The multilayer network structure, or architecture, or topology,
+  consists of an input layer, one or more hidden layers, and one output
+  layer.
+\item
+  The input nodes pass values to the first hidden layer, its nodes pass
+  the information on to the second and so on till we reach the output
+  layer.
+\end{enumerate}
+
+As a convention it is normal to call a network with one layer of input
+units, one layer of hidden units and one layer of output units as a
+two-layer network. A network with two layers of hidden units is called a
+three-layer network etc etc.
+
+For an MLP network there is no direct connection between the output
+nodes/neurons/units and the input nodes/neurons/units. Hereafter we will
+call the various entities of a layer for nodes. There are also no
+connections within a single layer.
+
+The number of input nodes does not need to equal the number of output
+nodes. This applies also to the hidden layers. Each layer may have its
+own number of nodes and activation functions.
+
+The hidden layers have their name from the fact that they are not linked
+to observables and as we will see below when we define the so-called
+activation \(\hat{z}\), we can think of this as a basis expansion of the
+original inputs \(\hat{x}\). The difference however between neural
+networks and say linear regression is that now these basis functions
+(which will correspond to the weights in the network) are learned from
+data. This results in an important difference between neural networks
+and deep learning approaches on one side and methods like logistic
+regression or linear regression and their modifications on the other
+side.
+
+\hypertarget{from-one-to-many-layers-the-universal-approximation-theorem}{%
+\subsection{From one to many layers, the universal approximation
+theorem}\label{from-one-to-many-layers-the-universal-approximation-theorem}}
+
+A neural network with only one layer, what we called the simple
+perceptron, is best suited if we have a standard binary model with clear
+(linear) boundaries between the outcomes. As such it could equally well
+be replaced by standard linear regression or logistic regression.
+Networks with one or more hidden layers approximate systems with more
+complex boundaries.
+
+As stated earlier, an important theorem in studies of neural networks,
+restated without proof here, is the
+\href{http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.441.7873\&rep=rep1\&type=pdf}{universal
+approximation theorem}.
+
+It states that a feed-forward network with a single hidden layer
+containing a finite number of neurons can approximate continuous
+functions on compact subsets of real functions. The theorem thus states
+that simple neural networks can represent a wide variety of interesting
+functions when given appropriate parameters. It is the multilayer
+feedforward architecture itself which gives neural networks the
+potential of being universal approximators.
+
+\hypertarget{deriving-the-back-propagation-code-for-a-multilayer-perceptron-model}{%
+\subsection{Deriving the back propagation code for a multilayer
+perceptron
+model}\label{deriving-the-back-propagation-code-for-a-multilayer-perceptron-model}}
+
+\textbf{Note: figures will be inserted later!}
+
+As we have seen now in a feed forward network, we can express the final
+output of our network in terms of basic matrix-vector multiplications.
+The unknowwn quantities are our weights \(w_{ij}\) and we need to find
+an algorithm for changing them so that our errors are as small as
+possible. This leads us to the famous
+\href{https://www.nature.com/articles/323533a0}{back propagation
+algorithm}.
+
+The questions we want to ask are how do changes in the biases and the
+weights in our network change the cost function and how can we use the
+final output to modify the weights?
+
+To derive these equations let us start with a plain regression problem
+and define our cost function as
+
+    \[
+{\cal C}(\hat{W})  =  \frac{1}{2}\sum_{i=1}^n\left(y_i - t_i\right)^2,
+\]
+
+    where the \(t_i\)s are our \(n\) targets (the values we want to
+reproduce), while the outputs of the network after having propagated all
+inputs \(\hat{x}\) are given by \(y_i\). Below we will demonstrate how
+the basic equations arising from the back propagation algorithm can be
+modified in order to study classification problems with \(K\) classes.
+
+\hypertarget{definitions}{%
+\subsection{Definitions}\label{definitions}}
+
+With our definition of the targets \(\hat{t}\), the outputs of the
+network \(\hat{y}\) and the inputs \(\hat{x}\) we define now the
+activation \(z_j^l\) of node/neuron/unit \(j\) of the \(l\)-th layer as
+a function of the bias, the weights which add up from the previous layer
+\(l-1\) and the forward passes/outputs \(\hat{a}^{l-1}\) from the
+previous layer as
+
+    \[
+z_j^l = \sum_{i=1}^{M_{l-1}}w_{ij}^la_j^{l-1}+b_j^l,
+\]
+
+    where \(b_k^l\) are the biases from layer \(l\). Here \(M_{l-1}\)
+represents the total number of nodes/neurons/units of layer \(l-1\). The
+figure here illustrates this equation. We can rewrite this in a more
+compact form as the matrix-vector products we discussed earlier,
+
+    \[
+\hat{z}^l = \left(\hat{W}^l\right)^T\hat{a}^{l-1}+\hat{b}^l.
+\]
+
+    With the activation values \(\hat{z}^l\) we can in turn define the
+output of layer \(l\) as \(\hat{a}^l = f(\hat{z}^l)\) where \(f\) is our
+activation function. In the examples here we will use the sigmoid
+function discussed in our logistic regression lectures. We will also use
+the same activation function \(f\) for all layers and their nodes. It
+means we have
+
+    \[
+a_j^l = f(z_j^l) = \frac{1}{1+\exp{-(z_j^l)}}.
+\]
+
+    \hypertarget{derivatives-and-the-chain-rule}{%
+\subsection{Derivatives and the chain
+rule}\label{derivatives-and-the-chain-rule}}
+
+From the definition of the activation \(z_j^l\) we have
+
+    \[
+\frac{\partial z_j^l}{\partial w_{ji}^l} = a_i^{l-1},
+\]
+
+    and
+
+    \[
+\frac{\partial z_j^l}{\partial a_i^{l-1}} = w_{ji}^l.
+\]
+
+    With our definition of the activation function we have that (note that
+this function depends only on \(z_j^l\))
+
+    \[
+\frac{\partial a_j^l}{\partial z_j^{l}} = a_j^l(1-a_j^l)=f(z_j^l)(1-f(z_j^l)).
+\]
+
+    \hypertarget{derivative-of-the-cost-function}{%
+\subsection{Derivative of the cost
+function}\label{derivative-of-the-cost-function}}
+
+With these definitions we can now compute the derivative of the cost
+function in terms of the weights.
+
+Let us specialize to the output layer \(l=L\). Our cost function is
+
+    \[
+{\cal C}(\hat{W^L})  =  \frac{1}{2}\sum_{i=1}^n\left(y_i - t_i\right)^2=\frac{1}{2}\sum_{i=1}^n\left(a_i^L - t_i\right)^2,
+\]
+
+    The derivative of this function with respect to the weights is
+
+    \[
+\frac{\partial{\cal C}(\hat{W^L})}{\partial w_{jk}^L}  =  \left(a_j^L - t_j\right)\frac{\partial a_j^L}{\partial w_{jk}^{L}},
+\]
+
+    The last partial derivative can easily be computed and reads (by
+applying the chain rule)
+
+    \[
+\frac{\partial a_j^L}{\partial w_{jk}^{L}} = \frac{\partial a_j^L}{\partial z_{j}^{L}}\frac{\partial z_j^L}{\partial w_{jk}^{L}}=a_j^L(1-a_j^L)a_k^{L-1},
+\]
+
+    \hypertarget{bringing-it-together-first-back-propagation-equation}{%
+\subsection{Bringing it together, first back propagation
+equation}\label{bringing-it-together-first-back-propagation-equation}}
+
+We have thus
+
+    \[
+\frac{\partial{\cal C}(\hat{W^L})}{\partial w_{jk}^L}  =  \left(a_j^L - t_j\right)a_j^L(1-a_j^L)a_k^{L-1},
+\]
+
+    Defining
+
+    \[
+\delta_j^L = a_j^L(1-a_j^L)\left(a_j^L - t_j\right) = f'(z_j^L)\frac{\partial {\cal C}}{\partial (a_j^L)},
+\]
+
+    and using the Hadamard product of two vectors we can write this as
+
+    \[
+\hat{\delta}^L = f'(\hat{z}^L)\circ\frac{\partial {\cal C}}{\partial (\hat{a}L)}.
+\]
+
+    This is an important expression. The second term on the right handside
+measures how fast the cost function is changing as a function of the
+\(j\)th output activation. If, for example, the cost function doesn't
+depend much on a particular output node \(j\), then \(\delta_j^L\) will
+be small, which is what we would expect. The first term on the right,
+measures how fast the activation function \(f\) is changing at a given
+activation value \(z_j^L\).
+
+Notice that everything in the above equations is easily computed. In
+particular, we compute \(z_j^L\) while computing the behaviour of the
+network, and it is only a small additional overhead to compute
+\(f'(z^L_j)\). The exact form of the derivative with respect to the
+output depends on the form of the cost function. However, provided the
+cost function is known there should be little trouble in calculating
+
+    \[
+\frac{\partial {\cal C}}{\partial (a_j^L)}
+\]
+
+    With the definition of \(\delta_j^L\) we have a more compact definition
+of the derivative of the cost function in terms of the weights, namely
+
+    \[
+\frac{\partial{\cal C}(\hat{W^L})}{\partial w_{jk}^L}  =  \delta_j^La_k^{L-1}.
+\]
+
+    \hypertarget{derivatives-in-terms-of-z_jl}{%
+\subsection{\texorpdfstring{Derivatives in terms of
+\(z_j^L\)}{Derivatives in terms of z\_j\^{}L}}\label{derivatives-in-terms-of-z_jl}}
+
+It is also easy to see that our previous equation can be written as
+
+    \[
+\delta_j^L =\frac{\partial {\cal C}}{\partial z_j^L}= \frac{\partial {\cal C}}{\partial a_j^L}\frac{\partial a_j^L}{\partial z_j^L},
+\]
+
+    which can also be interpreted as the partial derivative of the cost
+function with respect to the biases \(b_j^L\), namely
+
+    \[
+\delta_j^L = \frac{\partial {\cal C}}{\partial b_j^L}\frac{\partial b_j^L}{\partial z_j^L}=\frac{\partial {\cal C}}{\partial b_j^L},
+\]
+
+    That is, the error \(\delta_j^L\) is exactly equal to the rate of change
+of the cost function as a function of the bias. \#\# Bringing it
+together
+
+We have now three equations that are essential for the computations of
+the derivatives of the cost function at the output layer. These
+equations are needed to start the algorithm and they are
+
+\textbf{The starting equations.}
+
+    \hypertarget{_auto8}{}
+
+\[
+\begin{equation}
+\frac{\partial{\cal C}(\hat{W^L})}{\partial w_{jk}^L}  =  \delta_j^La_k^{L-1},
+\label{_auto8} \tag{13}
+\end{equation}
+\]
+
+    and
+
+    \hypertarget{_auto9}{}
+
+\[
+\begin{equation}
+\delta_j^L = f'(z_j^L)\frac{\partial {\cal C}}{\partial (a_j^L)},
+\label{_auto9} \tag{14}
+\end{equation}
+\]
+
+    and
+
+    \hypertarget{_auto10}{}
+
+\[
+\begin{equation}
+\delta_j^L = \frac{\partial {\cal C}}{\partial b_j^L},
+\label{_auto10} \tag{15}
+\end{equation}
+\]
+
+    An interesting consequence of the above equations is that when the
+activation \(a_k^{L-1}\) is small, the gradient term, that is the
+derivative of the cost function with respect to the weights, will also
+tend to be small. We say then that the weight learns slowly, meaning
+that it changes slowly when we minimize the weights via say gradient
+descent. In this case we say the system learns slowly.
+
+Another interesting feature is that is when the activation function,
+represented by the sigmoid function here, is rather flat when we move
+towards its end values \(0\) and \(1\) (see the above Python codes). In
+these cases, the derivatives of the activation function will also be
+close to zero, meaning again that the gradients will be small and the
+network learns slowly again.
+
+We need a fourth equation and we are set. We are going to propagate
+backwards in order to the determine the weights and biases. In order to
+do so we need to represent the error in the layer before the final one
+\(L-1\) in terms of the errors in the final output layer.
+
+\hypertarget{final-back-propagating-equation}{%
+\subsection{Final back propagating
+equation}\label{final-back-propagating-equation}}
+
+We have that (replacing \(L\) with a general layer \(l\))
+
+    \[
+\delta_j^l =\frac{\partial {\cal C}}{\partial z_j^l}.
+\]
+
+    We want to express this in terms of the equations for layer \(l+1\).
+Using the chain rule and summing over all \(k\) entries we have
+
+    \[
+\delta_j^l =\sum_k \frac{\partial {\cal C}}{\partial z_k^{l+1}}\frac{\partial z_k^{l+1}}{\partial z_j^{l}}=\sum_k \delta_k^{l+1}\frac{\partial z_k^{l+1}}{\partial z_j^{l}},
+\]
+
+    and recalling that
+
+    \[
+z_j^{l+1} = \sum_{i=1}^{M_{l}}w_{ij}^{l+1}a_j^{l}+b_j^{l+1},
+\]
+
+    we obtain
+
+    \[
+\delta_j^l =\sum_k \delta_k^{l+1}w_{kj}^{l+1}f'(z_j^l),
+\]
+
+    This is our final equation.
+
+We are now ready to set up the algorithm for back propagation and
+learning the weights and biases.
+
+\hypertarget{setting-up-the-back-propagation-algorithm}{%
+\subsection{Setting up the Back propagation
+algorithm}\label{setting-up-the-back-propagation-algorithm}}
+
+The four equations provide us with a way of computing the gradient of
+the cost function. Let us write this out in the form of an algorithm.
+
+First, we set up the input data \(\hat{x}\) and the activations
+\(\hat{z}_1\) of the input layer and compute the activation function and
+the pertinent outputs \(\hat{a}^1\).
+
+Secondly, we perform then the feed forward till we reach the output
+layer and compute all \(\hat{z}_l\) of the input layer and compute the
+activation function and the pertinent outputs \(\hat{a}^l\) for
+\(l=2,3,\dots,L\).
+
+Thereafter we compute the ouput error \(\hat{\delta}^L\) by computing
+all
+
+    \[
+\delta_j^L = f'(z_j^L)\frac{\partial {\cal C}}{\partial (a_j^L)}.
+\]
+
+    Then we compute the back propagate error for each \(l=L-1,L-2,\dots,2\)
+as
+
+    \[
+\delta_j^l =\sum_k \sum_k \delta_k^{l+1}w_{kj}^{l+1}f'(z_j^l).
+\]
+
+    Finally, we update the weights and the biases using gradient descent for
+each \(l=L-1,L-2,dots,2\) and update the weights and biases according to
+the rules
+
+    \[
+w_{jk}^l\leftarrow  = w_{jk}^l- \eta \delta_j^la_k^{l-1},
+\]
+
+    \[
+b_j^l \leftarrow b_j^l-\eta \frac{\partial {\cal C}}{\partial b_j^L},
+\]
+
+    The parameter \(\eta\) is the learning parameter discussed in connection
+with the gradient descent methods. Here it is convenient to use
+stochastic gradient descent (see the examples below) with mini-batches
+with an outer loop that steps through multiple epochs of training.
+
+\hypertarget{setting-up-a-multi-layer-perceptron-model-for-classification}{%
+\subsection{Setting up a Multi-layer perceptron model for
+classification}\label{setting-up-a-multi-layer-perceptron-model-for-classification}}
+
+We are now gong to develop an example based on the MNIST data base. This
+is a classification problem and we need to use our cross-entropy
+function we discussed in connection with logistic regression. The
+cross-entropy defines our cost function for the classificaton problems
+with neural networks.
+
+In binary classification with two classes \((0, 1)\) we define the
+logistic/sigmoid function as the probability that a particular input is
+in class \(0\) or \(1\). This is possible because the logistic function
+takes any input from the real numbers and inputs a number between 0 and
+1, and can therefore be interpreted as a probability. It also has other
+nice properties, such as a derivative that is simple to calculate.
+
+For an input \(\boldsymbol{a}\) from the hidden layer, the probability
+that the input \(\boldsymbol{x}\) is in class 0 or 1 is just. We let
+\(\theta\) represent the unknown weights and biases to be adjusted by
+our equations). The variable \(x\) represents our activation values
+\(z\). We have
+
+    \[
+P(y = 0 \mid \boldsymbol{x}, \boldsymbol{\theta}) = \frac{1}{1 + \exp (- \boldsymbol{x}} ,
+\]
+
+    and
+
+    \[
+P(y = 1 \mid \boldsymbol{x}, \boldsymbol{\theta}) = 1 - P(y = 0 \mid \boldsymbol{x}, \boldsymbol{\theta}) ,
+\]
+
+    where \(y \in \{0, 1\}\) and \(\boldsymbol{\theta}\) represents the
+weights and biases of our network.
+
+\hypertarget{defining-the-cost-function}{%
+\subsection{Defining the cost
+function}\label{defining-the-cost-function}}
+
+Our cost function is given as (see the Logistic regression lectures)
+
+    \[
+\mathcal{C}(\boldsymbol{\theta}) = - \ln P(\mathcal{D} \mid \boldsymbol{\theta}) = - \sum_{i=1}^n
+y_i \ln[P(y_i = 0)] + (1 - y_i) \ln [1 - P(y_i = 0)] = \sum_{i=1}^n \mathcal{L}_i(\boldsymbol{\theta}) .
+\]
+
+    This last equality means that we can interpret our \emph{cost} function
+as a sum over the \emph{loss} function for each point in the dataset
+\(\mathcal{L}_i(\boldsymbol{\theta})\).\\
+The negative sign is just so that we can think about our algorithm as
+minimizing a positive number, rather than maximizing a negative number.
+
+In \emph{multiclass} classification it is common to treat each integer
+label as a so called \emph{one-hot} vector:
+
+\(y = 5 \quad \rightarrow \quad \boldsymbol{y} = (0, 0, 0, 0, 0, 1, 0, 0, 0, 0) ,\)
+and
+
+\(y = 1 \quad \rightarrow \quad \boldsymbol{y} = (0, 1, 0, 0, 0, 0, 0, 0, 0, 0) ,\)
+
+i.e.~a binary bit string of length \(C\), where \(C = 10\) is the number
+of classes in the MNIST dataset (numbers from \(0\) to \(9\))..
+
+If \(\boldsymbol{x}_i\) is the \(i\)-th input (image), \(y_{ic}\) refers
+to the \(c\)-th component of the \(i\)-th output vector
+\(\boldsymbol{y}_i\).\\
+The probability of \(\boldsymbol{x}_i\) being in class \(c\) will be
+given by the softmax function:
+
+    \[
+P(y_{ic} = 1 \mid \boldsymbol{x}_i, \boldsymbol{\theta}) = \frac{\exp{((\boldsymbol{a}_i^{hidden})^T \boldsymbol{w}_c)}}
+{\sum_{c'=0}^{C-1} \exp{((\boldsymbol{a}_i^{hidden})^T \boldsymbol{w}_{c'})}} ,
+\]
+
+    which reduces to the logistic function in the binary case.\\
+The likelihood of this \(C\)-class classifier is now given as:
+
+    \[
+P(\mathcal{D} \mid \boldsymbol{\theta}) = \prod_{i=1}^n \prod_{c=0}^{C-1} [P(y_{ic} = 1)]^{y_{ic}} .
+\]
+
+    Again we take the negative log-likelihood to define our cost function:
+
+    \[
+\mathcal{C}(\boldsymbol{\theta}) = - \ln P(\mathcal{D} \mid \boldsymbol{\theta}) = - \sum_{i=1}^n \sum_{c=0}^{C-1}
+y_{ic} \ln[P(y_{ic} = 1)] = \sum_{i=1}^n
+\mathcal{L}_i(\boldsymbol{\theta}) .
+\]
+
+    The back propagation equations need now only a small change, namely the
+definition of a new cost function. We are thus ready to use the same
+equations as before! We leave it as an exercise in project 2 to derive
+these equations.
+
+\hypertarget{developing-a-code-for-doing-neural-networks-with-back-propagation}{%
+\subsection{Developing a code for doing neural networks with back
+propagation}\label{developing-a-code-for-doing-neural-networks-with-back-propagation}}
+
+One can identify a set of key steps when using neural networks to solve
+supervised learning problems:
+
+\begin{enumerate}
+\def\labelenumi{\arabic{enumi}.}
+\item
+  Collect and pre-process data
+\item
+  Define model and architecture
+\item
+  Choose cost function and optimizer
+\item
+  Train the model
+\item
+  Evaluate model performance on test data
+\item
+  Adjust hyperparameters (if necessary, network architecture)
+\end{enumerate}
+
+\hypertarget{collect-and-pre-process-data}{%
+\subsection{Collect and pre-process
+data}\label{collect-and-pre-process-data}}
+
+Here we will be using the MNIST dataset, which is readily available
+through the \textbf{scikit-learn} package. You may also find it for
+example \href{http://yann.lecun.com/exdb/mnist/}{here}.\\
+The \emph{MNIST} (Modified National Institute of Standards and
+Technology) database is a large database of handwritten digits that is
+commonly used for training various image processing systems.\\
+The MNIST dataset consists of 70 000 images of size 28x28 pixels, each
+labeled from 0 to 9.\\
+The scikit-learn dataset we will use consists of a selection of 1797
+images of size \(8\times 8\) collected and processed from this database.
+
+To feed data into a feed-forward neural network we need to represent the
+inputs as a feature matrix \(X = (n_{inputs}, n_{features})\). Each row
+represents an \emph{input}, in this case a handwritten digit, and each
+column represents a \emph{feature}, in this case a pixel. The correct
+answers, also known as \emph{labels} or \emph{targets} are represented
+as a 1D array of integers \(Y = (n_{inputs}) = (5, 3, 1, 8,...)\).
+
+As an example, say we want to build a neural network using supervised
+learning to predict Body-Mass Index (BMI) from measurements of height
+(in m)\\
+and weight (in kg). If we have measurements of 5 people the feature
+matrix could be for example:
+
+\[ X = \begin{bmatrix}
+1.85 & 81\\
+1.71 & 65\\
+1.95 & 103\\
+1.55 & 42\\
+1.63 & 56
+\end{bmatrix} ,\]
+
+and the targets would be:
+
+\[ Y = (23.7, 22.2, 27.1, 17.5, 21.1) \]
+
+Since each input image is a 2D matrix, we need to flatten the image
+(i.e. ``unravel'' the 2D matrix into a 1D array) to turn the data into a
+feature matrix. This means we lose all spatial information in the image,
+such as locality and translational invariance. More complicated
+architectures such as Convolutional Neural Networks can take advantage
+of such information, and are most commonly applied when analyzing
+images.
+
+    \begin{Verbatim}[commandchars=\\\{\}]
+{\color{incolor}In [{\color{incolor}2}]:} \PY{c+c1}{\PYZsh{} import necessary packages}
+        \PY{k+kn}{import} \PY{n+nn}{numpy} \PY{k}{as} \PY{n+nn}{np}
+        \PY{k+kn}{import} \PY{n+nn}{matplotlib}\PY{n+nn}{.}\PY{n+nn}{pyplot} \PY{k}{as} \PY{n+nn}{plt}
+        \PY{k+kn}{from} \PY{n+nn}{sklearn} \PY{k}{import} \PY{n}{datasets}
+        
+        
+        \PY{c+c1}{\PYZsh{} ensure the same random numbers appear every time}
+        \PY{n}{np}\PY{o}{.}\PY{n}{random}\PY{o}{.}\PY{n}{seed}\PY{p}{(}\PY{l+m+mi}{0}\PY{p}{)}
+        
+        \PY{c+c1}{\PYZsh{} display images in notebook}
+        \PY{o}{\PYZpc{}}\PY{k}{matplotlib} inline
+        \PY{n}{plt}\PY{o}{.}\PY{n}{rcParams}\PY{p}{[}\PY{l+s+s1}{\PYZsq{}}\PY{l+s+s1}{figure.figsize}\PY{l+s+s1}{\PYZsq{}}\PY{p}{]} \PY{o}{=} \PY{p}{(}\PY{l+m+mi}{12}\PY{p}{,}\PY{l+m+mi}{12}\PY{p}{)}
+        
+        
+        \PY{c+c1}{\PYZsh{} download MNIST dataset}
+        \PY{n}{digits} \PY{o}{=} \PY{n}{datasets}\PY{o}{.}\PY{n}{load\PYZus{}digits}\PY{p}{(}\PY{p}{)}
+        
+        \PY{c+c1}{\PYZsh{} define inputs and labels}
+        \PY{n}{inputs} \PY{o}{=} \PY{n}{digits}\PY{o}{.}\PY{n}{images}
+        \PY{n}{labels} \PY{o}{=} \PY{n}{digits}\PY{o}{.}\PY{n}{target}
+        
+        \PY{n+nb}{print}\PY{p}{(}\PY{l+s+s2}{\PYZdq{}}\PY{l+s+s2}{inputs = (n\PYZus{}inputs, pixel\PYZus{}width, pixel\PYZus{}height) = }\PY{l+s+s2}{\PYZdq{}} \PY{o}{+} \PY{n+nb}{str}\PY{p}{(}\PY{n}{inputs}\PY{o}{.}\PY{n}{shape}\PY{p}{)}\PY{p}{)}
+        \PY{n+nb}{print}\PY{p}{(}\PY{l+s+s2}{\PYZdq{}}\PY{l+s+s2}{labels = (n\PYZus{}inputs) = }\PY{l+s+s2}{\PYZdq{}} \PY{o}{+} \PY{n+nb}{str}\PY{p}{(}\PY{n}{labels}\PY{o}{.}\PY{n}{shape}\PY{p}{)}\PY{p}{)}
+        
+        
+        \PY{c+c1}{\PYZsh{} flatten the image}
+        \PY{c+c1}{\PYZsh{} the value \PYZhy{}1 means dimension is inferred from the remaining dimensions: 8x8 = 64}
+        \PY{n}{n\PYZus{}inputs} \PY{o}{=} \PY{n+nb}{len}\PY{p}{(}\PY{n}{inputs}\PY{p}{)}
+        \PY{n}{inputs} \PY{o}{=} \PY{n}{inputs}\PY{o}{.}\PY{n}{reshape}\PY{p}{(}\PY{n}{n\PYZus{}inputs}\PY{p}{,} \PY{o}{\PYZhy{}}\PY{l+m+mi}{1}\PY{p}{)}
+        \PY{n+nb}{print}\PY{p}{(}\PY{l+s+s2}{\PYZdq{}}\PY{l+s+s2}{X = (n\PYZus{}inputs, n\PYZus{}features) = }\PY{l+s+s2}{\PYZdq{}} \PY{o}{+} \PY{n+nb}{str}\PY{p}{(}\PY{n}{inputs}\PY{o}{.}\PY{n}{shape}\PY{p}{)}\PY{p}{)}
+        
+        
+        \PY{c+c1}{\PYZsh{} choose some random images to display}
+        \PY{n}{indices} \PY{o}{=} \PY{n}{np}\PY{o}{.}\PY{n}{arange}\PY{p}{(}\PY{n}{n\PYZus{}inputs}\PY{p}{)}
+        \PY{n}{random\PYZus{}indices} \PY{o}{=} \PY{n}{np}\PY{o}{.}\PY{n}{random}\PY{o}{.}\PY{n}{choice}\PY{p}{(}\PY{n}{indices}\PY{p}{,} \PY{n}{size}\PY{o}{=}\PY{l+m+mi}{5}\PY{p}{)}
+        
+        \PY{k}{for} \PY{n}{i}\PY{p}{,} \PY{n}{image} \PY{o+ow}{in} \PY{n+nb}{enumerate}\PY{p}{(}\PY{n}{digits}\PY{o}{.}\PY{n}{images}\PY{p}{[}\PY{n}{random\PYZus{}indices}\PY{p}{]}\PY{p}{)}\PY{p}{:}
+            \PY{n}{plt}\PY{o}{.}\PY{n}{subplot}\PY{p}{(}\PY{l+m+mi}{1}\PY{p}{,} \PY{l+m+mi}{5}\PY{p}{,} \PY{n}{i}\PY{o}{+}\PY{l+m+mi}{1}\PY{p}{)}
+            \PY{n}{plt}\PY{o}{.}\PY{n}{axis}\PY{p}{(}\PY{l+s+s1}{\PYZsq{}}\PY{l+s+s1}{off}\PY{l+s+s1}{\PYZsq{}}\PY{p}{)}
+            \PY{n}{plt}\PY{o}{.}\PY{n}{imshow}\PY{p}{(}\PY{n}{image}\PY{p}{,} \PY{n}{cmap}\PY{o}{=}\PY{n}{plt}\PY{o}{.}\PY{n}{cm}\PY{o}{.}\PY{n}{gray\PYZus{}r}\PY{p}{,} \PY{n}{interpolation}\PY{o}{=}\PY{l+s+s1}{\PYZsq{}}\PY{l+s+s1}{nearest}\PY{l+s+s1}{\PYZsq{}}\PY{p}{)}
+            \PY{n}{plt}\PY{o}{.}\PY{n}{title}\PY{p}{(}\PY{l+s+s2}{\PYZdq{}}\PY{l+s+s2}{Label: }\PY{l+s+si}{\PYZpc{}d}\PY{l+s+s2}{\PYZdq{}} \PY{o}{\PYZpc{}} \PY{n}{digits}\PY{o}{.}\PY{n}{target}\PY{p}{[}\PY{n}{random\PYZus{}indices}\PY{p}{[}\PY{n}{i}\PY{p}{]}\PY{p}{]}\PY{p}{)}
+        \PY{n}{plt}\PY{o}{.}\PY{n}{show}\PY{p}{(}\PY{p}{)}
+\end{Verbatim}
+
+
+    \begin{Verbatim}[commandchars=\\\{\}]
+inputs = (n\_inputs, pixel\_width, pixel\_height) = (1797, 8, 8)
+labels = (n\_inputs) = (1797,)
+X = (n\_inputs, n\_features) = (1797, 64)
+
+    \end{Verbatim}
+
+    \begin{center}
+    \adjustimage{max size={0.9\linewidth}{0.9\paperheight}}{output_98_1.png}
+    \end{center}
+    { \hspace*{\fill} \\}
+    
+    \hypertarget{train-and-test-datasets}{%
+\subsection{Train and test datasets}\label{train-and-test-datasets}}
+
+Performing analysis before partitioning the dataset is a major error,
+that can lead to incorrect conclusions.
+
+We will reserve \(80 \%\) of our dataset for training and \(20 \%\) for
+testing.
+
+It is important that the train and test datasets are drawn randomly from
+our dataset, to ensure no bias in the sampling.\\
+Say you are taking measurements of weather data to predict the weather
+in the coming 5 days. You don't want to train your model on measurements
+taken from the hours 00.00 to 12.00, and then test it on data collected
+from 12.00 to 24.00.
+
+    \begin{Verbatim}[commandchars=\\\{\}]
+{\color{incolor}In [{\color{incolor}3}]:} \PY{k+kn}{from} \PY{n+nn}{sklearn}\PY{n+nn}{.}\PY{n+nn}{model\PYZus{}selection} \PY{k}{import} \PY{n}{train\PYZus{}test\PYZus{}split}
+        
+        \PY{c+c1}{\PYZsh{} one\PYZhy{}liner from scikit\PYZhy{}learn library}
+        \PY{n}{train\PYZus{}size} \PY{o}{=} \PY{l+m+mf}{0.8}
+        \PY{n}{test\PYZus{}size} \PY{o}{=} \PY{l+m+mi}{1} \PY{o}{\PYZhy{}} \PY{n}{train\PYZus{}size}
+        \PY{n}{X\PYZus{}train}\PY{p}{,} \PY{n}{X\PYZus{}test}\PY{p}{,} \PY{n}{Y\PYZus{}train}\PY{p}{,} \PY{n}{Y\PYZus{}test} \PY{o}{=} \PY{n}{train\PYZus{}test\PYZus{}split}\PY{p}{(}\PY{n}{inputs}\PY{p}{,} \PY{n}{labels}\PY{p}{,} \PY{n}{train\PYZus{}size}\PY{o}{=}\PY{n}{train\PYZus{}size}\PY{p}{,}
+                                                            \PY{n}{test\PYZus{}size}\PY{o}{=}\PY{n}{test\PYZus{}size}\PY{p}{)}
+        
+        \PY{c+c1}{\PYZsh{} equivalently in numpy}
+        \PY{k}{def} \PY{n+nf}{train\PYZus{}test\PYZus{}split\PYZus{}numpy}\PY{p}{(}\PY{n}{inputs}\PY{p}{,} \PY{n}{labels}\PY{p}{,} \PY{n}{train\PYZus{}size}\PY{p}{,} \PY{n}{test\PYZus{}size}\PY{p}{)}\PY{p}{:}
+            \PY{n}{n\PYZus{}inputs} \PY{o}{=} \PY{n+nb}{len}\PY{p}{(}\PY{n}{inputs}\PY{p}{)}
+            \PY{n}{inputs\PYZus{}shuffled} \PY{o}{=} \PY{n}{inputs}\PY{o}{.}\PY{n}{copy}\PY{p}{(}\PY{p}{)}
+            \PY{n}{labels\PYZus{}shuffled} \PY{o}{=} \PY{n}{labels}\PY{o}{.}\PY{n}{copy}\PY{p}{(}\PY{p}{)}
+            
+            \PY{n}{np}\PY{o}{.}\PY{n}{random}\PY{o}{.}\PY{n}{shuffle}\PY{p}{(}\PY{n}{inputs\PYZus{}shuffled}\PY{p}{)}
+            \PY{n}{np}\PY{o}{.}\PY{n}{random}\PY{o}{.}\PY{n}{shuffle}\PY{p}{(}\PY{n}{labels\PYZus{}shuffled}\PY{p}{)}
+            
+            \PY{n}{train\PYZus{}end} \PY{o}{=} \PY{n+nb}{int}\PY{p}{(}\PY{n}{n\PYZus{}inputs}\PY{o}{*}\PY{n}{train\PYZus{}size}\PY{p}{)}
+            \PY{n}{X\PYZus{}train}\PY{p}{,} \PY{n}{X\PYZus{}test} \PY{o}{=} \PY{n}{inputs\PYZus{}shuffled}\PY{p}{[}\PY{p}{:}\PY{n}{train\PYZus{}end}\PY{p}{]}\PY{p}{,} \PY{n}{inputs\PYZus{}shuffled}\PY{p}{[}\PY{n}{train\PYZus{}end}\PY{p}{:}\PY{p}{]}
+            \PY{n}{Y\PYZus{}train}\PY{p}{,} \PY{n}{Y\PYZus{}test} \PY{o}{=} \PY{n}{labels\PYZus{}shuffled}\PY{p}{[}\PY{p}{:}\PY{n}{train\PYZus{}end}\PY{p}{]}\PY{p}{,} \PY{n}{labels\PYZus{}shuffled}\PY{p}{[}\PY{n}{train\PYZus{}end}\PY{p}{:}\PY{p}{]}
+            
+            \PY{k}{return} \PY{n}{X\PYZus{}train}\PY{p}{,} \PY{n}{X\PYZus{}test}\PY{p}{,} \PY{n}{Y\PYZus{}train}\PY{p}{,} \PY{n}{Y\PYZus{}test}
+        
+        \PY{c+c1}{\PYZsh{}X\PYZus{}train, X\PYZus{}test, Y\PYZus{}train, Y\PYZus{}test = train\PYZus{}test\PYZus{}split\PYZus{}numpy(inputs, labels, train\PYZus{}size, test\PYZus{}size)}
+        
+        \PY{n+nb}{print}\PY{p}{(}\PY{l+s+s2}{\PYZdq{}}\PY{l+s+s2}{Number of training images: }\PY{l+s+s2}{\PYZdq{}} \PY{o}{+} \PY{n+nb}{str}\PY{p}{(}\PY{n+nb}{len}\PY{p}{(}\PY{n}{X\PYZus{}train}\PY{p}{)}\PY{p}{)}\PY{p}{)}
+        \PY{n+nb}{print}\PY{p}{(}\PY{l+s+s2}{\PYZdq{}}\PY{l+s+s2}{Number of test images: }\PY{l+s+s2}{\PYZdq{}} \PY{o}{+} \PY{n+nb}{str}\PY{p}{(}\PY{n+nb}{len}\PY{p}{(}\PY{n}{X\PYZus{}test}\PY{p}{)}\PY{p}{)}\PY{p}{)}
+\end{Verbatim}
+
+
+    \begin{Verbatim}[commandchars=\\\{\}]
+Number of training images: 1437
+Number of test images: 360
+
+    \end{Verbatim}
+
+    \hypertarget{define-model-and-architecture}{%
+\subsection{Define model and
+architecture}\label{define-model-and-architecture}}
+
+Our simple feed-forward neural network will consist of an \emph{input}
+layer, a single \emph{hidden} layer and an \emph{output} layer. The
+activation \(y\) of each neuron is a weighted sum of inputs, passed
+through an activation function:
+
+\[ z = \sum_{i=1}^n w_i a_i ,\]
+
+\[ y = f(z) ,\]
+
+where \(f\) is the activation function, \(a_i\) represents input from
+neuron \(i\) in the preceding layer and \(w_i\) is the weight to input
+\(i\).\\
+The activation of the neurons in the input layer is just the features
+(e.g.~a pixel value).
+
+The simplest activation function for a neuron is the \emph{Heaviside}
+function:
+
+\[ f(z) = 
+\begin{cases}
+1,  &  z > 0\\
+0,  & \text{otherwise}
+\end{cases}
+\]
+
+A feed-forward neural network with this activation is known as a
+\emph{perceptron}.\\
+For a binary classifier (i.e.~two classes, 0 or 1, dog or not-dog) we
+can also use this in our output layer.\\
+This activation can be generalized to \(k\) classes (using e.g.~the
+\emph{one-against-all} strategy), and we call these architectures
+\emph{multiclass perceptrons}.
+
+However, it is now common to use the terms Single Layer Perceptron (SLP)
+(1 hidden layer) and\\
+Multilayer Perceptron (MLP) (2 or more hidden layers) to refer to
+feed-forward neural networks with any activation function.
+
+Typical choices for activation functions include the sigmoid function,
+hyperbolic tangent, and Rectified Linear Unit (ReLU).\\
+We will be using the sigmoid function \(\sigma(x)\):
+
+\[ f(x) = \sigma(x) = \frac{1}{1 + e^{-x}} ,\]
+
+which is inspired by probability theory (see logistic regression) and
+was most commonly used until about 2011.
+
+\hypertarget{layers}{%
+\subsection{Layers}\label{layers}}
+
+\begin{itemize}
+\tightlist
+\item
+  Input
+\end{itemize}
+
+Since each input image has 8x8 = 64 pixels or features, we have an input
+layer of 64 neurons.
+
+\begin{itemize}
+\tightlist
+\item
+  Hidden layer
+\end{itemize}
+
+We will use 50 neurons in the hidden layer receiving input from the
+neurons in the input layer.\\
+Since each neuron in the hidden layer is connected to the 64 inputs we
+have 64x50 = 3200 weights to the hidden layer.
+
+\begin{itemize}
+\tightlist
+\item
+  Output
+\end{itemize}
+
+If we were building a binary classifier, it would be sufficient with a
+single neuron in the output layer, which could output 0 or 1 according
+to the Heaviside function. This would be an example of a \emph{hard}
+classifier, meaning it outputs the class of the input directly. However,
+if we are dealing with noisy data it is often beneficial to use a
+\emph{soft} classifier, which outputs the probability of being in class
+0 or 1.
+
+For a soft binary classifier, we could use a single neuron and interpret
+the output as either being the probability of being in class 0 or the
+probability of being in class 1. Alternatively we could use 2 neurons,
+and interpret each neuron as the probability of being in each class.
+
+Since we are doing multiclass classification, with 10 categories, it is
+natural to use 10 neurons in the output layer. We number the neurons
+\(j = 0,1,...,9\). The activation of each output neuron \(j\) will be
+according to the \emph{softmax} function:
+
+\[ P(\text{class $j$} \mid \text{input $\boldsymbol{a}$}) = \frac{\exp{(\boldsymbol{a}^T \boldsymbol{w}_j)}}
+{\sum_{c=0}^{9} \exp{(\boldsymbol{a}^T \boldsymbol{w}_c)}} ,\]
+
+i.e.~each neuron \(j\) outputs the probability of being in class \(j\)
+given an input from the hidden layer \(\boldsymbol{a}\), with
+\(\boldsymbol{w}_j\) the weights of neuron \(j\) to the inputs.\\
+The denominator is a normalization factor to ensure the outputs
+(probabilities) sum up to 1.\\
+The exponent is just the weighted sum of inputs as before:
+
+\[ z_j = \sum_{i=1}^n w_ {ij} a_i+b_j.\]
+
+Since each neuron in the output layer is connected to the 50 inputs from
+the hidden layer we have 50x10 = 500 weights to the output layer.
+
+\hypertarget{weights-and-biases}{%
+\subsection{Weights and biases}\label{weights-and-biases}}
+
+Typically weights are initialized with small values distributed around
+zero, drawn from a uniform or normal distribution. Setting all weights
+to zero means all neurons give the same output, making the network
+useless.
+
+Adding a bias value to the weighted sum of inputs allows the neural
+network to represent a greater range of values. Without it, any input
+with the value 0 will be mapped to zero (before being passed through the
+activation). The bias unit has an output of 1, and a weight to each
+neuron \(j\), \(b_j\):
+
+\[ z_j = \sum_{i=1}^n w_ {ij} a_i + 1\cdot b_j.\]
+
+The bias weights \(\boldsymbol{b}\) are often initialized to zero, but a
+small value like \(0.01\) ensures all neurons have some output which can
+be backpropagated in the first training cycle.
+
+    \begin{Verbatim}[commandchars=\\\{\}]
+{\color{incolor}In [{\color{incolor}5}]:} \PY{c+c1}{\PYZsh{} building our neural network}
+        
+        \PY{n}{n\PYZus{}inputs}\PY{p}{,} \PY{n}{n\PYZus{}features} \PY{o}{=} \PY{n}{X\PYZus{}train}\PY{o}{.}\PY{n}{shape}
+        \PY{n}{n\PYZus{}hidden\PYZus{}neurons} \PY{o}{=} \PY{l+m+mi}{50}
+        \PY{n}{n\PYZus{}categories} \PY{o}{=} \PY{l+m+mi}{10}
+        
+        \PY{c+c1}{\PYZsh{} we make the weights normally distributed using numpy.random.randn}
+        
+        \PY{c+c1}{\PYZsh{} weights and bias in the hidden layer}
+        \PY{n}{hidden\PYZus{}weights} \PY{o}{=} \PY{n}{np}\PY{o}{.}\PY{n}{random}\PY{o}{.}\PY{n}{randn}\PY{p}{(}\PY{n}{n\PYZus{}features}\PY{p}{,} \PY{n}{n\PYZus{}hidden\PYZus{}neurons}\PY{p}{)}
+        \PY{n}{hidden\PYZus{}bias} \PY{o}{=} \PY{n}{np}\PY{o}{.}\PY{n}{zeros}\PY{p}{(}\PY{n}{n\PYZus{}hidden\PYZus{}neurons}\PY{p}{)} \PY{o}{+} \PY{l+m+mf}{0.01}
+        
+        \PY{c+c1}{\PYZsh{} weights and bias in the output layer}
+        \PY{n}{output\PYZus{}weights} \PY{o}{=} \PY{n}{np}\PY{o}{.}\PY{n}{random}\PY{o}{.}\PY{n}{randn}\PY{p}{(}\PY{n}{n\PYZus{}hidden\PYZus{}neurons}\PY{p}{,} \PY{n}{n\PYZus{}categories}\PY{p}{)}
+        \PY{n}{output\PYZus{}bias} \PY{o}{=} \PY{n}{np}\PY{o}{.}\PY{n}{zeros}\PY{p}{(}\PY{n}{n\PYZus{}categories}\PY{p}{)} \PY{o}{+} \PY{l+m+mf}{0.01}
+\end{Verbatim}
+
+
+    \hypertarget{feed-forward-pass}{%
+\subsection{Feed-forward pass}\label{feed-forward-pass}}
+
+Denote \(F\) the number of features, \(H\) the number of hidden neurons
+and \(C\) the number of categories.\\
+For each input image we calculate a weighted sum of input features
+(pixel values) to each neuron \(j\) in the hidden layer \(l\):
+
+\[ z_{j}^{l} = \sum_{i=1}^{F} w_{ij}^{l} x_i + b_{j}^{l},\]
+
+this is then passed through our activation function
+
+\[ a_{j}^{l} = f(z_{j}^{l}) .\]
+
+We calculate a weighted sum of inputs (activations in the hidden layer)
+to each neuron \(j\) in the output layer:
+
+\[ z_{j}^{L} = \sum_{i=1}^{H} w_{ij}^{L} a_{i}^{l} + b_{j}^{L}.\]
+
+Finally we calculate the output of neuron \(j\) in the output layer
+using the softmax function:
+
+\[ a_{j}^{L} = \frac{\exp{(z_j^{L})}}
+{\sum_{c=0}^{C-1} \exp{(z_c^{L})}} .\]
+
+\hypertarget{matrix-multiplication}{%
+\subsection{Matrix multiplication}\label{matrix-multiplication}}
+
+Since our data has the dimensions \(X = (n_{inputs}, n_{features})\) and
+our weights to the hidden layer have the dimensions\\
+\(W_{hidden} = (n_{features}, n_{hidden})\), we can easily feed the
+network all our training data in one go by taking the matrix product
+
+\[ X W^{h} = (n_{inputs}, n_{hidden}),\]
+
+and obtain a matrix that holds the weighted sum of inputs to the hidden
+layer for each input image and each hidden neuron.\\
+We also add the bias to obtain a matrix of weighted sums to the hidden
+layer \(Z^{h}\):
+
+\[ \hat{z}^{l} = \hat{X} \hat{W}^{l} + \hat{b}^{l} ,\]
+
+meaning the same bias (1D array with size equal number of hidden
+neurons) is added to each input image.\\
+This is then passed through the activation:
+
+\[ \hat{a}^{l} = f(\hat{z}^l) .\]
+
+This is fed to the output layer:
+
+\[ \hat{z}^{L} = \hat{a}^{L} \hat{W}^{L} + \hat{b}^{L} .\]
+
+Finally we receive our output values for each image and each category by
+passing it through the softmax function:
+
+\[ output = softmax (\hat{z}^{L}) = (n_{inputs}, n_{categories}) .\]
+
+    \begin{Verbatim}[commandchars=\\\{\}]
+{\color{incolor}In [{\color{incolor}6}]:} \PY{c+c1}{\PYZsh{} setup the feed\PYZhy{}forward pass, subscript h = hidden layer}
+        
+        \PY{k}{def} \PY{n+nf}{sigmoid}\PY{p}{(}\PY{n}{x}\PY{p}{)}\PY{p}{:}
+            \PY{k}{return} \PY{l+m+mi}{1}\PY{o}{/}\PY{p}{(}\PY{l+m+mi}{1} \PY{o}{+} \PY{n}{np}\PY{o}{.}\PY{n}{exp}\PY{p}{(}\PY{o}{\PYZhy{}}\PY{n}{x}\PY{p}{)}\PY{p}{)}
+        
+        \PY{k}{def} \PY{n+nf}{feed\PYZus{}forward}\PY{p}{(}\PY{n}{X}\PY{p}{)}\PY{p}{:}
+            \PY{c+c1}{\PYZsh{} weighted sum of inputs to the hidden layer}
+            \PY{n}{z\PYZus{}h} \PY{o}{=} \PY{n}{np}\PY{o}{.}\PY{n}{matmul}\PY{p}{(}\PY{n}{X}\PY{p}{,} \PY{n}{hidden\PYZus{}weights}\PY{p}{)} \PY{o}{+} \PY{n}{hidden\PYZus{}bias}
+            \PY{c+c1}{\PYZsh{} activation in the hidden layer}
+            \PY{n}{a\PYZus{}h} \PY{o}{=} \PY{n}{sigmoid}\PY{p}{(}\PY{n}{z\PYZus{}h}\PY{p}{)}
+            
+            \PY{c+c1}{\PYZsh{} weighted sum of inputs to the output layer}
+            \PY{n}{z\PYZus{}o} \PY{o}{=} \PY{n}{np}\PY{o}{.}\PY{n}{matmul}\PY{p}{(}\PY{n}{a\PYZus{}h}\PY{p}{,} \PY{n}{output\PYZus{}weights}\PY{p}{)} \PY{o}{+} \PY{n}{output\PYZus{}bias}
+            \PY{c+c1}{\PYZsh{} softmax output}
+            \PY{c+c1}{\PYZsh{} axis 0 holds each input and axis 1 the probabilities of each category}
+            \PY{n}{exp\PYZus{}term} \PY{o}{=} \PY{n}{np}\PY{o}{.}\PY{n}{exp}\PY{p}{(}\PY{n}{z\PYZus{}o}\PY{p}{)}
+            \PY{n}{probabilities} \PY{o}{=} \PY{n}{exp\PYZus{}term} \PY{o}{/} \PY{n}{np}\PY{o}{.}\PY{n}{sum}\PY{p}{(}\PY{n}{exp\PYZus{}term}\PY{p}{,} \PY{n}{axis}\PY{o}{=}\PY{l+m+mi}{1}\PY{p}{,} \PY{n}{keepdims}\PY{o}{=}\PY{k+kc}{True}\PY{p}{)}
+            
+            \PY{k}{return} \PY{n}{probabilities}
+        
+        \PY{n}{probabilities} \PY{o}{=} \PY{n}{feed\PYZus{}forward}\PY{p}{(}\PY{n}{X\PYZus{}train}\PY{p}{)}
+        \PY{n+nb}{print}\PY{p}{(}\PY{l+s+s2}{\PYZdq{}}\PY{l+s+s2}{probabilities = (n\PYZus{}inputs, n\PYZus{}categories) = }\PY{l+s+s2}{\PYZdq{}} \PY{o}{+} \PY{n+nb}{str}\PY{p}{(}\PY{n}{probabilities}\PY{o}{.}\PY{n}{shape}\PY{p}{)}\PY{p}{)}
+        \PY{n+nb}{print}\PY{p}{(}\PY{l+s+s2}{\PYZdq{}}\PY{l+s+s2}{probability that image 0 is in category 0,1,2,...,9 = }\PY{l+s+se}{\PYZbs{}n}\PY{l+s+s2}{\PYZdq{}} \PY{o}{+} \PY{n+nb}{str}\PY{p}{(}\PY{n}{probabilities}\PY{p}{[}\PY{l+m+mi}{0}\PY{p}{]}\PY{p}{)}\PY{p}{)}
+        \PY{n+nb}{print}\PY{p}{(}\PY{l+s+s2}{\PYZdq{}}\PY{l+s+s2}{probabilities sum up to: }\PY{l+s+s2}{\PYZdq{}} \PY{o}{+} \PY{n+nb}{str}\PY{p}{(}\PY{n}{probabilities}\PY{p}{[}\PY{l+m+mi}{0}\PY{p}{]}\PY{o}{.}\PY{n}{sum}\PY{p}{(}\PY{p}{)}\PY{p}{)}\PY{p}{)}
+        \PY{n+nb}{print}\PY{p}{(}\PY{p}{)}
+        
+        \PY{c+c1}{\PYZsh{} we obtain a prediction by taking the class with the highest likelihood}
+        \PY{k}{def} \PY{n+nf}{predict}\PY{p}{(}\PY{n}{X}\PY{p}{)}\PY{p}{:}
+            \PY{n}{probabilities} \PY{o}{=} \PY{n}{feed\PYZus{}forward}\PY{p}{(}\PY{n}{X}\PY{p}{)}
+            \PY{k}{return} \PY{n}{np}\PY{o}{.}\PY{n}{argmax}\PY{p}{(}\PY{n}{probabilities}\PY{p}{,} \PY{n}{axis}\PY{o}{=}\PY{l+m+mi}{1}\PY{p}{)}
+        
+        \PY{n}{predictions} \PY{o}{=} \PY{n}{predict}\PY{p}{(}\PY{n}{X\PYZus{}train}\PY{p}{)}
+        \PY{n+nb}{print}\PY{p}{(}\PY{l+s+s2}{\PYZdq{}}\PY{l+s+s2}{predictions = (n\PYZus{}inputs) = }\PY{l+s+s2}{\PYZdq{}} \PY{o}{+} \PY{n+nb}{str}\PY{p}{(}\PY{n}{predictions}\PY{o}{.}\PY{n}{shape}\PY{p}{)}\PY{p}{)}
+        \PY{n+nb}{print}\PY{p}{(}\PY{l+s+s2}{\PYZdq{}}\PY{l+s+s2}{prediction for image 0: }\PY{l+s+s2}{\PYZdq{}} \PY{o}{+} \PY{n+nb}{str}\PY{p}{(}\PY{n}{predictions}\PY{p}{[}\PY{l+m+mi}{0}\PY{p}{]}\PY{p}{)}\PY{p}{)}
+        \PY{n+nb}{print}\PY{p}{(}\PY{l+s+s2}{\PYZdq{}}\PY{l+s+s2}{correct label for image 0: }\PY{l+s+s2}{\PYZdq{}} \PY{o}{+} \PY{n+nb}{str}\PY{p}{(}\PY{n}{Y\PYZus{}train}\PY{p}{[}\PY{l+m+mi}{0}\PY{p}{]}\PY{p}{)}\PY{p}{)}
+\end{Verbatim}
+
+
+    \begin{Verbatim}[commandchars=\\\{\}]
+probabilities = (n\_inputs, n\_categories) = (1437, 10)
+probability that image 0 is in category 0,1,2,{\ldots},9 = 
+[3.89940599e-05 1.79115580e-01 1.47286800e-02 7.96733555e-01
+ 3.28982767e-04 1.49752254e-07 9.19699482e-05 4.42365585e-03
+ 3.57722690e-06 4.53485505e-03]
+probabilities sum up to: 1.0000000000000002
+
+predictions = (n\_inputs) = (1437,)
+prediction for image 0: 3
+correct label for image 0: 6
+
+    \end{Verbatim}
+
+    \hypertarget{choose-cost-function-and-optimizer}{%
+\subsection{Choose cost function and
+optimizer}\label{choose-cost-function-and-optimizer}}
+
+To measure how well our neural network is doing we need to introduce a
+cost function.\\
+We will call the function that gives the error of a single sample output
+the \emph{loss} function, and the function that gives the total error of
+our network across all samples the \emph{cost} function. A typical
+choice for multiclass classification is the \emph{cross-entropy} loss,
+also known as the negative log likelihood.
+
+In \emph{multiclass} classification it is common to treat each integer
+label as a so called \emph{one-hot} vector:
+
+\[ y = 5 \quad \rightarrow \quad \boldsymbol{y} = (0, 0, 0, 0, 0, 1, 0, 0, 0, 0) ,\]
+
+\[ y = 1 \quad \rightarrow \quad \boldsymbol{y} = (0, 1, 0, 0, 0, 0, 0, 0, 0, 0) ,\]
+
+i.e.~a binary bit string of length \(C\), where \(C = 10\) is the number
+of classes in the MNIST dataset.
+
+Let \(y_{ic}\) denote the \(c\)-th component of the \(i\)-th one-hot
+vector.\\
+We define the cost function \(\mathcal{C}\) as a sum over the
+cross-entropy loss for each point \(\boldsymbol{x}_i\) in the dataset:
+
+    \hypertarget{_auto11}{}
+
+\[
+\begin{equation} \mathcal{C}(\boldsymbol{\theta}) = \sum_{i=1}^N \mathcal{L}_i (\boldsymbol{\theta}) 
+\label{_auto11} \tag{16}
+\end{equation}
+\]
+
+    \hypertarget{_auto12}{}
+
+\[
+\begin{equation} 
+ = -\sum_{i=1}^N \sum_{c=0}^{C-1} y_{ic} \log P(y_{ic} = 1 \mid \boldsymbol{x}_i, \boldsymbol{\theta}) .
+\label{_auto12} \tag{17}
+\end{equation}
+\]
+
+    In the one-hot representation only one of the terms in the loss function
+is non-zero, namely the probability of the correct category \(c'\)\\
+(i.e.~the category \(c'\) such that \(y_{ic'} = 1\)). This means that
+the cross entropy loss only punishes you for how wrong you got the
+correct label. The probability of category \(c\) is given by the softmax
+function. The vector \(\boldsymbol{\theta}\) represents the parameters
+of our network, i.e.~all the weights and biases.
+
+A full derivation is given in the appendix at the end.
+
+\hypertarget{optimizing-the-cost-function}{%
+\subsection{Optimizing the cost
+function}\label{optimizing-the-cost-function}}
+
+The network is trained by finding the weights and biases that minimize
+the cost function. One of the most widely used classes of methods is
+\emph{gradient descent} and its generalizations. The idea behind
+gradient descent is simply to adjust the weights in the direction where
+the gradient of the cost function is large and negative. This ensures we
+flow toward a \emph{local} minimum of the cost function.\\
+Each parameter \(\theta\) is iteratively adjusted according to the rule
+
+\[ \theta_{i+1} = \theta_i - \eta \nabla \mathcal{C}(\theta_i) ,\]
+
+where \(\eta\) is known as the \emph{learning rate}, which controls how
+big a step we take towards the minimum.\\
+This update can be repeated for any number of iterations, or until we
+are satisfied with the result.
+
+A simple and effective improvement is a variant called \emph{Batch
+Gradient Descent}.\\
+Instead of calculating the gradient on the whole dataset, we calculate
+an approximation of the gradient on a subset of the data called a
+\emph{minibatch}.\\
+If there are \(N\) data points and we have a minibatch size of \(M\),
+the total number of batches is \(N/M\).\\
+We denote each minibatch \(B_k\), with \(k = 1, 2,...,N/M\). The
+gradient then becomes:
+
+\[ \nabla \mathcal{C}(\theta) = \frac{1}{N} \sum_{i=1}^N \nabla \mathcal{L}_i(\theta) \quad \rightarrow \quad
+\frac{1}{M} \sum_{i \in B_k} \nabla \mathcal{L}_i(\theta) ,\]
+
+i.e.~instead of averaging the loss over the entire dataset, we average
+over a minibatch.
+
+This has two important benefits:\\
+1. Introducing stochasticity decreases the chance that the algorithm
+becomes stuck in a local minima.
+
+\begin{enumerate}
+\def\labelenumi{\arabic{enumi}.}
+\setcounter{enumi}{1}
+\tightlist
+\item
+  It significantly speeds up the calculation, since we do not have to
+  use the entire dataset to calculate the gradient.
+\end{enumerate}
+
+\hypertarget{regularization}{%
+\subsection{Regularization}\label{regularization}}
+
+It is common to add an extra term to the cost function, proportional to
+the size of the weights. This is equivalent to constraining the size of
+the weights, so that they do not grow out of control. Constraining the
+size of the weights means that the weights cannot grow arbitrarily large
+to fit the training data, and in this way reduces \emph{overfitting}.
+
+We will measure the size of the weights using the so called
+\emph{L2-norm}, meaning our cost function becomes:
+
+\[ \nabla \mathcal{C}(\theta) = \frac{1}{N} \sum_{i=1}^N \nabla \mathcal{L}_i(\theta) \quad \rightarrow \quad
+\frac{1}{N} \sum_{i=1}^N \nabla \mathcal{L}_i(\theta) + \lambda \lvert \lvert \boldsymbol{w} \rvert \rvert_2^2 
+= \frac{1}{N} \sum_{i=1}^N \nabla \mathcal{L}(\theta) + \lambda \sum_{ij} w_{ij}^2,\]
+
+i.e.~we sum up all the weights squared. The factor \(\lambda\) is known
+as a regularization parameter.
+
+In order to train the model, we need to calculate the derivative of the
+cost function with respect to every bias and weight in the network. In
+total our network has \((64 + 1)\times 50=3250\) weights in the hidden
+layer and \((50 + 1)\times 10=510\) weights to the output layer (\(+1\)
+for the bias), and the gradient must be calculated for every parameter.
+We use the \emph{backpropagation} algorithm discussed above. This is a
+clever use of the chain rule that allows us to calculate the gradient
+efficently.
+
+\hypertarget{matrix-multiplication}{%
+\subsection{Matrix multiplication}\label{matrix-multiplication}}
+
+To more efficently train our network these equations are implemented
+using matrix operations.\\
+The error in the output layer is calculated simply as
+
+\[ \delta_L = \hat{y} - y = (n_{inputs}, n_{categories}) .\]
+
+The gradient for the output weights is calculated as
+
+\[ \nabla W_{L} = \hat{a}^T \delta_L   = (n_{hidden}, n_{categories}) ,\]
+
+where \(\hat{a} = (n_{inputs}, n_{hidden})\). This simply means that we
+are summing up the gradients for each input.\\
+Since we are going backwards we have to transpose the activation matrix.
+
+The gradient with respect to the output bias is then
+
+\[ \nabla \hat{b}_{L} = \sum_{i=1}^{n_{inputs}} \delta_L = (n_{categories}) .\]
+
+The error in the hidden layer is
+
+\[ \Delta_h = \delta_L W_{L}^T \circ f'(z_{h}) = \delta_L W_{L}^T \circ a_{h} \circ (1 - a_{h}) = (n_{inputs}, n_{hidden}) ,\]
+
+where \(f'(a_{h})\) is the derivative of the activation in the hidden
+layer. The matrix products mean that we are summing up the products for
+each neuron in the output layer. The symbol \(\circ\) denotes the
+\emph{Hadamard product}, meaning element-wise multiplication.
+
+This again gives us the gradients in the hidden layer:
+
+\[ \nabla W_{h} = X^T \delta_h = (n_{features}, n_{hidden}) ,\]
+
+\[ \nabla b_{h} = \sum_{i=1}^{n_{inputs}} \delta_h = (n_{hidden}) .\]
+
+    \begin{Verbatim}[commandchars=\\\{\}]
+{\color{incolor}In [{\color{incolor}7}]:} \PY{c+c1}{\PYZsh{} to categorical turns our integer vector into a onehot representation}
+        \PY{c+c1}{\PYZsh{}from keras.utils import to\PYZus{}categorical}
+        
+        \PY{c+c1}{\PYZsh{} calculate the accuracy score of our model}
+        \PY{k+kn}{from} \PY{n+nn}{sklearn}\PY{n+nn}{.}\PY{n+nn}{metrics} \PY{k}{import} \PY{n}{accuracy\PYZus{}score}
+        
+        \PY{c+c1}{\PYZsh{} one\PYZhy{}hot in numpy}
+        \PY{k}{def} \PY{n+nf}{to\PYZus{}categorical\PYZus{}numpy}\PY{p}{(}\PY{n}{integer\PYZus{}vector}\PY{p}{)}\PY{p}{:}
+            \PY{n}{n\PYZus{}inputs} \PY{o}{=} \PY{n+nb}{len}\PY{p}{(}\PY{n}{integer\PYZus{}vector}\PY{p}{)}
+            \PY{n}{n\PYZus{}categories} \PY{o}{=} \PY{n}{np}\PY{o}{.}\PY{n}{max}\PY{p}{(}\PY{n}{integer\PYZus{}vector}\PY{p}{)} \PY{o}{+} \PY{l+m+mi}{1}
+            \PY{n}{onehot\PYZus{}vector} \PY{o}{=} \PY{n}{np}\PY{o}{.}\PY{n}{zeros}\PY{p}{(}\PY{p}{(}\PY{n}{n\PYZus{}inputs}\PY{p}{,} \PY{n}{n\PYZus{}categories}\PY{p}{)}\PY{p}{)}
+            \PY{n}{onehot\PYZus{}vector}\PY{p}{[}\PY{n+nb}{range}\PY{p}{(}\PY{n}{n\PYZus{}inputs}\PY{p}{)}\PY{p}{,} \PY{n}{integer\PYZus{}vector}\PY{p}{]} \PY{o}{=} \PY{l+m+mi}{1}
+            
+            \PY{k}{return} \PY{n}{onehot\PYZus{}vector}
+        
+        \PY{c+c1}{\PYZsh{}Y\PYZus{}train\PYZus{}onehot, Y\PYZus{}test\PYZus{}onehot = to\PYZus{}categorical(Y\PYZus{}train), to\PYZus{}categorical(Y\PYZus{}test)}
+        \PY{n}{Y\PYZus{}train\PYZus{}onehot}\PY{p}{,} \PY{n}{Y\PYZus{}test\PYZus{}onehot} \PY{o}{=} \PY{n}{to\PYZus{}categorical\PYZus{}numpy}\PY{p}{(}\PY{n}{Y\PYZus{}train}\PY{p}{)}\PY{p}{,} \PY{n}{to\PYZus{}categorical\PYZus{}numpy}\PY{p}{(}\PY{n}{Y\PYZus{}test}\PY{p}{)}
+        
+        \PY{k}{def} \PY{n+nf}{feed\PYZus{}forward\PYZus{}train}\PY{p}{(}\PY{n}{X}\PY{p}{)}\PY{p}{:}
+            \PY{c+c1}{\PYZsh{} weighted sum of inputs to the hidden layer}
+            \PY{n}{z\PYZus{}h} \PY{o}{=} \PY{n}{np}\PY{o}{.}\PY{n}{matmul}\PY{p}{(}\PY{n}{X}\PY{p}{,} \PY{n}{hidden\PYZus{}weights}\PY{p}{)} \PY{o}{+} \PY{n}{hidden\PYZus{}bias}
+            \PY{c+c1}{\PYZsh{} activation in the hidden layer}
+            \PY{n}{a\PYZus{}h} \PY{o}{=} \PY{n}{sigmoid}\PY{p}{(}\PY{n}{z\PYZus{}h}\PY{p}{)}
+            
+            \PY{c+c1}{\PYZsh{} weighted sum of inputs to the output layer}
+            \PY{n}{z\PYZus{}o} \PY{o}{=} \PY{n}{np}\PY{o}{.}\PY{n}{matmul}\PY{p}{(}\PY{n}{a\PYZus{}h}\PY{p}{,} \PY{n}{output\PYZus{}weights}\PY{p}{)} \PY{o}{+} \PY{n}{output\PYZus{}bias}
+            \PY{c+c1}{\PYZsh{} softmax output}
+            \PY{c+c1}{\PYZsh{} axis 0 holds each input and axis 1 the probabilities of each category}
+            \PY{n}{exp\PYZus{}term} \PY{o}{=} \PY{n}{np}\PY{o}{.}\PY{n}{exp}\PY{p}{(}\PY{n}{z\PYZus{}o}\PY{p}{)}
+            \PY{n}{probabilities} \PY{o}{=} \PY{n}{exp\PYZus{}term} \PY{o}{/} \PY{n}{np}\PY{o}{.}\PY{n}{sum}\PY{p}{(}\PY{n}{exp\PYZus{}term}\PY{p}{,} \PY{n}{axis}\PY{o}{=}\PY{l+m+mi}{1}\PY{p}{,} \PY{n}{keepdims}\PY{o}{=}\PY{k+kc}{True}\PY{p}{)}
+            
+            \PY{c+c1}{\PYZsh{} for backpropagation need activations in hidden and output layers}
+            \PY{k}{return} \PY{n}{a\PYZus{}h}\PY{p}{,} \PY{n}{probabilities}
+        
+        \PY{k}{def} \PY{n+nf}{backpropagation}\PY{p}{(}\PY{n}{X}\PY{p}{,} \PY{n}{Y}\PY{p}{)}\PY{p}{:}
+            \PY{n}{a\PYZus{}h}\PY{p}{,} \PY{n}{probabilities} \PY{o}{=} \PY{n}{feed\PYZus{}forward\PYZus{}train}\PY{p}{(}\PY{n}{X}\PY{p}{)}
+            
+            \PY{c+c1}{\PYZsh{} error in the output layer}
+            \PY{n}{error\PYZus{}output} \PY{o}{=} \PY{n}{probabilities} \PY{o}{\PYZhy{}} \PY{n}{Y}
+            \PY{c+c1}{\PYZsh{} error in the hidden layer}
+            \PY{n}{error\PYZus{}hidden} \PY{o}{=} \PY{n}{np}\PY{o}{.}\PY{n}{matmul}\PY{p}{(}\PY{n}{error\PYZus{}output}\PY{p}{,} \PY{n}{output\PYZus{}weights}\PY{o}{.}\PY{n}{T}\PY{p}{)} \PY{o}{*} \PY{n}{a\PYZus{}h} \PY{o}{*} \PY{p}{(}\PY{l+m+mi}{1} \PY{o}{\PYZhy{}} \PY{n}{a\PYZus{}h}\PY{p}{)}
+            
+            \PY{c+c1}{\PYZsh{} gradients for the output layer}
+            \PY{n}{output\PYZus{}weights\PYZus{}gradient} \PY{o}{=} \PY{n}{np}\PY{o}{.}\PY{n}{matmul}\PY{p}{(}\PY{n}{a\PYZus{}h}\PY{o}{.}\PY{n}{T}\PY{p}{,} \PY{n}{error\PYZus{}output}\PY{p}{)}
+            \PY{n}{output\PYZus{}bias\PYZus{}gradient} \PY{o}{=} \PY{n}{np}\PY{o}{.}\PY{n}{sum}\PY{p}{(}\PY{n}{error\PYZus{}output}\PY{p}{,} \PY{n}{axis}\PY{o}{=}\PY{l+m+mi}{0}\PY{p}{)}
+            
+            \PY{c+c1}{\PYZsh{} gradient for the hidden layer}
+            \PY{n}{hidden\PYZus{}weights\PYZus{}gradient} \PY{o}{=} \PY{n}{np}\PY{o}{.}\PY{n}{matmul}\PY{p}{(}\PY{n}{X}\PY{o}{.}\PY{n}{T}\PY{p}{,} \PY{n}{error\PYZus{}hidden}\PY{p}{)}
+            \PY{n}{hidden\PYZus{}bias\PYZus{}gradient} \PY{o}{=} \PY{n}{np}\PY{o}{.}\PY{n}{sum}\PY{p}{(}\PY{n}{error\PYZus{}hidden}\PY{p}{,} \PY{n}{axis}\PY{o}{=}\PY{l+m+mi}{0}\PY{p}{)}
+        
+            \PY{k}{return} \PY{n}{output\PYZus{}weights\PYZus{}gradient}\PY{p}{,} \PY{n}{output\PYZus{}bias\PYZus{}gradient}\PY{p}{,} \PY{n}{hidden\PYZus{}weights\PYZus{}gradient}\PY{p}{,} \PY{n}{hidden\PYZus{}bias\PYZus{}gradient}
+        
+        \PY{n+nb}{print}\PY{p}{(}\PY{l+s+s2}{\PYZdq{}}\PY{l+s+s2}{Old accuracy on training data: }\PY{l+s+s2}{\PYZdq{}} \PY{o}{+} \PY{n+nb}{str}\PY{p}{(}\PY{n}{accuracy\PYZus{}score}\PY{p}{(}\PY{n}{predict}\PY{p}{(}\PY{n}{X\PYZus{}train}\PY{p}{)}\PY{p}{,} \PY{n}{Y\PYZus{}train}\PY{p}{)}\PY{p}{)}\PY{p}{)}
+        
+        \PY{n}{eta} \PY{o}{=} \PY{l+m+mf}{0.01}
+        \PY{n}{lmbd} \PY{o}{=} \PY{l+m+mf}{0.01}
+        \PY{k}{for} \PY{n}{i} \PY{o+ow}{in} \PY{n+nb}{range}\PY{p}{(}\PY{l+m+mi}{1000}\PY{p}{)}\PY{p}{:}
+            \PY{c+c1}{\PYZsh{} calculate gradients}
+            \PY{n}{dWo}\PY{p}{,} \PY{n}{dBo}\PY{p}{,} \PY{n}{dWh}\PY{p}{,} \PY{n}{dBh} \PY{o}{=} \PY{n}{backpropagation}\PY{p}{(}\PY{n}{X\PYZus{}train}\PY{p}{,} \PY{n}{Y\PYZus{}train\PYZus{}onehot}\PY{p}{)}
+            
+            \PY{c+c1}{\PYZsh{} regularization term gradients}
+            \PY{n}{dWo} \PY{o}{+}\PY{o}{=} \PY{n}{lmbd} \PY{o}{*} \PY{n}{output\PYZus{}weights}
+            \PY{n}{dWh} \PY{o}{+}\PY{o}{=} \PY{n}{lmbd} \PY{o}{*} \PY{n}{hidden\PYZus{}weights}
+            
+            \PY{c+c1}{\PYZsh{} update weights and biases}
+            \PY{n}{output\PYZus{}weights} \PY{o}{\PYZhy{}}\PY{o}{=} \PY{n}{eta} \PY{o}{*} \PY{n}{dWo}
+            \PY{n}{output\PYZus{}bias} \PY{o}{\PYZhy{}}\PY{o}{=} \PY{n}{eta} \PY{o}{*} \PY{n}{dBo}
+            \PY{n}{hidden\PYZus{}weights} \PY{o}{\PYZhy{}}\PY{o}{=} \PY{n}{eta} \PY{o}{*} \PY{n}{dWh}
+            \PY{n}{hidden\PYZus{}bias} \PY{o}{\PYZhy{}}\PY{o}{=} \PY{n}{eta} \PY{o}{*} \PY{n}{dBh}
+        
+        \PY{n+nb}{print}\PY{p}{(}\PY{l+s+s2}{\PYZdq{}}\PY{l+s+s2}{New accuracy on training data: }\PY{l+s+s2}{\PYZdq{}} \PY{o}{+} \PY{n+nb}{str}\PY{p}{(}\PY{n}{accuracy\PYZus{}score}\PY{p}{(}\PY{n}{predict}\PY{p}{(}\PY{n}{X\PYZus{}train}\PY{p}{)}\PY{p}{,} \PY{n}{Y\PYZus{}train}\PY{p}{)}\PY{p}{)}\PY{p}{)}
+\end{Verbatim}
+
+
+    \begin{Verbatim}[commandchars=\\\{\}]
+Old accuracy on training data: 0.16423103688239388
+
+    \end{Verbatim}
+
+    \begin{Verbatim}[commandchars=\\\{\}]
+/usr/local/lib/python3.7/site-packages/ipykernel\_launcher.py:4: RuntimeWarning: overflow encountered in exp
+  after removing the cwd from sys.path.
+
+    \end{Verbatim}
+
+    \begin{Verbatim}[commandchars=\\\{\}]
+New accuracy on training data: 0.10090466249130133
+
+    \end{Verbatim}
+
+    \hypertarget{improving-performance}{%
+\subsection{Improving performance}\label{improving-performance}}
+
+As we can see the network does not seem to be learning at all. It seems
+to be just guessing the label for each image.\\
+In order to obtain a network that does something useful, we will have to
+do a bit more work.
+
+The choice of \emph{hyperparameters} such as learning rate and
+regularization parameter is hugely influential for the performance of
+the network. Typically a \emph{grid-search} is performed, wherein we
+test different hyperparameters separated by orders of magnitude. For
+example we could test the learning rates
+\(\eta = 10^{-6}, 10^{-5},...,10^{-1}\) with different regularization
+parameters \(\lambda = 10^{-6},...,10^{-0}\).
+
+Next, we haven't implemented minibatching yet, which introduces
+stochasticity and is though to act as an important regularizer on the
+weights. We call a feed-forward + backward pass with a minibatch an
+\emph{iteration}, and a full training period going through the entire
+dataset (\(n/M\) batches) an \emph{epoch}.
+
+If this does not improve network performance, you may want to consider
+altering the network architecture, adding more neurons or hidden
+layers.\\
+Andrew Ng goes through some of these considerations in this
+\href{https://youtu.be/F1ka6a13S9I}{video}. You can find a summary of
+the video
+\href{https://kevinzakka.github.io/2016/09/26/applying-deep-learning/}{here}.
+
+\hypertarget{full-object-oriented-implementation}{%
+\subsection{Full object-oriented
+implementation}\label{full-object-oriented-implementation}}
+
+It is very natural to think of the network as an object, with specific
+instances of the network being realizations of this object with
+different hyperparameters. An implementation using Python classes
+provides a clean structure and interface, and the full implementation of
+our neural network is given below.
+
+    \begin{Verbatim}[commandchars=\\\{\}]
+{\color{incolor}In [{\color{incolor}8}]:} \PY{k}{class} \PY{n+nc}{NeuralNetwork}\PY{p}{:}
+            \PY{k}{def} \PY{n+nf}{\PYZus{}\PYZus{}init\PYZus{}\PYZus{}}\PY{p}{(}
+                \PY{n+nb+bp}{self}\PY{p}{,}
+                \PY{n}{X\PYZus{}data}\PY{p}{,}
+                \PY{n}{Y\PYZus{}data}\PY{p}{,}
+                \PY{n}{n\PYZus{}hidden\PYZus{}neurons}\PY{o}{=}\PY{l+m+mi}{50}\PY{p}{,}
+                \PY{n}{n\PYZus{}categories}\PY{o}{=}\PY{l+m+mi}{10}\PY{p}{,}
+                \PY{n}{epochs}\PY{o}{=}\PY{l+m+mi}{10}\PY{p}{,}
+                \PY{n}{batch\PYZus{}size}\PY{o}{=}\PY{l+m+mi}{100}\PY{p}{,}
+                \PY{n}{eta}\PY{o}{=}\PY{l+m+mf}{0.1}\PY{p}{,}
+                \PY{n}{lmbd}\PY{o}{=}\PY{l+m+mf}{0.0}\PY{p}{,}
+        
+            \PY{p}{)}\PY{p}{:}
+                \PY{n+nb+bp}{self}\PY{o}{.}\PY{n}{X\PYZus{}data\PYZus{}full} \PY{o}{=} \PY{n}{X\PYZus{}data}
+                \PY{n+nb+bp}{self}\PY{o}{.}\PY{n}{Y\PYZus{}data\PYZus{}full} \PY{o}{=} \PY{n}{Y\PYZus{}data}
+        
+                \PY{n+nb+bp}{self}\PY{o}{.}\PY{n}{n\PYZus{}inputs} \PY{o}{=} \PY{n}{X\PYZus{}data}\PY{o}{.}\PY{n}{shape}\PY{p}{[}\PY{l+m+mi}{0}\PY{p}{]}
+                \PY{n+nb+bp}{self}\PY{o}{.}\PY{n}{n\PYZus{}features} \PY{o}{=} \PY{n}{X\PYZus{}data}\PY{o}{.}\PY{n}{shape}\PY{p}{[}\PY{l+m+mi}{1}\PY{p}{]}
+                \PY{n+nb+bp}{self}\PY{o}{.}\PY{n}{n\PYZus{}hidden\PYZus{}neurons} \PY{o}{=} \PY{n}{n\PYZus{}hidden\PYZus{}neurons}
+                \PY{n+nb+bp}{self}\PY{o}{.}\PY{n}{n\PYZus{}categories} \PY{o}{=} \PY{n}{n\PYZus{}categories}
+        
+                \PY{n+nb+bp}{self}\PY{o}{.}\PY{n}{epochs} \PY{o}{=} \PY{n}{epochs}
+                \PY{n+nb+bp}{self}\PY{o}{.}\PY{n}{batch\PYZus{}size} \PY{o}{=} \PY{n}{batch\PYZus{}size}
+                \PY{n+nb+bp}{self}\PY{o}{.}\PY{n}{iterations} \PY{o}{=} \PY{n+nb+bp}{self}\PY{o}{.}\PY{n}{n\PYZus{}inputs} \PY{o}{/}\PY{o}{/} \PY{n+nb+bp}{self}\PY{o}{.}\PY{n}{batch\PYZus{}size}
+                \PY{n+nb+bp}{self}\PY{o}{.}\PY{n}{eta} \PY{o}{=} \PY{n}{eta}
+                \PY{n+nb+bp}{self}\PY{o}{.}\PY{n}{lmbd} \PY{o}{=} \PY{n}{lmbd}
+        
+                \PY{n+nb+bp}{self}\PY{o}{.}\PY{n}{create\PYZus{}biases\PYZus{}and\PYZus{}weights}\PY{p}{(}\PY{p}{)}
+        
+            \PY{k}{def} \PY{n+nf}{create\PYZus{}biases\PYZus{}and\PYZus{}weights}\PY{p}{(}\PY{n+nb+bp}{self}\PY{p}{)}\PY{p}{:}
+                \PY{n+nb+bp}{self}\PY{o}{.}\PY{n}{hidden\PYZus{}weights} \PY{o}{=} \PY{n}{np}\PY{o}{.}\PY{n}{random}\PY{o}{.}\PY{n}{randn}\PY{p}{(}\PY{n+nb+bp}{self}\PY{o}{.}\PY{n}{n\PYZus{}features}\PY{p}{,} \PY{n+nb+bp}{self}\PY{o}{.}\PY{n}{n\PYZus{}hidden\PYZus{}neurons}\PY{p}{)}
+                \PY{n+nb+bp}{self}\PY{o}{.}\PY{n}{hidden\PYZus{}bias} \PY{o}{=} \PY{n}{np}\PY{o}{.}\PY{n}{zeros}\PY{p}{(}\PY{n+nb+bp}{self}\PY{o}{.}\PY{n}{n\PYZus{}hidden\PYZus{}neurons}\PY{p}{)} \PY{o}{+} \PY{l+m+mf}{0.01}
+        
+                \PY{n+nb+bp}{self}\PY{o}{.}\PY{n}{output\PYZus{}weights} \PY{o}{=} \PY{n}{np}\PY{o}{.}\PY{n}{random}\PY{o}{.}\PY{n}{randn}\PY{p}{(}\PY{n+nb+bp}{self}\PY{o}{.}\PY{n}{n\PYZus{}hidden\PYZus{}neurons}\PY{p}{,} \PY{n+nb+bp}{self}\PY{o}{.}\PY{n}{n\PYZus{}categories}\PY{p}{)}
+                \PY{n+nb+bp}{self}\PY{o}{.}\PY{n}{output\PYZus{}bias} \PY{o}{=} \PY{n}{np}\PY{o}{.}\PY{n}{zeros}\PY{p}{(}\PY{n+nb+bp}{self}\PY{o}{.}\PY{n}{n\PYZus{}categories}\PY{p}{)} \PY{o}{+} \PY{l+m+mf}{0.01}
+        
+            \PY{k}{def} \PY{n+nf}{feed\PYZus{}forward}\PY{p}{(}\PY{n+nb+bp}{self}\PY{p}{)}\PY{p}{:}
+                \PY{c+c1}{\PYZsh{} feed\PYZhy{}forward for training}
+                \PY{n+nb+bp}{self}\PY{o}{.}\PY{n}{z\PYZus{}h} \PY{o}{=} \PY{n}{np}\PY{o}{.}\PY{n}{matmul}\PY{p}{(}\PY{n+nb+bp}{self}\PY{o}{.}\PY{n}{X\PYZus{}data}\PY{p}{,} \PY{n+nb+bp}{self}\PY{o}{.}\PY{n}{hidden\PYZus{}weights}\PY{p}{)} \PY{o}{+} \PY{n+nb+bp}{self}\PY{o}{.}\PY{n}{hidden\PYZus{}bias}
+                \PY{n+nb+bp}{self}\PY{o}{.}\PY{n}{a\PYZus{}h} \PY{o}{=} \PY{n}{sigmoid}\PY{p}{(}\PY{n+nb+bp}{self}\PY{o}{.}\PY{n}{z\PYZus{}h}\PY{p}{)}
+        
+                \PY{n+nb+bp}{self}\PY{o}{.}\PY{n}{z\PYZus{}o} \PY{o}{=} \PY{n}{np}\PY{o}{.}\PY{n}{matmul}\PY{p}{(}\PY{n+nb+bp}{self}\PY{o}{.}\PY{n}{a\PYZus{}h}\PY{p}{,} \PY{n+nb+bp}{self}\PY{o}{.}\PY{n}{output\PYZus{}weights}\PY{p}{)} \PY{o}{+} \PY{n+nb+bp}{self}\PY{o}{.}\PY{n}{output\PYZus{}bias}
+        
+                \PY{n}{exp\PYZus{}term} \PY{o}{=} \PY{n}{np}\PY{o}{.}\PY{n}{exp}\PY{p}{(}\PY{n+nb+bp}{self}\PY{o}{.}\PY{n}{z\PYZus{}o}\PY{p}{)}
+                \PY{n+nb+bp}{self}\PY{o}{.}\PY{n}{probabilities} \PY{o}{=} \PY{n}{exp\PYZus{}term} \PY{o}{/} \PY{n}{np}\PY{o}{.}\PY{n}{sum}\PY{p}{(}\PY{n}{exp\PYZus{}term}\PY{p}{,} \PY{n}{axis}\PY{o}{=}\PY{l+m+mi}{1}\PY{p}{,} \PY{n}{keepdims}\PY{o}{=}\PY{k+kc}{True}\PY{p}{)}
+        
+            \PY{k}{def} \PY{n+nf}{feed\PYZus{}forward\PYZus{}out}\PY{p}{(}\PY{n+nb+bp}{self}\PY{p}{,} \PY{n}{X}\PY{p}{)}\PY{p}{:}
+                \PY{c+c1}{\PYZsh{} feed\PYZhy{}forward for output}
+                \PY{n}{z\PYZus{}h} \PY{o}{=} \PY{n}{np}\PY{o}{.}\PY{n}{matmul}\PY{p}{(}\PY{n}{X}\PY{p}{,} \PY{n+nb+bp}{self}\PY{o}{.}\PY{n}{hidden\PYZus{}weights}\PY{p}{)} \PY{o}{+} \PY{n+nb+bp}{self}\PY{o}{.}\PY{n}{hidden\PYZus{}bias}
+                \PY{n}{a\PYZus{}h} \PY{o}{=} \PY{n}{sigmoid}\PY{p}{(}\PY{n}{z\PYZus{}h}\PY{p}{)}
+        
+                \PY{n}{z\PYZus{}o} \PY{o}{=} \PY{n}{np}\PY{o}{.}\PY{n}{matmul}\PY{p}{(}\PY{n}{a\PYZus{}h}\PY{p}{,} \PY{n+nb+bp}{self}\PY{o}{.}\PY{n}{output\PYZus{}weights}\PY{p}{)} \PY{o}{+} \PY{n+nb+bp}{self}\PY{o}{.}\PY{n}{output\PYZus{}bias}
+                
+                \PY{n}{exp\PYZus{}term} \PY{o}{=} \PY{n}{np}\PY{o}{.}\PY{n}{exp}\PY{p}{(}\PY{n}{z\PYZus{}o}\PY{p}{)}
+                \PY{n}{probabilities} \PY{o}{=} \PY{n}{exp\PYZus{}term} \PY{o}{/} \PY{n}{np}\PY{o}{.}\PY{n}{sum}\PY{p}{(}\PY{n}{exp\PYZus{}term}\PY{p}{,} \PY{n}{axis}\PY{o}{=}\PY{l+m+mi}{1}\PY{p}{,} \PY{n}{keepdims}\PY{o}{=}\PY{k+kc}{True}\PY{p}{)}
+                \PY{k}{return} \PY{n}{probabilities}
+        
+            \PY{k}{def} \PY{n+nf}{backpropagation}\PY{p}{(}\PY{n+nb+bp}{self}\PY{p}{)}\PY{p}{:}
+                \PY{n}{error\PYZus{}output} \PY{o}{=} \PY{n+nb+bp}{self}\PY{o}{.}\PY{n}{probabilities} \PY{o}{\PYZhy{}} \PY{n+nb+bp}{self}\PY{o}{.}\PY{n}{Y\PYZus{}data}
+                \PY{n}{error\PYZus{}hidden} \PY{o}{=} \PY{n}{np}\PY{o}{.}\PY{n}{matmul}\PY{p}{(}\PY{n}{error\PYZus{}output}\PY{p}{,} \PY{n+nb+bp}{self}\PY{o}{.}\PY{n}{output\PYZus{}weights}\PY{o}{.}\PY{n}{T}\PY{p}{)} \PY{o}{*} \PY{n+nb+bp}{self}\PY{o}{.}\PY{n}{a\PYZus{}h} \PY{o}{*} \PY{p}{(}\PY{l+m+mi}{1} \PY{o}{\PYZhy{}} \PY{n+nb+bp}{self}\PY{o}{.}\PY{n}{a\PYZus{}h}\PY{p}{)}
+        
+                \PY{n+nb+bp}{self}\PY{o}{.}\PY{n}{output\PYZus{}weights\PYZus{}gradient} \PY{o}{=} \PY{n}{np}\PY{o}{.}\PY{n}{matmul}\PY{p}{(}\PY{n+nb+bp}{self}\PY{o}{.}\PY{n}{a\PYZus{}h}\PY{o}{.}\PY{n}{T}\PY{p}{,} \PY{n}{error\PYZus{}output}\PY{p}{)}
+                \PY{n+nb+bp}{self}\PY{o}{.}\PY{n}{output\PYZus{}bias\PYZus{}gradient} \PY{o}{=} \PY{n}{np}\PY{o}{.}\PY{n}{sum}\PY{p}{(}\PY{n}{error\PYZus{}output}\PY{p}{,} \PY{n}{axis}\PY{o}{=}\PY{l+m+mi}{0}\PY{p}{)}
+        
+                \PY{n+nb+bp}{self}\PY{o}{.}\PY{n}{hidden\PYZus{}weights\PYZus{}gradient} \PY{o}{=} \PY{n}{np}\PY{o}{.}\PY{n}{matmul}\PY{p}{(}\PY{n+nb+bp}{self}\PY{o}{.}\PY{n}{X\PYZus{}data}\PY{o}{.}\PY{n}{T}\PY{p}{,} \PY{n}{error\PYZus{}hidden}\PY{p}{)}
+                \PY{n+nb+bp}{self}\PY{o}{.}\PY{n}{hidden\PYZus{}bias\PYZus{}gradient} \PY{o}{=} \PY{n}{np}\PY{o}{.}\PY{n}{sum}\PY{p}{(}\PY{n}{error\PYZus{}hidden}\PY{p}{,} \PY{n}{axis}\PY{o}{=}\PY{l+m+mi}{0}\PY{p}{)}
+        
+                \PY{k}{if} \PY{n+nb+bp}{self}\PY{o}{.}\PY{n}{lmbd} \PY{o}{\PYZgt{}} \PY{l+m+mf}{0.0}\PY{p}{:}
+                    \PY{n+nb+bp}{self}\PY{o}{.}\PY{n}{output\PYZus{}weights\PYZus{}gradient} \PY{o}{+}\PY{o}{=} \PY{n+nb+bp}{self}\PY{o}{.}\PY{n}{lmbd} \PY{o}{*} \PY{n+nb+bp}{self}\PY{o}{.}\PY{n}{output\PYZus{}weights}
+                    \PY{n+nb+bp}{self}\PY{o}{.}\PY{n}{hidden\PYZus{}weights\PYZus{}gradient} \PY{o}{+}\PY{o}{=} \PY{n+nb+bp}{self}\PY{o}{.}\PY{n}{lmbd} \PY{o}{*} \PY{n+nb+bp}{self}\PY{o}{.}\PY{n}{hidden\PYZus{}weights}
+        
+                \PY{n+nb+bp}{self}\PY{o}{.}\PY{n}{output\PYZus{}weights} \PY{o}{\PYZhy{}}\PY{o}{=} \PY{n+nb+bp}{self}\PY{o}{.}\PY{n}{eta} \PY{o}{*} \PY{n+nb+bp}{self}\PY{o}{.}\PY{n}{output\PYZus{}weights\PYZus{}gradient}
+                \PY{n+nb+bp}{self}\PY{o}{.}\PY{n}{output\PYZus{}bias} \PY{o}{\PYZhy{}}\PY{o}{=} \PY{n+nb+bp}{self}\PY{o}{.}\PY{n}{eta} \PY{o}{*} \PY{n+nb+bp}{self}\PY{o}{.}\PY{n}{output\PYZus{}bias\PYZus{}gradient}
+                \PY{n+nb+bp}{self}\PY{o}{.}\PY{n}{hidden\PYZus{}weights} \PY{o}{\PYZhy{}}\PY{o}{=} \PY{n+nb+bp}{self}\PY{o}{.}\PY{n}{eta} \PY{o}{*} \PY{n+nb+bp}{self}\PY{o}{.}\PY{n}{hidden\PYZus{}weights\PYZus{}gradient}
+                \PY{n+nb+bp}{self}\PY{o}{.}\PY{n}{hidden\PYZus{}bias} \PY{o}{\PYZhy{}}\PY{o}{=} \PY{n+nb+bp}{self}\PY{o}{.}\PY{n}{eta} \PY{o}{*} \PY{n+nb+bp}{self}\PY{o}{.}\PY{n}{hidden\PYZus{}bias\PYZus{}gradient}
+        
+            \PY{k}{def} \PY{n+nf}{predict}\PY{p}{(}\PY{n+nb+bp}{self}\PY{p}{,} \PY{n}{X}\PY{p}{)}\PY{p}{:}
+                \PY{n}{probabilities} \PY{o}{=} \PY{n+nb+bp}{self}\PY{o}{.}\PY{n}{feed\PYZus{}forward\PYZus{}out}\PY{p}{(}\PY{n}{X}\PY{p}{)}
+                \PY{k}{return} \PY{n}{np}\PY{o}{.}\PY{n}{argmax}\PY{p}{(}\PY{n}{probabilities}\PY{p}{,} \PY{n}{axis}\PY{o}{=}\PY{l+m+mi}{1}\PY{p}{)}
+        
+            \PY{k}{def} \PY{n+nf}{predict\PYZus{}probabilities}\PY{p}{(}\PY{n+nb+bp}{self}\PY{p}{,} \PY{n}{X}\PY{p}{)}\PY{p}{:}
+                \PY{n}{probabilities} \PY{o}{=} \PY{n+nb+bp}{self}\PY{o}{.}\PY{n}{feed\PYZus{}forward\PYZus{}out}\PY{p}{(}\PY{n}{X}\PY{p}{)}
+                \PY{k}{return} \PY{n}{probabilities}
+        
+            \PY{k}{def} \PY{n+nf}{train}\PY{p}{(}\PY{n+nb+bp}{self}\PY{p}{)}\PY{p}{:}
+                \PY{n}{data\PYZus{}indices} \PY{o}{=} \PY{n}{np}\PY{o}{.}\PY{n}{arange}\PY{p}{(}\PY{n+nb+bp}{self}\PY{o}{.}\PY{n}{n\PYZus{}inputs}\PY{p}{)}
+        
+                \PY{k}{for} \PY{n}{i} \PY{o+ow}{in} \PY{n+nb}{range}\PY{p}{(}\PY{n+nb+bp}{self}\PY{o}{.}\PY{n}{epochs}\PY{p}{)}\PY{p}{:}
+                    \PY{k}{for} \PY{n}{j} \PY{o+ow}{in} \PY{n+nb}{range}\PY{p}{(}\PY{n+nb+bp}{self}\PY{o}{.}\PY{n}{iterations}\PY{p}{)}\PY{p}{:}
+                        \PY{c+c1}{\PYZsh{} pick datapoints with replacement}
+                        \PY{n}{chosen\PYZus{}datapoints} \PY{o}{=} \PY{n}{np}\PY{o}{.}\PY{n}{random}\PY{o}{.}\PY{n}{choice}\PY{p}{(}
+                            \PY{n}{data\PYZus{}indices}\PY{p}{,} \PY{n}{size}\PY{o}{=}\PY{n+nb+bp}{self}\PY{o}{.}\PY{n}{batch\PYZus{}size}\PY{p}{,} \PY{n}{replace}\PY{o}{=}\PY{k+kc}{False}
+                        \PY{p}{)}
+        
+                        \PY{c+c1}{\PYZsh{} minibatch training data}
+                        \PY{n+nb+bp}{self}\PY{o}{.}\PY{n}{X\PYZus{}data} \PY{o}{=} \PY{n+nb+bp}{self}\PY{o}{.}\PY{n}{X\PYZus{}data\PYZus{}full}\PY{p}{[}\PY{n}{chosen\PYZus{}datapoints}\PY{p}{]}
+                        \PY{n+nb+bp}{self}\PY{o}{.}\PY{n}{Y\PYZus{}data} \PY{o}{=} \PY{n+nb+bp}{self}\PY{o}{.}\PY{n}{Y\PYZus{}data\PYZus{}full}\PY{p}{[}\PY{n}{chosen\PYZus{}datapoints}\PY{p}{]}
+        
+                        \PY{n+nb+bp}{self}\PY{o}{.}\PY{n}{feed\PYZus{}forward}\PY{p}{(}\PY{p}{)}
+                        \PY{n+nb+bp}{self}\PY{o}{.}\PY{n}{backpropagation}\PY{p}{(}\PY{p}{)}
+\end{Verbatim}
+
+
+    \hypertarget{evaluate-model-performance-on-test-data}{%
+\subsection{Evaluate model performance on test
+data}\label{evaluate-model-performance-on-test-data}}
+
+To measure the performance of our network we evaluate how well it does
+it data it has never seen before, i.e.~the test data.\\
+We measure the performance of the network using the \emph{accuracy}
+score.\\
+The accuracy is as you would expect just the number of images correctly
+labeled divided by the total number of images. A perfect classifier will
+have an accuracy score of \(1\).
+
+\[ \text{Accuracy} = \frac{\sum_{i=1}^n I(\hat{y}_i = y_i)}{n} ,\]
+
+where \(I\) is the indicator function, \(1\) if \(\hat{y}_i = y_i\) and
+\(0\) otherwise.
+
+    \begin{Verbatim}[commandchars=\\\{\}]
+{\color{incolor}In [{\color{incolor}9}]:} \PY{n}{epochs} \PY{o}{=} \PY{l+m+mi}{100}
+        \PY{n}{batch\PYZus{}size} \PY{o}{=} \PY{l+m+mi}{100}
+        
+        \PY{n}{dnn} \PY{o}{=} \PY{n}{NeuralNetwork}\PY{p}{(}\PY{n}{X\PYZus{}train}\PY{p}{,} \PY{n}{Y\PYZus{}train\PYZus{}onehot}\PY{p}{,} \PY{n}{eta}\PY{o}{=}\PY{n}{eta}\PY{p}{,} \PY{n}{lmbd}\PY{o}{=}\PY{n}{lmbd}\PY{p}{,} \PY{n}{epochs}\PY{o}{=}\PY{n}{epochs}\PY{p}{,} \PY{n}{batch\PYZus{}size}\PY{o}{=}\PY{n}{batch\PYZus{}size}\PY{p}{,}
+                            \PY{n}{n\PYZus{}hidden\PYZus{}neurons}\PY{o}{=}\PY{n}{n\PYZus{}hidden\PYZus{}neurons}\PY{p}{,} \PY{n}{n\PYZus{}categories}\PY{o}{=}\PY{n}{n\PYZus{}categories}\PY{p}{)}
+        \PY{n}{dnn}\PY{o}{.}\PY{n}{train}\PY{p}{(}\PY{p}{)}
+        \PY{n}{test\PYZus{}predict} \PY{o}{=} \PY{n}{dnn}\PY{o}{.}\PY{n}{predict}\PY{p}{(}\PY{n}{X\PYZus{}test}\PY{p}{)}
+        
+        \PY{c+c1}{\PYZsh{} accuracy score from scikit library}
+        \PY{n+nb}{print}\PY{p}{(}\PY{l+s+s2}{\PYZdq{}}\PY{l+s+s2}{Accuracy score on test set: }\PY{l+s+s2}{\PYZdq{}}\PY{p}{,} \PY{n}{accuracy\PYZus{}score}\PY{p}{(}\PY{n}{Y\PYZus{}test}\PY{p}{,} \PY{n}{test\PYZus{}predict}\PY{p}{)}\PY{p}{)}
+        
+        \PY{c+c1}{\PYZsh{} equivalent in numpy}
+        \PY{k}{def} \PY{n+nf}{accuracy\PYZus{}score\PYZus{}numpy}\PY{p}{(}\PY{n}{Y\PYZus{}test}\PY{p}{,} \PY{n}{Y\PYZus{}pred}\PY{p}{)}\PY{p}{:}
+            \PY{k}{return} \PY{n}{np}\PY{o}{.}\PY{n}{sum}\PY{p}{(}\PY{n}{Y\PYZus{}test} \PY{o}{==} \PY{n}{Y\PYZus{}pred}\PY{p}{)} \PY{o}{/} \PY{n+nb}{len}\PY{p}{(}\PY{n}{Y\PYZus{}test}\PY{p}{)}
+        
+        \PY{c+c1}{\PYZsh{}print(\PYZdq{}Accuracy score on test set: \PYZdq{}, accuracy\PYZus{}score\PYZus{}numpy(Y\PYZus{}test, test\PYZus{}predict))}
+\end{Verbatim}
+
+
+    \begin{Verbatim}[commandchars=\\\{\}]
+Accuracy score on test set:  0.9305555555555556
+
+    \end{Verbatim}
+
+    \hypertarget{adjust-hyperparameters}{%
+\subsection{Adjust hyperparameters}\label{adjust-hyperparameters}}
+
+We now perform a grid search to find the optimal hyperparameters for the
+network.\\
+Note that we are only using 1 layer with 50 neurons, and human
+performance is estimated to be around \(98\%\) (\(2\%\) error rate).
+
+    \begin{Verbatim}[commandchars=\\\{\}]
+{\color{incolor}In [{\color{incolor}10}]:} \PY{n}{eta\PYZus{}vals} \PY{o}{=} \PY{n}{np}\PY{o}{.}\PY{n}{logspace}\PY{p}{(}\PY{o}{\PYZhy{}}\PY{l+m+mi}{5}\PY{p}{,} \PY{l+m+mi}{1}\PY{p}{,} \PY{l+m+mi}{7}\PY{p}{)}
+         \PY{n}{lmbd\PYZus{}vals} \PY{o}{=} \PY{n}{np}\PY{o}{.}\PY{n}{logspace}\PY{p}{(}\PY{o}{\PYZhy{}}\PY{l+m+mi}{5}\PY{p}{,} \PY{l+m+mi}{1}\PY{p}{,} \PY{l+m+mi}{7}\PY{p}{)}
+         \PY{c+c1}{\PYZsh{} store the models for later use}
+         \PY{n}{DNN\PYZus{}numpy} \PY{o}{=} \PY{n}{np}\PY{o}{.}\PY{n}{zeros}\PY{p}{(}\PY{p}{(}\PY{n+nb}{len}\PY{p}{(}\PY{n}{eta\PYZus{}vals}\PY{p}{)}\PY{p}{,} \PY{n+nb}{len}\PY{p}{(}\PY{n}{lmbd\PYZus{}vals}\PY{p}{)}\PY{p}{)}\PY{p}{,} \PY{n}{dtype}\PY{o}{=}\PY{n+nb}{object}\PY{p}{)}
+         
+         \PY{c+c1}{\PYZsh{} grid search}
+         \PY{k}{for} \PY{n}{i}\PY{p}{,} \PY{n}{eta} \PY{o+ow}{in} \PY{n+nb}{enumerate}\PY{p}{(}\PY{n}{eta\PYZus{}vals}\PY{p}{)}\PY{p}{:}
+             \PY{k}{for} \PY{n}{j}\PY{p}{,} \PY{n}{lmbd} \PY{o+ow}{in} \PY{n+nb}{enumerate}\PY{p}{(}\PY{n}{lmbd\PYZus{}vals}\PY{p}{)}\PY{p}{:}
+                 \PY{n}{dnn} \PY{o}{=} \PY{n}{NeuralNetwork}\PY{p}{(}\PY{n}{X\PYZus{}train}\PY{p}{,} \PY{n}{Y\PYZus{}train\PYZus{}onehot}\PY{p}{,} \PY{n}{eta}\PY{o}{=}\PY{n}{eta}\PY{p}{,} \PY{n}{lmbd}\PY{o}{=}\PY{n}{lmbd}\PY{p}{,} \PY{n}{epochs}\PY{o}{=}\PY{n}{epochs}\PY{p}{,} \PY{n}{batch\PYZus{}size}\PY{o}{=}\PY{n}{batch\PYZus{}size}\PY{p}{,}
+                                     \PY{n}{n\PYZus{}hidden\PYZus{}neurons}\PY{o}{=}\PY{n}{n\PYZus{}hidden\PYZus{}neurons}\PY{p}{,} \PY{n}{n\PYZus{}categories}\PY{o}{=}\PY{n}{n\PYZus{}categories}\PY{p}{)}
+                 \PY{n}{dnn}\PY{o}{.}\PY{n}{train}\PY{p}{(}\PY{p}{)}
+                 
+                 \PY{n}{DNN\PYZus{}numpy}\PY{p}{[}\PY{n}{i}\PY{p}{]}\PY{p}{[}\PY{n}{j}\PY{p}{]} \PY{o}{=} \PY{n}{dnn}
+                 
+                 \PY{n}{test\PYZus{}predict} \PY{o}{=} \PY{n}{dnn}\PY{o}{.}\PY{n}{predict}\PY{p}{(}\PY{n}{X\PYZus{}test}\PY{p}{)}
+                 
+                 \PY{n+nb}{print}\PY{p}{(}\PY{l+s+s2}{\PYZdq{}}\PY{l+s+s2}{Learning rate  = }\PY{l+s+s2}{\PYZdq{}}\PY{p}{,} \PY{n}{eta}\PY{p}{)}
+                 \PY{n+nb}{print}\PY{p}{(}\PY{l+s+s2}{\PYZdq{}}\PY{l+s+s2}{Lambda = }\PY{l+s+s2}{\PYZdq{}}\PY{p}{,} \PY{n}{lmbd}\PY{p}{)}
+                 \PY{n+nb}{print}\PY{p}{(}\PY{l+s+s2}{\PYZdq{}}\PY{l+s+s2}{Accuracy score on test set: }\PY{l+s+s2}{\PYZdq{}}\PY{p}{,} \PY{n}{accuracy\PYZus{}score}\PY{p}{(}\PY{n}{Y\PYZus{}test}\PY{p}{,} \PY{n}{test\PYZus{}predict}\PY{p}{)}\PY{p}{)}
+                 \PY{n+nb}{print}\PY{p}{(}\PY{p}{)}
+\end{Verbatim}
+
+
+    \begin{Verbatim}[commandchars=\\\{\}]
+Learning rate  =  1e-05
+Lambda =  1e-05
+Accuracy score on test set:  0.18888888888888888
+
+Learning rate  =  1e-05
+Lambda =  0.0001
+Accuracy score on test set:  0.18333333333333332
+
+Learning rate  =  1e-05
+Lambda =  0.001
+Accuracy score on test set:  0.20277777777777778
+
+Learning rate  =  1e-05
+Lambda =  0.01
+Accuracy score on test set:  0.2
+
+Learning rate  =  1e-05
+Lambda =  0.1
+Accuracy score on test set:  0.12222222222222222
+
+Learning rate  =  1e-05
+Lambda =  1.0
+Accuracy score on test set:  0.18888888888888888
+
+Learning rate  =  1e-05
+Lambda =  10.0
+Accuracy score on test set:  0.1527777777777778
+
+Learning rate  =  0.0001
+Lambda =  1e-05
+Accuracy score on test set:  0.5916666666666667
+
+Learning rate  =  0.0001
+Lambda =  0.0001
+Accuracy score on test set:  0.5583333333333333
+
+Learning rate  =  0.0001
+Lambda =  0.001
+Accuracy score on test set:  0.5361111111111111
+
+Learning rate  =  0.0001
+Lambda =  0.01
+Accuracy score on test set:  0.5777777777777777
+
+Learning rate  =  0.0001
+Lambda =  0.1
+Accuracy score on test set:  0.6055555555555555
+
+Learning rate  =  0.0001
+Lambda =  1.0
+Accuracy score on test set:  0.6416666666666667
+
+Learning rate  =  0.0001
+Lambda =  10.0
+Accuracy score on test set:  0.8111111111111111
+
+Learning rate  =  0.001
+Lambda =  1e-05
+Accuracy score on test set:  0.8833333333333333
+
+Learning rate  =  0.001
+Lambda =  0.0001
+Accuracy score on test set:  0.9
+
+Learning rate  =  0.001
+Lambda =  0.001
+Accuracy score on test set:  0.8666666666666667
+
+Learning rate  =  0.001
+Lambda =  0.01
+Accuracy score on test set:  0.875
+
+Learning rate  =  0.001
+Lambda =  0.1
+Accuracy score on test set:  0.8666666666666667
+
+Learning rate  =  0.001
+Lambda =  1.0
+Accuracy score on test set:  0.9416666666666667
+
+Learning rate  =  0.001
+Lambda =  10.0
+Accuracy score on test set:  0.9416666666666667
+
+Learning rate  =  0.01
+Lambda =  1e-05
+Accuracy score on test set:  0.9472222222222222
+
+Learning rate  =  0.01
+Lambda =  0.0001
+Accuracy score on test set:  0.9333333333333333
+
+Learning rate  =  0.01
+Lambda =  0.001
+Accuracy score on test set:  0.9416666666666667
+
+Learning rate  =  0.01
+Lambda =  0.01
+Accuracy score on test set:  0.9361111111111111
+
+Learning rate  =  0.01
+Lambda =  0.1
+Accuracy score on test set:  0.9527777777777777
+
+Learning rate  =  0.01
+Lambda =  1.0
+Accuracy score on test set:  0.9333333333333333
+
+Learning rate  =  0.01
+Lambda =  10.0
+Accuracy score on test set:  0.23055555555555557
+
+
+    \end{Verbatim}
+
+    \begin{Verbatim}[commandchars=\\\{\}]
+/usr/local/lib/python3.7/site-packages/ipykernel\_launcher.py:4: RuntimeWarning: overflow encountered in exp
+  after removing the cwd from sys.path.
+
+    \end{Verbatim}
+
+    \begin{Verbatim}[commandchars=\\\{\}]
+Learning rate  =  0.1
+Lambda =  1e-05
+Accuracy score on test set:  0.10555555555555556
+
+Learning rate  =  0.1
+Lambda =  0.0001
+Accuracy score on test set:  0.10555555555555556
+
+Learning rate  =  0.1
+Lambda =  0.001
+Accuracy score on test set:  0.11666666666666667
+
+Learning rate  =  0.1
+Lambda =  0.01
+Accuracy score on test set:  0.07777777777777778
+
+Learning rate  =  0.1
+Lambda =  0.1
+Accuracy score on test set:  0.08611111111111111
+
+Learning rate  =  0.1
+Lambda =  1.0
+Accuracy score on test set:  0.10555555555555556
+
+Learning rate  =  0.1
+Lambda =  10.0
+Accuracy score on test set:  0.125
+
+
+    \end{Verbatim}
+
+    \begin{Verbatim}[commandchars=\\\{\}]
+/usr/local/lib/python3.7/site-packages/ipykernel\_launcher.py:44: RuntimeWarning: overflow encountered in exp
+/usr/local/lib/python3.7/site-packages/ipykernel\_launcher.py:45: RuntimeWarning: invalid value encountered in true\_divide
+
+    \end{Verbatim}
+
+    \begin{Verbatim}[commandchars=\\\{\}]
+Learning rate  =  1.0
+Lambda =  1e-05
+Accuracy score on test set:  0.07777777777777778
+
+Learning rate  =  1.0
+Lambda =  0.0001
+Accuracy score on test set:  0.07777777777777778
+
+Learning rate  =  1.0
+Lambda =  0.001
+Accuracy score on test set:  0.07777777777777778
+
+Learning rate  =  1.0
+Lambda =  0.01
+Accuracy score on test set:  0.07777777777777778
+
+Learning rate  =  1.0
+Lambda =  0.1
+Accuracy score on test set:  0.07777777777777778
+
+Learning rate  =  1.0
+Lambda =  1.0
+Accuracy score on test set:  0.10555555555555556
+
+Learning rate  =  1.0
+Lambda =  10.0
+Accuracy score on test set:  0.07777777777777778
+
+Learning rate  =  10.0
+Lambda =  1e-05
+Accuracy score on test set:  0.07777777777777778
+
+Learning rate  =  10.0
+Lambda =  0.0001
+Accuracy score on test set:  0.07777777777777778
+
+Learning rate  =  10.0
+Lambda =  0.001
+Accuracy score on test set:  0.07777777777777778
+
+Learning rate  =  10.0
+Lambda =  0.01
+Accuracy score on test set:  0.07777777777777778
+
+Learning rate  =  10.0
+Lambda =  0.1
+Accuracy score on test set:  0.07777777777777778
+
+Learning rate  =  10.0
+Lambda =  1.0
+Accuracy score on test set:  0.07777777777777778
+
+Learning rate  =  10.0
+Lambda =  10.0
+Accuracy score on test set:  0.07777777777777778
+
+
+    \end{Verbatim}
+
+    \hypertarget{visualization}{%
+\subsection{Visualization}\label{visualization}}
+
+    \begin{Verbatim}[commandchars=\\\{\}]
+{\color{incolor}In [{\color{incolor}11}]:} \PY{c+c1}{\PYZsh{} visual representation of grid search}
+         \PY{c+c1}{\PYZsh{} uses seaborn heatmap, you can also do this with matplotlib imshow}
+         \PY{k+kn}{import} \PY{n+nn}{seaborn} \PY{k}{as} \PY{n+nn}{sns}
+         
+         \PY{n}{sns}\PY{o}{.}\PY{n}{set}\PY{p}{(}\PY{p}{)}
+         
+         \PY{n}{train\PYZus{}accuracy} \PY{o}{=} \PY{n}{np}\PY{o}{.}\PY{n}{zeros}\PY{p}{(}\PY{p}{(}\PY{n+nb}{len}\PY{p}{(}\PY{n}{eta\PYZus{}vals}\PY{p}{)}\PY{p}{,} \PY{n+nb}{len}\PY{p}{(}\PY{n}{lmbd\PYZus{}vals}\PY{p}{)}\PY{p}{)}\PY{p}{)}
+         \PY{n}{test\PYZus{}accuracy} \PY{o}{=} \PY{n}{np}\PY{o}{.}\PY{n}{zeros}\PY{p}{(}\PY{p}{(}\PY{n+nb}{len}\PY{p}{(}\PY{n}{eta\PYZus{}vals}\PY{p}{)}\PY{p}{,} \PY{n+nb}{len}\PY{p}{(}\PY{n}{lmbd\PYZus{}vals}\PY{p}{)}\PY{p}{)}\PY{p}{)}
+         
+         \PY{k}{for} \PY{n}{i} \PY{o+ow}{in} \PY{n+nb}{range}\PY{p}{(}\PY{n+nb}{len}\PY{p}{(}\PY{n}{eta\PYZus{}vals}\PY{p}{)}\PY{p}{)}\PY{p}{:}
+             \PY{k}{for} \PY{n}{j} \PY{o+ow}{in} \PY{n+nb}{range}\PY{p}{(}\PY{n+nb}{len}\PY{p}{(}\PY{n}{lmbd\PYZus{}vals}\PY{p}{)}\PY{p}{)}\PY{p}{:}
+                 \PY{n}{dnn} \PY{o}{=} \PY{n}{DNN\PYZus{}numpy}\PY{p}{[}\PY{n}{i}\PY{p}{]}\PY{p}{[}\PY{n}{j}\PY{p}{]}
+                 
+                 \PY{n}{train\PYZus{}pred} \PY{o}{=} \PY{n}{dnn}\PY{o}{.}\PY{n}{predict}\PY{p}{(}\PY{n}{X\PYZus{}train}\PY{p}{)} 
+                 \PY{n}{test\PYZus{}pred} \PY{o}{=} \PY{n}{dnn}\PY{o}{.}\PY{n}{predict}\PY{p}{(}\PY{n}{X\PYZus{}test}\PY{p}{)}
+         
+                 \PY{n}{train\PYZus{}accuracy}\PY{p}{[}\PY{n}{i}\PY{p}{]}\PY{p}{[}\PY{n}{j}\PY{p}{]} \PY{o}{=} \PY{n}{accuracy\PYZus{}score}\PY{p}{(}\PY{n}{Y\PYZus{}train}\PY{p}{,} \PY{n}{train\PYZus{}pred}\PY{p}{)}
+                 \PY{n}{test\PYZus{}accuracy}\PY{p}{[}\PY{n}{i}\PY{p}{]}\PY{p}{[}\PY{n}{j}\PY{p}{]} \PY{o}{=} \PY{n}{accuracy\PYZus{}score}\PY{p}{(}\PY{n}{Y\PYZus{}test}\PY{p}{,} \PY{n}{test\PYZus{}pred}\PY{p}{)}
+         
+                 
+         \PY{n}{fig}\PY{p}{,} \PY{n}{ax} \PY{o}{=} \PY{n}{plt}\PY{o}{.}\PY{n}{subplots}\PY{p}{(}\PY{n}{figsize} \PY{o}{=} \PY{p}{(}\PY{l+m+mi}{10}\PY{p}{,} \PY{l+m+mi}{10}\PY{p}{)}\PY{p}{)}
+         \PY{n}{sns}\PY{o}{.}\PY{n}{heatmap}\PY{p}{(}\PY{n}{train\PYZus{}accuracy}\PY{p}{,} \PY{n}{annot}\PY{o}{=}\PY{k+kc}{True}\PY{p}{,} \PY{n}{ax}\PY{o}{=}\PY{n}{ax}\PY{p}{,} \PY{n}{cmap}\PY{o}{=}\PY{l+s+s2}{\PYZdq{}}\PY{l+s+s2}{viridis}\PY{l+s+s2}{\PYZdq{}}\PY{p}{)}
+         \PY{n}{ax}\PY{o}{.}\PY{n}{set\PYZus{}title}\PY{p}{(}\PY{l+s+s2}{\PYZdq{}}\PY{l+s+s2}{Training Accuracy}\PY{l+s+s2}{\PYZdq{}}\PY{p}{)}
+         \PY{n}{ax}\PY{o}{.}\PY{n}{set\PYZus{}ylabel}\PY{p}{(}\PY{l+s+s2}{\PYZdq{}}\PY{l+s+s2}{\PYZdl{}}\PY{l+s+s2}{\PYZbs{}}\PY{l+s+s2}{eta\PYZdl{}}\PY{l+s+s2}{\PYZdq{}}\PY{p}{)}
+         \PY{n}{ax}\PY{o}{.}\PY{n}{set\PYZus{}xlabel}\PY{p}{(}\PY{l+s+s2}{\PYZdq{}}\PY{l+s+s2}{\PYZdl{}}\PY{l+s+s2}{\PYZbs{}}\PY{l+s+s2}{lambda\PYZdl{}}\PY{l+s+s2}{\PYZdq{}}\PY{p}{)}
+         \PY{n}{plt}\PY{o}{.}\PY{n}{show}\PY{p}{(}\PY{p}{)}
+         
+         \PY{n}{fig}\PY{p}{,} \PY{n}{ax} \PY{o}{=} \PY{n}{plt}\PY{o}{.}\PY{n}{subplots}\PY{p}{(}\PY{n}{figsize} \PY{o}{=} \PY{p}{(}\PY{l+m+mi}{10}\PY{p}{,} \PY{l+m+mi}{10}\PY{p}{)}\PY{p}{)}
+         \PY{n}{sns}\PY{o}{.}\PY{n}{heatmap}\PY{p}{(}\PY{n}{test\PYZus{}accuracy}\PY{p}{,} \PY{n}{annot}\PY{o}{=}\PY{k+kc}{True}\PY{p}{,} \PY{n}{ax}\PY{o}{=}\PY{n}{ax}\PY{p}{,} \PY{n}{cmap}\PY{o}{=}\PY{l+s+s2}{\PYZdq{}}\PY{l+s+s2}{viridis}\PY{l+s+s2}{\PYZdq{}}\PY{p}{)}
+         \PY{n}{ax}\PY{o}{.}\PY{n}{set\PYZus{}title}\PY{p}{(}\PY{l+s+s2}{\PYZdq{}}\PY{l+s+s2}{Test Accuracy}\PY{l+s+s2}{\PYZdq{}}\PY{p}{)}
+         \PY{n}{ax}\PY{o}{.}\PY{n}{set\PYZus{}ylabel}\PY{p}{(}\PY{l+s+s2}{\PYZdq{}}\PY{l+s+s2}{\PYZdl{}}\PY{l+s+s2}{\PYZbs{}}\PY{l+s+s2}{eta\PYZdl{}}\PY{l+s+s2}{\PYZdq{}}\PY{p}{)}
+         \PY{n}{ax}\PY{o}{.}\PY{n}{set\PYZus{}xlabel}\PY{p}{(}\PY{l+s+s2}{\PYZdq{}}\PY{l+s+s2}{\PYZdl{}}\PY{l+s+s2}{\PYZbs{}}\PY{l+s+s2}{lambda\PYZdl{}}\PY{l+s+s2}{\PYZdq{}}\PY{p}{)}
+         \PY{n}{plt}\PY{o}{.}\PY{n}{show}\PY{p}{(}\PY{p}{)}
+\end{Verbatim}
+
+
+    \begin{Verbatim}[commandchars=\\\{\}]
+/usr/local/lib/python3.7/site-packages/ipykernel\_launcher.py:4: RuntimeWarning: overflow encountered in exp
+  after removing the cwd from sys.path.
+
+    \end{Verbatim}
+
+    \begin{center}
+    \adjustimage{max size={0.9\linewidth}{0.9\paperheight}}{output_117_1.png}
+    \end{center}
+    { \hspace*{\fill} \\}
+    
+    \begin{center}
+    \adjustimage{max size={0.9\linewidth}{0.9\paperheight}}{output_117_2.png}
+    \end{center}
+    { \hspace*{\fill} \\}
+    
+    \hypertarget{scikit-learn-implementation}{%
+\subsection{scikit-learn
+implementation}\label{scikit-learn-implementation}}
+
+\textbf{scikit-learn} focuses more on traditional machine learning
+methods, such as regression, clustering, decision trees, etc. As such,
+it has only two types of neural networks: Multi Layer Perceptron
+outputting continuous values, \emph{MPLRegressor}, and Multi Layer
+Perceptron outputting labels, \emph{MLPClassifier}. We will see how
+simple it is to use these classes.
+
+\textbf{scikit-learn} implements a few improvements from our neural
+network, such as early stopping, a varying learning rate, different
+optimization methods, etc. We would therefore expect a better
+performance overall.
+
+    \begin{Verbatim}[commandchars=\\\{\}]
+{\color{incolor}In [{\color{incolor}12}]:} \PY{k+kn}{from} \PY{n+nn}{sklearn}\PY{n+nn}{.}\PY{n+nn}{neural\PYZus{}network} \PY{k}{import} \PY{n}{MLPClassifier}
+         \PY{c+c1}{\PYZsh{} store models for later use}
+         \PY{n}{DNN\PYZus{}scikit} \PY{o}{=} \PY{n}{np}\PY{o}{.}\PY{n}{zeros}\PY{p}{(}\PY{p}{(}\PY{n+nb}{len}\PY{p}{(}\PY{n}{eta\PYZus{}vals}\PY{p}{)}\PY{p}{,} \PY{n+nb}{len}\PY{p}{(}\PY{n}{lmbd\PYZus{}vals}\PY{p}{)}\PY{p}{)}\PY{p}{,} \PY{n}{dtype}\PY{o}{=}\PY{n+nb}{object}\PY{p}{)}
+         
+         \PY{k}{for} \PY{n}{i}\PY{p}{,} \PY{n}{eta} \PY{o+ow}{in} \PY{n+nb}{enumerate}\PY{p}{(}\PY{n}{eta\PYZus{}vals}\PY{p}{)}\PY{p}{:}
+             \PY{k}{for} \PY{n}{j}\PY{p}{,} \PY{n}{lmbd} \PY{o+ow}{in} \PY{n+nb}{enumerate}\PY{p}{(}\PY{n}{lmbd\PYZus{}vals}\PY{p}{)}\PY{p}{:}
+                 \PY{n}{dnn} \PY{o}{=} \PY{n}{MLPClassifier}\PY{p}{(}\PY{n}{hidden\PYZus{}layer\PYZus{}sizes}\PY{o}{=}\PY{p}{(}\PY{n}{n\PYZus{}hidden\PYZus{}neurons}\PY{p}{)}\PY{p}{,} \PY{n}{activation}\PY{o}{=}\PY{l+s+s1}{\PYZsq{}}\PY{l+s+s1}{logistic}\PY{l+s+s1}{\PYZsq{}}\PY{p}{,}
+                                     \PY{n}{alpha}\PY{o}{=}\PY{n}{lmbd}\PY{p}{,} \PY{n}{learning\PYZus{}rate\PYZus{}init}\PY{o}{=}\PY{n}{eta}\PY{p}{,} \PY{n}{max\PYZus{}iter}\PY{o}{=}\PY{n}{epochs}\PY{p}{)}
+                 \PY{n}{dnn}\PY{o}{.}\PY{n}{fit}\PY{p}{(}\PY{n}{X\PYZus{}train}\PY{p}{,} \PY{n}{Y\PYZus{}train}\PY{p}{)}
+                 
+                 \PY{n}{DNN\PYZus{}scikit}\PY{p}{[}\PY{n}{i}\PY{p}{]}\PY{p}{[}\PY{n}{j}\PY{p}{]} \PY{o}{=} \PY{n}{dnn}
+                 
+                 \PY{n+nb}{print}\PY{p}{(}\PY{l+s+s2}{\PYZdq{}}\PY{l+s+s2}{Learning rate  = }\PY{l+s+s2}{\PYZdq{}}\PY{p}{,} \PY{n}{eta}\PY{p}{)}
+                 \PY{n+nb}{print}\PY{p}{(}\PY{l+s+s2}{\PYZdq{}}\PY{l+s+s2}{Lambda = }\PY{l+s+s2}{\PYZdq{}}\PY{p}{,} \PY{n}{lmbd}\PY{p}{)}
+                 \PY{n+nb}{print}\PY{p}{(}\PY{l+s+s2}{\PYZdq{}}\PY{l+s+s2}{Accuracy score on test set: }\PY{l+s+s2}{\PYZdq{}}\PY{p}{,} \PY{n}{dnn}\PY{o}{.}\PY{n}{score}\PY{p}{(}\PY{n}{X\PYZus{}test}\PY{p}{,} \PY{n}{Y\PYZus{}test}\PY{p}{)}\PY{p}{)}
+                 \PY{n+nb}{print}\PY{p}{(}\PY{p}{)}
+\end{Verbatim}
+
+
+    \begin{Verbatim}[commandchars=\\\{\}]
+/usr/local/lib/python3.7/site-packages/sklearn/neural\_network/multilayer\_perceptron.py:564: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn't converged yet.
+  \% self.max\_iter, ConvergenceWarning)
+
+    \end{Verbatim}
+
+    \begin{Verbatim}[commandchars=\\\{\}]
+Learning rate  =  1e-05
+Lambda =  1e-05
+Accuracy score on test set:  0.20833333333333334
+
+Learning rate  =  1e-05
+Lambda =  0.0001
+Accuracy score on test set:  0.16944444444444445
+
+Learning rate  =  1e-05
+Lambda =  0.001
+Accuracy score on test set:  0.18055555555555555
+
+Learning rate  =  1e-05
+Lambda =  0.01
+Accuracy score on test set:  0.17222222222222222
+
+Learning rate  =  1e-05
+Lambda =  0.1
+Accuracy score on test set:  0.23333333333333334
+
+Learning rate  =  1e-05
+Lambda =  1.0
+Accuracy score on test set:  0.2916666666666667
+
+Learning rate  =  1e-05
+Lambda =  10.0
+Accuracy score on test set:  0.1527777777777778
+
+Learning rate  =  0.0001
+Lambda =  1e-05
+Accuracy score on test set:  0.8666666666666667
+
+Learning rate  =  0.0001
+Lambda =  0.0001
+Accuracy score on test set:  0.8416666666666667
+
+Learning rate  =  0.0001
+Lambda =  0.001
+Accuracy score on test set:  0.9277777777777778
+
+Learning rate  =  0.0001
+Lambda =  0.01
+Accuracy score on test set:  0.8861111111111111
+
+Learning rate  =  0.0001
+Lambda =  0.1
+Accuracy score on test set:  0.8805555555555555
+
+Learning rate  =  0.0001
+Lambda =  1.0
+Accuracy score on test set:  0.875
+
+Learning rate  =  0.0001
+Lambda =  10.0
+Accuracy score on test set:  0.875
+
+Learning rate  =  0.001
+Lambda =  1e-05
+Accuracy score on test set:  0.9833333333333333
+
+Learning rate  =  0.001
+Lambda =  0.0001
+Accuracy score on test set:  0.9833333333333333
+
+Learning rate  =  0.001
+Lambda =  0.001
+Accuracy score on test set:  0.9805555555555555
+
+Learning rate  =  0.001
+Lambda =  0.01
+Accuracy score on test set:  0.9861111111111112
+
+Learning rate  =  0.001
+Lambda =  0.1
+Accuracy score on test set:  0.9805555555555555
+
+Learning rate  =  0.001
+Lambda =  1.0
+Accuracy score on test set:  0.9805555555555555
+
+Learning rate  =  0.001
+Lambda =  10.0
+Accuracy score on test set:  0.9527777777777777
+
+Learning rate  =  0.01
+Lambda =  1e-05
+Accuracy score on test set:  0.9861111111111112
+
+Learning rate  =  0.01
+Lambda =  0.0001
+Accuracy score on test set:  0.9916666666666667
+
+Learning rate  =  0.01
+Lambda =  0.001
+Accuracy score on test set:  0.9861111111111112
+
+Learning rate  =  0.01
+Lambda =  0.01
+Accuracy score on test set:  0.9833333333333333
+
+Learning rate  =  0.01
+Lambda =  0.1
+Accuracy score on test set:  0.9944444444444445
+
+Learning rate  =  0.01
+Lambda =  1.0
+Accuracy score on test set:  0.975
+
+Learning rate  =  0.01
+Lambda =  10.0
+Accuracy score on test set:  0.9416666666666667
+
+Learning rate  =  0.1
+Lambda =  1e-05
+Accuracy score on test set:  0.9416666666666667
+
+Learning rate  =  0.1
+Lambda =  0.0001
+Accuracy score on test set:  0.75
+
+Learning rate  =  0.1
+Lambda =  0.001
+Accuracy score on test set:  0.8972222222222223
+
+Learning rate  =  0.1
+Lambda =  0.01
+Accuracy score on test set:  0.8944444444444445
+
+Learning rate  =  0.1
+Lambda =  0.1
+Accuracy score on test set:  0.9277777777777778
+
+Learning rate  =  0.1
+Lambda =  1.0
+Accuracy score on test set:  0.8972222222222223
+
+Learning rate  =  0.1
+Lambda =  10.0
+Accuracy score on test set:  0.7388888888888889
+
+Learning rate  =  1.0
+Lambda =  1e-05
+Accuracy score on test set:  0.08611111111111111
+
+Learning rate  =  1.0
+Lambda =  0.0001
+Accuracy score on test set:  0.10555555555555556
+
+Learning rate  =  1.0
+Lambda =  0.001
+Accuracy score on test set:  0.11388888888888889
+
+Learning rate  =  1.0
+Lambda =  0.01
+Accuracy score on test set:  0.1527777777777778
+
+Learning rate  =  1.0
+Lambda =  0.1
+Accuracy score on test set:  0.15
+
+Learning rate  =  1.0
+Lambda =  1.0
+Accuracy score on test set:  0.08888888888888889
+
+Learning rate  =  1.0
+Lambda =  10.0
+Accuracy score on test set:  0.15
+
+Learning rate  =  10.0
+Lambda =  1e-05
+Accuracy score on test set:  0.1
+
+Learning rate  =  10.0
+Lambda =  0.0001
+Accuracy score on test set:  0.11944444444444445
+
+Learning rate  =  10.0
+Lambda =  0.001
+Accuracy score on test set:  0.18055555555555555
+
+Learning rate  =  10.0
+Lambda =  0.01
+Accuracy score on test set:  0.10555555555555556
+
+Learning rate  =  10.0
+Lambda =  0.1
+Accuracy score on test set:  0.08333333333333333
+
+Learning rate  =  10.0
+Lambda =  1.0
+Accuracy score on test set:  0.09166666666666666
+
+Learning rate  =  10.0
+Lambda =  10.0
+Accuracy score on test set:  0.08055555555555556
+
+
+    \end{Verbatim}
+
+    \hypertarget{visualization}{%
+\subsection{Visualization}\label{visualization}}
+
+    \begin{Verbatim}[commandchars=\\\{\}]
+{\color{incolor}In [{\color{incolor}13}]:} \PY{c+c1}{\PYZsh{} optional}
+         \PY{c+c1}{\PYZsh{} visual representation of grid search}
+         \PY{c+c1}{\PYZsh{} uses seaborn heatmap, could probably do this in matplotlib}
+         \PY{k+kn}{import} \PY{n+nn}{seaborn} \PY{k}{as} \PY{n+nn}{sns}
+         
+         \PY{n}{sns}\PY{o}{.}\PY{n}{set}\PY{p}{(}\PY{p}{)}
+         
+         \PY{n}{train\PYZus{}accuracy} \PY{o}{=} \PY{n}{np}\PY{o}{.}\PY{n}{zeros}\PY{p}{(}\PY{p}{(}\PY{n+nb}{len}\PY{p}{(}\PY{n}{eta\PYZus{}vals}\PY{p}{)}\PY{p}{,} \PY{n+nb}{len}\PY{p}{(}\PY{n}{lmbd\PYZus{}vals}\PY{p}{)}\PY{p}{)}\PY{p}{)}
+         \PY{n}{test\PYZus{}accuracy} \PY{o}{=} \PY{n}{np}\PY{o}{.}\PY{n}{zeros}\PY{p}{(}\PY{p}{(}\PY{n+nb}{len}\PY{p}{(}\PY{n}{eta\PYZus{}vals}\PY{p}{)}\PY{p}{,} \PY{n+nb}{len}\PY{p}{(}\PY{n}{lmbd\PYZus{}vals}\PY{p}{)}\PY{p}{)}\PY{p}{)}
+         
+         \PY{k}{for} \PY{n}{i} \PY{o+ow}{in} \PY{n+nb}{range}\PY{p}{(}\PY{n+nb}{len}\PY{p}{(}\PY{n}{eta\PYZus{}vals}\PY{p}{)}\PY{p}{)}\PY{p}{:}
+             \PY{k}{for} \PY{n}{j} \PY{o+ow}{in} \PY{n+nb}{range}\PY{p}{(}\PY{n+nb}{len}\PY{p}{(}\PY{n}{lmbd\PYZus{}vals}\PY{p}{)}\PY{p}{)}\PY{p}{:}
+                 \PY{n}{dnn} \PY{o}{=} \PY{n}{DNN\PYZus{}scikit}\PY{p}{[}\PY{n}{i}\PY{p}{]}\PY{p}{[}\PY{n}{j}\PY{p}{]}
+                 
+                 \PY{n}{train\PYZus{}pred} \PY{o}{=} \PY{n}{dnn}\PY{o}{.}\PY{n}{predict}\PY{p}{(}\PY{n}{X\PYZus{}train}\PY{p}{)} 
+                 \PY{n}{test\PYZus{}pred} \PY{o}{=} \PY{n}{dnn}\PY{o}{.}\PY{n}{predict}\PY{p}{(}\PY{n}{X\PYZus{}test}\PY{p}{)}
+         
+                 \PY{n}{train\PYZus{}accuracy}\PY{p}{[}\PY{n}{i}\PY{p}{]}\PY{p}{[}\PY{n}{j}\PY{p}{]} \PY{o}{=} \PY{n}{accuracy\PYZus{}score}\PY{p}{(}\PY{n}{Y\PYZus{}train}\PY{p}{,} \PY{n}{train\PYZus{}pred}\PY{p}{)}
+                 \PY{n}{test\PYZus{}accuracy}\PY{p}{[}\PY{n}{i}\PY{p}{]}\PY{p}{[}\PY{n}{j}\PY{p}{]} \PY{o}{=} \PY{n}{accuracy\PYZus{}score}\PY{p}{(}\PY{n}{Y\PYZus{}test}\PY{p}{,} \PY{n}{test\PYZus{}pred}\PY{p}{)}
+         
+                 
+         \PY{n}{fig}\PY{p}{,} \PY{n}{ax} \PY{o}{=} \PY{n}{plt}\PY{o}{.}\PY{n}{subplots}\PY{p}{(}\PY{n}{figsize} \PY{o}{=} \PY{p}{(}\PY{l+m+mi}{10}\PY{p}{,} \PY{l+m+mi}{10}\PY{p}{)}\PY{p}{)}
+         \PY{n}{sns}\PY{o}{.}\PY{n}{heatmap}\PY{p}{(}\PY{n}{train\PYZus{}accuracy}\PY{p}{,} \PY{n}{annot}\PY{o}{=}\PY{k+kc}{True}\PY{p}{,} \PY{n}{ax}\PY{o}{=}\PY{n}{ax}\PY{p}{,} \PY{n}{cmap}\PY{o}{=}\PY{l+s+s2}{\PYZdq{}}\PY{l+s+s2}{viridis}\PY{l+s+s2}{\PYZdq{}}\PY{p}{)}
+         \PY{n}{ax}\PY{o}{.}\PY{n}{set\PYZus{}title}\PY{p}{(}\PY{l+s+s2}{\PYZdq{}}\PY{l+s+s2}{Training Accuracy}\PY{l+s+s2}{\PYZdq{}}\PY{p}{)}
+         \PY{n}{ax}\PY{o}{.}\PY{n}{set\PYZus{}ylabel}\PY{p}{(}\PY{l+s+s2}{\PYZdq{}}\PY{l+s+s2}{\PYZdl{}}\PY{l+s+s2}{\PYZbs{}}\PY{l+s+s2}{eta\PYZdl{}}\PY{l+s+s2}{\PYZdq{}}\PY{p}{)}
+         \PY{n}{ax}\PY{o}{.}\PY{n}{set\PYZus{}xlabel}\PY{p}{(}\PY{l+s+s2}{\PYZdq{}}\PY{l+s+s2}{\PYZdl{}}\PY{l+s+s2}{\PYZbs{}}\PY{l+s+s2}{lambda\PYZdl{}}\PY{l+s+s2}{\PYZdq{}}\PY{p}{)}
+         \PY{n}{plt}\PY{o}{.}\PY{n}{show}\PY{p}{(}\PY{p}{)}
+         
+         \PY{n}{fig}\PY{p}{,} \PY{n}{ax} \PY{o}{=} \PY{n}{plt}\PY{o}{.}\PY{n}{subplots}\PY{p}{(}\PY{n}{figsize} \PY{o}{=} \PY{p}{(}\PY{l+m+mi}{10}\PY{p}{,} \PY{l+m+mi}{10}\PY{p}{)}\PY{p}{)}
+         \PY{n}{sns}\PY{o}{.}\PY{n}{heatmap}\PY{p}{(}\PY{n}{test\PYZus{}accuracy}\PY{p}{,} \PY{n}{annot}\PY{o}{=}\PY{k+kc}{True}\PY{p}{,} \PY{n}{ax}\PY{o}{=}\PY{n}{ax}\PY{p}{,} \PY{n}{cmap}\PY{o}{=}\PY{l+s+s2}{\PYZdq{}}\PY{l+s+s2}{viridis}\PY{l+s+s2}{\PYZdq{}}\PY{p}{)}
+         \PY{n}{ax}\PY{o}{.}\PY{n}{set\PYZus{}title}\PY{p}{(}\PY{l+s+s2}{\PYZdq{}}\PY{l+s+s2}{Test Accuracy}\PY{l+s+s2}{\PYZdq{}}\PY{p}{)}
+         \PY{n}{ax}\PY{o}{.}\PY{n}{set\PYZus{}ylabel}\PY{p}{(}\PY{l+s+s2}{\PYZdq{}}\PY{l+s+s2}{\PYZdl{}}\PY{l+s+s2}{\PYZbs{}}\PY{l+s+s2}{eta\PYZdl{}}\PY{l+s+s2}{\PYZdq{}}\PY{p}{)}
+         \PY{n}{ax}\PY{o}{.}\PY{n}{set\PYZus{}xlabel}\PY{p}{(}\PY{l+s+s2}{\PYZdq{}}\PY{l+s+s2}{\PYZdl{}}\PY{l+s+s2}{\PYZbs{}}\PY{l+s+s2}{lambda\PYZdl{}}\PY{l+s+s2}{\PYZdq{}}\PY{p}{)}
+         \PY{n}{plt}\PY{o}{.}\PY{n}{show}\PY{p}{(}\PY{p}{)}
+\end{Verbatim}
+
+
+    \begin{center}
+    \adjustimage{max size={0.9\linewidth}{0.9\paperheight}}{output_121_0.png}
+    \end{center}
+    { \hspace*{\fill} \\}
+    
+    \begin{center}
+    \adjustimage{max size={0.9\linewidth}{0.9\paperheight}}{output_121_1.png}
+    \end{center}
+    { \hspace*{\fill} \\}
+    
+    \hypertarget{building-neural-networks-in-tensorflow-and-keras}{%
+\subsection{Building neural networks in Tensorflow and
+Keras}\label{building-neural-networks-in-tensorflow-and-keras}}
+
+Now we want to build on the experience gained from our neural network
+implementation in NumPy and scikit-learn and use it to construct a
+neural network in Tensorflow. Once we have constructed a neural network
+in NumPy and Tensorflow, building one in Keras is really quite trivial,
+though the performance may suffer.
+
+In our previous example we used only one hidden layer, and in this we
+will use two. From this it should be quite clear how to build one using
+an arbitrary number of hidden layers, using data structures such as
+Python lists or NumPy arrays.
+
+\hypertarget{tensorflow}{%
+\subsection{Tensorflow}\label{tensorflow}}
+
+Tensorflow is an open source library machine learning library developed
+by the Google Brain team for internal use. It was released under the
+Apache 2.0 open source license in November 9, 2015.
+
+Tensorflow is a computational framework that allows you to construct
+machine learning models at different levels of abstraction, from
+high-level, object-oriented APIs like Keras, down to the C++ kernels
+that Tensorflow is built upon. The higher levels of abstraction are
+simpler to use, but less flexible, and our choice of implementation
+should reflect the problems we are trying to solve.
+
+\href{https://www.tensorflow.org/guide/graphs}{Tensorflow uses}
+so-called graphs to represent your computation in terms of the
+dependencies between individual operations, such that you first build a
+Tensorflow \emph{graph} to represent your model, and then create a
+Tensorflow \emph{session} to run the graph.
+
+In this guide we will analyze the same data as we did in our NumPy and
+scikit-learn tutorial, gathered from the MNIST database of images. We
+will give an introduction to the lower level Python Application Program
+Interfaces (APIs), and see how we use them to build our graph. Then we
+will build (effectively) the same graph in Keras, to see just how simple
+solving a machine learning problem can be.
+
+To install tensorflow on Unix/Linux systems, use pip as
+
+    \begin{Verbatim}[commandchars=\\\{\}]
+{\color{incolor}In [{\color{incolor}13}]:} \PY{n}{pip3} \PY{n}{install} \PY{n}{tensorflow}
+\end{Verbatim}
+
+
+    and/or if you use \textbf{anaconda}, just write (or install from the
+graphical user interface)
+
+    \begin{Verbatim}[commandchars=\\\{\}]
+{\color{incolor}In [{\color{incolor}14}]:} \PY{n}{conda} \PY{n}{install} \PY{n}{tensorflow}
+\end{Verbatim}
+
+
+    \hypertarget{collect-and-pre-process-data}{%
+\subsection{Collect and pre-process
+data}\label{collect-and-pre-process-data}}
+
+    \begin{Verbatim}[commandchars=\\\{\}]
+{\color{incolor}In [{\color{incolor}14}]:} \PY{c+c1}{\PYZsh{} import necessary packages}
+         \PY{k+kn}{import} \PY{n+nn}{numpy} \PY{k}{as} \PY{n+nn}{np}
+         \PY{k+kn}{import} \PY{n+nn}{matplotlib}\PY{n+nn}{.}\PY{n+nn}{pyplot} \PY{k}{as} \PY{n+nn}{plt}
+         \PY{k+kn}{from} \PY{n+nn}{sklearn} \PY{k}{import} \PY{n}{datasets}
+         
+         
+         \PY{c+c1}{\PYZsh{} ensure the same random numbers appear every time}
+         \PY{n}{np}\PY{o}{.}\PY{n}{random}\PY{o}{.}\PY{n}{seed}\PY{p}{(}\PY{l+m+mi}{0}\PY{p}{)}
+         
+         \PY{c+c1}{\PYZsh{} display images in notebook}
+         \PY{o}{\PYZpc{}}\PY{k}{matplotlib} inline
+         \PY{n}{plt}\PY{o}{.}\PY{n}{rcParams}\PY{p}{[}\PY{l+s+s1}{\PYZsq{}}\PY{l+s+s1}{figure.figsize}\PY{l+s+s1}{\PYZsq{}}\PY{p}{]} \PY{o}{=} \PY{p}{(}\PY{l+m+mi}{12}\PY{p}{,}\PY{l+m+mi}{12}\PY{p}{)}
+         
+         
+         \PY{c+c1}{\PYZsh{} download MNIST dataset}
+         \PY{n}{digits} \PY{o}{=} \PY{n}{datasets}\PY{o}{.}\PY{n}{load\PYZus{}digits}\PY{p}{(}\PY{p}{)}
+         
+         \PY{c+c1}{\PYZsh{} define inputs and labels}
+         \PY{n}{inputs} \PY{o}{=} \PY{n}{digits}\PY{o}{.}\PY{n}{images}
+         \PY{n}{labels} \PY{o}{=} \PY{n}{digits}\PY{o}{.}\PY{n}{target}
+         
+         \PY{n+nb}{print}\PY{p}{(}\PY{l+s+s2}{\PYZdq{}}\PY{l+s+s2}{inputs = (n\PYZus{}inputs, pixel\PYZus{}width, pixel\PYZus{}height) = }\PY{l+s+s2}{\PYZdq{}} \PY{o}{+} \PY{n+nb}{str}\PY{p}{(}\PY{n}{inputs}\PY{o}{.}\PY{n}{shape}\PY{p}{)}\PY{p}{)}
+         \PY{n+nb}{print}\PY{p}{(}\PY{l+s+s2}{\PYZdq{}}\PY{l+s+s2}{labels = (n\PYZus{}inputs) = }\PY{l+s+s2}{\PYZdq{}} \PY{o}{+} \PY{n+nb}{str}\PY{p}{(}\PY{n}{labels}\PY{o}{.}\PY{n}{shape}\PY{p}{)}\PY{p}{)}
+         
+         
+         \PY{c+c1}{\PYZsh{} flatten the image}
+         \PY{c+c1}{\PYZsh{} the value \PYZhy{}1 means dimension is inferred from the remaining dimensions: 8x8 = 64}
+         \PY{n}{n\PYZus{}inputs} \PY{o}{=} \PY{n+nb}{len}\PY{p}{(}\PY{n}{inputs}\PY{p}{)}
+         \PY{n}{inputs} \PY{o}{=} \PY{n}{inputs}\PY{o}{.}\PY{n}{reshape}\PY{p}{(}\PY{n}{n\PYZus{}inputs}\PY{p}{,} \PY{o}{\PYZhy{}}\PY{l+m+mi}{1}\PY{p}{)}
+         \PY{n+nb}{print}\PY{p}{(}\PY{l+s+s2}{\PYZdq{}}\PY{l+s+s2}{X = (n\PYZus{}inputs, n\PYZus{}features) = }\PY{l+s+s2}{\PYZdq{}} \PY{o}{+} \PY{n+nb}{str}\PY{p}{(}\PY{n}{inputs}\PY{o}{.}\PY{n}{shape}\PY{p}{)}\PY{p}{)}
+         
+         
+         \PY{c+c1}{\PYZsh{} choose some random images to display}
+         \PY{n}{indices} \PY{o}{=} \PY{n}{np}\PY{o}{.}\PY{n}{arange}\PY{p}{(}\PY{n}{n\PYZus{}inputs}\PY{p}{)}
+         \PY{n}{random\PYZus{}indices} \PY{o}{=} \PY{n}{np}\PY{o}{.}\PY{n}{random}\PY{o}{.}\PY{n}{choice}\PY{p}{(}\PY{n}{indices}\PY{p}{,} \PY{n}{size}\PY{o}{=}\PY{l+m+mi}{5}\PY{p}{)}
+         
+         \PY{k}{for} \PY{n}{i}\PY{p}{,} \PY{n}{image} \PY{o+ow}{in} \PY{n+nb}{enumerate}\PY{p}{(}\PY{n}{digits}\PY{o}{.}\PY{n}{images}\PY{p}{[}\PY{n}{random\PYZus{}indices}\PY{p}{]}\PY{p}{)}\PY{p}{:}
+             \PY{n}{plt}\PY{o}{.}\PY{n}{subplot}\PY{p}{(}\PY{l+m+mi}{1}\PY{p}{,} \PY{l+m+mi}{5}\PY{p}{,} \PY{n}{i}\PY{o}{+}\PY{l+m+mi}{1}\PY{p}{)}
+             \PY{n}{plt}\PY{o}{.}\PY{n}{axis}\PY{p}{(}\PY{l+s+s1}{\PYZsq{}}\PY{l+s+s1}{off}\PY{l+s+s1}{\PYZsq{}}\PY{p}{)}
+             \PY{n}{plt}\PY{o}{.}\PY{n}{imshow}\PY{p}{(}\PY{n}{image}\PY{p}{,} \PY{n}{cmap}\PY{o}{=}\PY{n}{plt}\PY{o}{.}\PY{n}{cm}\PY{o}{.}\PY{n}{gray\PYZus{}r}\PY{p}{,} \PY{n}{interpolation}\PY{o}{=}\PY{l+s+s1}{\PYZsq{}}\PY{l+s+s1}{nearest}\PY{l+s+s1}{\PYZsq{}}\PY{p}{)}
+             \PY{n}{plt}\PY{o}{.}\PY{n}{title}\PY{p}{(}\PY{l+s+s2}{\PYZdq{}}\PY{l+s+s2}{Label: }\PY{l+s+si}{\PYZpc{}d}\PY{l+s+s2}{\PYZdq{}} \PY{o}{\PYZpc{}} \PY{n}{digits}\PY{o}{.}\PY{n}{target}\PY{p}{[}\PY{n}{random\PYZus{}indices}\PY{p}{[}\PY{n}{i}\PY{p}{]}\PY{p}{]}\PY{p}{)}
+         \PY{n}{plt}\PY{o}{.}\PY{n}{show}\PY{p}{(}\PY{p}{)}
+\end{Verbatim}
+
+
+    \begin{Verbatim}[commandchars=\\\{\}]
+inputs = (n\_inputs, pixel\_width, pixel\_height) = (1797, 8, 8)
+labels = (n\_inputs) = (1797,)
+X = (n\_inputs, n\_features) = (1797, 64)
+
+    \end{Verbatim}
+
+    \begin{center}
+    \adjustimage{max size={0.9\linewidth}{0.9\paperheight}}{output_127_1.png}
+    \end{center}
+    { \hspace*{\fill} \\}
+    
+    \begin{Verbatim}[commandchars=\\\{\}]
+{\color{incolor}In [{\color{incolor}15}]:} \PY{k+kn}{from} \PY{n+nn}{keras}\PY{n+nn}{.}\PY{n+nn}{utils} \PY{k}{import} \PY{n}{to\PYZus{}categorical}
+         \PY{k+kn}{from} \PY{n+nn}{sklearn}\PY{n+nn}{.}\PY{n+nn}{model\PYZus{}selection} \PY{k}{import} \PY{n}{train\PYZus{}test\PYZus{}split}
+         
+         \PY{c+c1}{\PYZsh{} one\PYZhy{}hot representation of labels}
+         \PY{n}{labels} \PY{o}{=} \PY{n}{to\PYZus{}categorical}\PY{p}{(}\PY{n}{labels}\PY{p}{)}
+         
+         \PY{c+c1}{\PYZsh{} split into train and test data}
+         \PY{n}{train\PYZus{}size} \PY{o}{=} \PY{l+m+mf}{0.8}
+         \PY{n}{test\PYZus{}size} \PY{o}{=} \PY{l+m+mi}{1} \PY{o}{\PYZhy{}} \PY{n}{train\PYZus{}size}
+         \PY{n}{X\PYZus{}train}\PY{p}{,} \PY{n}{X\PYZus{}test}\PY{p}{,} \PY{n}{Y\PYZus{}train}\PY{p}{,} \PY{n}{Y\PYZus{}test} \PY{o}{=} \PY{n}{train\PYZus{}test\PYZus{}split}\PY{p}{(}\PY{n}{inputs}\PY{p}{,} \PY{n}{labels}\PY{p}{,} \PY{n}{train\PYZus{}size}\PY{o}{=}\PY{n}{train\PYZus{}size}\PY{p}{,}
+                                                             \PY{n}{test\PYZus{}size}\PY{o}{=}\PY{n}{test\PYZus{}size}\PY{p}{)}
+\end{Verbatim}
+
+
+    \begin{Verbatim}[commandchars=\\\{\}]
+Using TensorFlow backend.
+
+    \end{Verbatim}
+
+    \begin{Verbatim}[commandchars=\\\{\}]
+
+        ---------------------------------------------------------------------------
+
+        ModuleNotFoundError                       Traceback (most recent call last)
+
+         in ()
+    ----> 1 from keras.utils import to\_categorical
+          2 from sklearn.model\_selection import train\_test\_split
+          3 
+          4 \# one-hot representation of labels
+          5 labels = to\_categorical(labels)
+
+
+        /usr/local/lib/python3.7/site-packages/keras/\_\_init\_\_.py in ()
+          1 from \_\_future\_\_ import absolute\_import
+          2 
+    ----> 3 from . import utils
+          4 from . import activations
+          5 from . import applications
+
+
+        /usr/local/lib/python3.7/site-packages/keras/utils/\_\_init\_\_.py in ()
+          4 from . import data\_utils
+          5 from . import io\_utils
+    ----> 6 from . import conv\_utils
+          7 
+          8 \# Globally-importable utils.
+
+
+        /usr/local/lib/python3.7/site-packages/keras/utils/conv\_utils.py in ()
+          7 from six.moves import range
+          8 import numpy as np
+    ----> 9 from .. import backend as K
+         10 
+         11 
+
+
+        /usr/local/lib/python3.7/site-packages/keras/backend/\_\_init\_\_.py in ()
+         87 elif \_BACKEND == 'tensorflow':
+         88     sys.stderr.write('Using TensorFlow backend.\textbackslash{}n')
+    ---> 89     from .tensorflow\_backend import *
+         90 else:
+         91     \# Try and load external backend.
+
+
+        /usr/local/lib/python3.7/site-packages/keras/backend/tensorflow\_backend.py in ()
+          3 from \_\_future\_\_ import print\_function
+          4 
+    ----> 5 import tensorflow as tf
+          6 from tensorflow.python.framework import ops as tf\_ops
+          7 from tensorflow.python.training import moving\_averages
+
+
+        ModuleNotFoundError: No module named 'tensorflow'
+
+    \end{Verbatim}
+
+    \hypertarget{using-tensorflow-backend}{%
+\subsection{Using TensorFlow backend}\label{using-tensorflow-backend}}
+
+\begin{enumerate}
+\def\labelenumi{\arabic{enumi}.}
+\item
+  Define model and architecture
+\item
+  Choose cost function and optimizer
+\end{enumerate}
+
+    \begin{Verbatim}[commandchars=\\\{\}]
+{\color{incolor}In [{\color{incolor}17}]:} \PY{k+kn}{import} \PY{n+nn}{tensorflow} \PY{k}{as} \PY{n+nn}{tf}
+         
+         \PY{k}{class} \PY{n+nc}{NeuralNetworkTensorflow}\PY{p}{:}
+             \PY{k}{def} \PY{n+nf}{\PYZus{}\PYZus{}init\PYZus{}\PYZus{}}\PY{p}{(}
+                 \PY{n+nb+bp}{self}\PY{p}{,}
+                 \PY{n}{X\PYZus{}train}\PY{p}{,}
+                 \PY{n}{Y\PYZus{}train}\PY{p}{,}
+                 \PY{n}{X\PYZus{}test}\PY{p}{,}
+                 \PY{n}{Y\PYZus{}test}\PY{p}{,}
+                 \PY{n}{n\PYZus{}neurons\PYZus{}layer1}\PY{o}{=}\PY{l+m+mi}{100}\PY{p}{,}
+                 \PY{n}{n\PYZus{}neurons\PYZus{}layer2}\PY{o}{=}\PY{l+m+mi}{50}\PY{p}{,}
+                 \PY{n}{n\PYZus{}categories}\PY{o}{=}\PY{l+m+mi}{2}\PY{p}{,}
+                 \PY{n}{epochs}\PY{o}{=}\PY{l+m+mi}{10}\PY{p}{,}
+                 \PY{n}{batch\PYZus{}size}\PY{o}{=}\PY{l+m+mi}{100}\PY{p}{,}
+                 \PY{n}{eta}\PY{o}{=}\PY{l+m+mf}{0.1}\PY{p}{,}
+                 \PY{n}{lmbd}\PY{o}{=}\PY{l+m+mf}{0.0}\PY{p}{,}
+             \PY{p}{)}\PY{p}{:}
+                 
+                 \PY{c+c1}{\PYZsh{} keep track of number of steps}
+                 \PY{n+nb+bp}{self}\PY{o}{.}\PY{n}{global\PYZus{}step} \PY{o}{=} \PY{n}{tf}\PY{o}{.}\PY{n}{Variable}\PY{p}{(}\PY{l+m+mi}{0}\PY{p}{,} \PY{n}{dtype}\PY{o}{=}\PY{n}{tf}\PY{o}{.}\PY{n}{int32}\PY{p}{,} \PY{n}{trainable}\PY{o}{=}\PY{k+kc}{False}\PY{p}{,} \PY{n}{name}\PY{o}{=}\PY{l+s+s1}{\PYZsq{}}\PY{l+s+s1}{global\PYZus{}step}\PY{l+s+s1}{\PYZsq{}}\PY{p}{)}
+                 
+                 \PY{n+nb+bp}{self}\PY{o}{.}\PY{n}{X\PYZus{}train} \PY{o}{=} \PY{n}{X\PYZus{}train}
+                 \PY{n+nb+bp}{self}\PY{o}{.}\PY{n}{Y\PYZus{}train} \PY{o}{=} \PY{n}{Y\PYZus{}train}
+                 \PY{n+nb+bp}{self}\PY{o}{.}\PY{n}{X\PYZus{}test} \PY{o}{=} \PY{n}{X\PYZus{}test}
+                 \PY{n+nb+bp}{self}\PY{o}{.}\PY{n}{Y\PYZus{}test} \PY{o}{=} \PY{n}{Y\PYZus{}test}
+                 
+                 \PY{n+nb+bp}{self}\PY{o}{.}\PY{n}{n\PYZus{}inputs} \PY{o}{=} \PY{n}{X\PYZus{}train}\PY{o}{.}\PY{n}{shape}\PY{p}{[}\PY{l+m+mi}{0}\PY{p}{]}
+                 \PY{n+nb+bp}{self}\PY{o}{.}\PY{n}{n\PYZus{}features} \PY{o}{=} \PY{n}{X\PYZus{}train}\PY{o}{.}\PY{n}{shape}\PY{p}{[}\PY{l+m+mi}{1}\PY{p}{]}
+                 \PY{n+nb+bp}{self}\PY{o}{.}\PY{n}{n\PYZus{}neurons\PYZus{}layer1} \PY{o}{=} \PY{n}{n\PYZus{}neurons\PYZus{}layer1}
+                 \PY{n+nb+bp}{self}\PY{o}{.}\PY{n}{n\PYZus{}neurons\PYZus{}layer2} \PY{o}{=} \PY{n}{n\PYZus{}neurons\PYZus{}layer2}
+                 \PY{n+nb+bp}{self}\PY{o}{.}\PY{n}{n\PYZus{}categories} \PY{o}{=} \PY{n}{n\PYZus{}categories}
+                 
+                 \PY{n+nb+bp}{self}\PY{o}{.}\PY{n}{epochs} \PY{o}{=} \PY{n}{epochs}
+                 \PY{n+nb+bp}{self}\PY{o}{.}\PY{n}{batch\PYZus{}size} \PY{o}{=} \PY{n}{batch\PYZus{}size}
+                 \PY{n+nb+bp}{self}\PY{o}{.}\PY{n}{iterations} \PY{o}{=} \PY{n+nb+bp}{self}\PY{o}{.}\PY{n}{n\PYZus{}inputs} \PY{o}{/}\PY{o}{/} \PY{n+nb+bp}{self}\PY{o}{.}\PY{n}{batch\PYZus{}size}
+                 \PY{n+nb+bp}{self}\PY{o}{.}\PY{n}{eta} \PY{o}{=} \PY{n}{eta}
+                 \PY{n+nb+bp}{self}\PY{o}{.}\PY{n}{lmbd} \PY{o}{=} \PY{n}{lmbd}
+                 
+                 \PY{c+c1}{\PYZsh{} build network piece by piece}
+                 \PY{c+c1}{\PYZsh{} name scopes (with) are used to enforce creation of new variables}
+                 \PY{c+c1}{\PYZsh{} https://www.tensorflow.org/guide/variables}
+                 \PY{n+nb+bp}{self}\PY{o}{.}\PY{n}{create\PYZus{}placeholders}\PY{p}{(}\PY{p}{)}
+                 \PY{n+nb+bp}{self}\PY{o}{.}\PY{n}{create\PYZus{}DNN}\PY{p}{(}\PY{p}{)}
+                 \PY{n+nb+bp}{self}\PY{o}{.}\PY{n}{create\PYZus{}loss}\PY{p}{(}\PY{p}{)}
+                 \PY{n+nb+bp}{self}\PY{o}{.}\PY{n}{create\PYZus{}optimiser}\PY{p}{(}\PY{p}{)}
+                 \PY{n+nb+bp}{self}\PY{o}{.}\PY{n}{create\PYZus{}accuracy}\PY{p}{(}\PY{p}{)}
+             
+             \PY{k}{def} \PY{n+nf}{create\PYZus{}placeholders}\PY{p}{(}\PY{n+nb+bp}{self}\PY{p}{)}\PY{p}{:}
+                 \PY{c+c1}{\PYZsh{} placeholders are fine here, but \PYZdq{}Datasets\PYZdq{} are the preferred method}
+                 \PY{c+c1}{\PYZsh{} of streaming data into a model}
+                 \PY{k}{with} \PY{n}{tf}\PY{o}{.}\PY{n}{name\PYZus{}scope}\PY{p}{(}\PY{l+s+s1}{\PYZsq{}}\PY{l+s+s1}{data}\PY{l+s+s1}{\PYZsq{}}\PY{p}{)}\PY{p}{:}
+                     \PY{n+nb+bp}{self}\PY{o}{.}\PY{n}{X} \PY{o}{=} \PY{n}{tf}\PY{o}{.}\PY{n}{placeholder}\PY{p}{(}\PY{n}{tf}\PY{o}{.}\PY{n}{float32}\PY{p}{,} \PY{n}{shape}\PY{o}{=}\PY{p}{(}\PY{k+kc}{None}\PY{p}{,} \PY{n+nb+bp}{self}\PY{o}{.}\PY{n}{n\PYZus{}features}\PY{p}{)}\PY{p}{,} \PY{n}{name}\PY{o}{=}\PY{l+s+s1}{\PYZsq{}}\PY{l+s+s1}{X\PYZus{}data}\PY{l+s+s1}{\PYZsq{}}\PY{p}{)}
+                     \PY{n+nb+bp}{self}\PY{o}{.}\PY{n}{Y} \PY{o}{=} \PY{n}{tf}\PY{o}{.}\PY{n}{placeholder}\PY{p}{(}\PY{n}{tf}\PY{o}{.}\PY{n}{float32}\PY{p}{,} \PY{n}{shape}\PY{o}{=}\PY{p}{(}\PY{k+kc}{None}\PY{p}{,} \PY{n+nb+bp}{self}\PY{o}{.}\PY{n}{n\PYZus{}categories}\PY{p}{)}\PY{p}{,} \PY{n}{name}\PY{o}{=}\PY{l+s+s1}{\PYZsq{}}\PY{l+s+s1}{Y\PYZus{}data}\PY{l+s+s1}{\PYZsq{}}\PY{p}{)}
+             
+             \PY{k}{def} \PY{n+nf}{create\PYZus{}DNN}\PY{p}{(}\PY{n+nb+bp}{self}\PY{p}{)}\PY{p}{:}
+                 \PY{k}{with} \PY{n}{tf}\PY{o}{.}\PY{n}{name\PYZus{}scope}\PY{p}{(}\PY{l+s+s1}{\PYZsq{}}\PY{l+s+s1}{DNN}\PY{l+s+s1}{\PYZsq{}}\PY{p}{)}\PY{p}{:}
+                     \PY{c+c1}{\PYZsh{} the weights are stored to calculate regularization loss later}
+                     
+                     \PY{c+c1}{\PYZsh{} Fully connected layer 1}
+                     \PY{n+nb+bp}{self}\PY{o}{.}\PY{n}{W\PYZus{}fc1} \PY{o}{=} \PY{n+nb+bp}{self}\PY{o}{.}\PY{n}{weight\PYZus{}variable}\PY{p}{(}\PY{p}{[}\PY{n+nb+bp}{self}\PY{o}{.}\PY{n}{n\PYZus{}features}\PY{p}{,} \PY{n+nb+bp}{self}\PY{o}{.}\PY{n}{n\PYZus{}neurons\PYZus{}layer1}\PY{p}{]}\PY{p}{,} \PY{n}{name}\PY{o}{=}\PY{l+s+s1}{\PYZsq{}}\PY{l+s+s1}{fc1}\PY{l+s+s1}{\PYZsq{}}\PY{p}{,} \PY{n}{dtype}\PY{o}{=}\PY{n}{tf}\PY{o}{.}\PY{n}{float32}\PY{p}{)}
+                     \PY{n}{b\PYZus{}fc1} \PY{o}{=} \PY{n+nb+bp}{self}\PY{o}{.}\PY{n}{bias\PYZus{}variable}\PY{p}{(}\PY{p}{[}\PY{n+nb+bp}{self}\PY{o}{.}\PY{n}{n\PYZus{}neurons\PYZus{}layer1}\PY{p}{]}\PY{p}{,} \PY{n}{name}\PY{o}{=}\PY{l+s+s1}{\PYZsq{}}\PY{l+s+s1}{fc1}\PY{l+s+s1}{\PYZsq{}}\PY{p}{,} \PY{n}{dtype}\PY{o}{=}\PY{n}{tf}\PY{o}{.}\PY{n}{float32}\PY{p}{)}
+                     \PY{n}{a\PYZus{}fc1} \PY{o}{=} \PY{n}{tf}\PY{o}{.}\PY{n}{nn}\PY{o}{.}\PY{n}{sigmoid}\PY{p}{(}\PY{n}{tf}\PY{o}{.}\PY{n}{matmul}\PY{p}{(}\PY{n+nb+bp}{self}\PY{o}{.}\PY{n}{X}\PY{p}{,} \PY{n+nb+bp}{self}\PY{o}{.}\PY{n}{W\PYZus{}fc1}\PY{p}{)} \PY{o}{+} \PY{n}{b\PYZus{}fc1}\PY{p}{)}
+                     
+                     \PY{c+c1}{\PYZsh{} Fully connected layer 2}
+                     \PY{n+nb+bp}{self}\PY{o}{.}\PY{n}{W\PYZus{}fc2} \PY{o}{=} \PY{n+nb+bp}{self}\PY{o}{.}\PY{n}{weight\PYZus{}variable}\PY{p}{(}\PY{p}{[}\PY{n+nb+bp}{self}\PY{o}{.}\PY{n}{n\PYZus{}neurons\PYZus{}layer1}\PY{p}{,} \PY{n+nb+bp}{self}\PY{o}{.}\PY{n}{n\PYZus{}neurons\PYZus{}layer2}\PY{p}{]}\PY{p}{,} \PY{n}{name}\PY{o}{=}\PY{l+s+s1}{\PYZsq{}}\PY{l+s+s1}{fc2}\PY{l+s+s1}{\PYZsq{}}\PY{p}{,} \PY{n}{dtype}\PY{o}{=}\PY{n}{tf}\PY{o}{.}\PY{n}{float32}\PY{p}{)}
+                     \PY{n}{b\PYZus{}fc2} \PY{o}{=} \PY{n+nb+bp}{self}\PY{o}{.}\PY{n}{bias\PYZus{}variable}\PY{p}{(}\PY{p}{[}\PY{n+nb+bp}{self}\PY{o}{.}\PY{n}{n\PYZus{}neurons\PYZus{}layer2}\PY{p}{]}\PY{p}{,} \PY{n}{name}\PY{o}{=}\PY{l+s+s1}{\PYZsq{}}\PY{l+s+s1}{fc2}\PY{l+s+s1}{\PYZsq{}}\PY{p}{,} \PY{n}{dtype}\PY{o}{=}\PY{n}{tf}\PY{o}{.}\PY{n}{float32}\PY{p}{)}
+                     \PY{n}{a\PYZus{}fc2} \PY{o}{=} \PY{n}{tf}\PY{o}{.}\PY{n}{nn}\PY{o}{.}\PY{n}{sigmoid}\PY{p}{(}\PY{n}{tf}\PY{o}{.}\PY{n}{matmul}\PY{p}{(}\PY{n}{a\PYZus{}fc1}\PY{p}{,} \PY{n+nb+bp}{self}\PY{o}{.}\PY{n}{W\PYZus{}fc2}\PY{p}{)} \PY{o}{+} \PY{n}{b\PYZus{}fc2}\PY{p}{)}
+                     
+                     \PY{c+c1}{\PYZsh{} Output layer}
+                     \PY{n+nb+bp}{self}\PY{o}{.}\PY{n}{W\PYZus{}out} \PY{o}{=} \PY{n+nb+bp}{self}\PY{o}{.}\PY{n}{weight\PYZus{}variable}\PY{p}{(}\PY{p}{[}\PY{n+nb+bp}{self}\PY{o}{.}\PY{n}{n\PYZus{}neurons\PYZus{}layer2}\PY{p}{,} \PY{n+nb+bp}{self}\PY{o}{.}\PY{n}{n\PYZus{}categories}\PY{p}{]}\PY{p}{,} \PY{n}{name}\PY{o}{=}\PY{l+s+s1}{\PYZsq{}}\PY{l+s+s1}{out}\PY{l+s+s1}{\PYZsq{}}\PY{p}{,} \PY{n}{dtype}\PY{o}{=}\PY{n}{tf}\PY{o}{.}\PY{n}{float32}\PY{p}{)}
+                     \PY{n}{b\PYZus{}out} \PY{o}{=} \PY{n+nb+bp}{self}\PY{o}{.}\PY{n}{bias\PYZus{}variable}\PY{p}{(}\PY{p}{[}\PY{n+nb+bp}{self}\PY{o}{.}\PY{n}{n\PYZus{}categories}\PY{p}{]}\PY{p}{,} \PY{n}{name}\PY{o}{=}\PY{l+s+s1}{\PYZsq{}}\PY{l+s+s1}{out}\PY{l+s+s1}{\PYZsq{}}\PY{p}{,} \PY{n}{dtype}\PY{o}{=}\PY{n}{tf}\PY{o}{.}\PY{n}{float32}\PY{p}{)}
+                     \PY{n+nb+bp}{self}\PY{o}{.}\PY{n}{z\PYZus{}out} \PY{o}{=} \PY{n}{tf}\PY{o}{.}\PY{n}{matmul}\PY{p}{(}\PY{n}{a\PYZus{}fc2}\PY{p}{,} \PY{n+nb+bp}{self}\PY{o}{.}\PY{n}{W\PYZus{}out}\PY{p}{)} \PY{o}{+} \PY{n}{b\PYZus{}out}
+             
+             \PY{k}{def} \PY{n+nf}{create\PYZus{}loss}\PY{p}{(}\PY{n+nb+bp}{self}\PY{p}{)}\PY{p}{:}
+                 \PY{k}{with} \PY{n}{tf}\PY{o}{.}\PY{n}{name\PYZus{}scope}\PY{p}{(}\PY{l+s+s1}{\PYZsq{}}\PY{l+s+s1}{loss}\PY{l+s+s1}{\PYZsq{}}\PY{p}{)}\PY{p}{:}
+                     \PY{n}{softmax\PYZus{}loss} \PY{o}{=} \PY{n}{tf}\PY{o}{.}\PY{n}{reduce\PYZus{}mean}\PY{p}{(}\PY{n}{tf}\PY{o}{.}\PY{n}{nn}\PY{o}{.}\PY{n}{softmax\PYZus{}cross\PYZus{}entropy\PYZus{}with\PYZus{}logits\PYZus{}v2}\PY{p}{(}\PY{n}{labels}\PY{o}{=}\PY{n+nb+bp}{self}\PY{o}{.}\PY{n}{Y}\PY{p}{,} \PY{n}{logits}\PY{o}{=}\PY{n+nb+bp}{self}\PY{o}{.}\PY{n}{z\PYZus{}out}\PY{p}{)}\PY{p}{)}
+                     
+                     \PY{n}{regularizer\PYZus{}loss\PYZus{}fc1} \PY{o}{=} \PY{n}{tf}\PY{o}{.}\PY{n}{nn}\PY{o}{.}\PY{n}{l2\PYZus{}loss}\PY{p}{(}\PY{n+nb+bp}{self}\PY{o}{.}\PY{n}{W\PYZus{}fc1}\PY{p}{)}
+                     \PY{n}{regularizer\PYZus{}loss\PYZus{}fc2} \PY{o}{=} \PY{n}{tf}\PY{o}{.}\PY{n}{nn}\PY{o}{.}\PY{n}{l2\PYZus{}loss}\PY{p}{(}\PY{n+nb+bp}{self}\PY{o}{.}\PY{n}{W\PYZus{}fc2}\PY{p}{)}
+                     \PY{n}{regularizer\PYZus{}loss\PYZus{}out} \PY{o}{=} \PY{n}{tf}\PY{o}{.}\PY{n}{nn}\PY{o}{.}\PY{n}{l2\PYZus{}loss}\PY{p}{(}\PY{n+nb+bp}{self}\PY{o}{.}\PY{n}{W\PYZus{}out}\PY{p}{)}
+                     \PY{n}{regularizer\PYZus{}loss} \PY{o}{=} \PY{n+nb+bp}{self}\PY{o}{.}\PY{n}{lmbd}\PY{o}{*}\PY{p}{(}\PY{n}{regularizer\PYZus{}loss\PYZus{}fc1} \PY{o}{+} \PY{n}{regularizer\PYZus{}loss\PYZus{}fc2} \PY{o}{+} \PY{n}{regularizer\PYZus{}loss\PYZus{}out}\PY{p}{)}
+                     
+                     \PY{n+nb+bp}{self}\PY{o}{.}\PY{n}{loss} \PY{o}{=} \PY{n}{softmax\PYZus{}loss} \PY{o}{+} \PY{n}{regularizer\PYZus{}loss}
+         
+             \PY{k}{def} \PY{n+nf}{create\PYZus{}accuracy}\PY{p}{(}\PY{n+nb+bp}{self}\PY{p}{)}\PY{p}{:}
+                 \PY{k}{with} \PY{n}{tf}\PY{o}{.}\PY{n}{name\PYZus{}scope}\PY{p}{(}\PY{l+s+s1}{\PYZsq{}}\PY{l+s+s1}{accuracy}\PY{l+s+s1}{\PYZsq{}}\PY{p}{)}\PY{p}{:}
+                     \PY{n}{probabilities} \PY{o}{=} \PY{n}{tf}\PY{o}{.}\PY{n}{nn}\PY{o}{.}\PY{n}{softmax}\PY{p}{(}\PY{n+nb+bp}{self}\PY{o}{.}\PY{n}{z\PYZus{}out}\PY{p}{)}
+                     \PY{n}{predictions} \PY{o}{=} \PY{n}{tf}\PY{o}{.}\PY{n}{argmax}\PY{p}{(}\PY{n}{probabilities}\PY{p}{,} \PY{n}{axis}\PY{o}{=}\PY{l+m+mi}{1}\PY{p}{)}
+                     \PY{n}{labels} \PY{o}{=} \PY{n}{tf}\PY{o}{.}\PY{n}{argmax}\PY{p}{(}\PY{n+nb+bp}{self}\PY{o}{.}\PY{n}{Y}\PY{p}{,} \PY{n}{axis}\PY{o}{=}\PY{l+m+mi}{1}\PY{p}{)}
+                     
+                     \PY{n}{correct\PYZus{}predictions} \PY{o}{=} \PY{n}{tf}\PY{o}{.}\PY{n}{equal}\PY{p}{(}\PY{n}{predictions}\PY{p}{,} \PY{n}{labels}\PY{p}{)}
+                     \PY{n}{correct\PYZus{}predictions} \PY{o}{=} \PY{n}{tf}\PY{o}{.}\PY{n}{cast}\PY{p}{(}\PY{n}{correct\PYZus{}predictions}\PY{p}{,} \PY{n}{tf}\PY{o}{.}\PY{n}{float32}\PY{p}{)}
+                     \PY{n+nb+bp}{self}\PY{o}{.}\PY{n}{accuracy} \PY{o}{=} \PY{n}{tf}\PY{o}{.}\PY{n}{reduce\PYZus{}mean}\PY{p}{(}\PY{n}{correct\PYZus{}predictions}\PY{p}{)}
+             
+             \PY{k}{def} \PY{n+nf}{create\PYZus{}optimiser}\PY{p}{(}\PY{n+nb+bp}{self}\PY{p}{)}\PY{p}{:}
+                 \PY{k}{with} \PY{n}{tf}\PY{o}{.}\PY{n}{name\PYZus{}scope}\PY{p}{(}\PY{l+s+s1}{\PYZsq{}}\PY{l+s+s1}{optimizer}\PY{l+s+s1}{\PYZsq{}}\PY{p}{)}\PY{p}{:}
+                     \PY{n+nb+bp}{self}\PY{o}{.}\PY{n}{optimizer} \PY{o}{=} \PY{n}{tf}\PY{o}{.}\PY{n}{train}\PY{o}{.}\PY{n}{GradientDescentOptimizer}\PY{p}{(}\PY{n}{learning\PYZus{}rate}\PY{o}{=}\PY{n+nb+bp}{self}\PY{o}{.}\PY{n}{eta}\PY{p}{)}\PY{o}{.}\PY{n}{minimize}\PY{p}{(}\PY{n+nb+bp}{self}\PY{o}{.}\PY{n}{loss}\PY{p}{,} \PY{n}{global\PYZus{}step}\PY{o}{=}\PY{n+nb+bp}{self}\PY{o}{.}\PY{n}{global\PYZus{}step}\PY{p}{)}
+                     
+             \PY{k}{def} \PY{n+nf}{weight\PYZus{}variable}\PY{p}{(}\PY{n+nb+bp}{self}\PY{p}{,} \PY{n}{shape}\PY{p}{,} \PY{n}{name}\PY{o}{=}\PY{l+s+s1}{\PYZsq{}}\PY{l+s+s1}{\PYZsq{}}\PY{p}{,} \PY{n}{dtype}\PY{o}{=}\PY{n}{tf}\PY{o}{.}\PY{n}{float32}\PY{p}{)}\PY{p}{:}
+                 \PY{n}{initial} \PY{o}{=} \PY{n}{tf}\PY{o}{.}\PY{n}{truncated\PYZus{}normal}\PY{p}{(}\PY{n}{shape}\PY{p}{,} \PY{n}{stddev}\PY{o}{=}\PY{l+m+mf}{0.1}\PY{p}{)}
+                 \PY{k}{return} \PY{n}{tf}\PY{o}{.}\PY{n}{Variable}\PY{p}{(}\PY{n}{initial}\PY{p}{,} \PY{n}{name}\PY{o}{=}\PY{n}{name}\PY{p}{,} \PY{n}{dtype}\PY{o}{=}\PY{n}{dtype}\PY{p}{)}
+             
+             \PY{k}{def} \PY{n+nf}{bias\PYZus{}variable}\PY{p}{(}\PY{n+nb+bp}{self}\PY{p}{,} \PY{n}{shape}\PY{p}{,} \PY{n}{name}\PY{o}{=}\PY{l+s+s1}{\PYZsq{}}\PY{l+s+s1}{\PYZsq{}}\PY{p}{,} \PY{n}{dtype}\PY{o}{=}\PY{n}{tf}\PY{o}{.}\PY{n}{float32}\PY{p}{)}\PY{p}{:}
+                 \PY{n}{initial} \PY{o}{=} \PY{n}{tf}\PY{o}{.}\PY{n}{constant}\PY{p}{(}\PY{l+m+mf}{0.1}\PY{p}{,} \PY{n}{shape}\PY{o}{=}\PY{n}{shape}\PY{p}{)}
+                 \PY{k}{return} \PY{n}{tf}\PY{o}{.}\PY{n}{Variable}\PY{p}{(}\PY{n}{initial}\PY{p}{,} \PY{n}{name}\PY{o}{=}\PY{n}{name}\PY{p}{,} \PY{n}{dtype}\PY{o}{=}\PY{n}{dtype}\PY{p}{)}
+             
+             \PY{k}{def} \PY{n+nf}{fit}\PY{p}{(}\PY{n+nb+bp}{self}\PY{p}{)}\PY{p}{:}
+                 \PY{n}{data\PYZus{}indices} \PY{o}{=} \PY{n}{np}\PY{o}{.}\PY{n}{arange}\PY{p}{(}\PY{n+nb+bp}{self}\PY{o}{.}\PY{n}{n\PYZus{}inputs}\PY{p}{)}
+         
+                 \PY{k}{with} \PY{n}{tf}\PY{o}{.}\PY{n}{Session}\PY{p}{(}\PY{p}{)} \PY{k}{as} \PY{n}{sess}\PY{p}{:}
+                     \PY{n}{sess}\PY{o}{.}\PY{n}{run}\PY{p}{(}\PY{n}{tf}\PY{o}{.}\PY{n}{global\PYZus{}variables\PYZus{}initializer}\PY{p}{(}\PY{p}{)}\PY{p}{)}
+                     \PY{k}{for} \PY{n}{i} \PY{o+ow}{in} \PY{n+nb}{range}\PY{p}{(}\PY{n+nb+bp}{self}\PY{o}{.}\PY{n}{epochs}\PY{p}{)}\PY{p}{:}
+                         \PY{k}{for} \PY{n}{j} \PY{o+ow}{in} \PY{n+nb}{range}\PY{p}{(}\PY{n+nb+bp}{self}\PY{o}{.}\PY{n}{iterations}\PY{p}{)}\PY{p}{:}
+                             \PY{n}{chosen\PYZus{}datapoints} \PY{o}{=} \PY{n}{np}\PY{o}{.}\PY{n}{random}\PY{o}{.}\PY{n}{choice}\PY{p}{(}\PY{n}{data\PYZus{}indices}\PY{p}{,} \PY{n}{size}\PY{o}{=}\PY{n+nb+bp}{self}\PY{o}{.}\PY{n}{batch\PYZus{}size}\PY{p}{,} \PY{n}{replace}\PY{o}{=}\PY{k+kc}{False}\PY{p}{)}
+                             \PY{n}{batch\PYZus{}X}\PY{p}{,} \PY{n}{batch\PYZus{}Y} \PY{o}{=} \PY{n+nb+bp}{self}\PY{o}{.}\PY{n}{X\PYZus{}train}\PY{p}{[}\PY{n}{chosen\PYZus{}datapoints}\PY{p}{]}\PY{p}{,} \PY{n+nb+bp}{self}\PY{o}{.}\PY{n}{Y\PYZus{}train}\PY{p}{[}\PY{n}{chosen\PYZus{}datapoints}\PY{p}{]}
+                     
+                             \PY{n}{sess}\PY{o}{.}\PY{n}{run}\PY{p}{(}\PY{p}{[}\PY{n}{DNN}\PY{o}{.}\PY{n}{loss}\PY{p}{,} \PY{n}{DNN}\PY{o}{.}\PY{n}{optimizer}\PY{p}{]}\PY{p}{,}
+                                 \PY{n}{feed\PYZus{}dict}\PY{o}{=}\PY{p}{\PYZob{}}\PY{n}{DNN}\PY{o}{.}\PY{n}{X}\PY{p}{:} \PY{n}{batch\PYZus{}X}\PY{p}{,}
+                                            \PY{n}{DNN}\PY{o}{.}\PY{n}{Y}\PY{p}{:} \PY{n}{batch\PYZus{}Y}\PY{p}{\PYZcb{}}\PY{p}{)}
+                             \PY{n}{accuracy} \PY{o}{=} \PY{n}{sess}\PY{o}{.}\PY{n}{run}\PY{p}{(}\PY{n}{DNN}\PY{o}{.}\PY{n}{accuracy}\PY{p}{,}
+                                 \PY{n}{feed\PYZus{}dict}\PY{o}{=}\PY{p}{\PYZob{}}\PY{n}{DNN}\PY{o}{.}\PY{n}{X}\PY{p}{:} \PY{n}{batch\PYZus{}X}\PY{p}{,}
+                                            \PY{n}{DNN}\PY{o}{.}\PY{n}{Y}\PY{p}{:} \PY{n}{batch\PYZus{}Y}\PY{p}{\PYZcb{}}\PY{p}{)}
+                             \PY{n}{step} \PY{o}{=} \PY{n}{sess}\PY{o}{.}\PY{n}{run}\PY{p}{(}\PY{n}{DNN}\PY{o}{.}\PY{n}{global\PYZus{}step}\PY{p}{)}
+             
+                     \PY{n+nb+bp}{self}\PY{o}{.}\PY{n}{train\PYZus{}loss}\PY{p}{,} \PY{n+nb+bp}{self}\PY{o}{.}\PY{n}{train\PYZus{}accuracy} \PY{o}{=} \PY{n}{sess}\PY{o}{.}\PY{n}{run}\PY{p}{(}\PY{p}{[}\PY{n}{DNN}\PY{o}{.}\PY{n}{loss}\PY{p}{,} \PY{n}{DNN}\PY{o}{.}\PY{n}{accuracy}\PY{p}{]}\PY{p}{,}
+                         \PY{n}{feed\PYZus{}dict}\PY{o}{=}\PY{p}{\PYZob{}}\PY{n}{DNN}\PY{o}{.}\PY{n}{X}\PY{p}{:} \PY{n+nb+bp}{self}\PY{o}{.}\PY{n}{X\PYZus{}train}\PY{p}{,}
+                                    \PY{n}{DNN}\PY{o}{.}\PY{n}{Y}\PY{p}{:} \PY{n+nb+bp}{self}\PY{o}{.}\PY{n}{Y\PYZus{}train}\PY{p}{\PYZcb{}}\PY{p}{)}
+                 
+                     \PY{n+nb+bp}{self}\PY{o}{.}\PY{n}{test\PYZus{}loss}\PY{p}{,} \PY{n+nb+bp}{self}\PY{o}{.}\PY{n}{test\PYZus{}accuracy} \PY{o}{=} \PY{n}{sess}\PY{o}{.}\PY{n}{run}\PY{p}{(}\PY{p}{[}\PY{n}{DNN}\PY{o}{.}\PY{n}{loss}\PY{p}{,} \PY{n}{DNN}\PY{o}{.}\PY{n}{accuracy}\PY{p}{]}\PY{p}{,}
+                         \PY{n}{feed\PYZus{}dict}\PY{o}{=}\PY{p}{\PYZob{}}\PY{n}{DNN}\PY{o}{.}\PY{n}{X}\PY{p}{:} \PY{n+nb+bp}{self}\PY{o}{.}\PY{n}{X\PYZus{}test}\PY{p}{,}
+                                    \PY{n}{DNN}\PY{o}{.}\PY{n}{Y}\PY{p}{:} \PY{n+nb+bp}{self}\PY{o}{.}\PY{n}{Y\PYZus{}test}\PY{p}{\PYZcb{}}\PY{p}{)}
+\end{Verbatim}
+
+
+    \hypertarget{optimizing-and-using-gradient-descent}{%
+\subsection{Optimizing and using gradient
+descent}\label{optimizing-and-using-gradient-descent}}
+
+    \begin{Verbatim}[commandchars=\\\{\}]
+{\color{incolor}In [{\color{incolor}18}]:} \PY{n}{epochs} \PY{o}{=} \PY{l+m+mi}{100}
+         \PY{n}{batch\PYZus{}size} \PY{o}{=} \PY{l+m+mi}{100}
+         \PY{n}{n\PYZus{}neurons\PYZus{}layer1} \PY{o}{=} \PY{l+m+mi}{100}
+         \PY{n}{n\PYZus{}neurons\PYZus{}layer2} \PY{o}{=} \PY{l+m+mi}{50}
+         \PY{n}{n\PYZus{}categories} \PY{o}{=} \PY{l+m+mi}{10}
+         \PY{n}{eta\PYZus{}vals} \PY{o}{=} \PY{n}{np}\PY{o}{.}\PY{n}{logspace}\PY{p}{(}\PY{o}{\PYZhy{}}\PY{l+m+mi}{5}\PY{p}{,} \PY{l+m+mi}{1}\PY{p}{,} \PY{l+m+mi}{7}\PY{p}{)}
+         \PY{n}{lmbd\PYZus{}vals} \PY{o}{=} \PY{n}{np}\PY{o}{.}\PY{n}{logspace}\PY{p}{(}\PY{o}{\PYZhy{}}\PY{l+m+mi}{5}\PY{p}{,} \PY{l+m+mi}{1}\PY{p}{,} \PY{l+m+mi}{7}\PY{p}{)}
+\end{Verbatim}
+
+
+    \begin{Verbatim}[commandchars=\\\{\}]
+{\color{incolor}In [{\color{incolor}19}]:} \PY{n}{DNN\PYZus{}tf} \PY{o}{=} \PY{n}{np}\PY{o}{.}\PY{n}{zeros}\PY{p}{(}\PY{p}{(}\PY{n+nb}{len}\PY{p}{(}\PY{n}{eta\PYZus{}vals}\PY{p}{)}\PY{p}{,} \PY{n+nb}{len}\PY{p}{(}\PY{n}{lmbd\PYZus{}vals}\PY{p}{)}\PY{p}{)}\PY{p}{,} \PY{n}{dtype}\PY{o}{=}\PY{n+nb}{object}\PY{p}{)}
+                 
+         \PY{k}{for} \PY{n}{i}\PY{p}{,} \PY{n}{eta} \PY{o+ow}{in} \PY{n+nb}{enumerate}\PY{p}{(}\PY{n}{eta\PYZus{}vals}\PY{p}{)}\PY{p}{:}
+             \PY{k}{for} \PY{n}{j}\PY{p}{,} \PY{n}{lmbd} \PY{o+ow}{in} \PY{n+nb}{enumerate}\PY{p}{(}\PY{n}{lmbd\PYZus{}vals}\PY{p}{)}\PY{p}{:}
+                 \PY{n}{DNN} \PY{o}{=} \PY{n}{NeuralNetworkTensorflow}\PY{p}{(}\PY{n}{X\PYZus{}train}\PY{p}{,} \PY{n}{Y\PYZus{}train}\PY{p}{,} \PY{n}{X\PYZus{}test}\PY{p}{,} \PY{n}{Y\PYZus{}test}\PY{p}{,}
+                                               \PY{n}{n\PYZus{}neurons\PYZus{}layer1}\PY{p}{,} \PY{n}{n\PYZus{}neurons\PYZus{}layer2}\PY{p}{,} \PY{n}{n\PYZus{}categories}\PY{p}{,}
+                                               \PY{n}{epochs}\PY{o}{=}\PY{n}{epochs}\PY{p}{,} \PY{n}{batch\PYZus{}size}\PY{o}{=}\PY{n}{batch\PYZus{}size}\PY{p}{,} \PY{n}{eta}\PY{o}{=}\PY{n}{eta}\PY{p}{,} \PY{n}{lmbd}\PY{o}{=}\PY{n}{lmbd}\PY{p}{)}
+                 \PY{n}{DNN}\PY{o}{.}\PY{n}{fit}\PY{p}{(}\PY{p}{)}
+                 
+                 \PY{n}{DNN\PYZus{}tf}\PY{p}{[}\PY{n}{i}\PY{p}{]}\PY{p}{[}\PY{n}{j}\PY{p}{]} \PY{o}{=} \PY{n}{DNN}
+                 
+                 \PY{n+nb}{print}\PY{p}{(}\PY{l+s+s2}{\PYZdq{}}\PY{l+s+s2}{Learning rate = }\PY{l+s+s2}{\PYZdq{}}\PY{p}{,} \PY{n}{eta}\PY{p}{)}
+                 \PY{n+nb}{print}\PY{p}{(}\PY{l+s+s2}{\PYZdq{}}\PY{l+s+s2}{Lambda = }\PY{l+s+s2}{\PYZdq{}}\PY{p}{,} \PY{n}{lmbd}\PY{p}{)}
+                 \PY{n+nb}{print}\PY{p}{(}\PY{l+s+s2}{\PYZdq{}}\PY{l+s+s2}{Test accuracy: }\PY{l+s+si}{\PYZpc{}.3f}\PY{l+s+s2}{\PYZdq{}} \PY{o}{\PYZpc{}} \PY{n}{DNN}\PY{o}{.}\PY{n}{test\PYZus{}accuracy}\PY{p}{)}
+                 \PY{n+nb}{print}\PY{p}{(}\PY{p}{)}
+\end{Verbatim}
+
+
+    \begin{Verbatim}[commandchars=\\\{\}]
+{\color{incolor}In [{\color{incolor}20}]:} \PY{c+c1}{\PYZsh{} optional}
+         \PY{c+c1}{\PYZsh{} visual representation of grid search}
+         \PY{c+c1}{\PYZsh{} uses seaborn heatmap, could probably do this in matplotlib}
+         \PY{k+kn}{import} \PY{n+nn}{seaborn} \PY{k}{as} \PY{n+nn}{sns}
+         
+         \PY{n}{sns}\PY{o}{.}\PY{n}{set}\PY{p}{(}\PY{p}{)}
+         
+         \PY{n}{train\PYZus{}accuracy} \PY{o}{=} \PY{n}{np}\PY{o}{.}\PY{n}{zeros}\PY{p}{(}\PY{p}{(}\PY{n+nb}{len}\PY{p}{(}\PY{n}{eta\PYZus{}vals}\PY{p}{)}\PY{p}{,} \PY{n+nb}{len}\PY{p}{(}\PY{n}{lmbd\PYZus{}vals}\PY{p}{)}\PY{p}{)}\PY{p}{)}
+         \PY{n}{test\PYZus{}accuracy} \PY{o}{=} \PY{n}{np}\PY{o}{.}\PY{n}{zeros}\PY{p}{(}\PY{p}{(}\PY{n+nb}{len}\PY{p}{(}\PY{n}{eta\PYZus{}vals}\PY{p}{)}\PY{p}{,} \PY{n+nb}{len}\PY{p}{(}\PY{n}{lmbd\PYZus{}vals}\PY{p}{)}\PY{p}{)}\PY{p}{)}
+         
+         \PY{k}{for} \PY{n}{i} \PY{o+ow}{in} \PY{n+nb}{range}\PY{p}{(}\PY{n+nb}{len}\PY{p}{(}\PY{n}{eta\PYZus{}vals}\PY{p}{)}\PY{p}{)}\PY{p}{:}
+             \PY{k}{for} \PY{n}{j} \PY{o+ow}{in} \PY{n+nb}{range}\PY{p}{(}\PY{n+nb}{len}\PY{p}{(}\PY{n}{lmbd\PYZus{}vals}\PY{p}{)}\PY{p}{)}\PY{p}{:}
+                 \PY{n}{DNN} \PY{o}{=} \PY{n}{DNN\PYZus{}tf}\PY{p}{[}\PY{n}{i}\PY{p}{]}\PY{p}{[}\PY{n}{j}\PY{p}{]}
+         
+                 \PY{n}{train\PYZus{}accuracy}\PY{p}{[}\PY{n}{i}\PY{p}{]}\PY{p}{[}\PY{n}{j}\PY{p}{]} \PY{o}{=} \PY{n}{DNN}\PY{o}{.}\PY{n}{train\PYZus{}accuracy}
+                 \PY{n}{test\PYZus{}accuracy}\PY{p}{[}\PY{n}{i}\PY{p}{]}\PY{p}{[}\PY{n}{j}\PY{p}{]} \PY{o}{=} \PY{n}{DNN}\PY{o}{.}\PY{n}{test\PYZus{}accuracy}
+         
+                 
+         \PY{n}{fig}\PY{p}{,} \PY{n}{ax} \PY{o}{=} \PY{n}{plt}\PY{o}{.}\PY{n}{subplots}\PY{p}{(}\PY{n}{figsize} \PY{o}{=} \PY{p}{(}\PY{l+m+mi}{10}\PY{p}{,} \PY{l+m+mi}{10}\PY{p}{)}\PY{p}{)}
+         \PY{n}{sns}\PY{o}{.}\PY{n}{heatmap}\PY{p}{(}\PY{n}{train\PYZus{}accuracy}\PY{p}{,} \PY{n}{annot}\PY{o}{=}\PY{k+kc}{True}\PY{p}{,} \PY{n}{ax}\PY{o}{=}\PY{n}{ax}\PY{p}{,} \PY{n}{cmap}\PY{o}{=}\PY{l+s+s2}{\PYZdq{}}\PY{l+s+s2}{viridis}\PY{l+s+s2}{\PYZdq{}}\PY{p}{)}
+         \PY{n}{ax}\PY{o}{.}\PY{n}{set\PYZus{}title}\PY{p}{(}\PY{l+s+s2}{\PYZdq{}}\PY{l+s+s2}{Training Accuracy}\PY{l+s+s2}{\PYZdq{}}\PY{p}{)}
+         \PY{n}{ax}\PY{o}{.}\PY{n}{set\PYZus{}ylabel}\PY{p}{(}\PY{l+s+s2}{\PYZdq{}}\PY{l+s+s2}{\PYZdl{}}\PY{l+s+s2}{\PYZbs{}}\PY{l+s+s2}{eta\PYZdl{}}\PY{l+s+s2}{\PYZdq{}}\PY{p}{)}
+         \PY{n}{ax}\PY{o}{.}\PY{n}{set\PYZus{}xlabel}\PY{p}{(}\PY{l+s+s2}{\PYZdq{}}\PY{l+s+s2}{\PYZdl{}}\PY{l+s+s2}{\PYZbs{}}\PY{l+s+s2}{lambda\PYZdl{}}\PY{l+s+s2}{\PYZdq{}}\PY{p}{)}
+         \PY{n}{plt}\PY{o}{.}\PY{n}{show}\PY{p}{(}\PY{p}{)}
+         
+         \PY{n}{fig}\PY{p}{,} \PY{n}{ax} \PY{o}{=} \PY{n}{plt}\PY{o}{.}\PY{n}{subplots}\PY{p}{(}\PY{n}{figsize} \PY{o}{=} \PY{p}{(}\PY{l+m+mi}{10}\PY{p}{,} \PY{l+m+mi}{10}\PY{p}{)}\PY{p}{)}
+         \PY{n}{sns}\PY{o}{.}\PY{n}{heatmap}\PY{p}{(}\PY{n}{test\PYZus{}accuracy}\PY{p}{,} \PY{n}{annot}\PY{o}{=}\PY{k+kc}{True}\PY{p}{,} \PY{n}{ax}\PY{o}{=}\PY{n}{ax}\PY{p}{,} \PY{n}{cmap}\PY{o}{=}\PY{l+s+s2}{\PYZdq{}}\PY{l+s+s2}{viridis}\PY{l+s+s2}{\PYZdq{}}\PY{p}{)}
+         \PY{n}{ax}\PY{o}{.}\PY{n}{set\PYZus{}title}\PY{p}{(}\PY{l+s+s2}{\PYZdq{}}\PY{l+s+s2}{Test Accuracy}\PY{l+s+s2}{\PYZdq{}}\PY{p}{)}
+         \PY{n}{ax}\PY{o}{.}\PY{n}{set\PYZus{}ylabel}\PY{p}{(}\PY{l+s+s2}{\PYZdq{}}\PY{l+s+s2}{\PYZdl{}}\PY{l+s+s2}{\PYZbs{}}\PY{l+s+s2}{eta\PYZdl{}}\PY{l+s+s2}{\PYZdq{}}\PY{p}{)}
+         \PY{n}{ax}\PY{o}{.}\PY{n}{set\PYZus{}xlabel}\PY{p}{(}\PY{l+s+s2}{\PYZdq{}}\PY{l+s+s2}{\PYZdl{}}\PY{l+s+s2}{\PYZbs{}}\PY{l+s+s2}{lambda\PYZdl{}}\PY{l+s+s2}{\PYZdq{}}\PY{p}{)}
+         \PY{n}{plt}\PY{o}{.}\PY{n}{show}\PY{p}{(}\PY{p}{)}
+\end{Verbatim}
+
+
+    \begin{Verbatim}[commandchars=\\\{\}]
+{\color{incolor}In [{\color{incolor}21}]:} \PY{c+c1}{\PYZsh{} optional}
+         \PY{c+c1}{\PYZsh{} we can use log files to visualize our graph in Tensorboard}
+         \PY{n}{writer} \PY{o}{=} \PY{n}{tf}\PY{o}{.}\PY{n}{summary}\PY{o}{.}\PY{n}{FileWriter}\PY{p}{(}\PY{l+s+s1}{\PYZsq{}}\PY{l+s+s1}{logs/}\PY{l+s+s1}{\PYZsq{}}\PY{p}{)}
+         \PY{n}{writer}\PY{o}{.}\PY{n}{add\PYZus{}graph}\PY{p}{(}\PY{n}{tf}\PY{o}{.}\PY{n}{get\PYZus{}default\PYZus{}graph}\PY{p}{(}\PY{p}{)}\PY{p}{)}
+\end{Verbatim}
+
+
+    \hypertarget{using-keras}{%
+\subsection{Using Keras}\label{using-keras}}
+
+Keras is a high level
+\href{https://en.wikipedia.org/wiki/Application_programming_interface}{neural
+network} that supports Tensorflow, CTNK and Theano as backends.\\
+If you have Tensorflow installed Keras is available through the
+\emph{tf.keras} module.\\
+If you have Anaconda installed you may run the following command
+
+    \begin{Verbatim}[commandchars=\\\{\}]
+{\color{incolor}In [{\color{incolor}22}]:} \PY{n}{conda} \PY{n}{install} \PY{n}{keras}
+\end{Verbatim}
+
+
+    Alternatively, if you have Tensorflow or one of the other supported
+backends install you may use the pip package manager:
+
+    \begin{Verbatim}[commandchars=\\\{\}]
+{\color{incolor}In [{\color{incolor}23}]:} \PY{n}{pip3} \PY{n}{install} \PY{n}{keras}
+\end{Verbatim}
+
+
+    or look up the \href{https://keras.io/}{instructions here}.
+
+    \begin{Verbatim}[commandchars=\\\{\}]
+{\color{incolor}In [{\color{incolor}24}]:} \PY{k+kn}{from} \PY{n+nn}{keras}\PY{n+nn}{.}\PY{n+nn}{models} \PY{k}{import} \PY{n}{Sequential}
+         \PY{k+kn}{from} \PY{n+nn}{keras}\PY{n+nn}{.}\PY{n+nn}{layers} \PY{k}{import} \PY{n}{Dense}
+         \PY{k+kn}{from} \PY{n+nn}{keras}\PY{n+nn}{.}\PY{n+nn}{regularizers} \PY{k}{import} \PY{n}{l2}
+         \PY{k+kn}{from} \PY{n+nn}{keras}\PY{n+nn}{.}\PY{n+nn}{optimizers} \PY{k}{import} \PY{n}{SGD}
+         
+         \PY{k}{def} \PY{n+nf}{create\PYZus{}neural\PYZus{}network\PYZus{}keras}\PY{p}{(}\PY{n}{n\PYZus{}neurons\PYZus{}layer1}\PY{p}{,} \PY{n}{n\PYZus{}neurons\PYZus{}layer2}\PY{p}{,} \PY{n}{n\PYZus{}categories}\PY{p}{,} \PY{n}{eta}\PY{p}{,} \PY{n}{lmbd}\PY{p}{)}\PY{p}{:}
+             \PY{n}{model} \PY{o}{=} \PY{n}{Sequential}\PY{p}{(}\PY{p}{)}
+             \PY{n}{model}\PY{o}{.}\PY{n}{add}\PY{p}{(}\PY{n}{Dense}\PY{p}{(}\PY{n}{n\PYZus{}neurons\PYZus{}layer1}\PY{p}{,} \PY{n}{activation}\PY{o}{=}\PY{l+s+s1}{\PYZsq{}}\PY{l+s+s1}{sigmoid}\PY{l+s+s1}{\PYZsq{}}\PY{p}{,} \PY{n}{kernel\PYZus{}regularizer}\PY{o}{=}\PY{n}{l2}\PY{p}{(}\PY{n}{lmbd}\PY{p}{)}\PY{p}{)}\PY{p}{)}
+             \PY{n}{model}\PY{o}{.}\PY{n}{add}\PY{p}{(}\PY{n}{Dense}\PY{p}{(}\PY{n}{n\PYZus{}neurons\PYZus{}layer2}\PY{p}{,} \PY{n}{activation}\PY{o}{=}\PY{l+s+s1}{\PYZsq{}}\PY{l+s+s1}{sigmoid}\PY{l+s+s1}{\PYZsq{}}\PY{p}{,} \PY{n}{kernel\PYZus{}regularizer}\PY{o}{=}\PY{n}{l2}\PY{p}{(}\PY{n}{lmbd}\PY{p}{)}\PY{p}{)}\PY{p}{)}
+             \PY{n}{model}\PY{o}{.}\PY{n}{add}\PY{p}{(}\PY{n}{Dense}\PY{p}{(}\PY{n}{n\PYZus{}categories}\PY{p}{,} \PY{n}{activation}\PY{o}{=}\PY{l+s+s1}{\PYZsq{}}\PY{l+s+s1}{softmax}\PY{l+s+s1}{\PYZsq{}}\PY{p}{)}\PY{p}{)}
+             
+             \PY{n}{sgd} \PY{o}{=} \PY{n}{SGD}\PY{p}{(}\PY{n}{lr}\PY{o}{=}\PY{n}{eta}\PY{p}{)}
+             \PY{n}{model}\PY{o}{.}\PY{n}{compile}\PY{p}{(}\PY{n}{loss}\PY{o}{=}\PY{l+s+s1}{\PYZsq{}}\PY{l+s+s1}{categorical\PYZus{}crossentropy}\PY{l+s+s1}{\PYZsq{}}\PY{p}{,} \PY{n}{optimizer}\PY{o}{=}\PY{n}{sgd}\PY{p}{,} \PY{n}{metrics}\PY{o}{=}\PY{p}{[}\PY{l+s+s1}{\PYZsq{}}\PY{l+s+s1}{accuracy}\PY{l+s+s1}{\PYZsq{}}\PY{p}{]}\PY{p}{)}
+             
+             \PY{k}{return} \PY{n}{model}
+\end{Verbatim}
+
+
+    \begin{Verbatim}[commandchars=\\\{\}]
+{\color{incolor}In [{\color{incolor}25}]:} \PY{n}{DNN\PYZus{}keras} \PY{o}{=} \PY{n}{np}\PY{o}{.}\PY{n}{zeros}\PY{p}{(}\PY{p}{(}\PY{n+nb}{len}\PY{p}{(}\PY{n}{eta\PYZus{}vals}\PY{p}{)}\PY{p}{,} \PY{n+nb}{len}\PY{p}{(}\PY{n}{lmbd\PYZus{}vals}\PY{p}{)}\PY{p}{)}\PY{p}{,} \PY{n}{dtype}\PY{o}{=}\PY{n+nb}{object}\PY{p}{)}
+                 
+         \PY{k}{for} \PY{n}{i}\PY{p}{,} \PY{n}{eta} \PY{o+ow}{in} \PY{n+nb}{enumerate}\PY{p}{(}\PY{n}{eta\PYZus{}vals}\PY{p}{)}\PY{p}{:}
+             \PY{k}{for} \PY{n}{j}\PY{p}{,} \PY{n}{lmbd} \PY{o+ow}{in} \PY{n+nb}{enumerate}\PY{p}{(}\PY{n}{lmbd\PYZus{}vals}\PY{p}{)}\PY{p}{:}
+                 \PY{n}{DNN} \PY{o}{=} \PY{n}{create\PYZus{}neural\PYZus{}network\PYZus{}keras}\PY{p}{(}\PY{n}{n\PYZus{}neurons\PYZus{}layer1}\PY{p}{,} \PY{n}{n\PYZus{}neurons\PYZus{}layer2}\PY{p}{,} \PY{n}{n\PYZus{}categories}\PY{p}{,}
+                                                  \PY{n}{eta}\PY{o}{=}\PY{n}{eta}\PY{p}{,} \PY{n}{lmbd}\PY{o}{=}\PY{n}{lmbd}\PY{p}{)}
+                 \PY{n}{DNN}\PY{o}{.}\PY{n}{fit}\PY{p}{(}\PY{n}{X\PYZus{}train}\PY{p}{,} \PY{n}{Y\PYZus{}train}\PY{p}{,} \PY{n}{epochs}\PY{o}{=}\PY{n}{epochs}\PY{p}{,} \PY{n}{batch\PYZus{}size}\PY{o}{=}\PY{n}{batch\PYZus{}size}\PY{p}{,} \PY{n}{verbose}\PY{o}{=}\PY{l+m+mi}{0}\PY{p}{)}
+                 \PY{n}{scores} \PY{o}{=} \PY{n}{DNN}\PY{o}{.}\PY{n}{evaluate}\PY{p}{(}\PY{n}{X\PYZus{}test}\PY{p}{,} \PY{n}{Y\PYZus{}test}\PY{p}{)}
+                 
+                 \PY{n}{DNN\PYZus{}keras}\PY{p}{[}\PY{n}{i}\PY{p}{]}\PY{p}{[}\PY{n}{j}\PY{p}{]} \PY{o}{=} \PY{n}{DNN}
+                 
+                 \PY{n+nb}{print}\PY{p}{(}\PY{l+s+s2}{\PYZdq{}}\PY{l+s+s2}{Learning rate = }\PY{l+s+s2}{\PYZdq{}}\PY{p}{,} \PY{n}{eta}\PY{p}{)}
+                 \PY{n+nb}{print}\PY{p}{(}\PY{l+s+s2}{\PYZdq{}}\PY{l+s+s2}{Lambda = }\PY{l+s+s2}{\PYZdq{}}\PY{p}{,} \PY{n}{lmbd}\PY{p}{)}
+                 \PY{n+nb}{print}\PY{p}{(}\PY{l+s+s2}{\PYZdq{}}\PY{l+s+s2}{Test accuracy: }\PY{l+s+si}{\PYZpc{}.3f}\PY{l+s+s2}{\PYZdq{}} \PY{o}{\PYZpc{}} \PY{n}{scores}\PY{p}{[}\PY{l+m+mi}{1}\PY{p}{]}\PY{p}{)}
+                 \PY{n+nb}{print}\PY{p}{(}\PY{p}{)}
+\end{Verbatim}
+
+
+    \begin{Verbatim}[commandchars=\\\{\}]
+{\color{incolor}In [{\color{incolor}26}]:} \PY{c+c1}{\PYZsh{} optional}
+         \PY{c+c1}{\PYZsh{} visual representation of grid search}
+         \PY{c+c1}{\PYZsh{} uses seaborn heatmap, could probably do this in matplotlib}
+         \PY{k+kn}{import} \PY{n+nn}{seaborn} \PY{k}{as} \PY{n+nn}{sns}
+         
+         \PY{n}{sns}\PY{o}{.}\PY{n}{set}\PY{p}{(}\PY{p}{)}
+         
+         \PY{n}{train\PYZus{}accuracy} \PY{o}{=} \PY{n}{np}\PY{o}{.}\PY{n}{zeros}\PY{p}{(}\PY{p}{(}\PY{n+nb}{len}\PY{p}{(}\PY{n}{eta\PYZus{}vals}\PY{p}{)}\PY{p}{,} \PY{n+nb}{len}\PY{p}{(}\PY{n}{lmbd\PYZus{}vals}\PY{p}{)}\PY{p}{)}\PY{p}{)}
+         \PY{n}{test\PYZus{}accuracy} \PY{o}{=} \PY{n}{np}\PY{o}{.}\PY{n}{zeros}\PY{p}{(}\PY{p}{(}\PY{n+nb}{len}\PY{p}{(}\PY{n}{eta\PYZus{}vals}\PY{p}{)}\PY{p}{,} \PY{n+nb}{len}\PY{p}{(}\PY{n}{lmbd\PYZus{}vals}\PY{p}{)}\PY{p}{)}\PY{p}{)}
+         
+         \PY{k}{for} \PY{n}{i} \PY{o+ow}{in} \PY{n+nb}{range}\PY{p}{(}\PY{n+nb}{len}\PY{p}{(}\PY{n}{eta\PYZus{}vals}\PY{p}{)}\PY{p}{)}\PY{p}{:}
+             \PY{k}{for} \PY{n}{j} \PY{o+ow}{in} \PY{n+nb}{range}\PY{p}{(}\PY{n+nb}{len}\PY{p}{(}\PY{n}{lmbd\PYZus{}vals}\PY{p}{)}\PY{p}{)}\PY{p}{:}
+                 \PY{n}{DNN} \PY{o}{=} \PY{n}{DNN\PYZus{}keras}\PY{p}{[}\PY{n}{i}\PY{p}{]}\PY{p}{[}\PY{n}{j}\PY{p}{]}
+         
+                 \PY{n}{train\PYZus{}accuracy}\PY{p}{[}\PY{n}{i}\PY{p}{]}\PY{p}{[}\PY{n}{j}\PY{p}{]} \PY{o}{=} \PY{n}{DNN}\PY{o}{.}\PY{n}{evaluate}\PY{p}{(}\PY{n}{X\PYZus{}train}\PY{p}{,} \PY{n}{Y\PYZus{}train}\PY{p}{)}\PY{p}{[}\PY{l+m+mi}{1}\PY{p}{]}
+                 \PY{n}{test\PYZus{}accuracy}\PY{p}{[}\PY{n}{i}\PY{p}{]}\PY{p}{[}\PY{n}{j}\PY{p}{]} \PY{o}{=} \PY{n}{DNN}\PY{o}{.}\PY{n}{evaluate}\PY{p}{(}\PY{n}{X\PYZus{}test}\PY{p}{,} \PY{n}{Y\PYZus{}test}\PY{p}{)}\PY{p}{[}\PY{l+m+mi}{1}\PY{p}{]}
+         
+                 
+         \PY{n}{fig}\PY{p}{,} \PY{n}{ax} \PY{o}{=} \PY{n}{plt}\PY{o}{.}\PY{n}{subplots}\PY{p}{(}\PY{n}{figsize} \PY{o}{=} \PY{p}{(}\PY{l+m+mi}{10}\PY{p}{,} \PY{l+m+mi}{10}\PY{p}{)}\PY{p}{)}
+         \PY{n}{sns}\PY{o}{.}\PY{n}{heatmap}\PY{p}{(}\PY{n}{train\PYZus{}accuracy}\PY{p}{,} \PY{n}{annot}\PY{o}{=}\PY{k+kc}{True}\PY{p}{,} \PY{n}{ax}\PY{o}{=}\PY{n}{ax}\PY{p}{,} \PY{n}{cmap}\PY{o}{=}\PY{l+s+s2}{\PYZdq{}}\PY{l+s+s2}{viridis}\PY{l+s+s2}{\PYZdq{}}\PY{p}{)}
+         \PY{n}{ax}\PY{o}{.}\PY{n}{set\PYZus{}title}\PY{p}{(}\PY{l+s+s2}{\PYZdq{}}\PY{l+s+s2}{Training Accuracy}\PY{l+s+s2}{\PYZdq{}}\PY{p}{)}
+         \PY{n}{ax}\PY{o}{.}\PY{n}{set\PYZus{}ylabel}\PY{p}{(}\PY{l+s+s2}{\PYZdq{}}\PY{l+s+s2}{\PYZdl{}}\PY{l+s+s2}{\PYZbs{}}\PY{l+s+s2}{eta\PYZdl{}}\PY{l+s+s2}{\PYZdq{}}\PY{p}{)}
+         \PY{n}{ax}\PY{o}{.}\PY{n}{set\PYZus{}xlabel}\PY{p}{(}\PY{l+s+s2}{\PYZdq{}}\PY{l+s+s2}{\PYZdl{}}\PY{l+s+s2}{\PYZbs{}}\PY{l+s+s2}{lambda\PYZdl{}}\PY{l+s+s2}{\PYZdq{}}\PY{p}{)}
+         \PY{n}{plt}\PY{o}{.}\PY{n}{show}\PY{p}{(}\PY{p}{)}
+         
+         \PY{n}{fig}\PY{p}{,} \PY{n}{ax} \PY{o}{=} \PY{n}{plt}\PY{o}{.}\PY{n}{subplots}\PY{p}{(}\PY{n}{figsize} \PY{o}{=} \PY{p}{(}\PY{l+m+mi}{10}\PY{p}{,} \PY{l+m+mi}{10}\PY{p}{)}\PY{p}{)}
+         \PY{n}{sns}\PY{o}{.}\PY{n}{heatmap}\PY{p}{(}\PY{n}{test\PYZus{}accuracy}\PY{p}{,} \PY{n}{annot}\PY{o}{=}\PY{k+kc}{True}\PY{p}{,} \PY{n}{ax}\PY{o}{=}\PY{n}{ax}\PY{p}{,} \PY{n}{cmap}\PY{o}{=}\PY{l+s+s2}{\PYZdq{}}\PY{l+s+s2}{viridis}\PY{l+s+s2}{\PYZdq{}}\PY{p}{)}
+         \PY{n}{ax}\PY{o}{.}\PY{n}{set\PYZus{}title}\PY{p}{(}\PY{l+s+s2}{\PYZdq{}}\PY{l+s+s2}{Test Accuracy}\PY{l+s+s2}{\PYZdq{}}\PY{p}{)}
+         \PY{n}{ax}\PY{o}{.}\PY{n}{set\PYZus{}ylabel}\PY{p}{(}\PY{l+s+s2}{\PYZdq{}}\PY{l+s+s2}{\PYZdl{}}\PY{l+s+s2}{\PYZbs{}}\PY{l+s+s2}{eta\PYZdl{}}\PY{l+s+s2}{\PYZdq{}}\PY{p}{)}
+         \PY{n}{ax}\PY{o}{.}\PY{n}{set\PYZus{}xlabel}\PY{p}{(}\PY{l+s+s2}{\PYZdq{}}\PY{l+s+s2}{\PYZdl{}}\PY{l+s+s2}{\PYZbs{}}\PY{l+s+s2}{lambda\PYZdl{}}\PY{l+s+s2}{\PYZdq{}}\PY{p}{)}
+         \PY{n}{plt}\PY{o}{.}\PY{n}{show}\PY{p}{(}\PY{p}{)}
+\end{Verbatim}
+
+
+
+    % Add a bibliography block to the postdoc
+    
+    
+    
+    \end{document}
diff --git a/doc/pub/Recurrent/html/._Recurrent-bs002.html b/doc/pub/Recurrent/html/._Recurrent-bs002.html
new file mode 100644
index 000000000..95d697fdd
--- /dev/null
+++ b/doc/pub/Recurrent/html/._Recurrent-bs002.html
@@ -0,0 +1,146 @@
+
+
+
+
+
+
+
+
+Data Analysis and Machine Learning: Recurrent neural networks
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+    
+
+

+  

+    
+    Data Analysis and Machine Learning: Recurrent neural networks
+  

+
+  

+    
+  

+

+
)!M*aHo(Q3(M9$sksm?5AcN8-mU`l|2AoAfi?2<@h}qjPr*q
z^2irsC|9UE^3UV@*S{x&H_F@S1NFa8*Wk=a!%*vSs31Kl$U*-mcV4z??Dze2+yPB9
zMz-wkiK}7UTp{e*xd1go8DTDimMBJ_U9B{tFdOh4S804DhA*%@KI5MBn&-z{7R|!d
zMv?z+|Hml#)*jm!l_e{Wx#K_E42zSNc3B;!g(SMU#=2F*44oF^EiNWCpf!Qoa*0R#
z5WAYuJ6Qj}R@aWrl9>Lyrarc|4HOJaIsGH%_)2k5M~5U9YTtv}C7Wg=83a!qKFe|<
zXNH_}5dCjjgB$2s|!<(*B
zniHhAa>A9FXPQ6avlhmm4F8gTk4z$yz#_@y_wOskZrHY2p~8y<)eIcGHvvZ);#QVs
zl)t?q6}(Oz_3SnwJ6wb`vm-M+42PYv%YQc-Q4jpayl%ag*9*Cp9}-s?C5~5G!6l7w
z^jSmko&1ztDHy9noTt9iVn_HV@wz)Phk)Xcc^LKY0_=9OpB##^q=fE>bM5rOWt2bo
zE&DwL0>i#O@iuH`-lmi#`X7+=jUBmFW=+Q^fxEt2`j=l;y?GQ`D4P&bM)~B~g*|2i
zL(0cN`y!?>C20=t^_nHAd__a;+{dgB+wIH+=W$88;PMZj&8B}%9pORXg+iVkS1xF2
zR|-_aj6}Lq6ATm{EDtio1*T3s$g8wz8^GQ;RtUqM
zfz$7U#g>@A-MWBG5_Yg{QnB!kq|%aAIZd!f{qnpuC_3eI@I>TC;?p%)rM9rBV-N=C
z2?k3kox#C#Omy>i#06gHqeK6^?MZXU!Jii?p@)AjY)x|>}4PRs5k7OMa-tk?qmm_5gi>%uWmlbo(3nhBon@&1|6v<9r
zuXWNH-SUTWoPFpQt@W@9>3O$XIfcv3saN4q(PB9my`iP1w*oL-aZmu|;Hv^YJci9J
zwl+435wzc|B;s<%aqN&NJF^!Kl|_?!{_L8BkmQ?s>d6`nFgeA6X`acIplN=tYZ6y7
zfEgr}52-?;`EZI-qIB#Ci)vR`s(Z0As(|&37gAvE3O8@7rr!@ueij8G2VDvaXsx;h
z0E32}d9NE^?CKD&XToAbR^@xSJMh)J-zaxDzemg2UvBvqGmwc-UjIO&7^JiV+VygB
zJ*|t-{4wVCzu&0IBiZ8w{^&)A
zU}xoI{||F$fMhM1*!?!lp6A*d6{rf%1ZlfcL$IlZarrrEY!rw@0?TkS7um?d;tIiD
zA%{YW&PmJz{}PwY{n^pf$bTPtjO%p~ND{Gm-AM9o=*TqJx(38$kNh!N|D!KDaFw_76L%z-kH{dW+5p=TTQqZfzbx+K*Ea
z%LCUd!AG%4#?_o>;zYKS&~ew_mOJdmm>=}2>s*gQNRm_!N?nF^;~9zkf)CRg3YxUTxoHKM4hLE`m>+2`l>vEnCbM%12kboGs2?b906i
z{%~_>-**|LW<%?AT08r4v+Y`)=6JXy0
zi#MBTts?1WK{c@7Pv;zI)_XIV7a9`EkKlFQKwb20D`bHY(L*DHQ0`69;b*cR?lS}$r3i+9Dzg!5ykV&brr5DB!aqocDf<0P6kRpF73IV8*AUE0SHrxk9fO2VBD
zkOTFe``3Lezq}o$mH!B}PEwL!?7TrbM#=^{!_&S`ZLGuZE@*&6!5^tU951k(@UE}%
zOAc~TI@D})zh}nU?!Oo@GREHTJx?!m_#jQ8p#S|9sMe4U7fmP
ziCK(ZpeYW>JzjC`L*jSNy@m%fF!w$v>7U`n2kRx##&?Y=t*s?3^oo
z+aKQ0JnYXCT;afozWygJC99XRzbQq(B1qvzSQ}S2M?a1jJfgK|+m3VG>ksECK*uGm
z`MJEPBWZ<670(jkrj%cK81iYo?n2)6nk$bs3jg6FF}~LSrZ<)u)~eEvTWx(J4zstN
z2k8Kv45hX4R`XWQ^RU7`GYD@D`ntpLd`n5slLJ~@Dh&n!2O(lP?|qAO(N>Fq+Y{&4`$dLiSgk$Z>!-x~T{XQ5FyO-X6=4?a
z2hC$ZbQi*Q%7&<39~J={zvw3T#+cR(7~=?JH|2^9u_#)mA$RSKH`O?qJ0@a#5A5x^Q6N6eGQ<`A
zn*S$gb>LLy5)buTiKDILay0zag(SvP>8zSnYAh(qtuWKnk8q6}Y8L9u4K~!wL0lj?
z6N?=iYG6QP4M5I5X+r2PdR^%T0TGd9A5Dio8a*E$NFTC;c1$W6yh8myU$Th
z@M1HKoCa0ULU4TxY%p0#I_OaKR3JidyR-bQH*lZEkv$N#&IRPz-R#C9^RuC!8$w@f
zs90%Z;5KmYK|h+;7Qqz4zD{qFS0+xsOhFuXMXx(%gdWe2J4_F%#H06tjk;
zgBs$!W8!&B3PaI!)IZMK|13L?d%)URh_f><4(rQ&i%44n5RFyMaF4XT2)qtE_wDwe
z<0oM2SOY?i0+k+ct_2v!VB16`C3i33j;ZKZ?l#mFgtf7%Oj#M$A;b;{iHjoyrr^CO
zi7l~Qv&FpI4!XjK`{GgXruq$G%O4QIvD&n!T|tHCj0+0{kKP72yz$~7SXfa}WC8JV
z-u@x4E|pT?j*CSEj~I(Co`gG$4A!;K6&#bmMSSgu#)>!iFUQ*#`p3u?k;cWVWAT1E
zx2@mA!KzA4??x$8u&9m
z`!xKOmApq5VmW7+XQn?>-h?X|j%BQIle@&mS;!-Ba?XLJkb(k^A2@b(s)!eAmSFo6
zjj&Kh>>2`pDdD=eA!6>NvM@-gRy*G$9Pbx|pQo!|kyosG=|v+561mOVe}1+)Y9Y?z`~
zB=euoR|KalF-4-ry?g^4W(0nmVsQFSqa33E(aw4~Vf(kYv)s{7W4n_55CMOF_RvmM
z=lKNmS4GQAfto0-L0R|3w%IxSbWKikf@?h?+C-KoPBC}Sb*YA|N;Y{NTtvd~gDK0C
z#Ye}5J{*=&nsIY8FH-?{Z&PR*PD~ubDj4`GEgUdYlx6>g8Z3$AV~PrXR({%o|4so0
zgXhDBR2>H9_@-ivu{bslZt<3P`AWDx{!^D0IxK{E4BXa$&3vT9#2$oxQ@u#sThTETK@Qjq2Xlm3ol;mEVX3i+{
zcwV(?D9}FhgV=lsi^Za$81HUqnW=zw2*^kYw$n<_n5fm5Afa3%%Pr*)Ivg(ik0ea9
z__aK+a@Rus1s3wrB?Cb-Q;3lmRM|dMWo%8v%fxtHt9%?TO4{I(kB$co)L>6s>J3?4
zL2mPKnzjqZ+$>AN3UP=7qcgONkeo=aU*YwtOI~`0wH@ZJZs-
z=Zvt5OxYdc%H*f{TT4@ZaZ^4m-lDKZtnH*UB6$B<3rFgQ3I!r%iS|S(p20wj2z=0R
zH1*%~hP^*e&|owo3N#O$R5{}LF{yl7oWJUntGJF|2efPrN^zudX*>Hg1YLe`j8le+
zlD&+KCqmH=t!wxdnf|8JJst)aw}$d3vUnwpWbSkBq$Ey0&s$~bu5ywoz%5RC5ihXM
zC%W8JI?ZV!D;Dh+tK|JWq2i$638-IPo$X{$L?Kn&Vq{63&vq=*kM2hKB?8)}29{Fk
zt&|YQ$6>Xn1F(z_1;dVt!nEaziQqp#o|z{P5|hb@D{vIWV$*V6yhI8dgjK}2mA{(y44rnS{k8d+7MkR30|A`b
zH|a~+vYOF8>7M5uUT7u7%+;b3G?>)^n@uz9Vb7-Cvtb3WGvuh`OmCJaMMV|TA3mV3
z@X8vYK6cP_!xO|!`8RUhy%ue&>$7|dd)NB(YXQ%PAZXOZ|}0f
zWuL)&O>H0k%`?e|v?XO~G}x_(A5*l=lt1~8A!Rf<*d9YvQdi))-_ki!(~noU{q*?N
zQB%FI)}qHgLV~*i83LD38z!JTmaibdO776=dm1tvVs`gCQ(9;YVj;f_!&M1UgeT9T
zkFj^O@B8I1M3Mlw-gi&u`!tD$R@(c*iE8H=4%9^=WHYkrn%7+}x8_ILU3GQ!^QLM4
zu-nAO`>uty*{nt3Y%8s+vJ=Fd6zWHHu6@sCcVhNXIFgG2`Q`v^_Q4jTa`{D5kzaEZ
zTHorqjn|pV5R*yBHxxAgb7PgRFIEVl?&G^ajR7JJiJ&fh88_S5r6^*36OCAMxPweh
z$=A)xZtq}uj!dcw3lusw2MG&_gRwQ7fB+n`yqUd)t0f5=H#^t=3XyCrx&MR+YR~Nd
z2$7Eegh=-j7ymc~{)M4`kwZGc64wTmBywOvN9*(IF};RL%I0Oiu@Z%x561beI%cvh
zQl{yBEOL00u&~d|3ne4Bn=ZCiy_e^DkyFC%&juOpa#bfSwM;VRefh)8Q2YCBOCG!H
zMmDzYOXQ8;dPVa-z@9rFo&{0s%gaPiixc#q9`}p7828Ku&t
z+@JdH?k!femLu(T(DPr263c+uYXqS*rLnHGD4
zO@rf&YEp|h`xJUo-vpK~d&2h=9#WANWxEO^`OUw#%(-2&f#Z~uLG;$@%V{hEej={R
zeu0jonZ{iG`xq>I)_cNiuGU1~ts~x*4f6oYHf9EX=sPzR6CkG*;NAoT21)R0q
z=Fvad|2j4BCooK}asM3bE^%02osaPl$a?Q~O53`Y(P)LpF0Q9b*R0Km$4aeYF0ME^
z?tTP{})1E*_RhE3PR#luh7sWRdth1k%871QYz
zr01U+Yru9BKjM~SaS$*LMTDxU-tDy(m>$ZQbQAHO2e39-rL)sp_HyyRc~dmr1sN#p
zd$;c=TO_Wmw62^|F{@=kH4740AtQd~8j7yrr1HFh|Hy>0=g+_QCu!Oh(La+jm2bSKTwwB$pk{%D4$aff(45eq?IhM%vR^gsY`W1|7Tc#`6R?eJPhiUp&Ja80*
zQFeo27gc&VKF5CT9}JSX+K1QoKb>w*yk}
z&b_A5;#T_h&M%4@5m_>Fr^wB&`Ud7s3D~LU)gF66yWRSE6?YmS0kI|=yE+9Nfsw>H
z8CGO0j=pPttYqO?R-j!VVrw7sIcZ}sDG&%FBZcXvhwyRcX%6S?(^iSRPDYgpEIU-=r=-kTxNRAv!i06JAtWh*aktE
z2&tr*#@c#EW<0MF%4WpFfI4=Qvey!Kww1wlee$xtmg;cxLhA2!yBvv;ry{%u_YGYM
zDAd5YV(Bgk(|P0{s_vTMG08Q|9;ME}
zjacgQDdS-)UrHjqK}0ka8|exfGnklFluKN^r+idm#eI6Qdl2qmil59fKA!nqQVX%{
z=r~?IwD5oD^`+=(r-cYyg_fudpX+vi6K`Cnm`bTVP-UE<&2wO<3%2L)mmNoq0*xb=
zfn#foer7HXqb4vHPIVdH@sTxqY|~sFAZaUPsY>03@RS@xs+p9!7Evi{d_(@i2#d^O
zX{y*2{SD?HO!>d(>-ii;a%`S`Y(Z?QHZpIk=53q~ys~Uip|f5O#7H%u&n-RMUh-D>
zUUsZgvVyxzaZ3A?RuQ}F(0Eoc0AqkmF-81)A3M3mUum>?WsE(}tqAiNcRQh}OWS3V
zbD=9Dl5N}aU+oyi_PNVOR3RnFKd+#7*i~hThs<84+hh#1xB0&t7GsXH;FH?Ko(GF7
z*l_PTeii+o1%?9V5xdH?Fe;GrlEV)w)BaM7A1l{kL+Cz&izKU3=Tm$fL#e=!QsoPnDPjqLmDTS>^P&&<+p
zyb6N?y1~Ucy(-VeyYPh-0MkAjb1lxJqIbbUVe(SO9)Bo0+|9dOZxmoU{=B4~`HsPe
zDF=)stQv8!Xp`-0kj(>`;HIol_%nsfDaQOobYp)V!=E{x`8)W;9D9b`VK4Gvw3V_p
zRaWKL3U#A(e#Jn8(+N<%fosvedgI@L*Gdp+n&FBH$;hz>ERD?9-G0@@L+k1zR(1;b*u+FeyKYDVaYWuLRNaY?aKUHN-?Vjd7tE96Kyo9QXMX+uor;qs4$Y%#^fS7O%5vJMt$
zE2-I1na?2u>#L6DILPm2i1_Q{QM90=xVZd$V)mA#RDJu}$|DJ5zYB
zS^_)(5&8v0XsQ}yc|uX;A%6RUs%7p&SKyW1A*oMGdx)>Ar+yOY(ni>orbSwSR+wJ?
zPBPoHfYbHhPq-Q{>;`#==Ehh9wf%4n%WtI~&l^K2v6+@v^QPl@3tCsCzxo#M9+epg
zkPS%8+Iu$c`}KJeSBL&a`b$A5<#3k$XZ$bO?*qp?NFTZh}t-JAufnpUAdKLY9_
z4tBlr6!*Yb+?@W!$d(eXe*PDoChIfl@LSXA>qDP_kza@p>@(Pj(BO==b;wKhI1W-R
z(1qXMyg@)TW&;M}6H~`$OGK{>LY2Z(59Yk0nLX#~B;GZkATuA>rk0iP7(F#e$6NPJQd^!Z*J5|IUF
zv^XrbA7Xrb1^{Bva-qQW;S21~<*(4mg;JiTU$MFH;GxIgw~DyvO$pd#n3S7*xvtoT
zI%kEGqkqu1xQ>2$e&$jzkVF@2uQBo3UEOI~Jer8fkhz&{rpv07&vk|=*&six1?|)B
zb(^ee52jSIcx996QO#wNiQPy}=q5R7$W2r+m3(~e4t|5KRmnHQnc17VxH_8|+5OMz
ze?XVx|3$oPJgol_F9Wd8h4vrq&gGDi+eOo;)b3M_LBN99vYZZ^$6xp}&U4Ls^AvCh
z&eSSlR?^7XKO5GsMJl9>kyV~mV(;(Z4D~->8-INt-EO`khr3uodn|5L0-@w@4hmQt`!8Nq$WgZ)ixhwYW=~I$s7A
z3@eT?dVbEY4p4fjX#`wd&nQ(WUnY=!G);;ql&C8?I-k#k4Xy}8^Q#rkbFE(%M-2kk
z6BGpI>*K0}C(k^|SE4LmPD13PlSL{pdrQcPf7UPk1^OItk!Dk{GKm}%N?RMfou$w>
zKMmrRSz>)in21J>$K=wgAml4au-Ph9TV@|CtngMUfYN@RXVY^Et-soyLdjuwI+DcK
zekN}2Xr)LPnSzfyMRrZaCWyqs*;)39$tfex!n&k!TV)j@>ViAiuxB}n_)L8*=dLCg
zWq6nYVbk^sMARZ?iNd^x=lX@+RHiCy2>nnaGZmZw+m@`F?|4P1Rv}d|&KLxpMuV|j
z?Hbw?U;&NCIO&I3GgHES#!
zytGSuoPBDQo~o}nQdA<`(|f7Qs{GM)szIYqqCqNw(|nuUHX5|H=;F#>lSfF<)f*B)
zI#!JgXcriHw^q(7aGUK2$QA5{Lf8Kwu1(uHNl{lgp>8tHR5}e==gAP_D8yE3?J6X%&=gALf<1yWY
z)`tqe$OI0^-K2uclkCrdArRs`6NVY}AYrk5HkasbomTwzp>bF$qJm>PL^#6`qE~OIAFL1hc
z+wAdI>eBe%o=WsEf(^Pa-?1Q#F6=A=vx{G4rJ9zBq!MP>9H;ath(tU1G+Qq?b^^Q`
z2!i52GlsrnS@fP(7W)?QyMd;5}OcHF3`A
zqp5*7M_J+q3LL-tl_tCpf9gI2ezPZgl1p#irY5%NIwO2l-sOObSI%0i)eu^@-;vU^
z%|NHJXGtB2Zk-{Emm0-lw=Vmt8ZnMTPRMA&k}{a6)2soRX3GkkWud
z53CRBY0ASfSZM*L-Df-rj5SYRVstB(L0}RSj1NGpME&1D>Y;dq5tE%z;`l|VRUU2Y
z{@&*bqCAF<+q;`xeKiS*K?7Y!MM}t&vOb6*F09LF)QBUb5zN!jo?n&_`PR+`tG6jV
zWa~GPk=Nq|LS%evB)8<#v*)3RMF#kT+-N#r73cvjg6ROw;B(w<*>j5q(!EF@RvITCWn%h2T+C?oq-t9mj6^n&yKXI4}WuTc-@)xzp)xGzbG5$F>W^7
z4T_!_Q*UloW=P;BTkY<48@{*L*RbeTO&+y=*k7YCFMM|%`%2wqg}i!dQzVjtNk4+O
zO1BcORkrf@N7wHJ+2deE5QWxU7PZ}S6*=}Ohyw7`Jy0ij1n@1J-my9tFX49?y!PGj
zaF)RmSLgVqHi(+@rrS#91l|rdf9h2~+EN!6jVrmA+0>8<&C#}b5G`QF&J*-Pm`bA*
z)rK);ZLBv7AKo72NS+nQhP`d|5N63q+%7^?gzMa{aEZ<+)TW+46k!qR3GS0A-F*3X
zz=2Z*tzhkqH6;e;@JDuHRDI!V)a)ux^}W3lChvdEFkFSWI;Km&l=%ZP(&tWR_IR;4
zNANkgmB7sBh%5SqE%SM(EN%x}>}Q*OXuA`sL`dwKcI+Qk{zn!iV55Rkp&#MITU{+h
zw|9Ix;hJds6LLxkH|76_NJRgak+``2AFdz=YvQ~-EikMl=dv+|+Vif-QYz9+hF}O)
zTi6)R{j_4=Zd;tNWRFDdC`|^&iU1Xt`ucdfdc*iPGtejgC_3k=wzJoYs&Sbi{qu${
z{u}7{{rF~=GPA%@b1#s&$!PJvc3|WG+JTLo!xyP14qHs6SLQVn0&jKrt2M{&t6RwC
zmOypy>+0tFs>7nQ#;997rmJ!m1*8vUdU2?jI;JLG3Af{k!KL9YPO?OxhDL+@z?o@;K#U?a8o-*1|myhFx*WE{wJ9y}rS
z9#Oqw!5QD0I|erXHkHs_rrjQVC~^860Ct!#?c`%a=Z;+uQUv&Fs$2hYh*`P@a>8rD
z!0)&OLRzVQ9Bhf#ZdW|~s^qI_&Y5ncSi!_UUZr4EYRRi&5yaPMocjoa&Q7BaKc|CV
zov3W1xz~!vs8>AH7FNnc?o4#C=$%SdEJ#VxmGU$?so7Z0XoA1IoF>3*%jRRh0buYY
zOH#|;Sd2oFyAPkBIJw+R_zjv2abi&%1^*J5PqW~tmZu8+LW&OH=IIpCKmToRE@d&*v5_C?d^Y({A=cJU%!*;+AbUlG2w?{>eN>pKq)#?D
z5M_S7@RMCJ&{uTS3$N4-wacadIQTMegviypBin+)rU4uRVYNF-pOh`2VUb%IO9>K4!)Pc?TXVLA}OuJF871SVpb&iv(mQjZ&AD;WrV
z)d8ic_(cQi@$*RNXN%LHW2w$vH;e$oG%^W7BjuChEd?=?+6-|`^(;+Z`0Gt)arC%(
z%Fh!^nfNp|4HzT+rR&TUA_6A|gpJzut#wclbbwxJUShflg83;C7ii7KCmqvGE1vi@
ziV5v`a&-#U)F^xp{1Z$--Wd?93n-sGPsy@kq-W*lk5G#&&(C8(U-X6U&uth+*ODPm
zuK0EEEni|IWdwtp1}Y#@OOy12J9a%M6w!m&!5IV=C37$(tIyA7#rzkCU)Z1#hNu*8
z#08ZFn7O1hb6grxFkrLmyx}4Mbz#rI+(D$$JY$j^zM{a+!@SjXITc3ycq`{+RXt-i
zdDq4y(ZW4MA=!#IwZ}$$sA57>AK+{lUu0Jj!W+a?7i^3t8G)-aKI*lU8F*9&?Bsae
z#1h~xy3c;@=zwA4H!9Zp;jR%NBph6`C|3qhF4g%W9=oS(03pXgg*#!nSKY~sVbI;!
zXR$d_@wQrGT4)4X*l><>u8RR;LJZdw=XTHca+ZRW)(OKhr1L1!4;(LCao4_nmVC#%
zzfTBSNtY+o=%e%zK>82j@8StqWxW&Rky)YQkT(KXAnonF+
zaAS{__Ck2R0$D+^FHM~zH7Y6tG6%d)AQ^mAKhIElK4TW{yOnXmw41(Bmgfm>TV)I2
z+R~6RLc8wfVnp(%=M1P+A>&G$aHPRnaYg4)8c2G^O~~JI*tkzZi&6|#ZCG-uNsVqn
zd@<^PNd~<&k{Vm57d~4x#97!+b@9960RyN1Ql2?10QcISF6wLx(Sd)I_FEQ3z<TeT6a1C(L!VH>kq7-@Eb!5I9vJWv6^nEfpyb>;uGXN#PZnv>*Az3{NN77
z46{HcyFdnK_gQ%RlLk~soYWIOW6QTWe1Y2bL3hjto#3WvsKF^eZ?6<;bX8O*Fug`A
zJF0|SakUWNbiZ7$aDzG9@o|gIWHhAGxb(3Z@G9BoqqT5RAF?9-<-7Wh8U~YDG*uYe
z`tkjK-IJ){qUJ`PJ^20nb2WoLnS035ORZk@ksxzaDxU^Tqdeai`+
z*NTKQtIa=^hbj`{;eEIS_E>%sEWecUFrS>L#qGtHBZlthn;5
zD(Zj4Ze1yz`vCb3Q2NhN3rch?eJ|$#hv4SHs{I+3ZlIPnz1te)I8`x8i0Fi)KeX>8
zu9!JBbNLgjN~88LFU!X^Wd)dSk-AgIX^QZRh-0%3th-fHN>f;tiuI&Ge;3A8{l?;l
zny87a8Be(@;!dgKy-Nmx>whkaDqV^Zab+VJSwl@-R0CxvIGb>@p*TB;^6n+zH91*^
zqL7?x_*s=i%|C^AMVxDs($a4|Ik0(Gv<6wy$CNTUM#EwZ1VUdM_CnEn5tUj!R)H>{
zSDBuXCXZIH+dtJf4@lfGwV9);Z&ilgD*{-Zz6ZFza!L#65h
ztlJHutuEWwewMd^`FzxLmg50e8rhrGMt|F)(p7Lo7>R}JIyVzh=HbA4MT(NSi`IVc
zYzbn?%5~B2IEgeLx|OvYsm{~TxdtAw1H!
z^)0LKcw(?(&x5%ni)C7*_tYe)^aX0LdLf@LS+LynnllfZW%hSc>A!rxr@7hakoSQW
z8?(Fn9xXgV0ZaimllG72ePnJAKD`4vrGg+PRU=EkNXPux)C_quEFo^k(@i$^AFG47
zBM|#5yCehj>8c+@csd$-I#nIJ5yD?}+l+-AJW(|UXFEq<d@q?$
zo7B9eHJy~oxzIVpjtiHcqxt)VqcH$2W+fP;b`afcLpzL0L4$51%z!Nx#$^QbClQW~
zm#+KJ%mktUa8>E#dRJetjR8@6;G)-*>09r#_)LQ=GFL!QIXl)^iCn6~Ufamwc_tyc
zT6x$U?RKNwT%2hdres>@9dB9bOs?TsQh1vs;8`Q$h}Ul3${2FS{0+)uaSYTM&0#QJ
z@!pi$PxnqC_%K<|^_2t?ZkQV;XtpE0^)H`Sy3pA%N3P@s9(?vfO-rjILGaDLIUDZho`PwW4uYTr`0M7C9;m_rrZ($Xa12Z(~2~P4(
zy+jZ*`<#8$w)fr2KT_E1MPnpwpm3?CDhC#{o4kdc<+*Dt
z$KyHL)JccP%`F{09l4Q!SYd!fM=B}bEH$4J4vKp~6)&ey-u3Jskq*Z&M^H8RH_j%d
zN61dp{x2)y%;ksxA=yF0(dX+YH-9==Hx}1(R5wXf-G?x?wzxbPXp_;2%0uuF6vU7Q
z=P8+B9C)x)AF{U>A^@CFWhlsxOaXFrc~7>_k5U5k*dgvz`L9H;*2^yQcDL4tB)4IJ
z-#zJ%F0{saPCB?@-7PW|0{=8V8rgpzu=#Ngd!`{4gK)~Qd!&YH!Z$>H9Yop1hRlUO
z$6mcraOWyqfDOMB1qv)+(
zKY$R0Nvgkg&krUej
z@#TT4#He}yZp1T%0h$c>V$vp;Nma}f2%N&@+~1=n?kF7)n86fJMYHPL&4@9ih5l9L
z(*Xb@`}BIB8vtD3CexH%$S4>8^qeFNqjY7XAeo|s&XNSJ%^rYi%eo~h0kLQYQ606d
zx0edAOMC&j@YJ|b(n&rM%+f=z^XYUJym*0gE(u>y2G7!az0LJVb#mFmXb5Cm$#EEj
zsUkF(9-bRogBA&A7AR8hD`L=nPd@i$riG)5^O8<12JoTcFdFJQ()?|S=oA0KtEauI
za&)fi-CmT5FY%Lbk(Oo47O&q+jmMgWMkV{-gj4Qfuj0)yB+v7eUVDLSkifkG1-y}v?_Htm>EO_tkFdPK>I_0hPs`uMnxx!-f#76E~+
zd+oA20y>-OJ6l8-IYcpNgrhe|6!f_Ez5!ic3iM6FJ?vn{(wA@guAdcSag}m5Cl9P)
z|2c!4@M;Y+PA*nu*-{5FOWP&mXjE9p(j_VN8z=k8CYXJRLzo4}@3=9#3LL18u+Z_vO`emCC3nNH)IKU+_b
z0o({gCKqcW{!oh3pwJPCR*g@-M^u3dpXl176yBNSo@zpOWw891ply;~#ppbA678qJFCvye%*b9r;p_&s39e
z|8za*-(NwDNnNfHk0mzcZ&d(^02{|&1&rSg!@~MGi@pHLV9#`UB31cBc}tH7s`__?
zJp=?sA`#1qgNMeeFYOh-cbdRu-kKn5+t5$GAcUeEkYy+{6tK@e3_9F>7WgZC1`Ez7
z(H-?)TS#0_^U5Vyg$kWqhLZ`yDs&!;!^s03hqs&ztb!Yi|Qiqs13Zi!MDv4;0X(X#A3
z1Qp&B`C+w=@yE0b%hXUsV93M706e}|k=aS%2vuA&ie%}vk_do842udE7Jr@OV)-*j
zF@<}LO5V~k^)8GB!E4~uitnmtN1V@Kpve{PDSN=&dA=!IHCx&2EDl7jhp&2qs=vR1
zuGS?@+gIkQR@vp=tJPVuIij_h*}dzXLKkV!NjVoL<`@rt9t6|juns4gR(0A?|(+jgMMx;;+eVc%5uX(W9n3fmwV+H3-8t(RrbY
z@%9+`qo_RS;}E##VCY6{nzlpy+t(;PuanMv17oe0$C5zlKY`IOzOS7a*e^mVq02zj
zM15b3049tnnl?VoBuIEifT*h6C7?gz)E4DBo@LGpPgUFh^H)*=ncvst=b)7BqWx{!
zvoIMemaAVa@fGl7!2VK^_C}wOWnoD3J|Uc-Tai@2^D~?mPwcG72!7|Hl2}~KO@s8`
zzy1{T(H4*atUey_K}m#>-cY{mOI~;=0^D4)UugT3UX#x`Slwz~K(#M4D=D>x4RBRV
zd`RTKEp3G8>b@Y009=hnIY)&`S*P`qt#IB;8yDYhqn|Hrt2Ea!M>It=bBG@2ygmI(
zWX~mFo1%KOP_*FT<%kZP`c)8nfa^Mm9*kYL#IggHU-PxMl3@L6JENRd$usl$^GGt|
z+>F2_BL6NUDJZ-UxaAahbc`yw$%bInj|0ZQae@8TP$~=YGW=xX4YzfPIkCbaSo-)in5Av=A?e0UC+(4UG09u@##10VYjwi
zMJ!Jtu6CoR(!I!R5IEt45rvjpCRG;*^LLMBmOlny#c^Z<5Z55#Glk2Oal?7p@?VUO
zE+5R2^i`X?w2S!cKJiV$B7TN8r2n4COq@6b#J;`-i4=bxAV&IIp^Lh&dl5^Ub9U@?^hlsn=t`DRnw6wj>$T$zqT1h?@imBouh!6S
zp_HDZ2Ydh5eWIOLi|VktRHr7Gna}AfN?JT=hqLAuK;*y>RKgzPRAFKxt_?e4PCj|T
z(BgX`;)$~F!RXlnH9NkF_a=)%sAh|Q|Mjm>K0
zLI1cBVH?F^$5aNeg~%(TaynYRaalO~?93Y)Yp!4Z%UMa#4xVAK%oc1ZE#!K9eeQ`#
z&ntf?vZ|wF|39o!RxTcv{~e&*)K+lW;6mws*SwKdpvU;d(8FhEo^krzl+B@{B!d(s
zvC+gho_x^p`mBeRBrhE&J@FvFpiA*{a(M4@?^1_Npk3bcZV|Uu*g$cn=M7by48leB
z2V1*zP|}oTrllE&y!HSd
zZ8Qr?uVZig*DtE-d&L$H?er~1Y>x-T1)p}N6Z4L+Q?^`8^7xt3*90#Q(UTD2>2%Q4
zsA7|yD33S2?i$;*dHcAFw$_&w9cf!_$C5G5|0XRT%#}0LN%2d#-d8dsBX~_bYOJWD
zpH}N)2Dl2%@JK;LHI_U8^XBA|?X38gCRK!{5(!+s2Mtm2EYuk3c&;fvzCdbT_I!3;
zQ&ZNd5lMCy>5!~^EZ>j*xVrya%5}?_Oge@?yR-Tev(o3%3D_Fk*WTI&Nmc&85{q)A
zejR|oOyqQ$@Z&jW<3!!#K;2Vg&4Hc-kta)UZvG1}qo#1n!&p*!?aL4uQ^c0WcL~Ve
z8H*ndc}+YEam#6Tui!iU!SIcGb~rQq@LUv9K1su5`vK;&_!1&*T}D45QmRlOJu(K{
z)H4u(_KM^9s_=g}d&{WIVy90KcN%wlaCdiicXxMpd4Q&IcN%wh*QRlIcXt|S+?W2(
zyu0tr?3dkqxs_CIB~_I=C%LEcyPW4IpI1ME0e$OW4VzQglS0VqK8A
zeL85odddM;QX%dZ*4;Bn;e(+Q+V#$h+5|t`b{A2QrVY~Ncga)(&2>)G*DMnwYE4*f
z?qlUPDpn_tk&+sJP0Jv7i3rom1S=0{k(;j`_<1}$@GF+wkuC!1lcp$;Qck12!Z@U2
zQYl+UGaCiFdA`9ZVheXFAhWYJgZO7}Tb!FPGk$kflLMNuR&Y(Gz}nFLqL+0}ogVMg
z(`3RAM+;&p^BKmmm;tCR;3$zW3ldU8DUke=Oo)C
zs@5Q~Y|H}TiK7S7h^g!IU1G)Wk=va?Xh(j(a^5CNw_N1i|dEPDv2$m1|HG6j&>9dmT|+wbPjiAhyl
z0J4^Y=x;BUlCDkKdvx>|j_w`xH;eFNCJnViMiOs5*TRG0q
z6iQ~hI(=a7^NGjzGMaMO4FY*IN)ocGeU+6KH$i+F+2(?GY{!5poZN!73mU_zNcV)v
z!pb?%?*y)eJckm9NqCE-oRYjn#^2o88&)Pq=GaBQbBrjAB9xL-NR1{Vsx)&*UE&p;
z(SP*+{jK7r@m~MnCdjz7%7U4FG%+W(Rtg=?Am0l_oI^8x1Ns!_vt{F!77}O$E|K_H
z%s{PJ>3Z5<+_Ik3+<;0YR-iR}zM=U?5i6toeiGlN8yl^{mtt&K-3wU6R-1k|>c|Fo
zmN*KFvYT*~%o$8h@h4cKvomdDWh~cypbUybNRwl0K6t+{Bc5^`GN0O)3p==v4VvJ{
zGFJnm)_WsQ`Xeq~xd>pxJ+JDPhZ}_UlS|A}z4M~-6h`_DJv|{poFi$=&{!5TLg^~u5_&n$cee#OFB?k;@B
z3+}T+fNDVB!c$r0FoAP4t0vl7C8yYrrsSHrC@MSHUm$$Z`bqF|ouyQ%Y3eM$s$9J~
z%UJvkat`r;mmhgk0$yk(t)0m;U3hV`zW_aAaMJS(?m$u3Lff7KhaQ+WB){raVA8A)
z8Wz+4OD+^EGfPsuD-BRb*YTnSC1ARCrFBGVA`(kk<=fwQbm>emei&46f2qHKtGO23
z>Vp$hqQ_;=P}YS=+siXD`fpBRF6^RH?|5UUf1K0)d^|{xP1jxtJdd}^{3${=Pc7fv
z>{qa%chSADONB3P+}t#NRQ>DDzVMoUzpiqvT}I&7(uB^fyNROV-2k{K6iz>lW3
z(zM-UHg#G(S>;|};I=pLeS?R!?~HHjSxSdo#CrLgo>smrL6U!Sy*lif;&cmcF8=9p
zzU|LH5*VYhn*wuluG{?EB-7sOs|3jy;?xm8&WCa76m)utBy>UtA4fELeG3IsCBoMl$s5JP#G2r&_4^3={EL;Y4TMNnn6S%n;?HW(x5=2Pk9emZV0ZUM1
zeh4K8Oqavjq{*bXh7rvEn(wsL&os@R{qEdMNtt%4(NsO8)JtG^@EJatyw5W=P?-XI
z&bJ}5({AiHqRh?S9`g2sbI{n415&3N6(VL(@>x+NNf)s5H_sB|2Pt_9i-cY?j~IKn
zT4|%e-i|(wu6(^qh{gzd^i$->X-gA
zC~k*3C}xB*WeWu6Je@#OOpv1?rLFIanJ+!STzwV@mqYclW^*t@)o{IvY~_r2^1|F-
zynVE*U>^ugJEDCQ;FvT4!~SIoNol1SOHl*+zhlI>_*rvPbA>J0ihCPIPo5sdLqLlF
zsJ%*b63#UQ<)D8WJia-&z1KSsRA`+Zd`P4O5Me*&z5bCho%Ro2;o95TvPEY?dH2c#
z?sqq_RX@CblY?rXEfm(eiOmCR5b=nYs|+m%g}uLj58(n+thM{gD7GDIN9)%l0ag^)
z29F2v#Jw0gm3wG#j}yu;P`e%%xWh>5nSu$j<~DKFud~OSEevWwt%5h-wXi?m?2iB-
zb}9)k(eD?H@@IkBql>ccA^S1vwjZ4i{FokqV*q1jjgNfGe=qwc8-LA`d+qh1l|-of
zb8G(~Wynk#yNgoR;F{fPHDu}cGG=H`0Rt>G23efccCD3dyM6`=Kzr^j%fi}c@lC=?
zCpvs33O(&Yue!BQWDqC>ht(io5RCl
zgy5weGWCpd5K)H39w;pzGT|ODS*U;OCWgjlp`SYF1+YTREqJX?()S%Yl`0#fD6lWL
zP4P0y!RF4S=V2p!ysTQXfn?3G5~z=nzHl@ITAhOKgOr)AiPI~RI`Pc_$G|PXs5+OM
z$znz#pH5CgJ(7bpO%t_|MGT-4H2L)wR7-x;<7~6G!@bg$*;ldRtKa33{f>Y+(!E#i
z*GU+BTaO8WiMyS6@f?tGI-&X~Tmw
zKtAh7s$EQ=PJ`Dmfsjuou$Cp6dY~+tYs@{i<&Bh1eL&?MG}UmYe~uI>&a>&sxOh=L
zFsmuH#gtU^FMqd%U#({cxxA)=!w;2P$PtNHj&=-hJEr=hU9D`;vL7u$#8hIE85f=9
z+%t7Hp|KJ%lIKi9S+kw|PfDH;kGyT)hrvHuRTU3QO|h^Eb4wH2foRDxQAMYOm4qUP
zTr)_7$NM>V*ky6fuejrX0-A4UDB*6K0{hqV0sMXpaQlwQ8
zN$Kf*u$!&))L0M+!F#PzALoOBHtoL?NBbB4aOM$VGIycqWtIAWfW1JZP;Bn=oMr!D
zo>9&ur!M{A{qwko2K2gJcfBtTCq?9p1$VDkyx&*dHwV!*h8GK$Y~wrlUZb6dU0&1rV_#f?|HB?}|@r2vqLW
zgy3fo<}WCHD$a^{l4ZMSqmH0mC*J599^|Zh>#m&~2;h&0Mo)YOecxV=yGzdE_Lt-z
zQnEOX%T+$>s1BnOKwoWsvg|$3$~Hl4<1ocA+EOOe2K0m9;oJ3a?zbi`tQjMD1Tmv4
z86`sz&4jqPw
zA5uL`)ly6)8_cV1fv6)Dw=h>zn^cYx_%o=|NjV?X{i>k850W&{nI7p3n(rM&TJGS6
z2Q!pJW0{t%3VxZOf^>ZbxdNf$+F3P-)bDd_HgQHDBEX>i^QdoS^Yom-o`Kuz>&R1^
zaC&)lhkDJ)rBiajd5w79eF;B*dza)u`S#Sgv)z}39-|R5L?%rYx4-XVL|
z^4)j?Oo`JK&+%(t_WQToz|(FK^-4JX%s3GEWXo$I4^#*;f)TWa)Jg-z>|{Z$CsLDW
ztk~^+hJa@xbPTonbYzLQu0;LSTtj>GZ4B+ygax>gJqa1S{-)?fT6MbaLO=HJiiglQ
zXF_J6QAIgwAF&)q&|56EQJUGzsW_QVN>KRHhVJW}(a67qQye{`g>2A#WVypwDHd|4
zs$Fq-%+0mdp-e8yvkCPzJ(*?-kp$sleC4xp0Of3DK=ozhARd~OXn#T9hHN1(N>jMM
zyI%PXhBe#ZcADO6T~!X1$~gw*Kz-a>dsjxj8v^+5O^!AO0=MRqyQ9y`eT!(dL7-t|
zGGCT)iZFBqa`(cN@fwR=}PScA=wIMrN^
zfuAeE_2`NG>haEf^{@yjqq1#2bChDSOyQ3C41>V~FAggf2A33wxx{{peb#>bk+kVu
z*WOqA%d-bF@{fpdzk6PN@(#e}S7TYFr?Y4RJi3}c>3UyPEJ_h`H<(8Nx30u+@Bs4k
zLBr0p54LnXe0ePBxr#O?z#F8$Y0nf0P$bayEsDL#MNzYlsp2W^yKh;EMbRL%VHY1|
zw)+miZ5l~3k8hfpsTJ=yL;pxZieN6OwrYC3Ly~Cgr31_y>4Y*j3|BX1JdSvESdwa;K{+I3Z$4jChNJ&P
z&_XwyFz%wccgDQuUamufNPc>eSJ~GA~Kg0OI7x
zxd8^$Bng#_!A*;s`MMcxM%dX#Vs|`kPIyCeJ}o^M{Ri
z2e96%*L@WwvzCPRk$$&*`}&ij`nP>CFEcl3-KAt5`qq_cf?GPPqWZ_`;op=wq@(0W
zGGR-|Enk4eKl2U8rHt9M3cX@b_u2b{>12w(VZGrG|Lqfi8fWsi$<ye
z^gKnnX79?lS(rF-m`{)EaM_hA)k*i$ed`6bv)~*UIn>`(`+iXWw>_w9t&SwiCTK>v
zq)UQ`kjPP!-;avkKL5@NzW5E(>W&A7Cj^{8GK2m5n|(4|yOc?Uvf^{@m9_biHK1$n
z5Olq%XVV^j+Jl?Clq+WM0K%+uGN^&%5=a}tEP_hOIapzI|DLboX|A80x7d2#@6f*)
zR*(st$z%2_2JDf+?nO3ZFKYWMR?lf4TtT1Cy@$V1hM-a}FHV(N|EWmRMWKNw%=FO5v
z8Kr|4&TZsXDN#lJO#vP?Q51D9ihJy@VrTf0;
z5hmJiT6E>eB?`TWD8O;8WaK-nFW{N3-fkg%&+2-^RaxZrIEI
z8p*yS?4$g7+jkkXKUF**?%GxZs=&eTWs+@Z@G88AObVOJC!B!^Zz@ODmEY~^!h8de
zTmFsNGnnp$SvVcLJz$V$)dZP9ok~UqB!!`N(<(aoh?zM1n4OonB~S%nVx&cK&DAWt
zlH19WYl<|p6DiEN8mpnhCt?9^G)JagTe36XWORn5ep09AM<~4KL$0Np`x-qvjFq~m
zUUwDf-ikRO_m+&p?!b%lmUEEigX=9XEcX8~$8?1RH+e}n(!a{@G|EtLi(}CNG7f~j
z2oW@Ha7!m*RRIsD!-fNbRM+sxytVEofdRnAr&0RP^`I4pY?=IVfkRbY4@jg9
zw5QPZ*A8mzyOKa7k80>?WqKy4-x}+Ozt;9X>N3fd&FT_(&|}kXZd0{)yX>n;t65)%nYWVEimK&`-(UXh%tF>p)!pb6W+-%{}J
zhixJQcm#fQI`cg%A=^e$JnmA5W+5q$ImSq=9}-AWFAH4wO=LQRv;FRVVw+ni4|h9T
zeT>m1k4<1}mc_Cr)i4tQVxX)ogoV-TYA!SC(N$1Y8F)fBoQb~%N%A=6r~_5-$%t^c
z33p|0+HC>cl!)T9yuv3)S7H-I3D-Mp6IvRl+@ha$S}fR@5E$r20E(xf>Ct!mEBU}W
zw$VfaG5GnlYs+Ch8zJeXu+-+97dNqBSX;}!TnEH19`rmNtVRf+m6$?=58C_6A(B6b
zGH;HIKSJFuQ!DqC6FyGqbfS|r+~*HVXBO7nVP%M#Ipq9NHilT#;<^EIr`*y$dlX3y
zg%@g9At#M@ghY>qtRONpU(m$SP@LLhQ-u`dWpSwk*H!yza-1bGZNI+svp#`3@p^#-
z2gf89qG#dlzw8k}i*xdTR{9e95Ef-r-B6aYR2;SQbcKK;k5i;~E%M~37YeRubYbGK
z;1Xf0)$rMOyTj5l8kDv8DUj|0td!`z}K`qkvRty
zl8(;}8CZ^kzCM8FlLzo;<$A~2Jr{a96w4kJllU5)5@+zM4ySTSa=`8q10{@#%A9~bCsN{i
z7&ZFGo%hCJYSrfW9fYkBR9i7&+H^7aIt&QWn%l?d&W2PHXpTl`=n8qD^8^BWa!fSI
zF=9RqJNw1*MtdRky9GG+0tP|}5}8i&?D@cvUdl`szs3r#>1Ef~(*wpE7-%1?($<944U&m-O4dT3Hz1GzU
zLrhEOzO=!BwK%k14H062m|e&6=;AwfkV`%x&}Jzpl9EI{t40(jL73SX|ICA*AI)NO
zNF>Pe^AXIxhb!#1-UJzY1f%FzQYBuLEW?B8-U}1zW*-zlVq|nH+uA)RC(ww9N`s%-v7;ppV$;yguOxVSu^x6ryrU>{u5VAO1dGvwj46<<>^RH%bRXGPE^?kb>(2mSzWO8`YLethfQ>Yu&xME$@8Nr=!sG0#Co!jvX%{uzW
zN2sORL_9}(MLNp-V(hoy_)$2?MY+;3VIfHh7N}tPyyWhFqT)}0LbvLLUz>UL0=Gpi
zzMx{cxB9w|qCFgQwfmaDn1!Q|Bnm`WN>6S?N{Gh!Ogdw_swPWE;Hx;F!bvj-C)a5o
zNQ;a5>BGhTJXfOG^H5-ITXU9$l?Md8)5tebzf*BUV-7XnMAK%7MoTqSe>X)Tx~8&{Q{GtTPv+c
zLr2S|#NO2g@RcpLZf^eCVXWznh;p>3Ou`(i9c!9!SCA>nh@A@*po*rSM_3Br3gj_t
zB{-R_<>Xi%wT4kFg<7Y0KK0+IRiA6#;MZ7hei*%4blquZrrJkNaxn-Ho32gpx;@P2=_Y45*HVZaQ}dA&QP48yNFD}F7%Z}g`LN{|W
z`>2smfbUb8Ccu4$&wU{={F_T3fC6=!&F+N0#K;|nq6NM621~#g+#8v{OF`p3Ew8df
zVS0ij=Viw4%W+Tp0ne*lef5ox->$QTH>AF{)h;e-hs>1<9B3)RH~i3yBMu$F%yW$=
z)Mv<=%O3LNvfA(bg}#LSI>N3IS6cFu-eaC#
zISygUr6?$U85bRbezb}J?Hh*n8>brT>pE0Pg&7QZ4r8XOPX|K@
z!);SYR?EtzE^y}Z)jgwsl_z9<4j1}uiSAh$Dk06DHc?gRsc8M;Ow=l@1DsRrvr@J{
zqaegC^EPtU9XT#887TmMJ)wXtDok8$*J{La}S%f~RRo|7TiD)U$>>K*h
zZpZHRjMY_|$p`)mkKWByQQi+0m1KueX^+T~wkC)JcOc?mP#oC!^FjhnOQ~?O-99H>
zmF&QDb()D^`ax!F0TF#L&3;58du7Qf*GAc^g0d$nulrl|y1rNDC2He3G=WPRF8!E@
zMRlgtPKj8
z7r~IE<&FH@h_vK{P8MqBo8OwsWIyv%{|c@R8*yuX>SZOVT%@eP&7*m>tkjC>vTQ{U
z`Z5j6IGrNicNb9?%PVZta^3z`#?+M<;u%)(kZv#l1JSvTY-bT^^$gQ)O8UG!><5467g7k((F
zI{+HHaa*KJ^xyJv{PW;YiC&Q2BLvnHyVTKb5yll$RAfpBD({vQgL}{{WF~favmcus
zn?R0I0_=>0esJ$s?T3L
zg89Fu)_hr>{m^3x8}Ci#ZOg!A4D-i+zrRA(C4Dod6!xtSs08v&
zHXB9;CL~-qj2r!4=H6&HiryQQ4lKb1MK$DQ;R(Zl;}rD>>@$H~2V=4bCUfPQ!N#=8
z4j2we_m`OvKohoTL_+=^*E(i&B>dt>JzmX&ox-Hie^{@_vKIj!h_9H%ut`T8$w1#q
zL+FtAjVTCT{+sgwVD`@4^M%iurx!SYo0r7dML99;EKs^eEw5aN1opzTPFyZ^f=PQR
z<3Mfo8!J%-Q>Z*No!&W+7K>sPuY?T}{i4{(P-MpSQ0`Q<66+#~#Ew&DrdtZVa?p#t
zt4g-J)v~QjZcZe1AM@)%r-WVXfUwGLDl04+4#2guoTLvo9l5`#NwW_kHVwFBLp>o_
zi)R|iaH7>|*7@0|!2#XP=v9|folOy^E5Z77_|rob0v>X*P!Q81y8YlMvc~!XsQYj7
z`)kHOSHjtcJg4fK{aCkD!yD~YOX0;Zf0SVmgE9ke6egv}1zj79cH3$G3Re$B#NmP5
z;EWlUo|*5tPEwYdzw9T+SY!b^uho8sZdf}Hod(LG%Dy|@|Ao#^!es6o3j0p?%z8)A
zI`p*pNW9Vas$*Hcj77Uu!Jb67hT(+bDLbXZ5WQ=E(NBC$Ptnf_gSBNNpmY3c#?iTC
z21ED5yW_-tG`4+r)Nr+kB6P4}GM5_e~b&^1R3A&&xxS&}!LS-{q$0
zcfv?-0d%3H5zIY%)>{r>(PW|7nwN59-KLCKVe~z~OT++l`Z1kWO2IqRLc<1n8;8!I
zx*hZ==_@>%GZ>$xyfzhY3vcV>ov^$e1&MIr#bw)zA%MHjBVY*BSvr3OvvvA9)K%k9
zyHXiv3qHe!MN-b$Xd}bR*7Qjg%3cf@a82eO@VRb#x!n9ipecZ<0d?>j>d(d4exMj^
z0#6D#zf1qNIbk#gNr!)uEY21W48gSjJt@~q711>?xgoS9Cd!I^Quq1-xA#2fq}cr}
zw01FCdL&Axz?!?RpB5Rws%2NkV!?8z+6!~@%|XxU4X&oGJ3*O)9YGk}4d_8Wyve5i>ic>gmm2-f|%$$uc9Av
zH@Z4$7OJYB7g%4q?b#_Pm`^_uW#ABgr$?y&B|3ky1}+tgG5$qEGt5>pZxTWO$M7v@@*p6VwHK5&7RH4vd*8Not4`C~a?SX%5Gz
zX>Im}m&we;#01AEWo~V0=uMT~wWnP0ay}V&)#!rsgV=B5;fn)^={@E?-TeYGUT5j%Ma?0s?UV6Ts+m<#PuF
zSw=!y0tf>7Rp>zJKtMh}Kukp)oxENCGrEeo7eK<%QLx|3?k}%RKe~RX3ouot^7HHvE^BT^${a?X2Cr|J7msOHF(*DiCf3M}WM!
zi>Z~dgPFCfwSy%CK-|pQ&C$i$*bboXY7S6zu=8dBeDxA>bhHJCI(mFrUfuP}v-n_~
zATa;t$i%_Sjt`~;f+z3jVQudE^{zRiintb#QPs-b?aNVLEvB~kV3z+~eQOX*S$79Z
zfP_0h>1$+T04sq0pHaUQcKF`~{cmzyK@k4C2l2t&KzRQj=Tezc|3?l$1^D0G{2v9}
zzqIkRcC&JHcLO+?yVzU1x>`FrFyMoEft3IMwgMuY{J;5sDeVmqbus@k9S`6?s{jBr
z@do^_7WrQk0Z{$dVxUHZ``>Kwe>D70;~>Hz!r_B`ttJo%kk54xQ4lByNGM1MC@4rM
zXlN)HIAl0DSXempZ%7EpSm;1(Y%Fw4Ok4tTB3wLDd`wIL9e|XAlA4AZhlrkuo{EW_
zikj-56i{erXgC-+G&ndkDqKuls{haBvkwFX28_a0tlHH4p@FP!KR=aOAI+JJ&+FdJXLm
zY9tjEziqJiNfl`Pm`X|FuY(@|buh@9YSty<407<``*7;=SzVyB6HX&g
ztr^sdj~g0Oh{YNNaSn+`-z3MHsG0B72UUiRZcFGsRY<&|tH^8_dAOA0#=bzLvA6=R
zocG988?=*e=FfX)aaM({%(Kr{6fHe#b3@Jctcb}nPgAdsqq}_9i4xzHX#TGF#{K-t
zEnzEuV5f1!tn7lvablcS@@ddIO~n-Om|ldryl?uh-HcS0gz3Av8;Nz-33}MZZ8WPZ
z&lkYm(23P(P20Sf3QlM2#u_b8MRA9(^gFmtc;-@AM5(-_L#D2}hcy*>1bV*(q5WDR
z`u8REDjwBy?Xr|*e`3rHA^dphu>jZ6)6)9e(7WZ*$!_d`z%A(D*DW|hwwF{OY7${8
zmIL=5ET}g1yx3uLv)O=SErA-%7KN?xA_nphB7!Q}Qp0Q!{Yf+s%+0G7H}&1B
zKpia-QhqW0jDhduJNR$$@{Fr4Sc7rX0pI1^&rL%=Z|z&%Rns|OS|pJ{-L=a*IO3Dk3ww$eMnGx$_V<{3jW%0*hE#2?
z7{l(9lipQdh6Xe5V^Ty#!wB9Z-Lj?vA2>t?MwQn=E!;^CDs$q!C58AjnG>e7G`buT
z3t6{Z+>G9xy2M~JhIC_HHoFJ0-wodzDtB~maUBGO!yYPUqwbnPxsY*y`XL@3effLt
zTYqf@O_8^}s3DKOn+TQ9BQ%MJEp>6q=#bg{gyHI|r^)+JPeJvF2}1m`
zcctGXS-Nz`*0v-B&LkR0P6tHe7*f3`PWglS1~8uH(3lGrzSW-7d)JB{yPXRP
zo>c2-6MAm_B+7uQ%{g)cR`f*K!(X_*6->_=OIAzf-2H5tb*7Vm3kCFIL4{mQiq+c-
zA3QNuNEbJuW43JMyYo;rvj;Fvy%WnFAB)@*>dVo4fTKmtwr(zSxK82qQsCJcQoj!T
zrD}6H*U8wiijyf!;Rrv_*$)*t&?*Bk$ia;__x(z%=r`z}uiSq9Lc93wcGmzVc7VDs
z#4m)Zhcnnej3oX?!1_`Hw|ajii5^3Ht$WObexy-)sXDS2hT7FTE|;sAtqZac$E3{1
zmEm%m_=L~ELbLOha7xosw6;O!l6d%SnTOY7<~X^O5IEA?vYXqviCl_|T%TQQ_lLB>9sF4_P_
z)La3p#&7f>ekxH1HGJYjE3*s+%1zf>gM0qT&GULUzSY)zQ_z}#Y!IE_`kAe(Q*=vj
zJ@uufdo9ko@48C_v(#>=)eCZ~8OclfT_a95$j`?;&*NLwz{0|sZA+?fYAr(&#YL}Z
zU`_KaQl9r!DS5%7(aCVU&2V1dv-cSrn6yYFBf6l6GWAGo$y*R-$kW#
zGxsoST4YKF6l!oHw#c04<=EhQ;)%W{FeYdq|&FP
zJf`3{$5On*UwZ3;%=;hh;etsHKEHU97glA{jgcXdv^a$e`
z`Tk_j9$39Q=3fxdva%iJ^VO>on*W8s7Zkz9F$Iot_Rn|w@ngeL!2GDPvv--7969&6
z?rD5}79=1Hn>?St;J+ID_^!AFOO3i11wU>1&O4cM(8%G2Z-q=-oO^&XZSn9y=q9;B
zI_a1-(VRxIabaD}#_CYcahQorh7UoaLb+?5kXhxlU`j*K@f{}|4b*Owe0{p$B7c&3
zTpOr&7_a9k)
z5t^j$--{A^$vA8mgG;7sQYspng(nK7g{sAn#|b4gX=qXxN0*zu@y#`NUE`lOqX|E{
zwu|1TFY(_wyM$b}EMWDinRi}o|miey?+_szK*8W*T&{#T+zz8-3Qp&
zTEkl(4XyeX^W{9E1O&Gp#mAM6?nbZ{0>x@eQ;t;6H{kiF_EC|TY@)#o(e+AAT$M>FfA8TO2F%|ryXUW
z?nBiim}KuNm=tDO!B1a4KVKx#dY{a$UgN2URWKEBV=Gm*EX5;cV}+S1O2Si;>It+;
z#=m;@2ec2)*Z=4YA!4}K`5AF?RD29Xek{IGgkNNs61w#xs<`>N=dtKPA!gp_>$92@
zJ$&r7)j2Kj!_js#({g&5{Y?1hk}bvBiqFzVSoE2}c9;E+v)gN%)$i8$!s4p+X?Z!6
z4$qHGKov*sE-zT(BV;!`ePxEP-%@z;G-habgO*NO(IK0GmsTP%wlqel@iV;Eu~bAw6dB2XXL2HTVRfpN_%O5zmrd
zuLYJK-!<-frgnrP)LETF^{W@?hyTc*1Peurze7DvuPJv${N~71%#8y8F6;CC;i8dK
z!X?MC{E%
zDYldp6%T3Qtq;NYt9-#{F6p=Yg+63%5q6pPC8L&jNehlbwyIj7mNCi=^u`-}U#3Q!aFS
z@g-L}lX3PsHOZaYnwzZ-qDGq{p&{ovxML~VBmBf~jyK`aPMDHahVSpre|*0|(WILormF
zH?I=}0y1NT&vU!F!_oB?1G;@VnY&3zh&~GnL(`@R*|=Jff{aR1vzXnQ#7ar}_af_)
zdw>YJq|y-@2|r$dpe?CN2RQctY!#3&R)>8C+hvf@~Ek3p%bBh=Im9O`!
zRhTME3_PdRA^Fb)V3E}~e>YlP2!n`UbVu52hkCq>@&u9bWbAyQBc@?Sr+<(N{`h%r
zjNh{fQ8R_dqg+!mAL}h8;56Uz8;#O7w2kSGW=Ke2cjJ%aXoHh2*HWQlsvB4-Xs8W^XVOfx
zwq_JKkWJ*J0a3&bI<%+F$tpD`)ZAJXyx-g>ubf>h>&S4+9I)#0_(`5RoCXXIX_RwR
z6FEwStKAo~eCNGuO?G*6OF1!S)w>Smx-Dj18xi>ldfhWDu2X0X*{-~L&|G{tmj1pJ
z-eKMHQsBxTHi3=$>xe|9=FzFBIEG50dh)ld?j%sb0+}?T>5n5y6^m2Y6_jSqlbBh?
zG{nsYWPA{2lh_XygH2`&!Xv*GR68w;^UIt!l~B@@@1>4U+n$@gOV#>JhUA7{1GW5s
zd^fo&6GbErJFcghVW)l%jIs^Bay@YgO-QnFlS#8X1CVIT+)BTX%lTT1Z|`R&vGGmP
zai}XjfR@}g_?>j;2Fm&MO*!Nst7`-Z{xCrI{J7;e@Le1k
z=}ADRzf=gWnzHz**B9TrS~ua6Dx0Rv4zKsKKWD7zx9$8}+r>T2+b76!89%>*%kjW<
zz$2Z3`tG0yW*58i`J2&(KkkR^!Zd|7ye1coIbu86&C3
z`7Jp5k0RD7M_rR9_}Hc^%L+uDvX8QfTqGeZ8bL~1W|_>Y?0
z{_Y|@wA(6wiH|=z|JLTG+$9sBr*j)z9VK@g4s{C_5k;C9$+n9#&wl$uI)7SeuqA(E
zqy_Zxo-znQ4LCKhX3$wvA9rW?#FSsdr!k&
zm+hgfd3*s51Z*DscBDT^!4zER0p3wHbperP-(-hJr7m@(&Ly0Vf&?X>8!3P)H?LbR
z+{uf#6Fs&57YX*uV}zV#gbuGnWQv}nSH8r4$J4LBa+@TNXVN3@)5cB)`$(;`E8&cb
zFSQh=EW+(7cvI$a6BMkU-(n4N)uy9*95>_ok^^pakE+&RMgknqJd5817jD@&bb!xc
z-HKY;vY}AWO@(|*SA}hUro7wU6slpXgsV*Uk1oe9<;0_t8n#wk1I8yypCAkc0yKU;
zIJAtjDTxM|Bx|idDk4ch=i%lz1`gHiwFp8HZdJ6=wgePZHPQOgFA$H&MYWSV?*9$nN9SGsoXajqtif;#pGr&JJvU=+sYz`0w
zi}pXlWEizl5#(T-IFuy}KBmCsck)m9Y96a=4|ez$o^;92S?YTi9-SiAYzN6*L)7oE
zKSI-$f$a4efZ`JjytWJ~o64e)5-JAKL?MhwvP#*A*g=NHOB)#qAVG7ud)NXVxYo}R
zw&L#`=N!u{c|2-J>OeH+Q<8e{MR-w9{6&m_uv`1()?ZLshT?z60Kd}jH*ynQeF+-#
zBVrE{6`Q~WO4|uG5&oQrt6NDw8Rq!IA6MncLA71__C9CxPieh1`{P0!D|Bj4X|(Xo
zp3rrw4-Y~9per(KOFqgknjmE~VlkYqW9SH*u3y1!ER04G#Q-7(R)vPIl;(It>-R>8
zXCg+mXL(l(QlBt!bR^~!=XkJ_SYq2R=G{C`)$?x=WiK9AowxkGI`jKhJXiMOTy3&5
zG^BCNzCTl3b+U(vmZ$8aVY6DF?P$~QFzsmd9}
z0sm)DGleQMP2<^5k#M1)>`43+Ssm5TUOMwVBB_qDHAAA!*0`5^{iXN$+`w!*g-%E
z^YBxwfI@7O`$P4f)-`~!0%IaP<(f$PQgf(@|1te?7&diir^94sX!Q%L-8Lu-%b$U%
zT_`V@SVwQ1Y|;hLCt=AejO`~zTBfJcMh*Ld!}w`6mK-tLj){K60@~NCF1RlHM&m{4GV0^FtE9&W|=fh%-@oj6BT3GSsHPZfj5<
zPf%b_;jLriv2Hv}IlH7%v#n4U9I{Rfb8BvPS_*gd-ahps_Li(9S-#%*Jm)P-`i#P5
zVFix|DGw<5x;QO#(L(PX+El7?q0m1J_x;{MU96X#^b+E&2OVM)%61F!e|Iq#5r=+(L@8aA2OXgY+#
zAI%z4LNsU4z2+%4l2lRI3ytdufHGP7nLBtSb*s_h>~c+eIGh)LMfQeNRKm{j%;Oi7
z-O4p_;+T%8vR3#IEBqjU6J}(uoCesewxv#D#7`%?ciY2H5Yfox7W
zwV99VeAF49yU2YHPE@q+gk-EM!wAZo($h!9cdY7#aXkl{V#??iw90Bi>tk=pJ0@~1
zS*+7b&7x0G+)s}0ZOHKR?4oPQZ|tyuDGKvYd4GP^5rKDRLvhhI+j+KoAm&M
z382Qz+|XKhI(rX0dLs-h?$I2a^OFK
z_c|G^B_?0J@*eKFy$_Vr#+>xYm|3i}eNIg4io@l+fwYt<*Yhk6sU;)60yB
zTSGF5IndOtYucHNKb+ELDtFGEI*~#icYjMcw7HI{%Hi9iqWU?1WlwF$hPt*G!kY-o
zX0&8JXOZ;3n&+ALQJUmTo~VNW>>!DyQJ<*l=33}=oM-9M;FgM}&Wl2YG~%5{Uiq%I
zwETyl6-L!bQCk3{7xZ#pnV(f5=I`kjxWcnH7=Z9lwV9|VS8kfVj~{mpU4J`2oPYJ+
zR_(Q!u3YM!yqpZnd$~s=VLDc(0uqICv>j2yat^n&%@3!UX{3v&VmM4XbaM@+&1Jb#
zzcGkZq)*YB$ZIWddl0J*TR_fq*~6Bf0V>k043k+ZY1XecyD~gBHsGD8OnEqo;WhKOwM7;?j}^9BO1YWMlOaHQl1uU0dZDQ~5ad*v8SN)W
zgf$DXkPgv;kWGC02++R^(hC}5wA8r~lZ?u>*(|K-rl;4xKmFKF48jGEh;;m?j`!a)YK!PN
z=AOwfL~>w{a~C;69qOF^euz*{E;#3!RfApJmMs=seV?m)uQP2GmVb(M>Q$>Z9O|>7
zhf+-&CTYCQts9RWZ&{MoIC`x^N76a-$itO}NSX?!EDP3qm84{r(9C|!n^T6?{{Wa_
z52~M`YYv(9dpgEr#5Vd)=A$oTGL~@63UYqr9@aF7JDp}Q8yN^!)60Hb*I*v(n3t+B
zEP}I_qTYno*`V=wUVk+Yu=xI--y8Kmn6lT{aA8zt;{8kK&YttS^(4+MfS}o;B@?q(
zUn2HBoG&^S1xzrZX==;?7%C_UOFePtqPuJK@0fd#`Y~i6UoGgq76$Z`XkPcu?fmsQ
z#dL3(;n`=;p;8Ml!kg66{b^%Z;%;qA2oZQ-D8&Z9w$JS^Cw~Lc<4BO5u~|%Ls!=nGRHZfhXyWoVq)gPON%)0|99G)@^LEIXwy&yPj}H0@Oi
z!#Ahq>%Vu~=ibpSN!zNV;i{KXvq@%G)ip_~N4Reyu@}U*ozg^wYHa{~&YQF_
z>E+s1{{RAW4}Z#VYf{MP=+_bS`f^88Fl~CJ{KYDDl0|xErDSEeiW|EgTh+c(Me*w$
z1qNbb*Wp;pm~%)xHL$jDr=d=ceN}NvxA$Xw{{Y>fUljXRLOx6NNMr@#J+I7}sAb-+
zE+O<@Nc7(jJXshnPC~(8RG1S{FE@z21r&%6>%3#l`hO24=<83)3vV&beu$ZkdI7IG
zdF->M_v=%rC2JwA>Aa5Q;w&eeY7{G~%}irzp(5z$)ML3+hTOTkboG(<7O
z=sT&*ZGUeveEatiL{|(iA>T4k4P4Wo8k8tvI|q71(#zO%BpZNx=U{@eo~Ls->##SK
zGI_BiX6K{ITsOX%wRj2ipPCgvuTPNMMve;^0^9Q|DrLFdYT;e=t4*ckxv{{Z3`+co
zT17^U1JcAcD9~v=PpQ-kSXOUD8wygvlCG)k%zw?9uPNsKp9TEay>SQS*!66aicQRQ
zERKN^UC&e{CJH+Bw6ijx6FUg#rhvq&)97A#Y}n~ND89~C5L$f6usbInpjr99dr?0j
z?HI6Qc>e&RDvypC%wr-9f_iz=0x4)oxIGMu-rRZo&7m#CQ<@H=oio^M!pzjXyl
z(0@lr7Rq9Xs_nOF4K*dAPZ7%$k`l4)4G@c+6pP#&R%kfRY$IDzjpw(J6M5CtX&eCv
zy~byAaJ)qoj;#|jWOIrM
zJv9TRY25RHUY<#Ym2(b8X=+|f`_KA6Jb#{e;Et%>04H+wGLH`}})(~wD~LpF!=1e5xh{Knuv^Y5c%hsUb^d*_k{49gU#
zAD60b4O;`I0U~cB;IJ$P$KBUMTCBPe**HRaBkZR~xnR`-1$y8+~I-n-}Ie(y%
zI{^qgAlF%>?VRvX4bU2GA0hOd7q5(x0q$#xMs5{A+IYN2ut^;k%i3AYI2+0yX4V%X
z(_RWi>J+G~+Yw1b^(+%!fQsIXwPq32nzwIN=S=C8hQT~!?3+T+ASbhwLR0BgS5^=S
zsO?VTnxf}W$UIC+tnkNb^;utE=YM@k)APh#Rq86*jA(+@52yM}9fUdYkFfaR{O{KJ
z?@;H(eRJktNlZT{^cY}u4zr$u(PK)ItweN%nO(V=o|#?K$co;8zZk+=(Xbk*(ShWMDBJ~rXSoL+Cqzw_Y?nu+$78|aGt%X16hTW^!=#oR($3-kB58HgN9mwra8H=_uX(*nK2jCXhD++jGf(&+GWOKxZ!8%lEHLDDfbPeO>kPaaU
z2H|oKA>cO)n15~*fjC?R;cyZ|z*_^5&j*BRju=QB1?yV3eyk4tP69+wFL1~f2Ow%i
z5#3m)ey0WGa*K61Eow*;g~+{aa1~e>RHBjEA_DdV13bK-(klwwDH&pxM@Ll_>cGcJ
z^}BSyO{-)_p!53;5nXj0UDcOqxFJG+3E?v+^X97k|1-PFq+$=sdERDHwsHSV}iGq@Pl?jM}5a%Hb{
zmG0yY{mh#81Y5ZgJ;5>D9bVw>?i_b;fB(b)F%SR(0s;X90RsdA0s;a90003300RUO
z5(N+wAu$vnA|o&|K?XAwBs4-6KqWOp|Jncu0e=Ai0R;jp6=w;;6`Gownwp$v8OCv(
zW~T|qzNZF*GQ6L+Q&Uhv1OQJM
zo+;y;Clff%F_b}G;6bNZ#(OY)5Fh|gMt^UNHbB#$2LcxzM8h-8}hqG!8ciaYZjDOwv
zcF~+n@fcKzsJe~DXihUxzE|RFx2bMTHB64It1>#3wvVYS*e5#mYPzS7CdQo!fX>v{
zeN9b3C*h;Sz_AoV>Jv6+Datt$lFf)uZNTun=QlmWEwjdjxib4WiNSdu+JH`}Un;OY
zWoK35ynOU~IVMlzO%yfuRY0XOUw@Q#@!izv=U+&^b$mW+Db=w#a9N7HvVR%twxzaL
zTAa#sa^0a^%Lu}~rTDK@A_^=qaG0-z3FBGg5v$ZHX4nu~b74H9VcwxtRiUbJ(8*3V
zcX8;(?D0Dj0NmjULZ%fnP--IX71SQ1Y|bWpcR@7>j1JIx)YNtext&LH%YR8Y9WMOS
zm(VNvT6zv!o|)7Bim8>;vGNc6MK!Pej46+LUZcAF>1X3TLI8?;8KDU7{{TOD`5l^d
zJ0bYPtDCKGQg!2L;%UZOoRF!bE2`u9G0ecX#eIwAskq@{oXc3_o@>@C;V9#FHf0Zm
zk&Tj>6!k8j?@Xr}@qIu_FMnEc6@!LlNqs1()SjSm%+E7XfF}
z0Tz#Z7G&WilFSQMHZ`gWHr|$uC11&UTH9azC?&crBtJ;LxLZ3OCsIgPA0S_f)g5rG#%s3V6WsUq53z@^kk>d*9N|1Bcnt6v!fH6s$!>S3@XMjrHF
z25?vvCT~uz^}80#cj@iR6Ozv&PGG<{FTuySWb|s*J~mZE-#-I511zsWrsKAtU{7P%
z2rDKvL_mfGoFHFJKF1SKFq;q%Wu7q<`yU-wFO#vD^&76mrGIFryGHdYJBB>4Oocf2
z7m+nj36EO|
zqdZLwJ6w8xp5CR5GAXqUZ_zLD-JMsSjGDM%P<1fv7Gw1qO;=HPy1mT}k43z83#k>0
zavO)q>9~GFk$;$!E!VxL9}>%^E)8_{+!K9*f;|Y{+h9&p)X?lgI@PMHoB@Kkjl?_4
z6g)J5GiXCBqzD?so@8`AtZ2VJ=KJhG)0`m9nX@*2)@>JU!8JY$p4oSrsNTzaKjYY=
z?P&6+)9x{kG85NJwBqIYpm6P_mI1wX+}z}1x3+cBD}OmGawUE}dga1!jp_~dTL2zN
z&#~2z1X>>!_0csgrY8
zpMQ{zg#KA+D>9nw%!~R%E*{K1YqqBusjk)6v*8lzh@58_N1-+B4&1%g
zIU9!of8}l(c(9F0M!^>q9+PuZpS@r?Bsi8C{6qaaB$7zt?rFs2b`su&ws~4DvbOima)GBP7(vEyS)d^kP-sDHFS)tOZ?%JA
z{Ter}-Wi|MO_IuT>5G;@Os$8#%zwq|G@4&O9%Sbv$jICCJn=e#SctaPuXRx}ps+L~
zzz<^%;O%71eLh&$M&>c&b#Ju>5D3fZ6+1UY_g_z{iH%iQ>5_+!J
z!qA_wVkF{Vo~Fdun-gedkw2HO?zC
zI$kzIKSRgXPe-vP9!=do`hTxW(UlA<9I}eCE?Ai;)12}MTYa2>2D4-FUmF(nvW$NL#n>KCSF~0ScTCUvcMJ&FoP^{jAvsP_pHrNr
zmJ?LDmwSxHax?OuoZG-mQMtxtFCa#j+^m`8Fu9&d&v3YfqO>+fGk?fLW@aKNnuC0n
zzDoG}2Cu;=Gy`PLvs8(P`gQrD#`O%`X~qh-20Z$BDb&#unVw0?a+{o|In2Xho=BE^
z@>i>DU8_!AiS=RR)Qt(b%5z!CXIIVS-xH_fHdCA&cB>l)A04~1OB!U>mkQOeEL(RC
znwuQBgBrLtlgeOm34fUlp~_yNqk@e(K3|T`dYt1lxuyvHUbWMT*^QjFi#nI#hL4ey
zkhiS+xK%@G@&@YY`SW6Wi>Qi*3f6c;Y;zj62TuWE4UMo^#@^Sq=Q-t-IMH>D#LL(5
ztU|H9PIGZjqw~0a)qu}(x}L@qvW;5u!1d1|h2}+u`zT&_0e=9V;rX7YuE_*CoTD(C
zo15%R4b@#N6fp&dvQj%sg|IeMq&ZV+(OCWwvTQO?y7$
zF42#Gt@{<<62voR2Uy%DRjXQ@(aO7gv%(5kPO>TUt8G~uQ>JDu)$-J17))Qrz|%LT
z(QN+!jngVED1SaIX`6SJ)bz-AS^RA2nOa$+sPMvPBQ|Hb@9kBK>QcPN@P^-BKn;X6(#-1s81(6BuY)VjbDyR&bBTv@
zaZFcOefBenfK5e>9xtiCY|&aF+_DM1#hR9hU$z2qoPTEtkp5HD2;xlW*!~<|#Kw5-
z>zC>EYhR7sBHHM|$z?UF+Py-*#-1j$ztUwy%7ZMKWW+_{J%c@M3
zDp9Gp)n+zI0q{#_#pxT55cIR{DSwh@3`<)OPpe
zoQsz99Dl%jE}?Nb#j*z6m$7Fb9RV>n!p;{z|oZR#2o+sSzm1H*KC
zxk^;Pe#P{?L)@ryy0zn6CUrihDS`TcPMYU3wtuK=sN#7g)aKO|vL{Jf0nPYxrvp*c
zZo2;f>MJz!d&}e=s>__4e=0J_g6|?w!Ie`+)l89MHBFCuQ1x7y6&EWO(Fs?LQcuNU
zvO9!ZpUCjcd2@bMnbNI*nfXplHcWP&qgg%Am;}**V<-?TAZy&?IT1aQRw39j!#Pe<
zlz-&|P0yLeahiytahjZANK$y_LjFrv5yRIR#%d3ZnX@_S8%*c9$WB6XoV5*`gqR7O
zqO>+fvvZuj(26TaO96)FkO8vc12^gDbtWNWtO;zSebYkEf;7n^iM9
zoj8}bQw4*JhzkltS1N9gG0M}JIZ7FjJbz5#
z=>l%h2VuF&ZfrtC*qa@MFb%;J3C0ziV^$1r!em?IAVMJ%MUFKIKrUjjo7DbHo`4&g
zGe@&48TlD<8m{S=`LQQe#5pU8(p4KPBf~%BJaaCdnBuE&FxLXx8rL%CnGZL%{{Yu3
z{{SIR{{W7rbhg!;+I1kOD>cO(=@{Wg-;kY-g%U)(S
z**MutblgK98j7WD`u4QF=53lm1ljT@{sliBGyS$V6?K;T9fxAD5`kJc^VhE0nt=(6
zNii_pr~GxA`UVpt6{}i4a(}{Z(&0jfV
zOTElHcU!)N_DVh7b(12=bDLd6A20Kb?*^ID{{Y*T(!|)ZH=-bewxL;3;@Ju9Q*x|E
zX-`XbmGrvSE;1zH3dUKKoU2b(W`D9I#Qy+F`kvsPKqo27+nHA#H^z)UqfStr#aI=k
zC<=Wmq;Sd-slqjTo_{%IXM7ky8h%sqg_;&H7)X`sYqp?BQ`S8&Q68Q$&r!V;uH#|bz&*WIW=fOhG27HgP@+D1+98#kP0
zj%8aGKa)ng7>!gezGPl)DzP;=1%}m_pZN3{(?8vi->dL0)PJx+1ReDVigvv>1z1;7
zB-GR*5gI(UUHIi@e2HAg8l*52ln1kh)oR2)&0uTP)F>fA8mYjDie^1Lm{y`(RurkL
z+Ud4Fe^Passg;x=En)HcMclEtfH^?Bi|Q!idPlpm{{Rcv`d}rZ
z2q61R=-D5T>|u=<$1=v2{&r@6F)1F`;20U8+(jBGpns=Q&t|rqgf1btPcG+@D+s4?
zTa4yr%(aLKH3;r|nW?DWr!UDE>y+Fke_=N(q-n;Zv7F3P?F
zsr6Szdw*Dp$(8*1IJ;WJg@I%e&Gnz4_j=6?XfaqP$l!zD~iPf-5n|9y46ueNF;+;8Gyml(!a|Wq#PVrcP0-
z)Esi8&a<|xEpW9AdzGX=`&L#o&c-j;K
zeln}pnySZZTxk{FJ0>b+bt^Ao#LksUL;7`&5{rCqZRW%tp)sZuKI3|fV|GvFDxs~J
zba>!1a>e!bZf<V-+fP9&eC6$iJ`
zhvv%yD(7j?gBQ`?exlx3TFzM@y)3U1!%BT|u43d2ii=Ep8n|m?7!{jXs()GrS-6{q
zf9zN&s>QGq?97a)Kaf_KcP5D3(2W*qH*j=oajtWa?sYlBNslY7tYk#4wzpC3tyAc%NWiaR
zeR*9NJ3A|~#<7}nXw9|hS5z=`yST8TrE420m$i9TB~e>*9}3$dVSm6A-yv2lk`&wQrW
zW%>UA7uV9ucE_tx$EE)Ojo9>H^=dw|3D|>;)SHG!RCP=|!dcUWjRLKbtp>ZMwM5%%f?vHp6WgH!THlkdzqb6sCg@vdMM&=bC0J_b8s2C
zxRx?K{Yv{%YH%_Y{ZZ|Ob}^>(a5E0^YfPD!rLqIZwpE9H52koiJAXQyBt3iyjeU6){{WD?
zE+0?&Y^;t?hm(=MXTA^PS2_JKgVgp$rlQTvbNaTQ%&>vyFlB#}8+2>O=PDJ5lv>m1
zw+~FAgY;ZB7IX|&nIGeeGP+JqJ8>$1#jOCDSp10C>K!F0wv5|)Nw*iL^*S!-#T|NG
znOPlo5ize@&3~B9DZ0weJ;B(3VSWO}+~63TG-+-#*9}%MvX%xFGsMit38}2)a*lU6
z&NE|ZvkkaejeiUNrM_1$8^W7
z3k92x>}Ar`Wn=#Uk1va;^!LHD2O`QJ65pt!GaYOhR(~3g&tiVx@$9rK)M*cHZzLXD
z1jfF%JvSDttc4#$+GY9tXrr|+7l+A>(k!fHTUFF%Zu8Id7p^B$sV-_
z&ZdMv8=R*y51cr(ah!}0F;j@1S#6p=MT(+th&dJroksPP^(tNlS634|r&Ey?F6nug
z_bv4Usej5d4RsEZmYiC0FdDv~%vTxBT|(RJ$U;Sr?lOi8#g%MuOP#Hug8`*~ZTU|x
z0NGoOXq98Nv8+l4Mf{_$;uOWKnPRYrKT&>b8#Y*1A0Ic`Kb?AkR=lscj#hc0WigwV
z*zd*$oP-Cq^$GT|M6R}W$zZbF_q4rfGfFUtGvVRB!52YY-YRq1_2@*Qmqc}p5(G_Y#J8}8_KbE~PJ`<)*H$a1an
zvg>SgF>zKPR0hdqlOeV&
z^)lsooR2lKvP^yQI*g1U7n$O=QlhG@y=z-9XN9{vg*Mn-zp)6FjWaY=p1@r1(@Tpz
z7>yiz?mosbb|}!-)AAg335?RPM1QN`O>R2=IKRpW$hN0K!x;7T8$4J>e~)D%V_jd9
zsb^&`O2W(QIak8W#|E#Yr=?{Jt|!&4GG9Y`JBPB?%?>5vA4Xv|jf=_hvPBpq%5F8;
zh1FK|d!skJq^l;oG5EC%5@uxqHrW=BK=x6&L7sTl$Ff@T!!n!l
zIP|;=k&ag|G4j^XuPIt;SHMetddL1g3+|H`>OGntC;L0tGnCvPE3QNX5ji_tm7DG2
zY;$2VbA9&@z|^Z+GswlPZGXmTReYy0wLZ5uE;_FeeqEQ_QoJhaHTf`E2srq+UfnMm
zyg8aF$y+dRmRy*%49xE-=(}5q^=!|#-)uY^^4|}jQBl&8gQEbTQ7`8{~PfWNq
z4E7f%w>?UFjg`de6`0v*L*&*axv`kLW7I27!YPCJv~PmCh(?*UH-EUx6+~5qV`xD8
zm57kHzFwKk@)E4ujMH=a{+wQy9L-LTvD9nv-VCoT(k6JHR0uamvR!x)&n8T`
znJFH@U^S#rs{wNfxgy(>3mlP7BT)i!1RdTqqvU*uCZ`F*M~q%%s+!W-kcILVwGpe9
zRS47pj8u-Y#{3mbr+74|ytroTW7|hGXG-O)WT1?+!KI~OqTi;{Z
z0iObV%Od)PhLc7e?OG%<qj|;GtX4z`sl3D#WNAO?d5^*pCvZim-pK;HL2fC=Gfq+AyEN#W2
z5Cs0XJ%hw|#(x&b{!JBjQlwF1`3N4<-)3$iwemz=n{m>bxp9@9K0Tn0aqO11;U4Xx
zyo7#`YGF>IVt-j=w~cFdO3_`m8ybpU`+84FuMK|Z6a0nv_)S9t%H4Jt2LzQB9
zwf_K>N9HO!&jpjj(btUB63lTFG8=ih*%1gOek?N1#(#9zUAa2ZiEVJC@=*rAv?!rVw+;SFgWoGB}Phfoy
zaYzyKYe;jLnsPaVWIrd={+J+TMJQ(*ob~7H9zTTC9
zxHNvjokt()aa~%w35H`V$H*x@q!DG?o))rrD@k?WPbr=Q+ZnX$?l793#13I^RhhV~
z1b;0*k=mfRk^}~F_qb>ExDJE4w1m`eKszC>^P!TgcQ@|HlD5-7^27fCZG_t2#u8__
z=uHzW!l0WvcqQ>~tm3w&!4}&5`R;BB!*Y}&%+dOXLVgh3L>8%;8(@)`A`)71)N#sE
zun4stTqn6>5fMAFn}^x2)NPW#V4uXJ27i`WqlEq%{G!IP(&FoR)!Bz25Q+JH)uFMv
zFHIPg{?VW07|PVaTe-o*J}1Re4kxiv)~|R;ZROh*fh@GBYKu(1()S}6+N+OTF2A{
zjgdI>KM_Ba18~-76Dp_~WkFSVkiMX*Xii)XMS|M`ZE}zpYBNXd+;8@Yxo7edDZPWI
z2eF^u%%Y%WbnzTNJ2%0FDqmcdP*x^X=UdBF0+nF}jw1>#7UED1wB1WtMt_x4l&=o<
znE06}-fbM~3s!ztO-d*ch~|TlDXIKsiInYZxwJ)|sw2i22&k_)dUostsSy?3tUkPT
zWZ7LSMJ-i@RVi*mQ*)Hx-?_iEp4~qf-EKW%n!A%XH`t$Zb8wi(3d=^VxwEY!FC;_d
zzEeUZvc*|;M&V+`5CHaU@P8`F{G~rSO%+YlqGd6d!}4IdGhdTzME4X4Wg(;0#yatv
zfY8yg)uX9H{{Z#e+!k&dRbe}y-}X;qEB4ht>83$R`qvIMq`AOr4)}MqwX8Uip=KF4LZ|taTzsIrl
z_83ocdlLzkWY&_*$K8K{`pVvA<&F(ym&);iLg6LZ0yTxARk-C=Y`Fj=vH2B=-3!
zt&P}x)lwe&tR;!;wXuSOE5fz=S97GVK39t;S#rfA@
z+{m)c?rv}G-(xp7H#hbPh&KH=aL460H#Z5JoV^nXmfW?*$#!)-0sT9lAfQ_-GD<%P#sloURDfy|s7~S}RgA#_CCaY56BV@QY+6!Ux5dl4r$bIvjnh1tc}g(y
zEXws*C9JH#tPWf9zues1-(xp7H~S)B4Zb^RInGm*<$pH`n}ws>GOA@?A=~70yM95f
zkEyLv>*d$`N}ke`rBmwooi(v*N~M-hn7!Y%9E}>>$vbM!pB$=NZv%fJkX6kW<-Lt(
znU)*a&U4KYzrwWREFM8>uJs*%E*?3309%ilU2T#n$fs_DlpABgfC6!cgjdQdUeYZFE9>?|s>A&r*r#Ko0oS|r5HKHn(11W2n
zdAx&`vNEgKTXM(uWrZLQX{#MlJ<5UXYf9rCHGhF4F=7PN1K(%Dnjl0uWprWql
zj#EWQwT>}SBV(h+#;z?0Agg0SwYJ%}AgtWn+$I>K*C9X0Y_9A*OR82KN2V=->tNwE
zH-9&{s{o5<8DnjgCV-4#Gvt-_7E*+>R+T7Sp9zk^fUBD=c=Fzqask9@?S7v@nK-qV
zuMMq?IJ6^2zUKb`X=Vxh(ftR3rzzxx_c$?)YE*8`GMaKSWgR&ynY=ufTkM(3P6x4K
zpnHoPZk}!zEf}kH#o!Qk0v^_UZecko<$v5|R(m}^O1;`XHGfZK%*?0TRfwhP+Z817
zJK*9_SQW5KDSs8p+y1}3O-~N#eQnPdaWIVNtl!#hq@J6Ok~30G|<_aV9EFMrcg=ILNW-Ce_Ar6@Q3}
z?Yv^Je*sMb*2uTUkH?-CQl(dQ2YWIYFsNC}#thBITUy*(Yc^KEe%_{x)Ds=_)!peH
zc_)p+fH^)&*DuPStC0AeJrS(3wHdd|u9luks>mDTmo-ZKu9f5CrJY0FwijVPzWdjE
zPi~*dgCME$4>^sE<_~DZ$HwY)>3_&J>Q}~D{;Qt7R?7&L2ua&q>pZT&{!!M&2e%pg
zVR(JTsA`Ixp41g8v{Mx9dx}~HGSzM#!Ewa$8sC1Mh}oGK)U6fGLmDyEPtIFXvh^OA
zwK7u;To8Mf<8bU+7mVGNWfi1b>QRwnSv3}uTA<|KI5Z1@!^C`G7*}l00e{C~%{{&T
zP_0D1nF|$8Sap1ZP&4dX%5lQL*D$;(u9UXpL-DQD3L3
z3^juc&mAi)IN&$sD{cksEdoM@a5ISnEDI~i9wf*CUn}xW>$sSQ}eW?
z00PQ^uV#r;7=1>&W0{)V0%~is2tFo^;XExoaex!tK)|RIDZ
z6C(nQfmcMuLuX5-Wnw)?#>C^}8*Ij+g|EwQmD`BIvB9P2TXDQ&l#L%rn_o?
zc5bG`Lw$=eBLmX@s0yoAwmb1mPRXFma
zijwTA=OQ7qaIt~eN@W>AYi9}&-pbL&p?_-Kg#Q2;*PM2=@lK_X_ubJG5LKU>Mquf<
z#}^XYg<`yFI)$|l9%bLjp@h$vsVE^3{Oi<5DJ#Y{uYV;~wSD%w-pV;1lZY{~yNG97
z%3Ni9n;6m5uBtv^b6u+BB`Ykwo~LJM*FD(piQ}h*KTdcoFTTOlPclZ)JZBxtlrq<7
zQRmTYjZQ`2Kqr(WyD6>;n}bU}t9AZO1yjd3O-*;)`H5cO5iUy7%XuG}S_0orf^k&_
zHoFE1D1R!|*oB<>gl=kt#$oMdy($AdV^-7br1JJa1+DmjWdEvZO;P*aMULm;}e7;dw|XQbytSH!n!=Q3zIcBImE2AOm9g@
z6UGE^pjmBG%Us$wV*uGzp8-&j@dvU52Z=}IEIqRn*HJU(Qs$>tdg!s%*EO<+v+;K4
z1b^SEj8pE#++iASe8t>SV{b@K;n6QtH8`Hc`IOS`-%PcymbISDA|FL{025H4@6oOm
zW~)uceN4>drFt%Hciq=~je~>MUX{D4jEMMuQB8Dsjm|g|xm8VJ@dWSlD)i0dy2G_0
zj{J^bi8nHzmA#D)*todwk$y68su`J|jem^QTCTlhF{jxMoUr;nC)8T>3SS=9HDp~>
zXXM-5Q>G?r5eLY&3u=u;cCD4N7{X(LV`Sq!X2oTihPlMRA}{4Y0Yqk6$CB)(tE2LR
zreh#(pNeKjOIZBrZz^fTz?@>;JkpUdCoLfh!zQ)$mIo25Dw|_Gw
z-a{7OXgNhg2d>!y5!q57Mx(iF4R1ubF-XvrR>nP4th4#n#0K8Ssj@>P3$tBXgT46n
zP{u12Ju@t1WApWul4n-Mb;K1JM^$uFRy*5GK|?nYgfz
z%dmjtk8bkQxT>o{2rjcQ~IYR6sX
zXVdjKzF!+yeALlO*B;P<{GMHxL
zWtWTNEnKhAhy%Tpn%`5`YVoYeYdS%x*BZKEH3*}e)uMSUQkMsh?te7V*bt4xC1bwl
zxJim)H-0lvC=u-3MdclT9G{-eFzqzmk7%}|FZ`;$vt7G7=+%8L
z7f{8kE@n}Wf_`O3ihuEyuA5t&W}%+@5f|HPGD0;sH#aLfbjl`Ok6r6^2DXg8wJ{rn
zt!r$D#%yadE#GhNz9qVDq`-mdR4lyxren7oKs@Q%mCZ`wTLqS>J+UyX$v;JjhM8Hc
zLtUv~>sy0~hO2g7HaNelmzHH%(X2ZyH>kF-iC}1;LX(Vb2!F?LdW6l*L+~{WOnJ$5qQBns7|rGcC{jk)|`ZT4jQ2W(r#H+kPFtx=C(Ex`cnPNKKw&Rm7uT@!1T
zIXe&BdOZ&_jO7`oP7OITAEsl_*)O4M9~Jw#loIEcDs{gddV}q=;;zO%yO{%^j;U7TUAqapy6Z)$jEE7iR9Schk(E<-ObS5
z3?)dXGz=e*mXuCuDJd`C-~HYD`~fp-y=$FkpJ$)F_qjv#u8*AyXWTw$YRmXSkBX*4
zPxU-JJlvj$?sxDMMMYNztx7Z6hHvz>Zz#dlFIk{n8IOu2YxGe;TwjE6t>#AN)C{1S
zV-nE>KFt>v^?{$nm8Z(xPiJz9fu7o&(TJIghF41z@qh9rHtAGsjX$iLUj1S08G>ge
z5T$J~$Nox}ed9EAME|07CiTtrmAEt8f@OB_3`Ta&-j9!Ve~8tz%2&Px$AJ|F9ZaET
zf!_Rf6->R?uinkr_>Da0LAlmtmp|suInRl(=@3X23^iMeva8|l|Gfwzy#jcD*4!z)
z*nX`h!@7Gf9&^oiPiVQniR+TdvBlL@<+TM1lqw&Aw1C<<^PszVf@m)&zOqF7J4?M&
z*1`;6vo4--(-RgJ%LO}$YJVI1xZyHh!}6&?{R34mYW@t}L`=|L0|HKd7s$B)uW@+k
zwQ}gSnqY6zu=<5anNK+@iywe}bau}vXE*l9@mkv?ZL8EUU4CcSJYTdNP>Muxm{ms2*2W>uYq%OTPA-
z9bt@>{>8Rfv2+x;;sExx#$93w)UT;N7|TpGOJ{=vy49Z|r*0z#xd3go3>~vYsWI`>
z&seF@aVBr&tePzyD&mENxevGDHxq&ipAC*0hbmUR3B=0sc7+tVi#kXvE-eiLzFQut
z?}+!E%gi4I^y6t|px4&wjZokW+YB|~XQ01mzPEiA?FA3zDZb#EZJ%$@SLi~LZSs*E
zqL}vBoRS}vq9|!IsQ}{@A4oxFjcavHo8AlA*>MZkKfPoAY;ov2mDW!uR20QJaON?m
zIhE)mbSo9oDq9TINLi~8sQ(!+mZKYfCLnkj7iRw{8yD?Gv=k!J`_TeScG%5TDlXvO3Q5gv}c9
ziH?vfyqVfN(}~s}@VIFSuLO@=`lfUflU&u;RsT@%KQlB^$yzF{Rj_!b;5o8OcnoymYsQccj!H$uWPy*S~8>%(w7Swz7XXXa$`;mHa
z%mTOY)GQ(8?x!eALf`jyHqxz`!^al+?LlUv9zJQao5MdXD<3a^C`Vj!1#S4w)>kxM
zPGs?V*60Y>8JNaObk}hVP82^CX=Qj3*44Au1HWrtrQgl4uPsraqI?M>{2=PFrWd<~
zj&(5^tE7BAM(DO2(9~K|sW)Qe_LION@b#7xi`H^0s$kKKv7Xnln*)xzi$nWe>%=zPBY@Oc;Yl
z((({|9g>YZ{gx7d$SEg#`)l;H`@2TK@`6AQEQ@sIRrkPn|B3z%A?PS}#Uw?jxz_#Y
zqoExlfEs@1ialHv79ee?7w>wEENq*F8vtGfCP|5X?Pa^WyXz5q&&1!FM?ilP_1^o{
z`qT5DFw}F_T|DOwesOM8;`7RjMJqO0dZ|K+p%DzOnzsbLf>Xnkr@zX@o;GwDLGth3
z@*5>{pKYrKjWmbWu#Z77Ac-FB+>Fb}WznV91OTESLZ1$2;
zU}<1Yvhf?z`;}i$TdO_qAJhybbrLW8Q@)ys(=*moE$Z;som^BWQgt13bz(8~(-GuTO&n6e86HX!Ba>Jbm-xB2+!j$J|!Y#*ib}Z+8Bd)$o-cYLs83^QG
z&A$x-8)
z|5+swhQ5LGqIvzn0y$?U$*&VGFU!T+7d<IK`U@g{w*EdlfB$FvefItS@1UX)6TKjL&B#y6C$EDpz+_40Ca0kNe>}g>iTbjr
z%h8;|$E7wvJa(BA`;v}fQG7)1!i9;*L&F`zFSb2gPi}}`l4!9XOIuQmNEE)2Zqy-t
z5v9K<{#NId1LdINQE4mq<*hr3NSvhvS=`XBF*il%l25d|XTwQ$6LgNM|nqiBQd{UR{xzb_tqFw1+d8Rh+Y3ScK=uSWT=ZM
z&6pxU>d=eT&XFIdIUZq~x9CrReQEtbAnB%{2#w?6tlxrqWs$H$FNC6zHg_W5KG>9u
z$1WQ+kTlwZKE(+yvMH7N`&(5nj8}t);=i7^cj9g>F;?-9?}&tPKfMc-Vu+8poxY`Z
z7zI<*12
zx9_Lr{Sg1nv11jz2m6y4YUfKk)zyO$kgD@aM_O#e)otLpWdU0#rRqH~Z33GEY=fLE
z3^6QCl#{#LqdJPvJv>!igIqBePuZcK+4QZZyOilvsU+*VIHHjJkOxmyNi(-*b1CAZ
z1cmqwM$1i}sAA9JS=IW3)G6HVq2x!`j`gGslE_@j3{}pNWf*lgUjma)eBK6#KGym<
z!RgFS%SPdTV5`+_={NsW6JRY=Oj41Tm4DAjhp0!Km}U)Fc9f_n$3+%}4XVz$l{hiu
zdB!CLA2odKq~njnm1<(}IWwXX_l}bWofxU~atp1V<$Pk+YP(j`6d5
z<2H9L;xl0apt&gbhVjAlK6Qs`{||D%rT_TZpT_K`INei&?r#h`=y14xo?oZInC}^U
ze5Sp!Y3o)Rv~{VgrTFoQ;^#~qf?vTl4ju~STNW!ZUe?X5I3xQhbgcvC&%bR)Rqe$Q
zD}3r@3jbQqLGNaB4@mh4Gq-vBeg5nd47^F3J6c%Xixt!T<0TL
zA)=3@UXi`yfI8D0Q-G=8aK*@{;fE9j)lW*s5v!v!6Lb9k^a?d(yp4R^i#>dje;3Mx
z9tA|PZ0pohW*0pXS3lu0&rmI#8H5x_-pR+MJp=}+56+PTl_(AegHh=@50eGzAKz}8dVk)9zi)hKTaC77;^&69QPD@IadaNP
z`Fg9W=R1e*2V9uD&LO1P&h$7}Txrd@jAp~!loO7B=v*_xM?LxZ`2|h|KGbp&&-}HD
zHnkr;2%wj))Apcw)wBPLTdTpi=fzgxUW~A?NTJDa-nsu_N8j&y2jKZ5lx=hy+pm2S
zv!LGjkOuI?aJfVbG5&;|<;?o0WGB`z-aEp6QKd_Zd-K}p=#7&P){o`7MU&Q{eaZyC
z)1#%8I`6TR*k~;JW~7{x|ARbi{hNINzHoZq&?y`a&uADJIOUqHSsSRcYMq*_hh4;o
z;u?%HZwDmDS;oW!A~z^l1wCj>#Jv`Bt?JEBEdUI>ELgm{F4w9`it9J_X-a={#~!=4
zE6!Z*|4>5AFJNiTJ~5&+Qyfdz=Ffm>@XfRe{+Ty-Tio-MVa{hnFi;DgXtN%ap
z{ttLx{117V&;=~r$exi`_y0gI+W(+;GLSjS5X=}O+QumXEH0KzBZcC~71)b|ozq@7
zrsv_^Nef&1Y>{rH>6Db|7^-{tHbisc(HCr+OU6!@o8}W
zLF=?==nr$XnDEaf-r1ntD2{8_)cG);s#RQ(LJ_;0+y2Cp1~
zLuJmAl!~nZicF6
zuCnKaZ}cZGVY{b@56Ly_D&@d6YZ$JnA^5{S4zOAOK^ddkj8uGX_;y31`vACOP4YN-
zXuXM4XJcZqYBYOcHP<`sdJ!2PW_%|0Y3;KlFb=>RwE2R8?b^>?$)7k1OcI@L{a~KG
zZvL)zaL0n_)0BeB@da1cxBcHvAOHMA@vEV)7Ye(ntbdcMWW!?1fQBL2bYOgCe^d%H>uH9-~_zC!ynUxo55n2ej`_e
zkSnDl+h+MgPdf&TBY%F;w(3x4q?hSft{Ay9I45hrZ(e%u|3(sqhSCvfEMY%o;>{5cWuKA(O77Ke(Bm
zba+7wojUkwLm3{W>fW&BkLQ#s9YaWdT;Ar`zLSt%nYQ@coLD?0UKqSINd>a@%@uWN
zH=ne)lAWYfYH=Q|hN^WGRM9wXT!P|0l!kg_QFTRgE?syjZWl6rV`ud#J@1%RZ#XZN
z8I{N=RZANAR?qdujP_weN`$vr20U_IZSW12FjD6}m^?`4enCwRJua1jsY3-178kT)onix
zNvoDVGBV%9p4e+i-Yn$cQP-@9)(_70E0$~J>GI}pQ#OXhHF5T24*3py9}-uuS-|x9
zV>~^!n?3tG>4r)JZS{}Go!n{SS;KqL(rPT2i{xq_`d-=i6rNd7Ime9OCyX%!Atxez
zg=LhwClZd_-8a!K%YpjN!c_h=wtB~{mz|)7n~5>tLIY>1o|+lwB74b(W5wldrAyiFxqVW9v<-?6329?M?JKy^geQZ*m+b
zSqytF9hNNaS{k~3FZ@HfVEFpja{RxvC`lp|)E8*z06IDb`u`H6{udYZyq4`5wunfW
zUd!uPlJW_-G0Q0gC7}!I7Fvs_nY_z@g
z56}j4tf)x~uh1?I_N!L@GeS0#mX^h{XfNsxKOwf$&hBlia!T8+wNHR}K$q$sIT1B+
zCc)vH=aV%XZ!7e17v(NoF36F2nm5cFO|J^}1!ND#ahXZDcQM1zwp*4{#LS8d^*@a^EJl
zNIMaL2ADkI&UjTjI!O-EC7jlN45|4NX4WQQ&WJL|9p4?Cpf!rPKw>
z(TX1vHH*`2h8ZCX*MsMU-Gyb43dT2X#d<~J;@3KKkVm!IY^jy6zeqoaZ3<2Y0p_fP
z8=s8Le!s%^)?MZv)JEQlxk}U+-p0(x{7&x73vzT0zXG{wep_{NP^bnW(^hp0{h$iP
zstBA~OcrM$6Kjs%0h7fF_CZTK6LU(P3(AY*ex9NrWoz2IuV$Eya`Y;3K`NJrWk2W(
z|4@z(^ZYdlt{Q;#q1-b+Xew4{0AnxRF^I<{F>bJ%=8eC85_}eY_N8eTyw#1lQev7s
zz%7AqS}nSxD$_y#q2QW&3VVIh+n%eMq?n}M>EsezmH>ry1!+>S!kJxlB1b^cNRNh=
zbPLYRAN`h17YUX%)tJt`BfV!cQmbdSJj7?TsyCd*fyM#5Vk)0MCEQ+`g&L{2J9_BNS_
zArUbixC?J@^g*use`h
z`YpYsQ1OEfI3=x`(}Na}Fs}GwPLCTtTVn_ZPg!0E!`b{@#4Q
z&6w|NziV$RF`z`;XM|NB+2*QU9<`(VE;v{jtP9PHz}2eUrFZ8q_-tu5J6
zs2i*{wf^~tfra-`6{PVK)@TERIUWy=25_clYts4fL|jVEKQPmQq3&g~t2S!E31nK}
zX05Md+U@v5D|2-O*aqbaEh6NE$@9UvDMHCF%W@DgK*oo9^RMBZ2KQ*i7+-VCghF6F
zglAbRA}d8J$_ZZFzNmKgJQO&!ht<_J!!XA#`!HQ{C~mEJspn3`pN;lzg9${A5)BP=
zP`pSs`NpOJ_Cn<=v7JRZmu%ScaS_oXD;nZ2e<)PvlvlUI7-ubX?6);fY?99I;BF7T
zbtV9ynBt8E8|bwAW=9nJu4aURB0b01lCDRs+l{#y!N=@dfq9sWPxB$+$8W8kijqEF
z$z|_Ftper4mybLbEjuu0T-UyMnlm#q2m*9A2elh;E~{y>0Us#){{3WNU@(DP@p|R-
zhkyXT4Gv|EJ#3W3Dd9FWd{5u06S-(G7YqQ^r6f>+LJ;Dtj-8;wo?}5JH_4&l?60`@
zIt#<>`#X)TBV2-}@>fSQ@2
z4R@*_#ac2Xm4!BLC(HQ439>atnQ8(f_?2F+egxgt{YHC8y41dVDO|<`(aEcAUaCU=
zgPXfeS1*Bl?xNP<;&hLK@NK&_yIj8B?*USyh(q5{Hv&$)u)dc-p}n0Qc5og({glk#
z$-ZiGE8JSeRj9pP`#D*(6}s^TfMIXL!aPq#ThRgov+?W+kP>}%CN&p7|BhSMMfUP{
zzdr&hqq-NO2k$ma%-V_sE=x}%xZcSUCbH*he0m=dh^^N@ulJcXbZS}PKTwy?Q0zYx
z5C;=BpF1^O>H3)Jz|S`_+nmfYIk~sm*&_#9GhTiElhf!*vd?)Lq&
z#x{FPxrw9oJ#gh9X>WO%3L$6Jj%p2(Z{TG`dzB4%RY%jk~&cMJVrx~MlkDcxF
z_ODCigAy3(SW(p*h{x#**=lI0*NypEX^KEDUx*;9T2LIEwSzipy|&Vda4W72Z5S!>
z%%vF46;;K6nva>^y3?7iHxG`cqrRti*$)=FwP>OtDjyUE8FtbDwq|gD?(CU%g?o8H
z4dFd`$=)I7Y~28913Kfdy|1voKBW7T0rE*By5TbHUhR5T_^T>f8>EU8X3ojUX=D2R
zuoFZizJt~uN)!_AwH4gNH&b-2f_g25igHFPS$Q@;grjjY@w@|i>cjo^KNJ|q>_@yf
z?CC@JqMvF7(#Hrm?hjDmCXzz3ulJ=fiH9EU
zpplVIr~ZXa9fm+p9wzO9wg&OBCigTU#O!qg54ltp(@%gJRjwKLO8YjHe!P~#*iEEI
zy7On@*tCI{7ycz;M>lf1M_1DN)LI*qrx|iB|KowX<^tEI*$FB|fw_J$#YG
z>jfp6CfplshWz-4^6L|_jYP2g`2y0U-4FYJXE>GEeA{ARzvmb#nr9(&bPlM?FTu8v%cCb
zoRyvv~we}`%Y9n^t*2QO!&-X~3w27#Ak0uOF8yH(rXNW`yaRh_c-6pJRKGu4Uyd=J%g)OaCye~y<;mxE+=iseZCATV
zSJzTW2&)xzv}f_>PQ{%Zwqa1|A2tORF8u(b#^nY1iwI$~@;$-}@=T)%Fi~}<30D=v
z%Q!4Yydmo+9pzqzy{kHOq`qg7$UFe-Um$Nm^84dt3-leti9AVs$)J>w`0g`zNltv%
z#Bti`aEdWI0ZGXuui<(o>I>!iqWW=6OFjJN^`6wN3Oe>|6^oQ^!+4Q{*`fz~4NF}X
zyM}YON7Gr3crXwZHVPBaRv7!_4S9^yi+vd#;KV#ix<%vbGJoX^fjeo}FT;bD{o8EQ<(Ra)vno=ZT=p$V`<+DS@i5!LZ)vp(709%@
zhfpsZy4kON6-kTwe{~D(yxvrZ>%G>ZsZ>-l*$AG$1W8HrPl+1Fd9T;em?DWGM#|$`
z$RPRu5U1|@zd|A$aCA?Nx;6XzBwU9z?MT;Ld=*#Hh8p
zDDcI9jbuJ-b@+2W=*3b(x{oy{jT?T<@B{T?)wsXqua>e{CSo_yu758I1S
zbzD$t8s=OLiq8-GjyYhj$oQZPPt-2ff!f0e<;;uF6qkF$c9HvDP%9t5n3K$2W8)N%Yc|GiOg>w9GP
zcosi$EX3Q*o{X0MD$Iw`LN{XN%@HP!+vJzu!rN*RWnJO&>pb5V{M@zDq=GrhnvC6C%?Y(BV?sBFFLdsrDGH2Bovtv9
z#pPSSWzG`e34_MWB0q)vBOnw
zG~w9JYF61@p%|SpdbdCKk=ywm-Sj3P@gqMl7xw(F0!KTGpxnmp%voAtiYIwYiQ>e0
zFK6f;5>)V~jTv4MGS<1gVW`u?b+CGEjaREr%eJ9zPpQjqELr$ed_^#D42ud2iKPCZ
zApJ%L5-)54Q6uo&|~vvwIiYoov)Gu2Spa{R2_hP9%QJ
zQIYmXk%vtZYs@uIi3w->?L3h@h(&#}hE+20CVhy&uV@8sFL@QMf@d~9y#=hGPzi5z
zd-o95j(r%3sZeY3&O%>v@bTT!OVl}`^`Qu3@;`{
z&f*&k-dFa2%y+NKD-mShE1lOGX!|5HM!wp>#fr_5N5T4S1bv)4<$baK^9}G$spp37
zkxMf(xje+-9kdVFfLL!&hvV^VBz{&WYHUyV4{I_;%-y8EDd1aOii{kdqBJuIB(D3{
zP_+l+Ag!!Y33Tmt!BK
zzNT93fIV2M7j2nfseBtE;
z9}sRoheB!m~g`t)OSPxG5QS+fq0~^N1uiu)?mPoT+fMA7Oop;5&O?%K7
zzJ@IG)*qibhLeR#fROt>_dFJGew6LE^~g)8!NgyGw494o+j(WCs_D~ugi%3
zd%2$}?U9eKOodMeq)kT-rwgX)D5AKydDW4niwhZ={!n^&`^5>)Q&NGZ>df|=;|W_h
z4z;+PH1FhX*NqIMO>di^ik$*F!Esatur(k;2{maDdH6-{?N_ulaN0)c$;ONZA-Qv$
z|AhSF=93QF&-r}~T@@i6(#6j2>-={fYHRQ*aFLm#a>7`Em5iw`h7vBV1iwlHOT&7P
z-9MBt{af|D*@7V>mANTOaZJk=y=cOLh!0)|+rnL1800ERYm$Ej^KzaC=5@Kw%|9hz
z*peY1-kFE=SElLCV7zNb(5l<<8Nqfx8_?5Ej%ZkcF`5MNB@Vz2{XPbC%;4+4wTx_5PdWbCH)xC9k
zK;-mOy@3-+DIUt`EyS3=j@@SWhM1(C_JvP!R~blk-Es~?zx_jL;CK8zI2E0#o`l;%
zc~bmkTfbIAZ`4@zAYCuj*8WzP9Q~f%H6*2IgI4TJKE+2?*ydaR
z6jRu5ZXJSXFTusWOF$aX*zXH>ROwW?ha^;>W%2#)t_MRjKl%16pVBrs>-B!bQ&(yJ
zw%LJBH}5FPWmR|>fJYEIViH3I2#)^-C&hsLX+mG7KsaMTjf6MCaG9|LQt2+F>}##x
z68Ib_)Yq3#X=q3G0^xGkk_<#{hlg-8sSuRwAZV=BjNejf%`PS>&O~QoYOv~*&ad&r
z3=fdGl#hqNFutgOYtprmb&(L^PbL)te9`G_=u*)f;sE8)W7IY
zd3f~9TUF|9zbgRsJo+MrNd55YN`LJD2b!M@G_ql>+@(6g?fbG#`jmX`IKNn5%^U^U
zY7QQ}kNPBxi~Ayz(}jqgLo-jcSx6~8SkeZ(DMYeZm3p}Pp#9U@aw!4K3y3xZS^q;J
z!u5c({RI`r2SjA*-O3*YkA~IJkfAGzbIj3MDia0GsUbe^fWbTv$Owl{PBwSQMSSZl
zxj=JZPF*T_b60vnz^r&XAsa?_F@o2@8lzPOZj7N(gZF;^AIgDd5xEuyCd8dOOkoKD
zg$NHYYukSe~A!5wx1TA`q@?rhQCZrg8Z|wTsYm#iTbO;XE=KMif0@Dp`3Mo
zH@5hP;?fYynzE`XS#oLO@pFXd5G@_yiq?Xyi#s#I+i;`K#bzK3VE#qHfDjYI=WcN*
z4&6hIsp%JzarD-b6Hn`rgC-!?Ysjl(yiFM`xo&!*8~57;{^_yV7I7oJaB^a|t+IgI
z4t5IpqbrOG+2c7nr|Hwb#t*!NFH$}p^2nRe;1)VnL7ttx$uCBObUW1*{S>ASJT!;B
zb$`pwHd+~~WpxV!<_kpKq;gP*S!$c2Q`&EY+n6sedA@(pN!DJAo(fA9X590INKE*9
z6f35&I8q7A0JKa0CSw>r+1Sf(H;M@T*TX}e
z**dgFVvf$gPB@NzS7!cWU0;`XBLxXlgIf2rl3aR{R_%SKXBFj&gf%<@)HZ`@IZ%Pb
zCCX@Cac|unR2@x5?X@>vp$`ToiHF*Zm{%O7^^w0|4*j;XGRZj)BKwXI*hc0Ir|mr`
z66qY1u;2;2f?M;AHO7=Tc9`cDly%5?x*MBL&#b=nK?60tzGm>ggvjT+m%XX;qE@)9
zlY_iBlZKO%Q|bSq)rFGnqaGI`YcMD+@p-_tL?`SfK9y->HOuwS_KjeD8s@k5v3xc~
z_@CqHxl9r{>IPjs+N0m9-(?{QkSWIBRUEoUVg&$noCxX)*Vj81i7pgNH51eEaIeZD
znpM)f5FW+;rf95e{pNrv+G^qt)dJx)hM{5jRjKY5A%SB`M-4NP$Fke;$zKhX7Z9ao
z2(0p=OF;{knt_z@AR5u4lM15P@PeimkI%gCkoJO969WWqhVtycfH0~^KfjF-;tl^k
zNALhtkcX|s_`=UJCJS1J$(9K-H5kgSMDJDfJUMvEtLqC--^JqjaS`1!Ik)
zMTTdAI0mjfN9ZC4FdrqHb)s)LN%dAdo9(ryl#9!Ry|nkZXRq~`wTs9eIO;|SP_$&!
ze6=Uz4EnJpugDeqBO9))X{A$t>6caR5~TnnAzfhIUmxcxa@CK=-xB5Q<#LnJA^ROn
zSQ(`gC~*dx$2=o=<2p_XV`G1(zQFUy{c8Za5NyBV%qbbYHkSMHB~u!Ut|C4vxkX3s
z3!k5FDpfCv>kJb%Q9LWr@(j~nH2VBL!N4c>hHS}l+Rlpi$t~2L5OY?}#!K1@M8OEK
zsmm&Gvs>7m1Vcm6p$rS;zOb5>dxO!Q=_Hto*03T6b>GW{w+bIl_l`SyqzGpsm_gWd
zsyR0AOavHpAzgO&+>B~r6lJ`CV7o-zU-aEZMbj+dVJORl>rv3bdvW&s1_8VTFl){R
zmIMY<^aw*8=guS|p4xRj2)mE1BNec+67Ja?RL|Orv_H-nsJS|7Zp2`HKtY$yuG>yk
zu*$PdcIrp9h56;w;K~&(iF$#DR<$r1+QadzvDZ7d@tQ=Q2ev-Et5!*oj4p$0i;n1U
z7|h-#5H(_iC;CcmC}sRzCB$IDOfOYft)d*{3&BNfxSJ-fWOpBQ(>(8T1Qs@-wv`5-
zl=7MX5`-Dxd*H^K!hcrk9Gm3HjCrK@+1_w&T9GsX$T$tSzC8ft71xv4GhLiC2+?OQ
zVB(`G@dEkVFu6%P@kOI1n(NG|gfXHx&wwi+AQOq_fZ(vwM!E(x7EQ^XW1aBV+=pDh
zGN;V!`VhAvCUaD4eYY9F`x;}d9Venn?f7Ec>G|2Y4*-Jbw18CzhYXj~0#hsk3rHXRmNn&h!-eDZ`TJ#fGpuG@c@{5Yx?HyoouVg6_aLbM;!8kYL
z*>8@d)w=$#^+JPDcU8GWIhm_vw!_gd$}&ETDHi6ZKJvOMRf)^
zdb_^xNh?6jR7nt%-jY?=vm5i-4*Y)jGTA?_p5Bhjvj8GA1{fJ*qZ%
z>WGQNH+9E%kQ2
zSZ(Y?faj*LGqbKMlA6r)%-BG;U?=cJ=AeTuPu6#W&5Q{R;=CNwx1Ydu_)A$j9@0(?
z)Fr^X%}z5j)al{(e{wzWsOMeTbONM(e;r#UPO{y(+z!eYkO+
z9G=#JktPDriBxTT{8_3=T2Ld_lM^4F{+XK>a3rHPh=_ne)dygDO8vEF5iQ5xO=^uR
zRK3q!*AD4OGl@_kCY?*Qe+}+Ua@iV|<(iXO3?vccf!G6fuh58Uuh-a}X6Sz>hh)7k
zj<_qV=|sxVGlqQp(j1A47QR=a7hZxOcI+lk7rig-9*7s1Y#(eOq^
zaBBA7lYU^%nxJ@tG%&K&D&)i?)>I3>2g!o6Vs-7|YJck&d5y$^U@?w*+F+&wbXE1n
zzZ5@YYDn}Vh-c!_rkx8KyY`aHP*5?&myBe?uq4X^7=~k(gmKI@>}pK2kf5b
zfqR^}-s~Y7bT;|d)x82zbo)^u0*!pGZw2Or=eB-AmRoaDjOWplJQt%kCAah|L}I54
z903-we}mtpP5aVkJ8RZ+ZMNC;4+=Nj`3s}TPIysjh3!4kG$>XrlHU_F)I
zyMAV#3F@5)@1y#t+@T=wWwVYY93?u?eY&^e61oXzc`>2$-tp_B^6$T-6Z@S9d6~L#
zhfN<&rRQUbHv}8Pc@v@vbJMED0C$AaB`)9B4QJWd+PdAJ$YgRHI2jaxipwE=f?Q{R
zS)wd+K4`wyNQJW^EtEx*Q*JlUOas29YhM``>vinZ6tt({S{S6THP|}M%52D!w6&~l
zGZVcv?$L_6Hg>DHd|L)l7=Nwc=q|dSv74EH3~uF2Ny180RAyKl&4_y={>O!#zvU!1
z9J_F5Ts&@tmeF>bN816w@I^QOQ@N7b`cpcdQU6d7-V_?P&*|~@PgSlMua|mNzaFZQ
z9*EtWaoHaGHNOvA7?-dvtb6Q(a(87yhKG(v72;be-S|n6&DQoCDqTH9
zIirXQFADaZ*6*Q?gKZTbFa+b&n`B(fs&Qgkjz$jvsa
zyo@c-y>K%vUji=$xA5|+2v~WPru8yhz1Ss;cgn~aO_)wJcINES@OnxK`>6`nbQxP@
z=XE!!C2Qd8Ii;dx${UUYS$uLA%FK^UNEGAM;8is>YLw&e2bU>8S{(*}3XV`O;Lfg>
zxKvtQh2*pKd(m>~I3&g^2qeLyGZk7ZxNn;9DU&
zB3h*Dw5W%nHFc2J)<*bk^T#v$h+o+V#PK*Owk_!Qm~mipC;Yr}X}p?K
zN#hn~;~&TJrs~MX9hjbJJ-R3pUmLaq
z)NZmnqU?!L6tW;3Gc5IzN%lVS!>VC(5Ud@@2)Q4jSp?%KT!s8?|
zT)Y>lJ$Ux>@0=X9Khj+C0S^5i%dxbR>s`CFuwLAGk!32y;OviaEHk={&FpE
zTN}1UU<7@thZR!8Ml5#%E2y-@GsSou6{rla_
z1l^-QF0Bv;@eScb0zTJ*7K?JgK>zIZ;Tx(K4Q}&4t+K-L%6zKbl3*pIh0UZ=5NPFJ
z#2WXqYS1Gn(ZH*%(Ly(^hq3YS5YfbUcoZHNw5A&S>3OWwOQYwS74T#nO>6z}y1l0J
zlerAd=$jK8^)1RNA>*w4Ev+&#lU4cGyer`HBOMzhbCU{=NJZjV&%b7XdzHg7NQ}Fm
z@0Xe!S?B}XU0`zOORd<6&iNtvljxqLvnG5<(Olfi7d0#KS-R|v>G>@(0$8JOFFtYL
z4tf|Em?_Mn6=AHEInX%0jTQYGG*U5Oi~54G=1{K%RynLSK0*}f6kSULQ8w1xEz&?v
zg&z-E(RvWlC(SwHOdsz7$vj^3$xGLe1`?+|XRD>aK~JPRqEsp^e0Hb&W$t8Chkjk5
zaSy+j1rLG#4_A02MvqHFVbZEL&QV#I<@0)XoT{i)
z_JMv18$!y_^NuzpJ0hwk{clxOO*d@rqR;>;CGj4pN3>PFyHNsOvOZ36xBV55r-tb;
z8ky-mmNJha%8}vG7+Tle>8^^^WeUrwt8&o4^-qhW{c_O^9M9rGHlk;(;QxF6uj(BY
zI6|`8Q=N0L=!*JXY}&Oa3vd?!cACTgX~=S^`IMo
z`ExYnws?s@;;b??;6Zbe=<-R@fM`a56+p??L3@KFFbsxeSckYVd9e30@_o5{&G(Y6
zRW-d;|A7?4i`!Hz%*x3_e-`ALP%;6+)9cvQLxUn$;WycVgCyU8ap!WKx*F
zD?_vE-6jfj#eHOwWH!-B=qZ%ceRQt`-)G9zp*%OwAcI87$lCf7kwQ23;?dXvW;4dh
zRrXmeoOPVHI0>PjwR=O~OR~VmeY!dwVBu*yLV}eS;iz7ilsM~=eQ@Ool`u;{omibF
zI&uAFL-ot13^yrE#$*06j6}!OsRrx-mDc9MLpqbP4ZVWbMqdrF2GI1F*}2Ok=u8od
zWiRD*mN|6$8-Q3r^Slw~l*b~x552tkvmyY}=*SuP+Baw2B-C*wiGQDW+x5mw<0x%7
zx8Fzld;bv&2j+R0>%;eslKW6#q-&)|h_5#@N+<2Jx@N(a#;__2O~uUr!_ZkcH1&3I
zobDRkDM+&o>F(|hiNR1Q2S`Xbn$bvZgh-EW*Z|1^0wdHZAX5oJoQfbn1VMRU{)T(+
zbDn$7_k2F7-Xx7oQ6ePVqaNlwAD3fbG$peuv*=9#?Yf`HIy&%&JFn?st^2o>jE$Te
zK7=DXQU{^a2sucAv+Hr(ADZjhKDhMZO1QSw$u1
z6xVKMqFBXjf5PzW0A?{q`$#i_
z8PFqHbj0s4_e6b(vT2+9*!zIZ2Jfeis?xSDJK35f`_9qH$k4u)YJU%(?_^y
z*>Anm2h5By->(qwK8@tUJ3FV*!)oM%^Yuy3RU3F)P2W&8V2ecnauMd`ML1lC>AtcT
zyxcO3+&z0IMSzH-OUVQDP%W_qb+T_Nh9*y?SCopVQfXY&-25^O)2C7Iy>&hj2!0EC
z#-zVXU(C*-i!Dq7R^d%z1zPj(sv1YltxlM
z#?|p(1Znx88<=9%QDS)GW1nA^m@9paGHfh;T_PPI&IUaeqI?PCsP-U~vg(8_;
z`&jHKMOeBRg(ae`m?FOhn&lM%xY)mV&qbv;ailfZHNmRQteVUASs7IFL$hY8XKmx_jw;hbrTlnmautRpXM
z5$273xSYTA{tX4ndFZ*URGfYcc!s|qgPUo7xZIHw)qVr0c6=jk%at^LRQ?$xk{qG?
z@OflUGEK2(TLb*eCdBbTx(urc6tluHL#T%1kztwNh=1d|f=rjB1Bb@$n+Bw$?8){iFa;fn+&3U^ltAZRsk3X^?u
zEMQoF5=7IACGNM@
z|HzzieiyaFIF}Rm9pzgJ=gP7pxgNA*&*R)$tA}Bfq0>s`YIS%2IIrd{LUqaA04Z8I
z_+Bs2nG)zsLY^6$WSElTsdxhd{Y#fZ@wIXu833=D
zS|~-H@qkj94Y5K?D|aj}vvb6~1CV65`U0ouqXfpxp1emC?Z?a$e0JEbl({~sdo4M>
zErgRVG3nF`;jxG)DisZnXr5*BTZY1_i)F|QvDLCx`V^KS=-;;tRjI6_)nC6J$wBY%
z<;liJ30%@C^a}{@8(7og)t}w!J_s>Zk{(8O5Dlp_9ACwW&V`dPGc}jc=4K~+g)V1>
z0(Jq>qO~s-{nFgo|2mwgzDBv2VduZSdk#P65pRXd)Sn!;IS~d19^kZa;Rvsm5m3J2
z*mXmuHm|woTiwkD(DxLVYky4|mA@V6Id`9@bpmDkL5@T8LANkp?4ABcX3jqOI2(8O
z=_a}UFDhZg+#V}I6_SLl3aDbJmSl2wc9MR(pqz%O@a1|Mjo?ZB6U4$ta>;B8yWxLT
zS)-tIc2A@$hjZSI$#O)wHjUJwL%k)5ySx&b$UW&PQVz+3%7sZ@1pGmK?V2tsq0{(W
z`BMSAwct48a&c#!3RLM0FtGBw&+X3*3;g--f{A#tvmNo8nG^*4{-nAiR{;sb5bt?#JtUzdZR+r$T0=6e^6aHSNS#u
z|J6FaXIeH*0@&8SXysw_U$i3D1$DL5c?F5ak!Z6?g^zJsEy4bixp*m!P1X53Iv!c_
ztWS7k=O{M_i;4@0okkx3i;|yXG{S>Itw{Rfsa&W<#Hk5J%>F-1^*qGLjsJWE^&=xu
z|BOC2_Ax;OsZ2%N>Qxi;MCN1mxHf=@#iZUIUibmhtG$7&>B!jk{5Sm>{UF}jo8fWw
z-fz>5^({puKtdkSmhp4)+q)6mbra%p*qES@+H51_+y-j|$Z*cDXk|~KR&^z))Fg%s
zCm~dIEDGj(%6M3|LASEG4F^P6m8niacIj0;+*8yaBVSV<-
ztnmhFLu5ebcOdrtUUothfc8RH{2l$=L?AzUMy5=wWBz|+6dGvkblA63Y9|&!WjZAx
z3!AAZBV!;(c%3Jlr?GaBdFq^$eg+D9jwG0#9Z|e!54F+>lU9}ueQ^EUHGHtG0x&;k
z;8DIL^S;^Fqc1FSuLH;%Fth+8
z)Sz>RLoPUTBMqrqr{O0OD(c~sjt@+bY&w@)^k1coWr!<p)|uk5ACiuS&lf$A
zduNs8f%l7`%ct3aec{Swy5(=`g4;AzA{DH`B^HUiubGXkx5)H)+|)_ZDMxi5RFn=9
z9p4+)^bP|tM5F2!2f)L{`D)YdDE4r!y+hmt_BwQ~EuGPr>$VDnhl~B3d$j!h0p%^{
zU?mU4foBdXSt9!>(?zDYAu>x@p0QKN^fDQ?m}rCkd!Z3W#N!*(qjlbVklW5cplfD~aVAjl%
zW|;7~A{PA2QzH$Zg1Z#HZ7qW2bS_YFdfL;fOHK(%Z`(kA9{kTN($083
z<9e^&(gLZzHz*%9YkiN*kuN~|I1k=Nxlwb(zm~BHb|p;RIR|dcdY1{R9%wD`e-*MD
zx+&*To!m{E-S=0NF4XASa%_Tzsrw
zF@o%C>ApO)t;@gA)N8bCRu*%i?|RCN#Q7pePG*q*mWR1o%(4xzR!mVI@Ztajk!VlU
z5+j&ZuK7)r0z>tbiGRQ4hzSA
zBKi{62}*
zrL5e&$c#KyXu%uTSbl=9K+An2a}MKTjVNeJas_3k7&GqbJDmtxZMeu$_n!V`dp(HY
zLK#I!fdfE!1{C>;5{7c@NpTpX9~?yXyAod+Fv*(LX#$bG>w%-Wh~OPC{zsDGho-6=
zEMj?4Af2ovZm!yxdGA6Z0P_kzkI^cJzF}29NYQ}opbeNkEZ{!HGnzU!dP9Nsnnm}d
zD&&bai#8w-`}zrI{F0QGzp+Ze>v$?d2f;(PDGKiqj7V-luMA1JO#a%R-ed334{=6Z?Vn;bt^`fLn>jLmpr6
zEDG0ka1Il~;ni0;;3?p(zQ(wPAmu!8J@(PqVh!E*%W`3kDbPRM_nG?S*@yOd0ycdO
z$n4WD_jQ;cQ{8OqC!X(_hWkjSOnD9P*4=q_pJ|iX&NR_Ze0Mjc{aC=&Z
zmngy~k>P|kvkx4Wp(xOeJQxTpeJ~G8cPBpb-ghFgwpOqu4?lU1`b(!?=Uv3I|2Wc8
zwjqC7e1iq>&;gndEB|pUoY;$2L-XSo81DJQWyLr0`&yq99$7`eVr}2+{b7HP0Dx0w
zFECK%RKY29p_(Q%zeF`go)WV}v;Xu=z#+^23uvU(C!3Vd2ulx$Y|yrOroJT0^-cL!
zyILw#7CW><|Ek8+DLJM2CQ?fQ2!Y9d*K0MD>t`~S`eF29s5SG;>NU<^UJo~-y!p?{Fo^Vyylv|(p=By*Zt+*o%YCBJTcrk%M$hW(N>|2qtx`&RL-nS*
zrUH$=YZj@DH
zp-{mgP?g5=#G-lZ@PH#HpC%qJ)3~5KaXDp)vKugOa`Lu`-N}5Obn&jkf}*zXA&C*V
z_@~ku9L!hwr=nc?z;VOK3@{9L0aS7y^~+D$N@~FY=K&wL?=Jf3H}m$)`FG)=9rI_*
z;Q;r3B$}Tj6YeQvzKAyyNCh*68p2q#!HmOFKDSC)rCmw}1zT2$P1BRR2`q^`m`LaQ5lqm=)t(Ny`Rc8OQLnzWKV
zC#XDSW^J52N
zkXn*t$4>cBFD)0=8*ZhlDXYl$ds)H#=PS08)AwLS@Cp|frX`cLA|;aOLJUzbjV61y
zn@OB|PF2pHNGZbsn7l%PuAk%?(cd{y{xJM5Hr-FO-8WIa2;dD2Dru%BVBXz2bS8%*%M?c
z8oxz|@LPVkCi!B8Ztc$Thj3xp9i5j;Zi@9=<(i`W59}^&Q@gb5_*nd^#R^f6F(MJ@
ztjiRYwD$&DkhR*Pm3=<8*Q90_9`E_&jd7c1ioIOMF_*blp{ZXOF*5=(XI#2B3Xu3iW&DRVk3|x6_
z7VW|9OwpQ|@`jwIejLE-xGDVg2TIi>zWfk*Z2VXPUP~nwm3<-}OsD9m4ZCB&g}Bo`
zBlu=g9f6-Vst`!-)@m~5eq(fvc%6rz{tMi-mzVB(|3z6+zmd0`QlZbZ1?l@IYb-U=
zj%X|O$gc^+LT*9%{8Do8g_ups(fxfd4`)zfy`<+%OY8Xb2@lQlZryJ4M#@I?FUk2-
zOg|C?e;M~lJ+1i#9$Cl+Id-qW%j1Ysf+k+>w%|!!y0u(@-LJ}FrC317C-yPV*`+)3
zDL+NMs(DNG4I?2BSQ0-nAaM
z{<9}9XP7MSllXX^_&@$iQ1A6|phTOtvzgMHva?yjy8M|k;(l3!^O4>$s(r~ozvTW_
zhgW#N(!=CkciP;%_ZA7_f9e6*A-ax7&ep?NccTavzh=JLT_vLPz=R06h(|};>cKsF
zQy)ObC&Zb*kEG6usXOkFPPfQ=lJ5|{j4f-s##58-P6G{_450{q+@KY|#NtJ{#I;s_
zTCqdJb4j%=o*RK)AiBH7-1{!8$H2I4nY&UCD)xx0>26iZlDU!p*1zJT8sepO&dd9e
zvApHM&b9PkQKYa&=H9%Re`%=_Fd@>6(|RT!SyMwvu4
zN(Vg<_DC@7>17hEg(XT3aj9s=UoWF2ff*s4|a59@{lh8_|0Own&x=3
zd3E0?!X`hl39Tk}3|nM9fcH{xtCxm3XiKh07$cgV|K!(9$+mYw-3wifYzP(Luw(R#
zKdJbJi+LvO%&!wm9-6!&3ktrjtpGBIPhKB@T|1*}Sp3kmOAdGkl3$x@9$RGWe;NAP
zj|*88piA4&kmLn~vV}0m&(L5vUD*G~#;(c?e@9xK%U1h40S{zz;gV~P&N@%ED;Nao
zXx*#RzepEsrUKDRnp52F=~SK!ke~zm>ypwS3`izB7jF{cnbHHI-Y1M%)$^Lz$KyRp
zo4|qo@N7VL3EsA13EBKo`GM~i&ofPo$=jGuK`R~~8
z^}(|jv;eM$e6?(F9r~Ar^BkwU2hp?W5pTw14Zj=((gDiZvZlb*-EE3?ftQE=Iv^3}
ziA{Pw{ra7oSvn!}Ic|JgqK5><_Xf){H}1k1~Rhaq&d9$d(fEuW=>xI$iQ6{}oOV)VjD;F(laF)WIZk
zu0z?PtVG_9E$|*61pXER`Y6``^IkcBE%+Hb*EAJY`Rv!UsY9vM*qZfiP$u?Bc6Udt
zZh^$fIvFjg3qPBSnQl}ir1Jdp3C`Q05a1vygr8gz^$
z7ao=UkF3;RVT?E@_rvQbFl&mLeclJMfT`jYK^a&qM^LDnjh<4HzDBn714|PaZvy+-
zSum;0r?3>ZxALh&?a|T(VH~eiPOxj`DUA)%Xbjw?@l<|IpD)SETR`Tdbtk@_JFbyRwlmk(=A_2>0n8G_(y$j<#!eRF@w5A23ah2SKcG5QI~8*
zvTsY2inTQgs5K7U!fvy#jKzm@{;9~73L&dd$Wio*yp8iVeVc^Ne6Ry|Vzc=Po-)0y
z7V0Sw(t>~Z+AyBnf2rx6^^E(KBnq*IAB4~Fl^h!wGQ($>AS=a1f@=pFwdCB*KKs4O
zvPg*SY(Kd7`;5MD7pP3!#G?j_q#o?}O1_WQyTw2vm)9DnV0bllPubzJ;pP_9F8kn&
z{MjP(C=~F8Lwn*DMqpg|fw#3~*h4VUI(5qL-C)=!G_p~8CKqeriPSNAOc3T}fo}`x
z^E=(5#oTr6&JGN8-dBvt50OaiI}&+J$%(A$@cN>{*k|Rzkn}-`xphs>$}NZ6v}6{!LrJ8
zj2AYkni_1)4t?E`9by4*R7-6d?4m%n$g-Yeuf7VTo#QssRKAIRrD4BaBx3DLdf}uB
zJp-ftLaPyFt4$EsMZL%R#2P8KH=wzKM*gL)W17UZqI*lE<-=c1&v_y6zpC3i_mmap
z-%06m4`n~l|27be3GLYAiRY-m3^R$Ec=dTkFje00*@ua(=nqO80i#ty#nX!aj3uQ3
z1ewq2R>W5|@b8fCAI%7lAxn<^)JvsfZaT)%0*kJ9SpLg>VPrhGcSf%CUY%Wy92guU
z!S+a>jXd(_r|)>IN^}$q9*;tG&hLOW&dG8(iYcZ>#ZAnN7;>EWEK+68*;K+gEK0BI
zfOL*_@SK|kIDZN5LdQwgPF-8S;JqqkP=f3y5=}_w8{`ABt5@hKp3j2TA5`00eZU_#
z(OQ2p)%f$5)4zzUh;j2cmpLWfN}_I~5O^Zd=W$$~K8%n_M^ELz3u|}1P~UfbL)2uU
zey?x!X}4qvz%iqRx)Tj3%dEu_mbQoYS`7l2`u&!pirvUiGGg1ugG$Ol>b7iFr@2VFle;`S(DKTkq!GxD)m_*1C
zA_n<6;fwo3Y$_a^n?LVRQ9}aU+u>ltzdWVinXVlG^bhb1U6~+6{8I#-zA~G?;s?|2I3f5)eEe%8+KvE19q)5W)Oa5NJ}sns)yY
z?j_qycy?dTfpoan&|NaT{K?cjUaNnqTJCF2`F4)G^Eo{C`lDLfuiF~t#IDhb@z%xV=Y1?J1#
z|{U^+Qcyhjq>G6{4%&s7PC#99{#{VGhv;6B
z`H#JO^^x=3-(A~t{>`?5Exm;3e?o$%K!yf-V}x6obm=0WY)*vjWFf(MNw(~p6H9>D92q6mdp;{kpe^6a
zS{ZyWy}Dntrg*A+--+U~RhOi)km4&IKmRG7GGAlvc^^|)iF;N{YpL;a0DVGRnTC~%
znH=1X&?Hh~%}!vrq6AW}>HFW#)%|mnX1EdfDj7%kFWC*eeI_(fq8O<;!xlPT{Mmm3
zZ5ndb+#|wY`r0SSgmFClAb1ncDl13ZLH}%M_0n5D7d%=225TgnxO%Q^u4AU7aHP<2
ziU7Z(&s+Syr}sx8c#L!atJT-tt>68gS_RjrsC8l=a{>f4S;TGa>Tn55cf(e+eN%R3
z5wl<>D^0`B5Y)JWOVFs6}BhT@cE#U{~}M!dn(J1J%Nnmx)=yrzavyB;pY@LWarf_rok0^7#)UG;t|Q(v?=#>RJI>M7t15=W^T7-~kvurIF=#n3lgV_WHHXuSiH$3PgR|4d}+WH-yte=!ltk!*!Q2f-&*D{FnJjb}l4{zbR>Qlgh
zkH-olF?XNLCCmxz$3Pd~6(`Bqat2~yWxOZrHXBNCtpQXXnOE6ZJUQ`Y9|76C$A|h3
zg}%e2IIr}dOM|>d)5s2|z;KG^EaYwZL2tidyxIgZhlaDg22-;>XFcV^>Qn>SGPoW~
zt;3KqjX}E-TYHk{iT^aS&6&%@bC^Ovjzh1&U-#ZKC6cI+m1UIR_$ppx>dd%QS=!$n-#r{z-*f@+))XFV6Y9nZ6(2
zE2ZLLG7W=DkKp?wyNX8i50g9Vxdv9aWIa0;tC_r5#rhfL+lS`(h*EUxtsD5~(Hwan
zyR;=-R+URowkU`)7r%WFbGDhxEVOL#RO$ZO0CkAMK?3yEFdY3Gs4fC~IH^qcq+&+B
zaFYJA?2cwKF!z^{YGMDs*&t%CJx%q~y41gvUjTdHD@*+(oby?@q=Y}n{zpedGbPh9
zbB77S!0(PSkqg&HTX{cJTmbtxe|pA{QguG9AN8BtiQ0U({*p~nukCjZo|YV^B2Il9
zB==%YF66%S2)v<@Ivr3DdM_8pOnSfX(+MA#iZLsI=(ElFC~!Zb@^zMwT
z@dvo3chB~bAtbLPxJ5Sp$RP>dnxRao=-s=d=`3FCg`b!A8RTiYu$*?HdxO;5M<{yz
z$Xq*ftaE&}sPZ`Dn3%aI@a}r&Kjv`H`H1V1ql_nv)YajSksf0vFYJQ2NNS14QmNJX
z{lUNCL<=+U%4~JhBnUAWQXf%!4@owPO)1288!s}8wFv>}7~Q|Z_Z*IoX#uTq#&mS9
zOBSb?^tS{)lZ$I;TK$sLG0{};7PVm~9?b=n*BMDRYegPSQ1xeDc_a0i$d#f^L{D`-)WGn`=uTJmUSJ+gGt`vq@C-Y(9WhA*f+o8^d(
zp3FUC+9Ew#`UoNvRzZD2$(Ykr|5aw(kE^2}Rr(^nLMGQu<3Q&Vk!|aBD8;|Uh{1@c
zcsIq@?i%W)M+37^19OsQ=BG)sXVmf4ExL90?*{|lV@N)=^}+um^CRA2I_F@cJcW~(
z!{y#V4g{A8Jr}9t=5~GDXPl9{T)Sk>_CPbc$g4A&m~LlU$33^#wd69Boz}%k-YkCV
zxhDm0?5&<>{IfmqiYBPNuiM&ndV`uo7U1frttF+`r5f9V5I$VO+bInRL7@Y@1G=VS
zQ*Yx_rv;gNqB664zd8PEA})y!3Bp&TQYQt(EbP#g)xT%MLO3mDFVo13zFdbW7aT*>
zS|8tsl2Pj*E9`}Cg}~DpK9pCFmUn678Kj2ue$#%M;_IZafXRcLnsZXw*^kyYJwkCz
zUhird%AUwM{lkA2K63xa6E^#c)I=#|V!HSc<2l$GQlk+>O*rS`Tdc)WnUJNwbCD|fC87BVCGNFD++)Aacx6@@3Y5gClY%ohu|^V&wmEfM0sR%v9x
zngPci+5}RT?oqbj^q*afIC1t8*h5X>GqvbUOQbTBurF1)YE-5{90j>k6ow88HTN*?
zp`eG!+AJSmSTg&SG91vJ3y7P2bsIuiJMqc@U{?KbR*MiN?
z=L!CkISG4Gw5y#*$x2HJv(bBRU&K+4&EJOp^4d*N
zmpq_!=16n)weIbkP7C~&n5Q)od?ci=kr9#V7~ez+GB8~j_~5E^1xHe*gG@KJti3^Q
zeH{M<4ouz?MD;R?(aJ0h`Mxynh;5qE
z7Rp_A4{vaXOKT;Ravj``UEsr1(eAnnm2tKfjXnJz*+Hq_B&8TbZp(g7k!paPv*#Jt
zM1sSCpx7XxB+79cu2%3bzJN_$L?%3k@v2-^$sbY9fXo>jI^b%LCp_}DCdj5>$l0cj
zKnP6^1K$Xjk@C_LIEV1-l;$mjBDVBQMg)=Ts*`Q%-9u`ooSyCUmrj^BhLKY^({3D38Lf>G+k|ouGss_
zi>W|GU?b3Qw&epxK`RHCzEuJ`JQ8f+o*N-t@MCX~7@-vV-uvH-
z{InSJYEK((VfUtJ^{-QrhELM&g=mxzmcEWI3n*7F-kMHiDf}9$0sw81b#}w)3
z(L@O2SNNU$&tmV9$V6N4bTjXA$0m(z+HkfLe1u57zQB$>r)EDl=0+Cn>5F4D%oYm1
z;0@FdOd}F0|KmNto&hFs{CGM_tAYK%I}@-?EQR~V9!NoXi-E8Mfl7B>R(Lc(m(JwF
zN_`!Z1#gMboEjdu%jy{1k5|HUwrjFDUl*ysRp)V!0&ttDo2|a`L9i$I_a6+C@vt$2Gl
z?I{oOBSgF~+iaP
z#|41=6$bjI70DpfFB;YQ*-(Dl<|4zlAL)3zgCI-MIsnhf-*&?foWu*$w_sYbkj0&A
zF|JRfu#_|0Ox?NclMy!G77iI`V4r5g$lPvDg5Aw1a4*{>5uz3Ft+vS+Ah^Y#EOU&5#wgp8u>9G`ivXis%lk0`)VkOH>RXCUc-J>FEnv(n)$;UVXJSG29$C+
z#3?$>A9Nc_Fex75j``@z$Krl`4nZHL4%n&7yJ}dGJszacz2Cf==LslRMHR(}BM{!T4SgdK{74t8$uK!up<{h_@|h&KRgjf~tyXz(
zsVVKGiZqFjLQ84%(ShVERmXmr`XTU9)G3b}dJx$vpk5uUS5f#{r>Xd(SMF{z?V5uU
zrWca!xZB8V@^|%N&c^VtY*yqSTHzyAq$`D#4cmSah?Z-dxO9E7+<|?q=Nd2?#4d~_
zQVJZ|hqyWZd6^saWv&fA%>U8}#CZ-OKa(Nr30u)TgHY9gZdYJmu;&Z?FlO^!60lHjQmUdU*+P1#~wfG*m9Hj$f63
zRnWk*ef%tKXyXHAfn}dPAMitL{3cr4;-{#>qL@167N)pzM+I5%b0BEK@Y
zMDVWsCG5blz=xG2P^6NhmtyT7y>coaDltSLv6
z!ytet^*^f~n>6rMI{rOk`$j>j->{%%S%~givp+Q7XO<+&T9>kzEsq(tk+KK3mIQD2
zoS4FmYfK0*sM1TCd)Wn6Mnl@j#r!V#6(m$`Eh5?$aqDaB
z?aO~n-^fRcfSvu$g~f1kJ31Ul`7@4ByUz}EOgbv{O10tiEm(8kbn?wDh1l(#P;V+B
zL!i`$SLv9#b0&d8hb>m(b3cyvYL^w&OXz{V7JKf`@nckdy9`oBkp~-W0%p;IdD!{@
zzFf+vI+c?hIofRh$pf*@heR_uBM^I-Nv-+V)?~z2icBfXwuajy{bsH4${MlsUjrU6
z^P+>xmV4mgO)dueg6>MKL%Vx%xdJ&l^Znbs*rcTjT8tTS<{((>S%FkhZ7Hk@UlW(t
z5&$#w+y8dNkdq(M%LH1ZH?B{O-*tSN6LTO%(>S5%Ytw{cC;wc2w*wf%uNVeMDnasg
z`CnCVv!W_jzPK~c_4Y)6EmOLey3W4&-8jRg#~e=4UEy*{SYR%AnQ?5#k$$nCMC?sQ
zJoSa&Kru2|Q&sKEce%JLJ`;PH90MG8|E(a>wyv!CxZsXzgWSeVD$LbawG(R8lt@CA
zKEwMZMPnUHXPGS`sG<;Ys54CssnyJxu6iAZobCZyjjuu4Uge^>+kfGU5vdz--{0$Ww5>dbXl9
ztMtCcjM^Pj8EEyVVxR907gmHV-Iqy|#O8NFo=KJ+@f8CP36Gg1;7qNN4RGbg35PYE
zxnsinqOl=K)>9yjFpqJwJ_|3|+}p4t_KQ(VZCYCX9EFZF9_p{(*Pv_7N1SpYYQza~
zO7dZ4O$jfvenjIv_~nf68E#8Wpr=pu=ag4a{=Z!!fB0e+ok8vl0PHb>3TzK`@tHzR
z{Be5Rc%dH+xGzS87Z@SX&%cBQY}3#&+>*S8>`K#fVwoom%$j{?KX^AN9?iMRNxnz*
zyw0p|lN?`Vqkz;40SH3V>9fYmj8Aif4DxrcVfB2rO
z{c6`MmJCy(*@;MIu}yW|W6qX-QS-~PDk~^FG|{R|Qs^!dyjQU|MP{{IP0=@U5r>7L
z5UW|KuTS@iT9NW{f?|*MS*KlMygjYIv1qew{<5A(hU(0s;X>M7%q7r4~jgR$Rn;OC3
zv?mY3`|No#{9aD(Oe3l~v&4bC%NDWF^nS(JmM?8UQA+Bu5s=p6oL8c(0{m?InFvz2
zFg1a$x3pX_g{h@6S#g!Bk#K~f@TKvr0AbQMayk=v2#X-ZtE|ZOl8Ch~zSf
zFD>?bCjv5jeQr|^8~c$$bP4fD)wE=zn>M`~Lw6(`wW)v#7&BBS$BnW7q;kJxHt1Sc
zGGc`&oUeihWS&To%k6quni%W|0_YCEvuS%WFiJB~Oi0n;mysn;6SK)G$R|8ej`=wl;wb
zshl1ZKbY(t6sr(Ze2Cb4zfE_qDiQvph45lpY3{=l#ly1tm+g_O>}Y#XLUDGs))Z=C
zA|gzYd_X92v8?P(Ynv1{>;B*RmYqH72mb)n?faz`xYULsv*
zH&6jo9^<2w>1s6@%~8^-KvzloO&7C~i!;lyU>
zeR8!;GH17&q&Su+87us1G&S|dFtG>!S@z$G2H9s7l5$Hkr4oFgnm4rdeS|VYe$mWl
zz$db*-QtxL?w>H`yS9fwzH-kpRV6C&hk-;yC-h}2r1{G#29Ud7)&UXYjzZq&jGGLf
zJ=2>8B)qpA14CZeC6V`=2D#pc7Vt}41oV32-o#OWICh>$LAAvug
zmlon+$hxA}*3Bz>R;d%^E*_(~etu=Yy0PJGu(h_e`>O*#m>k?%A6Y-Q2~tL2fn!?N
z&6T3wlh|^l$YX-%S2CF-L-^9|gY3MZtX8XueITa@m5IDomOVTVQpeeZ4`Jjy79vI>
zB0qk7K6=$5jX{`90kgnOL{Cs-PJ@C~8heqf}8%5M}xj
z5*O$uxzx3NBfwvB84C~FA`?)J+_BpfxU}*yDaS*)+fbZkbuAsp+W*Y};0Mb_LXSc(
z?nnh@-trUDjdzIYl_97uION5hFdX_*$t-s@SLxVr`+@Yj?CBX8}*c`lm@rFDq;xhHEwN%e}Hc8Rr(lXP701{=QQu!v!+5r0w
zn!nXePTTlVBAwEL6VwoyX!3}f^eMZAVj*w`|M4T(hz@oJjB2f^Yq0mcC6TFtbuLJZ
zv?(rv++4GMIdsqo$L^HZ;f)WEOIzud1Z?r9(`dT7svoTA57eXX)e-TTJKLR7%2UEw
z9BcwNrw?({Rn|CONu737mD`1)ccAF3+7$u?
z&s7$a&&|hNIjRTUE`AP}_m_4b7Z3y_|X#s-D$*Oh;87VJbDB6&D8zI1Zz4o
zN9Iya#kZ4_DL?f1N*o-vF~l!SA+j&}zOfOPBCk(*^TPHhyaQH?_2*=ZJrl|C1`nbl
zr#KmxGlYKviyGZpRF2%IuoSGK(1o6m(Ekb~q{BI5y)S<3#=gQg|4W0v()C*5ktDj{
zgqzJfI<0aEmfu3x@o&h}1Fa|K+jyO^Dbq7omXTHtX-5izz`!cZ@%8_3$)cI{gMu6-?TrO$?T2
zp#`E<;j%*POPOPw8fEfx2T%X>bCHCu14v4!XBJ2LqHc=P6x7o%CrB=h~?
z-=*ndFMeoc0E@kCSw!t4CQ$6TPIo?tjCU4Zr39Q4KN!jm?^h6;n8ltVmfMlThNz)y
z=+<`|w(LIlXGnY$CZ{b^ujgSk-k|Sm@g&Y{JmPa#
zunddA8&QJO4&g{eOPvHn$b-RLLKM*bC!bYmTP|ozc9?%(b?)JM=8F&Db29oO(RSyl
zV!eBW?`ew0boTMA=!>g>H$(T@@9`lv@}7uR&!&jDJ(ucF5AN0TPc-e(8kbnIlG-(T
zEgw@b_%DP!RDD&k&`&?*~pVWm-
z)j*@a9Nb-ny=L3K5o8wD`{&W^S&EJ6(81V8Lyp@5!@sTYjHl
z-;j8djHKmTAza|Y{JQB{Vy&>Gp|s&CtwPoF|4I2~`!szcIFoMR&ObJkrgtI{
ze;C|dXKJMUckanQv^@IGLeMu~+j20N88$B3io8*dtNwXB3CE-wux(wZndE
zGISj=K%!9kU3F|ZpCSSbc_YyCVLs@g`N|kpw*FxQ|<+0^(H$
z;B8<6fQQ(7a=kue3$oHwIldNuKvNW*J;~zD)T~kgDYS6CP1b1oXxMERp
zcJ`6}YwpX&aBr$lcjulrMIYQ>^`_JRZG{aZ5;tlB55V&2c&o2A@WJ&Nom)VSg0ESc
zXY`Dd^u?Tn-DU(U=HT96Db!OAaFZoGvs9zh()q!ZawIAwrv+b#ADIzErc)mD;1Pv0
zf_l3wbiQf3E|Y`lb#0V9wJ}Vs-|fsH-rMY=K6%#fLUEy+Q6Z62CHuR+79K8U4$}Bm
zdnp_JDU7T-sBhaozO*4pHoFZN2Dd0?R5`}ydj#r(8mb@8Oeu@Dd_JSn3Ws~Dy4Mkh
zT9c&-|7YqgquP4HzTMzZ+@ZKz@Z#=8io3fN*Ptm5#WlEVkwPdCin|49krqfO5Ud3n
ztQ07;|HJdV=e+BD+28hBGqd+vvwm~mb6w5T?mqr&4$RzYXO_yb_x3I*S(0iX`y)U%BSqSy01$<%KFuK^x=zJ7d2_JDWA9C^8uLxVI(Ho4D>QNmdcg
z|Dg@d253`hrR-0DOzu#9O0U}~E_5tuTi6o=q+-k%w2~d@;If+r6AL
zCT!TOJECv!%ok`sw_wX|5+7OLFR78ZwUO8UcU-(>#Adc9@}Gf;niEC%&S#>Uhf
z(L)F)I_Sx`weM*2?~CK%eNqwBfB7FE)pC?^F;L9Cik%+%EX>h<154yx1E29AAx6YL;fdW->mlydT}IAJ@|SDl%I)*;kd^0YACCdkQ%(NrG!E`e
zl!RT>0f~YsoYmejME-lE)RP3^N)xLT6%i!6nnl=tR&jSk>ac^nb9PZZFPqdQ)Xpk7
z#;3jgwPqTKCK&bNx;3ujrK8@VyOCLv1Hol#qe~{UWGwqkX5mDO@v5|o#atNz!_tX_
zkkg{Xl(2E{1apSSY28*ufPeTt%O4dIahoFc2Qm>f89>ka`5m8|Z?mmI9KF38cdgTa
z3iq;cUp93E!B;7_hM}1j2;*qWRVqihspTFY372Nl1o%fhCCr`p>rt%Be*krmdZI&o
zj3Rkb&=tyauB02MIE@hdM66#20w^U3Z;8kh>2WFElszPXGDSBMt%`}?#m^+Z-EU5p
ztwiPYtr%^yi&@2Q&e$JH^EsucnK3_6xBEWRlucGMseYQ<2upzzv;>&*it=3==T*8`
zWK-S|gMF98?td%)h>$K-y+a5(g#IoWex}>-iB7Vb7g6X`crHFfv9y~7$SYA7w$7Z
z<TQ(&@*oCJVh63jj_hLM3w+yCDZ0nN@Ek|e*;OC
z5DPk)_gU|=H`VKm=Y6)#c1=HEBA)WXpp*Daoj}EZrN|oPvL8_
zZL|+Em@t6tkr3z?Ov13ho5jq<*Q82s0F|S9JmRzN+lfo}+N3|bAa$4Yea2d0OE3ww
zo5Ju98r1fD279I2U=X8<6XdQlZ*5=WFTBPaOlrI6Eu(c=(O=m;sWhbv883r?$(d(`
z%Ybr)IL)6Yjb~FMwFo?UsgIzKqPPH7cXRm`W$X)QQ274(R^TK~IJtUr)9wx&i{{9w
zy!vob45P=lg#1CQX8U0HNaB-Q0*XAon5dh7{n~`*FQ10kg$V~~*3Hp3Lt-s&?)+>1
zD1tuCF21-ZZ+t4$QR9)$(V+}@zNVbLFg2m~G8VY>&e3`Zn_Ag^&gD6h{C4z~F*Q7}
zxVN?3K@^rNZ`u-tf0-?Tt5u{`Pjf&1hpG*}?x$qCm&e{s^Zjaw$_f
z-8!p$V#Nbx83id<*`3H-tRp+)X-E{AA9k~wSWPvos2^S!6CYIhlQ0P$kLFrr+27`M{2dPUAR_GPnFV@K!Zta>RQv=|4>Fs2m$*+
z>BZGs&spzD8Wtv5aC1J`s^1Yq_Vb>PXFv}#OWz`3W1f91CXUEOdyDh6hIUsy82*u@
zL^EN_HkJBoMV9AoQ;_4_6S*AbC_RTEynl7ouAyjan#F~RnJq%(`10Gi6tTW2%Hqr8
zMaLLrBrd@s33zr=9)oc~u&D}dJai(~?&^KkWwl4W7S|?~9Lj4egE#n}Mqc#C(lY6efN7b99OBZSD+YT|k}}
zjJ@nN1qV#KRVQ%x!xBK8Bu4LQU9sz8ut)qUMwGdQvENJaE7cyN12o(M`6WzNXGsr?
zPW*|5v2Pa~?;SWTMAwKQFHsD_KZ=l%j?x5ZCMUbrJhoBNmf>%$#a{$cOPB69+^qpc+X?%k@>)7wWjJgePP^tjJz>g3A(&(&HB(!gJxcwKTamf*!TPw58+O
zdN_^f$**zW5^6!t_S+L%=}t%(AEy62%*N*N6;Hs4?As4D>Dg)Vy9{+u*{&NX27*;z@5p>O{_H)CUMwkwQG(bzDL!
zSP@gK0CgW{`|R_K*oJiw9tu*#phur%f-AKfy(W%Hf8jhjPOKG+AHf9*6`Z30QKW%N
z;zciw)_j7QlTxK;l_=`+FOsZv%`;9S-Nh`LH|^T7NIbrMaVwu!5wS1*SF_Qz{#Pr}
z{4J=&e$(H%_-A_gv@oR+f(2LGvS3jk%pQCS;P
zt{8LcBsxks^^hZU@wFZ;EI@ivtjk~xwBHseRZudO#NYbHCi|s`>!*LAsGlGf@fGNC
zFs%iY_#8ht?joL>NsiKj&Xe`#g{UTdRB-_7-N!hN=(J7=YGV`fHGYy?90mPS@UXK`
zKZDi#y`V=5-gr)JZNs+&_N2R_s}(9$0WZ9r#bs^TQ45#wop-oEI%@(msBsV7N&!Y>
znF5J=uyzc$>_=}^iPpqP3c#?y2}gXb1lG^V`4@`9QwtEGV&y-D(RC$mYJ5LY(FK`@
z^OG(_H0rj0F@qZ^yQqkU^0~&@!7qwa<{Ff_4%alwz)1^`y?=DAa|hpw>^WYKf`n|4
z7JcIl1DTVi`;!Z@>6U+c=#cmvt*(bi{8mLQ_*^H8b>T53B{T7lAh8MVLojxW@gP6L
zq<$aJQ>BnL+4y6B07S^PKj?7uL-)6mu%ZEP{8^E=OHN7&!4I#Cp{Rwcrz;v4vX=qr
zWa`084oQXlzS39n!gpkBxN;Z&0e-d6?Ceb<|53IsH3bUX73Z9ANWNd=okEY>W$FwT
z7Z?4u?FUaV3kkpBnIfkSyJv{DFmg22N!1xELD3l26I5#iCr-b*6DIsWgZK{s(73a8
zz`1xD&>aWv5jUA1Ie3cpX{q*NK?zg_yVxuI(S=F13vM$Dk3x2j)cB4Ix_r@25`1o@
z_Dl_QHyYRbftNv~V-$R*e&}mn=BKH-&Xg4jOsViQyFB|*@Q`Pf$yS7yQW#AYmy8JZ
zLwO0=p6+TqeTx5CgM|dck=!e~k-cD23<<_5dqSbZ1qc}R^q_KtV47+NY7m29ikM!z
z<0C}N|W~^6CQDo
zmF1dUVB5_UUPq`5dnsviAsr{+=^#QLKLP5L5pm6e_aA`tiJ?b?^#t1kJo$M6WKC%s
znnW&3r(^+(ze`M$_B2a3!q9kmp2z$e6&*E~K7oX;>ZwaIfuh@I#AOouuERofU)mj<
z-l)gXHzIvE(rDkycCxD;=X-FQPTxGF`X$tXtys>G{{d9P%@(E4Rte+EGKM=Ip}02B
zQkw`DsZJ_>w;lTr)W|mecrOt3{_{5NqzdmXj&1RxXrw}r0^EvfR}SJu1d+x_AC)#T
z{ye1chfdg-{$M-=&gd9kNj0WlNj5?1d(rA0`{gIG(BORv{SmV(xQN#V
zevruf&Auh5vya)8GidgFN(GJg9!%tRycVElOB_V!BT3jw!~V;$)An?OTjYIFhG_Ap
z?|Nb1hiA9&K{Q`6_0T7OqYdxh!z$^UwpzCDht|#!@ihPD^TDtBL@uucFZW(KjQtMi
zZ9NgD(O&)yYTDA(&>HI^836Ff6nPAO1(zCO+PhFQ3Z+UsAB@k4J07p@vWxFj_$I}4
zGLEAAX>Hc4Fo|RoH6Y+njOS5jNGIx%eP(A5ee6-B_Qt8xov5bQmZA?m0HvA?JTG)k
zQg)JM`)K~}B;yADjHRAo4GK10$a!9@+V{&S@-su#qwxFkn8*w3p#;C1S&p#TVz9tP
z!_36noRj(LU%Mc(49{Ls2dVMtk-D{J9Z
zcJ}gD=E=HED{*#3($BjMuL?3$wWkg{f^=^he9C48?mAlyD-9IeabJnH{no5t*m2i?
zCr87GKPHnTOJToo
zk^eWl{+RhFX13$JC)T3pOai=@Ma}hES%7`FmXFgR>Q{0F^1*}TqcEQ`F<8?^Nc$sM
zWSMEms1|o%_DFMa+&iCdET$4Y7k}F|TzD{3eB5nKmg&xh>~K@P<4r$_#2n=eXl%8f
zYF>Yt+#isLjW=4U4cHRMQWC$LnjPwo-Fy1*
zor#C(-H9sHMY!xdNgVq>z-Lmsc+4>l5<{C8pJZ_AhY{>8$CA33xaBpU(OLuCGPPi8
zJ2shLDMy}B7cPuZu$mU&^n}1iz015A3ksUu@u{wMGNI74LYBVvU}tQSta6fHslRS?c|gOJks`=4_t`@$^_i82QC-R4^y9)oB8Jz%Bu`zW&6U6(7U!||ma%b8*
zpu1^@w-C{OSr5*R`LIg)XNRdN=btM??aXFB9!HUwj;kqs6ONip7_qH8wBKCb04Y3GF=ni`BjXJg|F+9{Wi&UK`2~y_|mKe
z3v~rV`K74vCN~om{0G2ec#jVmfUkVu)x?*3ZP^sk`9xTh8Imkq_#)}Sc6N)UT)hFW6-WkRnB>R`y=N(d
ziqt=-487Y}mso3gogqT2cD=*;3lR4RsEyhk2LU($!oz_x!x5uu7(2?3JS=bMP)|la
z`7mCrx571{ckxfmSUz$LjQ^J!3&8!K-R#LXPtFDwC3|wsvj)c1fgiN2=^!4
z@@hD*-3N8=D!7_@(jAPB?z`)niIqU{?NXCG<`vUhwuA<2PD|}iO1hFohnbWva3WhB
z&&2EV8E?i-=KemshFiu>_ozaU)l-d>OH^^vq|xetE|1DLj=
z5x0R5N&9039oy%#zc$=~Gxj{I!9ljz!uy@a!0vOW&P~IIhyEF^2E->$JCKn4?duss
z%}V@Ll3kLGNcEv5S!Ya%UMWG@RM&Hlv}Xs9M>*fJ4*29o*xq+y1_q~@C#Q`46&aY%0Vb5)Df
zj^8xzTNcNE5eRTPHFA2?o|ecQ8OZkY%F$AzV~HYBC7DI;22wR*Gt=F!_xX7X%(S!7
z3%&i;GOYG~Zo|YObE}@P>z=xFaADesTtdWv^Lz3i`q328*mVQIwx)qR*oKbDXni;*{^sHE?2Q`)))ahdLvzj|0&A(~BjAy&Y>#>M12ERfxbrh+PGF43LWpPf
zVs7>#b4-Y5Kz7Fj_rW?U?!`-yiyz+9$80|#BJr|fh_%ynncz3p_PqnhE@7!yC<%KG-^YZApA)8+KPep08Oe(fgdQ~7LP{=B7e=Oa=
zFh-Y1y5gWGucM+WU8q(Iyq>3(Y>A7AbMi3uqmYqG`fUfd81=a!xfKdiabJ=~sNB0D
zxRDT-0Oyw*8+XreDK+gK|H4bHLR2`5wC3>H}!X9>rBw{
zBH_6Y1M^v;R;~KNNKyvGS2}EQ9VRB3USg-x#bWFg29ej#Cd?Y6dztNxz-`9=!-={uV9HE4h0uTsIBn3Y=;@c5uM^JNoSP3@x8m#?^|0NkwEQ9a-M}
z2ZQwi|52zN+O@2wd;NFsk|asdqiU1r1mzq~I8T02&DRo{#M^y0MYhNk4A1iAETDaL
zt+^aL@|b?6i@rKP
z0}w`NGKKhG{Flt{%^-`*=);b(0&}cv%A{?XtUIuq!fj9qoW$y{Eu3-yts&*NDKFc}
zd;lyNqm%owD<+*_@1w;N&X60~C%ncG5U>mtT`OT6$NKu%6!uWE=Rz#>8_J~O){mwz
z6s*f!^&rb5IB@H8hlZpXyM99pBaM~z*@DF=NCDDESnGoaJaT&E3B6WRi5c9{ATm&r
z7kK;6z;$Baoac4Ss82$rGf3?rHznm({_5NH!pi8)_NfClbPsy!KUIe8*$=Mu%-LKy
zVB*^r&sf3Z-kUH@^`T8d`V|>}+CNI`ff+Ssp{KVahCj0ymm>SSyZvPpHvO~U`2zbM
zm4}(Ws-S4@(rwBXfzBV1c?&1Y9G!o%30yO~=8|D2oO9!hzMqH7nff*R%m`0wJyVhoeUX(j_y#x<
z9^MiN%$IWAaVw=!j_PxDX^HjOAz%WzWL1Ad*OdgPnTK+OkKf>VOqykuV4X47Z`Z`iQbQ-A=2FwiWf4a)%9aIbHSkY#`~yIygQupxf~Tu?k65HJnX
zL%5bNTcC-m5?cv_<7ZEKQXybcn3B0-U3&Br@og~HgCWmuNzL=1r#B=Mq;S9haOz^(
z2qQtc-mFezMpUH=XOuCVPH6sI37fRL7E%0dITQXpUT5+qJPR?|Y11_mJQtj^YmZh7
z;Q9V(SpJFKk+HF_D1Nwa^lO7!+dNyJqiZ2&A)w!$X+Vo?N+
zReg5HKt3mFle-AP1}pd!etFq*fz<$M5-o>KG_fRaSzV}+X?>?*-kTuby|R3f1Xh|rjjXas3?u&u>}%*u?Z4JH-yQTk{h(s+5h=PL
zTsPxr`)AMA^-Eaypfz~Q_GH7Qu)K;!Na3abdRF7X6s{F-xG#%77(Df05~2C`<%N)@
z=?zBT{`3YMoVV*5Ok!;?F1F14exO&n-1!?9_g{mIa1Z(IkuaI_ndmXsPQ+&tql5Ay
zWYKgudvFe&t>jl21uYPPbFJ7gqD?=wF{GQY{vSXfcdAH@BW?Y&%O6S+>AAbHsx0A5
z1@*=5Gqn>3W4WipA;ormuD!js=zR+Y*Gkf~yCENM@Zu;-IG#Q2M7<)G(mJ&d)TjL8fxWayfr)&o&LYT=W7|vCcx=
z(|2K=EI(II;y%o!f4?!nwakG_tN5qSVPXW!Cwggjf_{OuYaiH^R;)YNi$_s1!<8`E
ziU*IsWE(>(9(G2)DUB5hZ+_!Ik|ETV&65f4`CgHY2X(VHC(wOpjoyqA%Yladx@A=4#*=8iy>wX?HA+t9E4y*;tBu$z}(QgLo`m9
zJp;97Bs(@4@)xtGo1v(m!
z-K>m$nw#ukkzWy)Q1KEU5<>#1&~k^f7#IayneA<@V+1
z#571V+>20{BAHco))xg?zlEOrV##z$cTZdGI~9;;(^D$?Ko@`sK&ZPF7^ldJs}kes
zrM-Ti92$^Yk;=Zjt29)GBRce1DF1dpXnD5y;zmL1U);eX!NuJTOWeWvOiPss;^68I
zEc`z%jQ^W!FmV1S;ryRm^Z&&fk=&ZDC+7M82{oAi6KZC?H#lV$4tIT&6_le~!31h+
z!p6Oi2)eR<`Ed5p&Gj^(qPX37Wa5p?4J9+z7^B?B+nC}PJv&G&h3SmGkYNO^ylte-
z^)0}e(fRa8g|wCO$=49hA_W!`qTE3=qPus@x_xAn$ZggNx*rmSLC
zjeKwPgepjO-2O}EU(Rp3;xNB|JjJ@n3lGx}TOp}peC&(B;7f0V6Uq}30sGl#5XjjB
z2E%!%koorlVifQnt!Q0c+xO2UNHCAde+iHtM89=q}&4Ug@to|D3M
z;gwUM3!|Ex>u;MF5R}uj_2$|yX@a@3*aiK}F
zEC2G17m4@SQDgz3-!&S{Fyl@M>4zs0_D*v$cNpL$9|XXY?N|@zO8TF?IwW=S1ajK<
zWPggJ1r3UrD=ynM={H?@O^S97L+x!vwm|SX3HpJ=@!r6KY)w0N7sLV
zg#=d+A;(g*)^-=%fNT2Y8(oOeDbq@L89iy-WLhm*8QA3OcStgmiVBJI23Ge3lL6P@
zuWP#9ep{;u`3iPNB+uBz&)(?D!*wX!fLj!#su%kA$Nc%2%@9+M)yjZp$>0p$B!xqc
zM?FUOz`I)HYl0V%SrHcr6Ki7=Uyb5f13ugYUvpMUN0n9iWbyO^v$2`-h(78mG9(SB
z7inP0OCCD+U&1RE`-hpfVOg*@@^MI6HRM4OT07*UHpu>;ze<2pKt?=Y3MEP-;Ua<(
z>5&(C#~>ocb7R`}oP&jmVnV%HKli6?_DOUdNEi)1PM$zL4OZv~qSd^KX|j`-QZ68xcvI6D=;MA5PO!VjRihdh|0fAO$~g
zT10g!*+m~42H|K+8*@A1w)qYw-Wie&xV7_4Sm1*n9
z;v)hqcfhV%m9~=hQM4B;X?nX%k#8rM0V7vr<8W$0)}fgh4Zq=N%T}F4QK2@JzoFYI
z^!tlB`cEg;4E0oL3`l(cpx}r7cD~(2efW5-XIbbtSv!^>sVsW(0jB`ytu$OywCwm$
z>44G#1Nu#fGpdsyu}?2$kKyb}27NpY+Ly?gXd?5F6uI_@=v!^ZxvZqK>jukYxnYWW
z;DUtIf`G^IN%F7D`$^waj+eKC%hHG56DgL69rg?#u9n&Q_Y}Z$0*_8}!TA-wHGQX_
zL%4BgthFUj3Y2TjdTgB|0N?&8NIb>1Bug1|JX*cG`LH}tK9D)&a@F;R-&SP4CF{lA
zpQ;gjCJdejt4rV)d8|f}TQGST1GaPMKZ`;2uevZO2x*>buildT)F#r||
zt`%kiC@12k#d%<7*~DyEtOZ|o@JMNfcP;T0u_7Qs_K&X_dqL`$C&b5>%r21P0;XX4
zWO)+VxSg=%zgm+;8l5B@uH1}Ah8)IjimK9H3vx%3!6eIAMYOjZVA~n}CQwv&E>J1p
zxGTdMb#Nzy()Ql*3XWd}hEiNRYj2NRXAUStFh3o1I3CH(+F?fNeC%MhZY`Xazvg1)Yb+qj-=Bgm}VI;cocdW5);aJEVVO0
za~6yDvbT@81A9vDJ{L5&Uy
z5FbX9R97W-9@()86OoxJ)-#Ha?{)RIR)IgC{3EJa2xdgd40QF_;br3s?z%dyrTBQy
zlI({iFC?w75F?z_Qon{(%2m<5k%&u}fj0@?X*u;;1){}UY@~IDbixW?%KCc^DPBi2
zBPL%Op;%y0@97ok1mZ$Gp=i8n^JH$>;O>LWa~FBsSsAVGz>rG671eXLX`bS{l#@@z
zV*SPCDklVNycZsCNJy2&?>A{cMCIC>A5UH&8FJWP2bP@*@T)O9?R=LE+Obf}7iPKq
z5AbvKpFV8=KR~!rFiyqXq#WY|)L&}sme%tuXovt@Sx0$p{@2!bvU{RP_#`OiZoM6z
zaDtWFTV+o}bU94wOpeszP3AHFo@<*fS}|(@W2jvo-~_s$;pMR6jGs?3-B_*++GG7a)B(hnnqAs=9!xgJ>9s7QLIjvbpAn_aQ}v!zZg
zKi>sn38L0EEz%s$%ZImjr|@rsXb_h~%e#&<+jT1|}QmkINVYX!p`s
zUZ5k6#m65vWqvTZW1e(py*%a6DC%V-OjubIU@1(%^$EsqQOk#
z(w)9V?KRp%${@zkhwf#+=-bdR*p8BU9rQc}U%#;vga|lm_a_eu=XuSSADDqLlM}60
zac^5b(Y}RMI8|;7-ilK5St11TC?FUuk+O(4?-_vT45b;?mJb&K7wr788b(iv5rUteJ^$x7QiH#Tv&{#3Z9+&58RC7Qa~Wf<{X>)6TH@S@T)tS3l>L;T=%r(t
z=($U5^z83!q95xcN~wK+v@Y}+{vHl`3E?uL|JdM;RffH);{B5{u1hHv+Anl=S*dxo
zsVKf=b;y&jWRP(Yp@dfk^tmpwI#8k=wQ{Qr+Q9{0%mJadIbf%v1(JV4^1ev@N4Khk
zznJn$pTL;@;OzILyEliAIHwcaC$bmz9XknMcR^2SgZuw)-us^-%KuhTp3HmyE8cr5
zp^)ueT>fA29$C8+^$U-{q9~PAGfL7w1ydO+lfU9uMD57^LY#`87>%J|ssIagWTkV4
z`!m9iw6V1e?u6;80lUpUYtQiSN{(v%W!@~}A;~(&KBOH7ja)w|(B|#p(0qEjH^Pxf
zKli%!wV#fR^X9r~(s6Cd=j8p(t9r+*ca*pbJoI9HZcS*I!oPceKbl+5woERl2FvL6
zLVIwRdDD#tZfMBUkM12ggg{YyZR6GTxRufcfH!nDj6MN`EX5mOcL5dq2AmFu$>6m<
z*`zN4>WAP!
zE1uZsAUAzIx2%^BfH)z_(j8Sr&Y*c7nmUAW*L*jBi6{SUEyeKPTy)0$HNFZ2*uUa2
z+2TmwVbw9!1xXcv#5(0CymYnZ(}m|EpxJQO_LuVs)tYwkgC%8B-nTd^&yJ$=YQh$I
z@<@z%*1&7ls@Z{-G39y^ox-j)E4IL;`+^}ye(E!w+TC_%*x3k~>BJd=M%Uk{#wdn8
zd&`v4F{(exYoGQgOy7tuzWI_}hihme0_Vl##I&|)lJ)dg`ZEzqftGx>IAzyj5_S~F
z?3yXo>^#tD5!_*%*XGzZGAE8VLA%Q|zDEo}f$>9Qxe#e)lW^Q!LU+1xe%d1+qqDT#
zTP|I*c(Z$oN6h|Y=9h#7m*O}Ix(%?kK+I9*x)6J}x0;u)WRO}z=bABwo!QF`S81?bY*%NAqr5&xuEGy7L%^FBMDJ_soXzDeF5
zeqEKyji1cmV8PV+r2DdWEGVX!C?2i~_`ogXSN7w`^ZR|e<@1a2&zcg{_LfOUsE*Xv
zG>p4!@U44i*-R|*2eUrwv4VA)2JunY~R|bU~#a{A@_67t?+nC1MW!q3xcvJt
z=EC2~^R~O}@c#Ht*f6o?J3KTi4fsXc8;5b;B~z#SgzsEQ^9h22)&%?1&X6%m#la2+
zFCRoSiF4+?zo$|c#*}7ZM{}TIq?MD#e*n0asW@v|6>Nm36h@NU?-ZNwR^)3y0jed+m-Ca?
z>_jHgys4nL+}P^4{{XvW*>nE^>ajWA1a)GdV|RNVCVd&M8u6DNahNOCDxGc~kh
z-3Np?cYgZYx^PPHNsxBB$t1TaX|wsLK~Sbyq_gVEfE3LNvnPzYfYk7ZZy#J)aid_+
z9>p9>SH8!4q9hZ4s()e?J!;${V5KmxOmg^>%}!4hmKH#>-dbzV-n_Dn64B^mie8Wu
z;I4ZAP)tXWr1Fn}9t^`s(@kcs7AZ+kCZ#BlN(%|KO5EcJpMY8R*e$%2Ol}#Z#}*
z<-Wkzby54w)hA;yz8P0HbXt9j7efn%BNm)2=e;IB>h~c{aQ@S8!2&|
z`<~B<^Q}Qa(3(i()0ppC2b*UgA%nSs2$50XN1K?9a9a)WPKS~Qury-6tm7$?Oo#Rz|rE;$6DvQ
zmA#}UYH|OMC3K$^_R7Qj3I&@@pxkFiGF)5lZUl!9!|JOEL2x>r(Q&>~A;TH+=4Var
z0EGTNf;LYkF3_EH@>L-zJa!@KEm1<+?vQ8`)(QK8{!j4To1gfgD{>W<933#`U7I7@?ZL$OZYD(F-)r(Mna&!wI3AGiG9U
zBR^llnL&xMO2&@^eU$D3QU}|GiuVqp`G%K`u%AX)wVNaU*%f8tE1fu66#2H2*s=9>
z>OrcnrW_c`mgT27yCfPk#d_z)z{)yi&ACh0TMmMFaKM+;a?>5ze-?5S(Zy)0BZ7WF
zhJyJ3hIJ;1>#yP{2VYf!(yK{%(knihf-xHE1lR1J%O{2=%cqQoebq+?BIlWex-I!v>qb0VY;a9ClcdDpq@1z
zYfJT{>v({K;h(`FW;V>%vpZ`dZzYU+7*s@*jAYMi1%lNHsJvhtoS^|VA)+V+ls)2x
z|AXagFrOdfPvT&!tX49H0@wbmQ)%*`2+CtoN5esIUWYTTo#mP8C*cT#LMqJz)_VX>
z2AGfEh5a!{u+B0aN_M8+CwRwJW}Q*%V4M^q^cr)WqLBN=ERt1dP(x(MU9iU?t+U~l
zY^yW)$<|8mB-uzQ%}9>!d_|*^e1hw?h{C3X9gY#(8*QVV6$o22443#$x*T>BM@u3L
z8{XP9AwNO&eOxu8Jws+&rI$0W$RGh_k4dxJa6E&G;}*Tspi2>r>V2(Sot`|Q
zZM`_B9aSU;LGO6AEAe}APEu!LZGv|xG~(K0M^;r%Nf=!-*v+DYN7ym-)mHlEU`}f+
zCtZB4%oBNcD@D0m;}sc2;F|6`L5(SC4i4%v`|Lb(4F(&oSB*|6GOJfiQ$DdFW&?j%
zOElU8RJ~MFO6E1?nuChZK4TZtu{Oiiu}htHu}~)Q%ak7S?9dmk$r;SNg?!q5fo)mC
zPVzj-^psm0v!W$O?Tv3wZ3hu~u8IGAOwQ?c)%BuU2UkBiCcRE}{|nrP+i
zKBS7aW;j-Si5l$-+hlPhGhFbnaC~#OGBqohT)jv0<AKi&
zhR9uTXA22#+OGL8i?X8N=C!C$FH$i(SUT)J#Z|$x&m(xls>dLj5@bFF`=x^WqhAN_
zjt5gI5Md-Vo~B$EtnZKqCd3#a8)l)U8c}xp&S$uS#}P0eoH3fI1>VEQmUC-RpqQ^Y
zF*W}crSR};U3#03{?kL}aI60_(6I7F!EHQl8X-saut)UsZ*4^J~wx$o&@&2IMZx!#;
zZczatd~k?CQ8^Rl}5WzZ{)rvG%$BB8M8lx)uy~UHLdL6@w%XY1Yt9|KhV&$=Kg<
zd%~`+sr2VNcT<9NDeB(_PpQLVj|h8i#I6;|p8p3RAJ8)duhMx;1jwYsW9fat4|@B}
zpWMfKNQH~}?hS8yI1PaIU0M&pB=NfrXVc9YECw@=?{RPivQDZ=qa!jLUE6&ALX-^F
zkh5ySXMDOpi85MiCX;jNnPqEO5vecyjkgjL%&}LI(s`90(6Z`hhtxuND{R;x$`Or#I05y1Gg#k)5eNgj^_t(RT
z!+u+P=Jq$3*AS^22hgGM&>cnUt(=Dt;MXO5Sg@4Omk|g;l+8qUN#o3>(rKRwHv|rBDl&h&E1;aj&BptCDKEZru6ji=+KrI1Da
zGLu%C+Eo!@XP4&CBCcioqAl6$A9!;PhQmAc1KELYQSTe+s}*h6=o1}`{v=qlZC<7i
zMZ&2#D@fthffyAL?m;8;X+FZKi!^^JCZDy&fw8+UF98;*{W>|JMnzEjI>t+?MUt|t
z5F2fxF6S%6hKd3!JEZ(_5MJx#S}KVM;pnde!2GaDgJkp$#XMb{yb*bG`R
zrDY&7_Z)XfKBsuI{`DI5hfO^e7SEzdMdxRtibn2cI6^|0z6_?wmo=Gcg)K&2C(_ki
zU>axv9=zMLqlh<OtsopM9(cGE0QC->$
z_?P%@!6UWkiXK7p?h|9Ds@GxZ%un(LQX@RxonpdqW_RZ>Bc!j4;N165QzHj~;W0uH5k%&B%VbPc-_y#rC?W73;VVkm_f2X&q-kP>E<+`>Gl#>~ftLw<^ps){zjW+Gxe7z;o
zb+CeKXf`2_f{DjX{MM&G-^C)+2WP+wFW&m+xw*H{@waf)bhX>T|5cezU)
z#q465fBhVt*UWr~oLep@3%V-6q%zBU$8lu&mT6Jfa0+vfOujFW!2z}|1;)@!r%I-i
zZjIpn;!mcJp)guNb|g#p2VzaTHRgY%qleYmua5a~f^TaYXwppa?jAxo`^&a@Nx{^u
zAx7PwdM$9?{e|I`BcX@}=B@~~4Y0czpID13_a)ZrYWdFZ0=+Ex#bohi?^^L+ZKSTW
zD(Ko5hlOL;AbBqXGW*TMVwjssuWMy|#}$D!V%8d{O;KgHe>GSZ
z1~|%YS9W5c6*ZP2)z|7XxasM5_}9d#XUy_J&kdzAWMumpse(3N-KyA}mgiZ1711(EF7cD_fFcE)(3|QF71imLp-8TlFZl^{xt`%vO|g>_E`s@q
zg@@dFS?FSUpuDk>HI2hS!GfdtOQolx2nZM4GB77|9sWaPV~Nm=h%=IlDKYdIK~&K|
zCSg!vXeBS>+ZRBE#Y{5onG||hsIYGhcN21ZitfqSgm*syEV9t?C@4l~e=(;J?>->a
za1#{xA^!l7f+~vfEJ`PXF%5&%)hER7V3C>3Gidi8vF4?XkTHZ|X2mdBj3LZWyhgt<
zN+uvROdtgvRI7FQAynX_?dcWu47pCAgphKmIy}xRj4$tR%uX*BP&yZ160FQab@Sq3
zDgesWazshYQj^2D^zxP4f9(B8l6=tsicqT`32AO@0NdD$f0d)4q@CX_j3x9*QQbUr=O@X($
zpLGof5C9gi)W{C)Telp)5RPoE1KpeXgc!iPh$>r%rsFGNyFN^~e|FcXNH5H|#Wj1I
z#ol4A-c76j0A;lIkpBRhhy=sTwYm~nu)C%ZxpOU!o|0VAe-h@ZmZJBF+^PqNos02M
z$*YutCocwBZK1O@{Y1i^X{mu*aAH0{I=#oP%hKu)062uWr>R86L8RDq0DvfIX?H0%
z2;2v`ieI=0xE?k|f7D9qG*yTu#xk{1Q=CD@!^8oKoqfOoRduLUqk7B{KzU=zR5fuQ
z1$BS{7d)mW)1i*3Ln`qOyg-Q6cARqZ0D}EBlqipflxca7LH;n$^D_c+&m0#Q|@WtyLF;do@F@*@g
zX&F@=zi!U=8opqz9rY@QZ*rh06kN7~jGe{Axo{`p^)Y2$g@AGzTerz&Rl!o!y-S8D
z+q!oFLpMS|@=r`gfB-K%hs-L5BlbOzhmUZ-7D3=(f2M;Ub1`CZa0JoDAqx4?04cUX
z9TpL{Hy?s6p`nEar8P5aR{gofHXo)1_7l`092?TnXGf%Xv*B)RYAQoj#Vrr!JNiKK;}3<396C8Ma~B@q#Sv<
zY8!cTf0(JOw{hl`;wWpo)Zn+ts7#`jM-f44d5j;XYh8R1G2gaW7Ed8E5eJ+=9hD5_
zV7C>##pry_IGko&k#b(37@M|5f&fwj1_rdLMPqXFHbvyH#i~X1Fb-LNxk8FPOhqr;
zb##=dn5{BRB6P>AgUD_Qv}>5H2gFAU9I(r?e+-8!`adi;fVYg+A#zgayL^9eDe%r@
zI6XkP6hLd#!VlCHTWv=;ssS#R0NuVI#H^Z-wI?$>pmO+lAQ%Nwu6-DsIznS4tymTS
zyBskq1iE2{ja(w7-3-mAymRWvlx*t~^>I$#b9U}uLvImSt&+V!P!fgwpE9sO0jACW
ze*hA=yO#A{A~Jl#!|%Azu<87=&8vNz{mq0Ka*$e0$5N@A8-f$a)1pOZcL*VdDC6892JC!7hehmjU;7wP0nPS&$|VU))5#TSYu&{eQ0nGb
zW4IPLgm8R7x}&-+XQbuNxbSN#+6R^Zf6+!UGzT%v^Hd!<4@4-Wg9?iPtR5kvVZ}zm
z2K$AQ1Z)#84&vv#(8}d9!)m*U=EJBNYW>6*pma+PLQq+#sX+UHyAf%TA;l(ZrOX<9
z=PEm><3YsMBd~!j@F1;u?iNUPx-dshyW%s^RTsp*xLn#sDM#GFg?-8kv-b)Le>LI-
zRDLDK!b%gCrOPGyl-f0(p~lGYu=g#>KQUn!=@irP0T*%et@Rbl@`-LlV&798i$xDL
z4p(kuwXx64TdW$i(x|TU4X8n|rQnA`F>=>gc$D0rSx%)*gpa2|2X*O-SU7{R#p#rk
z!5{H`z`g>*oc{pT6d4K=Rpp3ke^M2rJKQ3wi2JFe0Is0cS(VA-xHh0|X8{ay4_&(u
z@J$tK69}ec2Mi=t0>nCWZeq&L%{@fN=YiCsAfv=WGHmeTGp`5;#vdZ?Elr5>6Mo@r
z65mMNRdRoFp#)H{VG15c1wjA~6{Q+k;uyXly3v)tQy(vRm5eK+xnE_mf4}U?0QcZL
zlGzKjX^epr9l4rts-zOswTQHN7YUFzxDtRW?2ZudfFh;^<|pd_;Vg4^FN-`)8Usvj
zWNNU_cPK1XId7S1^H|4S{{Y@)3b^x3RkV@P{_)QHE~Z7U(3Xc~OvkI`{e}fcmn*hg
zAURZmCP9^kTAuBPm4{uM7avFLyDvqHIFFz)h*y9YUajf4;=IfM9^P)clY_
znNO+31PrCO2V`;PT6CDw&ovGVC$bEQVC{7prE8Td#{wKdi%Ws=8C>>~-bxi-BZb0R
zTYP*>4b3p7?u-|LBT_9wOo$aU+;JaK5JFYIv_P@}S#brt%3)sBdzfA~jGE>c`9CF`
zL#ttN>I<#Uj?oIte=1}wV|NbIHjpNdN7Ob^EoD|yEDbqf_fZE$w3Y8Lx0V7!)}E6X
zcZ@_8&6P1EOozA&lJHMZICtoXb(M&t)VI6D=O<3b%mLzAJP-GB)>5`)4S(L|2&jU9
z_k2dhftzYPbqOLGX_dO99Dyf4N7H7x=cTH
z0-GA_K(}w0{0@A;MvjIQw2rC!i59lE1|o|?y}uEgvukwoFSiX~7~-*Y5LFK{g?`Z0
z4^_u^P?F%Uf8sEUY0(7?%gS}?J}CT4K9zV*pyuGt2C7s+h>XJws1mY-ENZ119~U{O
z`GPrp$E7FC{^5v`V8x$CR0?0YWKh7-(6)4ryd&rtf(wMq$ixk+c@4T?UN0asTQ
z7V&GmOqKTHJ+J~^6=?2j)$s#pKN9UHs+ZjT#12aEe?x;`F{#t(i^3UEWV~Sz4Kp+L
z6QNI0jy{N@p=qo{jy)KhkTR>pS{)4&aTr7cIQ&7@m|0vx003?mx&a+70LtKjLEs{q
z4q^qVnQBN0S>ka+Dr-LA7O@4YmcT6>u06~Y^BGuv1*%o&_?VM@bjSFa^AzbZh83az
z0F6MXe=3f60-{rp
zJTks!wMWY+ATdpcF^rGQTetBmH+5S_f4OzDeUuQQu?6zILxU}e2+%n&HF9f7FA<$Dnce3c0lF9+OnWe>I*<7;cC@p7Mljd0E$
zK_IB65@4yX;wXU&$rVtpZ&w;sw%^n>)}qe2s7Pt&>C^(={{T?9q*6@(08mx{x`iHM
zrzc;HLbFGl36`?RxyL~f+DmF6fBS~YXlluE;2cI7F9r!13<(tqP(JEmAsOZza+NeS
zedbY!u-$AXwdON`w(AkbKio;mi7Pf96yE0_6n{ac*CeC?zkaGc7hgAhi|u2A4U|x7=vU7=@2=injy+s$*q-e;A{MOnjpx
zj?7CF2gImSayWsMD(bNl3{IEYCCd`lS)IUY<%9y^0|yYUjZAoak;@hMjhdPUE5uW1
zu+C?6uMi>1JxukVaCQ+~keMZ6)H=&3B`>H1)r(}Zf-pr)W2c#EbP-!IxxaB}nTlAI
zlf*WqX~vzD5HM^iJXfd~b!^AL6xx{CN8FKl~(cy47Ic@Z|&(fmuze|0I7G=Fm8mYW|;
zOmNsxQJa?`m*LYhbJ!xhwBiwFR#c+0k<1Nl&0SmsEWK1`LMPn1l9&Ft${`93F*0i8
zfD5#uN+oT}wRbWPI51RvS;d?{03MSY#H#TD12%Cum@N?n%(zfRPrK>
zKnO8d`u_k4>}ooI!5C1Z?qyX{%M}^LLt?;yE^vB_%$fN6e?p-zby?Iw{1CFhi3MDv
zi;9bXQO?nRW()`1Mb(CC@e*pEFif-JI_noQj>4yYJ=lGnEg8eBvP%Jcokd_NUlhtTtlR5Tx})2+UIjc4MBXgP1sfPD<^7-=^#X}w
zpP6Au)^i6_-7W3L{6p$)NV**g_XQ)_AOwMy9CJv-SyuH{vOU^$$_>M%9M5
zfA25_%f1WMyhX*A^Di_l;Y>gT2FBa|TlV|u|QoAl<`{fGQ$gMw)tQf0=wcRQbMZ&
zGJ%=ZaRIA${E#?*pvU@!ODSI!EBZ?Je~7nL)a+H5h^vWai{NEJhTLK#4T-q
zu}u1fc53z_RhAXU)E3S-mXx}IXAQ-Vo493$yiBaZuxaicsH2u;plCb|?l48jGtQ+c
zd8{(9Ev64Ln1K3<&0;d*gCGP|{dPrrqiZ>F7ruxjQQh*+Le_)O7
z19yKDowEia6h_Fi8F?esTbb(|q%u}hIgZ_z&Sj7dm7-qepHV9ZzjXs;85E6yc1Tf_?7d4Q_H
z6@W!0#uDT!N5%e9{Dp;SPfccf&e+Je3MCmg?
zvkj+ce&*A|-fmn*X^bBc0I4mq^Ai+RZ@AR3kT`+`Ist|cnT6XMYS+AxxXeqziIQcP
zV(~U6#Teq@@NH{%#jL%=N(_%vKSf)ZudKlUDoob|cw^T9>&)j?0~{~8kQmL=#B!0t
z*Ze>x#2D4bL10c67O4ZveG!!OFBRV@_9F%QWbg>9K$Gr4pX3YPOmCDa^U*Tk{F
z-rk`B6@LpZHa=zXU6~BdT5N3~0(C4d=Fz9732A|uRQy7pM-q)nS5AwRN|YKga6`aR!5r%te?546j;buXlNepQ
ziAFxOqBlnZQAP*oZyg|`tKm|1M`8mkK)W8iJskBMBZ
zNVItpjDT>D)I}I?KX6+>Uoi3mTe)B)!@>d9)yr!U!s$k`-TtBqFon3pty0^r;7U{z
zU240QdomWkQ!=q@yf{A;2MQBrP^+#;RvoJI^D7FQe>X2r5v8?lweET*0W)uKPFR@m
z$CV5r8zrlrdb(R%#J-RWO5}P&y@n6fv4nCc$4bEP?}dVCr++
zaJB9m`F6{2Z}BVT^#dq}iDJ=p#N7*y#08@S(H!_?(Yx~%75j+j{mihYeOxl99->uX
zjIzucf3kC!Q2{JAZYnu!mc3kGQ2`SavXro%rAtIm$P(vKv<8%*UAv8FNvPH3bV{SO
zgIn#Ngjx##Vt96jG#JvizGacNqKC9T=1hr+!Qhxw*ffkzjLr{lQ7IRoy?KIaFdREr
zat)aWJlwMyXxHK|4}+DyI!+?(c9`2r>K0CHe@6ar%UMd#;pH$AZO62L1yJjV2iuLN
zUSpC@=y>%HtO&Q5F2%a!xpvr9*x&URvyzi17=EFuT8ii9RcT&RpPpf)*fMta0C+hU
zwTgZ13E+tuY7%Xjuaj|mMS1=8aOukkt)o478`(^vWwm7J^a_;5s?ySsU-p9qN)6Bk
zDlCbKkpBR;7M42YURV9Zu`)9d;oUDXn=VxO&Zg0Pz-l75nvTTLao!Lsu!%Ime>M&-
z21QM*Dpn2hUZWDoO-+~GPej-i2{Pg{nWYXQOtlcj-`vD~rft^c!k_9{+`FzND)aQj
zI2gA;dxil>rVVCUjo}%GlMG*o$H=B6tX8q3+A^li6s%?c03Z1a62uq?z!>a1UIGJF
z+f%j`IxztF-X`>ph6w_LMxIAZe;Ck62g#@n@tf)_HC%B$Kinu+l?Xb$vmM5F{^!QC0xE
zabNaoIZtF=KOeh`pbOATOks?*@hqrO^k2!iS2}6)GcXf6HHS9~A?u;{>!l!@77mxlKEltoK@#!&Fs|_Z+cV!-0UH
z$+i9^m^D>m-!hG2EyLVr#BFlJBB-MHjVUTsXUB6ALrHM%(bTHR;Rn7{eENW*)n>=J
zV6x=%0g}EUfk*H=gm+gc$a4*V?QY3TwQ`*&xU&M-EqEZf2Mln?fBZqCg6h1oj@UhQ
z1^glxmeF7!)s1ltSR^Y*nOCUeL
ziG@a@hH3K~@S{*1gnPsp2Y_W*!tXKS-q`6Zw~BWw(;Icq@h2~(m3*VZC4905ee>cb!%^b4NVdX-^BZJmR
zbg0i7@iG-9Vpd%r`r#dhKbRa!`ofmoUMzw+;N=Dq0dTBRt~K@dC;$Wr
z01~|-45yy|0Ew3KMgbp!G(gq(B>b
zOPEnk3_&dYH(m)&E(>sbU&JL(D=fae9-`5n0AM`EgDyKisQG*)sQb7U>SZk^K^Vj~
zfc(cl2qt%S!pd)va4=eEzGdA&&X4AJH;~n9h~QfkG>vg9X+EmoBo3;Tj!Nf;p8mEqu(-zevLi;b98v
z)Na@v)z6_2sf`G-*=32o;{;Ww_bM~3QxL-k19H|>S1m;b1+}_xLqVJ^gY~&tR9(~X
z?ohK|>`W3<$&R{?z*mC(!3HyTn1?uGfmC{kH<4(_5NuROCct@1>G3Hz1sAefo4go5wdI2vXL7*wB^AA3FoQ)OjC7V@I?rC+ilMQrT
ze-Y3JbT{k~I!!hP5ZQ5}(~B)u>-|h!N(0?u^|(;0k#Wqtkke1@Iu^i(@nponV>vF`
z_Xki~;lHUwFs9rx?Bu%cT+h@T1kp!5f4-v96<~RlTY+E9u&fW5uI)@0N|eK{W1mku
zLg;&O^9P*O0cqYciC!RAmW^Yoh|Vr+zazPPn2nN!X-q;i1DcL-AA=s`9t_{9M|gOM
zF|k~G^#EfmaIKir5h(gD;{XD$A}Wqhv~_O15LKypSh%z(y65ID#g7Ie`vI+Ue-j!3
z&A@fhyO+Y-ikk<5JQ$`fAa#;IZ}Ak0jB0qekWpfwm*gwd#8y4TFe+V&S&wwBed;*f
zlD%Kam0kenq9yyaXI7^hfl?R+`%LFxSNeZYE}CRLN)}FE;9?Pz2`Sg0^gQ|bmi??F
zKE9(XO6J*7Y9n{dphn#DDMwDbe}T7`gmWB^I7ZcK``>hCC*amNCNpp0vu}(Y2WG}6+DlRgUmv4rh5T13l=!#borM|C#-ETpQ{RqgfBfX
zMz1f91Vw2zoGYm9O`Su!wDLm8pwPIPIyMJ_7X@9!T+z`2ygp$RtBw*Lf1uEULblMi
zSB568*MzNJ^ntW<+bSu1t3MFG
zDgZV+fT*pv<&Of0BKHD1eVg`13dolOvdqe}th6jlWX-!Id-gLS}OoZT3S!`=g&w67H)je|&KRNkrS1mtP;4
zMYoNo+K{Y26ipKg$%%z27%FU
z1$QXTN=2>Y&LzOoM&WtrOU8C~_by_{fCk&lqy`yoZiv=a
zk%KS-)(S?#_TS4YvT6$m=VN}QUhtvuaTnFw*_YBnV&(umXMW)#9GVg7JAmk|TBFF2
z_ghJ;@hUt5GPK);Ya7&MgtX_sNE^gP<_>Rpy+Cqh)+wq2s(m#fUBpj;Zc9TyJ>3Lpim
zF%Gau17}DTe}bh3n~xu{6(a26s4?w|YQQs=X5w&BkLHy+IO1kOxU+x(?Nbi4-e!_J
ze^GN<)eTYnLjWttz$4ok_m~~t8a`GCve|bZx#ZPs`~^=Ksu_6&pLGB_*d$>iiyk9@
z)2_ZpGPO%bE01t{xGPZdHJL%Vf8em>`GFnPCH7Ine|<;i39^Pk;H;AL{{ZZ@Y3es4
zD8bM|>LNMS;kRdU>B}c)P(w_uX*(r2QOlaFdDrn45rW(vSKQSTT>L>I$0Fr0FUr9U&tV+uZbiGO(@5u^SD_Lq9P!rM^f0Px?#R%0dtMe3c6R$BqHO$1WIH_qW
z6==51F!;Zeys>BtyQDXQy+RKTdn#O<44HPj(B
zwQ)t=^DT*AxG?~trg}+Dha|YnlC!cje=MI62W8@+Mg?Yh%r`ZcbADAVRZ12UiINuC
z13@%)s_~eH!lwnDvpo$;hgW>F8CSmEa`STpU;wrKz$?swi_YN{b+ZWZjlhSTBwsoH
z;k|m*{N`8|7s^5vhS7e{0FmXgI<-K|Rz%lqoPft+>e+uUJ
z>Ka(4Yp2|73DE_uig3;$}(&p9wq9+82vMyECElRBjCUT|A4Q}Y9b?Ksd6
zHTgjluR6W~Vw7hM7L@EQ$1>I;g;ieY;wcA2y5b2JkIGtUE8L|Wv{Bp)y`}?(6&#_c
zhAXcS3oc*OESGg`6^|b;F%mYLvZB@FmKYTIfRqWL%oSV9_Q8xtXUhtXf2CNbX2dy}
zEVlY(mmxz^pw&G~9mZ>k*lUbjRRwG>HNSfKfdFFzKqJUi!UogZ5<0Zny#!#2UR*X6
z&xkc^2EGYK(!{%jPAgL`nWeRPfhhBy>L37e
zv0e<;$kcCr%PV&G6}n$Bf7oY(#0CYlm1<4-ej#IETf47bVFm-P4Bx4I1-RgF2gH0P
z6`@X#Qnh%4w6KmDz+cQ&gjc{cmB{_Z&k!T6xO{n!X8>3SM-e$`O~J*n1@|fqaDb73
z#TuKCp^MB^%Lst65)!vC)hO;RL@`4(U9!RzahYo~xR7SXF6J6}e~sCJ(0s$kQwLwr
zOE?o4E<-}KR21%1B8I`C@_i9J48*bz);WR?E??VG0@AL4YK|w^Szo4lRs=*30d3LH@i0d6J6%EgB>u;lNkAKx{utejwofn`4}=og-5Q?3
zddwSX!rF2NRnGY?`6*D7^x^e|@DJq;fId68B`W18r~2
zrvWrvt4)Ky9wiJK<|2wL{Yw=HeaCdS1CjLr2$p&6{Y>sat-b&`mWoTIk6`46Kx!wM
zuCQuQs;?|9u?2%qm!c_?D1FQ@l^BE3&{hiOD7j>Va0M)Syo(w~#K;r%5aZ?jPIP>n
zr9kY-Ri`%>f98u)tua`Aug#RRZd+1_uqKyi@plCYc2xXCYhH3+1Tk{Uxx{1mVM}MRRRmuhe`)gem|pT(DyZFCD_T
z3I~gf02|Tc2M1a2mOEXp(}u4dL_x0Q|Gb^icnP}D(SE}O5j@iI$lzL4;K7@~gx
zj^-*>e+(Dyf{x(@fNVAy{ls1_o=59Xv3BWKFc}
zV6D>Zc09l#vfSS%reBvdb6h?k>7=o6`j-&BH@J=$?gPBN%PhRhHiu>AR*cY=0&(ZR
z!!X1y%YL(eGLqApfmz0wBB{tV@597)tU@4lf10njdBou4S3SZMmsJE#O54Rtw|1};
z`*~$rLB#`9KMX{z2D-MI{4+@cwL~449cxBc{kYIeX4_@gb7?~!5PZZjmmXsF!`aNl
z3gCrp!t-C4feo541H|s5;^but*20&VjIjzjn5k1h%oq@(BZvhoA3J^^ldV`?LJ(Zl
zf49`3LpQA+<`sIV1Z*t%fVBgqTTukbiMqyk
zjaDyQy~U>U$PX;Uh^QF-1iVVBiFQ5ze*kjG02&T6T)|=mWuuDP(MDBSt7{Gek2r!t
zZJW-aDzIO1a;;FXOz{S2^|fV_V*1WczN+`<$Qe7i}W2LjpEa{yo0L6_8D3Lc@&
z-BpGC)M#yaA>y0Fa||nt>xofAIzL;SP0M&aKy6WqAOf{4gzA+{sEyC^OoDSXe`xub
z@;pUZF0SH)??#WQZuD7yh(Jw^{Xnqe%%T3^+Ed&B6EQc@C5xemV?!(fUL%c#@f<<-
z3t%te6C3&(A)fx+u^U493ATMvxDm@9rxm;QzoYIIphg?VxmMC>Zk%V0`Hwn;=qj9c
z`SSz`W#(eBTC3IXQ9Gh43iFbZe+qniAD^kY6blNvEklR^M$Z8)pi&M3>)jb)NTs$6
z2?`Tts239gt!(2^2%?c4`RL|y90ra)?l2b}r_n3%q
z$NLoFZyBE8BAg_isD&s%Tc;PzmPQ7!2MXCx>Q~9z8mzto>RMb`zdD46Ld9*xrgJG5
zPnSJGyA5ulD$eNiBm73HQdzjMjF((O87xKAP_|I+HnmU)<#M-8WQmk$+bpmTK4vg8
zpC2CLqK6o_AN#~)yC_u3f58i{cFWMbp+X2~-9vC6a8{#jJR+C6%xTVU0!^1fSruz1
z=5@hF=?p#(K4XG7-UkE`r9e7&FDDUN)$JqUgO;WX`G{95f&;p%gL{Gz;fj&S%ip*W
zZ&x|W>K}c^#JaLFezxhwnuP~g1DU*(`ltnV3tE9ha_^`hWPbJSe}I;iWe9G*gW;Ba
zg+S??H*ddE`+}=>s|Yyg3svAFO?9~(Uyo2vB^3cGR7gY;EZJxqUI_70ywK=?5MN;}
zXy3T679155AytwIQ&Ycs5N!*+z^8-R=%QVC8_B9gi%dtO&UiJFp5{6e?+pbb#HtnJ9Tnyn58bG
zW-MO9Q|#j4M+b|8@LSKBW0nit1(dg<6QKA?@VgfiJhmlfSk=5Vu?n4AFo0IuTirl!
z6G#q8V~WG*gxVm5L#p!|Z&t5vrmdbjp@ztn5TH7JM%~p))}swgDv56SQ4N5%8(ar#
zC|EXN%(4kmf0NH&h#8H8jyy%7mADRI;be0ThgbYd>tVUXL{#nU@E)OQ)w0IcDY{VR
z6sufELQz#H-WjeB6AS>?xJrTTV3Osxul9l$Q)xjcg%_Xc5y1tABsDOR1BL#geNwKX
z1XTeI8VaUq>DvHs4M8o8-!mstZ0c8*{v#_jSRLH7f2&T^2T`KD0E`3(*Kg7miX5iG
zIyFM34s?jzB`@u~1oh#kun+ff%$$Qsd0Dt+6h5t4sy_!EMq#OCKE01KfHKIx{N9
zVUTwpe<62hqzk=?~i^Z)Bs|ixH|_C=59A%xmzkR`!wREV%Abp
zo!l#HqHI-qxFuK-K}T>QS?&u{HHqZ4XEE+ALi86Ia~(Ur5Bs>S1!Iio9DZ&pgf5j{^^U(X$HjY7T7ZhAe{EhmgF||4zkljF$!7Fk@fS@-Re<^2
z$pFK^JO2R5YrJb8!~z$xoBBGL1q5sV0AQm*tqjZL?VtA#3Dr8~j+J6vN~=jLsHips
zR5WQv%o>BD&->zA*efn$3%z6ne3b|l7sjQN+D7BwUQ~33N@#mB&XfjQqQ+H*#(|;#
ze*hCifN`9$@KkQjNX>}J?AA$;XtQFD|H&(P-7)n+S@9D
z_W4+}1qVIL-(PGMaE)*y@FTl$!q~Jif0h&j&GP}=*IO{K{qbI`%*l5wKJBgkYbIi=@xt9?T2qjzx+@mw@Sen&}m5|Yqk2El1aDdmEhSh<{
z!4EwEGFK0{@M?YC{$nG-wdVSjhKhmB)EFwvuR>9tVo}F}2@!ZqCx;NjGHrmYuc=0@
zr*f`R*kxID3>frP_<}h^LBnsxe~6ZX;O;mKp`xt+08vTI26c{Or~p$#z9N>xFAFp5
zrC42W_?LjOlk`3y9aMMf6IdIzFa1R>OEMtZ1~w8X=|M&W&<}861{-I|;fBnhtgF)j
z1!FZXoMBhYp%oUy;qC}K46Yt$6^nY}DsxwGMYo~mY~adr|(0vf=4a
zSZY@hIcmCh6#$O138BT%;sq>bzI3%TT(1noX%k@7TpWwIimP;t*d{7oveZTGZGIuA
zoXo^H$IK;wv6+F}u{Bmj@~)_e?-_Ysh?Fw&WO-nKG0M7PE@dlu3ZwH8RdAVPqNz&d
zgrTbNaqd*O_?3;V7^zs3e~%_UAw^)p5@_@Ynq+7+zqltV&Uk>8RnX>F1rOpI9W}j>
zMpib2C^+(Q6|_9zAMOq}LKr-BMQf7VFJ~+oqwWRjr<0guxDB-}vC9<(Skjy+vX#d&^H(Y?i8=Qqs=@YnR4RQbr(CIUvUO2F{a~w4>F0(^BiEj
z#)e4Z1K^?E^{H9!nUfY_Ifu*!2Zm%Rc*?+0utPBNuSaZgLzS7YFYrcHjRwrBN;!M^
zmnzyf6|f0wfIsRLe}hS8@Chs1BCs`#WZ`OM5)*c{X7e|pQ#P{z$9tiI|K3YrOc&;s&}`?y*QI|u!lg4Dno5Br#GTv%3&
zHxxJMT}(i5tN_dQ3YZ@vQj`u$518f03I;XK&8#*n+BEGU3k+>#g{uA`LAGDi#sG;K>O^D2<%*%nZ0KBsrd6lz|Z!l$*
zFFyH-nz=_tFKkq^dDE_A2)T7vOloG25vGjZ^BX$p;vrPqik@P>Vl`&#aiuem*L5AL
zB{qLhe}Smmh5Leqmnxzhpfp(0t@DJuMF?54qbLQgZ0ZMI&j4CjW4K{fbr*#r-$zkmL1uj)h@+PlmtwD2
zWkFE`;lxZR^#ClOIn6O(H_Es5Evm_q-4IGqe{8RDrKO{r)S!#ZUXEdbjsF0s!q_!z
zBCooxmobeDpd0YwGvY$MK;=CHwgwQjH~hs?v{xm@$X;{IxT^`6Y)rfXbD|xs4FK5y
zI@`lu5A`bx(APoo{{T?kTan-QD$&TKv9F!nRCc+&LxNFGZnZHfY-ZvNAa+CuY#t?K
zf3L+kAu@qR7iYVHu2R)Gh80$Cx|9wQz`vN}!3;OQ#1<&{ms%6S`C+3907J^nSus~n
zahBFhhum;j_WlltFYED`XAJ?19@+PFD@c`(nvaK}VY1Kr*U_<~M!wFDlEsIH#sNmu(j@wvj`XA{tyJ
z#8pi2wZl~jR%;QpLDi+($84-Bf#zODtAlZ_={zOwlQFTpS6s{it8ohVZ|WW^e@$)=
zu&_*W`BV?MwReh*D|IM6#x0Kwqfu%Bq5(!n#AL6OQGW=>UGpe8&I+_EVEjN^hXw<<
zDLAU!5UReO##KeZz)B)RgHXWT?93|VeV^)J0t~UAnUotcc=T}-6p6~|>NOb`JVd^o
z%8Xg|Fe5D?3Fl7qXOxKHo8m^`1Z+&yW9)}riJD($OoVw%nz5Y
zSQqeg2KJGdV*+?_1mGVK)R#+!BN$aNLN1Xr0)o?($8S>0W@L&UqQc;pf8Y)X8kMkP
z{{VTIKRKcJlp~kpH0AD8rsNLe00Ywj!YQ~aST)J@8d1Sx#k}_qrpi%EGDl^$Uw$RK
z<#`9EiAG_Uu^>>c%ukqg4?>GrRjje)b_0)&W~eX#;=bx$JK~QJ6>x@pKz}qHVd!wm
zP}-xSoKWfmHWZ@Uqr@B8e}^~UxqX`NdV#F@MT@Z6GFiHqGKSUKW-Li{2~;cX<(I}!
zsK#85bdTH+!);5ZEzV+-QoZ5S&Yww}GG;a|XAJi|v!Wg`^h?U@=KVUZZZpwR*jBuJ
zMY$0)6h*!&5He~kdozdx?uFJ-j>2D*wu@g;CD0sL+{LC3`!ftye@tE~4F(>M=3grF
zuA(&M*K);*4Qg08$(jIbnN5}-n61*;(-(?8W)U>zQ*=;3t+{D}G#Ak>R0O`o_%kVp
z=D#o8TaWu94wV*~{OE)^{^Rzei-N7Oc(zeYCb7h>*Dr}x&=0w#IL&4#qwy^kUTUxE
z8&Zo5jK=!7Q2ygPe=*2DBB~+Fv#WGKJihY(058m_&n9$_z+jJaA2wNmrz0{Y%{;&b
zM!m+oZK0t7dqJq2AhN~86BU1sSUEgQum{9^H*AUxyb=wv%oc1-iXBIM3gs#ZWHpYZ
zn%hIFl)RMgTa^a|ZImVhkl}{|EGHjK#F;eyK4VTI43SG%e=2s2Cy{V0@l&w%1jBl1
zcjhX%yqR*J0(I*bjujx$aIf@qp
zO7kn_zAb+VoaDV`(eCGFw}vaGWSRZcalfA27@;c}CTmgBfC
zRsjWje~TO0Ur`(5c+^D%>}9(SNfQ@ichsb9vxlj2G-#Rp)}l~UH3Jtg+D(^bcq5d4
zFB$BYz%({Uy^@sM^5u#iQ(bx{6lWDPl@>1J+gs;QMfYXubK-4TNV0wtlQ&V8R)XrL
z=`wz!5WH2Im{Ef+Vs?R1)JyINNHVcd`)r#4f1QNDRpX8ZX~FRf9F{H;fDBsHTA)&{
zwK0~CB~V+q17%-Q(B;y?5U0(=;amEM8iIqrOmr!9f0P{pY=ZFWV4|zb99%-s2Uh6v
z_?L{lVG~ZFdcVvJ}}UmhSZNS0
z4vQV|i0`1gI+ns3TqJInc)@_^@%|))%rIZcng0M&E#UP4X~d$n
znQQ|`xl8IC7`H4Xg~Us(e=>sTc|w*HVr5|g
zs$+SRi1jvWqqrZYnJPNSxFmUQcTEv8n)NMfSJMEzI59EDj8-KfBvAE;aOqW=`+`}^
zDZ85GY@M)mxR48QU%%pB*$vf5aE)?5h`ma3WVM(P~i0P`Wn*C7n+~;MNzR7)f@Ovz9NoX{Zp6>sCUvENyjo`S^_8)v@;m
z+nPHv@ESd^#-$%cwjc^Rw@?o5`9JC^c()O@Vxw>~VR(WN4GlmVE#>@w>~@O7T8R6a
z=smD-yzb9es1BCWTl7WdSsU=5%zta_8T*-FnvzL1yob!S1bQkZ72ah6oaMN&(v!I2
zZ)ekq>F^VKJ4#qBssM0fYwI4My3nZH8lTlEQT}X1kd=J9(G`--)Yx
zxFbcQ%eWv_Q2=Cusy-=%QJUjd=lX!5REp{X43^MEzdbHl
zGOsz5aJCFA0`p4_#@%g6t(iqj)Ix$OYxIGvF>P$_9MKivS5PMENFY7bpjansbAg3S0b37Yz}s3f^mzh}`Sc1C%xtc2F)C=Ki!vZwjqxeX4N~jUDXidyJ7u!&
z9yEk5jnh>*l^DgRAAc}BTCytLHg9p`P+2gKJ|P>4!7!tO_FSOt1yQItAZoHXG&o<_q7
z04=TVEs(1*SLBpE%;+mNFj6AmDwoBYwGN?*jTkei_#FYvc7F#~FLIj;URXA@L?c_e
z+yo4}Wu2_I!^9k2xA=yz;H|L2;`KMs`G;ctO2ugEHHbhpZk5!vin;=6YOv)VxQQlZ
zxf9FBr~?g43ys#OpnP1%)9|p{d^~)@GQxHRLi8>(R$LQSd*WGaag&xa
zRI{6&g0Qhzrt>b{)5N$cj~>z{09H8~&t1aC)KR7wU@
zC{c64Tz^?AHZhhi_vWDcHlmx=gQfta8Zy@_-F~H#`9=Y(9J4Q#ov*p3NG)Krd_frt
zko6r57AhT*05~GLD2VHXafZXpW3WpafDBOqY@s_B2*Jls3KqW=ErtzH1+RBi<}TwQ
zG~{9QT3?txSAHR@T6TmNHMd3R48rXUsLFaZIDY~;VG2#DS1#e;akT>F53@+oLhk}5
zU}|Yh0a4;I9w^MmGGz)6m?ln-<~WxXVUCm$Z!Trst2J;P{NAOhqh?x`bqAW3Rvd+M
zDbo)B0Mjcpri#=@%7P%uHVL1o!F8B^VvN>0BXu4Zwr@FkU;)AAUZ&Q3C7D25Q$sJ#
zPk)#SjIxq#mhFkSOX|_pED))Vpdob_?o_iDjSb4Pr2rGQH)(D6wv2dkRo7A_U
z?zQlD39`l!IZfiqvy95zw~;c&TyrQ)Gyed69(#^cu(s-)zPN!90I)z+H|Lp;#AqIv
zU#R+mkjws{w)xO@^$!U}hcCXOxw`RE=zrPrW??|fi<%c%<{;gx5}sldDkzs!L9`Y*
zaAI9JbK(x$=a{#4{Y9l^?o&+GU{&n4IF*V}q})IYh!`E%Lk91I%&Td=+{xBsf%7av
zpb?Op)LuxqOKM&M08_|(j?XYSdKc8Tg9NBWGw!3fvxUM=E>SX!xw6%gkxR3!27e5~
z%UQ4}{L1o#!872zLDd&1t0EOeYW`8-7uRt_Ev?TSEVqh?zy-W_GSt3X_>C~Y{M;@Y
zd)&%(!!bxxequVox|wj1^dHH9a51oUR79eUG}jM_U=O>6guK;51IcK5y@MI6`1Zr6
zznI236hICOy2buw(K7Eu+p^#`x_>}UY$ED4mQOp4eGk;gkt%Wl{@`)&@u-(+sfXJgajT4r3QB>}c5Fu3|QMvI~;|^AC35pmXN)5Sp+y
zOL*!&E>LUR7~8ABKlc!H=b@TQX0*GQV^PjlnONVbNXpsc*hixjKq^zzpbhX90<{JLz;VPRYyuF+5L_kz-!}-!d${x;%v$O)g$44(nA{`d
zxS0c>`*SPMFdVGc%-TNJh<`W(bm6FZhcwYBo5$N>8w@ntou5%&_u;
z-U}~R8oVB1PS>_#qpqWsqWoq!gXZGA#MVc|Z$A>#M*5WUC#hz#xPRBCr7kxR%G|a%
zy@42|m94=H?7YEXWQ2fA%Y)0*gVmzG89xYEL{vx2sQi*Z&>;)9-
z{;0z2$AiRcm>M-K6@M9lqqqyj>S6#dNAn5*8D8@cD=+0Ttly201*<(mO6u*dU>g>1
z4x#Emy{HG3;NE4C_QEXa$_va6l3?&UfQyNXnZNLH@f?BjOsY#+B
zN`orAmolI^mw)jpiosx!cp~^;XNcTZ8oaQswF4K_mkQ=7?gzx_QaWjx>zPJgguQr*
zm!|}52JTXL@h^6{oT>8v08+L?XYlj$2`J-;9-$DWiYD@s58>n7MAhYL;Gj=>VLE
zD|ZknNqC|T7Cn}A6zpck$H5;!bfN~z`P9o(>OExmAazU5B6I#HiEEsTs2~caE>d3L
z^)jLUFka9rs3Ma6My`zW4$7|!Q(#AEj47A-i;y-733O@0UZq%-)jxoRRt;!>xupt^
zPl-+ds(%I{u8o{Q2vc_XnTdO05iLDvz)eYXM|b>J?-pmngO
zqsK945OQu}vGiv!OK2xjq<@J)aN&0cCO3NaidzRr!}AWR52(YUJj|3pr7q=tub~54
zCyBiV8DF`;+oD>&#S4BW9(6w|s9S_Z?~+1OwTdM%T
zT|>4Vgnn0}`)!+WbH6qc|gW3)BL+rx8F;5na&a
z%(4q*lx8|fLRnPGwVdiD`Y^5Y3IeDDSKJQO>>g!7MBbn%d$W+)RWLwaI))zjRTM10
zI|Ixrir&ImR*8$eOo~OObj?E1xL9)mxI(EWF>DYSmitD(wA{g@2WSCWs8>sFrr{+)!(S!WP7uZzM)48>SyI@cB{0
zUBLeU!L^1ytTL$Y5H>kX$}gG6FQOy@t#08lG07i=sd%=QV8UY>o?uyPQsC|43YGVi
znBLyomSv8(l#LhimMvaaJdwoN6dFv-$&Y15q<{G4{fI_ZWEq4TFIF40{x1q#gmj??N=vmCg
zl_{;%FbV(&u`AyY(Je03;F(S!R{Ykbe%o+R`hn7=)D**9JNlPvp@26mCOaNsPHS`?
z$$KT`j7m0hDg$Nr7~w5N(|^>Zc_i|zAqXPX9Y^k+0hVWB*|XnLteV|^Y$*mt;J!kF
zvjrG-S$l(MWj2l_Z5wUndWWdT#RJcoQ`zPYs}HD^1BB3v*0WSmw#@z^&AQ#E;&mx
z?itm$Z-J5$&EmX;SL@=rvR*_fEaP>`vlpbG`K
zp>I;sQ<282IsvuJ^9CL&#|2GE*Qg_B&oZ{(1QW|=n9bhY>VGr7HOzqeg-RpTMQnK@
zYU&n2xKfl#n5zB48@{dr2Vl)h&Q>iESc!S$#wsn_*-eA$0!aYZV*mmogwFw5CBscJ{@unEHOix?eacu1n`?+K9YEOC
zhQ1(d@A1@k3f3nT#6W}5*%ofz>O2Qyw4!zj2rts&u~Q0-yb{KRM$$RquvmRV3Ibf8
z96*TJ+JBk)js|-!+McB!ZT9(?ShajgKngg8p(r^##qXF*76a94EUMOnsJ8T9mRT1E
zOcqqLf6P)GI;Sf&^Sey&pg4a=G$#rw8mCN
zwu?W+w)cb$4pH$)Z>^$<5=9Lu&VvO9A5
zj({dLW~wu
zuBGNN$ZYEMXHbikHbGasgM3T{VyInAV)}%(&P(P7lxVrB(3);^Jj{wZ<7)E2;73Bi
z9+^rASO7ujCr)8ziJvA68o5{)4l*0m4u9876gWeKaXHB*mT7yAC{c{)CqCQ
z-4JQL$KS~D76Vx&1}m;9SKwk3RYczewBd7Pza@15E;2SfDk_I8*cV^vF&IZyy6!027eT4ARChFu_>G8H5ckdMa#2QN3jo7#==hE;<9}#*
z59%~$xfI`Y!Bpfg=|?fRYQUxNJkOdu;#VskD^Y-C<^u}d&CHhyfMmWPGKSc#5b9UZ
zLt46nsKb__t*j=$;!;+}IF~D_9B(rQ)m;dRow+-iR$6f^_pQ{SRNf5tJFx75RQDR|?md9)sPGuWvXCb(0
z5}z}#5t|_AFixKtkM`FBDma@~&Za@T{7k}K*uw@=wH-hy3$W4
zzkJKyDW;6Joni&<5TC97;Wll7RO($`0P(~z^n$9$a3HAmmm89yH9k0$;2pFfRRj=b
zP({7K^g(HD#rFgejROqmBYzd9NC`5`L@hzks8@Zti&ofNTn1(iEslwI){K(G1)cW6
zoci+HPpCsc@1ffDE0mjg09;Tgf-^WBDzV)(LpOMPB~kN9*ryESWmkb1+iu|g+YihA
z!sC$+PKWm}H`8^@t_p!9Z~z0(m~aW2(|m3iWYrV|Emdfu`j0AwXn$!vd5S=#oUNNi
z5Me&xrrbBUhHL6OG(6zSdm1{QiD^D*eKvx&wsG7d?NV{Uej?@AU;_f)-aI^z!dUhr
z72_!75>}PIYfNo`g^Ig%`-|gk8);K@>g7>~S}~t6AQGE@AACZbriOqy=!y#!;Gb{t
z8wMS%7BmlW`-}q)$bWU_T{!Ub=J_E;P?v3TO5xZwy}_ihc$7TUM_3f%6$&c`*aCrm
zQkk^pQ+0f?TpHX8Sa6KUvBML4DGU!$clE%VkvPT}AxfCF7W$czgL!u~WSLqYF-k@#
zz>k*+ry9
z9@WY~uvE+S9kEk_QNQq%PScx~GblfVDr&lhpcF?{NVA41tJ!euJRIr@G+TSG>IP|w
zUE`7ts7kSb0Mv4Zt`TOPqA}$j+5Z5QFkisST{bbCL4S?|1C#xwx}qaAeM>9s3aVZ=
z&LMkanns0#6-&%I2gF06X5uNhixWDHK&h6DoxmZ`iH)WYlpgPFv(Z@8kk<=`Myym})Tm}NQAh;gi!jA$FN@paJ?I92-Pb$J%;Mqv
zz9Qa`GJm}f<%~kqQ=c8MrbOov`aU9oPX{o@FXaa#vR+%eN5PdY#B
zOK7N;D~oo^=Grx{%y*(zZstjJh7o;J)E{5O^SLP!|6rzs|V5>}^v_qxAA3-ci*CeoK$yOSJ+Y)Si
zKn&EeyO(g=)B=n*MA$*@EdlB|3XDQx*j?j|;(0QZ^EOSEcJUqnGbrT!Nf7Cf^*(D4>sLu(hZoGK&Hbj^8BK%s5=myl&36sl
zN-#@xh_n;{ZDOm813wa!fFc*##D6$Nfh*;#O~Ycw&M=!o%DcN^`GHYMRg_pHwDYV+
zL>#E(?a}uuSY4oj(i+EZ;(XYYt$7Sf9&&z1Z%_!|&@_Km_Z)n)N2sPFr0cb)LR76F
znzH%KDuG_EJ0@J~KA>8q87z<2RIsuK^KJ>Gokf>TbsrUd%LyX({RxLB#D5u3BRr^r
z@cEghQ4_K;6IToq4=c}I%I@H7FFZm{NWi-3aRQx^LJyR<`in~EI|JC6h>8}lO2a2K
zELWST%Lt-tZ!xQ1Eq?m1K|
zTsnn}aF2<*%nkYbl2s8!^O%KQOA`DL?;39GVEhEX_V;sKiOGqo9H1{t(033hAaxhZ
zQ*!}m0>XRh11#-H;WnocGZt}bURokOGVS2SXg3rk#}diLH44G3M1P70)U%aO5F$}>
zrZw(d$HFMI2KOPBz!1oslnZA2tx6KkFh_6`K=CjLqc;+pGTJhcD(N}sna@6<8J4AV
z;9hf3s4B@OO>SS5Ds#SBS%|S-8#%%wuTzlLmf|hi)b5yNE|j1YoKeIn56Oy;x4w-x
z0Zv5|Vd^cf8iU^E(0|B{hTw2KO42r$kfDIa)Ro%_AYzRrxo##)yBd&fjk7B{j1KXC
zP^6sPG|enQiujslxU_tCD4U@>bUXQnfr^CkF{_uBhHuJyhO3aOJ$U~BQkf*A{{U33
zfS^U{p;UT>4V8>c25tZumKp;MVTu*K-!kS)4&v4zO1J&)UVj1mV&I0%>R#Ovs~KpA
z2zZz=AdD_67#ss{H5QL3xPrz@ykfC~L`MM%Q%TA!2Wu;VtAI0Dpcv$1r-mLW_ZiM8
zyZM74vX-pl=2-*XWzF?5?WL+GBrRT80|cgI02Dhpmav)XE$G~E@cD~6j{c@g#aWlv
zr~z7f^#vs)27i2`RjX^8f#AR@_XP*-8WZX74v0kC1b(Ix#cc2oYY~Vm
z7M~HUm3hsa;+teb)hRdcjK{Uh)H@00sOzF9F%4EjKgl`cJqE$r!e82e>
z-4H~p%YVoEnea4y@}6$u3S^Mek@}VfOPPG)aLlR9wIvQ_!25*hGf>5VfUh#rfb^7+
zCjfVtN`;0|;TwZ6E?!F>DKibHg$odJ51-~VVC-IB9%ACooS$|70BU34TTYK2BO<79
zw&5DcGv?wH!qfp3SF>%(<3jVx+$1ey5fmnx?SD3!l$}FMl`%g4rLolDK=h4+NQ|f}
z$poO>F->-X);aHtc#Plv5pQcRGYA{zRt?%XV0i
zK%*;`QpN$K^ZS>zgJI{mC4ea|pEAYfN<2y%GZ-m;T+JYG&M=j>o(XMu)%8c1Na^6j
zF@I1V!q`oytL4;qEBvsvvEw~O>3o0^69I
z61Y6Es;CAwve)BLL2R>LO+co#891_Do@1(X=yezhq-z_NvSgY608@n?BUBgzRNNDx
z3pb6-1^miercjHO0+<=SBBjz<*evMDKm8`>
zQsK>#_hv}vk(kZgdr*urvc3&q2u>C7LN_nO6}V+W?UrCnOs%8J+`5fba6pU(&ropz
z8gnpiF&3p7mN2}r?Wnq>DJbd?!IU6qfYm^dRn=AeOAIqP#9gAc;Mg&e&tf;pDu0he
zS$NPrWcQE6Q39ORZo~ab32G}6kvAMjJ~Ao9v8wz)+{`S2nBy
zP1n(yXTnije;ivkE`lf(dpyG02OdytV43~cjjQW
z3;zJWY(%x5G{7Ta*GxG%3eKTb5`Xm9GV*$2C^>*^tYdKzeU>i6s*HygX@?o9QLD^5
zB4*>1pwH4;jWO^D^4zx0B7h7dn}%7*#7zbz4G1dh^u)^tmQa)qr3U%xQt@wcqmW3B
zF%q&2a)0ej(JLu%
z9*^*$OkfUWV`iHz-N1~$GL^KBzK}&KEG8j$SnfK@78?Ca8aazzT|lfL?;V~=-yaryNSzyY1LUEdlgz=U*>wijm8zgMqENjvTmyxnkCa
zWL1m1m_TQnn}Ky34mp)FlCh0pQ%{{WrlH$l8_q3BwW-}N8yiG&r=@&J|%kLOH*{3
zePR>&MXTfMC_GWAesU0Ul}AlH`-6P0v%Uk}=6~bi{nb_Bu44rW
zWdO5x;$`zcZ<$tdcf@MJJ!@KsyDsF;y=Zxc0?8PF=PsJ4flMr)V~AkWYOcpwTMLZl
z%Po}kE!SKHQB7fi$he!br^_TYmg-hwDka9)0u5AERJq#;EyFg(-ECY5wfg#wjmCxQ
zRC1%{ZcJwGxR(eZxqtO5yC)S1TAIO7t-(_7xXulS;%&MI9{&KS1XQZ6!kYOu_?rI!
zy+D`gB1!=2R)@r3tzeb9@fCt$pMB0i$5U=xMTM9(0{KO4rG*I|nC_$)#EPT=lpdp;
z+)W9_Wh}7ml28hO7W@%+ZsV|5?TOc*P9T+sCmDVwsRIdq#D7Lw`Xx(tW@kk6*SJ+s
zrRVt%nS1&jYiA<*6T*3gY8B_R$wBX-~=PkR5x$r;la6uU}
z@W4p?1KbXA`+p^xR_1|Vx(3;JMWhz!c$F4|4oqC#F^?l?OWii0xxboinUqz#h+E0z
zGx?TF`4MG-dVNe|L&R&yL`@Tce-SH};yD)nWrkb03p{mGBRNM?N?T=S+&~n>s?FD!
z2GOjdS70SLbH#HG34(GydWgjwJm(LZh)|_s-u|iwC4b_cLia54XyGfCqkHW@V#`B-
zq2w0GXUQ|5ZkPwI%qK1;KZ9*<-b$AH83h*W#93b7GKTtYDP0rDIQor~OIogp!i8Cf
z5HSG|I__E@0+?b-XB6fST~RQ+JDzY6nB=>FyJk?q9F;UKyX}{#bBmlM;$fJryi7p|
zYaGSzMt|#)3eu01pjM?XQ7ouDv6epFKz)9DfUIyPS$0c#9v(?#9!#A?aNHR})LNz`
zw0j7xW>#pjqNs&|%*z_TGW`B<&o}Lt@f8kBj>_RNU8NjgJDXfUF;}+|ke2r@bf3&-
zZ}f+ltk}!=I)%C2?Rk%dbTbK2p*r%Q#cV1Hxqp&*epm-%;DwqBxm48FC%A?S460scx7)pFC=gFMQ!<(D4}P4MAEsg
z*aU3^X&yBy3m2Jkg$t{B!~({ds6kY!y5ayhbC|&0mcj8krNu({B}@T`Xa|M@4M4~o
zoqxt~akk}%lt*o_QNog;Xl+|rbiFWkvOW5S2QjLnQ$4MEn$`pxL+*ZOQ8$!*4^bhe
z88e1asL_@VIha}eO#cAv)UC@p;{N3!Q5p{4+^>72*y7UwnaNaYywnLYE-IW3qs55Y
z`938nsN_(z)p0su$qp3V_=}2^rC4bpKz~UOo+;`x2dXO5Am$z4czAC+jAE#A8*v!k
z3>v)w6-5U0nZB`cg5Ym|h@iB=3w6R9v{xo9%uKpM)B`?0S?w
z?6uwX6w^Ias@ukvM-5krm>Ap^=}6!EmMyin;u-^05-%)Z^aNjUtBz$)mNf`$HGlj@
zX64_2->1ZNT`d>_*A)(E7(YT+?t$m1%pqQ41S`ZxSR&qH`9*1&K#3|-#8+~b$sDPW
z92kq$(H%TSD-FC%7qtnc{{Xv&)22)x+b_fvAxckp?C?v#JSU%s%?jGAuxe72zzH86OX-+~)IL|cbGVEJ3PlYdoZ#}z0v33?>2jGt(4m@65SmHgH6jslu9uhRPm49HaEap@&IO-krgz9Q3i>Q`Qscz_rcbD!u&*Cpg%NC-c
z_E=&XXQLSpGX=6@aZrc1rv-av46F>Ab*QMf%reT4*(?Ks`;Eeyc;*j77eu%X5eu%P
zM3KSFEUQf&u_n00_0Zcw`Xi1h6pJi>P&dVX5}Ti7QtZU&_2NI2-ha(N#WtZ>aK|f5
zWP#fl634+(s{|{Gg*?Q>r8=6{Ck_~uK{gz)L;MD%#~EtM`Al-@EIk~?2z;y`tv(4&
z>fJf`{!tU{wO$=Xr(;==TER#BL`S
zs+JY5nT@(|ZAnrGu}OLASTAHSpcf6_;x>TGlbraKx)sI5Ixz*5-9r>hOS`fP
zUZLF*<#~*QbWf6a5TPt?qO%DxZ%|Bf6`4tJ*~TEATRo4MVt+nXN&UkOJ-A2@iD&_e
zeBHny%cQn^%1TjH4&)9TJ@8gqJM#|NQmgj4+#L2I(B&hxB|Wia__&!A;8@1)DQcYo
z_?O=$-*afLE;_k-2=v^)JBs~918m~pr7t>)Ha(+OL6`wISynradBIAh&rvM2H)Nqf
zI}#xV^6?$4Tz}=e7q}T0IpR|63!)Ji#aB=O)?fy5O=wrK$oqlB2RS^#;6PHM1&Q)}
z&W0w1YTc7lRomt_22TuHUK?3*=EWswXcwBjN)~RT7XSwFOlScXpYE)AdU0zyATj!|t5MQhsD;bG~G5MB`c17hciABS{p;uv}y-nl|N>3LIM=CujUOSjShy+(-g-6y?OGj%9^06FQ1oRrd>$B`SCWY^epZgJzuKqr$eD
zc5GHZ+y!qpD#$2II8{8c<;#O0=wN9)pte{cMt>2Q@V^iT2!>orgf&w-mZ)M-ORUEn
zGJ_e7qVi5D2$im_L-f^CI<63wL0MJf!Z5Exh+#(HAN4_?HIJCW)7NoI+ZORG
zb&3Au=c#JAJCA&pL*i`E(82W`6PBeE^9U1AKH_ab?S+4$BW1V4%(OE2pwwN!3Ao?N
zI-L<1H8ZOb+YZ>NwUKn^xC>+6^SFza;eVF{358#so=9n}0Y>kcOr9K)od|ElLILUK
zhJyo#b9N48x;c+txQ*P_u4N0yEIXL|kc0k0o?@&v&Diibi@Iw8c=r*(U=I6ceLy39d7l`KWj;mW+
z_G`Io6$3I=%0s9X;yzCuLMb8s-#%BJI0ae0ZnVo|o`>X>6w4ae?94Mb5;Me#Cn)I#wr
z*@%3LN~luowL6N}QO2VsggW|MZ+;=`r7VTzPBCYOQ-UQ+U6geKf`4rtx_G%#V<0&t
z&#+e85e@KBR6NX*xeg(MZ)0crhz8V4@V=s2*z+7J>|{KcXsv)`Ae*8syg0<5BA()P
zi1@_fF5<*@@h(jt+`y}pn+Y2iw*=5=6(N9GW9
z<1t5@#7GUdc}}G=P=9?L+%S!Rei7`{$%FWZ!+MEd`%8B<%JRfgxS@i>aw6WE#8It1
zMjqc1u`SHX;a^&b6c|}e#L#Y8RyM^z$WcUHJ%%94U~7k*^IMc$Bf^HR31Rt%nMBV~
z9}
zjgb)IYK*~G1?
zzqnNEm{>$9i9(yG05uWlWwIbTF0(A4FG}FaW|YiPAzJ0}1Sl5EU}kUzi$5uT6uqCA
z!Xq$`TZoPF+#WcGz?oW&K#|w%!y@+77D3hrm=JLbG=D+s^o|HRqY+-NIk@H}W<6Yb
zF%QOKq`-}ho9RkT)>=IF@P~wi$wLBLzotl5Or$X*o
z66_cvv<|M~CK^q{-WhH_BE0t;d6-4q5uRY%zF}Tqk>QmY%sV?kAJn=6DY}hKp`goX
zKsJG1;28~dEUqF1Fb!f~#Hv|QZZj-|ekHD$ae5Mn&cJpAl%Aa`a(~gORdfFUh=SS9
zW`C7T~817?;kYSgbvCJnOP&&C)idxX1UVpM!b4EF}3bIdLY
z({F2*Hlab4jDW9;D@>hNB;;-^CqswaN`Eoz8h~#YnCy*K$?f@<-d_o4YQYdW)G@P&
z!F4Nc3t7xXUogGgt2@8&q0V9BD{-zTkjwKb1zj;gDXyw-Rqrzd+Aa>IJ8J}CrH!mT
zqA?m@sP@b|KTzo6Q85u$`GeTkrU`}f0>|dLR!B!`PiP_RLoTX3ko9TNhzBfPgnzm`
z$6Vv7ZD8JOl&_v6Ha9TE#rl`#GO0t%UFRgvm$)Ef5eL*)xDY{OP}#|dD656&W!qe!
zx`QfeDtSP-7tSGVBBKm#ceuygApA$#;xe%Y+{K5eI+QgU-QW1f5kZzYm7COfhW9VH
zvgn0to}x0jVAnG48z$T0W$v3VF@Ja?mei*)u~Ex9nC$q|1Y`9Z^)nJxrNK(8hk2yw
zBAHjZxt$HGp&xOko&@|W5VXNb&C=`HQA9|W+f68bCTn|!HFHdSM#s!kxIN3^EbdaO
zTlfg_pGY3V(KD$5R|>g>;s6-pWYjbU=CLcU8H{Gv#57*~OHg~Ojp_zo;(ygy>M-4)
z6^14ws5`Q}LSLDQr_4Awfma%OnfZp9%rk$*8KNhB%xl8ubYrSAbNW~ZVz
zKzvGWO73(MZ}kMmGbwqL;fKy$xt5}I%o}b}TJyPR3F>Pwu~#v%fPcKkIy0#Hb{<{M
zvDBIeJlqgG6ftN~)NG|z@0gV&46rkX6oGE$Jr0Bx7k|X9Ln);k$Ie;I5$NV?xW3_=
znbc!)G}b15=BUP3E1b>7mzh>{>N45fxOX$GLhDgl%(Nk960x#7aWaL8r5-X
z#*gx3<_*UwYObM`On)~rW+x6Jyxa!4gVn@ULa*@2yiAq2i$%^N@5E4x%Pw;k)!bP4
zi^R$u$Ed4Og9&Dft|g1N{3{-1H;J1fg{axRnHhYJQem*OV=y?4YcVSCc=W{~QeGXv
z1$_~AH3{maC67;!K64YIJqVr)Sq5}}MH
zh=m{HjuVvOIb+UAv+i(1al!sO!UEk#jS1+(J}LM9Uq-
zF)Dxm*^|+F6c;rhFd%PYY6?6&3NK7$ZfA68ATv2PFq4sUD1V(=S#RXV5q{raF?liv
zSkYv6^Rl08I1U`-Fl+&fz;*x*NoknnkOLko$-h2dbyf429gUp4NX@?M{;GO{-QzCU
z{pDRC|G&L|_vQCtvdjD|R`KrsVHfyW7GW&dg}czCeiEu(93{Te(eD1V`^CFIR(+K}
zj8(tC(>n5=_J1n%DsC5fOAX@pVqY19P3}vqF+4+us~L$OT1MFkzkddSy0VLHU2L-9cIxXuy9uX!
z?CWQ_6LUeMv)!x2d*QEcJTl+W6t?#HSafmVy~x6fytp;aaZk08Ks@Z=7Q(l4aqRyN8cPm4l+CAJL5r_N1_K5HDG5>y_#a^GEs#geg
zS5#T-hibe2FpYevCo*il!(!8^cg*SI)2in`(|<6o$jQ8+ZXX*N6vJMTvQZSrqW}sM
z3ad*{VA&meKac{b*c*;H?3~&OTR=({KqJ`G_Ck-iF%a2y=%LETX$j^Sa)jyvk#!A|2ih)_Y
z*`cc2(eZs4tFmst5_NK17V5{ktWQ-tU=z9EE&jY9tv3U(I9PI<9SpYOHe6EDENL~d_S}TG5d1Q=1E^;C+Kt}G=EtN`24hw
zpjds~O@c1}o);8+jx{E=f^z>><$cT71?J3~$F7IZI}PtI0kOf7s$8Fk=|xcuvN;Dn
z3t0U?t!U600s*29u%RX48-joZuMF9Rn@HRB+#-NQ{|TVbe@rBBVWHz#lTuSJJkD9>
z%RG#Po{XM|R5nqVc(gX(`-p+gL)bpPB1D%EYgednm
zXs9gnrZfMCCWpi~Y7gBVylPXGx3EyEdNlEJ-|+t|q2ZzHPcmteDc~^4Mbd}6C{W)0K2@Hq
zgq*lR0-{q!P9CmG91uFvmof&BvfuL-kb$itl7>^j3k8I?|CQ^2ubW7TM
zb_wu9-<`O%kSdvMQ!Q!lGOetKkFs~u@rqY8$7G=&t7LQWIB;7I
zG>Z(C#hEtSA#l5hlamGfr|3YN
z+qP}vjgyIO+qP|EV%xUPcX)Ny>R+(0_EX(e<+gMOQPzDT4Y41_bUJ(-xqw-m6Y^{V
zFq&R^J~A>VKM}uopCFL38=}2qW3uU)N6xo>fM|NZH{;#dVv(&ciE$@*9E;@1wcR^J
z9Ot{6yt*C!Divz2GDBE*FMm)ON{HoN?>+~+ljt;oPZ=aVTE8`LVHoX~fh4xM%v2e+
z%?eVCa?>mU#*@dR*l@NU_0FD*l(5|aL==eKP%jy?vcek$=6u%#6czPrNNRJmQH3cu
zh^BLfc3fOjD?0_9JMR_@Jp(
zJ|(F-aFpl56iYdrsa%m*?9}=IXe1$_8Ob;4Igx{K6&$ilP%+a+H%kzIVuoM=&3s+$
z+YP=EJDilUzB&3^O8;r`$l^W(_E}}>SnB#$cM?}BLw`p`Z80Jn!07jSoy&o3ny!SQ
zek9q(YvKw(7fvO1=dsJ&iN8lgbMOW=XNM4WD`6UPO>ldKl{jMEmqLk4G
z#UX2ulSo$Lv%A6lbtNxyCjZ5BoGli-!a<&?#$%-_ouyE_jfk{lf2}FS449L|@vbh0
zC16!7j&(qepLE{0q*pin@kZ?8w&Rc_%YF}r*b~uX#MYX`Rduv(Z}J`mKxIcyx16#;
zl8Ij~zXhVh|0Fq;=?Ci`i-y_{WMxm!&zp=T;Oa%2w~#5{)wS}Dvg**bwz16?s7!%G
zRpgU~@ytr#STe4BNG5sG*$zY4Y*bsBmzOiFr%(z=hUh87lOgDF6lQQ|Aemk{5n*(A
zZKpQn-|4le{r(Z7NJ9n(?0D#Xj!?2#EUbg%h>1Q|&~p*b!)}}gW-+kJqFH?yq6}2F
zCiUPysBACizboH_b<6FA+w>cm^_Zdl$C=4)fYDSls|&?v-9PT0Y1VI&ga)K-cw2|y
zzo(61#e?i!m&ZjujtUGfyC8i9V2?85?iF%x1f$BVKlKVRCK>$z21Nl}tZc{M`5aMh
z-`pt@%Y!wI>moU>@Mly)ROLr3h>84>KGH%QCPE=pCU2jqyr>TozMb{R0h$(=)0l+7
z(eY1?DH|oSMoe!~Co;sYJC^@U342fs-SHJ_;?yq7X_Mmyy4jMo?KF#&_e;zk*#D&E
zsDgQl>hLG1#%06-S68}Ecvvc*KWrLRDfeomsyb{z|Bk;Ls^h2DlIN@S$z_j*W2tBw
ziU19WunUrdBw^x-ULgcD6|(RaWA^>?MW|-mGLv>E92`xx&$=E}1rKCJd1ztJIJj=U
zz(Rh<$~4T_uZyciBkTF4oXT*A^vgfVp
zpOLxctSyQ&$jq8X3+ImD)g5||i|!OU|9#D3w%2ll9*jFxQ@w|V!5Odz?rXE^y{Kln
z*s9`#AGM
zaMAFn@qYC_UBo=6_%tPLrk>a#Rm*^`(0GI~UHuXQ?yPBw?CGq41PnfnE)PAZUChiW
zurme#w+%`OtQ|q@y83)+xC!J!hlQGeTbW@;6oZC7>WL~O5bWUE;{Aj8?7pVFRpP8V
zFxr4+^~X}b9L{TzV7`!P(``hhIQnLfKZO|(bg*UD@8dNORj@0W-N>o
z{jHG648sM%7PwtBewRMC`Ur{Nw;>8pgv34IWsmWXy}>GF=<;2V=-(JV60Oen?@KqM
zN&&M9t>DBZeW&wmz^|c|B9n}`7%_w^<9u;(uzpLOBF8UjP-XDKOF6&*X-f|%fUVEH
z=bkEA)pF*l(g}r7Bi|3kSt`a>r)%~b_XhpPM@(?}MT%}c!`C&OoTP~B-{gs(1n#T`tS@i%WXXp%^fqfTI&SrpuW3z)G>J2&3XSFFpalRcKn=
zoN5)qkZZ~mWvts9MeQax8}-psZNH5~4o-U6+_{XUDWnmv{W-k=c2qk*F
zv`SQe1nXlt36yIhx#y-zNGAd)m@J39}lJ;e%vyFr%Ruof&9|ZgbCcI
zl|b|>>rNycWP@D-nJHd+rW0_VH1e%_=L$Y#*0A*5^4>2G#rKZi+0Fq2FF{qQbwZF+
zyH@H`b2ea^3xzr=O$XhLbsE#WIa6u9^SKKpqQv7`k4u%(#w?LQ5m?+RfUYVfe
zQrjhU604_u>e>tq@{w~&BPu2#2zb-|t+<%5A!k$6(sgFYs`#6`O35D`!2cIO=tl(c
z_$c0{x0@{S>TG~uVz8S-8~y^9xTUIZCn%sYUkXqWyE>J*vIsh#7Cq|QU!>I$FF27N
zficW2R{wTxEFSn5jrUVJQKz=s!O>rD1Ee;pE!Wj0nl39e&KsoxRHMcUA!#mF>WpiG
zslXA5b=4Yf{{VhbvOv4aauIGFZFRdbX%CS70c8#u6u72dox7eH#v1o=oHOLd0AqlG
z5d$E>DKH?`qU~8Rk#ysRsRYNH&;eS7tP+^)%1IJS5>2+%$+VLeC-A>Pj3m11w8xTy
zxVCRi1zTU4MJh;b>KlIzC)OfiNwe)+l^cZryq;SDb30>RhW83z&wby_pc{3L7qXe7
zm{sp}+}@oJqON6@f4Lil1>GALg46wFmc5E(qURx6+H_$(|S_Bol+>tzD3{}2I-Mzi=cy9lO
zsBYCmQo!~!xVyT4lQ5ldPNZ-P{-;l>C(wz}Z{l;}p2J)ZaCKQ2-qj?1ujLMS3ZIy%
zMJ@w|g%$K>4VOrN(2Q%BP~Z>Pe*^%Ls|A^DnPd!^B(NTL{pH&yln0U|slC3Of3G5=
zC{j)Z?O9m<)mD6=vmjsHG0sahA@q+8L`BgaUFz*Ty9yBNpSSGC5nM?_4{Aq$iy8Ct
zwkm*yI6H+ozqG65t(_Igd35#~Rc7q8Y*t_qyfS6@GC816aU%T2B9o5$!wDd;&;2J-
z_ZNRgC2SI3KCrb^>|w~Qwp;m%urL=XVm5dBRjWNOGRTaoq6{u@0O@9`L3gzI(KC2r
z)|QtUg~xminCtfWF>X@IFsB(|?G>PE-U0`uD0<`~kkoVLOJbCW0A3Pu}a;)dt
zo1*hZG^T;!GBtTbYS~^ra|_UEw=iLlH|m37TYglqRbbRdgMeW~wJKX6@Fxu)*ARad
z*o9cHs3J(P@cxzmiz-f}4gN8G3*E>zWo|wi7s}N3HGUnqeYrW{|mbEBktNmH!o~*f+>=BmIKc-9<#(LLL+yKCJs3f{5MY3;e
zvi}0!(ca+ckTSpOFbCH~5!yJ(1(8m)lN&H)ZI1$ohWrP6iP2gu9t7oOpHOO4|b76oV;V-CPr4cOJhRqbM=hEKVV>Yon&Yyg*fL|&6SUkI0RdZ@2
z!%tS#+Nc#oThGt_Is|~EyeY9pJE?{D5&?Vf`?iKg7QyveDXvq6iJyd@p5BB%TS3^y
z#6qAuyK=v%1);Kn$C$TM5j?2alyz`;&mv$^qZtQya)_G4!&m^X^Hp&N*(e+=Vf9t|(c;|>Ga-Q$M2&5@NJbJR&Mn+i|
zG?lx3XEGV5h=rBtiz)RQLQNSjBm^&SBP_>ZD#D5wj+Zxxy@7I1)#gz(%iQ2mlo4
zRGk^Vz=Jgy_veJ7twQfih>#uG90?jOLNVMY>oq&c2U2+S9~|-M#9Zb0+)a6|P~MKa
z1lShse+dPQ3%cQKP`P>Wujn&oRUurP!N^o0+C3~Q0|J1qu^n>N`L61$K#;&ylAcCZ
z)L4i>$uoaC#(vz-%5!bgoqtPv!8y%;C~~L^4?E#tCPXWP1r;0_>?~6Z3l)M7|Dx=P
z97@7NWXhd0Hnp1ReuW{0Zwlc$^9EGJy$uMakg=(&iKm_AjnHp
zT4nU|;N2A6k!3xx%4DB)anoXLW}Zh=rJNcvHUoQG3n9FJcyG1Be
z+1A|34tFpRbzO}J@MFysO(}&^>D(S6^f`A;bVLppF=8;+@d#gvoc!ZWdTkM!X~B2I
z48u;Z{3qa(12MH1*t;r0h?g|0
zibA!M6lMKs=@R1#Mhc_C>NmKS#hPShE!umb<$Q|p*RIeEQp$PN)Z=6dYNT0U%q%mD
zyzygcJM|w0>!y=mgDf#Z6c&N{jK~})^
zQEZ&x7pnnREpREVbLwJXy$kN5=2##V79YetEw9O%%0AIdf;_oMp6Mgam5T)mJ&on7
zG50Dgvl7p$B}bEFue(tq8g!@{$IozlVp=Hae<~~%;NM>;+8NtKg`1!y6>YquXVbEu
zVLq*rp(UfuS3fxYSP0HIm>6K^;Q@db^2bq}GevA{jzN<^i);R8i^~@?=SldBpVg|9
z!#~bRRu)&^^_EopXvuNfvorfh)xmE5Xq~NuKpO>M)x#?@13U-fv8<7hb+l4TXjBIe
z?664hRR79e?P^0qJdVdxuXsFnM3lFyJz&%l*d|C1{)&GW&zAb9yymNLGJgR`q#nYu
zl&3lS*6DHb)GR6?N#v?^HZ)@e?$uh2!VhkZ!9eX7Hj^6G+>*sH8P0fuve6$~V~g8&5AOeKVbX+VtK0i@QH0Nle+f2p)xG(yDXB{6(avcI$X2K{Oo@pGRHR=3edJxXb2~<<_Z1A0-8W9Gd$8G3|+_hC7@6Q8ZS^fE0Ik#t{
zhgznGL?I^B+D3|kFO)MxPa?7SvM`c)bh=9kQyO1Ur$6{Y1I%4nJi~kf&0#~OC2G2^
z17E4PLMcDDxtFGg=)3zji8lAsRMvS~b48HSsuhd1)OJ6q`D>rjZu|Q4t;igafvCBs
z8qPqDXIVY#IVT!yGIC+5~Q)P))`TCpZgB#zvz(=YjztfJrR*vLBrl7i;MU0|i!3%WRp5I}+>
zIeAs!RwA
zFYd0;3Gs>Y**$+Ctkl_HY56r&+7X9KFhNl#yDYHWKq$XBLL3xh{wKgD>BPb%`9VQ~
zv9NJ5{jaU_f7lxLKe2V!V8P)Z&?c(_9#gboOtlLkMB)_?{GJ<7OmT*KwS5a&%h+M~
z*NbXHWT*JH4as;CWu!#IY+YnxS6R;1_kGOKsm7}V|F1S#m#k?+tjB8{4_R@Lf^(Vs
zWGS(dH%6DJ<)QoQ*tW+4$8!b$zrN+NvaTK`W^4dF8e3#5LHV<1S>aR#=EaLBo}0p4
zwuE{aPmZVvXZPNGSWjF^`{&OO0}oXr
zYrI3WkFU>r#4pQpDa%-mj{6Bn!}tv${3Cu9))?}mo~GU6ee#H$v}lq=^Uy4Qr%e_lcq~%
zyY#!d_NxxUD8gvE=!ZguVvp>u+e?nz$-OMaGDEBj1yU%|3dfgRrtFGNCXVT>YY}p0THoJaK(GU7Z
zl2N`$|M#J`Cws4@Sc94DKP4DvYCR}_i~u?JFdk%;yP0pgR-UydfPEvM0wC8l9Rg6B
z{*m}+W17^m@Ev%y6hK{sG$S%X)V
zZ4ELv2qhnQ(ghyfOXg;Y^toI(2qb`2J6JTiUV#Vqv@aM>JWb@GGDUW?Xj@vb{LmKZ
zo7|cA(q2jYoHX{>4bOAOz
zZKMAsruwz6aYz$Z&cVB462zb@PB5{?a(V1&>2nlGA8(f(v)|A8ESG21rVK#9UGr-E
z{LdB)FEOCB?Wt^hXNHF=eS5u66Cp`pk3-(5Ro
zqk{K2!*)m27@jX<$SLr0iVpz(U9iUF25)~#ZK^RN0?y@d>o`B8Jd8gT%Pv^rf%-D&
zCW!Jc5#c?o=mbCB2SM2#LXv=V4ZfvMZmfRP1{&3*g@csOf;v%uHiO)bR~+2Z)D-Ar
z&yk*wx4-&cZ}`VWfh_UCa#;Eh7$zH6PW@+2Blm-nNZ=mdlA-3dPdLDoWP?J0@4h)3
z-=x26A2V!%M4cBN3CJ@?CW60-Ngp?jWVm6lDE}lBgv$TXc=O>vo+rUF6&c@{#ky$(uxhUfSk24er0PO8A%Cl|m(iYhYljjBIFo&#PA!}^vG5qKM?={cnGfU@FB+_V%L-r{gQcxXb2xum&{oHv-S6P(aq}Pe=E%
z@=gN86)*tr@eJlWq|Wwft%U@4)GE#h+XUV+Z{y}Hkii!+nO%_IdB8Gofq$x07oMMh
z>IdxieIro%PyByzX6)>817Mad;a*u
zWYFM`2Q2@#df(xTxjcM#UOiRMqam{g4u`MEt3upMt0O8KcTMK3d3eQl-`qC$K
zu{l8flaB49BI_d!Rm5K+9%fpBymIYOSha<@_j~&~AS08Hsciv-KhO7}GB9gA(3NGc
zOgNjB^yjQ-+$>Xmb;%9pM;ElbM-saX7RZ
z(d2nrh1t#XFyzk8&fh@pfECTc!w+T1@)+eL_F{6OQWCs*rSIu?m^33_Z#I!W-!87a
z!D_7$>v06KHQ2RSb|*y>3M1QfAzudabiS$PQPiUAPqXVVm9}eZ$W8!F#?6anhNJHF
z8w6s}bRmL=Pn!5`+a#uZ`kHJBD8=oIQ)>nTPJl#{s}};{anv<8sHmHPg(Hp|#Oq(~
zA#Ngr>K@7Q_XB3Bil;`Ee8>0z`4RLMrHBAm1*)##d}`zQ3Y?Tb5=>_5;hN4Q)nKth
zhP|OY5Ji*bDP*j{n7Dw<`EYrq*E~No2H9t4xQA+*cS*$Gm+83
zuj~1nBlPb$QQCd6g=!fS0SLN^N6Qc_6W-OU8D10qLdj_C>(>B_;NJEe9B{=tGD+U8
zpRI6V=G8}x&@)$O(=(=;O#Zu-WybEzg9PGHq&rvueU&3lqI?Q^61~ij)$+2yG{5q)
ztY0WN%JMl3&sWVx^du=7`L_d2L9}k*qbcgM%t}iT^nU`X*|DI&$by(q>CmEDxqMdq
zJdW%b{zJH;<|UxA(2ski4@`n5n&Yy{z&HrOKePt%a@=ufXcgp8&&z<|lH3A0f{Y!t
zRi6fw^dv*te?%_+oxmaJ-@>t4hnOzS2i5q*@s;qIdSpt#fKaiP?76GKp4-K>GAO1?
zWOdG5wtT?qzL*cMK|oCW{qAHbP$E#6F8xy+T&b{dxEla^Ps2B-;t?5KvHZ!}h?Z}-
zfZ7DPatMSaaMej^$$lK_=NO@qj
z-la0U0-ocew;|r6G8g@LYZUI@RX$e{V_zNQbV#W^QbcrFNnX2^`Nb`$UxN~w2V5hu13-XIIIzvDXGGY`H`
zOtjT?euKCgQu(WS&NM}XY{l;d*91QdBcqegFP}m*bS@~X*YSfiA#SNt_81y<9H(}Z
zq9wL(5wLQ2gIpO0al%a7?}5^}*xH#YjpNXNSCgxM}UX4fW#SqdU
zuS0QlBTL0&WWw&7v7(@Hl+(#V&1e$7E~<_|~E$f5}X9FxFs
z`41wU9;(8c4gWwGf^<4kM+DG!v!Jw3ilu;soncxKvcg$F2A86kgUc0|1QL0rIf`N6
z&jwHRj7<#`$V`LlUYZm_iAZ@yAn6&70yR9K36aV33nUx(50;_q7k7FB>Q>2qXK@w;s6C@j=3fikr5FkDJ}XBCuNx3k
z8GaBJh+L2#P%9zW+au}cLp!bL3soOI`FfYL5=N9a8SKG)>j}e%{Cm?Qe%Kq5&F%1SbP&XZqeV^sNzDv$+FC6mpI4i1zUhJvomb
z8z9G2AZT2~0VDbWwfVlwf!CNz;ai@gsm}Z=u3PljqoQ^Lo6$Dey)7MqK
zVEz)fexUR6GLuyres$fPn-X=!`?-1=_jSv_;nwq|lUh^S||Ht?4@BVuH0EP)lG>GtR)TrCPDNc9$1|d|%1TUrPo|OH^@*fMW?t;7b%hy`Cq;L4!(xQ!%KWLXS8dm70ETv3iyf^FW9y@FA#QvqXam2n%d+wB46iPq0EFos0eHtj?
zINpAi+cwkzs2{IvH_w*77wYoILUB<`;b?(~OCk26B3gscG=rsvA2!HOdHGXS8HcI{
z1$gSEu09qHz>1ETY5D7zy^i1tpgOnu>gwKvk#7P-&<3s8IV&{DSp+`(!?ue3vIQ>6dT#5mM~-)vYO
zZ2mXuzMGL86o2XPnMNMy=^B!OJDNwZv08mGJldwl8Vb!~ro70YRHkfJO$bf}y-VW8
zTBeaM)ScfwQ&!(A>WN}WWjp1eo#Sjkpp-}P=*saQDkllL)*?rsM9;BH@D-N6j|DrJ*;bYNe|4{)^2`{W1w9#m&e`Jr%h8TL?k3=8nMa3!v}N)
z2@9}-?tE1aH}qN2cb8)dr&|6RX{&1fZ1}_=717-Qu-8IkA=Hvpr@d-^dvw|e4jo-I
zgoo1JoIIeCW7x|qD)CO_Q8FoiAiyNXT9X~&1>x#FnWJ6;2y*88f=+A9!O%ZCXDk@&
zNntgmAObz!nUgck=a6IRziQjh(RbXkR*z9
z7I38Hv%iJmS5tDp+g#8_K&Z8X#9K~6ybX%x`^fR@5Q5!i4XSK}NbmOQS^&4Ob3N3@
zCV|Nd*n6x4FQ-=8^)xOA+51Cn!R5WfzLt$u2os#RkjsB2GQ
zo`88{Sa)5sY3}p+LKo5wWIGWYha3rzB~WMH;!QJu;*{YhV_S4+dPin5_-6Pl&Eso?
zQ1g1HR@BTK`6ONx2X#7BC;)ri1OeTiK54i%eEqO-h$%CzaG`4qDwKF!A~DAlS8hAB
zB7NxyCI>Oj{Q!1jSHn5aXZ~sjD(iSmlyT9axM^j10$66iltdt|BMIBR`llziYJ^Y0
zZ-MCD_s}_!4$9hkU9XQl{J~Qo=Vta;40QATL@)QtriM+)i*;H90H7TuQQQ*T8VlZb
zV|K~Q9??qx0aRMkz4z(0A?R)WT#P+Ewls$5N(26e>Z_Vzo+DfHELpfV-Gu32?Uswh
zPFs#gy%d1u>$>Z~2Do1Td3WZv{hIdXwzNfBf42!+CA8!qboXq>0nHnL%AO9G61y1>nvY=c8zN;D`^Ctf2sC
zeYmI3a$WJw(VF&Ocj-G+7g#^sB7UI=?=I0~1dYjkl?FzMdOkS?}+fjJM0vOqyyJT)l8XT5>
zWmofrrn#jI02WTa76Upqf`gckY&o5bHx^kk6u{0oTO&4WYwAEJfhV?6zvqwo;sZS~
zxNHijQ2Y20wI6H!r7u52<=9Y5`EE@!9Z*Iq--C`EP>W-0ooLq=OWcq|IblnEz`dV`BN=|FZ4>
z?$EaWvyq+Ftx41u;?4PuA477Pm1Q^TN=@UMO;rTB3nAa*v%;(RE;uN>J^_K%ujxrA
z>s9bx1F-=Ia&yQ}!Q3KF61XkJ+h!^OEZ#m|(1+*kH5xo>u#n%1lL3{uy
znwqUuXJn*%AFmz~YBoR>N5lHKLz^0O9)FhHeDQ{c3;A#R4cg4FLp9Q3#!e`>t5P$q
zs#J67ur6)X{Jm7J>GN{zI9%65BW-}0YWawCBY)U1*hszNCe?xSltt92SOm~k+QH`{
zMaH(|?L5J~yJNQq_pto|b~?#bTE@&x?*A=xps~|rc!YId_oAk@?2#sO-uP{=cVk~T
z@+4k@>hh1FnjMxN(TWw#N}`}qR`nl@H$-KZzi2Qve7aiB{DO8;_HSzH+CU0klQunS
z)8Ha8my5}mt|l#H)L9s{4hF!XeUYb;Cgu1!dqoBY0@yeu*XeY&p)do81gDYNS+qSK
zTpyxp$8m@ujeWW^+js|#*S^B`fU6u!V-@P>682z&?W!Ul>z`+v=Esg&dB9C10jJto0nbOH?kFRsbN$ci1HI9q=i$
zT7$%5K}Vus+~7(JCg7ZpL^RQThgZPxmd$Snl~&LA#yJ1b?LX5Ik*T{8&6ycO3py7p
z28twmH5}CZ-YzVJ7KrUxj~2L|
zk6v{>5+`s!f=xa!oB;sQ+a4>~{13#S;f$Kzeh%uA`6bNDIPVdDilPadW4D$v=PVjWm8m1%h!l^~
ztd76!luDZ2091bHW87Uc_xr0qGeY)K6Y>_D#JkTK_ei6bApjZjfstna(o8|!LJpN6Xh#?#K^eSW)Np%xP)o9i&<5n;Yx#~2
z;7@#>_lTj~#DwWU9@ria)1zE3AU;I(;@7;LF^ND6DEd7_Tm;`o0~Xo=YR3B#RKohZ
zf`Wbp;XtNz3qZ~S0q{D-ej&`M)xs1O%VBW}JaT0{W2A_ojM-?_Hhy+K$QGjLiwulD
z(ClLJObP5CJJWcrq}>A*5gYE`$V2C*f`wK}Z0(%gqvKe8_r6c|1%tHb%9=~(zN)2Z
zwr}+jI76-M{8QgF2mFgh@@}yERe8l4q+F@m@cA|O9e}O31B8UXj=>;PRx8z+n3CFO
zq`DEJ|Mvw=v?jx%b!PeGhlO_oU1p{3!lp#`HBUF*F30D8xAMb0Aje|WF^h0v+>DBc
zSn#72NyvmJ?WiurtE6aog(&cQ=;zRH$+ZDA9G{(~X-^P?yLLPCTYS`XO1j5NN`FuO
z7K-yossZe$x6R4ZVeN}yc{zD@NaS@2nIXOmooAif>RN}%cB1s?*&<%E^g0c*Zn_~t
zU)))B9s&dyu@aq*9HdMRW%D7ugpJ5_FihW*rt|w*IlNQ`L8;YDI|K9l!=O
z*$%cbFRm~1BVqs37JajrE@5thm$$T{!e{LUZ>krq8UE|nv`JpJ*H;pA+*kiF%U}r8
zy{0rT)bg)|cwKyE&1bp!A~1*_HISPH4Z*jsq;EX+DScf=svp$5Grlhf6k+rZwr^u`
z9YAJWJu6(~jq4B1(hjugMkNef*0-E~%FJ~tb4+Y491P>U8lsQ-5g(^W0Ri8*uv@7R
zQu>q#^S{+y01plaZj=0P-Uu98tcZF*WyB{dX_{d!xE2Z-_&nNX+^YTC8SRe;gPu5_uo14
z*@pZT&m|}^PvsD<+6W|>0mhrb^pAvIxjCgjf<}#}+Zn<6(Sco6!NJ2=8A9jqw}59)
zgYBa(Heop{U~A(7(Dsx?cvyF@CH42P($8iQ7r!EtR{HutDj*0K44%n+XF?&G(JbOh
zkASSHVKa?*qTbpp=d3YlIC6-Uyi6cpaHsXXK*@wVzN->cspIqs?eScM&HL20ly*b)S@E;Vg$3HQ`xTW}&v4kXq;V&0&KL?S|j<##$S8JeDnx
zvWH$-RnZfo#<5zb=tA|H(8|Vp*~2obxcslzxg0HH%?&p&qcj^o-!ObYC?E(|mDaK*
zaI$=2su^3hNuKo1MF-av4hmDhtxGLw1Dua52igQvv8k5&)lR!1u&@h!O^l;I&f1&M
zh7l{2Al4QDET$^nHc+K*#&42T#bk^g5PBcc}3@853GK|hW>5DgyT(?-qhG#<#BU#bjZ
zjG*w*eQT{tLloP535k94^7eyIa7KM+ztS6Hh!%DZdURscwwu_u$Gl87+v-Lv$@Hhx
z{NI$)FJzv?cOIz2zG4Y!E>7e3xA_6ybLCfuZ7L4pTP#_^lo11nGa%529k{4!%CHHO
z1LPUYQ4#Ivn@HzW?g#nl611ON(X>mrll8UrN=FFzNKci+3`qxLz|271%xN2Gb8wK@
zu5cc)(N2F=Y+J-4;uLQ%5`37>mtS98wnswvoLbGgqX0N}kYyA!8OOoP4{Y|y
z2Z(x4BXptErF|k$Z5+yBH*khVFjmG$?`5b%1$2363oqLnR`j`r_;mUbs!eHQ?VfxK
zOy8hcZcvVB5|H*2n7}OTjR9InS|-0)*xG7I0actG{j(+Xq+z12pR;cO)8B7CC?IBF
z7yye*G*{swWE7+z`Q><{3wbVnc2CZ}dL_OOlC)Une}#5@6K_0;-nC4b6R)(0u>8|1
zbwTwS4hH#OQ=uf!=zSmhP*N|?c5v9Ai4gWfKBQb{Q$VA%?a4OKp+`fhlU}hpv%4kY
zBqw~rIlVYAP(xMsxjJ&`@KlP9Xo{g)xE7q*FdS&ga|Wifc8k_-+9m#AEHP}*Y;8jJ
zoju=ZAJuYk8q^$SghASyml^UP#HamK)@H9?3-u-bSRn!^cF?t4p&JLIwQaRwD4VRY
z4p68S4**O*aSc8v5zl{B3P{DXTtv*_D8+
zIxNFzmt55&V8;`mL7(HjRj{)-6FUEWb11u^QT&ArZlJ5Wa%<#D`SyeK=Op1I$jT-;
z*;t?Zh{Jo*;#&l?@QcTV1E8>*T|=@hAdJ`RA0NkrNUXjXgh%btnfY%b!FPH7WV3$-
ztLi8k>zl^)OsBb=uzhtF1fL98cGo#gVC3if>FIjse4E_dQfV?i^{g|fvnU6dVjiJv
znAnpRja&cT9{kl>N7SKy8yWB*hz!BMe!I9h=tBh#olV+LK~0*-giVs`g9Kyd;`raB
zv7)V&bSR4Ke``=6!G(_DMu5l*i^E6@CVI8_-+Llh>nzpuOT^JF!S?yHT*j-qcWm>q
zz%($ls!E%gnVb1}C0n}bELL(wrUs=A05N3rZmP%{}>L@*X+BKomsxtQ3fXKDjy0ITusQ-IrNW7?qr
zmxSTawYVRksVxsE{Hw~FZ3k-Aq?Nixb&w6z6R2DP7@IL+
zd9;Ow)*93u;FfP6T*kdB%Cp5UNtw6Ayo6&pRbSa{WudtwkhWGbMwPksZ;c@JE=(Z7
zMNJD~LMr!k>gkwY@fCKn8n`BlnQq@xw>eLLus(OWocWu0=nYWTbddh4ZZ<|ss#lEG
zn1n9e`}Kxn__hgVzU=IKq%J^pwJdiBjcqiAj(qAA5%P6k~e
zEmQVd?a9siudwsrnmo%vvNUyWZiRiiFWuPhLVVa0{G?seP#=}*g>90WgW*+Bj)rbZw6V=yu=j$>Fy!56OrCWd_}~ZQqtVT-ue2A0vD+&c7!xB5XY`
z*hXEO?|OU}z{L$MeVVV1t@c-k(xHJP0UZWa>Biqq`WVIQJ3^L^Jf^W<7wiU78Z9zY
z!Z&e4(@;4>2pFWRDN9%LdY}8ZT^lJgXUSN7g$=KNy0`75@##?l3iX~+j~t!ch?bu>
z-P-;HFPJH$ID!}&1@t?>SPnDCiy~6jBMl3a{$<^rIgwt@13|P*f=vL$4fxxDl3Hy@
z2dZQY^us^m;re;@XnfE|{Lww_r%!m$pGzyryBhRojOZf}Tr>UH1u^@dL2@(@@O7
z0G%!Q5M<&4Fc(#y5e7XZuI35ZzoQjRSeM!27R2)Dw2UczG30U}!Y+cSe;BL2Q>Kywpvu2`GrWJR;p_+KO&BK58BQM=
zjtphvam_2aCXB`|#EV$*VZ~o>?+>d`@2XU2!=t-UW_OEbwR)QslZUR40VWKux1PUn+OGPwZ&C=7>mk3?st&gpuG#-
zNZftcGKEToez>YpJ}S^Mfk9nu^
zWULIxT+D%{SPYfF8QZ@_gr#;!5X5s+jrY`D6xnEy*!u02rJK3#3hFpSd^OSi>Wk~E
zH&+8tz&?Mn3aE@G7Ia*!uYUQJR!ew@04D$)9z5x~VlcfAYpd$}tAB?BncRaDFO3Hu
z?$V+_lr%Q?why}~us1;pSQBx1P5ATHL>9KB^7ZOuO`cYPoXQOtD5!kSCOTJah(hwI
zGaI8Q7d!-tFnIg~2ay=n-D3{oFzuAyL40R^a1eigyaPsnIfQ&#>mgrzz~8iPOf#?JmTB;|PNyF4r^)w~M;P9H`wz?Z;HA)LRge
z5oLd4S9?53!S~nQac?#R?l6SC+difo+oM<=vMC#~sc9+|&S-{16@Kg~98goFGnJX;
zHFA~)1lwV{7&1S+@56tX5eGtIxJ8;NVMq}$Q43ia3_VO9-!;cCVCRH^Yk1B53~8?E
zdm$f{*wy)MG93Nl)W{IuwTVV9Br-7RVSRs(iR1p0?6B(?b|i7~s5X2>nK^pr>_$-*
z#84G?iWp^wqbtAaRE9K+3ctc|>+|#N>HzjwM+)?cM?9ZvgqxiIdnGf)?BSex+)|<-
zOk~sFV*=|dlpWgFyCEp@(Dipwr903ssyD!b^RVCoEKm_(VIT;{djS?;5QE6D@)Lij
zPS&=7;WT#dhE|)589V$ptG#z~T?emH;I)0b*-t!P)o%_>*hpQEqi-=>U`p+H{iD}l
zle556q%~6Btm1TD2dUB#Wl}(rqd~>M`YvMw5CiCbPJHFY#_2@w
zP%+&#>Ch1WbiDuxVfLomg!}LUdcU`0s62P!S!lU}>7riZ#IB*%4UD%D0~dW5h~DQ<
zQ*jYNx$C>`=;SczAPsRMEuwHXotX~Q?m^aqyO$6;?d^3-9s`_t$YFoxm{@ZC
zG{7mX1Gj{xuNM;FxG1;_faAcM5P5g6n?-y)Y!>H{$FIaAVV43_2$4^A4t>mrtX&Fy
zo%c?Y&zV#RMwmX*5`0L5GlkmPP2)+pKec!aYfg?I37kX;rU^|SXQX@xk;Z>2;;awC
z+{GY_7yGn$c;i1LasDe4{L6p7D!_qZ|H7__Lt$Is)$8_Q7U36Xin}cGapNuv(>I&5
zI#4+_thO7UA<3{WdBPqZu*4-S@gpG*lq1P7um387r!E(e+_jrUV%*1v$y5VJT8_1nww{6qjHq)k^Ouf|;$YKzPLGbB8m?#F~0huO;Q$fiXlf!3#&66;S
zgZ<2tA&R+Up&?dqeuPn6#})H_jDP>!9#-2_IN-pzy82RjB&ha>{@GN|>oDY}uN({k
zRayEBGS!#{5XD(MAVz;y_?4s}lKBetKyv9E!*9&0eb?7|$?j~UiQR!R|JMGl-t
zVnH3lyvI)yKtU%&_}Rw^aGH!jk1g&qz&R{5_%Pv6G-cMuJqQ6gAg9l^FtuAifL(-v
zU3-66#o~eMlm+-4EEELoro0CrY-5-SSzj|YvtcJR-7Y`4jbAep#b_R*dJ|}lKF1RE
z+pb+LMBn?-cMj3;7$b<_eMs#6fRrl@Wm}{1;Y>PE3=bpcIq-p$rR5Wo~D5
zXdp8)H#U=zb0~j}T1j)IMiRd3SG>8QuvASgKyhq)++#cJwkNEJiScrvp`JuE4>T;T
zmVfBC)h_&xd~mf7{6cR$SQ-n^69dXb3Z37;*Ou3?K}YHB!9>BxO9n`x-KmYR?K
zwT$YvtREXkJw;p5OYi=Ob>hl#pf(=pUyemNhOT1V4Ls#EhOc9a~}qJWrihkQlBpjxcGgKj8vLqcl4s3b3QVL70(8u;V><
z0;PX#c;n5O#F)aqnYM_jP;E_jka&{&WK^X`-ixx*pr
z_rb&Z-j#vVSpXl5kVO5|z8>5Vu92^K}umF#J=dl}@t8vG+a&7rc-QAI%
zk4R!*n)UP#^y*QE3m0i@CyX79qe|GQETpa>z^jz5PF8O_|Wh>BTH*G+a@Bd^`e882ga5X!KO^DEJlBP
zxK)YBj3uM;bAMm7^oJ-*~oRg8r=Z*hVdP$m~ik{l9vraGs){pe7xIWPs)
zIgXXC&2W)XkJ9Lt1pV}Q44$Cz?xyVmvhbr4}dsEJf8Q*V6J
z=eH5JGZ>F7wQ*1FfW9
zyzB&o+)(o=m^M_3o_AT9@V
z;3pHWw*t3GIEQ1SB$mQTyR&a5UbfGJ#G%vM0|)9xjD?n=+d^b|l_-!0Ob4k06zXQ_|?e0=smE
zorjoRVLpv&05$4=M|?4-a`Kv;ghZb~^hz3XgjmDTJj%3^NweXuha%Y=Ojd%Ls@^U$J+*PaK_Xoi4Q1wvItRFtarKq!8kyS
zS^u>=vitD&%_4q>Ac8!ab0}(W1=LEqDyp0p8_y4*k=~cXaJVORvc3Sri;oOS6?7L4
zYr2uxucQB7TaAk32g-kLNskMk$_p=kaa3XyjoL#__Aj-u@(soZWe<9J^`If
zF@_N!9LTvY@k2un5F&=LKA~K!1bD$R&rWEx4ui3Z8i2?X@H~IXE%IE>lTJl|B{V=e
zRK~zy3TOajk7k%Rt4Dg4AaLS4Ol|8$JuR!Qp97&{$(;~3(#tTFxai@$dPys!e#sb$
za9PpVOC#`>czyB3;7Wp|;k~4TeV`NqsntsHg3UpB;#A0t+5EGQHdF$P{XEj!FuF&2
zt9^cw`fFpGoFRX{aZ-}7{l_=Td%!Dwe&k>bB`71$&9P``bT;XJ>KsX6YfcbEER!Ag
zbPjo!V(#^hVtgr&6S8wq0~Xw2WxmnXVl*jMdc*C@~toKp%fZve1`qkNWKleP=wHw#pYU
zsJ^kw8@_a?%r$7e*>3Ru%ZI8s5g_*M8pxjk9G7jB(h{_2x9AZ_ErwT41Mg<-?uwk}
zEDmJt{2|Za7$T@wMm9oo(x)+>yk;Lu7Kgii001KmHL=(;wGe6W@Uu7R(qODp1MX{{rpIP8hH3fV;Cbt5Hk2U8~Rb7
zeMyK`c6=tptYB@P`3T;s`iXWy;K1M8?E58i#`UXG&c
z@gFaOGB$47q{76^yzf1Q31^)U%KB7bF$5{<(|vzmoM;+{fN0C65i*A%bg%B$q9Max
zNyfzN>2Mht;2{{qU)+cH?jdD_+Pws
zdIJ#vA9TPoHF%tTl8Ob%1S}2LdN*z*P)Km0ypgvjzAzb%WP=uTl?F#y!g86ssroQ9J^c=Dup?pRa;8VaRul1P91
zr-=+NYIzqadEmZgAQ|avQ21Qcu;X;kgH*wFm{=*AgMR>9D9%|Lz>`X1xYAq`QJ(#Y=JGE1c24RN
z#razjY#2-$(B;^$#NQW&(IYkvOg4Y{>tz9K-nf$^c~VDPoE#<~kd)trT^SE}cF*hT
z_EHT&X)A1OmKwrmSKU61@03pTM1WSiAzSdp=Z9DizjnbufR_%ba*=
z?1EBCBW%7@LUr(K58-pg9967rp^3b?Ex5)RQn1;-p%iau?7r_Nj(Do25&~H!=VoQu
z_0?=L$*Ah305__tdsxxRX@W^tneg1YBOXA8j-IR2E$0~pGAKnbp2LSEJ
z)&fS?p%*k6&OMqJ>LCHAuqT=BZbK?en3Hbc43_BI`2&{Sf5DPzCgM;L=du|%hJn+q
z?=}^GyGrM)kZ*C7epW)eDc+Yw_!PA+-p^4?_@ZU8Qny+b!UIcrhYQJ>Hskq4;JIr9Bin+iQTpbb7?c^BLvPnSi`C%;q}De#>0w
zY@YKg)meYMP-jhzQ2G2HL4k28WwCuVT#SCC(L#m_{(G*f?Y^w!o0=+p6!^3n1=t!k
zhV!SUoY9$ibo%c$_E>sc&VjO;UmFJe(uT7x!828K$r0b3YhbHg-1yWnm1d&QsGfb`#)GVYkUZSXL|nP
z%S!@#8VJ<=jam>eM5rv^9`#BsNGB4|yIpdF0+tJHA^7seSWgQ^5dLkx2xsWc^JQ-V
zdZNNtj#+cJpEO7@P8wW8xTT_gvg9LFD5dXu?(t^xtT;HWC
zDCPBzyqzp`DynL}-PR6n-z`^KilnVa@BX9}9@(t4N|h|;i=@kyQ7ZX;S;~YBdUZ_`
zZS?)LKlBsZj&{a3;2`XFHzwMovZUp&?b7;=XClwjvdU+Z{`cGuQ531j=KFvfx+s!=
zpV3gAC5IZruSK$Tt?dTacXi{&r7V*ZN4dNgeqV&19*`=2INUYdkDPzyY+o
zr7V)E#i)H*uHe^tif+H-;MQYJJ%DL{7~8urWh7)|X;t1LLm>hgd3Cpa;+HTIjI8EU
zj&~fp-X;gfl)Tx-qdQs27cPSf2QK!j%&~moiPY+@nO^`6QRP#BnXtbpbU)^xP7E@j@Vbe@o
zd$xyO9=PLkru2wGHX`}m0q!2Np91zP3o+Yz*ti!rHn#TN8-{CHrA4J48cqNK(>lZj
zzIc6z->gctU|@M$Z%+h&@))>(JwO9G8rOkk)b@4{JBVaJ8lo@v
zaZ$-9Z%1aW@M>`Fv9UkU%U{{nF#GHfjM~&3>!Wu8Ut{#Ax}|DU6{*gds%oMcEZM}Brg1c^C)0PHE`YvC;s2X!7iAt%8%+$@Z#Qa~PG(}4`G
z>2HKXBh3L1P0$$7n8F}`6A@j|`DqX#mKs;5s;@4Cj@U79oR8AG(M6gYMdkz@uu{c2
zmeAVTjbp6CQ*f1<(||F1qH^*Q6Vhx0y63A$7yUY|NtNcgo=-=4Q<&rzRAy`^#=}ON
zgPwub0OoJ-4f_yoH1>aoI6DVMnKZ9Bp?W}WKsFpZoWC6?E8duYkHnUls1km-v&o8w
z%+U8!xD^7<^-$xH3%Dp>2zNoM!5fNoXm)n;bwk5CfCvmDr23cPAI~$Y(h{Urv4Bhr
zR&Y$$2m6eyb3w;&J!P=bzK3)j**a5z0WqI`tT%|!mwIoJNis~@gx~?61p@v7Woux#
zQZwy`Q)G)B;gjo^PIA%?-lcSO{`p+pgeU2Y&`O6Bb`Yc
z$-S5wo&_yW6{z5uwQoTGI%t@i#f`H+4zzO{B}Oy4Rl|l~WX3Iub%{WeQE-tU+1TUg
z!;J=t{G6GrXS&?kAf8kJc_5P7Z;46}c0pgo$B{S(egJuYu!!J^Q@0ss3LUOYq+qZM
zG`TS{axgepmV68FikiUgei-RtnA*1Xrw8G>28V{QDInIE&et*h)Q%GptFeQl927Fo
zz`2JLtEbGiolr8KXcT+67tXFEbc1etR_;Rx>6!&_Sw
zIjG{|`?m@(g7LznMbsL9*523896L+;yaLC>eDohC$w;P7D>{%5Bgy6Lu>7lJ(V!{l_j7WeXZpXcCC=>%lAE*aZiHplcFdc9&fzo*Y
z4e?;5hswK^i@}mOR8btNGoMq+yH;^C0iz*&LM~J#pdjD?e8dkQlSl7c
zNYl@Mj$1-apF=oj^M50f{^f0bA`1hR`ptAoWJO41SW*XfwTx;CT08qnudt@zw^V@3
zWYdSbJJeY47)S(0hAN@8RE$58ajJj>tEvmnL;Zf_t6o;ge`}IMLDXC73r@1^g%sHA
zdkD+I`Yn7Nc^O;%PHO;(nq$s)tuCMT66|3~Co
zJ3hNDrc4!F&)J;yvVyGM4ZVl7VS3U+_wUVhv0Q=jnyte`S<@e-N!HZC^WD_YE27L3
z6K>DUyS4Vn?>VL@BUE)=)MbeqnT`kgA6;v?)$=;|iJ>G0zI({Vq}BObT<@>4*CpqF
zoC~V09c$Nc?LY-!jyF-yl28i;6yq*@B+0|qLjnorJJQt9!tX32WRTcTwN`#9j-efj
zW2%hri4pu{SrzJ*@`cYz_hm5HmD03NK7$
zZfA68G9WfFGnZga1r-E0Ff=)nk#i`2jdyiaRBhKj4Wa@fAkCpci5VKCQ=~(rahQN1
zW?*RP4h2C-`N0ApDL_d}OI;BR0ze>P0T4)tl#|mC>FNOg+d#@`1jo1_(I}~Z|9~iA
z;1E}=P8s5gMQWi@z<{0A+|f5(a1q0Qb=-xC<$#
z657cNgS4@A#U}de2=G980kEW`INxt~K*14?K|&!YKnvn(3wOk3ghCtu12hx~clG*L
z2p(x$S63$~K|v1>4*`gyivSvbVqntuTJfIn{s01JTs
z4)>?`uRuuDZ)XS;igt8@puCVM8vucHfCDvPmTwMfQkPg3^6#Nwid(SE;m=fC25sq?oA^laKG7(k_?7MIe7g8|2<_v
zO)X;sWt}^J_58P0Q4#G4c=Llr0e&G-5C9eeivi-;gYUls=|Yfye-`o|zUn9h8j$#-
zUhGBvRkQn_G~oGjNqB*O$I?b)*$W4F{vo?LNE8IczQF%y%Kvux|1tenl>eRV|6P)*
zn}fq|0MDO@{|5kZL^^o=3Bl6W%@w-}T4?M-p#B^B5dOykE214>|7}xug<%0n8C-zqS8@Gtg%35i^9-;3HOdAOiL=f12L
zq$GXvL-_TRF@+B!OHWQ_Dc6MxE^F+KX+ZXtGd?L#q-e$~8we5B-}!t7C}n=GyJ)%z
zEon42n?4do!E9)@YxO%l_T`6pZ}B+Js!4OzmL8SZZWD7?ZwOq&>x+IuOe-NF
zd<`$8Q6as5u~hM$!th9wI)}@ieBWj~lj`b8N?W&(=zRM`5g#scE`?!j?Yfi3e)^j1
ziey+A7tP*SiZV6rGUHQ~Tgo7R
zyEYOph?`>d*%hM+3(T6Y@I%FpERFr@J#X>IbN?cLb(7;Uza^jUp3OHBQo&9hM3GgO
z-z9m_{(a~mr)pa115DZ(#o!o3+j^u4M%`i40n)r4DTkUc#S=N2ud+<0$$dkci5K>8
zjjkhx`AmUPRA!O?n}DtO_09w1rkpZuae}jJZ<>|CENVX7li*XlKy?E!ZMO2_0~XJ*
zLNU>Q>Fgu_0Ew=t@R8#-`8@ryyad%eD)Z|CTsSYKb;Ook9qb+J@F^0-mAdLS8Od}v
zUJlh=`}sg#)gjg?U9WoGyd<^;GsA+D9o&$O-%FFC5Vw+Zu}o!&_-MNCo!OwUQ%gP-
zzwBV63M
zxKt{0EbY8ia<%of=oQbo)Vu6x7Q!>^k-~;
z;@`rZpw&=babCW)XC5KqDG)Ku2Q`=p;wJdn;=HMdA@4}ouWGa_SD5^f*c-H
zgQSxCWH{9{^jTVZqO^po-A$(-Bus52%;T2=y0!bll#KKB3xwgMX8n;=PX{e=(ho
z@Fmc)RgV#gr3>E3!OJddXJ&_v@#m^fYo`aaBur*i6wmg8Ob
zP&lJ(TE$wP6J8npWFznLUF@r7B06oF_L1<4*LiP5k0e27SMLzdI)?PY(VoA5)J&1r
zmD|ml0%;6)J`B|cq!M-<_s-duz88!xO@OHb!;Spl5z7YjUJ!K2tQhXm@pFZoJ;F#`G#verf%j*E1J!Y||XITpt`hD0WSANNWMr@*ZzMa>^x%Z4U8I^>*=@v3~A)aq~1QO{@*
zi@;?@$A3*-Zdpd~BG&uPwxSReDK0!A1tR_3Dn`T!5|~-r&HB!N)LZ7cj5%{}#B6>hV
zK6|7CQ1e^PM$NqxkpF?(lRG4uJo+i`{xhzRws#b%-l>DAPiUwsVC1{$EVe(E(MFL`
z`^8DRNFDtmZDSID_1YV45leeUirNcB#3%zwLy7UFqnPuIh>KD}LfKi;x|FY7fb(LY
zIwYm;74&Xtf4wJ1%RrlswOWJ1t4%-VpHrwEaBR&!E6&3XFDX0sBljnp6f<$E+Xz46
zVI-=hL)pjkT(Kv$Dcxs>Qd{UrnRwH-mj1_(G5?FZb0~{{pTr&}#6+reqz99aAJF<%
z9mG^;Rd3ZC?|q4hM}NoB&}){Lb{{WhKAzCR@l@OU`fSp5S(P$EyHe+-@D{?mGp?0i
z#PiNsM?KY#K{pWb`*kal@U^7?`ZxIJEMWTxe1v4EPwCTvAgu%%aMt4(%#R|)A1i7O
zd4s$iK2rC8O&&X=Hk}_Q7l7HP@8Uk0tA3HQYn}1*19aScpI1&U(G|a}$TY(^3l3ct
zxN?zU$?yshCrgyL8KGW&ll+<*^OgR<1n-wy8dfpt;gGVGtP7szBn~01m4r9b9rtYs
zxrfr)8P|AHGW>$YC`iM{83{9(o++MRC^vXa4ugDu+AR8KmFXAe#NW$?7+fejI2h3_
z63YDOo1ktJRj+4iFo-;`)Uo{Zm|{b60}ZfWZc?@#J5GOhgn)xmRM+Eg^30SV$Qq;F
z>F2dszRI@%MEFA7A(AZ=54)ee+~@J{zn=E?-mNGQM>9E?mf85s>@;{ON^Y{w%WR0w
zW5P~+STO&w25EfE~a{zF~`piAecEfW)~_vxW%qt
zw}m3VMSFc6m}IrJYorPE%UvyF=qiWQ_)Nu@WcM&z9oUT`o9E%qpKLcOXa?BUbvmQf
z3KsmvFWi=PEaOohou4Qu*yYQxmSx{%&$pI;M;cOPrR9hR>FbZP&v@TLyfik&-Pgz(
z6lpQ0qL$dLU5yXS@3X6w^>}yJi03v-xa5uC*72I42eaDtG%ai~uu>63GO6vkdd4S^
zccs_Q$oA)cPX#iW+qH9Nj+!^jZoazuj$6K4;dC^v@oK@3w*DT~ymzJFaL=vBxdc;x
zJK0h*F1*?ft0(!05u_Ym2-iHJD8A;^bET9(U44Ym1ioj2
zc+K)7az#+i#JiPm>#Xs~EV~22jr*xYrb_8VMgcgrJ<$7W{rs$x50ih?!sX~Nzu6lav`Zhk6J9y_}5uG+#t3wy_
zn(_-K*stMx+uKNm>7HcUX<;*w`ReslDC3G;8JEnivL96y_cdRZ@GIQM1ssw{^Vr-&
zc#QAR)+P#W^RvZhM%SO_X~kwwTL!tQLnY?`nYvAd`n+vx>zs_I3qCiV}{t=bDrpJL}PZFr}DN*uM`b^5o)UOq8NCmRhsS|i)Oet9ESK({H|&ArBX
z$YH&UXM#R`yOMh{^p&SmsXb$rd8b{t%UjD1Ey3<@Bf1%n+XzwR6dFF~AySl~A&*Cc
zStVMuHt@X`y02PSd=jA|z6N7AvwQJVuS-MnQhA{*pl~F~>)IE#0v1PqU_X~=CRrqY
z=>nKR4x~B6-!aI4xz@_1rPrt!V|v)|C0RMSg4}leA)nu!@Ot(T8S2i)5c+kxS9s~f
zaKpnsw>hKONNtz0c`Al=HsupeTey=%L)_!t5FfL$0_pyg2c}WM^Tet5wJ
z-X1AiF0*7fv$NFFVk2ysb=zb(l-u(?D^e`CY|`b`tG42a@22xVGlx(iZeo+X3H8C;
z1>c>`uhw1@=ykcVSkA||PZIcWtZjHn=?6`A%>Azk$(f3ic_8;Cv?p{_N*eRni-W82=Dinz+_?Q0{K*{Q6Z2!hm0}lsPaYLWw%C>NApH
zC~Rg7lz7yYei$%;Bn#rehzAyoWXdF}3U&o(OUnqleVmgdr+qMnCu7wrOIs!Q+UAIB
zZknh7!Naa!SvKo`EOeRaE&5Svg4OQhnVg*dQZ6n#LrO2CC9YY5g1uR|F~^LIC
zeOn0vB*i~5nIs%`>{^cKg3A3laq7sL9}N2r$X1ucK#La~RH1KVNh!nC!g_Ah+%p*v3=uXhcbB6$H=Wi_bL9ihP8Z7BZ_^
z5d0Vt=9S$&w$ZYc-)y@wG{s0^oqCp}@IlOzcJiTrmH1z?cS7tL}ZVw5C`HIl3vk6*Z^5YYhe
znXCk+7vJkDXudyLhS=0ccR@XQ>*-7_Aa@Zlxlbib4SaW(A4dDD<1&08y8kt9+tdKe
zb6-4vivA?!Gg(A=?Q;i(0afSi7;s)l4b#`tX7=4ZF6mFqf$^HU9-cS2`rF?)Rtt67
zO(Sk!(I(?`Dl1_s5sSTu_JB2Qw`dLsg^YkTfXRYk(!O*!Lt3|#`Sf7mEJvQ#(UGVx
z#W;|8#sKc)FVSi>z)Zk)O5WV^OUhu7>NpBX1H2zHi=-O)C-z?oG6oGwk{3ZvMFbQC@A$E
z{_&`%j@-+{k#fy7lP?q=8t>ks^xWuGuwvIDNu$iZoG&G4S+2e3Z8wzH_z};eT5K{n
zy%Z@l;Ko9^@t#b-JlG(|)!xT{?~!DG;t7fOfEuzA|BgeAA#SbClsl2L1X#ze5VLk9
zBGzy8I;5?^Wu!BFNy8%2NH6SkD2&oqSCLC4u>f4dG*9MT^huCOBc*#q=|#Jtot3}<
zUsrha)k&pvhAEiC)EYicaLn){%|yK-R0)~T1W`hfmL;!Kzuxd5={f0>ZBWC)2z-FZJLZ9`Fc*CvVQkH1LhkG8#O$CN>Djj(7cik1^oWQ#Ysr#RBt5j>GG!7$R*LY2(wmM
z4u2Crq(vi7Sv%gc_t=cqVxi<(t#vs{FE&8%WtHHN(R}pste01ya20cRJJFo6L>?bb
z-c`>#<11eiAw~$Q2`z*r-Ect<>7H~|ZJ$ES)&iv>Z~V~V+_7E5z~%wf3AuIx6d
zHBws@PSCrn1|2sh+PZ_tCvl2*(?jhS3*#RvKc5Y@-)sGJ{DHf#DO@I$q6{&v>htE1
z)#;6?&_|eg`TC68f(9cGPmt0k>BU9dYr+AQGfriw5VhBhr?Q@rVwWl6obaDZKg%F5
zK;SHu9ec%`tK|eZH(gnOJ&wcLQV6a}gL&PaM^`WzN8m0#xOFfncIF^SU->PTpZi;G
ze}i)_{*o(nhx@im9zFRokI42Xt^23N?kw+n&(g_nw}H5H&ruZ-gws^(1sm1-GHf;}
z=G@PIL^RXQAWz6)kW^Bjh%va4e1oxAF>bQzsa?Lh|1mSol
z30IrQ-A}b^tg^Z`lV)~S-;)@o+V%0^s*F|D#i?Enxr2cTnQLS`Wy`|tZ|3Atc}iUs
zncO8?ThSgmZT-1LWPLXj2BV}13J*^Q=<51Sty=SXqU;`yJ)U&H{!7YW60C?=Q-@y1
z?nQeWIDmE-LcaEYJtt`L@7|sq72nJ}<{=k6H7NSx?ZfC-M)uN)y;Hjv$X{0
zhvc*K{JPX`o03X9x4#v9Wc%KwnU?pMt|4F466`OVchQ4?^Omvddj&#;jbL0x{^!G9
z@#lrI0-^A$K@VZC#8qC@4UT|zYza->H8#G^-E_CBHmZICNlZn{eK=p}vy&N(j1nLW
zAnV-kSJzy$Y&VAwQ4KPE7JWS(*mgs?zqj+fEG_+`dn@F9+n?izWtaNClt{
zTJL>5;|)iDEe*bj9ZL2+qE?B1gW6WNS)-I$z<)zF662R$sgLM>#q=bzb<~ejU(M{n
z8au3$h4DS|cCwuDKyU5g(v{DM(wQ;YvE_<+M(Xjs!|}d^hF28Pf#!QT@@1jyxf}HZ
zIELbPJCkEo_0-=+G~)^*x~K-F`%dGlZ}
z5vAVSt6T^C8ArW{w$jvV*;PHg(cFa485#$F##zFfNjs}H{t0O*lWIQ*a`8G&_wVnP
z^}HvjWVp$2Z!yS((q+^F9GfO{)zY#j*t?{CnfWxSvn=aoH2bmH+T&(lWtE1&xE2)$O0)-Ry$=V?TX#2TKuHAOu=oNrWn(I=~FLllyrxS_6fZ|zMA66rhuh4fzj;JfR9eR
zbhCAKu0`g(?895ZeWFcP-?*Ob`!%PiM(E_v>()t`(D9U@l5vh*cH$1jsJEosTijBg
zyS|=?NTxJ6)_Y0BxbgP8$j5o*x(0&PSF-&nDyrGI%6`%4lNtFd`;BY27`%^v!4|Cr
zx3|0CHUpS<9Ic!z@1pt_zFvS|~M-wCreA8AR-?O#rNn%*^Zv6cnP4
zAfU5_ovj$q8N>_VayADj8auxQIs$)KnVESIC;*ZmTae@1q6xsr10WA_2C8}3gIEC6
zz<+>}os%nTwM%
zqn)D}qrLS%`qa!VoB(cij#htww^v7yHRzwlxY(M!wbR)g^sfZJw*(+(VGOc$0{sq>
zu>04d&08zqg5K=T{}J}q2(1TeL*2K`p=^n1@NZ2z&z3oA%Vh^wkG$h{4oErYz>TX$?3
zo!y=P@&27pSWJ!=z|G7FVBuj0Fux6{xUGq(osG?#YA1x>?Gv+j>yxvcqX*OfOtqD*
zotv%K|5{BgY)wpmcin%)#hyvs*22LBBrW!TzHcUkKQ=RvGk_TYasYtbjm?>UQ~fh3
zzs)Sa&2KI6@v^tG2bcn_oj^Vorl7YM1TQC`D+u81=mPTb`g`HO5dsSrz{JAX`EAJG
z4hq6Q*`;kw?EpOgGQTPL_tO7uKlML{l;-V(n%LP|djL#8rU-vb3U%sr%
zeQW%0wE!k%H7S1u4OP1T8JB;Y#BGi3Oe}280IVFG0HC8I&;x<_Z3b95H~?NOZ=-Ai
za{p&Y08EUwcFu1h0DBi_AAqTyBf{^E+W7{-8Goxj%>tz$E_%z43o3{6TMgivK~}Z+uFB&>Nr1
zAN0nj`Uk!7sr?7>yfFa(pf?7iKj@9Y_&b*~Yxx&^tKI4^_*UTGWN+$h{y>&DEq`TZd8^FsFZfoO{U0fBzV^Vk%h?)a
z>ioyb@_*KU9ryp3-$dDi94+kr>Kn^jp$>m0>o;-%IsJ1~{`kCA@AwydE7j>Q_?Gn#
z2ge(=v$-SaPj5NiBs#nOUBO#3T>gS@&2aq-zBPZt?XR7Ci+2ADzNPp03%)hO^B?%%
z=T+(VgX*7qnEB5E{=X;qKUme-(as8_X<_pA?&U8Rd7!hSg}W~E+a=2KW`Fzn_ka5T
zML_ZA{{JglM8wYBi-Db+6TraA!vSF7eB1W72DAD6SFG{B9;W|X;BPn0zwz%U82|)w
z2N{1OtSs6Y^95U`G>4Y?iWf|mLsRfDo>t-rXvxGut)x$P61)@3-6jPI76!EXXHXW}
z$w~3*`z8h07Hd%iV_Uy$wk1|g-Iyo~?Ern{eF;#+g~xI<7}WzaK+T+@Z0qSGNqCA~F_n%oOU(v6UN%eoq+t>4;-HPT}TCg0gG2EBY@Vf|>r@O$N
zU+=MEqJVwEE3|VT^FwjQWMJ%jd(d>~Qu5Ud8><%&O<v&TCQh#)hPIuAxn3r(Gjm{vE0RHz*1S-U}e&M9!LjY$E3F9s`M(&AjpesW}AIJ?(`
z&nYwz#YHN&y@e3oDgpGe(iVY$Uq_>kU`-=DzW}jJa6T1@4w{OH^b`_&a1H-Jf_CIx
zp?uTLJ~1kXQpshx?eB=hw(1BJjADNXz8dCIDyz=jsRnboXVF2L{C)r@&N$w3uh_4o@X?Ba#>ABT97gBQ
z-Z2_nE2B#nnwr1O8c6YdaC
z%XclEBY0&c-1vn~chT3o2O5be%wBVM%FoS41aKVg?HrNcTctE61fIXAo7F)fT)Jd!
zTuk?v4w-LgysZdGTT&bBbIQYed@j4k+(zG+noL5jBAftEK(N0Fgjp(rp80jr)zLp>
zL7#DuD_xIJNy#%Fc5Y
zfif7BUbkZlV_@;N4O1Vb#_pyP8UL5g;Ud`_GnWu!O8bOi&(NwzwvU1hrLMz&_9KWA
zI;vM!3jr9IrOAL&8Gwg%t(IR!7#f+;)$`lzrMCNv7EA~5E<)Y
zVRK)`&k>U!&4%CbrbNJtW_e;;j$C^~l4^9MCA+8WanAb|m_ByW$3eG<3JJlw8I^RZ
zspCRZlN78SGdO$QoRVy*LK@+JqvM;dmY3VFeC$>rV>mKjg|{WqSawNbZ}obAgmzx9
zjfI)3HfewPZIfZX~gO8wecXo0wo%|BCT`7X}Wq
zJerY;sKIxrpFYA5G}Etw=a4B7(Zy;aqBgAaXit6FYexV*E_6fiuGgj$>nkiV6#1#E
z#P4|61vxWaOWT!Tg4ara%|%NsCkY8;%|fOU24EG6$vx%tkx>!z3e|aqy3MDtsC(-9
z{W4Ssj@S92ab+y46>PhbwfygbT@J5DfMc=WS$}lUh|)Hwe)wvmDz1t@*qx{Sa=q|y
z?xHX;N)}z~T}CNGkVd|lz0vHrH)XPSmzhsP6O?+Vsb1|%G(eVr)}2%H`A~GE54BJT
zfO^>ph4qwralyf@Ln2)?x&ndti<+=PQeS~t@
z-+VM+H_}AFE|kBdC{A)b!CdG#MC1Wr;BBL6pgo0umz+6mrHg%px^y;rGNzk)
zSQDDPAH5pB(h&CkC^uL*o&t+5zaWVb!!IL4PTllQWq#G4Q36IxveE`0dx{93D`o1?
zut`^=5(=U_2L1L5z123*mq?lCMH3e68>FRBq95V+XW73&cms!j&k@!3Yhs!%B;0}J>bqc^Am#W*cR6X>}gg
zQ*b(et{ATA9KIk@8z_3-~uu0LFBrapvo~csm#XzDgUb_!`=sqYGM+z$8l+3!a
z{WO9>jP5|6Rlfm^9|B`57a4>bB89+fwoYnW-)2JPp;$MIoTX^6KY24Y;i~G+tBmJQ
z{Yo#9iYgL_k2pHZqrKeb7~boK88El{vrg)NB6V^*Pt!_O0G?P1(95*z%cAC<4e`Rs
z&k`py{XtBkXjskKt8SK+l(dZ~!r=sSfJE&&{HbDx6IwZj=OK(oDa9
zSLTc4jsmr3irz+)ydz3i%P@>Q&kxbq-H(=+Vy+NUyzKbed2I&yA)|^#?-C&OV5vEQ
zGZT;fa#~Tz$f7Z1nm$h5c{Wbnr4!0jdp(-q_w&oUxG?n>jjOZ)@>uJb!0{xtHAaa#
z;O@@KVbd7|$vI9l+fqbRRT@U_s=B#E1hPUkV5k$oK-%D(p^tXI&Hh-L)3|)?YtjDv+
zW@QgTTiO^yCC|CHrN>084ejqcDsKb>7P~5_d!L4_Y0`=`-edr%Fu|92(5WOo&yGe5
zLzp}G=FNMld{UGP{OtD=Sv;qC6thF+<+^y@XIp%oN*L*q6onscyHO6h#?soSy&6++
z^_w|FAAve>t(OIr-A>l)^dP~1BFd$6HLLNpamrq$r|yjPsevYm+vQ%bC<3_=y?Tm>
zYS8a^a!6dB>WtLxeLAHD_MzT1`B}F?x3i<}yc9x|4xeJYcg0oUFhXNUXE)8${!hDv
z@c{iW@|O#4ir8KkS8*G`50F!*W(0nLNZh8%$`(tK_%n9t$FR&zG3Wq)4)_dDbu7ww
z>5F%k>Zq0OSp!zpdDGMp1e36I1JF5Pii!^L3DdP_THT!DkqzSzVDw^r+qvm1Sy(?b
ze$?tKo%>2g><0l?qK);`bxoW=RIa~5UN*0iCt2KR+%TOm-vz$DoF2WXs6&3EsP{tT
z=3A|hL#m)6>^p7i5`V~l1^4XbCJ=%pls{e^vQz9t-8=`Qu%}>Ff)cTP%6l(d{SEr7
zP^;b!XqtPFJCL$Qu(PhV>QV>1E$%sxcjJgljalwaR;nuT(@%YP-m4uANB7RW7DLgD
z_m~(QaRX8=Q)s?190N#rJXjt5#vwI&RPa7VO)IEUtGa8S!Ml)uuWeM{#dq))fBdmq
z3E(x#!TGk1(`@d8yffmmvE*pNBJ$H;RHBzsm}5&*qwz!Rw#v`I-7q;cmFbf^S)*h8
zoiebn3@-hvBx5^gW`$5b!iov%&yD8LSkHvv8*7z5B)p-!)!j0eAK3jLQi0xdE)$t>9uw>Jl$-&l+
z_!flg4nlVz7{vCrAWj&HdfjOA5sW|mDAV+7OuqK@2iyEf!w()|g;u=kkY}!Fv%k@D
zPB5W#zV+i9x-=xVAy&bxvm3XSDJHf5s!2aGM;+u6i|fdL)<;ViVpABm7U5qI4DEVa
z>LMY$3U*WNNjt=JDN9?ADp8GxQ*(KBUK98X$w`>Vk78;KqYoq96{!5DUg%M|u%qc?
zyiqj{P5Y;Yu+ZH&XN*w=V_qwmWv`ixYDt23kH=Z@MaqaQSGTB)wc0#Vz=BprxorVi
zF})j#cNb}YK9AIKL&{X9;NbHx7&rB&0T6t;*&K2&0Aj{pq;T!b)9hS|0H5SLYo}E!
zK1c!>xME&u9X_9~LJqh&Mm+@I8o=T?^YnTe6LGo!Jq)^ZFz)V!k;0a2{xl@dII)9n8>!
z0DA$T6;}fm6Po4ubN3uC;+e}gN%A3|4E`EkhDPZOA}s;&k%+xs^OsSI*JLfAlyrEv
zmi?oDCu$DPDZ^RGKCTyIQo;`7jp}G%UV5epp}`#N@!%bEP|Wb~qO{#Sox?y7EiI$D01kA
z@amMpk|beHf+S-T?w*-S(`r3_fa6j3O>ii~;`5$nZ%p3vR630K~_*HT3;
zB&0QmazgZ8vsyXACXlt4HrcZXH}_+E`*8IR=Gw
zXJAox{jz&P7EXej4KiQy{@~1iV{Zs}56%SjN%CWH-##PkRtJ_wpTH@1YPVYK9vQK;
z1=jS`s}Tbk9kEfulIy@FPjlPdM+Db?@mS`p$jokw+2&thSr*heD*t{*QY(zs;4IO6
zFV2FiDJNQ5I_=PX!Y!GN2z_8fm~N+rWe7h`FMI2aBrUy}v(=@y%Z}fFV_+d9A;177
zs#VT30-23wWaj>ZOl6`f_6Sg-`YiQ4x_wo$lYCN)mHR!=G`|@s^Wpx3X5at;@)4?r
zv0RWCh(7wRV=zFu2D+Lp&nBUbeXqWauM;M@E?r{Ze;D=U9lnKK6RwN;OB=u`s*bqT
z0)mA`I^*?85;e-U>b@+0sc;U!{*1GVS-aTmicG3m6&FI`A%E2NQIJV*I)F=M4qt28
zE9OD!y$6v&g&&1BBcph0uifYEk*gU(-QK~X+)s%V)Q!*gyR?QjR7`HQ$6v%O=IGsh
zPt_7zCe4su$QIC{Etf6$;sBik0A{8^9&=3Iky%tV;mW|i9`;9nw;5Ov-9k1Py+*@^
z2;b5BFzF;GBD_XlZ>F7()gG&3Ng-C?hpwVz@A^g}lXO0+t;b?&DZp5dClo~upQI4x
zTk-IwYGi9ad9syV6QHLKZ7fzp))A#KQ-up*O!Vu?a*{KBcAJTWr+bGZ1-TW|JO;el
zrvO(94OTiw249?>alAJvYr%H4YUg=gT5LgXQT4qZ&+bbL#K&Hke&cf9`#
zxuTyq5AIIbRrh(I@ua*&HED7>=7oqz^j!Iz{s0ru~*gyYAGtSE7Nq?aJ@R
zO@pDNCc}q&DJ*aXlDJ;`1+sxQ>kztB^g1C~{R`tCuP<{C>pRn*mCRTS
zgx@VO856F5(eadVK{C&3orPc@j(u5WKgbICjGFVD4oC-vhP~kDYt^j*@oNR^geIWQ
zz&Ckbg4TL-t)$lt(sQ0w4#P094n8hR)Hbf}COX9&;_#!TUy}pRlQqQ&FkA&tY3sWEHO|A}e
z_kiOjV(dPuUa5|W{8Rg~D}?Qi%qVIN?4{%7LfJo!K566uBG{WsfyJoPr=z4i@7OU!
zD=S22EO<-EA2V%lW%ftgn8_BOH^FtZD6WMfV#07v>>`O!Uqz%I!DivCktM=nPePCV
zLQvs?FdX3z{nj%8&RkWq`*^q2|Sq?@!pJ!!}q=(@&3t!!do>{H6Zc{A|x>X_MSN
zzy4kN)eUb!Ta9u4#$2I-+;#Xy3o%(D70
zFKU;_GbHpqEriMbxK#di!dF`svq!Rv2huozv`{$+97#xVtvWC%s-CR!&fSUc{%|{{Ch2Y%2zsRNxo110g$VRhEmt$Wz^#^|rE0<#)?W|}
z!yfT7RgJ${Zl0CF(c#xg5sUeUt9Y+=4>{Ly_!m%-c0ZTS9D5z(=s%rtkJ{JM*(#rK
zX|N7q1w+jy#hskL0~442ob|eFd0nA@yY;>V+f&4hlpMzEkR%&auc6iA14qbWvNBA=
z6ESz!?Y5^ry~8yWUXI13WI~D3{Cl1xp97
znh#epO3M~?=uYf~;u(5WRG|VPnRhh#^UzzT{o3s8g0HTTncfz?>y2=BM60R!meqbN
zp!kDLOhq@HAbpAAEJAtoIa)SfF%tg_0jJF9a>)O2po~1shR+z+`}Rx%e;kTP{1I3d|4m#=Q`5+lLXL#v8vRgvbmFH!XE=
zG*`pjV5xZO
z--#Y}C?%o1zfs8)7sJzc;B4?04AkyqKGTovVhq{@pD
zAkm)qXiTaE9zdD05(iV8t^-n9J#{jSeiFh<;Vh16bT?N@QfO_EjEj;63}eyK4h6Do
z(EfB0_8Zt~R=hQTaN8o;TR;gwOa2k-FXi;j*~I3?FD>KOt6deA($B{Xk;*DglnQ&1
zi$MkQ7Q(>yWI$Kc+=61f_Mf+088w#g)-yzQDD-o5tx~&bB6-qQNbqT)!GLh^_LOwF
z8tP#7#)9j6@l>?kW_iS>OC>Eg#~y_bzRqwc`x``-*_sA_ExPJ5T_8y+=;voEJ{7|{
ztSViEt{`j_+2fgewr@4SHBIpD&f7S4D1;01dekOEQU1BkDN&tbdqRuX>q8h6B0qOM
z+ik;>vI|81pUpLt2U!O3qkj-c$HUEG)cw|#lq}4r<
zLx`7LYw#&CYyS=#=$eEcPZP>NzP88U(C8=4Bk0<$x1HKckx_c54Q%?jEQaP>(Zs5^
z^5a?hFcvNT)ryRXf4kxRxa3Ep<p$vW+J>3P}Gi=D*ns|86L{qtwO!J36knvZ9A^VvEcEjYU@qTXCdq0NqE;d
zTgrL&nSAkuJ9TlI2hQs@2CMcEZ&9>a`&Q)mGB`1>*hg%tZ~3TYD4*Y;Q4sP}eAA*K
zjz~~{;8rgzo;27mMEm1-5~Z|=U6;gTqj2)+-d1YyAgd<8_j2)@SX9b{w|5qrT?U0z
zxhb0Ly*^C2X$fvysMmuw@hty&;dxRi1mMMyJZ2~H6(zH;O87+ef^+d38R_HCR2LU-
zPObrE?H&|DHflgVlq@02KD;ehGSw)9`G@ekaZE=BW(
zBi&yG8x!(yks6)dBg=Ghs%Q?-=5a{MiTrqMHE85yz75=Hh!DP?)a-Ab*~Yj>*S5&z
ztq*?KdTEoCJyj2_zeH$-(w?CV9TQm9ErR%U*(oa?{@8<ud|DHV-YWZ
z?zABLHRN9Ur#k)VFaF>==*oNtm}D5U)-GtNEm%UVktkf_5Iuy3Sbx&2m%=G^^zdwq
zQpC#?4ZUmkLlhE6$wNq-dJEDIRh)I*+tf5?-fCQK{LHL%>>z
zZvQq&=yr*UicvmMbGa|?h^x_Ch>EE3QxLsN<)@j`jkNfqyQvRtWDC;AQc?)!s%sIc
z_ha8BnM??OnIgE3((GAfcB;^_1Y-E2U8B5!eb<94S32(YuVY+Qijt`5K`8crGnh9C
zf%4XFExyrZj$w@w$*kRg@R)uIlKoK9(Aw)-u|RcnCy1t8w`>*AdENgyC*p;xJr6;D6bN^**y{Bvaj=!LtVZP6_PtoEXR(Mhr8~^?S&Gk-
zQG8*n77YM--J47w4Ws!%ilXaXk~6ko%=g{%wTiQ1aj8bvbc7!qLwQE|cTkM5MrE8x
zbM~(*pF;?fY1-T$&?afWi0HZVq!C5?iEH56_dJpA2ooC!y^?Fk=bo~EzDIwDj#cne
zAYCrf!xl|pB&`w^V<*8ZzkW6=e7NMthqO*tieEo?=@JXfZ1YE%MBSWZc=1J!q(PFQ
z0zg})4uzXJE@{}=%czk~t4m%|P}Clxt-3*V
zg6C%im`o2d!x?zYY!$VCY1mJ@tDtU#
zUB8{_nmel(eX%QhNyH7fSP9bq41uZz71-6VKQ__-YP{;Wj`3+ngipVlhiCPRAE$0x
zD{hHt_m|7m1PAGgr*7$yyJ1VwcpIn2Vps!;NLIy==oa*WG8T;v<
zq7mp4GlZc!FT^B&A6y6$9q;MB>G(9Q@H?|HDm8wzK4|@p4wfjpkkO(e(e=woVTgJo
zcp|wvA1%yP=DinjlD8knBU7yGr-l26L&sI*^7m$)G~7Riu~fddfLJd>9CDOD?EUJa
zK<&cD61o|j5LX=Xwh_XE_=2t!^rig+?VU;Ho;HP9?1m_Ryq$BJ+i-a9YyVk8RFT*Z
z_#q`;ozsJiJr^YQ8zzOhSBGDa(a_BGRXmXy2#tZ>mlf4uZ46EJAA7Fb9L_wo0XOBx
z1IAz$!jO1^e4Fn~KrM;B{Ie~ytN5BKO{9z)uvrGqmmdBiDJ(WN4UZ>+?KyO`LMN+8
zE8}NED=L702iqRT%X|OUZ~jsYbHjKahGj7OWE1y(B(s-~x_@oQxwX~_hHNRg2BdY+
z2#zIoEO_EPJu1&VeyQ7vH`sEOL*1yA?rCYj<7LUz
z1VT+W2Z+QRW?P`PNfJmypX--=0r9
z=rfO-hOtB!h-CsSwILBV`oEc{F-{4f|b%HU$3F|HGXLiB}To2OkJc89LdgYtz?Y_4DXgo1dq3`u?APp3F6;1;zrA{*@J_2G81nWhWB;sKzYPa4&0thTW#!+WsC#QM;B
z#IOh*X^1;;`z2VMR<4p8Hs0Aq35IHo5@-*8Qf|;=Wu0y5CBFEmVF685crtUzp~ts>
zI5l~md?V6jJUlnWACz?0uC~I`REk?&r%DmnQ<~BL+*7F;siJ$=_OYJfby#|I9dr9}
z3+hZNC}dQvMq&jfzA?wo%Y4wcK8ANEb^K}0a7t%SkP@i;x#IFH4AFvqIt5H08>}bZ
zx+6{^L?6k0uf%1)WffVfctBVKrH5L7f125a3$Y1{Fur!5<6Zm#JV26O^y;4umff~|}p>K39IDjHmb
zzlB3(k`SO$Zx96aYPz*9tu5@3ZXOg0rfCD>)I+uN?deL?=!M)!uw2^kzI;!VN1;)a
z$4n+Er9}$%TpxVr#4;VqEAG^Ptpc}h0sGy86(-z_ZhXQw*E|%JP{rkR-IW-|<^$zJ
ztKOE_5|#*wO#rF-%4d4o@#is@A}&?H(RpLBr+sd2vKu6tUfmj<>>Z)1G)ZqZt&ew7
zIV-quJLgPRdlhc<7PRAR#w}i_HOp5anM@|0u-C=74JSVPXrlFo_2ra*)kUAr%VhAj
zQQ;zRYMab;WqiUoujS
zpF_QXS3u6^6OOsGIFqX-MkKH%k0hn+Rk|#m*7ffanwmwgu8)we&}dP}vK5a9%rfws
z8-J4-%k=Zr(65W5QuGRcu`o0Qcz_`d6&JUM{)`iFWM4>cnQ`n=sdQfy({zqT{_58*
z2m>FF!AoVw9#j##bmCF!YhpH!BnEqNU3g0Q0~AnG+PqMuNWGy8K_>kQ=rJQth0XDf
zhL4RJ35aHpdSZsJt{=um7iAy2p?cTT?LD8jzVVfsgh%=PTEZ4B4gLpt9!J
zsU=Cqw)Ya@H?A`A=M7kVWEU$vpJVJrZz+v6cg~rGK&%5{_(m_(82PI#$=Zn4(P5ZP
zX-sx(W;~tkMwYLC_95NT?(f2sg_zX)qmgoLh3k&xiOAvYkfbn7DU-Sez+?m*F0I-bAJw>tDDwJJ8{lkaZp0w1TROVlwPZ^LmHm@r^+
zGpiIMh%VU6^RF1sbZXF}$t-W*->o^c+{;o5(0hK!f{@C8bv+oBH{Lyv`q}`sSJL7z
zQKKH9;!)7bY!ffI;#<$NONca5fP0i!=HM1Hw*a}l5Y(gN{TkVHM3?wU-?}3w_tOdnVgTJjpRbqv%NhX84~9%P`t!nw-&Nq;^&k_g;jf#(Bb?0Sn$~;llxwi`uzj
z8?EJkNAUPiO5u?ok?XV$riTf@7zM(u$?NG5>7K%Dr#-5pVyVbl=XPH_xKb80Qo1>7
zYrqVK>1`a3{czB^jh3Hr5YGbBkc;>I!=kI2D_gcgSSmBwLOwo?$g_d7^*ZrD_9D6%
z*o~vY)P>>q?s@MYoijxgQrlVbmpN_gh*Kti&4}ZUeko6rGB{coVP92736uP|Di=IL
zROs-Ls4rqVJF!3z8H0Aq@4mSzQGQ`)|9P^?$-vaLfo*AV&88~s>u&)JXzi%E@8nztMB}7c0X-X=Wg|XB`MjA
z^qotuLIJ03!=5~Qr_*iHUGHF;IvDy)!e$vn+k%RMbiSMZ>!Jg`*_LfYO}io$E~`iu
ziVqV-;Y29^IW7BQXx!4P~|#1aA>YpIOb!_k2!ctlJle2<&_3`TI)8@ECRO+eCSFf^Zf21nPH+wxAq=xfvHGdDQHIAK0COw?SAe3evJXD2Q
zxlo$?SxYFlO`~lz3>Yxj!&f!+s@lhg#QE4dE$^`P>HhtbgP=&y?)H>(S91Qt~#29^x;UDkR2<^p|nzQk@SA|mgM}exV##F%9s+-J`ARJ730X{
z<56%2ur{CVJ+7xlQQ7zaJkKS8)UST8Ze|gNwM>D0w5<72eO4o@a+Ks|<)&Z1%)$b|
zKbWGB4Z2kFr#yX$v^7gvS)5<=kQQ&qBBcv>o+%mQ%(9tDom
z0?i(kr$o-Gyqfi=*i5*)WW-xWo>eqGr7PEX==o{vLNZb4fz0|vR;
ztNeIYJ}*H&%olm67v-TgG`4qH&eymwxG!0qzyvju(F>x+uu*
zswnz}Qin`rP}fZuP1t+hj+nNXsMUG%Qz
zlZJuV-GzS)Who;?m6fn-LkFuz7(e>aJOVkEl))ZhoGqYzcECUd+4))J6H~uT8&;qB
zNSFeD`vrcC0Pbt_5!`e6`ZnG?F;k6#m2m<)Fa8Gl6wd9X-4hgt(08$#gix1kC@0u5
z(9h#sUYfDwWS$;+Qr}OH#{Pc&maS1BV)^-S9(jY^UyXYnfdv?c
z9V3iC_#xBareSa((yX!V5pmw-gC#WM?cs@ke4$~G8a5Uy6;8cT_nLFVoltwk+MBFW
zAG|sRVk0y9HwoNU#4qz|db!;ZRHf=U)#`Z6jhN!+bRAomx4J`G+xAVQsEdD9m^G+P
zE`W@Y_Bgzh>{=rlf(RJh%L@;-0o1)E1rKa9IWp{#@XtBkfi!h
z>W}Z>)7@0vNMt`|1nC$hxYMP6vjJiNpy~Ooc`K6X+qNvp!7wKpuRs5^->ExyflKk&
zdxCax%-r{M^B&@t%Vt^-X2fed&Kpf?naTCLxaETg8VMFiX9jOYZVZ6&h}|sR2Y{vD
zVw>yU5w>=)xh_+TF2hU(fN8vc)+)YAKM*7MQ80}mY$P1ysODp^U#~OHRy+ve0AVHF
zPwE(Ix}sxcJG3&(zz;II8qj+oZc&M-5s}mCOlfoXi<1hLteUiBC82;IrqHz-%VVSb)DGcIQlOs!is*yJ`a2yuO
z=~q8faxX+LA_Jq(ctxEoa2@ji-3Rum9TA_Z-a99U&uPIJe->k(MD>^|omJJU9mwivF<L^s+R!Ujtiud>g@f$l!I3RqL?7#kq?Pgk4>d9485fs$aUEh6Ky*!P;
zyOVf;O1N&ZEwhncGcP?k{pTn+dq@&)f6i?l%Q`
zz8nwA+m!W8k=jf&FC?5ApKJy9&@R4^T%ZBiC+!Cpu~1AI!%X4^mCw&qZ5I
zBY)0q+I{dZ+Nj@XI)d?<)rSxx`8mQOShhM<Ex&Q
zV29Zk8o2;}r$aiWNGF_vPp8`hsnuIw8EhBO%Zc^Y5$??msR!L9-?xbgXyafe!QA)Y
zw7+tcFhgwm^_wDt81YRP(o8oK$%CgWzMOnb-}a)78+E;d_U76M-codk@bPOGH1r2g@PWCJKiwlPeAK
z!)QbCOr$n|hVeK(`WuC-uT#XpeB~@H<$5VTeO0ixOF&A{O}{Tf6<&25IoHQ|l}7D<
zZ#2xgUW8pCJ<=?J^CFz1f(bcKD#o?ZBqN6u#%XemHnmguK|r%ihJzCl>m>P`xs#5<
zD9Oav3H^s%gXfFRF$cfzi>eLftb<5_4sWFm;muZ5kTYQ!e
zWiC(r-G&8hVM>~ULK0T0W&8)05oRTztGiW>bhuU~@1cI2`u5cB!zWFdXilMO(>%fT
zOvmKG5l0Qez*v3Tb;PqwK=sw=IPhLf^w_w63EvAr
z`b;k5N^{fmG>r+2sr9>xyW9dYECg<~Z&Y@ZWa^!-mC$t)^$Yjo(mho~Glkt}Gz-z6
z!XH5&>|x;a_Iu9wrQ-R8r@BXK;(S7UB`bfp8g;#<;g)@G%`1TCGF@Pj=-H1+FhT^q
z+)t3n;*QA`SUD%5rbx&JCEK1L2rQ
zNZL#e$xBAlWHoy69`sn9U&{B)Y9MHz@we8L18}R}9+gqPFY;VRTIooK%N1jT_2vNX
zw$}Ww6N7WPgvvX?SLI$q;JKB&m3r>5%9>K1c-;!3$eB2aG<^hXyG;7uroogpffF9O
z%~yfxS|!SxTflGgt~T(0fo)kYMJ?*96DL?@2ilmHND~U*Y#8N~RMdS6)CuA91CKfM
zqKCIEpVu@9y$bXipI1P!SbI)2G7
zFUaWZ;!ZLRlka8ogs4|v2(7Rl=xnF97&hOLbhrGw?m&Q6RSRnA-2Jm_RS#YEoF
z;Y6LcdympBqow5EV-1JBWC&+y9q&%i2OlLAkDrhI%|{&Q;;b=eHWS^8Se8f2dq-0K
zNe${
z@7JHgq0zC4Ia~_QwG)MwX)fG<|B1z=J1W|@X;PlvLqd|Yn5l_Or%g&aV%p*MI=4hs
z7}*+lI+8cjdn3q*vdb6uBf1&SS3C1lR*bI<_o4UtcVsYs7sG{L#qpRXyq>7z*A)R)
zi$*(FCsoA*uLPb~KUQ=)x#pr?M4|Sv4yRw+JwwM6y(Tr3i*gT?p{yLxjufa5eP9Y|
zo`_}{d}gM1mkngsc`tYB$;+{p5^5n^`XK`%wv11rNL{tH{QN3U?}ma~MIqfGc^Mx@
z28`MHbCeW+Wg*J+=bvGC8~dY3U?(dc`cmcdy(W0Owk(4H1)JV?-DWNX=3WSsxtfW(
zNAFBSwFFCB*fg^a%Mam3624A)CrL6!&|i<(+tG;&F@N*Rgi5u;ynkWO|H!L$wV|@C
zIleV*mF-#=rH4Sgnq}=~`M3a91LbhV<7no01}BAoTeiWeF19XLl7T53iL5_DBm{#Z
zx=R7i@6R)uPnxV|KD+z_nyw!yB~S?Wgs6rU}Jmtvv!CTHXk;e|2Tg3Y`YjMsOC
zlR|=vTHladTO%T()llAZc@b`g|Frw2fB%Kw=i~solBL>cm5(#J`Jtnx!>ad?XrZVt
z#g*(!g&Of9lDz!YuS;a6eR7@mYv$v-Qe0~**}@MALBdPvMB-N==}QB|ry^b%>8wsHLtfX@S5&hZ{r%6kz3b*seTM94
z;{vIdA4fHs8ITbxmXSTO3!nO1Dy%ktSe4Um6jh+|Q6ziNe~~7+(94umVVoMOp1~zjs8@DX;o+tGbTF4`T
zI%w$;ucoS&E>aE<{(8y5tNXEh&%h6Ylg|v&yyaQ+S`(VNbZ}4l^8Nyvk1A!Bx&l@g
zQY2hm4U60?b0dOg(RZ{gPhq76U7heGOxcC)_c2=|kOAsrA8t|{jJU2V>Vf%x!gGKAttN@C
zsaqL=h=}|fyt<~j9R_)#!*x*}FGO_DyP)ZwinIp%l=iQf
zNulQ-pFzuu!Rl;CfW2@b_7wse}@TD-JpwP}p1Qv~AU
zeB2n=Rodoycuh(wX5cDvllMWd6nNC;Ziq4Dye3hNYMgyFSv8@mWkDdPKWLgkq@`M;g
zZAaVJjMaE`I)iuUx`;|BCaM{X^v5azJ;?n*Ei2;*_Gr;CY*9UgC*+)3aBX4-PkVkAF<$W9vaCM~_%$qkmUB2UVY
z28#OuQ5{m;{mzO1Z2R>!b9_^8PPi=q9weO_LIgt1&-p>!g+*Cmp~j4%sVyV>W`Vq?
zDFTM{2`O8x+e(rFVu8c!z8=_jjx>i2QrUW9ga_}J)*#=1{L&WB@Y^n6xdl56arZyX
zD>=?+rBbR#^My?(SM>?
zyu8YEHaGK(upJds7GwtDxqJV?#@y*!1XRtU?avr%l31&GpW8u8s|2ydGvEZcQF5R7
z`;7#&Y;IwHx7*(I<=B&5dyUs?(OMNELIPW_kP#CR1?~_Xlb$PJIs@q!01lV~yZ5`=
zuJFvkt%^gO1v)}T1Zei5fVCDe*hDpo`Z$XRtylb{*AGph@#f}2`RgCJN&EYgEquZh
z6H_}CHt$0<)BQP{0%oD(Mtj=~15
z6LWSpOR88!-yE;Fagv;H!D!}YA5L7F?TNvw@{2$X9KgD&hzS+K#*s-ka?>uZ(A$?V
zw)$b2#gEZvMJZWjHdZ&HRi}8(pBg4j@+i%DsMznZ`h|Ev=E2iqm?WKezEA8mbk!iT
za;uGhAV|pi4E##N#u2dCPc&f3-frQdl>!YRHV^RP6RT=Z-YUa-!@jRG?Kg90V&R;t
z3w&R!8Mm>SZe;kACY^awxjHI?qfY{C
zIo=sr2O>v~f^xc~#}LWPsT=@ZK%&1o4_&RXq;70X@f+Qn+O;ldGi$@J#GM)B48gD3
zuVP0g8@BI#2f*e{_^8eu6k=uPGbcy{@>L5pE|FLJ3BkTbU$jrI>>
zMuqNPe+4MBW5r2-MR{Vway$0s9y~q}-X*RX2>53t#|f<*S^`^FaCf&US=dIfB1dxg
z5HL`hV7~75A98cDI5TD}1;RXC3oWt^%V^F+^5O+_SE9Q2y>K3XB`d-$1K2?HlY;!r
zqIA+{P%%>qcB~m!%N|m~mjga4wfa24I8c>jfAH{>isBGBzQYNJ>;Z9Yw1Krw4ZaoW
zKcP}#2Z(31pd|bxL0aD;$=Cs^EFjm;{bA~|wuyYXip~hd^|}=sx~o@{{ks3^3qLE#(Lk4bPVxDBZ=wPh@IPs%GTM3B&U1hGjv*CUW<$S_Wtmzk*oxDjW;c&Qc
ze}WT7Co_9O1((!{&%JzF=DJ%MK+9I4VGX8M%(
zyF&!K#IVWRe|Xj}9g@bmx=2CX(-J3_kJY>fbjW5pivKo@EfQH*l@apT4GQAs!
zKX%6Z#83LHQjj=
z5f^K@t1%gcMP6m|1yQ4zgWJt-{wi}0^VNr@ibt8!xebmsrC8)(5upHHn<@qGu_d&c
zwZ3M!C3l<4`>lG^Wb%TYCMvT|q>zM1e8l~A=k^XBABTt^*StD<(%WRGZ~%2!e~gDu
zReU*DERn~Ys`9jyBvbi->Dz~v7osnC3yc;$N6oh$w9CL(dDDasJszfNcv
zLzo^eHcpz|XR^8(KP(->>J;${8+2^Ow`$hQwqs^70w>Rj&%&e
zW9VY1#C4I1k_g_@MK{hZkxm%-f6e$RRKxCqSEr*|d17V+=hSl71pmb@WnAe_2Wa%^
z?p-4DScp>bfQ(M|Gp8T^W!%a~`?C|0{!d?&a!eafn1b9m9_W_4=c|GV_m~)a=v!A0
zIVZo@BYk1&T-`K8&2N*RX)5l=Jv9hG(=A$YdC(=89kB7LH|u^05hJo?)`6X=?jLVwpw?YA=TS@@PaB#g*H;?7Yoe%^*{W=rJe}f%S^TeucgtGat
zdGaVF1HktePzvga$iBciWALaYWTVB-fSMuU@`4qkn0upo8bJPmIgFV+j1I>+7Vdh8
z!jz$Y&}|g;C9vH@fR%SMTN$s^Z)TMhKN>J&jUpn85S3+n0y*yBry2AfSdf_S{dMR8
ze59>U={u^2DZWT>fBVh#i>(>S&#BcY(FAZVPs8A+e}ZHGZXg^9=}Lou
z*pQawfY-4Sr~#vCRCe|uS@Tb2==_x2yd0$==hf~R&QPAEe_Sh*j@a5Uy6(PnpZYRX
zP>F?ZRI#@y?9Y-}p8Dt_)S-91T=es7D!9~^!dQEPL-{*9vN^fX5M{71hX`^NW5&6xNOsfxD+(d!8P#Wf
zu;xs5P~UIZqUotBIx!8EWFhMS)#>^nX;F%db#TSie=~eeq$I`dp9;>VjHJc>3k|L?
z&HvL32S8-1Q%WlJ62bDkTkfABfE5?nE6zZI!9k2>$qh>DKf(Nx|9b$=5B#&h?ew;g
z5|pO;B_HpIUFq+{%1!@Wpq#)OeF+u*bk0~ucJ~zDAo7%?$${~EZ7q}m0(%cmGjclI
zTrFy*f8YL@8NKIL7ggmvr_68GFbaafz;4_zk>Ak4129
z(>O}{Tno)=)e*OJPb7L
zQF_|@J|C!>Qka%9gJ{~m#m@~`gE4g)h=CTT3kL2XFNfdJ$7)wg8J`fSUmYc_8UgG~
zF@c>e$Fp4W+3@s|rc?w(3qhLrox9_-f0heh&-@
zSfznkeZua`Q>jr1o)!6-S+7e^lh=>K#Q8xqZ>f_FAS^#p(@96*YEy65R7>#4h}OA6
z&QJBP66=evZPgzhu8NYwX%Lgce*xyvx-*m}khV?PV#6CEg|d{cczQu%?BLs+j%*mwizxe`S`Ju2^pP
z;QQb6Xic7&(bH5zsAFg|Z_!!`lF2NsV&Lz9{z#Un`}q_gn&Bwh4eOvs;QA$M87rbd
z(Y#-k^)4ery4ycIr@s}sbj(bJxhh+QlD-9CCdkI@BvND?@_8U_^(~Y;!iBI0(Gy*V
zHAMpo_qLa&7VvthU&2@if4RfKcxEDI_4VV$cH~=;MaE%4a-)in#7OtfS+}eCgnRu+g8aC
zAoiAN&(u}Stu&j7g8PKCCDs=;mHT~k;F7@=IGEya%bi;;Qc$0){2z_y;12$4>zB+W+r1J9{;f7z+8^0e}lW^Tlhm11|{##R7MG}l0@i$7$fuI5jFX>OkQg~pV|@tY)*<--hv
zu8*Gna7D1ZY2eEsj9PHK8+{N-hT1a!u)|=%*mOKle``MfScgw}HJWXC9)JMb5Lt7U
z<_o{9a+Fo?n4>e&4NTA&g~G2Np?uZcY&hHC1Fy=|Ml(`}=}EHM5QGR}rmKv9NOM?b
z*YRp_Lq-HP{ty2H9O4Nu!gK0c)o5i&GDGfQe8uR;7@m008CES^4*4`UI%%5rWE1Rh
zpMX(zf6}DVvcoJ%Pf*>2z(KX*k_^1eqt#sl1`~
z7EDvOW(fl#GhC$j8Q$>55IU-mX%B-;AbZFr4eczKwKq7lvh7yu<`=&4d!6+26Z3E}
z(rR`yJv%pRO#6^}i3{IXEEuK>ouXG!0HZzve}~C`kpdPKLVI;`=m0C$6g3foqPt|Ug&orT$+L>Su;
zf1Noc$&W9|cy#tfp8Uaf4xDUw<_y3UBS78x7_YJ}QuxJx8DDN?Bop-~vRE3RSpF37
zCgbZo{7XOTnr{||3(%Jc3-W8RBT=*Wjdk1H<6#LDaEPpLPSU*Tr?Y4y=JDhVf3MzL
zjN7GTOU|Jt?RPqiKwlwY_Us$#`_J8+e~U9-0$Le_j4Gnv#@OOy*L4|(8ROR&i}xu-
zo>iis(}A+U%eJbh8l1DKa4MY2IaNf65e`UDpX$OUFvU|KKVeliy=I}`pmzy|H%BfLp|zbA(+hHv{e!mPEZ6Qwe4V4%armoF3r$%Rc*DEpDYI1C6&D%2
z>~sgBNQV#LF97)QX)fRd9|mZQlyQ465(K;HqbmZ%Y5qsISBE~=92duwf1Pw@3`L&J
z6h6<){hve7g-7GGpoellZqyXZvjr=aV;#(i4#{N?P!Z60cb3r7^F?OpI^oS^~Maozf>TR$_2f!k*c^~=z1ACib#ZN!NC45Lyg8`a%T2$=v>3vvfUk^O|>sptAE5f9gucgfj<^O~nB^
z9w-3r5ih&D;mA|nCEgdHYH)E6rtBiXMyy=2yv5uA2W9@oPjdgkYTwD`yY@BT*vde9
zLCc~AtP4;Ts&c*S`3t*!>MHp|{j8VhdzF3s2nY@p4Q*^lWLfzmN9nz!y#4)-_C$e}
z-VC}$EiG&#`;7Kcf2VnjSkE$gdQFNV+_kS68N~DMF-;E%rQ9UcAKetJe@-(?j$4;)
zT)NTa^c__hZT9;wF7flQN)3M(
zgX*h*F*kGJuIhzzWbfPE_lE@VUkje5PGgd_8=K*@h8giHqK${q!I2WiY54eG-0v!h
ztZr(KagR|>e=g#Gf9$eRz?!aJu~G1U~23;c?I15hCI)7=l^@HS@7M0SJ=iFMMz5udWQ-NJQ^
z7PfOQ09ls$B(M{C2GENqjJ#`*(6o^yP4-{Ydl3qATcUNJ^oN77czO%fi)QL^vX)_I
zG=k_af0k(_pMD)6f?DCU5`ql73RvT`a
zvKMhY7}Jx5rGJxysv}nyuY|9^U4V@;0bfGn8=fhI(hPFG~S6!~3e|{cStuTz)e5N|f8TuNe%OA>=WFTm&
zWJPCXxBMlwqh^CTz$A6Q0(~w|5ge_yj)zHK}0iS4gW38ihPZ7Ja{^By!86`uQ
zf0<`|N?MnO4V@MhSbb=uta@?;Mlm8o6*O=IaYmX+slZ)ttGB4eMBZG-Rr6cdpEwcv
zd-sUZ!Ou!zWw`FVW)I_sxvRg(ENG(G7NgwOXSffs2yL7nA0*NMQVTehlYqAH(MuOU
z?^ACBBo5Bss4tLtedG2W_Qjw}Gtq6ff3qB2)s36sd}Q&lOi!_*uQhxa{Z`N`#fAbMP1ks4Z#?V@6o*KEUfA~?=
z#3;vNS(*T6`kdT%6PmykjWvp_}hdq%8{t5#Y1Kwj$RD#p*5HPPz*NR*}k2P({P$s31iND?dz
z3d_~~t&~W%7Qv#d1dIgi6CbS`fBjDT;Qb`(&wu0Il+94WY58aK?F1+nXlrr)K9>Dc
zd-zCb-0h7o7;T|N!#8`UdAC4!I+Yu`lhBUBG${}#>#POMIoDIIk{Vhhf2Im`hc(B>
zHaj)SiN>3b+7;jNVo_GQz+d?CMDtwbDAlscx(58jAZNzf@=uT(
zjFk79+b9lP!-4q;<0%9#I*XAtcG*@oPw80WeoWkRC?H_MGuDu+LQ$CbA
zccVV^R8)Hk5)!e?e~>d%fp(QEB=N3IJMFrb%+RM2<1`0EZ0fmUA=k6?G%v_-_9fu7
z%aRS?Q_72Qt8(VPXKNCTGI>u}Gl_(IWziJM{}0b+SXyh#;w`6;FcKepQR=`*B~Gai$!`ODQqc;e}6c0_)G88`L@o<>$wEM
zJpq@f9th#ZJO`VL>Z9QK2MB7SJI!yevp-%R=}|9-uHZo&mM7AW7+$0SyA;$V=5CLP
zfv$uF9luog6_<@n=XHN*T7hy*Z#X(wqTp_$oAXemuPS%k=dl
zG=|elKP&&Vf167;{-CN%1W9nHQ-n!0*hBhnAmW?WaLS1zLl~94u4ClZc=tp7Ly?vd
zuuoO;-XbeZG5_q1_2~6bG2|Lgld42VO+71d&6Z`TDCExdk8o~{45~&fqzLFbcfyakbwtUl?SGFN2
zlez*siGsn{;rV_?hr2Q0)$66l6maC*?kS-5CW0!cs@%|MM4qdzoK7giVQN^(-VEB1
zagv2ef7(-3YfOzLKHKGkq<`&8iy218%I-JVxFeN1qDYf{U!pe+W_O|TXJRFyDVxIP
zPpvOe(jkw5K4Eh9Th!F?%RJQ9u2rSPit=j61Df{b<~1gVjW@m-lFqE(6|z4VTRuhb
zEUzB~hrq0F3t8ra%=Vsh}1NMKsGRIA%VUs0&LV@43>38SJp3PhMp*x?2lobnva}~#oIKUuu
znZjwCBPej~?o5Or%s`iTG{%qDDf3^Vg&b+Ae<{>g6#uw9fE~HkRlzbC$F>@EWnwvz
zf3D#*L?>0cczWX6xu~Opqi(Bftdad9vn%`=mlVTdl@i9KaUHJ;KM7dVnt0{1;9DTo
zLbv8cpQ6r;9Eui^g`V)X^oycV;0Ok%Jp*}7odZ4d|vn141e+1NvQ^ze=;xx
zH}~}TH;bwcb!5}8vdO0ZT
zdl}s3c(f-Pt~rauPCL0_@Q?)D6eUy687{m_A*g@XuP-f#?fGFmEWT7Sf2kq9>eyg+
zt=;RIC%lpvdl8f%xb=YGaEX13e?)8u&yE#H&8YX1@kQOn&}PDa0c~HrJ4oIGXvs`L
z3|>@xL;no;4MxR61d+<|c}y7L#(^vzEsDy3C~RY$xQl!?WLnLb5PxX@lE$4SdZ4q&
zRKgmzKW^_UQRa?zXG6S%ank9N^(j{q^SSF~D0tG>wTR|A|qiO>C)~R#A2X|
z<0U<(_Lq(*)cxVqYw@X_+XTpU=}H9JRMk&w1lqZ=ET|un2*rW?$#(Ab{jiX6b_o<0AGt%{N8;W
z$RZb3T1Q}C$Bkh5jhd-GqS@O6=&u}z|3L!|ct%l@D)}pSY_Fcc9?9&`T3XK*TYWi)
zFP+{gb0~butr7WF`DOjcjcI*wnse=4#<(Ilvb$HpK^ap#s<
zFsgGC!6B@cBuUFEYf73J&L$AE1hj_rgX!R!_8GeT7^{$QXDTmNq8Gh-4@uj~@*%-Y
zV1FF2t8<3u*8^R|x=m5+->?B}3F5i>QmF;@t{}}Bh;3^tLdfg622iwRH_`;%vA1z=
zd?eHi$4w&8fA6Q9ax@%@P@apmQ1Q{^hXk~`P%ym1rA<-a$Cc11qt`EN|Js-1N}!iP
zE+A`jPh1ti?|lv3TlW0drXbVEb4NZ2`7kZ3v?JSHgf)mQ`F{`}sm4(W@Wd?eGg&uF>6reCwJcLK9l!(ojIsJAab(o?4Ke?#
zaEYvi<+of}6Kie)*g6pQe$Lbj$xd`-!`!p$pJkOJ*J;7X`3|WV^v^#cW!m_eoAM%@
zM2=K8e`vHd^XKeg`|UX~mI5pvfjR-~`qu6=(0_Pr9rzf2$Mnu5)kzt>cx>ny
z(=jlHrh57eVK{rm6~q?vZSm|9SQLOv4G@G`f1qcCz#;J*Y9p9t_#@a@bw6KCD*;rO
z*(;r7XiiY0B420p?x$rUHP}}fkC&ge&Z{Mce~L$a1GLGU$a%OqW`lu}O*U0cnD2Ze
zwL|r8#_%!|Cf-f)o*#ME?jFTtPo=7_C9#or_ArTf_}stvd4Oll$P3vH&G^$=4vG%>
zEgY}P%o!ZHLLK4Vd&O{>RJLqWS(9MRGbm4?XPdud+^e4p6Cr6{
ze^aZIjwaw2Ev8-7vX%WTE=%;TPZoxL@&R;{Iv8_Utv{*914Zk!1VV(MP87CKRT(ro
zh|X_If7DV5LQwY4f~XW0mG
zDjm^5IiH0JE&_vw50apNaUlcqu@SqT6{{mQ;%=JC49lSx9PzL$^Ke4A3l4WLe>1rf
zv5K#tJKDcgc?kyW=)>9mGrxbyOpnK9)9Pd8?iK1QWTaoZ2bERFto%u-`zci8=Fsi2l7g)Yd^fUfl{5G^2p*5fG$3Mb|(aE7ZFsUy}$U0$U1fzl2j2gH4%`HQ~
z;cVV|t1P>?sC)@IjHGu3bn;j5e*m{58|Z6I5eY~Fe!|u6;_eZEzn%T%t%VxiXbfwHu%7i?YTBc+w40`LK~rwbc>qmJYTnag0ZxI+`#Q?~c5s*0f3*~a*n1rb
z{4j;Ivqm$mtVGB%dGqqOmhD8C#w>A=lky+50|fjokcm3IQmYM;x=$42ckmVJbqMCO
zKquAfRv*vRr{J}lbWtvmRq)eEA0vU)eG&Fzf{IL`cN;-L(#>=F)@ltO$fa0x2ffWj
zfvDv`@>RI-V((kgc-Kh7e-r*}i$rdPsSN6Nf33A`u(&sYK5y`#zsIc-+_>ek^%jrp
z=;qP}9)vni7*@Iaw!P3TZw|Agb9p}oR(X`H*EjQc+vQ8)#ntW}-xZyl+H`>m6^|_g
zRul3($RH(X8=9no&G=A#HN~OK%d_8ZI8GiTst1YjD~f1Z&5eWne_VbKMz&egB%RSL
zjhBCd%ZX9Qi;{@6J0%}2B?BQ~yswC@{{k5l6emnhcsMv6@usBji983Dj$b$>O2amc
zCtb{PzQ8wM^4U3B-RE`O&GZjhV%{d}iux1Cy;^w3d9;O@M7}&yEFj|5BoFPoLxiCb^j+&6ZfS*5Jde`!g5$mX`ShbQUDWM??b
z!sXs2aEmx-)nI6WIm-RuGQ(QZNpD^L2nnDSOXk$0E8VDY3FoojU@?67ja9oDTGZP{
zMgYE1v+(zdlR&4HU?LO?UV&)LZa9AGhBhp@z7o^AM;(?H(e!h`IYvB%*4Pcw`Z-sN
z?!DsL%b)k!e|YlcB@vifr_1OfWOiIcc3_*H$;Hz@bC9X97mpOHvl`_7sIoWGpA@n<
zlV@v`(S|QlsNcqda2aa^f4}=@;d2^^VWc7!ygQJs`VSNjd7h37Xk|s&z5YQ{Azg(t
ziy>S;0_9*r!d+gV%yb;X@EEz@VV}VZzrD{BP95F=e=J(G>A6SmW-=7ej>>tySoPJ*
zY~4bK({3*GmEl#>RS)^mu*N74&Q6!p-tpP*MNOmbBP
z_!gtiipcMf#=
z5K=%4Y9)lMeO6U?IVal}rQ`DFvdTm{of?VEP%iJafLdx@u4A6AgjGmNb|cXo9V(xQ
zgA{vysAXqu+BZ%``s_04izBjlhE|_7%91tAf5;*&>tj_AO6gAYxj+i7K){xnB**Zm
zBgwI3!0S=G3ijdZa@Ff0D#_G3O@P+1rCB86CcT5-LW~(-@{9|8y^7iYJG*>)OqVlV
zaa9hH3=Ub6ksY)v<~JODU{lOM@w@K*;SMtSW~dW%fJ4$8c7z);<(HP@Tu$-+}}hzxZ5(%BV(gB-KqhT*|kFLFpQ4N
zDHpoe$E~e$B?{TS#O^9*Oxml2QwUF`T;>
zv4J93b59)hNdBboT5Do#F|5kg-WLvfe{lkLkb?IZxeeG`W0I!;H=~YMULlQkT$pk}
zJKzDuQnr{7Tc#*YodySPr?L=nXRbJISS)z^O4gV2aS>D_i!_M!;#~aVsiJJTf^5Tw
zpNSr3%l+_aMYDB6C&rj_*iVJ3-!$&*SSOrWcUAIQ83Lq&6zl6fymI;lPKi>Pe|TT^
zr-|mzqcaLyPfvu`!%MsOefbWlooKkTNTo0LZLNPeDF7Pr=uK(6K}=!Nrno_lR6vHH
zP(o%LRZ;HAVEm2PH3uWKb5fbEbM1Zb>#@o+fdZM+-~U~!eWDqR9%uxeF2e!P^l@;L
z>R#~l@OH5I#FXC_MPjs%+*V;xe`41oiiy0|;xxaJT449UC&8R7Zf1^u_|n&#+!d&S
zMP<#^cN!}BH`yv*0NX;;DG
z)9z*SK`15RTDDEf-2UF7&hUhbI-G*?0~{}f7IYgLaN>QJ525bo;{lB3f7^Q0isKcl
zIkfSF?|$}3Z86jSr!iqd&=8Bdfhs@Ulm0RtstRiQ1cB7EW0GQ1VzZy2{&cD|tn1zb
z(v2w0ABxM_tn1>h>p0!8CTtgou#w;_1O(PRewctgDITMV^V#EHUM+R*LzfpLfZSZ_
zcY{8+G-bTOI2J)RihadUf1WlV9Ej5!+8%oi`=y%vA}Y^(dLG6G>qjy4O-LJKXPSOR
zzNia^o2&dmX*Pkk+^m0TiMMKzQ%oh!da`7e4ty~{Z)#^?zeFFgS56nZ*u5h@;O^Tw
z*Lx8IuQST)8r$OMr?*QM#r*kc5E*q5DCr>mU=FH3QYUuQF*VT*e+CF6dkFldn;@`Y*#6fs)-7VY`2m10xc!o
zqj7aN$bUvy^FnELzSZv}M{Z;qJ1A!Vnpj;P@~EV9nvNe|SoC2E@(`ooO8{pyEYZU{
z{ed0GXFj3-K3RuWUi7MU`}f?)$QreR0!9yI-74LT^3}&AVTWBPpnhau9)HsSfuh62
zpj-$WnJUQw2?03VT1vJ{HU)r7n;%$9OPC@cFUJc0xunk@U@2=)5GhfyVMHBvoa#(B
zgIw@b%w{Q8V2%}bEC3(g<-p}>a=^=94_Zv=IcJm6dJ_RTmtmy=77;l)3NK7$ZfA68
zG9WTGFgZ1sVNL}U1Tr@?HJ4z<1uTD@Thca+ySoHv+}$m>26y)+c;oKw?h@QBSkT}S
zg1fsr!3p+~GtbO9Gwb{NEmqT9rPr>iT{qB_q)MucB4!{Hpd`r7nUR%=g%2PnuL80)
zwqs>u6b0Fs0oa&WSU3iq8L2w-Dj;YEL-1V{nx
zfR67!_o4yh4Y^n
z?2L?mBK`3eWdg_=n_7e1oUAPY#&%`^StfZVfC9+vU1$lQ0oegefELC!<^Yg6Kn_=VNAeb8}<*?Be9i1akb$WN-5qKQ#+WCx9Es(Hijn?g)Rh0sd7O7dx}}
zaynZ8{}td*O8|0~ra(I<;2$SR(7%+n@1=ZqdY3!@huC`|od3kM`IkAs2?+eJG#18A
zfBDKODaiqBjV8<|6AqwpEUob
zE(&@-nVyZ8kFneTcFcd+&c(^|AGP`KW1E8PoGhK3o&FUO2r##_0sdj{^rvT*c7Ms_
zMHFNtB~;ZI<=zL+j!_=;UL89oXLskntpCIl5triw@UUSiwX%<6x3mJTjJ8S($KeHS78
zE&B{~2Cx8t4gjFLsRi?&RDVs%A2I76@p}n;yzD{t0CQs-C!mj|Iq>}r!OO|m6$o&4
zbOHK!{ZsMZ2!WLwU}kCR{66IG7X{(3=rVTZAOP>b#P3Z0t@^+A)BJTwY2RYxUY5u#Q|7&GzYiZ;0-**3Fq6z#%mFEA$
z+0seU(j91~Wa(^b@vo}>OD^MV{66#|cAss4?;ZI|r2gj~*}Mn|1Xy$1io_r36c*aDbE#NRiAGW~!5jLTm}5_YB_GfTVA05(o8
zfU%>au?GUn`wXygass?q-$&UD=>FG`0GOHVK+f+j0DBi_AAmW?5#diGxj5gW{}KHQ
zaRHb`|3=&ZX0g8!4}e+xKZu(Jz%2PUVg)cu{f*cG%rbu?4gj<4-{?Jq+~4RugZ$s<
zJ%hr35YK;m2Bp6d8-Q8mZ}gr)^>6f^LG5q!ej@e1(R&8X{~+FXWc)XJ&u8*C;sr39
z{s(cs>r6p5@4frCgoERc#Mbt2e^!=vie~?S?^OW)1HMx>{~NNs$2Yh9hx+{tpZ@{h
z!(05F?EMfH9`+VMyMI``%l_&AeirM0z<0tn|A2q**=+xYtnWYovHob
zx!;M|+q^I3|8)Lc@V<7O{u<@Ko!$vM{sX?VbNUB-@3Zqi;5%8Df53OLuK$4VWZnKz
zsduvO|A6mgJ^q6KeLa-^JQx02dn|w7m;ZZU{)JVY9YNMWO-r-)Ul#shB5&;MXz8xY
z^1gqwSl{LEU;q9||GyGY{=IJh;Vmi(a`$57c%NfNHXdF8E9d(Rv$C>t`}~)$>A#*`
zf9=cni~Vo>=fMF00^Nb82ut%IQ~qG9v~Qv1z7mC#70{HtOs7>iAGKr?pq4TxI|%W_
z^R~%=LPY^B{#jH-AUSD1eczM-yAmzRU~GSz*Y(!q>WLdOW#JuTUwL0b6bX@$Tn#4m
zz%2Pi|B6mB+9TQA0_{?c>8!SdHe!JK$gvo2N7rKv+xQ#$RVtZYMcZmVv|I5kYcrOk
z4Tk$78o_tM#mVpB&To%canZ)TB1?2LMg^fbBeF2|JzZ$pMAzgAIn!9&q>->E
zd~QBcd()eNAM}7QynLnjMTxb6dJ%tm3aCw@^1ciKnFdN*J~r>o788QC9gfUPuG&@8gCs&NH*u(Y(ia2j)AiDc$dM0>
z{7}Ha-U7%_t+S?%d_YjdoF9LQGMQ76cV@0|i`Q>@7umpARhzEx=BzBEqYOmL`P6+9
z6ZjcxMWQf?A%Q18Rx}-^kj}Z^3@4TH7ZlGuoYz%ocf_%Pg?aVOCoC`5nXA;;G^ii7
zxq;-XSYdqA6yqf3Ikp>>^99z++YQ+3<-`(<%QG;bfMZE51~xe$W~zTIoHRs%0Vx{%
zp@JjxoAHq40OAa7OQf#W69cbf0dJuuMjq%_@!b#WLx&ymp#7;-3n7N>lpiQUDH7@I
z6^wEvFnaV_pXA2lR>kIFX+k;Yh?s@s*91@&xXzj+_Go{ie@XIGP;_g8=ZP(R$<2pN
zYbscK!u>={vSzPyNyUHjl{M;^?dvQ~y;lm8nRS_;+pYX6u#
zTL;xJ!O>NwMTQ&6%7DQJak&Bb(+R4gDs$jyRzU+hX6tgffZl(^jeDJv`=CkD07q;G
zO!CnCIwn)1+{wkO$7TQ;3X+v_x+L;L+D~{rjfQ#Zpj24@2Am{q3(McPmULym&BdM<
zZVeohV;Bc?o|SN(7^G`+Lh#9x1_PUUPsDyM_Pq$J*3W+lJfd?l(U*AwACj{WJ@Ael
zeRlI=HoY9UbmLgjSher;SxO7>dZ4ZEc*X$j8jB#ZvyWB%AQx0&!`f7;SpOpUvB=G9
zYm16#!|BaKhm~oQ$Wu70+TlyNVGiM~vaeM=tJTf!K~6^OnUvkJ{745D6sRcCM{h}A
zJ`tD#&1HWpEnNvcv%ONj(|*lLLPQR{>dVdQZdK6ej~O9by!>fn%D*40?RFvTn}W8!
z-|Bj8%U2_J%>_8mNm9^i48ho<9V9DMD?1V6Q%Qf@k@(};ZD!9WQvFLq5x63o`Q%TX
zX~auH9^MegK``#D&MML*%`vYp^*US<)w~|lHx;;p6h9kuLsA87$m^pD8guL+qn
zg%-_Im%_osh1e!6b$cKSt4hM28gA$we
zN&cH_%B;DQ^6>$^$b1~CcU;F!VTUG?LesMOjOF<^8acxJ@F6@0;q9J*)-_F>WBNQG
zF_9S-!bzpWI~8MM#2&AKSN8tg-j&zjbKieNmP02@=Kf$m#)FlNowpk+Nv-I%{gnr-Q^>Vq7hu*)mr5^i-Zg_p}Pv9Ouo3f{eA~2CT=n-;TId
z_KQoCl;Xt_ci|{!9K4x9>R6+#9z55Z7`5^iJ;H}Rni5D?
z?j)N>gFYP9R6l8;gPwT$!XJ&r+QNUJxksZu8Zy3*TN28ssg!Pev|*+MY0Rdc0*DgT
zPEFnj{m$UVKSmH-3*LLhjs~F!f-KY8eZR_BMaw+ug~M{poHMZt18eW%1G&Nke*#^oN*C`Mzrq`Cal|Z*^_^FW(&4F
zdOf&Mu615xMS5mPT0%I>0DBrFElq?})1Zo6@4X}!6;;chR_a+7DA2z?ln>ESM?>in
zQ97Q&tz%J4iz2IoyLl)!tao;P5rM9lQT&BpNf9)f$5F0xS^vo&7p)8Mj|&dQ)GW!NuZX=x8%<0%#h<(&orV@CN|UhM1h54hd{JH9X9UL
z;7g&DM8dBe&5vJ|Z~OR;&gz7yDTLv^%o
zy*+%C&~L9p_1x|kD>1^nRKZjp%R>O(
zCa1va>_C2nfZ&L0{`{rySe+b$F9spo6hhQuH{N)CqHn@SwPmBl13
zmkT_$uY3Z05#A9ute8YqlBu=B(pt+NA8I-#Wh@x#D@PVQwt
zjS_B;y@^X?a1_$f=Y4)=9Lojx0A0T(=JRzpH}&uyq#B*r0&_qxkBwKxhK3n`!Yo!DVLuVcvJ(M1PXl2(~ir
zhoY^sANqodr;oyv*q%F`Gj{a@^!c&3tn2Vi^!7;$AvYXs4E+vd%&+%~SnZc8q2^fCphnq^o8ow?I>Uc0MyUj0lU|e&FD6@KS$w9^
z#RLLfX@zP!+_Ok*Zz#~a0D)KlN?o)1@N9SFetq{fOi|4MZ3PDvSkw6zKqmRsCkLT;
z2)TZhSzTOyyb*6PCmFtK%8_;T>XpOCI9-^Hy!?qzK#}W54&7hY}8P
z2O~MrD$Y|28B_RXaU$y0o-gSVf*F_>5>(Xw+|Pew;n=1jJdJ_5?kAV<
zt89O0G3Bd0A{xeIRT?r9k-RSyZ_uQJ0dZ*Vkjagv=&W5(CY%hzLE>Sm-54(13{9SWYGkrWB$EkcV=q&U|`+
zWvc*pgx4$h7l>5d7Mf^Gd7_^zhEwwpRM}2H>R@NH()TvIYf&7=-y8mr9lFlU%0mrv
z%)Ve*Qki3xclWqJHNvn+069R$zbI3Fa54%r1lOX>Asg{{8`eUGUID}{o
zxC;KpjO&=TPW0xHphTCSQF@91c=aJ-nS+5BXR;@X<}tKqhEtkso^w~+asOqhS*s=1l|9$Kr;0t8_(-)PXV$Jf@
z$>T3chj};)W%)*v`#Gz;%tt>j*6Y8CA9YNi+qZ;E4)&@Z5!C-KuBcBxVNu=IPgRyj
zlg~1cNz`}Cn=R@1b^SxY+*kyUIh2ck6$-72NHo@##RE8_Z^7B5B_rl^gFyl`IG1nm
zBipocB{05Yq9ud3iB%qD5z;aqUACj?X?!`(z-lLMtXAK!@RIYxkOABnhfe-V
z<|Kt~g68<)GVY)*zooG&8ddbuI6I@q#t=oYgkXz*ACG2~)uoWI2W;c`ITGrB;tsS{
z?|=wTP-ZaN;(?ydyB`fFBjxtbsj#-U4
zA&;a~D>^GgViF<9AJb7o&AFOif1+%p^Y{#J=zk5S6EqTvFPXk&YI*z!AO8$>vrzo>
z)6IG-jfOLjwtcz?Fv@tv)_XsHu%LGlYWW&5lkiiEKa=u4u8-{L(;R%Fec;zVkC@_;
z7?h&gFq(KxYoeo&vOXcqfm;t#i-L5MlXi#{R=
z!zy&eE7+jhLh3!dv?1hZ?45lVjj?DMq#w1gPC`~LwG*Y^punHMFJ-aGC)~b^Fo%St
zTN&eedq}tYT7;;#!jY{qzk7l?tK;&K*+k_lVUPHxjSF
z*E;0NR&e1Tlh7WtdLOso%l`N@K%wHBC_SGjjaG{
zRU@iGah*t)tp#}1}tn;kREOMhMB1dxS?8B}jSF?3a=O@#FI0Q6~qyuUC%
zztjIL?y`IUUK9*wnk;*OvW{z;R5ZVFHEqhw9hx>NQn9oRv8qgUrlR=?66@v05E{*<
zac`5wu@@_^L+R^(4W2qU8fu}`#VFab?pn))=g222VG(d@0&cmtS=9jF-^N@yN)~j9
zgn4=))($x$c>QBCSpZe}hV;NL0Va2bN=s;!lDZ1yTQsY~fN?(OL4VRSYy+*|Xao~&
zc)f-ggcXK4iIn|eSp^MKAYiRPd>I6!uUfj&#?G1TDJzD5WOh7$Kq~Li7-a1DgXo1J
zmaXO_jgQGxp;XeA(d#+&
z#1)<5jr%&itWA&#;s{sLNonzL9~h;KHaZh?Ps>jd29#YSB6ft)J@&uBh;q~-OQ33c
z@z}5y(*{$2C8>WCS%@rmjoyWuTD3zp@DYRdU7V-L7=eMKuYVw>VW%Gwr;XBiin$p(x|j@aRr*k|o}s2BBk{^
z^(QmeLDgZQ34AHnN)RNxC|xwENTWfF2HV8dskk?PET7Ykr{FROzTLfgF2CSOZyYU+b)e`;%tNX=JR2
zlo82)zmr)))+ZEma>A@ichu`^VhOhHS4+JX&4sC0_ZY@U7p{bxKn|lmL%NNnStcs(
z!-Vk9m-50gCC>_yzFRH{+PU+jcg<*BZi}q`?NTBoNmueA*FxBj1d}
z%E=u)N(BFsgw&&pJ{?4|KR4}_4xyZ+k7jm3aIchk6
zBHQdX+~@V}G+X!C;^^2f2m8JyUKcGxl?V)x+#4WHuB5(S^LJ2l=
zEgBOph+{!pPo4&urp3IpZo2Ox*D?uzgy&h9T0#gsf7T!0Hv9pX5|`2x
zm#1zM54*24!5tnEt|nli$WaCVeRW-??MNs5VCJ>0?6lMkP}px5jsa7o{24oS*o{=9m%CB;&
z0ck01jT`acBM;K_g~-LY`WFPWz`&}nAl12}N~fPqe<
zjRVYI*4>dj?VI)!L#Q5(kA;UFWK)V(-0#QK&a9M*Iie%FJoX4y4ri&&hIH#B5x{iQ
zQ1^DFzCB3w$d$|Fi!3jyR>xekNcz5Ofbv0ufc$XZ*bTU>-hlFOlTthh#ayJWR|S$szq#>|w8>AkYCTqxag8^3M(R=5
zq%g}!P7+Pc>k%o<=$&<^oCcN&etQ+cHI9QV5oWPkG!tK4tDaVUbuOTozVvOiAE5l!
z6&vuR6+&R>H=FE0xPsB!+L9>vHIWxZHqUN`vg4(0rB?~7B7J6mlG-hBvXi=P3v*I`
z-U5#W{)mTD7WEdhU&?n)_p9Tf>&ikJcIAOZ#1Un25({hY$!Xj-QOqHP5rLNTSFB-I
zbo!iqBpTF!Epl~QikmK7MGh)X<43VGbYb^B1pjaA(E7d6ghHmC%+?KHi|0}cUjPop
z4edf4hzmBW&nlpQEfj-ok<~so2i8#C1LI(V+g($bqtf0%g4YjzbHPxfcUw+#pcp}c
zA81?BsEvonh!0Ew*@-G@LCE8-
zOse3p<{0f$OtTylPSb}O+*L!$c8QSYxq8pr&#`sX%{wq
zQp2Pdg{em-gjni{0RjuM@OrHjiBd4*tF2VFE*$CCCU$lt_!olHqp2$9KTgxbO>0as
zUeH!tC0O;?p<5AoEMel{qeBB}9G)*Pw&@TrcA~U@qRr%PZy6PF35G2*vqP+BA{$aw
zbP#e-BcGbi$O+A8?uRm$4LR*vC$^`e3{d2kQl>WTakU%Mqx#gYGfIz9t4lMq@Syk|
zu-Y=Jb5z67s3j09NCsVS$8;@^BR;QMzIK&x(WoG#H3o((hm9|mt00^Ea>>*{9riz>
zH%PXBoN5*=Q!OI1p4O+bK_OEPd@f0*Jr*SP8@=3N2ba91?$Pb_>s}j-q}n3#{6dR(
zMM|)|fzCOwrMpeVJ7FlIrS}EFzdk?zz6*1^1v>pJ12bZdL6F1%2~g-2YnGBMbFd!6|>%Z>_)(&5BK4jE?-srl`H
z*Pv-LCFw#SX1W3ju5Y%J(UE`-d~uS{Iv`+FPTFXsXO)8*1}OpWKpoo#iV#%v2&|xc
z(y+9&h6B(;N~;Vyo%RKiry&L9_ghArmkon~3t*jjmFeWM*TovoK6^x*yT8>pc+HM{
z9Q1o2_B9)p^$l(`#;>;DW>@;ppRJUCVp2bPX_kZph~nWAE%qnGpq-P2gWQx#(+LaA
z4w`xU;UXz-m2chgGTB0Q$8(3VJKVt!Qa}pIfzGnuU>JgpXmczCEaJrlEXicV5T6Bu
z*9fl^{U_7(?wpd~6%W8|K-%yz!B8bs!X&BJSvY4ULvD6zC4rDBXvc;o-wTL;P~)?8
zLhAPs4ORG}nA|aTD33!TDSNHj`+M~f^b-uUDXe(tm~MnIG3?SLAi*8rNRXmFM7hVw
zd`NN^gC;ho8iBrMXm38kNTa!ikNSqDtsMEpEe8Xzry1)E{}B(yYjomKz2VnOWj3M(jOu9rWzQ2@7p+wXZ!L&1pTr@G^u<;uuXPEl1~w}z2^^{p4(Om3|r
z;qGz)t4RL9jk$yNb$usEyKjRckoHh<@18&LwPl#d5ZV=!Y{n{8TeH;9t#))QaH2P2
z7O0}Z&$X;yRY8XSQuG7wYheJrgM2(?fSBd0Ltnhvw~@5C(i&y1)Lg88?7GlxH|p)a
zOqvI@la?IV>};&T7O2HVrM%Zq5AfmN$Oj6GD5_o|P{a3uq+sXO=;0tExd3&hbV(>_2tg*^7b$V
zCMi@alI3svv-FbsXoJ8wa>as8m(~|{h#Bq*VyV1n9pO5-D)a31IC9HuF$wrXSI#_Z
zZ|CL=;@k+hud#uF#Co^i_+U~;ey?Cuq)u4}1R^va)tMw_hSkIVM6$Tv_qW2QTZH5VIxi4hoc|h}-GhC#q0y%hH`%m_-yq3j
z-s3mUku12F0cF`v+Ml4MGbWXNYd(FB0j`&w5^*P(rawg=sc$hxMpU1mz1VT7X+b{)
zr9|VuO4aa!J?syEeo9-`dZfr=o=9TaSOL%qKlq?phGDiM4E+wFjwBVGm|Q0MAdF$e
zGe$-@s@~@cFr`#IfA!Py4uN##&9LI=L`2p7ov{>lDCK6!n9O
z%jv4)5P4ugKT%RxL4w1;$BdG}sNj=$<)7zv;Lt5a;(Xi!G25!-EHJoHnbv&{TNz=S
zMJC#^PndL%`V118U8gVh2|uVe^*>rcKames-yu-pqqJ>#CQI1sezTUYDob?44fP2Cjt4d)S&Q?xZDeAz(eaY9KBe}^jqDp}x66}1Fw^WDJU2b^
zRp4Y}A+Q4y@%)md^-_bqnltmxz^NW@L|j{URWKB*!-KKXZ`*}z&Z;jJ
zw?;8dVN+*k1<7NyF@&^>EOH?l;|LNq`g8WBS8%yJyf(0VOTf->lLaTy>imvtOii)J
zbEN8j@&L59>)+pK5N5i%yRxoLnS_#}nb
zqUb#0Wf*LRo!xbS_V^f5%hCWt*9~-GbORj!xzypNaB#Tp(&)K8(@=Rpz
z&TU;t*FKXjQUi}#+e_P0xN2iz`B@4+q>5}IjF<9PzZ_>8cL=3Mm~KCa*i$)U`Jlu1
zL{3kFc)3AmeLr?BF#a;}ZFCvktAs+`=vZ-AL7>Ckr+&T8EWj0~%?m&6b5(S#Qmhew
zgp$#WEyEr#e5zZ0%y9l=qn`c2b18)UAqYtG!Ipdu{i=s(L#V>kY>@58d{ZhDn>+4I
z8w!1EOb42XrwQIvFpf-;FQ=>ilGsN~yL#Mb9;%nQm6}`|?yA
z%L+b2dL6wF?K<`JTn
z3;4ltBu$fEU$p1JszR2@)?tXz(*}LW2ZUYVT^2_YD!jkiVqbbLK)i6iGt8BLXJwQt4Uq)9NRfM7>3Fv_wytkqF)9MB-b9(GpLM=a
z*hu>c%+`bdm(X(rr!SYo)zmM)g{=Nax2TpPpN=wOL~T~unn4ILb4Kg3n5sM}9iX;p
zhqwlwjbxTyRZg*^Pe+VyRW7Fm*+V{7>0G&VB~%c~rCZrTcs)f=Fng+hqD0JpJ~W)F
z6E$V7o|^BQDc39du<5XL|ESGnU@kg5kH*Hj>B}Q;x3QGa4c^aP&#L5ii#^~j0eioU
zb=*x)duNF!aiJWKV`;two`S3Ku<@^A!L^PW
zmT$Dysd5_m?bDdq-%K)pedAG8XIS$dSXyF*Z0w2CioJzVVHlZS2B$Dw=|iYTc6jhr
zzs85+7Q`^xjLf*&p-;9U(JYg&FN`r(DX6TvZ_4T_mlzPwE&cvvLQhVA-x)Go`8~w?=vnxV
zt>f*DpiUcHKbf&y4)mugQ|>6q(ZM7T_%+xWv>?L|Nk5C_*MN=Qxq$@znZEJz8
zQJQ{;BZE%Jt1o;j)`ZO$Q`Ibg-RbAgyUXYgIQhr!`Oy4cSOOg3J5FyW-;#-2Z$64Hkbf<-=P+x%
zmv8g34ljrmNV7BRRfwXUw$NM%6tOi=tqPUcLS6HpBZY6Ag!yQeaik8Uy4|}1=+BFs
z^A>%#iS30`=9JwJ3TRvli`bgcm)*32Mu$C?PY87d(rCYbb()*mie!{9O-Hi)NMh`t
zWM0DhlG_)|s(3S9C}gT<6rN19_jAgglBU>uR%^p<>EZX?*HTo;Fb~%r+Mm%Xgi?dc
zIbEn>dyk`DoA}hUPH&cRP4N&n<(X>;73TS
zECrf}6ERWrxCxkjSwzL@DhaVSIreK+XXi1ppT01~`1=WO3-uq4_`AdiG=~Fq@1C9O
zP4FPOC`R
znVqjhUhYjqW|}*JIgdZ=DwIAW)4Y9*oY8yGSZ!6-D5UO2y~Y$kFDCfxFTlSf%F^w8
zXC4y~>y^6*ysJ7n2b0btuXRJWwm*6NYHRc{1|Ei3+Ui~>nsB^)8cj1Lx|Fj<03L@a
zQgLv9OiH97oNGio=P{D1XKItRl7mb?v5egrHWm^zahd*YT6;GsDWp2Gp)p$VMNrhA
zYttDQ)r37G=FGw#;4uJcqbI^4Hv~r<6w>82zuG%=n@r%?A{6%knsJK*u{Ph`wLz-$k@maQ*wiEQQ(Ru*&
zM}{As&->!@+HA$wpO##YZj+Sjo5K(eZrv1SaT%W3Q4rU|?J<)aE-ThP`7BH5)#MQA
zWcNi7B*}+U)^(pUo0z5NUG}LzDkK6EC6pB#gA!*pByhO4vWCY?Q))>s*tGV@W3V58
zzF+|(^2~R-b1tHj%`Gn&XlyGrcBq~Jxzyj=?X26r?jlfQZTk@ymmr^g9hyJm5HcJ5
zIjA3qcVIA9N=t{z+^%Rxd!}qN&vxf=NcUiV$12zvD>_>T@graR14mAT*Uy5umzNdF
zRyU5MvvP`-wCz1nJnJj<`Upj#T?w&&TwtoV@o$1|ldH_ida=U?lr&ttL
zo5mU~8;=V9j0wFKWsK_hnmkzK;N(Bc*N{V>7$H>YC%|-|k9l$`s7b5rtlz+}k2&0R
zO@8B;&Z{uG8CK76DEMQkemKr%y=gluzAfs?up^hfvZ|>Gtg=Qh^mJ}uY(&23CbE*W
zVDu-bam_lo$(mQO&LX*u+zJ?fc2?IJn-VX9p;bLw$aHW(PTWD-((XYok&I!igled&
zGETSdtL1g1hT9>f;pBMy@G1+gH;Z%x*U7L|i~8LwqM~kdKRca*Vk)da`?eK^@^BjX^W7
zEuY~0t|K6x`Udi(v=x1y&Zf8gE@t&{(Y%rC=1ZtYe6iKbvF`naH1PTyOzkVemuov1
zNAk2#pWEUrsf>^X(BlP=Fjicp$^%2L~
zrs~@a&0LiZEK<)29|cp%8nOC8G3G@hWkcTM8M`Zr$`Ez=5cPaSGS?WlIV(CbG
z3Leig2X*xH`O!gbigp>Y)R5~a%(D=ha@`^*WH6dLrIV|ctI;oA#nl|Z=AtcW_!Qr+
z*|vd(ACI`O;Uom+aL4}~|^5
z{S}%_DmhSHKVaN{xEoR9Dh}d_1bv}DnDdZnU|e_IlOYpB25c=3m(7EK?Mxc*D5!@M
z?UfyF#QIwK#;Bfg^ieqVs}n0Pnx*3UY{OZn;!)qmZv%<$7**$l+67*m5|s>xkr5ai
zBbM{WpbK`2G-b@L)eo}Lsh0_<`n46Whm}tIAvjGAGD7Wt2Ln_$j!q!@EZE61AkbQ-
zVAnQyktwvD#kdz+6xF4CGGeG>eE?d!h55%v&6fjv!JJo?XCd?%m7pQX=!`Nqe64P{W!kOa#4dbDc1r?CtMI0i
zos+zwcmgikfJq&lA{b%FfJLSJi{O^mV^IUr1c5ex)=esJ`dL#;J+e^*bS7xv)K(%`
za-LEOTHeaM$q*-!>u;R7w6I)P`ulNSdAwEEX&4p~@9X9HMb^oL?wB97WiNWdPh`Q&
z`w~}WK50m46yPcX)uK{&V@E(*NRde{L2lkfyaWWo{?WPI?IYZdVcv3F9!K<^Dx!(b
zIk(P#;h(<-S3YwcW?=H{jnp_Z47F~!E;TDiMzQSU3ox7U>Jvw%p}$9e
z`kv!c-D@P_RF2f@raHdG#DDKQov@Eyh5pimnQvwe!b`=
znVSFQmkX&vLj^lC2EVfzf0($AN#(Na3JD*J>{YRL;=p;?Ydy_m{Dc)RWz=9VPkzfP
z?k?>Rm(1mYIOrOy1dCY}$oe^>5(
z+?$2hfX*-_QeIy8&;PzoVJYfq^;1*)h4C^3NR%4N02H@l<
z3&%&RH1u$q1Wtc#Q
z{))rfwQ)X^7_c=C9b$74eRjyBD$Wqu0QZZOi5G$-QmNUgCUPn1Q3Bfvx7ubE>
ziq2u(hl}g9j27|e%8Vs`kCl#*E6|s_y^_XN=(nUsspp4D{Ouj2Frsh;bz0JLDMOvI
z278=m-?H@N+534{^zp;|aeFL?O5y+bD|Spv|7Afr<=w*!j54
zX>sacUtO99En>%rVNVlhAKlZoJuwp{{V7bU<2)JGgs|IrQ-~&XV5BN5C$`QA`n>Kr
z*mVS0ck_x*sw8}W@TJouaGw^ELWh4`HpL^=9E7dk&>5q+(1mHn!9@&Mz3!sT_+`f->)>Sbs3;zZRyO#e5r1A^(-
zUBBvWvZ*AL1POQz4ibN?v?d?DQ?V8yhaVZ7Ns2MVd~_#&RFlHyV>BaaueFJI=p!F>
zKp<2Ddh8tyL$xY}%B;t}*;zFdi_j}1M5%=)S%MFSuM4!{
zK^o%d3>dKof+Br61Yt+cuVmzS6mx-6>voGvBy1;VRE~wshE+T10h!y44kA<#@{^dO
z``e9c4;Z@TgHV-s^4ykP#CU8>Z_CWyidxKd*#8Bxl1hE4p@ezzd(77VcksM
zZWv-V=!Dc0AL>|P%q0D+7AaX&5=Zw+J5f6(bFWf=kkLJ3`jL>8jD7eqX)Ncc?FXhS
zY)I~K0y2~yVpf+$k4%*&>Jubs!cLS%O05wBvW%T*!}X5=QU0qUey7H)8|Yv*WsQr6
zko4OEAZydBu;hs+mxf}CfMO&iqp-pAiCPb>sd`abRi
z&3j3IPvk)wLFAYHAn;&TmIB6n8a%2w(zWw4LwLzeH`zzlA_u9@`3jO$@qYTyh}#nO
z-CDOOc}QA}t(Ca20Qeb>_@i{RsURje$mQ0N8a#N*)=Qi4hT?o3kUKiOo7Dc+U8(4|
z<>!G%m%ivRZWl!OkrFS3-1ukZM5+b)l_J7_hUSv*)gg09ZlcWPaGG5Y)#m20LC~#upJpQd2m}@-K9LHg)N)81Oe48{R^XAa3*Z7-+hSO&pQQw
zasZY*_OzIXYa|H1tTY(DypQe9KYJY`UB|vw8}a)(cSJWq3zP+)NO57Djd(zMqyaia
zCHB}Ls@)y#Xj^gZ%>yKz5)p@yb~q@45luuTwo-sCZ?
zdJxcTdaSIDEuCxBc+lhpraJaxFlelQNII`2rdj?ew3z~*E~4m}KoLX{Z3M~7FK&N1
zfpOaEF{bkAl`VSA#9w;2Vp1CUvJgQ4q+>V90FN+z{t{yNJ#Y
z`uEOMIa42ak2FoKY0M^3xlC42kWfk?{E^UR{3&
zqR24Grn#84f}Je*+aP{g<9URq)A^cjpkyLY=00&r^2v8ad#XDp@4Qhby=PS<+%_dp
z!a9E`Ceb;0Qkpn@NFck{f7(nIX)rN9EPVY5IWa9wX9vr
zG9f(i-FyD2k>tLA109i>yAF=3!Y>Bi_{|ll`Vlk>P
zwg|Kq8xDG@RmJIt*r|RxIa{qui;tN*ZMe=h7<##f?vGE$Y0FSgZmRRa#T+oQokbB0
z;_mLpX%ZxMB8WIYQKxf8T?^p{>LBWrPMa|7`27S%V$219YS&4AmQ&GJBv@YyL}Ij&
zco47l)AqDJ&xZ$Hi7YYCDtrsQS0AKTn9oY^fsfBa=URU89Y`Ai&*&E(Mu^7TaZgqo
zYQFC|Y!pRmciRNMIwWK>1}p7`;ND*=I(!k*u#M5{H<(4Vk9VC(e*BE;NX}!VN0Puw
z?fdMHtjYaK1-irbS%v5&31{{6$Z-VnODR*VU%6NMy8X1d)X(0nUJFE
zuIGR@c;pN}#mNdN{|gZZEkkiub)D_xs%Fi9DxY)B^}s<9*SNnB;mB_d;;GuM@VEW0
zu5((z`l3T_9>L~q-6=ELXuA^Z5^gmDH&D!leQ3&KJOyD0M4Xp86%{gEd?lIZVG|=|53c^MC*exl6_7HqWN-mNIc58Zn
zuBv=#cL6L!PzibN+$VMYTe1aXU(qNp`~2U`+kIG{O2N9ILNp#Gh!0*BW+1?&&r!+g
z;yfwu?c8Z8P;8Gv!L`pWf|%kDejJRFaoM{Ft(3TfH^=R`O6>ER9&x*GJZbd0wH-?Qt0*KFSYl3E$Bk57UI`0R6ydw_zY??<)9-pMVgZ2Jj8|SoL34kck*tTtS
zY}-~R9ox3i(TPrM+qP}nw%zfamwCGL6@|5{v?gFZiTH(@P#7VWoFgK>QekvvynfU=
zj)-t5aE}E|c0fOY^G8Mm&fHCJAI)b^y_9F^X`!8yy?SI}c}r=U6R+>P6x0)c$1O_s
zBOV1Jfot1TJPO4H!I8Bco$qVQ%K0xuyH4t=sLN#A0nNi*4#e^1NVKI!BCB}>_b*JB
zJFBh>_o6}NHzm9)9E;@s(Q?DFj>NSk48^(K@>2o24@0F$9eL#J2N8J<3{AArXhGj4
z5hvnneXEk((;$;+yy3b>LntqQZi(BVOb0&r)tSKVgX(c_SMhAZ-MpnX@;{>r>e=QB
z>e^_(B)v!AF?EM`1tyc5GlpT6ztxp|h3Kn;23P_gidBE0K>%HUE+=6(T?{Cu
z1`K0o_MdU{_VeVhy`60&+6IgNMj9Bx7n9=)=E|>OwHosIr=k*ne1<=Ly}4>2$@gNG&uuVl=B#
zk^&<^H#w!qBLJI~L)42j`0%B6INFp_Q9O1ay*^tMlKav#N-My>sNVt|PoveJzit98
z=-_Fdn`jBUM`L^4zHxTlcS7MSII0}SU&;hV5yUW$>9Vpd#|m18)kpHYAjPo_A
zK4$PUksbE8uGlyNuBQMTkvB2*skFn$2LNrq?4+i1F`S=cDFyo8VgzdcSzw|O&i;YM
zGD@J;yA%F@^Y^4qEz;_m3HGOoUXKmBP(bjS)J|eqh2!0~tJeR>DS7uhl9vrYkk)^P
znGuDHQu01v8x_|H3O@ki)T+6*-XQ@%zOj63q|^S+*lFQpyB{D@7pY@pv%9Tw`9?r<
z*1_m*(o5jVs=5%HaT#juL8z&%^zDEUW!S&ToLP8(yT)lNFbaS17y(WA{0zg4WDtX}
zGzG1zIKh4Wr(9$2DF|WhVEO!tEKZ~|zZ4$tEEQs`E}No5RcCHCsDq
zv}ApclJ)39wiPNOpZ92^94vJ%JpZ%^uv`eA_+VK#bB+HCqi
zjZ1?BIV0T-77#JN(vaf`2b7RXG3_y6-r$md7RA+fEFG;I^+Sov{;N}Hp8b9+&6)#K
z(s)%bJCFqWkLuP}1HwsQ+s$u!e&xHR9B*wPh4a*HWA~8tEt}Knz-SpxMA(V#w}*P^
z?GN@b^2Y*9O5&|G#sl%fzzGO62DqCg;`$ExA=an{&V$Va;UkTNe>SauG6?XrcC{D|7U5oc1=
zRAq}AX3AlYrd3%|V1G&lU~DaeMIv|AHy;`8my
zEBJaHpx0Oy1cm!;yt@w3;zI
zDnMT)SYELBwTSM-2fOeTASe{O2QIWez_VJ1$tIu<;KSU3j5
zbeXFnwdUA|40SZ$l)$>ya6a+qoj<3A{J0VVO2DFLlNiCZ>$U_Yp^XI^M(gPk!Wmo@
z203&;`+^-}VU{9sH8P;^)xqq#pv3a^R_ZKkaEc?I>Y?UAEN7S*qGOzY=t-ZpYiEQ;
z0^vjl$Mg6@$K*s>*T0~9Dv#Jni)xodDjj1r_3`b`Y#gcYXiu^en8K5){*|E>=>jv1
z^b@7Zs#1|ZvvbotT#KVhJwAIS2Y`ASF}FCjLPcB?lCAtG6viDL4Rf|L`hD=1cV`(3
zcFq*rdW&Sd8zZLbhh+?Z7mqGi^d>qdgA8Qs#)!af5%bk3?t$opTXEt^Zq-qcTB19h
zjq#a)*8X8(eh&yqAD2Vh^m!7+nZ5S4&=&f*G@rL?w?JpDbsafsr0}MO41HwGx$_Vk
zMfjwjm%H!UU};~h!`Uo&_?5O?+{)R7BcFavq3EmJnk>@hGP1}c+;
z7=OKFwu3cbZBt;n@vjA6Q!!A>ahWuY2#l#Kzx;==EWe=@Lmd?%XF
zNPl85q_Qx!&#+shoDf6e?d6agGfppy(f7D>lJZIp%13N}X4U>nJP%IYIPLknxiqum
zog}OsuMYlni*V|_)%fdb5OUEihjxlYdg+aB?9yQ!rev>yL9q>(1`8_8=u!s-;{&oIsoreJ@^|8@JAyE6Y|>{C_{^{~BY
z3OxA5ywHSyGjUL+;TYu%sqckqzdPcHfwFkD>;Eqi<#exdOfu%#D3Ux{xy^;-&zV}~
zt;3wa1(i9bYqSxmvpKvY{{cSW#cr9!#ZLBOD@B(m;9PY;d^Kv)8wjY%znM76N8P;b
zQkAz+HU$`RdRr^Ne1S)ODd%TcKl@?O)>u#=cLleViK`c{H=6W{$VgQwsG98^?`tsW#2#Kaj#09jtLEWk=|N&4JZNRx
zSGG3-M8dzL;j`U!1hHb!LjM7OJV)ocBo10=4{B+%sJ50&py}@a77hNO=|J!XOXvGP*8xqQw5SvKYX#X$d87AYzWC;Ri9G
zo#{4=hN2D!YbHRBB|610Y$j(idHaceDu$uH@j(F|n<0{fQnkA0M>=fed9f-g_d(VY
zRFZl3@G>nZt5_AsN9H6sC!^c^!pz|F64{l1f!;w$L8j!sH6-v6RZY^
zk`NlI?v3V=xTWw7uKlN1tI5Yk>d`1kH2F+J51RhqF-DT`iD`n$lR{@Y7Ca3Jbf(o6-jbrnPmR)O`jdtp#Ij=Z>
zTD0&FI4FXb2*KFSCMtg4+=01h4)`0q0}#!ifT_Fv}vC0`4_*>inHCTu)j^6#mDq)`S-nER0P18Z{7%df+{In
z=)`ghYmm!Zp(aGS#vH%C@hC@s^t8otJt4i~6HmtMP8VzaSxcjx4y{olM#55%P{<{+
zt+|b2q3}3)(SQ3?=@qdsOspjKWYBH1<-;}b2~e$+dbujtsq%Yehq^Buf#_qOO&mIg
z*omDk)r=n!FhPY^!xJ^6$k%>rXWKzlJ+ASZjGsSkLdaqCu1NZ()W0TwbO|FZ#n8b~
zQ4&*9uLT)P)Fg;QbuxHO`>P&l0Bzy-;KV$Ix{&c8?fX82uCBe5wNI4o!s_*crorL)
zfMiaXWG4g^@&=8yJ>6do<7#_uIxE`vk{>!aV}vqyj-P~k3drh?KjG?F(grCq$
zrOr}n?>gw9bQ!wm-DRw%Ydyb@-{@J#V$Me#)Rs*O>OOBASn0La77Nxr$a;O-~uLgVg%_#Zd0}`7wlAjC+c~C{``MX4TAI&j;Re})jR(>sHHI<&@
zfHMMU-laRu!Asn~he)Hmdq}A_O>#n$YJssxk`y4k^miv6gvX2R{Ig#Ya?serQ*Ir3
z^aqs_+wP=vT-uuU5ZB@tdSqzw@_8m%GL?O+(z^o>bNWgXjB%l-9e4#`Bc5A+5<)?o
z+_%MM6T3XWapWI=IKMqv4sHH`EqsEx07~9{UDDk8%1Dn_l}
zNV4x(6js_-%?;q&ZB;ih(6GPTS=s=HSXjXO>&+pEPAJh|TLbMX7{me*m3D16l|
z2?8qE$=eSU?~)Za#t6DJdX;~>D*vxhTlW_ABL^V*yCRT(u2gg3n|b;t;PGC?3Q^Il
z@R!@^Q{@(&+9JV#1TbKa2pGmh(0XM5&@s=}F
zw{2acpFvqxY5IY^TN2U_*2~;kf@iyD+P?FGn*5r;
zDn#ArC&kNuC-NdK-NaZcTGKmv6VkFZX|;v}H7_vvs2TY}E$_npVLAEz=p8Lp+-rMc
zsL}VVdi*>7SgzCk8~y%XEi0+(kCyAy*-v4xD%1h`-?NG65x0UYC+P=qcC^`#v{%bZ
zu{YjOQIR|LTh6WRO6QU~S1=+HuR3ds33PIg;Bwf{BV7h
z@;{^fv+eQ37eyHXo-p?r>55*uLc39MJ$}K$^d}WV?pZ1p?yO8kru41qQZW
z=_<^rgTgrGfIEL_XU_APiKHUWgcN;OxjsbM)2?Tc$Uc%PG5
zt$U}JtV1-o}o;M1Wrgn`Z*eJeilJuu{%G3
z?H%EPn)M&)GlV%ffwgn~ENWojPc=SemrFW)ynET_gDawWhQQ|Mn=aN3ksIV}Tui*y
zT$>wGe@ZM@Kdd^e8y$u>NI;TnqaI6ptU}(@{=knOP}KuuO9dVru=>dKb|s~v_}l63
znE_1Mk%p=d%SP$9_YtlKii+ETdVqUmFyWWT|jluZ)1TA1I4jq_NZR3U$HF@OI7)(A<9&&q6@IjJ5R5i8`%
zxpOUHzkNn-J5_3D#=K{=u6eb@xJ&T<9-^delhDT7QG=W;@CSfOF)4dO0B^G7xw$_7
ze+F@BrCq;Rm};&vJ#ZQss`fMMi|A?m(NPG&x3cV^JE+a|yn2Es48Jw6U){6A?WW4V
z`-9AD9-?4Z-2Nw*5DZMyF_6~JznHTtQtixUPT4?%IC=Zf=ua+@5xbf!i|z|Y2k|Ct
zlk#LLxIo-C$We&9fCIiQ_v)4!{mhiLf2-@~w(pqGY^Yr?5E|mrOuxlCh%!qaMXWF^
zA>sI|-=!sjs{Rjjr3gKa;)ew_&xd2Na(s{0Z<
z#4duv7$)_%*cmBj{hgN|XF@i%S1KJyd@hkX68)Tx!b#-;25kDfqPZqQTs+~sxC>cr
z5_fRe{?#}UQpb4jZ{XBYk++>8e~qlA-!IiFtr4o+b$?%ES2lf6dH0YN=xWKTp*}S1
zO9C7k?<-hSr7sf_ev4SM&p#-MFgStIV$Df-Xp%0m=qWIt`(H(FpW}YDt9Hj8c2SRo
zUGUgOeu&7|^yFO*4PCB07@TYNQIjj{RgEUV*yDgFQ(@}#&aLQKJI>_|e@mxH4}ad6
z!EI@bf}dA{kE-hmkKjgbd4;GPNBE}YP%Ql}1o;t>{l1Kq#kefx
zElw$(x3T0fuq;S08P>?sfA?qRT}?r063p*{!>n{bK)2~x*&~@yzUougi>_uT0DfbN
ziNAOhCBok-W$kEoEJikp0IQ%%u2GovVXL>?T{xWRC8P#;kry1w-35r(eFk+Hi$kil
z7PsdJaFj$MXqLE8Jc(MDRSuwp*t!I(uGh|o#K9~9Vbi~Td7AH8f3@8>+>FQ9?(zt+
zBo5OXK!BSqi10%C`T0T(5fv}weh>1KoFm4go%(TpC8qN;LtOhVo*wgJUNMSHhc?B%
z=T{&Rr-ARu;k$XYC)a}!Y$GI*i)y+Bz@8K{5G0FuBIJ_VY_yy{3hOS(^Hqu9_XFd<
z?(NiCkzMKT7TFxaf76ZbkG8$#|2;ifoobAnxwz6S{1YCI+m9|sSW+(^1}bh6$3qBd
zr%|nyx^8WW8$vIYZ*lf)gH68L6ge~q%beEaY2BAt|`MxadM
zLhf*660=j69geg~%{z^~uQ|X50)i)^wd=z-maB|(AXaWNaGP@C-*`eL0UA|DC>$rk
zp@|%cAv);MaTfnv=ZyC>vpz=%Xc2*wqU#cV#RL#NctDpR?md3C-Ix}N<$scP8i
zZ$;T@rGH@ZG%t8`w_Har+DDBw{F_HjcWC;(b*m_2e6!YRSt%zuB6tftG=bo@7>NI^
z&`}}L>GB`6v(qs5CMzMNq1HyVgN}={_EO$1aO+o$e`ibOKmRuR_!+lg2(;!;eGms2
z$Hk4-1J!3`u)EaF^gDbj%G7<0i;+QqePATKYv1j)3wBP8D;wkRVw54W8U-bSGE4*E
zBU}rfV6O<|bBSKu8bObo3RT;GPPStfbq+83dU9Hy3jHlcA&T}pTEv5yhm7#q-2G_5
z)9Soof6Em*PGAUxHT0;vwj}sL!-HKr?je~Qs(Ao8RTEeJwikI0*UI`M$gYN9woSNN
zVuYAdLL5o9_RL_W4%476fU5Rt)vmc^aB<;{*RO}KC(wYT^|f4l2a3#zDt#Pf$n+I?
z5!4zTEOmX0zoTQgr3tD<`Cd@UsZni=ytV*uf91><%b%64s*T+QUgnpG&wGv_E3%i4
zEoM1J=L5k|mQD1BGPI6^(Os}(zM=)eKWxhEx#hsa)>vCA5_-&)IfSaGxZ7{PZM<)sfF1OVrqK4sv#zym`zz(#wWC}XeU
z1a9le#yfle0J}`k
z9|f>IX=;p@lQ*_CQHFxJ`o38c0Qr?SH%H$v(fBMXR
zr}`Q+ujktI0$ugE~uy7nG(Brz3JIOq>Um3U7mG^XocO
z?k6=?iC{t&fb&*P-zlw-IuuK_wNIPXGkjl=k~$ade4wFa!<@TMG`$-2e;Mh~`0|vBL2MSHbO<#Z$!miEfb8@H8kiGouJ@p3
zkUw7x^HFHM!}4#-pNDsPzQD+Gy&V}?CI6}POXSaDekYR7zkCZ3I@C7T=`}d
z;*P_iD15N-@HCk;dA))`S?W{g$AYlZtXMWwr9Xim#oQh$c!`f6$+=e
zc`XBsALuGTK?l826bq`e03#HHq=k~Y+!aV?U7@I^5KqFnr{)ij4q3j9)qDJ6AI
znH!qqDv2W(B+xk~pSTq?A!)RB_oXeF8rh+6wGRaa9whwV~7Zvv+<-OJxnz
zxY?N8CR&LSEoE|CY1VL&|Bl19QClgD$5V~@3uy~^@AhZ
zrQ59UusdJtBCK`8%YGRLhy#8#k7UAK)myp%zQwwrf-;O4wQO`g2
z6dL)I`pS0f!N7jQBN|t9!m`_6Cz$1_1~-Y&>bX%>`g=ED!){H4N62HxOyDfd(d)%T
zQ|2=EuUmZYe^VQS?%jNq%~N^O`89>`gS5I+?mpGqF!J4+XL!Osnrizh;bl+{0#tq7
zjs%|AJ|2Hwk!%a_G0xlUN$gx}F?}*b<9(7OEkO`s`Yy&}#zX}_gm|}+kx4P_f%c^i
z?P!PT9}idVl(iqQDtyoep{WiT^0gjz-q_L)(LJhmf54oRK2A}YR=%cqbknLC7@73B
z!ksQMH}5tr6~_uSJ{ASGh})ant7h+)r-TDZlf7hV*yOfJbF|y3MXK(JXO*%B5q$dJ
z;lbfUQ&|eukL!=Kg2>MZ6P0Q3otMfU+EAryB65*;357>+e{DU>uM
zZ5+E2f1H}*n)P(HV!oBD8yUt$mFmP#x0%d_40o0{g^179vByiyG|u;~B6_v%zuXRL
z;tUv2gx{@}ep;a@J{y8+TQ84EdJ|WAu8L*gWpds%e}z_6+Q}e8qjG;L+%D;U0E<5~
z$~Lg8Y57jB>M_DXm=lkCS6he9jlbDe4Ss(@e@OA1Da=oo;=1oU@x9>;!`K-Jj^XaG
zoQI$CAL=#V6V+vDkOlP3*93;ZfM3%E6F(4gj`e0cpSn=Qyd&x=5-ktD3?obhOA9~
zf5>GTtga!ku(2}Vpm8hs#1wQo
zzxqc{87;jSC%#%eO(N}~I)O{tA~8=qTTMoG3;2}az$XnCxj=3&fz|P`WaLOl+|b)E
zogA}iS#Vp7t_kTm{Gb)7m~PCzL?nxUfA$%rc=vGq@7(95u<-8*iT&M7k9-rzhPD)K
zP(PO)edHe}_R$PdODc-Z5gA`DUgPD8R?rA*g{xUk*+qe-&8`y
zf2}&Uzf0HD9B9cXrJ=`aFer@dw&-!QURi$o=uHXL;t1uZwtMGj-5H0VHg|^Me=$O9
zfr{;t@L~(oOg0ouHql4}C`@i($A|2`t?L5prg^~9?pQ5?%SC)d0Tqu6+bj#C$B6wc
zR3?wMzDY|(mJfAAEye#dvmiZI85ov$;8dC~k~A@mkApPepQ@&EhupD2CgY^|ZW-Z_
zzRX!`a^$-s6l9W0HMp*8+je6We=-BbGJ<3`(eNYD
zL7$E3JR{0`HZtd3cp${VBcTeiCkZWNt$K3TEBYmUCj7y#ZA989`&tQ2${9GK0GQq6^{&XLNPYFr&eJ4j`Ge{xfUpz^G?
zfs+LlTh+}Tar2Nyr7?XtOEF46)6j~lIuY~^(M%}=%v9*fqe+1x_CFr<^#iUP@>jdu
zQHRU%vQ)I-Jc+yD5_s?o
z4lYijtFDWhqXM5yvU_j7|5hIlZGQVk(a|*CLRH}W)^Gjca!}0m7kSr=%Kt{!bsGr=
zAhl%$C&A>rmzpdQ!sa&?6cP_zv_?~6iX>UpDlJf>V~72QNo(EDe>~XYi(#0D)@F@u
zEw4#vcKMqV7_=fjfPSTa&j|)Uznr^*({R*yfMr8)lJETQfsPUi_DiSE^ldv{B`G8D
z%FY&OWb%F8xE?C&W`(0nx!&z}S?!?8h7iCMjt9Cln$;l^#3H^Op00mx0YI?l9KFhS
zqXGl2)rsTMF$Vvdf3s?nUylZhUW1frh0Q2*g8kP>kHcWs(zXDRzIk8NYUfdT^-J0Xctz?VC|#u>N}%D4=x?gJ?Iaz
zVC@v2A2uQ&|K^65sI|&&Rzuc~2-BFtpId{yPl3CnrdKahf1_sxJH~drcE`!Mz_o;b
zIK7>ua2^s*-vlk}EZ7nw;9o=o5pk#P!FJ^2u^~*O&QDsnOC}FZGx^pc2Nq8DKpjA*
zcpi^t=A6;spEvz+KyI6)5??>OxW*-7Cm1@7*jwyV+F@ZUyxI9?j*Pz&XsEtCkblf9=gdV_mm(f715I8UEaM8cEaHtvivrdZD^jK1F~!MyV3SEtqluW;
zHF3-3F=k_KiWn#do_}aln)DTiv63FPOFgl;
zMp^2N>cCXrih3(_<&DbX3jZVul;${5RbvbSpc$+OVZ#%6^|-je*}6<{0$1(B2yR()
zoILb>*};g~yak0SU?MN+bFj^
zff7W!oInFzchHC%H4(KvA_o3^d5B@}{XWQW!jFoAQbx=uHW^w5w2RMY$Xlrhzw1yo
ze}knHZHT#WV=DG;7CezYLwtrB@}vd!_smLvN+x)HNmx?j!uu##dj(W1v%v!*!462(
z>&(sGjt>~9Kb_5tSd8x+N^=Ha*kN4+I|i^=xd_<`gJe-vKC?mxhe_mYN`b|1DK%tf
zmeIK?OUu-KrBTByM11x6AH#WbFgiN4fBWW8oV@r-(m>0o0*AS6`h?&DP1uUom4UqC
z!w>i{Jz4gNTOCgmaKX*Y8zt~%=(7p(>cS5fG9kv==~xu8HB%u6c6oYtprWT6
zc>lR!C~)XY=O+_dTpnB;iz6K_q}KQ#(QHXRyYT0xa0_p4Z?rK+;voMmmG{E`e~IED
z4yL#=99)!pdxslSZlrM%OBFy}Zs9s+rvRQN^;~9!F3Sxex^|hD{*SGwzL`i9YD%4p
z6&&<%zWDY9eE}PgLLJ!g7l5=x@cBeC-a0W7Q>v$P3|nNchZoHaSs}dcq#*2%_n!DUiX
zF1WFgF06-r4OLz!?r=f-iXIom
z*1G>}kdMpFyFBzx1ytCg4k{O4l=?4a_x}Sc=%}HS(Rve;s_+e$Eye{15-~Y6Gzu?F
zWo~D5Xfhx&GdDSxVNL}V12r-+laX^Mf3;P2SQB}-_OlxWK~O+Nh9FfEk`Pcpkrs*&
zq!$HcNG1i6Oh_h_1P}pTk&Ys+3L;iemt~QnNVTAX3ep4=LF`KGF1Vt15^Q+y-v7QQ
zPcoVFp7PE)@A-|cwud*)mJWwPjxZ0w5%B~|z>P`;5f&vHfp|hT%p($TUJz5ne*pym
z5ls
zL<9^#`HcrGL~x;?5QQNglg)z+(GYu>A0uEhS%@r$DGn#gB#XAg1M5IqI24|e#U^4ueXov*9YRE{F%mpyAO)X>Y$FYa4N;e)+f+|LNt**13&~K
zA%Uzj|BIU*$_)5H4Gm=oV6MCXV8B8UzNLvtR8$n6DH0-hSir>dIdXnJe=N2Th=K*-
z0D2Wb97x_65s!|x6JbFU31kBT+}Jd9Dv&J55uO0KXe-em)Q)^7hBg9`<>XAb140Oz
zRE7l#<*{xa9&P{^Wb@EjgFG7Qh=7Pl2!zOO=og~vPY{FvdyzmOt3mw+3BH&4M%@lZ
zpDc(Yi36j4ofyay3B^D4e`fMy(_mC;LPR)`5ds)&4kTkQl+BFIliR4a?k)>Vym|KtmBKno-l)|Ii!(1-PT8NRg&w-8h34sMMCjYMS
za2_1Rll(sp3^tF>kj;WF;+y#L*bySg#o=ohYQp@mF(CvX08j(~e?`++CNctY{m9Hj
znHgJQ1-e;c^N0xrn;1!D^x+0hW)gN@Kw6Rn=GyC5LCAGSOu2SUdrH~Gp|
zj)Q6&J)hXJ6Ar);$>zUcsBCHBJV+=6$P-}@+L&LMqtlS_2231$sdm2975^?Lxj%(R
zgXwG@6CjzH1E4?v#$X7jMo4C6fP{!H03C{!iwH2m^I!xGe*yR+Bo1J}0*q|XtIPnC
zPysp<2;o2sBovXs6UfR*2%6?QVmb*y_&*Ty?}%IzKc`zY8TvUNc{21?$j=sKrazO_rU`KBxpxZ)3?da#*zu-v)bbl=RPtkUEaI^$RA_F**
zg*iGyvKbjzxsnt&DU>#Gl*$(s70%Nv`}ECs
zbWfXMFrFH}M3G|Ka?lU&o4B8PJE5cwYxs2C!J@$9V!vmQK3y
zMfuGhY*5L$JBMaO9qlAmsR}p?qHilLu3d7wt@c02moe4MG_c9G*Qg_;X#1j;bu;;m
z^-3E*e=c<`*(2E(H#$}5Wd62WNjRR-z2aUe5U<1%zucweaN2k5Lcf~h%EQ#{0*$=%
z{3T<}rU}%daJRlUWb;>}&va(D3^O*|D+w&JnZ}^?j_lr)hAj_O38|aff49}@^K9b}
zI`h+#KDyHUSu4bGZ#ORr8(+I4f1A1Q=cmaPe?V4EdU*V3|J$-QY-K>GS-HXQGpuH*
z4~0wPRl}UTyNAa)uHn5UpNvNQoriYaO8aoaLRHl^Z<+1It+e#|`T@OFmBC7%H@BKS
zzXS|>&BrKS89(72xMeloB&7OYC$WE{+8&|gY0MREX+>(QK{2jd@nvNE6EHG(w&YsB
ze_p;2dCcf+B4tEq=~OYV1v@QQvUqF#MB!?iu_iUUL@{s=ko7)!Hr4Qch~H_&6m)5`x8l*-am|aX!&gU_(jVPMvQXAP3n{K(E3a4PxjrJqG#(8)S!i}?bJ-0Bp^
z_MTGbo-4eF3}x@pBXicqP@<+>ekcyFo>8pZlv@p*A-v3mRIFO-u{8bfdZ}xQJ;N)-wwTxJ#4O|{v~JNQ^-u~#caK*XpHD)#8AKVMrxVi=H>3<>e=%<6zZ;x
z4%A*WKU2E5h?E+C){(JQt#uVof0v&Wt2kg}@+mV6@$YXe+tTv9d%SV3OAf1H#&qs>Wt`90d2>Ig@P&}iwYP2i&OEv>JA{@Bc#bnZyxefP&P)NDqNAm#e@qz(W6dyb
zP4;}Xzva!Hb;BDDMR;bf5no77+COjabZj3uuQPr6xc-##%$|3HmL0K@;^K;+f1G#w
z#5Ck4yp0~N#|K&XT)9yK)z%fvFV=j64_nVD%6F(Tz8(3#t7!bhu_|(FP^$IgZGl@-
zUk$X*QVyF@(cP&}pyYSDe^-Y%7HpcPone^l>Q{iT4UOnOcC=FCl}qbb<5r)k_QTx3
znftgb?uh5%>tBYOKaJ>QxMC9$X;qzJwYmh_ynUDa2uaDCrarj$(9y5!_jfF;b;`Wa
zSs~{)K7L#fM(%4SuTL#+j%DfJUOl)pB|qd*z(eciwgW|dMq!moYh?2uy!%cEc#xR71Ib@q
z?Iy+S-f1vsW1tbDCs;UG_$Z(r>~2453O3xGnV9y^wyqB!i^&bzW><*2w^n9(DcRI5
zC~l#)Wd&E~AN)h@f3@(jHn+>Kdo1&a%a>RBWg9v|cwPR#k^Y!}G3S#7^>|JbcHH%v
z^H6e2)1VS9>Q(9CywfU~xNuyXD1B@^B-m#9mx|(?bO&f2MIQqo)1VtlXL4>^XeB6@`AqhXkj?>zme^+{sEE6r0~27sR9w1S#H3
zh`d`?RdJ%P{Crd@PQ?g{rA%3m3tk}7ejW7pZOl-6cDv0HJoLKl-kURF+ojSc+s}J1
zGPr)we(CwXyEx>nNnb;s*u)_{q+>O0NuVODL#nXX}nMQnvSe#TKm
zcerpYW3Aq^#h;BrmKA4()o#HKmL_gKcr0yqrDtOAndgBUB%Tev%zYgi%-<+GAQy6$EGO_lS{iP9
z+4m)lf-2Xwx|=ko^^G?Gr}znF9JDB}N*{`ALc{c((o
zSF~QLEYMRgBmLf?St#*4mH5C?x-%>izc|9VEWvv9vUj*$d+u1(!06EV#4svj0=N`~ReLvD1pQ=ZHpV
z4Se>I-VA#7r2x1sKVyxLfok0-T<6llO>91P^@CZ?<^D2{-HSi2$cpvPtX>k(8?h_>
z;N|z_gJ-+=+m
zyxKW!!1++&-2Q`9R{O&#Z=-FnZXfns3M@>S-;_Dv%>2#DGr`Jq%nEZ~IavcBP(3no
z!x}cf6mH8EFU@>rC^ovW6so`U#_vMYT?(?Z>RNE;XeEjN+$(VkK_mN~$IW}JTGJLT
zL(viV%m6Tgo2QiDA7_7pT60}FP$9FQdw-{s?e&HH<<1YbtL$(jL-*E!`{(|#koD%4
zHs)4&w!zcC>^sg)Q(cy{y^8hs422si+p76$75@V$Dl*xVA@URfF_&Qu0uvK5GdVa4
zFHB`_XLM*XATcp8GB%fCP6ZSNF)}nXIhUcO0VjX8cXe3Q-PSgZbT`NdQqtYs-Q6)X
zGsJ*&3rI?dbT`scBHfL2mvl-ANPMHudybxSz3-pjHP_5>oFJ~Y-fZqH!Py@R`
zm@UlRplTq<#tsBzfVxP6oxNP`Y-}OFVsJ7u|BCd>TY?3kU~cIEe(mO92QUX&0Tfu2
zSpX{F*HEDyfF2A2SO9I!9jyUiYk(F|51@aoDXpOikkwGtR?}o)fdrz_AA_`5LfAS-A&A-2GO0{rR@K*`P$2yz4da*_f6qjZ9n66yq%
zL;g(+S_sImn2!IL1KfbX|43tN?)KYPNli@&;ACzGf&fA0AWNtr#2n)81~B_AgT8@Q
zbpH?p0wmpCU4P|J{+G)2-)a6uT>^g$U73-ikDvML|L&MM$lcBRuiE_QvMs?NH#;|o
z+dmP30Bbu(;4k)Wzj|f|`YlrySCN;I*3@EFg3ca@Ss4ti4u}Qf3HfdPE1tNNk^q2*
zp9{baegB$MX^@p9*vSdX+70zr`K0Wib%KChy;%Q`0e1j_UxR%9FRis5$jX2ER~M|@
zomsU(b}sHfd8z;Lfr?Q7$ZUWR02=`40swki+Oqyi{d;nLiP?XNp(XM2aRxgBtj!(W
zfPQw?K-{(rH7ctL7y=y*`u{(B?!R)$xH~$km^%UK|M$ZFx3#&G
zouk+PV*g)uy1-w&=~ck4PUepPrL%LBvGW94so6m+ZT~6qKk|Pm|9ly8(Sf%B(
z#5H9Y|Bqq%Z7dD41Y6mGYyceGJOFc7S932^Ht2+KaB~BE*r9`O1@!zqLI7445Eue=
z0XVxu`~cQqSJYo^=HUjgivJS*gLnX}5`Pdsl)=9dFB^bW<`3cou*&~IyZ~0^KZp;&
zs`78d#{po~_=BK+nty)~)KB-{h##84;17c8%>Ruz*q}kp9i46ei1?uri@yw@Y%QSo
z!XG_UVEISS0X4U@wfnb#iwlaOTlcg>(sTY(BWPx4psO9&>Msi@*T1N8Qie?cgH@89r0
zx0Kqi8|U`{%JzR}kNwY8{2OaRT)_@NT{|o250Sr2p!?3%&eM<$dit?L<b2rKoM&Yvhlq~T{dsvs26Q;0`cNSkOSBbI;4*`xxB76df~rqdRHmE;7B
z{gZ=0MS3)$1dflZ%}F1}FRj$Yw#@yN{mC$;#fP(XShR!FmFEM?+Nc=z6|(d6i@Cn0
zx5T$R2WSr;Nbi_cr!p#QSp722`h{$M;tMR~=@vgO
z+Pq7A;njbXSzZ|7umX~EXFJ|E*yuvh`zZQv>1=y&(Y8i$PhCQoj8!hoX^MMjAilw?
z0{Io@QF$R$JQ&zBifWe4Hnr%B&XZ>!tVsfOqa@$|WcRv&^}vClrl7?dvfNc!9vgV}
z?F(vymU{99YXo`fBj4*a!9f|_#mz4O5AXd&KfQlfYcya5joyGvswffi-sQxH)mSfz
znw>l1_2ya$K3lI>jDu*N3%6E?FZ=Z-9)G$Bq+wWdqt8Xh4N4aYMnw7?KI6>97`ZAuvJrTZ8iQS
zq4@)gBbLEOT2BNCgCh+AV!2cSW;p#F)p%uYkxBXkUC04AeDz_nHQzv7q;SqeV(XK&
zW;z7F3gujgqVgf0ldfU-#K+V^Z@f8*KJtGRfMiN)pVPE(G9my~#BrLf47gR8&JTt+
z@(&F+H8A0obAL2tl-KbFMZY0cpKnpMzldmgx`~0SvAzgE-e*8sLB6U~J?%=$KPvi9cl&BIJ}9m~tq|dtH;;uV
z>Nn}%B(TFh=h|$$?vK$)*);kz$NYcFG%%4Mg80acTKHQMRqUkjK)LmsLQZP&R=9G2
zRT<3BO@{|MS|J1(Ji4D{%n=QfvYrH&#BT$SF43nxK4)0Vm$fS&Ok9nuio~Ro^DhjZ
zG4*SXT)!UmhiE<}RhlU`QH6lZU}w-jr>AtMud`vOOeMbz316+d`}khyEhc|JIz8SD
zcWkI#9))E#v6ie()!IVpV+AV-a&>75PSMxdFlt%3kMYCmPgWa~N&@u8ro*2R=KOk+
zO+pFd()tcqXa;&;uOA%fg&<;JeTs}qk@U?-U4eVHkFvW^;fL|&!<-L%Vyi$d$A>yu
z_d-z?>#pL))%B(zRw{N4u#XZMzn#Z
z^7xXC*w=*@A%$ry(U|r83|cb?S8DoqAI3cIw!96g1$^<49~y0acV0GJ(8#$2<%eb+
zT8zmDv0dCQIhbAHC~Lxs4XRcoWl{tjRBE(R9Cv9sirHT=uk^obX-a?1=z$H#z4)r@
zl0lzBTYrqr=v$HI?9C<&WUFhIl_>~)l^f7ZoZzyack5JBOA(BpYJU$>DWsX9+bN&z
zGu9G&UjrqN6v8jhQsCXQo?3>D2^aw2H_$~m9B;BeN=K(
zzH^hDIKAylCj|pvL8X7oSc*?n{-v<>%whxfl`&;gb*cA)Z+eWcE?yPW%7vnwfEFX)
zCg|YXQ>G#|uBV^wv9XUy=sbp919kc8z7v-=SH_(w^alR;;`wB7@B%TSmQEmkta6!U
z-Gw*eCmb0ZtL95^7kUE|Vx#R3AMvG;{cmw7r5wIINW~gAP6mHbtdDJb0wb+%JwelT
zM5h(XV%*8JrtnA$!-BNKrie$T)g%tt4AGVqv?o8%{TcN|w+32Kx`_CY2I!n@7dqg7
z&aBzk=D1FCPK(3vC5p!AU5b=E+f~%W?h&bFi7W}yx4Cm36VT$7=J3gTx!pGGE>kTi
zx;)o$Ih6C3HCcZvdqLT-^+BHi^y61UB
zZ1`+6T6D%mzcCl46G*WMfd&t(f1@15%q`b@kbtsWB)fm(5JVY#7IU|el9jfU%B1^s
zE%64v&v{ewtoo2#D_o`{IzY7o)5Sw+DfYV+KJvV*pL@R!7R4P&)IznH2UjntgM#~W
zx;`^)XG#UBmn#(wLv@Fy-fHBcxvlgbr61tmp%|Ocs!=@ieH%Rcskl^2J8)x4MvsZb
z1;^32hNXXG#SbQ53Ur~5*(x2$96pqtgx^4{IlhkO)W3`&midTm6ufzJRz1Y08A-by
zYvotVnMu>eMW8w*9P3rzM?d)LA_u3zZ^>y?@DV;YdxhbHyu_G@0YM2Bsy@u+#%^wK
zZ4n$zxfy=iTS7LzmNuB<{ZXC3!&Ww9+TF7;3{QVV=23>M!z89B`@3>P-IK(dfd0Zo
zV~bkrzEG}<=-TXf_oiGDrjUZ2!wwc%YuP;`sqwLx{cNOHpScc^xrx2ju3FK%;Ic!4
zo*U0eTUAm}a`u-{DqYG$&>UW8VUDtq7;M(F4>^ZMpFY3v?eT)>KX;3O-C|**vY4Jn
zV)cJq?zHL_ai%x95sjIB&(G5i?sK-i7Sa5()`^B$kL
zSTPULYF!l+PCC2sE}8pn(kCly($cw9G}_*I?p(ROw|e`nT`z#kMzc?K)lLA(#{y!L
zx+Z>|pS1AOmsngL(~un;WxI2oH>BUs`@4VlgIeW^!V1v^b>IYN^z8;1pV=STwr!AH
zzt&;nd`B9X8&$VvwnmXbiAZaX^KLI;5kVb20m2=I*S{zL?fmgnk?FK2UKL*}y!R_a
ztDV}bqImIagbs}gFt4U~YO>1vt#l1Zd1bG?wRXK1!9O<}2Qbs?xXav~;qo+Pv^9UH
zDhEWgo3~uX?Fl^9vCPU^c&NFoVEE3%t31E~+U%#;ctTPNVW+-z&R(**6(&I)=6)*i
zl&xnr*Su=va5itUnDoyR+cw&kbb)z9o|Fw_Dp|G|78jMn~Ou>x99DKBy6jEE^Vw$?TSSLJOWx~
z=1$V5T8@=k5i}xA;mo!27Cg*;5g7=Wt{wt&Q;^XLW7^34;7>!k0ZU^9lus{QT=UC{
zziRR2%cElWm)60F?w}u6o6OABd%1Sw!W)T6ou%?F7LpnaNA!Mz>
zcaKbmMsqDksMvBNbM9-~%RGNp6UL6DMy%t>6Bgo_!&*^pxeC50l;$cI-s<8-MYPqa
z0EJawkk23mGQFR2-);!9Fj=B$w02SsHm?GPJxa5Za@Aqj4gFVfay1ilfHTzHer#VLO60>fY;mng|Y)7^%ZS%RDej
zs+-dvGfch!v#J;{9O&_ZkXroR0cS$5IRBZAIKvVgmG}OEh}@RV5BpJp
zfixU5MJ^_<%(gQ39;j7+vcm+hx)gXj)a6KMFBmWsUXj(4@;MvnBrHuTJuTU!clF-5
zD~JZ3?W`5OmNN+eTtmjne2nUd3yuMXGu^4Bp+JH}D^^>Da6W(Qm7nSC0W9&8MGOam
z@8uD*B=re;l`(VkAiQ7eZyyT6K7lZQ@>XgT-L!_4y&;a&Of~jmnKXGqzIa-doSn^(
z?}BL-bZsEFBQpK{BH(UE&<1RK|C8&-gCu?lGm@pL(E`O
zUozVyr#!RXe?se)tg7A<$TZ;zGYGxsfxvz=_$VruPZ
zx@-EzN2}Z)|GfDGAw;vXeDS7O)2uEtG*W_D}aAeL8gkAi8HjA?Y5D9BB(Yz
z{#>b=hGL?)>J$9BJ8AZlKg$)mBB1=JayB`L6L$Y$E%aWPKxKjje6kvWPQw|f})%>pPjL58!vu)dfl#Z8^
zd-_czrYgHEz0o%9K#Z;9!yR45{g*^Wb@R_!C|L`rlC_#Jl^#AlZHvAJ(V$-IkpPNMm9yiAk=dk;rgG$db`aRzv??F*d=Hsw>JpN80f
zt95@w7*eeVYv2x$-x_)BT?*P_UdaHae(-4NO`p9vOU+z@P5FK_q(_GRt|q5COo>=x
zdF4apy^n&MUD@E$P+5K*-MtXZ!u7oIyykSG{q3i4iaF+{4|~oxvc-nNN97j0@IQ_Q
zVK&g6FJ7A}o@oTb*s@OYliwCL_C`CU3Lk$QHn~@`@5~c0?&Mx1jC8ezQr;)3666nl
zHAsmLK&^6c=Uy%7dNG0=9ZQ-b`UpI6^dQaKD17;q&n;b(M#kAsnV+r8{EI#J`)1TI
zAE4DXmonf2F7<=pdT@sjTn9(OY|DlgSZy@f2DU_(W2D#;C)XWd{{>#`DM#d$lKOw8
z$($OQ${vW{W%*g##14HdM>1y`Fk_pl4Kgc7RB|o-(Q@Y0v{1|ZkYAvFE=Ev_H*mzI
z98cBzh*Jj@!`)_`K}v-@remi(wX=2Vd&hUX+>f^=a~;&_?B8Q$eZX5d0L+lA&
zxuzkEIuXbXG(#*ZxTWKsZAR^}50buZIjw5BE7Ec1u
znOa{d!_>)}B|aj|sX5kYu1c+})VurbCH1JMeRz{MT&e^NX=>TIP3f!#W+H#TnYPEP
zZ(_V6u$#}bWHri|yqdy)!WktznhD^*#gtoMYpcF&N-pD#+f;`eBD6uFl4$1S>o+S4=Il_NV7#pNdLyIiFc$J~1%Nu?#JL|&M;v+1Ng;a@miooW2
zTn6TZ>d&rAsDdB02Nvpb4f`z+Jgv$l=Xr*H#lue!PE$rYFDjL`6GL-yStys3ML3NOFIuUnpdnwkPbT
z-1gR$J|nobfYy^Cfp1O{=c&eykkqpS|KqXyP?yuadwU`UdX-+1c+L46O!Q`Cz0yu_
zu@Ii+je>Xb>xl419iEkSqSledE1VENwyk-HKkc^6^W$8lk5T8Cs!D9h`=gx(+>YZ6
zAGeo3FMDRtGvPZLUvPg%3K-Vj%x4O0>Y3j|rX&4|mbM5UPDn;Lytif|@DS(H1w(QR
zWsH<#^``S>>Q|e}%=01(2|&^L^Hc4Zhbt~z@l^xCDs^8KvjoqHQaxkIi>!=Z7}%!)
zg+;`NuV=!GWc^_ImJe`~)u&xNd7epOtq!-W#q1bqAe@({_!obcG*Lg@F5Q|j75N$Q
zJ*xG|xF>sQ$0kBZN9teaNa2lH)qk^hr)-}M=GHlU(~Ey~a%fhht-y3b6@0X+*Xf&O
zYC){Cp?M!@8p%P9M!L-TiYtf2C}z_biOdw1H>nWwfa+s2M=4%M{Gx#@_SmfG=Zu%4
zv4TK?mWA6|dCGt6Rm@{4DXSYESU7_AFfPG$8Ta%DB!X0-ycZ>@dLL@g`&;CSmQ08+P$L3u6&rMHaFx)ddXc|N42SO@juBZv-}1Y_rerpX3ao}pbd4{ijLlj4@8x(OSU>Ysnf;59W+;FYpVR`H8(!q+f#
z8R*D}AV}<&>m}f4KD_6kc<2Ibz<2Yb0r?HlBHJ&I8eHQknhX`;7Giy|H9an_e>y&&LQx`3F
zvhciIdTD>FjN;eL7KJcB^C=@V{hHa$?jrL;%@6#=5$zP{Z%e1$@np-`OOJbb8NTA~
zi$ZydPW(vQ&6q8J`Jrm~4euH83u;U!$Tb|>w|-s_ubHvW0ER$$zsCHFzP|i*s_kN&
zIw4~eO(gMzuno>2!sMJ(d-;c$ABfB{GXw;N>dPsmV#61h^
z;$v=KtcKB--~f1gBuMOO!X6$K(M~l6KIylD@X^Hs?~bruVR<1)R0rdqsE)0WYch8}-!#)C<6zE6ehX0lbWfqV_`PuP
z?TV3I-8Bk{dfeMLh2xSp6#>ycN#t&D3>c(mNh!L2I)rV)!~WWO73|?6@Owt9!5&}v
z`kew@BNQ*@PfL+j-Frqt>ktr`3@jc+vB4kT&0)H-l{n9ue7R^pK*3sXULaLz%oS6v
zQBArU!g7#IG_x-dE~XP3r=4>9{?w-#=lT;}Cm@Yp+an~7VR;z8^WBl!Bg#El$$#>E)u#&xGD>D>a8YHZPD^OAlkUzb0&L&
z_COlmuhyO+{V+||UvM4cTRHk+>uapicDAE`f+uGce{u+MXEGUo*7#6Sok3$*%a}Kr
z82~o7h9Q?e7uHFj@|Wa8az|;(aM_2Q_RA&b-PB?vVdS8hWv7?5$VTv0t$1JBsraU$
zX;oCT7k%j>2!L~yh&PMWD6@rrp^B2-kerN&Fgl1U>HUzP(aU2Sz*(a1phy;nLyv2J
z<3k-0!tlL4#!m}DGRr;BN1B}G0B)WRvIzkH$N140VP_rAM+r=N
z7RGb_xJxw4MB8hAezcx`C-iwvGkbmZ5~0Y~mpRK%j}`ZDYnWVeYAv7x0n+=F8bE>a
zc#=vu2i}3JWsr5<;(?4c9#V-A>z|I5yo`-(*(~23L+c;tj(`Dwx}$w+
z6)JQ<(SbPkY{*ci(=(4H!B9vNE-&5)$Skp_sB4#rQJBelXRnJeHjz{Ulz1YYI)g3X|Qrb4#{AGxH
zgf4{UgM6meu~ho=R@t@YLN@__`E+LR@;W~@ITXO8-4wPPW?Yz*-{4P42Y)3?E7m1?6&o45!2*dB4%;
zneVzTPq0=Q)fV^1H;Z3}a$&3omnx!-3|iG?;3}?I`Ic^Eh;N$UTCMmP)PT5_Y64Tt
zJ%u~qmGv5B?QlPch;cnyBR5Tt#9~JhtenJI=?E+q^OEw#VClnv^`hq@`daPU1Bt0V
z!`E&+{QQWug-y-nk@zTo`|J)mb!|{p~2V~^?$6g2Su;-DaLOTsy#;pWP@=T
zj%yF={oB98jbtNDaO7^tjP^?Ny5rRLycdAWLBW+DjC-<4rL3rR)@N@i@1VdyPSoBh
z6hVzuL0kJQ($ss!TI3ehX9i;l@!?sihS_&hry*OqQ#84Mi~rgDGwBD4Mzm|YU0^!*
zYK!nis&87WW_5J|1!g~9UJ?v)uTN@e5a+whw(%me0pwinxk{5b;){>$lBgg0?>>%m
zv^f=>QIx+0#YKnP5r6hmNTiE%VK(MGInTJ(>~D9FPy=y)LbyT`?TDK;(OVFm8$}sW
z%@>5tV+|L7CfN;GsOcMI`i`N}xE;rWzoe51nK-6)jdG9SDDHUjJGCPbsTdNoI3KmK
zGuiZchvPY~q>^Fk3$aKBP`Grh(j~zA%tbsQq**SM&nKbXzT-^%e2d{h6%;Aj^s&#I
zmJ*CQmxbx{Z3d;4pN=2BAlW%wgbuc`k}wdtp!=MEP>3QZimG*vR&ZBzh9T;64FcK~
zkM05}s^nuElQzW9Tgi42p$EUSChe^!r`;1+>cVwFGYKH|er+`aH
zHq(d}JjGjdk(`td0A*BoewAZKH8j(&Kb$ar~o>BJGH&fy;4XYaCANwl50mtE59md@Rt#IgZkEu>N`
zKEDejbKgNfVs|Xae~>|O>!J4Pis3{OP`D2$dPa>Q6GtKl2qchFF(I--*gH70Gg-7~
zmr-?zqAodRvpjGz?({wXSj?3o1)Gen#4rw~hpSJ{ttKIv5
z=g{_4tIxYD$|&^IERgE25)#E6B!^{B0RL3M_n?N`y<)b=PgWd6;U&
zO&K^s6ElKQ2jJ5TXZ2PiJk}=;dkkTbFJ^WhF*e$;yps~HT2(#wSZS$yCrE`RNWY0p
z6Ub-&Ac|~~Rx(!-`}}@oE?5RLEP3KoM@GRngq#y>9x)3B)K%lnRqH1?MWd^*MQ~I^{V@(H(~X%V7A`
zgm)+yeU~e17)T|zll9-TvAbTXt%Y0;ht}AxW8miS(Lq%c+7$RWi=`)Mzqv7w0q$i#_>m686B-J2!sm0`lZpY1go`%S69+J(0sh
zdcK{@MMkx(Oy@FF&4qYXoF)Q~A)1N8{ook+!@=QuO^8E}U0lB}7wsTx`>joY6sPMn
zY0CypKsaO2B_p01iG|32`uoF7cRwFM(_vP1VZ$seapC4GiH&~U!N_}u2yZb#T89Ii0Y;>hC2)Du;ctZ~^t>ev*FNb}DGQHe|UDi|lb&5iE{=jcYA5o5a98;I(k_R4_nDkp`aL|}%W1lFwrzt}896`b;
zKyQnFqp{yIoj9T1@Lk=RWl8^1_otQ#g}&(X2_2iJl~Ei`2;cueqH+GrEL~`SS~So?6G!n~%$J83H!Y{8
z80>+;t1qQG<0$8*1xyk9^aZ7Z6|*N(J#J)y5)|r$tll&HJ8xmVM!p!XI+|o?C`8$I
zjU+8U_cDez-)HI*WZ=!Sru&vpr|=TSPo8PP_=Df}*U#;qBG^sv+RhfziD(>WZmLo=
zgx~#&&=B5#cnV<_37-rQH~^I5yxJXus@|RB#6L7C`Q~{BwxJx{4^D=vL~`;A6$d@s
zYR)X2nlWN9Jy-QSGo!+Cmqs%3X)N>+7ARQrKv>a09>
zu>*|#sl)_yU1bl1rR6?F6;3K5kCCVig@xb;wj0VFUzuOa<{4)&dnQbeAZ|7~^dH
z2syuhhj7TNu3Sy8`VG4E_u%3pOwVuMi;A*R8h%_$MAr+5WsIznBVkjVoJILeAae8J
zkWsL)-h4gz?uf#Re=cOZahqeisp$bA+$?CzM3>ZW;}})Jofu&e;LtvYCyw8YCPz4h
zZ)bwf@_A6c>5lQ61omOVn9v9H+)oyiPd0vk!_+$6l>A!?`kUI34dy4VU~6K!$|NCO
zrC80Ap)=*7@^Nna$hZ4vZ2IcPe7H(P)mE_sdfyR5%Zfz~+j=~oi%SJWwo)IdG
zD0$#0RkvPm!4|}?qgA!alxugD7U}G(glo%~CK{(JzF^1k522exAG7oKrfqpdVz+63
zU65lVj4?(MIv85o#}jL!Cj;l|PrcGCsDT*piFm|usmw-LNa%Cj$ILS`GoJX_>OI8P
z;G@j7+~CH4OYmClx_&iivQE-nH0|_=WtZcWrZQjGoYKYVLg-O2%k@5?Um2c*`Q$`qMR3@>h3CZQI?fuP-x2dX
z-RW#CYxt$69TE7j`Rv4~7}ufWLcI^3B)IW?qEE^n`3qAqumq
z?;U%h_*;W@QA>kJVj4{OMBx5Qv-rbTL7uzBC@;#gJV8FfTd$DKCb9LhG2;+CTPfH<
zgV86+Qj~_DMnCR%9S|U_Zo`=WP*u{p=|V8Ufl=DqZojHQ;!WQM#(`nD_l@+%8b8wY
z+1ij&Asb!ttV)Fkn<$lkdpa6j-G7%OCq1hF+Q?$k7N)lTuB-J!);qknx-1eoU+!gO
z&fI`pm%@7dV%;sUvMmL?j$5Bat}+H9)|>L9q>&>=&=mu=EAi&nj@V`(?E_TsWq98v
zV%{OW4$Bx2^U3TMzbxaT-t$bHT@69nyM1=+q)e5%jyR{2Rsc=kw
z?25`yEO#)aC+f$oDyzguX-Vh3c-=fOt}cJjOK-WOAQ>Y0xid%W<9n6TsQtvl&uwOO
z7{?zv?jE+y$V`-_A6~A6c_F>5tYOTyzSm*2xe0Y^vHTT3I*K=)8Or#>1`h@{Jj5`T>N6X8(GYieAs8pErtO&H$9$N{R8W4W3r1^JqSH{N_!`y%-#1LiqVvYFdjiGTf}Sj?4@Bp|K;q@=8|jCteOttWKDJ_t%`~y>M1*0x(TO*R0mZHDgH%
zwZe|%r6+eaN(z~}aH;!K6PL7R`FI-zf1SoPbmP9yATkIJ{N+iucuYyXD2tD-9-K
zWm@t&g6d{{JputY&5vUVj~3q2Eg%U@3-=O~hFY<5pe
z)+Pg?Q;X>$zIA23En$Dk@XaibDeZG0B08gac@$^%yw~9GBO8go$)?xlbcoi^x@gQuYL9f?(H-AJvhBcV#IGcTwexsUb$Q+*d_>(
zsM7UQfIf0FIElp0uMe`7rTm0RBos+hxJR|qFU_)reVKQxQG*j7&GH?uzKG{B=5E!j
zp!!XVC0;1e&VBs#n454~*$-pSS0nJt6vq%{c!LNS9*)29qK3
z6ah1rVGaTm5jHn73NK7$ZfA68G9WQBG%%N8P6ZSLGBP!j(Gn?twN!ar6IUJ=s`4?c
zXSFJ>`otnc$i)qH(EtHq69khWR6K^vOEQqm#F>c!BUC}K3ZkOmffovjF0QimC@dNcvM{Lxz(^cK3l%~TLs^iK03H+xlA#_mXaS{v1uB>T)JoY*C5W6kLmjK+
z3sGUE*<_;VH(0`yDs`k62$#mFWB|r{fk?Gd#l2OKMEG@HAV!7!Tptw3r7L7AsVXs6
zCJ}La01~he(hR}1?S*X;LI&PJL#bMtGTIve4?WA8f<+>W#Uj+18CFQqI-$v6$EVU0
z46smiDnMU<8XBOT7&D0@I$1sJF2EfQkP{k6GLTD(pt_w#L?uc>cJ_B*hzOReY3L3I
z41_(}=rP8gE02wp10zO|EF>{fg92HMH8Wto-G+V;ckc!WL717QxgLtQF8cR2Z=r`$
zXlF?VYX)ZNePS4CW){82P0wv>D3T!jkQN}^`znQVBq5$0D=MlCsP@Thfzi&0?qI^_QDB76HC$Qq7TbEm82}B
z^?x|E1c__8Bf!ljk(wkHm|;}-n-pZ?y|d{c3&a2}05DCX7jcd4vf<1U&W!lTu$m|n
z&|(IE24)aih(0_kgDr%BrOhzI`o5#*iYEyGIH6&Y;?dOb?A4=4Ed_$Q%?Qa`*BdQ(
zcni@m@zKQM6lq8YIMnh)F%*lG;qm_&xL&;?%mzaYW`rIec>DPSjHa=4
zo*1da+s_YJB}nl&OtZ@eh=e4?q7+~nb#9WEH|Ha#fjg4
z;d*m%+1E;zrc&W70!P>Odm#ubO{C2hBe_VB9X)UV&FRs=?VZx!%MJ~t(yU@2@b*Ld
z@bL`-0e<3)9(kJX`@=3%MBdvqH@pDCG^pXVG*Oz*mZfYslDjukw(H`)er`d+Un)Nd
zNr+zCw`KFi+A*WTcl_iErnoH;)&-sjM_%XWe8}B1
z4UUf=xv_2boJ
z*Y+DQM>N*5XPwGu`{HbSP+QD@jO3>SE_7O5&S5@>E;$uXbWa>rwm)uQebUZ*=E8@%
zs`|38^T;i!%ZI)sd|kqqceD6|s-$4v@h}&CL5tq{*YD39o&Ivp9_jFZ
z%>vEsy1n_f+M;4y`n2!rKBUV;2x^xNA{%SvgI}DfSbD@Df6^8Ga)mX2qTjRZ!81m0
z8>Ff&JK=2Wx<1CmH?TkzaiO&$NmZE>_0|1GFSUEN(^emR*Uc`6kj1{z`=^*T>Aq8b
z?e`KNO*~rMB{pvxQv$aQo3M6GNLy{Ak7i_UQTDcq5;E4$^7G|_K631ttncj=xuxyX
zY_9Hw&z~jk7
z|NP~mM(fb5xeFbV_6}Koa}odP>k7GZVKFaxi$1esifgSl{#tl{v-SRScMn<3d5@Rd
z)pef=?)UetIXt=2T>e|i+}7Rw!feodIPl-90@G*#-Er%2%Esjn(~gw#70oi`$YYke
z?HgKTLz8r!JCf={UQez5H8=B{_Pix-4s20fLX7%_=14*s^HOrh1w7%!`C^SHLaL)Z
zPaT_itLnhm_$c0gxd-ZNC2^lSjUG2SzkXwBgV$wMK$WIQt3A-6@lM&*+Sqldd{kg{
z;H`mUJuj8V*Eh=s+P3wd()9ehzhvh>y5-MLzq>$E?%uFVerXq#B=JsI@I2pjhpfNk
zalYdm7kqo(WmprAH+xOKQeWzsQpF_9_91BL-C6#gpD*lxVD6b}P6U_yr|E2;)+udM
zvl9Y4@x|-+Zw#xkU7qpX`i(_R^u?5M8-`w~Sv<3@!0CoA&uRJNK{j*P>I;ISce^sY%auzhStH}Ae-430L7+HL1&~(2c>&g-
z9}e_&2QdJ_M45^o&<9Z!9eZm_07g)l4>}O*g8-l?IAA1UDgl_G0|`O|AcRH%?w}VG
z=>ecU02|N&f3UUIwXz2Et;}sLtc4{Afvx@h{Lq;HaM7{0vDFs?AX;WNx&UY=2I$*b
z+Z_MefG7fgPcgvEhM+&rLohscH`TS#vT?M~m6kj{10W3qfEX;|IPJf<2@se8zo`*i
zJuqnBp9ugVFC5NKMN%>_Fi^tNAB&SfV>~7NkU#m^e|RCVKp-090}!4V5DES)j6Vua
zC@0Pf{1xE1Ie-ZQ2BNUwv6CM9m(rI|N`ezXj{BP!p%A#^n8;t|02T!Qk;V&({po9B
zVPOLJLJ=q&h=QVE1VbDY=Z^*4e#!`65H9$OAPDIAV=%`#O#f10{!a53b!{}^WX{OY
zaA@Gae>(<6`C~)=sLel*4MU@_2rLf!D{DE|@@nEa*s&oBtdND#J3n6R>NGzu9Ez(EfxNi#H#Fd9PtKO6V2TzdXU
zf20}I7Zm!hb^W(7)E9va{@>>RGP4JdSqqt=F}_gbzjO$!9wG>YTOe>SuU{qpCI1yk
z3+0If0dZ+L2`O0xnO_>)<4r>nMw)OfA&xILf?n$1I>NNVd{7`33&_g-RDp!L{EIxH
z8OPKCNo!jZOEZY*f1jkEwz?=78je7De*!Y{3IG&?fd*4a5ynGCULFXQCd@n>4Ei}g
zfTRQpjU%`Ke*U;{zypn;I_|N8JRqrcEc%6%07=N-NKpomH294OIDmiohl
z&=?FN=zW$k
z{@hB!sQiVGFDn29gFqP7$S@kFe;VU`yDqLYLN~v^>?FUk#8L&bhJ#TW=}1<86FUne
zcZLVlEQqR)yem+EHZf4S7;!TSRph`Q!-V`X`7EQdZxe2Leij;G8o^GdtM&H2orG=l
zT~mBy*>fJ@&qntjI2OyjyW5b~zzNvC{i37X)UuZ>(|bg}ev8MstYJLwe`H|ckn~eV
z43Z%Tf10hC9pB$fj62$6Oi6;aX^n^sx;=0eR@(;_qjbp+DXmdov-~R@#+vpjZddg}Uw$g4f}8F}
z!@Lwl;-tt^<56eX17<#Qe=qd;>gk5}Rh(e3OVN9^i{Zv55%YRI*&t=ckpT>dTV{QXMTn{Izhx+ah%01#lho7Dng_!s~J=pOHl1veS3~SQr
z>1!qV)yiSe;tOLQJVR%;I7heXNtI
z+OM=(P_wHO*(O}lO?Lu}KeRAkzJ~H@$CI+EYPMulYp(zmA^xX&s7q|ggEi01j1Z^ItsB^0VSWrWe!f)5
zEZf|(ecOH7o+Yica)z3D&y4R?T&3hy(zJ!*3t2Y>F8A$Eb&=^BO{!7bf}dooH7m``
zM)gN-v?Ju_f4e?Z^xop}#yfn#3N-kdB()ncPh`w5W-!sxuUct|99l#@vdzp1YdpD?MM?FzO4EnkVO!8J_kxaB
zTdc*kb3WU`GXRUJ=IM~si&
zhHs%-e{V#u*Lx3Jc8L;?#ox)`Dw}~h8s4U}ue+8rK)xM4GH&Op8c}#1=3?&!sA8R#
z`7-WQ)>wU;m;RhPMR!mo#S)ulKgq~XWpwra<)OOdvkG@z>aQ}ncO-Y5%;^2JWU4$u
zFjp~?$c@F*C
zksSkK`9@v-Zzx#EfhreG3kYVVmriyS`eYqZ4#7y>tI*UK-6A
z1M?S5uH_+)?grr1#O!IRqlnA+%#tB`IK`Z)?NoN40-0NdoKsey+jmxr9x^4NVon#i
zfBNe-CFINM=|ZMm;pms}_qUy@E{KQt?LoV#%cZ=@wzP3#u|tf4O9H<1a`A1wxQIxP
zi@+YQwGKd&(>8N9C-`AMmH1(4l^}M*{D;0nvM28=5?`%R9cJT-j-?7z`DErLb89{k
z-L-1G8k5?WQ%dEi_gC#k<{Km4XE5*|e;81U&u+9|)qhJ*psUsFn7zxJVCg
z1wuaDBxigQlM}D1_F6z7*RAj+((f#0fAzT!nVprB(8Byd{+r5ZDtadkvvlf`KqE`$
z{EW$t_J#)++{yx~clqN2-RV4Y%4X9q&(_tEIcjf2z;kr9|Re;{3L|>Ei-XPwYLGXN}k*_mKpX1E$_;
zlU+#I&h1IlSiNoS6Jwvg#=F1tq)nubf8&%H6%I|_4-iQm+bv)0i7PB{m`xmMvgXzmH0b?#SaXKAfATbn3_`v6
z{i5RMcd@O3Hh?2i+ZD?lM>xw3kV;dyoqINQCc2wI}qgb>M{
z1>VRMV~wcI#bp5q)n$5pf4h0}vZQfbo-c7#+f1nl2e}dbuERJKhWC4uNahLQHRY=4
z=3q%m;x1q}e^;Tab7UgB{%BW9)=KCSYeMyBn{W6;3&)m5?b6qh6Z7MO8GhmI68UJj
zi=pLgac04AS?*YinekrnL*~HcnnrcuHa^bN{+lyd+2!}5*%{X^f4DF%cv!hf_r1}m
zxBP;jSy!y>EJHHX!uADwZ^{9?1$E`h67vb~@4AiQ=Yo^j2{SmYXf3yt9vQY($MUw1_1ne3ud6M8g`f?amw%Sau
zf%5S+(%)m|Ch9EOe|D}on$Fe9VSF}|{k~_u{B|X9u(x~^us;Rf;~w^^A`>Kg)1{Mr
zvqG)nDSq(LXF-$J^Vpbhx_!~UiwkYyTXBNQY?M1|wPJ;wl^QljedRa?%2u=f(0eJd
zD^USMtEfQHxq};=O_2eQ5$lEAjP#b|Fq>h7sCu=I?G^Xhf9p+NY?Z>7JYDhLzH@E9
z0$&?gGROq>1J1rYOe-IYxM_iU(S#=5J-G0
zmbt^+M0vH3wlZ9_;*;RF*Z{#P^VD)YIXqF*fwa>ZmGHQ4{H()i@w@sRR;yObQnt!~
z_`&Jq>3+l>Iz7hy&2{(k%aN=j?Y+hj1Xi3Ls^2g(f35%%(y!-L@DZecdI5r45b1fv
z@8fbKr&C&`b6-(+c-2x^{gZ5I=rC+G^#xRn4@DfIm^Vr5I5nFD}c}D7JB%8
zZf&G7ef74@ZJV@`&*L=7f(iT1
z1L+M;e?REtAQghFP*oW#|ap`?Sh=P(>EA9;K
zYwi0oGfxyJKdZBkbam=DYNJU4mE*O!i-NVYS*ec*
zWcnGZJ9`w)nUqgt0;1>k4i0#f6sh#-h(}BbwqoYA}}Wd_0Y$Ydg+zTeY?d`
zf1^12$)F_fe&z2fQD6JXJJ!Zd<>RwEIv5z#CAixPEjj0V`s6DZS%X#
zSrT1izJ3zw3ahX{G?s_`Ub@rT+{gn?a*Bi1U9*(#^#04l`<*iFv8d;$E<5&ki$|`f
zK+4-cWaST;X!c?k1RByF7l>t6bxNQwu6($uL#sIKSF)u$arb)ewM0*
zNOi;?DXMnH!oHaz9b=~a^@-@2q5W~9sjwM;r?jsj-dnnw#q`W9qO-S!$y2>;Lwj%S
zi|^%>@5fPJGXupkrIVhpJiYq3@(sJqI&X>{{=(YE6vpZ%DRI<~-HsgDDbY96aLQ2r
zt^=O3O#``6SJm$1!~@UtuBUUNe@{_(($GyHH+}weE`3O?
z#qAtos6|M&BV9XT-_Bt7@Lf>sl|lxa=N!b3=7uPU!pVnH1p1Y)l4Kq3eiIn1Sa7`W
znRfBxjDkg61^)`;=BN*gx}yw>PfUiPZ%&_S>x!Q<6IS-g!+RApXDwV_e+x6|%F=!_
zk#q689>dH;$vSKUDPm;OS`Jv-w4`0FJx^na=a#?yvLU*tt97>!Lv!cJ@ahMVY;OHl
z9<}0{gS}Vz6|3gU{mELjT119v7$2}HYxB;XLk5{fh`mGagb(}2_n#{@Dpg*%i*&vV
zfC#WgJ~Eomur%JRUOA`pe`em(CT(u{a4?-`HM=@k!OzO(@X}^jma~ubE3p_p!8`}*
zC5ywLjNIuZTNZz&W=b*IL1ndm=IGbWS*X0H+ty-hG5a~rcP!Wg3lBwim|11FvwPs8
zsnS}F`1}w-*GzDgTgUc?$&QsmK^f8a3y`^7BctA`aIYb%cXb|hf98d6xy9f0CS%mU
zU=DBG@{XB)%+Gx&5SZ*MBn)ryY#T4Z9QMccaD1oGptFpoe=MXb7CkWtMK9x0?R1?Mru=#Td4JTB0#u)@fS
zn(d@861TJ3e+q~W@AO33m1jySHL~)V?!*u7$h>qSt&}g=jXkqM%Fc`C{z4b4~{f5LQDN;1PUzQ_kGoT`(ZT9?*rek6cNP`*TN^@a?Rb51>9IZ
zQa|-ge<@LCa(sS?xfWr&(OVvx)Ii2_P+p-cI2f0%=GmQ4x2YMivp(~B%`fAmhy!(P^|l&ZVCbC-3NOV!Dmy3}6}
zA(dk-@{D2WT}r;Jnx!b6^}Z5=tm%E^-oW?GUQf2EXcuG%b0m#Vj^12`+!hcrdgas`
z#rfIN2z3)JxkeT_+JZEf1M9Bcp@BTQP)C=&vYFu;dJ1=G*f;Sy1*&q^2?x7Ve&0tL
ze@8~TUwp0`2s!D2n7s3r`s!Rl$wp5^^VEjK$5B=mdCDRElMPJo8|~t2mnZQgPQf9y
z$l1a0^?^ariaZj{?r=w6u~sX?1Fy4foaP^1ly*iH(J6J^>lS)}Axd1P5wYAauj-u?
zuSYJ$!@_1{AiwTZS^(C*L-6{{9f7(h5lTBLTyh}t>jzi$#92
zOxVVBTH)Qv#)6XvXU4HCmSzV%mqsMUr%#A^6g5Rrd*vAvp!NMkd!~)aD=$ZSw=yiU
z8=80eX#6O>Rwj8h)|t_XpH|$Hx{c!OX`|?iva_C=8Se8cuQbtij9c~RF?K4re<_s@
z;`Bcfh?yF<+3A$Fo!C;AcFZnDd{$nb_b%C|{xr6@IDZblJlNr(AVVU!oUDOTuK^TN
z3-nLFn9-{8wS;AqPw
zcM_EIIpJ(io~mC(nYQb+&v(Wne;p#JqGcQ+F=V72Qs==@$#ubr`9=kih*W%7V0r73OwS8yR7tx3AUQ)g^DXZor2E_R1#PQzZ(-Ve^iwoUgXSi
zCb}Oev68KW|GJ0}-6+yru+VJd$d5Q`IyA*>DP}@Ai}h=fsYOvf_H{}rQI3o%KSmxa
zV0%#?;@P8fD%?xgN6(kN=_`q
zLjG0=r?nV%CZA8ZSkAO|DIqB!-Hn8Fhk!_fARrxn
z15{MCT%0XHT%2rL
zKwD2I3wHoFJ0~YE3LTw{JJ14b?*fvw00V^q0$@9Ux)pyI;tR3huP^!GqFPgYkRu7y14%j7a46GT?JNvtdyFL8~~`t3Q*A1*7^NY2MB`jw`B#W
z=|J?q^FR!LyQ|9SNa+}A$Z>J}t^vRW@B+Gf*#A!ZU)<;+%m9C=L0oOzU7Y_)0AREO
zgI$F=IJ~{R*=;>N!0ayWw(PD>fAQ0?v-bdaySRTl0wAC6Kquf|jqwCoL)r?9RxTh9dk?V3-w}ZT8+#|rmk92bLD3UpmNIS~-G%
z9v%SRzukb4M*J6JNcVs91#l>7>d1d-=(7C3Yx9?#9LUPW+8$&J;O658Sh%}e_@Zz^
zc7dCZ58%%QS!`>d&tJ;|;9v*2fFUjbS5I&tz{bTL<@Z4O`2ZYJzeRr|egKE`A0z
zH~51f*$w}Lgdmm{e-Omd@(+Ua#nK(J=RmL%&<6ZR&hsDn-zVlj3P@+?LQ!-4!b{9K?Y&x>uLuC
z{lfwxv;PN#6yop?2x+?GKOm%0PXB-qy8kGK3qtNsWIl*12y!C-p@)C8#N`heh_lPT
z)R1Ca|EM9sTrD6Et$z)am+OC|{~jmg8sKmRy4$<_84xd|5w1><$Kjv&5Q*EL`TRzn
zK##vJlK`J?wq{Q~=@P;bec|VfT*^kW7CP@IjgZwsQymV+|oq1ADvt!vPY*^B)jWgx5bH
zWCY&-*glB2&p#j}yzf6CWITR<$U!3c0p0&j_3!&tAw=t{ds8r!&`q^+QrA8jTdr2*tmrt
z_cSl0bjV5x1pdp{>hJfUzn;2~C&T~Z-!CBmAkYVBg|fQjVkHvpkl7Mh86;OSQw2{a
z#C}>sAa0PXDyl
zl>BB9q#8tuAt!$|QK-kR8=9-S5?s{)HbBY0L(O!gT=>rLJ?~s@M{)-RKzHI;MyRXz
zK7o7c7xrZawMkXSMlrm1*#cK9p1Tu{&k7bvH|feuH#GRyJ>J`Rivg)s=K0s}A_*px
z5nTIwad)BP%OsxS71BtRD2aSN<4{I_krU294<3V-ZdBhUz2&gQGK0ldB_ekaN`f`wzogF2wBDZe%p>>u0ahrYmSZwIwgV2~LsrqwfhQ-%l2
zyKI03W#mzXe-YX82U=?o?z|s!#O$mj%Fi~fhNr6iLHp2u#bd{^9tn?cW=e#+Bchy@
zw4$}n{E5v|=6eF~=uS?wXX1}$eieKNF)ESCq}MU50y^8Hd#5XtMg|x~dL-i%Nj0Y4
zOk3|8razA*v@!L6Jf_1`YgXB6h|^n*eVR}Cxu*0|nD>;18^HGv;US%rIIRjr3`DS2
zr^o)KW?P?sLQCjIzq@U?JWwfBM7rYnQp*@AsYt%5SMK&Ws7d0xcP@c1PH(;eM*s{G4Q4GL`S
z%y}}`N01CRH>LVwo@O(?HtI1+BiQ$fv;dU0r+|Wb1qb`wOx_b82rouzew$b^4t+HR
zqDa)2FgqLwjn@YWE<2
zFMo+)dNZ2z@q7|fh)UcP@g_N_Zd<{td`Hmoh>DdGO0p0;`~zQti-`{Rdgbaj6SM@Z
z)m&K-vUVo*6%~5pRAnUn+TA3)0Hd~(F^|avA^AOYZT-D~m0P};ZRqozHf?Y~`E(LG
zH!af}3$&7}kCp{H?px`cCFUb|THJ$wzKh!F5>%XGC6jKI8m^TuGPUD#4k`$4$UZ3H
z(Hf7rA}|r39@ZI4+{h3ZM0_9TW#aOP_J1B~4qUos;etTWyp_vt=O?RBY6jA42hvEYq0z?ABB5yO|@FYi)8y?)ewx?1U(4&}*fjecAGAx;s
zZ|rYv%l)o8wpmk4mkRxm2-sI1oeb4Tov?Z?H*`V+a`?j#d{DW3#Rd&x2{DWSA5jB>
z{K1QUG~7-rJ3Q20SJYwCm~QreXIkA}KD@1B5h-KTZHy{T)>_6YjVHCf(pfG8
z-Z4!+Mw--5;ta$Y@Tb^(t`!%zk_JH)+V;K-W3($`yIVQ24vPaqcSQS`;$Bp}B-q2UXQ
zj``2_)CBe;nA{~G{lq&|SQ>h>28q#(N)OTQ!l-wTpJ**9`|~6$rKCf;nng<;LM#^f
z2WFs2Sl3=HFOz5zd_^^Xp7vzr&)XJh1Y7vc!9ZCh+vr%ySy}>oc7h{E5Xr6knPXh!e)zWfIyW
zwx+#St7~RyD*s??=KTnp6U8sd_ZtkmebkcTM74QEEC$}q0t)kgT=WaO49|WYFGg+@
zx6DHf=3kEaUtuInmG7+@qlpx7;Z=>Zp><22_+z1F^N2x7>BhvWEeyT>dKo6UB~azL
zZgHZ-RBai>R`+h%H;cgidYK`dct$*>nZj~Pa=_FF^Ywb@DXsP==b~@t)D&Muv8Lp)
zqUz!GotB~)+|;^%)*80$A9Y!1m8dq}RlmbA=a&}H!x%TD6DeLT!R~)up2K!r0D^zWGem_%-&zOAKjpLLaX%G!ndG
zFty$zjy_%triXEne2k7(-NCY0rjf6RmXvpFvf{Xd;-}1iE0vUqg+o}bzyXf=u@y?n
z=g(gVQJm~iJd1&k`DiJdmf(YSUQ%J53JYC)XMD;kg(ag!J{Ye_IG@jielPm7#$>cX
zZ$?BWe?=*Kx~b38KCjqFsV)Ba!um_zxhgpl+>U{+cpAB$ERC0lBaIIMm4+BAt4&5q8aHfxOA1)l5y
zJ=vr*oZ_)ZIvto(cBjpduu)TpS-WLJwJg?38v22jsk>?Ncgju)qZRSfo8pmQu{Yf4o-ka^T8~K=)b#^`glK18ab+#tOS767
zkB;Ymw8UeiG&f0UrZwX!XGs#k@}Hk<IdqD9
z&%SUpWg~~L*58QA6=Un|GFOHiHnHu0fI_vN&DI@4f-L9XhaNz0_tl06zln2p6rD}z
zE-`9BS2Fz_UF>IJPmHwb#euo%CBC&cFNEFhT47Vrq6Nr#!i%pv$KEq*G#2u^%5=mV
zXM{tSOon7SQ%Tm0JV?AY`!=eKRoSX3U3v#%*z47YzF%0`B}BP>?iEEQV2@RQADL9>
z*Qod;(36edi&Vz{e1p(N!Rb(R`?g>#OgSMhSHL+}JNNtJZg2KgfHH4+z8(W+=Ohmz
zo?8uWUeeT?1VrI@g$17%!st4$w?{9+VBS3m#y+Va)VW^6A$N%mj{f4hO6~?lBm5}-
zCo^GC>33yS0b@4l_1RW*KQq{Wrl+hE?JJLS+6wlK?^7E2DUOTZhHmx;&aPSH)%Ufv
zu0cMFaTnVu9NqKFU3M_^7xR%6dI@5nsiJNTtZMg-u3wn)Mp_
z74?MCEpTYYq6YFwbE8KLdsjb;+z*pV`bZ;V@Mfna%6uK|YZh=a=SPZvp>f?r6eb+7
zSAv=S)ilVG+UPu}{%bh)i}NEd{FOCwlecL0G3`4NsdwA?5@F^Kj@Iwb^OX^1)*8%e
zz=!^=yYU*C&!<6NX}GiphY~>PPYVSIn53zLV}lzi#_${x(!>UAa%s8o!oG;_=^9%@
ztW1|Qe!_L&4Ia%erg~+6>5Dv!yy`>iyH>q}QDM9xVLl&v9`sNa+|9Bl9Q16sD}Sup
z7AsD|$w)w=O`90C*~J2G_Cjf%vA#N?`Hnkla$}@3#XrPCUE_6nUKUL-E5f~Ez$=^K0qQ>Wq1$w(A`v|8U5TqJmyW^sd$
zpUm_cJya0f6LajUjL+oLuD~Asf&`Vzh_AFHs*ap8O>;xIfIibTTU6nV0iV|SYF<;MAG65Crvve|JTf*4x97b9spuT57*__%XMM&c
z<4DUKqA~tz$;&M@hxw{fVlUd}eYfRKyGsgeE7-T_Jd*MzO5N2wJ6(cjQBmig9_~tK
z(eL6t61#s$=_R7is)6;EhsoOX1jR_Ie*kh_u~Zj5?jtXM%YzLB7Tf%SBvM%%b+@Z}
z*rv?Kd$l#m>0h!&K!ihi4#RCHqXTX@Enbap0MQSbd}FeGuaa~9TguTFzC6h)yt8%5
zqQXzmi)OI@R_E`8`8L9#!AyYKO@2nnr1^%E)&C}@D+jj&P_4d-EYqCmRa~AEX4=7;
zLcqNK73On)E9{7jy6Q5Mef)SLfyzBeU&i?fdp$4IN6e@@FGKeqmGpj$yRSMKqTE=&
zP1=vlY$COklJD0^Ujo|RP$vt^D3of^7!u2|=r$YkMlQG&uZ;1)Coe8ve+!6|p9%An
z=^CCZjQ((>IUvVIU6m2YJ$U7pHbKEAeVg2%8g{XNm9B5&sM-UK81E}dvtZh!m}GGf
zY(VJW{G=Jc@l0V)w$6VOtmdKAsl&??0s6GySr!n3&ZwxoUHO({0BgC-tod
zFUQCgMq*}mSax(BI$hzoM!yJe^dC~;?>BYO%}SUO8$W_GH5sx9y57uvPw6ZN+8s2F
z%x|Z9FHgGL*|K$uZKeHea>e7j-7(s}wM6+8Us3lA
z%kKx)vDwSUXdpe{7BM%^qdd<`Giadf$yoJl{27}7kKW%;)=muZ+nasWV&-&AKAp;c
ze4gv!YCIdm`d`+arm1pX4z|30Pd7Ziw0?bf^cf)6HQnis>0Z!?0yM9s`E%?VMRRnpCZvY9eOB{m>s2}5axtBAo($xX=P826Kv7%+PJb#f)C24wFKZ)
z5=Nmk`5Stgj7Zl-f0?)M0xGS{&}D36~n(E+x@wb&5#A^n1Gglm35?dLc3I%>(Dj}bo1sH%r8SCD&@53!2Ww!j11_d
z;^iurlPs|7#F-<(#O+~2XS2ISB#-0uR!=YVvkwMx7#2z^{)0+Tr(2w~6QG$fc81s2
zE;td5t1M>59T~fRuvm_R5$U_fS~x5ckAWw%M|10tR$^MB_}Gjqn+Rodkp`O
z81PhxC*pI+7v&O>L?Lh*@Lo`{*LcR{mE>wlA#}Q;y&J7-L8Gz3=Tr|<~
z>wahj-mN+_vDt^SHG;7SS}B*Np93eyyF#3H
z)IXI?YCf`!`#)`eWoN;x_`MKb)ApV%z3__+A9ydr&gbr~@D%S3)pU5LHayVs*$r+H
z?q2E$OAYX^pY(^MFLX$-<&jnvPJimJLO5;;3U%Q)6v%v4TA?dHfh=!uM*&};VQz?5
zhesE#T`7lP?Md44GYcmt+N&Ua0^9=H_$%O;9ZjOEHbm>_&^kWb3#%c|m@
z`S#Nrd1X{K@!M9oTWit*>jt_P=3kC`K}Za{uFJ(bL7(4L^>^RaR=D!*wOoGQI+5gr
zS|)?a;|rjFC#$K8vrie_`*rPmXb8NP41sDQXa{`QE@nY$Sy+
z^Rk^aRJ(2!*1r5S17s3Xt$k5s2lA&1Q%v@~?2-Ki{}T*wmZBItDc6w
z^S|mIM>tj=+qzt9w4nFtrF|!d3CiA#ad{)IFbUg#sr|L0>~RiXC1(S(ID~nBUD*=P
zhKLM(ohkbg^gW+@(8EH4-6vk}1bu|ZNi09?lokPHXW1!zXW&I@Jl0kqRajHSiYq1k
zoI2mkgYF*DLum}CRp6lUjboy}p1s`Bs7Be2=e5l)`_~tfSzK=-(@WU#CQ1w^#jmdF
zT$GxBa^y>hHAhGieGPR)$4Yig|bS{<2D7TnfY!h{V9S6yImeFGEtNn2Kx|uvr;fQcris68#
zlnGF~TAo%>M}~D%C%dZLE%4`8VhF;
zHw@${dY6O092i9+K7ALH731urP~EeD`i8cHh4rm!R52juyTx&J!O3Kv5X$k+=9{yB
z0a=GBFc}-dbXJzZkWsR+!h=C9?ZYv(b#0zsYh)+q5R|8&(_0+r(2I#>?cuN30wXR%
zp`%eb{x0vbqlTuMd^vNDJUX-EI$u;29aaje(b1@8$_)e+eAk(%$YY>Fgf8W62E3~7
z&tOi-6p-Z!J2CEq!BZ9lHRo0^yC_S4=^nPpBQxIHTfrYX@7HGAm@<(2q+T`1w;k(R
zcPDQLDveUh^4+~n?!|R=SE4&vR>&RoL79uo*uWO`wAv^WVr
zGpDYPV8Tb_;gt%1uKGxc${w-E;&B;_(M(W7;LV7n8>XEU_!
z`yDaLE5OM`mQpcOLrIRak(D1~aV3qdY!`uJGPH3&7&*jMx9vm#Q99jVq_Fs^MNbOl
zMpI1Mu_NZiHx9*+@s|7Ko1Gyr>>VaUb?`b&nu7$7Q=*-LQS)^fpThTVrm#OJSyyxR
z=+V*-U&#i!&hG3QGIODSYG|K%tBT&=6WrqqY0`)ROvShd2O?FhKG_KqEdS_}nQUPH
zZa_L*3G1hdW6{02I0smr1jMgQMogfqhpI+?O||~9x$$B+{=?;waVUy~lLEDrX!GL<
zdvBN4U1px2N!cPtaZ`NKg=9PLiGd++_|3Vf)jOwGS7BqSEez~`I~)~{B^S*sQ?gMK
z7v69~^LAf$+Bp}dd)l8dz3tsEx5<=mjHR7lkzZPo&d(uw4tM4#+SQ{pe&&pnB;>h%
z(!6x2oj>1V1SiILQ^>6&8qwwCOic_+--3nZrFpV-xwU|Oi)mPu#*(#)g^Ji}6ICb!
z*L>xN-9+|StH0ELBRlB7cxil$!@z?6F&v(y+}c(;>5OA
zO+awpdpfa_x4{_$s*WxL1uHk>$S4iFD0BH0C;`Y$QfBX1e9+CNZ^{ISc6<8L)o_z>
z+9XNsmah_&z@uwxA+XceH}B
z6$JGF(9$ELc1J_IOEShY9X|aWSM#C%x<;K)#l3i;{a!n7WMw4!^4c~lwGibz$oj@y
z;Y%?B%g3SGIe%vrktM&<`RAeugSNNv!4v4Z^cBFd?VQXCH-Xf*qbwMnjwAi57f
zM(sd6V}YL4VPnx(dpV^P=l%)#fehjHaaDq7>`{#>`}#%{8Mw2`j287n-~VRmeg8lr
zC{!H7Y3?Uql{yHAzDyujfN&R{Ip4Y&RWU~HEKrDlH>k>yWBA2j0Rx9?B0b$@
z{1gWCAxpsx^#0LsYbPI}0t8W%I$3WsXMEQmahxHOPCrjl-Shd`FGjEq%{!kgG*?J>
zs2Hg0YyCb9f6LI6$NnX5AkE?ZvZeOW^MVl~XF>n^RU;CsB-=Eq`}Y(pac|#PoHr4+
zHr#A~T`;osfvru-NkXBLTR>;C#NsHHe*1go7zF>^u{&uo-@&lzdgyEv#JhT=rd@>x1$7p}NN*#>x^JlKytka@sFre6IPgD&|XZyNP
zZ%V2cMOg7CI$K!E1Z_L@z1l|hrY=EDT#TbO*s>axVFxW=^lr{Sq|Ef&-Zh>5_u#JaCO*kM$}Q$;8}$C^9%VU_DN7P9PUhq!WUc?kSmdcY
zOp7mUJap_4mWs}$}NRdkurm^&Y+wLEH
zW#-tR>8JV$jan~8X`goG#Rnaj3?rj&G){kIOSFI)!kIqYPM{ApohB(l^RYELNKxRE
zNH2B4+VkPyjasE%B7PdTUjI&-T+KL}bdE$aAM;`CG}$FxgY`Z8q`eyIXK`_Tcv@K@
zHF*FjZ*y3gsh-h0zWUBA4$qo@D=y@_4}P0UIx`JlgXup}5N`#sy{x~*>Opu}6l4&c
zO9HR3xDRr2Ue$(J9(><-{Hl}j`3db&&2|)nNwCGqZq2@7AY?~zVf8LJ|4Zg4k)uS4
zK~S%eXV2)(7juxEOoobFOJEq8P&cxCR4a&0ji9${!
zYE=b>ziKU`6RNBus;|pq?n-Y0&FVC8Xo+m9PR28#V572l*7IAzdRqirBwL{0v3BM6
zuXgN+MJAL1KXTwK@E3GaZ_7QS-2JWR7J*ZnswQ`(j;yjG1QN>Zti<*QlCJJRTwd&J
zwhO2mCAyBS%m^wO@L;7&48JYv4;eVV&L_!&@Hr|t3$%cS#x@?BDclz$Z|_Ah%4s5j#KGTTrgSnFY2`lkF19cwl4=0kooKP
z5797^sEvvARWJ*6x&6F%60MuY(YS*PjYWM6@GIoeJUYg0ClKdH5mEz(V_
z!;7{o1d=VY@%NkW%0AzKSuE*v=!i(KGfCe>>pQ&Bx0lzxTJ(pP)^OOaE5k6?{bR+)ul|s6;5waZPyoUWI8{tjw4aOZaz;6bv1AA`mEwL
z0bSueO3Qw>hIG5_AMG)B)ECiN&j1QsiE1jHBlY=+Ds7s7pv}stknJw^8E#6LSQQ<9
z@SBpqWkt-+;*v!GpuI+m{XyCFz+prpXb!TzD=3E
zXnF>l!%!ht*?cnJ=L9bXU2N9;i1)sGGQIIm3-5eVzrNvmC!)jkG_%yMn5rI;bWs{g
zu7MJRQq_!qretcAhp1BlWHJiw@^J8EGs$}QT-ZiDj_0FD+)MbLUis^ZAef3S>2?{b
z#*GTrFzYO?YoT-bQ?#(L6>}ds`c~=!o$GfFOQw^iXS6k+5#0kU*nTF#4=S>p#9v)@
zv_3geDI6-9G!FE#5g%-+OCt^J41>x>@2H29rLbjxr`)$4B`FIP4?Ox*j&VHr7}yWL
z--!rpwxTiJeb&d-o=vDQG)p~($x*Uzy9Dh-Ag_!wEbjHktL^KkospUgB#sNQdW)oc
zlblJ!wW^X*jz2Cheb+Wh#6^{#7w$yQ{Mso083t=(+l&Xv)GQ!I+GV8e^S2!N8diH7
zx=!DJH$veo*rf+K0dJEm2a5KL*$2WL5}oAs9aan*+NZaIkb@NFKz=PXb@Yo^rw=t6
zq|?NkiXwdv3q0_lcZ5Vqq+6gfs1!P?E}?%!noow3%;-Nxi38bVq~E9Q;?y
zU%SZaG?B*R~hA`m}O6Gh+*KNu#Vek>;N2q>n7$pEJjed#hhfnh22h)l
zSnW!_Cl*%{>3HkkKSAF;FWp06voG}0rJMK+$J4be{c3z=`L>)UMWg6E=B9RTUGKPm
zh}pRbAvgP;{p|(uuIqdBtK%Rkq8H5ZlQdfu<9pK{#TV7$PE!$5)nYk0yD9lgtU>Oj
ze&JaE6wOyoE3il;a|+#=&(i1mf{OUQH5%^K)_0A`nl!WuG|t`@0}S$?qA%c-(EA%J
zeFf@v)kSLz-6>9@kQ$V8M;CZ`ao$3JFZbK%pi*i(_5Tt;;oMwvyR3WvmMRfUL^tpM$7>TCBl7
zV-A;*A4V)?J>@#Xti>wT)&KgD+_#^T%889|z`KmFjxiH0ta!*i-lk#<3R)EJyMKjI
z5qILMwC$$|t)IXtg|Y4?mIYcHgU%9Ifzh+4CXn-zzCvY)Ok!=wsCmh>pH_$)fLNY!
z@$uN{+8I+@zNMvqjeKRTn!8|s6a^2m|f
z5RqFz+^hn1Me{D}aG`34ae*`N_yVoUX9^Y9GB=Hv3vEUiWsQ0yl0QsA1ovh9yHGn#
zitaf*aC%ybC7ZxaYu>nT-)&$2`wG}4{70!sCEw=~j0K|wG&qeV!<<`x=ub64D=h
 
+
+

+
+
 
 
 
 
+
+
+
+
+
Set up of an RNN 

+
+

+The figure here displays a simple example of an RNN, with inputs \( x_t \)
+at a given time \( t \) and outputs \( y_t \).  Introducing time as a variable
+offers an intutitive way of understanding these networks. In addition
+to the inputs \( x_t \), the layer at a time \( t \) receives also as input
+the output from the previous layer \( t-1 \), that is \( y_{t1} \).
+
+

+This means also that we need to have weights that link both the inputs \( x_t \) to the outputs \( y_t \) as well as weights that link
+the output from the previous time \( y_{t-1} \) and \( y_t \).   The figure here shows an example of a simple RNN.
+
+

+
+

+
+
+
  
+
+
+
+
+
+
+
+

+
+

+
+
+
+
+    
+
diff --git a/doc/pub/Recurrent/html/reveal.js/plugin/leap/leap.js b/doc/pub/Recurrent/html/reveal.js/plugin/leap/leap.js
new file mode 100644
index 000000000..48084ffb0
--- /dev/null
+++ b/doc/pub/Recurrent/html/reveal.js/plugin/leap/leap.js
@@ -0,0 +1,159 @@
+/*
+ * Copyright (c) 2013, Leap Motion, Inc.
+ * All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:
+ *
+ * Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.
+ * Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+ *
+ * Version 0.2.0 - http://js.leapmotion.com/0.2.0/leap.min.js
+ * Grab latest versions from http://js.leapmotion.com/
+ */
+
+!function(e,t,n){function i(n,s){if(!t[n]){if(!e[n]){var o=typeof require=="function"&&require;if(!s&&o)return o(n,!0);if(r)return r(n,!0);throw new Error("Cannot find module '"+n+"'")}var u=t[n]={exports:{}};e[n][0].call(u.exports,function(t){var r=e[n][1][t];return i(r?r:t)},u,u.exports)}return t[n].exports}var r=typeof require=="function"&&require;for(var s=0;s=this.size)return undefined;if(i>=this._buf.length)return undefined;return this._buf[(this.pos-i-1)%this.size]};CircularBuffer.prototype.push=function(o){this._buf[this.pos%this.size]=o;return this.pos++}},{}],3:[function(require,module,exports){var Connection=module.exports=require("./base_connection");Connection.prototype.setupSocket=function(){var connection=this;var socket=new WebSocket(this.getUrl());socket.onopen=function(){connection.handleOpen()};socket.onmessage=function(message){connection.handleData(message.data)};socket.onclose=function(){connection.handleClose()};return socket};Connection.prototype.startHeartbeat=function(){if(!this.protocol.sendHeartbeat||this.heartbeatTimer)return;var connection=this;var propertyName=null;if(typeof document.hidden!=="undefined"){propertyName="hidden"}else if(typeof document.mozHidden!=="undefined"){propertyName="mozHidden"}else if(typeof document.msHidden!=="undefined"){propertyName="msHidden"}else if(typeof document.webkitHidden!=="undefined"){propertyName="webkitHidden"}else{propertyName=undefined}var windowVisible=true;var focusListener=window.addEventListener("focus",function(e){windowVisible=true});var blurListener=window.addEventListener("blur",function(e){windowVisible=false});this.on("disconnect",function(){if(connection.heartbeatTimer){clearTimeout(connection.heartbeatTimer);delete connection.heartbeatTimer}window.removeEventListener(focusListener);window.removeEventListener(blurListener)});this.heartbeatTimer=setInterval(function(){var isVisible=propertyName===undefined?true:document[propertyName]===false;if(isVisible&&windowVisible){connection.sendHeartbeat()}else{connection.setHeartbeatState(false)}},this.opts.heartbeatInterval)}},{"./base_connection":1}],4:[function(require,module,exports){!function(process){var Frame=require("./frame"),CircularBuffer=require("./circular_buffer"),Pipeline=require("./pipeline"),EventEmitter=require("events").EventEmitter,gestureListener=require("./gesture").gestureListener,_=require("underscore");var Controller=module.exports=function(opts){var inNode=typeof process!=="undefined"&&process.title==="node";opts=_.defaults(opts||{},{inNode:inNode});this.inNode=opts.inNode;opts=_.defaults(opts||{},{frameEventName:this.useAnimationLoop()?"animationFrame":"deviceFrame",supressAnimationLoop:false});this.supressAnimationLoop=opts.supressAnimationLoop;this.frameEventName=opts.frameEventName;this.history=new CircularBuffer(200);this.lastFrame=Frame.Invalid;this.lastValidFrame=Frame.Invalid;this.lastConnectionFrame=Frame.Invalid;this.accumulatedGestures=[];if(opts.connectionType===undefined){this.connectionType=this.inBrowser()?require("./connection"):require("./node_connection")}else{this.connectionType=opts.connectionType}this.connection=new this.connectionType(opts);this.setupConnectionEvents()};Controller.prototype.gesture=function(type,cb){var creator=gestureListener(this,type);if(cb!==undefined){creator.stop(cb)}return creator};Controller.prototype.inBrowser=function(){return!this.inNode};Controller.prototype.useAnimationLoop=function(){return this.inBrowser()&&typeof chrome==="undefined"};Controller.prototype.connect=function(){var controller=this;if(this.connection.connect()&&this.inBrowser()&&!controller.supressAnimationLoop){var callback=function(){controller.emit("animationFrame",controller.lastConnectionFrame);window.requestAnimFrame(callback)};window.requestAnimFrame(callback)}};Controller.prototype.disconnect=function(){this.connection.disconnect()};Controller.prototype.frame=function(num){return this.history.get(num)||Frame.Invalid};Controller.prototype.loop=function(callback){switch(callback.length){case 1:this.on(this.frameEventName,callback);break;case 2:var controller=this;var scheduler=null;var immediateRunnerCallback=function(frame){callback(frame,function(){if(controller.lastFrame!=frame){immediateRunnerCallback(controller.lastFrame)}else{controller.once(controller.frameEventName,immediateRunnerCallback)}})};this.once(this.frameEventName,immediateRunnerCallback);break}this.connect()};Controller.prototype.addStep=function(step){if(!this.pipeline)this.pipeline=new Pipeline(this);this.pipeline.addStep(step)};Controller.prototype.processFrame=function(frame){if(frame.gestures){this.accumulatedGestures=this.accumulatedGestures.concat(frame.gestures)}if(this.pipeline){frame=this.pipeline.run(frame);if(!frame)frame=Frame.Invalid}this.lastConnectionFrame=frame;this.emit("deviceFrame",frame)};Controller.prototype.processFinishedFrame=function(frame){this.lastFrame=frame;if(frame.valid){this.lastValidFrame=frame}frame.controller=this;frame.historyIdx=this.history.push(frame);if(frame.gestures){frame.gestures=this.accumulatedGestures;this.accumulatedGestures=[];for(var gestureIdx=0;gestureIdx!=frame.gestures.length;gestureIdx++){this.emit("gesture",frame.gestures[gestureIdx],frame)}}this.emit("frame",frame)};Controller.prototype.setupConnectionEvents=function(){var controller=this;this.connection.on("frame",function(frame){controller.processFrame(frame)});this.on(this.frameEventName,function(frame){controller.processFinishedFrame(frame)});this.connection.on("disconnect",function(){controller.emit("disconnect")});this.connection.on("ready",function(){controller.emit("ready")});this.connection.on("connect",function(){controller.emit("connect")});this.connection.on("focus",function(){controller.emit("focus")});this.connection.on("blur",function(){controller.emit("blur")});this.connection.on("protocol",function(protocol){controller.emit("protocol",protocol)});this.connection.on("deviceConnect",function(evt){controller.emit(evt.state?"deviceConnected":"deviceDisconnected")})};_.extend(Controller.prototype,EventEmitter.prototype)}(require("__browserify_process"))},{"./circular_buffer":2,"./connection":3,"./frame":5,"./gesture":6,"./node_connection":16,"./pipeline":10,__browserify_process:18,events:17,underscore:20}],5:[function(require,module,exports){var Hand=require("./hand"),Pointable=require("./pointable"),createGesture=require("./gesture").createGesture,glMatrix=require("gl-matrix"),mat3=glMatrix.mat3,vec3=glMatrix.vec3,InteractionBox=require("./interaction_box"),_=require("underscore");var Frame=module.exports=function(data){this.valid=true;this.id=data.id;this.timestamp=data.timestamp;this.hands=[];this.handsMap={};this.pointables=[];this.tools=[];this.fingers=[];if(data.interactionBox){this.interactionBox=new InteractionBox(data.interactionBox)}this.gestures=[];this.pointablesMap={};this._translation=data.t;this._rotation=_.flatten(data.r);this._scaleFactor=data.s;this.data=data;this.type="frame";this.currentFrameRate=data.currentFrameRate;var handMap={};for(var handIdx=0,handCount=data.hands.length;handIdx!=handCount;handIdx++){var hand=new Hand(data.hands[handIdx]);hand.frame=this;this.hands.push(hand);this.handsMap[hand.id]=hand;handMap[hand.id]=handIdx}for(var pointableIdx=0,pointableCount=data.pointables.length;pointableIdx!=pointableCount;pointableIdx++){var pointable=new Pointable(data.pointables[pointableIdx]);pointable.frame=this;this.pointables.push(pointable);this.pointablesMap[pointable.id]=pointable;(pointable.tool?this.tools:this.fingers).push(pointable);if(pointable.handId!==undefined&&handMap.hasOwnProperty(pointable.handId)){var hand=this.hands[handMap[pointable.handId]];hand.pointables.push(pointable);(pointable.tool?hand.tools:hand.fingers).push(pointable)}}if(data.gestures){for(var gestureIdx=0,gestureCount=data.gestures.length;gestureIdx!=gestureCount;gestureIdx++){this.gestures.push(createGesture(data.gestures[gestureIdx]))}}};Frame.prototype.tool=function(id){var pointable=this.pointable(id);return pointable.tool?pointable:Pointable.Invalid};Frame.prototype.pointable=function(id){return this.pointablesMap[id]||Pointable.Invalid};Frame.prototype.finger=function(id){var pointable=this.pointable(id);return!pointable.tool?pointable:Pointable.Invalid};Frame.prototype.hand=function(id){return this.handsMap[id]||Hand.Invalid};Frame.prototype.rotationAngle=function(sinceFrame,axis){if(!this.valid||!sinceFrame.valid)return 0;var rot=this.rotationMatrix(sinceFrame);var cs=(rot[0]+rot[4]+rot[8]-1)*.5;var angle=Math.acos(cs);angle=isNaN(angle)?0:angle;if(axis!==undefined){var rotAxis=this.rotationAxis(sinceFrame);angle*=vec3.dot(rotAxis,vec3.normalize(vec3.create(),axis))}return angle};Frame.prototype.rotationAxis=function(sinceFrame){if(!this.valid||!sinceFrame.valid)return vec3.create();return vec3.normalize(vec3.create(),[this._rotation[7]-sinceFrame._rotation[5],this._rotation[2]-sinceFrame._rotation[6],this._rotation[3]-sinceFrame._rotation[1]])};Frame.prototype.rotationMatrix=function(sinceFrame){if(!this.valid||!sinceFrame.valid)return mat3.create();var transpose=mat3.transpose(mat3.create(),this._rotation);return mat3.multiply(mat3.create(),sinceFrame._rotation,transpose)};Frame.prototype.scaleFactor=function(sinceFrame){if(!this.valid||!sinceFrame.valid)return 1;return Math.exp(this._scaleFactor-sinceFrame._scaleFactor)};Frame.prototype.translation=function(sinceFrame){if(!this.valid||!sinceFrame.valid)return vec3.create();return vec3.subtract(vec3.create(),this._translation,sinceFrame._translation)};Frame.prototype.toString=function(){var str="Frame [ id:"+this.id+" | timestamp:"+this.timestamp+" | Hand count:("+this.hands.length+") | Pointable count:("+this.pointables.length+")";if(this.gestures)str+=" | Gesture count:("+this.gestures.length+")";str+=" ]";return str};Frame.prototype.dump=function(){var out="";out+="Frame Info:
";out+=this.toString();out+="

Hands:
";for(var handIdx=0,handCount=this.hands.length;handIdx!=handCount;handIdx++){out+="  "+this.hands[handIdx].toString()+"
"}out+="

Pointables:
";for(var pointableIdx=0,pointableCount=this.pointables.length;pointableIdx!=pointableCount;pointableIdx++){out+="  "+this.pointables[pointableIdx].toString()+"
"}if(this.gestures){out+="

Gestures:
";for(var gestureIdx=0,gestureCount=this.gestures.length;gestureIdx!=gestureCount;gestureIdx++){out+="  "+this.gestures[gestureIdx].toString()+"
"}}out+="

Raw JSON:
";out+=JSON.stringify(this.data);return out};Frame.Invalid={valid:false,hands:[],fingers:[],tools:[],gestures:[],pointables:[],pointable:function(){return Pointable.Invalid},finger:function(){return Pointable.Invalid},hand:function(){return Hand.Invalid},toString:function(){return"invalid frame"},dump:function(){return this.toString()},rotationAngle:function(){return 0},rotationMatrix:function(){return mat3.create()},rotationAxis:function(){return vec3.create()},scaleFactor:function(){return 1},translation:function(){return vec3.create()}}},{"./gesture":6,"./hand":7,"./interaction_box":9,"./pointable":11,"gl-matrix":19,underscore:20}],6:[function(require,module,exports){var glMatrix=require("gl-matrix"),vec3=glMatrix.vec3,EventEmitter=require("events").EventEmitter,_=require("underscore");var createGesture=exports.createGesture=function(data){var gesture;switch(data.type){case"circle":gesture=new CircleGesture(data);break;case"swipe":gesture=new SwipeGesture(data);break;case"screenTap":gesture=new ScreenTapGesture(data);break;case"keyTap":gesture=new KeyTapGesture(data);break;default:throw"unkown gesture type"}gesture.id=data.id;gesture.handIds=data.handIds;gesture.pointableIds=data.pointableIds;gesture.duration=data.duration;gesture.state=data.state;gesture.type=data.type;return gesture};var gestureListener=exports.gestureListener=function(controller,type){var handlers={};var gestureMap={};var gestureCreator=function(){var candidateGesture=gestureMap[gesture.id];if(candidateGesture!==undefined)gesture.update(gesture,frame);if(gesture.state=="start"||gesture.state=="stop"){if(type==gesture.type&&gestureMap[gesture.id]===undefined){gestureMap[gesture.id]=new Gesture(gesture,frame);gesture.update(gesture,frame)}if(gesture.state=="stop"){delete gestureMap[gesture.id]}}};controller.on("gesture",function(gesture,frame){if(gesture.type==type){if(gesture.state=="start"||gesture.state=="stop"){if(gestureMap[gesture.id]===undefined){var gestureTracker=new Gesture(gesture,frame);gestureMap[gesture.id]=gestureTracker;_.each(handlers,function(cb,name){gestureTracker.on(name,cb)})}}gestureMap[gesture.id].update(gesture,frame);if(gesture.state=="stop"){delete gestureMap[gesture.id]}}});var builder={start:function(cb){handlers["start"]=cb;return builder},stop:function(cb){handlers["stop"]=cb;return builder},complete:function(cb){handlers["stop"]=cb;return builder},update:function(cb){handlers["update"]=cb;return builder}};return builder};var Gesture=exports.Gesture=function(gesture,frame){this.gestures=[gesture];this.frames=[frame]};Gesture.prototype.update=function(gesture,frame){this.gestures.push(gesture);this.frames.push(frame);this.emit(gesture.state,this)};_.extend(Gesture.prototype,EventEmitter.prototype);var CircleGesture=function(data){this.center=data.center;this.normal=data.normal;this.progress=data.progress;this.radius=data.radius};CircleGesture.prototype.toString=function(){return"CircleGesture ["+JSON.stringify(this)+"]"};var SwipeGesture=function(data){this.startPosition=data.startPosition;this.position=data.position;this.direction=data.direction;this.speed=data.speed};SwipeGesture.prototype.toString=function(){return"SwipeGesture ["+JSON.stringify(this)+"]"};var ScreenTapGesture=function(data){this.position=data.position;this.direction=data.direction;this.progress=data.progress};ScreenTapGesture.prototype.toString=function(){return"ScreenTapGesture ["+JSON.stringify(this)+"]"};var KeyTapGesture=function(data){this.position=data.position;this.direction=data.direction;this.progress=data.progress};KeyTapGesture.prototype.toString=function(){return"KeyTapGesture ["+JSON.stringify(this)+"]"}},{events:17,"gl-matrix":19,underscore:20}],7:[function(require,module,exports){var Pointable=require("./pointable"),glMatrix=require("gl-matrix"),mat3=glMatrix.mat3,vec3=glMatrix.vec3,_=require("underscore");var Hand=module.exports=function(data){this.id=data.id;this.palmPosition=data.palmPosition;this.direction=data.direction;this.palmVelocity=data.palmVelocity;this.palmNormal=data.palmNormal;this.sphereCenter=data.sphereCenter;this.sphereRadius=data.sphereRadius;this.valid=true;this.pointables=[];this.fingers=[];this.tools=[];this._translation=data.t;this._rotation=_.flatten(data.r);this._scaleFactor=data.s;this.timeVisible=data.timeVisible;this.stabilizedPalmPosition=data.stabilizedPalmPosition};Hand.prototype.finger=function(id){var finger=this.frame.finger(id);return finger&&finger.handId==this.id?finger:Pointable.Invalid};Hand.prototype.rotationAngle=function(sinceFrame,axis){if(!this.valid||!sinceFrame.valid)return 0;var sinceHand=sinceFrame.hand(this.id);if(!sinceHand.valid)return 0;var rot=this.rotationMatrix(sinceFrame);var cs=(rot[0]+rot[4]+rot[8]-1)*.5;var angle=Math.acos(cs);angle=isNaN(angle)?0:angle;if(axis!==undefined){var rotAxis=this.rotationAxis(sinceFrame);angle*=vec3.dot(rotAxis,vec3.normalize(vec3.create(),axis))}return angle};Hand.prototype.rotationAxis=function(sinceFrame){if(!this.valid||!sinceFrame.valid)return vec3.create();var sinceHand=sinceFrame.hand(this.id);if(!sinceHand.valid)return vec3.create();return vec3.normalize(vec3.create(),[this._rotation[7]-sinceHand._rotation[5],this._rotation[2]-sinceHand._rotation[6],this._rotation[3]-sinceHand._rotation[1]])};Hand.prototype.rotationMatrix=function(sinceFrame){if(!this.valid||!sinceFrame.valid)return mat3.create();var sinceHand=sinceFrame.hand(this.id);if(!sinceHand.valid)return mat3.create();var transpose=mat3.transpose(mat3.create(),this._rotation);var m=mat3.multiply(mat3.create(),sinceHand._rotation,transpose);return m};Hand.prototype.scaleFactor=function(sinceFrame){if(!this.valid||!sinceFrame.valid)return 1;var sinceHand=sinceFrame.hand(this.id);if(!sinceHand.valid)return 1;return Math.exp(this._scaleFactor-sinceHand._scaleFactor)};Hand.prototype.translation=function(sinceFrame){if(!this.valid||!sinceFrame.valid)return vec3.create();var sinceHand=sinceFrame.hand(this.id);if(!sinceHand.valid)return vec3.create();return[this._translation[0]-sinceHand._translation[0],this._translation[1]-sinceHand._translation[1],this._translation[2]-sinceHand._translation[2]]};Hand.prototype.toString=function(){return"Hand [ id: "+this.id+" | palm velocity:"+this.palmVelocity+" | sphere center:"+this.sphereCenter+" ] "};Hand.Invalid={valid:false,fingers:[],tools:[],pointables:[],pointable:function(){return Pointable.Invalid},finger:function(){return Pointable.Invalid},toString:function(){return"invalid frame"},dump:function(){return this.toString()},rotationAngle:function(){return 0},rotationMatrix:function(){return mat3.create()},rotationAxis:function(){return vec3.create()},scaleFactor:function(){return 1},translation:function(){return vec3.create()}}},{"./pointable":11,"gl-matrix":19,underscore:20}],8:[function(require,module,exports){!function(){module.exports={Controller:require("./controller"),Frame:require("./frame"),Gesture:require("./gesture"),Hand:require("./hand"),Pointable:require("./pointable"),InteractionBox:require("./interaction_box"),Connection:require("./connection"),CircularBuffer:require("./circular_buffer"),UI:require("./ui"),glMatrix:require("gl-matrix"),mat3:require("gl-matrix").mat3,vec3:require("gl-matrix").vec3,loopController:undefined,loop:function(opts,callback){if(callback===undefined){callback=opts;opts={}}if(!this.loopController)this.loopController=new this.Controller(opts);this.loopController.loop(callback)}}}()},{"./circular_buffer":2,"./connection":3,"./controller":4,"./frame":5,"./gesture":6,"./hand":7,"./interaction_box":9,"./pointable":11,"./ui":13,"gl-matrix":19}],9:[function(require,module,exports){var glMatrix=require("gl-matrix"),vec3=glMatrix.vec3;var InteractionBox=module.exports=function(data){this.valid=true;this.center=data.center;this.size=data.size;this.width=data.size[0];this.height=data.size[1];this.depth=data.size[2]};InteractionBox.prototype.denormalizePoint=function(normalizedPosition){return vec3.fromValues((normalizedPosition[0]-.5)*this.size[0]+this.center[0],(normalizedPosition[1]-.5)*this.size[1]+this.center[1],(normalizedPosition[2]-.5)*this.size[2]+this.center[2])};InteractionBox.prototype.normalizePoint=function(position,clamp){var vec=vec3.fromValues((position[0]-this.center[0])/this.size[0]+.5,(position[1]-this.center[1])/this.size[1]+.5,(position[2]-this.center[2])/this.size[2]+.5);if(clamp){vec[0]=Math.min(Math.max(vec[0],0),1);vec[1]=Math.min(Math.max(vec[1],0),1);vec[2]=Math.min(Math.max(vec[2],0),1)}return vec};InteractionBox.prototype.toString=function(){return"InteractionBox [ width:"+this.width+" | height:"+this.height+" | depth:"+this.depth+" ]"};InteractionBox.Invalid={valid:false}},{"gl-matrix":19}],10:[function(require,module,exports){var Pipeline=module.exports=function(){this.steps=[]};Pipeline.prototype.addStep=function(step){this.steps.push(step)};Pipeline.prototype.run=function(frame){var stepsLength=this.steps.length;for(var i=0;i!=stepsLength;i++){if(!frame)break;frame=this.steps[i](frame)}return frame}},{}],11:[function(require,module,exports){var glMatrix=require("gl-matrix"),vec3=glMatrix.vec3;var Pointable=module.exports=function(data){this.valid=true;this.id=data.id;this.handId=data.handId;this.length=data.length;this.tool=data.tool;this.width=data.width;this.direction=data.direction;this.stabilizedTipPosition=data.stabilizedTipPosition;this.tipPosition=data.tipPosition;this.tipVelocity=data.tipVelocity;this.touchZone=data.touchZone;this.touchDistance=data.touchDistance;this.timeVisible=data.timeVisible};Pointable.prototype.toString=function(){if(this.tool==true){return"Pointable [ id:"+this.id+" "+this.length+"mmx | with:"+this.width+"mm | direction:"+this.direction+" ]"}else{return"Pointable [ id:"+this.id+" "+this.length+"mmx | direction: "+this.direction+" ]"}};Pointable.Invalid={valid:false}},{"gl-matrix":19}],12:[function(require,module,exports){var Frame=require("./frame");var Event=function(data){this.type=data.type;this.state=data.state};var chooseProtocol=exports.chooseProtocol=function(header){var protocol;switch(header.version){case 1:protocol=JSONProtocol(1,function(data){return new Frame(data)});break;case 2:protocol=JSONProtocol(2,function(data){return new Frame(data)});protocol.sendHeartbeat=function(connection){connection.send(protocol.encode({heartbeat:true}))};break;case 3:protocol=JSONProtocol(3,function(data){return data.event?new Event(data.event):new Frame(data)});protocol.sendHeartbeat=function(connection){connection.send(protocol.encode({heartbeat:true}))};break;default:throw"unrecognized version"}return protocol};var JSONProtocol=function(version,cb){var protocol=cb;protocol.encode=function(message){return JSON.stringify(message)};protocol.version=version;protocol.versionLong="Version "+version;protocol.type="protocol";return protocol}},{"./frame":5}],13:[function(require,module,exports){exports.UI={Region:require("./ui/region"),Cursor:require("./ui/cursor")}},{"./ui/cursor":14,"./ui/region":15}],14:[function(require,module,exports){var Cursor=module.exports=function(){return function(frame){var pointable=frame.pointables.sort(function(a,b){return a.z-b.z})[0];if(pointable&&pointable.valid){frame.cursorPosition=pointable.tipPosition}return frame}}},{}],15:[function(require,module,exports){var EventEmitter=require("events").EventEmitter,_=require("underscore");var Region=module.exports=function(start,end){this.start=new Vector(start);this.end=new Vector(end);this.enteredFrame=null};Region.prototype.hasPointables=function(frame){for(var i=0;i!=frame.pointables.length;i++){var position=frame.pointables[i].tipPosition;if(position.x>=this.start.x&&position.x<=this.end.x&&position.y>=this.start.y&&position.y<=this.end.y&&position.z>=this.start.z&&position.z<=this.end.z){return true}}return false};Region.prototype.listener=function(opts){var region=this;if(opts&&opts.nearThreshold)this.setupNearRegion(opts.nearThreshold);return function(frame){return region.updatePosition(frame)}};Region.prototype.clipper=function(){var region=this;return function(frame){region.updatePosition(frame);return region.enteredFrame?frame:null}};Region.prototype.setupNearRegion=function(distance){var nearRegion=this.nearRegion=new Region([this.start.x-distance,this.start.y-distance,this.start.z-distance],[this.end.x+distance,this.end.y+distance,this.end.z+distance]);var region=this;nearRegion.on("enter",function(frame){region.emit("near",frame)});nearRegion.on("exit",function(frame){region.emit("far",frame)});region.on("exit",function(frame){region.emit("near",frame)})};Region.prototype.updatePosition=function(frame){if(this.nearRegion)this.nearRegion.updatePosition(frame);if(this.hasPointables(frame)&&this.enteredFrame==null){this.enteredFrame=frame;this.emit("enter",this.enteredFrame)}else if(!this.hasPointables(frame)&&this.enteredFrame!=null){this.enteredFrame=null;this.emit("exit",this.enteredFrame)}return frame};Region.prototype.normalize=function(position){return new Vector([(position.x-this.start.x)/(this.end.x-this.start.x),(position.y-this.start.y)/(this.end.y-this.start.y),(position.z-this.start.z)/(this.end.z-this.start.z)])};Region.prototype.mapToXY=function(position,width,height){var normalized=this.normalize(position);var x=normalized.x,y=normalized.y;if(x>1)x=1;else if(x<-1)x=-1;if(y>1)y=1;else if(y<-1)y=-1;return[(x+1)/2*width,(1-y)/2*height,normalized.z]};_.extend(Region.prototype,EventEmitter.prototype)},{events:17,underscore:20}],16:[function(require,module,exports){},{}],17:[function(require,module,exports){!function(process){if(!process.EventEmitter)process.EventEmitter=function(){};var EventEmitter=exports.EventEmitter=process.EventEmitter;var isArray=typeof Array.isArray==="function"?Array.isArray:function(xs){return Object.prototype.toString.call(xs)==="[object Array]"};function indexOf(xs,x){if(xs.indexOf)return xs.indexOf(x);for(var i=0;i0&&this._events[type].length>m){this._events[type].warned=true;console.error("(node) warning: possible EventEmitter memory "+"leak detected. %d listeners added. "+"Use emitter.setMaxListeners() to increase limit.",this._events[type].length);console.trace()}}this._events[type].push(listener)}else{this._events[type]=[this._events[type],listener]}return this};EventEmitter.prototype.on=EventEmitter.prototype.addListener;EventEmitter.prototype.once=function(type,listener){var self=this;self.on(type,function g(){self.removeListener(type,g);listener.apply(this,arguments)});return this};EventEmitter.prototype.removeListener=function(type,listener){if("function"!==typeof listener){throw new Error("removeListener only takes instances of Function")}if(!this._events||!this._events[type])return this;var list=this._events[type];if(isArray(list)){var i=indexOf(list,listener);if(i<0)return this;list.splice(i,1);if(list.length==0)delete this._events[type]}else if(this._events[type]===listener){delete this._events[type]}return this};EventEmitter.prototype.removeAllListeners=function(type){if(arguments.length===0){this._events={};return this}if(type&&this._events&&this._events[type])this._events[type]=null;return this};EventEmitter.prototype.listeners=function(type){if(!this._events)this._events={};if(!this._events[type])this._events[type]=[];if(!isArray(this._events[type])){this._events[type]=[this._events[type]]}return this._events[type]}}(require("__browserify_process"))},{__browserify_process:18}],18:[function(require,module,exports){var process=module.exports={};process.nextTick=function(){var canSetImmediate=typeof window!=="undefined"&&window.setImmediate;var canPost=typeof window!=="undefined"&&window.postMessage&&window.addEventListener;if(canSetImmediate){return function(f){return window.setImmediate(f)}}if(canPost){var queue=[];window.addEventListener("message",function(ev){if(ev.source===window&&ev.data==="process-tick"){ev.stopPropagation();if(queue.length>0){var fn=queue.shift();fn()}}},true);return function nextTick(fn){queue.push(fn);window.postMessage("process-tick","*")}}return function nextTick(fn){setTimeout(fn,0)}}();process.title="browser";process.browser=true;process.env={};process.argv=[];process.binding=function(name){throw new Error("process.binding is not supported")};process.cwd=function(){return"/"};process.chdir=function(dir){throw new Error("process.chdir is not supported")}},{}],19:[function(require,module,exports){!function(){!function(){"use strict";var shim={};if(typeof exports==="undefined"){if(typeof define=="function"&&typeof define.amd=="object"&&define.amd){shim.exports={};define(function(){return shim.exports})}else{shim.exports=window}}else{shim.exports=exports}!function(exports){var vec2={};if(!GLMAT_EPSILON){var GLMAT_EPSILON=1e-6}vec2.create=function(){return new Float32Array(2)};vec2.clone=function(a){var out=new Float32Array(2);out[0]=a[0];out[1]=a[1];return out};vec2.fromValues=function(x,y){var out=new Float32Array(2);out[0]=x;out[1]=y;return out};vec2.copy=function(out,a){out[0]=a[0];out[1]=a[1];return out};vec2.set=function(out,x,y){out[0]=x;out[1]=y;return out};vec2.add=function(out,a,b){out[0]=a[0]+b[0];out[1]=a[1]+b[1];return out};vec2.sub=vec2.subtract=function(out,a,b){out[0]=a[0]-b[0];out[1]=a[1]-b[1];return out};vec2.mul=vec2.multiply=function(out,a,b){out[0]=a[0]*b[0];out[1]=a[1]*b[1];return out};vec2.div=vec2.divide=function(out,a,b){out[0]=a[0]/b[0];out[1]=a[1]/b[1];return out};vec2.min=function(out,a,b){out[0]=Math.min(a[0],b[0]);
+out[1]=Math.min(a[1],b[1]);return out};vec2.max=function(out,a,b){out[0]=Math.max(a[0],b[0]);out[1]=Math.max(a[1],b[1]);return out};vec2.scale=function(out,a,b){out[0]=a[0]*b;out[1]=a[1]*b;return out};vec2.dist=vec2.distance=function(a,b){var x=b[0]-a[0],y=b[1]-a[1];return Math.sqrt(x*x+y*y)};vec2.sqrDist=vec2.squaredDistance=function(a,b){var x=b[0]-a[0],y=b[1]-a[1];return x*x+y*y};vec2.len=vec2.length=function(a){var x=a[0],y=a[1];return Math.sqrt(x*x+y*y)};vec2.sqrLen=vec2.squaredLength=function(a){var x=a[0],y=a[1];return x*x+y*y};vec2.negate=function(out,a){out[0]=-a[0];out[1]=-a[1];return out};vec2.normalize=function(out,a){var x=a[0],y=a[1];var len=x*x+y*y;if(len>0){len=1/Math.sqrt(len);out[0]=a[0]*len;out[1]=a[1]*len}return out};vec2.dot=function(a,b){return a[0]*b[0]+a[1]*b[1]};vec2.cross=function(out,a,b){var z=a[0]*b[1]-a[1]*b[0];out[0]=out[1]=0;out[2]=z;return out};vec2.lerp=function(out,a,b,t){var ax=a[0],ay=a[1];out[0]=ax+t*(b[0]-ax);out[1]=ay+t*(b[1]-ay);return out};vec2.transformMat2=function(out,a,m){var x=a[0],y=a[1];out[0]=x*m[0]+y*m[1];out[1]=x*m[2]+y*m[3];return out};vec2.forEach=function(){var vec=new Float32Array(2);return function(a,stride,offset,count,fn,arg){var i,l;if(!stride){stride=2}if(!offset){offset=0}if(count){l=Math.min(count*stride+offset,a.length)}else{l=a.length}for(i=offset;i0){len=1/Math.sqrt(len);out[0]=a[0]*len;out[1]=a[1]*len;out[2]=a[2]*len}return out};vec3.dot=function(a,b){return a[0]*b[0]+a[1]*b[1]+a[2]*b[2]};vec3.cross=function(out,a,b){var ax=a[0],ay=a[1],az=a[2],bx=b[0],by=b[1],bz=b[2];out[0]=ay*bz-az*by;out[1]=az*bx-ax*bz;out[2]=ax*by-ay*bx;return out};vec3.lerp=function(out,a,b,t){var ax=a[0],ay=a[1],az=a[2];out[0]=ax+t*(b[0]-ax);out[1]=ay+t*(b[1]-ay);out[2]=az+t*(b[2]-az);return out};vec3.transformMat4=function(out,a,m){var x=a[0],y=a[1],z=a[2];out[0]=m[0]*x+m[4]*y+m[8]*z+m[12];out[1]=m[1]*x+m[5]*y+m[9]*z+m[13];out[2]=m[2]*x+m[6]*y+m[10]*z+m[14];return out};vec3.transformQuat=function(out,a,q){var x=a[0],y=a[1],z=a[2],qx=q[0],qy=q[1],qz=q[2],qw=q[3],ix=qw*x+qy*z-qz*y,iy=qw*y+qz*x-qx*z,iz=qw*z+qx*y-qy*x,iw=-qx*x-qy*y-qz*z;out[0]=ix*qw+iw*-qx+iy*-qz-iz*-qy;out[1]=iy*qw+iw*-qy+iz*-qx-ix*-qz;out[2]=iz*qw+iw*-qz+ix*-qy-iy*-qx;return out};vec3.forEach=function(){var vec=new Float32Array(3);return function(a,stride,offset,count,fn,arg){var i,l;if(!stride){stride=3}if(!offset){offset=0}if(count){l=Math.min(count*stride+offset,a.length)}else{l=a.length}for(i=offset;i0){len=1/Math.sqrt(len);out[0]=a[0]*len;out[1]=a[1]*len;out[2]=a[2]*len;out[3]=a[3]*len}return out};vec4.dot=function(a,b){return a[0]*b[0]+a[1]*b[1]+a[2]*b[2]+a[3]*b[3]};vec4.lerp=function(out,a,b,t){var ax=a[0],ay=a[1],az=a[2],aw=a[3];out[0]=ax+t*(b[0]-ax);out[1]=ay+t*(b[1]-ay);out[2]=az+t*(b[2]-az);out[3]=aw+t*(b[3]-aw);return out};vec4.transformMat4=function(out,a,m){var x=a[0],y=a[1],z=a[2],w=a[3];out[0]=m[0]*x+m[4]*y+m[8]*z+m[12]*w;out[1]=m[1]*x+m[5]*y+m[9]*z+m[13]*w;out[2]=m[2]*x+m[6]*y+m[10]*z+m[14]*w;out[3]=m[3]*x+m[7]*y+m[11]*z+m[15]*w;return out};vec4.transformQuat=function(out,a,q){var x=a[0],y=a[1],z=a[2],qx=q[0],qy=q[1],qz=q[2],qw=q[3],ix=qw*x+qy*z-qz*y,iy=qw*y+qz*x-qx*z,iz=qw*z+qx*y-qy*x,iw=-qx*x-qy*y-qz*z;out[0]=ix*qw+iw*-qx+iy*-qz-iz*-qy;out[1]=iy*qw+iw*-qy+iz*-qx-ix*-qz;out[2]=iz*qw+iw*-qz+ix*-qy-iy*-qx;return out};vec4.forEach=function(){var vec=new Float32Array(4);return function(a,stride,offset,count,fn,arg){var i,l;if(!stride){stride=4}if(!offset){offset=0}if(count){l=Math.min(count*stride+offset,a.length)}else{l=a.length}for(i=offset;i=1){if(out!==a){out[0]=ax;out[1]=ay;out[2]=az;out[3]=aw}return out}halfTheta=Math.acos(cosHalfTheta);sinHalfTheta=Math.sqrt(1-cosHalfTheta*cosHalfTheta);if(Math.abs(sinHalfTheta)<.001){out[0]=ax*.5+bx*.5;out[1]=ay*.5+by*.5;out[2]=az*.5+bz*.5;out[3]=aw*.5+bw*.5;return out}ratioA=Math.sin((1-t)*halfTheta)/sinHalfTheta;ratioB=Math.sin(t*halfTheta)/sinHalfTheta;out[0]=ax*ratioA+bx*ratioB;out[1]=ay*ratioA+by*ratioB;out[2]=az*ratioA+bz*ratioB;out[3]=aw*ratioA+bw*ratioB;return out};quat.invert=function(out,a){var a0=a[0],a1=a[1],a2=a[2],a3=a[3],dot=a0*a0+a1*a1+a2*a2+a3*a3,invDot=dot?1/dot:0;out[0]=-a0*invDot;out[1]=-a1*invDot;out[2]=-a2*invDot;out[3]=a3*invDot;return out};quat.conjugate=function(out,a){out[0]=-a[0];out[1]=-a[1];out[2]=-a[2];out[3]=a[3];return out};quat.len=quat.length=vec4.length;quat.sqrLen=quat.squaredLength=vec4.squaredLength;quat.normalize=vec4.normalize;quat.str=function(a){return"quat("+a[0]+", "+a[1]+", "+a[2]+", "+a[3]+")"};if(typeof exports!=="undefined"){exports.quat=quat}}(shim.exports)}()}()},{}],20:[function(require,module,exports){!function(){!function(){var root=this;var previousUnderscore=root._;var breaker={};var ArrayProto=Array.prototype,ObjProto=Object.prototype,FuncProto=Function.prototype;var push=ArrayProto.push,slice=ArrayProto.slice,concat=ArrayProto.concat,toString=ObjProto.toString,hasOwnProperty=ObjProto.hasOwnProperty;var nativeForEach=ArrayProto.forEach,nativeMap=ArrayProto.map,nativeReduce=ArrayProto.reduce,nativeReduceRight=ArrayProto.reduceRight,nativeFilter=ArrayProto.filter,nativeEvery=ArrayProto.every,nativeSome=ArrayProto.some,nativeIndexOf=ArrayProto.indexOf,nativeLastIndexOf=ArrayProto.lastIndexOf,nativeIsArray=Array.isArray,nativeKeys=Object.keys,nativeBind=FuncProto.bind;var _=function(obj){if(obj instanceof _)return obj;if(!(this instanceof _))return new _(obj);this._wrapped=obj};if(typeof exports!=="undefined"){if(typeof module!=="undefined"&&module.exports){exports=module.exports=_}exports._=_}else{root._=_}_.VERSION="1.4.4";var each=_.each=_.forEach=function(obj,iterator,context){if(obj==null)return;if(nativeForEach&&obj.forEach===nativeForEach){obj.forEach(iterator,context)}else if(obj.length===+obj.length){for(var i=0,l=obj.length;i2;if(obj==null)obj=[];if(nativeReduce&&obj.reduce===nativeReduce){if(context)iterator=_.bind(iterator,context);return initial?obj.reduce(iterator,memo):obj.reduce(iterator)}each(obj,function(value,index,list){if(!initial){memo=value;initial=true}else{memo=iterator.call(context,memo,value,index,list)}});if(!initial)throw new TypeError(reduceError);return memo};_.reduceRight=_.foldr=function(obj,iterator,memo,context){var initial=arguments.length>2;if(obj==null)obj=[];if(nativeReduceRight&&obj.reduceRight===nativeReduceRight){if(context)iterator=_.bind(iterator,context);return initial?obj.reduceRight(iterator,memo):obj.reduceRight(iterator)}var length=obj.length;if(length!==+length){var keys=_.keys(obj);length=keys.length}each(obj,function(value,index,list){index=keys?keys[--length]:--length;if(!initial){memo=obj[index];initial=true}else{memo=iterator.call(context,memo,obj[index],index,list)}});if(!initial)throw new TypeError(reduceError);return memo};_.find=_.detect=function(obj,iterator,context){var result;any(obj,function(value,index,list){if(iterator.call(context,value,index,list)){result=value;return true}});return result};_.filter=_.select=function(obj,iterator,context){var results=[];if(obj==null)return results;if(nativeFilter&&obj.filter===nativeFilter)return obj.filter(iterator,context);each(obj,function(value,index,list){if(iterator.call(context,value,index,list))results[results.length]=value});return results};_.reject=function(obj,iterator,context){return _.filter(obj,function(value,index,list){return!iterator.call(context,value,index,list)},context)};_.every=_.all=function(obj,iterator,context){iterator||(iterator=_.identity);var result=true;if(obj==null)return result;if(nativeEvery&&obj.every===nativeEvery)return obj.every(iterator,context);each(obj,function(value,index,list){if(!(result=result&&iterator.call(context,value,index,list)))return breaker});return!!result};var any=_.some=_.any=function(obj,iterator,context){iterator||(iterator=_.identity);var result=false;if(obj==null)return result;if(nativeSome&&obj.some===nativeSome)return obj.some(iterator,context);each(obj,function(value,index,list){if(result||(result=iterator.call(context,value,index,list)))return breaker});return!!result};_.contains=_.include=function(obj,target){if(obj==null)return false;if(nativeIndexOf&&obj.indexOf===nativeIndexOf)return obj.indexOf(target)!=-1;return any(obj,function(value){return value===target})};_.invoke=function(obj,method){var args=slice.call(arguments,2);var isFunc=_.isFunction(method);return _.map(obj,function(value){return(isFunc?method:value[method]).apply(value,args)})};_.pluck=function(obj,key){return _.map(obj,function(value){return value[key]})};_.where=function(obj,attrs,first){if(_.isEmpty(attrs))return first?null:[];return _[first?"find":"filter"](obj,function(value){for(var key in attrs){if(attrs[key]!==value[key])return false}return true})};_.findWhere=function(obj,attrs){return _.where(obj,attrs,true)};_.max=function(obj,iterator,context){if(!iterator&&_.isArray(obj)&&obj[0]===+obj[0]&&obj.length<65535){return Math.max.apply(Math,obj)}if(!iterator&&_.isEmpty(obj))return-Infinity;var result={computed:-Infinity,value:-Infinity};each(obj,function(value,index,list){var computed=iterator?iterator.call(context,value,index,list):value;computed>=result.computed&&(result={value:value,computed:computed})});return result.value};_.min=function(obj,iterator,context){if(!iterator&&_.isArray(obj)&&obj[0]===+obj[0]&&obj.length<65535){return Math.min.apply(Math,obj)}if(!iterator&&_.isEmpty(obj))return Infinity;var result={computed:Infinity,value:Infinity};each(obj,function(value,index,list){var computed=iterator?iterator.call(context,value,index,list):value;computedb||a===void 0)return 1;if(a>>1;iterator.call(context,array[mid])=0})})};_.difference=function(array){var rest=concat.apply(ArrayProto,slice.call(arguments,1));return _.filter(array,function(value){return!_.contains(rest,value)})};_.zip=function(){var args=slice.call(arguments);var length=_.max(_.pluck(args,"length"));var results=new Array(length);for(var i=0;i=0;i--){args=[funcs[i].apply(this,args)]}return args[0]}};_.after=function(times,func){if(times<=0)return func();return function(){if(--times<1){return func.apply(this,arguments)}}};_.keys=nativeKeys||function(obj){if(obj!==Object(obj))throw new TypeError("Invalid object");var keys=[];for(var key in obj)if(_.has(obj,key))keys[keys.length]=key;return keys};_.values=function(obj){var values=[];for(var key in obj)if(_.has(obj,key))values.push(obj[key]);return values};_.pairs=function(obj){var pairs=[];for(var key in obj)if(_.has(obj,key))pairs.push([key,obj[key]]);return pairs};_.invert=function(obj){var result={};for(var key in obj)if(_.has(obj,key))result[obj[key]]=key;return result};_.functions=_.methods=function(obj){var names=[];for(var key in obj){if(_.isFunction(obj[key]))names.push(key)}return names.sort()};_.extend=function(obj){each(slice.call(arguments,1),function(source){if(source){for(var prop in source){obj[prop]=source[prop]}}});return obj};_.pick=function(obj){var copy={};var keys=concat.apply(ArrayProto,slice.call(arguments,1));each(keys,function(key){if(key in obj)copy[key]=obj[key]});return copy};_.omit=function(obj){var copy={};var keys=concat.apply(ArrayProto,slice.call(arguments,1));for(var key in obj){if(!_.contains(keys,key))copy[key]=obj[key]}return copy};_.defaults=function(obj){each(slice.call(arguments,1),function(source){if(source){for(var prop in source){if(obj[prop]==null)obj[prop]=source[prop]}}});return obj};_.clone=function(obj){if(!_.isObject(obj))return obj;return _.isArray(obj)?obj.slice():_.extend({},obj)};_.tap=function(obj,interceptor){interceptor(obj);return obj};var eq=function(a,b,aStack,bStack){if(a===b)return a!==0||1/a==1/b;if(a==null||b==null)return a===b;if(a instanceof _)a=a._wrapped;if(b instanceof _)b=b._wrapped;var className=toString.call(a);if(className!=toString.call(b))return false;switch(className){case"[object String]":return a==String(b);case"[object Number]":return a!=+a?b!=+b:a==0?1/a==1/b:a==+b;case"[object Date]":case"[object Boolean]":return+a==+b;case"[object RegExp]":return a.source==b.source&&a.global==b.global&&a.multiline==b.multiline&&a.ignoreCase==b.ignoreCase}if(typeof a!="object"||typeof b!="object")return false;var length=aStack.length;while(length--){if(aStack[length]==a)return bStack[length]==b}aStack.push(a);bStack.push(b);var size=0,result=true;if(className=="[object Array]"){size=a.length;result=size==b.length;if(result){while(size--){if(!(result=eq(a[size],b[size],aStack,bStack)))break}}}else{var aCtor=a.constructor,bCtor=b.constructor;if(aCtor!==bCtor&&!(_.isFunction(aCtor)&&aCtor instanceof aCtor&&_.isFunction(bCtor)&&bCtor instanceof bCtor)){return false}for(var key in a){if(_.has(a,key)){size++;if(!(result=_.has(b,key)&&eq(a[key],b[key],aStack,bStack)))break}}if(result){for(key in b){if(_.has(b,key)&&!size--)break}result=!size}}aStack.pop();bStack.pop();return result};_.isEqual=function(a,b){return eq(a,b,[],[])};_.isEmpty=function(obj){if(obj==null)return true;if(_.isArray(obj)||_.isString(obj))return obj.length===0;for(var key in obj)if(_.has(obj,key))return false;return true};_.isElement=function(obj){return!!(obj&&obj.nodeType===1)};_.isArray=nativeIsArray||function(obj){return toString.call(obj)=="[object Array]"};_.isObject=function(obj){return obj===Object(obj)};each(["Arguments","Function","String","Number","Date","RegExp"],function(name){_["is"+name]=function(obj){return toString.call(obj)=="[object "+name+"]"}});if(!_.isArguments(arguments)){_.isArguments=function(obj){return!!(obj&&_.has(obj,"callee"))}}if(typeof/./!=="function"){_.isFunction=function(obj){return typeof obj==="function"}}_.isFinite=function(obj){return isFinite(obj)&&!isNaN(parseFloat(obj))};_.isNaN=function(obj){return _.isNumber(obj)&&obj!=+obj};_.isBoolean=function(obj){return obj===true||obj===false||toString.call(obj)=="[object Boolean]"};_.isNull=function(obj){return obj===null};_.isUndefined=function(obj){return obj===void 0};_.has=function(obj,key){return hasOwnProperty.call(obj,key)};_.noConflict=function(){root._=previousUnderscore;return this};_.identity=function(value){return value};_.times=function(n,iterator,context){var accum=Array(n);for(var i=0;i":">",'"':""","'":"'","/":"/"}};entityMap.unescape=_.invert(entityMap.escape);var entityRegexes={escape:new RegExp("["+_.keys(entityMap.escape).join("")+"]","g"),unescape:new RegExp("("+_.keys(entityMap.unescape).join("|")+")","g")};_.each(["escape","unescape"],function(method){_[method]=function(string){if(string==null)return"";return(""+string).replace(entityRegexes[method],function(match){return entityMap[method][match]})}});_.result=function(object,property){if(object==null)return null;var value=object[property];return _.isFunction(value)?value.call(object):value};_.mixin=function(obj){each(_.functions(obj),function(name){var func=_[name]=obj[name];_.prototype[name]=function(){var args=[this._wrapped];push.apply(args,arguments);return result.call(this,func.apply(_,args))}})};var idCounter=0;_.uniqueId=function(prefix){var id=++idCounter+"";return prefix?prefix+id:id};_.templateSettings={evaluate:/<%([\s\S]+?)%>/g,interpolate:/<%=([\s\S]+?)%>/g,escape:/<%-([\s\S]+?)%>/g};var noMatch=/(.)^/;var escapes={"'":"'","\\":"\\","\r":"r","\n":"n"," ":"t","\u2028":"u2028","\u2029":"u2029"};var escaper=/\\|'|\r|\n|\t|\u2028|\u2029/g;_.template=function(text,data,settings){var render;settings=_.defaults({},settings,_.templateSettings);var matcher=new RegExp([(settings.escape||noMatch).source,(settings.interpolate||noMatch).source,(settings.evaluate||noMatch).source].join("|")+"|$","g");var index=0;var source="__p+='";text.replace(matcher,function(match,escape,interpolate,evaluate,offset){source+=text.slice(index,offset).replace(escaper,function(match){return"\\"+escapes[match]});if(escape){source+="'+\n((__t=("+escape+"))==null?'':_.escape(__t))+\n'"}if(interpolate){source+="'+\n((__t=("+interpolate+"))==null?'':__t)+\n'"}if(evaluate){source+="';\n"+evaluate+"\n__p+='"}index=offset+match.length;return match});source+="';\n";if(!settings.variable)source="with(obj||{}){\n"+source+"}\n";source="var __t,__p='',__j=Array.prototype.join,"+"print=function(){__p+=__j.call(arguments,'');};\n"+source+"return __p;\n";try{render=new Function(settings.variable||"obj","_",source)}catch(e){e.source=source;throw e}if(data)return render(data,_);var template=function(data){return render.call(this,data,_)};template.source="function("+(settings.variable||"obj")+"){\n"+source+"}";return template};_.chain=function(obj){return _(obj).chain()};var result=function(obj){return this._chain?_(obj).chain():obj};_.mixin(_);each(["pop","push","reverse","shift","sort","splice","unshift"],function(name){var method=ArrayProto[name];_.prototype[name]=function(){var obj=this._wrapped;method.apply(obj,arguments);if((name=="shift"||name=="splice")&&obj.length===0)delete obj[0];return result.call(this,obj)}});each(["concat","join","slice"],function(name){var method=ArrayProto[name];_.prototype[name]=function(){return result.call(this,method.apply(this._wrapped,arguments))}});_.extend(_.prototype,{chain:function(){this._chain=true;return this},value:function(){return this._wrapped}})}.call(this)}()},{}],21:[function(require,module,exports){window.requestAnimFrame=function(){return window.requestAnimationFrame||window.webkitRequestAnimationFrame||window.mozRequestAnimationFrame||window.oRequestAnimationFrame||window.msRequestAnimationFrame||function(callback){window.setTimeout(callback,1e3/60)}}();Leap=require("../lib/index")},{"../lib/index":8}]},{},[21]);
+
+/*
+ * Leap Motion integration for Reveal.js.
+ * James Sun  [sun16]
+ * Rory Hardy [gneatgeek]
+ */
+
+(function () {
+  var body        = document.body,
+      controller  = new Leap.Controller({ enableGestures: true }),
+      lastGesture = 0,
+      leapConfig  = Reveal.getConfig().leap,
+      pointer     = document.createElement( 'div' ),
+      config      = {
+        autoCenter       : true,      // Center pointer around detected position.
+        gestureDelay     : 500,       // How long to delay between gestures.
+        naturalSwipe     : true,      // Swipe as if it were a touch screen.
+        pointerColor     : '#00aaff', // Default color of the pointer.
+        pointerOpacity   : 0.7,       // Default opacity of the pointer.
+        pointerSize      : 15,        // Default minimum height/width of the pointer.
+        pointerTolerance : 120        // Bigger = slower pointer.
+      },
+      entered, enteredPosition, now, size, tipPosition; // Other vars we need later, but don't need to redeclare.
+
+      // Merge user defined settings with defaults
+      if( leapConfig ) {
+        for( key in leapConfig ) {
+          config[key] = leapConfig[key];
+        }
+      }
+
+      pointer.id = 'leap';
+
+      pointer.style.position        = 'absolute';
+      pointer.style.visibility      = 'hidden';
+      pointer.style.zIndex          = 50;
+      pointer.style.opacity         = config.pointerOpacity;
+      pointer.style.backgroundColor = config.pointerColor;
+
+      body.appendChild( pointer );
+
+  // Leap's loop
+  controller.on( 'frame', function ( frame ) {
+    // Timing code to rate limit gesture execution
+    now = new Date().getTime();
+
+    // Pointer: 1 to 2 fingers. Strictly one finger works but may cause innaccuracies.
+    // The innaccuracies were observed on a development model and may not be an issue with consumer models.
+    if( frame.fingers.length > 0 && frame.fingers.length < 3 ) {
+      // Invert direction and multiply by 3 for greater effect.
+      size = -3 * frame.fingers[0].tipPosition[2];
+
+      if( size < config.pointerSize ) {
+        size = config.pointerSize;
+      }
+
+      pointer.style.width        = size     + 'px';
+      pointer.style.height       = size     + 'px';
+      pointer.style.borderRadius = size - 5 + 'px';
+      pointer.style.visibility   = 'visible';
+
+      tipPosition = frame.fingers[0].tipPosition;
+
+      if( config.autoCenter ) {
+        
+
+        // Check whether the finger has entered the z range of the Leap Motion. Used for the autoCenter option.
+        if( !entered ) {
+          entered         = true;
+          enteredPosition = frame.fingers[0].tipPosition;
+        }
+
+        pointer.style.top =
+          (-1 * (( tipPosition[1] - enteredPosition[1] ) * body.offsetHeight / config.pointerTolerance )) +
+            ( body.offsetHeight / 2 ) + 'px';
+
+        pointer.style.left =
+          (( tipPosition[0] - enteredPosition[0] ) * body.offsetWidth / config.pointerTolerance ) +
+            ( body.offsetWidth / 2 ) + 'px';
+      }
+      else {
+        pointer.style.top  = ( 1 - (( tipPosition[1] - 50) / config.pointerTolerance )) *
+          body.offsetHeight + 'px';
+
+        pointer.style.left = ( tipPosition[0] * body.offsetWidth / config.pointerTolerance ) +
+          ( body.offsetWidth / 2 ) + 'px';
+      }
+    }
+    else {
+      // Hide pointer on exit
+      entered                  = false;
+      pointer.style.visibility = 'hidden';
+    }
+
+    // Gestures
+    if( frame.gestures.length > 0 && (now - lastGesture) > config.gestureDelay ) {
+      var gesture = frame.gestures[0];
+
+      // One hand gestures
+      if( frame.hands.length === 1 ) {
+        // Swipe gestures. 3+ fingers.
+        if( frame.fingers.length > 2 && gesture.type === 'swipe' ) {
+          // Define here since some gestures will throw undefined for these.
+          var x = gesture.direction[0],
+              y = gesture.direction[1];
+
+          // Left/right swipe gestures
+          if( Math.abs( x ) > Math.abs( y )) {
+            if( x > 0 ) {
+              config.naturalSwipe ? Reveal.left() : Reveal.right();
+            }
+            else {
+              config.naturalSwipe ? Reveal.right() : Reveal.left();
+            }
+          }
+          // Up/down swipe gestures
+          else {
+            if( y > 0 ) {
+              config.naturalSwipe ? Reveal.down() : Reveal.up();
+            }
+            else {
+              config.naturalSwipe ? Reveal.up() : Reveal.down();
+            }
+          }
+
+          lastGesture = now;
+        }
+      }
+      // Two hand gestures
+      else if( frame.hands.length === 2 ) {
+        // Upward two hand swipe gesture
+        if( gesture.type === 'swipe' && gesture.direction[1] > 0 ) {
+          Reveal.toggleOverview();
+        }
+
+        lastGesture = now;
+      }
+    }
+  });
+
+  controller.connect();
+})();
diff --git a/doc/pub/Recurrent/html/reveal.js/plugin/remotes/remotes.js b/doc/pub/Recurrent/html/reveal.js/plugin/remotes/remotes.js
new file mode 100644
index 000000000..ba0dbad7b
--- /dev/null
+++ b/doc/pub/Recurrent/html/reveal.js/plugin/remotes/remotes.js
@@ -0,0 +1,39 @@
+/**
+ * Touch-based remote controller for your presentation courtesy 
+ * of the folks at http://remotes.io
+ */
+
+(function(window){
+
+    /**
+     * Detects if we are dealing with a touch enabled device (with some false positives)
+     * Borrowed from modernizr: https://github.com/Modernizr/Modernizr/blob/master/feature-detects/touch.js   
+     */
+    var hasTouch  = (function(){
+        return ('ontouchstart' in window) || window.DocumentTouch && document instanceof DocumentTouch;
+    })();
+
+    /**
+     * Detects if notes are enable and the current page is opened inside an /iframe
+     * this prevents loading Remotes.io several times
+     */
+    var isNotesAndIframe = (function(){
+        return window.RevealNotes && !(self == top);
+    })();
+
+    if(!hasTouch && !isNotesAndIframe){
+        head.ready( 'remotes.ne.min.js', function() {
+            new Remotes("preview")
+                .on("swipe-left", function(e){ Reveal.right(); })
+                .on("swipe-right", function(e){ Reveal.left(); })
+                .on("swipe-up", function(e){ Reveal.down(); })
+                .on("swipe-down", function(e){ Reveal.up(); })
+                .on("tap", function(e){ Reveal.next(); })
+                .on("zoom-out", function(e){ Reveal.toggleOverview(true); })
+                .on("zoom-in", function(e){ Reveal.toggleOverview(false); })
+            ;
+        } );
+
+        head.js('https://hakim-static.s3.amazonaws.com/reveal-js/remotes.ne.min.js');
+    }
+})(window);
\ No newline at end of file
diff --git a/doc/pub/Regression/html/reveal.js/css/theme/template/mixins.scss b/doc/pub/Regression/html/reveal.js/css/theme/template/mixins.scss
new file mode 100644
index 000000000..e0c560692
--- /dev/null
+++ b/doc/pub/Regression/html/reveal.js/css/theme/template/mixins.scss
@@ -0,0 +1,29 @@
+@mixin vertical-gradient( $top, $bottom ) {
+	background: $top;
+	background: -moz-linear-gradient( top, $top 0%, $bottom 100% );
+	background: -webkit-gradient( linear, left top, left bottom, color-stop(0%,$top), color-stop(100%,$bottom) );
+	background: -webkit-linear-gradient( top, $top 0%, $bottom 100% );
+	background: -o-linear-gradient( top, $top 0%, $bottom 100% );
+	background: -ms-linear-gradient( top, $top 0%, $bottom 100% );
+	background: linear-gradient( top, $top 0%, $bottom 100% );
+}
+
+@mixin horizontal-gradient( $top, $bottom ) {
+	background: $top;
+	background: -moz-linear-gradient( left, $top 0%, $bottom 100% );
+	background: -webkit-gradient( linear, left top, right top, color-stop(0%,$top), color-stop(100%,$bottom) );
+	background: -webkit-linear-gradient( left, $top 0%, $bottom 100% );
+	background: -o-linear-gradient( left, $top 0%, $bottom 100% );
+	background: -ms-linear-gradient( left, $top 0%, $bottom 100% );
+	background: linear-gradient( left, $top 0%, $bottom 100% );
+}
+
+@mixin radial-gradient( $outer, $inner, $type: circle ) {
+	background: $outer;
+	background: -moz-radial-gradient( center, $type cover,  $inner 0%, $outer 100% );
+	background: -webkit-gradient( radial, center center, 0px, center center, 100%, color-stop(0%,$inner), color-stop(100%,$outer) );
+	background: -webkit-radial-gradient( center, $type cover,  $inner 0%, $outer 100% );
+	background: -o-radial-gradient( center, $type cover,  $inner 0%, $outer 100% );
+	background: -ms-radial-gradient( center, $type cover,  $inner 0%, $outer 100% );
+	background: radial-gradient( center, $type cover,  $inner 0%, $outer 100% );
+}
\ No newline at end of file
diff --git a/doc/pub/Regression/html/reveal.js/css/theme/template/settings.scss b/doc/pub/Regression/html/reveal.js/css/theme/template/settings.scss
new file mode 100644
index 000000000..e706e19c5
--- /dev/null
+++ b/doc/pub/Regression/html/reveal.js/css/theme/template/settings.scss
@@ -0,0 +1,34 @@
+// Base settings for all themes that can optionally be
+// overridden by the super-theme
+
+// Background of the presentation
+$backgroundColor: #2b2b2b;
+
+// Primary/body text
+$mainFont: 'Lato', sans-serif;
+$mainFontSize: 30px;              /* changed (by hpl) from 36px; */
+$mainColor: #eee;
+
+// Headings
+$headingMargin: 0 0 20px 0;
+$headingFont: 'League Gothic', Impact, sans-serif;
+$headingColor: #eee;
+$headingLineHeight: 0.9em;
+$headingLetterSpacing: 0.02em;
+$headingTextTransform: none;      /* changed (by hpl) from uppercase; */
+$headingTextShadow: 0px 0px 6px rgba(0,0,0,0.2);
+$heading1TextShadow: $headingTextShadow;
+
+// Links and actions
+$linkColor: #13DAEC;
+$linkColorHover: lighten( $linkColor, 20% );
+
+// Text selection
+$selectionBackgroundColor: #FF5E99;
+$selectionColor: #fff;
+
+// Generates the presentation background, can be overridden
+// to return a background image or gradient
+@mixin bodyBackground() {
+	background: $backgroundColor;
+}
\ No newline at end of file
diff --git a/doc/pub/Regression/html/reveal.js/css/theme/template/theme.scss b/doc/pub/Regression/html/reveal.js/css/theme/template/theme.scss
new file mode 100644
index 000000000..b1dbc2736
--- /dev/null
+++ b/doc/pub/Regression/html/reveal.js/css/theme/template/theme.scss
@@ -0,0 +1,171 @@
+// Base theme template for reveal.js
+
+/*********************************************
+ * GLOBAL STYLES
+ *********************************************/
+
+body {
+	@include bodyBackground();
+	background-color: $backgroundColor;
+}
+
+.reveal {
+	font-family: $mainFont;
+	font-size: $mainFontSize;
+	font-weight: normal;
+	letter-spacing: -0.02em;
+	color: $mainColor;
+}
+
+::selection {
+	color: $selectionColor;
+	background: $selectionBackgroundColor;
+	text-shadow: none;
+}
+
+/*********************************************
+ * HEADERS
+ *********************************************/
+
+.reveal h1,
+.reveal h2,
+.reveal h3,
+.reveal h4,
+.reveal h5,
+.reveal h6 {
+	margin: $headingMargin;
+	color: $headingColor;
+
+	font-family: $headingFont;
+	line-height: $headingLineHeight;
+	letter-spacing: $headingLetterSpacing;
+
+	text-transform: $headingTextTransform;
+	text-shadow: $headingTextShadow;
+}
+
+.reveal h1 {
+        line-height: 1.2em;    /* added by hpl */
+	text-shadow: $heading1TextShadow;
+}
+
+
+/*********************************************
+ * LINKS
+ *********************************************/
+
+.reveal a:not(.image) {
+	color: $linkColor;
+	text-decoration: none;
+
+	-webkit-transition: color .15s ease;
+	   -moz-transition: color .15s ease;
+	    -ms-transition: color .15s ease;
+	     -o-transition: color .15s ease;
+	        transition: color .15s ease;
+}
+	.reveal a:not(.image):hover {
+		color: $linkColorHover;
+
+		text-shadow: none;
+		border: none;
+	}
+
+.reveal .roll span:after {
+	color: #fff;
+	background: darken( $linkColor, 15% );
+}
+
+
+/*********************************************
+ * IMAGES
+ *********************************************/
+
+.reveal section img {
+	margin: 15px 0px;
+	background: rgba(255,255,255,0.12);
+	border: 4px solid $mainColor;
+
+	box-shadow: 0 0 10px rgba(0, 0, 0, 0.15);
+
+	-webkit-transition: all .2s linear;
+	   -moz-transition: all .2s linear;
+	    -ms-transition: all .2s linear;
+	     -o-transition: all .2s linear;
+	        transition: all .2s linear;
+}
+
+	.reveal a:hover img {
+		background: rgba(255,255,255,0.2);
+		border-color: $linkColor;
+
+		box-shadow: 0 0 20px rgba(0, 0, 0, 0.55);
+	}
+
+
+/*********************************************
+ * NAVIGATION CONTROLS
+ *********************************************/
+
+.reveal .controls div.navigate-left,
+.reveal .controls div.navigate-left.enabled {
+	border-right-color: $linkColor;
+}
+
+.reveal .controls div.navigate-right,
+.reveal .controls div.navigate-right.enabled {
+	border-left-color: $linkColor;
+}
+
+.reveal .controls div.navigate-up,
+.reveal .controls div.navigate-up.enabled {
+	border-bottom-color: $linkColor;
+}
+
+.reveal .controls div.navigate-down,
+.reveal .controls div.navigate-down.enabled {
+	border-top-color: $linkColor;
+}
+
+.reveal .controls div.navigate-left.enabled:hover {
+	border-right-color: $linkColorHover;
+}
+
+.reveal .controls div.navigate-right.enabled:hover {
+	border-left-color: $linkColorHover;
+}
+
+.reveal .controls div.navigate-up.enabled:hover {
+	border-bottom-color: $linkColorHover;
+}
+
+.reveal .controls div.navigate-down.enabled:hover {
+	border-top-color: $linkColorHover;
+}
+
+
+/*********************************************
+ * PROGRESS BAR
+ *********************************************/
+
+.reveal .progress {
+	background: rgba(0,0,0,0.2);
+}
+	.reveal .progress span {
+		background: $linkColor;
+
+		-webkit-transition: width 800ms cubic-bezier(0.260, 0.860, 0.440, 0.985);
+		   -moz-transition: width 800ms cubic-bezier(0.260, 0.860, 0.440, 0.985);
+		    -ms-transition: width 800ms cubic-bezier(0.260, 0.860, 0.440, 0.985);
+		     -o-transition: width 800ms cubic-bezier(0.260, 0.860, 0.440, 0.985);
+		        transition: width 800ms cubic-bezier(0.260, 0.860, 0.440, 0.985);
+	}
+
+/*********************************************
+ * SLIDE NUMBER
+ *********************************************/
+.reveal .slide-number {
+  color: $linkColor;
+}
+
+
diff --git a/doc/pub/Regression/html/reveal.js/demo.html b/doc/pub/Regression/html/reveal.js/demo.html
new file mode 100644
index 000000000..505bb1882
--- /dev/null
+++ b/doc/pub/Regression/html/reveal.js/demo.html
@@ -0,0 +1,410 @@
+
+
+
+	
+		
+
+		reveal.js – The HTML Presentation Framework
+
+		
+		
+
+		
+		
+
+		
+
+		
+		
+
+		
+		
+
+		
+		
+
+		
+	
+
+	
+
+		

+
+			
+			

+				

+					
Reveal.js

+					
The HTML Presentation Framework

+					

+						Created by Hakim El Hattab and contributors
+					

+				

+
+				

+					
Hello There

+					

+						reveal.js enables you to create beautiful interactive slide decks using HTML. This presentation will show you examples of what it can do.
+					

+				

+
+				
+				

+					

+						
Vertical Slides

+						
Slides can be nested inside of each other.

+						
Use the Space key to navigate through all slides.

+						

+						
+							Down arrow
+						
+					

+					

+						
Basement Level 1

+						
Nested slides are useful for adding additional detail underneath a high level horizontal slide.

+					

+					

+						
Basement Level 2

+						
That's it, time to go back up.

+						

+						
+							Up arrow
+						
+					

+				

+
+				

+					
Slides

+					

+						Not a coder? Not a problem. There's a fully-featured visual editor for authoring these, try it out at https://slides.com.
+					

+				

+
+				

+					
Point of View

+					

+						Press ESC to enter the slide overview.
+					

+					

+						Hold down alt and click on any element to zoom in on it using zoom.js. Alt + click anywhere to zoom back out.
+					

+				

+
+				

+					
Touch Optimized

+					

+						Presentations look great on touch devices, like mobile phones and tablets. Simply swipe through your slides.
+					

+				

+
+				

+					
+				

+
+				

+					

+						
Fragments

+						
Hit the next arrow...

+						
... to step through ...

+						
... a fragmented slide.

+
+						
+					

+					

+						
Fragment Styles

+						
There's different types of fragments, like:

+						
grow

+						
shrink

+						
fade-out

+						
fade-up (also down, left and right!)

+						
current-visible

+						
Highlight red blue green

+					

+				

+
+				

+					
Transition Styles

+					

+						You can select from different transitions, like: 

+						None -
+						Fade -
+						Slide -
+						Convex -
+						Concave -
+						Zoom
+					

+				

+
+				

+					
Themes

+					

+						reveal.js comes with a few themes built in: 

+						
+						Black (default) -
+						White -
+						League -
+						Sky -
+						Beige -
+						Simple 

+						Serif -
+						Blood -
+						Night -
+						Moon -
+						Solarized
+					

+				

+
+				

+					

+						
Slide Backgrounds

+						

+							Set data-background="#dddddd" on a slide to change the background color. All CSS color formats are supported.
+						

+						
+							Down arrow
+						
+					

+					

+						
Image Backgrounds

+						
<section data-background="image.png">

+					

+					

+						
Tiled Backgrounds

+						
<section data-background="image.png" data-background-repeat="repeat" data-background-size="100px">

+					

+					

+						

+							
Video Backgrounds

+							
<section data-background-video="video.mp4,video.webm">

+						

+					

+					

+						
... and GIFs!

+					

+				

+
+				

+					
Background Transitions

+					

+						Different background transitions are available via the backgroundTransition option. This one's called "zoom".
+					

+					
Reveal.configure({ backgroundTransition: 'zoom' })

+				

+
+				

+					
Background Transitions

+					

+						You can override background transitions per-slide.
+					

+					
<section data-background-transition="zoom">

+				

+
+				

+					
Pretty Code

+					

+function linkify( selector ) {
+  if( supports3DTransforms ) {
+
+    var nodes = document.querySelectorAll( selector );
+
+    for( var i = 0, len = nodes.length; i < len; i++ ) {
+      var node = nodes[i];
+
+      if( !node.className ) {
+        node.className += ' roll';
+      }
+    }
+  }
+}
+					

+					
Code syntax highlighting courtesy of highlight.js.

+				

+
+				

+					
Marvelous List

+					
    
+						
  • No order here
    • 
+						
  • Or here
    • 
+						
  • Or here
    • 
+						
  • Or here
    • 
+					

+				

+
+				

+					
Fantastic Ordered List

+					
    
+						
  1. One is smaller than...
    2. 
+						
  2. Two is smaller than...
    3. 
+						
  3. Three!
    4. 
+					

+				

+
+				

+					
Tabular Tables

+					
+						
+							
+								
+								
+								
+							
+						
+						
+							
+								
+								
+								
+							
+							
+								
+								
+								
+							
+							
+								
+								
+								
+							
+						
+					
ItemValueQuantity
Apples$17
Lemonade$218
Bread$32

+				

+
+				

+					
Clever Quotes

+					

+						These guys come in two forms, inline: The nice thing about standards is that there are so many to choose from and block:
+					

+					

+						“For years there has been a theory that millions of monkeys typing at random on millions of typewriters would
+						reproduce the entire works of Shakespeare. The Internet has proven this theory to be untrue.”
+					

+				

+
+				

+					
Intergalactic Interconnections

+					

+						You can link between slides internally,
+						like this.
+					

+				

+
+				

+					
Speaker View

+					
There's a speaker view. It includes a timer, preview of the upcoming slide as well as your speaker notes.

+					
Press the S key to try it out.

+
+					
+				

+
+				

+					
Export to PDF

+					
Presentations can be exported to PDF, here's an example:

+					
+				

+
+				

+					
Global State

+					

+						Set data-state="something" on a slide and "something"
+						will be added as a class to the document element when the slide is open. This lets you
+						apply broader style changes, like switching the page background.
+					

+				

+
+				

+					
State Events

+					

+						Additionally custom events can be triggered on a per slide basis by binding to the data-state name.
+					

+					

+Reveal.addEventListener( 'customevent', function() {
+	console.log( '"customevent" has fired' );
+} );
+					

+				

+
+				

+					
Take a Moment

+					

+						Press B or . on your keyboard to pause the presentation. This is helpful when you're on stage and want to take distracting slides off the screen.
+					

+				

+
+				

+					
Much more

+					
+				

+
+				

+					
THE END

+					

+						- Try the online editor 

+						- Source code & documentation
+					

+				

+
+			

+
+		

+
+		
+		
+
+		
+
+	
+
diff --git a/doc/pub/Regression/html/reveal.js/plugin/multiplex/package.json b/doc/pub/Regression/html/reveal.js/plugin/multiplex/package.json
new file mode 100644
index 000000000..bbed77a67
--- /dev/null
+++ b/doc/pub/Regression/html/reveal.js/plugin/multiplex/package.json
@@ -0,0 +1,19 @@
+{
+  "name": "reveal-js-multiplex",
+  "version": "1.0.0",
+  "description": "reveal.js multiplex server",
+  "homepage": "http://revealjs.com",
+  "scripts": {
+    "start": "node index.js"
+  },
+  "engines": {
+    "node": "~4.1.1"
+  },
+  "dependencies": {
+    "express": "~4.13.3",
+    "grunt-cli": "~0.1.13",
+    "mustache": "~2.2.1",
+    "socket.io": "~1.3.7"
+  },
+  "license": "MIT"
+}
diff --git a/doc/pub/Regression/html/reveal.js/test/simple.md b/doc/pub/Regression/html/reveal.js/test/simple.md
new file mode 100644
index 000000000..c72a44079
--- /dev/null
+++ b/doc/pub/Regression/html/reveal.js/test/simple.md
@@ -0,0 +1,12 @@
+## Slide 1.1
+
+```js
+var a = 1;
+```
+
+
+## Slide 1.2
+
+
+
+## Slide 2
diff --git a/doc/pub/Regression/html/reveal.js/test/test-markdown-external.html b/doc/pub/Regression/html/reveal.js/test/test-markdown-external.html
new file mode 100644
index 000000000..859d0a199
--- /dev/null
+++ b/doc/pub/Regression/html/reveal.js/test/test-markdown-external.html
@@ -0,0 +1,36 @@
+
+
+
+	
+		
+
+		reveal.js - Test Markdown
+
+		
+		
+	
+
+	
+
+		

+  		

+
+		
+
+		
+		
+		
+		
+		
+		
+
+		
+
+	
+
diff --git a/doc/pub/Regression/html/reveal.js/test/test-markdown-external.js b/doc/pub/Regression/html/reveal.js/test/test-markdown-external.js
new file mode 100644
index 000000000..cab85c6f6
--- /dev/null
+++ b/doc/pub/Regression/html/reveal.js/test/test-markdown-external.js
@@ -0,0 +1,24 @@
+
+
+Reveal.addEventListener( 'ready', function() {
+
+	QUnit.module( 'Markdown' );
+
+	test( 'Vertical separator', function() {
+		strictEqual( document.querySelectorAll( '.reveal .slides>section>section' ).length, 2, 'found two slides' );
+	});
+
+	test( 'Horizontal separator', function() {
+		strictEqual( document.querySelectorAll( '.reveal .slides>section' ).length, 2, 'found two slides' );
+	});
+
+	test( 'Language highlighter', function() {
+		strictEqual( document.querySelectorAll( '.hljs-keyword' ).length, 1, 'got rendered highlight tag.' );
+		strictEqual( document.querySelector( '.hljs-keyword' ).innerHTML, 'var', 'the same keyword: var.' );
+	});
+
+
+} );
+
+Reveal.initialize();
+
diff --git a/doc/pub/Regression/html/reveal.js/test/test-markdown-options.html b/doc/pub/Regression/html/reveal.js/test/test-markdown-options.html
new file mode 100644
index 000000000..5b3be9758
--- /dev/null
+++ b/doc/pub/Regression/html/reveal.js/test/test-markdown-options.html
@@ -0,0 +1,41 @@
+
+
+
+	
+		
+
+		reveal.js - Test Markdown Options
+
+		
+		
+	
+
+	
+
+		

+		

+
+		
+
+		
+		
+		
+
+		
+
+	
+
diff --git a/doc/pub/Regression/html/reveal.js/test/test-markdown-options.js b/doc/pub/Regression/html/reveal.js/test/test-markdown-options.js
new file mode 100644
index 000000000..3ae13503a
--- /dev/null
+++ b/doc/pub/Regression/html/reveal.js/test/test-markdown-options.js
@@ -0,0 +1,26 @@
+Reveal.addEventListener( 'ready', function() {
+
+	QUnit.module( 'Markdown' );
+
+	test( 'Options are set', function() {
+		strictEqual( marked.defaults.smartypants, true );
+	});
+
+	test( 'Smart quotes are activated', function() {
+		var text = document.querySelector( '.reveal .slides>section>p' ).textContent;
+
+		strictEqual( /['"]/.test( text ), false );
+		strictEqual( /[“”‘’]/.test( text ), true );
+	});
+
+} );
+
+Reveal.initialize({
+	dependencies: [
+		{ src: '../plugin/markdown/marked.js' },
+		{ src: '../plugin/markdown/markdown.js' },
+	],
+	markdown: {
+		smartypants: true
+	}
+});
diff --git a/doc/pub/Regression/ipynb/.ipynb_checkpoints/Regression-checkpoint.ipynb b/doc/pub/Regression/ipynb/.ipynb_checkpoints/Regression-checkpoint.ipynb
new file mode 100644
index 000000000..29137fca4
--- /dev/null
+++ b/doc/pub/Regression/ipynb/.ipynb_checkpoints/Regression-checkpoint.ipynb
@@ -0,0 +1,939 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "\n",
+    "# Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis\n",
+    "\n",
+    "  \n",
+    "**Morten Hjorth-Jensen**, Department of Physics, University of Oslo and Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University\n",
+    "\n",
+    "Date: **Nov 21, 2017**\n",
+    "\n",
+    "Copyright 1999-2017, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license\n",
+    "\n",
+    "\n",
+    "\n",
+    "\n",
+    "## Regression analysis, overarching aims\n",
+    "\n",
+    "Regression modeling deals with the description of  the sampling distribution of a given random variable $y$ varies as function of another variable or a set of such variables $\\hat{x} =[x_0, x_1,\\dots, x_p]^T$. \n",
+    "The first variable is called the **dependent**, the **outcome** or the **response** variable while the set of variables $\\hat{x}$ is called the independent variable, or the predictor variable or the explanatory variable. \n",
+    "\n",
+    "A regression model aims at finding a likelihood function $p(y\\vert \\hat{x})$, that is the conditional distribution for $y$ with a given $\\hat{x}$. The estimation of  $p(y\\vert \\hat{x})$ is made using a data set with \n",
+    "* $n$ cases $i = 0, 1, 2, \\dots, n-1$ \n",
+    "\n",
+    "* Response (dependent or outcome) variable $y_i$ with $i = 0, 1, 2, \\dots, n-1$ \n",
+    "\n",
+    "* $p$ Explanatory (independent or predictor) variables $\\hat{x}_i=[x_{i0}, x_{i1}, \\dots, x_{ip}]$ with $i = 0, 1, 2, \\dots, n-1$   \n",
+    "\n",
+    " The goal of the regression analysis is to extract/exploit relationship between $y_i$ and $\\hat{x}_i$ in or to infer causal dependencies, approximations to the likelihood functions, functional relationships and to make predictions .\n",
+    "\n",
+    "\n",
+    "\n",
+    "\n",
+    "## General linear models\n",
+    "Before we proceed let us study a case from linear algebra where we aim at fitting a set of data $\\hat{y}=[y_0,y_1,\\dots,y_{n-1}]$. We could think of these data as a result of an experiment or a complicated numerical experiment. These data are functions of a series of variables $\\hat{x}=[x_0,x_1,\\dots,x_{n-1}]$, that is $y_i = y(x_i)$ with $i=0,1,2,\\dots,n-1$. The variables $x_i$ could represent physical quantities like time, temperature, position etc. We assume that $y(x)$ is a smooth function. \n",
+    "\n",
+    "Since obtaining these data points may not be trivial, we want to use these data to fit a function which can allow us to make predictions for values of $y$ which are not in the present set. The perhaps simplest approach is to assume we can parametrize our function in terms of a polynomial of degree $n-1$ with $n$ points, that is"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "$$\n",
+    "y=y(x) \\rightarrow y(x_i)=\\tilde{y}_i+\\epsilon_i=\\sum_{j=0}^{n-1} \\beta_i x_i^j+\\epsilon_i,\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "where $\\epsilon_i$ is the error in our approximation.\n",
+    "\n",
+    "\n",
+    "\n",
+    "\n",
+    "## Rewriting the fitting procedure as a linear algebra problem\n",
+    "For every set of values $y_i,x_i$ we have thus the corresponding set of equations"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "$$\n",
+    "\\begin{align*}\n",
+    "y_0&=\\beta_0+\\beta_1x_0^1+\\beta_2x_0^2+\\dots+\\beta_{n-1}x_0^{n-1}+\\epsilon_0\\\\\n",
+    "y_1&=\\beta_0+\\beta_1x_1^1+\\beta_2x_1^2+\\dots+\\beta_{n-1}x_1^{n-1}+\\epsilon_1\\\\\n",
+    "y_2&=\\beta_0+\\beta_1x_2^1+\\beta_2x_2^2+\\dots+\\beta_{n-1}x_2^{n-1}+\\epsilon_2\\\\\n",
+    "\\dots & \\dots \\\\\n",
+    "y_{n-1}&=\\beta_0+\\beta_1x_{n-1}^1+\\beta_2x_{n-1}^2+\\dots+\\beta_1x_{n-1}^{n-1}+\\epsilon_{n-1}.\\\\\n",
+    "\\end{align*}\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Rewriting the fitting procedure as a linear algebra problem, follows\n",
+    "Defining the vectors"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "2\n",
+    " \n",
+    "<\n",
+    "<\n",
+    "<\n",
+    "!\n",
+    "!\n",
+    "M\n",
+    "A\n",
+    "T\n",
+    "H\n",
+    "_\n",
+    "B\n",
+    "L\n",
+    "O\n",
+    "C\n",
+    "K"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "3\n",
+    " \n",
+    "<\n",
+    "<\n",
+    "<\n",
+    "!\n",
+    "!\n",
+    "M\n",
+    "A\n",
+    "T\n",
+    "H\n",
+    "_\n",
+    "B\n",
+    "L\n",
+    "O\n",
+    "C\n",
+    "K"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "$$\n",
+    "\\hat{\\epsilon} = [\\epsilon_0,\\epsilon_1, \\epsilon_2,\\dots, \\epsilon_{n-1}]^T,\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "and the matrix"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "$$\n",
+    "\\hat{X}=\n",
+    "\\begin{bmatrix} \n",
+    "1& x_{0}^1 &x_{0}^2& \\dots & \\dots &x_{0}^{n-1}\\\\\n",
+    "1& x_{1}^1 &x_{1}^2& \\dots & \\dots &x_{1}^{n-1}\\\\\n",
+    "1& x_{2}^1 &x_{2}^2& \\dots & \\dots &x_{2}^{n-1}\\\\                      \n",
+    "\\dots& \\dots &\\dots& \\dots & \\dots &\\dots\\\\\n",
+    "1& x_{n-1}^1 &x_{n-1}^2& \\dots & \\dots &x_{n-1}^{n-1}\\\\\n",
+    "\\end{bmatrix}\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "we can rewrite our equations as"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "$$\n",
+    "\\hat{y} = \\hat{X}\\hat{\\beta}+\\hat{\\epsilon}.\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Generalizing the fitting procedure as a linear algebra problem\n",
+    "We are obviously not limited to the above polynomial. We could replace the various powers of $x$ with elements of Fourier series, that is, instead of $x_i^j$ we could have $\\cos{(j x_i)}$ or $\\sin{(j x_i)}$, or time series or other orthogonal functions.\n",
+    "For every set of values $y_i,x_i$ we can then generalize the equations to"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "$$\n",
+    "\\begin{align*}\n",
+    "y_0&=\\beta_0x_{00}+\\beta_1x_{01}+\\beta_2x_{02}+\\dots+\\beta_{n-1}x_{0n-1}+\\epsilon_0\\\\\n",
+    "y_1&=\\beta_0x_{10}+\\beta_1x_{11}+\\beta_2x_{12}+\\dots+\\beta_{n-1}x_{1n-1}+\\epsilon_1\\\\\n",
+    "y_2&=\\beta_0x_{20}+\\beta_1x_{21}+\\beta_2x_{22}+\\dots+\\beta_{n-1}x_{2n-1}+\\epsilon_2\\\\\n",
+    "\\dots & \\dots \\\\\n",
+    "y_{i}&=\\beta_0x_{i0}+\\beta_1x_{i1}+\\beta_2x_{i2}+\\dots+\\beta_{n-1}x_{in-1}+\\epsilon_i\\\\\n",
+    "\\dots & \\dots \\\\\n",
+    "y_{n-1}&=\\beta_0x_{n-1,0}+\\beta_1x_{n-1,2}+\\beta_2x_{n-1,2}+\\dots+\\beta_1x_{n-1,n-1}+\\epsilon_{n-1}.\\\\\n",
+    "\\end{align*}\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Generalizing the fitting procedure as a linear algebra problem\n",
+    "We redefine in turn the matrix $\\hat{X}$ as"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "$$\n",
+    "\\hat{X}=\n",
+    "\\begin{bmatrix} \n",
+    "x_{00}& x_{01} &x_{02}& \\dots & \\dots &x_{0,n-1}\\\\\n",
+    "x_{10}& x_{11} &x_{12}& \\dots & \\dots &x_{1,n-1}\\\\\n",
+    "x_{20}& x_{21} &x_{22}& \\dots & \\dots &x_{2,n-1}\\\\                      \n",
+    "\\dots& \\dots &\\dots& \\dots & \\dots &\\dots\\\\\n",
+    "x_{n-1,0}& x_{n-1,1} &x_{n-1,2}& \\dots & \\dots &x_{n-1,n-1}\\\\\n",
+    "\\end{bmatrix}\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "and without loss of generality we rewrite again  our equations as"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "$$\n",
+    "\\hat{y} = \\hat{X}\\hat{\\beta}+\\hat{\\epsilon}.\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "The left-hand side of this equation forms know. Our error vector $\\hat{\\epsilon}$ and the parameter vector $\\hat{\\beta}$ are our unknow quantities. How can we obtain the optimal set of $\\beta_i$ values?\n",
+    "\n",
+    "\n",
+    "\n",
+    "\n",
+    "## Optimizing our parameters\n",
+    "We have defined the matrix $\\hat{X}$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "$$\n",
+    "\\begin{align*}\n",
+    "y_0&=\\beta_0x_{00}+\\beta_1x_{01}+\\beta_2x_{02}+\\dots+\\beta_{n-1}x_{0n-1}+\\epsilon_0\\\\\n",
+    "y_1&=\\beta_0x_{10}+\\beta_1x_{11}+\\beta_2x_{12}+\\dots+\\beta_{n-1}x_{1n-1}+\\epsilon_1\\\\\n",
+    "y_2&=\\beta_0x_{20}+\\beta_1x_{21}+\\beta_2x_{22}+\\dots+\\beta_{n-1}x_{2n-1}+\\epsilon_1\\\\\n",
+    "\\dots & \\dots \\\\\n",
+    "y_{i}&=\\beta_0x_{i0}+\\beta_1x_{i1}+\\beta_2x_{i2}+\\dots+\\beta_{n-1}x_{in-1}+\\epsilon_1\\\\\n",
+    "\\dots & \\dots \\\\\n",
+    "y_{n-1}&=\\beta_0x_{n-1,0}+\\beta_1x_{n-1,2}+\\beta_2x_{n-1,2}+\\dots+\\beta_1x_{n-1,n-1}+\\epsilon_{n-1}.\\\\\n",
+    "\\end{align*}\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Optimizing our parameters, more details\n",
+    "We well use this matrix to define the approximation $\\hat{\\tilde{y}}$ via the unknown quantity $\\hat{\\beta}$ as"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "$$\n",
+    "\\hat{\\tilde{y}}= \\hat{X}\\hat{\\beta},\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "and in order to find the optimal parameters $\\beta_i$ instead of solving the above linear algebra problem, we define a function which gives a measure of the spread between the values $y_i$ (which represent hopefully the exact values) and the parametrized values $\\tilde{y}_i$, namely"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "$$\n",
+    "Q(\\hat{\\beta})=\\sum_{i=0}^{n-1}\\left(y_i-\\tilde{y}_i\\right)^2=\\left(\\hat{y}-\\hat{\\tilde{y}}\\right)^T\\left(\\hat{y}-\\hat{\\tilde{y}}\\right),\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "or using the matrix $\\hat{X}$ as"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "$$\n",
+    "Q(\\hat{\\beta})=\\left(\\hat{y}-\\hat{X}\\hat{\\beta}\\right)^T\\left(\\hat{y}-\\hat{X}\\hat{\\beta}\\right).\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Interpretations and optimizing our parameters\n",
+    "The function"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "$$\n",
+    "Q(\\hat{\\beta})=\\left(\\hat{y}-\\hat{X}\\hat{\\beta}\\right)^T\\left(\\hat{y}-\\hat{X}\\hat{\\beta}\\right),\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "can be linked to the variance of the quantity $y_i$ if we interpret the latter as the mean value of for example a numerical  experiment. When linking below with the maximum likelihood approach below, we will indeed interpret $y_i$ as a mean value"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "$$\n",
+    "y_{i}=\\langle y_i \\rangle = \\beta_0x_{i,0}+\\beta_1x_{i,1}+\\beta_2x_{i,2}+\\dots+\\beta_{n-1}x_{i,n-1}+\\epsilon_i,\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "where $\\langle y_i \\rangle$ is the mean value. Keep in mind also that till now  we have treated $y_i$ as the exact value. Normally, the response (dependent or outcome) variable $y_i$ the outcome of a numerical experiment or another type of experiment and is thus only an approximation to the true value. It is then always accompanied by an error estimate, often limited to a statistical error estimate given by the standard deviation discussed earlier. In the discussion here we will treat $y_i$ as our exact value for the response variable.\n",
+    "\n",
+    "In order to find the parameters $\\beta_i$ we will then minimize the spread of $Q(\\hat{\\beta})$ by requiring"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "$$\n",
+    "\\frac{\\partial Q(\\hat{\\beta})}{\\partial \\beta_j} = \\frac{\\partial }{\\partial \\beta_j}\\left[ \\sum_{i=0}^{n-1}\\left(y_i-\\beta_0x_{i,0}-\\beta_1x_{i,1}-\\beta_2x_{i,2}-\\dots-\\beta_{n-1}x_{i,n-1}\\right)^2\\right]=0,\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "which results in"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "$$\n",
+    "\\frac{\\partial Q(\\hat{\\beta})}{\\partial \\beta_j} = -2\\left[ \\sum_{i=0}^{n-1}x_{ij}\\left(y_i-\\beta_0x_{i,0}-\\beta_1x_{i,1}-\\beta_2x_{i,2}-\\dots-\\beta_{n-1}x_{i,n-1}\\right)\\right]=0,\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "or in a matrix-vector form as"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "$$\n",
+    "\\frac{\\partial Q(\\hat{\\beta})}{\\partial \\hat{\\beta}} = 0 = \\hat{X}^T\\left( \\hat{y}-\\hat{X}\\hat{\\beta}\\right).\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Interpretations and optimizing our parameters\n",
+    "We can rewrite"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "$$\n",
+    "\\frac{\\partial Q(\\hat{\\beta})}{\\partial \\hat{\\beta}} = 0 = \\hat{X}^T\\left( \\hat{y}-\\hat{X}\\hat{\\beta}\\right),\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "as"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "$$\n",
+    "\\hat{X}^T\\hat{y} = \\hat{X}^T\\hat{X}\\hat{\\beta},\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "and if the matrix $\\hat{X}^T\\hat{X}$ is invertible we have the solution"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "$$\n",
+    "\\hat{\\beta} =\\left(\\hat{X}^T\\hat{X}\\right)^{-1}\\hat{X}^T\\hat{y}.\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Interpretations and optimizing our parameters\n",
+    "The residuals $\\hat{\\epsilon}$ are in turn given by"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "$$\n",
+    "\\hat{\\epsilon} = \\hat{y}-\\hat{\\tilde{y}} = \\hat{y}-\\hat{X}\\hat{\\beta},\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "and with"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "$$\n",
+    "\\hat{X}^T\\left( \\hat{y}-\\hat{X}\\hat{\\beta}\\right)= 0,\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "we have"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "$$\n",
+    "\\hat{X}^T\\hat{\\epsilon}=\\hat{X}^T\\left( \\hat{y}-\\hat{X}\\hat{\\beta}\\right)= 0,\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "meaning that the solution for $\\hat{\\beta}$ is the one which minimizes the residuals.  Later we will link this with the maximum likelihood approach.\n",
+    "\n",
+    "\n",
+    "\n",
+    "## The $\\chi^2$ function\n",
+    "\n",
+    "Normally, the response (dependent or outcome) variable $y_i$ the outcome of a numerical experiment or another type of experiment and is thus only an approximation to the true value. It is then always accompanied by an error estimate, often limited to a statistical error estimate given by the standard deviation discussed earlier. In the discussion here we will treat $y_i$ as our exact value for the response variable.\n",
+    "\n",
+    "Introducing the standard deviation $\\sigma_i$ for each measurement $y_i$, we define now the $\\chi^2$ function as"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "$$\n",
+    "\\chi^2(\\hat{\\beta})=\\sum_{i=0}^{n-1}\\frac{\\left(y_i-\\tilde{y}_i\\right)^2}{\\sigma_i^2}=\\left(\\hat{y}-\\hat{\\tilde{y}}\\right)^T\\frac{1}{\\hat{\\Sigma^2}}\\left(\\hat{y}-\\hat{\\tilde{y}}\\right),\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "where the matrix $\\hat{\\Sigma}$ is a diagonal matrix with $\\sigma_i$ as matrix elements.\n",
+    "\n",
+    "\n",
+    "\n",
+    "## The $\\chi^2$ function\n",
+    "\n",
+    "In order to find the parameters $\\beta_i$ we will then minimize the spread of $\\chi^2(\\hat{\\beta})$ by requiring"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "$$\n",
+    "\\frac{\\partial \\chi^2(\\hat{\\beta})}{\\partial \\beta_j} = \\frac{\\partial }{\\partial \\beta_j}\\left[ \\sum_{i=0}^{n-1}\\left(\\frac{y_i-\\beta_0x_{i,0}-\\beta_1x_{i,1}-\\beta_2x_{i,2}-\\dots-\\beta_{n-1}x_{i,n-1}}{\\sigma_i}\\right)^2\\right]=0,\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "which results in"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "$$\n",
+    "\\frac{\\partial \\chi^2(\\hat{\\beta})}{\\partial \\beta_j} = -2\\left[ \\sum_{i=0}^{n-1}\\frac{x_{ij}}{\\sigma_i}\\left(\\frac{y_i-\\beta_0x_{i,0}-\\beta_1x_{i,1}-\\beta_2x_{i,2}-\\dots-\\beta_{n-1}x_{i,n-1}}{\\sigma_i}\\right)\\right]=0,\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "or in a matrix-vector form as"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "$$\n",
+    "\\frac{\\partial \\chi^2(\\hat{\\beta})}{\\partial \\hat{\\beta}} = 0 = \\hat{A}^T\\left( \\hat{b}-\\hat{A}\\hat{\\beta}\\right).\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "where we have defined the matrix $\\hat{A} =\\hat{X}/\\hat{\\Sigma}$ with matrix elements $a_{ij} = x_{ij}/\\sigma_i$ and the vector $\\hat{b}$ with elements $b_i = y_i/\\sigma_i$.\n",
+    "\n",
+    "\n",
+    "\n",
+    "## The $\\chi^2$ function\n",
+    "\n",
+    "We can rewrite"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "$$\n",
+    "\\frac{\\partial \\chi^2(\\hat{\\beta})}{\\partial \\hat{\\beta}} = 0 = \\hat{A}^T\\left( \\hat{b}-\\hat{A}\\hat{\\beta}\\right),\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "as"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "$$\n",
+    "\\hat{A}^T\\hat{b} = \\hat{A}^T\\hat{A}\\hat{\\beta},\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "and if the matrix $\\hat{A}^T\\hat{A}$ is invertible we have the solution"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "$$\n",
+    "\\hat{\\beta} =\\left(\\hat{A}^T\\hat{A}\\right)^{-1}\\hat{A}^T\\hat{b}.\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## The $\\chi^2$ function\n",
+    "\n",
+    "If we then introduce the matrix"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "$$\n",
+    "\\hat{H} =  \\hat{A}^T\\hat{A},\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "we have then the following expression for the parameters $\\beta_j$ (the matrix elements of $\\hat{H}$ are $h_{ij}$)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "$$\n",
+    "\\beta_j = \\sum_{k=0}^{p-1}h_{jk}\\sum_{i=0}^{n-1}\\frac{y_i}{\\sigma_i}\\frac{x_{ik}}{\\sigma_i} = \\sum_{k=0}^{p-1}h_{jk}\\sum_{i=0}^{n-1}b_ia_{ik}\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "We state without proof the expression for the uncertainty  in the parameters $\\beta_j$ as"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "$$\n",
+    "\\sigma^2(\\beta_j) = \\sum_{i=0}^{n-1}\\sigma_i^2\\left( \\frac{\\partial \\beta_j}{\\partial y_i}\\right)^2,\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "resulting in"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "$$\n",
+    "\\sigma^2(\\beta_j) = \\left(\\sum_{k=0}^{p-1}h_{jk}\\sum_{i=0}^{n-1}a_{ik}\\right)\\left(\\sum_{l=0}^{p-1}h_{jl}\\sum_{m=0}^{n-1}a_{ml}\\right) = h_{jj}!\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## The $\\chi^2$ function\n",
+    "The first step here is to approximate the function $y$ with a first-order polynomial, that is we write"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "$$\n",
+    "y=y(x) \\rightarrow y(x_i) \\approx \\beta_0+\\beta_1 x_i.\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "By computing the derivatives of $\\chi^2$ with respect to $\\beta_0$ and $\\beta_1$ show that these are given by"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "$$\n",
+    "\\frac{\\partial \\chi^2(\\hat{\\beta})}{\\partial \\beta_0} = -2\\left[ \\sum_{i=0}^{1}\\left(\\frac{y_i-\\beta_0-\\beta_1x_{i}}{\\sigma_i^2}\\right)\\right]=0,\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "and"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "$$\n",
+    "\\frac{\\partial \\chi^2(\\hat{\\beta})}{\\partial \\beta_0} = -2\\left[ \\sum_{i=0}^{1}x_i\\left(\\frac{y_i-\\beta_0-\\beta_1x_{i}}{\\sigma_i^2}\\right)\\right]=0.\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## The $\\chi^2$ function\n",
+    "\n",
+    "We define then"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "$$\n",
+    "\\gamma =  \\sum_{i=0}^{1}\\frac{1}{\\sigma_i^2},\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "4\n",
+    "0\n",
+    " \n",
+    "<\n",
+    "<\n",
+    "<\n",
+    "!\n",
+    "!\n",
+    "M\n",
+    "A\n",
+    "T\n",
+    "H\n",
+    "_\n",
+    "B\n",
+    "L\n",
+    "O\n",
+    "C\n",
+    "K"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "4\n",
+    "1\n",
+    " \n",
+    "<\n",
+    "<\n",
+    "<\n",
+    "!\n",
+    "!\n",
+    "M\n",
+    "A\n",
+    "T\n",
+    "H\n",
+    "_\n",
+    "B\n",
+    "L\n",
+    "O\n",
+    "C\n",
+    "K"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "4\n",
+    "2\n",
+    " \n",
+    "<\n",
+    "<\n",
+    "<\n",
+    "!\n",
+    "!\n",
+    "M\n",
+    "A\n",
+    "T\n",
+    "H\n",
+    "_\n",
+    "B\n",
+    "L\n",
+    "O\n",
+    "C\n",
+    "K"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "$$\n",
+    "\\gamma_{xy} = \\sum_{i=0}^{1}\\frac{y_ix_{i}}{\\sigma_i^2},\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "and show that"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "4\n",
+    "4\n",
+    " \n",
+    "<\n",
+    "<\n",
+    "<\n",
+    "!\n",
+    "!\n",
+    "M\n",
+    "A\n",
+    "T\n",
+    "H\n",
+    "_\n",
+    "B\n",
+    "L\n",
+    "O\n",
+    "C\n",
+    "K"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "$$\n",
+    "\\beta_1 = \\frac{\\gamma_{xy}\\gamma-\\gamma_x\\gamma_y}{\\gamma\\gamma_{xx}-\\gamma_x^2}.\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "The LSM suffers often from both being underdetermined and overdetermined in the unknown coefficients $\\beta_i$.  A better approach is to use the Singular Value Decomposition (SVD) method discussed below.\n",
+    "\n",
+    "\n",
+    "\n",
+    "\n",
+    "\n",
+    "## The singular value decompostion\n",
+    "How can we use the singular value decomposition to find the parameters $\\beta_j$? More details will come. We first note that a general $m\\times n$ matrix $\\hat{A}$ can be written in terms of a diagonal matrix $\\hat{\\Sigma}$ of dimensionality $n\\times n$ and two orthognal matrices $\\hat{U}$ and $\\hat{V}$, where the first has dimensionality $m \\times n$ and the last dimensionality $n\\times n$. We have then"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "$$\n",
+    "\\hat{A} = \\hat{U}\\hat{\\Sigma}\\hat{V}\n",
+    "$$"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.6.5"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}
diff --git a/doc/pub/Reinforce/html/._Reinforce-bs002.html b/doc/pub/Reinforce/html/._Reinforce-bs002.html
new file mode 100644
index 000000000..b95a62053
--- /dev/null
+++ b/doc/pub/Reinforce/html/._Reinforce-bs002.html
@@ -0,0 +1,227 @@
+
+
+
+
+
+
+
+
+Data Analysis and Machine Learning: Reinforcement Learning
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+    
+
+

+  

+    
+    Data Analysis and Machine Learning: Reinforcement Learning
+  

+
+  

+    
+  

+

+
 
+
+

+
+
 
 
 
 
+
+
+
+
+
Code example 

+

+
+
+
"""
+A simple example for Reinforcement Learning using table lookup Q-learning method.
+An agent "o" is on the left of a 1 dimensional world, the treasure is on the rightmost location.
+Run this program and to see how the agent will improve its strategy of finding the treasure.
+View more on my tutorial page: https://morvanzhou.github.io/tutorials/
+"""
+
+import numpy as np
+import pandas as pd
+import time
+
+np.random.seed(2)  # reproducible
+
+
+N_STATES = 6   # the length of the 1 dimensional world
+ACTIONS = ['left', 'right']     # available actions
+EPSILON = 0.9   # greedy police
+ALPHA = 0.1     # learning rate
+GAMMA = 0.9    # discount factor
+MAX_EPISODES = 13   # maximum episodes
+FRESH_TIME = 0.3    # fresh time for one move
+
+
+def build_q_table(n_states, actions):
+    table = pd.DataFrame(
+        np.zeros((n_states, len(actions))),     # q_table initial values
+        columns=actions,    # actions's name
+    )
+    # print(table)    # show table
+    return table
+
+
+def choose_action(state, q_table):
+    # This is how to choose an action
+    state_actions = q_table.iloc[state, :]
+    if (np.random.uniform() > EPSILON) or ((state_actions == 0).all()):  # act non-greedy or state-action have no value
+        action_name = np.random.choice(ACTIONS)
+    else:   # act greedy
+        action_name = state_actions.idxmax()    # replace argmax to idxmax as argmax means a different function in newer version of pandas
+    return action_name
+
+
+def get_env_feedback(S, A):
+    # This is how agent will interact with the environment
+    if A == 'right':    # move right
+        if S == N_STATES - 2:   # terminate
+            S_ = 'terminal'
+            R = 1
+        else:
+            S_ = S + 1
+            R = 0
+    else:   # move left
+        R = 0
+        if S == 0:
+            S_ = S  # reach the wall
+        else:
+            S_ = S - 1
+    return S_, R
+
+
+def update_env(S, episode, step_counter):
+    # This is how environment be updated
+    env_list = ['-']*(N_STATES-1) + ['T']   # '---------T' our environment
+    if S == 'terminal':
+        interaction = 'Episode %s: total_steps = %s' % (episode+1, step_counter)
+        print('\r{}'.format(interaction), end='')
+        time.sleep(2)
+        print('\r                                ', end='')
+    else:
+        env_list[S] = 'o'
+        interaction = ''.join(env_list)
+        print('\r{}'.format(interaction), end='')
+        time.sleep(FRESH_TIME)
+
+
+def rl():
+    # main part of RL loop
+    q_table = build_q_table(N_STATES, ACTIONS)
+    for episode in range(MAX_EPISODES):
+        step_counter = 0
+        S = 0
+        is_terminated = False
+        update_env(S, episode, step_counter)
+        while not is_terminated:
+
+            A = choose_action(S, q_table)
+            S_, R = get_env_feedback(S, A)  # take action & get next state and reward
+            q_predict = q_table.loc[S, A]
+            if S_ != 'terminal':
+                q_target = R + GAMMA * q_table.iloc[S_, :].max()   # next state is not terminal
+            else:
+                q_target = R     # next state is terminal
+                is_terminated = True    # terminate this episode
+
+            q_table.loc[S, A] += ALPHA * (q_target - q_predict)  # update
+            S = S_  # move to next state
+
+            update_env(S, episode, step_counter+1)
+            step_counter += 1
+    return q_table
+
+
+if __name__ == "__main__":
+    q_table = rl()
+    print('\r\nQ-table:\n')
+print(q_table)
+

+

+
+

+
+

+
+
+
  
+
+
+
+
+
+
+
+

+
+

+
+
+
+
+    
+
diff --git a/doc/pub/Reinforce/html/reveal.js/plugin/leap/leap.js b/doc/pub/Reinforce/html/reveal.js/plugin/leap/leap.js
new file mode 100644
index 000000000..48084ffb0
--- /dev/null
+++ b/doc/pub/Reinforce/html/reveal.js/plugin/leap/leap.js
@@ -0,0 +1,159 @@
+/*
+ * Copyright (c) 2013, Leap Motion, Inc.
+ * All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:
+ *
+ * Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.
+ * Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+ *
+ * Version 0.2.0 - http://js.leapmotion.com/0.2.0/leap.min.js
+ * Grab latest versions from http://js.leapmotion.com/
+ */
+
+!function(e,t,n){function i(n,s){if(!t[n]){if(!e[n]){var o=typeof require=="function"&&require;if(!s&&o)return o(n,!0);if(r)return r(n,!0);throw new Error("Cannot find module '"+n+"'")}var u=t[n]={exports:{}};e[n][0].call(u.exports,function(t){var r=e[n][1][t];return i(r?r:t)},u,u.exports)}return t[n].exports}var r=typeof require=="function"&&require;for(var s=0;s=this.size)return undefined;if(i>=this._buf.length)return undefined;return this._buf[(this.pos-i-1)%this.size]};CircularBuffer.prototype.push=function(o){this._buf[this.pos%this.size]=o;return this.pos++}},{}],3:[function(require,module,exports){var Connection=module.exports=require("./base_connection");Connection.prototype.setupSocket=function(){var connection=this;var socket=new WebSocket(this.getUrl());socket.onopen=function(){connection.handleOpen()};socket.onmessage=function(message){connection.handleData(message.data)};socket.onclose=function(){connection.handleClose()};return socket};Connection.prototype.startHeartbeat=function(){if(!this.protocol.sendHeartbeat||this.heartbeatTimer)return;var connection=this;var propertyName=null;if(typeof document.hidden!=="undefined"){propertyName="hidden"}else if(typeof document.mozHidden!=="undefined"){propertyName="mozHidden"}else if(typeof document.msHidden!=="undefined"){propertyName="msHidden"}else if(typeof document.webkitHidden!=="undefined"){propertyName="webkitHidden"}else{propertyName=undefined}var windowVisible=true;var focusListener=window.addEventListener("focus",function(e){windowVisible=true});var blurListener=window.addEventListener("blur",function(e){windowVisible=false});this.on("disconnect",function(){if(connection.heartbeatTimer){clearTimeout(connection.heartbeatTimer);delete connection.heartbeatTimer}window.removeEventListener(focusListener);window.removeEventListener(blurListener)});this.heartbeatTimer=setInterval(function(){var isVisible=propertyName===undefined?true:document[propertyName]===false;if(isVisible&&windowVisible){connection.sendHeartbeat()}else{connection.setHeartbeatState(false)}},this.opts.heartbeatInterval)}},{"./base_connection":1}],4:[function(require,module,exports){!function(process){var Frame=require("./frame"),CircularBuffer=require("./circular_buffer"),Pipeline=require("./pipeline"),EventEmitter=require("events").EventEmitter,gestureListener=require("./gesture").gestureListener,_=require("underscore");var Controller=module.exports=function(opts){var inNode=typeof process!=="undefined"&&process.title==="node";opts=_.defaults(opts||{},{inNode:inNode});this.inNode=opts.inNode;opts=_.defaults(opts||{},{frameEventName:this.useAnimationLoop()?"animationFrame":"deviceFrame",supressAnimationLoop:false});this.supressAnimationLoop=opts.supressAnimationLoop;this.frameEventName=opts.frameEventName;this.history=new CircularBuffer(200);this.lastFrame=Frame.Invalid;this.lastValidFrame=Frame.Invalid;this.lastConnectionFrame=Frame.Invalid;this.accumulatedGestures=[];if(opts.connectionType===undefined){this.connectionType=this.inBrowser()?require("./connection"):require("./node_connection")}else{this.connectionType=opts.connectionType}this.connection=new this.connectionType(opts);this.setupConnectionEvents()};Controller.prototype.gesture=function(type,cb){var creator=gestureListener(this,type);if(cb!==undefined){creator.stop(cb)}return creator};Controller.prototype.inBrowser=function(){return!this.inNode};Controller.prototype.useAnimationLoop=function(){return this.inBrowser()&&typeof chrome==="undefined"};Controller.prototype.connect=function(){var controller=this;if(this.connection.connect()&&this.inBrowser()&&!controller.supressAnimationLoop){var callback=function(){controller.emit("animationFrame",controller.lastConnectionFrame);window.requestAnimFrame(callback)};window.requestAnimFrame(callback)}};Controller.prototype.disconnect=function(){this.connection.disconnect()};Controller.prototype.frame=function(num){return this.history.get(num)||Frame.Invalid};Controller.prototype.loop=function(callback){switch(callback.length){case 1:this.on(this.frameEventName,callback);break;case 2:var controller=this;var scheduler=null;var immediateRunnerCallback=function(frame){callback(frame,function(){if(controller.lastFrame!=frame){immediateRunnerCallback(controller.lastFrame)}else{controller.once(controller.frameEventName,immediateRunnerCallback)}})};this.once(this.frameEventName,immediateRunnerCallback);break}this.connect()};Controller.prototype.addStep=function(step){if(!this.pipeline)this.pipeline=new Pipeline(this);this.pipeline.addStep(step)};Controller.prototype.processFrame=function(frame){if(frame.gestures){this.accumulatedGestures=this.accumulatedGestures.concat(frame.gestures)}if(this.pipeline){frame=this.pipeline.run(frame);if(!frame)frame=Frame.Invalid}this.lastConnectionFrame=frame;this.emit("deviceFrame",frame)};Controller.prototype.processFinishedFrame=function(frame){this.lastFrame=frame;if(frame.valid){this.lastValidFrame=frame}frame.controller=this;frame.historyIdx=this.history.push(frame);if(frame.gestures){frame.gestures=this.accumulatedGestures;this.accumulatedGestures=[];for(var gestureIdx=0;gestureIdx!=frame.gestures.length;gestureIdx++){this.emit("gesture",frame.gestures[gestureIdx],frame)}}this.emit("frame",frame)};Controller.prototype.setupConnectionEvents=function(){var controller=this;this.connection.on("frame",function(frame){controller.processFrame(frame)});this.on(this.frameEventName,function(frame){controller.processFinishedFrame(frame)});this.connection.on("disconnect",function(){controller.emit("disconnect")});this.connection.on("ready",function(){controller.emit("ready")});this.connection.on("connect",function(){controller.emit("connect")});this.connection.on("focus",function(){controller.emit("focus")});this.connection.on("blur",function(){controller.emit("blur")});this.connection.on("protocol",function(protocol){controller.emit("protocol",protocol)});this.connection.on("deviceConnect",function(evt){controller.emit(evt.state?"deviceConnected":"deviceDisconnected")})};_.extend(Controller.prototype,EventEmitter.prototype)}(require("__browserify_process"))},{"./circular_buffer":2,"./connection":3,"./frame":5,"./gesture":6,"./node_connection":16,"./pipeline":10,__browserify_process:18,events:17,underscore:20}],5:[function(require,module,exports){var Hand=require("./hand"),Pointable=require("./pointable"),createGesture=require("./gesture").createGesture,glMatrix=require("gl-matrix"),mat3=glMatrix.mat3,vec3=glMatrix.vec3,InteractionBox=require("./interaction_box"),_=require("underscore");var Frame=module.exports=function(data){this.valid=true;this.id=data.id;this.timestamp=data.timestamp;this.hands=[];this.handsMap={};this.pointables=[];this.tools=[];this.fingers=[];if(data.interactionBox){this.interactionBox=new InteractionBox(data.interactionBox)}this.gestures=[];this.pointablesMap={};this._translation=data.t;this._rotation=_.flatten(data.r);this._scaleFactor=data.s;this.data=data;this.type="frame";this.currentFrameRate=data.currentFrameRate;var handMap={};for(var handIdx=0,handCount=data.hands.length;handIdx!=handCount;handIdx++){var hand=new Hand(data.hands[handIdx]);hand.frame=this;this.hands.push(hand);this.handsMap[hand.id]=hand;handMap[hand.id]=handIdx}for(var pointableIdx=0,pointableCount=data.pointables.length;pointableIdx!=pointableCount;pointableIdx++){var pointable=new Pointable(data.pointables[pointableIdx]);pointable.frame=this;this.pointables.push(pointable);this.pointablesMap[pointable.id]=pointable;(pointable.tool?this.tools:this.fingers).push(pointable);if(pointable.handId!==undefined&&handMap.hasOwnProperty(pointable.handId)){var hand=this.hands[handMap[pointable.handId]];hand.pointables.push(pointable);(pointable.tool?hand.tools:hand.fingers).push(pointable)}}if(data.gestures){for(var gestureIdx=0,gestureCount=data.gestures.length;gestureIdx!=gestureCount;gestureIdx++){this.gestures.push(createGesture(data.gestures[gestureIdx]))}}};Frame.prototype.tool=function(id){var pointable=this.pointable(id);return pointable.tool?pointable:Pointable.Invalid};Frame.prototype.pointable=function(id){return this.pointablesMap[id]||Pointable.Invalid};Frame.prototype.finger=function(id){var pointable=this.pointable(id);return!pointable.tool?pointable:Pointable.Invalid};Frame.prototype.hand=function(id){return this.handsMap[id]||Hand.Invalid};Frame.prototype.rotationAngle=function(sinceFrame,axis){if(!this.valid||!sinceFrame.valid)return 0;var rot=this.rotationMatrix(sinceFrame);var cs=(rot[0]+rot[4]+rot[8]-1)*.5;var angle=Math.acos(cs);angle=isNaN(angle)?0:angle;if(axis!==undefined){var rotAxis=this.rotationAxis(sinceFrame);angle*=vec3.dot(rotAxis,vec3.normalize(vec3.create(),axis))}return angle};Frame.prototype.rotationAxis=function(sinceFrame){if(!this.valid||!sinceFrame.valid)return vec3.create();return vec3.normalize(vec3.create(),[this._rotation[7]-sinceFrame._rotation[5],this._rotation[2]-sinceFrame._rotation[6],this._rotation[3]-sinceFrame._rotation[1]])};Frame.prototype.rotationMatrix=function(sinceFrame){if(!this.valid||!sinceFrame.valid)return mat3.create();var transpose=mat3.transpose(mat3.create(),this._rotation);return mat3.multiply(mat3.create(),sinceFrame._rotation,transpose)};Frame.prototype.scaleFactor=function(sinceFrame){if(!this.valid||!sinceFrame.valid)return 1;return Math.exp(this._scaleFactor-sinceFrame._scaleFactor)};Frame.prototype.translation=function(sinceFrame){if(!this.valid||!sinceFrame.valid)return vec3.create();return vec3.subtract(vec3.create(),this._translation,sinceFrame._translation)};Frame.prototype.toString=function(){var str="Frame [ id:"+this.id+" | timestamp:"+this.timestamp+" | Hand count:("+this.hands.length+") | Pointable count:("+this.pointables.length+")";if(this.gestures)str+=" | Gesture count:("+this.gestures.length+")";str+=" ]";return str};Frame.prototype.dump=function(){var out="";out+="Frame Info:
";out+=this.toString();out+="

Hands:
";for(var handIdx=0,handCount=this.hands.length;handIdx!=handCount;handIdx++){out+="  "+this.hands[handIdx].toString()+"
"}out+="

Pointables:
";for(var pointableIdx=0,pointableCount=this.pointables.length;pointableIdx!=pointableCount;pointableIdx++){out+="  "+this.pointables[pointableIdx].toString()+"
"}if(this.gestures){out+="

Gestures:
";for(var gestureIdx=0,gestureCount=this.gestures.length;gestureIdx!=gestureCount;gestureIdx++){out+="  "+this.gestures[gestureIdx].toString()+"
"}}out+="

Raw JSON:
";out+=JSON.stringify(this.data);return out};Frame.Invalid={valid:false,hands:[],fingers:[],tools:[],gestures:[],pointables:[],pointable:function(){return Pointable.Invalid},finger:function(){return Pointable.Invalid},hand:function(){return Hand.Invalid},toString:function(){return"invalid frame"},dump:function(){return this.toString()},rotationAngle:function(){return 0},rotationMatrix:function(){return mat3.create()},rotationAxis:function(){return vec3.create()},scaleFactor:function(){return 1},translation:function(){return vec3.create()}}},{"./gesture":6,"./hand":7,"./interaction_box":9,"./pointable":11,"gl-matrix":19,underscore:20}],6:[function(require,module,exports){var glMatrix=require("gl-matrix"),vec3=glMatrix.vec3,EventEmitter=require("events").EventEmitter,_=require("underscore");var createGesture=exports.createGesture=function(data){var gesture;switch(data.type){case"circle":gesture=new CircleGesture(data);break;case"swipe":gesture=new SwipeGesture(data);break;case"screenTap":gesture=new ScreenTapGesture(data);break;case"keyTap":gesture=new KeyTapGesture(data);break;default:throw"unkown gesture type"}gesture.id=data.id;gesture.handIds=data.handIds;gesture.pointableIds=data.pointableIds;gesture.duration=data.duration;gesture.state=data.state;gesture.type=data.type;return gesture};var gestureListener=exports.gestureListener=function(controller,type){var handlers={};var gestureMap={};var gestureCreator=function(){var candidateGesture=gestureMap[gesture.id];if(candidateGesture!==undefined)gesture.update(gesture,frame);if(gesture.state=="start"||gesture.state=="stop"){if(type==gesture.type&&gestureMap[gesture.id]===undefined){gestureMap[gesture.id]=new Gesture(gesture,frame);gesture.update(gesture,frame)}if(gesture.state=="stop"){delete gestureMap[gesture.id]}}};controller.on("gesture",function(gesture,frame){if(gesture.type==type){if(gesture.state=="start"||gesture.state=="stop"){if(gestureMap[gesture.id]===undefined){var gestureTracker=new Gesture(gesture,frame);gestureMap[gesture.id]=gestureTracker;_.each(handlers,function(cb,name){gestureTracker.on(name,cb)})}}gestureMap[gesture.id].update(gesture,frame);if(gesture.state=="stop"){delete gestureMap[gesture.id]}}});var builder={start:function(cb){handlers["start"]=cb;return builder},stop:function(cb){handlers["stop"]=cb;return builder},complete:function(cb){handlers["stop"]=cb;return builder},update:function(cb){handlers["update"]=cb;return builder}};return builder};var Gesture=exports.Gesture=function(gesture,frame){this.gestures=[gesture];this.frames=[frame]};Gesture.prototype.update=function(gesture,frame){this.gestures.push(gesture);this.frames.push(frame);this.emit(gesture.state,this)};_.extend(Gesture.prototype,EventEmitter.prototype);var CircleGesture=function(data){this.center=data.center;this.normal=data.normal;this.progress=data.progress;this.radius=data.radius};CircleGesture.prototype.toString=function(){return"CircleGesture ["+JSON.stringify(this)+"]"};var SwipeGesture=function(data){this.startPosition=data.startPosition;this.position=data.position;this.direction=data.direction;this.speed=data.speed};SwipeGesture.prototype.toString=function(){return"SwipeGesture ["+JSON.stringify(this)+"]"};var ScreenTapGesture=function(data){this.position=data.position;this.direction=data.direction;this.progress=data.progress};ScreenTapGesture.prototype.toString=function(){return"ScreenTapGesture ["+JSON.stringify(this)+"]"};var KeyTapGesture=function(data){this.position=data.position;this.direction=data.direction;this.progress=data.progress};KeyTapGesture.prototype.toString=function(){return"KeyTapGesture ["+JSON.stringify(this)+"]"}},{events:17,"gl-matrix":19,underscore:20}],7:[function(require,module,exports){var Pointable=require("./pointable"),glMatrix=require("gl-matrix"),mat3=glMatrix.mat3,vec3=glMatrix.vec3,_=require("underscore");var Hand=module.exports=function(data){this.id=data.id;this.palmPosition=data.palmPosition;this.direction=data.direction;this.palmVelocity=data.palmVelocity;this.palmNormal=data.palmNormal;this.sphereCenter=data.sphereCenter;this.sphereRadius=data.sphereRadius;this.valid=true;this.pointables=[];this.fingers=[];this.tools=[];this._translation=data.t;this._rotation=_.flatten(data.r);this._scaleFactor=data.s;this.timeVisible=data.timeVisible;this.stabilizedPalmPosition=data.stabilizedPalmPosition};Hand.prototype.finger=function(id){var finger=this.frame.finger(id);return finger&&finger.handId==this.id?finger:Pointable.Invalid};Hand.prototype.rotationAngle=function(sinceFrame,axis){if(!this.valid||!sinceFrame.valid)return 0;var sinceHand=sinceFrame.hand(this.id);if(!sinceHand.valid)return 0;var rot=this.rotationMatrix(sinceFrame);var cs=(rot[0]+rot[4]+rot[8]-1)*.5;var angle=Math.acos(cs);angle=isNaN(angle)?0:angle;if(axis!==undefined){var rotAxis=this.rotationAxis(sinceFrame);angle*=vec3.dot(rotAxis,vec3.normalize(vec3.create(),axis))}return angle};Hand.prototype.rotationAxis=function(sinceFrame){if(!this.valid||!sinceFrame.valid)return vec3.create();var sinceHand=sinceFrame.hand(this.id);if(!sinceHand.valid)return vec3.create();return vec3.normalize(vec3.create(),[this._rotation[7]-sinceHand._rotation[5],this._rotation[2]-sinceHand._rotation[6],this._rotation[3]-sinceHand._rotation[1]])};Hand.prototype.rotationMatrix=function(sinceFrame){if(!this.valid||!sinceFrame.valid)return mat3.create();var sinceHand=sinceFrame.hand(this.id);if(!sinceHand.valid)return mat3.create();var transpose=mat3.transpose(mat3.create(),this._rotation);var m=mat3.multiply(mat3.create(),sinceHand._rotation,transpose);return m};Hand.prototype.scaleFactor=function(sinceFrame){if(!this.valid||!sinceFrame.valid)return 1;var sinceHand=sinceFrame.hand(this.id);if(!sinceHand.valid)return 1;return Math.exp(this._scaleFactor-sinceHand._scaleFactor)};Hand.prototype.translation=function(sinceFrame){if(!this.valid||!sinceFrame.valid)return vec3.create();var sinceHand=sinceFrame.hand(this.id);if(!sinceHand.valid)return vec3.create();return[this._translation[0]-sinceHand._translation[0],this._translation[1]-sinceHand._translation[1],this._translation[2]-sinceHand._translation[2]]};Hand.prototype.toString=function(){return"Hand [ id: "+this.id+" | palm velocity:"+this.palmVelocity+" | sphere center:"+this.sphereCenter+" ] "};Hand.Invalid={valid:false,fingers:[],tools:[],pointables:[],pointable:function(){return Pointable.Invalid},finger:function(){return Pointable.Invalid},toString:function(){return"invalid frame"},dump:function(){return this.toString()},rotationAngle:function(){return 0},rotationMatrix:function(){return mat3.create()},rotationAxis:function(){return vec3.create()},scaleFactor:function(){return 1},translation:function(){return vec3.create()}}},{"./pointable":11,"gl-matrix":19,underscore:20}],8:[function(require,module,exports){!function(){module.exports={Controller:require("./controller"),Frame:require("./frame"),Gesture:require("./gesture"),Hand:require("./hand"),Pointable:require("./pointable"),InteractionBox:require("./interaction_box"),Connection:require("./connection"),CircularBuffer:require("./circular_buffer"),UI:require("./ui"),glMatrix:require("gl-matrix"),mat3:require("gl-matrix").mat3,vec3:require("gl-matrix").vec3,loopController:undefined,loop:function(opts,callback){if(callback===undefined){callback=opts;opts={}}if(!this.loopController)this.loopController=new this.Controller(opts);this.loopController.loop(callback)}}}()},{"./circular_buffer":2,"./connection":3,"./controller":4,"./frame":5,"./gesture":6,"./hand":7,"./interaction_box":9,"./pointable":11,"./ui":13,"gl-matrix":19}],9:[function(require,module,exports){var glMatrix=require("gl-matrix"),vec3=glMatrix.vec3;var InteractionBox=module.exports=function(data){this.valid=true;this.center=data.center;this.size=data.size;this.width=data.size[0];this.height=data.size[1];this.depth=data.size[2]};InteractionBox.prototype.denormalizePoint=function(normalizedPosition){return vec3.fromValues((normalizedPosition[0]-.5)*this.size[0]+this.center[0],(normalizedPosition[1]-.5)*this.size[1]+this.center[1],(normalizedPosition[2]-.5)*this.size[2]+this.center[2])};InteractionBox.prototype.normalizePoint=function(position,clamp){var vec=vec3.fromValues((position[0]-this.center[0])/this.size[0]+.5,(position[1]-this.center[1])/this.size[1]+.5,(position[2]-this.center[2])/this.size[2]+.5);if(clamp){vec[0]=Math.min(Math.max(vec[0],0),1);vec[1]=Math.min(Math.max(vec[1],0),1);vec[2]=Math.min(Math.max(vec[2],0),1)}return vec};InteractionBox.prototype.toString=function(){return"InteractionBox [ width:"+this.width+" | height:"+this.height+" | depth:"+this.depth+" ]"};InteractionBox.Invalid={valid:false}},{"gl-matrix":19}],10:[function(require,module,exports){var Pipeline=module.exports=function(){this.steps=[]};Pipeline.prototype.addStep=function(step){this.steps.push(step)};Pipeline.prototype.run=function(frame){var stepsLength=this.steps.length;for(var i=0;i!=stepsLength;i++){if(!frame)break;frame=this.steps[i](frame)}return frame}},{}],11:[function(require,module,exports){var glMatrix=require("gl-matrix"),vec3=glMatrix.vec3;var Pointable=module.exports=function(data){this.valid=true;this.id=data.id;this.handId=data.handId;this.length=data.length;this.tool=data.tool;this.width=data.width;this.direction=data.direction;this.stabilizedTipPosition=data.stabilizedTipPosition;this.tipPosition=data.tipPosition;this.tipVelocity=data.tipVelocity;this.touchZone=data.touchZone;this.touchDistance=data.touchDistance;this.timeVisible=data.timeVisible};Pointable.prototype.toString=function(){if(this.tool==true){return"Pointable [ id:"+this.id+" "+this.length+"mmx | with:"+this.width+"mm | direction:"+this.direction+" ]"}else{return"Pointable [ id:"+this.id+" "+this.length+"mmx | direction: "+this.direction+" ]"}};Pointable.Invalid={valid:false}},{"gl-matrix":19}],12:[function(require,module,exports){var Frame=require("./frame");var Event=function(data){this.type=data.type;this.state=data.state};var chooseProtocol=exports.chooseProtocol=function(header){var protocol;switch(header.version){case 1:protocol=JSONProtocol(1,function(data){return new Frame(data)});break;case 2:protocol=JSONProtocol(2,function(data){return new Frame(data)});protocol.sendHeartbeat=function(connection){connection.send(protocol.encode({heartbeat:true}))};break;case 3:protocol=JSONProtocol(3,function(data){return data.event?new Event(data.event):new Frame(data)});protocol.sendHeartbeat=function(connection){connection.send(protocol.encode({heartbeat:true}))};break;default:throw"unrecognized version"}return protocol};var JSONProtocol=function(version,cb){var protocol=cb;protocol.encode=function(message){return JSON.stringify(message)};protocol.version=version;protocol.versionLong="Version "+version;protocol.type="protocol";return protocol}},{"./frame":5}],13:[function(require,module,exports){exports.UI={Region:require("./ui/region"),Cursor:require("./ui/cursor")}},{"./ui/cursor":14,"./ui/region":15}],14:[function(require,module,exports){var Cursor=module.exports=function(){return function(frame){var pointable=frame.pointables.sort(function(a,b){return a.z-b.z})[0];if(pointable&&pointable.valid){frame.cursorPosition=pointable.tipPosition}return frame}}},{}],15:[function(require,module,exports){var EventEmitter=require("events").EventEmitter,_=require("underscore");var Region=module.exports=function(start,end){this.start=new Vector(start);this.end=new Vector(end);this.enteredFrame=null};Region.prototype.hasPointables=function(frame){for(var i=0;i!=frame.pointables.length;i++){var position=frame.pointables[i].tipPosition;if(position.x>=this.start.x&&position.x<=this.end.x&&position.y>=this.start.y&&position.y<=this.end.y&&position.z>=this.start.z&&position.z<=this.end.z){return true}}return false};Region.prototype.listener=function(opts){var region=this;if(opts&&opts.nearThreshold)this.setupNearRegion(opts.nearThreshold);return function(frame){return region.updatePosition(frame)}};Region.prototype.clipper=function(){var region=this;return function(frame){region.updatePosition(frame);return region.enteredFrame?frame:null}};Region.prototype.setupNearRegion=function(distance){var nearRegion=this.nearRegion=new Region([this.start.x-distance,this.start.y-distance,this.start.z-distance],[this.end.x+distance,this.end.y+distance,this.end.z+distance]);var region=this;nearRegion.on("enter",function(frame){region.emit("near",frame)});nearRegion.on("exit",function(frame){region.emit("far",frame)});region.on("exit",function(frame){region.emit("near",frame)})};Region.prototype.updatePosition=function(frame){if(this.nearRegion)this.nearRegion.updatePosition(frame);if(this.hasPointables(frame)&&this.enteredFrame==null){this.enteredFrame=frame;this.emit("enter",this.enteredFrame)}else if(!this.hasPointables(frame)&&this.enteredFrame!=null){this.enteredFrame=null;this.emit("exit",this.enteredFrame)}return frame};Region.prototype.normalize=function(position){return new Vector([(position.x-this.start.x)/(this.end.x-this.start.x),(position.y-this.start.y)/(this.end.y-this.start.y),(position.z-this.start.z)/(this.end.z-this.start.z)])};Region.prototype.mapToXY=function(position,width,height){var normalized=this.normalize(position);var x=normalized.x,y=normalized.y;if(x>1)x=1;else if(x<-1)x=-1;if(y>1)y=1;else if(y<-1)y=-1;return[(x+1)/2*width,(1-y)/2*height,normalized.z]};_.extend(Region.prototype,EventEmitter.prototype)},{events:17,underscore:20}],16:[function(require,module,exports){},{}],17:[function(require,module,exports){!function(process){if(!process.EventEmitter)process.EventEmitter=function(){};var EventEmitter=exports.EventEmitter=process.EventEmitter;var isArray=typeof Array.isArray==="function"?Array.isArray:function(xs){return Object.prototype.toString.call(xs)==="[object Array]"};function indexOf(xs,x){if(xs.indexOf)return xs.indexOf(x);for(var i=0;i0&&this._events[type].length>m){this._events[type].warned=true;console.error("(node) warning: possible EventEmitter memory "+"leak detected. %d listeners added. "+"Use emitter.setMaxListeners() to increase limit.",this._events[type].length);console.trace()}}this._events[type].push(listener)}else{this._events[type]=[this._events[type],listener]}return this};EventEmitter.prototype.on=EventEmitter.prototype.addListener;EventEmitter.prototype.once=function(type,listener){var self=this;self.on(type,function g(){self.removeListener(type,g);listener.apply(this,arguments)});return this};EventEmitter.prototype.removeListener=function(type,listener){if("function"!==typeof listener){throw new Error("removeListener only takes instances of Function")}if(!this._events||!this._events[type])return this;var list=this._events[type];if(isArray(list)){var i=indexOf(list,listener);if(i<0)return this;list.splice(i,1);if(list.length==0)delete this._events[type]}else if(this._events[type]===listener){delete this._events[type]}return this};EventEmitter.prototype.removeAllListeners=function(type){if(arguments.length===0){this._events={};return this}if(type&&this._events&&this._events[type])this._events[type]=null;return this};EventEmitter.prototype.listeners=function(type){if(!this._events)this._events={};if(!this._events[type])this._events[type]=[];if(!isArray(this._events[type])){this._events[type]=[this._events[type]]}return this._events[type]}}(require("__browserify_process"))},{__browserify_process:18}],18:[function(require,module,exports){var process=module.exports={};process.nextTick=function(){var canSetImmediate=typeof window!=="undefined"&&window.setImmediate;var canPost=typeof window!=="undefined"&&window.postMessage&&window.addEventListener;if(canSetImmediate){return function(f){return window.setImmediate(f)}}if(canPost){var queue=[];window.addEventListener("message",function(ev){if(ev.source===window&&ev.data==="process-tick"){ev.stopPropagation();if(queue.length>0){var fn=queue.shift();fn()}}},true);return function nextTick(fn){queue.push(fn);window.postMessage("process-tick","*")}}return function nextTick(fn){setTimeout(fn,0)}}();process.title="browser";process.browser=true;process.env={};process.argv=[];process.binding=function(name){throw new Error("process.binding is not supported")};process.cwd=function(){return"/"};process.chdir=function(dir){throw new Error("process.chdir is not supported")}},{}],19:[function(require,module,exports){!function(){!function(){"use strict";var shim={};if(typeof exports==="undefined"){if(typeof define=="function"&&typeof define.amd=="object"&&define.amd){shim.exports={};define(function(){return shim.exports})}else{shim.exports=window}}else{shim.exports=exports}!function(exports){var vec2={};if(!GLMAT_EPSILON){var GLMAT_EPSILON=1e-6}vec2.create=function(){return new Float32Array(2)};vec2.clone=function(a){var out=new Float32Array(2);out[0]=a[0];out[1]=a[1];return out};vec2.fromValues=function(x,y){var out=new Float32Array(2);out[0]=x;out[1]=y;return out};vec2.copy=function(out,a){out[0]=a[0];out[1]=a[1];return out};vec2.set=function(out,x,y){out[0]=x;out[1]=y;return out};vec2.add=function(out,a,b){out[0]=a[0]+b[0];out[1]=a[1]+b[1];return out};vec2.sub=vec2.subtract=function(out,a,b){out[0]=a[0]-b[0];out[1]=a[1]-b[1];return out};vec2.mul=vec2.multiply=function(out,a,b){out[0]=a[0]*b[0];out[1]=a[1]*b[1];return out};vec2.div=vec2.divide=function(out,a,b){out[0]=a[0]/b[0];out[1]=a[1]/b[1];return out};vec2.min=function(out,a,b){out[0]=Math.min(a[0],b[0]);
+out[1]=Math.min(a[1],b[1]);return out};vec2.max=function(out,a,b){out[0]=Math.max(a[0],b[0]);out[1]=Math.max(a[1],b[1]);return out};vec2.scale=function(out,a,b){out[0]=a[0]*b;out[1]=a[1]*b;return out};vec2.dist=vec2.distance=function(a,b){var x=b[0]-a[0],y=b[1]-a[1];return Math.sqrt(x*x+y*y)};vec2.sqrDist=vec2.squaredDistance=function(a,b){var x=b[0]-a[0],y=b[1]-a[1];return x*x+y*y};vec2.len=vec2.length=function(a){var x=a[0],y=a[1];return Math.sqrt(x*x+y*y)};vec2.sqrLen=vec2.squaredLength=function(a){var x=a[0],y=a[1];return x*x+y*y};vec2.negate=function(out,a){out[0]=-a[0];out[1]=-a[1];return out};vec2.normalize=function(out,a){var x=a[0],y=a[1];var len=x*x+y*y;if(len>0){len=1/Math.sqrt(len);out[0]=a[0]*len;out[1]=a[1]*len}return out};vec2.dot=function(a,b){return a[0]*b[0]+a[1]*b[1]};vec2.cross=function(out,a,b){var z=a[0]*b[1]-a[1]*b[0];out[0]=out[1]=0;out[2]=z;return out};vec2.lerp=function(out,a,b,t){var ax=a[0],ay=a[1];out[0]=ax+t*(b[0]-ax);out[1]=ay+t*(b[1]-ay);return out};vec2.transformMat2=function(out,a,m){var x=a[0],y=a[1];out[0]=x*m[0]+y*m[1];out[1]=x*m[2]+y*m[3];return out};vec2.forEach=function(){var vec=new Float32Array(2);return function(a,stride,offset,count,fn,arg){var i,l;if(!stride){stride=2}if(!offset){offset=0}if(count){l=Math.min(count*stride+offset,a.length)}else{l=a.length}for(i=offset;i0){len=1/Math.sqrt(len);out[0]=a[0]*len;out[1]=a[1]*len;out[2]=a[2]*len}return out};vec3.dot=function(a,b){return a[0]*b[0]+a[1]*b[1]+a[2]*b[2]};vec3.cross=function(out,a,b){var ax=a[0],ay=a[1],az=a[2],bx=b[0],by=b[1],bz=b[2];out[0]=ay*bz-az*by;out[1]=az*bx-ax*bz;out[2]=ax*by-ay*bx;return out};vec3.lerp=function(out,a,b,t){var ax=a[0],ay=a[1],az=a[2];out[0]=ax+t*(b[0]-ax);out[1]=ay+t*(b[1]-ay);out[2]=az+t*(b[2]-az);return out};vec3.transformMat4=function(out,a,m){var x=a[0],y=a[1],z=a[2];out[0]=m[0]*x+m[4]*y+m[8]*z+m[12];out[1]=m[1]*x+m[5]*y+m[9]*z+m[13];out[2]=m[2]*x+m[6]*y+m[10]*z+m[14];return out};vec3.transformQuat=function(out,a,q){var x=a[0],y=a[1],z=a[2],qx=q[0],qy=q[1],qz=q[2],qw=q[3],ix=qw*x+qy*z-qz*y,iy=qw*y+qz*x-qx*z,iz=qw*z+qx*y-qy*x,iw=-qx*x-qy*y-qz*z;out[0]=ix*qw+iw*-qx+iy*-qz-iz*-qy;out[1]=iy*qw+iw*-qy+iz*-qx-ix*-qz;out[2]=iz*qw+iw*-qz+ix*-qy-iy*-qx;return out};vec3.forEach=function(){var vec=new Float32Array(3);return function(a,stride,offset,count,fn,arg){var i,l;if(!stride){stride=3}if(!offset){offset=0}if(count){l=Math.min(count*stride+offset,a.length)}else{l=a.length}for(i=offset;i0){len=1/Math.sqrt(len);out[0]=a[0]*len;out[1]=a[1]*len;out[2]=a[2]*len;out[3]=a[3]*len}return out};vec4.dot=function(a,b){return a[0]*b[0]+a[1]*b[1]+a[2]*b[2]+a[3]*b[3]};vec4.lerp=function(out,a,b,t){var ax=a[0],ay=a[1],az=a[2],aw=a[3];out[0]=ax+t*(b[0]-ax);out[1]=ay+t*(b[1]-ay);out[2]=az+t*(b[2]-az);out[3]=aw+t*(b[3]-aw);return out};vec4.transformMat4=function(out,a,m){var x=a[0],y=a[1],z=a[2],w=a[3];out[0]=m[0]*x+m[4]*y+m[8]*z+m[12]*w;out[1]=m[1]*x+m[5]*y+m[9]*z+m[13]*w;out[2]=m[2]*x+m[6]*y+m[10]*z+m[14]*w;out[3]=m[3]*x+m[7]*y+m[11]*z+m[15]*w;return out};vec4.transformQuat=function(out,a,q){var x=a[0],y=a[1],z=a[2],qx=q[0],qy=q[1],qz=q[2],qw=q[3],ix=qw*x+qy*z-qz*y,iy=qw*y+qz*x-qx*z,iz=qw*z+qx*y-qy*x,iw=-qx*x-qy*y-qz*z;out[0]=ix*qw+iw*-qx+iy*-qz-iz*-qy;out[1]=iy*qw+iw*-qy+iz*-qx-ix*-qz;out[2]=iz*qw+iw*-qz+ix*-qy-iy*-qx;return out};vec4.forEach=function(){var vec=new Float32Array(4);return function(a,stride,offset,count,fn,arg){var i,l;if(!stride){stride=4}if(!offset){offset=0}if(count){l=Math.min(count*stride+offset,a.length)}else{l=a.length}for(i=offset;i=1){if(out!==a){out[0]=ax;out[1]=ay;out[2]=az;out[3]=aw}return out}halfTheta=Math.acos(cosHalfTheta);sinHalfTheta=Math.sqrt(1-cosHalfTheta*cosHalfTheta);if(Math.abs(sinHalfTheta)<.001){out[0]=ax*.5+bx*.5;out[1]=ay*.5+by*.5;out[2]=az*.5+bz*.5;out[3]=aw*.5+bw*.5;return out}ratioA=Math.sin((1-t)*halfTheta)/sinHalfTheta;ratioB=Math.sin(t*halfTheta)/sinHalfTheta;out[0]=ax*ratioA+bx*ratioB;out[1]=ay*ratioA+by*ratioB;out[2]=az*ratioA+bz*ratioB;out[3]=aw*ratioA+bw*ratioB;return out};quat.invert=function(out,a){var a0=a[0],a1=a[1],a2=a[2],a3=a[3],dot=a0*a0+a1*a1+a2*a2+a3*a3,invDot=dot?1/dot:0;out[0]=-a0*invDot;out[1]=-a1*invDot;out[2]=-a2*invDot;out[3]=a3*invDot;return out};quat.conjugate=function(out,a){out[0]=-a[0];out[1]=-a[1];out[2]=-a[2];out[3]=a[3];return out};quat.len=quat.length=vec4.length;quat.sqrLen=quat.squaredLength=vec4.squaredLength;quat.normalize=vec4.normalize;quat.str=function(a){return"quat("+a[0]+", "+a[1]+", "+a[2]+", "+a[3]+")"};if(typeof exports!=="undefined"){exports.quat=quat}}(shim.exports)}()}()},{}],20:[function(require,module,exports){!function(){!function(){var root=this;var previousUnderscore=root._;var breaker={};var ArrayProto=Array.prototype,ObjProto=Object.prototype,FuncProto=Function.prototype;var push=ArrayProto.push,slice=ArrayProto.slice,concat=ArrayProto.concat,toString=ObjProto.toString,hasOwnProperty=ObjProto.hasOwnProperty;var nativeForEach=ArrayProto.forEach,nativeMap=ArrayProto.map,nativeReduce=ArrayProto.reduce,nativeReduceRight=ArrayProto.reduceRight,nativeFilter=ArrayProto.filter,nativeEvery=ArrayProto.every,nativeSome=ArrayProto.some,nativeIndexOf=ArrayProto.indexOf,nativeLastIndexOf=ArrayProto.lastIndexOf,nativeIsArray=Array.isArray,nativeKeys=Object.keys,nativeBind=FuncProto.bind;var _=function(obj){if(obj instanceof _)return obj;if(!(this instanceof _))return new _(obj);this._wrapped=obj};if(typeof exports!=="undefined"){if(typeof module!=="undefined"&&module.exports){exports=module.exports=_}exports._=_}else{root._=_}_.VERSION="1.4.4";var each=_.each=_.forEach=function(obj,iterator,context){if(obj==null)return;if(nativeForEach&&obj.forEach===nativeForEach){obj.forEach(iterator,context)}else if(obj.length===+obj.length){for(var i=0,l=obj.length;i2;if(obj==null)obj=[];if(nativeReduce&&obj.reduce===nativeReduce){if(context)iterator=_.bind(iterator,context);return initial?obj.reduce(iterator,memo):obj.reduce(iterator)}each(obj,function(value,index,list){if(!initial){memo=value;initial=true}else{memo=iterator.call(context,memo,value,index,list)}});if(!initial)throw new TypeError(reduceError);return memo};_.reduceRight=_.foldr=function(obj,iterator,memo,context){var initial=arguments.length>2;if(obj==null)obj=[];if(nativeReduceRight&&obj.reduceRight===nativeReduceRight){if(context)iterator=_.bind(iterator,context);return initial?obj.reduceRight(iterator,memo):obj.reduceRight(iterator)}var length=obj.length;if(length!==+length){var keys=_.keys(obj);length=keys.length}each(obj,function(value,index,list){index=keys?keys[--length]:--length;if(!initial){memo=obj[index];initial=true}else{memo=iterator.call(context,memo,obj[index],index,list)}});if(!initial)throw new TypeError(reduceError);return memo};_.find=_.detect=function(obj,iterator,context){var result;any(obj,function(value,index,list){if(iterator.call(context,value,index,list)){result=value;return true}});return result};_.filter=_.select=function(obj,iterator,context){var results=[];if(obj==null)return results;if(nativeFilter&&obj.filter===nativeFilter)return obj.filter(iterator,context);each(obj,function(value,index,list){if(iterator.call(context,value,index,list))results[results.length]=value});return results};_.reject=function(obj,iterator,context){return _.filter(obj,function(value,index,list){return!iterator.call(context,value,index,list)},context)};_.every=_.all=function(obj,iterator,context){iterator||(iterator=_.identity);var result=true;if(obj==null)return result;if(nativeEvery&&obj.every===nativeEvery)return obj.every(iterator,context);each(obj,function(value,index,list){if(!(result=result&&iterator.call(context,value,index,list)))return breaker});return!!result};var any=_.some=_.any=function(obj,iterator,context){iterator||(iterator=_.identity);var result=false;if(obj==null)return result;if(nativeSome&&obj.some===nativeSome)return obj.some(iterator,context);each(obj,function(value,index,list){if(result||(result=iterator.call(context,value,index,list)))return breaker});return!!result};_.contains=_.include=function(obj,target){if(obj==null)return false;if(nativeIndexOf&&obj.indexOf===nativeIndexOf)return obj.indexOf(target)!=-1;return any(obj,function(value){return value===target})};_.invoke=function(obj,method){var args=slice.call(arguments,2);var isFunc=_.isFunction(method);return _.map(obj,function(value){return(isFunc?method:value[method]).apply(value,args)})};_.pluck=function(obj,key){return _.map(obj,function(value){return value[key]})};_.where=function(obj,attrs,first){if(_.isEmpty(attrs))return first?null:[];return _[first?"find":"filter"](obj,function(value){for(var key in attrs){if(attrs[key]!==value[key])return false}return true})};_.findWhere=function(obj,attrs){return _.where(obj,attrs,true)};_.max=function(obj,iterator,context){if(!iterator&&_.isArray(obj)&&obj[0]===+obj[0]&&obj.length<65535){return Math.max.apply(Math,obj)}if(!iterator&&_.isEmpty(obj))return-Infinity;var result={computed:-Infinity,value:-Infinity};each(obj,function(value,index,list){var computed=iterator?iterator.call(context,value,index,list):value;computed>=result.computed&&(result={value:value,computed:computed})});return result.value};_.min=function(obj,iterator,context){if(!iterator&&_.isArray(obj)&&obj[0]===+obj[0]&&obj.length<65535){return Math.min.apply(Math,obj)}if(!iterator&&_.isEmpty(obj))return Infinity;var result={computed:Infinity,value:Infinity};each(obj,function(value,index,list){var computed=iterator?iterator.call(context,value,index,list):value;computedb||a===void 0)return 1;if(a>>1;iterator.call(context,array[mid])=0})})};_.difference=function(array){var rest=concat.apply(ArrayProto,slice.call(arguments,1));return _.filter(array,function(value){return!_.contains(rest,value)})};_.zip=function(){var args=slice.call(arguments);var length=_.max(_.pluck(args,"length"));var results=new Array(length);for(var i=0;i=0;i--){args=[funcs[i].apply(this,args)]}return args[0]}};_.after=function(times,func){if(times<=0)return func();return function(){if(--times<1){return func.apply(this,arguments)}}};_.keys=nativeKeys||function(obj){if(obj!==Object(obj))throw new TypeError("Invalid object");var keys=[];for(var key in obj)if(_.has(obj,key))keys[keys.length]=key;return keys};_.values=function(obj){var values=[];for(var key in obj)if(_.has(obj,key))values.push(obj[key]);return values};_.pairs=function(obj){var pairs=[];for(var key in obj)if(_.has(obj,key))pairs.push([key,obj[key]]);return pairs};_.invert=function(obj){var result={};for(var key in obj)if(_.has(obj,key))result[obj[key]]=key;return result};_.functions=_.methods=function(obj){var names=[];for(var key in obj){if(_.isFunction(obj[key]))names.push(key)}return names.sort()};_.extend=function(obj){each(slice.call(arguments,1),function(source){if(source){for(var prop in source){obj[prop]=source[prop]}}});return obj};_.pick=function(obj){var copy={};var keys=concat.apply(ArrayProto,slice.call(arguments,1));each(keys,function(key){if(key in obj)copy[key]=obj[key]});return copy};_.omit=function(obj){var copy={};var keys=concat.apply(ArrayProto,slice.call(arguments,1));for(var key in obj){if(!_.contains(keys,key))copy[key]=obj[key]}return copy};_.defaults=function(obj){each(slice.call(arguments,1),function(source){if(source){for(var prop in source){if(obj[prop]==null)obj[prop]=source[prop]}}});return obj};_.clone=function(obj){if(!_.isObject(obj))return obj;return _.isArray(obj)?obj.slice():_.extend({},obj)};_.tap=function(obj,interceptor){interceptor(obj);return obj};var eq=function(a,b,aStack,bStack){if(a===b)return a!==0||1/a==1/b;if(a==null||b==null)return a===b;if(a instanceof _)a=a._wrapped;if(b instanceof _)b=b._wrapped;var className=toString.call(a);if(className!=toString.call(b))return false;switch(className){case"[object String]":return a==String(b);case"[object Number]":return a!=+a?b!=+b:a==0?1/a==1/b:a==+b;case"[object Date]":case"[object Boolean]":return+a==+b;case"[object RegExp]":return a.source==b.source&&a.global==b.global&&a.multiline==b.multiline&&a.ignoreCase==b.ignoreCase}if(typeof a!="object"||typeof b!="object")return false;var length=aStack.length;while(length--){if(aStack[length]==a)return bStack[length]==b}aStack.push(a);bStack.push(b);var size=0,result=true;if(className=="[object Array]"){size=a.length;result=size==b.length;if(result){while(size--){if(!(result=eq(a[size],b[size],aStack,bStack)))break}}}else{var aCtor=a.constructor,bCtor=b.constructor;if(aCtor!==bCtor&&!(_.isFunction(aCtor)&&aCtor instanceof aCtor&&_.isFunction(bCtor)&&bCtor instanceof bCtor)){return false}for(var key in a){if(_.has(a,key)){size++;if(!(result=_.has(b,key)&&eq(a[key],b[key],aStack,bStack)))break}}if(result){for(key in b){if(_.has(b,key)&&!size--)break}result=!size}}aStack.pop();bStack.pop();return result};_.isEqual=function(a,b){return eq(a,b,[],[])};_.isEmpty=function(obj){if(obj==null)return true;if(_.isArray(obj)||_.isString(obj))return obj.length===0;for(var key in obj)if(_.has(obj,key))return false;return true};_.isElement=function(obj){return!!(obj&&obj.nodeType===1)};_.isArray=nativeIsArray||function(obj){return toString.call(obj)=="[object Array]"};_.isObject=function(obj){return obj===Object(obj)};each(["Arguments","Function","String","Number","Date","RegExp"],function(name){_["is"+name]=function(obj){return toString.call(obj)=="[object "+name+"]"}});if(!_.isArguments(arguments)){_.isArguments=function(obj){return!!(obj&&_.has(obj,"callee"))}}if(typeof/./!=="function"){_.isFunction=function(obj){return typeof obj==="function"}}_.isFinite=function(obj){return isFinite(obj)&&!isNaN(parseFloat(obj))};_.isNaN=function(obj){return _.isNumber(obj)&&obj!=+obj};_.isBoolean=function(obj){return obj===true||obj===false||toString.call(obj)=="[object Boolean]"};_.isNull=function(obj){return obj===null};_.isUndefined=function(obj){return obj===void 0};_.has=function(obj,key){return hasOwnProperty.call(obj,key)};_.noConflict=function(){root._=previousUnderscore;return this};_.identity=function(value){return value};_.times=function(n,iterator,context){var accum=Array(n);for(var i=0;i":">",'"':""","'":"'","/":"/"}};entityMap.unescape=_.invert(entityMap.escape);var entityRegexes={escape:new RegExp("["+_.keys(entityMap.escape).join("")+"]","g"),unescape:new RegExp("("+_.keys(entityMap.unescape).join("|")+")","g")};_.each(["escape","unescape"],function(method){_[method]=function(string){if(string==null)return"";return(""+string).replace(entityRegexes[method],function(match){return entityMap[method][match]})}});_.result=function(object,property){if(object==null)return null;var value=object[property];return _.isFunction(value)?value.call(object):value};_.mixin=function(obj){each(_.functions(obj),function(name){var func=_[name]=obj[name];_.prototype[name]=function(){var args=[this._wrapped];push.apply(args,arguments);return result.call(this,func.apply(_,args))}})};var idCounter=0;_.uniqueId=function(prefix){var id=++idCounter+"";return prefix?prefix+id:id};_.templateSettings={evaluate:/<%([\s\S]+?)%>/g,interpolate:/<%=([\s\S]+?)%>/g,escape:/<%-([\s\S]+?)%>/g};var noMatch=/(.)^/;var escapes={"'":"'","\\":"\\","\r":"r","\n":"n"," ":"t","\u2028":"u2028","\u2029":"u2029"};var escaper=/\\|'|\r|\n|\t|\u2028|\u2029/g;_.template=function(text,data,settings){var render;settings=_.defaults({},settings,_.templateSettings);var matcher=new RegExp([(settings.escape||noMatch).source,(settings.interpolate||noMatch).source,(settings.evaluate||noMatch).source].join("|")+"|$","g");var index=0;var source="__p+='";text.replace(matcher,function(match,escape,interpolate,evaluate,offset){source+=text.slice(index,offset).replace(escaper,function(match){return"\\"+escapes[match]});if(escape){source+="'+\n((__t=("+escape+"))==null?'':_.escape(__t))+\n'"}if(interpolate){source+="'+\n((__t=("+interpolate+"))==null?'':__t)+\n'"}if(evaluate){source+="';\n"+evaluate+"\n__p+='"}index=offset+match.length;return match});source+="';\n";if(!settings.variable)source="with(obj||{}){\n"+source+"}\n";source="var __t,__p='',__j=Array.prototype.join,"+"print=function(){__p+=__j.call(arguments,'');};\n"+source+"return __p;\n";try{render=new Function(settings.variable||"obj","_",source)}catch(e){e.source=source;throw e}if(data)return render(data,_);var template=function(data){return render.call(this,data,_)};template.source="function("+(settings.variable||"obj")+"){\n"+source+"}";return template};_.chain=function(obj){return _(obj).chain()};var result=function(obj){return this._chain?_(obj).chain():obj};_.mixin(_);each(["pop","push","reverse","shift","sort","splice","unshift"],function(name){var method=ArrayProto[name];_.prototype[name]=function(){var obj=this._wrapped;method.apply(obj,arguments);if((name=="shift"||name=="splice")&&obj.length===0)delete obj[0];return result.call(this,obj)}});each(["concat","join","slice"],function(name){var method=ArrayProto[name];_.prototype[name]=function(){return result.call(this,method.apply(this._wrapped,arguments))}});_.extend(_.prototype,{chain:function(){this._chain=true;return this},value:function(){return this._wrapped}})}.call(this)}()},{}],21:[function(require,module,exports){window.requestAnimFrame=function(){return window.requestAnimationFrame||window.webkitRequestAnimationFrame||window.mozRequestAnimationFrame||window.oRequestAnimationFrame||window.msRequestAnimationFrame||function(callback){window.setTimeout(callback,1e3/60)}}();Leap=require("../lib/index")},{"../lib/index":8}]},{},[21]);
+
+/*
+ * Leap Motion integration for Reveal.js.
+ * James Sun  [sun16]
+ * Rory Hardy [gneatgeek]
+ */
+
+(function () {
+  var body        = document.body,
+      controller  = new Leap.Controller({ enableGestures: true }),
+      lastGesture = 0,
+      leapConfig  = Reveal.getConfig().leap,
+      pointer     = document.createElement( 'div' ),
+      config      = {
+        autoCenter       : true,      // Center pointer around detected position.
+        gestureDelay     : 500,       // How long to delay between gestures.
+        naturalSwipe     : true,      // Swipe as if it were a touch screen.
+        pointerColor     : '#00aaff', // Default color of the pointer.
+        pointerOpacity   : 0.7,       // Default opacity of the pointer.
+        pointerSize      : 15,        // Default minimum height/width of the pointer.
+        pointerTolerance : 120        // Bigger = slower pointer.
+      },
+      entered, enteredPosition, now, size, tipPosition; // Other vars we need later, but don't need to redeclare.
+
+      // Merge user defined settings with defaults
+      if( leapConfig ) {
+        for( key in leapConfig ) {
+          config[key] = leapConfig[key];
+        }
+      }
+
+      pointer.id = 'leap';
+
+      pointer.style.position        = 'absolute';
+      pointer.style.visibility      = 'hidden';
+      pointer.style.zIndex          = 50;
+      pointer.style.opacity         = config.pointerOpacity;
+      pointer.style.backgroundColor = config.pointerColor;
+
+      body.appendChild( pointer );
+
+  // Leap's loop
+  controller.on( 'frame', function ( frame ) {
+    // Timing code to rate limit gesture execution
+    now = new Date().getTime();
+
+    // Pointer: 1 to 2 fingers. Strictly one finger works but may cause innaccuracies.
+    // The innaccuracies were observed on a development model and may not be an issue with consumer models.
+    if( frame.fingers.length > 0 && frame.fingers.length < 3 ) {
+      // Invert direction and multiply by 3 for greater effect.
+      size = -3 * frame.fingers[0].tipPosition[2];
+
+      if( size < config.pointerSize ) {
+        size = config.pointerSize;
+      }
+
+      pointer.style.width        = size     + 'px';
+      pointer.style.height       = size     + 'px';
+      pointer.style.borderRadius = size - 5 + 'px';
+      pointer.style.visibility   = 'visible';
+
+      tipPosition = frame.fingers[0].tipPosition;
+
+      if( config.autoCenter ) {
+        
+
+        // Check whether the finger has entered the z range of the Leap Motion. Used for the autoCenter option.
+        if( !entered ) {
+          entered         = true;
+          enteredPosition = frame.fingers[0].tipPosition;
+        }
+
+        pointer.style.top =
+          (-1 * (( tipPosition[1] - enteredPosition[1] ) * body.offsetHeight / config.pointerTolerance )) +
+            ( body.offsetHeight / 2 ) + 'px';
+
+        pointer.style.left =
+          (( tipPosition[0] - enteredPosition[0] ) * body.offsetWidth / config.pointerTolerance ) +
+            ( body.offsetWidth / 2 ) + 'px';
+      }
+      else {
+        pointer.style.top  = ( 1 - (( tipPosition[1] - 50) / config.pointerTolerance )) *
+          body.offsetHeight + 'px';
+
+        pointer.style.left = ( tipPosition[0] * body.offsetWidth / config.pointerTolerance ) +
+          ( body.offsetWidth / 2 ) + 'px';
+      }
+    }
+    else {
+      // Hide pointer on exit
+      entered                  = false;
+      pointer.style.visibility = 'hidden';
+    }
+
+    // Gestures
+    if( frame.gestures.length > 0 && (now - lastGesture) > config.gestureDelay ) {
+      var gesture = frame.gestures[0];
+
+      // One hand gestures
+      if( frame.hands.length === 1 ) {
+        // Swipe gestures. 3+ fingers.
+        if( frame.fingers.length > 2 && gesture.type === 'swipe' ) {
+          // Define here since some gestures will throw undefined for these.
+          var x = gesture.direction[0],
+              y = gesture.direction[1];
+
+          // Left/right swipe gestures
+          if( Math.abs( x ) > Math.abs( y )) {
+            if( x > 0 ) {
+              config.naturalSwipe ? Reveal.left() : Reveal.right();
+            }
+            else {
+              config.naturalSwipe ? Reveal.right() : Reveal.left();
+            }
+          }
+          // Up/down swipe gestures
+          else {
+            if( y > 0 ) {
+              config.naturalSwipe ? Reveal.down() : Reveal.up();
+            }
+            else {
+              config.naturalSwipe ? Reveal.up() : Reveal.down();
+            }
+          }
+
+          lastGesture = now;
+        }
+      }
+      // Two hand gestures
+      else if( frame.hands.length === 2 ) {
+        // Upward two hand swipe gesture
+        if( gesture.type === 'swipe' && gesture.direction[1] > 0 ) {
+          Reveal.toggleOverview();
+        }
+
+        lastGesture = now;
+      }
+    }
+  });
+
+  controller.connect();
+})();
diff --git a/doc/pub/Reinforce/html/reveal.js/plugin/remotes/remotes.js b/doc/pub/Reinforce/html/reveal.js/plugin/remotes/remotes.js
new file mode 100644
index 000000000..ba0dbad7b
--- /dev/null
+++ b/doc/pub/Reinforce/html/reveal.js/plugin/remotes/remotes.js
@@ -0,0 +1,39 @@
+/**
+ * Touch-based remote controller for your presentation courtesy 
+ * of the folks at http://remotes.io
+ */
+
+(function(window){
+
+    /**
+     * Detects if we are dealing with a touch enabled device (with some false positives)
+     * Borrowed from modernizr: https://github.com/Modernizr/Modernizr/blob/master/feature-detects/touch.js   
+     */
+    var hasTouch  = (function(){
+        return ('ontouchstart' in window) || window.DocumentTouch && document instanceof DocumentTouch;
+    })();
+
+    /**
+     * Detects if notes are enable and the current page is opened inside an /iframe
+     * this prevents loading Remotes.io several times
+     */
+    var isNotesAndIframe = (function(){
+        return window.RevealNotes && !(self == top);
+    })();
+
+    if(!hasTouch && !isNotesAndIframe){
+        head.ready( 'remotes.ne.min.js', function() {
+            new Remotes("preview")
+                .on("swipe-left", function(e){ Reveal.right(); })
+                .on("swipe-right", function(e){ Reveal.left(); })
+                .on("swipe-up", function(e){ Reveal.down(); })
+                .on("swipe-down", function(e){ Reveal.up(); })
+                .on("tap", function(e){ Reveal.next(); })
+                .on("zoom-out", function(e){ Reveal.toggleOverview(true); })
+                .on("zoom-in", function(e){ Reveal.toggleOverview(false); })
+            ;
+        } );
+
+        head.js('https://hakim-static.s3.amazonaws.com/reveal-js/remotes.ne.min.js');
+    }
+})(window);
\ No newline at end of file
diff --git a/doc/pub/Splines/html/reveal.js/css/theme/template/mixins.scss b/doc/pub/Splines/html/reveal.js/css/theme/template/mixins.scss
new file mode 100644
index 000000000..e0c560692
--- /dev/null
+++ b/doc/pub/Splines/html/reveal.js/css/theme/template/mixins.scss
@@ -0,0 +1,29 @@
+@mixin vertical-gradient( $top, $bottom ) {
+	background: $top;
+	background: -moz-linear-gradient( top, $top 0%, $bottom 100% );
+	background: -webkit-gradient( linear, left top, left bottom, color-stop(0%,$top), color-stop(100%,$bottom) );
+	background: -webkit-linear-gradient( top, $top 0%, $bottom 100% );
+	background: -o-linear-gradient( top, $top 0%, $bottom 100% );
+	background: -ms-linear-gradient( top, $top 0%, $bottom 100% );
+	background: linear-gradient( top, $top 0%, $bottom 100% );
+}
+
+@mixin horizontal-gradient( $top, $bottom ) {
+	background: $top;
+	background: -moz-linear-gradient( left, $top 0%, $bottom 100% );
+	background: -webkit-gradient( linear, left top, right top, color-stop(0%,$top), color-stop(100%,$bottom) );
+	background: -webkit-linear-gradient( left, $top 0%, $bottom 100% );
+	background: -o-linear-gradient( left, $top 0%, $bottom 100% );
+	background: -ms-linear-gradient( left, $top 0%, $bottom 100% );
+	background: linear-gradient( left, $top 0%, $bottom 100% );
+}
+
+@mixin radial-gradient( $outer, $inner, $type: circle ) {
+	background: $outer;
+	background: -moz-radial-gradient( center, $type cover,  $inner 0%, $outer 100% );
+	background: -webkit-gradient( radial, center center, 0px, center center, 100%, color-stop(0%,$inner), color-stop(100%,$outer) );
+	background: -webkit-radial-gradient( center, $type cover,  $inner 0%, $outer 100% );
+	background: -o-radial-gradient( center, $type cover,  $inner 0%, $outer 100% );
+	background: -ms-radial-gradient( center, $type cover,  $inner 0%, $outer 100% );
+	background: radial-gradient( center, $type cover,  $inner 0%, $outer 100% );
+}
\ No newline at end of file
diff --git a/doc/pub/Splines/html/reveal.js/css/theme/template/settings.scss b/doc/pub/Splines/html/reveal.js/css/theme/template/settings.scss
new file mode 100644
index 000000000..e706e19c5
--- /dev/null
+++ b/doc/pub/Splines/html/reveal.js/css/theme/template/settings.scss
@@ -0,0 +1,34 @@
+// Base settings for all themes that can optionally be
+// overridden by the super-theme
+
+// Background of the presentation
+$backgroundColor: #2b2b2b;
+
+// Primary/body text
+$mainFont: 'Lato', sans-serif;
+$mainFontSize: 30px;              /* changed (by hpl) from 36px; */
+$mainColor: #eee;
+
+// Headings
+$headingMargin: 0 0 20px 0;
+$headingFont: 'League Gothic', Impact, sans-serif;
+$headingColor: #eee;
+$headingLineHeight: 0.9em;
+$headingLetterSpacing: 0.02em;
+$headingTextTransform: none;      /* changed (by hpl) from uppercase; */
+$headingTextShadow: 0px 0px 6px rgba(0,0,0,0.2);
+$heading1TextShadow: $headingTextShadow;
+
+// Links and actions
+$linkColor: #13DAEC;
+$linkColorHover: lighten( $linkColor, 20% );
+
+// Text selection
+$selectionBackgroundColor: #FF5E99;
+$selectionColor: #fff;
+
+// Generates the presentation background, can be overridden
+// to return a background image or gradient
+@mixin bodyBackground() {
+	background: $backgroundColor;
+}
\ No newline at end of file
diff --git a/doc/pub/Splines/html/reveal.js/css/theme/template/theme.scss b/doc/pub/Splines/html/reveal.js/css/theme/template/theme.scss
new file mode 100644
index 000000000..b1dbc2736
--- /dev/null
+++ b/doc/pub/Splines/html/reveal.js/css/theme/template/theme.scss
@@ -0,0 +1,171 @@
+// Base theme template for reveal.js
+
+/*********************************************
+ * GLOBAL STYLES
+ *********************************************/
+
+body {
+	@include bodyBackground();
+	background-color: $backgroundColor;
+}
+
+.reveal {
+	font-family: $mainFont;
+	font-size: $mainFontSize;
+	font-weight: normal;
+	letter-spacing: -0.02em;
+	color: $mainColor;
+}
+
+::selection {
+	color: $selectionColor;
+	background: $selectionBackgroundColor;
+	text-shadow: none;
+}
+
+/*********************************************
+ * HEADERS
+ *********************************************/
+
+.reveal h1,
+.reveal h2,
+.reveal h3,
+.reveal h4,
+.reveal h5,
+.reveal h6 {
+	margin: $headingMargin;
+	color: $headingColor;
+
+	font-family: $headingFont;
+	line-height: $headingLineHeight;
+	letter-spacing: $headingLetterSpacing;
+
+	text-transform: $headingTextTransform;
+	text-shadow: $headingTextShadow;
+}
+
+.reveal h1 {
+        line-height: 1.2em;    /* added by hpl */
+	text-shadow: $heading1TextShadow;
+}
+
+
+/*********************************************
+ * LINKS
+ *********************************************/
+
+.reveal a:not(.image) {
+	color: $linkColor;
+	text-decoration: none;
+
+	-webkit-transition: color .15s ease;
+	   -moz-transition: color .15s ease;
+	    -ms-transition: color .15s ease;
+	     -o-transition: color .15s ease;
+	        transition: color .15s ease;
+}
+	.reveal a:not(.image):hover {
+		color: $linkColorHover;
+
+		text-shadow: none;
+		border: none;
+	}
+
+.reveal .roll span:after {
+	color: #fff;
+	background: darken( $linkColor, 15% );
+}
+
+
+/*********************************************
+ * IMAGES
+ *********************************************/
+
+.reveal section img {
+	margin: 15px 0px;
+	background: rgba(255,255,255,0.12);
+	border: 4px solid $mainColor;
+
+	box-shadow: 0 0 10px rgba(0, 0, 0, 0.15);
+
+	-webkit-transition: all .2s linear;
+	   -moz-transition: all .2s linear;
+	    -ms-transition: all .2s linear;
+	     -o-transition: all .2s linear;
+	        transition: all .2s linear;
+}
+
+	.reveal a:hover img {
+		background: rgba(255,255,255,0.2);
+		border-color: $linkColor;
+
+		box-shadow: 0 0 20px rgba(0, 0, 0, 0.55);
+	}
+
+
+/*********************************************
+ * NAVIGATION CONTROLS
+ *********************************************/
+
+.reveal .controls div.navigate-left,
+.reveal .controls div.navigate-left.enabled {
+	border-right-color: $linkColor;
+}
+
+.reveal .controls div.navigate-right,
+.reveal .controls div.navigate-right.enabled {
+	border-left-color: $linkColor;
+}
+
+.reveal .controls div.navigate-up,
+.reveal .controls div.navigate-up.enabled {
+	border-bottom-color: $linkColor;
+}
+
+.reveal .controls div.navigate-down,
+.reveal .controls div.navigate-down.enabled {
+	border-top-color: $linkColor;
+}
+
+.reveal .controls div.navigate-left.enabled:hover {
+	border-right-color: $linkColorHover;
+}
+
+.reveal .controls div.navigate-right.enabled:hover {
+	border-left-color: $linkColorHover;
+}
+
+.reveal .controls div.navigate-up.enabled:hover {
+	border-bottom-color: $linkColorHover;
+}
+
+.reveal .controls div.navigate-down.enabled:hover {
+	border-top-color: $linkColorHover;
+}
+
+
+/*********************************************
+ * PROGRESS BAR
+ *********************************************/
+
+.reveal .progress {
+	background: rgba(0,0,0,0.2);
+}
+	.reveal .progress span {
+		background: $linkColor;
+
+		-webkit-transition: width 800ms cubic-bezier(0.260, 0.860, 0.440, 0.985);
+		   -moz-transition: width 800ms cubic-bezier(0.260, 0.860, 0.440, 0.985);
+		    -ms-transition: width 800ms cubic-bezier(0.260, 0.860, 0.440, 0.985);
+		     -o-transition: width 800ms cubic-bezier(0.260, 0.860, 0.440, 0.985);
+		        transition: width 800ms cubic-bezier(0.260, 0.860, 0.440, 0.985);
+	}
+
+/*********************************************
+ * SLIDE NUMBER
+ *********************************************/
+.reveal .slide-number {
+  color: $linkColor;
+}
+
+
diff --git a/doc/pub/Splines/html/reveal.js/demo.html b/doc/pub/Splines/html/reveal.js/demo.html
new file mode 100644
index 000000000..505bb1882
--- /dev/null
+++ b/doc/pub/Splines/html/reveal.js/demo.html
@@ -0,0 +1,410 @@
+
+
+
+	
+		
+
+		reveal.js – The HTML Presentation Framework
+
+		
+		
+
+		
+		
+
+		
+
+		
+		
+
+		
+		
+
+		
+		
+
+		
+	
+
+	
+
+		

+
+			
+			

+				

+					
Reveal.js

+					
The HTML Presentation Framework

+					

+						Created by Hakim El Hattab and contributors
+					

+				

+
+				

+					
Hello There

+					

+						reveal.js enables you to create beautiful interactive slide decks using HTML. This presentation will show you examples of what it can do.
+					

+				

+
+				
+				

+					

+						
Vertical Slides

+						
Slides can be nested inside of each other.

+						
Use the Space key to navigate through all slides.

+						

+						
+							Down arrow
+						
+					

+					

+						
Basement Level 1

+						
Nested slides are useful for adding additional detail underneath a high level horizontal slide.

+					

+					

+						
Basement Level 2

+						
That's it, time to go back up.

+						

+						
+							Up arrow
+						
+					

+				

+
+				

+					
Slides

+					

+						Not a coder? Not a problem. There's a fully-featured visual editor for authoring these, try it out at https://slides.com.
+					

+				

+
+				

+					
Point of View

+					

+						Press ESC to enter the slide overview.
+					

+					

+						Hold down alt and click on any element to zoom in on it using zoom.js. Alt + click anywhere to zoom back out.
+					

+				

+
+				

+					
Touch Optimized

+					

+						Presentations look great on touch devices, like mobile phones and tablets. Simply swipe through your slides.
+					

+				

+
+				

+					
+				

+
+				

+					

+						
Fragments

+						
Hit the next arrow...

+						
... to step through ...

+						
... a fragmented slide.

+
+						
+					

+					

+						
Fragment Styles

+						
There's different types of fragments, like:

+						
grow

+						
shrink

+						
fade-out

+						
fade-up (also down, left and right!)

+						
current-visible

+						
Highlight red blue green

+					

+				

+
+				

+					
Transition Styles

+					

+						You can select from different transitions, like: 

+						None -
+						Fade -
+						Slide -
+						Convex -
+						Concave -
+						Zoom
+					

+				

+
+				

+					
Themes

+					

+						reveal.js comes with a few themes built in: 

+						
+						Black (default) -
+						White -
+						League -
+						Sky -
+						Beige -
+						Simple 

+						Serif -
+						Blood -
+						Night -
+						Moon -
+						Solarized
+					

+				

+
+				

+					

+						
Slide Backgrounds

+						

+							Set data-background="#dddddd" on a slide to change the background color. All CSS color formats are supported.
+						

+						
+							Down arrow
+						
+					

+					

+						
Image Backgrounds

+						
<section data-background="image.png">

+					

+					

+						
Tiled Backgrounds

+						
<section data-background="image.png" data-background-repeat="repeat" data-background-size="100px">

+					

+					

+						

+							
Video Backgrounds

+							
<section data-background-video="video.mp4,video.webm">

+						

+					

+					

+						
... and GIFs!

+					

+				

+
+				

+					
Background Transitions

+					

+						Different background transitions are available via the backgroundTransition option. This one's called "zoom".
+					

+					
Reveal.configure({ backgroundTransition: 'zoom' })

+				

+
+				

+					
Background Transitions

+					

+						You can override background transitions per-slide.
+					

+					
<section data-background-transition="zoom">

+				

+
+				

+					
Pretty Code

+					

+function linkify( selector ) {
+  if( supports3DTransforms ) {
+
+    var nodes = document.querySelectorAll( selector );
+
+    for( var i = 0, len = nodes.length; i < len; i++ ) {
+      var node = nodes[i];
+
+      if( !node.className ) {
+        node.className += ' roll';
+      }
+    }
+  }
+}
+					

+					
Code syntax highlighting courtesy of highlight.js.

+				

+
+				

+					
Marvelous List

+					
    
+						
  • No order here
    • 
+						
  • Or here
    • 
+						
  • Or here
    • 
+						
  • Or here
    • 
+					

+				

+
+				

+					
Fantastic Ordered List

+					
    
+						
  1. One is smaller than...
    2. 
+						
  2. Two is smaller than...
    3. 
+						
  3. Three!
    4. 
+					

+				

+
+				

+					
Tabular Tables

+					
+						
+							
+								
+								
+								
+							
+						
+						
+							
+								
+								
+								
+							
+							
+								
+								
+								
+							
+							
+								
+								
+								
+							
+						
+					
ItemValueQuantity
Apples$17
Lemonade$218
Bread$32

+				

+
+				

+					
Clever Quotes

+					

+						These guys come in two forms, inline: The nice thing about standards is that there are so many to choose from and block:
+					

+					

+						“For years there has been a theory that millions of monkeys typing at random on millions of typewriters would
+						reproduce the entire works of Shakespeare. The Internet has proven this theory to be untrue.”
+					

+				

+
+				

+					
Intergalactic Interconnections

+					

+						You can link between slides internally,
+						like this.
+					

+				

+
+				

+					
Speaker View

+					
There's a speaker view. It includes a timer, preview of the upcoming slide as well as your speaker notes.

+					
Press the S key to try it out.

+
+					
+				

+
+				

+					
Export to PDF

+					
Presentations can be exported to PDF, here's an example:

+					
+				

+
+				

+					
Global State

+					

+						Set data-state="something" on a slide and "something"
+						will be added as a class to the document element when the slide is open. This lets you
+						apply broader style changes, like switching the page background.
+					

+				

+
+				

+					
State Events

+					

+						Additionally custom events can be triggered on a per slide basis by binding to the data-state name.
+					

+					

+Reveal.addEventListener( 'customevent', function() {
+	console.log( '"customevent" has fired' );
+} );
+					

+				

+
+				

+					
Take a Moment

+					

+						Press B or . on your keyboard to pause the presentation. This is helpful when you're on stage and want to take distracting slides off the screen.
+					

+				

+
+				

+					
Much more

+					
+				

+
+				

+					
THE END

+					

+						- Try the online editor 

+						- Source code & documentation
+					

+				

+
+			

+
+		

+
+		
+		
+
+		
+
+	
+
diff --git a/doc/pub/Splines/html/reveal.js/plugin/multiplex/package.json b/doc/pub/Splines/html/reveal.js/plugin/multiplex/package.json
new file mode 100644
index 000000000..bbed77a67
--- /dev/null
+++ b/doc/pub/Splines/html/reveal.js/plugin/multiplex/package.json
@@ -0,0 +1,19 @@
+{
+  "name": "reveal-js-multiplex",
+  "version": "1.0.0",
+  "description": "reveal.js multiplex server",
+  "homepage": "http://revealjs.com",
+  "scripts": {
+    "start": "node index.js"
+  },
+  "engines": {
+    "node": "~4.1.1"
+  },
+  "dependencies": {
+    "express": "~4.13.3",
+    "grunt-cli": "~0.1.13",
+    "mustache": "~2.2.1",
+    "socket.io": "~1.3.7"
+  },
+  "license": "MIT"
+}
diff --git a/doc/pub/Splines/html/reveal.js/test/simple.md b/doc/pub/Splines/html/reveal.js/test/simple.md
new file mode 100644
index 000000000..c72a44079
--- /dev/null
+++ b/doc/pub/Splines/html/reveal.js/test/simple.md
@@ -0,0 +1,12 @@
+## Slide 1.1
+
+```js
+var a = 1;
+```
+
+
+## Slide 1.2
+
+
+
+## Slide 2
diff --git a/doc/pub/Splines/html/reveal.js/test/test-markdown-external.html b/doc/pub/Splines/html/reveal.js/test/test-markdown-external.html
new file mode 100644
index 000000000..859d0a199
--- /dev/null
+++ b/doc/pub/Splines/html/reveal.js/test/test-markdown-external.html
@@ -0,0 +1,36 @@
+
+
+
+	
+		
+
+		reveal.js - Test Markdown
+
+		
+		
+	
+
+	
+
+		

+  		

+
+		
+
+		
+		
+		
+		
+		
+		
+
+		
+
+	
+
diff --git a/doc/pub/Splines/html/reveal.js/test/test-markdown-external.js b/doc/pub/Splines/html/reveal.js/test/test-markdown-external.js
new file mode 100644
index 000000000..cab85c6f6
--- /dev/null
+++ b/doc/pub/Splines/html/reveal.js/test/test-markdown-external.js
@@ -0,0 +1,24 @@
+
+
+Reveal.addEventListener( 'ready', function() {
+
+	QUnit.module( 'Markdown' );
+
+	test( 'Vertical separator', function() {
+		strictEqual( document.querySelectorAll( '.reveal .slides>section>section' ).length, 2, 'found two slides' );
+	});
+
+	test( 'Horizontal separator', function() {
+		strictEqual( document.querySelectorAll( '.reveal .slides>section' ).length, 2, 'found two slides' );
+	});
+
+	test( 'Language highlighter', function() {
+		strictEqual( document.querySelectorAll( '.hljs-keyword' ).length, 1, 'got rendered highlight tag.' );
+		strictEqual( document.querySelector( '.hljs-keyword' ).innerHTML, 'var', 'the same keyword: var.' );
+	});
+
+
+} );
+
+Reveal.initialize();
+
diff --git a/doc/pub/Splines/html/reveal.js/test/test-markdown-options.html b/doc/pub/Splines/html/reveal.js/test/test-markdown-options.html
new file mode 100644
index 000000000..5b3be9758
--- /dev/null
+++ b/doc/pub/Splines/html/reveal.js/test/test-markdown-options.html
@@ -0,0 +1,41 @@
+
+
+
+	
+		
+
+		reveal.js - Test Markdown Options
+
+		
+		
+	
+
+	
+
+		

+		

+
+		
+
+		
+		
+		
+
+		
+
+	
+
diff --git a/doc/pub/Splines/html/reveal.js/test/test-markdown-options.js b/doc/pub/Splines/html/reveal.js/test/test-markdown-options.js
new file mode 100644
index 000000000..3ae13503a
--- /dev/null
+++ b/doc/pub/Splines/html/reveal.js/test/test-markdown-options.js
@@ -0,0 +1,26 @@
+Reveal.addEventListener( 'ready', function() {
+
+	QUnit.module( 'Markdown' );
+
+	test( 'Options are set', function() {
+		strictEqual( marked.defaults.smartypants, true );
+	});
+
+	test( 'Smart quotes are activated', function() {
+		var text = document.querySelector( '.reveal .slides>section>p' ).textContent;
+
+		strictEqual( /['"]/.test( text ), false );
+		strictEqual( /[“”‘’]/.test( text ), true );
+	});
+
+} );
+
+Reveal.initialize({
+	dependencies: [
+		{ src: '../plugin/markdown/marked.js' },
+		{ src: '../plugin/markdown/markdown.js' },
+	],
+	markdown: {
+		smartypants: true
+	}
+});
diff --git a/doc/pub/Splines/ipynb/.ipynb_checkpoints/Splines-checkpoint.ipynb b/doc/pub/Splines/ipynb/.ipynb_checkpoints/Splines-checkpoint.ipynb
new file mode 100644
index 000000000..19b2da72b
--- /dev/null
+++ b/doc/pub/Splines/ipynb/.ipynb_checkpoints/Splines-checkpoint.ipynb
@@ -0,0 +1,1341 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "\n",
+    "# Data Analysis and Machine Learning  Lectures: Optimization and  Gradient Methods\n",
+    "\n",
+    "  \n",
+    "**Morten Hjorth-Jensen**, Department of Physics, University of Oslo and Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University\n",
+    "\n",
+    "Date: **Sep 20, 2018**\n",
+    "\n",
+    "Copyright 1999-2018, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license\n",
+    "\n",
+    "\n",
+    "\n",
+    "\n",
+    "## Optimization, the central part of any Machine Learning algortithm\n",
+    "\n",
+    "Almost every problem in machine learning and data science starts with\n",
+    "a dataset $X$, a model $g(\\beta)$, which is a function of the\n",
+    "parameters $\\beta$ and a cost function $C(X, g(\\beta))$ that allows\n",
+    "us to judge how well the model $g(\\beta)$ explains the observations\n",
+    "$X$. The model is fit by finding the values of $\\beta$ that minimize\n",
+    "the cost function. Ideally we would be able to solve for $\\beta$\n",
+    "analytically, however this is not possible in general and we must use\n",
+    "some approximative/numerical method to compute the minimum.\n",
+    "\n",
+    "## Steepest descent\n",
+    "\n",
+    "The method of steepest descent The basic idea of gradient descent is\n",
+    "that a function $F(\\mathbf{x})$, \n",
+    "$\\mathbf{x} \\equiv (x_1,\\cdots,x_n)$, decreases fastest if one goes from $\\bf {x}$ in the\n",
+    "direction of the negative gradient $-\\nabla F(\\mathbf{x})$.\n",
+    "\n",
+    "It can be shown that if"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "$$\n",
+    "\\mathbf{x}_{k+1} = \\mathbf{x}_k - \\gamma_k \\nabla F(\\mathbf{x}_k), \\ \\ \\gamma_k > 0\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "for $\\gamma_k$ small enough, then $F(\\mathbf{x}_{k+1}) \\leq\n",
+    "F(\\mathbf{x}_k)$. This means that for a sufficiently small $\\gamma_k$\n",
+    "we are always moving towards smaller function values, i.e a minimum.\n",
+    "\n",
+    "\n",
+    "## More on Steepest descent\n",
+    "\n",
+    "The previous observation is the basis of the method of steepest\n",
+    "descent, which is also referred to as just gradient descent (GD). One\n",
+    "starts with an initial guess $\\mathbf{x}_0$ for a minimum of $F$ and\n",
+    "computes new approximations according to"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "$$\n",
+    "\\mathbf{x}_{k+1} = \\mathbf{x}_k - \\gamma_k \\nabla F(\\mathbf{x}_k), \\ \\ k \\geq 0.\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "The parameter $\\gamma_k$ is often referred to as the step length or\n",
+    "the learning rate within the context of Machine Learning.\n",
+    "\n",
+    "\n",
+    "## The ideal\n",
+    "\n",
+    "Ideally the sequence $\\{ \\mathbf{x}_k \\}_{k=0}$ converges to a global\n",
+    "minimum of the function $F$. In general we do not know if we are in a\n",
+    "global or local minimum. In the special case when $F$ is a convex\n",
+    "function, all local minima are also global minima, so in this case\n",
+    "gradient descent can converge to the global solution. The advantage of\n",
+    "this scheme is that it is conceptually simple and straightforward to\n",
+    "implement. However the method in this form has some severe\n",
+    "limitations:\n",
+    "\n",
+    "In machine learing we are often faced with non-convex high dimensional\n",
+    "cost functions with many local minima. Since GD is deterministic we\n",
+    "will get stuck in a local minimum, if the method converges, unless we\n",
+    "have a very good intial guess. This also implies that the scheme is\n",
+    "sensitive to the chosen initial condition.\n",
+    "\n",
+    "Note that the gradient is a function of $\\mathbf{x} =\n",
+    "(x_1,\\cdots,x_n)$ which makes it expensive to compute numerically.\n",
+    "\n",
+    "\n",
+    "\n",
+    "## The sensitiveness of the gradient descent\n",
+    "\n",
+    "GD is sensitive to the choice of learning rate $\\gamma_k$. This is due\n",
+    "to the fact that we are only guaranteed that $F(\\mathbf{x}_{k+1}) \\leq\n",
+    "F(\\mathbf{x}_k)$ for sufficiently small $\\gamma_k$. The problem is to\n",
+    "determine an optimal learning rate. If the learning rate is chosen too\n",
+    "small the method will take a long time to converge and if it is too\n",
+    "large we can experience erratic behavior.\n",
+    "\n",
+    "Many of these shortcomings can be alleviated by introducing\n",
+    "randomness. One such method is that of Stochastic Gradient Descent\n",
+    "(SGD), see below.\n",
+    "\n",
+    "## Gradient Descent Example\n",
+    "\n",
+    "We revisit now our simple linear regression example with a linear polynomial."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "[[3.79420435]\n",
+      " [3.13985512]]\n",
+      "[[3.79420435]\n",
+      " [3.13985512]]\n"
+     ]
+    },
+    {
+     "data": {
+      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYwAAAEWCAYAAAB1xKBvAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3XmYHGW1x/HvmckeQvZAQpIJSwirbEFWISwJiCjqdQEDFxCNRK8rylXjgkhUXMAVERGIJCIYFxTwmgiThASSMNkgK4SQjYTs+56Zc/+omrEzdPfUTFd3dc/8Ps/Tz3TXeqq65z1V7/tWlbk7IiIiDSlLOgARESkNShgiIhKJEoaIiESihCEiIpEoYYiISCRKGCIiEokShhScmS03s8vD9183swcTimOIma1OYt0thZk9YmZ3JR2HxEMJQw5hZtea2Qwz22Vm68P3nzYzy8f63P177v6JXJdjZgPMzM2sVRxxJU0FrRQjJQypY2a3AT8DfgQcCRwB3ApcALTJME95wQIUkUQpYQgAZtYZuBP4tLuPd/cdHpjj7sPdfV843SNm9msze8bMdgGXmNl7zGyOmW03s1Vmdke9Zd9gZivMbJOZjao37g4zG5vy+Vwze8HMtprZPDMbkjJukpl918ymmdkOM5tgZj3C0VPCv1vNbKeZnZdmG9uH8W8xs4XA2fXG9zGzP5vZBjN7w8w+lzLunWZWFW7jOjO7J2XchSkxrzKzm8Lhbc3sx2a2MpznfjNrH44bYmarzey28ExurZndHI4bAQwHbg+35R8ZvrMTzGyimW02syVm9pFweBszm2tmnw0/l4f77Fsp2/JiGO9aM/ulmbVJWa6HZ5Wvhfv5u2Z2bLiN283sidrpU7bj62a2MaxuHJ4u3nD6q8PYtobLe0emaaUIubteegFcCRwEWjUw3SPANoKzjjKgHTAEODX8/A5gHfD+cPqTgJ3ARUBb4J5wPZeH4+8AxobvjwI2AVeFyxoafu4Zjp8EvA4cD7QPP/8gHDcA8GzxAz8Ange6Af2A+cDqcFwZMAv4FsHZ1DHAMuCKcPyLwA3h+8OAc8P3FcAO4DqgNdAdOD0cdy/w93B9nYB/AN8Pxw0J98Od4XxXAbuBrin7+a4s29IRWAXcDLQCzgA2AieF408BtgAnAqOA6UB5OO4s4NxwvgHAIuALKct24EngcOBkYB/wbLhPOgMLgRvrbcc94fd7MbALGFR/O8IY1wPnAOXAjcByoG3Sv3+9or10hiG1egAb3f1g7YCUo+Y9ZnZRyrRPuvs0d69x973uPsndXwk/vww8RlBwAHwIeMrdp3hwlvJNoCZDDNcDz7j7M+GyJgJVBIVprYfd/VV33wM8AZzeiG38CDDa3Te7+yrg5ynjziZITHe6+353Xwb8Frg2HH8AOM7Merj7TnefHg7/GPBvd3/M3Q+4+yZ3nxu2+YwAvhiubwfwvZTl1S7zznC+ZwgS66CI23I1sNzdH3b3g+4+B/gz8GEAd58P3AX8DfgyQbKrDsfNcvfp4XzLgd/wn++r1g/dfbu7LyBIrBPcfZm7bwP+SVD4p/qmu+9z98nA0+G+rm8E8Bt3n+Hu1e4+hiAZnRtxmyVhShhSaxPQI7XR2N3Pd/cu4bjU38qq1BnN7BwzqwyrcrYRtHvUVhX1SZ3e3XeFy0unAvhwmKS2mtlW4EKgd8o0b6W8301wtB/VIbEAK+qtu0+9dX+doB0H4BaCM5vFZvaSmV0dDu9HcNZTX0+gAzArZXn/Fw6vtSk1QTdyeyqAc+rFO5yg7anWmHC6Z9z9tdqBZna8mT1lZm+Z2XaCRNaDQ61Leb8nzefUOLeE32utFQT7Ol3Mt9WLuV+GaaUIKWFIrRcJjvauiTBt/Vsc/4Gg6qWfu3cG7gdqe1WtJSgUADCzDgTVNumsAh519y4pr47u/oMmxJTOIbEA/eut+4166+7k7lcBuPtr7n4d0Au4GxhvZrXVQsemWddGgoL15JTldXb3qAmhoe1ZBUyuF+9h7j4yZZr7gKeAK8zswpThvwYWAwPd/XCCxJhLL7iu4b6o1R9YkyHm0fVi7uDuj+WwbikgJQwBwN23At8B7jOzD5lZJzMrM7PTCerLs+kEbHb3vWb2ToJqmlrjgavDhuE2BHX2mX53Y4H3mtkVYUNtu7BRtW+ETdhAUNV1TJZpngC+ZmZdw2V+NmXcTGCHmf1v2DhebmanmNnZAGZ2vZn1dPcaYGs4Tw0wDrjczD5iZq3MrLuZnR5O91vgXjPrFS7jKDO7IsK2QHBEn21bngKOt6BDQevwdbaZnRiu6waCtoqbgM8BY8ysNll1ArYDO83sBGDk2xffaN8JG9vfRVBd9qc00/wWuDU8IzUz62hBh4lOMaxfCkAJQ+q4+w+BLwG3ExRY6wjqt/8XeCHLrJ8G7jSzHQSNxk+kLHMB8BmCs5C1BA2xaS+WC9sVriE44t1AcET6FSL8Tt19NzAamBZWd6SrF/8OQXXJG8AE4NGU+asJCrrTw/EbgQcJGnkh6BSwwMx2EnQ9vtbd97j7SoI2ltuAzcBc4LRwnv8FlgLTw6qffxO9jeJ3wEnhtvwtzfbuAIYRtImsIaiquxtoa2b9gZ8C/x22t/yBoC3o3nD2LxMk9R0EhfjjEWPK5C2C73UNQQK91d0Xp4m5Cvgk8Mtw+qUECU1KhLnrAUoi0jQWdHse6+5RzgKlxOkMQ0REIlHCEBGRSFQlJSIikegMQ0REIimpO3v26NHDBwwYkHQYIiIlZdasWRvdvWfDU2ZXUgljwIABVFVVJR2GiEhJMbMVDU/VMFVJiYhIJEoYIiISiRKGiIhEooQhIiKRKGGIiEgkShgiIhKJEoaIiESihCEiIpEoYYiISCRKGCIiEkneE4aZPWRm681sfppxt5mZm1n9B9CLiEiRKcQZxiMEj7c8hJn1I3jE5MoCxCAiIjnKe8Jw9ykEzzqu716CZ0frgRwiIiUgkTYMM7sGeNPd50WYdoSZVZlZ1YYNGwoQnYiIpFPwhGFmHYCvA9+KMr27P+Dug919cM+eOd/OXUREmiiJM4xjgaOBeWa2HOgLzDazIxOIRUREIir4A5Tc/RWgV+3nMGkMdveNhY5FRESiK0S32seAF4FBZrbazG7J9zpFRCR+eT/DcPfrGhg/IN8xiIhI7nSlt4iIRKKEISIikShhiIhIJEoYIiISiRKGiIhEooQhIiKRKGGIiEgkShgiIhKJEoaIiESihCEiIpEoYYiISCRKGCIiEokShoiIRKKEISIikShhiIhIJEoYIiISiRKGiIhEooQhIiKRKGGIiEgkShgiIhJJ3hOGmT1kZuvNbH7KsB+Z2WIze9nM/mpmXfIdh4iI5KYQZxiPAFfWGzYROMXd3wG8CnytAHGIiEgO8p4w3H0KsLnesAnufjD8OB3om+84REQkN8XQhvFx4J+ZRprZCDOrMrOqDRs2FDAsERFJlWjCMLNRwEFgXKZp3P0Bdx/s7oN79uxZuOBEROQQrZJasZndBFwNXObunlQcIiISTSIJw8yuBG4HLnb33UnEICIijVOIbrWPAS8Cg8xstZndAvwS6ARMNLO5ZnZ/vuMQEWlWxo2DAQOgrCz4Oy5jzX5sCtFL6jp37+3urd29r7v/zt2Pc/d+7n56+Lo133GIiBS1xiSAceNgxAhYsQLcg78jRuQ9aRRDLykRkZatsQlg1CjYXa82f/fuYHgeKWGIiCStsQlg5crGDY+JEoaISNIamwD692/c8JgoYYiIJK2xCWD0aOjQ4dBhHToEw/NICUNEJGmNTQDDh8MDD0BFBZhB9+7Qvj3ccENee0wpYYiIJK1+AqioCD4PH559nuXL4dFHYc8e2LQp7z2mrJQush48eLBXVVUlHYaISPEYMCBIEvVVVAQJBTCzWe4+ONdV6QxDRKSUFbDHlBKGiEgpK2CPKSUMEZFSVsAeU0oYItIyJXAvprystykN5k2kRm8RaXlqb8WRenV1hw55K2iTXm9cjd5KGCLS8kToWVTQ9ZaXw5gxeUsa6iUlItJUCd2LKePyq6sLcrfZXClhiEjLk9C9mLIuvwB3m82VEoaIJC9bQ3A+GqcTuhdT2vWmyvcZTo6UMEQkWdmeBZGvBwXls2dRtgRXu97y8vTz5vsMJ0dq9BaRZGVrgIZkGqebKmovqAL3llIvKRFpHsrKgrOH+syCv5nG1dTkN66maEzvq3HjgjaLlSuDM4vRo9VLSkQkq2wN0JnGlZUV/oK7KBrT+6r2brM1NcHffF7/EZO8Jwwze8jM1pvZ/JRh3cxsopm9Fv7tmu84RKRIZWuAztRIXF2d91t5N0lTel/l+Yrz6v3VsS2rEGcYjwBX1hv2VeBZdx8IPBt+FpGWKFsDdP1x6RqLd++GG28sjjOOxva+ylOj/htTVvGb4VP4UN8X6dFuR07LOoS75/0FDADmp3xeAvQO3/cGlkRZzllnneUi0oKZuQdFa+ZXhw7uY8cmF+PYse4VFUGsFRXZY6moyLwdDc2bYuuKrf7Xr073kSdP9mNbLa9bRL/y1f7xgVMcqPIYyvKCNHqb2QDgKXc/Jfy81d27hO8N2FL7Oc28I4ARAP379z9rRboGJRFpGTI1KtdXrL2o6svU4F8rQ8+pg3sPMnPMIiY8tomJs7sxY8dJVNOKw9jBkF6LGPauPQz9eD8GXXk0VmbNp9Hbg4yVcY+5+wPuPtjdB/fs2bOAkYlI0WnowrdacV0Al+872jZ03UXK1d+vP7eCX183hQ/0mU739ru44NZTuXPyRRysKeOrF0xl8s/nsWlHW/6x7p18dvzFnHDVMViZxRpuq1iXFt06M+vt7mvNrDewPqE4RKSU1B5p13ZHLSsLGsDri+MCuPrXStS2L6TGkavRo99+PUY9vmIlx7ZeyRsHK4AKKspX89ET5jH03a257DMn0O3YU+KJJYKkzjD+DtwYvr8ReDKhOESk1KR2Rx0zJn+3+Bg16u0Fedz3exo+PGiwt8xnAqvoy6k91vLLD0/m1QnLeWP/UTyw6CI+fM95dDu2sB1M836GYWaPAUOAHma2Gvg28APgCTO7BVgBfCTfcYhIM1T/jCPOC+DyfEdbr3Fem7icI8f8hcMztGNUt25Hn9/cxZM3nxPLOnOlK71FJD4FvHo57/LwzIzNr2/h2V8tZsIzB5i49BhWVPelmjLKMjXjjh0by/5rNo3eItJM5OtGgXFpbAN2ugb21q1h587Iy9i/cz+TfzaXb1w4iXcetoAex3XmI/eexxNLTuPMXqv59XVTqO7VJ/3MFRXFl2zj6JtbqJeuwxApYpmuKaioSDqy4HqGDh0af71G6jUV3bu7t2mTdRk11TW+6OnX/WcfnORX95rhHdnh4F7OAb+g0zy/Y0ilv/Cbl/3AngO5x9YIlNJ1GHFRlZRIEct2E8GkbxQYR/VShmVU9+nL+A89zoR/VjNh2bGsrg7OGI5rvZxhg1Yw9Op2XDLyBDr375x52XmuytPdakWkuCT1nOwosl0gF7WdIMMyajDKqaGLbeWyPosYNuQAQ0cczdEX9csx6PioDUNEiktST7GLItt1GQ20s3iNs+DJpWxv1yv9eIwaytjc7zTG372MEWMvKqpkESclDBGJRz6fYperbFeIp7m2Yv2CDfzhM9O4eeDz9G39Fqe8/zhu3fMT9tD+bbOXU4Ph2MqVxdXInweqkhKRlmHcOLj++rSj3Izn7q5iwvjtTHzlCObsORGAbraZy/suZugl1QwdcTQVyyc3fJV5rlVweWjPUJWUiEhjDB/+n8e+1rPK+3L57Wdy78zzObzNPkYPncTMRxay4eGnebzsY3zi0YupGH5hMHHtVeaZGvLrX9jXmO68Rd41WQlDRFqEt15ez9TjbmSftT1k+B7aManff/PUt19i89r9TNp6Ol+fMISzW82h7NO3Zi68ozwsqbEJoBC3I8mBqqREpFnas3kPz/9mIRP/soMJ83vz8t5BAHyS3zC6/Nv0qF5P9ZF9aPXju9NX+TTU66v+zQnh7bcjb2zPsTx1TVa3WhGRFDUHa3jlL68xYcxaJk7vxJTNJ7OPdrRhHxd0Wciwc7Yx7IYjOP2jgyhrFaFyJUrh3VB7Q2MTQJ66JseVMJK6vbmIxKU53b+pkdbMfot/37+UCRONiSuOZ70PAgZxctvX+PSZ0xl6TUcuuvUkOvY6o/EL798/feGdWuVU+xjZXJaRKt3tzoulazLo1iAiJS3O20o05tGiCdm1YZf/87sv+RfPrPRT2r5at8k9bb1/rGKqP/KJ5331S2viWVkc+3bkyPS3Sxk5Mvt6Y/4eiOnWIIkngca8lDBE6onr/k0FuJ9RU1QfqPZZYxf6D66s9Eu7zvI27HVwb8sev7xbld/97kqf88fFXn2gOj8B5Fp4F8n9tZQwRCQoyNIVSGaNW06RFGzu7qtmrvGHbp7i11VM9R62oS6UU9st8dvOqvR/fa/Kd23Y1fQVZEsCjU0QDU0f1/eTIyUMEQnuoBpHQZ9gwbZz3U5/+o6Z/vnTJ/mJbZbWrfqIsnV+/dHP++8/9byvmfNWPCtLdyZlFlQRNfYsK8r0DSXiAlUDKmGItHRjx779dtvg3rp1UVedVB+o9pfGLPDvDav0IV1me2v2Obi3Y7cP6/6S//jqSp/3pyVeU10T+7ozbmft7csbsw+i7LNsSaWA1YBKGCItXaYCq3v3xi9r5Mi3n2XEWHiteGG1P3jjFP9Iv2ne3TbWreL09ov8K2dX+sS7Z/meLXtiWVdWmc6ksr0ynWVFPSvLdBZRwCSthCHS0sVRjTR2bPoj69pqmvrTRqw+2f7mdv/7N2b4Z98xyQe1eb1usSP5lW9odYTXYH6wd9/CN6pnKqSzvXI5w8imgNWAShgixapQ3VNzLbDSVYk0pWrF3Q/uO+gzHprv372s0i/qPMdbsd/BvT27/N09Z/o911T6yi/+xGuS7ok1dmzjzjJybcPIRmcYjVw5fBFYAMwHHgPaZZteCUOKXiG7p+arwEp3pJth2u0dj/APHfWCd7XNdYPPbL/Qv3pupT/749m+d9vehtdX6J5Y6arfMsWVay+phuZVG0bkZHEU8AbQPvz8BHBTtnmUMKToFbpQzKXAaqjQTI05w7TVmPctf9NvHjjFH/vsNF+/cEPj11fgLqbu/p/91thqqHzFoV5SkRLGKqAbwS1KngKGZZtHCaPIlMCVwQVXTIViQ7IVmB06+MGHf+8v/vYV/84llb7Weqedbl/PPtF7MxXLGUaqIr1gMW4lnzCCbeDzwE5gAzAuwzQjgCqgqn///rHuRMlBC/lHa7S4rosohDTfYQ347taH+z1d7vDObA1yHdU+qvXdvq+sbW7fd7H+ZlrAgU/JJwygK/Ac0BNoDfwNuD7bPDrDKCLFeLSYtFyui0io0Nr509/6rsOP9BrMV3OUX8dYB/f+5av8E4Mm++NfmOYbX90UX4wtoHAuRs0hYXwY+F3K5/8G7ss2jxJGESmlqpfGyKVAa+p1EQU88t6/a79PvW+ef/viSj/vsJe9jIMO7oex3d97xHT/xYcm+eJnXs/PRXOSmOaQMM4Je0h1AAwYA3w22zxKGBGUSpfOYpRrwd3UJJrHfVlTXeOvTnjDf/XRSX7NkdP98LCaqYyDfk7HV/wbF1b6lF/M9f279ue8LileJZ8wgm3gO8DisFvto0DbbNMrYTSglLp0FqNcCu6xY93Ly5s2f5RE04gDgc3LtvifvvSCjzhhsg9otbJucQNarfQRJ0z28V9+wTcv29LwNknh5emAr1kkjMa+lDAaUEpdOotRU88Qsl0AFyWJRrlBXZbkvH/Xfp/yi7n+jQsr/ZyOr9RVMx3OVr/myOn+q49O8tf+vVzVTMUujwdhBUsYwETgtDhWlutLCaMBzbVdoVCamnAzzVdeHu2fvaGCIsPyq63Mf3T4HX4Y2+uqmc477GX/9sWVPvW+eapmKjV5POArZMI4E6gEHgZ6x7HSpr6UMBrQHNsVCqmpR3hxVCllG5/lArtdtPff9fmG/+X2F33L8q3x7AdJRh4P+ApeJQX8FzAP+Hbt1dmFfilhNKA5tis0VVOry5oyX45VSuns27HPK++d418/v9LXZLhormQPCJpbVWZcmsMZRrAuDDgFuBXYCKwGbogjgMa8lDAi0D9j4RNnE6uUUguCmuoaX/iPpf7TD0zy9/Sa4R3ZEdRqccC/3fZ7b79oLt1RaCl83zqoyayZtGFMA9aEbRnfBa4GjgN+ATwQRxBRX0oYEkkSVXNNqFKqMfPHPjvNbx44xY8qW1M3amDrZf6ZUyf537423beu2Pqf5WfqhVVKha+qTbMr8l5SFiwrMzM7GVjoaSY0s0XufmLWBcRo8ODBXlVVVajVSakqKwuKoXTGjoXhwwsbz4ABsGLF2wYvp4KjWU5X28JlfRYz7NIDDB1xDAMu7Jt+OePGwYgRsHt39vVVVMDy5TmHnReZvhszqKkpfDwthJnNcvfBuS6nrKEJ3H1BumQRek+uAYjErn//zONGjAgK3gLwGmfBk0t5pvv17KXdIeP20o45J13PjIcWsGHv4fxp9Xl88vcXZU4WECS6Bx4IEoJZ5ulWrmx8sOPGBYmtrCz4m699lOm7yfadSfGI4zSlUC9VSUkkjXkwUMzeemW9jx051W889nnvk1LN9KXye3xT6+BJc9VH9YunqiGu6h1d8NnsoQv3RLIYOzZzwojxupQ9W/b4xLtn+VfOrvTT2y+qW0U32+Qf7TfNH7xxiq94YXVs6ztEXIWvLvhs9pQwJF7N8Z84DwVhTXWNz/vTEv/x1ZU+rPtL3o7dDu6t2edDusz27w2r9JfGLPCD+w7GthlZxfG9FfsFn83xt1lgShgSn+ZaTRDTdq2dt85//6nn/fqjn/cjy96qW9RJbV/zz58+yZ++Y6bvWLsjvpgLXTgWc8+l5vrbLDAlDIlPMRcY9TW2QG1CAbxrwy7/v7te8tvOqvRT2y2p2x09bINfVzHVH7p5iq+auSaGjUkTaxKFYzEXyqX02yxiShjSeJkKz2KvkqiVp4Kt+kC1z/njYr/73ZV+ebcqb8seB/c27PVLu87yH1xZ6bP/sMirD1THtCEZJFk4Fmu1T6n8NotcXAmjweswiomuw8hBuj78HToE3TRHjUp7nUDR9efPcD1DU+JcM/stJv56KRMmGv9eeTzrvScAp7R9jWGnvMnQ9x/GRbeeRIceHXKPOypdo/B2MX7nLVlc12EkftbQmJfOMHKQ7ei12Kok8nAmtHPdTn/mOzP9C2dM8pPbvlo3ay9b78MHTPUxn3ze35y1tmlxxUXVL29XbL/NEoWqpBKSj0IjyjJzXW9DhW2xVEmkKyDM3EeObFSBWn2g2meNXejfv6LSL+ky29uw18G9LXt8aLcq/+FVlT738cXRq5kaU3DlcuNDFY5vVyy/zRKmhJGEfPxDR1lmHOvN59FrnP/QmeKsTRpZ9sPK6W/6726a4tf2n+Y9bEPdJO9ot9i/PLjS//W9Kt+9aXe8cdXff7l+VyocJQ+UMJKQj0I3yjLjWG++jl7jXm6WZz/UFaBhgVrdt5/P/uCd/rnTJvkJbZbWTXZk2Vt+wzHP+6O3TvW189bltn0NxVW/Oqw5VSspeTUbShhJyEePjSjLzOXRoan/8LXVOnEWAHEXkJmWR3B315mPLPDRQyv94s5zvDX7HNzbsduv6P6S/+R9lf7y+CX5eRRp1O1sLr16VD3WrChhNJcLnPJ1hlGof/i4C8ixYzMucyV96z6e0X6h3/7OSv/3D2f5ni174t2mTHFF2Z/N5QyjuWyHuHszSRhAF2A8sBhYBJyXbfq6hNGcLnDKVxtG9+6F+YePuWDZtmqbv37mB72m3vJ20d7v7/VNH/fpqb5u/vpYNyGyqJ0TmsOReXM5UxJ3bz4JYwzwifB9G6BLtunrEkZzu8Ap7l5SBbrxXt26ciggD+476NMffMXvvLTS33X4XG/Ffgf3m/idv1V2pNdgvq9nH695tIQK3OZQ968zjGal5BMG0Bl4A4KLB6O86hJGMR79FFMhkaUdIC//8I3c9mWTV/pvhk/2/zrqBe9iW4Kvjmo/q8MC/+q5lf7cT2b73m17449TomsuZ0ri7s0jYZwOzAQeAeYADwId00w3AqgCqvr37x9sfbEd/RTbP1e2nka1MRUwwW1dsdX/+tXp/ulTJvlxrd+oC6Vv+Zv+8YFT/I+fm+YbFm/M2/qliYrpIEhy0hwSxmDgIHBO+PlnwHezzZN4G0YmxZbAMsXTvXswPs/778CeAz7t/pf9jiGVfn6neV7OAQf3juzwq3vN8J99cJIvevr1/PRmiktLKSxbyna2cM0hYRwJLE/5/C7g6WzzJN5LKpNiqyJrKCHkIcEtfXa533ftZP9A7xe9M1vrqpnO7jjfR11Q6ZN/Ptf37dgXy+blXbEdkORLS9lOKf2EEWwDzwODwvd3AD/KNn3i12FkUmxnGO7ZE2oMCW7L8q3+56+86J86cbIf02p53SL6l6/yT54w2Z/44gu+8dVNsW9WQTT0fRbTwUouivF3K3kRV8JI9G61ZnY6QdtFG2AZcLO7b8k0fdHerTbbnWCHD08urkyacAfQA7sPMOORRUx8fDMTZndn5s6TqKGcTmznkiMXMexdexl6S38GDh2AlVlew8+7bHeNffTR0vqus9HdcVsM3a222JTSUWeEqoia6hp/dcIb/ssPT/L3HTndO7HNwb2Mg37uYS/7N99V6c//ap7v37U/ue3Il2xH3s3pqLw5bYtkRXOokmrsK6eEEVeBXkqJIZs027Fp6Wb/05de8E+eMNkrylfVlR9Ht1rhnzpxso//8gu+edmWoog17+vLlFCLrb0qF2rDaDGUMBojrn+MZvYPtm/HPp/887k+6oJKf2fHV9yodnA/nK3+/t4v+n3XTvalzy5v+griKOiTvKo/9Wr57t3/sz3N6ai8uRwASVZKGI0R1z95iRcWNdU1vujp1/3n/zXJr+41wzuyw8G9nAN+fqd5fseQSp92/8t+YM+B3FcWV0Gf1D7PFH8Dt1gXKUZxJYyW8YjWuBr3SrCRcOOSTTx73xIm/LOaicuOYVX1UQB5pX+pAAAPjUlEQVQc13o5Q49fybD3tuWSkSfQuX/neFcc16M1k9rn2eIfPTp4rO3KldC/f/C51Bq8pUWJq9G7ZSSMuAqvEni+8L7t+3jhwYVMHL+NCfN6MXv3CThldLGtXNZnEUMvPsDQTw7gmCH98xtIXAV9Uvu8BA8ORDKJK2GUxRFM0Rs9Ouj6mKpDh2B4EsvJ1bhxQUFaVoZXVLD6tnv46Qcm855eL9Gt80Euve0MfvTiBbRvdYDvXDqF6Q/OZ8Puwxi/+jw+Ne6i/CcLCI68GzM8k6T2eVzxizQncdRrFeqlXlLB+mvatT+kDn0nHfw6xvrxrZf5Z06d5E9+fbpvW7WtsHGliTO2uv4k9nkz6+AgLRtqw2g59m7dy7TfLmTin7fzuZkfo4+vfds01Yd3oXxbxmsekzFuXGnX9Zd6/CIhVUk1Y17jzP/ra9xzzSSu7FFFt641XH77mfxkxgUcmSZZAJRv3xoUcNmkVGUxYEDD0+dq+PCgnaGmJvjb1MK20HHXiit+kWZCCSNpYWHoZWXs7HQkv+71LY5qvY5TPziQ2/4+hBU7uvLJ017iH9+cyeY391JWXp55WaNGZV/PiBFBA7J78HfEiMIUvrkU+EnGLSKHUJVUQvZs3sPSkT9h0Pi7aFOzr274btrzu2630/F9lzH01mPpd06fQ2e0LPdpytaDJ6neRrneZ6sEeqaJFDt1qy0xXuO8PP5VJv5+LRNe7MTzm09iEScygEYWhpkK0IbmK8brGaIU+OreKpIztWGUgLVz1/H7EVO54Zip9G69gdM/OoivPD2EN3d15tYzZlLByvQzrswwHIKG1zZt3j68devsXU2T6iaaaVuybWMqdW8VKRpKGDHavXE3/xpdxW2DJ3Fqu9foc8YR3PjbC/nX8kFc1u81Hr5lKqtfWsuCvcdx7+yLsYomFIbDh8NDD0H37v8Z1r07PPxw9iqeUr2eoViufRGRFnQdRh5UH6j22X9Y5D+4stIv6zrL27DXwb0te/yyrrP87ndX+pw/LvbqA9XpF1Dovv7FfD1DttiSvvZFpMShmw8mY/VLa/zhW5736yqmek9bX1cGntpuid92VqX/310v+a4Nu6IvsCUUhg1toy6SE8mruBJGq6TPcIrdrvW7mPzrhUx8cjcTFh7Fwn3HAb05omw9VwxYwrBhS7j81oH0Pv144Pikwy1Ow4dnry4bNerQXlQQfB41Stc+iBQRJYx6ag7WMOePS5jw6DomzOzMtK0nc4CzacceLuq+gJsvn8Swm/pw6gcHYmW9cltZ/S6ntdcYQMsqKHNtGBeRglC3WmDVjDVM+PXrTHyunH+vHsQmDxqUT2u3hGGnrmXoBztx4YiTaN+tfbwr1jUGAe0HkbxSt9oc7Fizg6e+NZPPnTaZE9u+Tv9z+/CJMe9iypvH8J6jFzF25DTeemUDc/cM4oczhzD0q2c1nCyacjWzjqwDhewJldRtRkSagzgaQnJ5AeXAHOCphqZtaqP3wX0HfcZD8/2uyyv9os5zvBX7Hdzbs8uv7DHT77mm0l/5y6teU13TpOU3qtE2tQG4vPzQeUrsCX6xitr4n0snATWuSwtFc7lbrZl9CRgMHO7uV2ebtjFVUsunrmbiA8uY8Fxrnl1zAlu8KwBntl/E0HesY9iHO3P+LSfSrku7XDchepVKuttk1NeY22a0NLrNiEiTxFUllfTZRV/gWeBScjzD2LZqmz/59en+mVMn+cDWy+oOII8qW+M3D5zif/ifab5+4YbGJuZozNKfKZgdOl2m51OXl5dOt9okuwHn+nzvqN+TSDNDM+lW+1PgdqBTpgnMbAQwAqB/ytXBB/cepGrsYib8YSMTZ3Xlxe0nU805dGAXQ3ou5DMXrmToTUdx4tXHYmW987sV/funP3KtfzVzpraJmprivi9S7XMhVqwI7uFUe1Za6F5dcdxmJMr3JCLpxZF1mvICrgbuC98PIcIZxinHn+r3f2yyf7DPi96ZrcHBIdU+uMMC/9p5lV557xzfu21vfGk5qqh14005Qk76wr5025ZUm0uuZxhqw5AWilK/0hv4PrAaWA68BewGxmaf5ywH937lq/2W46f441+Y5hsWb4xzvzZdlIK9sQVWUwu4OJNMpkI6iSqdOAr8pBOwSAJKPmEcEkTEM4x+XY73xc+83vTeTMWgMQVWU89I4jyKzlTvn1SvLhX4Io0WV8JIvJcUgJkNAb7sMfaSahaa8iyIuHsCZXv+BqhXl0gJaFYX7rn7pIaSRYvUlFuDx30xYLqL6mqf+ldRoWQh0oIURcKQDJpyBXTcDxwaPjxIChUVQaKoqIBHHw3OfJYvV7IQaUGUMLJJ+jYS6Qrrho7o83GbjeHDg+RQU6MkIdKCJX0dRvEqljvJNnRr8HTTQ3DdxMqVwZnF6NEq5EUkZ0XR6B1VQRu9dRsJEWkmmlWjd1HSnWRFRA6hhJFJ3I3HIiIlTgkjkyiNx0k3iouIFJASRiYN9VCqbRRfsSLoYlrbKK6kISLNlBq9m0qN4iJSItTonTQ1iotIC6OE0VRqFBeRFqY0E0YxNDbn44pqEZEiVnoJo1gam5ty2w4RkRJWeo3eGzeqsVlEpBFabqO3GptFRBJReglDjc0iIokovYShxmYRkUSUXsJQY7OISCJK83kYjX1GhIiI5Kz0zjBERCQRiSUMM+tnZpVmttDMFpjZ55OKRUREGpZkldRB4DZ3n21mnYBZZjbR3RcmGJOIiGSQ2BmGu69199nh+x3AIuCopOIREZHsiqINw8wGAGcAM9KMG2FmVWZWtWHDhkKHJiIiocQThpkdBvwZ+IK7b68/3t0fcPfB7j64Z8+ehQ9QRESAhBOGmbUmSBbj3P0vScYiIiLZJdlLyoDfAYvc/Z6k4hARkWiSPMO4ALgBuNTM5oavqxKMR0REskisW627TwUsqfWLiEjjJN7oLSIipUEJQ0REIlHCEBGRSJQwREQkEiUMERGJRAlDREQiUcIQEZFIlDBERCQSJQwREYlECUNERCJRwhARkUiUMEREJBIlDBERiUQJQ0REIlHCEBGRSJQwREQkEiUMERGJRAlDREQiUcIQEZFIlDBERCSSRBOGmV1pZkvMbKmZfTXJWEREJLvEEoaZlQO/At4NnARcZ2YnJRWPiIhkl+QZxjuBpe6+zN33A38ErkkwHhERyaJVgus+CliV8nk1cE79icxsBDAi/LjPzOYXILZc9QA2Jh1EBIozPqUQIyjOuJVKnIPiWEiSCSMSd38AeADAzKrcfXDCITVIccarFOIshRhBccatlOKMYzlJVkm9CfRL+dw3HCYiIkUoyYTxEjDQzI42szbAtcDfE4xHRESySKxKyt0Pmtn/AP8CyoGH3H1BA7M9kP/IYqE441UKcZZCjKA449ai4jR3j2M5IiLSzOlKbxERiUQJQ0REIimahNHQbULMrK2ZPR6On2FmA1LGfS0cvsTMrkgwxi+Z2UIze9nMnjWzipRx1WY2N3zltXE/Qpw3mdmGlHg+kTLuRjN7LXzdmHCc96bE+KqZbU0ZV5D9aWYPmdn6TNf/WODn4Ta8bGZnpowr5L5sKM7hYXyvmNkLZnZayrjl4fC5cXW/zCHOIWa2LeW7/VbKuILdSihCnF9JiXF++HvsFo4ryP40s35mVhmWOQvM7PNppon39+nuib8IGr1fB44B2gDzgJPqTfNp4P7w/bXA4+H7k8Lp2wJHh8spTyjGS4AO4fuRtTGGn3cW0b68Cfhlmnm7AcvCv13D912TirPe9J8l6BhR6P15EXAmMD/D+KuAfwIGnAvMKPS+jBjn+bXrJ7gdz4yUccuBHkWyP4cAT+X6e8l3nPWmfS/wXKH3J9AbODN83wl4Nc3/eqy/z2I5w4hym5BrgDHh+/HAZWZm4fA/uvs+d38DWBour+Axunulu+8OP04nuLak0HK55coVwER33+zuW4CJwJVFEud1wGN5iiUjd58CbM4yyTXA7z0wHehiZr0p7L5sME53fyGMA5L7bUbZn5kU9FZCjYwzqd/mWnefHb7fASwiuINGqlh/n8WSMNLdJqT+htdN4+4HgW1A94jzFirGVLcQZPZa7cysysymm9n78xBfrahx/ld4ijrezGovoCzUvmzUusKqvaOB51IGF2p/NiTTdhRyXzZW/d+mAxPMbJYFt+JJ2nlmNs/M/mlmJ4fDinJ/mlkHgoL2zymDC74/LaiiPwOYUW9UrL/Por81SCkys+uBwcDFKYMr3P1NMzsGeM7MXnH315OJkH8Aj7n7PjP7FMGZ26UJxRLFtcB4d69OGVZM+7NkmNklBAnjwpTBF4b7shcw0cwWh0fYSZhN8N3uNLOrgL8BAxOKJYr3AtPcPfVspKD708wOI0hYX3D37flaDxTPGUaU24TUTWNmrYDOwKaI8xYqRszscmAU8D5331c73N3fDP8uAyYRHA3kQ4NxuvumlNgeBM6KOm8h40xxLfVO+Qu4PxuSaTuK7tY3ZvYOgu/7GnffVDs8ZV+uB/5Kfqp0I3H37e6+M3z/DNDazHpQhPszlO23mff9aWatCZLFOHf/S5pJ4v195rthJmLjTSuCRpej+U+D1sn1pvkMhzZ6PxG+P5lDG72XkZ9G7ygxnkHQMDew3vCuQNvwfQ/gNfLUYBcxzt4p7z8ATPf/NIS9EcbbNXzfLak4w+lOIGhEtCT2Z7iOAWRupH0PhzYqziz0vowYZ3+C9r3z6w3vCHRKef8CcGWCcR5Z+10TFLQrw30b6fdSqDjD8Z0J2jk6JrE/w/3ye+CnWaaJ9feZt53dhI2/iqCV/3VgVDjsToIjdYB2wJ/CH/1M4JiUeUeF8y0B3p1gjP8G1gFzw9ffw+HnA6+EP/JXgFsS3pffBxaE8VQCJ6TM+/FwHy8Fbk4yzvDzHcAP6s1XsP1JcPS4FjhAUM97C3ArcGs43ggeBPZ6GMvghPZlQ3E+CGxJ+W1WhcOPCffjvPA3MSrhOP8n5bc5nZQEl+73klSc4TQ3EXS4SZ2vYPuToFrRgZdTvter8vn71K1BREQkkmJpwxARkSKnhCEiIpEoYYiISCRKGCIiEokShoiIRKKEISIikShhiIhIJEoYIjkIn0cwNHx/l5n9IumYRPJFNx8Uyc23gTvDG82dAbwv4XhE8kZXeovkyMwmA4cBQzx4LoFIs6QqKZEcmNmpBE8+269kIc2dEoZIE4VPLhtH8FSznWaWtyfqiRQDJQyRJgiftPYX4DZ3XwR8l6A9Q6TZUhuGiIhEojMMERGJRAlDREQiUcIQEZFIlDBERCQSJQwREYlECUNERCJRwhARkUj+Hz+GMgV/Yq0JAAAAAElFTkSuQmCC\n",
+      "text/plain": [
+       "
"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    }
+   ],
+   "source": [
+    "%matplotlib inline\n",
+    "\n",
+    "\n",
+    "# Importing various packages\n",
+    "from random import random, seed\n",
+    "import numpy as np\n",
+    "import matplotlib.pyplot as plt\n",
+    "from mpl_toolkits.mplot3d import Axes3D\n",
+    "from matplotlib import cm\n",
+    "from matplotlib.ticker import LinearLocator, FormatStrFormatter\n",
+    "import sys\n",
+    "\n",
+    "x = 2*np.random.rand(100,1)\n",
+    "y = 4+3*x+np.random.randn(100,1)\n",
+    "\n",
+    "xb = np.c_[np.ones((100,1)), x]\n",
+    "beta_linreg = np.linalg.inv(xb.T.dot(xb)).dot(xb.T).dot(y)\n",
+    "print(beta_linreg)\n",
+    "beta = np.random.randn(2,1)\n",
+    "\n",
+    "eta = 0.1\n",
+    "Niterations = 1000\n",
+    "m = 100\n",
+    "\n",
+    "for iter in range(Niterations):\n",
+    "    gradients = 2.0/m*xb.T.dot(xb.dot(beta)-y)\n",
+    "    beta -= eta*gradients\n",
+    "\n",
+    "print(beta)\n",
+    "xnew = np.array([[0],[2]])\n",
+    "xbnew = np.c_[np.ones((2,1)), xnew]\n",
+    "ypredict = xbnew.dot(beta)\n",
+    "ypredict2 = xbnew.dot(beta_linreg)\n",
+    "plt.plot(xnew, ypredict, \"r-\")\n",
+    "plt.plot(xnew, ypredict2, \"b-\")\n",
+    "plt.plot(x, y ,'ro')\n",
+    "plt.axis([0,2.0,0, 15.0])\n",
+    "plt.xlabel(r'$x$')\n",
+    "plt.ylabel(r'$y$')\n",
+    "plt.title(r'Gradient descent example')\n",
+    "plt.show()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## And a corresponding example using **scikit-learn**"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "[[4.20374363]\n",
+      " [2.83470272]]\n",
+      "[4.21610176] [2.82633808]\n"
+     ]
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "/usr/local/lib/python3.7/site-packages/sklearn/linear_model/stochastic_gradient.py:117: DeprecationWarning: n_iter parameter is deprecated in 0.19 and will be removed in 0.21. Use max_iter and tol instead.\n",
+      "  DeprecationWarning)\n"
+     ]
+    }
+   ],
+   "source": [
+    "# Importing various packages\n",
+    "from random import random, seed\n",
+    "import numpy as np\n",
+    "import matplotlib.pyplot as plt\n",
+    "from sklearn.linear_model import SGDRegressor\n",
+    "\n",
+    "x = 2*np.random.rand(100,1)\n",
+    "y = 4+3*x+np.random.randn(100,1)\n",
+    "\n",
+    "xb = np.c_[np.ones((100,1)), x]\n",
+    "beta_linreg = np.linalg.inv(xb.T.dot(xb)).dot(xb.T).dot(y)\n",
+    "print(beta_linreg)\n",
+    "sgdreg = SGDRegressor(n_iter = 50, penalty=None, eta0=0.1)\n",
+    "sgdreg.fit(x,y.ravel())\n",
+    "print(sgdreg.intercept_, sgdreg.coef_)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "\n",
+    "## Convex functions\n",
+    "\n",
+    "Ideally we want our cost/loss function to be convex(concave).\n",
+    "\n",
+    "First we give the definition of a convex set: A set $C$ in\n",
+    "$\\mathbb{R}^n$ is said to be convex if, for all $x$ and $y$ in $C$ and\n",
+    "all $t \\in (0,1)$ , the point $(1 − t)x + ty$ also belongs to\n",
+    "C. Geometrically this means that every point on the line segment\n",
+    "connecting $x$ and $y$ is in $C$ as discussed below.\n",
+    "\n",
+    "The convex subsets of $\\mathbb{R}$ are the intervals of\n",
+    "$\\mathbb{R}$. Examples of convex sets of $\\mathbb{R}^2$ are the\n",
+    "regular polygons (triangles, rectangles, pentagons, etc...).\n",
+    "\n",
+    "## Convex function\n",
+    "\n",
+    "**Convex function**: Let $X \\subset \\mathbb{R}^n$ be a convex set. Assume that the function $f: X \\rightarrow \\mathbb{R}$ is continuous, then $f$ is said to be convex if $$f(tx_1 + (1-t)x_2) \\leq tf(x_1) + (1-t)f(x_2) $$ for all $x_1, x_2 \\in X$ and for all $t \\in [0,1]$. If $\\leq$ is replaced with a strict inequaltiy in the definition, we demand $x_1 \\neq x_2$ and $t\\in(0,1)$ then $f$ is said to be strictly convex. For a single variable function, convexity means that if you draw a straight line connecting $f(x_1)$ and $f(x_2)$, the value of the function on the interval $[x_1,x_2]$ is always below the line as illustrated below.\n",
+    "\n",
+    "## Conditions on convex functions\n",
+    "\n",
+    "In the following we state first and second-order conditions which\n",
+    "ensures convexity of a function $f$. We write $D_f$ to denote the\n",
+    "domain of $f$, i.e the subset of $R^n$ where $f$ is defined. For more\n",
+    "details and proofs we refer to: [S. Boyd and L. Vandenberghe. Convex Optimization. Cambridge University Press](http://stanford.edu/boyd/cvxbook/, 2004).\n",
+    "\n",
+    "**First order condition.**\n",
+    "\n",
+    "Suppose $f$ is differentiable (i.e $\\nabla f(x)$ is well defined for\n",
+    "all $x$ in the domain of $f$). Then $f$ is convex if and only if $D_f$\n",
+    "is a convex set and $$f(y) \\geq f(x) + \\nabla f(x)^T (y-x) $$ holds\n",
+    "for all $x,y \\in D_f$. This condition means that for a convex function\n",
+    "the first order Taylor expansion (right hand side above) at any point\n",
+    "a global under estimator of the function. To convince yourself you can\n",
+    "make a drawing of f(x) = x^2+1 and draw the tangent line to $f(x)$ and\n",
+    "note that it is always below the graph.\n",
+    "\n",
+    "\n",
+    "\n",
+    "**Second order condition.**\n",
+    "\n",
+    "Assume that $f$ is twice\n",
+    "differentiable, i.e the Hessian matrix exists at each point in\n",
+    "$D_f$. Then $f$ is convex if and only if $D_f$ is a convex set and its\n",
+    "Hessian is positive semi-definite for all $x\\in D_f$. For a\n",
+    "single-variable function this reduces to $f''(x) \\geq\n",
+    "0$. Geometrically this means that $f$ has nonnegative curvature\n",
+    "everywhere.\n",
+    "\n",
+    "\n",
+    "\n",
+    "This condition is particularly useful since it gives us an procedure for determining if the function under consideration is convex, apart from using the definition.\n",
+    "\n",
+    "## More on convex functions\n",
+    "\n",
+    "The next result is of great importance to us and the reason why we are\n",
+    "going on about convex functions. In machine learning we frequently\n",
+    "have to minimize a loss/cost function in order to find the best\n",
+    "parameters for the model we are considering. \n",
+    "\n",
+    "Ideally we want the\n",
+    "global minimum (for high-dimensional models it is hard to know\n",
+    "if we have local or global minimum). However, if the cost/loss function\n",
+    "is convex the following result provides invaluable information:\n",
+    "\n",
+    "**Any minimum is global for convex functions.**\n",
+    "\n",
+    "Consider the problem of finding $x \\in \\mathbb{R}^n$ such that $f(x)$\n",
+    "is minimal, where $f$ is convex and differentiable. Then, any point\n",
+    "$x^*$ that satisfies $\\nabla f(x^*) = 0$ is a global minimum.\n",
+    "\n",
+    "\n",
+    "\n",
+    "This result means that if we know that the cost/loss function is convex and we are able to find a minimum, we are guaranteed that it is a global minimum.\n",
+    "\n",
+    "## Some simple problems\n",
+    "\n",
+    "1. Show that $f(x)=x^2$ is convex for $x \\in \\mathbb{R}$ using the definition of convexity. Hint: If you re-write the definition, $f$ is convex if the following holds for all $x,y \\in D_f$ and any $\\lambda \\in [0,1] $ $\\lambda f(x) + (1-\\lambda)f(y) - f(\\lambda x + (1-\\lambda) y ) \\geq 0. $\n",
+    "\n",
+    "2. Using the second order condition show that the following functions are convex on the specified domain.\n",
+    "\n",
+    " * $f(x) = e^x$ is convex for $x \\in \\mathbb{R}$.\n",
+    "\n",
+    " * $g(x) = -\\ln(x)$ is convex for $x \\in (0,\\infty)$.\n",
+    "\n",
+    "\n",
+    "3. Let $f(x) = x^2$ and $g(x) = e^x$. Show that $f(g(x))$ and $g(f(x))$ is convex for $x \\in \\mathbb{R}$. Also show that if $f(x)$ is any convex function than $h(x) = e^{f(x)}$ is convex.\n",
+    "\n",
+    "4. A norm is any function that satisfy the following properties\n",
+    "\n",
+    " * $f(\\alpha x) = |\\alpha| f(x)$ for all $\\alpha \\in \\mathbb{R}$.\n",
+    "\n",
+    " * $f(x+y) \\leq f(x) + f(y)$\n",
+    "\n",
+    " * $f(x) \\leq 0$ for all $x \\in \\mathbb{R}^n$ with equality if and only if $x = 0$\n",
+    "\n",
+    "\n",
+    "Using the definition of convexity, try to show that a function satisfying the properties above is convex (the third condition is not needed to show this).\n",
+    "\n",
+    "\n",
+    "## Revisiting our first homework\n",
+    "\n",
+    "We will use linear regression as a case study for the gradient descent\n",
+    "methods. Linear regression is a great test case for the gradient\n",
+    "descent methods discussed in the lectures since it has several\n",
+    "desirable properties such as:\n",
+    "\n",
+    "1. An analytical solution (recall homework set 1).\n",
+    "\n",
+    "2. The gradient can be computed analytically.\n",
+    "\n",
+    "3. The cost function is convex which guarantees that gradient descent converges for small enough learning rates\n",
+    "\n",
+    "We revisit the example from homework set 1 where we had"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "$$\n",
+    "y_i = 5x_i^2 + 0.1\\xi_i, \\ i=1,\\cdots,100\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "with $x_i \\in [0,1] $ chosen randomly with a uniform distribution. Additionally $\\xi_i$ represents stochastic noise chosen according to a normal distribution $\\cal {N}(0,1)$. \n",
+    "The linear regression model is given by"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "$$\n",
+    "h_\\beta(x) = \\hat{y} = \\beta_0 + \\beta_1 x,\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "such that"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "$$\n",
+    "\\hat{y}_i = \\beta_0 + \\beta_1 x_i.\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "\n",
+    "## Gradient descent example\n",
+    "\n",
+    "Let $\\mathbf{y} = (y_1,\\cdots,y_n)^T$, $\\mathbf{\\hat{y}} = (\\hat{y}_1,\\cdots,\\hat{y}_n)^T$ and $\\beta = (\\beta_0, \\beta_1)^T$\n",
+    "\n",
+    "t is convenient to write $\\mathbf{\\hat{y}} = X\\beta$ where $X \\in \\mathbb{R}^{100 \\times 2} $ is the design matrix given by"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "\n",
+    "
\n",
+    "\n",
+    "$$\n",
+    "\\begin{equation}\n",
+    "X \\equiv \\begin{bmatrix}\n",
+    "1 & x_1  \\\\\n",
+    "\\vdots & \\vdots  \\\\\n",
+    "1 & x_{100} &  \\\\\n",
+    "\\end{bmatrix}.\n",
+    "\\label{_auto1} \\tag{1}\n",
+    "\\end{equation}\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "The loss function is given by"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "$$\n",
+    "C(\\beta) = ||X\\beta-\\mathbf{y}||^2 = ||X\\beta||^2 - 2 \\mathbf{y}^T X\\beta + ||\\mathbf{y}||^2 = \\sum_{i=1}^{100} (\\beta_0 + \\beta_1 x_i)^2 - 2 y_i (\\beta_0 + \\beta_1 x_i) + y_i^2\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "and we want to find $\\beta$ such that $C(\\beta)$ is minimized.\n",
+    "\n",
+    "## The derivative of the cost/loss function\n",
+    "\n",
+    "Computing $\\partial C(\\beta) / \\partial \\beta_0$ and $\\partial C(\\beta) / \\partial \\beta_1$ we can show  that the gradient can be written as"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "$$\n",
+    "\\nabla_\\beta C(\\beta) = (\\partial C(\\beta) / \\partial \\beta_0, \\partial C(\\beta) / \\partial \\beta_1)^T = 2\\begin{bmatrix} \\sum_{i=1}^{100} \\left(\\beta_0+\\beta_1x_i-y_i\\right) \\\\\n",
+    "\\sum_{i=1}^{100}\\left( x_i (\\beta_0+\\beta_1x_i)-y_ix_i\\right) \\\\\n",
+    "\\end{bmatrix} = 2X^T(X\\beta - \\mathbf{y}),\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "where $X$ is the design matrix defined above.\n",
+    "\n",
+    "## The Hessian matrix\n",
+    "The Hessian matrix of $C(\\beta)$ is given by"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "$$\n",
+    "\\hat{H} \\equiv \\begin{bmatrix}\n",
+    "\\frac{\\partial^2 C(\\beta)}{\\partial \\beta_0^2} & \\frac{\\partial^2 C(\\beta)}{\\partial \\beta_0 \\partial \\beta_1}  \\\\\n",
+    "\\frac{\\partial^2 C(\\beta)}{\\partial \\beta_0 \\partial \\beta_1} & \\frac{\\partial^2 C(\\beta)}{\\partial \\beta_1^2} &  \\\\\n",
+    "\\end{bmatrix} = 2X^T X.\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "This result implies that $C(\\beta)$ is a convex function since the matrix $X^T X$ always is positive semi-definite.\n",
+    "\n",
+    "## Simple program\n",
+    "\n",
+    "We can now write a program that minimizes $C(\\beta)$ using the gradient descent method with a constant learning rate $\\gamma$ according to"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "$$\n",
+    "\\beta_{k+1} = \\beta_k - \\gamma \\nabla_\\beta C(\\beta_k), \\ k=0,1,\\cdots\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "We can use the expression we computed for the gradient and let use a\n",
+    "$\\beta_0$ be chosen randomly and let $\\gamma = 0.001$. Stop iterating\n",
+    "when $||\\nabla_\\beta C(\\beta_k) || < \\epsilon = 10^{-8}$. \n",
+    "\n",
+    "And finally we can compare our solution for $\\beta$ with the analytic result given by \n",
+    "$\\beta= (X^TX)^{-1} X^T \\mathbf{y}$."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "[-0.72135172  4.78496149]\n"
+     ]
+    }
+   ],
+   "source": [
+    "import numpy as np\n",
+    "\n",
+    "\"\"\"\n",
+    "The following setup is just a suggestion, feel free to write it the way you like.\n",
+    "\"\"\"\n",
+    "\n",
+    "#Setup problem described in the exercise\n",
+    "N  = 100 #Nr of datapoints\n",
+    "M  = 2 #Nr of features\n",
+    "x  = np.random.rand(N) #Uniformly generated x-values in [0,1]\n",
+    "y  = 5*x**2 + 0.1*np.random.randn(N)\n",
+    "X  = np.c_[np.ones(N),x] #Construct design matrix\n",
+    "\n",
+    "#Compute beta according to normal equations to compare with GD solution\n",
+    "Xt_X_inv = np.linalg.inv(np.dot(X.T,X))\n",
+    "Xt_y     = np.dot(X.transpose(),y)\n",
+    "beta_NE = np.dot(Xt_X_inv,Xt_y)\n",
+    "print(beta_NE)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "\n",
+    "## Gradient descent and Ridge\n",
+    "\n",
+    "We have also discussed Ridge regression where the loss function contains a regularized given by the $L_2$ norm of $\\beta$,"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "$$\n",
+    "C_{\\text{ridge}}(\\beta) = ||X\\beta -\\mathbf{y}||^2 + \\lambda ||\\beta||^2, \\ \\lambda \\geq 0.\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "In order to minimize $C_{\\text{ridge}}(\\beta)$ using GD we only have adjust the gradient as follows"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "$$\n",
+    "\\nabla_\\beta C_{\\text{ridge}}(\\beta)  = 2\\begin{bmatrix} \\sum_{i=1}^{100} \\left(\\beta_0+\\beta_1x_i-y_i\\right) \\\\\n",
+    "\\sum_{i=1}^{100}\\left( x_i (\\beta_0+\\beta_1x_i)-y_ix_i\\right) \\\\\n",
+    "\\end{bmatrix} + 2\\lambda\\begin{bmatrix} \\beta_0 \\\\ \\beta_1\\end{bmatrix} = 2 (X^T(X\\beta - \\mathbf{y})+\\lambda \\beta).\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "We can now extend our program to minimize $C_{\\text{ridge}}(\\beta)$ using gradient descent and compare with the analytical solution given by"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "$$\n",
+    "\\beta_{\\text{ridge}} = \\left(X^T X + \\lambda I_{2 \\times 2} \\right)^{-1} X^T \\mathbf{y},\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "for $\\lambda = {0,1,10,50,100}$ ($\\lambda = 0$ corresponds to ordinary least squares). \n",
+    "We can then compute $||\\beta_{\\text{ridge}}||$ for each $\\lambda$."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 4,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "[-0.76259836  4.89939464]\n",
+      "4.958389265060428\n"
+     ]
+    }
+   ],
+   "source": [
+    "import numpy as np\n",
+    "\n",
+    "\"\"\"\n",
+    "The following setup is just a suggestion, feel free to write it the way you like.\n",
+    "\"\"\"\n",
+    "\n",
+    "#Setup problem described in the exercise\n",
+    "N  = 100 #Nr of datapoints\n",
+    "M  = 2   #Nr of features\n",
+    "x  = np.random.rand(N)\n",
+    "y  = 5*x**2 + 0.1*np.random.randn(N)\n",
+    "\n",
+    "\n",
+    "#Compute analytic beta for Ridge regression \n",
+    "X    = np.c_[np.ones(N),x]\n",
+    "XT_X = np.dot(X.T,X)\n",
+    "\n",
+    "l  = 0.1 #Ridge parameter lambda\n",
+    "Id = np.eye(XT_X.shape[0])\n",
+    "\n",
+    "Z = np.linalg.inv(XT_X+l*Id)\n",
+    "beta_ridge = np.dot(Z,np.dot(X.T,y))\n",
+    "\n",
+    "print(beta_ridge)\n",
+    "print(np.linalg.norm(beta_ridge)) #||beta||"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Stochastic Gradient Descent\n",
+    "\n",
+    "Stochastic gradient descent (SGD) and variants thereof address some of\n",
+    "the shortcomings of the Gradient descent method discussed above.\n",
+    "\n",
+    "The underlying idea of SGD comes from the observation that the cost\n",
+    "function, which we want to minimize, can almost always be written as a\n",
+    "sum over $n$ datapoints $\\{\\mathbf{x}_i\\}_{i=1}^n$,"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "$$\n",
+    "C(\\mathbf{\\beta}) = \\sum_{i=1}^n c_i(\\mathbf{x}_i,\n",
+    "\\mathbf{\\beta}).\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Computation of gradients\n",
+    "\n",
+    "This in turn means that the gradient can be\n",
+    "computed as a sum over $i$-gradients"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "$$\n",
+    "\\nabla_\\beta C(\\mathbf{\\beta}) = \\sum_i^n \\nabla_\\beta c_i(\\mathbf{x}_i,\n",
+    "\\mathbf{\\beta}).\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Stochasticity/randomness is introduced by only taking the\n",
+    "gradient on a subset of the data called minibatches.  If there are $n$\n",
+    "datapoints and the size of each minibatch is $M$, there will be $n/M$\n",
+    "minibatches. We denote these minibatches by $B_k$ where\n",
+    "$k=1,\\cdots,n/M$.\n",
+    "\n",
+    "## SGD example\n",
+    "As an example, suppose we have $10$ datapoints $( \\mathbf{x}_1,\n",
+    "\\cdots, \\mathbf{x}_{10} )$ and we choose to have $M=5$ minibathces,\n",
+    "then each minibatch contains two datapoints. In particular we have\n",
+    "$B_1 = (\\mathbf{x}_1,\\mathbf{x}_2), \\cdots, B_5 =\n",
+    "(\\mathbf{x}_9,\\mathbf{x}_{10})$. Note that if you choose $M=1$ you\n",
+    "have only a single batch with all datapoints and on the other extreme,\n",
+    "you may choose $M=n$ resulting in a minibatch for each datapoint, i.e\n",
+    "$B_k = \\mathbf{x}_k$.\n",
+    "\n",
+    "The idea is now to approximate the gradient by replacing the sum over\n",
+    "all datapoints with a sum over the datapoints in one the minibatches\n",
+    "picked at random in each gradient descent step"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "$$\n",
+    "\\nabla_\\beta\n",
+    "C(\\mathbf{\\beta}) = \\sum_{i=1}^n \\nabla_\\beta c_i(\\mathbf{x}_i,\n",
+    "\\mathbf{\\beta}) \\rightarrow \\sum_{i \\in B_k}^n \\nabla_\\beta\n",
+    "c_i(\\mathbf{x}_i, \\mathbf{\\beta}).\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## The gradient step\n",
+    "\n",
+    "Thus a gradient descent step now looks like"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "$$\n",
+    "\\beta_{j+1} = \\beta_j - \\gamma_j \\sum_{i \\in B_k}^n \\nabla_\\beta c_i(\\mathbf{x}_i,\n",
+    "\\mathbf{\\beta})\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "where $k$ is picked at random with equal\n",
+    "probability from $[1,n/M]$. An iteration over the number of\n",
+    "minibathces (n/M) is commonly referred to as an epoch. Thus it is\n",
+    "typical to choose a number of epochs and for each epoch iterate over\n",
+    "the number of minibatches, as exemplified in the code below.\n",
+    "\n",
+    "## Simple example code"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 5,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import numpy as np \n",
+    "\n",
+    "n = 100 #100 datapoints \n",
+    "M = 5   #size of each minibatch\n",
+    "m = int(n/M) #number of minibatches\n",
+    "n_epochs = 10 #number of epochs\n",
+    "\n",
+    "j = 0\n",
+    "for epoch in range(1,n_epochs+1):\n",
+    "    for i in range(m):\n",
+    "        k = np.random.randint(m) #Pick the k-th minibatch at random\n",
+    "        #Compute the gradient using the data in minibatch Bk\n",
+    "        #Compute new suggestion for \n",
+    "        j += 1"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Taking the gradient only on a subset of the data has two important\n",
+    "benefits. First, it introduces randomness which decreases the chance\n",
+    "that our opmization scheme gets stuck in a local minima. Second, if\n",
+    "the size of the minibatches are small relative to the number of\n",
+    "datapoints ($M < n$), the computation of the gradient is much\n",
+    "cheaper since we sum over the datapoints in the k-th minibatch and not\n",
+    "all $n$ datapoints.\n",
+    "\n",
+    "## When do we stop?\n",
+    "\n",
+    "A natural question is when do we stop the search for a new minimum?\n",
+    "One possibility is to compute the full gradient after a given number\n",
+    "of epochs and check if the norm of the gradient is smaller than some\n",
+    "threshold and stop if true. However, the condition that the gradient\n",
+    "is zero is valid also for local minima, so this would only tell us\n",
+    "that we are close to a local/global minimum. However, we could also\n",
+    "evaluate the cost function at this point, store the result and\n",
+    "continue the search. If the test kicks in at a later stage we can\n",
+    "compare the values of the cost function and keep the $\\beta$ that\n",
+    "gave the lowest value.\n",
+    "\n",
+    "## Slightly different approach\n",
+    "\n",
+    "Another approach is to let the step length $\\gamma_j$ depend on the\n",
+    "number of epochs in such a way that it becomes very small after a\n",
+    "reasonable time such that we do not move at all.\n",
+    "\n",
+    "As an example, let $e = 0,1,2,3,\\cdots$ denote the current epoch and let $t_0, t_1 > 0$ be two fixed numbers. Furthermore, let $t = e \\cdot m + i$ where $m$ is the number of minibatches and $i=0,\\cdots,m-1$. Then the function $$\\gamma_j(t; t_0, t_1) = \\frac{t_0}{t+t_1} $$ goes to zero as the number of epochs gets large. I.e. we start with a step length $\\gamma_j (0; t_0, t_1) = t_0/t_1$ which decays in *time* $t$.\n",
+    "\n",
+    "In this way we can fix the number of epochs, compute $\\beta$ and\n",
+    "evaluate the cost function at the end. Repeating the computation will\n",
+    "give a different result since the scheme is random by design. Then we\n",
+    "pick the final $\\beta$ that gives the lowest value of the cost\n",
+    "function."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 6,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import numpy as np \n",
+    "\n",
+    "def step_length(t,t0,t1):\n",
+    "    return t0/(t+t1)\n",
+    "\n",
+    "n = 100 #100 datapoints \n",
+    "M = 5   #size of each minibatch\n",
+    "m = int(n/M) #number of minibatches\n",
+    "n_epochs = 500 #number of epochs\n",
+    "t0 = 1.0\n",
+    "t1 = 10\n",
+    "\n",
+    "gamma_j = t0/t1\n",
+    "j = 0\n",
+    "for epoch in range(1,n_epochs+1):\n",
+    "    for i in range(m):\n",
+    "        k = np.random.randint(m) #Pick the k-th minibatch at random\n",
+    "        #Compute the gradient using the data in minibatch Bk\n",
+    "        #Compute new suggestion for beta\n",
+    "        t = epoch*m+i\n",
+    "        gamma_j = step_length(t,t0,t1)\n",
+    "        j += 1\n",
+    "\n",
+    "print(\"gamma_j after %d epochs: %g\" % (n_epochs,gamma_j))"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Conjugate gradient (CG) method\n",
+    "The success of the CG method  for finding solutions of non-linear problems is based\n",
+    "on the theory of conjugate gradients for linear systems of equations. It belongs\n",
+    "to the class of iterative methods for solving problems from linear algebra of the type"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "$$\n",
+    "\\hat{A}\\hat{x} = \\hat{b}.\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "In the iterative process we end up with a problem like"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "$$\n",
+    "\\hat{r}= \\hat{b}-\\hat{A}\\hat{x},\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "where $\\hat{r}$ is the so-called residual or error in the iterative process.\n",
+    "\n",
+    "When we have found the exact solution, $\\hat{r}=0$.\n",
+    "\n",
+    "\n",
+    "\n",
+    "\n",
+    "## Conjugate gradient method\n",
+    "\n",
+    "The residual is zero when we reach the minimum of the quadratic equation"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "$$\n",
+    "P(\\hat{x})=\\frac{1}{2}\\hat{x}^T\\hat{A}\\hat{x} - \\hat{x}^T\\hat{b},\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "with the constraint that the matrix $\\hat{A}$ is positive definite and symmetric.\n",
+    "If we search for a minimum of the quantum mechanical  variance, then the matrix \n",
+    "$\\hat{A}$, which is called the Hessian, is given by the second-derivative of the function we want to minimize.  This quantity is always positive definite.  In our case this corresponds normally to the second derivative of the energy.\n",
+    "\n",
+    "\n",
+    "\n",
+    "\n",
+    "\n",
+    "\n",
+    "\n",
+    "## Conjugate gradient method, Newton's method first\n",
+    "We seek the minimum of the energy or the variance as function of various variational parameters. \n",
+    "In our case we have thus a function $f$ whose minimum we are seeking.\n",
+    "In Newton's method we set $\\nabla f = 0$ and we can thus compute the next iteration point"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "$$\n",
+    "\\hat{x}-\\hat{x}_i=\\hat{A}^{-1}\\nabla f(\\hat{x}_i).\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Subtracting this equation from that of $\\hat{x}_{i+1}$ we have"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "$$\n",
+    "\\hat{x}_{i+1}-\\hat{x}_i=\\hat{A}^{-1}(\\nabla f(\\hat{x}_{i+1})-\\nabla f(\\hat{x}_i)).\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Conjugate gradient method\n",
+    "In the CG method we define so-called conjugate directions and two vectors \n",
+    "$\\hat{s}$ and $\\hat{t}$\n",
+    "are said to be\n",
+    "conjugate if"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "$$\n",
+    "\\hat{s}^T\\hat{A}\\hat{t}= 0.\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "The philosophy of the CG method is to perform searches in various conjugate directions\n",
+    "of our vectors $\\hat{x}_i$ obeying the above criterion, namely"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "$$\n",
+    "\\hat{x}_i^T\\hat{A}\\hat{x}_j= 0.\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Two vectors are conjugate if they are orthogonal with respect to \n",
+    "this inner product. Being conjugate is a symmetric relation: if $\\hat{s}$ is conjugate to $\\hat{t}$, then $\\hat{t}$ is conjugate to $\\hat{s}$.\n",
+    "\n",
+    "\n",
+    "\n",
+    "## Conjugate gradient method\n",
+    "An example is given by the eigenvectors of the matrix"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "$$\n",
+    "\\hat{v}_i^T\\hat{A}\\hat{v}_j= \\lambda\\hat{v}_i^T\\hat{v}_j,\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "which is zero unless $i=j$.\n",
+    "\n",
+    "\n",
+    "\n",
+    "\n",
+    "## Conjugate gradient method\n",
+    "Assume now that we have a symmetric positive-definite matrix $\\hat{A}$ of size\n",
+    "$n\\times n$. At each iteration $i+1$ we obtain the conjugate direction of a vector"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "$$\n",
+    "\\hat{x}_{i+1}=\\hat{x}_{i}+\\alpha_i\\hat{p}_{i}.\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "We assume that $\\hat{p}_{i}$ is a sequence of $n$ mutually conjugate directions. \n",
+    "Then the $\\hat{p}_{i}$  form a basis of $R^n$ and we can expand the solution \n",
+    "$  \\hat{A}\\hat{x} = \\hat{b}$ in this basis, namely"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "$$\n",
+    "\\hat{x}  = \\sum^{n}_{i=1} \\alpha_i \\hat{p}_i.\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Conjugate gradient method\n",
+    "The coefficients are given by"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "$$\n",
+    "\\mathbf{A}\\mathbf{x} = \\sum^{n}_{i=1} \\alpha_i \\mathbf{A} \\mathbf{p}_i = \\mathbf{b}.\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Multiplying with $\\hat{p}_k^T$  from the left gives"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "$$\n",
+    "\\hat{p}_k^T \\hat{A}\\hat{x} = \\sum^{n}_{i=1} \\alpha_i\\hat{p}_k^T \\hat{A}\\hat{p}_i= \\hat{p}_k^T \\hat{b},\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "and we can define the coefficients $\\alpha_k$ as"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "$$\n",
+    "\\alpha_k = \\frac{\\hat{p}_k^T \\hat{b}}{\\hat{p}_k^T \\hat{A} \\hat{p}_k}\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Conjugate gradient method and iterations\n",
+    "\n",
+    "If we choose the conjugate vectors $\\hat{p}_k$ carefully, \n",
+    "then we may not need all of them to obtain a good approximation to the solution \n",
+    "$\\hat{x}$. \n",
+    "We want to regard the conjugate gradient method as an iterative method. \n",
+    "This will us to solve systems where $n$ is so large that the direct \n",
+    "method would take too much time.\n",
+    "\n",
+    "We denote the initial guess for $\\hat{x}$ as $\\hat{x}_0$. \n",
+    "We can assume without loss of generality that"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "$$\n",
+    "\\hat{x}_0=0,\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "or consider the system"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "$$\n",
+    "\\hat{A}\\hat{z} = \\hat{b}-\\hat{A}\\hat{x}_0,\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "instead.\n",
+    "\n",
+    "\n",
+    "\n",
+    "\n",
+    "## Conjugate gradient method\n",
+    "One can show that the solution $\\hat{x}$ is also the unique minimizer of the quadratic form"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "$$\n",
+    "f(\\hat{x}) = \\frac{1}{2}\\hat{x}^T\\hat{A}\\hat{x} - \\hat{x}^T \\hat{x} , \\quad \\hat{x}\\in\\mathbf{R}^n.\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "This suggests taking the first basis vector $\\hat{p}_1$ \n",
+    "to be the gradient of $f$ at $\\hat{x}=\\hat{x}_0$, \n",
+    "which equals"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "$$\n",
+    "\\hat{A}\\hat{x}_0-\\hat{b},\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "and \n",
+    "$\\hat{x}_0=0$ it is equal $-\\hat{b}$.\n",
+    "The other vectors in the basis will be conjugate to the gradient, \n",
+    "hence the name conjugate gradient method.\n",
+    "\n",
+    "\n",
+    "\n",
+    "\n",
+    "## Conjugate gradient method\n",
+    "Let  $\\hat{r}_k$ be the residual at the $k$-th step:"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "$$\n",
+    "\\hat{r}_k=\\hat{b}-\\hat{A}\\hat{x}_k.\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Note that $\\hat{r}_k$ is the negative gradient of $f$ at \n",
+    "$\\hat{x}=\\hat{x}_k$, \n",
+    "so the gradient descent method would be to move in the direction $\\hat{r}_k$. \n",
+    "Here, we insist that the directions $\\hat{p}_k$ are conjugate to each other, \n",
+    "so we take the direction closest to the gradient $\\hat{r}_k$  \n",
+    "under the conjugacy constraint. \n",
+    "This gives the following expression"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "$$\n",
+    "\\hat{p}_{k+1}=\\hat{r}_k-\\frac{\\hat{p}_k^T \\hat{A}\\hat{r}_k}{\\hat{p}_k^T\\hat{A}\\hat{p}_k} \\hat{p}_k.\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Conjugate gradient method\n",
+    "We can also  compute the residual iteratively as"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "$$\n",
+    "\\hat{r}_{k+1}=\\hat{b}-\\hat{A}\\hat{x}_{k+1},\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "which equals"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "$$\n",
+    "\\hat{b}-\\hat{A}(\\hat{x}_k+\\alpha_k\\hat{p}_k),\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "or"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "$$\n",
+    "(\\hat{b}-\\hat{A}\\hat{x}_k)-\\alpha_k\\hat{A}\\hat{p}_k,\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "which gives"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "$$\n",
+    "\\hat{r}_{k+1}=\\hat{r}_k-\\hat{A}\\hat{p}_{k},\n",
+    "$$"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.7.0"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}
diff --git a/doc/src/.DS_Store b/doc/src/.DS_Store
new file mode 100644
index 0000000000000000000000000000000000000000..b3d346bdc606d52e74599f3ec01668cf44204156
GIT binary patch
literal 6148
zcmeHK!EVzq82;TAmJTL*K-!MEAaUrS84MvIq$(>CyP%0oZ~(M9ZM6l3cxqCjQHpZm
zA>a*o6iys@5FP-&-}X?O4sl}>@?Y8CZ~Oo3=5uVvL?pt=Fd%9Zk%z?CTtjw_aXY6a
zYuK6=Q0P8#N=Q<+o2tQ5w5xCmI0Y`70=#xT3bATWX+(4DH%?WIHr^N>Al?3{4Ia@t
z^$|P3TIYC08ST>;HQ@8gpJRThF|X5O#Fw;1PZ2F?h@K=BBo}#NqsD8|6U0w2ilkKg
zMUCGa-K#}B&O
zvyMLvy20I!zkUDCY}WENZUql_58s@=o4%iY_=LxT32d+0Zdg2n&v4EW=WR63Qk5N{
zm6;yxK{`dRA>sgX*QQDQK+OCEuj%PI(vzB>P4c4P4Y|aLcp{PbwcvbEAE)uH#zE29
z#VKH4D(~9G{RKC&Q@|;3NfqGz!AD~3S*#4|tpl090sxz+)&`$H8JOc+>{+Y~q6H=l
z6=(ayVS?ePmBC&B?Wl&R)+2dF{d=zgYX+xXO4zOpj
TGKd
+  Do premises $A, B, \ldots \to$ hypothesis, $H$? 
+
+- Deductive inference: Premises allow definite determination of truth/falsity of H (syllogisms, symbolic logic, Boolean algebra) 
+  $B(H|A,B,...) = 0$ or $1$
+
+- Inductive inference: Premises bear on truth/falsity of H, but don’t allow its definite determination (weak syllogisms, analogies)
+  $A, B, C, D$ share properties $x, y, z$; $E$ has properties $x, y$
+  $\to$ $E$ probably has property $z$.
+!eblock
+
+!split
+===== Statistical Inference  =====
+!bblock
+* Quantify the strength of inductive inferences from facts, in the form of data ($D$), and other premises, e.g. models, to hypotheses about the phenomena producing the data.
+* Quantify via probabilities, or averages calculated using probabilities. Frequentists ($\mathcal{F}$) and Bayesians ($\mathcal{B}$) use probabilities very differently for this.
+* To the pioneers such as Bernoulli, Bayes and Laplace, a probability represented a *degree-of-belief* or plausability: how much they thought that something as true based on the evidence at hand. This is the Bayesian approach.
+* To the 19th century scholars, this seemed too vague and subjective. They redefined probability as the *long run relative frequency* with which an event occurred, given (infinitely) many repeated (experimental) trials.
+!eblock
+
+!split
+===== Some history =====
+Adapted from D.S. Sivia[^Sivia]:
+
+[^Sivia]: Sivia, Devinderjit, and John Skilling. Data Analysis : A Bayesian Tutorial, OUP Oxford, 2006
+
+!bquote
+Although the frequency definition appears to be more objective, its range of validity is also far more limited. For example, Laplace used (his) probability theory to estimate the mass of Saturn, given orbital data that were available to him from various astronomical observatories. In essence, he computed the posterior pdf for the mass M , given the data and all the relevant background information I (such as a knowledge of the laws of classical mechanics): prob(M|{data},I); this is shown schematically in the figure [Fig. 1.2]. 
+!equote
+
+!split
+FIGURE: [fig/sivia_fig_1_2.png, width=700 frac=0.9]
+
+!split
+!bquote
+To Laplace, the (shaded) area under the posterior pdf curve between $m_1$ and $m_2$ was a measure of how much he believed that the mass of Saturn lay in the range $m_1 \le M \le m_2$. As such, the position of the maximum of the posterior pdf represents a best estimate of the mass; its width, or spread, about this optimal value gives an indication of the uncertainty in the estimate. Laplace stated that: ‘ . . . it is a bet of 11,000 to 1 that the error of this result is not 1/100th of its value.’ He would have won the bet, as another 150 years’ accumulation of data has changed the estimate by only 0.63%! 
+!equote
+
+!split
+!bquote
+According to the frequency definition, however, we are not permitted to use probability theory to tackle this problem. This is because the mass of Saturn is a constant and not a random variable; therefore, it has no frequency distribution and so probability theory cannot be used.
+
+If the pdf [of Fig. 1.2] had to be interpreted in terms of the frequency definition, we would have to imagine a large ensemble of universes in which everything remains constant apart from the mass of Saturn. 
+!equote
+
+!split
+!bquote
+As this scenario appears quite far-fetched, we might be inclined to think of [Fig. 1.2] in terms of the distribution of the measurements of the mass in many repetitions of the experiment. Although we are at liberty to think about a problem in any way that facilitates its solution, or our understanding of it, having to seek a frequency interpretation for every data analysis problem seems rather perverse.
+For example, what do we mean by the ‘measurement of the mass’ when the data consist of orbital periods? Besides, why should we have to think about many repetitions of an experiment that never happened? What we really want to do is to make the best inference of the mass given the (few) data that we actually have; this is precisely the Bayes and Laplace view of probability.
+!equote
+
+!split
+!bquote
+Faced with the realization that the frequency definition of probability theory did not permit most real-life scientific problems to be addressed, a new subject was invented — statistics! To estimate the mass of Saturn, for example, one has to relate the mass to the data through some function called the statistic; since the data are subject to ‘random’ noise, the statistic becomes the random variable to which the rules of probability the- ory can be applied. But now the question arises: How should we choose the statistic? The frequentist approach does not yield a natural way of doing this and has, therefore, led to the development of several alternative schools of orthodox or conventional statis- tics. The masters, such as Fisher, Neyman and Pearson, provided a variety of different principles, which has merely resulted in a plethora of tests and procedures without any clear underlying rationale. This lack of unifying principles is, perhaps, at the heart of the shortcomings of the cook-book approach to statistics that students are often taught even today.
+!equote
+
+!split
+===== The Bayesian recipe =====
+!bblock
+Assess hypotheses by calculating their probabilities $p(H_i | \ldots)$ conditional on known and/or presumed information using the rules of probability theory.
+!eblock
+!bblock
+Probability Theory Axioms:
+- Product (AND) rule : $p(A, B | I) = p(A|I) p(B|A, I) = p(B|I)p(A|B,I)$
+  Should read $p(A,B|I)$ as the probability for propositions $A$ AND $B$ being true given that $I$ is true.
+- Sum (OR) rule: $p(A + B | I) = p(A | I) + p(B | I) - p(A, B | I)$
+  $p(A+B|I)$ is the probability that proposition $A$ OR $B$ is true given that $I$ is true.
+- Normalization: $p(A|I) + p(\bar{A}|I) = 1$
+  $\bar{A}$ denotes the proposition that $A$ is false.
+ 
+!eblock
+
+!split
+===== Bayes' theorem =====
+!bblock
+Bayes' theorem follows directly from the product rule
+!bt
+$$
+p(A|B,I) = \frac{p(B|A,I) p(A|I)}{p(B|I)}.
+$$
+!et
+The importance of this property to data analysis becomes apparent if we replace $A$ and $B$ by hypothesis($H$) and data($D$):
+!bt
+\begin{align}
+p(H|D,I) &= \frac{p(D|H,I) p(H|I)}{p(D|I)}.
+label{eq:bayes}
+\end{align}
+!et
+The power of Bayes’ theorem lies in the fact that it relates the quantity of interest, the probability that the hypothesis is true given the data, to the term we have a better chance of being able to assign, the probability that we would have observed the measured data if the hypothesis was true.
+!eblock
+
+!split
+!bblock
+The various terms in Bayes’ theorem have formal names. 
+* The quantity on the far right, $p(H|I)$, is called the *prior* probability; it represents our state of knowledge (or ignorance) about the truth of the hypothesis before we have analysed the current data. 
+* This is modified by the experimental measurements through $p(D|H,I)$, the *likelihood* function, 
+* The denominator $p(D|I)$ is called the *evidence*. It does not depend on the hypothesis and can be regarded as a normalization constant.
+* Together, these yield the *posterior* probability, $p(H|D, I )$, representing our state of knowledge about the truth of the hypothesis in the light of the data. 
+
+In a sense, Bayes’ theorem encapsulates the process of learning.
+!eblock
+
+!split
+===== The friends of Bayes' theorem =====
+!bblock
+- Normalization: $\sum_i p(H_i|\ldots) = 1$.
+- Marginalization: $\sum_i p(A,H_i|I) = \sum_i p(H_i|A,I) p(A|I) = p(A|I)$.
+- Marginalization (continuum limit): $\int dx p(A,H(x)|I) = p(A|I)$.
+
+In the above, $H_i$ is an exclusive and exhaustive list of hypotheses. For example,let’s imagine that there are five candidates in a presidential election; then $H_1$ could be the proposition that the first candidate will win, and so on. The probability that $A$ is true, for example that unemployment will be lower in a year’s time (given all relevant information $I$, but irrespective of whoever becomes president) is then given by $\sum_i p(A,H_i|I)$.
+
+In the continuum limit of propositions we must understand $p(\ldots)$ as a pdf (probability density function).
+
+Marginalization is a very powerful device in data analysis because it enables us to deal with nuisance parameters; that is, quantities which necessarily enter the analysis but are of no intrinsic interest. The unwanted background signal present in many experimental measurements are examples of nuisance parameters.
+!eblock
+
+!split
+===== Inference With Parametric Models  =====
+!bblock
+Inductive inference with parametric models is a very important tool in the natural sciences.
+* Consider $N$ different models $M_i$ ($i = 1, \ldots, N$), each with parameters $\boldsymbol{\alpha}_i$. Each of them implies a sampling distribution (conditional predictive distribution for possible data)
+!bt
+$$
+p(D|\boldsymbol{\alpha}_i, M_i)
+$$
+!et
+* The $\boldsymbol{\alpha}_i$ dependence when we fix attention on the actual, observed data ($D_\mathrm{obs}$) is the likelihood function:
+!bt
+$$
+\mathcal{L}_i (\boldsymbol{\alpha}_i) \equiv p(D_\mathrm{obs}|\boldsymbol{\alpha}_i, M_i)
+$$
+!et
+* We may be uncertain about $i$ (model uncertainty),
+* or uncertain about $\boldsymbol{\alpha}_i$ (parameter uncertainty).
+!eblock
+
+!split
+!bblock
+- Parameter Estimation: Premise = choice of model (pick specific $i$)
+  $\Rightarrow$ What can we say about $\boldsymbol{\alpha}_i$?
+- Model comparison: Premise = $\{M_i\}$ 
+  $\Rightarrow$ What can we say about $i$?
+- Model adequacy: Premise = $M_1$ 
+  $\Rightarrow$ Is $M_1$ adequate?
+- Hybrid Uncertainty: Models share some common params: $\boldsymbol{\alpha}_1 = \{ \boldsymbol{\varphi}, \boldsymbol{\eta}_i\}$
+  $\Rightarrow$ What can we say about $\boldsymbol{\varphi}$? (Systematic error is an example)
+!eblock
+
+!split
+===== Illustrative examples with python code  =====
+!bblock
+* Is this a fair coin? (analytical)
+* Flux from a star (single parameter, MCMC)
+* The lighthouse problem (two parameters, MCMC)
+* Linear fit with outliers (nuisance parameters)
+* ...
+!eblock
+
+!split
+===== Example: Is this a fair coin? =====
+Let us begin with the analysis of data from a simple coin-tossing experiment. 
+Given that we had observed 6 heads in 8 flips, would you think it was a fair coin? By fair, we mean that we would be prepared to lay an even 1 : 1 bet on the outcome of a flip being a head or a tail. If we decide that the coin was fair, the question which follows naturally is how sure are we that this was so; if it was not fair, how unfair do we think it was? Furthermore, if we were to continue collecting data for this particular coin, observing the outcomes of additional flips, how would we update our belief on the fairness of the coin?
+
+A sensible way of formulating this problem is to consider a large number of hypotheses about the range in which the bias-weighting of the coin might lie. If we denote the bias-weighting by $H$, then $H = 0$ and $H = 1$ can represent a coin which produces a tail or a head on every flip, respectively. There is a continuum of possibilities for the value of H between these limits, with $H = 0.5$ indicating a fair coin. Our state of knowledge about the fairness, or the degree of unfairness, of the coin is then completely summarized by specifying how much we believe these various propositions to be true. 
+
+Let us perform a computer simulation of a coin-tossing experiment. This provides the data that we will be analysing.
+
+@@@CODE src/coinflipping.py from-to: start import modules@end import modules
+@@@CODE src/coinflipping.py from-to: start simulation@end simulation
+
+In the light of this data, our inference about the fairness of this coin is summarized by the conditional pdf: $p(H|D,I)$. This is, of course, shorthand for the limiting case of a continuum of propositions for the value of $H$; that is to say, the probability that $H$ lies in an infinitesimally narrow range is given by $p(H|D,I) dH$. 
+
+To estimate this posterior pdf, we need to use Bayes’ theorem (ref{eq:bayes}). We will ignore the denominator $p(D|I)$ as it does not involve bias-weighting explicitly, and it will therefore not affect the shape of the desired pdf. At the end we can evaluate the missing constant subsequently from the normalization condition 
+!bt
+\begin{equation}
+\int_0^1 p(H|D,I) dH = 1.
+label{eq:coin_posterior_norm}
+\end{equation}
+!et
+
+The prior pdf, $p(H|I)$, represents what we know about the coin given only the information $I$ that we are dealing with a ‘strange coin’. We could keep a very open mind about the nature of the coin; a simple probability assignment which reflects this is a uniform, or flat, prior
+!bt
+\begin{equation}
+p(H|I) = \left\{ \begin{array}{ll}
+1 & 0 \le H \le 1, \\
+0 & \mathrm{otherwise}.
+\end{array} \right.
+label{eq:coin_prior_uniform}
+\end{equation}
+!et
+We will get back later to the choice of prior and its effect on the analysis.
+
+This prior state of knowledge, or ignorance, is modified by the data through the likelihood function $p(D|H,I)$. It is a measure of the chance that we would have obtained the data that we actually observed, if the value of the bias-weighting was given (as known). If, in the conditioning information $I$, we assume that the flips of the coin were independent events, so that the outcome of one did not influence that of another, then the probability of obtaining the data `R heads in N tosses' is given by the binomial distribution (we leave a formal definition of this to a statistics textbook)
+!bt
+\begin{equation}
+p(D|H,I) \propto H^R (1-H)^{N-R}.
+\end{equation}
+!et
+It seems reasonable because $H$ is the chance of obtaining a head on any flip, and there were $R$ of them, and $1-H$ is the corresponding probability for a tail, of which there were $N-R$. We note that this binomial distribution also contains a normalization factor, but we will ignore it since it does not depend explicitly on $H$, the quantity of interest. It will be absorbed by the normalization condition (ref{eq:coin_posterior_norm}).
+
+We perform the setup of this Bayesian framework on the computer.
+ 
+@@@CODE src/coinflipping.py from-to: start bayesian setup@end bayesian setup
+
+The next step is to confront this setup with the simulated data. To get a feel for the result, it is instructive to see how the posterior pdf evolves as we obtain more and more data pertaining to the coin. The results of such an analyses is shown in Fig. ref{fig:coinflipping}. 
+
+@@@CODE src/coinflipping.py from-to: start plotting@end plotting
+FIGURE:[fig/coinflipping_fig_1.png, width=500 frac=0.95] The evolution of the posterior pdf for the bias-weighting of a coin, as the number of data available increases. The figure on the top left-hand corner of each panel shows the number of data included in the analysis. label{fig:coinflipping}
+
+The panel in the top left-hand corner shows the posterior pdf for $H$ given no data, i.e., it is the same as the prior pdf of Eq. (ref{eq:coin_prior_uniform}). It indicates that we have no more reason to believe that the coin is fair than we have to think that it is double-headed, double-tailed, or of any other intermediate bias-weighting.
+
+The first flip is obviously tails. At this point we have no evidence that the coin has a side with heads, as indicated by the pdf going to zero as $H \to 1$. The second flip is obviously heads and we have now excluded both extreme options $H=0$ (double-tailed) and $H=1$ (double-headed). We can note that the posterior at this point has the simple form $p(H|D,I) = H(1-H)$ for $0 \le H \le 1$.
+
+The remainder of Fig. ref{fig:coinflipping} shows how the posterior pdf evolves as the number of data analysed becomes larger and larger. We see that the position of the maximum moves around, but that the amount by which it does so decreases with the increasing number of observations. The width of the posterior pdf also becomes narrower with more data, indicating that we are becoming increasingly confident in our estimate of the bias-weighting. For the coin in this example, the best estimate of $H$ eventually converges to 0.6, which, of course, was the value chosen to simulate the flips.
+
+!split
+===== A few words on different priors =====
+* uniform
+* Gaussian
+* Jeffrey's prior
+Repeat the coin flipping experiment with other priors.
+
+!split
+===== Bayesian parameter estimation (single parameter) =====
+!bblock
+We will now consider the very important task of model parameter estimation using statistical inference. 
+[CF: maybe stress that model parameters are not random variables, and the meaning of parameter estimation is therefore very different between frequentist and bayesian approaches.]
+
+Throughout this section we will consider a specific example that involves a model with a single parameter: ``Measured flux from a star''.
+!eblock
+
+!split
+=== Example: Measured flux from a star ===
+Adapted from the blog "Pythonic Perambulations": "http://jakevdp.github.io" by Jake VanderPlas.
+
+Imagine that we point our telescope to the sky, and observe the light coming from a single star. For the time being, we'll assume that the star's true flux is constant with time, i.e. that is it has a fixed value $F_\mathrm{true}$ (we'll also ignore effects like sky noise and other sources of systematic error). We'll assume that we perform a series of $N$ measurements with our telescope, where the ith measurement reports the observed photon flux $F_i$ and error $e_i$[^errors].
+The question is, given this set of measurements $D = \{F_i, e_i\}$, what is our best estimate of the true flux $F_\mathrm{true}$?
+
+[^errors]: We'll make the reasonable assumption that errors are Gaussian. In a Frequentist perspective, $e_i$ is the standard deviation of the results of a single measurement event in the limit of repetitions of *that event*. In the Bayesian perspective, $e_i$ is the standard deviation of the (Gaussian) probability distribution describing our knowledge of that particular measurement given its observed value.
+
+Because the measurements are number counts, a Poisson distribution is a good approximation to the measurement process:
+
+@@@CODE src/singlephotoncount.py from-to: start generate data@end generate data
+
+Now let's make a simple visualization of the ``observed'' data, see Fig. ref{fig:flux}.
+
+@@@CODE src/singlephotoncount.py from-to: start visualize data@end visualize data
+
+FIGURE:[fig/singlephotoncount_fig_1.png, width=400 frac=0.8] Single photon counts (flux measurements). label{fig:flux}
+
+These measurements each have a different error $e_i$ which is estimated from Poisson statistics using the standard square-root rule. In this toy example we already know the true flux $F_\mathrm{true}$, but the question is this: given our measurements and errors, what is our best estimate of the true flux?
+
+Let's take a look at the frequentist and Bayesian approaches to solving this.
+
+=== Simple Photon Counts: Frequentist Approach ===
+We'll start with the classical frequentist maximum likelihood approach. Given a single observation $D_i = (F_i, e_i)$, we can compute the probability distribution of the measurement given the true flux Ftrue given our assumption of Gaussian errors
+!bt
+\begin{equation}
+p(D_i | F_\mathrm{true}, I) = \frac{1}{\sqrt{2\pi e_i^2}} \exp \left( \frac{-(F_i-F_\mathrm{true})^2}{2e_i^2} \right).
+\end{equation}
+!et
+This should be read ``the probability of $D_i$ given $F_\mathrm{true}$
+equals ...''. You should recognize this as a normal distribution with mean $F_\mathrm{true}$ and standard deviation $e_i$.
+
+We construct the *likelihood function* by computing the product of the probabilities for each data point
+!bt
+\begin{equation}
+\mathcal{L}(D | F_\mathrm{true}, I) = \prod_{i=1}^N p(D_i | F_\mathrm{true}, I),
+\end{equation}
+!et
+here $D = \{D_i\}$ represents the entire set of measurements. Because the value of the likelihood can become very small, it is often more convenient to instead compute the log-likelihood. Combining the previous two equations and computing the log, we have
+!bt
+\begin{equation}
+\log\mathcal{L} = -\frac{1}{2} \sum_{i=1}^N \left[ \log(2\pi e_i^2) +  \frac{(F_i-F_\mathrm{true})^2}{e_i^2} \right].
+\end{equation}
+!et
+
+What we'd like to do is determine $F_\mathrm{true}$ such that the likelihood is maximized. For this simple problem, the maximization can be computed analytically (i.e. by setting $d\log\mathcal{L}/d F_\mathrm{true} = 0$). This results in the following observed estimate of $F_\mathrm{true}$
+!bt
+\begin{equation}
+F_\mathrm{est} = \sum_{i=1}^N w_i F_i; \quad w_i = 1/e_i^2.
+\end{equation}
+!et
+Notice that in the special case of all errors $e_i$ being equal, this reduces to
+!bt
+\begin{equation}
+F_\mathrm{est} = \frac{1}{N} \sum_{i=1} F_i.
+\end{equation}
+!et
+That is, in agreement with intuition, $F_\mathrm{est}$ is simply the mean of the observed data when errors are equal.
+
+We can go further and ask what the error of our estimate is. In the frequentist approach, this can be accomplished by fitting a Gaussian approximation to the likelihood curve at maximum; in this simple case this can also be solved analytically (the sum of Gaussians is also a Gaussian). It can be shown that the standard deviation of this Gaussian approximation is
+!bt
+\begin{equation}
+\sigma_\mathrm{est} = \sum_{i=1}^N w_i.
+\end{equation}
+!et
+These results are fairly simple calculations; let's evaluate them for our toy dataset:
+
+@@@CODE src/singlephotoncount.py from-to: start frequentist@end frequentist
+`F_true = 1000` 
+`F_est = 998 +/- 4 (based on 50 measurements)` 
+
+We find that for 50 measurements of the flux, our estimate has an error of about 0.4% and is consistent with the input value.
+
+
+=== Simple Photon Counts: Bayesian Approach ===
+The Bayesian approach, as you might expect, begins and ends with probabilities. Our hypothesis is that the star has a constant flux $F_\mathrm{true}$. It recognizes that what we fundamentally want to compute is our knowledge of the parameters in question given the data and other information (such as our knowledge of uncertainties for the observed values), i.e. in this case, $p(F_\mathrm{true} | D,I)$.
+Note that this formulation of the problem is fundamentally contrary to the frequentist philosophy, which says that probabilities have no meaning for model parameters like $F_\mathrm{true}$. Nevertheless, within the Bayesian philosophy this is perfectly acceptable.
+
+To compute this result, Bayesians next apply Bayes' Theorem (ref{eq:bayes}).
+If we set the prior $p(F_\mathrm{true}|I) \propto 1$ (a flat prior), we find
+$p(F_\mathrm{true}|D,I) \propto p(D | F_\mathrm{true},I) \equiv \mathcal{L}(D | F_\mathrm{true},I)$
+and the Bayesian probability is maximized at precisely the same value as the frequentist result! So despite the philosophical differences, we see that (for this simple problem at least) the Bayesian and frequentist point estimates are equivalent.
+
+=== A note about priors ===
+The prior allows inclusion of other information into the computation, which becomes very useful in cases where multiple measurement strategies are being combined to constrain a single model. The necessity to specify a prior, however, is one of the more controversial pieces of Bayesian analysis.
+A frequentist will point out that the prior is problematic when no true prior information is available. Though it might seem straightforward to use a noninformative prior like the flat prior mentioned above, there are some "surprisingly subtleties": "http://normaldeviate.wordpress.com/2013/07/13/lost-causes-in-statistics-ii-noninformative- priors/comment-page-1/" involved. It turns out that in many situations, a truly noninformative prior does not exist! Frequentists point out that the subjective choice of a prior which necessarily biases your result has no place in statistical data analysis.
+A Bayesian would counter that frequentism doesn't solve this problem, but simply skirts the question. Frequentism can often be viewed as simply a special case of the Bayesian approach for some (implicit) choice of the prior: a Bayesian would say that it's better to make this implicit choice explicit, even if the choice might include some subjectivity.
+
+=== Simple Photon Counts: Bayesian approach in practice ===
+Leaving these philosophical debates aside for the time being, let's address how Bayesian results are generally computed in practice. For a one parameter problem like the one considered here, it's as simple as computing the posterior probability $p(F_\mathrm{true} | D,I)$ as a function of $F_\mathrm{true}$: this is the distribution reflecting our knowledge of the parameter $F_\mathrm{true}$.
+But as the dimension of the model grows, this direct approach becomes increasingly intractable. For this reason, Bayesian calculations often depend on sampling methods such as Markov Chain Monte Carlo (MCMC). For this practical example, let us apply an MCMC approach using Dan Foreman-Mackey's "emcee": "http://dan.iel.fm/emcee/current/" package. Keep in mind here that the goal is to generate a set of points drawn from the posterior probability distribution, and to use those points to determine the answer we seek.
+To perform this MCMC, we start by defining Python functions for the prior $p(F_\mathrm{true} | I)$, the likelihood $p(D | F_\mathrm{true},I)$, and the posterior $p(F_\mathrm{true} | D,I)$, noting that none of these need be properly normalized. Our model here is one-dimensional, but to handle multi-dimensional models we'll define the model in terms of an array of parameters $\boldsymbol{\alpha}$, which in this case is $\boldsymbol{\alpha} = [F_\mathrm{true}]$
+
+@@@CODE src/singlephotoncount.py from-to: start bayesian setup@end bayesian setup
+
+Now we set up the problem, including generating some random starting guesses for the multiple chains of points.
+
+@@@CODE src/singlephotoncount.py from-to: start bayesian mcmc@end bayesian mcmc
+
+If this all worked correctly, the array sample should contain a series of 50,000 points drawn from the posterior. Let's plot them and check. See results in Fig. ref{fig:flux-bayesian}.
+
+@@@CODE src/singlephotoncount.py from-to: start visualize bayesian@end visualize bayesian
+FIGURE:[fig/singlephotoncount_fig_2.png, width=400 frac=0.8] Bayesian posterior pdf (represented by a histogram of MCMC samples) from flux measurements. label{fig:flux-bayesian}
+
+=== Best estimates and confidence intervals ===
+The posterior distribution from our Bayesian data analysis is the key quantity that encodes our inference about the values of the model parameters, given the data and the relevant background information. Often, however, we wish to summarize this result with just a few numbers: the best estimate and a measure of its reliability. 
+
+There are a few different options for this. The choice of the most appropriate one depends mainly on the shape of the posterior distribution:
+
+*Symmetric posterior pdfs*: Since the probability (density) associated with any particular value of the parameter is a measure of how much we believe that it lies in the neighbourhood of that point, our best estimate is given by the maximum of the posterior pdf. If we denote the quantity of interest by $X$, with a posterior pdf $P =p(X|D,I)$, then the best estimate of its value $X_0$ is given by the condition $dP/dX|_{X=X_0}=0$. Strictly speaking, we should also check the sign of the second derivative to ensure that $X_0$ represents a maximum.
+
+To obtain a measure of the reliability of this best estimate, we need to look at the width or spread of the posterior pdf about $X_0$. When considering the behaviour of any function in the neighbourhood of a particular point, it is often helpful to carry out a Taylor series expansion; this is simply a standard tool for (locally) approximating a complicated function by a low-order polynomial. The linear term is zero at the maximum and the quadratic term is often the dominating one determining the width of the posterior pdf. Ignoring all the higher-order terms we arrive at the Gaussian approximation
+!bt
+\begin{equation}
+p(X|D,I) \approx \frac{1}{\sigma\sqrt{2\pi}} \exp \left[ -\frac{(x-\mu)^2}{2\sigma^2} \right],
+\end{equation}
+!et
+where the mean $\mu = X_0$ and the variance $\sigma = \left( - \left. \frac{d^2L}{dX^2} \right|_{X_0} \right)^{-1/2}$, where $L$ is the logarithm of the posterior $P$. Our inference about the quantity of interest is conveyed very concisely, therefore, by the statement $X = X_0 \pm \sigma$, and 
+!bt
+$$
+p(X_0-\sigma < X < X_0+\sigma | D,I) = \int_{X_0-\sigma}^{X_0+\sigma} p(X|D,I) dX \approx 0.67.
+$$
+!et
+
+*Asymmetric posterior pdfs*: While the maximum of the posterior ($X_0$) can still be regarded as giving the best estimate, the true value is now more likely to be on one side of this rather than the other. Alternatively one can compute the mean value, $\langle X \rangle = \int X p(X|D,I) dX$, although this tends to overemphasise very long tails. The best option is probably a compromise that can be employed when having access to a large sample from the posterior (as provided by an MCMC), namely to give the median of this ensamble.
+
+Furthermore, the concept of an error-bar does not seem appropriate in this case, as it implicitly entails the idea of symmetry. A good way of expressing the reliability with which a parameter can be inferred, for an asymmetric posterior pdf, is rather through a *confidence interval*. Since the area under the posterior pdf between $X_1$ and $X_2$ is proportional to how much we believe that $X$ lies in that range, the shortest interval that encloses 67% of the area represents a sensible measure of the uncertainty of the estimate. Obviously we can choose to provide some other degree-of-belief that we think is relevant for the case at hand. Assuming that the posterior pdf has been normalized, to have unit area, we need to find $X_1$ and $X_2$ such that: 
+!bt
+$$
+p(X_1 < X < X_2 | D,I) = \int_{X_1}^{X_2} p(X|D,I) dX \approx 0.67, 
+$$
+!et
+where the difference $X_2 - X_1$ is as small as possible. The region $X_1 < X < X_2$ is then called the shortest 67% confidence interval. 
+
+*Multimodal posterior pdfs*: We can sometimes obtain posteriors which are multimodal; i.e. contains several disconnected regions with large probabilities. There is no difficulty when one of the maxima is very much larger than the others: we can simply ignore the subsidiary solutions, to a good approximation, and concentrate on the global maximum. The problem arises when there are several maxima of comparable magnitude. What do we now mean by a best estimate, and how should we quantify its reliability? The idea of a best estimate and an error-bar, or even a confidence interval, is merely an attempt to summarize the posterior with just two or three numbers; sometimes this just can’t be done, and so these concepts are not valid. For the bimodal case we might be able to characterize the posterior in terms of a few numbers: two best estimates and their associated error-bars, or disjoint confidence intervals. For a general multimodal pdf, the most honest thing we can do is just display the posterior itself.
+
+=== Simple Photon Counts: Best estimates and confidence intervals ===
+To compute these numbers for our example, you would run:
+
+@@@CODE src/singlephotoncount.py from-to: start bayesian CI@end bayesian CI
+`F_true = 1000` 
+`Based on 50 measurements the posterior point estimates are:` 
+`...F_est = 998 +/- 4` 
+`or using credible intervals:` 
+`...F_est = 998          (posterior median)`  
+`...F_est in [993, 1002] (67% credible interval)`  
+`...F_est in [989, 1006] (95% credible interval)`  
+
+In this particular example, the posterior pdf is actually a Gaussian (since it is constructed as a product of Gaussians), and the mean and variance from the quadratic approximation will agree exactly with the frequentist approach.
+
+From this final result you might come away with the impression that the Bayesian method is unnecessarily complicated, and in this case it certainly is. Using an MCMC sampler to characterize a one-dimensional normal distribution is a bit like using the Death Star to destroy a beach ball, but we did this here because it demonstrates an approach that can scale to complicated posteriors in many, many dimensions, and can provide nice results in more complicated situations where an analytic likelihood approach is not possible.
+
+Furthermore, as data and models grow in complexity, the two approaches can diverge greatly. 
+
+!split
+===== Bayesian parameter estimation (multiple parameters, covariance) =====
+!bblock
+* multidimensional posterior pdf:s
+* nuisance parameters (e.g. background subtraction?)
+* corner plots, covariance, correlations
+* best example?
+!eblock
+
+!split
+===== Bayesian model selection =====
+!bblock
+* Bayesian evidence
+* Occam's razor
+* Best example? How many spectral lines are there?
+!eblock
+
+
diff --git a/doc/src/BoltzmannMachines/BM.do.txt~ b/doc/src/BoltzmannMachines/BM.do.txt~
new file mode 100644
index 000000000..167e6d97b
--- /dev/null
+++ b/doc/src/BoltzmannMachines/BM.do.txt~
@@ -0,0 +1,1440 @@
+TITLE: Machine Learning and Boltzmann machines with applications
+AUTHOR: Morten Hjorth-Jensen {copyright, 1999-present|CC BY-NC} at Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University and Department of Physics, University of Oslo, Norway
+DATE: today 
+
+
+
+!split
+===== Types of Machine Learning, a repetition =====
+
+!bblock
+The approaches to machine learning are many, but are often split into two main categories. 
+In *supervised learning* we know the answer to a problem,
+and let the computer deduce the logic behind it. On the other hand, *unsupervised learning*
+is a method for finding patterns and relationship in data sets without any prior knowledge of the system.
+Some authours also operate with a third category, namely *reinforcement learning*. This is a paradigm 
+of learning inspired by behavioural psychology, where learning is achieved by trial-and-error, 
+solely from rewards and punishment.
+
+Another way to categorize machine learning tasks is to consider the desired output of a system.
+Some of the most common tasks are:
+
+  * Classification: Outputs are divided into two or more classes. The goal is to   produce a model that assigns inputs into one of these classes. An example is to identify  digits based on pictures of hand-written ones. Classification is typically supervised learning.
+
+  * Regression: Finding a functional relationship between an input data set and a reference data set.   The goal is to construct a function that maps input data to continuous output values.
+
+  * Clustering: Data are divided into groups with certain common traits, without knowing the different groups beforehand.  It is thus a form of unsupervised learning.
+
+  * Other unsupervised learning algortihms, here Boltzmann machines
+!eblock
+
+
+!split
+===== Why Boltzmann machines? =====
+
+What is known as restricted Boltzmann Machines (RMB) have received a lot of attention lately. 
+One of the major reasons is that they can be stacked layer-wise to build deep neural networks that capture complicated statistics.
+
+The original RBMs had just one visible layer and a hidden layer, but recently so-called Gaussian-binary RBMs have gained quite some popularity in imaging since they are capable of modeling continuous data that are common to natural images. 
+
+Furthermore, they have been used to solve complicated quantum mechanical many-particle problems or classical statistical physics problems like the Ising and Potts classes of models. 
+
+
+!split
+===== An intermediate step, the Hopfield network and links to the Ising and Potts models =====
+
+More material on Hopfield networks will come here later. 
+
+!split
+===== A brief review on Markov Chains, Metropolis and Gibbs sampling =====
+
+
+* We want to study a physical system which evolves towards equilibrium, from given  initial conditions.
+* We start with a PDF $w(x_0,t_0)$  and we want to understand how the system evolves with time.
+* We want to reach a situation where after a given number of time steps we obtain a steady state. This means that the system reaches its most likely state (equilibrium situation)
+* Our PDF is normally a multidimensional object whose normalization constant is impossible to find.
+* Analytical calculations from $w(x,t)$ are not possible.
+* To sample directly from from $w(x,t)$ is not possible/difficult.
+* The transition probability $W$ is also not  known.
+* How can we establish that we have reached a steady state?   Sounds impossible!
+
+_Use Markov chain Monte Carlo_
+
+!split 
+===== Brownian motion and Markov processes =====
+A Markov process is a random walk with a selected probability for making a
+move. The new move is independent of the previous history of the system.
+
+The Markov process is used repeatedly in Monte Carlo simulations in order to generate
+new random states.
+
+The reason for choosing a Markov process is that when it is run for a
+long enough time starting with a random state, we will eventually reach the most likely state of the system.
+
+In thermodynamics, this means that after a certain number of Markov processes
+we reach an equilibrium distribution.
+
+This mimicks the way a real system reaches
+its most likely state at a given temperature of the surroundings.
+
+!split 
+===== Brownian motion and Markov processes, Ergodicity and Detailed balance =====
+
+To reach this distribution, the Markov process needs to obey two important conditions, that of
+_ergodicity_ and _detailed balance_. These conditions impose then constraints on our algorithms
+for accepting or rejecting new random states.
+
+
+The Metropolis algorithm discussed here
+abides to both these constraints.
+
+The Metropolis algorithm is widely used in Monte Carlo
+simulations and the understanding of it rests within
+the interpretation of random walks and Markov processes.
+
+!split 
+===== Brownian motion and Markov processes, jargon =====
+
+In a random walk one defines a mathematical entity called a _walker_, 
+whose  attributes
+completely define the state of the system in question. 
+
+The state of the system  can refer to any physical quantities,
+from the vibrational state of a molecule specified by a set of quantum numbers, 
+to the brands of coffee in your favourite supermarket.
+
+
+The walker moves in an appropriate state space by a combination of 
+deterministic and random displacements from its previous
+position.
+
+This sequence of steps forms a _chain_.
+
+!split 
+===== Brownian motion and Markov processes, sequence of ingredients =====
+
+* We want to study a physical system which evolves towards equilibrium, from given  initial conditions.
+* Markov chains are intimately linked with the physical process of diffusion. 
+* From a Markov chain we can then derive the conditions for detailed balance and ergodicity. These are the conditions needed for obtaining a steady state.
+* The widely used algorithm for doing this is the so-called Metropolis algorithm, in its refined form the Metropolis-Hastings algorithm.
+
+!split
+===== Applications: almost every field in science =====
+
+* Financial engineering, see for example Patriarca *et al*, Physica _340_, "page 334 (2004)":"http://www.sciencedirect.com/science/article/pii/S0378437104004327".
+* Neuroscience, see for example Lipinski, Physics Medical Biology _35_, "page 441 (1990)":"http://iopscience.iop.org/article/10.1088/0031-9155/35/3/012/meta;jsessionid=FA91B191036E1F10948F7C42B6A6D295.c1" or Farnell and Gibson, Journal of Computational Physics _208_, "page 253 (2005)":"http://www.sciencedirect.com/science/article/pii/S0021999105001087"
+* Tons of applications in physics
+* and chemistry
+* and biology, medicine
+* Nobel prize in economy to Black and Scholes
+!bt
+\[
+\frac{\partial V}{\partial t}+\frac{1}{2}\sigma^{2}S^{2}\frac{\partial^{2} V}{\partial S^{2}}+rS\frac{\partial V}{\partial S}-rV=0.
+\]
+!et
+The Black and Scholes equation is a partial differential equation, which describes the price
+of the option over time. It is a diffusion equation with a random term.
+
+The list of applications is endless.
+
+
+!split
+===== Markov processes =====
+!bblock
+A Markov process allows in principle for a microscopic description of Brownian motion.
+As with the random walk studied in the previous section, we consider a particle 
+which moves along the  $x$-axis in the form of a series of jumps with step length 
+$\Delta x = l$. Time and space are discretized and the subsequent moves are
+statistically independent, i.e., the new move depends only on the previous step
+and not on the results from earlier trials. 
+We start at a position $x=jl=j\Delta x$ and move to 
+a new position $x =i\Delta x$ during a step $\Delta t=\epsilon$, where 
+$i\ge  0$ and $j\ge 0$ are integers. 
+The original probability distribution function (PDF) of the particles is given by  
+$w_i(t=0)$ where $i$ refers to a specific position on the grid in 
+!eblock
+The function $w_i(t=0)$ is now the discretized version of $w(x,t)$.
+We can regard the discretized PDF as a vector.
+
+!split
+===== Markov processes =====
+!bblock
+For the Markov process we have a transition probability from a position
+$x=jl$ to a position $x=il$ given by
+!bt
+\begin{equation*}
+   W_{ij}(\epsilon)=W(il-jl,\epsilon)=\left\{\begin{array}{cc}\frac{1}{2} & |i-j| = 1\\
+             0 & \mathrm{else} \end{array} \right. ,
+\end{equation*}
+!et
+where $W_{ij}$ is normally called 
+the transition probability and we can represent it, see below,
+as a matrix. 
+_Here we have specialized to a case where the transition probability is known_.
+
+Our new PDF $w_i(t=\epsilon)$ is now related to the PDF at
+$t=0$ through the relation
+
+!bt
+\begin{equation*} 
+   w_i(t=\epsilon) =\sum_{j} W(j\rightarrow i)w_j(t=0).
+\end{equation*}
+!et
+ 
+This equation represents the discretized time-development of an original 
+PDF with equal probability of jumping left or right.
+!eblock
+
+!split
+===== Markov processes, the probabilities =====
+!bblock
+
+Since both $W$ and $w$ represent probabilities, they have to be normalized, i.e., we require
+that at each time step we have
+
+!bt
+\begin{equation*} 
+   \sum_i w_i(t) = 1, 
+\end{equation*}
+!et
+and
+
+!bt
+\begin{equation*} 
+   \sum_j W(j\rightarrow i) = 1,
+\end{equation*}
+!et
+which applies for all $j$-values.
+The further constraints are
+$0 \le W_{ij} \le 1$  and  $0 \le w_{j} \le 1$.
+Note that the probability for remaining at the same place is in general 
+not necessarily equal zero. 
+!eblock
+
+!split
+===== Markov processes =====
+!bblock
+The time development of our initial PDF can now be represented through the action of
+the transition probability matrix applied $n$ times. At a 
+time  $t_n=n\epsilon$ our initial distribution has developed into
+
+!bt
+\begin{equation*}
+   w_i(t_n) = \sum_jW_{ij}(t_n)w_j(0),
+\end{equation*}
+!et
+and defining
+
+!bt
+\begin{equation*}
+   W(il-jl,n\epsilon)=(W^n(\epsilon))_{ij}
+\end{equation*}
+!et
+we obtain
+
+!bt
+\begin{equation*}
+   w_i(n\epsilon) = \sum_j(W^n(\epsilon))_{ij}w_j(0),
+\end{equation*}
+!et
+or in matrix form
+!bt
+\begin{equation} label{eq:wfinal}
+   \hat{w}(n\epsilon) = \hat{W}^n(\epsilon)\hat{w}(0).
+\end{equation}
+!et
+!eblock
+
+!split
+===== An Illustrative Example =====
+!bblock
+
+The following simple example may help in understanding the meaning of 
+the transition matrix $\hat{W}$ and the vector $\hat{w}$.
+Consider the $4\times 4$ matrix $\hat{W}$
+
+!bt
+\begin{equation*}
+   \hat{W} = \left(\begin{array}{cccc} 1/4 & 1/9 & 3/8 & 1/3 \\                   
+                                       2/4 & 2/9 & 0 & 1/3\\                   
+                                       0   & 1/9 & 3/8 & 0\\
+                                       1/4 & 5/9&  2/8 & 1/3 \end{array} \right),
+\end{equation*}
+!et
+and we choose our initial state as
+
+!bt
+\begin{equation*}
+\hat{w}(t=0)=  \left(\begin{array}{c} 1\\                   
+                                 0\\
+                                 0 \\                   
+                                 0 \end{array} \right).
+\end{equation*}
+!et
+!eblock
+
+!split
+===== An Illustrative Example =====
+!bblock
+We note that both the vector and the matrix are properly normalized. Summing the vector elements gives one and
+summing over columns for the matrix results also in one.  Furthermore, the largest eigenvalue is one.
+We act then on $\hat{w}$ with $\hat{W}$.
+The first iteration is
+
+!bt
+\begin{equation*}
+   \hat{w}(t=\epsilon) = \hat{W}\hat{w}(t=0),
+\end{equation*}
+!et
+ 
+resulting in
+
+!bt
+\begin{equation*}
+\hat{w}(t=\epsilon)=  \left(\begin{array}{c} 1/4\\                   
+                                1/2 \\
+                                0 \\                   
+                                1/4 \end{array} \right).
+\end{equation*}
+!et
+!eblock
+
+!split
+===== An Illustrative Example, next step =====
+!bblock
+
+The next iteration results in
+
+!bt
+\begin{equation*}
+   \hat{w}(t=2\epsilon) = \hat{W}\hat{w}(t=\epsilon),
+\end{equation*}
+!et
+ 
+resulting in
+
+!bt
+\begin{equation*}
+\hat{w}(t=2\epsilon)=  \left(\begin{array}{c} 0.201389\\
+   0.319444 \\
+   0.055556 \\
+   0.423611 \end{array} \right).
+\end{equation*}
+!et
+Note that the vector $\hat{w}$ is always normalized to $1$. 
+!eblock
+!split
+===== An Illustrative Example, the steady state =====
+!bblock
+We find the steady state of the system by solving the set of equations
+
+!bt
+\begin{equation*} 
+w(t=\infty) = Ww(t=\infty),
+\end{equation*}
+!et
+which is an eigenvalue problem with eigenvalue equal to _one_!
+This set of equations reads
+!bt
+\begin{align}
+ W_{11}w_1(t=\infty) +W_{12}w_2(t=\infty) +W_{13}w_3(t=\infty)+ W_{14}w_4(t=\infty)=&w_1(t=\infty) \nonumber \\
+W_{21}w_1(t=\infty) + W_{22}w_2(t=\infty) + W_{23}w_3(t=\infty)+ W_{24}w_4(t=\infty)=&w_2(t=\infty) \nonumber \\
+W_{31}w_1(t=\infty) + W_{32}w_2(t=\infty) + W_{33}w_3(t=\infty)+ W_{34}w_4(t=\infty)=&w_3(t=\infty) \nonumber \\
+W_{41}w_1(t=\infty) + W_{42}w_2(t=\infty) + W_{43}w_3(t=\infty)+ W_{44}w_4(t=\infty)=&w_4(t=\infty) \nonumber \\
+\end{align}
+!et
+with the constraint that
+
+!bt
+\begin{equation*}
+   \sum_i w_i(t=\infty) = 1, 
+\end{equation*}
+!et
+yielding as solution
+
+!bt
+\begin{equation*}
+\hat{w}(t=\infty)=  \left(\begin{array}{c}0.244318 \\                   
+                                 0.319602 \\  0.056818 \\  0.379261 \end{array} \right).
+\end{equation*}
+!et
+!eblock
+
+!split
+===== An Illustrative Example, iterative steps =====
+!bblock
+
+The table here  demonstrates the convergence as a function of the number of iterations or
+time steps. After  twelve iterations we have reached the exact value with six leading digits. 
+
+|-------------------------------------------------------------------------------------------------------------|
+| Iteration           | $w_1$               | $w_2$               | $w_3$               | $w_4$               |
+|---------r--------------------l--------------------l--------------------l--------------------l---------------|
+| 0                   | 1.000000            | 0.000000            | 0.000000            | 0.000000            |
+| 1                   | 0.250000            | 0.500000            | 0.000000            | 0.250000            |
+| 2                   | 0.201389            | 0.319444            | 0.055556            | 0.423611            |
+| 3                   | 0.247878            | 0.312886            | 0.056327            | 0.382909            |
+| 4                   | 0.245494            | 0.321106            | 0.055888            | 0.377513            |
+| 5                   | 0.243847            | 0.319941            | 0.056636            | 0.379575            |
+| 6                   | 0.244274            | 0.319547            | 0.056788            | 0.379391            |
+| 7                   | 0.244333            | 0.319611            | 0.056801            | 0.379255            |
+| 8                   | 0.244314            | 0.319610            | 0.056813            | 0.379264            |
+| 9                   | 0.244317            | 0.319603            | 0.056817            | 0.379264            |
+| 10                  | 0.244318            | 0.319602            | 0.056818            | 0.379262            |
+| 11                  | 0.244318            | 0.319602            | 0.056818            | 0.379261            |
+| 12                  | 0.244318            | 0.319602            | 0.056818            | 0.379261            |
+| $\hat{w}(t=\infty)$ | 0.244318            | 0.319602            | 0.056818            | 0.379261            |
+|-------------------------------------------------------------------------------------------------------------|
+
+!eblock
+
+!split
+===== An Illustrative Example, what does it mean? =====
+!bblock
+
+We have after $t$-steps
+
+!bt
+\begin{equation*}
+   \hat{w}(t) = \hat{W}^t\hat{w}(0),
+\end{equation*}
+!et
+with $\hat{w}(0)$ the distribution at $t=0$ and $\hat{W}$ representing the 
+transition probability matrix. 
+
+!eblock
+
+!split
+===== An Illustrative Example, understanding the basics =====
+!bblock
+
+We can always expand $\hat{w}(0)$ in terms of the right eigenvectors 
+$\hat{v}$ of $\hat{W}$ as
+
+!bt
+\begin{equation*}
+    \hat{w}(0)  = \sum_i\alpha_i\hat{v}_i,
+\end{equation*}
+!et
+resulting in
+
+!bt
+\begin{equation*}
+   \hat{w}(t) = \hat{W}^t\hat{w}(0)=\hat{W}^t\sum_i\alpha_i\hat{v}_i=
+\sum_i\lambda_i^t\alpha_i\hat{v}_i,
+\end{equation*}
+!et
+with $\lambda_i$ the $i^{\mathrm{th}}$ eigenvalue corresponding to  
+the eigenvector $\hat{v}_i$. 
+
+If we assume that $\lambda_0$ is the largest eigenvector we see that in the limit $t\rightarrow \infty$,
+$\hat{w}(t)$ becomes proportional to the corresponding eigenvector 
+$\hat{v}_0$. This is our steady state or final distribution. 
+
+!eblock
+
+
+
+!split
+===== The Metropolis Algorithm and Detailed Balance  =====
+!bblock
+
+Let us recapitulate some of our results about Markov chains and random walks.
+
+  * The time development of our PDF $w(t)$, after 
+one time-step from $t=0$ is given by  
+!bt
+\begin{equation*}    
+w_i(t=\epsilon) = W(j\rightarrow i)w_j(t=0).
+\end{equation*}
+!et
+ 
+This equation represents the discretized time-development of an original 
+PDF.  We can rewrite this as a
+
+!bt
+\begin{equation*}
+   w_i(t=\epsilon) = W_{ij}w_j(t=0).
+\end{equation*}
+!et
+with the transition matrix $W$ for a random walk given by
+
+!bt
+\begin{equation*}
+   W_{ij}(\epsilon)=W(il-jl,\epsilon)=\left\{\begin{array}{cc}\frac{1}{2} & |i-j| = 1\\
+                                             0 & \mathrm{else} \end{array} \right.
+\end{equation*}
+!et
+
+!eblock
+
+!split
+===== The Metropolis Algorithm and Detailed Balance  =====
+!bblock
+
+We call $W_{ij}$ for the transition probability and we represent it
+as a matrix. 
+  * Both  $W$ and $w$ represent probabilities and they have to be normalized, meaning that at each time step we have   
+!bt 
+\begin{equation*}    
+\sum_i w_i(t) = 1,
+\end{equation*}
+!et
+and
+
+!bt
+\begin{equation*} 
+   \sum_j W(j\rightarrow i) = 1.
+\end{equation*}
+!et
+Here we have written the previous matrix $W_{ij}=W(j\rightarrow i)$.
+
+!eblock
+
+!split
+===== The Metropolis Algorithm and Detailed Balance  =====
+!bblock
+
+The further constraints are
+$0 \le W_{ij} \le 1$  and  $0 \le w_{j} \le 1$.
+  * We can thus write the action of $W$ as   
+!bt 
+\begin{equation*}    
+w_i(t+1) = \sum_jW_{ij}w_j(t),
+\end{equation*}
+!et
+or as vector-matrix relation
+
+!bt
+\begin{equation*}
+   \hat{w}(t+1) = \hat{W\hat{w}}(t),
+\end{equation*}
+!et
+and if we have that $||\hat{w}(t+1)-\hat{w}(t)||\rightarrow 0$, we say that 
+we have reached the most likely state of the system, the so-called steady state or equilibrium state.
+!eblock
+
+!split
+===== The Metropolis Algorithm and Detailed Balance  =====
+!bblock
+
+Another way of phrasing this is
+!bt
+\begin{equation} 
+w(t=\infty) = Ww(t=\infty). 
+\end{equation}   
+!et
+!eblock
+
+!split
+===== The Metropolis Algorithm and Detailed Balance  =====
+!bblock
+
+The question then is how can we model anything under such a severe lack of knowledge? The Metropolis algorithm comes to our rescue here. Since $W(j\rightarrow i)$ is unknown, we model it as the product of two probabilities, 
+a probability for accepting the proposed move from the state $j$ to the state $j$, and a probability for making the transition to the state $i$ being in the state $j$. We label these probabilities $A(j\rightarrow i)$ and $T(j\rightarrow i)$, respectively.  Our total transition probability is then
+
+!bt
+\begin{equation*}
+W(j\rightarrow i)=T(j\rightarrow i)A(j\rightarrow i).
+\end{equation*}
+!et
+The algorithm can then be expressed as
+
+  * We make a suggested move to the new state $i$ with some transition or moving probability $T_{j\rightarrow i}$.
+
+  * We accept this move to the new state with an acceptance probability $A_{j \rightarrow i}$. The new state $i$ is in turn used as our new starting point for the next move.  We reject this proposed moved with a $1-A_{j\rightarrow i}$ and the original state $j$ is used again as a sample.
+!eblock
+
+!split
+===== The Metropolis Algorithm and Detailed Balance  =====
+!bblock
+
+We wish to derive the required properties of the probabilities $T$ and $A$ such that
+$w_i^{(t\rightarrow \infty)} \rightarrow w_i$, starting
+from any distribution, will lead us to the correct distribution.
+
+We can now derive the dynamical process towards 
+equilibrium. To obtain this equation we note that after $t$ time steps the probability for being in a state $i$ is related 
+to the probability of being in a state $j$ and performing a transition to the new state together with the probability of actually being in the state $i$ and making a move to any of the possible states $j$ from the previous time step.
+!eblock
+
+!split
+===== The Metropolis Algorithm and Detailed Balance  =====
+!bblock
+
+We can express this as, assuming that $T$ and $A$ are time-independent,
+
+!bt
+\begin{equation*}
+w_i(t+1) = \sum_j \left [
+w_j(t)T_{j\rightarrow i} A_{j\rightarrow i} 
++w_i(t)T_{i\rightarrow j}\left ( 1- A_{i\rightarrow j} \right)
+\right ] \,.
+\end{equation*}
+!et
+
+!eblock
+
+!split
+===== The Metropolis Algorithm and Detailed Balance  =====
+!bblock
+
+All probabilities are normalized, meaning that 
+$\sum_j T_{i\rightarrow j} = 1$. Using the latter, we can rewrite the previous equation as
+
+!bt
+\begin{equation*}
+w_i(t+1) = w_i(t) +
+ \sum_j \left [
+w_j(t)T_{j\rightarrow i} A_{j\rightarrow i} 
+-w_i(t)T_{i\rightarrow j}A_{i\rightarrow j}\right ] \,,
+\end{equation*}
+!et
+which can be rewritten as
+
+!bt
+\begin{equation*}
+w_i(t+1)-w_i(t) =  \sum_j \left [w_j(t)T_{j\rightarrow i} A_{j\rightarrow i} 
+-w_i(t)T_{i\rightarrow j}A_{i\rightarrow j}\right ] .
+\end{equation*}
+!et
+
+!eblock
+
+!split
+===== The Metropolis Algorithm and Detailed Balance  =====
+!bblock
+
+The last equation is very similar to the so-called Master equation, which relates the temporal dependence of 
+a PDF $w_i(t)$ to various transition rates. The equation can be derived from the so-called 
+Chapman-Einstein-Enskog-Kolmogorov equation. The equation is given as 
+!bt
+\begin{equation}
+label{eq:masterequation}
+\frac{d w_i(t)}{dt} = \sum_j\left[ W(j\rightarrow i)w_j-W(i\rightarrow j)w_i\right],
+\end{equation}
+!et
+which simply states that the rate at which the systems moves from a state $j$
+to a final state $i$ (the first term on the right-hand side of the last equation) is balanced by the rate at which the system undergoes transitions from the state $i$ to a state $j$ (the second term). If we have reached the so-called steady state, then the temporal development is zero. This  means that in equilibrium we have
+
+!bt
+\begin{equation*}
+\frac{d w_i(t)}{dt} = 0.
+\end{equation*}
+!et
+
+!eblock
+
+!split
+===== The Metropolis Algorithm and Detailed Balance  =====
+!bblock
+
+In the limit $t\rightarrow \infty$ we require that the  two distributions $w_i(t+1)=w_i$ and $w_i(t)=w_i$
+and we have
+
+!bt
+\begin{equation*}
+ \sum_j w_jT_{j\rightarrow i} A_{j\rightarrow i}= \sum_j w_iT_{i\rightarrow j}A_{i\rightarrow j},
+\end{equation*}
+!et
+which is the condition for balance when the most likely state (or steady state) has been reached.
+We see also that the right-hand side  can be  rewritten as
+
+!bt
+\begin{equation*} 
+\sum_j w_iT_{i\rightarrow j}A_{i\rightarrow j}= \sum_j w_iW_{i\rightarrow j},
+\end{equation*}
+!et
+and using the property that $\sum_j W_{i\rightarrow j}=1$, we can rewrite our equation
+as
+
+!bt
+\begin{equation*}
+w_i= \sum_j w_jT_{j\rightarrow i} A_{j\rightarrow i}= \sum_j w_j W_{j\rightarrow i},
+\end{equation*}
+!et
+which is nothing but the standard equation for a Markov chain when the steady state has been reached.
+
+!eblock
+
+!split
+===== The Metropolis Algorithm and Detailed Balance  =====
+!bblock
+
+However, the condition that the rates should equal each other is in general not sufficient
+to guarantee that we, after many simulations, generate the correct distribution.
+We may risk to end up with so-called cyclic solutions. To avoid this
+we therefore introduce an additional condition, namely that of detailed balance
+
+!bt
+\begin{equation*} W(j\rightarrow i)w_j= W(i\rightarrow j)w_i.  \end{equation*}
+!et
+These equations were derived by Lars Onsager when studying irreversible processes.
+At equilibrium detailed balance gives thus
+
+!bt
+\begin{equation*} \frac{W(j\rightarrow i)}{W(i\rightarrow j)}=\frac{w_i}{w_j}.  \end{equation*}
+!et
+Rewriting the last equation in terms of our transition probabilities $T$ and 
+acceptance probobalities $A$ we obtain
+
+!bt
+\begin{equation*} 
+w_j(t)T_{j\rightarrow i}A_{j\rightarrow i}= w_i(t)T_{i\rightarrow j}A_{i\rightarrow j}.
+\end{equation*}
+!et
+!eblock
+
+!split
+===== The Metropolis Algorithm and Detailed Balance  =====
+!bblock
+
+Since we normally have an expression 
+for the probability distribution functions $w_i$,  we can rewrite the last equation as
+
+!bt
+\begin{equation*} 
+\frac{T_{j\rightarrow i}A_{j\rightarrow i}}{T_{i\rightarrow j}A_{i\rightarrow j}}= \frac{w_i}{w_j}.
+\end{equation*}
+!et
+!eblock
+
+!split
+===== The Metropolis Algorithm and Detailed Balance  =====
+!bblock
+
+In statistical physics this condition ensures that it is e.g., the 
+Boltzmann distribution which is generated when equilibrium is reached.
+
+We introduce  now the Boltzmann distribution
+
+!bt
+\begin{equation*}
+   w_i= \frac{\exp{(-\beta(E_i))}}{Z},
+\end{equation*}
+!et
+which states that the probability of finding the system in a state $i$ with energy $E_i$ 
+at an inverse temperature $\beta = 1/k_BT$ is $w_i\propto \exp{(-\beta(E_i))}$.
+The denominator $Z$ is a normalization constant which ensures that the sum of all
+probabilities is normalized to one. It is defined as the sum of probabilities over all microstates
+$j$ of the system
+
+!bt
+\begin{equation*}
+   Z=\sum_j \exp{(-\beta(E_i))}.
+\end{equation*}
+!et
+!eblock
+
+!split
+===== The Metropolis Algorithm and Detailed Balance  =====
+!bblock
+
+From the partition function we can in principle generate all interesting quantities
+for a given system in equilibrium with its surroundings at a temperature $T$.
+
+With the probability distribution given by the Boltzmann distribution we are now in a position
+where we can generate expectation values for a given variable $A$ through the
+definition
+
+!bt
+\begin{equation*}
+   \langle A \rangle = \sum_jA_jw_j=
+    \frac{\sum_jA_j\exp{(-\beta(E_j)}}{Z}.
+\end{equation*}
+!et
+In general, most systems have an infinity of microstates making thereby the computation
+of $Z$ practically impossible and 
+a brute force Monte Carlo calculation over a given number of randomly selected microstates
+may therefore not yield those microstates which are important 
+at equilibrium. 
+To select the most important contributions we need to  
+use the condition for detailed balance. Since this is just given by the ratios of probabilities,
+we never need to evaluate the partition function $Z$.
+
+!eblock
+
+!split
+===== The Metropolis Algorithm and Detailed Balance  =====
+!bblock
+
+For the 
+Boltzmann distribution, detailed balance results in
+
+!bt
+\begin{equation*} \frac{w_i}{w_j}= \exp{(-\beta(E_i-E_j))}. \end{equation*}
+!et
+
+Let us now specialize to a system whose energy is defined by the orientation of single spins.
+Consider the state $i$, with given energy $E_i$ represented by the following $N$ spins
+
+!bt
+\begin{equation*}
+\begin{array}{cccccccccc}
+\uparrow&\uparrow&\uparrow&\dots&\uparrow&\downarrow&\uparrow&\dots&\uparrow&\downarrow\\
+1&2&3&\dots& k-1&k&k+1&\dots&N-1&N\end{array}
+\end{equation*}
+!et
+!eblock
+
+!split
+===== The Metropolis Algorithm and Detailed Balance  =====
+!bblock
+
+We are interested in the transition with one single  spinflip to a new state $j$ with energy $E_j$
+
+!bt
+\begin{equation*}
+\begin{array}{cccccccccc}
+\uparrow&\uparrow&\uparrow&\dots&\uparrow&\uparrow&\uparrow&\dots&\uparrow&\downarrow\\
+1&2&3&\dots& k-1&k&k+1&\dots&N-1&N\end{array}
+\end{equation*}
+!et
+This change from one microstate $i$ (or spin configuration)  to another microstate $j$ is the
+configuration space analogue to a random walk on a lattice. Instead of jumping from 
+one place to another in space, we 'jump' from one microstate to another.
+!eblock
+
+!split
+===== The Metropolis Algorithm and Detailed Balance  =====
+!bblock
+
+However, the selection of states has to generate a final distribution which is the
+Boltzmann distribution. This is again the same we saw for a random walker, for the discrete case we had 
+always a binomial distribution, whereas for the continuous case we had a normal distribution.
+The way we sample configurations should result, when equilibrium is established, in the 
+Boltzmann distribution. Else, our algorithm for selecting microstates is wrong.
+ 
+
+As stated above, we do in general not know the closed-form expression of the transition rate and we are free to model it as
+     $W(i\rightarrow j)=T(i\rightarrow j)A(i\rightarrow j)$.
+Our ratio between probabilities gives us
+
+!bt
+\begin{equation*} 
+\frac{A_{j\rightarrow i}}{A_{i\rightarrow j}}= \frac{w_iT_{i\rightarrow j}}{w_jT_{j\rightarrow i}}.
+\end{equation*}
+!et
+The simplest form of the Metropolis algorithm (sometimes called for brute force Metropolis) assumes that 
+the transition probability $T(i\rightarrow j)$ is symmetric, implying that $T(i\rightarrow j)=T(j\rightarrow i)$.
+
+!eblock
+
+!split
+===== The Metropolis Algorithm and Detailed Balance  =====
+!bblock
+
+We obtain then (using the Boltzmann distribution)
+
+!bt
+\begin{equation*}
+\frac{A(j\rightarrow i)}{A(i\rightarrow j)}= \exp{(-\beta(E_i-E_j))} .
+\end{equation*}
+!et
+We are in this case interested in a new state $E_j$ whose energy is lower than 
+$E_i$, viz., $\Delta E = E_j-E_i \le 0$. A simple test would then be to accept only those
+microstates which lower the energy.  
+Suppose we have ten microstates with energy $E_0 \le E_1 \le E_2 \le E_3 \le \dots \le E_9$.
+Our desired energy is $E_0$.
+
+!eblock
+
+!split
+===== The Metropolis Algorithm and Detailed Balance  =====
+!bblock
+
+At a given temperature $T$ we start our simulation by randomly choosing state
+$E_9$. Flipping spins we may then find a path from $E_9\rightarrow E_8 \rightarrow E_7 \dots \rightarrow E_1 \rightarrow E_0$. 
+This would however lead to biased statistical averages since it would violate the ergodic hypothesis discussed
+in the previous section. This principle states  that 
+it should be possible for any Markov process to reach every possible state of the system
+from any starting point if the simulations is carried out for a long enough time.
+
+Any state in a Boltzmann distribution has a probability different from zero and if such 
+a state cannot be reached from a given starting point, then the system is not ergodic.
+This means that another possible path to $E_0$ could be 
+$E_9\rightarrow E_7 \rightarrow E_8 \dots \rightarrow E_9 \rightarrow E_5 \rightarrow E_0$ and so forth.
+Even though such a path could have a negligible probability it is still a possibility, and if
+we simulate long enough it should be included in our computation of an expectation value.
+!eblock
+
+!split
+===== The Metropolis Algorithm and Detailed Balance  =====
+!bblock
+
+Thus, we require that our algorithm should satisfy the principle of detailed balance and be ergodic. 
+The problem with our ratio
+
+!bt
+\begin{equation*}
+\frac{A(j\rightarrow i)}{A(i\rightarrow j)}= \exp{(-\beta(E_i-E_j))}, 
+\end{equation*}
+!et
+is that we do not know the acceptance probability. This equation only specifies the ratio of pairs of probabilities. Normally we want an algorithm which is as efficient as possible and maximizes the number of accepted moves. 
+Moreover, we know that the acceptance probability has $0$ as its smallest value and $1$ as its largest. 
+If we assume that the largest possible acceptance probability is $1$,  we adjust thereafter the other acceptance probability
+to this constraint. 
+!eblock
+
+!split
+===== The Metropolis Algorithm and Detailed Balance  =====
+!bblock
+
+To understand this better, assume that we have two energies, $E_i$ and $E_j$, with $E_i < E_j$. This means that the largest acceptance value must be 
+$A(j\rightarrow i)$ since we move to a state with lower energy.  It follows from also from the fact that the probability $w_i$ is larger than $w_j$. 
+The trick then is to fix this value to $A(j\rightarrow i)=1$. It means that 
+the other acceptance probability has to be
+
+!bt
+\begin{equation*}
+A(i\rightarrow j)= \exp{(-\beta(E_j-E_i))}.
+\end{equation*}
+!et
+
+!eblock
+
+!split
+===== The Metropolis Algorithm and Detailed Balance  =====
+!bblock
+
+One possible way to encode this equation reads
+
+!bt
+\begin{equation*}       
+A(j\rightarrow i)=\left\{\begin{array}{cc}
+\exp{(-\beta(E_i-E_j))} & E_i-E_j > 0 \\ 1 & else \end{array} \right.,  
+\end{equation*}
+!et
+implying that if we move to a state with a lower energy, we always accept
+this move with acceptance probability $A(j\rightarrow i)=1$. If the energy is higher, we need to check
+this acceptance probability with the ratio between the probabilities  from our PDF.  From a practical point of view, 
+the above ratio is compared with a random number.
+If the ratio is smaller than a given random number we accept the move to a higher energy, else we stay in the same state. 
+
+!eblock
+
+!split
+===== The Metropolis Algorithm and Detailed Balance  =====
+!bblock
+
+Nothing hinders us obviously in choosing another acceptance ratio, like a weighting  of the two energies via
+
+!bt
+\begin{equation*}
+A(j\rightarrow i)=\exp{(-\frac{1}{2}\beta(E_i-E_j))}.
+\end{equation*}
+!et
+However, it is easy to see that such an acceptance ratio would result in 
+fewer accepted moves.
+!eblock
+
+!split
+===== Brief Summary =====
+
+The Monte Carlo approach, combined with the theory for Markov chains can be summarized as follows:
+A Markov chain Monte Carlo method for the simulation of a distribution $w$ is any method producing an 
+ergodic Markov chain of events $x$ whose stationary distribution is $w$. The Metropolis algorithm can be phrased as
+
+  * Generate an initial value $x^{(i)}$.
+
+  * Generate a trial value $y_t$ with probability $T(y_t|x^{(i)})$. The latter quantity represents the probability of generating $y_t$ given $x^{(i)}$.
+
+  * Take a new value   
+!bt 
+\begin{equation*}  
+x^{(i+1)}= \left\{\begin{array}{cc} y_t & \mathrm{with\hspace{0.1cm}probability} = A(x^{(i)}\rightarrow y_t) \\                                           x^{(i)}    & \mathrm{with \hspace{0.1cm}probability} = 1-A(x^{(i)}\rightarrow y_t)\end{array}\right .
+\end{equation*}
+!et 
+
+  * We have defined the transition (acceptance) probability as   
+!bt 
+\begin{equation*}    A(x\rightarrow y)= \mathrm{min}\left\{\frac{w(y)T(x|y)}{w(x)T(y|x)},1\right\}.
+\end{equation*}
+!et
+
+
+!split
+===== Gibbs sampling =====
+
+More text to come. 
+
+
+!split
+===== Boltzmann Machines =====
+
+Why use a generative model rather than the more well known discriminative deep neural networks (DNN)? 
+
+* Discriminitave methods have several limitations: They are mainly supervised learning methods, thus requiring labeled data. And there are tasks they cannot accomplish, like drawing new examples from an unknown probability distribution.
+
+* A generative model can learn to represent and sample from a probability distribution. The core idea is to learn a parametric model of the probability distribution from which the training data was drawn. As an example
+
+  o A model for images could learn to draw new examples of cats and dogs, given a training dataset of images of cats and dogs.
+  o Generate a sample of an ordered or disordered Ising model phase, having been given samples of such phases.
+  o Model the trial function for Monte Carlo calculations
+
+
+!split
+===== Some similarities and differences from DNNs =====
+
+ o Both use gradient-descent based learning procedures for minimizing cost functions
+ o Energy based models don't use backpropagation and automatic differentiation for computing gradients, instead turning to Markov Chain Monte Carlo methods.
+ o DNNs often have several hidden layers. A restricted Boltzmann machine has only one hidden layer, however several RBMs can be stacked to make up Deep Belief Networks, of which they constitute the building blocks.
+
+History: The RBM was developed by amongst others Geoffrey Hinton, called by some the "Godfather of Deep Learning", working with the University of Toronto and Google.
+
+
+!split
+=====  Boltzmann machines (BM) =====
+
+!bblock
+A BM is what we would call an undirected probabilistic graphical model
+with stochastic continuous or discrete units.
+!eblock
+!bblock
+It is interpreted as a stochastic recurrent neural network where the
+state of each unit(neurons/nodes) depends on the units it is connected
+to. The weights in the network represent thus the strength of the
+interaction between various units/nodes.
+!eblock
+!bblock
+It turns into a Hopfield network if we choose deterministic rather
+than stochastic units. In contrast to a Hopfield network, a BM is a
+so-called generative model. It allows us to generate new samples from
+the learned distribution.
+!eblock
+
+!split
+===== A standard BM setup =====
+
+!bblock
+A standard BM network is divided into a set of observable and visible units $\hat{x}$ and a set of unknown hidden units/nodes $\hat{h}$.
+!eblock
+
+!bblock
+Additionally there can be bias nodes for the hidden and visible layers. These biases are normally set to $1$.
+!eblock
+
+!bblock
+BMs are stackable, meaning they cwe can train a BM which serves as input to another BM. We can construct deep networks for learning complex PDFs. The layers can be trained one after another, a feature which makes them popular in deep learning
+!eblock
+
+However, they are often hard to train. This leads to the introduction of so-called restricted BMs, or RBMS.
+Here we take away all lateral connections between nodes in the visible layer as well as connections between nodes in the hidden layer. The network is illustrated in the figure below.
+
+
+
+!split
+===== The structure of the RBM network =====
+
+FIGURE: [figures/RBM.pdf, width=800 frac=1.0]
+
+
+
+!split
+===== The network =====
+
+_The network layers_:
+  o A function $\mathbf{x}$ that represents the visible layer, a vector of $M$ elements (nodes). This layer represents both what the RBM might be given as training input, and what we want it to be able to reconstruct. This might for example be the pixels of an image, the spin values of the Ising model, or coefficients representing speech.
+  o The function $\mathbf{h}$ represents the hidden, or latent, layer. A vector of $N$ elements (nodes). Also called "feature detectors".
+
+!split
+===== Goals =====
+
+The goal of the hidden layer is to increase the model's expressive power. We encode complex interactions between visible variables by introducing additional, hidden variables that interact with visible degrees of freedom in a simple manner, yet still reproduce the complex correlations between visible degrees in the data once marginalized over (integrated out).
+
+Examples of this trick being employed in physics: 
+  o The Hubbard-Stratonovich transformation
+  o The introduction of ghost fields in gauge theory
+  o Shadow wave functions in Quantum Monte Carlo simulations
+_The network parameters, to be optimized/learned_:
+  o $\mathbf{a}$ represents the visible bias, a vector of same length as $\mathbf{x}$.
+  o $\mathbf{b}$ represents the hidden bias, a vector of same lenght as $\mathbf{h}$.
+  o $W$ represents the interaction weights, a matrix of size $M\times N$.
+
+
+
+
+!split
+===== Joint distribution =====
+The restricted Boltzmann machine is described by a Bolztmann distribution
+!bt
+\begin{align}
+	P_{rbm}(\mathbf{x},\mathbf{h}) = \frac{1}{Z} e^{-\frac{1}{T_0}E(\mathbf{x},\mathbf{h})},
+\end{align}
+!et
+where $Z$ is the normalization constant or partition function, defined as 
+!bt
+\begin{align}
+	Z = \int \int e^{-\frac{1}{T_0}E(\mathbf{x},\mathbf{h})} d\mathbf{x} d\mathbf{h}.
+\end{align}
+!et
+It is common to ignore $T_0$ by setting it to one. 
+
+
+
+
+!split
+===== Network Elements, the energy function  =====
+
+The function $E(\mathbf{x},\mathbf{h})$ gives the _energy_ of a
+configuration (pair of vectors) $(\mathbf{x}, \mathbf{h})$. The lower
+the energy of a configuration, the higher the probability of it. This
+function also depends on the parameters $\mathbf{a}$, $\mathbf{b}$ and
+$W$. Thus, when we adjust them during the learning procedure, we are
+adjusting the energy function to best fit our problem.
+
+An expression for the energy function is
+!bt
+\[
+E(\hat{x},\hat{h}) = -\sum_{ia}^{NA}b_i^a \alpha_i^a(x_i)-\sum_{jd}^{MD}c_j^d \beta_j^d(h_j)-\sum_{ijad}^{NAMD}b_i^a \alpha_i^a(x_i)c_j^d \beta_j^d(h_j)w_{ij}^{ad}.
+\]
+!et
+
+Here $\beta_j^d(h_j)$ and $\alpha_i^a(x_j)$ are so-called transfer functions that map a given input value to a desired feature value. The labels $a$ and $d$ denote that there can be multiple transfer functions per variable. The first sum depends only on the visible units. The second on the hidden ones. _Note_ that there is no connection between nodes in a layer.
+
+The quantities $b$ and $c$ can be interpreted as the visible and hidden biases, respectively.
+
+The connection between the nodes in the two layers is given by the weights $w_{ij}$. 
+
+!split
+===== Defining different types of RBMs =====
+There are different variants of RBMs, and the differences lie in the types of visible and hidden units we choose as well as in the implementation of the energy function $E(\mathbf{x},\mathbf{h})$. 
+
+!bblock Binary-Binary RBM:
+
+RBMs were first developed using binary units in both the visible and hidden layer. The corresponding energy function is defined as follows:
+!bt
+\begin{align}
+	E(\mathbf{x}, \mathbf{h}) = - \sum_i^M x_i a_i- \sum_j^N b_j h_j - \sum_{i,j}^{M,N} x_i w_{ij} h_j,
+\end{align}
+!et
+where the binary values taken on by the nodes are most commonly 0 and 1.
+!eblock
+!bblock Gaussian-Binary RBM:
+
+Another varient is the RBM where the visible units are Gaussian while the hidden units remain binary:
+!bt
+\begin{align}
+	E(\mathbf{x}, \mathbf{h}) = \sum_i^M \frac{(x_i - a_i)^2}{2\sigma_i^2} - \sum_j^N b_j h_j - \sum_{i,j}^{M,N} \frac{x_i w_{ij} h_j}{\sigma_i^2}. 
+\end{align}
+!et
+!eblock
+
+!split
+===== More about RBMs =====
+o Useful when we model continuous data (i.e., we wish $\mathbf{x}$ to be continuous)
+o Requires a smaller learning rate, since there's no upper bound to the value a component might take in the reconstruction
+
+Other types of units include:
+ o Softmax and multinomial units
+ o Gaussian visible and hidden units
+ o Binomial units
+ o Rectified linear units
+
+
+!split
+===== Sampling: Metropolis sampling =====
+In order to sample from the RBM probability distribution it is common to use Markov Chain Monte Carlo (MCMC) algorithms such as Metropolis-Hastings or Gibbs sampling.
+
+
+
+Metropolis sampling starts by suggesting a new configuration $\bm{x}^{k+1}$. In the brute force method this is done by some random change of the visible units. The new configuration is then accepted with the acceptance probability
+!bt
+\begin{align}
+	A(\bm{x}^k \rightarrow \bm{x}^{k+1}) = \text{min} (1, \frac{P(\bm{x}^{k+1})}{P(\bm{x}^k)}),
+\end{align} 
+!et
+where we need the marginalized probability
+!bt
+\begin{align}
+	P(\bm{x})  &= \sum_\mathbf{h} P_{rbm}(\mathbf{x}, \mathbf{h}) \\
+				&= \frac{1}{Z}\sum_\mathbf{h} e^{-E(\mathbf{x}, \mathbf{h})}.
+\end{align}
+!et
+
+!split
+===== Sampling: Gibbs sampling =====
+
+In this method we sample from the joint probability $P_{rbm} (\mathbf{x}, \mathbf{h})$ by way of a two step sampling process. We alternately update the visible and hidden units.
+New samples are generated according to the conditional probabilities $P(x_i|\mathbf{h})$ and $P(h_j|\mathbf{x})$ respectively and accepted with the probability of $1$. While the the visible nodes are dependent on the hidden nodes and vice versa, the nodes are independent of other nodes within the same layer. This is due to there being no intra layer interactions in the restricted Boltzmann machine.
+
+The conditional probabilities are often referred to as the activitation functions in the neural networks context due to their role in determining the node outputs. For the binary-binary RBM they are
+!bt
+\begin{align}
+	P(h_j = 1 | \bm{x}) &= \frac{1}{1 + e^{-b_j - \sum_i x_i w_{ij}}} \\
+	P(x_i = 1 | \bm{h}) &= \frac{1}{1 + e^{-a_j - \sum_j h_j w_{ij}}},
+\end{align}
+!et
+where we recognize the logistic sigmoid function $\sigma (x) = 1/(1+exp(-x))$.
+
+!split
+===== Gaussian RBM =====
+For the Gaussian-Binary RBM the conditional probabilities are
+!bt
+\begin{align}
+	P(x_i|\mathbf{h}) &= \mathcal{N}(x_i; a_i+ \sum_j h_j w_{ij}, \sigma^2) \\
+	P(h_j=1|\mathbf{x}) &=  \frac{1}{1+e^{-b_j-\frac{1}{\sigma^2} \sum_i x_i w_{ij}}},
+\end{align}
+!et
+while the visible units now follow a normal distribution, we see the hidden units again follow the logistic sigmoid function.
+
+!split
+===== Cost function =====
+
+When working with a training dataset, the most common training approach is maximizing the log-likelihood of the training data. The log likelihood characterizes the log-probability of generating the observed data using our generative model. Using this method our cost function is chosen as the negative log-likelihood. The learning then consists of trying to find parameters that maximize the probability of the dataset, and is known as Maximum Likelihood Estimation (MLE).
+Denoting the parameters as $\bm{\theta} = a_1,...,a_M,b_1,...,b_N,w_{11},...,w_{MN}$, the log-likelihood is given by
+!bt
+\begin{align}
+	\mathcal{L}(\{ \theta_i \}) &= \langle \text{log} P_\theta(\bm{x}) \rangle_{data} \\
+	&= - \langle E(\bm{x}; \{ \theta_i\}) \rangle_{data} - \text{log} Z(\{ \theta_i\}),
+\end{align}
+!et
+where we used that the normalization constant does not depend on the data, $\langle \text{log} Z(\{ \theta_i\}) \rangle = \text{log} Z(\{ \theta_i\})$
+Our cost function is the negative log-likelihood, $\mathcal{C}(\{ \theta_i \}) = - \mathcal{L}(\{ \theta_i \})$
+
+!split
+===== Optimization / Training =====
+
+The training procedure of choice often is Stochastic Gradient Descent (SGD). It consists of a series of iterations where we update the parameters according to the equation
+!bt
+\begin{align}
+	\bm{\theta}_{k+1} = \bm{\theta}_k - \eta \nabla \mathcal{C} (\bm{\theta}_k)
+\end{align}
+!et
+at each $k$-th iteration. There are a range of variants of the algorithm which aim at making the learning rate $\eta$ more adaptive so the method might be more efficient while remaining stable.
+
+We now need the gradient of the cost function in order to minimize it. We find that
+!bt
+\begin{align}
+	\frac{\partial \mathcal{C}(\{ \theta_i\})}{\partial \theta_i}
+	&= \langle \frac{\partial E(\bm{x}; \theta_i)}{\partial \theta_i} \rangle_{data}
+	+ \frac{\partial \text{log} Z(\{ \theta_i\})}{\partial \theta_i} \\
+	&= \langle O_i(\bm{x}) \rangle_{data} - \langle O_i(\bm{x}) \rangle_{model},
+\end{align}
+!et
+where in order to simplify notation we defined the "operator"
+!bt
+\begin{align}
+	O_i(\bm{x}) = \frac{\partial E(\bm{x}; \theta_i)}{\partial \theta_i}, 
+\end{align}
+!et
+and used the statistical mechanics relationship between expectation values and the log-partition function:
+!bt
+\begin{align}
+	\langle O_i(\bm{x}) \rangle_{model} = \text{Tr} P_\theta(\bm{x})O_i(\bm{x}) = - \frac{\partial \text{log} Z(\{ \theta_i\})}{\partial \theta_i}.
+\end{align}
+!et
+
+!split 
+=====  More on RBMs =====
+
+The data-dependent term in the gradient is known as the positive phase of the gradient, while the model-dependent term is known as the negative phase of the gradient. The aim of the training is to lower the energy of configurations that are near observed data points (increasing their probability), and raising the energy of configurations that are far from observed data points (decreasing their probability).
+
+The gradient of the negative log-likelihood cost function of a Binary-Binary RBM is then
+!bt
+\begin{align}
+	\frac{\partial \mathcal{C} (w_{ij}, a_i, b_j)}{\partial w_{ij}} =& \langle x_i h_j \rangle_{data} - \langle x_i h_j \rangle_{model} \\
+	\frac{\partial \mathcal{C} (w_{ij}, a_i, b_j)}{\partial a_{ij}} =& \langle x_i \rangle_{data} - \langle x_i \rangle_{model} \\
+	\frac{\partial \mathcal{C} (w_{ij}, a_i, b_j)}{\partial b_{ij}} =& \langle h_i \rangle_{data} - \langle h_i \rangle_{model}. \\
+\end{align}
+!et
+To get the expecation values with respect to the *data*, we set the visible units to each of the observed samples in the training data, then update the hidden units according to the conditional probability found before. We then average over all samples in the training data to calculate expectation values with respect to the data. 
+
+!split
+===== Which sampling to use =====
+
+To get the expectation values with respect to the *model*, we use Gibbs sampling. We can either initialize the $\bm{x}$ randomly or with a training sample. While we ideally want a large number of Gibbs iterations $n\rightarrow n$, one might decide to truncate it earlier for efficiency. Doing this while having intialized $\bm{x}$ with a training data vector is referred to as contrastive divergence (CD), because one is then closer to approximating the gradient of this function than the negative log-likelihood. The contrastive divergence function is the difference between two Kullback-Leibler divergences (also called relative entropy), which measure how one probability distribution diverges from a second, expected probability distribution (in this case the estimated one from the ground truth one).
+
+
+!split
+===== Kullback-Leibler relative entropy =====
+
+When the goal of the training is to approximate a probability
+distribution, as it is in generative modeling, another relevant
+measure is the _Kullback-Leibler divergence_, also known as the
+relative entropy or Shannon entropy. It is a non-symmetric measure of the
+dissimilarity between two probability density functions $p$ and
+$q$. If $p$ is the unkown probability which we approximate with $q$,
+we can measure the difference by
+!bt
+\begin{align}
+	\text{KL}(p||q) = \int_{-\infty}^{\infty} p (\bm{x}) \log \frac{p(\bm{x})}{q(\bm{x})}  d\bm{x}.
+\end{align}
+!et
+
+Thus, the Kullback-Leibler divergence between the distribution of the
+training data $f(\bm{x})$ and the model distribution $p(\bm{x}|
+\bm{\theta})$ is
+
+!bt
+\begin{align}
+	\text{KL} (f(\bm{x})|| p(\bm{x}| \bm{\theta})) =& \int_{-\infty}^{\infty}
+	f (\bm{x}) \log \frac{f(\bm{x})}{p(\bm{x}| \bm{\theta})} d\bm{x} \\
+	=& \int_{-\infty}^{\infty} f(\bm{x}) \log f(\bm{x}) d\bm{x} - \int_{-\infty}^{\infty} f(\bm{x}) \log
+	p(\bm{x}| \bm{\theta}) d\bm{x} \\
+	%=& \mathbb{E}_{f(\bm{x})} (\log f(\bm{x})) - \mathbb{E}_{f(\bm{x})} (\log p(\bm{x}| \bm{\theta}))
+	=& \langle \log f(\bm{x}) \rangle_{f(\bm{x})} - \langle \log p(\bm{x}| \bm{\theta}) \rangle_{f(\bm{x})} \\
+	=& \langle \log f(\bm{x}) \rangle_{data} + \langle E(\bm{x}) \rangle_{data} + \log Z \\
+	=& \langle \log f(\bm{x}) \rangle_{data} + \mathcal{C}_{LL} .
+\end{align}
+!et
+
+The first term is constant with respect to $\bm{\theta}$ since $f(\bm{x})$ is independent of $\bm{\theta}$. Thus the Kullback-Leibler Divergence is minimal when the second term is minimal. The second term is the log-likelihood cost function, hence minimizing the Kullback-Leibler divergence is equivalent to maximizing the log-likelihood.
+
+!split
+===== Optimizing the cost function =====
+
+To further understand generative models it is useful to study the
+gradient of the cost function which is needed in order to minimize it
+using methods like stochastic gradient descent. 
+
+The partition function is the generating function of
+expectation values, in particular there are mathematical relationships
+between expectation values and the log-partition function. In this
+case we have
+!bt
+\begin{align}
+	\langle \frac{ \partial E(\bm{x}; \theta_i) } { \partial \theta_i} \rangle_{model}
+	= \int p(\bm{x}| \bm{\theta}) \frac{ \partial E(\bm{x}; \theta_i) } { \partial \theta_i} d\bm{x} 
+	= -\frac{\partial \log Z(\theta_i)}{ \partial  \theta_i} .
+\end{align}
+!et
+
+Here $\langle \cdot \rangle_{model}$ is the expectation value over the model probability distribution $p(\bm{x}| \bm{\theta})$.
+
+!split
+===== Setting up for gradient descent calculations =====
+
+Using the previous relationship we can express the gradient of the cost function as
+
+!bt
+\begin{align}
+	\frac{\partial \mathcal{C}_{LL}}{\partial \theta_i}
+	=& \langle \frac{ \partial E(\bm{x}; \theta_i) } { \partial \theta_i} \rangle_{data} + \frac{\partial \log Z(\theta_i)}{ \partial  \theta_i} \\
+	=& \langle \frac{ \partial E(\bm{x}; \theta_i) } { \partial \theta_i} \rangle_{data} - \langle \frac{ \partial E(\bm{x}; \theta_i) } { \partial \theta_i} \rangle_{model} \\
+	%=& \langle O_i(\bm{x}) \rangle_{data} - \langle O_i(\bm{x}) \rangle_{model}
+\end{align}
+!et
+
+This expression shows that the gradient of the log-likelihood cost
+function is a _difference of moments_, with one calculated from
+the data and one calculated from the model. The data-dependent term is
+called the _positive phase_ and the model-dependent term is
+called the _negative phase_ of the gradient. We see now that
+minimizing the cost function results in lowering the energy of
+configurations $\bm{x}$ near points in the training data and
+increasing the energy of configurations not observed in the training
+data. That means we increase the model's probability of configurations
+similar to those in the training data.
+
+!split
+===== More interpretations =====
+
+The gradient of the cost function also demonstrates why gradients of
+unsupervised, generative models must be computed differently from for
+those of for example FNNs. While the data-dependent expectation value
+is easily calculated based on the samples $\bm{x}_i$ in the training
+data, we must sample from the model in order to generate samples from
+which to caclulate the model-dependent term. We sample from the model
+by using MCMC-based methods. We can not sample from the model directly
+because the partition function $Z$ is generally intractable.
+
+As in supervised machine learning problems, the goal is also here to
+perform well on _unseen_ data, that is to have good
+generalization from the training data. The distribution $f(x)$ we
+approximate is not the _true_ distribution we wish to estimate,
+it is limited to the training data. Hence, in unsupervised training as
+well it is important to prevent overfitting to the training data. Thus
+it is common to add regularizers to the cost function in the same
+manner as we discussed for say linear regression.
+
+!split
+===== Recent examples: RBMs for the quantum many body problem =====
+
+The idea of applying RBMs to quantum many body problems was presented by G. Carleo and M. Troyer, working with ETH Zurich and Microsoft Research.
+
+Some of their motivation included
+
+* "The wave function $\Psi$ is a monolithic mathematical quantity that contains all the information on a quantum state, be it a single particle or a complex molecule. In principle, an exponential amount of information is needed to fully encode a generic many-body quantum state."
+* There are still interesting open problems, including fundamental questions ranging from the dynamical properties of high-dimensional systems to the exact ground-state properties of strongly interacting fermions.
+* The difficulty lies in finding a general strategy to reduce the exponential complexity of the full many-body wave function down to its most essential features. That is
+ o $\rightarrow$ Dimensional reduction
+ o $\rightarrow$ Feature extraction
+* Among the most successful techniques to attack these challenges, artifical neural networks play a prominent role.
+* Want to understand whether an artifical neural network may adapt to describe a quantum system.
+
+!split
+===== Choose the right RBM ===== 
+
+Carleo and Troyer applied the RBM to the quantum mechanical spin lattice systems of the Ising model and Heisenberg model, with encouraging results. Our goal is to test the method on systems of moving particles. For the spin lattice systems it was natural to use a binary-binary RBM, with the nodes taking values of 1 and -1. For moving particles, on the other hand, we want the visible nodes to be continuous, representing position coordinates. Thus, we start by choosing a Gaussian-binary RBM, where the visible nodes are continuous and hidden nodes take on values of 0 or 1. If eventually we would like the hidden nodes to be continuous as well the rectified linear units seem like the most relevant choice.
+
+
+
+!split
+===== Representing the wave function =====
+The wavefunction should be a probability amplitude depending on $\bm{x}$. The RBM model is given by the joint\
+ distribution of $\bm{x}$ and $\bm{h}$
+!bt
+\begin{align}
+        F_{rbm}(\mathbf{x},\mathbf{h}) = \frac{1}{Z} e^{-\frac{1}{T_0}E(\mathbf{x},\mathbf{h})}.
+\end{align}
+!et
+To find the marginal distribution of $\bm{x}$ we set:
+!bt
+\begin{align}
+        F_{rbm}(\mathbf{x}) &= \sum_\mathbf{h} F_{rbm}(\mathbf{x}, \mathbf{h}) \\
+                                &= \frac{1}{Z}\sum_\mathbf{h} e^{-E(\mathbf{x}, \mathbf{h})}.
+\end{align}
+!et
+
+Now this is what we use to represent the wave function, calling it a neural-network quantum state (NQS)
+!bt
+\begin{align}
+        \Psi (\mathbf{X}) &= F_{rbm}(\mathbf{x}) \\
+        &= \frac{1}{Z}\sum_{\bm{h}} e^{-E(\mathbf{x}, \mathbf{h})} \\
+        &= \frac{1}{Z} \sum_{\{h_j\}} e^{-\sum_i^M \frac{(x_i - a_i)^2}{2\sigma^2} + \sum_j^N b_j h_j + \sum_\
+{i,j}^{M,N} \frac{x_i w_{ij} h_j}{\sigma^2}} \\
+        &= \frac{1}{Z} e^{-\sum_i^M \frac{(x_i - a_i)^2}{2\sigma^2}} \prod_j^N (1 + e^{b_j + \sum_i^M \frac{x\
+_i w_{ij}}{\sigma^2}}). \\
+\end{align}
+!et
+
+
+!split
+===== Choose the cost function =====
+Now we don't necessarily have training data (unless we generate it by using some other method). However, what we do have is the variational principle which allows us to obtain the ground state wave function by minimizing the expectation value of the energy of a trial wavefunction (corresponding to the untrained NQS). Similarly to the traditional variational Monte Carlo method then, it is the local energy we wish to minimize. The gradient to use for the stochastic gradient descent procedure is
+!bt
+\begin{align}
+	G_i = \frac{\partial \langle E_L \rangle}{\partial \theta_i}
+	= 2(\langle E_L \frac{1}{\Psi}\frac{\partial \Psi}{\partial \theta_i} \rangle - \langle E_L \rangle \langle \frac{1}{\Psi}\frac{\partial \Psi}{\partial \theta_i} \rangle ),
+\end{align}
+!et
+where the local energy is given by
+!bt
+\begin{align}
+	E_L = \frac{1}{\Psi} \hat{\mathbf{H}} \Psi.
+\end{align}
+!et
+
+
+!split
+===== Running the codes =====
+!bblock
+You can find the codes for the simple two-electron case at the Github repository URL:"https://github.com/mhjensenseminars/MachineLearningTalk/tree/master/doc/Programs/MLcpp/src". Python codes to come, only c++ as of now. 
+
+The trial wave function is based on the product of a Slater determinant with Gaussian orbitals, a simple Jastrow factor $\exp{(r_{ij})}$ and the reduced Boltzmann machines. 
+
+The Broyden-Fletcher-Goldfarb-Shanno algorithm was used to perform the minimization. We used $14$ hidden nodes in the calculations below.
+
+!eblock
+
+
+
+
+!split
+===== Energy as function of iterations, $N=2$ electrons  =====
+!bblock
+FIGURE: [figures/figN2.pdf, width=700 frac=0.9]
+!eblock
+
+
diff --git a/doc/src/BoltzmannMachines/figures/RMBenergy.dat~ b/doc/src/BoltzmannMachines/figures/RMBenergy.dat~
new file mode 100644
index 000000000..f85552e62
--- /dev/null
+++ b/doc/src/BoltzmannMachines/figures/RMBenergy.dat~
@@ -0,0 +1,100 @@
+5.59097
+5.38551
+5.14253
+4.92049
+4.80505
+4.62185
+4.48617
+4.40662
+4.2445
+4.17527
+4.09137
+4.0624
+3.96858
+3.92182
+3.85881
+3.82252
+3.76908
+3.73743
+3.69007
+3.62217
+3.63773
+3.58399
+3.57401
+3.53176
+3.51544
+3.48965
+3.4685
+3.45745
+3.44185
+3.43143
+3.40337
+3.38956
+3.37394
+3.35488
+3.29924
+3.2945
+3.2913
+3.29587
+3.28508
+3.27126
+3.25836
+3.24323
+3.24687
+3.21526
+3.2367
+3.23167
+3.21595
+3.21003
+3.18076
+3.18035
+3.18534
+3.15874
+3.17502
+3.14463
+3.17313
+3.16107
+3.13238
+3.1367
+3.15066
+3.11037
+3.11367
+3.10958
+3.10312
+3.09725
+3.09705
+3.09602
+3.08657
+3.09206
+3.0703
+3.07878
+3.10305
+3.06263
+3.06851
+3.09004
+3.0673
+3.05662
+3.047
+3.04692
+3.06387
+3.06418
+3.06383
+3.04044
+3.06274
+3.04205
+3.04167
+3.04238
+3.04487
+3.02483
+3.05472
+3.02654
+3.03249
+3.04819
+3.03152
+3.03445
+3.03407
+3.02488
+3.00511
+3.01687
+3.03151
+3.03374
\ No newline at end of file
diff --git a/doc/src/BoltzmannMachines/figures/plot.py~ b/doc/src/BoltzmannMachines/figures/plot.py~
new file mode 100644
index 000000000..bf33e2968
--- /dev/null
+++ b/doc/src/BoltzmannMachines/figures/plot.py~
@@ -0,0 +1,14 @@
+import numpy as np
+import matplotlib.pyplot as plt
+from IPython.display import display
+
+data = np.loadtxt('RMBenergy.dat')
+x = data[:,0]
+y = data[:,1]
+plt.plot(x, y,'ro')
+plt.axis([0,101,3, 7])
+plt.xlabel(r'Iterations')
+plt.ylabel(r'Energy')
+plt.savefig('MLrbm.pdf')
+plt.show()
+
diff --git a/doc/src/BoltzmannMachines/figures/plotEnergies.py~ b/doc/src/BoltzmannMachines/figures/plotEnergies.py~
new file mode 100644
index 000000000..394649f06
--- /dev/null
+++ b/doc/src/BoltzmannMachines/figures/plotEnergies.py~
@@ -0,0 +1,62 @@
+import sys
+
+import numpy as np
+import matplotlib.pyplot as plt
+
+from mpl_toolkits.axes_grid1.inset_locator import inset_axes
+
+try:
+    dataFileName = sys.argv[1]
+except IndexError:
+    print("USAGE: python plotEnergies.py 'filename'")
+    sys.exit(0)
+
+HFEnergy3 = 3.161921401722216
+HFEnergy6 = 20.71924844033019
+
+numParticles = \
+        int(dataFileName[dataFileName.find('N')+1:dataFileName.find('E')-1])
+hfenergyFound = False 
+if (numParticles == 2):
+    HFEnergy = 3.161921401722216
+    hfenergyFound = True 
+elif (numParticles == 6):
+    HFEnergy = 20.71924844033019
+    hfenergyFound = True 
+else:
+    hfenergyFound = False
+
+data = np.loadtxt(dataFileName, dtype=np.float64)
+data[:,1] = np.sqrt(data[:,1])
+
+n = len(data[:,0])
+x = np.arange(0,n)
+
+fig = plt.figure()
+
+if (hfenergyFound):
+    yline = np.zeros(n)
+    yline.fill(HFEnergy)
+    plt.plot(x, yline, 'r--', label="HF Energy")
+
+msize = 1.0
+
+ax = fig.add_subplot(111)
+plt.errorbar(x, data[:,0], yerr=data[:,1],  fmt='bo', markersize=msize, label="VMC Energy")
+plt.fill_between(x, data[:,0]-data[:,1], data[:,0]+data[:,1])
+
+plt.xlim(0,n)
+plt.xlabel('Iteration')
+plt.ylabel('$E_0[a.u]$')
+plt.legend(loc='best')
+
+minSub = 80
+maxSub = 120
+inset_axes(ax, width="50%", height=1.0, loc='right')
+plt.errorbar(x[minSub:maxSub], data[minSub:maxSub,0],
+        yerr=data[minSub:maxSub,1],  fmt='bo', markersize=msize, label="VMC "
+        "Energy")
+plt.plot(x[minSub:maxSub], yline[minSub:maxSub], 'r--', label="HF Energy")
+
+plt.show()
+plt.save(rbm.pdf)
diff --git a/doc/src/BoltzmannMachines/src/Hudson_Bay.py~ b/doc/src/BoltzmannMachines/src/Hudson_Bay.py~
new file mode 100644
index 000000000..acd44518e
--- /dev/null
+++ b/doc/src/BoltzmannMachines/src/Hudson_Bay.py~
@@ -0,0 +1,43 @@
+import numpy as np
+import matplotlib.pyplot as plt
+
+def solver(m, H0, L0, dt, a, b, c, d, t0):
+    """Solve the difference equations for H and L over m years
+    with time step dt (measured in years."""
+
+    num_intervals = int(m/float(dt))
+    t = np.linspace(t0, t0 + m, num_intervals+1)
+    H = np.zeros(t.size)
+    L = np.zeros(t.size)
+
+    print 'Init:', H0, L0, dt
+    H[0] = H0
+    L[0] = L0
+
+    for n in range(0, len(t)-1):
+        H[n+1] = H[n] + a*dt*H[n] - b*dt*H[n]*L[n]
+        L[n+1] = L[n] + d*dt*H[n]*L[n] - c*dt*L[n]
+    return H, L, t
+
+# Load in data file
+data = np.loadtxt('Hudson_Bay.csv', delimiter=',', skiprows=1)
+# Make arrays containing x-axis and hares and lynx populations
+t_e = data[:,0]
+H_e = data[:,1]
+L_e = data[:,2]
+
+# Simulate using the model
+H, L, t = solver(m=20, H0=34.91, L0=3.857, dt=0.1,
+                 a=0.4807, b=0.02482, c=0.9272, d=0.02756,
+                 t0=1900)
+
+# Visualize simulations and data
+plt.plot(t_e, H_e, 'b-+', t_e, L_e, 'r-o', t, H, 'm--', t, L, 'k--')
+plt.xlabel('Year')
+plt.ylabel('Numbers of hares and lynx')
+plt.axis([1900, 1920, 0, 140])
+plt.title(r'Population of hares and lynx 1900-1920 (x1000)')
+plt.legend(('H_e', 'L_e', 'H', 'L'), loc='upper left')
+plt.savefig('Hudson_Bay_sim.pdf')
+plt.savefig('Hudson_Bay_sim.png')
+plt.show()
diff --git a/doc/src/How2ReadData/How2ReadData.do.txt b/doc/src/How2ReadData/How2ReadData.do.txt
index 29d61b965..12541a443 100644
--- a/doc/src/How2ReadData/How2ReadData.do.txt
+++ b/doc/src/How2ReadData/How2ReadData.do.txt
@@ -4,6 +4,7 @@ DATE: today
 
 
 
+
 ===== Introduction =====
 
 Our emphasis throughout this series of lectures  
@@ -18,47 +19,148 @@ and use these to introduce statistical data analysis
 concepts and machine learning algorithms before we delve into the
 algorithms themselves. The examples we will use in the beginning, start with simple
 polynomials with random noise added. We will use the Python
-software package "Scikit-learn":"http://scikit-learn.org/stable/" and
+software package "Scikit-Learn":"http://scikit-learn.org/stable/" and
 introduce various machine learning algorithms to make fits of
 the data and predictions. We move thereafter to more interesting
-cases such as the simulation of financial transactions or disease
-models. These are examples where we can easily set up the data and
+cases such as nuclear binding energies.
+These are examples where we can easily set up the data and
 then use machine learning algorithms included in for example
-_scikit-learn_. 
+_Scikit-Learn_. 
 
 These examples will serve us the purpose of getting
 started. Furthermore, they allow us to catch more than two birds with
 a stone. They will allow us to bring in some programming specific
-topics and tools as well as showing the power of various Python (and
-R) packages for machine learning and statistical data analysis. In the
-lectures on linear algebra we cover in more detail various programming
-features of languages like Python and C++ (and other), we will also
-look into more specific linear functions which are relevant for the
-various algorithms we will discuss. Here, we will mainly focus on two
-specific Python packages for Machine Learning, scikit-learn and
-tensorflow (see below for links etc).  Moreover, the examples we
+topics and tools as well as showing the power of various Python 
+packages for machine learning and statistical data analysis.  
+
+Here, we will mainly focus on two
+specific Python packages for Machine Learning, Scikit-Learn and
+Tensorflow (see below for links etc).  Moreover, the examples we
 introduce will serve as inputs to many of our discussions later, as
 well as allowing you to set up models and produce your own data and
 get started with programming.
 
 
 
+===== What is Machine Learning?  =====
+
+Statistics, data science and machine learning form important fields of
+research in modern science.  They describe how to learn and make
+predictions from data, as well as allowing us to extract important
+correlations about physical process and the underlying laws of motion
+in large data sets. The latter, big data sets, appear frequently in
+essentially all disciplines, from the traditional Science, Technology,
+Mathematics and Engineering fields to Life Science, Law, education
+research, the Humanities and the Social Sciences. 
+
+It has become more
+and more common to see research projects on big data in for example
+the Social Sciences where extracting patterns from complicated survey
+data is one of many research directions.  Having a solid grasp of data
+analysis and machine learning is thus becoming central to scientific
+computing in many fields, and competences and skills within the fields
+of machine learning and scientific computing are nowadays strongly
+requested by many potential employers. The latter cannot be
+overstated, familiarity with machine learning has almost become a
+prerequisite for many of the most exciting employment opportunities,
+whether they are in bioinformatics, life science, physics or finance,
+in the private or the public sector. This author has had several
+students or met students who have been hired recently based on their
+skills and competences in scientific computing and data science, often
+with marginal knowledge of machine learning.
+
+Machine learning is a subfield of computer science, and is closely
+related to computational statistics.  It evolved from the study of
+pattern recognition in artificial intelligence (AI) research, and has
+made contributions to AI tasks like computer vision, natural language
+processing and speech recognition. Many of the methods we will study are also 
+strongly rooted in basic mathematics and physics research. 
+
+Ideally, machine learning represents the science of giving computers
+the ability to learn without being explicitly programmed.  The idea is
+that there exist generic algorithms which can be used to find patterns
+in a broad class of data sets without having to write code
+specifically for each problem. The algorithm will build its own logic
+based on the data.  You should however always keep in mind that
+machines and algorithms are to a large extent developed by humans. The
+insights and knowledge we have about a specific system, play a central
+role when we develop a specific machine learning algorithm. 
+
+Machine learning is an extremely rich field, in spite of its young
+age. The increases we have seen during the last three decades in
+computational capabilities have been followed by developments of
+methods and techniques for analyzing and handling large date sets,
+relying heavily on statistics, computer science and mathematics.  The
+field is rather new and developing rapidly. Popular software packages
+written in Python for machine learning like
+"Scikit-learn":"http://scikit-learn.org/stable/",
+"Tensorflow":"https://www.tensorflow.org/",
+"PyTorch":"http://pytorch.org/" and "Keras":"https://keras.io/", all
+freely available at their respective GitHub sites, encompass
+communities of developers in the thousands or more. And the number of
+code developers and contributors keeps increasing. Not all the
+algorithms and methods can be given a rigorous mathematical
+justification, opening up thereby large rooms for experimenting and
+trial and error and thereby exciting new developments.  However, a
+solid command of linear algebra, multivariate theory, probability
+theory, statistical data analysis, understanding errors and Monte
+Carlo methods are central elements in a proper understanding of many
+of algorithms and methods we will discuss.
+
+
+
+===== Types of Machine Learning =====
+
+
+The approaches to machine learning are many, but are often split into
+two main categories.  In *supervised learning* we know the answer to a
+problem, and let the computer deduce the logic behind it. On the other
+hand, *unsupervised learning* is a method for finding patterns and
+relationship in data sets without any prior knowledge of the system.
+Some authours also operate with a third category, namely
+*reinforcement learning*. This is a paradigm of learning inspired by
+behavioral psychology, where learning is achieved by trial-and-error,
+solely from rewards and punishment.
+
+Another way to categorize machine learning tasks is to consider the
+desired output of a system.  Some of the most common tasks are:
+
+  * Classification: Outputs are divided into two or more classes. The goal is to   produce a model that assigns inputs into one of these classes. An example is to identify  digits based on pictures of hand-written ones. Classification is typically supervised learning.
+
+  * Regression: Finding a functional relationship between an input data set and a reference data set.   The goal is to construct a function that maps input data to continuous output values.
+
+  * Clustering: Data are divided into groups with certain common traits, without knowing the different groups beforehand.  It is thus a form of unsupervised learning.
+
+
+The methods we cover have three main topics in common, irrespective of
+whether we deal with supervised or unsupervised learning. The first
+ingredient is normally our data set (which can be subdivided into
+training and test data), the second item is a model which is normally a
+function of some parameters.  The model reflects our knowledge of the system (or lack thereof). As an example, if we know that our data show a behavior similar to what would be predicted by a polynomial, fitting our data to a polynomial of some degree would then determin our model. 
+
+The last ingredient is a so-called _cost_
+function which allows us to present an estimate on how good our model
+is in reproducing the data it is supposed to train.  
+At the heart of basically all ML algorithms there are so-called minimization algorithms, often we end up with various variants of _gradient_ methods.
+
+
+
+
+
 
 
 ===== Software and needed installations =====
 
 We will make extensive use of Python as programming language and its
 myriad of available libraries.  You will find
-IPython/Jupyter notebooks invaluable in your work.  You can run _R_
+Jupyter notebooks invaluable in your work.  You can run _R_
 codes in the Jupyter/IPython notebooks, with the immediate benefit of
 visualizing your data. You can also use compiled languages like C++,
-Rust, Fortran etc if you prefer. The focus in these lectures will be
-on Python, but we will provide many code examples for those of you who
-prefer R or compiled languages. You can integrate C++ codes and R in for example
-a Jupyter notebook. 
+Rust, Julia, Fortran etc if you prefer. The focus in these lectures will be
+on Python.
 
 
-If you have Python installed (we recommend Python3) and you feel
+If you have Python installed (we strongly recommend Python3) and you feel
 pretty familiar with installing different packages, we recommend that
 you install the following Python packages via _pip_ as 
 
@@ -103,24 +205,36 @@ distribution for scientific and analytic computing distribution and
 analysis environment, available for free and under a commercial
 license.
 
-===== Useful Python packages =====
-Here we list several useful Python packages.
+Furthermore, "Google's Colab":"https://colab.research.google.com/notebooks/welcome.ipynb" is a free Jupyter notebook environment that requires 
+no setup and runs entirely in the cloud. Try it out!
 
-And more text to come in order to get started with Python
+===== Useful Python libraries =====
+Here we list several useful Python libraries we strongly recommend (if you use anaconda many of these are already there)
 
+* "NumPy":"https://www.numpy.org/" is a highly popular library for large, multi-dimensional arrays and matrices, along with a large collection of high-level mathematical functions to operate on these arrays
+* "The pandas":"https://pandas.pydata.org/" library provides high-performance, easy-to-use data structures and data analysis tools 
+* "Xarray":"http://xarray.pydata.org/en/stable/" is a Python package that makes working with labelled multi-dimensional arrays simple, efficient, and fun!
+* "Scipy":"https://www.scipy.org/" (pronounced “Sigh Pie”) is a Python-based ecosystem of open-source software for mathematics, science, and engineering. 
+* "Matplotlib":"https://matplotlib.org/" is a Python 2D plotting library which produces publication quality figures in a variety of hardcopy formats and interactive environments across platforms.
+* "Autograd":"https://github.com/HIPS/autograd" can automatically differentiate native Python and Numpy code. It can handle a large subset of Python's features, including loops, ifs, recursion and closures, and it can even take derivatives of derivatives of derivatives
+* "SymPy":"https://www.sympy.org/en/index.html" is a Python library for symbolic mathematics. 
+* "scikit-learn":"https://scikit-learn.org/stable/" has simple and efficient tools for machine learning, data mining and data analysis
+* "TensorFlow":"https://www.tensorflow.org/" is a Python library for fast numerical computing created and released by Google
+* "Keras":"https://keras.io/" is a high-level neural networks API, written in Python and capable of running on top of TensorFlow, CNTK, or Theano
+* And many more such as "pytorch":"https://pytorch.org/",  "Theano":"https://pypi.org/project/Theano/" etc 
 
 ===== Installing R, C++, cython or Julia =====
 
-You will also find it convenient to utilize R. Although we will mainly
-use Python during lectures and in various projects and exercises, we
-provide a full R set of codes for the same examples. Those of you
-already familiar with R should feel free to continue using R, keeping
+You will also find it convenient to utilize _R_. We will mainly
+use Python during lectures and in various projects and exercises.
+Those of you
+already familiar with _R_ should feel free to continue using _R_, keeping
 however an eye on the parallel Python set ups. Similarly, if you are a
-Python afecionado, feel free to explore R as well.  Jupyter/Ipython
+Python afecionado, feel free to explore _R_ as well.  Jupyter/Ipython
 notebook allows you to run _R_ codes interactively in your
 browser. The software library _R_ is tuned to statistically analysis
 and allows for an easy usage of the tools we will discuss in these
-texts.
+lectures.
 
 To install _R_ with Jupyter notebook 
 "follow the link here":"https://mpacer.org/maths/r-kernel-for-ipython-notebook"
@@ -138,13 +252,13 @@ yourself, you can thus opt for either Python or C++ (or Fortran or other compile
 languages.
 
 To add more entropy, _cython_ can also be used when running your
-notebooks. It means that Python with the Jupyter/IPython notebook
+notebooks. It means that Python with the jupyter notebook
 setup allows you to integrate widely popular softwares and tools for
 scientific computing. Similarly, the 
 "Numba Python package":"https://numba.pydata.org/" delivers increased performance
 capabilities with minimal rewrites of your codes.  With its
 versatility, including symbolic operations, Python offers a unique
-computational environment. Your Jupyter/IPython notebook can easily be
+computational environment. Your jupyter notebook can easily be
 converted into a nicely rendered _PDF_ file or a Latex file for
 further processing. For example, convert to latex as 
 
@@ -156,13 +270,437 @@ And to add more versatility, the Python package "SymPy":"http://www.sympy.org/en
 
 Finally, if you wish to use the light mark-up language 
 "doconce":"https://github.com/hplgit/doconce" you can convert a standard ascii text file into various HTML 
-formats, ipython notebooks, latex files, pdf files etc with minimal edits.
+formats, ipython notebooks, latex files, pdf files etc with minimal edits. These lectures were generated using _doconce_.
 
 
-===== Simple linear regression model using _scikit-learn_ =====
 
-We start with perhaps our simplest possible example, using _scikit-learn_ to perform linear regression analysis on a data set produced by us. 
-What follows is a simple Python code where we have defined  function $y$ in terms of the variable $x$. Both are defined as vectors of dimension $1\times 100$. The entries to the vector $\hat{x}$  are given by random numbers generated with a uniform distribution with entries $x_i \in [0,1]$ (more about probability distribution functions later). These values are then used to define a function $y(x)$ (tabulated again as a vector) with a linear dependence on $x$ plus a random noise added via the normal distribution.
+===== Numpy examples and Important Matrix and vector handling packages =====
+
+There are several central software packages for linear algebra and eigenvalue problems. Several of the more
+popular ones have been wrapped into ofter software packages like those from the widely used text _Numerical Recipes_. The original source codes in many of the available packages are often taken from the widely used
+software package LAPACK, which follows two other popular packages
+developed in the 1970s, namely EISPACK and LINPACK.  We describe them shortly here.
+
+  * LINPACK: package for linear equations and least square problems.
+  * LAPACK:package for solving symmetric, unsymmetric and generalized eigenvalue problems. From LAPACK's website URL: "http://www.netlib.org" it is possible to download for free all source codes from this library. Both C/C++ and Fortran versions are available.
+  * BLAS (I, II and III): (Basic Linear Algebra Subprograms) are routines that provide standard building blocks for performing basic vector and matrix operations. Blas I is vector operations, II vector-matrix operations and III matrix-matrix operations. Highly parallelized and efficient codes, all available for download from URL: "http://www.netlib.org".
+
+
+===== Basic Matrix Features =====
+
+!bblock Matrix properties reminder
+!bt
+\[
+ \mathbf{A} =
+      \begin{bmatrix} a_{11} & a_{12} & a_{13} & a_{14} \\
+                                 a_{21} & a_{22} & a_{23} & a_{24} \\
+                                   a_{31} & a_{32} & a_{33} & a_{34} \\
+                                  a_{41} & a_{42} & a_{43} & a_{44}
+             \end{bmatrix}\qquad
+\mathbf{I} =
+      \begin{bmatrix} 1 & 0 & 0 & 0 \\
+                                 0 & 1 & 0 & 0 \\
+                                 0 & 0 & 1 & 0 \\
+                                 0 & 0 & 0 & 1
+             \end{bmatrix}
+\]
+!et
+!eblock
+
+===== Basic Matrix Features =====
+!bblock
+
+The inverse of a matrix is defined by
+
+!bt
+\[
+\mathbf{A}^{-1} \cdot \mathbf{A} = I
+\]
+!et
+!eblock
+
+
+===== Basic Matrix Features =====
+
+!bblock Matrix Properties Reminder
+
+|----------------------------------------------------------------------|
+|       Relations      |       Name      | matrix elements             |
+|----------------------------------------------------------------------|
+| $A = A^{T}$         | symmetric       | $a_{ij} = a_{ji}$            |
+| $A = \left (A^{T} \right )^{-1}$ | real orthogonal | $\sum_k a_{ik} a_{jk} = \sum_k a_{ki} a_{kj} = \delta_{ij}$ |
+| $A = A^{ * }$          | real matrix     | $a_{ij} = a_{ij}^{ * }$       |
+| $A = A^{\dagger}$     |  hermitian      | $a_{ij} = a_{ji}^{ * }$       |
+| $A = \left (A^{\dagger} \right )^{-1}$ | unitary | $\sum_k a_{ik} a_{jk}^{ * } = \sum_k a_{ki}^{ * } a_{kj} = \delta_{ij}$ |
+|----------------------------------------------------------------------|
+
+!eblock
+
+
+===== Some famous Matrices =====
+
+  * Diagonal if $a_{ij}=0$ for $i\ne j$
+  * Upper triangular if $a_{ij}=0$ for $i > j$
+  * Lower triangular if $a_{ij}=0$ for $i < j$
+  * Upper Hessenberg if $a_{ij}=0$ for $i > j+1$
+  * Lower Hessenberg if $a_{ij}=0$ for $i < j+1$
+  * Tridiagonal if $a_{ij}=0$ for $|i -j| > 1$
+  * Lower banded with bandwidth $p$: $a_{ij}=0$ for $i > j+p$
+  * Upper banded with bandwidth $p$: $a_{ij}=0$ for $i < j+p$
+  * Banded, block upper triangular, block lower triangular....
+
+
+===== Basic Matrix Features =====
+
+!bblock Some Equivalent Statements
+For an $N\times N$ matrix  $\mathbf{A}$ the following properties are all equivalent
+
+  * If the inverse of $\mathbf{A}$ exists, $\mathbf{A}$ is nonsingular.
+  * The equation $\mathbf{Ax}=0$ implies $\mathbf{x}=0$.
+  * The rows of $\mathbf{A}$ form a basis of $R^N$.
+  * The columns of $\mathbf{A}$ form a basis of $R^N$.
+  * $\mathbf{A}$ is a product of elementary matrices.
+  * $0$ is not eigenvalue of $\mathbf{A}$.
+!eblock
+
+
+===== Numpy and arrays =====
+"Numpy":"http://www.numpy.org/" provides an easy way to handle arrays in Python. The standard way to import this library is as
+
+!bc pycod
+import numpy as np
+!ec
+Here follows a simple example where we set up an array of ten elements, all determined by random numbers drawn according to the normal distribution,
+!bc pycod
+n = 10
+x = np.random.normal(size=n)
+print(x)
+!ec
+We defined a vector $x$ with $n=10$ elements with its values given by the Normal distribution $N(0,1)$.
+Another alternative is to declare a vector as follows
+!bc pycod
+import numpy as np
+x = np.array([1, 2, 3])
+print(x)
+!ec
+Here we have defined a vector with three elements, with $x_0=1$, $x_1=2$ and $x_2=3$. Note that both Python and C++
+start numbering array elements from $0$ and on. This means that a vector with $n$ elements has a sequence of entities $x_0, x_1, x_2, \dots, x_{n-1}$. We could also let (recommended) Numpy to compute the logarithms of a specific array as
+!bc pycod
+import numpy as np
+x = np.log(np.array([4, 7, 8]))
+print(x)
+!ec
+
+In the last example we used Numpy's unary function $np.log$. This function is
+highly tuned to compute array elements since the code is vectorized
+and does not require looping. We normaly recommend that you use the
+Numpy intrinsic functions instead of the corresponding _log_ function
+from Python's _math_ module. The looping is done explicitely by the
+_np.log_ function. The alternative, and slower way to compute the
+logarithms of a vector would be to write
+
+!bc pycod
+import numpy as np
+from math import log
+x = np.array([4, 7, 8])
+for i in range(0, len(x)):
+    x[i] = log(x[i])
+print(x)
+!ec
+We note that our code is much longer already and we need to import the _log_ function from the _math_ module. 
+The attentive reader will also notice that the output is $[1, 1, 2]$. Python interprets automagically our numbers as integers (like the _automatic_ keyword in C++). To change this we could define our array elements to be double precision numbers as
+!bc pycod
+import numpy as np
+x = np.log(np.array([4, 7, 8], dtype = np.float64))
+print(x)
+!ec
+or simply write them as double precision numbers (Python uses 64 bits as default for floating point type variables), that is
+!bc pycod
+import numpy as np
+x = np.log(np.array([4.0, 7.0, 8.0])
+print(x)
+!ec
+To check the number of bytes (remember that one byte contains eight bits for double precision variables), you can use simple use the _itemsize_ functionality (the array $x$ is actually an object which inherits the functionalities defined in Numpy) as 
+!bc pycod
+import numpy as np
+x = np.log(np.array([4.0, 7.0, 8.0])
+print(x.itemsize)
+!ec
+
+
+===== Matrices in Python =====
+
+Having defined vectors, we are now ready to try out matrices. We can
+define a $3 \times 3 $ real matrix $\hat{A}$ as (recall that we user
+lowercase letters for vectors and uppercase letters for matrices)
+
+!bc pycod
+import numpy as np
+A = np.log(np.array([ [4.0, 7.0, 8.0], [3.0, 10.0, 11.0], [4.0, 5.0, 7.0] ]))
+print(A)
+!ec
+If we use the _shape_ function we would get $(3, 3)$ as output, that is verifying that our matrix is a $3\times 3$ matrix. We can slice the matrix and print for example the first column (Python organized matrix elements in a row-major order, see below) as
+!bc pycod
+import numpy as np
+A = np.log(np.array([ [4.0, 7.0, 8.0], [3.0, 10.0, 11.0], [4.0, 5.0, 7.0] ]))
+# print the first column, row-major order and elements start with 0
+print(A[:,0]) 
+!ec
+We can continue this was by printing out other columns or rows. The example here prints out the second column
+!bc pycod
+import numpy as np
+A = np.log(np.array([ [4.0, 7.0, 8.0], [3.0, 10.0, 11.0], [4.0, 5.0, 7.0] ]))
+# print the first column, row-major order and elements start with 0
+print(A[1,:]) 
+!ec 
+Numpy contains many other functionalities that allow us to slice, subdivide etc etc arrays. We strongly recommend that you look up the "Numpy website for more details":"http://www.numpy.org/". Useful functions when defining a matrix are the _np.zeros_ function which declares a matrix of a given dimension and sets all elements to zero
+!bc pycod
+import numpy as np
+n = 10
+# define a matrix of dimension 10 x 10 and set all elements to zero
+A = np.zeros( (n, n) )
+print(A) 
+!ec 
+or initializing all elements to 
+!bc pycod
+import numpy as np
+n = 10
+# define a matrix of dimension 10 x 10 and set all elements to one
+A = np.ones( (n, n) )
+print(A) 
+!ec 
+or as unitarily distributed random numbers (see the material on random number generators in the statistics part)
+!bc pycod
+import numpy as np
+n = 10
+# define a matrix of dimension 10 x 10 and set all elements to random numbers with x \in [0, 1]
+A = np.random.rand(n, n)
+print(A) 
+!ec 
+
+As we will see throughout these lectures, there are several extremely useful functionalities in Numpy.
+As an example, consider the discussion of the covariance matrix. Suppose we have defined three vectors
+$\hat{x}, \hat{y}, \hat{z}$ with $n$ elements each. The covariance matrix is defined as 
+!bt
+\[
+\hat{\Sigma} = \begin{bmatrix} \sigma_{xx} & \sigma_{xy} & \sigma_{xz} \\
+                              \sigma_{yx} & \sigma_{yy} & \sigma_{yz} \\
+                              \sigma_{zx} & \sigma_{zy} & \sigma_{zz} 
+             \end{bmatrix},
+\]
+!et
+where for example
+!bt
+\[
+\sigma_{xy} =\frac{1}{n} \sum_{i=0}^{n-1}(x_i- \overline{x})(y_i- \overline{y}).
+\]
+!et
+The Numpy function _np.cov_ calculates the covariance elements using the factor $1/(n-1)$ instead of $1/n$ since it assumes we do not have the exact mean values. 
+The following simple function uses the _np.vstack_ function which takes each vector of dimension $1\times n$ and produces a $3\times n$ matrix $\hat{W}$
+!bt
+\[
+\hat{W} = \begin{bmatrix} x_0 & y_0 & z_0 \\
+                          x_1 & y_1 & z_1 \\
+                          x_2 & y_2 & z_2 \\
+                          \dots & \dots & \dots \\
+                          x_{n-2} & y_{n-2} & z_{n-2} \\
+                          x_{n-1} & y_{n-1} & z_{n-1}
+             \end{bmatrix},
+\]
+!et
+
+which in turn is converted into into the $3\times 3$ covariance matrix
+$\hat{\Sigma}$ via the Numpy function _np.cov()_. We note that we can also calculate
+the mean value of each set of samples $\hat{x}$ etc using the Numpy
+function _np.mean(x)_. We can also extract the eigenvalues of the
+covariance matrix through the _np.linalg.eig()_ function.
+
+!bc pycod
+# Importing various packages
+import numpy as np
+
+n = 100
+x = np.random.normal(size=n)
+print(np.mean(x))
+y = 4+3*x+np.random.normal(size=n)
+print(np.mean(y))
+z = x**3+np.random.normal(size=n)
+print(np.mean(z))
+W = np.vstack((x, y, z))
+Sigma = np.cov(W)
+print(Sigma)
+Eigvals, Eigvecs = np.linalg.eig(Sigma)
+print(Eigvals)
+!ec
+
+
+!bc pycod
+import numpy as np
+import matplotlib.pyplot as plt
+from scipy import sparse
+eye = np.eye(4)
+print(eye)
+sparse_mtx = sparse.csr_matrix(eye)
+print(sparse_mtx)
+x = np.linspace(-10,10,100)
+y = np.sin(x)
+plt.plot(x,y,marker='x')
+plt.show()
+!ec
+
+
+===== Meet the Pandas =====
+
+
+FIGURE: [fig/pandas.jpg, width=600 frac=0.8]
+
+Another useful Python package is
+"pandas":"https://pandas.pydata.org/", which is an open source library
+providing high-performance, easy-to-use data structures and data
+analysis tools for Python. _pandas_ stands for panel data, a term borrowed from econometrics and is an efficient library for data analysis with an emphasis on tabular data.
+_pandas_ has two major classes, the _DataFrame_ class with two-dimensional data objects and tabular data organized in columns and the class _Series_ with a focus on one-dimensional data objects. Both classes allow you to index data easily as we will see in the examples below. 
+_pandas_ allows you also to perform mathematical operations on the data, spanning from simple reshapings of vectors and matrices to statistical operations. 
+
+The following simple example shows how we can, in an easy way make tables of our data. Here we define a data set which includes names, place of birth and date of birth, and displays the data in an easy to read way. We will see repeated use of _pandas_, in particular in connection with classification of data. 
+
+!bc pycod 
+import pandas as pd
+from IPython.display import display
+data = {'First Name': ["Frodo", "Bilbo", "Aragorn II", "Samwise"],
+        'Last Name': ["Baggins", "Baggins","Elessar","Gamgee"],
+        'Place of birth': ["Shire", "Shire", "Eriador", "Shire"],
+        'Date of Birth T.A.': [2968, 2890, 2931, 2980]
+        }
+data_pandas = pd.DataFrame(data)
+display(data_pandas)
+!ec
+
+In the above we have imported _pandas_ with the shorthand _pd_, the latter has become the standard way we import _pandas_. We make then a list of various variables
+and reorganize the aboves lists into a _DataFrame_ and then print out  a neat table with specific column labels as *Name*, *place of birth* and *date of birth*.
+Displaying these results, we see that the indices are given by the default numbers from zero to three.
+_pandas_ is extremely flexible and we can easily change the above indices by defining a new type of indexing as
+!bc pycod
+data_pandas = pd.DataFrame(data,index=['Frodo','Bilbo','Aragorn','Sam'])
+display(data_pandas)
+!ec
+Thereafter we display the content of the row which begins with the index _Aragorn_
+!bc pycod
+display(data_pandas.loc['Aragorn'])
+!ec
+
+We can easily append data to this, for example
+!bc pycod
+new_hobbit = {'First Name': ["Peregrin"],
+              'Last Name': ["Took"],
+              'Place of birth': ["Shire"],
+              'Date of Birth T.A.': [2990]
+              }
+data_pandas=data_pandas.append(pd.DataFrame(new_hobbit, index=['Pippin']))
+display(data_pandas)
+!ec
+
+
+Here are other examples where we use the _DataFrame_ functionality to handle arrays, now with more interesting features for us, namely numbers. We set up a matrix 
+of dimensionality $10\times 5$ and compute the mean value and standard deviation of each column. Similarly, we can perform mathematial operations like squaring the matrix elements and many other operations. 
+!bc pycod
+import numpy as np
+import pandas as pd
+from IPython.display import display
+np.random.seed(100)
+# setting up a 10 x 5 matrix
+rows = 10
+cols = 5
+a = np.random.randn(rows,cols)
+df = pd.DataFrame(a)
+display(df)
+print(df.mean())
+print(df.std())
+display(df**2)
+!ec
+
+Thereafter we can select specific columns only and plot final results
+!bc pycod
+df.columns = ['First', 'Second', 'Third', 'Fourth', 'Fifth']
+df.index = np.arange(10)
+
+display(df)
+print(df['Second'].mean() )
+print(df.info())
+print(df.describe())
+
+from pylab import plt, mpl
+plt.style.use('seaborn')
+mpl.rcParams['font.family'] = 'serif'
+
+df.cumsum().plot(lw=2.0, figsize=(10,6))
+plt.show()
+
+
+df.plot.bar(figsize=(10,6), rot=15)
+plt.show()
+!ec
+We can produce a $4\times 4$ matrix
+!bc pycod
+b = np.arange(16).reshape((4,4))
+print(b)
+df1 = pd.DataFrame(b)
+print(df1)
+!ec
+and many other operations. 
+
+The _Series_ class is another important class included in
+_pandas_. You can view it as a specialization of _DataFrame_ but where
+we have just a single column of data. It shares many of the same features as _DataFrame. As with _DataFrame_,
+most operations are vectorized, achieving thereby a high performance when dealing with computations of arrays, in particular labeled arrays.
+As we will see below it leads also to a very concice code close to the mathematical operations we may be interested in.
+For multidimensional arrays, we recommend strongly "xarray":"http://xarray.pydata.org/en/stable/". _xarray_ has much of the same flexibility as _pandas_, but allows for the extension to higher dimensions than two. We will see examples later of the usage of both _pandas_ and _xarray_. 
+
+
+
+===== Reading Data and fitting  =====
+
+In order to study various Machine Learning algorithms, we need to
+access data. Acccessing data is an essential step in all machine
+learning algorithms. In particular, setting up the so-called _design
+matrix_ (to be defined below) is often the first element we need in
+order to perform our calculations. To set up the design matrix means
+reading (and later, when the calculations are done, writing) data
+in various formats, The formats span from reading files from disk,
+loading data from databases and interacting with online sources
+like web application programming interfaces (APIs).
+
+In handling various input formats, as discussed above, we will mainly stay with _pandas_,
+a Python package which allows us, in a seamless and painless way, to
+deal with a multitude of formats, from standard _csv_ (comma separated
+values) files, via _excel_, _html_ to _hdf5_ formats.  With _pandas_
+and the _DataFrame_  and _Series_ functionalities we are able to convert text data
+into the calculational formats we need for a specific algorithm. And our code is going to be 
+pretty close the basic mathematical expressions.
+
+Our first data set is going to be a classic from nuclear physics, namely all
+available data on binding energies. Don't be intimidated if you are not familiar with nuclear physics. It serves simply as an example here of a data set. 
+
+We will show some of the
+strengths of packages like _Scikit-Learn_ in fitting nuclear binding energies to
+specific functions using linear regression first. Then, as a teaser, we will show you how 
+you can easily implement other algorithms like decision trees and random forests and neural networks.
+
+But before we really start with nuclear physics data, let's just look at some simpler polynomial fitting cases, such as,
+(don't be offended) fitting straight lines!
+
+
+=== Simple linear regression model using _scikit-learn_ ===
+
+We start with perhaps our simplest possible example, using _Scikit-Learn_ to perform linear regression analysis on a data set produced by us. 
+
+What follows is a simple Python code where we have defined a function
+$y$ in terms of the variable $x$. Both are defined as vectors with  $100$ entries. 
+The numbers in the vector $\hat{x}$ are given
+by random numbers generated with a uniform distribution with entries
+$x_i \in [0,1]$ (more about probability distribution functions
+later). These values are then used to define a function $y(x)$
+(tabulated again as a vector) with a linear dependence on $x$ plus a
+random noise added via the normal distribution.
 
 
 The Numpy functions are imported used the _import numpy as np_
@@ -182,7 +720,7 @@ y = 2x+N(0,1),
 !et
 
 where $N(0,1)$ represents random numbers generated by the normal
-distribution.  From _scikit-learn_ we import then the
+distribution.  From _Scikit-Learn_ we import then the
 _LinearRegression_ functionality and make a prediction $\tilde{y} =
 \alpha + \beta x$ using the function _fit(x,y)_. We call the set of
 data $(\hat{x},\hat{y})$ for our training data. The Python package
@@ -237,7 +775,7 @@ y = 10x+0.01 \times N(0,1),
 !et
 
 where $x$ is defined as before.  Does the fit look better? Indeed, by
-reducing the role of the normal distribution we see immediately that
+reducing the role of the noise given by the normal distribution we see immediately that
 our linear prediction seemingly reproduces better the training
 set. However, this testing 'by the eye' is obviouly not satisfactory in the
 long run. Here we have only defined the training data and our model, and 
@@ -247,7 +785,7 @@ We need more rigorous criteria in defining whether we have succeeded or
 not in modeling our training data.  You will be surprised to see that
 many scientists seldomly venture beyond this 'by the eye' approach. A
 standard approach for the *cost* function is the so-called $\chi^2$
-function 
+function (a variant of the mean-squared error (MSE))
 
 !bt 
 \[ \chi^2 = \frac{1}{n}
@@ -276,14 +814,14 @@ many practitioners minimize the above function ''by the eye', popularly dubbed a
 the  $\chi^2$ function becomes smaller. 
 
 There are many ways to define the cost function. A simpler approach is to look at the relative difference between the training data and the predicted data, that is we define 
-the relative error as
+the relative error (why would we prefer the MSE instead of the relative error?) as
 
 !bt
 \[
 \epsilon_{\mathrm{relative}}= \frac{\vert \hat{y} -\hat{\tilde{y}}\vert}{\vert \hat{y}\vert}.
 \]
 !et
-We can modify easily the above Python code and plot the relative instead
+We can modify easily the above Python code and plot the relative error instead
 !bc pycod
 import numpy as np
 import matplotlib.pyplot as plt
@@ -308,13 +846,13 @@ have a small or larger relative error. Try to play around with
 different training data sets and study (graphically) the value of the
 relative error.
 
-As mentioned above, _scikit-learn_ has an impressive functionality.
+As mentioned above, _Scikit-Learn_ has an impressive functionality.
 We can for example extract the values of $\alpha$ and $\beta$ and
 their error estimates, or the variance and standard deviation and many
 other properties from the statistical data analysis. 
 
 Here we show an
-example of the functionality of scikit-learn.
+example of the functionality of _Scikit-Learn_.
 !bc pycod 
 import numpy as np 
 import matplotlib.pyplot as plt 
@@ -376,8 +914,8 @@ where we have defined the mean value  of $\hat{y}$ as
 \bar{y} =  \frac{1}{n} \sum_{i=0}^{n - 1} y_i.
 \]
 !et
-Another quantity will meet again in our discussions of regression analysis is 
- mean absolute error (MAE), a risk metric corresponding to the expected value of the absolute error loss or what we call the $l1$-norm loss. In our discussion above we presented the relative error.
+Another quantity taht we will meet again in our discussions of regression analysis is 
+ the mean absolute error (MAE), a risk metric corresponding to the expected value of the absolute error loss or what we call the $l1$-norm loss. In our discussion above we presented the relative error.
 The MAE is defined as follows
 !bt
 \[
@@ -434,691 +972,358 @@ def error(a):
 print (error(y))
 !ec
 
-Similarly, using _R_, we can perform similar studies. The following _R_ code illustrates this.
-(more details on _R_ will be inserted later).
-
-===== Non-Linear Least squares in R  =====
-!bblock
-!bc r
-set.seed(1485)
-len = 24
-x = runif(len)
-y = x^3+rnorm(len, 0,0.06)
-ds = data.frame(x = x, y = y)
-str(ds)
-plot( y ~ x, main ="Known cubic with noise")
-s  = seq(0,1,length =100)
-lines(s, s^3, lty =2, col ="green")
-m = nls(y ~ I(x^power), data = ds, start = list(power=1), trace = T)
-class(m)
-summary(m)
-power = round(summary(m)$coefficients[1], 3)
-power.se = round(summary(m)$coefficients[2], 3)
-plot(y ~ x, main = "Fitted power model", sub = "Blue: fit; green: known")
-s = seq(0, 1, length = 100)
-lines(s, s^3, lty = 2, col = "green")
-lines(s, predict(m, list(x = s)), lty = 1, col = "blue")
-text(0, 0.5, paste("y =x^ (", power, " +/- ", power.se, ")", sep = ""), pos = 4)
-!ec
-!eblock
-
-In our lectures on regression analysis (and other ones as well), we will discuss in more details various _R_ functionalities. 
 
 
-Another useful Python package is
-"pandas":"https://pandas.pydata.org/", which is an open source library
-providing high-performance, easy-to-use data structures and data
-analysis tools for Python. The following simple example shows how we can, in an easy way make tables of our data. Here we define a data set which includes names, city of residence and age, and displays the data in an easy to read way. We will see repeated use of _pandas_, in particular in connection with classification of data. 
 
-!bc pycod 
+=== To our real data: nuclear binding energies. Brief reminder on masses and binding energies ===
+
+Let us now dive into  nuclear physics and remind ourselves briefly about some basic features about binding
+energies.  A basic quantity which can be measured for the ground
+states of nuclei is the atomic mass $M(N, Z)$ of the neutral atom with
+atomic mass number $A$ and charge $Z$. The number of neutrons is $N$. There are indeed several sophisticated experiments worldwide which allow us to measure this quantity to high precision (parts per million even). 
+
+Atomic masses are usually tabulated in terms of the mass excess defined by
+!bt
+\[
+\Delta M(N, Z) =  M(N, Z) - uA,
+\]
+!et
+where $u$ is the Atomic Mass Unit 
+!bt
+\[
+u = M(^{12}\mathrm{C})/12 = 931.4940954(57) \hspace{0.1cm} \mathrm{MeV}/c^2.
+\]
+!et
+The nucleon masses are
+!bt 
+\[
+m_p =  1.00727646693(9)u,
+\] 
+!et
+and
+!bt 
+\[
+m_n = 939.56536(8)\hspace{0.1cm} \mathrm{MeV}/c^2 = 1.0086649156(6)u.
+\]
+!et
+
+In the "2016 mass evaluation of by W.J.Huang, G.Audi, M.Wang, F.G.Kondev, S.Naimi and X.Xu":"http://nuclearmasses.org/resources_folder/Wang_2017_Chinese_Phys_C_41_030003.pdf"
+there are data on masses and decays of 3437 nuclei.
+
+The nuclear binding energy is defined as the energy required to break
+up a given nucleus into its constituent parts of $N$ neutrons and $Z$
+protons. In terms of the atomic masses $M(N, Z)$ the binding energy is
+defined by
+
+
+!bt
+\[
+BE(N, Z) = ZM_H c^2 + Nm_n c^2 - M(N, Z)c^2 ,
+\]
+!et
+where $M_H$ is the mass of the hydrogen atom and $m_n$ is the mass of the neutron.
+In terms of the mass excess the binding energy is given by
+!bt
+\[
+BE(N, Z) = Z\Delta_H c^2 + N\Delta_n c^2 -\Delta(N, Z)c^2 ,
+\]
+!et
+where $\Delta_H c^2 = 7.2890$ MeV and $\Delta_n c^2 = 8.0713$ MeV.
+
+
+A popular and physically intuitive model which can be used to parametrize 
+the experimental binding energies as function of $A$, is the so-called 
+_liquid drop model_. The ansatz is based on the following expression
+
+!bt
+\[ 
+BE(N,Z) = a_1A-a_2A^{2/3}-a_3\frac{Z^2}{A^{1/3}}-a_4\frac{(N-Z)^2}{A},
+\]
+!et
+
+where $A$ stands for the number of nucleons and the $a_i$s are parameters which are determined by a fit 
+to the experimental data.  
+
+
+
+
+To arrive at the above expression we have assumed that we can make the following assumptions:
+
+ * There is a volume term $a_1A$ proportional with the number of nucleons (the energy is also an extensive quantity). When an assembly of nucleons of the same size is packed together into the smallest volume, each interior nucleon has a certain number of other nucleons in contact with it. This contribution is proportional to the volume.
+
+ * There is a surface energy term $a_2A^{2/3}$. The assumption here is that a nucleon at the surface of a nucleus interacts with fewer other nucleons than one in the interior of the nucleus and hence its binding energy is less. This surface energy term takes that into account and is therefore negative and is proportional to the surface area.
+
+
+ * There is a Coulomb energy term $a_3\frac{Z^2}{A^{1/3}}$. The electric repulsion between each pair of protons in a nucleus yields less binding. 
+
+ * There is an asymmetry term $a_4\frac{(N-Z)^2}{A}$. This term is associated with the Pauli exclusion principle and reflects the fact that the proton-neutron interaction is more attractive on the average than the neutron-neutron and proton-proton interactions.
+
+We could also add a so-called pairing term, which is a correction term that
+arises from the tendency of proton pairs and neutron pairs to
+occur. An even number of particles is more stable than an odd number. 
+
+
+=== Organizing our data ===
+
+Let us start with reading and organizing our data. 
+We start with the compilation of masses and binding energies from 2016.
+After having downloaded this file to our own computer, we are now ready to read the file and start structuring our data.
+
+
+We start with preparing folders for storing our calculations and the data file over masses and binding energies. We import also various modules that we will find useful in order to present various Machine Learning methods. Here we focus mainly on the functionality of _scikit-learn_.
+!bc pycod
+# Common imports
+import numpy as np
 import pandas as pd
-from IPython.display import display
-data = {'Name': ["John", "Anna", "Peter", "Linda"], 'Location': ["Nairobi", "Napoli", "London", "Buenos Aires"], 'Age':[51, 21, 34, 45]}
-data_pandas = pd.DataFrame(data)
-display(data_pandas)
+import matplotlib.pyplot as plt
+import sklearn.linear_model as skl
+from sklearn.model_selection import train_test_split
+from sklearn.metrics import mean_squared_error, r2_score, mean_absolute_error
+import os
+
+# Where to save the figures and data files
+PROJECT_ROOT_DIR = "Results"
+FIGURE_ID = "Results/FigureFiles"
+DATA_ID = "DataFiles/"
+
+if not os.path.exists(PROJECT_ROOT_DIR):
+    os.mkdir(PROJECT_ROOT_DIR)
+
+if not os.path.exists(FIGURE_ID):
+    os.makedirs(FIGURE_ID)
+
+if not os.path.exists(DATA_ID):
+    os.makedirs(DATA_ID)
+
+def image_path(fig_id):
+    return os.path.join(FIGURE_ID, fig_id)
+
+def data_path(dat_id):
+    return os.path.join(DATA_ID, dat_id)
+
+def save_fig(fig_id):
+    plt.savefig(image_path(fig_id) + ".png", format='png')
+
+infile = open(data_path("MassEval2016.dat"),'r')
 !ec
 
 
+Before we proceed, we define also a function for making our plots. You can obviously avoid this and simply set up various _matplotlib_ commands every time you need them. You may however find it convenient to collect all such commands in one function and simply call this function. 
+!bc pycod 
+from pylab import plt, mpl
+plt.style.use('seaborn')
+mpl.rcParams['font.family'] = 'serif'
 
-===== Examples =====
+def MakePlot(x,y, styles, labels, axlabels):
+    plt.figure(figsize=(10,6))
+    for i in range(len(x)):
+        plt.plot(x[i], y[i], styles[i], label = labels[i])
+        plt.xlabel(axlabels[0])
+        plt.ylabel(axlabels[1])
+    plt.legend(loc=0)
+!ec
 
-We present here several examples, with pertinent Python codes that we
-will use to illustrate various machine learning methods and ways to
-analyze, from simple to complex, various data sets. Many of these
-examples allow us to generate the data we want to analyze, following
-much of the same philosophy we discussed above when
-fitting various polynomials.
-
-We start with a simple exponential growth model that is meant to mimick an ecoli lab experiment.
-We can easily model this system and then produce the data used to train various machine learning algorithms.
-Another model from the life sciences is the so-called predator-prey model from ecology. Thereafter we present 
-a simple model for financial transactions before moving to a random walk model and ending with 
-the simulation of velocities of a non-interacting atom or molecule confined to move in  a one-dimensional region.
+Our next step is to read the data on experimental binding energies and
+reorganize them as functions of the mass number $A$, the number of
+protons $Z$ and neutrons $N$ using _pandas_.  Before we do this it is
+always useful (unless you have a binary file or other types of compressed
+data) to actually open the file and simply take a look at it!
 
 
-=== Ecoli lab experiment ===
+In particular, the program that outputs the final nuclear masses is written in Fortran with a specific format. It means that we need to figure out the format and which columns contain the data we are interested in. Pandas comes with a function that reads formatted output. After having admired the file, we are now ready to start massaging it with _pandas_. The file begins with some basic format information.
+!bc pycod 
+"""                                                                                                                         
+This is taken from the data file of the mass 2016 evaluation.                                                               
+All files are 3436 lines long with 124 character per line.                                                                  
+       Headers are 39 lines long.                                                                                           
+   col 1     :  Fortran character control: 1 = page feed  0 = line feed                                                     
+   format    :  a1,i3,i5,i5,i5,1x,a3,a4,1x,f13.5,f11.5,f11.3,f9.3,1x,a2,f11.3,f9.3,1x,i3,1x,f12.5,f11.5                     
+   These formats are reflected in the pandas widths variable below, see the statement                                       
+   widths=(1,3,5,5,5,1,3,4,1,13,11,11,9,1,2,11,9,1,3,1,12,11,1),                                                            
+   Pandas has also a variable header, with length 39 in this case.                                                          
+"""
+!ec
 
-
-A typical pattern seen in population models is that the population grows faster and faster. "Why? Is there an underlying (general) mechanism":"http://www.zo.utexas.edu/courses/Thoc/PopGrowth.html"?
-Here we will construct a model for cell growth based on a simple difference equation for the growth. We make the following assumptions
-!bblock
- o Cells divide after $T$ seconds on average (one generation)
- o $2N$ celles divide into twice as many new cells $\Delta N$ in a time
-   interval $\Delta t$ as $N$ cells would: $\Delta N \propto N$
- o $N$ cells result in twice as many new individuals $\Delta N$ in
-   time $2\Delta t$ as in time $\Delta t$: $\Delta N \propto\Delta t$
- o Same proportionality with respect to  death 
- o Proposed model: $\Delta N = b\Delta t N - d\Delta tN$ for some unknown
-   constants $b$ (births) and $d$ (deaths)
- o Describe evolution in discrete time: $t_n=n\Delta t$
- o Program-friendly notation: $N$ at $t_n$ is $N^n$
- o Math model: $N^{n+1} = N^n + r\Delta t\, N$ (with $\ r=b-d$)
- o Program model: `N[n+1] = N[n] + r*dt*N[n]`
-!eblock
-
-The difference equation can be programmed in a simple way, and in order to get started we
-set $r=1.5$, $N^0=1$, $\Delta t=0.5$. The program reads
+The data we are interested in are in columns 2, 3, 4 and 11, giving us
+the number of neutrons, protons, mass numbers and binding energies,
+respectively. We add also for the sake of completeness the element name. The data are in fixed-width formatted lines and we will
+covert them into the _pandas_ DataFrame structure.
 
 !bc pycod 
-import numpy as np
+# Read the experimental data with Pandas
+Masses = pd.read_fwf(infile, usecols=(2,3,4,6,11),
+              names=('N', 'Z', 'A', 'Element', 'Ebinding'),
+              widths=(1,3,5,5,5,1,3,4,1,13,11,11,9,1,2,11,9,1,3,1,12,11,1),
+              header=39,
+              index_col=False)
 
-t = np.linspace(0, 10, 21)  # 20 intervals in [0, 10]
-dt = t[1] - t[0]
-N = np.zeros(t.size)
-N[0] = 1
-r = 0.5
+# Extrapolated values are indicated by '#' in place of the decimal place, so
+# the Ebinding column won't be numeric. Coerce to float and drop these entries.
+Masses['Ebinding'] = pd.to_numeric(Masses['Ebinding'], errors='coerce')
+Masses = Masses.dropna()
+# Convert from keV to MeV.
+Masses['Ebinding'] /= 1000
 
-for n in range(0, N.size-1, 1):
-    N[n+1] = N[n] + r*dt*N[n]
-    print('N[%d]=%.1f' % (n+1, N[n+1]))
+# Group the DataFrame by nucleon number, A.
+Masses = Masses.groupby('A')
+# Find the rows of the grouped DataFrame with the maximum binding energy.
+Masses = Masses.apply(lambda t: t[t.Ebinding==t.Ebinding.max()])
 !ec
-and it generates the following output
-!bc
-N[1]=1.2
-N[2]=1.6
-N[3]=2.0
-N[4]=2.4
-N[5]=3.1
-N[6]=3.8
-N[7]=4.8
-N[8]=6.0
-N[9]=7.5
-N[10]=9.3
-N[11]=11.6
-N[12]=14.6
-N[13]=18.2
-N[14]=22.7
-N[15]=28.4
-N[16]=35.5
-N[17]=44.4
-N[18]=55.5
-N[19]=69.4
-N[20]=86.7
-!ec
-This forms our data which later will define our training set. 
-In this case we defined the value of the parameter $r$. We could alternatively assume that we just received the 
-above data file and where asked to find $r$. How can we estimate $r$ from data? This will be one of our tasks later.
 
-We can use the difference equation with the experimental data
-!bt
-\[ N^{n+1} = N^n + r\Delta t N^n\]
-!et
-Suppose now that  $N^{n+1}$ and $N^n$ are known from data. Then we could solve with respect to  $r$ as follows
-!bt
-\[ r = \frac{N^{n+1}-N^n}{N^n\Delta t} \]
-!et
-Suppose we set $t_1=600$, $t_2=1200$,
-$N^1=140$ and $N^2=250$. 
-The following code plots the data 
+We have now read in the data, grouped them according to the variables we are interested in. 
+We see how easy it is to reorganize the data using _pandas_. If we
+were to do these operations in C/C++ or Fortran, we would have had to
+write various functions/subroutines which perform the above
+reorganizations for us.  Having reorganized the data, we can now start
+to make some simple fits using both the functionalities in _numpy_ and
+_Scikit-Learn_ afterwards. 
+
+Now we define five variables which contain
+the number of nucleons $A$, the number of protons $Z$ and the number of neutrons $N$, the element name and finally the energies themselves.
 !bc pycod 
-import numpy as np
-import matplotlib.pyplot as plt
-
-# Estimate r
-data = np.loadtxt('ecoli.csv', delimiter=',')
-t_e = data[:,0]
-N_e = data[:,1]
-i = 2  # Data point (i,i+1) used to estimate r
-r = (N_e[i+1] - N_e[i])/(N_e[i]*(t_e[i+1] - t_e[i]))
-print('Estimated r=%.5f' % r)
-# Can experiment with r values and see if the model can
-# match the data better
-T = 1200     # cell can divide after T sec
-t_max = 5*T  # 5 generations in experiment
-t = np.linspace(0, t_max, 1000)
-dt = t[1] - t[0]
-N = np.zeros(t.size)
-
-N[0] = 100
-for n in range(0, len(t)-1, 1):
-    N[n+1] = N[n] + r*dt*N[n]
-
-plt.plot(t, N, 'r-', t_e, N_e, 'bo')
-plt.xlabel('time [s]');  plt.ylabel('N')
-plt.legend(['model', 'experiment'], loc='upper left')
-plt.show()
-
+A = Masses['A']
+Z = Masses['Z']
+N = Masses['N']
+Element = Masses['Element']
+Energies = Masses['Ebinding']
+print(Masses)
 !ec
-We can then change the parameter $r$ in the program and play around to make a better fit. By now we know that this
-'search bythe eye' approach is not the most optimal one. 
-
-
-=== Predator-Prey model from ecology ===
-
-
-The population dynamics of a simple predator-prey system is a
-classical example shown in many biology textbooks when ecological
-systems are discussed. The system contains all elements of the
-scientific method:
-
- * The set up of a specific hypothesis combined with
- * the experimental methods needed (one can study existing data or perform experiments)
- * analyzing and interpreting the data and performing further experiments if needed
- * trying to extract general behaviors and extract eventual laws or patterns
- * develop mathematical relations for the uncovered regularities/laws and test these by per forming new experiments
-
-Lots of data about populations of hares and lynx collected from furs in Hudson Bay, Canada, are available. It is known that the populations oscillate. Why?
-Here we start by
-
- o plotting the data
- o derive a simple model for the population dynamics
- o (fitting parameters in the model to the data)
- o using the model predict the evolution other predator-pray systems
-
-Most mammalian predators rely on a variety of prey, which complicates mathematical modeling; however, a few predators have become highly specialized and seek almost exclusively a single prey species. An example of this simplified predator-prey interaction is seen in Canadian northern forests, where the populations of the lynx and the snowshoe hare are intertwined in a life and death struggle.
-
-One reason that this particular system has been so extensively studied is that the Hudson Bay company kept careful records of all furs from the early 1800s into the 1900s. The records for the furs collected by the Hudson Bay company showed distinct oscillations (approximately 12 year periods), suggesting that these species caused almost periodic fluctuations of each other's populations. The table here shows data from 1900 to 1920.
-
-
-|------------------------------------------------------|
-|      Year         |     Hares (x1000)  | Lynx (x1000)|
-|---------l-----------------------r--------------r------|
-|  1900 | 30.0  | 4.0  |
-|  1901 | 47.2  | 6.1 |
-|  1902 | 70.2  | 9.8  |
-|  1903 | 77.4  | 35.2  |
-|  1904 | 36.3  | 59.4  |
-|  1905 | 20.6  | 41.7  |
-|  1906 | 18.1  | 19.0  |
-|  1907 | 21.4  | 13.0  |
-|  1908 | 22.0  | 8.3  |
-|  1909 | 25.4  | 9.1  |
-|  1910 | 27.1  | 7.4 |
-|  1911 | 40.3  | 8.0  |
-|  1912 | 57  | 12.3  |
-|  1913 | 76.6  | 19.5  |
-|  1914 | 52.3  | 45.7  |
-|  1915 | 19.5  | 51.1  |
-|  1916 | 11.2  | 29.7  |
-|  1917 | 7.6  | 15.8  |
-|  1918 | 14.6  | 9.7  |
-|  1919 | 16.2  | 10.1  |
-|  1920 | 24.7  | 8.6  |
-|------------------------------------------------------|
-
-
-
-@@@CODE src/plot_Hudson.py
-
-FIGURE: [fig/Hudson_Bay_data, width=700 frac=0.9]
-
-
-We see from the plot that there are indeed fluctuations.
-We would like to create a mathematical model that explains these
-population fluctuations. Ecologists have predicted that in a simple
-predator-prey system that a rise in prey population is followed (with
-a lag) by a rise in the predator population. When the predator
-population is sufficiently high, then the prey population begins
-dropping. After the prey population falls, then the predator
-population falls, which allows the prey population to recover and
-complete one cycle of this interaction. Thus, we see that
-qualitatively oscillations occur. Can a mathematical model predict
-this? What causes cycles to slow or speed up? What affects the
-amplitude of the oscillation or do you expect to see the oscillations
-damp to a stable equilibrium? The models tend to ignore factors like
-climate and other complicating factors. How significant are these?
-
- * We see oscillations in the data
- * What causes cycles to slow or speed up?
- * What affects the amplitude of the oscillation or do you expect to see the oscillations damp to a stable equilibrium?
- * With a model we can better *understand the data*
- * More important: Can we understand the ecology dynamics of predator-pray populations?
-
-The classical way (in all books) is to present the Lotka-Volterra equations:
-
-!bt
-\begin{align*}
-\frac{dH}{dt} &= H(a - b L)\\
-\frac{dL}{dt} &= - L(d - c  H)
-\end{align*}
-!et
-
-Here,
-
- * $H$ is the number of preys
- * $L$ the number of predators
- * $a$, $b$, $d$, $c$ are parameters
-
-
-The population of hares evolves due to births and deaths exactly as a bacteria population:
-
-!bt
-\[
-\Delta H = a \Delta t H^n
-\]
-!et
-However, hares have an additional loss in the population because
-they are eaten by lynx.
-All the hares and lynx can form
-$H\cdot L$ pairs in total. When such pairs meet during a time
-interval $\Delta t$, there is some
-small probablity that the lynx will eat the hare.
-So in fraction $b\Delta t HL$, the lynx eat hares. This
-loss of hares must be accounted for. Subtracted in the equation for hares:
-
-!bt
-\[ \Delta H = a\Delta t H^n - b \Delta t H^nL^n\]
-!et
-
-We assume that the primary growth for the lynx population depends on sufficient food for raising lynx kittens, which implies an adequate source of nutrients from predation on hares. Thus, the growth of the lynx population does not only depend of how many lynx there are, but on how many hares they can eat.
-In a time interval $\Delta t HL$ hares and lynx can meet, and in a
-fraction $b\Delta t HL$ the lynx eats the hare. All of this does not
-contribute to the growth of lynx, again just a fraction of
-$b\Delta t HL$ that we write as
-$d\Delta t HL$. In addition, lynx die just as in the population
-dynamics with one isolated animal population, leading to a loss
-$-c\Delta t L$.
-The accounting of lynx then looks like
-!bt
-\[ \Delta L = d\Delta t H^nL^n - c\Delta t L^n\]
-!et
-
-By writing up the definition of $\Delta H$ and $\Delta L$, and putting
-all assumed known terms $H^n$ and $L^n$ on the right-hand side, we have
-
-!bt
-\[ H^{n+1} = H^n + a\Delta t H^n - b\Delta t H^n L^n \]
-!et
-
-!bt
-\[ L^{n+1} = L^n + d\Delta t H^nL^n - c\Delta t L^n \]
-!et
-
-Note:
-
- * These equations are ready to be implemented!
- * But to start, we need $H^0$ and $L^0$ (which we can get from the data)
- * We also need values for $a$, $b$, $d$, $c$
-
- * As always, models tend to be general - as here, applicable
-   to ``all'' predator-pray systems
- * The critical issue is whether the *interaction* between hares and lynx
-   is sufficiently well modeled by $\hbox{const}HL$
- * The parameters $a$, $b$, $d$, and $c$ must be
-   estimated from data
-
-!bblock
-@@@CODE src/Hudson_Bay.py
-!eblock
-
-FIGURE: [fig/Hudson_Bay_sim, width=700 frac=0.9]
-
-We will later perform a least-square fitting. Then we can find optimal
-values for the parameters $a$, $b$, $d$, $c$. In our calculations here
-we set $a=0.4807$, $b=0.02482$, $d=0.9272$ and $c=0.02756$. These
-parameters result in a slightly modified initial conditions, namely
-$H(0) = 34.91$ and $L(0)=3.857$. 
-
-
-The following Python code demonstrates how we can use linear regression to fit for example the population of lynx.
-Similarly, we have also used a decision tree algorithm to fit the lynx population data. As expected, the linear regression is not exactly impressive 
+The next step, and we will define this mathematically later, is to set up the so-called _design matrix_. We will throughout call this matrix $\bm{X}$.
+It has dimensionality $p\times n$, where $n$ is the number of data points and $p$ are the so-called predictors. In our case here they are given by the number of polynomials in $A$ we wish to include in the fit. 
 !bc pycod
-import numpy as np
-import matplotlib.pyplot as plt
-from IPython.display import display
-import sklearn
-from sklearn.linear_model import LinearRegression
-from sklearn.tree import DecisionTreeRegressor
-
-
-data = np.loadtxt('src/Hudson_Bay.csv', delimiter=',', skiprows=1)
-x = data[:,0]
-y = data[:,1]
-line = np.linspace(1900,1920,1000,endpoint=False).reshape(-1,1)
-reg = DecisionTreeRegressor(min_samples_split=3).fit(x.reshape(-1,1),y.reshape(-1,1))
-plt.plot(line, reg.predict(line), label="decision tree")
-regline = LinearRegression().fit(x.reshape(-1,1),y.reshape(-1,1))
-plt.plot(line, regline.predict(line), label= "Linear Regression")
-plt.plot(x, y, label= "Linear Regression")
-plt.show()
+# Now we set up the design matrix X
+X = np.zeros((len(A),5))
+X[:,0] = 1
+X[:,1] = A
+X[:,2] = A**(2.0/3.0)
+X[:,3] = A**(-1.0/3.0)
+X[:,4] = A**(-1.0)
 !ec
-
-
-
-The similar code for linear regression in _R_ reads (more details to come)
-!bc r
-HudsonBay = read.csv("src/Hudson_Bay.csv",header=T)
-fix(HudsonBay)
-dim(HudsonBay)
-names(HudsonBay)
-plot(HudsonBay$Year, HudsonBay$Hares..x1000.)
-attach(HudsonBay)
-plot(Year, Hares..x1000.)
-plot(Year, Hares..x1000., col="red", varwidth=T, xlab="Years", ylab="Haresx 1000")
-summary(HudsonBay)
-summary(Hares..x1000.)
-library(MASS)
-library(ISLR)
-scatter.smooth(x=Year, y = Hares..x1000.)
-linearMod = lm(Hares..x1000. ~ Year)
-print(linearMod)
-summary(linearMod)
-plot(linearMod)
-confint(linearMod)
-predict(linearMod,data.frame(Year=c(1910,1914,1920)),interval="confidence")
-!ec
-
-=== Simulating financial transactions ===
-
-The aim here is to simulate financial transactions among financial agents
-using Monte Carlo methods. The final goal is to extract a distribution of income  as function
-of the income $m$.   From Pareto's work ("V.~Pareto, 1897":"http://www.institutcoppet.org/2012/05/08/cours-deconomie-politique-1896-de-vilfredo-pareto") it is known from empirical studies
-that the higher end of the distribution of money follows a distribution 
-!bt
-\[
-w_m\propto m^{-1-\alpha},
-\]
-!et
-with $\alpha\in [1,2]$. We will here follow the analysis made by "Patriarca and collaborators":"http://www.sciencedirect.com/science/article/pii/S0378437104004327". 
-
-Here we will study numerically the relation between the micro-dynamic relations among financial 
-agents and the  resulting macroscopic money distribution.
-
-We assume we have $N$ agents that exchange money in pairs $(i,j)$. We assume also that all agents
-start with the same amount of money $m_0 > 0$. At a given 'time step', we choose randomly a pair
-of agents $(i,j)$ and let a transaction take place. This means that agent $i$'s money $m_i$ changes
-to $m_i'$ and similarly we have $m_j\rightarrow m_j'$. 
-Money is conserved during a transaction, meaning that
-!bt
-\begin{equation}
-  m_i+m_j=m_i'+m_j'.
-  label{eq:conserve}
-\end{equation}
-!et
-The change is done via a random reassignement (a random number) $\epsilon$, meaning that
-
-!bt
-\begin{equation*}
-m_i' = \epsilon(m_i+m_j),
-\end{equation*}
-!et
-leading to
-
-!bt
-\begin{equation*}
-m_j'= (1-\epsilon)(m_i+m_j).
-\end{equation*}
-!et
-The number $\epsilon$ is extracted from a uniform distribution.
-In this simple model, no agents are left with a debt, that is $m\ge 0$.
-Due to the conservation law above, one can show that the system relaxes toward an equilibrium
-state given by a Gibbs distribution
-
-!bt
-\begin{equation*}
-w_m=\beta \exp{(-\beta m)},
-\end{equation*}
-!et
-with
-
-!bt
-\begin{equation*}
-\beta = \frac{1}{\langle m\rangle},
-\end{equation*}
-!et
-and $\langle m\rangle=\sum_i m_i/N=m_0$, the average money.
-It means that after equilibrium has been reached that the majority of agents is left with a small
-number of money, while the number of richest agents, those with $m$ larger than a specific value $m'$,
-exponentially decreases with $m'$.
-
-We assume that we have $N=500$ agents.   In each simulation, we need a sufficiently large number of transactions, say $10^7$. Our aim is find the final equilibrium distribution $w_m$. In order to do that we would need
-several runs of the above simulations, at least $10^3-10^4$ runs (experiments).
-
-Our task is to first set up an algorithm which simulates the above transactions with an initial
-  amount $m_0$.
-  The challenge here is to figure out a Monte Carlo  simulation  based on the
-  above equations.
-  You will in particular need to make an algorithm which sets up a histogram as function of $m$.
-  This histogram contains the number of times a value $m$ is registered and represents
-  $w_m\Delta m$. You will need to set up a value for the interval $\Delta m$  (typically $0.01-0.05$).
-  That means you need to account for the number of times you register an income in the interval
-  $m,m+\Delta m$. The number of times you register this income, represents the value that enters the histogram.
-
-!bc pycod
-#!/usr/bin/env python
-import numpy as np
-import matplotlib.mlab as mlab
-import matplotlib.pyplot as plt
-import random
-
-# initialize the rng with a seed
-random.seed()
-# Hard coding of input parameters
-Agents  = 500
-MCcounts = 1000
-Transactions = 100000
-startMoney = 1.0
-Lambda = 0.0
-FinancialAgents = startMoney*np.ones(Agents)
-for i in range (1, MCcounts, 1):
-    for j in range (1, Transactions, 1):
-        agent_i = int(Agents*random.random())
-        agent_j = int(Agents*random.random())
-        epsilon = random.random()
-        if agent_i != agent_j:
-           m1 = Lambda*FinancialAgents[agent_i] + (1-Lambda)*epsilon*(FinancialAgents[agent_i] + FinancialAgents[agent_j])
-           m2 = Lambda*FinancialAgents[agent_j] + (1-Lambda)*(1-epsilon)*(FinancialAgents[agent_i] + FinancialAgents[agent_j])
-           FinancialAgents[agent_i] = m1
-           FinancialAgents[agent_j] = m2
-
-# the histogram of the data
-n, bins, patches = plt.hist(FinancialAgents, 50, facecolor='green')
-
-plt.xlabel('$x$')
-plt.ylabel('Distribution of wealth')
-plt.title(r'Money')
-plt.axis([0, 10, 0, 500])
-plt.grid(True)
-plt.show()
-
-!ec
-
-
-We can then change our model to allow for a saving criterion, meaning that the agents save
-  a fraction $\lambda$ of the money they have before the transaction is made. The final distribution will then no longer be given by Gibbs distribution. It could also include a taxation on financial transactions.
-
-  The conservation law of Eq. (ref{eq:conserve}) holds, but the money to be shared in a transaction between
-  agent $i$ and agent $j$ is now $(1-\lambda)(m_i+m_j)$. This means that we have
-
-!bt
-\begin{equation*}
-  m_i' = \lambda m_i+\epsilon(1-\lambda)(m_i+m_j),
-  \end{equation*}
-!et
-  and
-
-!bt
-\begin{equation*}
-  m_j' = \lambda m_j+(1-\epsilon)(1-\lambda)(m_i+m_j),
-  \end{equation*}
-!et
-  which can be written as
-
-!bt
-\begin{equation*}
-  m_i'=m_i+\delta m
-  \end{equation*}
-!et
-  and
-
-!bt
-\begin{equation*}
-  m_j'=m_j-\delta m,
-  \end{equation*}
-!et
-  with
-
-!bt
-\begin{equation*}
-  \delta m=(1-\lambda)(\epsilon m_j-(1-\epsilon)m_i),
-  \end{equation*}
-!et
-  showing how money is conserved during a transaction.
-  Select values of $\lambda =0.25,0.5$ and $\lambda=0.9$ and try to extract the corresponding
-  equilibrium distributions and compare these with the Gibbs distribution. We will use this model to 
-extract a parametrization of the above curves, see for example "Patriarca and collaborators":"http://www.sciencedirect.com/science/article/pii/S0378437104004327".
-
-
-=== Particle in one dimension and velocity distribution ===
+With _scikitlearn_ we are now ready to use linear regression and fit our data.
 !bc pycod 
-# Program to test the Metropolis algorithm with one particle at given temp in one dimension
-import numpy as np
-import matplotlib.mlab as mlab
-import matplotlib.pyplot as plt
-import random
-from math import sqrt, exp, log
-# initialize the rng with a seed
-random.seed()
-# Hard coding of input parameters
-MCcycles = 100000
-Temperature = 2.0
-beta = 1./Temperature
-InitialVelocity = -2.0
-CurrentVelocity = InitialVelocity
-Energy = 0.5*InitialVelocity*InitialVelocity
-VelocityRange = 10*sqrt(Temperature)
-VelocityStep = 2*VelocityRange/10.
-AverageEnergy = Energy
-AverageEnergy2 = Energy*Energy
-VelocityValues = np.zeros(MCcycles)
-# The Monte Carlo sampling with Metropolis starts here
-for i in range (1, MCcycles, 1):
-    TrialVelocity = CurrentVelocity + (2.0*random.random() - 1.0)*VelocityStep
-    EnergyChange = 0.5*(TrialVelocity*TrialVelocity -CurrentVelocity*CurrentVelocity);
-    if random.random() <= exp(-beta*EnergyChange):
-        CurrentVelocity = TrialVelocity
-        Energy += EnergyChange
-        VelocityValues[i] = CurrentVelocity
-    AverageEnergy += Energy
-    AverageEnergy2 += Energy*Energy
-#Final averages
-AverageEnergy = AverageEnergy/MCcycles
-AverageEnergy2 = AverageEnergy2/MCcycles
-Variance = AverageEnergy2 - AverageEnergy*AverageEnergy
-print(AverageEnergy, Variance)
-n, bins, patches = plt.hist(VelocityValues, 400, facecolor='green')
+clf = skl.LinearRegression().fit(X, Energies)
+fity = clf.predict(X)
+!ec
+Pretty simple!  
+Now we can print measures of how our fit is doing, the coefficients from the fits and plot the final fit together with our data.
+!bc pycod
+# The mean squared error                               
+print("Mean squared error: %.2f" % mean_squared_error(Energies, fity))
+# Explained variance score: 1 is perfect prediction                                 
+print('Variance score: %.2f' % r2_score(Energies, fity))
+# Mean absolute error                                                           
+print('Mean absolute error: %.2f' % mean_absolute_error(Energies, fity))
+print(clf.coef_, clf.intercept_)
 
-plt.xlabel('$v$')
-plt.ylabel('Velocity distribution P(v)')
-plt.title(r'Velocity histogram at $k_BT=2$')
-plt.axis([-5, 5, 0, 600])
-plt.grid(True)
+Masses['Eapprox']  = fity
+# Generate a plot comparing the experimental with the fitted values values.
+fig, ax = plt.subplots()
+ax.set_xlabel(r'$A = N + Z$')
+ax.set_ylabel(r'$E_\mathrm{bind}\,/\mathrm{MeV}$')
+ax.plot(Masses['A'], Masses['Ebinding'], alpha=0.7, lw=2,
+            label='Ame2016')
+ax.plot(Masses['A'], Masses['Eapprox'], alpha=0.7, lw=2, c='m',
+            label='Fit')
+ax.legend()
+save_fig("Masses2016")
 plt.show()
-
 !ec
 
 
+=== Seeing the wood for the trees ===
+
+As a teaser, let us now see how we can do this with decision trees using _scikit-learn_. Later we will switch to so-called _random forests_!
 
 
-=== Random walk model ===
 !bc pycod
-import numpy as np
-import matplotlib.pyplot as plt
-from sklearn.preprocessing import PolynomialFeatures
-from sklearn.linear_model import LinearRegression
-
-steps=250
-
-distance=0
-x=0
-distance_list=[]
-steps_list=[]
-while x_W@ct=@PwI}>4Z5AGxG^EGeg77Ibm*?n}!-{H_yBK&F2!vE_OanHbEIT
zTP89NE;4x!hyUqL#?B$Y#VNqeMkcGIA;|Xc8p=k+#@XE&Xyfcc#>v74!6zZ71Oo%{
z2~Y||P_PgPT>R|ZY}~x&L{Rt;FbbN=vi~7aGMfJtuqxupQsVz%kevTR|Ky10e+cRt
za*B%oVQ~M0ssD$xwCZU^#3!@|80x^hkcJvS-foBZJa&a$y_Ym9BqL>TW2R0A}DW&s{g-rfQqE_
zAN!vo`;bYxS$--`fb4%PfXvK??EgU>WOV<_5a>~n{y#MF|JCq+s0I}Y6^RJy(~=-y
zApUJZNJ7BDz{0`8z`?=7!NbELAfY26AtEASe?db*|B8)=_Z1r#mw=d>lz@BL=>C6(e?t%$2vD5RXV8!o
z5KtJ9&=`>aMj(hj*@XV@@cn1{{{zr4ke~Js2aoVc#fOCWkNCex|CNwX&@iz7HX%@;
zAt9j9q0v8E9)Kl^OQr!Wz|Skf+s*-$R44En~$C{P_@4Ni|=kMBQs}PbN$|;gxQ)v>wuoK
zhf?pEiqNfO(Sh{#a{E4x6DV-HXgzjjGg;rcETx9uld!$XC{SJ2?I-sEtrwNM8W~-oB3k=?c
z|Ngd%WX$!NftgB@@zsg{CnBUS{j$_?Tbsp*a|5vk!!C`z=?V_|7%GZ7)oSx%DbchC
z0p?F{%!Ru*9e(8kA8UPPm5^AW$bf`})vd(3BCxG1Rn
zf>AZ_uRZm9#eoMAz{<&lo;uOTj_ru-gv@W_b$sk3L
z4Sa2#@=y>F8!Uy{xfb)On
z(EXtqO6mynP)t}1(*S>TcpOdoM#T17i=ggsJ(U?pce8KGjCrC(ZnZ9^0f8RqlUTr4
z#?=d3f^Syo3uL+8BR>-|veNFkCz(>$j$Usa)Ni~T;HLJGh*2FcxHH!LJ^D&Sa7BH;
zr+f538RV5-vgI}y;$qgSrntvudjBBUmE@EB+Oe@lnXa$FNx#uCnW?E)YtF6pUIyz@
zoPCI;k2Qi5yHw1nHHaM|Oeg82K}3FRW0B25y92y8dK8}BxomP5+UO`UhpZ3GgU}l^
z%p&H+3u?Up}$JJNbinW2Ugt3PFpqXop7;3e>v@cncl4{DXHJHri-H2
zF{S{lc*h0Rx80)^`rKAfM<0pg9Su*s7+y^Y!#DjViQ#zUW&*HXjI(>Yi4&ZmswL?~
zHBkAjsAO4qMA|Z<)3RXDLz8mFQUx;JlgQm`=rJ9O6
zNAn>q>8S6mK*yvb)Pv(Q?)%tPg@0tZK_GU2<_vuE9l;BPyO33Afg7>|ZR&A=b`F-hd?%cUXV6SrCdKxZZ51CK(CW4211(W*8GyT-#Y%^hGgu4Kky_{-9*K7!vK((VV>J^zh@
zumnSDXlQB502Pn@%J=g5`t<7NHu0GfIk7q^^l1_qZ3c#nmC3a>A0kWb17On2P8`YK
z-o4WI`D>yN-d-`cU28;=ldiYPKSVzO$5I`=FKe|#K0i$F-zGDf8WQreZy6Qc9|Ili
zZINwH#y0%Qgz}#;0>5`0$9J6yIG>NbWcHRQcS+;MJbu$S!^-%zb$5uSM7XC@Y03Hl
zI1(8;+?unc%#?}Vt#l8@GOI6P)|-}LLnXJ+l(bMJFXDeh?e&;CFaC1Rz~Wv@1;+xg
ztVj6*%A3NMpP|@vRI;+^Q-wj2(qvyZlc674MYepI3&F@HF(*4?TPvfLLDu8WJjup9
zgsV?B%R5juE6KG%p1*#1xk_d9Ia}QLMW`8B%~s5hr&80tnuMD7HPQkgi%d&tAkrb5
z^yW1j*fqM`6y6g?%JQg}6Mc44c8dO7b_YOSVVM)V_b08s%lYZK;z=WA+2ZHBksmjH
z>b=`DFB0x-zmsb{zs7wbp0jFCv$^iO`ZqG}!f3D8G5q5Gm)%CFEs?midQ)a$KCRQs
zb8C>AGk>o)BKZlrJE5T}%h+V9mq1IlZXaaDtPK-7cagY_pjeN=C2@_Oo4B>M#t(Ea
zHcKCMEv+SXi%#iuor(&}e7vgapA8(Ma!EC)b(Hke*IW*o!c<_b)$XLN>J>(C@tv}1
zEmFugsG>tE-&k;hr4OQ=k9;eb+PIjUBpA*3s4wQ!tytiQ><&igSeMlkTi>pvtTx@~%_tShUBh)v8Sv3Sb#%u%9!oB;=ay}oy
zKXboNyB^Zg7@<+XP)nK7%EriLZ{`kx{E0M*kp~=i|TXNrU9J8iFcL
zA6gE)GP)yB8f`A&hBb=~qr!`3p(3!79xzX{>Z^dLU!2*>`0>fs425p^7!*|qNZsRc
zv9*pWR9D;d=6ZLIyzGE`C>!UcaFn1`yF3@TDa1u`47L!D*A--T+)zsJ|
zSSuex7}g(EVBe+GL8U80*n(2LN}p?19qICReC0fK=d`0@_&%>f$(Sx$WU5|RgcYjE
zN=z^Z=4yHof~f3gy?6GGY*d98mMG@pDcFxxV!GJm;m#T!PCRsfoyy=keFR-#P|#Vc
zeP6mXolbu3vS2$_QF)k^hZ%BUu(a+-P)%!;D$A>-wn;f`%B+`Xg_hc$J(5XK%c`7U
zQ3w+TirQ1Ebwl&7(!^`gD_5EReA=Tt-d&A*?6!Wzu_wawXK(lBzV-8Yvx!vah)3qN
zIi}7@F17g*;LfNIV-)>^=|p$)*g%j~i8v;Sid!gR!aUOC{06P~#@}mu`jJbFo-HZ~
z;}u}l7PEAx
znNqZRNKLJrMdjHtG`bjp?z|e4hvle=QnN!-dTk1H>A&}&y##mG?!%Rm&VlxVv)YP0KzF~R^w{Pqx(;>2H#W9~oR{+Y=xs_EGE>qb#rnICjM&x}_7E`&8
zR=%4p7DvT=DXw!_lNOf0Y%Ql{sQQ^PJ#T+${_(BBcQ&jb>Q|7Ce~{2!f!a(dh0{Lp
zJU8;(|A|$h*-x=QF}W2@A#pZ!aeo9BYmHxJ@U%*(qwM~1VHS_bEDN8$#?zYLj@n*U
zcw3ddSD1v*%y{7Bhp>zO(nyuCp*fEtY+bzw@eK=nfB3zyk>AS1M1L|i^R@E#+BvHn
zgQ28>jm8E9j8OsmIx5&?l$k!hX@Q~=~l{)({zSF};Cf=4pKW+W+mC`)+0%urn8(biv(X_++q@bi)3R$F?K4XS
zd}PX?C2iyCKhMkk9FF}@<*p(%%RUA9I1j9X_YHO=JBc`#)XT8{AVf}QB9gWf76unn
zp@`YmddB9Q*mq|h`l-%dt(*SvE{3j}j=2SDI{O@Lp;eA6&I$?K2w8z^PN>V0>1_+y
z#E7({kIrGzii@KRKspwVXY_pC3;_@mIeL#Mk9m5VF=udNT<@ct4USERU$l$w^0SpI
z&H;DxZkOP*pBMyZ#aT6{T+Py_e~1-4qq;3T7Xflx<`8ARqy7hkG@eOaJk_(uwQuYr
zPjo80(g+Gia89PvHk)t;m`b7%=jv1SDLIhc_e%LW6gx<5<^Ipq3m4n78o8GSr(=CzBjq$v-W!-r43=IT2S
zxIU!CZ&2k`4HNVCffJDT4y4uZ-qEm7H@*zBc|l(_`gVg-Dm$8?OIKUC!^e3lxBGAG
zCsly6kcDDdsQWPD8*xUCD%D?(ju&p~6%4kzxtw5%R~*A5}pEG
z%IR1nQ^as$sA?3V6GmB9uI=P$f{5FqJR-q_&^kF2TxFp=mpp46g#sFAnpo`T6iv`8
z$dX<}D>#9X_l|2FKj3tXrGHP6{m6RUE=U3T5w{dYCmf{!TA@TLx`=mBZqB4NZRGwK
z=LaC4))p$lbzb`oyyOYb>AbfE5Wt)&_vp@PwF@ntG4*JUkHLmBm0EPBofMVMP_mkE
z7|%DdbVtrNt>d+n#9>I{pa$1QM6Flk`@kCxL`&zQ#&&^x>O|?!*m$~A^2_o)xhbsi
z99IhOUgnyFcS-YBPHQjQf8SpCe<@q4e08le+aDX#I^{lGD62jDiHlXJ>Z@lbQbO_G
zf-tni7?&`?Xo#L<7;$c234;(Rt=}OuHwCNC8_R*LhOA1FKtAi>D(K|UXuYmcq7Im3
za`(&ij$+&~GZC9>MNJt99A^HQ+s{h)>S5wpEB5U*YU0oHj*xhn27kRr
zj-Saf%aKDLU?NeHw#e@4Wp-#*mva-kq??!);VDO|m_}-c|F8NZBQTJ)8fPXd{THd+
zwf0!6@N?GnIAX@wez)1e*aqhxKKl^R*8mo_F0sPz<4YQ_CId#jqsk(@)pC2FYlp|`U+*~iEr=`K(p|e*YyH}am&iJc7nU2Tr`ZbZ!+n=j@D{h46nm-cpo03?5V^bcHu{X)%$e7oOI
z4|kBs6czv@`3@I*B4r9KylRF{33AO_=CVdXwvytrW9Ew+|J6P`U3&HVo>0o=aVe#H
znh@8gRAIgFPJfsSNoJfv+>%Pn*5E!{NMA3={*rs=mQ(#Cgeq=Xl{R~^UL*Co{go|FQR)LYVw@TK5^2dudazbyLzTL1bd
z&}|bq`vIOn1}^!8I3|>m@HVY)Vhk7X1D5G_vUIU|;FhgqvPuQ!g$HCb
zO`FNGyee%+B>Y!l0C#f+CP~k9?&&MWLCvN#d3<+lWe0MY4N(Z03vSGSq88MmuC-oD
zbdHO|hyC$C2+5eWcE_BhuzmdMB6!9E8(AHlu2B`m`=+l%IOk2jR_lSGQsAl$DZB4ksH?e)sLgB}0n8
zwZ6XvaG%Y`xvUhzr)8u@V^k!k_h%TL7XI$iaRnO}fi$VLIrChV^S`f`3++Ic;M1(t
zZgxmLR)Xs_b1Eu-`#lTb*HP0L_x&ch>FPsETOI>FOlIlUwgxCGUd9X`_lC5l(XZVz
zsb)`HLY`-Tty`Xsi`JLqh~{4a*V%7o$BBZLtQfrJ@&uA4NW2vITe}jfrVh4BieLGI
zECG%a5#8;0L%9gkR;?By*2xb)a!#}VsOB6g=2wP7T#-eWsTNdToami-fUM7#_A_G2
z=VB$oxqjx!8okuFwZ2Cd;>fJXsp>P$r7gRXR@*73
zKatc6)i6j<6!f^Ob={U-Z*Y;<3qqOp>yo8370@b&!VGt7B~~4F`_j=mi&l|sTN+zw
z#-}LY4n!@Vwd_*+!5*bi3RujUuxEDhL$CRwFE;1CF`Q8V$*@$5lB&yKyV}6h!y&}O
zv!7R~=))I%zyqtJv4yy)mFlfcm~VR3jdSw43OnxT+wphh_g)o^_=HTwyeqeK?5)sz
zCGH7Upeg@7x63Te3QP^Fo?7UZ+r73pPn?aNK_0KBj}|dfo?@EKEO!pRVSNDg$Iv}^
zv#@tkm>c%qAkp-`gjkz>=uk4G)~Y`V8)5DtDi$gL(|qm}n#}YNjUhobrlCZ4De+Vv
z)gv?|6@b8`X9E4L1jIxEQJ)24{=9{8$x9M9{
z#gBfZk%NCtD;vg*+?0gLK~Tn{YiES+p5jDfm%t{EQc^J(>&PzHqDz_eIQ=pm7tGNh$VzXnc6oQxQc
z3(Xo{qKlTINHrxw8r$C4I>JO^XWf)lKHoY?*(Lt_iQ)8!Z#hWiOvtN#>7^DK3p0T;
zx#|(0ZfBM$T6IB@{g>=VKY=mx5uR7&lM+^QjtrL>B)7mLl2xuK!8|KALVN`6<>^3!
zH4~DntjuaQrtdA8#k*mv8-;h?M%-wPfBxq05k87$HLtC^m}*%4{7^WZ%fIh_q?%(6
zF5D=l#%j?vrg#ztu?pbADzX*X+RZ@XG0-6@2)mvZ_t-z&X89$(vRjZDGoSV2!abzf
z7(LHjzd!Pix;tkdY*^11sUfos<${+!Y|V9kIyovJV0fF*jl%tCpTB0MgOegn|J*mGotXO|@AD1K(>ebPS)J6$p6trf2DA~H(_z01QdHzp#;0NuEMuKd
zl!4~D)-;k9+TyT~H9Rb6^eovGF$XQjQuEmtku=}ggtTRylz!9AY0Bo%Jw
z2;o1M+sbx+ys7mM0t$#WqGo1zeCdA#??ov^toDiVtD&O&J?*9*Br_dPtO-n*V;qjXV>`&o_
zne0l--Fg=C2cvHfd+0*Ccr3~igC-hTSgNeNQGKx3QD~RF=CiM8)MuA~DP(8%xxlFG
zcwfi{0iUau^5;wSN@{!Zy7S(Iqn|BW{P|vrTCO&oxVOwf(qZ5=zVKi@#<%fy
zPc0F%jkE)6W4WMXt%b)wx|bmXts&odMy*|u`7T!JiQda39bpN;E|f;Gb0~|^>bmm{y8mlJ^#r@L0XbR-IpJJ
zs-ooAc#Y=X{~(GP5)<%e%qdX5`=64K$Oo1!Hw^%N$S;hV{Dj&$F8Y;bx*Q8)6Vu&lbY19q!B2jdFL-o
z5rjk#BwN!hy$1bKL0uqlBqe_UeQk(KfxNMgk#WKi+w|_0ZImPaI=!ho!$&wpVzv5}
zV%Z2u`U1PMb_+vJ>X8Y{C^%*}uFVPqoYUc~d8&Ax7BMJfag{5vKt|h=CZT?=sjY>M
zSb44dGpG1eWkN_%zV_F+-J`Ct@o9kE^8JXR<^{uMoXz(Yw2D_n;xPWZ#Ku1d@3*Tj
zw<@Qj0l*VuIYm+02RsvxFD?1s7tQ5U>ZIJmvkkkl^Ro??Ct`Mke-Kt3+rX^8J_}Wu
zqHk{ppqxh|w(9VKJ9Fxc)C;UkLo(LvS{A7%739edQ|rsE8J~19RSwWJWBAb9GwGCy){Hn(E`I4xq^_pRAcZ
z#rFNW@RE1%L8vfcD|dE?uLb@R7#u*!J4Gcb3Qzzsfj;8kgptxH>nRahc@#5EqD|&<
zu%KgfMWTam5{Ps4hA%KPDPrMPikmr7I%szEk*uiHeug9TouM#I3}D%nDj+Q>N=qiR
zIMnxBSeI-^(_AOlZL$gnDVH^HopwrRg)Nx^m>rsl$k}DRzjE0{>>GWBSF#j|3a0k0
zWR@i#;?qw^aPm$0+F3@Uog2)>8ME6@XhjlKG*B&k;|WCS@|>fx7E}Z^ZGeC08VwPR
zQwSFEokwUfLsL9r73w^Cfv$H7ji_n12;7C
zeO}}&C+rYD1K0x>OvwR#=%ji0FuMf2Z$z+{dL4naUp|zfq*Pd4CUD`&?4&VvbSeTX
zV2|8lJ@}aIzF~z=^f|iV=-_D-CvQ?fJ;6W=^%gEMB5l*-n`Ow#YT{lDjLSdvxDP1R
zSjZOMzoB9ndJrA9tSYhJ9HAI4VDiv$`!3|sjYE_#Oo3>mPx_E{bdVp(SivX8k%JXG
zYhki3Vs8#6vi55dzO&%)79NXBxa9tvhLZlCidli3bGOZx9mp
zZ*DC8I+=GS-Mx6hH$?Mpf<50ExA(k&ls%z6UAPJRxn8h{+eA+1Y9jkYUc8Ij7{s&B
zUKG}yf<3_*H|aVzM6xc-JK$;J$?d#Jjey}t?wd{kXD8mu4JoL>&9cEg{pRS)$4t&M
zjD8n}@F4p(vC-#}L|!gIpII*K=nzm)P>?W?&@i7R6!d>dE)dW#7?`l=Smf;3WE7Na
z9O7`CI8Xj~u03|1cLo$3SU3(3EIxH)rTMBV0oU{aVsBeIE==4Vp367~vW
zRc0x&*RbegC=r5AOLvz^oJl?tWO@(#EtgFD|E;F*!8+=vBpBgd
z7oGtgOxRGobtD#kC%`E&E9Lvn!wD$&W{<;59oGX?ONCY{)~;~JLW07mB+=}C_Aq_D
z@qQ9A8I~AKWF`o_A&+daL-pE*1IF#VcrJnz@9;FDE4$;zY?L`w`?Y`CcT_nkaVsX&
z)^G@8JRyQf8w7jsw4X
zPkDuhnOnxfyExwXq6d9QHiMO`j2Rk*Ev+?w_WD2g@zVefm9V*=G{wp|jMF^tcEIPS
z#^3rq{6*~PSDN@;bYps(gYYlg%i6W|KvV8+JD)F`Vogfy(_5qSaIS!|a9mNd}n54vEj>g~X
z(^lfXEck-Ko(NrevToCIe8ep8%b5#|uX?^;^z*Zq|Hf3MFE{;0dza>J_=uxV^ffB!
zem}Q96qbO9@xw1dxcLVm>it5T;~_}?XSgP}&PC1R(DtY8m>wvps)Vun@P&P5pkp*3
z_PDC^zzj64bC_?x%6LF`=*eS=7M>iRl8ybyQgEwSL2wr!CeNzph7%fLDHo(#Mm9Ck
z&#LT)s-v+651Vj@o$AS8vPsosPPj
znM(9qf2Y1639Z^UZBjDhm=$6#Q#7kTMw_9)?IWuh^QLiEd#oso^8PVSunwVkf$91v
zaC01?Ya`yrT!052yd8t#$F;3`J<{Hl8n>bw>U*>{X5YS#wf+2r))d*8w8jS@IuN4q
zM)`$5WXHEK>Eu;;wC;V(sAlz9qelL21V!SPOu2fI}f
z9!;q|QD{DBrUvoy!}dZGfcZKCBMrBdSOhGH%ve!m97}~Q7aiR)pB?*+lvtZJwJGb7
zuzs}6rlLpYkwGv*s8KPT@ACmJ3lU`(^SQS?Hbqo
zkw8gX)w{cI7Bx}2*Aux4C)SNHa`L=0P%=3yjd`uFf)sPQoCVeWX&Wox`*&f4-7OPk
zb@0NKguqI54G8jkXZ(oZG`0{|FsF?qQz-($NWO);aIMV`k(0M8v1Jh1Y<3Gz;Hu+S
z$3boplN6*ZYhh&$q;vZ26;^|y<<*B1hv^CXF
zG5=Nv*8~wLr#lVi^}F&pG;=Oe_dwzEK$8h-iP11&5~Hq^iVg_@io
zHy2dMj_>g&`_EuyujeZ$04I+6u~w6?2dTzNiG+~)EneS2-z{ufr@drW8J!8st){m<
zpgW;24xN{3w37r=gTO$a^rC`VV=7VbBPO2k!avs
zM`d3ZF+5w>W>eUFKq5Vu&}(0FZh#xE=Zm08Fg)0vmw@E@t{{xWy#^_+VVWq2Nl?h^
zetRQ_&ADAOU*O6yX9PZGQF3>E{Zir|gcZx<3jmn>eS!(Y=qnBGBVz7H-oQdroMP7`
z*QzN=i#_NxE2Z7(3o{|HBHxkp@2WZ@90GejPWCAioUgZRZy?&JVO{j-tnCI)SfOn0
z;Hu5bjI!Cw%9OuVsX*<5r1-GEy0(lQIaf2WuaYc6=Z>F`FF}_Vc6Rpd75G*4D{CzW
zm={qXYdqN^?yE1`eXIP0k&r>hP6XZ;B>Qi1Cg#w>^FY<6@ud1O!$8KrXbP6ysd|;U
z1$$~O&cCO)*_&qPl@!#*IMYxsr-d53azl^MWWlHWUVDynssW?5wlxHde9xV)#z#Le
zF16r%AS_#@n~`a=q9y)vtS!uR;tWq)VsG}2uIN5Nc2yN|(PC%VD2>}D?Ung_7`~qW
zc!}M7Lt@pMX@29Jn_V1$z$
zRf;SE_nzJ?yZbbArT~SmEh|#>>hl!=J7YrJ$L6>o$kBBYfOe}EnwR4Is}~BwIwk
zWko@k6Y8JQFaPHau3?al3)*6?MI$=$)#Igj>=9+#b_}TlPUzK}9=wQ^oi8`#mJ6!j
zpBXmI|CZtRTf4@=utwQ1`qA_V9FN#F1=O9i)}B4u!>TysQ3nxu))2yz68E$y5Lm2c
zBCRws>3gB(qXW&Zkyjyqv74-NvBQHpZ{DYVLiuy?y
z0Z<8BZf@?C;t|=QDBhhEVLW63h9Cb!#cy>Unvv}jh`HGh;Ta)ddce1~3Ug#vCd+m^
z5hXv7LzczF;JP=QJiU_m)2
zFggG$$k4xE<<0XFch!IvC-n;@SFNBBPNp
z@9WFV^GAE*_#KMPv%yJA?&{4xi63M6G_60DETyJGnX;`U(kI+zEaBsI*5a+`^#i7>
z71J41t2$S#c;t^Y=VjA&qbCnZ>12Mm^0o({Z2h77lU6Wx`*^5l=kKTW8jKYMl8FL)
zJ|BT%GaiTr{8)-y`A3`KwvWhcYEk++%urFsMV(Gg-hA_m@jL5zn|PXH=E>p%zXz5T
zFVM1!;z7g|yM`XUyXKdVwbSfxi&XGWxrMrhx#H@4Lo??yNIHaO9do=Oz}cGg!GIoi
z_v@0^7ziAF=#$*81a`L_8qj!%l+%a21}o>Ba^A5ME8ur!_=CU8x+0c()KeKE!1sv&tqRP5y`q?A{!RgU~IknMV
zgDWdr`WiM}=iJtVW?VP969Fo%R-$+H4iPgr9bws0v%zBwottnYJ-5aB5F=)_Sb$9P=JLy6VZRzivg=TJ@y
z@TOf;P7V@M>$${XY^a0Sh+{Y)b1f9-Aq)K@G6`}l=>%>>F&49?T<~AT{9wWyPT3Yl
z{1<#Tg8K8!mnK@x5q`$#B~3y`%3;wXAs|bB*3TLjwl%0Nf;EoQY&72XPx%=CL68If
zK_ryBx37RUJ;*8;$+?E?T(=>rN@(K(5H{PG)Z&E)&7+T+F#)b3*3U6kvTq}m=QU39
zFpS*7W9Ix>Hw!w;6H>p~KOc~XVao|kn<+yW_ZKm{(tqBq4NGTm)~?450kJ?|tS0kM
zHi0$@Qs!T`s|d|A#_^2$P8$yl$!OzbRLe~$iN|8MRH8rc%5XVsH(y-bVtKLd{UaE8
zIhjwb>#y&Bxo%BsU<8Z{S$e(R5Buax@;hn!pj-?#U;IvpOg-r2t(!Ko;@RNaP;lRt
zBqfL`b9{Yj?K^SI5r^6MP5T#Cl$i6QofpR=tyYJFQ)+i2
zURV+$1m-6VKL_TmVLHqtsb(rY)!6=oHjHRd;0=$jGr((ey{
zk{90bNTrwr<{C0xq*1C?p7;f3sCS(f9PIi
z;h;J|8q-Y!)c~}tf|{vD{z1sU=I`f=(b7+EWor}jj>Ufn^oAq6m&x=dkK
z4RQPB^IKZ`f?&0%JksH
zYq0cN7&$UDar47QtYj7ZdB9&5-AUT3`NaZm6o(;xM2PY1n$7C|ztJ2fRnN9CrrMH_pgZagJVlU`^`pWh+|ARnYe>I^lMj(4rJlh~mx-yN&
z8|N6y`0HsE#78Q$DLTsv_&CuBi#69
zS5@v-uukG#rft&%nesB0+K!bpfo9aNQQ+J_2S?=PuM273X@P~Q7$WZTEk3i>Njmq1{UNraOV|KpEIv|>Np+}w~E
z9z@IUgtVfjGxzu>`8H15r~LGxVe4UHFo%4@2v;?imA+;sM1o3lZZ1kO?PR!`aB6N?z_bqB9LvD
z!d`K?v%w3|*OfmtZ*~J~RToZ9zPJ5PSp`@wty|(2$B<#O^V!y;=G$OKw5m>rhzWh5
zKFFH1@o6KvS
zHi7zV=KlDjK}W&EFUQO>)Ui&Zh1gKK{V=CaA);Kt2~6PWBrgZ=msoW*y3s(of@f`?
z|D&{Vn0+nPUy><)^T#oo`S0C>$0w&RCyY4U)`7fMg5nuNxJ+EXhZa`;FpVi?w5j;K
zw`<3i*$r5zDVgOAXEb&mB5r6c9Cyt`5eAu>P^^cg79B}4y#Nkj{WHHCT3HUhK?={i
z!eERIz1j$px~95~X(GCHteX^UDI4}%%JI$7Kt6eIVnYRT)U4)IkJ}sA;_M32Fj#sp
zvYL&|c$5oM}^QXKe>#mJLsD27wph0w2ydx6EfI(l=u
zWPMBxn8We`uJvX|C^4~$egvKN=6E={I@Nk`x}CWZL039-)uWAAS2J;yqDfj=UNlkL
z3-P#iye{tcuJypA8xQ^=xkSdgS_`TA_72tSqm@gN`l2+q`e46*5I8kpPw)>qs$ZJP
z#(J;vnNU}l&ljKnj@Mqb-<%%}XTAIRtymA8Su39K0Ah&;W?>~K6j1o&808#y^~1zP
z!e?qr)v7eY*5uuV#|gBL_e>=F*B{R90xu2q-D$enhZ(0%!u5B7mc~O){>nnTVbxiy
zoN2BiwRF6=LPhKtSMF#1I=(d6{~+S()X!?a4K$bq_!!~|3Nkj2HRS+pX4}rSHQ3yH
zIvb12>LQ4rqug
zKp6&J5(()3t?#6&>S^29+so%^3T{oVkt*~4dfE}{*ZrDvvY-G5Tkf^LL5_7*Efr?
z%P?3KBPYHr+I8PHuBeN<_)xuW@9jxC@W>12-pw?VA>ch+aj<)r@?q=&H_-O9>>t_(q=
zbr9J)?@{X)#ME@{aa+D}9bYRR1z65TiG2;Tb&0(&*c%!bT26#>D1t8v%S!Ij9cxrtYnODmS<{~!`sV=Et?2V3ANrbsdO
zDoZ$t9q_dX66ehV_wKG*8Ns$_PR>*$`Y>bt9m2f0wY70I(s+LqIlDWIhbEhE?`ric
z6*sYH3NM#;_qd;r)TU!MV|)st`X@=+;OEN2Lhd<43M518$o~Jj5hwPp~TkJ
zgoJeB4Mzm>Xu5-oLS9`_C}dfX;|qWlVnaXPCf&TPsa76N`n{y#C-!J`t2ATC=9;+5%H<%^VoV$nzZ2lFd);J!O$0!
zEn9*WmSktApPucYnafWij$(Pjj?$*
z)dL?fSzS*fx5N-9Hm7Q#_4ShP=jK_%;=2Lru$;CggF?}#YkgEG+g1W+lcqBp|2<7H
zuFEn!`o-pH)LQq5qrs8?LTBLcr;AS4dg%@zXQl}t@7mb9H7;1J4@f+Qfu(KrM*WmuSi$0+}u|_
z-B*uCD88Zh2X&v{uFvy6J2zo5tCw!`T{D0qhrVgPABzGbF2*VFPGbqS2putcliO>R2=
zIKRpW$hN0K!x;7T8$4J>e~)D%V_jd9sb^&`O2W(QIak8W#|E#Yr=?{Jt|!&4GG9Y`
zJBPB?%?>5vA4Xv|jf=_hvPBpq%5F8;h1FK|d!skJq^l;oG5EC%5@uxqHrW=BK=x6&
z6C-eZ$iYP>q<
zQZJ^PUYjtwm~b*do_N;BvRd-PGMn-^^t=m^j#n@-^48F=DOze*z)OC5$NoMG?vohm
zJ(?dU`#ab(l-wUHu0#V7IXhgHo9*Ikb73=cefJN*)T>%E$i=K}#%Wc2r!loYw>K_2
zuMmD+m)lajD(W@)Fj)vV__to&FB-f#nkmU!FmRS!n6(Vd?(0uA1u_mf}%t$mkfhN9a#XxHSy+7bmwpN_vfz#Of88*=R%L)+M>In7d=tD^9{G
zgZQ*>g1U%CnYA~#%N0abg=1(y`;~~0x4vGP%<>Yf+lWU89d*^q_u7PS$pmQ@JU0gP0Rvc~)sOs9TAXArTD`&7GE{F74|ytroTW
z7|hGXG-O)WT1?+!KI~OqTi;{Z0iObV%Od)PhLc7e?OG%<2jMNg$aTGEedAZpU2qk_jGS0?y
z*W;Oy{>weqD}g5lp4JBRymXd}jbF~g{#Ms*_C%p
z>Gj~%_nH1mzoQH@0FNTmMn!jYl!vD3HVUkCP-(|vZmD(i6#J>UAa2ZiEVJC@=*rAv
z?!rVw+;SFgWoGB}PhfoyaYzyKYe;jLnsPaVWIrd={+u(U@o%i+wx+=r+Wh(MZVAJ3lp@U0`iDY(5Zpu-
zshJyKk(eS9T5{BJ%2KchwH{n2xnmI#JF%OG*{{@XlD}Y|#G?k5S)+vh8vLThveM$~
zdDYp6AP|Z9ebu3{x-U%_mHyG6>V)r$Vw-N3*?cLe^lVj*eAL?D4San`Ws^zFTPTev3ucHFtY6@m}I5*}S&jMjf3
zX5$z?k(>*;$P172T{pyme&RT`NT`RfSb4ZbMUZl;7XEzqFp+KN#I^Jz|=>lQ%cmpL26?n8pgr
zMy3xLL{=qS$0O@V#N>u_H6Jf$^4~1I!zT#)S_iEn8Wg5x-(ysY()1I
z31uOp)y6vUn}E>KvDKrgL;nEv+}sv!8&zRDpWpURV=MO6Kk24HN&43gHKe(~YY!%i
zDbH4wWlyNxl;nEmjSY3Qd|8q=b(Q(wAO5|V{{T)`KKk+K`nTrStH(l)M;dwVD)x8v
zu2B)}MqwX8Uip=KF4L
zZ|taTzsIrl_83ocdlLzkWY&_*$K8K{`pVvA<&F(ym&);iLg6LZ0yTxARk-C=Y`Fj=
zvH2%taZ!VS)jJ=e8&(rqmaRSULIa6gRjgi$x
z@iVzUQOffvim`6|SeIp8LkmS+#Sr=q<)BH%V|s6a&s9{uOLK7ic5rR@eBE
z>QTk{*I?Ypvd!*pZ|&b>H#avo_6dkK{Wx&PYD~cl$9LR-P&GOA@?A=~70
zyM95fkEyLv>*d$`N}ke`rBmwooi(v*N~M-hn7!Y%9E}>>$vbM!pB$=NZv%fJkX6kW
z<-Lt(nU)*a&U4KYzrwWREFM8>uJs*%E*?3309%ilU2T#n$fs_DlpABgfC6!c$yI~YW|e?pqtMiQeQ1~lp@hQGPrqzd
zjRw}W8^$p(`ilzgOnmD6RnY8O$>nv8bF>qEw>S14$Myv2zwNE3I2s0=p=e$;qAHdH
zDQlT|yn~jqGOO5Ia>w^&g&+@Us~u83%7N@_O5+_hfg>?u1k?lHXZ|w?fr*Z)mF4J_
z9GcvJrd06>kU2r_awNaxTjX(zU0Sd5eM<+YACvra{9mle%{s}V-WIsx*k_Hq(^#OQ
zuIG+ZMM$-dF;OF9qsGRrEeIg1V?wpI*|#99+}zwI7^BxAKgVpY>^)1WRvkyCErIJ`
z;WalmxT^q*XBlH{l_r3UVKd~F_7+lvvsRTTU7rb#!GNoqEqLeQnPdaWIVNtl!#hq@J6Ok~30G|<_aV9EFMrcg=ILNW-Ce_Ar6^M)N
zykf9_0Zjtd$hXFi$DS5arB`(adoma>s9DR#49&({THIS}HderX-lmMy6CL!`-RT~A
zCyl~@IX+6)FUp^*kocWF5v;Pc8Mn)>mYzzg$Q$FAHA?)hmE+^3okQKW7hylX``3F<
zZlB46AgS^XIgO3x4`{^4#_DzH$TsR%###QWp1oGf2$cv)+g$5BuE73L*2M?68T?^*
zeZ{D1ik+U+6)Ln-6zqG7S_U%JZXUsL#PS;7ew~QfnHbcq70p8$G1O1aTT-(19+}YdzRyH>{=I$-IZk(q+9Aykz-jk7L!__0D1nF|$8Sap1ZP&4dX%5lQL*D$;#p#7jcip>U#F@JHG>S#
z9V;w2;5X$fZUyWu0z!sxGl>K&3oFSUB*+0@EAn=HsZKR@#<&4yR)yJ9^R%S^0?L7}
zW{Fc6eMY-unVQ@JYHPCyJ|>LeJS{wNfD_z6z^D@`!qzJ-844VGlAvznMXL>>7Tjs5
zyN~f!O_jlxh8w9f*8R&zu%kkdV&ekePfA3zR>sD8^o+&F1*;`#7F=#MqLJYfi3Zc4
zhf8Dfj;Csyo9)r4^JgcaH!`y8$4t(>V;}h}qTMeovazW;9b2h=BGJ@7ro6ySwSOCq
z+QKeBLmX@s0yoAwmb1mPRXFmaijwTA=OQ7q
zaIt~eN@W>AYi9}&-pbL&p?_-Kg#Q2;*PM2=@lK_X_ubJG5LKU>Mquf<#}^XYg<`yF
zI)$|l9%bLjp@h$vsVE^3{Oi<5DJ#Y{uO(HrefGNE$~hjBh%vFdh-X{MTxEQl7}3|ID=fX9r)Ox_J=pJwO?TY+iC*9lE=tnNc^{ct0^d%8aa9I3y9Nm;D%IG9
zoce@rYJ|pN?Pk3y13Y6^)9j@9?_bOy5NS!ota!6gV?*+wn5noaUB8+YuoY)BfzwcD
z9e4i#8f|5RIYuCi{*h;rj+1FUKc>@$EaN(b0jo}QH3RkU5!x}_192*WXT;X!6{=R`
zGk^;oE
z=#4XiOKr~s1#r|S8RHX#B71<%`gK=^y~4UYwF{FqH#x+tv`lYFNE5~caG+UjQ_Ec1
zH)8SV{b@K;n6QtH8`Hc`IOS`-%PcymbISDA|FL{025H4@6oOmW~)uceN4>drFt%H
zciq=~je~>MUX{D4jEMMuQB8Dsjm|g|xm8VJ@dWSlD)i0dy2G_0j{J^bi8nHzmA#D)
z*todwk$y68su`J|jf~Y=uDxV2r`Zmiu=+kH)LQfkUmn&qWL;Ee^`1Vl7D-}I6EM#Nz^_7xm
zR>gJ16&XiWbW>J4+f6}3KNZNx{M?o*o0V->Q;gICRwG;x9qd40Fh_1EetUn6fY`#M
zS@GMgbrFYM7`YkE{yNOAv`(jwKpDXe7|hiHWx5tWkeRr!kIS<|1pd8(%v-wB);CN2
zO1hpeg5t1=sjqdRbC?yDoZrI_9r}fX#f@G^Qyc#Pm?Bfu>b764WDhE1CY(ERZ_=%d
z_=9^VzbLw9b6h9cO?LMmjtk8bkQxT>o{2rjcQ~IYR6sXXVdjKzF!+yeALlO*B;P<
z{GMHxqT(8iG1HF`*-&5CW
z@vO;fIzg$|8oFUM2&0_UqIoP*mj{pTG||`)jl?BmzUR0}if1S-ocA6Q)AF2Wh{at8
z8YucI+Z?AjZ=P9i78YaH(LC7YGyNx1Z9aKQpSe{ib8L8$dki+Lg^p;adfksXehUtjRw`iH4b3tV3O?U+Y_giH56oUN$(ttCyB#
zSkbIIEjOsPu!&%3phA<3Z3xG3dW6l*L+~{WOnJ$5qQBns7|rGcC{jk)|`
zZT4jQ2W(r#H+kPFtx=C(Ex`cnPNKKw&Rm7uT@!1TIXe&BdOZ&_jO7`oP7OITAEsl_
z*)O4M9~Jw#loIEcDs{gddV}q=;;zO%yO?7S6Z(g+8-lR~U``?rQ*&{QIRL|WS!MqKi(ASw
z!b$AY@+0ODHvltdVzy(4ZIoEn7a|VDyMIY8{0m2!%lci9WMYS>XA9Gm#K6zE_qSA!#Q*(23asCkdw(?)Ges?)W(Wi3n)10HfuAIq-)AF(z+7^!Up3dSO
zM`&7O;9T|dA-~3znvW2c<0FjTmp>$J#_S#gxYLv*_Vqb#C)i<|g3|??oafJGY4Z_F
zPp90of{jH&#cNzu@P2kNX{aK>_8CdjXL`tU>6>t`o2UJs)o#@`Cs7VOpvJajHz-RA
z==t)26ZJ3Gt@=U(n4zJg3N1w<@|To>k_a
zGXet^uqBFoZahgwPX>J_WTVq@Q#Ca>okeXoP_xTs;cv~YC3mt+V;I2!pVJ-Ai-G5|
zs+AS`ek<6Ot!oT*-0F4Rk7`@8-GC5AFh-i1F=b|Ma-63qT63Ir-{KRwsZCtlNm#bV
zXaqzoRILCsU*yBtfJK66>)&csF0RVWEs<|Iy-qCC9QfNIcwN7_BLoK4a
z6R7lISAD=?>`di3!tp-V?0v%vPnZO*u2aYs3CCx~wGEgA1Gf8UPC{~m#-2k1EY!Nz
zvQ|R9$`s--66ye4zi00i?4Wzo12Z;_Sbfc^W!O2ZKbi&
zkEo)IYb%~3mTja~s#2*!mZ`T`OA{umssYSe*{1|pW-%%ve0B~GW6Bij-W7i_?+4oXiqsu>0RgjJmuwQA6=8st$Eqg6NO9LtQPv*
zY8xWkdM#8@jQ%{9KdaR;{+f$#YZ;i?i}!&5`;29uGPEp3e#Vj=u|?!oq@`}VpV}gP
z!#DSaI^_Van}xOejwv*B~RP90W+f0T2poHacc=oVfuZy
z_BYSi4}Hze!5grM2`^h9J*%>YqaYjaxK<85o;SOVsb8A5Dr_J_+f$4`vrVR^ve5O~
zio<+Mt!5Fazj$w{RncvN*?vxgt+@%%GrS|M@OUPN9#*A=aXb;(s
z%aw>6E)KB9KaKCRlYOe3rxZxu$?v|vO?}(gGQVY-6WkfT#LdI*R^<;W?2O!7lQg!j
za_nYUX{vo1mHKc9$E$^XJy=tYaP7*Wxzv?cb)Z|3kixpA=H}+$hv)6zW8ZUq!;u>2
zVBN9r@z-Sm@h0XAZo4r#Eddensq)!kCqEuMp%%u-1uU&d4tX=t#~1J3r#SxMQXwiq
zNk}nSk}dG$N`c6>gsmvetLg0e+dOGnEWof*nYq5QZt_{!8M(gLLe;Yy4~9M2FXgu~
z`zulR0!L%q_aAzg%LdkQ3<^zlA8Ig{C%MXU6Py;XhMMhPr6AWz-nQcSrFJ+AXXlZV
zWtK%w6wUV&``q6CoN|VfO#R*DQT(A9Ft#fOO
z*Tml}#lK{p+k^u&L_&lEs7&%oGCXX(Pu5OQwsR0z1Z<5W@!Wl<;PyR-W7>~&G_&B$Z}vCZ%`XAmW=L&Q!RCK>vOd^Zi_ywPhH#Y60+iYFU9VG9?FWtWRL7n;O-R7Ke?ncUClPX7Rj3D-=T7xY=R
zpT!&gkfzs@)fT7Oxn{=;?DW|e*L+?q?=J}OcZf&j@|0}aS0$Q7WfaF^v|iR~c8xi7
zYIWHz{{WIM#WuPl{SKFG?ng*+5;l!tIuwn@=2|TlgxYwDa&(9I@8OculzgmRV)13>
z3g59ex4^Xq?
zHyg`C6KlZ-x+$NHPR)mCp4og32=rw94m)u|Lsg9~E3xBMb(#_=Rq`~7%4hmEvtwtZ
z&5YYwvJ$jvDtC=F*(XG$e0*Z8*v+g(#*VP@y1WYG+99hk&E+%2b#6CPAW1X9C8<3(XEjW{{T
zO1fD5hU*=R6l!UI+5}yB;eCyaUP%0!pRoyWs>aj7V`9X-n)}HQ#Ke0>b-kD{gNW065yTm%?#>kSo
z=$4+(IqNCqVv6Tw^N@sTc^uKbzXg4iQ5jJ^ILL5u#E$|ZMPEdeQ~o02;>0J#DW#yIB#@6#=q2;L`Eofnu&E6Cer8on@(B6yCT;VK@CWJ~ed
z@M%wHLi@zyxN+L+G10YTG@}xzo|>QgAswo*s_tBx#9_OnM@Ef^qJ_mu)p4Y4BMWqm
zk%H{9u)G~HjS4)qA9(Ot7&&&0Ju2>5N5L&L-SC`uBTu>ZOHMKpu7bItbAiY>pk;&YVV8alWqGND*N|XyB!HE*YWn2Wh46^czDuv
zn;fs)vQSbQtUNId5RTPcYl0bN;pzOLLATkd1q`p#+-}Ol7E)w=*739CiL2+05khyt
zB8ITBDIRfW`#g9xBe`~ZdwY0-eS3DDj9&PfVx>}8>MH3M11P*zlhYz1E7$l@3_D6D-=u!ME8s>?}vBdZ(!(*q_;Dk^3xoqQvO%
z@wYZ_L)14L*#7gm;g<1u-Nwt}#5OiwB{94uh(~c4DCNa6T0cjpG&a|g&pa$r#bpW>
zZLSj3_(PPWDv7w>S$KKA8?k&Z*!d)Q9JpEJ(~i~0F*5HmW3DR^)A?eGTx!YbeI7lp
zP;(m@zv$jAPqJFS1LL9Ri#0VipSq*JbdW)O?ehkgXy6Y-QdUmpwUlFK1Cx%@xJ5%bjtG6R!eynMFxc92`9HzhOGz
zw4$?FMx39|6@}olYJCkgm%_&Hm6es4vc$yiEYz*v#IItw(a^DF@{2bLSIHyfeUgn~
zi;I5B;T9${a~z~Ob=aL^v0m?A1q+&Gj!aT4-A9vfvnbLRd|qt7X-M2|EOP$sKP
zD5HXImLRm3o(uXtHDq|>6XBLoq^40=Omlg%iYp5qpyXYjlFCQpiyp|tQA5O)^RXha
zW}@?WvTn~DZP{3SWpH24e4P>H;T++kjk-=zW6hLGv1d3|HJhWck&fo9Q8etVC7MMI
zBz8qt8Eb-sKJbx5qhuDDbB$C}eUVt9qoT@8PK{V=4-Qqx{Qm%GQ%@FIdA!yZ=IBz{
zWtXzgBjE8z=XDKP^PZ37zoIb8EYxs3Yb?}QhE7d0&uWc-XW-8li%Pc2LzH6a=7fsE
zf0b;llFb!_v0FU6w2-4piYTG1EJ;wYHa>VG4I(JgQ1FpO2_u2x10N^$dTK;`5s5_-
zikD>kUYp=vNv_4^Um|zWLPzcLEWNYn>uqN9{{VWisMKD|8nG%U^7cYBuPCy_p5}*+
zmMM{LSghG4sZ%`{7IoRBd=d37kdsT2PCi}}Aw6C8KfTRbBg(~=a6PtNj71F#be@z(
z4ZC?_8WaT@Lt7ph5^ol1(c`7H3`)sDEPe&>cwwZHwMV4j>~Ejsg!ORyD|AP?@kJFw
zBZrF}WgoeVELJSbGUF665_abZH+2?0hjdxvbp@4oP+?*?!gG<(G;p6lgRO;~p|YWLG$qE7bDv3Vn^x3?JfYQ9nx;qr7|7T&ur2v~SzkrKxtY-tJz
zLsmZMjBByFKfI#PBsCs4@HqVjfkm&AQQ4(4N+#puS0;MWoR*%8;8V6plx>bO
zgCWqZbZQeXeeJX7+p%ScL(Jx&xh3j(Iea70e`lpN(-)F)?2OajR8Ylb{T^RFmr+*5
z;+K)?@)1?h>DOl$hKWjYvg3RUZInHW4;AgFxoo=SkC&;I*bIx6mt3;BrG*f#fdr=7nL#j
zIJADAZg+UHZ}V7jvN7Y>OBI#bTWbybGxMnQB{}mW)KUCJe$8ywQh2Ti?f#ixXz=+x
zHh+<;t_JLs<6V)HoVhBF&aqgbAuP9E)SU@$f2of{l)Rh@F@aK<>7|A_4VE@1wPKi-
z{Y3u&Nf*%k5{f-JeG%}r%@JNk>2!J$bEz&5N;hj|ixhNrGO2J?_3b`h>3Jz|O5}Pe
zJ7TxO6cl-ozl`=fFPh55?7h@BPtPmh^(FR3(xe
z%PzlY;=kzSmsmu^xv~B2ye*bFn#bpgJxe79*%eD;!Wn0Hq3~ZAZi~(PkINr{%Fc*+
zH9kbX&X4SJa^eJGr?I@Dlrfekv25cEyehKy&dU|iMPV#Uy^UtX_jzR{`%Cgmfpy7$
zLWTAv718~j;`Xdw%ECsRC&8&2udT?vh+-cGkw;^Ae#PR>PH@oEV)6Zz^d$cP=ytj&
zy}e?~y_b$QS*W6gheJXue2Uwoy7nF~7q*JTq$K2UjtVOg%GspoVwTD}H4T&`JgIvX
zip_0!SVeACTPpi5`aLs7J_NQ|Wr>pVMklMQ;OFQ|HHtKpP}&;2Q4iFnxfhl{W1rU5
zweLb7ShDeWMTy@7R6peCNTP?f@j{Q;%=+kFB(}KWExZ?>@u-i4QAky>ZGPPpQCX~f
zRvHr7L&M5P?|;VX#bMzUiYTmB8nI%9vhe+h7nhpK_MXoguX2hW7TVtNW08B=MHEq9
z`&(AWhAppU;{BJFzDpOqU+Ax4YwHcu3GC%3doCkD40Idd)o2dCVR2knZ@KImN
zB^Mm0hgU=gB929el5(PepjlxBta4r%UMmH8<)Hex6a%g@8CyNXDWdpTU(&QK_>i|f
zZ4N5LB6O8*3#;W(k!y;iD`a{fl%G0XWLt%9s-{%>F#44rr#}>PJk+_Np6gDjJn|*%
z+gF&=$}d<-9C2+d3+Ry@5^^CY2*_QHaPYipq1cx0hcl
zna{&vweo8RGJ_fAf2hLc>K#*YFzj-#k!5ZhaamQX(w$1>Cc`VLaKn?C^qV96xgyfE
z8x%)1Ci0~<%l$?V-3J4d#d3!Qvdcg;P!yj^(#n;%{g+1dR+HDx(}>VQ(80A^ILfMv
za|+l<_f{$#EAwT@l53H~a1-V`1o|IVt167I!(~vZ!Bu>(#u_f60Ts^b>NCE6BRxnS
zVKk%sQALxMtSfUII$EfJULFdD)U8;RDx6hR@{U)^oE8efS_;#Apd7D0>w7;7Y2tpL
z+Z3Mub67KIH{%&q7T}FXs`!lrc0(5m?c5U{IggOP9ntKkrdL+^7=<*GO;sB(Zx-T=zBM-LD7hpFHA2muWi9&J;yWX;e5*~^FlMhMm2
zddT))xZ~KSKd6;3Q2jOsxhdGVcd!fa>c_xRk%4WUPX9$=VfH+3btE#Xs)E?La)WF
zTx8e%1qTaeN-9m)Wwd8yWo+%m$$cxAn@$~gE{F6y*8C$KMOAX9)ZJ%Gf{zqWMYd0{
z%TT%v#@yAf4hUDLLbo;*50ch`os%?@45pl_i!r^E*1%{>a47-$7O82YK2l2RSK?NrF7+}^WnO_SI(-OL>n6;fsQtt
z72)Dn2-v9^IOW+uZ<}RhWo2(RPZXlYF{k{YOe;Z5X$8$Dp=`rwUXZmto4YN#g7|I@
zRR%e%pqxa72*n1byn<9?n7A^z5bhV4jV{-Tj)^+22F
ziQRvv$MpR!^0(z>ZVGV1jvMe@T~w+n@YPjm#BQdowP!`?MtS*nS65fg%G`AsZSxKe
z4Y`$7CXDK|I|-yMSXf{lyithRSJi@ltfdlFGNCG}@UJKisA?(LN1c&tvhevicHku+
z)L{BeALA%lzdIn=AGsvN1Eg0h8me5;j575w@Y3LI6`
zuE7r9CsL+W{Q9Hem>pE0(dp)cS|D&vez2q!RsI~iWm>8_J=axSsHhLRZBeb$qTzBF
zfkUno${1fz3cMxF*zU@>i$)mE1%eqtSMu-Y77s|eWGFzNQ|rVNS%>O*gmDfl3x&cAf1zq=X$XRz4Ia@x`Ow0w
z6=DjmL)}$sX}a+giedy>;RR}EB_q}p9H_TtMJF!jG7}JcZWCTP@xThEy%Y@V!+fe$
zK1%0;)>C^XgwfkF%qqQj#*F^}Dk&le
zFmf6lrQpj-#MdM~sQBHEZ`Sw=e3<}(L&b&18B(J0Hng*J4
zMoV5#CGlrZWu=xqe5gi=zUXD$%UsD#!+bXj`7JH5sU=esCgI^rk5mHmH}I*Gz49OQ
z1&RLvSj!Y*G!?Pr4(Zr-jP{j0)U9l>{*+>VR5@)c;wfoYS6iI61-R!ZCwt`H^ve81
zMOAF5tHFK?D|uAova+#Ki`S`Ks><8)Hp;8F0BdegfxE7CD!>Fm;pz?Z#h#i4PxP6Dc_{6fVa<(4P5MZ=xc@KC67r=YD5`xT_;
z>MPW*#1pavA;)TYHC9$szIRv3{!NOq^o9FN_S7}7N&>l5t1D$*D=RB!T)^tQDiAs4
z0>gw5gNAkEg0(QyqV`7t_B;Uq@Ekj-@Yco&P4~_@oa@nJ`{e%sWh!0IjyYxiV}{v!
zN~p?sS?$ApM=cOM$|&H!mqC@)(Ek7&>+(hVt;q|h7bHRaH?_$qr=*P-y;~dPT|s
zIPt?pD^n82p)
z;eMhRN-b#B?59Ky>M&&@WivnE?Yvv7)if24PbTw&1^%C~e9YE5BMC26wpas6C^
zH9C4Kl-6?QgBTzTor!?$-3o37Ea9H$IoQ~N#@x^ishTO%tEgI-(WPx9DF&zj2DPj=
zYN1k9I2;xrJkzU1r#VEU$1eDIT~9R9+0bo1ZqQtNDa9iUce;Tmx`_c?l_5vdlnjHc
zw>XoOH3N+AvWzjLa7DtA!o!4TNkdsJ39Zy!ht*O40F~k5ZX=qs1*{0=lbiq|)yr_p
zU@F`@j4g(9A*-_4yPPJn54l$0zn4LrHCD0BQnQi9#`^-N>CNws*~zkNo&n|x!)87a
z#&^>t4}#nrL1!p<9_XK>tLjuhe-&Vf{XM!Z!!C-sLUZP6997Ev6cS3+i4S*9L2G}Y
ze{0)NbHDdYe!(IUv+diAY=#C(Q0wl<)}6S_=Y`~!fbadaVId=HcHj{
zhsXG}dR^};P7{b!A`IQlib^
zKKVdc`aN)f$Hig8JTy}Mt?snpw7q>S2dH1
zF5zK>q-&rf-+qR3|jtWe0+IspL^;Wg3S74HYOm
z8*{oYLpNb_lnyQ_)_DbaVX;89%Tz&Bqm&!@1%{Y|pH;24ij>FU;sIN8Q{qoLsKq*c
zxuJ{U{5(5PH2g#KgdO^$JmGvCF#IxP5Gp$?ZyJp;3Ctuyal-v5bN>LCDcFu_)b7cp
z-l3mm997|rL0+m`Y~&TcDMdiHITZ6MO?RMI*3?~mertDL#UuS$MT$6QEBVJY`bt%$&c-6V$?=*LZbvB
zisr+e{Vi_$1YGxhkY+K}h5rBwMT%}V0QijM)nQ@=IgxYGHwle)+n7@74yZAf%UJaf
zA+K*FAFopiYhoZ|3$9iiC*e$L7g8}=JE8ug^$x`CWr>A@h#F%}IW0S$4>8SrLsQEq
zR#Uc#GEM`>*`3+$Y^wA7#2iqWEt6EjXJVsEb}DrGG!uIQ|yA
zH%L%(hOPI@SR2}jD^DYq>vdgMLZm3wk*c}@qEBV79vtqRg4(N`HBrS?PHh)4QnhJ1
z7AY`BnGKZja{xGBdz|B
zkMQWaE{HCw*;W*?-4r>sR#Y7Td}o$v#*mrKnp`XBPuF6Qu$a)zXj=*rm08y3n^iO
z6h4m1{9g|fN$#Xc_FoV|{3;bds;s6VVQI9K{4>g69QC-Mv-(rvxcMxoRgW
zDKu&VA&Qxv025j{EK%86Mi@ux#OCGU;?yZRk7-)8HJW*qrw(JPanC@`Fq(hrK;xeg
z-_qE|x7cOB38(C~>SwYi1>xT7P9tH;UBM3A6-+POUb#@kT9x0*rgm1%(e0HKggPq9
z=y>}sUzITch5M}gIa7DBM+X*%
zl=WDSJs}t>sk}j3C<)rGPznWtrc$rLOPv1DuQ|>oFvs?$5X=ndnLTr#4MrFZ9hTCF
z$CopdQ9}T4adod0mpL?8T+c-ONPumXE~KaeN^U3XDgFpGjRB*l6Q1r!y#R|qbiemL
zRJl{+4yz~@Rn95wtA8i5s)P?t8>*fSqH?bmhU
zd022mRJ%Q+ZeXu`<@~O#4HgQ@naMyk2Vy>4c?CoY96B*avL&+MziIyfg;w}X=hUFa
zSx42;7dZxDw^ND_#D`;b>Et!NPX!f6k>5M{GoGC3mwd0CZnoDtWe7WiV@Y0v!>N{1
z(*;SosvJydu}7pfv%!Cd;eOSq)a
zcTmf9S{X)g45+EdK@qQuuBBs+_|ZB$W_6L+ULF_(TbT&kqWmL?xTR@R<2D`$y!S3
zms6BzYWPwW1p6p(gOj4%Ej{vrsG=ypifmgT!U5`{!=`Tv>SBuEDhckO1MkN{E>$zd
zOsJoXY$!d^{Gc1?B?r>Bbw^FzBiNmf3E
z$f{8|I>w-?Da^&tIG@rFc_Mz5A)u`~cB)w0A*58Pax_MHWLCF+6#oFi!jGv~X%qvh
zZ~C{Ae~9FUgDoOQ4yRG78)QpbNEc7k*^xujikSvYCQ+r<-!6;C#1wC*&4XnHfJa5G
zmr%;Dh*ULqTc|QHikBw@Jc=nf3RxHFiR8O>Q_qZWhV(;#Wo1U!s;s=C9`>+w3W3f&
z*G#t+RZyjEl@6+mM*dOonguQu%r6o22xXY`5}A_^MAsWM
z7bcfhl2Glzsr5XXzwi`ktooC%@Q#rG09SQ}@Z&QM2+Vsb+-22UE
zAd*#0dIh-sg4#5(cg~A(P}RGd95hkKQw#SM!BSO3s88MsyHgCQ{+jYD!=`C
zs>Ty~3$LZ#<$vi;cUMiDWw0pU>H3|v?nJaG)
zC@D^!O13mamx4QIMdCCP*{hW5-TYA3WMSHmMK3q9fn@&x5DIftx%n!ij(vxjQ}{vb
zg2{!i8x#aXIVsg0VMY*v)k~^-zN4xyYh(quN0nLyO|1)H*8c$MPNEf01qM`d8ihjIs#R1q#3N8?%?T)sHp+<7
zmya}sXiNIlNIRxPFL-`P@
zfoONxZNn$rpvQpG&?H?Rla00goQVP?^FS7ScEGJhr2KfL5R1p!C)%Xb(>T-ly8Xnk^2~*#akngd|
z#ak+H_EZ_|3*tUcK)xOj_j34;G1AM_w>eo;h7}QUKy-3iT+-9sNO%G&wIKE%cJx$1pK_)ouXP$i^shivU|l_x6%`m(^BhuD6zBo%
zp~|fV7hw%9MXnI;C|h``mdmZ%u{w0hf>s+NNNS8QXQB;zjvg#qkSVXlaC5qfNKE>g
z$_~ntTEItZzQ~L+1PO;4sBj?VUyyGW+O5vgj+J$(q8r^4`hzY*!?lVBn^v}c(HyK1
zW2`cUtyqMG4Xc!L@`HZMMC`KDi0wb*j*_rgM%b#Qrm6?0qVzLgC9TAh&4C*?qVA{=
zjE11Kw{v?hNy=(77|z5FB*va((4e@-6x$D(
zO&_N9H~U9ZbbFzO+MZ{CVlY#m6?kASMD*&P@)w4DX+*508c(kc~H=eSHlQ9^%X53B|U{bcZjEyY8WkjO&yG
z3v-?MS1R%|y7eoZq{;SLK!AXmJqI1Aj){`DQB#00dU=!>!2od6A_x$UCJ@`xBIk+(
z*60qzcPO%(JEt7Hs9=(}8gx`%CM&e`9P+B5$l_8aGWX`8zL8}DsvQ8wGMxir(m*5y
zx|wY=odUAlKvi<*ot3yX>e9i@Y;%(}nW`fy2pvkCXGvQ!AyQPmY5Of9ONADuFMl(-
zI*s`R=Cngxnl;B&CC+f{f$3m5D4()23y(!6oZ$yL1tN;bAFnIr(5j9aaMpnc!Y8|w
zVq!JF;>Apw=ZI?UoW!m5?6$N8Cj0;dKbx=QltN;jXp(Ki0cd4zk%%F0#V6Tp1eIlFYG-}{hsZ99^+3jx%}J?LSI%8i
zlr>AJrz&|=3aa69T#($GzUjUjiKS84;#Fv=S9e68Bl!-6R}FItj$D=3sZs4)o<7TS
zm%?cGE8=1jwSUTuO_f$^e^oCzQ#$8d+0a*mRwJMyFdL+GDtu=maXoD?iUE*bQPoF<
zApmaEMW|FNTRJWO0EMNr9kK!?PCLKtQHCHb>MY)w(Oqf6s`!Ij)fnHx_?Td=T2JL|
z5UEg(qYBd2)YW5usu1s>CO1^Eo|y|vn8yj5Elk3A9Jz(yvACLX1!tNi->1X?b1IGU
z=M&Hr#d4~;0CVQI-BD3X?Ug*LsB2W?80?{qRa9r?dO`*V;R5Xg`ao!5M@EqDkc_&X
z>Ww4civC41x}j8C-WU~eNniU!VK#mED5|6d@m((Hwul^0_RP5QPA)%XiH-WGac?B-
z91qgALE849*NyI}MLF2m#khv8Y;r}dG6ar2zP_rcWwz^HA=1^O39vfwLvjPE5(kKH
z#Ze@p<`O$2=`+=8AW2Z$uZgTbqH8DNw!s6em_>s~5xY@Z+~!a5P+($pmA0opM6cQk
z(VzJ!H&;}qRa8*xj%vAR^COamMCTb$a2aTH^+l`^H-RbIU8n_FjHR>U{{UQ6`#s?#)65#`&4nOiD0RN}l2
z$}rm}sa%!#ZJP<;HBQz0+^3>G5r{SRS~0h=X~74%C=hhwHEttPaLO7KT<}^~QlRC2
z1DVBZXfB$-4&s>3O%a})Tu66CuMOIoQFA?QvKsCLDd%2c^*n0IBJGkol7pIHqlfyTt-Q)U9wCEffVPMV0*2?LQ--7k
zrvyiSXNd4(hd~myv=40ImBn?tEKEVlu5*gUz`r^vb=QdeCY1~j7RL~^YIBr0{k*y=
zna^PRql5aQ4@psmei`M%NTMDay*69M<*yc)K>*Mi(6*oauS$mi-*qi?EjWYO$}ub#
z!`KfdsOCqIB*0r`Ivcd97S7>Fud;9s(%)CQCD%DwL%lwIN~aZ7IAahVl(
zFtDqu)e9Bf8Upyvsjx!}39#8kZuE+kv_nOdoB8g;f7N6Tqu~>Nh`3^d?sJS>PYu7y
zB>>!F_H%G*_jA<1-S2A3VHv0C0kW0_4l!Kx9{jsntdptQZ`lBNIw207YU
zKZOT6C~HfUKOLUJrBjzOw|*4(McpZ7`!4(~%_Fg0{E8~@(D`t3s8pi(WbH9?+i&N1++TMdY<-J9RMK)?A4ZTnt40z<4;m;>GvT$%vo2rj}
zN**lBd4;Gs&wn7V5}XG$Un0IhDhUVvrTcJuBh!Jw&W6GmW=6Y0GJxW*C58Zsesa>m
zsPt9hl_dktQDAJa{YLBg6VB+o>7JQVjq0^y?>*Lx5#ckkhA)qap`DhjIDc}gMDz+U
zu;Fe$jIOWd*UFL=36&I8RXIS)c!6-#-eCaSEBsC%O_Y@;%3nq|l9Tx2@LYYfmr(Uu
zqf@)(t)RM-3SU=~C$ea5)IQ-s61lhELR)c?4ebsoa#f*FBCQ
zR_ZgAwS7MwZVan&~zeL~6DPpc#GLF>6&G^
zpsiR`T44EADxO7^LZYrz%7RyQrIqiL0IOV6{VjNhaXys<)5!;+MD?nioH@6M78MVn
z;H%&*XfX3oXxpjfkS5BWlTq>#6J;PQ>MQCCVKBFS
zveasZ(F^_uO!WiySdJz%jvDPMl~o)RuNY`Sqly-nv=EE`08eh8FD8xZwBb{}R8xsp
ze;+FOR4HLYUnRIm=C+v#f{7l5Jp76(sWws#CCg(yQ27Tz`bGK;b_DfaX;lZD3FO3KPm}b2TY{UD8d!$tH)j{WxAMy
z`^Xn5!f^H~bm^b($?rG#xHI1Oll&udZ
zXpkc1VB%oWxPoJ%cgWR4Xfx>ErA8H9)k}{U^Mu(E>eN393hm`-tg5q0*UDq&qZ1Ru
zZ&d9rE^NnS!dgR%XCUW#6lyWi8WZvWfBX}_PV=Zm26tZ|WLq!XdG6FgK^srjnkE!#?g{VI@Gb-A=(QkK6VwCd`rwJ_@JhK+&WN{n|%
z?1;e|Q*fGn{#uG0WM=ldt-N<$J{!p-?q^Kn7tu6YnxErOrW9M~y+bTFM!!X`UKoo@hC;EPpybMo_&b-#fLGFsbk?^$*RHD_53BETE8)MUoH+-&v2Er1;1~yp^@EC3GAkYVd<5m-(x!t-fW|@}XB%JUU>1v8uVN
zN%uqyCjHSZx%oMc8f8%-SzS~8r6bF#r^<$k1yAc)9RpDM2{{ZAUqy0J7i2YjAi}UKg3aE3?J*2a55D_X&Nb12bkLkp@+SCd5}}Vq-z^
z6+(oPyPNNlakfirGv$lmqi@n$+&+^dz(!S@v2o6zQaN}&kv`%C5*`*rOn6fDGhmrO
zuKK$9A;ZvX>S+H0K!~#m5PN3sW$4Vs|sMCU6>N$oW++uE7qHt8lc#1(o!71V?%3Rbj;#l!2Qc)UiRM0=U
zOtafBL_6`aS*-C5F>5M
z&PBtpOjbcW#hA+v18H|rzi=@+Qa2!jJkCt8Sli@M12I%OW0tV$V2DYxkrmUaLk(V7
zAM9g?OLKsVjwf!=oBCzu7eXpwgzcy(CDzTDR`_NbhFqc*DRTAkCw6sH0?)YM-?$gy
z5yiOf<(Nm*M=)7{Ai7}ai}5(i1Et^=@Lc8)P0V08nVURIrXwTRcsyAK4q!DcU&H~$
zhH9X;c;wXeB8?@Lp*uktQ?$E|W=%mnMXg3@6T$S4mAmSGPiZO{{UF}
zg7b}T^K`^@D(kVr<_!yW?E8q;-@1!`t*yeSw>JT`$D)0pHlrA+M$-hNxso3dD{)Ac
zki^8SM&&bGmP%~_Sc&0*8{EVQfs7Hb1j`U+trFQTY=FUpmy{ni1UdMfY~hu+34;aW
z)l~5tYSl5G3_#)uk`fm?nAoXFG6N3d)rqVp`I#8~xvPa(S0oa|JEYibzS_C&ELX^E9j**2jGG7PCbog7R%Hf7Gytx6u^sTaqA&<57en$v)LcA
z`eJm0>ND`fp&IyRJ6@=x40kx0A*Jl8Ej1ji3-_#$Q$?mUFGQB^sB21t%P6oW1U
z3S7j8?pwK#xYJN>D`6<*j@emMwxB8|ctCbC!y1l-77WL?mQoa%tFI7#Qqw3-6@AC<
zmgF%XfN?%=pmMN)29jE&X>+kS!yL2d%YDQHw}~^7d{H-rB%KoJ1$krcW7Hu|ptN`aN4jL(SHS?~
zgNml{h%b_0MTO2Fqd8SnMdeFurUo2+d_TeP(U?Bv5(Fb~6?IH>zfj853<6kIBQP<~
zWWl(J@$ef$qVZ-1skzj%p3McBmJwUrdc~b{E^V2V3YEMQ9a7_3+X})GR$G-Csht?E
z;O#pyvRuFcz08j(e80HW-;(ID6a>P2YB_lY7rPRr?JTOfU9vKUb*;KeyVJIBX
zzR;obsKWmMRK#HTpSQ$S@mS$ABrW$dD!PkV2fUzoSE-!0a06_@FP1lPB2g&XPnoj$GY>4J_*Wo2u3ps_r`jO1A{dg)`Oi@G?
zl9OZ^$4*Hva`hK6f(=5s9m~!%O8SY4B3xsb03@+!W$JAco5c@L~
z-#A;#Qq_+E%M3lD9n86ia|AUxS#=l9NF+iYHp-Ql_#w}gsR@A&wq^X)3Y&rtyPWCi
z?ki<98H_4=a~t{yEwdXQ1EA_Y!TuSbs(R}E?|RL68ur|Hqx0@N4dxA)}Z$nBB_5VqVe#mQ`EGyX~a9kZiri_
zx-pGHy#AwxJQzq?)jrvyG3HTU5ZW+b@hP*eW^ND{z#
ziNsxz*#SWE1RjXS;}6s}Vtu)rW19+`B@9?pQz+HRGlRI7Uloych!=wSW)2zZRfeT)
zw=
ztwi@T-;NSY9KH+9lC7aNWiQh(WSZ<=C*-5a8jaR!+{^&=;UXvE4gfR|CakAfCnUi&
z{&{dffK)ov;Mj(cZq#P0>J%IVZe*bCOO$Q$22`z
zhb#|%7b?aB*wo4{$$YIt-4V(qMlPB
zGCiV^=!omWo+8u~A-p~(%6C(m)A2Fw_=z~{+(U1|bR+6Gcx9ZS8e|`cTZ{q{<6W}b
zjY@!>XhV6JsIPB3QxzCSxMlo7Foy|KID@34)b%5bg)n`Jbw8|+WZ74G;Y-#?WR&KSbml4Ot
zL0Oi|DcYuAMd-|!8%4x0^h%ZolbJ=lSz`<;xRJZ&o54oMPl
zl)IW34PX<6F2Qot5v29YIh^aC7AT%shdP61nQL$!XDyg;C%Sg0w-C)nDLb`<5lWBT
z?w*VsjO`HWHgLm!MCx2K`)Y7d`;=TDTWrC`Sxr<{T)}r0zM&L#D(>eFE7D^DqqqkV
zLz%TytWk?HhS-CcHlaCSA8a65Q-sW9dnKj!E6QQd2X_j1lwv&c$@*W^M4X-=!%Kak
zLJm_un5*p*;FmXe5|DSv5+aNUrtTWk7$DyfE9Fp4mnmX;nDHOfPd}tO*X=!kji?1~
z4X7eFWWx~{Dj??UKkRHTSaR`kD;c({_iwq=TMpeYwMlb4)&?Cy+7}5_ZGXB+U5f!
zRBA6-h~`>~nSdjSkqr2TTtb+KGb{_O9Y*?qaLRm5sQmajc-+j)qd0Xv)7>lC78OA;
zuGsd2&5K;uF&t5s>Iz`;r}imH0XH#E<&+f>oMM@`TfjG)AzmPu7JWg{Pg6iw)U?G%
zN(Q&o36>(f
zvo3J}>6GOxWvZ@Wk49U&mBlK!_&`?k4Vc7Vn7Z-9nElPS6{kmuLAgr|b6JvOh}_&z
z+&dxEN1D)Jzmza*7Wzt+%=1wK{7XRh7&@6KfWN4eD|G-wZV;rGiD);9aU2M-FDy~X
z@L(Ua7kv{Q7>Djs+MQu=l_SHyOfy=SaCdELE5J7@#6rnT5=sKe?gXPYHkv^g;|yf%
zz_~sHn&S9|a`P>x%tW0^-_B*0Uvi4*cy~hr29t{%DZvYcT|iUu981_$D+RIh{1DQ$
zh!vSbaV9cjF^oY1jeH|QD-wNpxI^s|+!9?Q=?b`jI-G+WaRV0Ibz|cK@C|+UXsH)JxxEhWKcSg
zXC{{{+)c_&F)K>IW{$UA4;WEY-
z3ic~N_XViuF}R`W3|7`}hGY#o@ixq!p^Es1fV_n0+}W5hR7SEP?YUkS3q)ze&U(yZ
zdH&(b9-*uhx72Q&`-(EbFeI|1c*$lnj54CRbiB%)2<-Zn9uvVa{lx6UqJYJ3P>dT*
zvZI)Z^R8Iea2yPw#i`#H?1n<+m<)z2IWpT_#}Zz`Ma8XyAg#=LiuD&SJ-ii_;U96&
zEUmkT&SRl6Ga0p0u3he67THoo9Wy>G`%E;;@|fmmxWj%q=ueg%(v0Me=y(i*F&3j&
zGUNq!714SnMYg+EKM=Lxt6VIpK%k$wnLtMNOD@utgWLd^z97tZ3h%;^M@(@G+>cP6
z=7u|p(aGU}6vPsUR#|W%6P+*p>R&qdx44>p;-+54rXX3}WQ*Gdh!ArX{HVG;N=%9P
zB=Er~(Y_!G=mOyRYCKFqgVf^BQ3qaL;n{6*8Ham=YIfHXOYy_puvAM#&*BaTFyO*H
z7~CHjBZX6_M^I$9DFTi@a^|#TlxfFsgjn2kT;)^h52=?BfVY02nfwSipS
zDx;0UBG?WQZ&77?cP!q#Zrrh`H>euS$qHv-Dk9j8vZaf*ir9k2gAHjgvNZECaa92IZ0U8&QW)=P|meUWw|B3vh&Ea-%#=o;b@Y73(u3
zt2csWD!I1d3DKQKQ-Fb{n1i1J5o>TN>_4eOE3*tlChbGLwyB2a#kvwvG7kO9l^q8J
z(!^mbKg7B-+5JmGX6n`fs@v7nr@?*3%JQ(t<=jKnFQZwcTAYcRRtu{;lmk#gIHjSb
zjybaa4kmK8O16@-3L8ktSS6QhGhqg0K@u_{m1VeUzV1F|TL_G{c1%$&;_Gt9GL9fJ
zy%g{Uv_+pCJ%;CrSp4Qa
zq18(;e$t|{H!@i?MOO}`T&L7c5e_5t2!|1AtTf8rF&?no!*Lfu1)?g{Q^=O^%AtZt
zP3~*77+gr^J6yW|0Btro#C7vD{{WeWnIO0lW;o6zH<-k!UI}*-kV7m~>=A9Vv6fHe
zC$N(W+FZUR{Y@9c1DA~i^x=k@oePZMA56%TwES9BGXpONOhFBE^*O$iDmW7ve}yfV
z6C0a@4t`0dyy)f=ij4+(%qgVFGr#)5N|{fzNTpL;O?Tmjpndq(tVsf&vTy1OFv|Iw
zOfqU2n)#QODCSbwLq`#}RS}vN?z1exP9-~k5%*)>;9bgKK&BY5#$m3^Wvs{b4Q>%w
zvzVwF>M3qoDM-r9M;n4N3~WMh4m>gowd*8#;q*K9CJ3Bn`S+A&$+qwGh@Whx@?7V>xk;^T+U|`4r4h%PyK>HRZs5!
z0OWQk4PSruV8{dNQP&y2b1zmj{?V^8@q*8AtZba5v2!#1HwK$vg03Q)(ZEY8_tZ6!
z4G*Ni^%gvGkIYGlyTXV<>sn{kn@aHolB09P_rL(FO%Tik!oG>bKX8nYH59
zC9L6=wy_5lxm-&&Iw~$1_RTF=@6izP^V%{lGR_qhxTl5>mw|HJRorVWMs)zz;)dMD
zpahl@ic^N>bu0Oohj9{fznHBf=5q*b5Hz@zfZl$kC>6;ThM~ro4jEH8tgT%~Rq~gb
zoP!VvCgfpRe$aS$XyRMJ$eW!J+=V!0r9*JpZaqW`qdD#i2&Sgp4xe(x@tA_B%-hs)
zV9jsPCB4oF&@}MfZXPP(J;=*fE~49(Iv^TpPx?-Rt~{X`4q%)ikC}P6A91LM`hhir
z+Ce*F(4UabZVl}T*rXezD=eAM^
zf;u3(ge_(U>4j6ce>2db*jG(i=JG^@iL)4({0
z%)&#L8-TAnr9>9fL;i@;jLgFPFyc@wD6ZP)GbKQz7?uMqvPP$vU-^f;f^tvXP9}6<
zr7;cxNdutzCAGaFO@Z!cu46WXz)nQP%nPOhF22ym7W~Rw&mI@j1|Xhex3*RC`GdH5+-o#b
zDXeeOBX~x{;%zW=#*AUatw_gdM3~IP7#8B8UEJ;0scM;8%-htkK%Rcc!wgDj
z>KPY)VR`A)II&()1E@x;5Jm|`5)evou3-#?c=Aj#jhI|^HnM;)V1To!QPIp`l<0us
zSO<%Y+DUsN`JxetYd%u$R7|(ISVM6RV5S+L;xF7(w7(xRkQY5XOKE2bMqUxD--z{L
zUL`%wMp(q?2tJTpz!~Vxxs|H^%mwOQ!Z1RmLS={=#I;KrBg*)eBmO&$gOX9Bpd~lk
z7vf?)WT>yFNL*bHS+u@Rl~^yZnRo6LrCpnW%Sc0E+x?R6o!t6NipXekcNmsxRG$gy
zf`G>JC>IkD2kKtM_$HD|qw3&wTJQ=o2(2}7rm-2k3^6irdzu@R*DxuBea5xXs+DeF
zhjSnz8K=QqxNZ$@QxdU1aK&Q;&>y)=7npYDa9sCCa^G&HOKr>rO*85Qy9j0G9$9-r
zu(1prpKyF7wiPzuvZBI&o&|S#3ywmoN=(!2aekR;_bQ4q&JU%Tdfx>R_rij1Y2n^o2k?d5;b)Wy|26
zm;sK|mC0>VDWX68`-layw-F)4|Q%tdQjfv_I~xqYC*($-f;
zgw%$h#J=DouoJoD3=dUTfYSye#Ec>RKqw8E|0P46w_vX)bPFE#Gi}angU+
z4AiG6a+G6J7%Ez1OJWO`LzKb_4cs%6=37~q)+icSQu-Vdf*GSdL?y{|mHz;gVJ*De
z#w+bx?qr9-4HCy*V#&Ch7$2EINzHKrI7hql1M2B=sGQmbCH*8uPYp2X_(n5Lx
zVauh~Ov70FC7Foh%(cTAlkh=84S&N7V$PC~9MI-Dp8*!8UgfbfDwtwU#j`T8?887F
zt}&G&MOmG>m<6@0=>g@aKO_q30W3Sd0t(T*XERayqYo%mt14;r0iMcb?rag~6Fk2p
zxEsDRXl4&WU?8dD?(xLsDSiEOmA
z#ljK1r|vcbJ_$fS2O0_2SWP9x%xEv9mZ+Bv!ok^`a8F@JptYOpm^8(ykeADlviLr1!q42$UGc#K2@MR4T}P)=@*z2w?H5Wj;E1?~
zKcajK#2-kiSzkCM0Oic9VRH69=Q8+evI`u(qOSl|EtbVRGAc913;0cfIyn%gQL?Xc
ztRZ95Hk3`HPjfL&n3b^bTe6p!Pg39v!Hq>gnnJTra){99gpa^A2?`5P$*DsFsZC}T
z_@Y_YztHM;v16u}1}w-vQiCJ@E?B2>N^kO|pcrEOHsIlBGcCu~CO#IniD^LVv}(9)
ztl1ZO6PLKR3?M~YvRZKrn*RW4x7?|y-7nZ|*TR
zN7Q*@%Tlui%apuVJw$aI%2qTE^ozYiM787f0WaD(5;p-bYIY+KbU+XRa_`^l%(*pn
zb8vc@nXSj+G1&J2iW8Co!Mh?wM?(sVQyEEuDfbXHZWBqA{-H1kQcr)1AFoqkBNmTR
zzz?lHPGD-|n=>(cD&Oh`Ypv6mX#o4tsB;(O98BFdtEpZx-r1--`Gz+djm?DTAI>2b
z7gUu~eI$JassbB+BeGfWmV~#6DTJuOijSY-Tbbd*
zQJ0
zukg5mS8Jn4TG#Z&yv}24$7~9NL>MJ|%NubG=^QXcjt_%qbi~LI!)}SPismS8Tee*Q
z%q;Q2$ILsk!JI4UotEH=SXB3m{@yo(@+bIF{{S$82P5IL{ltox9dwZB0`V$JMj@os
zX$AWx^#QOhMYA^Ee-mM@cN0$WRcG}se-y3{{WP+
zP~W&{e}IiOM)1ov!{R)D!YF*t_(WnJoU*9j$V0~r>RvfbE}{<)Qxd!%r4l!BML6_L
z!03{*D6!d(jXHS}tCG8kck=-^8;OB|1G$e4fQ;?{?4Hlu9@$Er%kb1IFv^n#-h_U-
zJ?-}sxYfcvGe9K$?m3h!Ru2fLE3U>pW1)bJT`_K1Amflo?c-;na>o|zR;!Br#i}w&
zTOA4SVrs!LLP0;gkFEGoY*BoZ{H=CMUmlNB8C(V&)j|td!_=VRWZc~l9ZPHs9Hrxg
zxs7GZ_>BO@9}w}$dsNFRmFrUuV0|eR`OC>N)8OWb0%xVcZV0Ahof-JJ}F3n4>
zB{Ll^(HCH0!55WZL>9+hMsSl@ho`BOM9pIfd#aT=OI1fJPt>I!lS~rS3i|^v1<*iY
zCyPG$?t?QibzLMl&zoP0qSLuUDtyoPGe|v0JuU7ntn-L6)(MQt?BhTAEo!{#=5U`t
zKH{R{?1Gy)m!DG-yi*ZorAc_(;yo1}D6eqNJYRv~#fZf+3`-p`3DC;Gn1PGTNiXmu
zMhpky2{QgPRe4A?JC(SzJc9a(0IwL)2lOwY)OT-^I7-QV!|xjMJGi$O#pY+&zyZ~B@T&Mw|FTf^c}Lj?ACD)9QI*FxeMwQaIdouPodluqY~Rb9ByK5EQ_``
z5ST2&9tu^#?pnU5+a77J4t`;^3PM@m$JS4Q{JbEf?*~1Cc-MLBWBBU=3e+
zsn4lN@(F%F@)A6VV3n@85NazKTa1tIAl&g*0(+rUwuG<~2M%Fi2#qSI{HO&E5~k41
zY35xtCZf<$tB#(BZ5X;1wTWd*qQaMmxHt&oz!l()2H}4ag+g(1OK6K+CPpoQ=2WsK
zJ4%7&uM31)Gc3G*V!De}LA4>DPf-TnO_Sj#gMgYv`hoFiGDGxC;aKpm_M_poMKue_
zl_jq381-EBFy~JcyU+WZ3h4K>g+CvRxuku|C5K%lNux)<0=Z*xEQ@Yt-W7Xd^q->P
z7l-cyQx8nCC1uRP&gqq6RL2v(S?ydNp`7y(=3ZIO3*3w4099>MGGGb_daigS3#aul
z4i5`wDp$p9)K;Aln(j!OO-X+7UlO`8P1?+CU$$)4xPes8v-c=5Tf2h*=)p1qpbi15
zjb_#BxLc|lhPJnfTeX6R4q~k}1HnLQ@ehD1IsVd=uf)5W?}Bfl89$;g8uNy#R24;9w*nh06F-h&
zK4+dU>L49UN^=~(&&(yj%MUCB0m0d*ry1}{bWhyeSvPl_OxLU#Y#mH|s>c^_OA0=r
zLwviHIUeQ!KQRkoeUIBL&c*?~_ZqJZ?C@ul1Q!fOZDlCB%(HT?kh(zVfUTkxw9XYO
zTCB>P3|kK&vkNS%w#FIUCJ5K5*vz1H6O7)jJR?=lGn1K4nPM{)8c=2piAO2LWtGbF
zFKE!T1#2zldu7qLsP-^8A8>ELGYEc{!#Iz?3Hp^ZI)E^X@62VKGqctZ}9T49yL!MTkLQG&9^4#IgsPXlx=Ixp|Gk!$Jc#
z$Ahz(_7+nGNaCXU5G6mV_KM600mL;T1t&2py~ZnRXehd)qy>Z#j%;U(#0~0^6of7|
zTO5LAKh8z=%Os9jVbEvjf>&ixX}%$-v(!r^vgDJ3I7rxe`I!44e**Y)Y$7RtqEX`R
z<_MkmWx{4KtqcIjWw#a~ZiXUTTQ1zkFH!LgGYJSD=0+h6Einm+NErge?Q@Mi%b1q+
z+^I)5Gh2p4M`gL0Yn|#*uTs=->UYH2WUeA<#HH4r<(O;gTYY~?iOlX%eSfKiGXpMM
zvCBGCB9dFo$1Gm87p3@0!cz#-m_Ox+E?ez}(DhQGQl82+d4=%YhT|L7{^9fuZd%82
z{*&&7ehUShZh4qlyhXiFW)5eN%d>3`!<@bCu_sdz*P2v*BU^SFf7)EPzi{$+!r%O>#LMjRvsLaofG
z6=f_vx`0QinC`y>vx)m$m_Q5xFHET0%Mx-Z2t+S7{-4P_4iCCM8o^sQz6Z=pWsqi6
z3gKmwbi>zjl)zlQ7=?`D#8d_uW~+$~?g$A0ScSq#gmozHVjXRl1@Qw5ilw6IDZXG8
zth1Svy?U1^i&r`3J>T3A+tO?T{mOB*adUcCF49;zIF#N|
zp=fAoFxt54UZ0|B{{UdzGz|9>5Cik-4Q?yeV&)vp)Uf!1zDQ597WEWz;uYcmR8TNS
zPjdV926nGh2U6o#(w_k-z^zP@eOwM;IE(R)(@;yGh5b(ETXobo+M(OxQ`4Ein2yAu
z1xzI<*CwQ^W{6%D9^iqaduD937_>e{A;Hypg2EgPuyclqvFlR(rQFLRwj9nTg1{Rg
zNl&Pnoc6_SYZOW<`hx&N=l*Lmj__hPsd;E+00ljmmrxxfYk!0a-IkoYmaaLBL4s}=
z>pYQxUsEl~xQt@+NE8H5P`jd~beR*t{^C1AD)B2*{mkSZ?pWRzZN>8Hr2{5KM#^uv
zTD7TbojGO&)b|w`%ACZqu4^sjIa@3m!@l66s{4Yiv#6=gH2XoxhH@F|SsTN`35~j!
zkMW;zg$po-y~)J)pQJ9^AKd+^D434WK@;PsMPikNS(V9C^iOso+{OAZumu8M!OBzz
z+yqA>P?RpjHCmFm9e!R|4LIuk)(aaFpOl<|KP9@2rP+lVf?l=Ueeh>KP|
ziAG$o>ZSw2#jRl|d&fj@0RmJAEsi~^dd3w`GuD3PP-$;vgUAB{RdhpVxQV#kL!oL}
zRFpIMC0{G@!|Gx$QD(xNLa3MH{1~{GnQ^&ssDBX6<_jE4QE)&@d#G-5GC^32aUPT1
zBMrEYElf&w4%w+zAxIo?FL&Bp{lhH6TAA);>1cPDHHoy;&?jihVVNO!D-AGLW&V;=
zxD`r8_RImk;^Ol$Q>4T;>K+49fheGL++qvqG>zmT+L)MpPBOx#7|pEN1}cNMiBo^N
z+y&Rk61n#U68A(Fn2_T@BDgt6$&TmDK{)0{aeM>x80NY45df+{RlmQ)8q5d-OSI)=
zG{11!xdK?&<1sBW+X~d-Vr3|<`j=UwWy|#}ZIoY9fE*+t2J=GZSSme1XnHTWt*bf?
zRQ`}PHpj$tW5n5l*{NSFW^jJY+TB5@MShE_-N6+&hCCqQExRLa5(X)1Q*aKQI*6MO
zh^K4$n~PmVNAmG^6bP_Y?H!Z+hrc%#E+@Bxkr&+FVj_=;LeQ=l7lJf^v4Xu+wMG4C
zg$f@L%5I}9yXZ#k`sN;2a>%byOj|2ZO}UkU7GGqq8lBWG%nLU#8M(WGR3KanYY?zP
zt0AECQxlB;00iELTZGg})ysaO#ol}!^#pm25sl6Y38l^@W|{H^HloFB8cwE=Le<^~
z7_Vkx%XK{wTEO;Cw`o%%kA&NBJ1Szz;r<~N-|B6F^~A%W_bPt65MQSgDIO6ltl;K3
zR=v>T18fqlN}F$i+bl;F7*oR@QCiZ}F-vU;YV;maMCuEIIK^u9D;Y5Xq*Akx9_0kd
zeXu;Dli#?iUKDzrb8cmj2d*Y%qd|gRuXn?#)v+Loa_x7B3dm?N-;pueA2Np#^-EUn
zI>pS^Wy{RHPt0)4{5&ygEc_Dg2)dsH8|Z7pE$6`65!ZmRg|Q_VfoTC0Gma%_I%HuB
z&&Fm6k#^_10o?6kVAi{3iUAQ>;K#U>eUkAAn}+K(ObHozRJ$s#aR{0mFSuW=l~{JC^A}gB
zwV0_JIGk!fmygmtrIOxTi{hpS&`~eAv0{l*;-n2l{LgQS*tfSThjRG`9_}`nRstqX
z5aoR_2Nk~D%l`Pythv%ZR%5$n5+1qS0~_v8YgbbF$cuC5#;Ug($WxhKW$ed651QnQ
zrqN#`Q)PsvlUQ73O(HO30n!P!A;Na1$LL(K7t(%XTGfJ)Mq#W{ljRqdIGxkEdxi;6
z0*OtesvSR!dw7lr?*9Opix@p%n_-(~w4UY6#RqJpX)_zmUgKUeZ@GHes60vUAT7!v
zI0M_bvzD*rg(!7qqHY_bOx%#Z(=Kz|zV4EMfqFm_0&$y>wyMS$C5py{C&so_9&j=dh+^g7*lwh|UN|kFq)kpaT
z?8D0=m8r{z4NC9|;Fv1rB+DEpu<=HJM4d&{aAhyZ24{(mOGCDaZ3?_20O(Zo0D6Tn
ziwg+08X;i#8Z9F>PDyd9%EZ^u606PpM$;^HA2L*6_Z}g{E8#s8m@M@Z-{PL;Jw>dV
zKa@NPr@2U9gaO&xEqCDIY+M8#LNF%X(S*36_Y514QQHsPpkNc)H~N>H%c!)nr%)E;
zdb|X(;tD21a^~0&8%K20@aP7^rHM&9H8zRYBz(STCuz;^TOW8;U*0v*5A0ean{z
zxJTT>=4v~hjwXq6GJ|IBCuQ{m5pWTejp`^;#IY_^FQG8r@%REF!lk&z3z+K*GKO{)
z;v)feIwt7Nw97rOLp2cq+n&f`w|7r74-2kE{-Y{6>O1yUU-1c%w%9|CO%-g(8?REb
z!SF!fl6xNz>CuW1P)nVbFIkxsQ^A%PVm2x`OcCc0NmutEFyg@9xP;av0FQ`NHrEWU
zA-nKs5o;4Xu2<>Dj&?wjzX6Q@01*30oMr;bI=(1k96{|5gvQsezF5~+2rZ~1PnQh|
zI-C0@NQP4fc@YLFnytX9|^DqV;7euB&4j2H&)otz)_`6Cx2L~F8
zEv$nV1?lY2WzeSX6f0{t+`l3N0k<pj
zZpw!UL5sr?GFiHrWUFjdUcs5a`BQnM_&{|LXd&E=Lx`xL=WJu(1_%lyCH`XKCVxk=p)E&Zv#YD{ukzS6%n&F24Zb62n
zwsQkrmLnl;7FvrhnP$N4U-viit3Q$@Z7&Alaexj^f>z@Vi736D@=BYBW^QbJOO+)Z
zT@0}~THaG8kDhl0}c%Hc8VS(#amBGH4PF6fn_
z$g6VC5~xrydyRk>8+r;MFO#77GaH*S5
zXy3R+7SnnqY9E=0GTf$RXaP6b09043r&+n&C3hatyMlu4x7@JINBUN
z122EHoEU7W+Ho@qjgZ6S5#Iot;*!pR7VmHkT~6pQ8J1d=HZKGLZ)Dtvt!2omVy5TM
zxDkuxm8@bR7ITpWsH)h7(Cq?qWv>!T^J+5AqXd=&VaC?PIhh%J@o>YcGIq?rdjo^+_a1Ld?;#o{CTNfMx&Y~A(
z%8DgBMPM!i$1s>9_oYyKmIB5j^8vYK#pv9`cs3BaV9|vgPk{o(7t~)kLZ+@*aUSMr
z$HY-R^;0OTxQh8c48Zb9Vvhd+sp-Jsh@z}ya(zTfFhzkZSnRq#nPUj{A;&KYW%DiZ
zsnZz27GbaI7jBqzGUWG^kHm18p^(x{rq1I_g1?qkS-Xp1V3qQ(xo!|}(5ldA&$R6f
zQCF#SOrT9>Gl;n`ygUA;njl=jxt1?-@YGuA04aS!f6FK4iHv54sY}1*FccpIMJzjq
zP>f=s6s62-3l#7j+;WI=-rk_Q`>ZIqh(*CLhcczyS}~-Gqb*B=5|An-#WmQ?!2C>V
zA5i61A)lH}uYe%-<@em)6V8!BvgV}?ekJLoF;;{KuW<^ln?DnAF2^>7O1>A|Df!ks
z9t5R=
z2r*1y)3g!o4%mCJEwRMzHE>9$?_#gaQFgH@)HcacjCk=1=wSpY&90?P2WoQ?tk%mY
z`oWt}Ytx9X7uhWqX%qi-LNzR1<@N0{X|GI*Z9)`b8-5D)l4R^_lfE*S^7$b
zS=2Eg3@mkAuK@?7-XnwqNfa@oa(qQ}1?u04rvk}j;mqFNGD8S(^)Pd(
zouQ*pS|lTeIk{Ahd~!{akYfJ;@=>e^9(UoEq0B_+@RX?t%NtJ2M#88Fp_}awsNDs-
zSe$WshT-6?CNsevhZ7JRJQ4)~!miiq8KKJ{wqJD{LaT-$zGpg(gXNe*-e8)Rqp973
zbgBZq$NNYbvG%YUt>Ee!`IWkXKyfnZa)FP;0gTEja>FXv;q@v2rGpOj+IaOfxI$%y
zSi@um;}&|zw?w;m`FkgC$fJ$Q#Bpd$KBEfyI*YXrGQp^IpK(Kn_=j+XcXK8Q4dw%x
zU1k(=#(1LAt*Jtp`s|k|4+AZ5TUx(}ZLl#*%X6w>bWYA?O+`3|^KZn$n@q$QB|WQ|
zn^*WQTx1}{H~Wa0i!&iCi;cMZQaENOlAsaCVJKL&6^GJrsH@-(Q9@O0m%t2#H|`i2
zHM}YHECMlxzG7nAbuLiQ%|)49Ad~1rGQ2QTQl1nVh3z>-co4;`+su>4u)iuQa5pPV
zrf5Af!vsNUyNp>d7#oWfsDo0PsQ&umwh;N_(ylpSn3X7IpmjB
zb?87o*dEd@yCQ(8p8w@GM;DPskaRu`2g5)3Z3~
z6v?R2xy&pkr7e*lsd8pHj<>!HzEPtt>Ln}tpnxd#NntYqPU+O5zqGG|+_z#*WHgKk
zK@!=5<)nK|2uD&4loT%2z`%j+KN*uzQQ)}T=L~Bqy|U{N+{<$Q^B9b6;{se`&yqmK
zDw0}!0n9;Zb8^8H3(bWd?=X<0SEn`ZSi!jh4#Yi+e17{x(J1lt{R99--j&fKC|hF7aG!UB7VS|9nC)(h_>uot7atzfnD
zEk&z!MM2HQWsV@5ZYmuT%Z9uZIh8EN=8^fc73v-SW9l!No=NUNAXhFZmpfQE!YX%?
z2z*Ovl*yitQ6P=WBZRKd(TF+Bt-0?bE~6Qrm=|QKr!i1K+ce7zQ)EcGGHxz*OuK~V
zL&ZvcNabqwl!PI@?K_1ql3Y#>AW(6?nG%arG~FM*t&jVGWB1(MLMpfAm)xZdA(4p9
ztC+V!nYR9A+w4sciLzLpn9OsyimG@y%W~>#j^fIe%qCh{?k$sS!h1sHq8ATD*a@gb1+!%8B3I^i!s|Ow;JU(#&6e#z1=eR
z011#Ft5m^~KWEhJgDyk}P(o^0{{Wcn
z*e}B2a)|i~SXL`tTzVTn7gFy|s10Fq>NtWs8GNlY{$LXbcX1eL3c{HoZOUp>l?*8d
zW8@VT+{wvL5DJf6qsEYeBl0D1%k-H1O`P(}pHOkqS$s+jmvtzQL|J?x?{gDOVBr^6
z1ySXxfMwT+wCVyc_M7lZ$AKR(YttLB-N$BcNY1>uK9eyFfHYtWxXEs!lLPlTnfgv&
z1f|FLH47I#yu(SH$}NUX`Fmr#quDG0?99dNFKot(c`6muSSx
z1Tf4sw5l@H5f|LC@#fis5vEijq8VTaVl1*agz8Hqh7O|+3Cwh)s{RAOwK)gSO=_+>
zidMOqcu!DrVbEC3M1Nca-(AL_rFG_L&s+#V_Di{K+}zlmWacs~gv?9nO@Fo5!#yN0
zIgaUvI`s()zzLER)Z(Uy54h%M{3e-^6EdYrhLky!N>pjVGjf@0il@wv%(}xIn1lo7
zu*`^$!yG^24hUa^;#I)4grTYf7oTh`h!IS0UUrAY2
zM^PxusZb_OK*6{wI5D!|P_wz!)`XY=~h=if7aoP7!UV3H3f3
zWxO~D5{;b^i4il=hmoYLV-!IcBW^JJO~NUCOVr)ABoQ^MAknHRWo@WjPk|X;WhSCwyFn(;Qn!y4r3u$JWu?8xsh1qUsq%;;
zoK#rhDGnt@g~REB6S!s#*XDj>{{TdIM1)_0eLfL|a`NJ6$(geSn28Gn>N2@XmjId$
zcd2=eFtuZt**fJ3vK@S;4AU|*_=3lTisPt@I-cT?x6)DFA~IQm*~ebxI4NO=LscHw
z#<#c87MsMN9YWB_*mj4^38`>aC^0GVQj(rySC13;4Qj?4SpS8~E}+mdVsA|L&d*wo{=iliyQ7Tg$#xrd}0!E+Zf^)4vYTtz}G
zZblI&)T}P9E+dtQqeSa2@C_aY1~mzLwidcG58>A`_WO*Gw`8$hmBhR^vNFeyXf*gv
z(w&msMAggp<(93+`-(7UUglN47Zz2u5TQZQNpaz0!D0dKV=tM3G
zCMJ?TV--cgrpbgKa4$bpsK66aQy%`ig`SsJ7&(HuxCR>M6Hqgl?h@{FZf6$tG)EHx
zxC0zp(It`fB~iQ;s6N6|EnUxU;ch+GhFpAX+;aB`K#u8Bel1
z6#V8XTDRYUi)q;z?C=j}Hf;(Zom>gTvR%aiZ*sgzXbQMZLZwfALIrYWO3GSZp>HqL
z9&-;|&1yME@twJzv&28eOP9i%mk}PwFPkhol?ABL#8SwJdPfOpC>7?i{4oXqAZAuP
zsXzqqVt9As)!K2y1$PldsHfoH%mWtYVVt1l#4-W3ZJ81KGNIk%zUDR1WrN}of|v}m
zD%JB-C;cf!GVoI)503Tb-cSukY9^EODtigHMzJY@duj+2;^wC2O6KJ`%AuShm5773
z@qR4Ms$g2Z1QkGD$zvMa5%<9N9Ef*un6|5&Uop6mJXEQB2=R|K1WqPYpbKIZQuFF8
z=iZ9KtWns@hF@>`ge$JjaZ?@ryvz)}$Fyx$UM1$qK`CZBhK>B8AVY)*=wV=>AdC0{
zbAPokh4e)8u$m?K6Fp0p8SyO@Sg(j~6?3ViUo|leGV;x)(?+F<;#3|YH)C1CQbcLK
zBbBttsSDTm`Gb00ai~~FPgsYO^6Qy%(%e>dO0zqdZY-$ysuH49JltZ;R;^_l)8KEw
z%kw~sh&M|$?had{tBbAr6M8i+EV+1#n4sDkAo&iZ%fufBkhEMx_zjoik{ZH{vgM7M
zircusXty6zVBF1KSMSuNtz=BN2N+amgD@hl
z3unW>7ntAb0%8}GTZMu)JMdrOSA)M2)Cib|IA4$TVn35n@07tvwp4J&WuSv2Z@7baBo0xkqGSZ4eFVR662Cwu
zF9feQDnFv-?r=qAaVj-a)5s6xQR;6EF9i3?VEJDpS25HKmq_9pFX@)tdSEyS?j2!W
ztCAtW52?Q1tC-d+h?>s%*k)M^IGNG7_9f&awc@4@51dXy)ytzMSuevmw9-2Bu$zNX
zi-OCSFUfZXq85?$EoY)H7n;J4Oa-Ry#Mgn|gQ9q1Ufsjmi(%ZjDFScGxLaHNO($^b
zI*PQRi;szd5!BlQ6S5gUi{MKTcWL;PZU{$`enTTBH7Qk|t_)lq&vd~VigT5fdOqhW
z4a8~-6@(;w;7T)?TehoFB4JQ9|lQSl+8{jF+;%2sQEJz
zrAjd2l$L8m0Vg2AYwOKGNdh~V0@~_)doPIPBX!MdOc%ItIK(osPtHLO{Q`5Q6R{Kwy^|%vA~cwX$)!Lg<iRaUL$M3crIJqSGU5YyN$$Dd|@>yWtv_cr5N=^^W$;xau2~0DO7Im^C%uJT)AY&o0TiYK`I_&
z3h9*ll39%*mIsHGd=!C(+jDN>X)avr;6=-qE?l{C<;#Omzmf77zu|Ka0W}=OUJEYe
z%a`Ki%a;^jmjYBtaYYSyxIO}D;KIKQxpIQzWtaQ`@JC(g^R+---vI?w#N^}VVDvyh
zA5xB21wIL0p{7$;Q1vLw!~AYsUo3nb_;i{cMvop-9|qqpd@%59{{W%mpBz3P{vq(r
zfB2j5H93cYuMBnPJU_(wzr$yN{GOafv0T}QC}6TVBAO4Vl^7%dL&Zuw2!pa=jCu1P
z2mb&M|Jncy0|5X600RI301&5%*hUOv&g&3x&Gwm8h2ch;*mmv*l4$2DMV+pp@oaGm
zWJHM0IdL)F>3!fW%?GUeHInus+AT`s0)0kRua=B4n9%4iv9B`LDR18K>H*-TA%L@7sbF
z{>qboO}8-d@-6YEAR>cZj}?OIi~J;dA`F;592K7uDi8;cCJItpE0vi!9^6>0^e
zOT(ZUNjDjFWuZw3Lm5!)Utn6%a9tFLbvo$deIx7{4cMZ
zeR}X(y@PJ`$Rl>hm@Dw5$HR({>dK(>;CK*>Xtpt5H^RohqtvJ;RgW9wLq~nBig?RY
zF5G0STuE)BJQfo|p#9uY1+mNTbSFl7jjTrR@^Djhgr!j36OM{oCw4fVk*H(WOCrAI#=0B?^vN#XjZFy
zg7ygpdK0mYGkpNLwp0vV!2I6+X(U{YH=u|XH%C$rcg7m&;iGEMob}&RpJ02Px&gw(
z^3$wP<#+iqb>BIt?7t1CR%oe8V+bHcC7!WgHexlol)@#Z*(jvraavn$`XGe+TL=sf
zrF>7=I2f!H;wYgRr_zC4mN*)-Q1X^0$z6(wsxD`)3Zi
zFDXC`e@0s_NxBevn^bhbM<#KPzo?jjs%+jLR5i8m{jEh~Cjy6TW?WOx80@`W28e2c
zRM0Se=8(HZ+N-;KIeRtq?A#>vlu4oa6mZ
z-Mx3p!$+1Zrgw{SO1(Lfg#arE5KJxVfOYa!JWkUZaV0$e03wD5QRpaTa;C#l`rs$y
z{7f+tev+&%eKGM9lsB}g?fojN2j|oh)ENw>KIi@b7ladHH?PiPrg3C3zSL$JT`Ogn
zCzT+HrO+;bp{O&`>&1om_)kGDbTNkauVf=3m56_cvK9
zz|`<+-*7*ZvqO8Kh$n#EdE@xnUMGqm>{)xE`O1zZJithKNTCtV=RZ;2yLIeUz
zz$bTat&D%nCDDe&yTG+NiBYd|jJ<8e8Y;r6+tQ$2;on*JY-O_tml&U_!wuux^qNiX
zJM2l{c=*O&ikC6DI%g!V%66VS3gw80@6nwxT~)}s=X(xR!b3maFMGs;zg(l`C0`?&
zr1nF^f$qQ^sPobdNbv_rbW83bQ&)2Lq71L2rvwJ4w9zC|WfA4#1`4;X?~IH6WT_&C
zrJGw%>U>Vk^a>@U`C`-9vsQ@8NSLIx!v6re6v2&TgQ{+wzrlCaOAz)`kz?CE@vdza
z)N?MeaC>ftK`4f(43z%>2OF;lcu+jb@RY+D=`(iUt-*s#6|kYSnF_{U;@jynaD2(2
zV2O2M-XgZ0{{Yu1(}YOjtd9~nlMDkBj5Z%72r!FtU0uzZCAM&DJr28l9Xk+}YRD6E
zmPE8cy{g#NHPoqVj4&5N^WPx!0D=4fqI=-NcmN{+FTQcdzInhy0pCFUCm(+k*K@QMAxNc*2rx5J!rJZ?t0nA6q3w|coQWA$I${Dfwl^(0h-B|)i3;SG{4P!lU7JMFxlYDCM*5!^KNr{J
z^D^0oWNUJ5voR+e<%86Fr-%|$rT6=*sNHFl}Y8B@p?UyiuIx^eT>x((Qkk`O8@$KvVNiNQ^
zbUk@ojj?42!Qf_0BKH7j#w}Z|(_%zo086psBzT6>c*COxk`u%AV*GyXyXqY@yYcQ5
z+>ciAWgjg23xw>m1Zdj&BxGhwkgR+H!RkL2dXRXF;%7UP
z>~_t1um)n=wgr%QDn`{Wt2O9=`U{H*9xZnX;bScDH+nuM;w|`ZoqDhs5R&<{
zPu06`z_MNLWSh0K)%Lc|HV&5jLZ>mq6Uha4CzF<;*O@tl_v2+gA$o@aXq?ZEEMA~`
zzGn2G?z8tP0f5#C&inDzvaC5R%iXQG5kHIz)Ixh)
z;d_^g@d2g8yPwJIDO$3$Ip0;IvcX
z0^Lp)mS9PoT>F#bO1W_znqW41B)ft2bGhR+;_Z#Xb+~^20Md3|ABZe$9xbOgaW&H6
znL-#tUms|Y0SzC<6s|XoL(Y1&Sbln
z9I}QIGyMoi9EJRrgI;V$CwCi>dbcms8*p&XvN6<|9Kr+F{HJ#Nnc-s%F7~n=L1nq_5>PFp
zjniikk_qHxw`Q_*iBe?V?l0U@KjR^DG7=8}IOYS@yyPc4Oc8^&S;d5mCVY4hc!g=*
z`n&Ulami@&eZXvko?>`I`;mu-_dN36^UKn}_1)`V1hV0>>u<%I+_m2yFR39TyJY&m
zcIQ^_J_lYhSP}V)iIA=C>9Y@GY{pp|gL1fo8yA8mM9Vw3;ual@L*q+kJWt76@?Qc7
zT$smM%z>{{9mTiqQ*^mL9PM8GI7}Qx$g=DAzfkRt9L{>Vv$JnGJ=t%L%#$nQXP)Ja
zZ9;f_CYs$7sVlJQdXmfP&m*wLkg+9Bt9Qn>=K>Jl%0F@n&~!m{>cP
zUM`_J?hbX&h~3CzCh`|T+int!@t)as&%uc&4(vqpBFDcctE3%;Y#EkWV)ZP)cXz3A
zER#26-({K>Kzp}jW2t^@-w7RfV~(KS?xUG=iB@$Zcf$1tXOk?wNh!>4koM0%mh2Wh
zTWL8>T#pt>EcGD+)rTwqPj-070$U%u?oi>k!3!U|B#)#%$qS{E*ngCm#&B@PxJ=t*
zoX=P844;In3v+_V>nDhp54b;8KBJ3#?E03xerFq!UCu*jbb^>>cj#IRej$8*VZkleCF0^*=CVJnl~{nm}0u5Y|_ge||y=
z_!|+)Hd~R+mjpd*mDWhL@B^inr!kzk!>1pH+F9^J5J%`19^u0g;PO~Bj(CtJjcyVx
zmx;l#o`?`V*g-$S2O)6@)?0L$;%9wY?=TLJ1QBzV9s_1Jd=SsVOQ`r;+An|t!wPUV
zAS7rYOFCm5H_}_8bJMxw;4P(RsJx<`Fu#!PFATZH4ZLxy_aCZuqkAi{zc*{M<3iFQdm`
zT>9Z$WY%`Gyjp+ST4q8vbGS42byS6x-V%{Yt{ZEW{XFT1%b~*3Ie8FU}+tChDPFmqXtU7WaN#5
zj^K*xz)x=(0rMSM4v8FQN3wm1ClXQF<9`6M;!X3H0|(WK-QR`-b>JSdb1w4QwsEXC
zxQ@OL9`<9C8>!}Z2dFPryfS_~Ws{uwFKK)2F)j^;dAY1PmtPwUo`}x9+;h}eSWkp@
zA@973#t#f!d$_mC&5UMGh+8k;!%togqo@|l
z@WY`ckuZ43A+atAeX^~YGHH-_XMqD7eaY@_eP9}R+Z@~nFZQ$^rMBM;!V8bbvRWJv
z=Tp6JGF}+R5~Ro{+r|tzv}Z{TQ}A45xLS(bkm`2d3H&wSWU)5byIQg~nK`o{9BxM0
zB7ZRSdGWF{nQRUV3&o#?N#&49bJfg)v2ynuTE5Wj>dym?mYJL|&rp3O$Za8C)EVyf
zUq}NPbMEhme^MU2Z;r2m4YF6m;Nn(T%0&E@!Y!u@adhqv+i1;faF`aH&YhgT8F2gp
zJY|Xkks#tS@V$2Qqx>KiPZH^Dp#2tC>HWtLgsJ-$^W
zBbPRgo9wrHm(};O?`;D)usE~ES>s`p5sw)YK5VkfEQ2hvW?2Q6StXtUWtLgtxonUY
zcx9GB<2+
z%jMu~Ks`WvWC=smFH*Mr+Zdaer`ZZOc!n&$&-bAd?)tKht~|WI
z`-^=`IELHCTU+4>aXPT1sj-gjBPL6FAZK>wSCq&etUM%P(Q3q=lHIdp9x^#|Zx~PB
z)uzbXR%~>+AM)iZ=&d*I5?3w@W{gMMT8k4
z8EGcmb~W+a00oIVYmDa7bKp8d&Rkv|gzkDu`<}mWo!~-sIK0N53xkNx%x9_UIbsM*
zB0#(rCtS!(VLjcI@?mjf978)~_FH}2!=9xl69feYE}JaAH^OwjLQ!t*cZor1g$eF*
z+=plT-sD>R%cnN6lktFQ4r;;XdXx)p^X?nRW)qjW%!CWJ#v~rDPX~tNaLk5XAmz!K
zxn#mx4x>B?J4yKv`j$8?PnI5?@vPYyXKXy*+W!FT@I!>V!5=C5ha;I8%y9oBA{{YK_WOxfB&U{DRvCK~;#z=rmmr?5-(s!E~4}rvqq7a)?*XWmq9YH}3
zV%P(-!yQj0y!7)J;JI8`c|4sWVB9Cc@a?vIp5@iD%ugmUctSpNr~d$YljNV-`?AYC
zceumg9}w8drTa%$2x||B?{|5Iox^hV8v`sb5x5C@xm+Iuq=FU-2Z93a{Er)bqywuP
z2Ur_of7uVrm*>D|z~SIW5!5a4dX9PUgTvIp!`D%QAaJzxbMqvgID4C)Y>p?nERO^o
zk_@^`J{yI;i?qHn~
z5bAiw%Nfk_xdTjLH(Uk)5ky2R?TXiw4@
z349^ngP1v&S;fn6Bd9gLH^LZE^KLyezU;lWK4SJ`n_kO_q+2jH1?op7Vd-JUH!>rJ
zGfWxBeRw@?iFvtuiPwX-e*)tPyKtI$&j&mgd&p*4obdykwqGX7EG})oJ{Wo6?q75r
z)tz}Ik5?;xu2N%st_>m0!sN*hBzNVI+XKZ)43N=Kr%o$
zb18E((#BePnC1xsW&q$rpGiMS=6q=&4~wlimTj;b0g*7+<;BZ7U#K4L%=IIzk3z)a
z6yn%G-tMfOPPv#Px$DE!P)tG=PY!MR;>_i_U2U@rw+IKT?2hh%I$a-fH;x$*yJ2v$
z_c+W(F`MV*Kjafv0kYmMBzv$4gtttG;T!DoIqv7ic;hvY!>15uHZb#NdAHN31@WF3
zU9&lkWdbp026!$*axCY9g*LckW%JT)Evd@8Sbd^$`0d
zffPHq8vxCe^F7CK6T^YVGbgz0j}0LMAe7eM2M-p$;8+Jf19_J&Ui@nuws=GW>J8`d
z9%Gl7lpS*!hCB{Bf!}t9{YS)>MX(MdWF{QuP$N=hK0-KY%RA=8a}Lh~mKXOUT*pqQ
zczd&vEI)APv5aEkPng#-bpdj?MjRIG&l~Y?jQnx$y91v8042j%>N)=a
zR~qUkUReDoH&KTci3Z`07Y7!EJ7k^*JLA=01(sNAvC#jNu
zmlgpP?s9N5zU~g7pMnkcl9%0|;Hgexr
zLz6Eb@&5qr)BaAl~g&co}&}*R{dOU#IwVeFY0oA
z_|>hJF<6Slpvli9^@U^dGPSh#BfhB$8z*WIccPDEp
zlW~jz$B=wOo%vjx!>!r_mBpM~f7*LPY|aZTCq3I7Sll|>lLd#ldRPoR8v>Ev7g2f{
zek{4$PjS_-@jXS@F&57Wb7h$ulJR%r^P#h}vz#yY3aaCnkJZz1YZc%t*qW+@?zB#C6Y}=ZH7EImvoF
zHru8bHZcQEU{)B_#OZ`w(r9{E=ELM
zrOr#0$)Wp#n|{1xn{l2n*s)^9cx9e42y*Vv0$Bs++2diJxVm^;A{>ps9s}kMEp71M
z8yn!KbEq=kDYj>J+v2_(G1>lXCbGlo^{3#xdB`oYe6k7RIkP3Y@}4`g%O%4l#}@3G
zU3qzgE|z&4Wu8{!*4uvs-y=^9vC|dkcOlAgY1x+#P#c15vdN6WWVyY>-A6Mf+mR`L
zXNTj5a(vF`7JSL^1n$W8j|}P6zdwk#`bY3i8~x?HX)XpF?)qB>!Jc+apD6l=!b=k}
zb-6C=@t#KV**-JQS>-2#&QFZ+c|Ql6ENt_ZS>!(#z*%LUa^D6l@X0K)_mE+h%MSi7
zJZ;=g{AHhm{Q^8c(0>ANe~+&Z;PvJF1?$7z>N@aVuHS>#;CTPU045Lt00II60s;a8
z0|5X40000101+WEK~Z6GfsvuH!O;-m@bUlJ00;pA00BP`tzi{3XK|K$L5Nio5BikF
zj6uZog3>Wv5pHy0mUSD-brf({?s}eLjAjZ8x)*+;8)gfOLU7=yY}5(}_?hOSZ#jU?
z2iy>_!L*6za=-Hv!Su>qf$k|xOZ6{$+-_#0t<>#c9LiQpi8z6*K<^Q`sMAd6AO|A*
zg!r1MYNq}o*G?hhaCNBo<{b#a)e?YD<_IH~QHB{V0#O%@!4xCrC~nzg*BneXs}g`W
zS!EdkPF0OH6yvyT2qEq{c~;ysI}8y=-#KE`zGNY2<5AF*gl3ATF*T)JUR6@$a@trV
z%aO&k-#{xQw=G?q?>OY8b+C*hM@Uz~Sa!bh6^%tiqtA2A(*S
z0Du@<9Bgn4ADGx?D?whR1+HJjbJVvwh1PIH<$rNN)8ak?@t}v`*gpg_>{0`wlv_m
zQ#B32ETD{1SAL*aMIFEp3rn~InZhsPbz15@iN2t-n!S@>Gt)3^%D$$YY^xDUks5E0
z?q#aW4HclwP1ZpKonGN?fOo62vITgl!9xHI7wQ7(lw}N%V;`8;GMY$u>LDKIQxRL<
zBOS+x@GM;64q%0FJwR5jV(TCvQKldVGQhkh8KfpUX>0%ugL_zJ;-3*4M(=fqy#e^B
z_{#AIJl@oYQ2xn$jQR5jL6UAiuEnvJh%1+VVUvSMWLhap%#qu&kIZDP
zNXmntMqPrunVUCi_XR-;=6sT&cFiY}4pJx7)LKH1aaetvOf|4CWTDbrEYA6qy`J2Gtzx+BzYtWz?AsPgZ&9I9eIk$AR1;u(xqE5j7v^?OQ5b@#@ghnky~BA0G{nQk
zNmPYe#@89_FcT}OlOcgXWS9;j;v>k^R0GJ$EK$vWu2Pb$V64B%GY7*L;2px@4VU5w
zb$%t=iB-hR2H*>37{V$S=4?^yiQxAoOI3nZ`4apBxrQD{3kmTv1wBipxYTh1z#gM?
zUS*0#@fTFY?mHoitCe*RO~UUo&VO>1+u|eJ+$n5%h-D225ZLBb%0kBsJkSE9BeoMk
zyLybLQ3tQYqLcAo%wQoLm}}e-YD7`4_UZIYsPNi;qQ#_BB5vXhvtT&V`{DqxO5u+(
zBDFKnKB36{LD~6_lNoZ{JAd_t-M(v|h$aYz8(&u)u!0RjUSO1%62q9jpK!D$E^eFY
z=4j~oi&z@)!qG-G6<%0{=Hjl1YZe^fAV@EnY5RaJxa8ho
z8Y43+!NegY3mExBU!+|_n6$MXBktoRmKD0CVc_ya@Yar4t?A>-7T#StjH=zKXxV@2Ezs2YoSM+$0TH${w9m$Z2mHgdAM433nA*aPq;u7KLk#wx21K#%`{eGW**Bsf7h-6^R2@sknlQIUn%jP`=>lZ_GzXlmZd#gcL5xuV2)}V)ViH
zmh3v)^Ft8H^&_6B%%Uome-J5)x|TS>GgX;?ij7-bMny`sYBV%_%TA@%M^qB>*-=J1
zlm)!Z^#oCuP?{ECa1#b9s
zh0231Em{fV6=Ijc6F$7I8)r+Km9D69P+B49%g>fYa`hcVhF0l6;KkYwD7`$B&%(MY60*v?AMZ|2`s_Fq1eGp
zA=GLhU#WK{Qmfpjgb#NJ4M9&Z?8>-ZnZy^xj28)TKayNy^%B5U@<5y*BM>}Pr#Q$F
zawl+iCAFBK0jQKV1XMmkH*|+_b)^$Fgk}?zroCLbLDe}W9L6l>pWtOsyyM(n*F3W<
zHRB7F3iWYY1u3mciXg`vbtfJm4?uchD|a6rU|W{&RVpD(e4gVQBdu1QAq(pgQ8wMG
zKB)N8JfV1SLYYAE%Ea>`3{yBLh0N@ah$^QVP1S5cQumSFV;I;e2Z5#5Y
z2+sSEj+yuqls{315t@b;2=l`#uHXsW21~O!hZ1=L7*d-fDo&9mu(3->REiiQmHFhJ<321#%0f|e79-uIpgFp`wlsAcVbyrihL)ht
zipv6%4`k%6*@&~3xB`}UFz*D=j8qgen5?Y<4*^x6BWNkK$Ax3d0{|586*MtwsyJCa
zMsa7TR40}-dncqtv1ZFDvGAe_WNeJhf~W>db;d;6C;>;f1u}DnJ8INCz;3jOEy~dL
zKl-Q=E)KSouUbKEfv@rBV>LjE*-5hd2_*Vic{h
zGa@X&n40{eVx!XGg{KuWZ@QTn(LLduc5am|s8g~g?B+Qo+N$*}zU6kLpbL1+7$?BO
zDVi=&#}TLsr)B`#PUG7bk~x8KW)H+ol@H8*%X^9pgNfk1F{cCSBIHi(_<-#Qc!r><
zTE^gkfZPtH!{EN)(v=ytVCocmeBTbEUh#Ox^#Y^izI!A^9JtgDFVVZ`#Fyd#;N=08S%1R`Gtse?P&@XvU
zl@vzgAo&0Vj43K(BK(joqgacw5bQ*^E%+guT8yG!QiXjkYXa30l_oUE+#oKzMKHZ)
zEwx{Xu%-79+OTt&2TU1ZHl`Ce91-rYO%_Uq>IP-a;sg`8<-rYJQQhuhJqxKtP|hjp
zv0vbXz?piKJ0K0A+y^nk{XqsRxDFMTL&=`)4$qioJ9&f+@eBnA;wvikD4?G)vXl<3
zi#2(Z0tjL^J$+qNF=b^7DhZ(ZfuI~-8;z`bOWf*V0#+8JTIW0n>|
z`J!c`Ogsx*q%$O46nY~Gw!)xf4X-mK4`K~T395${GfBbbS)XBr0kOvfs4=|e1bw8S
zSx#0YS_5HA2uF?TJZ4!wJNHu3IX*1ZOR6nJ#W*7)6`Nj3|a%tQp5Vc-MeQ-aSEXFMXZG-QU19w?P5
zNw;yVj=o_$Cx#hVW!y!TS(hV2eZY&(%8kb3Ar3Y7am`9QV5>kml?qGd6~Vp|7{yBn
z2gr$OK9Zl(wzz7do2H>%Kw+s?U`VUI3=KSX%(V&d-{*55$*?}R`le!@VtcoZKm*9Dh
zRrxA)C?j?hh#lyw`X(MS+!4m6L?m|xb#5=
z>A6Y=_$LBCaobTY!ihkmb53893jx0nz8@D9zd&W^DNxb}BXG)djKGaYf8huOJ7uLk
z63U_KR93piV|D|~VXDh=W67Ceyp;fO76e>TSYyi-aD3q~teT6OY`EwcA)EuJfC3pE$^iGqMEcNem~Fc%+*V;~mSQC47Cc!NiS
zJb&z?gJEVmvrvzqfqX(xhn#MZN~kie$>Ww^p#arKxCYfg=W&Pu4wBqerL!3{(MOrV
z{lZ=gv9ZOA8sGTEEf|MH-n{D-c7Emywe>RdJ`fuJ01>-PP}3M^!*Dr=f&)y}2y%+v
zZxL)-y+I`&_<$&_cNPE_s)@A;#0{#hVpQ430~=Lm=z+{V5U>=)QRWZ@OI(vGmEt)A
zy<$Rx(Fhp&ggHmlQ^d)Ojhoa1d5AP`Qrd)rRs_@BIFdsA+@N9Zh#e?YQy-8cKxw3e
zoWPGH)k~^{Z58SuI1BMBpdw#jtV5LpETAit5Mj9>TmZ^*?krd82nJr|5Fc{cNa|p}
z@fVd(aoEZRobxJpiPu+CRi-yGGT#1Ww>lVR*4qeT;=Jlv0eFN|`$K4gZ%hS&
z1d5{UW&Lv~P~1y_07nxMVNFUTuF8dhk{=B~0BelE;cJVc8mT}S6w4{O%QQ?&5wA5J
z+e4T8jsPKJoZ}NlZU?aMJ~K3-RC>;Qw*m}Y+kG)O1gmDyM|B?K8t!NX!;ra@P3eU;
zlhmdj!4LE@tjDXEATgIYiFE>26%@YB%1pK)A4l_usk1}%Ew3;JK)6QWq2R9G*yI7B
zwp`A`t&2kS)TJB^kxR3ei~54CR-)SOVnOmp50ZeXx_IpJZUalF09$Ba@mMZbq|Y5k
zi9_<13$U?9z0C$#t<78zYv~8cUr@f^q`QxCWqYrt8K6E~)ER_dQTl^gA=&YZy?V?P
zB%Ke;pm716d1i9I8HsE!h|L!L62uCtfdsSh@c>tdxk`|AQI@h^qa5)R%?_K6bX!Rb
z!3NB9AIc)zVk;mxhTY+X{e-A4UKXG+xha)&RcP#js38jlmNl3BK}%nR4OOz8#Yayy
z$r=W|Jdl{#q-)d_wAoA_A83}uK4J1uiZEZ5veFNTH2Gm8nr_>Nub!7P6EF
z%(G~&mMC!^GXUFT)XYf1Y>Sk2_=TUagtdH#B(eB}We1W|pxu^1M&HU?LeAN6{)DQ_
zXNZ>a8QT`B&>XNrY2}*7?GdF9Q3ojBmg*^NG^m47E&X{Dxr~
zn^Yq&YCJiO$|_~cmEQj{?w4W5q^byMF8D4&f1Y
zuAZlZuF}7AAJiIsZW>;`)WBg+n4vQ}cziOkS6*Ni-X_5%_!T8WUM}mYOGc<1D5+OD
zw>6875)74jhWVz$Wx~8a?gKy(zNM5;MV^c$0ocKW04hj@f0%n-W#yQvgI4sl4h;<9
z-^>LBSHkg81+IZZ<}KK_Dty6Xe4gKIz>JSRFU7#XOhaN;klb2_R2V~L_x#C
zOE|fd75jt0FI$`K{l?m#id3x~kz}j8oDfhEL`pl~;soWsQQ2L9*Y^xdUs=pWMyFwL
zbt3kflrC>YhBl^$M823-ASM~UBfg|Pf!hF_J)WYgt*uaRWB7=`0i
zQ1W04Fhv63Fa%pHv#1CSM--7r18bl>u;?qcACs66f}5(V2u;|C(yKj7UYK@w1fkq0
z@o|C~cZ|B!wc9>xa-xyWQ!K_cJ{>^=r)lBle6hj*0JAl5i-BzoUx@hc81}P9<(O07
z_XH2XzdD30DcIsEsN8fE<|`r}bsYHx(+Ye%4eQYzS%=B)0J#BpW&<{n(k=rglr_M0
z#Hmec1WqijKfi+^*8mLWCWw$k076v%VZdZ0J{K)PPB0OFf(!-J4+Vars!lfZ1!+jVhtUV!J8Cpi@05=2_^?rfxtc`8
z=#?k|Q>|QF71imLp-8TlFZl^{xt`%vO|g>_E`s@qg@@dFS?FSUpuDk>HI2hS!Gfdt
zOQolx2nZM4GB77|9sWaPV~Nm=h%=IlDKYdIK~&K|CSg!vXeBS>+ZRBE#Y{5onG||h
zsIYGhcN21ZitfqSgm*syEV9t?C@4l~F{cpkJ|NX_6BPI%{{WDJDvI$eN+*Lc4TIFx
zC&ceyk(tahX!jqn=B160F@#}e#V}cnANFzeeQ4Ztf0*)>RV(y`I#RI*VL|p_q4UIZpYaZ_)Q`#v?lA%@hlZ)uK{=_
zM{)+4eT_)|=1>82_Q&uD8X60KgSJvblFUtkx4EBn4F?ba7O>RF4((gF9KR5bY^?*`
zoB4zoz`KYlTZpFPD`C4nOt^N}s7No&xWzSlo5kK?uHH?n{{UsQ_mKYpnur9$%(c1_
zS+KjN5V>DuKs0y)24zajG0ADu#5@%*#S;9~YPYy+g0UQZ%$P=$@<-kr
zyLIXfw;PW4%qthXzou7kR*$&gT3oRuJgR2$BC~7GW=Iv4{{ZS;stqfC63C-$@z=Qe
zre!yb{@^QHP!^S6+$-?K>m4yt)}1kh2*7C>RUN->&i5L=V6GkYDu-`!pePhvwt|eE
z#l*RAC*k!mWnP7VavEE=$z@f+Qq;XmhA7**cL75;LO}9QOh$kJFFc3LDuyHWJ&=cw
zaK9Eo;9#bMA9FEcac~6D#vuy%(Eus7K^+zmw>KYxEuo=>2BkGKYgYZiYK6SNmL=2C
z`i^QX@pC9w$q5ZFxH#RtOEqi~+hN;bcf3bZ+kiAeO$Gs_P5@M>AepRl+-S<&-c>=p
zGmcd(CBdA=L_p>^KMAUl!bQ#pGNc@NxoR7EbC{{Cw{hl`;wWpo)Zn+ts7#`jM-f44
zd5j;XYh8R1G2gaW7Ed8E5eJ+=9hD5_V7C>##pry_IGko&k#b(37@M|5f&fwj1_rdL
zMPqXFHbvyH#i~X1Fb-LNxk8FPOhqr;b##=dn5{BRB6P>AgUD_Qv}>5H2gFAU9I(r?
z42LWFKP)$Zw~W>ya#HBKe1C8$@Xlm7JwUh=Kx@>(57ZS~ZAUn&0WOvR-M%2iteTLu
zCo?;sa`<>47zI+UeHffNLSrPYSQY@g95E{dx?zTmTq33249%y!bLz;HZ0i#BaZcWI
zcJ5z8ZxL9nlD$As5{3MqGO$1arp^BV61cmT^
zT204Nshb;u7VR#iM(Yf8XZHcDk<<3Bp-?-PrdHFUMQ3*iA%-a9+#m+*d_sps>~mlH
z7*GMt_I%1E2}{$-6=`eT#TiiQ=2&C67C3}(d_cORx-Dm<<KQUn!=@irP
z0T*%et@Rbl@`-LlV&798i$xDL4p(kuwXx64TdW$i(x|TU4X8n|rQnA`F>=>gc$D0r
zSx%)*gpa2|2X*O-SU7{R#p#rk!5{H`z`g>*oc{pT6d4K=Rpp3kQWc{++#;%o`>CV=
zuAtUgmC56{HlS^10Ss~vUAqwQO%-bs2&QBQ3?x+o#5#0tV#?0VJw(Unfz+ZPqr^fo
zZ1CbUuLuanA0qB8O^EUne&K8q-$>k5a({B61W>SH3LZxVK>!XFr5ah{7``C7(Ureb
zA1`^8j4PwLUuChs?8*T5;5?Gq3$j2>_b9XO`JWUz{Om1Xqu+MiWELAygnQ8M_$6fyb-en57^GsE=k<$M0&igK=
zMXu16hhXHs5?W*ljUY%U*o0qDD-b+6gDJr~lvL{bm5>~-bxi-BZb0RTYP*>4b3p7?u-|LBT_9wOo$aU+;JaK5JFYIv_P@}S#brt
z%3)sBdzfA~jGE>c`9CF`L#ttN>I<#Uj?oItDr78UcMj7wkS32u)HYEqWmZ!x4LM==
zQ3pk|mG3aOmI6c8o|71Nj6@a9l`$kthqw!p@J~=Ucj$<9m58I%x4XpWCr-!A0peLa
z5BGA`Qnq9bf8OQ@sDgm^d`88An`%6D(c&CXyDcOu6$j=59D2I;JY3yD@Q)Bn0m*y0
zN01nH{{Xl32B^z__jCTL*x^tDn87#2z&(yH6%zq$;6Zi?Z{lNqVRAhkM1@|jpK%za
zwSB^imv?%Z$r!p!KXn3|8tg!~ZE;*(A5uB$9GVY;IHB^i)qmX49m)O>OLs^OFor&PN3%C&IYPfL5PgQ
z45$*age+>M8Xp%qsQH39eaEFI%>LnskzmE2MpO!4xn@FqM1~X0O4Gikn9or9(zQwn
zc)3Yu;th&EmH}5+6&CSpyiAq$;ytheUKMEWYt``sXg?C|C#sj+{lpGR@I!-NF{#t(
zi^3UEWV~Sz4Kp+L6QNI0jy{N@p=qo{jy)KhkTR>pS{)4&aTr7cIQ&7@m|0vx003?m
zx&a+70LtKjLEs{q4q^qVnQBN0S>ka+Dr-LA7O@4YmcT6>u06~Y^BGuv1*%o&_?VM@
zbjSFa^AzbZh83az0F6MXDvo#pqEn`}MDLYsW)-N=wks65Kp&vIMXDf2f&~|2=?oqq
zQjSjMX=nw^rt)v*RR9B|du}h()vu);yXsN={=+cUH_0rl2C*I?^M?>IcpO;?DW?$u
zS$A+T3B#r}4J;$4xrVnWt>Q5R^N$7jl;L+A`G#`ySba;N%+UIY^i`QsDZQlmv;QRzO4NggZ<^WXl0m@ck^Bv0CQCQ5**>o@!2K>Pfwg}y?a;ZVgIk?G;
zBr|r@dg1phmU?4)Q~|PF#XDon)=+tcdppD}g|BlW6|tIRGaE;!w5q66PVgQZnX5BL=cCTPsifrppN4a&geUuQQu?6WaLyhma@pX$3YR=OKKqdhRSGa$#LKuMj0;#2^b6s6$(&3>R}-n
z<{WaBG&OzZQHZeJY$mnlGk~`15yn5>UHXh|fdkiq+$t*IVxT`=InK$FQ+ptHa;M=754_0IncM-B#z8W69>eoQgS$flq%}66AVt5+9k^p)>)mvYUP9i;R6Q{u8mB1e38o)_>G#H
z1}nr<`=F}c5S
zXqk#wm6OCarD~vzD5HM^iJXfd~b!^AL6xx{CN8FKl~(cy47Ic@Z|&(fmuzbt#iHe{$iL
zn;%R}aM(~$o0lP%;nOp7*dn~N;t^(6RHCwx%nfeMU0eh#y;NsHC)~P{m;SiQAqovK
zGHT?23$&t2C2h;KcQOw+FjRb5#hgF@9+Mlys__8>HgPzZEfEFfIqqcHT%o~v(_Upn
zjgDo@7u6Ug*r55SfB-$cN{X5$GB?2}?k$hSSX3(5}@RBp#8X_%niakD_(SyGEw$Imm_&(8RJ+&C2%5
z8yP?dF;mFL0Yf+6E9Yd3*YFC
zMI}k+mNXHY$&G^3l2m&UH=Xwq*>>L*+(99B*95_R@HVejM9yWJD(N;kmp~HAWu?;+
z-xc9*U^4=WxR!#zyHNq{_Yf^rZYrD6t{D59w3Z*I^Df@P(5rgE6cbG%x$;}7p23CV
z*ZPLFXNa4(Eqj`J;dPl_#c5Q2Na?T5AeIHSh2|IlH)_O&)y<)JgH%ibdrV1yL!#Ed
ze2`{qS%2!5s?1^Dp?L(sjbi>_noQj>4yYJ09z78DdU-hv^XUJpt9s7web}p4Tk=sIsvMMk>c4jQ+>x0d2TN4
zCwI(2R(;EC@7!IB9tl8!aQ&oi?^4}{WH6PBTTWxK2LY$*TB}wZ3WH0o
z%f1WMyhX*A^Di_l;Y>gT2FBa|TlV|u|QoAl<`{fGQ$gMw)tQf0=wcRQbMZ&GJ%=Z
zaRIA${E#?*pvU@!ODSI!EBZ?Jh__YL>{XbEtBGcd;AKIE++rjRiRKtBq+$TEO!|d(
zYW5;kmKDd;7S1@9l)8au4aJX}xMhaCOsvANY3?1Uqn2f$Xgm$>Fh$5S&ZQ}NtTM1I
zrVlchfclDJ4=-$ELY<@;H+muhrxt1@?*uq{NM*K)Hxh`Z`pm0RfF5~ZjqU??e-oXv
z1|k$j$g>%FBh_1(>l~yqR#Q2S-IvZ~kPVfhUgn=sD+lERf<4MjReFxGaEdb(S5%vm
z?j2=#g?E$8rGKdOaohwsGpGlnsfq;boa~k>2rOu?6ND?yB{y5d3fy^ss=*b2MJ2|w
zR~5qiK&5`7wio6l0uC_8alpP92O|2oK7Pf78$YF1xmk}lU4N=
zIx~<3$_QH4A%M^>%9bPLps&&(%~M$
z0b4uqi$94-4UaW1fHr2~Tc}6?TCTbu;}P^7*eY0`p@8|7+y>SBMCmg?vkj+ce&*A|
z-fmn*X^bBc0I4mq^Ai+RZ@AR3kT`+`Ist|cnT6XMYS+AxxXeqziIQcPV(~U6#Teq@
z@NH{%#jL%=N(_%vKSf)ZudKlUDoob|cw^T9>&)j?0~{~8kQmL=#B!0t*Ze>x#2D4b
zL10c67O4ZvCobi~FUq1-EfmKw56K&aZJAy(xpWl@mh(m>)Er&c#IeBM-k|{%e+w=)
zK4tM;nGDWaY;7O{bu2FC(Wj;fX@Qwk{6e**SJbwoREGqlwc!QaE{Lqyb2$`V5!m^0
z95aF-fXa39ZYcgo5{*h%PK%UElo~N`L%>nN9P1c8czcejEW48!UAu`)pJaZ_cq3gX
z^%2TOB+TjV2}?t`1h`K|F_LG61~%@`DlJyAv%E5nv)oeHdgcMS8X}ehFnL5U
zaR&l?#7M*4##{<}f!Xs9D`U(y6gVyrc6*D!KFNK6jWpUUS%eEKxrJHi%N%x(aRD>P
z%*z2f983+Fy1JMZK4o0cXuV77x?-{$nM(ae<<%TiID(?>X*!Es+liazh?t-%y9}$JmwaIuNEGVFb|Z!+$>qTEnZrbf(|hBSh`z*q*RHVSI8e09})Zr3NJ}q
zFnE>T57y>*oXX4eFdFrN$C1)YrOqBlnZQAP*oZy
zg|`tmeFYP*(uG8Vs6GO=sC
zI6o8z3KM2ftFA~^9jf#5D+-%8FHaGrwQaTTdL{ugZ*WdnnDEDyBDEU0B3n<0N;NFS
zN)Hi$G{lrKT+3E!5p`8CgP%}3C4m$%no%afUo$L%19f2PbKG#X?i=}b%W!Y;E9LbA
zD2It+(RIY#3y#DEqXf|$_+`<%^A#2Qi0J*yu%>-nGN&G*RbY&=%o?(Dm{9>NHf}08
zY?i%TUr_-Q6ta}Co~27fP{pk2F-Xi2Em<#bA;wS!yjpM+Wq0AhG{hBO$`
zx4vbOwxWl$KITk`iNWBQRM<3(PK?eEZ&4{1puKs5YA_r-SaJ=S2Rz)f8fe$zFAsy2
zzB*1K?RJ>kO6nF)Y)1ZY%UMd#;pH$AZO62L1yJjV2iuLNUSpC@=y>%HtO&Q5F2%a!
zxpvr9*x&URvyzi17=EFuT8ii9RcT&RpPpf)*fMta0C+hUwTgZ13E+tuY7%Xju
zaj|mMS1=8aOukkt)o478`(^vWwm7J^a_;5s?ySsU-p9G7kH$RRcXRBO6En2mLVI|gTzfjjVTir`>LTp`1a5kFGW>T*?J0ns>2VW3GvsJ<-(upS=_s>B`Wju#5fqYKzoJ(NTv;DS&iWthm#Cn
zh{wpLB&=4kquMg2%@nL<{{SEP3lhW_2*4QZJ6-|Be2g#@n@tf)_HC%B$Kinu+l?Xb$vmM5F{^!QC0xEabNaoIZtF=KOeh`pbOATOks?*
z@hqrO^k2!iP)#3wpUqZjSR1hVO`7T?dQv
zC@h-xBB5?upNHyPTS(>Id_|ib)<_JN+QoFxs|LdXmDF~L!SNmZ!Y_JuVY~OZRBVPj
zh*WdP>C{GGXWR`YU|VXt>LAs*57e~F;2CWwPGf0NP0=;EOISDkEvjUvVE5
z1FPc%v^~SRcsaREJD05YT9w08Rgd=^v01}`fS}2>{w0_-Rbt;Vjbbgs+-JmXa>F91
zqWFy|DphC4a}q;IaPHC6s>$I8zEphrfTGoA$GKp#QeS18DH4S?-#
z$xO9!ohP`n0@y8hAh-t%aLD{Yqk`(ZvX0n2bp`w)7nadrAk~d=4Ok>ANaQouaW0~x
z1@>#^52!S7kOgQS*h9p66}o<}?j{QI11oK9OUj>7lYc_MJA_=ixPRM;BA||)rb>=(
z9OJ~N&I{&e^HWZ~A<;@;DL@5Pi`EGD?xD6s0axYQIS~jHV;?{2Cc4^47HS#60U<$%
zTQTg^pfB+e@be2ox?@)cJByfB!nM>nOCUeLiG@a@hH3K~@S{*1gnPsp2Y_W*!tXKS
z-q`6Zw~BWw(;Icq@
zh2~(m3*VZC4905eH^>yt9J0@0f#_!j|1$
zEP^@U01=Y!XX(epxbJ8MmEY7|=S0YKGS%2!56>_bMK~|?_XD{*
zbN;{FX6aJzSLBQeEQ4>_kT*nFIVHUPM=Y^eOq=rru9h`%fNAHLANRPc9+}KuTgF%-
z;Y#fi&M!Sdys)-be!0}UEAvob+X%m?cmDtqy&?>!p8o)emh(mdAA&SM)%fGYLkA7q
zOGmTBvZxpm+@8cxwcai&KAgpCR-HSRCwg4Ih8ame9X|lNW%-^VG8TiZrB~w&!G^hjR>;YWr@Dy1XZW^Dl@H95W@!p
za@JH=Eky=8OmHU<#caiY_UEmrIOOkGL?-D35)
zP^*z~%)F4(PwqMvz=!c<#K2=YF5348P+H-?sYEcQ+%oLsy6#-h)EorSM?Jox(iLEN
zm0N*d%&@Evn6B+i7fO`Fu4A81J3{Dtaq|b9)d6YVGKpRwSC)-qs))`mYriA8e3*@r
zg=tJeGy|HBa36yn_KuOcdrP_%V!y%1HY
zd04o#D7xq7F2#=qBKrZYa}yc?&A@fhyO+Y-ikk<5JQ$`fAa#;IZ}Ak0jB0qekWpfw
zm*gwd#8y4TFe+V&S&wwBed;*flD%Kam0kenq9yyaXI7^hfl?R+`%LFxSNeZYE}CRL
zN)}FE;9?Pz2`Sg0^gQ|bmi??FKE9(XO6J*7Y9n{dphn#DDMwDbfwz~0a~zL2M%8Nj
zmnsE2SD(yGV9MQ@e7r-+OyXB10d4Fh&Qj?}0{KJ&9BU70-|8O~Jdclq%tCRddjT^G
z7C7Z}`Ik&5tZgx$s|tyPFFi0uuP=@SMQJshE2!;FokO~`@snc$5*(N6AHB4zcRwtjmKE?av8e~MTcISKk+E-y|((xGh1yY!>l!o
z%G$>$l%M-6^lE`aK{Y26ipKg$%%z27%FU1$QXTN=7uDO@m(oIF<^Vir
ze&Helj%VXO2!+U@FHmS-nIRNb~+qT!dT
z%%z}Q8o3=84+sh%1*4fbM=OtTd$=o5@->-3xqslWLNMS;kRdU>B}c)P(w_uX*(r2QOlaFdDrn45rW(vSKQST
zT@yEf5xMHlI10vn!chPQ?ijSEE(ml0yzC%}(=k!B
zNI5*gkok(Fu(5UUYw8l4&|)4&JzRiL5C(cO%)!HZ7V0EK7sEC%oN|e!
zL*lA0lFLOvfz${0HtEz$B{H9^O3Mp$y-FPK$qHC2S!xv_|w=tc0zWBksUz}bEc-i@_#TZ##Cqk=n-~(q7rdl4Px|Xas7F8CSmEa`STpU;wrKz$?swi_YN{b+ZWZ
zjlhSTBwsoH;k|m*{N`8|7s^5vhS7e{0FmXgI<-K|Rz%lqo
zPft+>3g-9f8d#=lr`&7_(FLuFc!Ek{D$8QnUP*Wy3pT@b1jC@kd%DpeJm-mWDA{Vt
z>0dvHnX#xGyoe&2Dp+Oz05cT`=GRUp5M7X!V^;GhR!{}J*5N41GHf3nCF;T${WF~`
z0Z*PI;LcuC&u_$`82;V&C~m77hgAkvB?xAA&3cxZrqtZZ4+l?(DUqIs`G8(@CMK)*^*fUg+W}2SmE!2^WvbT4^iXr5&_U+zh>@
z1BMkGp{Rx{uMi6^U(_s@b!-)nA1^TyHk-1d)#H{J6#0OZ38BmtTg&#rj7Mk73XY{%
zr)I=CnJl*YWtSmCQlQm6OC82*iP&q5TvY{ZFEzh<`GEjq13)9lRl)|-+Y&mo+Pwr|
zie6kc70-w@YvZGj<^?NMbJV4!q}uks#A}GEyt1E?W(K|qM$*K)gib3{E}5mZd4VYN
zp6Va~a3SM-e$`O~J*n1@|fq
zaDb73#TuKCp^MB^%Lst65)!vC)hO;RL@`4(U9!RzahYo~xR7SXF6J6}joE?Fe8b06
z2Vc-jI1?ByLqfGw6z)_ahQXooeGxni#Ig_8If4%^U)xav(yoAPc|TJ+Y79Fu^pwF_
zlYeoVi<>p-A80XIU#5Ci1Vj%3ZPC#2Fh=q_T|xRJ{>Pa~KpU3+7~PGwPcRw}gc8}^
z8lJ&=%qnjWa?rM*3d(oq5HNsraUCwfOt1>?MybaV+BIRc=!=U+?4l{TTEEOh2&(`$
z)O>!0Ie+gld#ARCUCWtX`+&e>_P+rz64s=3G|gFzxo&wc%&U>uZN3R>WYgEowZ0=U
zahgguID&RAFQ@|2+-$*)%fy+qdx2P5r(XvXn+0r|q46la0=s>s8>DhE-V*m@u>);y
z%%=e~T&qokzaAwF8s;L3Ed5Ir2z|$Nw*!&&00@?O?fp#dK&`$2IhKk`rH^3bhCpg3
znXa&EP^zyiEwKfIPnV)8lPG=6FqIgC(a=^3<|w&jgKz~bdc2DoN5sez^$_Fb{Z4dz
zoTWhQ$yETTKv%z~Hy7rMQmrvqeXq@wvu;~bh_EJ?Xz_Oi33gQcL~CAhUj#97%elmR
z%qh7@T|1jYnB~pf7XSr=c~wpJ#ZlAymUn}wDyit>)CA$fsYP>bU$4}BLxd^-FI=!=
z2rnJNxC#f0i~t+pj#`x}HTjRI0h+mTc2M1a2mOEXp(}u4dL_x0Q|Gb^icnP}D(SE}O5j@iI$lzL4;K7@~gxj^-*>3>WT#j^PD>Y&IGF#9kxrJTQS|
zO|
zap%9oFvKp)ezSivlGB=jS;m+msmL|&!^C#1LLhaTuef={;N(|5!W5TP1WroZ#Y?w#
zuoU}wWm-YS15`f@M6Cw8wwnAiNdmP*9hV(zMpyl~&`W09W!H0QLmm)(#4(p1V)nz?
z%)|=dg>Ay~UzmXnnlJ;z?xW)5WeV29mza#P3ObmnQ$WlZ5ThfA1uP#sejt;rSY1L8
zT-CSKp+h|6xRG`obqUY~xAiGBJ&|LfBflXAl8vg3f22}nyB&Jym$g;fZEgHY8J^$Z
zm;<--;PV$fSr3JK^(%qkgs*5MGM=I>Y*Mch%YstzD&`e>s03^*`GB;FXOz{S2^|fV_V*1WczN+`<$Qe7i}W2LjpEa{yo0L6_8D3Lc@&
z-BpGC)M#yaA>y0Fa||nt>xofAIzL;SP0M&aKy6WqAOf{4gzA+{sEyC^OoDSXX!)4(
zJVjY9uHuC6Mvtj(^jUw1KuwMPK(OP?q5k07Q``U(F*ngAi=l{PLo5MaBaMae96|RB
zU@zhm8~Pd{p8nji8$$UBwtZ2!5z8K@6}$GoqwW=;MjOYuR?=v0oM(;sk2-|tDx7xt
z^8^WH=3=p0tJUvOJEAHI^OBMZe0v|Csksyj3c4*rhyX^<0WF|X4g%}l8DU7JwhRdh
z6K1Fv69TPl<4_2qoZKN^;*7sgB2_`xu^=dSFWV1V9%8HJC1(7I)UhE?IDm|vc!F-O
zD{|bmJ;4IoR$unxJIZz9FCJ9S=2HRmwe^xyN?POaGQ#9^2z)T5J$a;YwfC5aaL4-;
z;cpq9;3AwPo~VT=KwGC5&6Y+6um=j+Q0iC7+ZwFC0_s{^S-(1jheE||#iny97f+Wx
zLAwoZqAJel^dtO6s#00Fv5c2oLK!SY)KIoi?l!eh2<39OO=O9bXxl8X4?bovGoK$G
z;-ZHbw;%h&WVK}c^#JaLFezxhwnuP~g1DU*(`ltnV3tE9ha_^`hWPbJSfR>eI2yVWE;g)@cKF3)y|TRj0N$g-DgxR60DG3$4QM$0
zN~Wkab@AK~gDh6QqGu(k==+3GO=(RUM-MQHSDr+&u61vGB|CL;ZkVMmqh>5#!c*+x
z;713GgYa9=nPZj<+y#`kq7$I_O7OcE6FjyhW?0p{G_eYuTQGoD+gsg0ZxcukNn?t`
z=!DuJg+r?I8*f&xZltVUXL{#nU@E)OQ)w0IcDY{VR6sufELQz#H
z-WjeB6AS>?xJrTTV3Osxul9l$Q)xjcg%_Xc5y1tABsDOR1BL#geNwKX1XTeI8VaUq
z>DvHs4M8o8-!mstZ0c8*{v#_jSRLH7t4`DhQKGy6j06bRZ_*Zu9HzoLHA1Eibcoy~
zFY_*T;#3!?&oU;anM|fHa^WGb+#h*6$MY27s+2pW;t@yCR3`MW9N&HjtX_if9w7>l
zI`=cNP+5A#x$`K27_>)HGb+Ymkar&;j^z!p
z481{Sv(#kfaYcu?TLkZqekIfZVxPD>2NC9OH($A1Dlz*s;-zBNQc|7VD{G=`ReHE3
zSP?--a3NXl3sW_TzcR~1AUhx-AMpc0M+{pmLz&rl{$!ol8
zAH)I|vYYxknFR!E{{Ud4L9GnSN6Z?7qR;!{
zTi7cuV+*}x1bmeU6&J>(liEh(-(FO7hDvCAGR~9+TcXBQhQ@)R{{Rz2fN`9$@KkQj
zNX>}J?AA$;XtQFD|H&(P-7)n+S@9D_W4+}1qVIL-(PGM
zaE)*y@FTl$!q~JimJ|cc^8wu1U9%(*QPB!kSGFdG;0v02f=R-sJ+nc7aBe7Wv|#)|
zZZ(b>Xh72wkf=6z=$B+UMVMWM{{TAZh|rTa)DTibcU%<+D`O4A$sY_0WPnZIsi4v2
z1W=+qPz}NYJ!HZ?ZE*ZXqn&GP9Wp?`?Rcr8Ppqzvh#%o%@h)5re-fi7!wA@D(W^U&
zo)mM;%<8$95fBI^TnF5vGwxWL)rysn(UFfdFk*0k*P4dafyluRJpeLS54iAZeck?J
zBf+)i`jv)?fz8wyD$K7!QJ!K^$ASqFcuXgU5W_NUfUK{nMy;oEu2R@#S#=B;^i}wR
zIYdFjZ^nq0g5d5r456Z|{{T@*%m#IiW2gXAL%t%G!!HXn?4?*;Z}^vhv6J*ZARSb9
z>JwNSwlDofE=w{X+6FcfDCt2)1kev~UD~~pscIY0R>|bzI3%TT(1noX%k@7TpWwIimP;t*d{7oveZTGZGIuAoXo^H$IK;wv6+F}u{Bmj
z@~)_e?-_Ysh?Fw&WO-nKG0M7PE@dlu3ZwH8RdAVPqNz&dgrTbNaqd*O_?3;V7^zs3
zk0w4LMPR`aX!Ho0WN0+MxF;&kcz~5v(B@YK58@jgHNB8VRyKqvIP!57v^?P-?hZFX
z7(8@EYm(b9XDk|{?gi?nlbB?<4Ye(?%M}c|*sH3B%l89zx+SKr{`wuxcMEX(?h)6m
z>!Z;cx)$icA7u_k2F-weJi_-@N_c~(9^)2AkW4%+8&*C-oHc$3JBiVPJ*wO0BbLxN
z7y|ATue_tpJRg~I;?;E*JD*>11}ib9<9-h^iOusIV7$hLNa6$Fq22YVS?`&X7GXJu
z%mxRBWGQ&cz)`S6F!HZQY;i-CnXfPKMpcal%&JN`d-<0t+BX%j32T5q>J@`YX7WP8
zv2`-$&WUw1g)p?RV|zJfSOePXJ)|A6gd04{dJ+`w%7go5{$M--j}tvmEo%FW(fw`(
zEq8};z!>DVSY@Ez@Uq$z=nk%Kp}Z-da=>6mCvXu=yKA^TgaL^X2e6j1$iU)P05kB@
z1W{tLHoNS<#2I$YBPag=B~awjR(Ssa$!yhQ&l4+GK(h90FsfKGKTx(|V5YFSTChs1BCs`#WZ`OM5)*c{X7ddtpG#?2(GzUmVS
znhAK&0`h}sJW3DyxLON42mP6X)W8}K`(iilnzS|
znB~U`1~tyjtTrp!H0>b^3~gnFs{SECwqMl7x1$$$_>SArEG`rw3%iX)5{aV8oU>*|
zS1uI2OkilkK;OX>U<1vc}kl$l`M11deW7at29)UZ-0u44$fbyrMk
zW{(l3jNbDbI_cseRNIQ4V!vWFX6tdKGmzJH9jYZZe^7y_+lBjrg_kO#9H2J0H#hHM
z^#+762DD?m=pgGMkU^mLQ^)0H&
zlHCwWQEabqrKO{r)S!#ZUXEdbjsF0s!q_!zBCooxmobeDpd0YwGvY$MK;=CHwgwQj
zH~hs?v{xm@$X;{IxT^`6Y)rfXbD|xs4FK5yI@`lu5A`bx(APoo{{T?kTan-QD$&TK
zv9F!nRCc+&LxNFGZnZHfY-ZvNAa+CuY#t?Kuf;hbGJ!@HXS;!}Qq?(z6;^M$lnxTW
zznJ8~3^%{T7AW|aS`)$fVWSEFL(0urF;`DoqE?u+}g8m=|T#AKL`TL1%OqJ_f!feJvngoxSxd&qF}*
zTf3M^O$6fO%InD~SD`5OUTZaY%xG6ZQ+z_h1p2k&0!N5u)LxWJj-R#UN<$a&(
zU;+%WpP7^!GI;cH6BLQc>FPBZ7d%A1oyv?^_AnzYAq)l+3@C@iZxZg7yM^bWFR&h2
zbc(ySUmcD0D;;?H%wCGXDcUiuQ@w%uN5;WmBKsIcf$A_z10Fp}3^WTZg>Fl7qXOxK
zHo8m^`1Z+&yW9)}riJD($OoVw%nz5YSQqeg2KJGdV*+?_1mGVK)R#+!BN$aNLN1Xr
z0)o?($8S>0W@L&UqQc;p;0_2Hm9S&~0C|`{IidKJBbVbeG1CNeo
zs4xKHzUp2(;*Ss&aE5$9e>5Fo=y1wV+M}YJQ0fCV6r$Us#2eX%H{ZE^n(umntocQY
zu-P(Mx|lMC)!Jq(Np%TSEA8c%#!slmT#j^)+z`WUOQtQ(Vv|z6;ndEbNt-feHZErj
z_dK(r9x?Pw%IxO-I<9Ur(Nfq}ynIEu5i}GiVbR5ILVp-Yne@!ADFGu+0z$_J!TO!
z=2LV~K&`oHf;1P=E>r}*#rQKRh~~d9+*^Kjsv42;J5xKRG%J2A*UBB~+Fv#WGKJihY(058m_
z&n9$_z+jJaA2wNmrz0{Y%{;&bM!m+oZK0t7dqJq2AhN~86BU1sSUEgQum{9^H*AUx
zyb=wv%oc1-iXBIM3gs#ZWHpYZn%hIFl)RMgTa^a|ZImVhkl}{|EGHjK#F;eyK4VTI
z43SG%Dt3$~k#H>WQ?T^}!+L3V<|??nnR3;r%tqupB|=Sf%=Td>qM$(6CqWRsvW*R2
z`i_H5*iiZCjH#V9`Ee1k^q)Uh80MP@qA)hs8q{klS`+gCrdXw>0w_M1gb?!Nit^@O
zt(L(#86si_&oLziGl7Y-jN}}Z)*S9C!)a%$T@W7x3c;$<3LwO`B}@TLoAvP+Lq*2m
zimM8AVljtj(}=I>Oz1g^7XwQ3E9K@9LR*NYyOuqm4S-AJk9Af50KCgG+p&Z(qKkSQ}WaUS^hARs*$~RJ5-W6yY{@(>Boz9`R#ZKTsG-
z4j0e*fM42V?)Vs3$!aXq?hY0SE~Ynmh}x($MorLmnOdc!?mj0kZsGt>JQ$ik9QJE>-~rdy5;{Ur`(5c+^D%>}9(SNfQ@ichsb9vxlj2G-#Rp
z)}l~UH3Jtg+D(^bcq5d4FB$BYz%({Uy^@sM^5u#iQ(bx{6lWDPl@>1J+gs;QMfYXu
zbK-4TNV0wtlQ&V8R)XrL=`wz!5WH2Im{Ef+Vs?R1)JyINNHVcd`)r#4orJ(u3(@o%=ZHS(Aly{=8qk85
z+3A4grXBG75DtqS@QClAyE>M_8eAl9mw3T|=<)s}gv>B(R}o#_CCM+MAwNeZDMcO)
zM@Dm)n!vW$gXYk58eigiVxXBy5LY&J;tbNnu8UUs%)Azj6-pXBaZpmSlCOJ`<7mbm
z8YLPCst^0jvGKJX4S{Q~%zCmEc^90-!jM&=NA6^*8=6-~IA{{T}h
z;Pn7$#GQ8HH}48UMeD~
zVBRk?eU4}|{-9J`EPGx!m&9|xx>&w){l;$%I1X>k+_6O`UJVrjKbpUkDw8tm{{VSH
zmK0)TVF9XRd6S6sHfy7}AEucqI>@*rd2V-25i*+fEo)cP0K7OcF~^KnB_SkG^@wok
zRhs*PS<5NAn&oVruyweQ3vge*;$GPe)ktuSazBWCFgb?Pm8jH}D9yY9ScI^^s&CJj
z3#DRH%hDvtZ*U@4EYWIE$WXdB10|hLLg3aHp%_Vam9v&FxM`>mj_X!Jv@C6Pc=`B@
z-PN)82HToDGVmHbu*RhyMYbRcI=4^`?)g9JDtNaMwqm1jGhujw5Dg7L8ZG7gf9!UO
z!&->@n&>^SaJ=r%SEvq_(p&UJ=2;u?pUi9Q8T*-FnvzL1yob!S1bQkZ72ah6oaMN&
z(v!I2Z)ekq>F^VKJ4#qBssM0fYwI4My3nZH8lTlEQT}X1kd=J9(G`
z--)YxxFbcQ%eWv_Q2=Cusy-=%QJUjd=lX!5REp{X4787^
zJuX=?uQ`-(whSx+^GgoK-EB#&nMF&~LV_u4^nt7~ZEWrw(G}oVP$uf-P**YgATWO7
zD+_T+Xho=F!6-P_oy>8-m$sgg(Zhj)Ql11CC5I1+YD$@~H+5#`S^Bb<>q}%mxav8)Pa3mOjnPF9$Sv%US*!`6i&nmg*4!7!tO_FSOt1yQItAZoHXG&o<_q7
z04=TVEs(1*SLBpE%;+mNFj6AmDwoBYwGN?*jTkei_#FYvb_Z84a+?cYST?mpBU`%M
z1Pr@novgRR#2j6>_=d3Ht+2x4^*7M@hhqIo#c1j^h(I-NmDIJ0x&moxu;m`Oi6&*a
z6U)b_0}V?Hjn=53d|b!V@UYu_Jbc13!gdAZ<+{|N+c$+6!ZBqvcAL4!{fI!dQy@uA
zVE05|rf)aY_)TVHz&U956opGOXA$CA-V~>7n2{kN;;?u
zUaDa7V93Iuv+{-^mJ0^kzZ;ei5ie!}kPEN`t50tH$B7>5M)2=76vZ``9llNbguixL
z$n-oxLZUo|S&@4XAi{Zbn8<+q$E+TVN|iV%YVI{g;6n5+Gge#^R(s-EY;lv8GgPyi
zo`SHkSf=wX-P6RlDvuu0CID7B8qZzA#?(=!7+?ixwxAUxX&zu2nn*mfR(?;!5GYY|
z!CYA?HZhhi_vWDcHlmx=gQfta8Zy@_-F~H#`9=Y(9J4Q#ov*p3NG)Krd_frtko6r5
z7AhT*05~GLD2VHXafZXpW3WpafDBOqY@s_B2*Jls3KqW=ErtzH1+RBi<}TwQG~{9Q
zT3?txSAHR@T6TmNHMd3R48rXUsLFaZI089g3Qeh3F5%#DwF2c2vq;fG?*b-ZYH3UX
zQQ|TlD9pz)WeN|NCQgs$IF}V+j+7B^E@j=THEXt
z(Kx1jE|@OKHa
z#t}J9;>xp(%G|e+GR9nUC`>c|0DT^Nj#IF<>YTo~fe-+&Kvg&AnUBP19++RK`ht+l
z{-Cz`(026?2}Or5zM{Fh@lxp7@@8Q`%Zr*9S>_BIdEcKICJ6-
z+vk|KcKt=AW$sf<)?iiawm6lFP^8>I3y2sU*h2>IgUqXGz1+#xV}bK5LZA_lo77%N
zxJznY0svFUe2&jBIC>YC~Y{M;@Yd)&%(!!bxx
zequVox|wj1^dHH9a51oUR79eUG}jM_U=O>6guK;51IcK5y@MI6`1Zr6znI236hICO
zy2buw(K7Eu+p^#`x}i!B;6FTIF~o899#
zAR4MqiMn3X46x-qt8*6)V;3##XxQMcVm5iQ3zGr!4|d?7bLR69ny@xYcW_w;-FzFTh9cl$Na5iCcEkB5w}H
zRfD*(GePh>nqfEGx4$yY7ThgfX#QsP0hQdxnxHW%Ac`6S**ZR^R`;1+qTtv^qZB|Y
zQ`Ddh@D&2J1_How#3XD25XTT)CIH_z2+4c6^dHPx>N14|^2M0kBjdQ41EBkJE6^|;
ztk=xiKG%pi19aiLqmTWKl-F9_T2@Q=nWnrAT$FKG;JEqZ?D=cl&CTUKw@bX-TP&kC
z?6*NJbV3Ry@ZaJjX|9h&#(=0m;V<}$F={qAyh=~7t(tydi_Ea{g5C=+R~oz?VNTb!
zVxz93m7@G+ID_Wmyu{W=#BVgVmzG89xYEL{vx2sQi*Z&>;)9-{;0z2$AiRcm>M-K
z6&Zn}xC_PVVgN5k^9leNUh@zuFXb|<-;Iz3t35(W>g}#z8y0U4q3S@rs0Wqc-er;Y
z!Yt^@3(O9ZVDLJCi;0Wlm*sdMam6Y%yJ{A%mDsvZ<_C?Yfv#p8sJ#AVRN1V+@V!eK
z0#&~l+}g}v1F33&HH#uGqV&SzWr1OsNnV8sK*H%lTbFSCj@&5|RhBkVKIMKh%mB$<
zNGl%O2St8iw>A4lT=>VLED|ZknNqC|T7Cn}A6zpck
z$H5;!bfN~z`P9o(>OExmAazU5B6I#HiEEsTs2~caE>d3L^)jLUFka9rs3Ma6My`zW
z4$7|!Q(#AEj47A-i;y-733O@0UZq%-)jxoRRt;!>xupt^Pl-+dss6xS`hz03B}ET2v1)D0*bWbu8s0vmPUF2bO7u%@HOF=r5RZep?YXD~}>CsL$;
zi9vATcLydndiIK22S~&74yq5R!=gOQlt85}<$bTA16n7Ey#^Uyxxm|^TE4{#ekLAu
zKPsqOghlU?LQ}Pk?CuSP)BgY<8pJ#gf&%Z`2M(s}_fyDIN5vcmxF@!76$lj?0l&Cx
zp_GQ(ssIh^h61+t%oN&Sd18}^&wPAENTuAMd_!`P#3nGmT8b+|)CR^hLd$-m0F7DM
z8CwY8=&4Oo6>vO2Hj?Km@9}b#fMHWEC6vp|Uh2x&j$K2x9fW>Yqx+YZg?S!gb}8T+
z`GxR@NUDztEj>)!+WbH6qc|gW3)BL+rx8F;5na&a%(4q*lx8|fLRnPGwVdiD`Y^5Y
z3IeDDSKJQO>>g!7MBbn%d$W+)RWLwaI))zjRTM10I|Ixrir&ImR*8$eOo~OObj?E1
zxL9)mxI3RW;^!KqOh
zn!L+4D{AM6QJqUg(gY!`Td#?PNY!F!f!=0i3Uw}2UR*;-XM9H>gwkLoSQ=9zw*UxU
zi)l9r1VU9XGNIJ2;P5ylWdV<<1=e2Ti!R^-%$F=j%BJg>Xeb&iyx*B>)mrf??ER62
zm4POR4CSbncJSO#YlFfT#F}p;Mk*VoA2IOxQNvxp{{X?YhCZw^sP7OqIZVngna3}p
zBm%8&;W07EABCxSww7SRV;Y`dS!+_@?cxfR_mr64-rJUCj<}SK7xI=ZUROMk#Mu-Y
zOw7rXwLz*l3dhU2z%3fcxV>J85z)e*f>a83om{>nq<9E{1XMWqqFfV0q2s8E!0MTq
zYvvlHtAIkTSuJ-2W;l*jinpP}VV4IB80cBd#g!?o)G!JF2(c^Q5Ya6z)!>;mu|Q9910^YIZHF{8P&IMfszu<;=F}d>*7>FWq&3WgHgp*
zj}N|KlngG&3xfF|30>wiu#Z@ma73A)`;^;D0e{k7z_>WXTMRbSwD
zh`K+CZYkIimAt5bq}4#Z;$tfp7lJ=Tvu?X1NDUNmE80zXAWFIf2h8Yip5+uda$hpbC8maF3S7VzYhjp&OA6vUv4WX`N6wjqV#c|p
zjmHKVNcM;743F#ts!W+J>0taBtOU_m;5m8&p5Za3gjKyxT$WXP
zi73@a_L&jxQyd})Y8b|7{J>nM9K{eO-Hhf_RJ&N*7h%m3s_N_UCcTbuG*@L~APO5V
zf{Hy)I|gmnuzukR28+x?^*|eeu}gu!u~S!kd`dtHIEA4o
zIXuPhm`oM})oLuN)`O_F^k0@)7Y9rhRI`7~QXJ#Cfa5qjmLTBA0HfA15N>R96j7`;
zIO+?A0?;@`n9n@HtLEEnTC~PiMYfAS#J2Z@4Gvfs<;s^VFSv?;fq0cuekB^ItBqWo
zi-?&@ASqgALKrKlk5Lztwg3x_uuDDCm$&oRGS>dn3l-dGwHW>ps3almgas82rUKRA
zhn{dLW~wuuBGNN$ZYEMXHbikHbGasgM3T{VyInAV)}%(&P(P7
zlxVrB(3);^Jj{wZ<7)E2;73Bi9+^rASO7ujCr)8ziJvA68o5{)4l*0m4%bW+I75VS
zImsrLX?u?461i9D)J^Zy331BZ5NW-~-^lS616d^oE3PS5;9?V1MBfCo;d5laC3OHU
zGB!OcBfQ3h0E6m0;sGQojv^INSMFcy5~>Q=7hmZy7)MsR?kL(9L8-D-cQlRojfhte
z_s!~ZQAT150K;(T_>L{(Xn7CnG-tUK-*mxLwG(E2$iBGX~XN2#TG#JDFBm
zaV+<()S*<~4EH;*?1AL$c!xNm85Ji)0HUTalF4wQq$0Gh5yC6EV}ps7BAcNi-3mt;
zEO@
zT{7k}K*uw@=wH-hy3$W4zkJKyDW;6Joni&<5TC97;Wll7RO($`0P(~z^n$9$a3HAm
zmm89yH9k0$;2pFfRRj=bP({7K^g(HD#rFgejROqmBNe7d2{Oz?EkV$zSADsQR@huz
z24)T|j)`~HjFQ9!o%X?;`tsXPs6#;Sq1yE;l$&_~Tu>;2GdLY8vE4L7H+Xv`QS(XI
zrwrs}SAiJYZs7jg56k_+N_+%;L3X%I-iMYK52b6g0{AC+#>B#alw8f
z<=9{Y0^Z&{Jdnaz_9GSJDCH7XmA-3CZGeS}yLJ1E<82#hQ+4X)QHNSFpD-X2n|~jC
zLY$_CfH~-j3l-p>Z}A%j9jz8L4{`g90}jY_=3O}O^XB;>Mo^b+a!TRYHNC;4v3Qg`
z)JIqp;uQ)j2G|0Deo~pV=Tmijv0NJ53RrNA$g#r{dnpVLQFrygn~^xi7$HiSwHErB
zl7o47HDsAuA2CWsD8Q17IeBeSW~&ZJf?5Y9%A5L$?MgDMTk!-bwNS^*VzSxfgG>tvC_jWMYPyD?6h~D^vxX_F*>LPU
z9O?=*TYInS25E|2`Q2P%jViOugrI%R&M4=s_rm=E$5hpDy+a2IWDRV7C0ibmRU6k
zZ&pp~>#0tGg4j3mgn}UFb$mS}LKv@2N>+Y3a|Koe{{XiIKyzzIkcj0&gUoua%3vAr
zoaxZUAQ1~@Z{`ajTOoGy)B>Jh7o;J)E{5O^SLP!|6rzs|V5>}^v_qxAA3-ci*CeoK
z$yOSJ+Y)SiKn&EeyO(g=)B=n*MA$*@EdlB|3XDQx*j?j|;(0QZ^EOSEcJUqAn%dh>285evX>H}m-Qn2T8roiM_8|J
z(#1|MW8z|~&UjV3{7ha&7tsD=OUZg%V~DF(11L<_sRiF>{{Y3rb#NmtL}fVh0IR>^
zQc~tcm*O~^l%a%J%(jM@wNaFi)l5}G;w)?(#0QyJ*hm=Z3IL0EJYrE7l`eHq7+>Z5
z#}wGqvQt`Mc(CKlvfX0U(476n#bL~}G6ucDWE+E^@%xpER|{xe`kFMvRfg5-E~Fa9
zv-*gmi+2Fo?hq7T6*eNc8}yevKp%(HtOSve3gFN80jGm;=0yHw>=XmV%rfT?Zli)*
zSj~40+)6M@b%?YS0BvHci~~Oslz<`^+Qc|Tfh*;#O~Ycw&M=!o%DcN^`GHYMRg_pH
zwDYV+L>#E(?a}uuSY4oj(i+EZ;(XYYt$7Sf9&&z1Z%_!|&@_Km_Z)n)N2sPFr0cb)
zLR76FnzH%KDuG_EJ0@J~KA>8q87z<2RIsuK^KJ>Gokf>TbsrUd%LyX({RxLB#2HW{
zJg9>3`I)9s6S6TAR}2#mE6-iZ?%-@MJVH)Lz`E&i0-cgV50tq2i%RD^1K62}iWaa+
z!zVN>SDUEI2%>9mF{@uLKBi)kE2O|i@eqv7R9M~UTB_nT>qiHuy2`9BUv45dG#C};
zIp;8PBWNxiqw57>&>4pm92jrg%p1dHpM$mpB52#{N=Zdp;EfcG!DPUB4Z&rUFkkLD
zR4ZIMg^X~IiMz}V`TLSp5k>Qugje6&M-%C6F~7W2%|R=n=;xmkt*pq=$X$xp&6E?
zbl_fdP^c=&CQWW%lqz$+Sy_m&UK=^WBd=4C)|TQe+SKltWiFJU6r54SDG$ktkGH;!
zHvvvX6JhEtuNs5i=FrHEhTw2KO42r$kfDIa)Ro%_AYzRrxo##)yBd&fjk7B{j1KXC
zP^6sPG|enQiujslxU_tCD4U@>bUXQnfr^CkF{_uBhHuJyhO3aOJ$U~BQkf*A{{U33
zfS^U{p;UT>4V8>c25tZumKp;MVTu*K-!kS)4&v4zO1J&)UIF`J;D*fVUfmL_8EA(H
zc$hIDj4mq}90PAP7LO>ng2qd{VzGlnM*#{`Ny;n-Yb$}PfHPR2802H8h8`*R8O|uX
z`GX*`maOCESp(f=&Gj+urK%?+EnZjy1g2yF6gxSVu$k&D=-hDd`HMP^{-#UCS(n$S
z0a|+X1tlZ~e4|yXYny@Kz$*6z2kjaY>M=sBxu~oP;2I8yMB4;@rV_<$@DFPdh$|MK
z5v-MY&B0L3>(LO=SAZv~czK(Y%ads@U?yMG2&Kd*sk&zNzRutxlWQa41skfA$^mIX
zsOgK;z+m>2=b9^4@s4KmfFTX$>LKOEpP0*Rn5D0nm^i#d)ppJvCUtN&%t+k`4qKEL
zf-9FKq!7x{h*GvnoyDavCm72QG2f>@=uBr={EV)Q`ZRVyi7n@Y8e0z|8%f{+PG3G{
z**pTayS~lG>|2XdW82gs#)iwLG(j}P3OBZ0KuqbmbjoDL$b@eZ4x&{>0ervt72Oa-
ztINmwnea4y@}6$u3S^Mek@}VfOPPG)aLlR9wIvQ_!25*hGf>5VfUh#rfb^7+CjfVt
zN`;0|;TwZ6E?!F>DKibHg$odJ51-~VVC-IB9%ACooS$|70BU34TTYK2BO<79w&5Dc
zGv?wH!qfp3SF>%(<3jVx+$1ey5fmnx?KYZ}okL5NF+TpKvDDx|^o@f^jHoNg1fbk8
zO?Mr;TZot*XlMa&1}0%ZN~dvM3c1wYC7{$c?0P1YQypq|I)o|yM5hePc36=>qbrtD
z#sQ@B`EOgMP)L9S
zqbxMsq!hx~O{lBo)OaiWu(h${Jw@q!m@k@@pyp8KrG#y*^%W;I<_H4Yn41!~Jh7^%
z1~#(S<559uvtCU=rnMP3vRyP1n(yXTnije;ivkE`lf(dpyG02OdytV43~cjjQW3;zJWY(%x5
zG{7Ta*GxG%3eKTb67<(H@_J$@Ie=`eV{s9EmM+7pjE5F!hZ(6+tIRthX5*Bg&(d0r
zG4KfT+_uglfD9v>hFQtPO$H?m2rBFJ#LEYkP?Qd(2Knkz@o#dYkVuX(6103w8jZ^c
z)UdCBzflVX?UZ7cE%6vj`nu~<-mOL`?)M#A+i~5R@%c_Hct+|dIxP!T3)`nzNi@TUWXPcXW
zbsG*jmimuE$H>*~m7pA}9K7wUO2KBEimmyg0GLf%h+8~7_Y656Vkt!z+(C6W;S)?R
z%NJb43@~Ms)t{+Z1Z9SCjgpWW#UxzRQ%B*CSI<))06ryp;Y(9=ntfsu`9-k5$L0$C
zF~6uYtoV-tHYhw%seW=0a+OC-Jo|%uuCu-a-R9%r{nb_Bu44rWWdO5x;$`zcZ<$td
zcf@MJJ!@KsyDsF;y=Zxc0?8PF=PsJ4flMr)V~AkWYOcpwTMLZl%Po}kE!SKHQB7fi
z$he!br^_TYmg-hwDka9)0u5AERJq#;EyFg(-ECY5wfg#wjmCxQRC1%{ZcJwGxR(eZ
zx%Df%Clv`=n!!-5!BX$I&JBm+ZMp{@{{W~2RI07Qn)x>Pn*RX3K$qzvN&xCshs0p5
zV3oS@6@p=(ea=9~Q*K;Eg_t!0`9*D|g$W**?xYyRilhOQ9;2MxO$o+jEU@j8Pzrz+
z{1JC<npWn4
zV7dm`ctxZZ=y;VDgAPnw-7$|NXiMETpt--AZJCr+yNFxKFyJk?q9F;UKyX}{#bBmlM;$fJryi7p|YaGSzM(dIa(vOs&R;4dd
zET}xOmOkA;eSUj@tZ*h-c1w949!X>#Or1n<+!;dDTBaqmdkC#&R%o)KsD**d%NoBj
z{QhvyH|>}46%I>|%Hc6xr5s>8n_NIKSGN+7miI1npUh@&^oN+N*vt4jg}L4Bd5?v3
zGYL_lI`W{!Y$^)5l6ihu2V>xcnhLp8)etLG?^6=N-x9f8ItXDjHHl``3ezv)w6-5U0nZB`cg5Ym|h@iB=
z3w6R9v{xo9%uKpM)B`?0S?w?6uwX6w^Ias@ukvM-5krm>Ap^
z=}6!EmMyin;u-^05-%)Z^aNjUtBz$)mNf`$HT*_q<==qcr^Iz#Ef@mV6%J?^KSEdT
zf#;~qAzoqxE5t`wBHm*8MQNEpi7HdXS8|rg9I22T7>m`>9Xv)W4ZKVjwF#yF0K10M
zrc58(FT@lfN>6y~@JqlvC!dJT3fio&YEqQI4^(HlVWN#SFnE~Xf*U_XTZcYi`CGV?
zRb|H&C^QTP4uLq9jYJG40r@l26l(9xxn$&Z5GuBJ!z}>IQ;Vm#@&<=6t$_J}f^cs!
zRdBU&jzj3q(+~OO0igv>9h@K?}amduW4MtrR=57zDB8V!)!`ty3
zFEMDuAr!I*plC}ksAUR^278rpa5*E%2&>>wt(Dq*#SjIyhmCfaXKtdasc=Mr(ci);y9`p|ucxRr;1pKRs2&nDw3T44Eap@&
zIO-krgz9Q3i>Q`Qscz_rcbD!u&*Cpg%NC-c_E=&XXQLSpGX=6@aZrc1rv-av46F>A
zb*QMf%reT4*(?Ks`;Eeyc;*j77eu%X5eu%PM3KSFEUQf&u_n00_0Zcw`Xi1h6pJi>
zP&dVX5}Ti7QtZU&_2NI2-pxS8HlbK>$16-^f!i1o$H7vo1S^V#JjBGMI-1re4j7d|
zHXN`+{061R8EVS;OmgWgJsidee5@a>J_$|g-8uOFQ4{UADvQvA2N4(MDRg3MYW40g
z!GMTw%nB+kIS*(fHv!}ya3EPyt=zyD$^Pp292#b6?K#bBeo?yv1RzUnH1ny#_lO*@#0NP%!r(wsq6LZae9nd@g=*cCQ&rpMHwI4(T3#Dja^}S)
zXlNIjy-F5tqZa@M@l0p|7NK8Ab%JM@shgLoqAas`)EUKQCy8|&DT{Q(t{}Mq%}ix#
zr{rK18go+#e-W)qD6go1G*idwB17fkWTQv;H4RE@1;rzy5s6wcbqu4i&BbWFCX&9`
zBwJ_Z0YE)AdU0zyATj!|t5MQhs
zD;bG~G5MB`c17hciABS{p;uv}y-nl|N>3k6gnRt0co2*ts1(9?&AIGUkOqe^*7ZK;re
zvLUPvUx;Bw;2-rtpf!(}!qeArO4}CkEOm+gqLG6Wqq9bLu!_2fY_@LBXzzMkD$~v797&SAi5!(*fskM=G=eP@F-t)MNmf@EJ
z358#so=9n}0Y>kcOr9K)od|ElLILUKhJyo#b9N48x;c+txQ*P_u4N0yEIXL|kc0k0
zo?@&v&Diibi@Iw8c=r*(U=I6cjK$B~?Pi1K!a)DJvA|y@aiC7OEld5<&fDcpT
z?p0#PXEEM%?>Q#-Z$T?55l{R>uzB%T6y`JHcrj$;Az)THOYKj
z8W(Z1CU`qjN9r2H3eP4WbN(C&Tl$C#QNu*I#Ar1hfnAu;h-B_~AlseAcGrj=Ac2CW
z*{n;zTsw_n(xpm~R4*fNGT|9*rX%5lo0*7Bd6pdCse>HzGo$JRwTwN(vk_0}l&b4<
zlQwYKL4$Y*%hd%c;H8D%WJk!V;G+vyb}lct`9*Nyg@+K0J~72bH8_-9FG-5FU8cWs
zwL0hMHSQEr%P?2I{6@AKDNf#?6>o6$3I=%0s9X;yzCuLMb8s-#%BJI0ae0ZnVo|o`
z>X>6w4ae?94Mb5;Me#Cn)I#wr*@%3LN~luowL6N}QO2VsggW|MZ+;=`r7VTzPBCYO
zQ-UQ+U6geKf^8kTc)3zzAUP$^uvXg<4e(J^Jj{~04k3bXV`ut^2GmRNzM@*#^BgMd
zWIUK?t$<`8o1!heIK-eLp5k?g_{8Ea;>36HE=?cYz^kfUEb?;&2%T`Aqcp(W92uq6
ze$7P5<*Sux;YzRObzp`^<`8w`F-M!kNDa4nPNgzXeI48|jevd;?9|DF_=m%KiC_Cm
zcQwlL#8SASg2Qqm-kQWwtvyB_-x9Gc%*x?kT8R`GSxv;yZdq0~#X!hWL|r|GAj)8C
zhn(|Ulw2dihOP-=`G=WA&ru$){2Wv+QGYe;%bb+R|U%$iE2d^TR9>mFjQro(~e^2pe2-!312Ng
zaTYA{(*XG3NBNF}Vamw_0G<;BeSmL<7R)M=cB8{D&XmR<4z0w)Y!i3Olh!3P%mZuS
zZ{DNn48Y!nKh#x?uX|x!t%3gloV_Wx&CW8(xJW9*#Ie!}m%BKK2C_vepqtH0wqmWX
z977&*tXys`2~#Gym?h>?zqnNEm{>$9i9(yG05uWlWwIbTF0(A4FG}FaW|YiPAzJ0}
z1Sl5EU}kUzi$5uT6uqCA!Xq$`TZoPF+#WcGz?oW&K#|w%!y@+77D3hrm=JLbG(qe1
zjtDxV5nir2xaK8hJzRP*55{7oz>STY?o_O*9MNBlLg+v@SHFqx8EU7HH!%D-mTDWe
z8G>yi1xIp{ZSGOy^<~0jcYs_IOnHwAI+e?V9aq}tW&LIgnryv6;pSFn+;*>FEceW?
zBi-zR3%)p6trVNwceJ5kIX&l9+^VI?d+2~*6$$Y!s)#cxGb_*dSZHr*K#CwHCsyZa
zM&JcGh7}pWOcg^YIn+v+aSte*qjrVx~*W(6Di`sev!jw
z69pajFlP`umxzqho)YXBBD4;!;wBnR!`>NgJ|evL9C?^U+!3B&+rD95V3FaK8O%F7
zKp)h)0x7zUO`)L6Xh1fBUf>xGbu6wT1TYO^U&N|eQEoFVgnlKim~na%h|a)v1eBhg
zDsq3(sa13T0EmLw&SsT(iE7ajvQh!WM~i~2OM^$2FB0E!(cW%bsl7}xdV_zd+o;Yz
ziLNCSIbm_D^#St00n{uLxsohSqkxQaMZgV|m>T~817?;kYSgbvCJnOP&&C)idxX1U
zVpM!b4EF}3bIdLY({F2*Hlab4jDW9;D@>hNB;;-^CqswaN-^vjfNvO>?2T5*?fIA9
zUkPVw!4NsrF|&xlbt`TQSM-4)6^14ws5`Q}LSLDQr_4Awfma%OnfZp9%rk$*8KNhB%x$pKe9m~p+jf~2r
z?*S`jr=mANd`fOg?sOAx^#sNT58uS23}Gyv8~+
zsQPvuUCy!8ng=}G5IhtyXi?N`rB?5ll_U(XGlmp_Zst7>gccWn#H~Xqr5wl3SvO94yg^A@8
z!cyx@E4{{#@?_=>$0=&Ap_NQGGG-?ZBD~xNxr5cjRYI@u$-GRJxQj*3BJaddi_0!^
z7S-HX_>08K9mlAvQG*F)i>@V$xBM#}WjBeNB88~gy_p$&j#6Q;vtuwgjcYL~?|AgZ
zAyQr)zy-^6v3sk$KM<+0ce#KyShW%4>&_xzh1RnLnPuEjfz->WRV=g>BRPaz;}Jo`
zw&lyrA*&UXUZ58Q+6*v|P7uhb8q8~S;u6X(CXCFoiLf+4U)21`v;G(&YA0}2-}o)E
zcNg^sA8?DfFpH^rj-$+VI6;s29N32AvN~a??Th>$_~AH3{maC67;!K64YIJqVr)Sq
z5}}MHh=m{HjuVvOInGJ5?r=kK!TvkK0_G9h9nR)*02S2X2n9|d76Axx2QCSwD5RTU
zHA2*C7f>}5PDuKYaSDRB3r5_*@F-?Gh*rtF1OVb|Gk~$C

literal 0
HcmV?d00001

diff --git a/doc/src/How2ReadData/src/Hudson_Bay.py~ b/doc/src/How2ReadData/src/Hudson_Bay.py~
new file mode 100644
index 000000000..acd44518e
--- /dev/null
+++ b/doc/src/How2ReadData/src/Hudson_Bay.py~
@@ -0,0 +1,43 @@
+import numpy as np
+import matplotlib.pyplot as plt
+
+def solver(m, H0, L0, dt, a, b, c, d, t0):
+    """Solve the difference equations for H and L over m years
+    with time step dt (measured in years."""
+
+    num_intervals = int(m/float(dt))
+    t = np.linspace(t0, t0 + m, num_intervals+1)
+    H = np.zeros(t.size)
+    L = np.zeros(t.size)
+
+    print 'Init:', H0, L0, dt
+    H[0] = H0
+    L[0] = L0
+
+    for n in range(0, len(t)-1):
+        H[n+1] = H[n] + a*dt*H[n] - b*dt*H[n]*L[n]
+        L[n+1] = L[n] + d*dt*H[n]*L[n] - c*dt*L[n]
+    return H, L, t
+
+# Load in data file
+data = np.loadtxt('Hudson_Bay.csv', delimiter=',', skiprows=1)
+# Make arrays containing x-axis and hares and lynx populations
+t_e = data[:,0]
+H_e = data[:,1]
+L_e = data[:,2]
+
+# Simulate using the model
+H, L, t = solver(m=20, H0=34.91, L0=3.857, dt=0.1,
+                 a=0.4807, b=0.02482, c=0.9272, d=0.02756,
+                 t0=1900)
+
+# Visualize simulations and data
+plt.plot(t_e, H_e, 'b-+', t_e, L_e, 'r-o', t, H, 'm--', t, L, 'k--')
+plt.xlabel('Year')
+plt.ylabel('Numbers of hares and lynx')
+plt.axis([1900, 1920, 0, 140])
+plt.title(r'Population of hares and lynx 1900-1920 (x1000)')
+plt.legend(('H_e', 'L_e', 'H', 'L'), loc='upper left')
+plt.savefig('Hudson_Bay_sim.pdf')
+plt.savefig('Hudson_Bay_sim.png')
+plt.show()
diff --git a/doc/src/How2ReadData/src/plot_Hudson.py~ b/doc/src/How2ReadData/src/plot_Hudson.py~
new file mode 100644
index 000000000..3b57c3277
--- /dev/null
+++ b/doc/src/How2ReadData/src/plot_Hudson.py~
@@ -0,0 +1,19 @@
+import numpy as np
+from  matplotlib import pyplot as plt
+
+# Load in data file
+data = np.loadtxt('src/Hudson_Bay.dat', delimiter=',', skiprows=1)
+# Make arrays containing x-axis and hares and lynx populations
+year = data[:,0]
+hares = data[:,1]
+lynx = data[:,2]
+
+plt.plot(year, hares ,'b-+', year, lynx, 'r-o')
+plt.axis([1900,1920,0, 100.0])
+plt.xlabel(r'Year')
+plt.ylabel(r'Numbers of hares and lynx ')
+plt.legend(('Hares','Lynx'), loc='upper right')
+plt.title(r'Population of hares and lynx from 1900-1920 (x1000)}')
+plt.savefig('Hudson_Bay_data.pdf')
+plt.savefig('Hudson_Bay_data.png')
+plt.show()
diff --git a/doc/src/Intro2Course/#Intro2Course.do.txt# b/doc/src/Intro2Course/#Intro2Course.do.txt#
new file mode 100644
index 000000000..e31513ed2
--- /dev/null
+++ b/doc/src/Intro2Course/#Intro2Course.do.txt#
@@ -0,0 +1,167 @@
+TITLE: Applied Data Analysis and Machine Learning: Introduction to the course, Logistics and Practicalities
+AUTHOR: Morten Hjorth-Jensen  {copyright, 1999-present|CC BY-NC} at Department of Physics, University of Oslo & Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University
+DATE: today
+
+
+
+!split
+===== Overview of first week =====
+
+!bblock
+  * Thursday: First lecture: Presentation of the course, aims and content
+  * Thursday: Second Lecture: Start with simple linear regression and repetition of linear algebra
+  * Friday: Linear regression   
+  * Computer lab: Tuesday. First time: Tuesday August 27. 
+!eblock
+
+
+!split
+===== Lectures and ComputerLab =====
+
+!bblock
+  * Lectures: Thursday (12.15pm-2pm) and Friday (12.15pm-2pm).
+  * Weekly reading assignments needed to solve projects and exercises.
+  * Weekly exercises when not working on projects. You can hand in exercises if you want.
+  * First hour of each lab session may be used to discuss technicalities, address questions etc linked with projects and exercises.
+  * Detailed lecture notes, exercises, all programs presented, projects etc can be found at the homepage of the course.
+  * Computerlab: Tuesday (8am-6pm), VB IT-auditorium 3
+  * Weekly plans and all other information are on the official webpage.
+  * No final exam, three projects that are graded and have to be approved. 
+!eblock
+
+
+
+
+!split
+===== Course Format =====
+
+!bblock
+  * Three compulsory projects. Electronic reports only using "devilry":"https://devilry.ifi.uio.no/" to hand in projects and "Git":"https://github.com/" for repository and all your material.
+  * Evaluation and grading: The three projects are graded and each counts 1/3 of the final mark. No final written or oral exam.
+    o For the last project Each group/participant submits a proposal or works with suggested (by us) proposals for the project.
+    o If possible, we would like to organize the last project as a workshop where each group makes a poster and presents this to all other participants of the course
+    o Poster session where all participants can study and discuss the other proposals.
+    o Based on feedback etc, each group finalizes the report and submits for grading. 
+  * Python is the default programming language, but feel free to use C/C++ and/or Fortran or other programmin languages. All source codes discussed during the lectures can be found at the webpage and "github address":"https://github.com/CompPhysics/MachineLearning/tree/master/doc/Programs" of the course. 
+!eblock
+
+
+
+
+!split
+===== Teachers and ComputerLab =====
+
+!bblock
+
+_Teachers :_
+
+o "Hanna Svennevik":"https://www.researchgate.net/profile/Hanna_Svennevik"
+o "Morten Hjorth-Jensen":"http://mhjgit.github.io/info/doc/web/"
+o "Lucas Charpentier":"https://no.linkedin.com/in/lucas-charpentier-176206171"
+o "Stian Bilek":"https://www.researchgate.net/profile/Stian_Bilek"
+
+
+
+|------------------------------------------------------|
+|      day         |        Time |
+|----------------------------------------------------|
+| Group 1: Tuesday | 8am-10am |
+| Group 2: Tuesday | 10am-12pm |  
+| Group 3: Tuesday | 12pm-2pm   | 
+| Group 4: Tuesday | 2pm-4pm   | 
+|-------------------------------------------------|
+
+!eblock
+
+!split
+===== Deadlines for projects (tentative) =====
+
+!bblock
+
+o Project 1: September 30 (graded with feedback)
+o Project 2: November 4 (graded with feedback)
+o Project 3: December 2  (graded with feedback)
+
+Projects are handed in using devilry.ifi.uio.no. We use Github as repository for codes, benchmark calculations etc. Comments and feedback on projects only via devilry. 
+
+!eblock
+
+8am-10am |
+| Group 2: Tuesday | 10am-12pm |  
+| Group 3: Tuesday | 12pm-2pm   | 
+| Group 4: Tuesday | 2pm-4pm   | 
+|-------------------------------------------------|
+
+!eblock
+
+!split
+===== Deadlines for projects (end of day) =====
+
+!bblock
+
+o Project 1: October 1 (graded with feedback)
+o Project 2: November 5 (graded with feedback)
+o Project 3: November 30, tentative (graded with feedback)
+
+Projects are handed in using devilry.ifi.uio.no. We use Github as repository for codes, benchmark calculations etc. Comments and feedback on projects only via devilry. 
+
+!eblock
+
+
+
+!split
+===== Learning outcomes =====
+
+!bblock
+The course introduces a variety of central algorithms and methods essential for studies of data analysis and machine learning. The course is project based and through the various projects, normally three, you will be exposed to fundamental research problems in these fields, with the aim to reproduce state of the art scientific results. You will learn to develop and structure large codes for studying these systems, get acquainted with computing facilities and learn to handle large scientific projects. A good scientific and ethical conduct is emphasized throughout the course. More specifically, after this course you will
+
+* Learn about basic data analysis, Bayesian statistics, Monte Carlo methods, data optimization and machine learning;
+* Be capable of extending the acquired knowledge to other systems and cases;
+* Have an understanding of central algorithms used in data analysis and machine learning;
+* Gain knowledge of central aspects of Monte Carlo methods, Markov chains, Gibbs samplers and their possible applications, from numerical integration to simulation of stock markets;
+* Understand linear methods for regression and classification;
+* Learn about neural network, genetic algorithms and Boltzmann machines;
+* Work on numerical projects to illustrate the theory. The projects play a central role and you are expected to know modern programming languages like Python or C++.
+
+!eblock
+
+!split
+===== Topics covered in this course: Statistical analysis and optimization of data =====
+
+!bblock 
+The following topics will be covered
+
+* Basic concepts, expectation values, variance, covariance, correlation functions and errors;
+* Simpler models, binomial distribution, the Poisson distribution, simple and multivariate normal distributions;
+* Central elements of Bayesian statistics and modeling;
+* Central elements from linear algebra
+* Cubic splines and gradient methods for data optimization
+* Monte Carlo methods, Markov chains, Metropolis-Hastings algorithm, ergodicity;
+* Linear methods for regression and classification;
+* Estimation of errors using blocking, bootstrapping and jackknife methods;
+!eblock
+
+
+!split
+===== Topics covered in this course: Machine Learning =====
+
+!bblock 
+* Linear and non-linear regression 
+* Gaussian and Dirichlet processes;
+* Boltzmann machines;
+* Neural networks;
+* Decisions trees and nearest neighbor algorithms
+* Support vector machines
+
+!eblock
+
+!split
+===== Extremely useful tools, strongly recommended =====
+
+!bblock and discussed at the lab sessions
+  * GIT for version control (see webpage)
+  * ipython/jupyter notebook
+  * Devilry for handing in projects, next week
+  * Anaconda and other Python environments
+!eblock
+
diff --git a/doc/src/Intro2Course/Intro2Course.do.txt~ b/doc/src/Intro2Course/Intro2Course.do.txt~
new file mode 100644
index 000000000..0e01dfeab
--- /dev/null
+++ b/doc/src/Intro2Course/Intro2Course.do.txt~
@@ -0,0 +1,171 @@
+TITLE: Applied Data Analysis and Machine Learning: Introduction to the course, Logistics and Practicalities
+AUTHOR: Morten Hjorth-Jensen  {copyright, 1999-present|CC BY-NC} at Department of Physics, University of Oslo & Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University
+DATE: today
+
+
+
+!split
+===== Overview of first week =====
+
+!bblock
+  * Thursday: First lecture: Presentation of the course, aims and content
+  * Thursday: Second Lecture: Start with simple linear regression and repetition of linear algebra
+  * Friday: Linear regression   
+  * Computer lab: Tuesday. First time: Tuesday August 27. 
+!eblock
+
+
+!split
+===== Lectures and ComputerLab =====
+
+!bblock
+  * Lectures: Thursday (12.15pm-2pm) and Friday (12.15pm-2pm).
+  * Weekly reading assignments needed to solve projects and exercises.
+  * Weekly exercises when not working on projects. You can hand in exercises if you want.
+  * First hour of each lab session may be used to discuss technicalities, address questions etc linked with projects and exercises.
+  * Detailed lecture notes, exercises, all programs presented, projects etc can be found at the homepage of the course.
+  * Computerlab: Tuesday (8am-4pm), VB IT-auditorium 3. Depending on how many enlist we may extend the lab sessions
+  * Weekly plans and all other information are on the official webpage.
+  * No final exam, three projects that are graded and have to be approved. 
+!eblock
+
+
+
+
+!split
+===== Course Format =====
+
+!bblock
+  * Three compulsory projects. Electronic reports only using "devilry":"https://devilry.ifi.uio.no/" to hand in projects and "Git":"https://github.com/" for repository and all your material.
+  * Evaluation and grading: The three projects are graded and each counts 1/3 of the final mark. No final written or oral exam.
+    o For the last project Each group/participant submits a proposal or works with suggested (by us) proposals for the project.
+    o If possible, we would like to organize the last project as a workshop where each group makes a poster and presents this to all other participants of the course
+    o Poster session where all participants can study and discuss the other proposals.
+    o Based on feedback etc, each group finalizes the report and submits for grading. 
+  * Python is the default programming language, but feel free to use C/C++ and/or Fortran or other programmin languages. All source codes discussed during the lectures can be found at the webpage and "github address":"https://github.com/CompPhysics/MachineLearning/tree/master/doc/Programs" of the course. 
+!eblock
+
+
+
+
+!split
+===== Teachers and ComputerLab =====
+
+!bblock
+
+_Teachers :_
+
+o "Hanna Svennevik":"https://www.researchgate.net/profile/Hanna_Svennevik"
+o "Morten Hjorth-Jensen":"http://mhjgit.github.io/info/doc/web/"
+o "Lucas Charpentier":"https://no.linkedin.com/in/lucas-charpentier-176206171"
+o "Stian Bilek":"https://www.researchgate.net/profile/Stian_Bilek"
+
+
+
+|------------------------------------------------------|
+|      day         |        Time |
+|----------------------------------------------------|
+| Group 1: Tuesday | 8am-10am |
+| Group 2: Tuesday | 10am-12pm |  
+| Group 3: Tuesday | 12pm-2pm   | 
+| Group 4: Tuesday | 2pm-4pm   | 
+|-------------------------------------------------|
+
+!eblock
+
+!split
+===== Deadlines for projects (tentative) =====
+
+!bblock
+
+o Project 1: September 30 (graded with feedback)
+o Project 2: November 4 (graded with feedback)
+o Project 3: December 2  (graded with feedback)
+
+Projects are handed in using devilry.ifi.uio.no. We use Github as repository for codes, benchmark calculations etc. Comments and feedback on projects only via devilry. 
+
+!eblock
+
+
+
+!split
+===== Learning outcomes =====
+
+!bblock
+
+* Learn about basic data analysis, statistical analysis, Bayesian statistics, Monte Carlo sampling, data optimization and machine learning
+* Be capable of extending the acquired knowledge to other systems and cases
+* Have an understanding of central algorithms used in data analysis and machine learning
+* Gain knowledge of central aspects of Monte Carlo methods, Markov chains, Gibbs samplers and their possible applications
+* Understand linear methods for regression and classification, from ordinary least squares, via Lasso and Ridge to Logistic regression
+* Learn about various neural networks and deep  learning methods for supervised and unsupervised learning
+* Learn about about decision trees and random forests
+* Learn about support vector machines and kernel transformations
+* Reduction of data sets, from PCA to clustering, supervised and unsupervided methods
+* Work on numerical projects to illustrate the theory. The projects play a central role and you are expected to know modern programming languages like Python or C++
+
+!eblock
+
+!split
+===== Topics covered in this course: Statistical analysis and optimization of data =====
+
+!bblock 
+* Basic concepts, expectation values, variance, covariance, correlation functions and errors
+* Simpler models, binomial distribution, the Poisson distribution, simple and multivariate normal distributions
+* Central elements of Bayesian statistics and modeling
+* Gradient methods for data optimization
+* Monte Carlo methods, Markov chains, Metropolis-Hastings algorithm
+* Linear methods for regression and classification
+* Estimation of errors using cross-validation, blocking, bootstrapping and jackknife methods
+* Practical optimization using Singular-value decomposition and least squares for parameterizing data
+!eblock
+
+
+!split
+===== Topics covered in this course: Machine Learning =====
+
+!bblock 
+The following topics will be covered
+* Linear Regression and Logistic Regression
+* Neural networks and deep learning
+* Decisions trees and nearest neighbor algorithms
+* Support vector machines
+* Bayesian Neural Networks
+* Boltzmann Machines
+* Dimensionality reduction, from PCA to cluster models
+!eblock
+
+
+!split
+===== Extremely useful tools, strongly recommended =====
+
+!bblock and discussed at the lab sessions
+  * GIT for version control, highly recommended
+  * Devilry for handing in projects, next week
+  * Anaconda and other Python environments, see intro slides
+!eblock
+
+
+
+
+
+!split
+===== Other courses on Data science and Machine Learning  at UiO =====
+
+The link here URL:"https://www.mn.uio.no/english/research/about/centre-focus/innovation/data-science/studies/"  gives an excellent overview of courses on Machine learning at UiO.
+
+o "STK2100 Machine learning and statistical methods for prediction and classification":"http://www.uio.no/studier/emner/matnat/math/STK2100/index-eng.html". 
+o "IN3050 Introduction to Artificial Intelligence and Machine Learning":"https://www.uio.no/studier/emner/matnat/ifi/IN3050/index-eng.html". Introductory course in machine learning and AI with an algorithmic approach. 
+o "STK-INF3000/4000 Selected Topics in Data Science":"http://www.uio.no/studier/emner/matnat/math/STK-INF3000/index-eng.html". The course provides insight into selected contemporary relevant topics within Data Science. 
+o "IN4080 Natural Language Processing":"https://www.uio.no/studier/emner/matnat/ifi/IN4080/index.html". Probabilistic and machine learning techniques applied to natural language processing. 
+o "STK-IN4300 Statistical learning methods in Data Science":"https://www.uio.no/studier/emner/matnat/math/STK-IN4300/index-eng.html". An advanced introduction to statistical and machine learning. For students with a good mathematics and statistics background.
+o "INF4490 Biologically Inspired Computing":"http://www.uio.no/studier/emner/matnat/ifi/INF4490/". An introduction to self-adapting methods also called artificial intelligence or machine learning. 
+o "IN-STK5000  Adaptive Methods for Data-Based Decision Making":"https://www.uio.no/studier/emner/matnat/ifi/IN-STK5000/index-eng.html". Methods for adaptive collection and processing of data based on machine learning techniques. 
+o "IN5400/INF5860 Machine Learning for Image Analysis":"https://www.uio.no/studier/emner/matnat/ifi/IN5400/". An introduction to deep learning with particular emphasis on applications within Image analysis, but useful for other application areas too.
+o "TEK5040 Deep learning for autonomous systems":"https://www.uio.no/studier/emner/matnat/its/TEK5040/". The course addresses advanced algorithms and architectures for deep learning with neural networks. The course provides an introduction to how deep-learning techniques can be used in the construction of key parts of advanced autonomous systems that exist in physical environments and cyber environments.
+o "STK4051 Computational Statistics":"https://www.uio.no/studier/emner/matnat/math/STK4051/index-eng.html"
+o "STK4021 Applied Bayesian Analysis and Numerical Methods":"https://www.uio.no/studier/emner/matnat/math/STK4021/index-eng.html"
+
+
+
+
diff --git a/doc/src/Intro2Course/back.do.txt b/doc/src/Intro2Course/back.do.txt
new file mode 100644
index 000000000..b6f0d63ba
--- /dev/null
+++ b/doc/src/Intro2Course/back.do.txt
@@ -0,0 +1,161 @@
+TITLE: Applied Data Analysis and Machine Learning: Introduction to the course, Logistics and Practicalities
+AUTHOR: Morten Hjorth-Jensen  {copyright, 1999-present|CC BY-NC} at Department of Physics, University of Oslo & Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University
+DATE: today
+
+
+
+!split
+===== Overview of first week =====
+
+!bblock
+  * Thursday: First lecture: Presentation of the course, aims and content
+  * Thursday: Second Lecture: Start with simple linear regression and repetition of linear algebra
+  * Friday: Linear regression   
+  * Computer lab: Tuesday. First time: Tuesday August 27. 
+!eblock
+
+
+!split
+===== Lectures and ComputerLab =====
+
+!bblock
+  * Lectures: Thursday (12.15pm-2pm) and Friday (12.15pm-2pm).
+  * Weekly reading assignments needed to solve projects and exercises.
+  * Weekly exercises when not working on projects. You can hand in exercises if you want.
+  * First hour of each lab session may be used to discuss technicalities, address questions etc linked with projects and exercises.
+  * Detailed lecture notes, exercises, all programs presented, projects etc can be found at the homepage of the course.
+  * Computerlab: Tuesday (8am-6pm), VB IT-auditorium 3
+  * Weekly plans and all other information are on the official webpage.
+  * No final exam, three projects that are graded and have to be approved. 
+!eblock
+
+!split
+===== Course Format =====
+
+!bblock
+  * Three compulsory projects. Electronic reports only using "devilry":"https://devilry.ifi.uio.no/" to hand in projects and "Git":"https://github.com/" for repository and all your material.
+  * Evaluation and grading: The three projects are graded and each counts 1/3 of the final mark. No final written or oral exam.
+    o For the last project Each group/participant submits a proposal or works with suggested (by us) proposals for the project.
+    o If possible, we would like to organize the last project as a workshop where each group makes a poster and presents this to all other participants of the course
+    o Poster session where all participants can study and discuss the other proposals.
+    o Based on feedback etc, each group finalizes the report and submits for grading. 
+  * Python is the default programming language, but feel free to use C/C++ and/or Fortran or other programmin languages. All source codes discussed during the lectures can be found at the webpage and "github address":"https://github.com/CompPhysics/MachineLearning/tree/master/doc/Programs" of the course. 
+!eblock
+
+!split
+===== Teachers and ComputerLab =====
+
+!bblock
+
+_Teachers :_
+
+o "Hanna Svennevik":"https://www.researchgate.net/profile/Hanna_Svennevik"
+o "Morten Hjorth-Jensen":"http://mhjgit.github.io/info/doc/web/"
+o "Lucas Charpentier":"https://no.linkedin.com/in/lucas-charpentier-176206171"
+o "Stian Bilek":"https://www.researchgate.net/profile/Stian_Bilek"
+
+
+
+|------------------------------------------------------|
+|      day         |        Time |
+|----------------------------------------------------|
+| Group 1: Tuesday | 8am-10am |
+| Group 2: Tuesday | 10am-12pm |  
+| Group 3: Tuesday | 12pm-2pm   | 
+| Group 4: Tuesday | 2pm-4pm   | 
+|-------------------------------------------------|
+
+!eblock
+
+!split
+===== Deadlines for projects (end of day) =====
+
+!bblock
+
+o Project 1: September 30 (graded with feedback)
+o Project 2: November 4 (graded with feedback)
+o Project 3: December 2  (graded with feedback)
+
+Projects are handed in using devilry.ifi.uio.no. We use Github as repository for codes, benchmark calculations etc. Comments and feedback on projects only via devilry. 
+
+!eblock
+
+
+
+!split
+===== Learning outcomes =====
+
+!bblock
+
+* Learn about basic data analysis, statistical analysis, Bayesian statistics, Monte Carlo sampling, data optimization and machine learning
+* Be capable of extending the acquired knowledge to other systems and cases
+* Have an understanding of central algorithms used in data analysis and machine learning
+* Gain knowledge of central aspects of Monte Carlo methods, Markov chains, Gibbs samplers and their possible applications
+* Understand linear methods for regression and classification, from ordinary least squares, via Lasso and Ridge to Logistic regression
+* Learn about various neural networks and deep  learning methods for supervised and unsupervised learning
+* Learn about about decision trees and random forests
+* Learn about support vector machines and kernel transformations
+* Reduction of data sets, from PCA to clustering, supervised and unsupervided methods
+* Work on numerical projects to illustrate the theory. The projects play a central role and you are expected to know modern programming languages like Python or C++
+
+!eblock
+
+!split
+===== Topics covered in this course: Statistical analysis and optimization of data =====
+
+!bblock 
+* Basic concepts, expectation values, variance, covariance, correlation functions and errors
+* Simpler models, binomial distribution, the Poisson distribution, simple and multivariate normal distributions
+* Central elements of Bayesian statistics and modeling
+* Gradient methods for data optimization
+* Monte Carlo methods, Markov chains, Metropolis-Hastings algorithm
+* Linear methods for regression and classification
+* Estimation of errors using cross-validation, blocking, bootstrapping and jackknife methods
+* Practical optimization using Singular-value decomposition and least squares for parameterizing data
+!eblock
+
+
+!split
+===== Topics covered in this course: Machine Learning =====
+
+!bblock 
+The following topics will be covered
+* Linear Regression and Logistic Regression
+* Neural networks and deep learning
+* Decisions trees and nearest neighbor algorithms
+* Support vector machines
+* Bayesian Neural Networks
+* Boltzmann Machines
+* Dimensionality reduction, from PCA to cluster models
+!eblock
+
+!split
+===== Extremely useful tools, strongly recommended =====
+
+!bblock and discussed at the lab sessions
+  * GIT for version control, highly recommended
+  * Devilry for handing in projects, next week
+  * Anaconda and other Python environments, see intro slides
+!eblock
+
+
+!split
+===== Other courses on Data science and Machine Learning  at UiO =====
+
+The link here URL:"https://www.mn.uio.no/english/research/about/centre-focus/innovation/data-science/studies/"  gives an excellent overview of courses on Machine learning at UiO.
+
+o "STK2100 Machine learning and statistical methods for prediction and classification":"http://www.uio.no/studier/emner/matnat/math/STK2100/index-eng.html". 
+o "IN3050 Introduction to Artificial Intelligence and Machine Learning":"https://www.uio.no/studier/emner/matnat/ifi/IN3050/index-eng.html". Introductory course in machine learning and AI with an algorithmic approach. 
+o "STK-INF3000/4000 Selected Topics in Data Science":"http://www.uio.no/studier/emner/matnat/math/STK-INF3000/index-eng.html". The course provides insight into selected contemporary relevant topics within Data Science. 
+o "IN4080 Natural Language Processing":"https://www.uio.no/studier/emner/matnat/ifi/IN4080/index.html". Probabilistic and machine learning techniques applied to natural language processing. 
+o "STK-IN4300  Statistical learning methods in Data Science":"https://www.uio.no/studier/emner/matnat/math/STK-IN4300/index-eng.html". An advanced introduction to statistical and machine learning. For students with a good mathematics and statistics background.
+o "INF4490 Biologically Inspired Computing":"http://www.uio.no/studier/emner/matnat/ifi/INF4490/". An introduction to self-adapting methods also called artificial intelligence or machine learning. 
+o "IN-STK5000  Adaptive Methods for Data-Based Decision Making":"https://www.uio.no/studier/emner/matnat/ifi/IN-STK5000/index-eng.html". Methods for adaptive collection and processing of data based on machine learning techniques. 
+o "IN5400/INF5860  Machine Learning for Image Analysis":"https://www.uio.no/studier/emner/matnat/ifi/IN5400/". An introduction to deep learning with particular emphasis on applications within Image analysis, but useful for other application areas too.
+o "TEK5040  Dyp lring for autonome systemer":"https://www.uio.no/studier/emner/matnat/its/TEK5040/". The course addresses advanced algorithms and architectures for deep learning with neural networks. The course provides an introduction to how deep-learning techniques can be used in the construction of key parts of advanced autonomous systems that exist in physical environments and cyber environments.
+o "STK4051 Computational Statistics":"https://www.uio.no/studier/emner/matnat/math/STK4051/index-eng.html"
+o "STK4021 Applied Bayesian Analysis and Numerical Methods":"https://www.uio.no/studier/emner/matnat/math/STK4021/index-eng.html"
+
+
+
+
diff --git a/doc/src/Linalg/Linalg.do.txt~ b/doc/src/Linalg/Linalg.do.txt~
new file mode 100644
index 000000000..b24f84427
--- /dev/null
+++ b/doc/src/Linalg/Linalg.do.txt~
@@ -0,0 +1,1121 @@
+TITLE: Data analysis and Machine Learning  Lectures: Linear Algebra and Handling of Arrays
+AUTHOR: Morten Hjorth-Jensen {copyright, 1999-present|CC BY-NC} at Department of Physics, University of Oslo & Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University
+DATE: today
+
+
+!split
+===== Introduction =====
+The aim of this set of lectures is to review some central linear algebra algorithms that we will need in our 
+data analysis part and in the construction of Machine Learning algorithms (ML). 
+This will allow us to introduce some central programming features of high-level languages like Python and 
+compiled languages like C++ and/or Fortran. 
+
+As discussed in the introductory notes, these series of lectures focuses both on using
+central Python packages like _tensorflow_ and _scikit-learn_ as well
+as writing your own codes for some central ML algorithms. The
+latter can be written in a language of your choice, be it Python, Julia, R,
+Rust, C++, Fortran etc. In order to avoid confusion however, in these lectures we will limit our
+attention to Python, C++ and Fortran.  
+
+
+!split 
+===== Important Matrix and vector handling packages =====
+
+There are several central software packages for linear algebra and eigenvalue problems. Several of the more
+popular ones have been wrapped into ofter software packages like those from the widely used text _Numerical Recipes_. The original source codes in many of the available packages are often taken from the widely used
+software package LAPACK, which follows two other popular packages
+developed in the 1970s, namely EISPACK and LINPACK.  We describe them shortly here.
+
+  * LINPACK: package for linear equations and least square problems.
+  * LAPACK:package for solving symmetric, unsymmetric and generalized eigenvalue problems. From LAPACK's website URL: "http://www.netlib.org" it is possible to download for free all source codes from this library. Both C/C++ and Fortran versions are available.
+  * BLAS (I, II and III): (Basic Linear Algebra Subprograms) are routines that provide standard building blocks for performing basic vector and matrix operations. Blas I is vector operations, II vector-matrix operations and III matrix-matrix operations. Highly parallelized and efficient codes, all available for download from URL: "http://www.netlib.org".
+
+When dealing with matrices and vectors a central issue is memory
+handling and allocation. If our code is written in Python the way we
+declare these objects and the way they are handled, interpreted and
+used by say a linear algebra library, requires codes that interface
+our Python program with such libraries. For Python programmers,
+_Numpy_ is by now the standard Python package for numerical arrays in
+Python as well as the source of functions which act on these
+arrays. These functions span from eigenvalue solvers to functions that
+compute the mean value, variance or the covariance matrix. If you are
+not familiar with how arrays are handled in say Python or compiled
+languages like C++ and Fortran, the sections in this chapter may be
+useful. For C++ programmer, _Armadillo_ is widely used library for
+linear algebra and eigenvalue problems. In addition it offers a
+convenient way to handle and organize arrays. We discuss this library
+as well.   Before we proceed we believe  it may be convenient to repeat some basic features of 
+ matrices and vectors.
+
+!split 
+===== Basic Matrix Features =====
+
+!bblock Matrix properties reminder
+!bt
+\[
+ \mathbf{A} =
+      \begin{bmatrix} a_{11} & a_{12} & a_{13} & a_{14} \\
+                                 a_{21} & a_{22} & a_{23} & a_{24} \\
+                                   a_{31} & a_{32} & a_{33} & a_{34} \\
+                                  a_{41} & a_{42} & a_{43} & a_{44}
+             \end{bmatrix}\qquad
+\mathbf{I} =
+      \begin{bmatrix} 1 & 0 & 0 & 0 \\
+                                 0 & 1 & 0 & 0 \\
+                                 0 & 0 & 1 & 0 \\
+                                 0 & 0 & 0 & 1
+             \end{bmatrix}
+\]
+!et
+!eblock
+!split
+===== Basic Matrix Features =====
+!bblock
+
+The inverse of a matrix is defined by
+
+!bt
+\[
+\mathbf{A}^{-1} \cdot \mathbf{A} = I
+\]
+!et
+!eblock
+
+!split
+===== Basic Matrix Features =====
+
+!bblock Matrix Properties Reminder
+
+|----------------------------------------------------------------------|
+|       Relations      |       Name      | matrix elements             |
+|----------------------------------------------------------------------|
+| $A = A^{T}$         | symmetric       | $a_{ij} = a_{ji}$            |
+| $A = \left (A^{T} \right )^{-1}$ | real orthogonal | $\sum_k a_{ik} a_{jk} = \sum_k a_{ki} a_{kj} = \delta_{ij}$ |
+| $A = A^{ * }$          | real matrix     | $a_{ij} = a_{ij}^{ * }$       |
+| $A = A^{\dagger}$     |  hermitian      | $a_{ij} = a_{ji}^{ * }$       |
+| $A = \left (A^{\dagger} \right )^{-1}$ | unitary | $\sum_k a_{ik} a_{jk}^{ * } = \sum_k a_{ki}^{ * } a_{kj} = \delta_{ij}$ |
+|----------------------------------------------------------------------|
+
+!eblock
+
+!split
+===== Some famous Matrices =====
+
+  * Diagonal if $a_{ij}=0$ for $i\ne j$
+  * Upper triangular if $a_{ij}=0$ for $i > j$
+  * Lower triangular if $a_{ij}=0$ for $i < j$
+  * Upper Hessenberg if $a_{ij}=0$ for $i > j+1$
+  * Lower Hessenberg if $a_{ij}=0$ for $i < j+1$
+  * Tridiagonal if $a_{ij}=0$ for $|i -j| > 1$
+  * Lower banded with bandwidth $p$: $a_{ij}=0$ for $i > j+p$
+  * Upper banded with bandwidth $p$: $a_{ij}=0$ for $i < j+p$
+  * Banded, block upper triangular, block lower triangular....
+
+!split
+===== Basic Matrix Features =====
+
+!bblock Some Equivalent Statements
+For an $N\times N$ matrix  $\mathbf{A}$ the following properties are all equivalent
+
+  * If the inverse of $\mathbf{A}$ exists, $\mathbf{A}$ is nonsingular.
+  * The equation $\mathbf{Ax}=0$ implies $\mathbf{x}=0$.
+  * The rows of $\mathbf{A}$ form a basis of $R^N$.
+  * The columns of $\mathbf{A}$ form a basis of $R^N$.
+  * $\mathbf{A}$ is a product of elementary matrices.
+  * $0$ is not eigenvalue of $\mathbf{A}$.
+!eblock
+
+!split
+===== Numpy and arrays =====
+"Numpy":"http://www.numpy.org/" provides an easy way to handle arrays in Python. The standard way to import this library is as
+!bc pycod
+import numpy as np
+n = 10
+x = np.random.normal(size=n)
+print(x)
+!ec
+Here we have defined a vector $x$ with $n=10$ elements with its values given by the Normal distribution $N(0,1)$.
+Another alternative is to declare a vector as follows
+!bc pycod
+import numpy as np
+x = np.array([1, 2, 3])
+print(x)
+!ec
+Here we have defined a vector with three elements, with $x_0=1$, $x_1=2$ and $x_2=3$. Note that both Python and C++
+start numbering array elements from $0$ and on. This means that a vector with $n$ elements has a sequence of entities $x_0, x_1, x_2, \dots, x_{n-1}$. We could also let (recommended) Numpy to compute the logarithms of a specific array as
+!bc pycod
+import numpy as np
+x = np.log(np.array([4, 7, 8]))
+print(x)
+!ec
+
+Here we have used Numpy's unary function $np.log$. This function is
+highly tuned to compute array elements since the code is vectorized
+and does not require looping. We normaly recommend that you use the
+Numpy intrinsic functions instead of the corresponding _log_ function
+from Python's _math_ module. The looping is done explicitely by the
+_np.log_ function. The alternative, and slower way to compute the
+logarithms of a vector would be to write
+
+!bc pycod
+import numpy as np
+from math import log
+x = np.array([4, 7, 8])
+for i in range(0, len(x)):
+    x[i] = log(x[i])
+print(x)
+!ec
+We note that our code is much longer already and we need to import the _log_ function from the _math_ module. 
+The attentive reader will also notice that the output is $[1, 1, 2]$. Python interprets automacally our numbers as integers (like the _automatic_ keyword in C++). To change this we could define our array elements to be double precision numbers as
+!bc pycod
+import numpy as np
+x = np.log(np.array([4, 7, 8], dtype = np.float64))
+print(x)
+!ec
+or simply write them as double precision numbers (Python uses 64 bits as default for floating point type variables), that is
+!bc pycod
+import numpy as np
+x = np.log(np.array([4.0, 7.0, 8.0])
+print(x)
+!ec
+To check the number of bytes (remember that one byte contains eight bits for double precision variables), you can use simple use the _itemsize_ functionality (the array $x$ is actually an object which inherits the functionalities defined in Numpy) as 
+!bc pycod
+import numpy as np
+x = np.log(np.array([4.0, 7.0, 8.0])
+print(x.itemsize)
+!ec
+
+!split
+===== Matrices in Python =====
+Having defined vectors, we are now ready to try out matrices. We can define a $3 \times 3 $ real matrix $\hat{A}$
+as (recall that we user lowercase letters for vectors and uppercase letters for matrices)
+!bc pycod
+import numpy as np
+A = np.log(np.array([ [4.0, 7.0, 8.0], [3.0, 10.0, 11.0], [4.0, 5.0, 7.0] ]))
+print(A)
+!ec
+If we use the _shape_ function we would get $(3, 3)$ as output, that is verifying that our matrix is a $3\times 3$ matrix. We can slice the matrix and print for example the first column (Python organized matrix elements in a row-major order, see below) as
+!bc pycod
+import numpy as np
+A = np.log(np.array([ [4.0, 7.0, 8.0], [3.0, 10.0, 11.0], [4.0, 5.0, 7.0] ]))
+# print the first column, row-major order and elements start with 0
+print(A[:,0]) 
+!ec
+We can continue this was by printing out other columns or rows. The example here prints out the second column
+!bc pycod
+import numpy as np
+A = np.log(np.array([ [4.0, 7.0, 8.0], [3.0, 10.0, 11.0], [4.0, 5.0, 7.0] ]))
+# print the first column, row-major order and elements start with 0
+print(A[1,:]) 
+!ec 
+Numpy contains many other functionalities that allow us to slice, subdivide etc etc arrays. We strongly recommend that you look up the "Numpy website for more details":"http://www.numpy.org/". Useful functions when defining a matrix are the _np.zeros_ function which declares a matrix of a given dimension and sets all elements to zero
+!bc pycod
+import numpy as np
+n = 10
+# define a matrix of dimension 10 x 10 and set all elements to zero
+A = np.zeros( (n, n) )
+print(A) 
+!ec 
+or initializing all elements to 
+!bc pycod
+import numpy as np
+n = 10
+# define a matrix of dimension 10 x 10 and set all elements to one
+A = np.ones( (n, n) )
+print(A) 
+!ec 
+or as unitarily distributed random numbers (see the material on random number generators in the statistics part)
+!bc pycod
+import numpy as np
+n = 10
+# define a matrix of dimension 10 x 10 and set all elements to random numbers with x \in [0, 1]
+A = np.random.rand(n, n)
+print(A) 
+!ec 
+
+As we will see throughout these lectures, there are several extremely useful functionalities in Numpy.
+As an example, consider the discussion of the covariance matrix. Suppose we have defined three vectors
+$\hat{x}, \hat{y}, \hat{z}$ with $n$ elements each. The covariance matrix is defined as 
+!bt
+\[
+\hat{\Sigma} = \begin{bmatrix} \sigma_{xx} & \sigma_{xy} & \sigma_{xz} \\
+                              \sigma_{yx} & \sigma_{yy} & \sigma_{yz} \\
+                              \sigma_{zx} & \sigma_{zy} & \sigma_{zz} 
+             \end{bmatrix},
+\]
+!et
+where for example
+!bt
+\[
+\sigma_{xy} =\frac{1}{n} \sum_{i=0}^{n-1}(x_i- \overline{x})(y_i- \overline{y}).
+\]
+!et
+The Numpy function _np.cov_ calculates the covariance elements using the factor $1/(n-1)$ instead of $1/n$ since it assumes we do not have the exact mean values. For a more in-depth discussion of the covariance and covariance matrix and its meaning, we refer you to the lectures on statistics. 
+The following simple function uses the _np.vstack_ function which takes each vector of dimension $1\times n$ and produces a $ 3\times n$ matrix $\hat{W}$
+!bt
+\[
+\hat{W} = \begin{bmatrix} x_0 & y_0 & z_0 \\
+                          x_1 & y_1 & z_1 \\
+                          x_2 & y_2 & z_2 \\
+                          \dots & \dots & \dots \\
+                          x_{n-2} & y_{n-2} & z_{n-2} \\
+                          x_{n-1} & y_{n-1} & z_{n-1}
+             \end{bmatrix},
+\]
+!et
+
+which in turn is converted into into the $3 times 3$ covariance matrix
+$\hat{\Sigma}$ via the Numpy function _np.cov()_. In our review of
+statistical functions and quantities we will discuss more about the
+meaning of the covariance matrix. Here we note that we can calculate
+the mean value of each set of samples $\hat{x}$ etc using the Numpy
+function _np.mean(x)_. We can also extract the eigenvalues of the
+covariance matrix through the _np.linalg.eig()_ function.
+
+!bc pycod
+# Importing various packages
+import numpy as np
+
+n = 100
+x = np.random.normal(size=n)
+print(np.mean(x))
+y = 4+3*x+np.random.normal(size=n)
+print(np.mean(y))
+z = x**3+np.random.normal(size=n)
+print(np.mean(z))
+W = np.vstack((x, y, z))
+Sigma = np.cov(W)
+print(Sigma)
+Eigvals, Eigvecs = np.linalg.eig(Sigma)
+print(Eigvals)
+!ec
+
+
+!bc pycod
+import numpy as np
+import matplotlib.pyplot as plt
+from scipy import sparse
+eye = np.eye(4)
+print(eye)
+sparse_mtx = sparse.csr_matrix(eye)
+print(sparse_mtx)
+x = np.linspace(-10,10,100)
+y = np.sin(x)
+plt.plot(x,y,marker='x')
+plt.show()
+!ec
+
+
+
+
+!split
+===== Matrix Handling in C/C++, Static and Dynamical allocation =====
+
+!bblock Static
+We have  an $N\times N$ matrix A  with $N=100$
+In C/C++ this would be  defined as
+
+!bc cppcod
+   int N = 100;
+   double A[100][100];
+   //   initialize all elements to zero
+   for(i=0 ; i < N ; i++) {
+      for(j=0 ; j < N ; j++) {
+         A[i][j] = 0.0;
+
+!ec
+Note the way the matrix is organized, row-major order.
+!eblock
+
+!split
+===== Matrix Handling in C/C++ =====
+
+!bblock Row Major Order, Addition
+We have  $N\times N$ matrices A, B and C and we wish to
+evaluate $A=B+C$.
+
+!bt
+\[
+\mathbf{A}= \mathbf{B}\pm\mathbf{C}  \Longrightarrow a_{ij} = b_{ij}\pm c_{ij},
+\]
+!et
+In C/C++ this would be coded like
+
+!bc cppcod
+   for(i=0 ; i < N ; i++) {
+      for(j=0 ; j < N ; j++) {
+         a[i][j] = b[i][j]+c[i][j]
+
+!ec
+!eblock
+
+!split
+===== Matrix Handling in C/C++ =====
+
+!bblock Row Major Order, Multiplication
+We have  $N\times N$ matrices A, B and C and we wish to
+evaluate $A=BC$.
+
+!bt
+\[
+\mathbf{A}=\mathbf{BC}   \Longrightarrow a_{ij} = \sum_{k=1}^{n} b_{ik}c_{kj},
+\]
+!et
+In C/C++ this would be coded like
+
+!bc cppcod
+   for(i=0 ; i < N ; i++) {
+      for(j=0 ; j < N ; j++) {
+         for(k=0 ; k < N ; k++) {
+            a[i][j]+=b[i][k]*c[k][j];
+
+!ec
+!eblock
+
+
+!split
+===== Dynamic memory allocation in C/C++ =====
+
+At least three possibilities in this course
+
+  * Do it yourself
+  * Use the functions provided in the library package lib.cpp
+  * Use Armadillo URL: "http://arma.sourceforgenet" (a C++ linear algebra library, discussion both here and at lab). 
+
+!split
+===== Matrix Handling in C/C++, Dynamic Allocation =====
+
+!bblock Do it yourself
+!bc cppcod
+int N;
+double **  A;
+A = new double*[N]
+for ( i = 0; i < N; i++)
+    A[i] = new double[N];
+!ec
+Always free space when you don't need an array anymore.
+
+!bc cppcod
+for ( i = 0; i < N; i++)
+    delete[] A[i];
+delete[] A;
+!ec
+!eblock
+
+!split
+===== Armadillo, recommended!! =====
+
+  * Armadillo is a C++ linear algebra library (matrix maths) aiming towards a good balance between speed and ease of use. The syntax is deliberately similar to Matlab.
+  * Integer, floating point and complex numbers are supported, as well as a subset of trigonometric and statistics functions. Various matrix decompositions are provided through optional integration with LAPACK, or one of its high performance drop-in replacements (such as the multi-threaded MKL or ACML libraries).
+  * A delayed evaluation approach is employed (at compile-time) to combine several operations into one and reduce (or eliminate) the need for temporaries. This is accomplished through recursive templates and template meta-programming.
+  * Useful for conversion of research code into production environments, or if C++ has been decided as the language of choice, due to speed and/or integration capabilities.
+  * The library is open-source software, and is distributed under a license that is useful in both open-source and commercial/proprietary contexts.
+
+!split
+===== Armadillo, simple examples =====
+
+!bc cppcod
+#include 
+#include 
+
+using namespace std;
+using namespace arma;
+
+int main(int argc, char** argv)
+  {
+  mat A = randu(5,5);
+  mat B = randu(5,5);
+
+  cout << A*B << endl;
+
+  return 0;
+
+!ec
+
+!split
+===== Armadillo, how to compile and install =====
+
+For people using Ubuntu, Debian, Linux Mint, simply go to the synaptic package manager and install
+armadillo from there.
+You may have to install Lapack as well.
+For Mac and Windows users, follow the instructions from the webpage
+URL: "http://arma.sourceforge.net".
+To compile, use for example (linux/ubuntu)
+
+!bc cppcod
+c++ -O2 -o program.x program.cpp  -larmadillo -llapack -lblas
+!ec
+where the `-l` option indicates the library you wish to link to.
+
+For OS X users you may have to declare the paths to the include files and the libraries as
+!bc cppcod
+c++ -O2 -o program.x program.cpp  -L/usr/local/lib -I/usr/local/include -larmadillo -llapack -lblas
+!ec
+
+!split
+===== Armadillo, simple examples =====
+
+!bc cppcod
+#include 
+#include "armadillo"
+using namespace arma;
+using namespace std;
+
+int main(int argc, char** argv)
+  {
+  // directly specify the matrix size (elements are uninitialised)
+  mat A(2,3);
+  // .n_rows = number of rows    (read only)
+  // .n_cols = number of columns (read only)
+  cout << "A.n_rows = " << A.n_rows << endl;
+  cout << "A.n_cols = " << A.n_cols << endl;
+  // directly access an element (indexing starts at 0)
+  A(1,2) = 456.0;
+  A.print("A:");
+  // scalars are treated as a 1x1 matrix,
+  // hence the code below will set A to have a size of 1x1
+  A = 5.0;
+  A.print("A:");
+  // if you want a matrix with all elements set to a particular value
+  // the .fill() member function can be used
+  A.set_size(3,3);
+  A.fill(5.0);  A.print("A:");
+!ec
+
+!split
+===== Armadillo, simple examples =====
+
+!bc cppcod
+  mat B;
+
+  // endr indicates "end of row"
+  B << 0.555950 << 0.274690 << 0.540605 << 0.798938 << endr
+    << 0.108929 << 0.830123 << 0.891726 << 0.895283 << endr
+    << 0.948014 << 0.973234 << 0.216504 << 0.883152 << endr
+    << 0.023787 << 0.675382 << 0.231751 << 0.450332 << endr;
+
+  // print to the cout stream
+  // with an optional string before the contents of the matrix
+  B.print("B:");
+
+  // the << operator can also be used to print the matrix
+  // to an arbitrary stream (cout in this case)
+  cout << "B:" << endl << B << endl;
+  // save to disk
+  B.save("B.txt", raw_ascii);
+  // load from disk
+  mat C;
+  C.load("B.txt");
+  C += 2.0 * B;
+  C.print("C:");
+!ec
+
+!split
+===== Armadillo, simple examples =====
+
+!bc cppcod
+  // submatrix types:
+  //
+  // .submat(first_row, first_column, last_row, last_column)
+  // .row(row_number)
+  // .col(column_number)
+  // .cols(first_column, last_column)
+  // .rows(first_row, last_row)
+
+  cout << "C.submat(0,0,3,1) =" << endl;
+  cout << C.submat(0,0,3,1) << endl;
+
+  // generate the identity matrix
+  mat D = eye(4,4);
+
+  D.submat(0,0,3,1) = C.cols(1,2);
+  D.print("D:");
+
+  // transpose
+  cout << "trans(B) =" << endl;
+  cout << trans(B) << endl;
+
+  // maximum from each column (traverse along rows)
+  cout << "max(B) =" << endl;
+  cout << max(B) << endl;
+
+!ec
+
+!split
+===== Armadillo, simple examples =====
+
+!bc cppcod
+  // maximum from each row (traverse along columns)
+  cout << "max(B,1) =" << endl;
+  cout << max(B,1) << endl;
+  // maximum value in B
+  cout << "max(max(B)) = " << max(max(B)) << endl;
+  // sum of each column (traverse along rows)
+  cout << "sum(B) =" << endl;
+  cout << sum(B) << endl;
+  // sum of each row (traverse along columns)
+  cout << "sum(B,1) =" << endl;
+  cout << sum(B,1) << endl;
+  // sum of all elements
+  cout << "sum(sum(B)) = " << sum(sum(B)) << endl;
+  cout << "accu(B)     = " << accu(B) << endl;
+  // trace = sum along diagonal
+  cout << "trace(B)    = " << trace(B) << endl;
+  // random matrix -- values are uniformly distributed in the [0,1] interval
+  mat E = randu(4,4);
+  E.print("E:");
+
+!ec
+
+!split
+===== Armadillo, simple examples =====
+
+!bc cppcod
+  // row vectors are treated like a matrix with one row
+  rowvec r;
+  r << 0.59499 << 0.88807 << 0.88532 << 0.19968;
+  r.print("r:");
+
+  // column vectors are treated like a matrix with one column
+  colvec q;
+  q << 0.81114 << 0.06256 << 0.95989 << 0.73628;
+  q.print("q:");
+
+  // dot or inner product
+  cout << "as_scalar(r*q) = " << as_scalar(r*q) << endl;
+
+    // outer product
+  cout << "q*r =" << endl;
+  cout << q*r << endl;
+
+
+  // sum of three matrices (no temporary matrices are created)
+  mat F = B + C + D;
+  F.print("F:");
+
+    return 0;
+
+!ec
+
+!split
+===== Armadillo, simple examples =====
+
+!bc cppcod
+#include 
+#include "armadillo"
+using namespace arma;
+using namespace std;
+
+int main(int argc, char** argv)
+  {
+  cout << "Armadillo version: " << arma_version::as_string() << endl;
+
+  mat A;
+
+  A << 0.165300 << 0.454037 << 0.995795 << 0.124098 << 0.047084 << endr
+    << 0.688782 << 0.036549 << 0.552848 << 0.937664 << 0.866401 << endr
+    << 0.348740 << 0.479388 << 0.506228 << 0.145673 << 0.491547 << endr
+    << 0.148678 << 0.682258 << 0.571154 << 0.874724 << 0.444632 << endr
+    << 0.245726 << 0.595218 << 0.409327 << 0.367827 << 0.385736 << endr;
+
+  A.print("A =");
+
+  // determinant
+  cout << "det(A) = " << det(A) << endl;
+!ec
+
+!split
+===== Armadillo, simple examples =====
+
+!bc cppcod
+  // inverse
+  cout << "inv(A) = " << endl << inv(A) << endl;
+  double k = 1.23;
+
+  mat    B = randu(5,5);
+  mat    C = randu(5,5);
+
+  rowvec r = randu(5);
+  colvec q = randu(5);
+
+
+  // examples of some expressions
+  // for which optimised implementations exist
+  // optimised implementation of a trinary expression
+  // that results in a scalar
+  cout << "as_scalar( r*inv(diagmat(B))*q ) = ";
+  cout << as_scalar( r*inv(diagmat(B))*q ) << endl;
+
+  // example of an expression which is optimised
+  // as a call to the dgemm() function in BLAS:
+  cout << "k*trans(B)*C = " << endl << k*trans(B)*C;
+
+    return 0;
+
+!ec
+
+!split
+===== Gaussian Elimination =====
+
+We start with the linear set of equations
+
+!bt
+\[
+   \mathbf{A}\mathbf{x} = \mathbf{w}.
+\]
+!et
+We assume also that the matrix $\mathbf{A}$ is non-singular and that the
+matrix elements along the diagonal satisfy $a_{ii} \ne 0$. Simple $4\times 4 $ example
+
+!bt
+\[
+\begin{bmatrix}
+                           a_{11}& a_{12} &a_{13}& a_{14}\\
+                           a_{21}& a_{22} &a_{23}& a_{24}\\
+                           a_{31}& a_{32} &a_{33}& a_{34}\\
+                           a_{41}& a_{42} &a_{43}& a_{44}\\
+                      \end{bmatrix} \begin{bmatrix}
+                           x_1\\
+                           x_2\\
+                           x_3 \\
+                           x_4  \\
+                      \end{bmatrix}
+  =\begin{bmatrix}
+                           w_1\\
+                           w_2\\
+                           w_3 \\
+                           w_4\\
+                      \end{bmatrix}.
+\]
+!et
+
+!split
+===== Gaussian Elimination =====
+or
+
+!bt
+\begin{align}
+ a_{11}x_1 +a_{12}x_2 +a_{13}x_3 + a_{14}x_4=&w_1 \nonumber \\
+a_{21}x_1 + a_{22}x_2 + a_{23}x_3 + a_{24}x_4=&w_2 \nonumber \\
+a_{31}x_1 + a_{32}x_2 + a_{33}x_3 + a_{34}x_4=&w_3 \nonumber \\
+a_{41}x_1 + a_{42}x_2 + a_{43}x_3 + a_{44}x_4=&w_4. \nonumber
+\end{align}
+!et
+
+!split
+===== Gaussian Elimination =====
+
+The basic idea of Gaussian elimination is to use the first equation to eliminate the first unknown $x_1$
+from the remaining $n-1$ equations. Then we use the new second equation to eliminate the second unknown
+$x_2$ from the remaining $n-2$ equations. With $n-1$ such eliminations
+we obtain a so-called upper triangular set of equations of the form
+
+!bt
+\begin{align}
+ b_{11}x_1 +b_{12}x_2 +b_{13}x_3 + b_{14}x_4=&y_1 \nonumber \\
+ b_{22}x_2 + b_{23}x_3 + b_{24}x_4=&y_2 \nonumber \\
+b_{33}x_3 + b_{34}x_4=&y_3 \nonumber \\
+b_{44}x_4=&y_4. \nonumber
+label{eq:gaussbacksub}
+\end{align}
+!et
+We can solve this system of equations recursively starting from $x_n$ (in our case $x_4$) and proceed with
+what is called a backward substitution. 
+
+!split
+===== Gaussian Elimination =====
+This process can be expressed mathematically as
+
+!bt
+\begin{equation}
+   x_m = \frac{1}{b_{mm}}\left(y_m-\sum_{k=m+1}^nb_{mk}x_k\right)\quad m=n-1,n-2,\dots,1.
+\end{equation}
+!et
+To arrive at such an upper triangular system of equations, we start by eliminating
+the unknown $x_1$ for $j=2,n$. We achieve this by multiplying the first equation by $a_{j1}/a_{11}$ and then subtract
+the result from the $j$th equation. We assume obviously that $a_{11}\ne 0$ and that
+$\mathbf{A}$ is not singular.
+
+!split
+===== Gaussian Elimination =====
+
+Our actual $4\times 4$ example reads after the first operation
+
+!bt
+\[
+\begin{bmatrix}
+                           a_{11}& a_{12} &a_{13}& a_{14}\\
+                           0& (a_{22}-\frac{a_{21}a_{12}}{a_{11}}) &(a_{23}-\frac{a_{21}a_{13}}{a_{11}}) & (a_{24}-\frac{a_{21}a_{14}}{a_{11}})\\
+0& (a_{32}-\frac{a_{31}a_{12}}{a_{11}})& (a_{33}-\frac{a_{31}a_{13}}{a_{11}})& (a_{34}-\frac{a_{31}a_{14}}{a_{11}})\\
+0&(a_{42}-\frac{a_{41}a_{12}}{a_{11}}) &(a_{43}-\frac{a_{41}a_{13}}{a_{11}}) & (a_{44}-\frac{a_{41}a_{14}}{a_{11}}) \\
+                      \end{bmatrix} \begin{bmatrix}
+                           x_1\\
+                           x_2\\
+                           x_3 \\
+                           x_4  \\
+                      \end{bmatrix} 
+  =\begin{bmatrix}
+                           y_1\\
+                           w_2^{(2)}\\
+                           w_3^{(2)} \\
+                           w_4^{(2)}\\
+                      \end{bmatrix},
+\]
+!et
+or
+
+!bt
+\begin{align}
+ b_{11}x_1 +b_{12}x_2 +b_{13}x_3 + b_{14}x_4=&y_1 \nonumber \\
+ a^{(2)}_{22}x_2 + a^{(2)}_{23}x_3 + a^{(2)}_{24}x_4=&w^{(2)}_2 \nonumber \\
+ a^{(2)}_{32}x_2 + a^{(2)}_{33}x_3 + a^{(2)}_{34}x_4=&w^{(2)}_3 \nonumber \\
+ a^{(2)}_{42}x_2 + a^{(2)}_{43}x_3 + a^{(2)}_{44}x_4=&w^{(2)}_4, \nonumber \\
+\end{align}
+!et
+
+!split
+===== Gaussian Elimination =====
+
+The new coefficients are
+
+!bt
+\begin{equation}
+   b_{1k} = a_{1k}^{(1)} \quad k=1,\dots,n,
+\end{equation}
+!et
+where each $a_{1k}^{(1)}$ is equal to the original $a_{1k}$ element. The other coefficients are
+
+!bt
+\begin{equation}
+a_{jk}^{(2)} = a_{jk}^{(1)}-\frac{a_{j1}^{(1)}a_{1k}^{(1)}}{a_{11}^{(1)}} \quad j,k=2,\dots,n,
+\end{equation}
+!et
+with a new right-hand side given by
+
+!bt
+\begin{equation}
+y_{1}=w_1^{(1)}, \quad w_j^{(2)} =w_j^{(1)}-\frac{a_{j1}^{(1)}w_1^{(1)}}{a_{11}^{(1)}} \quad j=2,\dots,n.
+\end{equation}
+!et
+We have also set $w_1^{(1)}=w_1$, the original vector element.
+We see that the system of unknowns $x_1,\dots,x_n$ is transformed into an $(n-1)\times (n-1)$ problem.
+
+!split
+===== Gaussian Elimination =====
+
+This step is called forward substitution.
+Proceeding with these substitutions, we obtain the
+general expressions for the new coefficients
+
+!bt
+\begin{equation}
+   a_{jk}^{(m+1)} = a_{jk}^{(m)}-\frac{a_{jm}^{(m)}a_{mk}^{(m)}}{a_{mm}^{(m)}} \quad j,k=m+1,\dots,n,
+\end{equation}
+!et
+with $m=1,\dots,n-1$ and a
+right-hand side given by
+
+!bt
+\begin{equation}
+   w_j^{(m+1)} =w_j^{(m)}-\frac{a_{jm}^{(m)}w_m^{(m)}}{a_{mm}^{(m)}}\quad j=m+1,\dots,n.
+\end{equation}
+!et
+This set of $n-1$ elimations leads us to an equations which is solved by back substitution.
+If the arithmetics is exact and the matrix $\mathbf{A}$ is not singular, then the computed answer will be exact.
+
+Even though the matrix elements along the diagonal are not zero,
+numerically small numbers may appear and subsequent divisions may lead to large numbers, which, if added
+to a small number may yield losses of precision. Suppose for example that our first division in $(a_{22}-a_{21}a_{12}/a_{11})$
+results in $-10^{-7}$ and that $a_{22}$ is one.
+one. We are then
+adding $10^7+1$. With single precision this results in $10^7$.
+
+
+
+!split
+===== Linear Algebra Methods =====
+
+  * Gaussian elimination, $O(2/3n^3)$ flops, general matrix
+  * LU decomposition, upper triangular and lower tridiagonal matrices, $O(2/3n^3)$ flops, general matrix. Get easily the inverse, determinant and can solve linear equations with back-substitution only, $O(n^2)$ flops
+  * Cholesky decomposition. Real symmetric or hermitian positive definite matrix, $O(1/3n^3)$ flops.
+  * Tridiagonal linear systems, important for differential equations. Normally positive definite and non-singular. $O(8n)$ flops for symmetric. Special case of banded matrices.
+  * Singular value decomposition
+  * the QR method will be discussed in chapter 7 in connection with eigenvalue systems. $O(4/3n^3)$ flops.
+
+!split
+===== LU Decomposition =====
+
+The LU decomposition method means that we can rewrite
+this matrix as the product of two matrices $\mathbf{L}$ and $\mathbf{U}$
+where
+
+!bt
+\[
+   \begin{bmatrix}
+                          a_{11} & a_{12} & a_{13} & a_{14} \\
+                          a_{21} & a_{22} & a_{23} & a_{24} \\
+                          a_{31} & a_{32} & a_{33} & a_{34} \\
+                          a_{41} & a_{42} & a_{43} & a_{44}
+                      \end{bmatrix}
+                      = \begin{bmatrix}
+                              1  & 0      & 0      & 0 \\
+                          l_{21} & 1      & 0      & 0 \\
+                          l_{31} & l_{32} & 1      & 0 \\
+                          l_{41} & l_{42} & l_{43} & 1
+                      \end{bmatrix}
+                        \begin{bmatrix}
+                          u_{11} & u_{12} & u_{13} & u_{14} \\
+                               0 & u_{22} & u_{23} & u_{24} \\
+                               0 & 0      & u_{33} & u_{34} \\
+                               0 & 0      &  0     & u_{44}
+             \end{bmatrix}.
+\]
+!et
+
+!split
+===== LU Decomposition =====
+
+LU decomposition forms the backbone of other algorithms in linear algebra, such as the
+solution of linear equations given by
+
+!bt
+\begin{align}
+ a_{11}x_1 +a_{12}x_2 +a_{13}x_3 + a_{14}x_4=&w_1 \nonumber \\
+a_{21}x_1 + a_{22}x_2 + a_{23}x_3 + a_{24}x_4=&w_2 \nonumber \\
+a_{31}x_1 + a_{32}x_2 + a_{33}x_3 + a_{34}x_4=&w_3 \nonumber \\
+a_{41}x_1 + a_{42}x_2 + a_{43}x_3 + a_{44}x_4=&w_4.  \nonumber
+\end{align}
+!et
+The above set of equations is conveniently solved by using LU decomposition as an intermediate step.
+
+The matrix $\mathbf{A}\in \mathbb{R}^{n\times n}$ has an LU factorization if the determinant
+is different from zero. If the LU factorization exists and $\mathbf{A}$ is non-singular, then the LU factorization
+is unique and the determinant is given by
+
+!bt
+\[
+det\{\mathbf{A}\}=det\{\mathbf{LU}\}= det\{\mathbf{L}\}det\{\mathbf{U}\}=u_{11}u_{22}\dots u_{nn}.
+\]
+!et
+
+!split
+===== LU Decomposition, why? =====
+
+There are at least three main advantages with LU decomposition compared with standard Gaussian elimination:
+
+  * It is straightforward to compute the determinant of a matrix
+  * If we have to solve sets of linear equations with the same matrix but with different vectors $\mathbf{y}$, the number of FLOPS is of the order $n^3$.
+  * The inverse is such an operation 
+
+!split
+===== LU Decomposition, linear equations =====
+
+With the LU decomposition it is rather
+simple to solve a system of linear equations
+
+!bt
+\begin{align}
+ a_{11}x_1 +a_{12}x_2 +a_{13}x_3 + a_{14}x_4=&w_1 \nonumber \\
+a_{21}x_1 + a_{22}x_2 + a_{23}x_3 + a_{24}x_4=&w_2 \nonumber \\
+a_{31}x_1 + a_{32}x_2 + a_{33}x_3 + a_{34}x_4=&w_3 \nonumber \\
+a_{41}x_1 + a_{42}x_2 + a_{43}x_3 + a_{44}x_4=&w_4. \nonumber
+\end{align}
+!et
+
+This can be written in matrix form as
+
+!bt
+\[ \mathbf{Ax}=\mathbf{w}. \]
+!et
+
+where $\mathbf{A}$ and $\mathbf{w}$ are known and we have to solve for
+$\mathbf{x}$. Using the LU dcomposition we write
+
+!bt
+\[ \mathbf{A} \mathbf{x} \equiv \mathbf{L} \mathbf{U} \mathbf{x} =\mathbf{w}. \]
+!et
+
+!split
+===== LU Decomposition, linear equations =====
+
+The previous equation can be calculated in two steps
+
+!bt
+\[ \mathbf{L} \mathbf{y} = \mathbf{w};\qquad \mathbf{Ux}=\mathbf{y}. \]
+!et
+
+To show that this is correct we use to the LU decomposition
+to rewrite our system of linear equations as
+
+!bt
+\[  \mathbf{LUx}=\mathbf{w}, \]
+!et
+and since the determinant of $\mathbf{L}$ is equal to 1 (by construction
+since the diagonals of $\mathbf{L}$ equal 1) we can use the inverse of
+$\mathbf{L}$ to obtain
+
+!bt
+\[
+   \mathbf{Ux}=\mathbf{L^{-1}w}=\mathbf{y},
+\]
+!et
+which yields the intermediate step
+
+!bt
+\[
+   \mathbf{L^{-1}w}=\mathbf{y}
+\]
+!et
+and as soon as we have $\mathbf{y}$ we can obtain $\mathbf{x}$
+through $\mathbf{Ux}=\mathbf{y}$.
+
+!split
+===== LU Decomposition, why? =====
+
+For our four-dimentional example this takes the form
+
+!bt
+\begin{align}
+ y_1=&w_1 \nonumber\\
+l_{21}y_1 + y_2=&w_2\nonumber \\
+l_{31}y_1 + l_{32}y_2 + y_3 =&w_3\nonumber \\
+l_{41}y_1 + l_{42}y_2 + l_{43}y_3 + y_4=&w_4. \nonumber
+\end{align}
+!et
+
+and
+
+!bt
+\begin{align}
+ u_{11}x_1 +u_{12}x_2 +u_{13}x_3 + u_{14}x_4=&y_1 \nonumber\\
+u_{22}x_2 + u_{23}x_3 + u_{24}x_4=&y_2\nonumber \\
+u_{33}x_3 + u_{34}x_4=&y_3\nonumber \\
+u_{44}x_4=&y_4  \nonumber
+\end{align}
+!et
+
+This example shows the basis for the algorithm
+needed to solve the set of $n$ linear equations.
+
+!split
+===== LU Decomposition, linear equations =====
+
+The algorithm goes as follows
+
+  * Set up the matrix $\bf A$ and the vector $\bf w$ with their correct dimensions. This determines the dimensionality of the unknown vector $\bf x$.
+  * Then LU decompose the matrix $\bf A$ through a call to the function `ludcmp(double a, int n, int indx, double &d)`. This functions returns the LU decomposed matrix $\bf A$, its determinant and the vector indx which keeps track of the number of interchanges of rows. If the determinant is zero, the solution is malconditioned.
+  * Thereafter you call the function  `lubksb(double a, int n, int indx, double w)` which uses the LU decomposed matrix $\bf A$ and the vector $\bf w$ and returns $\bf x$ in the same place as $\bf w$. Upon exit the original content in $\bf w$ is destroyed. If you wish to keep this information, you should make a backup of it in your calling function.
+
+!split
+===== LU Decomposition, the inverse of a matrix =====
+
+If the inverse exists then
+
+!bt
+\[
+   \mathbf{A}^{-1}\mathbf{A}=\mathbf{I},
+\]
+!et
+the identity matrix. With an LU decomposed matrix we can rewrite the last equation as
+
+!bt
+\[
+   \mathbf{LU}\mathbf{A}^{-1}=\mathbf{I}.
+\]
+!et
+
+!split
+===== LU Decomposition, the inverse of a matrix =====
+
+If we assume that the first column (that is column 1) of the inverse matrix
+can be written as a vector with unknown entries
+
+!bt
+\[
+    \mathbf{A}_1^{-1}= \begin{bmatrix}
+
+                              a_{11}^{-1} \\
+                              a_{21}^{-1} \\
+                              \dots \\
+                              a_{n1}^{-1} \\
+                    \end{bmatrix},
+\]
+!et
+then we have a linear set of equations
+
+!bt
+\[
+    \mathbf{LU}\begin{bmatrix}
+
+                              a_{11}^{-1} \\
+                              a_{21}^{-1} \\
+                              \dots \\
+                              a_{n1}^{-1} \\
+                    \end{bmatrix} =\begin{bmatrix}
+                               1 \\
+                              0 \\
+                              \dots \\
+                              0 \\
+                    \end{bmatrix}.
+\]
+!et
+
+!split
+===== LU Decomposition, the inverse =====
+
+In a similar way we can compute the unknow entries of the second column,
+
+!bt
+\[
+    \mathbf{LU}\begin{bmatrix}
+
+                              a_{12}^{-1} \\
+                              a_{22}^{-1} \\
+                              \dots \\
+                              a_{n2}^{-1} \\
+                    \end{bmatrix}=\begin{bmatrix}
+                                0 \\
+                              1 \\
+                              \dots \\
+                              0 \\
+                    \end{bmatrix},
+\]
+!et
+and continue till we have solved all $n$ sets of linear equations.
+
+
+!split
+===== "Using Armadillo to perform an LU decomposition":"https://github.com/CompPhysics/ComputationalPhysicsMSU/blob/master/doc/Programs/CppQtCodesLectures/MatrixTest/main.cpp" =====
+!bc cppcod
+#include 
+#include "armadillo"
+using namespace arma;
+using namespace std;
+
+int main()
+  {
+   mat A = randu(5,5);
+   vec b = randu(5);
+
+  A.print("A =");
+  b.print("b=");
+  // solve Ax = b
+  vec x = solve(A,b);
+  // print x
+  x.print("x=");
+  // find LU decomp of A, if needed, P is the permutation matrix
+  mat L, U;
+  lu(L,U,A);
+  // print l
+  L.print(" L= ");
+  // print U
+  U.print(" U= ");
+  //Check that A = LU
+  (A-L*U).print("Test of LU decomposition");
+    return 0;
+  }
+!ec
+
+
+
+
diff --git a/doc/src/NeuralNet/.DS_Store b/doc/src/NeuralNet/.DS_Store
new file mode 100644
index 0000000000000000000000000000000000000000..9871ec283c6f42187332180a04bd72088cd645d2
GIT binary patch
literal 6148
zcmeHKu};G<5Pc4n+6p0cqih+OkeFGbsthbFd;qv@P!WM5nhMym@G*Q1AHxU0JD*ii
zw}=4=33Ml&pJU&<*e^~T18{@I
z1?1W7U<8jbuJQT#)p$gGQq9JrYBpk&ERiGn{(CUifIgWexi>8DQDaKfddXFA&ec${
z_X@XM4|DcECHY>Ge0$&IoAVAntgna78QpG@vBP;)VT?M4>FNmFa}wWWcF&P|TyosP
zoF|O8gwf`VJ+&*th?4x7PojG`W#octd1jY-O`g6n8x+r86Fx7^OaW8C6!@_M+_S|x
z+lp420;Yf|uv9?44>?`1h}bBGPX~)|1R%E99E^4OQ4~%dv543xvWMoBN>r+gM+~QQ
zw&yXfh}bA99WEX|T)eZ3ClnX%&ir`@hbt7VGzCn7RRy;6vM2lh@z>}7Rgztq0;a&f
zQoyx_XTt%f6!+G($;n=u(=X^^64xj$DXfI8n6a`I_vyjdo=b&TL~In
+
+
+
+
+
+
+
+Data Analysis and Machine Learning: Recurrent neural networks
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+    
+
+

+  

+    
+    Data Analysis and Machine Learning: Recurrent neural networks
+  

+
+  

+    
+  

+

+
 
+
+

+
+
 
 
 
 
+
+
+
+
+
+
+

+
Data Analysis and Machine Learning: Recurrent neural networks
  
+
+

+
+
+

+Morten Hjorth-Jensen [1, 2]
+

+
+

+
+
+
[1] Department of Physics, University of Oslo

+
[2] Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University

+

+

+
Jan 8, 2019
 
+

+

+
+
+
Read »

+
+
+
 
+
+

+
+

+
+
+
  
+
+
+
+
+
+
+
+

+ © 1999-2019, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license
+

+
+
+
+
+    
+
diff --git a/doc/src/Recurrent/._Recurrent-bs001.html b/doc/src/Recurrent/._Recurrent-bs001.html
new file mode 100644
index 000000000..e41f16735
--- /dev/null
+++ b/doc/src/Recurrent/._Recurrent-bs001.html
@@ -0,0 +1,158 @@
+
+
+
+
+
+
+
+
+Data Analysis and Machine Learning: Recurrent neural networks
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+    
+
+

+  

+    
+    Data Analysis and Machine Learning: Recurrent neural networks
+  

+
+  

+    
+  

+

+
 
+
+

+
+
 
 
 
 
+
+
+
+
+
Recurrent neural networks: Overarching view 

+
+

+Till now our focus has been, including convolutional neural networks as well, 
+on feedforward neural networks. The output or the
+activations flow only in one direction, from the input layer to the
+output layer.
+
+

+A recurrent neural network (RNN) looks very much like a feedforward
+neural network, except that it also has connections pointing
+backward.
+
+

+RNNs are used to analyze time series data such as stock prices, and
+tell you when to buy or sell. In autonomous driving systems, they can
+anticipate car trajectories and help avoid accidents. More generally,
+they can work on sequences of arbitrary lengths, rather than on
+fixed-sized inputs like all the nets we have discussed so far. For
+example, they can take sentences, documents, or audio samples as
+input, making them extremely useful for natural language processing
+systems such as automatic translation and speech-to-text.
+
+

+

+
+

+
+
+
  
+
+
+
+
+
+
+
+

+
+

+
+
+
+
+    
+
diff --git a/doc/src/Recurrent/._Recurrent-bs002.html b/doc/src/Recurrent/._Recurrent-bs002.html
new file mode 100644
index 000000000..95d697fdd
--- /dev/null
+++ b/doc/src/Recurrent/._Recurrent-bs002.html
@@ -0,0 +1,146 @@
+
+
+
+
+
+
+
+
+Data Analysis and Machine Learning: Recurrent neural networks
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+    
+
+

+  

+    
+    Data Analysis and Machine Learning: Recurrent neural networks
+  

+
+  

+    
+  

+

+
 
+
+

+
+
 
 
 
 
+
+
+
+
+
Set up of an RNN 

+
+

+The figure here displays a simple example of an RNN, with inputs \( x_t \)
+at a given time \( t \) and outputs \( y_t \).  Introducing time as a variable
+offers an intutitive way of understanding these networks. In addition
+to the inputs \( x_t \), the layer at a time \( t \) receives also as input
+the output from the previous layer \( t-1 \), that is \( y_{t1} \).
+
+

+This means also that we need to have weights that link both the inputs \( x_t \) to the outputs \( y_t \) as well as weights that link
+the output from the previous time \( y_{t-1} \) and \( y_t \).   The figure here shows an example of a simple RNN.
+
+

+
+

+
+
+
  
+
+
+
+
+
+
+
+

+
+

+
+
+
+
+    
+
diff --git a/doc/src/Recurrent/Recurrent.aux b/doc/src/Recurrent/Recurrent.aux
new file mode 100644
index 000000000..257b3fd3b
--- /dev/null
+++ b/doc/src/Recurrent/Recurrent.aux
@@ -0,0 +1,18 @@
+\relax 
+\providecommand\hyper@newdestlabel[2]{}
+\providecommand\HyperFirstAtBeginDocument{\AtBeginDocument}
+\HyperFirstAtBeginDocument{\ifx\hyper@anchor\@undefined
+\global\let\oldcontentsline\contentsline
+\gdef\contentsline#1#2#3#4{\oldcontentsline{#1}{#2}{#3}}
+\global\let\oldnewlabel\newlabel
+\gdef\newlabel#1#2{\newlabelxx{#1}#2}
+\gdef\newlabelxx#1#2#3#4#5#6{\oldnewlabel{#1}{{#2}{#3}}}
+\AtEndDocument{\ifx\hyper@anchor\@undefined
+\let\contentsline\oldcontentsline
+\let\newlabel\oldnewlabel
+\fi}
+\fi}
+\global\let\hyper@last\relax 
+\gdef\HyperFirstAtBeginDocument#1{#1}
+\providecommand\HyField@AuxAddToFields[1]{}
+\providecommand\HyField@AuxAddToCoFields[2]{}
diff --git a/doc/src/Recurrent/Recurrent.dlog b/doc/src/Recurrent/Recurrent.dlog
new file mode 100644
index 000000000..2280cf3e7
--- /dev/null
+++ b/doc/src/Recurrent/Recurrent.dlog
@@ -0,0 +1,12 @@
+translating doconce text in Recurrent.do.txt to html
+output in Recurrent-reveal.html
+translating doconce text in Recurrent.do.txt to html
+output in Recurrent-solarized.html
+translating doconce text in Recurrent.do.txt to html
+output in Recurrent.html
+translating doconce text in Recurrent.do.txt to html
+output in Recurrent-bs.html
+translating doconce text in Recurrent.do.txt to ipynb
+output in Recurrent.ipynb
+translating doconce text in Recurrent.do.txt to pdflatex
+output in Recurrent.p.tex
diff --git a/doc/src/Recurrent/Recurrent.do.txt~ b/doc/src/Recurrent/Recurrent.do.txt~
new file mode 100644
index 000000000..e37213cb8
--- /dev/null
+++ b/doc/src/Recurrent/Recurrent.do.txt~
@@ -0,0 +1,38 @@
+TITLE: Data Analysis and Machine Learning: Recurrent neural networks
+AUTHOR: Morten Hjorth-Jensen {copyright, 1999-present|CC BY-NC} at Department of Physics, University of Oslo & Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University
+DATE: today
+
+
+!split
+===== Recurrent neural networks: Overarching view =====
+
+Till now our focus has been, including convolutional neural networks as well, 
+on feedforward neural networks. The output or the
+activations flow only in one direction, from the input layer to the
+output layer.
+
+A recurrent neural network (RNN) looks very much like a feedforward
+neural network, except that it also has connections pointing
+backward. 
+
+RNNs are used to analyze time series data such as stock prices, and
+tell you when to buy or sell. In autonomous driving systems, they can
+anticipate car trajectories and help avoid accidents. More generally,
+they can work on sequences of arbitrary lengths, rather than on
+fixed-sized inputs like all the nets we have discussed so far. For
+example, they can take sentences, documents, or audio samples as
+input, making them extremely useful for natural language processing
+systems such as automatic translation and speech-to-text.
+
+
+!split
+=====  Set up of an RNN =====
+
+The figure here displays a simple example of an RNN, with inputs $x_t$
+at a given time $t$ and outputs $y_t$.  Introducing time as a variable
+offers an intutitive way of understanding these networks. In addition
+to the inputs $x_t$, the layer at a time $t$ receives also as input
+the output from the previous layer $t-1$, that is $y_{t1}$.
+
+This means also that we need to have weights that link both the inputs $x_t$ to the outputs $y_t$ as well as weights that link
+the output from the previous time $y_{t-1}$ and $y_t$.
diff --git a/doc/src/Recurrent/Recurrent.idx b/doc/src/Recurrent/Recurrent.idx
new file mode 100644
index 000000000..e69de29bb
diff --git a/doc/src/Recurrent/Recurrent.log b/doc/src/Recurrent/Recurrent.log
new file mode 100644
index 000000000..df68a93b2
--- /dev/null
+++ b/doc/src/Recurrent/Recurrent.log
@@ -0,0 +1,819 @@
+This is pdfTeX, Version 3.14159265-2.6-1.40.16 (TeX Live 2015) (preloaded format=pdflatex 2015.5.24)  8 JAN 2019 10:54
+entering extended mode
+ \write18 enabled.
+ %&-line parsing enabled.
+**Recurrent
+(./Recurrent.tex
+LaTeX2e <2015/01/01>
+Babel <3.9l> and hyphenation patterns for 79 languages loaded.
+(/usr/local/texlive/2015/texmf-dist/tex/latex/base/article.cls
+Document Class: article 2014/09/29 v1.4h Standard LaTeX document class
+(/usr/local/texlive/2015/texmf-dist/tex/latex/base/size10.clo
+File: size10.clo 2014/09/29 v1.4h Standard LaTeX file (size option)
+)
+\c@part=\count79
+\c@section=\count80
+\c@subsection=\count81
+\c@subsubsection=\count82
+\c@paragraph=\count83
+\c@subparagraph=\count84
+\c@figure=\count85
+\c@table=\count86
+\abovecaptionskip=\skip41
+\belowcaptionskip=\skip42
+\bibindent=\dimen102
+)
+(/usr/local/texlive/2015/texmf-dist/tex/latex/relsize/relsize.sty
+Package: relsize 2013/03/29 ver 4.1
+)
+(/usr/local/texlive/2015/texmf-dist/tex/latex/base/makeidx.sty
+Package: makeidx 2014/09/29 v1.0m Standard LaTeX package
+)
+(/usr/local/texlive/2015/texmf-dist/tex/latex/graphics/color.sty
+Package: color 2014/10/28 v1.1a Standard LaTeX Color (DPC)
+
+(/usr/local/texlive/2015/texmf-dist/tex/latex/latexconfig/color.cfg
+File: color.cfg 2007/01/18 v1.5 color configuration of teTeX/TeXLive
+)
+Package color Info: Driver file: pdftex.def on input line 142.
+
+(/usr/local/texlive/2015/texmf-dist/tex/latex/pdftex-def/pdftex.def
+File: pdftex.def 2011/05/27 v0.06d Graphics/color for pdfTeX
+
+(/usr/local/texlive/2015/texmf-dist/tex/generic/oberdiek/infwarerr.sty
+Package: infwarerr 2010/04/08 v1.3 Providing info/warning/error messages (HO)
+)
+(/usr/local/texlive/2015/texmf-dist/tex/generic/oberdiek/ltxcmds.sty
+Package: ltxcmds 2011/11/09 v1.22 LaTeX kernel commands for general use (HO)
+)
+\Gread@gobject=\count87
+))
+(/usr/local/texlive/2015/texmf-dist/tex/latex/setspace/setspace.sty
+Package: setspace 2011/12/19 v6.7a set line spacing
+)
+(/usr/local/texlive/2015/texmf-dist/tex/latex/amsmath/amsmath.sty
+Package: amsmath 2013/01/14 v2.14 AMS math features
+\@mathmargin=\skip43
+
+For additional information on amsmath, use the `?' option.
+(/usr/local/texlive/2015/texmf-dist/tex/latex/amsmath/amstext.sty
+Package: amstext 2000/06/29 v2.01
+
+(/usr/local/texlive/2015/texmf-dist/tex/latex/amsmath/amsgen.sty
+File: amsgen.sty 1999/11/30 v2.0
+\@emptytoks=\toks14
+\ex@=\dimen103
+))
+(/usr/local/texlive/2015/texmf-dist/tex/latex/amsmath/amsbsy.sty
+Package: amsbsy 1999/11/29 v1.2d
+\pmbraise@=\dimen104
+)
+(/usr/local/texlive/2015/texmf-dist/tex/latex/amsmath/amsopn.sty
+Package: amsopn 1999/12/14 v2.01 operator names
+)
+\inf@bad=\count88
+LaTeX Info: Redefining \frac on input line 210.
+\uproot@=\count89
+\leftroot@=\count90
+LaTeX Info: Redefining \overline on input line 306.
+\classnum@=\count91
+\DOTSCASE@=\count92
+LaTeX Info: Redefining \ldots on input line 378.
+LaTeX Info: Redefining \dots on input line 381.
+LaTeX Info: Redefining \cdots on input line 466.
+\Mathstrutbox@=\box26
+\strutbox@=\box27
+\big@size=\dimen105
+LaTeX Font Info:    Redeclaring font encoding OML on input line 566.
+LaTeX Font Info:    Redeclaring font encoding OMS on input line 567.
+\macc@depth=\count93
+\c@MaxMatrixCols=\count94
+\dotsspace@=\muskip10
+\c@parentequation=\count95
+\dspbrk@lvl=\count96
+\tag@help=\toks15
+\row@=\count97
+\column@=\count98
+\maxfields@=\count99
+\andhelp@=\toks16
+\eqnshift@=\dimen106
+\alignsep@=\dimen107
+\tagshift@=\dimen108
+\tagwidth@=\dimen109
+\totwidth@=\dimen110
+\lineht@=\dimen111
+\@envbody=\toks17
+\multlinegap=\skip44
+\multlinetaggap=\skip45
+\mathdisplay@stack=\toks18
+LaTeX Info: Redefining \[ on input line 2665.
+LaTeX Info: Redefining \] on input line 2666.
+)
+(/usr/local/texlive/2015/texmf-dist/tex/latex/amsfonts/amsfonts.sty
+Package: amsfonts 2013/01/14 v3.01 Basic AMSFonts support
+\symAMSa=\mathgroup4
+\symAMSb=\mathgroup5
+LaTeX Font Info:    Overwriting math alphabet `\mathfrak' in version `bold'
+(Font)                  U/euf/m/n --> U/euf/b/n on input line 106.
+)
+(/usr/local/texlive/2015/texmf-dist/tex/latex/amsfonts/amssymb.sty
+Package: amssymb 2013/01/14 v3.01 AMS font symbols
+)
+(/usr/local/texlive/2015/texmf-dist/tex/latex/xcolor/xcolor.sty
+Package: xcolor 2007/01/21 v2.11 LaTeX color extensions (UK)
+
+(/usr/local/texlive/2015/texmf-dist/tex/latex/latexconfig/color.cfg
+File: color.cfg 2007/01/18 v1.5 color configuration of teTeX/TeXLive
+)
+Package xcolor Info: Driver file: pdftex.def on input line 225.
+
+(/usr/local/texlive/2015/texmf-dist/tex/latex/colortbl/colortbl.sty
+Package: colortbl 2012/02/13 v1.0a Color table columns (DPC)
+
+(/usr/local/texlive/2015/texmf-dist/tex/latex/tools/array.sty
+Package: array 2014/10/28 v2.4c Tabular extension package (FMi)
+\col@sep=\dimen112
+\extrarowheight=\dimen113
+\NC@list=\toks19
+\extratabsurround=\skip46
+\backup@length=\skip47
+)
+\everycr=\toks20
+\minrowclearance=\skip48
+)
+LaTeX Info: Redefining \color on input line 702.
+\rownum=\count100
+Package xcolor Info: Model `cmy' substituted by `cmy0' on input line 1337.
+Package xcolor Info: Model `hsb' substituted by `rgb' on input line 1341.
+Package xcolor Info: Model `RGB' extended on input line 1353.
+Package xcolor Info: Model `HTML' substituted by `rgb' on input line 1355.
+Package xcolor Info: Model `Hsb' substituted by `hsb' on input line 1356.
+Package xcolor Info: Model `tHsb' substituted by `hsb' on input line 1357.
+Package xcolor Info: Model `HSB' substituted by `hsb' on input line 1358.
+Package xcolor Info: Model `Gray' substituted by `gray' on input line 1359.
+Package xcolor Info: Model `wave' substituted by `hsb' on input line 1360.
+)
+(/usr/local/texlive/2015/texmf-dist/tex/latex/tools/bm.sty
+Package: bm 2014/10/28 v1.1c Bold Symbol Support (DPC/FMi)
+\symboldoperators=\mathgroup6
+\symboldletters=\mathgroup7
+\symboldsymbols=\mathgroup8
+LaTeX Font Info:    Redeclaring math alphabet \mathbf on input line 141.
+LaTeX Info: Redefining \bm on input line 207.
+)
+(/usr/local/texlive/2015/texmf-dist/tex/latex/ltablex/ltablex.sty
+Package: ltablex 2014/08/13 v1.1 Modified tabularx
+
+(/usr/local/texlive/2015/texmf-dist/tex/latex/tools/longtable.sty
+Package: longtable 2014/10/28 v4.11 Multi-page Table package (DPC)
+\LTleft=\skip49
+\LTright=\skip50
+\LTpre=\skip51
+\LTpost=\skip52
+\LTchunksize=\count101
+\LTcapwidth=\dimen114
+\LT@head=\box28
+\LT@firsthead=\box29
+\LT@foot=\box30
+\LT@lastfoot=\box31
+\LT@cols=\count102
+\LT@rows=\count103
+\c@LT@tables=\count104
+\c@LT@chunks=\count105
+\LT@p@ftn=\toks21
+)
+(/usr/local/texlive/2015/texmf-dist/tex/latex/tools/tabularx.sty
+Package: tabularx 2014/10/28 v2.10 `tabularx' package (DPC)
+\TX@col@width=\dimen115
+\TX@old@table=\dimen116
+\TX@old@col=\dimen117
+\TX@target=\dimen118
+\TX@delta=\dimen119
+\TX@cols=\count106
+\TX@ftn=\toks22
+))
+(/usr/local/texlive/2015/texmf-dist/tex/latex/microtype/microtype.sty
+Package: microtype 2013/05/23 v2.5a Micro-typographical refinements (RS)
+
+(/usr/local/texlive/2015/texmf-dist/tex/latex/graphics/keyval.sty
+Package: keyval 2014/10/28 v1.15 key=value parser (DPC)
+\KV@toks@=\toks23
+)
+\MT@toks=\toks24
+\MT@count=\count107
+LaTeX Info: Redefining \textls on input line 766.
+\MT@outer@kern=\dimen120
+LaTeX Info: Redefining \textmicrotypecontext on input line 1285.
+\MT@listname@count=\count108
+
+(/usr/local/texlive/2015/texmf-dist/tex/latex/microtype/microtype-pdftex.def
+File: microtype-pdftex.def 2013/05/23 v2.5a Definitions specific to pdftex (RS)
+
+LaTeX Info: Redefining \lsstyle on input line 915.
+LaTeX Info: Redefining \lslig on input line 915.
+\MT@outer@space=\skip53
+)
+Package microtype Info: Loading configuration file microtype.cfg.
+
+(/usr/local/texlive/2015/texmf-dist/tex/latex/microtype/microtype.cfg
+File: microtype.cfg 2013/05/23 v2.5a microtype main configuration file (RS)
+))
+(/usr/local/texlive/2015/texmf-dist/tex/latex/graphics/graphicx.sty
+Package: graphicx 2014/10/28 v1.0g Enhanced LaTeX Graphics (DPC,SPQR)
+
+(/usr/local/texlive/2015/texmf-dist/tex/latex/graphics/graphics.sty
+Package: graphics 2014/10/28 v1.0p Standard LaTeX Graphics (DPC,SPQR)
+
+(/usr/local/texlive/2015/texmf-dist/tex/latex/graphics/trig.sty
+Package: trig 1999/03/16 v1.09 sin cos tan (DPC)
+)
+(/usr/local/texlive/2015/texmf-dist/tex/latex/latexconfig/graphics.cfg
+File: graphics.cfg 2010/04/23 v1.9 graphics configuration of TeX Live
+)
+Package graphics Info: Driver file: pdftex.def on input line 94.
+)
+\Gin@req@height=\dimen121
+\Gin@req@width=\dimen122
+)
+(/usr/local/texlive/2015/texmf-dist/tex/latex/base/fontenc.sty
+Package: fontenc 2005/09/27 v1.99g Standard LaTeX package
+
+(/usr/local/texlive/2015/texmf-dist/tex/latex/base/t1enc.def
+File: t1enc.def 2005/09/27 v1.99g Standard LaTeX file
+LaTeX Font Info:    Redeclaring font encoding T1 on input line 48.
+))
+(/usr/local/texlive/2015/texmf-dist/tex/latex/ucs/ucs.sty
+Package: ucs 2013/05/11 v2.2 UCS: Unicode input support
+
+(/usr/local/texlive/2015/texmf-dist/tex/latex/ucs/data/uni-global.def
+File: uni-global.def 2013/05/13 UCS: Unicode global data
+)
+\uc@secondtry=\count109
+\uc@combtoks=\toks25
+\uc@combtoksb=\toks26
+\uc@temptokena=\toks27
+)
+(/usr/local/texlive/2015/texmf-dist/tex/latex/base/inputenc.sty
+Package: inputenc 2015/03/17 v1.2c Input encoding file
+\inpenc@prehook=\toks28
+\inpenc@posthook=\toks29
+
+(/usr/local/texlive/2015/texmf-dist/tex/latex/ucs/utf8x.def
+File: utf8x.def 2004/10/17 UCS: Input encoding UTF-8
+))
+(/usr/local/texlive/2015/texmf-dist/tex/latex/lm/lmodern.sty
+Package: lmodern 2009/10/30 v1.6 Latin Modern Fonts
+LaTeX Font Info:    Overwriting symbol font `operators' in version `normal'
+(Font)                  OT1/cmr/m/n --> OT1/lmr/m/n on input line 22.
+LaTeX Font Info:    Overwriting symbol font `letters' in version `normal'
+(Font)                  OML/cmm/m/it --> OML/lmm/m/it on input line 23.
+LaTeX Font Info:    Overwriting symbol font `symbols' in version `normal'
+(Font)                  OMS/cmsy/m/n --> OMS/lmsy/m/n on input line 24.
+LaTeX Font Info:    Overwriting symbol font `largesymbols' in version `normal'
+(Font)                  OMX/cmex/m/n --> OMX/lmex/m/n on input line 25.
+LaTeX Font Info:    Overwriting symbol font `operators' in version `bold'
+(Font)                  OT1/cmr/bx/n --> OT1/lmr/bx/n on input line 26.
+LaTeX Font Info:    Overwriting symbol font `letters' in version `bold'
+(Font)                  OML/cmm/b/it --> OML/lmm/b/it on input line 27.
+LaTeX Font Info:    Overwriting symbol font `symbols' in version `bold'
+(Font)                  OMS/cmsy/b/n --> OMS/lmsy/b/n on input line 28.
+LaTeX Font Info:    Overwriting symbol font `largesymbols' in version `bold'
+(Font)                  OMX/cmex/m/n --> OMX/lmex/m/n on input line 29.
+LaTeX Font Info:    Overwriting math alphabet `\mathbf' in version `normal'
+(Font)                  OT1/cmr/bx/n --> OT1/lmr/bx/n on input line 31.
+LaTeX Font Info:    Overwriting math alphabet `\mathsf' in version `normal'
+(Font)                  OT1/cmss/m/n --> OT1/lmss/m/n on input line 32.
+LaTeX Font Info:    Overwriting math alphabet `\mathit' in version `normal'
+(Font)                  OT1/cmr/m/it --> OT1/lmr/m/it on input line 33.
+LaTeX Font Info:    Overwriting math alphabet `\mathtt' in version `normal'
+(Font)                  OT1/cmtt/m/n --> OT1/lmtt/m/n on input line 34.
+LaTeX Font Info:    Overwriting math alphabet `\mathbf' in version `bold'
+(Font)                  OT1/cmr/bx/n --> OT1/lmr/bx/n on input line 35.
+LaTeX Font Info:    Overwriting math alphabet `\mathsf' in version `bold'
+(Font)                  OT1/cmss/bx/n --> OT1/lmss/bx/n on input line 36.
+LaTeX Font Info:    Overwriting math alphabet `\mathit' in version `bold'
+(Font)                  OT1/cmr/bx/it --> OT1/lmr/bx/it on input line 37.
+LaTeX Font Info:    Overwriting math alphabet `\mathtt' in version `bold'
+(Font)                  OT1/cmtt/m/n --> OT1/lmtt/m/n on input line 38.
+)
+(/usr/local/texlive/2015/texmf-dist/tex/latex/hyperref/hyperref.sty
+Package: hyperref 2012/11/06 v6.83m Hypertext links for LaTeX
+
+(/usr/local/texlive/2015/texmf-dist/tex/generic/oberdiek/hobsub-hyperref.sty
+Package: hobsub-hyperref 2012/05/28 v1.13 Bundle oberdiek, subset hyperref (HO)
+
+
+(/usr/local/texlive/2015/texmf-dist/tex/generic/oberdiek/hobsub-generic.sty
+Package: hobsub-generic 2012/05/28 v1.13 Bundle oberdiek, subset generic (HO)
+Package: hobsub 2012/05/28 v1.13 Construct package bundles (HO)
+Package hobsub Info: Skipping package `infwarerr' (already loaded).
+Package hobsub Info: Skipping package `ltxcmds' (already loaded).
+Package: ifluatex 2010/03/01 v1.3 Provides the ifluatex switch (HO)
+Package ifluatex Info: LuaTeX not detected.
+Package: ifvtex 2010/03/01 v1.5 Detect VTeX and its facilities (HO)
+Package ifvtex Info: VTeX not detected.
+Package: intcalc 2007/09/27 v1.1 Expandable calculations with integers (HO)
+Package: ifpdf 2011/01/30 v2.3 Provides the ifpdf switch (HO)
+Package ifpdf Info: pdfTeX in PDF mode is detected.
+Package: etexcmds 2011/02/16 v1.5 Avoid name clashes with e-TeX commands (HO)
+Package etexcmds Info: Could not find \expanded.
+(etexcmds)             That can mean that you are not using pdfTeX 1.50 or
+(etexcmds)             that some package has redefined \expanded.
+(etexcmds)             In the latter case, load this package earlier.
+Package: kvsetkeys 2012/04/25 v1.16 Key value parser (HO)
+Package: kvdefinekeys 2011/04/07 v1.3 Define keys (HO)
+Package: pdftexcmds 2011/11/29 v0.20 Utility functions of pdfTeX for LuaTeX (HO
+)
+Package pdftexcmds Info: LuaTeX not detected.
+Package pdftexcmds Info: \pdf@primitive is available.
+Package pdftexcmds Info: \pdf@ifprimitive is available.
+Package pdftexcmds Info: \pdfdraftmode found.
+Package: pdfescape 2011/11/25 v1.13 Implements pdfTeX's escape features (HO)
+Package: bigintcalc 2012/04/08 v1.3 Expandable calculations on big integers (HO
+)
+Package: bitset 2011/01/30 v1.1 Handle bit-vector datatype (HO)
+Package: uniquecounter 2011/01/30 v1.2 Provide unlimited unique counter (HO)
+)
+Package hobsub Info: Skipping package `hobsub' (already loaded).
+Package: letltxmacro 2010/09/02 v1.4 Let assignment for LaTeX macros (HO)
+Package: hopatch 2012/05/28 v1.2 Wrapper for package hooks (HO)
+Package: xcolor-patch 2011/01/30 xcolor patch
+Package: atveryend 2011/06/30 v1.8 Hooks at the very end of document (HO)
+Package atveryend Info: \enddocument detected (standard20110627).
+Package: atbegshi 2011/10/05 v1.16 At begin shipout hook (HO)
+Package: refcount 2011/10/16 v3.4 Data extraction from label references (HO)
+Package: hycolor 2011/01/30 v1.7 Color options for hyperref/bookmark (HO)
+)
+(/usr/local/texlive/2015/texmf-dist/tex/generic/ifxetex/ifxetex.sty
+Package: ifxetex 2010/09/12 v0.6 Provides ifxetex conditional
+)
+(/usr/local/texlive/2015/texmf-dist/tex/latex/oberdiek/auxhook.sty
+Package: auxhook 2011/03/04 v1.3 Hooks for auxiliary files (HO)
+)
+(/usr/local/texlive/2015/texmf-dist/tex/latex/oberdiek/kvoptions.sty
+Package: kvoptions 2011/06/30 v3.11 Key value format for package options (HO)
+)
+\@linkdim=\dimen123
+\Hy@linkcounter=\count110
+\Hy@pagecounter=\count111
+
+(/usr/local/texlive/2015/texmf-dist/tex/latex/hyperref/pd1enc.def
+File: pd1enc.def 2012/11/06 v6.83m Hyperref: PDFDocEncoding definition (HO)
+)
+\Hy@SavedSpaceFactor=\count112
+
+(/usr/local/texlive/2015/texmf-dist/tex/latex/latexconfig/hyperref.cfg
+File: hyperref.cfg 2002/06/06 v1.2 hyperref configuration of TeXLive
+)
+Package hyperref Info: Option `final' set `true' on input line 4319.
+Package hyperref Info: Hyper figures OFF on input line 4443.
+Package hyperref Info: Link nesting OFF on input line 4448.
+Package hyperref Info: Hyper index ON on input line 4451.
+Package hyperref Info: Plain pages OFF on input line 4458.
+Package hyperref Info: Backreferencing OFF on input line 4463.
+Package hyperref Info: Implicit mode ON; LaTeX internals redefined.
+Package hyperref Info: Bookmarks ON on input line 4688.
+\c@Hy@tempcnt=\count113
+
+(/usr/local/texlive/2015/texmf-dist/tex/latex/url/url.sty
+\Urlmuskip=\muskip11
+Package: url 2013/09/16  ver 3.4  Verb mode for urls, etc.
+)
+LaTeX Info: Redefining \url on input line 5041.
+\XeTeXLinkMargin=\dimen124
+\Fld@menulength=\count114
+\Field@Width=\dimen125
+\Fld@charsize=\dimen126
+Package hyperref Info: Hyper figures OFF on input line 6295.
+Package hyperref Info: Link nesting OFF on input line 6300.
+Package hyperref Info: Hyper index ON on input line 6303.
+Package hyperref Info: backreferencing OFF on input line 6310.
+Package hyperref Info: Link coloring OFF on input line 6315.
+Package hyperref Info: Link coloring with OCG OFF on input line 6320.
+Package hyperref Info: PDF/A mode OFF on input line 6325.
+LaTeX Info: Redefining \ref on input line 6365.
+LaTeX Info: Redefining \pageref on input line 6369.
+\Hy@abspage=\count115
+\c@Item=\count116
+\c@Hfootnote=\count117
+)
+
+Package hyperref Message: Driver (autodetected): hpdftex.
+
+(/usr/local/texlive/2015/texmf-dist/tex/latex/hyperref/hpdftex.def
+File: hpdftex.def 2012/11/06 v6.83m Hyperref driver for pdfTeX
+\Fld@listcount=\count118
+\c@bookmark@seq@number=\count119
+
+(/usr/local/texlive/2015/texmf-dist/tex/latex/oberdiek/rerunfilecheck.sty
+Package: rerunfilecheck 2011/04/15 v1.7 Rerun checks for auxiliary files (HO)
+Package uniquecounter Info: New unique counter `rerunfilecheck' on input line 2
+82.
+)
+\Hy@SectionHShift=\skip54
+)
+Package hyperref Info: Option `breaklinks' set `true' on input line 44.
+Package hyperref Info: Option `colorlinks' set `true' on input line 44.
+Package hyperref Info: Option `pdfmenubar' set `true' on input line 44.
+Package hyperref Info: Option `pdftoolbar' set `true' on input line 44.
+
+(/usr/local/texlive/2015/texmf-dist/tex/latex/fancyhdr/fancyhdr.sty
+\fancy@headwidth=\skip55
+\f@ncyO@elh=\skip56
+\f@ncyO@erh=\skip57
+\f@ncyO@olh=\skip58
+\f@ncyO@orh=\skip59
+\f@ncyO@elf=\skip60
+\f@ncyO@erf=\skip61
+\f@ncyO@olf=\skip62
+\f@ncyO@orf=\skip63
+)
+
+Package Fancyhdr Warning: \fancyfoot's `E' option without twoside option is use
+less on input line 53.
+
+\@indexfile=\write3
+\openout3 = `Recurrent.idx'.
+
+Writing index file Recurrent.idx
+(/usr/local/texlive/2015/texmf-dist/tex/latex/idxlayout/idxlayout.sty
+Package: idxlayout 2012/03/30 v0.4d Configurable index layout
+
+(/usr/local/texlive/2015/texmf-dist/tex/latex/etoolbox/etoolbox.sty
+Package: etoolbox 2015/05/04 v2.2 e-TeX tools for LaTeX (JAW)
+\etb@tempcnta=\count120
+)
+(/usr/local/texlive/2015/texmf-dist/tex/latex/tools/multicol.sty
+Package: multicol 2015/03/31 v1.8m multicolumn formatting (FMi)
+\c@tracingmulticols=\count121
+\mult@box=\box32
+\multicol@leftmargin=\dimen127
+\c@unbalance=\count122
+\c@collectmore=\count123
+\doublecol@number=\count124
+\multicoltolerance=\count125
+\multicolpretolerance=\count126
+\full@width=\dimen128
+\page@free=\dimen129
+\premulticols=\dimen130
+\postmulticols=\dimen131
+\multicolsep=\skip64
+\multicolbaselineskip=\skip65
+\partial@page=\box33
+\last@line=\box34
+\maxbalancingoverflow=\dimen132
+\mult@rightbox=\box35
+\mult@grightbox=\box36
+\mult@gfirstbox=\box37
+\mult@firstbox=\box38
+\@tempa=\box39
+\@tempa=\box40
+\@tempa=\box41
+\@tempa=\box42
+\@tempa=\box43
+\@tempa=\box44
+\@tempa=\box45
+\@tempa=\box46
+\@tempa=\box47
+\@tempa=\box48
+\@tempa=\box49
+\@tempa=\box50
+\@tempa=\box51
+\@tempa=\box52
+\@tempa=\box53
+\@tempa=\box54
+\@tempa=\box55
+\c@columnbadness=\count127
+\c@finalcolumnbadness=\count128
+\last@try=\dimen133
+\multicolovershoot=\dimen134
+\multicolundershoot=\dimen135
+\mult@nat@firstbox=\box56
+\colbreak@box=\box57
+\mc@col@check@num=\count129
+)
+\c@idxcols=\count130
+\indexcolsep=\skip66
+\indexrule=\skip67
+\ila@indentunit=\skip68
+\ila@initsep=\skip69
+)
+(/usr/local/texlive/2015/texmf-dist/tex/latex/tocbibind/tocbibind.sty
+Package: tocbibind 2010/10/13 v1.5k extra ToC listings
+Package tocbibind Info: The document has section divisions on input line 50.
+
+
+Package tocbibind Note: Using section or other style headings.
+
+) (/usr/local/texlive/2015/texmf-dist/tex/latex/ms/ragged2e.sty
+Package: ragged2e 2009/05/21 v2.1 ragged2e Package (MS)
+
+(/usr/local/texlive/2015/texmf-dist/tex/latex/ms/everysel.sty
+Package: everysel 2011/10/28 v1.2 EverySelectfont Package (MS)
+)
+\CenteringLeftskip=\skip70
+\RaggedLeftLeftskip=\skip71
+\RaggedRightLeftskip=\skip72
+\CenteringRightskip=\skip73
+\RaggedLeftRightskip=\skip74
+\RaggedRightRightskip=\skip75
+\CenteringParfillskip=\skip76
+\RaggedLeftParfillskip=\skip77
+\RaggedRightParfillskip=\skip78
+\JustifyingParfillskip=\skip79
+\CenteringParindent=\skip80
+\RaggedLeftParindent=\skip81
+\RaggedRightParindent=\skip82
+\JustifyingParindent=\skip83
+)
+(./Recurrent.aux)
+\openout1 = `Recurrent.aux'.
+
+LaTeX Font Info:    Checking defaults for OML/cmm/m/it on input line 88.
+LaTeX Font Info:    ... okay on input line 88.
+LaTeX Font Info:    Checking defaults for T1/cmr/m/n on input line 88.
+LaTeX Font Info:    ... okay on input line 88.
+LaTeX Font Info:    Checking defaults for OT1/cmr/m/n on input line 88.
+LaTeX Font Info:    ... okay on input line 88.
+LaTeX Font Info:    Checking defaults for OMS/cmsy/m/n on input line 88.
+LaTeX Font Info:    ... okay on input line 88.
+LaTeX Font Info:    Checking defaults for OMX/cmex/m/n on input line 88.
+LaTeX Font Info:    ... okay on input line 88.
+LaTeX Font Info:    Checking defaults for U/cmr/m/n on input line 88.
+LaTeX Font Info:    ... okay on input line 88.
+LaTeX Font Info:    Checking defaults for PD1/pdf/m/n on input line 88.
+LaTeX Font Info:    ... okay on input line 88.
+LaTeX Font Info:    Try loading font information for T1+lmr on input line 88.
+ (/usr/local/texlive/2015/texmf-dist/tex/latex/lm/t1lmr.fd
+File: t1lmr.fd 2009/10/30 v1.6 Font defs for Latin Modern
+)
+(/usr/local/texlive/2015/texmf-dist/tex/context/base/supp-pdf.mkii
+[Loading MPS to PDF converter (version 2006.09.02).]
+\scratchcounter=\count131
+\scratchdimen=\dimen136
+\scratchbox=\box58
+\nofMPsegments=\count132
+\nofMParguments=\count133
+\everyMPshowfont=\toks30
+\MPscratchCnt=\count134
+\MPscratchDim=\dimen137
+\MPnumerator=\count135
+\makeMPintoPDFobject=\count136
+\everyMPtoPDFconversion=\toks31
+)
+LaTeX Info: Redefining \microtypecontext on input line 88.
+Package microtype Info: Generating PDF output.
+Package microtype Info: Character protrusion enabled (level 2).
+Package microtype Info: Using default protrusion set `alltext'.
+Package microtype Info: Automatic font expansion enabled (level 2),
+(microtype)             stretch: 20, shrink: 20, step: 1, non-selected.
+Package microtype Info: Using default expansion set `basictext'.
+Package microtype Info: No adjustment of tracking.
+Package microtype Info: No adjustment of interword spacing.
+Package microtype Info: No adjustment of character kerning.
+ (/usr/local/texlive/2015/texmf-dist/tex/latex/microtype/mt-cmr.cfg
+File: mt-cmr.cfg 2013/05/19 v2.2 microtype config. file: Computer Modern Roman 
+(RS)
+)
+(/usr/local/texlive/2015/texmf-dist/tex/latex/oberdiek/epstopdf-base.sty
+Package: epstopdf-base 2010/02/09 v2.5 Base part for package epstopdf
+
+(/usr/local/texlive/2015/texmf-dist/tex/latex/oberdiek/grfext.sty
+Package: grfext 2010/08/19 v1.1 Manage graphics extensions (HO)
+)
+Package grfext Info: Graphics extension search list:
+(grfext)             [.png,.pdf,.jpg,.mps,.jpeg,.jbig2,.jb2,.PNG,.PDF,.JPG,.JPE
+G,.JBIG2,.JB2,.eps]
+(grfext)             \AppendGraphicsExtensions on input line 452.
+
+(/usr/local/texlive/2015/texmf-dist/tex/latex/latexconfig/epstopdf-sys.cfg
+File: epstopdf-sys.cfg 2010/07/13 v1.3 Configuration of (r)epstopdf for TeX Liv
+e
+))
+(/usr/local/texlive/2015/texmf-dist/tex/latex/ucs/ucsencs.def
+File: ucsencs.def 2011/01/21 Fixes to fontencodings LGR, T3
+)
+\AtBeginShipoutBox=\box59
+Package hyperref Info: Link coloring ON on input line 88.
+
+(/usr/local/texlive/2015/texmf-dist/tex/latex/hyperref/nameref.sty
+Package: nameref 2012/10/27 v2.43 Cross-referencing by name of section
+
+(/usr/local/texlive/2015/texmf-dist/tex/generic/oberdiek/gettitlestring.sty
+Package: gettitlestring 2010/12/03 v1.4 Cleanup title references (HO)
+)
+\c@section@level=\count137
+)
+LaTeX Info: Redefining \ref on input line 88.
+LaTeX Info: Redefining \pageref on input line 88.
+LaTeX Info: Redefining \nameref on input line 88.
+
+(./Recurrent.out) (./Recurrent.out)
+\@outlinefile=\write4
+\openout4 = `Recurrent.out'.
+
+ ABD: EverySelectfont initializing macros
+LaTeX Info: Redefining \selectfont on input line 88.
+LaTeX Font Info:    Try loading font information for OT1+lmr on input line 114.
+
+
+(/usr/local/texlive/2015/texmf-dist/tex/latex/lm/ot1lmr.fd
+File: ot1lmr.fd 2009/10/30 v1.6 Font defs for Latin Modern
+)
+LaTeX Font Info:    Try loading font information for OML+lmm on input line 114.
+
+
+(/usr/local/texlive/2015/texmf-dist/tex/latex/lm/omllmm.fd
+File: omllmm.fd 2009/10/30 v1.6 Font defs for Latin Modern
+)
+LaTeX Font Info:    Try loading font information for OMS+lmsy on input line 114
+.
+
+(/usr/local/texlive/2015/texmf-dist/tex/latex/lm/omslmsy.fd
+File: omslmsy.fd 2009/10/30 v1.6 Font defs for Latin Modern
+)
+LaTeX Font Info:    Try loading font information for OMX+lmex on input line 114
+.
+
+(/usr/local/texlive/2015/texmf-dist/tex/latex/lm/omxlmex.fd
+File: omxlmex.fd 2009/10/30 v1.6 Font defs for Latin Modern
+)
+LaTeX Font Info:    External font `lmex10' loaded for size
+(Font)              <10> on input line 114.
+LaTeX Font Info:    External font `lmex10' loaded for size
+(Font)              <7> on input line 114.
+LaTeX Font Info:    External font `lmex10' loaded for size
+(Font)              <5> on input line 114.
+LaTeX Font Info:    Try loading font information for U+msa on input line 114.
+
+(/usr/local/texlive/2015/texmf-dist/tex/latex/amsfonts/umsa.fd
+File: umsa.fd 2013/01/14 v3.01 AMS symbols A
+)
+(/usr/local/texlive/2015/texmf-dist/tex/latex/microtype/mt-msa.cfg
+File: mt-msa.cfg 2006/02/04 v1.1 microtype config. file: AMS symbols (a) (RS)
+)
+LaTeX Font Info:    Try loading font information for U+msb on input line 114.
+
+(/usr/local/texlive/2015/texmf-dist/tex/latex/amsfonts/umsb.fd
+File: umsb.fd 2013/01/14 v3.01 AMS symbols B
+)
+(/usr/local/texlive/2015/texmf-dist/tex/latex/microtype/mt-msb.cfg
+File: mt-msb.cfg 2005/06/01 v1.0 microtype config. file: AMS symbols (b) (RS)
+)
+LaTeX Font Info:    External font `lmex10' loaded for size
+(Font)              <9> on input line 119.
+LaTeX Font Info:    External font `lmex10' loaded for size
+(Font)              <6> on input line 119.
+Package atveryend Info: Empty hook `BeforeClearDocument' on input line 170.
+LaTeX Font Info:    Try loading font information for OMS+lmr on input line 170.
+
+
+(/usr/local/texlive/2015/texmf-dist/tex/latex/lm/omslmr.fd
+File: omslmr.fd 2009/10/30 v1.6 Font defs for Latin Modern
+)
+LaTeX Font Info:    Font shape `OMS/lmr/m/n' in size <8> not available
+(Font)              Font shape `OMS/lmsy/m/n' tried instead on input line 170.
+ [1
+
+{/usr/local/texlive/2015/texmf-var/fonts/map/pdftex/updmap/pdftex.map}]
+Package atveryend Info: Empty hook `AfterLastShipout' on input line 170.
+ (./Recurrent.aux)
+Package atveryend Info: Executing hook `AtVeryEndDocument' on input line 170.
+
+
+ *File List*
+ article.cls    2014/09/29 v1.4h Standard LaTeX document class
+  size10.clo    2014/09/29 v1.4h Standard LaTeX file (size option)
+ relsize.sty    2013/03/29 ver 4.1
+ makeidx.sty    2014/09/29 v1.0m Standard LaTeX package
+   color.sty    1999/02/16
+   color.cfg    2007/01/18 v1.5 color configuration of teTeX/TeXLive
+  pdftex.def    2011/05/27 v0.06d Graphics/color for pdfTeX
+infwarerr.sty    2010/04/08 v1.3 Providing info/warning/error messages (HO)
+ ltxcmds.sty    2011/11/09 v1.22 LaTeX kernel commands for general use (HO)
+setspace.sty    2011/12/19 v6.7a set line spacing
+ amsmath.sty    2013/01/14 v2.14 AMS math features
+ amstext.sty    2000/06/29 v2.01
+  amsgen.sty    1999/11/30 v2.0
+  amsbsy.sty    1999/11/29 v1.2d
+  amsopn.sty    1999/12/14 v2.01 operator names
+amsfonts.sty    2013/01/14 v3.01 Basic AMSFonts support
+ amssymb.sty    2013/01/14 v3.01 AMS font symbols
+  xcolor.sty    2007/01/21 v2.11 LaTeX color extensions (UK)
+   color.cfg    2007/01/18 v1.5 color configuration of teTeX/TeXLive
+colortbl.sty    2012/02/13 v1.0a Color table columns (DPC)
+   array.sty    2014/10/28 v2.4c Tabular extension package (FMi)
+      bm.sty    2014/10/28 v1.1c Bold Symbol Support (DPC/FMi)
+ ltablex.sty    2014/08/13 v1.1 Modified tabularx
+longtable.sty    2014/10/28 v4.11 Multi-page Table package (DPC)
+tabularx.sty    2014/10/28 v2.10 `tabularx' package (DPC)
+microtype.sty    2013/05/23 v2.5a Micro-typographical refinements (RS)
+  keyval.sty    2014/10/28 v1.15 key=value parser (DPC)
+microtype-pdftex.def    2013/05/23 v2.5a Definitions specific to pdftex (RS)
+microtype.cfg    2013/05/23 v2.5a microtype main configuration file (RS)
+graphicx.sty    2014/10/28 v1.0g Enhanced LaTeX Graphics (DPC,SPQR)
+graphics.sty    2014/10/28 v1.0p Standard LaTeX Graphics (DPC,SPQR)
+    trig.sty    1999/03/16 v1.09 sin cos tan (DPC)
+graphics.cfg    2010/04/23 v1.9 graphics configuration of TeX Live
+ fontenc.sty
+   t1enc.def    2005/09/27 v1.99g Standard LaTeX file
+     ucs.sty    2013/05/11 v2.2 UCS: Unicode input support
+uni-global.def    2013/05/13 UCS: Unicode global data
+inputenc.sty    2015/03/17 v1.2c Input encoding file
+   utf8x.def    2004/10/17 UCS: Input encoding UTF-8
+ lmodern.sty    2009/10/30 v1.6 Latin Modern Fonts
+hyperref.sty    2012/11/06 v6.83m Hypertext links for LaTeX
+hobsub-hyperref.sty    2012/05/28 v1.13 Bundle oberdiek, subset hyperref (HO)
+hobsub-generic.sty    2012/05/28 v1.13 Bundle oberdiek, subset generic (HO)
+  hobsub.sty    2012/05/28 v1.13 Construct package bundles (HO)
+ifluatex.sty    2010/03/01 v1.3 Provides the ifluatex switch (HO)
+  ifvtex.sty    2010/03/01 v1.5 Detect VTeX and its facilities (HO)
+ intcalc.sty    2007/09/27 v1.1 Expandable calculations with integers (HO)
+   ifpdf.sty    2011/01/30 v2.3 Provides the ifpdf switch (HO)
+etexcmds.sty    2011/02/16 v1.5 Avoid name clashes with e-TeX commands (HO)
+kvsetkeys.sty    2012/04/25 v1.16 Key value parser (HO)
+kvdefinekeys.sty    2011/04/07 v1.3 Define keys (HO)
+pdftexcmds.sty    2011/11/29 v0.20 Utility functions of pdfTeX for LuaTeX (HO)
+pdfescape.sty    2011/11/25 v1.13 Implements pdfTeX's escape features (HO)
+bigintcalc.sty    2012/04/08 v1.3 Expandable calculations on big integers (HO)
+  bitset.sty    2011/01/30 v1.1 Handle bit-vector datatype (HO)
+uniquecounter.sty    2011/01/30 v1.2 Provide unlimited unique counter (HO)
+letltxmacro.sty    2010/09/02 v1.4 Let assignment for LaTeX macros (HO)
+ hopatch.sty    2012/05/28 v1.2 Wrapper for package hooks (HO)
+xcolor-patch.sty    2011/01/30 xcolor patch
+atveryend.sty    2011/06/30 v1.8 Hooks at the very end of document (HO)
+atbegshi.sty    2011/10/05 v1.16 At begin shipout hook (HO)
+refcount.sty    2011/10/16 v3.4 Data extraction from label references (HO)
+ hycolor.sty    2011/01/30 v1.7 Color options for hyperref/bookmark (HO)
+ ifxetex.sty    2010/09/12 v0.6 Provides ifxetex conditional
+ auxhook.sty    2011/03/04 v1.3 Hooks for auxiliary files (HO)
+kvoptions.sty    2011/06/30 v3.11 Key value format for package options (HO)
+  pd1enc.def    2012/11/06 v6.83m Hyperref: PDFDocEncoding definition (HO)
+hyperref.cfg    2002/06/06 v1.2 hyperref configuration of TeXLive
+     url.sty    2013/09/16  ver 3.4  Verb mode for urls, etc.
+ hpdftex.def    2012/11/06 v6.83m Hyperref driver for pdfTeX
+rerunfilecheck.sty    2011/04/15 v1.7 Rerun checks for auxiliary files (HO)
+fancyhdr.sty    
+idxlayout.sty    2012/03/30 v0.4d Configurable index layout
+etoolbox.sty    2015/05/04 v2.2 e-TeX tools for LaTeX (JAW)
+multicol.sty    2015/03/31 v1.8m multicolumn formatting (FMi)
+tocbibind.sty    2010/10/13 v1.5k extra ToC listings
+ragged2e.sty    2009/05/21 v2.1 ragged2e Package (MS)
+everysel.sty    2011/10/28 v1.2 EverySelectfont Package (MS)
+   t1lmr.fd    2009/10/30 v1.6 Font defs for Latin Modern
+supp-pdf.mkii
+  mt-cmr.cfg    2013/05/19 v2.2 microtype config. file: Computer Modern Roman (
+RS)
+epstopdf-base.sty    2010/02/09 v2.5 Base part for package epstopdf
+  grfext.sty    2010/08/19 v1.1 Manage graphics extensions (HO)
+epstopdf-sys.cfg    2010/07/13 v1.3 Configuration of (r)epstopdf for TeX Live
+ ucsencs.def    2011/01/21 Fixes to fontencodings LGR, T3
+ nameref.sty    2012/10/27 v2.43 Cross-referencing by name of section
+gettitlestring.sty    2010/12/03 v1.4 Cleanup title references (HO)
+Recurrent.out
+Recurrent.out
+  ot1lmr.fd    2009/10/30 v1.6 Font defs for Latin Modern
+  omllmm.fd    2009/10/30 v1.6 Font defs for Latin Modern
+ omslmsy.fd    2009/10/30 v1.6 Font defs for Latin Modern
+ omxlmex.fd    2009/10/30 v1.6 Font defs for Latin Modern
+    umsa.fd    2013/01/14 v3.01 AMS symbols A
+  mt-msa.cfg    2006/02/04 v1.1 microtype config. file: AMS symbols (a) (RS)
+    umsb.fd    2013/01/14 v3.01 AMS symbols B
+  mt-msb.cfg    2005/06/01 v1.0 microtype config. file: AMS symbols (b) (RS)
+  omslmr.fd    2009/10/30 v1.6 Font defs for Latin Modern
+ ***********
+
+Package atveryend Info: Executing hook `AtEndAfterFileList' on input line 170.
+Package rerunfilecheck Info: File `Recurrent.out' has not changed.
+(rerunfilecheck)             Checksum: D41D8CD98F00B204E9800998ECF8427E;0.
+Package atveryend Info: Empty hook `AtVeryVeryEnd' on input line 170.
+ ) 
+Here is how much of TeX's memory you used:
+ 9683 strings out of 493089
+ 145050 string characters out of 6134842
+ 281360 words of memory out of 5000000
+ 12916 multiletter control sequences out of 15000+600000
+ 55018 words of font info for 82 fonts, out of 8000000 for 9000
+ 1141 hyphenation exceptions out of 8191
+ 41i,14n,43p,1002b,389s stack positions out of 5000i,500n,10000p,200000b,80000s
+{/usr/local/texlive/2015/texmf-dist/fonts/enc/dvips/lm/lm-mathsy.enc}{/usr/lo
+cal/texlive/2015/texmf-dist/fonts/enc/dvips/lm/lm-ec.enc}{/usr/local/texlive/20
+15/texmf-dist/fonts/enc/dvips/lm/lm-rm.enc}{/usr/local/texlive/2015/texmf-dist/
+fonts/enc/dvips/lm/lm-mathit.enc}
+Output written on Recurrent.pdf (1 page, 182262 bytes).
+PDF statistics:
+ 69 PDF objects out of 1000 (max. 8388607)
+ 53 compressed objects within 1 object stream
+ 4 named destinations out of 1000 (max. 500000)
+ 18433 words of extra memory for PDF output out of 20736 (max. 10000000)
+
diff --git a/doc/src/Recurrent/Recurrent.out b/doc/src/Recurrent/Recurrent.out
new file mode 100644
index 000000000..e69de29bb
diff --git a/doc/src/Recurrent/Recurrent.tex.old~~ b/doc/src/Recurrent/Recurrent.tex.old~~
new file mode 100644
index 000000000..889367f59
--- /dev/null
+++ b/doc/src/Recurrent/Recurrent.tex.old~~
@@ -0,0 +1,171 @@
+%%
+%% Automatically generated file from DocOnce source
+%% (https://github.com/hplgit/doconce/)
+%%
+%%
+
+
+%-------------------- begin preamble ----------------------
+
+\documentclass[%
+oneside,                 % oneside: electronic viewing, twoside: printing
+final,                   % draft: marks overfull hboxes, figures with paths
+10pt]{article}
+
+\listfiles               %  print all files needed to compile this document
+
+\usepackage{relsize,makeidx,color,setspace,amsmath,amsfonts,amssymb}
+\usepackage[table]{xcolor}
+\usepackage{bm,ltablex,microtype}
+
+\usepackage[pdftex]{graphicx}
+
+\usepackage[T1]{fontenc}
+%\usepackage[latin1]{inputenc}
+\usepackage{ucs}
+\usepackage[utf8x]{inputenc}
+
+\usepackage{lmodern}         % Latin Modern fonts derived from Computer Modern
+
+% Hyperlinks in PDF:
+\definecolor{linkcolor}{rgb}{0,0,0.4}
+\usepackage{hyperref}
+\hypersetup{
+    breaklinks=true,
+    colorlinks=true,
+    linkcolor=linkcolor,
+    urlcolor=linkcolor,
+    citecolor=black,
+    filecolor=black,
+    %filecolor=blue,
+    pdfmenubar=true,
+    pdftoolbar=true,
+    bookmarksdepth=3   % Uncomment (and tweak) for PDF bookmarks with more levels than the TOC
+    }
+%\hyperbaseurl{}   % hyperlinks are relative to this root
+
+\setcounter{tocdepth}{2}  % levels in table of contents
+
+% --- fancyhdr package for fancy headers ---
+\usepackage{fancyhdr}
+\fancyhf{} % sets both header and footer to nothing
+\renewcommand{\headrulewidth}{0pt}
+\fancyfoot[LE,RO]{\thepage}
+% Ensure copyright on titlepage (article style) and chapter pages (book style)
+\fancypagestyle{plain}{
+  \fancyhf{}
+  \fancyfoot[C]{{\footnotesize \copyright\ 1999-2019, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license}}
+%  \renewcommand{\footrulewidth}{0mm}
+  \renewcommand{\headrulewidth}{0mm}
+}
+% Ensure copyright on titlepages with \thispagestyle{empty}
+\fancypagestyle{empty}{
+  \fancyhf{}
+  \fancyfoot[C]{{\footnotesize \copyright\ 1999-2019, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license}}
+  \renewcommand{\footrulewidth}{0mm}
+  \renewcommand{\headrulewidth}{0mm}
+}
+
+\pagestyle{fancy}
+
+
+% prevent orhpans and widows
+\clubpenalty = 10000
+\widowpenalty = 10000
+
+% --- end of standard preamble for documents ---
+
+
+% insert custom LaTeX commands...
+
+\raggedbottom
+\makeindex
+\usepackage[totoc]{idxlayout}   % for index in the toc
+\usepackage[nottoc]{tocbibind}  % for references/bibliography in the toc
+
+%-------------------- end preamble ----------------------
+
+\begin{document}
+
+% matching end for #ifdef PREAMBLE
+
+\newcommand{\exercisesection}[1]{\subsection*{#1}}
+
+
+% ------------------- main content ----------------------
+
+
+
+% ----------------- title -------------------------
+
+\thispagestyle{empty}
+
+\begin{center}
+{\LARGE\bf
+\begin{spacing}{1.25}
+Data Analysis and Machine Learning: Recurrent neural networks
+\end{spacing}
+}
+\end{center}
+
+% ----------------- author(s) -------------------------
+
+\begin{center}
+{\bf Morten Hjorth-Jensen${}^{1, 2}$} \\ [0mm]
+\end{center}
+
+\begin{center}
+% List of all institutions:
+\centerline{{\small ${}^1$Department of Physics, University of Oslo}}
+\centerline{{\small ${}^2$Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University}}
+\end{center}
+    
+% ----------------- end author(s) -------------------------
+
+% --- begin date ---
+\begin{center}
+Jan 8, 2019
+\end{center}
+% --- end date ---
+
+\vspace{1cm}
+
+
+% !split
+\subsection{Recurrent neural networks: Overarching view}
+
+Till now our focus has been, including convolutional neural networks as well, 
+on feedforward neural networks. The output or the
+activations flow only in one direction, from the input layer to the
+output layer.
+
+A recurrent neural network (RNN) looks very much like a feedforward
+neural network, except that it also has connections pointing
+backward. 
+
+RNNs are used to analyze time series data such as stock prices, and
+tell you when to buy or sell. In autonomous driving systems, they can
+anticipate car trajectories and help avoid accidents. More generally,
+they can work on sequences of arbitrary lengths, rather than on
+fixed-sized inputs like all the nets we have discussed so far. For
+example, they can take sentences, documents, or audio samples as
+input, making them extremely useful for natural language processing
+systems such as automatic translation and speech-to-text.
+
+
+% !split
+\subsection{Set up of an RNN}
+
+The figure here displays a simple example of an RNN, with inputs $x_t$
+at a given time $t$ and outputs $y_t$.  Introducing time as a variable
+offers an intutitive way of understanding these networks. In addition
+to the inputs $x_t$, the layer at a time $t$ receives also as input
+the output from the previous layer $t-1$, that is $y_{t1}$.
+
+This means also that we need to have weights that link both the inputs $x_t$ to the outputs $y_t$ as well as weights that link
+the output from the previous time $y_{t-1}$ and $y_t$.   The figure here shows an example of a simple RNN.
+
+% ------------------- end of main content ---------------
+
+\end{document}
+
diff --git a/doc/src/Recurrent/reveal.js/css/theme/template/mixins.scss b/doc/src/Recurrent/reveal.js/css/theme/template/mixins.scss
new file mode 100644
index 000000000..e0c560692
--- /dev/null
+++ b/doc/src/Recurrent/reveal.js/css/theme/template/mixins.scss
@@ -0,0 +1,29 @@
+@mixin vertical-gradient( $top, $bottom ) {
+	background: $top;
+	background: -moz-linear-gradient( top, $top 0%, $bottom 100% );
+	background: -webkit-gradient( linear, left top, left bottom, color-stop(0%,$top), color-stop(100%,$bottom) );
+	background: -webkit-linear-gradient( top, $top 0%, $bottom 100% );
+	background: -o-linear-gradient( top, $top 0%, $bottom 100% );
+	background: -ms-linear-gradient( top, $top 0%, $bottom 100% );
+	background: linear-gradient( top, $top 0%, $bottom 100% );
+}
+
+@mixin horizontal-gradient( $top, $bottom ) {
+	background: $top;
+	background: -moz-linear-gradient( left, $top 0%, $bottom 100% );
+	background: -webkit-gradient( linear, left top, right top, color-stop(0%,$top), color-stop(100%,$bottom) );
+	background: -webkit-linear-gradient( left, $top 0%, $bottom 100% );
+	background: -o-linear-gradient( left, $top 0%, $bottom 100% );
+	background: -ms-linear-gradient( left, $top 0%, $bottom 100% );
+	background: linear-gradient( left, $top 0%, $bottom 100% );
+}
+
+@mixin radial-gradient( $outer, $inner, $type: circle ) {
+	background: $outer;
+	background: -moz-radial-gradient( center, $type cover,  $inner 0%, $outer 100% );
+	background: -webkit-gradient( radial, center center, 0px, center center, 100%, color-stop(0%,$inner), color-stop(100%,$outer) );
+	background: -webkit-radial-gradient( center, $type cover,  $inner 0%, $outer 100% );
+	background: -o-radial-gradient( center, $type cover,  $inner 0%, $outer 100% );
+	background: -ms-radial-gradient( center, $type cover,  $inner 0%, $outer 100% );
+	background: radial-gradient( center, $type cover,  $inner 0%, $outer 100% );
+}
\ No newline at end of file
diff --git a/doc/src/Recurrent/reveal.js/css/theme/template/settings.scss b/doc/src/Recurrent/reveal.js/css/theme/template/settings.scss
new file mode 100644
index 000000000..e706e19c5
--- /dev/null
+++ b/doc/src/Recurrent/reveal.js/css/theme/template/settings.scss
@@ -0,0 +1,34 @@
+// Base settings for all themes that can optionally be
+// overridden by the super-theme
+
+// Background of the presentation
+$backgroundColor: #2b2b2b;
+
+// Primary/body text
+$mainFont: 'Lato', sans-serif;
+$mainFontSize: 30px;              /* changed (by hpl) from 36px; */
+$mainColor: #eee;
+
+// Headings
+$headingMargin: 0 0 20px 0;
+$headingFont: 'League Gothic', Impact, sans-serif;
+$headingColor: #eee;
+$headingLineHeight: 0.9em;
+$headingLetterSpacing: 0.02em;
+$headingTextTransform: none;      /* changed (by hpl) from uppercase; */
+$headingTextShadow: 0px 0px 6px rgba(0,0,0,0.2);
+$heading1TextShadow: $headingTextShadow;
+
+// Links and actions
+$linkColor: #13DAEC;
+$linkColorHover: lighten( $linkColor, 20% );
+
+// Text selection
+$selectionBackgroundColor: #FF5E99;
+$selectionColor: #fff;
+
+// Generates the presentation background, can be overridden
+// to return a background image or gradient
+@mixin bodyBackground() {
+	background: $backgroundColor;
+}
\ No newline at end of file
diff --git a/doc/src/Recurrent/reveal.js/css/theme/template/theme.scss b/doc/src/Recurrent/reveal.js/css/theme/template/theme.scss
new file mode 100644
index 000000000..b1dbc2736
--- /dev/null
+++ b/doc/src/Recurrent/reveal.js/css/theme/template/theme.scss
@@ -0,0 +1,171 @@
+// Base theme template for reveal.js
+
+/*********************************************
+ * GLOBAL STYLES
+ *********************************************/
+
+body {
+	@include bodyBackground();
+	background-color: $backgroundColor;
+}
+
+.reveal {
+	font-family: $mainFont;
+	font-size: $mainFontSize;
+	font-weight: normal;
+	letter-spacing: -0.02em;
+	color: $mainColor;
+}
+
+::selection {
+	color: $selectionColor;
+	background: $selectionBackgroundColor;
+	text-shadow: none;
+}
+
+/*********************************************
+ * HEADERS
+ *********************************************/
+
+.reveal h1,
+.reveal h2,
+.reveal h3,
+.reveal h4,
+.reveal h5,
+.reveal h6 {
+	margin: $headingMargin;
+	color: $headingColor;
+
+	font-family: $headingFont;
+	line-height: $headingLineHeight;
+	letter-spacing: $headingLetterSpacing;
+
+	text-transform: $headingTextTransform;
+	text-shadow: $headingTextShadow;
+}
+
+.reveal h1 {
+        line-height: 1.2em;    /* added by hpl */
+	text-shadow: $heading1TextShadow;
+}
+
+
+/*********************************************
+ * LINKS
+ *********************************************/
+
+.reveal a:not(.image) {
+	color: $linkColor;
+	text-decoration: none;
+
+	-webkit-transition: color .15s ease;
+	   -moz-transition: color .15s ease;
+	    -ms-transition: color .15s ease;
+	     -o-transition: color .15s ease;
+	        transition: color .15s ease;
+}
+	.reveal a:not(.image):hover {
+		color: $linkColorHover;
+
+		text-shadow: none;
+		border: none;
+	}
+
+.reveal .roll span:after {
+	color: #fff;
+	background: darken( $linkColor, 15% );
+}
+
+
+/*********************************************
+ * IMAGES
+ *********************************************/
+
+.reveal section img {
+	margin: 15px 0px;
+	background: rgba(255,255,255,0.12);
+	border: 4px solid $mainColor;
+
+	box-shadow: 0 0 10px rgba(0, 0, 0, 0.15);
+
+	-webkit-transition: all .2s linear;
+	   -moz-transition: all .2s linear;
+	    -ms-transition: all .2s linear;
+	     -o-transition: all .2s linear;
+	        transition: all .2s linear;
+}
+
+	.reveal a:hover img {
+		background: rgba(255,255,255,0.2);
+		border-color: $linkColor;
+
+		box-shadow: 0 0 20px rgba(0, 0, 0, 0.55);
+	}
+
+
+/*********************************************
+ * NAVIGATION CONTROLS
+ *********************************************/
+
+.reveal .controls div.navigate-left,
+.reveal .controls div.navigate-left.enabled {
+	border-right-color: $linkColor;
+}
+
+.reveal .controls div.navigate-right,
+.reveal .controls div.navigate-right.enabled {
+	border-left-color: $linkColor;
+}
+
+.reveal .controls div.navigate-up,
+.reveal .controls div.navigate-up.enabled {
+	border-bottom-color: $linkColor;
+}
+
+.reveal .controls div.navigate-down,
+.reveal .controls div.navigate-down.enabled {
+	border-top-color: $linkColor;
+}
+
+.reveal .controls div.navigate-left.enabled:hover {
+	border-right-color: $linkColorHover;
+}
+
+.reveal .controls div.navigate-right.enabled:hover {
+	border-left-color: $linkColorHover;
+}
+
+.reveal .controls div.navigate-up.enabled:hover {
+	border-bottom-color: $linkColorHover;
+}
+
+.reveal .controls div.navigate-down.enabled:hover {
+	border-top-color: $linkColorHover;
+}
+
+
+/*********************************************
+ * PROGRESS BAR
+ *********************************************/
+
+.reveal .progress {
+	background: rgba(0,0,0,0.2);
+}
+	.reveal .progress span {
+		background: $linkColor;
+
+		-webkit-transition: width 800ms cubic-bezier(0.260, 0.860, 0.440, 0.985);
+		   -moz-transition: width 800ms cubic-bezier(0.260, 0.860, 0.440, 0.985);
+		    -ms-transition: width 800ms cubic-bezier(0.260, 0.860, 0.440, 0.985);
+		     -o-transition: width 800ms cubic-bezier(0.260, 0.860, 0.440, 0.985);
+		        transition: width 800ms cubic-bezier(0.260, 0.860, 0.440, 0.985);
+	}
+
+/*********************************************
+ * SLIDE NUMBER
+ *********************************************/
+.reveal .slide-number {
+  color: $linkColor;
+}
+
+
diff --git a/doc/src/Regression/.Regression-bs_html_file_collection b/doc/src/Regression/.Regression-bs_html_file_collection
new file mode 100644
index 000000000..49f9aeeb4
--- /dev/null
+++ b/doc/src/Regression/.Regression-bs_html_file_collection
@@ -0,0 +1,101 @@
+Regression-bs.html
+._Regression-bs000.html
+._Regression-bs001.html
+._Regression-bs002.html
+._Regression-bs003.html
+._Regression-bs004.html
+._Regression-bs005.html
+._Regression-bs006.html
+._Regression-bs007.html
+._Regression-bs008.html
+._Regression-bs009.html
+._Regression-bs010.html
+._Regression-bs011.html
+._Regression-bs012.html
+._Regression-bs013.html
+._Regression-bs014.html
+._Regression-bs015.html
+._Regression-bs016.html
+._Regression-bs017.html
+._Regression-bs018.html
+._Regression-bs019.html
+._Regression-bs020.html
+._Regression-bs021.html
+._Regression-bs022.html
+._Regression-bs023.html
+._Regression-bs024.html
+._Regression-bs025.html
+._Regression-bs026.html
+._Regression-bs027.html
+._Regression-bs028.html
+._Regression-bs029.html
+._Regression-bs030.html
+._Regression-bs031.html
+._Regression-bs032.html
+._Regression-bs033.html
+._Regression-bs034.html
+._Regression-bs035.html
+._Regression-bs036.html
+._Regression-bs037.html
+._Regression-bs038.html
+._Regression-bs039.html
+._Regression-bs040.html
+._Regression-bs041.html
+._Regression-bs042.html
+._Regression-bs043.html
+._Regression-bs044.html
+._Regression-bs045.html
+._Regression-bs046.html
+._Regression-bs047.html
+._Regression-bs048.html
+._Regression-bs049.html
+._Regression-bs050.html
+._Regression-bs051.html
+._Regression-bs052.html
+._Regression-bs053.html
+._Regression-bs054.html
+._Regression-bs055.html
+._Regression-bs056.html
+._Regression-bs057.html
+._Regression-bs058.html
+._Regression-bs059.html
+._Regression-bs060.html
+._Regression-bs061.html
+._Regression-bs062.html
+._Regression-bs063.html
+._Regression-bs064.html
+._Regression-bs065.html
+._Regression-bs066.html
+._Regression-bs067.html
+._Regression-bs068.html
+._Regression-bs069.html
+._Regression-bs070.html
+._Regression-bs071.html
+._Regression-bs072.html
+._Regression-bs073.html
+._Regression-bs074.html
+._Regression-bs075.html
+._Regression-bs076.html
+._Regression-bs077.html
+._Regression-bs078.html
+._Regression-bs079.html
+._Regression-bs080.html
+._Regression-bs081.html
+._Regression-bs082.html
+._Regression-bs083.html
+._Regression-bs084.html
+._Regression-bs085.html
+._Regression-bs086.html
+._Regression-bs087.html
+._Regression-bs088.html
+._Regression-bs089.html
+._Regression-bs090.html
+._Regression-bs091.html
+._Regression-bs092.html
+._Regression-bs093.html
+._Regression-bs094.html
+._Regression-bs095.html
+._Regression-bs096.html
+._Regression-bs097.html
+._Regression-bs098.html
+._Regression-bs099.html
diff --git a/doc/src/Regression/.Regression-reveal_html_file_collection b/doc/src/Regression/.Regression-reveal_html_file_collection
new file mode 100644
index 000000000..dcec8b30b
--- /dev/null
+++ b/doc/src/Regression/.Regression-reveal_html_file_collection
@@ -0,0 +1,2 @@
+Regression-reveal.html
+reveal.js
diff --git a/doc/src/Regression/.Regression-solarized_html_file_collection b/doc/src/Regression/.Regression-solarized_html_file_collection
new file mode 100644
index 000000000..e7f421255
--- /dev/null
+++ b/doc/src/Regression/.Regression-solarized_html_file_collection
@@ -0,0 +1 @@
+Regression-solarized.html
diff --git a/doc/src/Regression/.Regression.copyright b/doc/src/Regression/.Regression.copyright
new file mode 100644
index 000000000..0507b6521
--- /dev/null
+++ b/doc/src/Regression/.Regression.copyright
@@ -0,0 +1 @@
+{'holder': ['Morten Hjorth-Jensen'], 'year': '1999-2019', 'license': 'Released under CC Attribution-NonCommercial 4.0 license', 'cite doconce': False}
\ No newline at end of file
diff --git a/doc/src/Regression/.Regression_html_file_collection b/doc/src/Regression/.Regression_html_file_collection
new file mode 100644
index 000000000..b2083b0ad
--- /dev/null
+++ b/doc/src/Regression/.Regression_html_file_collection
@@ -0,0 +1 @@
+Regression.html
diff --git a/doc/src/Regression/._Regression-bs000.html b/doc/src/Regression/._Regression-bs000.html
new file mode 100644
index 000000000..aef0fd910
--- /dev/null
+++ b/doc/src/Regression/._Regression-bs000.html
@@ -0,0 +1,452 @@
+
+
+
+
+
+
+
+
+Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+    
+
+

+  

+    
+    Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis
+  

+
+  

+    
+  

+

+
 
+
+

+
+
 
 
 
 
+
+
+
+
+
+
+

+
Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis
  
+
+

+
+
+

+Morten Hjorth-Jensen [1, 2]
+

+
+

+
+
+
[1] Department of Physics, University of Oslo

+
[2] Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University

+

+

+
Jul 22, 2019
 
+

+

+
+
+
Read »

+
+
+
 
+
+

+
+

+
+
+
  
+
+
+
+
+
+
+
+

+ © 1999-2019, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license
+

+
+
+
+
+    
+
diff --git a/doc/src/Regression/._Regression-bs001.html b/doc/src/Regression/._Regression-bs001.html
new file mode 100644
index 000000000..3401b90d4
--- /dev/null
+++ b/doc/src/Regression/._Regression-bs001.html
@@ -0,0 +1,446 @@
+
+
+
+
+
+
+
+
+Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+    
+
+

+  

+    
+    Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis
+  

+
+  

+    
+  

+

+
 
+
+

+
+
 
 
 
 
+
+
+
+
+
Why Linear Regression (aka Ordinary Least Squares and family) 

+
+

+Fitting a continuous function with linear parameterization in terms of the parameters  \( \boldsymbol{\beta} \).
+
+
    
+
  •  Method of choice for fitting a continuous function!
    • 
+
  •  Gives an excellent introduction to central Machine Learning features with understandable pedagogical links to other methods like Neural Networks, Support Vector Machines etc
    • 
+
  •  Analytical expression for the fitting parameters \( \boldsymbol{\beta} \)
    • 
+
  •  Analytical expressions for statistical propertiers like mean values, variances, confidence intervals and more
    • 
+
  •  Analytical relation with probabilistic interpretations
    •  
+
  •  Easy to introduce basic concepts like bias-variance tradeoff, cross-validation, resampling and regularization techniques and many other ML topics
    • 
+
  •  Easy to code! And links well with classification problems and logistic regression and neural networks
    • 
+
  •  Allows for easy hands-on understanding of gradient descent methods
    • 
+
  •  and many more features
    • 
+

+
+For more discussions of Ridge and Lasso regression, Wessel van Wieringen's article is highly recommended.
+Similarly, Mehta et al's article is also recommended.
+
+

+

+
+

+
+
+
  
+
+
+
+
+
+
+
+

+
+

+
+
+
+
+    
+
diff --git a/doc/src/Regression/._Regression-bs002.html b/doc/src/Regression/._Regression-bs002.html
new file mode 100644
index 000000000..dee6ad935
--- /dev/null
+++ b/doc/src/Regression/._Regression-bs002.html
@@ -0,0 +1,450 @@
+
+
+
+
+
+
+
+
+Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+    
+
+

+  

+    
+    Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis
+  

+
+  

+    
+  

+

+
 
+
+

+
+
 
 
 
 
+
+
+
+
+
Regression analysis, overarching aims  

+

+

+
 
+
+

+Regression modeling deals with the description of  the sampling distribution of a given random variable \( y \) and how it varies as function of another variable or a set of such variables \( \boldsymbol{x} =[x_0, x_1,\dots, x_{n-1}]^T \). 
+The first variable is called the dependent, the outcome or the response variable while the set of variables \( \boldsymbol{x} \) is called the independent variable, or the predictor variable or the explanatory variable.
+
+

+A regression model aims at finding a likelihood function \( p(\boldsymbol{y}\vert \boldsymbol{x}) \), that is the conditional distribution for \( \boldsymbol{y} \) with a given \( \boldsymbol{x} \). The estimation of  \( p(\boldsymbol{y}\vert \boldsymbol{x}) \) is made using a data set with 
+
+
    
+
  •  \( n \) cases \( i = 0, 1, 2, \dots, n-1 \)
    •  
+
  •  Response (target, dependent or outcome) variable \( y_i \) with \( i = 0, 1, 2, \dots, n-1 \)
    •  
+
  •  \( p \) so-called explanatory (independent or predictor) variables \( \boldsymbol{x}_i=[x_{i0}, x_{i1}, \dots, x_{ip-1}] \) with \( i = 0, 1, 2, \dots, n-1 \) and explanatory variables running from \( 0 \) to \( p-1 \). See below for more explicit examples.
    •    
+

+
+ The goal of the regression analysis is to extract/exploit relationship between \( \boldsymbol{y} \) and \( \boldsymbol{X} \) in or to infer causal dependencies, approximations to the likelihood functions, functional relationships and to make predictions, making fits and many other things.
+

+

+
+
+

+

+
+

+
+
+
  
+
+
+
+
+
+
+
+

+
+

+
+
+
+
+    
+
diff --git a/doc/src/Regression/._Regression-bs003.html b/doc/src/Regression/._Regression-bs003.html
new file mode 100644
index 000000000..88f261dee
--- /dev/null
+++ b/doc/src/Regression/._Regression-bs003.html
@@ -0,0 +1,459 @@
+
+
+
+
+
+
+
+
+Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+    
+
+

+  

+    
+    Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis
+  

+
+  

+    
+  

+

+
+ +
+ +

 

 

 

+ + + + +

Regression analysis, overarching aims II

+
+
+

+ +

+Consider an experiment in which \( p \) characteristics of \( n \) samples are +measured. The data from this experiment, for various explanatory variables \( p \) are normally represented by a matrix +\( \mathbf{X} \). + +

+The matrix \( \mathbf{X} \) is called the design +matrix. Additional information of the samples is available in the +form of \( \boldsymbol{y} \) (also as above). The variable \( \boldsymbol{y} \) is +generally referred to as the response variable. The aim of +regression analysis is to explain \( \boldsymbol{y} \) in terms of +\( \boldsymbol{X} \) through a functional relationship like \( y_i = +f(\mathbf{X}_{i,\ast}) \). When no prior knowledge on the form of +\( f(\cdot) \) is available, it is common to assume a linear relationship +between \( \boldsymbol{X} \) and \( \boldsymbol{y} \). This assumption gives rise to +the linear regression model where \( \boldsymbol{\beta} = [\beta_0, \ldots, +\beta_{p-1}]^{T} \) are the regression parameters. + +

+Linear regression gives us a set of analytical equations for the parameters \( \beta_j \). + +

+

+
+ + +

+

+ +

+ + +
+ + + + + + + +
+ +
+ + + + + + diff --git a/doc/src/Regression/._Regression-bs004.html b/doc/src/Regression/._Regression-bs004.html new file mode 100644 index 000000000..6e9d3daca --- /dev/null +++ b/doc/src/Regression/._Regression-bs004.html @@ -0,0 +1,456 @@ + + + + + + + + +Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis + + + + + + + + + + + + + + + + + + + + + + + +
+
+ + Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis +
+ +
+ +
+
+ + +
+ +

 

 

 

+ + + + +

Examples

+
+
+

+In order to understand the relation among the predictors \( p \), the set of data \( n \) and the target (outcome, output etc) \( \boldsymbol{y} \), +consider the model we discussed for describing nuclear binding energies. + +

+There we assumed that we could parametrize the data using a polynomial approximation based on the liquid drop model. +Assuming +$$ +BE(A) = a_0+a_1A+a_2A^{2/3}+a_3A^{-1/3}+a_4A^{-1}, +$$ + +we have five predictors, that is the intercept, the \( A \) dependent term, the \( A^{2/3} \) term and the \( A^{-1/3} \) and \( A^{-1} \) terms. +This gives \( p=0,1,2,3,4 \). Furthermore we have \( n \) entries for each predictor. It means that our design matrix is a +\( p\times n \) matrix \( \boldsymbol{X} \). + +

+Here the predictors are based on a model we have made. A popular data set which is widely encountered in ML applications is the +so-called credit card default data from Taiwan. The data set contains data on \( n=30000 \) credit card holders with predictors like gender, marital status, age, profession, education, etc. In total there are \( 24 \) such predictors or attributes leading to a design matrix of dimensionality \( 24 \times 30000 \) + +

+

+
+ + +

+

+ +

+ + +
+ + + + + + + +
+ +
+ + + + + + diff --git a/doc/src/Regression/._Regression-bs005.html b/doc/src/Regression/._Regression-bs005.html new file mode 100644 index 000000000..35f7c9bdb --- /dev/null +++ b/doc/src/Regression/._Regression-bs005.html @@ -0,0 +1,449 @@ + + + + + + + + +Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis + + + + + + + + + + + + + + + + + + + + + + + +
+
+ + Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis +
+ +
+ +
+
+ + +
+ +

 

 

 

+ + + + +

General linear models

+
+
+

+Before we proceed let us study a case from linear algebra where we aim at fitting a set of data \( \boldsymbol{y}=[y_0,y_1,\dots,y_{n-1}] \). We could think of these data as a result of an experiment or a complicated numerical experiment. These data are functions of a series of variables \( \boldsymbol{x}=[x_0,x_1,\dots,x_{n-1}] \), that is \( y_i = y(x_i) \) with \( i=0,1,2,\dots,n-1 \). The variables \( x_i \) could represent physical quantities like time, temperature, position etc. We assume that \( y(x) \) is a smooth function. + +

+Since obtaining these data points may not be trivial, we want to use these data to fit a function which can allow us to make predictions for values of \( y \) which are not in the present set. The perhaps simplest approach is to assume we can parametrize our function in terms of a polynomial of degree \( n-1 \) with \( n \) points, that is +$$ +y=y(x) \rightarrow y(x_i)=\tilde{y}_i+\epsilon_i=\sum_{j=0}^{n-1} \beta_j x_i^j+\epsilon_i, +$$ + +where \( \epsilon_i \) is the error in our approximation. + +

+

+
+ + +

+

+ +

+ + +
+ + + + + + + +
+ +
+ + + + + + diff --git a/doc/src/Regression/._Regression-bs006.html b/doc/src/Regression/._Regression-bs006.html new file mode 100644 index 000000000..cf2ff155f --- /dev/null +++ b/doc/src/Regression/._Regression-bs006.html @@ -0,0 +1,449 @@ + + + + + + + + +Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis + + + + + + + + + + + + + + + + + + + + + + + +
+
+ + Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis +
+ +
+ +
+
+ + +
+ +

 

 

 

+ + + + +

Rewriting the fitting procedure as a linear algebra problem

+
+
+

+For every set of values \( y_i,x_i \) we have thus the corresponding set of equations +$$ +\begin{align*} +y_0&=\beta_0+\beta_1x_0^1+\beta_2x_0^2+\dots+\beta_{n-1}x_0^{n-1}+\epsilon_0\\ +y_1&=\beta_0+\beta_1x_1^1+\beta_2x_1^2+\dots+\beta_{n-1}x_1^{n-1}+\epsilon_1\\ +y_2&=\beta_0+\beta_1x_2^1+\beta_2x_2^2+\dots+\beta_{n-1}x_2^{n-1}+\epsilon_2\\ +\dots & \dots \\ +y_{n-1}&=\beta_0+\beta_1x_{n-1}^1+\beta_2x_{n-1}^2+\dots+\beta_{n-1}x_{n-1}^{n-1}+\epsilon_{n-1}.\\ +\end{align*} +$$ +

+
+ + +

+

+ +

+ + +
+ + + + + + + +
+ +
+ + + + + + diff --git a/doc/src/Regression/._Regression-bs007.html b/doc/src/Regression/._Regression-bs007.html new file mode 100644 index 000000000..ee1cfb7c7 --- /dev/null +++ b/doc/src/Regression/._Regression-bs007.html @@ -0,0 +1,473 @@ + + + + + + + + +Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis + + + + + + + + + + + + + + + + + + + + + + + +
+
+ + Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis +
+ +
+ +
+
+ + +
+ +

 

 

 

+ + + + +

Rewriting the fitting procedure as a linear algebra problem, more details

+
+
+

+Defining the vectors +$$ +\boldsymbol{y} = [y_0,y_1, y_2,\dots, y_{n-1}]^T, +$$ + +and +$$ +\boldsymbol{\beta} = [\beta_0,\beta_1, \beta_2,\dots, \beta_{n-1}]^T, +$$ + +and +$$ +\boldsymbol{\epsilon} = [\epsilon_0,\epsilon_1, \epsilon_2,\dots, \epsilon_{n-1}]^T, +$$ + +and the design matrix +$$ +\boldsymbol{X}= +\begin{bmatrix} +1& x_{0}^1 &x_{0}^2& \dots & \dots &x_{0}^{n-1}\\ +1& x_{1}^1 &x_{1}^2& \dots & \dots &x_{1}^{n-1}\\ +1& x_{2}^1 &x_{2}^2& \dots & \dots &x_{2}^{n-1}\\ +\dots& \dots &\dots& \dots & \dots &\dots\\ +1& x_{n-1}^1 &x_{n-1}^2& \dots & \dots &x_{n-1}^{n-1}\\ +\end{bmatrix} +$$ + +we can rewrite our equations as +$$ +\boldsymbol{y} = \boldsymbol{X}\boldsymbol{\beta}+\boldsymbol{\epsilon}. +$$ + +The above design matrix is called a Vandermonde matrix. +

+
+ + +

+

+ +

+ + +
+ + + + + + + +
+ +
+ + + + + + diff --git a/doc/src/Regression/._Regression-bs008.html b/doc/src/Regression/._Regression-bs008.html new file mode 100644 index 000000000..2976e41a3 --- /dev/null +++ b/doc/src/Regression/._Regression-bs008.html @@ -0,0 +1,463 @@ + + + + + + + + +Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis + + + + + + + + + + + + + + + + + + + + + + + +
+
+ + Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis +
+ +
+ +
+
+ + +
+ +

 

 

 

+ + + + +

Generalizing the fitting procedure as a linear algebra problem

+
+
+

+ +

+We are obviously not limited to the above polynomial expansions. We +could replace the various powers of \( x \) with elements of Fourier +series or instead of \( x_i^j \) we could have \( \cos{(j x_i)} \) or \( \sin{(j +x_i)} \), or time series or other orthogonal functions. For every set +of values \( y_i,x_i \) we can then generalize the equations to + +$$ +\begin{align*} +y_0&=\beta_0x_{00}+\beta_1x_{01}+\beta_2x_{02}+\dots+\beta_{n-1}x_{0n-1}+\epsilon_0\\ +y_1&=\beta_0x_{10}+\beta_1x_{11}+\beta_2x_{12}+\dots+\beta_{n-1}x_{1n-1}+\epsilon_1\\ +y_2&=\beta_0x_{20}+\beta_1x_{21}+\beta_2x_{22}+\dots+\beta_{n-1}x_{2n-1}+\epsilon_2\\ +\dots & \dots \\ +y_{i}&=\beta_0x_{i0}+\beta_1x_{i1}+\beta_2x_{i2}+\dots+\beta_{n-1}x_{in-1}+\epsilon_i\\ +\dots & \dots \\ +y_{n-1}&=\beta_0x_{n-1,0}+\beta_1x_{n-1,2}+\beta_2x_{n-1,2}+\dots+\beta_{n-1}x_{n-1,n-1}+\epsilon_{n-1}.\\ +\end{align*} +$$ + +

+Note that we have \( p=n \) here. The matrix is symmetric. This is generally not the case! +

+
+ + +

+

+ +

+ + +
+ + + + + + + +
+ +
+ + + + + + diff --git a/doc/src/Regression/._Regression-bs009.html b/doc/src/Regression/._Regression-bs009.html new file mode 100644 index 000000000..12a0d62fa --- /dev/null +++ b/doc/src/Regression/._Regression-bs009.html @@ -0,0 +1,460 @@ + + + + + + + + +Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis + + + + + + + + + + + + + + + + + + + + + + + +
+
+ + Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis +
+ +
+ +
+
+ + +
+ +

 

 

 

+ + + + +

Generalizing the fitting procedure as a linear algebra problem

+
+
+

+We redefine in turn the matrix \( \boldsymbol{X} \) as +$$ +\boldsymbol{X}= +\begin{bmatrix} +x_{00}& x_{01} &x_{02}& \dots & \dots &x_{0,n-1}\\ +x_{10}& x_{11} &x_{12}& \dots & \dots &x_{1,n-1}\\ +x_{20}& x_{21} &x_{22}& \dots & \dots &x_{2,n-1}\\ +\dots& \dots &\dots& \dots & \dots &\dots\\ +x_{n-1,0}& x_{n-1,1} &x_{n-1,2}& \dots & \dots &x_{n-1,n-1}\\ +\end{bmatrix} +$$ + +and without loss of generality we rewrite again our equations as +$$ +\boldsymbol{y} = \boldsymbol{X}\boldsymbol{\beta}+\boldsymbol{\epsilon}. +$$ + +The left-hand side of this equation is kwown. Our error vector \( \boldsymbol{\epsilon} \) and the parameter vector \( \boldsymbol{\beta} \) are our unknow quantities. How can we obtain the optimal set of \( \beta_i \) values? +

+
+ + +

+

+ +

+ + +
+ + + + + + + +
+ +
+ + + + + + diff --git a/doc/src/Regression/._Regression-bs010.html b/doc/src/Regression/._Regression-bs010.html new file mode 100644 index 000000000..96f43a65d --- /dev/null +++ b/doc/src/Regression/._Regression-bs010.html @@ -0,0 +1,462 @@ + + + + + + + + +Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis + + + + + + + + + + + + + + + + + + + + + + + +
+
+ + Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis +
+ +
+ +
+
+ + +
+ +

 

 

 

+ + + + +

Optimizing our parameters

+
+
+

+We have defined the matrix \( \boldsymbol{X} \) via the equations +$$ +\begin{align*} +y_0&=\beta_0x_{00}+\beta_1x_{01}+\beta_2x_{02}+\dots+\beta_{n-1}x_{0n-1}+\epsilon_0\\ +y_1&=\beta_0x_{10}+\beta_1x_{11}+\beta_2x_{12}+\dots+\beta_{n-1}x_{1n-1}+\epsilon_1\\ +y_2&=\beta_0x_{20}+\beta_1x_{21}+\beta_2x_{22}+\dots+\beta_{n-1}x_{2n-1}+\epsilon_1\\ +\dots & \dots \\ +y_{i}&=\beta_0x_{i0}+\beta_1x_{i1}+\beta_2x_{i2}+\dots+\beta_{n-1}x_{in-1}+\epsilon_1\\ +\dots & \dots \\ +y_{n-1}&=\beta_0x_{n-1,0}+\beta_1x_{n-1,2}+\beta_2x_{n-1,2}+\dots+\beta_{n-1}x_{n-1,n-1}+\epsilon_{n-1}.\\ +\end{align*} +$$ + +

+As we noted above, we stayed with a system with the design matrix + \( \boldsymbol{X}\in {\mathbb{R}}^{n\times n} \), that is we have \( p=n \). For reasons to come later (algorithmic arguments) we will hereafter define +our matrix as \( \boldsymbol{X}\in {\mathbb{R}}^{n\times p} \), with the predictors refering to the column numbers and the entries \( n \) being the row elements. + +

+

+
+ + +

+

+ +

+ + +
+ + + + + + + +
+ +
+ + + + + + diff --git a/doc/src/Regression/._Regression-bs011.html b/doc/src/Regression/._Regression-bs011.html new file mode 100644 index 000000000..9c489cc58 --- /dev/null +++ b/doc/src/Regression/._Regression-bs011.html @@ -0,0 +1,523 @@ + + + + + + + + +Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis + + + + + + + + + + + + + + + + + + + + + + + +
+
+ + Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis +
+ +
+ +
+
+ + +
+ +

 

 

 

+ + + + +

Our model for the nuclear binding energies

+ +

+In our introductory notes we looked at the so-called liguid drop model. Let us remind ourselves about what we did by looking at the code. + +

+We restate the parts of the code we are most interested in. +

+ + +

# Common imports
+import numpy as np
+import pandas as pd
+import matplotlib.pyplot as plt
+from IPython.display import display
+import os
+
+# Where to save the figures and data files
+PROJECT_ROOT_DIR = "Results"
+FIGURE_ID = "Results/FigureFiles"
+DATA_ID = "DataFiles/"
+
+if not os.path.exists(PROJECT_ROOT_DIR):
+    os.mkdir(PROJECT_ROOT_DIR)
+
+if not os.path.exists(FIGURE_ID):
+    os.makedirs(FIGURE_ID)
+
+if not os.path.exists(DATA_ID):
+    os.makedirs(DATA_ID)
+
+def image_path(fig_id):
+    return os.path.join(FIGURE_ID, fig_id)
+
+def data_path(dat_id):
+    return os.path.join(DATA_ID, dat_id)
+
+def save_fig(fig_id):
+    plt.savefig(image_path(fig_id) + ".png", format='png')
+
+infile = open(data_path("MassEval2016.dat"),'r')
+
+
+# Read the experimental data with Pandas
+Masses = pd.read_fwf(infile, usecols=(2,3,4,6,11),
+              names=('N', 'Z', 'A', 'Element', 'Ebinding'),
+              widths=(1,3,5,5,5,1,3,4,1,13,11,11,9,1,2,11,9,1,3,1,12,11,1),
+              header=39,
+              index_col=False)
+
+# Extrapolated values are indicated by '#' in place of the decimal place, so
+# the Ebinding column won't be numeric. Coerce to float and drop these entries.
+Masses['Ebinding'] = pd.to_numeric(Masses['Ebinding'], errors='coerce')
+Masses = Masses.dropna()
+# Convert from keV to MeV.
+Masses['Ebinding'] /= 1000
+
+# Group the DataFrame by nucleon number, A.
+Masses = Masses.groupby('A')
+# Find the rows of the grouped DataFrame with the maximum binding energy.
+Masses = Masses.apply(lambda t: t[t.Ebinding==t.Ebinding.max()])
+A = Masses['A']
+Z = Masses['Z']
+N = Masses['N']
+Element = Masses['Element']
+Energies = Masses['Ebinding']
+
+# Now we set up the design matrix X
+X = np.zeros((len(A),5))
+X[:,0] = 1
+X[:,1] = A
+X[:,2] = A**(2.0/3.0)
+X[:,3] = A**(-1.0/3.0)
+X[:,4] = A**(-1.0)
+# Then nice printout using pandas
+DesignMatrix = pd.DataFrame(X)
+DesignMatrix.index = A
+DesignMatrix.columns = ['1', 'A', 'A^(2/3)', 'A^(-1/3)', '1/A']
+display(DesignMatrix)
+
+

+With \( \boldsymbol{\beta}\in {\mathbb{R}}^{p\times 1} \), it means that we will hereafter write our equations for the approximation as +$$ +\boldsymbol{\tilde{y}}= \boldsymbol{X}\boldsymbol{\beta}, +$$ + +throughout these lectures. + +

+

+ +

+ + +
+ + + + + + + +
+ +
+ + + + + + diff --git a/doc/src/Regression/._Regression-bs012.html b/doc/src/Regression/._Regression-bs012.html new file mode 100644 index 000000000..26a0feef8 --- /dev/null +++ b/doc/src/Regression/._Regression-bs012.html @@ -0,0 +1,469 @@ + + + + + + + + +Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis + + + + + + + + + + + + + + + + + + + + + + + +
+
+ + Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis +
+ +
+ +
+
+ + +
+ +

 

 

 

+ + + + +

Optimizing our parameters, more details

+
+
+

+With the above we use the design matrix to define the approximation \( \boldsymbol{\tilde{y}} \) via the unknown quantity \( \boldsymbol{\beta} \) as +$$ +\boldsymbol{\tilde{y}}= \boldsymbol{X}\boldsymbol{\beta}, +$$ + +and in order to find the optimal parameters \( \beta_i \) instead of solving the above linear algebra problem, we define a function which gives a measure of the spread between the values \( y_i \) (which represent hopefully the exact values) and the parameterized values \( \tilde{y}_i \), namely +$$ +C(\boldsymbol{\beta})=\frac{1}{n}\sum_{i=0}^{n-1}\left(y_i-\tilde{y}_i\right)^2=\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{\tilde{y}}\right)^T\left(\boldsymbol{y}-\boldsymbol{\tilde{y}}\right)\right\}, +$$ + +or using the matrix \( \boldsymbol{X} \) and in a more compact matrix-vector notation as +$$ +C(\boldsymbol{\beta})=\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{X}^T\boldsymbol{\beta}\right)^T\left(\boldsymbol{y}-\boldsymbol{X}^T\boldsymbol{\beta}\right)\right\}. +$$ + +This function is one possible way to define the so-called cost function. + +

+It is also common to define +the function \( Q \) as + +$$ +C(\boldsymbol{\beta})=\frac{1}{2n}\sum_{i=0}^{n-1}\left(y_i-\tilde{y}_i\right)^2, +$$ + +since when taking the first derivative with respect to the unknown parameters \( \beta \), the factor of \( 2 \) cancels out. +

+
+ + +

+

+ +

+ + +
+ + + + + + + +
+ +
+ + + + + + diff --git a/doc/src/Regression/._Regression-bs013.html b/doc/src/Regression/._Regression-bs013.html new file mode 100644 index 000000000..e4d1547cc --- /dev/null +++ b/doc/src/Regression/._Regression-bs013.html @@ -0,0 +1,489 @@ + + + + + + + + +Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis + + + + + + + + + + + + + + + + + + + + + + + +
+
+ + Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis +
+ +
+ +
+
+ + +
+ +

 

 

 

+ + + + +

Interpretations and optimizing our parameters

+
+
+

+ +

+The function +$$ +C(\boldsymbol{\beta})=\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)^T\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)\right\}, +$$ + +can be linked to the variance of the quantity \( y_i \) if we interpret the latter as the mean value. +When linking below with the maximum likelihood approach below, we will indeed interpret \( y_i \) as a mean value (see exercises) +$$ +y_{i}=\langle y_i \rangle = \beta_0x_{i,0}+\beta_1x_{i,1}+\beta_2x_{i,2}+\dots+\beta_{n-1}x_{i,n-1}+\epsilon_i, +$$ + +

+where \( \langle y_i \rangle \) is the mean value. Keep in mind also that +till now we have treated \( y_i \) as the exact value. Normally, the +response (dependent or outcome) variable \( y_i \) the outcome of a +numerical experiment or another type of experiment and is thus only an +approximation to the true value. It is then always accompanied by an +error estimate, often limited to a statistical error estimate given by +the standard deviation discussed earlier. In the discussion here we +will treat \( y_i \) as our exact value for the response variable. + +

+In order to find the parameters \( \beta_i \) we will then minimize the spread of \( C(\boldsymbol{\beta}) \), that is we are going to solve the problem +$$ +{\displaystyle \min_{\boldsymbol{\beta}\in +{\mathbb{R}}^{p}}}\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)^T\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)\right\}. +$$ + +In practical terms it means we will require +$$ +\frac{\partial C(\boldsymbol{\beta})}{\partial \beta_j} = \frac{\partial }{\partial \beta_j}\left[ \frac{1}{n}\sum_{i=0}^{n-1}\left(y_i-\beta_0x_{i,0}-\beta_1x_{i,1}-\beta_2x_{i,2}-\dots-\beta_{n-1}x_{i,n-1}\right)^2\right]=0, +$$ + +which results in +$$ +\frac{\partial C(\boldsymbol{\beta})}{\partial \beta_j} = -\frac{2}{n}\left[ \sum_{i=0}^{n-1}x_{ij}\left(y_i-\beta_0x_{i,0}-\beta_1x_{i,1}-\beta_2x_{i,2}-\dots-\beta_{n-1}x_{i,n-1}\right)\right]=0, +$$ + +or in a matrix-vector form as +$$ +\frac{\partial C(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}} = 0 = \boldsymbol{X}^T\left( \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right). +$$ + +

+

+
+ + +

+

+ +

+ + +
+ + + + + + + +
+ +
+ + + + + + diff --git a/doc/src/Regression/._Regression-bs014.html b/doc/src/Regression/._Regression-bs014.html new file mode 100644 index 000000000..fd3112740 --- /dev/null +++ b/doc/src/Regression/._Regression-bs014.html @@ -0,0 +1,471 @@ + + + + + + + + +Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis + + + + + + + + + + + + + + + + + + + + + + + +
+
+ + Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis +
+ +
+ +
+
+ + +
+ +

 

 

 

+ + + + +

Interpretations and optimizing our parameters

+
+
+

+We can rewrite +$$ +\frac{\partial C(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}} = 0 = \boldsymbol{X}^T\left( \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right), +$$ + +as +$$ +\boldsymbol{X}^T\boldsymbol{y} = \boldsymbol{X}^T\boldsymbol{X}\boldsymbol{\beta}, +$$ + +and if the matrix \( \boldsymbol{X}^T\boldsymbol{X} \) is invertible we have the solution +$$ +\boldsymbol{\beta} =\left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}. +$$ + +

+We note also that since our design matrix is defined as \( \boldsymbol{X}\in +{\mathbb{R}}^{n\times p} \), the product \( \boldsymbol{X}^T\boldsymbol{X} \in +{\mathbb{R}}^{p\times p} \). In the above case we have that \( p \ll n \), +in our case \( p=5 \) meaning that we end up with inverting a small +\( 5\times 5 \) matrix. This is a rather common situation, in many cases we end up with low-dimensional +matrices to invert. The methods discussed here and for many other +supervised learning algorithms like classification with logistic +regression or support vector machines, exhibit dimensionalities which +allow for the usage of direct linear algebra methods such as LU decomposition or Singular Value Decomposition (SVD) for finding the inverse of the matrix +\( \boldsymbol{X}^T\boldsymbol{X} \). + +

+

+
+ + +

+

+ +

+ + +
+ + + + + + + +
+ +
+ + + + + + diff --git a/doc/src/Regression/._Regression-bs015.html b/doc/src/Regression/._Regression-bs015.html new file mode 100644 index 000000000..82a9581f1 --- /dev/null +++ b/doc/src/Regression/._Regression-bs015.html @@ -0,0 +1,464 @@ + + + + + + + + +Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis + + + + + + + + + + + + + + + + + + + + + + + +
+
+ + Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis +
+ +
+ +
+
+ + +
+ +

 

 

 

+ + + + +

Interpretations and optimizing our parameters

+
+
+

+The residuals \( \boldsymbol{\epsilon} \) are in turn given by +$$ +\boldsymbol{\epsilon} = \boldsymbol{y}-\boldsymbol{\tilde{y}} = \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}, +$$ + +and with +$$ +\boldsymbol{X}^T\left( \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)= 0, +$$ + +we have +$$ +\boldsymbol{X}^T\boldsymbol{\epsilon}=\boldsymbol{X}^T\left( \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)= 0, +$$ + +meaning that the solution for \( \boldsymbol{\beta} \) is the one which minimizes the residuals. Later we will link this with the maximum likelihood approach. + +

+

+
+ + +

+Let us now return to our nuclear binding energies and simply code the above equations. + +

+

+ +

+ + +
+ + + + + + + +
+ +
+ + + + + + diff --git a/doc/src/Regression/._Regression-bs016.html b/doc/src/Regression/._Regression-bs016.html new file mode 100644 index 000000000..45a289cd6 --- /dev/null +++ b/doc/src/Regression/._Regression-bs016.html @@ -0,0 +1,474 @@ + + + + + + + + +Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis + + + + + + + + + + + + + + + + + + + + + + + +
+
+ + Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis +
+ +
+ +
+
+ + +
+ +

 

 

 

+ + + + +

Own code for Ordinary Least Squares

+ +

+It is rather straightforward to implement the matrix inversion and obtain the parameters \( \boldsymbol{\beta} \). After having defined the matrix \( \boldsymbol{X} \) we simply need to +write +

+ + +

# matrix inversion to find beta
+beta = np.linalg.inv(X.T.dot(X)).dot(X.T).dot(Energies)
+# and then make the prediction
+ytilde = X @ beta
+
+

+Alternatively, you can use the least squares functionality in Numpy as +

+ + +

fit = np.linalg.lstsq(X, Energies, rcond =None)[0]
+ytildenp = np.dot(fit,X.T)
+
+

+And finally we plot our fit with and compare with data +

+ + +

Masses['Eapprox']  = ytilde
+# Generate a plot comparing the experimental with the fitted values values.
+fig, ax = plt.subplots()
+ax.set_xlabel(r'$A = N + Z$')
+ax.set_ylabel(r'$E_\mathrm{bind}\,/\mathrm{MeV}$')
+ax.plot(Masses['A'], Masses['Ebinding'], alpha=0.7, lw=2,
+            label='Ame2016')
+ax.plot(Masses['A'], Masses['Eapprox'], alpha=0.7, lw=2, c='m',
+            label='Fit')
+ax.legend()
+save_fig("Masses2016OLS")
+plt.show()
+
+

+

+ +

+ + +
+ + + + + + + +
+ +
+ + + + + + diff --git a/doc/src/Regression/._Regression-bs017.html b/doc/src/Regression/._Regression-bs017.html new file mode 100644 index 000000000..0ee4b2985 --- /dev/null +++ b/doc/src/Regression/._Regression-bs017.html @@ -0,0 +1,473 @@ + + + + + + + + +Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis + + + + + + + + + + + + + + + + + + + + + + + +
+
+ + Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis +
+ +
+ +
+
+ + +
+ +

 

 

 

+ + + + +

Adding error analysis and training set up

+ +

+We can easily test our fit by computing the \( R2 \) score that we discussed in connection with the functionality of _Scikit_Learn_ in the introductory slides. +Since we are not using _Scikit-Learn here we can define our own \( R2 \) function as +

+ + +

def R2(y_data, y_model):
+    return 1 - np.sum((y_data - y_model) ** 2) / np.sum((y_data - np.mean(y_model)) ** 2)
+
+

+and we would be using it as +

+ + +

print(R2(Energies,ytilde))
+
+

+We can easily add our MSE score as +

+ + +

def MSE(y_data,y_model):
+    n = np.size(y_model)
+    return np.sum((y_data-y_model)**2)/n
+
+print(MSE(Energies,ytilde))
+
+

+and finally the relative error as +

+ + +

def RelativeError(y_data,y_model):
+    return abs((y_data-y_model)/y_data)
+print(RelativeError(Energies, ytilde))
+
+

+

+ +

+ + +
+ + + + + + + +
+ +
+ + + + + + diff --git a/doc/src/Regression/._Regression-bs018.html b/doc/src/Regression/._Regression-bs018.html new file mode 100644 index 000000000..7a9d6f4d6 --- /dev/null +++ b/doc/src/Regression/._Regression-bs018.html @@ -0,0 +1,465 @@ + + + + + + + + +Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis + + + + + + + + + + + + + + + + + + + + + + + +
+
+ + Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis +
+ +
+ +
+
+ + +
+ +

 

 

 

+ + + + +

The \( \chi^2 \) function

+
+
+

+ +

+Normally, the response (dependent or outcome) variable \( y_i \) is the +outcome of a numerical experiment or another type of experiment and is +thus only an approximation to the true value. It is then always +accompanied by an error estimate, often limited to a statistical error +estimate given by the standard deviation discussed earlier. In the +discussion here we will treat \( y_i \) as our exact value for the +response variable. + +

+Introducing the standard deviation \( \sigma_i \) for each measurement +\( y_i \), we define now the \( \chi^2 \) function (omitting the \( 1/n \) term) +as + +$$ +\chi^2(\boldsymbol{\beta})=\frac{1}{n}\sum_{i=0}^{n-1}\frac{\left(y_i-\tilde{y}_i\right)^2}{\sigma_i^2}=\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{\tilde{y}}\right)^T\frac{1}{\boldsymbol{\Sigma^2}}\left(\boldsymbol{y}-\boldsymbol{\tilde{y}}\right)\right\}, +$$ + +where the matrix \( \boldsymbol{\Sigma} \) is a diagonal matrix with \( \sigma_i \) as matrix elements. + +

+

+
+ + +

+

+ +

+ + +
+ + + + + + + +
+ +
+ + + + + + diff --git a/doc/src/Regression/._Regression-bs019.html b/doc/src/Regression/._Regression-bs019.html new file mode 100644 index 000000000..3be490b51 --- /dev/null +++ b/doc/src/Regression/._Regression-bs019.html @@ -0,0 +1,461 @@ + + + + + + + + +Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis + + + + + + + + + + + + + + + + + + + + + + + +
+
+ + Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis +
+ +
+ +
+
+ + +
+ +

 

 

 

+ + + + +

The \( \chi^2 \) function

+
+
+

+ +

+In order to find the parameters \( \beta_i \) we will then minimize the spread of \( \chi^2(\boldsymbol{\beta}) \) by requiring +$$ +\frac{\partial \chi^2(\boldsymbol{\beta})}{\partial \beta_j} = \frac{\partial }{\partial \beta_j}\left[ \frac{1}{n}\sum_{i=0}^{n-1}\left(\frac{y_i-\beta_0x_{i,0}-\beta_1x_{i,1}-\beta_2x_{i,2}-\dots-\beta_{n-1}x_{i,n-1}}{\sigma_i}\right)^2\right]=0, +$$ + +which results in +$$ +\frac{\partial \chi^2(\boldsymbol{\beta})}{\partial \beta_j} = -\frac{2}{n}\left[ \sum_{i=0}^{n-1}\frac{x_{ij}}{\sigma_i}\left(\frac{y_i-\beta_0x_{i,0}-\beta_1x_{i,1}-\beta_2x_{i,2}-\dots-\beta_{n-1}x_{i,n-1}}{\sigma_i}\right)\right]=0, +$$ + +or in a matrix-vector form as +$$ +\frac{\partial \chi^2(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}} = 0 = \boldsymbol{A}^T\left( \boldsymbol{b}-\boldsymbol{A}\boldsymbol{\beta}\right). +$$ + +where we have defined the matrix \( \boldsymbol{A} =\boldsymbol{X}/\boldsymbol{\Sigma} \) with matrix elements \( a_{ij} = x_{ij}/\sigma_i \) and the vector \( \boldsymbol{b} \) with elements \( b_i = y_i/\sigma_i \). +

+
+ + +

+

+ +

+ + +
+ + + + + + + +
+ +
+ + + + + + diff --git a/doc/src/Regression/._Regression-bs020.html b/doc/src/Regression/._Regression-bs020.html new file mode 100644 index 000000000..ea9d4bb03 --- /dev/null +++ b/doc/src/Regression/._Regression-bs020.html @@ -0,0 +1,459 @@ + + + + + + + + +Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis + + + + + + + + + + + + + + + + + + + + + + + +
+
+ + Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis +
+ +
+ +
+
+ + +
+ +

 

 

 

+ + + + +

The \( \chi^2 \) function

+
+
+

+ +

+We can rewrite +$$ +\frac{\partial \chi^2(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}} = 0 = \boldsymbol{A}^T\left( \boldsymbol{b}-\boldsymbol{A}\boldsymbol{\beta}\right), +$$ + +as +$$ +\boldsymbol{A}^T\boldsymbol{b} = \boldsymbol{A}^T\boldsymbol{A}\boldsymbol{\beta}, +$$ + +and if the matrix \( \boldsymbol{A}^T\boldsymbol{A} \) is invertible we have the solution +$$ +\boldsymbol{\beta} =\left(\boldsymbol{A}^T\boldsymbol{A}\right)^{-1}\boldsymbol{A}^T\boldsymbol{b}. +$$ +

+
+ + +

+

+ +

+ + +
+ + + + + + + +
+ +
+ + + + + + diff --git a/doc/src/Regression/._Regression-bs021.html b/doc/src/Regression/._Regression-bs021.html new file mode 100644 index 000000000..89068013d --- /dev/null +++ b/doc/src/Regression/._Regression-bs021.html @@ -0,0 +1,464 @@ + + + + + + + + +Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis + + + + + + + + + + + + + + + + + + + + + + + +
+
+ + Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis +
+ +
+ +
+
+ + +
+ +

 

 

 

+ + + + +

The \( \chi^2 \) function

+
+
+

+ +

+If we then introduce the matrix +$$ +\boldsymbol{H} = \left(\boldsymbol{A}^T\boldsymbol{A}\right)^{-1}, +$$ + +we have then the following expression for the parameters \( \beta_j \) (the matrix elements of \( \boldsymbol{H} \) are \( h_{ij} \)) +$$ +\beta_j = \sum_{k=0}^{p-1}h_{jk}\sum_{i=0}^{n-1}\frac{y_i}{\sigma_i}\frac{x_{ik}}{\sigma_i} = \sum_{k=0}^{p-1}h_{jk}\sum_{i=0}^{n-1}b_ia_{ik} +$$ + +We state without proof the expression for the uncertainty in the parameters \( \beta_j \) as (we leave this as an exercise) +$$ +\sigma^2(\beta_j) = \sum_{i=0}^{n-1}\sigma_i^2\left( \frac{\partial \beta_j}{\partial y_i}\right)^2, +$$ + +resulting in +$$ +\sigma^2(\beta_j) = \left(\sum_{k=0}^{p-1}h_{jk}\sum_{i=0}^{n-1}a_{ik}\right)\left(\sum_{l=0}^{p-1}h_{jl}\sum_{m=0}^{n-1}a_{ml}\right) = h_{jj}! +$$ +

+
+ + +

+

+ +

+ + +
+ + + + + + + +
+ +
+ + + + + + diff --git a/doc/src/Regression/._Regression-bs022.html b/doc/src/Regression/._Regression-bs022.html new file mode 100644 index 000000000..968b64515 --- /dev/null +++ b/doc/src/Regression/._Regression-bs022.html @@ -0,0 +1,457 @@ + + + + + + + + +Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis + + + + + + + + + + + + + + + + + + + + + + + +
+
+ + Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis +
+ +
+ +
+
+ + +
+ +

 

 

 

+ + + + +

The \( \chi^2 \) function

+
+
+

+The first step here is to approximate the function \( y \) with a first-order polynomial, that is we write +$$ +y=y(x) \rightarrow y(x_i) \approx \beta_0+\beta_1 x_i. +$$ + +By computing the derivatives of \( \chi^2 \) with respect to \( \beta_0 \) and \( \beta_1 \) show that these are given by +$$ +\frac{\partial \chi^2(\boldsymbol{\beta})}{\partial \beta_0} = -2\left[ \frac{1}{n}\sum_{i=0}^{n-1}\left(\frac{y_i-\beta_0-\beta_1x_{i}}{\sigma_i^2}\right)\right]=0, +$$ + +and +$$ +\frac{\partial \chi^2(\boldsymbol{\beta})}{\partial \beta_1} = -\frac{2}{n}\left[ \sum_{i=0}^{n-1}x_i\left(\frac{y_i-\beta_0-\beta_1x_{i}}{\sigma_i^2}\right)\right]=0. +$$ +

+
+ + +

+

+ +

+ + +
+ + + + + + + +
+ +
+ + + + + + diff --git a/doc/src/Regression/._Regression-bs023.html b/doc/src/Regression/._Regression-bs023.html new file mode 100644 index 000000000..301726fe5 --- /dev/null +++ b/doc/src/Regression/._Regression-bs023.html @@ -0,0 +1,491 @@ + + + + + + + + +Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis + + + + + + + + + + + + + + + + + + + + + + + +
+
+ + Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis +
+ +
+ +
+
+ + +
+ +

 

 

 

+ + + + +

The \( \chi^2 \) function

+
+
+

+ +

+For a linear fit (a first-order polynomial) we don't need to invert a matrix!! +Defining +$$ +\gamma = \sum_{i=0}^{n-1}\frac{1}{\sigma_i^2}, +$$ + + +$$ +\gamma_x = \sum_{i=0}^{n-1}\frac{x_{i}}{\sigma_i^2}, +$$ + + +$$ +\gamma_y = \sum_{i=0}^{n-1}\left(\frac{y_i}{\sigma_i^2}\right), +$$ + + +$$ +\gamma_{xx} = \sum_{i=0}^{n-1}\frac{x_ix_{i}}{\sigma_i^2}, +$$ + + +$$ +\gamma_{xy} = \sum_{i=0}^{n-1}\frac{y_ix_{i}}{\sigma_i^2}, +$$ + +

+we obtain + +$$ +\beta_0 = \frac{\gamma_{xx}\gamma_y-\gamma_x\gamma_y}{\gamma\gamma_{xx}-\gamma_x^2}, +$$ + + +$$ +\beta_1 = \frac{\gamma_{xy}\gamma-\gamma_x\gamma_y}{\gamma\gamma_{xx}-\gamma_x^2}. +$$ + +

+This approach (different linear and non-linear regression) suffers +often from both being underdetermined and overdetermined in the +unknown coefficients \( \beta_i \). A better approach is to use the +Singular Value Decomposition (SVD) method discussed below. Or using +Lasso and Ridge regression. See below. + +

+

+
+ + +

+

+ +

+ + +
+ + + + + + + +
+ +
+ + + + + + diff --git a/doc/src/Regression/._Regression-bs024.html b/doc/src/Regression/._Regression-bs024.html new file mode 100644 index 000000000..2a1d4586f --- /dev/null +++ b/doc/src/Regression/._Regression-bs024.html @@ -0,0 +1,457 @@ + + + + + + + + +Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis + + + + + + + + + + + + + + + + + + + + + + + +
+
+ + Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis +
+ +
+ +
+
+ + +
+ +

 

 

 

+ + + + +

Fitting an Equation of State for Dense Nuclear Matter

+ +

+Before we continue, let us introduce yet another example. We are going to fit the +nuclear equation of state using results from many-body calculations. +The equation of state we have made available here, as function of +density, has been derived using modern nucleon-nucleon potentials with +the addition of three-body +forces. This +time the file is presented as a standard csv file. + +

+The beginning of the Python code here is similar to what you have seen before, +with the same initializations and declarations. We use also pandas +again, rather extensively in order to organize our data. + +

+The difference now is that we use Scikit-Learn's regression tools +instead of our own matrix inversion implementation. Furthermore, we +sneak in Ridge regression (to be discussed below) which includes a +hyperparameter \( \lambda \), also to be explained below. + +

+

+ +

+ + +
+ + + + + + + +
+ +
+ + + + + + diff --git a/doc/src/Regression/._Regression-bs025.html b/doc/src/Regression/._Regression-bs025.html new file mode 100644 index 000000000..336b45fca --- /dev/null +++ b/doc/src/Regression/._Regression-bs025.html @@ -0,0 +1,535 @@ + + + + + + + + +Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis + + + + + + + + + + + + + + + + + + + + + + + +
+
+ + Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis +
+ +
+ +
+
+ + +
+ +

 

 

 

+ + + + +

The code

+ +

+ + +

# Common imports
+import os
+import numpy as np
+import pandas as pd
+import matplotlib.pyplot as plt
+import matplotlib.pyplot as plt
+import sklearn.linear_model as skl
+from sklearn.metrics import mean_squared_error, r2_score, mean_absolute_error
+
+# Where to save the figures and data files
+PROJECT_ROOT_DIR = "Results"
+FIGURE_ID = "Results/FigureFiles"
+DATA_ID = "DataFiles/"
+
+if not os.path.exists(PROJECT_ROOT_DIR):
+    os.mkdir(PROJECT_ROOT_DIR)
+
+if not os.path.exists(FIGURE_ID):
+    os.makedirs(FIGURE_ID)
+
+if not os.path.exists(DATA_ID):
+    os.makedirs(DATA_ID)
+
+def image_path(fig_id):
+    return os.path.join(FIGURE_ID, fig_id)
+
+def data_path(dat_id):
+    return os.path.join(DATA_ID, dat_id)
+
+def save_fig(fig_id):
+    plt.savefig(image_path(fig_id) + ".png", format='png')
+
+infile = open(data_path("EoS.csv"),'r')
+
+# Read the EoS data as  csv file and organize the data into two arrays with density and energies
+EoS = pd.read_csv(infile, names=('Density', 'Energy'))
+EoS['Energy'] = pd.to_numeric(EoS['Energy'], errors='coerce')
+EoS = EoS.dropna()
+Energies = EoS['Energy']
+Density = EoS['Density']
+#  The design matrix now as function of various polytrops
+X = np.zeros((len(Density),4))
+X[:,3] = Density**(4.0/3.0)
+X[:,2] = Density
+X[:,1] = Density**(2.0/3.0)
+X[:,0] = 1
+
+# We use now Scikit-Learn's linear regressor and ridge regressor
+# OLS part
+clf = skl.LinearRegression().fit(X, Energies)
+ytilde = clf.predict(X)
+EoS['Eols']  = ytilde
+# The mean squared error                               
+print("Mean squared error: %.2f" % mean_squared_error(Energies, ytilde))
+# Explained variance score: 1 is perfect prediction                                 
+print('Variance score: %.2f' % r2_score(Energies, ytilde))
+# Mean absolute error                                                           
+print('Mean absolute error: %.2f' % mean_absolute_error(Energies, ytilde))
+print(clf.coef_, clf.intercept_)
+
+# The Ridge regression with a hyperparameter lambda = 0.1
+_lambda = 0.1
+clf_ridge = skl.Ridge(alpha=_lambda).fit(X, Energies)
+yridge = clf_ridge.predict(X)
+EoS['Eridge']  = yridge
+# The mean squared error                               
+print("Mean squared error: %.2f" % mean_squared_error(Energies, yridge))
+# Explained variance score: 1 is perfect prediction                                 
+print('Variance score: %.2f' % r2_score(Energies, yridge))
+# Mean absolute error                                                           
+print('Mean absolute error: %.2f' % mean_absolute_error(Energies, yridge))
+print(clf_ridge.coef_, clf_ridge.intercept_)
+
+fig, ax = plt.subplots()
+ax.set_xlabel(r'$\rho[\mathrm{fm}^{-3}]$')
+ax.set_ylabel(r'Energy per particle')
+ax.plot(EoS['Density'], EoS['Energy'], alpha=0.7, lw=2,
+            label='Theoretical data')
+ax.plot(EoS['Density'], EoS['Eols'], alpha=0.7, lw=2, c='m',
+            label='OLS')
+ax.plot(EoS['Density'], EoS['Eridge'], alpha=0.7, lw=2, c='g',
+            label='Ridge $\lambda = 0.1$')
+ax.legend()
+save_fig("EoSfitting")
+plt.show()
+
+

+The above simple polynomial in density \( \rho \) gives an excellent fit +to the data. Can you give an interpretation of the various powers of \( \rho \)? + +

+We note also that there is a small deviation between the +standard OLS and the Ridge regression at higher densities. We discuss this in more detail +below. + +

+

+ +

+ + +
+ + + + + + + +
+ +
+ + + + + + diff --git a/doc/src/Regression/._Regression-bs026.html b/doc/src/Regression/._Regression-bs026.html new file mode 100644 index 000000000..ca22d9f9b --- /dev/null +++ b/doc/src/Regression/._Regression-bs026.html @@ -0,0 +1,517 @@ + + + + + + + + +Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis + + + + + + + + + + + + + + + + + + + + + + + +
+
+ + Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis +
+ +
+ +
+
+ + +
+ +

 

 

 

+ + + + +

Splitting our Data in Training and Test data

+ +

+It is normal in essentially all Machine Learning studies to split the +data in a training set and a test set (sometimes also an additional +validation set). Scikit-Learn has an own function for this. There +is no explicit recipe for how much data should be included as training +data and say test data. An accepted rule of thumb is to use +approximately \( 2/3 \) to \( 4/5 \) of the data as training data. We will +postpone a discussion of this splitting to the end of these notes and +our discussion of the so-called bias-variance tradeoff. Here we +limit ourselves to repeat the above equation of state fitting example +but now splitting the data into a training set and a test set. + +

+ + +

import os
+import numpy as np
+import pandas as pd
+import matplotlib.pyplot as plt
+from sklearn.model_selection import train_test_split
+# Where to save the figures and data files
+PROJECT_ROOT_DIR = "Results"
+FIGURE_ID = "Results/FigureFiles"
+DATA_ID = "DataFiles/"
+
+if not os.path.exists(PROJECT_ROOT_DIR):
+    os.mkdir(PROJECT_ROOT_DIR)
+
+if not os.path.exists(FIGURE_ID):
+    os.makedirs(FIGURE_ID)
+
+if not os.path.exists(DATA_ID):
+    os.makedirs(DATA_ID)
+
+def image_path(fig_id):
+    return os.path.join(FIGURE_ID, fig_id)
+
+def data_path(dat_id):
+    return os.path.join(DATA_ID, dat_id)
+
+def save_fig(fig_id):
+    plt.savefig(image_path(fig_id) + ".png", format='png')
+
+def R2(y_data, y_model):
+    return 1 - np.sum((y_data - y_model) ** 2) / np.sum((y_data - np.mean(y_model)) ** 2)
+def MSE(y_data,y_model):
+    n = np.size(y_model)
+    return np.sum((y_data-y_model)**2)/n
+
+infile = open(data_path("EoS.csv"),'r')
+
+# Read the EoS data as  csv file and organized into two arrays with density and energies
+EoS = pd.read_csv(infile, names=('Density', 'Energy'))
+EoS['Energy'] = pd.to_numeric(EoS['Energy'], errors='coerce')
+EoS = EoS.dropna()
+Energies = EoS['Energy']
+Density = EoS['Density']
+#  The design matrix now as function of various polytrops
+X = np.zeros((len(Density),5))
+X[:,0] = 1
+X[:,1] = Density**(2.0/3.0)
+X[:,2] = Density
+X[:,3] = Density**(4.0/3.0)
+X[:,4] = Density**(5.0/3.0)
+# We split the data in test and training data
+X_train, X_test, y_train, y_test = train_test_split(X, Energies, test_size=0.2)
+# matrix inversion to find beta
+beta = np.linalg.inv(X_train.T.dot(X_train)).dot(X_train.T).dot(y_train)
+# and then make the prediction
+ytilde = X_train @ beta
+print("Training R2")
+print(R2(y_train,ytilde))
+print("Training MSE")
+print(MSE(y_train,ytilde))
+ypredict = X_test @ beta
+print("Test R2")
+print(R2(y_test,ypredict))
+print("Test MSE")
+print(MSE(y_test,ypredict))
+
+

+

+ +

+ + +
+ + + + + + + +
+ +
+ + + + + + diff --git a/doc/src/Regression/._Regression-bs027.html b/doc/src/Regression/._Regression-bs027.html new file mode 100644 index 000000000..50e915fc9 --- /dev/null +++ b/doc/src/Regression/._Regression-bs027.html @@ -0,0 +1,464 @@ + + + + + + + + +Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis + + + + + + + + + + + + + + + + + + + + + + + +
+
+ + Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis +
+ +
+ +
+
+ + +
+ +

 

 

 

+ + + + +

The singular value decomposition

+ +

+

+
+

+ +

+The examples we have looked at so far are cases where we normally can +invert the matrix \( \boldsymbol{X}^T\boldsymbol{X} \). Using a polynomial expansion as we +did both for the masses and the fitting of the equation of state, +leads to row vectors of the design matrix which are essentially +orthogonal due to the polynomial character of our model. This may +however not the be case in general and a standard matrix inversion +algorithm based on say LU decomposition may lead to singularities. We will see an example of this below when we try to fit +the coupling constant of the widely used Ising model. +There is however a way to partially circumvent this problem and also gain some insight about the ordinary least squares approach. + +

+This is given by the Singular Value Decomposition algorithm, perhaps +the most powerful linear algebra algorithm. Let us look at a +different example where we may have problems with the standard matrix +inversion algorithm. Thereafter we dive into the math of the SVD. + +

+

+
+ + +

+

+ +

+ + +
+ + + + + + + +
+ +
+ + + + + + diff --git a/doc/src/Regression/._Regression-bs028.html b/doc/src/Regression/._Regression-bs028.html new file mode 100644 index 000000000..39cfa7990 --- /dev/null +++ b/doc/src/Regression/._Regression-bs028.html @@ -0,0 +1,489 @@ + + + + + + + + +Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis + + + + + + + + + + + + + + + + + + + + + + + +
+
+ + Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis +
+ +
+ +
+
+ + +
+ +

 

 

 

+ + + + +

The Ising model

+ +

+The one-dimensional Ising model with nearest neighbor interaction, no +external field and a constant coupling constant \( J \) is given by + +$$ +\begin{align} + H = -J \sum_{k}^L s_k s_{k + 1}, +\tag{1} +\end{align} +$$ + +

+where \( s_i \in \{-1, 1\} \) and \( s_{N + 1} = s_1 \). The number of spins +in the system is determined by \( L \). For the one-dimensional system +there is no phase transition. + +

+We will look at a system of \( L = 40 \) spins with a coupling constant of +\( J = 1 \). To get enough training data we will generate 10000 states +with their respective energies. + +

+ + +

import numpy as np
+import matplotlib.pyplot as plt
+from mpl_toolkits.axes_grid1 import make_axes_locatable
+import seaborn as sns
+import scipy.linalg as scl
+from sklearn.model_selection import train_test_split
+import tqdm
+sns.set(color_codes=True)
+cmap_args=dict(vmin=-1., vmax=1., cmap='seismic')
+
+L = 40
+n = int(1e4)
+
+spins = np.random.choice([-1, 1], size=(n, L))
+J = 1.0
+
+energies = np.zeros(n)
+
+for i in range(n):
+    energies[i] = - J * np.dot(spins[i], np.roll(spins[i], 1))
+
+

+Here we use ordinary least squares +regression to predict the energy for the nearest neighbor +one-dimensional Ising model on a ring, i.e., the endpoints wrap +around. We will use linear regression to fit a value for +the coupling constant to achieve this. + +

+

+ +

+ + +
+ + + + + + + +
+ +
+ + + + + + diff --git a/doc/src/Regression/._Regression-bs029.html b/doc/src/Regression/._Regression-bs029.html new file mode 100644 index 000000000..336d2cacf --- /dev/null +++ b/doc/src/Regression/._Regression-bs029.html @@ -0,0 +1,482 @@ + + + + + + + + +Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis + + + + + + + + + + + + + + + + + + + + + + + +
+
+ + Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis +
+ +
+ +
+
+ + +
+ +

 

 

 

+ + + + +

Reformulating the problem to suit regression

+ +

+A more general form for the one-dimensional Ising model is + +$$ +\begin{align} + H = - \sum_j^L \sum_k^L s_j s_k J_{jk}. +\tag{2} +\end{align} +$$ + +

+Here we allow for interactions beyond the nearest neighbors and a state dependent +coupling constant. This latter expression can be formulated as +a matrix-product +$$ +\begin{align} + \boldsymbol{H} = \boldsymbol{X} J, +\tag{3} +\end{align} +$$ + +

+where \( X_{jk} = s_j s_k \) and \( J \) is a matrix which consists of the +elements \( -J_{jk} \). This form of writing the energy fits perfectly +with the form utilized in linear regression, that is + +$$ +\begin{align} + \boldsymbol{y} = \boldsymbol{X}\boldsymbol{\beta} + \boldsymbol{\epsilon}, +\tag{4} +\end{align} +$$ + +

+We split the data in training and test data as discussed in the previous example + +

+ + +

X = np.zeros((n, L ** 2))
+for i in range(n):
+    X[i] = np.outer(spins[i], spins[i]).ravel()
+y = energies
+X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)
+
+

+

+ +

+ + +
+ + + + + + + +
+ +
+ + + + + + diff --git a/doc/src/Regression/._Regression-bs030.html b/doc/src/Regression/._Regression-bs030.html new file mode 100644 index 000000000..559929638 --- /dev/null +++ b/doc/src/Regression/._Regression-bs030.html @@ -0,0 +1,480 @@ + + + + + + + + +Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis + + + + + + + + + + + + + + + + + + + + + + + +
+
+ + Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis +
+ +
+ +
+
+ + +
+ +

 

 

 

+ + + + +

Linear regression

+ +

+In the ordinary least squares method we choose the cost function + +$$ +\begin{align} + C(\boldsymbol{X}, \boldsymbol{\beta})= \frac{1}{n}\left\{(\boldsymbol{X}\boldsymbol{\beta} - \boldsymbol{y})^T(\boldsymbol{X}\boldsymbol{\beta} - \boldsymbol{y})\right\}. +\tag{5} +\end{align} +$$ + +

+We then find the extremal point of \( C \) by taking the derivative with respect to \( \boldsymbol{\beta} \) as discussed above. +This yields the expression for \( \boldsymbol{\beta} \) to be + +$$ + \boldsymbol{\beta} = \frac{\boldsymbol{X}^T \boldsymbol{y}}{\boldsymbol{X}^T \boldsymbol{X}}, +$$ + +

+which immediately imposes some requirements on \( \boldsymbol{X} \) as there must exist +an inverse of \( \boldsymbol{X}^T \boldsymbol{X} \). If the expression we are modeling contains an +intercept, i.e., a constant term, we must make sure that the +first column of \( \boldsymbol{X} \) consists of \( 1 \). We do this here + +

+ + +

X_train_own = np.concatenate(
+    (np.ones(len(X_train))[:, np.newaxis], X_train),
+    axis=1
+)
+X_test_own = np.concatenate(
+    (np.ones(len(X_test))[:, np.newaxis], X_test),
+    axis=1
+)
+
+

+ + +

def ols_inv(x: np.ndarray, y: np.ndarray) -> np.ndarray:
+    return scl.inv(x.T @ x) @ (x.T @ y)
+beta = ols_inv(X_train_own, y_train)
+
+

+

+ +

+ + +
+ + + + + + + +
+ +
+ + + + + + diff --git a/doc/src/Regression/._Regression-bs031.html b/doc/src/Regression/._Regression-bs031.html new file mode 100644 index 000000000..0a77e9794 --- /dev/null +++ b/doc/src/Regression/._Regression-bs031.html @@ -0,0 +1,519 @@ + + + + + + + + +Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis + + + + + + + + + + + + + + + + + + + + + + + +
+
+ + Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis +
+ +
+ +
+
+ + +
+ +

 

 

 

+ + + + +

Singular Value decomposition

+ +

+Doing the inversion directly turns out to be a bad idea since the matrix +\( \boldsymbol{X}^T\boldsymbol{X} \) is singular. An alternative approach is to use the singular +value decomposition. Using the definition of the Moore-Penrose +pseudoinverse we can write the equation for \( \boldsymbol{\beta} \) as + +$$ + \boldsymbol{\beta} = \boldsymbol{X}^{+}\boldsymbol{y}, +$$ + +

+where the pseudoinverse of \( \boldsymbol{X} \) is given by + +$$ + \boldsymbol{X}^{+} = \frac{\boldsymbol{X}^T}{\boldsymbol{X}^T\boldsymbol{X}}. +$$ + +

+Using singular value decomposition we can decompose the matrix \( \boldsymbol{X} = \boldsymbol{U}\boldsymbol{\Sigma} \boldsymbol{V}^T \), +where \( \boldsymbol{U} \) and \( \boldsymbol{V} \) are orthogonal(unitary) matrices and \( \boldsymbol{\Sigma} \) contains the singular values (more details below). +where \( X^{+} = V\Sigma^{+} U^T \). This reduces the equation for +\( \omega \) to +$$ +\begin{align} + \boldsymbol{\beta} = \boldsymbol{V}\boldsymbol{\Sigma}^{+} \boldsymbol{U}^T \boldsymbol{y}. +\tag{6} +\end{align} +$$ + +

+Note that solving this equation by actually doing the pseudoinverse +(which is what we will do) is not a good idea as this operation scales +as \( \mathcal{O}(n^3) \), where \( n \) is the number of elements in a +general matrix. Instead, doing \( QR \)-factorization and solving the +linear system as an equation would reduce this down to +\( \mathcal{O}(n^2) \) operations. + +

+ + +

def ols_svd(x: np.ndarray, y: np.ndarray) -> np.ndarray:
+    u, s, v = scl.svd(x)
+    return v.T @ scl.pinv(scl.diagsvd(s, u.shape[0], v.shape[0])) @ u.T @ y
+
+

+ + +

beta = ols_svd(X_train_own,y_train)
+
+

+When extracting the \( J \)-matrix we need to make sure that we remove the intercept, as is done here + +

+ + +

J = beta[1:].reshape(L, L)
+
+

+A way of looking at the coefficients in \( J \) is to plot the matrices as images. + +

+ + +

fig = plt.figure(figsize=(20, 14))
+im = plt.imshow(J, **cmap_args)
+plt.title("OLS", fontsize=18)
+plt.xticks(fontsize=18)
+plt.yticks(fontsize=18)
+cb = fig.colorbar(im)
+cb.ax.set_yticklabels(cb.ax.get_yticklabels(), fontsize=18)
+plt.show()
+
+

+It is interesting to note that OLS +considers both \( J_{j, j + 1} = -0.5 \) and \( J_{j, j - 1} = -0.5 \) as +valid matrix elements for \( J \). +In our discussion below on hyperparameters and Ridge and Lasso regression we will see that +this problem can be removed, partly and only with Lasso regression. + +

+In this case our matrix inversion was actually possible. The obvious question now is what is the mathematics behind the SVD? + +

+

+ +

+ + +
+ + + + + + + +
+ +
+ + + + + + diff --git a/doc/src/Regression/._Regression-bs032.html b/doc/src/Regression/._Regression-bs032.html new file mode 100644 index 000000000..e2d8e0601 --- /dev/null +++ b/doc/src/Regression/._Regression-bs032.html @@ -0,0 +1,483 @@ + + + + + + + + +Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis + + + + + + + + + + + + + + + + + + + + + + + +
+
+ + Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis +
+ +
+ +
+
+ + +
+ +

 

 

 

+ + + + +

Linear Regression Problems

+ +

+One of the typical problems we encounter with linear regression, in particular +when the matrix \( \boldsymbol{X} \) (our so-called design matrix) is high-dimensional, +are problems with near singular or singular matrices. The column vectors of \( \boldsymbol{X} \) +may be linearly dependent, normally referred to as super-collinearity. +This means that the matrix may be rank deficient and it is basically impossible to +to model the data using linear regression. As an example, consider the matrix +$$ +\begin{align*} +\mathbf{X} & = \left[ +\begin{array}{rrr} +1 & -1 & 2 +\\ +1 & 0 & 1 +\\ +1 & 2 & -1 +\\ +1 & 1 & 0 +\end{array} \right] +\end{align*} +$$ + +

+The columns of \( \boldsymbol{X} \) are linearly dependent. We see this easily since the +the first column is the row-wise sum of the other two columns. The rank (more correct, +the column rank) of a matrix is the dimension of the space spanned by the +column vectors. Hence, the rank of \( \mathbf{X} \) is equal to the number +of linearly independent columns. In this particular case the matrix has rank 2. + +

+Super-collinearity of an \( (n \times p) \)-dimensional design matrix \( \mathbf{X} \) implies +that the inverse of the matrix \( \boldsymbol{X}^T\boldsymbol{x} \) (the matrix we need to invert to solve the linear regression equations) is non-invertible. If we have a square matrix that does not have an inverse, we say this matrix singular. The example here demonstrates this +$$ +\begin{align*} +\boldsymbol{X} & = \left[ +\begin{array}{rr} +1 & -1 +\\ +1 & -1 +\end{array} \right]. +\end{align*} +$$ + +We see easily that \( \mbox{det}(\boldsymbol{X}) = x_{11} x_{22} - x_{12} x_{21} = 1 \times (-1) - 1 \times (-1) = 0 \). Hence, \( \mathbf{X} \) is singular and its inverse is undefined. +This is equivalent to saying that the matrix \( \boldsymbol{X} \) has at least an eigenvalue which is zero. + +

+

+ +

+ + +
+ + + + + + + +
+ +
+ + + + + + diff --git a/doc/src/Regression/._Regression-bs033.html b/doc/src/Regression/._Regression-bs033.html new file mode 100644 index 000000000..07e2b51a9 --- /dev/null +++ b/doc/src/Regression/._Regression-bs033.html @@ -0,0 +1,460 @@ + + + + + + + + +Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis + + + + + + + + + + + + + + + + + + + + + + + +
+
+ + Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis +
+ +
+ +
+
+ + +
+ +

 

 

 

+ + + + +

Fixing the singularity

+ +

+If our design matrix \( \boldsymbol{X} \) which enters the linear regression problem +$$ +\begin{align} +\boldsymbol{\beta} & = (\boldsymbol{X}^{T} \boldsymbol{X})^{-1} \boldsymbol{X}^{T} \boldsymbol{y}, +\tag{7} +\end{align} +$$ + +has linearly dependent column vectors, we will not be able to compute the inverse +of \( \boldsymbol{X}^T\boldsymbol{X} \) and we cannot find the parameters (estimators) \( \beta_i \). +The estimators are only well-defined if \( (\boldsymbol{X}^{T}\boldsymbol{X})^{-1} \) exits. +This is more likely to happen when the matrix \( \boldsymbol{X} \) is high-dimensional. In this case it is likely to encounter a situation where +the regression parameters \( \beta_i \) cannot be estimated. + +

+A cheap ad hoc approach is simply to add a small diagonal component to the matrix to invert, that is we change +$$ +\boldsymbol{X}^{T} \boldsymbol{X} \rightarrow \boldsymbol{X}^{T} \boldsymbol{X}+\lambda \boldsymbol{I}, +$$ + +where \( \boldsymbol{I} \) is the identity matrix. When we discuss Ridge regression this is actually what we end up evaluating. The parameter \( \lambda \) is called a hyperparameter. More about this later. + +

+

+ +

+ + +
+ + + + + + + +
+ +
+ + + + + + diff --git a/doc/src/Regression/._Regression-bs034.html b/doc/src/Regression/._Regression-bs034.html new file mode 100644 index 000000000..de4b3316b --- /dev/null +++ b/doc/src/Regression/._Regression-bs034.html @@ -0,0 +1,471 @@ + + + + + + + + +Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis + + + + + + + + + + + + + + + + + + + + + + + +
+
+ + Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis +
+ +
+ +
+
+ + +
+ +

 

 

 

+ + + + +

Basic math of the SVD

+ +

+From standard linear algebra we know that a square matrix \( \boldsymbol{X} \) can be diagonalized if and only it is +a so-called normal matrix, that is if \( \boldsymbol{X}\in {\mathbb{R}}^{n\times n} \) +we have \( \boldsymbol{X}\boldsymbol{X}^T=\boldsymbol{X}^T\boldsymbol{X} \) or if \( \boldsymbol{X}\in {\mathbb{C}}^{n\times n} \) we have \( \boldsymbol{X}\boldsymbol{X}^{\dagger}=\boldsymbol{X}^{\dagger}\boldsymbol{X} \). +The matrix has then a set of eigenpairs + +$$ +(\lambda_1,\boldsymbol{u}_1),\dots, (\lambda_n,\boldsymbol{u}_n), +$$ + +and the eigenvalues are given by the diagonal matrix +$$ +\boldsymbol{\Sigma}=\mathrm{Diag}(\lambda_1, \dots,\lambda_n). +$$ + +The matrix \( \boldsymbol{X} \) can be written in terms of an orthogonal/unitary transformation \( \boldsymbol{U} \) +$$ +\boldsymbol{X} = \boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{V}^T, +$$ + +with \( \boldsymbol{U}\boldsymbol{U}^T=\boldsymbol{I} \) or \( \boldsymbol{U}\boldsymbol{U}^{\dagger}=\boldsymbol{I} \). + +

+Not all square matrices are diagonalizable. A matrix like the one discussed above +$$ +\boldsymbol{X} = \begin{bmatrix} +1& -1 \\ +1& -1\\ +\end{bmatrix} +$$ + +is not diagonalizable, it is a so-called defective matrix. It is easy to see that the condition +\( \boldsymbol{X}\boldsymbol{X}^T=\boldsymbol{X}^T\boldsymbol{X} \) is not fulfilled. + +

+

+ +

+ + +
+ + + + + + + +
+ +
+ + + + + + diff --git a/doc/src/Regression/._Regression-bs035.html b/doc/src/Regression/._Regression-bs035.html new file mode 100644 index 000000000..12ca2541d --- /dev/null +++ b/doc/src/Regression/._Regression-bs035.html @@ -0,0 +1,463 @@ + + + + + + + + +Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis + + + + + + + + + + + + + + + + + + + + + + + +
+
+ + Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis +
+ +
+ +
+
+ + +
+ +

 

 

 

+ + + + +

The SVD, a Fantastic Algorithm

+ +

+However, and this is the strength of the SVD algorithm, any general +matrix \( \boldsymbol{X} \) can be decomposed in terms of a diagonal matrix and +two orthogonal/unitary matrices. The Singular Value Decompostion +(SVD) theorem +states that a general \( m\times n \) matrix \( \boldsymbol{X} \) can be written in +terms of a diagonal matrix \( \boldsymbol{\Sigma} \) of dimensionality \( n\times n \) +and two orthognal matrices \( \boldsymbol{U} \) and \( \boldsymbol{V} \), where the first has +dimensionality \( m \times m \) and the last dimensionality \( n\times n \). +We have then + +$$ +\boldsymbol{X} = \boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{V}^T +$$ + +

+As an example, the above defective matrix can be decomposed as + +$$ +\boldsymbol{X} = \frac{1}{\sqrt{2}}\begin{bmatrix} 1& 1 \\ 1& -1\\ \end{bmatrix} \begin{bmatrix} 2& 0 \\ 0& 0\\ \end{bmatrix} \frac{1}{\sqrt{2}}\begin{bmatrix} 1& -1 \\ 1& 1\\ \end{bmatrix}=\boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{V}^T, +$$ + +

+with eigenvalues \( \sigma_1=2 \) and \( \sigma_2=0 \). +The SVD exits always! + +

+

+ +

+ + +
+ + + + + + + +
+ +
+ + + + + + diff --git a/doc/src/Regression/._Regression-bs036.html b/doc/src/Regression/._Regression-bs036.html new file mode 100644 index 000000000..f0d68b336 --- /dev/null +++ b/doc/src/Regression/._Regression-bs036.html @@ -0,0 +1,470 @@ + + + + + + + + +Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis + + + + + + + + + + + + + + + + + + + + + + + +
+
+ + Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis +
+ +
+ +
+
+ + +
+ +

 

 

 

+ + + + +

Another Example

+ +

+Consider the following matrix which can be SVD decomposed as + +$$ +\boldsymbol{X} = \frac{1}{15}\begin{bmatrix} 14 & 2\\ 4 & 22\\ 16 & 13\end{matrix}=\frac{1}{3}\begin{bmatrix} 1& 2 & 2 \\ 2& -1 & 1\\ 2 & 1& -2\end{bmatrix} \begin{bmatrix} 2& 0 \\ 0& 1\\ 0 & 0\end{bmatrix}\frac{1}{5}\begin{bmatrix} 3& 4 \\ 4& -3\end{bmatrix}=\boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{V}^T. +$$ + +

+This is a \( 3\times 2 \) matrix which is decomposed in terms of a +\( 3\times 3 \) matrix \( \boldsymbol{U} \), and a \( 2\times 2 \) matrix \( \boldsymbol{V} \). It is easy to see +that \( \boldsymbol{U} \) and \( \boldsymbol{V} \) are orthogonal (how?). + +

+And the SVD +decomposition (singular values) gives eigenvalues +\( \sigma_i\geq\sigma_{i+1} \) for all \( i \) and for dimensions larger than \( i=2 \), the +eigenvalues (singular values) are zero. + +

+In the general case, where our design matrix \( \boldsymbol{X} \) has dimension +\( n\times p \), the matrix is thus decomposed into an \( n\times n \) +orthogonal matrix \( \boldsymbol{U} \), a \( p\times p \) orthogonal matrix \( \boldsymbol{V} \) +and a diagonal matrix \( \boldsymbol{\Sigma} \) with \( r=\mathrm{min}(n,p) \) +singular values \( \sigma_i\lg 0 \) on the main diagonal and zeros filling +the rest of the matrix. There are at most \( p \) singular values +assuming that \( n > p \). In our regression examples for the nuclear +masses and the equation of state this is indeed the case, while for +the Ising model we have \( p > n \). These are often cases that lead to +near singular or singular matrices. + +

+The columns of \( \boldsymbol{U} \) are called the left singular vectors while the columns of \( \boldsymbol{V} \) are the right singular vectors. + +

+

+ +

+ + +
+ + + + + + + +
+ +
+ + + + + + diff --git a/doc/src/Regression/._Regression-bs037.html b/doc/src/Regression/._Regression-bs037.html new file mode 100644 index 000000000..87e2a3326 --- /dev/null +++ b/doc/src/Regression/._Regression-bs037.html @@ -0,0 +1,457 @@ + + + + + + + + +Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis + + + + + + + + + + + + + + + + + + + + + + + +
+
+ + Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis +
+ +
+ +
+
+ + +
+ +

 

 

 

+ + + + +

Economy-size SVD

+ +

+If we assume that \( n > p \), then our matrix \( \boldsymbol{U} \) has dimension \( n +\times n \). The last \( n-p \) columns of \( \boldsymbol{U} \) become however +irrelevant in our calculations since they are multiplied with the +zeros in \( \boldsymbol{\Sigma} \). + +

+The economy-size decomposition removes extra rows or columns of zeros +from the diagonal matrix of singular values, \( \boldsymbol{\Sigma} \), along with the columns +in either \( \boldsymbol{U} \) or \( \boldsymbol{V} \) that multiply those zeros in the expression. +Removing these zeros and columns can improve execution time +and reduce storage requirements without compromising the accuracy of +the decomposition. + +

+If \( n > p \), we keep only the first \( p \) columns of \( \boldsymbol{U} \) and \( \boldsymbol{\Sigma} \) has dimension \( p\times p \). +If \( p > n \), then only the first \( n \) columns of \( \boldsymbol{V} \) are computed and \( \boldsymbol{\Sigma} \) has dimension \( n\times n \). +The \( n=p \) case is obvious, we retain the full SVD. +In general the economy-size SVD leads to less FLOPS and still conserving the desired accuracy. + +

+

+ +

+ + +
+ + + + + + + +
+ +
+ + + + + + diff --git a/doc/src/Regression/._Regression-bs038.html b/doc/src/Regression/._Regression-bs038.html new file mode 100644 index 000000000..882badd18 --- /dev/null +++ b/doc/src/Regression/._Regression-bs038.html @@ -0,0 +1,484 @@ + + + + + + + + +Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis + + + + + + + + + + + + + + + + + + + + + + + +
+
+ + Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis +
+ +
+ +
+
+ + +
+ +

 

 

 

+ + + + +

Mathematical Properties

+ +

+There are several interesting mathematical properties which will be +relevant when we are going to discuss the differences between say +ordinary least squares (OLS) and Ridge regression. + +

+We have from OLS that the parameters of the linear approximation are given by +$$ +\boldsymbol{\tilde{y}} = \boldsymbol{X}\boldsymbol{\beta} = \boldsymbol{X}\left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}. +$$ + +

+The matrix to invert can be rewritten in terms of our SVD decomposition as + +$$ +\boldsymbol{X}^T\boldsymbol{X} = \boldsymbol{V}\boldsymbol{\Sigma}^T\boldsymbol{U}^T\boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{V}^T. +$$ + +Using the orthogonality properties of \( \boldsymbol{U} \) we have + +$$ +\boldsymbol{X}^T\boldsymbol{X} = \boldsymbol{V}\boldsymbol{\Sigma}^T\boldsymbol{\Sigma}\boldsymbol{V}^T = \boldsymbol{V}\boldsymbol{D}\boldsymbol{V}^T, +$$ + +with \( \boldsymbol{D} \) being a diagonal matrix with values along the diagonal given by the singular values squared. + +

+This means that +$$ +(\boldsymbol{X}^T\boldsymbol{X})\boldsymbol{V} = \boldsymbol{V}\boldsymbol{D}, +$$ + +that is the eigenvectors of \( (\boldsymbol{X}^T\boldsymbol{X}) \) are given by the columns of the right singular matrix of \( \boldsymbol{X} \) and the eigenvalues are the squared singular values. It is easy to show (show this) that +$$ +(\boldsymbol{X}\boldsymbol{X}^T)\boldsymbol{U} = \boldsymbol{U}\boldsymbol{D}, +$$ + +that is, the eigenvectors of \( (\boldsymbol{X}\boldsymbol{X})^T \) are the columns of the left singular matrix and the eigenvalues are the same. + +

+Going back to our OLS equation we have +$$ +\boldsymbol{X}\boldsymbol{\beta} = \boldsymbol{X}\left(\boldsymbol{V}\boldsymbol{D}\boldsymbol{V}^T \right)^{-1}\boldsymbol{X}^T\boldsymbol{y}=\boldsymbol{U\Sigma V^T}\left(\boldsymbol{V}\boldsymbol{D}\boldsymbol{V}^T \right)^{-1}(\boldsymbol{U\Sigma V^T})^T\boldsymbol{y}=\boldsymbol{U}\boldsymbol{U}^T\boldsymbol{y}. +$$ + +We will come back to this expression when we discuss Ridge regression. + +

+

+ +

+ + +
+ + + + + + + +
+ +
+ + + + + + diff --git a/doc/src/Regression/._Regression-bs039.html b/doc/src/Regression/._Regression-bs039.html new file mode 100644 index 000000000..b779e21a2 --- /dev/null +++ b/doc/src/Regression/._Regression-bs039.html @@ -0,0 +1,490 @@ + + + + + + + + +Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis + + + + + + + + + + + + + + + + + + + + + + + +
+
+ + Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis +
+ +
+ +
+
+ + +
+ +

 

 

 

+ + + + +

Ridge and LASSO Regression

+ +

+Let us remind ourselves about the expression for the standard Mean Squared Error (MSE) which we used to define our cost function and the equations for the ordinary least squares (OLS) method, that is +our optimization problem is +$$ +{\displaystyle \min_{\boldsymbol{\beta}\in {\mathbb{R}}^{p}}}\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)^T\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)\right\}. +$$ + +or we can state it as +$$ +{\displaystyle \min_{\boldsymbol{\beta}\in +{\mathbb{R}}^{p}}}\frac{1}{n}\sum_{i=0}^{n-1}\left(y_i-\tilde{y}_i\right)^2=\frac{1}{n}\vert\vert \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\vert\vert_2^2, +$$ + +where we have used the definition of a norm-2 vector, that is +$$ +\vert\vert \boldsymbol{x}\vert\vert_2 = \sqrt{\sum_i x_i^2}. +$$ + +

+By minimizing the above equation with respect to the parameters +\( \boldsymbol{\beta} \) we could then obtain an analytical expression for the +parameters \( \boldsymbol{\beta} \). We can add a regularization parameter \( \lambda \) by +defining a new cost function to be optimized, that is + +$$ +{\displaystyle \min_{\boldsymbol{\beta}\in +{\mathbb{R}}^{p}}}\frac{1}{n}\vert\vert \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\vert\vert_2^2+\lambda\vert\vert \boldsymbol{\beta}\vert\vert_2^2 +$$ + +

+which leads to the Ridge regression minimization problem where we +require that \( \vert\vert \boldsymbol{\beta}\vert\vert_2^2\le t \), where \( t \) is +a finite number larger than zero. By defining + +$$ +C(\boldsymbol{X},\boldsymbol{\beta})=\frac{1}{n}\vert\vert \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\vert\vert_2^2+\lambda\vert\vert \boldsymbol{\beta}\vert\vert_1, +$$ + +

+we have a new optimization equation +$$ +{\displaystyle \min_{\boldsymbol{\beta}\in +{\mathbb{R}}^{p}}}\frac{1}{n}\vert\vert \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\vert\vert_2^2+\lambda\vert\vert \boldsymbol{\beta}\vert\vert_1 +$$ + +which leads to Lasso regression. Lasso stands for least absolute shrinkage and selection operator. + +

+Here we have defined the norm-1 as +$$ +\vert\vert \boldsymbol{x}\vert\vert_1 = \sum_i \vert x_i\vert. +$$ + +

+

+ +

+ + +
+ + + + + + + +
+ +
+ + + + + + diff --git a/doc/src/Regression/._Regression-bs040.html b/doc/src/Regression/._Regression-bs040.html new file mode 100644 index 000000000..79442e146 --- /dev/null +++ b/doc/src/Regression/._Regression-bs040.html @@ -0,0 +1,491 @@ + + + + + + + + +Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis + + + + + + + + + + + + + + + + + + + + + + + +
+
+ + Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis +
+ +
+ +
+
+ + +
+ +

 

 

 

+ + + + +

More on Ridge Regression

+ +

+Using the matrix-vector expression for Ridge regression, + +$$ +C(\boldsymbol{X},\boldsymbol{\beta})=\frac{1}{n}\left\{(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})^T(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})\right\}+\lambda\boldsymbol{\beta}^T\boldsymbol{\beta}, +$$ + +

+by taking the derivatives with respect to \( \boldsymbol{\beta} \) we obtain then +a slightly modified matrix inversion problem which for finite values +of \( \lambda \) does not suffer from singularity problems. We obtain + +$$ +\boldsymbol{\beta}^{\mathrm{Ridge}} = \left(\boldsymbol{X}^T\boldsymbol{X}+\lambda\boldsymbol{I}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}, +$$ + +

+with \( \boldsymbol{I} \) being a \( p\times p \) identity matrix with the constraint that + +$$ +\sum_{i=0}^{p-1} \beta_i^2 \leq t, +$$ + +

+with \( t \) a finite positive number. + +

+We see that Ridge regression is nothing but the standard +OLS with a modified diagonal term added to \( \boldsymbol{X}^T\boldsymbol{X} \). The +consequences, in particular for our discussion of the bias-variance +are rather interesting. + +

+Furthermore, if we use the result above in terms of the SVD decomposition (our analysis was done for the OLS method), we had +$$ +(\boldsymbol{X}\boldsymbol{X}^T)\boldsymbol{U} = \boldsymbol{U}\boldsymbol{D}. +$$ + +

+We can analyse the OLS solutions in terms of the eigenvectors (the columns) of the right singular value matrix \( \boldsymbol{U} \) as +$$ +\boldsymbol{X}\boldsymbol{\beta} = \boldsymbol{X}\left(\boldsymbol{V}\boldsymbol{D}\boldsymbol{V}^T \right)^{-1}\boldsymbol{X}^T\boldsymbol{y}=\boldsymbol{U\Sigma V^T}\left(\boldsymbol{V}\boldsymbol{D}\boldsymbol{V}^T \right)^{-1}(\boldsymbol{U\Sigma V^T})^T\boldsymbol{y}=\boldsymbol{U}\boldsymbol{U}^T\boldsymbol{y} +$$ + +

+For Ridge regression this becomes + +$$ +\boldsymbol{X}\boldsymbol{\beta}^{\mathrm{Ridge}} = \boldsymbol{U\Sigma V^T}\left(\boldsymbol{V}\boldsymbol{D}\boldsymbol{V}^T+\lambda\boldsymbol{I} \right)^{-1}(\boldsymbol{U\Sigma V^T})^T\boldsymbol{y}=\sum_{j=0}^{p-1}\boldsymbol{u}_j\boldsymbol{u}_j^T\frac{\sigma_j^2}{\sigma_j^2+\lambda}\boldsymbol{y}, +$$ + +

+with the vectors \( \boldsymbol{u}_j \) being the columns of \( \boldsymbol{U} \). + +

+

+ +

+ + +
+ + + + + + + +
+ +
+ + + + + + diff --git a/doc/src/Regression/._Regression-bs041.html b/doc/src/Regression/._Regression-bs041.html new file mode 100644 index 000000000..4f7064b9e --- /dev/null +++ b/doc/src/Regression/._Regression-bs041.html @@ -0,0 +1,456 @@ + + + + + + + + +Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis + + + + + + + + + + + + + + + + + + + + + + + +
+
+ + Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis +
+ +
+ +
+
+ + +
+ +

 

 

 

+ + + + +

Interpreting the Ridge results

+ +

+Since \( \lambda \geq 0 \), it means that compared to OLS, we have + +$$ +\frac{\sigma_j^2}{\sigma_j^2+\lambda} \leq 1. +$$ + +

+Ridge regression finds the coordinates of \( \boldsymbol{y} \) with respect to the +orthonormal basis \( \boldsymbol{U} \), it then shrinks the coordinates by +\( \frac{\sigma_j^2}{\sigma_j^2+\lambda} \). Recall that the SVD has +eigenvalues ordered in a descending way, that is \( \sigma_i \geq +\sigma_{i+1} \). + +

+For small eigenvalues \( \sigma_i \) it means that their contributions become less important, a fact which can be used to reduce the number of degrees of freedom. +Actually, calculating the variance of \( \boldsymbol{X}\boldsymbol{v}_j \) shows that this quantity is equal to \( \sigma_j^2/n \). +With a parameter \( \lambda \) we can thus shrink the role of specific parameters. + +

+

+ +

+ + +
+ + + + + + + +
+ +
+ + + + + + diff --git a/doc/src/Regression/._Regression-bs042.html b/doc/src/Regression/._Regression-bs042.html new file mode 100644 index 000000000..0c995529e --- /dev/null +++ b/doc/src/Regression/._Regression-bs042.html @@ -0,0 +1,469 @@ + + + + + + + + +Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis + + + + + + + + + + + + + + + + + + + + + + + +
+
+ + Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis +
+ +
+ +
+
+ + +
+ +

 

 

 

+ + + + +

More interpretations

+ +

+For the sake of simplicity, let us assume that the design matrix is orthonormal, that is + +$$ +\boldsymbol{X}^T\boldsymbol{X}=(\boldsymbol{X}^T\boldsymbol{X})^{-1} =\boldsymbol{I}. +$$ + +

+In this case the standard OLS results in +$$ +\boldsymbol{\beta}^{\mathrm{OLS}} = \boldsymbol{X}^T\boldsymbol{y}=\sum_{i=0}^{p-1}\boldsymbol{u}_j\boldsymbol{u}_j^T\boldsymbol{y}, +$$ + +

+and + +$$ +\boldsymbol{\beta}^{\mathrm{Ridge}} = \left(\boldsymbol{I}+\lambda\boldsymbol{I}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}=\left(1+\lambda\right)^{-1}\boldsymbol{\beta}^{\mathrm{OLS}}, +$$ + +

+that is the Ridge estimator scales the OLS estimator by the inverse of a factor \( 1+\lambda \), and +the Ridge estimator converges to zero when the hyperparameter goes to +infinity. + +

+We will come back to more interpreations after we have gone through some of the statistical analysis part. + +

+For more discussions of Ridge and Lasso regression, Wessel van Wieringen's article is highly recommended. +Similarly, Mehta et al's article is also recommended. + +

+

+ +

+ + +
+ + + + + + + +
+ +
+ + + + + + diff --git a/doc/src/Regression/._Regression-bs043.html b/doc/src/Regression/._Regression-bs043.html new file mode 100644 index 000000000..c44c78991 --- /dev/null +++ b/doc/src/Regression/._Regression-bs043.html @@ -0,0 +1,449 @@ + + + + + + + + +Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis + + + + + + + + + + + + + + + + + + + + + + + +
+
+ + Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis +
+ +
+ +
+
+ + +
+ +

 

 

 

+ + + + +

Where are we going?

+ +

+Before we proceed, we need to rethink what we have been doing. In our +eager to fit the data, we have omitted several important elements in +our regression analysis. In what follows we will + +

    +
  1. look at statistical properties, including a discussion of mean values, variance and the so-called bias-variance tradeoff
    2. +
  2. introduce resampling techniques like cross-validation, bootstrapping and jackknife and more
    3. +
+ +This will allow us to link the standard linear algebra methods we have discussed above to a statistical interpretation of the methods. + +

+

+ +

+ + +
+ + + + + + + +
+ +
+ + + + + + diff --git a/doc/src/Regression/._Regression-bs044.html b/doc/src/Regression/._Regression-bs044.html new file mode 100644 index 000000000..fc2bf6ad7 --- /dev/null +++ b/doc/src/Regression/._Regression-bs044.html @@ -0,0 +1,453 @@ + + + + + + + + +Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis + + + + + + + + + + + + + + + + + + + + + + + +
+
+ + Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis +
+ +
+ +
+
+ + +
+ +

 

 

 

+ + + + +

Resampling methods

+
+
+

+Resampling methods are an indispensable tool in modern +statistics. They involve repeatedly drawing samples from a training +set and refitting a model of interest on each sample in order to +obtain additional information about the fitted model. For example, in +order to estimate the variability of a linear regression fit, we can +repeatedly draw different samples from the training data, fit a linear +regression to each new sample, and then examine the extent to which +the resulting fits differ. Such an approach may allow us to obtain +information that would not be available from fitting the model only +once using the original training sample. +

+
+ + +

+

+ +

+ + +
+ + + + + + + +
+ +
+ + + + + + diff --git a/doc/src/Regression/._Regression-bs045.html b/doc/src/Regression/._Regression-bs045.html new file mode 100644 index 000000000..082121081 --- /dev/null +++ b/doc/src/Regression/._Regression-bs045.html @@ -0,0 +1,462 @@ + + + + + + + + +Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis + + + + + + + + + + + + + + + + + + + + + + + +
+
+ + Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis +
+ +
+ +
+
+ + +
+ +

 

 

 

+ + + + +

Resampling approaches can be computationally expensive

+
+
+

+ +

+Resampling approaches can be computationally expensive, because they +involve fitting the same statistical method multiple times using +different subsets of the training data. However, due to recent +advances in computing power, the computational requirements of +resampling methods generally are not prohibitive. In this chapter, we +discuss two of the most commonly used resampling methods, +cross-validation and the bootstrap. Both methods are important tools +in the practical application of many statistical learning +procedures. For example, cross-validation can be used to estimate the +test error associated with a given statistical learning method in +order to evaluate its performance, or to select the appropriate level +of flexibility. The process of evaluating a model’s performance is +known as model assessment, whereas the process of selecting the proper +level of flexibility for a model is known as model selection. The +bootstrap is widely used. + +

+

+
+ + +

+

+ +

+ + +
+ + + + + + + +
+ +
+ + + + + + diff --git a/doc/src/Regression/._Regression-bs046.html b/doc/src/Regression/._Regression-bs046.html new file mode 100644 index 000000000..77b3f2369 --- /dev/null +++ b/doc/src/Regression/._Regression-bs046.html @@ -0,0 +1,449 @@ + + + + + + + + +Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis + + + + + + + + + + + + + + + + + + + + + + + +
+
+ + Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis +
+ +
+ +
+
+ + +
+ +

 

 

 

+ + + + +

Why resampling methods ?

+
+
+

+ +

    +
  • Our simulations can be treated as computer experiments. This is particularly the case for Monte Carlo methods
    • +
  • The results can be analysed with the same statistical tools as we would use analysing experimental data.
    • +
  • As in all experiments, we are looking for expectation values and an estimate of how accurate they are, i.e., possible sources for errors.
    • +
+
+
+ + +

+

+ +

+ + +
+ + + + + + + +
+ +
+ + + + + + diff --git a/doc/src/Regression/._Regression-bs047.html b/doc/src/Regression/._Regression-bs047.html new file mode 100644 index 000000000..927b9be13 --- /dev/null +++ b/doc/src/Regression/._Regression-bs047.html @@ -0,0 +1,455 @@ + + + + + + + + +Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis + + + + + + + + + + + + + + + + + + + + + + + +
+
+ + Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis +
+ +
+ +
+
+ + +
+ +

 

 

 

+ + + + +

Statistical analysis

+
+
+

+ +

    +
  • As in other experiments, many numerical experiments have two classes of errors:
    • + +
      +
    • Statistical errors
      • +
    • Systematical errors
      • +
    + +
  • Statistical errors can be estimated using standard tools from statistics
    • +
  • Systematical errors are method specific and must be treated differently from case to case.
    • +
+
+
+ + +

+

+ +

+ + +
+ + + + + + + +
+ +
+ + + + + + diff --git a/doc/src/Regression/._Regression-bs048.html b/doc/src/Regression/._Regression-bs048.html new file mode 100644 index 000000000..6beabb135 --- /dev/null +++ b/doc/src/Regression/._Regression-bs048.html @@ -0,0 +1,464 @@ + + + + + + + + +Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis + + + + + + + + + + + + + + + + + + + + + + + +
+
+ + Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis +
+ +
+ +
+
+ + +
+ +

 

 

 

+ + + + +

Statistics

+
+
+

+The probability distribution function (PDF) is a function +\( p(x) \) on the domain which, in the discrete case, gives us the +probability or relative frequency with which these values of \( X \) occur: +$$ +p(x) = \mathrm{prob}(X=x) +$$ + +In the continuous case, the PDF does not directly depict the +actual probability. Instead we define the probability for the +stochastic variable to assume any value on an infinitesimal interval +around \( x \) to be \( p(x)dx \). The continuous function \( p(x) \) then gives us +the density of the probability rather than the probability +itself. The probability for a stochastic variable to assume any value +on a non-infinitesimal interval \( [a,\,b] \) is then just the integral: +$$ +\mathrm{prob}(a\leq X\leq b) = \int_a^b p(x)dx +$$ + +Qualitatively speaking, a stochastic variable represents the values of +numbers chosen as if by chance from some specified PDF so that the +selection of a large set of these numbers reproduces this PDF. +

+
+ + +

+

+ +

+ + +
+ + + + + + + +
+ +
+ + + + + + diff --git a/doc/src/Regression/._Regression-bs049.html b/doc/src/Regression/._Regression-bs049.html new file mode 100644 index 000000000..70d02b6a2 --- /dev/null +++ b/doc/src/Regression/._Regression-bs049.html @@ -0,0 +1,456 @@ + + + + + + + + +Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis + + + + + + + + + + + + + + + + + + + + + + + +
+
+ + Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis +
+ +
+ +
+
+ + +
+ +

 

 

 

+ + + + +

Statistics, moments

+
+
+

+A particularly useful class of special expectation values are the +moments. The \( n \)-th moment of the PDF \( p \) is defined as +follows: +$$ +\langle x^n\rangle \equiv \int\! x^n p(x)\,dx +$$ + +The zero-th moment \( \langle 1\rangle \) is just the normalization condition of +\( p \). The first moment, \( \langle x\rangle \), is called the mean of \( p \) +and often denoted by the letter \( \mu \): +$$ +\langle x\rangle = \mu \equiv \int\! x p(x)\,dx +$$ +

+
+ + +

+

+ +

+ + +
+ + + + + + + +
+ +
+ + + + + + diff --git a/doc/src/Regression/._Regression-bs050.html b/doc/src/Regression/._Regression-bs050.html new file mode 100644 index 000000000..fd305b0a5 --- /dev/null +++ b/doc/src/Regression/._Regression-bs050.html @@ -0,0 +1,471 @@ + + + + + + + + +Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis + + + + + + + + + + + + + + + + + + + + + + + +
+
+ + Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis +
+ +
+ +
+
+ + +
+ +

 

 

 

+ + + + +

Statistics, central moments

+
+
+

+A special version of the moments is the set of central moments, +the n-th central moment defined as: +$$ +\langle (x-\langle x \rangle )^n\rangle \equiv \int\! (x-\langle x\rangle)^n p(x)\,dx +$$ + +The zero-th and first central moments are both trivial, equal \( 1 \) and +\( 0 \), respectively. But the second central moment, known as the +variance of \( p \), is of particular interest. For the stochastic +variable \( X \), the variance is denoted as \( \sigma^2_X \) or \( \mathrm{var}(X) \): +$$ +\begin{align} +\sigma^2_X\ \ =\ \ \mathrm{var}(X) & = \langle (x-\langle x\rangle)^2\rangle = +\int\! (x-\langle x\rangle)^2 p(x)\,dx +\tag{8}\\ +& = \int\! \left(x^2 - 2 x \langle x\rangle^{2} + + \langle x\rangle^2\right)p(x)\,dx +\tag{9}\\ +& = \langle x^2\rangle - 2 \langle x\rangle\langle x\rangle + \langle x\rangle^2 +\tag{10}\\ +& = \langle x^2\rangle - \langle x\rangle^2 +\tag{11} +\end{align} +$$ + +The square root of the variance, \( \sigma =\sqrt{\langle (x-\langle x\rangle)^2\rangle} \) is called the standard deviation of \( p \). It is clearly just the RMS (root-mean-square) +value of the deviation of the PDF from its mean value, interpreted +qualitatively as the spread of \( p \) around its mean. +

+
+ + +

+

+ +

+ + +
+ + + + + + + +
+ +
+ + + + + + diff --git a/doc/src/Regression/._Regression-bs051.html b/doc/src/Regression/._Regression-bs051.html new file mode 100644 index 000000000..1f7e91c6d --- /dev/null +++ b/doc/src/Regression/._Regression-bs051.html @@ -0,0 +1,464 @@ + + + + + + + + +Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis + + + + + + + + + + + + + + + + + + + + + + + +
+
+ + Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis +
+ +
+ +
+
+ + +
+ +

 

 

 

+ + + + +

Statistics, covariance

+
+
+

+Another important quantity is the so called covariance, a variant of +the above defined variance. Consider again the set \( \{X_i\} \) of \( n \) +stochastic variables (not necessarily uncorrelated) with the +multivariate PDF \( P(x_1,\dots,x_n) \). The covariance of two +of the stochastic variables, \( X_i \) and \( X_j \), is defined as follows: +$$ +\begin{align} +\mathrm{cov}(X_i,\,X_j) &\equiv \langle (x_i-\langle x_i\rangle)(x_j-\langle x_j\rangle)\rangle +\nonumber\\ +&= +\int\!\cdots\!\int\!(x_i-\langle x_i \rangle)(x_j-\langle x_j \rangle)\, +P(x_1,\dots,x_n)\,dx_1\dots dx_n +\tag{12} +\end{align} +$$ + +with +$$ +\langle x_i\rangle = +\int\!\cdots\!\int\!x_i\,P(x_1,\dots,x_n)\,dx_1\dots dx_n +$$ +

+
+ + +

+

+ +

+ + +
+ + + + + + + +
+ +
+ + + + + + diff --git a/doc/src/Regression/._Regression-bs052.html b/doc/src/Regression/._Regression-bs052.html new file mode 100644 index 000000000..bb072b2f6 --- /dev/null +++ b/doc/src/Regression/._Regression-bs052.html @@ -0,0 +1,465 @@ + + + + + + + + +Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis + + + + + + + + + + + + + + + + + + + + + + + +
+
+ + Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis +
+ +
+ +
+
+ + +
+ +

 

 

 

+ + + + +

Statistics, more covariance

+
+
+

+If we consider the above covariance as a matrix \( C_{ij}=\mathrm{cov}(X_i,\,X_j) \), then the diagonal elements are just the familiar +variances, \( C_{ii} = \mathrm{cov}(X_i,\,X_i) = \mathrm{var}(X_i) \). It turns out that +all the off-diagonal elements are zero if the stochastic variables are +uncorrelated. This is easy to show, keeping in mind the linearity of +the expectation value. Consider the stochastic variables \( X_i \) and +\( X_j \), (\( i\neq j \)): +$$ +\begin{align} +\mathrm{cov}(X_i,\,X_j) &= \langle(x_i-\langle x_i\rangle)(x_j-\langle x_j\rangle)\rangle +\tag{13}\\ +&=\langle x_i x_j - x_i\langle x_j\rangle - \langle x_i\rangle x_j + \langle x_i\rangle\langle x_j\rangle\rangle +\tag{14}\\ +&=\langle x_i x_j\rangle - \langle x_i\langle x_j\rangle\rangle - \langle \langle x_i\rangle x_j\rangle + +\langle \langle x_i\rangle\langle x_j\rangle\rangle +\tag{15}\\ +&=\langle x_i x_j\rangle - \langle x_i\rangle\langle x_j\rangle - \langle x_i\rangle\langle x_j\rangle + +\langle x_i\rangle\langle x_j\rangle +\tag{16}\\ +&=\langle x_i x_j\rangle - \langle x_i\rangle\langle x_j\rangle +\tag{17} +\end{align} +$$ +

+
+ + +

+

+ +

+ + +
+ + + + + + + +
+ +
+ + + + + + diff --git a/doc/src/Regression/._Regression-bs053.html b/doc/src/Regression/._Regression-bs053.html new file mode 100644 index 000000000..bb2d31a84 --- /dev/null +++ b/doc/src/Regression/._Regression-bs053.html @@ -0,0 +1,459 @@ + + + + + + + + +Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis + + + + + + + + + + + + + + + + + + + + + + + +
+
+ + Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis +
+ +
+ +
+
+ + +
+ +

 

 

 

+ + + + +

Statistics, independent variables

+
+
+

+If \( X_i \) and \( X_j \) are independent, we get +\( \langle x_i x_j\rangle =\langle x_i\rangle\langle x_j\rangle \), resulting in \( \mathrm{cov}(X_i, X_j) = 0\ \ (i\neq j) \). + +

+Also useful for us is the covariance of linear combinations of +stochastic variables. Let \( \{X_i\} \) and \( \{Y_i\} \) be two sets of +stochastic variables. Let also \( \{a_i\} \) and \( \{b_i\} \) be two sets of +scalars. Consider the linear combination: +$$ +U = \sum_i a_i X_i \qquad V = \sum_j b_j Y_j +$$ + +By the linearity of the expectation value +$$ +\mathrm{cov}(U, V) = \sum_{i,j}a_i b_j \mathrm{cov}(X_i, Y_j) +$$ +

+
+ + +

+

+ +

+ + +
+ + + + + + + +
+ +
+ + + + + + diff --git a/doc/src/Regression/._Regression-bs054.html b/doc/src/Regression/._Regression-bs054.html new file mode 100644 index 000000000..941746562 --- /dev/null +++ b/doc/src/Regression/._Regression-bs054.html @@ -0,0 +1,465 @@ + + + + + + + + +Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis + + + + + + + + + + + + + + + + + + + + + + + +
+
+ + Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis +
+ +
+ +
+
+ + +
+ +

 

 

 

+ + + + +

Statistics, more variance

+
+
+

+Now, since the variance is just \( \mathrm{var}(X_i) = \mathrm{cov}(X_i, X_i) \), we get +the variance of the linear combination \( U = \sum_i a_i X_i \): +$$ +\begin{equation} +\mathrm{var}(U) = \sum_{i,j}a_i a_j \mathrm{cov}(X_i, X_j) +\tag{18} +\end{equation} +$$ + +And in the special case when the stochastic variables are +uncorrelated, the off-diagonal elements of the covariance are as we +know zero, resulting in: +$$ +\mathrm{var}(U) = \sum_i a_i^2 \mathrm{cov}(X_i, X_i) = \sum_i a_i^2 \mathrm{var}(X_i) +$$ + +$$ +\mathrm{var}(\sum_i a_i X_i) = \sum_i a_i^2 \mathrm{var}(X_i) +$$ + +which will become very useful in our study of the error in the mean +value of a set of measurements. +

+
+ + +

+

+ +

+ + +
+ + + + + + + +
+ +
+ + + + + + diff --git a/doc/src/Regression/._Regression-bs055.html b/doc/src/Regression/._Regression-bs055.html new file mode 100644 index 000000000..beaf91621 --- /dev/null +++ b/doc/src/Regression/._Regression-bs055.html @@ -0,0 +1,461 @@ + + + + + + + + +Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis + + + + + + + + + + + + + + + + + + + + + + + +
+
+ + Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis +
+ +
+ +
+
+ + +
+ +

 

 

 

+ + + + +

Statistics and stochastic processes

+
+
+

+A stochastic process is a process that produces sequentially a +chain of values: +$$ +\{x_1, x_2,\dots\,x_k,\dots\}. +$$ + +We will call these +values our measurements and the entire set as our measured +sample. The action of measuring all the elements of a sample +we will call a stochastic experiment since, operationally, +they are often associated with results of empirical observation of +some physical or mathematical phenomena; precisely an experiment. We +assume that these values are distributed according to some +PDF \( p_X^{\phantom X}(x) \), where \( X \) is just the formal symbol for the +stochastic variable whose PDF is \( p_X^{\phantom X}(x) \). Instead of +trying to determine the full distribution \( p \) we are often only +interested in finding the few lowest moments, like the mean +\( \mu_X^{\phantom X} \) and the variance \( \sigma_X^{\phantom X} \). +

+
+ + +

+

+ +

+ + +
+ + + + + + + +
+ +
+ + + + + + diff --git a/doc/src/Regression/._Regression-bs056.html b/doc/src/Regression/._Regression-bs056.html new file mode 100644 index 000000000..572f7309b --- /dev/null +++ b/doc/src/Regression/._Regression-bs056.html @@ -0,0 +1,459 @@ + + + + + + + + +Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis + + + + + + + + + + + + + + + + + + + + + + + +
+
+ + Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis +
+ +
+ +
+
+ + +
+ +

 

 

 

+ + + + +

Statistics and sample variables

+
+
+

+In practical situations a sample is always of finite size. Let that +size be \( n \). The expectation value of a sample, the sample mean, is then defined as follows: +$$ +\bar{x}_n \equiv \frac{1}{n}\sum_{k=1}^n x_k +$$ + +The sample variance is: +$$ +\mathrm{var}(x) \equiv \frac{1}{n}\sum_{k=1}^n (x_k - \bar{x}_n)^2 +$$ + +its square root being the standard deviation of the sample. The +sample covariance is: +$$ +\mathrm{cov}(x)\equiv\frac{1}{n}\sum_{kl}(x_k - \bar{x}_n)(x_l - \bar{x}_n) +$$ +

+
+ + +

+

+ +

+ + +
+ + + + + + + +
+ +
+ + + + + + diff --git a/doc/src/Regression/._Regression-bs057.html b/doc/src/Regression/._Regression-bs057.html new file mode 100644 index 000000000..48e567830 --- /dev/null +++ b/doc/src/Regression/._Regression-bs057.html @@ -0,0 +1,454 @@ + + + + + + + + +Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis + + + + + + + + + + + + + + + + + + + + + + + +
+
+ + Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis +
+ +
+ +
+
+ + +
+ +

 

 

 

+ + + + +

Statistics, sample variance and covariance

+
+
+

+Note that the sample variance is the sample covariance without the +cross terms. In a similar manner as the covariance in Eq. (12) is a measure of the correlation between +two stochastic variables, the above defined sample covariance is a +measure of the sequential correlation between succeeding measurements +of a sample. + +

+These quantities, being known experimental values, differ +significantly from and must not be confused with the similarly named +quantities for stochastic variables, mean \( \mu_X \), variance \( \mathrm{var}(X) \) +and covariance \( \mathrm{cov}(X,Y) \). +

+
+ + +

+

+ +

+ + +
+ + + + + + + +
+ +
+ + + + + + diff --git a/doc/src/Regression/._Regression-bs058.html b/doc/src/Regression/._Regression-bs058.html new file mode 100644 index 000000000..a0dc42c2a --- /dev/null +++ b/doc/src/Regression/._Regression-bs058.html @@ -0,0 +1,465 @@ + + + + + + + + +Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis + + + + + + + + + + + + + + + + + + + + + + + +
+
+ + Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis +
+ +
+ +
+
+ + +
+ +

 

 

 

+ + + + +

Statistics, law of large numbers

+
+
+

+The law of large numbers +states that as the size of our sample grows to infinity, the sample +mean approaches the true mean \( \mu_X^{\phantom X} \) of the chosen PDF: +$$ +\lim_{n\to\infty}\bar{x}_n = \mu_X^{\phantom X} +$$ + +The sample mean \( \bar{x}_n \) works therefore as an estimate of the true +mean \( \mu_X^{\phantom X} \). + +

+What we need to find out is how good an approximation \( \bar{x}_n \) is to +\( \mu_X^{\phantom X} \). In any stochastic measurement, an estimated +mean is of no use to us without a measure of its error. A quantity +that tells us how well we can reproduce it in another experiment. We +are therefore interested in the PDF of the sample mean itself. Its +standard deviation will be a measure of the spread of sample means, +and we will simply call it the error of the sample mean, or +just sample error, and denote it by \( \mathrm{err}_X^{\phantom X} \). In +practice, we will only be able to produce an estimate of the +sample error since the exact value would require the knowledge of the +true PDFs behind, which we usually do not have. +

+
+ + +

+

+ +

+ + +
+ + + + + + + +
+ +
+ + + + + + diff --git a/doc/src/Regression/._Regression-bs059.html b/doc/src/Regression/._Regression-bs059.html new file mode 100644 index 000000000..65d131b1d --- /dev/null +++ b/doc/src/Regression/._Regression-bs059.html @@ -0,0 +1,455 @@ + + + + + + + + +Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis + + + + + + + + + + + + + + + + + + + + + + + +
+
+ + Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis +
+ +
+ +
+
+ + +
+ +

 

 

 

+ + + + +

Statistics, more on sample error

+
+
+

+Let us first take a look at what happens to the sample error as the +size of the sample grows. In a sample, each of the measurements \( x_i \) +can be associated with its own stochastic variable \( X_i \). The +stochastic variable \( \overline X_n \) for the sample mean \( \bar{x}_n \) is +then just a linear combination, already familiar to us: +$$ +\overline X_n = \frac{1}{n}\sum_{i=1}^n X_i +$$ + +All the coefficients are just equal \( 1/n \). The PDF of \( \overline X_n \), +denoted by \( p_{\overline X_n}(x) \) is the desired PDF of the sample +means. +

+
+ + +

+

+ +

+ + +
+ + + + + + + +
+ +
+ + + + + + diff --git a/doc/src/Regression/._Regression-bs060.html b/doc/src/Regression/._Regression-bs060.html new file mode 100644 index 000000000..af1206cdd --- /dev/null +++ b/doc/src/Regression/._Regression-bs060.html @@ -0,0 +1,454 @@ + + + + + + + + +Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis + + + + + + + + + + + + + + + + + + + + + + + +
+
+ + Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis +
+ +
+ +
+
+ + +
+ +

 

 

 

+ + + + +

Statistics

+
+
+

+The probability density of obtaining a sample mean \( \bar x_n \) +is the product of probabilities of obtaining arbitrary values \( x_1, +x_2,\dots,x_n \) with the constraint that the mean of the set \( \{x_i\} \) +is \( \bar x_n \): +$$ +p_{\overline X_n}(x) = \int p_X^{\phantom X}(x_1)\cdots +\int p_X^{\phantom X}(x_n)\ +\delta\!\left(x - \frac{x_1+x_2+\dots+x_n}{n}\right)dx_n \cdots dx_1 +$$ + +And in particular we are interested in its variance \( \mathrm{var}(\overline X_n) \). +

+
+ + +

+

+ +

+ + +
+ + + + + + + +
+ +
+ + + + + + diff --git a/doc/src/Regression/._Regression-bs061.html b/doc/src/Regression/._Regression-bs061.html new file mode 100644 index 000000000..201c00019 --- /dev/null +++ b/doc/src/Regression/._Regression-bs061.html @@ -0,0 +1,458 @@ + + + + + + + + +Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis + + + + + + + + + + + + + + + + + + + + + + + +
+
+ + Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis +
+ +
+ +
+
+ + +
+ +

 

 

 

+ + + + +

Statistics, central limit theorem

+
+
+

+It is generally not possible to express \( p_{\overline X_n}(x) \) in a +closed form given an arbitrary PDF \( p_X^{\phantom X} \) and a number +\( n \). But for the limit \( n\to\infty \) it is possible to make an +approximation. The very important result is called the central limit theorem. It tells us that as \( n \) goes to infinity, +\( p_{\overline X_n}(x) \) approaches a Gaussian distribution whose mean +and variance equal the true mean and variance, \( \mu_{X}^{\phantom X} \) +and \( \sigma_{X}^{2} \), respectively: +$$ +\begin{equation} +\lim_{n\to\infty} p_{\overline X_n}(x) = +\left(\frac{n}{2\pi\mathrm{var}(X)}\right)^{1/2} +e^{-\frac{n(x-\bar x_n)^2}{2\mathrm{var}(X)}} +\tag{19} +\end{equation} +$$ +

+
+ + +

+

+ +

+ + +
+ + + + + + + +
+ +
+ + + + + + diff --git a/doc/src/Regression/._Regression-bs062.html b/doc/src/Regression/._Regression-bs062.html new file mode 100644 index 000000000..d7f70e544 --- /dev/null +++ b/doc/src/Regression/._Regression-bs062.html @@ -0,0 +1,462 @@ + + + + + + + + +Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis + + + + + + + + + + + + + + + + + + + + + + + +
+
+ + Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis +
+ +
+ +
+
+ + +
+ +

 

 

 

+ + + + +

Statistics, more technicalities

+
+
+

+The desired variance +\( \mathrm{var}(\overline X_n) \), i.e. the sample error squared +\( \mathrm{err}_X^2 \), is given by: +$$ +\begin{equation} +\mathrm{err}_X^2 = \mathrm{var}(\overline X_n) = \frac{1}{n^2} +\sum_{ij} \mathrm{cov}(X_i, X_j) +\tag{20} +\end{equation} +$$ + +We see now that in order to calculate the exact error of the sample +with the above expression, we would need the true means +\( \mu_{X_i}^{\phantom X} \) of the stochastic variables \( X_i \). To +calculate these requires that we know the true multivariate PDF of all +the \( X_i \). But this PDF is unknown to us, we have only got the measurements of +one sample. The best we can do is to let the sample itself be an +estimate of the PDF of each of the \( X_i \), estimating all properties of +\( X_i \) through the measurements of the sample. +

+
+ + +

+

+ +

+ + +
+ + + + + + + +
+ +
+ + + + + + diff --git a/doc/src/Regression/._Regression-bs063.html b/doc/src/Regression/._Regression-bs063.html new file mode 100644 index 000000000..853fd1a25 --- /dev/null +++ b/doc/src/Regression/._Regression-bs063.html @@ -0,0 +1,460 @@ + + + + + + + + +Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis + + + + + + + + + + + + + + + + + + + + + + + +
+
+ + Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis +
+ +
+ +
+
+ + +
+ +

 

 

 

+ + + + +

Statistics

+
+
+

+Our estimate of \( \mu_{X_i}^{\phantom X} \) is then the sample mean \( \bar x \) +itself, in accordance with the the central limit theorem: +$$ +\mu_{X_i}^{\phantom X} = \langle x_i\rangle \approx \frac{1}{n}\sum_{k=1}^n x_k = \bar x +$$ + +Using \( \bar x \) in place of \( \mu_{X_i}^{\phantom X} \) we can give an +estimate of the covariance in Eq. (20) +$$ +\mathrm{cov}(X_i, X_j) = \langle (x_i-\langle x_i\rangle)(x_j-\langle x_j\rangle)\rangle +\approx\langle (x_i - \bar x)(x_j - \bar{x})\rangle, +$$ + +resulting in +$$ +\frac{1}{n} \sum_{l}^n \left(\frac{1}{n}\sum_{k}^n (x_k -\bar x_n)(x_l - \bar x_n)\right)=\frac{1}{n}\frac{1}{n} \sum_{kl} (x_k -\bar x_n)(x_l - \bar x_n)=\frac{1}{n}\mathrm{cov}(x) +$$ +

+
+ + +

+

+ +

+ + +
+ + + + + + + +
+ +
+ + + + + + diff --git a/doc/src/Regression/._Regression-bs064.html b/doc/src/Regression/._Regression-bs064.html new file mode 100644 index 000000000..841a90caa --- /dev/null +++ b/doc/src/Regression/._Regression-bs064.html @@ -0,0 +1,472 @@ + + + + + + + + +Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis + + + + + + + + + + + + + + + + + + + + + + + +
+
+ + Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis +
+ +
+ +
+
+ + +
+ +

 

 

 

+ + + + +

Statistics and sample variance

+
+
+

+By the same procedure we can use the sample variance as an +estimate of the variance of any of the stochastic variables \( X_i \) +$$ +\mathrm{var}(X_i)=\langle x_i - \langle x_i\rangle\rangle \approx \langle x_i - \bar x_n\rangle\nonumber, +$$ + +which is approximated as +$$ +\begin{equation} +\mathrm{var}(X_i)\approx \frac{1}{n}\sum_{k=1}^n (x_k - \bar x_n)=\mathrm{var}(x) +\tag{21} +\end{equation} +$$ + +

+Now we can calculate an estimate of the error +\( \mathrm{err}_X^{\phantom X} \) of the sample mean \( \bar x_n \): +$$ +\begin{align} +\mathrm{err}_X^2 +&=\frac{1}{n^2}\sum_{ij} \mathrm{cov}(X_i, X_j) \nonumber \\ +&\approx&\frac{1}{n^2}\sum_{ij}\frac{1}{n}\mathrm{cov}(x) =\frac{1}{n^2}n^2\frac{1}{n}\mathrm{cov}(x)\nonumber\\ +&=\frac{1}{n}\mathrm{cov}(x) +\tag{22} +\end{align} +$$ + +which is nothing but the sample covariance divided by the number of +measurements in the sample. +

+
+ + +

+

+ +

+ + +
+ + + + + + + +
+ +
+ + + + + + diff --git a/doc/src/Regression/._Regression-bs065.html b/doc/src/Regression/._Regression-bs065.html new file mode 100644 index 000000000..08333bcdf --- /dev/null +++ b/doc/src/Regression/._Regression-bs065.html @@ -0,0 +1,468 @@ + + + + + + + + +Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis + + + + + + + + + + + + + + + + + + + + + + + +
+
+ + Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis +
+ +
+ +
+
+ + +
+ +

 

 

 

+ + + + +

Statistics, uncorrelated results

+
+
+

+ +

+In the special case that the measurements of the sample are +uncorrelated (equivalently the stochastic variables \( X_i \) are +uncorrelated) we have that the off-diagonal elements of the covariance +are zero. This gives the following estimate of the sample error: +$$ +\mathrm{err}_X^2=\frac{1}{n^2}\sum_{ij} \mathrm{cov}(X_i, X_j) = +\frac{1}{n^2} \sum_i \mathrm{var}(X_i), +$$ + +resulting in +$$ +\begin{equation} +\mathrm{err}_X^2\approx \frac{1}{n^2} \sum_i \mathrm{var}(x)= \frac{1}{n}\mathrm{var}(x) +\tag{23} +\end{equation} +$$ + +where in the second step we have used Eq. (21). +The error of the sample is then just its standard deviation divided by +the square root of the number of measurements the sample contains. +This is a very useful formula which is easy to compute. It acts as a +first approximation to the error, but in numerical experiments, we +cannot overlook the always present correlations. +

+
+ + +

+

+ +

+ + +
+ + + + + + + +
+ +
+ + + + + + diff --git a/doc/src/Regression/._Regression-bs066.html b/doc/src/Regression/._Regression-bs066.html new file mode 100644 index 000000000..6c7f7f24e --- /dev/null +++ b/doc/src/Regression/._Regression-bs066.html @@ -0,0 +1,462 @@ + + + + + + + + +Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis + + + + + + + + + + + + + + + + + + + + + + + +
+
+ + Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis +
+ +
+ +
+
+ + +
+ +

 

 

 

+ + + + +

Statistics, computations

+
+
+

+For computational purposes one usually splits up the estimate of +\( \mathrm{err}_X^2 \), given by Eq. (22), into two +parts +$$ +\mathrm{err}_X^2 = \frac{1}{n}\mathrm{var}(x) + \frac{1}{n}(\mathrm{cov}(x)-\mathrm{var}(x)), +$$ + +which equals +$$ +\begin{equation} +\frac{1}{n^2}\sum_{k=1}^n (x_k - \bar x_n)^2 +\frac{2}{n^2}\sum_{k < l} (x_k - \bar x_n)(x_l - \bar x_n) +\tag{24} +\end{equation} +$$ + +The first term is the same as the error in the uncorrelated case, +Eq. (23). This means that the second +term accounts for the error correction due to correlation between the +measurements. For uncorrelated measurements this second term is zero. +

+
+ + +

+

+ +

+ + +
+ + + + + + + +
+ +
+ + + + + + diff --git a/doc/src/Regression/._Regression-bs067.html b/doc/src/Regression/._Regression-bs067.html new file mode 100644 index 000000000..1f4897191 --- /dev/null +++ b/doc/src/Regression/._Regression-bs067.html @@ -0,0 +1,455 @@ + + + + + + + + +Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis + + + + + + + + + + + + + + + + + + + + + + + +
+
+ + Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis +
+ +
+ +
+
+ + +
+ +

 

 

 

+ + + + +

Statistics, more on computations of errors

+
+
+

+Computationally the uncorrelated first term is much easier to treat +efficiently than the second. +$$ +\mathrm{var}(x) = \frac{1}{n}\sum_{k=1}^n (x_k - \bar x_n)^2 = +\left(\frac{1}{n}\sum_{k=1}^n x_k^2\right) - \bar x_n^2 +$$ + +We just accumulate separately the values \( x^2 \) and \( x \) for every +measurement \( x \) we receive. The correlation term, though, has to be +calculated at the end of the experiment since we need all the +measurements to calculate the cross terms. Therefore, all measurements +have to be stored throughout the experiment. +

+
+ + +

+

+ +

+ + +
+ + + + + + + +
+ +
+ + + + + + diff --git a/doc/src/Regression/._Regression-bs068.html b/doc/src/Regression/._Regression-bs068.html new file mode 100644 index 000000000..fed0a66ef --- /dev/null +++ b/doc/src/Regression/._Regression-bs068.html @@ -0,0 +1,466 @@ + + + + + + + + +Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis + + + + + + + + + + + + + + + + + + + + + + + +
+
+ + Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis +
+ +
+ +
+
+ + +
+ +

 

 

 

+ + + + +

Statistics, wrapping up 1

+
+
+

+Let us analyze the problem by splitting up the correlation term into +partial sums of the form: +$$ +f_d = \frac{1}{n-d}\sum_{k=1}^{n-d}(x_k - \bar x_n)(x_{k+d} - \bar x_n) +$$ + +The correlation term of the error can now be rewritten in terms of +\( f_d \) +$$ +\frac{2}{n}\sum_{k < l} (x_k - \bar x_n)(x_l - \bar x_n) = +2\sum_{d=1}^{n-1} f_d +$$ + +The value of \( f_d \) reflects the correlation between measurements +separated by the distance \( d \) in the sample samples. Notice that for +\( d=0 \), \( f \) is just the sample variance, \( \mathrm{var}(x) \). If we divide \( f_d \) +by \( \mathrm{var}(x) \), we arrive at the so called autocorrelation function +$$ +\kappa_d = \frac{f_d}{\mathrm{var}(x)} +$$ + +which gives us a useful measure of pairwise correlations +starting always at \( 1 \) for \( d=0 \). +

+
+ + +

+

+ +

+ + +
+ + + + + + + +
+ +
+ + + + + + diff --git a/doc/src/Regression/._Regression-bs069.html b/doc/src/Regression/._Regression-bs069.html new file mode 100644 index 000000000..67c484a91 --- /dev/null +++ b/doc/src/Regression/._Regression-bs069.html @@ -0,0 +1,466 @@ + + + + + + + + +Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis + + + + + + + + + + + + + + + + + + + + + + + +
+
+ + Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis +
+ +
+ +
+
+ + +
+ +

 

 

 

+ + + + +

Statistics, final expression

+
+
+

+The sample error (see eq. (24)) can now be +written in terms of the autocorrelation function: +$$ +\begin{align} +\mathrm{err}_X^2 &= +\frac{1}{n}\mathrm{var}(x)+\frac{2}{n}\cdot\mathrm{var}(x)\sum_{d=1}^{n-1} +\frac{f_d}{\mathrm{var}(x)}\nonumber\\ &=& +\left(1+2\sum_{d=1}^{n-1}\kappa_d\right)\frac{1}{n}\mathrm{var}(x)\nonumber\\ +&=\frac{\tau}{n}\cdot\mathrm{var}(x) +\tag{25} +\end{align} +$$ + +and we see that \( \mathrm{err}_X \) can be expressed in terms the +uncorrelated sample variance times a correction factor \( \tau \) which +accounts for the correlation between measurements. We call this +correction factor the autocorrelation time: +$$ +\begin{equation} +\tau = 1+2\sum_{d=1}^{n-1}\kappa_d +\tag{26} +\end{equation} +$$ +

+
+ + +

+

+ +

+ + +
+ + + + + + + +
+ +
+ + + + + + diff --git a/doc/src/Regression/._Regression-bs070.html b/doc/src/Regression/._Regression-bs070.html new file mode 100644 index 000000000..b5cb2bae8 --- /dev/null +++ b/doc/src/Regression/._Regression-bs070.html @@ -0,0 +1,458 @@ + + + + + + + + +Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis + + + + + + + + + + + + + + + + + + + + + + + +
+
+ + Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis +
+ +
+ +
+
+ + +
+ +

 

 

 

+ + + + +

Statistics, effective number of correlations

+
+
+

+For a correlation free experiment, \( \tau \) +equals 1. From the point of view of +eq. (25) we can interpret a sequential +correlation as an effective reduction of the number of measurements by +a factor \( \tau \). The effective number of measurements becomes: +$$ +n_\mathrm{eff} = \frac{n}{\tau} +$$ + +To neglect the autocorrelation time \( \tau \) will always cause our +simple uncorrelated estimate of \( \mathrm{err}_X^2\approx \mathrm{var}(x)/n \) to +be less than the true sample error. The estimate of the error will be +too good. On the other hand, the calculation of the full +autocorrelation time poses an efficiency problem if the set of +measurements is very large. +

+
+ + +

+

+ +

+ + +
+ + + + + + + +
+ +
+ + + + + + diff --git a/doc/src/Regression/._Regression-bs071.html b/doc/src/Regression/._Regression-bs071.html new file mode 100644 index 000000000..c8d786d91 --- /dev/null +++ b/doc/src/Regression/._Regression-bs071.html @@ -0,0 +1,470 @@ + + + + + + + + +Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis + + + + + + + + + + + + + + + + + + + + + + + +
+
+ + Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis +
+ +
+ +
+
+ + +
+ +

 

 

 

+ + + + +

Linking the regression analysis with a statistical interpretation

+ +

+Finally, we are going to discuss several statistical properties which can be obtained in terms of analytical expressions. +The +advantage of doing linear regression is that we actually end up with +analytical expressions for several statistical quantities. +Standard least squares and Ridge regression allow us to +derive quantities like the variance and other expectation values in a +rather straightforward way. + +

+It is assumed that \( \varepsilon_i +\sim \mathcal{N}(0, \sigma^2) \) and the \( \varepsilon_{i} \) are +independent, i.e.: +$$ +\begin{align*} +\mbox{Cov}(\varepsilon_{i_1}, +\varepsilon_{i_2}) & = \left\{ \begin{array}{lcc} \sigma^2 & \mbox{if} +& i_1 = i_2, \\ 0 & \mbox{if} & i_1 \not= i_2. \end{array} \right. +\end{align*} +$$ + +The randomness of \( \varepsilon_i \) implies that +\( \mathbf{y}_i \) is also a random variable. In particular, +\( \mathbf{y}_i \) is normally distributed, because \( \varepsilon_i \sim +\mathcal{N}(0, \sigma^2) \) and \( \mathbf{X}_{i,\ast} \, \boldsymbol{\beta} \) is a +non-random scalar. To specify the parameters of the distribution of +\( \mathbf{y}_i \) we need to calculate its first two moments. + +

+Recall that \( \boldsymbol{X} \) is a matrix of dimensionality \( n\times p \). The +notation above \( \mathbf{X}_{i,\ast} \) means that we are looking at the +row number \( i \) and perform a sum over all values \( p \). + +

+

+ +

+ + +
+ + + + + + + +
+ +
+ + + + + + diff --git a/doc/src/Regression/._Regression-bs072.html b/doc/src/Regression/._Regression-bs072.html new file mode 100644 index 000000000..ea9f32047 --- /dev/null +++ b/doc/src/Regression/._Regression-bs072.html @@ -0,0 +1,452 @@ + + + + + + + + +Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis + + + + + + + + + + + + + + + + + + + + + + + +
+
+ + Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis +
+ +
+ +
+
+ + +
+ +

 

 

 

+ + + + +

Assumptions made

+ +

+The assumption we have made here can be summarized as (and this is going to useful when we discuss the bias-variance trade off) +that there exists a function \( f(\boldsymbol{x}) \) and a normal distributed error \( \boldsymbol{\varepsilon}\sim \mathcal{N}(0, \sigma^2) \) +which describes our data +$$ +\boldsymbol{y} = f(\boldsymbol{x})+\boldsymbol{\varepsilon} +$$ + +

+We approximate this function with our model from the solution of the linear regression equations, that is our +function \( f \) is approximated by \( \boldsymbol{\tilde{y}} \) where we want to minimize \( (\boldsymbol{y}-\boldsymbol{\tilde{y}})^2 \), our MSE, with +$$ +\boldsymbol{\tilde{y}} = \boldsymbol{X}\boldsymbol{\beta}. +$$ + +

+

+ +

+ + +
+ + + + + + + +
+ +
+ + + + + + diff --git a/doc/src/Regression/._Regression-bs073.html b/doc/src/Regression/._Regression-bs073.html new file mode 100644 index 000000000..f6c1a7d10 --- /dev/null +++ b/doc/src/Regression/._Regression-bs073.html @@ -0,0 +1,467 @@ + + + + + + + + +Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis + + + + + + + + + + + + + + + + + + + + + + + +
+
+ + Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis +
+ +
+ +
+
+ + +
+ +

 

 

 

+ + + + +

Expectation value and variance

+ +

+We can calculate the expectation value of \( \boldsymbol{y} \) for a given element \( i \) +$$ +\begin{align*} +\mathbb{E}(y_i) & = +\mathbb{E}(\mathbf{X}_{i, \ast} \, \boldsymbol{\beta}) + \mathbb{E}(\varepsilon_i) +\, \, \, = \, \, \, \mathbf{X}_{i, \ast} \, \beta, +\end{align*} +$$ + +while +its variance is +$$ +\begin{align*} \mbox{Var}(y_i) & = \mathbb{E} \{ [y_i +- \mathbb{E}(y_i)]^2 \} \, \, \, = \, \, \, \mathbb{E} ( y_i^2 ) - +[\mathbb{E}(y_i)]^2 \\ & = \mathbb{E} [ ( \mathbf{X}_{i, \ast} \, +\beta + \varepsilon_i )^2] - ( \mathbf{X}_{i, \ast} \, \boldsymbol{\beta})^2 \\ & += \mathbb{E} [ ( \mathbf{X}_{i, \ast} \, \boldsymbol{\beta})^2 + 2 \varepsilon_i +\mathbf{X}_{i, \ast} \, \boldsymbol{\beta} + \varepsilon_i^2 ] - ( \mathbf{X}_{i, +\ast} \, \beta)^2 \\ & = ( \mathbf{X}_{i, \ast} \, \boldsymbol{\beta})^2 + 2 +\mathbb{E}(\varepsilon_i) \mathbf{X}_{i, \ast} \, \boldsymbol{\beta} + +\mathbb{E}(\varepsilon_i^2 ) - ( \mathbf{X}_{i, \ast} \, \boldsymbol{\beta})^2 +\\ & = \mathbb{E}(\varepsilon_i^2 ) \, \, \, = \, \, \, +\mbox{Var}(\varepsilon_i) \, \, \, = \, \, \, \sigma^2. +\end{align*} +$$ + +Hence, \( y_i \sim \mathcal{N}( \mathbf{X}_{i, \ast} \, \boldsymbol{\beta}, \sigma^2) \), that is \( \boldsymbol{y} \) follows a normal distribution with +mean value \( \boldsymbol{X}\boldsymbol{\beta} \) and variance \( \sigma^2 \) (not be confused with the singular values of the SVD). + +

+

+ +

+ + +
+ + + + + + + +
+ +
+ + + + + + diff --git a/doc/src/Regression/._Regression-bs074.html b/doc/src/Regression/._Regression-bs074.html new file mode 100644 index 000000000..0d34e7d64 --- /dev/null +++ b/doc/src/Regression/._Regression-bs074.html @@ -0,0 +1,517 @@ + + + + + + + + +Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis + + + + + + + + + + + + + + + + + + + + + + + +
+
+ + Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis +
+ +
+ +
+
+ + +
+ +

 

 

 

+ + + + +

Expectation value and variance for \( \boldsymbol{\beta} \)

+ +

+With the OLS expressions for the parameters \( \boldsymbol{\beta} \) we can evaluate the expectation value +$$ +\mathbb{E}(\boldsymbol{\beta}) = \mathbb{E}[ (\mathbf{X}^{\top} \mathbf{X})^{-1}\mathbf{X}^{T} \mathbf{Y}]=(\mathbf{X}^{T} \mathbf{X})^{-1}\mathbf{X}^{T} \mathbb{E}[ \mathbf{Y}]=(\mathbf{X}^{T} \mathbf{X})^{-1} \mathbf{X}^{T}\mathbf{X}\boldsymbol{\beta}=\boldsymbol{\beta}. +$$ + +This means that the estimator of the regression parameters is unbiased. + +

+We can also calculate the variance + +

+The variance of \( \boldsymbol{\beta} \) is +$$ +\begin{eqnarray*} +\mbox{Var}(\boldsymbol{\beta}) & = & \mathbb{E} \{ [\boldsymbol{\beta} - \mathbb{E}(\boldsymbol{\beta})] [\boldsymbol{\beta} - \mathbb{E}(\boldsymbol{\beta})]^{T} \} +\\ +& = & \mathbb{E} \{ [(\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{Y} - \boldsymbol{\beta}] \, [(\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{Y} - \boldsymbol{\beta}]^{T} \} +\\ +% & = & \mathbb{E} \{ [(\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{Y}] \, [(\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{Y}]^{T} \} - \boldsymbol{\beta} \, \boldsymbol{\beta}^{T} +% \\ +% & = & \mathbb{E} \{ (\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{Y} \, \mathbf{Y}^{T} \, \mathbf{X} \, (\mathbf{X}^{T} \mathbf{X})^{-1} \} - \boldsymbol{\beta} \, \boldsymbol{\beta}^{T} +% \\ +& = & (\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \, \mathbb{E} \{ \mathbf{Y} \, \mathbf{Y}^{T} \} \, \mathbf{X} \, (\mathbf{X}^{T} \mathbf{X})^{-1} - \boldsymbol{\beta} \, \boldsymbol{\beta}^{T} +\\ +& = & (\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \, \{ \mathbf{X} \, \boldsymbol{\beta} \, \boldsymbol{\beta}^{T} \, \mathbf{X}^{T} + \sigma^2 \} \, \mathbf{X} \, (\mathbf{X}^{T} \mathbf{X})^{-1} - \boldsymbol{\beta} \, \boldsymbol{\beta}^{T} +% \\ +% & = & (\mathbf{X}^T \mathbf{X})^{-1} \, \mathbf{X}^T \, \mathbf{X} \, \boldsymbol{\beta} \, \boldsymbol{\beta}^T \, \mathbf{X}^T \, \mathbf{X} \, (\mathbf{X}^T % \mathbf{X})^{-1} +% \\ +% & & + \, \, \sigma^2 \, (\mathbf{X}^T \mathbf{X})^{-1} \, \mathbf{X}^T \, \mathbf{X} \, (\mathbf{X}^T \mathbf{X})^{-1} - \boldsymbol{\beta} \boldsymbol{\beta}^T +\\ +& = & \boldsymbol{\beta} \, \boldsymbol{\beta}^{T} + \sigma^2 \, (\mathbf{X}^{T} \mathbf{X})^{-1} - \boldsymbol{\beta} \, \boldsymbol{\beta}^{T} +\, \, \, = \, \, \, \sigma^2 \, (\mathbf{X}^{T} \mathbf{X})^{-1}, +\end{eqnarray*} +$$ + +

+where we have used that \( \mathbb{E} (\mathbf{Y} \mathbf{Y}^{T}) = +\mathbf{X} \, \boldsymbol{\beta} \, \boldsymbol{\beta}^{T} \, \mathbf{X}^{T} + +\sigma^2 \, \mathbf{I}_{nn} \). From \( \mbox{Var}(\boldsymbol{\beta}) = \sigma^2 +\, (\mathbf{X}^{T} \mathbf{X})^{-1} \), one obtains an estimate of the +variance of the estimate of the \( j \)-th regression coefficient: +\( \hat{\sigma}^2 (\hat{\beta}_j ) = \hat{\sigma}^2 \sqrt{ +[(\mathbf{X}^{T} \mathbf{X})^{-1}]_{jj} } \). This may be used to +construct a confidence interval for the estimates. + +

+In a similar way, we cna obtain analytical expressions for say the +expectation values of the parameters \( \boldsymbol{\beta} \) and their variance +when we employ Ridge regression, and thereby a confidence interval. + +

+It is rather straightforward to show that +$$ +\mathbb{E} \big[ \boldsymbol{\beta}^{\mathrm{Ridge}} \big]=(\mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I}_{pp})^{-1} (\mathbf{X}^{\top} \mathbf{X})\boldsymbol{\beta}^{\mathrm{OLS}}. +$$ + +We see clearly that +\( \mathbb{E} \big[ \boldsymbol{\beta}^{\mathrm{Ridge}} \big] \not= \boldsymbol{\beta}^{\mathrm{OLS}} \) for any \( \lambda > 0 \). We say then that the ridge estimator is biased. + +

+We can also compute the variance as + +$$ +\mbox{Var}[\boldsymbol{\beta}^{\mathrm{Ridge}}]=\sigma^2[ \mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I} ]^{-1} \mathbf{X}^{T} \mathbf{X} \{ [ \mathbf{X}^{\top} \mathbf{X} + \lambda \mathbf{I} ]^{-1}\}^{T}, +$$ + +and it is easy to see that if the parameter \( \lambda \) goes to infinity then the variance of Ridge parameters \( \boldsymbol{\beta} \) goes to zero. + +

+With this, we can compute the difference + +$$ +\mbox{Var}[\boldsymbol{\beta}^{\mathrm{OLS}}]-\mbox{Var}(\boldsymbol{\beta}^{\mathrm{Ridge}})=\sigma^2 [ \mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I} ]^{-1}[ 2\lambda\mathbf{I} + \lambda^2 (\mathbf{X}^{T} \mathbf{X})^{-1} ] \{ [ \mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I} ]^{-1}\}^{T}. +$$ + +The difference is non-negative definite since each component of the +matrix product is non-negative definite. +This means the variance we obtain with the standard OLS will always for \( \lambda > 0 \) be larger than the variance of \( \boldsymbol{\beta} \) obtained with the Ridge estimator. This has interesting consequences when we discuss the so-called bias-variance trade-off below. + +

+

+ +

+ + +
+ + + + + + + +
+ +
+ + + + + + diff --git a/doc/src/Regression/._Regression-bs075.html b/doc/src/Regression/._Regression-bs075.html new file mode 100644 index 000000000..63a117fc0 --- /dev/null +++ b/doc/src/Regression/._Regression-bs075.html @@ -0,0 +1,466 @@ + + + + + + + + +Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis + + + + + + + + + + + + + + + + + + + + + + + +
+
+ + Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis +
+ +
+ +
+
+ + +
+ +

 

 

 

+ + + + +

Cross-validation

+ +

+Instead of choosing the penalty parameter to balance model fit with +model complexity, cross-validation requires it (i.e. the penalty +parameter) to yield a model with good prediction +performance. Commonly, this performance is evaluated on novel +data. Novel data need not be easy to come by and one has to make do +with the data at hand. + +

+The setting of original and novel data is +then mimicked by sample splitting: the data set is divided into two +(groups of samples). One of these two data sets, called the +training set, plays the role of original data on which the model is +built. The second of these data sets, called the test set, plays the +role of the novel data and is used to evaluate the prediction +performance (often operationalized as the log-likelihood or the +prediction error or its square or the R2 score) of the model built on the training data set. This +procedure (model building and prediction evaluation on training and +test set, respectively) is done for a collection of possible penalty +parameter choices. The penalty parameter that yields the model with +the best prediction performance is to be preferred. The thus obtained +performance evaluation depends on the actual split of the data set. To +remove this dependence the data set is split many times into a +training and test set. For each split the model parameters are +estimated for all choices of \( \lambda \) using the training data and +estimated parameters are evaluated on the corresponding test set. The +penalty parameter that on average over the test sets performs best (in +some sense) is then selected. + +

+

+ +

+ + +
+ + + + + + + +
+ +
+ + + + + + diff --git a/doc/src/Regression/._Regression-bs076.html b/doc/src/Regression/._Regression-bs076.html new file mode 100644 index 000000000..1307d54da --- /dev/null +++ b/doc/src/Regression/._Regression-bs076.html @@ -0,0 +1,444 @@ + + + + + + + + +Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis + + + + + + + + + + + + + + + + + + + + + + + +
+
+ + Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis +
+ +
+ +
+
+ + +
+ +

 

 

 

+ + + + +

Computationally expensive

+ +

+The validation set approach is conceptually simple and is easy to implement. But it has two potential drawbacks: + +

    +
  • The validation estimate of the test error rate can be highly variable, depending on precisely which observations are included in the training set and which observations are included in the validation set.
    • +
  • In the validation approach, only a subset of the observations, those that are included in the training set rather than in the validation set are used to fit the model. Since statistical methods tend to perform worse when trained on fewer observations, this suggests that the validation set error rate may tend to overestimate the test error rate for the model fit on the entire data set.
    • +
+ +

+ +

+ + +
+ + + + + + + +
+ +
+ + + + + + diff --git a/doc/src/Regression/._Regression-bs077.html b/doc/src/Regression/._Regression-bs077.html new file mode 100644 index 000000000..f9822b325 --- /dev/null +++ b/doc/src/Regression/._Regression-bs077.html @@ -0,0 +1,453 @@ + + + + + + + + +Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis + + + + + + + + + + + + + + + + + + + + + + + +
+
+ + Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis +
+ +
+ +
+
+ + +
+ +

 

 

 

+ + + + +

Various steps in cross-validation

+ +

+When the repetitive splitting of the data set is done randomly, +samples may accidently end up in a fast majority of the splits in +either training or test set. Such samples may have an unbalanced +influence on either model building or prediction evaluation. To avoid +this \( k \)-fold cross-validation structures the data splitting. The +samples are divided into \( k \) more or less equally sized exhaustive and +mutually exclusive subsets. In turn (at each split) one of these +subsets plays the role of the test set while the union of the +remaining subsets constitutes the training set. Such a splitting +warrants a balanced representation of each sample in both training and +test set over the splits. Still the division into the \( k \) subsets +involves a degree of randomness. This may be fully excluded when +choosing \( k=n \). This particular case is referred to as leave-one-out +cross-validation (LOOCV). + +

+

+ +

+ + +
+ + + + + + + +
+ +
+ + + + + + diff --git a/doc/src/Regression/._Regression-bs078.html b/doc/src/Regression/._Regression-bs078.html new file mode 100644 index 000000000..bfe54afc5 --- /dev/null +++ b/doc/src/Regression/._Regression-bs078.html @@ -0,0 +1,468 @@ + + + + + + + + +Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis + + + + + + + + + + + + + + + + + + + + + + + +
+
+ + Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis +
+ +
+ +
+
+ + +
+ +

 

 

 

+ + + + +

How to set up the cross-validation for Ridge and/or Lasso

+ +
    +
  • Define a range of interest for the penalty parameter.
    • +
  • Divide the data set into training and test set comprising samples \( \{1, \ldots, n\} \setminus i \) and \( \{ i \} \), respectively.
    • +
  • Fit the linear regression model by means of ridge estimation for each \( \lambda \) in the grid using the training set, and the corresponding estimate of the error variance \( \boldsymbol{\sigma}_{-i}^2(\lambda) \), as
    • +
+ +$$ +\begin{align*} +\boldsymbol{\beta}_{-i}(\lambda) & = ( \boldsymbol{X}_{-i, \ast}^{T} +\boldsymbol{X}_{-i, \ast} + \lambda \boldsymbol{I}_{pp})^{-1} +\boldsymbol{X}_{-i, \ast}^{T} \boldsymbol{y}_{-i} +\end{align*} +$$ + + +
    +
  • Evaluate the prediction performance of these models on the test set by \( \log\{L[y_i, \boldsymbol{X}_{i, \ast}; \boldsymbol{\beta}_{-i}(\lambda), \boldsymbol{\sigma}_{-i}^2(\lambda)]\} \). Or, by the prediction error \( |y_i - \boldsymbol{X}_{i, \ast} \boldsymbol{\beta}_{-i}(\lambda)| \), the relative error, the error squared or the R2 score function.
    • +
  • Repeat the first three steps such that each sample plays the role of the test set once.
    • +
  • Average the prediction performances of the test sets at each grid point of the penalty bias/parameter by computing the cross-validated log-likelihood. It is an estimate of the prediction performance of the model corresponding to this value of the penalty parameter on novel data. It is defined as
    • +
+ +$$ +\begin{align*} +\frac{1}{n} \sum_{i = 1}^n \log\{L[y_i, \mathbf{X}_{i, \ast}; \boldsymbol{\beta}_{-i}(\lambda), \boldsymbol{\sigma}_{-i}^2(\lambda)]\}. +\end{align*} +$$ + + +
    +
  • The value of the penalty parameter that maximizes the cross-validated log-likelihood is the value of choice. Or we can use the MSE or the R2 score functions.
    • +
+ +

+ +

+ + +
+ + + + + + + +
+ +
+ + + + + + diff --git a/doc/src/Regression/._Regression-bs079.html b/doc/src/Regression/._Regression-bs079.html new file mode 100644 index 000000000..79e1d53f1 --- /dev/null +++ b/doc/src/Regression/._Regression-bs079.html @@ -0,0 +1,455 @@ + + + + + + + + +Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis + + + + + + + + + + + + + + + + + + + + + + + +
+
+ + Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis +
+ +
+ +
+
+ + +
+ +

 

 

 

+ + + + +

Resampling methods: Jackknife and Bootstrap

+ +

+Two famous +resampling methods are the independent bootstrap and the jackknife. + +

+The jackknife is a special case of the independent bootstrap. Still, the jackknife was made +popular prior to the independent bootstrap. And as the popularity of +the independent bootstrap soared, new variants, such as the dependent bootstrap. + +

+The Jackknife and independent bootstrap work for +independent, identically distributed random variables. +If these conditions are not +satisfied, the methods will fail. Yet, it should be said that if the data are +independent, identically distributed, and we only want to estimate the +variance of \( \overline{X} \) (which often is the case), then there is no +need for bootstrapping. + +

+

+ +

+ + +
+ + + + + + + +
+ +
+ + + + + + diff --git a/doc/src/Regression/._Regression-bs080.html b/doc/src/Regression/._Regression-bs080.html new file mode 100644 index 000000000..efecd3647 --- /dev/null +++ b/doc/src/Regression/._Regression-bs080.html @@ -0,0 +1,451 @@ + + + + + + + + +Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis + + + + + + + + + + + + + + + + + + + + + + + +
+
+ + Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis +
+ +
+ +
+
+ + +
+ +

 

 

 

+ + + + +

Resampling methods: Jackknife

+ +

+The Jackknife works by making many replicas of the estimator \( \widehat{\theta} \). +The jackknife is a resampling method where we systematically leave out one observation from the vector of observed values \( \boldsymbol{x} = (x_1,x_2,\cdots,X_n) \). +Let \( \boldsymbol{x}_i \) denote the vector +$$ +\boldsymbol{x}_i = (x_1,x_2,\cdots,x_{i-1},x_{i+1},\cdots,x_n), +$$ + +

+which equals the vector \( \boldsymbol{x} \) with the exception that observation +number \( i \) is left out. Using this notation, define +\( \widehat{\theta}_i \) to be the estimator +\( \widehat{\theta} \) computed using \( \vec{X}_i \). + +

+

+ +

+ + +
+ + + + + + + +
+ +
+ + + + + + diff --git a/doc/src/Regression/._Regression-bs081.html b/doc/src/Regression/._Regression-bs081.html new file mode 100644 index 000000000..d69fce222 --- /dev/null +++ b/doc/src/Regression/._Regression-bs081.html @@ -0,0 +1,468 @@ + + + + + + + + +Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis + + + + + + + + + + + + + + + + + + + + + + + +
+
+ + Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis +
+ +
+ +
+
+ + +
+ +

 

 

 

+ + + + +

Jackknife code example

+

+ + +

from numpy import *
+from numpy.random import randint, randn
+from time import time
+
+def jackknife(data, stat):
+    n = len(data);t = zeros(n); inds = arange(n); t0 = time()
+    ## 'jackknifing' by leaving out an observation for each i                                                                                                                      
+    for i in range(n):
+        t[i] = stat(delete(data,i) )
+
+    # analysis                                                                                                                                                                     
+    print("Runtime: %g sec" % (time()-t0)); print("Jackknife Statistics :")
+    print("original           bias      std. error")
+    print("%8g %14g %15g" % (stat(data),(n-1)*mean(t)/n, (n*var(t))**.5))
+
+    return t
+
+
+# Returns mean of data samples                                                                                                                                                     
+def stat(data):
+    return mean(data)
+
+
+mu, sigma = 100, 15
+datapoints = 10000
+x = mu + sigma*random.randn(datapoints)
+# jackknife returns the data sample                                                                                                                                                
+t = jackknife(x, stat)
+
+

+

+ +

+ + +
+ + + + + + + +
+ +
+ + + + + + diff --git a/doc/src/Regression/._Regression-bs082.html b/doc/src/Regression/._Regression-bs082.html new file mode 100644 index 000000000..88c2e3683 --- /dev/null +++ b/doc/src/Regression/._Regression-bs082.html @@ -0,0 +1,454 @@ + + + + + + + + +Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis + + + + + + + + + + + + + + + + + + + + + + + +
+
+ + Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis +
+ +
+ +
+
+ + +
+ +

 

 

 

+ + + + +

Resampling methods: Bootstrap

+
+
+

+Bootstrapping is a nonparametric approach to statistical inference +that substitutes computation for more traditional distributional +assumptions and asymptotic results. Bootstrapping offers a number of +advantages: + +

    +
  1. The bootstrap is quite general, although there are some cases in which it fails.
    2. +
  2. Because it does not require distributional assumptions (such as normally distributed errors), the bootstrap can provide more accurate inferences when the data are not well behaved or when the sample size is small.
    3. +
  3. It is possible to apply the bootstrap to statistics with sampling distributions that are difficult to derive, even asymptotically.
    4. +
  4. It is relatively simple to apply the bootstrap to complex data-collection plans (such as stratified and clustered samples).
    5. +
+
+
+ + +

+

+ +

+ + +
+ + + + + + + +
+ +
+ + + + + + diff --git a/doc/src/Regression/._Regression-bs083.html b/doc/src/Regression/._Regression-bs083.html new file mode 100644 index 000000000..f13ff3412 --- /dev/null +++ b/doc/src/Regression/._Regression-bs083.html @@ -0,0 +1,448 @@ + + + + + + + + +Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis + + + + + + + + + + + + + + + + + + + + + + + +
+
+ + Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis +
+ +
+ +
+
+ + +
+ +

 

 

 

+ + + + +

Resampling methods: Bootstrap background

+ +

+Since \( \widehat{\theta} = \widehat{\theta}(\boldsymbol{X}) \) is a function of random variables, +\( \widehat{\theta} \) itself must be a random variable. Thus it has +a pdf, call this function \( p(\boldsymbol{t}) \). The aim of the bootstrap is to +estimate \( p(\boldsymbol{t}) \) by the relative frequency of +\( \widehat{\theta} \). You can think of this as using a histogram +in the place of \( p(\boldsymbol{t}) \). If the relative frequency closely +resembles \( p(\vec{t}) \), then using numerics, it is straight forward to +estimate all the interesting parameters of \( p(\boldsymbol{t}) \) using point +estimators. + +

+

+ +

+ + +
+ + + + + + + +
+ +
+ + + + + + diff --git a/doc/src/Regression/._Regression-bs084.html b/doc/src/Regression/._Regression-bs084.html new file mode 100644 index 000000000..563450133 --- /dev/null +++ b/doc/src/Regression/._Regression-bs084.html @@ -0,0 +1,454 @@ + + + + + + + + +Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis + + + + + + + + + + + + + + + + + + + + + + + +
+
+ + Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis +
+ +
+ +
+
+ + +
+ +

 

 

 

+ + + + +

Resampling methods: More Bootstrap background

+ +

+In the case that \( \widehat{\theta} \) has +more than one component, and the components are independent, we use the +same estimator on each component separately. If the probability +density function of \( X_i \), \( p(x) \), had been known, then it would have +been straight forward to do this by: + +

    +
  1. Drawing lots of numbers from \( p(x) \), suppose we call one such set of numbers \( (X_1^*, X_2^*, \cdots, X_n^*) \).
    2. +
  2. Then using these numbers, we could compute a replica of \( \widehat{\theta} \) called \( \widehat{\theta}^* \).
    3. +
+ +By repeated use of (1) and (2), many +estimates of \( \widehat{\theta} \) could have been obtained. The +idea is to use the relative frequency of \( \widehat{\theta}^* \) +(think of a histogram) as an estimate of \( p(\boldsymbol{t}) \). + +

+

+ +

+ + +
+ + + + + + + +
+ +
+ + + + + + diff --git a/doc/src/Regression/._Regression-bs085.html b/doc/src/Regression/._Regression-bs085.html new file mode 100644 index 000000000..ae9cc591e --- /dev/null +++ b/doc/src/Regression/._Regression-bs085.html @@ -0,0 +1,453 @@ + + + + + + + + +Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis + + + + + + + + + + + + + + + + + + + + + + + +
+
+ + Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis +
+ +
+ +
+
+ + +
+ +

 

 

 

+ + + + +

Resampling methods: Bootstrap approach

+ +

+But +unless there is enough information available about the process that +generated \( X_1,X_2,\cdots,X_n \), \( p(x) \) is in general +unknown. Therefore, Efron in 1979 asked the +question: What if we replace \( p(x) \) by the relative frequency +of the observation \( X_i \); if we draw observations in accordance with +the relative frequency of the observations, will we obtain the same +result in some asymptotic sense? The answer is yes. + +

+Instead of generating the histogram for the relative +frequency of the observation \( X_i \), just draw the values +\( (X_1^*,X_2^*,\cdots,X_n^*) \) with replacement from the vector +\( \boldsymbol{X} \). + +

+

+ +

+ + +
+ + + + + + + +
+ +
+ + + + + + diff --git a/doc/src/Regression/._Regression-bs086.html b/doc/src/Regression/._Regression-bs086.html new file mode 100644 index 000000000..784594eb9 --- /dev/null +++ b/doc/src/Regression/._Regression-bs086.html @@ -0,0 +1,457 @@ + + + + + + + + +Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis + + + + + + + + + + + + + + + + + + + + + + + +
+
+ + Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis +
+ +
+ +
+
+ + +
+ +

 

 

 

+ + + + +

Resampling methods: Bootstrap steps

+ +

+The independent bootstrap works like this: + +

    +
  1. Draw with replacement \( n \) numbers for the observed variables \( \boldsymbol{x} = (x_1,x_2,\cdots,x_n) \).
    2. +
  2. Define a vector \( \boldsymbol{x}^* \) containing the values which were drawn from \( \boldsymbol{x} \).
    3. +
  3. Using the vector \( \boldsymbol{x}^* \) compute \( \widehat{\theta}^* \) by evaluating \( \widehat \theta \) under the observations \( \boldsymbol{x}^* \).
    4. +
  4. Repeat this process \( k \) times.
    5. +
+ +When you are done, you can draw a histogram of the relative frequency +of \( \widehat \theta^* \). This is your estimate of the probability +distribution \( p(t) \). Using this probability distribution you can +estimate any statistics thereof. In principle you never draw the +histogram of the relative frequency of \( \widehat{\theta}^* \). Instead +you use the estimators corresponding to the statistic of interest. For +example, if you are interested in estimating the variance of \( \widehat +\theta \), apply the etsimator \( \widehat \sigma^2 \) to the values +\( \widehat \theta ^* \). + +

+

+ +

+ + +
+ + + + + + + +
+ +
+ + + + + + diff --git a/doc/src/Regression/._Regression-bs087.html b/doc/src/Regression/._Regression-bs087.html new file mode 100644 index 000000000..a2bb69c80 --- /dev/null +++ b/doc/src/Regression/._Regression-bs087.html @@ -0,0 +1,496 @@ + + + + + + + + +Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis + + + + + + + + + + + + + + + + + + + + + + + +
+
+ + Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis +
+ +
+ +
+
+ + +
+ +

 

 

 

+ + + + +

Code example for the Bootstrap method

+ +

+The following code starts with a Gaussian distribution with mean value +\( \mu =100 \) and variance \( \sigma=15 \). We use this to generate the data +used in the bootstrap analysis. The bootstrap analysis returns a data +set after a given number of bootstrap operations (as many as we have +data points). This data set consists of estimated mean values for each +bootstrap operation. The histogram generated by the bootstrap method +shows that the distribution for these mean values is also a Gaussian, +centered around the mean value \( \mu=100 \) but with standard deviation +\( \sigma/\sqrt{n} \), where \( n \) is the number of bootstrap samples (in +this case the same as the number of original data points). The value +of the standard deviation is what we expect from the central limit +theorem. + +

+ + +

from numpy import *
+from numpy.random import randint, randn
+from time import time
+import matplotlib.mlab as mlab
+import matplotlib.pyplot as plt
+
+# Returns mean of bootstrap samples                                                                                                                                                
+def stat(data):
+    return mean(data)
+
+# Bootstrap algorithm
+def bootstrap(data, statistic, R):
+    t = zeros(R); n = len(data); inds = arange(n); t0 = time()
+    # non-parametric bootstrap         
+    for i in range(R):
+        t[i] = statistic(data[randint(0,n,n)])
+
+    # analysis    
+    print("Runtime: %g sec" % (time()-t0)); print("Bootstrap Statistics :")
+    print("original           bias      std. error")
+    print("%8g %8g %14g %15g" % (statistic(data), std(data),mean(t),std(t)))
+    return t
+
+
+mu, sigma = 100, 15
+datapoints = 10000
+x = mu + sigma*random.randn(datapoints)
+# bootstrap returns the data sample                                    
+t = bootstrap(x, stat, datapoints)
+# the histogram of the bootstrapped  data                                                                                                    
+n, binsboot, patches = plt.hist(t, 50, normed=1, facecolor='red', alpha=0.75)
+
+# add a 'best fit' line  
+y = mlab.normpdf( binsboot, mean(t), std(t))
+lt = plt.plot(binsboot, y, 'r--', linewidth=1)
+plt.xlabel('Smarts')
+plt.ylabel('Probability')
+plt.axis([99.5, 100.6, 0, 3.0])
+plt.grid(True)
+
+plt.show()
+
+

+

+ +

+ + +
+ + + + + + + +
+ +
+ + + + + + diff --git a/doc/src/Regression/._Regression-bs088.html b/doc/src/Regression/._Regression-bs088.html new file mode 100644 index 000000000..712697d73 --- /dev/null +++ b/doc/src/Regression/._Regression-bs088.html @@ -0,0 +1,532 @@ + + + + + + + + +Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis + + + + + + + + + + + + + + + + + + + + + + + +
+
+ + Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis +
+ +
+ +
+
+ + +
+ +

 

 

 

+ + + + +

Code Example for Cross-validation and \( k \)-fold Cross-validation

+ +

+The code here uses Ridge regression with cross-validation (CV) resampling and \( k \)-fold CV in order to fit a specific polynomial. +

+ + +

import numpy as np
+import matplotlib.pyplot as plt
+from sklearn.model_selection import KFold
+from sklearn.linear_model import Ridge
+from sklearn.model_selection import cross_val_score
+from sklearn.preprocessing import PolynomialFeatures
+
+# A seed just to ensure that the random numbers are the same for every run.
+# Useful for eventual debugging.
+np.random.seed(3155)
+
+# Generate the data.
+nsamples = 100
+x = np.random.randn(nsamples)
+y = 3*x**2 + np.random.randn(nsamples)
+
+## Cross-validation on Ridge regression using KFold only
+
+# Decide degree on polynomial to fit
+poly = PolynomialFeatures(degree = 6)
+
+# Decide which values of lambda to use
+nlambdas = 500
+lambdas = np.logspace(-3, 5, nlambdas)
+
+# Initialize a KFold instance
+k = 5
+kfold = KFold(n_splits = k)
+
+# Perform the cross-validation to estimate MSE
+scores_KFold = np.zeros((nlambdas, k))
+
+i = 0
+for lmb in lambdas:
+    ridge = Ridge(alpha = lmb)
+    j = 0
+    for train_inds, test_inds in kfold.split(x):
+        xtrain = x[train_inds]
+        ytrain = y[train_inds]
+
+        xtest = x[test_inds]
+        ytest = y[test_inds]
+
+        Xtrain = poly.fit_transform(xtrain[:, np.newaxis])
+        ridge.fit(Xtrain, ytrain[:, np.newaxis])
+
+        Xtest = poly.fit_transform(xtest[:, np.newaxis])
+        ypred = ridge.predict(Xtest)
+
+        scores_KFold[i,j] = np.sum((ypred - ytest[:, np.newaxis])**2)/np.size(ypred)
+
+        j += 1
+    i += 1
+
+
+estimated_mse_KFold = np.mean(scores_KFold, axis = 1)
+
+## Cross-validation using cross_val_score from sklearn along with KFold
+
+# kfold is an instance initialized above as:
+# kfold = KFold(n_splits = k)
+
+estimated_mse_sklearn = np.zeros(nlambdas)
+i = 0
+for lmb in lambdas:
+    ridge = Ridge(alpha = lmb)
+
+    X = poly.fit_transform(x[:, np.newaxis])
+    estimated_mse_folds = cross_val_score(ridge, X, y[:, np.newaxis], scoring='neg_mean_squared_error', cv=kfold)
+
+    # cross_val_score return an array containing the estimated negative mse for every fold.
+    # we have to the the mean of every array in order to get an estimate of the mse of the model
+    estimated_mse_sklearn[i] = np.mean(-estimated_mse_folds)
+
+    i += 1
+
+## Plot and compare the slightly different ways to perform cross-validation
+
+plt.figure()
+
+plt.plot(np.log10(lambdas), estimated_mse_sklearn, label = 'cross_val_score')
+plt.plot(np.log10(lambdas), estimated_mse_KFold, 'r--', label = 'KFold')
+
+plt.xlabel('log10(lambda)')
+plt.ylabel('mse')
+
+plt.legend()
+
+plt.show()
+
+

+

+ +

+ + +
+ + + + + + + +
+ +
+ + + + + + diff --git a/doc/src/Regression/._Regression-bs089.html b/doc/src/Regression/._Regression-bs089.html new file mode 100644 index 000000000..76d15ca46 --- /dev/null +++ b/doc/src/Regression/._Regression-bs089.html @@ -0,0 +1,498 @@ + + + + + + + + +Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis + + + + + + + + + + + + + + + + + + + + + + + +
+
+ + Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis +
+ +
+ +
+
+ + +
+ +

 

 

 

+ + + + +

The bias-variance tradeoff

+ +

+We will discuss the bias-variance tradeoff in the context of +continuous predictions such as regression. However, many of the +intuitions and ideas discussed here also carry over to classification +tasks. Consider a dataset \( \mathcal{L} \) consisting of the data +\( \mathbf{X}_\mathcal{L}=\{(y_j, \boldsymbol{x}_j), j=0\ldots n-1\} \). + +

+Let us assume that the true data is generated from a noisy model + +$$ +\boldsymbol{y}=f(\boldsymbol{x}) + \boldsymbol{\epsilon} +$$ + +

+where \( \epsilon \) is normally distributed with mean zero and standard deviation \( \sigma^2 \). + +

+In our derivation of the ordinary least squares method we defined then +an approximation to the function \( f \) in terms of the parameters +\( \boldsymbol{\beta} \) and the design matrix \( \boldsymbol{X} \) which embody our model, +that is \( \boldsymbol{\tilde{y}}=\boldsymbol{X}\boldsymbol{\beta} \). + +

+Thereafter we found the parameters \( \boldsymbol{\beta} \) by optimizing the means squared error via the so-called cost function +$$ +C(\boldsymbol{X},\boldsymbol{\beta}) =\frac{1}{n}\sum_{i=0}^{n-1}(y_i-\tilde{y}_i)^2=\mathbb{E}\left[(\boldsymbol{y}-\boldsymbol{\tilde{y}})^2\right]. +$$ + +

+We can rewrite this as +$$ +\mathbb{E}\left[(\boldsymbol{y}-\boldsymbol{\tilde{y}})^2\right]=\frac{1}{n}\sum_i(f_i-\mathbb{E}\left[\boldsymbol{\tilde{y}}\right])^2+\frac{1}{n}\sum_i(\tilde{y}_i-\mathbb{E}\left[\boldsymbol{\tilde{y}}\right])^2+\sigma^2. +$$ + +

+The three terms represent the square of the bias of the learning +method, which can be thought of as the error caused by the simplifying +assumptions built into the method. The second term represents the +variance of the chosen model and finally the last terms is variance of +the error \( \boldsymbol{\epsilon} \). + +

+To derive this equation, we need to recall that the variance of \( \boldsymbol{y} \) and \( \boldsymbol{\epsilon} \) are both equal to \( \sigma^2 \). The mean value of \( \boldsymbol{\epsilon} \) is by definition equal to zero. Furthermore, the function \( f \) is not a stochastics variable, idem for \( \boldsymbol{\tilde{y}} \). +We use a more compact notation in terms of the expectation value +$$ +\mathbb{E}\left[(\boldsymbol{y}-\boldsymbol{\tilde{y}})^2\right]=\mathbb{E}\left[(\boldsymbol{f}+\boldsymbol{\epsilon}-\boldsymbol{\tilde{y}})^2\right], +$$ + +and adding and subtracting \( \mathbb{E}\left[\boldsymbol{\tilde{y}}\right] \) we get +$$ +\mathbb{E}\left[(\boldsymbol{y}-\boldsymbol{\tilde{y}})^2\right]=\mathbb{E}\left[(\boldsymbol{f}+\boldsymbol{\epsilon}-\boldsymbol{\tilde{y}}+\mathbb{E}\left[\boldsymbol{\tilde{y}}\right]-\mathbb{E}\left[\boldsymbol{\tilde{y}}\right])^2\right], +$$ + +which, using the abovementioned expectation values can be rewritten as +$$ +\mathbb{E}\left[(\boldsymbol{y}-\boldsymbol{\tilde{y}})^2\right]=\mathbb{E}\left[(\boldsymbol{y}-\mathbb{E}\left[\boldsymbol{\tilde{y}}\right])^2\right]+\mathrm{Var}\left[\boldsymbol{\tilde{y}}\right]+\sigma^2, +$$ + +that is the rewriting in terms of the so-called bias, the variance of the model \( \boldsymbol{\tilde{y}} \) and the variance of \( \boldsymbol{\epsilon} \). + +

+

+ +

+ + +
+ + + + + + + +
+ +
+ + + + + + diff --git a/doc/src/Regression/._Regression-bs090.html b/doc/src/Regression/._Regression-bs090.html new file mode 100644 index 000000000..4262a3a91 --- /dev/null +++ b/doc/src/Regression/._Regression-bs090.html @@ -0,0 +1,492 @@ + + + + + + + + +Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis + + + + + + + + + + + + + + + + + + + + + + + +
+
+ + Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis +
+ +
+ +
+
+ + +
+ +

 

 

 

+ + + + +

Example code for Bias-Variance tradeoff

+

+ + +

import matplotlib.pyplot as plt
+import numpy as np
+from sklearn.linear_model import LinearRegression, Ridge, Lasso
+from sklearn.preprocessing import PolynomialFeatures
+from sklearn.model_selection import train_test_split
+from sklearn.pipeline import make_pipeline
+from sklearn.utils import resample
+
+np.random.seed(2018)
+
+n = 500
+n_boostraps = 100
+degree = 18  # A quite high value, just to show.
+noise = 0.1
+
+# Make data set.
+x = np.linspace(-1, 3, n).reshape(-1, 1)
+y = np.exp(-x**2) + 1.5 * np.exp(-(x-2)**2) + np.random.normal(0, 0.1, x.shape)
+
+# Hold out some test data that is never used in training.
+x_train, x_test, y_train, y_test = train_test_split(x, y, test_size=0.2)
+
+# Combine x transformation and model into one operation.
+# Not neccesary, but convenient.
+model = make_pipeline(PolynomialFeatures(degree=degree), LinearRegression(fit_intercept=False))
+
+# The following (m x n_bootstraps) matrix holds the column vectors y_pred
+# for each bootstrap iteration.
+y_pred = np.empty((y_test.shape[0], n_boostraps))
+for i in range(n_boostraps):
+    x_, y_ = resample(x_train, y_train)
+
+    # Evaluate the new model on the same test data each time.
+    y_pred[:, i] = model.fit(x_, y_).predict(x_test).ravel()
+
+# Note: Expectations and variances taken w.r.t. different training
+# data sets, hence the axis=1. Subsequent means are taken across the test data
+# set in order to obtain a total value, but before this we have error/bias/variance
+# calculated per data point in the test set.
+# Note 2: The use of keepdims=True is important in the calculation of bias as this 
+# maintains the column vector form. Dropping this yields very unexpected results.
+error = np.mean( np.mean((y_test - y_pred)**2, axis=1, keepdims=True) )
+bias = np.mean( (y_test - np.mean(y_pred, axis=1, keepdims=True))**2 )
+variance = np.mean( np.var(y_pred, axis=1, keepdims=True) )
+print('Error:', error)
+print('Bias^2:', bias)
+print('Var:', variance)
+print('{} >= {} + {} = {}'.format(error, bias, variance, bias+variance))
+
+plt.plot(x[::5, :], y[::5, :], label='f(x)')
+plt.scatter(x_test, y_test, label='Data points')
+plt.scatter(x_test, np.mean(y_pred, axis=1), label='Pred')
+plt.legend()
+plt.show()
+
+

+

+ +

+ + +
+ + + + + + + +
+ +
+ + + + + + diff --git a/doc/src/Regression/._Regression-bs091.html b/doc/src/Regression/._Regression-bs091.html new file mode 100644 index 000000000..900edcfc0 --- /dev/null +++ b/doc/src/Regression/._Regression-bs091.html @@ -0,0 +1,483 @@ + + + + + + + + +Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis + + + + + + + + + + + + + + + + + + + + + + + +
+
+ + Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis +
+ +
+ +
+
+ + +
+ +

 

 

 

+ + + + +

Understanding what happens

+

+ + +

import matplotlib.pyplot as plt
+import numpy as np
+from sklearn.linear_model import LinearRegression, Ridge, Lasso
+from sklearn.preprocessing import PolynomialFeatures
+from sklearn.model_selection import train_test_split
+from sklearn.pipeline import make_pipeline
+from sklearn.utils import resample
+
+np.random.seed(2018)
+
+n = 40
+n_boostraps = 100
+maxdegree = 14
+
+
+# Make data set.
+x = np.linspace(-3, 3, n).reshape(-1, 1)
+y = np.exp(-x**2) + 1.5 * np.exp(-(x-2)**2)+ np.random.normal(0, 0.1, x.shape)
+error = np.zeros(maxdegree)
+bias = np.zeros(maxdegree)
+variance = np.zeros(maxdegree)
+polydegree = np.zeros(maxdegree)
+x_train, x_test, y_train, y_test = train_test_split(x, y, test_size=0.2)
+
+for degree in range(maxdegree):
+    model = make_pipeline(PolynomialFeatures(degree=degree), LinearRegression(fit_intercept=False))
+    y_pred = np.empty((y_test.shape[0], n_boostraps))
+    for i in range(n_boostraps):
+        x_, y_ = resample(x_train, y_train)
+        y_pred[:, i] = model.fit(x_, y_).predict(x_test).ravel()
+
+    polydegree[degree] = degree
+    error[degree] = np.mean( np.mean((y_test - y_pred)**2, axis=1, keepdims=True) )
+    bias[degree] = np.mean( (y_test - np.mean(y_pred, axis=1, keepdims=True))**2 )
+    variance[degree] = np.mean( np.var(y_pred, axis=1, keepdims=True) )
+    print('Polynomial degree:', degree)
+    print('Error:', error[degree])
+    print('Bias^2:', bias[degree])
+    print('Var:', variance[degree])
+    print('{} >= {} + {} = {}'.format(error[degree], bias[degree], variance[degree], bias[degree]+variance[degree]))
+
+plt.plot(polydegree, np.log10(error), label='Error')
+plt.plot(polydegree, bias, label='bias')
+plt.plot(polydegree, variance, label='Variance')
+plt.legend()
+plt.show()
+
+

+

+ +

+ + +
+ + + + + + + +
+ +
+ + + + + + diff --git a/doc/src/Regression/._Regression-bs092.html b/doc/src/Regression/._Regression-bs092.html new file mode 100644 index 000000000..15286cf43 --- /dev/null +++ b/doc/src/Regression/._Regression-bs092.html @@ -0,0 +1,462 @@ + + + + + + + + +Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis + + + + + + + + + + + + + + + + + + + + + + + +
+
+ + Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis +
+ +
+ +
+
+ + +
+ +

 

 

 

+ + + + +

Summing up

+ +

+The bias-variance tradeoff summarizes the fundamental tension in +machine learning, particularly supervised learning, between the +complexity of a model and the amount of training data needed to train +it. Since data is often limited, in practice it is often useful to +use a less-complex model with higher bias, that is a model whose asymptotic +performance is worse than another model because it is easier to +train and less sensitive to sampling noise arising from having a +finite-sized training dataset (smaller variance). + +

+The above equations tell us that in +order to minimize the expected test error, we need to select a +statistical learning method that simultaneously achieves low variance +and low bias. Note that variance is inherently a nonnegative quantity, +and squared bias is also nonnegative. Hence, we see that the expected +test MSE can never lie below \( Var(\epsilon) \), the irreducible error. + +

+What do we mean by the variance and bias of a statistical learning +method? The variance refers to the amount by which our model would change if we +estimated it using a different training data set. Since the training +data are used to fit the statistical learning method, different +training data sets will result in a different estimate. But ideally the +estimate for our model should not vary too much between training +sets. However, if a method has high variance then small changes in +the training data can result in large changes in the model. In general, more +flexible statistical methods have higher variance. + +

+

+ +

+ + +
+ + + + + + + +
+ +
+ + + + + + diff --git a/doc/src/Regression/._Regression-bs093.html b/doc/src/Regression/._Regression-bs093.html new file mode 100644 index 000000000..96afec3c3 --- /dev/null +++ b/doc/src/Regression/._Regression-bs093.html @@ -0,0 +1,506 @@ + + + + + + + + +Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis + + + + + + + + + + + + + + + + + + + + + + + +
+
+ + Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis +
+ +
+ +
+
+ + +
+ +

 

 

 

+ + + + +

Another Example rom Scikit-Learn's Repository

+

+ + +

"""
+============================
+Underfitting vs. Overfitting
+============================
+
+This example demonstrates the problems of underfitting and overfitting and
+how we can use linear regression with polynomial features to approximate
+nonlinear functions. The plot shows the function that we want to approximate,
+which is a part of the cosine function. In addition, the samples from the
+real function and the approximations of different models are displayed. The
+models have polynomial features of different degrees. We can see that a
+linear function (polynomial with degree 1) is not sufficient to fit the
+training samples. This is called **underfitting**. A polynomial of degree 4
+approximates the true function almost perfectly. However, for higher degrees
+the model will **overfit** the training data, i.e. it learns the noise of the
+training data.
+We evaluate quantitatively **overfitting** / **underfitting** by using
+cross-validation. We calculate the mean squared error (MSE) on the validation
+set, the higher, the less likely the model generalizes correctly from the
+training data.
+"""
+
+print(__doc__)
+
+import numpy as np
+import matplotlib.pyplot as plt
+from sklearn.pipeline import Pipeline
+from sklearn.preprocessing import PolynomialFeatures
+from sklearn.linear_model import LinearRegression
+from sklearn.model_selection import cross_val_score
+
+
+def true_fun(X):
+    return np.cos(1.5 * np.pi * X)
+
+np.random.seed(0)
+
+n_samples = 30
+degrees = [1, 4, 15]
+
+X = np.sort(np.random.rand(n_samples))
+y = true_fun(X) + np.random.randn(n_samples) * 0.1
+
+plt.figure(figsize=(14, 5))
+for i in range(len(degrees)):
+    ax = plt.subplot(1, len(degrees), i + 1)
+    plt.setp(ax, xticks=(), yticks=())
+
+    polynomial_features = PolynomialFeatures(degree=degrees[i],
+                                             include_bias=False)
+    linear_regression = LinearRegression()
+    pipeline = Pipeline([("polynomial_features", polynomial_features),
+                         ("linear_regression", linear_regression)])
+    pipeline.fit(X[:, np.newaxis], y)
+
+    # Evaluate the models using crossvalidation
+    scores = cross_val_score(pipeline, X[:, np.newaxis], y,
+                             scoring="neg_mean_squared_error", cv=10)
+
+    X_test = np.linspace(0, 1, 100)
+    plt.plot(X_test, pipeline.predict(X_test[:, np.newaxis]), label="Model")
+    plt.plot(X_test, true_fun(X_test), label="True function")
+    plt.scatter(X, y, edgecolor='b', s=20, label="Samples")
+    plt.xlabel("x")
+    plt.ylabel("y")
+    plt.xlim((0, 1))
+    plt.ylim((-2, 2))
+    plt.legend(loc="best")
+    plt.title("Degree {}\nMSE = {:.2e}(+/- {:.2e})".format(
+        degrees[i], -scores.mean(), scores.std()))
+plt.show()
+
+

+

+ +

+ + +
+ + + + + + + +
+ +
+ + + + + + diff --git a/doc/src/Regression/._Regression-bs094.html b/doc/src/Regression/._Regression-bs094.html new file mode 100644 index 000000000..f84d240d0 --- /dev/null +++ b/doc/src/Regression/._Regression-bs094.html @@ -0,0 +1,560 @@ + + + + + + + + +Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis + + + + + + + + + + + + + + + + + + + + + + + +
+
+ + Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis +
+ +
+ +
+
+ + +
+ +

 

 

 

+ + + + +

The one-dimensional Ising model

+ +

+Let us bring back the Ising model again, but now with an additional +focus on Ridge and Lasso regression as well. We repeat some of the +basic parts of the Ising model and the setup of the training and test +data. The one-dimensional Ising model with nearest neighbor +interaction, no external field and a constant coupling constant \( J \) is +given by + +$$ +\begin{align} + H = -J \sum_{k}^L s_k s_{k + 1}, +\tag{27} +\end{align} +$$ + +where \( s_i \in \{-1, 1\} \) and \( s_{N + 1} = s_1 \). The number of spins in the system is determined by \( L \). For the one-dimensional system there is no phase transition. + +

+We will look at a system of \( L = 40 \) spins with a coupling constant of \( J = 1 \). To get enough training data we will generate 10000 states with their respective energies. + +

+ + +

import numpy as np
+import matplotlib.pyplot as plt
+from mpl_toolkits.axes_grid1 import make_axes_locatable
+import seaborn as sns
+import scipy.linalg as scl
+from sklearn.model_selection import train_test_split
+import sklearn.linear_model as skl
+import tqdm
+sns.set(color_codes=True)
+cmap_args=dict(vmin=-1., vmax=1., cmap='seismic')
+
+L = 40
+n = int(1e4)
+
+spins = np.random.choice([-1, 1], size=(n, L))
+J = 1.0
+
+energies = np.zeros(n)
+
+for i in range(n):
+    energies[i] = - J * np.dot(spins[i], np.roll(spins[i], 1))
+
+

+A more general form for the one-dimensional Ising model is + +$$ +\begin{align} + H = - \sum_j^L \sum_k^L s_j s_k J_{jk}. +\tag{28} +\end{align} +$$ + +

+Here we allow for interactions beyond the nearest neighbors and a more +adaptive coupling matrix. This latter expression can be formulated as +a matrix-product on the form +$$ +\begin{align} + H = X J, +\tag{29} +\end{align} +$$ + +

+where \( X_{jk} = s_j s_k \) and \( J \) is the matrix consisting of the +elements \( -J_{jk} \). This form of writing the energy fits perfectly +with the form utilized in linear regression, viz. +$$ +\begin{align} + \boldsymbol{y} = \boldsymbol{X}\boldsymbol{\beta} + \boldsymbol{\epsilon}. +\tag{30} +\end{align} +$$ + +We organize the data as we did above +

+ + +

X = np.zeros((n, L ** 2))
+for i in range(n):
+    X[i] = np.outer(spins[i], spins[i]).ravel()
+y = energies
+X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.96)
+
+X_train_own = np.concatenate(
+    (np.ones(len(X_train))[:, np.newaxis], X_train),
+    axis=1
+)
+
+X_test_own = np.concatenate(
+    (np.ones(len(X_test))[:, np.newaxis], X_test),
+    axis=1
+)
+
+

+We will do all fitting with Scikit-Learn, + +

+ + +

clf = skl.LinearRegression().fit(X_train, y_train)
+
+

+When extracting the \( J \)-matrix we make sure to remove the intercept +

+ + +

J_sk = clf.coef_.reshape(L, L)
+
+

+And then we plot the results +

+ + +

fig = plt.figure(figsize=(20, 14))
+im = plt.imshow(J_sk, **cmap_args)
+plt.title("LinearRegression from Scikit-learn", fontsize=18)
+plt.xticks(fontsize=18)
+plt.yticks(fontsize=18)
+cb = fig.colorbar(im)
+cb.ax.set_yticklabels(cb.ax.get_yticklabels(), fontsize=18)
+plt.show()
+
+

+The results perfectly with our previous discussion where we used our own code. + +

+

+ +

+ + +
+ + + + + + + +
+ +
+ + + + + + diff --git a/doc/src/Regression/._Regression-bs095.html b/doc/src/Regression/._Regression-bs095.html new file mode 100644 index 000000000..51156d4a3 --- /dev/null +++ b/doc/src/Regression/._Regression-bs095.html @@ -0,0 +1,460 @@ + + + + + + + + +Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis + + + + + + + + + + + + + + + + + + + + + + + +
+
+ + Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis +
+ +
+ +
+
+ + +
+ +

 

 

 

+ + + + +

Ridge regression

+ +

+Having explored the ordinary least squares we move on to ridge +regression. In ridge regression we include a regularizer. This +involves a new cost function which leads to a new estimate for the +weights \( \boldsymbol{\beta} \). This results in a penalized regression problem. The +cost function is given by + +$$ +\begin{align} + C(\boldsymbol{X}, \boldsymbol{\beta}; \lambda) = (\boldsymbol{X}\boldsymbol{\beta} - \boldsymbol{y})^T(\boldsymbol{X}\boldsymbol{\beta} - \boldsymbol{y}) + \lambda \boldsymbol{\beta}^T\boldsymbol{\beta}. +\tag{31} +\end{align} +$$ + +

+ + +

_lambda = 0.1
+clf_ridge = skl.Ridge(alpha=_lambda).fit(X_train, y_train)
+J_ridge_sk = clf_ridge.coef_.reshape(L, L)
+fig = plt.figure(figsize=(20, 14))
+im = plt.imshow(J_ridge_sk, **cmap_args)
+plt.title("Ridge from Scikit-learn", fontsize=18)
+plt.xticks(fontsize=18)
+plt.yticks(fontsize=18)
+cb = fig.colorbar(im)
+cb.ax.set_yticklabels(cb.ax.get_yticklabels(), fontsize=18)
+
+plt.show()
+
+

+

+ +

+ + +
+ + + + + + + +
+ +
+ + + + + + diff --git a/doc/src/Regression/._Regression-bs096.html b/doc/src/Regression/._Regression-bs096.html new file mode 100644 index 000000000..9b8d9e10e --- /dev/null +++ b/doc/src/Regression/._Regression-bs096.html @@ -0,0 +1,462 @@ + + + + + + + + +Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis + + + + + + + + + + + + + + + + + + + + + + + +
+
+ + Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis +
+ +
+ +
+
+ + +
+ +

 

 

 

+ + + + +

LASSO regression

+ +

+In the Least Absolute Shrinkage and Selection Operator (LASSO)-method we get a third cost function. + +$$ +\begin{align} + C(\boldsymbol{X}, \boldsymbol{\beta}; \lambda) = (\boldsymbol{X}\boldsymbol{\beta} - \boldsymbol{y})^T(\boldsymbol{X}\boldsymbol{\beta} - \boldsymbol{y}) + \lambda \sqrt{\boldsymbol{\beta}^T\boldsymbol{\beta}}. +\tag{32} +\end{align} +$$ + +

+Finding the extremal point of this cost function is not so straight-forward as in least squares and ridge. We will therefore rely solely on the function ``Lasso`` from Scikit-Learn. + +

+ + +

clf_lasso = skl.Lasso(alpha=_lambda).fit(X_train, y_train)
+J_lasso_sk = clf_lasso.coef_.reshape(L, L)
+fig = plt.figure(figsize=(20, 14))
+im = plt.imshow(J_lasso_sk, **cmap_args)
+plt.title("Lasso from Scikit-learn", fontsize=18)
+plt.xticks(fontsize=18)
+plt.yticks(fontsize=18)
+cb = fig.colorbar(im)
+cb.ax.set_yticklabels(cb.ax.get_yticklabels(), fontsize=18)
+
+plt.show()
+
+

+It is quite striking how LASSO breaks the symmetry of the coupling +constant as opposed to ridge and OLS. We get a sparse solution with +\( J_{j, j + 1} = -1 \). + +

+

+ +

+ + +
+ + + + + + + +
+ +
+ + + + + + diff --git a/doc/src/Regression/._Regression-bs097.html b/doc/src/Regression/._Regression-bs097.html new file mode 100644 index 000000000..9399d436b --- /dev/null +++ b/doc/src/Regression/._Regression-bs097.html @@ -0,0 +1,477 @@ + + + + + + + + +Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis + + + + + + + + + + + + + + + + + + + + + + + +
+
+ + Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis +
+ +
+ +
+
+ + +
+ +

 

 

 

+ + + + +

Performance as function of the regularization parameter

+ +

+We see how the different models perform for a different set of values for \( \lambda \). + +

+ + +

lambdas = np.logspace(-4, 5, 10)
+
+train_errors = {
+    "ols_sk": np.zeros(lambdas.size),
+    "ridge_sk": np.zeros(lambdas.size),
+    "lasso_sk": np.zeros(lambdas.size)
+}
+
+test_errors = {
+    "ols_sk": np.zeros(lambdas.size),
+    "ridge_sk": np.zeros(lambdas.size),
+    "lasso_sk": np.zeros(lambdas.size)
+}
+
+plot_counter = 1
+
+fig = plt.figure(figsize=(32, 54))
+
+for i, _lambda in enumerate(tqdm.tqdm(lambdas)):
+    for key, method in zip(
+        ["ols_sk", "ridge_sk", "lasso_sk"],
+        [skl.LinearRegression(), skl.Ridge(alpha=_lambda), skl.Lasso(alpha=_lambda)]
+    ):
+        method = method.fit(X_train, y_train)
+
+        train_errors[key][i] = method.score(X_train, y_train)
+        test_errors[key][i] = method.score(X_test, y_test)
+
+        omega = method.coef_.reshape(L, L)
+
+        plt.subplot(10, 5, plot_counter)
+        plt.imshow(omega, **cmap_args)
+        plt.title(r"%s, $\lambda = %.4f$" % (key, _lambda))
+        plot_counter += 1
+
+plt.show()
+
+

+We see that LASSO reaches a good solution for low +values of \( \lambda \), but will "wither" when we increase \( \lambda \) too +much. Ridge is more stable over a larger range of values for +\( \lambda \), but eventually also fades away. + +

+

+ +

+ + +
+ + + + + + + +
+ +
+ + + + + + diff --git a/doc/src/Regression/._Regression-bs098.html b/doc/src/Regression/._Regression-bs098.html new file mode 100644 index 000000000..205ff2efa --- /dev/null +++ b/doc/src/Regression/._Regression-bs098.html @@ -0,0 +1,474 @@ + + + + + + + + +Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis + + + + + + + + + + + + + + + + + + + + + + + +
+
+ + Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis +
+ +
+ +
+
+ + +
+ +

 

 

 

+ + + + +

Finding the optimal value of \( \lambda \)

+ +

+To determine which value of \( \lambda \) is best we plot the accuracy of +the models when predicting the training and the testing set. We expect +the accuracy of the training set to be quite good, but if the accuracy +of the testing set is much lower this tells us that we might be +subject to an overfit model. The ideal scenario is an accuracy on the +testing set that is close to the accuracy of the training set. + +

+ + +

fig = plt.figure(figsize=(20, 14))
+
+colors = {
+    "ols_sk": "r",
+    "ridge_sk": "y",
+    "lasso_sk": "c"
+}
+
+for key in train_errors:
+    plt.semilogx(
+        lambdas,
+        train_errors[key],
+        colors[key],
+        label="Train {0}".format(key),
+        linewidth=4.0
+    )
+
+for key in test_errors:
+    plt.semilogx(
+        lambdas,
+        test_errors[key],
+        colors[key] + "--",
+        label="Test {0}".format(key),
+        linewidth=4.0
+    )
+plt.legend(loc="best", fontsize=18)
+plt.xlabel(r"$\lambda$", fontsize=18)
+plt.ylabel(r"$R^2$", fontsize=18)
+plt.tick_params(labelsize=18)
+plt.show()
+
+

+From the above figure we can see that LASSO with \( \lambda = 10^{-2} \) +achieves a very good accuracy on the test set. This by far surpasses the +other models for all values of \( \lambda \). + +

+

+ +

+ + +
+ + + + + + + +
+ +
+ + + + + + diff --git a/doc/src/Regression/._Regression-bs099.html b/doc/src/Regression/._Regression-bs099.html new file mode 100644 index 000000000..89e30924b --- /dev/null +++ b/doc/src/Regression/._Regression-bs099.html @@ -0,0 +1,650 @@ + + + + + + + + +Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis + + + + + + + + + + + + + + + + + + + + + + + +
+
+ + Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis +
+ +
+ +
+
+ + +
+ +

 

 

 

+ + + + +

Further Exercises

+ +

Exercise 1

+ +

+We will generate our own dataset for a function \( y(x) \) where \( x \in [0,1] \) and defined by random numbers computed with the uniform distribution. The function \( y \) is a quadratic polynomial in \( x \) with added stochastic noise according to the normal distribution \( \cal {N}(0,1) \). +The following simple Python instructions define our \( x \) and \( y \) values (with 100 data points). +

+ + +

x = np.random.rand(100,1)
+y = 5*x*x+0.1*np.random.randn(100,1)
+
+
    +
  1. Write your own code (following the examples above) for computing the parametrization of the data set fitting a second-order polynomial.
    2. +
  2. Use thereafter scikit-learn (see again the examples in the regression slides) and compare with your own code.
    3. +
  3. Using scikit-learn, compute also the mean square error, a risk metric corresponding to the expected value of the squared (quadratic) error defined as
    4. +
+ +$$ MSE(\hat{y},\hat{\tilde{y}}) = \frac{1}{n} +\sum_{i=0}^{n-1}(y_i-\tilde{y}_i)^2, +$$ + +and the \( R^2 \) score function. +If \( \tilde{\hat{y}}_i \) is the predicted value of the \( i-th \) sample and \( y_i \) is the corresponding true value, then the score \( R^2 \) is defined as +$$ +R^2(\hat{y}, \tilde{\hat{y}}) = 1 - \frac{\sum_{i=0}^{n - 1} (y_i - \tilde{y}_i)^2}{\sum_{i=0}^{n - 1} (y_i - \bar{y})^2}, +$$ + +where we have defined the mean value of \( \hat{y} \) as +$$ +\bar{y} = \frac{1}{n} \sum_{i=0}^{n - 1} y_i. +$$ + +

+You can use the functionality included in scikit-learn. If you feel +for it, you can use your own program and define functions which +compute the above two functions. Discuss the meaning of these +results. Try also to vary the coefficient in front of the added +stochastic noise term and discuss the quality of the fits. + +

Exercise 2, variance of the parameters \( \beta \) in linear regression

+ +

+Show that the variance of the parameters \( \beta \) in the linear regression method (chapter 3, equation (3.8) of Trevor Hastie, Robert Tibshirani, Jerome H. Friedman, The Elements of Statistical Learning, Springer) is given as + +$$ +\mathrm{Var}(\hat{\beta}) = \left(\hat{X}^T\hat{X}\right)^{-1}\sigma^2, +$$ + +with +$$ +\sigma^2 = \frac{1}{N-p-1}\sum_{i=1}^{N} (y_i-\tilde{y}_i)^2, +$$ + +where we have assumed that we fit a function of degree \( p-1 \) (for example a polynomial in \( x \)). + +

Exercise 3

+ +

+This exercise is a continuation of exercise 1. We will +use the same function to generate our data set, still staying with a +simple function \( y(x) \) which we want to fit using linear regression, +but now extending the analysis to include the Ridge and the Lasso +regression methods. You can use the code under the Regression as an example on how to use the Ridge and the Lasso methods. + +

+We will thus again generate our own dataset for a function \( y(x) \) where +\( x \in [0,1] \) and defined by random numbers computed with the uniform +distribution. The function \( y \) is a quadratic polynomial in \( x \) with +added stochastic noise according to the normal distribution \( \cal{N}(0,1) \). + +

+The following simple Python instructions define our \( x \) and \( y \) values (with 100 data points). +

+ + +

x = np.random.rand(100,1)
+y = 5*x*x+0.1*np.random.randn(100,1)
+
+
    +
  1. Write your own code for the Ridge method and compute the parametrization for different values of \( \lambda \). Compare and analyze your results with those from exercise 1. Study the dependence on \( \lambda \) while also varying the strength of the noise in your expression for \( y(x) \).
    2. +
  2. Repeat the above but using the functionality of scikit-learn. Compare your code with the results from scikit-learn. Remember to run with the same random numbers for generating \( x \) and \( y \).
    3. +
  3. Our next step is to study the variance of the parameters \( \beta_1 \) and \( \beta_2 \) (assuming that we are parametrizing our function with a second-order polynomial. We will use standard linear regression and the Ridge regression. You can now opt for either writing your own function that calculates the variance of these paramaters (recall that this is equal to the diagonal elements of the matrix \( (\hat{X}^T\hat{X})+\lambda\hat{I})^{-1} \)) or use the functionality of scikit-learn and compute their variances. Discuss the results of these variances as functions
    4. +
  4. Repeat the previous step but add now the Lasso method. Discuss your results and compare with standard regression and the Ridge regression results.
    5. +
  5. Try to implement the cross-validation as well.
    6. +
  6. Finally, using scikit-learn or your own code, compute also the mean square error, a risk metric corresponding to the expected value of the squared (quadratic) error defined as
    7. +
+ +$$ MSE(\hat{y},\hat{\tilde{y}}) = \frac{1}{n} +\sum_{i=0}^{n-1}(y_i-\tilde{y}_i)^2, +$$ + +and the \( R^2 \) score function. +If \( \tilde{\hat{y}}_i \) is the predicted value of the \( i-th \) sample and \( y_i \) is the corresponding true value, then the score \( R^2 \) is defined as +$$ +R^2(\hat{y}, \tilde{\hat{y}}) = 1 - \frac{\sum_{i=0}^{n - 1} (y_i - \tilde{y}_i)^2}{\sum_{i=0}^{n - 1} (y_i - \bar{y})^2}, +$$ + +where we have defined the mean value of \( \hat{y} \) as +$$ +\bar{y} = \frac{1}{n} \sum_{i=0}^{n - 1} y_i. +$$ + +Discuss these quantities as functions of the variable \( \lambda \) in the Ridge and Lasso regression methods. + +

Exercise 4

+ +

+We will study how +to fit polynomials to a specific two-dimensional function called +Franke's +function. This +is a function which has been widely used when testing various interpolation and fitting +algorithms. Furthermore, after having established the model and the +method, we will employ resamling techniques such as the cross-validation and/or +the bootstrap methods, in order to perform a proper assessment of our models. + +

+The Franke function, which is a weighted sum of four exponentials reads as follows +$$ +\begin{align*} +f(x,y) &= \frac{3}{4}\exp{\left(-\frac{(9x-2)^2}{4} - \frac{(9y-2)^2}{4}\right)}+\frac{3}{4}\exp{\left(-\frac{(9x+1)^2}{49}- \frac{(9y+1)}{10}\right)} \\ +&+\frac{1}{2}\exp{\left(-\frac{(9x-7)^2}{4} - \frac{(9y-3)^2}{4}\right)} -\frac{1}{5}\exp{\left(-(9x-4)^2 - (9y-7)^2\right) }. +\end{align*} +$$ + +

+The function will be defined for \( x,y\in [0,1] \). Our first step will +be to perform an OLS regression analysis of this function, trying out +a polynomial fit with an \( x \) and \( y \) dependence of the form \( [x, y, +x^2, y^2, xy, \dots] \). We will also include cross-validation and +bootstrap as resampling techniques. As in homeworks 1 and 2, we +can use a uniform distribution to set up the arrays of values for \( x \) +and \( y \), or as in the example below just a fix values for \( x \) and \( y \) with a given step size. +In this case we will have two predictors and need to fit a +function (for example a polynomial) of \( x \) and \( y \). Thereafter we will +repeat much of the same procedure using the the Ridge and +Lasso regression methods, introducing thus a dependence on the bias +(penalty) \( \lambda \). + +

+The Python fucntion for the Franke function is included here (it performs also a three-dimensional plot of it) +

+ + +

from mpl_toolkits.mplot3d import Axes3D
+import matplotlib.pyplot as plt
+from matplotlib import cm
+from matplotlib.ticker import LinearLocator, FormatStrFormatter
+import numpy as np
+from random import random, seed
+
+fig = plt.figure()
+ax = fig.gca(projection='3d')
+
+# Make data.
+x = np.arange(0, 1, 0.05)
+y = np.arange(0, 1, 0.05)
+x, y = np.meshgrid(x,y)
+
+
+def FrankeFunction(x,y):
+    term1 = 0.75*np.exp(-(0.25*(9*x-2)**2) - 0.25*((9*y-2)**2))
+    term2 = 0.75*np.exp(-((9*x+1)**2)/49.0 - 0.1*(9*y+1))
+    term3 = 0.5*np.exp(-(9*x-7)**2/4.0 - 0.25*((9*y-3)**2))
+    term4 = -0.2*np.exp(-(9*x-4)**2 - (9*y-7)**2)
+    return term1 + term2 + term3 + term4
+
+
+z = FrankeFunction(x, y)
+
+# Plot the surface.
+surf = ax.plot_surface(x, y, z, cmap=cm.coolwarm,
+                       linewidth=0, antialiased=False)
+
+# Customize the z axis.
+ax.set_zlim(-0.10, 1.40)
+ax.zaxis.set_major_locator(LinearLocator(10))
+ax.zaxis.set_major_formatter(FormatStrFormatter('%.02f'))
+
+# Add a color bar which maps values to colors.
+fig.colorbar(surf, shrink=0.5, aspect=5)
+
+plt.show()
+
+

+We will thus again generate our own dataset for a function \( \mathrm{FrankeFunction}(x,y) \) where +\( x,y \in [0,1] \) could be defined by random numbers computed with the uniform +distribution. The function \( f(x,y) \) is the Franke function. You should explore also the addition +an added stochastic noise to this function using the normal distribution \( \cal{N}(0,1) \). + +

+Write your own code (using either a matrix inversion or a singular value decomposition from e.g., numpy ) or use your code from exercises 1 and 3 +and perform a standard least square regression analysis using polynomials in \( x \) and \( y \) up to fifth order. Find the confidence intervals of the parameters \( \beta \) by computing their variances, evaluate the Mean Squared error (MSE) +$$ MSE(\hat{y},\hat{\tilde{y}}) = \frac{1}{n} +\sum_{i=0}^{n-1}(y_i-\tilde{y}_i)^2, +$$ + +and the \( R^2 \) score function. +If \( \tilde{\hat{y}}_i \) is the predicted value of the \( i-th \) sample and \( y_i \) is the corresponding true value, then the score \( R^2 \) is defined as +$$ +R^2(\hat{y}, \tilde{\hat{y}}) = 1 - \frac{\sum_{i=0}^{n - 1} (y_i - \tilde{y}_i)^2}{\sum_{i=0}^{n - 1} (y_i - \bar{y})^2}, +$$ + +where we have defined the mean value of \( \hat{y} \) as +$$ +\bar{y} = \frac{1}{n} \sum_{i=0}^{n - 1} y_i. +$$ + +

+Perform a resampling of the data where you split the data in training data and test data. Implement the \( k \)-fold cross-validation algorithm and/or the bootstrap algorithm +and evaluate again the MSE and the \( R^2 \) functions resulting from the test data. Evaluate also the bias and variance of the final models. + +

+Write then your own code for the Ridge method, either using matrix +inversion or the singular value decomposition as done for standard OLS. Perform the same analysis as in the +previous exercise (for the same polynomials and include resampling +techniques) but now for different values of \( \lambda \). Compare and +analyze your results with those obtained with standard OLS. Study the +dependence on \( \lambda \) while also varying eventually the strength of +the noise in your expression for \( \mathrm{FrankeFunction}(x,y) \). + +

+Then perform the same studies but now with Lasso regression. Use the functionalities of +scikit-learn. Give a critical discussion of the three methods and a +judgement of which model fits the data best. + +

+ +

+ + +
+ + + + + + + +
+ +
+ + + + + + diff --git a/doc/src/Regression/README.txt b/doc/src/Regression/README.txt new file mode 100644 index 000000000..f3bf58066 --- /dev/null +++ b/doc/src/Regression/README.txt @@ -0,0 +1,2 @@ +This IPython notebook Regression.ipynb does not require any additional +programs. diff --git a/doc/src/Regression/Regression-bs.html b/doc/src/Regression/Regression-bs.html new file mode 100644 index 000000000..aef0fd910 --- /dev/null +++ b/doc/src/Regression/Regression-bs.html @@ -0,0 +1,452 @@ + + + + + + + + +Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis + + + + + + + + + + + + + + + + + + + + + + + +
+
+ + Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis +
+ +
+ +
+
+ + +
+ +

 

 

 

+ + + + + + +
+

Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis

+ +

+ + +

+Morten Hjorth-Jensen [1, 2] +
+ +

+ + +

[1] Department of Physics, University of Oslo
+
[2] Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University
+
+

+

Jul 22, 2019

+
+

+ + +

Read »

+ + +
+ +

+ +

+ + +
+ + + + + + + +
+ © 1999-2019, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license +
+ + + + + + diff --git a/doc/src/Regression/Regression-minted.pdf b/doc/src/Regression/Regression-minted.pdf new file mode 100644 index 0000000000000000000000000000000000000000..33108d9e96248362f104e589782c244ff925c76f GIT binary patch literal 459187 zcma&NQKxyrWf{?8fRSGzBEZj#KAj9g{B^OW<|uz#lcSWe;W*=gteWUxeF1a zgq^XQxtO`BqnSC3fB=lEn~S-zJ&f1CRxMfAtx@FOOU-^t)0!yelVD936*O6DbZUSZ z504f%T<|`F2%Qk9*1~lC+}cv8zmn~w0`$OO z8XuJ*AOEw>@Ynpg{<2-!B#hyaN;!O9J;@u|fS2Xfx1C^{tyT0_?8XBtv_F%q@4aP` z-NSaxt5ckOQ4gXMe9E83ww@Wm?*vGzzC=c?b8^C~T+D%J4sEb3=U5;9@JdRf>M-yN zmoWFJ_L4G*2lXXRM1<=!=k0S4S%o`GlO;<`eNqlora=Er^T8aPY@+t%k~3CJ*hZ_~ z6l1fobkEBRiqK~Ipa}R>KcKIU?A=&lHJaqDC%AMVV)naQ+zD!oi(Hr;dh4KReha~_ z|Ce*K(PFy^S=^kzt(}<6b6WJujb`o)Ck~wn1RtGIB<(S?iN8_g{uj-bvkz**+ zU@oP79;b}JN>kW{_K40+xq@8@vm}LuikW%a(ZiTp4r_qW$&|k^5~b7oO>2f^Ui3nU zNZvRKIpd}*eE}X1vLbCyRG-VE#a)L3maIVg-Bc36zkkdv4w1>60u}$lxaR-BUxgwD zC5mDn2tAP_?DE+{)QGLfGVJ}NpjTB+nna6g#B`67$lnyn$UA$9dTWl((yELKu(Lnh zwf}S9mKjY59#YfrB@ia`;}#z>?icWhmx%~{n52N-R_{; zcOHND-m*GUxKmJr#li4gf2k8D5XIo-Cq|`&mx7Tr0_zu2A-e0Zn2y>_;l`g zZJmsFmBsLB<0t!GTUD-Dx>%wS%`H;f<<2ow(?+R{x?W$L=hDx^cog60M!~`GHdkI9 z?~A|eM7Std>0pj39mf4W?Ko%$!>>)YE5BUCQhqz;qjH8ZD~5MSBm0=A>@5;X=ywD)V32q1^Y2O+}( zY-7ZbId5{+Rm#AX*6t*%a9jL}5)^0B`n*OCZieO=dq(Ol7cY#DDW3u7zqv8^2D}|-#yn`JbG#{m0ynmGq?(1+ zb`oeVb1lL4s!dovSwJsHdKE3~NIL}bED*@K7fjn|a!a2I3DZ71#N{Hd(?4iU3V1G+ zA(B*_Q-FNFmN?NLdnYxDK{EMojdI}h`t$hX+~CGODpZ(z(u~TL^L(f`q77Owv(%&eSDT>o2PUeS?FJ#0hn{nR`g zubS>3#vRWLrc2Y|GQ)$e&_6iKmAgGW|YP!nW>q!cO`D%yp8R;@bMrF)${bT~RVo zCH0WMdr(jYPp8-TQePYG5+~J(pFCDcNZiKG(pd}C(R0FGeCVrGw7~vT*t)ww+bI}= z4EAwsVzGSGt;)kT4h?-|lQA=(HIn*I(_#F5IN544ia_fY~VQVq*N%aN=UA>RT2Qw*zJCR8$xNMP==u^0E=B&O#Z1bcPDT| z@vUhc0Sbp$jk?qN&VeQZX*s}Z8w*_p@g2+Wp$TKO-CUv)p_ z+vlW*h|4 zU7<5XQc&SoAin6bz2!_TMaX&Dgz64(JS}RG6$lyw_LD006nHptIZ2Vxc@|O~@R}Y( z5G`KMaUTmR%RDbWub;tT8n8|7EwGc!BLzElq<|U6&5o?Q*Ng99_e~duWr@Pt%$86Y z85`HUJuGh&u1cpp!GM~W$`)E80l^l`Aqw3%g=_CtL+3c8YF?TQnJEYgm`nQUA;`GH z(#bN4v^UGbDf$6cwM;%soU+&=l6a8EHsVZimsG#$1nvb5RR>p1DzxiA9&-_)E!wHH zXm)e(R%X3Zux|@}cG<3+yNLy~Q@cfUWLP01~~^x*B7*uCut~r%(S68g`a!s z9h#89H=uaaNCtjUL5T{sMs3WjInbcRAwoibfLZRKJSz+0rC-nsz0l^Fo7H5297Yi~ zs}qH(v%%xg&xQ@;C1x9x?;knODCIG1RiIbdXCE0@5rrfzPPphtGFBT}u_aLK{3F!O ze5TLHfk4(9;ZBZNH=51p+dW=_OTy?$Y(b4yceXZjlL=Ko%8w;XkdXK+>U$Ee5f1p6 zoGPs{L|6D%K;LwN)Cj(Wm+OT|`9YYXfEpMj93X74@3L9ZUx}d~V3QSrX&_>yy?iG6 zt%T#GAJZ(bPn|%$qq1l4YELsvX^jJt5v(1!;+&&A8W^!d|Ji#P03=UGjl}8jUme0R zj(CbtCXl+sLRmBO`l2C@@N783=ogUCkg)TY-bg}?BO2I!z4y?dWYoYlVSM8EwWQdO zt2eehqW_?a)smH+5y8M0$m%eL^!i5`L&8go zu~>-T_7|fH5I({bkl=^Hx=PHl7JXR#$qRvdro0xu>nXP z%x^HhNKk2iJSOQf(5AY@9H=2wVbY{3O!cZVwfBKW$d)|qt6W{>x8I}2hk(ue07a$iDLb1r^b65G(z8|=S@#dYkkqi&6cz{IbIY>v z)5-mw(a<%(h%j_-l7f%QCMT4x!?$FVTAd@dnFS}zZb$^8d#tx%HY}(aWhFnkk#ur) z+3irjw<|3HWfk~Wbg7N!RpIFFY9Zpq%5-!2#z-z`AYVa0Hsu{}66o8lODj7x(ymq( z6QDarT>=(Fn7X9}BU%k)G3I%M5aib6+T-Zb3qqKLs%*k@jeS=OjL#imymt%()0YDyR#7jsyvnWOPzQY4t0M>QqURBYyaNGtz1T=q z^BemqChl0Q;DdcOSdQ-po}5kh%CAu(O+cWNV?Jv~(A+HWw$F=`c9Qv_^>ivr)^0e$ zx(Z-OH!nTt)T1ilc#TZR%yb?F6TejY#3LZq|g)}+79sc2M|v_MYV;qOAFKbtEjazlW|82frA z;F*fjQI19L6e;?Rx$ai{A8lQc;0*em$_mT4nk%$L=@{;h`t~@y>~}*75|(-|m%@X^ zMzVEBj_mWY)T6rYhr#M%0{pdss0M-RK}BKCH@FI5F1OtH>}d^NyPJ>iq#K~C%z zc_(20VoW*_z|6;>_mMYs-~+`uv68cv*oF7ti3puN;(hqy@es@h!$gEa^N)niH$o#1 zF)Di}ZgfLzGEWOU(>CJYArM>gfU&O^r*!dxxiwaHF5}EY=1(_XsAHM^P;cya+RiD? z0nfhmks}@Jl33`*R71f2WueT_HP>WGe^O?u9T0| zOUP(aFi?WsG&t7n4Ss9JWJWzt`T)U-A~XIUt>)&Dx& zIA!GqnoSFahr;l*`_4XxR+2acy=Rn+{1m)KKD#LpDLI{q7e}0tM4ZW3z%Ch5nXat6 zA}o6fBD64Nz|A6Xj5mc73wcj+xn5L|2FB4-)k)@KWR0~sSP9}aOh#uO-u4Y1A>)48 zY>N!dv>C5oqz&Mr|sOGKzR(~2CHa*>n1cCB6B6((~jF8h~Ri?|H+aX^M=I+1yaJZKIo8LY;JO+JF~k=zELUQ&Dyi zang8Ofw!PT#sk-qW@wkC0>=Qz=(FWkYzVtH?LGa7@s5jbVXmYpQIHzW~ zorJ-s+twkl6D4fXo5N;Pb6TsUZGAjQBOexRlAt~R;^afGtNUP1CS)pu0J&3BA62FY z!ml9@#!Jb2{0_>0<7*wCZ^h^Ek& z%r5M#Phq%01UQZRJ&QrQ4Dg0g;0p}KwF2Juu^JpOFW79v8c#{to<%#5@f~A^M4#S% zYD2sp*e;n1FkWoOw)Rp`0~Bb_O2hCnW8J2E7_;Dz`%5KXa*!PwKqc$wnw((J%4CSI z(;IJ`x}BNj_YDmB?HHMe0`IHYZV((q z9?ZiP6ULXPA+{y?oTwKT##sSfp>Nb^y!YN~Pn)j0YsUGB_L;R~O5oJ>_^^ZUvEu0rTjuy1VU~$D9`~!YfIpmrfBV>f>liiN_4|}mO+C5^N50&--(c4?UKqz-=G8vCskn&Vl4K)M+OcJ~gn`tN zDybyAFobV=-=DX_a8~PV6^q^29g<~@%3{3R3a2f&k0$E zQ*6=eOP{MrUtYbD%q0Boc$tTG2B+O<`iOzFsT3`9BA5^qpFlS2EC&YXJ8Hj9DEQ>g zz7$|j9kskd7V9v6Fv?y6N$~Ckk!RUk&lh*-^Q}|m;LtMNXw~qY)mhX??-*z zx|)?bYd~hNspo&9S99@E(4eEf8SY^Ypt<#6L}U!?`k2ym%)*YvIyy{2m5`^~;|v&N z&hJ1W%cW^$xMg6d#dz*|Qt?VAnqHL_QIsNS$YCL&hL)g7v30ltlMl0iRX`j0VuxsC zq9TMpu-Zu94qA&=FA6dK1}aeEBJEr#)m?`9Lq-;Sr@&T|wK7E)>tm1m5f;K0Ni9@) zPa-Thd-CL)YpyK&6;(KTdBMcfm`jiZ2$OsN{-=JQID(@w1@S(gT94WWb|J{CqEpepUEZVD^t!HU*gGKC%=mXTYBOq=~`IKdejKC zEDAQMw%y1;#Eu?T<5s)Vr^jYXIb`rF0if}O4S@@6fYB0oo8+z#7=fq+>3-RVprmS% z>@f8jgrQyCn%=Pb*nR0kLi_LE<@O)zC@v*cmd3Au)6aHu<;ZMeh~ky2A)L#m8ixy} zrhbOkUj1rm`soBp0<;SFCB94t3!>Ivgj2Q#F9 z@MF$|5v~$q96)n|s+5JqQ3$#zhzy2US~Ph)Q^F7-K7kDRBT}e?oH&t6w}r6*!+;Xn zwcd^8{rukz4u*@msrW#G4)A}d4#tw0mx~58Rs)>IvYb7gn~pK7uYY6p7ig2Rqu*uT z5|s>s>)?ogzU^C^avR0=XW*EwTJFO}dcHOdzg?X@+p9E2 z>TL4L<27)(6xQ_WVrDAbl260Y8A(kRcIZA%5O;gDzRn`*uDl`YT0#Pl#;h%RK0#;V z$XHPMvpX4Ha5+`qL7i(a+crK{)1SXeTzUrL~yCoubv1B;%g$sPz~b?K}4vao7##9FfP? z@w9i<_rQmvc1yeL;iR*Ba$yRs`U~SMfC!TTNC!xFe(mH?gI}=+=x%mD|HziRE? zE>nAUpx4eOjN~Kxnm`pVlsO0jv2!Hcfv|9MwS!y-nccnsSe_gwP4rhddk~D10`ZJ; ztC2wnOcVo|j96nLW$o&|lj~~iwKqu_IGDi3pOd&CYom+36XSbUk*igwP~AaR$1c|~ z2*)o5tgyq}sAln6PO&H-!Ik2%*6VS<^|do=0bKOPn(pTf2-mf|u&^rI(Oxu8a7o%0 zP)bk?H(0&&_5RD5CSicVoC47m^>bldU!BXH_m|$s(yr}*@4ejcz-Exi&jcVgEBM8K z#<{r<`nqB8q2InM)q(e;tD9Izt+bLSCk7b$ zj#;~jZTlPF8Lt_a{mMGAIt*_S!S+~Zff_Tbu1PFr&kk1s zvEH;^jflJ_4R{|2vtE8ssUL;-Ve1Sq{y=msCjI|4PZ0!nDsIoem z-Mkb2OpZ6I(Eh^bcxkGjKGdesv9cGTZD&(Wia_u!qqNm zcG(U)Rz}=QVB>Nnx<@TyT4KcY1V&p;Iz=&@w>3IKB!fThbXvWsGUJY!efx+^ZW+2D zmQIE$6)4EGU$(ete*%@;0t=2IjT(8b(U@A3m{?p%q6I0{D)~fb95*>~IF^6%DsfPW zB5E}iaQPlQA9du(F-shWK!`f;>uF0!z@w|J)hQJ>Yxo*YsCw1lQ`AQUtY8GXyG)4m=!H!$EmP1q|^=HN<`~&74>dnfTdch zBGQg3Tx~J`&4^T5>tL8D7t!=4HFo`zi!Ox)N8U<649C#o9vVmz1nEL1UU5D-`O5jI zAQSGGzMA5het>4cWc< zvF8@#)&)^YE3hVJMpp3T&RqV}gBgkcbLnl@zpp3r2#d0G+{^vfeG0ZZ+z@Pdj9j6& z=8nIV^Z~z@;iTE!ehI)6w`-RJOIhse2EA`}Y|o4)&~oPb)epI_AKPZ-MY(e|bqAv6 zL=n{4Ah>Wrd|kRcOvosHViKRoe5g~1fIfMD-4R1@8Jlw!1VTzVnmhZ^YC2i8pgF|! z``pk&e*m0^W{SYy`^9{OIA@dcwOp> zlWTtp>Asm8LdNjA5kal&X^qmL|AH1h#h3r>pp`%~!I(Nb{P$E(QcnFG3VWc*K3O8Wj?f%x*+qqYLq$|2t*HFzXP z0$qsn7jHQ*NfMhXdIj|G`F8mfwnXSp);b*xVed?(f?& zmis%d-%sxBFxCDUt9fT$FiSR{7L=pzWic9NPhF7dFz%*zxlv<^f}(>M5s>#QLQ10G zUNFN+?G6~jd3hxXQ8|lQOa;8t-F`HO(ejCc=XWFfX#33%7cD54vj^sR6;a$^dYWC$ zO%tLJLbN?^g-Y;eXW*a62Y=mg8rJK+V{>mc|B7tJJXfVpmmbF=oCIL)~-!L#tENNaRvNO9r@=Y4kT~qr}7v?E%UO_S}c~|q)aFQ#Xn{87>#5X30?OG@3G!(Z zm7aaQ<$kovGr&U!q2T-{Stp?9sTrB6hV`>hqtgM<`Kc0oIsZD$`bm%-=>MF&)hg?{ zK;i(Q3Xn9^Z1fkH@Mp4F)|Ginhts}Dmbw1`Nb^Ss@7>-=X9o!H%@}C^>zos&V!(dR z4kKUj{!h4YjOza!{;TkXTnHOD%>6bb0Q)oV{iUoJA;^lr{@*sNFJr^I)5H77Z^3=U z&k(`i(Erx3{xq=t5dEs)z$=Im$r#T0LE`h|Yd{p4sj7%`EhSRF@o(<(`AY*Ok_Rm{ z8K(*VH*KYQ-8Hl{qVcWHQwh|QdH=+p%cOfgY+9z2JfBb?Xs#N&`oiXr#U7MA?s){i zp7YQ*l>B98u9Ah3Co8>S>?}ts4OGCR_&UEB2?RpitatyPM8nShKZ%Bg`G4=u{U4&) zFBe)pJg;E9i?e05&JOD%AAFq9hjkcDfUYA$JTpO zO~17xPi|nb8$k0tMVJjEt+3;J7=NY5N zBhR(I)%TqZI6w0gHN(>#)eM?BWu%fUO_T#%f=$z4tLGMq4=1ezW!cj!G{P!15bAqc zi-?lzZr(Ymj79wK$>=lEFjn*hs&V+OII0U6@}xznLOrEwg_}wVd7dy?^0hEYBH17U zieYEu10p}4!RM$k8!rw#TQO0|K1MIDgvAwCY)^>gZ&4ZrnkJYLHbq9%A!+`v9QE=X ztS}Z&@vU1<13}~>O>Tluk3x_;43YEb2oNm>CN&+2hgPB;r{szeC8{(RjeT4A;kdP| zvFRV_lD8$~P6?{gFq$7TH3vp_c$VjcemwY&5c%5np^7b#T${voG+kp=yI~3)OFVZg z6Cpc@KN4N|{nVLWi_pAm$q~2H>7>TY`YOZ%ka^U_5cj}^p)=QEH?eGzpb|<-Vc5N# zb~lln&pQnh(P>xA9AmWI8ubyYVgZz~y0(Y+`sSb2gyEw9kL*hTCw zz}-9f&XYo9)sxNySR65~g_A^2k#aKd@#DWbl>eqN>|>npHA0?Pf*i3i!O(OQO`pn; z?HbM*dO(cyxRvD+hP}WmnJ-WA85|tyKt&2~i3r-z6WtVNw7zO2o56v5SZpI!0bpX` z18-AyM~&YL-iis+#u&bh=*^?Xxh(zQTjs$x#V@Gb=}~#xL_TBVRbodR(Sl}2d)g#p z1+*@>CwO=svjdsjrz#fm+a8%J$0U%eb;GDlxfIPp-YFgi<7fE$%Z68JwPZBTZRHS(pwJTUY#lG^F;;edt%;SS>8`XAGZ56ulJre{=!N7?nm$!SAvt1 z*yIz)6^L39^9BUfJJdU)LawpH@}wrjIX>KVSR)+Knz zG*E=_^?Bevbr)WphdgD~1?0;_QuB~|_dkSNH+#)C!vvQJRiE%Az`{vQP%z@X+DBJHbBcnozR>;M zblo)r`cXpF8S+TBLt(xjYaf9Y#nm}_i|OzE&(Qu=i*Kfnj9^T{9#4`{RQU~`1{kAR zSqCi{f{rQ$!NBQ~|E=f{Mlann;+dqToQCk@Pb!5c7ROrps?G|+M@lIw6)sU1lf{nZ ztKTdZBeCXjX7b9agI{BV$K05#BLxygFfi$riWyIiUS|GH_^edIy| z?b?3S*}UNSRqm1L9LDR?N5k`4@wgey8~a+TIa1cgI^RzAW#m-Yn5Vi%y^abqlwFwy zmg%1d9S6kz^o-LP-^aB&$Y~4X{xU>C?WvS~c zSeUu)x8r%T?)@ga5^)(166O9IZ1qLRI9O#I2s(V{1Uek$i3{TQChu1g9Z#GfiY2GRtMqad9)e|r zAi{()kG8_%L?mZb;Lq{J4B{2s1!JsW zj170b@@VRf24K*U+FYM=eoY=8Ja78hWa*8@`?Byx86$lbSCJ<3qiZ6vZLRsdukHu9 zX@*H0JGJgpr5^4*@>NTLB@LP+uB<|=xI~1_^*Nw`CMQdQS{|ooUZQQX4>R4@Be#Na zgIFo!??WARfG0l_#|5xzG*~fFg88qk7!*?KQZYo_6LBWC0=Eo!_B58TUSklp6CpDx zRjT&fRep^yXtc_3aUjf#N*x9^8ig>*oZgcd0^CY6Tc;>$l+3=>G;Gaaax*knY|66w z15Q{yNQbgHjWZbwL=$m_x2DVz6eBCaVZ^45B+F1Dvo8oE%Dy<)Ac@h72hWg5S7bvX zn>=W>uT#MYuSFCy-pf^ud8y6bH_B_lo$0ZDaGXv5({uz*UwNp(SU$8^)Yav%8@NHK zQcS0_%J4At$KWX&nLsJ=y(TMJ0DHu+B=vMDIJc3s2TJpI24DOdux!(#Zc4n;bB(qN z(M0on3gW1gSPs6hJ7Rj1X{%VJ(V4*Ut3iqVUn1Z)xJbWQ_8Q3q*VIYFTm5%}v zPDsMhFxd5^q5@7VoH-c`pHwPui?Z1`Df4|C6`U4uBe%I!S?_7LPDjUQ-I>Fs{HeiY zm%Tok!7kV2;w^g@wem!dBMZ3;?)omf_^IK8`3~gEbA@x<6B_03&XK;dW`R2!-B=vS z0Gd>+vGBdfj%3LL+$f1>h|3ER73QY?pd3vDiA5unplBMxbZ;z^D%rbmGeiYscL}=flUr+xNCKF6po7-R#oR*xu0pEHv1_BYHO|{UHvHEp&bVOPexWPn zg@)ac=De!#CNAB^v;Y7jZy7UnRLUG?zsT0g7|;fh=Xpo4P>6|s!i5qO(d3|hw+@AE zR|7R%C){i-XOU^SjbXl_MQM>Zg?65D z3Jmq8mvhjd@9;V`-}-8nY*($QZhzF_dTSIi+3g{&8I*d7!8V&E^TSccHb<-3P;stp z;2BU76|f6C?5_FuVX+$Wwpzj@tIHs}_N5tpiv%SJSSlVbV?+~oGE}KE%1TOzo*Q97 zP_a|$!KHLl(7R$lc_hvPN_Jfn)f$vy^Q078El`QLQ;x+*n zjr_(5I8Nz2ywt08k$((2EO=#Q;1Fza?M?59R}!5GwpcOIzHuFS2oGr3xoA{W33Xty zLapQH4{`^Kq>xMZO>b=NEbv!6pU^OW5I;e}01ia+`H{Z}PAdK|3eum1YJbf0g$#uZ z{4w)PMI?Md3LahwH=m}nqc|JNKjR3@=cP@d7*mxxDL{{x+URxP-|QR^7vF7j6K6{9o)E)A8fAupri9kF5YM4WqSaRlu*F>hZ&~g z0+tq2T9n5BdU0QslelXwHChmI4LC9W1A#Rca}72~9f+egixx!W4Wm)uZts`Aa4fEm zWJvJ0uL-a@rOU^l6ug^Jxn{ZBOaabsjWL2TXm=~(rLl6wzOwmNTX|7WkR9R=lZ{k$ zg`d&m^X1AL4ll={XOZKg?s>8pKhHK%zi-W0bp5QCadsQrA6-?r@u5_8?egg1s7=S$ zR|0sZ!z=D_L&7b`&*V8F1~nYsCwt@I>Y6eP(`|HfXnQV22X|8ziwE2_QYDG#$L>C%Q_-Am@=;bBAM#2# zXX!zN(i_JTyRO~u^peP}!ouzEwa;2PYRMSOeIQ`3N+JT@xZbUDe$HR^GjU@UX`NSx zXa7>%%u*n*hCn*VP^J*D%Y4&zl{w;EIHvp`ng zqswy0xolI?hF|AHxfAtmD*Nuby0vs1qdTv4{@@b9l;zg6sUNB3 z^mq#;Z$3fD6mUbIeWy^JhV0;{< z)l#Ct6WemB3ysGCKJRUo=FuF4070Y%o$xsg?517mLq317(u0s)@QUyBx%5Kksmv)uXKF`uvhRlnPjWysC?$uiUSDrEqW=%ox$5;@ z(#}~nPg>)a`_|`x*I-%9#&pl~KjWR-kx5Ev9m#9V38R@qq=@xCc!`_$!gWVeYL$(m zmP{!*_ogoHB0dhiYE>0cfK%l8XtN8XC}wV@jo*;Rwizy|7|9>t*h&L*oGePT0dXPr%ybL zZ$-g)&6eNBdR_J`$CG*#oP35(iBgsF^e^{6pS%Dx2uWDDiQ}a!6BCxqnR^ZK6t92e z;JZZY>%Ic2uMYrtKd_a+Ohch1*WKD?q&gL&ZB|l29X&p9Xnu2eZy!>+y)3*u`M!>^ z?X)B0+2u;p@2rogzTADs{A{c7vsHyR@40jg?D%}dG*sEB(6n02jQb|9W?tdSdAg9LB7(&qVyD$KQ0`tGKv) zjd#4ADy}IUoW6eh{tSt&$f@__a&u_>gCntGfB25%HYvF$EG3QBPL0?X25iD&q>g0D z9_K9b{T62xh7yL!m;WNFf6&n3p@yl-#QB`hhndGnm{W)|HbN>B6nor9FbPo@>C<1B zh)q&U9wScD$5=VOpdI9V*QmN-Sq2(Y9>Jio)8;o;6F&HsMuH@Bb*F9tQzW6n{_=Ku zEt3nqKP?Ba5z!|5n(BsGm+M%>DY2dx+OTubEJ2#;8Zhj&1IVwuYg(zjO=hc|ouuu5 z6r-l|{=S@Xagz*vwb_t@jd% z!0kDk&J0W+MIDmHce)a!C&;r(+X7w};#PFOuR#)sxI+HTG8zpYs7UdfrOe>Ru}G_F zP_8$Mo*aZdF};h~bQiKnMMc(JgAkOhzXA$+=W*xQem&so_1m!NpXaQ>pMGgPOC~4( z&3j8=EQC;1W->f(1it`(QCJf{Mwg{Vo+4W$gi>Ad`IC}N-yh9lfTg><}G90fnGP1oz<|0eN>$>QK#NXL;0w*oP7_}S=3-D zwa#ng+S&^_^2!<50eD}NZxK|AZZ#qgV%4i2@d)w$RwFhBm` z9D^3R(reL@gu1{XQRZy!h;2gVjE$yTYB7$k$6HnfO?Z#)k`m6dMY`VT9~|090kgeA zcx2niLbM>>dU_ms+ABq(IX9U9*FN*=h@vbSMNzKLG?woT>H^JIIsuS@>3DTenw(SA6(N(z@P}8Yqfpky)R4kxp~^|_ zzs+!z+97->dJ?J+4#dZ*wY0E}-wRG4>Yx@_5 z6=nxe5F60CzByoJRgj@}_2bZl&K-1Lcls!!u}maynSlyOMc&Oyu0L^5_233~BivHbx$nSkvq(!ZMlQks-I>E>WMw4uJ=)>728p8mM|c7luiVV+CR zBhKIBW_2Y%TD7FJ0jWdUGL`C&HqtOr5HqnN7{P$WKpYg|IXsJQxagp8v<3wsOIR}f zelaGWnlm``Kf}YFL~W|!(xxJ*T-EH%r`{BOOuuvQ;a;s@B6`V&QSH-=!|dF|YXCl!J2W zUeMEkr__75!k6AII6d97m7=sGnF%T=SZ8UvpSlK3j)sH!fZ?WLVxz%?CK0AQIT z550Q3BTXT<{*HBYy$t(w%YtYmNpZue3mrEJ2H+)HomjF>gLGgF zGPc!sr_sa615Efj^`ZZBN+RfId6FZ2!m!i-)AQt8CI{0)ZMw0?wLd-rC%1uZUBrN0_qj8*>+Klhn5)1rctxRyg8>y&m}#3Y3bF3!eN)Y7^%)< zG2{+v_vVkRD@_M7&~He?=fU-s=zi8x&%8F2QUCbfIZ5222Ny&ymqQ;x^Treby}pyN zACry03#YG+uyPXPmctCoUFaR|izf~j67E+1*UW~HN3I%ZUjc&SFdr^0rD&?JZM%0^ z7cb)TDjz@6ok~d4BxtK;cs=ak>5P=-Ro<}-k*D7>y?#Dms4Lg#o*t%9Zdl-I8gvo zi?#91-ClF}5Hl$^!#*X;WAw7uzlw!}xh-xJaI*QuxV$190;NZgsY@@$NLVOB++PYk zt>rP;*qhd=u;DB$qf%ceZe+foRWUYH$CpD(Yqp za!W932<^H#!b9)d(qGJYfjt$<*01gZwv@tgjjq3y!W95MhKLL%lCpL~aJ{;T=>%BO z?o@yHDX&tczf%F25gONof51jAWy*>k;nHv*D(gfC9D!ESw?}tbx77$Ql(*irUTxDH zcQuPe)ja@R7kQi*S|vFKitwNx0t$OX+pQ)Qk@kgvJt44CSOf~P9Z+=Yi8(X+%v)Fq zYbf(A@IPi3O>+hhC0#_}@IJ5V9OHH*a?xxP`%|CyNxUZqjW66FsEG3?%n^Bd9B{m3 z=@Ci(fBN%c0>p+TFyL+0tQlnkW zgTG(1^yS$54`NrQAbtu(AR$CFmLUF_oI1ez@VNJFdH8q=vCzj}}LfI*{WIRHL^wjlE(kCZbc!B+|wI$K% zAikmwF8vR?D_4*2bUfGpJ?HJ1MEssJ@h>onF!9)m&%(DM;K0*75&2_i$TFkOYl>H+SXk4g@G@#2rx}T^LK;_Y%l0d_9+!E)l1m zE$rvr48L^qz?t9QKrI?JxE?!01(Ifxc(w@gSvY@LwwP~eS^<}EPaybI$snz_m(m}%(i@&3J~ ze&8c8#8mwj7K|>exA`23Ccez4(J?{K1G4+K^TS#cikV(3ixyrf4nZU$R34DHknHU5N z!?+2M&{WwJ2Q1-}pXM}ezLpz9#d_^VKvRXRi#0~`Onzg|*etYMxlF%&D47Jx74SaF ztP>@2fdv`Yo->|)*=x8m@oU&HR3Q=Bq*!7!aXlUZdaNKFLT0pz6IkFAGc0ODWJZfP z!O{&SCU9B(rB&Nej5(0u)h|Hl88OGP<@{x-1xkXH?7@tAkSCpF_5=IWqa`E!r$s=j z-^BE1l|XzO#8ZSm=~<$=Kzx)j-U~+{cfJ$YY({|j(&HXxF5>Wrhne>4+OTUx)Eqz6 zMu!*s=kRmfiLLmvnRYF(F0wJvszO?mv~pJ%&n$`;dlHOD>v4$Y9AJ-ImBE z6IdE6wt{XD2R>1O*dQvg=#)j0Vw{IBZ?MCtOCb|eRiwz&XXs{-z+>tn8Z+Dx>pI^))=DzXc%C zYa*R&Ci9RA>iGCIUZP|s&S}{2NCO6j-?VI%ZMokX>9B# zvvObjR>K04gMOwBPTXh3)%#U9sC`2ue6=A9F07uJL|ngi8@2g_e436 z;ajIQaY4NAIejnGB*O;1Gglb~un5&C^%~I6vb%nJ);i-kd$d&gB3m?Hi2RdOY;LB@ zeP;}cm~7j*yNALTPW3g!#QB$lk0Th}c=TQpkK)#Q`+${ZQ|<{|%ylLPDAR?UBEKQ` zb%`?8SGU$x;E(Y^l1z-;sH%xmu3%bRN!g?!G){J7tuXUf%>(S+-QG(0ki>LEnSn={ zaw)ZZZ^+`+!^p7mhiEdeBdwRi=bteyW(5pZUI1UOc+p(rRNc7^BXMnQdE1%p(+x>y0-@ijkej zP|lOP2k}e1ygOm9cE{hWkD)7LEnw!-seU0c*6}!z{1$>_sjRxW;%i4hioR#;AoUKM6LkS}U$Rso4cLWDP9eJ5U21TN}1V3;RCLKgl+VL=R7< zfkQ74bPfkSJmk8{re~Dft?ye#J%6}!=AJb>gK9_NTIj!Ov^F7Mtbd34Ev8y4fS6$3 z=J;vdI=!$7?Db`3I_*Lb1KRRa(oHY#XM65@v+*VlfZlh=rDAlajSRQ>2ufl_3qRi{ zjMcZ=s(AvO(2q;1Zh4GGQsZYaSeCt7z3!N&fp1MU3^2iJjty-l2Vn{)h-JjC*tdx+ z+I=oB+ead|f6oU}#^3k(>amtHn?Wo`$>-r>Is{TMrUkhgW?x8FZ6zlkL{{*X%gF_i_8eZLvzdUb;DJH2e9v#Y!!b z4MHG==urpxQp4n@#;Ve^8?>9EJwdgnxZ(HD#Q6!zqcO_|fpl%B(EA$D{F? zD``s3o|=BX)9jgzZ}xIkWo#KWd*yKrscGLni7n2VDuqC(+-KsCTq9)@VOs3pPI#@j zM!PQ{-l5Mvwb@lJz1vN*>?Pj0W(VYH5=Wx-@n}#HJ=M*VT>TL&lBmSlo)Zt!Xb$^c-dc>EFrot_?NG78<6ht`S2lW;=*QrhB*J z!G0c%jMvu=+s+Jyz~yjRhwVF-c=jDzb>CmvT%h5bntT$e99$Z1EYyLdBR0Cvcy%>U zPz5aYG;a1`CN@D~i>(zyNk{aigWne|N!!(Jr;q(Pl%@Jjv6-*6vy-_XT2EkSkqSh9 z{=a~QG(bV%1v5v+ZM67+Vtkd|{O35iCSAO|S(CgENJ`cXRc#eYy^>Y|$w=lQxEHb1v3CHt z5`n)-JoL~}YC{+dqJO3J92UQ*kMO2kCGwO9U}c==em98}!3S|&DyZ$z_+C&ft*Y)4 z%#`yZEkcH|U9(FdYfn3~ueo--mH$=0y8zJ4)ogk_z3G(Ujjg3u3zkS+9rep= zyF=Ofe97?VM5@C}N2sTD4oby3!>1@I8Nzec;wAw%vK(TtL!*cV$@Oj`VZWEOGtz}c zEI&^8_g`4Qve_orC($!9sBWrbgnWP!&(TTvuA?ZNtlM?iqw6N-ao7@BuGsWvI4^TM zuHKp=GjpIj1MO6U@fb+a6pYY)d|##VMjoP<`4RAycOEeX`aMIZfSsIX+}>Z0cU40( zJILmm%fX zSUx0#YIYRE4m&3!gR(K}GgF9@Y>BUpwZwEYw-9EG$9!#cwHX2ehO21>ED)3nBZ7Ck zZMtAN8u;_wg})fr`5Du?O(Dxz)IGldNe_AgE9E-YW4bJEf5klP>y;@TXsO*>a2u-`MF@&Uvn&s{sqRD zK!i*gXkeVf_Zq_`C7?sVgWxa`JWUp9f+J5Kh%(GoDm^ysh^R!vXCGl9^L;kO7CKMPmaI=GYDSsu3}X>`v*dx(=gt!4rih5Y!NeXS9U3gI(p?-jAF>xI zK^nWA_^`@QzjeG5f|tIH8bd3#+UlFgd=!b)RcxH$qdp3$N!;t%_5ZP@?l2=P+n>x-1iw8Akim-e`NaEjy|}S1!%S^X>7`oG>UKx@cNYVRgw!WE3_WU zRiPzC=B4q&C@xEq8AYVNi+LD%F}-^_&LCxYnI#Pe(?N@6&wa3Y)5; z^e>5Gu3@?qp+l%0g=agfCVf9MxuXv1M0jGy`x;)z6k0rzv7+03K$>vxA1IoFbP$1X zU|G-qHAsa8Fz`c+Q-lur6giC52Ci*(*v^9%g6s z+sp~|o0SGV(PD(Kr0Ki5aPeYJ?4hNxDkkW#N(;xc)1>1Q>*q1AMolw5mhAfHlRzg7 zS+8Ul;^Y+JWFOTBlEAuRgS{Rdo{oP-HS{E@82ZVj$0PHKw|dK|T+V7w)8trzZ6Kw7 zbTJ-sPxY!*`zL{#9FE!4F1Fhmp0^({zhKL*m&SJ8A>3)4K$^I8-VXS%hZ+YAgNyqN z1ImsD(JUde^Le5l;voBSdC{el$?}xP2E6OJQ`cG_IJ^+^`dd=DbyiQ{?@mP%wn6JJ zE*r5;;b*>bt7p@xupDp6E`F@-ok$e6a1P}|=Pr8TP^c)T9gO2?J@2hGo{_K=1qkq^sL~SfRry`>(E?uRdU950Kin z9mq}E^hEZ4!@E0gl7(|P6KCYQ{Q~+Vk^j=@kVb`gD)OOlf?iqOe#i{*!EnF|^U*A# zZw^o4Ns%)qX5Wzqo!Y~d;{9+eFdDn#)brj^?Q_lJW{u_R=VK-kr1Q@bK#^_`v44A3 zfdqe{006Yr6DMpT(7dOpnm;`K)HstG{?BNT`o4JkwDPQXE7VmR+65Aw3PCB>+jr(5s)7yZiU0! zSomRJxjXoRltt4G^wG>roAfNS65zef`0_k2l*7h!-!A^)w(zawZ1oE|K=%1BFnUMl_|5jKT|NSZ;bIB+vAVK9V>n9`x6N zpf3Ws-afOO8;4l6@$yIC+Kse^4T3{lx^a5BGaCnS6?*b)G=4a0DCvffE|>Nt>4eR4 zWh@Y)`az@|MS+7%k*w%!_UG+JxCRi7&(_WCv+eCdaWaK}Yhx2vYytf|ndqai53mfx z_~(k?@VBt+Kc=ncuaZJL=t3mlCnpUm^Lx^G_i_|MuXQpNI*?h5dlZZK^}i%=A4paQe5b^tWUTdee+Q8iImy^P(XNP(;Ptp|(}{2=Ng&lG`m=i!eF{p8{} z|3JVC?uyf(QW2IfN`rr!tB=bayC9swPfJ|xYhJh-VjWfNP`v27=V~g@4w#E@t}Ckc z#qxbFB#L|$o44V0sL{Ua*``hR7us_|Gu~vPQw4i`t z7BvxWWz7mY|F=-3_1jH&)Bt;n7evaX6vIF{`%1;f>M(!uwR5P$aPZPyJcF%OYED#5 zM5>Q~iyS2IJFNy^LieAUwt8eN{T*VRcG3|xo8L5#jwzrti)q2Zn8>5_v4K5zA#Z-G zvJs3uVG@L@Jrc_n*(k>QTM=UCaJmeliHH_l?G13O`G`mKsnhm75k~BsJZWsoQ*RL} zG8zS@e3*Myv7&}Ki+)*TyCgBwg6ftX)nq7!PbMg)Ove}BG?w-a+tLlkR3*8;2>BEa zo|!1DyE>y=^^ZN!$oi70cVipGSnfTLftoPWoXMH!cxF5OM`KhrpXg0H^+24n#~eOHrr=Pw1*a;3vKmOc#nTk1|3tF(AN(hP{Vj_6tIr z7-8-9;~+u1Wnd~!gRLMKUFJkFsoLd&u<1kvQJL%X0Cmj6f!B;fp492}69`q%1SXS6 zS9pNbltZ#kZY{TISA#c)bx|oyN)o;+?;)EJ6nJWOU?q=mpTO2Y^T6Ry6eG}L#IW)u zm339RFb1dBym?@5kUeYTB*{kHG~wHx%!`GZ5IVoFh)ol%Y{CNuZT$w6-(Dx$$)P=1 zZ%atmR~nmiwm6d&#*@e&ezvOczJRwFb1nh{hK>@|m&T7gDXZ|TYKyDlJ3J0r#9SCu zj=-~6C7Su=Xl7a&v})j=fx-Qok?4B56pEuM>GixX>#@Oj=Fgy%r zqJV`+vDqVmaoN%^`fj&qpI@imY^O^MYy6W4C)^CFWhc=IoHsRthr1!-6-&G z%85@+>Fk~t%0Oy*UaJslD75rP55&Si`W36>6WZ$pGJTud$qrFu)5M`KVjQ{R7JJKA2d-w9gQ{+4Y75@9_BF>RIzX z6@Vk`&(OGEONWG|I|)IUvutZbu;K#4wiw!QcrQZt&$%NF3qLgX`}=F7bO<9$_}ED{ zzr5XdHHfI+(POS=sgfH58)-5hjH)7UvGVcj@@CnCbP&cc?xB|bS{pzx;c(l=WN;BX?FG{pFFCZ)fBnKEO?bGlFR!uWCM5W z1hGw23r0GJnutV8kFj|8h(D0scFB6F<{Nc5x%)}tUG2r5ctT&|40}f8cpyy415@KZ zg$#U=$?#m9nss-#S7iaJ7s0aMJWP|1Yn4gK9|w_kMi*aw|5m~q&G`Tov;GxRrdx9q zg(N^zvUPJ0fWHdkhMUMkWl{wuH3_sL&XvaPtsBTSZAD z%a|ZLE*zwW=h&RrGgYYxhq*yeB5ual2W-}Qh(F&3g|)!pWX42qQ>o^DIu@@;2UR|g zTUFb6LbX%So?K&>O+eLrI=iXTOQn-)q`9(Xn>EGN&qlqS=*8oS#g1atRU`sPbzhZ= z1r_u`c_Xmlt2lqBhEPhu|2V41VY>@qMLSq}Y)WU65e`1VROVKa_^;)q#`%MsGXJ83g) zdk=>9!JDzkE^f5oC{KXp0ho|3;DkFZU44KXM7%~6IkSHkn?Vzt5)c`rJ!FaHFQBS>_s+ z5j=-cb4Q#HHEcQU7t<(9kz@HkQcx5v`5fd(8YV4^Tv_c_eRkMd$UrCZBx}wkIYwG~ z1`)ELs1c2)7{g8I*xjF3I+HKDx%Vh7l!W@5oR`>D1a;(B+!()N+j%%%ZM%L~Y^N^M z^>p*&u_2%-zTDvWS5T<5OrQE**;Q1;j~6zgEbFcUY3NR-4z865$Tk{e|mFW}(La>Ay{nlOf4KT)0mEYawz6e9-EK78&Kv0-XN>U#o&A@IAAm{M!c}fUxqfdko(vdkW=D8Pq zRn?;i>{q-nd>cshfdYy#U$B`B%q`zjc74C>5|@o0UUWA#%{@uQCHpnnoyp5F7Xb15Q-lVdmdw1OQ8*(+MXs`SeS$zLpD0!T?-TyF7fr z4@<=SUmZjOAz9|gfL@xvco-z^hmL=tDO5M?an9au9!hR};Fv5Oy*sU=n`foi>qr0a zr!~tLF4sVrhx+<3=Ba=B;;-4Z29v6`ESuOwu&$+F&9pO~@xHbziN_ zOKg~}iJCvoYx1F!PbiW*lA&<^-u~3FXU&zU?v56jx`zb8D%^<=>-r$a#~G&7 z6vkm&>GXYl$_f&HoyoUHPv`0NJpE|4Zf>4+*1(0I?x%vU569ETkFIW3^hTo5q09e4 z9X;7v;F|2vzU`#rl1F5#kbOeu)-S-3`E8FDUFqQt5-Ht9(_I~@g+QnJE}*NeT;nNZ zk1<++hNq-j9W(I5x-8XfW8N!UrKzxldQx}Y9WAeh&wTs3NcR;CtK-8scr2vp}llASwT2rSt zdM&vC@5Qf$BWSS>CY|`s>fsNW4B4kE5KMu~sJMx~zAkxK5rO$Wx)P_vI}&E7{&c!f?nJ9vZ|!L+M0YfS6Y-2MYD&XwAOK~{WA(dEOy zPOQnVgZ;v>=K(cqrJmeC^ei~iIKQ;@catrF{cqM9q9S+zae98PGPj?^(Y`X+&Xa5X z`9X9i)_9*fh92^g|M(3TW){DXZ(0|ev3P($7sX_tKSwC8*Jrk_YBfc>$Ok9QL;RDs zrSyz2N_Ax%Sv-4spfB(}OuMS!I;|pjzY3^{eHm}WHjsCUJW-LQQzq1_lLr8rtmc(e zAr5DCP#?Qgd}prf>m?FBhsTs$jc7Ccc=)|&XwXp-WO}R!aDFd3o-fq^crH-|J+6GGk@j#+M>+4p#jcUV=u84bmyTw2!2{#XJ_AFx zZY{oQd_jBG2p5)J9IGE&i)O+}_t0Fr+T0M0Y^eKfPoR_#8 zSWU&}PPkvw*@-}%&6!IOT`q3|rD;p&WnsBL@mGttRK4judi;Q-31;$$+Jyh6#&Rnj7vld``w|L>Y5&R%_*{e)up>@?+03Y!Fv~XE?26m1cMo0 zUp*7Sv{pyNt;Au;Wo04GxIX_=;QZQ7B0SguDfEPklBHBfX#Dkk!w!1&_Sjale1$eq zEWnTe-@@;oK&e;m+XGu}vj9uFbA`%YY;gWSxi|StpBfLERJ3YxytG>(X@YoHFLJp7twFeX8LNnKKes5*$KUey|OBu+C$eZ zSe2#61G=U%kv=^E>^fQSG&nEfiM+otRONjjf|WhGaEiz7y-s3k!ZAB`QHDhie%0KQ zf)nv2^}Wk@GfjdhK4c-^Z~g$m45|SCO$+`P#1In)EA#)V1yi*&GiQ&8l}3lpSI0SPa8#*G|bgY<=(Jv zpES+CL-O@?`JHXQBVCV3M7N+~cZ1I^RL;TtKJr(SMzy28mhCt42L8pavVmoz`3^GM` zoty~8Jq&BSki6G@6`8ThkJ6Aba2N&iWyDdm#FBOZn81>}Ui67d`6u z>r%Jo+{kjNDVX$TqQl;j1!p3#2M!#?bF?Y$R&k+-Kh0HEpXzgc`mbP#dH4OCtSxfO z$>K+WO+`bc#dj~i℞uMQC2v9}=XP@MjjNcXIV>&$V@5^G`pMK)d|H;7xoRapMLH zb8i-?YDKtz02!z6o2^&D=osIa-T>_Bq&a4ri*$wV>So(J{8L&`H<6Z-GZVTmuZQZhD<)qrwG4%%%&&QLj*@6dXUT#vKb`N zpfAAmQcy~^>3)okB6whi7oR?M1S)PEk%*5>GRb>^=UJ!l6Ca1zVtV{4cCQmGvX6Zo z&Pi|&^kGR9V0l~*Q&FM}TwO5d8T~DxOOD_&;S&Z)NnxKxU|0zqj{H%B&;6jIR}$He z7YlIrCmbS1&p`dz*$O9%4LycCxIEi{Db@ZxOYt>n5ahqk^K6xI4r-0&a3p#VtH;%R5dgz*eZ1M}xc1T3mLkB7 z?$=n5$B0&s4A(B@;GjUC{X<3yhf`GuWZapnt;nu@qgmil?q8>&d(ZF_7Xp6$dWFoP zb|b)gMT0($rqd3sH(iH$UEdm;VZT{Ms92*0;fxG7qQO2|=?S~0fT>_g-;J@tUWYY$ zcW!U20nF|87WH;{f`Y(quNz;r0Bm85VQ>8VHt;N>0<$B;xBAy0;7P7rF8T)g8x6P? zxN-GrYW)+gv=N%wRCc76sc4OJoA$jvx19 z$S&AZ!o$W@RQ5T8G|`bDEr${S2crwGzl^wXgiYHw1l4O7Vlb%)M$wbo_&V$M{JWSA zS2L;8Gd+cpbr^jyle_}*skVSdluPg=OShO$C=vP%5uR1s^Ntq*ov$rW3erJf=s4%QfBJ#n=?g%naqo4FA0`3WSa^5q{ zV20W}%%G5=-yul!02pld^umURDcW-T1zB5NJ&JHH6P=q$4mmC*e;`b-Dhu)0_*Y4Z z$>!QZSbR?6Q@5K(lj*ZJc1|zVt;nl=JQe_A8g*u)$ti64u1$&KZ-_TA!9wqK=s{H#z>+6aGZr95*?>Q z1;1qrPvsGx&;+F{w9}JdRjcdB9JSfBf3UA0rAY5FWn=~NMUpO|Qn3-K)KYh5X_$c} zOp82kqBeuPm?{mO55MaSes9IUXf4=6IMQ6GH~lEDKlR*MWpt1~D8^fy)f7t7B@>E% zdJfQ1W$PShyp)v_oI*2J0?kQ{ya>0@31}S+@lm%qc z$LOMgHfXpG6YexGi=ho+R#q;P2vSH+m^{gnCZZypsM&&5X(07U4uU4l;7Dp!ZHcQj zkvHv=iiN_H6`1VHb$m5P5&$U1Fs58^djed*No+im5n;^KpR=?)vPSW^u{GxQ2gIae znfiUoKyA+hrhl&UfqBrKowr){(+>8+^p=Q8If2g88SdL; zSJu;!3|vO7N@hL>cvy%3&iFo2yU>O$$O+WWNEW&yZSeXa<8y?Xyv6|2q05+VhFR2p)K^e;7A$r>_pK;P>3VO zSy6xNfrmED5FxI6oYi1#*S{bJ0kB`&IQ6I2M=FY~$Qjymeq2csEt*HajeRKe4#xK1 z>^dJv>FJ&`uyzH}N~GS+uVWBSdz@c8wM+oc-3OfjoF}l;RBe8KOcIkW9^=Q;rl(2M zb^*_GB_h@`X*5fk6kQVcoAV~Tva$QRr4Ns@H#$!L2ee^$^Uh{?4UT>UqJ0AbFO9w1 ziC+~s7bXyH;puEwmlMRF{)yA=^}W-}M6DC3JHayNG2x~J;-is!jF$W3J$=4ezDC*C zt~o=j)gW2;(h>xjPDO~3dh3#D3Xsq9Obj|)P|X<-l>w)a=v-R-@V#bbg9X0B4#=TO zP_w*tFb5Zm?7m5Pe8Xt{ooeADjbhHhqc0$`a0ZY;<2T#_J#pRP;p2P;IhT$%!hVC? z_UbJ}?@|KdCgd#q!Q9NyP6YmQB{Csxp}ucTv-9GRLTEep(~K_P2Mx>Qur5p6Nl!t*ZbfGciE!NSlvi0!g)*Ngr${&jQk+ z^m<7cK9`Di?D6nv#vip&EU3Zir*VI{2lP(CW?hPWr1}+`hCeGHu!cm_UDc;iV;{@g zGA-)^dZ9f%=s1vs=3-B(+?k+=C17dGe}=Gj%eq|r)>^IC1q6_k&;Mq!Ag*!vArf$| zL*^Q6TA~ruwhn;wxxr={^_;-#u2pL!1RgGbu_Yz!98P7_utN76EVzt^w=E~2Xxaoc zbAGqh;t-nVU^Kc7Ex?O;Nwj&5FZ{S=OH`mh%^f3t9WFI{11QZxKR!SFu158n?t#8T z2X}4&qwSTS9~|ADj-N0T7*j0%b_Jj;s%zjgfCNhY0^F1E8zp z`aWVJ|uw?m19L>&t8^nKg>rX=qoj7VSK&OVnIDxD&|9Cl&0%5Dze}UcF5QP7W>Sq3Ls+*bff1X6I*8B%UY5Nax zdn&QQqs{}#M2U|?d%a56Te7gC&8iz*3)m84I)Q1t{_pqmxfn2x5lC!P=Y2nLP^_7I z(dT9O+lAY(`fuCb{;Rz@`tCQuRfU-=<>d16wA5LYCZT$9y$n@j$3~SO-H)?o`G9Us zs?xA<=jOcCs(y{B>XZ9hS+rTw)1n(;8ePrj;CIcno#kW4?MV4KonY*F zPT2;N>TIFkR1{S#2coJ832NB6*GeSea8%<4^C@*bk`yDYS}_}pNZSVR{IV&iQ=&&J z-Pn1@k3(niGYg9?y34|FNoG(Dd|5j5rGxEQp|db^sBiy1K5aizGB)u$N~XOo_nec% zA+MEM+hzH;*p#AQeZqFyaz%(fDXe6?ey8oWLs;%R$h5;%VdjLpE(|)+eY<@}o6oqGlDxAU@K(#(K|+Ptox&K0 zlb1+5`;UDI1iCV=<-}D7dBM?{{n6lG_nOdXz(b#cI}D4?+byrf8{KdrR>Bazn8)U8 z$}BcJ^mAd2`9pw^TL;UiG|m^ho}4qdZ0`>p=^dPJe~X(&zA&zg4JO?rsOdTU4%ki| zYh9x*YbP9L?zqU-eCP;#g$%avp}8y=*v~)h3&^y=?0%grbvw9sPG_L&m}reylV>cM zXCOD6;moOqVVC*itaxiJH=6_BrMBDy;hGUT98?;%F+&MQ)OVIZR`sXDt-+LzjqS&yq z5E}GxrR|Y3C@?Q#wGH!?E`Vxd#Ey{n9crx63BSf*hka(L<#7b3lqniY4rW6!$$t>F zB4oR%G;LLnBLS+aLu#39Re*k^=xvMpIZ?M@LG&4xOJUdtQQq#b7(}83Ui6)i20`A! zmSi3-;Qhsfzbt++t7#Lrk7XbM%&T5&sLt5T+EdZG0F(kXBYXy=nE~q>3A~U?J84R> z8V-Zbj8_uGE|3dWN+y`9G%}gKoavVPVWsN^u3*EY-{glx2;?6ET5=qV(;@|+xygZO zsRH0BxCkon&*n#g7YNSQje;@)r!jQkO<{(KHsJ?~6_(52o-UVdcp+#~^ zfCGcN9;`$8y0tz$x!tY{?fX%jiV(e?g01FVRJ1Yo6F@eG~g<8`ej&2 z3@6;X1_)azOlerfMACgTgL7e6Wih7US09Bxkt;n5*pUQ#+YZ3E19V;R)gkG^@y>}a!~AA^-98!U^C-{Mc$u$ zrr2C?X+N-i&0%r3%;;R`XA0`&nd^U;rW~5bQU=U|n;yE8OUH=_Fv_Z?kN}@KK|;{N zHR;9@6dtAjn2|o+B;=84DRrdOR}ay45&c@(aone&%`$^ZqJdsp(PAcxt8lpCxH&8$Uw+V1LSA1q=N~ zF2))ELKb&rw-EY<(b>#LD!2=odh&PtjUNmYIr4Bm^Px?y?4befUxiZ81~=@T^~in3 zG~I{6>>X6v>dNvNj$9?B$@fcJ9&mLA1lV(%+S?)h4mjf7X8z09){aD%4$(EJ_w{{o za5|W`w|CM~A9m$uN7z9)T zhqqw50VFd@Dgvc4Yun~5)OE?kBAz>AS%W+ZY{%$E4D{ETAYBXLzXt)=EnvJaPFpbd}T056!6wiDwf2( zPoHOo=xUIZS)sq-J$vMTf&!*>F{o)mw%ZDd`)7C%pFSP^D3S{d(N;*)5uyll7XwaN z7Nz>QKXHtdbOFkiuIOb04nzm;h4&beg(W3*`0ttOL4cA!w*53l4#l43P{ZG(ZhB z6nJ^T?S>sav;=`od1sl&Xr&RVAEZeSIwr$(CZQHhO z}VC=qdE? zv6OwbZ7?O1HH3nV5Thj9OqSiXa(1W_o1wEMOZS16>-;xzK(BLC=uReQ zzrxO9f@cKw>#h0wo?Xz&By?!$i7nV`0|tCh6z;q*>*gJj(+0w<&THpj579HRqJ^kLTNzMa8t<#6T@&5LAsBVv{2dmZ6j5P8H=HxPcSzDGGX85FQBj!5zqgIYX5()va*N02|k^i zp{0_u4HTU$J_9{H6rHGrqmwf}I|DluowSLqnX@@Q0|(20Eu~LuXxU-0q5X4v?o9%1 zh8@Rqhqa2#5!$S51Hd8(pBuIYD3DJu<1Vd0EVMcNe*220xf91}OImL8t_>FOCZ0az zIE@id{S73nENsA5Ry1yCP<3#g;5??GGE06{FhPx0pshFxbks1(5;azaQw`NJ>LQFI zGZO)vm5wx~jEscM9jAn8n(-JWj!H0$emol;U^QhjjGpB%a-g1&br@g*J+7}A*2Gm( zhX~EfZafRKr~Y<-O`j5~GfxU^q6Oo6z{tZ-n{{s>_a{4g00G5U(+`K^5C#S32n>P> zRq+=Qbfpv)L6ihdCmG>}lBeNNoZpMUBY-l&pbD?psf1XjK8cfGC+Q+9yGV?bj{uMr zWe~De$Oys|QKh~V5Q$Z12F$~E2I9r4WW2)^9q%)979_R{M=5y$i!$bh#2rU0sq;uK z;1q&PpYtM;MOf|kT;aFK!-R!_OrG<97X(T*pmm6rIM8?R3q&N1y3Efo`J;hApbi0U zHXw@77l_s`avtC^CLza9G&W?YM#V$#e#c1QNjTJO?=wc9=(3j;lfKS@F+U+!mkB+< zp_)Js4iOZZhKu1z!X=I#6b_g+D*h(n-kFI>$6%ley>AF*U<~;fZb$uv-iUtE-!~#2 zXO9u6b;nfw1{X;BFd!KvnyT*Kit(Gz_ea8Wk4)X`=_;dQq~fE+XS3oW#1kDsz)sm^ z-Z|}JlJ5So%=7cIOLh5jeAH9fXK|*(bA`qL12DFN?YW^_NBWY4&~o{{;%82}$M$xK zO8^9e%hZiy;0+WkaNTO{alLAo`4*wPL0|K`7yFvJeIUA`c5^t)vp#c1TlKd_S9Ny} zG*fTlVNtf5Xl7&ZaE0VLxa9#3Y=C&<-yp-=i%%FwuY6A7TvFgO46{-W0WmAUgh|wiAi@q_j>3T=nDt~myCfK<3JPEtIX zCH;dnViJK3(!jsgm2R%z&tAM>l|R8kZC)56)3&v>bnpZgAuxaq1W?fVFj8E?8h90b zsh6${xpF1Vh0p1$Ni~(|Ovfv=za5u7IBrQbq18o3$~yk~lHwlq2fv{)vSpn%gSEb! zp#rVZDf!`>tkHR_HW8y;7D3N5wj1HBt%!#}6w@}Vf@ltx&H19wp7XM!EZ7k1dfH_d zHa@G@kIr<=+8gk&=&=zoUP7JfS3NpsBYyLiS!FPnF|y)hB2x2qLQcaaUOZzSF%!5T`T{( z-^w#^6v=`G#}{|TX8?So%4+f%9`^1%ZWl}CJUMJG)=@sF>q2u|*p^GcHU8ZzR$W-^ zGA*qe;;f0X%;iC?e=gEYAJ{zSwYPitcoJ$45IXVV@Z&QN)9+)eB}ruheu1C}4AUc# zM`@jLv~#=QfFt}6tcDXz9u3KgY!DZU^9%NFF?YfQJII~A3U=VPbDZa-@G!>Sv*}-a z`J#Dp&_;H0Py&jlf7u zM!hOL2L-rm>@xk9*d9&irO8&q504*4^EQU3ub+)?r4@0f_ruNM#DpIaQhk;X!;T55 zMW!M0k*Wp@m>u4jhP<$8)UJK%pE#bfP zF<3xEttAp_gdP_W$>z!lB+kZCQKN>v2=hE(Q=(p}d`RQIb2r;yrOZ$gP~0Suy%Rnd zZxG2*{t71!%)O-5tDF#0AHoQ)pc?2TTOUo<={6wpjGMNLCE`3hNi2{J5)3@OB7es? z(G5P$PMVozec}Hz>c-Hwb8j9IHB2L{lt!1*4Q3WaOd9W@xX@;&VOveM!&t4%meRU> z*9L))bqLMjz8^XDu)I8GX;Z?aj0#~L=2%}G4V+=8Y9_r}P_iMA-7DAdOGb2r`DB+5 z?g585AY5gz$i9W>i*aJn0&`(YGk6iOPf)zXQFX-0SDqOOOVnK7&VW7MluJ9q)4Es5 zjMCr#B%8AB9$$N&x_+?0!!bGnUi3jUcBt$4Njcw{P29wNKZ(w(<58FyRk7@U!H3Tf zcI~NMGBw3J1ukbCc!+VqqE~c#|8!1n8tvf&ZpKEekClkMTUL=P6wc zLX&B^A3oKYV>=uh$F)2C0h#XP{Vw=@n52`e+&_DUC|r6(I2Il{e?)cZv`(MkY)tZ}$o2*Dad9`yr3MSA5wjk%pR#aLN zEeDb-65RJ4lrz9I3q=1MBBUg7-Fm&-=Kh52@kbTgX^4=84(F5Si+x3vohlt4r!p75 zzYboH%ddg8>}DY=hAyAJ!^Os=lWG~P(&F76)vCQ@wV5|JYpH{tL@~0uwa_Wa*Yzc< z6+17VUhZfTe5_$5w<~^IJ#QCYwyL01j0yX76jYAsj5_y=FpOsHfwf-g`O*_BK2$U0QbXsehzaRaI@D zDqLlZ7Huwvw>}@}ne)tvB&S&iwrP_OqOovp4qzd}4O8j|#*1+t2z1@X@|CS{hS=iu}}^<&~&EIlI4 z&dXH^SEEjlv*L?YeG7ShMc0>9`C&t;BDT08_wDs@&~bM}w&j=bB-H(miPL-&>g#;) zYGq&fq5ga}*#2l>j^!N&UTcV9uH{XGmL_ zArC69(d(Jzh^j!`QI681#4$30m?#0O3)KsHI=LU5{Y;~7T0)UG3MQ2S<|Yno7A6-5 zB*rj<`s|4Wi(PkY$#nU=9+MKbU;M4DC+6C_KjQB;z>l2`@ucs?dvyVrgJ>A+4~Tms z)cD7r)XM6sVot))Gbjh6e$A@PXZ%Ux6T^UgH79B)7R)bYrpGuEq+TQ&~}ARcgK6kf?8dL1+_tX;=#oP2sv$UB(}H+7mU z`m7=y6?l{+Aa!6ThcSp7fN`*!AjQ{(11Km> z&0dh0;+<-g{Mvoppu^d_XTxHog}7^f;I6Q@S&KvpDt=N3<1C%@0iq0W;qe5;@bhVa zpW}?2d8%mwgA@H0%3Uqd3tc)uui)qu&?B8ps;RFFrh6%b+AGSTB6~V}FGGT3YUzAT zPL8bq3JB+ebjekPRAc^!JEC}34{Ov!dhmRp2E>E^6l|D|->RF*C+qS#qN4$acc_?b z-n>O$NTv;s;-G~G_lcl(9W}Hm=labRPznHxF*m~I3z|2)$zpp3&OLGr zuCPxQz#*!R%?B3L3-)w2PL~@%VL(!wUAr#ToW$_LkW>7b$q6I1vCW z5u7RV<4|KDtVPT=&e<1D!xYfNm~f;&FCU!gphihT2tRW84<6>qC3zy$WHh??AB>dv zCk7LCv4JBGB_#V~M%{li5X|d_KXj(qYNZ(=wBX3)x#4n z+;fX-t$5|a&C^HX+;`A7n3+Dbtzky%EjQH{W&^hN`H29mu`-$Eb1r%g)9{@mD!OJD z7@(g=mXtsFVBq$JEQd)%z%T6vMa19)hC2e+u*k6Pm^E3!x$?wRs6(LpbB5X8V1rp$Cz1YM+n*K^OsM*w8jOgrw;N zKSyPv%@G)`q-e>=;k+E{DWNAtj~3)suL$;xoBhSGulDhir0#Lbz8r(GCx6TcF6_i@ zdH&v~P{>^v?cmBP`#TB|h+PPXkk(8y|8YE5Sdf+V zCl0txve^aV*k&-hgx^Zsq>Yv?#3u((!i$u9r$ZcJBkk z0S+)*B@GZHnH0qIIhK_QI&egtP|p_Uz-0z_YnBnIMUw_#<(8Ol&m$l(bJzQHnl-~uE?m+GTOg_EKki?3|zT8M7 zQ3gguE|t8_nVRdhiYO^|=8n|anulvvTzE433o7H3zk{elo;n&I3uwjp8cD0 zq8z%OrjY<;_ap9M%`dgWb2l|KYY?>=C*$Y5=*Moj!|6mr;)oyMZf+ZKaKy$0v_GLN zeP9%Ci+W?IHi#FUczs64SC!xzvwD}Dt=IkYa!acln04h0inM2OgRw6*yAmVoo8yJt z^(*bmFhYT?hb9m7Px;1v_O|dqaQt|El7b#|FBvm$X+aC5d-eVE>FQ|kcHZ(mp9_R- z-5!!RZn0-k!4#a~F|A}-*Nh$utGi}>eO~fdVuI4 z^K$KQ>ntOT4LyIq;Ii+`jDOW_Q@7%@kqb9fz{ntv3M%-C{HVIfC-U~k9dh=iWdM}c z1GgVzCt?vtaa)DfpdC8C2JhvkOv*p1r#7shrewoK6-9`}=cycsxlO|ke_+4B&Z6SP zpW)6UM@mx8kh2kWyoXi}{Kp@MVQrtIOr*^w?CCqHb+HPH<2wif$Z`Gq*b>X@op1RM z@mW63o+Y5x?F6FGR*(T2G^w~rRz^D;^b9PfwEP}v?DJv_Wp01rkZP?>Sz*g%wg zdm9AmB$EXuM-mz)dv1CT!deWl`X}_#6r1JV-}gdT@*Ias`Fj=PIeh*&Zum1RaglOO zAL+nTBZ3)pvVYnxX?!QD9l&aXVCZqpZDNNWuDXiYh(;BZa<)kMdx*;VGXK^jca)s- zzjqd89D z@*h|R%YTrEnVA0jW!YxU?WD~XM8AIlou2a&G$C(9eQpo)OimZmkAp4dT#1i9u6eg? z@uiexZ`YrXA2a|7e{e}R3C>&w0o=NO_m5FK^a(Y$O7G|I$}PU$Zr|r+tr9g8RkW^u zq)xk@deke{&w99MocXX)@H`s2xnrl+S66R$RSm0u`s1aJ>w0si8=F`6`}8BMc1IF( z;lY`GmdQX^`ZP$ZZY&!*Atie_s$zy|m&HcMrIYT)MvG6|rcHPCy2ZAZp50Upi(=}d zlxlF*RigdIs`tfx*S-9*!Vc_@&*Dxjav)$4ar;|7a(FRxBW2M|*YL_u960sWm)XV&Q8jO{x2JW}STP@J z-Brbl8ry!F_VK6CscXEMN%@IZpOSs;`lit8a_i`-I#m%mMvwy=2C@Z$$<6dcl-V)v z=fxUJ-u9X;S5bq_i_0ck7^^PlPq$y0ZK=WLiq@;6(C2r9J4m*PAlE$(#WZOE!;kP} zeYudBrDBN2Z{r9-uWRO?H1`w(X7&1cNGddGQ^~vV>p(>KL@3?8#_&}2V}G^pU8QEH zK_Z;;8M25f*g^Wc-x#JVTa-CogCAp(72qn@lZx?4X-2;$nai)ic)vNuQ^KNyx#*(g zOSkIJ3mp)#J9(GU6SPX%&WcA?1&u@0ARQz6oF4UEwQpm(8|2yC>;ib2ilq$d>}}7ALYbL8^bF--$_MF`diLB1{eJHtBVyWB zEh(l`icF7@Blh$Cz%o(O=#2KNt1cY}qXuoUDAvDm8kK>mEiSuT&9+ywX6tPvhnS1h zgdY8EBxxB(^)s2@gEi$|qfhR7IAME6kr^SBe;HFI=e$H9l}OTDe4(>0J%6`Fq8&3a zY{_$+ipG|*<8~40)b6BcqWNfN_9`^`q1=pDAsa+ST?2vbry&aQEk-kZjve83l@Z_unZbwb-2w>_ zFs@r-W(VGszS{%I-q4Tk2;f(|${yEL#H>kBTOH6e2qyZeGI>q*)2$E~S+z|V)n@+i z6%D|IC8P=Ae~g4NVTMtHdWjlN7UKr~Nu$ubLu%hJmv??tC*%y2g;XfLOD7{gmzH9h zs#mT<&P5IH!5Z0PypS`a-*mBXVjK*rbXcTB97iM;pDpn0(Y^cF!@gHY6(J243+v0b z9LUGrN3h9~qrlnM|^el`5p^?-j2JLd_r!>E?5% zWK55un#nW*j?$NM-88w6IQj4niQ`e(m;!8orgVHlkt|F(+fB&VwG}|C0`Ytg0X4`M8v7+DOQE5BrGM@CYEOJK-1N^ zUL3oHD*9Jq3BU$iGdnk^MsCiJtlCGtmb~7)@os-wJGhK@dg_3)$~hQD`^`w`p(R_V zFb<8~H@#gw=Pb#smfdB^FF>4JzPF|jj^&LAIMgZOL<{e@T!8%xm7GkbgGX{Js-_wV z)n?V!t!H}#T}i~3DiL}fk46{_(JsSo0ARZ0AV7i#0drWhHKg~Mprr(8@cR&&1*%OM z0tPWB)@k$5*zW!;r5_kNAWwbN0RU>qf5kGSfx`gGG{d}uDo{vU=#M|c6p1hvv`|kJ zN&Aib^bpVE;?bPQOc)~Y6vMsbT?#-tL5TOzz=*NR3;pZk6Bs}>P&B$sZ@zC zE~nsFFo4848%*~)>*_;ctRzRGZGQq&-lRH9*|&^Y3wN{(A*GMf;8qS8vY~!%;H}l% zlFiehmId*}eENjJ{ci`oHm*4aY~tN$8xIQvAA8~mTB3_el9(IKiuUfCQPfkAqv+Ia zUE6%%2!fiEP^MZI`{)_}llZi-90`B+FB;dgwx(828zt^a%chNYW|fa!;k$c*C_-5_ z!X)7bx)q`^VzzVBiQw+ZYx;EIRLE4as+_SbXlJ?C*+*dEo-?Bpvn$074IqsyqH#^+ z2a|q`h?Ze6xscvSl}+#NtFLc!?$yj(ba&zBMF5bj#ivy@-|lBej;7eo?K*aqyaRri z?)>l%7L`BCA&V3n?;%nW@YXCwd(QFY=gAH4m4jyTi1#2T-oGjenQqJ7e95y=sMrU<3#bhpbcFq#;C>L#a{H@Npzu-4ust>34Y*mR+nxTJg zFDF`CrCJbz%R_=D%Tj7NHLUcaiqdw zPy7h>9Sd1*@x6J$q4a^G_N`zl{M~q(2(|6WEqvjK7i%?um`Z?2YM(|`h=mfyi3Gr; zaZ2DPGQD+$v$9BwE#ognej))*DgA4$d*%nr_VXw|OfstAp`u0$iY+_()`Na*V`4#a z@x@(p*IVtmG}TfQMzP(Q(3uA(`GfcSy=`JgaIX&8wTF16b&>8C#$zl@coXM;t;(vwm zdIpbWCz<~nDSn!v?c!Ex@b>emR+)Fb^fGr%fxh6rF!|<1P$zbwXl-t*VOo=58iqG( zsW6c0-j!h-SpdDrFbt#fF?ODQ&Hj#gi_9T<*cW-H*ebKB+ZHutK$SPAfSL{lX#xIm=Yqbt5QXOO!vMJ4(KF}#_%hIzM2os$DP#jn1}EMQzoII9RX#XefJ6)q%_OG{wxMd zyATb=%SgDP|E3gpHuD{AHBivDGzo@&J@6+R;`QS=u%EyQkTlB-E1UmV0gckbmV28s zk|d)UDZ`MCkzIflAB6WuD;gi1;-a&O^Lj_ zax<&~6IvM2ZS?#`FRC0ggP)jLhzGQ*5laMEEQXd0IhpvGD~Q8x!DnM!TW&lyjFoa+ zlH1}{bhTzYSF@>|O${rQ^TEm!<)$EPGz$bdIo?_l%WR1Z`GU&q;&489!+4(G_5tQQ zqy1+hD?nqc_5`~9eN+q*t!|Fr^c zTz=Bwdcq=C3$OlPhZw4;&dkIe9N2xGxQhE&`Qq&Pm(Q46D(YcW7&+75DW+_HzEMcS zNp3rrm&3qy0N%8VU#Da}v45wG4c0&Za*wd#+)0*ifN{~OUxy#dzW`hHORoQcJhA?V zc@riErvG{hds$P%}xDg1#%I9y7vlgN?!+V`}3f`SM;XCtpZ8wY-W&L6Ipy{MUDn(zrbyC$KEP=oA5 zLYbr~4B3&j3IEL$v`_P%rd?imiWgyLaX)-S`QcU5KcC_o`{~MCkm&}t)>^v1YT|RQhcx3en(`-C_o%>$G9_62p^(^LL3ieSj|P8qm(b5pb>1E!Jrn|&X?j72d#uq z7cs8XUvi7M)>`K|gu*3WSc>zm_;d+m5R9KQro%OlhE)`WF+>Q6k0u1%2xl)!poLQR zMUB>6IS&hl$yX{41-zzh2D`HFUx-$6Ouny$95LWlrR!rClN>Vua2yHmBLzz`fs%l^ zLfsoz=FIxG+Qgb9^Jrgh3%f}hbyRCNyIf^cISqrUwYyySkp14|n3>FDL*>+C7p8Yfz+)oeC8Sfv2kg5VD=u78T@4^fFc4R+*u^0i;qd}sQAYLC!)*;c-U@LIo z_7DIC@SRTX`Z&>N#eqY1}{bnQm85VBf|va1`-@3!*uVT zhjQ~)u0SLLD~ceEv{dNg{RS^u+`v{6zQ|C>b-mE#&^i0iDMlYM9S3S+s+C{{#%w{8 z)dmt6liHURE(~>0cQ`fzJN_8AYDc%Fbnd&8XJy*JIqae5>f>{$+q=Hi3IMsX*|cxT zu`X1Cd4NE1jr*f7G?m0cYNpr;O~${z)e+{a6GWc#spK{|IRUx#+*#2cmTmUSamCEqAFXeZOvLTAP27_7M#rNvH1D;nqa`%(^wrQqOund?>SpuPz!4ZwWpMS3R9k*0sJ z(a1|oeRvwLLz^=>0}fAU@>Bmm2~*Of>zV7QODE8mW{;|^-PYxp2z^yo*aofJP&yz& zBDy}peelkLy5Z}Z4cWq2oXuwwJEU?fQ^QCPtq$}-QM`c~i#h{sg8r_$2;zJL<#-@* zLkWIB!=dPDX=A)Gs7EtFSHvqTLm^4_^F;^FqCJQ~!+jL1P+`uZ+Q*4+%mm(qjm$ic z^Xb_x#uGE$l$+a~uc^s!mu&r;%bf-V)dh!|A}tMsX`fp$0f+cobvl+)=zas-*i z^uA>vHDRHXf-vdnq=L2NO!i#IZvIfD(52vJXtMRi4#&1Mc*N{e(va(poo|h_9Iv+T zl9&=0{KYOIAaNLkLGy{Cd$*Dijo-C3HyOJJdtWs4i_vW7mhGE#8!?DmrpqB6;Q2C$RK%;FSz` zJikyp|0}*g3_c)vQ6Uh@-(V%^$1+uDuY$tvG5S^@=rWT)!IP`)<8?YSqb9&Z6n}H_ zd_whKQ?HlzQo_y*4a9Nr3kM3?ggX^#31ut)gnngzz8E=Lh5P(Q)Dp>0%EmXkr8L4aIyLn%j-7sx zItpH(6G&+Qum{3nZ$#g0wCC76z;9v|pmpT31;Ob|OqTnM`84{*Z^hB+XrRXv|9j9W zYwZgVrleg<)8G5Q@qMvm-a__>2t=}LHX%^%|9myL&wS2=ld_x%CStP25%}V$Q>Hwc zdwaqwV5b$K2ZO`DHx;?G0#b}#@f?vN<#@zEHSN3DL(bTuq@R-H?0dd*R$VUAL?T(>29H2ewPQ(oDsKh4^)7tHhq0paXf)q9zG;0v z2(~b*xxVLk)7WT?FLSuo-OZ3aQU)67bp&t@%)Z+}Vce`ruM=>mZ#Sk(Sg5%(kfTq2 z=o|T}FR=#CI3=H^NIf&;K%<#{>lvw% zqV3`Q({Q6=fUmNnVh$Cf!HGR6OA)M!h4F`7^mSat0$CxXa=)QPxqaPGFou{0pnE|8 zsiF?I-*Vp4-RXg#^8o5yLSQIi%PjPR4+s~?UBCVR{viQaBCqle6h#-9u2U{k;LNkt z70u)2!5G31qh}ITXgW#Of5hzzd^A|V=nMFWn+7$VHJjyDkAA+?gY1%J(AWq5wMKLRJ>qN6~&?Nq`qlyF(@RpksASlq5oO@L!C+(-AxoH$%B z&g{U8D9D^Kb{&6ayyI5z-agT2(}#SHdjn3H#nuSpN1u`fbKu&Dv_Kp&b7eiRH4BUd zh%Ily>)dZbZV5XH(eZFC)5Z5f8e4kVyxpDj-%2>TrSx9<7PiD6v_7?(twz06mAw7i zMYBD@_%KG1g)nIcR(j}3T+QN+NiaKQ?pl1d+QT+0DsR7mqBuew!`1aq5#QxTonS(` zKmhKi#D zU=`rT5l9@zXfWnrg?mP4d^pUl*Y;yD4Ig6O-L~XV0K3Qx7i2i^9UIpASC5w9cxm#Z z(Kt!5NpS~18dN9%n_*+r+BNfG6y|b!LY-)(>HOkilUhLHRfKqBPI!nL0adR!Dm>N0 zcSu2f7kC5=$ABhBwm~L(c=5ih%A@${TVG~S1*x?zC?xopQVa9G3@E)Hk zxy{_O$24Pk{(auivY zvK}Yv?5xLB9$>d2nT4WCU?(lj#KON{pMzz>%O|sENjf45aQ}%B++&!A|!kA$59xOIqX^WXpg)C=t3!C0;Y~ke1ei3nUqAH%| z-7!x>{Zv{Xc`dY&15ZcX?HDcWd>`G`>-o@a8%N7NgZ`VY#9Qpw)sDJi<~I~JnS1ac zg$;Fzsm55`XKR~svYusRT*!-9EAu{=Y@@ZzEbmJRC-<5A_}OuuL2GpAPNeO+aU;b} zZ6EO2(Qcek0fg-o<9q25OCK>22ESh!@+hK}$L+?`^b404j0&F6wWt4{p~m z3Z%z7k~{f2R&C79Mv+5CY;ccz@1Wf_Nje8)I%K+O6wkj;(&!qRKqpV9Pf;wIBfqvcwHIl(Zw zPuvbZzaViFC<&QhbNPL4Z+!?&E`SiJDAm$WvzS}zyhEcU)Z2=Hdo^BURXAmTYuGF1 zGT^$()I@jNxn{Y4^;bRrkf_41{rV#Gjl@T4#m->4a<(YiXlr`r$#-8`-xtbsY?|hD z$|}7bOP0{Yu7J{tva3#PX_gjFraLJcHz46>02VZnaLg81Hk9M^5Ny}x(#WSRf!nLy zrgGN-xw5hKwpT|%cLBbd16yv$s2dPEEjI4t=-~`|a;^8|&;CdE3x&**dlq;SqwhS6 zY#Wkf{nlABkSAQNy#MH!sB`Qb1l0HW#Ib}yI^eV1S$F45P;xFNZ_<>}{0NQPasfA3 zb7h>GaDn4sm7|kFFleG28hZ(B6@=QmRc;hXCPy3ks+^{Sq@?5CP7zj+(!x%H$RNQB zXzt#I3fu+Y0K$Oo37+ki?At85d0<*H@Tc5rH{dG1H{Yh)f&W${@PrP^r}KLSFwp&U z+y`JGyruz;G9J*pkq>b1Y(MMY5Y~je3X2IIXNx{Wf;U$O0T*K5pawzkwgwt?kML4% zDpH$Z~+=5Q;2)Z-Y)U9@~MfB8~X9UCVp32-875l_Jo?NPY^CeO_}C7YVoQ5o{GG|nLtJnglU$feBUI91C|Ly9~678H}bRX0{1)w#($G( zBH01oGLZdzq4o$)QpCJ2G6IJNPZ?l__)7v3{Qe4Zb-%p4hv#(s1UW)ytlSM=y8v-Y z^MFw*_wsmAK3Ak}$f7_VXIUS%EcjAg<1dpW8t}1@h9L+P!05z$`&yEKCk6{9XLuyG zT4E5DGb#LX))1aSwgZj99wtu1xt<&pvzTp7R6k1eBtEsc&xA7lAgABSuam-cB1{k% z1zTb~e&=y2tthhK*SK*k!5a^Y$bqM4)rbaES>M=AlO}kTx|%o!&B+*r28xcpq;Syc z{EwiwO*CUym4iyiujfV-l#iv3UW6iIatb)+`22R(%*QuP#WQI5{P*%Nz`Z$?bi_bJiX411aIEDj z%Xpz~vqPLW0@pB*cbRhh3+~Pjp}aralM*Ne@#oU|wKkLnH!X0b&=T789K}$VK?(aC zkAvDJTZ3j;-xzn$3X-e~6{BVsAu@f6l+W0QVld2s<_FVwz?w^YsG;Q-q~Fibf8I;N+@5P&u=QdyQ;CaSe!@Cd;l$ zJFC~6(Yw1uU9ju?_>xRP@*_Hu1`s3(`Z-^C_BrV^s?-Pp#mj5PH9|3B+M1e?4yV>C zUI{oL#7$1Vt^Wnq8vb4RP&@Dgnfn9Jy^cztnYu0W(7%L9>_|m5jdQLFWK%~4JZ5279vqeqheJ@sl|!?Fc#=s8 zc&H)vKC1_v59{8#0d{BQ@}i_<^aNs>RYSS?d=WU%21LKyNq}X3LQ>xneOK1cxyt*} zuVP>ULOM7u zHn~m!qpv4IvC8zW#{-D*?U8>k505>l)n@2^Q^Be&)@E{rw+L;&jwA~ufghuAuukyY zCemx51u`AEcwvg}O8MskVA`^CO<3ja8ic!r0^Tdk`jtEjzwDzQva&phhh_Ou}O#Pdn2)CoQ-SoxO~f%4ADRE$b;8fa;hf?@jIv|`hA2Cmr`5ZrG6 zwqx^AAJ0{bbnFv#oe~66W0upq5lW$qdIEjrh#r5#wqdN zPEf^ANGsOXB#o$Of@(~o?30M`+%XLKp!C9Jjq3?%(oA_ILWMh#p1K38K85t(A4EO0 zK#Hja9w9$ek=A(p^}{Na7!U*KX%SWy6tU(ZXg z-aT+`01`J{11a|rsH`9F)LokR2l}aBBSoeej?>d@vgE{#y-yL};8*CcDHgAh*+k!6 zD1$yvP<6Vac15I9=#ZF>bKFb$o=VPO;AGmEGC)M60VpNkkl&?a+iJ#- zs%g9jcFfJf46qCH=e^?lxe}lWz;+-YrqGZi4QKnP|g7A6P z#R&vjW!LT*-|Xv~sO@8I2-eC>^qYV~mGj-?Wu5l+Bh!CqqqWhoesd+}*W~l%q4d(O z_boGu$jX5P!i-V(GL-USRDuI{HI5O;HO~m&F8a+R zovoh9z?YLu8QOX_lc7PJtB>6a)`aoI14n)Pq72msT%L&uqd&E7q8Y(2Q|D6IjOQT~x&JjWb$J!D&F()@da@4f24I8*3L zroxnLTNEbP!1Va6n;~ikq?omdoCXqRrWGk{92A4Bjc)F(Z*Ssf27}UUdD9uT#oZu~ zs1p6ER%Y26|2P5Po7`9Eevr=E}r3jP?9?{khGX$&#W(ogl26{f(iPzNUw9yUk?kB zu!*>3%i}z>O&5*<&%=7%C-vUWTERaQeE+QLi!kJVdbZpjareF&EKxqlpqu zJ0bJL*x@F;=k~2e=7@;wr{c%7IGx|FK`~J8aPYOi&Mwd0e%_06>9U*NiF~LJrHmtF zbF0TQc0RjrvmM%MyycQ_9&Cz{+1=YK$PuPt+vqOopPdcn zB+FM9-?$uD?H=dI4u+#|OZFMAqqP?sayFad-!~U3mtWUWqn{Nl+dgj_S%U}O#|wNO zK^yB0T?K-+1GfYzT5kf4k8*aeSt4nZHexK+Zupz0AEe<%S`;eHmC687B{gLwp2e}@yy$ZsHR!{+EJM0g1?wab{@Be+T-@|=%s4#>~8K+}-+*(z1 zzF!fhQeSU02Xgo_ye)-RQ)yfaof9>*KPQ)w5Wb-q^wD9-%@QXnm#lXD>%zn={Vp^b zQUc`@vvQ(<7@$$e9Bafp8AIDi7-#zwHRqCv^Uh?6uv5a1hvk#;*pcaRMZUXaC}CcI zvdpr^BvK*p)s2UZEf0{Wku~idR*u_jw$b5Aq==&kg=qnt(RWk;sUwxYPO6UOoZvRc zb`8EF8xLOH-#R|v9ReGgr;TPIc5O)kH;PFEIC!-o^u*UC;zx^^Pao!p@{ax{#7+1A zjaI3{eaXe6Q@|(j(4uuj8>$%c%vN8?Iwq0v;1y1G!c31!3m} zY)i{Uwa19$xS)ObTSJ4;05h0(yYVdzxf)vqxW#l6uL@G&c8kP+VNIGRX!MTF9BUdi z-?ph>4AV-aw=tH$KtB|aF<&2Gt=u6AfI7O{N7N%7*jUI<>8)AViQ4zr3AWHjeLS~z zUhX%_A{JA^&#I#aJvVK7Q}B19hlSKMb@;Uo6FErIC@`40~Y|nk6Wlav&YoO4_ z_d7lv%5v!f2DxdTL=HUo|1kCrO`gyOSapg4!yVmu(MVPD zPeCdHMtHP!%7uv12;do`AFB;c3JIO-l>paaXgg9(6 zEq`R*6_!CS(i`pMT>49rM1sLYx^NQ?*9l9))D`}<6|O#ODh8DV=m#O_JJ)a2r+Pzo zM#=uz+PjOk+}lV0G->#5t4150m>w>syOz-7)*{Suw@!WeG*&Rj!PbL3i4{RK)7DtK z+-a^086>)>(ZsKAMDPZk3t8{9Jj7V5w~u})2p#JB=cwSsMA1NXBFuOScX0V))si04 z;JIGfye-i099V3nkGQ+$rIw_Skef4Poc?1LF`jO5}{ZHK1wO~cqj zz^%#zB$w(60VH3}GhV13KolSWxnpX3;jHGKKDJA2_6y1?PsL7k?Uv1g>$76;1a!>E>GH8rt-thCqoX0(dmPIM9>AJxrgGzIcQ&`<%6Rk`G zY1#Q8tRsPv4K^_`w)u5Vy6B{e%H3Q0IDkmRxk>Dm(d(78ga2a&L`qzC-(w7?0Nqj! z3h?-XbyOmV4{pBhuW{E<1o|RDp=$uW z)}weml5=HuTG+S=_;P2@jtFL>!{-6B@;Cr1uAfJuRYY;tQFXeF%KYi0i8UmM`SR`a zk3-tn)R4z4jV#Zezz3X`2C-#{I6540BH@D(B{PGZ64Qx2l@@=`bH~W#t;-_o@q{z2 zYt;I>Ezwf_i%Gm$wLcpDVTDQm$*r(l-ZtabVvpRN+*?;QGZ%X1-rN%*tH@MJGa*`z zneC)X2PU!32V$mh)%twz|Mqd23EO*f;jwvQt?ho0)hh9F#u3iw@^lN)r|TWZ+r$fA zeDim%K`v9yZPExA$8B<#lqh1T+^jHg!l^cujZfdIrJ%$AM$4AI=GT1DGKIj_F}*c} zu=A6g)U1$D;MgA+Z0wNaKXf7o%YV4S>>LdLt1CRCrTs7CkLtIlKOk;pLpU4@uz4?? z-I*-I{b%%Y`Cs(5o!VYts3vE==LJK z_U02z@7v|`TD`7f5{8kK+OdO3TS>N^&90!$x5+zTj$A``6Eh(GE6!r}bQaJ#xlE5W|#sBoA2{f@nY55}OLCI%QY^t6m6Qy42+dN|ITjI1R@l09jh# z{(ff9&fc$MAU-O_Oa^MOV^WF=Qk|x2Z};m?Ik9H{q)k(>ePKgfx^qT=dVH45+xgM{ z7P3HKNlH`_s$e^AcbF#yy8X~p43sf6BZL}hP_j_4y@V{Zv)r1K>R{qlq6k+M6Qf#! zHc0);pq|v?K4i9nDCJ(_JZ?On6J-tj6I!Su{hrN|x3qr`sM5OoJSF513Ub3-ay-5T z4=a3eSWBSa#9|md(jX$`m)Ix40KX@YbdYel6FS_uMjX0ebt-eUY~_ij{S*-^I5ip~ zhM0;)QGgZ}93!Mn`|Z?jfF;9v0y+zRWe#rgZc|8e;6b$~-*e4JW1OiD>}^g)@AYP9 z%uIvZDWFe5dq!M}f6V63Abb z)9zFDv0m&Yd?wku(Ux|u>U1n9+Uo=XGRIIQV-8zMd630*i_=1%2K0^01{B~{q1U-S zE*`(@#V*)t>#%2nzc$<3#OYd3dwJ}v{jwIYv%n2rV1lWR&3l<%sr6R5oQF!-ad07r z&>eg%#svdngcUsW;jIakO6cUMx691|{Tt8Y*!2wacC%(_K}WEJ5JzadHuT(~ucFfo zLl?R+OL?M|ygn)6-5*WsnD4+dn>44yKtvLF0*<3p+)#lmIEtGqT_Ii=_d5;?T{HJd z6^Mi2QBq56JaL#>&_pwA{GJZ+o-Yo=N1g5~NGUWzGMb6bYbw6HOXk_@nn5dvqp(5YIZ4i6iCNYAZ!sIqaMz)rTgO|kB7=alXa4(foKTLyV{jGwMjsUfm7%6~seG3Rg zP#pnDu@a&-Gr0o7&9qv)@p!ZgZG6P(JAM+LJYkVRV8Bi&j_~jPy(1;A=U0LT9HXa_ z#);2}X?KB}-h)N5+!lLlwSy!Mp0@a={e2yLO5A#La=B@LqljakCqb2jPDm=%t+l_Q z?tAx zrML)hFlpdQ_cLi0H6@5$4{w=|?s0oWMgy);0Fv9jKC@|S_W@0rS-l_t&6Vl!+sH!LN1Ykf=OqL&8^*h{NaD!2YDECurk zsQyWFmi1a^8BgA75`%c^tX&@$Z{!~5%h)OT-`2^_%ORC1z@Cfq;t+;7WX?OhewQ07jQ&;hj}h0M&O z-^e5yd@G9kUt?7?6*LkP3)+~JqYVr1m|=vQ^Ot{)bSsqw!HqR4(@#BXCSMT}=a-22 z7}_O=qsXddVyWI7!vf%B)6l&^U2W?W^!}p0IctTQ&uHc%xiJ;zUvM&BQCR5wgbU9X z%2Weg@C8J&L%0BdXTlhc>NJACITe_{y!n3ycP`>j%0Gu=YR!EeWTD)P#un3)V~{>Y zf;h7jrS>JGZ^=Bdp=#%Yyqq_nvg6{^r%K)=`(Q3Bn8S;YiZG~=_9_QJmB|$_7yAag zRtJ_!{RudoKfm(O)ZtZJtcYAh4}$Rg^LlShAmj@fApvV)_UMKfzptm0rSp||rr}JJ zHPhhd&cKvD9cWNYnRwg!GT}^ALi$n6K(uLtR)})QOTg8V!qJGGYkcHT5hEI@1$So*0aIOKfL>=j zo6JR5{D*YB37Iy|e9a(!5F~rFtYKiJNuIUe~S*Zwx?f{xE9PPQ?H;Z5+@`%5d8mh5ycXdC=0xbV0aPi|S)Bw!qWq$qi%C3DG`hI!4DCoU37 zesoC$K^791N)ieMf3zeS>r@PTbt1?fIiPRRgpjcjkhnM?5=c2nId4SSD>ToANW$d> zlqe`+kMpqxWlnnYlkVm*U=B8|=bPQ&KoE5o<0-q9tIC3xy5w?)w@5%YN|xQvbD*Id z>cVKuA!3cmeRVhve>|S#>zIvJ?A7bq>(-UiWqK$I^bkjWyuK=jRxd3iDp(F>O#i2s zFG_yefg(VhB6|WmLgLMQ^w4^kqnO#bZWqDY6?jjQ z%-D0iyM#kJ`|8}QD!?4bVQxUQpwp{F5;8_DA`!EUKG9rcl52!kB)zRL)hH{3RRtW% zO@Rst&U5)xd~edk$EK9y`~{WKLTB=cAyykmXP$=EV3bm*9DZyfR{1Iva2Eu+zu*%- z!N&eWEdK-S{NE7@CXWAkRJjII*Zzp@fwxb{zQqxd+D}piK6(hXo%Ut^Fo3Ne?e8wI zF=H#1rW9$1<;~h2OnAz_T(8YMUuJRG7dI<{Mp7$)@QqI6SL{`%a$r(A6j68rPb|mzg9!Z-fPpcw{dnF zdg*ij{rTu&aJ~X;QQ%ZQ1$r)%nRF=|vF4@))e z714_yK3mvu%Rlj`|4|s2ToEkRJuAWZ{PH`zW(SdZW)w|J zfaJ53mr^?>n@Baw2c8>7HlP&nE+QprbI>)zmga|dV_&>@S0E-0;hdjwDVH`<90wTAqq`k##TmL2)*Y0G0!`1fAZ4f3|tTq4)37iDvm~)4NZ0s zcqa~W{e$dyJBpk!1dSDe4ntBubF;kOVE5|6&;X)V$ktkCi`@a*_@kPt)s&8ORl*=$ z$Zx+i2vBOBe2|5yOJj4oHn$aXOv-Fv1`RY+Al#7~UM_D>uNT9W^X&#D5BUPhPwyh7 z|2{f!X6BW@wqw|cI*>_I+fA4knbsr&+(mPi1%gcdP==YjNn$xGVutoWKTd5Xk_b^M z=m7sztGlEGCVjjdTyI<1kI%PS)FvZ3O3g0c+9u8(WU9Nw7>21V29w8Bpgx`wBaLX7 z+qS#K6KL>()O>U8k-y{c_wt3&53=5Ojf0Db8mylDVOtj#O0^5x3>#h0;JaeV#k6~N(VXx5HC3Q&|Y{u1%v0v75K&-&&L6kGiok^Ca2 z$}s5}3uykI>2q8$7GSlKCe|O6?tYW*I;YWJqFIT_iFsCX0){JQ!3mjyvnM`N&DM)^ zpcg=N{erNK8jZUrJ6T}NC7;1Z!x3IWjZ;yOH?xr>Xg9s@Cn+9oPOnj zv?~Oidz+(#YpN~v=FhoT{)IyKp$g1k!qsc2Gg2zt!VC&R3c1chD^>NA2iU~v$xi`6 zV8DA@s!!^fchk@cq^Ph%o0+QNX1bn@=!BVEwJWG`laY-Vy_Sl@!P4WL6O@PvAU!Rn z)IK`@QK(S4IYHQyLJI8!ofGbi!_1DD49RK7g7j`P|a45byzi6I3}TmU|nZsh+2KiU;foq zJ=zVd0;F#xz&@T|U-dzY_0uu{{@M$Wwa2ax`cY6}51Lfb7|$F1m6~a(_Hfnc^01SV z<`F+SyA-=vK&9XZ^(aQFRY}Y>(=A%`Eiaf7bq)tnB5!7QST#;Zs{ta7fE05W5O16|?N`1+^P0y0%G}Y;E<#G_nWa zUG&)b6h{@bqawhe6Bm*?5wt6LDyZ^DG)1d7YnddFmK54Cw>^ zde1!ui!_%Z{pO3D!td*n^pjLgJVJ)74;^3)i4udRwdGrd+Z=YW4Le9Jh8iCgl=NQn zV+HYI!4rnML&}fz3XdhVQ}|cWD}N0?4PEWoavdwOruNx~(mEDG%g(zn-dX2^6XBx| zN6W_F9*{?(IT%)(L$>dQLLtVc#t|#EVRpG(C16Ak0q1Vgu6gO(XuSXSaAfHG?_U#=!0VN(3a0qL^!F*rkO~?)?l7|DhEX(EbIqMd(CyyZd*T0ePH4sgv z2Nn_}J_HAXkkY*zA$_98Pb^DjfO|xQXhoF@LWZ*}?-_7huP6&`^3JlW#zZplES2Sh z=HePcjC(*HW#YigQFUvkh+j2dRq=H5PuyX0W>1K#)6yV4VP$)H|aC2gF&P6 zg(FQQu~-7Z%5N^6Au7g16V5}H61GcA(N6gIcfSSHBW~{QF8ToBNDIcw1FRH#ZTP#W3@{$K?_S9=58~d8Zl;tf6f>j04i0E3P?%jv6 z!r$m;e+ZR9&!B@%_*KN>Xy8h|2+(_G7hF0;iNCK=%*SNEIUea`l6~1>EZ^kwYkeNN zct7#F2{-OI0!-xPk$>h9buXEY6zY(=`iu@)v&Y6-{U9zNH(}4&N!~mOK@-EVBV4E= zeB2}6Z*q2O+RR~a{jkk#8&WI=*Y8&l!2vk7z{9e7x^8U^poCQBuT!hWon5Gk9znc8 zXgjS|6g}bQaDzHyKS)TD!Nppyhc7k+2mhK<$y;PdzS2p7b?pBPHj+F5$J$#lrOC_b z(gpmuxTP!EIh2 z%5$xvtg70u@CoVNoO}NU0JrWH)pZJUy;rB{nT(P8>Np$)+oSS6DL+F2_KSli`ds~t zDIl5fQ@ro~N~U}~FaE{xykth0<$fr~XGgU2npjI;-2<67N0_>b0kKn~fF|V{3Jp^| zh@Aeb!~44b2&0~@En6tFF47kc+&48ceDc^I`f5T1i9H0*f0J$Gc%EMwHn~3RI|j^8WxL*`B>N_g@`<*Zj+c4ONOpu8x*|cGtNs=e^1YaY_-)$>M^RA|Ms=|PIE(4;$^x?)0%)r!#2jB#85;ju2f;kEBzpid2K5(Y}sswh*yUpGu z^*EHV7IF9rIb+#32Xd~U^?_1_6%K45=y?n<6}=V07WA|jfBTW2h(`P*q6IblvK9Bp zve(oD5#x2EkBx=Y4DDActW;6lU}+OzkM+DF84|*tV!BhVqA8Z>K0Q0AjyMDlUVU=K z`t&&8LC|$>J`=J81_ z7eknk`?2!3o|A+Ea|o2oE_HyF1!X#B*;w$w!-(UwHkyXp&PFS|Zy0d9v2hgeCwVxH zLnUedD(L99ic;+bl1!q0d_Sc406w2f5Gs@f1Pu_c)DSG+yXcwCy+6GlPwky@$E!4|M5NA3#I3g(8n^L*XFb!y?L4qT&m7EkYSbrk{ z+y7}OMJ%&3&>sS$(u!mCq~F6bSGC)w8pi|_JG+tuc8fp=|4AXMNK+~dx2>Y00~zKA zLrq6GUjC!t&eKc4+)?O}J@_~51d%;dGfrUgTjE2js+)3+&^B9E(>3LgAc&UWp^n&G zo@PsFD%xg{Mc2#z^k6wvLm0nmZp9%HyJxS20M5~w@O=K9DcE; z0!r0lq79qDz;9eI_m_4DMUF_tgldIYOLs&_Q>Xi|y)6iL&JZFwAD2zwChbZ~Vl;p^ zV8!?q!Zk;+h^}#Ww2m>BDo5kTUd0evp*EW5(C6>myyZ6IiPAC5v-35Q>E-k}VnbAj z5|vG2C4V`SQIn9(*plR8ydZ_6xPnb8yY|T~A4q(dT${{G?kL7>$HnW8pck5L^m6P8 zq<)7;O>?d@Nc$f|1JBfNmRWH=*QL!tN;_M_3T{Y&(hX;BvM*(3Y|}l5o>$`W6t&bA zj{}}}wZWu|3rv_rGSP&4WPkbGi!OEOCE-NG7u%{v<`ONp! zvKHr7YM>H=SM~K=^^1lMM@h$i)A-BS0v7{Md&?$YtjQQ3^Rq(4)&PlCbDlyq^jypA zQWk3z$?B4jt$?atV-bP$#}?SVu8UB_F8i{f3IHj54jt~)#R?{Mao{&x3WAgmaaj^? zIV4iTzucT8y0@bGrEZNMm;F1-1UiSJb=qftJJksAFI{`U=2jvg5|~UExygDJemOz$ zJ?Ua)zQvd!N#7b*&UK%^UKErM&T9R( ziZC0}cQ>WY`Fi_sp8){1VuYa_v%f0UH)ai6gKabJTGzT=`Y%$)Fsk?$yT3mpXw#bY zmR|i>5KiC+8+Bfdx;_Uf1sI&aJd-DJ%As2xS=|zcJHEc&zCgWNSLC5RD6xTauuQGr zZ||k`mf6kOWC`L}Xy#A->5?@XxCXqoE*=S z+~6C;cmWR}v*52_Xo1kYsto;Z-DQkbY%hlczqEq&yV~7yCuaSCvT1XgZl#qgfb?o! zuQ>U(J}Eog<=QQLQB3I!zQ)D@Nx6SO%ESP-J8}HRFkfvnT}Ii2sljjg zhYAJ-*yafhJ&ug&A9Q$@s(F_n%-S2qOQZ;V|Sw$-TGV5NIS7>oBgcoW`a_lKWN9FqXj+Bd#`u{?C&lQKG$xX>AYBStEdkkkze zZ#^BT$uSs>-ksL$5?zNIu*P68AOWMqzvZcNo>a%~0N`!(|x~iKW#S>e}9YDj{ zOGx#Enx^pUEIV;Yy)YErI|cbx$ETICgH2q})Zv`;jXa}kdNC*{7W$j_Bo3wd72(rn#6*Og>9d*WcVGYii$ z&!31Nbx|f#IVf-cRdRF#*#JS;Kjx zw7_*EPa0|Zm8DUMA-`C%uH3fdv&DDC(tcS`S#<#2Nd&0=B&^P>6IN+!aqM9U^;^vB zvB34dh0i|_+SjN<$vTyyv00-teQqn4uE@?gHA_rZJ^@0pQgz}D=sDwNrkS)}fKY8@qZ|X@@k<^tcTl!|;pLLUJy1g+b0>iAbTWv$0?vtzU ztYy0tR+$fk@91bd;18+S{j7z z^9qqM)scFoWb~m>swzN~;qp;Sx_VlW)>Liqop#)=-Pi>gNZeLnRBCrp)*U~wgQOdH z(ljE4%{M@MuIE!SZM(a_jTU@0ub^3tgky^W`~rCWA6-=b<-r$ewVr$Wvwk%8W}A{A zN@o|jO5?*U*?Ue0qsJxZI2;Ac}tY2ljV!Wd;AF`k`U!UB7hxt4>) zi?djsHf260T7hX8PHoL{@4nV!zFV0x|q zV56U&Y{@G(h884|uT1vy%F^nmOGeDiV;G~jwF)zY#3^1<66%NLc!;Ctv@@C`=`w+aajf@a z7~uwKTMUH1xU?sPY6S(|aiAY(5-{vV?cGl;mLq%T+u_;Sp)1>YmA28N#T@WU+`;jpe74 z1eh(M0%1bJkH6=rX8SMG1m2LaA9&KiIhi|Gs0cv$kwk!0wP?GV^qdp>oQ$X0&J*oP zj&jX@)t#_1j*!RVLbn%6U$+7Um4+nWan6x0SEG;q>N7{Uu_w!jG>^DR40l)_#U$8| zxFMYn6~G_z*BJ?AxqEFNE-FVE6`J^Bka8E+sFol`6#`z(B^<%TW#k2r#gm110XV`$(XnnG_$ZfdVJL%y}i|n zM9n)4hc6t@Gp@V5qlp?qwh5X3vc8-#xJwc6^Ly{TQ4+71zg|qJahdbKB0})!#plB< zihTL-HpE`Ti5xNuQg!g`3^*~nxdXy8a!x)qZB=#r*9noO)kCn~czY}q&M!1q?;zl;Fh`te=!O6&af>KOjDWL%J$az{lD z6;b2tWXLt(a(cJkXrW_SLCFI6Wz0f=U}RCOuPo_Rjxm&#Mo?;i$$ATg=`#h}C?S#V@Ka|LIDs8;qf?sx=X&CdcgzWUa8XV#@dKwWxR6EZGOybjMZ{q<`g2y%F z_M7$0XyGE}SUUm0m73;qF&>?5)Gxj-H7eX>+~0AArj>!3k2Wa$nWDE(%%hY}_4q_Q zX)aPNVK8GOjan-{v%cdBYkccqknBU9zF?iM?V}UaO`A90XP1EK;{c}S`Mtn_|G<(O zWnD+ykeIkD?q8^)nWx4nSW1NB_6|21B>Sl+dIX1O*mop$rojiV^y33YB@TaXWIs<2 ziPOnkxq$lW2L-8i#_x0(l6(=91Ji*CHYm7(W2Y<~VXf6W&^z8|O) z3K(%R<76lkZ>A;Tdys{Z>&+ah{0~9L^dEwbnel&5&Ry$BCU3E!^}g24CssBfTt(}0$}rul zlv1f?O5&H2O$4T`+i_`eC2dZ9eR#$LkOlxvsA=nsPK*@yho98+f8uSrJU9uF`}Y+d zyw5zZsLy3)avZX(C0}MklFkm2a~UF&fh5`#rV{FuPdr;aI6L}k%bIUz9QLiaF>mdC z;GcK}(zm{aLUEdoxV63ZovnCH>Nf19Ny~R^R@YCUed_-0gaz&m2&ZPdPWZI_?xahX zuTMo=xqW3AtXuniyWm8BUx6Bfa3#~Fd-VO_K$%6db=?Hd#LSUo62|b6@me^@BsmLF zN_~NxwyoO1?cONNy1iVZsIz*vvr-jwn1-w$A&$kcCD)=FtieCd_D*Q2?_Mvmx-(Uu zVJ}B8%v2A6gzAqM1e^zHf-Z~NTgCo*H@ogj?>e%86^|3e5w%8CLOxY#Oj|o?rCWQf z0E#s|Mtp4zlQ>V>&CVup$A0B3pF|vSSv;`wfRiLu3Li4bHibU+v8Q>4EjqM(WF|@=t-kC5KI`Gwb2L(+Vt72dZU5}vF--vk! zVGq3rQ@YS6;{vsjiW~xr_yGVsjA_^qPqq&kWC3OZO>h)AD$2qv71EfT85aI%LM$Vl zv7wn(YtS_F_Fy<;fFT#rERG<(23HL9c~m%+S18Q1Ad$VIV{gNVgP}&g6a&I!*^jAe zURee#Z5)SWRsalYYQa}Yml4E3hF3^$>QY#z@-#mb1qrSV8SKmZhm%tA3Tk+lSdlRD z^zT{^Ft-q4HCT|C!3@|_3puJJLhxOUjkI?*rVN7h;XtD_5Z1R^UJgCYB81HC)pQuXGyZZ&}on%^xbJOvxatZ4)%blhP%TsbD@W!l?|>%@FH- z8Bpgpc{2=YkdRAF5mo|QRx;a1#gZC40^3YyY;KC{u4$Jjg*TsGUN^AFjHp!`WGvUG z!jUbRdBys&3lmy4Eb3hQQA*;;Krc=Z@>)DIY8s}rFcBHED2XwmnbYC&=GG!e%=i-aX4g z3Nr0ywu81Y_Y8ie`2s}ZvyOn4ldu5hMwiJ5NVonl}3k;czT-VV)Kr@7AtCdT%U4dc3AOs)c<+3fEZSvriAwyi3k+YjIxkBC7HI%wWY8B9)#Y;PV%!Cr+BQaits( z$dU(gHok52YtRx6sI+LDdjk0~f<{&`W%??0mrnZjt3%Od^%jsYqq^XXq^B5&1R3o^ z5Tzl3iv6=F?u1L3p)fDrPfD~w86u4OYA0^*v%em$eb9Nqk5PyE{9niV^BC~CeA=of zUZ=qU^2Uk7=3`Tz^DE#$3sU^y>XPJXxTW{u92ajNQaqd|4;06FZ`4qhG z_r}cP&6=SD-Uc=kh^vr#D!q5quT2B|7THRIAqJwMuA#2R`twLNpnZfN7Z&%|W<6Et z6cELf`f1ObeO}by;5(|H+mNRL=E7MM9-Zacx!pG)QBeg9(i|bPn8DSmWD9IN?GXUH z;v)28LQJGeB~#d9(*uC}G)XQPk@*_J3*Ad&!0TXuHqIdRJ$Ne62Dfsq(~+e{Kt!aQ zcf@*~qf0RW$@=P#R&BvQ`NL#l0$bl+y&HB_rjP@};`}zg%Eqb^%>&5#Y8IBS(lLw! z7%vA?@eDX2O$AawIV<}L8P!)~2gwP=4papXDcOWi=~4_Je_qb{0mfbiA0!lw=;lft zIM_#&eJp+%u8%V9eX<$lU?hlIW}hK$7yCb+A#ZJpKMLp5Y&KziR0GNN&g5Ye&pCLU zgXdwP)j0B~)NJ5D$YS2@rb(B0A-T6^nS1*w{9`)jmBMIm z&6;%Vup)1A%?2RhxbatDISw?|KTc)i*Blfsp0U<#O1x5VxW(V(G<;9f|6nIdW+b1P zq}GQtN`>)Y`J;3382zi0;z=QNn1GnOT`~42W5eLMvdt*macll;cuoWPyfc3jci55odNTon!u%8TzX2dvhVS>3Oh zVO#1whX|G>AD|UDCOfYIYG|4gh2nVO-rcoG!Pq3W1 zzq!X0P*zN+!m=??5h?gWOiQ$$L{Ty)y#Gm~#XH=w5?-G53U=}uHDM{y`iF9H@JqAs zoNRh@wDA01`3P^n{30-}67R75!s!1!fa0Kpu#Vt)STk=O<2DZ{MB51JK&3d2p*MAa zrjN9%V9}Un1v|@c1ax(bjfQ>T-h7kW81tr7e|-HyeD~K5oV3AZ8$~VbkrM-SLaOX( zOxR%%X-TwG|A`ql2oZ%O33Yv9X41|y5`=Q`eABth;wPEt!v zXn|=*o-)A{GE%$|I*R$p<~a$aQoN*;il!u{mAsS@9I1LC;uXkno8fU1J?h|&BtQ9X z(Ei-PD65?x4ugzkR{@#*Jc}iuYu9(DCcB$WC@qyKs?O^eT3J(geIH(cy`YK3!&04< zsm|>W{f2+{2S$qsTwNx8!90S7w`i5u$yO6(6Dh3i`LKaIEh3aCq7+hc4GI2;Q?Ome zQXEE+L!<2~=67N~kTFn_ClAAY&@dRJJnk<^2nVgE6wV%>teD+RY*Uj7IjbkH^tY6SaDK3rA14RKW1)-eMNuLN@sdVC`M2TsXGgTpWesdKuZ?o5e-`n$z)%UI$!Va^LVZWa#T<#x}5TxH@z z%J%j|om%#r6VB984-{OjgU);7ot}4+X5X2s(|gADX|!R&~?OX^wAsMuSOaNV!xhyPWb8y5U-dVL}E8w<1*E9Y39Ga=R-DTt|O2MGqxH@$c<0RKQVJ2lY)S0zgZR(r*-A2`2 zaa*bM{a5~>JzKHUoBz8#eimxItcSXgn-I`6Gy+qhWazm@EbE7_6=n}0|QsqCtvagq%De2GR;(8 zQ7rnz<<>|N)t?XvOhp55q?9})$pX^#hOfT?C@Wdq2R!SU28o$O9R+qytr}DU=1~*? zyHA5jNt~cY27@4}gKRK^wmX{++0nr<6C)a*Qn{BSauuy+$iyroFdR<^qSYU(5N}grAQg7?*cijJ>RSO}V z+bB};D{(n`yw*mUx8M)}G^jy5^z)TrGL$*@Go}SZn80{maW2`#$0PtS=Fa@NEQD82J81p)MGyvFcGzlDPw_7l|9x>r&T zJ43*%8`5Jd17dTC)186SIU}~i2>;6`*{Y9}TF?N8=KQwq0?Kb$>9W}cw(xS$ucO(- z+e~E-zo(L9GBlAE<$vX-^%dh-i=oVOwJs815>waY3sZ2~Vk;!3?8GTDUn2lw05kL( z(vFBKxyHKt_7<5TP(iSTJg2iXG#}6l($I+we`JuQ%^Df$l1<eWus^FeF6s%7ZbIXh?jW*!5z`{aKl znckh)eIfwUJx*AFEvq~Q4$?!T1vQYA#mZ`jCxYX!$v^5KQ$0HKnD>kDDJ8T?)&^4$ zmG#C(>5vP2r2yzkKcvGKDHFw@g*9HvJ4etRNr6Fh1*HHagwR`wccwJdFgC|{Pg3RX)F2*IW6jJfd%QsoNT06-o?U{ zw-XTuO%Qv*E!^n3!0loua12RC@e4)Fjsze0KtMOVF;lhH5puMTdUj_>^t2Kz8 zG-5d_1XB5kIl6YrkM=i*UJWnME16(L$xJPFLHnd54`Ujn-cU*~z)33SFc8e)eML{T zyIwtYPQsk*%@MVH$xlM>MD?Tz>3OM)!`$=lb6F+>tC)ZqQ$ldesiB!lH=y>w0EDT= z6I2Vf1cYrZwAp~4u4`Df=jT4sm8z=E_I#y<^qN;U{n0sL&n#oltT5y#+}rl%9`8Dx zHtX1^Ut48%={AaPhhJ<7jwxU%49*(iaXrD{GPoyAgDIOf6;VspOE1ti_&RC98#`pOzEXiE5(z)(L&-Xsev0jf z=z8gNg8iD*7iOLhYvMvFO+|=8+G&E1LZLuyDI@kHAmit6)ke%jX`E7foy<~SE}7*9 zin*!s@2iQj4pH`DvG~}<>%1-WIAH*n5cU}W{w?*H#lpSG|kT^iYy^BYI7oq*@BFHDhQ3oR#@yp0jQS2bVNdt zfxz%l_!L?(4i~xyjmd~lItDf;->*{Zi4*4Dg~e# ze+I$}A|OzMYke?OI4U=VjRL~1}lb94b8VEs}) z*Zn0jmZt#7B-th{_(~`f7NB1w5wy3eHfcI7GfQu>Be6#Beo(wHQAybKOn*DaJt_1& zut9265w#5XHp`O$-3HWG=Pz(BryPV!6^1Rli8EN|BMNu4aRrCwTg(y0K2pt?!j57F z>z9#2dhUO>6k;gBFt)%pNg1RrS$>3@W(anSJSfL|cI=|Bhs+A&Ht0c4_V=?2Lroe~ zf@bY$TEQqf9#(uUw4a$Lk4!JQ-D}4ffl6Hrb;pyMr;moR6*;ENS5lnj6T7fP&kw4N#{_g+}GO6VIV*VS=N|D!`HXjZ zw%5b`9*3y!cIPo6%v3(IZ!-~rR#cuX-(x&%hb+vej8=vPpMS`zKYLZgh{L}bmg~8m z5nPbOr59$^PlD!^^{@;J%7MwYdB3)_rP&gqLZoGEjQZ~fANv1L_6|&O{kEk+U_(6BF9IF}5~$&4kS&z)h;a4Z zXc$YF6jhk_Y#2auPJrmu>?FeKCc{{!&9iI-Ofe@eu6DnW`=i>MNu!%`^BZ*UuG5NF z6%{Vs&|B8c!djbF_njNE-A^zQI&6MQj8G~cFT~sve}_sPvJ6I7xPXB`tc2>i#390?nQn(QZfw4wqsr_8kQ}QV#UVb9zq71+<~rBiK1qVP1;4G{F1%Tr zea?Z&vs6$|wpY6aeQxhVr5`8=se%a(M5h+y!s{buL6U4&*L!Dk z8gh_l8cxiuAXiQxc(NbgGgW@THDttW|3RFy{)f#16FcMoSe*YuN7!$T|6k|`ivG-# zZFkECc-l-|8$}e`wf@reYhdC9H0%f$2`agdtIr%xW)w>0)+LjG+bGsoPUqXbf~0dg zslvNDq~Fi7tSb9nWu`>Zn;f7M-KPUW8kyH8auvo<otjvFf{TRtmwz8&vV?0XY5R3ZtL%Q-YiA=*h0ub4W8rpcE&$6!t1 zIuF9aXX$Uf)a>n*+s1>ns~o0SZ~9I>TqS6c0IZ^j%p)yitkUXKa^zBBAzdarx$BkG^lL!r(Pg85lD(C zng&QDh7OHVqIwc~u1&;)%@I{`uLQJlRWtkPLq+&%Y@q0|b_pkK)@GQ#>85Mjm~KC+ z93LVlPrf-C2?6JrfW%w2i1?K$Df1yAYqPVGCH1SbbN4D76P8!y364WKwkN@&#M*e2 zUV6S;x34!)bH3+(ccz8tCWTDYO6T>oOolP1*n1>Yy75O?hf7#n*@b>j)C4$+CR}Bx z990YpT2RXF&VT7wm1j?c&=yQ!Ym=l&s0v7ep`Pj83$$IIFHbMOQ*b!tMMNomk>*=D zb-ur(k!%^ou?nMjJRPXv^I>yHHpTh%zK66QpB#S(v`OMk7>RvfL@Ek~2>KAH=1%Qd zl=U#P1Ykr#7pvwYEB1WhjhTfKT-%T$}G7!la1Mc@GNwJfNh9Hk~oW41MJlpaU zts!kGkRS)43X+o5cWE*Mq=F|N1hyp49^ZgD+s%X{^=WkyEqQohv9CmFRkg}VjG3a% ziLve4w4^CwWmDY~Ad@nkd2@27Y$O6dG&}N3DkNgKYCOb5XGCFDLxMvp#xI4AJKlj% zd2^EcrMnfxAL}ZM-p?-2qqR9G@~+N7A*~Y3UqWI>rzJ8Y7?71VEgH{d6G^6mH;8h^ zKcaf_GN8?Cy}-UP6$O9IzDV#e98{(HX;ypez@u^S+T%nH_%)_?~Cq+GZA_)~R7 zZIS=fM1|e-@3i6IUY*9K>B0OtAPX_b#$@iRz>mM^r>QBFD257Je=2wZR7b@zyhw}v z^SGkK^y?FHl0@2I)|4{&d$jxgXOy`$Cv$=hf3d|D485>Jj8 z6KbXVAm#x=#azyUR)%|@&6?%QGfu2;!jPmNGTa#uJ`;uLy@&cO`kO(>p3i#(-CJrpVWLf3+)*(7RppH?AJU^5dyDycSIG<-d2V#pS)P;~@ zmdo41`LDBA_2aYTJaZ9zmi`3QS7a(dRaK;dvDv-T{R2uix)I%^{yHoAdf6@46+V$Z zoh}0H;r_X|Pl1N~%59kW6OQEv0e%PX-T8GmmE>YK9&j}iV|CwBemgeR7FW#3P}X~&m4Mjkqk_C$C}v|+-ZFk+f7J~ZNRAc6 z(J?F&HYLEYG9hy-8j`2pWPg&#y6Uja`#i9+cjV_iRyk|@%%%t7GnPDyP*x^%GShH} z&DPNR<|c^z_ukRcuF%ox(Y>DEVdpkouXdA|O`=PwU#+{Se)dhjYw%mv8vEeMm@Avj z)0cI1yD*nn|1r%}izl0&Sg2}?N#T~o%R&tk6RV1jQmO2-j6w+E(NxJ-^4v}HG8ZRA`m)7CRV}AX>OEf=gNbX^#r0>kf z%#-)l!%i$>r{MzO`uk^JLM*#!)Ud86aOg~y^<9tm?FAP{0Y;3=xPDc&eqi$N0@Y9; zM*AxOh5{c@RPGk&nH5g{o`?dAZl8&NNjt!P)1uD}A!6$@FuR=pG+2 z^;ELRSkX(9(?SzoK_D~Gc~jxuUabucG6WU#ky&n_7oDp(RT4cxBsbFehaC2fN`z6~ ziJ~q|ua1hNK$vqH3ID?gA(8O0+mr(6xb=-Hb)dkyQG#a@!8@`j8Bvnr?RR+3Iex#h zPKfWqmI{4?^2U}^+D&f6&stzj5F{=7^V(x91OCq~$G$Vu6+9I%27$hiH9yiT1!(tI z#2`t>I5KnDoBC?x1vy1YeMh=JJ|4^{&wbBSu|w^J=-@B-7(3w<+Wuq)4(@RyNC(|< zNU^m_(6dLu=f~K}2j}JA_#l4W4iju8_zvn2ondcs>1!CWdfc^7$T5;=jw*pZ%MQQm zZ}dPzoc$jNZp-@aw@?zn!V~6UfgPAX%YaBKxTHI_;;6#KLVm{!l4$rC`NA}AHCBDE z;5grRu|+h0*2P{vE$7N~2WUBbcRq!#UfMYgi#GL@*wf-urzwS^nXD#&6(^0z6$;_j z!}QD^-pHZuQR|#s^=g#vLd5PuDl*v(tIa*`(?3tSk#nRU4O6TJ3df1-m) z=9r-~5+3p<=U0AGEmlOof|B5rh^R1=m?^tx;bo{sGuL&w@jg)^GpeD$xfs0?yC0~; z4KjQFES0|BP4@cFGQdh4SFg#A<={6+I`zk3r}1upAoFlUh;8}rE$VHg6)zHaz^shPE({wbF}f}N_V|1NYHS=jzd zo?gYn!IXer-pESX#TJTQj)0MY;eSI3oSa<M_4oJQP49V~Q`SOi`nH&OIBhgSi4lEkt)&O-QP%KF&5ONL? z`KGw{2@(S%#RCmSRLFJ+zZ6@zyXb}xY`M^emqcu23{EBnJgpU3UHZKR8|cU^1O~?0 zvD+XKaCSpDLs&>^Hj>;NgW>D-+hzhQa#oI0&U<5+YmGN*Lp-sQ(k%AIgogV@gT}rW zQ@!i-!*)BbAQ~#-PawZEXL5#y-TLuqWW4mWKPlcr1=# z&I4R8sm8Du5Bh=JT~h`^_cd@-3?V~!3j`1`j9l;*F^@1vz=RRFAD_pDBUL)Vetuwm z0EH2uJRmLbHZ$R0N`z`r8x@24=yfB)TtlUSgtw#yhz4TAziMtgXnJl0oERGc$NMJW ziNd9jj3nYVNkPXD=!na(ke%_wsNBK<_=*<7`(Dle&P?2qwh)a(6NmW#g@rZ0vtDzGM3&kGQ zdYb()o&}kEzQ4c;CoaPa@#Ta@zK)(Bo=(5L2C9!1j-48Kd9w0aBUjqF^6FJ?oa)WJ zUk6UlPwuR`Et788Nv9J=y-MQZl^%_UIK018Zv-os$D%HWT0res?^p4UmkY0&w+u>d{yD^Y@x3CtOED5bx^oUKQ zgy9T2hC(e`O#HD2u;Dw#x%8bE^d?YZ*a%K&KboFYFSe-jp1#4%wb+lHC;EZA5fb4Y z14uu8^rWStxw|o@K=a(%_lE>^Ebyk!pu;^^REZZ(HWPMUlTHte-fMN;8mF&L$4)gL zAI#INJF79=ULNKV5oyD)f1*<<<1&(axR|8E>(G zE#TAoTz8f6fa2n;1!e|P&`RZo=w~{gOq{2tHDy2pR0T7jmxu!BhMH~wY*DK~SGaP( zT@n(h{9}y&xe){T3>aDs#0?Q^VRx55b=Bb_pSI>~pJ6?vqj9>-w1#Ha2by;47UJll zjsbrTbtG_iYP^gI_NJys1#BaKw+dY>DE-Tz1NRvG^)$Knna)I>+>k21Ak2kOuo8MtsxmQ(g6FcRSRv<=S5$N43yNf7V!R?Pvixia;+wM=DlXQlNvd}Kz1bJU6VHn zj&jJOpdA!}vG0VN4hfH`jGOMgx}YLO2E-A}?UA5d9HK|K!TgbIY2I#)oq;-jZ360X z<1V4r-dNcURYN++dOZVbfN>2AMxYJ;@|U(ts#=okQw2OV9a^UXv}xs`u@Nd-Hb;mQ zL)GgHbg+#o%rr_fxw9(4n+CmR@1XFc;UBSEV`zl$#fr;eK`k)IWK;*-ES5`Eh?c3S z2Be7M3V{kAK158pb49L$L`*W>=g8y^cQzN#+C3CgnqiX^BZdqESD+rL%OeI|VO)lG z-8}ipqz@p~YE4#qk0FM@JFM0=V05Btgs)0_>u2MOW_`<+8$7FJzs91`DOEOO= z2g{@+pc16$b>%{Obv-a?WkXeKsPKw)R5g%Sw``j|)BSbSdZRfx$>CjE1aoW)hk-42 z$8PCRjGgw@zjOq5vAJD1GnKF&c*r|EWgnyY3(1!_ooO@wyt7#-g-RJ}6J{k}&>>!t zc>7(WH_XjTT2gSM?oRv5PV=6}sWCOL>{Wg-?~Hu~)?tXa;9nymF`-mI((kMRtP~K~ zCCKl%y+m*@wuy`Pyym+Gt2{bAI#ld;?f0h<6>tID<|6M(*7G>N!gDt_ffm@nXQ3%X z_YdrhdLXBzH5BU)a6l7x!+)RdOl<#ky8qvaCuX*PGynfQ|KGr&6>Z6+?N-#@EA?jk z>qx9oI0ZV~YpLsh2Jo`#%66mo{)+;ZW-rWFqL_!PM_vG(e#0hlytDWz1QTF*`#t}I zs8{ITw_8y8Is4u2w+C%XG@<%Mj1KolJHlk6*^dy`SV>bfdpmS~bpG5*WoI7NtD~tC zDNbdZ*D$Wwm##M+wXILt4>f$a&!LAjcE8^T?E^GY>Y8Sdtfoa{{oejqtSyR$`bp-x z`zy4quOdD`f3%Tz33S`B_{K>!&63_D7zObS}3(ukB-tqQ&F ze(UrBb7REXwap15uU&6B6iTL)E$la-EFiAz-_b;+e=1q8&s`j_CTaWgyHB(1}6onXX?qLyPN$ zTr4XyrMU;OEZLSx_Ag5y&OAwCqNnK|r_C-B)R<|U#{^z_-4Ti2GW`f%?884Jd;dPk zw+Oco>{Y)aeZ< zn23Nd*mW&~zO`&Tka;o~qWFQcH9Y@sP9kx66j(65tboY4?d@=eY!L@JY+j9FN3f*U>T=vD^qJ z#`H%nj8=e5ARu7mBXh=r85}AIdZTR9(M`@w85mOZl#URPH1dLF%tqnF;6XWoHWQ>v zu52)I&UwNZ3(zWZTWWgd@F1=TO`zt~3}=9J-%Jxce)14++S2SqkY5TKZf0V5C((~e zYNTUt&$!CTkOc>cZSksi&n7}v>mx$2tMf>TRsw)6SK)>c)P;K|jv|)26*t7vAKzLw zWwfn3x`IkHqnTE8XSxk;hUF7zy}MJ(WKx)vxNwp`Rkl}l9`{4~eKEkn%VSm+6B&^q zf+7M=PdIs-LRO9$O*GCcz_8^ivy4aLD!tK*?KOWsQ8YP?=tW*FLnpK}$p{-^7Qc;a zEog;=ESljQjY4VR>`bOp30rU}Nc>i#qG==PSvp)qk%Ll}`m zx#k;*n;$3kakv2=YvbMi zaPEN9aE!=lFI0f^h8+VN^z8cwWG_p@zlFc>C!5LIwmuDwxCsu+ zcOdyn*(us6-I$QUbK~;9OR_iRJGhLrXJ%N;!FdpFH`j?iuz(0O?$h!$>F}6{WCfzk z!;WPbE+SSrYM>aCX(oNT?45!2%;BRwS2NU5EF;%aM+AL2)&@t`L7`w-;g=uBJMlYa z+0w+17rf~vU+Tm^Xi&}px`T{ef)it{))~(U31gdF0ejjzJwDTz^bc^wz2z>nGMv;3 zf2|B!h2UsppUgG7SYBEC1M|z|d0B#I=ISXNPGIkG}JJ^ z8?kqAaE_=^C7&jxWWaEU0ELeeoBo|r!oojlypOAZrNi#Fc7aC`EPLLaNkX;@*`I*!QCTguBc)LioP@WZ0Md5(E%ELoK zici5aXwK}$q1Iib@XT5E!;Wg(*4@$ROl<7a zA6xqQsl$_MZV|8}r+(!F0cBu?kiQ4VkAs5HQtJ+MPvoJ%4r-Pe$j-Dl8WYDFt80UR zxDQ^x7J1eO{feG;kER1}=U{=;0Swze^(SZX{6V7 zV%eU*ab=Ix?H_IF#Qe<^C^T`v_hBtv7D0>CJ)jp#Oib|=p3V;=@N8(badF5q6te~B z!N|{x%|F0gLd|%NFyaQwlCLXe(&B;#HjE^mE>;8D-bN9Myj7}}DL-iZ5Qm!!(4g!B z>qk#gg>wMQ2h4{2Ov~(y!i$CA%jCZ@sRRc_dUl@qb;fczmA9MMy-UwuEER4~F^=qt z0_}MEtG&z%9pF0+(J=lbNvxTAI8BW$EX#J5Vc^Ga6A zWnyc|VfS_+17WZff;TX4piC~sWaqkzIY|U7h8s~w-Y?+q1MC+m`IpJRy^w2xTyHf= zOw0qCV>6fkRktod$~xlja5TPUGqy}1-K(p|+zkwZRr_LGGv?qexxY&dxu?TWj7&S~ z+j%#^mIW%E4J^AxBU$NXfhssSdrIBAjQ72fUCW z;SYmusycm0?j@@$(XAWo;+4D8d+}C&c$!0quR}4+8VkqMY0#&`QccmU?E)xck#qQj zbo88V$%{iORUY?_ZDOcMCYH)vC)T2n)vNbzP8A{Rak*rK_;R=hVmz1gx^QbW2(`-s zL+AcbA%I+sMyE8`w(FYN1+!9%u(q5m{(9Ot<3O|xwLrR+ArWBcU3=osU+auHf4P|A zDWYrHMUQBbU^sD1E}SP%WILwr%q2F+G2qDHG8hs50HT@Td36e=S-n^dpvzKGj^zYf z1W3sNn>H1tFaSf*WS9DfY3(3L73s%EmiD{U%Lx{9M44ILE`f8Q0TLcIQB-E(-2H*a zSzSVzdYy@EY+b@>d2;|agApg9KLT?8GU^1>scD!cgD2}>FZ*@Mg=q?S$oAl5deuX= z=kQZ0eHzM2@0O#6M}xM5U#qVa>Lvb+>B+aoGJ+t5F3e^S4{H<1>VR;uqWp?|>+zEC zjqnp)n=uns?(3y0K-kw`~j8T=Ga}BmydMP?u2fKy!9N3%|(NHVXT{C2lT*K~2=Eo8Rznr#JR*o^)GwOt6LR zj8LzObJ$cr%4;OIDro!$XAV;erPk)@8n$dXg?vWHy2Wgjch<8GtNF=?MWk~dVNrP4 zQO&C{0%gDL5;v}s#+$XSo#GdrY%d<@8mm8E5wh&}_remcDcBgPL0TuJ=aJubkCd^y zck)GkPX3l7ziWWzTYei6QHQd}FZxYjxKv{>5~318v*KFC6Q@LAKj^^hT6_QDILh&# zEYHkL|6?N7jh1E{u_%)7oxb9ynY2pcD_|yiGt44iKTyx2RDaOeOY_W|g}2O5-`P~r zWQyB|MVmGZ68+9eW%)QqZ+LY0zYK&*^T%my>979Ip3hHe#w21{NA2yNdNN1VeY9G& zkq|5eJ^X%jecuEtZBoa#x*fstSQ~^ev|C@tkreg#H?k#JyzWLy<2Z# z!=c@uT44UluiX!jedoq3X=h1#lGY}avtQj``?N}tSgji^TzJ<@9#6eSaJqjx{pRJb z_v$Gj+7)U2e!ptZErq_egSHLS&*|DD`HghZw$1Gw_T(&R7)@u81vW+B|04HPu{erp z$lF7z7gk~>$f>;&E65VB1W;p*ZX;Ydjphjwty%Hv@`yl!H)(de>(Ju!LQW5RR?WC{ z+o^Ka1}G#m9MLTb@QDK#kSc1KMzbfaIDIl~XtIs7zdpdNZmqcU>AAnztPTX&?0LP> zy9Z6~+b$4@gB|%s(N{JFP$|2a&IqPeojo_7bRAgDRDf!BFfwU$eF)D-ZPy?pN+pH zdI^TBIB25SbU_@!g$V=bcVfk&36%^gyD)21t^}bnMJdprIdH~NWR(%nfi*+aZMyL8 zv_xzJQR$=L#Fjgq2CbGHEet%%rfK20G0}8#It7&_*{2Ivp+oACtkzKHkV2`>p5~(r zrw?a!nxoH{jSE^1z93VgoMW_ZLl%@$(0lrD#LD=!}@q69Ug{HTHBtURN ztqXnZ%JVH)fr#_YL?7pr4{(5k6b8yv<=^$eXYbM(8W&eG;!XatiVDn?L4ajv3Y#`% zX5uIUHi_6q@IJ4e@=kXQdMyV+r6EFYgO8B;Gmene-13917L0CVBBa%H$f@_Cz|cjg z)m59e3$UAr3%%)&=R;*-J&UNs;77}W5D)Xv*Rod*mv)9G+0Xh_JBt#lP>L0p4k2>q z$c9lWKzC|l8G#Z~qE8Ke75^>@{Z}`J#EZ{##8d1AuM~faym0MUdo6}GW>98R?8WVu zQ%9M@2-F(l)lHhD-83(2cdS=vQ%5!{m@*O4^v*i`89QQ)Us+UG;sc>bM6D{Uvg7L@ zi@W}8=2620tg2F}dQFDAuo_}vUn3HXK01V<#dd8|JdaeU)z??er!no|MQJI`C)^|w z2a|lWG5VwqG=;>l-#d(=91ls45gL0id1r3$Py>z62wuKOFSi19KPyU(Qsk_=otJ=G z%TUkMk3w6}wYoo6W00V7n=?aXA}T{>T@cexS2;PA7hvuQNT9w@mizSbwym8>G@THj zVjq29W*5x-y1z3-2_P$aY0U--Gwb{T+fZCDB1e7Y7KvjV7iX5^W$MuGwq9B*hsl*H zRqg264NZHlSsP{9WZVJl4zheIHq`|NH)t3e2{*O%6jW4q3i2jq{>wzOOyMD{aWTRPDPN|`^T6)_I)b1}Aa_3>Ml4H!02_(L+`Ea9q0Q-u_gdLbyfpS~D_Rs}USI5+~ zTQ82-^Rb9s8@#BsyJ~04Ev!@W_!x^$4<4%zuiiNV@0$TvB{ zQ`an&Tzey2 zHjUVQA@1Gbhq3;n8H1I~<4@Wj`*~I;k6(*d0=+S~(%IysqK_@fd zLeGt7_WcD!MA_#N)On*^(cVZxfVX>RhC#&8eO+Kku;4=B(%YZBKASVLAO^Ufq8!SQ zup0)`9!V_E!ioMZEX2YrQXSlsda<6JBS^C)W(XwQbG%G`hCP~+3iVPjPDDN$yuzs2RtDu+~benR9=h% zH~i1165`LbrgDy`>im~nryG9)r6wF!1#(Kk4+`^!ruHN?tYgzImPhu zUn4f{IVFJt&ET{<;$Yh;a&b&A2ZzvJZWVBS7xtlx zN3WIHINJws{5fE7b;iJePwfzUJgYi zid%OKJueaBfs46z2pP;31hB9BsJsUqi13XtF*j?9h+9I_6o%)&&N-BEGFCXotz)L1 zW*-U&fzTy#6Z5RjQ-;9}(A<7U9ar@8hfFnZJQqsrM1(|R7c^stj43C-X z$1zXvF&LLSA!O`g<+Tvn7dNr{3r?OQ`UzDCbPD#>*-jG7-8l4 z*QABcR~5s|#Imyavz+yF(2Si|Do@p1e;#5EBBD?&5hBIOf#ybwX#Ywc#?Bu3KPWBE z|8Vio#K!VJ?s~n^n)(+mL*02*XW(`n8APyWK(7-I-%>|`x=wdCE$VmPg zUpw0uMYNH+Z1uwX7eliP2cWxyMn40XZt9|$>+PBx+UwiV@p;T#h*nbi2bId&iguD~ zX39v+WT0!xa(!7k4D;#Od-|NI^mFVv)}n)(sl55eM0;J?Sa#~F8cJ@2jv0n4&pj!okXcS;E_BKrfHQ)+ zZ1G8&>h|jmX7H~ z-JOjZdE6nT|6`o4U=oRviOoHLKqlH+EP^s9Zq1Aw+@PC%I*gGM=|O3DoPo@Xofr+J^fw;-O4ehIjqTXYO!YGy z3tf@?YV{X4xK3*eHcaYl-s~iI$pT9ItL#!MPyo82{okAHeT6G%3Vn)T>1?;X>TKb~ zXP?oX2UH_+(rzEjVo>EbFJ>4B(CZm+J3xX1y8PR8+1dIj4MBh(A~pmrWF`$}7!pS? z8X91QVmVn)PMFY?wQ5Q+oPvcVooZg7(x*(I=c~+Oo^1>tIBLEbbb)O zo&H#Grh^>e^oCP_Z^Em7eljvIT{uJX^k^MKnIXcbVgqtua?F~V_|`c7D0#dR#4uwT zz93xa!>xRiCZnkgoxH7}`1OKAg{^OsskxQ#LV+Uf zz3#5oCUtTC&O@LcWeGK@UrMk!h^F^qUbfK6e`z+>ch(^)UZfzRk!DaUwqw9_16*$jltID{nkKcVg%x#V9XFgP& zV`a=3SDwS>=GUHg={z(6vEreE5X{4{8`pTyF}Mtv2AAiEppu;Q1Wcm-0vF>17(*1d z6=I29ntv#)_4C?~u)QtJKiq!H&YBzE6%*jSC#ScOzWr60O%R>p?cGG8{=^DR7)Twm zvf#|^&${}&d=RQn->dvoim`Gl>hA|X#bV*rQTa}-bN_pi=p>!lizCVnB*qF~RzE%{ zJ(@kd;R9JT9Aic23>1t3oDFnHWLeMVN?DGvItHJ}!AQRn5Fw%i?6|;EyGZ|>|NeIO z=2FwkPH#MYHHuyd!uiDEjr{QTRz2Y8DQYP3bxezLG1#nrg_r|yk%S^fP4fZBqAQ^1{mMwri{e5Cewpp(A#0lrydd2fa}_%AI&YBny5rx~ z>Jm8WG7G*UjHb>bPb2=#HiyjvZ7*Tc3v42l*M|j?hZ@%miCQfE5h!1OOhe-3FF=n; zYyL)KPle5>+k7h`SS3eVWpBQ(Owan+t)0n-@JG}BTQs0G zAiL~Ptx#MJIdKH-_vh@vaWa0@rExc6_^oo6Ko}v>FuXUxV&bLA@0chjB9a#+wK1jK z$AAMDG;z$VrEyM_J`+RiIPAh4>EC)FbfUlTKA4Dc+bTO4Q{fx%!Z&zL7TREOWYH$0 zBHf5eLAGe+$r@0Q>EtDYF|raPZx?j@VJXSnfI{iNzw@>AP?YKO3Yk7x=Em>i92g6uQ{CB1h7 z&+c&is5;1$5VKvm-yU)U*O!=|Q}x1d{P%wTnw;(fpVa7v2m?oJ;tyiL_YV1QItj5; zSn-mM%b>3}nV5jan&UZosj@~u9S%79KSpCZv;5JId?h=GZ#sietDOU6e2B#HAjNBy zX;IrmH7;xEd*c$8k>9zp)BY14nEH5_T=|i!{kzuj1L8B{o4cqU8d>e)aJe@Yy`rq> z3UhI6hbb>+eUgsPRgK#6w~R;}pzQj6tn_m6{H|;AEXVEDGrmGC>*)EWTK+LGUoK>* zsxd?FuV6GZVn$FTO7cIhaT3Hgbz(Ak&H__`WNc0<7R=Fyb+}eChy)FVbd?nl7wy6r zUR0swlRYHYyXm}TuGbZM+{VYq39UTeSt>bR(@8{BzOo)r&WjgfivMgHb3p5hrZ|BQ z(Whrplm`&y&WafvM8)j36wzHCX$iQb-+Hp|cv0d9pb)hZ-m@i|)z-B8qT@mr<`oDf zfXjXm;M%X{k8hk7Pj2XE2l6lVj5VEibMQ{TuNDDma45CEpFZ}fTn1}Y8m0wc{hM`& z@J*cTeRZ9iX_&rU))HcYC0eg`1-yA5&|$P8c;UsAdu`|`>*0p|M$Nn> zQ7RitGqtp&Iz_0ev@&O&$RKE{Us5@A9pt|VZYyuJYnZkmY#Ls%k0`c41a0ZTn?=$spusHxYw#B~@dno()rgrFE#qAuU5Jhe_a!7ht`8w(TEEJMM3DQ*kO$S%exo~9 z4if%%k;=@-@L#;A|4$z;8#D9&)SYspr5(35`hUQuy^7_m2ixY+rzHZoEEG`9n)!f< zJo89vYDH6-$=z;#K7B)&dZE&iU2ZX40(v4&9L}fH9lmGbqAz6BjdymD?%$6$GwgLB za~$S+)(JbiraDq)F;mg4|42?dwM}^s_^~IBOE~ORrz1v`HM97a4?n(5e#p6UuLqOp zecQgjGt_p>bKQugC1X&v^_mHW*r1z<=3-o@D0+7SpY|(=RUuRIu)02aw^KJ`{|+~Q z-&&!HUY`%1Tebdrqvz_=2hZ?_O3r+{IrD(Ih0pt>Bdv9MyOv*BMG@n?Bv`46Gi7}T z=E6HBHS*a_(&7SNi{b)VWMln$K&x{$WQM3DT z@_M{2 z_c=SAWV#6bBn&f`g+o2!`-`gxaf*Z8_!Be2YB2qU*aM1r2zl6`*c(szGh=2SQ>awN z(pL-)jqH|R5-v_({GN`!!`k-xNz^ z1&IL{CkjqFJIhr1@ADU|0( zL|+x2B>`Hr{ko7iTNhQGlRI>+lam*e`l3TmzWrwn6bxzLWRl@o$BW#d5(*q(COOKr zmAR1RmCJPyyExn}Nu_+i$V)cxM0;Rf&^M~y4OLU;$|~I4``V&q75~kX2CY9++6;Sx zw<=D--L8P*E<}G(+~axliwG!RtJIv*NXXKISjkvwCxJL{0hl#~J0ejG500-rp&QO!1zkm;^&TS9P~wz>-%>n0=uZQBzQ3R6O56E@9&hIVFF1aau>P>A#=i4*CBPVFuzN8`vEG^ zos}+CCL%&032E3gcy?4EP#7qt5?L%b7fi-iQv z4mCRyp-zev=N40z8ANV!tEejs?|J?;sK}92jGrz|Bg3OGXmS2=an z6wdSJK|4F71G$zv1@9DArIZ~R7I^d);)~dH!YJr%ecd+ex5zUKDJv9^+8D<>ioi=Q z)9lw@B6p!GJHW`;Oqc(09Km_E=tHTOzO*6GqYUx}Q`Nb2$iGflf*VVJ4nSL1ZWvp@ zJWb4N;l|nE30$BAV?N3quU18!{)9{hfamaslk#=%(o9(EEAf;(vmeh#0M7jD9=X=& zRedYB<$#au><&?qJ#m*HHzoUBZ&~@z{^ct1l7NL|kGo}?u4Knp7Q;jO?QL`T*YR}t zXo-RYGDFQN+nPuLj1H6}jixhO>(eB#y=wV=+A3pBIEu_*e}e_;{QC1SNmNP_*%dOo zI)0|2m~T%6v|Z`b2^+5x{?&DbrH5UPbbq@zf(kdK+6;uE#6>u9*hfFMYgHtx65DSq zbYaysoXLi}YR_cnhQF_?KuKfdwLqUIS(-;7+_ZP<6@2kq*d1f#4Fj9X8=cylUFD4~ z6EAc;<%6tJ{ArMv)hxvRo~%c6*3GPhbvk1`1GkN;^X z90d#r6`d4^(+WZ6D&8hS!ctbNLS~X1@=o;L*6~s z7m2C-?8iGG{Bp<2%2x83lz3MO0fzBrz&%dqD9?yZ{Xjt8 z+J-}4=70kesP`dR#$VqyG8YNf5t@)glpA%?#YBbiYI`0#%TJK0Gr*|EjSDoGfUTr?W~ba#Q{z&3X31@ETjb5+k(oMc zL63GUR{6iUTjgVylWVueFa$LstxTuz86f&G$X3ZBt0tP1m^3Nzy;7}OnDSyyiD#wh zmDA8#3#>kROws0CptFg#*jpwW-fJMC%Vio>$%7wl_Vo{9vE0~sP^9b^wgXKa+6K7C zvmZa-Z8Su(rKdw%RS&}ALNrZ|>F6^b+pFSOBK9$}^`erIg6tvo@?chS`r0GiGxSAV z^B!|u=WYnfljzj(9$PREGtj(c1ziIITE7Z-Fy6|=6MZXD=m{jNbGPLDaZK;OLpC{Ahu+LVF+Dq9h zLq2}J%X8@TR^nu)-DepQOh)MPL8z44%H5nvTjZIHY^}(CKcDrG1-AJ486RwRxU7DC z4C07|=kHQajJWmriu?d_7pf>1TK>%zhEJ96cRpi?%gk;LE3OMlr#CV|q0T6Lt$)sP&^v6P0w9z^o+7Kl-hgNNG;%+TZWZX3BTAw!hdp1`k)2qBbuWhWb zmB#QSljc{#TdP}@_HbTgmTu1vdFp=empu>2rHPzW`cU>d(+^`h*jkUt`IS$D9T~;{ zi?VlW%tY&&c4OPNZQHhO+qP|+9dz8W@7T6Fw#~2i@w*P5{U_!eHAY=kh|?aE`*wsG zL~`Ljd>&kyW*6scjV+O;vY~bB+xE8aU+siu^_*jP>u=HeFejKZTEI474(^cEz|6H8 z<{s@9te zVBrJQN|+*k^_Q`b8#B;3O+=ptSnF&2alJ~1pt&Kr3!JhK82`Nib>-f2s2cK<7CSC( zfac0@kAhhm&M4P%cqI^@mmL2@!r90YrSG;s3S|6z+Io$|4Wn#uWu0)F5c)h;ijW{W zOckQ&bWw~xncnE_A-?nthR#-t){z5%`L-FRf*||tW}SrzH<-a@76yP4r)Q~vvaAmh z#bIM4CCh842HkUEIQDSW3MuSza~=F5RD8nI%q;1(i&>_LDHRxQzcLat+qkU*&XB zu~R<~ohY|SAmDI;kIb>!y}UPGcWr&#-*=g2VGDw?y zO%STOO`>bxL_heerwSZ^k0GtBQv=mIkpVj=?4?Ngv}y69+T16c*EsJqx57yeXTIV- z0K=tuRdZzu2@+l(S?c;B4v3-@e<`fXqe;R-Ws_P;gdzOm&Q#$w zR)5o`i1Xt48{{W~Ht2kN7NEMTYyej3FEnBMExoia4+LH$45P`_S<0lePY}pM^yE)- z1%_|W>@FN*WH zqa420i8S1_Or+z=R#|m&t;G&%oZX5n@$$*Ztg5n;J3WP{UInQZ7YrJjY2#TG;{qmh zFwBExs%1yJa;nRG?eTJ$+Thv1HMCve8!{oy%2phq}|4<1-PM#8rgK;MpR5XmZY#L zP(`KA&j?e3TAvngt7{P&JH((W!9LnQ+^wx@Y1y!90G9SUg3?hG7_;b19?-NdVeb*> zCm*^&KU1f8D;U(?^>!;+l7)SwZi;RTq<{9Oh6Mbf<-u8z)MNJLnBL&X=(X%9unP_< zi_2g^vty$({iNDM-u;qkxr6}mZY0ejKE5(kF0B#F$$e;I`D2C)dKy!QUZuHgOm|V) zc()yfcg(U!o1Kq?PV;lZ@~Yv_NI}mqspp{OD)^OJ7su~+X*c1(c+)nTqTcKCo~L|y zP;wD}UAU0-2v3W)XMbHh>1 zI9%{2jzFZ98PN3Vx+#mwSPpEvd1)nWtoo~X_V}J96xq+gW`J{6|@2wV3?%qc{ zQMPBbLyzQFT^t1X+YnJE3sxUJ+#SpD*J}rjdLt6YTqz?GKLH&PUCVKLdy7_-(84sS z+PSOi_Qs8Fp;>f%i6e4F$4ZJGPCV5PGsB*C6hyt)>DGMyIM~KlD-rIP5)#;VRCTdy z#gi;0oy?Kae`-Cc5Bk&$LWhpSpat!$b5+jIhf6X_WlruT@53fp6o}@;Cbg=_nD;3H zgXXfhM8q~j%5zWpqV-oX;Ux!Z`QRqdlKooV*2ld*Py~ej8IRNgExi+qV6?e6_3`Z| zaF9K~@40q7r?0mbRmOgpSMxXBM3A$eA=jW@Q`{2lnh@Y&-{V3JvmyB|H<~$_coe*{3v|S@o(i9P7Ga zU*M?fU%!0Y1EA!!2Ka#~|7eveCrruAUl%XN9U-tsYS5#Q5FxEN_;(c9+0uxfGI?1& z07}~KZ$(9or2K8hd>CmG{{?ZG2OmEOUQPxpJJ&?Nu;-~gNV@!vox?O)#fhnQ0R{e| zvqexHJ;u?Ddx&mhhWsv-ydCDj*b_4L_P>$B$6AgT#a{CIQ%P5tE-SIisxj$wt{@v| zWH%}89uhr!?!$UE;6djC>!7h(*2U(?FKWk(Sl^%Eq7D&f#-n-+56Ju7>vYpSCvI#77spTOb?{CIia zy>W`k!?C1Hj2}IPF-34?OBExTo4C9M+$3)@M36#e(7$}y1w(c7C$GH&_|CCn>P!BX{Z3Aw3B`G^=ozh z25TLtCL6puE6YQzMpl1(DT87Znnzd!8MDrKt(YY`FM8zSBBp_&K2cPumy=+9vMHs( zmQZI#QHLjKS?zAX3taE-8};4?4p_{I8*>a8!Z?tQFd!j`9OJz4%+Dbj&~CtLX~kk_ zav+9^;$}4?5+8zU%7%dX4Wl?pBvm-r>=8~#et6l4V@a4iaZmT&E@_bFyV5W=)sPqX zN_;X@$7r3;Q{wYA2yt$zI?BzABjU^IBlh5j{kMmuWF|eigXf4fWeP$4RBW~&E)H*g zE6*Pyidm|eoEbvUcqAs0Uk}dNU4|e(eu`hECE7%cE3pyQS;W!h37lS=GRpwe? zjTr%XEDtE!1bJYzd_LTvfG?WU!%kwA7V8R*vJ&rre-jJUDAN_8g5#tky`$Yr7+M5e z?E@*6%p_T85mMK&N7d29$X>Kc5ZjyaocMGZI9jU76TSp&hX1VDd3whF2@L4lAoQs5 zM2uYaIN0T@LumUxtJAU5qrfE1gsEW{GD3o+NG_Rhd^Dz5a%&9u#nV7Cc#P`Ca|i#c zJ)u3f2=k_zZQp1;bv$<37pfu-VKWlWtqVs50xz9LzYO4oL`lz*oWTJ^znMM%wgKYU z3W<2g`@#JCGxNy9^^1vg(f>$^Dfslc5@3(jY-sO5uws-=iWfZO`{VDM5C3!hA*g*O z28>;VEi3_*b6dn=h_yv@+7f#I#Q4(h!16mIQalNS+(=NGUv^tl^5V8Yd_BO+wPzcD zo#0Bk5%>k!d(40U)af<_cTVlE@W((b3DC)03*yh~rssF@`%!@A)eF$=PG5slnduE? z*Ra0(FAx2V5iALqTKR5A!G<`~hGpuOMAqs`-NkoAyw`U^a=t`6!7;3_f77Cdd1> zSSJ+3tE}Y5&56Uyeuxq@iV*_E@EOSvzJ#BeXw;3;rHALW6-Y_Nh&$Hz-E+JrSuK(e zyK9KgBBGcakvM(aCawy$DN;^bOS}zxec=uaBrrIRcS*wI=iUST{DC6|Hjx;}yxT*> zPUi}-rfYcML*aN3MzHA~)N^^dt|@t9Clcbe+N0+7UqMs-c@F`{3S_S?JyC=3PB!#V z13yOlSU=!7l;0)dE3`B$%cN&_rG7;g$`3&uqT18Hl4+J8?!cU}G+0}5@VQec0GvUu zu>c6%OKja)>C`R@5Im!lt@8O9`iO;)88m{%&(NH`wo?JVvDzZUf}7rNXP@RUn0J{{ z_ExwUF9d<41V?0Kl2#{Ug$;E!1=Ml~)N+b?-$7cW-VhMK|6;C8;d0@rS_el%W~VV) zS-bhq8&-vCW(xJUnKqKp(24}v_~4C09R?OPBWRqpJ4t57jZB*SCH5#?an{WlDUQ4N z>rdD`*Q-hLS36!BfXfQPDXHMVNk$@^gnU^_B0PlCoFh*5cZ{bqJqf64SO&l>0+EE4 zP}bsAiv=(%$-q1~K~8p19?{BV!R)a#_)6L!es|D876Gk*4de|>u^@e$+a!GwzrZ95 zp0CVPR6-s=Ns`hmSINS%rZr1V(YQbfhZc2#0?Fz=43^jRyNa0Zk<#8jqBTXq-d=S? zkM+LUyaZeE$h*lFwbgZQsvWS%(X)e4^f2M)t=RBxW+z}{3V@h(7Qv3`c>3iUatF``^ED(UJ6P`W` zGX_uKOw%QFtjV8zq1KI2UA*<{YA_TKb0pAPUl!;Hvm&-v)^sNqj=2BV#mVmvszOR^ z&Qn=GDx$pVkT&(!h^0q6By?Uf_-KyE1u-XG1UXg9y%5mg?8v^ULkd#t1u{vCq$W|h39n_TFy{7g-JrD|gHGvXQzQCp zPNljmTK82+3%?%OvagD_|F7dG7A}cg9(lO9@%*HhK^O{)Tl!>5pEeeww?rdzaqq*$ zHnw#U2271Xb9scP8h}Y&u|k25HqqZSe*eR;$22O%#C-VaCZU^$CUp@<@aIz!A%4w( zpqhIj&O-=FV`hg@1ci-QQ7os-R;g);>U#^cvgt^3B`9^RqTEP8VGrj3$8NOLYP^hT z2g*B$kBcb45mLXqxBGQ=cpMbhlM?4LziXqTyUWWrobu+c{w zPt2dkwvI)JO`qR5!^B5vz^g1&LqVOZdg7%8nhMbvJf(RloU%1wCC}X+G>)#tA8fne za3qUc_X$i*sd-PSmK2J)~~u#}wy9sVA9Gz(r5WJD?n zBOX0mL4c+9)hoQ*^~gJe`#Msuen(z(UIx;=ta^AP;=~scA6AHbta!r?w5>!SAGAg! zew{lU@nB#Nb3n8SD#2%gh}8Q<-Zx4kL@;t17jV8AS52FIT^RpyV;X#EnME}4Dl!h` z-$PyT6!aq7gN!|GY-6yg1rS?faIPvdAe-`BnEugqzU(6ALZAfziV{R}YxaydQp$OC zU=%&_g9sYj3LLv1nQA@0na@1k9rsR~p%qzo&20*6x_mc(E}+47fByIcq6Wxp@ED|v z#|x9`@LD5AWTxpq)fu3XRj8h`No_N<&_x6w8E#7BaQ5=!{csMx=fhi{AwO3<;0B{p_mo8yNxZ& zK0aJx@rsXfj}FLLIvooOlP9d%8Y%d!ZB5V><&WUb1bW;;?sot18xC?Gf=y z99+K#I^j+z9^DL=zfIsVYIH7FBtv)edx$57)5ixHEAHm0`y>y9t-xm41tudca2R3j z5*$hU*R_RTla0RBGbcktOL~gaEd!!$CNto8-@o@Lzom{z%KPj9RF*$BChW6ok9CDSrxA3m*69u zxLlv`M#gxK>UV!^)@TLv_4qy=|CaI8PUF7v@I>t5n&9h>@wBOoi&i=DWRNDcPubq~ zet8gZjLFqCJ@9AmuOEL_pPe<|3Ea;vKFaQ;Gv7}`Y&r`Lj9iT0$m9!{>iX0}F!y|z z+}_>oUJj6mM^TBB2|{ZlJJPR3-Fd)+l-L}zwCK2kI<)R%48In#Uj~xCF)_v6gWCy& zNz0Jm6M_1EqL6;^);yXP##*=VP)Q8eOj6MrD&YhPwffr z{Ob}Tk_N&-PK)gXQh_pH@1?EDmXzdz1Dr*!wM!l5pL=07kY*0pkxTu{mOxg|pt&6pYfviFye21FW!j~Mry_=E(u6DTZ6Ax`hny0l8uFYYcM-4t zXtww(e$>7<`(!(NNZ{#y^pSS5tDNUtP2(w&?Ha+}WdT1Y^e-}`=~G~KQ^!xvEPeWp zTXv+m4Ad;7hb#DQw_NJ0o;I0d-32Z~bZf$dtOjYS|e+a}&Tfr}z(!(56l`&ms7O18_181{y z`EI!1vM+Qo^#0PH*T`Gmr-6^3Qa_Vl%>U3{2#dy9GPwQLdER1F{p~CK=PcGa8eUq# zdyS9$!3JHnzI*nEu!r#tbmie`$I|d#;0IE@CFwgoCIR=}6kc!Aap3jOXYabqEl~k> zMo?T@nROc0q}Dz{YJ6NLX?0>;K!#Z9DJmp4({$GPXk9sJ>|}6Jsl_Q& zAFK&@(5Vm|5L^*w$OQ^~UL^ksi=!D;mKZ6M>~X)`EKvMT+l#0HI}77!B&VRHzX?gD z`0*FeKV6CRIhY*9Y;~))%$|7v4o#ka5MSt1;AgZzy(5PEpUAh4K;lu z)NVRv;`L#qj1`nhit%MDRYRtteWr^vhzAGM&T&x6pq?xa;=Uo+1|(ABV~o;a`Y*?? z{IM2_*nPW>*gu*Wx)UNl4KRb5S~_!ex#w_LiBMa_8X6zPAP4rmEp(O=lQPeh-6!fs zbVW-2XXb)x)5(;G_KTSf9{})t1q}h|u!_fSbUU^b;+E4* zFlOpyN)^8+Kgy~#vzPJF{m*xPPnTR)`$i5C6KvRc?d;NeiVfLT6+<^(f!%MSIXVQA5_FsG(>tZ~Z@!&w>dknqq*$?Iw(SD$)8e)b{Y&0gL zD}Qddr9rNH*QiOQLpw8q^5Oz149Y+H|G-Wj!CmEfq*V7~3|?(HC(3HNTx#%&xuGoH z>AfK*yrLlxlIj_B@Yt7@s=f7(xj^CGTcAfdZ@*evv&YwID*WaZ@S$!t1q^3)T=Pq1 zmF5|AIke?^^#P1j<+)4@jAuWpyq$)dJsr1e(^Uh4kH!Z;BuYHLPoq#eVU9}&nAF+i z-I|@pWlZPyQ~Gf@d#dmR(u_#P90M%T@${w~|KbGbE7quubN}2nPcGY0gkVz$bP8)rJ3{o6PEn# zeI@GVx-|+GJe7q79_w8P%KHtYI=`5%4^V?K-^$wf6sUVd4yc{dEtY#7Yp>+hM@nH> zEu9s1Sb<0#7$~N?g<+8ls^~ay_(Dh&FcHi_Bc?mlSXhp0?3c)-UbJ=+N)oLM9`*xj zcJ4+CkbRyuG)xJovD>R^Z;@*5>)2rLa6pQI(ijh64%&oxmxJ;@RB|EJQ*;ArdMCxV zJGTuCG;#erH9<1z=_ z5-1-U(cQQEH*G?UyL--k+wNqteVdJ}p)$OkhrX6X-%=}P@gpIFmA5`O@63p5|EKa~ zr|4#SoWh6|hoL7%HAPnWRt_svOg4s-LIL?7P;+3yiGM0xS{@l%F?~zeLq2K(vJAns zxP&ilxIC3qiQJVa%=N{IW=kh-RAxj*+yeOB0hrzhnZ}QI09`EDYbOdc8!>iS0@G@g zvJ=d(*>xX{=`QDA^89|t8Z51R(M)9!m@30@V-nG4Fex3?o0EaF6|oB#806wTu?ela z5H&`PQVcB1O^vUMUE!P>#-4S=x~6a2-~pFwe_Lg1$VTL2vJ}uG!z&cw-?u2UXYWXY zH(jIzsNLsob}zz&dUzZ43+$tkwRP{`mZr{Ncu*^(#ef6S@y@IuhgN7E?&n3}tNpyY zG|2Z2V2sfAHCt=I-1M)`SUW?_>bvyPQ4W>H!kQ4NV8E>E1&IB$b5u151;Yf)hS@g2W*cm3PdIN9MDdkdRLEJgz>?~N8*)B7LCpkYi)7cWDk%nB zabTM|?{M>u-tcO!e*V=P?-`=g7zzYB2=7JnL}f*7E5qhZ8UYm&4K42B@RZ?|$dK#k z=`seJP?839xw{8g4{*nN^o{iw(hr9m*DGXT%HhoWIXkg{dudG5BMSsSX)WyK-zE5%$F5dVeCjAw*(FtP! ziw}X~GVen~lSz`^g{>cehO=qitGfoBOijV)6etWdOqLwO=I+yxiUT`s@m2LZzQ_UX zx-5&`$!9UKV^eB?b;Ym+Xi-wzPZZUn42I+)g5{y?5EJS5d=LU^F6XL)4HY#n@M17X%G|;MH%qk zp)Geoc+Kp>MFNL;ZFg*${oR%Lxf=oq$0OWyr%_9a&4zG79n_WV`fl;$G!s4ceDz8Pm-hjY|-*>ioqD8D)+@q6<1xxsdaxTwb!v zV=+i-M&Py=k8&Vwm_1m>%b4oAbCR$|c^siJ6M3SjKQ)L;J;u*haH#gLNs$hAY}q30 zYL)zRZ{d)h!YUYXP#_m7n+7Zi7dfa%Bg1JaSpN9$NmLX%%1GudrerkE9B?57#G~Y$ zZ3?12F7pd!o5s_6&-wt!^V4Uv{}}(-nE&ti&-(wxQ~!@qmD4!-|6qW-?LLu2G@IWY zg=Dsm#rFFv!N8#7#Pmsr!IXE`r?+C^Xjkohy{x>=WSOIjZwqKwu|r=u`S~~5+itF1 z(wu?5$qe5=E>HW*xb&RiRCzABHM#L@9b;u1T)EKYSI6fKZ^UoR>FdvxHI5_h^*dvB zy;5Zh145n2xYn+JS$KilA%PqdH}f^LU3zOy4-Qr~%IwZXz3X%d=oBmK=wEbSyM3kg zuNvQ`LLXNVSN+|O2XevZFh+Pn;*>rzjk~XEZnX>Uh@zaUGq7-J)fAvI@$2SjsuDje zxYQu+&dPQs4y6iPXs5V5&HZZ=+m_&Y#}DOp^`TRF3fwjC2j8)Lu=+uDgb49bEE1#O z_=yI73HIyR7Nak|Zw-Rf1X*toA>lfO_xfU~dbA+bPSXm)ZC{12`b~WTjOo(OalR!Ec_gc&kae8z& zw-fp{t{#Gt5qd{`qbNVjntX)a;8&FrM{uMVC!(7?Waa{-!yPnX1LwLw8g#(}Ny zsOT}^BU`EOqM9P~ntz)`xKSstSkSvk2uo3LqeOhVS=gQUSKYlqgri9vsMr)reT@dH z|2*1^!foR6)q#jNz6Gch5=*XSp48s&V1`rby=qif+z5n(H91@3cKUBX|Jx`ZnEehkaqe`s2$SnwUc9y)R2UaLXkrd{T*E9|`#>-CGY zSB9@eLufnz-+XQb#00#IMAXsWrf$qZWeor7eNe(5P@%XNLCVB1fmR7^JyKD`5$WZ8 zgNb;i#WFU$W-Fme%274wF}BQb%wDM;hMEsc=rzKY9FKBLPVA4#$qMl`U zgsQndvtg6W_tf1Ie`U%9;ACx5)Kl;tq;5vvtx@^xkDdrNDkJig2@%xz_L@OECb$=j z=l?ZTa||S5aWEVg2ePPI-JjJtR1dfO9b>@~I$ekOrSp;s%n~fX=7?B_NB8DLvCcBk z0N$E)L`~G+qBAQ38Dgu#V#*6Db9tq9T?O++jtIJy9UDYIXp)A%=EvCX&xQ|&w!zt| zJO`Wr$rih9HOF&WiHbQi;jFrld57iU4Hv~@KQcv>t53GN)7EAQR{St#he;Q@HqA#3 zq6ibm7K3E1gqT_=J~2rGX1|+%QK2m-gnpgvKU@ZAK@Uesq-qXs2YTOcuBCp+11drx znbhy5k{Tiv80+s-^&!)6qO%wMcNb24hzso#b>CD+_g6*QTl-B|LqKip3h(jzSe>>0 zh!2vh2*{8xDFaZs>cZkS_dcW?EjxyvqH5!NxO{MkqX^nq@58*%XRqY&C(fNwJ=36?02Fb*kj-NR%YbV+wa%rNr;L~+_cS6zm5^O6i9&zY^>Xv`&c89WjHLl{ zMkm&UG*EH|fO@{U!oVA#OhDKampVOY@?OJYa)kkd>z+DvHckR@dwp#c*t!Qn`lDI= zp)Vcm)HPO$Io?JU7i`6k$-~%7_cFo=4;pZW1P zaL)l{|J-O-(XUGSBP#`0qoBt#nSit@{i%k!MXd#|y zY00_)W#42K-ECz+oT((v_QMX9I+-H4Us&2l_t{f1a{#kKgNdR-gMrCKIfOQv{@asB z{M~j24}ZE|a|#m6nRkLv#WP$@s8}wXo0E1kcE{!ZS5<$!P?Cb;2#jR;inL z!f+7KIQ65(N<|ebtr-nS3zuNyu%VoMdE;?#Y1gVGnpo!_W_6G4Ow0ooIgbPNr#oSx z>9B#85pxTI02--Hol4QRFum8eqF@-h}YihRGp>`)6*)p#gY2>ptj9i$J)n_)sKOL6~!2ob4v zI~r_vj{HWsk$TMZ%)lXEkgO_%=9I-CTsplQWfNZ#E$~5FLpVZ^T_*{c&S7@86QW$A zk#@2y`*fp=I1>q1ZH#%J$Zz~t-k@H9OCn2YNjAER&<-Sq8DzGWaR8!?SWtkZjgVim zsMhfhMgM#iajOia?(=Lq>zEouB8yIT@(^2C?-PtJS>^cn12y=n-&?!N4X| zZYUadCFv9?ml5|LUxD0$U=^{zTmFVwyEegECKi|^3JD2IQHg}_lPj2BjWTh%z{*7! z5jRg1*i1ccJebUURxb&PHlkEmuW|JXA-u+1U>~)D!=Elbe}nK;QN7#n`$LjHflEEK zW4!ZHH)=FfQ#GyzkvalQOqjCT%TKGZBH=KPF# zGI{*&U{Iu1F~-i^BMpgq2-IAas(F<9iQ*UL)z}*5Be@$^>We#d;qEer9KJNQxb)jy zlqh;pFl2`{^CN<63>WaK1@xYmllk7DGp1tXz zRD%LsH9cv{)S04s>U{D_X0Ja)1s^6XCrC~cmMI-db1!}@n0s4I=qx{O#M~=Y=qn6q z@6RsCGNE0A{}7vhbB%Zb#SL#1)VU0rX+|D;)MCiZ-i#Q%g_YEDyk#V91^ASAHLyI&DGVN6XewPdcGHSEghkOY z6onlQvfm!*h-BXlBbsVnQHb0UQ2W&RY-ce|ymajNL_%|E`-A_r_`btXqCbek^FNXABjz94zr=c<^8TYQEW}golGT3!q6Pri8 zC{4ky&*ccSLhTS75P5#Pt?h-2^T>b$Y`JYfS6cLvPPof7RPRyA1=L6KKH`okT9xh= zKTC|76#?Ob-m3YJO)bablx_FkyWFl;J^svr{l1r^`q)!v%?Pfjl{nXB*KhUc`pZRs z(eQuQg{q)_Ox(7viti4abKcz$!hAde_yQJgADX|8 ztrh3#y1jm`ccc(lT1|fL>m_4Qs1+O9$zHb3d$F~VoyAH@+n2-HnPoZk4sY^38K;=^7z>!MUew6-x*08(Q*l!vi$9rt7m? zVkrFmpZ0exQlIJmOPX;0UunX^$-@4>N|P(>Ntc86yX*SBa|wm_g;)@Td{wEQnt`y8ox{OVQnVSc(kJ-0d_kZrS_RaRP9n^v z`XnwC`W&FDJ|PajwJXe;GMsD_jDDhw4(Jx4$g|LZYhjE$Kq#_9%@u$Gk$_ET7>ToI zP>HoH;DVQ@^^0Vu{^pHCsC5Nd(?G$a#RBh*jwxrDJ7@(G3I`1(04X9@u)8Wy+5)5H za?h%=#ZV`cqZZvjLzM}TYfRqNW(DClYfR0;c`CC)iyryZ!PBgZg^JqF_vpqEcRX46=cw8L%X@By&yPzp2~>ea+D zStPt<-Kyq})d zg0+O2w%O%J#lJq%-HayLbX$gdFDEkuK0eZ=VhaSGhcj&qzf?R@{{-1Ba_F}qNNQ}B zFt&|}rRiMv6Y`W`e6DrY5okzB>u+-wP`q!GSfp#wtLk89YL z%D=110woe~$8!z`7b)swZ^6wk~cQ(0S=4uIiz|v+2})P)2|gG* z6Tq*KPCHyp$@F=spl0`GSZ|j@u0KzjXrD(rdA5LKrplavsbI(#>|gMm0N(GSjoJe* z06_6ZI+!|ME1Oqn1B?B8ontI%{(J2V)o_bc+%Xen=tONE88a{my;mpg^K?}ya|8JT zt2Y<=7=RZ>2%IGm6b;9jzVajzfF3qTxB9SH18C**fsnz%*I)wjUsNb-8(O)@UJP0_ zhrL#kn@ugF_tLsP)&;?Il4b$*h4j2;Ok%h&^g?xMkCydmi7+#F)43dkYca)s#U`cm zlICXCv9Fax@Zr;ARVW7D3k(jVxC+^Js4l4pm!Ah$Y5@;BgLD&a-I-BtWjcihGlM+= z^Cevm`B&Gx=;^P%y6Oe$fHou&%rce?2g!&y9#OfQnlInNda61(eU2qO^ zMW=e!_`7?2z+^Wp(9x$M;xY6cpTCjZA5L%Y{c6ZCuOkR5?>jk8G@i^a96eu_u$7Ni z6{Ln>)`uiI+3AW%?;*qq(`_1XZ{<@;=!kNOjRjTwU0Tr>~%VdwlY#ASft3 z6Wx1cN)w&i!egRpH3xsIZ(>?+mV5HSulRH$q;GbZjfeW+Yh7&KIp#B)2X+NM#JzwG zeOX6`=w>;@jL>PYg125Z3=Xh_vimPZ7B#(^LtG!Qx;Pf2O%u}y=YdSS2{Tjgpon!2 zl&PR=RJ6XB6@s>PVt@5TX;r?W{hog+_&Fl224qv7XSPV2lqEcqi3DWZcIlazR z?I~xrcE`qup^SOrPsNL4bu$SH@9%=nm4Cz) z`ir=ENm~gwjx`IzP_+Qm7vaEaPD|vMkSicidZid5bP4t36arU3#0GgfgX>T>iIV#z z-djb_$j1Q}i5v?pTotnXZAiG-{d%x=wV=ZP^z^}FEYG-`ksyjGJp10bX(2=Cp#sX+ zZ?O8UNWaLl%#EmUQUP$f%^d}d1zeY{KG3x*F@gB&cihNKSo)jR)fgsNG58SNpCqu- zB32l)f*iP!Y(_Q-VXS=~vzIb>>)Pr)$P$)@Vn(tEm*>yXY5i!6Lep7DEIjm3-mDJ=5selS+UVT#A=zEj=16$Cy?FT7zw)$| zW@J@_xFA><&xMW${XwN;SBM}JnStN?5_)wcO;r++bj3T(n-ea|6fFubQvG9*3?)|| z>@Rl3$$=ZkvVY*Ye`pMy=4IGo1Zjr36 zlA_&Q%a0$fHtEodD)$L?%6H!N%5^*+Q8UAyHLuIUq50dn^;C?;^(62|`E0ifCbg@J z7?PmL*r}zU+4g-aL(2nyqQ1;>;RudJBO5n^_RfBNeII?;Vm(&aWa7{MmY~jbJAq8D z6M1vfv3qo-3sQ94t9vcKq8oxRi7rx)_48W6xknz%jnSk z6SWt8DT30H;cAfG=^deDu6}xeKRn5|N+O9VMY6W2M1GU``dm}J} z-iRoTZr3I(lxntF`$l1e#!C;WcqjF&Qgs<}jd$0OcC)0nBDw@ZoPT?J5wjIxkQSxx zkMSJ)M~afkBC*kL1U7bCODQ|`;Q~uAF<8rx+s4^t`d59}?)(lpGcgK}=(Qk%38GZa zql%(7vf(Ogh@VZ0cHLwTF_sAI^CM^^SC+%AJDk^xWyH zaseE_;%MKGYX%1yVK+>c2W5dniPSn2co9tHGhxA{IV^m_7qN5R98=G#Zbm-q#rtk( z3D&%#SxUD<3v0bE6bVtr8qqffG|{^}>9Oux>{*0auSXIAke;57cf`HgfZ+d_dfESr za$x@7DTiepS;xbF4D?O?sVQzl$G)ha9TFIp8D|zyhaDW*fCGX{n;q>UJ$H(a<#Ebx zLA{g={ls~Or=<%NoR(dTI7aN#YxOU?hn8`Uz;}vV{@v}shbzP_J3LpN)9J5+h?+d^ zbVoObtfaT&fW9|t+VyF{n;$+EdzNE5o%IMuFNbW7)H}(zz}?-gXg!A-20p`Er_GA4 zzngVFTxrE^j;B?v!;`QEcmk1zU#@;%L6O@We*<@)?C)60Q#b1v->WZb^!1KRmjy>T zhwqM*mKFc55`=D?|6S6Q-=3HF_iT9E>8$np{^$kyQXjQi$=XoK0euI|l%ipV-@FG}2h)&hSXxy_lEK~x{>l>*?0$Bsgc#rU>d0;0sub&rpm;|M zxQy*>-(-&z?I3+)h|QF(04ZgREV<3z)gD<-L;fnhpImh^y0FPvoS2eW1M5*vOmL z*=rXW5IAQZS=FAo)+Pu=H<>-@D!k!GiRPNUzBQh=z6JjGTdUwnG{?`E?!$5eXZO16 z#^L&&ZnL*^*QCiG&X>xA^Yr6(iG#A?KvP!U(nU&zrSys-Qhm>T?ZdFfM-zgls=^I4 z`8dwDZQOmtWfXK?bSontY3px~+Zx^?$@STZTojjzfsr(ChAJD$HYI*16)6g5l~-PI0kpHRc^M^&K4Iz;Pc^C9XjzbbIknM zszlq9J*TjVQPzyArhYDlZc z&`}vrL-<;cb3k*#7T3lX1zeP7>ifBRTzH(X_$NIp_zw6=pJKOSwjSw|pn)R4XgwaG zzq!^3*Q>r~vWp%~Wv%Dfe)k`tajSByeg!YrXL`p90wF@LgJw^k5Tz4Yu#Jq__CWu= zgL1c@3@n`R!Gt`pt9u2ZGiG~4B9@K$=fnbuj>pP$2uqX7xe|1s6WO7@ZKpjqwp9Nz zKJ-037paMwRh687Ou>nc!$$AHvD0_%8HV$JSi%^+ZLU7U7KU;MqP#S5Og$b(C|*=w8u-yQ)v+PEJaH#sQ!jmhH73+-S*jzXIUrk$Kq zfknz23|w9vQKSrycC6aE%Y4p}{D2#;)EEQm;dj;#WBlyP?Y(VrV0aY0J73bv;eJ_c z1uZ?FohE|5cpy0m6j_vt9hwW?$`d1;MK% z4i=NRKEzQNI+eXh6GqfNgdyV9Ae2m_>C&DL1AiDz`fzP`^^18{iI}Zk0^Pv@mhCv7 zzi8nV#Hay9f4MavibCiAF?LQdqJ-PFZriqP+qP}nwr$(iYTH`R5!p#`s(!gNF|}pL$PNIp$0RZs3;^_ zig4BT!==r8dq$!rGgO>fPg5Drg$RiYH6t;A803I}$yxGi>xh{-m_;(*xv2Lwi16BV zW=sQ&6&0|0XBxQ`nx2?>21H@bCl{JCFSqK57j4w6Or5H@J;txF8XH>U~IXyPZd2a06V)nX*xjASp-7=OxzeDU|5kskq?AQ zW(@NAIEM(!V3NCpN4UTY`bzN690z$>0?dbCUsuQB+Qq_xiHKH@t8HQpEnN-mLd#GQ z51@e58Z*Fz$DX~#`J9%5;4n_N<%?;T(_|Nuqxv(^mGWDxuB9hx3n@8oowfEP05yQb z7@7#mzRjFGp4T=%e4I_tFKoI?qw6k_!xhmb)d4$a6#}5b0tL6q9#3D+mu7zyNm&~R z_$CSkiM#-sY;Z0spn`s~2hlkei$p|C>n2@Ibf$0`{ZH0e@B?A=lHlwz;lA8J(?Bm7 z@H&7o6L&N}gp!Sf`0l$jFyg}OUYb+^F})|AA_n2v=$xQmkT>eXc0(5a2k5V)i9T9{ zr-0xGOR0zsreBxU&nroREkx@R{VAoItO%ww#K@gFW}w3A7q_hlFB$|UfdKCth!SQ$ z$*VmWnSo4!BrP&?E9^Jhw%ki&R1GZ;b@22g^nyFIc3vufCJ6;-jg$g)o&uCFz}AO`IgqB7in?t)cGa_M9qA zoWz-^g|@A*#uk)D?V9N1M0Rc1i}GPzxv0!Bckc#^V#!K(39w z2g3Pe?;4L1#d%U$;d2sjHefFmAGv&{PZTcYlz*i1Wv%-yQw9Lsuq6v#@W`8jIQv5> zBQF-+Ci|W`5UvG=4_S6M^$@cD(b)>oZ#-k3)RU!db_6FQRS)(|=?`cmYq3(##}hp1 zBGnZ1Tp3cN|EDqN36!XJM!9M~sX9ty%-S09wqLGgmLl-Onu_-*$4Eq5skFdyax4cLQb`pLe7v`r|fHcKIA=3zI$E_#3J6b5D5v+b5I-#ZqrQjEEfn8n>a0c<75x z#rGSnuS8(t6C=;OBn=VevGh{cv!%NZVMexdsubxd6XG4TUO=eX?M|#fzMLh01&xVA zHSP_)gzyz$%FoI#${Sqz4b2hy%mFR zLgUfV#Wd@leL3#o_c{|Mg3S3ffYKK+VLuH#C;(9kd+&N7JTv~oaw`~w>zunpMV~cl zv)=F(%vTQa{FlzZ1VvQu%pk&x2qWmGXU8_UDB)7*A^sal_Yh-l`1m(uS#wkUtd{qT zZ43W?U@qX+9$w1BheARdcuMCZAL6!#xx+TJ`RFce7&POryp-zS-m#c@2jK|0838ka z@2ITk9$4(wef~T?XTG~0E0HpJKa~&;FxJlq5Dl}7wydlFL51!+gi6c#R<>ijQC_*C zG`#Qgi#23b(b@Lwzp|NQ2CS+>%}OI3-=)PxS|&NuMzR~NgJ0@g8Tw-ZVJ-Eh3RSjc zL1r>BZ_6?3pG#p{ddL>pm@VjZ1~-LoVY2>5mkz6}Mn+BR(Gd-Mppx$DL)BI6rxgm) zUpw^33s?A?U)6j`yPEmoQq{cdT`A6HYO2NHPf4zDgKbsmd?*$lmv>(=zPCjf!?47NfORTCf=oZ{Kj$Vx? zIgZYaysm12$BASf(jYQT+E^ba?Bm_G2q5A}G1JYi%86jdiuLo^CeY`tdAsPs=KJ(> z^?qD$8~g*MreUex_DWp{OWmxgd6=PM=Gvv*GYO*okwF{Uiszxii<}xzxv$c5ePD8PC7Y ziX5w1k3b80I3*YHj_~*WApa>fY0bF5m0n|4U$5J`qvlzdXxutGzP_4h9IV74$5<1r zV+*o;n+v6*sm^R2wqiPqeiqYcYyTfQCJiOOog}|Rq)UQ`-5X$^6P?ATMx%^ zQsQrMjCn4jLotU%iO_mDBomI%bX|xeXPX(29VtBf+ic!Ul`ID}${3%{-jBuKl=~WK zwa=8x3ph7)NY6F=#4JW6gP_`lHa4vJ? zH37NzVIWv;w#q%481=M6EcTW-L5MY}PXo@8HWv7u4+6B;yuRmZyc+96x=Dqlq_5_%gWkKuls-=0E3Z6Xbc)@ltrwHp)NpC$kf%1O@y@qIFB^>0d$1hDJ0m+ z%X<6)$gY0DgJh%n)YdpzTb^#^6FTiJDx7Us+N!M_Q*={_mGhUp{gNy(mK*WDD+2Kb z6c!T=twLu-b_CbhQdE4TnacPr3~E0RBB=tSr~)cv{vi*qAYrbRn1#<_8w4o-ETnRG zNoooWo~W!tlB@8jv|Sa4gJv{uV7O~Rq<7I>E}jlo&DKp2Y`;OH)j@1|r$N%TBDX!o zjFJc@fepC)05k=*d*5`WV~(kDuy`)y!Fk;gIt=l)%GJ0a%5b_ObvR!gT%&G&FP@4A z?59HyRNh~67Kx(3Gz0;$n`UdKi4H0duySh44C zv$F}T84O^?0nUL&5iIbD%T3XWSf&ZqFd&L^LOkt4IrD~W2e{E!vVz)+dBgbrvgh4q zdJ41^Q~;?lEZJ2@3;5l(L7pcb>z}oG3dh_$_{@uPl!i}#&+^gmL3IQ@GnQi9{=qbH zD}I$BV~sN}8w{PAu(>%3R9&09yw_{LU9U-w&OqDCCe$tlAe4ISApHWLC?!tNV7kv* zo@Za#wC$cdfOn1ozz)YHkNj067%f0#Qz;gmOkYwjwmpJl8!ZV-h?%nI&P}h{|@5Vx^MB0Eb~+vMK8OjHMXo#2L|GjrOZk zS2-wna)RPN;{0{a3#=BN6bd*-Y@ ze^e4Xj=wwV0AuY1B!d*tn)4*y8!P@%l0nsQZy|DQ{Op$EMhc7-)PoO~V@_ zF;J}cqx<<2g!*c^B2^ED$V)62xB0}LADJ) z|I6vVi=3`(BL1Nek5!{zpf~wzAfcMK$Z#>(Qe@2o*6Pg{YYq|TjK&}+jk3BO!5W3R z?chCy$sT=6n1E$m!^~(W6yq{qYxE`t`$dd7#wTXUC&D{w?Pm9C@ znDNwZww?l5QDh^v0GiALnt>1sbr&_x(roiUEYnD3@DKytZ$`xRhIXi{(nQ_W#4&Y< zatzRPP5{?+;j7z=1P%rMP^`l>pY6EElJLWn4x^V3oaBJ894Zpt7RbK4we!!>`2{Y{ zF?b zI5F=6R&c*uTqZ%!6HSp5*f?><*H!o$fN+}3bCE^Jhav`^H1aFffS)J7n*~SVoYx8J zve59kcDxSPeLc}vSQ(djT8v+O{`T%C8M=(va1O>~dCMxA*;LKa9a7a07x7M#> zKytp@z^Cvlsq)0+#NiGT?cGYpG2*K@;qyc!LjNy?)R1olWuLE7j*~SbzTBVX@C1)d zxMUxXsQxw!=CM{654NkV)o?GFWmQ{sTr@<6J_z5^{dIQ452Mh$zXf-#)}MHaw}O=OTI~d7Ob9a$3Eg{AI6wu)hhq#q4(0T!TQ51z5Q;>Pv7==2kCq2B4`(qJR)NiU@VtVR^#pfsh z`K&P)kOQ)1Pf+8Be6LiTTOddvhWt( zm@tB=D)XS&34cPox_Y->KC9BBpy_us9@_T0J!qRM*Zxrd zV*HJX3dIqJ&yn&-g&z&SLSw+ySlX#43r1Ij4;kugUHKVe!bXwJcoq$paP%U@YF3|y zy1T$?*&{{$w7i|r3sjN^_iA`ROYi(k-?=(hb+@E{Rv{AlC_ntH>7*VTM5-VYS`n zSDqx~I{jLZuAJ%-IGi*Ml-#KlDXTd!@Az@L*IcN6N1X-;vDc##6Otq1mF{~a=_5eE z31}z!oLrb^z^0xx4%Ke&iygaf18;_dLntX3NH+pDP@9QXPjivUm5g~^Zr(23x*bZ5 zxp(gFg9v;g8Na)&f}RUb;QNbXCFC*7f?fx_pk?V;eDGY?2i8DNX0Fb?330M5LW)? z7V}vUUX1VQc6edf*Vu976$%t>Vsx?}5r0>I1pc!jxD*xeEK;Htlo1TYzNsE$eGVe> zEZRBzMJ*|pAU$NuRUYS}vRznC#Hb*aB;rQH9dT;#cX{`;9Rz}6aal0$nVL>l!Y5r) zQ*@BiFzVD_pPwe&N(cqP3k(zR#O*npRG1+1#O`ECxe!m!V;j2kq23GM7n`=hQbkH2mx77VXOi{c1j^f6lQA}Sr0#YRH9A@#KJ4C+3ThPKg0EO za(E**x8nwfdhx5q837Ka%UXQ#s@K+!BHPjUZ<%eO$8Iae_8umRPeQQ7%5*#9FIw^W zCVlzg!>HFSeeU(sd9^3cU#ANky8HVlyu5Di^u=-=3?_$+#NHg0Yb0x9yYB*%aE?9b=AURuS6Jlbf=%QLj0c{r*TY=v)IH@ zr30}>aNA)$WK9Y|M{{5)6$$&rROv?$fh}tb+i5>Iem_*xR3I5?bI2#A zCtjy2FSATCB`~X1 zAl5Ybch}ZVF}AnE2Tz`qcpit${&}P^>9!+z+;VnVbZRy^k0p^EICV&Sg*mm0-~~ip zk-{h{v3IeQ9L63bN+WoH$c~lN_?U;{k|LP4#A{WNJkXZZ^9q1}kS||<`2aVN4-ySx zQj20^ZV%^eVt22yp3SCkAbana6g1)3mBndo6^s1QUN@2jNstcdJ=4} zwjIg`BaY_a#-luQ;^>-eXVqeDHV`v;zOfx}c7E_*`XrSbjz)K0`ItOruq{k`+(2-&eGP)q=_6~?*@*{Cdu;VR8xCJY^PEgSwMSNa3x(L? zC>VU2Ewzye-;(|SnUdGw? z8ocHAI=hd#=8JXH;T&|coOGXJR9YZuDo)CsdMRmU-0k}ef4{6ZEvyEe?$)46GXDE+ zmRQ;TvSdAcS@n_H@>N{(=~`-I@_YQ5Tw}=U&Ghb*UfNrIu?xi+$ zwJ+NB=%1j8g02~4WGMdYJE=F(t-WTKdU?6e&@S|Ke?~6oI`#FxOn!PlX1%`lPcCae z>$SeogHGF4Z+Cm1CztWJeIKnn7ZH#D5>?#~x6>^GGoj{=sVE6KG7j%cG7u*<|B5UL z6sv6YDQDRYgx=E|U^hJ^OAv5s_0HHC3Zx@M`Lu)?Q(aj0@w$@zCJOl2F`l0X^5Xjc z-UFif!+||`AXftHiPOd(@4|=s0Je!I@Ba63{QFz&KRA*(7&!jh3`JF zpg(Y*qHOGUq|n}hTWERHbbD|8i4@TR6jUU~_iy!mzI_*VN;Yt;d6DL@;Nz0hW)0Py zJMhEN@X;Rk@zrZT-!8t-L)dwAOlhPT1WQ4|j$_s;WofE3})ta-PZ>jZGC6S)}n7VRvv# zj+B{kO`=6C$RmdD8Yh#fDGRuirBNGREi&Y0~p43NkkapLJigvNNa{U=KnV<^q$g)Ac^fKreGju6AasmyO6%pTYG zJyGT$gPez&Lj090%CL}YqF6;AO*dv1N>d2OIh8WtB$9kwV$8JY#2JX?CKYy&CWrH1 z?qPYT1XJjcO3oml86?RLNPwbQvv&)c5IL~vaB@>wSpiiwVhVLU@MNte?+zWS%w7PK zVWQY*%qoIV02??$Kqdd;_#O+XzOWk|02zNql$Y~vzifM{cXP0Xm0=Ky13k0fK# zkj;a!rPIz$2sCFbBEvvKU_Lfw41U`m8)4*}NC~r+4!>R(#wr}_JaX~3iYV=}8>zkE z^;%~PPp@NgL?>9*8%w%)h1SlAO>_BnQuUEqe9#ojX9=uKMgREaSClu9Vn2?GqZ$IE z7e#gAsH7MP(Yh2NC?2o=Me$D%V*xNgpb<*MCc(v_nZUbyz-l1Ke0^Zzpowg8-oLn~ z^WnZ_R6gV>^I8ji9ELcn5f1#=T>d*L3qd~o3UF)ItUwgG^y_}Kf(X2&KQjz6AU1n) za8fcVdA!EE2vug7L9)eDW03STc)-D8%X#9hSfd&@Gk#Qd3hhI=PK;vPXZ$0kr!%4H z(vUY~cGI!S*xCUxo@u(*-WqI!(eqS_dtg9@fS5f<=NU5i$7$g;TSv#U%R!O{pgvFB@SrmfCk!Dx~epPeq!cFx85v)Kud z#R#zhO?Xa~aim({aIeZq5~FdBf3glNnZzX~(}@wqVM*G8S&}Kr_-7`&nc#P#c=_e*0hrq4qc02~t34jE%kbF)vAY$sts+AlQ+s>OE6bCq1 zTM8X7337Z6-0YtRrJGdjqQ|{EaJ^x(EtJTCb9F*cc)1oj-5D|pZ9qI6X@3hzQ|mUk z&=A5sj?SjHn^NPqHCC4y^=C&hL#x)*hFqxDQn+R7^WOHhY{ZRTzBRYup@z$gqsuPu zM`wI*!*TE&fyGPlf1G3EOl(~scQxh79(a=pQHZL~>uyKPI$QLQu9m90tP7LN8p zB(t`-M+K@VffQb3OR5^}xBnplX8Y&Z`whnw|8;sXBZJiHU2kVeT>bA$%bQKuncc#@--7v?@WT^D|0m%0C%a>fYWgk~BJ){Y{TAE(*{{$1 zgJ0q>{w5u0+S9SH?}0bVsh=jaIbzp*#oaHF{3WuN38n;IG*Cq#+2w343~yW-9YF3+ zJm}azH#T4A#dvJ(%T<*Lzh=GWhggs!pWzee4uWN#^1iGF}lCM*QC%=szVkJ;WxvPC%+$MJ>1t2p5^&0u1cI3$9V0{lFyNwV@1>q;d5HWOEyo@-T>}0g{q~D$O*j~cyZZ0kIfmE=9=`8mw6fkGxMZ> zv~*5`18Rz=(V|X_&qqjA207q zGV|rnwX60G=(kU7E9J$vp0xZRWp$D+_tw&~>-vNC5+T7fl^}-SdbE@Cy`Lvv?6_L_8>&6y_UhX$~j_F2Y-Jr~l zEsfGj^K+-I8C}x%`eFQKp!SYMT8uRe>i{W!Dc_8|(?n`kSbj2wpt2jp-c{-%af?p!+8UPq$)X5d~G<%yy4 zM|avgnry$1ZDoRXn->+alWrYFMcjz*7@Q`74+Y-D5$jR)?fhY<0#`yf6Br#zoy0o< zmi@lpPwk4(*#XMaYA|bc@J*7FrAtT{7``X2V{yv=FHGiOsEzfq2wER}dvlR@m{zj3e&{L5T$B+&S4xGrnP9cGWOb|#c@leSDwt-k_20LGx za~qha$Pt!O!((u;=R@u>4F-IP!e=m-YD1JLAfMdLc$#tk9%-Au{Wb*0JfwZeAi)$p0D02fZ7wh6Dj=a4A++z6G3 zO$GW+l-NxN_MERIZ|gW;gX#x{0^iI>zximzZy?>v9M0=MjcVhD-H~>z^Um5*9||1D z)}nOlvbsZmqS#AmBc16_&Q>^cW|wkGd|P`}ZgCc~tcmZ^eCj>ZO)^8celeJ!koG#p znbn9BXiRPIjwsaLa8*~voR(wd% zjzonR=c?1=aCO2${Ecp3pH?|wICB|Q>NI_Oye-SKCEo?QKqBSp0=9w{3x^n5<8I-e znz5es>V|k&N&KJ# zj0%v{Ao9Z5kE%l_R}*{?1AiUdRqECf*z<67-m@ps9}hWyyFrep*pFIR9C1^PQbgdM~DKqD4cb~eaFc4+Q)d=UZL*uHkU9)lEfyiyeEe3}&<*V#DdmK@>`}5ep1GJ2f9YmS#iomS z6x;Uie-(^RPM&!whtw_p61tIf`|h@GCtBD6^95R-7EJ&KF{b9crt1XP z!H;OqF&=59&>s79X-fgGFVJ8P;*}JcjPWGHoSP>XidZCfnn*~JeTE*v2Ebqi{xX=a zdU~Ltm=w(hmdM6~u+0)?X4w9*9o@~;F3|{224P~o zCZz`X!ltj;<0N5MitCR!Dz5etjZ`{A5vNjnUUeLuTc&0!!dt6nT%Tb_c`GTihEC@W z%<8fp3tL=r^(6Tx3|vEHaoa!Bmbd@{B%npI90oP7p}H^bsy<*soX$63bY4)uD%8QF zx_4X}VN~=a9l~Rs4q~%Yc=Wp$M+QUM3@@Lm)!k;IFE+6WE-5u2rYT2v#ZAsGrVVLr zP6dcBS^{p$r$pGok)=TYxnkx>qIWn_gmf55%qQF&?>8%jR0oO-vB;dcqQSqm**G=k zPPT-K+1S;9(vZM8{+Ep)V1fYl4EmZ0WiX)(_fmw;qbtL5`0KZAg8$~nU|t+Nk~z0T$??`I!dndKt(hctKmLR! zO;GHRRyGUQw}3xN-vFK!xK##XFnEKgXU48Y zTHNT^eQO_xN8&fb>*_Z!dcG9??;w+wDT)uqXE<#)MawEssBY2%4KRYO~@wJ{^1r`z{N4-5KL z{lBML7KZ;|mStjRW&dyG;bE<9`(J#i|EoUXRx+CVBr-qgvje=Pqa`+ogf{jWV7hmL z`>`&mmL;*}_1pIc9H(%kmi0A=#3nkusM65B&$rnT$%d%p(`t2y-;a;;^PzQ~80Tax z!~C@F-%=9>CN5KJGCoBv74y>I!0mpcDp0n&7`6&ra32{2Mggnj*GrBCB^E?hf zwg0^@e8b>*E{_unMlDrxRL9e*j`}n}zd%K+9-)IZZKI3_uI;gEl_TmGJ|P8)DItrF z*^?<7n+*U$j5H;44>9zq*#s2#0GZwY_pJiYJF|0TS)8kT@1esHv>(vn;(lyafgf3~ z+KSy$4i|~r_0Vf2|FqD0!Cu~5w*jV(|6(eRhVD?yVZjrQpJZauQQk|kI80mvzIGfA zhR^or&{~^mfib~FjHYG=iOdx(aJY$Paz5)@)z&@Dt!h7tyS?<#PI3F6Oy`EgNRP*l z8V5em*WKDWANUaPx!V~1Ywh@eahs~Li<%izDueuL5#GL zhpPl9_o`^P1J&1V=|IYA8Be}Ig|e*^l8!_Z|XF~dW)P(Be9h<=Qmp?4M}7e2T%QxoOw?pdljuT3^aO-B-53e66|OC7dO!Q-ydRVRl_Uw(SG!_YZLaii zSH+pkWI&`@bSmaIdw8H4n#UNa?L-}nvcL(cJCMpre!WezD7k2Xd*vk-da}aGtpi|&GK}pj&n4Bo3>F{%Zq!ud zfRwE-ML6$%GU;4#iP=_0d4ajQQacULL|x~fQEp?3&~~Z$6xv~_F$jkd_UP1$VGYV& z(V8Rr;fbq0H0t3WwiKTx4a;5@wTmBfrnd_P%KR{?y{VG+TMKsGFDK{cbF05{nL$-F zN(M2RGsV;U7j3)A7SPtMYDs(T@OGz!NPE~Fu}7}If3M_e22U4=lpxK2qdy1*pb_le zi~`_Yd607l>i~Z=sG2lG&~^fJT)60yOreydD;vFZS+%K_o%M~$Y7OY%MPnDyz)l>k z8`!P}aIbUyhXvU(?j0%M32~>0OMpps7OH)v`w(gi=-de~pf*?wOwb>oVr)9DwdZi1 zoYQaxF8AI&K*6<2^DZb*R}`^`yK-HhV}Xmw&P*#B(hL6pbRxYKgx0#a($WYV6f_h~ zNPjxXg*&2?_nRcQiSD+!)k9>LuE=064Cww^;1{wIrex?IRbi1{N^tjF;g>^RB$uJ6 z6)OrvM)T!TsWl9$@VPjqGeLQ-8+u2;^A`*~HdBX3@kS|N3KsbB2*zCrlEiZy(wMFR z-gfiio+W%1*}fDEOXK3r*QUL$tF)3O3rxF4jzh$~apbl;f%ySZi^Qd}D@ zYNj&2NO>MDy0n}&3~gY5?9?c9hXSVbmr}Gx8w@nVYHmM~Ifiog)u?Hj$PGxEUJUR}J6VM6dH_uh3P``t6y*i=L8w|xK|U)Mc!^KF z_Ho%03olw%FdwFb^9x=}NU|;2Nkt$vaxchW3zUOms2uhiz_QCSA=r+Qgy#|leIhR54h0NM8bU-0p9bA$gYU5}};{G!w9BAt{qSdq7* z_F~m7v-ao6G7x${v^z)Yi0=H=#PA->G5r#4io)F3vCb-btPU^;(*d+ij=ZhE_|1!m zX73X((PdY8(Fh2voF;dKp*&sdSO{%S4x^S&;4}#Y(xl)t#A-{ zN*9!S2yUJ;*C0zDWQmW(y#d081WRMgXw^GyH#&DBCS|~Vir&0ow5BN?J)N^I5f*} z{2O9**|hx1#V*V_Jv_SNib!{p=zOaHeKHV8`+u{(l!7`yz~IjNZ~p`zKaGzBvt!(h z>;X(_N64($hY6%Sz zcV=Qnms8&DTybong|0&VkX;8*A}mtv!@%!oeN#a4Af>^?>L_s{0w7Ip7Y5l6Ef~Rj zFSFxS!`)aP-JOqz2rWLG<%b6!Ifnfdn(j|^!Jn?z(=iQL(25Q&@DjITpqRtRJs7Q1 zVXzPF9JL+fM?gLln>|M2n9>My z03ebLX>nZqlv%dlZ$BEm1uKo~f0&9Hnf|xd%go63-`BYPzX;iHkDC^u2l7}NePvB< z2~zrDZnxF7iX-U^%%oS0$5DLf zXLoD&>%w^kY9vB63jGxx6gfy`14Tr?D2^br*Z>G)y9vyHnL^Ca{8$ED2POg`slMZ{l=ne44*e&W2vSOXW)3}QN5?8zo^v$#U z)qiFO4{m6hjNIZ3F)?6Lc_1}3 z?g*WE^K|TEfe?cn%EzSV3?=w(n?WWe^D@p!B!Y{{`_DD3fCPc{5X3E15uteHA!eSM zQAVJBNv3>q%&4*j77TvDZ_S>~d+BttAhCCw&mOBTppFkOuX}5@D5tSK9V)CFSQIyR zH$W12DERi1U4{pBPZEe=#WSy<8BBDreg;Od$i2u z<7Fhy21q#2TO!g=$YF3rv_d%E$eqwM06bx=Bgkb+S4^Fz!iCB53U#92MYRIzD!x+j z${hgx9}@3@$f@GnoKJ;fagTqsuZt_h9=Ogt zhq6;&$I&hOKtgtclnnu%pfW-&|A6!M@+R2Uvwq8?YTYaN&+a48K}-;8sW8r4fSVJ9 z_keYR_ijx9#8CpN2y}Rb+(ZQ8y$)gtk{)b=zOP&Elq8ci2MCBPbny*}zaY#=%VE3q zMY!;y>6~^7@M#LhoR3hEe#>r4_;Fcf*S62g5o!VS^ulq)TrZ@~Qar1Mo(Tm0$i?6N z@!6uQtkK#5hb?Z}J%v8YRat!Ekq$IaQk5vh2Ql6P$R*vo-7_L2+C`8_kuNcvNPgHf z&tSd3u-^j+!3`DY`1vU^N1sq(F;Nf$KYI2bJ^l}zpm?paJx9`{^r?G~gcRxxg~^a90&aicM0fWkpqvF>&`9Pc41P@~}=6`jL#XnhP7> z7mda%Xn*Bfmg8)h#-Q;76k}aDU7@sh^RQlk@WgV59u%%IlW7zBJDQGGH4UpVF*l(4d0 zf<|DhDPZ8iy)zS5d^q0pd{8bSyZ{R^VBjTn+%1xUaGBgsjsc`;07fbf0Xg*I`jl@L z1{=xSthwxJ6axe6s~ROIQ-0T7S-GVR5dd%pIC8uy84c;K6krH2Jf{p+xTRqbRs_H2 z@V}wieTp>+1V9O*J&J)&k+f~EqQ>+hPZr9Ss)!@=g$r*iE|UR(vm$_T@kA0J)Wn}+ z&9Z_Z>YSP{u0bw;1qOS825HhNz2gahss_qEJVTp)em7Xqv9;?t>Tk~>py08W!8`>}D zH{mt7#nHrfklY9bI(A_?H{=eZG)Ch~wZVn0ps>~jX$5h?0c(UEQu@8st|Fpj+LS_x ztD@5ksayY%I%7x7ZGroa0jLl}hjt~#-HD4uk_OGq^~Xh6bER!0BnbA z42hK?l9Ij_{TdjdFDj!_d!}LeR|59CGh~m3)rGsE2$Q4A$aWZ;V`dgJ~7y&U}->OL)So{p9!HT@FB}L)WJn|%$#?YkWo7(9*?V~z(lc_HLL)?k5L@qs@=yaE40(2OXM z_fQD>*O=n97oNL6#UN9onVkNK5JGbAYFEt1_qA!G;I z)1gm#XI>W007QxjQGty|sMIZa5xzzT^sTrFM!A~$V|`i!P4Zw@;u%^hsP=%_U4b2T28Uj^-4!1x}R?M_V6@Qle`|H!{Ty(PD4ZC&3G*u z1a55J9XJpiqVjUWwg!r0u`pZX8NY0_=;40MJ-~Y^&^dwz%jS!=%XO89B{?Q%T+hL)A9 zkJs;2D0FKEwyk@%LiM(7mqv8UxkIo+xx;5w7E=TSc|aLSA7ecstS=*6tY?#VS-9=0 zeXq4nez+^~yhs2naiK&Y6%($7tlVsn-=NP^mExk(T{L{nJV9FssWv zCxO6mC^)9+0Hx2>fqreE2?;XFV+z_z(2+}McZUUB3zkjg1jb=T3=a8Ap8w{ydkGm0 zZ|ILNkc4)5`!*M(vcHTFF>^>lkQ<@@0gIv6CbYjE4+MikSFYbT7{N^<5KhxEO2p^; zHLPFCj|XQZPeKM zz!f+_qy9H9`0b0u>M(-nteat)&Uve}+4;!Iq7H=>;fMu#3MSYR>ozaT05Se=Wjku- znVFUlAW!sq$xAvct-(Mt;zR{CcgAMSIH!XcFE8070h@m)oP`2yZ2=o^^^|l4NWgwW zyjR#hkyKa(_WD%)6Y)s|TufMW`vrduu%h!zcso206#|WPGn~%O(4Mpt6#qT{`KR59yLsEltIPXj^$&i1mJfy%~DROg|l`HBI zwafJGB&L#Y`=ElhijrQL)IfJrDc?(o1YU{!ch*A<-x@X`^nl7Lul%(>o_@OdmSzu5 ztKf_6E`?3L2B+BU^z`0tm;AU#K`0V3?$qL?@?nUa;n$LRpWs8{J8oj*B+~~WCPX8zO`4NrOjFUhhg{!6vO^j{?2%&ZLmXGhh4B;LfW|H1Fg)q`uyd%Gcr z&`VtQ3F5_c`x{+y3DhnzFV^8**p4$F{8lSH`-RJw77~pkM5S3G4Uxw6>an#E`;ncX zdY?zAvAurY59f`wMcN$4RZWrXt)Hcm^CB};B!AsX=89N_4BJ6A#ic^S+dozi7LCV&uv=6 z-NiI;UhgWyI*nEx?{h)(U-i&~RsPKU`gibsYFN)9`fC(V!#oNedHrT*oij#-8@1r6L12}dthznU zyGORk>MNDwoFkRet#5XOM>o`)%M{ z-$7Qp$Q|cT?1yGXJhE2RR}dR8z_Wy?s5$5&Q^azY?RA{VdU<`9PC126hPB>arD1|T zT_4omScO%`M8x!qIQt=0#SkgQbF&nfFB2djfCfbIlLh3ZgUEXTO8=>$R)uP3uvcCO zE9oV_)5o>3e#BdkQ|`h(A9VLo8>D|b9*32m?F1}<4-I4ay>Yh%&b4oeC2&3KneH|i zW~E}5I)U+VWsM|@Uo}v!!f5v^IK|6&bTveVRC6(Jnam7~Hb&`pOo0B_?>J9`9w$Vk zPcRI+jASDfLZYS=MSBk=q<$p=5UaL(&$qHmDQPDpGQh=195?dyHY9?65akm2b|DE! z4M(ZGkdn_%WembXb9i z&0VaB{SK?!3x+z{He1i3HNoUT6Ahg7V0PAFP851DZ!+?4TKj}xi_i-$DH?4**3qSt zP^9g_LhatN8)hlb5B0i(%7|j0r{Fi+&zYrLz8fp^GaFJ-l(cET30PiTQ)xt?nRsLn zYmF_siHG6c-v~H#1KJv!tN$qzWB&qNPH|})filyTk61R}DB>voeTni_Fe=-g=-Dl5lmzgI4%1@{!>j9xT7lG08hsLn-nM(iY1Wk`?a=`=?G9>R@W9!-8AoOp zBZDt1*MT5NJ43~hyQ&R>t@USG2Yd>q`*XK@k#F}~SdRVN&RSdrO^8V$C6d%Wp|NT-m4&G0yZ# zP|4hjUn;&QoPve+_SZ%p-iP5=ojU8x%s9)M=a%FqDe8U*7VQH4qzI8GXl=mIiT)VR zmc0tb!KRu)wR47>dfXs1)-Xid=A9uoG_fzo;jqPO@BpFZ0WjP-iLmUBO96dP*7I2@ACH&UJ7urbp1DwvfspN`?8SsuKkqSuwFA#y*jPw@i0Y8&SJwOArJ7m7w zmpfwm2Vxks6Sy*&M#&B(*u3kHgB5Nv+Ojr5UsxRZ>Z-?K_^9_WnZM-TOj8gpaDte) z!%6CR#Q|ySVR0^dqos6}U$)&MmJ7}&hHXM3Me4g6Q5X!xrpA2>a4tHW%BE#UQo;)- za&gXE3V6`9qAEO8!>;+gRrvDu&IS7FOx-Ns1Ii=pW^TGc9- zk#3bNysbmlWC0}xg93_D93ZF{?EK6|>Coqogd`xFinET1|KN^qjwS827)(THAnK`Z zPnEO)MfGpDQv)xr;a0BU&f>N=gh4Df5<{2uA`Iix?dj4HfHP)i*xR)bE3^%VRcl=D z#pux0*3vuGg5avp+T&M%JCYm+*tn}%-9mpF6-zpC6t zK8@Tzi8DL>dYeI#QVH`!XQsz*qSzyM$Ec|nfcv=na*7eA(#teb?Gt6dR0-(=4wKlT zuvw{CkZ>YU2LKV95!r`vhDVKmI`rM|%=9WXaX%nlN5mQW1(0c%$y)EN>oul8+kzfO zol0ASNXC34K1hW<6AaLNa2qM|F`P}{o-#pp=LJF@r8-Mj0RUzojLmuSd>sofT+A=M zO>Yq(Lf_7OfTrhxVM)vp7zq0*_`H1!&b;nVx3Z95cm_#SjNeEO6^B2lg@f4-_b{UW zwI~<*OYla-p;RcHyJ-?I>gMR`)SXfk%;=<4;Vf<%Ti0K>8I!TtrG_|hh=HtLQ`TC( z6Ky^#-(r}lSK98pOy((Ox@PCZgtAlM>Ort)TI^Anjq0jlLRMBeOSAS^8q8;sdkT5N zdet!<`=K-8l>?G#_e@P9b8A>&TpM@z83ZP069mB@=_0!kF2JQ{nki*TXHf{F3Xuma z%(axeouJ59ATbgi2|?Tda&R50ZS@qwHuqPE`L5A z?=SweD;EYn1Pju|TiByQC;9biKwU@N=qY{nIWGg0wIJk%lNqaK=1fEp4;@p(GRRO9dE z0)!k6n)~J_f|_Vt@A3Q~)~mXU&&v(W^}8+7vOYezGQHcX^c&`{Fy0rZRzdv*i?nrO zPJy=A;|AtjPef7ug~{OLtl;RzAOa|lR)7sfW(2V+L-0WZ3xskkl59bVugYG7_ZMqx z+ClfeZh#cUe5X{}?3H(9UhH&92dIz?rCBtlF}eBWi9d>o5G%+u%~@tXb647&yE!CX ziNtqKaZ08!Ae%Dax5?z+lkZdcxdKt(BKr7(3CjF%kW>X5SHK9uniM&~=a8b>nDD)9 zV6I4wDZW7j(OV@JJ+cl}hkV~mo(mp&i`>6hXG`HCb@fSQmM8(O>KYtITR-Bo>WU)eDnR1V{ zyf@_JsS4rx4rX59=Pydv-n;F^Ow38E`LPqin3gWS|B3PL`ivJ3@RnYZ`}T2)I9^7W z<`r~{LJ^+w<(1x&L4(xfdeEIBG++$7$x0q`u;n!)C8fp{dJLz)C62tc={>F9@RXL9O(T)WHtp%iiA0BH!xO63``5K z4Nw&Wf0>TT**6|bJjGbAki#Eq>`uucH(n}cACE-r=)lQ8o;`>7)4M4zSc+LNCar8aXuD)ONOsA^KPwm*4_R4Zo08n50cwy>WC?~yFDAS zI1R0FQ=Ke1y<1~u+Z4XCqT2RYvix#S6fNCvb**A(V6;cnWbzzW%~OlEyiF@OC9`7l zcU`a(&5k_r`B+-nZ+5(2o~X-1q$o%dMF)VsE`Ok8A_NmT%{=mrBFQb|FAMd_)anJ- zWTcr_L&^)%ri)ENLaTcYnD|+9v8Gd_R2gP|0b^d)49In`fJMftO3i)eX7oEeAGgv1 zN_@!7XEexgSd`{8!`Ag4&(FNMw0j3`%ri;Urtm#yrluGuuhzU?Uzaz|_Txw&@kPeL zlNKX3>g9h`I4YquEdar(aT0@NG$}dFi_;Uyu1|1Jc%qWv*AMHAu{XjP!tfGg>`kG8 zNkW>@m#*=Vr2NitkHb8Xo( z45QdD<;_PzvLQ!J+9y$PveBDzrb<)4b_DSGRdf#bT=;JYN($h0MCkHg3KA7fyFS&_ z44g3W*u88F*Vwk?LogwEEkcjQ%}hLt8oRtbbYv6=t0znZSO~x*kOD?#K5#3nFc@zd z)OvDj4#z&-({-h6kam0Px#|da@e@TjWSt4ksoS`$s_KAikrdaJyH*~X!U5B8qqPu& zOC9uK5mXZ3edFlzsa_StBrRDRGG`eN*8&_0aWgXuKqIm}EXo+jn@Ny2gnD+5u;;jR ze%*&t8E`HHG&}_B2+J+maC>&YRNAb_ZI~*hUc`Wf!Af~LFI8wR{c}PDD-5WA@z$`b zpiRivMxKvJAIBIbS#7!;!(exsjPPK4S95KD=*5W=dC&ih3hQXuYk$+4VfM=Rz2&%x#!!p-@Q@ z2raLTChQ7Ii>!|p!j`giUw(qDJb!_A`^-+f$XDCx9#~6Cnkvmi)%zxA2igvO)vRtgLWp}#FJxv)`Y_!V8bH>|m zHtUEbBsR-R$F(P|Tp|i{(~@c^%@Pe_{(vP{r4rCS&@v;5D+OFRK@J9P;9LFIA`e>Y z*j>Ed_Vmb2St4q{(CmwTb0Bt2kvJ6w6P7?4V{_~L7=@HNf_mocLPmb@dVoh1!=ULeGoG=0a|Xqm}HWTMywhV zSzH99%1bROkoSSBn3*|bj4#LLEgdsO(J96J{vh#`JO=?#Cj(=#>j?yX#Pi989W@J8 z)u0(Vkq)NEBaj1=3VotdNK!I81eOZ8><6FoC{aRpBHT+sQ~;GI_6<{>Z6y_+a zF8fgJ<&Z&^!WZz6X-gtigE*Dh(@kP(|2hyN%&XH8i5Zc-HD_)kyo0l*QPJ(y0xz1t z94|h_mo`0OgX%yJWc%roU&y^v+F^wI&m%YC+%JC?_19+=t_K+5<}Naf+N~L#QVL!1 zb0&W(hLv+o7zL|#VQLA^QOS^RB3v`d#K@TW2`3SK4 zmRTha8%d~tlBTnj)J!d%xHviVH9%MU2HV5KRq(TKBB8gt+^>!~kv-in&@6W|2NZAP z`mjClg$fO#fIe8RFu^Nm?6Kmd**hx9SirQQljx z@`3Ubba=TMV0en48`@tf7Vf?e-@rfFx;SoOv`5V3IeHyq#K#unKDl=x=riOUMd8gh z@UTb!Nj_enlS3~7%yF}1nQnmMYr1M!r@IxehAB$vp9ZJDlOA0;V`y{c?88We`+m*> z;p($8l`hi-IiZ^MCcLQ?mKYm*y`U`c=Sdz{hCJH@wC8KWYA*~C=1SSv;n_86*hg1> zBTAW3l+}n6DU*S?`&J|X@4r5Mo!^2rdqVE$q&B=EXpq;c=?vLEzcTFBQzh5fa@5}? zC>Vt^NRV;3?NJi(=cxdPXL#S^wmjQfd9jMJj5mjJl+kjODXh%md1CtzK2-!q|EXA0 z-e%u!>J5_HKkSeJd|3;)?ej>JE-?HJPw#>~slSZ?=ezEdR|>grfs-{Y(^{!R*M6M+ zsIw}2?10mBU5=r8eo+=o2o!Fvj?LkBQGPA}fu-oECzJEGrAs5X0d4drO_br)MK1xl zy8fzIHAgJ;Mj}?Vz87A;aj?hIYu3<^n~$+sX%OvhZ#WjZU0CbISpn{ca+W=oLtwJ@XLS z1Ta5THn~{r0Yz~{bag10@ke@7m61*2jszQ6;SPh538ldFD29s7J>u_f?isO+4+VUa z^YX*`D*=cZmgSe^wbsIOq-A^QBOzo(Qc@?9gMf#7I3_4n25ebj6p&u3DeTz6T)vP@ zV16ybontRV;~vRa2A?rZu_F^yi2R^AD+o_8I1h%t?od>*O~M8UL2Jot*}XLucd`{d zg4|zfWyfoPA7#e2I@uCq4Aro2rv4Zj%{Hh#++mbINCC=gJ30E31$x(4zKHiSL~pj4 zY-`~|dBx&?Q^u2_G?#RW8b`tQqTtpH!~;CVni-EHN*uY)!|o!q|GW|GIzVh(tJ(x7 z$u*Er@!Gm_+F&C<2Qig4y9YL_uS3wbXYFJ{4fg#O?6~($%KJg#jz0Psw zNA$d^R;%N8g^ylOlGg&y>r`^vXbQOq8ebi+6Xh!e#wx0|U zN(czwk%yf3ij+jx>y4rD5>aD-IFMl$bgbRT+!r}a;uyhl#Nx1TAT+~5xK-djw+zE07ju7o^Xpfe{Bo0A5 z7z1XWIov447O4t7HBwH;vyD)nv!{SDrKM=zFOk~gC#mSbRt+pf1IGn2ua{>cqPt`p`M3#wC(2k42$K| zhd$8L#?)AIFxl=Hx(Z(YG!EnS2S@l=+Jyu>{sVLNe&sKKCf*J?Q#J7aXb8Sog_C*W zGy}dh%_*h!8x8IeP}>8TFKf~QL1=>-0A;LquF42z+}>1FJg%tN_>P7D`_@5vjhRMN z#IJfMhY(wp*!#!#^asu33X}0uMzPtZt07PIeeY>fDU}*-1eOgb(sVPZst@0PXw#$? za20iJ5^_2c^Emwci2Y$x=G!NMV)t}DyAN!$lnjcX9`kb|@a*z$hhFVhw@H#dC9)Ii zOgK$c2LkNzM5YKK;1|>rCVvi&=)Uw&FOTe(d%|KY`S2UQ+#~we-H^`HLAZVH`^h*5 zz|BA|`Mcrvg#y{a>kSI;lbd892@JD59t&48CO}ouN8L@W+Rujba>PQB|J@H0`M@X% z@A|`3J48hBTpM0q`L|ebJepJ3c3yV_O)Q`pZ3153=MSj-zhkEVQmC{1H-Z4e|2-yc zT0<)K*K7436qf>Qm3a?7P?>LgB~!2V=kuJlHS)%N7crzVqa= zY(2{LzP~7X*0qU$VBP8!9vS3Btv3J~NQ>s*z3e>plC+S_`<0k&JhEXnL(vXupVeN+ zvw2krKZ)k8&Kq0q~27pQ0VmY3CoNUQPMOg$ze69Y+ovz z3MtBoV+F=>xNmjjcxNA8%K$VrFS_k8I8G`V(*UXtw1LUxOP1iW+DsL6pn zr*F-nePF2Ak7aLDn#uY|>(5-cNibxTb_3B`7_t}8QLFPbK7kM?1lI+cuI@{%C!O>J zYfjEEA?OWselX;WeL?@ab3i;RPuNbe!ab=VNE_|Ecs;5XQ6Desav+PET}`cTXT~67 z@Rol+0+c>~q7A%S@26xV6dB;Yh=1+%-&4ZHy2e|9e`Yr|z5_HleX{a%dT-q*`60Z( zJkfmza=>yvxVv}gfV15w?$1By?l+XHdX#9@#F8FLn5iKKnZ_Vw#j81D`K#0xg}Vbt z=Qtt0Xt17sS}%-tTXqB&!xQ^0kMz06@sWp2Y80Uz)+bBus##U)Gxzdz3-nQY;}SFq zo@7s1M=80KdAbPx=|b{`M?a@zby%4OSV)Mw3Cv)+84;qc5>D5B)G;QKfYZTV5yufh z5f*~?O}1&S32^sSlK(ROH<6%K(133=AD%dKKyrX6^u5QM#i4%G6(EZM_Cd|`8D5I> z`*MPTioJ7KCA-Vu)Xq*WEjeO^{M6}z|}FMT~1>IaK36T^22xqzVfVf z_u-e~dM03xh}FoJ&U|^|M);MF)c2JSbMd`zv0cS*@b%x)s+bN<`JFOe zrnSn_#{MLhrV%7C^9-uP7n)Ts8B-9 zxz1f){CaTa$Y&YqX&k(Y`NsrbmKkcxi}PZ1=`7uI(JB0Ue9)6s|JB@w5ikUcQ+~^2 zmJ+0Hz~`ER>bGMQ_BNmR*DhitbEUGAbj3zDjORA58_c47ZS;Hx2X?sn5ui=L*Px0e zoD^|&vcs-=w5uyed0_Z)EcG+igSW%=@e>j(Brx0c$qoN0V9xsY|HOn2fl$kF*{wGA34aw_2PP|{kl&Ump(E@a0}VHs&!kd)#hQN9 z@c{pGcq#0o6%b0l$}20f;=C&`94wB8;2nH6mV-}+9z=J4PFt)x1YS5iXu-am0)^{5 z{$Jj)N!;F|3gPSGr<{L0619*|gipW558s-_nb98uVhYM>=RqgKgx-TQVQtN4XM!&9KXk>T?V({ex#vHK_#5GUJ>k zoVvJ~u{im#Hs`f3o4@Ru)^N@|s)Jv`L(ng8gNfMPJ~S^M;5e~n`u4hg1F*mQO=;sxIu^hK<`@$Uqe25f%P76b$%E1#npF3MR&46~9&3Iul|7kbMgwc82lWS& zJThh{_Q$uU68-fm)rGlYC|Y{%oT2q`Qp9$Mzl>$j;Eu?ywZbT=FmF8&GBju)zEFRK z+<^dCb98;bf#t6__MNyLmZb<0z82w+cDv6>-`~N;0DG`*x7Qd_er&=%W-yU&9d%tH zC&P@GrU&HLqG0c{vfXD^C22x3$u?j%j*XbnJ~}Qp#PqpSwKRn%`xl=Gbs|4IkC#py z^aea_TH=T^X-hOx)TZlr>GE6vXYN{R4+>}f84chnL4)S<3LVFLF+qIQj(@n`=iL83 zm5z@#K>(B(%<;~Czg|oIz5A3!uQdD#A|+I&0aSV57h~E(P3&Q8Nh&LUN#%(TJihMb zGC`*LFE@za5TpN}LBvSF$jr(3zeFM?0(LfL=Kp>B)rI}CR$HB>)jOA8J+oR~{IjM;3(SWq zb)g%A%6kkbwE+0?NO?tt=41dM00BbY0(k^{*qMkZM^JC0F|(#Xf&}^YX@@^T6odo> zOW87*K^?&7eFgwBa%%qm2m%7iiUQI|5Fijh0Z2apdk_--Ok>cN@N;{><^u)@Bz&1j zn_K9BF3v(*%Wr0oy8uo=AS5J1?%zScD4PTj6c`}Dr~dM?2(6tF`oHNK2yh`n@;yKC zfx_#M!H%hiCoj*>$H47`Z@Um;8nSr)>?lVd4E_X!5cFZZyDwt^I&s*CH}kKzYXG9M z@`rkG!fn74*g^gP*Dy8m3?Lz0(04&=!U+DI=K#X6Hho5J5SR3I&w2oOeLJDR2)oFy zY8}0We&m4z-ynbc^xHDIA&#MgIe~EUY7qL2RBZT?;3L2R0@r@P0=46BJKf{ZAS+@`1)yhz(Ju|YzF_J6MBFeCi^ zZ)Ubqe{LpS0{XiRe19%+@)*2AcZ~jdbv$4X=HcZaz#jE7x{-bU%E1UJ{2>q^P>@l< z1$Y1u@avF&Zh%C4uH*P+19px(F@Zie_pSj#I~fT8<-y0c@O$u)4nYC)5OU%GdVj87 zXK{P{0s8QuL;)`eV&pNtdmDGs3E}*%Zm#hGp8(YXZcfPo0KUJ!tV~=+8N4{AmS1*m z`PHUsD=Q33N+!Qv#(o^Bs0Mrhe7lGW0CWWr2?Tx@7=ysI72Rt+dfIo>1n^;3MtMkoMm3;w^|MhL0JLQx z2zY$@UTYLYq>|5v;KfS;b@_;{^WA{`gO)*!0yh7bIKMX+4WJeP;Z?7P+a*BfItTs` zm;9%Vz-|9+Hi3$PVe`9n0_4UPBS^sC`|gJB&6XVK7P&34kay?=J`(WLn}Oq;4#b{a z8=wy`rtin6umHeUxU3NSwDl=gP~dhZD6nH)_&e>p5aSouf(Wy0dxL}y=_ z-~$ogl)I4|EhAW25v2s*hvc8;X{o>U(WalZh046d9y=%%otU?$9?^&FUv@m7GD6zd ztOczW4JID$5y%%u@>yPPV$r}cS4~w-C#(=J^25JI zm%b4jwJhbOUr-N5DaIwtL%KsRGa)XdvX<#{OyKA91OpN7Cfi_EtqOH5W}bg{h_q$; zh&uqD#iEwmeD|z_E#V|gXnca%&K~FTBC*lH21S^iy{=<%;Wu93qBhz-kq<0LKn+_% zw(xQ$;X!G)!Iam7?J>O~K1B9tn-C?muZd{7fE>a__iy zoMLWg#o9IKBwZ0-!;~&y_14aot;un(ce?DP?9Us^s@wd8Fu5cx7pV~2*&>j-P$esn zKk!Bh3B{|$yYv$1wkzK}AvrvDU?uFL47e)%EbLlFMfg`}#dsVNw>$J`*h~EcddR!r z8N*a#N$WDg_NAQQ8!&MTQoYa{=zQY{A$Q`d*nM}?OS6n^_fkx!cc3e>({^AokYtCe zUA`zcGdU!h0~S)?jLlelkaf?iVH500RtK+c?KS7&eVMcVEMH%$u#I5)ng>@sGU)eU z(*EGgIA#NInwyo$wYT*%twJp*@q32L`%wFo6r?M;Jc&lwdPK>w-)bq;jf(_>*R+te zPGi(Oh)h%Qoj-@Cu1Lcjh=moC+h#n+_lRXRNfd+(D8K&Qf`*ibj+x#ke%4Vr@`PkX zOFVu-zt^Qx@F@f1L-qVACAB%`oNHCY>_`rZZN_*Trng9swI3CaTtYP+)+{|VPM2bG zTr5}cs(tS&ZX01E()k0i4xy|Pjv}*44>|yg&$Q)gbO`CVd_8aAhuKgZiI~bFWe)YW zXQ_P;P)aCGd{UYf>;^gVcUh1Duk$U%2JlK-tG3h{h9;s z=$HCzsZPs3V1R;QwiW?@8@Nq8$SSC5V$8|zw`ZzM&b#H_#=kNb9w3bH5;mwI;>==5kx8+i;g6Zw5BpDGCQeV9j>`~Hj9Kj(2( z#>Pft7n_)6r|WM~VVRI>{Z#43prpe&^a*#kqpOwJnhaNHDP7@?$fYATS0$&WT^0s% z%a#9^g$7HqvZ=VQtKh%3E&;@O%{;wt+_r*zKJV3uH?Le=Y@S z(Jsi*BmJzx5o)?tB0;C`&R$FAl-9MPgW2+H+2va7>wF5+_ew!!H||2xk*?)34b|GpgWr$B*Fmbzv6nR|i5^ukt9bhD7Iw|Ho8i5xT2Q0%ucyH$ij!)Vd;Fn5rq-W; z1oB!|sjCpWYe+gH5;{pyL1hLsC{(|IX&p!Tv+uDUnUTny|njE_( z%5Km9K&?BVVBLIfh$v!_;JY!3x=ix)u(ud(t&AKj;ABr5$`e>8wHsO!T|qU82elvq z^E~rqpC=!nO=`pdtdVb>3A{O9{H&A7LFIfX^nCk0R6aZ-QW?j!i|^=jfxEL@Ei?VpBTF4mfiDiR0s&8?wa0Kq4&<1TM>`|q_YwQ zIepy5XU-p_!Di^seyQbfqtS`A=#e-cEwP|Vh1hoY=EphfkOqLO5MZ+I7E#hvpPCbQ zGd4@`$OYBJG@67cH4(XLYT=3p7&XC0&b^_)D;xC0I_Kr*O;%hwB^41M7Mw_L3XzR5yX-mN**-GDNV0Nlh zCVzxj3@Ha977?GwgZPSMO=Nqdt?0;rL%K~25wSo{L}j}P3X2jypl|Jb2+a%}a7=SQPD8Jw_Cv(1`N+LI^(SaG&)D}cE||}kM4i%|9VC%MZby6IN|Im0tS(5 z!7gR4+@KP6f^r9-W}URa$6r*B5+#G%P@fo1F90j@8hk6pGJ>>9XNNuIZ=-ciG3hy? z;im2QQ6-&0nzy17_{YeY0wL$iFBx#$$@XLzqU6AEKGvY(yTWKp{yS4MuzCA?jEOkU zO4|QCEvx*Ji)^Gh8)NXCX$eK!QdZ~q&R_CtcWFJsIhCTLY3U7DT*z`R>ehh2*0eIB zT;9`5Y8BQfgrsdqoktzKCDO9VbPtPNl~Tn%{~peU(V8J%(1m*qcYu*CXOL*&4-q6y z@+0Z1SbI?uqoT-pD)M+KC<)f$%@~L(Lcg$x)Jk|UWEP;%7B`?m&DK}dI62khsbx%LKO=CcH zy)goX16aFOAh{zER_9N#aCh$Yux>;0j5p^8xJ!!NAOowxdLr3qxu4o*r3<&4c}g{F z8-_prDbwq^ScRZ<#cNab_}-ccd+;yHB8lXD_Gq@QewhheRe6fwr)JH)wM9z(1W~kw z!}rAF+>cQ;TJ5$B?~6<(P^EZlvZnLd8`jy3)v2j&+RkE{(ivkQk3sbNLH!E0W66zO zc$JnT`)Fi47)bKRl!}f9kB1m}{GRl+lo8_^czi}3pXM~-%mDxMmNM=y9mpJ?8Z6r`Tw}7%l12C3&A?fv6h;L16=8ip-u$$h+V#=Fj00GU2?Rlw_!^^^ zAtXg1=y71+%yhb)59;_x3J(OnlfHvY2a1EUgm<0*H~!1h{B?+-gNXJ3KbV!<#U|PF zM`0T)Id7C7pkk92O)z*9ySqPRU3j%h-kY_J3YWKxjUOX^%QUo+&Fe%83E3=^11q`< zgX?r7?!|g+7FaK%kE=8=(H8CWM&K|1sV6fwUbhozYy`j@-8~tnkc-TK{L1gAMm?C7 zeq`AKPKkp3C`%F(;D8F(QE6@KD9j1dnH`$(ZJs7Bb7oI6vhsh27tzc6 zoTss$ExRnDKOJ&5rrVB=R3QEkHE4#%0hYZHO;om=gCf7bLYo$}g2H7@cdgyFt3%?mdOMUGWf>7>PX&eeo?~ z;XofVQ%mYNo-*!1Bd5pBZk2eve0(myNxDFv(PN)d@oz6guC|VdZ=QPwHDZ5tUkAL7 z(#ONXAZGPv=J2t(uvlmysbE&lF1xu-7#0a3~id& z+&j1F9EDyKD9*@&ZY6ITWzvdc<@0Kh8j#7mto1G@jc9p&_gs0pSsBF8XyB3CzGE!Y zohCoY?;ST5mQ(!@*>c`#PY%Vs@}Efa`B&G3A>QyS0#dqPza5M-syxyZK2cF#Zw+RY zwk3Dy3Hy-t0cCZh_k#r2cW{G${+jajF|TFeNTzjpzslECv-)FeuPfKe87U+~B;%^^ zw8ud?BwVbJ0#Z0F(1~%gW>dGK=Art7X|f1!F^|mu_T*eyUoEB-)aI{AenH5s!~Veg zlm&?VGQr~JjFUd9@rQ1>ccY0bg@Oc#uX@OaSjQP+$G&59;qmfOCC>R}*KKuk! zeU5uihO`#zqMVWyW4iCm(@YOoZwVT}(am0vKc`x&_Xe#e{}S0GP5ju|)IEzIx&XvE zAgpSyhrND9gY@7M2}P=$Ue>;qFSYKdNkYo27Hy(!j1+z+v(qJoe)*x)jk6dEX-1`c z%PR(Em#OpZQH6S^o!|=N(iRmZjf`%Kt~X`wPP2?|wcSCOfp;HM3#ub1?fMq+VzCe8 zVkU+HMuYuIZoR__$*hOIN5yT9Z)fliuzv|>y%U012Tr0l2@DJbFB&AKsQ*<7}0Q9j_%Y4ch^Q=KGQV`iY8Zb zqan^QnA0a9SUrz6BMML1qC|u)6=SpJdw(+S$g^J_hh~tK)WPsZuEL_md8a^UDMBL>nw&r|&ywdwStnUq_**l*us{bEj z=MXGPv|!n5+qP}nwr$(CZQFR)wr%rW+cx@DcT`0U{-B3B%|S+BEn8l>k zf$c~*iVv96jg87`7w%%*hml?d~9kl?Mv5t zz?s#IGX?)h1bLqrJYQ|u#tuEWskVGOE(t@9I{Vc{HY>$(2hkgd+_ST<^VT`JC=yG2p2;P5(@3|tUqw)>*U1qL!GGDr?tTTaD`Anr=)#k0%q%RTFQ3VV2)ys(|MLs9J@ve*84*C)g$F8EH768 zW*&={oUrj~cpHBNVzK#Qj(y~3j$5ZVlwiWwsq7F>N^ch^R5o}SCvG9cMdFL{1FdQe zozPA6nx85m1!e6>8@Q(4s}uJ%5w08BWK`M}1|Vd4<_mqgW8$zDTbaJ<&9k}F0ncY< z#!G{Nb?4Av5!C*&iZ8YCopS7n0Vwm;N5Z)O80aM8+LVE~!zS%+>plBdPjX7xOK z?9sPWFZMa_ZohD{`tGyWGT@T?w2!qnW^hPC7u45V68up^EH~CoKeSn-Mh9;yME=zq zQOJcDtwsjf0?NJbLOF9H@4#+|M!O16dEWLC8&i^zUzMijjbes(?XVO*_4#pu?e};^ z&Mkw-=6;`(}HU(_de?G2tLs?Z}(f6o^FqW_b5xdLN8V*-{JZMm83W=*b7;bUvC&QY|&Px3w^%JKU4 z(@l6&9e_zmc&_f;_1&`ARS6($Y#p-y?f1bb6cdnkl&gm z=a|V8=U?#%$jUup>~UlfY;COiUyVentz*?o_G})1t)7{hll+M6YlGy7dL>0~&Nh&T z3~%()428Wgc1zW2;HmG_sLK(>LY)!)M`yLI+7KEGa<5IlXNS?f;N_f6ZBEXF6LMtY zq*QJF%j*vKz6)IEzd-4`>uQPzLWEe7bXa59@x$e;0>;&f0q?SFBfI# z@zyq75w5TC4s7gCy8JUsOn56H)9iu{4&e^VhkYL}+{6i5oJ?I09(>gl)eKYIq-iZX z%5c4C{h~($Yr$d{m$~D$5c1WqFy-gKg*FUIMf82vP;?r^G40?aL9STu`~q)Mi6-u3 z&#pZ@5I4feg;5i=$ACoKNicYsVTJVspb9EHTEsmO3}zF*L$l=Kg(`4$SNBJ_YHjc3 z*>9 z@T(`R>sCuFxt3q;)d`8zV1FW-PB=kNMM4_IvO--lq%bE&=VlI#UfjZWypWe}V9gqRRB<4!eTD;Utf2o&&km~63RerGPCH&3EkL#O2;XH3ZS!_6UH}J4&)^!E<*tRmO0-xg9^K+Mr|{M z%&vhaH%93gXEM^U;z_HYmZ;JiG(n=nLP7^ubNhFvd=Tp@OZ1OC%~0cNcWpMed*iM( z#xusQ%uR{ZBu`;zwwyDRmTKZ1NQ}hV9Dl12%`qkwYjcjZPLpDJz4rYh-fosGEg1@G zmRAMg(o=~gp|6r}1g_2=jVtS~aKj!7gN#P!V_fcb0ehRUPor66^Xvoq@BtPo^@ktH z-zYCG)S4txQsc4cPI|X(69P}49J*(~+#A%aEeNz5oEpaJPAz*+b;1MqO7DU;n~`it zc#ftB&vb}Bm?y!n?xACn<45T-9ba z)d*aEv$}s+`XJg@maG|^FJgKn<-1?WY?Bc!B%A6F7&OBw=11_ElM03^CRvB(doBuB9@fSz)_;ed6LSKd!>>x|O(kVueMFyLg-LNRIzj zi%AJ)Qgp-PrUYA%-bD8o?*P*ak*5ku?4OO79FO30Ho;tX?or9y^qpz+eH8WE2>2E) zGTzCZLmZZ;Ia1eUZNHP5f>$_`WLI8pkLHo=&)K0geIFL9g_yy9F~u5sm>RAo$CQ3@ z59?N>FFNwEnX4<2UQ1)mj-%{zDu}m-%Bm>)WB3}nyR^V2BC48h9p~{C%^8HutZ6Ti zP`!bbM2qXGTm;i$0*uIAQF_})9O|izS}BA{@1fV)p{>uBTiCJ;@~sSG*4oIm8=b-D zb5mK~Z^KxHadgIumBQdQa1D+#2YTO7hbDS3Z6Kq~$L!p3^-p@bl0a@46REh-n2kbuD(|1(lDucrv%EacMMGY)Ra+|IT)iRHG}HE z8Wj09O#I=5yP))HpJU-6*8naJP9Vx7dcNk$DMBvefW!?`C zjM&W$uNO4&XUtm4^fxh`W~7F2c;1kY+YOe)*^jfczaY{x(&DRX1@ROOX?E2zbF zxG086lc`mQms}HnuKYz`#>iIg1NebmZg)H+juV$2bTguz)3NcNwvDy;wofPe!oB>p znU{u4LVJN)F}1g_>r#@DI?4yNe9Y+y9&pLJ;{2!}R<{M(8W_^+Q$wg^KY|{uh+_$e zZ2i>Up2L5$yq~-i-yd~vYcLi46f3^yfgJG&oOopR$GS6?V7#^5lCyBBjBdH)Cpe@h zwc6-rI`swjIA-uBG?VEClKjS?D964B-}lcDV^eJ|`et5)on2DhE&k4e! z1bt+RZ6|NgEJ4cD-MuwAJY`1w`zr4Fm#VTSR^6<0J%|D0_o;YIT#~7tR~lss&P-nY zz;l)zCWMHL-rh)$PfFiZJ}VqlUbDdlX(4s`^qy>KkBV1-K5)gJ1bM+`B-qcYOp z5TWijBazL6oa79Cx3?;2IX1VAdU4rvEu_eD%aKdl%zjPEO|IBIVa6u2rRrH)D4TT2 zsJcnv={aXM&K>dAf1qhoiREzd(P7fA70C&E3=m^}+<1ShMpW5Pk@Qfx>aZwfTP$hH`%%N3rv-%& z^AA6e<1R9C2IY${E5fvh#@E|Z|MnDv9S;uml*l=q45P7fzDd!Tn&z4{vAJM1m7u=% zp-1Ppn9M_l>=1vLo=S@3c5M8Kd(IhI4d|YRmazWpu44#vxGF|dD{ZIG@+keDA*!~CjU?O1UWcwfH)om z6icMPRosM)9Syeve%}|zjKd{^=Ft5li z0HEj?P;j8c0GkHQz|)}0h)`}|7(|5*7@Mc~lpLlw^A_rqetz`y^aSf8xDg^7Q;zpQ z-Fp_`0f5uEVGki2Kz^_=^MGCbKQdq>djJ@m{eOSgVH4df=oz5E`Z03$6)*;Ik@o}Z zLkj^i4giE*X#;AhVeY~IJcIW`zj?C*2o~)1Eq%xSq(a2K;lhLoCDPRzguvq%!q*3L z4*_%%&=PE7&p-nZ+xP15V|{cC`*MT&rZXUczH}06 z0l23T>H*quVIIZzPT^ic0Ou*-H3D{j-M$`02q+-xLxmRrAg1x+2;St~Vld5KPh&Ls zpl)Cp(EAs7np#ldeQq;o( znvG;6Q}oLt6pLtQyxKznzXd{&U?Btb!=MAy3l~B7T_Y!_<6nngFyRC&LxqzTEQAKHTP z5x>L!O0)ll3(9f8YSduiblBk`4ST&GmFwZF^jLRFnStlY)?V7HL|E>;Ycfru+v1oa zs<6x^f=^@>7>__?R!=eZw%98+h3fld?~F;*kUA!KoGP-2~f7s~DErF$swB{v0cl0{rp>-<@Ju_xHU* zbJp{jG~7;kADG6TrP##R{Mz}r*OnIs#*%Gg=PfQ-oS6QgR0znw%!r2iaq(Ih49cRn!Nmt8}Heot;QpI>Lw@ z)m}9;H}o zes5lz{#vXHJZAB7r=5^~L$Dxu?uZIeHM8nO^seBP`gYa@>44%sh@=I<{yZ?88%-zW z=IT0s?4rC9T&SsatWJg}Q@RUDD*b~5#gt=b?fdJFR`YpZN~YPKRd=%9uIhOxcKvV+ zNppQ>Fv=%vhE>LXL>;P8vIXpY#1-GTO{;C0g6@q&pC3Nv5F4Y|7HfN+5Q-}MTZP+6 z!9$acQHOC$zMS4myPh*c)sfsiC>-Ub1Dr1Kai^;6A_?tI3v~Ii-jyU(*qcV45c!0W z8%M%|2xxh0ms$L;3+5#J1r`$6{s-8+WE08*i|yKJ@Mbtb0WF8iqn$mbHVsJEVrNAQ6VA6e)x`bN z`e6?Lr(3cMc&`XN&4~(tlI4V70*7%req zb}T8#gk9M)Apg4KHBx9Z<|ZW+divXRR{f!gk#RUZ)2ENcH4X--;h3lqe?KSK}Bga@19ad?VuH|i|N3P^+DlnMn z5O!HeQGdOe2^;X(D1DXRW|?zaSVDSLLlfOP<(X%7UQ?I}Zg|ih0O)ZrI?g;BYlL|x zv^tlFYl+X`ZI!FsUE1tjIUv8fSW6#A9c{jSZ|P5+rn~CQ?P+6b!lu%MbY)rjNr+5u zn33!@sD@AA>`gArrxDecEH1NJhfDUuMOL?7$(MSG{A??(^A9x>S*F~KzFO|e=Gh)? zY!0%qUdzB5EZq;!vnJbw%s9yuJDGz<|4|PW@Wd6r$zVZM$R^aem7_Sd+2~N*14PpD zi(%Q@l&~t5O)j>fG@~+zUIib~YuR#gqsM$3BeE*svdNYw7-Rjz)(9Uw0dz0%I-pW2JPUGE!^2yaH;&4}Az5dt+8LVt zNYxUS5%@X&#KY+wc6mB2@69B}H&|Mlg-m6ew39w>O6!lxl~9#j>N@c^5WYY9a$bcP)GwYDghiJhMVVPencp)81@hJ)<*F3H&eyKIve@S$MC|U}^?<+asjQ&!pc+v87L0e5 z1FqVgUmjnD9=D9CqYJ|`rqaK;G>nqBJO@y`$$F0~|7MhUiVqvm69pGgR36_)O=Wai z4fm?FF4T_ppw%D1t(;&&;Uln4VpnVm37WRnlQw4@*|*eDh;tebhHUMRkuf(Tb{D<# zn2h4fkJvlViaTz(l`dM_Z0?wWNE604EDm&^-Y2Yl5)SCq*Ng~{PB?({E z2m4aywi?L>zuzlgFGK!(dq*?`uzP=Wn7R)GDw9OL6-P7W&8BUtmCJIO0!X^+mcmtq^?z z-IP7GO5#V#)X_%1#*;? zAth(ah|PMMBU_nSJM6Jw;l}Vuxf&zj$k|*1p$01WU>XQ*CUTDQdK?z6t6Q{@hDKFU z=W1k?*5{fdZ?RdDEm71M-WzHR@&dh3TC!~4l#gr)<}ef3BMfcf)Q!&%JN`ZqgFOFb zzQ(fe(MkaeMobB9ad|uy_wAb(7i+Su! zhCI)sTAHseNTuWa*Kq%HK=u)u$V+m3pogWu`e|X}7Bi=KeluU)J}2^e`=Dnpf6Jft z6U8i=jeK~eJftm&PLm#myPw9pf|fJeZLV|#{7_elNGs_}Ul;4!h9UdAXnMLqVJrGJ z$g)#Dg(QC1^Jd#C6dX(94a!8GYTm%ax3d01l+?&xaeLW!$)>fO?tkN$*QQ& zu=b*2b?rwM%%l$6WcYbG+qMsZ%-!k%ar)xT!3aH6eJk3wlFQb&V`*27oB zdUZq+q>s~+i`6t0L!o!ey+B-E294zae(mH6TZiHt&OZS;f8XUj&^Yfqj2L9Bd>~p$ zNCG%@F<}caE2ziAKf!qM!n{p1fDg95OH0jMvIWE8qXt0M8p$*g3Uqf!P{60pN$lEzDJ8v$Kbd_0gn^kG!>G? z+Lb;(Wo$D7dO)>vt)R9<#D5i7-M#s!Wis3>r%zp$CP|yiBapr0apl8-!v2HyRiJh3Io5d=A674;bLCIa77G1fWaz{N_ zweZPCqLXuYpW>DldF{X68nEF~jg`ezYimBJ%?G_Ik6g8MXrcCfw14y@;zBf=J{WgM z;?T^#E1dNt+sc*GPWKc;+`YxZ_BS*^+awwxGsev{6y4ll*+dNel0?&(Eh*zug)HIosNjj z6+uI_WmAOpWv=Qa$tVw3(V{N!-{eh7&`-s0@1LkIGIF8YpX*Tf?rW}RVQM!-a^L&9 zW(zK2TYx}87Aaa`R{74TMtqi-avSaW2_1N|4je4G!W&{x*>-@*H|-6XCgXqoEwqgl z;tcw^0XU@+)esG;*0io_md&hgR2>%LnCklI^y38H;3mY6#;;%?72Aa5-ilp7XG~(# z*7J!riSE0CMlQM3_UwBjOY)-YG^p=7!ZwYzD(_PtYQg!e!g{eG%dU}=RvlgX7%@iy zF}A!YluoX)zdOEVANKWJ;fKD`&wzONI|=@`-&=}cw49%l)`zI4LZfHQ%fb3RAvQ4@ zt~POOB?AZ8d>}cdsc<>VC7R zAGfOirc!mw@vFWM70!B{dyYlQfZ$dRR!ShmYH)YH619*mf5QF6W|*S8%5?*}Q#F0h zT!pEXAivo6sJYtMe|mi8jy0g>@fZ;Q@!oYRJ7+N-NN?)l;154Gz~VXNs7^B2?N=AK z3X#!M5O&k^6u%w7tP6~$J^sS$>7k)>rgW65(u_5pRLZ|yd8EFt2eGgT;l!j20aqIe z5w>FYQPVi_SX)#e4%yKuR-t;y2k!k;*iq1RHRyCCU4E~;&%di7k)f%>4*VXzT{qO6 z3(1Iw;wz_gm>UBJsXK`Y(Nf>__oJYGjiBs?J4-dA=T=zwjiq~{4eq;;HO=}u8pH*2 zoI;SNaAy?uPnl}I`Hwx2#_R*EeKkra0(_e*KViB@HvzpEPu!nuM`{yK1u;Fkvic8N zxV5kG18B#H;_^7Jue&tx;5blto4l${JJ-0VNV#}=>ZmVmtZLs-VP z%Kjp`l%#)NX@1+e3gDFwxpxLNF%;@xsMIdN!yHPL*r(ZC!uOZqaK$QIr1}OU0o@gQ zMmM@UcBv9Ihh)0F)d!vxSCMO2#@FXjYZ`~-CYrs*2t|fq*QpOZzY+nnnVU`ofAc6G zBWOuQ8XA6?3jE5zw0QsB_jWrK{ET00Tk?8oSlW%1y8jV-pLbk{atjX~rg~J&--hXv z#E*=4t~84~5U?z^Ko)ghr95jv@7jFqpgkYX82XNX@vyB{!5Qx2`|zxTV3wP`C3aLC z%~CND;0#NKn@fUdg;qUeh`6}9cRdgh4gKAgIKyBjOvY*Y&c>kR2FH$vuO_p%b9AWn zB4L$F{oP|Z(Wga2_s$Uz^ClEqV&`>Deka~qm&E#xcW0Krb(G7e+Nj3~-EVzqkS4p6npWQ|4v_5 zNDn&2_{xidATB6AH%p07I`>WA+IG6SiWKBcwQfa$~FHdrT3b4{=odAB~Z=DS7-7hgA{p_ zy$*DnLcxu>@)oHUjA?w341%mHdtQRaqrCR?aigz&GqjMnR5pZQgFY{xF$p ze=1mg+zG@N=9198_^0BL6L#S&V0VC{~#-VRQI$gn^x0so{gt=4_dQeupwc z&7|CtMNWS-;P#dYGUD2lv!Si2TH{p_*1Yz_m349PHMs{5p*Ka%Gx7O=M;d$oto)`? zfh)ARc>3;kp@?JUJvQRUd%qeB$=XSx*A2^g5_hcFoW9}N(3_2SjqZBoBR$&OBPJZy zhE_NbUtVk3{U&iRm9)uQO+aU?&QlOCj|V}#x{epK%Uq6yKhPaw>$3(7j*nX3!*ytC zV_9K0d|W{N*>-y9=ewgik-Yg{;(5QxSDB|m9?)WE4N?|M@$xo zv(xjU-T!FN%3%p1zYza0=~z6O9J70Qpf0M5p@9EW>xCcROM`GwE0Y=L<20YC=><)R z(p~8Ye13%7%&3 zZF=kOW14Q_xEuJmK=5f=V|>^OLKE#M5(rvsqY+DQs9YW6D7HaIl^CuC>rI|U7-_a} zUdi)~{SjnQHMH{LF8*-1RF}wFHZt?DiI0iwmRX`4W+)JK64{!}QBu5w;40FKQq|op zxyeDMu^8nETMb+l?an&Y1z6SHTOCGBR>7b!qKJ!WMrE|@81WAyXs<-Dgq0`<{_Fo%3p&W3AyN8VDyhgXNS?h%N8-4 z@J6U-p^WZc868PdsovLg4i=@eUFh}=Z=i}AQyI5(jQ6;whKLOf#WYBjPY z7tI6+=qZH1G98K;H_^6N2@P2qPaF-qE>G_xD%x!faDQVtZU#mw|3!}qK)A^big@w7 zzOMM3#@~)V?bZ5U%+iwRI5o7~7GXCyrJbex3mV9Rsc6iEnJMyy-1#mW(XIoAZV7Zz z_PamH%0r1~F2M$56|&IX_wirH7cA1!F8qJ5e3{tL~@gWF-AzYd5HNs)fkYjBY%0E@7#7j7cg|dfvZ0C<= zy|9Q@t*Le&nDj`bALVh$*1<*2$z}!nglV+W;TwJcv=-cdirF~+Q_RN3!2aK5Y9<0s zPIiv}e)``r8zUnF)Bkg4|9`Zs+dvg^wvbq(wC4#1FmaG<9NgWZ=`h1ENf?5=y1?PF zbb>*`0SMc>v>hPQ_DIg=61``8e!W$1S5#)MR=izrZ+5Etvf>lfl*4r=k?a5y!-sU% z195Z)0WeuvNi7NfnRuhagQKIdqZ1X^Mn~Wt=D^U3@rEZhz~GR7=%ZqgL5?0wl0iFo z7+3Zf0vfCTNl1VJ9-x6iLps_4wsmxbehCM2LjW5I*s-Mpu)qZ{;et7g87KR3d2((3 z{L#&W`mOi4K%e`C93Zl~ zJ~$wlo;bg`nFVZgKMCO0jBIKI+_u&E1GtG`9+8YajC3cy0N|Ryd&J-3Xut^MQe&I* zySWf3th+5JhtQuqFbf6~$jSfSK1f3-7f^l($R$+;01N)FYTbCD55m^J`xg*^GvGJm zbL%r+fUTZCSI~fAdj}BYu{lr^uz<}CAt0=qDm*$I8XEwl=_?qb(K(F&J$PGy&5aq{M7&El@{g(?(qj)$krfjo*$~g#dXgIw#m^Ylw#sX){DvT+qx;J9pJ4U1k@v> z9S}hrU>MkD=6wvYGY>9-pFPn3A9Z>6>=wWcP#_;F;A2~mP>xSyZ=Qe+vUhLJMxgWb-3;)aR3G zpuiz;SLc(D^_-vc_a_Z-YFBs=>TOr!;*R`T2!Q4{>TgJNJoMKS=+`gpJHPJFZpx2% z%CCLH?=Iy;*XG8zq``ak{VzelmF*SJZyC?T+RCvd%JQB(FyLoj+LtqX97}%k)XBx}@5~ti1Q4;XNNRPPZHp(!O8r z9`}rF`zHI!9M}aj;@pEhfHA>iqDujuYj1S^j>&AuewIVGbewZr9_e{eZ3%+t(8mZf zsnW^Un9wah`J|>$7fqu@-bsuO^hTYZ)*g&J0`>DzZkZlZ<#J$sJm2#lp48}3X8XW# z?_Vd1)H zR|%kt6mJal!1jqisV)H}?~UAbmON;aZ|`c4b>F}s;cA2Ce$P^9*6QfwQ|QQ>9ccd04aQIn^kQ^%~)-&|gE#I}@ytWINTT<%oTNFHZ zmM@?uOannxLDZb0jCN-X#k?Z|j73u&W`$0C+vfe;0rAVk&zkLoZ9sN-d%G=`)e6)J zX=OFiG%J+S*;RGH$e2r-k<2}Qjlsn8FX*aU^`PUQl8gM$1#jQ-uHDBxQE z5qDZ#uQr$nB}sgdsJfqoFFPib z8d$hQ9nV8vK0jyn-D!1@pG?82#>qsf>z8K%Gbw!f$J%w*d`x3%r`lA4WS}uA*8WRB zyaeB>b2IWiWz1G3p2IIH6&o#|8f&=NtVXC7oj}4A5eCZcLFIA_8o0% zuB5w_=$0bgraBrYTUcn8=SC;zlk~njX|YUD}Y zjE?a{5VyS6G1fQg9LdvFV{&G^1e_O{(dDhG0Pp=!7)P@2sBL+9`5eLUwXh%kp*ETg z9_8>#Bs6!MM*huj`{RCs3fxMb>0RW~%;7mRP*8PkCRkzG-)zhHEoQ>K6k}r9aKP5; zxvM*aM;2A-r7}+;#S!Dc8=w`uY-v3igWPcln$?evs6^^{GG`dx!%%FQj=u@WT*<26 z+c>+IlPyWBOxl)ui&^%s=$(T&x#%xTMwdsO7Ydb;(mrqYdcxOLzOGMSz!OWUubg z)lc`Vh-Uk3-2Qs%aKns05#S~}m%M>92Evdh=C>n}3%Sagwyz>NK$TJNo&&@+xhiDl%~ueDs`9ap*hncA`liRi`r*XAtZx70{UCbr}zk(*fW?sQJb`f!8 z@?owWSI=3?ZTn2MjCJ?oOLa{h?Idq#K~ZYqB^6snpwD%rtRV2AP4Dhgblwkh#LJhg z5T1SlJ>W>48=ky+f1lfsa^*Xn3jO(5#vu6DVF`XCeB^uoSrStvO>#6fhu3r(=*q)q1aQ_Xz(}OP7MG=VEvy`FcX_<`qfYN8_)JsW}v=gYO zW1C_h3HE#AWWaBjp>)iHI5wvoLX>I=@z{V`P2cyM+`{v#OdqBU3a;!v;%RUxVYaXI zlK=BNtG9}o=1pD|^B73Zy+EIgjX~*`h|H9Ek^BA+xeXGbDhEifssn*mmxCI ztPVM79qUZAuUeXxb_1MG=*Scf>AZUm-}P-bxW^xx_$qlLii$1-hL@1CbujnA?#3o< z*gml25k2WOq1N#e7wzkx=e~&#eSR~|x$)7PPH^$J2eLau_qbT*kkvdhhZfN6VlOzQ z>+%t-`}+VK_8Nq8icl|OZM!9hbf(^{FUzO`ZS|ljSv8Gc$kx}K+j71h!ttVkzmh03 zk81@pOqF8yNz2!Mlb1m{z=c~}T4EzTY*GCvdL~UnV(Sw5zyu8%)ZBu8zSgZTz>k4J zs8?|$`{0I&fC!z(c1sSL7PG+?-L)2R*3ex7iB1G@qh|x<)l>VERuWA&+ZcuZfj9M` zN7)rV)wT0T)=0lVA4VMBDc@sp-FY~k!SJdH+jLaFd~GB2*|hk$bC39bkX2ITul{zw@OyIFs z`kl`#<$9fM^{aBp^2NCKr7E8B&dlf#48yT|q!n3%vL=5Kn3;`!mV?ZRekqI)8`-_6 zdIHC8CH@#g#ENYCPF5Meq|D!Y346A@L|xuPklG~G+{H3i_t(tS55cN+Z&^G&c`D#S zHmD|hspnB})_}i=ulV2MIJ3;Yq%$TavM-(@)iCw&iV-{YH>rPXOrdY_VbpS73RB|& z7vFkM3r>=0T&nGrpNH-Fv>IwvHmg@)-rUFd#jwrxXTkLF_kM;wNg__?zmje*R*yJL z-8qIEMe?4T@;o_N)`ki%XibAB0&LqK$(#IGXZP)yvo`d4D(ba-Q1d?#_D>55^_m8n zgLJLL$Md#^=4dXwJR;{CXwPG9m+YFuR6_7f+q{7%&6Ic>PS2z)uK>|SEv#=nlilzE ztl5w=-6ZQ=r1)@Lm@6$?q{lFITV{IFef7i5dn)EmUa7iM`9xwo3V1`l5@^sVp>d16 zSx-6((eW#9bb2I393{vK;|veot0@*Z>OgJ&8hHWUcbFJ6aWTYt3BBr0WfAQirgC$y64F_s$my4O z)U4OuY);szr=!Me?nn=vE11R4HM4z*u1w8#bo8J~Yl~&Gb1K&(sQ;L*9<3UjEYgZ6 zo6oG{6wiWix7ehhV+4Lc_bhcWoszsePCicDI+vs;ru4fokWPUL$hTB~s`j(btUE$o zw3+}I!?HWf&lMIru>f|yu{Kg28(cN6&h$4Bu;W$@&1#C`VjvL;W@+>C!`9L}s!c8geVd6E+r_GpJl$S3kaT}YiF442>TS<+2>Nc5l z;XPOea^QaSp^BTl)9t)+;X;&I;;+Oo6Kmr;FtQ)W{GXYtSDX~$bbK#_$qPc!#V0l;%uO@ z&2enNu)bKmF(iFS&ju1foIOiPcV*L=s|+N&gb^XcA7#u7uMUAtXd%i$D4nW@Ebd-f zb2(so|JPusG_coo>sIjC(RYtEhCfq#aJ$<6AuB^C&jz#nlNMJ#qAJAC&??IP^jg>_ zggRp->b(3|thVOk`C4zd{zY;F;jhssbt(m6%iyDg+DrQSJ<>WgGeD&Ko zJvFQK#G&4FC90H9bWE2@W!)EE$<&9?Scg=a-CZ%!R-0?S9fqO;Bt@_;KLbGRn270++t1A`}c&!9XpEeoGd!%dpLpIfDzL-;g+D!^3fl7_J+Le#XOVe-mEziXJ$+@&?2{+Fo0EH0=+Opj8H z_L$|wL&-z(s@bNuXK7uDSaBgvg?LdZ%j=1SZ;iDXludj?rki-gy9)cz9#iIW@EOey zIENW%P+G>OAK6u#Ze!)uhT$_(PQhQK%l<-_QwB*zUNyz_U-9dkm2@kv;ePrxcSH^5lnvyu$3BW^&F2PfROu)^kbqL%y)$2Q_d{3 zM(^?z>MO{-4nNb~?w2S@;{VVkua;X|&oU0Mq2PVmXp%)6jV#(7 z17H$z*nPxWMX7QAK}z?Ar0vi-+r*LvfJ8ks*lHa2JycYAZRd;u7CyP4Eh0!!Yl%`! zOd|#wt_2!U^=+m6>I942Y`*I1CaISNH2O@FmI~OeRp;gIFL=&bnu3GYpIFR6EuCwS z4Nlg|qPcgbNlY}@YVcozkV#;AgRyv$z%WSNrt|=E$-D??!lw0fEL$n8NSm-s=9-2TB57s<{M= zl{lLYcYq7o)&ftNy0%k^#bf=%7LcZ~Zb`m=nHwjh zEk?mH3-!aToMu9r&5`%*TGM^5u5Q{`e{e+TzPd2_fe*jV0+$cV(OoNR8Lzg(u3Dk7 z9%KT?n3snHr9f`{FXLoTMcPf2#zChD8Sm|;-}?**eU*k;1zmW@!&tnAd+g4Ak<&IKP~`-L zVM=557Gz_ZBk|+SK_Z3rH)x${SGl_+W4Vbl%Vtbv&|8y+JuY`Ga#X^5gGdVf&k%pQ zIcS`-U&Ef`&PB=`Cr3o7dT4&3xLSWSrkHeR6)i-sV(l&AnlnB4C#6*?F*cGB4ZlRJ zhk=6;z@d&DL2F{U!PeWbmC(ud5~DBn`yq|dVX%p~Z}+pTo2=2AF=1wM`mPlBd25&o zb9B%6rqtVXz|X-5JTXK*1$OI2fs+eGv{4fKM=?;O=>bk*g1O*FS_hDmC2caZ4*6<@ zm>re)BX)>gYphr7Lc#xG>>Q#q0k*Xr+vwP~ZQJg!V}G%2+qRvKZQHhOCx4&8zwY3k z!JX7#Pij=Fs@C4`^CZEw4z7G$F=^MYjZz{k0&V(7+LJBeHS(j5qRo*m5$g9@8Zh`@ zzMl4)*(=|3MC{xR`yi0#Mc%|rsYMKHO=C|!y3kDT)Q1eXPzVewbiH(ur$l~1eUuwJ z-|AT7ypdeXX6gHMu{&|5QgfGjF2^cq_q{#=cO3!2ADLyQ-jKnOi)YrlFhS=3LkH}` zdOBw&4Mv)2L>i6Rw7u5*lbZ!6Eag9nNmQdkwSy3Y-W(h?p;=}v9ZDpxu&V@F+K;uR z?xJ++U>~;r>G1QuoQIgHoibz?OKSusF8`>npb#5$=|oQz+12dXxY;50+WSt$eA;X} zDpD)Is~+Zltm|VjEQQ!chf$x-x%)#d{IkcT!{`{9qUn;Y+pQE8yMXzaXX(Kby0e3;LjPobYv7x+?}xSr zbmS$~sqG2{9j;IKM|psV4nC>Jb&C}mqJVGZ{W0n+f$zfuZCyLrKYNFf8G9lMMvtVv zP)CAHmT#@)USV2RU!7BeO8C5|OOd}C|1C;cK}a&c71j84(|s8YGneXla?+Y~wW9$L z`_As1#~sV)AP!(><#C4C?NeJm9N@b|9j=;aaL9YGzd7)CZ>g0XIP9a%QgcTT8Wkja z=kJU%V2;PB*O$siQJSV>jTr09ta@TGyZ5}ix~SZHO_iskE5KW;GB{tHD)#Sn({m!7 zd+{sDOA_w>Vw(S*Gcv7iVMOh?6hYu_dl_I1B6Sv?-iYzU)J?ky-cY|zun_vZw#A8& zp`Snq?Xd2UE=dHsD&fh@^ZB0TA)K*fs*b@;&H7~RF0$D==OW5raYrwNxAgJv*ZWmD z-GWjd&v*B7i+;RzOt2(OxyO3BF;B4Gq6&+I7-h2?R96Kt_&se^Ma*r^FuvA)*!UPj zM`wr5K=yD&Jk!y_xz4XYQIYuG`C(c1**n$|p46pzRIWYG&BRB>z>u1(Do`w4LJ^Iq z48ktWB2Ja`RwCM%>SLvd<5NaL`{375g;#2u0@ZQFOCQZ>|HqIV8h>x!EN3S9-I!^o zzPaFS4a1NF)rdP%e+#~hp61mscQ1A8A;5xIF{N}$H}(bg$a%qQ5PMgl7OLt7ZJ1C{ zTWsfYSz^0DC)WEedG{|OpJdIdDA<%UlyY%tjCR*LWNz{sNLJm=G@i@{T}8BzsK6JP zSL6&*+S`Z2j$!_n>agS1y*ZBQkYWadMfELnx1|qyCT3x_1JQND9K;8AGf+U~zPHZS z20{_4kmAW?BfKrcUsw;8D&=2;VnI8W`{S|$=j74%Z#Y-KLkmyK;;5IZ+Yr0QZj5`F zRHIpB7A~p*_4~v8f@F^edc67fE`qpifzTDtW|`eMKXle6+Nze_g$=aaN1HgJ=zKXw z_`&SfM_&{ACkRhCxoe>G8Ws?@*-F_WR&jkg)`Wah&n&^Z$XSR|FZU_h^17d9oxwL) zhx4OI?c@ih$(TMdY}{$FID4EhvSvxUV9sT|Fj8B`8cs3rT!WoO%5d4X*&&vQE;=!9 z@h%_T@G0oq)@1+OI^x94oEJf`+QGcih+n%l3ExG0VRWcN2{q`GGd1J%pVr9xMej?; z)(kLv;si&8@$XYfpMs9zc2t2$;i&uNZb+4o5>s$Q)|e-kfyM@-1NnH1Tx#pcfch9z zXp*dc#bWLcyDn*DdbgoqT-&M3SzNny-oZ;xTgAyls+8-T`=wbYy!zA}ulIk3lJW%~ zp18NRc|*}~Kh}UgcPfV+I2Vu|!+@U@X>5_w;GKR3^c(LH539`$nrjf?q0V+r=vU{y zKH3Rz)l0LBpim`5%K=(saEn5AcD4gQuGF5#iZZHLrZ(cKWrx@2@0hfc8KcK@ESCa9 zexY(83hv-ZTw9JV(qK#*`m=#teGO&<^!{xbE6 zCl@Xl?4De;jzE^J!{Sz^(*`h+az^u6F;S@ifGk$PDSQ5S!6s2UVkESMdE3*!`7S=l zvgF(8N2R&899cDI#aM|I#gv^tSKrO4yP`c4y3^z+#78mc)ohhQ2Flo2_65`+{srWj zG13$v2L`vlBSoG^9ia!;^cJSQ4ZOF>NM0dOlez;;_Q42ZJMQgTmSXR?ws4Y?FEHE) zE_vdT2&k<8()CM5Pq}`rUZP2wlf_DG#Y9;UhmpwXxB^gRvB^>-3fG>IR#8o-y ztJ>I+(lG})-`zd>luV|ZoJ))%RBn<4={GUN(t0^QXhDJm&mw{w~VwL=%fDLMQ>NICu>;G`1v*F!rPK2!@nx2@L2;!ykj6 zmDaj#T}zD>D-9EVTu(epH~Yn( zZ}3j6o1>CpBES#WPkn)*SL%(|wP=4VAKiFf5PRR5dF6*Mso#2%>{(qa?Gsf6SGzpm zs3D+Kj!ut&IjU{BUZKw1&xxxr!sv-PI=44LNF=K?6!>7LIN8_m&(XBR9ot7%E$Yc} z`UFyoW1DEFYxioE-uFggP><6&?*>b7{~XW3B_Ls7(k<^inP`&!EAJ#cZxX!i_awQitY{pz-(+4IXfwZg$3XSTDUk;c;QN!I``+6hLwibbuVL7UZA$ zD_SlVRHe+_OICkdVl&)xcF*C{@9;U@=fkrw19j|0%H593kBgcT$^7-(sK>nd^Q)mU zIE56{6-eVX3SIsI5QaI1w~Kmcc5NxUo9g4v>umnd%tW+pjG=qC9Pc)JILe}#FsT8tAH>X;}N$3t=ZF;59_6$I%@!A0@JZGLKjAjml^C)>tt-^=p zwuDz(XQzIVA#jM*Y}Ch0W)J+8uH>`hAAmNprM8KUv8n)`Js?= z)16%*1N*tzA$Ime++-Q6r6)&>)O}G0W2hWN`iWkxIsR8O;vT`_Eo6U84r+^NQ_Ktm zT}~-SAbEwYL%clkj01G(7=KJ7WN!JB(SAF9l>f2i_`36A7P>iF4}^KJJ@)#tEPLrT zqYwPM<&wOikCY>Dp4CM7{p}}84(7`vqRONo`V~3VVTY;~Va$A_GD)AlV>VA-&4j4YXM-{0(|a9-R7}-tcT4vgtMc3K6XBcPUls($pieK}$uI6Ul)V zeo=v@Bp4vBc5wPS@J|`(Lf+vkdBTL8{wKQ*g}dMTf-przIeBJ2GvOtmt?E=wEd~Q- ze%d+&3Sx=Hn%4c?Z13R(meBD5QtaQe%Jy`t8;n||nA-;xetn3;603+cYbz`%A`%9+KvyaYqI2IxAx@Te_;+Xi?QRX@U2NwQ;@NO31?uMmai} zL7EqB@~Mzo%@5@Tj;B{k{)*#Bq*R@2LEb2Z*xyLvGkDPHD_?CCEN+ywy26hj1zi<5 z$s46%5>d%Nf(-7xj-KElCSEiQ6BPK|ZWqP8^V}QnxRHSLG2Sg*iHF9$S1ko8oedN1 z>AjXiZe(wLyV+^*X6Q@QIi{jnE~r1?A%9y6^Rrfw@-y0*RULY@dODet(f4GjIXU}H zs}!C>iig33^tGF1-X2cIID!eDVg*7wp5#N|Q6!^Hi+X5nMuy{tXNoEE2setJACyHb zPh%>wrR+7VsSqIZ*qV{e+3ud>-~7PedM=M5ky^oa<6S9-6!PleT|lX6%wBf5V;=W8 zGP=L(ts0})v8R=^=uZTO{HdC!cs(7H${Z8bYz&hEARfDHBG6ImW*8zmY|dl*bL!Z1 za?p}^_shXeMj-i&{}PDCGuJ$@NIj2cc8Pzx(>T8E1Zmt zWpxencwyWo?DWil1HLZW8)akv4G|q>B>Mp6vrBDNX_GI4jUHr8oTBuqF+IOsY$=sb=}U3s+|s zFS@5Lb&$r9MPbxanVm@icf^z^K=O~i($GwEjZGmf=>(?BZ1C@Z>YEVJ=PFKde|6^- zqikn;4;i&LjHuItqYm=bp1uhD4eXoKAXB_4RP0CzzPUwj@z;)hyJYNZgN`$en}BP~x|H zMlO6MY%!5JIq6{O^9v`wxOWj^` zD)|M4Wnbd-g0roz9T_s9$fsST$y%>&GvB!4NUJu690(3Am)#M?_SVmpGktp>+O zH?OVyVX4+xS9VoN9*y+e2g!|5bMcSM^0(56I2N{^obD;W(@$wBTP=lpye=T5wG_`D zjl)l7{la=bn>Ai{d6~nnnRG?ha0_=flzrXagQ~ikUty4iQsYAD?hVde#2sX;M%*F+ zA27)oxDu|~Mm0HmlH>zOUOEYvWHnhlL2zUMocOB>a57Y$MDA+8g$7`lM2`a`-Lv4m zH~yIlo?fF!*(nNzQdsWBT|l(>6T zXIZ<42#^>L!z-hqlQ~|sDtK9#@u{aDszjwOw$d%Fl08+?jqSx37x+BC=irHkUd-dL zI16AEJnYncsQH-=7mBIa#}tC6>{lLMkE&3#w;zS|2RYeAPW~yG6r5Jf-(^w4kt*~R zmG9j>hGTJ`LJbXgMA_@pu^ivOvV#-}^Qf1+a*V0ijF_AVGvgsX(6{ACq^k<#b@Cl2 zSID=27A>+os}FozqK)w&~EJsepA#l)7q!PCoS2_%%bC zt2oQ($vxw)AG6#STFevX-qhL(xsD(xfu0Y(%b~2|&ul`_wq7GcE$}t-SY~|ugeHkhfx-uwn(Yp1{r<+Ssn5E56RT0l#4}3KAA__wrHl7vihwK`7 zGGb+>wyI^CP$!aJ0NZJfwULeU&EGa{JmQ1p${CN;@KZn0fWRo#ck zz-j3)=$-dE3$PYTH&nlBd}X?`T{Uiu41!WK|9)P-f;>iv__H$TvYP_Q>q@s{eD-Ct zqU+Vj+WR1g)$Wy`z8@Ztn2K~8*@ZFQ3=N)!KiL-}rVCDh2W111PZ_wyx(Q3`mOmFv z8%W>`a7#^%*pzA{vi?xcWo6_P454yUVeF2KDqz>R)4l>BJr48_wAUy{EQ;xVhi+HCDDI04gzm?ZVW6*sWu2U4^RsckL@22?!HpF&0)M>^ z2)1&L^dL^;Xw`7qF>^2wQL`5-GPP%#fVC&lF6>kUzO-U7cO4p~_2y{6X8bc>w7r>m zo}WlLEtU-UG6KgQF6_s7$kuCS;ZOJh#b5P_{x74roc~*y_#fdB8}t7kzGWt4XZ`QO zBMuh!|GV(W^&j!ZYK5JMjw%~;ev1xIIyGsL5sqy-mj>&wSPx2t(I6278A%G7&8Lo(Lk||fGU<84*w9=-ciEf;nN%J zHV_O51qQ~Uzh?{heA_@D3JMAw69oC%0AMYSGz)8h04mfQmggJ1tMQN4xEo<;C`qb} zc6vnt^Wdz4j0Evdc;xDbwTgI&1LOkyX^CY3_6YG~A(xZ{X|RWI_c39B;3!7u-)#V- zz6TEMpYNlI;adi~is3hfz}RC1#26Bo@9Pi$4FU!Ft%D#athz6-_wDur1Lptj0udm{ z2;fA*2J>$R)*;d}@T-sUrzA!qf&&uP^#S(R^I@zKU<87?1K~CT?va56aa#O`=eG{^ zeSZLb6W)aqP!b4Rsl^}i3GUNtWM33RJ-&huA^w!~T`C2?0}rfw!6$t-Yt2!_M8Upq zIfU_VZ~e9$7$Tr|!GO8DgWQ>W5A3Uu{F!U>M+U;OcjuyV2M1b(1VR97i~N!{Kn8>S zVVC|?Hu8jm^&i*;(y+$aWelUi{2+$E1BLJdhSK-7(;WR_|I{WD;ak)Wlh4I(>!o6=4 zHg=t$U|1Ngps2Dqu8fuiv>)y>((?@?x!`Cn^Wcwt7pQYE@ZnFzFHbpN4cyyMkWb%d z5un=}O>stA`#zv=UpHF{dJ=^GZ^=*J#;=x%pXkG$f=>bW?;Z5BS5Nn^=7%5fSAk1G zK$ylX0a!}wH`FBxW`A8J&=;T#{k3*#$(`6rrs(wn-TkN7K7rk)vx;_%$=_u>fCzi)`nk{?G8R-|R_)aYSjVSY2V#~!LB)>-&Y%trNy%H;ur!XIUi1IYVo+ueum^%)ZT1meYB=O)nxz* zPSp}s>GKTQ2+I+plD!6r-mt zr%K1Wy@IZR*CiI%)TFW(@Dj;vD+LuhYELnDH#m3EZC{wDp4As6WWXJB3ePwgUnBR40QQ zE?P*NNu*hSS<~j1Gm1SPjt1x(+bqhk)UF#^^8LTT+&{XO)Xybj{L9}Xx!Ub_0`(5m zh|X6P2qFec|87d~@rnC3oG&v?gSp+v#Lv(3Brw+TVeUCWBP!# zwWMXz`Q)#+QV7(y+{@p}Pvbn(R=uy<(=}jIso4+U);Q~1UpyLFsuh6X+}n{>r!Y`B zBf%LYS?Pg=PcL1{_;m9$ZX<1?*a%f8x>{BYyeRIeHnrW((@9Ug`+%8CK!CT&kyybd z_v0Qpu~=aL)wLvhbztr0&_rvZaKgxhp8is>oFbqrXg zr`1-T^lOgf)D2B#g96I%OswdNfoF)+cz}TN(i?GR+GpdklGL;7PVXEoERCP6F8PZU z?ReP$cwT#$Lbpp*8*A>kH-B*Lc=+uCTKjXsUpjj4`Q%(5;s#=GV#cbQK>28@?Zg2U zM1q&is>y7bi)rdvz6X*+M-5k07Jm33QJn8)zCiHNV;}YaK!xE{S%DbtTA|Vu$hVv| zzm(h3`1|xo0>JIHH(IvH7p)oMTG?E3n%iGpSq%{|bd;<#(#}W`M5OONQHpA&{nH6Z z?u2F&A3AFqaSi}xD9Di>odC>jQ#CKUc)InrxprpPl#jq!jzVlkEbx`JMZW~+n0HBuLYG|l@39v0tZFst=zJ~)~Hst zKf(}+_=^RvFSkYJ(L>%S4-}G-k_M$NtBFPYBI$aW&K$Mco}SbNP50~RRtCeFVWT?X z#d7nHK84FojHL~Jki4KVkF3lvJH;FCq7HltsQt>~&=x~7t<^BP&W5QLZ`l?!7#2jE z9xVs~%V>D%#)+@JQN716czk4pKia>QMSlOFrJT?uf+~mmn8Lz)##N(KeAAuW67fwO z)yZ1y-mS5xPFo|<{z)g+a_Z?v!9$uQCg_|xS8=?zMb)Zszad1A`^dTdvg?bzy`ssw zL4P@`X@J%cP|Q7R7nOC7Ii~+Z4=KVFgAgroY7lNdwg-77yz<5xX!6mhh>SoBIyP23 zthWqassfVb-N?3rDv03$%T4J1QpT7zyO`a|~x;LvXc&Mp; zbYRAZ1_X6f-qt!O+BRf|C0G<>%JB}YWJ}l1&b3FgUMR7w(aW!JjTjEBlLI6^K9+nB z_fHG=v1Nsv;I;`4`a3P+(P&0Ogr*v#-pmz zy;-NXaB);=9r7iwT_3Our+^n!uc$MkImzRsjz!|EVXbm+3UMgZUJpX?+-C#3p>?pm zE^~Pxg}E+@;7(sN!pU(c7ahBOAZTG_s-k@dGqDEtJaA5r)VwXp%uoTG4#b*nB2U!g zm9RnnBV{N&?=(+D4+e))pvIS$Z#6+M*0gH-OL|RdgiTWusqiQ!L_Fx3;AOd-t}IQ+ zU2YK;o^uW}5Fj<=MyN#aS3X124~yLEq`ZChd?YqXh%uIf92?g;H)DZgg*B1n*dC=1 z6A+A_lX(Dr-Mw(PI`5l<6JWDqivy|)O)&w0Z2!bvQWxtM5AThZW^(!adIP+@zUBfM zVFnPKt3abv(uI5{#MEwoqMp&FR;WgMpI#qVOFrXDJ$z*KB=mc_2P}gC{`CugbBAac zNZ(YnNzLkYZhvg}haZo4vokkXGL%P#(MLvD0N!Uf0ib^xQ38|1!M>eA8O~Xv(2gif z(`w)bVakNFvxqahd7AC5=w9SyLHa+|iUif`VbN2InsFTW@ zk2|)YM<4It`86t9IGc`D4!?c#&|FC?Jr;S`Z@ykC3$J5N-wRc&JYYHkPo2P$>7;h z9`=dhv-VN8Xu7+lUep9B_4nYThr2{H#5S}?L)R_R+pj=xreGc$%NWR%P7$8Wal+!7>ep(&*D;=^zh2~tL)vEW|Lk`Q!9u#z~!FE!&?8Rmti!Ni18!LxZh?JS#3 z@bbyMO3ud_om0X91@bB8(2q+$c!5GE&9~-_fu84sZ%Qrxn;B$b|*F! zYoxp1;>);}Id|x2S!b#!2E$>Ys(dyD*+I8puKNWel9yyQCoQcB31O8|4R%<-8(cTK z$6coY`)tj7tMK`VyNSM|v-0D@qC9;><3dn7Mgasb%z-*-+pFlrfG`S8Fb*0)GI{X$ zszgG?`&*8W8i8{5?s25j)I4@t8)3)nU$@jW%7j1sa&kz-2ReXr2Bl(!_jQymeZiCX z80L8AWba_;Sp%%pc*QOSc+YD~fS`oiue^foZ~=|t%@;PZ*aOP2te3StadSVRZFv8* zZ7c=)nwDNXG#_vDq6aC%6!)UGqL~;8ld;bf zb}~XrwEMnz1<>S1-UJwa$Jy)|+wsMU^~PJt+#qpwB45LlOUX0~e-t=qP1jnm3)D{emZ_z>1#>8wnV6qbx9ueH)@%<);uPiQUsVb5@4*KvjrTEW%tL77%XQiaB zCK001DfXNYzAWa8wIfA$tv_d{y^XcEOkT|54~;`W#mj1Q1_C7?b!S#$&%eMg@&={qY8 zF0@)VNdER1yZYP^)74dVIcET#LITCSNHVfjkLs=P!0{sai1=A$wXadYR$+ z=QcX3+hBpw@%S^mk`!CjHSdS?llDg)LB9{H4wZC!q-satFEPU;)qSr6q8q7UwWgo! z6Sf&aX#)$;?V+Gpi=iy15~d8c^pItkc)OTI7WcI1O^%WStUhO4^)ovprIY0T2o6EK z$$G_GRs8be8187dp+w>F!eUZs=*{*C+L&cI*)+j`4Pl|+*Gz2=iq6klEDH{{-cjK+R{kBupLNTJlhsuU+N3-rLoczNeGM^?;OC48%0 z6r8V}Rc(GMY)>50_|cz_k9}(cu*_00&!Ujz%<|ZH7@>r4=olzP;Fh{0OW+-aU>|mN zqS)ta*>kCXxds}~5mSBO2w6co%}=oWW}jeBY@9sKZTFeoe$4uno_hjEVAFhD|FPas zo(nA3@cZpvCm@-5i+*LM6a3K1*6XYg^BSv-$i=en7 zIG4jo1^yfu?M{4jFt5HLcrnaCN&Xz(vr})w-@r_4Np8TB<|x?+FQTnHjz@sZ7C>oM z<--KOH>u+>g?0Nn%PjUV+u_5!L<51-(8s3Ye*|P1M6LWdIKS6HGf5`z^QD( zU_cTSd3^^)%z%ha7^38n-6zzQm2f$ELm**}FN&)RkItRbTxz;x!-XxnuIlrC_{Y|d zKs%q?m#m&M#=Z#;o~hlB&zQ?o`>tx!!?>u?*ijVN?V`|l3uV$o6SY24O&;h7m=%==%f*|4;J zzK|;XE6+wB*KI{}$E)paTtR7qVj$F3T{`@u>R3pb+cuE`P>)lq(7Jxh@hgho*tW{u z#C-U$tVyfA!zo1YQ!U5AZN;kbp)2e9rkla%y#;ICO5&{VpZrAaUHZfXFjnmiyMetz z^Y=5NP|tXfmPQI@xx?FCT3Od;t76;4t(UQG6(+YEgTbYL`GVA5wFE+`SAm+SoRLc! zE6l-&ycjI}g6tn08){UuwD&9}R!(;*tiV%?EL9A;Xy1$rBxnbbp*7+K3V@rQt_e~+ z@H_FXslsYJ><&#oV-i&%jEn5;Bpd*PbGEau^E7af@zZl*QUo_jDkaH2`gQSp%#;$;~_pB7$JXHP0yTN?# zkIuG(f7urVW$8o9h`+=p`wKLw9gK(FD= zZQ<}28d-{j3g_trk42s7bTRFEx9(b6Jng9(E8%a}zxzDX zq2yEiLF4)UsNd}+z!n2ppSL^z$kQu-I9?h|O9j1omo@^;wVm5&vW5nkCR#l>$P(#G z6<&M!WGGu*boos~!;wS(Y*Ig%CJ|VjT0ejTHiP_#=E;HpM#aSY4=e2L6`b>=YS%mk z^t2kE_ioZIWjfEI@6&h~ToNqC@w=%f(-`+{Ija2a-U%05YR$Ykpw6|LRH>^x`aEmq z#UUDFMQNM?g3$G$8jjmi2anmQ$PiOB-61LQBdxK=&#Rdt!d3a4jlFu?2kS15+YOyZ zDSt5JfPh^|10=|65%DRswEJ?cB&-Ik3BgRKCnt#0Eq%6wbqDH+jVIlFSroQ| zg#txq?aQHuRSIXPgH2@im=+JZrScRwd6L=_VomzAjeLsCgjXo8y`hq=snWgNh2^R? zGa7O6iY~!0&1Y}g)cv(x=%R&!I5f-HBxlzMG~8i&^9jkJjP`eL@aU!EY)A-I9WiJ zRBx7xDE|vh{x%mPccNh_Qg@wZo!K#&%--~>8Owfc{ZMd2vJcIWC+KC>PRj|a;dzO( zu;to1dCx12w5-A|;MoFJ_}ARM`Q+TwK_x6|!7B0UW*cuKxeB5SXTZn~{Jd#kiI9nv3C0Lf1Yw2zi(^Ss|kF=;twpp8FjPZR zRD%(t;6#RIUVg3bgv1}Dfra|^>E+;N2|gc#+dbUYwHm_FII#8%XYk%;xDy_mC+AV9 zl;1YzvcmIg!NopAo9Uu`jGA@cdt<~$qNK*|b~s^qy~Q)-1zTob@c`bM$!(&|QSUCf z)~;bR@jY0c^lxoo$8U0Jn<*4N>>)z^<>|l<9rV)A(;4QrXQ@*)6l=-8=aU9Xtmdt4 z>(JQtj9&1*sHzpj$j6c>+`sVK$w_WQ8UldYei92*Axv}GWeD5x3#MguP70svCMJd& za{y_#)Lf?UKe|%^d}t_ zx_jU&HX=*IQ6p)`(%)R9*yzI9nne`dOv@mXQ*m?eIycaXm3%$*b zL-&?Y{u>C-Tc|R%9C*t#;3YY*4QAY@RJPW^2XlD7bUbuw|%NfZ>eudLcvyMXRAO$@vZ6JEjPh#%tY3Vzj}| zb4@dpXmwxNYTIAL0ec5^N~oWXq2V#k;?$(}H#Mnypxe2d=I-cY(S?x-^;^<{c3)Zi z#wQ(;O4cGpOplRL^ifK0iG8fC1gDzyWPW`_Co?owz6oDduPCz`Z^RW9-0n^GFi6Eo z19ga*yO7|>E7<~w(T0p0%#;E3&M$qiX5j$dEPuzmoaUwfVUnQaB$ryYw}{FxD8Uj$ z3R>;B2UHY0Mh}SnHQWm%Je`}pG{}+l8GbE5?YKi<#s9iZS}@VWOC|)xdTQ<_ZGZx5d`l(E)~&)wlHzTm^n5FV^qoKlrSn2G1v+W7WmSj9`8AWXROfD$p83#*3g9=L^s3)|v zJ&idy-cM->u!1l!`WGA!EGR;A1i|jmTnikG*7frOvc-_`ODKn3orCTj?(dHjg^mLW z@RK(v#_AnJ%>!~`Qh#v*Y5*~h`T-t)o~Dl?AD%W~Lo)sY4l*C0J9zmIVN^v+tLrmj zU}pS@ch=HEi7L3fx;VVOwKfc60LKhE%1p%wjCQrvCrkqr%?D)w@|U%l4RSRTbkTnu z2%ZgEjmtCGfdOSDgiX{VdN8bh`j;4*OZkR71#O$D>Q@ z{{wUDDFt+|ULqLBa@&$v^wpE(va24D-RqF*iwerof8;8=;hPSc<=^`wX=HKv8C%=g z(7@X}y$W*(0)`bR+ZRGVOkXn%ROoBG%P|}Et`Fu9#H2w{y2FL`t#R~t!}=qmyNIuo z-d){NTmRu(o5nP{w02Ru{b{yV@=Rz(V}^0!D+UHKS#9m#v&QqI$3inJzzmKoh9w2f z<}B`hyX%jMe%ZGL%+%)A`f2pVlM~j;OW+q84BI!mt07rDE{ed-A7y`zzmxQ>jw_x z1O~(>OPB6j`fjG|6RG7J*+=(}uTl!mjx4eDJz>Awq^(9&3sgY+N3(&N*&_FXPi zcruQGsmVDAzQe7!&^tvt+~_8L?*{`@+SpIUx7nM;_gc~4T|7%u#D*uZG~lWDH1sRo z-T}dfAG6pm4SFdJZEZ0l9>OVK_Q#&;sH-Y#zMqBb9@)QsJ>~PC1k>u%3F`QQEG9G< zDiglWPx=m67)H0Vk$$w+{=4XJ5=D2O1Zx41zr0R0zxW%0qN;Fpeu_b73d-T3PA}nh z-&1jrHUoe5VQoJF%|ND1&`fl6$nQVcslHL9f+f$U>V&XBIRDs3t7BuUsNvwepw?Mg zKmhey%1mIsZ!8ob7Sz#g@JkRoJLfy_v<)7jFFi?md7#Eoz+4X!qMz0$;yyT|%qNb0 zBXC;r51|WC<0Qc#utCaKU@Oq%y)P1npVl5Ch?(MNU@unEceGnCmg*0o9Ym7=RF=>e zB1q%%Yhdpi>wijD|0$j9gkek$y&{5WXM747$SnAhAlUJ%+^T;YO^x@17_fi9Hh`r0 z`w~)U!s-bdnN9dktoEV%X#=dRUs5qz#UFe{rguiSm(CdYR!8<3pi1s22q9aoWfUM~ z2u~Sc>n6_`pss_T^pSMxrv5GUSO1oK{Y}k2vQPAMbe+7f+)V>*&slx&#(*}SPo=;I zZlLUc#HRCEKzdmkH576Ahw5)v-9NNR*5SDw*syO_ct`Bxu_{eGs~a39d86x$NRy|~)bN1%$i(DOeArhvWAKT-+MlnnT>%uuZ?3hU`(NlyFSn+LK(QPoI1Chi z$*`RnRaKXl-9LL)@3mI)8T<3DrU)l`2s6$`DVMZVKg=}vRi%sbf zx6Spr0R(e6(DAix8M4?yS}-!e=(>l0LmS$_xr=cdwQQhmNP(hjB@8?9wqBU56KgS3 z%MahtZFV01Y(<`EW!7g~Gx4R&Li2>Dl<*Mg>fO{I6hwcBrkou|!3f*;jQ4Xrx)ff_ zm!g;Xn!h&K89-p@Zpp|Rmx_Z5&)q^*bq|gG4)~@O>5o5UC5#vsy=KN_qT}uRrJO?c z6FJ@M($qaWk0@{5EV!^#I=b)nZF38xvj|C7z}BP*0AQcLjPUA`7YVt2EMODlp;r#O zYt+W)I>KgK?#~0o!4ja?pW4+P7G!aDXKz~(9sk0L*vnV-yUr73mjUjwb{Y~qsGmC( zaeIKO#W%8M*(@`x7jjgZyCx&#Mo{JLl z(*J7!DAEA;@rIS8o68JTErDs<(>^&CHAF4)@SPEnc_C3%LW|F<%_xpqFEKK!*jE15 zvW{!cBj^q0k*`~%@ySHt|IIZUm!%#zafot&S~}Ln+wNAg0f-mfD5}?rA7@^CnZIOI+AnZ5?x(9PM`XEAYFs))j*jMd{IEX`qep z29??G!NBz+ZhxD+nuzK&D#}Ow1^#XyTzf#ciA#9+AK-5}SIzwf~Djc@ta z@cN=L%(>ZqI$`UO;cEKE!}Rkv5lu+d7jJMvU5u*E2ZqJprtO)Y<+Xwy+vNfkz^OT2 zI9%4t=6KlUQD69-=j3YzS8wmBLfk*+B{)cli(-aM&Oypp;<1~97_38b)Woq&;JsL$ zGiND=`dN?pMTJ6;$UG-+v5kN)xw60C+>L{!DGc>mgVA;2{N`K!=e4)Oh+Vx-O|VrM zRkFx@oyO^jgbPXjJ9EpQP-J5%-KL2NfPdOdW_q>QZLZ?J$fFvi1J-D8rhFKjmg1*7 zfBzMn6zRy#W=s=XLrz^qH`~&{@Lro_7D+y57JSdKOr#*%#9ex0+kDW!{ZCNGM3VNnyVh}9rJJXn0oS=@D&s7)x(AEIbBSv z4)RJkKRDZgk`-eOHNW;+AOeD3X#wUO^D=J66;?$Nt`u<+SR4mmQAH-1r<$ig`W91W zZx?3L^~ZdvCvnv;I5UUlO?S9;r4lEhA*t5^!ex=wEK zq!87Dl10w;`P1Jf3tjTEI={f3P}P_Q;L~vQ<0042H1b(aS<6;YoRye(n%y5uyA>)! zfxK;_5aY6Iqeq%kSN8#ECG?3LPWWpAZ`}F29-D zJ=px2c-WxX`ul3UvQsMoh-NhrsD(F%R$k-f#^KDAK3>1A9M-d$H@=c}1!ZuuzGY+% zB6l|YU9oiAmY5(7NxaR)`hosNB4A#YK0Y|g&9O10|A>)zd|B%-_&_J%7x82$+7tE z&_xEBPp!81cWTcFe=VNeP6_!g?D>63N+3%xB^O{hefTh9I7Cn{jof5U^@UiB9EaPZ zNcbnT?h~u}oRii$vzef}!&mgL&Dn-X-}9xO!}wg2AYHB&-0rAdQCW&~=pNvYXYelH zRhfjboN7g3NMVlepa!iwms}zZI2rk{PfyXKW>q86YpNy?@?{uMIbeHSKpPxd6<3$? z2qk53S>(t12DRBSxic5%dT65ObP;snb}ypcF(nYtVk-oWD87;hWBvP~rv#KB>pT09 z`AMP3>r{-#Aeo)h5xK4}_JxY`O;KZ()o@02$VB~8jN8#h-_DIsgQF7!oX5eY&Hhng z8-$o3({QIVRS4Wt^JN2%Ece7JFT`aBmh6D?Nq*d|K-a|2EVO6GcIhHVgtO%g2LvPg zaS(QR>02&L$^Fv7noCARfZ2dJ11S<9euE!n5rf$sy@Cm|-rgNpwhE@pWrmO_B;~_h zTqvfC&(I%roas(?bQuO+uO%)1c~u1<~jLH!78<(Q~PI5Oh(3IY33 ziu0`a*|I%?nm;u}i7^-!{AXR%@ZiBiRz0L0`s{1;Uc@bid7SU~g(DFHupg+sdBLju z*!5+S1*6h$`NtEQYIvxC{r&n>>=4V8LaNh81`V*Y9^s0!(`k2EP+xn-<)&QLua-xi zYfvo|lwMKZ2~kjvh3-%f?k*u@t{5SLKP! zC^T;6aQ|}dmzxSy2qyG9P8MRulVc4-a(7D6Fe_i8>RiF2v9moYQGe(4cA(Lp5SK%W z7sZ~P=sXyyxW0ov55C~D3Lp&Lt7-2u98fHZD>cq-9#Xq=Yvs-kK*YO=k-p?d`zd=3 zgSfdQiK}hB(p>rF;O+csJftFdR(`t;cB;UDVw8e z7M$9^cZ3;VK}p{vp^9yLO^@H4H<*}jz~&!&!~8SLKfR^kj|B#@*N<+5u@H9ysqt^s zDs2>D6S0%bIp__yg;0@n!cA4u!G1;@o<21LD>X1!El$Xu<3tQ<(FAlU1%XuPWZiUm zJaDHQF*&fqyY6@l<J5ultiOwZf#sK*LXfBAG*`c>TJXevR}$lp|@kN@Si+T#tKf z_YxtZwvElj#xuB#xycqvMXcn2TT?1}^nl*}my*T-{9&zmp}V{iaesEi#}Aq}<+tS2r2S$VLW>&NBWV06IX$zfX&oPpGCw;?Y-E>kC2-D%yi zoU!nOUSCcQUsN@q*D33~k$L!6D&d5DDJi4~3(hU}F((0-gj zP&!buD8qEzLo-~2%=iwp~sGhsm5bqd`tj&lHofET;H-!!CJpBmBEq;VNddPQ$F z2C@_N%1#YGv7N8P=;uxqfX^fscl|qVlcg`}_K54+qLUe`=uLo_WG|Hn=cblsgFyV2 z>P^s2m^`}b^zkn_lOw|Ia)_`jZiCAdQ_GG9918W$9D~DYs2}Uhw;W8>NEC=g5?P~D z-t?tb9qpTbfj6p6Xn2>iPf@n=Rr6j6-qABbp5)znHCZBIH0@|VQY9b7PLOMBYXu^w zAGO9qSBGuYUCe$y2Ht%#@XEYcOGo6@!k9vnMra@SQWTCS-BWOJpZ9(u0R2(vdvaqK zLN$ruG?Bt(G$V)BGmO^*?VO+pZl0^<(HcCy@?%R7`SE!(M16(34mn+}#*sT_J9HyN z2h3nq_!*oK+qr{4tGFq9#kL>(m?8Q#*ckr%y4>|g-B%&<8%%%Yu=kmqH{B~UuY zToX~GrxE1c#W`f_=9(0qDUej1%L!s;I$ez#bQVwgPAtsP<=xO-1ZzCM9LjC4rYMbVXZ8zL9S&K zsX`UE2#Plu#&wM)e&u?9gstbpx^^Go^CgaO@YABz9(47E?FOkMR)y zpCt*~Q9_EkDNseVzMQSCdj#V<%R5_5yg8akECf3xs_i88Qb3jO$HmffhO#<2t4#`; zzWzIWxY1Inn7z{@5#6a?gZX=i6^T{@Zbm2BkxFlVo)o?Yem}p1Vo;2Eyr2ifxAS2v zhxgL;#)zfF?0N@$@nj#^lCcjn?lFyIqamW#s~ZwusiDH9d#^g~&b&(Zp0qU%8Cz^u zGvs9TTazM%FXK~C4OC;!B2#DJv8;|n)#T16obOL+Uhax<(4fLj(t14+?6@0dL5F6E zxSmh63B>ywas^ut>l1>vuVU+YN%X+pS9iB3y1DU>sg&8o>{rpeQF_+aOk2JWq(a6l z$vPsq8>fJg?T``fQ%W{Kf#g zx*8nC^cY%tERanYmqsNCfBUljp)8ApA_nS+Q?QI<^Ouq5Rr`R)>^_OTv>j5+)Ed>L zM*O5SrPgRUhk7{d$z}z=z=~Ue0$f!Tp2yP~9bR*}XtScMHZ} ztSkg9$VhlIxvtGZ*V4=CzM9=PZ{@Yp56y&-Cd4Gv{=f)P_Wo|=MGVL%V%>6+4Emsq z%p`ClQBEcO+m9p=d;qEllm zdYM|B_v&F;S)%J)vIosjxrJU_yY5u8Za_ZhaYDkwfaZtdXTD3p*gA(-<{pe=6e{fp z1@^U0JRCKg#9!;$-Mf)h5xPY&Qgcelr|R;=sY}kXJOP#wU#bR0xnBbDt(|LrRR_S= zUwY|qAXhZqZf8s34L^CY-!cmAhh}y(OQQ*j{_)!$VEPVQ)*{DX!yB zYxELcsooqn*mGp$Cx%wpJ5HUSNwM%GWaH0?JMb*HY}flDhBpb)l$EQF51$)8gtVeS z*-qjcR&kOPpLFPRJaxvo4aQ*-kgJ@~lcwO6{wR1nPN@hB`B+xDP7-|=+oe&t%8N7? z+iOu@dE^*%2lta5M=kiw3}RvvVE~8DEz51AhbS|w9LT#UPOmR2b)nom{!SnZ@BpC@&K}+qcZ>RRuC?&W7=X8+8Rhw@F zNlxK$MeZ*?-~AdVRjYfDn`fvvDwkf#dhod|Tp9pk_k8`zY*=Ld*o`YSZ4tIt^-~N} zPSk-J@BWlAklyLc)5k_L$8q}Wdo>LTdZX8Mxkyi(F&D2F9itLkXvgP-09hB*it@dR@NA<_3gnJMF&sDO?f|+}^|D6F zSv>@6oARe$gn^Hz;Y&o$dbcD48ON`*G^jSOMI-n%g@N>r-4|>jy+s&a)bh+H#wO-l z`x>w5Ha$|JiLgxqE>wDkCbaY@iKN6h!;-TqnFqHhFv<%$nRk$bV{R}fW8>DPrzSZe zl^Vw+C(DG}`0kv}FdrI1%P~GeI_)xXPPl=#n05=VaM?fv$@-{xUA_wAu^)Ui1l2-^ zqgB#)yI5!&S?Ce4=^=^)eDzNyYaT@=20Z{gE?gUiTaP33NI8o+(q0xvZ!^h@5l+iw zl)UHlBHqgi7D)JOI`C7Ix`7m91-b5*+{eE4&Q}nu#1;s z$_6#V`nt+%X+%kzB?o9E@+}(s85F#7e?*d#(nLxo{B6nvOD{{YLgD@A#=wFPSvs*YsXp9Z+^*; z=Y-qqVO*BGrm17&J(3gc%vvXYv2dY$17%`%WJCDC>8Wum9EoP7@_e#sH|m#>E{up3 zEn`m0pivTb73@M`Oa|R)bMV4|ETo!)$~FWQT{43&GvjSg_S4MUvr>fV4I+P%Y%dB1-i<{35sL-*DN7O{mgbhx!*;q_}C0;nzd z2BEp7S?LuE1gcfY7d1SrHwDShD=Cb2ZzbqA&r7=j@>OveW^@OS*pY)AbV+P)N#T3Q zmMH5}29z`&@!Gj2@~E*4XiB%xDN}y}mlv-<&qkPlneQs^nzyY7B-oS3@O$&QC4O2F zTlc5$c5b*y>yk=C6tnP6!}H%+B$hQNQ3~&lFl*86#6!&sB0nhzzNvG{Wze2rdO-M~ zK9Muc-2PSSg%Hy4DUcDkUV~9;5A`??k8r!wxrQDjavSMAMvT9VsTNC@|IUk!KfZ^M z4-a|v-pbhI+x}U}h@FULBO-mv9ahD7qkf6UO+rokQVa3No-W6Ly6yEHUAGxzAB6Dh zmo;rbi-u`FA+etQRr;0Q;7T+iWKWXNFSRa)(6#7_Pxw7aor~06`o(hDQ%cc!H|%_f z!o^&I<&DSgqfysTJL${BJhW~MRX?_ajTJN%1Z#`da^xjjS4Z2*;D_VR>`upY-=_FW zWIyAyV|eQ)kL1|YPGVp!$Z_lHkm%Xrc?pWD8qjf=P{|`)SMM0_7MwvtNID=(8A8kx zgi}+xv^`|!OGeP3gzBAYpXo%l<BUbWJNsY$p{C-Dj%W3j0Qu{K#kB^2#Gm?fMPt07nxAuut(Jjf>l8bWNo`HuguIPrgMo9gY8J1LHSR52wSj@AGrP`8>t# zJc>=#Ib84V1so<#$>C=-+Ig++%#?(!v^b$CXzEU zgxi)&Sis=qiIg_!z*k}0hlp9e+cm)=ad>FfVJZrT7$glQPt<5uN?hO2E}x$Q6CsZv09o~!VmA8$t=90 zlPO>(LPW;!_37C69iqe#BsaPD#*o#yfPATOTMECX>1`&DGns;eWbtPsV62U9-#Z2i zmZb2@l*G*3wfKlrks$$h_GDLbYS+H8EZ)@Ymj_qFJ)M8zK3w9V8V(GQhWnT>9O>p) z@`5PH`#yHE{*-pbLz1YaL8xVxp)fb z7=H$yQJ}cjAsfBywZpfETF*j@C&ARZ)2iEoomsELkHHol;JR}@^F};P9?^GWipN@4 z<9*`d`ZuhFGyz|wi7F9SNq7Z6NDbki17;oaqQ;+HZ7mgQVOd5>K_NGvGpAFY?ln~g`vYtCB8smc%6;po zFK3c8<^Dll)|a=aq0*YNuqs(KBLryq57Cii4Vd-x5#>}b^1kAPC5+Mmcvfm8i)Zx% zJZH5HHt0S2F87J0VbBP8PPIk}^^Xh)hGUZxUqI8`SW#@r6DkJv?<5V5Qp#!jSlBea zi(c+oSHucGDrIWvWd~2C(3Ytt6mvMiD-x$%7d0DS>d6Sf!?P^yNJ$H^QWKjlV=M1B zUXS(V#XqGf^Ss{MydPHeUeQCORY!mjBgUXfM#*#1rRXTI#n$-ATV&I1z4gM7l}Me1 z%W`asloza!`WBX2rMQkNTv*m%V}vgHb^!&1nWnJael-|ZQO=^bSRXQLaYI6@gE8fRsf{#3l*|iET(?=9Og!Y}tOW%vgWK0FFYeEg7B?Ck?-Yh%EyB)Jg z`AX>fUQ$MrNgqB-BB%Ju5vUSG`P*Y5)~=vAxU|ZSM%CI++gK6Mkx60*pXJadw6BzU>FZZfGS4&W=9G05~)g!OY$(TzSP_T>~ z#|bme)=x9{(N5AOyec_QtC=uVMr+2Q$nCGsd^X^n!XYiFc6~UbuaucA_pMEHN1pVU z7dMU*s?6|MVBp8m{zaymlCBHuxaXxhAU(A6D()61>#=3$eeod|*E#3FS?dkR_tr9& z$N6!%#D$f!VV-lg3)9AWxs%b|#&c$iR&{EftA(J(8E%Nu@lL80HH056Usplf5N}fT z)QOBAS`bIKS~@x65*1lVq z$USv!bL19M_9#w=2jqT=y`oBxi)<4d!$%|9?0k);)Kn;3;<&<^GIm`BzRmvV)eji5 z2Uk^Qxzl+)xdewjEC9#d^tFQpfYBjIjOZ%nAEU#OeAG$=@*jHl5m%>Hkycy{e zPbrp_KVKTU9anCS%48JF^%*kE*d4eF z-MiHwCBiJgv~bO~E<5g0HwHfdU?~zRoN{Is7a(Gx))@04WWk<@h?8h0fv^hZli;9- zdvg^<(CQI%aKYCcDAtAE!n^*)u#yPcmSRIL9un|$Y`Ie+#Z9$&wXl*FaGGY(|VgL3nj*`Sm`K;ah}%DCu0C6%pWsXYWThUI}2> zdRkP$$o<;@&XiW_Ho+wl- zK36t}BEW8c+TgGe$F2gz1b4)?+oT1{yDF%wA1(B+eKW|HzU-GAq7n{+!PJO?MBQUT zjU-47$01F;U&bf*rdOw}EG4I&jK@pRaplNuDS2>v|Zu z;5;yGdO~0AU%6wED&X;p_@KhFC%0*9)+MhPQ;T;ddfJUVVzEcC}=h5IDB(1~1(;K)|A9{sNId)f5V|<%2igI8- z=T+m~6{JFSYnRKTcY2@+P*H(5o_FtNa@?E!XYmCMINoPLk9!*i3V(-%YpW#^Kz%Tp zzj&Us;pOq^QXd3d%ft0wZe_IuoDHQe`jWwNEzV#^jvghUd!cY~F6ss7DREE@4Q(Eh|%b= zhL`v-C{H$ujy*a@q$O59|NP?wre&9bF72#l$!@uunXg`9O98hUQdhhKlY^ALRUCkM z7m#CJOM4G{cWHnKu-^s67sag}S;huDh{bro`jsKwM}FyOyLqBEyI~yiytC(+U7i}f z#1IMk#7cI+lnP@eQ>6Fcf;qhH8)fX6`-gtvf4Az%P=*LteN04ob)S6%BK7U4M(f(EKe5)KN7J;% zic}wV>?W!sn=f!lDRd*9QreR5q=es+(g8L%&bnZ)J)2N+BCTEMjUO>dNfGUqdu5A(k@q{VO@1eQcuE zSFV#VHQr;&m6g2B631f+b*mzapP27Prf8o71=o@Y)xNUFZ5n=7=U9?a^BQ(#9h!?l zeqFwH;jcRd&`MY|vtZV;sM)Oaj<=zD5~@kAH(^m_(4lm^2_k#%@av&2S}`NCdk|FD zb$dPe!Y?Xu7zi&H%3bzp$FDjo=@*QcTQQuc!z?vN4K=m)}b-UXMh)G|CC)Tb&hho73sk6XHxft&0w<5%C62?^{75 zAN-BSHJ;hYQuV`@`A|*N#$Y@_LTV zv$tZ2czTV@AAY1G+f!eW`o!bbkrY1UWrJ>Mm_arikLS@=0+UMU+fCSL{2oRX17-so z-L+ZjLeyF88#_tThvduR$Z}qZ@NkkVWkzZj83vjx2*lu1`_Vv(FxRLPSqxY}YIfsRd9wm~3xJR9ICD2>v1Tp2t8hf#Ewm*b2X0pU-l@||sc{9jR<`=D$ zoG@CpyadO#n!IpRw!uWVod!i0IEsO81KJ+3wlweU;+#_?A!fIpeErbd!~q0s z?2nS{FEvIP*S*V8MReu3Ms;6!@7jvX;r$y-yB^f97bq}CW#oyx^eFI9e7B#qq`kC? z-jiJ+M4+mPs$oE3=!ahU@LyG+_gGK)&A2oB^w%Nbyq=+aeohnau+xiUGppVw(&fyS zMWn?QjJ-ATl)zt+*sC;q-AD8su(HvE?eR6)p(%{e3+OX$UBS~EI_1`ORazK%7RC?H zi49Ez0qK+(>f5Lo$!EGe^PM!67`SGI2iGCvP*DV zO!!!Kyc%DN>&%|+?<-mg%GTkosvy8;@Dtj(rX<1-D;R=(!B3e(1m7ZA74tj?juDr) zUwlHZj}UW?-?{3f(JPncwgK^Bnm?Dl!fS1Awl23R98`Hf}9?2Ih>6u^dxVBYRU z!vD@K#;@d8+(6!UJ31*skNeopzVcAqxyv$;NjNCThJ@p#Up9X3p|6^>&fypDUPjT- zKFxy1Pj5q#($RLl<9_K{S65i4bBTKLh=MQX~2K^M<-DY zr19$Ib3`)Tw52B3>%;JRmXR9Z3T_#eq5O2c6|fV2iAXm+LKL5~McOWwnSWBtNTFeo z()1fQll0fXuG$crZ+0X}vl$&ii6!nRNHq=Ppsfaj!b2d6FXjys$^U5SpDA zy4Ix*-Z@$?<*s`1B3%d5`CL^qqaH|kmmkXEtum7>MJNk{2+B^dMcGVsUD!|D!a69 z>n&)#=3Oy%q1A8590HNcu}Q_AVK-2>of^DDH~FAfKHWpXMD622gE3`X?2HodHb`u7642eX%P1ibA+0tNKsFkBtQIH^ z;t!MK!N~Vd`hse?B*|Aj26U^+%?<{oFoMQDil4IYc0bg-!6iSCB-_XLDf08*jL|p1 zm!;d3J8Qr5v1MzCySjh@Db`ISGdZ}&;eH91h+%LJIBhDobH~_~Dz2Oye`%P0?tdO1 zW%Iu7F3~eZ$6yVP#u}w@%ZR&{^h! z>u*-<3PhARu70WIkS{H3gA9V&Kf&^%o(@|L@mky^>CDt6qr7I~emq|uI&V&%b1nv?tF`JtgyUKgf8?`=Qj@Vlf6PK=1JQ!|aP5khwL>Z1@ z$`Em*@+7&jVH?97%DX-p%LWA^g}CZXTfM8Gam$kUZc%MpUxO>SPUyVy=l|o2yOiE5 zj*D+W=g+sYIe7}>$~Xa?SUczQG@EdR4SpEOSZgVouOdztwLy+g6r6qe{^-x!@>;0g z`_?kn+7!$x`)pVtJjuzAkNcX?dHHy~B02Z8v!ANP<4gF{G7{nq^4J9X7RWh~jtY=V zqxy2D15U99EOC)4`(o`a3bQ&oF`2)HxNu5l_e@HziU>aq%s@w79@D!H!jr~hVS%k zeOR~RI`p-O@G*trGT%ubp~0BgRXM32`98`*aZkDQ&6zDuwdG@;%fNs*I?f^7fCybB zn~k!gG2>q7F!@(ZhpT+}n_laqEKGuiwez7R5`t0hG-GsIh(svK>-UT$i=o{{_8)wA zi91O5n|fOYvMR#Os11Qs8-dGB2KwI%AV;_?9(N`Y=02Zc_%gbrMz-z{i^NKO8*1{0 z|Dd<(=D9dV*!hX}tu?8TbhbjsQ}jt2{udnwV^%2mwx)oDh$+};7XctV(a&@Y>k4K& zdjxu%{umoar34a7h&SoeUuqkZCN*JSlgf1^2lhA}7V$zWVOqVetW4cVn~E=>gB@ou z_4Au?)vR*dqRF1xh>lxmEQm+--O5Zj3+v}IT|RoOq&tefChxu@k+H&_o^l~>PJUCy z+ZOY2BTqcVcugs{&Oe2K%IVN7dbyd28(&Ktz&h+_Oc6aI-Ov@G;a>KLBy?u~Gp_P@ zDch(s4^T0Ie&&KXH21oQL8Xy%veQG#+$aXjtA1){Yr7zaIhaZ^(f`~p#)>jGzezDS zU!=muHRKSDgMr;HPl-*&%(2ZMGh|Fdv?RK)UCa075kgU9Mc7atJ!cDA2^AA97CnJx z5mD4UdMc5c{T_A>1OAB@m-0(Y$JT% z5z_Q!+ z=!wJXaGYn4Kqi&J7~vNm-K0@^q~%TM6xFcp5@M^SK{7NkPjg)Np|PepUde1|Uk&C+ zCqJHZnN{3`gq2Z-ZEX^V*BuH?(T3#pJE?kh;>S4w zB&S5*OHyVe-vHU>uykGm<(ftxM6BS?REM9uUWPGM1+>aylRNl$ zr7i-I-r36nW;yhlbD4``wQ?WN5^DS!MKh2cRfNBIWldgNZLnNRxxvR&sLbvf?y#vL zom?m%l3|W_2I?A(VSMWH~84F4Nq94 zyQWK$58%Oi+V<@v;R+_*Ka|kekC8UTQGujJ9{b~XX z9=g;~n$EoqHP>>a2uVoelEt#VY9-DRN9DW04nlSp89hSi=2-ukFEV!Lo+RE6wxm4ZPD2Qzn9lg_;p6=x; zt5cj*ZNdrBY?^>N>&P(~#7rp^ZU11kcuNnek8a6N@5BTtDn*g07RNxfu*-WInO322 z(6Ou`1npz%g%HiPSasF%8OVCJP>%{?`KCw+FymnJ)=d%{M7fd0>8QhN_J7ezJY{nOjaV63+NN z8ms&_soYKLAL%&S&xXrc>O4y!2C+Yy3eJt|N+PxPeadB5ws?m-FW{uc?%cOR?vRR@lk{uFmzj~RHkJ+g@DmUJ`_#A}Xc_R3 zK;-Y_EmzJJedtqBj-c+%i>yuq4}R?7?bcCwY|=R8txOO!_-eu0GH)`YK5IQSayy64 z&;kz5lv+-w+}^us1F)+MJ1gpSPu}A~z(B^&p{DeaTJ~Xw5+uIee3M2K2Yc7bT^yO! zurI;Akm(H(FADHs?!9=wb>p~wH9CY#g@V#|1ts}4YsI%bVaMc5Z^mO$x*tnHh}>AC zBZrXdh7&YynMQf=of7Scsv;k1JOe=$Q`seZ!*K&k4-)0}M`GA7-<;$Qsy5{va@Ikn zlEn?=ZtSFV2g29ozEeTnjiLnBnj4Hg!3%A+#8a-mnw9uh@^YF-c6V|v(?))kvm-`O z%M^~he1`)Qy1oV6Lwl8) z$x2q$pS@-Eu5e)^QDbLMn$g6csl+d7q5 zQC;Ui)cP!oDNQKgRGs_6yvh;c`@Y#!{rr0S$oZAr+|R{+%%HBFnN|;5M^|zsZmkX> zBRHLnu7Lh&4Q4lk%CGd-6AEl}#ZLw|_F(4{Y4a<|Xi2n;r1CzKyWCsd??URytQ$_} zFb>=t6n7*q`BV`~M3|FzFP0)g#l>~UHF;VqvlFiQ^ai^cKk2YD?idr@Pe@+`rFyQ~ z+^bP-@mWI>hU+M6xrPlk9;gL`r>%RuHGNA_DJR|y2-LR~lKE|{EaeD1F^ZyCV=f48 z4&bTOe{}WvF64@&(cU&vacT-koJv{UN!!I!@WxSW0DasJDaU5Tu%uz7xC>6!Qg&xMXX;pbc4J&k0)d3$Nr1CmcvxTDSCA6&&R)D zr!r3zmrh|Y+S0#|cF+>nOixewPk|2%$c?(f(;YXoyuuQK zwwrVH%^T^CQR&sV{k-@3_KrKP42jU22JgFc$w~&Qyb-Ae)McVW6Wust*mf{?(hk=m z6%glBHzrA`GDb+g<=z!vwOa^?yX{8}taCGbCm4+lT=k+TOl)vbj}3O=3Zd|rT1O2$ z`HnOzwu^8P5y$eF63*JpXWdnuoP*!xg+g76I1l>+E)^nXr*HN^Q7hHU9zQYrYPR@X z)J(}g6UwWZAn8QZYHljRn*u?Rv^*&ei8|al^LWKmYj`~fgdxOGT%1gjM9M#gx3Tv@ z{^4scS72D~6_3HgE#%g*InE}QB?jE%N96d3!?CujZPAjCL;|WXBCto<9Bhq+_MMpP z3~|am2yW9IN)Y1Fmz?rNiz^(uX9-5_CG~As=R=xt_el)`%}2eAAJMj{P`GN0z4?Nq zs#&IJXW@{hFfbojXJCPKbxHdrJ)fB3T2UO78yT{%;ysD3ZuQYwYt`c;p4YN+qP}n*2K2WiETUk zRc-Ct{)+B?IHwTiQ%nQcxHI&(sPP|~G9)-~V63bN`8eC z$oO|k;#2wF85C{X8cVA}G)T6DS-!ENxXU=ogRXBB74YH`gPj0OY{9`PNxY={T+fk2 zszj^WqX_HmWkgx@N%d#3I|QSDD6R+Oysbr^Uh+y01y2s$%J^bW;zNKZUfetvZJ~fu ziH@A6H;?N$#9y!gFLaL}K7LSqN6$Eoa!^K?+tNa&Dg#@Nu&tilGsry!@Y($R}D*Bk?gzsP#X8%jBiT(ZdThH={)4CK@xMTf7|{G}vmyJMX2+#bVC zWkRqDEkW!_@rHmS>2rG;A5@tkIQtd}uc!ib^SgX3S|fkdpaCX!@UWMx>~&FdDOEwN z_k5e~9u`uj;tZy)i%SpLga~=n`eYN1)`B{(40N>&JR^mgo`FC@FeL;5Hl5JVVqzgh zD#8Yk89{#ix$iuypS5z`&INYZZ=H-0k-+C#9O;22B!7xO7iz{2`X*_qTD(wQv~qZ3 ze~onOG23uI*1L0fKl33C?PX2DcD4us2d0P0S3>YvkFzVlndYsNC3O2f2=nNNd(Et834zUEB!GT7(BbZcwg2q)%{c!gnv z>UH$~SyKA0^jSXV#F6*wTOwqxu3VNe#t=I8FYIUsQU^wQeFFPyCnz&9Gt#wztYeN5 zT^TXpHkQ0I(SrNvFu{6HLGg>>#rBRls=+rrJeUj) zpyXFz1@^s|n?5*-0~AcT1%ojX)Gb=!?FlrLrUxElMJb`HD8#qb@fz zY*QsgI4h$EVg!w6K8|iUy4_$Lu%N#0}r z$zDqd1_B{;#{Eu5&lu)~1U@Ts@$%{Sm}As} zpes)vrK)ZNH~;!}vr`fzapqM6{OdZ#UFJeBNrz{ zdUvg(3-T2)r*~JhV@7|-BLoCscgJFyVHlG05*^Ffn8znbbJ@ZK>;7}VawVheL&w8L z6DW&XY-bJyg`SeaHf5b>En4M+j8D4xrL+uh+miG>2cL%!&y9wWc>yW1w=@YgaaQQC zQLvDR4Chhpww4sJn1MBODmOyZU@kB2+x=QAf^ayrjhKU9S~(Ydh`6x&893 zF(bYACyOPup;?vxTkA{cBLe~MJbhy-Baa=~07{sf)yfv61#>(Z!}xQSwVpYs>OJVw zpL|Z}XwxJ91!6Dqm)}6zBc)%0%y8Vnf%B4dWnqqfU3%n8Y6!1G(~xxI%kb0JiqG?Z z7WBn}XG?JmED?&BRk$8(3&Srhd$1vxz@cti-yHqK+^pk{u;|k3IV0_uXDOTv2KgOwac>JTX)ox=nQ{O z1DurkBY~#5uq=Be09r6DX~m@*tgtADA>8@{X=Pk5FQuF}V60rZXV3heBRCWEeWn1& zl~jFB<|PRSH8$_9>w>yYi8@AfBRmHRiv!4*;=$q5(ivF7Ca_8^Kw0 z{hEG3qwY+T_%ZQlFFGuK;%8`X=s?}`5vrV?a_XW>mKXM$xJ0E;2R3);AGv9~Zds{5 zQTslnayF&Xmp&5|n_Td69om7r^bS*#t^ofD16{aZparEtt373U0#TB99cz^p&Ll_T z*PfP;YFe%=`fJVebp3-b!X2p5(C&B77&2uQugiUJ0*M0}V15=D?#sPrl6#kh4*xhU zh$#lp=Ct)^GPVBCA2l?telEm?fFuwR8Q;oL;UVQ-{@tMY(6B^8p8#F4FxA@3xWC=- zPM&14Z`tN-#`*~fGf?Q#!Sra8S84fKIQ}aIk7@R@?qb-9e zdJh9DyOl7fjxxWV{*pOrsDZW_j{s zvb4^0$p#TU*+i#(iUU$t-!N zMEto=n>}dNTvbxnAM-yZ1yJMpkM#1yu0Mo8gr=lfyFVVhL*Xx=9HyuKQ*3ikPk5@n ze^q&g=#jYd$Qh*pW|H6NYpp5Y=jPZqyMZT44JaWB?XQ6TY}bf(domG8BYMsvHW#Oa;irCfCpN z^AxK5sBRlJsfV~{cM=1!0JCuP#NFJt&yhC$2r3hv3^sb*-&hM98ID72Ux9bxo@lU} zO%!8wUwBs{4Koru{n5dY27+mx9T0;649;c{X%F%9G|)8Qm-^|yM{<$}{BNRjO1h50 z_i!~K1g;RJqhalU!}SY>x-m18njLcTM*U;*yHPsvnLBn+$Zng`(^+wOqm_%y%4t<2 z+D4!~6?5I{W{C+nL2)x=Bn27Ib*0z2$dRv-i?oMm<`>aAUXADMBo?ZJ%4$5MP{qYowyJ7V5nH+3o8Eb6?c>0IOUiL`? zh~E*O_LISUfj3VE-|ZS{C}3SL59$~Ep$D@Rl+t>Fc<@f$3qSJ>$s#zITd+j+`kE^* z#M(HoROATF3QR8E?c*gw6ScJ)k}0zR>nq3$hd7%M%tkRg03L{8)LG<2;E9zDXZ4vO zrLT()Shx|HLZ);|M?JPN8>T>33M+$n3*j-i+A^jO=xnb<5HpmYiPEDqkFGVNCk5DU zUe}fSvu@^p0r!GYT|#WY-MsNi$2ulfU|_@ZWBf_%5DXBJf-#Ktl@j(n)CEjTabFT3 z`j;1TTkVGV>rItcHP?wDNk7PA7so<%KT{bRBEa5pMSjjg7^E6JU@!b*KK6_P=+b$M z9q6U-H$u@vgZi|uWBotlBCqZUi;hv6X$$LQc*Sx%A)0@r*sHGvHceWK;?%3A)Hq9U zoVBx-m%CiGje~X>|H2~5(p7585qW)d-F>Zb1&B1G?AEhT-O~OH_KJX_e2KPqSAnP^ zskmw9SbC^iycV+)J^L49A67Bg#qwc+)HP2>$t-NjivdSYnrP|l2KHf9NDkhG``^M@ z)(~rE^J^dyd6S!nF4$Vvu?5kVYEJ%0+|-!Dy;aLUQk zhDt%Q*U&wfEm*+`O#A~NZQXoySk-#uMiM^84y&2I=C7{kmVmN#dczB-9wP*P?rGUw(;7wU%UIYZQHhO+qP}nwr$(CHT_T3OwD4lOYS0zBvt3; z;JepaE`Iqb2UzP+Y8$zTpqb7-qr$B6$%h8gtEUFE93$7PYORlI#EJ{CjxrhG)!Gjf zR1mGI_1INjX~S;Ne@+zMo=c3K)w0>78f_^IWBnp1z%xsRe-e&d`Ns>A_9Om=?tC1g zWg=uPb)J+8GFnfC&Y~LN*lm20vzMBAJZ?pKUj6p`*QF)-hTA@2h-5TP zK92=)`gYw)|AR1-5LJH;z;eM*-A(0pU>Bq&4|`be^aE?JZ77Q9XTmLAzmA+b(btFH zBtIqM;Z|lD1AzOzY8(0kwz;FizpAqH%^Xd?dkaph)(gWLPe8cDu8Wn`@D#f+Yn+2? z zo~n2Jlo-;yiHIG&yCeX+bzzN?8&C@V(wxXHa0u%jr3^kSMH^VC2TQ_+>!`nNv7=}< zkw(cHdvJ_csdN13?kDgaFy&2$C0hbQ%Zg3I1}lqzep6V7Fp?cw!C{;0YSX^wG=8}r z4PJ+l!eMg;K&5TxwmWSTIno5!bgd;m2@NCRo-o3zuBq>9IQO(6jjgis)V*1Zm2`;lnkZ%QIK$e3_dna^k8g!XCuWeeCZ`5 z@$7^bNrNSXI9HM}z8J9Qj5&lZ_lxPchP0^0L>A47z}biU%8bS4;mM~Ylu?j+ogj%7 z2K8w^reEm6)_U3;07%tfeV&lyxT8DF0Xu9egB|kpeeuta;^8W#3DEt;b`ZLeK&G3h zA4rKu$YMO2Zg{9b;W$ltvsIEX%C!o3s(1_Ggmz=28)N zByow|plqO4p`#eEix|7Dh>P3-AcW-F#vdBCQAnZqN38kc*Qe$>FY z*vvdSHIBDXm`PJ%qBX18JnBr2ssWu^tQu?UhWIMu55UPtxQ`F?E}YO93*ME@Esi}F z19JtYihYT7W5Usm%@^*3>4LV+r$gJn!T*y!VW5WOjn}>iVUFZgdj(PGBw*Ad&(J(#v46x+?ON+*1;1cgtbn+* zqt>^s(1qKo7dh1JH3LzgY*Ei>zvVjMBZe7Nc!kJm3DB7yyALK8!&p(&YJNJ5*4iAq z2fRmlSqqi{6B}|${NhFy923tXTXbexKEFw1YZ zu$id_=&sjtCa`UjUJZa|;7`e)?tC(i%eQ7&iy)tR@XZnC5f-6zNhbrj-w@(3R?9}i zFjyoFQw7!Il^i0dhW%mZ$T`KPzC6&%Xq!ox(2HQ5%=6nzpnQ7IEdX+Zvej-lBG)A8 zL%whmwlDv|=EWKOL8CC*@)#EWQBEj49vlkCI_B;k3oVl!VS0LJ7!}ssk*`poQb5bY zU8T;ouNT@e!b&Zc6{!$~XcTV#1vA0c(?V}#-wCos`BiS+S>!^lid zfHfwVIWA(8*qgtQ{Tt}oQD`)xTwCb$+rwshO$ZZ_sUxJA@c}xFTMcfR{2kygVa?e! zV8U>)g;%>L^s*rCUv`Elp*QnxACl+wcW1?=cZh>kq-Ce7$!@h^785p{2;^rnlAfi1 zbXWGG9R-wP7MR=6Kh`GdhL7VK7zGM^g0|1k72~D9|_n(V*f$Hn$JmQVEeN zDiYD719(LzY8r_tx>HCa{-C`94Fk(ei9{vDfUE9atIuqT9AB~Sgj^L>yO#xZQ<P)i~lO&N~idCP(1 zy4vNqhJu4Mu&LAJ^=~)EQn&dPZsmRxnZa>OilmWsF+j0rRCf5B5a%F2jZ)QOuFt6u zgU*BvdT-#TXsr9>X+l411yPhl+rN=1Vt5E^#qB`i0A{q&pZ(a)oX9jh`jLx^F}nk? z*%nOffq*z8V_Fe6`QKK1h@2Lv)5NU%{-vVP+0Dd?|;RuYIOI!JQ*#!xIx?J50p7k9f(AGnwK>HpO z;3Xk2o@UZBa(d_lg7gb1X;iS(xRKj_yreYh*l1%)S5JK-yi-iH4o{cV%MCI(ixlU~ z*YHhzi+7IObP3ygIOq*coquh|pE<0=)~i-3kyk@@<0f=fu^ztKsiI*EZl@fzy$U5k zEBY*7rhjpLPjX2MO-Qx+7-)x(g5-)4_J$Ptk%ey;D z4)Ha=q=IC7|K66p0~o+u;;I%)_3>`ra}|xqLiH&i(LnGUzLsQdHT0X5;p%%WH6eU? z`NOuH;Q7JOB2GHXfkSsTkPS1d zUluU{TJ~-62PKlhsiKL7dnx*SbDX}XVnPFx?BHS&sj$LqpWO`wmp#7TvkNshdNyVb z*8;nQ2jo$;$DF9p9?Dvx4W^V;HkjWD3ce1P13Wf*V!pd=WgY#!_D+lA z|LcA*#!2bl7k~t~_OYHa1wl1~zt!>f9ufdzr zr`kH5RQ%APEXi6XjMWqL<;kt+^DzqL)A=`rCffUbiEBfXM*b^cu@Zwf*&hD=wR|QD zyLD93JJm=_h%8}gkm(*Nub%|6xtoXx5itR3ep$&6=$MB39~WBRUQc*(v>6+3o$?=q z*H8JE8`NYLU3C@uFFt~?+EC_?Wq&UsP|D_aamIqlpRW1bG>PMf3YkLKjC!@RceK?@ z(YYx5+j;&^uNn{0Z)}0irKBtMHF<$l^~?kRAzD{MnTqADbO-qt_h`jgG)0V5`C`nnaM{2ztyOFGiPI+F zwim*J;xBWB(W(|gc00_aEhAPw7yf8D5*ud^Qqt|!$<24t3B5_25UBtOa5<}v*B!=YA zR2U$m`X#7I;bsNHaLPodAH?cUzxc$*1unyRiQE;Qhd(!{7n~M~r2tV4tW%?CB9>0P zQIxc@gh*~OYQ_UXYsqGrGEGjHepRQqR<609|6;asw{<(i%s{SopgUcY-s%KC4 z9J{~NWa8e%bj$)`j|1JaXYK8QPJrs^i@KE^yx(Zev{xbB*$KK|L=!YKOvkv_cF!A&9mcizKy zkZ+|o5T413>TrY{UCy$-xj-s$AlvB*d z$u`(yzEc+UYwrQ~@3j6cxMg{qAXX0 z-HWofbFwo>%6;#JT#_dE?)@CW1DPFJTXl-Yrjy56U>-E8W{JDFy3i2Q8k?Px!pX`uNdj0FojmkDMk#x~GB0 z^;M9hD!8Rpl5|PQu7lggDJj|E1?*6c(3y@8!bB`{Xkq?2>09rLw^}&iPF@LOZVkvh z#Sb#N2i|MTdTu9%ZUZY6)D|5&Ig;tuI{)4tE3>M_oyqJZ4}kVV#(F5u=1fOW8}TzK z;440^tC$(D48n^nov!g&D4z>eWwyYrZA!kq9?IkB<@^gpo3wNZ;ED~J`mGbMSBnCQy51|qQ zSdMEolOrDbV@Ep{vWfvKp5;RvtIO3Q`CO)l$_04egr?W}xiC=3J$>A7qTuZLm|&1@ zlCCUV6dYjtR!1nxx?Or7z|U6QRH~M<(%;cjuX7KH2PO{BC9T}qb#-NGCYg?Sx?n@gBwplE~-o^W$0Q6#Szw52rM*~C?X_y-Pf#o z)+r;mHe4mB^avyRE|~z;*1=20(JZx3>Y0LR+Woz(TrRcBEkjY9FjmPf>vfvOgW`5z zjE{<&6HTRv2|rri8U|JrQ?I z5qKl05vboy+jA}sON(mB)tfL(bNBV&)U|?^raRj*Vq=-} z$|4zvjcu8?0!(_0qn9_3 zZ=|MLSW9B-<9lNFTxO|5XS*rx2qLxK&#Qq(4sJ5%AHYjxe?l>$m*dL-0%Y`Qq$MBF zWKnG)*-7I0vS5Gz$G({yYtM)#597}?d8FSWaW=_O{b>Z27|PKyU-`4n#H2Pa2SAql z;#Joj-x%CsKMNtEO3vO$nQv)6v(dC;6Nb@#pORFHCpoq5l1ZRm?Zj^F`rZTCS46-w zw`^v-e>rUr+_i1-A~sn{$r*|5$$q*lJV^3B#_q`FbW3e^d$ui&dSM5Vd04vaK~1jq zA?mSyW_Ms3{R0_SE#;2x1GrFtUT3I4m7?PFSmW#_;|JEjD&bMuOY?;|nsgMeT*Wdx zUQNq#Cs!s6v?_<$9iLE}L33z_qK-2vj+Z{#MwSPE!LoXiA^Tz|Uq^68&rlJAN zi7~wjw>p}CP%MnTS{jCuUv0S%m5c~6&IiYKci z$$|E@AexS0pJY5A#WCl8@G1t3f7(fIBxz%9%>`TvZsL2kk;oYOcDZUyACWctz8YoF zeh7sU-2o74PzbO!uw4*kJ6~+Dbte><<^O(9OYUBu>SPjMaVwc3t zHZ36?N99R@!AhR zJ-vSngtr{cx2g^&Dn(Zk#x(r9A!?`n%nvm_dfl@jrU^2W4?ByUYA?T@BA zpKIH`7}w&oOqplTEQ*&>f@G;F7{O!}rNi@Bu9OSsc|Ypq5!j+uM%|Q+PwEiTz$D;c zLXVWXFuRruf+cbCDgHA9?CiZQ{oMYM__(chZPcSZsXK#C2aS>4NllMDmcH&NtR(*>ikw z(*vVX_GiA~LdU%}n(>iBaGo;MH*6N_-b?atCh7E`&sG~Hc;FH285*r{U1Nls0<)6Z z=B|nYw=Py;zW?Q-ynPA30iYY08S`(GKr_i6FBhTyP9i4jC;aY+#*r2CTDt;^EUYJB_#mJ*G|8#c-)K?G`bk;M5ejBp#l8OdR0 z2VPf}w+8r}LKlo38{=jh{!@$+hU&_V%~}Hl2&hj^4CsUvY@c{3ycmqq6qs-$0{Y2A z%|wO7ow@O>pu|R1gmQ?f(JNl0tR5u#^nAT69BzR@{sITRv?vreK9y>$UBZ2=LV~6P zZu8qLDG3{e*=pcmW{*g^EujZAKXB!_gJE9jQIy!l1pg>L4#2iyp3w~cGg*c(oIr_w z?mMf`GmN^6MW-IIIjDQwUJ@>d+sgUXMz)-1C1dlADK*NKyaXK4KaMZVU78q61*$rgW`7v`B91 zAqXb%4ifF)!d>4$+Pi-Jwl(U|Wvn{9qh^28jaV#vAhR5etHhZBzwqx!>C$XqJgiX6 zJs@FDAWGjeo=)jRFJUBA1kYIu z%Cb!28MSXj4D^@4y;M&eyZM~QN(e~hU$<4U0fn#DjmmC==UAxRn))a}KR!<+6Y?dDMu*$;)W3V9nT1d`0B_@Bp+`Q4#MQY}+zb?t$uY6vsuI z1lodahS_wU@mMu0te1~uWtdAhpOXeJ3wqp`1}80VO^+I`peJ8HB0fw{4wYtVjYqe) z#nI)gZOhA-*fFCGddIArdR#io7HV8CcG`IbKGRa$rR{jdilf_rE1Bm~SEXHH&2Iy1 z;QJ*R$C)9Y;FZY5eO$m)d?ghoHc>a?;(tOj(N^Qg`8rjL#fd(XfAo$sN|&C zvKmz=x`A`0QqgdJ6WSJZ$_=V(7^m0azD*6_Q400GK21#iieaO_HpG@Mwerwjk_29W zWIC<}zg;dSjSVLG3Yl>vz1f7>HawqcL$SawPN1fL)kY%1#X*5Q<^a+I3RMCcRDhSK z&Oznq*maFQ^RWkMmM+aB6c9n)CuK7#6kJvT&P6a15;d`?gcV$A+TLK^{ z6r%sT;q@;MzC~e=mpJ}IO|&53niB7HeR(^SRL{xyXX>(PQ~tRE9c^90`Kj}}m!U1B zW_J_GSkacdbn}Ec*3-7Th}AwK2-H981zX~lYk{C6CP8aevGEVWGl$KjPQi7jE981yw$=_c0U6-c_0Gi?9?> z!yO!`rdKhiIBbH4bumem*|D49UOoiI#7-?fy=>P6IN@gh-_iwk-O$St$52kQXLpaq5pix^pq8XYkiuWR_)F#^~)3DAaTtZdL&m_>CbO;sfBWGfZ9|`^i#==bO}tn-CbJe6n2^-6tfSwMigc^9T>-lcu?? zzWX7w_NzProkKFI4*mX#(0-g%UBueC$`;>O7tOdQNzzIke<{!%*p1R|t>Er80}Z5f zFh%|KtQex)m1ndt5^O>I_>^C-KV=e(L2T@W3+t7FmSd_l?n~ zp)90xI?+SEK37KUd#;S@Y*nf~~m!9-wWd*ae*H%79K4$@e1eY9cyz($vG1nXyMZ_HMD68JRf=- zv)iE> zj+AP6c2zExmSpWoq}^rIC)bQ z|L2xH$>6rRnb^D=h!()tbnG2FbEqNb;9N$?EL5 z)>|mvd08w6g)C%lc%DqIlSM5TxR5H7P~lzj=_X_DeF_RHDm^-ZIn#VYUW=xFu!h!t zinCXIG{A8UJ-r;a!4^Eq`E|*70$xygC_MF|kr*e3cC(Tv&4IIXcpdSGyArX62b%00 zk@Ge}J!qoJVk~g(xyrWFE&1#HJtBw4qV9ZHzqQ0g1c+x0R5s+ibH%y^A=I&+a>C$p zDY4DRt@GC{%_Y>?3sZ*Rf}7AEWFJqRIlVDKZsZ%hZt&5XMZTLbT}C^?=G$Bt+;;(3 zHqNSazfpyNRAg3MIy8#4M|izz*l0m=i1X80uh(J!2iTyOq_0ws9D9q>r= zq{)K#hk?=~bpu{!n;$$5X+)(dsE1`+;31)(*LeisN|0G-XtZYL-$f%vV71%>dAM=z z4=3}Qa9^o8l!;-Eb>7)+fU!#x^R;(rq5M&8gc`HeQfrwIg5+%PGe>}Anz)#QvI0*F zVj*q*p1EUmAVevQcSSpYPtBLs`~GH~x;)2!xe--Yw5s52n2trZb8Iu81{n2<5g$DH zCgoQEe1Hh*(Y=N%JevIjY$LFW_y4Ju|NdXql7;^Ns+P=*|9i@aPxn77_zY}xEX?@- zIiP5T%`6>_?D1)ZE%h9Y1dR-A42__;xuF~!?Tz%Tpj_7*+l>@6H|CkdAz}ggs9k?y z%Ps0w4_nNTVUkuP|6h>tm$&2qadC0!$m-g0sGq^z+2KH!Vw^Bmsw#g1C=d%fOfm(H z;sxWSk8J{`Cc*(m<5d?%Ff6i}8Wp4jss9H=H%Cub+Zz`TwE_5qGs(M5_zuTQ1JDsb z!8<(*_*>GgtFOBoNCX-RT;=DiTZGWti-HgAIIH5^1jqnH{`doQkM9Al4(#Poi)m`5 zZwy-q(g6`*sd+UiDT(vTS{4_5{fD*ugTpJoGYfQRX`=@=+rkVaIzKiGXvjCMn}^{G zWC%+KkbbU#3CueTV3f)zUP4$SCsId8JLu)G-w7u=wCmc z50Lv8mRD0q?Cr(HD3^)V2MN=+l0o$+HBKw~6bQwM0L)DMD#Y(j}|i-0KPR!`i#$C_dUb z2JpbY^)Z!vy85|KN=``uQ$5R2;{d37s)FqchI@yLeg^33G2BI)4gI|NWi9nYI68K) z8Ghu}zEX~P%Aoyvlf3F2A7)*?U&78bIM#i>L~OoWECphly3)D=zIG7&01Oxo5MYmOYVzRj_=*#>=Dtq{_^` z-C0@Pl-R(oV5zEVfYy<_a?C*OpG?F6?(7z^z^kCT2dBG$bYN?JU)-Y1$^c!vHgh|n zu)MU7;B^6MKlpF}U88t|khy4IwE89hvk_l>kpS*@JfV=hKX35*V6zgxJPd&Q7!SlG z%@?@en$b5tUB@LJh;hmX?LI)m#1Ee?_t5WR$s@SmTjDQrdG*)lPpa0_uX|bXg=Xg; z_l*ZKJ+uoCbfW*NGYGYe;ZN0P@{I=xJz*1;w1np)3})Le{Nuap4fq4t2a5jVC%OAN zZ2Th|K*{h14iduhB>+Y^}#ry-IDL z!2T0Eq@Py5c#N#0b}Y+My^3wb^O6g-EAU=j2iv zhf!AQR)+&>8*J^HIU@ci4$zab;>UVm`VeA0OZN3xrdwLEd{Ky#iD|@kuIckF4G}x9 zQ9;fL`bc(ms52RhJ4gC)?@=Fl4>HDf)RIhko|Ve$)1j|gRix510xNzM)s9_9_8R5}S-00BD9{z!A5|yoDE>vTmojUe zx2FkY%NokU>$Qmz2l+>(!i8QUULlyYkF=g{qLAAqV~QCQ%7?AcmN+v#+E$|Se9Yq< zYI>?06XN*8DCqI6P>#iFOk39#SajRY zCE*a2e_`AqkYzC$_R?yUO=BaaZo;+-Co+E<7kem71YeqBJo5R)w5IW(u0483AKxA= zXZ#(LG$txU-CaSW+IV7z>&NY+rw%I}+@-7uPERS#WREm)&`6ok{K?huHsA+5rDoBD z9P~}|SqTS%5cTX!vD>5Q4#Z~iSgm(Dv_P7DICkloXgcEgT^!QeL*Zq7&ud0+N!0k^ z_tQWBiJN8`H>uxKZs_Y3U>X#K@+La#L4h4Q;1^2|5W@;aqdFldB8U^vSxM}I5z z%v6GVhu3opN_@oyFVJK#{tk>oa?EyFu&FcY5ZiBIM6el`X+T(s8O0+@AJvqk%z}Ri zQr8l9|EDkZy2268yNu!K?ww|YJE+!tfH`M1&X@R@T2pU7%~XrM$2NnZc7teMpMqCc z2mz5|^>1dB^)E&b9-Wf3!MANh!1>Se-C(SN_uT--ilW5u1KXIWsiY>!r{#t#;2y{l z7nhw`PmagLy>YuaLC0?|p6vD3)<&;)2j-Hw0BFez|KaP`)EDT{K@FS84O>NSc?s)+ zYv^S$drkdA%FcZ{k!tOn*3jYI?QVU8A^E?vMaH_9BQS?uU8`g(P3EKqUL}abB`~G1 zHC-a6p)o=k(uaXx#>TtYAI~jo<;463?2g}KNz-kfW@kmUUa!mWhFgG=kW{Vz;UOpS z!5O0ufARJ5FH{lnc|5i;9Ee$T#iY^}ge3l=PG3vkYy_4F4%&ghi|HS1mqm)djay}Y zxAGqAEn%!>rwER~jQsYn z5h6z;F9q{)Jy63H9iQ;{&g*eq(U1%;vN7svj%ebhD(LSbc+$2r$gg7=Ds>D5hn`G>QNy;m^fP6`y9LC(1){q%_WXE$*2>dS z4+l$9Y9p@k=15RsPpEWVn605WZwhMcoL?du=sna*t27}DT^Xirtv==67uoIUpBB?# zJx~Z^tEUa)36!SU##!1uY@7;C3PU`bnitG1d@SuFT0z;7?}YV2)$*D zN;7lnoT8Y5cdzWuZ@-BlJR1m>MU#X7ccz|OI!a{5sp`Z*dbqqi+uf`r>{@f*dwr*S z?j?nZY&;!twZ#HhMaWk9RsHA-;J2wdeLnZjL=H2pD1*$pef;|A6?N!m)n8hj^!N=& zvf_$d?x47!=A1xyn2@fvML)0r%mBGruW0}e7B-uVJZ>gad@8aGecQZ*+xs_Gd%1zY^ACYz+OUxS?SG=9dtO+kz zO*gzEaV4ySq+9kNgT~NKJnV4TN{q}=#8#W#Zy34nI)5g%4tm>IuN|CE|u3lmM^Ghp<~EbD7rk(;7-oMuClGyX|t%e zg;qteYfliCK&|O4{)Dm=)x*<}8in2<+>)1Xle3?vI{xYSu$;HzAWVvjn9A4=K*!P+TV+fQur3O31rsW%9q6MZSc=x=&lGDdc17hiufgB4&2Fm z64V!V2;7OO)m5NpALiu>T||hafzKf+Q9Mzlk5Q zUt<`L(u%G1e)Ka~GBGs#A9@#kcxnA86exNo1vRVkrR z0&Y&6ijzYuNPO0KS6FW2hQa~Zc#)xv+5?CYq7z^BXee_&ER3OcC^&_OuTwGkzjCj{ z7!B)hNov*xAtJfQ#TxImE#`DX87b|(e1XS@AxEJ-Y*ri0A|nhOOEH?=-OJj@o0Equ zhU;C>T}ad%aaa+0^4H5~!Od}F^+n30dRr4K)&&f_HaB42DHdXNqd*93tmH~Ook&@l z`)rmG#QG)Ovu*0p~1cTY2`eHZpS2WV|7Xg93&U8BiPJC!y-F|F3zw~RC2BC*gjc{ z!oA;|PEXt|48!dVw5qRBY$L7;mZP&n zt1kbTe7RDU50Oy$!))x5g-l60qLUtuTm}}GEnBiLepy?>6T>Y(tCs8JbafX3fZZeJ z>jH(Vw**5;l^{>B?3(pNwpc(Q7*%wzZluM zyBhR^c*+06>bZq^znl9Dw!`~JHpZ$RXTJDr!jnK~X$Fc~M{n`jWS+;_CNhB=t-P1M=E6~zz zhrCCMj23;kg>K5CHP}4s{nGE-Y_`Q)l@@6Lgmi8TI1liu^Q=0TV8bp!s3I;j<7HUu zkCBd&c)_&-G~M|Akz>!5l~DQ*gVKP>odiv_xd?pv?D{UoAn-yG1Nv`KC|IbovugVt zq4!Vk$y)T>>7VNwu#0w+4bN}?uRYL_h}5D9!`jb7qr>FvsE9cn7h(d z`5wiF7o|rTmbpP36yV5@PV6jOHek|c!x-u=!%o9J->c)Z+}3pxyByi+cwOISrf?D~ zQTg~Cd|#(4CKWN&8{JQVdBdJ?D6^y7da3y~*AE6;GWjAaN`I$)kGXd}7ljRVXksm- zY6Ql5QaSwap9kf);?_501pd(TzTZuXoDN!89YG!fBy^zU)M81|8Zgo9zuEz*yeq=j zfk~ww!rbr^wP>$(uZ4<(QY5FUc8(Y8pPmoqUO9ZJ62eZigl zM88Gz3WOlIpg7)0*TTFu8h0SRcVVQ(jc+dBCBJ@_a$HTvBdf#g0!TsJe;9J!_Zr`I{t#kUe_RR^QqH z-kUPDwS?`Z3In|?M+)NZ3=8sEwL^yRINz{~0cEJ(9?Xc6s(?W|fz7JBO(wDFT4JoQ zNnU`_V)SfCj{ZE)79BpU)n(rgH2|;j%SA$4x@hT;S5mK(_Tg{Qa``k4{1m~(EfPos z0gfX+YxBi|4o8%+wqkr%oQ3217F_f2RIhP$W-sEhmU|`zw0Ncy2z5*OBZpqRSPb)9 z09fFnd2C8t_5N%a6Tp$?`L}Lss5s#g)Ie0CfJNtyP3A;FrkXhdU1MCvZHKFf@f@!O zQ3tcSNl3j`U?!?zpFUZ%;L)?xFctN5OOi=wVn6FHMrJ|lhhT^-kx)1iNAWg~M(u;_ zh%|zG(}U5()VxxX6c~vE4zC`=o;$H@A9jQsD(_Te6S^JfH*x@?1C^q@F#q@CKLacu zS5$40rp*;MDxvY7AQr8L+J}6ME=^Ws2>{IqU8%og_3nMr?n{@^3(q#D?1*aRKqH^+ z+Cl-^tu(0<5N5NvCD+d{Wo=j3{;pMos>eIQU2rq*?Cd>g8rL+KEahc!k)--fy8Qb% z3qa{`P<7+pGT+ge>UeO11UyI^^G_@dSc@Z3V~CwAcDb3+PL%p~SrFhu+IGMm~tan!CZ} zrG(`*GC-^FhLYMQw{D{*3=XT1{lB8-p@L#zmrl@Zk$HDaLl@!Z^N^oRs{CPNHPI60qy2NZnk!GnFV`ja^ zTKh=5x4r+)JWf87hWM!}e0KVa$m`%}S6I29(j8=jTk+1o66l)n&3@1831CzY0E|$i zMe~#;W9S%e3T)TZC!Y*~dlkO3#rV~gjFm`M{V`Rm`~C`4;T@`Bn&kC<2JZKVZWs7Q zg*#jGwPqwb58D6%=b-r4q^xXD)V;jM;lVdZ!T*@y)NKX9W3`fob`LNaASm7ZNwBa! zlTu?awzLYIGL;5=(YFy&iX+ytCvx}&uN-h2CF)XBQB!(f_&|*tVuZg?0`FzxczOzK zTfOuSzRV{_tZk$)%7p+-9`53vZq*u`Jy!x;77BP7%CR2kV&(EuMT)VC-VDxj=*8dM z;arDrx;(NMdr#cuDBrzY{N#3J@}Y$GY&=ancfY!P8Tz~+lv9Rt4px{MONHzu_cU$G zn->a80Hp)b8R(48SxM$+%f4TpV<*vyD~EHB3n$zsF>*mM54TBwJL;DKX)pA$q#usW#0B`UmKiraYfb5kssfFra>T?Xua}a+eg`nf+c>BPzlQeTbhD2ASwW}fEb5H;dfB0p+R_K zIMP6OO~pj$=q+U-LO?GX$<{Z_EKDW zppwY_+CZ3JWu$hvjyEm~?0%`)Qhp@g=8TLQG+B$2ib}>w64CyiaWQtjt?*4{H0g!y`W_!a3>(T@yEQKvJmcNQ z>-`5tQc!D>^h_l^@jO?IM?AWr@j^BIl&+&rrMqIAjA%$67QRpKrx;&912aS=?o zD$~+Zo*~|BJe~{7AeGrm42O~ABq$k1BH?q1v**zwX zqg|6getDk1rS#A3Joi199poP9Yw7+FCdHutJ)#271ENGEBcdr+9+U9Ss~f-BMmzaG z)}^2D6Q?ZW;kBXBeLnx}w_~b-(>{zA@?C)PV>24(QN_QUyW6YJe=BGWa3S;{~H8FfQk@R@R4!o^3r?|KA#dD zo^CUFi;+KaM7`T`VWwia_xS!OH(2f%gc0|{WpPgqAsbA4`s$w!@7phZYgH6y}d@tL6NhqA92 z4Rfw_a1}Q<41kFD09*u5PuTJR+emab{pf}be|vW%POz_pLB3(e=tLapB30tHL_YgD zpUYWXPz zDTFI6atmAj5K|*@lMSw1Afve`_gMAK$eTA~kL|s=1j|PLB3_UW_OP(NZwwPWIik?7 zNj{cHY89R~GnMGD`$P`WOlmrNSzs5aD386pjYT}uPd^afheyBNwtn7hieJHOT?mZd z=t^E)&vuVL)tbb$83nY!jL>i!E!K(@adSrSRoEe3&> z>ILb{rJg8#7Ir63y+Y}2;-OyPe46>Jd%k4$aj$DJJ`1eJ!XAmIB+Gm^8YMJG&qe|& zzQX-i0Co&8me5%)wCc=5Ik7fG{jG7?&?Ttdm#=(i@|c{6tK&uAHXWZa?h5j~iDxXd zRpgEKne@C2=NF2TY{yShrk66z%Pm+KN*K|1tLkmrA*J6mbHi=ry)K7IeTy?HN)V?; zeSYyPKM5eH-h4~nLhf+Ek)`cz3e6p`z_z3QrKO%vZ%O&hi^Kj->U8S9c68vd0jh%+ ziZblxhw3-vTqGQ92qf%;kFHpaBrgPPN$)&Y$ez(vuMRX)ueyKG1Ciyd;MuH*N&;}{ zf-1UcpkCD-7;dzRRxyHTE&ZMzV^O(ZGEH`)Z-65oz zJ)cyC$z}2ZVL93Hvkr$DyIb0Y*3S=QXhTV${fXH<#kXNHwN}>$V|{y#4`1LZah=*c_k<38!mX} z;Afq9%O`*s__EVi`gM^yzfM=~ZNyD^pd^tEr7@iMz0gTcrrQN_zsq%;ltT@HH+DcH zD&D{d+91+GQQL9D=y_6jgJTX2f6u!`hRA^8=|11O5Z2MKRya!dviAYM9>^jQKNR@X zPGJt$$*DRfO|*4WtZY8cbEOeaNV6BGm%h696Wj+PXy?TZhAyA!&8+?ckx>yfv<9S2 zy8ryaOWn!5ZCxpu$8~VExOj8sjo5mB;Nt#9?x8Sf+bv+mD*N3)(%_M_h*ni)jpBZY zw+QigkrccMOfXw9x|uV9vxuZP!X+g69+jruhLFa-)6OHa&*N!=K{wR~%I$Zo?$Nx% zKFio&i{6P9CyR+uxs#Yl_x8^z>|*nNl}+iGj>UovH@3Wy-HE_Ev#(2jQFh<85z0GX zTdtVyaz>@9kVjMlw>cRjxfiY9NJtV`8~6&ub7}2;+|sd%_rzUh>_D+F6JCRf@2n4# zF)U-#^+nn!c=C|aE=!9XTf2+fZ1(fj#ubWyNi?Ag0u;CrIF#IK7ME{J=-s9%2l6|L z>qJ)y;wv@FEx*k<%f3)t?G8_yj^p|ytGUaacn9F7+IrdpfX2#iLNjrX@_<+sa3jyD z@OG~;eK$5au#tEsCC&QJ7iBYaSoT|?gP@*NzV#V2FBgyK^58d9ic@#cuoG~CqUL5t zED4iyQrI`)ZMLM3kkL@aWofMGWisbyi`9WC4xQ{wBqY|}$*s}t5sL5j`ZD%9_P>|5 zESgrhkA+9t)<%b^F-)BB2FD^9ed&)WEV*9KCi{?@_9~Hffc`F(@zLrJFj!^c@>bP%N zS6E#z>vI!ay^QUy9zT^o+4rIq&DbBLs1?)FAVjW0?m{d%SZhbcnpy-VWJle@v_U8V zzp%V6iOfnfr4TxJp(*bo9xD`Xw8lA1!u-7aAb_kdc+%J!nFzb2j}|YF;t6y{Gj=Q9 zMvzb5CZo^$Vba@f-hgm!m$1G?(m}31$>&Ol$va6b+q2@G7hpu2o+EK=ZVF8^?5|)m zvCQ;Z3R#4+yraydC#XhXP1wH}iIm_z|75Nebr9gV4^2ohx~{2vXKN#2c#R`@8m4+! zU&$*P0~vSIdX99PMjr9RqtJ>|Kpt$=%v>2BSi_z<{Dxd4x3%CwtV5=Xib|rz_XO$0 zIX`0q(9`D^A~@x_G(FRk7s-fx-Yj&a`!s4QuvgaZzr*FO(J`jJSAg}ycuMEMeZSD^ zN`s-51QAA#hy3{s)67-4cD#N308Nh(4rG;&P0iI{LZt98TtfWXdqmMG9!)971xDj3 z9!|%1v~3{FFOG*1aE4Kp+c3~*7fD_k!L*q@BAEVUb-{byM8OC-?uMs!zqjiDW6yZPm1_+a3f{Damo-g^xT(NHKxOccF4L6Qw_+;-Nf52 zbaunkAPDvTBuVpi49jjNjT$37+)Fd|t(9^~`t{QzTAo*CC8?x!)r9)Ab*V`A5|MEJ zD`Wn&C^t&prSdH;EJ6j+t&W;gw=4fi@-v0wC*%}rj?+-cJAJ@*(Cy{+e*gdDUm2(x zF)g}d(9YkB32lEs(=<@!su8V5=1L_@w7HFp1%)l@D^o$D1Y|x>p|Dt9Rq{sf7PU5c;GzA=>U7!` zS>ryB4X`rFIa?@K^QT7_Y3r(ec(-)xqPw$pb`4FeY6F&g=F>=i{ zq^zG=7Znm%YbT9#!?V3eyM?GfF-A8)5gg2L@y`Um>z0CfV|~400vCm|E-Vr>AHz6O{ud5v1*mqOrWOR6s56@-NR!a%uOGWmYR z>o)?UO6U2j{8NVlO8H(TD#F}GYPVkVv-aD{k_R)__VR32aY<62c09=nmfYJe7T=4T z$}JtK%?4Y@6vlY`DX#(s0y0DPGl`r+ zbN-b`XQZa=Cmgc$8Ny2n@;=3qwVuM6-m4kw_Jk%xGkn?uRs$$*H_jCLb-2&_5d8IF> zIo4SaB-F+-VZ-5o$oFsT{>h3)CvNy{tc3Gt4g;Z97>NdzMZqXOWFn@UPwv=)cc=bX zX3p;NbTTu0D(E=s!&G9#5%fkxZ?1pvtRHi^-d(Z-y|4Kd1}eR6iJeuuPjNO0Llp~a zQGW*91WUT9B9(X{*4#YUEwq5(*Qd|!bVIuZJrE$`!`s}-{#8CZFp_GR2PuQzy zrhFWr#Jt1fLoiz!?9WFtcIzR^Cf0Fptmj(K`De65PakvqW1(KGn|cquNjc#~;DuYO zN<-{oV8M&gN!%5cAU;`lfI&a{Ol+)*aAcNU(aqg$VMP)91)6i2E{a6d&WnzU3;ZUv zu;BiV?c9PAVswCW_^NX)#ElMFY;Z-BIW6)OYn>U(;E{Q~i`JvLAI3pM5X-x{qt5X~ zA83`V z)>+NO=gG*T2woUfnqQN5QORd)*yR$1Brn~+z&>Y*U@s%KCZ9LH>dG^|To-xGBiV8P zsssf{=vZp$%i-#dCQ5KWuz{PZcOyl_K8$@@T=S_&)cKSNDk|q8^v5g%CGYzlaw@|a zi*ooI8IEnEgCe!!7Wfv{u^rk(>Gq^@5lo1kHg+?9cQn1_)TQx(MefNhG6j%X2QDI3?B;t2GR*VkE`JEybZF^KwQ-a7l?hG$uKmnh!tax4)> zZFpPW{ObV8k`4nr`6Ur$)q6${=@@>7@qif`OSv!E15N8zKGc8$zd1;j+LK}D2}6HR z!>NH{L0VMkFAcr3XPS7U=}fwpU2>N4{rfCze56c0M;SMEyH z@g4RI)78HHVmP4V65}gI;NPt`Uhk&2q)&C{m`hS*Wl}E-Q&6E z|Gu4s>?b1m8q1R7?ruU2@1b(4v5rpRo``ucqAaSRz;}er0XAvStbYj`w-9c91)@FB zlnW!hRA440$=FiIC3Z{=pRwWVY6f;Zd`KYyj#Yu_eoAg<{#%|K7$)Juj_7eGu93&` z9weJV1YUbOrsj7h4UdbNi%@2%l))`$EKW2j{F>!KsI=Xy&K$QvkS?j;=InV#=RYFW zL9bJ#7o$<1w-kC{HkLEGZQHWNFj_I)!8a$xd3Cyj65fPDJNpLkk}nk`l-E{IT5|89 zsZ^kR3LLq{kR44I{fQ#;wbZ;P2mxu)BZUoV7 zupIL=M@#sz8A8p{5KV5)akhS-nlDKY;!l@SLB{>}KRM|fpHmJntDlJvAJfowXAd=I z)nG;3=U-=ikxkT&#T4hiB{<|qK=K})h>&o zb)4tw^2|0uh(;7-c{WI3=nePJag?+~A)=|LULW=>1A6IQ*D*ik%0~wuEvhnGy_s9& zFWveqDG~fa@!ID`>ww%M()hzEo65_^I^Sutn(Vv4JWJ-Vt^iMEhF6iiNmXQ*xUwB@ zgS4pdwPM3ZcYA^;-@e|g8JY*D#5BBUB*6Erf|)5hf&Q7UyO>dGDBWTU*r~G z6c#J~V&QRlDlh&^`#)p+BG%!17#OT*P}KP(72e^MvJC7`+O^I=?te8t3_*KCV-a4? zzozR{y+r;c?nJkO%PI)SR-^XrY)?J$)i84UeySvn)`zz5R9P#8`tv8W5qDz=8l)+i z%J5ZxI-Khe*vf#SAGh5FMn9w2MVdm1X`8*Pv)!rb60xBI)UlcI7GlO&LmHk)&Z%=6 zC%*>xs!wUyG8^^(B*;Vlg^F&h(b_S%>@y&6WC2BgoH2)|+gS`B1k` zRPwM)$)|{Uf8)s9MVZyNeC}asCcl%I!_W?5YBlx2ylPze1e z{rC#NkR(c6pSk&Du+0MJ`m`4Y)wSQ?7NM8dS}{MN$DDDk@8Xs{JX=wHBupamx7REN zmin=072!mgAD~%3r;p9+=DD4$$5l(6bH@EpW@}N_7AjhdCLe4>9!7`#;P0QlaN!?d z^mNM3Ut&S$kiKoTBTdAz5%r`QQr_Z7CS2racwHqTVz->P!Cr@VyHO#K6kAoseIKDR zYe65lypR3dpxCB2;N|}D7oA8ZE1r*uN|0!-Gj)Je-d)=~=f;_`v9;|s<5z2*Yo=}p zS-oN#@j#aU{MAS>W#_;AOUIXDPpch@gfx5Pxnt#K3Q<%2mu9%tvpToN-K8TnR>;J0 zC+hAK%pNF|ajD46yjE+6nu@&p&0a%msvO+xR!RO)Jv+-PQbG}V`70q|?yE4e`mnpQ z>5s2yy=~Z7s<)o(=-hPb#M=Hbf`XZY-^_?=rpz}mq@u}7w9|gpQOIR(M*vZ693hfr zMroNkO|2H6>p5=RVBK~gZpshj&hg(I{=DKPC)C1pwXn_@aYj3BynCR*SPOuGBs^kl~x=}v@a+mLqt|~lh)eEoo!;%vi*J* z8C_g9JxFckHweNaY#Q*R_(DjyR&^x$g>?NW%X(w8+J~V^JiFYxN(S1$mgjs7qE4Ji zl|GUoH4el$U){+Z2*K3K8^uWK@vX)RzX{*TlzfA{!f1}!P?*2R$Z3i;b*&7;!k{7# zypeeE*{QDD8$aqqg=zAxVl%En58%ma!&Ckd!5Y<3sX4|g7DqRMyX|q0Sd?a3U;P>J zd({xD-Jm-az4Xt4Z?spL2CZ4xiQ}X-B1J)sSy+n>dm4^r^hA8ro9nK;MG8jNr6P;Fos-6!9Zke#4my(yO-wvZ7epz`I{Y7X0)_^~5MUUV{ebs*qlLF*UI zIj+fnGw@aV?To!dmdJ6Mm90mMLp%r)g^ZOS)_{-N9eEfE09g+P;iBAU(_}5`)TUq}mufJrG28(62shZSGFvkBeLr zYF57$qA5?Z4G^!L7va5b1fHC-Jj;*xE?Vb_B zZxZ{frBvlKG_X+Xt;Nn|FEdU?LV*oanf-JSx1a3PGDwA|-aM%z=ggm{zZvaU+~P&> z?du7rNI&n+f;xE)9fgpTz1HeUaE63K%3(y=SW{DH{kJ9eYPf^v)iHM7#F)N%{g|ZC zV9F{FQ=q{(x2o2x-n>-b{M%p(XgdZ7jxlan)E~4=0!*06a5LEf1S)hxS*kXMi zm#fm8w#Vw*n+R~2grKY{a6(7}9=9QI&Z~oZ;$b#t>4t7cou@2vk~m|vte&IR^dIm{ zk|t{yR4C8ZHmu4l{Ou$VaHzQoU!lIcsysB|N^IhN{}`byg!`W2^k_H%u|R~BM19=q z9sAhaLZ=>LQ)tR@fH$Pu`0U6MchMOzxF* ztfbKy0+iX=&(wYH$xe*9`23aZvyyFLZOkTt!`D)!Y>Hs$mrU@fu z?417dQEyMZkH?hT+sIr_HLgGzMMex0Mh@KK#C(AVMI zp>Oj++zZS2A!;3`pA2UxhSp(8n}euF!n!xx6}wpyI%qx;QmWO5&i9Ghv8Nu+CxH?A za-F_DE&QzAIOe7hm#r)tT9115RWSTd^vU!l@W@ltI)M zwj+E-Lfb)U_#NR*d>(V963-?DJc+-@DC2F~*ed(WH>-(uij?u`Gv4Cz z4=VRFfZH%bzHBb@4Oct*FO7as9R;|Q-y$_Jel~438J><1S6052d>9gSm|_f=`Tk`> zy!8@p`}$zi*YCZsD1l6(Km~5oN4ut$PdXP?ym_BKdEFt5!)eqf%MZCcvV!ghxZa@} zUv&2Ht6&$zoFNN{>%!Y-8iOimP70SA6)Z=O^e#c>hgTHfmj2b!!gtcPkr|Px>RW(xRBJ^{&dq^5G5y`Z8 zW0KPganxJpvDHEKl4dG3jzDDUiCOJjup$|B6V(z)(*OEC%z%AX7$P6J$`nkVmrm4{ z7)kdL4`)WAQd--w)e`O%;zg$AaPEf0^+Uy;Fir*8lxsQCpG+KPcCuP2Xx}G}re?C3mL45j z>M8R}7SyR;0Kd$3G-s!)&2~go8*7~mu7#WLs(tA{M&pliEkAjQ>6xFCRb09CHz(Pi zyGu%wLcrDiN=k@KdDJQ{j&3>pIR?S`qPr~QRRyUq$j1LBA1Zwbf5dv)&74SEj=i*_Y@NkQnZvdb#9Ww2I&A#+mz3>&OR4 ztRp$wrQx^g3|yK`G5S_aFf4z;Vb*?}+rsEC+%qq|R$b<@d}EI7~vJX`5qUFZGqIm%JFJS3^01K7|f2G=kSHZ$JxeT1EijU|5EfjFOEMuZ#j zpXOr)(F?gk;9qMND_BBKqT1bl0l;oKei(=5ptZn5Gsv|uNaf|m4OA&P8)P4H>GxSY z6_cO&JZG)2^>`vT7uZ;McB04BMC~{X!)pkaRRt;C_a-_HL=-xNj|;iaL5(E*JhPEL zZnh37TCtEg%oqV-(j9Ybf;+=BETfLYO&@mm3&Yly?wt}WpNZ?wPNR#OoK*adJtdv| zVcLFpzkoDH6REaN^(>Y7KfqTr= zL(>z^)q*2B_wnNh+A-@4uAmi#mOEwTVaXWDP3)lh)z+8yC(?NeX>O|hcsqCqSe8?G zAJ89oP=RK}@2qu2=J-Hnh8W$HV_f`S>jw!FEhRFa(d$t5t6ufQs)Z-Hw1TaH~^DMSrL*%leVk> z0TyZ`5*yoduJi%KE5SRe>r0TK9Aj%^?QaEn?e!7HdWKbEEZv>&@@*0)RWoU0R>^kq z=PP5s@@SjjD%FZcE$5={F?>NZ;>?`;U{UR_cQavDk6|qm&GuNO#I(Sk=#@m|BUSQP zIn}Ffvx~I8fn^G~T{>p!cd}Na_&Hyk#L`E1*wySiA9XG^+UbR9T_IEPnTO-VToqi* zCJ%P$E2XWdMU382RePpclGmhL-a6f z_fY(-cO(TP{68O3Q~jV!{Dl&**?f`)JNQNKg8JQsB-yfWJG>CB6nX-lX(Xr3`*{=g ztMJ35j6vFGK}qhjj{5Yb%RsnulIZ}~c>TRx-HQ*OjdhAjr)zWf{Vn#<)ZFcnAGqc* z0>7G4r?OiPzUz8h#bMArnv2e3-Ns-X&OvoiMU$6Rb69Pa)fN%Dp)psCGwQ2yi5*G} z(+l0#*A^YJkpOI?J%VHzSG)_eMdhH~knHdJFj1ONYMe#fjvsE-Rx0g6ghE;bxTOAI zDIlKSH)=~D(O3z347DhC4ib*R@rKgdo`E8tXuS4SS~yzQj0YENF8G={RWx8)9nBvoWUnAt_F(!)XbHCmO z?@O?$%A38!hCwJI`H8#X%I-3A?shhJ)ARIY0|Fx9qEDsZ%2BC@R-pkO_WDhG3yFks zSoxhtB5dVb<{;|vGM6AZniMMMAm zMxFsRuWgl*&HY~7+R^C|6+7_9zJDX4am0s%-YMbzrBdZ}$7+$<25GOb#5JQRHk38d z)onw?r|TjP*!1keT$k2-`9HC% zb9ioY)1F#>?g)nkFAwx&o8qwA-MX9WNI_JfV$+gZt&d~-jwggLNk_}+Q>Ed3PzW&l zwYKfgR25jrCh0O&>b!+(Z(%Z)@Lz*5vu!n!#C}dCAt}`el<5=KsU;J@uKVnYb%{D(avX!}5FtgKJ!H-O=8yz6#$O0;R(<2GKuCsGhm7ssaC9qQh{otM9iAX1pbogFhJjbA{*P-1h%_ga`UisX`ULG!92P9GB6|d6IJWikwR}Z961lZht5g&L4H}fIGG)qP6t!kD zJ|KY^ddG!x+xM! zIdd8t;Nf3t|EvJLB>wz(ui}`Qui2uVQ`QJQHo1Y`(v7DFr~&2lsd73G%=hFB%1T;a za=Y8DyFP<>2jBJL-e^nkz)8(dkIt9&2hFf$CP3tHhVLK5}I?=-116ykCCCi+k z7o;zoxMC*=1+R-liF2jvAbVKK%d0U8Our{$u)Pyk*heWf_+C`k!B&S_LL`&J;x+J18Fu}(N z*q4bN3Xz_;c%tsV?1wPjK=IKz-*CwmQWN&N)Y{`a$3xP>8&-~_wX-ie64uGd0<@aQxtkyP|(3(pkL+10tD!WucCPU$mmI^W1WIh3dJ)n z`E*@lNhDgO#XQWg?hT$iSAsu{YXt^z;q6ub5~UHxd*0Hx_^&>A@9VGJhzChC_@qd0 zMT`uyH^IfzdXy)f&`e=tLy81wAxn3We-)G5q+H53-VqI1yuxv!3j0{-Rk#INEfK8= zY;))993dwfK8DdJSL8tuNGsMjCpDVt3kI+6IC*{=lPyf%PNS27r+{r*eazEehjGa> z1uS~!_W*vk28hW+DYjk|^u5uaWv2CRz;#8d1+y1aGwk1QkK7Q5&2`Qy-(rDt5waHm&~A$92zRfsTw3-k0j&(P=p6zv=bg1BwbI-1& z8}~=&ROk0m#HJd44KS|zj!?B@+HRgv6Nnb-6eJ0T3Dj`ScZaP!#AHpc&t0ve>FFRs|`Q zQ)Y^i?svHDJ_nkED)j`hyzjP4rdNBfX_EzLXN3i;=&)$28uOqc(%Tfh$?O`w?VdSb zA01iF^KexoX=sV3u$cNr-mV27(^reYrTR4^Nm)yUnlNqLxl_TM9+H6EjEi$mILb z3i~ptyuQr(smB*9xsHwXqaG-QpfV3>IwSBf146pvLO%nQ^T6BtUcQNa?#s8Jp6=@IH$N+`U28RpEZ42J`#>p*h(2sb ztDY}8zO%XG!8GXO1uc2wT@4sc@KurBt1WT=P&*%2jo1&bKoSUhLvO=LJ3>g31=JSo z(H={f#IMh+&XQB=uN&zTv)o|l1cI@En%u5dYuvneySP0M@NAyCuK^AzDuav~h56b0fD z?#^RGHNfg zz8*0~=M!h$Rz}o}{Vp&@P216r{mvl(yT07;fDc;9FpDxYKPsHUE~nUjQ7;ln5;)$F z9Q2&C$laiCSlhXqaw0=e$z5`#fIR=#sb88sbQUayDVg#@vE*?$7hBX{s5W5QM3of9 z7=N;&^k`GOM3BLNz%S=q>tWqA0iaqp;pL0`>|ISsTSZNwVP4mFwR5O=sF0(C2=2dG zA99bIS`8Pi5SP!cp)+OJh(KL~sAVs$jfrEUMW1gKQ}y&$^)fo-j1f|FsDtHe>TUt| zc^A@Nze!rzeK?N$L|?$$;iAQPYMK_sHYwvs@W7;s%%r`c1xID$J?{pS^fsQjn;Q9+ zuhG8as|;Tx1r->v(wCtgpXKw!>P9KT%WFRrz7e`VPCd;L!T&d3oQS z7gXBRgz(b-L*Bl~@8p&zAv>fjS$z-XIfO12suEQJGkrY+n)F?cnY-9@-9#=HO}HC5 z1Zp(g+t4N(&D&nb!`w*~DXri}|@)cbcI}-_j&SO#WbU4RzN0rke+0POW zuZTGw8*BT|Z=%(SE~rmUK`VLazNe^k>xa8UTS3JF|EMwSqK**|hAenFF%caqovaK% zsdLf=;A)$fe}Sj$A^jg`fz(aH(Kv%m9K($p@pu=7Aw5Jaa57uI@5%ump>{oq+z^LC z0SZ_$ib+_PI3pMpAX!ZM2$OyCvf3tjlVMle^&)2h6gs%(Ho)QfwNRrM&MLXS(~nva zXT=MGmL-=7+nbfO!dRlykWqPQf)%!GSjpZD+K_RQg-O~|RclO*B|h8dV7gf!?IRM+ z*!$o6Ctx+CDCIHK98+h>x(m$qr@N1UanO5UMutZd$AEYafc5Ii=w)MWT>5o-rQpu1 zBuYiHfpkrT6^2NGcc{qucF)}G>$nvwkVDg9>DRQB6L7iCeu&-LK&_k((`)2ZRjR05 zMh$)%aEpBo`zv4dU!FyA#AdG>R2;r+G_qAZE<%HWjY`rT(VIz9T20`|zf(~iCjOYl zZWqLyYQ?zg$ii_Jg&rc3?aR=f52;d@GJ-@Cry&{ANKHktHvL`kOP0a)&_A>Beipc+ z2x=VFmXG-mbq-VJ8g#x415D)ewZ&}%8xISl&=7uZEy4cP?dbdk#HSSN{-_Dl?LFFg z!UL3NJ)Nl}!;S9HB?*g6uJC?{#=Kmg(;56&`zQ3MeRMWPmzM`@1j4>klv4iKn zfuZ&HH~TzWW8B>I6;{K)<>oCpVp8j*z+x>c4u&bLJuCS;2QppzBEHsAlcqX3dq(@R zHi|J@zO2a4a^GLI!NB!(^r?CO1o5-Mcx3y?xhr?ulEl352xqCmdb8u)tX8rM%+wQ* z`XYYE!l*a}`2NkwZ}xlh?0}%aLP2V%oDFyQ2uuk^qDisgA?jiJB_{;!GlqCifbg&d z&cqr-{(uHa_LQ~M*llt(k5k3rB`=g!LYyz3megHSiu{vZJAjEb4HTW_sURuRt+wya z1iiz<7LaPo1t!bDqa?jD&1F-(4Tz>KTv07#=2j7Lf7sCr1s7{9*$2XWebsZAszGbV zL(%GEUP)OJDe;|-jA)tKyVnu+L_ags?#mIbVb|(&78d5WFh3I-%^1?izlW7MC3~f#b`{n&Yiu7;LIC0F3-bltSG6a_kSk%o- zwUU6*qskTy?GUN2uuDuT<1a;#Dr$kOhWTB-8SK}E5H-$I%M1GS3)#h}Wh|8>YGe+U zCA_oCXhR7YziAqndJ3RKN+?wD*c*pCvW6~`6vyz4S5+m6S=lxu2$DTUeK1-GnV2&! zyL&iJV380tj2>`3pTkxwz~E zwJ3UZ`D8q79XoGfzMz)Ru{aOY7_3Fp=fGK!7Drm&Er`n_pW`<=0b5+6ntpRFa@z5U zWdlrIZyN=P2?8i1yeB2w0qU;vFd8K)@zHtMK?$wb_#HW-PE(8+$J6yedd^;lB8p*R zwBN(hp?(aJPF;94H`29y@8UYT&c{~u(du2NMcixJU*BAk09xN`+Z@9;7acxX5l2iw zddyws!Y%cNCpn*E?Eg>ZVf|ZFX5K(36B&H?^_B5eTB>H8D71IZfFAl>Ux2dyqE_!_ zh>XL^A=$W9xj~Zx^k^RE;$t=14J4}ad>*LKK_(tA(MVUCkB|UJ`D@g#E~=(9qckwI zWWza1z`~8mEZRTp{jzysBWJ$uQVTI@B_a#@-MEm8YExFu?di32QF18bna;!Yp{tCsOuC>rtP} zF6)0od5H^XM>h*w$?8t0bg)@nhr;##+0m!K1F#&qB3B!ZUVYj*6L4Kk=va(WyLx03 zUo~d^(G&w=da~*XWyMc8qu3kRNalL(Y4688(Iqp}_P?O}6SOOzR3(9{hT+#eArW<} zCd?8JHj%OtU$SmiVEIR#A0RRXu&F>Q59)VG^v6X)`m$?9$VLR_r(gwpijYnBDja%5 zQo){x&k@q+`@liH-+Lmtc+k>H1N!!IxyZA14H?FMP&iJUFqrI5U&>}Lvn?irVsP}5 zFUbU-nTnWNnLE~n>xM%$2AziwRZU1~y^sXU?BEf=;7}B@D7K%xu;twyww>0u{w^Hd z`L+&fOVKg%$~PJzQ{pX5bJk1^XskqjsB;jh+a#n&wKqO1u4OM?#*d?7(uh#3w}a#& ze*X9NGNM2$AmPVx5qVQ}M~i5NS#a9PdkJIdZ3MRI)Xout_~<6QaA2>evnA-CyyaeQ z)zsSO*Fn3zQ)^|QL_ju1OsjN>$|N}$#PNSG+b_Y*n{iFOMudfgytmMuzY8~PQe`yQ z|4DsRf=P3Qs{ue%_d0vqgDS;_SpyIj1 zTfqd_wl&IRKdB~CRV4GyEPTE?0Rsp>mtDA5)w(*6;WGZub~){P{#zKz_J|#2_{$Q1 z`7@D_%>+n*&`O7Gh9wxmE!Fr*COxROvcrx?K_MfR7MiPxk}4W1&|w5;;6SE1ml^&a zC1az?U3}-~T90-UfKX(*BK~9%&+^|;dvh|iBcgv(lRRr;*s68%*3sJTe0GcQIY#j| zX3j)*gbRr{$LiPQn>~wx$)Zz;8&qNhGO}i*jTVY~6`40Tz7A;XvZ&tDPh$Su={gsI zniJfiiNZoxw~*jbFKbiKuSQ>U7UgK;1?s_L2@f^H=#HaD>irz+$&`C!j)9NS^8>TLN@$T- z#OPhXVM)&hxHx6f#wS`SwERR+D7zOo*{PD7Jr7&B6QLjt>kR9wj3D?gNlG%r4bWr$ zv}pW#qTRkOaE&L}kz}ze*txPiYP~&yUUO;d%RC0O1c9DNzX0)|R(m@0qhUsU4@=!J z+t3ze$ZBf*DtCb6^0$(}yeONGqD>;GERd!(W=Ah#hALhIPwPeLLS4S~gNS%}t&_uI zqkWetbfxpa74j|S$s|2N2Lm)VkO^0FI~nEJ%*z{d8Zb@5b?EQbxIXo?b%Y%;?CRdL zbngZFp}~DW+}o(51vr8y>oF8gD#x`3Kr$7hK7yZ`ki20Rwq)uX<~Mqmj2$7(NGmw| zBK!-)rl&9r#SD5Sn)5HbQB@q#ABfz;xJB`;957bOh2-purp`53KNoT9e92iGmhjzW z1aSs-Pi-vU_DqpdTphqDNBa|7pn&pAB1VdA|qCZXQ zq9{8f{@J|6OZ6x@5zLensY~acD<*0AF|&*tZQvcwAwW51AMfNDFVP2MwsrWD_-KMW z#a&2Qz>!E&pSoA0mAUc%^W}7?v&FmYHb9$vX%ZzhpRSmv=9xBYsxGpGcX}k<5|Ae0 zV5UVqJ87pW;AGpW@I{Z~T%Z_{-VM#)COF-Yx_W2Jvp>OCQH?Y-Y#KwQ8kfa)BAP2n ziy}oY21Kakl!!*1^GePh1#IDcqYD1FBL2m2;C*#q;oFtPpFcZANfuy2)$>}D zTD|6r>?4{f^bZfuF(p(&zXYvc&wWO>#KeFp^@LDnJ`D+jZFS0jirwyL-Gc>?oT#DZ zvA9rC^bw>?NOkv&vpd|kmbJ`b$?GPn^tniiT6(;2WK%rj=1f-1n>~^q=d*9qGdk9N zDWC0fsd&RVhx2^m82ka1TKMyID;J8yle1%q+2spu{(FCw`X&ycJxS(SoUr3!OYo z6^Q_r3aEDZ-9?xBYqoO*CX!x<&f3HB_MUV25@3)8T%Pa2zTTdh2KJ_+eM1{rSTRci z7Z!L+0p&QR4A4+}I(c6MXm*P3|Ho)wIky!HTmYJF@50vaQ?5v@era3pQ^P=HGzhZI z!;pdFX@tg8T1HT{Em_zZ9{Gk&;{q3Obu&sgdra{i zvS@u1%48FORMnBO$B9lV4RT2AURt7y$EL~lHIl~LiB}|H3nGL(Vvf8|`P4{Flhxwr zckOJ|7s&1PO?HzZFky~wVMM`O(p^y~JrzBNh20v~GNTBGDyI=g)ld9hn;kE3sC~Hb zsb~q4lVj~_;i9C^j$Ny8#8JCx$52!Nuz0bOA8q29 z6VkTvu1{^aUL=~pGUya~gi3wHZTxBLdt>GjWkt1S%0G(8IiOzs22hZROUNJ%?~|}s z$M$N36GD@$nkNhC^t2rXRc6||#A*f`^k{Q}40ZPFiWI6O(Gb9!nOIqxXt9=jsf{X# z86f9F)(uZ__Rhj8STqKz3~K*GyHJzgHAx`34XImLnaTb@ok<W#I~MO;GtQf4s0LTz0so%c9K#~%43mg zD#U8f;fr*TJ>3NmckJ+48gtE8C#R=)TJIpCH%S+0{_YK7Z>rjPf z$MZ_g)ml?t3oI$-Jmqu^>~~y89V}MJB*1O(s2W{#*&vSRg7+H7;uQQwlW)Jmmq%6E zcmJ*^+#vj<9Y8<}wPU2&IPx!cmV5J}wkMS?J zWZGT)QX0pTy(s*pxMU=B+~3020eIXY=JUMF<0LaP@GG`%wz8=m~HF4~$Hml_O`j(mD)|#W z&Qs@=y|*CHna9ft1|4J3bg`nL#s=k>9h8VyQM6v*i73m9^pO#eNc6@RSD)omWE<8k+niAs zB7Q{p%u5L^no@Q*yvF>>quRqG%jLbl_h`vx8UlU?i&s_`5Tp-Sqn3T*Z<7~^X*r!S zy=+9pK*Om;I1Ida>I%Mbm125sMPQj37`Vaczg@Z%AT zwb(ym+Kg*P%Y6{oM>1TlcvtP{+f|C!>ZtMun1hy@gS|d z10!L2{xA^BsG6s?jOW8#_ZIB+K8{hi$3bJ35VM+O;na5hU79!_2wVW5&)^d9?J;||r=(o> zTQ7VT^*JtH1rM&y$Zx($WJ!JUj66mSR09a4v2?miipX4kfqlBV4tVovROyl;h6a7m z0Ujvpk9c%i0^UZsl;w7d%haX7ms)*phEN#Ksiu#`)>6}hXoWZDH!AO<=X`vXJ5_j= zB1dsZJNz1=e!x}1&pij%#Rz;V+a?SNz1zRm=efq830B)}Vra(6Ryiq`M?yjMlNa<# zVC{3-w<<3h!P+7js7&9zjGaS}D6pD^+qP}nHgCIc+qP}nwr$(CZQC~I)>OTk&0EYa zsVtIBQkDOl^M7bp6)hVFYjH02B*EsOa>CU+w@{xjbN~v-e{u@;%4(ErvLhELIlUaK z!Z01L^m*^4sL(<1@OZEm^wN zk&|BZw0aDqGb}Y=AEIRZ6*Y@SQU}i~CT-Bvi6zMd*hT<7#cGpt-Fo)_^m+hxr#{N9 z^a!cjy$xz}OOU{%KGOi@{1Pzh&rZ7dbVM6xJdcJZm4w>NHxCV)izwi^2xn?fxH1*w zvggHEeiPE1x*Sbp3r~aHUNGCtUAL-NXbu$x<%vbtwuV(ej# zli;L+*j@2bs$*-wznWjSQq~c+I2I$=R&CLGBm&Wr*C6= zQ{QEXC0S$F^o8C0NNHTOn^@k;%*Ci~C(stpzi|*<@+hqWs|>GFC>+<4Ofd|dNuiKy zNEEng|E}0$8Of*Qe5_5Qsw&RmXkUFju>7%E#a1kI>1cl=qXqPUmlZV%i&+FY$&!8< z8pwA!fSLzlrb&jY(ukvGkkSVGQM!kn$EWG7cZ2R~k#Cn@9%>d)>1Iju`3uk0)=pf* z@XBunShgfGVSYmgCygmuMncy^>@_K)?KB?U_ZC^dbQ2N%;I%!=z>o`X{XW(L>va=nCk3 zvGN_e#)RbuoA%78@6KdsW&dwgaU0>N)KPv;2vO!&_m%xIT$2nz=UG4&H{@KAae>YFS(maK1;bu>ThhZIgivCM31}6GCRSBamqC1r_ z;H;)Z9#-`G-`#)`D}Fn7Ge z?uR%c`0A+s!Fpr0oVn?~)6=le+1%(t%qxu&)lz|7ro|y^P8sSYl)M4b-6G3{dqaw3 z8hupOq?xATv@7eXfSk(V?OoF6p4+6dJN9bjxj8Auc4#Kt?h;YK-1RbXASJ0MjLb-SBAg>?8ht1)0jg{SILuZ8r-&Qc{c>iCodx!J*s2T z^MK#z9W?oP7r38XWW3`4yGT=xRA>Enw61I$26Q0syg4z+2r(pTsZJRjY`Q{ui^M>8 z#5Nn`Z9tVuknEICh0#v5|K^O#Hw}{_`%HL{KzD3%`8bLir`eD%+>TI_VKtyfkw%<$ zuPVG3?TCWNEwP4Vp6{6Yv1Pd@&pJw|?DAiPCdxaZ+dR}`?GVDNV1jDq(bWVu9(48W zT69QyeDw@o)%{@_uIjTR!1`EDM3Yr21QN}CAclL*gEs{ zCIvjGk%=x8CEC61^@6=0O^GrNPZRXG&m-H9h}VT>$9nBc5-#VX;xKKO5!?g+vBQ;; z2ld^NV?s~fQJzx01nBhJ!@+$xA`e-LgZQs<)>L;%`-$FjGH^jIYF&o@!0bqn3 z+a^6U38jReg*eX&^g!g4aFOhTC9e%n*zEi<58j_!D2)?mR6t2u16vq9`KTy8bVA#! zDvgS-S9va;DTj(n-mbv2@3G2J41hXglI1N2X!pYw2V=Fr`rBeym}TlAUl;O^Q>bLN zHAZKk8=5O*Y8`)!vx0q%zFsJ|aL`p5dLe^%2BK8`7s9XgLW-E(u08Sw->Wz05k$9i zbI=sOXdzm6>G48Gec(x%%5TMNzrpsI;t0HF2A^r|AkO-uVj|KTRusVo0P~0A$=WI|(+(UJx*s^M2mLQbj&ol^m$}Nxc`T0UbmqIHcwEz9_BO z4{b?VCUmP;{&*?qz-Cv5WC>iN6yko#`AIStkkcgQ9FYXa7zyw}Blm3y>Avtw@>DOYUJ1l?(EK_YT`>(bgZby^gpu=8_N&DjeQ%%JU`mH-h-B&tjD@TPZ7So-oVZA+(ZMabfCU-e1ZlV3Zd+ZK&=+SUc0&fIdsmiILvvlIUu}_~!t_H~<{A2`5N9S`Rub@)1^|y<$oScC1+_A1#*hy$sGMx{2G9syiy`wlF@J5Lr&!D*OfFL5ZNc<@uF^)S zePDCa;RTknQK{>nHP3@`S&MMQ%@QAcc+-%ZHYy7ZN7(ttE8*;exM4*uy%P1YT7Xa) zpj)^>%FwrsHL$Z^Z8q_*ul$_IxE8$-z7WW!bYqk4p>{VTO5YEnlhn^urhYWE_mJxS zGuUJO?^Q6jB7OE<(~$3aFDS2OUtb~gCuG{^15S%bh)B+3l_~#tV8+z0l4xcf>tRa; zCKV7Kwr7dJsVECf5DU_-g>*(KHpINJ89U`Wt6^PN>Me0Y*5}VObTN6>4!ow(jl@G! zjFlLn+dZGZY9)_B&-{(e&%@Vb}9?pJMG4~T!%C-UU zA%2{~y&CN^no<16m8!k7)`T{MEXl4eI(*1K2Dw$fgF5u%d<}4F0Etc6QKLa(iW-L? zBukwVxG%=Qd=N*BDePtQekc*+0XpfL zrWHv-IT&xRJ?zGzSReiye3;~!GY*ptsgDR|)2RK85*+!rOldp4?6vRp zP(TzUDofCWSt^Bz1q(f2>|gwk2Wt^ofJC=~F=iMcnUI{Dxv4KC+A4~ysaC!F0l>yM zs8WR>USROwf`s`Fl0D{7ifqNUPFaIO5ZPqeLzrO0rFUez2Bix;OwIb%%e=JIwj2|m zBK~c&F~XRNPvh98ur%WC$EPT~F|XvqvZ4}?_nRSCzeEgPpdpTr8e;BbJ4k;v#=a4!T}=G%8*)220FDpBMgq>L~qJ2tfrk>B!#ZS3WEBkD-L zhNvXYmx#LiN01zRU@bYnOU8f33CT~Vdrqac004f0zY^0Vse*HcBbB)sOyFC!&dKC& z%r9MG`eeW1q#!E;C4)l$O!)^?GmK9tk6p40?1^o)P@D`GhZshY^H0C) zlCb%1pgVR+{!OtIJRbZI@x7&bJ|Hk*K{jEU#DJU2t&T}DjR!>tlAXMXaUlr7%m>aX zjP4$<9gV3AjM_K_SPe_jo|@o_(D0#h695&x@ZQ_xL2mPLaJeEl+OcCKZSWlP9hA@C zCgxU@D>YQy{>TbN(s_McvIH#&o{JsebP=&MP1W>{o=)`oFQ$iAHg1P&Ch_;FiRwM0J&f9+H7zgQfmRb;pTK;qde z{!H0XK%l@3C9Pp}VbM zFG44}kN+%5D>caQ=@A^2W=Q@QVt;V|qwPL@=jiR-1C3f^-Pu>>X=~b9^Gf)$O!;>* zuja?a2}hGB?EROy+?XYY%IeVOQb$3o%LfQ>t{B|5jwIx{zvMBDm9G(94DvqAPRw?! zrH7aMPZw0kxnwQquGC!Dc^2fYV*68Dv^XwuJ@F2v}j-$L38lmA17{)T3zc%8;4w zb^V4EZ|Q*)xrMzS-iy;elvMC0_R!O1*6Ml2Rcvt5{_CrpyJ7P2#RJ7By$ozY7s;|I zRX(+*PuITVZp!({I&A`3oY7XcWB&Hy$Fty0w(HT&hS1XCPk;*R>=Cu47_&uQt&ilSIx9Vyp@qGFdwdc4<%Z$z)Z~z&Mu4~dhvtr#VZS13Sj@ad- zS*6e%e~duf{OpFOM`Wh2bVTtLceXNji~1LP1j0ea$$!LFN9DBH{~HXF?Z3ewS((`W zgM4HnVC7``uPjJ*CN_@$MM3^QSdh&r?mSAj*stRoBvKP8PFGSU7?WxmEG`Yk?o;|b03E)hnmD?|+`q?Sf<>k=tna2Npo!ax}a z))0AtuH=}Eeh7HrYk60QYAk-@z#$ZhePaTmy8!ZwtE>ym2qAGm*lk>#`;fn8f0-Cq z6p%ra_PIi+04CH>y3cIxnxx$N*v)?*R|KGLqO37>K|d#;=M$6i9734$Qzw2_X4_^HdH6S03c*L{jbo zh+8CwDLjFUD0heQxyyUwH`saD^@|TZhk=9SEe~B(nV5zMcX$=CApbKOZ3Oh)vYsF2 z$KmGyCXd(hVgL^0^D^s?>G7{K(8%g06KM!RKWMckUnzTzc&H*!}>jbcC~*rECRUX>5Br%If&|f8KWYpMu3Bt z1`({UeTzR*wFAQlronMcLq+2!o)(Rq7#!3TY!he5r}`nm{qhy+dNInat#R!r(9s1h zQn4eF(t48;E9~>I5f*zY3aV?oZhTu8XP>D!Gp9vtNGP;1jk;ez%0kMVp zHEreYvB$@Kp?_b6@fS$okcAHR5B(B_#!b;5`4RmT-moVo{RDm!%0JjXWsd(Gi+#!b zY(*ut{KoMgB4+zQCw;@ezsA5*27N>R#!)=$Ud4^{;=+f2Sq=_T{X)6_DG36hEZ_|| zixDjF!OkyI10QY01@sp+(8{WVnB6@lLS29q`H_W3 zp>hvPsJN>X@|A7a>0i7#t*8C*Z2EPFeRH=KAlbu}C&M5)C;dnGTWCXOz*`-XFr`Z} zBLDvWoP3Cdee>m~gL9{*I^gZk_^X4@ z0}F^U0c;G7_2t*fi|>5XMdiae_NAC2Rgy8E6gYlcA=gOBpRp%MX|z)X z<3jwxEZ|8%(3>Le zCd*8_O(S0M*&lg^uI2Z46P>jEqN5rje8#(E*$pr93!ON&e5=tTAC(+(4H*wO3Wv26 zd}lch`L$Jc2U@%<06yYLt#nu4e6h&IqRgBf6J}LA@XNcu?f$`?Z_9Xsz}{ z@_k6HyOGQmL}%whpoZg=C{LB^1>H`1!mW+s+0oq|WZpvD zY&lyrcoA>tqGhz4IySmIe%F?I|6SMgf-sDFj(EU~l;45YO^RN^03SvV=?x>xZfEpo zJ1GU3B8sHYhp6Lwe7j4j!?W#DzIXYXI;=O45zN0P*Tlas%oa;UBi?29^Z$x7!9>>*(1{ZjQI zw0J4f!X6tF)~AS8(_E$9YPCg$&UasruGH&lcy@RAa(k_hgh5;u&wM>AcdYa@vDe=H zJrjwuuxx<|bKdvNRmAz>U84iM>t^^|-QwPl<9>_{*P~HCbtOGR{)lCF0PQZEvGDXR zA-w1<2aER=MaE~GI-4Dw8jBFr-YIExh$pJW5k2D@bdiUy9oNGAR75THXPswOx=@}Z zjlNPVh@coT(-OBMGKHC7(sDu+;R>y9z(eBPx1qNM8~#!&Rs)xEC~rhM@JiP^;B&s|pu`BN$e`<~zx zC+o!?Wi}nhhwpR@C+9{##n9!I(6QC8a{kazHkMT0O%#S&xown|zuWP>q#Q^r^Mo2X zpVjOH988}djR`vsCF9HZz#~N0_}Lo~?&vcikfTW}b#UvF(5-JtW1pZ(+vE*LTiMDd zLSGD-zg@nj8C(zVo5a#CZc2Bmjpz3j#Cl;^>#fi7p`Du+#YqX~hJyaygHLD&i|Km? zcYSSLcW5#>&)(_S!+}&-?N7WT+7D9(i=cRX+ga@4Z);1oP&{Oy9Xy&Z8($2~O1eh| z_L(5r(hInPLX2O)YZb%8?w2BRvZwzB8@xjQQtXJJgdkM~I|h~$jh1X`^F zc|O-@tskKAI=1rEnKG&Pse0IteQJQbz90of_Wy$Ba#I`s_--MF1jcN;Gb~En<@Ic+ z?9X5Go6$}!aEhy3e>-a2SN));6t$li&cC2gJ0r(%1T=cJ>?=@m5F%WsKUIv0U8$cL zQ3{e7p0C;jz2i=p!n14_P4OKgx8FTcXrJ$&&D954g3tY zuBNnALlrO>d;cw{4qw`RK&S}el{X(UAP*^eXT{N z;C#0%utf;`VNmI85}rN3^Pe|0S9ETQ(cUV? zz&ph49#DR)(;TvK?()<0LSH$mL=i#h9mQ_m%ig$3Q(@Nt4L9gmarul09Y=?s)HBhy z{DG;)&F{X6JPdA}#Jrm6x1~uhbQ&$}x$c!dYZ3 z1~NA=Q1@*e+zXiESitZ(PTF^4LKJz{5%Aaa?1j0EXsG70)jxSm6G*O^H#o$4`eRclK5d*tD8 z7y|gIu2Y3dULY-^|3)cVSS4WX>f>&_J^HWf@(-hzs|}uOS2l3RERqqm6Hvo$`=CTv zi`(`xk07m|vR58-O$Q2*A1hnz@f%*+rH16lGKsRUJ0VTcHB0Y9Is39xVCB6PKbRb9 z#=Eftd^U+{fl;2a^pQs|)7wt;Rm>{ero{DtW{I$54H1l270UkV8QnZwaVS*GW$0bn z>?#8J_{|H{{l1#Ay$3^-(l>{C(0epV-n{}f^L97~%X^Ixrz#r$G1=`=6XTBP+*X|< z@7+LLLLU{q^KV2~lBA<~!8Y+PEwRcAW65{V1;z}NuX4fpLds0b%VLX{`k4GnGuD=i zgA#yrJ++!8>9BR>)F$KJYiu* zXjg^wRcrH98nkVoY_A0G9Kh;(|Bj_L@vCz&gGBL-kk?JU95l^b@o8c&^_nHde`AiRdLpEyo_+^~xV&bXXGmY3B_v}rc&I<@b?_D#cbCl#B z&h|`(?463RLASo)_Dv0M9~vbI8E6)^_ba6nBa+vmn!ybw3?ez$q}L18fF9mjxEGFn zG)1wU@pd#)R8RIZ?9c_2e-Ps~FM?~=8x;bWn@cIc7u|De`hyz+{7yRG(&&`Q-({#& z?9=N5q3O@~W#bQhjleB-hK9Y)?9yY3%|KRv{`2RoB6+i3GTV%l@gfm_VnQ{4PK z9l4o+ZqRHIuG-WXwhpG@9Xl!%wG4j{laOYSb`rbfb&oR7I;p?1?-PxtP78B0+<_}R zY>Gf$Tj!s%52&lH+37Pp|Al2fDrRB^w=&L0yjQ%vf0qtNAO@^L0t7 zWt~b!I6QVq6mbVtHlV#@3-Li{7;#jpS#Og8tI4ME2UaBq^-Y0sr}@vAJ6f?(m07)% zv?{x^aaXsMOZ+9j_b#rJ@~{kY>*6{vr7gu69q3yBCmwk_8kD$BXsNG@@f3TZ z(Jk4LnwP#4&v_|as>>(4IDvf*aUNtMtkjM_Di;)Poq<;5X;NF@b+^u9XoPpP_5eU!e}iSnRM!S>RX7a~X*)I6<2k2v3 zsg=5(NuJ6&dpeYmeaaZI!h885PwAb>OI<$FeSIP#_5=A2O=+%P(pK~DCFJX97aP&Z zWqykrPl-;?ZgD1hyo78-#7L-$Zk9hHXv2@KS*eH{jpotp2b)ZTwyj&W973IWK#Qjo zj#k5_W1>{*^Z)7zq+vx*E{E+$qZ2(Sm71pD#mki5XQiO?Hao5kQvn{n#HCD^pmB~de3FxTL8y*98zAHp*JmRd&J^E+nl=sV*tVweR4}u#cm4Lu57@PEXu$aTy?1 zJLryfu3i0HKZ&?vM|?TNoX8ecEkW9LO+To!)=;rZ(`R#qLana9>PdnUW zZW>mBe1Q(iViUb*$?q3=`*hfgiezOjw|WSUz8Pcvp;*|IqF3Qs+b;xup9d&hx2c_= zj0~L=uN2A=CHG(n@~QXDFTcww-%6jcD_5&2VAXG9Pk9$XUEF#fvh;TQJ69?YJv)N} zb4~g=s{Gub9>-W*Dt4=tk7-@|`}T(m0y|AooF0wi=jq`9Z@PJsNqz8cuk3w4s!S&a zh#5Pj_EpvKmEUScqDG4#;F}N@l3VE8)%Lc9D#X7MDcgwIJmlOynJw90Ex8KHk?x|t zsq?ik_OT~qY}D|YZyandsx7%EAU65Ap|N1pq5+M4J7}tio)Xh(AZBtT&RCX1Y>T==xX@!lFxKg(MIIb&g$57cP45t1p95%7V zPO7%=1K*oR_2%N#6@JOf9Nxa%P2Cn4#o=AnNzRV9ToVA^t$Q>$G~~1)Qq_fdD;M+D zT%-%jRiAZI?MIJ;T7i8wj0)ox3JAWA7T)qkADMG{a#U)*4vH-^*Vqk02h zj-z7BYG!IWt4smM(uQ3n$-@vHws3a5!$iP%#RU$_#N`wGMROxCkKA9~!{920e3Vvg z@zzpgFdgWj2ELs){fgvvrePzHZ6=q#M$qyB4yy!(-D6G)g&?y_H;tbOvS^#(*Fe&^ z%5=Hf9Lu`hBs;oEh2h4CEtQo$1!uW6X8dlEJ}DAWT?!pmerSsxxIHqAbJQ_EIJHn4 zR4S*zMSO%C6-(2)zszX|{spn`C?Nlw@G+J;g+9jZ6JcP++B%re!XSu+YV(A!oFiFm zqh#miiT^r_Oc&$p;(1Q7S${YhJTA-Z>7KBJWJ+aJhz$cOJ2R(En{IWXJKgwTThpp& zwV~|QBr8gmfO%A0LC+y=yuGFH`di(CJbmZqHMhTZX3D4VJ7|~p)>c+~5eUo6B$IrI z@qojbt$inqug>}zv=9cD+LJvw8T+Q4wXIr>a=1V0h}HEnzP1~s<4T5#J3 zyAE+wKbNZ#)9UZ|5SMu$XP|W6)V&3MwJMpGL2oXAs*>Oty-!6dsCvj;$Y30wV5LLO zp+`2K_pW*}2(2rX5iibhTT=|jLknEV^j679X{f6lo;C;3v-dn#<@Yi4N6TWEs=mFA zS1c+qLrE1?Z4Wjr5^B^04%Vl}1nJFIWR7|s#-tLor@7Dl%Jue3CG*xgKGop@&!}t# z$||TrnipZ_!-UTy|2iiDtv2VfyN0es{3!jIlLMk*ea=`Zz^tu_Y~wc9k?lWGh8?Bw zR&ePy?!9lh+sq+XV+Jc5cVtPtiB5#g3)#`a?q2LLX*{ zdbn8{{95egD{L0N1&6&Z+Al**GlqFlDaQOcD?QN-x3dnHAxzn?CK=vCQnR?a)QxYF z6`N-#WR2B}fA_xXVp0$)o-H9|FMiTKwKW%%8Gn9?bAdA5?%Y8-fhd(Ka_jw>dicnJ z#oODkF!-iPiD)Z#46C+cA52H;vQsPvGm&KW?dxn0Q#P6^<20iXQm5306FSw* zLdiX|J{A*o+zd@ruz5(mGNv)ch)5xkC^SV9k4}%zM2z=POIEIM#C6jXs4= z>7hE;o8V~o_2hapkdv8N28qSr05EpXb_0O2=J??vR2|cb00lBPxel^&Myswr?)2?b zr6SLV&*Mg~NX*We?efLAQ zN_=oJ7|tw<@)4M2wJ8wdxzoQ3s_^)JJdpC{Xh+K2RILWDZC8y^s|nAA5YB;et!s^W zC$3OX0@In@Fw~hBsxH{K_6yyLnwE->vkms_I0Y#GEE8tXzM$-B**8+v*cdRmxlQbv zg}+|FmwD57J~w0Y&jSc#poUPT9Kj+ZM;`07Idm470c&3$ug8r^EnpX z3)}sv{M=YY2N*&*&I&H2-ii(s=;rB*ggy_!E{H)mV=9m}HJ%)NxZ@2eaHE?lW=CN_Ei+EvQvw%bQjGJ6R~8^__(^aCwjYVbN|@rxU5c_Wf<7sUZ)N{gzpb6*m&@3^STB{ch?7lujXT ziAHgB{ZGeDUma^*th*HpJt>31=}Cqxc{JepbWwSeLM;_EBfyS9P*zvFiKnQ zWHoh&c*7t`J4!}+q_m{m;he(Nx0Aa0Q(9SNY@DSVgaDHj9VdQyU@0Q$Sh)Z&nKY)y z;lv%Cw9|9Qy*M4{yzkjG?>Wj~m6CJmc9lYv?1;feZ`=K3op3glB)Dsk9LwIX3OT|{ zO3FKV1^J6akvl2_4c1Y8CoUArIN&*~pfvMPC_iB5cU=V_gpHt;I~oQ1oKv zR!+ta1oUE7`cB3o#)h^=#!!5GP>xOx#`@M!ZX3-l&T2{98+_8Pl41XL+}@6Ml4OTt zU=fC;rQIgj#$CW&*d`^FgkIQwj-K_|t@gz4eVx5cGd#(8ytvd_)wnDvkykcFV{c>! zl-LX;oVud10{T~ZBqbP9fCFl3W->N4Q~_3RTLa1lxT3+avNBs*N{}TW7@ENQLW2N= z1Xyz?NC9w;0|qfN1#$oqka?Rg17sp#7=x@}2TpSXND@HE3kd}d!xaGk0x#R(7EY7# zn~`UQu+j$wn-|t5xyI$ms-mGG>@~DwV*ZVJ+SCjHC4gsXu6K28sRsh+Hvw=*BSrIT z=*Ctu1I=p$2bLB{5zut>>*f>yE&`rYN>xwhN_X8pRZAt@%T$*-gup84nm2jK2Q zJvy>_=Dzk_76Yz^xhrKUt87o~-sXn{a7sZ43k0C9%*@1&431Q(J%%*6J?j7Pg=vFb+kw4$_BEfb?lU>*RqdY|_E$!> zIz9tR@ciWGlSkaZPX~nn(gXIx0VHr$lv>#>|4q-@HF()Ipdj7RfqQ`iaLlV_9^0J2 z0rAdr=LFUk1PDun9^1UM9rh(+anbQhPo@OndYX$4e35hFtZV1>`_(ltBmJ%X$hFTq z)bqa~_c$wIZ3N%+0GJXuF{_$*K{&IU`~Gb_{h>~6ima|?nu8O0{Uv+pt@d|OX~Fx2 z|NL!O1pBVZlwIE$hco-s<)St zgso?D8}#Ct?%oDibNA~1zr!Vw)-tO=VkG4Dfa^(4(4! zzT;QyYfi(@7h!4PXg3U)Yp{xJ&hjJ{kBZ^D@YF9OQ$yohvk%mI1I0t(%U;hk@Wt?D zG!Sn#0q6k4P4SQlZUa9I~YvgrBWC}mog(nmNiaBz29fU^F3@_-#=BR%cg8i;iM zhCXq3?l|_tb$hRuAPN4n_srko`dR?Zclc8#W)L6x%*N0efJ?B-uf;^=#P{u89L}Mk z;n$Lr!>99SnT|UNrnGlSUM!pD`gcKi-Tp@kp4GLb;VsgDj)HimMc}j-!#MOL`0E%L z77(!GYx&pK|9Dq%|@j)J7f7d7y67InIhcAmjFD8^!48{gkyd8eDRnn zdT;l9YfxJFOJ{z&vc3YB31wf{2&}1@&nnk-$>tKr#wWCMVLkBcgpcQDe}9<2VnLRC zYZ`jHNJ*}?yVE#10j6-ata`!Y;QJ=0PrmVUN`5`0-cHE^@M3-=eRbyn5XNwgB3f8% zZK>Yox!M`qmc*%iD`4U?a*y+}0YdzJ8+Ey4{JRUP{PN`r z_}l)3eh~X#FG^09MvOx@x*B8(;FIDb(NNycThXFT`#BYh_BOyterUg6@oXE^q(7~+ zsWADSoaK=Grp|0q#Dtq}HIhgMHmuyXhuH+{u4mq5Tj{pZA0V_Q(#SRs3gy_|8xr~R z9!SA7-aZCEiXX@0ba3Cd=n+GU;uD>x25>}h6pUFq_-0}%YF!~;s;u5Ai&ED9#JOE?aUepUd-5ms2x<%7eu9_9r%(X4pvzVIoW?bp@0Y$L&L&V{;!HKv_V;Kmtwou_) zCLJGp*}Z$ZcadPBvQlxcYD=Q#HkWnE)?fDyNSDCZ_!#i$KtqKMonG$=O~AK7;pQ7e zx3b)EmnMSXSyaH$70FFlW(;lmGepa+E_sXWGl5 z)ASNOEfK>5yj4HB+DOjt&tE40;!H)QoeGb43$s*KOi{)A>lXgoZcHRE*sA_YmV+A; zWm$M=!5~TPKA5NPpWeSl?@y3)w)c&_UoDq3zc?VAI~@r1&%fvE{|UoD9oW8G=U!Bd zC=@qR*iCN;XPuQ|Pu@lV^~l}UmsU+ss5_ln|G|@0hoplVBT9X#h{0fbwT;x{!g>Sz z*Y`*1U8g*qX|4yx1+I{mx1HTP8dlMrEuM?o;xib!`k`^8sU z(%>d(R-Ge3_n>>Ofz#ZTjLZ46zX8wV@s_1Tn5EERLBxbkviZ1s>pVRoz%Q)(ws8Y2 zDSxUwd_1B#gQS$VJCh(^J7wg)*@5uy;_x$R_^lLKa}+1JK2Ck7HrUhS9!wOibM z!z+v6sM)F%eklJD9BUP>q0jYWL#orBNgreT*H%uJqWIS0&`+5+VlbV?S)M~VqVOjc zad_5)1g)HDnHE6tF&(@$!wzs)A8f8IEF^&M6sz(3CW4n7n>KS#JELN&J)UC!km@2GV=$-``%4Y8#vhvf!UWhLc-!$FSy3l%9-Kh)~H9qd? z* zPU_a=V$>uyzbp((yDfRrs>P=cz9;5URKp^UUM0f(9Y)?{{W?FXnK5TXppjV9p#CT# z!h)S>hQ*I4GR+t)a<@70|HC?>fcdUkaIYRdfQ*W0*tjnJ2fRj|Qe+%3%<^S&^Ri;v zaC6|P5Tv1}i@=Zz21!4`E!h>K3mQjany>5*ihMLXb0JmO z2C%@>F`@WYo#`ZxgWlsSaSEsJ{b?cZUTAvg45{G? zp8!B|uU6!eSp-*YVW+ETv`)uZ4Y-3dS;%CtgqL3LK}z|q^xeH@=(J#?o1G-%3ZjPp z;mAq_#W^_3HAR>AIIrXpQx|bwA?!ZR2>p%Ljk&o75RL>9(4>}=v!|54!h3TQv5;k? z_AE39$7fZtsXLz_rV=!wu!zk_ggOQ|<_Hv1Y4fTC*p_om6#bk{5R>gBn%F4+zAff3 zyv7Tatjc^324Fj?MNsW6r%rXW2l23^UmR>q7`^Z^R-LJ?11u-b9~WP}DlJ!DUL+ND zukt2K=F)5gAmO#bye7|VCL0RZ1>75C8}EoF1O$UhTNz1L9jMC_@I)qa`SF^W!fMCx zvF2>fv3?;%k**MuAPgVfmX7j2`Of}hf=vBtj*AHL0>sVxDDRUXVOBD<+Y$Bpir>)p z{nI6%`tet6^xZ34+u}{- zPY-48c&PLIJ1=q$%}x{uRx!b(3)U?Q&hF7I!meWkzo17~01DAUAApo zb0Q{g#5~-GdE7rCcjj8l5vUT5oZ^O>`;21{qlxXiyl_x*VlvK>t9FWoC2;$ z{4zfBswAF_6ni~a;mpNaw6E(X%p3WBB;^&Ai(_mfR)`CNs}+L>*meZDz>+}MP)(Qo z9^I#2_U+?>@L1zM$6amSxjbsdcjdhkm0k0^cI}uvV6dxMVMb}P!R98p2oa*b*s8ge zETX0JSQ~SgBUg=??D<_|r~mt^9f{*nS{}x7#L97b{VUHKm1#DR!iAOSuA+@E*)JZ< zbqws53T68)s_0hzkzb>NP#hY_)iDk>w)p&uu6SL;?Fh0LuU&%o{llEfhL*DeVP~8J@bNLHG=l*X_Pq2slhdVHK&t&Gwws6Wrc+49vBoyG zDbu;@9VttSm_fRZke4rMAY8U*>{M$4qwA-C#c@sh2JMRPrN}&8?cO_X+`N`xtg!wl z#jVcnQ8jH2?&iT-vsl;GF6)Ayuc6nM6zz+$*j8En{{EdYCP8_=MH%$HrQ)>JQ`O(_ zH`beLhORi`X++zVcyN4b#o^anCSKBmoD5|4J*C>01rxwEsKe_Vk@j0PwWdk6JP>=E+jifG5VCw7y-z;nihwVWx~2l^{9L+U>_E|K>A zV%>@nde*$NJg7Y#z9w;A1WZBrtkVeBOG+#!q@lA}^8>{{U|!u*QZGdD1q`n>?i=76 zNEJwQnzRmENg?rsv^gFd|K40LQ>lzgJ)comTE$h1SC*;`mYn%Fy;H(+2TZ!fblrucJkCYCFHY(4f^s8#U-NG1YEUo^MoMNpUI_b zwy%}%VwU1kzdc^2IW7ats{(J8Oy7HSme$Jk&+_?*D6|&n>o7MYaT$lH_WCk}eJ6_y zQ9Y_k?}H3Lb|XF``n6#m5^#Bcop6O>4-oHlR4za^j5Od=FHor-%T*)4f>k4}rs#M& zp=929rbr>-Hm^cvC%#kjddi&10)!NTU??UW@+U#2+x`rfvGg-pp6TsaO>Dr#RqRN zA`98|{{AiZS`1bmqgFpz_bRQnY{};PLtH4)yLMK;RGYhSJR9r=d+{z%w`I!e*VzcS z16>e1A82^nE&8v|5c@E0?ALPqrO4(5?I{Z_yl@4J#cN}*(IUs+AgZhln_+JE(X5`? z27c*P6b5x;``*Rz ziYaCJO2ra%nDrpN3(5*kofoU_OW_m)CdVXPt32CT+hKnP_rYt*Plc@CMah>IgaDH? ztO{vAx$-Jy^8Qe?c85S2P77tfyzk)+DPLcV9D9U0tBUe>p@U2$Dy&2Tt4%L zL*yqAoWd@*zR&O7d{8rL&XmDBd0>|KzERD+PiTkm>|pC-Fh@D$sbo5`)R$F%=olhT z-^z`%p-Mr#JHZ>e`MDL{SvVzopMvCsx*@1=`(XgD^rfX7x9<9KEZbU&!V)RHlA#>(y19ggyS#q~ns2Y(BZeeF|C!u8I;B+xSuVz^==LxE z*_NzPFaMV~_o&~U8CWE(utg+1`OByhbSy=aZAN?A{+_5(ul%-bRw5XHu`b0=hqjcN z<>WLMFHo9T4gU-+x51LpI64shK5Gv zas>oR-H){XLTVUe1jfLuH?J3Kl>x#-2i&a@V`_teHR$n+P5E_}2E_J#F<54fkaMeP zdK1qXO3N=SD#{5^)zYmB@g7T80h|A@>f5Ucjp$XWlpwVoPFO)F$Ks==O#*&IkqaB0 zC4RtH`tOC)h8bkRe%GWA27F&KwCElq22-^FgY5mC*sJgpzJ7jlE7JX6c1g>H?szV?(FzMla{Mt9H+0aJSgK&(hzsk{8|Gxum})6*7tn;W*}XcEG*eE+=>qE%!C)=*Q=d0Cgq^QS+H%4@n;SN7);*1J$|Np~WlFM4 zq~y9zbFDyjBdo{?xaW8T5Avh5&%0I;CSj5m@Nk{ z1cGm-YqS%W?fwuGd=A}HsAgc+w^pAg zvk}TUVIn(y&}~32w}9!0V2@G1m~c8uiVQNIG0K7!@3=z8@CGPAhlJ=*TuM{bB@o4|Nr_>#|9JdOr!wcB1kz4ES;}rk%QqkTaqwB? zKv+EFXLD5lNKBk3@t5GgkB<;1gHgC|!66H2qD5^r$od~r_ftd}@qpLOQwW^0%0(kH zbV>s==wnh)E&TxVe=L}}GBXQI5QPRxEf7vy3uU3Z4Z0b;v84So9{+Mg3pQ5lhwh~O zb^5r35SF-wFFaOVTM<%*N=RwaisZQ4S7YUEX$uj^O-tpF{d&IrfNCU&UWs#>(Vd+| ztHsGpzy`5jKJ-%Xa~YY#jwPaU+mJ^Q^fjX3IO%{UJkCXnehO7J)$)in)V>sa>LsMq zbuBS1L^?S0{P;VsRA7i@%kAkCmX_=Iu(xQJ4q2Yr+@!geD-U0imO?#vdP{QJ5=_j~ln-9R_ip>vaGhx)34I&9#`-UiQO zqfb4lk2uM4i$CCP3}`WCPWp162cCbCl?KcckMU0i4gR{176c6!uy8$pC-9t)pra@= zm%)UyOlXWf2N&f_{fZV7Z>^Wseevk~w6CGN&d4$|S>Q)JsGNHpH~(4fi;yk8TxJhK z7i~+UxG@~Lghy8)SM-+(M#{!WP8%Hk_oW$=M|e?O31#sIi=T-|E^&F1W8fVaLtcl1 zCgo!is!Nn74fE33G!=`A%F6C(n*NAHKX6 z0bx5ZM~7Sv{SegLJ)G@#B_4b|EBD=$j0JDV#ZC6+TM1+Nf7gUU=%*;`*1=3m4E8zKnVcW=`d>(nj6}TO{rzRrJYnpI z^0A_rTHZmbAP&0V4a$cQIO31g2#)}e{T3e{gIS&%FLpiY-5WjFCN@Qch4VI3lPL`4 zPa7YTrhiPYD9p(xqFDS5jU=~oZUsIgftV7qtt*fw+Bu{Mb||RcgSRo5Qo` z-4kAnZtxfU`vw0+J!3}C4MV8*vk4aL#~kguX0?HnNm0SsrgeLH$DY$i6mU63ABgC% z=;jP*ljGe-8?i{eHFUgiSq3p>jvet{j9EX|J_Zd_HAX6!wJeAS(7k%-K7{@-)la}H z=`rM|&EW6(8=JTqcL@pekqrqzXjBb8=3Iiw^qEOfbOX4*33E9l*Ijm@>glF}TGe{PuUYcspcz+PZlBJy%+f4&Ivo zZ3(YI5X;cD@XEXZ zF6)9?kY%ijzP2o1am6D29Il6IaSivBLEn+FF1*NEC)<#kOv#Oc@o+x3I$K*;!ytkq z>1db!(sr8Qwu#=rnk+LOd$q3-84|B}n|-qed(=`ii^=(M=|Tui3@b~ED@GZMF4=^L znaW>27LQ=kF;QyMnqRwgZmaFE{WSg)OMAH~6%HTR{RT^;B~MXQf?4_$LVI?Cx{TS& z=f3woLNcByC^eHb(CKNE9wFD@y&>Tp^(cRp(xxg?Ct&BeyPo%lV%e(~|1y|eHyp9# zc4GCII493zrSYiY9>Vb&N5EO234pMYS>Das)lp27(YI3w#Z?Y)^%mba5PJ73S9MPc zt`nXY!2PZM%r03hp})p-;^+7>s)W^;mO!!_Hz|N74J)Z|xCBYkB$jxsyko1F15}jX zM4h+T$7N;s5okg@9(q?2+i~5{bvuMaXuPex-!12SUEYES$lzk#SNNt`yk5nxfK8=k z%ir=MzTsQowK9zS1Dy^!5H3|l(ui^05ja-CCo=0@ymgbGVGxV=FEP>_ORvJs2e~;v zS`t#CZs(UWN&;m*+LDFXg9pj8fK6%h-H4XpE8lDBf-;$RJP>J5N_moZmO%_@wv}x0 zox$dUXuX^xyE_>|Xh%GOe=*6FzS6~-<9SiREWyk@e2 zg9QMD`2D{cPd#vvAjH0br7VC_`N553K-hw{9Wjg$ys171|6?tpmNorabXV)Q?%Y(K zJT!-c;uI{jlSG2EdCrMfgFOV`7#i3{r08LK5E(P1p`h_6n$NEjOC4ssi`t)Y-Qk-A zRqv1o4;I1x*kU_3G>=cujWUn`a2szAv%{H>!VXQQi2ggL%)`CZxIIQ(w)0gdBvOrv zfWh#afE)1D2o4Qf#P6JimErlh0_=w3?1^#XDoM@DdXJJt?2Z87V!V4!=7Nzym)$<56NiI^Mt)h=6m!sg6u=1b@jtHZBXP>jHxyM{tvKf)O2BESyRI*$; z?-jL5!^hgs{1cQLqgVPPRw3g?z4|!fhzk&bCQN+hO7Kq4r*x4<3p1;Rd>P~uT0vzA1O;Ud_m|g2rrFW*dzH62van^E-*Am-o%qjxO(iv78txp`v+ilA4_@D}} z`RB-`@+|yvG|6vNX_CetGJ43S-;(S-_i0muiUvmh2ohUR40c2eC-?9x*moIS7_kmp z2lZZ}l|x@z#pk3TPdNc_Zfr!q-&qml2Fk7)H#jsF(*5ZmBPC2+3rH7Cknq(hDO^h; zIX{_FSbZhZTy`jq@xzHEeCOl6}4JO^=BX`1SvF;!XKXhD0^l8JZf#6S>1m%tw65iEQ9jn zsj{!^ya0m!*R|RZ&SOY0L)}8Qob# zkNBmMJn_S4)IWkC070MSHQ+@dzFD&UvnDZ$Oc803agqVf9BM1mFmc99YFi=Kf1&?0 zlF0)TFFCkf`f5@hu)5z-zsBM-3pvH|J4GtY+f5KJ#w_seyoE)5TQ4w8 zCS?`iq`x_L(5L_Ytk6U{BLhB@b?XCkxQc2ft089Se@0)so>Ch@Q*#dN9}P9aHM!$! z_Jyl*>B*uqBD+vn@F_LtOT?ou3nF!h9N{?HPWJh|=zoSEyg)!!Vkwp#t8-<-%20P! zsUxu*E2Ld-X8*epKE@|Pn9=v7>?*8is{$*x7h5JKxW4{C`7<4CP$J zXy1nvR%YbTMaFj3TmMAhE`J+dpP2L9(9I+7q|v&+e23_yS&z(D*`vFsuZ_{?2~J@p zi1`%O+`1(b4a7uH(54saC+kBi)BD#_3jLeJlAboJz>>W4D`g3$%Ta)Z) zi0S+Zf|R~#LlQwEvPi943ipX`Mtv-_PpcoaVzUU>pD1!v-Cn>wIGPUBHodpDUmovG zavp2mE=HQXI_5`NZ5(odD3b6PXtA1Vq;0S59l3q|$RUicKi=etmfdWjshyDUZ5^1w z^4t4}@oBuo(x+~)*ffRPP-ilFnTLIOpS|ngYveK5Y12b;0UswvWp(6HWhWID$@GIl z_UO}24ycA2`mI_PoL#)Y7V*5rURSKL+L*%Z;yJ7kzQguCjj8@RlFr#L%%*VTER9Jb zhTBH1l*p{rWTbH%a;4C)h1biTCP%Hk)4vnD>qUgU(e^^skmaE$GbkE4XD+D?jK-6@ zhP2bnlWAgigCL7WwVgefWr@FWXk*JKAT`rzOJ@l_Bd+S%)R)WjJMCg; z%Rg3X4<>TxHU6!36#uS&J+WSa)XL1M=gBqpmyjOSerDMHH>B-c`O43{AE#7P(LQWY zot7eJrgC_?sT+jHdCZv9qTkw^Z^^j=<#30;i6Vy6`0ZGm0<1q|?)ca7>QZ-qr!mqc zfsNbr#}1z%@LxNBGxSC%?Pwab^#S#=d&3;f(*%ya&Z{4fV6;fiE>p5ESzl&er3thN zLvEj1Q?krL83-9$%s9a|p%T+6X9W}0rc7e5CMk#v4ra2An-|(025TL%Crlc{tjmmd z6^tUw2lV#-E=Av9KwFLJo2K%qMTLQ*E)|hFa+LPhNG6UM4&9B4>_e3ni~pgkf9|HN zgRjNC7Mz!<#-?v&XWyWy-)Y6UIv!dKqmT?%xxz*U! zET2lvex^r;hmVKTlKze5jz?RtqL&^42g=rOARsXQfcf=#0^GU+a=1J-C?kXZ&OEW$ z2l%l)MGyXunZyR7C&`?|s0Yyf6)6Wncm>c*`+Hgmwb^krZZWNj$=KAg{&PeLPPZ32 zzS<5JUi6T7SgzC+@acQG#{7A-{2NcTNNQ%_2Ox{VeG}Xv$VqHpGhEb<+;h2Com$nP zuj9A{sqrvyNc49iOVd>^JUILau8Pba$)@J*q*;s6&w)uaWBRXzV)y;jc8>r=+grHc zUfW(he!E#+s_wesR=&Cv=tH2;${A^hC1(_gAD~#UEX!6;_BV!&12d03zOttLO(U?Z zM8*|3M;Q)%nTAo}!Oc|br>+`Em4_8(@td7VJ=FP=*AsuN8svJo6CwQ5NFnUG`DmcO zKCwL4m}?~wekML72f1lfn-X=$n7?x+k)P2v`yRS02Hy>bMUeQr4FJ9~5W7IRReMD-pP#@zDACo{7fAkx++-Gl6 zEZ}GwaB^Akf{GHNo|Oqt<;cp82V?f6f`C0zI{CQ2?f<}9riGsEquYG$?BN&yzbq6CJ zysHxbSD=yYKA zJ9?-S%8cUe>^JCVn~NcNrKePM&zf`815vPjVilG^_Pw|aPf@eHE*^m(DkpIim%YGA z{c>xp*xPoQACm@wbEVDB;ABo3`rMcSCLLei zd$#VnxXTcZJ!t@$^P`h=1s{3DO{MD{ycZVVMd2L}uMGNIsKSQ91LHRVh}ep4Ge_%c zU#d-pxox|do^}E4bQNSlsOyppntzZ19iG1h;$gdf5&bqbiiKy@o=v&Gm-pRE7~Bjb zhCdQuHv5rup*ddrNGxvZfscz`j2JKD;&V=)S=}e|(@0+KQ|xmx=JE&l1ZVcx1^peV zM{Xsra75Qv$}bBPPvGS_tchgZgjLMWpM1Hwc4|X;9vspTv24WuSK2!1EQH>~M}vI! zL=fXm=>Z>lclJ%T#T0CepO1==cS1!z$1?PI+4fArZTDv7GW3kVLGOpBq=X`FveG^h zg3s`u3abOS?7CEBQu`eH(VVHDCo9d`d*G^np&P{zK$2EcTuXF*NY~ z*xf*LtMYI7D@R$IXa3$Djeq*luqww05k?-W za#>&9k(v=BMe4?!p;ms_Q>CkQ$~%kQV>+|0ht>ENyPiW^ zly6dvoLcr}-7YVbmzsVBn-AL!mm)jG$$&jRb#1T~&essb#zVphWnH>xu+?!73+_@a zK_2jhvK6ph`WVHdyuoE}6WY>ZPp&6m67Iyk@BTv2RaD64e0WWxK#!o}G>N7o%WxNr zIcn`3Q{$NXlKhaaq*~=n0s?V+LPLF3XseZO^_OgzouTuL81=r+jmbpNiQ&Akv2vvC z^KmF z6j7afv)0StcngtkKKSIdj#ssl5>%;sG^0?`K&glXNI#@~1!7A}s1IS<;u5c}1#e}# zBA~Y$F=F`N!djP@>1Oib1;SZeYYgU|tZ!QpiKMYE0SuV6eOEs>ctlQE?^WMIg7p_)Ra8 zivrxN(lTBTymG?ve=wcvFumVvqJeKRTgwlR*msDdYoWL(eUEhq;gFYbD3GPbnH8n^ zG%T)KVO^VD2v_yB*yfK!Ok1-04j_yKt3LdQ4G-VAL@sAAPs93wNU-H63n)t603?Z* zz=vw3bQZ=isrxecpM}9}QYc4xG5M*k&rL!J;*b-8B!)?||2;|(cTE1=i8|AJ1bHN@ zFsEG}$Wt*ffABsl#o2^FRirC<5AJ(!TFR_iRh9j`7qSsf(zc^;NU7%)Dtd7?uJ)>& zy)Fe!3FLfma7Ru`%WI{9nuPS(rWk)Mdyr)iY9>4^7R0FeU zKPP559c}iD`?<)l&l~FdqEi&j%FQsx4t{4$DvPOcSniY??Y+M(OCqPq3ic9b(p__) zo6YAa&`r1Kl6Wo|8*ZSN2!2kPhO-X?+7P zg1Avj`{?UK|EbZIHF72gBJDCKKW3}~hwoSLNYKBv()Igo4+;V*IN}CW(`gKQN|9D@ z%k~`7sebVdf$9KQdsD?qhSd?sJRi(}apQoN=S(!GEjlm# zlV}O-b<1TZO1+l73E}w)KAC|y=dyhN?}WLsn{gH=4J|_2s$TlmAP{b#G%iFY^TkU$ zek-jb9kwTMC@vuK-xVjOs74^-BJj;6t4rpbLPsAjN`l*TmW`9qHNkd8{9V|Xt!ZgJ zBLz0Lv0BSsNz;zG4fd?p9^&>PE++=q>#Hu3`4CFSV#ZdKK<6;ro7Ls(<>cKUYg-7 z>_-du^9ExbuB>WSFDJNV_STcsJ!=O8N)9&tPR;#o^Vil(`)c)?D~p;{uw(_0!N2Dz z$zmw+Nf9-4PxSYd?6J8Afk<||=}54{ka|*((?@SXRIVehbzmWVUh!bv;}+HAbZqqm*|FIAtmT=x1RuyOk?kSW;v!uz`vmHJT>BF%ts4Yq`W74 z>?4?_R@5h%G)q7iCLg`q{W$Oc5@pLLd_D4Nu=`3pQWdLjX+TCI0XS>(yBdRXU4!| zwnv4+1vUX;(^oL}36{+IbZy3GpB+lA)7uT|)BjMI4%MX7`5$*L1$JX&Ei<6eMy zuTL=rpzupiqovVnq?OjK+qnl$jG+eTvM5SW52Ye~6$v$QCxp|QVPLZC1Y8;$Pkh4{ zJ<3_+jmK9&lZnuhFxfpNCWhxv()p14#NIvNT0>%_f0EC}=QMIY9nYJlVw6bp&1rft z3_)5*$AuMqOAR~d$fJM0mSs2-(K+~y^)uzF{Am_Ej;s06^GVKY0pq0l#dO!`ARJ1k zHOQp|dY^83mw@Ha{3#cKzs20V;2-^l)(pYWc!mGq{s~9&#v@mC>5z=gglHL6R`88iD?34!N$)M6=v%DGc zqrP5-VF>5Tvc@NM+sktcl*nb~NiTNTE75O>ElQyz0j%(Te$mL&|-37pkLz&@F*!3Uz0xq_s5+%ZT+@%twHJ8U@0;St*4$c#gz%*K6}4x6!5Zh@El5 z0P^gK*l2QR{mo-OitWs#JZBvyDChKpoQlt&0vNyca9H>=4uIHIyLsY{G<;uHXIv{s zy_HmWQE>n7g&s!GcrY7sJ3=B}oZ+O`Sm`!QESYuN7#Pjrh3;#hMk z>P}C>L5p--;Y)__N<;^~x3TW~WpZ3l%(SYPoQcZOLKU5;!2%cPiY9Z;CE4qG^H06z ze<(0L@_BnZ!Gy(I*~}%)S5+sZ%|5)6Iw)cq(8VZ8bhy|gUQ-K22)mAh@h^@zcGBdD za1k*pwxX1ITd1TaTE30TJIE}*Xs#gr7`r%yaYB#+{_c|$n6dbg3I!F*u^A6`d{nn9 z+EMta2j3*Ia)A=94>!h_o%uD@C8@6ewalqQ*7TjLKvjec7i{fnYbzutjL=CB1wkux zp?G~E;h)t;9*zjHti&$?0cn%jb2~B4b1Rz*!VXzAVl7(b_RH;79k{g@Glq(0UcQ`n zgjnb+{*~Ae#XrZ+x1#Bd)A_JOoeCW@`&b%|G#p|`a%Z{2?*07y|5hI-ILp8a3=doBfGnwNZ)8K)Fm9s` zz*Ik^7Ca3VHp%YenvfweHVSW>+DCL?F^XxiQRS;=P7DEv)KZJ&P?%^u3jDKg$0f+} zBg=C`5TlucS4@3zrIVHzd4>!sf8l=}4<7436`C5R6pRi3MW#+at7w`NJc{a8>~fY} z0-@o|;87hp7v?@6Bq&oonkG|KFE!WxbA~!D=QV6)fi1pYbuBj>2ch)V>OiPS|47^E zbFtjCU(ITixIX#%G*CD-u*5Gr34M5Ar7&(X(cVI8vkTw(de^*6Ch^R1HY^bw*!OXh z(qpM2`{qsV--8gm(!e{Od@1=KO6*~BMsReK5R#A%Tgo)~{pn-0*L-jPHK)y@q{h`z zBenl4yJ#N$2{E+#FjoO1X;7lu`vTYW(sfw`Hd=(@nrbeuHzdYvK}GzAs1pgYIFr=FoF_g-{2@a}pbb zQkX35=1L#++#~jd8$`bMxw`PjKocNzpPQqz7LE`H*A(@LfB;3k9`{mNyVg z4>tCNqBCo-|0J$(o7H&4Rbq>zM~NG$ck??aPitq5)E$}jH0O*Yd5VJQ)iV1ZDHkhd z_A}VAU9i0V#C4@ke&?1-AEZJpDL%#-ZreEWX{?ZQT_rcLwbjIhd!xc)geI%4V=C$! zb)M`pX}|(#GXc!>exDbAhS7^x+AB9x=;hLdQ-5jFm3hxnu}!Ae+bF8!R8iMh?5=VM z2n4T@7DYGgfTBxn!AQX9(LjFIp&k?kpB=ihH||O-@U>4vqhb)~tES-;y!4X(1MO4& zExG6-IgqS;PF6iatEx0~)b5IPO055gHhYOOI|6)M&z?Juqoge(+6{Um9`aAE$C%j^ z!?R{$6?{_HfTq$m4HtcV$S?mR*)D*Ogsjl3y?&RXe!Aip#zGTpReyPiX2D>t)FLV{ zwWa%MH*Oy+CPdIi(sR{i4;}d&Bp0Y=CV<1EVk!=#*gL_b+!V?!JAgf|L%WIg<`Y--7FHfEA6({ z21;sw#e98cQ|Vo?%JY;u{?~t?%iIe&{&rd~%Q&DNsz|TQ+R=upxPAK(2=}kK`CpiA zF2?_b*=FKk;`kpjo0*WAiRFLwQU51R{=dZb|BsJa)tguK97lo9lsH{CYT=xCSa+Lw zJZIF^=qQcTY&K&t#(8u)Ga*7*M<*%?tuw-kRHr&gSNOVBDnfCx)>ejETDviFH&dxg zCvRK8|G$4Au2bLMx#Lg%+gDHcbY{>9d0bW)Ckf;jNiwCgfVOzVhnixnhL1DP4$RJUm8zT`$FkV71lzmRf2x%Z6 zLO2sLC?e{=aFc&zRb!Bg1ep*-Bw1qw$vq-qsb!G{=6Pk)hO#A7Oq0ZTl7SQwW+V=W zeK3KL5%#1Wh=6-7aP2~XjnRM$3GpfSKq&wG6&euy8vyVJ+S|YVGY-c{v>ypdDojNE zF+4&RfPQ8O(a!zexCLaiq=%A|(>7Ek$+1`+?NhHfbt_$yf}Hn(b2KkA`+9NpwJ>ok??fvV8DnF z0sS8Mi!KN(p_>UMB)ARrQ0PB$yA0H0$OZ~oq><3w1v_RyClYes?TW$3gM?8_exR9$ zFf5;8@=maUcLgx<1YsnIIY0;|y4FqtVT8hjMAf#BILwqq4h2k)k|?P@bjm{dQ8jIT zF-zIo1Rpd^%6M~$3W^-*F(u&jc-*>9NgOtk;lWJ;=3od+YRr(9gB3Gw%P-#x=iUZ{ zF^{u~FZ?8Qe?v%0i=XU0^}yBPwHUa3asmYJZ^G>=!bxu-3pcARo6s_cKKdC_Pzd(m z50xFC?URx4 z+8-3+fB!ql0F`vl{vsyEl)pyhgh^%Sa{&}_4-cp6%7*RKQ|;$n3FUACR)gaPPjmO-=l1y9?(KYb{;MH@}x$R z&a$&VXT6>ciiC#P0FB74)M~p?N|Np_mAozAhCze9k)WS?Rt^0!{dF?XuwwNMUQ6Mx z1Hpw>s$P0jODo#5x-gWCSVn9f1i#n}p4_9+MBm(fSnccsLjFm5Ip^$b#odsL*3P$D z##7SnmN^k{8X1((gz@TB#asm!&|mt8o_*Q%@VvQ81>3$U-IlV}%SD>0wmjHNOosGd z%&q!r)%>kpjZ%NC{8whwN*UrTFK&31!;91VAr5AvFtqI*BtPY9old+Klx-8 zyr9O{Ra~6Aov8M2j;01>_g8D)=T--OPFV#Ia7`bdI-AZ(%Wto}ZKF@7W?iiOA(|Gx zIg`cKXoK_lM}FSxCF9Qq(Fu2H0?D?}x9U_2uSuhE<2$Rlrc<%XjEVWzxqEZ~DfavI zdi{FeNLZh5l3rJd+~Od0osa$V0uD<6jzSy0dXLhQO)-HamZmQHoABIAd^2QR;HAN7 zNk(NJrk`cy%J`L6^18q0&q&VN-Z?kP@mARCV}^@;JbHdvyXL&H?|k)!6KMF!71QGi zpke^OPxtVJ*<;~>50wSG8O*Qf)BJHRwam6**KWHv(%?~?i`uxobNNc5_oS}L+-mo> zG(V2Xm}oMtC#~(YeuD&&SsFI{;ku@c)MRy+$du||F?G?2fFaj@TOOjO_DpYOhQETb zDdj*=U74ut`#g=uiPlN*gurziNSSxjbbc0>bO>_SbeaPr`v7Je1&*Z1@zpE479{BXSkLj!`l=I#O!EkOjuUc!vY8mUHc4x6a zJT;B?*LR85>Nj%1FCudPzFuH&tl;XTX+Ja2ab#=jZ+7b?{kI~Rc21bgcT~Uf?;)(# zUd_8+$i1$&Ka|E!z(f5wAkTfNT2y(?@=ouK9&_8bKB@U^kIi3dTTRty?=Phfz!1az zcyAltMwE^>O}XMGA6a&Lmo_@OT#s$a$coVxyT?@O{?KKsK5g+Pa*U7gW0UVlJyLN-gjk zu%dOELK(HSOTL+}D0fdc_b5DCq8_$Rdr5OY$_<^u^UjAgA?`W)ur^LjD6PR5rATkh zi>|Z>e0u5h?ZtVD%Fcxz=$cz9ey!0z&kq*g3R{YX#?KDu?WyZeJRgDHE<2Z=yy3d{ zEMYq>60IN1JdkklAtw!#Ot)0#IQnm4_3{iO=o4nj;*EEg7%v|j)xPCq{swPx0}yvM zE4{bF33xBtmTzOI`#f0<)hNC$H6P#Nk+tUfYW(^CmNG^j`3eXZ^65DQxNTh#t)csD zO<&7qk|*IK&D%y9VlSqqEozkJ56D;!)OT*y$%$nN;9 z_3^BU*w2t+{pI`JPrqWa;B+3niQtXB*=0zxq6$%mM^J~zEjX)Fx3%y1ebPEitR7W9 zKXI)2;~FlYePnO5v|tQ>bfv{RR@3gRGjmw>Df>sQe)Wn8C0(j%HN>W~o`*!qsE#S2 z*ZLio(WKyEP~}e{Kj+3;D=V1Dj+2RR@?(Y9S4TLoUbcmWh!OtETFDxnQC(g_ zNS24e?^y3MTWmb9qNp!y$yktxTw=!nPBP7`P(hkQ!eUrwHWrXmt$X# zx~_;}L{V~%RMa~7PktxPH_;SbT#C$7n&qXQhi|*77U%C^MBXhDx%ig8Qg-7Zu0n@{ zp`q6geVYlnRXH5|+Z2cp(H_2LrwV`<#%}G7?{*D}lL}7Fbzz*&N_WyfIF%?351F^X z_dcB^MO0_E;xqo~X+{UA?ve!q{7$=a=JIOgNw}&8fA!+O_=&a#!vO{}elNrpke#=c zc`KDC>O8HKGOo^mqqRC7U3Grz8(y31)x*Z_Dbw@~#7JaUswnh7UF+f52MgfO9kMDwY1M6t$ExpnZQlBA+-*<4$2pgZf5(|faJrHj&+a>f zmV1F^1lrcY^)-4ox3agmR-`Iq`&!{Nz*oYw_Rz6&@McVFLZF%McGIowIl`#-w;%eF zM&S^hb!IT)mZA-DnmfhUAbx3s^ZjlCz@sD93Hf1pdS-}~wc*ARzrDQVS98Sjabi5} zb*X_JzW58-{VPZOzYRbCS(^y$jjUjJdH-kJVIpMWU}gQE7c(IXGZ)wYy#8Nb6)OwZ z|9^}}^}iU;^=2EnN@NQ2AWIOUSfLVKJ?%dr!HY%^AoGM-5UL+S z&;Z>h!-oKLNpP$LJjAzf5Q~J+k&p!;())UQ|iFq>{C>N zxM7aZR)BIj4bD@>P>l`&hl)DP0ucrd5e7>gqW;Z zWs;Idzmt$aAxWAf1m{j390EauwVQ!p-HK8`>}v?jf>bTyDio!mSk;TjAfkD!y!AK;*kUshc zGuDO(`;F=cBI*s``(zl8@PQNt8wf{5bU<~c?BvM)0eur#K~40P`nTtoS0K8LL zB>mPWW~3D(A>9fZ?SuF=zlHxukgHH9!hu*x+PcEi>CF&e{FyCGBoY^CLy2^*1Ge&R z{$Ui-9RXJoB1iqVq{9-ZsPog0FcTDvE6BOoX~@V<{hK3c-AhRw7e=8J8BiKWA7IK@ zU%&_-ftSV2Er^L?#ETE7RfHIzShk?42qkFVL<|HH3w#)(semZh?yPE!qAUU_nBa%( z3M{&(fLsAgSl$v?gGMfhoXaETb630h)=$a41qn7{b#TKI{&sBg}`TXY*J5eEhROy2Ov3CdSix-t37!tUVnxc5-z_YlV=slp?y7YS)_^xR*{DT&+I^Z1SRPX;Kr zxzGUuqn6rdD3dGYyEP@n{q~BX%LgM8;yPMU1>sZN{ zS-Oc5o=eb}cXEeKLo{VCN2H^t@n@PT`a&0m)>M-am)Byp@h}oviW|3Aez$GgzmMYWgF{~R*$zg> zn;;Clc3~7Gsa{7?bu@$b$r33T>e4nG#ispV3=pF7n3L=% z|A(=63eL3aqIP53wr!_l+crD4lP9)qr(@f;ZL?z=yWgt4YoB~o|9>BjdA`=V$EZ0j z-8?P0U^#%2qUf?awcoVk?*x@B{Zc)HBHbB=Pl7X6)qj36T!Zyvu5seu1FPtWRYK6& z(b>?Y6Y2s;#hJ~lKpm$U_(IXHaOZY?{1I9TSHc4l`nTCtGeLnX;cJTs^KidVyz?U6E+NHmlV znE`JiIhCt0?q|z?yLZUS7gWwRnEf}Jvb$$)$J5-3){5U&JERM45Qt&X363_Oj~2V) zGq1yX$i`0!SBM2D?mT32y}3f50MsO1be7E%ZS^IWW6GgT`EM2WUHs-g`Psa!>ZdcC zGQo1pWSDvf1z^;sI zCjl$Pmw9mLVYc5gWgOnOiyx)CuikE!QwVwDYvm9y0-rvPwZfs}(-Hm_x zM|W3qKlRNgStmlIVHrD~xg-pM@Ga}w{NyF71ge2;cf}rgty5o#6Ak(1(Y*hLnt$R3 zkB(;%vs^c*W2cv!`+b-4FGx(mh!5R*CkH@iGG-@wmm( zqOB)j8(ue*Upy0bKOG98docM)hHfb73|AH$J0cnCSrL|U9BczhmR}G^KO_rmF?m7+ zsH4=?Q$N{>zuO*i53vpix_T^KZgo>oMaGTfDJ+fK8VytnN_#Hsm5g#NLPdxHPg&WR zn~{L)?n2}C{lK~5P7Z$dOVN<9=FEgJ8Hf#~mV}Jyw(rn$Ujx&N5}w&JmE?zF&|Po5 ze~Z~UE;ZVz8x23vh{q}lo@YuCuu^Y^@PZv3D5mU1utXEmxC` zX}_m(rbK?HJ;Lb6K%urx%DXK^={!7r>!Wq0BVN9DqIlfn6R7I3@jg{c=K zYRZBd>qN+zKE6|IwnwX8m+ac!+z<7;W3u_$R`RQ_X?J4ff*1cm_m0=10r{L7pxfHm4gRYrDQ+N0-GpU+dzwJ#L ze?oN>ewmdzjq|Ecb78mj1A*!SCVJRul?IW=F$P;sZYC{~T9W(97YLw|#;8}5=LBzG zmCc#UOW@Mxb-jNAW?OK!b32M?wsrNau6lJ3cj;m1zPGZt=|rhq28mb-m<_bGy+&vH z{-DUU;^F<%qa^ru8IQ2VwC!$d=1lY^5^5y|->Uhe*D)@OAJOi$gCj3JfBgA;J4o$$H?FO5UaZ%4%E9*nXnwkD_UUVxyj7*c z%s>QE;um8+SW6ji6UVAn=|It5Af&{y=|Gh^!_HY@I_2&T<&mfmEBeqPu2W_AfT3yF zNUGkx*KL19LmZ0t!>{w{(s<4strzB$497XQveEU$yYn&G@$iBWO{>LfyY_E|8h5MZ zw3w}^LlAuqf^l+2HFNuw&ulXd`|R9Lq5Fb~!f`ZhMq5z(45yi}=xcOe_f`>@~1-o5WQRrwR97zsPb|6 z_i?WOYb1BOz~FD2{2ZDz5cLSzbhEi7Ccv;LGZh!hUQpNGU$5{Y z9u=Qqw#&PoxxTBbnj&h8ckwN8-gjm$FGgx5Aw~DrOy;X8euEDjyYi8NPk9Pf8?<=R zHhn=IIq2`|`j8Vnm=+y)|20cAz7N!YK&xw$?lqYoBs2}#Zhq0?ii_Ba`9Ab}7R>U> zvrT!-`a-;C#^F4%(1)XE7jsS()x;fK>}iQcF=uIei-G44;r^>`dpxEdlhc{+`W*AM zCiYMK`ubos4&NGF0|bx$lrm^K1Uc=8-aLZ5R3$A6Kj|L zyS^2NF~20cov0orOawf8$$Bta6TfCpV-RmGWu>I?&Lm*`k}6Xz$(4E9*~F}3-G{8& z3lewU^Gdi?2R5Af|C>2YSyHNd{CFvykC zqZraBhcsIjM~LS>y=p0!c|JDg4Q<{NDvLkn7R#pQY`*cf90tc6`xkxIx~{zRMxYh! z{WQ*B7P=iu^Mwa>@eh?e8EBA$@@A0NpS(3Pi-ShH%GnB5GBAe1D7_kEf24~oa{73$EEH^YL zc7SaDQZMH;FFRjc!Z^u>yP`)W{c`!1K@oSXXujN4fsMF9y)*@%_R z^yEkf)a(;-fwbiz}L2?aZL2VRldPsHin%*FrZ(y;w6mxi00>;Lm*{zE2n z{7;vL^B?H<{~DR>?y8yOi_4)$qD-Q^^GL=aPP)Af4+F=s77pJ^j2~GXED0kSS?mvm z0um<~R4f@3+yqr9N%aozyDO1*>fI~QYclF~({GQRMuT@4JvWelSEyLG zhThjVhZHJteqfYKg^8`i8k`kJ2?s0Ke_(9P_W60P2?iH8kk4-DF@z386C(j~0%gxP zFo_)i4n^U__UX?764QtB=+L)L$RxUFgz>!tMFkZ21xA5|r@w?9N?L|&$%%&Jn6XzF zd-ptAxE24ph=o#ESehW;z1-(dFf|N%81*0s%*2K_2Z|dAOd8E76b>SY#kLnMIKZa| z98GH&PzOvm%|1N^Y|CmQgZeOOC~W9VXm1bqETFqIO;6V>1_ZB03M&U>L=Ix9lc|&K zr%f~m_H9Svp8zfd$P6>|IJYbNUGUGeD1;8l-T?&yw* z))znpA|fNW@dXFDdwK%zeA~Me#i`)2oXwhS`#DfVN{fZ0+(pAb0S9xw*%q^G`Fbk~^@6gUX!6<|XUKfeN zF@{Zj5G+ZRR+584(4@~A>2;ks2A7C?qqh43%Zy$w#Ftb(7O zJ?pU4fFpX5ku|(GSO$awlOa&pESe2W3@}?N0Dxj0N&>KqZiDB6zJAu*0LKgWcaLm= zp2UJbjWs_Up`HbNUqAK_fDA2Cd99#mCQZil(_R;Wv+U{0ZZz4%_~PnmDpM4F9h zRN!!7@_u8Kg@azU)Y7aIDXiZE^}#&+YfxVGfD5R_^Me5KJqQ2bBL(z8#WX+yCEH;# z570b|(z5z8>i2Ha8IZeA`oy9 zvqvE8-c=AHgf6pY<|4tQBq&ml$~+?(tYGxFcmoR1&?jdi7!gJ5Q3|-Iv?H|0C3lMQUqYn1u9t4D?QHZ|q zRo~bXaswSa;R-`ToC)M}0o{G$D`od#hBw3Im7#D7?;! zXJ+R*3Z$uDx?B7>zy1uJc+ic28pQ;n68Hkrkvd|;-qxd#IS{1dQT2bm+^z_rUjc*| z0FTOYqfZdA0{^l$G$XWp7DYjEX}VX!ScYDN{k%%~^xmI?MN}j~qE93SHkCsx%(ulVfkkeK)k4$u^IPV}Dq}vevcO?^8XHCJ^m8g3X}$hOo|b+c6ToP`}1P zf1W*OZysW(S%=~8JSh{rExww!P-QTOEhLCX0`!M ze#^PGpY+Ba3_}8#E=%ro>8&%TDPQPuf>u~f__4nz{;q4UQHJS+-wDK~qaGSEs+9)o z^P-D3JYFbyw0;dYWiB?=GzCLrv%Mj;wD%pmfW~POhWm~cPi)A-{?!t|g`Oyp^ycob za7ScGSaWlxT`V0i|MONx(ru2k(xWa5%E{Mt9GHw8iB{Bw8`RV1XY!Z+|3E9Owo9K^xGoH+q zNnluAx^=&j{cDs+CY_4}Uer|22-gm6ei1*-=ZC=!k*B(}G#UN*Z3+8We&jzk$!abz zm-&b?=S4LYUrUouV!7^+hsA?`bX(Kf#@%9)$kG1&+pL~>wT7BZ9;Q*E5H>?TJ|!>^ zo30IVX^=BW;@|A0qF8!WW~%;THBJTp;(tbUUEA7!VLRQEP&wVDOt@pP2NW$b+R$|m z{#I6fi`F9qmL9LWD&u6yEbbUmJ06qgEcbLO)`$6nK31o7bxfCfIYieM9ozC394fom zumhA&$F}p6o1#bBFYmo0)pi?uPwi@ygIk&OOAo!X9eu14L&Qx<|E)H;)QVS7%zw5$ zfR#Ko`FjFSMrBCq=GldAP^%)~6n8aBlS~_;rRWxneABb46Q{IQiG86lQ_QIEf;kA~+>sg5YV{~f zT0=vw(>L8=T zU!#2|Px+oaDLECrqRW1}2Qfv?S$%x$^%b6+L;&2#OjxMU(m94D)sn#*j(t-Z+S1yc zKPUIBgV0yHeRU@I;@do!8R1>W0At17WOPVph@vw>rsLyMY7PI(Q+xqp1D?wgP3DuX z_0Q{hU}zh~Ohk_SyYXJV7tL(NDe$M;O^vVT1~PlrF+qk}gA?H=Ixlz*&~fz&;{jywXqaVhX zVetKAb-5O;#9MqJ7$&4O8Btv0M_-!Lk7M=vGmHzVeZaL&43@QM;u9aMK#abs-Pu~> zE9`<60=&{`j`xB0mrd*t6LoO%+7nE^ImB%uo6vTg8~|T%9D^ zJ@)lMlMbKKeTBo-ynp;M4X)F(%kfonZvD!0=kyG#1;SzOa!6#`RcP-_V4|qBeGatn8S{TSd|Og*oBgE+ zK|upUr9IiOAhd7c8_l?e`-dmRX+2OkcZ$P4JYNSTXA{)L=}tR>gavkLm6HVx=Z*8I zf3p>L8=@eGkgyJuI*#WcML}A6LxLopu?w!V5d+8f5w%E<=xTTbk=x7QGE*^QpK#SP z*;n@DWC8oPf7fDnqaiEYR6Jxuv;FQ%=Bgi%{?v>@otg(gs%fJgZGE_cIjBWn0EG;N zCBLZiE%V>jA4!Z4an&|NhVHNi`J8J4Vv{1l;CVUzWvTL$@V`a^*p|cYvw!5Kvs`Tv zy`9z0csl*b+4!ZzPU#QbrOT`@=%Si0D8*>;gr$(I;W*2Ia7nAX+1Fx~qVM^s&8{8B4Yi$os_`Os)Qcf>;=lY8 z;geo?6d7)5U$%yC-@kq%p?(Qs!@zK(R$MUz0M{FKv4t zR~vXsnvc>vYbQ#S%syOg1PF*jK|cbV!}z1mrs8_kuPGicGZ`H#-+`m=*y}a?#y?Oe zeDpK95=#_LW?)2?Co*{4wT?D+cbW|y@njfgYJP;2z3j)AnvP=xNcl%gnaNi~5_y0l zKxI$Eyzn8P%_$c@w^MSa6R+pT-D(n@-oM&s4jyA(^mnaJmuE>+fJBx?lvN zKF?AKPi5W=qx?ZJSgii9IaL zZG{u3Bnl%T5}A)J$q@iwkkYtn^%@eEWFi|gc$8R8#kFJ)y~Cacs-WTAMH6_`nOC=> zQ3NRlv{kD1IoLv2bH@Ugj8rI$sN=}mr493HFZiA}O!w770_!CrnD{Y!4YP>{whfW= zSCK_d;fWlX>Q5csz@IEav`;H9EMe1GBEaMykf9oQ1`*Q>i``wCgwsh>Lxv*XiCgQL zaK8twg?c~4{V3Cvt@kC4*3Y*(uIXSyo7=wLVez@bqPI)q*lJ)QdJvxZ8ZLPX<@KU7 z+%$^n9MPEtdvY_VxZgRTyWdLu&gimkAsL>_YFyS^@1HvUUeC@ces}E$w^uJ8qYxHQ z3TS6$L1da=4yv;bR!94|1j{BaJx71pc{Ef&+K}_U4He3)Q6%Ou%^6_Jcf9gu&>X z3(5XaXsHqWPAo9{_+zxseTdioK>O%IDJd}+K@inFog_~Ar`Z7 zi0u!8az#SS@f2zU)B1u|Mw2KGNKjj{cJx{Koud_$P*UQv;8gHk$5&RdEg((wnUbG! z1>}ATd-8TspYn9dwIn*q3PWlqs+U`)&PHTy@&%;ZRPkg|x3_fxlyzPFh8=*+0!f|{ z?L5=1@Td?tpZ(JBlUsaLoS{gA-kasXSG&F78w+ZxZ0mhWpgf$qAtzC|>)=|q zDm%fI7qGt=%cqePJ#4q(<%;mTs;_SHv`)UIc@hV{=w6vRCrBr}rJwEKZ(Z3wd?Itp z<*$enlU|~6Pv18W_K_xz{)+MM9&VL}AyxeiO*do@wys)86`%SiDA`|1OxpcxICFfq zd0wgPf;c<`OSpQDU)yQz&%;}#rv2SKyltKd%B$3k(q>7&18Om96?SqW1ia_pAf3N- zOe$;v*YQ`PnbF(9?h>mH2c{^ru6Rq@&gyK~uAP6^v?!Z44Y3J9%^;DU(PDDWmsApb z5X;@#2^>`Uj=XmKs-54duE1wg)B1$X#o$!VGKzmOCZ#o@bTjH+P{_n`){!Idzs`q` z=nJ8XORD0sn=p@?fu*N*SiA7EZ}vi5pQFd3bh-*jCrE>Ss*GP{@8|8V{hx4w`ZWFuTFc#b%$hiMP`;hnU_*M! zS13)@bSxdKq^6VlXJhm|4`hKBWDMsANyiqx8z2@@X+>F-0G~rdn0h?wZynP~uUa@dO>4;Q;_B1; z?@l$yYB}+wc0SL{&2DZU5>iWFD%=u3%|%%4yi>Z*BvwvMYxX?1dD zv@sm2a(d_^4S>i4ZP|$`B`$H;?9dYX`0m}a-9r$fm4BjtxR)kx!+AwaAJgj_J}!8d zu{qap+?Y*IlCS7jJaBqRQb^P$sl1oDs7ZSvN=DyKx27bEYm_m0;%KGVbJokIH^>C^5p zw6&pQbUk)-Ou0jWO6q7ZU_qT|(Z=2KFc@>S4Za2YO1Wc6=^A6@!CAWRcL{|Ueo-os zcozm6(bB9)x4FWtDLG+wOy&KsrNvnl+OSgViI_+~Hxu5pxv}hfFsblRrQ+ftF-!F- zMB8Z4-oV^PvS#^>Y<{xwS{-p^R>4&7r(|`Azn#SjDT^Rozi`vmgbu-yxzp6cC^8L= z=!9n)Hg@&?^1TX?7lMcH>zHN4A(hxLsRI?b zi{d_qt=3(X#?9&0dyDIss^)Be3M4DrNbEGh}JI}whjdkDnn0u6N>0?Cb4%#5EH#Lrdl|axlnOUnwW$aKI zkiA^qL}vvBmAEpfH!)C%Xoe8_Hfx|ISenXql?U2*Mkqhyq3~)T+K>6xwIMHfUg?&r zW=nkjaXmbFcn`bH;<_Pd|DI&tfQ`x?&iRrd$^Mpo5fSgUZ>efr*%o9ua@I`v6LE7N zq@A3Z)mZ=-{}deQxhL-VuvMAqGeVoMWdB!|mNB@TcL&>^jH|}ty9&KqbCbw}DG?q?OWMAPD!pLGJURYaFg;Ju!RQ<-+)0a?_U*@`b6&gKX@*t(H zWt2J_+l(JR6T!W-N6E(Wf&E++j(FsOVmgS@#3KCMyx==iy);g)+&$c(Gi)}J!_H#4 z;HIx%RR-HDRp?P2>D(zK(Vu!1Oq-kS`H~YD%W1uENo8`g8dXDWdgsLdjcd~1`V@W5 zC<-xQtfM9xC&zUJA4;Jjw=NxBoGUza3Tl6>=1ZTCi9OriA;{>v=lCDCg$e)4VFN_)9MB0#rUCezI~!Khyv`@gj`D$tK&e@k zEAR4s1fgmYwT1NI%yf@)*5zlH(j9Dfu8I_Tf%(bMQ;9Ibylci@tGlUJ z-e?tU?u+fA+p{W>I_2LFUQdRre!)2QwHxU;PHqk9#5v>1Slqq~m;kYb_9C3+q2UxR zam!QEPLWQ+0>&4njBDGtXLSf_3*g9Rx4$x3WB zt*F&X*56?VioJsC;R@R7|XPp{4F~S6vx1+20@S;(Ke=$7)=L z;drXh(|Ye0EcCM|_xP337txHUnq;GT@1P+nQ`(&(%F2M6dnWgiUF+jArIhxioM$`~ za}j3t%;PuOneJI!fseTPP^iR0SBUkH%|w^d!;t(Zld_!uNAr-!zPHZ8Buk72{T@CI|)-TJelJ_W&K zhCzVwyVz)=GLIxDk4dsx?GLVBz@H6IpVgzk58ugz4)@$)v|hA(pVqMBJm8&|LXUgJ z?`{Y6Zt^REZNuI=L&QJGC`YxyNBx{dm@{gO%j)so`8O5e>%F%ulhOL}ciZ3fJPd0I zS~b(rSitvFLrz1lf{^q)sXt)wxM8JTr1JRLH+r`Vieyt!TKwkpW^HvhpH%nkKGUK~ zZnVo~MvcD&5=dL^y-y($Of{@dLTnz6SM+R;u;N9wkuL}#67Y{a4#suBYdN16t(Lf8 zcqX>oFtEoVIk7;{av$inW$?M{t#kG*zH-Dnl#Vr)pa5>c%THgRzv_Fc^d6gHu@76D z@D>BJtAV8&+(Ot!roF!E9lZ!yZ6H+2V5cBiP zPAu&Y`X&a-AevknSsj7vnCt5th?JEhJAvc`S7H%RD2Rv%{c%mf6*)iyZ2M_7H9e52 zffocJ^IssqK08wd{1qYqhH9l)rA;?q^*`!^Y)I!aJ7Vp_}4 z&e;=QMo0JZf3;wkfQ)6Nuk(1dZD@na;!OcN$wtZokxL-f&EW_ho&hm|YiF!!0pZaD z;RxdU|LG{sRWA7od*OiAv z6@3{11F@+S#iJ6kHim^kCY5B-hL6t=CytL_Gy37Euhak`N?_^$y)J@VLVi{<*&&?l zIR#ev19)Gz(ftFPvFG~HUpFNu?(9qW+ma1Q>+l{4jrWB8{XUvLyt(_(p*}URXi-;t zZO~O!(SV*QRuG8tgCc^kJ&4D^j!s}ae$wDR;n_;Rvxtz0u1^l`c|#9=Ds;b5PXOgn z+<~mDbq{x}cds|G8C-4cUOz<~Kk}PX^Zm;k=tu9Z5=4Ci8wjsi-4{2xTD+g+Drjnn zDT%79hN?h_xu8Xbq-{J1vP24V29H)i-vVeDD+nhTz1H^ECqq?#4X-XCl2cyujt!9y2s6l$ zzzl(ba)1RDSZPmofqyd7c8yndjU9+TeX{X#fG{ZBY6JzWVToQr=SP=^@S%X(+kn4( zzRO=Zg-wqk(si-|`ic;C7{URbX|8n+V7K4&5Ug(o%Rg|y(*`;z7+ng}`1qGyKqrE* zqUPihNFHRC-#_cWf6!$nrf1~GhX^De09Nn2uyU8xS6$!TkG-jupn%2_B`249&iW6= zoaV~pX39C;VH{KQC->h!sgr@SeGiQ1hQA@fT8|kMPHyPUAanI@-%P$fUO|`|oE*N= z5cpEfTMz^_G=N5T%>^BZzoVGj-h1Cb($SNbOpDFVes0!2o8=e!r{=dd`ce8u#=)5! z9hvWhO@U`Ya49MK!qEBFP6TIrL&5u|j`pBHvVgL;1pRy$hzGz-GM_?SK<48=M0OZJjZ1_>Fovn$L2E!$EC3{4f30W410bf!FCjyQ zdjL{;)BqvaN#QGGKs(ic(mx_HpCJRye}nV?2G9QnWk3H0IR8l}|4C2(B+DP6y|a!V zp)SLm@qgOy_kUV?hyN)aR0F8C|M%GX59yEh=%)m*o9#2<5VV{5i6vxi*-wgt!)NN+ z<`4AP1jx+JCm|9Z%Ihd77yMo?iNn_lZ)e8Zm&5=91!yiNFX{)t12}E^0MQh5v7zq! zFy)KW$N$kZBu0HV8xUGRjK1wglWo`cnXzHgt!GT~~kDyR>JT{aw~SlP%kq zw$h6;(f=;60BA+^fn(wY)~mw*VG9SBbgNR5%*|d<3}g}u>O_{OF_RpC6qG)UwuVHmUlR)5%j(QDBzvmunWEbnnF=8 zhn6=U6vPd2YTPJl)PEe{Vt**qaz@DEO8F~+xBT-d3-+O zEx(@z_Ky#}$5nc`9_nw*AHQTEe?f(O?1O{sB5XjJ+1G#qqQ9M@?r0knplh6C;At( z61kYfeRDcnP$N1f2<^tWCF0S2l#76qtQjXIM-mX5#6Mt}4)68h+K4_g!{QhmO{NBn z5udb$X0J*6nvoYnK57#(gKU#fp#Dw$uc1trD24^OUPhQv?1UiK^K*|*P;4EVPc)In zq-pmT^~94MPg|)6AbEj*;Lr%DKra8lvH@ir!5ka^@WX`VqVOyjeC}A749$a$YYbH~ zUSjziOsDW&f9CevzDd91O4`DDyle)~uC4AoJDL8p-;7t)srYv5B%j+VNaH~QDe@@s z@p>L)8ZY@{teMjeS8lC$x~;VqwY-6^4d8&)x>tffSOZ^>E+N{Ni{ z0-oWR;~7lr4tRsv0Uao|K44%HWQdUaqa~H%CWRW1dh~XJsNy63(yszgGO~RaCbX)cavshbY=95@Jt;Mc|(qHc4lH7 z!+D$z)P|c-iT807U2(>HD{EL={Q6kSi$e5sBv=$uWh{^RdOw=H-zHQw2`!b^@hEcT zLPp&j;?~UXzj^+yCSJ<;&8ST17|et*Y19Y5I*LL05LLCappR509;cDP=vq|dz;C}@ zD-?nU!`d+}v+a|x3|VWN&BXCnfa!Ny46F&Oj=+*E++)Ph4SGQ(LWHniHKWyD_MIDs z9LtGrG2Zzsu)zt`$0O01#|BC~7NCDnflx=UDi}^d) zHRwc3?=2~*A{Z?2*zhp&Pee6u-?h21FiK6TBUT_4R8r2MXkFk9t*`A+&z_WV7*>+v zqbvn=)j&PqDZ7;*C)Df+3UDH|j=uLpq2TO}q#=sv$8g6FS;OL9xkEN0PxxU@p-=q1 z3Si<>rec7JHHyZrPW*a$>&HeJfnv(MLnkT^AKRmyo!r4u4RMnft>Q7`SuNw-Jvs{I zV|FhO?9i23Cv~-d$o+$~om($khF0T!-_Xk;Q|Nc0j>;!--w|B4~{E!?W)NWY*9tjzGJS6~pkt?np-NGhZFS zz2h0)o(VQ`b6e(_ID)K~jSIClMq`Zm3t0y(tqk_7Rb_4X)46fw2;vqav4-}FUn({d zU+K3Dn|KLM+#(2*5t2T#T4WamKy2QZ4%>nraXk*(K>HO^Xdi=v(|nn7dt9C8fYBLr zj}#HF&U5LP>!G<`>FtK-%#^{8HCBte#@nxZfWY8_+xkO0-KpjL-btPGqLHb06iCI< ziCeAms$imyGQ2B$phPcKU_4kJRN;WBn}Ny=FG!$?TmMuv0!5r;8=euZTl&Qpsx@KuHs?rUnG(s_V}MQx$-(14ROCu)s@aalG%nmYx%1o3Etf28oLh%< zu-n;H8ZAzLkCrZRU9M!6ejq>3d|R29KM$D!fN-vglRImB0E0K<16^y3E3@&bTsy0i za@4)Qu9&z%90#1&ncX+!y~wK8F$g=5ui18Ain`QKf-8A#v-{TQ1cYs*K808DGpUAnlo4K*+LLj$>SM_EY4b!Wkf1QIe!Cz>Md{r)C} zntup7WU)92sYj=$(5j9c36G=OSZz)muNOiT zJ1UAu=8mnK9dCaOENz7K2fm)y(>(B!Dl@2|;3se&84oucbmOhOu^SdPUD>e6W}D5d zU}4wezVrc#IQ@7TpTT>w_0EO?Ebe$AE-^J$73$VG>9~{IxS6kOvz8L~H201nUZqL0 zqENLwrT+S(MFBcqnJHH0VejjmZ(_3@wkUu~3VaC~p2-p!L;}Q5uG^)9w9T9~ot75p$*voVFaP%s%IBP3ew7kN@`*H)1q5*-WOaws3cc)*q#+H zpRQoHqx>f4?6eE@2awr1`8`KmwMTpgQMzKO-Zl%I=yFvp$bRu$`QR_s-U?o%*IA>=@QFjOzv8 zWz-eyE`(VC1go?^`u-az<0UynszSk%Pt{+}ljj4F@%T#p@wJjW|AFt-Gh7ZdPO24% z=Fwo-2F7i^wz1M{=91lw6ACpYLfcuY+c2ncdlY53%FrZvR9OPA2~J%8%PIpF;BPq& zSvS+a68qYnp8c>fuu=RU+erli0d9=XE@0}Q^*j1viu7Lgb$SaYTGs~YZ%K~1qWo|G zJ#hr2iuY)UMVF&x%fa47{hmCHd$WG_QC1mLsv0&QUJ%d#yDU!Lzy4%ze8zt{uxj*? zn+U`{X$#8VH!aXD#66G^@>O*UJV}fE9m4n%6gJ#xTDmxOWh~Jcm@!9kgFaFQB@CiT zxrUP=_tgCg%UV*I6dzb47zK^G*rV`@bSPZpEQ8N* zUajzL$pzOJ&cK?ld6}*$jL(WT4+{LFO0qG2-LO6suO|`W2Ij={qVUy^SG3-ev^uE! ze=aOm`RRNxw^LL(PEYpeO-cO82x5knvMH(YB3q%7K^!et4vtTN#}!>TmK|v!0s)iC zlSeb4sg#3tyR4;43Zjbca=$}n%_J!oOP&{a-BMH;vFI$NIps4Lc zdNto|h4U>|DW5%cLa~osR)7tmBY>A~r9>}32O#;`H}O-2egEyO&>YVJeXDO0UtDg-}%?YbMQE#$?0ZL91kUtYn24nvsV8_q)|d53zbqdgj>hYa7k zVv(yfn7noE{Zd{5AhNJ#tGqq^!LB?t^x|0O7s>i5Fz7*Tf6%@|kPbKr^}nuqX&+UK zwV)itX&tS8uDPr?7pY?cW~E2SB{4IdGls-(TLCWcnjiv*`rW%N?1?LwL|3NeR!w|) z%uWbVUsGMokJW#)4ZPP~PmLT7v>Y$R#Scejq7p-Dv|f3Ny|4HL8NNNts{cmEsgh>iJJa?vEozD5)I6x z{0|nG*5Vn!a&hW!SeE%dyrWvAZVm0atVZ%)rwZGrORjx=2U$N;(6|2p1Bn^Yruul# z@`N)EvLnZ>g9b~46XpfZt{X!IdK^nR*h1zFsnyZ}GFcfPq$zI!?M%l`m_aJ|xx*}n z*9R=+;-;0IY7=WLHlll3C5lS7=_Bvn^U2yHI{~~KuZf>?Eh3QUuVQBeDVGUK^A8_d zkqj;0#CV1Vmnov**u}`=mYtfrdXWz#a~10$xf42eV(U4F!GS$z)aA^l7Hq}xu5h9q zq#J$BRCMFDsrX=#)-LF$LtdBOwuS?~YS&8#zSseNA&ZT52SbGxE=@&=ore|P>IaanCl(+Uv}I_?g5KzSj^BhBNU6% z+E(Ae5(?q1A)Svu_oM=ZWo|MoiE71f^-sEAX)E{tp&+QA9Zb=v}~5;t!cKQo(?A3#;88lC!*P6 zo}}4o$Q2ykhT7M zcRMv$%r94?caG>6m3vs~#ec^NSEKk{v8+yUdduT{;3vFBWVYOIPj#+vi6!e6qun+&Ls zSi5eTW1cquYz?>|X7svJc@Y0LZYa7p<>}>LIXUHMW1k{CZ5&5sGz=MOhaV}cA$m=& zK%&*we4KHWfUBi5Xj;-F;<8OF_X6I#mv>XA3^|QSbbRZmGjq%;0)2Q62BL5V>vU@+ zWpJfF4oTA7u{7esOMRo>`fas3(MCcT3Y6r& z>Q$5I8EOV{ak?sk8C0$#`y4YBx(?ro zS{67x&w&)F_x-E``!Q-+v_@X16~qjaf@*O^8O*SInWNS8uw2bWZ>wG4{zSue?ij}8 z==i8SFSma+Ufgk7vgxbLVyPsIYpB4`@azUli7~-syLHN;4ca5bYk8P7MJD3$q9aUP zE-0HeyVikk0*E^%d*dK_>w2|?tt~ycaZ|sx)m`wHcOgk|KB?cRpW>~(V=QtbDBGN< zf`f+^_y}oZs<>Pmk)h9D>SR;%p0)yD;Dg!C>(yQ++$(uZ3$%yv)+KeDx%vaAs=s8z zXQ`d^`~5;LumX3~(HqLnQWVaLZaZU#oyQ&j+EK7%T!wP6aJLI!7nL;zPG=FD;Iwfr zy_nv4dVl^O05?F$zXWK*Ij;D&v{e=gKXea)UIe!*l(fN^LGEW3IGh@K_!VzX)I3ar zqGV>U9UHsL|EpItk>@i9(VWPngOOin=+g)NRpH%1w$WxR1Gj`5pG4#rUNau3!`l0wqHrx4}{(dpl zdw3&r!3X*>C-jt=0{W=!lH?q0y0MT0%n|RY!1K)XCiRB}Aw$ES;Z**w*n5S-!M0W? zqe=d3nAgx4xm}CIy)e)y$y|CV#Huc+2Fqj??{Y2O)S3u#Zk6fqjXs|);S7aQh&q1# z)qQlh!^>ht_Djy7s*Z8k=V?7N18Twh1^1ew<89@U91``%24%BlCb7(gw4M)FtZ}v^HTfIJC20)JN+ASdv}}On2RMPusx#yt&YPb z@h5q@*>6)81xU#uF$&#&e{*Y2*lk`*`9k+5MjY4|(Z!>ce9nk$V_L}O2 zZ3NF0AP&aU+DfPO{OXrqMu2o&(xp`yCLmWbAIWIv$M({CnA+B&8%_lxODu1POsoCt zkjin$JD;nzVc0|Eq4mq9dJ`(o4vHd4G>6Q!D8~RH=~3tw_096q$0=-utQEAvAiAH+ za;8`octj}6w3(L*1Ol7uuRdQDB?=5ziHeU=T-+JH~UP2FleTQl% zv+1PF`c74ncn+@W0A9GWwgxW|@|+@r7LEH|*_YDdY@{vTtv-*y`ydUYFg4=?)2Yo7 zm}1*m@f)rQmPdNj=fzSNh;mxc;rX;3v}4Q75avLluk%4%wkK+HSNIwe(VdLg!XYu@ zVuAga`q^tI|8~}w0E=y;YD$5rWLmxKLN>uQq-9Ka`$-y;rA_O;vSlMF_WXXnlqB75 zH&FwidZiSFjLNTrfe})#UK%K)RH5kGp&hXgWeKC7NSNHV7;hs(eB?MVoNECNXAhwaW(9yRm24#G+PD~Cf^NvPyco7e`)&fqu zC2Yn(M2xWG85uggdWrhd4?58l4~Jyt)j8hZ!&_K-A>4TEK46LkUyRJF_ibWu_dE6m z4@PA9I2L6_^p1V?V#zviZOM#jc~OwDS;(tIK_;Cj)8Uc!TBao>j)Dl{y_B-(cCWZU zg*qaVMv%g7LA?(IjhWz7otZ)HAkL+_S|;{ScxdeCQ?LBli@V-$27#0&h!bI4Z^Eia_3TFT4;GU6w zd(|~vG$c*WRQv%~kU26mXyPlnP)kzr9VD*LTK1~RR^&nF+$xIK6lTTcP+v-nu(y$2 zYaIypnEdi%QZ-8_)5ebg^cGU;+fLX`5y8s36p-Im^|BbhWlgaMy?ios?N^D8i9$B= z)1AsUr%?s?M-BJdomDmNq}8Mf5h0doPOQ2sRn<8SbMOWOv| zFr~hc^A?!(&A|7W%vk2gR(8(Itao2d7%2}kehAxZCMXksi5#o}Ssl?l8zHydZwbD; z0vugr$QIJp7H8S(nR(L`ej>M&XvMKjgw*c>Ap|*THXd=qiKXiF=NDdm(-K9zQ56)k zZH{{Jib2$+xn(=rw$cTL6roX91};OT*a)-R#arpU-ETbrE%ERF(V7e>QIP?Uoe!#9)7&(~I7bcfF!A8C3c=vhdRO7Pa|j+i=H)$Y=B zybVfcnG3(hCR~U#aUALBa)jROOUUm++JZ1l+4mKe7x1D0cbLL%> zzs`X`k*0~-RUeA9L#`wLn3rTUt6&$}GDMHRL8GD;A$>dmn*DJOx#%sLda8B5wqapM z;lW%YF*K_hu&YNW>oAbI2mSE_*#hbk5!I0<#>t63{mZ(-*@&<w9v!8lX{?5KRM0NGfShky6l9a~LXJ*{Z@cy03zgQ?Es^bQ8Bq@Z zBD>9zK)VOZS>{nPYhmV8_^MBrqMmoe#GC8v_;%<`EdPuVkNZ5^NnlfdiBQjQUj6+K zQb+;BJHtEZ0>s)r8YRKKwPf79QHOha#3YDPDTuGzHc>iuyJ$mQRy6N)e;LZQhwcW~ zIzbzfj^h*1<%>C-Z3^msI=}6Q>HPz zfy{Mz>3A|E`RN$jH=2MBT|kxC`ORgksYle_6;t?$1p?~D3egu3(vOnuVEYRq=Y(Y& zHCyjPxFfUqL0vDprC*VL?EjhVwNqz*=`4SiWp6%?Ue z)s}u8@gH=X8IaZ#g@0Y>m1ui8vO<}|vRjDdW3G}WC-FgN|9btdLl${pj+ATY$xQIg zWAvjK&7Q;^%8aXP-NjK{@y}>KL0C3o`$ptB_od-lR)Pv<6g&`uT_C~Lg!eRBCTn31 zQT&uCoKfSHS}77#Ujwap%i~)q(U zfBo{3S4?&u=rv-9yfs$C7T)<|Hci{*9psB{3!ZA!e(-HrVqrif~*wjhJngl#xFf;u@d|~GkdS+ z>YRx`LUqGaHNZqpZD?HQYkp1C#^|}ajyCUr=$j3h!{-KbV@3LpqDm|W`gbL^3=;e} z!o!{C5s?M*P=X4v+3JZ*q<#ZXwR#%>+n0tVbD9wrw-nyXnul*Mr%vQ~+2RX$)`D)Tk*9w}rjE~? z*(dBTM=hszQ7y57QR2;_4`usQ=Y69PX1~_I&^+XIq4rQ^g~rqC@*;9MMZ zk&PPc3Tza7i^xrd{e%lw8MKw9xV%XGDjsmF;U*1Y z+YF;VjjA2R8~b1dY&m66Sr#Fs`JT8;pylp^NOr~fv9y}k(24*i)6^%+A0y^dUjpi0 z?fzI)sR7bi14|cWUi%wWOtnxsd`&fKRZh*Zb#x65%nB2l!FBqX4OQvtwM?kp;?CV) z=ZlUCwE2A0^xheskzEI~$S?{A+4*iP`F9J^J8~9S+<>BB97TqXUU}Emltlo zotKb&mKB#YHPPW?Ao&{3n`hAhjl=K=|FE0wA3=&%60Zd0m23^9T?Gl~>8t|`C#p@D zf(1($;=@_ROf+KhuKMc-8I80jvy!JoZ$=mzFi`5L%HzDXEE_CL}^uDv9 zCOgIPp6k6QWJ=G3BAcPuWR-#+$xY2^bg=~9tsuixtWQr%@|Tus;$%2|o6k9x!=E8~ zvP||Uhu7WAON(|pLEN}rljNcj;i9IfUnkAnjdz1p7ZxtHrL3UW%Y64g*5ggqWn+Qj zEj{mMeYdn^ah!}*T+t$=h=WdcEu>%%#J<=fJj6%b1z{dHq!JKHW@6T?UbOG3m~Y zRjY?%(r{G5d*nj}v!3t9HPNtO3*$*8tVhy^n2(T<0k9<;_TxiRyrNVmj`@$qecqDV z8A>f$l9O&KdXEHsV==6dJZTbkeO|XtW3=2Oc>0R$9;lZaKGvX#lD;?y&fY{00z(Nj z8zg18d9THv$^`e{1Y!f}uyGgrpNpj>6EAYLg}$hs+hz>y{%m#RW;KDc7Jb|jDBC9t zjbZ;xw7=^6*<<8K?k4|e#A|F1gCc84vM5i6=^JzR>Jw~uzw*f}27Z_KkEV!Ck)$8C zt)JBI_0b=fP@`wRqH84OzIJP49BbXm^f`IaxI?!= zOmD;|hBt|QNczd&O2*RIUcomk66SX5QgA@bTds9MXqv2d-E&LH?Q%Hz;jzY?DrM4V z@F-Dccw;loPIN0SkKH)__(S$5rgPOwgO1pYf}&l3d*4X(_O>rb<;+@>%M0BxJE~$kp~l5 zqwttQXj)%w!Z$ubv>jZ7WV&JIMl7G0C8te1LLPJ6STa>6-N!44f1VO+^lK-{*2XQo zk-{Ucj+FB-T5jWhd#{KeO7Z&Z!$+5Ro$wG1$MdkRD_K&_z!CSc5Gw=OOtZZh>I2~%SzpK9u9-wL-#nXO4Ip0iV?DaKfY8!%@f z_gtkyeh!x~eYZ~_4%cKZ@Z2fM1B3i%Xp+b(h&2pnU$p8)pI%S5o{X3`Cciwr>59F% zQAF{uW~!9}e{&MbP%1uuN7nJ-i);M%~&y!HMd)6Fc!SgBUOWdOJDqQ5yO(veZ^AhzC zhb~G`grGX{ug8#3jwg8+m|z&lQ#-lBpJw!Av*@mrRCe3YTH;CW`UiB22Bj6bZLdB0 zA(`w@oE>igHiCs)TPzLK1W$KD1$kXBbHyYPQC9`}gbeATFm|n$v0T5ppH7BTAo9ta zq=VC8{n|Q;DwtDuyu>IeIl^N)+or2b_%ZB|#B1k6F(y;WS=)+F@|fvGiqaRA>C+=* z3sNl!q)HZg(Vg_s(Jr9*H*m0<9Feb#5y~3r+6G66%5qFMR|(lzAl?>mJs?bnfO7U8 z{gp#llhlmkATirgAhXfqwq?~}hcZs^AkwG{Z(T?xnA)~ra0KXJZ@k`i-0-Yy3GE6SHENdwlRic`m$r*?ie01Vm>B%; zUlYoN`Z)TLOfJ4+vVP=V%In;t71n@dHMAy?<T%~mh3@9#m;={y6n#Jv9zDcn-0+U!$%?J>_gCRX)p2oi~0%; z0hj8)`~VIwyb_8BC%uO zGQKl#9Xbu|l($x#+Nb6<`N4PyN#&I#63W9#%V0+&{VCW0PV;ksn8&`pKt%Xvszmk= zDa-{E_HuHV?{H1oXzbz;c8{AVylkNsH*v%rVysvMPU@&yh56jcF@rDUZm&?DpRLmZ zU8~bGJXV%Ona#9$@NPWoM45hwh{M3u1&K&qUt(-JR82>jG9z1cHt62>n^@A~mu#z* zatCDFDc-(2yS#c;L=wSzq3k zePky%ICS&WkRh6^Z`?haDNM@|dG_+1pgM-@^o^sdxEr9-H9mW7Fq@o{dSe1owN5Nd z`-DAajW^a7=@@dCa{O>--A1_^l_`uZ)W z{>&2_LK7Fn1*LNH?iOhs_oP|2iA_-5W{7N`v6pV1(MYOK3sk;1vP_QA^wmx?-M5{1?`#nE~ z+Xso`%$dcM2NQZKS4q9mQNs;RK=za@j3UNwlG?nNAVA!*^6`Om(%sY_=x0&3%3;6E zKinotcV+h(mSWc}RH;O`vLhZDS>%Q_ncXf)adN+-Dsw*S6_eiAZx7R(8wP$$aE^*cG}aL&aR+Jmy^l;yk;|qU!|egFbsV*%ODSmPr4WhiEyhWe5R7Qv84reLUe6OzU|>#P=<>GF9bi zZsAX=oFAkcEKJrzpB5@TjJ9PclfRpBO1{%Z=B;cdh=twJ>kP}Nu}b0MC&i*wvIv)u zS}Gl*OvJ>DP|{Dt;frJVG%=Z$v#(gynQL5p{Y%d6YF zh<*51<{6xo`1=-zD(Wh4QNl{{^5&++6o_qs#ffC}y#TqO%Cq`Z*Q(a`kH6c#if!6E>|Ke#LUZSckectNrp* zA2U3~er2C*xF@Z^zsWwqiBr{rk|)IV^bCbr;ylsJ;chTtE{SDg5cPu6I#7;WT5sna zvD7n=AnT>jNnPlr7B5X^F8<}w58Br=T(G(7dFOqCY0zx5-jB)1le+Y;e4B46{QTY5 z$MuR$3s_cr2%Y386EawBU9i6|mCyj)#-G)bSD-+uI43qrj>S77f}+Ht99NwlW&HrFmz(A!uFp$o*X$bD#1x zEk9hPAp}Vg1tZE-q&zk72I(e1ZCIQXi7Kj5<1~JMrIGvb8l;M0P^a2vfJptB8e@50uZ?EK5Yx}FB z$+&&mnUzQYckhWpZvdgBt@e0Xw`GQ#N6ucokdsRGF=yYAZc z&XbK#YO%u+0aVI46Q4rqQqT*c3~!CcJ!Tl&zSEs*$iLTCX*j0D-zIJV09!_?mldjV z*`H+hjRw*JkP1rOcj9Fg1?28%dvfb-A1pCN)aP?+^FyN|I-esjwWA2&AEsnne=xW$ zakwtkg^;t5Oez|`Gz*(d>+Kb=h)tajDm6(}ip3j_=Lp@vM1>#>^h}{VKBKQ27o&+e zD;P*TXMOf3i-sa)&Djf|+>(ol7{G=!y&8bzg#!I5V*lPvQrTzfyfI~+V*NgCI&fho zClJOJDIuMrZI)S4KEFzqaB8W~1vpQds2xcNDkS2ldsCAF!Ro3n)G)VwE_r=vs^Agw zmC16nv-r}ZU(>q?u{G;qO#hR%*CfSzxzhY3vfD{Z{uQ$w!E0cpJ*qt6O$H zE8AL98s4>2&o6LBnK7N2#M=?r3;);<|0pmv!^&BKjbhCpR089XGylbs6gxCfE3%ty zs*da0^6j8x786dO>8BnZ+K+f3V)+{vOXJV*90Z`FjW6#^!dRNerE%0vyEbl`1jgE6 zBv&~>XB-0S?)%a9ynD>N<(MojQEzhsTVM0PD5CA%rS7~3Z6$Kqxca>qQpF17B`F9VJHE$EaGI_&ArI1ZK4dEnj5qg$8U^;+!&>1UxiwR7 zVvi}piI*Fft$FOJS6GO7@R@2F&-#sH$+UNpzxl$`S96t+{#gVmD)U_J zGh*xwbbS4?rwZmP^s{aNeR%3%N>Om1?btg{XIhKSEm{w!1*;#r4zAv690w$y`sl*1 z($l;tijifMjz64HIjl!s?HEh!D^1wgn4K;Nd+I#6R({c+fc`}r_&P9?eWpfY`sB}q zE-IeHf5ZEOO?2)_J^!PnFI764Z{m4U9l~eMA)=D3eXrFc>$C*npnS#=Np?&)0slg> z>-di%OgA6Df1>@!c0osFQu(f%sQvARU+6wjNT+@1uc0rt!75|3M7o9sImm@$urCcM zU;7MKuh*_VFA9DQ5mekK54W`Y%!t=a`s_?HkfiU&i6p-*wF|sD7RS%-U!xjb1$9jI z8c!C(x=oUNpuycXxo1#r57$B<9oY~m`ux-`H^^9Xdgt)XrTTM|cE71ULA{u5?mHL1czSTXdgO@LL9`pR{A(g9dqiHa!j2JPmx}QdI4h#~-Ym&0CC+v-T%-|LZ z?&(HN`=ChaM&-sV43ZhbYE<|y@+7xDPX+c8I>t>5;6uq2Cei@J+fNjKPSHti%)n*?&a++g9SH|H}BBja~N zCZl?62!T9z3^O=I9X`#TK?k)R7@**Y^8HcFHE_J&LjCT6`5Aaff!BG7WEr z9O&F>T%m-&;k0L@$fRvnU_7C(aF-KZ`91;HgD@YhsQ0#%R!+Rs(|U>^q z%ZP%+CbdEYcLgQ|jg1ner^+Hq0sq};2DWI!8?E`tc+WV-dpEAcU*YauPhSFgtesH=Rlmrrx5{5|HgKfXUxP%j<|3FC+ObQHoV*b7r)k?oFcy9^ zRyi4&ojnKeqHfocV_p+-Vru3Szt2UTX>&+sHErq(iSjhx&H4Guk(3{`lr;RaL3!Ri zIRwL5;Yi+^NF{_oFa_XKNoVN0DhRfcPi@IsA^Un%UBBp0t3?9>_w&T2JIhyCf~U1` zG4rZ@WBBf#Dt5534II83X0&|OPu!rvlU)B0MO`ori?tS%41|#|_m77DU-@auO(&Hf*7UBvP!qzoT@w^w zb?SAoAHi(}w1<#CFu&%9D+JRp`{XOCq(5BEvAQ>3=){c4=!F!a4Y+x^du9ZK#^^Sn z%yL;3UN-^*QJ91<+c*)>V@yuiYolEYh37`>4UnbouR3$~*^ygIu z!}*_Y2(?aLeq>zt6bPEg=kg+tE^K2%tPpp-UYWxVAptlGy;uCk0k^@;sT&Cv;VULq%h zh<%2S{*%ypI(JKJib2F6ii-i&A^$wgjg}1I5E=&0_sC*%tVv7LN0~!Kz0}>7j~c;y z2w`(yg|O^i*O1aFiFCTx`OMKPDNkx>h3&6vm>on?lenUt__hi|Ek<=KA=>p(CbV#z z*jLODlnk8}`*d$_oA^*k2iYh#)FFZQoiV5$#}ou_hA1{k^I%%A?cQZ7dz~|OO!@R! zQf?%jSG97e-`wLe;S%rhGf~p2D#z}A@E>#cM%sjBrx#uTBzUm+;(pLz#t4A5{K|&r z<8FjUf%)}Aq-kyku1v^qX($nk`69t95Ym)S*Q5LmszBoBnr}$?XQJJiHe>l4d^DaY zoD|l-$(H<&X?`XTMW!aBknwG55`4EJ3E0S;|!EpM2mf)$HYIDVMd8gSG$ z#uH%1I`Ly{=1%VDitWRqj$UJ=1EB9Iw^t)-AwNipi$9yCL)1e|PDtkp;JQuZhgKA} z(i(or7R-pvAZxz1sjbVrdB+qTy8q+N7bA6KT-qvoaqv5JJil@kn;PY7a><$Uhj{2! z*ET}e{w#%;4*KNhpnG|MaA!mKbr0qq4MX_nDo|mhWy!sEh)6t$%PvlUlctFMFve5gISC=P2-rOJIGfW{rW7ow9;!uVFtGJE6Gv>_DvvzqV*>-SZZ!c`?RNI$U1CmbS#QeHf-eJVvinR&JQ)Gg5Cvz;JhtCe zjSPP**hT07r4`si`7w$G@bT4!Lto$2F?^+mer7eSLMfWNEJ3Cn3r0wYr2>RS=p9e)^}$}!2pf~+{#3-0&+9A{H)Myi$86BU&_7g9Q?eH6ah=MOvHU3+ zy?rt^wk+{e(_*myV!p_@u|%M?y-H$jqmRaiMFRI_C2ka30cg;n7Dug1B}1d8t+UEQZ}zk&zU`g_){MX8YKE_6INnRq&X zSe)VHNPU(d;uXRBS$}I5?S1NH+w_wopiJ7E-{CI8iC)OTy~c(D?%4CEM)G@YhrMK! zzSl-muH=0WO`;T+xquC!>j$`J%qA=WpYVQ6Ob%iLYQ*;F`I|BsD(*cC<5 zmH@hJYcJcjZQHhO+qP}nwr$(C@owkz-XF|VG8>gj1|*hS*QNum)9akb6B=@%Bs~z< zWhU9y(3Bp$qlaA6Zf|d{K!{fSqRmw7zlTPBu?Gck`__Eq!JXFBDt7*mMkH5dmAgP= z!V!{XH-}vHZ8ikegr}uXIfd&&}tiDyAY%j{Cl+rfQy9-{px3qzyy zk+l@<9Ue!fetn7QW1n;Oq|kE(}9$q6@R9vwDws&YWx^^<=d$T-3^OV}ba`P7aL8^|DxFT%ir~ZK{ozOu$z1tOl(c|}FR90?5lu|((9!FD4PE`_|W$P{; zX!~IhcM+ZoIq~Ddw+j;W z6QA4vonFGuHYfNIJ1y=;mGRY?WBz1_^U32-3wB^0PU{(i5%d%_iVk7tZJE@*OqggI z$bt+!?j}Ak(2PedM8x0t?hIUx<_k02K)OP+t0}2clVmqrp+CIv`!uoJ6^e}3SkOXA zbp6ktpLZaJ{q@kM9jUZNZcy;CJY@guii=tT(Y?C>USD{#68y3b?-FEdh5yWQDF_Ga zr`DMzeh6J@$OUD}cfS*$z6vN{Y(}n&xdMHTHK?q9wp+EhujTOlL|3U;^%T*KsC0d4 z<^pASv5V8lqHn4w>|CPJTMSxpiTu<3Bcf7fw{9w;jBCut;ht zGfz5@0GKkvuV)`|XX+R-YC2h0c(|HS;S?dp(L6Q!8u}fOHSC5pquLo60Nbmo7^_`8 zd-uGXkb6+j1)@(%NxXn{3g8K{BY{=PR$#)4^AuMr4T@0h?~D~dZKqtS_@6zk?>Q;| zVMY+09;lSxsWVMzYVLe)3G^ZtkIk1fUQcUyM?o1=Ix!xN2!bIb>fAVKt0-K84eOSE zwN|CHDdeFiw%rAo+$9`R+?+yb2%s55F<(T@VDJD9jJY zg;Mc*&`(1r9Ko36C1N@NPfdu?mGKwum$qSXorN8<^R#qYv-pqFelhz=;DJCk-V=)dmz+Ps=7)Z8dM(TQVhaIVdkj$ zXO>EHhK~U1hMzX)8$jg8!4ry3AuUD%LUSspL7$DJ8KtHQyox99d8VUz_ZW;KH>CWE zk8{*eIue`<)Z=2tA2)q;R_+v}9pb1paGUz>_6p0KzXbdI$YG;b9S@Wal?M`N9?=_L z_$vK*6x^0gp1DLW;;R=q~KfHZT^nA<5bygd;+Q zwadTbMmI!P2(8`DgA*->%7IoMeGP%BJ8VVj@J#2%SDx7nK8qhyHbaX@XDT=^*Smqr zzeEclY<6igh4tJBgrj-EuMX^B6`kLVv1L`QWY(IW#TvlR{`dEE+@8l+2@+$eZ3%$x z23pP&8}`Lh{4@-y7>EH)t^x9Qq8fyG(i~S-)XNr^8xQ;CQQ~z zQ!E@CEbbJZfPLSmxRcc6^_2X5Hc-wFG3>&>jK|leZxD8Q2`Tk4bjbHZ+z;>w=&rz- z&+gFzugxi=BJ|2(>`-X?Ag;tINAfD_IP-ZfLm$LoR(yQ2w;to;>C}H6t`1|$5h9}j!x_=^l1&F}ZOu-T)8|T=6P#jJ8u8!8wNY*m`j5G{bV+bim*0GkKx%?tzMp&mb)aK zfrQ`5bOkkyIWykk;|Yv&5%6|(!&%blSmQ?FJsqKU1f95=`){S;Hp#w7u^RdES>wa_ z;S~2J1=0^BdX`b=HPiu#+nnu3C9%Kvu4)5dd$infr?Fp|y?LB;h9PS`;K0rLHF0~2 z*W2dyU%KD5fjH*nI9SM|istU0x2i-PYnp1mQkPXgZN8F$BKq^|g5vedItCDvnh=_R zQE}=tReOMHq8t=fVF0~*Jg{QE_e=Y=h=+`dXHM4wrU~Bi&*|7Rlr62)q$E?$jss|d zUb5#@lIbbPC8?N}zBTpJiRDgJBoj8p;y>Iso%eEyC}Lk0VKaevP=i_!GHJ4p#&;OXY^?jp1g)Jdr|oqkDtc+g zOZKT3ba~401ixq)nO)2xYc`i;{0hLDbbV{$SYf60tD1qlYo;A5fX>+>Bk`0-JlB-# zb3z<}TU*OHb2w96SqrU(ey;C25x^y4iy&^5>htax$-Twifx&^IOx`$o)FGKe%8kpds}~y#W=?+z zDjxKIbuwv%7N4Bnk16Pdp(5lAq3_ml1dC#=wq^h;7BdF|vT~;?LIL;}8n%eWQ&4nh zcyCVqr>0L3yllLiK$3UG7fAanzpl8SkbRG*P+r>j`nKhpFt0D3Q0f^n&XbFHR7JsV zHqPEQY%5eQ-$&ys^KQw-4&_D$&y;u-=(ai8khszgk(q8JfCZEx+{wkeKh$XSL7VuOr%3K~5X9fD!RNJ&oFmSCRr&{BdycpH}lxnZQ( z%{N*(3uGO|R(#%5a))6E`?jzyXP#h%*U%10I|7pB$l9?uLkvo$uDBGix&NH=Q}jIl z*$+Odg=|@Ex6_pLD|QnSM+1W+qt)ijw~{R8&3G66(2kuiPU^inK8Bv!<&tYH)e%eIU zIsqSES0;gKc~pHM!jyV}(52QW?2&^rQ|I>6+c2@9h}!h&n->wiGDM2|I9Hj%cZYoF z8@wt)dw}{)gz6cy7xFk?uJ|hNK!9Q(l^sB7$5H2&1I${Akz3KGZH__pNBjpGr8U77 z1N*?-|7)yg`HeiCgUWiI_{=861*laTbMc<h2Mxnw2tP+cQkj>GPGJ`2^$DH!3qkYOHA=8p6&9j!NN7Nw+3Kt@f<$KHu9i@rMA$CqPfa7)E`4V;9nS)EIa>rkyR;6CMNCXyqG!qv7`aYHH z8?-9I1Dgj|Hk~sB6N#U<$)~*9gr14R_VaDTLM{Yvp@v8riDIe8zgy_6ZF3eM3_@5F=5k4?QiASkda==iaSfpM z60(^(-rV`|10%U3Cn&PL-rIjYIQq7h8Gtif7ZIy%44_e+RFqX~VRd%7wv)t=>D;@q zVX6?DI3jtb0%QyQ&N+8_jg~JlPrLS^!rC5zGv%sZ9o_suXF!L5DIv9Hr2yv$A`J=JbpbJhXsN zuUbcBB-`+ULWiKatDNcDS4utHl89Ml(32jI)I}(o?|`)!hD&wFq0BK9g_KvBS!D#| z5a*@jnvp#L1q9^&ddjq9L>|v%GHTuusVH1>7o2Q!>u2CWMEv`Q+&-K1+*KB30ADgK~kk$^FP# zxAXjS9SpPInNX{D>2dtzGAp$yxzP1&h3EO?K5CbD*C&zCg9HwDIJ{sEfbgRhnUCpD zP2C4-P&N<^WDl`(qh^G==qwUS9J~_ke)BqL0ug%!@$-hkR6v~Peh|Aum~ra$ac4b( z&m5vSt&%=lfgOjeeF-*1f-_H}whSu8BO^kbsA+$b+S?`uavNxvH{5OWyL$*v7BMZ^ zPVZW=@Iwc73*v{#K|DYQY&8ur-Ka%X^V6#F1j?Xw=ybB|H&wdObK*>K%#XyQ=p=PC zKR|uxb%?UZifJ15An@#7nGrcg7QH`b&sunr4)9(Lf0Ia@ypL5v>z0aG9J`%!dvVQ| zDwuj8@%VA*X+3KkWt<>^SRDByH98=vHMp8JE3oTAaU;}d&h{r+rMspX>yew;#jZmS zSLryKa*+~`IPqFZdQ$|{gn@niQcHJ4(V*_`K~)J*Wv&BTl(ep9HJsOD{8m9ORllbaPn^_MmY$7DZ+_wCpAmUqDBr~H^|cOka*EM-~Ax$boV__f^7whw~) z>S>vW8yl6g;k z?^$dYC_!(1ae}IP04zUAZ6v6JHO&}4nUeM5ANG6{HU=wEbek5H2v+pmu(zXR=x*9^ zMKLLE!XfnYqYFMA4}0&DEt$!F>6~?G4TaM1Q9W{*`Q6GH;Ox1zR`a$125R&# z_eyrQK5#w7Yn<%Es(`R=0||mHAq=)|G817D>`QGN-{vgz z)>M&hZ7XEYAbvb_IBTg=N4N^otkjUY|~%|0nDQaV0Q{?8E5e{Q5Y9k`e~FQz6o$e^T; zg{aR-js!JRSM#W7@nb|U`s|*V1gn{Z^sno-+_U~Ft2y}nyRKCSJts^Bf_(81V*8oF zB1kQINPdTusL%iQ5%X^gtx)0?n#i&myK|j~@g&0MD#UV`g%z#J;$=(V!K+%hubc#@ zVOO-nX9?arRax^$$e|qlaemlKiIx&(t%rJSE9i#Z*}MzR&(DNJ%79p+R*9Et<%w04 zLU;bG8Xa z2OlL`nST@`V%sA~;UwHUJ(q+|9$3TlM3wr!C2Dr$Ul${#GdsL zl-7Opx`n-=4+wj&rbk_m#o$}LQ5`yH8xRQ1(4xvxHs53!DK{Z1aJ_2_XRav-3d*~v z1yi?A23oEptgsunR-^14Q2}#-Kf3YoEWpBfcRdVhaNg!;BI&ZUiJuJcSq&CYtD0p% z)zZAMfWv=3)shiGKk1SK5=TpRfsLtjPwi{a`8N}()eKE5F!6HSTK0HoN=9bZpM*o$ zdKw~N>HE|)`VU0x^zAXfo?-pkYdYLxTO6z@R-6Ush!NSfxTL}s15bJkYP>{tVRXo%%K<`j-K$`2#-%bTDMaoCgiavQxwnXr;jv+yr8 z*s7qRvG2(8;|OsHyLo|lq{7Pug_hP}x<7-;bkkwImoUiIX5UhEP68I)g!4TW%T&3) zHupD^_PP5D+N2j4k?I5W7rQ2X{<5DA`*lAzag{(%U|zhm5M0kpg|+Cuav__xhXd+S zFG5rlYw=AWkgmy6Edk~Vsb(sL<*qeTa>sEpr|FqA_2hCIWW>RfroU2`O z-nOgZ0_OW`c-9Lxkt!%S zfL9d=QH&K~iTh0{9#A4cD19)=^ReN!B%E=_`(@xT0mO`#cuE+v0&V=Bkl#p_p!h7) zJ1*XEu;j}AYOG>qy9rMZ4U|pj09HV$zw4?ygRG`5sx6-4QlAQz8duK!erP{fkD;;9 z#{K{wR#55aKTND?Lx;E8)I#@M*-86ghszHeMYoVcr+;0ALC%qisA-Y^Y_n$-&cp0P za+_IP9W4v-vI~Y#F#>bsEY&gOm#@pSa<@_|E|>=Hoes=1Tm*{96ndWrlx;Q#@t$5D z&^&F7U)gG|v*N`3xtU_XHKMb|o>iL3R4@KehD9Bx%*V|JOa2r*GitTl~yx9;Anav z3vgfPLfg)H`pnrB(1Wu{MzPDhOR9>)9JK*H(aWbZDp(L89_(4KMhz-4pS^4KPCM*(T<5u zwHT2`+@%|xQOI;=`Pk6MeeWd?0#&vNvMIS#6s`yt;&&#v!bIo{Qt*Vskt6&7g$Uxa zENyH6r6^S{s{qt#GkK@+iW0%ip&jYXbTsy!=*=HWEOL6=${zN2^h$K20uWm}29Tc- zW`*YH19bAAfGDVpQ#~B+d&!fl1SQqErd+>4P7Aj4c!=H5&Vo=H?Fvr~HoBxds}YHC z5Ef&dDQ(jmVZxf+&R}WRTLywUyVG4iFFrZ_@nf~uM7{#GO8SgthyuGeH8uU!93f`U5Q z6DR;Cw5Z&WhdznZa}}hUtj+Y}8X-!&yCB1*^R&aOB21u{5Xa>rWt6;k4cFbL_p3#Hpf&VEuXz7oII9$$qI7)~q6pUp|WgQtZB;UOnKE$_}g1j?iUAmw; zEgPt(Rrofi&9!pa0e^esL$HnQjZW2wL6B5Ncnoq#cfjm{TR~mUb2f)O^Q>B2yspLh zaq^Co74qQoFvQ;`7Wr^*E59xm9eXN~bj1Uv*QM(Xks(Dvm z`D@qR#nM6J7q`7NxD`50x_-zCdM!ophCYLnHW#M9V>~y+Ta{n1+CtxTxFmdZ>I7-I z8A_4ByEFEa_e~myWVB=9>ZO>Wd>q&VObQ@7_z{;4h3PsdjE**e8@{(;8BV2SViShI z!nvF#TnA_p9^YGzP&iIjqnz9FJDCk?l=J3q(7dJpDwxEQim%UpSrdC~Lru7hcW?xj z-4yb(!!DZZMYldQ93Ff}ZGN>9Ce@N>Z-6w{q8GK_0KiQDw)e4d+n`4c(?GMbvwddy zb*b+ePS`vylDtff$u0)FgnAd?K#>50Nu$8{=&sh)0RH;>wajt6Fz_8+hhXJE zX!@K|ObdfcbGd_tk+`{#a05lu_yjt>1 zazj#DASL!yL*|!Sg>r^e-k8KE{fL$rGeu(Hj&rZnz=glESWTz+07Vk5M^Uo31)s6+G8x+syV=LgpA#c~1bENjH3#P1B2g-MnD zhp`aNMI&jZ3qawYp&qzBrdChwh*Fb-<4$X~&}$UI|r_!U>r z;`H`io;TZoV*kv~_DR8j`fAtz?tsy;167E+E@wy#-C-nA;#2hIIOCx6gNrgf`u=ud zeWF=HM8y%|BSqLDk&!8~8^I*AEIcz5*Jn6^zOht?Nubz4Qf3SpP2mU@0NyR*fwdYL zU>41hTdu5%C?uz^8EaFge&d>3Pb^61EB)h#ChNtNsbTdZSZ>2rhyB!${m2;lEv7Kd z(7!$6MU>C~6Ql8nukaW;d76pTz7z{3ivVRM!InlqPAUxE453tnkQ*NV0!`~#qQic^%WY~6MLjj1c@W)RYF^xJ{DR0BjRWuyq_gy*ub;n`oc2mL%%RSCGdItL> zJra9Gig%8>oszb00Y^A2pLW6H)Lp>QaP-DYThcUw=fSxrXp?YA$)8XDfCuVWb0b@d z$~{zi!b+Z~OSB*1(#?kiPUo&RdX?$YW3wLxI^9|6J-r0dwPAFP^0SI&3h0vnE@NqP zwrUNaJKC7{h-S-^*q85zEkmc)fzHbYjryfTc1{PPRmt3f*N zcIx-<3@(?>8iwR0P`EUBVGr@Wq+_(2`}~G?5aJSRQPQa4Uk>}dVNvj*-ZZT1B$yWM zVgxs{O102Q!0X8BoDs736?ndJA+%sHUM~;fw$L`!6S30q|F<31+|Dw#D@N5Pxeau;hZ;AUR@`kq%%5(nY1{6R7 zjpTEQRCpo=iDJ%c+wM-T3HN-{ZaQ<19r&|yp0NCzm7~loj9K?Lc4Gr%DL*q?vD<-a zW6q4yx*4N3{rHRX#`5#Ei?{2wk=UZNno+274r&Ti#DZtxZ@++uVYt)J5yUGqG}n^a z!~ExHot%T6Gd;>oM=-7&SRX(AFa=LqP6kYdI|VY$9-MwJNAzLl9RX3-=v&i8!9_YU zr03-rAhMAeP5C(5_%PEsGSyrstJ#EK&inFysT>v?-)I&Ia=DhmiK|_X z(+5+xHpwy8T-#H!Fb9(@(6b6q9jM^pK-t{7qXoK_E>4AdLtiv0{Yryy!rSM{;5JII zx)@L+&-HQXDj`P~c~sDE)G9raX}aFVJ)yLy5gbBEl&9jgS1%SHFYngH);*-e?-s;F zOcH!W8&=xPxTq$B60=G`6smC|t1)&iscTY@x=is+%w50M*qrD2>HlS+wcs8^rCD?r z-P3QHy3~$u^o>8;n4ZBYCKU6k z1jU_&21}?BtIwUATHT`QwkM9X0q{)u;RdI0oy)|+YE0PnOnbewBGq={WvBsSuz4kW ze7;-Zh6v4p*>c^g?^dPp$hbDvQQ&Mn6AqdBJM!&#oKpADV1n`@HB67-kx;U7)Hym88g4P z#{v*)zE0w|c?rqyJ!tG4F2d(*e*-sBGa9Of+~Q)hcKgT{Q*bj zC7BWpC#sRnqun@IsVDC9Dm?9ogQm#5v3e!b{hE8hVX{0O6S|?A)Yt2wK3MrE2%1@f zv*s76I?M!`8R}$Q=U$gM0Cv`TBJQlSY4986UWZIL1bEiPk3OjB;!5{M-(EMT7l#0&H6}9) z`V%m*9JkX1uw}U#m3!e2GET75Z!3Pl+LH`fNP?}?ca|&FuDD3+7N7MmTU(Gjgtd1_ zo+ZYxtsXTW!z>Fr)|mpR@~1Rm=N&uCwqg1&z0~0$sQV|y+KwCh?lKW)7`Gtu z7XFmj>GVM)DJBc?B_{yCTuD6B9o_TmkStjOln&D|x%tFUF=`00{;XsJY5qNdAOb>U?!Hlzp(2;_dU_ZlJwKFh=cYThp zpDIfXF6yqY636x8o8*meu%$OrtLn6n8*SHm=ZHh~iD?YWk^ZAm%Z?R+HiL=Gw{rvjs$HSsG31bPL z$7pU*SY=>{;s!r8OcVr2^jFZi6Pxmm3SByf3*)^rWwwVWF7-(Z)yvR%P*-XuXp%g@ z;7=V_wO$khU(!0IEvk~2*6Gp@SpTjm*YrRGw$%rI?14F33Y;2{ILsF(N)1Blsbk|^ zkdi08XI5~I?N4-^6^GK0h`(L9fSCwLxQFY004X%U;q|Xk;b|trMVJvO>5H@_ zI%j4qOM{yYqm7Y@UBqQrsPoxv>I2Ov^*o7YQoa>tHDSJq5TsD2b;Jm8RIFO>*;;1l zrp(@qu{~sOOO{XS>DP24yO%Cek^VyHfcM21!GAuL)O`yq$j)ap80IkK!y-@}I5?^2 zw$0NlQcQt?p}w*?IfRWtdRi^Nc!+!T-dKsS9j?FYOT)Q2DN*Q_w-i^hh91nt*XlP1B>2v~V{QaA zS#IrG2yMn9Hc6F#{wWB~$^19v0y?wJkEWew(qk`gQz!5wuXRl4N6PPs#6Xh_eDt$a zDuv_8MT456x_JO}Wtn7tIECnwO9@_vH?T^rD|5T&QRhE0;)Ym3jGiycXfl7pK{r-o zLSt2ML52x*h;2YnG^`iE?+PWdzX_`o>0avxH`9`R7Os^t{>NX`#$(yCUT$x2vn}#$ z=vYAs4nyNn#YYT{g+b&8=eA-uUXQK#7H-QJ=M_XEXu{bwVU|01g8I?>KCfK zAl-J=-h4lRztwD3h#X(pk#S->+%W`AH~P(XJ8DZx#l9Cxzl3cnlUPAOvI)lYug0$U zUCR4D$qiv^lt~KF*yh$XCOi8B(JB|(UmyCnhWiqy8_RiWa(ntsm~E49AUsZwYqF*f zc$~=YAV^CE`vK5^Bg#Kru^;|&+O)OqRY1Lj;`>I@Q7dJod>MpEtMK=cm%RRGF{>!T0@Wl*e*no z48Hm1Q7|N1Gt-m=)u=7)%p)(S7kJOE&r$hAd_y5t;%7*gY`S>NR zi;aV6)p78Bb`&Qc9#45v%}{5Ai!0|zqmj!G+n=zq{-g2aAbXFav2zb}f%tA&RoYyT z48NwNtL+{rT_J&Fn*3sf8VMBO&~uBVS>Xd8E9W*`_2Z$QNypTBRQhnmttq9LaJB|? z;Nhb0&=cs)7mp{(o#2LXjic3a@pz8|<#{7{{B(_8PKjPgu6rLz)ew8tg}LNI=x>UH z2j=!l+&mkgqZ)CIB0(RZS$UTrXclW}XWd3DY?A<09m#vx z0JCB542ZXPBN^a|^i_o-7fjVESsWd>nO}=C$mlz0oLR*$*yf3$N}L*3kc_%sDd?^E8}mTw)Bor%*z zN3$sx67=@}u@G8Ko`0NlPlIq2n(af)Qz^|eeYj+0^{z!~K8QKC8cDI1&*|b73JIjR zelUXRw7|i%L1~G#N06;>xRbUFfl$nbk^yTab}$Ub+yu+}5ho0zZn8bsjXiOnirv!x z*?v<`UeW9W9|vlh=c0B(P10%){ZmxT@FFq)@*x*6$m)|J^#T;VF)>!Q#GVgDJ9SQ= zA{qbM#1X9xCDSE*5c5PI`Aou`JiLvp?@1oLblRwkHwX9)6`+o()>BFKt{} zJ4|7C^3b7pyt5>8c_M2vsdpO_p7N{DT=K0ilOH3jQ(C6Iejf~#8S8xhm%MW^3q)Ye z@;WAac-0l7t3dBU3Hpg!p0>Ly!{z?|gsGWWbNzbuWN#@20q8upt`+6+ZpI*g29wqH z1cknuEoz*cv`Tgl2auRa@l8GgaUUgVh=%4M>j2fm73~Z6iui*UM+W`lZ&zKS$iEyU2;(1dj`PKJp)@8(6>2G>6CPb&)Q(SXwETqNth^A7Fd0@;^}Nby^K(*K z7sF`Jj=|YQ4y(z3)9h-NIYJ7S$e8TNqF7<>xc+tWDrXR>?Pj{6h+5B+gy?=UJ3guP zwL~#L%})ZB$VwSpS|f`o!a=Hk2T_Hz*6;=Rr10;SS5snkOPW6e7PX%9IEIK3- z3WS`3qaMB)PrMG4sgL!J&oiv_>z>C(u;L$MWV!8*ex+yMC1lpG0Ku*R@5#0tMw9NiR_rPqfuSThG|BA?$_ zGft(&5Nk@*HE2x?!H>4B;$alVS@FHw;6M}{p6Y{FWaz;rd~?CU{0pjEIw3Q;a)F8^ zWiMfTG9!4ag)?E%`11Ki`*VN2-HUE#)3B0&1?;c1|H-aq9*aq;)|85*r@z&PEp}BOr zYE!4dmJFK)=qM|GaU-H{H|-qJri29tMejGwbcSZqt6rc5Ik{WWE8JXAAXdlky=X(= z@2O{}p-F7nZD@Hh@ANjF)@1G0Fw2d7O7wn-&wOcoKk-(_rC3$iowMu5ad@_n+~~d zZCx1L{J+@yHUM(98thTqMJH~m-N2c7l&!{o{Uaq+JvZhd%9;G%Xe`5|#3AG42>E=& z2C%_bD^U($Jj&gMIDw)YThClSq)4|D7P2ie5q>dSuC9vNLGwL5wJLooPZ*|ySZm?% zS0YpO-bIy$T^vRf9gaY{yY9fzQLMbMHWPrcW7tm}LRx1$x6gT5Cg532 zX#&fkB*jpBZBY(>e8@kjbWG}b6p7?y5_OyfzDwuM`V%|D&ZD>tl{SQe7kTk@BN#8eU};uU+anVNn6teyiw)-*$ikZU&ZzL#WU4(TK1K$FJy3H(9_4zx;E&%# zL4O?=Mm}m{E#AWNM*=|9igH_vYc)eAr-?b{=2Lk~!jkn|Rl-arapsK)nFniE%Vi&7 z8crRM#Nga0B0PKB#nqyWDeEo+KNgZFJa=ZmCY15!rUh)87O+0>bo73t(=6C8RMEu{yi6L z3&3ZaNR=}8+nsRlxSpUc;TVqf$=;p30Qi}{DS|e?;=Sbi(j8AG+Sx9~(zQxRrQ}kZ zv=Qpw`r9IAYz2)$qZvT*bC&MH)NY%f=(N?ZN-+3<2m(`UTAEWf6~-hUSo;J1RuMX} zA}D2_BG!BeIZ-u#UlSfGTV68ilNaBX)0{~(R+I<2-EXQRIVsP{M1?9S(bUNXm{}Ve z8m?BC-vP|Je!6E&PlKy}CLn!L*jsx~Zn{>CDk=u&AQhv;buo#fcCj)QWDg324yL99 z!FK7kgi0%f0v!Swtb4h|Mm9^@vy+Y2Ef}+bNs3S4UA@MIed4L#$v57`!@|>30y|P4 zNfbvw;qg981n~4bXZXrW>-1kmBLY9Hd8xo81MRnFr7*?l-lg+ky_MD!D)o&a zzLa%Nm&863H|BD>vE`xqT@^(KNFksE$Za+O0|+Pkg|U#MN^E=xv>b|j`xu4Bhb(5v zDm^gT&zQ1>sewNJKxV%Qaem4!)?W8HgN3z1d1^9ONNZn;k{va0uR!EINR?mL_O^NN zb}=zpnc&qyWe4EIx!)$MZkV-Pc8q*by*V_;#7XQcD_l<(H^;Nh@jP;>-DvF6YTwrC zW+t>5rL{6$T*$A;5QH<6wK-bMssYrbWT3)cABybF1M1+5W{RHC$} z%*SuheHY9ixUts55Al!U*>X6yIMaOKN=$UE=PX+NVvN5&)rA$_kb>|s4+CX*Mc>JT zHb2IO?|jUtzprd3HB(vCr?Ztvxx{`|QZ2LZEA0vAUMSw5y#;4#L}$b+0pE(l>lnvE!=GIlo%OzX73GT zrtxea)_j}FUgMO2MMhsUyH|-akg9DQeasafv>xEelG<77nQ;?oK-Roq`+r>haPS^T zsx)LQ^WQI1mcH&1COF95mA6%a(ceFk8MhY8mNY_wq7fEtrqkvv%lF&2HA7xKTA$RU zTBn?pbmPt}j31Mfky^&Bv@@0tFU8xhk8hb`f&wo||DGkMSvZ~bE1BCYtGn0$+M$khIU^`JkLB&XE>Hx*X;fstMt(H-?#qrCUxQyC>OxSdn z@QBiStYnK_ubv+f2Y1n6ZO3HhOi)UiT*xx~jD?IKuLnKFJ|pASyhqrSXD&Is`q0W# zcvXbT3eiM3>F8(IznbY=W%_797I)~W3aPT=4K4chO5J06&NwE)^^gr1>Y&V8$<40= zagV+_tNP1!ZIBnH-q=FVE%BquFPdy7fqmaA!I^WW)|&WC=CQgt1kcBj3BSunyCga$ z#U>B3^$r+~e_dMV=VIO*Hu_O6MTn{!ItX2d{aWXxq5Zrm{v%k6|_j61?vK zg0;LW#E$g7JKJ9&zpX@3Uj&^vB-(C;RTs<9sP1YwaDqe7)mLmux<@R>oeqw_&DgIc z+Uc}%O#0dVfEf?vjN*2@EjStd30600v&LlwY`GEdf%LUlEj^c-By2EKJi(!p3@(1x z*Y$=!ego?~ll3TkTaC!-L3n2chAjgF&uRht3V4-&(un z$hvtL!)J2egd>pa@Os3V!7kXNS3a>Z*Mmqsk{-yW4yA?0_{g_uAF)Fpq>IX?YkKH( z$&=~!TiVq>Gr*kaMOud{;ykqdO<8wEmL=z|(1C8wuSl*y4;i+Lj5M>7U2Es|xP(n{ z>+)ICTo}SZ4wKeTE%^TeX$qG0x`qCaO0Z2&c1=fYD0+w}JRAD;RyCNbL0)XO0ZMyx z{!ZU>xWJp+vj7{$)zv*_FNOccvR6WH7JEtFdkS!TL>GLe%Ic8Oxdhono^rb?fu37l zHIN1pi0ryASJ%IgtO*uS2Wi%L!(Iky?bUMqRSDexShGd2v1C%6%<-|Ei%;A(Y_J~! zCrPVAh^g-R3KCJuo{DFbcPomFNqRaXAkVMm-f(Bdf~jIXmIs)?eX}H%owCYIax0QT zIu>f=sw3`x|DpUE2?gQ%HF5MOmh)37$B%WfkeGsnx~BTe_j2eaNZMmvW_w9R9bg7& zct-3z1V>v8McZrjUv*w5tJ#@c!q}L~8nyGVSj^>H_G9|w4TgoP5O?QPi6@3yS~pdUcQ&I>~i(33;#$F@>qMl6_LYuv4LG2 zi5>l*q9qWe8}N}vvw0qHz$nmAhg}*?X|O~mP?Bkh1b*zNys|+++l=s9ewmK9iT;Ig zEJge@!4w)hv>+tJHw}nVxsp>N=i3V#E{VfBbx+N`ksN+jnB&#^5_xujc(_a0gswC!_dn&B( zk7pDlT{*@er{%=okCTeSR7QKgr6gOYJhDW5Els12q%!bJY9?-Qw^eSHpO~s;`TfOi zvdK1bL7l$(4V~zErIT08n;Jz+H;UAZ-?l|Jrq>P&!QyeJz7Yaag|zimy}wM>t=6R? zm%fL3{Yab#7y)yqkAYyZX zdzNi^y3^m2a@`#QzXYp_RxA@{$@I~+(CFUCxXPNUipWC5{R5ZB?V$`bmw((YMAf-s z>l31CW+~1+MShgT$?P_432NqBR^x(x7tik5-K?RCYgQ#RDRNtsa;fZdWbFTn+wA%i zM>tB-R+L*3p##H~>gpB>TcD?Rg)liM@T9SN7$})i!-8-wLg=P_x5{O6_&Xi+E)PIx zi-_v3)}eR|E;Q38QjG~wyQ)RQu=B6NWDS(cz;t*`_?Olo-~7@R&+rovJTPW<_U^0S z6!=;*7brKWzp*JH#MAXUVAcbg)6@yzuT}^WZfbL8wyj_^rIfP(==FKV_1J?#JW0wH zL4)Zfgd%s73}}h-2*`NXSFRIwD|iJd=*@h9S7yP5z){ul&+epuiXfoXADyjkHxbE1 z?x}qXMIAzZ4NHTumiRDi6z~QA5*y|hd%3?+{kmdrdjC3*`}9gragvkJu=S!nNZu4L8(P&Hxa}i~- zNPeMG9zL-IWVPm=>nSr?%vYLI?`5=E&E?0ZO1|_$!E=-Mts?1(eUfoiA1y~i;xp-_ z7eLaVkj^G9@O+R4e_%!g72Po^wnu7X4V9L)weX*-Bd)8Ac&%SIy~;?UPk>mtPFvKz z4veBvk@%1YSH^Ol4bfU8v`XcU=n*rHm~#fZ)Ygoo2;a~CBfS}ni`>W6Y_+<%f2a$nX@+A`2Y8 zI)D|F6Ak=SF_0iKeMylpss2QT(cmW4Wu3v*s_bqa?(K?WuN!@&JA+7cv!G(q&fWrf zoi>`6cmgkAvLl#Byyc3%FSyE$q-en*{fdooJh$DUlR=2{$z)t`R4lklW+l##)!2Hs z8IfjLg4GQk;;H`lR9lMQQ9Q#(;U{zbbAfZg9w%?V9f|PX03`I5IWEld35`h$IB;`m)a3ekJj5T6n~d=QE05E4 zD!pPzQYU+)Mod}1yR=RRy7p`IobQzPG^~Gf$UDaXcWOAeZ-h)y%j`wP8d8L~&yUjk zpEjqDdA|1`0fKYbK(Vh2RyJzsJDFPPW0MSs{yz&x0o+4;Q!}aDbI*_uNF-~=zX6A^!j+fC_XRs zO1JAbqD@jrfyJzN92U97Nnt%K>0}`HjR-N}De&9fdsnwymsFi~e2@v$YWz5~j9D*U zCG{sN8I!)lbhZAiykt-0vFf;t@zJ2xIBmXA-q|h#Uo^Rv(6RU5zo+Gq^#C^t9fEn*8JI0vG zJJz0U(-y0zMu{pNyNT@l>cF`B$Uk*_hRET+e|paWv<0Yh~4-C=|#?4Eed6BWOHhpWkh9TZ)9Z(K0XR_baG{3Z3=kW&0Je^ z95=3h*RSZu-P+1D@0U}qt+IUIwH>eR#7V=5^#>oo?q=)ek?iE8 zY@yjC2$BE@fM82DhDJ3u#>RPiq$Xv0}N+Gx}SDg@L-8eLIMN>uP> z$0^ll?_J{*(Frb7f#?Jx<`mJ1G=!sZbxf!sI?14h=%k1mqLVeILF>tZYP4>QQI;s@ z##maCGpaFOp~*HT(4w7ojfn;|o(h2o>d=^^Q4^`)QIi^zxfjvN4mCt4XT9SLt;Eoz zoFPV9T7fgf8S7{jU?F(qzzAniLz8nD)DX8l^>CKh>xuu&_n?7;g3mkXK%Cdq#LS|} z+kjcp=r&>{ke0>3^k#{K1IHY7@E9G;@?dR4BhqZ~@H>Q`FiD33P)HqI0tFWuk6Ac+ z@j@?C0Zm>tK662nB7nFz&v0Sco@NgRobREL)8f+EYq&r3faU+Pdh@L4N$r~Bt0^^uO};F`!dPem$JnM z?&~3+zMi&(>&qnBfj-Hhr=u~j>PH}M(XJ^Yll>AbT44KPz1n}(ZJKwqs{0R{_I3Ap z(+rGy{rgFWTJE=p-M~Dv+iIq-(7uf-hsVAnV! z6|YAolPzXOshRR&J7n^rvrN3-4w)#EE8GrQTQW-NPccdp)vJujm4>J`K9x$2s(nip zb4p)Ov7B$IVu-A^Rk3rpMQgI3IdZ1gBKP9*Dv@mMc9hBO&JZeWMVUggnXz#@%2vl2 zBC4&Bjl}}>AEsi6K`v9bO2w}7hc{KdsoxuI=4};GoAZi@==Uay*xo%pUJIC~f`TPu5UmdlFC+*&9^|8${(a|O*3IL<}JI(fX6Xs1vk*FaJCLXbJl7!3r|AhHM@wMLG z3QTcJh#ECH8oMR3SLA5)cE~>Ync0>y#o05f^Onfu9FsEJA)An9-_!=w`@6OQ+qj=? zV&BW7pxf?rb0MmAM&H)unC+w5bGSu+C{U#%-*(8}^18)8g*X4<(*I;1o^Ie350jQ1Sg zW@k-t%Vwc7zCNeY-!j&zL+DzUZg`Tb^>fs{+v<{27j@~jCpKiqwvoUebw;T--&IHD z`WRdvk88#c4E_>RN6<9w$>IBx_2FLgZdgluzx!>m-@Saa`wK|DLn3cRLD;*H)GRca zVbByjG^NJV34wxC6KAPZQPheTZ7DzseUrncD=;|E4LQzC-$!01$lmE$EU?F_Y0)9o z^b}=YUjgg+3GCNIlkH)t5-w!xP%i6NjwX}oJd%Rv3fiHzV7lyPt_c|np>Q7U$qJ{) zD&&II6G!U26v=P;CVEvy$(0O9SF5QoOd9Ali)-dg1i&>QETSAsj?U-4YA?OGHuB7S z=CHiD_i9wvnOs+pX4A!2OSBP z1sXjB<{*!t408qJguwSeP#oi!w!v&TRt{88CMBV@hBcq)xQJQca)VZY8w<@+N|2jp z=_OEffw_-J2RA6V>!A@mNeMF7nug6yqA6HhJE4iysfOAHtfV}oE=Jx+ytM)nn#4i~ zK15gu$84m1KuSW3A}oR_v6?1UFNK{+2el^g$|9?BA>lD&9E&V-LYLVm7r;+r*gDaf zH)0#pBDe}q%WMyUM}uN|$K#Vwh+xK;0cS%pSg45}st#qKjgb?pCRznQ5}R>4!<1sw ztlDU$(Hh*hB!zKOfWS#zW+k6PZDLQ9E}dNlTNCd;ZO8!m%O92kAp8W6{q^;*r$;m4 zm2>6f1wx9XPo6CttVA{}808F2IV(svdNGV*TvK3eQrsIFrxh*I;w}m=COEG15ME|^ z&^Zyba6Di$iY*bqCrtyA)WrTKHp~~xfwxGotP%{10cJ=CR>HKNRX_R*KPjNy>rrp( z5r9EUWw~g^l=_~k;tkt77GO)M#&R&C1@eeB1vuv*>y^$PaOtYylGxrJRC+eP_nwiZjvHh-Kj7GRrI& z$~KmnhpK#jRo?f~6xo!rfx8%`DYdnXg9<~9$fEQdumuB$(ZFUf6xr0RMYFgpuxJSM zLE~Ol1ksz8Vsl_vV(+Fot`caI_%~JF1ur}7>C#{7>z>7AIbKA8$YS+SBB~3}vq}l9 z>n$5%&w}?+JTt?}j7L@gfyH|ZtZ3GdV}QzuX zC!Rp0HW}`kN4}(&LewhcF4jzRsn&rC)_j$|69BE{h)$p^I5M_X3_b4(E|-n<6|m4` zSoTgB%k}7p-(Z5+hG{l|o+OJTHu*b_FW4*yo=u3$yrDhH5mGSh9&$xV_`v}(VRn0l3XRtp- zQr0qt?R$s78o_qn5jWtBM1GSdvaJQRw-ima2T-{QSJ#CqEoz_L%KNfgRCnRg#&?jWxubGdj zv|)1p;G7V}^Ql0=DeSPC!#{@AB4q503rO18yuS?M(3Ip1$_lA@x0BA1^drJ`(5F3~ z*apYZXK70@E@`x7L#+@3C&kAreAz?su4YBWnShOs7MNwhzKn%6awujj+o5s2m%xgH z-td9pBCJH?Xj}DAw6AV0bhs?72)D@^@S!Gq%nB!uFquQm&uZK;(ix+ZNx67}=1@Wnw3)rn+!g9#j!QgmN}5N4 zF*6K5q>M&}jZ3ZlB~Hk^f^4OH4%S zN-`I0fKn(16lp3>Z+WmdIT(nNCgzk%qA&@ek7OuK$yy*IVmWLvoCSd?HNz1hJ6AYD zTrCM(@VefC0!OaMamE`Y7o6e5m1T-XF{k~Qu8XnA(MqUFDlK@Vj2UJ z;j1frT8nIg^GL@b*_HZz7Fm|E``GC~1E4*M%_>Z|WqIS%Y$Pvz(LSvZ&``l-Mq-gm znRBII1%-~lWCAmcol}q~QFKMewr$(CZQHhOn{RB}wmtL4wrwjfm82?F`APrX@2>8; zwaz|kyM*W&&MU>OH~oe6{-f~_p9Dy}BCR&jh=IA-$k2V5{+9q1HOQ2Id?(ov%CuBr z60(AJP9{-tvR!k_btO5fKU0OdC}l*9aW+JC(3aCd#nf6O2vyFobqSoL6e=01EDy76 z6dk85(cd{Ud8M$;B2@TuFt;-PZgpsbV>^EnO$1X%Z!_L~b~0PM3cfb25KGPp@GwE$SKL zhR)>Q*X%$tmPT+88X;id982k7L^q*e^AgNq_&TB^IMaBbL)V}C*`k=0>2}VKW&aZ0 zt~#LZsvu+p0uGJD=@^0Vm)#)z@Usr#AYm_t3XT1b*Oti}NChW}!J{b&>SUM}wi|F{ zU5v-E6BG+W07uin-_M0zOwLy)CVa?RXM?27F2TYFA1!;Cs<@K=n?nBIhLS^Ubdjov z0@PjoC@!ZRm^=U&ay~iY#~u|LPQ(Y*b?SeWY$WE);}e&|X)J`qwP{o3#T@IpFEHBE zG-W+!9OFzw?{#6s=118W)tGO$$FN$s9Ku6n;ylQxhM* zbNQczJzvq_+lN3>;&nfhg6u)qasm4&I9Qa>$-Q7uxPgq$jb%Y{aJ5RvsJD7?208mx z$_oy5$jK3(tonBjvB#W9;`Sn4VTX!~i-~^E5xGMZ<=z)TV+z!B8F!Ub4ELF^B{YL3 z*^tRZe#9{~tZ4?U)o%afR`eG*Y(3p&hFZj>&4;Ux4&#n*M80TMXqo{jn1fR>d@%Nj zypnF&-=}w4kH@6`CbK6D7;Xy{YmNP{9*K3d#C{zzkLgaonN0dk6$|L#kXL}bWNvi6 z5@-iTr3cmqvX7Np+6MSUp710ju>OAgZ&e5)<4%@SWBDY{jj=$dVEb}R&i|&MDB6c-YaAs5ITj;n2kT)@{?{GrvP_&nW!cU`A}0A7vsm{CgXBJa zfXMjkWFtuDBrl9fQF}b3?Iz_k~GAgd$9O zZ_PuolgA#4FpfdZpjtzgHhe^$9kx9{WaKv?z&TFN6WTzhd=seBP!Z{aZ7wSM<;+iP zjLz>*m=6kg_mh$dhM0!M1MVN83oVaZU@MPE<5l$s1IYReBaKGG69xL#3daoL&fNYu z$sED6=_n|0X_k~+e+n*h;_?HkT}}Zx)}<7cg zX_k}3Xb&I9gpY4fGWgE!Nr?TgN%_PRBU+O1oL!{T|<1!jH9jf_4u& zzvnt7F>hr#8lgS+X7mrpL`P4Mk+k&=Ns$goas=k4(h`OpJ8ILM4qTnA#FGM1N7u~4 zlbU^CX;3iHWHID9aB3gBrby2uzi89U4YlOvIQe!l(a7sLw@VDuUEhBo#W2?Y^jpLv zya@zON$O8V*081jlM_VP#|vRRj#e^69w2Ci=IW$N3F6IG6cqb14GIvhYo>zSY0l zUssLn&v$Rr`~lzIHARC*UqbI&Q}^BbDpCEe|2+IooAXe;j}m=vq)0Y!ep;oU9HvHZ zd2i07Bik}zeC_?+KV^TlV4Tan(jz72r4bLWOPqH{kfkT}J`TEReSK#~@U!r@*=_Z{ z)@#dX+nzW#>fpM`>-62tjbhnsdTD!c{`^dVJSUU!@cupR>e_`yu0^nI`W{;}r&kw$G_L90)TM=|(@&RP3w+atW%wdN*RFw$es9%A z81(Hw_*VaH_Fdt#@3Yr$t+SXQy2;z%jP&W-*2hhBfZwRSj)Olb!{4}h{BiOVG4k)@ z(rydC4u2WCCCBePbesRN|7_506~A`uvQi(@^lev9c#-zR?LT=C&66Sm{#tMY-(bKmry zz6`%Ea{lsDbo`p>x4)`Qyc{h0Bm85P_F?rj{@w}~`M2~uuB zP$3P&O3y*Y63S?7LL+DVrLChEJ+@lRfj-!4VS>3&^1ynOOi z?sLt%Y@3n%U70&@P*P0nWk9W1lhG5u_>wx}pJvZP1J~{1*l`KJe19--^lHrh<-34a z)9X{Q_l`cFGAywcZgV_1yxck1+BEFS#iw~Wo!YL1sa|AW@S?EcuPXrRi)K@u$L3r` z=lA>l+9@h5EacZ;khhL1Ql#@8m|wHSjqOwtACZ!|zK?$4x52mao~BEW84e#YAcE^8 z-Fh&Md#~rs20nM#p0WH*21UgFC*^WtB1aZ-^SL#U`*U*077FC9;m*Es)~D!hZ+VfuPXE`sQO^IGX&K@B$i8{}s8;7lTP@94DPnGsb!VKF z-^cNFe)}Bj@VW4cG?Jh6$rcTBCNit>$gFvEoCc^eVyB-D?u=)uizIG?iy} z+@8<(=i$C+Ox4DNR{0KmW89GH$JpqL{raMwRQ^4w4NANAJ|j((_P6!`twF{q=JGJj zs6fo0?Vhh*EBg6oeg2d5z2s35PQLKkk2j0H3IO>Ro%Z)PmZ;#EmGE@l$6prZd^d=w z(E6}&r;v6|j(%7I+6hI;&2vDj{k+j{|k3B5-_r}F#ZqxW+GtZU}s_epCdEDKSKWBSe%iQgO%<7 ziN%|{OjWXUS6QRl1RN>g2Dbn6ft0J?yFrQS>Ou)WP|H5JQQFEW6ou=_Q_s8o_7)R! zGMy>$zV3L5P)eYvVvI=VP6r~z6)*rp6Eo8b=nkq3?tzR57DTl;w=zF~F)%eTGZW9x zM{owq45+~sI3P11pbvny0aaiE?(6E+W?+6KPyi?nAjp60TEH>5fK&hol3n7(4Wt3E zO7;ga1xIHHu85rK!h&dQ1_QL*(H+M5x0g{vThs4vT+hhx75lWRoq|YUZDe-2cVlD? z%K(@GY=Vi30T=~$bDPf^umK3d0IHR-rU`&U18^Z=8xWQmQi&xXr79_`85?%bTc24~ zR22N^zeH48T1zMjkASYIk_G_4dJd?BWp&|?zZ{tV`aydR7 zNg-X&@U06R0J;a};@0q!^|EVK2B(Epn{IyrodukUGU>KV`z$qE4jk`XYy13N%w zppG7J&%gGGYa#bs2NVdvsBV7R$BFWzeEg+v`P1a082*}izUldy+5Y<(ml2?&+x;VM z|MS{7gCjtI(Mmeh`R(g0s^FYy{#R z@Z0c%Eia;zi{3vo4Qa4%2+q)#tO8_0cVb}wtJeZOxcd^R-S-N4wRbh~^n15~cC-ig z@Z;|YZJp3DvOAmA#l>V9P@SU_$aL^)_zg$+#cl=(0-O#gKo0PCXgccLzSDp00p08! zy02|ub#s0I(ZJB?6zHj?6`)rip<7pGD*&LriTdyAr$5zCTS)&PC_@_?a3AuwuLiWQz^46cA%E`W1BC{S`C@LLYT((mu|df#!9OG}e7 zO9QyluebE?_U!P^#_XfN=1*2A+MRvPQWV_m3z^$*X_H%G^9w*GwRZ4K_0RH}|2~sJ zvinVR0L^v4ZPo*P@x6QY)sE+<-;2gCZzlkZj?HhWmw9mpEyURYkiqefIH=F?~O>#PltX!SNn|M^m@*2F}?% zydr>13<3CE7j*wT>J@aogM-*xR|by`09kV1Uw$NR_<#vOkT3fT%>E;b2fkQ00q8sb zBAVa8A8Y$xWHZBmuZHtCZT$WQ+Rgn9-^vakd6WO(SFikGm-9D#aQe6B;g2m-3ev0ze=i{4b25=fFS={lDrq%B!qAk$n|T z8c_H{%VTpv+jWohuN{AwNph+jpMU&nJ1k^q7ya&n5c6d(oC3t}I)xijT{xv;o& zk+2wgr*kJ*nuH9bqDe@;QCpELJ+A}9uKRgJHGNpAR1V#1UYQosQ`xq(2c;R9CL46C z7X$&9Q|Z!FfML^1urL6lv8{- zfT)oe{&Yh8&zY0X*v`KOU;0YK$qse@d`;u#5xZf0NY%R>nocuV!$URVosoOpQ^{I) zpsiPzA+>B(Ldc+Ct49#ztrDtTeD7(98+*e~Y7~!WRg>&`AiJB6qlg_K&aO>SbsGd-y=AiraXC^W_&O>x0dI{Q8_$C>_OtI zps#g2wLiE`cFgR0yu~uCDW@wYjUc5cNdd62at{Ys+(?H86uumd5@>e z#p;-K*vZK1jm%pp+x*7cxa()Yyp^S;)`!+K1ZRAoL5#M)BwCAUN8Yl2k=HwMEgN2pLDhdFcD?0@m zJVso?iF|`TQOJ?UV>O~kQnr!-8VlYU64_ud#9YN2hA(Dq39lFwY84Q7xQr4z&9NmX zTQ#L2Xm8kE@w4mc^?H1TCrNJ8*Hgb?DsJ>fvLZFaJ0_zl-H1iL_M{SPeeAH$Vs@;mw1t$ujA|%&Sj@jh3k1pZt8rzzF>hH%xn_ z5Pi$w<16P7&2~$RkVRL<_zFWolXcd!4Z5ij7Om5Bu^d+%r9^q7ZGzkh-rIBLGmap*P@-JH) zKjU^ykFIGA;SCu)Fi*c)*J$;Hf}j27!(=LxTCfs5%Kcu-V)W2(tQ0RsA1@cJ~|g1an~?0h&)rrC6n^7y?dN&PyB1REH0^@yJ;36G4m${ubZR z_b_X>eVh+5z6L>MVi(qcw6dZ-tyvJ~lIZOv+ADb?MGI15;hf+gw(r{)Z zeJ~;7nh8iiiP8?Rtj?sP`^ApM0 zwRlm+wQM&uNApuTGE45gEPo|ITL=IWGpbd0j=C<8tZ|2wBiV8ZLT(*ZzAhfcJ<8bW zDf2whnPOlAZajg+GiDMY-Z9qOh!&2(o%dgPquKFfVPL;Rak~vy+{KTg+uikL!pmS8 zY$JI|ix^WzWL0Tj3Yn%hTC%fV-4@DOEp+-6X4EwurXv6>s=6^%>$KsOmzsc5>|r^v zDw*@rK(`6cwl&@PQI`7IO$ETSq_Bp}y z2Mc8p(6Z9?&Aasxyv-?5Y(8FrjJ*DB?wW5P$jjdbr|5D&q$2>fHd6vJ)n3lJt<uN*P$Jqq!k?B+$MF7@@AFyjiA+WfXqr_pD<(f~MQ{x*!`Ls$M z$4iZVbS44ipAFgI7^IXlNW=qHc{})Riz8 z@AZ)Tr=4OD5fSanVN$3p-Mxw>pd= z?a;L(23T3`TlYGJg}4_iiZxh`ahdJxAgdfG_Z@o34lvjgH^=do0{qI+55~0?oF9>_6%pr*%a7Y(e-o#EA zVAf)X>w;(2%K^G2{rU2L{X@=DqA+z)bK-d>!f*r|;SX0}6UQ+S&Z?O@$F$SuSJa5x z56Z_GM+c(m_cjmeAl8-~dEEmM5O|Vm-e3v;X@;`tDY*O*!!DhucEV}{m>+e&je&7g zuO=jeGa0K*@W~`kP zgzsu2 zHtJHx59bMH$*`7vHJiApbzb8y#m0B#(_?jo;R;zS2sSi;wc> zMd94pzV%P~8q!jTsWy45A&#cV=lal4t}jI67qK}8wB0UriXuJSd2suP+$p;M@dDC^ z1R?B9Igg=q;0FGoJ%Zr6R^`nit)9U_yQPor;1q^;=iNcO&qJ2)tn{ zgCBq2&^ZJCTQ+UG4p~2HFRmyv99eTSPm*7@3SwxY}Yh_7IhSCVt} z`9iH?p-k@752l$y?=tdG^ysi$IF|0iX$jsVcNmJ+*mT>&K=c%SYPOl2jOgo?Vh zmU1JbIa(DinxCMM4MQ#V`x&rND~HM2c!K~@^zV2^qrrE|Z?{DTIm@lj0<7IgU5lNN z7M+4ro@Df#r_TSBE#gh395{(Hnx6J-La!mbgU9UTQ^BcMI?%KEI?G5fGP1?bcms zUo8r;(jZM+tiuD#^&9vIw%e@LblZ%5E>Eq#)np z`{Pk$4-*8Tg1>nx0yKf$bDl9hsA`KYuDn?NDQj4^Fd-XjXrAA@|Y5C;=$%cBcU@!P!Jdu+A)mwQ5W zwbGdfLXaW}4Ih1fDt8RXQ8qP`PDC%)!K^a##{Zzq^YX|55x;oX#4PM%sxs7F=?<3e z%0HI3euh-SMeTi7^VGd_qDh)q%}=SHm6+D2u28b-`8HtTrP_FqEs?<&-4wl^PT6!y z@S%3-UG6Ma9x{rTIO%2|Z+dl^RkpkMs5t)G44s8GgN38fr7uooB;nqtFhy4+($J2~ z+mp4)T@k`KF&B%AeqD12p}&-+b5eGz)yw1GK@jC*SETRVhI|B_vtP!pi)kdEqjd|W z<(MM7DPGDdjsc}!mM?2l5iFGRuB13#q4kG zCMRUW-BVD47O(o2Fxw~aYyw4}Z0UVJsmRi3$vJ+67nCiZJ;_ykX{JUd#A zpewj0t!qqw94f^f-=gMpUB*#eiij)?{wXL@IwBYt@G{&A3AL_!fg{JVjT1gz-vh{# zaH$Uo{I}~H>W)C8dTh`OpMX0|im1UEbbo4~pQ4YDQajK2-QVsef&lF^>|yx?ky}Q+ zyfHHgZRR)CuoMj+`@d!lS~!-0`y zeBo8Uhb@Ufw>TrsouFbV@lmKZrU84+=hsEwwS2_y=o*LU0L_MgGmGd!ihUZH2GOn= z!cGq$8Y;h&KFq2;Pg9MaTi(@3V>s%gx$UT>Ess}Q)b-a57?`|>GoIsCnQjsU4QIVR z_J+TH>QJlBD2s^vOU?_i@RoBuc>~U<5RIk*=2ptM{CGlR0XY5jGxf~8!yK?R)N|LY zNhu23Ae=Msp4O~L!|R@WVnjF%)=gg_Bocn#OckM1{7SQ$OE6|6v7SZdv=lLlQ*tS9 z+Zfi^#N*G9wi6#}oCCGOJX6F{OZI3Veh{vBoNDD2Ec+rQIcTcg=Z^-CtTc`{Flder zye+~LD3$mT1*fh*S9h4LeJGDdEiis`ba8qsGb`m*tE@pWc;ZmW3X9<^zucfh%co>h z5Wfs1v>i|NYQx`tRu->-!!rGMu+5k==J5-r{iv?xKU^<(KbiHOt@1~CC0h7sdKeH= zmYg+ViJI6$c#ax@{3%3;9A*hxVxK)ty$z1Bcj0#*zUk-ZQ$&2eXW`O6>8wCKfVMjx zNjnlTgMNQ6oxUbg(mBMtU3IU)3SlOdkdPO&^_d$hMsMbz0=9%w~~Rd*=F z;Gb}#4*yC->1DnA=mEbinL8%XOucQygL1IVA(x;RWOBZX8j*1Fp93GA97L|~Cm-ui zXqx&A?cP?wE3>et^TD|{!l6H>3W!~)|2T9=L-?p{V8nhl7VLKIgyfUiHLl983@zIi z6B>p{sS-Hrl_3756M*}&@uN8Pusf`hp;_whMENu4F(y{%)e$`0qwDQEVJ*(6(;k%B z{`VM|@#zA47FI2y9oN8hz6(pjwY;ncAqUAs8*n(8)t21?InF}WTSaTc#@{qU{yG3g z5|QdpEL%b7%qwm@*J7;BCq+@Y`pQ|8_yrcX(4*z?U+#dtiX{OuSax1PAX?*8}lqpd$hVx{G2vzSi`||OF$sF{cJSsgc2O5s-_vC6{;Jh zJ<>WR^a&=>n)8-&dH2$`_G`%b1Q#>hI5AbvmuQZ4+`9G@t|BZ;bsCstn0v22L`BM!0mQdZi|QU z<4Eayw4>N6a+SL*YSyyMl+xD2=^)9x4|e#luviFy=$3^Pagb7%@m?GF88pt?lS}x z%|(4vnC>*Bk+0R}A;0THZ7^&?yR+_-0l zJ{?bWO7Z2HjRB1)eKA#^N%sha&RWLu-2E`+C>It{B};q&yND1b9%OL5rzW z4|u(p!KjviG=TOCEDL-B_&>8%oYQ+WA>B4gcklEbFYG>@=64&K88afah& z@Y^c=oS`=K7g3Kw-kZ)Raq0|Sp`h0P*gQ{FINW)VFY1IuyNa^l040wpwb6D)HIjye zjtpH(dvPt*t&<-&(J|+;`#7e}%jhwEm?IT|8fF`gkF2cerIs=r(1oK{q)~r(-oJHS0u!NTm|4_NGgdjn zXU7>zOybMD^DW-}X%|`H9bteQKPM~YdnU73NeqDUdy;D%GO#c)y%1jMY8HpAaZF?6Noji}Qt8 z{$(5PsF3r_d%EJ~Lua*ED4qZ`5= zdug4$`fl9|gMzn(8sTIDZ3#FnXzF%lsziLcddiFA+f%aF%xK{}1I6d-B7^@*06+*n zf%>c_hi_qy%VRiW9I^K|)$&&IXQr+uSiqC1oUt8nih|be&1ZOOH0IoE}l>PPUFOR{8ndnZ6g8 zUZ`kQGhw%yxoo;#CxxZF6@&#;tKV5#L}eQaag>W6RgEwtT}3L)IS*#B4}%Q$jFUak zVig)vcq1L__3Ib~yLE9oi#VxrC*@mDKZ%w_*I@b|!P{NUAuiE|Iv8zy$v8DU(XLE)PfT1{vsou8>UWR3=dcmQzC&olUjxj?_3DhK z@gVVOjYskG2VW}15m?W$e+WtfOFB!g+Gk;=qSJ6yN8hiFV7MCssdxc`G9_S(&-PNU z2Lt9$o75;;-p2=ta8zjezfBN%wT7sN7#}1i#a+_8DO&CbhsQ~@-ZZz*y;DT;hah? zT^qK`j@0&qNA{)}&w7mm*TebpFL8ZUI~M=rOxm!2`4|LThL$v^X64%6ljv8Xlk=DK z8Tx+}3-w0tm)4BwcTHALY|VY(M`#^;E%w80F|C{hQUgwpW-7&9M&O=U0j(B?=ARzw z*$N4Yko7EPdsF_B=l5laLAD2qx*nQ;Bc8I^_pyILO)Af{o#ndV@1nJ}E9G2M#Te^+ zzE1<%nSy067ADfp?K>(jL_@7&kAU(~&s2tdRUF4+H1>{7sT(_Ai=rSGn@=J|C{mHl zUBIs-B5;(}^T@w?)r%b-kWt{Xs%{!I@McF}K`YkzI3x-&(&hS&*{*6jgE8BId|Qt? z^j#7xP)IX?!aRm)TyQRv*lILdkFcPyGomPTFy1)yA3U;Hwh_xQ^E55|(F2y}wUGzb z$HGh7k8MRBSnMmthzf|)Eou}bdLTfojDmTF0g_^OIZ&@9eAgYeT=lUT>nJ<#{L@j< z^@8q2&OR~GH4z+4>j8 zmLda;&+#<4<)s&Pw~J!qyu%rk=^o zthjaN%@C{8na>nF+P|(`G@LFVQ4olK@9*=7!4@3m=k!a+d=!0je&0`DM~yM!QC2Tz zp*61j-AQ*R!erT|Ds|Z#L2B<298b1$q@~I~C=SbKRc)_fZG)w6TAi#zD)lBBB}CTa z%c^YDXl}VmwfIwxMKAM}?^^vT*pj1^Eu$*D0}5tIYcz1QTZssyvz&S8XT*B?eXeMG z?F0=tt#!G)4e?7u%4C;rag3Htbw~*FxucXkBGK61XY6A;P_l@@0Cz~yk|c5ezUYb&!847zu+_C3zr@bXFl!DG ztw_VeT5OsR2?%NfCW7>%adB71QUP)wQ%Uffgnw>eth2Q3@B)*Q+nXK2mQ!fZcbih~ zxlL=3)|ouPH~N(@(wN_eqiwa#*>5XsIl(g^CU@oY5Uht=5~N4wX)F4-hMKlzdl9{Z zO_Is6$(0M~(ZUz6kmG=*MQ8>-7)`bE-l6bajZqA+9(#l1)`zvR>YWRs4OV<6RacqS zgBZJ8_S_gxiHYBQ1rtAG`(pr-T>eSqc^Ehd>IkZZMTTe_cl}9w{PxF7d#hONku5u? z?K~?m5G~ps1vL_-;E!jaLf;S} zkOED}ym=RcTE%)g2jvHbE_6{laL1B;Zivm?OLgcP*b(v{+S&ZZL zHyi_R(wCkPRvK2ZTUf>R)6S!B+nYVEgdx$~7l8}O<20z24XY)KfSKm2ugMs*5??GH z)fQ9j3a+p}B=T?M@n=kH^n_6p|L%*)bAoWb>z>et3>G)b2ozsvm*jNe7xE?Vfx8T6 zN)n1*+MY&!Zn@L7q2!&XkJvGjAMRsf5lRL|ih;ow!>-qA1vzvTs zG8B-Nl*<3we85k%$k=8B5W35DME+hcmP{X40gWL5ix+4S8lf(BWdcvySnmZ|C~MXEs9PS_B#@_$9hZVNvgbfQGB(t)-M@-}7xGk^Fpka`=qy z%~xa!Jovn6v-3M9ZrnIpln+Z2z*Cb3P#x8bR@z#FpTS49$(f$SwVI#7mrdy7Jjqf_ z>h=VZTfLmzn6fSxGHh5sBzW9jk+zibN~JheF4|)1&HF;k3%Di4Z9NvPuezID8}nmm zzeJ(X_D9L7E_QhpG|qd3jV~!{&6l+WEuM&WTNBKpA*aJ)Vbu})KPKh;&aW|LCX6JmHkb9h^*ll zFBYgzRIO9gOJxZ08n8!H#1Y9B;X$6&E?yxvL08uPTAqP;GlcA|$V)D+65k4Dp@=98 z!^PQ$4S{6Hf1ZLm+p-J{-?y3Sor}d}VZsn7oL9pppF4{sxlhp$q<~#$mpv5MC7+;y zj^B=&^=8>XKVeo>nt}G-VyO^1)bkqTmy(c}&04-ZfmV=tEGl-=OZUp#(zsJ@JQ$@| z2f4(X^k)+Y(PqjWV0cS+$Em3h&QG<#g&d1fg-I@h2AchIezu-Db}1woF1BNIXc792 z`RVCIt$gt+sv-=%S^IXt9LbQbNXUv03|U_KG7kLur%7`LIzA6WF#D^7zg}y=_@SVt z*{c9jw|+E*|5I@gNvSNnAjBENBO?9J&5SpAGDEkT^V`}vA&mADNL|~?6s1CN&qlWe zRYVC)&f5DIHaby zeSu8TUj!~5;OaVT^0O7-GasZxM2Y*kqO(&LE2dbBnPE5B@~V0VML~1rbIBgSRZ zo+Jj|zXKpr6n{zkg$v22c_ZeMDa+IsOy9$=Nn~VZuG^pYnlT@HzMhmv9W(8`fw<3A zROWC=VgGV6jG$E zu33X=?E+JfaNRspR&p7^=5@LDQMAQ^8eMZ+CqV}0^=r0Stq_ui$cZD~@*DYR`q#Sf z((>}eZUpE}lfdUj+~=k`AJzY@Ch;nup z=1Se|v}ihJ9ic61QEpFiTx-pljz|Es0a3=nCUoAb>orgO$zgbP6~+>gzr9stdbjm7 zv%sK4qaE0euQiKJBv>(Y*~Gy%LP$N;#G7H87~gA!$*=d$Q&fIBOn*~+FA@_0E%@jA z=~}}(^9nn2@x3E`5y7oY6Nl1wOQ&I8Ng$7yROYxQ2P%acm}^)=QGf0wuorWkU8IeLwzo`Bl)H|u zk}y~MM8encC>`_57#(#1Z`Sz)ap!Mc7AMj^Exu^?@@s)KTQIl!V#v7_1jzaJN#(`q zBhe~^XQMO?>5woBtdT|B;S!h~omG6szyLf^kB{Ap2Q%P(ODevk(&On$I57$%qM;Or z76f6&2SBFIPafX!@{fjb|0||eSJQkvXMfkdgGx+yKB;B*$ppR*(-FArTRGKV@(5m{))%HF!*A-6QgcEP7+gL1}KqMvx^RS!G6 zKlA&XQZymgPIQU#3D`8xPhParRv<>)F0yY?&K(dds>(dazRIl0+dpjocs ze{xSnfyhdNW_K?z9OZG6qWA@0I!G=!0gkb~FVV{H&9oz_g;V^`)TKCOLE!64x-pjy zRTk57g6&HH9PprzDUX`;LV|3Ef}0CkJUdjp1^Dp-3tAMC>Sj=CC-)GcA58>+~|bY zHsRo8AoX|C+*51LDn<0KAB!V3o(d3@I3K0~Q7$TG`&V9c zEy`Y&WmK<4G-&H*)+%xl&xfcSNkr}eEZ}(IQ?+!3!SjVsu|l9YF)i+cp0Mcmz@yHv ziuPfAKS!9MX7U>%YR>`9CdTC*mYCQ%pUIMSA+zgR;5W|p)oFgGCsm-3!$}y>NCh=|6e}+u89NG zTGAWjE;y;(a%dQ;cl)$hgyx{SMoc|{ph_-Hp<#C}ukvTsrTK0TojTs4iQcO|BfiCN znrhhk^Rh%LfYwKXc6w$Sky>+fYT`LP93~Z3uLDtSqe49lt?qPi8%i;ZzY+KnY4>}_mS#EmBDl7O@SnZTG$ffAm0OFC@ zIXO{<2}Il5#V&_PTO*D>l?e!6*Ai-b`{?oUCzgZqv~vT84JoO$vGYo=-A)o06WAY3 z5yz9cobhskW%;w*Mi}Ho1md^ICoGlT?KsNL-Ih8>hU#Ds(6*u7@=RU61EmygkD5y} z`^gUXuUC0-qg@dlQnjPHO7_d)N`h^29~*)_M5XsG55wUS&KMvfb#~Gh@+;&;o#h5- zF9OUeFKn`_73_`PoF^@nc#XXU2KjvS<3d?I`h1YV!Uskw0muf1mVVQ-l=*)@EuMXZ zk|@OLA!i=*qB=B};|au?fAl;CtKiW9T+SLp%Z9Ff@~Z>pL@kK}2C>Q#QEDB7UBm>J z!))e*4oa|bk%;7u>im>pTSAz{OI`h3bRM?F@#9nYxOxpgO*Tp1;{rAbKc?harKk7x z_lK@y=YxEC2`Bpe&|zNMeWf+Zhj+iJWGLY}p)RT)QYDJMJ{p*Hb_0&jIQEO_m z=5oLw%XXJ3K81&70|KUas#exW0?{|!&Q4RslEz4p{SDK)VNYRnhIsRaeUmm0L#GCV z_wle`b%sV;$4@i?l zNG0moBDW*Q3#CAZQ?pQp{KXRd+XmouBagq*?r4d&r@c^TOf3{)wt(oD?}pk-9(l|6Wt%6LMe z>=sozU4BA;INqGy+QPd^vmxzqfbeV9^ZZ|1l$4ey`yHWubG{mlah?aQvtSV=>eKlYiysTX^FQ3m8~Qt+{ zUWK7vnic#67C22dC469jtc-o)`KJ>(nDFx3LqvKIe0=W5Hs%@6*d%aqjL382MN6(( zeEV|GP~3Ouuo$KE!ODLdF%-&74eX&J@i%!~rW+FaG~4JAe#1^n>_nSrzv)Rpx>S1} z%l{JA5Fx&$qnF1;7Q1vakR4P<#{c98FCZqCAoDXe=LkhM&$v4w>5EG$69oQqeNv$Z zN8yC5+0fHPe?2dArv2T6W33XX#Ym2Qs7+Q)14nQcH9THe?u zO}{Oat86Cnya-;=-uW zuWr{dN7*|spSauK z1%zYWr9>-G^Fkv}z?N&dCaKl7zABog)2sQ2g&W7p;U4Zwa|rBo)k>LBxRVF8l{uxrIfzO<&8L8Biy1xuM@P3 z4)d(Hwvt&C3*6M7kp)NiH&wwi(Lqc9%NT422y{;NtQN0x_SMaY;4(yX{874vVyLlr`6;sprXmgW?>!m=P&!o)smofwjv z3j&(|=$lE{vEOoYjSrk4IN%QV>(qIPw;R-56)&F}86U zJ5bm^nZ8|6S!c7MA z7HbD7TzoY6x8q_zBtTPb=@dBNi~5k zNIsErpevyLM%KnsT{=yZH+u-0_1Z)y4+C(K&J2Oq?oNi|XnJC&2LWV(@l8s|Sbcg+ z5MWZZ8wgkpcSH1F%ouXdo`Beg?J|FF9Jhv+k!hs2qMF22l`mXN4$PCnE4fmA6RSFB zj|DN~SO7#|7xP!|&+T64e9pouHP5nM0yq3z#;e$k0tZryU$rz=PF#zd8#R8?PE;s* zGvo{gdg4q0hZxFE_uIGkP)5;!f!f{zS&_|-{QjV=tCq)B$vJX&(UAMYd!|j+`z{W> ziR%ZyVo_k|j->(|MFN=JV}5X30?tMQyNObnDmlq#<*jY5C*f%V5YqnMys)^>#X2c` zR-`s-rMa*8$UK-A!MJ-ok^XLRcYlF5luh@YjTZKz7UJNdDq`kxq?j#5J{XaaIlm-b z{?fz~!Y`Mpgz~k=8e>Wnx(OpP7gm>;rzEcIQU`4?rNw(;;*)i%Vi|P~ha1l-+Ytdh zG4HwH2;Czqd;G(C+$kFm^b)6G;tzj)1`?wwln(Io5{pFngmQ@xk ztEYj~1=vOA2pqZXnYz*UKijWeC3eetlBavpZK=H4n}HZY`3tTb$C0Y6MMcO;u;j&X ze$VTu%g}}$5YNZxRa6V1S4imT&$PyfZBn8V^2j~u$ukN!QZ41*MU7rhVsh*W;G^I9 zM$@Lhj5HJ0pgLDAgwL+iEd0>#lco3;=Ce1fhtd}7fLD!tk{yyDMdy?eXDq~i-*u9j zaQzy30CMwyDw;-6^dAr?uK%NrBiizQQhI41-k0=o%%WVh?F$ zB4{#KG_>LzCyIlSJNHl}#+u$5UXIUK#Sx??Kn-zl^EW$fKD;jIAcTP;*zlnqUy9P(8 zx4b1On0z(Z8o2jOyHoS$jb-#3bVgVR@nOTyMhY8+Zxv>VZmpo% zzDPgpp3r|;HV6|~ocPc0ez??Ipb8Gt3jR~(ZrO|sM86U*s5N_IcV)x8FMg0}sj*>` zrM?`r&tP1{XIEOhM;g2l>&FGzScPaVM1~6pqg)X0(me8amgGZ%QCqfgpt0d_eXF{R zN)*0f>Xn}ar)kyzXV*+TzcZ`K9^jENj!c0_dAGjRpCUaX#Ey9)@3Lq%`0kUCzB@vr z-g)T$MHCGfG(%}eKGo@snb=(4NBWe$yZOg4`mgMYt#W4@WSA-*S{5$z7XGI|Db~?L z3q}VuEUwxJa(ge^XCx3LnI9rjccU`=yX%LNoQDke)@vyyj@tQP{Xi7`b@&Zac`a4`kv|yc?z@+onoj7J&OZlpL`;Qtks$;Ygp+y0FsVC;5E(gjaIqWE_RVvOIAM zN9_CceL_z7$@W4$77s-Rc#P`<$?(K@&)25Qi~T7kjejwm!ZjAN%>#*sSc%V1<>hnc zIWvr&zTod$@eNr`@(3`^2835e3@-r$6io8+m-c{~XS!0@NXk%l?=pOQs z7IB0kg4EkJsVvH_4So0oo5rCo^^Q|#X8q;RVJ&4+SNTxX+fhDOUG4sg7KHY_-gbZ( zz0zWF>=Bnqw#af;-u^(vVcMQTTUT_u9VQ;kl$_q#7c7jCVdPI8T*fcugO7jbjeI=5 z<2;5?L+b1WvM2nTcV6h3U`gYHa{4O|9xM3f6?55(W+Rh%n^hPo!v>7_ov*G*o~{`9 z&(uA7W$(E%uX>wWu*%ZR9YiJtI*hW1EAXL)1Q%yK4x~9Ytqzw70#3I2ct}u0Y;1?5 zORYir@^iz$&L;%&>|zWKwVZyia+N!G<3jwC_gwU-8G1GgKbm_rT{oWqlx9sdz0;+v zn&R($Q%Kvj=usFR61yz|Lz{YIW=dx34|N?LJqrTkQyo$@$Ah!D!nJ%V?Q6Ekja4af zOVbC+e7B?PNg7}vN}5#q?zgT0Jlped-1Z+9dPPz73Wqi5-Lm}fJ&6m#oR&r~VMbs* zDYw{f+Ty2$`D5u2X*`1&C%MLSrSk{2yR28v#s^q1(CQi}N6oFISFceqHTsTi<9BnH z02dZ^Aj<=40$R(%&W=jcg?HPuAI}`(X2n{N;_?###Qf7eDhjbNUtvd?8inEaY ziAlvo9gBnh2uc@tZeZGTpXs8!z1pMYM_a&+KmS5u%DE-BPVG{xLL3viH?zz@@9PcA zV!ureJ@i?R%ny&tS$K!n%R(an2-4}Sp%WJ;voVGD*PE_5&!lh;@w^2l%0BgKX`&mb zGJ`kkVY2z(MpL-|!2hAKy$IEypUdJ-iizNiC*=HLc&g_io~K*%vKr6o9;*pnmh0i{ z#bjc~JF$H1f_N*=_z5MXD0-2Ou-J3{BDn z%R1nr2u%2K>=E``a|WJ};edRsm@pnC2#w`Iw#{**@x=(vy>zdlN$I))sZNeTq$ZmR z^3P90U2-cA2ry(K!XNatubhD%5&P+CdFc-pU3m=}hiQWY$tNwraMyYE!G*iY)I9QU zD>qHevNeq6uzEOUub0~Xh4xqD1~8==3Qcr`r{$${v|=g@0*SO-=u*%B;sh8rR7AyF z?1Tf8%Z8-ltBC~2?VIob@JYUT&R{bj?42QAgu^Q7Azr-`-A^HPw1+2?2J8(w=&@l% z#l<2f{^{LCy~Y@rrc^ITu_T96Nnj(42U5F=Qz*yau=u#XpAxab1RfI=zY;(n?TI!^ z$gr*pzp#-#x9Wt504w6Mv^m<0quej^3}^A1*_dk_6%Z<_;klJ&u+v+8jW=(%74}Fg zK1ob-EMB*2!^|_XROs%sN*%#n z_l>l}o68&BI|d!+^yC#-J-Yocej=%rh@oI{C$bbl81cuI;7uN9Wp!-t1nersfOSop zGSV%A(V``E^l>h8i&6IrE)ZJlf%}{!Sn7&65LwsjlJvI*+)`HyHX{ei*K%iLW5~Q5 znvdb08Fh`#WjJ0Wo4lVX1 z>t{Ojs+>!IN)3DsW9EHDl-TBaQ7vU$fQJKz;dOU&_~G32xrK78kMS_Fw`s3x(KInt zC&*S8>tCmX3pGO8Fxp4jgI&)^qvr$X^3WD|jnzy4fFzhcawCPiw`9bOSg@~{Fp`rA z!!J>x(&R%If?Ab7@zLboGA=YoreOkida!*bk_s^-s?I|epKD7XD_U1DXx5x(j1+=W zhuU*_kW>@|uH%Gz(E_OE;hTb}rrpy@g1*0)B%nNtLM@Qi`E9LFg99u3{25GNl4D0) z+0mlJB%Bz@E&3oguHn=~dGk7)HE}WNkNn;JuxgXj9S6 zU@MrknGsf;zOzpj=+e{PP&IaYRz2Y;mqSh9m=n($pfiHjdmQimR)`aKz*epS##TJv zRD=yAkwPNX)r3k7fn5!(&(c)NHQ|BcnX)-RLZ}f~mJVbY#luP?W!nnq;qUUVkvq<* zsTR@js5?``3~zhqf6}$v>RQ;71oX8~jpe3w{0H0NWUh`oqA9C}Dy}?j$%%D-`rmmc zCO&M)Kpq!eV6EV)unGa#2D>ATuw%>VN?-IdffSBKN%ne&HJvM~_$^l>ih@H0 zLAJ%`vUwmZvck=C1tHv2w2FeaItj=c*;`m#uN)4>I5_}9l&x^`6Ch8>+FtZex$fZV z5vk0Kw5ePd^@29!h|=ZSbw6#8{)3rDgFP4veC7`y_yp{vh=QD(nQV7u_2KvY z*_gmDbR0Z#(ycI|-Sf9dAVkmqAUMLO(TLW;a=hQ6oir+U>PhMP+CNBe-C06-TJXKB zNx7}8GcHH-<_PI}SRP03$c$T7O-f@n^ydQ<|F8bGpK?9-kb7#4F?F#zbs;5Mz3a)E zfWCHK(nUqjSb19&*3C|7iLV~4X*Vka3p(VP_f69>sHgRLU4vpH8ru-xBdDzE|AydL z=o$WB5F9JZ|Fod|2f;D1GyKoz{{g`-BfCBtuY-O1J!@&6f&aA3$_27+w<}rV?$pf;p(xjuZcDDHAgHRyF zUQ&p_TVCv%8tR;X6$1mq&t;2w)(WQn)eRM}>oXIhQ2H>8z~hWm^uQH)O_Vr5nRHh+P! zU$7bb{e--vSlmFiery2pTbx~6&x{NloSckW?HwFV!MHT(x3+#q%4{u8K9Gu+OdTS0!S`txJ}$F(wm1Z4a9Bog=iIBs^!y`?;xxc>O`?S$F?DlES0 z?ER4u-tlaxV{(2^C#k5WfUl!#_-Fa!8OEP!<~P8bN;r z?w-x&s`>qFpa>%w6Ok-0r627Eff*?L_tpU!xVpIhVt$qWHv|XYKlBOMKRN}e?}Jynxz#mJ^e>-cWO9S2-xUo4>%eA zr}mr$`Os&B#%~5d1#k`k(3z##_-*l~M&mba?Kf@5{O-OHbOU%g+A7PBN9DA?ge|--0{P39<`=?FN=m0!jEuFI${L4xm?pHPCKRp1j>zAQ+C6iFfil3i^>>qSf2mL-e$amkX@CQ!gFCwO(B&C&g;3sivmku(HVr>fA z2vG0j1AxKBiRFdg*n8oh69v6L@=9vKl>eny0HB^Opnuke;@=b11mBNy9sX{{#Nh!* zt&eaY_i&dPh1!334ffmhE5|+QH~LVwsQ!n&@0X+Gw`MsTtB;+ZnzFHl`NJoO5}kuv zv;CsJXFT|7>gV*2+0R6&Du&e5tS}GkKn`2 zwQG+gevsbn9Am;DSyS?F5_QhQ3GYf>)xsFR+TRj{t)X$p7+74)dVU%4S++FJ3OIn3 zDi#zxNg)I4C0f8zsrrx-CA6<=Lh5o=eUta*lA6-xj?mOOjy_`Mt7%XI{_5C&AeCi+ zz5%F~q_j=#9N$eHb7o8>aOy=Tr!aN`bvaIyrmbrAp1p~QTH&{|a=v8oFnU^(0vZ|M zzq(VF;9*B(3}qxfIPJ9HC}NRMOY3DY)4#}hao`rP&Q$X@9ps0nR1d3ERRM?t3LV3q zbE#&$T?5vOK4?2`!!%G9c%fB|Ip>cI7^wMa`NuWt`#{@9$D1UQ>pGrKdxL928}B*I zuwZt=BJbdm_+gqfGDgYFb-3n;>fXeE>@>OBeyI!;=xuEXPi-1eQuaJTGFyVmLCIJb z3C9NyQROXIe2f#E)s|twG3-GLvC>AO`ZZ70TjKg9R=pR5=a_{r_uRk1U?D$y@WzwIMzf?s7i7x78YQlF%~my76wIgxDdH425iQj7GR2v+VOtUrLv7mB2LyL0oekZ@${2mGa2Iy# zj2@RJ-d#d50gSO!bH4Bx?D_#frOsWg`u>Tgv)Vvg}5Al8&5EX&OVuZsMT$IGk! zM;7Zoh9A1+@?{m&7W}Sh@=($zO1w4}Z(|C1-WM%Z#!R2};1||M@Y)WWgp(h>>W|n! zU{EmdOlL=+iU%+LU}i?^wp}Rb9Aa{HUo`hc@Q{p!vt}AXB;On^b@_4a^4Kf1(^b^a zgI;l?70kT8nnswp>y#7_`DW@u*v&}O$MF0R+;sjMJ-e=*tgYtO@%Mo78KT6PKu^=H<^K6pCPS4B2`7aStPJ0-J2~d6p?YWp?JL#Md>3zW622- z?HAeTC!5Vc`EaSwE;w8&k$cBUeHc}p>kPKPA=$=B@y zpmEU5>+dLDHe_Q(-C8YK09gGvYk5V4aF)()^4Hv3B-ED{Wo*=#%NXqiMR0b_bL|w2 zL}>E)V=vo)&CVQ;w95xy;S|(wZo{|P!%zsR;0}W!cnH&6?E%jq06Digw!8AN2KIdC zMgSgHe}`Yrj*-N8Y`K(#R*Qs8(cg9kyXdT*I?g>0+Tf?V$_EqImhkMqX+QgGt*6wS z)^14`wv3>uYKO5$XGY0QW};a)S;n4^pGhuF9f8_vg(hoxSdgNrEO}tRHerbDv)C9T zPowoYN0#2=n5AFtZJZ{dfUR<6yvjpg(Ki?y7c|C>G^P3JwFugV|Je7}N3)^L&2z08 zi+p`rM01&Vwrl@$#`@kV_CKo9r`3U&Jd0hwmQ~a9M9G8{#^{IVcg?cAWdUCatmB*GTOaM2JdEDp4rYm|py~A%4^+Y^zo|<{ z8BvS+zc<7eVQ^=}Pq3`UHX~ht*__&I?s(B7jlW(uEJvD$P!O+mCAblp0^ps?Jbjq6 zlp^1OIJhG(x6j$r{}l&x6J3_thQbQET_mNbF*J4NeQ~9rLy;LyitNgWw+!VP6S;km zEyA&XxbLU9;MXRq83RBe;+=8gYGNb z8CG16Gaq`Q503sL*SGw;RIM#zLCpM+C@omawfh?JT%Htg;4((nvqy;`T~9|~GBv8j zjD0Sj;1niYN`6~t8G`+8r;0)fE$*JjrP{nILP<-A2ant?Wq@#Wl2(~vPG3RV4A<1u z#+7YWp5w8GKo&-$dI^hMBJycl5FjZ>+4}4t&weoz%N%Q(q))58Z-7mpC}+=#IV>Q-^%pEvNbs3 ze~A~dYWJbW(6(tHrf_(Bq0hwX3ALp3XA-%49-glUj|cPo8%;YG?yRbXO6*9dL`f zr?@rOpA;FXU{(USU^``^LJu-ND#BH)TuV{hK`gS@HNgbrEvlQpwWAdLuKyNTNB=(N z!1!?o#+9`C=Xts}sGgp$3~4S74q|;zF50J6p8G&`{9lxBWvCcb5^T983q!hCL%TnX zeI=U}P!hj(BOv?dA8;CUVPI2D2C%>YJ;Or5=N(nf#1OMY{$9YHQ6k=Xr2DqJVG&)< zC;^U@*CF*04B9`W>qDcXB+wL_&iUQ8y=J zBPpehO_-UuM{*f@GV|3p1P6-y8j``&=MSH{W*0nKUseKlq&$9;97gVBHtA56YD}Ad zLY~eRf@9(>(@ecrE-b0?pqE8Xdkxl!B2?paF0GtywB3Hf?4v2t(O%y`a_<=6_AS0V zD9}NN-9Z0>Zl$H>`OpZ8#TF|@C7-`~uk!RG7*e9xqWJ8se6PN*g=hGrZq*XUx_b8seObtC!rG=W`K4CQCROi9;kA7O=Ab z9fU&+B)2(g&;?~WHLx~q5u(CTUO}m#$AR5ZxFw;R-IX(tc&eT&vkF%Tk)E>6*R+9b z;dk^Eq^pehudUgs`G0RGYoM4nPg#4X5xXrYy=nxqN^73zOsPz;>1MO+`|`Y} z5wa-S)~_uD9RC@2c!^?@36=2j&)g+UqxvQneq9Rc5Z_MMv4(WBSGa4(@em&-8f|b1 zj^$@Bxn;pgY-D=>sjIB2sjeA|pQ?q3`Hcv06(JKUogbw&!H;><#QKu3Cchtz5fK6hfX#W& z>|4b*DdipU4u*XIn%RvU#lK7>S3C2u4`zEGPwBZFBfJK#EfjiQ@G>Wfj~rAOlUG;e z^`yO&1UI!{_A>ct>H7P)q{O8qjg?dWj$-p7duQlAM2Jc-Q6y^`zcV}L><)b6FT6Jq zF03A}P=~h``9n6Lxu7*Z6AqA2&V=$we^j7!5UQPCTTZ^pGQON#14;i|8_6tA1qN1w z9q^0PMlHQt>2XTWf%I=A`$a5fy}m@o;s3Y775Q6@Za zj!@a9zO}>e#WR~OEe|5hzG4vjYI08@5!Sh^Mi^opnNLUDjv}fU^w&BE)yl;a( z*9}s}9H3kRJ=#)vX!50=Pxw8R>{ZAo5|uaNQ-UM{K4wC7@$LuF5mbRm9kX)Q%Qmw_ zB`7G0kRS^b$XP#D z${1i{1%A854VuDx+2QuY5tk$O48b=I1QTf8?E_)czvY(1U2=-WA{%5+hP)sc9g0pc zLaksNwb=S}_amn-F$gnBh?W9H$6zDogDg|iBR0{jwdg<^+~QLHCVL4Ko+N_#syZ*9 zW1Z>0o7NG65{ThsvPJvQTnHH#oHjke?~$pj@7iQ_%f4nBHg?BNGJ{U|oreM=r1xl% zdUtz~2SKhU)ESSq{L_g*MyP*1#p|=B?ak*wpro`-Jlq{&k|Xgj6%7;W{b(Q0T6m|x z2R3!kys>Y9E-7t4JBl%IjN?K^ff_wdXww%O&FX3$mIapdP(-P77i8V0I6j6l9pE2+ zLYZAH_1CHd%cCweE^Z99bj(g7aEO2j_1t~wT z&LBm3Eep{ZYhS!uD#Ck}Nizz`pvSd+7UiwK1Sz)OO=jy7QM^9H(G5|!^dV@!`JjS0 zHTuVs;gS+*g8mT7r2845JY6LSyuA6oNvL2j#7enB^2^bn~m8!dfOPUR;aE8pPnpR4bp3Bm@F-IH)zI$?cPbef!#GK4RJ_A$H zw?7iI31y@!Ij8A1iB&Y_ZIu;_qJD4y#oRuHyfebC9k`aA(wQ zMW3l_lP>2}Pm`&HkfK}@wC|0!rw64=F=Jy@e z+C&#+IEbt5Sm6_Xqox;4Q83v&2io0JD5#{|Hli45uT+)%yVOAzS5>l2!%mu@mK~`> zM*3e~rM>4+-{wj`cZX3L3~f1j2;?)ThN(o=O)VqGafD5r24%J{nL)@mpqE6BC!CL1Fw)I1SW_KC_QD-3 z{B??}#Ch%}iLP5u5EPnKedTJVl6tp4W>%va`^sK8Ve3*4|6@XE_DrZUDi*N6GCTq{ z(;X%HC2WnZVIn$>Ou_`jCaFQjQl4yp9Yj12CM&Gps-UGt;UzIbk3;D%%fk9CqaQW6 zO8v#uS^q8Chm*~@zQC*R#*npT&p(7NW9p(5e_|GoV(+qK8w2;n1m~F0W3Z|ULfg(M z!t!LpaBZ@K)b~rlT;h79Yd5dQ?LWH9j;(mQ$erx&3P+G1B8-%x1oNP=lZlKDb)tZk z)Y>I2A@d5S8wOG(g|9uzx*uEig}SFPW2k&?#Ud&;)2zfi2QVzu0&;%RL(=zFBq962 z#|RQhd$Bq018D`T`1aXq;T>QJ@SW@7kro`S632=sR6EHijDCJdyu)W{wv3BNAY}yl z?>Zc2t0C(?=5_n&BiP`Kz&B{Y_q?KAWJ1X-eP&%mOOM_bI=9#dW2`CN@QF$W6><`m z+m6ieT4YQ7#M2bty(1+mdn3(`*8&c2NfgXRd(FJ9swg0~uhLe?VOtJIP&oEWmX*wR z=4R$?TSvXd<|yqM?aUDY6ry$yKvNa!|Ei9hLL@%(FG{gn38s6G&3Kf&Uc zg?A6A4w4ff6fx_;oRe22_Q9o-NA-#;e0zST4@>QI6v|L%I(2(aR=`~YDVdvgS(J(s zvhxfK`0V;{x%JOYhWxu6d|rK{tVxfXzh4fz8VlE0$wb)**-{cV;83lNa0uf_)?{d^ zP*z<#E$S1GP!dM)a;j5|nA&C)ru~&)ekG_1R>Rx7o&vXaLke>Qg(xFA?B(Beox&gx zI$~&`^lS-R83d6p5V8F3cH&9|zxx&&4y=zw^j3%gg8Of}Z9%pH_lyvWN_r!R_o2=C0yQ%L?~DuiTHXzw6#pd+HrSvV#hncn z_Iys=?nW7^6sBxrdS2^Gg{u&27@NCH-4PB1lxBhH9`G=2_XFYOdaA%_Fk*y-mU=={-v(yu)jz@^kX;x5Nb}Dp9Tq@OTS6<^755ZlaqygwzIId>&DgAunjQY^ zm%V?rkfS5yd!!mIM*wH=Y0fgq(;K8vCmVc zGk?jz~1NNd#tq;p z?#*pI!Y(@MWe{HXRml^0{!Ur1c&x9yeyyQfVp?M4%KU93;j=ck<(L$&AX2D`S8)hw z3$zX~{I3agypSw0oAGim)BZJp5Ggk4}Niv5N>h6{?94A zcwO~LF_WW6nZT+G`B|G7h7Y*mR8a(KHc4`e{!C8#{@q5+>w`uM0hJK_*59ykoRC)R zG|7dLVI$P}9i)V>?9|lPx5A-P-{5mxvNHMi@Q*;Y9v5h(q0rPHI;+H6X$YKr1zB@$ zBT1s>)qx<~4RSE#zXBBLPVOQzFtXsSOJ^3L&U`T!rxaN~8+q1up-~fefv!Z|L88Kw zKRh=#jS`Hz8w9BE1EsK1w(y%lsxQ5oZa$a2b*==M!B;qJPih!nH_<7+&Fhiz1Rk3U zd9@Dfo4Q#464pOVH#LSK-gg?>v#3g!uS7J{o=F+131Y}<>?bb;o z9FHxeLb0+4Bl{(-0=u3dJoGysIIj1XRBsD-nCHE%0|ZzveR@^z;?Sn7`?)GjhP-8U zAwT=`3WE=ODq12sy+6Yu3jpO(3x6|&rLZJ$QcJKLMNRJDkZ$*;8piaHOn@9}Sj}*2 z@yWYNig5pF@Q8CD5K-1mB){0EkthPM4}&Ry>{gvu#fCJroi3-#@gP&+wax`!k+BJi z&;3@08(g1RiWOa^CGF@z#Loj%cV?$8&>V;UQk=-wfh&m*MWF0GL#X$%zms>G?&k%;|mROyk$H%PVc8|jn;WEN! zR$hp9$wU4E!#A$&^mX_dq&kMZApb4&Pf_nWT}fYsBYQj*)7ZCO4HMTX1?MOt@#lR$ zHOfW@nd1JDB*Xu-w=Nhr5{kD(4gy@c)_q+xM&nDGmY)5a!kk3R=>q zyFj9e=;;LU;YuMXV=u(;trimX6Jn?_Hc-lEtYTxZ6HR4}&m3tf+3~#P40J;I$JCtS z8Jdvg#b4+rpZmdPy-xmxlO{)g$i(5f0&L<6UA!`|5Y&FFUj%>td-*9`v6*a^%7C*9 z0rxdfROnaz;1-Jp`vIcx!bjTbp{U7mLBb@%-`-``y%QbSTR# zh-Kv`>VZ-u-v@?zQJPd4tY6p@gsf{)+fW7Zjq&&uw&@#91R{b>4ybdh`(E?V|2 z9cM4Bk?!$ryxPd4-b;2ChvT$ar{)>V^cdyTh+RRFP0!#GF0CpA;J?daNQGD1_icOvneVy})6fH{G8#xkerWnXsR z<8j$(4Do7CkCNcXf6dbw!58tnZq_%Inr0s%sFc9|Jx!>m*5i2I z6weB@JRdcg4)A+9#0qU!a2^`bM`~!-O6@a10idKsAO21J*c%(9m(U-`9*HfdjSRkJ z#b;emj-a_5_rJ1+RPH#ekbI?kG4w#E*Y4Sx&?idqxMy@B$g@(s3;i??{`H-V9cU1l zAeYwH+=GRKu6ao|5S->8_EgYm?4?-|VKkA-t7VGw$$zGAtDHMp_R5p5yOO*^Dy z3>RHD%APn+Tf2uhGT{?;u&W_${JTzPWjo%6V?$yISL9hdAY*HFI`!+04Jf%&^z1Qp z{K33xt>Gvba335QEgi~$fuUdmt?Ekd4|$8U)gy96gJTdC!w&DeY16f;q)(mT69_9> z2GbT>@L}CmQtOrmTzeszgH(qy-WQ<>!`$q@%EtABthWBGE||a|x-Z{&ZR7&A-4JBq z!WSvc+;@^4zWi2j0yagJ@2_>(REx-A3NQ%zS+q@`oC5-@f-o zTu8mnPgEJ}r%INwntV?9309PAM&+#u<%SP0!k%;q=%47=fZ!cw8Cfk(s{f$_coLqS zXnSdL_k3S9HfgG0+k*2;{{diz| z<)@6WZ+*KR2H@fx-(0jPsri)Lj=4UywJNzC1)WA|Y>dF4`6+zV2|Cmh^>=#fZ1kor zu$7$3vb+jogivJ;fp^xYSPCe;RwJ{aPrC6Ip&4K5ZRH zLY*MlPB47g{Ac(+iwRsVN%)idx@QBgG+me1F|p1uLEQVva>y)8$Sufy*J zX)7ybjA(C$^b_>+%EojVeA2ge4fJ+7kdTbipE z(K*OMf#!666+3?UzzIIJ+8VX-K6%OCZrscY*EuQZ$NkF#wwt$#S8m#O8b88|oYyNn zA#0A560JR8Kx#`^X^t{^_RDWE)3+}bilcagz187DksM8$iMPWn(DIB!HP7)TbSx4- zQCHp(!R%fMk5kf%3xdvYrqr;HG(D%nsnHduhXRi~lU z41e+aSpZ5z2ZCOAGxxe_UOuf#CSu~lRp1iTRW6@p%;b@&`AI$d8BkoCYp=m)B)^`_+N~jLvSd- zwndX0+sTb>+qP}nwr$(CZQHhO+sVI^SM>*P@P<9@sy>50d+n9$qv%yMViPFw8RjdE zlc=I^e%;>UcBJ=#Jy~S5fK&@6)3e@InKZImTw*lVVRkerR?Ku*&U!qWWTyh;QkOE; z!i#BMn9)kF-veAubpIz1G7kB)4#^POP)H`MHZ}Ge=mm}k__?ME_~T;oq^I|9!k$tCb#!N4A=G%2 z(5`>3S*({D3VR|h*tCy}DZjvh-%abpExE4&78G2)bUeLk%{a3tHlCmBhv-5_RRS35@rviHf*qMhFzYOsM(-i2(Cdyfjjk+D#+z$)m-H1i${#0JFKCesoJC7b6 zT6nYUEI64XL>B-O=MDesX@DDV%u}}oDQZWCq)0p`7xIx0=37f5(lKJu0KPfh3;I1_ zo-)?U?Iby)NS}eSMjuP1rh#%F(`fztDn>P2>~AJJ>bwM2u-oT5Jkc&h4WvrJDRD^S z?+Xf&m(fB2**fAQ(4@!?(mV@Kp&1hsoU~iTl`cGc9?u8HVc6s`({}wZt0pl0@m!J+ zsar8n>n@}(&BEL1X7+qL?dC~lrxt| zo`dK_TAXi5`F?G}w_AM0%T2f@dKEqE8}?DGFD)5mNxC0ecX&8+WhGzc0p;lB0MB@bMlv5*_@oHAn@=K|8tFsoR}@pl}bVS{emgZedMSnr^^h zLm-aOaxC+nic(s!tZs)Yp1oWqecCcqAg`W<6$F=l4vu~MuMHv8-9X}R`;>H2-%fHt zfpSa9qJr^Y@1J_)&->V_p}2TlF&y+H2AQI7L9*oeH$PZmQAoZwqnsR1VBI@B_{ovR zOX3nJLbsY)s&!&KOCX7}t&Ow6KicocirwBh>f z?m`kV){PjSWSb9ytsSgk%Hm5&qwqo+gsoD$9%D5q$gpJVYBVA6h#yB)3mK5cu}{`yD;kLfES4 zi^?H$K%!(PDPa0c0&Qb`@WEyq8#?skBGb$hboy#Xt(8YR$j1-@<3PbrRfAUHUe52I zOHT*d{tc(#hOLpBOB3X}ZoQr(jn=X0;j`#=1aqd9hL;1QQ>@+f_wDY?lEciTq@zmB z@`fmIs%j_{)g>a|r`K63d#SN!77Mt4qrIavc-!C-&yUL(LH6jTK3PaIEAT2W25pE- z^YT=;_(W>J6B0>(g0%vLfTQ5%$I zwjw5TNMU6d5$s1y=%Nz`RpAI8T%{a|6Ooi|dkk3ebcTnfiY-4Mlr1AVH*TzKMM%6JN=g--TlkYFIrh=SHf!&e z{A#0aBQa4G6j){1I7*@#Z!#Iu1ll9hWL1;g`OHA)+Py))pa>Sd>cdd`v%imw>qj6ac!YgsF=(2 z)2y5l(Ek9-pb0u_1llP0KT@UN6)uAwS z_%gGus3f_fJo(lyXh08Y-Dr3)+om(02;xJ?zEbfQY_w7vY$r8R9WX^`sP}hOU~$k4 zC0fU&2*ab;J4epqC!sP!Q}dt4>17&?U8pW_VAif0|L&gjDWL0W)o#{<8P@Ym<}By} zFI#(S&xxqyP&a||7@Gdm0-7;>HWC6~J0@iJ3^nF&Y1 zGxofNe?XE!)OmC;wtg?sCwjvta&RiTFkr>lNk5;f_}h1@@A+@du|4BCrJVT@t`*b* zH5{Clx4-Qh@`XBsw=FaA1c@<@0 zbKtES;=hVq4k+5^7x`orclh94-(=$EtgUiEos+D>nF1Ra*&Y zVnzs5-_kIyDHkK-*zbQS8=@DteiLAebThs{YPo5_)D)6wz*D|DN~PEc9xKsVYIZmk zh?cuI9Xglc7kIl9JK7k%lG;W(CzaR5MI0QnO0hWy{5F(^*}LKt?&kwYu}BTExva` z3RBkjma{Pv?Q`hDDxQ6YGJzVo4-OXV%8!7}9R_A0S@2L{`qYLb;>2`UT!D(1)Uq6X zImP`$B~KttV!5>>Dr6v%S(PPQ#7eihGmL4a1YR?JZ}KpLTJ(HXRy^xCeEdDmn2s{f z8>}WgmWX>K`S0_KJYV6W&d2&7gR$=Ng4$OMBjyl^g<$Xdc!x zT!$#xi`yfaSr<3B+H~B`xIJ=_lCWtoe`bF?7v>_NZZXz_9cQu8;f`=t*(9(_7B!3- z{9xB5Z~Y<`51f?n)ra?DfvTQ$+q;v>pkIT!^)pE3r5} zi6w*0#Wwu*6=2@8uB+HDrWHr(k{>yA;`*2--Y`LN)XEKIBn^uI)sCLq+V_RuUS`_b z^1F3JM6d-TL^sl&c)|74Y$kVlT=_I$cyBuf*Y2M)#J+OR;j)TE{ za#gpGu$;6>Z$*MET(W~*a`u@9#Lgzla0oKEBnGsDd2&$DDW*I-9XB*fok|3k?(i~D z&^@k9p6QZCu)G2jj#^ZcW?vEd+2i_P;Vp=EM_PlYOjpJu<5=O zx`5h*J>gsW!!n#VW0#gjoMEQ6lsc(Ab_w;&AU&O<+VOVA>tKE_EXFqR^W!EJ62Y^2 z|6+0C*|Nho5zPA;#y=18jKlcPx~Qy(pU4mX-*d3jGrd64sfl@gD60qP(GxQ6-D6iy z;p2tSG0fZSR6ZWz{42N}NUeN`cm377hWWT^jDL1Hgw`6J#Ul8{e9SACo!~K|Sm)%0 z)FDAsvx)@&&&0hC-IEH?TzeZ}LdU6l zW+7`I?t*s!c1J$6Ev;ddVe%pFX{a;{R3i>Y6l$n)W|k={;*YTui643}VV5J0A_G{% zO66(SR0prig!biJ>8x5#^3_*8E_43Tp?P;e$fVBu!MTF34?dr|`F_?3!I#aAM^N1$ zb*C(JTk`qC&#t3nC=)YrofUI!z|iVJyrG;w1ST~qv^tVmDXX9lgQm-?&JG`9njpHc0e^e4uAO}IFlbK+CC}&@HCt6l4mGg5AK@>tkc-2Ii3It$jQsEe zvU`^07I`-iH(@tFuucOfSSzbT(>Vyl>;_ghG`FQlKwcl7UO7sjtr?~2bnGdDRkxx6Xipi9s&e23}X(}X<9rL<9bRVL?R^E~L^ z<_Q>G+EPW#F#Z}YiO(%Ep_~Jvaj*p6mplgox|Ffvx)i(#wEyAtta&~CZ=C5ra1_th zzygws3zAmK*v8b!j2@4kjhW>?Ck8wQdUgip{~Z56XUf1x$Ikpea;8oHIMYncRpuy5 z``3Sn>CFu?96Vkduq(AfUYn$SXaTiO1d1!ATtuhjeQJ*5=i~Rch=_ynwDDtx=XG{e z{G5WtzqzNuuIYD_mHp4vZG|NOfTB!oDw>DHTZuY0G6;AEx@T~>x5uAfoa_Pwh_7c# zijD_N1iTlz_GgJ2zqY&<0ShaDtO>w94r%Oq*&L+q4zMf$j}IFc4W85w@GUie{|Z*W z8K9%LgI|V4j}-pUGNP$5G8S`f1ofd5l$G)_SHicUcnv^;Ptwzr*42{^;LHd_&Op%^ zsEpmSJ3tR0uo0>6j}eqp9VnY7kP{y$fVQgnrW-AUHr<* ze6lIvm{`U6*dIO*DgbwF>ulay4!ly;f9xUvbGGVlE>Eq$?}D;OKIo@ueQyv#iQ(n!c48J$<-bjz|+Sr3J`~T+viUx%dhul%8bm7o@?0W zE($+@WgXBNoaZaAC8)~>lw$I~GQ-5>6@6uQY%Wq2h|xC)&i@ z*y9U8r#G~LPmRfq^}%ai2A#|Guh4NQJ+R-&BX{D1tEbxluG)SY*jmz&-ivM*zqJnk zAMAlo3GjpCuUlk5fKJ-6Yumlv;^jBTrmtp8wx8-}-kGglE9zV(7}2{#S?mO=|JdZ1#Vw&JRB zSVV#J02qomH~X>EJ*y4_Qxp*V#J(l?QU#z5JFd{vW?FJ)Bvq9R#LPavWj;hF_3?Nw zZkFvitL5Rn2^xwK@8Yk4R%5Qkb?b2JBY^FAAoKp1n@+Zxd$g)ZCW^y}mgLww9pc8U z>NXqLwVlYcp=_ea#O%4aRL-rS*FK$b;u7@70kJOxV&K%;Z8lDKYfl5r8UI6!&s=wh zaBgFom*HC~rh9g&AOj8ASIv=_fSJn*bo)VL?OvbE#gP(vY2x5Im9H$qK&bi`UEa%@ zbM~jT07WP%;KFTgy{D==1 zX>jykcTwg8+wx=19z{qH1F3dB5q5iq0ij$GDIqgKk6L;Jw>P?+B4D{2iW~15$!sOm3$6>9l^x2 zd3wngPCbp*ie$m`02%2Ww|UJkZg-H!`1*vvTfyK9)Y-DBTBTK&Q9J%xI zN8Ay=w;!X@>qbzSMjx)v?3P}WqUDdhr>41=$LF?Wr?Ak*Z!v*yVQ9Q48Ev7^p(Er> zvgwKtJI~$5DJHfhgXKSD@5CTBb|;upi@$9plw+OYU;xL|R1B3;G);mEVDUYu4=AV9uob?hup1r3ZiVrbL7 zaS*GpRE2GY`+fyIF?^cD=TE8sdnTQ^l@QWPDN$JkY9HQ)nZ=VX#XpQ^n39Oi0KtOD zc`%m%oKc~mdQVWvp0~29TeeC9YSawP#H7E3-_>{1C|FBHZxWq4tsmZQUa+gX!n6 zsuKl@rd=~&zRAF9LDE0_t~@%b{{TB6a;66xZ!yP7p%xK?4T^AkC=!en@p2%lrK^!b zwA%fgrD&E>%vNPcKnn4;ZB8^>$v5-W{UX3HXI$m-!Pq+S1$((Y@XjxPE}bs;^?=@$ zd_zF8qmL@(9c5Gui#yVArgwr=@F)8@nYZ6SeDlk^6jD(WlqXrS651tzs0lJU2|G#7 zBcEnoFoPW<0q*teMgJ5j(|S=pur!|2R^J~tPt%X8_|C1yKZ`P?>>F{@u1tdSPm@iy zx~0Kod~vYVt)YUf0^Oed@BGq7I&Tn%Fvush;=1Ry^#ej+u2oijg17;_9nOX%EQGLu z{06A|SU)Kmf2sf>hiNghe1z-mZG_0bx}e&I!jWK)x}4Kr79%I<=mktI!H zO6#~zbfoy0ouG}pt&bO=M6$VDg5JNfTDI}3!TY8aMw6H?O|{BYJ#eynf;^Uod{>cO zJua;UMvvn9-bGLSh$&$0twM;lWTVID*wt5pTfPq-_7SfR%zt5lIx7cghq$_0GjMvFyxezoL@Q!*!tEs#{G z&kS*5_sa@1oQ9(M(Re0E1avX-{>(|ZUxrfUeI)|}F;;V7dC$ zh$!p&Z3PNu!b_8phJ~d+6wB(UgBlA~W$f9^TQ6Ss2%%@_hLjsELogrAK4 zl~-MB&$TrI@3FHS!n_G_r()J${3L*jKv7B(m5yuCy;~jGyeiZ&;Ka+CSqj4Sc_puK zM^9}f1V!4i=OklRr*F)zU$7x0fkncmZsaFs-G*U43FuX<$L{{pAAx(OtJzEee-l*K0y-K!OX+!`W47UmVBrv3ebm+Hhj?t#UL6vF zo~xxEWytXyDF{>H{V#gSM6Pc}kpX$rM~AVdq)QFZ@b*W~%3hnrE19}yh0k(zZVsO1 z9IRuSd|!LLxmq?uB!S=$w)h-U4`H8>IsvFHRtuK4Rj8Nf&s{llgoJsWvWzh-u0H#q zQ8xlKL0g=>!s?>ONsL{0%%)T{jnY@rP%JiY(cK;E~|A&$0Z@I zuQ4{rh)76g11diT#8>E+UJp(S&d&`C1;x z!n1-5{8tCS>FI(Qan*LZNK<&0)1+7tsQaVUGOVTy-M5-yBSVRFq&z?wS{k6(%vR~(L)}4hWMgj$|5+1s^6nd zyX%@Hz20gx)|18d3S`6__8=Sx9vip38pC^5t_HRrUe>+Z-%A12nX%zp8jRHx*=@r# zSempW}%V-)|)XyD;OS0sdsIG;J%^fMZNho0312f|O0Bf10h z;ShfDjST7Fob+{I*^>7RYW|WBqj)tfW7uR&Ltrd=R8ej_aW{C9LL{{q%kQcNZT9#$ zDGRAD!!Cj!y>gLa+JDdVFZL&mS89yH@y6`_k*@#B2P*NP77=qlfOes6M#PbvAlrqz zD;O9kMt&^kn&X97vgq2$UA`6)Edj4>+yTO9g~= zd_u8Km`+(m5~B*GP^60cQK1UcQU{a!9kVTTNc*P07DQy}betgQTR%R{mQUuXhR(S5~tJl3(0YWQJhB-noh) zPkgXdGmA@`N=3Vgq-^UtQQbH>&SB_c>po0Ssa_en9=KV@Nr7Pi(BMy!CY{Rt$8Zs? zQ@>uIuaUSobO_eR6rQWXhfm&$DI}~nfZ0LNL@%C*Fv5_?>DyI}9C>!*a5TMdd=JB6 zay8#EYsD+c9+ymQYvlUEwSf%Km^!Et`M`jzePkguh`J}k)4~U;J_E4^P0_NS%t~wz zZZ!ZdJg-=jp?%AY27m9iL}{h38ELkckn~qhu4)rKxBWoqLX-DWk_m)Oq}jSN8iPwQ zP9f$ZJWxm+tR41HLi6>CO%!fX~NO2eVS)Brv(VFmVcypKdI`6b5 zPo5L**t}wd_Dc^wpo*zmw{1>F>u6Cux@=$4*ibM_zU_79O z=a0O)pH(CuKBsWLhcjh7vb3;85K||@%oUvK3$V>dv!kx8C_nEe_N|raijDZmvt*pH zpjtc1uXNW2CQe&x`OkQ(z#HOxOIHdT_#R-DqS`2}FDM=|svr36*VKa1c5L(`q9jJ3 z;SAxE{DLvHFbN>MlP0y-1_E(!9oDY1oso$_XuKgEu-_t^pX8rqm|)*^F9U;m?#qG3 zyC;rO(EFs8JuN8wLXR|bqHEix#njzA6NPXsE&Yy2nH+7OSd6;#8(xl+J3vbL!#Yrx z!&kxI-UT5~2B@|q$ETvAzJx;OETc()K@%t5N~gwk8OU3fY1_sj&l52g9ofeAJ-^!F z7j(u23P0896vENZJpu>=*8T0VhtN`mu$Xg8d5Q`9Q3;PhjNe${1~Bsm@l44u>iUUt z_{(=)P>+|KHQnxOs4v*BYko4cL1>p!eY#=EI1(-6ZdHUf8EFsW1Fr_JfC&&!^`eoe zd9mBvuHxlG>0moe&FlvuH*gWEol)WH4Z2vC{cS>11ymiNK__(uRmSE5F zO=z05AMGmEuJO~R-9_w*U2kP`kxj+NKC9@8`^Ni-yiVvQC?F`>yYyap1!Rw{G4*$& za%Ra;p8WI4ySEzqwmL!4eieWJ#nL>jZ(+$PF+UVQZFJ70!B;{B*1x55-xvrytMq~2 zyv$)cSd)l#i%eznha<(??R<{TH$QJy%P@;ikHF|y83FV)ThZxv7ju@;zC|`Z>5Til zo)>XYR0`%=e46{sD89x^cX?|GjBh=N`7Q-+jjvGi#cPQt)~`Mwdq+z{?z87qb!>BN zvuZXBmj47}1?*8)ylxksc?Kp26W!`DR>3s)>fzSK0$Q zkPD(y(k;7A5aBAlN7bq&^TI>UemZQT)BRaS#CMp`)?Xu^i)FmnzpFdaw)a16xrk-6JmoQ;+3vzCjE znpvoIkPWl*e(1>t4h1*m7%ox9HSMHHvylL^l+s_~XM)#$*aM}!kFw&gl%}zTQV|;m(3ont`<$wLy>3Q&Z0mUy=gVHrKwnF z(K5;~bvjmWre{mCx#8H^C4mqXU65AwGH3;HrPl~yj+YKxVBDk*TN3%rkbZjwWRn+*Mz#v2&uy5PzvwQ^^D!*fXW3L`JU| z7@92(oNQ+&fu5;wC%Ild%#PYENii`3Hirlpiu%4Pf!i~W^SG=EAWkp`4m=@r!vyp z+;r7EkM<2iRfphc(E3d^<<``wP$3wip6k+bACXyc4nhlVEj5MMwu(=rV_1 zW+rnwE?{2jtHoeVH+M`75oqA9Tb)lAwtW?jEe=n%^hw6WpGy}u;(AqbH=_Vsv~8Lw z4F}HgmNBvXK4(P<1yQ8S&7+MDP%xfM$q3JHUx=2NcAl(gS08rAZ6D(h&J>j{6p)xyj4-nlt!LuepU6c~g- zz?M?hvlq0(z@D)3j;SSSgFNU?ONm@+i}=8V9kVS7v0Ja>Hb5te5)_Xn5yB|KmLrn7 zU=S~%iyBt6W5Mt!La8;Xu!K>_ZvNN`Gf3(cb{sk1XjuhA zp-x-qJgl*IMf=bfKx2h)Cs(vH_t=cMZo~k0An{(c&nm$p+;6mq!jY%(8F<8r^s$jR zw|udEc}xKLH(?Ci<8ABhfYwsLc}aV0a0+T|3~i_hd9C}#l1V&;i&HLy)-}0GX3vVA z$Z?a%o~a8jy;Q?4xR9NJY@#<3xU!m>`N}C}!V1E;MGyEZiG#$`$321oX+P5Hc8_{>uqxlBQ zzJF;GkZe}YL6jh;WMGH8r8E2Qr@Vh+^lBMQuIkL1`8zSn;ll^xk{?c~^@Np!liJn| z>UaKgw$-FPyx7V=RB>9v3Bt6*{%hP1RRVj~$l1+55#?P{`eB)4MxU=?L1@O?U^8FM z>p`3Q`;z{YoCK`lDtXXLM;)rg#DV@q{KBQzp}- z4dB{XJ^E}gdd9?(-HaCbVXsMDG4B!RTc;mVK)1v%b{4F9BWLA&83qjjokTijF10hn z?^59O&!bbL-&fr)g3o!m6yy9dn>Zbgso$QC#_sjngJ$pzNZSkZNsv-_lKaH+RO8d@ znj=CkbOP{?eZe+ssB<)&z!T7~f1~;cyfpVDysCvLfj=Wf>5t=k*T`E)10xxick=K! z2DgBZl<_)pw=n`A5J+D&6#PwGO%f0<<3Aj(|JcwzKSmFt54A}QxEP_8i0tniVDW;? z<{<$UJxX3WZMXnU9-C~F(pzBf6FYNZiz zdqM>!dCh&wpnJK0Fp``p94?O+K*@A+B_0_P%5j;cQQGUJj>X+!YboHTLL&%n%fKbF zs28r4T(95uq4w{PZ+V{ED+MLY!m*JBczI(es2uKwH8OU&R3a~jMJgXfh77FaxJcf`XH|Dbs5q;uG={FtNy3gE)lS))1b4~i(EFeC~qu3`&jH_dOs?p8gG1$KW zA<^@8iVcxYnj0I`XB8T5s4UE2f6XL_IyG=lSJLNL?|o9V|6H!jpM93h>A1eP_p)isRTN} z=ES1FgmBvk&3scRT<%t%Ihh_&X)pDrIGTU60=Km7YuJzi?wObB8$NuNQ$;(N`(bur zHr9G&0Gw2c3KjoRTxXUigpr-6fyJj+oX^;a0h^vT?4CF*X2gz*eEQ$1yCDELpvo)w!C%PDTU4h*1 zWc`}1Ln6KXnIV6T46;2uc@G#i5##27qcJV8+iAPr9q(h*q{ege6U*pc608i^$ftpy z!!5UeIe-c|rUudb)l|6EfRjsZNha@hj%QZk!Cd=S%JvQ*;C_zBgE!k)W-l?%uV#)P zYO8M%lkZ#PC9;SO1a<>Z&1_-Hy-npke&8#UhApQQ-*ECil$hwDqX_laQBaSm0R+Ca z$jx)YRt9x7E4}+(I%Vh~4Bo!n;iZ@~SI@VQv&O{*$=U+LS9UZ-k=9w3ODle^-RZ9@ z#y~QR*Tou&uT^{Dq|nNBo{AHGPgnTpxGA6j*sU44TWr|~uHg?3LhnHTi$~7XI0W>0 z7N?s?r@N47digDN>U+dfH=VEZV;0ImJ7<}jhM9!5+9)zLmE?!7of*`Ncun8GrTN&v z5|L9zo3M_v2BNfnJ3WXYqRdwg>SRR;c7HrrL2x<`+@RG!SNu@-Q_@m;em3V^twzGv z_$_wdTJRWB7e~1r7$L!YMXnVV!#=N{HBY}3iXHTf#`d_VH5voQ$`lsS-wXPi$aK>| zmP>!_vznIe@836f0lRM&>X?aAT1kDsz^sY=bg3vwJGbl0!S^?@fH@&+nq4uxfo+X_ zC{6oP>+w`A<`j~((mxypabl9e;QnY?R6IE?S*J|Ym{CO1N&35_tLG!!VN`786js|D z819PPN*Q{yNLlPCYS@#0gA-g4K8~6)9&|1^wp1wC=x<(<*nT%}L+pUE0-z~N7;YE2 zn=48O3;SpkRvC(L$g9PRJO&ejBYUIv)UxuXsY=ZWy@b$H%7|uUU^pN|f~6T3?JsUs z7O6>%w>ur7f9aFYTFmEZPtgkgy)+HFRc>2|J8g!M*a{8fF3Z7QcHFL`PMO>iXxe!j z>I#_lY5$d;pAdE8Emy$_ox{Rdv&0<7p-%)yvt<7?Sw}m4cwKmLnCZyW^-+c=pq$nE zq|XlBE#=H`cTnZiS*lR6cz z0ZlWRC7tKwte$Zd^SpkZkee!z?ZajA8;Wr{>s_XK$b`1HWW<`2bpgziI0uAdB)kOp7kN>}aCxZo5udWp{>&{AD>aOeM2XSAOqgYOfT@GElI)mo|adT@+L?i^d>z@YK2x?`* zgQVi_6Z-c$7+uj4SXu|A(fzR=ADwkhLZEM(v-Fgl9I?|vqZ`jfI;m$X%O^DR@{uY3w$SmWRt_j@2HSw_7P=Zh?OZ&7VbYLH&tO(FE1|O)^%VI5UgQ zy$TlP&s|2dI8 zGL=PL;Fx#aZ7`LNZY+(ASB4YxB1T8;&)pNzSChH15TPGn^dz^GegRwR?n0ZNGNTT0 z&Slmre(G#i8^pOs6{cMt4(b#(fx@Hlz%60S+*>Q0$4sCSc)j>y6QrBWCpAEJ-twx) z>mm_4-EN>)>h8H&vRELe>$9whSHlxO%-cdpmB1Sm?Oso+=bD(#*-4U1wn?pHfMggG zqoBjXRR6okdFwAoGfaGO+U9VLB|uhsZl$PWu$*p;;DKru#mX(MI%$}$vfO};XPF9h z7tCGF%GjhAHK3Wm{vO+m;UaWrBEs2ZJ|W2dlf(0i?BVEAy+WTW?^jA!&FKvne0<}R zXUl+Y8ch|wJrWajnFtc?HZ-@NQ=H|qhF{y15b(pz+(nXlz|x=oud=&hH^-rN>WX|v zuC739S*NPN)Exu5h&6W}GHpWKzM!Xj#VUoiI{eW9oib(kPiKo9u>+{sL^r};x zrfpACRX`v1#gZG7KPggrE55V!c44EUR+!u?L()vF1iprm{yVk9KGJL?a-sa%eo-R3aYdhhpfK1EOBqyC z$yJv2!T2;qKF7BJ1G(tdq=c+cOsxG|k=^6F;b?qRVG|9JSDGABQBHrIP{#5L<5<$Q z+-k*sQZKmrtNa;_!18R!XG+wT2Y>PGF%wX;4x7`3h$Ygv;7q0e=LXf%8F@fB!06!S z>N5$mmlXI3JHfN2i@6U~7`*id1o?A73e5Y0?vO-fd!EJqDjqQi=Mx&GcZqan#OvH$ z6YPGIl@&dKrTN~i%d)xxC_RuYB-F3AGIJNST7?z7cg||$2bkrAJNsGiwNzf>^uz2l zIxxgiAOn7dxQ_6oy6KJ9C#J}4j`o6g{4=>vct@zBNo5{F8d5@Doa~92n+2Cu9sa_W z=4p|V*vEPh^8?{QdfLEd5S3h}2R-iQPRxBo49eu2sQBusf74&^iMpW6ed>743nCn+ z1gy=qOA8%@j!EQ}`k>BL_=aXW5X5!fDz0vm^L=hT%@>vJ^+OrIQ&sus zwJ28MP|3(H{qwm5ywGc;1YOC~Pp)}8wd*f+Vn#NFnalz8Ty_3U-7-@AK(gdWX$R?|Ys&}k>2HMw5GXU9W7iY7yxUe`3kON(R-di>Mu(+<}|U7FW8|ast%NnXwCTNzq+sOc~E;f=~&QUQET8%X{%qq ziT=36$C$AIVIihB)bY15!f<)C*RK!u^(R<&qqj1IL(`$Tfq19hu}7xAhFjN=>O2&- zJ8YBZ9ZpJmYkdkWQ91nUti!Y+L(|FIOYGpt4Q+aVgfB`V zAB7z|Fv43wUy4G%Fozv?6MejygfTNkOT!@m=}0V{nL8+fJ(`Tr(nnXEoXa9kE5HsJ z+xcYaR|z8FYm2^Xgu;n3q_=P~2-q;ZQ=s06D{9l{OBFa5`5I^&U z1})=&fPT982fqL2WcP;2HR;qmpIj!A9DR&=R`~NK+JUQ}kMDIic-j)U4t(@$u5Px! zXUIlV?OdYdZ?$*r1n!W@W@g8Z}Ru;BGEQY1Xd@ zsB~laeCCy-mD#G@&}!L30%W45dlYn$am6s$muzbCoDE9f(Kz*@nU`H2h#^F?O0f`G zZ=1g**`Glx&oA)`)SpB>h1u;xlJCQ>sG!*>W=n}L{97hE-#c$WBt@7p&+)zg*cl|w zErisjw_&(#GQN+S(CZ!fQENn94a(@41&HM}m?iZ#Evh$T{c76if8>%yxSfaXHgc(f z1eDny_TE@Uk=<>@G<3MS&emjTgwjb0`n2KoisWM^^ z@Z5HNfCCNqQZSY6S{L!dcXs@(^APn0uD{ z?7CG~A0MJ_Bm*WPYZX}&MUQ>BZ!hz+~~?d*u{DM^UHe{d;wJxH84 z?c~6m+g%7=Mcl~@__af|0Ve13jri#sXkI4ORk<1rtciGJZd91 zDB~Ve%v$vp|u6W7)pk@DUTa$$X({iJ{ zHUq)Z$=ZS}UYdi1v8iZ57nvnJ(Vx?_Y5ovjJWdUiS}A=BC%1he8Pi%_;gR}EzeHN| z)X#Hxm#7#f=~em_70M==u-B3hdv9OGVjc2=%2Yal7EE(D^g(4-aHx;!NMmz0gM*+% zp@?C9ZrOYC{#2zcEb;+xRa}b3Uk+`v|Fmttisl(k(M{VQ+w)BsaZ!{P+$X*M6&YuI zG;M^#xaOGcpa4FNSv_bV>@4GGOnJr^z8qN9*`}|18ztxeD7&ZTT)1dUz_D%Hwr$(l zv2EM7Z5umI-q^Nn?$}9x)m7)BtIkbdu0OD9tvPEvV+gQ_T-?r&rYs{Tjirx?*}E-$ zw2)_M=!)B7Tqb^rwWt#JlOE@lY^OL!nKLcA+}#jJRVI;)kVkMs*i@C*mb_?7{VDXzr}95&ns5QfJ^-Y=B6t)6{u5Y{ zChj!d*zRjdAulZZZv`1=m@RVVQSB?$O+es2#IAlPTDL{7vNNP8zE>8hB! zbVK}1?1U8%PW?RY@UIgRf*x-FaQelS`OzoFURJ`?)bG{EdBB~nr?9O2k_1;L(7xXM_E7l;P~ zgEDSBLG`=qKg=M{hCX}%;uz7FwV#&Hwc5@7F>!3fsh5H8_|cCMdQ^y0cBZ&TQJIu% z>O(7lB45mvUPwyqJ#Gq9UzKlU#Ruc-xFym{IT}-wu87@Oq1f$}5SS8{R>sk)X@851 z_W5D8s>iFGc283PYE~sRVt%3KsifC%vxSpS+Hgy{cy_)5ek}NCe`-t-*zDHQRw##-M~f)e#P^Z}$+>pt+;^o2W_NHe~Pk zq9<&K5T(Hw?#N;_U)j4;OPfu!v!fPj_cz6>hEi}3X#pQo?kk#6U(0;`xrE9+%bCK1 z4Z;50BALBXMhQjdvu456T*!P}Vi}Se%N>6yH+)r2k^Pu?Z$={Z|CU+ajSv}%X`Tkr z?_c_6a)8>ePj(WG2MB9s1tZP5Tou(RHlBog1&8r8@@NAi7dn*Ea#|R60tNZmL$s4% z6-hTAw?HsJuS1|{K#?2$d34QpPV2R1-!bV*>KqtD9Fqz=aAa?Vg9|p4(k)mErGQdc zT!+E$CgSYIU02fWO3VQSyZ?%?(al1e#ZkugNGXDoQ@u%N&~oGUKX#;|dTAzKFs%Wa zPQQObcaaE#lmC84mS4X=O@oviKxCEx7mo$!NBs4Qw*;8nwHCPl@Jgkk@zl>9`E1F6TWZF+*?T(g}Ot^WSZL zPo*uM&?KyfeFBZ@+g#@k4<;Ly&E;f!h_NB_{IgKwnm8=YcB3;B*WNm~EkmU@asXSQ zD-E5mJRN6LlYBw9=vq~i|K2%=%=7y=AFmd*NkN)kmo0B-=rIOexdI&$g_P^Zlo&_U zc>!IdLdeTkqa$m@Ts4pPr1jNI7b)0HxJg6tIN`yY*3g-o z4miV_;;Bxdh;TPXwwQS0Shlc?HTp?X6d{i22c=*7N?iX_Y4MfWXs>F3U)hd5M9fdW zUz2B>D^S+vPpuKmQ~Gzs6Md#FpVqL_Bc_(a?eg=p_z37G!wYo5gGfmMIW8Zw)qh$G=k()n2!A$oi<_-; zp>-2Nt$+>I)C`qwPu?IZ8eHAqlf^L4rT*T0eK)Yzv{IVg_Hg!JiOkpcLR7CAcCS?a zWI12btJ2jsYVz9dG1TEdblYw@0p$$S+GBZlL}G!8{jD0bUZ%%WPn>0g&t%kkh6utw ze2!o_&JPK$U1Z;ZbSwY&ms8N#>+MMlOBa!+kt_a9F-aM&RAPexJ0Zxx47J!TjUy8JnBmau>F|3uQCq=(y1cGQCwdaCJS$+Nnt?*JBP}ECF%?S zH=iAg<|NqL@r9%TdQ%P`MbWi8|2gD7C!V>muFX#N&S1wO%i^th zPYw&C!s|5I%H1U0tzMEjtPPZsrVaeVx1E@7Pf1fz{gy`|mV=8u_;=H{i{|j7Lm`K_ zz2692mky>jpW zyy+Dj1bE~2Lih`j;W7i566+)uB^{0$J3BQWD~*qb;I>X9AA`Ja=Lwt_uv~1<66V*- zK2)64vxxE6DR94NI(XdmJmtS{WZmkR9Vj=Cf?f}`x?(X+{Buf3^?C%b3LK+3 zPICNu0(?`vh~e*_PirP`*_U4@d5v3m9)aZ0ETU~V$N`@AP2Li=9y6ogeO>{`k+wAt zm9bwv$N73%Vn)XXY|HgL{POh`1rrKAhwkz?A}M1dB(BEutebV+Uwtl7r!BVlaT@fg z8Gm|4?A&QQIJPDd{D#jl|KOD0WR~9H34^_Dbv?#zNN60(8tkfnq_sRddQ zKg@L(z=6Y95A^S_Q{uOav4`%EKmJ7&SQZgqkYQ=@tDtZX>{SlK6BT*De*B|)jFdSu$N=<{3`GJBQ zhVST(*wI5vyd3w_x&XO7b(M=}wzDi^!VhZ&xaHLuY7m{)N-P;1(2?0JGLx83!FMGH z*H6)7*O~A-M=)gGP*Vgf@U z3tJFF?wwK09YpEsv=<|b6xnCibM^oc6%n6sUbnDOx*2^|twSWSBzY<=M? zq>#q`{`c1))aHrmUV!&9I6W9bOZtf~`PfY0-F$<+zS%|MKG;qaL9Zb^SIhhxZquk% zWB1`_aC|yvCj-*-ygYo0qK%Ub^=ST)$!AAk+J z+0|jPV8T!?j5cRDaV4$a(c*-CJW(s*g@)FoVw7w?H1sz;+4AwV+*vJ#kbRIB$bN*e z#Wk5TUa7i9wYnHHjT5)Yb38zxQ`P;L6tINEL&rX%+(;tsQfJ7|)sZ*KcKgUFiSZa? z-@PUyFR*z)9zOQDXfxfVUWbHadokOpb@e)$YSMLkl#c;f$j&Ia9lm;hUH2H2z@j8B z9j0U&z&`xlHRI-n6L^;9KQP9fY|Z)#AqC;aHi`h4TTvkk%-4sdP1u8@)RL zYJP*+(3NJX29^c?x#h8-S{kP^c!Rk-J5bD_h5X`0N6r;B@Y%n83OVd~KA{h)VmUve2V_PpiH@UE=v3uRQG}5b_(T1B9EF!$O z7=%M_749t-K{5UvA0LUm0Xj{N?4cfJw^Vehw)2z6W;QX*aoR2yF#Ti>Mm&4^F5u)} zeD~pzw-0}diif=v3j&7@_w=!xmxSa@=cJZf4|X|yn4VBzTNveLG!Fx!D3FAW&F1wA z*^zfC9~Pd|Qr;xO_NF4;y^c=QHIJIMsBy&o#J6axX=dlTp0P+U+y;r@r)bWf+BHF} zS9hxpRwqd1iGwT$T92miJw(|P0i%)#2Qo<$%C9VbgXc`z+SNTG=KQK*Dk}r8$7x$7 zjhw&~T3z-2w@xJ>@s6ejzqVemBwNgXEq>;RMG#II>~4N6m2@f7R**uHy;j*S3|AN8 zOd-s(a5Z215qDh=_40yYpM0U@SsSXhu+>k*d|sQuMq`e2xGd-GX1B^{5T9|J zs?xTULsGWXg8Dh9^hFX^4Z@^`UtRFigipk_jTt0ns5~6Pdat%zI|w9K&>HR_cz9R= z;^ORM^ng-O`t9ne?8qjciAA0#r8Bn%rJ(V@4pgOZwGGTzEty2HD%ewR+1s_2$|))L zbe$Ki?hXjvYY+8e*<^INZ+{SKyaMFU>j(=9J+fi+9KkbN`6>%~@H~XnuL8kBMA~(6 z>I%Z_iFZm&0f;aRw$0|m_g?DYOlPOuid6`n8jx_nm%bB8ZY62Z2;2BLN_}$T-h6cx zXWQ>}6_b|lA#ImCeE-KRVhLW68j*SW`b?1 z2`L_nrIdf>q%2fg7CX(|TUZ>Xv$YwamzTip!7?ir3cK(8YWf zXpM;3>Rhz}=wz8mizlvuJ-586NV&mowthXTj2X_2aorEU+~x*Frm3g6!rP(SSmCsQ z(WrF_xX;sLO4O@{@Iqc!#TZS-Ky(H|)5_v})7=HeYwqCc02~gVo3D~SVc&=bz))!( zso=>YbhY+hYU;gD$|CltzCliN#~OKf0i8V}iF`_lH)zKohxdO{v5zJH6!4%K;G|7J zqc zkw;G9tLMIA72r*$>NgCTewk7_J{aW;gaAy3kQkKQkGXHD$S%*IBJ7Kj8nrg(=K1$2 z{K4ymT4wO>9GSwlA#hi=tk+KJ^|r!F|z7u&nOhKl0`hLYiivu5%#MbODqeT z)SE^F^QUm)Z=S$|%^=n7e6*Yma!f-jI2@-Gs^u#I#P+g0(?=Z|v1{hLUfrX)K?^VN zAM{0NtBG^VVn%U>srklI+tg~AF>NwY;z4>6GbVZrCK3$nAXsj%9d%CAA;j{MpLCCB zN+hI$YG;V-ZOvTZTv@>gYVpa}p+)xH1$U9~B)(2Y!2bH54#cc+MfEAZ7-KM2_^S4a zOF`M~IX+4Fn{RL??^AbBqMo>CW6W5GDvgt<8<0gFRT#G;#fFw(MRRN5c%SqbXLM=& zEbr@@%?*Ofx0j@`f;Fh|Ynnd*j@f&96v+9(4ZG1}3X>L_Lvx>_(;1Em3J1Ht8}2E8 zowjUobL_f(0NbAzubjD$Cwz0C1YaXNCi&$9%^}MDi>R&0-ha?HF~e$b#e*OCNY6M% zFPjFB+EfsW4!UN{$?b}VFfZPmkW)vq+Oy^Yc<@X@Do)P=<4d~nd0|^QFWF~OAb@xS znL;f`m<(XzRW$2b5CB<6>8Ulvq7u^-IBv7A(~WkE1p4`eTw&BPB|o)Wnuj0tb@AIi ziL&;TVa#j3M|M%##S2C^&e#OZvrgDm3zXSwG2ibcBhQ0&nrTeh$FdU7=6$yY(`9gN+S)%Q4=$VCG1CFnn-Q~}IZ`S_J= z^t23HxPSaV?f79jIBREK^wlR-ptdv*;2y4ow@mh8Mh6O7!FYCFs!tczatru12XGmk z+Z#wEt*&VTUUeh__hKmVA~qv(3%f`>-<}kJW-|!uW~A_+Bd&_6gWYlGl2r3qipR^!$;pLul1d%AvE}6quR%UZkl$tVZx4J<%S5D6Eyr696WxE-dX{u9{ zSzs)cMS^Nd(2tRy=5AO}ebwv!eq&R0ECPDRQwP)B^i(0Ke13XWHA{pKY}@OVdj86MSKRA9=Tsf{y8U@KOg5Y@%jAoal z?_GzGPaimvs9#cAuiVu;!FtU{_zZzpf&Mu$So3juEFN0POICyalPVA5QIKiv_b7g` zh#((3vl0FffI1O;j*XLZH~L4u^D*G5cbHG5e6(j>Z)18vf0e zs;9)U3-MYsCo2s=H^{F0Q)DD9zgSV z5w30_6mWTs--O^1Bl=g$O#w_VaX&n6V>pd%XM+RdcNn!mD|t2C=obF;fai(|dk~AN zEAH?(W=BgV6aM+NB;I0lQ`&1!;uShFWvCG<6$N!ds&xRmZ0lZZwL>EI_bEQ3| z{dD$({8vrvWQzkT98#E35v&uv!)V-`At>3}BBY@ST#;ZB=(Xd&PY#qeb4AU@knI#o zoUQp&osz7&$fJ(Ba2(TY-CZy`33+n^Uo|uN-6h_r=JU^zpEqn)P&jtHMAD%-YQIYxxJc`hUGKv{J2#)(%ceO$ApXG@%H;q{_@uhQ8r^N& zSmkMPojs33b2Sg?&!!H$uW~ZQ*@Z9n15MGC>D_$4*&}pX;w=@vF(BW5z~!iKLgdhL zs44dlGOuwps~3TboIhu<)Q784cn7CO^aSDd-4UqHv>bZtM&%SgvpDIG9eIz1vH8yr z%RQETtk@3zo?sBDaQNc?OHMWuVaUkC;B9PT~k5T4Gpk1HK7~PGsDA`P82V41Uqs z+cmoUGaSXAB=o9VlwQW^@|JXcvf$B{*Bt87X(pyrgAnxRp@tY#AR|nNL)hgC+1Sqkg|% zGcHlFij|}ZXZSMrKW#--&1Wz z=Eb4%WeBSfr6|8%vU|0rKt2`Lge=;qT1TPkRcMa9q8Tta7_3-LYjoStfONt{ZzPmQ zWvnE0gV)RW6`J)(0vl%8jICZ00 zx#NtHMD0_7qmF3=zb=S?!x4Vq_Z!XYlc(gsS~aD7-zq*|4D{6q9IV5jZh$k@pocQ@8jB{Pz;u8G(zEG5<8}w zuU=647hZd|ob8PC1|ohinah+<#oAriRGPs-XSmKOuCGd4hZbz@_6{q-N1s0C%Yj_1 za&aqHJkvDfKYqi5Il#^fRRD=30(@e*YDy4`M8wJ#vqUV>@+kE9*tKj*u2oHIPLR8+i|5LeA_a;?1bc4 zVpvkdL+I!UaIVz!O~rThQLdGXS>qDFp}SJ-Ij(iSMRPWh%wJ7KN<)-82$%|ISnwix z>&(-V=*@KoTw3(4n!T(TjZnZBk?f#Zd71+g?pAvKa`{WEYOk{O38{%Hr`0Ddlfq1- z0l94@ZKMAjmeo6%TR0#`PshXDb?a6;#7^VfDE>rGy~U06zg%WNA%Jr|)=jI~GL5GB5m zy@u^ew+a45!UrJX7b44SX*PvOvDqKVKNdW8QCoIIiT5sUvBy7t$2Zia_T6^fek}QS zoHuAD+Dt{D;>}+2^z4Y{bSnF-cPJYr1c9E3m~`v1iqfdoh@0BECbgWzo|iALfpPih zaAP~(WIsiWy-)~?)hX)dtKEbYqh6|RqI#}R2AgoR}$`k=6j|O zl}k!#<7uN&D@$`hJL>qAOJf$EAsc@A@Wx%x*Fe`>yE*N57G*iF6lu5N+NDw#J$MaRea}#!%ETvrneb!I_&8XT)jNnIH&h)$Y>oU; zuvIAM`Ysv0ADTW+>l|8xXgilnh8>rh9mN0#`s0`8`f^Q{rrzg}1pMkm`8HGw~h~c1xN1l)tAB>p!zWCf10bkm51h382piddspt{t>_$m4^~l>NvSxM zYL*SJ%iTZYod<^f`x^O|SsVJLU*$9yFs9=?9U4_!M)5Yi!xKHx*rYS?dxk#CpEnxU zvjn7jLOQ#FOd-kKt}ZjVp8)zMh@c0~!{?P<`%^asQ1|Jv_l+#F&4?Z#j%A1~RK+)B ztF&cF&Lo0PQyH(5c(li9#zS>E*}FC)rsz9*hp_Yc$vywEp(;6Po(?X=A+7n>BArg8 znYHYXcwgEf9$q9;FQPEdc3t4={l0=e5gH;9{^yUjkPgA4xpWPU9xZfK=&6&jjxyc$;7-?r*odK zcpIcQ%Z1PI#=g^S_-(&i^={piaJ*dlcUNPt8f0JBuM4Wgr0^F;;+3p32*&xZ^-lgH zXGA)z0^L*HYQg?kz+Y6qHXy{!PL8LXwp|Hm)w^+fb}lv;uPh*mM`C|kXjI=y zxkGi6n!wow#$T}rk0=f4fQ3`6)&X0);ww+B-er)4;OtcR1E{g;ZROhz0cj~fsUyc;dFz`a>M>~@PmvHVg_Bk zT!K1FrF2s>8}w7yg=|LKsPNEiCD|T7YN?0D%JrzkU2nWw6C63K7p!$*b0Rx;56FZ$ z8t|i#IU85ZzBF&+qCOCa{I{G6ksZ)%CnT&a?4)$yTKPz|)DD*)O_U%38Sciw1$Ebl z|4P^R#}71pH#le4WMoRAi#Ot(1IX1GAU1@#8lH@p;mk^fYacYzLl~Hzp`&%kZ;@pR zaG(L2RLx2L=ijKzq3r(q>j%xELB%tcGIW6XMEGI}V+|$Z%{k-yn|@7fvj8KaVWW*h z{mW7Ou~kY-h0@|6`IR$OuV1R}jH@d$XiUy4^UBk=L3?H~?PuM_X>p2YD7cjaLS`kXjsAOmaA!p$}}V7^23=jMUY6D)JT8^c~3u?Lc!hL zYRQu}E1x00)DW$U8FwLL><~HCoj3%_0|V&I$r$qSi`#=6z#|K5be}wFMIcREykIC^ zs~Z1T*Cz2z8?9mfpos&# z03%7$TCBJ>{JH#A>2y_e!KiJn&J&`9UbJHoQ}L`aW?f9gskB_^*O9rW3G%dw z85^r_R2F|MR1Qejxda$B!_e=BE%RV1H?Jf+{^?HmQ&NmI_qjV@Fw?hO$sWKr+-G~Y z+-m0k;6#%mxurY~SWT9b97nVR_VjdM@#(}{E(}wtW&Hu( zVXngHZ!SqO`@$a*(GS}dPFHUlEvguvy^K+5TqS z#uYK_b-@|eZqgHZ<a@nrKamcEi{g4#wWd0y3sx-w1swEx z#OWOwYb2*Um*V#Ab**FzmQuJHmf+e{m?h;mq$yeOy96j{)^_rHgoDP5c*8lW0-v%O z0-Uww5d}27kW1cwmLj)maEO`kvh~uDne!lB8x_gf%L!=pS$~0J)c8W9lfVghtD9!kjgXKsW2V4m=R*1!2-d>@+(%Dh|HR9 z^9;te>#_YaKX@|01BE|42+nnBwfY#OOK=Ej`sluBXtpv4gI_3j8oE2BF4JG8!v|id z7)H+>@MKNKg_m;B+Ceh(xZ{c7fyW4>48rG6eV$1$B$_PR(8IQcvAw{0*ZT<@=bV)H zet|ZUW_12HubTaT@v2$=bEy9ZsQ%BZW@qAH{-5Lj<5hF8GIRWYc-6R@UF@-B;>1=`?VVr6Bz2Rc4h7iyDm|MNs6;IzPUFVoGU#40jCd z$fFB{CZD^fx<{Is*>NxdoGip!c+|awW_S&p2+4z6<+To)1*|3!08E~k!rI%EwA77j zX<=^#S?aHd48GyRrIW48^;Bj4e5+YVFevu$Q{@{W%RekNLK*j zOe|sF|K32h1pTsMVsJ0NJ92FU`l;zJkOv;u&IY>44Gc&o6Z&=D>`Qpyzu39`0u1j$ z9(+}l&j1F26@>N!TAP@hKWH@-q@@uIZA`7s;96OmS-lWlIh@^{K(Kzyd;);vsei?h zppe|0T)&n?fA$#0e(fF|E=dYs)BM`*Ka7uG_IS-r&km2@6E?oin&pMu719-E6@IIs zz%sQrfqol3K3Xi&-2*m`!_Z5~ht*fkPjg#TV?A`T$8+N7_XP2+%j^*{P zu~%;B@c#rZ8vmkx+Xw${nIpJDFaZZ502OvefgcsP2UMLTWW6W6`@tFXTnoPdv6A9a z?qkZb?th~8WTYg~BJE$FgM1x6D~<(_c$zB&R+U9PcF*r2!hRX7l-*1V0o?{<{l_M6 zJqw}~eq0r*018x86Z7IA`_U{R<|=`l{V~B-eg)620p9;jZ`P@>HbAO=d`f@uElh2! zj_wJQe$k7dzlPFefR8P0OdkydWGSI#JqoLm%UYsF01%&y-K{MP-7jG`({_P(IS(1q z-*ZmQUTZg>J0|aEmmu@3zt~p+Z@c7H#&(dd&LEA>Und}a2?qcuz}yd_9#Bb7cx`Pn z)Xi^67GQArMoy?a-Tf2@!yO?osiY(cl5eRSNo#!rSbzF1n*+S-w+s!4Gz5SAogdur z?(hrT5W+e1wWlHvKY{H7w^tJ9H?9LXmf|;NeU~@0r=SE-l9iPS6kGlqI1MDR|10wHDqv28ITrA? zN2~Rvcjsq^LGXeh)Eyw!vqkp{%ncN)@f&z|ZTkbfcd_vUpS3sfr5F9}r{{jB`;Pd_ z%R9ffvzH?L>)6uU-VDj36a3ReaO%&i`&FZ%63Y5rFFW-u=bv zrdW6c`1-9PD--0rFvtK~+FSf6XwL*RR=1Fj<5XH%ef+$9Bp`W_T1d)z=#{^s&VJbk z{Jy`^_WoK5{k)z$CDP21jy7@C_K{XP(OQv7z^{C4T>&h-QC z3u-Wcz`8)?_cKm~DSCu8Cgh3B1P{Md^zaTTqT8&TUdb!QekL z{ugNAMBLN_`~_kG-6Zr6M07c8&xFuvf(`O%VQq1~$9_{!d7izk@hlRQ#Y zh-S@?tuY(rOS;CokwBLvlEvG0d5Fq%s1BAH(xnAcIvOrTJ1`bm027|g!fwXm!**0U zg|-l00KcJxB+fGwxqyOEQID}@#PdANQno9c2_0@! zm6@oCG2)@HeY0cN(G@)rWb-bm8L6y`uRPU(S?J`TE4erN3Ibkfl-U5bOk;{51-+h8 zZi3SH0?{>OKQuB822aY>>Y_w0bAptCCajo zv$BOXY?*k*h1{U8v@u8Ri}8{ddMv{P3%NU3-Uif#T@~Z?_kI)BDdoeXPI#pe z%)p&1rG`+0N7r-Jy@spAqC-JaHPIPp@(HY=M=Pzw^Nz~jf zBdVP_%u&Bsa3lS<;aO7lfO5<2=lWr(_@n@_9-Z!9a+>H?sc}_G;y3)q)gRt9hK2Ay z(RcRh*=Ymo6kCJzUNP7S%IXHbrcR};P5TW?*griJ;{)f3DC^(rZLw!ZbINO#2eH%8 z<6VV|A7PO16+PelVj!gaVw~c5tHL0|M#sg|jLl6>|LR2o53%NY;B1%iLZ^HGVl=OkTAKr62sbx1-W(yE*K=;evE;53aZtnEN8 zXDCEp1JI&Ykd5XR5Zd0X?B_6XF0kN$LN%(WK=bW`Ide?S(_>ZP`&H1`q?~&m1{lg6 zQi=G`3e>H3w=w$`;u*@iSDkr(?-YUlhOBhpMh)B7{Y&z5_7FPfSRrzSRFKNdR>X?J zP%u=ps4?PT);)iu6G>1agr*J*FOI%2RXZFO6^AXv7>=+U+ZHjxWoqqt8N|~UAx8;% z1l6m(ufs%EXZ7Na)nxzq_eAO*t>2%B(= z3_i;jzmS3+$W7*(R2gwR%zkLp|CMyKFZ(r?dkskIADH&;ld zLF6JOlJDDN){gkP7CzvvD*M&R`4AG!@mP2{GAz16c&@UgZQFvjy0} zYcA^kw&5amuj+ez;jxvz9op*Kvr?$*rf`oBev)hWLN#=3#;!UC!Gk~7@#}> zpAhx|Sheo_In^*5xhMx^3%2W$b#ZtH(@!_EiN5l)ljqS(o;TiR!y8we2~ax5&;9JU z9Oc=ue5*XvsUr1MAC0Vu!;xm3yN#7g-2NbHIs6Oo6YxIe72>Z=)CD+muhoYt4pUydmVu-fI&Hp7@gpsvQoMYL4BYigwdQZk6L4r)yx+(UIDi!H-|MO+fHUOot6jQFFOsH z@XLOHAP?X%^o_f%mv)pJbVM}X$Qz?R(fu)mdb%|t=+J_31dy3!>N#dVOnPr6O85;T zzo1pG2P?Lshs1r6V~OZKCP@!bm>-GCjX1lN=pUkf#{5K0-ixnPQTzv$ns^hy^ONdg z0-q4%b_QqD_j3c1o%|GBpVk5LIxV@+LqyRhmV>F}vV1*mcMqWQEdOSN9#|pDUH1JkXgAg4n-;A`Lvq(b`wYos7o9`hMZ3eb(=9usDz;fl?eJHU zJ}#NTkG`)izoH_#LLmUjK~?XA59!9OxAU9iE=)UbIY;$VAqn9%avJCL$0^o(UU3(j zaf=N`f0ti~df9i^7crZVunfbfqT)nXgqyiWM(v)VS>StiYv^(Q({_>!c}c=70Z$`5 zXF=mT!&cFulT9&KGFl(f$lcEaDloLFlDX;x2M1zsR6ZVu^jMldoJ=nmGLNO`TY~UX z6L)se&o7B0LT+1g%iHKe{t_ybt~j|l)etkXM=Q#o1V(YbAT zo?`NKOB-c*+0|B>{$m|OX?89dDcCLm{2!@TJ)Dt6KrvmjPaae|dkwv9G9;=lf{K>D zDts{BCYA7Osdr;Mdr{vGtO}XP3QeC;m)0KopR_kHAbr%dPT`E@wsIzFL}hfC(wZA>t#ifa{GGYDBP?mb^rN( z>4Yku7eiTEFS*f1GYk2{-rsowl3i7w%*&_C8%$+;619r^Vrl|cx%HbhsShcS(q;dd zE7cw`kl@tAC`rb4=;erD4Fz=u=gd-r_(R8!lj(X%Y9*vsIdQVL6*Yrz**6-QkkLDh zsRuQOP{@&}lC3JHo(z9$td4A;)r#B9O=87~KQKFYpGFS)eiz3WIgDAaSGbsz%+Ajf>* zdF*7ZRfgKFj{hYw?6idWXtQB(lNP;tcqSp}#CLVP}-hVXJt>5_}-MWq~T`OXFweVab)I0lS(qE5aA%`-=YRp?rd81`mS)f@YL0BUkE1j$wL6mXSrUdQYT8mG zDWk*WU*)G);$_B!ZPZn5k~{O+fuTw$iYwHJT$})`v;~^&T7U#rsC-HH_N2PBTJ{p^ z;~z`MH{jqby&~M$dSYE$g42;)_ZaBC~dHMtDd2tVkfuS^$Z6%KT>jAW4iY7-EiXDN#xjG z7C|N-x{m$3Y%);YaHl0zY4B4v76BVBcF8U;g(8K*E$Yh7v+TJzplu&;k>|CK7KHnG{S04y z4jB#=E=h(2+&Uu0nj{`Uk^bHniIsErq{aUj=&=(_$_CE(5`k+oY_tSVB8t!-44U zJt~{+Iau%OwhsioFbUg;YEOAQ-8_3I%Nw|RiiLugXs(-`1m#ubXmD`d6YD4Y4yWa= zJp;uySOO5jjd2WY zVR2{HgGY%!fj>oLJkN5hRbzajv^-Igy<)sLSUSteV#OdwT;q%qpkLY#EXgBXynz`{ z(O0@fhp8c&QhcZ#Y@>&4gf5>KH*B~!nALz^=^%MKw{gdi$kPY*X9D@R@?^5D46U|$ z#p||4ET-|={KWD=V9zM7OYN?y(HY6dWpy0cBsS1}f(e_QA0$APZE+}ukLv1Prd={> z1x})0$iObem_}7qBZhE8M~F4-;jWXe$`rvZ*6pg$30H89Vyv?i)Gt?DR+I09jXgAnP%-sbZP*wv(Hf)iKF}NF~nb&%Qi;?HCP~HX zk?#F*yHII3_&Kob7E5T}zC^$o<~<|4X8kJ0>Cql@E<1bv)K_Qq>vlkMjhi?+J3Kv+ z*&I5(CyG5jx)IyIG6>^2xlqnag`iUcq%;-mF&ZKuvU^KLj!A>MH)oCBn)vUCBGkud z7&>89C~ISY(2Y-}#a~?eF7oX+VruT@AFg_GY8PG&B~D*X2GLAjm~kh zL=04||0*KJErdR1ynpo0=F^wT&h97pCXmW#2n8vDpOc5CgfD79Q3rr~GL*Wj_ABjy z58LnMzm8FMvCN-XAPPc!pwI%@x1D@S_gOWEbJMl5>+|pMUeHBTCez;!+xZc$IiyCD zGr@}7Q))_9q>UTNb4iI!YK6h{Ocl6LSd*bxvX1PD7$zy`d>Zx*LFl9hJ_sq)&lRI7 z__uvNVOTO_)J)GcVqm>Q#R(j$s-O4SCaTX>O1%jk;ef;_u1C-!a$&R%fP0wpBj5iq z)snlW1^qt&IY7q0=y?GRK8={&_{)M1)wwXz4!B8B5S69N^p@k1omYovrzOTPsfJ^Q9v*xN%blDBpSy`&~U%fh2- zd^qBQ#=I3hEJxA|W?Q2ZRpgFl`H;s{r|FLy8=XAZ8ezQ4l1u{FZ$tMgYaaZT^>c0A zR|NfA6%*98MS|;(&qi9EbX3||O(0leZ`8DmbFbUsL(5vvoAWV$R&LGa+nVN9i53K! zqyyCTpEkD5=gi8-*{~!hC7SYsm$xZSP_At+)T5f?1Yu`$nuW6BWcW{=-yKhp|Ewy}BG9VwC5)YiQ;>@4d1^*X;aULN32 zF)TLOMl;Zgf|1KKNtAf_i7D=eYM#=eqe|^b1x8=4$bRBvXxtMG4=lf;)I4@OrEzQ( zHACUYG*2TGWb@A!1m#?GF%(1H z4qHfrHXL-B?<^~UYlQ&iJ%(zj;v3>8pXahTa36d`(nfGI{doR4bA(N>cU;rk57-5V zmBkkkpP=}g`!vDIQ1aIKH<#F8ae7r*DTaVIJ0x_S6J&V=xHT!KO(d&sL<@2}#*r`t zOO6YrfL(K1>HSgyu|2Bn@-g1MQQQwEnx28Z!<`qRB@icszi3rK7RCDfAxx@id2E#w z*k*DWGew=pfPvz{2DBDRU0V=;%%fT-Sj|xvJp5Ar+~*NQEUHD7=cE%FoV@F_kNaS+ zq!_*Y1kHqU5!x*uAOW_)`LmJ*OA_r$OvdX=1$#GNeHngFpb5dey_{$NV8EHQ#_{g* zT@N8#&rK5mpWq=8-7hUbu6}Wl1Sna}FvWgr%Y-)1q=sj=IuKQM52eu8Tu=uG7#XV4 zN+m?fk}ne9mZtHxEMy{xRY~Q=TRqu2iRxN;XtSK9m?ti$neu2AT}LuALk6l;B%}8;5?jvcC@N4D-&glY&@?Mx4F?5&zu>;j#*jfxmIS<_wn zTc=iB*``;06WYHwjs}2g6@#bQq;{nhQoNF{jVxn|F(^I(X-YjAo1EYUv$MmXen`Fk zYNB+wn9Pbh^Ct#rn&*z|p~DTqq9k9?=Q!-?0$U0UaKGlvzH86}3FxEfwXm!=dLu9o zIs$|nYh`mE+jWu;MNz*U;>bUilZiaW*HXE}zn9{GHrxv}s;*~7l^0)jRCNm9eR93Z zs7k4XcAAP`YP+FGUAF`>yghM^SoFq)X~hOm{mk?@;k~~qk8`Tt^-tfNK^0QZb`?nc zxy3Z)UaLaG9DweFdWHN7_UkiTnbJ{*UoG`KIu1T`3l~>N&E`)tpV>t(SaBsn6u9 zwbU{=3N+%W1*J#g7mZ^@(|%@%Ef} zV}v1~9{l66o_4%TxkHG&DKN>jxuQX2V_+cHno*6pUKRmv*#0WLrk+JDcYWgIsxOUr zo3Hz8G~($NX)TJuM5)kB>=sek2t8uE$gvkHLMkgin6P?SxWcp%Z1*u!X@F4SO3Tz^ zs{}q+=3S4Og{IR-2dHnwp3wL8fhMv_{3&lVTpZlQunmH*$;i_x8TxSck_5nsl9f+z zPIc|e*h|(YB_hMkuu+D)kEnLset39055bv53gyMok-_3w(y97g`XBXzUztkwtg1~OdNS)GwGB>AVACCgu$UyLHzDD=3^PGoe9PJmGv2f z=VM9Bi^0Jz5pQBiMkD18|VrPMcIKBNYv}sBq+|L4)&I?*KITUHc zGZE#9Wy~A02YzB&o1MXK(BqS6XjysTJkeTy0{ki~S|(ieF6u|e6hxLbvlvo(4}6sq zGq{b9B82_UM9)H;=G+I3n4~_j{Iwn4F&ZjkW``fK#VhPwP5ao&k@A-*_luq}`c6jI zT$bP*Xb7i-xwAQFL!x$XG#bC3Op}ir_d}4-jLS@zeU7IKu2$l4zYssfH9LusTcqoK zq*`W^HK73p;W{93N~5GLS$r(MXF6{9#3v@0!BHq$XRnOanZiOJgV=u$&@dLBT$(3r?qIr;)iZiLSGRT>%VNWSoqSyFS zeyyiEH5Sy?XiA?c17W?NZOz3%y?47M3i)R3?@n zBhRK?HVolO57d<#MQSY!D+pr8P>SM-kQvKf&T|ELZhl^<Wm<5@WbZsP5k z-B3%JTShgpkq{K`)CCd9^nhx-J2=CpoJh5WG~qcv%!_`h*bK{+aIFE=5nbQ%13XN zc{JiwN}8g83|26OE~_hk;lbQ~kS(B6RBY~ucbY?iVoPntsx*B`QW;p4tVy}}z!{1ecW?EW#+aoKD-WSV+jLWW}51TOg3rd%o?KNg}pMi#)f4ov+C`K zvWRcMczc8)%G1xn?ihp>H1y9~JtiF-2XV(5)MS*A_w!0Hh&^$vj$I}8&ohXLhFo3) z-+RVJPS`wcoC(5Q6ZG-Q6uQ$BxgFce%VDs-bk>IRID_(08LDYw5T zk_lOHpp^2S;^uEa1Y7)RGoku)m)S!h)+hlNpI?P0wHC6Ygkz<7*kPq6KQ^{he&z9l z{Z`q;hIVgXpu%&o2@;{H`4S4&y`n2m7H8mAj)|QpdAf1dbk6rh6*zG_dXYyu=eMyT z=}uA_%Z@#YKy{DOD*t8PYA3P8QGy*+P2g>JP3f)YdHFkzd!5dgO|&|TV~7KlTrX7o zzJ;^c?+f+-Z088^xil9VeU=cdm*oDqh`QRfu$?4|jF`7PZyjZ2{u`~5hSb3(tjHT% zQOHE8H2o$0=Iw3h6+vk}%CoAuoQk>lz>}!to?`ce_d z-%J9=%?c%`&&PGi=xN^|Y#e|Z_-&sbfr7e*F4Ga@>Vb7VBr*}RITb6TEWyZC==x>8 z+HQboc=J{o^>YX>k+S5Se)~kCW>m4|8@?=$w-xjNDb00=>n#y#*8#d}y{|nTl|o-C zj(%d3w?9Cv^uf>`Bu&Dz>ktRS2M*pne_)TEyLDO}h8g_z=!RB)lip~U@dX!rj*;6n4MGHv=kF{!tlEu;uU1Pm?Rxx+PLcT9ecg6>{kKDhRoy#y8dX+Ii?bqoR{p*LysTY}m?rhtl9wc^=uv1So; z&z-CIb8b_*s_Td&t--~f5NI*xm^*wrXknY zhXKEHrOx@)Dyn$74AsQK4AjEO+bw!9*#)G1YZ`rJN3`VqBx%!TQe`u^PE*yq+=fBT zq#0UN-Dw9#Ga)o7Z?x6I5eTi_MihPCyAbn?JH_&7@td44vj;z# zpVox+piEjDpAmhb8_d$ndn_D(Bd%ME)FnFMDKpQrPfkYrdk&5i4@@Dd7#nIYo}H;3 zH-SMXZ#Yv#HoW#GUsx3}^j1V3;i)%A38yrEiLkr}n4h&%kkGdN%jG47gGkel^H-|z zj?K^G@*pB=w`8wrXU_KKhWo_*s%{8H5R{9Yi_6+)Rf!= zHeC~WdJ?9YTVc(!>dM!{^wB9Wu!>!`G-J+sOC<mxdOcjt|hyotvor_p$oF2I%_@%wX2TVcx#}M&wjJ9mRfJ!a+ zd0W}chCyqDH||z(qmDy>J?xxD@YTDYNO5#z3{g8$OsFAr9Ey)S?u2$WybJA z3Yf;3l4K_EhV6Fa{2B-?6_B!i_59>fc$-+Yy)hu9v%n5$C=NgUVbxUA#y%FFfCi~G z-me(yxCn01ATgDMd5og%k%^qTj~Ju^p>An$1VZg{VoR7POE&kd;c90(b`?gs&nyg- zZz^9>>%1X!alJhpLlU*HH<7B2s#LTqwhmqo4?J(?9NI&g7NVC^b38Hj|FD*Cr~0~4{nBkU1|Bm0QI4DFux!D$gDS_!538HP-mi}jClvjw>J$Emr zH;UYHNNvV=$7;&8qqF;FWPHW839Un_K#ga*2Ek|PJy)-Vji0%QJ9I(t8>E)IjI0Q< zG5-CsGbMMsh)bFJg$ZJ9HfGi*@@veFhyXY8Aq8{&R+p2OHnBY(-!)3E-*k>G^dIg9 zpdGi)Lmeb@flt721yAQw8`2&pPk&gG38z%(sN73JmX=d{xX3Juq$HP{s~>&GiZ-22 z*r92L^z6MpSGKMU?*nd60dYCQ-DD- z8Ee509Bx#Bvp!MA0RKj=)w0bj1Iqo2K=+QFpJ8|)7(F8IQLYxDN52&MFLINcunRu{ z>2p|jvpE)-w|*(BH0nvClS54;s~dH?YAa%MxnJ*L8W(CAeYAir@Le$jRbK-vY_b)#d%MOD5NF|$q`QB(7{9oyFhAsW+Yxz zd?qZazVm*tXgs5DCXI-OPdU+Mj8_h_MkEpPPWPp5?jHTYwQ-ZTohva{`At0YB-~E? z-$C5GLVG1F9i%KnkI0w(afr#aTCzT1%BVj3?zPB0{hu%mbCmOiJjS8cwse>U)r4xsVtg=g*B7H!2AOEtPeV4I3(cH2E#?`~vs zMLNUsldWu?trC{OL=Clji_)0;0^PmInAWL)wE77jVtn>D=DPXnOy&bq7nU67eci&X z^N%UcT8k?yW54y~6Z&+h@?>R145SE@ER+Id?9lk=mvaFeM%UX-Xqto4wDFM>J$#VY%U zs3RZC`sDS*MZ)`XDqjY4(PRN;a{fvqi~ zpUCy}kX9sjm>M4WA3BLQ>Kj8nP8#Rv*Orh;!HXRkM=jJ*Kg_!36yw2a3+s$>p6n9)bCJmoh)Y$lv6jFZur{l1(`UK(&%z#W7ckbUvNoGp1Y4Vr z4Qm%>#v6V!Da7DC`EI0Uw_Wx17HA6E3k z*E-k{M2o9EKPx<%MM!R-^OSEUawh$7t33)8J374M_Zss_+zh*xw<`)a$9M%AL{r|M z@xW9BzBHDcM&`S>@j@4OyA)#D)93++*7=iD&=ejC7Tp27p4>2A#g0S_^Q1QphRJ2& z!|?>hBiX3SYY`W$$N{Q#=VimzWhA_Rl^H{c)tNj|WlCP9THz+*{)VCX@f@T~!L}-- z&^_CiCQ6Thwo;FYy@&qPHX)MC<>*Bx;VaGy^u0al4k7R0nJGg5Q_T|KYj?qvW^zE& zj{0hEdWn^SlnD=p4kb$c0sq&l{Dfz&y;So+KBS^uc`^aY%*qrbAD7{?BL+PlB#Av9 z28%H}5yf8xcc1tlirN+StqpnMt`up|X2#z_#Lqj3$c1F@F)%!GG$~Aj1_GG<%_)*w z)pX^8SIXobd^ulj_#;{D^<-V-Y^MwXwk?6oNC60zQ1u!Wwfu)bjL$e@$5cjXeCGb; z&+QTp3I{E`L0*#wZ`Y_EG~}~Ylzy&K4NUZ}h-0f5TTh!%r&58z!)nzMOE7T_+3vA3 zht=(;cV68Vwh!y`u)>46F0bVq;r!4LdvRK{{=~o4mBok6Rpvx3Q#d^GpOr&>$QdFF zOxU1eM$Fhci*CM4Jc+p;T+-=Rfpw4Q!`IgK9TKu{xcc0_%7?%bz;uE(8+zIY&{~)= zp)EBm6^&Y-u51c{LfluK%YjAX<_v;|ez(rJxcKUL=p~sfPMH@oo!$!@xW^B=1C{3D z4#&-jay&7oJ*InT+Dgx{lS@fhlu*Vr1uaQzUX}-9E&!bNc<>N2$n(TS&R=zVqF3zP zH{e0W2@0rw#$3E?UNe0=d(IwFc+3#Ue4SWnvWqK@*k>|Y5W2*Yy8;O4GE$=!zP~Zr zd=zS7@EkwAddCcMWo8Y6VDNd+xQ?t~C)lHsGrWF6V6_#% zF_1()pi&0trnd_JL#$E>bxP-$Bb`b7E;a7*DY zkXlTLi>s|P@5MXVX1^pSXQ`uN87HH4c<~M}`XVwWTIZJ39CW+TOFIqI4ZBvuEQ~gq zdG$MaTHvCq;W`FLk6Zd;WkP=g+OYw@)ap~$A@ROY3~1GVfNr#zSm(TQFn{`)H=+UH_nqsICE^{Ad3;SNrJMz}$` zrM$9NQP#GtQ|d-8A@xP-|M;}ReWzZSPxclT{9|!2tu*V&r+WOQ0yDL^yThFzggO69 z=MD0J)q?!ELfgl_W}%RD@ELHM@2HL2P2It!{s8W8=Gy`55LxLAzmW87$Jg->w8|0Y zoO*VeLZEkO1BVGI)?daJwns_Q9g=wu*!`pK-GHxQ*|ZCBs_>xniG>Bj~Y; zjs^>c57=m8ZP<2>QP=7tdE|Y?UrS3`l5p>=hYDdu$pyw7Ek3&0if@Pb^W#Enz)T%` zgEA_oz{RRmdqfVt?FS?3@zl2$zQwP&2KGGzOyC7r^U2?a5L>sDTA+=~ZRwTB=9j4= zv-ZMW?-n^GtDGKHQj6I5exfXHDfsN;wzSG5iv%%~4d&QY49u)$=VR=(w1 z8MQi99?HLwC7UE756?PqX9OMc;}r%97KWt?+C~Saz&W4?pIJU()b?+!>wXM<+#pRh z_PA)KGSEL>U%r1bhCWF1vW$jML8oQC)`yrug*iE{?Pa}ysxq4pbNICJSyWo%W1APc zx)+Ze#_uBhC<+1|XC}`Iu0}*;bqXU2dm`)-ywk^{!Cy8Rv#HnoWCyubtf#ADWF>IS zSI90;71u^M*aEQI%s%~=W#UvCT$2cMh_uUa6+ZQ1P@UmWuMt1>F^yv$1b+T*vLg}A zzXoh(Raj+!ZI2CYx1H3^p(j=#1(nM(p{M{fq)qX zNe;pO>XU6H%k8KFoaTJ-*22#ejg$kltm(T2# z`W$z)qx6WYkf8izMPOpf5Ae_Av9i}a+b)r?2@$L(!re-1a zkkfBh10n^==epz%-cTvlwRcDV@)G+_aA`xY#effv9=G=;o1}oUwh9&z&OHI`Nu=ZI>(lq7k{u>{5MRL|1XTcqJe~Q& zhslT!$GFiYBh``k-%8>MvxM%yr!^e-0FJ?#k481cpCP1fkd=0%^D$oI(12g$X{~Or*O}=Wc!L@ zfG_P3-QBJs`dTp4B)j16CghJFHy#9@QXfqK7qyGQ7j;+xbZ1fIN;V>4=HhO({fN0Y zgF!Q88rDSqqNFuegPUChk7Rq>IB}gK8&B%(kz&QWfld+hNg0wm5@p;9?e{3kE&M#y zbi)2)y~Wy9t#~znH{VaJW0O%~N;VaXEWGUpXIHq@EkJld-jwh1&mR!SgsL)&^!0KW zJbl805XAF+mcK#BJH+@K)}DHsMw3~UJQ)r~7R#?O*j?S?in^u^$h?u)0cVwKpSVNq zm-@vPDir$ojjGgBye>bvuN^n3+w&TehrlOPtKVGwyoS%eFh%-|_wmq=%KMCII$FX4U;e5%=(o?S#pFZ+LhLEopV-i&oi+2JFa}xNIU@dH|1oW z=Pt^ZRpt{!@Sm$vZz6VS&2ODeMZQtWR=}wlshQ18D&}I1YFAiaY-Qd2U*gb>DxEQeeKHS#NPkjCc3fJ;;E%!Q+64>;Fo=lc@vrW$s z)G*OKa6}J+FUW{M`K)9|ufzD(>FU_UV#&-nJMWujr-1^(q#MQPrI;lVyVIQFXgYG2 zFq_d&$W9|*zGArA?D;+okFTW{N+sPPE@`ia_}46)=NKeRvC8B`_*-bl8Me6C!#SFS z6k)&L3^BR{<^>LC8V)>Fu36PC!9%zjrYfqVowYyICI-B+2>4`Qspq@0fR`S9Zue#} zyMZS?gD$%7KP+E98%w$o=0xdR(kNBa7nC;=O3Dk$b03~cC}Y{~kB;*7;3G2({NO-s+uysi&v{v8bD;sW>FIIp@qB@-+jZ&{w*2u!=fnt>~!PaS*N$FNDx zE4=>P7RCGf*daptlKP9VEe#v^a+Sksh0NE8LBq>CB$3pDJ0q-9Blh#ktKp&KSy#AC z2CSVNOFsyzmhHV+D0l*|Q<0w<=W^JHInJ~c}7fQMkl+|HHQ7O_RXG5J89l@F2NU?U*bD1a5;^@Ly27o!At=He27!{n85I{gLds`^h8K0&=MqWb;UB~Ff5)gCwuD+S zUlE?FR)6A{PHCeJ@)3ifV1&xD6t!zy{31LwrnH4*yP0=72rOkO2YAxT&e;y91V1SEI`C9*!o3L{#It4UQ$i%k;B%?q| z-KgZ{b+yj;gT*=wxytY#$ruopH6;RfqT!04z+QKdRV4sx_*ll%)@hb~w@1H_z>KCX z^T!sB|Br9P9X*7*z#Va2O`i}~Ro&D8z$wIZa*vgt%D zr*H!qY>#wVWioYn-zp!%$>g1r6NL<`Goqhh@uxJ(%hkuM zw)tdvMO!U$it;{b*DQmW6FJ(p41uj29z!_aNAbA=e@n$(&Q^VlX2Ob868H7isT=u< zA)EQ=7u3CBERzGY+6=K@&Tk9dFHui@_K=C6wY!blxn0X_%vWyF%%ama z6tgl_fqwmz0H&+&Ri!0@6<~6aF8q5?k4u4-X!)AI6?Z&Ia)yTz>BX3$|D?AV3|iAM zL%ac(t|BuY>dk$RBo$u3C6S;Unok$ze9ppP5}k5xEPC*l$bNq0FgmZi+D`}>OVvTprWUIR&vRLTw7SNf(}jkJjauyh(m0dAK{^P|zd! z)#?#soAx$u&Ug0A7?Jd=Qze#L2d0L?r^wl`77%exXgqi9R*=P%b(?+DoitEw*R4=lsW9WC85i(EqoiR$KJ1p+ zks0S`K>po6^OLqc$MEo5!_pDd(OdK8YoJxO?548f>ukK~^Z^dme$@TOYGJTQ0^-$+ z?MXoh;o)!oSfd)sI}W0gIjcGB|y{!1`4|Bt|cdy$%piIiw4FEW9O%VUxo!$~+HE+u8=s)h(6 zC0@{$ciR8g#yND?0VrxTwr$&LY&1MMvDw(RZQHhWq9%>e#Yf7oJukM$S=}P+RH`M943&#up^g3VIgOp4U=9- zmP)aBhk=#++oF)@q+3OH9%4609pZExo_JQ@qi-Q%c_-H8(Z(L&s+?Z^ECRzv;7K~8 z`i<)Be_YNar+^fPR>j3O`h~C3(L(gWuzS>?kIz^)=r14bBu5j?T0EASX-)w)qR6+m z6|$596NcfJG$qCn-q;uYQe-FxxW`=4la>w~QP`)y@Z{8!XaaqtnOXNvQ^+WGDX?8z*?i zWqW+ua1pszp3Q?5(Eds}f4|0t|Kej-IJ{NK>s8BCfQ_{Jptch%L=;j2#5Wk8Y3qgC z$ML-6!Vrm~r9nOgyFkh`qdNnNt3C>tFwCxByd#{Wgr54~n5=hZwKD6>0HXtE@|3#= zL%2-AAL!xO3|h?51=V%|RRW8w6G2MF3DjO@np|TjNPl~ zBf{%z2=D18M##uIz0Ox8Q-yrKy*c7vHOD@wxD}M0%uYH*h_JY#xH%gm<+S*Y^{kbR zE~qaTGb3hI6tl6$OYldK`yf`4V4d(JSUN#ooV&Stkkv4krz*Voz~C4T>@cinrb3wH zr!Rk2_HDu`A?>T5MG&J+#=pGvni*TdL(|+8`FohMJCW#x({!n%*1h_u^#bp%!UUId z&pDk0Tifig;L^|0V?=;4YObD zpY|Wg341B0i+r~9tOtQuF>7y(LECk7xcdwKFZdx-`XHY$k|rkxz@eO0xqD4tP;Df8 zTsWVmNa$8n(!4L<zGh=N>|89%Vo%y(4*p~Gn5}~uWFA;nWtp2pI1gW zdrc43+33wli)1vGJPyY&BHv#f!-Sg82GhrsFO}b30MWhbGzsU^1z(5{8F15z z=Rq$#Z;o+?HOE+_ig7{k`)Q_LVuZ&ObrUNi)B?Ij9F+i>L%GXOI2a1oO8IXzN$SC< z)~Dx1&o(YA+{iw&(L$Zsa-~issj9NLb##~qpx3n-qjFSG{@K)+lN77xFiJYj; z2AS8Gk?fw&5HxYr3DoiC^yHY_yh2kNJ;mC#*g%f5uocQ+zZV=FlCgjnVih!0iowRI zQg~vLgh1VNp325uXKhD_3Irp#D7Od;M zR2+kPp`3+hKnfR}F8ON_mzFe9mAJgq3L~0Q3GK1=|4faFgDO##s40I#(9>g$jfQn^ zGKaUazLHM)!${~hr$TUqzHId+^YQ?6$KkFskWu1z$fuC3lo&4R)v|XvCw*!g3j8N= z4VJ+mWEzjNwD0>03oN5F9WTUNtr}w;lU?}9q^J^>rVkcK12k{+u6c*32`Xt0isoGn zm=7Q9?75okdv@)rK7LZ@9EANWtZKPsaWm6K-Eu5GbcZyf#<6-mqCINFt(Boo#uRsy zAKb5YN4!c&-i$BRDC~42Bq5_n^B&+C^lf<(7ar*ok#k@T>av%I#P)t*T_>y?$uHew z_#A36z$)->)EW5w6+BPBpxw3YJwTijDdmEDiE8`$twRr+dP~zbVqt^*;t{_+iISTMT7|r3kBUf z$)M_hk9&n6L~xbO(+i^nzUv2#&oq`)&VXw~8)lN0_lZDK2!~|ZuK_vZ!_w!8@;Ks~ zlKh^UvWL3H%k|Y2{jS8+UFg51x{_Pc_X8B`T>*|A`2~snMI^UAtHP&%dP666M!mml zK8yR^?B(||itiJfmV6*e!oV+d_I!))<~b#(AUwu5dymA2T04c%9c2-wC6L_{^W#$e zd>yTU88)%ezI*Edn*=FTR!Eu-dqJo=po=VfZvhH{tm2ntPbNvRSoggG2L^~eq{ zRnVP7wsx@syx9e^scn#Z04K4K@ME19-@&mOVQ(Wo-I1D>AUjegBmX885<0W27>pEqLDaE?j5g*R}(Auk*EE{LOx2-{;ek%bB49Lc9^b>{T1wYrP;Ni8fef~6Yf z^jVF>HCCG2y(AqwUWH{DDSqk0413bm2^s*3tuQYL*buRJt#?nTV!oa#zLc!O0KIOa zhzB1h7>0t3?oJUD6@2p)5zDs-_WyNfA}&;!+437*WBQ1z!On7aj(n6;xm}{ zFF6=LSrjAxi`(hx{tm0{{T}Xx0$z;DVbC6Ja!O@V+TLZ2jCf z-j;p%*0$eC{F4kom!65g1lS}NaTtuwcq&>L>MyN$R)3f;z~O$Vp(JD*TP zm26;tL_47#dJN2Aj`aF}GWrS!U7ZzW&f60%18)rtb!Kh1`So9(3$|r%_c}-cpVUrq zT&T3$wcHX|yQB`E))(S1T?CLse2;STe+|f*nulxKA&K(L%FY~c2J{U}H$uEYZXQw8&tY3^f$ zF`{WT%>(cliM99qC`ZCgm1Yu#EddNgVgsEbzs~IT;bZidQ|G&sf7?f?hjqb?XSAu8K35F4JO=CCi- zc1r&be0$`eUi24aDO1w7j8MBL0|O1YvZyAugE>tS3rP#tH5R;lRf2N}5k?t9z3Dj} zh^*|KYa(7*rfa}*%3C3K!-T@l610sXW>Hi*qOmd+s*VHYCUi&&^l}n$C%D#0n(}Yr zRCjb8^vI1qsLWxE-l+c2NDFMHb8l+*aZO}(Vag}uvwTdgR`aS!13H(9V?k_&LZ?C^ z(j@vDh)UH?RKDOag+BhQ4O+Y+$1T|YHZxH8tnu%;UG>LCqgas#|MGrf*3g!p$CKKO zfAS-a%z!OKld)*?ddla5LSgt;ial-i{nPhEGNHcb6RNXwA->dg4E2CX zNe^c{cUp4n&FcLCIwBe#mlwu-u|o&0ci}lGGc-wKYBgK%8c55l)wQRI3;nx}Oz$fs zPmN@b1D!3PH>9?kFAgo?+b>b02;|j~pANBKEmFt{4B1SO=+$6<<0II+a3&IlQ=h6f zRR()b)EO2}=Ksh{SnSY}*P5%cDVh3Twdb#G@H;2hGByEKpgMBxrJ1TC;uzrB_L~LA2axakZC|?Kqzz6YJQP^^b>F)xxRMik3&1NLQN32#UhFZf zq^S`xqP#S}2v|TSTXdN}NRes*_tITwytlYQ&}PF%MrgHUq+WUNM#BJp9QbhRo=@sB zVWvc#k>JvJHJ@Wi9y%U;{hs`a7o-z}C^gg5%(){}o&Ds=vULblfva?q{~Pv&egYz4 z{wayT3n!4<&UL0plpe#;J+SNJ_V*5{#*25Ad|qf|Oc4CXaGP)E?K8^g`Nkli1na5f zW{VUv`Ri%$>iQPB0^G;>(Qhb&Kt|`4hs#i4xC&{?OOsi$(60@SvX6vSGrwlAl_rv& z_HFB{#{n(VEWJykmMeZj$E134WH>t}@5}JG9j4hC$!NfZPX{Bdn7d$503BtxiIi9wyJW9^lRu-voVUai^Z)+Umw>vI(u^qWYAgdLX6+) z@5{PxRPVL+FD=Xl{EB{@k>mo_(G7b%zGuKa2zFGi4f0Uau}z}E;H%))Rq%&5{2O3X zJDayIk>3=J2q`Rk7f2G9KWIx5Oe@_=70OLOnvN)PpfhzPw89e|;sabuusGR%@V*v7Uln z*_5};!y5Z;l=w7)64(Rp@s7xbrl_RDcXwdI5e;gDrQNyp&(UR})QS z9E2LiM~O}m>z07PBekM--$@6V+XzB^8dSDxG3GQp;+WSf58zvo!>{-S3j)#Eufx!>@QA%#}@r|#(*;0&}~6uPTyRj!B9Rc@`7Vd|0_3^c0cf`G8AMp z^)Ly_FE9?2OGmFwVdQRk~Yp0+Z_#0ZKb z6ev5}UOhsM-nXO{*Z*G@`P2sfEZwfg_M^6%PD-)BDz87Zasz+qOjjCiG=fdSC)k-A zSae@sst{@IbMi`4r1+i5iy=|EmOGPnFK$)X94Mh*{`@bBX+@1#)cm_6rua&6fy%ZJ zTaQD@Yxj)+?-Y`~oB+hBv8M=rkj15L(ZomJRj~v{OV$7u^+ah>Ft1FyGSjcc;x~`v z!@p5uROfy)QFe4F;SpB>Dwux8`M)G_i_i;ymoG4S553BTHeo(OV`JamRy`h`2YQKZ z6X8G-rpCR^@7LI$euZ)!X>wnWv_A^f~!* z<+qbNdV~GKk+G2V$tncmBJGrXEFuNodq2dUfTjWR__P)rss39-o??CGb_vvfMO+sr zneP#A*Q2eRc4t90@)+ zV*x3u7$GCH1_C5Zq}r|LNwO`{)V()o^EZ#MNRm7TX>fBPmdn0Eu6*w9UhK%7&6~=K zm!o?>1~=75SWB!Ed-gR5u#;e|EE0xS>9>X*sb83`8ou9+##Z-4ocABW5t78JKaN}k zcnI%}*j||kB~%;Te?$b%syV}O6tFQog$ZkoQtcbMzI^#%wYSRmDjC5C=m39jP&C@Y z$76Gf{Wdb=Fg?x+VfY!xm)m*M={92tnPV=zjk0D}EC%Gde^Je&SodUSQ_m}RIMi_U zCa1S0H3{SR3!mlAz+EyVi^3?`1NU+*F=iP32Q*PO4uwAcpp~m7-=5`T#s5{a_E)G# zTpiak1iNpK@;NSZG3BjAGh1G8QjFbGjA+5yCg8Sh+qP{RyKURuyKURHZQHhO+n#-9 z=3GqjU(8JSpV)6BobDVAX)B{1Uq4OvG2~cglUY3lys3YcU>3Vd+ zQ#o&o`EAXMXVvVwlN>I-qXl?29ZjF6{}_LbP|Fvdi^Rl8<~T!)-^}4C>mFWki$yLO zCu600G$1YEJ5VH_7<0&C#Z_e%bfOk|HnHP};A`*_cOgxzXae0+#ck8~_Y&t)^{%Jg z7N?D{YAU(yV`t0Td4BLX=hV(YfIQh}W;tl@4?F^~392B;8?g#j+YR!2gg-Rd`xbRz zDYEK2Ia8?dp5bAK(wlD7eW6XI?!mdY1V|l}_}8KidwW0 z3wax(Y7>*Xe)#rKxT=pLFkwQPBSsA^ni{2zG4@M#VlA-G+^#EbTzA=?vd)2o7+ChYO(GF`DzP@#NsykzWkj zEa%6F>P+}CcR3d_J=166jvbg7=wHJJh=;=}@9h(38Dv``-ZNSy7Lt=$RK@HXlT4%) z|FZ)v#oLeLjC@U#Q}x}vpO9E(6aG%dN5j&Q+uK%M3#|&EcalZduY+EY#D@he_h8*X z;Z_>3;+b9;lmz&(xHU(?VH?}!qG~VmxHVfPP8p*2HuQ&yawW@G0xuqZ>wbpQS!AHe zd|kw#js2b+RLs{v4`p8~%05i0kradkdmO5h8fV@LG#NFM*bc4dljisCGY%lB^T(M5 z5(hL3ItOPVv_XRyLcqfG;-;9S%oj>8C*xKw8{8~yCzk&rXQm2!jFldVdNv8ml2UoW?YB0EW4!zMjX zR&i`Ipqqzh|B)RhCPt&!?ZU#%U>e`F>O}&7rge`?J%Bi8W`586Sj_AW825zJlF14I z-E~IPE({ZQ2mK{miwmajLs7uSi;@Ky)mmQl9n@~B`YZs0wS5Gx?%+l^zii^Nb-wAd z%KMx97M#P5WN6WXQ}B#ReYvP5|D3;P-cb`}KF?hBgX^v5u{mdejoybtNT0v|LECym z?5dsG9A4X3a^gTCO#IIOKCSl3pwWVpVd{^TUR4mrJ4zbuqLu?b}iW;CH zG#A{K7ou|@23&T=7P*$v86O7wERisDyM3bNy|b6Xab6F_CB|d;pha+53?5q>Az6;b zOmughO{sZXgW9nq>~Sx#!kUk{*UUx(I~#tr(&-3q_dsiS#Qa{ctxXZO zu~z-%+88`UEOsSq4XE6A%Ep`@D$befk(Wf_ZNUSTt*%i+1B;4;H*g{gXUHHIW2&lu zIT(?oNv~&&8|Nj{nyIeW2(G(uf!Tt6-<;S~+qs-8?7OJ&sjCkm8YNFo@Z0cJ1*bOS zP|=bCbxIV2Q*?S9c)MHgDc67A0u2L;aKFM`>?4@%8q7xey{JPz;69?}&NuiqwRYx@ z$X;km7Xx6%nJikbp)`4dFi}WV12GaZ|B4x9N=e@?Vvu`{ysFBL5@EJaT#y%dZ?hBg zyQg{yqKJ}c{FXHz&Zs<`v&RnJt>j!kCQx9LVqcIhm4lUexO)<5-AcCdu{s}ZAnmQ^ zBBHWZ%+O_rsT-Gp7WF1*e8qW3yKnC)BvooFU{^K(VGdG|F>jqck$tF7h5Ip_8Rk(nuX`~-@Vrc85z>#(wrfJ-%<ffM>P207Rng$RBYj)hF+5}Iy zxb_8_)N;%-e69;lk|a;ItEzg-`=|Qn%vXM`RFvytpWnsugBF}lBAZaUr9sLa%IuS88icA*bzR|mhG6@(R zVED*Jj<3UXNexejUCg35G-mtY6Tcy$05jXL0W67ezvK)nxfVV?GmltNs#riIj+rs4h2Z z(KfES3*hBDaqs>UbyYZBFTmHKA8TpBzjm`YQe?(26B9yF6wKl25%^jZ#_zELldF2j z=5x>EbGf2aZlduz!RXj$>pn1f`>B!a640zirWj%2S6DnMwC$4>JUDv zz)={A@yGh_Aw&q#$w`%aZE)Tlk=#LAE=g?+CyhO|hv~mdy!lP~5WKF*BHQ;>cVzt| zf7^KBxZrdfj4~5%Ko@Xru^5jy)@0A?+AZ z$(m=VI52LD!X7j%-KVJmRJoQG6qXd=iJjb?AL6KQ+-VJ>p0cC3@@AHII*F z;Tg<-PO|=;!R{EBBNa@i{>C=ap_Qc9?1;25GEZt}V{afuYQEPu&dBYmW zoe0Z+bYB_jDOpxm6QPqH=9S%OR{EIF<{S=ng(}5U@4BE6!sQOrm*C>nAV%0gYB86q zlUDPSB$8^(D1EapUc+TeGQS@wD)=ySOK+He&$E-W78C4JFR%8a%?=ZvG75!Ob}F7P zLK$sn%W;uf&+5Sq;0?{^;s$(ccGmHLvC%dM5TKyMf%(a*cxM)*uo4|rPZA&G!AASdczejU*C?+JW9AavXWt- z-f1Yy?uGbEsWQ7yAi=O5Oc&@Q0CvlLP$R#!6OH0q)RBTf&)( zm<7nrCh_5oJwJSAB~G!38T|ze9ZeLmrQv}7JZyy7tfz?Z0Gtm%q_|y07y~!I;F$BS z$Obj&$1CoLwI>kUmD{s$DBhBk zTO#zyMh@*~+(vCtw0Bm@nriYk)=j?G9Y|~$lNyT3mVroIhSWHBwRX1~sZ64iHH|CE zHCF>7JlJkqf5A+jzsWaQWP(K+4sRXYl3L@)qm-GK(u0dsXQRSH3O`5TI+!&bGaD|3d z|D2Wj1B8v?4L9k@_s1X77H!fy0O_Z-56VRE4)2b_p)jA^F$-zD5fw7e`i75ZHh@}R;Fq)S>DJ7Dq)GXG@tlF>5SX1c`n$~<5_>R>CfVf5q8PAphG(ic zmSx`R^u)Svx8>COP0c?k(1fb%Wg9c>@VNY6j-1?6zx+^aIqwwKTFl+3vqy919_K)? zd?_B1#znQWjq9z0oNi)@QM+uoeP#2)Ore?iToFTXM`9Ky2;DNjc0lY~Ejtm`_4 z$<)WZBdcfhq(>jpC8>b}`2snc`V0@&)VuZ>i;1Q!!MuooU;~wkqbfn zBYlHWQn)~OW)wb@G(u$nWu_IPIQ9JYc*m*y^UG|?35 z*n278n7cb8fXoWBh}dfF3u4vGfki9xIvASt24WPK+eh)WTF;BT#J)?%gI_3>tI%@y zAyWdXI`XmngR1#5Jm8Z`VQRw3gKDEe%^B}uJ8CIQjO1mabaE_Y|A}uG7qRp|Av)~; zBcj93_FpyUAELv;!NmOE>Hmx9FmN#b{~|i=E-IOu9lTL!Zm$9kZfRt=nBYV-mZQ!bk5FZE9jj(DDYXz_9c{NQH&|onZgq z_;hrF+Pn*WGwa#GJg6r7W>#0QIwmIO#zAF;FizlTohqEG;bNzV&U z5N2k09#8=&b9iSTw+uiT9zZAobR!peaRO++jW_zdD*#6W21c;%PX8A)wHJYmR}@BY zad|UpXz5gb)-?=_AC-?p+6jm_R<>pbd+_x?fKk`e01Z)4Q2?UguWp5?ejELG`an%A zEv(=iX~0YV8vwAZkgBYHNfikp4Op;;o(8Py;^N?^zxzZbB{hVi@CfLNDyaYftmlAA zSe6&Qy377Aw_mhpfDsq}RdlE&$|B1yDWvNdzT3e8p!;CYU>ZLDx1!VYe-)jr_5Pb$ zFaS$wbaZiDGO~Z~RSa4^4!SU~5wx)ITJLL#wpswd4crX7?{gW`9QvQ4Q}bf_PtmCd z`H!L_3OJ^f4sowT__fZ`!o)PD^dsR|IAG`E5y(2+fen+ z>CwU7)%cUIvyIX1p6%_o@AZk=i|L6kncd07SdoF<@ex!K;&1Mm8}ipK8+0DDA0~h^ z5CE?XZTe5sr!=ka6pioHtT@jq@^_1VeQr@kUT>Op9C@amSL;AY%Lo4*el{v_2r zex?&@bR{XjSXaGQOmri+8|d||b)Y@;6MAIP{F~!1I<2>6vCFxI(0zRaBM*8#Zju=r zYruj80NAg3bbFr|KaqEzSHE+h4CfM3)YO#=-}uvCsAL9q7ylmL4p9Gi4+Mjo8{2b$ zxfj|$IS_bn`k}^xId4=?3Xp+ugS)M(6aYs@#|!WTZsyKcEinhcBdWq0%i|F#411pmfokm}p_>d^KB+_Px^iQUk{ z{_q9;Q)->`i<9Y_4Kd|=dFIO}|1)4oHv`8eKug{DI=kQ-M1j%Cq2+PL)N5y!;`aG@ z=(~IVD-ImdD_`Ji92vpo?#^7-*a(R6$Kwwm%@xj-uWioeE5D=cuS?UfOHvo~kMDi1 z6Dk16UjW&C-)Zi&hu}Y@tf9%Dk|;k_ekFSbJmmvuODVE`AIa~Hi2&+6c9Ad?Dky7M zaiSVW1oYFDx2Z_xdIhqG0VbavD=FH%9!4cZT{c&xd*XgHgiI-w<7J3R#dR18^Ri-2 z6q#Ci$`gFq%Qc`^Gn(3AkG33xm$o~FG(a!|%YBuwr?h-I6sbpj`)7^-r2h`PudO9#mCo6WH? zbsvrj8AIcayMh_8oOU7ENvW19~rBohZf1L&>x~Y18w!{}Ag4SGfrlSc}CjrEOEsaC_og_=(T1KW)%3 zBXBx(YQn3;7b%i9OAw{nH(AQ+N8KiVANWcGjrNGT!zI#MWK<;)u`u+T+S2hYxNqBk zcOhNiP-%zOOZ%$O2Jk7q$3xM+m@tU4m+bWeBj+;ss1|H41kxG89RAin%H5Q^G?&Ch|%MVpAT?aTBTNf}zs(1)s*f zZ+Stubq{bh3;_@(p-B6oz7}ga|EA_al#F_ozg9XS*;fJ@?IUFT3h!qgOc~9r@sCD~Mz(Pj^9Mmfc!{u@?JAY(SAa zL<@#3Y9wI}zW)8rQ(8W`Ek;H-&x=e<@q}c(n}q)Wc;Ilrk=Q{qj!P0DXN-BCO`t?Ktr;ULsJSB|W&t>$x0h*SW)_c?{C~wBG*#lPBRW zxRpf!jt}Ke6{M4ustS&6Cm^i44cdPw%uWJ>CW0>E`FD4WiXC@RRoFi2vtG7p9lhb? zd7dj_cJ5lME>CQ-{hT94$&UGut|`c9UF)aOT{>10-j_r?m0=3tVW}3~jNMtk>v)lxjlyRvhMHkvQ+-m#YFD!0b&ITp8h& zsx$|yVk3Q5@Zj)11yV3exL4mLTchw=C8Ov_mvS~0D*6XTH&RwQ*MC-{@qkC^dDTV? z%?fj(_|jWwaIn+?5-E5}0fBtl@zZJr}X#*wlkM&YFr zTc%G&gOP##&%447x7hsC;LAQEB{3=}dW#~C+Z9f0RcGaH=4-p$kP0N_(OSZj8(Klu ziY5c3R4gEH6)?1#&e_^_vt+Gtzq2TH5F^ik&bcKzZbw*X|BNfWH&;e<6PX&e7+%u7 z;5k)vk+Ih}#}_!vquna^#$Q$hdl^g;vQ0Nl8tdvqkky`j*U!c=>>Xuin+q60& zWKmhi`dc&j$(zngOmZcX1R1ED0Y^ zFKh^-pC~P%FOvJZb5w+@?;DCq<-OG(k+0{7)o&hH&sJn~Jcf)IUq!}}9n5BUTtNSn zsG#;v?j~Tx@W;Js0&e{cMpGqdGz`6st<5h3 z*B7~Dm^cf`Ru`iT_VdDEX)TcgmX>EHpgGs2GUTkErk7O8M>ay@UXX%cz5l`qm!wl- zftfD9q+N569gog&T4&X2z9~0gf9YCPkDAk%Zuh%gVO*(t`iai0+Hjuzb5ePzL-K*D z5j&IicLn7eD^Uebe>ns!LcAWu%L{|6ZlTgazN=>tKn-smP47ux8Q%_WtaC&$JdBgE zNN-$-)QlxvYSQ4^{Hskc!yvYl5)qwjL=V)6aj0axV0|ICGp@+5P#KRj_N#(J2Sq7w zEHPTElxxi{v*+ROmOQkf#a9a?q;}?6aH_oGAPmM{dPCSw|Gav(HC?wuvfV)a2Q2F! zUN*~PdBDWHSR*Iap|D{i_wnj!r`76{T?N8d5DZBG_aB?50#v02T!%deqcr?v#A;{& zFuVQQG2r!=wV13A^%qEd#5=CGU_WF_dj%7gOtj=&m~Zkv(tY~Vn}6&_UMZ&8pp%;T zlH&8=BGR4?hgM**o)_2&9&pgPo2bs?RcBmJ<@k6-dcDna2o%xPBEGoxmRNdtb0uE{{)qdX3(>n}1Zvc547a z#Cg=l^JW7Qu|2>yLmd+stau@hT$R=&)i0Cky!%;b#E>k(Yyjjm-#BD&R+V>^C|xZ; z&M9R|tJ*{4*X(o1V9;g%p3#SQ6?mwm3=LCrKs-Y7lQ(YaU(a2!)Svm(m6UXeffiz* z!GHp2U@}=@urPUcVVFIvo}YKgr=mS-kZOyS>gGZ74)HRVVW(~ZXW!G*=?OFTK(B~N zw7k*IoH{gZ8A}pU-fXFtSL|eIxg1IIqMsF|!4T0Y#Irj#R|OPet!C3R1-Ou*0LGfX z99MI1L>F!%YSv4l4{K)^OlY{~GGq*E2~R!so=~;?N*wumV0fE<{$#D6BFoRBy!;JO z`$)6b|Hm-_oQ4O_*1_&;NvmVWnb$;2YhU%SJZGf?QVNP+Fw8S%hJZ&Bwh5qvtCcN1Mp2+j1qjp0Yl*cY_g?CR)+DbETAcoh!r zVFz;lu1OA7 zc$Nx++A6tN|BL_jXRol}%d5skc0a_o#1oUEWuPS`PLXBb(F@Z=8nORE~b21d#|pCZ~x2fS}zn5 zf;%DLa1?SAZ93L~nnJ;Vq}*6uEAa_@Eoi6_S30oK)5=MuxSr|qP*tL7fEzi;EZNn? z5n!vzb&%=tni@a{On$liJDQu?GyS{w#IL|5F{?0H*F>#%U&^CV1GJ+) znAUU2BrBF!#||6n_lzR%0QfFZtevi(XZeD44w;`LfD}P`JF01)sq;x`VO^YzvenB> zlG2GKFpSEhAs>h8#vct!D$U>UbS^6)MItOZF(m5q!XH7$^@Mk7)u+6!lK61n8&l-W59&{{8~4=n#$=R0@bAh* zEH@MGF=tOhSboIxjb=?59-4JuRe*Eu0Tp3qb7fDgR2PxzA9O->@}BYJu85u`gIM`j zhV9&hitO{&2ToFFggzo1duid#sam|O`ntT$Z1XAyc4KLmMNdV+N_z^`dTd2JG%$?z z_zJeY@aTiXg!>Azxeml|B;1n`w056ZcQif8szmjw#pC$I;<*qVQRyjQv9*S8Oul+> z=aa5K`-TWY15S*^~&Z; zE@Hm=OH0;n}^pAMeelz+Y zn}??a6utn;z)Y_P1z`hf9z2RRzGoTqpz1YpymTsKdloYBF1D~$*Q!tD+EYlx6`T&J zzyWW%LPX*+n?@GN2v;Bk!piKuw-dnLmqb}h2`_$nK-CX>&Vfo9oz{bueWR%jXYhVb zNqCArCzO3g>q_X(6Mpg5&e1noEZbFNXKu5eK36%hzH!0=pkH&rTk;`#-rpp#;cK#N z`m%iI)Qj~W>}hL;ec);~1NzF1o%u+yEz%}a@wMESu%ah1k&BM;k-Z}HC5o*DB1ouw zDF;Aji9=LL>K7u%YiVQgT?pM<=;)n05!BHwsr3>Na!`V%579HI1t)=38FGhmVgJ&) zrv90l6voTI2na(?T#Ex(7yO8C7n#4iVZ1YS)6s~dDRRkl9HqRrY?O6T>E(*jY zw`6^;kXr7X$*W_UmfmGn2J@E=sxTbt_0Z9P8H(C+X-1uSp6~|1QArbx+zrU$Isd8YGy<#&AQ)5){4%WdGuU zvZK>G1k+aiiEX8TL%|l$AAyaiFY}AcWRb;T0Zqpf)h=luu1N=z&sm>nD%$F>QCM(M8RS6qSP(m*15(r>lbTeiECn~w{HIy72cOUL zLS-BW&mC`iDLH4ac^;BHG?qf@N65~zniIQ>)L7a=cYL$h6B>yBS#9|^czfblHS{nM z2anr24#da=$kB4*98h%<%0=W=S^G5o9H7{QfKoto#ASjV4lf((*7)_kb6Q$H8f|zv z=z!V+g=E%n-<&?hs`%rtn;|wEyU#_p@vd1L?q@^;QSti5E@bvj2YG1<J3Si1{h^W{DgdEj}_5IjguZH0wUdSdB?l9 z-z6r@n6=80^S}BrKaG`&*m4KY$5b*p9-yS&_J3Rs9OfEG!J~8=4OEkE;{;tutWUG@ zTX!R%v#YhO0|^-q*hI-hi^CAcWxg(rf_ssnlSWAL*^|-&8?fcOIMhtb=~W+w0odI} zcw6%_4z?jqJH7*UnPq+xfQz0{_Mt$)eU75DuDd)Mw-3z4k#EtxvQ%iC*tk7#-`=?@>_ z>g91FTlF|>=6MU|aurWylF%+lx9=jGaFPnaaJ?SDE@7fgMtMow~tW0751LDt3H0v5WOrR510FgQT+9*E*^X zWQgN6MS+QRP~)iDG+?E6W9nURTqr^cY}cHRo_Pbd0t8Tc_d2a|(6t4bAtX6aTpcm1 zXAK?$`XZm8bY}hDb-WWAP?;%Oq}FdmZHez+t=U*$h1$p-LUz=0I9ve8cx3QG|Dbn{ zp&>hwl758d$mMD&8c*zNVrVHq|B>8gft)kwpccrqiQ>_w#_of|;UGAH-Qo7jN|>Ez0f#_GrNY>{^0u zJ%3u`y*g*||B3(0NqW?-A|gX^e@-|wLtweU^!zPxp)GlnBXta3gSeK1*RmcrKtq
+ + + +
+ + + + + + + + + + + + + + +
+ + + + +

Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis

+ +

+ + +

+Morten Hjorth-Jensen [1, 2] +
+ +

 
+ + +

[1] Department of Physics, University of Oslo
+
[2] Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University
+
+

 
+

Jul 22, 2019

+
+

+ +

+ © 1999-2019, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license +
+
+ + +
+

Why Linear Regression (aka Ordinary Least Squares and family)

+ +

+Fitting a continuous function with linear parameterization in terms of the parameters \( \boldsymbol{\beta} \). + +

    +

  • Method of choice for fitting a continuous function!
    • +

  • Gives an excellent introduction to central Machine Learning features with understandable pedagogical links to other methods like Neural Networks, Support Vector Machines etc
    • +

  • Analytical expression for the fitting parameters \( \boldsymbol{\beta} \)
    • +

  • Analytical expressions for statistical propertiers like mean values, variances, confidence intervals and more
    • +

  • Analytical relation with probabilistic interpretations
    • +

  • Easy to introduce basic concepts like bias-variance tradeoff, cross-validation, resampling and regularization techniques and many other ML topics
    • +

  • Easy to code! And links well with classification problems and logistic regression and neural networks
    • +

  • Allows for easy hands-on understanding of gradient descent methods
    • +

  • and many more features
    • +
+

+ +For more discussions of Ridge and Lasso regression, Wessel van Wieringen's article is highly recommended. +Similarly, Mehta et al's article is also recommended. +

+ + +
+

Regression analysis, overarching aims

+
+ +

+Regression modeling deals with the description of the sampling distribution of a given random variable \( y \) and how it varies as function of another variable or a set of such variables \( \boldsymbol{x} =[x_0, x_1,\dots, x_{n-1}]^T \). +The first variable is called the dependent, the outcome or the response variable while the set of variables \( \boldsymbol{x} \) is called the independent variable, or the predictor variable or the explanatory variable. + +

+A regression model aims at finding a likelihood function \( p(\boldsymbol{y}\vert \boldsymbol{x}) \), that is the conditional distribution for \( \boldsymbol{y} \) with a given \( \boldsymbol{x} \). The estimation of \( p(\boldsymbol{y}\vert \boldsymbol{x}) \) is made using a data set with + +

    +

  • \( n \) cases \( i = 0, 1, 2, \dots, n-1 \)
    • +

  • Response (target, dependent or outcome) variable \( y_i \) with \( i = 0, 1, 2, \dots, n-1 \)
    • +

  • \( p \) so-called explanatory (independent or predictor) variables \( \boldsymbol{x}_i=[x_{i0}, x_{i1}, \dots, x_{ip-1}] \) with \( i = 0, 1, 2, \dots, n-1 \) and explanatory variables running from \( 0 \) to \( p-1 \). See below for more explicit examples.
    • +
+

+ + The goal of the regression analysis is to extract/exploit relationship between \( \boldsymbol{y} \) and \( \boldsymbol{X} \) in or to infer causal dependencies, approximations to the likelihood functions, functional relationships and to make predictions, making fits and many other things. +

+
+ + +
+

Regression analysis, overarching aims II

+
+ +

+Consider an experiment in which \( p \) characteristics of \( n \) samples are +measured. The data from this experiment, for various explanatory variables \( p \) are normally represented by a matrix +\( \mathbf{X} \). + +

+The matrix \( \mathbf{X} \) is called the design +matrix. Additional information of the samples is available in the +form of \( \boldsymbol{y} \) (also as above). The variable \( \boldsymbol{y} \) is +generally referred to as the response variable. The aim of +regression analysis is to explain \( \boldsymbol{y} \) in terms of +\( \boldsymbol{X} \) through a functional relationship like \( y_i = +f(\mathbf{X}_{i,\ast}) \). When no prior knowledge on the form of +\( f(\cdot) \) is available, it is common to assume a linear relationship +between \( \boldsymbol{X} \) and \( \boldsymbol{y} \). This assumption gives rise to +the linear regression model where \( \boldsymbol{\beta} = [\beta_0, \ldots, +\beta_{p-1}]^{T} \) are the regression parameters. + +

+Linear regression gives us a set of analytical equations for the parameters \( \beta_j \). + + +

+
+ + +
+

Examples

+
+ +

+In order to understand the relation among the predictors \( p \), the set of data \( n \) and the target (outcome, output etc) \( \boldsymbol{y} \), +consider the model we discussed for describing nuclear binding energies. + +

+There we assumed that we could parametrize the data using a polynomial approximation based on the liquid drop model. +Assuming +

 
+$$ +BE(A) = a_0+a_1A+a_2A^{2/3}+a_3A^{-1/3}+a_4A^{-1}, +$$ +

 
+ +we have five predictors, that is the intercept, the \( A \) dependent term, the \( A^{2/3} \) term and the \( A^{-1/3} \) and \( A^{-1} \) terms. +This gives \( p=0,1,2,3,4 \). Furthermore we have \( n \) entries for each predictor. It means that our design matrix is a +\( p\times n \) matrix \( \boldsymbol{X} \). + +

+Here the predictors are based on a model we have made. A popular data set which is widely encountered in ML applications is the +so-called credit card default data from Taiwan. The data set contains data on \( n=30000 \) credit card holders with predictors like gender, marital status, age, profession, education, etc. In total there are \( 24 \) such predictors or attributes leading to a design matrix of dimensionality \( 24 \times 30000 \) + + +

+
+ + +
+

General linear models

+
+ +

+Before we proceed let us study a case from linear algebra where we aim at fitting a set of data \( \boldsymbol{y}=[y_0,y_1,\dots,y_{n-1}] \). We could think of these data as a result of an experiment or a complicated numerical experiment. These data are functions of a series of variables \( \boldsymbol{x}=[x_0,x_1,\dots,x_{n-1}] \), that is \( y_i = y(x_i) \) with \( i=0,1,2,\dots,n-1 \). The variables \( x_i \) could represent physical quantities like time, temperature, position etc. We assume that \( y(x) \) is a smooth function. + +

+Since obtaining these data points may not be trivial, we want to use these data to fit a function which can allow us to make predictions for values of \( y \) which are not in the present set. The perhaps simplest approach is to assume we can parametrize our function in terms of a polynomial of degree \( n-1 \) with \( n \) points, that is +

 
+$$ +y=y(x) \rightarrow y(x_i)=\tilde{y}_i+\epsilon_i=\sum_{j=0}^{n-1} \beta_j x_i^j+\epsilon_i, +$$ +

 
+ +where \( \epsilon_i \) is the error in our approximation. + + +

+
+ + +
+

Rewriting the fitting procedure as a linear algebra problem

+
+ +

+For every set of values \( y_i,x_i \) we have thus the corresponding set of equations +

 
+$$ +\begin{align*} +y_0&=\beta_0+\beta_1x_0^1+\beta_2x_0^2+\dots+\beta_{n-1}x_0^{n-1}+\epsilon_0\\ +y_1&=\beta_0+\beta_1x_1^1+\beta_2x_1^2+\dots+\beta_{n-1}x_1^{n-1}+\epsilon_1\\ +y_2&=\beta_0+\beta_1x_2^1+\beta_2x_2^2+\dots+\beta_{n-1}x_2^{n-1}+\epsilon_2\\ +\dots & \dots \\ +y_{n-1}&=\beta_0+\beta_1x_{n-1}^1+\beta_2x_{n-1}^2+\dots+\beta_{n-1}x_{n-1}^{n-1}+\epsilon_{n-1}.\\ +\end{align*} +$$ +

 
+

+
+ + +
+

Rewriting the fitting procedure as a linear algebra problem, more details

+
+ +

+Defining the vectors +

 
+$$ +\boldsymbol{y} = [y_0,y_1, y_2,\dots, y_{n-1}]^T, +$$ +

 
+ +and +

 
+$$ +\boldsymbol{\beta} = [\beta_0,\beta_1, \beta_2,\dots, \beta_{n-1}]^T, +$$ +

 
+ +and +

 
+$$ +\boldsymbol{\epsilon} = [\epsilon_0,\epsilon_1, \epsilon_2,\dots, \epsilon_{n-1}]^T, +$$ +

 
+ +and the design matrix +

 
+$$ +\boldsymbol{X}= +\begin{bmatrix} +1& x_{0}^1 &x_{0}^2& \dots & \dots &x_{0}^{n-1}\\ +1& x_{1}^1 &x_{1}^2& \dots & \dots &x_{1}^{n-1}\\ +1& x_{2}^1 &x_{2}^2& \dots & \dots &x_{2}^{n-1}\\ +\dots& \dots &\dots& \dots & \dots &\dots\\ +1& x_{n-1}^1 &x_{n-1}^2& \dots & \dots &x_{n-1}^{n-1}\\ +\end{bmatrix} +$$ +

 
+ +we can rewrite our equations as +

 
+$$ +\boldsymbol{y} = \boldsymbol{X}\boldsymbol{\beta}+\boldsymbol{\epsilon}. +$$ +

 
+ +The above design matrix is called a Vandermonde matrix. +

+
+ + +
+

Generalizing the fitting procedure as a linear algebra problem

+
+ +

+We are obviously not limited to the above polynomial expansions. We +could replace the various powers of \( x \) with elements of Fourier +series or instead of \( x_i^j \) we could have \( \cos{(j x_i)} \) or \( \sin{(j +x_i)} \), or time series or other orthogonal functions. For every set +of values \( y_i,x_i \) we can then generalize the equations to + +

 
+$$ +\begin{align*} +y_0&=\beta_0x_{00}+\beta_1x_{01}+\beta_2x_{02}+\dots+\beta_{n-1}x_{0n-1}+\epsilon_0\\ +y_1&=\beta_0x_{10}+\beta_1x_{11}+\beta_2x_{12}+\dots+\beta_{n-1}x_{1n-1}+\epsilon_1\\ +y_2&=\beta_0x_{20}+\beta_1x_{21}+\beta_2x_{22}+\dots+\beta_{n-1}x_{2n-1}+\epsilon_2\\ +\dots & \dots \\ +y_{i}&=\beta_0x_{i0}+\beta_1x_{i1}+\beta_2x_{i2}+\dots+\beta_{n-1}x_{in-1}+\epsilon_i\\ +\dots & \dots \\ +y_{n-1}&=\beta_0x_{n-1,0}+\beta_1x_{n-1,2}+\beta_2x_{n-1,2}+\dots+\beta_{n-1}x_{n-1,n-1}+\epsilon_{n-1}.\\ +\end{align*} +$$ +

 
+ +

+Note that we have \( p=n \) here. The matrix is symmetric. This is generally not the case! +

+
+ + +
+

Generalizing the fitting procedure as a linear algebra problem

+
+ +

+We redefine in turn the matrix \( \boldsymbol{X} \) as +

 
+$$ +\boldsymbol{X}= +\begin{bmatrix} +x_{00}& x_{01} &x_{02}& \dots & \dots &x_{0,n-1}\\ +x_{10}& x_{11} &x_{12}& \dots & \dots &x_{1,n-1}\\ +x_{20}& x_{21} &x_{22}& \dots & \dots &x_{2,n-1}\\ +\dots& \dots &\dots& \dots & \dots &\dots\\ +x_{n-1,0}& x_{n-1,1} &x_{n-1,2}& \dots & \dots &x_{n-1,n-1}\\ +\end{bmatrix} +$$ +

 
+ +and without loss of generality we rewrite again our equations as +

 
+$$ +\boldsymbol{y} = \boldsymbol{X}\boldsymbol{\beta}+\boldsymbol{\epsilon}. +$$ +

 
+ +The left-hand side of this equation is kwown. Our error vector \( \boldsymbol{\epsilon} \) and the parameter vector \( \boldsymbol{\beta} \) are our unknow quantities. How can we obtain the optimal set of \( \beta_i \) values? +

+
+ + +
+

Optimizing our parameters

+
+ +

+We have defined the matrix \( \boldsymbol{X} \) via the equations +

 
+$$ +\begin{align*} +y_0&=\beta_0x_{00}+\beta_1x_{01}+\beta_2x_{02}+\dots+\beta_{n-1}x_{0n-1}+\epsilon_0\\ +y_1&=\beta_0x_{10}+\beta_1x_{11}+\beta_2x_{12}+\dots+\beta_{n-1}x_{1n-1}+\epsilon_1\\ +y_2&=\beta_0x_{20}+\beta_1x_{21}+\beta_2x_{22}+\dots+\beta_{n-1}x_{2n-1}+\epsilon_1\\ +\dots & \dots \\ +y_{i}&=\beta_0x_{i0}+\beta_1x_{i1}+\beta_2x_{i2}+\dots+\beta_{n-1}x_{in-1}+\epsilon_1\\ +\dots & \dots \\ +y_{n-1}&=\beta_0x_{n-1,0}+\beta_1x_{n-1,2}+\beta_2x_{n-1,2}+\dots+\beta_{n-1}x_{n-1,n-1}+\epsilon_{n-1}.\\ +\end{align*} +$$ +

 
+ +

+As we noted above, we stayed with a system with the design matrix + \( \boldsymbol{X}\in {\mathbb{R}}^{n\times n} \), that is we have \( p=n \). For reasons to come later (algorithmic arguments) we will hereafter define +our matrix as \( \boldsymbol{X}\in {\mathbb{R}}^{n\times p} \), with the predictors refering to the column numbers and the entries \( n \) being the row elements. + + +

+
+ + +
+

Our model for the nuclear binding energies

+ +

+In our introductory notes we looked at the so-called liguid drop model. Let us remind ourselves about what we did by looking at the code. + +

+We restate the parts of the code we are most interested in. +

+ + +

# Common imports
+import numpy as np
+import pandas as pd
+import matplotlib.pyplot as plt
+from IPython.display import display
+import os
+
+# Where to save the figures and data files
+PROJECT_ROOT_DIR = "Results"
+FIGURE_ID = "Results/FigureFiles"
+DATA_ID = "DataFiles/"
+
+if not os.path.exists(PROJECT_ROOT_DIR):
+    os.mkdir(PROJECT_ROOT_DIR)
+
+if not os.path.exists(FIGURE_ID):
+    os.makedirs(FIGURE_ID)
+
+if not os.path.exists(DATA_ID):
+    os.makedirs(DATA_ID)
+
+def image_path(fig_id):
+    return os.path.join(FIGURE_ID, fig_id)
+
+def data_path(dat_id):
+    return os.path.join(DATA_ID, dat_id)
+
+def save_fig(fig_id):
+    plt.savefig(image_path(fig_id) + ".png", format='png')
+
+infile = open(data_path("MassEval2016.dat"),'r')
+
+
+# Read the experimental data with Pandas
+Masses = pd.read_fwf(infile, usecols=(2,3,4,6,11),
+              names=('N', 'Z', 'A', 'Element', 'Ebinding'),
+              widths=(1,3,5,5,5,1,3,4,1,13,11,11,9,1,2,11,9,1,3,1,12,11,1),
+              header=39,
+              index_col=False)
+
+# Extrapolated values are indicated by '#' in place of the decimal place, so
+# the Ebinding column won't be numeric. Coerce to float and drop these entries.
+Masses['Ebinding'] = pd.to_numeric(Masses['Ebinding'], errors='coerce')
+Masses = Masses.dropna()
+# Convert from keV to MeV.
+Masses['Ebinding'] /= 1000
+
+# Group the DataFrame by nucleon number, A.
+Masses = Masses.groupby('A')
+# Find the rows of the grouped DataFrame with the maximum binding energy.
+Masses = Masses.apply(lambda t: t[t.Ebinding==t.Ebinding.max()])
+A = Masses['A']
+Z = Masses['Z']
+N = Masses['N']
+Element = Masses['Element']
+Energies = Masses['Ebinding']
+
+# Now we set up the design matrix X
+X = np.zeros((len(A),5))
+X[:,0] = 1
+X[:,1] = A
+X[:,2] = A**(2.0/3.0)
+X[:,3] = A**(-1.0/3.0)
+X[:,4] = A**(-1.0)
+# Then nice printout using pandas
+DesignMatrix = pd.DataFrame(X)
+DesignMatrix.index = A
+DesignMatrix.columns = ['1', 'A', 'A^(2/3)', 'A^(-1/3)', '1/A']
+display(DesignMatrix)
+
+

+With \( \boldsymbol{\beta}\in {\mathbb{R}}^{p\times 1} \), it means that we will hereafter write our equations for the approximation as +

 
+$$ +\boldsymbol{\tilde{y}}= \boldsymbol{X}\boldsymbol{\beta}, +$$ +

 
+ +throughout these lectures. +

+ + +
+

Optimizing our parameters, more details

+
+ +

+With the above we use the design matrix to define the approximation \( \boldsymbol{\tilde{y}} \) via the unknown quantity \( \boldsymbol{\beta} \) as +

 
+$$ +\boldsymbol{\tilde{y}}= \boldsymbol{X}\boldsymbol{\beta}, +$$ +

 
+ +and in order to find the optimal parameters \( \beta_i \) instead of solving the above linear algebra problem, we define a function which gives a measure of the spread between the values \( y_i \) (which represent hopefully the exact values) and the parameterized values \( \tilde{y}_i \), namely +

 
+$$ +C(\boldsymbol{\beta})=\frac{1}{n}\sum_{i=0}^{n-1}\left(y_i-\tilde{y}_i\right)^2=\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{\tilde{y}}\right)^T\left(\boldsymbol{y}-\boldsymbol{\tilde{y}}\right)\right\}, +$$ +

 
+ +or using the matrix \( \boldsymbol{X} \) and in a more compact matrix-vector notation as +

 
+$$ +C(\boldsymbol{\beta})=\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{X}^T\boldsymbol{\beta}\right)^T\left(\boldsymbol{y}-\boldsymbol{X}^T\boldsymbol{\beta}\right)\right\}. +$$ +

 
+ +This function is one possible way to define the so-called cost function. + +

+It is also common to define +the function \( Q \) as + +

 
+$$ +C(\boldsymbol{\beta})=\frac{1}{2n}\sum_{i=0}^{n-1}\left(y_i-\tilde{y}_i\right)^2, +$$ +

 
+ +since when taking the first derivative with respect to the unknown parameters \( \beta \), the factor of \( 2 \) cancels out. +

+
+ + +
+

Interpretations and optimizing our parameters

+
+ +

+The function +

 
+$$ +C(\boldsymbol{\beta})=\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)^T\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)\right\}, +$$ +

 
+ +can be linked to the variance of the quantity \( y_i \) if we interpret the latter as the mean value. +When linking below with the maximum likelihood approach below, we will indeed interpret \( y_i \) as a mean value (see exercises) +

 
+$$ +y_{i}=\langle y_i \rangle = \beta_0x_{i,0}+\beta_1x_{i,1}+\beta_2x_{i,2}+\dots+\beta_{n-1}x_{i,n-1}+\epsilon_i, +$$ +

 
+ +

+where \( \langle y_i \rangle \) is the mean value. Keep in mind also that +till now we have treated \( y_i \) as the exact value. Normally, the +response (dependent or outcome) variable \( y_i \) the outcome of a +numerical experiment or another type of experiment and is thus only an +approximation to the true value. It is then always accompanied by an +error estimate, often limited to a statistical error estimate given by +the standard deviation discussed earlier. In the discussion here we +will treat \( y_i \) as our exact value for the response variable. + +

+In order to find the parameters \( \beta_i \) we will then minimize the spread of \( C(\boldsymbol{\beta}) \), that is we are going to solve the problem +

 
+$$ +{\displaystyle \min_{\boldsymbol{\beta}\in +{\mathbb{R}}^{p}}}\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)^T\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)\right\}. +$$ +

 
+ +In practical terms it means we will require +

 
+$$ +\frac{\partial C(\boldsymbol{\beta})}{\partial \beta_j} = \frac{\partial }{\partial \beta_j}\left[ \frac{1}{n}\sum_{i=0}^{n-1}\left(y_i-\beta_0x_{i,0}-\beta_1x_{i,1}-\beta_2x_{i,2}-\dots-\beta_{n-1}x_{i,n-1}\right)^2\right]=0, +$$ +

 
+ +which results in +

 
+$$ +\frac{\partial C(\boldsymbol{\beta})}{\partial \beta_j} = -\frac{2}{n}\left[ \sum_{i=0}^{n-1}x_{ij}\left(y_i-\beta_0x_{i,0}-\beta_1x_{i,1}-\beta_2x_{i,2}-\dots-\beta_{n-1}x_{i,n-1}\right)\right]=0, +$$ +

 
+ +or in a matrix-vector form as +

 
+$$ +\frac{\partial C(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}} = 0 = \boldsymbol{X}^T\left( \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right). +$$ +

 
+ + +

+
+ + +
+

Interpretations and optimizing our parameters

+
+ +

+We can rewrite +

 
+$$ +\frac{\partial C(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}} = 0 = \boldsymbol{X}^T\left( \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right), +$$ +

 
+ +as +

 
+$$ +\boldsymbol{X}^T\boldsymbol{y} = \boldsymbol{X}^T\boldsymbol{X}\boldsymbol{\beta}, +$$ +

 
+ +and if the matrix \( \boldsymbol{X}^T\boldsymbol{X} \) is invertible we have the solution +

 
+$$ +\boldsymbol{\beta} =\left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}. +$$ +

 
+ +

+We note also that since our design matrix is defined as \( \boldsymbol{X}\in +{\mathbb{R}}^{n\times p} \), the product \( \boldsymbol{X}^T\boldsymbol{X} \in +{\mathbb{R}}^{p\times p} \). In the above case we have that \( p \ll n \), +in our case \( p=5 \) meaning that we end up with inverting a small +\( 5\times 5 \) matrix. This is a rather common situation, in many cases we end up with low-dimensional +matrices to invert. The methods discussed here and for many other +supervised learning algorithms like classification with logistic +regression or support vector machines, exhibit dimensionalities which +allow for the usage of direct linear algebra methods such as LU decomposition or Singular Value Decomposition (SVD) for finding the inverse of the matrix +\( \boldsymbol{X}^T\boldsymbol{X} \). + + +

+
+ + +
+

Interpretations and optimizing our parameters

+
+ +

+The residuals \( \boldsymbol{\epsilon} \) are in turn given by +

 
+$$ +\boldsymbol{\epsilon} = \boldsymbol{y}-\boldsymbol{\tilde{y}} = \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}, +$$ +

 
+ +and with +

 
+$$ +\boldsymbol{X}^T\left( \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)= 0, +$$ +

 
+ +we have +

 
+$$ +\boldsymbol{X}^T\boldsymbol{\epsilon}=\boldsymbol{X}^T\left( \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)= 0, +$$ +

 
+ +meaning that the solution for \( \boldsymbol{\beta} \) is the one which minimizes the residuals. Later we will link this with the maximum likelihood approach. + + +

+ +

+Let us now return to our nuclear binding energies and simply code the above equations. +

+ + +
+

Own code for Ordinary Least Squares

+ +

+It is rather straightforward to implement the matrix inversion and obtain the parameters \( \boldsymbol{\beta} \). After having defined the matrix \( \boldsymbol{X} \) we simply need to +write +

+ + +

# matrix inversion to find beta
+beta = np.linalg.inv(X.T.dot(X)).dot(X.T).dot(Energies)
+# and then make the prediction
+ytilde = X @ beta
+
+

+Alternatively, you can use the least squares functionality in Numpy as +

+ + +

fit = np.linalg.lstsq(X, Energies, rcond =None)[0]
+ytildenp = np.dot(fit,X.T)
+
+

+And finally we plot our fit with and compare with data +

+ + +

Masses['Eapprox']  = ytilde
+# Generate a plot comparing the experimental with the fitted values values.
+fig, ax = plt.subplots()
+ax.set_xlabel(r'$A = N + Z$')
+ax.set_ylabel(r'$E_\mathrm{bind}\,/\mathrm{MeV}$')
+ax.plot(Masses['A'], Masses['Ebinding'], alpha=0.7, lw=2,
+            label='Ame2016')
+ax.plot(Masses['A'], Masses['Eapprox'], alpha=0.7, lw=2, c='m',
+            label='Fit')
+ax.legend()
+save_fig("Masses2016OLS")
+plt.show()
+
+
+ + +
+

Adding error analysis and training set up

+ +

+We can easily test our fit by computing the \( R2 \) score that we discussed in connection with the functionality of _Scikit_Learn_ in the introductory slides. +Since we are not using _Scikit-Learn here we can define our own \( R2 \) function as +

+ + +

def R2(y_data, y_model):
+    return 1 - np.sum((y_data - y_model) ** 2) / np.sum((y_data - np.mean(y_model)) ** 2)
+
+

+and we would be using it as +

+ + +

print(R2(Energies,ytilde))
+
+

+We can easily add our MSE score as +

+ + +

def MSE(y_data,y_model):
+    n = np.size(y_model)
+    return np.sum((y_data-y_model)**2)/n
+
+print(MSE(Energies,ytilde))
+
+

+and finally the relative error as +

+ + +

def RelativeError(y_data,y_model):
+    return abs((y_data-y_model)/y_data)
+print(RelativeError(Energies, ytilde))
+
+
+ + +
+

The \( \chi^2 \) function

+
+ +

+Normally, the response (dependent or outcome) variable \( y_i \) is the +outcome of a numerical experiment or another type of experiment and is +thus only an approximation to the true value. It is then always +accompanied by an error estimate, often limited to a statistical error +estimate given by the standard deviation discussed earlier. In the +discussion here we will treat \( y_i \) as our exact value for the +response variable. + +

+Introducing the standard deviation \( \sigma_i \) for each measurement +\( y_i \), we define now the \( \chi^2 \) function (omitting the \( 1/n \) term) +as + +

 
+$$ +\chi^2(\boldsymbol{\beta})=\frac{1}{n}\sum_{i=0}^{n-1}\frac{\left(y_i-\tilde{y}_i\right)^2}{\sigma_i^2}=\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{\tilde{y}}\right)^T\frac{1}{\boldsymbol{\Sigma^2}}\left(\boldsymbol{y}-\boldsymbol{\tilde{y}}\right)\right\}, +$$ +

 
+ +where the matrix \( \boldsymbol{\Sigma} \) is a diagonal matrix with \( \sigma_i \) as matrix elements. + + +

+
+ + +
+

The \( \chi^2 \) function

+
+ +

+In order to find the parameters \( \beta_i \) we will then minimize the spread of \( \chi^2(\boldsymbol{\beta}) \) by requiring +

 
+$$ +\frac{\partial \chi^2(\boldsymbol{\beta})}{\partial \beta_j} = \frac{\partial }{\partial \beta_j}\left[ \frac{1}{n}\sum_{i=0}^{n-1}\left(\frac{y_i-\beta_0x_{i,0}-\beta_1x_{i,1}-\beta_2x_{i,2}-\dots-\beta_{n-1}x_{i,n-1}}{\sigma_i}\right)^2\right]=0, +$$ +

 
+ +which results in +

 
+$$ +\frac{\partial \chi^2(\boldsymbol{\beta})}{\partial \beta_j} = -\frac{2}{n}\left[ \sum_{i=0}^{n-1}\frac{x_{ij}}{\sigma_i}\left(\frac{y_i-\beta_0x_{i,0}-\beta_1x_{i,1}-\beta_2x_{i,2}-\dots-\beta_{n-1}x_{i,n-1}}{\sigma_i}\right)\right]=0, +$$ +

 
+ +or in a matrix-vector form as +

 
+$$ +\frac{\partial \chi^2(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}} = 0 = \boldsymbol{A}^T\left( \boldsymbol{b}-\boldsymbol{A}\boldsymbol{\beta}\right). +$$ +

 
+ +where we have defined the matrix \( \boldsymbol{A} =\boldsymbol{X}/\boldsymbol{\Sigma} \) with matrix elements \( a_{ij} = x_{ij}/\sigma_i \) and the vector \( \boldsymbol{b} \) with elements \( b_i = y_i/\sigma_i \). +

+
+ + +
+

The \( \chi^2 \) function

+
+ +

+We can rewrite +

 
+$$ +\frac{\partial \chi^2(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}} = 0 = \boldsymbol{A}^T\left( \boldsymbol{b}-\boldsymbol{A}\boldsymbol{\beta}\right), +$$ +

 
+ +as +

 
+$$ +\boldsymbol{A}^T\boldsymbol{b} = \boldsymbol{A}^T\boldsymbol{A}\boldsymbol{\beta}, +$$ +

 
+ +and if the matrix \( \boldsymbol{A}^T\boldsymbol{A} \) is invertible we have the solution +

 
+$$ +\boldsymbol{\beta} =\left(\boldsymbol{A}^T\boldsymbol{A}\right)^{-1}\boldsymbol{A}^T\boldsymbol{b}. +$$ +

 
+

+
+ + +
+

The \( \chi^2 \) function

+
+ +

+If we then introduce the matrix +

 
+$$ +\boldsymbol{H} = \left(\boldsymbol{A}^T\boldsymbol{A}\right)^{-1}, +$$ +

 
+ +we have then the following expression for the parameters \( \beta_j \) (the matrix elements of \( \boldsymbol{H} \) are \( h_{ij} \)) +

 
+$$ +\beta_j = \sum_{k=0}^{p-1}h_{jk}\sum_{i=0}^{n-1}\frac{y_i}{\sigma_i}\frac{x_{ik}}{\sigma_i} = \sum_{k=0}^{p-1}h_{jk}\sum_{i=0}^{n-1}b_ia_{ik} +$$ +

 
+ +We state without proof the expression for the uncertainty in the parameters \( \beta_j \) as (we leave this as an exercise) +

 
+$$ +\sigma^2(\beta_j) = \sum_{i=0}^{n-1}\sigma_i^2\left( \frac{\partial \beta_j}{\partial y_i}\right)^2, +$$ +

 
+ +resulting in +

 
+$$ +\sigma^2(\beta_j) = \left(\sum_{k=0}^{p-1}h_{jk}\sum_{i=0}^{n-1}a_{ik}\right)\left(\sum_{l=0}^{p-1}h_{jl}\sum_{m=0}^{n-1}a_{ml}\right) = h_{jj}! +$$ +

 
+

+
+ + +
+

The \( \chi^2 \) function

+
+ +

+The first step here is to approximate the function \( y \) with a first-order polynomial, that is we write +

 
+$$ +y=y(x) \rightarrow y(x_i) \approx \beta_0+\beta_1 x_i. +$$ +

 
+ +By computing the derivatives of \( \chi^2 \) with respect to \( \beta_0 \) and \( \beta_1 \) show that these are given by +

 
+$$ +\frac{\partial \chi^2(\boldsymbol{\beta})}{\partial \beta_0} = -2\left[ \frac{1}{n}\sum_{i=0}^{n-1}\left(\frac{y_i-\beta_0-\beta_1x_{i}}{\sigma_i^2}\right)\right]=0, +$$ +

 
+ +and +

 
+$$ +\frac{\partial \chi^2(\boldsymbol{\beta})}{\partial \beta_1} = -\frac{2}{n}\left[ \sum_{i=0}^{n-1}x_i\left(\frac{y_i-\beta_0-\beta_1x_{i}}{\sigma_i^2}\right)\right]=0. +$$ +

 
+

+
+ + +
+

The \( \chi^2 \) function

+
+ +

+For a linear fit (a first-order polynomial) we don't need to invert a matrix!! +Defining +

 
+$$ +\gamma = \sum_{i=0}^{n-1}\frac{1}{\sigma_i^2}, +$$ +

 
+ +

 
+$$ +\gamma_x = \sum_{i=0}^{n-1}\frac{x_{i}}{\sigma_i^2}, +$$ +

 
+ +

 
+$$ +\gamma_y = \sum_{i=0}^{n-1}\left(\frac{y_i}{\sigma_i^2}\right), +$$ +

 
+ +

 
+$$ +\gamma_{xx} = \sum_{i=0}^{n-1}\frac{x_ix_{i}}{\sigma_i^2}, +$$ +

 
+ +

 
+$$ +\gamma_{xy} = \sum_{i=0}^{n-1}\frac{y_ix_{i}}{\sigma_i^2}, +$$ +

 
+ +

+we obtain + +

 
+$$ +\beta_0 = \frac{\gamma_{xx}\gamma_y-\gamma_x\gamma_y}{\gamma\gamma_{xx}-\gamma_x^2}, +$$ +

 
+ +

 
+$$ +\beta_1 = \frac{\gamma_{xy}\gamma-\gamma_x\gamma_y}{\gamma\gamma_{xx}-\gamma_x^2}. +$$ +

 
+ +

+This approach (different linear and non-linear regression) suffers +often from both being underdetermined and overdetermined in the +unknown coefficients \( \beta_i \). A better approach is to use the +Singular Value Decomposition (SVD) method discussed below. Or using +Lasso and Ridge regression. See below. + + +

+
+ + +
+

Fitting an Equation of State for Dense Nuclear Matter

+ +

+Before we continue, let us introduce yet another example. We are going to fit the +nuclear equation of state using results from many-body calculations. +The equation of state we have made available here, as function of +density, has been derived using modern nucleon-nucleon potentials with +the addition of three-body +forces. This +time the file is presented as a standard csv file. + +

+The beginning of the Python code here is similar to what you have seen before, +with the same initializations and declarations. We use also pandas +again, rather extensively in order to organize our data. + +

+The difference now is that we use Scikit-Learn's regression tools +instead of our own matrix inversion implementation. Furthermore, we +sneak in Ridge regression (to be discussed below) which includes a +hyperparameter \( \lambda \), also to be explained below. +

+ + +
+

The code

+ +

+ + +

# Common imports
+import os
+import numpy as np
+import pandas as pd
+import matplotlib.pyplot as plt
+import matplotlib.pyplot as plt
+import sklearn.linear_model as skl
+from sklearn.metrics import mean_squared_error, r2_score, mean_absolute_error
+
+# Where to save the figures and data files
+PROJECT_ROOT_DIR = "Results"
+FIGURE_ID = "Results/FigureFiles"
+DATA_ID = "DataFiles/"
+
+if not os.path.exists(PROJECT_ROOT_DIR):
+    os.mkdir(PROJECT_ROOT_DIR)
+
+if not os.path.exists(FIGURE_ID):
+    os.makedirs(FIGURE_ID)
+
+if not os.path.exists(DATA_ID):
+    os.makedirs(DATA_ID)
+
+def image_path(fig_id):
+    return os.path.join(FIGURE_ID, fig_id)
+
+def data_path(dat_id):
+    return os.path.join(DATA_ID, dat_id)
+
+def save_fig(fig_id):
+    plt.savefig(image_path(fig_id) + ".png", format='png')
+
+infile = open(data_path("EoS.csv"),'r')
+
+# Read the EoS data as  csv file and organize the data into two arrays with density and energies
+EoS = pd.read_csv(infile, names=('Density', 'Energy'))
+EoS['Energy'] = pd.to_numeric(EoS['Energy'], errors='coerce')
+EoS = EoS.dropna()
+Energies = EoS['Energy']
+Density = EoS['Density']
+#  The design matrix now as function of various polytrops
+X = np.zeros((len(Density),4))
+X[:,3] = Density**(4.0/3.0)
+X[:,2] = Density
+X[:,1] = Density**(2.0/3.0)
+X[:,0] = 1
+
+# We use now Scikit-Learn's linear regressor and ridge regressor
+# OLS part
+clf = skl.LinearRegression().fit(X, Energies)
+ytilde = clf.predict(X)
+EoS['Eols']  = ytilde
+# The mean squared error                               
+print("Mean squared error: %.2f" % mean_squared_error(Energies, ytilde))
+# Explained variance score: 1 is perfect prediction                                 
+print('Variance score: %.2f' % r2_score(Energies, ytilde))
+# Mean absolute error                                                           
+print('Mean absolute error: %.2f' % mean_absolute_error(Energies, ytilde))
+print(clf.coef_, clf.intercept_)
+
+# The Ridge regression with a hyperparameter lambda = 0.1
+_lambda = 0.1
+clf_ridge = skl.Ridge(alpha=_lambda).fit(X, Energies)
+yridge = clf_ridge.predict(X)
+EoS['Eridge']  = yridge
+# The mean squared error                               
+print("Mean squared error: %.2f" % mean_squared_error(Energies, yridge))
+# Explained variance score: 1 is perfect prediction                                 
+print('Variance score: %.2f' % r2_score(Energies, yridge))
+# Mean absolute error                                                           
+print('Mean absolute error: %.2f' % mean_absolute_error(Energies, yridge))
+print(clf_ridge.coef_, clf_ridge.intercept_)
+
+fig, ax = plt.subplots()
+ax.set_xlabel(r'$\rho[\mathrm{fm}^{-3}]$')
+ax.set_ylabel(r'Energy per particle')
+ax.plot(EoS['Density'], EoS['Energy'], alpha=0.7, lw=2,
+            label='Theoretical data')
+ax.plot(EoS['Density'], EoS['Eols'], alpha=0.7, lw=2, c='m',
+            label='OLS')
+ax.plot(EoS['Density'], EoS['Eridge'], alpha=0.7, lw=2, c='g',
+            label='Ridge $\lambda = 0.1$')
+ax.legend()
+save_fig("EoSfitting")
+plt.show()
+
+

+The above simple polynomial in density \( \rho \) gives an excellent fit +to the data. Can you give an interpretation of the various powers of \( \rho \)? + +

+We note also that there is a small deviation between the +standard OLS and the Ridge regression at higher densities. We discuss this in more detail +below. +

+ + +
+

Splitting our Data in Training and Test data

+ +

+It is normal in essentially all Machine Learning studies to split the +data in a training set and a test set (sometimes also an additional +validation set). Scikit-Learn has an own function for this. There +is no explicit recipe for how much data should be included as training +data and say test data. An accepted rule of thumb is to use +approximately \( 2/3 \) to \( 4/5 \) of the data as training data. We will +postpone a discussion of this splitting to the end of these notes and +our discussion of the so-called bias-variance tradeoff. Here we +limit ourselves to repeat the above equation of state fitting example +but now splitting the data into a training set and a test set. + +

+ + +

import os
+import numpy as np
+import pandas as pd
+import matplotlib.pyplot as plt
+from sklearn.model_selection import train_test_split
+# Where to save the figures and data files
+PROJECT_ROOT_DIR = "Results"
+FIGURE_ID = "Results/FigureFiles"
+DATA_ID = "DataFiles/"
+
+if not os.path.exists(PROJECT_ROOT_DIR):
+    os.mkdir(PROJECT_ROOT_DIR)
+
+if not os.path.exists(FIGURE_ID):
+    os.makedirs(FIGURE_ID)
+
+if not os.path.exists(DATA_ID):
+    os.makedirs(DATA_ID)
+
+def image_path(fig_id):
+    return os.path.join(FIGURE_ID, fig_id)
+
+def data_path(dat_id):
+    return os.path.join(DATA_ID, dat_id)
+
+def save_fig(fig_id):
+    plt.savefig(image_path(fig_id) + ".png", format='png')
+
+def R2(y_data, y_model):
+    return 1 - np.sum((y_data - y_model) ** 2) / np.sum((y_data - np.mean(y_model)) ** 2)
+def MSE(y_data,y_model):
+    n = np.size(y_model)
+    return np.sum((y_data-y_model)**2)/n
+
+infile = open(data_path("EoS.csv"),'r')
+
+# Read the EoS data as  csv file and organized into two arrays with density and energies
+EoS = pd.read_csv(infile, names=('Density', 'Energy'))
+EoS['Energy'] = pd.to_numeric(EoS['Energy'], errors='coerce')
+EoS = EoS.dropna()
+Energies = EoS['Energy']
+Density = EoS['Density']
+#  The design matrix now as function of various polytrops
+X = np.zeros((len(Density),5))
+X[:,0] = 1
+X[:,1] = Density**(2.0/3.0)
+X[:,2] = Density
+X[:,3] = Density**(4.0/3.0)
+X[:,4] = Density**(5.0/3.0)
+# We split the data in test and training data
+X_train, X_test, y_train, y_test = train_test_split(X, Energies, test_size=0.2)
+# matrix inversion to find beta
+beta = np.linalg.inv(X_train.T.dot(X_train)).dot(X_train.T).dot(y_train)
+# and then make the prediction
+ytilde = X_train @ beta
+print("Training R2")
+print(R2(y_train,ytilde))
+print("Training MSE")
+print(MSE(y_train,ytilde))
+ypredict = X_test @ beta
+print("Test R2")
+print(R2(y_test,ypredict))
+print("Test MSE")
+print(MSE(y_test,ypredict))
+
+
+ + +
+

The singular value decomposition

+ +

+

+ +

+The examples we have looked at so far are cases where we normally can +invert the matrix \( \boldsymbol{X}^T\boldsymbol{X} \). Using a polynomial expansion as we +did both for the masses and the fitting of the equation of state, +leads to row vectors of the design matrix which are essentially +orthogonal due to the polynomial character of our model. This may +however not the be case in general and a standard matrix inversion +algorithm based on say LU decomposition may lead to singularities. We will see an example of this below when we try to fit +the coupling constant of the widely used Ising model. +There is however a way to partially circumvent this problem and also gain some insight about the ordinary least squares approach. + +

+This is given by the Singular Value Decomposition algorithm, perhaps +the most powerful linear algebra algorithm. Let us look at a +different example where we may have problems with the standard matrix +inversion algorithm. Thereafter we dive into the math of the SVD. + + +

+
+ + +
+

The Ising model

+ +

+The one-dimensional Ising model with nearest neighbor interaction, no +external field and a constant coupling constant \( J \) is given by + +

 
+$$ +\begin{align} + H = -J \sum_{k}^L s_k s_{k + 1}, +\tag{1} +\end{align} +$$ +

 
+ +

+where \( s_i \in \{-1, 1\} \) and \( s_{N + 1} = s_1 \). The number of spins +in the system is determined by \( L \). For the one-dimensional system +there is no phase transition. + +

+We will look at a system of \( L = 40 \) spins with a coupling constant of +\( J = 1 \). To get enough training data we will generate 10000 states +with their respective energies. + +

+ + +

import numpy as np
+import matplotlib.pyplot as plt
+from mpl_toolkits.axes_grid1 import make_axes_locatable
+import seaborn as sns
+import scipy.linalg as scl
+from sklearn.model_selection import train_test_split
+import tqdm
+sns.set(color_codes=True)
+cmap_args=dict(vmin=-1., vmax=1., cmap='seismic')
+
+L = 40
+n = int(1e4)
+
+spins = np.random.choice([-1, 1], size=(n, L))
+J = 1.0
+
+energies = np.zeros(n)
+
+for i in range(n):
+    energies[i] = - J * np.dot(spins[i], np.roll(spins[i], 1))
+
+

+Here we use ordinary least squares +regression to predict the energy for the nearest neighbor +one-dimensional Ising model on a ring, i.e., the endpoints wrap +around. We will use linear regression to fit a value for +the coupling constant to achieve this. +

+ + +
+

Reformulating the problem to suit regression

+ +

+A more general form for the one-dimensional Ising model is + +

 
+$$ +\begin{align} + H = - \sum_j^L \sum_k^L s_j s_k J_{jk}. +\tag{2} +\end{align} +$$ +

 
+ +

+Here we allow for interactions beyond the nearest neighbors and a state dependent +coupling constant. This latter expression can be formulated as +a matrix-product +

 
+$$ +\begin{align} + \boldsymbol{H} = \boldsymbol{X} J, +\tag{3} +\end{align} +$$ +

 
+ +

+where \( X_{jk} = s_j s_k \) and \( J \) is a matrix which consists of the +elements \( -J_{jk} \). This form of writing the energy fits perfectly +with the form utilized in linear regression, that is + +

 
+$$ +\begin{align} + \boldsymbol{y} = \boldsymbol{X}\boldsymbol{\beta} + \boldsymbol{\epsilon}, +\tag{4} +\end{align} +$$ +

 
+ +

+We split the data in training and test data as discussed in the previous example + +

+ + +

X = np.zeros((n, L ** 2))
+for i in range(n):
+    X[i] = np.outer(spins[i], spins[i]).ravel()
+y = energies
+X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)
+
+
+ + +
+

Linear regression

+ +

+In the ordinary least squares method we choose the cost function + +

 
+$$ +\begin{align} + C(\boldsymbol{X}, \boldsymbol{\beta})= \frac{1}{n}\left\{(\boldsymbol{X}\boldsymbol{\beta} - \boldsymbol{y})^T(\boldsymbol{X}\boldsymbol{\beta} - \boldsymbol{y})\right\}. +\tag{5} +\end{align} +$$ +

 
+ +

+We then find the extremal point of \( C \) by taking the derivative with respect to \( \boldsymbol{\beta} \) as discussed above. +This yields the expression for \( \boldsymbol{\beta} \) to be + +

 
+$$ + \boldsymbol{\beta} = \frac{\boldsymbol{X}^T \boldsymbol{y}}{\boldsymbol{X}^T \boldsymbol{X}}, +$$ +

 
+ +

+which immediately imposes some requirements on \( \boldsymbol{X} \) as there must exist +an inverse of \( \boldsymbol{X}^T \boldsymbol{X} \). If the expression we are modeling contains an +intercept, i.e., a constant term, we must make sure that the +first column of \( \boldsymbol{X} \) consists of \( 1 \). We do this here + +

+ + +

X_train_own = np.concatenate(
+    (np.ones(len(X_train))[:, np.newaxis], X_train),
+    axis=1
+)
+X_test_own = np.concatenate(
+    (np.ones(len(X_test))[:, np.newaxis], X_test),
+    axis=1
+)
+
+

+ + +

def ols_inv(x: np.ndarray, y: np.ndarray) -> np.ndarray:
+    return scl.inv(x.T @ x) @ (x.T @ y)
+beta = ols_inv(X_train_own, y_train)
+
+
+ + +
+

Singular Value decomposition

+ +

+Doing the inversion directly turns out to be a bad idea since the matrix +\( \boldsymbol{X}^T\boldsymbol{X} \) is singular. An alternative approach is to use the singular +value decomposition. Using the definition of the Moore-Penrose +pseudoinverse we can write the equation for \( \boldsymbol{\beta} \) as + +

 
+$$ + \boldsymbol{\beta} = \boldsymbol{X}^{+}\boldsymbol{y}, +$$ +

 
+ +

+where the pseudoinverse of \( \boldsymbol{X} \) is given by + +

 
+$$ + \boldsymbol{X}^{+} = \frac{\boldsymbol{X}^T}{\boldsymbol{X}^T\boldsymbol{X}}. +$$ +

 
+ +

+Using singular value decomposition we can decompose the matrix \( \boldsymbol{X} = \boldsymbol{U}\boldsymbol{\Sigma} \boldsymbol{V}^T \), +where \( \boldsymbol{U} \) and \( \boldsymbol{V} \) are orthogonal(unitary) matrices and \( \boldsymbol{\Sigma} \) contains the singular values (more details below). +where \( X^{+} = V\Sigma^{+} U^T \). This reduces the equation for +\( \omega \) to +

 
+$$ +\begin{align} + \boldsymbol{\beta} = \boldsymbol{V}\boldsymbol{\Sigma}^{+} \boldsymbol{U}^T \boldsymbol{y}. +\tag{6} +\end{align} +$$ +

 
+ +

+Note that solving this equation by actually doing the pseudoinverse +(which is what we will do) is not a good idea as this operation scales +as \( \mathcal{O}(n^3) \), where \( n \) is the number of elements in a +general matrix. Instead, doing \( QR \)-factorization and solving the +linear system as an equation would reduce this down to +\( \mathcal{O}(n^2) \) operations. + +

+ + +

def ols_svd(x: np.ndarray, y: np.ndarray) -> np.ndarray:
+    u, s, v = scl.svd(x)
+    return v.T @ scl.pinv(scl.diagsvd(s, u.shape[0], v.shape[0])) @ u.T @ y
+
+

+ + +

beta = ols_svd(X_train_own,y_train)
+
+

+When extracting the \( J \)-matrix we need to make sure that we remove the intercept, as is done here + +

+ + +

J = beta[1:].reshape(L, L)
+
+

+A way of looking at the coefficients in \( J \) is to plot the matrices as images. + +

+ + +

fig = plt.figure(figsize=(20, 14))
+im = plt.imshow(J, **cmap_args)
+plt.title("OLS", fontsize=18)
+plt.xticks(fontsize=18)
+plt.yticks(fontsize=18)
+cb = fig.colorbar(im)
+cb.ax.set_yticklabels(cb.ax.get_yticklabels(), fontsize=18)
+plt.show()
+
+

+It is interesting to note that OLS +considers both \( J_{j, j + 1} = -0.5 \) and \( J_{j, j - 1} = -0.5 \) as +valid matrix elements for \( J \). +In our discussion below on hyperparameters and Ridge and Lasso regression we will see that +this problem can be removed, partly and only with Lasso regression. + +

+In this case our matrix inversion was actually possible. The obvious question now is what is the mathematics behind the SVD? +

+ + +
+

Linear Regression Problems

+ +

+One of the typical problems we encounter with linear regression, in particular +when the matrix \( \boldsymbol{X} \) (our so-called design matrix) is high-dimensional, +are problems with near singular or singular matrices. The column vectors of \( \boldsymbol{X} \) +may be linearly dependent, normally referred to as super-collinearity. +This means that the matrix may be rank deficient and it is basically impossible to +to model the data using linear regression. As an example, consider the matrix +

 
+$$ +\begin{align*} +\mathbf{X} & = \left[ +\begin{array}{rrr} +1 & -1 & 2 +\\ +1 & 0 & 1 +\\ +1 & 2 & -1 +\\ +1 & 1 & 0 +\end{array} \right] +\end{align*} +$$ +

 
+ +

+The columns of \( \boldsymbol{X} \) are linearly dependent. We see this easily since the +the first column is the row-wise sum of the other two columns. The rank (more correct, +the column rank) of a matrix is the dimension of the space spanned by the +column vectors. Hence, the rank of \( \mathbf{X} \) is equal to the number +of linearly independent columns. In this particular case the matrix has rank 2. + +

+Super-collinearity of an \( (n \times p) \)-dimensional design matrix \( \mathbf{X} \) implies +that the inverse of the matrix \( \boldsymbol{X}^T\boldsymbol{x} \) (the matrix we need to invert to solve the linear regression equations) is non-invertible. If we have a square matrix that does not have an inverse, we say this matrix singular. The example here demonstrates this +

 
+$$ +\begin{align*} +\boldsymbol{X} & = \left[ +\begin{array}{rr} +1 & -1 +\\ +1 & -1 +\end{array} \right]. +\end{align*} +$$ +

 
+ +We see easily that \( \mbox{det}(\boldsymbol{X}) = x_{11} x_{22} - x_{12} x_{21} = 1 \times (-1) - 1 \times (-1) = 0 \). Hence, \( \mathbf{X} \) is singular and its inverse is undefined. +This is equivalent to saying that the matrix \( \boldsymbol{X} \) has at least an eigenvalue which is zero. +

+ + +
+

Fixing the singularity

+ +

+If our design matrix \( \boldsymbol{X} \) which enters the linear regression problem +

 
+$$ +\begin{align} +\boldsymbol{\beta} & = (\boldsymbol{X}^{T} \boldsymbol{X})^{-1} \boldsymbol{X}^{T} \boldsymbol{y}, +\tag{7} +\end{align} +$$ +

 
+ +has linearly dependent column vectors, we will not be able to compute the inverse +of \( \boldsymbol{X}^T\boldsymbol{X} \) and we cannot find the parameters (estimators) \( \beta_i \). +The estimators are only well-defined if \( (\boldsymbol{X}^{T}\boldsymbol{X})^{-1} \) exits. +This is more likely to happen when the matrix \( \boldsymbol{X} \) is high-dimensional. In this case it is likely to encounter a situation where +the regression parameters \( \beta_i \) cannot be estimated. + +

+A cheap ad hoc approach is simply to add a small diagonal component to the matrix to invert, that is we change +

 
+$$ +\boldsymbol{X}^{T} \boldsymbol{X} \rightarrow \boldsymbol{X}^{T} \boldsymbol{X}+\lambda \boldsymbol{I}, +$$ +

 
+ +where \( \boldsymbol{I} \) is the identity matrix. When we discuss Ridge regression this is actually what we end up evaluating. The parameter \( \lambda \) is called a hyperparameter. More about this later. +

+ + +
+

Basic math of the SVD

+ +

+From standard linear algebra we know that a square matrix \( \boldsymbol{X} \) can be diagonalized if and only it is +a so-called normal matrix, that is if \( \boldsymbol{X}\in {\mathbb{R}}^{n\times n} \) +we have \( \boldsymbol{X}\boldsymbol{X}^T=\boldsymbol{X}^T\boldsymbol{X} \) or if \( \boldsymbol{X}\in {\mathbb{C}}^{n\times n} \) we have \( \boldsymbol{X}\boldsymbol{X}^{\dagger}=\boldsymbol{X}^{\dagger}\boldsymbol{X} \). +The matrix has then a set of eigenpairs + +

 
+$$ +(\lambda_1,\boldsymbol{u}_1),\dots, (\lambda_n,\boldsymbol{u}_n), +$$ +

 
+ +and the eigenvalues are given by the diagonal matrix +

 
+$$ +\boldsymbol{\Sigma}=\mathrm{Diag}(\lambda_1, \dots,\lambda_n). +$$ +

 
+ +The matrix \( \boldsymbol{X} \) can be written in terms of an orthogonal/unitary transformation \( \boldsymbol{U} \) +

 
+$$ +\boldsymbol{X} = \boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{V}^T, +$$ +

 
+ +with \( \boldsymbol{U}\boldsymbol{U}^T=\boldsymbol{I} \) or \( \boldsymbol{U}\boldsymbol{U}^{\dagger}=\boldsymbol{I} \). + +

+Not all square matrices are diagonalizable. A matrix like the one discussed above +

 
+$$ +\boldsymbol{X} = \begin{bmatrix} +1& -1 \\ +1& -1\\ +\end{bmatrix} +$$ +

 
+ +is not diagonalizable, it is a so-called defective matrix. It is easy to see that the condition +\( \boldsymbol{X}\boldsymbol{X}^T=\boldsymbol{X}^T\boldsymbol{X} \) is not fulfilled. +

+ + +
+

The SVD, a Fantastic Algorithm

+ +

+However, and this is the strength of the SVD algorithm, any general +matrix \( \boldsymbol{X} \) can be decomposed in terms of a diagonal matrix and +two orthogonal/unitary matrices. The Singular Value Decompostion +(SVD) theorem +states that a general \( m\times n \) matrix \( \boldsymbol{X} \) can be written in +terms of a diagonal matrix \( \boldsymbol{\Sigma} \) of dimensionality \( n\times n \) +and two orthognal matrices \( \boldsymbol{U} \) and \( \boldsymbol{V} \), where the first has +dimensionality \( m \times m \) and the last dimensionality \( n\times n \). +We have then + +

 
+$$ +\boldsymbol{X} = \boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{V}^T +$$ +

 
+ +

+As an example, the above defective matrix can be decomposed as + +

 
+$$ +\boldsymbol{X} = \frac{1}{\sqrt{2}}\begin{bmatrix} 1& 1 \\ 1& -1\\ \end{bmatrix} \begin{bmatrix} 2& 0 \\ 0& 0\\ \end{bmatrix} \frac{1}{\sqrt{2}}\begin{bmatrix} 1& -1 \\ 1& 1\\ \end{bmatrix}=\boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{V}^T, +$$ +

 
+ +

+with eigenvalues \( \sigma_1=2 \) and \( \sigma_2=0 \). +The SVD exits always! +

+ + +
+

Another Example

+ +

+Consider the following matrix which can be SVD decomposed as + +

 
+$$ +\boldsymbol{X} = \frac{1}{15}\begin{bmatrix} 14 & 2\\ 4 & 22\\ 16 & 13\end{matrix}=\frac{1}{3}\begin{bmatrix} 1& 2 & 2 \\ 2& -1 & 1\\ 2 & 1& -2\end{bmatrix} \begin{bmatrix} 2& 0 \\ 0& 1\\ 0 & 0\end{bmatrix}\frac{1}{5}\begin{bmatrix} 3& 4 \\ 4& -3\end{bmatrix}=\boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{V}^T. +$$ +

 
+ +

+This is a \( 3\times 2 \) matrix which is decomposed in terms of a +\( 3\times 3 \) matrix \( \boldsymbol{U} \), and a \( 2\times 2 \) matrix \( \boldsymbol{V} \). It is easy to see +that \( \boldsymbol{U} \) and \( \boldsymbol{V} \) are orthogonal (how?). + +

+And the SVD +decomposition (singular values) gives eigenvalues +\( \sigma_i\geq\sigma_{i+1} \) for all \( i \) and for dimensions larger than \( i=2 \), the +eigenvalues (singular values) are zero. + +

+In the general case, where our design matrix \( \boldsymbol{X} \) has dimension +\( n\times p \), the matrix is thus decomposed into an \( n\times n \) +orthogonal matrix \( \boldsymbol{U} \), a \( p\times p \) orthogonal matrix \( \boldsymbol{V} \) +and a diagonal matrix \( \boldsymbol{\Sigma} \) with \( r=\mathrm{min}(n,p) \) +singular values \( \sigma_i\lg 0 \) on the main diagonal and zeros filling +the rest of the matrix. There are at most \( p \) singular values +assuming that \( n > p \). In our regression examples for the nuclear +masses and the equation of state this is indeed the case, while for +the Ising model we have \( p > n \). These are often cases that lead to +near singular or singular matrices. + +

+The columns of \( \boldsymbol{U} \) are called the left singular vectors while the columns of \( \boldsymbol{V} \) are the right singular vectors. +

+ + +
+

Economy-size SVD

+ +

+If we assume that \( n > p \), then our matrix \( \boldsymbol{U} \) has dimension \( n +\times n \). The last \( n-p \) columns of \( \boldsymbol{U} \) become however +irrelevant in our calculations since they are multiplied with the +zeros in \( \boldsymbol{\Sigma} \). + +

+The economy-size decomposition removes extra rows or columns of zeros +from the diagonal matrix of singular values, \( \boldsymbol{\Sigma} \), along with the columns +in either \( \boldsymbol{U} \) or \( \boldsymbol{V} \) that multiply those zeros in the expression. +Removing these zeros and columns can improve execution time +and reduce storage requirements without compromising the accuracy of +the decomposition. + +

+If \( n > p \), we keep only the first \( p \) columns of \( \boldsymbol{U} \) and \( \boldsymbol{\Sigma} \) has dimension \( p\times p \). +If \( p > n \), then only the first \( n \) columns of \( \boldsymbol{V} \) are computed and \( \boldsymbol{\Sigma} \) has dimension \( n\times n \). +The \( n=p \) case is obvious, we retain the full SVD. +In general the economy-size SVD leads to less FLOPS and still conserving the desired accuracy. +

+ + +
+

Mathematical Properties

+ +

+There are several interesting mathematical properties which will be +relevant when we are going to discuss the differences between say +ordinary least squares (OLS) and Ridge regression. + +

+We have from OLS that the parameters of the linear approximation are given by +

 
+$$ +\boldsymbol{\tilde{y}} = \boldsymbol{X}\boldsymbol{\beta} = \boldsymbol{X}\left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}. +$$ +

 
+ +

+The matrix to invert can be rewritten in terms of our SVD decomposition as + +

 
+$$ +\boldsymbol{X}^T\boldsymbol{X} = \boldsymbol{V}\boldsymbol{\Sigma}^T\boldsymbol{U}^T\boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{V}^T. +$$ +

 
+ +Using the orthogonality properties of \( \boldsymbol{U} \) we have + +

 
+$$ +\boldsymbol{X}^T\boldsymbol{X} = \boldsymbol{V}\boldsymbol{\Sigma}^T\boldsymbol{\Sigma}\boldsymbol{V}^T = \boldsymbol{V}\boldsymbol{D}\boldsymbol{V}^T, +$$ +

 
+ +with \( \boldsymbol{D} \) being a diagonal matrix with values along the diagonal given by the singular values squared. + +

+This means that +

 
+$$ +(\boldsymbol{X}^T\boldsymbol{X})\boldsymbol{V} = \boldsymbol{V}\boldsymbol{D}, +$$ +

 
+ +that is the eigenvectors of \( (\boldsymbol{X}^T\boldsymbol{X}) \) are given by the columns of the right singular matrix of \( \boldsymbol{X} \) and the eigenvalues are the squared singular values. It is easy to show (show this) that +

 
+$$ +(\boldsymbol{X}\boldsymbol{X}^T)\boldsymbol{U} = \boldsymbol{U}\boldsymbol{D}, +$$ +

 
+ +that is, the eigenvectors of \( (\boldsymbol{X}\boldsymbol{X})^T \) are the columns of the left singular matrix and the eigenvalues are the same. + +

+Going back to our OLS equation we have +

 
+$$ +\boldsymbol{X}\boldsymbol{\beta} = \boldsymbol{X}\left(\boldsymbol{V}\boldsymbol{D}\boldsymbol{V}^T \right)^{-1}\boldsymbol{X}^T\boldsymbol{y}=\boldsymbol{U\Sigma V^T}\left(\boldsymbol{V}\boldsymbol{D}\boldsymbol{V}^T \right)^{-1}(\boldsymbol{U\Sigma V^T})^T\boldsymbol{y}=\boldsymbol{U}\boldsymbol{U}^T\boldsymbol{y}. +$$ +

 
+ +We will come back to this expression when we discuss Ridge regression. +

+ + +
+

Ridge and LASSO Regression

+ +

+Let us remind ourselves about the expression for the standard Mean Squared Error (MSE) which we used to define our cost function and the equations for the ordinary least squares (OLS) method, that is +our optimization problem is +

 
+$$ +{\displaystyle \min_{\boldsymbol{\beta}\in {\mathbb{R}}^{p}}}\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)^T\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)\right\}. +$$ +

 
+ +or we can state it as +

 
+$$ +{\displaystyle \min_{\boldsymbol{\beta}\in +{\mathbb{R}}^{p}}}\frac{1}{n}\sum_{i=0}^{n-1}\left(y_i-\tilde{y}_i\right)^2=\frac{1}{n}\vert\vert \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\vert\vert_2^2, +$$ +

 
+ +where we have used the definition of a norm-2 vector, that is +

 
+$$ +\vert\vert \boldsymbol{x}\vert\vert_2 = \sqrt{\sum_i x_i^2}. +$$ +

 
+ +

+By minimizing the above equation with respect to the parameters +\( \boldsymbol{\beta} \) we could then obtain an analytical expression for the +parameters \( \boldsymbol{\beta} \). We can add a regularization parameter \( \lambda \) by +defining a new cost function to be optimized, that is + +

 
+$$ +{\displaystyle \min_{\boldsymbol{\beta}\in +{\mathbb{R}}^{p}}}\frac{1}{n}\vert\vert \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\vert\vert_2^2+\lambda\vert\vert \boldsymbol{\beta}\vert\vert_2^2 +$$ +

 
+ +

+which leads to the Ridge regression minimization problem where we +require that \( \vert\vert \boldsymbol{\beta}\vert\vert_2^2\le t \), where \( t \) is +a finite number larger than zero. By defining + +

 
+$$ +C(\boldsymbol{X},\boldsymbol{\beta})=\frac{1}{n}\vert\vert \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\vert\vert_2^2+\lambda\vert\vert \boldsymbol{\beta}\vert\vert_1, +$$ +

 
+ +

+we have a new optimization equation +

 
+$$ +{\displaystyle \min_{\boldsymbol{\beta}\in +{\mathbb{R}}^{p}}}\frac{1}{n}\vert\vert \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\vert\vert_2^2+\lambda\vert\vert \boldsymbol{\beta}\vert\vert_1 +$$ +

 
+ +which leads to Lasso regression. Lasso stands for least absolute shrinkage and selection operator. + +

+Here we have defined the norm-1 as +

 
+$$ +\vert\vert \boldsymbol{x}\vert\vert_1 = \sum_i \vert x_i\vert. +$$ +

 
+

+ + +
+

More on Ridge Regression

+ +

+Using the matrix-vector expression for Ridge regression, + +

 
+$$ +C(\boldsymbol{X},\boldsymbol{\beta})=\frac{1}{n}\left\{(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})^T(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})\right\}+\lambda\boldsymbol{\beta}^T\boldsymbol{\beta}, +$$ +

 
+ +

+by taking the derivatives with respect to \( \boldsymbol{\beta} \) we obtain then +a slightly modified matrix inversion problem which for finite values +of \( \lambda \) does not suffer from singularity problems. We obtain + +

 
+$$ +\boldsymbol{\beta}^{\mathrm{Ridge}} = \left(\boldsymbol{X}^T\boldsymbol{X}+\lambda\boldsymbol{I}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}, +$$ +

 
+ +

+with \( \boldsymbol{I} \) being a \( p\times p \) identity matrix with the constraint that + +

 
+$$ +\sum_{i=0}^{p-1} \beta_i^2 \leq t, +$$ +

 
+ +

+with \( t \) a finite positive number. + +

+We see that Ridge regression is nothing but the standard +OLS with a modified diagonal term added to \( \boldsymbol{X}^T\boldsymbol{X} \). The +consequences, in particular for our discussion of the bias-variance +are rather interesting. + +

+Furthermore, if we use the result above in terms of the SVD decomposition (our analysis was done for the OLS method), we had +

 
+$$ +(\boldsymbol{X}\boldsymbol{X}^T)\boldsymbol{U} = \boldsymbol{U}\boldsymbol{D}. +$$ +

 
+ +

+We can analyse the OLS solutions in terms of the eigenvectors (the columns) of the right singular value matrix \( \boldsymbol{U} \) as +

 
+$$ +\boldsymbol{X}\boldsymbol{\beta} = \boldsymbol{X}\left(\boldsymbol{V}\boldsymbol{D}\boldsymbol{V}^T \right)^{-1}\boldsymbol{X}^T\boldsymbol{y}=\boldsymbol{U\Sigma V^T}\left(\boldsymbol{V}\boldsymbol{D}\boldsymbol{V}^T \right)^{-1}(\boldsymbol{U\Sigma V^T})^T\boldsymbol{y}=\boldsymbol{U}\boldsymbol{U}^T\boldsymbol{y} +$$ +

 
+ +

+For Ridge regression this becomes + +

 
+$$ +\boldsymbol{X}\boldsymbol{\beta}^{\mathrm{Ridge}} = \boldsymbol{U\Sigma V^T}\left(\boldsymbol{V}\boldsymbol{D}\boldsymbol{V}^T+\lambda\boldsymbol{I} \right)^{-1}(\boldsymbol{U\Sigma V^T})^T\boldsymbol{y}=\sum_{j=0}^{p-1}\boldsymbol{u}_j\boldsymbol{u}_j^T\frac{\sigma_j^2}{\sigma_j^2+\lambda}\boldsymbol{y}, +$$ +

 
+ +

+with the vectors \( \boldsymbol{u}_j \) being the columns of \( \boldsymbol{U} \). +

+ + +
+

Interpreting the Ridge results

+ +

+Since \( \lambda \geq 0 \), it means that compared to OLS, we have + +

 
+$$ +\frac{\sigma_j^2}{\sigma_j^2+\lambda} \leq 1. +$$ +

 
+ +

+Ridge regression finds the coordinates of \( \boldsymbol{y} \) with respect to the +orthonormal basis \( \boldsymbol{U} \), it then shrinks the coordinates by +\( \frac{\sigma_j^2}{\sigma_j^2+\lambda} \). Recall that the SVD has +eigenvalues ordered in a descending way, that is \( \sigma_i \geq +\sigma_{i+1} \). + +

+For small eigenvalues \( \sigma_i \) it means that their contributions become less important, a fact which can be used to reduce the number of degrees of freedom. +Actually, calculating the variance of \( \boldsymbol{X}\boldsymbol{v}_j \) shows that this quantity is equal to \( \sigma_j^2/n \). +With a parameter \( \lambda \) we can thus shrink the role of specific parameters. +

+ + +
+

More interpretations

+ +

+For the sake of simplicity, let us assume that the design matrix is orthonormal, that is + +

 
+$$ +\boldsymbol{X}^T\boldsymbol{X}=(\boldsymbol{X}^T\boldsymbol{X})^{-1} =\boldsymbol{I}. +$$ +

 
+ +

+In this case the standard OLS results in +

 
+$$ +\boldsymbol{\beta}^{\mathrm{OLS}} = \boldsymbol{X}^T\boldsymbol{y}=\sum_{i=0}^{p-1}\boldsymbol{u}_j\boldsymbol{u}_j^T\boldsymbol{y}, +$$ +

 
+ +

+and + +

 
+$$ +\boldsymbol{\beta}^{\mathrm{Ridge}} = \left(\boldsymbol{I}+\lambda\boldsymbol{I}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}=\left(1+\lambda\right)^{-1}\boldsymbol{\beta}^{\mathrm{OLS}}, +$$ +

 
+ +

+that is the Ridge estimator scales the OLS estimator by the inverse of a factor \( 1+\lambda \), and +the Ridge estimator converges to zero when the hyperparameter goes to +infinity. + +

+We will come back to more interpreations after we have gone through some of the statistical analysis part. + +

+For more discussions of Ridge and Lasso regression, Wessel van Wieringen's article is highly recommended. +Similarly, Mehta et al's article is also recommended. +

+ + +
+

Where are we going?

+ +

+Before we proceed, we need to rethink what we have been doing. In our +eager to fit the data, we have omitted several important elements in +our regression analysis. In what follows we will + +

    +

  1. look at statistical properties, including a discussion of mean values, variance and the so-called bias-variance tradeoff
    2. +

  2. introduce resampling techniques like cross-validation, bootstrapping and jackknife and more
    3. +
+

+ +This will allow us to link the standard linear algebra methods we have discussed above to a statistical interpretation of the methods. +

+ + +
+

Resampling methods

+
+ +

+Resampling methods are an indispensable tool in modern +statistics. They involve repeatedly drawing samples from a training +set and refitting a model of interest on each sample in order to +obtain additional information about the fitted model. For example, in +order to estimate the variability of a linear regression fit, we can +repeatedly draw different samples from the training data, fit a linear +regression to each new sample, and then examine the extent to which +the resulting fits differ. Such an approach may allow us to obtain +information that would not be available from fitting the model only +once using the original training sample. +

+
+ + +
+

Resampling approaches can be computationally expensive

+
+ +

+Resampling approaches can be computationally expensive, because they +involve fitting the same statistical method multiple times using +different subsets of the training data. However, due to recent +advances in computing power, the computational requirements of +resampling methods generally are not prohibitive. In this chapter, we +discuss two of the most commonly used resampling methods, +cross-validation and the bootstrap. Both methods are important tools +in the practical application of many statistical learning +procedures. For example, cross-validation can be used to estimate the +test error associated with a given statistical learning method in +order to evaluate its performance, or to select the appropriate level +of flexibility. The process of evaluating a model’s performance is +known as model assessment, whereas the process of selecting the proper +level of flexibility for a model is known as model selection. The +bootstrap is widely used. + + +

+
+ + +
+

Why resampling methods ?

+
+Statistical analysis. +
    +

  • Our simulations can be treated as computer experiments. This is particularly the case for Monte Carlo methods
    • +

  • The results can be analysed with the same statistical tools as we would use analysing experimental data.
    • +

  • As in all experiments, we are looking for expectation values and an estimate of how accurate they are, i.e., possible sources for errors.
    • +
+
+
+ + +
+

Statistical analysis

+
+ +
    +

  • As in other experiments, many numerical experiments have two classes of errors:
    • + +
      + +

    • Statistical errors
      • + +

    • Systematical errors
      • +
    +

  • Statistical errors can be estimated using standard tools from statistics
    • +

  • Systematical errors are method specific and must be treated differently from case to case.
    • +
+
+
+ + +
+

Statistics

+
+ +

+The probability distribution function (PDF) is a function +\( p(x) \) on the domain which, in the discrete case, gives us the +probability or relative frequency with which these values of \( X \) occur: +

 
+$$ +p(x) = \mathrm{prob}(X=x) +$$ +

 
+ +In the continuous case, the PDF does not directly depict the +actual probability. Instead we define the probability for the +stochastic variable to assume any value on an infinitesimal interval +around \( x \) to be \( p(x)dx \). The continuous function \( p(x) \) then gives us +the density of the probability rather than the probability +itself. The probability for a stochastic variable to assume any value +on a non-infinitesimal interval \( [a,\,b] \) is then just the integral: +

 
+$$ +\mathrm{prob}(a\leq X\leq b) = \int_a^b p(x)dx +$$ +

 
+ +Qualitatively speaking, a stochastic variable represents the values of +numbers chosen as if by chance from some specified PDF so that the +selection of a large set of these numbers reproduces this PDF. +

+
+ + +
+

Statistics, moments

+
+ +

+A particularly useful class of special expectation values are the +moments. The \( n \)-th moment of the PDF \( p \) is defined as +follows: +

 
+$$ +\langle x^n\rangle \equiv \int\! x^n p(x)\,dx +$$ +

 
+ +The zero-th moment \( \langle 1\rangle \) is just the normalization condition of +\( p \). The first moment, \( \langle x\rangle \), is called the mean of \( p \) +and often denoted by the letter \( \mu \): +

 
+$$ +\langle x\rangle = \mu \equiv \int\! x p(x)\,dx +$$ +

 
+

+
+ + +
+

Statistics, central moments

+
+ +

+A special version of the moments is the set of central moments, +the n-th central moment defined as: +

 
+$$ +\langle (x-\langle x \rangle )^n\rangle \equiv \int\! (x-\langle x\rangle)^n p(x)\,dx +$$ +

 
+ +The zero-th and first central moments are both trivial, equal \( 1 \) and +\( 0 \), respectively. But the second central moment, known as the +variance of \( p \), is of particular interest. For the stochastic +variable \( X \), the variance is denoted as \( \sigma^2_X \) or \( \mathrm{var}(X) \): +

 
+$$ +\begin{align} +\sigma^2_X\ \ =\ \ \mathrm{var}(X) & = \langle (x-\langle x\rangle)^2\rangle = +\int\! (x-\langle x\rangle)^2 p(x)\,dx +\tag{8}\\ +& = \int\! \left(x^2 - 2 x \langle x\rangle^{2} + + \langle x\rangle^2\right)p(x)\,dx +\tag{9}\\ +& = \langle x^2\rangle - 2 \langle x\rangle\langle x\rangle + \langle x\rangle^2 +\tag{10}\\ +& = \langle x^2\rangle - \langle x\rangle^2 +\tag{11} +\end{align} +$$ +

 
+ +The square root of the variance, \( \sigma =\sqrt{\langle (x-\langle x\rangle)^2\rangle} \) is called the standard deviation of \( p \). It is clearly just the RMS (root-mean-square) +value of the deviation of the PDF from its mean value, interpreted +qualitatively as the spread of \( p \) around its mean. +

+
+ + +
+

Statistics, covariance

+
+ +

+Another important quantity is the so called covariance, a variant of +the above defined variance. Consider again the set \( \{X_i\} \) of \( n \) +stochastic variables (not necessarily uncorrelated) with the +multivariate PDF \( P(x_1,\dots,x_n) \). The covariance of two +of the stochastic variables, \( X_i \) and \( X_j \), is defined as follows: +

 
+$$ +\begin{align} +\mathrm{cov}(X_i,\,X_j) &\equiv \langle (x_i-\langle x_i\rangle)(x_j-\langle x_j\rangle)\rangle +\nonumber\\ +&= +\int\!\cdots\!\int\!(x_i-\langle x_i \rangle)(x_j-\langle x_j \rangle)\, +P(x_1,\dots,x_n)\,dx_1\dots dx_n +\tag{12} +\end{align} +$$ +

 
+ +with +

 
+$$ +\langle x_i\rangle = +\int\!\cdots\!\int\!x_i\,P(x_1,\dots,x_n)\,dx_1\dots dx_n +$$ +

 
+

+
+ + +
+

Statistics, more covariance

+
+ +

+If we consider the above covariance as a matrix \( C_{ij}=\mathrm{cov}(X_i,\,X_j) \), then the diagonal elements are just the familiar +variances, \( C_{ii} = \mathrm{cov}(X_i,\,X_i) = \mathrm{var}(X_i) \). It turns out that +all the off-diagonal elements are zero if the stochastic variables are +uncorrelated. This is easy to show, keeping in mind the linearity of +the expectation value. Consider the stochastic variables \( X_i \) and +\( X_j \), (\( i\neq j \)): +

 
+$$ +\begin{align} +\mathrm{cov}(X_i,\,X_j) &= \langle(x_i-\langle x_i\rangle)(x_j-\langle x_j\rangle)\rangle +\tag{13}\\ +&=\langle x_i x_j - x_i\langle x_j\rangle - \langle x_i\rangle x_j + \langle x_i\rangle\langle x_j\rangle\rangle +\tag{14}\\ +&=\langle x_i x_j\rangle - \langle x_i\langle x_j\rangle\rangle - \langle \langle x_i\rangle x_j\rangle + +\langle \langle x_i\rangle\langle x_j\rangle\rangle +\tag{15}\\ +&=\langle x_i x_j\rangle - \langle x_i\rangle\langle x_j\rangle - \langle x_i\rangle\langle x_j\rangle + +\langle x_i\rangle\langle x_j\rangle +\tag{16}\\ +&=\langle x_i x_j\rangle - \langle x_i\rangle\langle x_j\rangle +\tag{17} +\end{align} +$$ +

 
+

+
+ + +
+

Statistics, independent variables

+
+ +

+If \( X_i \) and \( X_j \) are independent, we get +\( \langle x_i x_j\rangle =\langle x_i\rangle\langle x_j\rangle \), resulting in \( \mathrm{cov}(X_i, X_j) = 0\ \ (i\neq j) \). + +

+Also useful for us is the covariance of linear combinations of +stochastic variables. Let \( \{X_i\} \) and \( \{Y_i\} \) be two sets of +stochastic variables. Let also \( \{a_i\} \) and \( \{b_i\} \) be two sets of +scalars. Consider the linear combination: +

 
+$$ +U = \sum_i a_i X_i \qquad V = \sum_j b_j Y_j +$$ +

 
+ +By the linearity of the expectation value +

 
+$$ +\mathrm{cov}(U, V) = \sum_{i,j}a_i b_j \mathrm{cov}(X_i, Y_j) +$$ +

 
+

+
+ + +
+

Statistics, more variance

+
+ +

+Now, since the variance is just \( \mathrm{var}(X_i) = \mathrm{cov}(X_i, X_i) \), we get +the variance of the linear combination \( U = \sum_i a_i X_i \): +

 
+$$ +\begin{equation} +\mathrm{var}(U) = \sum_{i,j}a_i a_j \mathrm{cov}(X_i, X_j) +\tag{18} +\end{equation} +$$ +

 
+ +And in the special case when the stochastic variables are +uncorrelated, the off-diagonal elements of the covariance are as we +know zero, resulting in: +

 
+$$ +\mathrm{var}(U) = \sum_i a_i^2 \mathrm{cov}(X_i, X_i) = \sum_i a_i^2 \mathrm{var}(X_i) +$$ +

 
+ +

 
+$$ +\mathrm{var}(\sum_i a_i X_i) = \sum_i a_i^2 \mathrm{var}(X_i) +$$ +

 
+ +which will become very useful in our study of the error in the mean +value of a set of measurements. +

+
+ + +
+

Statistics and stochastic processes

+
+ +

+A stochastic process is a process that produces sequentially a +chain of values: +

 
+$$ +\{x_1, x_2,\dots\,x_k,\dots\}. +$$ +

 
+ +We will call these +values our measurements and the entire set as our measured +sample. The action of measuring all the elements of a sample +we will call a stochastic experiment since, operationally, +they are often associated with results of empirical observation of +some physical or mathematical phenomena; precisely an experiment. We +assume that these values are distributed according to some +PDF \( p_X^{\phantom X}(x) \), where \( X \) is just the formal symbol for the +stochastic variable whose PDF is \( p_X^{\phantom X}(x) \). Instead of +trying to determine the full distribution \( p \) we are often only +interested in finding the few lowest moments, like the mean +\( \mu_X^{\phantom X} \) and the variance \( \sigma_X^{\phantom X} \). +

+
+ + +
+

Statistics and sample variables

+
+ +

+In practical situations a sample is always of finite size. Let that +size be \( n \). The expectation value of a sample, the sample mean, is then defined as follows: +

 
+$$ +\bar{x}_n \equiv \frac{1}{n}\sum_{k=1}^n x_k +$$ +

 
+ +The sample variance is: +

 
+$$ +\mathrm{var}(x) \equiv \frac{1}{n}\sum_{k=1}^n (x_k - \bar{x}_n)^2 +$$ +

 
+ +its square root being the standard deviation of the sample. The +sample covariance is: +

 
+$$ +\mathrm{cov}(x)\equiv\frac{1}{n}\sum_{kl}(x_k - \bar{x}_n)(x_l - \bar{x}_n) +$$ +

 
+

+
+ + +
+

Statistics, sample variance and covariance

+
+ +

+Note that the sample variance is the sample covariance without the +cross terms. In a similar manner as the covariance in Eq. (12) is a measure of the correlation between +two stochastic variables, the above defined sample covariance is a +measure of the sequential correlation between succeeding measurements +of a sample. + +

+These quantities, being known experimental values, differ +significantly from and must not be confused with the similarly named +quantities for stochastic variables, mean \( \mu_X \), variance \( \mathrm{var}(X) \) +and covariance \( \mathrm{cov}(X,Y) \). +

+
+ + +
+

Statistics, law of large numbers

+
+ +

+The law of large numbers +states that as the size of our sample grows to infinity, the sample +mean approaches the true mean \( \mu_X^{\phantom X} \) of the chosen PDF: +

 
+$$ +\lim_{n\to\infty}\bar{x}_n = \mu_X^{\phantom X} +$$ +

 
+ +The sample mean \( \bar{x}_n \) works therefore as an estimate of the true +mean \( \mu_X^{\phantom X} \). + +

+What we need to find out is how good an approximation \( \bar{x}_n \) is to +\( \mu_X^{\phantom X} \). In any stochastic measurement, an estimated +mean is of no use to us without a measure of its error. A quantity +that tells us how well we can reproduce it in another experiment. We +are therefore interested in the PDF of the sample mean itself. Its +standard deviation will be a measure of the spread of sample means, +and we will simply call it the error of the sample mean, or +just sample error, and denote it by \( \mathrm{err}_X^{\phantom X} \). In +practice, we will only be able to produce an estimate of the +sample error since the exact value would require the knowledge of the +true PDFs behind, which we usually do not have. +

+
+ + +
+

Statistics, more on sample error

+
+ +

+Let us first take a look at what happens to the sample error as the +size of the sample grows. In a sample, each of the measurements \( x_i \) +can be associated with its own stochastic variable \( X_i \). The +stochastic variable \( \overline X_n \) for the sample mean \( \bar{x}_n \) is +then just a linear combination, already familiar to us: +

 
+$$ +\overline X_n = \frac{1}{n}\sum_{i=1}^n X_i +$$ +

 
+ +All the coefficients are just equal \( 1/n \). The PDF of \( \overline X_n \), +denoted by \( p_{\overline X_n}(x) \) is the desired PDF of the sample +means. +

+
+ + +
+

Statistics

+
+ +

+The probability density of obtaining a sample mean \( \bar x_n \) +is the product of probabilities of obtaining arbitrary values \( x_1, +x_2,\dots,x_n \) with the constraint that the mean of the set \( \{x_i\} \) +is \( \bar x_n \): +

 
+$$ +p_{\overline X_n}(x) = \int p_X^{\phantom X}(x_1)\cdots +\int p_X^{\phantom X}(x_n)\ +\delta\!\left(x - \frac{x_1+x_2+\dots+x_n}{n}\right)dx_n \cdots dx_1 +$$ +

 
+ +And in particular we are interested in its variance \( \mathrm{var}(\overline X_n) \). +

+
+ + +
+

Statistics, central limit theorem

+
+ +

+It is generally not possible to express \( p_{\overline X_n}(x) \) in a +closed form given an arbitrary PDF \( p_X^{\phantom X} \) and a number +\( n \). But for the limit \( n\to\infty \) it is possible to make an +approximation. The very important result is called the central limit theorem. It tells us that as \( n \) goes to infinity, +\( p_{\overline X_n}(x) \) approaches a Gaussian distribution whose mean +and variance equal the true mean and variance, \( \mu_{X}^{\phantom X} \) +and \( \sigma_{X}^{2} \), respectively: +

 
+$$ +\begin{equation} +\lim_{n\to\infty} p_{\overline X_n}(x) = +\left(\frac{n}{2\pi\mathrm{var}(X)}\right)^{1/2} +e^{-\frac{n(x-\bar x_n)^2}{2\mathrm{var}(X)}} +\tag{19} +\end{equation} +$$ +

 
+

+
+ + +
+

Statistics, more technicalities

+
+ +

+The desired variance +\( \mathrm{var}(\overline X_n) \), i.e. the sample error squared +\( \mathrm{err}_X^2 \), is given by: +

 
+$$ +\begin{equation} +\mathrm{err}_X^2 = \mathrm{var}(\overline X_n) = \frac{1}{n^2} +\sum_{ij} \mathrm{cov}(X_i, X_j) +\tag{20} +\end{equation} +$$ +

 
+ +We see now that in order to calculate the exact error of the sample +with the above expression, we would need the true means +\( \mu_{X_i}^{\phantom X} \) of the stochastic variables \( X_i \). To +calculate these requires that we know the true multivariate PDF of all +the \( X_i \). But this PDF is unknown to us, we have only got the measurements of +one sample. The best we can do is to let the sample itself be an +estimate of the PDF of each of the \( X_i \), estimating all properties of +\( X_i \) through the measurements of the sample. +

+
+ + +
+

Statistics

+
+ +

+Our estimate of \( \mu_{X_i}^{\phantom X} \) is then the sample mean \( \bar x \) +itself, in accordance with the the central limit theorem: +

 
+$$ +\mu_{X_i}^{\phantom X} = \langle x_i\rangle \approx \frac{1}{n}\sum_{k=1}^n x_k = \bar x +$$ +

 
+ +Using \( \bar x \) in place of \( \mu_{X_i}^{\phantom X} \) we can give an +estimate of the covariance in Eq. (20) +

 
+$$ +\mathrm{cov}(X_i, X_j) = \langle (x_i-\langle x_i\rangle)(x_j-\langle x_j\rangle)\rangle +\approx\langle (x_i - \bar x)(x_j - \bar{x})\rangle, +$$ +

 
+ +resulting in +

 
+$$ +\frac{1}{n} \sum_{l}^n \left(\frac{1}{n}\sum_{k}^n (x_k -\bar x_n)(x_l - \bar x_n)\right)=\frac{1}{n}\frac{1}{n} \sum_{kl} (x_k -\bar x_n)(x_l - \bar x_n)=\frac{1}{n}\mathrm{cov}(x) +$$ +

 
+

+
+ + +
+

Statistics and sample variance

+
+ +

+By the same procedure we can use the sample variance as an +estimate of the variance of any of the stochastic variables \( X_i \) +

 
+$$ +\mathrm{var}(X_i)=\langle x_i - \langle x_i\rangle\rangle \approx \langle x_i - \bar x_n\rangle\nonumber, +$$ +

 
+ +which is approximated as +

 
+$$ +\begin{equation} +\mathrm{var}(X_i)\approx \frac{1}{n}\sum_{k=1}^n (x_k - \bar x_n)=\mathrm{var}(x) +\tag{21} +\end{equation} +$$ +

 
+ +

+Now we can calculate an estimate of the error +\( \mathrm{err}_X^{\phantom X} \) of the sample mean \( \bar x_n \): +

 
+$$ +\begin{align} +\mathrm{err}_X^2 +&=\frac{1}{n^2}\sum_{ij} \mathrm{cov}(X_i, X_j) \nonumber \\ +&\approx&\frac{1}{n^2}\sum_{ij}\frac{1}{n}\mathrm{cov}(x) =\frac{1}{n^2}n^2\frac{1}{n}\mathrm{cov}(x)\nonumber\\ +&=\frac{1}{n}\mathrm{cov}(x) +\tag{22} +\end{align} +$$ +

 
+ +which is nothing but the sample covariance divided by the number of +measurements in the sample. +

+
+ + +
+

Statistics, uncorrelated results

+
+ +

+In the special case that the measurements of the sample are +uncorrelated (equivalently the stochastic variables \( X_i \) are +uncorrelated) we have that the off-diagonal elements of the covariance +are zero. This gives the following estimate of the sample error: +

 
+$$ +\mathrm{err}_X^2=\frac{1}{n^2}\sum_{ij} \mathrm{cov}(X_i, X_j) = +\frac{1}{n^2} \sum_i \mathrm{var}(X_i), +$$ +

 
+ +resulting in +

 
+$$ +\begin{equation} +\mathrm{err}_X^2\approx \frac{1}{n^2} \sum_i \mathrm{var}(x)= \frac{1}{n}\mathrm{var}(x) +\tag{23} +\end{equation} +$$ +

 
+ +where in the second step we have used Eq. (21). +The error of the sample is then just its standard deviation divided by +the square root of the number of measurements the sample contains. +This is a very useful formula which is easy to compute. It acts as a +first approximation to the error, but in numerical experiments, we +cannot overlook the always present correlations. +

+
+ + +
+

Statistics, computations

+
+ +

+For computational purposes one usually splits up the estimate of +\( \mathrm{err}_X^2 \), given by Eq. (22), into two +parts +

 
+$$ +\mathrm{err}_X^2 = \frac{1}{n}\mathrm{var}(x) + \frac{1}{n}(\mathrm{cov}(x)-\mathrm{var}(x)), +$$ +

 
+ +which equals +

 
+$$ +\begin{equation} +\frac{1}{n^2}\sum_{k=1}^n (x_k - \bar x_n)^2 +\frac{2}{n^2}\sum_{k < l} (x_k - \bar x_n)(x_l - \bar x_n) +\tag{24} +\end{equation} +$$ +

 
+ +The first term is the same as the error in the uncorrelated case, +Eq. (23). This means that the second +term accounts for the error correction due to correlation between the +measurements. For uncorrelated measurements this second term is zero. +

+
+ + +
+

Statistics, more on computations of errors

+
+ +

+Computationally the uncorrelated first term is much easier to treat +efficiently than the second. +

 
+$$ +\mathrm{var}(x) = \frac{1}{n}\sum_{k=1}^n (x_k - \bar x_n)^2 = +\left(\frac{1}{n}\sum_{k=1}^n x_k^2\right) - \bar x_n^2 +$$ +

 
+ +We just accumulate separately the values \( x^2 \) and \( x \) for every +measurement \( x \) we receive. The correlation term, though, has to be +calculated at the end of the experiment since we need all the +measurements to calculate the cross terms. Therefore, all measurements +have to be stored throughout the experiment. +

+
+ + +
+

Statistics, wrapping up 1

+
+ +

+Let us analyze the problem by splitting up the correlation term into +partial sums of the form: +

 
+$$ +f_d = \frac{1}{n-d}\sum_{k=1}^{n-d}(x_k - \bar x_n)(x_{k+d} - \bar x_n) +$$ +

 
+ +The correlation term of the error can now be rewritten in terms of +\( f_d \) +

 
+$$ +\frac{2}{n}\sum_{k < l} (x_k - \bar x_n)(x_l - \bar x_n) = +2\sum_{d=1}^{n-1} f_d +$$ +

 
+ +The value of \( f_d \) reflects the correlation between measurements +separated by the distance \( d \) in the sample samples. Notice that for +\( d=0 \), \( f \) is just the sample variance, \( \mathrm{var}(x) \). If we divide \( f_d \) +by \( \mathrm{var}(x) \), we arrive at the so called autocorrelation function +

 
+$$ +\kappa_d = \frac{f_d}{\mathrm{var}(x)} +$$ +

 
+ +which gives us a useful measure of pairwise correlations +starting always at \( 1 \) for \( d=0 \). +

+
+ + +
+

Statistics, final expression

+
+ +

+The sample error (see eq. (24)) can now be +written in terms of the autocorrelation function: +

 
+$$ +\begin{align} +\mathrm{err}_X^2 &= +\frac{1}{n}\mathrm{var}(x)+\frac{2}{n}\cdot\mathrm{var}(x)\sum_{d=1}^{n-1} +\frac{f_d}{\mathrm{var}(x)}\nonumber\\ &=& +\left(1+2\sum_{d=1}^{n-1}\kappa_d\right)\frac{1}{n}\mathrm{var}(x)\nonumber\\ +&=\frac{\tau}{n}\cdot\mathrm{var}(x) +\tag{25} +\end{align} +$$ +

 
+ +and we see that \( \mathrm{err}_X \) can be expressed in terms the +uncorrelated sample variance times a correction factor \( \tau \) which +accounts for the correlation between measurements. We call this +correction factor the autocorrelation time: +

 
+$$ +\begin{equation} +\tau = 1+2\sum_{d=1}^{n-1}\kappa_d +\tag{26} +\end{equation} +$$ +

 
+

+
+ + +
+

Statistics, effective number of correlations

+
+ +

+For a correlation free experiment, \( \tau \) +equals 1. From the point of view of +eq. (25) we can interpret a sequential +correlation as an effective reduction of the number of measurements by +a factor \( \tau \). The effective number of measurements becomes: +

 
+$$ +n_\mathrm{eff} = \frac{n}{\tau} +$$ +

 
+ +To neglect the autocorrelation time \( \tau \) will always cause our +simple uncorrelated estimate of \( \mathrm{err}_X^2\approx \mathrm{var}(x)/n \) to +be less than the true sample error. The estimate of the error will be +too good. On the other hand, the calculation of the full +autocorrelation time poses an efficiency problem if the set of +measurements is very large. +

+
+ + +
+

Linking the regression analysis with a statistical interpretation

+ +

+Finally, we are going to discuss several statistical properties which can be obtained in terms of analytical expressions. +The +advantage of doing linear regression is that we actually end up with +analytical expressions for several statistical quantities. +Standard least squares and Ridge regression allow us to +derive quantities like the variance and other expectation values in a +rather straightforward way. + +

+It is assumed that \( \varepsilon_i +\sim \mathcal{N}(0, \sigma^2) \) and the \( \varepsilon_{i} \) are +independent, i.e.: +

 
+$$ +\begin{align*} +\mbox{Cov}(\varepsilon_{i_1}, +\varepsilon_{i_2}) & = \left\{ \begin{array}{lcc} \sigma^2 & \mbox{if} +& i_1 = i_2, \\ 0 & \mbox{if} & i_1 \not= i_2. \end{array} \right. +\end{align*} +$$ +

 
+ +The randomness of \( \varepsilon_i \) implies that +\( \mathbf{y}_i \) is also a random variable. In particular, +\( \mathbf{y}_i \) is normally distributed, because \( \varepsilon_i \sim +\mathcal{N}(0, \sigma^2) \) and \( \mathbf{X}_{i,\ast} \, \boldsymbol{\beta} \) is a +non-random scalar. To specify the parameters of the distribution of +\( \mathbf{y}_i \) we need to calculate its first two moments. + +

+Recall that \( \boldsymbol{X} \) is a matrix of dimensionality \( n\times p \). The +notation above \( \mathbf{X}_{i,\ast} \) means that we are looking at the +row number \( i \) and perform a sum over all values \( p \). +

+ + +
+

Assumptions made

+ +

+The assumption we have made here can be summarized as (and this is going to useful when we discuss the bias-variance trade off) +that there exists a function \( f(\boldsymbol{x}) \) and a normal distributed error \( \boldsymbol{\varepsilon}\sim \mathcal{N}(0, \sigma^2) \) +which describes our data +

 
+$$ +\boldsymbol{y} = f(\boldsymbol{x})+\boldsymbol{\varepsilon} +$$ +

 
+ +

+We approximate this function with our model from the solution of the linear regression equations, that is our +function \( f \) is approximated by \( \boldsymbol{\tilde{y}} \) where we want to minimize \( (\boldsymbol{y}-\boldsymbol{\tilde{y}})^2 \), our MSE, with +

 
+$$ +\boldsymbol{\tilde{y}} = \boldsymbol{X}\boldsymbol{\beta}. +$$ +

 
+

+ + +
+

Expectation value and variance

+ +

+We can calculate the expectation value of \( \boldsymbol{y} \) for a given element \( i \) +

 
+$$ +\begin{align*} +\mathbb{E}(y_i) & = +\mathbb{E}(\mathbf{X}_{i, \ast} \, \boldsymbol{\beta}) + \mathbb{E}(\varepsilon_i) +\, \, \, = \, \, \, \mathbf{X}_{i, \ast} \, \beta, +\end{align*} +$$ +

 
+ +while +its variance is +

 
+$$ +\begin{align*} \mbox{Var}(y_i) & = \mathbb{E} \{ [y_i +- \mathbb{E}(y_i)]^2 \} \, \, \, = \, \, \, \mathbb{E} ( y_i^2 ) - +[\mathbb{E}(y_i)]^2 \\ & = \mathbb{E} [ ( \mathbf{X}_{i, \ast} \, +\beta + \varepsilon_i )^2] - ( \mathbf{X}_{i, \ast} \, \boldsymbol{\beta})^2 \\ & += \mathbb{E} [ ( \mathbf{X}_{i, \ast} \, \boldsymbol{\beta})^2 + 2 \varepsilon_i +\mathbf{X}_{i, \ast} \, \boldsymbol{\beta} + \varepsilon_i^2 ] - ( \mathbf{X}_{i, +\ast} \, \beta)^2 \\ & = ( \mathbf{X}_{i, \ast} \, \boldsymbol{\beta})^2 + 2 +\mathbb{E}(\varepsilon_i) \mathbf{X}_{i, \ast} \, \boldsymbol{\beta} + +\mathbb{E}(\varepsilon_i^2 ) - ( \mathbf{X}_{i, \ast} \, \boldsymbol{\beta})^2 +\\ & = \mathbb{E}(\varepsilon_i^2 ) \, \, \, = \, \, \, +\mbox{Var}(\varepsilon_i) \, \, \, = \, \, \, \sigma^2. +\end{align*} +$$ +

 
+ +Hence, \( y_i \sim \mathcal{N}( \mathbf{X}_{i, \ast} \, \boldsymbol{\beta}, \sigma^2) \), that is \( \boldsymbol{y} \) follows a normal distribution with +mean value \( \boldsymbol{X}\boldsymbol{\beta} \) and variance \( \sigma^2 \) (not be confused with the singular values of the SVD). +

+ + +
+

Expectation value and variance for \( \boldsymbol{\beta} \)

+ +

+With the OLS expressions for the parameters \( \boldsymbol{\beta} \) we can evaluate the expectation value +

 
+$$ +\mathbb{E}(\boldsymbol{\beta}) = \mathbb{E}[ (\mathbf{X}^{\top} \mathbf{X})^{-1}\mathbf{X}^{T} \mathbf{Y}]=(\mathbf{X}^{T} \mathbf{X})^{-1}\mathbf{X}^{T} \mathbb{E}[ \mathbf{Y}]=(\mathbf{X}^{T} \mathbf{X})^{-1} \mathbf{X}^{T}\mathbf{X}\boldsymbol{\beta}=\boldsymbol{\beta}. +$$ +

 
+ +This means that the estimator of the regression parameters is unbiased. + +

+We can also calculate the variance + +

+The variance of \( \boldsymbol{\beta} \) is +

 
+$$ +\begin{eqnarray*} +\mbox{Var}(\boldsymbol{\beta}) & = & \mathbb{E} \{ [\boldsymbol{\beta} - \mathbb{E}(\boldsymbol{\beta})] [\boldsymbol{\beta} - \mathbb{E}(\boldsymbol{\beta})]^{T} \} +\\ +& = & \mathbb{E} \{ [(\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{Y} - \boldsymbol{\beta}] \, [(\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{Y} - \boldsymbol{\beta}]^{T} \} +\\ +% & = & \mathbb{E} \{ [(\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{Y}] \, [(\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{Y}]^{T} \} - \boldsymbol{\beta} \, \boldsymbol{\beta}^{T} +% \\ +% & = & \mathbb{E} \{ (\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{Y} \, \mathbf{Y}^{T} \, \mathbf{X} \, (\mathbf{X}^{T} \mathbf{X})^{-1} \} - \boldsymbol{\beta} \, \boldsymbol{\beta}^{T} +% \\ +& = & (\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \, \mathbb{E} \{ \mathbf{Y} \, \mathbf{Y}^{T} \} \, \mathbf{X} \, (\mathbf{X}^{T} \mathbf{X})^{-1} - \boldsymbol{\beta} \, \boldsymbol{\beta}^{T} +\\ +& = & (\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \, \{ \mathbf{X} \, \boldsymbol{\beta} \, \boldsymbol{\beta}^{T} \, \mathbf{X}^{T} + \sigma^2 \} \, \mathbf{X} \, (\mathbf{X}^{T} \mathbf{X})^{-1} - \boldsymbol{\beta} \, \boldsymbol{\beta}^{T} +% \\ +% & = & (\mathbf{X}^T \mathbf{X})^{-1} \, \mathbf{X}^T \, \mathbf{X} \, \boldsymbol{\beta} \, \boldsymbol{\beta}^T \, \mathbf{X}^T \, \mathbf{X} \, (\mathbf{X}^T % \mathbf{X})^{-1} +% \\ +% & & + \, \, \sigma^2 \, (\mathbf{X}^T \mathbf{X})^{-1} \, \mathbf{X}^T \, \mathbf{X} \, (\mathbf{X}^T \mathbf{X})^{-1} - \boldsymbol{\beta} \boldsymbol{\beta}^T +\\ +& = & \boldsymbol{\beta} \, \boldsymbol{\beta}^{T} + \sigma^2 \, (\mathbf{X}^{T} \mathbf{X})^{-1} - \boldsymbol{\beta} \, \boldsymbol{\beta}^{T} +\, \, \, = \, \, \, \sigma^2 \, (\mathbf{X}^{T} \mathbf{X})^{-1}, +\end{eqnarray*} +$$ +

 
+ +

+where we have used that \( \mathbb{E} (\mathbf{Y} \mathbf{Y}^{T}) = +\mathbf{X} \, \boldsymbol{\beta} \, \boldsymbol{\beta}^{T} \, \mathbf{X}^{T} + +\sigma^2 \, \mathbf{I}_{nn} \). From \( \mbox{Var}(\boldsymbol{\beta}) = \sigma^2 +\, (\mathbf{X}^{T} \mathbf{X})^{-1} \), one obtains an estimate of the +variance of the estimate of the \( j \)-th regression coefficient: +\( \hat{\sigma}^2 (\hat{\beta}_j ) = \hat{\sigma}^2 \sqrt{ +[(\mathbf{X}^{T} \mathbf{X})^{-1}]_{jj} } \). This may be used to +construct a confidence interval for the estimates. + +

+In a similar way, we cna obtain analytical expressions for say the +expectation values of the parameters \( \boldsymbol{\beta} \) and their variance +when we employ Ridge regression, and thereby a confidence interval. + +

+It is rather straightforward to show that +

 
+$$ +\mathbb{E} \big[ \boldsymbol{\beta}^{\mathrm{Ridge}} \big]=(\mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I}_{pp})^{-1} (\mathbf{X}^{\top} \mathbf{X})\boldsymbol{\beta}^{\mathrm{OLS}}. +$$ +

 
+ +We see clearly that +\( \mathbb{E} \big[ \boldsymbol{\beta}^{\mathrm{Ridge}} \big] \not= \boldsymbol{\beta}^{\mathrm{OLS}} \) for any \( \lambda > 0 \). We say then that the ridge estimator is biased. + +

+We can also compute the variance as + +

 
+$$ +\mbox{Var}[\boldsymbol{\beta}^{\mathrm{Ridge}}]=\sigma^2[ \mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I} ]^{-1} \mathbf{X}^{T} \mathbf{X} \{ [ \mathbf{X}^{\top} \mathbf{X} + \lambda \mathbf{I} ]^{-1}\}^{T}, +$$ +

 
+ +and it is easy to see that if the parameter \( \lambda \) goes to infinity then the variance of Ridge parameters \( \boldsymbol{\beta} \) goes to zero. + +

+With this, we can compute the difference + +

 
+$$ +\mbox{Var}[\boldsymbol{\beta}^{\mathrm{OLS}}]-\mbox{Var}(\boldsymbol{\beta}^{\mathrm{Ridge}})=\sigma^2 [ \mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I} ]^{-1}[ 2\lambda\mathbf{I} + \lambda^2 (\mathbf{X}^{T} \mathbf{X})^{-1} ] \{ [ \mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I} ]^{-1}\}^{T}. +$$ +

 
+ +The difference is non-negative definite since each component of the +matrix product is non-negative definite. +This means the variance we obtain with the standard OLS will always for \( \lambda > 0 \) be larger than the variance of \( \boldsymbol{\beta} \) obtained with the Ridge estimator. This has interesting consequences when we discuss the so-called bias-variance trade-off below. +

+ + +
+

Cross-validation

+ +

+Instead of choosing the penalty parameter to balance model fit with +model complexity, cross-validation requires it (i.e. the penalty +parameter) to yield a model with good prediction +performance. Commonly, this performance is evaluated on novel +data. Novel data need not be easy to come by and one has to make do +with the data at hand. + +

+The setting of original and novel data is +then mimicked by sample splitting: the data set is divided into two +(groups of samples). One of these two data sets, called the +training set, plays the role of original data on which the model is +built. The second of these data sets, called the test set, plays the +role of the novel data and is used to evaluate the prediction +performance (often operationalized as the log-likelihood or the +prediction error or its square or the R2 score) of the model built on the training data set. This +procedure (model building and prediction evaluation on training and +test set, respectively) is done for a collection of possible penalty +parameter choices. The penalty parameter that yields the model with +the best prediction performance is to be preferred. The thus obtained +performance evaluation depends on the actual split of the data set. To +remove this dependence the data set is split many times into a +training and test set. For each split the model parameters are +estimated for all choices of \( \lambda \) using the training data and +estimated parameters are evaluated on the corresponding test set. The +penalty parameter that on average over the test sets performs best (in +some sense) is then selected. +

+ + +
+

Computationally expensive

+ +

+The validation set approach is conceptually simple and is easy to implement. But it has two potential drawbacks: + +

    +

  • The validation estimate of the test error rate can be highly variable, depending on precisely which observations are included in the training set and which observations are included in the validation set.
    • +

  • In the validation approach, only a subset of the observations, those that are included in the training set rather than in the validation set are used to fit the model. Since statistical methods tend to perform worse when trained on fewer observations, this suggests that the validation set error rate may tend to overestimate the test error rate for the model fit on the entire data set.
    • +
+
+ + +
+

Various steps in cross-validation

+ +

+When the repetitive splitting of the data set is done randomly, +samples may accidently end up in a fast majority of the splits in +either training or test set. Such samples may have an unbalanced +influence on either model building or prediction evaluation. To avoid +this \( k \)-fold cross-validation structures the data splitting. The +samples are divided into \( k \) more or less equally sized exhaustive and +mutually exclusive subsets. In turn (at each split) one of these +subsets plays the role of the test set while the union of the +remaining subsets constitutes the training set. Such a splitting +warrants a balanced representation of each sample in both training and +test set over the splits. Still the division into the \( k \) subsets +involves a degree of randomness. This may be fully excluded when +choosing \( k=n \). This particular case is referred to as leave-one-out +cross-validation (LOOCV). +

+ + +
+

How to set up the cross-validation for Ridge and/or Lasso

+ +
    +

  • Define a range of interest for the penalty parameter.
    • +

  • Divide the data set into training and test set comprising samples \( \{1, \ldots, n\} \setminus i \) and \( \{ i \} \), respectively.
    • +

  • Fit the linear regression model by means of ridge estimation for each \( \lambda \) in the grid using the training set, and the corresponding estimate of the error variance \( \boldsymbol{\sigma}_{-i}^2(\lambda) \), as
    • +
+

 
+$$ +\begin{align*} +\boldsymbol{\beta}_{-i}(\lambda) & = ( \boldsymbol{X}_{-i, \ast}^{T} +\boldsymbol{X}_{-i, \ast} + \lambda \boldsymbol{I}_{pp})^{-1} +\boldsymbol{X}_{-i, \ast}^{T} \boldsymbol{y}_{-i} +\end{align*} +$$ +

 
+ + +

    +

  • Evaluate the prediction performance of these models on the test set by \( \log\{L[y_i, \boldsymbol{X}_{i, \ast}; \boldsymbol{\beta}_{-i}(\lambda), \boldsymbol{\sigma}_{-i}^2(\lambda)]\} \). Or, by the prediction error \( |y_i - \boldsymbol{X}_{i, \ast} \boldsymbol{\beta}_{-i}(\lambda)| \), the relative error, the error squared or the R2 score function.
    • +

  • Repeat the first three steps such that each sample plays the role of the test set once.
    • +

  • Average the prediction performances of the test sets at each grid point of the penalty bias/parameter by computing the cross-validated log-likelihood. It is an estimate of the prediction performance of the model corresponding to this value of the penalty parameter on novel data. It is defined as
    • +
+

 
+$$ +\begin{align*} +\frac{1}{n} \sum_{i = 1}^n \log\{L[y_i, \mathbf{X}_{i, \ast}; \boldsymbol{\beta}_{-i}(\lambda), \boldsymbol{\sigma}_{-i}^2(\lambda)]\}. +\end{align*} +$$ +

 
+ + +

    +

  • The value of the penalty parameter that maximizes the cross-validated log-likelihood is the value of choice. Or we can use the MSE or the R2 score functions.
    • +
+
+ + +
+

Resampling methods: Jackknife and Bootstrap

+ +

+Two famous +resampling methods are the independent bootstrap and the jackknife. + +

+The jackknife is a special case of the independent bootstrap. Still, the jackknife was made +popular prior to the independent bootstrap. And as the popularity of +the independent bootstrap soared, new variants, such as the dependent bootstrap. + +

+The Jackknife and independent bootstrap work for +independent, identically distributed random variables. +If these conditions are not +satisfied, the methods will fail. Yet, it should be said that if the data are +independent, identically distributed, and we only want to estimate the +variance of \( \overline{X} \) (which often is the case), then there is no +need for bootstrapping. +

+ + +
+

Resampling methods: Jackknife

+ +

+The Jackknife works by making many replicas of the estimator \( \widehat{\theta} \). +The jackknife is a resampling method where we systematically leave out one observation from the vector of observed values \( \boldsymbol{x} = (x_1,x_2,\cdots,X_n) \). +Let \( \boldsymbol{x}_i \) denote the vector +

 
+$$ +\boldsymbol{x}_i = (x_1,x_2,\cdots,x_{i-1},x_{i+1},\cdots,x_n), +$$ +

 
+ +

+which equals the vector \( \boldsymbol{x} \) with the exception that observation +number \( i \) is left out. Using this notation, define +\( \widehat{\theta}_i \) to be the estimator +\( \widehat{\theta} \) computed using \( \vec{X}_i \). +

+ + +
+

Jackknife code example

+

+ + +

from numpy import *
+from numpy.random import randint, randn
+from time import time
+
+def jackknife(data, stat):
+    n = len(data);t = zeros(n); inds = arange(n); t0 = time()
+    ## 'jackknifing' by leaving out an observation for each i                                                                                                                      
+    for i in range(n):
+        t[i] = stat(delete(data,i) )
+
+    # analysis                                                                                                                                                                     
+    print("Runtime: %g sec" % (time()-t0)); print("Jackknife Statistics :")
+    print("original           bias      std. error")
+    print("%8g %14g %15g" % (stat(data),(n-1)*mean(t)/n, (n*var(t))**.5))
+
+    return t
+
+
+# Returns mean of data samples                                                                                                                                                     
+def stat(data):
+    return mean(data)
+
+
+mu, sigma = 100, 15
+datapoints = 10000
+x = mu + sigma*random.randn(datapoints)
+# jackknife returns the data sample                                                                                                                                                
+t = jackknife(x, stat)
+
+
+ + +
+

Resampling methods: Bootstrap

+
+ +

+Bootstrapping is a nonparametric approach to statistical inference +that substitutes computation for more traditional distributional +assumptions and asymptotic results. Bootstrapping offers a number of +advantages: + +

    +

  1. The bootstrap is quite general, although there are some cases in which it fails.
    2. + +

  2. Because it does not require distributional assumptions (such as normally distributed errors), the bootstrap can provide more accurate inferences when the data are not well behaved or when the sample size is small.
    3. + +

  3. It is possible to apply the bootstrap to statistics with sampling distributions that are difficult to derive, even asymptotically.
    4. +

  4. It is relatively simple to apply the bootstrap to complex data-collection plans (such as stratified and clustered samples).
    5. +
+
+
+ + +
+

Resampling methods: Bootstrap background

+ +

+Since \( \widehat{\theta} = \widehat{\theta}(\boldsymbol{X}) \) is a function of random variables, +\( \widehat{\theta} \) itself must be a random variable. Thus it has +a pdf, call this function \( p(\boldsymbol{t}) \). The aim of the bootstrap is to +estimate \( p(\boldsymbol{t}) \) by the relative frequency of +\( \widehat{\theta} \). You can think of this as using a histogram +in the place of \( p(\boldsymbol{t}) \). If the relative frequency closely +resembles \( p(\vec{t}) \), then using numerics, it is straight forward to +estimate all the interesting parameters of \( p(\boldsymbol{t}) \) using point +estimators. +

+ + +
+

Resampling methods: More Bootstrap background

+ +

+In the case that \( \widehat{\theta} \) has +more than one component, and the components are independent, we use the +same estimator on each component separately. If the probability +density function of \( X_i \), \( p(x) \), had been known, then it would have +been straight forward to do this by: + +

    +

  1. Drawing lots of numbers from \( p(x) \), suppose we call one such set of numbers \( (X_1^*, X_2^*, \cdots, X_n^*) \).
    2. +

  2. Then using these numbers, we could compute a replica of \( \widehat{\theta} \) called \( \widehat{\theta}^* \).
    3. +
+

+ +By repeated use of (1) and (2), many +estimates of \( \widehat{\theta} \) could have been obtained. The +idea is to use the relative frequency of \( \widehat{\theta}^* \) +(think of a histogram) as an estimate of \( p(\boldsymbol{t}) \). +

+ + +
+

Resampling methods: Bootstrap approach

+ +

+But +unless there is enough information available about the process that +generated \( X_1,X_2,\cdots,X_n \), \( p(x) \) is in general +unknown. Therefore, Efron in 1979 asked the +question: What if we replace \( p(x) \) by the relative frequency +of the observation \( X_i \); if we draw observations in accordance with +the relative frequency of the observations, will we obtain the same +result in some asymptotic sense? The answer is yes. + +

+Instead of generating the histogram for the relative +frequency of the observation \( X_i \), just draw the values +\( (X_1^*,X_2^*,\cdots,X_n^*) \) with replacement from the vector +\( \boldsymbol{X} \). +

+ + +
+

Resampling methods: Bootstrap steps

+ +

+The independent bootstrap works like this: + +

    +

  1. Draw with replacement \( n \) numbers for the observed variables \( \boldsymbol{x} = (x_1,x_2,\cdots,x_n) \).
    2. +

  2. Define a vector \( \boldsymbol{x}^* \) containing the values which were drawn from \( \boldsymbol{x} \).
    3. +

  3. Using the vector \( \boldsymbol{x}^* \) compute \( \widehat{\theta}^* \) by evaluating \( \widehat \theta \) under the observations \( \boldsymbol{x}^* \).
    4. +

  4. Repeat this process \( k \) times.
    5. +
+

+ +When you are done, you can draw a histogram of the relative frequency +of \( \widehat \theta^* \). This is your estimate of the probability +distribution \( p(t) \). Using this probability distribution you can +estimate any statistics thereof. In principle you never draw the +histogram of the relative frequency of \( \widehat{\theta}^* \). Instead +you use the estimators corresponding to the statistic of interest. For +example, if you are interested in estimating the variance of \( \widehat +\theta \), apply the etsimator \( \widehat \sigma^2 \) to the values +\( \widehat \theta ^* \). +

+ + +
+

Code example for the Bootstrap method

+ +

+The following code starts with a Gaussian distribution with mean value +\( \mu =100 \) and variance \( \sigma=15 \). We use this to generate the data +used in the bootstrap analysis. The bootstrap analysis returns a data +set after a given number of bootstrap operations (as many as we have +data points). This data set consists of estimated mean values for each +bootstrap operation. The histogram generated by the bootstrap method +shows that the distribution for these mean values is also a Gaussian, +centered around the mean value \( \mu=100 \) but with standard deviation +\( \sigma/\sqrt{n} \), where \( n \) is the number of bootstrap samples (in +this case the same as the number of original data points). The value +of the standard deviation is what we expect from the central limit +theorem. + +

+ + +

from numpy import *
+from numpy.random import randint, randn
+from time import time
+import matplotlib.mlab as mlab
+import matplotlib.pyplot as plt
+
+# Returns mean of bootstrap samples                                                                                                                                                
+def stat(data):
+    return mean(data)
+
+# Bootstrap algorithm
+def bootstrap(data, statistic, R):
+    t = zeros(R); n = len(data); inds = arange(n); t0 = time()
+    # non-parametric bootstrap         
+    for i in range(R):
+        t[i] = statistic(data[randint(0,n,n)])
+
+    # analysis    
+    print("Runtime: %g sec" % (time()-t0)); print("Bootstrap Statistics :")
+    print("original           bias      std. error")
+    print("%8g %8g %14g %15g" % (statistic(data), std(data),mean(t),std(t)))
+    return t
+
+
+mu, sigma = 100, 15
+datapoints = 10000
+x = mu + sigma*random.randn(datapoints)
+# bootstrap returns the data sample                                    
+t = bootstrap(x, stat, datapoints)
+# the histogram of the bootstrapped  data                                                                                                    
+n, binsboot, patches = plt.hist(t, 50, normed=1, facecolor='red', alpha=0.75)
+
+# add a 'best fit' line  
+y = mlab.normpdf( binsboot, mean(t), std(t))
+lt = plt.plot(binsboot, y, 'r--', linewidth=1)
+plt.xlabel('Smarts')
+plt.ylabel('Probability')
+plt.axis([99.5, 100.6, 0, 3.0])
+plt.grid(True)
+
+plt.show()
+
+
+ + +
+

Code Example for Cross-validation and \( k \)-fold Cross-validation

+ +

+The code here uses Ridge regression with cross-validation (CV) resampling and \( k \)-fold CV in order to fit a specific polynomial. +

+ + +

import numpy as np
+import matplotlib.pyplot as plt
+from sklearn.model_selection import KFold
+from sklearn.linear_model import Ridge
+from sklearn.model_selection import cross_val_score
+from sklearn.preprocessing import PolynomialFeatures
+
+# A seed just to ensure that the random numbers are the same for every run.
+# Useful for eventual debugging.
+np.random.seed(3155)
+
+# Generate the data.
+nsamples = 100
+x = np.random.randn(nsamples)
+y = 3*x**2 + np.random.randn(nsamples)
+
+## Cross-validation on Ridge regression using KFold only
+
+# Decide degree on polynomial to fit
+poly = PolynomialFeatures(degree = 6)
+
+# Decide which values of lambda to use
+nlambdas = 500
+lambdas = np.logspace(-3, 5, nlambdas)
+
+# Initialize a KFold instance
+k = 5
+kfold = KFold(n_splits = k)
+
+# Perform the cross-validation to estimate MSE
+scores_KFold = np.zeros((nlambdas, k))
+
+i = 0
+for lmb in lambdas:
+    ridge = Ridge(alpha = lmb)
+    j = 0
+    for train_inds, test_inds in kfold.split(x):
+        xtrain = x[train_inds]
+        ytrain = y[train_inds]
+
+        xtest = x[test_inds]
+        ytest = y[test_inds]
+
+        Xtrain = poly.fit_transform(xtrain[:, np.newaxis])
+        ridge.fit(Xtrain, ytrain[:, np.newaxis])
+
+        Xtest = poly.fit_transform(xtest[:, np.newaxis])
+        ypred = ridge.predict(Xtest)
+
+        scores_KFold[i,j] = np.sum((ypred - ytest[:, np.newaxis])**2)/np.size(ypred)
+
+        j += 1
+    i += 1
+
+
+estimated_mse_KFold = np.mean(scores_KFold, axis = 1)
+
+## Cross-validation using cross_val_score from sklearn along with KFold
+
+# kfold is an instance initialized above as:
+# kfold = KFold(n_splits = k)
+
+estimated_mse_sklearn = np.zeros(nlambdas)
+i = 0
+for lmb in lambdas:
+    ridge = Ridge(alpha = lmb)
+
+    X = poly.fit_transform(x[:, np.newaxis])
+    estimated_mse_folds = cross_val_score(ridge, X, y[:, np.newaxis], scoring='neg_mean_squared_error', cv=kfold)
+
+    # cross_val_score return an array containing the estimated negative mse for every fold.
+    # we have to the the mean of every array in order to get an estimate of the mse of the model
+    estimated_mse_sklearn[i] = np.mean(-estimated_mse_folds)
+
+    i += 1
+
+## Plot and compare the slightly different ways to perform cross-validation
+
+plt.figure()
+
+plt.plot(np.log10(lambdas), estimated_mse_sklearn, label = 'cross_val_score')
+plt.plot(np.log10(lambdas), estimated_mse_KFold, 'r--', label = 'KFold')
+
+plt.xlabel('log10(lambda)')
+plt.ylabel('mse')
+
+plt.legend()
+
+plt.show()
+
+
+ + +
+

The bias-variance tradeoff

+ +

+We will discuss the bias-variance tradeoff in the context of +continuous predictions such as regression. However, many of the +intuitions and ideas discussed here also carry over to classification +tasks. Consider a dataset \( \mathcal{L} \) consisting of the data +\( \mathbf{X}_\mathcal{L}=\{(y_j, \boldsymbol{x}_j), j=0\ldots n-1\} \). + +

+Let us assume that the true data is generated from a noisy model + +

 
+$$ +\boldsymbol{y}=f(\boldsymbol{x}) + \boldsymbol{\epsilon} +$$ +

 
+ +

+where \( \epsilon \) is normally distributed with mean zero and standard deviation \( \sigma^2 \). + +

+In our derivation of the ordinary least squares method we defined then +an approximation to the function \( f \) in terms of the parameters +\( \boldsymbol{\beta} \) and the design matrix \( \boldsymbol{X} \) which embody our model, +that is \( \boldsymbol{\tilde{y}}=\boldsymbol{X}\boldsymbol{\beta} \). + +

+Thereafter we found the parameters \( \boldsymbol{\beta} \) by optimizing the means squared error via the so-called cost function +

 
+$$ +C(\boldsymbol{X},\boldsymbol{\beta}) =\frac{1}{n}\sum_{i=0}^{n-1}(y_i-\tilde{y}_i)^2=\mathbb{E}\left[(\boldsymbol{y}-\boldsymbol{\tilde{y}})^2\right]. +$$ +

 
+ +

+We can rewrite this as +

 
+$$ +\mathbb{E}\left[(\boldsymbol{y}-\boldsymbol{\tilde{y}})^2\right]=\frac{1}{n}\sum_i(f_i-\mathbb{E}\left[\boldsymbol{\tilde{y}}\right])^2+\frac{1}{n}\sum_i(\tilde{y}_i-\mathbb{E}\left[\boldsymbol{\tilde{y}}\right])^2+\sigma^2. +$$ +

 
+ +

+The three terms represent the square of the bias of the learning +method, which can be thought of as the error caused by the simplifying +assumptions built into the method. The second term represents the +variance of the chosen model and finally the last terms is variance of +the error \( \boldsymbol{\epsilon} \). + +

+To derive this equation, we need to recall that the variance of \( \boldsymbol{y} \) and \( \boldsymbol{\epsilon} \) are both equal to \( \sigma^2 \). The mean value of \( \boldsymbol{\epsilon} \) is by definition equal to zero. Furthermore, the function \( f \) is not a stochastics variable, idem for \( \boldsymbol{\tilde{y}} \). +We use a more compact notation in terms of the expectation value +

 
+$$ +\mathbb{E}\left[(\boldsymbol{y}-\boldsymbol{\tilde{y}})^2\right]=\mathbb{E}\left[(\boldsymbol{f}+\boldsymbol{\epsilon}-\boldsymbol{\tilde{y}})^2\right], +$$ +

 
+ +and adding and subtracting \( \mathbb{E}\left[\boldsymbol{\tilde{y}}\right] \) we get +

 
+$$ +\mathbb{E}\left[(\boldsymbol{y}-\boldsymbol{\tilde{y}})^2\right]=\mathbb{E}\left[(\boldsymbol{f}+\boldsymbol{\epsilon}-\boldsymbol{\tilde{y}}+\mathbb{E}\left[\boldsymbol{\tilde{y}}\right]-\mathbb{E}\left[\boldsymbol{\tilde{y}}\right])^2\right], +$$ +

 
+ +which, using the abovementioned expectation values can be rewritten as +

 
+$$ +\mathbb{E}\left[(\boldsymbol{y}-\boldsymbol{\tilde{y}})^2\right]=\mathbb{E}\left[(\boldsymbol{y}-\mathbb{E}\left[\boldsymbol{\tilde{y}}\right])^2\right]+\mathrm{Var}\left[\boldsymbol{\tilde{y}}\right]+\sigma^2, +$$ +

 
+ +that is the rewriting in terms of the so-called bias, the variance of the model \( \boldsymbol{\tilde{y}} \) and the variance of \( \boldsymbol{\epsilon} \). +

+ + +
+

Example code for Bias-Variance tradeoff

+

+ + +

import matplotlib.pyplot as plt
+import numpy as np
+from sklearn.linear_model import LinearRegression, Ridge, Lasso
+from sklearn.preprocessing import PolynomialFeatures
+from sklearn.model_selection import train_test_split
+from sklearn.pipeline import make_pipeline
+from sklearn.utils import resample
+
+np.random.seed(2018)
+
+n = 500
+n_boostraps = 100
+degree = 18  # A quite high value, just to show.
+noise = 0.1
+
+# Make data set.
+x = np.linspace(-1, 3, n).reshape(-1, 1)
+y = np.exp(-x**2) + 1.5 * np.exp(-(x-2)**2) + np.random.normal(0, 0.1, x.shape)
+
+# Hold out some test data that is never used in training.
+x_train, x_test, y_train, y_test = train_test_split(x, y, test_size=0.2)
+
+# Combine x transformation and model into one operation.
+# Not neccesary, but convenient.
+model = make_pipeline(PolynomialFeatures(degree=degree), LinearRegression(fit_intercept=False))
+
+# The following (m x n_bootstraps) matrix holds the column vectors y_pred
+# for each bootstrap iteration.
+y_pred = np.empty((y_test.shape[0], n_boostraps))
+for i in range(n_boostraps):
+    x_, y_ = resample(x_train, y_train)
+
+    # Evaluate the new model on the same test data each time.
+    y_pred[:, i] = model.fit(x_, y_).predict(x_test).ravel()
+
+# Note: Expectations and variances taken w.r.t. different training
+# data sets, hence the axis=1. Subsequent means are taken across the test data
+# set in order to obtain a total value, but before this we have error/bias/variance
+# calculated per data point in the test set.
+# Note 2: The use of keepdims=True is important in the calculation of bias as this 
+# maintains the column vector form. Dropping this yields very unexpected results.
+error = np.mean( np.mean((y_test - y_pred)**2, axis=1, keepdims=True) )
+bias = np.mean( (y_test - np.mean(y_pred, axis=1, keepdims=True))**2 )
+variance = np.mean( np.var(y_pred, axis=1, keepdims=True) )
+print('Error:', error)
+print('Bias^2:', bias)
+print('Var:', variance)
+print('{} >= {} + {} = {}'.format(error, bias, variance, bias+variance))
+
+plt.plot(x[::5, :], y[::5, :], label='f(x)')
+plt.scatter(x_test, y_test, label='Data points')
+plt.scatter(x_test, np.mean(y_pred, axis=1), label='Pred')
+plt.legend()
+plt.show()
+
+
+ + +
+

Understanding what happens

+

+ + +

import matplotlib.pyplot as plt
+import numpy as np
+from sklearn.linear_model import LinearRegression, Ridge, Lasso
+from sklearn.preprocessing import PolynomialFeatures
+from sklearn.model_selection import train_test_split
+from sklearn.pipeline import make_pipeline
+from sklearn.utils import resample
+
+np.random.seed(2018)
+
+n = 40
+n_boostraps = 100
+maxdegree = 14
+
+
+# Make data set.
+x = np.linspace(-3, 3, n).reshape(-1, 1)
+y = np.exp(-x**2) + 1.5 * np.exp(-(x-2)**2)+ np.random.normal(0, 0.1, x.shape)
+error = np.zeros(maxdegree)
+bias = np.zeros(maxdegree)
+variance = np.zeros(maxdegree)
+polydegree = np.zeros(maxdegree)
+x_train, x_test, y_train, y_test = train_test_split(x, y, test_size=0.2)
+
+for degree in range(maxdegree):
+    model = make_pipeline(PolynomialFeatures(degree=degree), LinearRegression(fit_intercept=False))
+    y_pred = np.empty((y_test.shape[0], n_boostraps))
+    for i in range(n_boostraps):
+        x_, y_ = resample(x_train, y_train)
+        y_pred[:, i] = model.fit(x_, y_).predict(x_test).ravel()
+
+    polydegree[degree] = degree
+    error[degree] = np.mean( np.mean((y_test - y_pred)**2, axis=1, keepdims=True) )
+    bias[degree] = np.mean( (y_test - np.mean(y_pred, axis=1, keepdims=True))**2 )
+    variance[degree] = np.mean( np.var(y_pred, axis=1, keepdims=True) )
+    print('Polynomial degree:', degree)
+    print('Error:', error[degree])
+    print('Bias^2:', bias[degree])
+    print('Var:', variance[degree])
+    print('{} >= {} + {} = {}'.format(error[degree], bias[degree], variance[degree], bias[degree]+variance[degree]))
+
+plt.plot(polydegree, np.log10(error), label='Error')
+plt.plot(polydegree, bias, label='bias')
+plt.plot(polydegree, variance, label='Variance')
+plt.legend()
+plt.show()
+
+
+ + +
+

Summing up

+ +

+The bias-variance tradeoff summarizes the fundamental tension in +machine learning, particularly supervised learning, between the +complexity of a model and the amount of training data needed to train +it. Since data is often limited, in practice it is often useful to +use a less-complex model with higher bias, that is a model whose asymptotic +performance is worse than another model because it is easier to +train and less sensitive to sampling noise arising from having a +finite-sized training dataset (smaller variance). + +

+The above equations tell us that in +order to minimize the expected test error, we need to select a +statistical learning method that simultaneously achieves low variance +and low bias. Note that variance is inherently a nonnegative quantity, +and squared bias is also nonnegative. Hence, we see that the expected +test MSE can never lie below \( Var(\epsilon) \), the irreducible error. + +

+What do we mean by the variance and bias of a statistical learning +method? The variance refers to the amount by which our model would change if we +estimated it using a different training data set. Since the training +data are used to fit the statistical learning method, different +training data sets will result in a different estimate. But ideally the +estimate for our model should not vary too much between training +sets. However, if a method has high variance then small changes in +the training data can result in large changes in the model. In general, more +flexible statistical methods have higher variance. +

+ + +
+

Another Example rom Scikit-Learn's Repository

+

+ + +

"""
+============================
+Underfitting vs. Overfitting
+============================
+
+This example demonstrates the problems of underfitting and overfitting and
+how we can use linear regression with polynomial features to approximate
+nonlinear functions. The plot shows the function that we want to approximate,
+which is a part of the cosine function. In addition, the samples from the
+real function and the approximations of different models are displayed. The
+models have polynomial features of different degrees. We can see that a
+linear function (polynomial with degree 1) is not sufficient to fit the
+training samples. This is called **underfitting**. A polynomial of degree 4
+approximates the true function almost perfectly. However, for higher degrees
+the model will **overfit** the training data, i.e. it learns the noise of the
+training data.
+We evaluate quantitatively **overfitting** / **underfitting** by using
+cross-validation. We calculate the mean squared error (MSE) on the validation
+set, the higher, the less likely the model generalizes correctly from the
+training data.
+"""
+
+print(__doc__)
+
+import numpy as np
+import matplotlib.pyplot as plt
+from sklearn.pipeline import Pipeline
+from sklearn.preprocessing import PolynomialFeatures
+from sklearn.linear_model import LinearRegression
+from sklearn.model_selection import cross_val_score
+
+
+def true_fun(X):
+    return np.cos(1.5 * np.pi * X)
+
+np.random.seed(0)
+
+n_samples = 30
+degrees = [1, 4, 15]
+
+X = np.sort(np.random.rand(n_samples))
+y = true_fun(X) + np.random.randn(n_samples) * 0.1
+
+plt.figure(figsize=(14, 5))
+for i in range(len(degrees)):
+    ax = plt.subplot(1, len(degrees), i + 1)
+    plt.setp(ax, xticks=(), yticks=())
+
+    polynomial_features = PolynomialFeatures(degree=degrees[i],
+                                             include_bias=False)
+    linear_regression = LinearRegression()
+    pipeline = Pipeline([("polynomial_features", polynomial_features),
+                         ("linear_regression", linear_regression)])
+    pipeline.fit(X[:, np.newaxis], y)
+
+    # Evaluate the models using crossvalidation
+    scores = cross_val_score(pipeline, X[:, np.newaxis], y,
+                             scoring="neg_mean_squared_error", cv=10)
+
+    X_test = np.linspace(0, 1, 100)
+    plt.plot(X_test, pipeline.predict(X_test[:, np.newaxis]), label="Model")
+    plt.plot(X_test, true_fun(X_test), label="True function")
+    plt.scatter(X, y, edgecolor='b', s=20, label="Samples")
+    plt.xlabel("x")
+    plt.ylabel("y")
+    plt.xlim((0, 1))
+    plt.ylim((-2, 2))
+    plt.legend(loc="best")
+    plt.title("Degree {}\nMSE = {:.2e}(+/- {:.2e})".format(
+        degrees[i], -scores.mean(), scores.std()))
+plt.show()
+
+
+ + +
+

The one-dimensional Ising model

+ +

+Let us bring back the Ising model again, but now with an additional +focus on Ridge and Lasso regression as well. We repeat some of the +basic parts of the Ising model and the setup of the training and test +data. The one-dimensional Ising model with nearest neighbor +interaction, no external field and a constant coupling constant \( J \) is +given by + +

 
+$$ +\begin{align} + H = -J \sum_{k}^L s_k s_{k + 1}, +\tag{27} +\end{align} +$$ +

 
+ +where \( s_i \in \{-1, 1\} \) and \( s_{N + 1} = s_1 \). The number of spins in the system is determined by \( L \). For the one-dimensional system there is no phase transition. + +

+We will look at a system of \( L = 40 \) spins with a coupling constant of \( J = 1 \). To get enough training data we will generate 10000 states with their respective energies. + +

+ + +

import numpy as np
+import matplotlib.pyplot as plt
+from mpl_toolkits.axes_grid1 import make_axes_locatable
+import seaborn as sns
+import scipy.linalg as scl
+from sklearn.model_selection import train_test_split
+import sklearn.linear_model as skl
+import tqdm
+sns.set(color_codes=True)
+cmap_args=dict(vmin=-1., vmax=1., cmap='seismic')
+
+L = 40
+n = int(1e4)
+
+spins = np.random.choice([-1, 1], size=(n, L))
+J = 1.0
+
+energies = np.zeros(n)
+
+for i in range(n):
+    energies[i] = - J * np.dot(spins[i], np.roll(spins[i], 1))
+
+

+A more general form for the one-dimensional Ising model is + +

 
+$$ +\begin{align} + H = - \sum_j^L \sum_k^L s_j s_k J_{jk}. +\tag{28} +\end{align} +$$ +

 
+ +

+Here we allow for interactions beyond the nearest neighbors and a more +adaptive coupling matrix. This latter expression can be formulated as +a matrix-product on the form +

 
+$$ +\begin{align} + H = X J, +\tag{29} +\end{align} +$$ +

 
+ +

+where \( X_{jk} = s_j s_k \) and \( J \) is the matrix consisting of the +elements \( -J_{jk} \). This form of writing the energy fits perfectly +with the form utilized in linear regression, viz. +

 
+$$ +\begin{align} + \boldsymbol{y} = \boldsymbol{X}\boldsymbol{\beta} + \boldsymbol{\epsilon}. +\tag{30} +\end{align} +$$ +

 
+ +We organize the data as we did above +

+ + +

X = np.zeros((n, L ** 2))
+for i in range(n):
+    X[i] = np.outer(spins[i], spins[i]).ravel()
+y = energies
+X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.96)
+
+X_train_own = np.concatenate(
+    (np.ones(len(X_train))[:, np.newaxis], X_train),
+    axis=1
+)
+
+X_test_own = np.concatenate(
+    (np.ones(len(X_test))[:, np.newaxis], X_test),
+    axis=1
+)
+
+

+We will do all fitting with Scikit-Learn, + +

+ + +

clf = skl.LinearRegression().fit(X_train, y_train)
+
+

+When extracting the \( J \)-matrix we make sure to remove the intercept +

+ + +

J_sk = clf.coef_.reshape(L, L)
+
+

+And then we plot the results +

+ + +

fig = plt.figure(figsize=(20, 14))
+im = plt.imshow(J_sk, **cmap_args)
+plt.title("LinearRegression from Scikit-learn", fontsize=18)
+plt.xticks(fontsize=18)
+plt.yticks(fontsize=18)
+cb = fig.colorbar(im)
+cb.ax.set_yticklabels(cb.ax.get_yticklabels(), fontsize=18)
+plt.show()
+
+

+The results perfectly with our previous discussion where we used our own code. +

+ + +
+

Ridge regression

+ +

+Having explored the ordinary least squares we move on to ridge +regression. In ridge regression we include a regularizer. This +involves a new cost function which leads to a new estimate for the +weights \( \boldsymbol{\beta} \). This results in a penalized regression problem. The +cost function is given by + +

 
+$$ +\begin{align} + C(\boldsymbol{X}, \boldsymbol{\beta}; \lambda) = (\boldsymbol{X}\boldsymbol{\beta} - \boldsymbol{y})^T(\boldsymbol{X}\boldsymbol{\beta} - \boldsymbol{y}) + \lambda \boldsymbol{\beta}^T\boldsymbol{\beta}. +\tag{31} +\end{align} +$$ +

 
+ +

+ + +

_lambda = 0.1
+clf_ridge = skl.Ridge(alpha=_lambda).fit(X_train, y_train)
+J_ridge_sk = clf_ridge.coef_.reshape(L, L)
+fig = plt.figure(figsize=(20, 14))
+im = plt.imshow(J_ridge_sk, **cmap_args)
+plt.title("Ridge from Scikit-learn", fontsize=18)
+plt.xticks(fontsize=18)
+plt.yticks(fontsize=18)
+cb = fig.colorbar(im)
+cb.ax.set_yticklabels(cb.ax.get_yticklabels(), fontsize=18)
+
+plt.show()
+
+
+ + +
+

LASSO regression

+ +

+In the Least Absolute Shrinkage and Selection Operator (LASSO)-method we get a third cost function. + +

 
+$$ +\begin{align} + C(\boldsymbol{X}, \boldsymbol{\beta}; \lambda) = (\boldsymbol{X}\boldsymbol{\beta} - \boldsymbol{y})^T(\boldsymbol{X}\boldsymbol{\beta} - \boldsymbol{y}) + \lambda \sqrt{\boldsymbol{\beta}^T\boldsymbol{\beta}}. +\tag{32} +\end{align} +$$ +

 
+ +

+Finding the extremal point of this cost function is not so straight-forward as in least squares and ridge. We will therefore rely solely on the function ``Lasso`` from Scikit-Learn. + +

+ + +

clf_lasso = skl.Lasso(alpha=_lambda).fit(X_train, y_train)
+J_lasso_sk = clf_lasso.coef_.reshape(L, L)
+fig = plt.figure(figsize=(20, 14))
+im = plt.imshow(J_lasso_sk, **cmap_args)
+plt.title("Lasso from Scikit-learn", fontsize=18)
+plt.xticks(fontsize=18)
+plt.yticks(fontsize=18)
+cb = fig.colorbar(im)
+cb.ax.set_yticklabels(cb.ax.get_yticklabels(), fontsize=18)
+
+plt.show()
+
+

+It is quite striking how LASSO breaks the symmetry of the coupling +constant as opposed to ridge and OLS. We get a sparse solution with +\( J_{j, j + 1} = -1 \). +

+ + +
+

Performance as function of the regularization parameter

+ +

+We see how the different models perform for a different set of values for \( \lambda \). + +

+ + +

lambdas = np.logspace(-4, 5, 10)
+
+train_errors = {
+    "ols_sk": np.zeros(lambdas.size),
+    "ridge_sk": np.zeros(lambdas.size),
+    "lasso_sk": np.zeros(lambdas.size)
+}
+
+test_errors = {
+    "ols_sk": np.zeros(lambdas.size),
+    "ridge_sk": np.zeros(lambdas.size),
+    "lasso_sk": np.zeros(lambdas.size)
+}
+
+plot_counter = 1
+
+fig = plt.figure(figsize=(32, 54))
+
+for i, _lambda in enumerate(tqdm.tqdm(lambdas)):
+    for key, method in zip(
+        ["ols_sk", "ridge_sk", "lasso_sk"],
+        [skl.LinearRegression(), skl.Ridge(alpha=_lambda), skl.Lasso(alpha=_lambda)]
+    ):
+        method = method.fit(X_train, y_train)
+
+        train_errors[key][i] = method.score(X_train, y_train)
+        test_errors[key][i] = method.score(X_test, y_test)
+
+        omega = method.coef_.reshape(L, L)
+
+        plt.subplot(10, 5, plot_counter)
+        plt.imshow(omega, **cmap_args)
+        plt.title(r"%s, $\lambda = %.4f$" % (key, _lambda))
+        plot_counter += 1
+
+plt.show()
+
+

+We see that LASSO reaches a good solution for low +values of \( \lambda \), but will "wither" when we increase \( \lambda \) too +much. Ridge is more stable over a larger range of values for +\( \lambda \), but eventually also fades away. +

+ + +
+

Finding the optimal value of \( \lambda \)

+ +

+To determine which value of \( \lambda \) is best we plot the accuracy of +the models when predicting the training and the testing set. We expect +the accuracy of the training set to be quite good, but if the accuracy +of the testing set is much lower this tells us that we might be +subject to an overfit model. The ideal scenario is an accuracy on the +testing set that is close to the accuracy of the training set. + +

+ + +

fig = plt.figure(figsize=(20, 14))
+
+colors = {
+    "ols_sk": "r",
+    "ridge_sk": "y",
+    "lasso_sk": "c"
+}
+
+for key in train_errors:
+    plt.semilogx(
+        lambdas,
+        train_errors[key],
+        colors[key],
+        label="Train {0}".format(key),
+        linewidth=4.0
+    )
+
+for key in test_errors:
+    plt.semilogx(
+        lambdas,
+        test_errors[key],
+        colors[key] + "--",
+        label="Test {0}".format(key),
+        linewidth=4.0
+    )
+plt.legend(loc="best", fontsize=18)
+plt.xlabel(r"$\lambda$", fontsize=18)
+plt.ylabel(r"$R^2$", fontsize=18)
+plt.tick_params(labelsize=18)
+plt.show()
+
+

+From the above figure we can see that LASSO with \( \lambda = 10^{-2} \) +achieves a very good accuracy on the test set. This by far surpasses the +other models for all values of \( \lambda \). +

+ + +
+

Further Exercises

+ +

Exercise 1

+ +

+We will generate our own dataset for a function \( y(x) \) where \( x \in [0,1] \) and defined by random numbers computed with the uniform distribution. The function \( y \) is a quadratic polynomial in \( x \) with added stochastic noise according to the normal distribution \( \cal {N}(0,1) \). +The following simple Python instructions define our \( x \) and \( y \) values (with 100 data points). +

+ + +

x = np.random.rand(100,1)
+y = 5*x*x+0.1*np.random.randn(100,1)
+
+
    +

  1. Write your own code (following the examples above) for computing the parametrization of the data set fitting a second-order polynomial.
    2. +

  2. Use thereafter scikit-learn (see again the examples in the regression slides) and compare with your own code.
    3. + +

  3. Using scikit-learn, compute also the mean square error, a risk metric corresponding to the expected value of the squared (quadratic) error defined as
    4. +
+

 
+$$ MSE(\hat{y},\hat{\tilde{y}}) = \frac{1}{n} +\sum_{i=0}^{n-1}(y_i-\tilde{y}_i)^2, +$$ +

 
+ +and the \( R^2 \) score function. +If \( \tilde{\hat{y}}_i \) is the predicted value of the \( i-th \) sample and \( y_i \) is the corresponding true value, then the score \( R^2 \) is defined as +

 
+$$ +R^2(\hat{y}, \tilde{\hat{y}}) = 1 - \frac{\sum_{i=0}^{n - 1} (y_i - \tilde{y}_i)^2}{\sum_{i=0}^{n - 1} (y_i - \bar{y})^2}, +$$ +

 
+ +where we have defined the mean value of \( \hat{y} \) as +

 
+$$ +\bar{y} = \frac{1}{n} \sum_{i=0}^{n - 1} y_i. +$$ +

 
+ +

+You can use the functionality included in scikit-learn. If you feel +for it, you can use your own program and define functions which +compute the above two functions. Discuss the meaning of these +results. Try also to vary the coefficient in front of the added +stochastic noise term and discuss the quality of the fits. + +

Exercise 2, variance of the parameters \( \beta \) in linear regression

+ +

+Show that the variance of the parameters \( \beta \) in the linear regression method (chapter 3, equation (3.8) of Trevor Hastie, Robert Tibshirani, Jerome H. Friedman, The Elements of Statistical Learning, Springer) is given as + +

 
+$$ +\mathrm{Var}(\hat{\beta}) = \left(\hat{X}^T\hat{X}\right)^{-1}\sigma^2, +$$ +

 
+ +with +

 
+$$ +\sigma^2 = \frac{1}{N-p-1}\sum_{i=1}^{N} (y_i-\tilde{y}_i)^2, +$$ +

 
+ +where we have assumed that we fit a function of degree \( p-1 \) (for example a polynomial in \( x \)). + +

Exercise 3

+ +

+This exercise is a continuation of exercise 1. We will +use the same function to generate our data set, still staying with a +simple function \( y(x) \) which we want to fit using linear regression, +but now extending the analysis to include the Ridge and the Lasso +regression methods. You can use the code under the Regression as an example on how to use the Ridge and the Lasso methods. + +

+We will thus again generate our own dataset for a function \( y(x) \) where +\( x \in [0,1] \) and defined by random numbers computed with the uniform +distribution. The function \( y \) is a quadratic polynomial in \( x \) with +added stochastic noise according to the normal distribution \( \cal{N}(0,1) \). + +

+The following simple Python instructions define our \( x \) and \( y \) values (with 100 data points). +

+ + +

x = np.random.rand(100,1)
+y = 5*x*x+0.1*np.random.randn(100,1)
+
+
    +

  1. Write your own code for the Ridge method and compute the parametrization for different values of \( \lambda \). Compare and analyze your results with those from exercise 1. Study the dependence on \( \lambda \) while also varying the strength of the noise in your expression for \( y(x) \).
    2. +

  2. Repeat the above but using the functionality of scikit-learn. Compare your code with the results from scikit-learn. Remember to run with the same random numbers for generating \( x \) and \( y \).
    3. +

  3. Our next step is to study the variance of the parameters \( \beta_1 \) and \( \beta_2 \) (assuming that we are parametrizing our function with a second-order polynomial. We will use standard linear regression and the Ridge regression. You can now opt for either writing your own function that calculates the variance of these paramaters (recall that this is equal to the diagonal elements of the matrix \( (\hat{X}^T\hat{X})+\lambda\hat{I})^{-1} \)) or use the functionality of scikit-learn and compute their variances. Discuss the results of these variances as functions
    4. +

  4. Repeat the previous step but add now the Lasso method. Discuss your results and compare with standard regression and the Ridge regression results.
    5. +

  5. Try to implement the cross-validation as well.
    6. +

  6. Finally, using scikit-learn or your own code, compute also the mean square error, a risk metric corresponding to the expected value of the squared (quadratic) error defined as
    7. +
+

 
+$$ MSE(\hat{y},\hat{\tilde{y}}) = \frac{1}{n} +\sum_{i=0}^{n-1}(y_i-\tilde{y}_i)^2, +$$ +

 
+ +and the \( R^2 \) score function. +If \( \tilde{\hat{y}}_i \) is the predicted value of the \( i-th \) sample and \( y_i \) is the corresponding true value, then the score \( R^2 \) is defined as +

 
+$$ +R^2(\hat{y}, \tilde{\hat{y}}) = 1 - \frac{\sum_{i=0}^{n - 1} (y_i - \tilde{y}_i)^2}{\sum_{i=0}^{n - 1} (y_i - \bar{y})^2}, +$$ +

 
+ +where we have defined the mean value of \( \hat{y} \) as +

 
+$$ +\bar{y} = \frac{1}{n} \sum_{i=0}^{n - 1} y_i. +$$ +

 
+ +Discuss these quantities as functions of the variable \( \lambda \) in the Ridge and Lasso regression methods. + +

Exercise 4

+ +

+We will study how +to fit polynomials to a specific two-dimensional function called +Franke's +function. This +is a function which has been widely used when testing various interpolation and fitting +algorithms. Furthermore, after having established the model and the +method, we will employ resamling techniques such as the cross-validation and/or +the bootstrap methods, in order to perform a proper assessment of our models. + +

+The Franke function, which is a weighted sum of four exponentials reads as follows +

 
+$$ +\begin{align*} +f(x,y) &= \frac{3}{4}\exp{\left(-\frac{(9x-2)^2}{4} - \frac{(9y-2)^2}{4}\right)}+\frac{3}{4}\exp{\left(-\frac{(9x+1)^2}{49}- \frac{(9y+1)}{10}\right)} \\ +&+\frac{1}{2}\exp{\left(-\frac{(9x-7)^2}{4} - \frac{(9y-3)^2}{4}\right)} -\frac{1}{5}\exp{\left(-(9x-4)^2 - (9y-7)^2\right) }. +\end{align*} +$$ +

 
+ +

+The function will be defined for \( x,y\in [0,1] \). Our first step will +be to perform an OLS regression analysis of this function, trying out +a polynomial fit with an \( x \) and \( y \) dependence of the form \( [x, y, +x^2, y^2, xy, \dots] \). We will also include cross-validation and +bootstrap as resampling techniques. As in homeworks 1 and 2, we +can use a uniform distribution to set up the arrays of values for \( x \) +and \( y \), or as in the example below just a fix values for \( x \) and \( y \) with a given step size. +In this case we will have two predictors and need to fit a +function (for example a polynomial) of \( x \) and \( y \). Thereafter we will +repeat much of the same procedure using the the Ridge and +Lasso regression methods, introducing thus a dependence on the bias +(penalty) \( \lambda \). + +

+The Python fucntion for the Franke function is included here (it performs also a three-dimensional plot of it) +

+ + +

from mpl_toolkits.mplot3d import Axes3D
+import matplotlib.pyplot as plt
+from matplotlib import cm
+from matplotlib.ticker import LinearLocator, FormatStrFormatter
+import numpy as np
+from random import random, seed
+
+fig = plt.figure()
+ax = fig.gca(projection='3d')
+
+# Make data.
+x = np.arange(0, 1, 0.05)
+y = np.arange(0, 1, 0.05)
+x, y = np.meshgrid(x,y)
+
+
+def FrankeFunction(x,y):
+    term1 = 0.75*np.exp(-(0.25*(9*x-2)**2) - 0.25*((9*y-2)**2))
+    term2 = 0.75*np.exp(-((9*x+1)**2)/49.0 - 0.1*(9*y+1))
+    term3 = 0.5*np.exp(-(9*x-7)**2/4.0 - 0.25*((9*y-3)**2))
+    term4 = -0.2*np.exp(-(9*x-4)**2 - (9*y-7)**2)
+    return term1 + term2 + term3 + term4
+
+
+z = FrankeFunction(x, y)
+
+# Plot the surface.
+surf = ax.plot_surface(x, y, z, cmap=cm.coolwarm,
+                       linewidth=0, antialiased=False)
+
+# Customize the z axis.
+ax.set_zlim(-0.10, 1.40)
+ax.zaxis.set_major_locator(LinearLocator(10))
+ax.zaxis.set_major_formatter(FormatStrFormatter('%.02f'))
+
+# Add a color bar which maps values to colors.
+fig.colorbar(surf, shrink=0.5, aspect=5)
+
+plt.show()
+
+

+We will thus again generate our own dataset for a function \( \mathrm{FrankeFunction}(x,y) \) where +\( x,y \in [0,1] \) could be defined by random numbers computed with the uniform +distribution. The function \( f(x,y) \) is the Franke function. You should explore also the addition +an added stochastic noise to this function using the normal distribution \( \cal{N}(0,1) \). + +

+Write your own code (using either a matrix inversion or a singular value decomposition from e.g., numpy ) or use your code from exercises 1 and 3 +and perform a standard least square regression analysis using polynomials in \( x \) and \( y \) up to fifth order. Find the confidence intervals of the parameters \( \beta \) by computing their variances, evaluate the Mean Squared error (MSE) +

 
+$$ MSE(\hat{y},\hat{\tilde{y}}) = \frac{1}{n} +\sum_{i=0}^{n-1}(y_i-\tilde{y}_i)^2, +$$ +

 
+ +and the \( R^2 \) score function. +If \( \tilde{\hat{y}}_i \) is the predicted value of the \( i-th \) sample and \( y_i \) is the corresponding true value, then the score \( R^2 \) is defined as +

 
+$$ +R^2(\hat{y}, \tilde{\hat{y}}) = 1 - \frac{\sum_{i=0}^{n - 1} (y_i - \tilde{y}_i)^2}{\sum_{i=0}^{n - 1} (y_i - \bar{y})^2}, +$$ +

 
+ +where we have defined the mean value of \( \hat{y} \) as +

 
+$$ +\bar{y} = \frac{1}{n} \sum_{i=0}^{n - 1} y_i. +$$ +

 
+ +

+Perform a resampling of the data where you split the data in training data and test data. Implement the \( k \)-fold cross-validation algorithm and/or the bootstrap algorithm +and evaluate again the MSE and the \( R^2 \) functions resulting from the test data. Evaluate also the bias and variance of the final models. + +

+Write then your own code for the Ridge method, either using matrix +inversion or the singular value decomposition as done for standard OLS. Perform the same analysis as in the +previous exercise (for the same polynomials and include resampling +techniques) but now for different values of \( \lambda \). Compare and +analyze your results with those obtained with standard OLS. Study the +dependence on \( \lambda \) while also varying eventually the strength of +the noise in your expression for \( \mathrm{FrankeFunction}(x,y) \). + +

+Then perform the same studies but now with Lasso regression. Use the functionalities of +scikit-learn. Give a critical discussion of the three methods and a +judgement of which model fits the data best. +

+ + + +
+
+ + + + + + + + + + + + diff --git a/doc/src/Regression/Regression-solarized.html b/doc/src/Regression/Regression-solarized.html new file mode 100644 index 000000000..6d7cff2dc --- /dev/null +++ b/doc/src/Regression/Regression-solarized.html @@ -0,0 +1,4279 @@ + + + + + + + + +Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis + + + + + + + + + + + + + + + + + + + + + + + + + + + + +

Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis

+ +

+ + +

+Morten Hjorth-Jensen [1, 2] +
+ +

+ + +

[1] Department of Physics, University of Oslo
+
[2] Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University
+
+

+

Jul 22, 2019

+
+

+









+ +

Why Linear Regression (aka Ordinary Least Squares and family)

+ +

+Fitting a continuous function with linear parameterization in terms of the parameters \( \boldsymbol{\beta} \). + +

    +
  • Method of choice for fitting a continuous function!
    • +
  • Gives an excellent introduction to central Machine Learning features with understandable pedagogical links to other methods like Neural Networks, Support Vector Machines etc
    • +
  • Analytical expression for the fitting parameters \( \boldsymbol{\beta} \)
    • +
  • Analytical expressions for statistical propertiers like mean values, variances, confidence intervals and more
    • +
  • Analytical relation with probabilistic interpretations
    • +
  • Easy to introduce basic concepts like bias-variance tradeoff, cross-validation, resampling and regularization techniques and many other ML topics
    • +
  • Easy to code! And links well with classification problems and logistic regression and neural networks
    • +
  • Allows for easy hands-on understanding of gradient descent methods
    • +
  • and many more features
    • +
+ +For more discussions of Ridge and Lasso regression, Wessel van Wieringen's article is highly recommended. +Similarly, Mehta et al's article is also recommended. + +

+









+ +

Regression analysis, overarching aims

+
+ +

+ +

+Regression modeling deals with the description of the sampling distribution of a given random variable \( y \) and how it varies as function of another variable or a set of such variables \( \boldsymbol{x} =[x_0, x_1,\dots, x_{n-1}]^T \). +The first variable is called the dependent, the outcome or the response variable while the set of variables \( \boldsymbol{x} \) is called the independent variable, or the predictor variable or the explanatory variable. + +

+A regression model aims at finding a likelihood function \( p(\boldsymbol{y}\vert \boldsymbol{x}) \), that is the conditional distribution for \( \boldsymbol{y} \) with a given \( \boldsymbol{x} \). The estimation of \( p(\boldsymbol{y}\vert \boldsymbol{x}) \) is made using a data set with + +

    +
  • \( n \) cases \( i = 0, 1, 2, \dots, n-1 \)
    • +
  • Response (target, dependent or outcome) variable \( y_i \) with \( i = 0, 1, 2, \dots, n-1 \)
    • +
  • \( p \) so-called explanatory (independent or predictor) variables \( \boldsymbol{x}_i=[x_{i0}, x_{i1}, \dots, x_{ip-1}] \) with \( i = 0, 1, 2, \dots, n-1 \) and explanatory variables running from \( 0 \) to \( p-1 \). See below for more explicit examples.
    • +
+ + The goal of the regression analysis is to extract/exploit relationship between \( \boldsymbol{y} \) and \( \boldsymbol{X} \) in or to infer causal dependencies, approximations to the likelihood functions, functional relationships and to make predictions, making fits and many other things. +
+ + +

+









+ +

Regression analysis, overarching aims II

+
+ +

+ +

+Consider an experiment in which \( p \) characteristics of \( n \) samples are +measured. The data from this experiment, for various explanatory variables \( p \) are normally represented by a matrix +\( \mathbf{X} \). + +

+The matrix \( \mathbf{X} \) is called the design +matrix. Additional information of the samples is available in the +form of \( \boldsymbol{y} \) (also as above). The variable \( \boldsymbol{y} \) is +generally referred to as the response variable. The aim of +regression analysis is to explain \( \boldsymbol{y} \) in terms of +\( \boldsymbol{X} \) through a functional relationship like \( y_i = +f(\mathbf{X}_{i,\ast}) \). When no prior knowledge on the form of +\( f(\cdot) \) is available, it is common to assume a linear relationship +between \( \boldsymbol{X} \) and \( \boldsymbol{y} \). This assumption gives rise to +the linear regression model where \( \boldsymbol{\beta} = [\beta_0, \ldots, +\beta_{p-1}]^{T} \) are the regression parameters. + +

+Linear regression gives us a set of analytical equations for the parameters \( \beta_j \). + + +

+ + +

+









+ +

Examples

+
+ +

+In order to understand the relation among the predictors \( p \), the set of data \( n \) and the target (outcome, output etc) \( \boldsymbol{y} \), +consider the model we discussed for describing nuclear binding energies. + +

+There we assumed that we could parametrize the data using a polynomial approximation based on the liquid drop model. +Assuming +$$ +BE(A) = a_0+a_1A+a_2A^{2/3}+a_3A^{-1/3}+a_4A^{-1}, +$$ + +we have five predictors, that is the intercept, the \( A \) dependent term, the \( A^{2/3} \) term and the \( A^{-1/3} \) and \( A^{-1} \) terms. +This gives \( p=0,1,2,3,4 \). Furthermore we have \( n \) entries for each predictor. It means that our design matrix is a +\( p\times n \) matrix \( \boldsymbol{X} \). + +

+Here the predictors are based on a model we have made. A popular data set which is widely encountered in ML applications is the +so-called credit card default data from Taiwan. The data set contains data on \( n=30000 \) credit card holders with predictors like gender, marital status, age, profession, education, etc. In total there are \( 24 \) such predictors or attributes leading to a design matrix of dimensionality \( 24 \times 30000 \) + + +

+ + +

+









+ +

General linear models

+
+ +

+Before we proceed let us study a case from linear algebra where we aim at fitting a set of data \( \boldsymbol{y}=[y_0,y_1,\dots,y_{n-1}] \). We could think of these data as a result of an experiment or a complicated numerical experiment. These data are functions of a series of variables \( \boldsymbol{x}=[x_0,x_1,\dots,x_{n-1}] \), that is \( y_i = y(x_i) \) with \( i=0,1,2,\dots,n-1 \). The variables \( x_i \) could represent physical quantities like time, temperature, position etc. We assume that \( y(x) \) is a smooth function. + +

+Since obtaining these data points may not be trivial, we want to use these data to fit a function which can allow us to make predictions for values of \( y \) which are not in the present set. The perhaps simplest approach is to assume we can parametrize our function in terms of a polynomial of degree \( n-1 \) with \( n \) points, that is +$$ +y=y(x) \rightarrow y(x_i)=\tilde{y}_i+\epsilon_i=\sum_{j=0}^{n-1} \beta_j x_i^j+\epsilon_i, +$$ + +where \( \epsilon_i \) is the error in our approximation. + + +

+ + +

+









+ +

Rewriting the fitting procedure as a linear algebra problem

+
+ +

+For every set of values \( y_i,x_i \) we have thus the corresponding set of equations +$$ +\begin{align*} +y_0&=\beta_0+\beta_1x_0^1+\beta_2x_0^2+\dots+\beta_{n-1}x_0^{n-1}+\epsilon_0\\ +y_1&=\beta_0+\beta_1x_1^1+\beta_2x_1^2+\dots+\beta_{n-1}x_1^{n-1}+\epsilon_1\\ +y_2&=\beta_0+\beta_1x_2^1+\beta_2x_2^2+\dots+\beta_{n-1}x_2^{n-1}+\epsilon_2\\ +\dots & \dots \\ +y_{n-1}&=\beta_0+\beta_1x_{n-1}^1+\beta_2x_{n-1}^2+\dots+\beta_{n-1}x_{n-1}^{n-1}+\epsilon_{n-1}.\\ +\end{align*} +$$ +

+ + +

+









+ +

Rewriting the fitting procedure as a linear algebra problem, more details

+
+ +

+Defining the vectors +$$ +\boldsymbol{y} = [y_0,y_1, y_2,\dots, y_{n-1}]^T, +$$ + +and +$$ +\boldsymbol{\beta} = [\beta_0,\beta_1, \beta_2,\dots, \beta_{n-1}]^T, +$$ + +and +$$ +\boldsymbol{\epsilon} = [\epsilon_0,\epsilon_1, \epsilon_2,\dots, \epsilon_{n-1}]^T, +$$ + +and the design matrix +$$ +\boldsymbol{X}= +\begin{bmatrix} +1& x_{0}^1 &x_{0}^2& \dots & \dots &x_{0}^{n-1}\\ +1& x_{1}^1 &x_{1}^2& \dots & \dots &x_{1}^{n-1}\\ +1& x_{2}^1 &x_{2}^2& \dots & \dots &x_{2}^{n-1}\\ +\dots& \dots &\dots& \dots & \dots &\dots\\ +1& x_{n-1}^1 &x_{n-1}^2& \dots & \dots &x_{n-1}^{n-1}\\ +\end{bmatrix} +$$ + +we can rewrite our equations as +$$ +\boldsymbol{y} = \boldsymbol{X}\boldsymbol{\beta}+\boldsymbol{\epsilon}. +$$ + +The above design matrix is called a Vandermonde matrix. +

+ + +

+









+ +

Generalizing the fitting procedure as a linear algebra problem

+
+ +

+ +

+We are obviously not limited to the above polynomial expansions. We +could replace the various powers of \( x \) with elements of Fourier +series or instead of \( x_i^j \) we could have \( \cos{(j x_i)} \) or \( \sin{(j +x_i)} \), or time series or other orthogonal functions. For every set +of values \( y_i,x_i \) we can then generalize the equations to + +$$ +\begin{align*} +y_0&=\beta_0x_{00}+\beta_1x_{01}+\beta_2x_{02}+\dots+\beta_{n-1}x_{0n-1}+\epsilon_0\\ +y_1&=\beta_0x_{10}+\beta_1x_{11}+\beta_2x_{12}+\dots+\beta_{n-1}x_{1n-1}+\epsilon_1\\ +y_2&=\beta_0x_{20}+\beta_1x_{21}+\beta_2x_{22}+\dots+\beta_{n-1}x_{2n-1}+\epsilon_2\\ +\dots & \dots \\ +y_{i}&=\beta_0x_{i0}+\beta_1x_{i1}+\beta_2x_{i2}+\dots+\beta_{n-1}x_{in-1}+\epsilon_i\\ +\dots & \dots \\ +y_{n-1}&=\beta_0x_{n-1,0}+\beta_1x_{n-1,2}+\beta_2x_{n-1,2}+\dots+\beta_{n-1}x_{n-1,n-1}+\epsilon_{n-1}.\\ +\end{align*} +$$ + +

+Note that we have \( p=n \) here. The matrix is symmetric. This is generally not the case! +

+ + +

+









+ +

Generalizing the fitting procedure as a linear algebra problem

+
+ +

+We redefine in turn the matrix \( \boldsymbol{X} \) as +$$ +\boldsymbol{X}= +\begin{bmatrix} +x_{00}& x_{01} &x_{02}& \dots & \dots &x_{0,n-1}\\ +x_{10}& x_{11} &x_{12}& \dots & \dots &x_{1,n-1}\\ +x_{20}& x_{21} &x_{22}& \dots & \dots &x_{2,n-1}\\ +\dots& \dots &\dots& \dots & \dots &\dots\\ +x_{n-1,0}& x_{n-1,1} &x_{n-1,2}& \dots & \dots &x_{n-1,n-1}\\ +\end{bmatrix} +$$ + +and without loss of generality we rewrite again our equations as +$$ +\boldsymbol{y} = \boldsymbol{X}\boldsymbol{\beta}+\boldsymbol{\epsilon}. +$$ + +The left-hand side of this equation is kwown. Our error vector \( \boldsymbol{\epsilon} \) and the parameter vector \( \boldsymbol{\beta} \) are our unknow quantities. How can we obtain the optimal set of \( \beta_i \) values? +

+ + +

+









+ +

Optimizing our parameters

+
+ +

+We have defined the matrix \( \boldsymbol{X} \) via the equations +$$ +\begin{align*} +y_0&=\beta_0x_{00}+\beta_1x_{01}+\beta_2x_{02}+\dots+\beta_{n-1}x_{0n-1}+\epsilon_0\\ +y_1&=\beta_0x_{10}+\beta_1x_{11}+\beta_2x_{12}+\dots+\beta_{n-1}x_{1n-1}+\epsilon_1\\ +y_2&=\beta_0x_{20}+\beta_1x_{21}+\beta_2x_{22}+\dots+\beta_{n-1}x_{2n-1}+\epsilon_1\\ +\dots & \dots \\ +y_{i}&=\beta_0x_{i0}+\beta_1x_{i1}+\beta_2x_{i2}+\dots+\beta_{n-1}x_{in-1}+\epsilon_1\\ +\dots & \dots \\ +y_{n-1}&=\beta_0x_{n-1,0}+\beta_1x_{n-1,2}+\beta_2x_{n-1,2}+\dots+\beta_{n-1}x_{n-1,n-1}+\epsilon_{n-1}.\\ +\end{align*} +$$ + +

+As we noted above, we stayed with a system with the design matrix + \( \boldsymbol{X}\in {\mathbb{R}}^{n\times n} \), that is we have \( p=n \). For reasons to come later (algorithmic arguments) we will hereafter define +our matrix as \( \boldsymbol{X}\in {\mathbb{R}}^{n\times p} \), with the predictors refering to the column numbers and the entries \( n \) being the row elements. + + +

+ + +

+









+ +

Our model for the nuclear binding energies

+ +

+In our introductory notes we looked at the so-called liguid drop model. Let us remind ourselves about what we did by looking at the code. + +

+We restate the parts of the code we are most interested in. +

+ + +

# Common imports
+import numpy as np
+import pandas as pd
+import matplotlib.pyplot as plt
+from IPython.display import display
+import os
+
+# Where to save the figures and data files
+PROJECT_ROOT_DIR = "Results"
+FIGURE_ID = "Results/FigureFiles"
+DATA_ID = "DataFiles/"
+
+if not os.path.exists(PROJECT_ROOT_DIR):
+    os.mkdir(PROJECT_ROOT_DIR)
+
+if not os.path.exists(FIGURE_ID):
+    os.makedirs(FIGURE_ID)
+
+if not os.path.exists(DATA_ID):
+    os.makedirs(DATA_ID)
+
+def image_path(fig_id):
+    return os.path.join(FIGURE_ID, fig_id)
+
+def data_path(dat_id):
+    return os.path.join(DATA_ID, dat_id)
+
+def save_fig(fig_id):
+    plt.savefig(image_path(fig_id) + ".png", format='png')
+
+infile = open(data_path("MassEval2016.dat"),'r')
+
+
+# Read the experimental data with Pandas
+Masses = pd.read_fwf(infile, usecols=(2,3,4,6,11),
+              names=('N', 'Z', 'A', 'Element', 'Ebinding'),
+              widths=(1,3,5,5,5,1,3,4,1,13,11,11,9,1,2,11,9,1,3,1,12,11,1),
+              header=39,
+              index_col=False)
+
+# Extrapolated values are indicated by '#' in place of the decimal place, so
+# the Ebinding column won't be numeric. Coerce to float and drop these entries.
+Masses['Ebinding'] = pd.to_numeric(Masses['Ebinding'], errors='coerce')
+Masses = Masses.dropna()
+# Convert from keV to MeV.
+Masses['Ebinding'] /= 1000
+
+# Group the DataFrame by nucleon number, A.
+Masses = Masses.groupby('A')
+# Find the rows of the grouped DataFrame with the maximum binding energy.
+Masses = Masses.apply(lambda t: t[t.Ebinding==t.Ebinding.max()])
+A = Masses['A']
+Z = Masses['Z']
+N = Masses['N']
+Element = Masses['Element']
+Energies = Masses['Ebinding']
+
+# Now we set up the design matrix X
+X = np.zeros((len(A),5))
+X[:,0] = 1
+X[:,1] = A
+X[:,2] = A**(2.0/3.0)
+X[:,3] = A**(-1.0/3.0)
+X[:,4] = A**(-1.0)
+# Then nice printout using pandas
+DesignMatrix = pd.DataFrame(X)
+DesignMatrix.index = A
+DesignMatrix.columns = ['1', 'A', 'A^(2/3)', 'A^(-1/3)', '1/A']
+display(DesignMatrix)
+
+

+With \( \boldsymbol{\beta}\in {\mathbb{R}}^{p\times 1} \), it means that we will hereafter write our equations for the approximation as +$$ +\boldsymbol{\tilde{y}}= \boldsymbol{X}\boldsymbol{\beta}, +$$ + +throughout these lectures. + +

+









+ +

Optimizing our parameters, more details

+
+ +

+With the above we use the design matrix to define the approximation \( \boldsymbol{\tilde{y}} \) via the unknown quantity \( \boldsymbol{\beta} \) as +$$ +\boldsymbol{\tilde{y}}= \boldsymbol{X}\boldsymbol{\beta}, +$$ + +and in order to find the optimal parameters \( \beta_i \) instead of solving the above linear algebra problem, we define a function which gives a measure of the spread between the values \( y_i \) (which represent hopefully the exact values) and the parameterized values \( \tilde{y}_i \), namely +$$ +C(\boldsymbol{\beta})=\frac{1}{n}\sum_{i=0}^{n-1}\left(y_i-\tilde{y}_i\right)^2=\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{\tilde{y}}\right)^T\left(\boldsymbol{y}-\boldsymbol{\tilde{y}}\right)\right\}, +$$ + +or using the matrix \( \boldsymbol{X} \) and in a more compact matrix-vector notation as +$$ +C(\boldsymbol{\beta})=\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{X}^T\boldsymbol{\beta}\right)^T\left(\boldsymbol{y}-\boldsymbol{X}^T\boldsymbol{\beta}\right)\right\}. +$$ + +This function is one possible way to define the so-called cost function. + +

+It is also common to define +the function \( Q \) as + +$$ +C(\boldsymbol{\beta})=\frac{1}{2n}\sum_{i=0}^{n-1}\left(y_i-\tilde{y}_i\right)^2, +$$ + +since when taking the first derivative with respect to the unknown parameters \( \beta \), the factor of \( 2 \) cancels out. +

+ + +

+









+ +

Interpretations and optimizing our parameters

+
+ +

+ +

+The function +$$ +C(\boldsymbol{\beta})=\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)^T\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)\right\}, +$$ + +can be linked to the variance of the quantity \( y_i \) if we interpret the latter as the mean value. +When linking below with the maximum likelihood approach below, we will indeed interpret \( y_i \) as a mean value (see exercises) +$$ +y_{i}=\langle y_i \rangle = \beta_0x_{i,0}+\beta_1x_{i,1}+\beta_2x_{i,2}+\dots+\beta_{n-1}x_{i,n-1}+\epsilon_i, +$$ + +

+where \( \langle y_i \rangle \) is the mean value. Keep in mind also that +till now we have treated \( y_i \) as the exact value. Normally, the +response (dependent or outcome) variable \( y_i \) the outcome of a +numerical experiment or another type of experiment and is thus only an +approximation to the true value. It is then always accompanied by an +error estimate, often limited to a statistical error estimate given by +the standard deviation discussed earlier. In the discussion here we +will treat \( y_i \) as our exact value for the response variable. + +

+In order to find the parameters \( \beta_i \) we will then minimize the spread of \( C(\boldsymbol{\beta}) \), that is we are going to solve the problem +$$ +{\displaystyle \min_{\boldsymbol{\beta}\in +{\mathbb{R}}^{p}}}\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)^T\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)\right\}. +$$ + +In practical terms it means we will require +$$ +\frac{\partial C(\boldsymbol{\beta})}{\partial \beta_j} = \frac{\partial }{\partial \beta_j}\left[ \frac{1}{n}\sum_{i=0}^{n-1}\left(y_i-\beta_0x_{i,0}-\beta_1x_{i,1}-\beta_2x_{i,2}-\dots-\beta_{n-1}x_{i,n-1}\right)^2\right]=0, +$$ + +which results in +$$ +\frac{\partial C(\boldsymbol{\beta})}{\partial \beta_j} = -\frac{2}{n}\left[ \sum_{i=0}^{n-1}x_{ij}\left(y_i-\beta_0x_{i,0}-\beta_1x_{i,1}-\beta_2x_{i,2}-\dots-\beta_{n-1}x_{i,n-1}\right)\right]=0, +$$ + +or in a matrix-vector form as +$$ +\frac{\partial C(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}} = 0 = \boldsymbol{X}^T\left( \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right). +$$ + + +

+ + +

+









+ +

Interpretations and optimizing our parameters

+
+ +

+We can rewrite +$$ +\frac{\partial C(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}} = 0 = \boldsymbol{X}^T\left( \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right), +$$ + +as +$$ +\boldsymbol{X}^T\boldsymbol{y} = \boldsymbol{X}^T\boldsymbol{X}\boldsymbol{\beta}, +$$ + +and if the matrix \( \boldsymbol{X}^T\boldsymbol{X} \) is invertible we have the solution +$$ +\boldsymbol{\beta} =\left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}. +$$ + +

+We note also that since our design matrix is defined as \( \boldsymbol{X}\in +{\mathbb{R}}^{n\times p} \), the product \( \boldsymbol{X}^T\boldsymbol{X} \in +{\mathbb{R}}^{p\times p} \). In the above case we have that \( p \ll n \), +in our case \( p=5 \) meaning that we end up with inverting a small +\( 5\times 5 \) matrix. This is a rather common situation, in many cases we end up with low-dimensional +matrices to invert. The methods discussed here and for many other +supervised learning algorithms like classification with logistic +regression or support vector machines, exhibit dimensionalities which +allow for the usage of direct linear algebra methods such as LU decomposition or Singular Value Decomposition (SVD) for finding the inverse of the matrix +\( \boldsymbol{X}^T\boldsymbol{X} \). + + +

+ + +

+









+ +

Interpretations and optimizing our parameters

+
+ +

+The residuals \( \boldsymbol{\epsilon} \) are in turn given by +$$ +\boldsymbol{\epsilon} = \boldsymbol{y}-\boldsymbol{\tilde{y}} = \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}, +$$ + +and with +$$ +\boldsymbol{X}^T\left( \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)= 0, +$$ + +we have +$$ +\boldsymbol{X}^T\boldsymbol{\epsilon}=\boldsymbol{X}^T\left( \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)= 0, +$$ + +meaning that the solution for \( \boldsymbol{\beta} \) is the one which minimizes the residuals. Later we will link this with the maximum likelihood approach. + + +

+ + +

+Let us now return to our nuclear binding energies and simply code the above equations. + +

+









+ +

Own code for Ordinary Least Squares

+ +

+It is rather straightforward to implement the matrix inversion and obtain the parameters \( \boldsymbol{\beta} \). After having defined the matrix \( \boldsymbol{X} \) we simply need to +write +

+ + +

# matrix inversion to find beta
+beta = np.linalg.inv(X.T.dot(X)).dot(X.T).dot(Energies)
+# and then make the prediction
+ytilde = X @ beta
+
+

+Alternatively, you can use the least squares functionality in Numpy as +

+ + +

fit = np.linalg.lstsq(X, Energies, rcond =None)[0]
+ytildenp = np.dot(fit,X.T)
+
+

+And finally we plot our fit with and compare with data +

+ + +

Masses['Eapprox']  = ytilde
+# Generate a plot comparing the experimental with the fitted values values.
+fig, ax = plt.subplots()
+ax.set_xlabel(r'$A = N + Z$')
+ax.set_ylabel(r'$E_\mathrm{bind}\,/\mathrm{MeV}$')
+ax.plot(Masses['A'], Masses['Ebinding'], alpha=0.7, lw=2,
+            label='Ame2016')
+ax.plot(Masses['A'], Masses['Eapprox'], alpha=0.7, lw=2, c='m',
+            label='Fit')
+ax.legend()
+save_fig("Masses2016OLS")
+plt.show()
+
+

+









+ +

Adding error analysis and training set up

+ +

+We can easily test our fit by computing the \( R2 \) score that we discussed in connection with the functionality of _Scikit_Learn_ in the introductory slides. +Since we are not using _Scikit-Learn here we can define our own \( R2 \) function as +

+ + +

def R2(y_data, y_model):
+    return 1 - np.sum((y_data - y_model) ** 2) / np.sum((y_data - np.mean(y_model)) ** 2)
+
+

+and we would be using it as +

+ + +

print(R2(Energies,ytilde))
+
+

+We can easily add our MSE score as +

+ + +

def MSE(y_data,y_model):
+    n = np.size(y_model)
+    return np.sum((y_data-y_model)**2)/n
+
+print(MSE(Energies,ytilde))
+
+

+and finally the relative error as +

+ + +

def RelativeError(y_data,y_model):
+    return abs((y_data-y_model)/y_data)
+print(RelativeError(Energies, ytilde))
+
+

+









+ +

The \( \chi^2 \) function

+
+ +

+ +

+Normally, the response (dependent or outcome) variable \( y_i \) is the +outcome of a numerical experiment or another type of experiment and is +thus only an approximation to the true value. It is then always +accompanied by an error estimate, often limited to a statistical error +estimate given by the standard deviation discussed earlier. In the +discussion here we will treat \( y_i \) as our exact value for the +response variable. + +

+Introducing the standard deviation \( \sigma_i \) for each measurement +\( y_i \), we define now the \( \chi^2 \) function (omitting the \( 1/n \) term) +as + +$$ +\chi^2(\boldsymbol{\beta})=\frac{1}{n}\sum_{i=0}^{n-1}\frac{\left(y_i-\tilde{y}_i\right)^2}{\sigma_i^2}=\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{\tilde{y}}\right)^T\frac{1}{\boldsymbol{\Sigma^2}}\left(\boldsymbol{y}-\boldsymbol{\tilde{y}}\right)\right\}, +$$ + +where the matrix \( \boldsymbol{\Sigma} \) is a diagonal matrix with \( \sigma_i \) as matrix elements. + + +

+ + +

+









+ +

The \( \chi^2 \) function

+
+ +

+ +

+In order to find the parameters \( \beta_i \) we will then minimize the spread of \( \chi^2(\boldsymbol{\beta}) \) by requiring +$$ +\frac{\partial \chi^2(\boldsymbol{\beta})}{\partial \beta_j} = \frac{\partial }{\partial \beta_j}\left[ \frac{1}{n}\sum_{i=0}^{n-1}\left(\frac{y_i-\beta_0x_{i,0}-\beta_1x_{i,1}-\beta_2x_{i,2}-\dots-\beta_{n-1}x_{i,n-1}}{\sigma_i}\right)^2\right]=0, +$$ + +which results in +$$ +\frac{\partial \chi^2(\boldsymbol{\beta})}{\partial \beta_j} = -\frac{2}{n}\left[ \sum_{i=0}^{n-1}\frac{x_{ij}}{\sigma_i}\left(\frac{y_i-\beta_0x_{i,0}-\beta_1x_{i,1}-\beta_2x_{i,2}-\dots-\beta_{n-1}x_{i,n-1}}{\sigma_i}\right)\right]=0, +$$ + +or in a matrix-vector form as +$$ +\frac{\partial \chi^2(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}} = 0 = \boldsymbol{A}^T\left( \boldsymbol{b}-\boldsymbol{A}\boldsymbol{\beta}\right). +$$ + +where we have defined the matrix \( \boldsymbol{A} =\boldsymbol{X}/\boldsymbol{\Sigma} \) with matrix elements \( a_{ij} = x_{ij}/\sigma_i \) and the vector \( \boldsymbol{b} \) with elements \( b_i = y_i/\sigma_i \). +

+ + +

+









+ +

The \( \chi^2 \) function

+
+ +

+ +

+We can rewrite +$$ +\frac{\partial \chi^2(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}} = 0 = \boldsymbol{A}^T\left( \boldsymbol{b}-\boldsymbol{A}\boldsymbol{\beta}\right), +$$ + +as +$$ +\boldsymbol{A}^T\boldsymbol{b} = \boldsymbol{A}^T\boldsymbol{A}\boldsymbol{\beta}, +$$ + +and if the matrix \( \boldsymbol{A}^T\boldsymbol{A} \) is invertible we have the solution +$$ +\boldsymbol{\beta} =\left(\boldsymbol{A}^T\boldsymbol{A}\right)^{-1}\boldsymbol{A}^T\boldsymbol{b}. +$$ +

+ + +

+









+ +

The \( \chi^2 \) function

+
+ +

+ +

+If we then introduce the matrix +$$ +\boldsymbol{H} = \left(\boldsymbol{A}^T\boldsymbol{A}\right)^{-1}, +$$ + +we have then the following expression for the parameters \( \beta_j \) (the matrix elements of \( \boldsymbol{H} \) are \( h_{ij} \)) +$$ +\beta_j = \sum_{k=0}^{p-1}h_{jk}\sum_{i=0}^{n-1}\frac{y_i}{\sigma_i}\frac{x_{ik}}{\sigma_i} = \sum_{k=0}^{p-1}h_{jk}\sum_{i=0}^{n-1}b_ia_{ik} +$$ + +We state without proof the expression for the uncertainty in the parameters \( \beta_j \) as (we leave this as an exercise) +$$ +\sigma^2(\beta_j) = \sum_{i=0}^{n-1}\sigma_i^2\left( \frac{\partial \beta_j}{\partial y_i}\right)^2, +$$ + +resulting in +$$ +\sigma^2(\beta_j) = \left(\sum_{k=0}^{p-1}h_{jk}\sum_{i=0}^{n-1}a_{ik}\right)\left(\sum_{l=0}^{p-1}h_{jl}\sum_{m=0}^{n-1}a_{ml}\right) = h_{jj}! +$$ +

+ + +

+









+ +

The \( \chi^2 \) function

+
+ +

+The first step here is to approximate the function \( y \) with a first-order polynomial, that is we write +$$ +y=y(x) \rightarrow y(x_i) \approx \beta_0+\beta_1 x_i. +$$ + +By computing the derivatives of \( \chi^2 \) with respect to \( \beta_0 \) and \( \beta_1 \) show that these are given by +$$ +\frac{\partial \chi^2(\boldsymbol{\beta})}{\partial \beta_0} = -2\left[ \frac{1}{n}\sum_{i=0}^{n-1}\left(\frac{y_i-\beta_0-\beta_1x_{i}}{\sigma_i^2}\right)\right]=0, +$$ + +and +$$ +\frac{\partial \chi^2(\boldsymbol{\beta})}{\partial \beta_1} = -\frac{2}{n}\left[ \sum_{i=0}^{n-1}x_i\left(\frac{y_i-\beta_0-\beta_1x_{i}}{\sigma_i^2}\right)\right]=0. +$$ +

+ + +

+









+ +

The \( \chi^2 \) function

+
+ +

+ +

+For a linear fit (a first-order polynomial) we don't need to invert a matrix!! +Defining +$$ +\gamma = \sum_{i=0}^{n-1}\frac{1}{\sigma_i^2}, +$$ + + +$$ +\gamma_x = \sum_{i=0}^{n-1}\frac{x_{i}}{\sigma_i^2}, +$$ + + +$$ +\gamma_y = \sum_{i=0}^{n-1}\left(\frac{y_i}{\sigma_i^2}\right), +$$ + + +$$ +\gamma_{xx} = \sum_{i=0}^{n-1}\frac{x_ix_{i}}{\sigma_i^2}, +$$ + + +$$ +\gamma_{xy} = \sum_{i=0}^{n-1}\frac{y_ix_{i}}{\sigma_i^2}, +$$ + +

+we obtain + +$$ +\beta_0 = \frac{\gamma_{xx}\gamma_y-\gamma_x\gamma_y}{\gamma\gamma_{xx}-\gamma_x^2}, +$$ + + +$$ +\beta_1 = \frac{\gamma_{xy}\gamma-\gamma_x\gamma_y}{\gamma\gamma_{xx}-\gamma_x^2}. +$$ + +

+This approach (different linear and non-linear regression) suffers +often from both being underdetermined and overdetermined in the +unknown coefficients \( \beta_i \). A better approach is to use the +Singular Value Decomposition (SVD) method discussed below. Or using +Lasso and Ridge regression. See below. + + +

+ + +

+









+ +

Fitting an Equation of State for Dense Nuclear Matter

+ +

+Before we continue, let us introduce yet another example. We are going to fit the +nuclear equation of state using results from many-body calculations. +The equation of state we have made available here, as function of +density, has been derived using modern nucleon-nucleon potentials with +the addition of three-body +forces. This +time the file is presented as a standard csv file. + +

+The beginning of the Python code here is similar to what you have seen before, +with the same initializations and declarations. We use also pandas +again, rather extensively in order to organize our data. + +

+The difference now is that we use Scikit-Learn's regression tools +instead of our own matrix inversion implementation. Furthermore, we +sneak in Ridge regression (to be discussed below) which includes a +hyperparameter \( \lambda \), also to be explained below. + +

+









+ +

The code

+ +

+ + +

# Common imports
+import os
+import numpy as np
+import pandas as pd
+import matplotlib.pyplot as plt
+import matplotlib.pyplot as plt
+import sklearn.linear_model as skl
+from sklearn.metrics import mean_squared_error, r2_score, mean_absolute_error
+
+# Where to save the figures and data files
+PROJECT_ROOT_DIR = "Results"
+FIGURE_ID = "Results/FigureFiles"
+DATA_ID = "DataFiles/"
+
+if not os.path.exists(PROJECT_ROOT_DIR):
+    os.mkdir(PROJECT_ROOT_DIR)
+
+if not os.path.exists(FIGURE_ID):
+    os.makedirs(FIGURE_ID)
+
+if not os.path.exists(DATA_ID):
+    os.makedirs(DATA_ID)
+
+def image_path(fig_id):
+    return os.path.join(FIGURE_ID, fig_id)
+
+def data_path(dat_id):
+    return os.path.join(DATA_ID, dat_id)
+
+def save_fig(fig_id):
+    plt.savefig(image_path(fig_id) + ".png", format='png')
+
+infile = open(data_path("EoS.csv"),'r')
+
+# Read the EoS data as  csv file and organize the data into two arrays with density and energies
+EoS = pd.read_csv(infile, names=('Density', 'Energy'))
+EoS['Energy'] = pd.to_numeric(EoS['Energy'], errors='coerce')
+EoS = EoS.dropna()
+Energies = EoS['Energy']
+Density = EoS['Density']
+#  The design matrix now as function of various polytrops
+X = np.zeros((len(Density),4))
+X[:,3] = Density**(4.0/3.0)
+X[:,2] = Density
+X[:,1] = Density**(2.0/3.0)
+X[:,0] = 1
+
+# We use now Scikit-Learn's linear regressor and ridge regressor
+# OLS part
+clf = skl.LinearRegression().fit(X, Energies)
+ytilde = clf.predict(X)
+EoS['Eols']  = ytilde
+# The mean squared error                               
+print("Mean squared error: %.2f" % mean_squared_error(Energies, ytilde))
+# Explained variance score: 1 is perfect prediction                                 
+print('Variance score: %.2f' % r2_score(Energies, ytilde))
+# Mean absolute error                                                           
+print('Mean absolute error: %.2f' % mean_absolute_error(Energies, ytilde))
+print(clf.coef_, clf.intercept_)
+
+# The Ridge regression with a hyperparameter lambda = 0.1
+_lambda = 0.1
+clf_ridge = skl.Ridge(alpha=_lambda).fit(X, Energies)
+yridge = clf_ridge.predict(X)
+EoS['Eridge']  = yridge
+# The mean squared error                               
+print("Mean squared error: %.2f" % mean_squared_error(Energies, yridge))
+# Explained variance score: 1 is perfect prediction                                 
+print('Variance score: %.2f' % r2_score(Energies, yridge))
+# Mean absolute error                                                           
+print('Mean absolute error: %.2f' % mean_absolute_error(Energies, yridge))
+print(clf_ridge.coef_, clf_ridge.intercept_)
+
+fig, ax = plt.subplots()
+ax.set_xlabel(r'$\rho[\mathrm{fm}^{-3}]$')
+ax.set_ylabel(r'Energy per particle')
+ax.plot(EoS['Density'], EoS['Energy'], alpha=0.7, lw=2,
+            label='Theoretical data')
+ax.plot(EoS['Density'], EoS['Eols'], alpha=0.7, lw=2, c='m',
+            label='OLS')
+ax.plot(EoS['Density'], EoS['Eridge'], alpha=0.7, lw=2, c='g',
+            label='Ridge $\lambda = 0.1$')
+ax.legend()
+save_fig("EoSfitting")
+plt.show()
+
+

+The above simple polynomial in density \( \rho \) gives an excellent fit +to the data. Can you give an interpretation of the various powers of \( \rho \)? + +

+We note also that there is a small deviation between the +standard OLS and the Ridge regression at higher densities. We discuss this in more detail +below. + +

+









+ +

Splitting our Data in Training and Test data

+ +

+It is normal in essentially all Machine Learning studies to split the +data in a training set and a test set (sometimes also an additional +validation set). Scikit-Learn has an own function for this. There +is no explicit recipe for how much data should be included as training +data and say test data. An accepted rule of thumb is to use +approximately \( 2/3 \) to \( 4/5 \) of the data as training data. We will +postpone a discussion of this splitting to the end of these notes and +our discussion of the so-called bias-variance tradeoff. Here we +limit ourselves to repeat the above equation of state fitting example +but now splitting the data into a training set and a test set. + +

+ + +

import os
+import numpy as np
+import pandas as pd
+import matplotlib.pyplot as plt
+from sklearn.model_selection import train_test_split
+# Where to save the figures and data files
+PROJECT_ROOT_DIR = "Results"
+FIGURE_ID = "Results/FigureFiles"
+DATA_ID = "DataFiles/"
+
+if not os.path.exists(PROJECT_ROOT_DIR):
+    os.mkdir(PROJECT_ROOT_DIR)
+
+if not os.path.exists(FIGURE_ID):
+    os.makedirs(FIGURE_ID)
+
+if not os.path.exists(DATA_ID):
+    os.makedirs(DATA_ID)
+
+def image_path(fig_id):
+    return os.path.join(FIGURE_ID, fig_id)
+
+def data_path(dat_id):
+    return os.path.join(DATA_ID, dat_id)
+
+def save_fig(fig_id):
+    plt.savefig(image_path(fig_id) + ".png", format='png')
+
+def R2(y_data, y_model):
+    return 1 - np.sum((y_data - y_model) ** 2) / np.sum((y_data - np.mean(y_model)) ** 2)
+def MSE(y_data,y_model):
+    n = np.size(y_model)
+    return np.sum((y_data-y_model)**2)/n
+
+infile = open(data_path("EoS.csv"),'r')
+
+# Read the EoS data as  csv file and organized into two arrays with density and energies
+EoS = pd.read_csv(infile, names=('Density', 'Energy'))
+EoS['Energy'] = pd.to_numeric(EoS['Energy'], errors='coerce')
+EoS = EoS.dropna()
+Energies = EoS['Energy']
+Density = EoS['Density']
+#  The design matrix now as function of various polytrops
+X = np.zeros((len(Density),5))
+X[:,0] = 1
+X[:,1] = Density**(2.0/3.0)
+X[:,2] = Density
+X[:,3] = Density**(4.0/3.0)
+X[:,4] = Density**(5.0/3.0)
+# We split the data in test and training data
+X_train, X_test, y_train, y_test = train_test_split(X, Energies, test_size=0.2)
+# matrix inversion to find beta
+beta = np.linalg.inv(X_train.T.dot(X_train)).dot(X_train.T).dot(y_train)
+# and then make the prediction
+ytilde = X_train @ beta
+print("Training R2")
+print(R2(y_train,ytilde))
+print("Training MSE")
+print(MSE(y_train,ytilde))
+ypredict = X_test @ beta
+print("Test R2")
+print(R2(y_test,ypredict))
+print("Test MSE")
+print(MSE(y_test,ypredict))
+
+

+









+ +

The singular value decomposition

+ +

+

+ +

+ +

+The examples we have looked at so far are cases where we normally can +invert the matrix \( \boldsymbol{X}^T\boldsymbol{X} \). Using a polynomial expansion as we +did both for the masses and the fitting of the equation of state, +leads to row vectors of the design matrix which are essentially +orthogonal due to the polynomial character of our model. This may +however not the be case in general and a standard matrix inversion +algorithm based on say LU decomposition may lead to singularities. We will see an example of this below when we try to fit +the coupling constant of the widely used Ising model. +There is however a way to partially circumvent this problem and also gain some insight about the ordinary least squares approach. + +

+This is given by the Singular Value Decomposition algorithm, perhaps +the most powerful linear algebra algorithm. Let us look at a +different example where we may have problems with the standard matrix +inversion algorithm. Thereafter we dive into the math of the SVD. + + +

+ + +

+









+ +

The Ising model

+ +

+The one-dimensional Ising model with nearest neighbor interaction, no +external field and a constant coupling constant \( J \) is given by + +$$ +\begin{align} + H = -J \sum_{k}^L s_k s_{k + 1}, +\label{_auto1} +\end{align} +$$ + +

+where \( s_i \in \{-1, 1\} \) and \( s_{N + 1} = s_1 \). The number of spins +in the system is determined by \( L \). For the one-dimensional system +there is no phase transition. + +

+We will look at a system of \( L = 40 \) spins with a coupling constant of +\( J = 1 \). To get enough training data we will generate 10000 states +with their respective energies. + +

+ + +

import numpy as np
+import matplotlib.pyplot as plt
+from mpl_toolkits.axes_grid1 import make_axes_locatable
+import seaborn as sns
+import scipy.linalg as scl
+from sklearn.model_selection import train_test_split
+import tqdm
+sns.set(color_codes=True)
+cmap_args=dict(vmin=-1., vmax=1., cmap='seismic')
+
+L = 40
+n = int(1e4)
+
+spins = np.random.choice([-1, 1], size=(n, L))
+J = 1.0
+
+energies = np.zeros(n)
+
+for i in range(n):
+    energies[i] = - J * np.dot(spins[i], np.roll(spins[i], 1))
+
+

+Here we use ordinary least squares +regression to predict the energy for the nearest neighbor +one-dimensional Ising model on a ring, i.e., the endpoints wrap +around. We will use linear regression to fit a value for +the coupling constant to achieve this. + +

+









+ +

Reformulating the problem to suit regression

+ +

+A more general form for the one-dimensional Ising model is + +$$ +\begin{align} + H = - \sum_j^L \sum_k^L s_j s_k J_{jk}. +\label{_auto2} +\end{align} +$$ + +

+Here we allow for interactions beyond the nearest neighbors and a state dependent +coupling constant. This latter expression can be formulated as +a matrix-product +$$ +\begin{align} + \boldsymbol{H} = \boldsymbol{X} J, +\label{_auto3} +\end{align} +$$ + +

+where \( X_{jk} = s_j s_k \) and \( J \) is a matrix which consists of the +elements \( -J_{jk} \). This form of writing the energy fits perfectly +with the form utilized in linear regression, that is + +$$ +\begin{align} + \boldsymbol{y} = \boldsymbol{X}\boldsymbol{\beta} + \boldsymbol{\epsilon}, +\label{_auto4} +\end{align} +$$ + +

+We split the data in training and test data as discussed in the previous example + +

+ + +

X = np.zeros((n, L ** 2))
+for i in range(n):
+    X[i] = np.outer(spins[i], spins[i]).ravel()
+y = energies
+X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)
+
+

+









+ +

Linear regression

+ +

+In the ordinary least squares method we choose the cost function + +$$ +\begin{align} + C(\boldsymbol{X}, \boldsymbol{\beta})= \frac{1}{n}\left\{(\boldsymbol{X}\boldsymbol{\beta} - \boldsymbol{y})^T(\boldsymbol{X}\boldsymbol{\beta} - \boldsymbol{y})\right\}. +\label{_auto5} +\end{align} +$$ + +

+We then find the extremal point of \( C \) by taking the derivative with respect to \( \boldsymbol{\beta} \) as discussed above. +This yields the expression for \( \boldsymbol{\beta} \) to be + +$$ + \boldsymbol{\beta} = \frac{\boldsymbol{X}^T \boldsymbol{y}}{\boldsymbol{X}^T \boldsymbol{X}}, +$$ + +

+which immediately imposes some requirements on \( \boldsymbol{X} \) as there must exist +an inverse of \( \boldsymbol{X}^T \boldsymbol{X} \). If the expression we are modeling contains an +intercept, i.e., a constant term, we must make sure that the +first column of \( \boldsymbol{X} \) consists of \( 1 \). We do this here + +

+ + +

X_train_own = np.concatenate(
+    (np.ones(len(X_train))[:, np.newaxis], X_train),
+    axis=1
+)
+X_test_own = np.concatenate(
+    (np.ones(len(X_test))[:, np.newaxis], X_test),
+    axis=1
+)
+
+

+ + +

def ols_inv(x: np.ndarray, y: np.ndarray) -> np.ndarray:
+    return scl.inv(x.T @ x) @ (x.T @ y)
+beta = ols_inv(X_train_own, y_train)
+
+

+









+ +

Singular Value decomposition

+ +

+Doing the inversion directly turns out to be a bad idea since the matrix +\( \boldsymbol{X}^T\boldsymbol{X} \) is singular. An alternative approach is to use the singular +value decomposition. Using the definition of the Moore-Penrose +pseudoinverse we can write the equation for \( \boldsymbol{\beta} \) as + +$$ + \boldsymbol{\beta} = \boldsymbol{X}^{+}\boldsymbol{y}, +$$ + +

+where the pseudoinverse of \( \boldsymbol{X} \) is given by + +$$ + \boldsymbol{X}^{+} = \frac{\boldsymbol{X}^T}{\boldsymbol{X}^T\boldsymbol{X}}. +$$ + +

+Using singular value decomposition we can decompose the matrix \( \boldsymbol{X} = \boldsymbol{U}\boldsymbol{\Sigma} \boldsymbol{V}^T \), +where \( \boldsymbol{U} \) and \( \boldsymbol{V} \) are orthogonal(unitary) matrices and \( \boldsymbol{\Sigma} \) contains the singular values (more details below). +where \( X^{+} = V\Sigma^{+} U^T \). This reduces the equation for +\( \omega \) to +$$ +\begin{align} + \boldsymbol{\beta} = \boldsymbol{V}\boldsymbol{\Sigma}^{+} \boldsymbol{U}^T \boldsymbol{y}. +\label{_auto6} +\end{align} +$$ + +

+Note that solving this equation by actually doing the pseudoinverse +(which is what we will do) is not a good idea as this operation scales +as \( \mathcal{O}(n^3) \), where \( n \) is the number of elements in a +general matrix. Instead, doing \( QR \)-factorization and solving the +linear system as an equation would reduce this down to +\( \mathcal{O}(n^2) \) operations. + +

+ + +

def ols_svd(x: np.ndarray, y: np.ndarray) -> np.ndarray:
+    u, s, v = scl.svd(x)
+    return v.T @ scl.pinv(scl.diagsvd(s, u.shape[0], v.shape[0])) @ u.T @ y
+
+

+ + +

beta = ols_svd(X_train_own,y_train)
+
+

+When extracting the \( J \)-matrix we need to make sure that we remove the intercept, as is done here + +

+ + +

J = beta[1:].reshape(L, L)
+
+

+A way of looking at the coefficients in \( J \) is to plot the matrices as images. + +

+ + +

fig = plt.figure(figsize=(20, 14))
+im = plt.imshow(J, **cmap_args)
+plt.title("OLS", fontsize=18)
+plt.xticks(fontsize=18)
+plt.yticks(fontsize=18)
+cb = fig.colorbar(im)
+cb.ax.set_yticklabels(cb.ax.get_yticklabels(), fontsize=18)
+plt.show()
+
+

+It is interesting to note that OLS +considers both \( J_{j, j + 1} = -0.5 \) and \( J_{j, j - 1} = -0.5 \) as +valid matrix elements for \( J \). +In our discussion below on hyperparameters and Ridge and Lasso regression we will see that +this problem can be removed, partly and only with Lasso regression. + +

+In this case our matrix inversion was actually possible. The obvious question now is what is the mathematics behind the SVD? + +

+









+ +

Linear Regression Problems

+ +

+One of the typical problems we encounter with linear regression, in particular +when the matrix \( \boldsymbol{X} \) (our so-called design matrix) is high-dimensional, +are problems with near singular or singular matrices. The column vectors of \( \boldsymbol{X} \) +may be linearly dependent, normally referred to as super-collinearity. +This means that the matrix may be rank deficient and it is basically impossible to +to model the data using linear regression. As an example, consider the matrix +$$ +\begin{align*} +\mathbf{X} & = \left[ +\begin{array}{rrr} +1 & -1 & 2 +\\ +1 & 0 & 1 +\\ +1 & 2 & -1 +\\ +1 & 1 & 0 +\end{array} \right] +\end{align*} +$$ + +

+The columns of \( \boldsymbol{X} \) are linearly dependent. We see this easily since the +the first column is the row-wise sum of the other two columns. The rank (more correct, +the column rank) of a matrix is the dimension of the space spanned by the +column vectors. Hence, the rank of \( \mathbf{X} \) is equal to the number +of linearly independent columns. In this particular case the matrix has rank 2. + +

+Super-collinearity of an \( (n \times p) \)-dimensional design matrix \( \mathbf{X} \) implies +that the inverse of the matrix \( \boldsymbol{X}^T\boldsymbol{x} \) (the matrix we need to invert to solve the linear regression equations) is non-invertible. If we have a square matrix that does not have an inverse, we say this matrix singular. The example here demonstrates this +$$ +\begin{align*} +\boldsymbol{X} & = \left[ +\begin{array}{rr} +1 & -1 +\\ +1 & -1 +\end{array} \right]. +\end{align*} +$$ + +We see easily that \( \mbox{det}(\boldsymbol{X}) = x_{11} x_{22} - x_{12} x_{21} = 1 \times (-1) - 1 \times (-1) = 0 \). Hence, \( \mathbf{X} \) is singular and its inverse is undefined. +This is equivalent to saying that the matrix \( \boldsymbol{X} \) has at least an eigenvalue which is zero. + +

+









+ +

Fixing the singularity

+ +

+If our design matrix \( \boldsymbol{X} \) which enters the linear regression problem +$$ +\begin{align} +\boldsymbol{\beta} & = (\boldsymbol{X}^{T} \boldsymbol{X})^{-1} \boldsymbol{X}^{T} \boldsymbol{y}, +\label{_auto7} +\end{align} +$$ + +has linearly dependent column vectors, we will not be able to compute the inverse +of \( \boldsymbol{X}^T\boldsymbol{X} \) and we cannot find the parameters (estimators) \( \beta_i \). +The estimators are only well-defined if \( (\boldsymbol{X}^{T}\boldsymbol{X})^{-1} \) exits. +This is more likely to happen when the matrix \( \boldsymbol{X} \) is high-dimensional. In this case it is likely to encounter a situation where +the regression parameters \( \beta_i \) cannot be estimated. + +

+A cheap ad hoc approach is simply to add a small diagonal component to the matrix to invert, that is we change +$$ +\boldsymbol{X}^{T} \boldsymbol{X} \rightarrow \boldsymbol{X}^{T} \boldsymbol{X}+\lambda \boldsymbol{I}, +$$ + +where \( \boldsymbol{I} \) is the identity matrix. When we discuss Ridge regression this is actually what we end up evaluating. The parameter \( \lambda \) is called a hyperparameter. More about this later. + +

+









+ +

Basic math of the SVD

+ +

+From standard linear algebra we know that a square matrix \( \boldsymbol{X} \) can be diagonalized if and only it is +a so-called normal matrix, that is if \( \boldsymbol{X}\in {\mathbb{R}}^{n\times n} \) +we have \( \boldsymbol{X}\boldsymbol{X}^T=\boldsymbol{X}^T\boldsymbol{X} \) or if \( \boldsymbol{X}\in {\mathbb{C}}^{n\times n} \) we have \( \boldsymbol{X}\boldsymbol{X}^{\dagger}=\boldsymbol{X}^{\dagger}\boldsymbol{X} \). +The matrix has then a set of eigenpairs + +$$ +(\lambda_1,\boldsymbol{u}_1),\dots, (\lambda_n,\boldsymbol{u}_n), +$$ + +and the eigenvalues are given by the diagonal matrix +$$ +\boldsymbol{\Sigma}=\mathrm{Diag}(\lambda_1, \dots,\lambda_n). +$$ + +The matrix \( \boldsymbol{X} \) can be written in terms of an orthogonal/unitary transformation \( \boldsymbol{U} \) +$$ +\boldsymbol{X} = \boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{V}^T, +$$ + +with \( \boldsymbol{U}\boldsymbol{U}^T=\boldsymbol{I} \) or \( \boldsymbol{U}\boldsymbol{U}^{\dagger}=\boldsymbol{I} \). + +

+Not all square matrices are diagonalizable. A matrix like the one discussed above +$$ +\boldsymbol{X} = \begin{bmatrix} +1& -1 \\ +1& -1\\ +\end{bmatrix} +$$ + +is not diagonalizable, it is a so-called defective matrix. It is easy to see that the condition +\( \boldsymbol{X}\boldsymbol{X}^T=\boldsymbol{X}^T\boldsymbol{X} \) is not fulfilled. + +

+









+ +

The SVD, a Fantastic Algorithm

+ +

+However, and this is the strength of the SVD algorithm, any general +matrix \( \boldsymbol{X} \) can be decomposed in terms of a diagonal matrix and +two orthogonal/unitary matrices. The Singular Value Decompostion +(SVD) theorem +states that a general \( m\times n \) matrix \( \boldsymbol{X} \) can be written in +terms of a diagonal matrix \( \boldsymbol{\Sigma} \) of dimensionality \( n\times n \) +and two orthognal matrices \( \boldsymbol{U} \) and \( \boldsymbol{V} \), where the first has +dimensionality \( m \times m \) and the last dimensionality \( n\times n \). +We have then + +$$ +\boldsymbol{X} = \boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{V}^T +$$ + +

+As an example, the above defective matrix can be decomposed as + +$$ +\boldsymbol{X} = \frac{1}{\sqrt{2}}\begin{bmatrix} 1& 1 \\ 1& -1\\ \end{bmatrix} \begin{bmatrix} 2& 0 \\ 0& 0\\ \end{bmatrix} \frac{1}{\sqrt{2}}\begin{bmatrix} 1& -1 \\ 1& 1\\ \end{bmatrix}=\boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{V}^T, +$$ + +

+with eigenvalues \( \sigma_1=2 \) and \( \sigma_2=0 \). +The SVD exits always! + +

+









+ +

Another Example

+ +

+Consider the following matrix which can be SVD decomposed as + +$$ +\boldsymbol{X} = \frac{1}{15}\begin{bmatrix} 14 & 2\\ 4 & 22\\ 16 & 13\end{matrix}=\frac{1}{3}\begin{bmatrix} 1& 2 & 2 \\ 2& -1 & 1\\ 2 & 1& -2\end{bmatrix} \begin{bmatrix} 2& 0 \\ 0& 1\\ 0 & 0\end{bmatrix}\frac{1}{5}\begin{bmatrix} 3& 4 \\ 4& -3\end{bmatrix}=\boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{V}^T. +$$ + +

+This is a \( 3\times 2 \) matrix which is decomposed in terms of a +\( 3\times 3 \) matrix \( \boldsymbol{U} \), and a \( 2\times 2 \) matrix \( \boldsymbol{V} \). It is easy to see +that \( \boldsymbol{U} \) and \( \boldsymbol{V} \) are orthogonal (how?). + +

+And the SVD +decomposition (singular values) gives eigenvalues +\( \sigma_i\geq\sigma_{i+1} \) for all \( i \) and for dimensions larger than \( i=2 \), the +eigenvalues (singular values) are zero. + +

+In the general case, where our design matrix \( \boldsymbol{X} \) has dimension +\( n\times p \), the matrix is thus decomposed into an \( n\times n \) +orthogonal matrix \( \boldsymbol{U} \), a \( p\times p \) orthogonal matrix \( \boldsymbol{V} \) +and a diagonal matrix \( \boldsymbol{\Sigma} \) with \( r=\mathrm{min}(n,p) \) +singular values \( \sigma_i\lg 0 \) on the main diagonal and zeros filling +the rest of the matrix. There are at most \( p \) singular values +assuming that \( n > p \). In our regression examples for the nuclear +masses and the equation of state this is indeed the case, while for +the Ising model we have \( p > n \). These are often cases that lead to +near singular or singular matrices. + +

+The columns of \( \boldsymbol{U} \) are called the left singular vectors while the columns of \( \boldsymbol{V} \) are the right singular vectors. + +

+









+ +

Economy-size SVD

+ +

+If we assume that \( n > p \), then our matrix \( \boldsymbol{U} \) has dimension \( n +\times n \). The last \( n-p \) columns of \( \boldsymbol{U} \) become however +irrelevant in our calculations since they are multiplied with the +zeros in \( \boldsymbol{\Sigma} \). + +

+The economy-size decomposition removes extra rows or columns of zeros +from the diagonal matrix of singular values, \( \boldsymbol{\Sigma} \), along with the columns +in either \( \boldsymbol{U} \) or \( \boldsymbol{V} \) that multiply those zeros in the expression. +Removing these zeros and columns can improve execution time +and reduce storage requirements without compromising the accuracy of +the decomposition. + +

+If \( n > p \), we keep only the first \( p \) columns of \( \boldsymbol{U} \) and \( \boldsymbol{\Sigma} \) has dimension \( p\times p \). +If \( p > n \), then only the first \( n \) columns of \( \boldsymbol{V} \) are computed and \( \boldsymbol{\Sigma} \) has dimension \( n\times n \). +The \( n=p \) case is obvious, we retain the full SVD. +In general the economy-size SVD leads to less FLOPS and still conserving the desired accuracy. + +

+









+ +

Mathematical Properties

+ +

+There are several interesting mathematical properties which will be +relevant when we are going to discuss the differences between say +ordinary least squares (OLS) and Ridge regression. + +

+We have from OLS that the parameters of the linear approximation are given by +$$ +\boldsymbol{\tilde{y}} = \boldsymbol{X}\boldsymbol{\beta} = \boldsymbol{X}\left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}. +$$ + +

+The matrix to invert can be rewritten in terms of our SVD decomposition as + +$$ +\boldsymbol{X}^T\boldsymbol{X} = \boldsymbol{V}\boldsymbol{\Sigma}^T\boldsymbol{U}^T\boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{V}^T. +$$ + +Using the orthogonality properties of \( \boldsymbol{U} \) we have + +$$ +\boldsymbol{X}^T\boldsymbol{X} = \boldsymbol{V}\boldsymbol{\Sigma}^T\boldsymbol{\Sigma}\boldsymbol{V}^T = \boldsymbol{V}\boldsymbol{D}\boldsymbol{V}^T, +$$ + +with \( \boldsymbol{D} \) being a diagonal matrix with values along the diagonal given by the singular values squared. + +

+This means that +$$ +(\boldsymbol{X}^T\boldsymbol{X})\boldsymbol{V} = \boldsymbol{V}\boldsymbol{D}, +$$ + +that is the eigenvectors of \( (\boldsymbol{X}^T\boldsymbol{X}) \) are given by the columns of the right singular matrix of \( \boldsymbol{X} \) and the eigenvalues are the squared singular values. It is easy to show (show this) that +$$ +(\boldsymbol{X}\boldsymbol{X}^T)\boldsymbol{U} = \boldsymbol{U}\boldsymbol{D}, +$$ + +that is, the eigenvectors of \( (\boldsymbol{X}\boldsymbol{X})^T \) are the columns of the left singular matrix and the eigenvalues are the same. + +

+Going back to our OLS equation we have +$$ +\boldsymbol{X}\boldsymbol{\beta} = \boldsymbol{X}\left(\boldsymbol{V}\boldsymbol{D}\boldsymbol{V}^T \right)^{-1}\boldsymbol{X}^T\boldsymbol{y}=\boldsymbol{U\Sigma V^T}\left(\boldsymbol{V}\boldsymbol{D}\boldsymbol{V}^T \right)^{-1}(\boldsymbol{U\Sigma V^T})^T\boldsymbol{y}=\boldsymbol{U}\boldsymbol{U}^T\boldsymbol{y}. +$$ + +We will come back to this expression when we discuss Ridge regression. + +

+









+ +

Ridge and LASSO Regression

+ +

+Let us remind ourselves about the expression for the standard Mean Squared Error (MSE) which we used to define our cost function and the equations for the ordinary least squares (OLS) method, that is +our optimization problem is +$$ +{\displaystyle \min_{\boldsymbol{\beta}\in {\mathbb{R}}^{p}}}\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)^T\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)\right\}. +$$ + +or we can state it as +$$ +{\displaystyle \min_{\boldsymbol{\beta}\in +{\mathbb{R}}^{p}}}\frac{1}{n}\sum_{i=0}^{n-1}\left(y_i-\tilde{y}_i\right)^2=\frac{1}{n}\vert\vert \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\vert\vert_2^2, +$$ + +where we have used the definition of a norm-2 vector, that is +$$ +\vert\vert \boldsymbol{x}\vert\vert_2 = \sqrt{\sum_i x_i^2}. +$$ + +

+By minimizing the above equation with respect to the parameters +\( \boldsymbol{\beta} \) we could then obtain an analytical expression for the +parameters \( \boldsymbol{\beta} \). We can add a regularization parameter \( \lambda \) by +defining a new cost function to be optimized, that is + +$$ +{\displaystyle \min_{\boldsymbol{\beta}\in +{\mathbb{R}}^{p}}}\frac{1}{n}\vert\vert \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\vert\vert_2^2+\lambda\vert\vert \boldsymbol{\beta}\vert\vert_2^2 +$$ + +

+which leads to the Ridge regression minimization problem where we +require that \( \vert\vert \boldsymbol{\beta}\vert\vert_2^2\le t \), where \( t \) is +a finite number larger than zero. By defining + +$$ +C(\boldsymbol{X},\boldsymbol{\beta})=\frac{1}{n}\vert\vert \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\vert\vert_2^2+\lambda\vert\vert \boldsymbol{\beta}\vert\vert_1, +$$ + +

+we have a new optimization equation +$$ +{\displaystyle \min_{\boldsymbol{\beta}\in +{\mathbb{R}}^{p}}}\frac{1}{n}\vert\vert \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\vert\vert_2^2+\lambda\vert\vert \boldsymbol{\beta}\vert\vert_1 +$$ + +which leads to Lasso regression. Lasso stands for least absolute shrinkage and selection operator. + +

+Here we have defined the norm-1 as +$$ +\vert\vert \boldsymbol{x}\vert\vert_1 = \sum_i \vert x_i\vert. +$$ + +

+









+ +

More on Ridge Regression

+ +

+Using the matrix-vector expression for Ridge regression, + +$$ +C(\boldsymbol{X},\boldsymbol{\beta})=\frac{1}{n}\left\{(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})^T(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})\right\}+\lambda\boldsymbol{\beta}^T\boldsymbol{\beta}, +$$ + +

+by taking the derivatives with respect to \( \boldsymbol{\beta} \) we obtain then +a slightly modified matrix inversion problem which for finite values +of \( \lambda \) does not suffer from singularity problems. We obtain + +$$ +\boldsymbol{\beta}^{\mathrm{Ridge}} = \left(\boldsymbol{X}^T\boldsymbol{X}+\lambda\boldsymbol{I}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}, +$$ + +

+with \( \boldsymbol{I} \) being a \( p\times p \) identity matrix with the constraint that + +$$ +\sum_{i=0}^{p-1} \beta_i^2 \leq t, +$$ + +

+with \( t \) a finite positive number. + +

+We see that Ridge regression is nothing but the standard +OLS with a modified diagonal term added to \( \boldsymbol{X}^T\boldsymbol{X} \). The +consequences, in particular for our discussion of the bias-variance +are rather interesting. + +

+Furthermore, if we use the result above in terms of the SVD decomposition (our analysis was done for the OLS method), we had +$$ +(\boldsymbol{X}\boldsymbol{X}^T)\boldsymbol{U} = \boldsymbol{U}\boldsymbol{D}. +$$ + +

+We can analyse the OLS solutions in terms of the eigenvectors (the columns) of the right singular value matrix \( \boldsymbol{U} \) as +$$ +\boldsymbol{X}\boldsymbol{\beta} = \boldsymbol{X}\left(\boldsymbol{V}\boldsymbol{D}\boldsymbol{V}^T \right)^{-1}\boldsymbol{X}^T\boldsymbol{y}=\boldsymbol{U\Sigma V^T}\left(\boldsymbol{V}\boldsymbol{D}\boldsymbol{V}^T \right)^{-1}(\boldsymbol{U\Sigma V^T})^T\boldsymbol{y}=\boldsymbol{U}\boldsymbol{U}^T\boldsymbol{y} +$$ + +

+For Ridge regression this becomes + +$$ +\boldsymbol{X}\boldsymbol{\beta}^{\mathrm{Ridge}} = \boldsymbol{U\Sigma V^T}\left(\boldsymbol{V}\boldsymbol{D}\boldsymbol{V}^T+\lambda\boldsymbol{I} \right)^{-1}(\boldsymbol{U\Sigma V^T})^T\boldsymbol{y}=\sum_{j=0}^{p-1}\boldsymbol{u}_j\boldsymbol{u}_j^T\frac{\sigma_j^2}{\sigma_j^2+\lambda}\boldsymbol{y}, +$$ + +

+with the vectors \( \boldsymbol{u}_j \) being the columns of \( \boldsymbol{U} \). + +

+









+ +

Interpreting the Ridge results

+ +

+Since \( \lambda \geq 0 \), it means that compared to OLS, we have + +$$ +\frac{\sigma_j^2}{\sigma_j^2+\lambda} \leq 1. +$$ + +

+Ridge regression finds the coordinates of \( \boldsymbol{y} \) with respect to the +orthonormal basis \( \boldsymbol{U} \), it then shrinks the coordinates by +\( \frac{\sigma_j^2}{\sigma_j^2+\lambda} \). Recall that the SVD has +eigenvalues ordered in a descending way, that is \( \sigma_i \geq +\sigma_{i+1} \). + +

+For small eigenvalues \( \sigma_i \) it means that their contributions become less important, a fact which can be used to reduce the number of degrees of freedom. +Actually, calculating the variance of \( \boldsymbol{X}\boldsymbol{v}_j \) shows that this quantity is equal to \( \sigma_j^2/n \). +With a parameter \( \lambda \) we can thus shrink the role of specific parameters. + +

+









+ +

More interpretations

+ +

+For the sake of simplicity, let us assume that the design matrix is orthonormal, that is + +$$ +\boldsymbol{X}^T\boldsymbol{X}=(\boldsymbol{X}^T\boldsymbol{X})^{-1} =\boldsymbol{I}. +$$ + +

+In this case the standard OLS results in +$$ +\boldsymbol{\beta}^{\mathrm{OLS}} = \boldsymbol{X}^T\boldsymbol{y}=\sum_{i=0}^{p-1}\boldsymbol{u}_j\boldsymbol{u}_j^T\boldsymbol{y}, +$$ + +

+and + +$$ +\boldsymbol{\beta}^{\mathrm{Ridge}} = \left(\boldsymbol{I}+\lambda\boldsymbol{I}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}=\left(1+\lambda\right)^{-1}\boldsymbol{\beta}^{\mathrm{OLS}}, +$$ + +

+that is the Ridge estimator scales the OLS estimator by the inverse of a factor \( 1+\lambda \), and +the Ridge estimator converges to zero when the hyperparameter goes to +infinity. + +

+We will come back to more interpreations after we have gone through some of the statistical analysis part. + +

+For more discussions of Ridge and Lasso regression, Wessel van Wieringen's article is highly recommended. +Similarly, Mehta et al's article is also recommended. + +

+









+ +

Where are we going?

+ +

+Before we proceed, we need to rethink what we have been doing. In our +eager to fit the data, we have omitted several important elements in +our regression analysis. In what follows we will + +

    +
  1. look at statistical properties, including a discussion of mean values, variance and the so-called bias-variance tradeoff
    2. +
  2. introduce resampling techniques like cross-validation, bootstrapping and jackknife and more
    3. +
+ +This will allow us to link the standard linear algebra methods we have discussed above to a statistical interpretation of the methods. + +

+









+ +

Resampling methods

+
+ +

+Resampling methods are an indispensable tool in modern +statistics. They involve repeatedly drawing samples from a training +set and refitting a model of interest on each sample in order to +obtain additional information about the fitted model. For example, in +order to estimate the variability of a linear regression fit, we can +repeatedly draw different samples from the training data, fit a linear +regression to each new sample, and then examine the extent to which +the resulting fits differ. Such an approach may allow us to obtain +information that would not be available from fitting the model only +once using the original training sample. +

+ + +

+









+ +

Resampling approaches can be computationally expensive

+
+ +

+ +

+Resampling approaches can be computationally expensive, because they +involve fitting the same statistical method multiple times using +different subsets of the training data. However, due to recent +advances in computing power, the computational requirements of +resampling methods generally are not prohibitive. In this chapter, we +discuss two of the most commonly used resampling methods, +cross-validation and the bootstrap. Both methods are important tools +in the practical application of many statistical learning +procedures. For example, cross-validation can be used to estimate the +test error associated with a given statistical learning method in +order to evaluate its performance, or to select the appropriate level +of flexibility. The process of evaluating a model’s performance is +known as model assessment, whereas the process of selecting the proper +level of flexibility for a model is known as model selection. The +bootstrap is widely used. + + +

+ + +

+









+ +

Why resampling methods ?

+
+Statistical analysis. +

+ +

    +
  • Our simulations can be treated as computer experiments. This is particularly the case for Monte Carlo methods
    • +
  • The results can be analysed with the same statistical tools as we would use analysing experimental data.
    • +
  • As in all experiments, we are looking for expectation values and an estimate of how accurate they are, i.e., possible sources for errors.
    • +
+
+ + +

+









+ +

Statistical analysis

+
+ +

+ +

    +
  • As in other experiments, many numerical experiments have two classes of errors:
    • + +
      +
    • Statistical errors
      • +
    • Systematical errors
      • +
    + +
  • Statistical errors can be estimated using standard tools from statistics
    • +
  • Systematical errors are method specific and must be treated differently from case to case.
    • +
+
+ + +

+









+ +

Statistics

+
+ +

+The probability distribution function (PDF) is a function +\( p(x) \) on the domain which, in the discrete case, gives us the +probability or relative frequency with which these values of \( X \) occur: +$$ +p(x) = \mathrm{prob}(X=x) +$$ + +In the continuous case, the PDF does not directly depict the +actual probability. Instead we define the probability for the +stochastic variable to assume any value on an infinitesimal interval +around \( x \) to be \( p(x)dx \). The continuous function \( p(x) \) then gives us +the density of the probability rather than the probability +itself. The probability for a stochastic variable to assume any value +on a non-infinitesimal interval \( [a,\,b] \) is then just the integral: +$$ +\mathrm{prob}(a\leq X\leq b) = \int_a^b p(x)dx +$$ + +Qualitatively speaking, a stochastic variable represents the values of +numbers chosen as if by chance from some specified PDF so that the +selection of a large set of these numbers reproduces this PDF. +

+ + +

+









+ +

Statistics, moments

+
+ +

+A particularly useful class of special expectation values are the +moments. The \( n \)-th moment of the PDF \( p \) is defined as +follows: +$$ +\langle x^n\rangle \equiv \int\! x^n p(x)\,dx +$$ + +The zero-th moment \( \langle 1\rangle \) is just the normalization condition of +\( p \). The first moment, \( \langle x\rangle \), is called the mean of \( p \) +and often denoted by the letter \( \mu \): +$$ +\langle x\rangle = \mu \equiv \int\! x p(x)\,dx +$$ +

+ + +

+









+ +

Statistics, central moments

+
+ +

+A special version of the moments is the set of central moments, +the n-th central moment defined as: +$$ +\langle (x-\langle x \rangle )^n\rangle \equiv \int\! (x-\langle x\rangle)^n p(x)\,dx +$$ + +The zero-th and first central moments are both trivial, equal \( 1 \) and +\( 0 \), respectively. But the second central moment, known as the +variance of \( p \), is of particular interest. For the stochastic +variable \( X \), the variance is denoted as \( \sigma^2_X \) or \( \mathrm{var}(X) \): +$$ +\begin{align} +\sigma^2_X\ \ =\ \ \mathrm{var}(X) & = \langle (x-\langle x\rangle)^2\rangle = +\int\! (x-\langle x\rangle)^2 p(x)\,dx +\label{_auto8}\\ +& = \int\! \left(x^2 - 2 x \langle x\rangle^{2} + + \langle x\rangle^2\right)p(x)\,dx +\label{_auto9}\\ +& = \langle x^2\rangle - 2 \langle x\rangle\langle x\rangle + \langle x\rangle^2 +\label{_auto10}\\ +& = \langle x^2\rangle - \langle x\rangle^2 +\label{_auto11} +\end{align} +$$ + +The square root of the variance, \( \sigma =\sqrt{\langle (x-\langle x\rangle)^2\rangle} \) is called the standard deviation of \( p \). It is clearly just the RMS (root-mean-square) +value of the deviation of the PDF from its mean value, interpreted +qualitatively as the spread of \( p \) around its mean. +

+ + +

+









+ +

Statistics, covariance

+
+ +

+Another important quantity is the so called covariance, a variant of +the above defined variance. Consider again the set \( \{X_i\} \) of \( n \) +stochastic variables (not necessarily uncorrelated) with the +multivariate PDF \( P(x_1,\dots,x_n) \). The covariance of two +of the stochastic variables, \( X_i \) and \( X_j \), is defined as follows: +$$ +\begin{align} +\mathrm{cov}(X_i,\,X_j) &\equiv \langle (x_i-\langle x_i\rangle)(x_j-\langle x_j\rangle)\rangle +\nonumber\\ +&= +\int\!\cdots\!\int\!(x_i-\langle x_i \rangle)(x_j-\langle x_j \rangle)\, +P(x_1,\dots,x_n)\,dx_1\dots dx_n +\label{eq:def_covariance} +\end{align} +$$ + +with +$$ +\langle x_i\rangle = +\int\!\cdots\!\int\!x_i\,P(x_1,\dots,x_n)\,dx_1\dots dx_n +$$ +

+ + +

+









+ +

Statistics, more covariance

+
+ +

+If we consider the above covariance as a matrix \( C_{ij}=\mathrm{cov}(X_i,\,X_j) \), then the diagonal elements are just the familiar +variances, \( C_{ii} = \mathrm{cov}(X_i,\,X_i) = \mathrm{var}(X_i) \). It turns out that +all the off-diagonal elements are zero if the stochastic variables are +uncorrelated. This is easy to show, keeping in mind the linearity of +the expectation value. Consider the stochastic variables \( X_i \) and +\( X_j \), (\( i\neq j \)): +$$ +\begin{align} +\mathrm{cov}(X_i,\,X_j) &= \langle(x_i-\langle x_i\rangle)(x_j-\langle x_j\rangle)\rangle +\label{_auto12}\\ +&=\langle x_i x_j - x_i\langle x_j\rangle - \langle x_i\rangle x_j + \langle x_i\rangle\langle x_j\rangle\rangle +\label{_auto13}\\ +&=\langle x_i x_j\rangle - \langle x_i\langle x_j\rangle\rangle - \langle \langle x_i\rangle x_j\rangle + +\langle \langle x_i\rangle\langle x_j\rangle\rangle +\label{_auto14}\\ +&=\langle x_i x_j\rangle - \langle x_i\rangle\langle x_j\rangle - \langle x_i\rangle\langle x_j\rangle + +\langle x_i\rangle\langle x_j\rangle +\label{_auto15}\\ +&=\langle x_i x_j\rangle - \langle x_i\rangle\langle x_j\rangle +\label{_auto16} +\end{align} +$$ +

+ + +

+









+ +

Statistics, independent variables

+
+ +

+If \( X_i \) and \( X_j \) are independent, we get +\( \langle x_i x_j\rangle =\langle x_i\rangle\langle x_j\rangle \), resulting in \( \mathrm{cov}(X_i, X_j) = 0\ \ (i\neq j) \). + +

+Also useful for us is the covariance of linear combinations of +stochastic variables. Let \( \{X_i\} \) and \( \{Y_i\} \) be two sets of +stochastic variables. Let also \( \{a_i\} \) and \( \{b_i\} \) be two sets of +scalars. Consider the linear combination: +$$ +U = \sum_i a_i X_i \qquad V = \sum_j b_j Y_j +$$ + +By the linearity of the expectation value +$$ +\mathrm{cov}(U, V) = \sum_{i,j}a_i b_j \mathrm{cov}(X_i, Y_j) +$$ +

+ + +

+









+ +

Statistics, more variance

+
+ +

+Now, since the variance is just \( \mathrm{var}(X_i) = \mathrm{cov}(X_i, X_i) \), we get +the variance of the linear combination \( U = \sum_i a_i X_i \): +$$ +\begin{equation} +\mathrm{var}(U) = \sum_{i,j}a_i a_j \mathrm{cov}(X_i, X_j) +\label{eq:variance_linear_combination} +\end{equation} +$$ + +And in the special case when the stochastic variables are +uncorrelated, the off-diagonal elements of the covariance are as we +know zero, resulting in: +$$ +\mathrm{var}(U) = \sum_i a_i^2 \mathrm{cov}(X_i, X_i) = \sum_i a_i^2 \mathrm{var}(X_i) +$$ + +$$ +\mathrm{var}(\sum_i a_i X_i) = \sum_i a_i^2 \mathrm{var}(X_i) +$$ + +which will become very useful in our study of the error in the mean +value of a set of measurements. +

+ + +

+









+ +

Statistics and stochastic processes

+
+ +

+A stochastic process is a process that produces sequentially a +chain of values: +$$ +\{x_1, x_2,\dots\,x_k,\dots\}. +$$ + +We will call these +values our measurements and the entire set as our measured +sample. The action of measuring all the elements of a sample +we will call a stochastic experiment since, operationally, +they are often associated with results of empirical observation of +some physical or mathematical phenomena; precisely an experiment. We +assume that these values are distributed according to some +PDF \( p_X^{\phantom X}(x) \), where \( X \) is just the formal symbol for the +stochastic variable whose PDF is \( p_X^{\phantom X}(x) \). Instead of +trying to determine the full distribution \( p \) we are often only +interested in finding the few lowest moments, like the mean +\( \mu_X^{\phantom X} \) and the variance \( \sigma_X^{\phantom X} \). +

+ + +

+ + +

Statistics and sample variables

+
+ +

+In practical situations a sample is always of finite size. Let that +size be \( n \). The expectation value of a sample, the sample mean, is then defined as follows: +$$ +\bar{x}_n \equiv \frac{1}{n}\sum_{k=1}^n x_k +$$ + +The sample variance is: +$$ +\mathrm{var}(x) \equiv \frac{1}{n}\sum_{k=1}^n (x_k - \bar{x}_n)^2 +$$ + +its square root being the standard deviation of the sample. The +sample covariance is: +$$ +\mathrm{cov}(x)\equiv\frac{1}{n}\sum_{kl}(x_k - \bar{x}_n)(x_l - \bar{x}_n) +$$ +

+ + +

+









+ +

Statistics, sample variance and covariance

+
+ +

+Note that the sample variance is the sample covariance without the +cross terms. In a similar manner as the covariance in Eq. \eqref{eq:def_covariance} is a measure of the correlation between +two stochastic variables, the above defined sample covariance is a +measure of the sequential correlation between succeeding measurements +of a sample. + +

+These quantities, being known experimental values, differ +significantly from and must not be confused with the similarly named +quantities for stochastic variables, mean \( \mu_X \), variance \( \mathrm{var}(X) \) +and covariance \( \mathrm{cov}(X,Y) \). +

+ + +

+









+ +

Statistics, law of large numbers

+
+ +

+The law of large numbers +states that as the size of our sample grows to infinity, the sample +mean approaches the true mean \( \mu_X^{\phantom X} \) of the chosen PDF: +$$ +\lim_{n\to\infty}\bar{x}_n = \mu_X^{\phantom X} +$$ + +The sample mean \( \bar{x}_n \) works therefore as an estimate of the true +mean \( \mu_X^{\phantom X} \). + +

+What we need to find out is how good an approximation \( \bar{x}_n \) is to +\( \mu_X^{\phantom X} \). In any stochastic measurement, an estimated +mean is of no use to us without a measure of its error. A quantity +that tells us how well we can reproduce it in another experiment. We +are therefore interested in the PDF of the sample mean itself. Its +standard deviation will be a measure of the spread of sample means, +and we will simply call it the error of the sample mean, or +just sample error, and denote it by \( \mathrm{err}_X^{\phantom X} \). In +practice, we will only be able to produce an estimate of the +sample error since the exact value would require the knowledge of the +true PDFs behind, which we usually do not have. +

+ + +

+









+ +

Statistics, more on sample error

+
+ +

+Let us first take a look at what happens to the sample error as the +size of the sample grows. In a sample, each of the measurements \( x_i \) +can be associated with its own stochastic variable \( X_i \). The +stochastic variable \( \overline X_n \) for the sample mean \( \bar{x}_n \) is +then just a linear combination, already familiar to us: +$$ +\overline X_n = \frac{1}{n}\sum_{i=1}^n X_i +$$ + +All the coefficients are just equal \( 1/n \). The PDF of \( \overline X_n \), +denoted by \( p_{\overline X_n}(x) \) is the desired PDF of the sample +means. +

+ + +

+









+ +

Statistics

+
+ +

+The probability density of obtaining a sample mean \( \bar x_n \) +is the product of probabilities of obtaining arbitrary values \( x_1, +x_2,\dots,x_n \) with the constraint that the mean of the set \( \{x_i\} \) +is \( \bar x_n \): +$$ +p_{\overline X_n}(x) = \int p_X^{\phantom X}(x_1)\cdots +\int p_X^{\phantom X}(x_n)\ +\delta\!\left(x - \frac{x_1+x_2+\dots+x_n}{n}\right)dx_n \cdots dx_1 +$$ + +And in particular we are interested in its variance \( \mathrm{var}(\overline X_n) \). +

+ + +

+









+ +

Statistics, central limit theorem

+
+ +

+It is generally not possible to express \( p_{\overline X_n}(x) \) in a +closed form given an arbitrary PDF \( p_X^{\phantom X} \) and a number +\( n \). But for the limit \( n\to\infty \) it is possible to make an +approximation. The very important result is called the central limit theorem. It tells us that as \( n \) goes to infinity, +\( p_{\overline X_n}(x) \) approaches a Gaussian distribution whose mean +and variance equal the true mean and variance, \( \mu_{X}^{\phantom X} \) +and \( \sigma_{X}^{2} \), respectively: +$$ +\begin{equation} +\lim_{n\to\infty} p_{\overline X_n}(x) = +\left(\frac{n}{2\pi\mathrm{var}(X)}\right)^{1/2} +e^{-\frac{n(x-\bar x_n)^2}{2\mathrm{var}(X)}} +\label{eq:central_limit_gaussian} +\end{equation} +$$ +

+ + +

+









+ +

Statistics, more technicalities

+
+ +

+The desired variance +\( \mathrm{var}(\overline X_n) \), i.e. the sample error squared +\( \mathrm{err}_X^2 \), is given by: +$$ +\begin{equation} +\mathrm{err}_X^2 = \mathrm{var}(\overline X_n) = \frac{1}{n^2} +\sum_{ij} \mathrm{cov}(X_i, X_j) +\label{eq:error_exact} +\end{equation} +$$ + +We see now that in order to calculate the exact error of the sample +with the above expression, we would need the true means +\( \mu_{X_i}^{\phantom X} \) of the stochastic variables \( X_i \). To +calculate these requires that we know the true multivariate PDF of all +the \( X_i \). But this PDF is unknown to us, we have only got the measurements of +one sample. The best we can do is to let the sample itself be an +estimate of the PDF of each of the \( X_i \), estimating all properties of +\( X_i \) through the measurements of the sample. +

+ + +

+









+ +

Statistics

+
+ +

+Our estimate of \( \mu_{X_i}^{\phantom X} \) is then the sample mean \( \bar x \) +itself, in accordance with the the central limit theorem: +$$ +\mu_{X_i}^{\phantom X} = \langle x_i\rangle \approx \frac{1}{n}\sum_{k=1}^n x_k = \bar x +$$ + +Using \( \bar x \) in place of \( \mu_{X_i}^{\phantom X} \) we can give an +estimate of the covariance in Eq. \eqref{eq:error_exact} +$$ +\mathrm{cov}(X_i, X_j) = \langle (x_i-\langle x_i\rangle)(x_j-\langle x_j\rangle)\rangle +\approx\langle (x_i - \bar x)(x_j - \bar{x})\rangle, +$$ + +resulting in +$$ +\frac{1}{n} \sum_{l}^n \left(\frac{1}{n}\sum_{k}^n (x_k -\bar x_n)(x_l - \bar x_n)\right)=\frac{1}{n}\frac{1}{n} \sum_{kl} (x_k -\bar x_n)(x_l - \bar x_n)=\frac{1}{n}\mathrm{cov}(x) +$$ +

+ + +

+









+ +

Statistics and sample variance

+
+ +

+By the same procedure we can use the sample variance as an +estimate of the variance of any of the stochastic variables \( X_i \) +$$ +\mathrm{var}(X_i)=\langle x_i - \langle x_i\rangle\rangle \approx \langle x_i - \bar x_n\rangle\nonumber, +$$ + +which is approximated as +$$ +\begin{equation} +\mathrm{var}(X_i)\approx \frac{1}{n}\sum_{k=1}^n (x_k - \bar x_n)=\mathrm{var}(x) +\label{eq:var_estimate_i_think} +\end{equation} +$$ + +

+Now we can calculate an estimate of the error +\( \mathrm{err}_X^{\phantom X} \) of the sample mean \( \bar x_n \): +$$ +\begin{align} +\mathrm{err}_X^2 +&=\frac{1}{n^2}\sum_{ij} \mathrm{cov}(X_i, X_j) \nonumber \\ +&\approx&\frac{1}{n^2}\sum_{ij}\frac{1}{n}\mathrm{cov}(x) =\frac{1}{n^2}n^2\frac{1}{n}\mathrm{cov}(x)\nonumber\\ +&=\frac{1}{n}\mathrm{cov}(x) +\label{eq:error_estimate} +\end{align} +$$ + +which is nothing but the sample covariance divided by the number of +measurements in the sample. +

+ + +

+









+ +

Statistics, uncorrelated results

+
+ +

+ +

+In the special case that the measurements of the sample are +uncorrelated (equivalently the stochastic variables \( X_i \) are +uncorrelated) we have that the off-diagonal elements of the covariance +are zero. This gives the following estimate of the sample error: +$$ +\mathrm{err}_X^2=\frac{1}{n^2}\sum_{ij} \mathrm{cov}(X_i, X_j) = +\frac{1}{n^2} \sum_i \mathrm{var}(X_i), +$$ + +resulting in +$$ +\begin{equation} +\mathrm{err}_X^2\approx \frac{1}{n^2} \sum_i \mathrm{var}(x)= \frac{1}{n}\mathrm{var}(x) +\label{eq:error_estimate_uncorrel} +\end{equation} +$$ + +where in the second step we have used Eq. \eqref{eq:var_estimate_i_think}. +The error of the sample is then just its standard deviation divided by +the square root of the number of measurements the sample contains. +This is a very useful formula which is easy to compute. It acts as a +first approximation to the error, but in numerical experiments, we +cannot overlook the always present correlations. +

+ + +

+









+ +

Statistics, computations

+
+ +

+For computational purposes one usually splits up the estimate of +\( \mathrm{err}_X^2 \), given by Eq. \eqref{eq:error_estimate}, into two +parts +$$ +\mathrm{err}_X^2 = \frac{1}{n}\mathrm{var}(x) + \frac{1}{n}(\mathrm{cov}(x)-\mathrm{var}(x)), +$$ + +which equals +$$ +\begin{equation} +\frac{1}{n^2}\sum_{k=1}^n (x_k - \bar x_n)^2 +\frac{2}{n^2}\sum_{k < l} (x_k - \bar x_n)(x_l - \bar x_n) +\label{eq:error_estimate_split_up} +\end{equation} +$$ + +The first term is the same as the error in the uncorrelated case, +Eq. \eqref{eq:error_estimate_uncorrel}. This means that the second +term accounts for the error correction due to correlation between the +measurements. For uncorrelated measurements this second term is zero. +

+ + +

+









+ +

Statistics, more on computations of errors

+
+ +

+Computationally the uncorrelated first term is much easier to treat +efficiently than the second. +$$ +\mathrm{var}(x) = \frac{1}{n}\sum_{k=1}^n (x_k - \bar x_n)^2 = +\left(\frac{1}{n}\sum_{k=1}^n x_k^2\right) - \bar x_n^2 +$$ + +We just accumulate separately the values \( x^2 \) and \( x \) for every +measurement \( x \) we receive. The correlation term, though, has to be +calculated at the end of the experiment since we need all the +measurements to calculate the cross terms. Therefore, all measurements +have to be stored throughout the experiment. +

+ + +

+









+ +

Statistics, wrapping up 1

+
+ +

+Let us analyze the problem by splitting up the correlation term into +partial sums of the form: +$$ +f_d = \frac{1}{n-d}\sum_{k=1}^{n-d}(x_k - \bar x_n)(x_{k+d} - \bar x_n) +$$ + +The correlation term of the error can now be rewritten in terms of +\( f_d \) +$$ +\frac{2}{n}\sum_{k < l} (x_k - \bar x_n)(x_l - \bar x_n) = +2\sum_{d=1}^{n-1} f_d +$$ + +The value of \( f_d \) reflects the correlation between measurements +separated by the distance \( d \) in the sample samples. Notice that for +\( d=0 \), \( f \) is just the sample variance, \( \mathrm{var}(x) \). If we divide \( f_d \) +by \( \mathrm{var}(x) \), we arrive at the so called autocorrelation function +$$ +\kappa_d = \frac{f_d}{\mathrm{var}(x)} +$$ + +which gives us a useful measure of pairwise correlations +starting always at \( 1 \) for \( d=0 \). +

+ + +

+









+ +

Statistics, final expression

+
+ +

+The sample error (see eq. \eqref{eq:error_estimate_split_up}) can now be +written in terms of the autocorrelation function: +$$ +\begin{align} +\mathrm{err}_X^2 &= +\frac{1}{n}\mathrm{var}(x)+\frac{2}{n}\cdot\mathrm{var}(x)\sum_{d=1}^{n-1} +\frac{f_d}{\mathrm{var}(x)}\nonumber\\ &=& +\left(1+2\sum_{d=1}^{n-1}\kappa_d\right)\frac{1}{n}\mathrm{var}(x)\nonumber\\ +&=\frac{\tau}{n}\cdot\mathrm{var}(x) +\label{eq:error_estimate_corr_time} +\end{align} +$$ + +and we see that \( \mathrm{err}_X \) can be expressed in terms the +uncorrelated sample variance times a correction factor \( \tau \) which +accounts for the correlation between measurements. We call this +correction factor the autocorrelation time: +$$ +\begin{equation} +\tau = 1+2\sum_{d=1}^{n-1}\kappa_d +\label{eq:autocorrelation_time} +\end{equation} +$$ +

+ + +

+









+ +

Statistics, effective number of correlations

+
+ +

+For a correlation free experiment, \( \tau \) +equals 1. From the point of view of +eq. \eqref{eq:error_estimate_corr_time} we can interpret a sequential +correlation as an effective reduction of the number of measurements by +a factor \( \tau \). The effective number of measurements becomes: +$$ +n_\mathrm{eff} = \frac{n}{\tau} +$$ + +To neglect the autocorrelation time \( \tau \) will always cause our +simple uncorrelated estimate of \( \mathrm{err}_X^2\approx \mathrm{var}(x)/n \) to +be less than the true sample error. The estimate of the error will be +too good. On the other hand, the calculation of the full +autocorrelation time poses an efficiency problem if the set of +measurements is very large. +

+ + +

+ + +

Linking the regression analysis with a statistical interpretation

+ +

+Finally, we are going to discuss several statistical properties which can be obtained in terms of analytical expressions. +The +advantage of doing linear regression is that we actually end up with +analytical expressions for several statistical quantities. +Standard least squares and Ridge regression allow us to +derive quantities like the variance and other expectation values in a +rather straightforward way. + +

+It is assumed that \( \varepsilon_i +\sim \mathcal{N}(0, \sigma^2) \) and the \( \varepsilon_{i} \) are +independent, i.e.: +$$ +\begin{align*} +\mbox{Cov}(\varepsilon_{i_1}, +\varepsilon_{i_2}) & = \left\{ \begin{array}{lcc} \sigma^2 & \mbox{if} +& i_1 = i_2, \\ 0 & \mbox{if} & i_1 \not= i_2. \end{array} \right. +\end{align*} +$$ + +The randomness of \( \varepsilon_i \) implies that +\( \mathbf{y}_i \) is also a random variable. In particular, +\( \mathbf{y}_i \) is normally distributed, because \( \varepsilon_i \sim +\mathcal{N}(0, \sigma^2) \) and \( \mathbf{X}_{i,\ast} \, \boldsymbol{\beta} \) is a +non-random scalar. To specify the parameters of the distribution of +\( \mathbf{y}_i \) we need to calculate its first two moments. + +

+Recall that \( \boldsymbol{X} \) is a matrix of dimensionality \( n\times p \). The +notation above \( \mathbf{X}_{i,\ast} \) means that we are looking at the +row number \( i \) and perform a sum over all values \( p \). + +

+









+ +

Assumptions made

+ +

+The assumption we have made here can be summarized as (and this is going to useful when we discuss the bias-variance trade off) +that there exists a function \( f(\boldsymbol{x}) \) and a normal distributed error \( \boldsymbol{\varepsilon}\sim \mathcal{N}(0, \sigma^2) \) +which describes our data +$$ +\boldsymbol{y} = f(\boldsymbol{x})+\boldsymbol{\varepsilon} +$$ + +

+We approximate this function with our model from the solution of the linear regression equations, that is our +function \( f \) is approximated by \( \boldsymbol{\tilde{y}} \) where we want to minimize \( (\boldsymbol{y}-\boldsymbol{\tilde{y}})^2 \), our MSE, with +$$ +\boldsymbol{\tilde{y}} = \boldsymbol{X}\boldsymbol{\beta}. +$$ + +

+









+ +

Expectation value and variance

+ +

+We can calculate the expectation value of \( \boldsymbol{y} \) for a given element \( i \) +$$ +\begin{align*} +\mathbb{E}(y_i) & = +\mathbb{E}(\mathbf{X}_{i, \ast} \, \boldsymbol{\beta}) + \mathbb{E}(\varepsilon_i) +\, \, \, = \, \, \, \mathbf{X}_{i, \ast} \, \beta, +\end{align*} +$$ + +while +its variance is +$$ +\begin{align*} \mbox{Var}(y_i) & = \mathbb{E} \{ [y_i +- \mathbb{E}(y_i)]^2 \} \, \, \, = \, \, \, \mathbb{E} ( y_i^2 ) - +[\mathbb{E}(y_i)]^2 \\ & = \mathbb{E} [ ( \mathbf{X}_{i, \ast} \, +\beta + \varepsilon_i )^2] - ( \mathbf{X}_{i, \ast} \, \boldsymbol{\beta})^2 \\ & += \mathbb{E} [ ( \mathbf{X}_{i, \ast} \, \boldsymbol{\beta})^2 + 2 \varepsilon_i +\mathbf{X}_{i, \ast} \, \boldsymbol{\beta} + \varepsilon_i^2 ] - ( \mathbf{X}_{i, +\ast} \, \beta)^2 \\ & = ( \mathbf{X}_{i, \ast} \, \boldsymbol{\beta})^2 + 2 +\mathbb{E}(\varepsilon_i) \mathbf{X}_{i, \ast} \, \boldsymbol{\beta} + +\mathbb{E}(\varepsilon_i^2 ) - ( \mathbf{X}_{i, \ast} \, \boldsymbol{\beta})^2 +\\ & = \mathbb{E}(\varepsilon_i^2 ) \, \, \, = \, \, \, +\mbox{Var}(\varepsilon_i) \, \, \, = \, \, \, \sigma^2. +\end{align*} +$$ + +Hence, \( y_i \sim \mathcal{N}( \mathbf{X}_{i, \ast} \, \boldsymbol{\beta}, \sigma^2) \), that is \( \boldsymbol{y} \) follows a normal distribution with +mean value \( \boldsymbol{X}\boldsymbol{\beta} \) and variance \( \sigma^2 \) (not be confused with the singular values of the SVD). + +

+









+ +

Expectation value and variance for \( \boldsymbol{\beta} \)

+ +

+With the OLS expressions for the parameters \( \boldsymbol{\beta} \) we can evaluate the expectation value +$$ +\mathbb{E}(\boldsymbol{\beta}) = \mathbb{E}[ (\mathbf{X}^{\top} \mathbf{X})^{-1}\mathbf{X}^{T} \mathbf{Y}]=(\mathbf{X}^{T} \mathbf{X})^{-1}\mathbf{X}^{T} \mathbb{E}[ \mathbf{Y}]=(\mathbf{X}^{T} \mathbf{X})^{-1} \mathbf{X}^{T}\mathbf{X}\boldsymbol{\beta}=\boldsymbol{\beta}. +$$ + +This means that the estimator of the regression parameters is unbiased. + +

+We can also calculate the variance + +

+The variance of \( \boldsymbol{\beta} \) is +$$ +\begin{eqnarray*} +\mbox{Var}(\boldsymbol{\beta}) & = & \mathbb{E} \{ [\boldsymbol{\beta} - \mathbb{E}(\boldsymbol{\beta})] [\boldsymbol{\beta} - \mathbb{E}(\boldsymbol{\beta})]^{T} \} +\\ +& = & \mathbb{E} \{ [(\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{Y} - \boldsymbol{\beta}] \, [(\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{Y} - \boldsymbol{\beta}]^{T} \} +\\ +% & = & \mathbb{E} \{ [(\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{Y}] \, [(\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{Y}]^{T} \} - \boldsymbol{\beta} \, \boldsymbol{\beta}^{T} +% \\ +% & = & \mathbb{E} \{ (\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{Y} \, \mathbf{Y}^{T} \, \mathbf{X} \, (\mathbf{X}^{T} \mathbf{X})^{-1} \} - \boldsymbol{\beta} \, \boldsymbol{\beta}^{T} +% \\ +& = & (\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \, \mathbb{E} \{ \mathbf{Y} \, \mathbf{Y}^{T} \} \, \mathbf{X} \, (\mathbf{X}^{T} \mathbf{X})^{-1} - \boldsymbol{\beta} \, \boldsymbol{\beta}^{T} +\\ +& = & (\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \, \{ \mathbf{X} \, \boldsymbol{\beta} \, \boldsymbol{\beta}^{T} \, \mathbf{X}^{T} + \sigma^2 \} \, \mathbf{X} \, (\mathbf{X}^{T} \mathbf{X})^{-1} - \boldsymbol{\beta} \, \boldsymbol{\beta}^{T} +% \\ +% & = & (\mathbf{X}^T \mathbf{X})^{-1} \, \mathbf{X}^T \, \mathbf{X} \, \boldsymbol{\beta} \, \boldsymbol{\beta}^T \, \mathbf{X}^T \, \mathbf{X} \, (\mathbf{X}^T % \mathbf{X})^{-1} +% \\ +% & & + \, \, \sigma^2 \, (\mathbf{X}^T \mathbf{X})^{-1} \, \mathbf{X}^T \, \mathbf{X} \, (\mathbf{X}^T \mathbf{X})^{-1} - \boldsymbol{\beta} \boldsymbol{\beta}^T +\\ +& = & \boldsymbol{\beta} \, \boldsymbol{\beta}^{T} + \sigma^2 \, (\mathbf{X}^{T} \mathbf{X})^{-1} - \boldsymbol{\beta} \, \boldsymbol{\beta}^{T} +\, \, \, = \, \, \, \sigma^2 \, (\mathbf{X}^{T} \mathbf{X})^{-1}, +\end{eqnarray*} +$$ + +

+where we have used that \( \mathbb{E} (\mathbf{Y} \mathbf{Y}^{T}) = +\mathbf{X} \, \boldsymbol{\beta} \, \boldsymbol{\beta}^{T} \, \mathbf{X}^{T} + +\sigma^2 \, \mathbf{I}_{nn} \). From \( \mbox{Var}(\boldsymbol{\beta}) = \sigma^2 +\, (\mathbf{X}^{T} \mathbf{X})^{-1} \), one obtains an estimate of the +variance of the estimate of the \( j \)-th regression coefficient: +\( \hat{\sigma}^2 (\hat{\beta}_j ) = \hat{\sigma}^2 \sqrt{ +[(\mathbf{X}^{T} \mathbf{X})^{-1}]_{jj} } \). This may be used to +construct a confidence interval for the estimates. + +

+In a similar way, we cna obtain analytical expressions for say the +expectation values of the parameters \( \boldsymbol{\beta} \) and their variance +when we employ Ridge regression, and thereby a confidence interval. + +

+It is rather straightforward to show that +$$ +\mathbb{E} \big[ \boldsymbol{\beta}^{\mathrm{Ridge}} \big]=(\mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I}_{pp})^{-1} (\mathbf{X}^{\top} \mathbf{X})\boldsymbol{\beta}^{\mathrm{OLS}}. +$$ + +We see clearly that +\( \mathbb{E} \big[ \boldsymbol{\beta}^{\mathrm{Ridge}} \big] \not= \boldsymbol{\beta}^{\mathrm{OLS}} \) for any \( \lambda > 0 \). We say then that the ridge estimator is biased. + +

+We can also compute the variance as + +$$ +\mbox{Var}[\boldsymbol{\beta}^{\mathrm{Ridge}}]=\sigma^2[ \mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I} ]^{-1} \mathbf{X}^{T} \mathbf{X} \{ [ \mathbf{X}^{\top} \mathbf{X} + \lambda \mathbf{I} ]^{-1}\}^{T}, +$$ + +and it is easy to see that if the parameter \( \lambda \) goes to infinity then the variance of Ridge parameters \( \boldsymbol{\beta} \) goes to zero. + +

+With this, we can compute the difference + +$$ +\mbox{Var}[\boldsymbol{\beta}^{\mathrm{OLS}}]-\mbox{Var}(\boldsymbol{\beta}^{\mathrm{Ridge}})=\sigma^2 [ \mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I} ]^{-1}[ 2\lambda\mathbf{I} + \lambda^2 (\mathbf{X}^{T} \mathbf{X})^{-1} ] \{ [ \mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I} ]^{-1}\}^{T}. +$$ + +The difference is non-negative definite since each component of the +matrix product is non-negative definite. +This means the variance we obtain with the standard OLS will always for \( \lambda > 0 \) be larger than the variance of \( \boldsymbol{\beta} \) obtained with the Ridge estimator. This has interesting consequences when we discuss the so-called bias-variance trade-off below. + +

+ + +

Cross-validation

+ +

+Instead of choosing the penalty parameter to balance model fit with +model complexity, cross-validation requires it (i.e. the penalty +parameter) to yield a model with good prediction +performance. Commonly, this performance is evaluated on novel +data. Novel data need not be easy to come by and one has to make do +with the data at hand. + +

+The setting of original and novel data is +then mimicked by sample splitting: the data set is divided into two +(groups of samples). One of these two data sets, called the +training set, plays the role of original data on which the model is +built. The second of these data sets, called the test set, plays the +role of the novel data and is used to evaluate the prediction +performance (often operationalized as the log-likelihood or the +prediction error or its square or the R2 score) of the model built on the training data set. This +procedure (model building and prediction evaluation on training and +test set, respectively) is done for a collection of possible penalty +parameter choices. The penalty parameter that yields the model with +the best prediction performance is to be preferred. The thus obtained +performance evaluation depends on the actual split of the data set. To +remove this dependence the data set is split many times into a +training and test set. For each split the model parameters are +estimated for all choices of \( \lambda \) using the training data and +estimated parameters are evaluated on the corresponding test set. The +penalty parameter that on average over the test sets performs best (in +some sense) is then selected. + +

+









+ +

Computationally expensive

+ +

+The validation set approach is conceptually simple and is easy to implement. But it has two potential drawbacks: + +

    +
  • The validation estimate of the test error rate can be highly variable, depending on precisely which observations are included in the training set and which observations are included in the validation set.
    • +
  • In the validation approach, only a subset of the observations, those that are included in the training set rather than in the validation set are used to fit the model. Since statistical methods tend to perform worse when trained on fewer observations, this suggests that the validation set error rate may tend to overestimate the test error rate for the model fit on the entire data set.
    • +
+ + + +

Various steps in cross-validation

+ +

+When the repetitive splitting of the data set is done randomly, +samples may accidently end up in a fast majority of the splits in +either training or test set. Such samples may have an unbalanced +influence on either model building or prediction evaluation. To avoid +this \( k \)-fold cross-validation structures the data splitting. The +samples are divided into \( k \) more or less equally sized exhaustive and +mutually exclusive subsets. In turn (at each split) one of these +subsets plays the role of the test set while the union of the +remaining subsets constitutes the training set. Such a splitting +warrants a balanced representation of each sample in both training and +test set over the splits. Still the division into the \( k \) subsets +involves a degree of randomness. This may be fully excluded when +choosing \( k=n \). This particular case is referred to as leave-one-out +cross-validation (LOOCV). + +

+ + +

How to set up the cross-validation for Ridge and/or Lasso

+ +
    +
  • Define a range of interest for the penalty parameter.
    • +
  • Divide the data set into training and test set comprising samples \( \{1, \ldots, n\} \setminus i \) and \( \{ i \} \), respectively.
    • +
  • Fit the linear regression model by means of ridge estimation for each \( \lambda \) in the grid using the training set, and the corresponding estimate of the error variance \( \boldsymbol{\sigma}_{-i}^2(\lambda) \), as
    • +
+ +$$ +\begin{align*} +\boldsymbol{\beta}_{-i}(\lambda) & = ( \boldsymbol{X}_{-i, \ast}^{T} +\boldsymbol{X}_{-i, \ast} + \lambda \boldsymbol{I}_{pp})^{-1} +\boldsymbol{X}_{-i, \ast}^{T} \boldsymbol{y}_{-i} +\end{align*} +$$ + + +
    +
  • Evaluate the prediction performance of these models on the test set by \( \log\{L[y_i, \boldsymbol{X}_{i, \ast}; \boldsymbol{\beta}_{-i}(\lambda), \boldsymbol{\sigma}_{-i}^2(\lambda)]\} \). Or, by the prediction error \( |y_i - \boldsymbol{X}_{i, \ast} \boldsymbol{\beta}_{-i}(\lambda)| \), the relative error, the error squared or the R2 score function.
    • +
  • Repeat the first three steps such that each sample plays the role of the test set once.
    • +
  • Average the prediction performances of the test sets at each grid point of the penalty bias/parameter by computing the cross-validated log-likelihood. It is an estimate of the prediction performance of the model corresponding to this value of the penalty parameter on novel data. It is defined as
    • +
+ +$$ +\begin{align*} +\frac{1}{n} \sum_{i = 1}^n \log\{L[y_i, \mathbf{X}_{i, \ast}; \boldsymbol{\beta}_{-i}(\lambda), \boldsymbol{\sigma}_{-i}^2(\lambda)]\}. +\end{align*} +$$ + + +
    +
  • The value of the penalty parameter that maximizes the cross-validated log-likelihood is the value of choice. Or we can use the MSE or the R2 score functions.
    • +
+ +









+ +

Resampling methods: Jackknife and Bootstrap

+ +

+Two famous +resampling methods are the independent bootstrap and the jackknife. + +

+The jackknife is a special case of the independent bootstrap. Still, the jackknife was made +popular prior to the independent bootstrap. And as the popularity of +the independent bootstrap soared, new variants, such as the dependent bootstrap. + +

+The Jackknife and independent bootstrap work for +independent, identically distributed random variables. +If these conditions are not +satisfied, the methods will fail. Yet, it should be said that if the data are +independent, identically distributed, and we only want to estimate the +variance of \( \overline{X} \) (which often is the case), then there is no +need for bootstrapping. + +

+









+ +

Resampling methods: Jackknife

+ +

+The Jackknife works by making many replicas of the estimator \( \widehat{\theta} \). +The jackknife is a resampling method where we systematically leave out one observation from the vector of observed values \( \boldsymbol{x} = (x_1,x_2,\cdots,X_n) \). +Let \( \boldsymbol{x}_i \) denote the vector +$$ +\boldsymbol{x}_i = (x_1,x_2,\cdots,x_{i-1},x_{i+1},\cdots,x_n), +$$ + +

+which equals the vector \( \boldsymbol{x} \) with the exception that observation +number \( i \) is left out. Using this notation, define +\( \widehat{\theta}_i \) to be the estimator +\( \widehat{\theta} \) computed using \( \vec{X}_i \). + +

+









+ +

Jackknife code example

+

+ + +

from numpy import *
+from numpy.random import randint, randn
+from time import time
+
+def jackknife(data, stat):
+    n = len(data);t = zeros(n); inds = arange(n); t0 = time()
+    ## 'jackknifing' by leaving out an observation for each i                                                                                                                      
+    for i in range(n):
+        t[i] = stat(delete(data,i) )
+
+    # analysis                                                                                                                                                                     
+    print("Runtime: %g sec" % (time()-t0)); print("Jackknife Statistics :")
+    print("original           bias      std. error")
+    print("%8g %14g %15g" % (stat(data),(n-1)*mean(t)/n, (n*var(t))**.5))
+
+    return t
+
+
+# Returns mean of data samples                                                                                                                                                     
+def stat(data):
+    return mean(data)
+
+
+mu, sigma = 100, 15
+datapoints = 10000
+x = mu + sigma*random.randn(datapoints)
+# jackknife returns the data sample                                                                                                                                                
+t = jackknife(x, stat)
+
+

+









+ +

Resampling methods: Bootstrap

+
+ +

+Bootstrapping is a nonparametric approach to statistical inference +that substitutes computation for more traditional distributional +assumptions and asymptotic results. Bootstrapping offers a number of +advantages: + +

    +
  1. The bootstrap is quite general, although there are some cases in which it fails.
    2. +
  2. Because it does not require distributional assumptions (such as normally distributed errors), the bootstrap can provide more accurate inferences when the data are not well behaved or when the sample size is small.
    3. +
  3. It is possible to apply the bootstrap to statistics with sampling distributions that are difficult to derive, even asymptotically.
    4. +
  4. It is relatively simple to apply the bootstrap to complex data-collection plans (such as stratified and clustered samples).
    5. +
+
+ + +

+









+ +

Resampling methods: Bootstrap background

+ +

+Since \( \widehat{\theta} = \widehat{\theta}(\boldsymbol{X}) \) is a function of random variables, +\( \widehat{\theta} \) itself must be a random variable. Thus it has +a pdf, call this function \( p(\boldsymbol{t}) \). The aim of the bootstrap is to +estimate \( p(\boldsymbol{t}) \) by the relative frequency of +\( \widehat{\theta} \). You can think of this as using a histogram +in the place of \( p(\boldsymbol{t}) \). If the relative frequency closely +resembles \( p(\vec{t}) \), then using numerics, it is straight forward to +estimate all the interesting parameters of \( p(\boldsymbol{t}) \) using point +estimators. + +

+









+ +

Resampling methods: More Bootstrap background

+ +

+In the case that \( \widehat{\theta} \) has +more than one component, and the components are independent, we use the +same estimator on each component separately. If the probability +density function of \( X_i \), \( p(x) \), had been known, then it would have +been straight forward to do this by: + +

    +
  1. Drawing lots of numbers from \( p(x) \), suppose we call one such set of numbers \( (X_1^*, X_2^*, \cdots, X_n^*) \).
    2. +
  2. Then using these numbers, we could compute a replica of \( \widehat{\theta} \) called \( \widehat{\theta}^* \).
    3. +
+ +By repeated use of (1) and (2), many +estimates of \( \widehat{\theta} \) could have been obtained. The +idea is to use the relative frequency of \( \widehat{\theta}^* \) +(think of a histogram) as an estimate of \( p(\boldsymbol{t}) \). + +

+









+ +

Resampling methods: Bootstrap approach

+ +

+But +unless there is enough information available about the process that +generated \( X_1,X_2,\cdots,X_n \), \( p(x) \) is in general +unknown. Therefore, Efron in 1979 asked the +question: What if we replace \( p(x) \) by the relative frequency +of the observation \( X_i \); if we draw observations in accordance with +the relative frequency of the observations, will we obtain the same +result in some asymptotic sense? The answer is yes. + +

+Instead of generating the histogram for the relative +frequency of the observation \( X_i \), just draw the values +\( (X_1^*,X_2^*,\cdots,X_n^*) \) with replacement from the vector +\( \boldsymbol{X} \). + +

+









+ +

Resampling methods: Bootstrap steps

+ +

+The independent bootstrap works like this: + +

    +
  1. Draw with replacement \( n \) numbers for the observed variables \( \boldsymbol{x} = (x_1,x_2,\cdots,x_n) \).
    2. +
  2. Define a vector \( \boldsymbol{x}^* \) containing the values which were drawn from \( \boldsymbol{x} \).
    3. +
  3. Using the vector \( \boldsymbol{x}^* \) compute \( \widehat{\theta}^* \) by evaluating \( \widehat \theta \) under the observations \( \boldsymbol{x}^* \).
    4. +
  4. Repeat this process \( k \) times.
    5. +
+ +When you are done, you can draw a histogram of the relative frequency +of \( \widehat \theta^* \). This is your estimate of the probability +distribution \( p(t) \). Using this probability distribution you can +estimate any statistics thereof. In principle you never draw the +histogram of the relative frequency of \( \widehat{\theta}^* \). Instead +you use the estimators corresponding to the statistic of interest. For +example, if you are interested in estimating the variance of \( \widehat +\theta \), apply the etsimator \( \widehat \sigma^2 \) to the values +\( \widehat \theta ^* \). + +

+









+ +

Code example for the Bootstrap method

+ +

+The following code starts with a Gaussian distribution with mean value +\( \mu =100 \) and variance \( \sigma=15 \). We use this to generate the data +used in the bootstrap analysis. The bootstrap analysis returns a data +set after a given number of bootstrap operations (as many as we have +data points). This data set consists of estimated mean values for each +bootstrap operation. The histogram generated by the bootstrap method +shows that the distribution for these mean values is also a Gaussian, +centered around the mean value \( \mu=100 \) but with standard deviation +\( \sigma/\sqrt{n} \), where \( n \) is the number of bootstrap samples (in +this case the same as the number of original data points). The value +of the standard deviation is what we expect from the central limit +theorem. + +

+ + +

from numpy import *
+from numpy.random import randint, randn
+from time import time
+import matplotlib.mlab as mlab
+import matplotlib.pyplot as plt
+
+# Returns mean of bootstrap samples                                                                                                                                                
+def stat(data):
+    return mean(data)
+
+# Bootstrap algorithm
+def bootstrap(data, statistic, R):
+    t = zeros(R); n = len(data); inds = arange(n); t0 = time()
+    # non-parametric bootstrap         
+    for i in range(R):
+        t[i] = statistic(data[randint(0,n,n)])
+
+    # analysis    
+    print("Runtime: %g sec" % (time()-t0)); print("Bootstrap Statistics :")
+    print("original           bias      std. error")
+    print("%8g %8g %14g %15g" % (statistic(data), std(data),mean(t),std(t)))
+    return t
+
+
+mu, sigma = 100, 15
+datapoints = 10000
+x = mu + sigma*random.randn(datapoints)
+# bootstrap returns the data sample                                    
+t = bootstrap(x, stat, datapoints)
+# the histogram of the bootstrapped  data                                                                                                    
+n, binsboot, patches = plt.hist(t, 50, normed=1, facecolor='red', alpha=0.75)
+
+# add a 'best fit' line  
+y = mlab.normpdf( binsboot, mean(t), std(t))
+lt = plt.plot(binsboot, y, 'r--', linewidth=1)
+plt.xlabel('Smarts')
+plt.ylabel('Probability')
+plt.axis([99.5, 100.6, 0, 3.0])
+plt.grid(True)
+
+plt.show()
+
+

+









+ +

Code Example for Cross-validation and \( k \)-fold Cross-validation

+ +

+The code here uses Ridge regression with cross-validation (CV) resampling and \( k \)-fold CV in order to fit a specific polynomial. +

+ + +

import numpy as np
+import matplotlib.pyplot as plt
+from sklearn.model_selection import KFold
+from sklearn.linear_model import Ridge
+from sklearn.model_selection import cross_val_score
+from sklearn.preprocessing import PolynomialFeatures
+
+# A seed just to ensure that the random numbers are the same for every run.
+# Useful for eventual debugging.
+np.random.seed(3155)
+
+# Generate the data.
+nsamples = 100
+x = np.random.randn(nsamples)
+y = 3*x**2 + np.random.randn(nsamples)
+
+## Cross-validation on Ridge regression using KFold only
+
+# Decide degree on polynomial to fit
+poly = PolynomialFeatures(degree = 6)
+
+# Decide which values of lambda to use
+nlambdas = 500
+lambdas = np.logspace(-3, 5, nlambdas)
+
+# Initialize a KFold instance
+k = 5
+kfold = KFold(n_splits = k)
+
+# Perform the cross-validation to estimate MSE
+scores_KFold = np.zeros((nlambdas, k))
+
+i = 0
+for lmb in lambdas:
+    ridge = Ridge(alpha = lmb)
+    j = 0
+    for train_inds, test_inds in kfold.split(x):
+        xtrain = x[train_inds]
+        ytrain = y[train_inds]
+
+        xtest = x[test_inds]
+        ytest = y[test_inds]
+
+        Xtrain = poly.fit_transform(xtrain[:, np.newaxis])
+        ridge.fit(Xtrain, ytrain[:, np.newaxis])
+
+        Xtest = poly.fit_transform(xtest[:, np.newaxis])
+        ypred = ridge.predict(Xtest)
+
+        scores_KFold[i,j] = np.sum((ypred - ytest[:, np.newaxis])**2)/np.size(ypred)
+
+        j += 1
+    i += 1
+
+
+estimated_mse_KFold = np.mean(scores_KFold, axis = 1)
+
+## Cross-validation using cross_val_score from sklearn along with KFold
+
+# kfold is an instance initialized above as:
+# kfold = KFold(n_splits = k)
+
+estimated_mse_sklearn = np.zeros(nlambdas)
+i = 0
+for lmb in lambdas:
+    ridge = Ridge(alpha = lmb)
+
+    X = poly.fit_transform(x[:, np.newaxis])
+    estimated_mse_folds = cross_val_score(ridge, X, y[:, np.newaxis], scoring='neg_mean_squared_error', cv=kfold)
+
+    # cross_val_score return an array containing the estimated negative mse for every fold.
+    # we have to the the mean of every array in order to get an estimate of the mse of the model
+    estimated_mse_sklearn[i] = np.mean(-estimated_mse_folds)
+
+    i += 1
+
+## Plot and compare the slightly different ways to perform cross-validation
+
+plt.figure()
+
+plt.plot(np.log10(lambdas), estimated_mse_sklearn, label = 'cross_val_score')
+plt.plot(np.log10(lambdas), estimated_mse_KFold, 'r--', label = 'KFold')
+
+plt.xlabel('log10(lambda)')
+plt.ylabel('mse')
+
+plt.legend()
+
+plt.show()
+
+

+









+ +

The bias-variance tradeoff

+ +

+We will discuss the bias-variance tradeoff in the context of +continuous predictions such as regression. However, many of the +intuitions and ideas discussed here also carry over to classification +tasks. Consider a dataset \( \mathcal{L} \) consisting of the data +\( \mathbf{X}_\mathcal{L}=\{(y_j, \boldsymbol{x}_j), j=0\ldots n-1\} \). + +

+Let us assume that the true data is generated from a noisy model + +$$ +\boldsymbol{y}=f(\boldsymbol{x}) + \boldsymbol{\epsilon} +$$ + +

+where \( \epsilon \) is normally distributed with mean zero and standard deviation \( \sigma^2 \). + +

+In our derivation of the ordinary least squares method we defined then +an approximation to the function \( f \) in terms of the parameters +\( \boldsymbol{\beta} \) and the design matrix \( \boldsymbol{X} \) which embody our model, +that is \( \boldsymbol{\tilde{y}}=\boldsymbol{X}\boldsymbol{\beta} \). + +

+Thereafter we found the parameters \( \boldsymbol{\beta} \) by optimizing the means squared error via the so-called cost function +$$ +C(\boldsymbol{X},\boldsymbol{\beta}) =\frac{1}{n}\sum_{i=0}^{n-1}(y_i-\tilde{y}_i)^2=\mathbb{E}\left[(\boldsymbol{y}-\boldsymbol{\tilde{y}})^2\right]. +$$ + +

+We can rewrite this as +$$ +\mathbb{E}\left[(\boldsymbol{y}-\boldsymbol{\tilde{y}})^2\right]=\frac{1}{n}\sum_i(f_i-\mathbb{E}\left[\boldsymbol{\tilde{y}}\right])^2+\frac{1}{n}\sum_i(\tilde{y}_i-\mathbb{E}\left[\boldsymbol{\tilde{y}}\right])^2+\sigma^2. +$$ + +

+The three terms represent the square of the bias of the learning +method, which can be thought of as the error caused by the simplifying +assumptions built into the method. The second term represents the +variance of the chosen model and finally the last terms is variance of +the error \( \boldsymbol{\epsilon} \). + +

+To derive this equation, we need to recall that the variance of \( \boldsymbol{y} \) and \( \boldsymbol{\epsilon} \) are both equal to \( \sigma^2 \). The mean value of \( \boldsymbol{\epsilon} \) is by definition equal to zero. Furthermore, the function \( f \) is not a stochastics variable, idem for \( \boldsymbol{\tilde{y}} \). +We use a more compact notation in terms of the expectation value +$$ +\mathbb{E}\left[(\boldsymbol{y}-\boldsymbol{\tilde{y}})^2\right]=\mathbb{E}\left[(\boldsymbol{f}+\boldsymbol{\epsilon}-\boldsymbol{\tilde{y}})^2\right], +$$ + +and adding and subtracting \( \mathbb{E}\left[\boldsymbol{\tilde{y}}\right] \) we get +$$ +\mathbb{E}\left[(\boldsymbol{y}-\boldsymbol{\tilde{y}})^2\right]=\mathbb{E}\left[(\boldsymbol{f}+\boldsymbol{\epsilon}-\boldsymbol{\tilde{y}}+\mathbb{E}\left[\boldsymbol{\tilde{y}}\right]-\mathbb{E}\left[\boldsymbol{\tilde{y}}\right])^2\right], +$$ + +which, using the abovementioned expectation values can be rewritten as +$$ +\mathbb{E}\left[(\boldsymbol{y}-\boldsymbol{\tilde{y}})^2\right]=\mathbb{E}\left[(\boldsymbol{y}-\mathbb{E}\left[\boldsymbol{\tilde{y}}\right])^2\right]+\mathrm{Var}\left[\boldsymbol{\tilde{y}}\right]+\sigma^2, +$$ + +that is the rewriting in terms of the so-called bias, the variance of the model \( \boldsymbol{\tilde{y}} \) and the variance of \( \boldsymbol{\epsilon} \). + +

+









+ +

Example code for Bias-Variance tradeoff

+

+ + +

import matplotlib.pyplot as plt
+import numpy as np
+from sklearn.linear_model import LinearRegression, Ridge, Lasso
+from sklearn.preprocessing import PolynomialFeatures
+from sklearn.model_selection import train_test_split
+from sklearn.pipeline import make_pipeline
+from sklearn.utils import resample
+
+np.random.seed(2018)
+
+n = 500
+n_boostraps = 100
+degree = 18  # A quite high value, just to show.
+noise = 0.1
+
+# Make data set.
+x = np.linspace(-1, 3, n).reshape(-1, 1)
+y = np.exp(-x**2) + 1.5 * np.exp(-(x-2)**2) + np.random.normal(0, 0.1, x.shape)
+
+# Hold out some test data that is never used in training.
+x_train, x_test, y_train, y_test = train_test_split(x, y, test_size=0.2)
+
+# Combine x transformation and model into one operation.
+# Not neccesary, but convenient.
+model = make_pipeline(PolynomialFeatures(degree=degree), LinearRegression(fit_intercept=False))
+
+# The following (m x n_bootstraps) matrix holds the column vectors y_pred
+# for each bootstrap iteration.
+y_pred = np.empty((y_test.shape[0], n_boostraps))
+for i in range(n_boostraps):
+    x_, y_ = resample(x_train, y_train)
+
+    # Evaluate the new model on the same test data each time.
+    y_pred[:, i] = model.fit(x_, y_).predict(x_test).ravel()
+
+# Note: Expectations and variances taken w.r.t. different training
+# data sets, hence the axis=1. Subsequent means are taken across the test data
+# set in order to obtain a total value, but before this we have error/bias/variance
+# calculated per data point in the test set.
+# Note 2: The use of keepdims=True is important in the calculation of bias as this 
+# maintains the column vector form. Dropping this yields very unexpected results.
+error = np.mean( np.mean((y_test - y_pred)**2, axis=1, keepdims=True) )
+bias = np.mean( (y_test - np.mean(y_pred, axis=1, keepdims=True))**2 )
+variance = np.mean( np.var(y_pred, axis=1, keepdims=True) )
+print('Error:', error)
+print('Bias^2:', bias)
+print('Var:', variance)
+print('{} >= {} + {} = {}'.format(error, bias, variance, bias+variance))
+
+plt.plot(x[::5, :], y[::5, :], label='f(x)')
+plt.scatter(x_test, y_test, label='Data points')
+plt.scatter(x_test, np.mean(y_pred, axis=1), label='Pred')
+plt.legend()
+plt.show()
+
+

+









+ +

Understanding what happens

+

+ + +

import matplotlib.pyplot as plt
+import numpy as np
+from sklearn.linear_model import LinearRegression, Ridge, Lasso
+from sklearn.preprocessing import PolynomialFeatures
+from sklearn.model_selection import train_test_split
+from sklearn.pipeline import make_pipeline
+from sklearn.utils import resample
+
+np.random.seed(2018)
+
+n = 40
+n_boostraps = 100
+maxdegree = 14
+
+
+# Make data set.
+x = np.linspace(-3, 3, n).reshape(-1, 1)
+y = np.exp(-x**2) + 1.5 * np.exp(-(x-2)**2)+ np.random.normal(0, 0.1, x.shape)
+error = np.zeros(maxdegree)
+bias = np.zeros(maxdegree)
+variance = np.zeros(maxdegree)
+polydegree = np.zeros(maxdegree)
+x_train, x_test, y_train, y_test = train_test_split(x, y, test_size=0.2)
+
+for degree in range(maxdegree):
+    model = make_pipeline(PolynomialFeatures(degree=degree), LinearRegression(fit_intercept=False))
+    y_pred = np.empty((y_test.shape[0], n_boostraps))
+    for i in range(n_boostraps):
+        x_, y_ = resample(x_train, y_train)
+        y_pred[:, i] = model.fit(x_, y_).predict(x_test).ravel()
+
+    polydegree[degree] = degree
+    error[degree] = np.mean( np.mean((y_test - y_pred)**2, axis=1, keepdims=True) )
+    bias[degree] = np.mean( (y_test - np.mean(y_pred, axis=1, keepdims=True))**2 )
+    variance[degree] = np.mean( np.var(y_pred, axis=1, keepdims=True) )
+    print('Polynomial degree:', degree)
+    print('Error:', error[degree])
+    print('Bias^2:', bias[degree])
+    print('Var:', variance[degree])
+    print('{} >= {} + {} = {}'.format(error[degree], bias[degree], variance[degree], bias[degree]+variance[degree]))
+
+plt.plot(polydegree, np.log10(error), label='Error')
+plt.plot(polydegree, bias, label='bias')
+plt.plot(polydegree, variance, label='Variance')
+plt.legend()
+plt.show()
+
+

+ + +

Summing up

+ +

+The bias-variance tradeoff summarizes the fundamental tension in +machine learning, particularly supervised learning, between the +complexity of a model and the amount of training data needed to train +it. Since data is often limited, in practice it is often useful to +use a less-complex model with higher bias, that is a model whose asymptotic +performance is worse than another model because it is easier to +train and less sensitive to sampling noise arising from having a +finite-sized training dataset (smaller variance). + +

+The above equations tell us that in +order to minimize the expected test error, we need to select a +statistical learning method that simultaneously achieves low variance +and low bias. Note that variance is inherently a nonnegative quantity, +and squared bias is also nonnegative. Hence, we see that the expected +test MSE can never lie below \( Var(\epsilon) \), the irreducible error. + +

+What do we mean by the variance and bias of a statistical learning +method? The variance refers to the amount by which our model would change if we +estimated it using a different training data set. Since the training +data are used to fit the statistical learning method, different +training data sets will result in a different estimate. But ideally the +estimate for our model should not vary too much between training +sets. However, if a method has high variance then small changes in +the training data can result in large changes in the model. In general, more +flexible statistical methods have higher variance. + +

+









+ +

Another Example rom Scikit-Learn's Repository

+

+ + +

"""
+============================
+Underfitting vs. Overfitting
+============================
+
+This example demonstrates the problems of underfitting and overfitting and
+how we can use linear regression with polynomial features to approximate
+nonlinear functions. The plot shows the function that we want to approximate,
+which is a part of the cosine function. In addition, the samples from the
+real function and the approximations of different models are displayed. The
+models have polynomial features of different degrees. We can see that a
+linear function (polynomial with degree 1) is not sufficient to fit the
+training samples. This is called **underfitting**. A polynomial of degree 4
+approximates the true function almost perfectly. However, for higher degrees
+the model will **overfit** the training data, i.e. it learns the noise of the
+training data.
+We evaluate quantitatively **overfitting** / **underfitting** by using
+cross-validation. We calculate the mean squared error (MSE) on the validation
+set, the higher, the less likely the model generalizes correctly from the
+training data.
+"""
+
+print(__doc__)
+
+import numpy as np
+import matplotlib.pyplot as plt
+from sklearn.pipeline import Pipeline
+from sklearn.preprocessing import PolynomialFeatures
+from sklearn.linear_model import LinearRegression
+from sklearn.model_selection import cross_val_score
+
+
+def true_fun(X):
+    return np.cos(1.5 * np.pi * X)
+
+np.random.seed(0)
+
+n_samples = 30
+degrees = [1, 4, 15]
+
+X = np.sort(np.random.rand(n_samples))
+y = true_fun(X) + np.random.randn(n_samples) * 0.1
+
+plt.figure(figsize=(14, 5))
+for i in range(len(degrees)):
+    ax = plt.subplot(1, len(degrees), i + 1)
+    plt.setp(ax, xticks=(), yticks=())
+
+    polynomial_features = PolynomialFeatures(degree=degrees[i],
+                                             include_bias=False)
+    linear_regression = LinearRegression()
+    pipeline = Pipeline([("polynomial_features", polynomial_features),
+                         ("linear_regression", linear_regression)])
+    pipeline.fit(X[:, np.newaxis], y)
+
+    # Evaluate the models using crossvalidation
+    scores = cross_val_score(pipeline, X[:, np.newaxis], y,
+                             scoring="neg_mean_squared_error", cv=10)
+
+    X_test = np.linspace(0, 1, 100)
+    plt.plot(X_test, pipeline.predict(X_test[:, np.newaxis]), label="Model")
+    plt.plot(X_test, true_fun(X_test), label="True function")
+    plt.scatter(X, y, edgecolor='b', s=20, label="Samples")
+    plt.xlabel("x")
+    plt.ylabel("y")
+    plt.xlim((0, 1))
+    plt.ylim((-2, 2))
+    plt.legend(loc="best")
+    plt.title("Degree {}\nMSE = {:.2e}(+/- {:.2e})".format(
+        degrees[i], -scores.mean(), scores.std()))
+plt.show()
+
+

+









+ +

The one-dimensional Ising model

+ +

+Let us bring back the Ising model again, but now with an additional +focus on Ridge and Lasso regression as well. We repeat some of the +basic parts of the Ising model and the setup of the training and test +data. The one-dimensional Ising model with nearest neighbor +interaction, no external field and a constant coupling constant \( J \) is +given by + +$$ +\begin{align} + H = -J \sum_{k}^L s_k s_{k + 1}, +\label{_auto17} +\end{align} +$$ + +where \( s_i \in \{-1, 1\} \) and \( s_{N + 1} = s_1 \). The number of spins in the system is determined by \( L \). For the one-dimensional system there is no phase transition. + +

+We will look at a system of \( L = 40 \) spins with a coupling constant of \( J = 1 \). To get enough training data we will generate 10000 states with their respective energies. + +

+ + +

import numpy as np
+import matplotlib.pyplot as plt
+from mpl_toolkits.axes_grid1 import make_axes_locatable
+import seaborn as sns
+import scipy.linalg as scl
+from sklearn.model_selection import train_test_split
+import sklearn.linear_model as skl
+import tqdm
+sns.set(color_codes=True)
+cmap_args=dict(vmin=-1., vmax=1., cmap='seismic')
+
+L = 40
+n = int(1e4)
+
+spins = np.random.choice([-1, 1], size=(n, L))
+J = 1.0
+
+energies = np.zeros(n)
+
+for i in range(n):
+    energies[i] = - J * np.dot(spins[i], np.roll(spins[i], 1))
+
+

+A more general form for the one-dimensional Ising model is + +$$ +\begin{align} + H = - \sum_j^L \sum_k^L s_j s_k J_{jk}. +\label{_auto18} +\end{align} +$$ + +

+Here we allow for interactions beyond the nearest neighbors and a more +adaptive coupling matrix. This latter expression can be formulated as +a matrix-product on the form +$$ +\begin{align} + H = X J, +\label{_auto19} +\end{align} +$$ + +

+where \( X_{jk} = s_j s_k \) and \( J \) is the matrix consisting of the +elements \( -J_{jk} \). This form of writing the energy fits perfectly +with the form utilized in linear regression, viz. +$$ +\begin{align} + \boldsymbol{y} = \boldsymbol{X}\boldsymbol{\beta} + \boldsymbol{\epsilon}. +\label{_auto20} +\end{align} +$$ + +We organize the data as we did above +

+ + +

X = np.zeros((n, L ** 2))
+for i in range(n):
+    X[i] = np.outer(spins[i], spins[i]).ravel()
+y = energies
+X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.96)
+
+X_train_own = np.concatenate(
+    (np.ones(len(X_train))[:, np.newaxis], X_train),
+    axis=1
+)
+
+X_test_own = np.concatenate(
+    (np.ones(len(X_test))[:, np.newaxis], X_test),
+    axis=1
+)
+
+

+We will do all fitting with Scikit-Learn, + +

+ + +

clf = skl.LinearRegression().fit(X_train, y_train)
+
+

+When extracting the \( J \)-matrix we make sure to remove the intercept +

+ + +

J_sk = clf.coef_.reshape(L, L)
+
+

+And then we plot the results +

+ + +

fig = plt.figure(figsize=(20, 14))
+im = plt.imshow(J_sk, **cmap_args)
+plt.title("LinearRegression from Scikit-learn", fontsize=18)
+plt.xticks(fontsize=18)
+plt.yticks(fontsize=18)
+cb = fig.colorbar(im)
+cb.ax.set_yticklabels(cb.ax.get_yticklabels(), fontsize=18)
+plt.show()
+
+

+The results perfectly with our previous discussion where we used our own code. + +

+









+ +

Ridge regression

+ +

+Having explored the ordinary least squares we move on to ridge +regression. In ridge regression we include a regularizer. This +involves a new cost function which leads to a new estimate for the +weights \( \boldsymbol{\beta} \). This results in a penalized regression problem. The +cost function is given by + +$$ +\begin{align} + C(\boldsymbol{X}, \boldsymbol{\beta}; \lambda) = (\boldsymbol{X}\boldsymbol{\beta} - \boldsymbol{y})^T(\boldsymbol{X}\boldsymbol{\beta} - \boldsymbol{y}) + \lambda \boldsymbol{\beta}^T\boldsymbol{\beta}. +\label{_auto21} +\end{align} +$$ + +

+ + +

_lambda = 0.1
+clf_ridge = skl.Ridge(alpha=_lambda).fit(X_train, y_train)
+J_ridge_sk = clf_ridge.coef_.reshape(L, L)
+fig = plt.figure(figsize=(20, 14))
+im = plt.imshow(J_ridge_sk, **cmap_args)
+plt.title("Ridge from Scikit-learn", fontsize=18)
+plt.xticks(fontsize=18)
+plt.yticks(fontsize=18)
+cb = fig.colorbar(im)
+cb.ax.set_yticklabels(cb.ax.get_yticklabels(), fontsize=18)
+
+plt.show()
+
+

+









+ +

LASSO regression

+ +

+In the Least Absolute Shrinkage and Selection Operator (LASSO)-method we get a third cost function. + +$$ +\begin{align} + C(\boldsymbol{X}, \boldsymbol{\beta}; \lambda) = (\boldsymbol{X}\boldsymbol{\beta} - \boldsymbol{y})^T(\boldsymbol{X}\boldsymbol{\beta} - \boldsymbol{y}) + \lambda \sqrt{\boldsymbol{\beta}^T\boldsymbol{\beta}}. +\label{_auto22} +\end{align} +$$ + +

+Finding the extremal point of this cost function is not so straight-forward as in least squares and ridge. We will therefore rely solely on the function ``Lasso`` from Scikit-Learn. + +

+ + +

clf_lasso = skl.Lasso(alpha=_lambda).fit(X_train, y_train)
+J_lasso_sk = clf_lasso.coef_.reshape(L, L)
+fig = plt.figure(figsize=(20, 14))
+im = plt.imshow(J_lasso_sk, **cmap_args)
+plt.title("Lasso from Scikit-learn", fontsize=18)
+plt.xticks(fontsize=18)
+plt.yticks(fontsize=18)
+cb = fig.colorbar(im)
+cb.ax.set_yticklabels(cb.ax.get_yticklabels(), fontsize=18)
+
+plt.show()
+
+

+It is quite striking how LASSO breaks the symmetry of the coupling +constant as opposed to ridge and OLS. We get a sparse solution with +\( J_{j, j + 1} = -1 \). + +

+









+ +

Performance as function of the regularization parameter

+ +

+We see how the different models perform for a different set of values for \( \lambda \). + +

+ + +

lambdas = np.logspace(-4, 5, 10)
+
+train_errors = {
+    "ols_sk": np.zeros(lambdas.size),
+    "ridge_sk": np.zeros(lambdas.size),
+    "lasso_sk": np.zeros(lambdas.size)
+}
+
+test_errors = {
+    "ols_sk": np.zeros(lambdas.size),
+    "ridge_sk": np.zeros(lambdas.size),
+    "lasso_sk": np.zeros(lambdas.size)
+}
+
+plot_counter = 1
+
+fig = plt.figure(figsize=(32, 54))
+
+for i, _lambda in enumerate(tqdm.tqdm(lambdas)):
+    for key, method in zip(
+        ["ols_sk", "ridge_sk", "lasso_sk"],
+        [skl.LinearRegression(), skl.Ridge(alpha=_lambda), skl.Lasso(alpha=_lambda)]
+    ):
+        method = method.fit(X_train, y_train)
+
+        train_errors[key][i] = method.score(X_train, y_train)
+        test_errors[key][i] = method.score(X_test, y_test)
+
+        omega = method.coef_.reshape(L, L)
+
+        plt.subplot(10, 5, plot_counter)
+        plt.imshow(omega, **cmap_args)
+        plt.title(r"%s, $\lambda = %.4f$" % (key, _lambda))
+        plot_counter += 1
+
+plt.show()
+
+

+We see that LASSO reaches a good solution for low +values of \( \lambda \), but will "wither" when we increase \( \lambda \) too +much. Ridge is more stable over a larger range of values for +\( \lambda \), but eventually also fades away. + +

+









+ +

Finding the optimal value of \( \lambda \)

+ +

+To determine which value of \( \lambda \) is best we plot the accuracy of +the models when predicting the training and the testing set. We expect +the accuracy of the training set to be quite good, but if the accuracy +of the testing set is much lower this tells us that we might be +subject to an overfit model. The ideal scenario is an accuracy on the +testing set that is close to the accuracy of the training set. + +

+ + +

fig = plt.figure(figsize=(20, 14))
+
+colors = {
+    "ols_sk": "r",
+    "ridge_sk": "y",
+    "lasso_sk": "c"
+}
+
+for key in train_errors:
+    plt.semilogx(
+        lambdas,
+        train_errors[key],
+        colors[key],
+        label="Train {0}".format(key),
+        linewidth=4.0
+    )
+
+for key in test_errors:
+    plt.semilogx(
+        lambdas,
+        test_errors[key],
+        colors[key] + "--",
+        label="Test {0}".format(key),
+        linewidth=4.0
+    )
+plt.legend(loc="best", fontsize=18)
+plt.xlabel(r"$\lambda$", fontsize=18)
+plt.ylabel(r"$R^2$", fontsize=18)
+plt.tick_params(labelsize=18)
+plt.show()
+
+

+From the above figure we can see that LASSO with \( \lambda = 10^{-2} \) +achieves a very good accuracy on the test set. This by far surpasses the +other models for all values of \( \lambda \). + +

+









+ +

Further Exercises

+ +

Exercise 1

+ +

+We will generate our own dataset for a function \( y(x) \) where \( x \in [0,1] \) and defined by random numbers computed with the uniform distribution. The function \( y \) is a quadratic polynomial in \( x \) with added stochastic noise according to the normal distribution \( \cal {N}(0,1) \). +The following simple Python instructions define our \( x \) and \( y \) values (with 100 data points). +

+ + +

x = np.random.rand(100,1)
+y = 5*x*x+0.1*np.random.randn(100,1)
+
+
    +
  1. Write your own code (following the examples above) for computing the parametrization of the data set fitting a second-order polynomial.
    2. +
  2. Use thereafter scikit-learn (see again the examples in the regression slides) and compare with your own code.
    3. +
  3. Using scikit-learn, compute also the mean square error, a risk metric corresponding to the expected value of the squared (quadratic) error defined as
    4. +
+ +$$ MSE(\hat{y},\hat{\tilde{y}}) = \frac{1}{n} +\sum_{i=0}^{n-1}(y_i-\tilde{y}_i)^2, +$$ + +and the \( R^2 \) score function. +If \( \tilde{\hat{y}}_i \) is the predicted value of the \( i-th \) sample and \( y_i \) is the corresponding true value, then the score \( R^2 \) is defined as +$$ +R^2(\hat{y}, \tilde{\hat{y}}) = 1 - \frac{\sum_{i=0}^{n - 1} (y_i - \tilde{y}_i)^2}{\sum_{i=0}^{n - 1} (y_i - \bar{y})^2}, +$$ + +where we have defined the mean value of \( \hat{y} \) as +$$ +\bar{y} = \frac{1}{n} \sum_{i=0}^{n - 1} y_i. +$$ + +

+You can use the functionality included in scikit-learn. If you feel +for it, you can use your own program and define functions which +compute the above two functions. Discuss the meaning of these +results. Try also to vary the coefficient in front of the added +stochastic noise term and discuss the quality of the fits. + +

Exercise 2, variance of the parameters \( \beta \) in linear regression

+ +

+Show that the variance of the parameters \( \beta \) in the linear regression method (chapter 3, equation (3.8) of Trevor Hastie, Robert Tibshirani, Jerome H. Friedman, The Elements of Statistical Learning, Springer) is given as + +$$ +\mathrm{Var}(\hat{\beta}) = \left(\hat{X}^T\hat{X}\right)^{-1}\sigma^2, +$$ + +with +$$ +\sigma^2 = \frac{1}{N-p-1}\sum_{i=1}^{N} (y_i-\tilde{y}_i)^2, +$$ + +where we have assumed that we fit a function of degree \( p-1 \) (for example a polynomial in \( x \)). + +

Exercise 3

+ +

+This exercise is a continuation of exercise 1. We will +use the same function to generate our data set, still staying with a +simple function \( y(x) \) which we want to fit using linear regression, +but now extending the analysis to include the Ridge and the Lasso +regression methods. You can use the code under the Regression as an example on how to use the Ridge and the Lasso methods. + +

+We will thus again generate our own dataset for a function \( y(x) \) where +\( x \in [0,1] \) and defined by random numbers computed with the uniform +distribution. The function \( y \) is a quadratic polynomial in \( x \) with +added stochastic noise according to the normal distribution \( \cal{N}(0,1) \). + +

+The following simple Python instructions define our \( x \) and \( y \) values (with 100 data points). +

+ + +

x = np.random.rand(100,1)
+y = 5*x*x+0.1*np.random.randn(100,1)
+
+
    +
  1. Write your own code for the Ridge method and compute the parametrization for different values of \( \lambda \). Compare and analyze your results with those from exercise 1. Study the dependence on \( \lambda \) while also varying the strength of the noise in your expression for \( y(x) \).
    2. +
  2. Repeat the above but using the functionality of scikit-learn. Compare your code with the results from scikit-learn. Remember to run with the same random numbers for generating \( x \) and \( y \).
    3. +
  3. Our next step is to study the variance of the parameters \( \beta_1 \) and \( \beta_2 \) (assuming that we are parametrizing our function with a second-order polynomial. We will use standard linear regression and the Ridge regression. You can now opt for either writing your own function that calculates the variance of these paramaters (recall that this is equal to the diagonal elements of the matrix \( (\hat{X}^T\hat{X})+\lambda\hat{I})^{-1} \)) or use the functionality of scikit-learn and compute their variances. Discuss the results of these variances as functions
    4. +
  4. Repeat the previous step but add now the Lasso method. Discuss your results and compare with standard regression and the Ridge regression results.
    5. +
  5. Try to implement the cross-validation as well.
    6. +
  6. Finally, using scikit-learn or your own code, compute also the mean square error, a risk metric corresponding to the expected value of the squared (quadratic) error defined as
    7. +
+ +$$ MSE(\hat{y},\hat{\tilde{y}}) = \frac{1}{n} +\sum_{i=0}^{n-1}(y_i-\tilde{y}_i)^2, +$$ + +and the \( R^2 \) score function. +If \( \tilde{\hat{y}}_i \) is the predicted value of the \( i-th \) sample and \( y_i \) is the corresponding true value, then the score \( R^2 \) is defined as +$$ +R^2(\hat{y}, \tilde{\hat{y}}) = 1 - \frac{\sum_{i=0}^{n - 1} (y_i - \tilde{y}_i)^2}{\sum_{i=0}^{n - 1} (y_i - \bar{y})^2}, +$$ + +where we have defined the mean value of \( \hat{y} \) as +$$ +\bar{y} = \frac{1}{n} \sum_{i=0}^{n - 1} y_i. +$$ + +Discuss these quantities as functions of the variable \( \lambda \) in the Ridge and Lasso regression methods. + +

Exercise 4

+ +

+We will study how +to fit polynomials to a specific two-dimensional function called +Franke's +function. This +is a function which has been widely used when testing various interpolation and fitting +algorithms. Furthermore, after having established the model and the +method, we will employ resamling techniques such as the cross-validation and/or +the bootstrap methods, in order to perform a proper assessment of our models. + +

+The Franke function, which is a weighted sum of four exponentials reads as follows +$$ +\begin{align*} +f(x,y) &= \frac{3}{4}\exp{\left(-\frac{(9x-2)^2}{4} - \frac{(9y-2)^2}{4}\right)}+\frac{3}{4}\exp{\left(-\frac{(9x+1)^2}{49}- \frac{(9y+1)}{10}\right)} \\ +&+\frac{1}{2}\exp{\left(-\frac{(9x-7)^2}{4} - \frac{(9y-3)^2}{4}\right)} -\frac{1}{5}\exp{\left(-(9x-4)^2 - (9y-7)^2\right) }. +\end{align*} +$$ + +

+The function will be defined for \( x,y\in [0,1] \). Our first step will +be to perform an OLS regression analysis of this function, trying out +a polynomial fit with an \( x \) and \( y \) dependence of the form \( [x, y, +x^2, y^2, xy, \dots] \). We will also include cross-validation and +bootstrap as resampling techniques. As in homeworks 1 and 2, we +can use a uniform distribution to set up the arrays of values for \( x \) +and \( y \), or as in the example below just a fix values for \( x \) and \( y \) with a given step size. +In this case we will have two predictors and need to fit a +function (for example a polynomial) of \( x \) and \( y \). Thereafter we will +repeat much of the same procedure using the the Ridge and +Lasso regression methods, introducing thus a dependence on the bias +(penalty) \( \lambda \). + +

+The Python fucntion for the Franke function is included here (it performs also a three-dimensional plot of it) +

+ + +

from mpl_toolkits.mplot3d import Axes3D
+import matplotlib.pyplot as plt
+from matplotlib import cm
+from matplotlib.ticker import LinearLocator, FormatStrFormatter
+import numpy as np
+from random import random, seed
+
+fig = plt.figure()
+ax = fig.gca(projection='3d')
+
+# Make data.
+x = np.arange(0, 1, 0.05)
+y = np.arange(0, 1, 0.05)
+x, y = np.meshgrid(x,y)
+
+
+def FrankeFunction(x,y):
+    term1 = 0.75*np.exp(-(0.25*(9*x-2)**2) - 0.25*((9*y-2)**2))
+    term2 = 0.75*np.exp(-((9*x+1)**2)/49.0 - 0.1*(9*y+1))
+    term3 = 0.5*np.exp(-(9*x-7)**2/4.0 - 0.25*((9*y-3)**2))
+    term4 = -0.2*np.exp(-(9*x-4)**2 - (9*y-7)**2)
+    return term1 + term2 + term3 + term4
+
+
+z = FrankeFunction(x, y)
+
+# Plot the surface.
+surf = ax.plot_surface(x, y, z, cmap=cm.coolwarm,
+                       linewidth=0, antialiased=False)
+
+# Customize the z axis.
+ax.set_zlim(-0.10, 1.40)
+ax.zaxis.set_major_locator(LinearLocator(10))
+ax.zaxis.set_major_formatter(FormatStrFormatter('%.02f'))
+
+# Add a color bar which maps values to colors.
+fig.colorbar(surf, shrink=0.5, aspect=5)
+
+plt.show()
+
+

+We will thus again generate our own dataset for a function \( \mathrm{FrankeFunction}(x,y) \) where +\( x,y \in [0,1] \) could be defined by random numbers computed with the uniform +distribution. The function \( f(x,y) \) is the Franke function. You should explore also the addition +an added stochastic noise to this function using the normal distribution \( \cal{N}(0,1) \). + +

+Write your own code (using either a matrix inversion or a singular value decomposition from e.g., numpy ) or use your code from exercises 1 and 3 +and perform a standard least square regression analysis using polynomials in \( x \) and \( y \) up to fifth order. Find the confidence intervals of the parameters \( \beta \) by computing their variances, evaluate the Mean Squared error (MSE) +$$ MSE(\hat{y},\hat{\tilde{y}}) = \frac{1}{n} +\sum_{i=0}^{n-1}(y_i-\tilde{y}_i)^2, +$$ + +and the \( R^2 \) score function. +If \( \tilde{\hat{y}}_i \) is the predicted value of the \( i-th \) sample and \( y_i \) is the corresponding true value, then the score \( R^2 \) is defined as +$$ +R^2(\hat{y}, \tilde{\hat{y}}) = 1 - \frac{\sum_{i=0}^{n - 1} (y_i - \tilde{y}_i)^2}{\sum_{i=0}^{n - 1} (y_i - \bar{y})^2}, +$$ + +where we have defined the mean value of \( \hat{y} \) as +$$ +\bar{y} = \frac{1}{n} \sum_{i=0}^{n - 1} y_i. +$$ + +

+Perform a resampling of the data where you split the data in training data and test data. Implement the \( k \)-fold cross-validation algorithm and/or the bootstrap algorithm +and evaluate again the MSE and the \( R^2 \) functions resulting from the test data. Evaluate also the bias and variance of the final models. + +

+Write then your own code for the Ridge method, either using matrix +inversion or the singular value decomposition as done for standard OLS. Perform the same analysis as in the +previous exercise (for the same polynomials and include resampling +techniques) but now for different values of \( \lambda \). Compare and +analyze your results with those obtained with standard OLS. Study the +dependence on \( \lambda \) while also varying eventually the strength of +the noise in your expression for \( \mathrm{FrankeFunction}(x,y) \). + +

+Then perform the same studies but now with Lasso regression. Use the functionalities of +scikit-learn. Give a critical discussion of the three methods and a +judgement of which model fits the data best. + + + + +

+ © 1999-2019, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license +
+ + + + + + diff --git a/doc/src/Regression/Regression.aux b/doc/src/Regression/Regression.aux new file mode 100644 index 000000000..2e3e588c5 --- /dev/null +++ b/doc/src/Regression/Regression.aux @@ -0,0 +1,119 @@ +\relax +\providecommand\hyper@newdestlabel[2]{} +\providecommand\zref@newlabel[2]{} +\providecommand\HyperFirstAtBeginDocument{\AtBeginDocument} +\HyperFirstAtBeginDocument{\ifx\hyper@anchor\@undefined +\global\let\oldcontentsline\contentsline +\gdef\contentsline#1#2#3#4{\oldcontentsline{#1}{#2}{#3}} +\global\let\oldnewlabel\newlabel +\gdef\newlabel#1#2{\newlabelxx{#1}#2} +\gdef\newlabelxx#1#2#3#4#5#6{\oldnewlabel{#1}{{#2}{#3}}} +\AtEndDocument{\ifx\hyper@anchor\@undefined +\let\contentsline\oldcontentsline +\let\newlabel\oldnewlabel +\fi} +\fi} +\global\let\hyper@last\relax +\gdef\HyperFirstAtBeginDocument#1{#1} +\providecommand\HyField@AuxAddToFields[1]{} +\providecommand\HyField@AuxAddToCoFields[2]{} +\@writefile{toc}{\contentsline {paragraph}{}{2}{section*.3}} +\@writefile{toc}{\contentsline {paragraph}{}{2}{section*.5}} +\@writefile{toc}{\contentsline {paragraph}{}{2}{section*.7}} +\@writefile{toc}{\contentsline {paragraph}{}{3}{section*.9}} +\@writefile{toc}{\contentsline {paragraph}{}{3}{section*.11}} +\@writefile{toc}{\contentsline {paragraph}{}{4}{section*.13}} +\@writefile{toc}{\contentsline {paragraph}{}{4}{section*.15}} +\@writefile{toc}{\contentsline {paragraph}{}{5}{section*.17}} +\@writefile{toc}{\contentsline {paragraph}{}{5}{section*.19}} +\@writefile{toc}{\contentsline {paragraph}{}{6}{section*.22}} +\@writefile{toc}{\contentsline {paragraph}{}{6}{section*.24}} +\@writefile{toc}{\contentsline {paragraph}{}{7}{section*.26}} +\@writefile{toc}{\contentsline {paragraph}{}{7}{section*.28}} +\@writefile{toc}{\contentsline {paragraph}{}{8}{section*.32}} +\@writefile{toc}{\contentsline {paragraph}{}{8}{section*.34}} +\@writefile{toc}{\contentsline {paragraph}{}{9}{section*.36}} +\@writefile{toc}{\contentsline {paragraph}{}{9}{section*.38}} +\@writefile{toc}{\contentsline {paragraph}{}{9}{section*.40}} +\@writefile{toc}{\contentsline {paragraph}{}{10}{section*.42}} +\@writefile{toc}{\contentsline {paragraph}{}{11}{section*.47}} +\@writefile{toc}{\contentsline {paragraph}{}{20}{section*.65}} +\@writefile{toc}{\contentsline {paragraph}{}{20}{section*.67}} +\@writefile{toc}{\contentsline {paragraph}{Statistical analysis.}{20}{section*.69}} +\@writefile{toc}{\contentsline {paragraph}{}{20}{section*.71}} +\@writefile{toc}{\contentsline {paragraph}{}{21}{section*.73}} +\@writefile{toc}{\contentsline {paragraph}{}{21}{section*.75}} +\@writefile{toc}{\contentsline {paragraph}{}{21}{section*.77}} +\@writefile{toc}{\contentsline {paragraph}{}{22}{section*.79}} +\newlabel{eq:def_covariance}{{12}{22}{}{equation.0.12}{}} +\@writefile{toc}{\contentsline {paragraph}{}{22}{section*.81}} +\@writefile{toc}{\contentsline {paragraph}{}{23}{section*.83}} +\@writefile{toc}{\contentsline {paragraph}{}{23}{section*.85}} +\newlabel{eq:variance_linear_combination}{{18}{23}{}{equation.0.18}{}} +\@writefile{toc}{\contentsline {paragraph}{}{23}{section*.87}} +\@writefile{toc}{\contentsline {paragraph}{}{24}{section*.89}} +\@writefile{toc}{\contentsline {paragraph}{}{24}{section*.91}} +\@writefile{toc}{\contentsline {paragraph}{}{24}{section*.93}} +\@writefile{toc}{\contentsline {paragraph}{}{25}{section*.95}} +\@writefile{toc}{\contentsline {paragraph}{}{25}{section*.97}} +\@writefile{toc}{\contentsline {paragraph}{}{25}{section*.99}} +\newlabel{eq:central_limit_gaussian}{{19}{25}{}{equation.0.19}{}} +\@writefile{toc}{\contentsline {paragraph}{}{25}{section*.101}} +\newlabel{eq:error_exact}{{20}{25}{}{equation.0.20}{}} +\@writefile{toc}{\contentsline {paragraph}{}{26}{section*.103}} +\@writefile{toc}{\contentsline {paragraph}{}{26}{section*.105}} +\newlabel{eq:var_estimate_i_think}{{21}{26}{}{equation.0.21}{}} +\newlabel{eq:error_estimate}{{22}{26}{}{equation.0.22}{}} +\@writefile{toc}{\contentsline {paragraph}{}{27}{section*.107}} +\newlabel{eq:error_estimate_uncorrel}{{23}{27}{}{equation.0.23}{}} +\@writefile{toc}{\contentsline {paragraph}{}{27}{section*.109}} +\newlabel{eq:error_estimate_split_up}{{24}{27}{}{equation.0.24}{}} +\@writefile{toc}{\contentsline {paragraph}{}{27}{section*.111}} +\@writefile{toc}{\contentsline {paragraph}{}{28}{section*.113}} +\@writefile{toc}{\contentsline {paragraph}{}{28}{section*.115}} +\newlabel{eq:error_estimate_corr_time}{{25}{28}{}{equation.0.25}{}} +\newlabel{eq:autocorrelation_time}{{26}{28}{}{equation.0.26}{}} +\@writefile{toc}{\contentsline {paragraph}{}{28}{section*.117}} +\@writefile{toc}{\contentsline {paragraph}{}{33}{section*.130}} +\@writefile{toc}{\contentsline {paragraph}{Exercise 1.}{39}{section*.148}} +\@writefile{toc}{\contentsline {paragraph}{Exercise 2, variance of the parameters $\beta $ in linear regression.}{39}{section*.149}} +\@writefile{toc}{\contentsline {paragraph}{Exercise 3.}{40}{section*.150}} +\@writefile{toc}{\contentsline {paragraph}{Exercise 4.}{41}{section*.151}} +\gdef\minted@oldcachelist{, + default.pygstyle, + 51639A41B11E5FE7D4CD027BA48FB6C64B967B05670D715D94F4392BD3176A49.pygtex, + 4C68C697EB414B135747B9ECA18C3B5B4B967B05670D715D94F4392BD3176A49.pygtex, + 439DCAB036A38F6389948C8DF1465C424B967B05670D715D94F4392BD3176A49.pygtex, + 0A3C1B39ACFAF4BAB627404CB6156A144B967B05670D715D94F4392BD3176A49.pygtex, + F5688422A51FDCF3C06A3FF7EF8F9E6E4B967B05670D715D94F4392BD3176A49.pygtex, + FE87C84792E25A9D98329B989214E79F4B967B05670D715D94F4392BD3176A49.pygtex, + F9FF5FEA9F501D7189D0F820FC8D1D3D4B967B05670D715D94F4392BD3176A49.pygtex, + 2702C77D52D3F655EDD2BB83D8F1240B4B967B05670D715D94F4392BD3176A49.pygtex, + E81889C7775BD173862927052713983D4B967B05670D715D94F4392BD3176A49.pygtex, + 31D43D24D5F9D55728C4E563E0E557924B967B05670D715D94F4392BD3176A49.pygtex, + F6068C18D7EFD701D08693F8BD52B4664B967B05670D715D94F4392BD3176A49.pygtex, + 9CC2E80F086A0B9D080182D504A931C14B967B05670D715D94F4392BD3176A49.pygtex, + 4C6F7D24872AF7644FF40FC87BFFAB2B4B967B05670D715D94F4392BD3176A49.pygtex, + C8DBC01C383B0800E51E0923C05691E04B967B05670D715D94F4392BD3176A49.pygtex, + E06035EDB37DBFD5697CD9DD1C5CA4A34B967B05670D715D94F4392BD3176A49.pygtex, + 363FA9AE05CC4E390538E08524572CF34B967B05670D715D94F4392BD3176A49.pygtex, + 367BFC34E39C7EA712AF1384CAA96F674B967B05670D715D94F4392BD3176A49.pygtex, + 09054641A8F49E0FADA410EE6C9BC1BA4B967B05670D715D94F4392BD3176A49.pygtex, + A7435FEC44C0E9C1E9660C02CDC74B3F4B967B05670D715D94F4392BD3176A49.pygtex, + 16523CD54121DE1835E829B872835EA64B967B05670D715D94F4392BD3176A49.pygtex, + 5A860BD7323291C7CF7221BB9B86BDC74B967B05670D715D94F4392BD3176A49.pygtex, + 457F2C4C88437DDFAF88B1D28783CF034B967B05670D715D94F4392BD3176A49.pygtex, + 066BCC1A504ED48C98D58B2EC6A448284B967B05670D715D94F4392BD3176A49.pygtex, + 5D2065327E88AF636A043C8526ED982B4B967B05670D715D94F4392BD3176A49.pygtex, + B0F453236926744E43E08A3FC3D889CE4B967B05670D715D94F4392BD3176A49.pygtex, + BFFA2954B53E806314EE31A7A4153BE34B967B05670D715D94F4392BD3176A49.pygtex, + F0C6E7177EB3BF181C2BCDCF6C126D174B967B05670D715D94F4392BD3176A49.pygtex, + 679F13D40F197D86A49DDBAE2064874A4B967B05670D715D94F4392BD3176A49.pygtex, + ECE8CFF2AC201FE83D423B89AB24D84D4B967B05670D715D94F4392BD3176A49.pygtex, + 90854007E426D557D1191356516A553B4B967B05670D715D94F4392BD3176A49.pygtex, + D10574B3CE6341E3B1CE27A555FC959C4B967B05670D715D94F4392BD3176A49.pygtex, + 207A7B5C5A05397AD9CC324488E0F0A34B967B05670D715D94F4392BD3176A49.pygtex, + 0FE2839ED408ED62CECC4C3D9FE2DDE84B967B05670D715D94F4392BD3176A49.pygtex, + EAD510BC3ABCC6143355164B81A3C2ED4B967B05670D715D94F4392BD3176A49.pygtex, + EAD510BC3ABCC6143355164B81A3C2ED4B967B05670D715D94F4392BD3176A49.pygtex, + 795F82986CD7DD6E2E776380A1AFB2624B967B05670D715D94F4392BD3176A49.pygtex} diff --git a/doc/src/Regression/Regression.dlog b/doc/src/Regression/Regression.dlog new file mode 100644 index 000000000..b34391d33 --- /dev/null +++ b/doc/src/Regression/Regression.dlog @@ -0,0 +1,146 @@ +translating doconce text in Regression.do.txt to html +*** replacing \bm{...} by \boldsymbol{...} (\bm is not supported by MathJax) +*** warning: math block in HTML must have space around <: +\begin{equation} +\frac{1}{n^2}\sum_{k=1}^n (x_k - \bar x_n)^2 +\frac{2}{n^2}\sum_{k + + + + + + + +Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis + + + + + + + + + + + + + + + + + + + + + + + +

Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis

+ +

+ + +

+Morten Hjorth-Jensen [1, 2] +
+ +

+ + +

[1] Department of Physics, University of Oslo
+
[2] Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University
+
+

+

Jul 22, 2019

+
+

+









+ +

Why Linear Regression (aka Ordinary Least Squares and family)

+ +

+Fitting a continuous function with linear parameterization in terms of the parameters \( \boldsymbol{\beta} \). + +

    +
  • Method of choice for fitting a continuous function!
    • +
  • Gives an excellent introduction to central Machine Learning features with understandable pedagogical links to other methods like Neural Networks, Support Vector Machines etc
    • +
  • Analytical expression for the fitting parameters \( \boldsymbol{\beta} \)
    • +
  • Analytical expressions for statistical propertiers like mean values, variances, confidence intervals and more
    • +
  • Analytical relation with probabilistic interpretations
    • +
  • Easy to introduce basic concepts like bias-variance tradeoff, cross-validation, resampling and regularization techniques and many other ML topics
    • +
  • Easy to code! And links well with classification problems and logistic regression and neural networks
    • +
  • Allows for easy hands-on understanding of gradient descent methods
    • +
  • and many more features
    • +
+ +For more discussions of Ridge and Lasso regression, Wessel van Wieringen's article is highly recommended. +Similarly, Mehta et al's article is also recommended. + +

+









+ +

Regression analysis, overarching aims

+
+ +

+ +

+Regression modeling deals with the description of the sampling distribution of a given random variable \( y \) and how it varies as function of another variable or a set of such variables \( \boldsymbol{x} =[x_0, x_1,\dots, x_{n-1}]^T \). +The first variable is called the dependent, the outcome or the response variable while the set of variables \( \boldsymbol{x} \) is called the independent variable, or the predictor variable or the explanatory variable. + +

+A regression model aims at finding a likelihood function \( p(\boldsymbol{y}\vert \boldsymbol{x}) \), that is the conditional distribution for \( \boldsymbol{y} \) with a given \( \boldsymbol{x} \). The estimation of \( p(\boldsymbol{y}\vert \boldsymbol{x}) \) is made using a data set with + +

    +
  • \( n \) cases \( i = 0, 1, 2, \dots, n-1 \)
    • +
  • Response (target, dependent or outcome) variable \( y_i \) with \( i = 0, 1, 2, \dots, n-1 \)
    • +
  • \( p \) so-called explanatory (independent or predictor) variables \( \boldsymbol{x}_i=[x_{i0}, x_{i1}, \dots, x_{ip-1}] \) with \( i = 0, 1, 2, \dots, n-1 \) and explanatory variables running from \( 0 \) to \( p-1 \). See below for more explicit examples.
    • +
+ + The goal of the regression analysis is to extract/exploit relationship between \( \boldsymbol{y} \) and \( \boldsymbol{X} \) in or to infer causal dependencies, approximations to the likelihood functions, functional relationships and to make predictions, making fits and many other things. +
+ + +

+









+ +

Regression analysis, overarching aims II

+
+ +

+ +

+Consider an experiment in which \( p \) characteristics of \( n \) samples are +measured. The data from this experiment, for various explanatory variables \( p \) are normally represented by a matrix +\( \mathbf{X} \). + +

+The matrix \( \mathbf{X} \) is called the design +matrix. Additional information of the samples is available in the +form of \( \boldsymbol{y} \) (also as above). The variable \( \boldsymbol{y} \) is +generally referred to as the response variable. The aim of +regression analysis is to explain \( \boldsymbol{y} \) in terms of +\( \boldsymbol{X} \) through a functional relationship like \( y_i = +f(\mathbf{X}_{i,\ast}) \). When no prior knowledge on the form of +\( f(\cdot) \) is available, it is common to assume a linear relationship +between \( \boldsymbol{X} \) and \( \boldsymbol{y} \). This assumption gives rise to +the linear regression model where \( \boldsymbol{\beta} = [\beta_0, \ldots, +\beta_{p-1}]^{T} \) are the regression parameters. + +

+Linear regression gives us a set of analytical equations for the parameters \( \beta_j \). + + +

+ + +

+









+ +

Examples

+
+ +

+In order to understand the relation among the predictors \( p \), the set of data \( n \) and the target (outcome, output etc) \( \boldsymbol{y} \), +consider the model we discussed for describing nuclear binding energies. + +

+There we assumed that we could parametrize the data using a polynomial approximation based on the liquid drop model. +Assuming +$$ +BE(A) = a_0+a_1A+a_2A^{2/3}+a_3A^{-1/3}+a_4A^{-1}, +$$ + +we have five predictors, that is the intercept, the \( A \) dependent term, the \( A^{2/3} \) term and the \( A^{-1/3} \) and \( A^{-1} \) terms. +This gives \( p=0,1,2,3,4 \). Furthermore we have \( n \) entries for each predictor. It means that our design matrix is a +\( p\times n \) matrix \( \boldsymbol{X} \). + +

+Here the predictors are based on a model we have made. A popular data set which is widely encountered in ML applications is the +so-called credit card default data from Taiwan. The data set contains data on \( n=30000 \) credit card holders with predictors like gender, marital status, age, profession, education, etc. In total there are \( 24 \) such predictors or attributes leading to a design matrix of dimensionality \( 24 \times 30000 \) + + +

+ + +

+









+ +

General linear models

+
+ +

+Before we proceed let us study a case from linear algebra where we aim at fitting a set of data \( \boldsymbol{y}=[y_0,y_1,\dots,y_{n-1}] \). We could think of these data as a result of an experiment or a complicated numerical experiment. These data are functions of a series of variables \( \boldsymbol{x}=[x_0,x_1,\dots,x_{n-1}] \), that is \( y_i = y(x_i) \) with \( i=0,1,2,\dots,n-1 \). The variables \( x_i \) could represent physical quantities like time, temperature, position etc. We assume that \( y(x) \) is a smooth function. + +

+Since obtaining these data points may not be trivial, we want to use these data to fit a function which can allow us to make predictions for values of \( y \) which are not in the present set. The perhaps simplest approach is to assume we can parametrize our function in terms of a polynomial of degree \( n-1 \) with \( n \) points, that is +$$ +y=y(x) \rightarrow y(x_i)=\tilde{y}_i+\epsilon_i=\sum_{j=0}^{n-1} \beta_j x_i^j+\epsilon_i, +$$ + +where \( \epsilon_i \) is the error in our approximation. + + +

+ + +

+









+ +

Rewriting the fitting procedure as a linear algebra problem

+
+ +

+For every set of values \( y_i,x_i \) we have thus the corresponding set of equations +$$ +\begin{align*} +y_0&=\beta_0+\beta_1x_0^1+\beta_2x_0^2+\dots+\beta_{n-1}x_0^{n-1}+\epsilon_0\\ +y_1&=\beta_0+\beta_1x_1^1+\beta_2x_1^2+\dots+\beta_{n-1}x_1^{n-1}+\epsilon_1\\ +y_2&=\beta_0+\beta_1x_2^1+\beta_2x_2^2+\dots+\beta_{n-1}x_2^{n-1}+\epsilon_2\\ +\dots & \dots \\ +y_{n-1}&=\beta_0+\beta_1x_{n-1}^1+\beta_2x_{n-1}^2+\dots+\beta_{n-1}x_{n-1}^{n-1}+\epsilon_{n-1}.\\ +\end{align*} +$$ +

+ + +

+









+ +

Rewriting the fitting procedure as a linear algebra problem, more details

+
+ +

+Defining the vectors +$$ +\boldsymbol{y} = [y_0,y_1, y_2,\dots, y_{n-1}]^T, +$$ + +and +$$ +\boldsymbol{\beta} = [\beta_0,\beta_1, \beta_2,\dots, \beta_{n-1}]^T, +$$ + +and +$$ +\boldsymbol{\epsilon} = [\epsilon_0,\epsilon_1, \epsilon_2,\dots, \epsilon_{n-1}]^T, +$$ + +and the design matrix +$$ +\boldsymbol{X}= +\begin{bmatrix} +1& x_{0}^1 &x_{0}^2& \dots & \dots &x_{0}^{n-1}\\ +1& x_{1}^1 &x_{1}^2& \dots & \dots &x_{1}^{n-1}\\ +1& x_{2}^1 &x_{2}^2& \dots & \dots &x_{2}^{n-1}\\ +\dots& \dots &\dots& \dots & \dots &\dots\\ +1& x_{n-1}^1 &x_{n-1}^2& \dots & \dots &x_{n-1}^{n-1}\\ +\end{bmatrix} +$$ + +we can rewrite our equations as +$$ +\boldsymbol{y} = \boldsymbol{X}\boldsymbol{\beta}+\boldsymbol{\epsilon}. +$$ + +The above design matrix is called a Vandermonde matrix. +

+ + +

+









+ +

Generalizing the fitting procedure as a linear algebra problem

+
+ +

+ +

+We are obviously not limited to the above polynomial expansions. We +could replace the various powers of \( x \) with elements of Fourier +series or instead of \( x_i^j \) we could have \( \cos{(j x_i)} \) or \( \sin{(j +x_i)} \), or time series or other orthogonal functions. For every set +of values \( y_i,x_i \) we can then generalize the equations to + +$$ +\begin{align*} +y_0&=\beta_0x_{00}+\beta_1x_{01}+\beta_2x_{02}+\dots+\beta_{n-1}x_{0n-1}+\epsilon_0\\ +y_1&=\beta_0x_{10}+\beta_1x_{11}+\beta_2x_{12}+\dots+\beta_{n-1}x_{1n-1}+\epsilon_1\\ +y_2&=\beta_0x_{20}+\beta_1x_{21}+\beta_2x_{22}+\dots+\beta_{n-1}x_{2n-1}+\epsilon_2\\ +\dots & \dots \\ +y_{i}&=\beta_0x_{i0}+\beta_1x_{i1}+\beta_2x_{i2}+\dots+\beta_{n-1}x_{in-1}+\epsilon_i\\ +\dots & \dots \\ +y_{n-1}&=\beta_0x_{n-1,0}+\beta_1x_{n-1,2}+\beta_2x_{n-1,2}+\dots+\beta_{n-1}x_{n-1,n-1}+\epsilon_{n-1}.\\ +\end{align*} +$$ + +

+Note that we have \( p=n \) here. The matrix is symmetric. This is generally not the case! +

+ + +

+









+ +

Generalizing the fitting procedure as a linear algebra problem

+
+ +

+We redefine in turn the matrix \( \boldsymbol{X} \) as +$$ +\boldsymbol{X}= +\begin{bmatrix} +x_{00}& x_{01} &x_{02}& \dots & \dots &x_{0,n-1}\\ +x_{10}& x_{11} &x_{12}& \dots & \dots &x_{1,n-1}\\ +x_{20}& x_{21} &x_{22}& \dots & \dots &x_{2,n-1}\\ +\dots& \dots &\dots& \dots & \dots &\dots\\ +x_{n-1,0}& x_{n-1,1} &x_{n-1,2}& \dots & \dots &x_{n-1,n-1}\\ +\end{bmatrix} +$$ + +and without loss of generality we rewrite again our equations as +$$ +\boldsymbol{y} = \boldsymbol{X}\boldsymbol{\beta}+\boldsymbol{\epsilon}. +$$ + +The left-hand side of this equation is kwown. Our error vector \( \boldsymbol{\epsilon} \) and the parameter vector \( \boldsymbol{\beta} \) are our unknow quantities. How can we obtain the optimal set of \( \beta_i \) values? +

+ + +

+









+ +

Optimizing our parameters

+
+ +

+We have defined the matrix \( \boldsymbol{X} \) via the equations +$$ +\begin{align*} +y_0&=\beta_0x_{00}+\beta_1x_{01}+\beta_2x_{02}+\dots+\beta_{n-1}x_{0n-1}+\epsilon_0\\ +y_1&=\beta_0x_{10}+\beta_1x_{11}+\beta_2x_{12}+\dots+\beta_{n-1}x_{1n-1}+\epsilon_1\\ +y_2&=\beta_0x_{20}+\beta_1x_{21}+\beta_2x_{22}+\dots+\beta_{n-1}x_{2n-1}+\epsilon_1\\ +\dots & \dots \\ +y_{i}&=\beta_0x_{i0}+\beta_1x_{i1}+\beta_2x_{i2}+\dots+\beta_{n-1}x_{in-1}+\epsilon_1\\ +\dots & \dots \\ +y_{n-1}&=\beta_0x_{n-1,0}+\beta_1x_{n-1,2}+\beta_2x_{n-1,2}+\dots+\beta_{n-1}x_{n-1,n-1}+\epsilon_{n-1}.\\ +\end{align*} +$$ + +

+As we noted above, we stayed with a system with the design matrix + \( \boldsymbol{X}\in {\mathbb{R}}^{n\times n} \), that is we have \( p=n \). For reasons to come later (algorithmic arguments) we will hereafter define +our matrix as \( \boldsymbol{X}\in {\mathbb{R}}^{n\times p} \), with the predictors refering to the column numbers and the entries \( n \) being the row elements. + + +

+ + +

+









+ +

Our model for the nuclear binding energies

+ +

+In our introductory notes we looked at the so-called liguid drop model. Let us remind ourselves about what we did by looking at the code. + +

+We restate the parts of the code we are most interested in. +

+ + +

# Common imports
+import numpy as np
+import pandas as pd
+import matplotlib.pyplot as plt
+from IPython.display import display
+import os
+
+# Where to save the figures and data files
+PROJECT_ROOT_DIR = "Results"
+FIGURE_ID = "Results/FigureFiles"
+DATA_ID = "DataFiles/"
+
+if not os.path.exists(PROJECT_ROOT_DIR):
+    os.mkdir(PROJECT_ROOT_DIR)
+
+if not os.path.exists(FIGURE_ID):
+    os.makedirs(FIGURE_ID)
+
+if not os.path.exists(DATA_ID):
+    os.makedirs(DATA_ID)
+
+def image_path(fig_id):
+    return os.path.join(FIGURE_ID, fig_id)
+
+def data_path(dat_id):
+    return os.path.join(DATA_ID, dat_id)
+
+def save_fig(fig_id):
+    plt.savefig(image_path(fig_id) + ".png", format='png')
+
+infile = open(data_path("MassEval2016.dat"),'r')
+
+
+# Read the experimental data with Pandas
+Masses = pd.read_fwf(infile, usecols=(2,3,4,6,11),
+              names=('N', 'Z', 'A', 'Element', 'Ebinding'),
+              widths=(1,3,5,5,5,1,3,4,1,13,11,11,9,1,2,11,9,1,3,1,12,11,1),
+              header=39,
+              index_col=False)
+
+# Extrapolated values are indicated by '#' in place of the decimal place, so
+# the Ebinding column won't be numeric. Coerce to float and drop these entries.
+Masses['Ebinding'] = pd.to_numeric(Masses['Ebinding'], errors='coerce')
+Masses = Masses.dropna()
+# Convert from keV to MeV.
+Masses['Ebinding'] /= 1000
+
+# Group the DataFrame by nucleon number, A.
+Masses = Masses.groupby('A')
+# Find the rows of the grouped DataFrame with the maximum binding energy.
+Masses = Masses.apply(lambda t: t[t.Ebinding==t.Ebinding.max()])
+A = Masses['A']
+Z = Masses['Z']
+N = Masses['N']
+Element = Masses['Element']
+Energies = Masses['Ebinding']
+
+# Now we set up the design matrix X
+X = np.zeros((len(A),5))
+X[:,0] = 1
+X[:,1] = A
+X[:,2] = A**(2.0/3.0)
+X[:,3] = A**(-1.0/3.0)
+X[:,4] = A**(-1.0)
+# Then nice printout using pandas
+DesignMatrix = pd.DataFrame(X)
+DesignMatrix.index = A
+DesignMatrix.columns = ['1', 'A', 'A^(2/3)', 'A^(-1/3)', '1/A']
+display(DesignMatrix)
+
+

+With \( \boldsymbol{\beta}\in {\mathbb{R}}^{p\times 1} \), it means that we will hereafter write our equations for the approximation as +$$ +\boldsymbol{\tilde{y}}= \boldsymbol{X}\boldsymbol{\beta}, +$$ + +throughout these lectures. + +

+









+ +

Optimizing our parameters, more details

+
+ +

+With the above we use the design matrix to define the approximation \( \boldsymbol{\tilde{y}} \) via the unknown quantity \( \boldsymbol{\beta} \) as +$$ +\boldsymbol{\tilde{y}}= \boldsymbol{X}\boldsymbol{\beta}, +$$ + +and in order to find the optimal parameters \( \beta_i \) instead of solving the above linear algebra problem, we define a function which gives a measure of the spread between the values \( y_i \) (which represent hopefully the exact values) and the parameterized values \( \tilde{y}_i \), namely +$$ +C(\boldsymbol{\beta})=\frac{1}{n}\sum_{i=0}^{n-1}\left(y_i-\tilde{y}_i\right)^2=\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{\tilde{y}}\right)^T\left(\boldsymbol{y}-\boldsymbol{\tilde{y}}\right)\right\}, +$$ + +or using the matrix \( \boldsymbol{X} \) and in a more compact matrix-vector notation as +$$ +C(\boldsymbol{\beta})=\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{X}^T\boldsymbol{\beta}\right)^T\left(\boldsymbol{y}-\boldsymbol{X}^T\boldsymbol{\beta}\right)\right\}. +$$ + +This function is one possible way to define the so-called cost function. + +

+It is also common to define +the function \( Q \) as + +$$ +C(\boldsymbol{\beta})=\frac{1}{2n}\sum_{i=0}^{n-1}\left(y_i-\tilde{y}_i\right)^2, +$$ + +since when taking the first derivative with respect to the unknown parameters \( \beta \), the factor of \( 2 \) cancels out. +

+ + +

+









+ +

Interpretations and optimizing our parameters

+
+ +

+ +

+The function +$$ +C(\boldsymbol{\beta})=\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)^T\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)\right\}, +$$ + +can be linked to the variance of the quantity \( y_i \) if we interpret the latter as the mean value. +When linking below with the maximum likelihood approach below, we will indeed interpret \( y_i \) as a mean value (see exercises) +$$ +y_{i}=\langle y_i \rangle = \beta_0x_{i,0}+\beta_1x_{i,1}+\beta_2x_{i,2}+\dots+\beta_{n-1}x_{i,n-1}+\epsilon_i, +$$ + +

+where \( \langle y_i \rangle \) is the mean value. Keep in mind also that +till now we have treated \( y_i \) as the exact value. Normally, the +response (dependent or outcome) variable \( y_i \) the outcome of a +numerical experiment or another type of experiment and is thus only an +approximation to the true value. It is then always accompanied by an +error estimate, often limited to a statistical error estimate given by +the standard deviation discussed earlier. In the discussion here we +will treat \( y_i \) as our exact value for the response variable. + +

+In order to find the parameters \( \beta_i \) we will then minimize the spread of \( C(\boldsymbol{\beta}) \), that is we are going to solve the problem +$$ +{\displaystyle \min_{\boldsymbol{\beta}\in +{\mathbb{R}}^{p}}}\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)^T\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)\right\}. +$$ + +In practical terms it means we will require +$$ +\frac{\partial C(\boldsymbol{\beta})}{\partial \beta_j} = \frac{\partial }{\partial \beta_j}\left[ \frac{1}{n}\sum_{i=0}^{n-1}\left(y_i-\beta_0x_{i,0}-\beta_1x_{i,1}-\beta_2x_{i,2}-\dots-\beta_{n-1}x_{i,n-1}\right)^2\right]=0, +$$ + +which results in +$$ +\frac{\partial C(\boldsymbol{\beta})}{\partial \beta_j} = -\frac{2}{n}\left[ \sum_{i=0}^{n-1}x_{ij}\left(y_i-\beta_0x_{i,0}-\beta_1x_{i,1}-\beta_2x_{i,2}-\dots-\beta_{n-1}x_{i,n-1}\right)\right]=0, +$$ + +or in a matrix-vector form as +$$ +\frac{\partial C(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}} = 0 = \boldsymbol{X}^T\left( \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right). +$$ + + +

+ + +

+









+ +

Interpretations and optimizing our parameters

+
+ +

+We can rewrite +$$ +\frac{\partial C(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}} = 0 = \boldsymbol{X}^T\left( \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right), +$$ + +as +$$ +\boldsymbol{X}^T\boldsymbol{y} = \boldsymbol{X}^T\boldsymbol{X}\boldsymbol{\beta}, +$$ + +and if the matrix \( \boldsymbol{X}^T\boldsymbol{X} \) is invertible we have the solution +$$ +\boldsymbol{\beta} =\left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}. +$$ + +

+We note also that since our design matrix is defined as \( \boldsymbol{X}\in +{\mathbb{R}}^{n\times p} \), the product \( \boldsymbol{X}^T\boldsymbol{X} \in +{\mathbb{R}}^{p\times p} \). In the above case we have that \( p \ll n \), +in our case \( p=5 \) meaning that we end up with inverting a small +\( 5\times 5 \) matrix. This is a rather common situation, in many cases we end up with low-dimensional +matrices to invert. The methods discussed here and for many other +supervised learning algorithms like classification with logistic +regression or support vector machines, exhibit dimensionalities which +allow for the usage of direct linear algebra methods such as LU decomposition or Singular Value Decomposition (SVD) for finding the inverse of the matrix +\( \boldsymbol{X}^T\boldsymbol{X} \). + + +

+ + +

+









+ +

Interpretations and optimizing our parameters

+
+ +

+The residuals \( \boldsymbol{\epsilon} \) are in turn given by +$$ +\boldsymbol{\epsilon} = \boldsymbol{y}-\boldsymbol{\tilde{y}} = \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}, +$$ + +and with +$$ +\boldsymbol{X}^T\left( \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)= 0, +$$ + +we have +$$ +\boldsymbol{X}^T\boldsymbol{\epsilon}=\boldsymbol{X}^T\left( \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)= 0, +$$ + +meaning that the solution for \( \boldsymbol{\beta} \) is the one which minimizes the residuals. Later we will link this with the maximum likelihood approach. + + +

+ + +

+Let us now return to our nuclear binding energies and simply code the above equations. + +

+









+ +

Own code for Ordinary Least Squares

+ +

+It is rather straightforward to implement the matrix inversion and obtain the parameters \( \boldsymbol{\beta} \). After having defined the matrix \( \boldsymbol{X} \) we simply need to +write +

+ + +

# matrix inversion to find beta
+beta = np.linalg.inv(X.T.dot(X)).dot(X.T).dot(Energies)
+# and then make the prediction
+ytilde = X @ beta
+
+

+Alternatively, you can use the least squares functionality in Numpy as +

+ + +

fit = np.linalg.lstsq(X, Energies, rcond =None)[0]
+ytildenp = np.dot(fit,X.T)
+
+

+And finally we plot our fit with and compare with data +

+ + +

Masses['Eapprox']  = ytilde
+# Generate a plot comparing the experimental with the fitted values values.
+fig, ax = plt.subplots()
+ax.set_xlabel(r'$A = N + Z$')
+ax.set_ylabel(r'$E_\mathrm{bind}\,/\mathrm{MeV}$')
+ax.plot(Masses['A'], Masses['Ebinding'], alpha=0.7, lw=2,
+            label='Ame2016')
+ax.plot(Masses['A'], Masses['Eapprox'], alpha=0.7, lw=2, c='m',
+            label='Fit')
+ax.legend()
+save_fig("Masses2016OLS")
+plt.show()
+
+

+









+ +

Adding error analysis and training set up

+ +

+We can easily test our fit by computing the \( R2 \) score that we discussed in connection with the functionality of _Scikit_Learn_ in the introductory slides. +Since we are not using _Scikit-Learn here we can define our own \( R2 \) function as +

+ + +

def R2(y_data, y_model):
+    return 1 - np.sum((y_data - y_model) ** 2) / np.sum((y_data - np.mean(y_model)) ** 2)
+
+

+and we would be using it as +

+ + +

print(R2(Energies,ytilde))
+
+

+We can easily add our MSE score as +

+ + +

def MSE(y_data,y_model):
+    n = np.size(y_model)
+    return np.sum((y_data-y_model)**2)/n
+
+print(MSE(Energies,ytilde))
+
+

+and finally the relative error as +

+ + +

def RelativeError(y_data,y_model):
+    return abs((y_data-y_model)/y_data)
+print(RelativeError(Energies, ytilde))
+
+

+









+ +

The \( \chi^2 \) function

+
+ +

+ +

+Normally, the response (dependent or outcome) variable \( y_i \) is the +outcome of a numerical experiment or another type of experiment and is +thus only an approximation to the true value. It is then always +accompanied by an error estimate, often limited to a statistical error +estimate given by the standard deviation discussed earlier. In the +discussion here we will treat \( y_i \) as our exact value for the +response variable. + +

+Introducing the standard deviation \( \sigma_i \) for each measurement +\( y_i \), we define now the \( \chi^2 \) function (omitting the \( 1/n \) term) +as + +$$ +\chi^2(\boldsymbol{\beta})=\frac{1}{n}\sum_{i=0}^{n-1}\frac{\left(y_i-\tilde{y}_i\right)^2}{\sigma_i^2}=\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{\tilde{y}}\right)^T\frac{1}{\boldsymbol{\Sigma^2}}\left(\boldsymbol{y}-\boldsymbol{\tilde{y}}\right)\right\}, +$$ + +where the matrix \( \boldsymbol{\Sigma} \) is a diagonal matrix with \( \sigma_i \) as matrix elements. + + +

+ + +

+









+ +

The \( \chi^2 \) function

+
+ +

+ +

+In order to find the parameters \( \beta_i \) we will then minimize the spread of \( \chi^2(\boldsymbol{\beta}) \) by requiring +$$ +\frac{\partial \chi^2(\boldsymbol{\beta})}{\partial \beta_j} = \frac{\partial }{\partial \beta_j}\left[ \frac{1}{n}\sum_{i=0}^{n-1}\left(\frac{y_i-\beta_0x_{i,0}-\beta_1x_{i,1}-\beta_2x_{i,2}-\dots-\beta_{n-1}x_{i,n-1}}{\sigma_i}\right)^2\right]=0, +$$ + +which results in +$$ +\frac{\partial \chi^2(\boldsymbol{\beta})}{\partial \beta_j} = -\frac{2}{n}\left[ \sum_{i=0}^{n-1}\frac{x_{ij}}{\sigma_i}\left(\frac{y_i-\beta_0x_{i,0}-\beta_1x_{i,1}-\beta_2x_{i,2}-\dots-\beta_{n-1}x_{i,n-1}}{\sigma_i}\right)\right]=0, +$$ + +or in a matrix-vector form as +$$ +\frac{\partial \chi^2(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}} = 0 = \boldsymbol{A}^T\left( \boldsymbol{b}-\boldsymbol{A}\boldsymbol{\beta}\right). +$$ + +where we have defined the matrix \( \boldsymbol{A} =\boldsymbol{X}/\boldsymbol{\Sigma} \) with matrix elements \( a_{ij} = x_{ij}/\sigma_i \) and the vector \( \boldsymbol{b} \) with elements \( b_i = y_i/\sigma_i \). +

+ + +

+









+ +

The \( \chi^2 \) function

+
+ +

+ +

+We can rewrite +$$ +\frac{\partial \chi^2(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}} = 0 = \boldsymbol{A}^T\left( \boldsymbol{b}-\boldsymbol{A}\boldsymbol{\beta}\right), +$$ + +as +$$ +\boldsymbol{A}^T\boldsymbol{b} = \boldsymbol{A}^T\boldsymbol{A}\boldsymbol{\beta}, +$$ + +and if the matrix \( \boldsymbol{A}^T\boldsymbol{A} \) is invertible we have the solution +$$ +\boldsymbol{\beta} =\left(\boldsymbol{A}^T\boldsymbol{A}\right)^{-1}\boldsymbol{A}^T\boldsymbol{b}. +$$ +

+ + +

+









+ +

The \( \chi^2 \) function

+
+ +

+ +

+If we then introduce the matrix +$$ +\boldsymbol{H} = \left(\boldsymbol{A}^T\boldsymbol{A}\right)^{-1}, +$$ + +we have then the following expression for the parameters \( \beta_j \) (the matrix elements of \( \boldsymbol{H} \) are \( h_{ij} \)) +$$ +\beta_j = \sum_{k=0}^{p-1}h_{jk}\sum_{i=0}^{n-1}\frac{y_i}{\sigma_i}\frac{x_{ik}}{\sigma_i} = \sum_{k=0}^{p-1}h_{jk}\sum_{i=0}^{n-1}b_ia_{ik} +$$ + +We state without proof the expression for the uncertainty in the parameters \( \beta_j \) as (we leave this as an exercise) +$$ +\sigma^2(\beta_j) = \sum_{i=0}^{n-1}\sigma_i^2\left( \frac{\partial \beta_j}{\partial y_i}\right)^2, +$$ + +resulting in +$$ +\sigma^2(\beta_j) = \left(\sum_{k=0}^{p-1}h_{jk}\sum_{i=0}^{n-1}a_{ik}\right)\left(\sum_{l=0}^{p-1}h_{jl}\sum_{m=0}^{n-1}a_{ml}\right) = h_{jj}! +$$ +

+ + +

+









+ +

The \( \chi^2 \) function

+
+ +

+The first step here is to approximate the function \( y \) with a first-order polynomial, that is we write +$$ +y=y(x) \rightarrow y(x_i) \approx \beta_0+\beta_1 x_i. +$$ + +By computing the derivatives of \( \chi^2 \) with respect to \( \beta_0 \) and \( \beta_1 \) show that these are given by +$$ +\frac{\partial \chi^2(\boldsymbol{\beta})}{\partial \beta_0} = -2\left[ \frac{1}{n}\sum_{i=0}^{n-1}\left(\frac{y_i-\beta_0-\beta_1x_{i}}{\sigma_i^2}\right)\right]=0, +$$ + +and +$$ +\frac{\partial \chi^2(\boldsymbol{\beta})}{\partial \beta_1} = -\frac{2}{n}\left[ \sum_{i=0}^{n-1}x_i\left(\frac{y_i-\beta_0-\beta_1x_{i}}{\sigma_i^2}\right)\right]=0. +$$ +

+ + +

+









+ +

The \( \chi^2 \) function

+
+ +

+ +

+For a linear fit (a first-order polynomial) we don't need to invert a matrix!! +Defining +$$ +\gamma = \sum_{i=0}^{n-1}\frac{1}{\sigma_i^2}, +$$ + + +$$ +\gamma_x = \sum_{i=0}^{n-1}\frac{x_{i}}{\sigma_i^2}, +$$ + + +$$ +\gamma_y = \sum_{i=0}^{n-1}\left(\frac{y_i}{\sigma_i^2}\right), +$$ + + +$$ +\gamma_{xx} = \sum_{i=0}^{n-1}\frac{x_ix_{i}}{\sigma_i^2}, +$$ + + +$$ +\gamma_{xy} = \sum_{i=0}^{n-1}\frac{y_ix_{i}}{\sigma_i^2}, +$$ + +

+we obtain + +$$ +\beta_0 = \frac{\gamma_{xx}\gamma_y-\gamma_x\gamma_y}{\gamma\gamma_{xx}-\gamma_x^2}, +$$ + + +$$ +\beta_1 = \frac{\gamma_{xy}\gamma-\gamma_x\gamma_y}{\gamma\gamma_{xx}-\gamma_x^2}. +$$ + +

+This approach (different linear and non-linear regression) suffers +often from both being underdetermined and overdetermined in the +unknown coefficients \( \beta_i \). A better approach is to use the +Singular Value Decomposition (SVD) method discussed below. Or using +Lasso and Ridge regression. See below. + + +

+ + +

+









+ +

Fitting an Equation of State for Dense Nuclear Matter

+ +

+Before we continue, let us introduce yet another example. We are going to fit the +nuclear equation of state using results from many-body calculations. +The equation of state we have made available here, as function of +density, has been derived using modern nucleon-nucleon potentials with +the addition of three-body +forces. This +time the file is presented as a standard csv file. + +

+The beginning of the Python code here is similar to what you have seen before, +with the same initializations and declarations. We use also pandas +again, rather extensively in order to organize our data. + +

+The difference now is that we use Scikit-Learn's regression tools +instead of our own matrix inversion implementation. Furthermore, we +sneak in Ridge regression (to be discussed below) which includes a +hyperparameter \( \lambda \), also to be explained below. + +

+









+ +

The code

+ +

+ + +

# Common imports
+import os
+import numpy as np
+import pandas as pd
+import matplotlib.pyplot as plt
+import matplotlib.pyplot as plt
+import sklearn.linear_model as skl
+from sklearn.metrics import mean_squared_error, r2_score, mean_absolute_error
+
+# Where to save the figures and data files
+PROJECT_ROOT_DIR = "Results"
+FIGURE_ID = "Results/FigureFiles"
+DATA_ID = "DataFiles/"
+
+if not os.path.exists(PROJECT_ROOT_DIR):
+    os.mkdir(PROJECT_ROOT_DIR)
+
+if not os.path.exists(FIGURE_ID):
+    os.makedirs(FIGURE_ID)
+
+if not os.path.exists(DATA_ID):
+    os.makedirs(DATA_ID)
+
+def image_path(fig_id):
+    return os.path.join(FIGURE_ID, fig_id)
+
+def data_path(dat_id):
+    return os.path.join(DATA_ID, dat_id)
+
+def save_fig(fig_id):
+    plt.savefig(image_path(fig_id) + ".png", format='png')
+
+infile = open(data_path("EoS.csv"),'r')
+
+# Read the EoS data as  csv file and organize the data into two arrays with density and energies
+EoS = pd.read_csv(infile, names=('Density', 'Energy'))
+EoS['Energy'] = pd.to_numeric(EoS['Energy'], errors='coerce')
+EoS = EoS.dropna()
+Energies = EoS['Energy']
+Density = EoS['Density']
+#  The design matrix now as function of various polytrops
+X = np.zeros((len(Density),4))
+X[:,3] = Density**(4.0/3.0)
+X[:,2] = Density
+X[:,1] = Density**(2.0/3.0)
+X[:,0] = 1
+
+# We use now Scikit-Learn's linear regressor and ridge regressor
+# OLS part
+clf = skl.LinearRegression().fit(X, Energies)
+ytilde = clf.predict(X)
+EoS['Eols']  = ytilde
+# The mean squared error                               
+print("Mean squared error: %.2f" % mean_squared_error(Energies, ytilde))
+# Explained variance score: 1 is perfect prediction                                 
+print('Variance score: %.2f' % r2_score(Energies, ytilde))
+# Mean absolute error                                                           
+print('Mean absolute error: %.2f' % mean_absolute_error(Energies, ytilde))
+print(clf.coef_, clf.intercept_)
+
+# The Ridge regression with a hyperparameter lambda = 0.1
+_lambda = 0.1
+clf_ridge = skl.Ridge(alpha=_lambda).fit(X, Energies)
+yridge = clf_ridge.predict(X)
+EoS['Eridge']  = yridge
+# The mean squared error                               
+print("Mean squared error: %.2f" % mean_squared_error(Energies, yridge))
+# Explained variance score: 1 is perfect prediction                                 
+print('Variance score: %.2f' % r2_score(Energies, yridge))
+# Mean absolute error                                                           
+print('Mean absolute error: %.2f' % mean_absolute_error(Energies, yridge))
+print(clf_ridge.coef_, clf_ridge.intercept_)
+
+fig, ax = plt.subplots()
+ax.set_xlabel(r'$\rho[\mathrm{fm}^{-3}]$')
+ax.set_ylabel(r'Energy per particle')
+ax.plot(EoS['Density'], EoS['Energy'], alpha=0.7, lw=2,
+            label='Theoretical data')
+ax.plot(EoS['Density'], EoS['Eols'], alpha=0.7, lw=2, c='m',
+            label='OLS')
+ax.plot(EoS['Density'], EoS['Eridge'], alpha=0.7, lw=2, c='g',
+            label='Ridge $\lambda = 0.1$')
+ax.legend()
+save_fig("EoSfitting")
+plt.show()
+
+

+The above simple polynomial in density \( \rho \) gives an excellent fit +to the data. Can you give an interpretation of the various powers of \( \rho \)? + +

+We note also that there is a small deviation between the +standard OLS and the Ridge regression at higher densities. We discuss this in more detail +below. + +

+









+ +

Splitting our Data in Training and Test data

+ +

+It is normal in essentially all Machine Learning studies to split the +data in a training set and a test set (sometimes also an additional +validation set). Scikit-Learn has an own function for this. There +is no explicit recipe for how much data should be included as training +data and say test data. An accepted rule of thumb is to use +approximately \( 2/3 \) to \( 4/5 \) of the data as training data. We will +postpone a discussion of this splitting to the end of these notes and +our discussion of the so-called bias-variance tradeoff. Here we +limit ourselves to repeat the above equation of state fitting example +but now splitting the data into a training set and a test set. + +

+ + +

import os
+import numpy as np
+import pandas as pd
+import matplotlib.pyplot as plt
+from sklearn.model_selection import train_test_split
+# Where to save the figures and data files
+PROJECT_ROOT_DIR = "Results"
+FIGURE_ID = "Results/FigureFiles"
+DATA_ID = "DataFiles/"
+
+if not os.path.exists(PROJECT_ROOT_DIR):
+    os.mkdir(PROJECT_ROOT_DIR)
+
+if not os.path.exists(FIGURE_ID):
+    os.makedirs(FIGURE_ID)
+
+if not os.path.exists(DATA_ID):
+    os.makedirs(DATA_ID)
+
+def image_path(fig_id):
+    return os.path.join(FIGURE_ID, fig_id)
+
+def data_path(dat_id):
+    return os.path.join(DATA_ID, dat_id)
+
+def save_fig(fig_id):
+    plt.savefig(image_path(fig_id) + ".png", format='png')
+
+def R2(y_data, y_model):
+    return 1 - np.sum((y_data - y_model) ** 2) / np.sum((y_data - np.mean(y_model)) ** 2)
+def MSE(y_data,y_model):
+    n = np.size(y_model)
+    return np.sum((y_data-y_model)**2)/n
+
+infile = open(data_path("EoS.csv"),'r')
+
+# Read the EoS data as  csv file and organized into two arrays with density and energies
+EoS = pd.read_csv(infile, names=('Density', 'Energy'))
+EoS['Energy'] = pd.to_numeric(EoS['Energy'], errors='coerce')
+EoS = EoS.dropna()
+Energies = EoS['Energy']
+Density = EoS['Density']
+#  The design matrix now as function of various polytrops
+X = np.zeros((len(Density),5))
+X[:,0] = 1
+X[:,1] = Density**(2.0/3.0)
+X[:,2] = Density
+X[:,3] = Density**(4.0/3.0)
+X[:,4] = Density**(5.0/3.0)
+# We split the data in test and training data
+X_train, X_test, y_train, y_test = train_test_split(X, Energies, test_size=0.2)
+# matrix inversion to find beta
+beta = np.linalg.inv(X_train.T.dot(X_train)).dot(X_train.T).dot(y_train)
+# and then make the prediction
+ytilde = X_train @ beta
+print("Training R2")
+print(R2(y_train,ytilde))
+print("Training MSE")
+print(MSE(y_train,ytilde))
+ypredict = X_test @ beta
+print("Test R2")
+print(R2(y_test,ypredict))
+print("Test MSE")
+print(MSE(y_test,ypredict))
+
+

+









+ +

The singular value decomposition

+ +

+

+ +

+ +

+The examples we have looked at so far are cases where we normally can +invert the matrix \( \boldsymbol{X}^T\boldsymbol{X} \). Using a polynomial expansion as we +did both for the masses and the fitting of the equation of state, +leads to row vectors of the design matrix which are essentially +orthogonal due to the polynomial character of our model. This may +however not the be case in general and a standard matrix inversion +algorithm based on say LU decomposition may lead to singularities. We will see an example of this below when we try to fit +the coupling constant of the widely used Ising model. +There is however a way to partially circumvent this problem and also gain some insight about the ordinary least squares approach. + +

+This is given by the Singular Value Decomposition algorithm, perhaps +the most powerful linear algebra algorithm. Let us look at a +different example where we may have problems with the standard matrix +inversion algorithm. Thereafter we dive into the math of the SVD. + + +

+ + +

+









+ +

The Ising model

+ +

+The one-dimensional Ising model with nearest neighbor interaction, no +external field and a constant coupling constant \( J \) is given by + +$$ +\begin{align} + H = -J \sum_{k}^L s_k s_{k + 1}, +\label{_auto1} +\end{align} +$$ + +

+where \( s_i \in \{-1, 1\} \) and \( s_{N + 1} = s_1 \). The number of spins +in the system is determined by \( L \). For the one-dimensional system +there is no phase transition. + +

+We will look at a system of \( L = 40 \) spins with a coupling constant of +\( J = 1 \). To get enough training data we will generate 10000 states +with their respective energies. + +

+ + +

import numpy as np
+import matplotlib.pyplot as plt
+from mpl_toolkits.axes_grid1 import make_axes_locatable
+import seaborn as sns
+import scipy.linalg as scl
+from sklearn.model_selection import train_test_split
+import tqdm
+sns.set(color_codes=True)
+cmap_args=dict(vmin=-1., vmax=1., cmap='seismic')
+
+L = 40
+n = int(1e4)
+
+spins = np.random.choice([-1, 1], size=(n, L))
+J = 1.0
+
+energies = np.zeros(n)
+
+for i in range(n):
+    energies[i] = - J * np.dot(spins[i], np.roll(spins[i], 1))
+
+

+Here we use ordinary least squares +regression to predict the energy for the nearest neighbor +one-dimensional Ising model on a ring, i.e., the endpoints wrap +around. We will use linear regression to fit a value for +the coupling constant to achieve this. + +

+









+ +

Reformulating the problem to suit regression

+ +

+A more general form for the one-dimensional Ising model is + +$$ +\begin{align} + H = - \sum_j^L \sum_k^L s_j s_k J_{jk}. +\label{_auto2} +\end{align} +$$ + +

+Here we allow for interactions beyond the nearest neighbors and a state dependent +coupling constant. This latter expression can be formulated as +a matrix-product +$$ +\begin{align} + \boldsymbol{H} = \boldsymbol{X} J, +\label{_auto3} +\end{align} +$$ + +

+where \( X_{jk} = s_j s_k \) and \( J \) is a matrix which consists of the +elements \( -J_{jk} \). This form of writing the energy fits perfectly +with the form utilized in linear regression, that is + +$$ +\begin{align} + \boldsymbol{y} = \boldsymbol{X}\boldsymbol{\beta} + \boldsymbol{\epsilon}, +\label{_auto4} +\end{align} +$$ + +

+We split the data in training and test data as discussed in the previous example + +

+ + +

X = np.zeros((n, L ** 2))
+for i in range(n):
+    X[i] = np.outer(spins[i], spins[i]).ravel()
+y = energies
+X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)
+
+

+









+ +

Linear regression

+ +

+In the ordinary least squares method we choose the cost function + +$$ +\begin{align} + C(\boldsymbol{X}, \boldsymbol{\beta})= \frac{1}{n}\left\{(\boldsymbol{X}\boldsymbol{\beta} - \boldsymbol{y})^T(\boldsymbol{X}\boldsymbol{\beta} - \boldsymbol{y})\right\}. +\label{_auto5} +\end{align} +$$ + +

+We then find the extremal point of \( C \) by taking the derivative with respect to \( \boldsymbol{\beta} \) as discussed above. +This yields the expression for \( \boldsymbol{\beta} \) to be + +$$ + \boldsymbol{\beta} = \frac{\boldsymbol{X}^T \boldsymbol{y}}{\boldsymbol{X}^T \boldsymbol{X}}, +$$ + +

+which immediately imposes some requirements on \( \boldsymbol{X} \) as there must exist +an inverse of \( \boldsymbol{X}^T \boldsymbol{X} \). If the expression we are modeling contains an +intercept, i.e., a constant term, we must make sure that the +first column of \( \boldsymbol{X} \) consists of \( 1 \). We do this here + +

+ + +

X_train_own = np.concatenate(
+    (np.ones(len(X_train))[:, np.newaxis], X_train),
+    axis=1
+)
+X_test_own = np.concatenate(
+    (np.ones(len(X_test))[:, np.newaxis], X_test),
+    axis=1
+)
+
+

+ + +

def ols_inv(x: np.ndarray, y: np.ndarray) -> np.ndarray:
+    return scl.inv(x.T @ x) @ (x.T @ y)
+beta = ols_inv(X_train_own, y_train)
+
+

+









+ +

Singular Value decomposition

+ +

+Doing the inversion directly turns out to be a bad idea since the matrix +\( \boldsymbol{X}^T\boldsymbol{X} \) is singular. An alternative approach is to use the singular +value decomposition. Using the definition of the Moore-Penrose +pseudoinverse we can write the equation for \( \boldsymbol{\beta} \) as + +$$ + \boldsymbol{\beta} = \boldsymbol{X}^{+}\boldsymbol{y}, +$$ + +

+where the pseudoinverse of \( \boldsymbol{X} \) is given by + +$$ + \boldsymbol{X}^{+} = \frac{\boldsymbol{X}^T}{\boldsymbol{X}^T\boldsymbol{X}}. +$$ + +

+Using singular value decomposition we can decompose the matrix \( \boldsymbol{X} = \boldsymbol{U}\boldsymbol{\Sigma} \boldsymbol{V}^T \), +where \( \boldsymbol{U} \) and \( \boldsymbol{V} \) are orthogonal(unitary) matrices and \( \boldsymbol{\Sigma} \) contains the singular values (more details below). +where \( X^{+} = V\Sigma^{+} U^T \). This reduces the equation for +\( \omega \) to +$$ +\begin{align} + \boldsymbol{\beta} = \boldsymbol{V}\boldsymbol{\Sigma}^{+} \boldsymbol{U}^T \boldsymbol{y}. +\label{_auto6} +\end{align} +$$ + +

+Note that solving this equation by actually doing the pseudoinverse +(which is what we will do) is not a good idea as this operation scales +as \( \mathcal{O}(n^3) \), where \( n \) is the number of elements in a +general matrix. Instead, doing \( QR \)-factorization and solving the +linear system as an equation would reduce this down to +\( \mathcal{O}(n^2) \) operations. + +

+ + +

def ols_svd(x: np.ndarray, y: np.ndarray) -> np.ndarray:
+    u, s, v = scl.svd(x)
+    return v.T @ scl.pinv(scl.diagsvd(s, u.shape[0], v.shape[0])) @ u.T @ y
+
+

+ + +

beta = ols_svd(X_train_own,y_train)
+
+

+When extracting the \( J \)-matrix we need to make sure that we remove the intercept, as is done here + +

+ + +

J = beta[1:].reshape(L, L)
+
+

+A way of looking at the coefficients in \( J \) is to plot the matrices as images. + +

+ + +

fig = plt.figure(figsize=(20, 14))
+im = plt.imshow(J, **cmap_args)
+plt.title("OLS", fontsize=18)
+plt.xticks(fontsize=18)
+plt.yticks(fontsize=18)
+cb = fig.colorbar(im)
+cb.ax.set_yticklabels(cb.ax.get_yticklabels(), fontsize=18)
+plt.show()
+
+

+It is interesting to note that OLS +considers both \( J_{j, j + 1} = -0.5 \) and \( J_{j, j - 1} = -0.5 \) as +valid matrix elements for \( J \). +In our discussion below on hyperparameters and Ridge and Lasso regression we will see that +this problem can be removed, partly and only with Lasso regression. + +

+In this case our matrix inversion was actually possible. The obvious question now is what is the mathematics behind the SVD? + +

+









+ +

Linear Regression Problems

+ +

+One of the typical problems we encounter with linear regression, in particular +when the matrix \( \boldsymbol{X} \) (our so-called design matrix) is high-dimensional, +are problems with near singular or singular matrices. The column vectors of \( \boldsymbol{X} \) +may be linearly dependent, normally referred to as super-collinearity. +This means that the matrix may be rank deficient and it is basically impossible to +to model the data using linear regression. As an example, consider the matrix +$$ +\begin{align*} +\mathbf{X} & = \left[ +\begin{array}{rrr} +1 & -1 & 2 +\\ +1 & 0 & 1 +\\ +1 & 2 & -1 +\\ +1 & 1 & 0 +\end{array} \right] +\end{align*} +$$ + +

+The columns of \( \boldsymbol{X} \) are linearly dependent. We see this easily since the +the first column is the row-wise sum of the other two columns. The rank (more correct, +the column rank) of a matrix is the dimension of the space spanned by the +column vectors. Hence, the rank of \( \mathbf{X} \) is equal to the number +of linearly independent columns. In this particular case the matrix has rank 2. + +

+Super-collinearity of an \( (n \times p) \)-dimensional design matrix \( \mathbf{X} \) implies +that the inverse of the matrix \( \boldsymbol{X}^T\boldsymbol{x} \) (the matrix we need to invert to solve the linear regression equations) is non-invertible. If we have a square matrix that does not have an inverse, we say this matrix singular. The example here demonstrates this +$$ +\begin{align*} +\boldsymbol{X} & = \left[ +\begin{array}{rr} +1 & -1 +\\ +1 & -1 +\end{array} \right]. +\end{align*} +$$ + +We see easily that \( \mbox{det}(\boldsymbol{X}) = x_{11} x_{22} - x_{12} x_{21} = 1 \times (-1) - 1 \times (-1) = 0 \). Hence, \( \mathbf{X} \) is singular and its inverse is undefined. +This is equivalent to saying that the matrix \( \boldsymbol{X} \) has at least an eigenvalue which is zero. + +

+









+ +

Fixing the singularity

+ +

+If our design matrix \( \boldsymbol{X} \) which enters the linear regression problem +$$ +\begin{align} +\boldsymbol{\beta} & = (\boldsymbol{X}^{T} \boldsymbol{X})^{-1} \boldsymbol{X}^{T} \boldsymbol{y}, +\label{_auto7} +\end{align} +$$ + +has linearly dependent column vectors, we will not be able to compute the inverse +of \( \boldsymbol{X}^T\boldsymbol{X} \) and we cannot find the parameters (estimators) \( \beta_i \). +The estimators are only well-defined if \( (\boldsymbol{X}^{T}\boldsymbol{X})^{-1} \) exits. +This is more likely to happen when the matrix \( \boldsymbol{X} \) is high-dimensional. In this case it is likely to encounter a situation where +the regression parameters \( \beta_i \) cannot be estimated. + +

+A cheap ad hoc approach is simply to add a small diagonal component to the matrix to invert, that is we change +$$ +\boldsymbol{X}^{T} \boldsymbol{X} \rightarrow \boldsymbol{X}^{T} \boldsymbol{X}+\lambda \boldsymbol{I}, +$$ + +where \( \boldsymbol{I} \) is the identity matrix. When we discuss Ridge regression this is actually what we end up evaluating. The parameter \( \lambda \) is called a hyperparameter. More about this later. + +

+









+ +

Basic math of the SVD

+ +

+From standard linear algebra we know that a square matrix \( \boldsymbol{X} \) can be diagonalized if and only it is +a so-called normal matrix, that is if \( \boldsymbol{X}\in {\mathbb{R}}^{n\times n} \) +we have \( \boldsymbol{X}\boldsymbol{X}^T=\boldsymbol{X}^T\boldsymbol{X} \) or if \( \boldsymbol{X}\in {\mathbb{C}}^{n\times n} \) we have \( \boldsymbol{X}\boldsymbol{X}^{\dagger}=\boldsymbol{X}^{\dagger}\boldsymbol{X} \). +The matrix has then a set of eigenpairs + +$$ +(\lambda_1,\boldsymbol{u}_1),\dots, (\lambda_n,\boldsymbol{u}_n), +$$ + +and the eigenvalues are given by the diagonal matrix +$$ +\boldsymbol{\Sigma}=\mathrm{Diag}(\lambda_1, \dots,\lambda_n). +$$ + +The matrix \( \boldsymbol{X} \) can be written in terms of an orthogonal/unitary transformation \( \boldsymbol{U} \) +$$ +\boldsymbol{X} = \boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{V}^T, +$$ + +with \( \boldsymbol{U}\boldsymbol{U}^T=\boldsymbol{I} \) or \( \boldsymbol{U}\boldsymbol{U}^{\dagger}=\boldsymbol{I} \). + +

+Not all square matrices are diagonalizable. A matrix like the one discussed above +$$ +\boldsymbol{X} = \begin{bmatrix} +1& -1 \\ +1& -1\\ +\end{bmatrix} +$$ + +is not diagonalizable, it is a so-called defective matrix. It is easy to see that the condition +\( \boldsymbol{X}\boldsymbol{X}^T=\boldsymbol{X}^T\boldsymbol{X} \) is not fulfilled. + +

+









+ +

The SVD, a Fantastic Algorithm

+ +

+However, and this is the strength of the SVD algorithm, any general +matrix \( \boldsymbol{X} \) can be decomposed in terms of a diagonal matrix and +two orthogonal/unitary matrices. The Singular Value Decompostion +(SVD) theorem +states that a general \( m\times n \) matrix \( \boldsymbol{X} \) can be written in +terms of a diagonal matrix \( \boldsymbol{\Sigma} \) of dimensionality \( n\times n \) +and two orthognal matrices \( \boldsymbol{U} \) and \( \boldsymbol{V} \), where the first has +dimensionality \( m \times m \) and the last dimensionality \( n\times n \). +We have then + +$$ +\boldsymbol{X} = \boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{V}^T +$$ + +

+As an example, the above defective matrix can be decomposed as + +$$ +\boldsymbol{X} = \frac{1}{\sqrt{2}}\begin{bmatrix} 1& 1 \\ 1& -1\\ \end{bmatrix} \begin{bmatrix} 2& 0 \\ 0& 0\\ \end{bmatrix} \frac{1}{\sqrt{2}}\begin{bmatrix} 1& -1 \\ 1& 1\\ \end{bmatrix}=\boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{V}^T, +$$ + +

+with eigenvalues \( \sigma_1=2 \) and \( \sigma_2=0 \). +The SVD exits always! + +

+









+ +

Another Example

+ +

+Consider the following matrix which can be SVD decomposed as + +$$ +\boldsymbol{X} = \frac{1}{15}\begin{bmatrix} 14 & 2\\ 4 & 22\\ 16 & 13\end{matrix}=\frac{1}{3}\begin{bmatrix} 1& 2 & 2 \\ 2& -1 & 1\\ 2 & 1& -2\end{bmatrix} \begin{bmatrix} 2& 0 \\ 0& 1\\ 0 & 0\end{bmatrix}\frac{1}{5}\begin{bmatrix} 3& 4 \\ 4& -3\end{bmatrix}=\boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{V}^T. +$$ + +

+This is a \( 3\times 2 \) matrix which is decomposed in terms of a +\( 3\times 3 \) matrix \( \boldsymbol{U} \), and a \( 2\times 2 \) matrix \( \boldsymbol{V} \). It is easy to see +that \( \boldsymbol{U} \) and \( \boldsymbol{V} \) are orthogonal (how?). + +

+And the SVD +decomposition (singular values) gives eigenvalues +\( \sigma_i\geq\sigma_{i+1} \) for all \( i \) and for dimensions larger than \( i=2 \), the +eigenvalues (singular values) are zero. + +

+In the general case, where our design matrix \( \boldsymbol{X} \) has dimension +\( n\times p \), the matrix is thus decomposed into an \( n\times n \) +orthogonal matrix \( \boldsymbol{U} \), a \( p\times p \) orthogonal matrix \( \boldsymbol{V} \) +and a diagonal matrix \( \boldsymbol{\Sigma} \) with \( r=\mathrm{min}(n,p) \) +singular values \( \sigma_i\lg 0 \) on the main diagonal and zeros filling +the rest of the matrix. There are at most \( p \) singular values +assuming that \( n > p \). In our regression examples for the nuclear +masses and the equation of state this is indeed the case, while for +the Ising model we have \( p > n \). These are often cases that lead to +near singular or singular matrices. + +

+The columns of \( \boldsymbol{U} \) are called the left singular vectors while the columns of \( \boldsymbol{V} \) are the right singular vectors. + +

+









+ +

Economy-size SVD

+ +

+If we assume that \( n > p \), then our matrix \( \boldsymbol{U} \) has dimension \( n +\times n \). The last \( n-p \) columns of \( \boldsymbol{U} \) become however +irrelevant in our calculations since they are multiplied with the +zeros in \( \boldsymbol{\Sigma} \). + +

+The economy-size decomposition removes extra rows or columns of zeros +from the diagonal matrix of singular values, \( \boldsymbol{\Sigma} \), along with the columns +in either \( \boldsymbol{U} \) or \( \boldsymbol{V} \) that multiply those zeros in the expression. +Removing these zeros and columns can improve execution time +and reduce storage requirements without compromising the accuracy of +the decomposition. + +

+If \( n > p \), we keep only the first \( p \) columns of \( \boldsymbol{U} \) and \( \boldsymbol{\Sigma} \) has dimension \( p\times p \). +If \( p > n \), then only the first \( n \) columns of \( \boldsymbol{V} \) are computed and \( \boldsymbol{\Sigma} \) has dimension \( n\times n \). +The \( n=p \) case is obvious, we retain the full SVD. +In general the economy-size SVD leads to less FLOPS and still conserving the desired accuracy. + +

+









+ +

Mathematical Properties

+ +

+There are several interesting mathematical properties which will be +relevant when we are going to discuss the differences between say +ordinary least squares (OLS) and Ridge regression. + +

+We have from OLS that the parameters of the linear approximation are given by +$$ +\boldsymbol{\tilde{y}} = \boldsymbol{X}\boldsymbol{\beta} = \boldsymbol{X}\left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}. +$$ + +

+The matrix to invert can be rewritten in terms of our SVD decomposition as + +$$ +\boldsymbol{X}^T\boldsymbol{X} = \boldsymbol{V}\boldsymbol{\Sigma}^T\boldsymbol{U}^T\boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{V}^T. +$$ + +Using the orthogonality properties of \( \boldsymbol{U} \) we have + +$$ +\boldsymbol{X}^T\boldsymbol{X} = \boldsymbol{V}\boldsymbol{\Sigma}^T\boldsymbol{\Sigma}\boldsymbol{V}^T = \boldsymbol{V}\boldsymbol{D}\boldsymbol{V}^T, +$$ + +with \( \boldsymbol{D} \) being a diagonal matrix with values along the diagonal given by the singular values squared. + +

+This means that +$$ +(\boldsymbol{X}^T\boldsymbol{X})\boldsymbol{V} = \boldsymbol{V}\boldsymbol{D}, +$$ + +that is the eigenvectors of \( (\boldsymbol{X}^T\boldsymbol{X}) \) are given by the columns of the right singular matrix of \( \boldsymbol{X} \) and the eigenvalues are the squared singular values. It is easy to show (show this) that +$$ +(\boldsymbol{X}\boldsymbol{X}^T)\boldsymbol{U} = \boldsymbol{U}\boldsymbol{D}, +$$ + +that is, the eigenvectors of \( (\boldsymbol{X}\boldsymbol{X})^T \) are the columns of the left singular matrix and the eigenvalues are the same. + +

+Going back to our OLS equation we have +$$ +\boldsymbol{X}\boldsymbol{\beta} = \boldsymbol{X}\left(\boldsymbol{V}\boldsymbol{D}\boldsymbol{V}^T \right)^{-1}\boldsymbol{X}^T\boldsymbol{y}=\boldsymbol{U\Sigma V^T}\left(\boldsymbol{V}\boldsymbol{D}\boldsymbol{V}^T \right)^{-1}(\boldsymbol{U\Sigma V^T})^T\boldsymbol{y}=\boldsymbol{U}\boldsymbol{U}^T\boldsymbol{y}. +$$ + +We will come back to this expression when we discuss Ridge regression. + +

+









+ +

Ridge and LASSO Regression

+ +

+Let us remind ourselves about the expression for the standard Mean Squared Error (MSE) which we used to define our cost function and the equations for the ordinary least squares (OLS) method, that is +our optimization problem is +$$ +{\displaystyle \min_{\boldsymbol{\beta}\in {\mathbb{R}}^{p}}}\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)^T\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)\right\}. +$$ + +or we can state it as +$$ +{\displaystyle \min_{\boldsymbol{\beta}\in +{\mathbb{R}}^{p}}}\frac{1}{n}\sum_{i=0}^{n-1}\left(y_i-\tilde{y}_i\right)^2=\frac{1}{n}\vert\vert \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\vert\vert_2^2, +$$ + +where we have used the definition of a norm-2 vector, that is +$$ +\vert\vert \boldsymbol{x}\vert\vert_2 = \sqrt{\sum_i x_i^2}. +$$ + +

+By minimizing the above equation with respect to the parameters +\( \boldsymbol{\beta} \) we could then obtain an analytical expression for the +parameters \( \boldsymbol{\beta} \). We can add a regularization parameter \( \lambda \) by +defining a new cost function to be optimized, that is + +$$ +{\displaystyle \min_{\boldsymbol{\beta}\in +{\mathbb{R}}^{p}}}\frac{1}{n}\vert\vert \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\vert\vert_2^2+\lambda\vert\vert \boldsymbol{\beta}\vert\vert_2^2 +$$ + +

+which leads to the Ridge regression minimization problem where we +require that \( \vert\vert \boldsymbol{\beta}\vert\vert_2^2\le t \), where \( t \) is +a finite number larger than zero. By defining + +$$ +C(\boldsymbol{X},\boldsymbol{\beta})=\frac{1}{n}\vert\vert \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\vert\vert_2^2+\lambda\vert\vert \boldsymbol{\beta}\vert\vert_1, +$$ + +

+we have a new optimization equation +$$ +{\displaystyle \min_{\boldsymbol{\beta}\in +{\mathbb{R}}^{p}}}\frac{1}{n}\vert\vert \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\vert\vert_2^2+\lambda\vert\vert \boldsymbol{\beta}\vert\vert_1 +$$ + +which leads to Lasso regression. Lasso stands for least absolute shrinkage and selection operator. + +

+Here we have defined the norm-1 as +$$ +\vert\vert \boldsymbol{x}\vert\vert_1 = \sum_i \vert x_i\vert. +$$ + +

+









+ +

More on Ridge Regression

+ +

+Using the matrix-vector expression for Ridge regression, + +$$ +C(\boldsymbol{X},\boldsymbol{\beta})=\frac{1}{n}\left\{(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})^T(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})\right\}+\lambda\boldsymbol{\beta}^T\boldsymbol{\beta}, +$$ + +

+by taking the derivatives with respect to \( \boldsymbol{\beta} \) we obtain then +a slightly modified matrix inversion problem which for finite values +of \( \lambda \) does not suffer from singularity problems. We obtain + +$$ +\boldsymbol{\beta}^{\mathrm{Ridge}} = \left(\boldsymbol{X}^T\boldsymbol{X}+\lambda\boldsymbol{I}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}, +$$ + +

+with \( \boldsymbol{I} \) being a \( p\times p \) identity matrix with the constraint that + +$$ +\sum_{i=0}^{p-1} \beta_i^2 \leq t, +$$ + +

+with \( t \) a finite positive number. + +

+We see that Ridge regression is nothing but the standard +OLS with a modified diagonal term added to \( \boldsymbol{X}^T\boldsymbol{X} \). The +consequences, in particular for our discussion of the bias-variance +are rather interesting. + +

+Furthermore, if we use the result above in terms of the SVD decomposition (our analysis was done for the OLS method), we had +$$ +(\boldsymbol{X}\boldsymbol{X}^T)\boldsymbol{U} = \boldsymbol{U}\boldsymbol{D}. +$$ + +

+We can analyse the OLS solutions in terms of the eigenvectors (the columns) of the right singular value matrix \( \boldsymbol{U} \) as +$$ +\boldsymbol{X}\boldsymbol{\beta} = \boldsymbol{X}\left(\boldsymbol{V}\boldsymbol{D}\boldsymbol{V}^T \right)^{-1}\boldsymbol{X}^T\boldsymbol{y}=\boldsymbol{U\Sigma V^T}\left(\boldsymbol{V}\boldsymbol{D}\boldsymbol{V}^T \right)^{-1}(\boldsymbol{U\Sigma V^T})^T\boldsymbol{y}=\boldsymbol{U}\boldsymbol{U}^T\boldsymbol{y} +$$ + +

+For Ridge regression this becomes + +$$ +\boldsymbol{X}\boldsymbol{\beta}^{\mathrm{Ridge}} = \boldsymbol{U\Sigma V^T}\left(\boldsymbol{V}\boldsymbol{D}\boldsymbol{V}^T+\lambda\boldsymbol{I} \right)^{-1}(\boldsymbol{U\Sigma V^T})^T\boldsymbol{y}=\sum_{j=0}^{p-1}\boldsymbol{u}_j\boldsymbol{u}_j^T\frac{\sigma_j^2}{\sigma_j^2+\lambda}\boldsymbol{y}, +$$ + +

+with the vectors \( \boldsymbol{u}_j \) being the columns of \( \boldsymbol{U} \). + +

+









+ +

Interpreting the Ridge results

+ +

+Since \( \lambda \geq 0 \), it means that compared to OLS, we have + +$$ +\frac{\sigma_j^2}{\sigma_j^2+\lambda} \leq 1. +$$ + +

+Ridge regression finds the coordinates of \( \boldsymbol{y} \) with respect to the +orthonormal basis \( \boldsymbol{U} \), it then shrinks the coordinates by +\( \frac{\sigma_j^2}{\sigma_j^2+\lambda} \). Recall that the SVD has +eigenvalues ordered in a descending way, that is \( \sigma_i \geq +\sigma_{i+1} \). + +

+For small eigenvalues \( \sigma_i \) it means that their contributions become less important, a fact which can be used to reduce the number of degrees of freedom. +Actually, calculating the variance of \( \boldsymbol{X}\boldsymbol{v}_j \) shows that this quantity is equal to \( \sigma_j^2/n \). +With a parameter \( \lambda \) we can thus shrink the role of specific parameters. + +

+









+ +

More interpretations

+ +

+For the sake of simplicity, let us assume that the design matrix is orthonormal, that is + +$$ +\boldsymbol{X}^T\boldsymbol{X}=(\boldsymbol{X}^T\boldsymbol{X})^{-1} =\boldsymbol{I}. +$$ + +

+In this case the standard OLS results in +$$ +\boldsymbol{\beta}^{\mathrm{OLS}} = \boldsymbol{X}^T\boldsymbol{y}=\sum_{i=0}^{p-1}\boldsymbol{u}_j\boldsymbol{u}_j^T\boldsymbol{y}, +$$ + +

+and + +$$ +\boldsymbol{\beta}^{\mathrm{Ridge}} = \left(\boldsymbol{I}+\lambda\boldsymbol{I}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}=\left(1+\lambda\right)^{-1}\boldsymbol{\beta}^{\mathrm{OLS}}, +$$ + +

+that is the Ridge estimator scales the OLS estimator by the inverse of a factor \( 1+\lambda \), and +the Ridge estimator converges to zero when the hyperparameter goes to +infinity. + +

+We will come back to more interpreations after we have gone through some of the statistical analysis part. + +

+For more discussions of Ridge and Lasso regression, Wessel van Wieringen's article is highly recommended. +Similarly, Mehta et al's article is also recommended. + +

+









+ +

Where are we going?

+ +

+Before we proceed, we need to rethink what we have been doing. In our +eager to fit the data, we have omitted several important elements in +our regression analysis. In what follows we will + +

    +
  1. look at statistical properties, including a discussion of mean values, variance and the so-called bias-variance tradeoff
    2. +
  2. introduce resampling techniques like cross-validation, bootstrapping and jackknife and more
    3. +
+ +This will allow us to link the standard linear algebra methods we have discussed above to a statistical interpretation of the methods. + +

+









+ +

Resampling methods

+
+ +

+Resampling methods are an indispensable tool in modern +statistics. They involve repeatedly drawing samples from a training +set and refitting a model of interest on each sample in order to +obtain additional information about the fitted model. For example, in +order to estimate the variability of a linear regression fit, we can +repeatedly draw different samples from the training data, fit a linear +regression to each new sample, and then examine the extent to which +the resulting fits differ. Such an approach may allow us to obtain +information that would not be available from fitting the model only +once using the original training sample. +

+ + +

+









+ +

Resampling approaches can be computationally expensive

+
+ +

+ +

+Resampling approaches can be computationally expensive, because they +involve fitting the same statistical method multiple times using +different subsets of the training data. However, due to recent +advances in computing power, the computational requirements of +resampling methods generally are not prohibitive. In this chapter, we +discuss two of the most commonly used resampling methods, +cross-validation and the bootstrap. Both methods are important tools +in the practical application of many statistical learning +procedures. For example, cross-validation can be used to estimate the +test error associated with a given statistical learning method in +order to evaluate its performance, or to select the appropriate level +of flexibility. The process of evaluating a model’s performance is +known as model assessment, whereas the process of selecting the proper +level of flexibility for a model is known as model selection. The +bootstrap is widely used. + + +

+ + +

+









+ +

Why resampling methods ?

+
+Statistical analysis. +

+ +

    +
  • Our simulations can be treated as computer experiments. This is particularly the case for Monte Carlo methods
    • +
  • The results can be analysed with the same statistical tools as we would use analysing experimental data.
    • +
  • As in all experiments, we are looking for expectation values and an estimate of how accurate they are, i.e., possible sources for errors.
    • +
+
+ + +

+









+ +

Statistical analysis

+
+ +

+ +

    +
  • As in other experiments, many numerical experiments have two classes of errors:
    • + +
      +
    • Statistical errors
      • +
    • Systematical errors
      • +
    + +
  • Statistical errors can be estimated using standard tools from statistics
    • +
  • Systematical errors are method specific and must be treated differently from case to case.
    • +
+
+ + +

+









+ +

Statistics

+
+ +

+The probability distribution function (PDF) is a function +\( p(x) \) on the domain which, in the discrete case, gives us the +probability or relative frequency with which these values of \( X \) occur: +$$ +p(x) = \mathrm{prob}(X=x) +$$ + +In the continuous case, the PDF does not directly depict the +actual probability. Instead we define the probability for the +stochastic variable to assume any value on an infinitesimal interval +around \( x \) to be \( p(x)dx \). The continuous function \( p(x) \) then gives us +the density of the probability rather than the probability +itself. The probability for a stochastic variable to assume any value +on a non-infinitesimal interval \( [a,\,b] \) is then just the integral: +$$ +\mathrm{prob}(a\leq X\leq b) = \int_a^b p(x)dx +$$ + +Qualitatively speaking, a stochastic variable represents the values of +numbers chosen as if by chance from some specified PDF so that the +selection of a large set of these numbers reproduces this PDF. +

+ + +

+









+ +

Statistics, moments

+
+ +

+A particularly useful class of special expectation values are the +moments. The \( n \)-th moment of the PDF \( p \) is defined as +follows: +$$ +\langle x^n\rangle \equiv \int\! x^n p(x)\,dx +$$ + +The zero-th moment \( \langle 1\rangle \) is just the normalization condition of +\( p \). The first moment, \( \langle x\rangle \), is called the mean of \( p \) +and often denoted by the letter \( \mu \): +$$ +\langle x\rangle = \mu \equiv \int\! x p(x)\,dx +$$ +

+ + +

+









+ +

Statistics, central moments

+
+ +

+A special version of the moments is the set of central moments, +the n-th central moment defined as: +$$ +\langle (x-\langle x \rangle )^n\rangle \equiv \int\! (x-\langle x\rangle)^n p(x)\,dx +$$ + +The zero-th and first central moments are both trivial, equal \( 1 \) and +\( 0 \), respectively. But the second central moment, known as the +variance of \( p \), is of particular interest. For the stochastic +variable \( X \), the variance is denoted as \( \sigma^2_X \) or \( \mathrm{var}(X) \): +$$ +\begin{align} +\sigma^2_X\ \ =\ \ \mathrm{var}(X) & = \langle (x-\langle x\rangle)^2\rangle = +\int\! (x-\langle x\rangle)^2 p(x)\,dx +\label{_auto8}\\ +& = \int\! \left(x^2 - 2 x \langle x\rangle^{2} + + \langle x\rangle^2\right)p(x)\,dx +\label{_auto9}\\ +& = \langle x^2\rangle - 2 \langle x\rangle\langle x\rangle + \langle x\rangle^2 +\label{_auto10}\\ +& = \langle x^2\rangle - \langle x\rangle^2 +\label{_auto11} +\end{align} +$$ + +The square root of the variance, \( \sigma =\sqrt{\langle (x-\langle x\rangle)^2\rangle} \) is called the standard deviation of \( p \). It is clearly just the RMS (root-mean-square) +value of the deviation of the PDF from its mean value, interpreted +qualitatively as the spread of \( p \) around its mean. +

+ + +

+









+ +

Statistics, covariance

+
+ +

+Another important quantity is the so called covariance, a variant of +the above defined variance. Consider again the set \( \{X_i\} \) of \( n \) +stochastic variables (not necessarily uncorrelated) with the +multivariate PDF \( P(x_1,\dots,x_n) \). The covariance of two +of the stochastic variables, \( X_i \) and \( X_j \), is defined as follows: +$$ +\begin{align} +\mathrm{cov}(X_i,\,X_j) &\equiv \langle (x_i-\langle x_i\rangle)(x_j-\langle x_j\rangle)\rangle +\nonumber\\ +&= +\int\!\cdots\!\int\!(x_i-\langle x_i \rangle)(x_j-\langle x_j \rangle)\, +P(x_1,\dots,x_n)\,dx_1\dots dx_n +\label{eq:def_covariance} +\end{align} +$$ + +with +$$ +\langle x_i\rangle = +\int\!\cdots\!\int\!x_i\,P(x_1,\dots,x_n)\,dx_1\dots dx_n +$$ +

+ + +

+









+ +

Statistics, more covariance

+
+ +

+If we consider the above covariance as a matrix \( C_{ij}=\mathrm{cov}(X_i,\,X_j) \), then the diagonal elements are just the familiar +variances, \( C_{ii} = \mathrm{cov}(X_i,\,X_i) = \mathrm{var}(X_i) \). It turns out that +all the off-diagonal elements are zero if the stochastic variables are +uncorrelated. This is easy to show, keeping in mind the linearity of +the expectation value. Consider the stochastic variables \( X_i \) and +\( X_j \), (\( i\neq j \)): +$$ +\begin{align} +\mathrm{cov}(X_i,\,X_j) &= \langle(x_i-\langle x_i\rangle)(x_j-\langle x_j\rangle)\rangle +\label{_auto12}\\ +&=\langle x_i x_j - x_i\langle x_j\rangle - \langle x_i\rangle x_j + \langle x_i\rangle\langle x_j\rangle\rangle +\label{_auto13}\\ +&=\langle x_i x_j\rangle - \langle x_i\langle x_j\rangle\rangle - \langle \langle x_i\rangle x_j\rangle + +\langle \langle x_i\rangle\langle x_j\rangle\rangle +\label{_auto14}\\ +&=\langle x_i x_j\rangle - \langle x_i\rangle\langle x_j\rangle - \langle x_i\rangle\langle x_j\rangle + +\langle x_i\rangle\langle x_j\rangle +\label{_auto15}\\ +&=\langle x_i x_j\rangle - \langle x_i\rangle\langle x_j\rangle +\label{_auto16} +\end{align} +$$ +

+ + +

+









+ +

Statistics, independent variables

+
+ +

+If \( X_i \) and \( X_j \) are independent, we get +\( \langle x_i x_j\rangle =\langle x_i\rangle\langle x_j\rangle \), resulting in \( \mathrm{cov}(X_i, X_j) = 0\ \ (i\neq j) \). + +

+Also useful for us is the covariance of linear combinations of +stochastic variables. Let \( \{X_i\} \) and \( \{Y_i\} \) be two sets of +stochastic variables. Let also \( \{a_i\} \) and \( \{b_i\} \) be two sets of +scalars. Consider the linear combination: +$$ +U = \sum_i a_i X_i \qquad V = \sum_j b_j Y_j +$$ + +By the linearity of the expectation value +$$ +\mathrm{cov}(U, V) = \sum_{i,j}a_i b_j \mathrm{cov}(X_i, Y_j) +$$ +

+ + +

+









+ +

Statistics, more variance

+
+ +

+Now, since the variance is just \( \mathrm{var}(X_i) = \mathrm{cov}(X_i, X_i) \), we get +the variance of the linear combination \( U = \sum_i a_i X_i \): +$$ +\begin{equation} +\mathrm{var}(U) = \sum_{i,j}a_i a_j \mathrm{cov}(X_i, X_j) +\label{eq:variance_linear_combination} +\end{equation} +$$ + +And in the special case when the stochastic variables are +uncorrelated, the off-diagonal elements of the covariance are as we +know zero, resulting in: +$$ +\mathrm{var}(U) = \sum_i a_i^2 \mathrm{cov}(X_i, X_i) = \sum_i a_i^2 \mathrm{var}(X_i) +$$ + +$$ +\mathrm{var}(\sum_i a_i X_i) = \sum_i a_i^2 \mathrm{var}(X_i) +$$ + +which will become very useful in our study of the error in the mean +value of a set of measurements. +

+ + +

+









+ +

Statistics and stochastic processes

+
+ +

+A stochastic process is a process that produces sequentially a +chain of values: +$$ +\{x_1, x_2,\dots\,x_k,\dots\}. +$$ + +We will call these +values our measurements and the entire set as our measured +sample. The action of measuring all the elements of a sample +we will call a stochastic experiment since, operationally, +they are often associated with results of empirical observation of +some physical or mathematical phenomena; precisely an experiment. We +assume that these values are distributed according to some +PDF \( p_X^{\phantom X}(x) \), where \( X \) is just the formal symbol for the +stochastic variable whose PDF is \( p_X^{\phantom X}(x) \). Instead of +trying to determine the full distribution \( p \) we are often only +interested in finding the few lowest moments, like the mean +\( \mu_X^{\phantom X} \) and the variance \( \sigma_X^{\phantom X} \). +

+ + +

+ + +

Statistics and sample variables

+
+ +

+In practical situations a sample is always of finite size. Let that +size be \( n \). The expectation value of a sample, the sample mean, is then defined as follows: +$$ +\bar{x}_n \equiv \frac{1}{n}\sum_{k=1}^n x_k +$$ + +The sample variance is: +$$ +\mathrm{var}(x) \equiv \frac{1}{n}\sum_{k=1}^n (x_k - \bar{x}_n)^2 +$$ + +its square root being the standard deviation of the sample. The +sample covariance is: +$$ +\mathrm{cov}(x)\equiv\frac{1}{n}\sum_{kl}(x_k - \bar{x}_n)(x_l - \bar{x}_n) +$$ +

+ + +

+









+ +

Statistics, sample variance and covariance

+
+ +

+Note that the sample variance is the sample covariance without the +cross terms. In a similar manner as the covariance in Eq. \eqref{eq:def_covariance} is a measure of the correlation between +two stochastic variables, the above defined sample covariance is a +measure of the sequential correlation between succeeding measurements +of a sample. + +

+These quantities, being known experimental values, differ +significantly from and must not be confused with the similarly named +quantities for stochastic variables, mean \( \mu_X \), variance \( \mathrm{var}(X) \) +and covariance \( \mathrm{cov}(X,Y) \). +

+ + +

+









+ +

Statistics, law of large numbers

+
+ +

+The law of large numbers +states that as the size of our sample grows to infinity, the sample +mean approaches the true mean \( \mu_X^{\phantom X} \) of the chosen PDF: +$$ +\lim_{n\to\infty}\bar{x}_n = \mu_X^{\phantom X} +$$ + +The sample mean \( \bar{x}_n \) works therefore as an estimate of the true +mean \( \mu_X^{\phantom X} \). + +

+What we need to find out is how good an approximation \( \bar{x}_n \) is to +\( \mu_X^{\phantom X} \). In any stochastic measurement, an estimated +mean is of no use to us without a measure of its error. A quantity +that tells us how well we can reproduce it in another experiment. We +are therefore interested in the PDF of the sample mean itself. Its +standard deviation will be a measure of the spread of sample means, +and we will simply call it the error of the sample mean, or +just sample error, and denote it by \( \mathrm{err}_X^{\phantom X} \). In +practice, we will only be able to produce an estimate of the +sample error since the exact value would require the knowledge of the +true PDFs behind, which we usually do not have. +

+ + +

+









+ +

Statistics, more on sample error

+
+ +

+Let us first take a look at what happens to the sample error as the +size of the sample grows. In a sample, each of the measurements \( x_i \) +can be associated with its own stochastic variable \( X_i \). The +stochastic variable \( \overline X_n \) for the sample mean \( \bar{x}_n \) is +then just a linear combination, already familiar to us: +$$ +\overline X_n = \frac{1}{n}\sum_{i=1}^n X_i +$$ + +All the coefficients are just equal \( 1/n \). The PDF of \( \overline X_n \), +denoted by \( p_{\overline X_n}(x) \) is the desired PDF of the sample +means. +

+ + +

+









+ +

Statistics

+
+ +

+The probability density of obtaining a sample mean \( \bar x_n \) +is the product of probabilities of obtaining arbitrary values \( x_1, +x_2,\dots,x_n \) with the constraint that the mean of the set \( \{x_i\} \) +is \( \bar x_n \): +$$ +p_{\overline X_n}(x) = \int p_X^{\phantom X}(x_1)\cdots +\int p_X^{\phantom X}(x_n)\ +\delta\!\left(x - \frac{x_1+x_2+\dots+x_n}{n}\right)dx_n \cdots dx_1 +$$ + +And in particular we are interested in its variance \( \mathrm{var}(\overline X_n) \). +

+ + +

+









+ +

Statistics, central limit theorem

+
+ +

+It is generally not possible to express \( p_{\overline X_n}(x) \) in a +closed form given an arbitrary PDF \( p_X^{\phantom X} \) and a number +\( n \). But for the limit \( n\to\infty \) it is possible to make an +approximation. The very important result is called the central limit theorem. It tells us that as \( n \) goes to infinity, +\( p_{\overline X_n}(x) \) approaches a Gaussian distribution whose mean +and variance equal the true mean and variance, \( \mu_{X}^{\phantom X} \) +and \( \sigma_{X}^{2} \), respectively: +$$ +\begin{equation} +\lim_{n\to\infty} p_{\overline X_n}(x) = +\left(\frac{n}{2\pi\mathrm{var}(X)}\right)^{1/2} +e^{-\frac{n(x-\bar x_n)^2}{2\mathrm{var}(X)}} +\label{eq:central_limit_gaussian} +\end{equation} +$$ +

+ + +

+









+ +

Statistics, more technicalities

+
+ +

+The desired variance +\( \mathrm{var}(\overline X_n) \), i.e. the sample error squared +\( \mathrm{err}_X^2 \), is given by: +$$ +\begin{equation} +\mathrm{err}_X^2 = \mathrm{var}(\overline X_n) = \frac{1}{n^2} +\sum_{ij} \mathrm{cov}(X_i, X_j) +\label{eq:error_exact} +\end{equation} +$$ + +We see now that in order to calculate the exact error of the sample +with the above expression, we would need the true means +\( \mu_{X_i}^{\phantom X} \) of the stochastic variables \( X_i \). To +calculate these requires that we know the true multivariate PDF of all +the \( X_i \). But this PDF is unknown to us, we have only got the measurements of +one sample. The best we can do is to let the sample itself be an +estimate of the PDF of each of the \( X_i \), estimating all properties of +\( X_i \) through the measurements of the sample. +

+ + +

+









+ +

Statistics

+
+ +

+Our estimate of \( \mu_{X_i}^{\phantom X} \) is then the sample mean \( \bar x \) +itself, in accordance with the the central limit theorem: +$$ +\mu_{X_i}^{\phantom X} = \langle x_i\rangle \approx \frac{1}{n}\sum_{k=1}^n x_k = \bar x +$$ + +Using \( \bar x \) in place of \( \mu_{X_i}^{\phantom X} \) we can give an +estimate of the covariance in Eq. \eqref{eq:error_exact} +$$ +\mathrm{cov}(X_i, X_j) = \langle (x_i-\langle x_i\rangle)(x_j-\langle x_j\rangle)\rangle +\approx\langle (x_i - \bar x)(x_j - \bar{x})\rangle, +$$ + +resulting in +$$ +\frac{1}{n} \sum_{l}^n \left(\frac{1}{n}\sum_{k}^n (x_k -\bar x_n)(x_l - \bar x_n)\right)=\frac{1}{n}\frac{1}{n} \sum_{kl} (x_k -\bar x_n)(x_l - \bar x_n)=\frac{1}{n}\mathrm{cov}(x) +$$ +

+ + +

+









+ +

Statistics and sample variance

+
+ +

+By the same procedure we can use the sample variance as an +estimate of the variance of any of the stochastic variables \( X_i \) +$$ +\mathrm{var}(X_i)=\langle x_i - \langle x_i\rangle\rangle \approx \langle x_i - \bar x_n\rangle\nonumber, +$$ + +which is approximated as +$$ +\begin{equation} +\mathrm{var}(X_i)\approx \frac{1}{n}\sum_{k=1}^n (x_k - \bar x_n)=\mathrm{var}(x) +\label{eq:var_estimate_i_think} +\end{equation} +$$ + +

+Now we can calculate an estimate of the error +\( \mathrm{err}_X^{\phantom X} \) of the sample mean \( \bar x_n \): +$$ +\begin{align} +\mathrm{err}_X^2 +&=\frac{1}{n^2}\sum_{ij} \mathrm{cov}(X_i, X_j) \nonumber \\ +&\approx&\frac{1}{n^2}\sum_{ij}\frac{1}{n}\mathrm{cov}(x) =\frac{1}{n^2}n^2\frac{1}{n}\mathrm{cov}(x)\nonumber\\ +&=\frac{1}{n}\mathrm{cov}(x) +\label{eq:error_estimate} +\end{align} +$$ + +which is nothing but the sample covariance divided by the number of +measurements in the sample. +

+ + +

+









+ +

Statistics, uncorrelated results

+
+ +

+ +

+In the special case that the measurements of the sample are +uncorrelated (equivalently the stochastic variables \( X_i \) are +uncorrelated) we have that the off-diagonal elements of the covariance +are zero. This gives the following estimate of the sample error: +$$ +\mathrm{err}_X^2=\frac{1}{n^2}\sum_{ij} \mathrm{cov}(X_i, X_j) = +\frac{1}{n^2} \sum_i \mathrm{var}(X_i), +$$ + +resulting in +$$ +\begin{equation} +\mathrm{err}_X^2\approx \frac{1}{n^2} \sum_i \mathrm{var}(x)= \frac{1}{n}\mathrm{var}(x) +\label{eq:error_estimate_uncorrel} +\end{equation} +$$ + +where in the second step we have used Eq. \eqref{eq:var_estimate_i_think}. +The error of the sample is then just its standard deviation divided by +the square root of the number of measurements the sample contains. +This is a very useful formula which is easy to compute. It acts as a +first approximation to the error, but in numerical experiments, we +cannot overlook the always present correlations. +

+ + +

+









+ +

Statistics, computations

+
+ +

+For computational purposes one usually splits up the estimate of +\( \mathrm{err}_X^2 \), given by Eq. \eqref{eq:error_estimate}, into two +parts +$$ +\mathrm{err}_X^2 = \frac{1}{n}\mathrm{var}(x) + \frac{1}{n}(\mathrm{cov}(x)-\mathrm{var}(x)), +$$ + +which equals +$$ +\begin{equation} +\frac{1}{n^2}\sum_{k=1}^n (x_k - \bar x_n)^2 +\frac{2}{n^2}\sum_{k < l} (x_k - \bar x_n)(x_l - \bar x_n) +\label{eq:error_estimate_split_up} +\end{equation} +$$ + +The first term is the same as the error in the uncorrelated case, +Eq. \eqref{eq:error_estimate_uncorrel}. This means that the second +term accounts for the error correction due to correlation between the +measurements. For uncorrelated measurements this second term is zero. +

+ + +

+









+ +

Statistics, more on computations of errors

+
+ +

+Computationally the uncorrelated first term is much easier to treat +efficiently than the second. +$$ +\mathrm{var}(x) = \frac{1}{n}\sum_{k=1}^n (x_k - \bar x_n)^2 = +\left(\frac{1}{n}\sum_{k=1}^n x_k^2\right) - \bar x_n^2 +$$ + +We just accumulate separately the values \( x^2 \) and \( x \) for every +measurement \( x \) we receive. The correlation term, though, has to be +calculated at the end of the experiment since we need all the +measurements to calculate the cross terms. Therefore, all measurements +have to be stored throughout the experiment. +

+ + +

+









+ +

Statistics, wrapping up 1

+
+ +

+Let us analyze the problem by splitting up the correlation term into +partial sums of the form: +$$ +f_d = \frac{1}{n-d}\sum_{k=1}^{n-d}(x_k - \bar x_n)(x_{k+d} - \bar x_n) +$$ + +The correlation term of the error can now be rewritten in terms of +\( f_d \) +$$ +\frac{2}{n}\sum_{k < l} (x_k - \bar x_n)(x_l - \bar x_n) = +2\sum_{d=1}^{n-1} f_d +$$ + +The value of \( f_d \) reflects the correlation between measurements +separated by the distance \( d \) in the sample samples. Notice that for +\( d=0 \), \( f \) is just the sample variance, \( \mathrm{var}(x) \). If we divide \( f_d \) +by \( \mathrm{var}(x) \), we arrive at the so called autocorrelation function +$$ +\kappa_d = \frac{f_d}{\mathrm{var}(x)} +$$ + +which gives us a useful measure of pairwise correlations +starting always at \( 1 \) for \( d=0 \). +

+ + +

+









+ +

Statistics, final expression

+
+ +

+The sample error (see eq. \eqref{eq:error_estimate_split_up}) can now be +written in terms of the autocorrelation function: +$$ +\begin{align} +\mathrm{err}_X^2 &= +\frac{1}{n}\mathrm{var}(x)+\frac{2}{n}\cdot\mathrm{var}(x)\sum_{d=1}^{n-1} +\frac{f_d}{\mathrm{var}(x)}\nonumber\\ &=& +\left(1+2\sum_{d=1}^{n-1}\kappa_d\right)\frac{1}{n}\mathrm{var}(x)\nonumber\\ +&=\frac{\tau}{n}\cdot\mathrm{var}(x) +\label{eq:error_estimate_corr_time} +\end{align} +$$ + +and we see that \( \mathrm{err}_X \) can be expressed in terms the +uncorrelated sample variance times a correction factor \( \tau \) which +accounts for the correlation between measurements. We call this +correction factor the autocorrelation time: +$$ +\begin{equation} +\tau = 1+2\sum_{d=1}^{n-1}\kappa_d +\label{eq:autocorrelation_time} +\end{equation} +$$ +

+ + +

+









+ +

Statistics, effective number of correlations

+
+ +

+For a correlation free experiment, \( \tau \) +equals 1. From the point of view of +eq. \eqref{eq:error_estimate_corr_time} we can interpret a sequential +correlation as an effective reduction of the number of measurements by +a factor \( \tau \). The effective number of measurements becomes: +$$ +n_\mathrm{eff} = \frac{n}{\tau} +$$ + +To neglect the autocorrelation time \( \tau \) will always cause our +simple uncorrelated estimate of \( \mathrm{err}_X^2\approx \mathrm{var}(x)/n \) to +be less than the true sample error. The estimate of the error will be +too good. On the other hand, the calculation of the full +autocorrelation time poses an efficiency problem if the set of +measurements is very large. +

+ + +

+ + +

Linking the regression analysis with a statistical interpretation

+ +

+Finally, we are going to discuss several statistical properties which can be obtained in terms of analytical expressions. +The +advantage of doing linear regression is that we actually end up with +analytical expressions for several statistical quantities. +Standard least squares and Ridge regression allow us to +derive quantities like the variance and other expectation values in a +rather straightforward way. + +

+It is assumed that \( \varepsilon_i +\sim \mathcal{N}(0, \sigma^2) \) and the \( \varepsilon_{i} \) are +independent, i.e.: +$$ +\begin{align*} +\mbox{Cov}(\varepsilon_{i_1}, +\varepsilon_{i_2}) & = \left\{ \begin{array}{lcc} \sigma^2 & \mbox{if} +& i_1 = i_2, \\ 0 & \mbox{if} & i_1 \not= i_2. \end{array} \right. +\end{align*} +$$ + +The randomness of \( \varepsilon_i \) implies that +\( \mathbf{y}_i \) is also a random variable. In particular, +\( \mathbf{y}_i \) is normally distributed, because \( \varepsilon_i \sim +\mathcal{N}(0, \sigma^2) \) and \( \mathbf{X}_{i,\ast} \, \boldsymbol{\beta} \) is a +non-random scalar. To specify the parameters of the distribution of +\( \mathbf{y}_i \) we need to calculate its first two moments. + +

+Recall that \( \boldsymbol{X} \) is a matrix of dimensionality \( n\times p \). The +notation above \( \mathbf{X}_{i,\ast} \) means that we are looking at the +row number \( i \) and perform a sum over all values \( p \). + +

+









+ +

Assumptions made

+ +

+The assumption we have made here can be summarized as (and this is going to useful when we discuss the bias-variance trade off) +that there exists a function \( f(\boldsymbol{x}) \) and a normal distributed error \( \boldsymbol{\varepsilon}\sim \mathcal{N}(0, \sigma^2) \) +which describes our data +$$ +\boldsymbol{y} = f(\boldsymbol{x})+\boldsymbol{\varepsilon} +$$ + +

+We approximate this function with our model from the solution of the linear regression equations, that is our +function \( f \) is approximated by \( \boldsymbol{\tilde{y}} \) where we want to minimize \( (\boldsymbol{y}-\boldsymbol{\tilde{y}})^2 \), our MSE, with +$$ +\boldsymbol{\tilde{y}} = \boldsymbol{X}\boldsymbol{\beta}. +$$ + +

+









+ +

Expectation value and variance

+ +

+We can calculate the expectation value of \( \boldsymbol{y} \) for a given element \( i \) +$$ +\begin{align*} +\mathbb{E}(y_i) & = +\mathbb{E}(\mathbf{X}_{i, \ast} \, \boldsymbol{\beta}) + \mathbb{E}(\varepsilon_i) +\, \, \, = \, \, \, \mathbf{X}_{i, \ast} \, \beta, +\end{align*} +$$ + +while +its variance is +$$ +\begin{align*} \mbox{Var}(y_i) & = \mathbb{E} \{ [y_i +- \mathbb{E}(y_i)]^2 \} \, \, \, = \, \, \, \mathbb{E} ( y_i^2 ) - +[\mathbb{E}(y_i)]^2 \\ & = \mathbb{E} [ ( \mathbf{X}_{i, \ast} \, +\beta + \varepsilon_i )^2] - ( \mathbf{X}_{i, \ast} \, \boldsymbol{\beta})^2 \\ & += \mathbb{E} [ ( \mathbf{X}_{i, \ast} \, \boldsymbol{\beta})^2 + 2 \varepsilon_i +\mathbf{X}_{i, \ast} \, \boldsymbol{\beta} + \varepsilon_i^2 ] - ( \mathbf{X}_{i, +\ast} \, \beta)^2 \\ & = ( \mathbf{X}_{i, \ast} \, \boldsymbol{\beta})^2 + 2 +\mathbb{E}(\varepsilon_i) \mathbf{X}_{i, \ast} \, \boldsymbol{\beta} + +\mathbb{E}(\varepsilon_i^2 ) - ( \mathbf{X}_{i, \ast} \, \boldsymbol{\beta})^2 +\\ & = \mathbb{E}(\varepsilon_i^2 ) \, \, \, = \, \, \, +\mbox{Var}(\varepsilon_i) \, \, \, = \, \, \, \sigma^2. +\end{align*} +$$ + +Hence, \( y_i \sim \mathcal{N}( \mathbf{X}_{i, \ast} \, \boldsymbol{\beta}, \sigma^2) \), that is \( \boldsymbol{y} \) follows a normal distribution with +mean value \( \boldsymbol{X}\boldsymbol{\beta} \) and variance \( \sigma^2 \) (not be confused with the singular values of the SVD). + +

+









+ +

Expectation value and variance for \( \boldsymbol{\beta} \)

+ +

+With the OLS expressions for the parameters \( \boldsymbol{\beta} \) we can evaluate the expectation value +$$ +\mathbb{E}(\boldsymbol{\beta}) = \mathbb{E}[ (\mathbf{X}^{\top} \mathbf{X})^{-1}\mathbf{X}^{T} \mathbf{Y}]=(\mathbf{X}^{T} \mathbf{X})^{-1}\mathbf{X}^{T} \mathbb{E}[ \mathbf{Y}]=(\mathbf{X}^{T} \mathbf{X})^{-1} \mathbf{X}^{T}\mathbf{X}\boldsymbol{\beta}=\boldsymbol{\beta}. +$$ + +This means that the estimator of the regression parameters is unbiased. + +

+We can also calculate the variance + +

+The variance of \( \boldsymbol{\beta} \) is +$$ +\begin{eqnarray*} +\mbox{Var}(\boldsymbol{\beta}) & = & \mathbb{E} \{ [\boldsymbol{\beta} - \mathbb{E}(\boldsymbol{\beta})] [\boldsymbol{\beta} - \mathbb{E}(\boldsymbol{\beta})]^{T} \} +\\ +& = & \mathbb{E} \{ [(\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{Y} - \boldsymbol{\beta}] \, [(\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{Y} - \boldsymbol{\beta}]^{T} \} +\\ +% & = & \mathbb{E} \{ [(\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{Y}] \, [(\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{Y}]^{T} \} - \boldsymbol{\beta} \, \boldsymbol{\beta}^{T} +% \\ +% & = & \mathbb{E} \{ (\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{Y} \, \mathbf{Y}^{T} \, \mathbf{X} \, (\mathbf{X}^{T} \mathbf{X})^{-1} \} - \boldsymbol{\beta} \, \boldsymbol{\beta}^{T} +% \\ +& = & (\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \, \mathbb{E} \{ \mathbf{Y} \, \mathbf{Y}^{T} \} \, \mathbf{X} \, (\mathbf{X}^{T} \mathbf{X})^{-1} - \boldsymbol{\beta} \, \boldsymbol{\beta}^{T} +\\ +& = & (\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \, \{ \mathbf{X} \, \boldsymbol{\beta} \, \boldsymbol{\beta}^{T} \, \mathbf{X}^{T} + \sigma^2 \} \, \mathbf{X} \, (\mathbf{X}^{T} \mathbf{X})^{-1} - \boldsymbol{\beta} \, \boldsymbol{\beta}^{T} +% \\ +% & = & (\mathbf{X}^T \mathbf{X})^{-1} \, \mathbf{X}^T \, \mathbf{X} \, \boldsymbol{\beta} \, \boldsymbol{\beta}^T \, \mathbf{X}^T \, \mathbf{X} \, (\mathbf{X}^T % \mathbf{X})^{-1} +% \\ +% & & + \, \, \sigma^2 \, (\mathbf{X}^T \mathbf{X})^{-1} \, \mathbf{X}^T \, \mathbf{X} \, (\mathbf{X}^T \mathbf{X})^{-1} - \boldsymbol{\beta} \boldsymbol{\beta}^T +\\ +& = & \boldsymbol{\beta} \, \boldsymbol{\beta}^{T} + \sigma^2 \, (\mathbf{X}^{T} \mathbf{X})^{-1} - \boldsymbol{\beta} \, \boldsymbol{\beta}^{T} +\, \, \, = \, \, \, \sigma^2 \, (\mathbf{X}^{T} \mathbf{X})^{-1}, +\end{eqnarray*} +$$ + +

+where we have used that \( \mathbb{E} (\mathbf{Y} \mathbf{Y}^{T}) = +\mathbf{X} \, \boldsymbol{\beta} \, \boldsymbol{\beta}^{T} \, \mathbf{X}^{T} + +\sigma^2 \, \mathbf{I}_{nn} \). From \( \mbox{Var}(\boldsymbol{\beta}) = \sigma^2 +\, (\mathbf{X}^{T} \mathbf{X})^{-1} \), one obtains an estimate of the +variance of the estimate of the \( j \)-th regression coefficient: +\( \hat{\sigma}^2 (\hat{\beta}_j ) = \hat{\sigma}^2 \sqrt{ +[(\mathbf{X}^{T} \mathbf{X})^{-1}]_{jj} } \). This may be used to +construct a confidence interval for the estimates. + +

+In a similar way, we cna obtain analytical expressions for say the +expectation values of the parameters \( \boldsymbol{\beta} \) and their variance +when we employ Ridge regression, and thereby a confidence interval. + +

+It is rather straightforward to show that +$$ +\mathbb{E} \big[ \boldsymbol{\beta}^{\mathrm{Ridge}} \big]=(\mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I}_{pp})^{-1} (\mathbf{X}^{\top} \mathbf{X})\boldsymbol{\beta}^{\mathrm{OLS}}. +$$ + +We see clearly that +\( \mathbb{E} \big[ \boldsymbol{\beta}^{\mathrm{Ridge}} \big] \not= \boldsymbol{\beta}^{\mathrm{OLS}} \) for any \( \lambda > 0 \). We say then that the ridge estimator is biased. + +

+We can also compute the variance as + +$$ +\mbox{Var}[\boldsymbol{\beta}^{\mathrm{Ridge}}]=\sigma^2[ \mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I} ]^{-1} \mathbf{X}^{T} \mathbf{X} \{ [ \mathbf{X}^{\top} \mathbf{X} + \lambda \mathbf{I} ]^{-1}\}^{T}, +$$ + +and it is easy to see that if the parameter \( \lambda \) goes to infinity then the variance of Ridge parameters \( \boldsymbol{\beta} \) goes to zero. + +

+With this, we can compute the difference + +$$ +\mbox{Var}[\boldsymbol{\beta}^{\mathrm{OLS}}]-\mbox{Var}(\boldsymbol{\beta}^{\mathrm{Ridge}})=\sigma^2 [ \mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I} ]^{-1}[ 2\lambda\mathbf{I} + \lambda^2 (\mathbf{X}^{T} \mathbf{X})^{-1} ] \{ [ \mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I} ]^{-1}\}^{T}. +$$ + +The difference is non-negative definite since each component of the +matrix product is non-negative definite. +This means the variance we obtain with the standard OLS will always for \( \lambda > 0 \) be larger than the variance of \( \boldsymbol{\beta} \) obtained with the Ridge estimator. This has interesting consequences when we discuss the so-called bias-variance trade-off below. + +

+ + +

Cross-validation

+ +

+Instead of choosing the penalty parameter to balance model fit with +model complexity, cross-validation requires it (i.e. the penalty +parameter) to yield a model with good prediction +performance. Commonly, this performance is evaluated on novel +data. Novel data need not be easy to come by and one has to make do +with the data at hand. + +

+The setting of original and novel data is +then mimicked by sample splitting: the data set is divided into two +(groups of samples). One of these two data sets, called the +training set, plays the role of original data on which the model is +built. The second of these data sets, called the test set, plays the +role of the novel data and is used to evaluate the prediction +performance (often operationalized as the log-likelihood or the +prediction error or its square or the R2 score) of the model built on the training data set. This +procedure (model building and prediction evaluation on training and +test set, respectively) is done for a collection of possible penalty +parameter choices. The penalty parameter that yields the model with +the best prediction performance is to be preferred. The thus obtained +performance evaluation depends on the actual split of the data set. To +remove this dependence the data set is split many times into a +training and test set. For each split the model parameters are +estimated for all choices of \( \lambda \) using the training data and +estimated parameters are evaluated on the corresponding test set. The +penalty parameter that on average over the test sets performs best (in +some sense) is then selected. + +

+









+ +

Computationally expensive

+ +

+The validation set approach is conceptually simple and is easy to implement. But it has two potential drawbacks: + +

    +
  • The validation estimate of the test error rate can be highly variable, depending on precisely which observations are included in the training set and which observations are included in the validation set.
    • +
  • In the validation approach, only a subset of the observations, those that are included in the training set rather than in the validation set are used to fit the model. Since statistical methods tend to perform worse when trained on fewer observations, this suggests that the validation set error rate may tend to overestimate the test error rate for the model fit on the entire data set.
    • +
+ + + +

Various steps in cross-validation

+ +

+When the repetitive splitting of the data set is done randomly, +samples may accidently end up in a fast majority of the splits in +either training or test set. Such samples may have an unbalanced +influence on either model building or prediction evaluation. To avoid +this \( k \)-fold cross-validation structures the data splitting. The +samples are divided into \( k \) more or less equally sized exhaustive and +mutually exclusive subsets. In turn (at each split) one of these +subsets plays the role of the test set while the union of the +remaining subsets constitutes the training set. Such a splitting +warrants a balanced representation of each sample in both training and +test set over the splits. Still the division into the \( k \) subsets +involves a degree of randomness. This may be fully excluded when +choosing \( k=n \). This particular case is referred to as leave-one-out +cross-validation (LOOCV). + +

+ + +

How to set up the cross-validation for Ridge and/or Lasso

+ +
    +
  • Define a range of interest for the penalty parameter.
    • +
  • Divide the data set into training and test set comprising samples \( \{1, \ldots, n\} \setminus i \) and \( \{ i \} \), respectively.
    • +
  • Fit the linear regression model by means of ridge estimation for each \( \lambda \) in the grid using the training set, and the corresponding estimate of the error variance \( \boldsymbol{\sigma}_{-i}^2(\lambda) \), as
    • +
+ +$$ +\begin{align*} +\boldsymbol{\beta}_{-i}(\lambda) & = ( \boldsymbol{X}_{-i, \ast}^{T} +\boldsymbol{X}_{-i, \ast} + \lambda \boldsymbol{I}_{pp})^{-1} +\boldsymbol{X}_{-i, \ast}^{T} \boldsymbol{y}_{-i} +\end{align*} +$$ + + +
    +
  • Evaluate the prediction performance of these models on the test set by \( \log\{L[y_i, \boldsymbol{X}_{i, \ast}; \boldsymbol{\beta}_{-i}(\lambda), \boldsymbol{\sigma}_{-i}^2(\lambda)]\} \). Or, by the prediction error \( |y_i - \boldsymbol{X}_{i, \ast} \boldsymbol{\beta}_{-i}(\lambda)| \), the relative error, the error squared or the R2 score function.
    • +
  • Repeat the first three steps such that each sample plays the role of the test set once.
    • +
  • Average the prediction performances of the test sets at each grid point of the penalty bias/parameter by computing the cross-validated log-likelihood. It is an estimate of the prediction performance of the model corresponding to this value of the penalty parameter on novel data. It is defined as
    • +
+ +$$ +\begin{align*} +\frac{1}{n} \sum_{i = 1}^n \log\{L[y_i, \mathbf{X}_{i, \ast}; \boldsymbol{\beta}_{-i}(\lambda), \boldsymbol{\sigma}_{-i}^2(\lambda)]\}. +\end{align*} +$$ + + +
    +
  • The value of the penalty parameter that maximizes the cross-validated log-likelihood is the value of choice. Or we can use the MSE or the R2 score functions.
    • +
+ +









+ +

Resampling methods: Jackknife and Bootstrap

+ +

+Two famous +resampling methods are the independent bootstrap and the jackknife. + +

+The jackknife is a special case of the independent bootstrap. Still, the jackknife was made +popular prior to the independent bootstrap. And as the popularity of +the independent bootstrap soared, new variants, such as the dependent bootstrap. + +

+The Jackknife and independent bootstrap work for +independent, identically distributed random variables. +If these conditions are not +satisfied, the methods will fail. Yet, it should be said that if the data are +independent, identically distributed, and we only want to estimate the +variance of \( \overline{X} \) (which often is the case), then there is no +need for bootstrapping. + +

+









+ +

Resampling methods: Jackknife

+ +

+The Jackknife works by making many replicas of the estimator \( \widehat{\theta} \). +The jackknife is a resampling method where we systematically leave out one observation from the vector of observed values \( \boldsymbol{x} = (x_1,x_2,\cdots,X_n) \). +Let \( \boldsymbol{x}_i \) denote the vector +$$ +\boldsymbol{x}_i = (x_1,x_2,\cdots,x_{i-1},x_{i+1},\cdots,x_n), +$$ + +

+which equals the vector \( \boldsymbol{x} \) with the exception that observation +number \( i \) is left out. Using this notation, define +\( \widehat{\theta}_i \) to be the estimator +\( \widehat{\theta} \) computed using \( \vec{X}_i \). + +

+









+ +

Jackknife code example

+

+ + +

from numpy import *
+from numpy.random import randint, randn
+from time import time
+
+def jackknife(data, stat):
+    n = len(data);t = zeros(n); inds = arange(n); t0 = time()
+    ## 'jackknifing' by leaving out an observation for each i                                                                                                                      
+    for i in range(n):
+        t[i] = stat(delete(data,i) )
+
+    # analysis                                                                                                                                                                     
+    print("Runtime: %g sec" % (time()-t0)); print("Jackknife Statistics :")
+    print("original           bias      std. error")
+    print("%8g %14g %15g" % (stat(data),(n-1)*mean(t)/n, (n*var(t))**.5))
+
+    return t
+
+
+# Returns mean of data samples                                                                                                                                                     
+def stat(data):
+    return mean(data)
+
+
+mu, sigma = 100, 15
+datapoints = 10000
+x = mu + sigma*random.randn(datapoints)
+# jackknife returns the data sample                                                                                                                                                
+t = jackknife(x, stat)
+
+

+









+ +

Resampling methods: Bootstrap

+
+ +

+Bootstrapping is a nonparametric approach to statistical inference +that substitutes computation for more traditional distributional +assumptions and asymptotic results. Bootstrapping offers a number of +advantages: + +

    +
  1. The bootstrap is quite general, although there are some cases in which it fails.
    2. +
  2. Because it does not require distributional assumptions (such as normally distributed errors), the bootstrap can provide more accurate inferences when the data are not well behaved or when the sample size is small.
    3. +
  3. It is possible to apply the bootstrap to statistics with sampling distributions that are difficult to derive, even asymptotically.
    4. +
  4. It is relatively simple to apply the bootstrap to complex data-collection plans (such as stratified and clustered samples).
    5. +
+
+ + +

+









+ +

Resampling methods: Bootstrap background

+ +

+Since \( \widehat{\theta} = \widehat{\theta}(\boldsymbol{X}) \) is a function of random variables, +\( \widehat{\theta} \) itself must be a random variable. Thus it has +a pdf, call this function \( p(\boldsymbol{t}) \). The aim of the bootstrap is to +estimate \( p(\boldsymbol{t}) \) by the relative frequency of +\( \widehat{\theta} \). You can think of this as using a histogram +in the place of \( p(\boldsymbol{t}) \). If the relative frequency closely +resembles \( p(\vec{t}) \), then using numerics, it is straight forward to +estimate all the interesting parameters of \( p(\boldsymbol{t}) \) using point +estimators. + +

+









+ +

Resampling methods: More Bootstrap background

+ +

+In the case that \( \widehat{\theta} \) has +more than one component, and the components are independent, we use the +same estimator on each component separately. If the probability +density function of \( X_i \), \( p(x) \), had been known, then it would have +been straight forward to do this by: + +

    +
  1. Drawing lots of numbers from \( p(x) \), suppose we call one such set of numbers \( (X_1^*, X_2^*, \cdots, X_n^*) \).
    2. +
  2. Then using these numbers, we could compute a replica of \( \widehat{\theta} \) called \( \widehat{\theta}^* \).
    3. +
+ +By repeated use of (1) and (2), many +estimates of \( \widehat{\theta} \) could have been obtained. The +idea is to use the relative frequency of \( \widehat{\theta}^* \) +(think of a histogram) as an estimate of \( p(\boldsymbol{t}) \). + +

+









+ +

Resampling methods: Bootstrap approach

+ +

+But +unless there is enough information available about the process that +generated \( X_1,X_2,\cdots,X_n \), \( p(x) \) is in general +unknown. Therefore, Efron in 1979 asked the +question: What if we replace \( p(x) \) by the relative frequency +of the observation \( X_i \); if we draw observations in accordance with +the relative frequency of the observations, will we obtain the same +result in some asymptotic sense? The answer is yes. + +

+Instead of generating the histogram for the relative +frequency of the observation \( X_i \), just draw the values +\( (X_1^*,X_2^*,\cdots,X_n^*) \) with replacement from the vector +\( \boldsymbol{X} \). + +

+









+ +

Resampling methods: Bootstrap steps

+ +

+The independent bootstrap works like this: + +

    +
  1. Draw with replacement \( n \) numbers for the observed variables \( \boldsymbol{x} = (x_1,x_2,\cdots,x_n) \).
    2. +
  2. Define a vector \( \boldsymbol{x}^* \) containing the values which were drawn from \( \boldsymbol{x} \).
    3. +
  3. Using the vector \( \boldsymbol{x}^* \) compute \( \widehat{\theta}^* \) by evaluating \( \widehat \theta \) under the observations \( \boldsymbol{x}^* \).
    4. +
  4. Repeat this process \( k \) times.
    5. +
+ +When you are done, you can draw a histogram of the relative frequency +of \( \widehat \theta^* \). This is your estimate of the probability +distribution \( p(t) \). Using this probability distribution you can +estimate any statistics thereof. In principle you never draw the +histogram of the relative frequency of \( \widehat{\theta}^* \). Instead +you use the estimators corresponding to the statistic of interest. For +example, if you are interested in estimating the variance of \( \widehat +\theta \), apply the etsimator \( \widehat \sigma^2 \) to the values +\( \widehat \theta ^* \). + +

+









+ +

Code example for the Bootstrap method

+ +

+The following code starts with a Gaussian distribution with mean value +\( \mu =100 \) and variance \( \sigma=15 \). We use this to generate the data +used in the bootstrap analysis. The bootstrap analysis returns a data +set after a given number of bootstrap operations (as many as we have +data points). This data set consists of estimated mean values for each +bootstrap operation. The histogram generated by the bootstrap method +shows that the distribution for these mean values is also a Gaussian, +centered around the mean value \( \mu=100 \) but with standard deviation +\( \sigma/\sqrt{n} \), where \( n \) is the number of bootstrap samples (in +this case the same as the number of original data points). The value +of the standard deviation is what we expect from the central limit +theorem. + +

+ + +

from numpy import *
+from numpy.random import randint, randn
+from time import time
+import matplotlib.mlab as mlab
+import matplotlib.pyplot as plt
+
+# Returns mean of bootstrap samples                                                                                                                                                
+def stat(data):
+    return mean(data)
+
+# Bootstrap algorithm
+def bootstrap(data, statistic, R):
+    t = zeros(R); n = len(data); inds = arange(n); t0 = time()
+    # non-parametric bootstrap         
+    for i in range(R):
+        t[i] = statistic(data[randint(0,n,n)])
+
+    # analysis    
+    print("Runtime: %g sec" % (time()-t0)); print("Bootstrap Statistics :")
+    print("original           bias      std. error")
+    print("%8g %8g %14g %15g" % (statistic(data), std(data),mean(t),std(t)))
+    return t
+
+
+mu, sigma = 100, 15
+datapoints = 10000
+x = mu + sigma*random.randn(datapoints)
+# bootstrap returns the data sample                                    
+t = bootstrap(x, stat, datapoints)
+# the histogram of the bootstrapped  data                                                                                                    
+n, binsboot, patches = plt.hist(t, 50, normed=1, facecolor='red', alpha=0.75)
+
+# add a 'best fit' line  
+y = mlab.normpdf( binsboot, mean(t), std(t))
+lt = plt.plot(binsboot, y, 'r--', linewidth=1)
+plt.xlabel('Smarts')
+plt.ylabel('Probability')
+plt.axis([99.5, 100.6, 0, 3.0])
+plt.grid(True)
+
+plt.show()
+
+

+









+ +

Code Example for Cross-validation and \( k \)-fold Cross-validation

+ +

+The code here uses Ridge regression with cross-validation (CV) resampling and \( k \)-fold CV in order to fit a specific polynomial. +

+ + +

import numpy as np
+import matplotlib.pyplot as plt
+from sklearn.model_selection import KFold
+from sklearn.linear_model import Ridge
+from sklearn.model_selection import cross_val_score
+from sklearn.preprocessing import PolynomialFeatures
+
+# A seed just to ensure that the random numbers are the same for every run.
+# Useful for eventual debugging.
+np.random.seed(3155)
+
+# Generate the data.
+nsamples = 100
+x = np.random.randn(nsamples)
+y = 3*x**2 + np.random.randn(nsamples)
+
+## Cross-validation on Ridge regression using KFold only
+
+# Decide degree on polynomial to fit
+poly = PolynomialFeatures(degree = 6)
+
+# Decide which values of lambda to use
+nlambdas = 500
+lambdas = np.logspace(-3, 5, nlambdas)
+
+# Initialize a KFold instance
+k = 5
+kfold = KFold(n_splits = k)
+
+# Perform the cross-validation to estimate MSE
+scores_KFold = np.zeros((nlambdas, k))
+
+i = 0
+for lmb in lambdas:
+    ridge = Ridge(alpha = lmb)
+    j = 0
+    for train_inds, test_inds in kfold.split(x):
+        xtrain = x[train_inds]
+        ytrain = y[train_inds]
+
+        xtest = x[test_inds]
+        ytest = y[test_inds]
+
+        Xtrain = poly.fit_transform(xtrain[:, np.newaxis])
+        ridge.fit(Xtrain, ytrain[:, np.newaxis])
+
+        Xtest = poly.fit_transform(xtest[:, np.newaxis])
+        ypred = ridge.predict(Xtest)
+
+        scores_KFold[i,j] = np.sum((ypred - ytest[:, np.newaxis])**2)/np.size(ypred)
+
+        j += 1
+    i += 1
+
+
+estimated_mse_KFold = np.mean(scores_KFold, axis = 1)
+
+## Cross-validation using cross_val_score from sklearn along with KFold
+
+# kfold is an instance initialized above as:
+# kfold = KFold(n_splits = k)
+
+estimated_mse_sklearn = np.zeros(nlambdas)
+i = 0
+for lmb in lambdas:
+    ridge = Ridge(alpha = lmb)
+
+    X = poly.fit_transform(x[:, np.newaxis])
+    estimated_mse_folds = cross_val_score(ridge, X, y[:, np.newaxis], scoring='neg_mean_squared_error', cv=kfold)
+
+    # cross_val_score return an array containing the estimated negative mse for every fold.
+    # we have to the the mean of every array in order to get an estimate of the mse of the model
+    estimated_mse_sklearn[i] = np.mean(-estimated_mse_folds)
+
+    i += 1
+
+## Plot and compare the slightly different ways to perform cross-validation
+
+plt.figure()
+
+plt.plot(np.log10(lambdas), estimated_mse_sklearn, label = 'cross_val_score')
+plt.plot(np.log10(lambdas), estimated_mse_KFold, 'r--', label = 'KFold')
+
+plt.xlabel('log10(lambda)')
+plt.ylabel('mse')
+
+plt.legend()
+
+plt.show()
+
+

+









+ +

The bias-variance tradeoff

+ +

+We will discuss the bias-variance tradeoff in the context of +continuous predictions such as regression. However, many of the +intuitions and ideas discussed here also carry over to classification +tasks. Consider a dataset \( \mathcal{L} \) consisting of the data +\( \mathbf{X}_\mathcal{L}=\{(y_j, \boldsymbol{x}_j), j=0\ldots n-1\} \). + +

+Let us assume that the true data is generated from a noisy model + +$$ +\boldsymbol{y}=f(\boldsymbol{x}) + \boldsymbol{\epsilon} +$$ + +

+where \( \epsilon \) is normally distributed with mean zero and standard deviation \( \sigma^2 \). + +

+In our derivation of the ordinary least squares method we defined then +an approximation to the function \( f \) in terms of the parameters +\( \boldsymbol{\beta} \) and the design matrix \( \boldsymbol{X} \) which embody our model, +that is \( \boldsymbol{\tilde{y}}=\boldsymbol{X}\boldsymbol{\beta} \). + +

+Thereafter we found the parameters \( \boldsymbol{\beta} \) by optimizing the means squared error via the so-called cost function +$$ +C(\boldsymbol{X},\boldsymbol{\beta}) =\frac{1}{n}\sum_{i=0}^{n-1}(y_i-\tilde{y}_i)^2=\mathbb{E}\left[(\boldsymbol{y}-\boldsymbol{\tilde{y}})^2\right]. +$$ + +

+We can rewrite this as +$$ +\mathbb{E}\left[(\boldsymbol{y}-\boldsymbol{\tilde{y}})^2\right]=\frac{1}{n}\sum_i(f_i-\mathbb{E}\left[\boldsymbol{\tilde{y}}\right])^2+\frac{1}{n}\sum_i(\tilde{y}_i-\mathbb{E}\left[\boldsymbol{\tilde{y}}\right])^2+\sigma^2. +$$ + +

+The three terms represent the square of the bias of the learning +method, which can be thought of as the error caused by the simplifying +assumptions built into the method. The second term represents the +variance of the chosen model and finally the last terms is variance of +the error \( \boldsymbol{\epsilon} \). + +

+To derive this equation, we need to recall that the variance of \( \boldsymbol{y} \) and \( \boldsymbol{\epsilon} \) are both equal to \( \sigma^2 \). The mean value of \( \boldsymbol{\epsilon} \) is by definition equal to zero. Furthermore, the function \( f \) is not a stochastics variable, idem for \( \boldsymbol{\tilde{y}} \). +We use a more compact notation in terms of the expectation value +$$ +\mathbb{E}\left[(\boldsymbol{y}-\boldsymbol{\tilde{y}})^2\right]=\mathbb{E}\left[(\boldsymbol{f}+\boldsymbol{\epsilon}-\boldsymbol{\tilde{y}})^2\right], +$$ + +and adding and subtracting \( \mathbb{E}\left[\boldsymbol{\tilde{y}}\right] \) we get +$$ +\mathbb{E}\left[(\boldsymbol{y}-\boldsymbol{\tilde{y}})^2\right]=\mathbb{E}\left[(\boldsymbol{f}+\boldsymbol{\epsilon}-\boldsymbol{\tilde{y}}+\mathbb{E}\left[\boldsymbol{\tilde{y}}\right]-\mathbb{E}\left[\boldsymbol{\tilde{y}}\right])^2\right], +$$ + +which, using the abovementioned expectation values can be rewritten as +$$ +\mathbb{E}\left[(\boldsymbol{y}-\boldsymbol{\tilde{y}})^2\right]=\mathbb{E}\left[(\boldsymbol{y}-\mathbb{E}\left[\boldsymbol{\tilde{y}}\right])^2\right]+\mathrm{Var}\left[\boldsymbol{\tilde{y}}\right]+\sigma^2, +$$ + +that is the rewriting in terms of the so-called bias, the variance of the model \( \boldsymbol{\tilde{y}} \) and the variance of \( \boldsymbol{\epsilon} \). + +

+









+ +

Example code for Bias-Variance tradeoff

+

+ + +

import matplotlib.pyplot as plt
+import numpy as np
+from sklearn.linear_model import LinearRegression, Ridge, Lasso
+from sklearn.preprocessing import PolynomialFeatures
+from sklearn.model_selection import train_test_split
+from sklearn.pipeline import make_pipeline
+from sklearn.utils import resample
+
+np.random.seed(2018)
+
+n = 500
+n_boostraps = 100
+degree = 18  # A quite high value, just to show.
+noise = 0.1
+
+# Make data set.
+x = np.linspace(-1, 3, n).reshape(-1, 1)
+y = np.exp(-x**2) + 1.5 * np.exp(-(x-2)**2) + np.random.normal(0, 0.1, x.shape)
+
+# Hold out some test data that is never used in training.
+x_train, x_test, y_train, y_test = train_test_split(x, y, test_size=0.2)
+
+# Combine x transformation and model into one operation.
+# Not neccesary, but convenient.
+model = make_pipeline(PolynomialFeatures(degree=degree), LinearRegression(fit_intercept=False))
+
+# The following (m x n_bootstraps) matrix holds the column vectors y_pred
+# for each bootstrap iteration.
+y_pred = np.empty((y_test.shape[0], n_boostraps))
+for i in range(n_boostraps):
+    x_, y_ = resample(x_train, y_train)
+
+    # Evaluate the new model on the same test data each time.
+    y_pred[:, i] = model.fit(x_, y_).predict(x_test).ravel()
+
+# Note: Expectations and variances taken w.r.t. different training
+# data sets, hence the axis=1. Subsequent means are taken across the test data
+# set in order to obtain a total value, but before this we have error/bias/variance
+# calculated per data point in the test set.
+# Note 2: The use of keepdims=True is important in the calculation of bias as this 
+# maintains the column vector form. Dropping this yields very unexpected results.
+error = np.mean( np.mean((y_test - y_pred)**2, axis=1, keepdims=True) )
+bias = np.mean( (y_test - np.mean(y_pred, axis=1, keepdims=True))**2 )
+variance = np.mean( np.var(y_pred, axis=1, keepdims=True) )
+print('Error:', error)
+print('Bias^2:', bias)
+print('Var:', variance)
+print('{} >= {} + {} = {}'.format(error, bias, variance, bias+variance))
+
+plt.plot(x[::5, :], y[::5, :], label='f(x)')
+plt.scatter(x_test, y_test, label='Data points')
+plt.scatter(x_test, np.mean(y_pred, axis=1), label='Pred')
+plt.legend()
+plt.show()
+
+

+









+ +

Understanding what happens

+

+ + +

import matplotlib.pyplot as plt
+import numpy as np
+from sklearn.linear_model import LinearRegression, Ridge, Lasso
+from sklearn.preprocessing import PolynomialFeatures
+from sklearn.model_selection import train_test_split
+from sklearn.pipeline import make_pipeline
+from sklearn.utils import resample
+
+np.random.seed(2018)
+
+n = 40
+n_boostraps = 100
+maxdegree = 14
+
+
+# Make data set.
+x = np.linspace(-3, 3, n).reshape(-1, 1)
+y = np.exp(-x**2) + 1.5 * np.exp(-(x-2)**2)+ np.random.normal(0, 0.1, x.shape)
+error = np.zeros(maxdegree)
+bias = np.zeros(maxdegree)
+variance = np.zeros(maxdegree)
+polydegree = np.zeros(maxdegree)
+x_train, x_test, y_train, y_test = train_test_split(x, y, test_size=0.2)
+
+for degree in range(maxdegree):
+    model = make_pipeline(PolynomialFeatures(degree=degree), LinearRegression(fit_intercept=False))
+    y_pred = np.empty((y_test.shape[0], n_boostraps))
+    for i in range(n_boostraps):
+        x_, y_ = resample(x_train, y_train)
+        y_pred[:, i] = model.fit(x_, y_).predict(x_test).ravel()
+
+    polydegree[degree] = degree
+    error[degree] = np.mean( np.mean((y_test - y_pred)**2, axis=1, keepdims=True) )
+    bias[degree] = np.mean( (y_test - np.mean(y_pred, axis=1, keepdims=True))**2 )
+    variance[degree] = np.mean( np.var(y_pred, axis=1, keepdims=True) )
+    print('Polynomial degree:', degree)
+    print('Error:', error[degree])
+    print('Bias^2:', bias[degree])
+    print('Var:', variance[degree])
+    print('{} >= {} + {} = {}'.format(error[degree], bias[degree], variance[degree], bias[degree]+variance[degree]))
+
+plt.plot(polydegree, np.log10(error), label='Error')
+plt.plot(polydegree, bias, label='bias')
+plt.plot(polydegree, variance, label='Variance')
+plt.legend()
+plt.show()
+
+

+ + +

Summing up

+ +

+The bias-variance tradeoff summarizes the fundamental tension in +machine learning, particularly supervised learning, between the +complexity of a model and the amount of training data needed to train +it. Since data is often limited, in practice it is often useful to +use a less-complex model with higher bias, that is a model whose asymptotic +performance is worse than another model because it is easier to +train and less sensitive to sampling noise arising from having a +finite-sized training dataset (smaller variance). + +

+The above equations tell us that in +order to minimize the expected test error, we need to select a +statistical learning method that simultaneously achieves low variance +and low bias. Note that variance is inherently a nonnegative quantity, +and squared bias is also nonnegative. Hence, we see that the expected +test MSE can never lie below \( Var(\epsilon) \), the irreducible error. + +

+What do we mean by the variance and bias of a statistical learning +method? The variance refers to the amount by which our model would change if we +estimated it using a different training data set. Since the training +data are used to fit the statistical learning method, different +training data sets will result in a different estimate. But ideally the +estimate for our model should not vary too much between training +sets. However, if a method has high variance then small changes in +the training data can result in large changes in the model. In general, more +flexible statistical methods have higher variance. + +

+









+ +

Another Example rom Scikit-Learn's Repository

+

+ + +

"""
+============================
+Underfitting vs. Overfitting
+============================
+
+This example demonstrates the problems of underfitting and overfitting and
+how we can use linear regression with polynomial features to approximate
+nonlinear functions. The plot shows the function that we want to approximate,
+which is a part of the cosine function. In addition, the samples from the
+real function and the approximations of different models are displayed. The
+models have polynomial features of different degrees. We can see that a
+linear function (polynomial with degree 1) is not sufficient to fit the
+training samples. This is called **underfitting**. A polynomial of degree 4
+approximates the true function almost perfectly. However, for higher degrees
+the model will **overfit** the training data, i.e. it learns the noise of the
+training data.
+We evaluate quantitatively **overfitting** / **underfitting** by using
+cross-validation. We calculate the mean squared error (MSE) on the validation
+set, the higher, the less likely the model generalizes correctly from the
+training data.
+"""
+
+print(__doc__)
+
+import numpy as np
+import matplotlib.pyplot as plt
+from sklearn.pipeline import Pipeline
+from sklearn.preprocessing import PolynomialFeatures
+from sklearn.linear_model import LinearRegression
+from sklearn.model_selection import cross_val_score
+
+
+def true_fun(X):
+    return np.cos(1.5 * np.pi * X)
+
+np.random.seed(0)
+
+n_samples = 30
+degrees = [1, 4, 15]
+
+X = np.sort(np.random.rand(n_samples))
+y = true_fun(X) + np.random.randn(n_samples) * 0.1
+
+plt.figure(figsize=(14, 5))
+for i in range(len(degrees)):
+    ax = plt.subplot(1, len(degrees), i + 1)
+    plt.setp(ax, xticks=(), yticks=())
+
+    polynomial_features = PolynomialFeatures(degree=degrees[i],
+                                             include_bias=False)
+    linear_regression = LinearRegression()
+    pipeline = Pipeline([("polynomial_features", polynomial_features),
+                         ("linear_regression", linear_regression)])
+    pipeline.fit(X[:, np.newaxis], y)
+
+    # Evaluate the models using crossvalidation
+    scores = cross_val_score(pipeline, X[:, np.newaxis], y,
+                             scoring="neg_mean_squared_error", cv=10)
+
+    X_test = np.linspace(0, 1, 100)
+    plt.plot(X_test, pipeline.predict(X_test[:, np.newaxis]), label="Model")
+    plt.plot(X_test, true_fun(X_test), label="True function")
+    plt.scatter(X, y, edgecolor='b', s=20, label="Samples")
+    plt.xlabel("x")
+    plt.ylabel("y")
+    plt.xlim((0, 1))
+    plt.ylim((-2, 2))
+    plt.legend(loc="best")
+    plt.title("Degree {}\nMSE = {:.2e}(+/- {:.2e})".format(
+        degrees[i], -scores.mean(), scores.std()))
+plt.show()
+
+

+









+ +

The one-dimensional Ising model

+ +

+Let us bring back the Ising model again, but now with an additional +focus on Ridge and Lasso regression as well. We repeat some of the +basic parts of the Ising model and the setup of the training and test +data. The one-dimensional Ising model with nearest neighbor +interaction, no external field and a constant coupling constant \( J \) is +given by + +$$ +\begin{align} + H = -J \sum_{k}^L s_k s_{k + 1}, +\label{_auto17} +\end{align} +$$ + +where \( s_i \in \{-1, 1\} \) and \( s_{N + 1} = s_1 \). The number of spins in the system is determined by \( L \). For the one-dimensional system there is no phase transition. + +

+We will look at a system of \( L = 40 \) spins with a coupling constant of \( J = 1 \). To get enough training data we will generate 10000 states with their respective energies. + +

+ + +

import numpy as np
+import matplotlib.pyplot as plt
+from mpl_toolkits.axes_grid1 import make_axes_locatable
+import seaborn as sns
+import scipy.linalg as scl
+from sklearn.model_selection import train_test_split
+import sklearn.linear_model as skl
+import tqdm
+sns.set(color_codes=True)
+cmap_args=dict(vmin=-1., vmax=1., cmap='seismic')
+
+L = 40
+n = int(1e4)
+
+spins = np.random.choice([-1, 1], size=(n, L))
+J = 1.0
+
+energies = np.zeros(n)
+
+for i in range(n):
+    energies[i] = - J * np.dot(spins[i], np.roll(spins[i], 1))
+
+

+A more general form for the one-dimensional Ising model is + +$$ +\begin{align} + H = - \sum_j^L \sum_k^L s_j s_k J_{jk}. +\label{_auto18} +\end{align} +$$ + +

+Here we allow for interactions beyond the nearest neighbors and a more +adaptive coupling matrix. This latter expression can be formulated as +a matrix-product on the form +$$ +\begin{align} + H = X J, +\label{_auto19} +\end{align} +$$ + +

+where \( X_{jk} = s_j s_k \) and \( J \) is the matrix consisting of the +elements \( -J_{jk} \). This form of writing the energy fits perfectly +with the form utilized in linear regression, viz. +$$ +\begin{align} + \boldsymbol{y} = \boldsymbol{X}\boldsymbol{\beta} + \boldsymbol{\epsilon}. +\label{_auto20} +\end{align} +$$ + +We organize the data as we did above +

+ + +

X = np.zeros((n, L ** 2))
+for i in range(n):
+    X[i] = np.outer(spins[i], spins[i]).ravel()
+y = energies
+X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.96)
+
+X_train_own = np.concatenate(
+    (np.ones(len(X_train))[:, np.newaxis], X_train),
+    axis=1
+)
+
+X_test_own = np.concatenate(
+    (np.ones(len(X_test))[:, np.newaxis], X_test),
+    axis=1
+)
+
+

+We will do all fitting with Scikit-Learn, + +

+ + +

clf = skl.LinearRegression().fit(X_train, y_train)
+
+

+When extracting the \( J \)-matrix we make sure to remove the intercept +

+ + +

J_sk = clf.coef_.reshape(L, L)
+
+

+And then we plot the results +

+ + +

fig = plt.figure(figsize=(20, 14))
+im = plt.imshow(J_sk, **cmap_args)
+plt.title("LinearRegression from Scikit-learn", fontsize=18)
+plt.xticks(fontsize=18)
+plt.yticks(fontsize=18)
+cb = fig.colorbar(im)
+cb.ax.set_yticklabels(cb.ax.get_yticklabels(), fontsize=18)
+plt.show()
+
+

+The results perfectly with our previous discussion where we used our own code. + +

+









+ +

Ridge regression

+ +

+Having explored the ordinary least squares we move on to ridge +regression. In ridge regression we include a regularizer. This +involves a new cost function which leads to a new estimate for the +weights \( \boldsymbol{\beta} \). This results in a penalized regression problem. The +cost function is given by + +$$ +\begin{align} + C(\boldsymbol{X}, \boldsymbol{\beta}; \lambda) = (\boldsymbol{X}\boldsymbol{\beta} - \boldsymbol{y})^T(\boldsymbol{X}\boldsymbol{\beta} - \boldsymbol{y}) + \lambda \boldsymbol{\beta}^T\boldsymbol{\beta}. +\label{_auto21} +\end{align} +$$ + +

+ + +

_lambda = 0.1
+clf_ridge = skl.Ridge(alpha=_lambda).fit(X_train, y_train)
+J_ridge_sk = clf_ridge.coef_.reshape(L, L)
+fig = plt.figure(figsize=(20, 14))
+im = plt.imshow(J_ridge_sk, **cmap_args)
+plt.title("Ridge from Scikit-learn", fontsize=18)
+plt.xticks(fontsize=18)
+plt.yticks(fontsize=18)
+cb = fig.colorbar(im)
+cb.ax.set_yticklabels(cb.ax.get_yticklabels(), fontsize=18)
+
+plt.show()
+
+

+









+ +

LASSO regression

+ +

+In the Least Absolute Shrinkage and Selection Operator (LASSO)-method we get a third cost function. + +$$ +\begin{align} + C(\boldsymbol{X}, \boldsymbol{\beta}; \lambda) = (\boldsymbol{X}\boldsymbol{\beta} - \boldsymbol{y})^T(\boldsymbol{X}\boldsymbol{\beta} - \boldsymbol{y}) + \lambda \sqrt{\boldsymbol{\beta}^T\boldsymbol{\beta}}. +\label{_auto22} +\end{align} +$$ + +

+Finding the extremal point of this cost function is not so straight-forward as in least squares and ridge. We will therefore rely solely on the function ``Lasso`` from Scikit-Learn. + +

+ + +

clf_lasso = skl.Lasso(alpha=_lambda).fit(X_train, y_train)
+J_lasso_sk = clf_lasso.coef_.reshape(L, L)
+fig = plt.figure(figsize=(20, 14))
+im = plt.imshow(J_lasso_sk, **cmap_args)
+plt.title("Lasso from Scikit-learn", fontsize=18)
+plt.xticks(fontsize=18)
+plt.yticks(fontsize=18)
+cb = fig.colorbar(im)
+cb.ax.set_yticklabels(cb.ax.get_yticklabels(), fontsize=18)
+
+plt.show()
+
+

+It is quite striking how LASSO breaks the symmetry of the coupling +constant as opposed to ridge and OLS. We get a sparse solution with +\( J_{j, j + 1} = -1 \). + +

+









+ +

Performance as function of the regularization parameter

+ +

+We see how the different models perform for a different set of values for \( \lambda \). + +

+ + +

lambdas = np.logspace(-4, 5, 10)
+
+train_errors = {
+    "ols_sk": np.zeros(lambdas.size),
+    "ridge_sk": np.zeros(lambdas.size),
+    "lasso_sk": np.zeros(lambdas.size)
+}
+
+test_errors = {
+    "ols_sk": np.zeros(lambdas.size),
+    "ridge_sk": np.zeros(lambdas.size),
+    "lasso_sk": np.zeros(lambdas.size)
+}
+
+plot_counter = 1
+
+fig = plt.figure(figsize=(32, 54))
+
+for i, _lambda in enumerate(tqdm.tqdm(lambdas)):
+    for key, method in zip(
+        ["ols_sk", "ridge_sk", "lasso_sk"],
+        [skl.LinearRegression(), skl.Ridge(alpha=_lambda), skl.Lasso(alpha=_lambda)]
+    ):
+        method = method.fit(X_train, y_train)
+
+        train_errors[key][i] = method.score(X_train, y_train)
+        test_errors[key][i] = method.score(X_test, y_test)
+
+        omega = method.coef_.reshape(L, L)
+
+        plt.subplot(10, 5, plot_counter)
+        plt.imshow(omega, **cmap_args)
+        plt.title(r"%s, $\lambda = %.4f$" % (key, _lambda))
+        plot_counter += 1
+
+plt.show()
+
+

+We see that LASSO reaches a good solution for low +values of \( \lambda \), but will "wither" when we increase \( \lambda \) too +much. Ridge is more stable over a larger range of values for +\( \lambda \), but eventually also fades away. + +

+









+ +

Finding the optimal value of \( \lambda \)

+ +

+To determine which value of \( \lambda \) is best we plot the accuracy of +the models when predicting the training and the testing set. We expect +the accuracy of the training set to be quite good, but if the accuracy +of the testing set is much lower this tells us that we might be +subject to an overfit model. The ideal scenario is an accuracy on the +testing set that is close to the accuracy of the training set. + +

+ + +

fig = plt.figure(figsize=(20, 14))
+
+colors = {
+    "ols_sk": "r",
+    "ridge_sk": "y",
+    "lasso_sk": "c"
+}
+
+for key in train_errors:
+    plt.semilogx(
+        lambdas,
+        train_errors[key],
+        colors[key],
+        label="Train {0}".format(key),
+        linewidth=4.0
+    )
+
+for key in test_errors:
+    plt.semilogx(
+        lambdas,
+        test_errors[key],
+        colors[key] + "--",
+        label="Test {0}".format(key),
+        linewidth=4.0
+    )
+plt.legend(loc="best", fontsize=18)
+plt.xlabel(r"$\lambda$", fontsize=18)
+plt.ylabel(r"$R^2$", fontsize=18)
+plt.tick_params(labelsize=18)
+plt.show()
+
+

+From the above figure we can see that LASSO with \( \lambda = 10^{-2} \) +achieves a very good accuracy on the test set. This by far surpasses the +other models for all values of \( \lambda \). + +

+









+ +

Further Exercises

+ +

Exercise 1

+ +

+We will generate our own dataset for a function \( y(x) \) where \( x \in [0,1] \) and defined by random numbers computed with the uniform distribution. The function \( y \) is a quadratic polynomial in \( x \) with added stochastic noise according to the normal distribution \( \cal {N}(0,1) \). +The following simple Python instructions define our \( x \) and \( y \) values (with 100 data points). +

+ + +

x = np.random.rand(100,1)
+y = 5*x*x+0.1*np.random.randn(100,1)
+
+
    +
  1. Write your own code (following the examples above) for computing the parametrization of the data set fitting a second-order polynomial.
    2. +
  2. Use thereafter scikit-learn (see again the examples in the regression slides) and compare with your own code.
    3. +
  3. Using scikit-learn, compute also the mean square error, a risk metric corresponding to the expected value of the squared (quadratic) error defined as
    4. +
+ +$$ MSE(\hat{y},\hat{\tilde{y}}) = \frac{1}{n} +\sum_{i=0}^{n-1}(y_i-\tilde{y}_i)^2, +$$ + +and the \( R^2 \) score function. +If \( \tilde{\hat{y}}_i \) is the predicted value of the \( i-th \) sample and \( y_i \) is the corresponding true value, then the score \( R^2 \) is defined as +$$ +R^2(\hat{y}, \tilde{\hat{y}}) = 1 - \frac{\sum_{i=0}^{n - 1} (y_i - \tilde{y}_i)^2}{\sum_{i=0}^{n - 1} (y_i - \bar{y})^2}, +$$ + +where we have defined the mean value of \( \hat{y} \) as +$$ +\bar{y} = \frac{1}{n} \sum_{i=0}^{n - 1} y_i. +$$ + +

+You can use the functionality included in scikit-learn. If you feel +for it, you can use your own program and define functions which +compute the above two functions. Discuss the meaning of these +results. Try also to vary the coefficient in front of the added +stochastic noise term and discuss the quality of the fits. + +

Exercise 2, variance of the parameters \( \beta \) in linear regression

+ +

+Show that the variance of the parameters \( \beta \) in the linear regression method (chapter 3, equation (3.8) of Trevor Hastie, Robert Tibshirani, Jerome H. Friedman, The Elements of Statistical Learning, Springer) is given as + +$$ +\mathrm{Var}(\hat{\beta}) = \left(\hat{X}^T\hat{X}\right)^{-1}\sigma^2, +$$ + +with +$$ +\sigma^2 = \frac{1}{N-p-1}\sum_{i=1}^{N} (y_i-\tilde{y}_i)^2, +$$ + +where we have assumed that we fit a function of degree \( p-1 \) (for example a polynomial in \( x \)). + +

Exercise 3

+ +

+This exercise is a continuation of exercise 1. We will +use the same function to generate our data set, still staying with a +simple function \( y(x) \) which we want to fit using linear regression, +but now extending the analysis to include the Ridge and the Lasso +regression methods. You can use the code under the Regression as an example on how to use the Ridge and the Lasso methods. + +

+We will thus again generate our own dataset for a function \( y(x) \) where +\( x \in [0,1] \) and defined by random numbers computed with the uniform +distribution. The function \( y \) is a quadratic polynomial in \( x \) with +added stochastic noise according to the normal distribution \( \cal{N}(0,1) \). + +

+The following simple Python instructions define our \( x \) and \( y \) values (with 100 data points). +

+ + +

x = np.random.rand(100,1)
+y = 5*x*x+0.1*np.random.randn(100,1)
+
+
    +
  1. Write your own code for the Ridge method and compute the parametrization for different values of \( \lambda \). Compare and analyze your results with those from exercise 1. Study the dependence on \( \lambda \) while also varying the strength of the noise in your expression for \( y(x) \).
    2. +
  2. Repeat the above but using the functionality of scikit-learn. Compare your code with the results from scikit-learn. Remember to run with the same random numbers for generating \( x \) and \( y \).
    3. +
  3. Our next step is to study the variance of the parameters \( \beta_1 \) and \( \beta_2 \) (assuming that we are parametrizing our function with a second-order polynomial. We will use standard linear regression and the Ridge regression. You can now opt for either writing your own function that calculates the variance of these paramaters (recall that this is equal to the diagonal elements of the matrix \( (\hat{X}^T\hat{X})+\lambda\hat{I})^{-1} \)) or use the functionality of scikit-learn and compute their variances. Discuss the results of these variances as functions
    4. +
  4. Repeat the previous step but add now the Lasso method. Discuss your results and compare with standard regression and the Ridge regression results.
    5. +
  5. Try to implement the cross-validation as well.
    6. +
  6. Finally, using scikit-learn or your own code, compute also the mean square error, a risk metric corresponding to the expected value of the squared (quadratic) error defined as
    7. +
+ +$$ MSE(\hat{y},\hat{\tilde{y}}) = \frac{1}{n} +\sum_{i=0}^{n-1}(y_i-\tilde{y}_i)^2, +$$ + +and the \( R^2 \) score function. +If \( \tilde{\hat{y}}_i \) is the predicted value of the \( i-th \) sample and \( y_i \) is the corresponding true value, then the score \( R^2 \) is defined as +$$ +R^2(\hat{y}, \tilde{\hat{y}}) = 1 - \frac{\sum_{i=0}^{n - 1} (y_i - \tilde{y}_i)^2}{\sum_{i=0}^{n - 1} (y_i - \bar{y})^2}, +$$ + +where we have defined the mean value of \( \hat{y} \) as +$$ +\bar{y} = \frac{1}{n} \sum_{i=0}^{n - 1} y_i. +$$ + +Discuss these quantities as functions of the variable \( \lambda \) in the Ridge and Lasso regression methods. + +

Exercise 4

+ +

+We will study how +to fit polynomials to a specific two-dimensional function called +Franke's +function. This +is a function which has been widely used when testing various interpolation and fitting +algorithms. Furthermore, after having established the model and the +method, we will employ resamling techniques such as the cross-validation and/or +the bootstrap methods, in order to perform a proper assessment of our models. + +

+The Franke function, which is a weighted sum of four exponentials reads as follows +$$ +\begin{align*} +f(x,y) &= \frac{3}{4}\exp{\left(-\frac{(9x-2)^2}{4} - \frac{(9y-2)^2}{4}\right)}+\frac{3}{4}\exp{\left(-\frac{(9x+1)^2}{49}- \frac{(9y+1)}{10}\right)} \\ +&+\frac{1}{2}\exp{\left(-\frac{(9x-7)^2}{4} - \frac{(9y-3)^2}{4}\right)} -\frac{1}{5}\exp{\left(-(9x-4)^2 - (9y-7)^2\right) }. +\end{align*} +$$ + +

+The function will be defined for \( x,y\in [0,1] \). Our first step will +be to perform an OLS regression analysis of this function, trying out +a polynomial fit with an \( x \) and \( y \) dependence of the form \( [x, y, +x^2, y^2, xy, \dots] \). We will also include cross-validation and +bootstrap as resampling techniques. As in homeworks 1 and 2, we +can use a uniform distribution to set up the arrays of values for \( x \) +and \( y \), or as in the example below just a fix values for \( x \) and \( y \) with a given step size. +In this case we will have two predictors and need to fit a +function (for example a polynomial) of \( x \) and \( y \). Thereafter we will +repeat much of the same procedure using the the Ridge and +Lasso regression methods, introducing thus a dependence on the bias +(penalty) \( \lambda \). + +

+The Python fucntion for the Franke function is included here (it performs also a three-dimensional plot of it) +

+ + +

from mpl_toolkits.mplot3d import Axes3D
+import matplotlib.pyplot as plt
+from matplotlib import cm
+from matplotlib.ticker import LinearLocator, FormatStrFormatter
+import numpy as np
+from random import random, seed
+
+fig = plt.figure()
+ax = fig.gca(projection='3d')
+
+# Make data.
+x = np.arange(0, 1, 0.05)
+y = np.arange(0, 1, 0.05)
+x, y = np.meshgrid(x,y)
+
+
+def FrankeFunction(x,y):
+    term1 = 0.75*np.exp(-(0.25*(9*x-2)**2) - 0.25*((9*y-2)**2))
+    term2 = 0.75*np.exp(-((9*x+1)**2)/49.0 - 0.1*(9*y+1))
+    term3 = 0.5*np.exp(-(9*x-7)**2/4.0 - 0.25*((9*y-3)**2))
+    term4 = -0.2*np.exp(-(9*x-4)**2 - (9*y-7)**2)
+    return term1 + term2 + term3 + term4
+
+
+z = FrankeFunction(x, y)
+
+# Plot the surface.
+surf = ax.plot_surface(x, y, z, cmap=cm.coolwarm,
+                       linewidth=0, antialiased=False)
+
+# Customize the z axis.
+ax.set_zlim(-0.10, 1.40)
+ax.zaxis.set_major_locator(LinearLocator(10))
+ax.zaxis.set_major_formatter(FormatStrFormatter('%.02f'))
+
+# Add a color bar which maps values to colors.
+fig.colorbar(surf, shrink=0.5, aspect=5)
+
+plt.show()
+
+

+We will thus again generate our own dataset for a function \( \mathrm{FrankeFunction}(x,y) \) where +\( x,y \in [0,1] \) could be defined by random numbers computed with the uniform +distribution. The function \( f(x,y) \) is the Franke function. You should explore also the addition +an added stochastic noise to this function using the normal distribution \( \cal{N}(0,1) \). + +

+Write your own code (using either a matrix inversion or a singular value decomposition from e.g., numpy ) or use your code from exercises 1 and 3 +and perform a standard least square regression analysis using polynomials in \( x \) and \( y \) up to fifth order. Find the confidence intervals of the parameters \( \beta \) by computing their variances, evaluate the Mean Squared error (MSE) +$$ MSE(\hat{y},\hat{\tilde{y}}) = \frac{1}{n} +\sum_{i=0}^{n-1}(y_i-\tilde{y}_i)^2, +$$ + +and the \( R^2 \) score function. +If \( \tilde{\hat{y}}_i \) is the predicted value of the \( i-th \) sample and \( y_i \) is the corresponding true value, then the score \( R^2 \) is defined as +$$ +R^2(\hat{y}, \tilde{\hat{y}}) = 1 - \frac{\sum_{i=0}^{n - 1} (y_i - \tilde{y}_i)^2}{\sum_{i=0}^{n - 1} (y_i - \bar{y})^2}, +$$ + +where we have defined the mean value of \( \hat{y} \) as +$$ +\bar{y} = \frac{1}{n} \sum_{i=0}^{n - 1} y_i. +$$ + +

+Perform a resampling of the data where you split the data in training data and test data. Implement the \( k \)-fold cross-validation algorithm and/or the bootstrap algorithm +and evaluate again the MSE and the \( R^2 \) functions resulting from the test data. Evaluate also the bias and variance of the final models. + +

+Write then your own code for the Ridge method, either using matrix +inversion or the singular value decomposition as done for standard OLS. Perform the same analysis as in the +previous exercise (for the same polynomials and include resampling +techniques) but now for different values of \( \lambda \). Compare and +analyze your results with those obtained with standard OLS. Study the +dependence on \( \lambda \) while also varying eventually the strength of +the noise in your expression for \( \mathrm{FrankeFunction}(x,y) \). + +

+Then perform the same studies but now with Lasso regression. Use the functionalities of +scikit-learn. Give a critical discussion of the three methods and a +judgement of which model fits the data best. + + + + +

+ © 1999-2019, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license +
+ + + + + + diff --git a/doc/src/Regression/Regression.idx b/doc/src/Regression/Regression.idx new file mode 100644 index 000000000..e69de29bb diff --git a/doc/src/Regression/Regression.ipynb b/doc/src/Regression/Regression.ipynb new file mode 100644 index 000000000..6db4e651c --- /dev/null +++ b/doc/src/Regression/Regression.ipynb @@ -0,0 +1,5838 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\n", + "# Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis\n", + "\n", + " \n", + "**Morten Hjorth-Jensen**, Department of Physics, University of Oslo and Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University\n", + "\n", + "Date: **Jul 22, 2019**\n", + "\n", + "Copyright 1999-2019, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license\n", + "\n", + "\n", + "\n", + "\n", + "## Why Linear Regression (aka Ordinary Least Squares and family)\n", + "\n", + "Fitting a continuous function with linear parameterization in terms of the parameters $\\boldsymbol{\\beta}$.\n", + "* Method of choice for fitting a continuous function!\n", + "\n", + "* Gives an excellent introduction to central Machine Learning features with **understandable pedagogical** links to other methods like **Neural Networks**, **Support Vector Machines** etc\n", + "\n", + "* Analytical expression for the fitting parameters $\\boldsymbol{\\beta}$\n", + "\n", + "* Analytical expressions for statistical propertiers like mean values, variances, confidence intervals and more\n", + "\n", + "* Analytical relation with probabilistic interpretations \n", + "\n", + "* Easy to introduce basic concepts like bias-variance tradeoff, cross-validation, resampling and regularization techniques and many other ML topics\n", + "\n", + "* Easy to code! And links well with classification problems and logistic regression and neural networks\n", + "\n", + "* Allows for **easy** hands-on understanding of gradient descent methods\n", + "\n", + "* and many more features\n", + "\n", + "For more discussions of Ridge and Lasso regression, [Wessel van Wieringen's](https://arxiv.org/abs/1509.09169) article is highly recommended.\n", + "Similarly, [Mehta et al's article](https://arxiv.org/abs/1803.08823) is also recommended.\n", + "\n", + "\n", + "## Regression analysis, overarching aims\n", + "\n", + "Regression modeling deals with the description of the sampling distribution of a given random variable $y$ and how it varies as function of another variable or a set of such variables $\\boldsymbol{x} =[x_0, x_1,\\dots, x_{n-1}]^T$. \n", + "The first variable is called the **dependent**, the **outcome** or the **response** variable while the set of variables $\\boldsymbol{x}$ is called the independent variable, or the predictor variable or the explanatory variable. \n", + "\n", + "A regression model aims at finding a likelihood function $p(\\boldsymbol{y}\\vert \\boldsymbol{x})$, that is the conditional distribution for $\\boldsymbol{y}$ with a given $\\boldsymbol{x}$. The estimation of $p(\\boldsymbol{y}\\vert \\boldsymbol{x})$ is made using a data set with \n", + "* $n$ cases $i = 0, 1, 2, \\dots, n-1$ \n", + "\n", + "* Response (target, dependent or outcome) variable $y_i$ with $i = 0, 1, 2, \\dots, n-1$ \n", + "\n", + "* $p$ so-called explanatory (independent or predictor) variables $\\boldsymbol{x}_i=[x_{i0}, x_{i1}, \\dots, x_{ip-1}]$ with $i = 0, 1, 2, \\dots, n-1$ and explanatory variables running from $0$ to $p-1$. See below for more explicit examples. \n", + "\n", + " The goal of the regression analysis is to extract/exploit relationship between $\\boldsymbol{y}$ and $\\boldsymbol{X}$ in or to infer causal dependencies, approximations to the likelihood functions, functional relationships and to make predictions, making fits and many other things.\n", + "\n", + "\n", + "\n", + "## Regression analysis, overarching aims II\n", + "\n", + "\n", + "Consider an experiment in which $p$ characteristics of $n$ samples are\n", + "measured. The data from this experiment, for various explanatory variables $p$ are normally represented by a matrix \n", + "$\\mathbf{X}$.\n", + "\n", + "The matrix $\\mathbf{X}$ is called the *design\n", + "matrix*. Additional information of the samples is available in the\n", + "form of $\\boldsymbol{y}$ (also as above). The variable $\\boldsymbol{y}$ is\n", + "generally referred to as the *response variable*. The aim of\n", + "regression analysis is to explain $\\boldsymbol{y}$ in terms of\n", + "$\\boldsymbol{X}$ through a functional relationship like $y_i =\n", + "f(\\mathbf{X}_{i,\\ast})$. When no prior knowledge on the form of\n", + "$f(\\cdot)$ is available, it is common to assume a linear relationship\n", + "between $\\boldsymbol{X}$ and $\\boldsymbol{y}$. This assumption gives rise to\n", + "the *linear regression model* where $\\boldsymbol{\\beta} = [\\beta_0, \\ldots,\n", + "\\beta_{p-1}]^{T}$ are the *regression parameters*. \n", + "\n", + "Linear regression gives us a set of analytical equations for the parameters $\\beta_j$.\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "## Examples\n", + "In order to understand the relation among the predictors $p$, the set of data $n$ and the target (outcome, output etc) $\\boldsymbol{y}$,\n", + "consider the model we discussed for describing nuclear binding energies. \n", + "\n", + "There we assumed that we could parametrize the data using a polynomial approximation based on the liquid drop model.\n", + "Assuming" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "BE(A) = a_0+a_1A+a_2A^{2/3}+a_3A^{-1/3}+a_4A^{-1},\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "we have five predictors, that is the intercept, the $A$ dependent term, the $A^{2/3}$ term and the $A^{-1/3}$ and $A^{-1}$ terms.\n", + "This gives $p=0,1,2,3,4$. Furthermore we have $n$ entries for each predictor. It means that our design matrix is a \n", + "$p\\times n$ matrix $\\boldsymbol{X}$.\n", + "\n", + "Here the predictors are based on a model we have made. A popular data set which is widely encountered in ML applications is the\n", + "so-called [credit card default data from Taiwan](https://www.sciencedirect.com/science/article/pii/S0957417407006719?via%3Dihub). The data set contains data on $n=30000$ credit card holders with predictors like gender, marital status, age, profession, education, etc. In total there are $24$ such predictors or attributes leading to a design matrix of dimensionality $24 \\times 30000$\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "## General linear models\n", + "Before we proceed let us study a case from linear algebra where we aim at fitting a set of data $\\boldsymbol{y}=[y_0,y_1,\\dots,y_{n-1}]$. We could think of these data as a result of an experiment or a complicated numerical experiment. These data are functions of a series of variables $\\boldsymbol{x}=[x_0,x_1,\\dots,x_{n-1}]$, that is $y_i = y(x_i)$ with $i=0,1,2,\\dots,n-1$. The variables $x_i$ could represent physical quantities like time, temperature, position etc. We assume that $y(x)$ is a smooth function. \n", + "\n", + "Since obtaining these data points may not be trivial, we want to use these data to fit a function which can allow us to make predictions for values of $y$ which are not in the present set. The perhaps simplest approach is to assume we can parametrize our function in terms of a polynomial of degree $n-1$ with $n$ points, that is" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "y=y(x) \\rightarrow y(x_i)=\\tilde{y}_i+\\epsilon_i=\\sum_{j=0}^{n-1} \\beta_j x_i^j+\\epsilon_i,\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "where $\\epsilon_i$ is the error in our approximation.\n", + "\n", + "\n", + "\n", + "\n", + "## Rewriting the fitting procedure as a linear algebra problem\n", + "For every set of values $y_i,x_i$ we have thus the corresponding set of equations" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "\\begin{align*}\n", + "y_0&=\\beta_0+\\beta_1x_0^1+\\beta_2x_0^2+\\dots+\\beta_{n-1}x_0^{n-1}+\\epsilon_0\\\\\n", + "y_1&=\\beta_0+\\beta_1x_1^1+\\beta_2x_1^2+\\dots+\\beta_{n-1}x_1^{n-1}+\\epsilon_1\\\\\n", + "y_2&=\\beta_0+\\beta_1x_2^1+\\beta_2x_2^2+\\dots+\\beta_{n-1}x_2^{n-1}+\\epsilon_2\\\\\n", + "\\dots & \\dots \\\\\n", + "y_{n-1}&=\\beta_0+\\beta_1x_{n-1}^1+\\beta_2x_{n-1}^2+\\dots+\\beta_{n-1}x_{n-1}^{n-1}+\\epsilon_{n-1}.\\\\\n", + "\\end{align*}\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Rewriting the fitting procedure as a linear algebra problem, more details\n", + "Defining the vectors" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "\\boldsymbol{y} = [y_0,y_1, y_2,\\dots, y_{n-1}]^T,\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "and" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "\\boldsymbol{\\beta} = [\\beta_0,\\beta_1, \\beta_2,\\dots, \\beta_{n-1}]^T,\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "and" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "\\boldsymbol{\\epsilon} = [\\epsilon_0,\\epsilon_1, \\epsilon_2,\\dots, \\epsilon_{n-1}]^T,\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "and the design matrix" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "\\boldsymbol{X}=\n", + "\\begin{bmatrix} \n", + "1& x_{0}^1 &x_{0}^2& \\dots & \\dots &x_{0}^{n-1}\\\\\n", + "1& x_{1}^1 &x_{1}^2& \\dots & \\dots &x_{1}^{n-1}\\\\\n", + "1& x_{2}^1 &x_{2}^2& \\dots & \\dots &x_{2}^{n-1}\\\\ \n", + "\\dots& \\dots &\\dots& \\dots & \\dots &\\dots\\\\\n", + "1& x_{n-1}^1 &x_{n-1}^2& \\dots & \\dots &x_{n-1}^{n-1}\\\\\n", + "\\end{bmatrix}\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "we can rewrite our equations as" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "\\boldsymbol{y} = \\boldsymbol{X}\\boldsymbol{\\beta}+\\boldsymbol{\\epsilon}.\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "The above design matrix is called a [Vandermonde matrix](https://en.wikipedia.org/wiki/Vandermonde_matrix).\n", + "\n", + "\n", + "\n", + "\n", + "## Generalizing the fitting procedure as a linear algebra problem\n", + "\n", + "We are obviously not limited to the above polynomial expansions. We\n", + "could replace the various powers of $x$ with elements of Fourier\n", + "series or instead of $x_i^j$ we could have $\\cos{(j x_i)}$ or $\\sin{(j\n", + "x_i)}$, or time series or other orthogonal functions. For every set\n", + "of values $y_i,x_i$ we can then generalize the equations to" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "\\begin{align*}\n", + "y_0&=\\beta_0x_{00}+\\beta_1x_{01}+\\beta_2x_{02}+\\dots+\\beta_{n-1}x_{0n-1}+\\epsilon_0\\\\\n", + "y_1&=\\beta_0x_{10}+\\beta_1x_{11}+\\beta_2x_{12}+\\dots+\\beta_{n-1}x_{1n-1}+\\epsilon_1\\\\\n", + "y_2&=\\beta_0x_{20}+\\beta_1x_{21}+\\beta_2x_{22}+\\dots+\\beta_{n-1}x_{2n-1}+\\epsilon_2\\\\\n", + "\\dots & \\dots \\\\\n", + "y_{i}&=\\beta_0x_{i0}+\\beta_1x_{i1}+\\beta_2x_{i2}+\\dots+\\beta_{n-1}x_{in-1}+\\epsilon_i\\\\\n", + "\\dots & \\dots \\\\\n", + "y_{n-1}&=\\beta_0x_{n-1,0}+\\beta_1x_{n-1,2}+\\beta_2x_{n-1,2}+\\dots+\\beta_{n-1}x_{n-1,n-1}+\\epsilon_{n-1}.\\\\\n", + "\\end{align*}\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "**Note that we have $p=n$ here. The matrix is symmetric. This is generally not the case!**\n", + "\n", + "\n", + "\n", + "\n", + "## Generalizing the fitting procedure as a linear algebra problem\n", + "We redefine in turn the matrix $\\boldsymbol{X}$ as" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "\\boldsymbol{X}=\n", + "\\begin{bmatrix} \n", + "x_{00}& x_{01} &x_{02}& \\dots & \\dots &x_{0,n-1}\\\\\n", + "x_{10}& x_{11} &x_{12}& \\dots & \\dots &x_{1,n-1}\\\\\n", + "x_{20}& x_{21} &x_{22}& \\dots & \\dots &x_{2,n-1}\\\\ \n", + "\\dots& \\dots &\\dots& \\dots & \\dots &\\dots\\\\\n", + "x_{n-1,0}& x_{n-1,1} &x_{n-1,2}& \\dots & \\dots &x_{n-1,n-1}\\\\\n", + "\\end{bmatrix}\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "and without loss of generality we rewrite again our equations as" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "\\boldsymbol{y} = \\boldsymbol{X}\\boldsymbol{\\beta}+\\boldsymbol{\\epsilon}.\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "The left-hand side of this equation is kwown. Our error vector $\\boldsymbol{\\epsilon}$ and the parameter vector $\\boldsymbol{\\beta}$ are our unknow quantities. How can we obtain the optimal set of $\\beta_i$ values?\n", + "\n", + "\n", + "\n", + "\n", + "## Optimizing our parameters\n", + "We have defined the matrix $\\boldsymbol{X}$ via the equations" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "\\begin{align*}\n", + "y_0&=\\beta_0x_{00}+\\beta_1x_{01}+\\beta_2x_{02}+\\dots+\\beta_{n-1}x_{0n-1}+\\epsilon_0\\\\\n", + "y_1&=\\beta_0x_{10}+\\beta_1x_{11}+\\beta_2x_{12}+\\dots+\\beta_{n-1}x_{1n-1}+\\epsilon_1\\\\\n", + "y_2&=\\beta_0x_{20}+\\beta_1x_{21}+\\beta_2x_{22}+\\dots+\\beta_{n-1}x_{2n-1}+\\epsilon_1\\\\\n", + "\\dots & \\dots \\\\\n", + "y_{i}&=\\beta_0x_{i0}+\\beta_1x_{i1}+\\beta_2x_{i2}+\\dots+\\beta_{n-1}x_{in-1}+\\epsilon_1\\\\\n", + "\\dots & \\dots \\\\\n", + "y_{n-1}&=\\beta_0x_{n-1,0}+\\beta_1x_{n-1,2}+\\beta_2x_{n-1,2}+\\dots+\\beta_{n-1}x_{n-1,n-1}+\\epsilon_{n-1}.\\\\\n", + "\\end{align*}\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "As we noted above, we stayed with a system with the design matrix \n", + " $\\boldsymbol{X}\\in {\\mathbb{R}}^{n\\times n}$, that is we have $p=n$. For reasons to come later (algorithmic arguments) we will hereafter define \n", + "our matrix as $\\boldsymbol{X}\\in {\\mathbb{R}}^{n\\times p}$, with the predictors refering to the column numbers and the entries $n$ being the row elements.\n", + "\n", + "\n", + "\n", + "\n", + "## Our model for the nuclear binding energies\n", + "\n", + "In our [introductory notes](https://compphysics.github.io/MachineLearningMSU/doc/pub/Introduction/html/Introduction.html) we looked at the so-called [liguid drop model](https://en.wikipedia.org/wiki/Semi-empirical_mass_formula). Let us remind ourselves about what we did by looking at the code.\n", + "\n", + "We restate the parts of the code we are most interested in." + ] + }, + { + "cell_type": "code", + "execution_count": 1, + "metadata": { + "collapsed": false + }, + "outputs": [], + "source": [ + "%matplotlib inline\n", + "\n", + "# Common imports\n", + "import numpy as np\n", + "import pandas as pd\n", + "import matplotlib.pyplot as plt\n", + "from IPython.display import display\n", + "import os\n", + "\n", + "# Where to save the figures and data files\n", + "PROJECT_ROOT_DIR = \"Results\"\n", + "FIGURE_ID = \"Results/FigureFiles\"\n", + "DATA_ID = \"DataFiles/\"\n", + "\n", + "if not os.path.exists(PROJECT_ROOT_DIR):\n", + " os.mkdir(PROJECT_ROOT_DIR)\n", + "\n", + "if not os.path.exists(FIGURE_ID):\n", + " os.makedirs(FIGURE_ID)\n", + "\n", + "if not os.path.exists(DATA_ID):\n", + " os.makedirs(DATA_ID)\n", + "\n", + "def image_path(fig_id):\n", + " return os.path.join(FIGURE_ID, fig_id)\n", + "\n", + "def data_path(dat_id):\n", + " return os.path.join(DATA_ID, dat_id)\n", + "\n", + "def save_fig(fig_id):\n", + " plt.savefig(image_path(fig_id) + \".png\", format='png')\n", + "\n", + "infile = open(data_path(\"MassEval2016.dat\"),'r')\n", + "\n", + "\n", + "# Read the experimental data with Pandas\n", + "Masses = pd.read_fwf(infile, usecols=(2,3,4,6,11),\n", + " names=('N', 'Z', 'A', 'Element', 'Ebinding'),\n", + " widths=(1,3,5,5,5,1,3,4,1,13,11,11,9,1,2,11,9,1,3,1,12,11,1),\n", + " header=39,\n", + " index_col=False)\n", + "\n", + "# Extrapolated values are indicated by '#' in place of the decimal place, so\n", + "# the Ebinding column won't be numeric. Coerce to float and drop these entries.\n", + "Masses['Ebinding'] = pd.to_numeric(Masses['Ebinding'], errors='coerce')\n", + "Masses = Masses.dropna()\n", + "# Convert from keV to MeV.\n", + "Masses['Ebinding'] /= 1000\n", + "\n", + "# Group the DataFrame by nucleon number, A.\n", + "Masses = Masses.groupby('A')\n", + "# Find the rows of the grouped DataFrame with the maximum binding energy.\n", + "Masses = Masses.apply(lambda t: t[t.Ebinding==t.Ebinding.max()])\n", + "A = Masses['A']\n", + "Z = Masses['Z']\n", + "N = Masses['N']\n", + "Element = Masses['Element']\n", + "Energies = Masses['Ebinding']\n", + "\n", + "# Now we set up the design matrix X\n", + "X = np.zeros((len(A),5))\n", + "X[:,0] = 1\n", + "X[:,1] = A\n", + "X[:,2] = A**(2.0/3.0)\n", + "X[:,3] = A**(-1.0/3.0)\n", + "X[:,4] = A**(-1.0)\n", + "# Then nice printout using pandas\n", + "DesignMatrix = pd.DataFrame(X)\n", + "DesignMatrix.index = A\n", + "DesignMatrix.columns = ['1', 'A', 'A^(2/3)', 'A^(-1/3)', '1/A']\n", + "display(DesignMatrix)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "With $\\boldsymbol{\\beta}\\in {\\mathbb{R}}^{p\\times 1}$, it means that we will hereafter write our equations for the approximation as" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "\\boldsymbol{\\tilde{y}}= \\boldsymbol{X}\\boldsymbol{\\beta},\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "throughout these lectures. \n", + "\n", + "\n", + "## Optimizing our parameters, more details\n", + "With the above we use the design matrix to define the approximation $\\boldsymbol{\\tilde{y}}$ via the unknown quantity $\\boldsymbol{\\beta}$ as" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "\\boldsymbol{\\tilde{y}}= \\boldsymbol{X}\\boldsymbol{\\beta},\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "and in order to find the optimal parameters $\\beta_i$ instead of solving the above linear algebra problem, we define a function which gives a measure of the spread between the values $y_i$ (which represent hopefully the exact values) and the parameterized values $\\tilde{y}_i$, namely" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "C(\\boldsymbol{\\beta})=\\frac{1}{n}\\sum_{i=0}^{n-1}\\left(y_i-\\tilde{y}_i\\right)^2=\\frac{1}{n}\\left\\{\\left(\\boldsymbol{y}-\\boldsymbol{\\tilde{y}}\\right)^T\\left(\\boldsymbol{y}-\\boldsymbol{\\tilde{y}}\\right)\\right\\},\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "or using the matrix $\\boldsymbol{X}$ and in a more compact matrix-vector notation as" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "C(\\boldsymbol{\\beta})=\\frac{1}{n}\\left\\{\\left(\\boldsymbol{y}-\\boldsymbol{X}^T\\boldsymbol{\\beta}\\right)^T\\left(\\boldsymbol{y}-\\boldsymbol{X}^T\\boldsymbol{\\beta}\\right)\\right\\}.\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "This function is one possible way to define the so-called cost function.\n", + "\n", + "\n", + "\n", + "It is also common to define\n", + "the function $Q$ as" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "C(\\boldsymbol{\\beta})=\\frac{1}{2n}\\sum_{i=0}^{n-1}\\left(y_i-\\tilde{y}_i\\right)^2,\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "since when taking the first derivative with respect to the unknown parameters $\\beta$, the factor of $2$ cancels out.\n", + "\n", + "\n", + "\n", + "\n", + "## Interpretations and optimizing our parameters\n", + "\n", + "The function" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "C(\\boldsymbol{\\beta})=\\frac{1}{n}\\left\\{\\left(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta}\\right)^T\\left(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta}\\right)\\right\\},\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "can be linked to the variance of the quantity $y_i$ if we interpret the latter as the mean value. \n", + "When linking below with the maximum likelihood approach below, we will indeed interpret $y_i$ as a mean value (see exercises)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "y_{i}=\\langle y_i \\rangle = \\beta_0x_{i,0}+\\beta_1x_{i,1}+\\beta_2x_{i,2}+\\dots+\\beta_{n-1}x_{i,n-1}+\\epsilon_i,\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "where $\\langle y_i \\rangle$ is the mean value. Keep in mind also that\n", + "till now we have treated $y_i$ as the exact value. Normally, the\n", + "response (dependent or outcome) variable $y_i$ the outcome of a\n", + "numerical experiment or another type of experiment and is thus only an\n", + "approximation to the true value. It is then always accompanied by an\n", + "error estimate, often limited to a statistical error estimate given by\n", + "the standard deviation discussed earlier. In the discussion here we\n", + "will treat $y_i$ as our exact value for the response variable.\n", + "\n", + "In order to find the parameters $\\beta_i$ we will then minimize the spread of $C(\\boldsymbol{\\beta})$, that is we are going to solve the problem" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "{\\displaystyle \\min_{\\boldsymbol{\\beta}\\in\n", + "{\\mathbb{R}}^{p}}}\\frac{1}{n}\\left\\{\\left(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta}\\right)^T\\left(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta}\\right)\\right\\}.\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "In practical terms it means we will require" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "\\frac{\\partial C(\\boldsymbol{\\beta})}{\\partial \\beta_j} = \\frac{\\partial }{\\partial \\beta_j}\\left[ \\frac{1}{n}\\sum_{i=0}^{n-1}\\left(y_i-\\beta_0x_{i,0}-\\beta_1x_{i,1}-\\beta_2x_{i,2}-\\dots-\\beta_{n-1}x_{i,n-1}\\right)^2\\right]=0,\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "which results in" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "\\frac{\\partial C(\\boldsymbol{\\beta})}{\\partial \\beta_j} = -\\frac{2}{n}\\left[ \\sum_{i=0}^{n-1}x_{ij}\\left(y_i-\\beta_0x_{i,0}-\\beta_1x_{i,1}-\\beta_2x_{i,2}-\\dots-\\beta_{n-1}x_{i,n-1}\\right)\\right]=0,\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "or in a matrix-vector form as" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "\\frac{\\partial C(\\boldsymbol{\\beta})}{\\partial \\boldsymbol{\\beta}} = 0 = \\boldsymbol{X}^T\\left( \\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta}\\right).\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Interpretations and optimizing our parameters\n", + "We can rewrite" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "\\frac{\\partial C(\\boldsymbol{\\beta})}{\\partial \\boldsymbol{\\beta}} = 0 = \\boldsymbol{X}^T\\left( \\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta}\\right),\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "as" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "\\boldsymbol{X}^T\\boldsymbol{y} = \\boldsymbol{X}^T\\boldsymbol{X}\\boldsymbol{\\beta},\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "and if the matrix $\\boldsymbol{X}^T\\boldsymbol{X}$ is invertible we have the solution" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "\\boldsymbol{\\beta} =\\left(\\boldsymbol{X}^T\\boldsymbol{X}\\right)^{-1}\\boldsymbol{X}^T\\boldsymbol{y}.\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "We note also that since our design matrix is defined as $\\boldsymbol{X}\\in\n", + "{\\mathbb{R}}^{n\\times p}$, the product $\\boldsymbol{X}^T\\boldsymbol{X} \\in\n", + "{\\mathbb{R}}^{p\\times p}$. In the above case we have that $p \\ll n$,\n", + "in our case $p=5$ meaning that we end up with inverting a small\n", + "$5\\times 5$ matrix. This is a rather common situation, in many cases we end up with low-dimensional\n", + "matrices to invert. The methods discussed here and for many other\n", + "supervised learning algorithms like classification with logistic\n", + "regression or support vector machines, exhibit dimensionalities which\n", + "allow for the usage of direct linear algebra methods such as **LU** decomposition or **Singular Value Decomposition** (SVD) for finding the inverse of the matrix\n", + "$\\boldsymbol{X}^T\\boldsymbol{X}$.\n", + "\n", + "\n", + "\n", + "## Interpretations and optimizing our parameters\n", + "The residuals $\\boldsymbol{\\epsilon}$ are in turn given by" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "\\boldsymbol{\\epsilon} = \\boldsymbol{y}-\\boldsymbol{\\tilde{y}} = \\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta},\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "and with" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "\\boldsymbol{X}^T\\left( \\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta}\\right)= 0,\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "we have" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "\\boldsymbol{X}^T\\boldsymbol{\\epsilon}=\\boldsymbol{X}^T\\left( \\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta}\\right)= 0,\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "meaning that the solution for $\\boldsymbol{\\beta}$ is the one which minimizes the residuals. Later we will link this with the maximum likelihood approach.\n", + "\n", + "\n", + "\n", + "\n", + "Let us now return to our nuclear binding energies and simply code the above equations. \n", + "\n", + "## Own code for Ordinary Least Squares\n", + "\n", + "It is rather straightforward to implement the matrix inversion and obtain the parameters $\\boldsymbol{\\beta}$. After having defined the matrix $\\boldsymbol{X}$ we simply need to \n", + "write" + ] + }, + { + "cell_type": "code", + "execution_count": 2, + "metadata": { + "collapsed": false + }, + "outputs": [], + "source": [ + "# matrix inversion to find beta\n", + "beta = np.linalg.inv(X.T.dot(X)).dot(X.T).dot(Energies)\n", + "# and then make the prediction\n", + "ytilde = X @ beta" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Alternatively, you can use the least squares functionality in **Numpy** as" + ] + }, + { + "cell_type": "code", + "execution_count": 3, + "metadata": { + "collapsed": false + }, + "outputs": [], + "source": [ + "fit = np.linalg.lstsq(X, Energies, rcond =None)[0]\n", + "ytildenp = np.dot(fit,X.T)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "And finally we plot our fit with and compare with data" + ] + }, + { + "cell_type": "code", + "execution_count": 4, + "metadata": { + "collapsed": false + }, + "outputs": [], + "source": [ + "Masses['Eapprox'] = ytilde\n", + "# Generate a plot comparing the experimental with the fitted values values.\n", + "fig, ax = plt.subplots()\n", + "ax.set_xlabel(r'$A = N + Z$')\n", + "ax.set_ylabel(r'$E_\\mathrm{bind}\\,/\\mathrm{MeV}$')\n", + "ax.plot(Masses['A'], Masses['Ebinding'], alpha=0.7, lw=2,\n", + " label='Ame2016')\n", + "ax.plot(Masses['A'], Masses['Eapprox'], alpha=0.7, lw=2, c='m',\n", + " label='Fit')\n", + "ax.legend()\n", + "save_fig(\"Masses2016OLS\")\n", + "plt.show()" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Adding error analysis and training set up\n", + "\n", + "We can easily test our fit by computing the $R2$ score that we discussed in connection with the functionality of _Scikit_Learn_ in the introductory slides.\n", + "Since we are not using _Scikit-Learn here we can define our own $R2$ function as" + ] + }, + { + "cell_type": "code", + "execution_count": 5, + "metadata": { + "collapsed": false + }, + "outputs": [], + "source": [ + "def R2(y_data, y_model):\n", + " return 1 - np.sum((y_data - y_model) ** 2) / np.sum((y_data - np.mean(y_model)) ** 2)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "and we would be using it as" + ] + }, + { + "cell_type": "code", + "execution_count": 6, + "metadata": { + "collapsed": false + }, + "outputs": [], + "source": [ + "print(R2(Energies,ytilde))" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "We can easily add our **MSE** score as" + ] + }, + { + "cell_type": "code", + "execution_count": 7, + "metadata": { + "collapsed": false + }, + "outputs": [], + "source": [ + "def MSE(y_data,y_model):\n", + " n = np.size(y_model)\n", + " return np.sum((y_data-y_model)**2)/n\n", + "\n", + "print(MSE(Energies,ytilde))" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "and finally the relative error as" + ] + }, + { + "cell_type": "code", + "execution_count": 8, + "metadata": { + "collapsed": false + }, + "outputs": [], + "source": [ + "def RelativeError(y_data,y_model):\n", + " return abs((y_data-y_model)/y_data)\n", + "print(RelativeError(Energies, ytilde))" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## The $\\chi^2$ function\n", + "\n", + "Normally, the response (dependent or outcome) variable $y_i$ is the\n", + "outcome of a numerical experiment or another type of experiment and is\n", + "thus only an approximation to the true value. It is then always\n", + "accompanied by an error estimate, often limited to a statistical error\n", + "estimate given by the standard deviation discussed earlier. In the\n", + "discussion here we will treat $y_i$ as our exact value for the\n", + "response variable.\n", + "\n", + "Introducing the standard deviation $\\sigma_i$ for each measurement\n", + "$y_i$, we define now the $\\chi^2$ function (omitting the $1/n$ term)\n", + "as" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "\\chi^2(\\boldsymbol{\\beta})=\\frac{1}{n}\\sum_{i=0}^{n-1}\\frac{\\left(y_i-\\tilde{y}_i\\right)^2}{\\sigma_i^2}=\\frac{1}{n}\\left\\{\\left(\\boldsymbol{y}-\\boldsymbol{\\tilde{y}}\\right)^T\\frac{1}{\\boldsymbol{\\Sigma^2}}\\left(\\boldsymbol{y}-\\boldsymbol{\\tilde{y}}\\right)\\right\\},\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "where the matrix $\\boldsymbol{\\Sigma}$ is a diagonal matrix with $\\sigma_i$ as matrix elements.\n", + "\n", + "\n", + "\n", + "## The $\\chi^2$ function\n", + "\n", + "In order to find the parameters $\\beta_i$ we will then minimize the spread of $\\chi^2(\\boldsymbol{\\beta})$ by requiring" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "\\frac{\\partial \\chi^2(\\boldsymbol{\\beta})}{\\partial \\beta_j} = \\frac{\\partial }{\\partial \\beta_j}\\left[ \\frac{1}{n}\\sum_{i=0}^{n-1}\\left(\\frac{y_i-\\beta_0x_{i,0}-\\beta_1x_{i,1}-\\beta_2x_{i,2}-\\dots-\\beta_{n-1}x_{i,n-1}}{\\sigma_i}\\right)^2\\right]=0,\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "which results in" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "\\frac{\\partial \\chi^2(\\boldsymbol{\\beta})}{\\partial \\beta_j} = -\\frac{2}{n}\\left[ \\sum_{i=0}^{n-1}\\frac{x_{ij}}{\\sigma_i}\\left(\\frac{y_i-\\beta_0x_{i,0}-\\beta_1x_{i,1}-\\beta_2x_{i,2}-\\dots-\\beta_{n-1}x_{i,n-1}}{\\sigma_i}\\right)\\right]=0,\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "or in a matrix-vector form as" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "\\frac{\\partial \\chi^2(\\boldsymbol{\\beta})}{\\partial \\boldsymbol{\\beta}} = 0 = \\boldsymbol{A}^T\\left( \\boldsymbol{b}-\\boldsymbol{A}\\boldsymbol{\\beta}\\right).\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "where we have defined the matrix $\\boldsymbol{A} =\\boldsymbol{X}/\\boldsymbol{\\Sigma}$ with matrix elements $a_{ij} = x_{ij}/\\sigma_i$ and the vector $\\boldsymbol{b}$ with elements $b_i = y_i/\\sigma_i$.\n", + "\n", + "\n", + "\n", + "## The $\\chi^2$ function\n", + "\n", + "We can rewrite" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "\\frac{\\partial \\chi^2(\\boldsymbol{\\beta})}{\\partial \\boldsymbol{\\beta}} = 0 = \\boldsymbol{A}^T\\left( \\boldsymbol{b}-\\boldsymbol{A}\\boldsymbol{\\beta}\\right),\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "as" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "\\boldsymbol{A}^T\\boldsymbol{b} = \\boldsymbol{A}^T\\boldsymbol{A}\\boldsymbol{\\beta},\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "and if the matrix $\\boldsymbol{A}^T\\boldsymbol{A}$ is invertible we have the solution" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "\\boldsymbol{\\beta} =\\left(\\boldsymbol{A}^T\\boldsymbol{A}\\right)^{-1}\\boldsymbol{A}^T\\boldsymbol{b}.\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## The $\\chi^2$ function\n", + "\n", + "If we then introduce the matrix" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "\\boldsymbol{H} = \\left(\\boldsymbol{A}^T\\boldsymbol{A}\\right)^{-1},\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "we have then the following expression for the parameters $\\beta_j$ (the matrix elements of $\\boldsymbol{H}$ are $h_{ij}$)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "\\beta_j = \\sum_{k=0}^{p-1}h_{jk}\\sum_{i=0}^{n-1}\\frac{y_i}{\\sigma_i}\\frac{x_{ik}}{\\sigma_i} = \\sum_{k=0}^{p-1}h_{jk}\\sum_{i=0}^{n-1}b_ia_{ik}\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "We state without proof the expression for the uncertainty in the parameters $\\beta_j$ as (we leave this as an exercise)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "\\sigma^2(\\beta_j) = \\sum_{i=0}^{n-1}\\sigma_i^2\\left( \\frac{\\partial \\beta_j}{\\partial y_i}\\right)^2,\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "resulting in" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "\\sigma^2(\\beta_j) = \\left(\\sum_{k=0}^{p-1}h_{jk}\\sum_{i=0}^{n-1}a_{ik}\\right)\\left(\\sum_{l=0}^{p-1}h_{jl}\\sum_{m=0}^{n-1}a_{ml}\\right) = h_{jj}!\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## The $\\chi^2$ function\n", + "The first step here is to approximate the function $y$ with a first-order polynomial, that is we write" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "y=y(x) \\rightarrow y(x_i) \\approx \\beta_0+\\beta_1 x_i.\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "By computing the derivatives of $\\chi^2$ with respect to $\\beta_0$ and $\\beta_1$ show that these are given by" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "\\frac{\\partial \\chi^2(\\boldsymbol{\\beta})}{\\partial \\beta_0} = -2\\left[ \\frac{1}{n}\\sum_{i=0}^{n-1}\\left(\\frac{y_i-\\beta_0-\\beta_1x_{i}}{\\sigma_i^2}\\right)\\right]=0,\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "and" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "\\frac{\\partial \\chi^2(\\boldsymbol{\\beta})}{\\partial \\beta_1} = -\\frac{2}{n}\\left[ \\sum_{i=0}^{n-1}x_i\\left(\\frac{y_i-\\beta_0-\\beta_1x_{i}}{\\sigma_i^2}\\right)\\right]=0.\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## The $\\chi^2$ function\n", + "\n", + "For a linear fit (a first-order polynomial) we don't need to invert a matrix!! \n", + "Defining" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "\\gamma = \\sum_{i=0}^{n-1}\\frac{1}{\\sigma_i^2},\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "\\gamma_x = \\sum_{i=0}^{n-1}\\frac{x_{i}}{\\sigma_i^2},\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "\\gamma_y = \\sum_{i=0}^{n-1}\\left(\\frac{y_i}{\\sigma_i^2}\\right),\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "\\gamma_{xx} = \\sum_{i=0}^{n-1}\\frac{x_ix_{i}}{\\sigma_i^2},\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "\\gamma_{xy} = \\sum_{i=0}^{n-1}\\frac{y_ix_{i}}{\\sigma_i^2},\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "we obtain" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "\\beta_0 = \\frac{\\gamma_{xx}\\gamma_y-\\gamma_x\\gamma_y}{\\gamma\\gamma_{xx}-\\gamma_x^2},\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "\\beta_1 = \\frac{\\gamma_{xy}\\gamma-\\gamma_x\\gamma_y}{\\gamma\\gamma_{xx}-\\gamma_x^2}.\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "This approach (different linear and non-linear regression) suffers\n", + "often from both being underdetermined and overdetermined in the\n", + "unknown coefficients $\\beta_i$. A better approach is to use the\n", + "Singular Value Decomposition (SVD) method discussed below. Or using\n", + "Lasso and Ridge regression. See below.\n", + "\n", + "\n", + "\n", + "\n", + "## Fitting an Equation of State for Dense Nuclear Matter\n", + "\n", + "Before we continue, let us introduce yet another example. We are going to fit the\n", + "nuclear equation of state using results from many-body calculations.\n", + "The equation of state we have made available here, as function of\n", + "density, has been derived using modern nucleon-nucleon potentials with\n", + "[the addition of three-body\n", + "forces](https://www.sciencedirect.com/science/article/pii/S0370157399001106). This\n", + "time the file is presented as a standard **csv** file.\n", + "\n", + "The beginning of the Python code here is similar to what you have seen before,\n", + "with the same initializations and declarations. We use also **pandas**\n", + "again, rather extensively in order to organize our data.\n", + "\n", + "The difference now is that we use **Scikit-Learn's** regression tools\n", + "instead of our own matrix inversion implementation. Furthermore, we\n", + "sneak in **Ridge** regression (to be discussed below) which includes a\n", + "hyperparameter $\\lambda$, also to be explained below.\n", + "\n", + "## The code" + ] + }, + { + "cell_type": "code", + "execution_count": 9, + "metadata": { + "collapsed": false + }, + "outputs": [], + "source": [ + "# Common imports\n", + "import os\n", + "import numpy as np\n", + "import pandas as pd\n", + "import matplotlib.pyplot as plt\n", + "import matplotlib.pyplot as plt\n", + "import sklearn.linear_model as skl\n", + "from sklearn.metrics import mean_squared_error, r2_score, mean_absolute_error\n", + "\n", + "# Where to save the figures and data files\n", + "PROJECT_ROOT_DIR = \"Results\"\n", + "FIGURE_ID = \"Results/FigureFiles\"\n", + "DATA_ID = \"DataFiles/\"\n", + "\n", + "if not os.path.exists(PROJECT_ROOT_DIR):\n", + " os.mkdir(PROJECT_ROOT_DIR)\n", + "\n", + "if not os.path.exists(FIGURE_ID):\n", + " os.makedirs(FIGURE_ID)\n", + "\n", + "if not os.path.exists(DATA_ID):\n", + " os.makedirs(DATA_ID)\n", + "\n", + "def image_path(fig_id):\n", + " return os.path.join(FIGURE_ID, fig_id)\n", + "\n", + "def data_path(dat_id):\n", + " return os.path.join(DATA_ID, dat_id)\n", + "\n", + "def save_fig(fig_id):\n", + " plt.savefig(image_path(fig_id) + \".png\", format='png')\n", + "\n", + "infile = open(data_path(\"EoS.csv\"),'r')\n", + "\n", + "# Read the EoS data as csv file and organize the data into two arrays with density and energies\n", + "EoS = pd.read_csv(infile, names=('Density', 'Energy'))\n", + "EoS['Energy'] = pd.to_numeric(EoS['Energy'], errors='coerce')\n", + "EoS = EoS.dropna()\n", + "Energies = EoS['Energy']\n", + "Density = EoS['Density']\n", + "# The design matrix now as function of various polytrops\n", + "X = np.zeros((len(Density),4))\n", + "X[:,3] = Density**(4.0/3.0)\n", + "X[:,2] = Density\n", + "X[:,1] = Density**(2.0/3.0)\n", + "X[:,0] = 1\n", + "\n", + "# We use now Scikit-Learn's linear regressor and ridge regressor\n", + "# OLS part\n", + "clf = skl.LinearRegression().fit(X, Energies)\n", + "ytilde = clf.predict(X)\n", + "EoS['Eols'] = ytilde\n", + "# The mean squared error \n", + "print(\"Mean squared error: %.2f\" % mean_squared_error(Energies, ytilde))\n", + "# Explained variance score: 1 is perfect prediction \n", + "print('Variance score: %.2f' % r2_score(Energies, ytilde))\n", + "# Mean absolute error \n", + "print('Mean absolute error: %.2f' % mean_absolute_error(Energies, ytilde))\n", + "print(clf.coef_, clf.intercept_)\n", + "\n", + "# The Ridge regression with a hyperparameter lambda = 0.1\n", + "_lambda = 0.1\n", + "clf_ridge = skl.Ridge(alpha=_lambda).fit(X, Energies)\n", + "yridge = clf_ridge.predict(X)\n", + "EoS['Eridge'] = yridge\n", + "# The mean squared error \n", + "print(\"Mean squared error: %.2f\" % mean_squared_error(Energies, yridge))\n", + "# Explained variance score: 1 is perfect prediction \n", + "print('Variance score: %.2f' % r2_score(Energies, yridge))\n", + "# Mean absolute error \n", + "print('Mean absolute error: %.2f' % mean_absolute_error(Energies, yridge))\n", + "print(clf_ridge.coef_, clf_ridge.intercept_)\n", + "\n", + "fig, ax = plt.subplots()\n", + "ax.set_xlabel(r'$\\rho[\\mathrm{fm}^{-3}]$')\n", + "ax.set_ylabel(r'Energy per particle')\n", + "ax.plot(EoS['Density'], EoS['Energy'], alpha=0.7, lw=2,\n", + " label='Theoretical data')\n", + "ax.plot(EoS['Density'], EoS['Eols'], alpha=0.7, lw=2, c='m',\n", + " label='OLS')\n", + "ax.plot(EoS['Density'], EoS['Eridge'], alpha=0.7, lw=2, c='g',\n", + " label='Ridge $\\lambda = 0.1$')\n", + "ax.legend()\n", + "save_fig(\"EoSfitting\")\n", + "plt.show()" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "The above simple polynomial in density $\\rho$ gives an excellent fit\n", + "to the data. Can you give an interpretation of the various powers of $\\rho$?\n", + "\n", + "We note also that there is a small deviation between the\n", + "standard OLS and the Ridge regression at higher densities. We discuss this in more detail\n", + "below.\n", + "\n", + "\n", + "## Splitting our Data in Training and Test data\n", + "\n", + "It is normal in essentially all Machine Learning studies to split the\n", + "data in a training set and a test set (sometimes also an additional\n", + "validation set). **Scikit-Learn** has an own function for this. There\n", + "is no explicit recipe for how much data should be included as training\n", + "data and say test data. An accepted rule of thumb is to use\n", + "approximately $2/3$ to $4/5$ of the data as training data. We will\n", + "postpone a discussion of this splitting to the end of these notes and\n", + "our discussion of the so-called **bias-variance** tradeoff. Here we\n", + "limit ourselves to repeat the above equation of state fitting example\n", + "but now splitting the data into a training set and a test set." + ] + }, + { + "cell_type": "code", + "execution_count": 10, + "metadata": { + "collapsed": false + }, + "outputs": [], + "source": [ + "import os\n", + "import numpy as np\n", + "import pandas as pd\n", + "import matplotlib.pyplot as plt\n", + "from sklearn.model_selection import train_test_split\n", + "# Where to save the figures and data files\n", + "PROJECT_ROOT_DIR = \"Results\"\n", + "FIGURE_ID = \"Results/FigureFiles\"\n", + "DATA_ID = \"DataFiles/\"\n", + "\n", + "if not os.path.exists(PROJECT_ROOT_DIR):\n", + " os.mkdir(PROJECT_ROOT_DIR)\n", + "\n", + "if not os.path.exists(FIGURE_ID):\n", + " os.makedirs(FIGURE_ID)\n", + "\n", + "if not os.path.exists(DATA_ID):\n", + " os.makedirs(DATA_ID)\n", + "\n", + "def image_path(fig_id):\n", + " return os.path.join(FIGURE_ID, fig_id)\n", + "\n", + "def data_path(dat_id):\n", + " return os.path.join(DATA_ID, dat_id)\n", + "\n", + "def save_fig(fig_id):\n", + " plt.savefig(image_path(fig_id) + \".png\", format='png')\n", + "\n", + "def R2(y_data, y_model):\n", + " return 1 - np.sum((y_data - y_model) ** 2) / np.sum((y_data - np.mean(y_model)) ** 2)\n", + "def MSE(y_data,y_model):\n", + " n = np.size(y_model)\n", + " return np.sum((y_data-y_model)**2)/n\n", + "\n", + "infile = open(data_path(\"EoS.csv\"),'r')\n", + "\n", + "# Read the EoS data as csv file and organized into two arrays with density and energies\n", + "EoS = pd.read_csv(infile, names=('Density', 'Energy'))\n", + "EoS['Energy'] = pd.to_numeric(EoS['Energy'], errors='coerce')\n", + "EoS = EoS.dropna()\n", + "Energies = EoS['Energy']\n", + "Density = EoS['Density']\n", + "# The design matrix now as function of various polytrops\n", + "X = np.zeros((len(Density),5))\n", + "X[:,0] = 1\n", + "X[:,1] = Density**(2.0/3.0)\n", + "X[:,2] = Density\n", + "X[:,3] = Density**(4.0/3.0)\n", + "X[:,4] = Density**(5.0/3.0)\n", + "# We split the data in test and training data\n", + "X_train, X_test, y_train, y_test = train_test_split(X, Energies, test_size=0.2)\n", + "# matrix inversion to find beta\n", + "beta = np.linalg.inv(X_train.T.dot(X_train)).dot(X_train.T).dot(y_train)\n", + "# and then make the prediction\n", + "ytilde = X_train @ beta\n", + "print(\"Training R2\")\n", + "print(R2(y_train,ytilde))\n", + "print(\"Training MSE\")\n", + "print(MSE(y_train,ytilde))\n", + "ypredict = X_test @ beta\n", + "print(\"Test R2\")\n", + "print(R2(y_test,ypredict))\n", + "print(\"Test MSE\")\n", + "print(MSE(y_test,ypredict))" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## The singular value decomposition\n", + "\n", + "\n", + "The examples we have looked at so far are cases where we normally can\n", + "invert the matrix $\\boldsymbol{X}^T\\boldsymbol{X}$. Using a polynomial expansion as we\n", + "did both for the masses and the fitting of the equation of state,\n", + "leads to row vectors of the design matrix which are essentially\n", + "orthogonal due to the polynomial character of our model. This may\n", + "however not the be case in general and a standard matrix inversion\n", + "algorithm based on say LU decomposition may lead to singularities. We will see an example of this below when we try to fit\n", + "the coupling constant of the widely used Ising model. \n", + "There is however a way to partially circumvent this problem and also gain some insight about the ordinary least squares approach. \n", + "\n", + "This is given by the **Singular Value Decomposition** algorithm, perhaps\n", + "the most powerful linear algebra algorithm. Let us look at a\n", + "different example where we may have problems with the standard matrix\n", + "inversion algorithm. Thereafter we dive into the math of the SVD.\n", + "\n", + "\n", + "\n", + "## The Ising model\n", + "\n", + "The one-dimensional Ising model with nearest neighbor interaction, no\n", + "external field and a constant coupling constant $J$ is given by" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\n", + "
\n", + "\n", + "$$\n", + "\\begin{equation}\n", + " H = -J \\sum_{k}^L s_k s_{k + 1},\n", + "\\label{_auto1} \\tag{1}\n", + "\\end{equation}\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "where $s_i \\in \\{-1, 1\\}$ and $s_{N + 1} = s_1$. The number of spins\n", + "in the system is determined by $L$. For the one-dimensional system\n", + "there is no phase transition.\n", + "\n", + "We will look at a system of $L = 40$ spins with a coupling constant of\n", + "$J = 1$. To get enough training data we will generate 10000 states\n", + "with their respective energies." + ] + }, + { + "cell_type": "code", + "execution_count": 11, + "metadata": { + "collapsed": false + }, + "outputs": [], + "source": [ + "import numpy as np\n", + "import matplotlib.pyplot as plt\n", + "from mpl_toolkits.axes_grid1 import make_axes_locatable\n", + "import seaborn as sns\n", + "import scipy.linalg as scl\n", + "from sklearn.model_selection import train_test_split\n", + "import tqdm\n", + "sns.set(color_codes=True)\n", + "cmap_args=dict(vmin=-1., vmax=1., cmap='seismic')\n", + "\n", + "L = 40\n", + "n = int(1e4)\n", + "\n", + "spins = np.random.choice([-1, 1], size=(n, L))\n", + "J = 1.0\n", + "\n", + "energies = np.zeros(n)\n", + "\n", + "for i in range(n):\n", + " energies[i] = - J * np.dot(spins[i], np.roll(spins[i], 1))" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Here we use ordinary least squares\n", + "regression to predict the energy for the nearest neighbor\n", + "one-dimensional Ising model on a ring, i.e., the endpoints wrap\n", + "around. We will use linear regression to fit a value for\n", + "the coupling constant to achieve this.\n", + "\n", + "## Reformulating the problem to suit regression\n", + "\n", + "A more general form for the one-dimensional Ising model is" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\n", + "
\n", + "\n", + "$$\n", + "\\begin{equation}\n", + " H = - \\sum_j^L \\sum_k^L s_j s_k J_{jk}.\n", + "\\label{_auto2} \\tag{2}\n", + "\\end{equation}\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Here we allow for interactions beyond the nearest neighbors and a state dependent\n", + "coupling constant. This latter expression can be formulated as\n", + "a matrix-product" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\n", + "
\n", + "\n", + "$$\n", + "\\begin{equation}\n", + " \\boldsymbol{H} = \\boldsymbol{X} J,\n", + "\\label{_auto3} \\tag{3}\n", + "\\end{equation}\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "where $X_{jk} = s_j s_k$ and $J$ is a matrix which consists of the\n", + "elements $-J_{jk}$. This form of writing the energy fits perfectly\n", + "with the form utilized in linear regression, that is" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\n", + "
\n", + "\n", + "$$\n", + "\\begin{equation}\n", + " \\boldsymbol{y} = \\boldsymbol{X}\\boldsymbol{\\beta} + \\boldsymbol{\\epsilon},\n", + "\\label{_auto4} \\tag{4}\n", + "\\end{equation}\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "We split the data in training and test data as discussed in the previous example" + ] + }, + { + "cell_type": "code", + "execution_count": 12, + "metadata": { + "collapsed": false + }, + "outputs": [], + "source": [ + "X = np.zeros((n, L ** 2))\n", + "for i in range(n):\n", + " X[i] = np.outer(spins[i], spins[i]).ravel()\n", + "y = energies\n", + "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Linear regression\n", + "\n", + "In the ordinary least squares method we choose the cost function" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\n", + "
\n", + "\n", + "$$\n", + "\\begin{equation}\n", + " C(\\boldsymbol{X}, \\boldsymbol{\\beta})= \\frac{1}{n}\\left\\{(\\boldsymbol{X}\\boldsymbol{\\beta} - \\boldsymbol{y})^T(\\boldsymbol{X}\\boldsymbol{\\beta} - \\boldsymbol{y})\\right\\}.\n", + "\\label{_auto5} \\tag{5}\n", + "\\end{equation}\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "We then find the extremal point of $C$ by taking the derivative with respect to $\\boldsymbol{\\beta}$ as discussed above.\n", + "This yields the expression for $\\boldsymbol{\\beta}$ to be" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "\\boldsymbol{\\beta} = \\frac{\\boldsymbol{X}^T \\boldsymbol{y}}{\\boldsymbol{X}^T \\boldsymbol{X}},\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "which immediately imposes some requirements on $\\boldsymbol{X}$ as there must exist\n", + "an inverse of $\\boldsymbol{X}^T \\boldsymbol{X}$. If the expression we are modeling contains an\n", + "intercept, i.e., a constant term, we must make sure that the\n", + "first column of $\\boldsymbol{X}$ consists of $1$. We do this here" + ] + }, + { + "cell_type": "code", + "execution_count": 13, + "metadata": { + "collapsed": false + }, + "outputs": [], + "source": [ + "X_train_own = np.concatenate(\n", + " (np.ones(len(X_train))[:, np.newaxis], X_train),\n", + " axis=1\n", + ")\n", + "X_test_own = np.concatenate(\n", + " (np.ones(len(X_test))[:, np.newaxis], X_test),\n", + " axis=1\n", + ")" + ] + }, + { + "cell_type": "code", + "execution_count": 14, + "metadata": { + "collapsed": false + }, + "outputs": [], + "source": [ + "def ols_inv(x: np.ndarray, y: np.ndarray) -> np.ndarray:\n", + " return scl.inv(x.T @ x) @ (x.T @ y)\n", + "beta = ols_inv(X_train_own, y_train)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Singular Value decomposition\n", + "\n", + "Doing the inversion directly turns out to be a bad idea since the matrix\n", + "$\\boldsymbol{X}^T\\boldsymbol{X}$ is singular. An alternative approach is to use the **singular\n", + "value decomposition**. Using the definition of the Moore-Penrose\n", + "pseudoinverse we can write the equation for $\\boldsymbol{\\beta}$ as" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "\\boldsymbol{\\beta} = \\boldsymbol{X}^{+}\\boldsymbol{y},\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "where the pseudoinverse of $\\boldsymbol{X}$ is given by" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "\\boldsymbol{X}^{+} = \\frac{\\boldsymbol{X}^T}{\\boldsymbol{X}^T\\boldsymbol{X}}.\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Using singular value decomposition we can decompose the matrix $\\boldsymbol{X} = \\boldsymbol{U}\\boldsymbol{\\Sigma} \\boldsymbol{V}^T$,\n", + "where $\\boldsymbol{U}$ and $\\boldsymbol{V}$ are orthogonal(unitary) matrices and $\\boldsymbol{\\Sigma}$ contains the singular values (more details below).\n", + "where $X^{+} = V\\Sigma^{+} U^T$. This reduces the equation for\n", + "$\\omega$ to" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\n", + "
\n", + "\n", + "$$\n", + "\\begin{equation}\n", + " \\boldsymbol{\\beta} = \\boldsymbol{V}\\boldsymbol{\\Sigma}^{+} \\boldsymbol{U}^T \\boldsymbol{y}.\n", + "\\label{_auto6} \\tag{6}\n", + "\\end{equation}\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Note that solving this equation by actually doing the pseudoinverse\n", + "(which is what we will do) is not a good idea as this operation scales\n", + "as $\\mathcal{O}(n^3)$, where $n$ is the number of elements in a\n", + "general matrix. Instead, doing $QR$-factorization and solving the\n", + "linear system as an equation would reduce this down to\n", + "$\\mathcal{O}(n^2)$ operations." + ] + }, + { + "cell_type": "code", + "execution_count": 15, + "metadata": { + "collapsed": false + }, + "outputs": [], + "source": [ + "def ols_svd(x: np.ndarray, y: np.ndarray) -> np.ndarray:\n", + " u, s, v = scl.svd(x)\n", + " return v.T @ scl.pinv(scl.diagsvd(s, u.shape[0], v.shape[0])) @ u.T @ y" + ] + }, + { + "cell_type": "code", + "execution_count": 16, + "metadata": { + "collapsed": false + }, + "outputs": [], + "source": [ + "beta = ols_svd(X_train_own,y_train)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "When extracting the $J$-matrix we need to make sure that we remove the intercept, as is done here" + ] + }, + { + "cell_type": "code", + "execution_count": 17, + "metadata": { + "collapsed": false + }, + "outputs": [], + "source": [ + "J = beta[1:].reshape(L, L)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "A way of looking at the coefficients in $J$ is to plot the matrices as images." + ] + }, + { + "cell_type": "code", + "execution_count": 18, + "metadata": { + "collapsed": false + }, + "outputs": [], + "source": [ + "fig = plt.figure(figsize=(20, 14))\n", + "im = plt.imshow(J, **cmap_args)\n", + "plt.title(\"OLS\", fontsize=18)\n", + "plt.xticks(fontsize=18)\n", + "plt.yticks(fontsize=18)\n", + "cb = fig.colorbar(im)\n", + "cb.ax.set_yticklabels(cb.ax.get_yticklabels(), fontsize=18)\n", + "plt.show()" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "It is interesting to note that OLS\n", + "considers both $J_{j, j + 1} = -0.5$ and $J_{j, j - 1} = -0.5$ as\n", + "valid matrix elements for $J$.\n", + "In our discussion below on hyperparameters and Ridge and Lasso regression we will see that\n", + "this problem can be removed, partly and only with Lasso regression. \n", + "\n", + "In this case our matrix inversion was actually possible. The obvious question now is what is the mathematics behind the SVD?\n", + "\n", + "\n", + "## Linear Regression Problems\n", + "\n", + "One of the typical problems we encounter with linear regression, in particular \n", + "when the matrix $\\boldsymbol{X}$ (our so-called design matrix) is high-dimensional, \n", + "are problems with near singular or singular matrices. The column vectors of $\\boldsymbol{X}$ \n", + "may be linearly dependent, normally referred to as super-collinearity. \n", + "This means that the matrix may be rank deficient and it is basically impossible to \n", + "to model the data using linear regression. As an example, consider the matrix" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "\\begin{align*}\n", + "\\mathbf{X} & = \\left[\n", + "\\begin{array}{rrr}\n", + "1 & -1 & 2\n", + "\\\\\n", + "1 & 0 & 1\n", + "\\\\\n", + "1 & 2 & -1\n", + "\\\\\n", + "1 & 1 & 0\n", + "\\end{array} \\right]\n", + "\\end{align*}\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "The columns of $\\boldsymbol{X}$ are linearly dependent. We see this easily since the \n", + "the first column is the row-wise sum of the other two columns. The rank (more correct,\n", + "the column rank) of a matrix is the dimension of the space spanned by the\n", + "column vectors. Hence, the rank of $\\mathbf{X}$ is equal to the number\n", + "of linearly independent columns. In this particular case the matrix has rank 2.\n", + "\n", + "Super-collinearity of an $(n \\times p)$-dimensional design matrix $\\mathbf{X}$ implies\n", + "that the inverse of the matrix $\\boldsymbol{X}^T\\boldsymbol{x}$ (the matrix we need to invert to solve the linear regression equations) is non-invertible. If we have a square matrix that does not have an inverse, we say this matrix singular. The example here demonstrates this" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "\\begin{align*}\n", + "\\boldsymbol{X} & = \\left[\n", + "\\begin{array}{rr}\n", + "1 & -1\n", + "\\\\\n", + "1 & -1\n", + "\\end{array} \\right].\n", + "\\end{align*}\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "We see easily that $\\mbox{det}(\\boldsymbol{X}) = x_{11} x_{22} - x_{12} x_{21} = 1 \\times (-1) - 1 \\times (-1) = 0$. Hence, $\\mathbf{X}$ is singular and its inverse is undefined.\n", + "This is equivalent to saying that the matrix $\\boldsymbol{X}$ has at least an eigenvalue which is zero.\n", + "\n", + "\n", + "## Fixing the singularity\n", + "\n", + "If our design matrix $\\boldsymbol{X}$ which enters the linear regression problem" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\n", + "
\n", + "\n", + "$$\n", + "\\begin{equation}\n", + "\\boldsymbol{\\beta} = (\\boldsymbol{X}^{T} \\boldsymbol{X})^{-1} \\boldsymbol{X}^{T} \\boldsymbol{y},\n", + "\\label{_auto7} \\tag{7}\n", + "\\end{equation}\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "has linearly dependent column vectors, we will not be able to compute the inverse\n", + "of $\\boldsymbol{X}^T\\boldsymbol{X}$ and we cannot find the parameters (estimators) $\\beta_i$. \n", + "The estimators are only well-defined if $(\\boldsymbol{X}^{T}\\boldsymbol{X})^{-1}$ exits. \n", + "This is more likely to happen when the matrix $\\boldsymbol{X}$ is high-dimensional. In this case it is likely to encounter a situation where \n", + "the regression parameters $\\beta_i$ cannot be estimated.\n", + "\n", + "A cheap *ad hoc* approach is simply to add a small diagonal component to the matrix to invert, that is we change" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "\\boldsymbol{X}^{T} \\boldsymbol{X} \\rightarrow \\boldsymbol{X}^{T} \\boldsymbol{X}+\\lambda \\boldsymbol{I},\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "where $\\boldsymbol{I}$ is the identity matrix. When we discuss **Ridge** regression this is actually what we end up evaluating. The parameter $\\lambda$ is called a hyperparameter. More about this later. \n", + "\n", + "\n", + "\n", + "## Basic math of the SVD\n", + "\n", + "\n", + "From standard linear algebra we know that a square matrix $\\boldsymbol{X}$ can be diagonalized if and only it is \n", + "a so-called [normal matrix](https://en.wikipedia.org/wiki/Normal_matrix), that is if $\\boldsymbol{X}\\in {\\mathbb{R}}^{n\\times n}$\n", + "we have $\\boldsymbol{X}\\boldsymbol{X}^T=\\boldsymbol{X}^T\\boldsymbol{X}$ or if $\\boldsymbol{X}\\in {\\mathbb{C}}^{n\\times n}$ we have $\\boldsymbol{X}\\boldsymbol{X}^{\\dagger}=\\boldsymbol{X}^{\\dagger}\\boldsymbol{X}$.\n", + "The matrix has then a set of eigenpairs" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "(\\lambda_1,\\boldsymbol{u}_1),\\dots, (\\lambda_n,\\boldsymbol{u}_n),\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "and the eigenvalues are given by the diagonal matrix" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "\\boldsymbol{\\Sigma}=\\mathrm{Diag}(\\lambda_1, \\dots,\\lambda_n).\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "The matrix $\\boldsymbol{X}$ can be written in terms of an orthogonal/unitary transformation $\\boldsymbol{U}$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "\\boldsymbol{X} = \\boldsymbol{U}\\boldsymbol{\\Sigma}\\boldsymbol{V}^T,\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "with $\\boldsymbol{U}\\boldsymbol{U}^T=\\boldsymbol{I}$ or $\\boldsymbol{U}\\boldsymbol{U}^{\\dagger}=\\boldsymbol{I}$.\n", + "\n", + "Not all square matrices are diagonalizable. A matrix like the one discussed above" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "\\boldsymbol{X} = \\begin{bmatrix} \n", + "1& -1 \\\\\n", + "1& -1\\\\\n", + "\\end{bmatrix}\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "is not diagonalizable, it is a so-called [defective matrix](https://en.wikipedia.org/wiki/Defective_matrix). It is easy to see that the condition\n", + "$\\boldsymbol{X}\\boldsymbol{X}^T=\\boldsymbol{X}^T\\boldsymbol{X}$ is not fulfilled. \n", + "\n", + "\n", + "## The SVD, a Fantastic Algorithm\n", + "\n", + "\n", + "However, and this is the strength of the SVD algorithm, any general\n", + "matrix $\\boldsymbol{X}$ can be decomposed in terms of a diagonal matrix and\n", + "two orthogonal/unitary matrices. The [Singular Value Decompostion\n", + "(SVD) theorem](https://en.wikipedia.org/wiki/Singular_value_decomposition)\n", + "states that a general $m\\times n$ matrix $\\boldsymbol{X}$ can be written in\n", + "terms of a diagonal matrix $\\boldsymbol{\\Sigma}$ of dimensionality $n\\times n$\n", + "and two orthognal matrices $\\boldsymbol{U}$ and $\\boldsymbol{V}$, where the first has\n", + "dimensionality $m \\times m$ and the last dimensionality $n\\times n$.\n", + "We have then" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "\\boldsymbol{X} = \\boldsymbol{U}\\boldsymbol{\\Sigma}\\boldsymbol{V}^T\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "As an example, the above defective matrix can be decomposed as" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "\\boldsymbol{X} = \\frac{1}{\\sqrt{2}}\\begin{bmatrix} 1& 1 \\\\ 1& -1\\\\ \\end{bmatrix} \\begin{bmatrix} 2& 0 \\\\ 0& 0\\\\ \\end{bmatrix} \\frac{1}{\\sqrt{2}}\\begin{bmatrix} 1& -1 \\\\ 1& 1\\\\ \\end{bmatrix}=\\boldsymbol{U}\\boldsymbol{\\Sigma}\\boldsymbol{V}^T,\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "with eigenvalues $\\sigma_1=2$ and $\\sigma_2=0$. \n", + "The SVD exits always! \n", + "\n", + "\n", + "## Another Example\n", + "\n", + "Consider the following matrix which can be SVD decomposed as" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "\\boldsymbol{X} = \\frac{1}{15}\\begin{bmatrix} 14 & 2\\\\ 4 & 22\\\\ 16 & 13\\end{matrix}=\\frac{1}{3}\\begin{bmatrix} 1& 2 & 2 \\\\ 2& -1 & 1\\\\ 2 & 1& -2\\end{bmatrix} \\begin{bmatrix} 2& 0 \\\\ 0& 1\\\\ 0 & 0\\end{bmatrix}\\frac{1}{5}\\begin{bmatrix} 3& 4 \\\\ 4& -3\\end{bmatrix}=\\boldsymbol{U}\\boldsymbol{\\Sigma}\\boldsymbol{V}^T.\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "This is a $3\\times 2$ matrix which is decomposed in terms of a\n", + "$3\\times 3$ matrix $\\boldsymbol{U}$, and a $2\\times 2$ matrix $\\boldsymbol{V}$. It is easy to see\n", + "that $\\boldsymbol{U}$ and $\\boldsymbol{V}$ are orthogonal (how?). \n", + "\n", + "And the SVD\n", + "decomposition (singular values) gives eigenvalues \n", + "$\\sigma_i\\geq\\sigma_{i+1}$ for all $i$ and for dimensions larger than $i=2$, the\n", + "eigenvalues (singular values) are zero.\n", + "\n", + "In the general case, where our design matrix $\\boldsymbol{X}$ has dimension\n", + "$n\\times p$, the matrix is thus decomposed into an $n\\times n$\n", + "orthogonal matrix $\\boldsymbol{U}$, a $p\\times p$ orthogonal matrix $\\boldsymbol{V}$\n", + "and a diagonal matrix $\\boldsymbol{\\Sigma}$ with $r=\\mathrm{min}(n,p)$\n", + "singular values $\\sigma_i\\lg 0$ on the main diagonal and zeros filling\n", + "the rest of the matrix. There are at most $p$ singular values\n", + "assuming that $n > p$. In our regression examples for the nuclear\n", + "masses and the equation of state this is indeed the case, while for\n", + "the Ising model we have $p > n$. These are often cases that lead to\n", + "near singular or singular matrices.\n", + "\n", + "The columns of $\\boldsymbol{U}$ are called the left singular vectors while the columns of $\\boldsymbol{V}$ are the right singular vectors.\n", + "\n", + "## Economy-size SVD\n", + "\n", + "If we assume that $n > p$, then our matrix $\\boldsymbol{U}$ has dimension $n\n", + "\\times n$. The last $n-p$ columns of $\\boldsymbol{U}$ become however\n", + "irrelevant in our calculations since they are multiplied with the\n", + "zeros in $\\boldsymbol{\\Sigma}$.\n", + "\n", + "The economy-size decomposition removes extra rows or columns of zeros\n", + "from the diagonal matrix of singular values, $\\boldsymbol{\\Sigma}$, along with the columns\n", + "in either $\\boldsymbol{U}$ or $\\boldsymbol{V}$ that multiply those zeros in the expression. \n", + "Removing these zeros and columns can improve execution time\n", + "and reduce storage requirements without compromising the accuracy of\n", + "the decomposition.\n", + "\n", + "If $n > p$, we keep only the first $p$ columns of $\\boldsymbol{U}$ and $\\boldsymbol{\\Sigma}$ has dimension $p\\times p$. \n", + "If $p > n$, then only the first $n$ columns of $\\boldsymbol{V}$ are computed and $\\boldsymbol{\\Sigma}$ has dimension $n\\times n$.\n", + "The $n=p$ case is obvious, we retain the full SVD. \n", + "In general the economy-size SVD leads to less FLOPS and still conserving the desired accuracy.\n", + "\n", + "## Mathematical Properties\n", + "\n", + "There are several interesting mathematical properties which will be\n", + "relevant when we are going to discuss the differences between say\n", + "ordinary least squares (OLS) and **Ridge** regression.\n", + "\n", + "We have from OLS that the parameters of the linear approximation are given by" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "\\boldsymbol{\\tilde{y}} = \\boldsymbol{X}\\boldsymbol{\\beta} = \\boldsymbol{X}\\left(\\boldsymbol{X}^T\\boldsymbol{X}\\right)^{-1}\\boldsymbol{X}^T\\boldsymbol{y}.\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "The matrix to invert can be rewritten in terms of our SVD decomposition as" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "\\boldsymbol{X}^T\\boldsymbol{X} = \\boldsymbol{V}\\boldsymbol{\\Sigma}^T\\boldsymbol{U}^T\\boldsymbol{U}\\boldsymbol{\\Sigma}\\boldsymbol{V}^T.\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Using the orthogonality properties of $\\boldsymbol{U}$ we have" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "\\boldsymbol{X}^T\\boldsymbol{X} = \\boldsymbol{V}\\boldsymbol{\\Sigma}^T\\boldsymbol{\\Sigma}\\boldsymbol{V}^T = \\boldsymbol{V}\\boldsymbol{D}\\boldsymbol{V}^T,\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "with $\\boldsymbol{D}$ being a diagonal matrix with values along the diagonal given by the singular values squared. \n", + "\n", + "This means that" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "(\\boldsymbol{X}^T\\boldsymbol{X})\\boldsymbol{V} = \\boldsymbol{V}\\boldsymbol{D},\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "that is the eigenvectors of $(\\boldsymbol{X}^T\\boldsymbol{X})$ are given by the columns of the right singular matrix of $\\boldsymbol{X}$ and the eigenvalues are the squared singular values. It is easy to show (show this) that" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "(\\boldsymbol{X}\\boldsymbol{X}^T)\\boldsymbol{U} = \\boldsymbol{U}\\boldsymbol{D},\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "that is, the eigenvectors of $(\\boldsymbol{X}\\boldsymbol{X})^T$ are the columns of the left singular matrix and the eigenvalues are the same. \n", + "\n", + "Going back to our OLS equation we have" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "\\boldsymbol{X}\\boldsymbol{\\beta} = \\boldsymbol{X}\\left(\\boldsymbol{V}\\boldsymbol{D}\\boldsymbol{V}^T \\right)^{-1}\\boldsymbol{X}^T\\boldsymbol{y}=\\boldsymbol{U\\Sigma V^T}\\left(\\boldsymbol{V}\\boldsymbol{D}\\boldsymbol{V}^T \\right)^{-1}(\\boldsymbol{U\\Sigma V^T})^T\\boldsymbol{y}=\\boldsymbol{U}\\boldsymbol{U}^T\\boldsymbol{y}.\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "We will come back to this expression when we discuss Ridge regression. \n", + "\n", + "\n", + "## Ridge and LASSO Regression\n", + "\n", + "Let us remind ourselves about the expression for the standard Mean Squared Error (MSE) which we used to define our cost function and the equations for the ordinary least squares (OLS) method, that is \n", + "our optimization problem is" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "{\\displaystyle \\min_{\\boldsymbol{\\beta}\\in {\\mathbb{R}}^{p}}}\\frac{1}{n}\\left\\{\\left(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta}\\right)^T\\left(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta}\\right)\\right\\}.\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "or we can state it as" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "{\\displaystyle \\min_{\\boldsymbol{\\beta}\\in\n", + "{\\mathbb{R}}^{p}}}\\frac{1}{n}\\sum_{i=0}^{n-1}\\left(y_i-\\tilde{y}_i\\right)^2=\\frac{1}{n}\\vert\\vert \\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta}\\vert\\vert_2^2,\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "where we have used the definition of a norm-2 vector, that is" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "\\vert\\vert \\boldsymbol{x}\\vert\\vert_2 = \\sqrt{\\sum_i x_i^2}.\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "By minimizing the above equation with respect to the parameters\n", + "$\\boldsymbol{\\beta}$ we could then obtain an analytical expression for the\n", + "parameters $\\boldsymbol{\\beta}$. We can add a regularization parameter $\\lambda$ by\n", + "defining a new cost function to be optimized, that is" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "{\\displaystyle \\min_{\\boldsymbol{\\beta}\\in\n", + "{\\mathbb{R}}^{p}}}\\frac{1}{n}\\vert\\vert \\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta}\\vert\\vert_2^2+\\lambda\\vert\\vert \\boldsymbol{\\beta}\\vert\\vert_2^2\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "which leads to the Ridge regression minimization problem where we\n", + "require that $\\vert\\vert \\boldsymbol{\\beta}\\vert\\vert_2^2\\le t$, where $t$ is\n", + "a finite number larger than zero. By defining" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "C(\\boldsymbol{X},\\boldsymbol{\\beta})=\\frac{1}{n}\\vert\\vert \\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta}\\vert\\vert_2^2+\\lambda\\vert\\vert \\boldsymbol{\\beta}\\vert\\vert_1,\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "we have a new optimization equation" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "{\\displaystyle \\min_{\\boldsymbol{\\beta}\\in\n", + "{\\mathbb{R}}^{p}}}\\frac{1}{n}\\vert\\vert \\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta}\\vert\\vert_2^2+\\lambda\\vert\\vert \\boldsymbol{\\beta}\\vert\\vert_1\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "which leads to Lasso regression. Lasso stands for least absolute shrinkage and selection operator. \n", + "\n", + "Here we have defined the norm-1 as" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "\\vert\\vert \\boldsymbol{x}\\vert\\vert_1 = \\sum_i \\vert x_i\\vert.\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## More on Ridge Regression\n", + "\n", + "Using the matrix-vector expression for Ridge regression," + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "C(\\boldsymbol{X},\\boldsymbol{\\beta})=\\frac{1}{n}\\left\\{(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta})^T(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta})\\right\\}+\\lambda\\boldsymbol{\\beta}^T\\boldsymbol{\\beta},\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "by taking the derivatives with respect to $\\boldsymbol{\\beta}$ we obtain then\n", + "a slightly modified matrix inversion problem which for finite values\n", + "of $\\lambda$ does not suffer from singularity problems. We obtain" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "\\boldsymbol{\\beta}^{\\mathrm{Ridge}} = \\left(\\boldsymbol{X}^T\\boldsymbol{X}+\\lambda\\boldsymbol{I}\\right)^{-1}\\boldsymbol{X}^T\\boldsymbol{y},\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "with $\\boldsymbol{I}$ being a $p\\times p$ identity matrix with the constraint that" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "\\sum_{i=0}^{p-1} \\beta_i^2 \\leq t,\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "with $t$ a finite positive number. \n", + "\n", + "We see that Ridge regression is nothing but the standard\n", + "OLS with a modified diagonal term added to $\\boldsymbol{X}^T\\boldsymbol{X}$. The\n", + "consequences, in particular for our discussion of the bias-variance\n", + "are rather interesting.\n", + "\n", + "Furthermore, if we use the result above in terms of the SVD decomposition (our analysis was done for the OLS method), we had" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "(\\boldsymbol{X}\\boldsymbol{X}^T)\\boldsymbol{U} = \\boldsymbol{U}\\boldsymbol{D}.\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "We can analyse the OLS solutions in terms of the eigenvectors (the columns) of the right singular value matrix $\\boldsymbol{U}$ as" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "\\boldsymbol{X}\\boldsymbol{\\beta} = \\boldsymbol{X}\\left(\\boldsymbol{V}\\boldsymbol{D}\\boldsymbol{V}^T \\right)^{-1}\\boldsymbol{X}^T\\boldsymbol{y}=\\boldsymbol{U\\Sigma V^T}\\left(\\boldsymbol{V}\\boldsymbol{D}\\boldsymbol{V}^T \\right)^{-1}(\\boldsymbol{U\\Sigma V^T})^T\\boldsymbol{y}=\\boldsymbol{U}\\boldsymbol{U}^T\\boldsymbol{y}\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "For Ridge regression this becomes" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "\\boldsymbol{X}\\boldsymbol{\\beta}^{\\mathrm{Ridge}} = \\boldsymbol{U\\Sigma V^T}\\left(\\boldsymbol{V}\\boldsymbol{D}\\boldsymbol{V}^T+\\lambda\\boldsymbol{I} \\right)^{-1}(\\boldsymbol{U\\Sigma V^T})^T\\boldsymbol{y}=\\sum_{j=0}^{p-1}\\boldsymbol{u}_j\\boldsymbol{u}_j^T\\frac{\\sigma_j^2}{\\sigma_j^2+\\lambda}\\boldsymbol{y},\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "with the vectors $\\boldsymbol{u}_j$ being the columns of $\\boldsymbol{U}$. \n", + "\n", + "## Interpreting the Ridge results\n", + "\n", + "Since $\\lambda \\geq 0$, it means that compared to OLS, we have" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "\\frac{\\sigma_j^2}{\\sigma_j^2+\\lambda} \\leq 1.\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Ridge regression finds the coordinates of $\\boldsymbol{y}$ with respect to the\n", + "orthonormal basis $\\boldsymbol{U}$, it then shrinks the coordinates by\n", + "$\\frac{\\sigma_j^2}{\\sigma_j^2+\\lambda}$. Recall that the SVD has\n", + "eigenvalues ordered in a descending way, that is $\\sigma_i \\geq\n", + "\\sigma_{i+1}$.\n", + "\n", + "For small eigenvalues $\\sigma_i$ it means that their contributions become less important, a fact which can be used to reduce the number of degrees of freedom.\n", + "Actually, calculating the variance of $\\boldsymbol{X}\\boldsymbol{v}_j$ shows that this quantity is equal to $\\sigma_j^2/n$.\n", + "With a parameter $\\lambda$ we can thus shrink the role of specific parameters. \n", + "\n", + "\n", + "## More interpretations\n", + "\n", + "For the sake of simplicity, let us assume that the design matrix is orthonormal, that is" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "\\boldsymbol{X}^T\\boldsymbol{X}=(\\boldsymbol{X}^T\\boldsymbol{X})^{-1} =\\boldsymbol{I}.\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "In this case the standard OLS results in" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "\\boldsymbol{\\beta}^{\\mathrm{OLS}} = \\boldsymbol{X}^T\\boldsymbol{y}=\\sum_{i=0}^{p-1}\\boldsymbol{u}_j\\boldsymbol{u}_j^T\\boldsymbol{y},\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "and" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "\\boldsymbol{\\beta}^{\\mathrm{Ridge}} = \\left(\\boldsymbol{I}+\\lambda\\boldsymbol{I}\\right)^{-1}\\boldsymbol{X}^T\\boldsymbol{y}=\\left(1+\\lambda\\right)^{-1}\\boldsymbol{\\beta}^{\\mathrm{OLS}},\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "that is the Ridge estimator scales the OLS estimator by the inverse of a factor $1+\\lambda$, and\n", + "the Ridge estimator converges to zero when the hyperparameter goes to\n", + "infinity.\n", + "\n", + "We will come back to more interpreations after we have gone through some of the statistical analysis part. \n", + "\n", + "For more discussions of Ridge and Lasso regression, [Wessel van Wieringen's](https://arxiv.org/abs/1509.09169) article is highly recommended.\n", + "Similarly, [Mehta et al's article](https://arxiv.org/abs/1803.08823) is also recommended.\n", + "\n", + "## Where are we going?\n", + "\n", + "Before we proceed, we need to rethink what we have been doing. In our\n", + "eager to fit the data, we have omitted several important elements in\n", + "our regression analysis. In what follows we will\n", + "1. look at statistical properties, including a discussion of mean values, variance and the so-called bias-variance tradeoff\n", + "\n", + "2. introduce resampling techniques like cross-validation, bootstrapping and jackknife and more\n", + "\n", + "This will allow us to link the standard linear algebra methods we have discussed above to a statistical interpretation of the methods. \n", + "\n", + "\n", + "\n", + "\n", + "\n", + "## Resampling methods\n", + "Resampling methods are an indispensable tool in modern\n", + "statistics. They involve repeatedly drawing samples from a training\n", + "set and refitting a model of interest on each sample in order to\n", + "obtain additional information about the fitted model. For example, in\n", + "order to estimate the variability of a linear regression fit, we can\n", + "repeatedly draw different samples from the training data, fit a linear\n", + "regression to each new sample, and then examine the extent to which\n", + "the resulting fits differ. Such an approach may allow us to obtain\n", + "information that would not be available from fitting the model only\n", + "once using the original training sample.\n", + "\n", + "\n", + "\n", + "## Resampling approaches can be computationally expensive\n", + "\n", + "Resampling approaches can be computationally expensive, because they\n", + "involve fitting the same statistical method multiple times using\n", + "different subsets of the training data. However, due to recent\n", + "advances in computing power, the computational requirements of\n", + "resampling methods generally are not prohibitive. In this chapter, we\n", + "discuss two of the most commonly used resampling methods,\n", + "cross-validation and the bootstrap. Both methods are important tools\n", + "in the practical application of many statistical learning\n", + "procedures. For example, cross-validation can be used to estimate the\n", + "test error associated with a given statistical learning method in\n", + "order to evaluate its performance, or to select the appropriate level\n", + "of flexibility. The process of evaluating a model’s performance is\n", + "known as model assessment, whereas the process of selecting the proper\n", + "level of flexibility for a model is known as model selection. The\n", + "bootstrap is widely used.\n", + "\n", + "\n", + "\n", + "## Why resampling methods ?\n", + "**Statistical analysis.**\n", + "\n", + "\n", + "* Our simulations can be treated as *computer experiments*. This is particularly the case for Monte Carlo methods\n", + "\n", + "* The results can be analysed with the same statistical tools as we would use analysing experimental data.\n", + "\n", + "* As in all experiments, we are looking for expectation values and an estimate of how accurate they are, i.e., possible sources for errors.\n", + "\n", + " \n", + "\n", + "## Statistical analysis\n", + "\n", + "* As in other experiments, many numerical experiments have two classes of errors:\n", + "\n", + " * Statistical errors\n", + "\n", + " * Systematical errors\n", + "\n", + "\n", + "* Statistical errors can be estimated using standard tools from statistics\n", + "\n", + "* Systematical errors are method specific and must be treated differently from case to case.\n", + "\n", + " \n", + "\n", + "## Statistics\n", + "The *probability distribution function (PDF)* is a function\n", + "$p(x)$ on the domain which, in the discrete case, gives us the\n", + "probability or relative frequency with which these values of $X$ occur:" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "p(x) = \\mathrm{prob}(X=x)\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "In the continuous case, the PDF does not directly depict the\n", + "actual probability. Instead we define the probability for the\n", + "stochastic variable to assume any value on an infinitesimal interval\n", + "around $x$ to be $p(x)dx$. The continuous function $p(x)$ then gives us\n", + "the *density* of the probability rather than the probability\n", + "itself. The probability for a stochastic variable to assume any value\n", + "on a non-infinitesimal interval $[a,\\,b]$ is then just the integral:" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "\\mathrm{prob}(a\\leq X\\leq b) = \\int_a^b p(x)dx\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Qualitatively speaking, a stochastic variable represents the values of\n", + "numbers chosen as if by chance from some specified PDF so that the\n", + "selection of a large set of these numbers reproduces this PDF.\n", + "\n", + "\n", + "\n", + "\n", + "## Statistics, moments\n", + "A particularly useful class of special expectation values are the\n", + "*moments*. The $n$-th moment of the PDF $p$ is defined as\n", + "follows:" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "\\langle x^n\\rangle \\equiv \\int\\! x^n p(x)\\,dx\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "The zero-th moment $\\langle 1\\rangle$ is just the normalization condition of\n", + "$p$. The first moment, $\\langle x\\rangle$, is called the *mean* of $p$\n", + "and often denoted by the letter $\\mu$:" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "\\langle x\\rangle = \\mu \\equiv \\int\\! x p(x)\\,dx\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Statistics, central moments\n", + "A special version of the moments is the set of *central moments*,\n", + "the n-th central moment defined as:" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "\\langle (x-\\langle x \\rangle )^n\\rangle \\equiv \\int\\! (x-\\langle x\\rangle)^n p(x)\\,dx\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "The zero-th and first central moments are both trivial, equal $1$ and\n", + "$0$, respectively. But the second central moment, known as the\n", + "*variance* of $p$, is of particular interest. For the stochastic\n", + "variable $X$, the variance is denoted as $\\sigma^2_X$ or $\\mathrm{var}(X)$:" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\n", + "
\n", + "\n", + "$$\n", + "\\begin{equation}\n", + "\\sigma^2_X\\ \\ =\\ \\ \\mathrm{var}(X) = \\langle (x-\\langle x\\rangle)^2\\rangle =\n", + "\\int\\! (x-\\langle x\\rangle)^2 p(x)\\,dx\n", + "\\label{_auto8} \\tag{8}\n", + "\\end{equation}\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\n", + "
\n", + "\n", + "$$\n", + "\\begin{equation} \n", + " = \\int\\! \\left(x^2 - 2 x \\langle x\\rangle^{2} +\n", + " \\langle x\\rangle^2\\right)p(x)\\,dx\n", + "\\label{_auto9} \\tag{9}\n", + "\\end{equation}\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\n", + "
\n", + "\n", + "$$\n", + "\\begin{equation} \n", + " = \\langle x^2\\rangle - 2 \\langle x\\rangle\\langle x\\rangle + \\langle x\\rangle^2\n", + "\\label{_auto10} \\tag{10}\n", + "\\end{equation}\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\n", + "
\n", + "\n", + "$$\n", + "\\begin{equation} \n", + " = \\langle x^2\\rangle - \\langle x\\rangle^2\n", + "\\label{_auto11} \\tag{11}\n", + "\\end{equation}\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "The square root of the variance, $\\sigma =\\sqrt{\\langle (x-\\langle x\\rangle)^2\\rangle}$ is called the *standard deviation* of $p$. It is clearly just the RMS (root-mean-square)\n", + "value of the deviation of the PDF from its mean value, interpreted\n", + "qualitatively as the *spread* of $p$ around its mean.\n", + "\n", + "\n", + "\n", + "## Statistics, covariance\n", + "Another important quantity is the so called covariance, a variant of\n", + "the above defined variance. Consider again the set $\\{X_i\\}$ of $n$\n", + "stochastic variables (not necessarily uncorrelated) with the\n", + "multivariate PDF $P(x_1,\\dots,x_n)$. The *covariance* of two\n", + "of the stochastic variables, $X_i$ and $X_j$, is defined as follows:" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "\\mathrm{cov}(X_i,\\,X_j) \\equiv \\langle (x_i-\\langle x_i\\rangle)(x_j-\\langle x_j\\rangle)\\rangle\n", + "\\nonumber\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\n", + "
\n", + "\n", + "$$\n", + "\\begin{equation} \n", + "=\n", + "\\int\\!\\cdots\\!\\int\\!(x_i-\\langle x_i \\rangle)(x_j-\\langle x_j \\rangle)\\,\n", + "P(x_1,\\dots,x_n)\\,dx_1\\dots dx_n\n", + "\\label{eq:def_covariance} \\tag{12}\n", + "\\end{equation}\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "with" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "\\langle x_i\\rangle =\n", + "\\int\\!\\cdots\\!\\int\\!x_i\\,P(x_1,\\dots,x_n)\\,dx_1\\dots dx_n\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Statistics, more covariance\n", + "If we consider the above covariance as a matrix $C_{ij}=\\mathrm{cov}(X_i,\\,X_j)$, then the diagonal elements are just the familiar\n", + "variances, $C_{ii} = \\mathrm{cov}(X_i,\\,X_i) = \\mathrm{var}(X_i)$. It turns out that\n", + "all the off-diagonal elements are zero if the stochastic variables are\n", + "uncorrelated. This is easy to show, keeping in mind the linearity of\n", + "the expectation value. Consider the stochastic variables $X_i$ and\n", + "$X_j$, ($i\\neq j$):" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\n", + "
\n", + "\n", + "$$\n", + "\\begin{equation}\n", + "\\mathrm{cov}(X_i,\\,X_j) = \\langle(x_i-\\langle x_i\\rangle)(x_j-\\langle x_j\\rangle)\\rangle\n", + "\\label{_auto12} \\tag{13}\n", + "\\end{equation}\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\n", + "
\n", + "\n", + "$$\n", + "\\begin{equation} \n", + "=\\langle x_i x_j - x_i\\langle x_j\\rangle - \\langle x_i\\rangle x_j + \\langle x_i\\rangle\\langle x_j\\rangle\\rangle \n", + "\\label{_auto13} \\tag{14}\n", + "\\end{equation}\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\n", + "
\n", + "\n", + "$$\n", + "\\begin{equation} \n", + "=\\langle x_i x_j\\rangle - \\langle x_i\\langle x_j\\rangle\\rangle - \\langle \\langle x_i\\rangle x_j\\rangle +\n", + "\\langle \\langle x_i\\rangle\\langle x_j\\rangle\\rangle\n", + "\\label{_auto14} \\tag{15}\n", + "\\end{equation}\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\n", + "
\n", + "\n", + "$$\n", + "\\begin{equation} \n", + "=\\langle x_i x_j\\rangle - \\langle x_i\\rangle\\langle x_j\\rangle - \\langle x_i\\rangle\\langle x_j\\rangle +\n", + "\\langle x_i\\rangle\\langle x_j\\rangle\n", + "\\label{_auto15} \\tag{16}\n", + "\\end{equation}\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\n", + "
\n", + "\n", + "$$\n", + "\\begin{equation} \n", + "=\\langle x_i x_j\\rangle - \\langle x_i\\rangle\\langle x_j\\rangle\n", + "\\label{_auto16} \\tag{17}\n", + "\\end{equation}\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Statistics, independent variables\n", + "If $X_i$ and $X_j$ are independent, we get \n", + "$\\langle x_i x_j\\rangle =\\langle x_i\\rangle\\langle x_j\\rangle$, resulting in $\\mathrm{cov}(X_i, X_j) = 0\\ \\ (i\\neq j)$.\n", + "\n", + "Also useful for us is the covariance of linear combinations of\n", + "stochastic variables. Let $\\{X_i\\}$ and $\\{Y_i\\}$ be two sets of\n", + "stochastic variables. Let also $\\{a_i\\}$ and $\\{b_i\\}$ be two sets of\n", + "scalars. Consider the linear combination:" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "U = \\sum_i a_i X_i \\qquad V = \\sum_j b_j Y_j\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "By the linearity of the expectation value" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "\\mathrm{cov}(U, V) = \\sum_{i,j}a_i b_j \\mathrm{cov}(X_i, Y_j)\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Statistics, more variance\n", + "Now, since the variance is just $\\mathrm{var}(X_i) = \\mathrm{cov}(X_i, X_i)$, we get\n", + "the variance of the linear combination $U = \\sum_i a_i X_i$:" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\n", + "
\n", + "\n", + "$$\n", + "\\begin{equation}\n", + "\\mathrm{var}(U) = \\sum_{i,j}a_i a_j \\mathrm{cov}(X_i, X_j)\n", + "\\label{eq:variance_linear_combination} \\tag{18}\n", + "\\end{equation}\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "And in the special case when the stochastic variables are\n", + "uncorrelated, the off-diagonal elements of the covariance are as we\n", + "know zero, resulting in:" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "1\n", + "1\n", + "3\n", + " \n", + "<\n", + "<\n", + "<\n", + "!\n", + "!\n", + "M\n", + "A\n", + "T\n", + "H\n", + "_\n", + "B\n", + "L\n", + "O\n", + "C\n", + "K" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "\\mathrm{var}(\\sum_i a_i X_i) = \\sum_i a_i^2 \\mathrm{var}(X_i)\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "which will become very useful in our study of the error in the mean\n", + "value of a set of measurements.\n", + "\n", + "\n", + "\n", + "## Statistics and stochastic processes\n", + "A *stochastic process* is a process that produces sequentially a\n", + "chain of values:" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "\\{x_1, x_2,\\dots\\,x_k,\\dots\\}.\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "We will call these\n", + "values our *measurements* and the entire set as our measured\n", + "*sample*. The action of measuring all the elements of a sample\n", + "we will call a stochastic *experiment* since, operationally,\n", + "they are often associated with results of empirical observation of\n", + "some physical or mathematical phenomena; precisely an experiment. We\n", + "assume that these values are distributed according to some \n", + "PDF $p_X^{\\phantom X}(x)$, where $X$ is just the formal symbol for the\n", + "stochastic variable whose PDF is $p_X^{\\phantom X}(x)$. Instead of\n", + "trying to determine the full distribution $p$ we are often only\n", + "interested in finding the few lowest moments, like the mean\n", + "$\\mu_X^{\\phantom X}$ and the variance $\\sigma_X^{\\phantom X}$.\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "## Statistics and sample variables\n", + "In practical situations a sample is always of finite size. Let that\n", + "size be $n$. The expectation value of a sample, the *sample mean*, is then defined as follows:" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "\\bar{x}_n \\equiv \\frac{1}{n}\\sum_{k=1}^n x_k\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "The *sample variance* is:" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "\\mathrm{var}(x) \\equiv \\frac{1}{n}\\sum_{k=1}^n (x_k - \\bar{x}_n)^2\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "its square root being the *standard deviation of the sample*. The\n", + "*sample covariance* is:" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "\\mathrm{cov}(x)\\equiv\\frac{1}{n}\\sum_{kl}(x_k - \\bar{x}_n)(x_l - \\bar{x}_n)\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Statistics, sample variance and covariance\n", + "Note that the sample variance is the sample covariance without the\n", + "cross terms. In a similar manner as the covariance in Eq. ([12](#eq:def_covariance)) is a measure of the correlation between\n", + "two stochastic variables, the above defined sample covariance is a\n", + "measure of the sequential correlation between succeeding measurements\n", + "of a sample.\n", + "\n", + "These quantities, being known experimental values, differ\n", + "significantly from and must not be confused with the similarly named\n", + "quantities for stochastic variables, mean $\\mu_X$, variance $\\mathrm{var}(X)$\n", + "and covariance $\\mathrm{cov}(X,Y)$.\n", + "\n", + "\n", + "\n", + "## Statistics, law of large numbers\n", + "The law of large numbers\n", + "states that as the size of our sample grows to infinity, the sample\n", + "mean approaches the true mean $\\mu_X^{\\phantom X}$ of the chosen PDF:" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "\\lim_{n\\to\\infty}\\bar{x}_n = \\mu_X^{\\phantom X}\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "The sample mean $\\bar{x}_n$ works therefore as an estimate of the true\n", + "mean $\\mu_X^{\\phantom X}$.\n", + "\n", + "What we need to find out is how good an approximation $\\bar{x}_n$ is to\n", + "$\\mu_X^{\\phantom X}$. In any stochastic measurement, an estimated\n", + "mean is of no use to us without a measure of its error. A quantity\n", + "that tells us how well we can reproduce it in another experiment. We\n", + "are therefore interested in the PDF of the sample mean itself. Its\n", + "standard deviation will be a measure of the spread of sample means,\n", + "and we will simply call it the *error* of the sample mean, or\n", + "just sample error, and denote it by $\\mathrm{err}_X^{\\phantom X}$. In\n", + "practice, we will only be able to produce an *estimate* of the\n", + "sample error since the exact value would require the knowledge of the\n", + "true PDFs behind, which we usually do not have.\n", + "\n", + "\n", + "\n", + "\n", + "## Statistics, more on sample error\n", + "Let us first take a look at what happens to the sample error as the\n", + "size of the sample grows. In a sample, each of the measurements $x_i$\n", + "can be associated with its own stochastic variable $X_i$. The\n", + "stochastic variable $\\overline X_n$ for the sample mean $\\bar{x}_n$ is\n", + "then just a linear combination, already familiar to us:" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "\\overline X_n = \\frac{1}{n}\\sum_{i=1}^n X_i\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "All the coefficients are just equal $1/n$. The PDF of $\\overline X_n$,\n", + "denoted by $p_{\\overline X_n}(x)$ is the desired PDF of the sample\n", + "means.\n", + "\n", + "\n", + "\n", + "## Statistics\n", + "The probability density of obtaining a sample mean $\\bar x_n$\n", + "is the product of probabilities of obtaining arbitrary values $x_1,\n", + "x_2,\\dots,x_n$ with the constraint that the mean of the set $\\{x_i\\}$\n", + "is $\\bar x_n$:" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "p_{\\overline X_n}(x) = \\int p_X^{\\phantom X}(x_1)\\cdots\n", + "\\int p_X^{\\phantom X}(x_n)\\ \n", + "\\delta\\!\\left(x - \\frac{x_1+x_2+\\dots+x_n}{n}\\right)dx_n \\cdots dx_1\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "And in particular we are interested in its variance $\\mathrm{var}(\\overline X_n)$.\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "## Statistics, central limit theorem\n", + "It is generally not possible to express $p_{\\overline X_n}(x)$ in a\n", + "closed form given an arbitrary PDF $p_X^{\\phantom X}$ and a number\n", + "$n$. But for the limit $n\\to\\infty$ it is possible to make an\n", + "approximation. The very important result is called *the central limit theorem*. It tells us that as $n$ goes to infinity,\n", + "$p_{\\overline X_n}(x)$ approaches a Gaussian distribution whose mean\n", + "and variance equal the true mean and variance, $\\mu_{X}^{\\phantom X}$\n", + "and $\\sigma_{X}^{2}$, respectively:" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\n", + "
\n", + "\n", + "$$\n", + "\\begin{equation}\n", + "\\lim_{n\\to\\infty} p_{\\overline X_n}(x) =\n", + "\\left(\\frac{n}{2\\pi\\mathrm{var}(X)}\\right)^{1/2}\n", + "e^{-\\frac{n(x-\\bar x_n)^2}{2\\mathrm{var}(X)}}\n", + "\\label{eq:central_limit_gaussian} \\tag{19}\n", + "\\end{equation}\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Statistics, more technicalities\n", + "The desired variance\n", + "$\\mathrm{var}(\\overline X_n)$, i.e. the sample error squared\n", + "$\\mathrm{err}_X^2$, is given by:" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\n", + "
\n", + "\n", + "$$\n", + "\\begin{equation}\n", + "\\mathrm{err}_X^2 = \\mathrm{var}(\\overline X_n) = \\frac{1}{n^2}\n", + "\\sum_{ij} \\mathrm{cov}(X_i, X_j)\n", + "\\label{eq:error_exact} \\tag{20}\n", + "\\end{equation}\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "We see now that in order to calculate the exact error of the sample\n", + "with the above expression, we would need the true means\n", + "$\\mu_{X_i}^{\\phantom X}$ of the stochastic variables $X_i$. To\n", + "calculate these requires that we know the true multivariate PDF of all\n", + "the $X_i$. But this PDF is unknown to us, we have only got the measurements of\n", + "one sample. The best we can do is to let the sample itself be an\n", + "estimate of the PDF of each of the $X_i$, estimating all properties of\n", + "$X_i$ through the measurements of the sample.\n", + "\n", + "\n", + "\n", + "\n", + "## Statistics\n", + "Our estimate of $\\mu_{X_i}^{\\phantom X}$ is then the sample mean $\\bar x$\n", + "itself, in accordance with the the central limit theorem:" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "\\mu_{X_i}^{\\phantom X} = \\langle x_i\\rangle \\approx \\frac{1}{n}\\sum_{k=1}^n x_k = \\bar x\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Using $\\bar x$ in place of $\\mu_{X_i}^{\\phantom X}$ we can give an\n", + "*estimate* of the covariance in Eq. ([20](#eq:error_exact))" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "\\mathrm{cov}(X_i, X_j) = \\langle (x_i-\\langle x_i\\rangle)(x_j-\\langle x_j\\rangle)\\rangle\n", + "\\approx\\langle (x_i - \\bar x)(x_j - \\bar{x})\\rangle,\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "resulting in" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "\\frac{1}{n} \\sum_{l}^n \\left(\\frac{1}{n}\\sum_{k}^n (x_k -\\bar x_n)(x_l - \\bar x_n)\\right)=\\frac{1}{n}\\frac{1}{n} \\sum_{kl} (x_k -\\bar x_n)(x_l - \\bar x_n)=\\frac{1}{n}\\mathrm{cov}(x)\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Statistics and sample variance\n", + "By the same procedure we can use the sample variance as an\n", + "estimate of the variance of any of the stochastic variables $X_i$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "\\mathrm{var}(X_i)=\\langle x_i - \\langle x_i\\rangle\\rangle \\approx \\langle x_i - \\bar x_n\\rangle\\nonumber,\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "which is approximated as" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\n", + "
\n", + "\n", + "$$\n", + "\\begin{equation}\n", + "\\mathrm{var}(X_i)\\approx \\frac{1}{n}\\sum_{k=1}^n (x_k - \\bar x_n)=\\mathrm{var}(x)\n", + "\\label{eq:var_estimate_i_think} \\tag{21}\n", + "\\end{equation}\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Now we can calculate an estimate of the error\n", + "$\\mathrm{err}_X^{\\phantom X}$ of the sample mean $\\bar x_n$:" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "\\mathrm{err}_X^2\n", + "=\\frac{1}{n^2}\\sum_{ij} \\mathrm{cov}(X_i, X_j) \\nonumber\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "\\approx\\frac{1}{n^2}\\sum_{ij}\\frac{1}{n}\\mathrm{cov}(x) =\\frac{1}{n^2}n^2\\frac{1}{n}\\mathrm{cov}(x)\\nonumber\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\n", + "
\n", + "\n", + "$$\n", + "\\begin{equation} \n", + "=\\frac{1}{n}\\mathrm{cov}(x)\n", + "\\label{eq:error_estimate} \\tag{22}\n", + "\\end{equation}\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "which is nothing but the sample covariance divided by the number of\n", + "measurements in the sample.\n", + "\n", + "\n", + "\n", + "## Statistics, uncorrelated results\n", + "\n", + "In the special case that the measurements of the sample are\n", + "uncorrelated (equivalently the stochastic variables $X_i$ are\n", + "uncorrelated) we have that the off-diagonal elements of the covariance\n", + "are zero. This gives the following estimate of the sample error:" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "\\mathrm{err}_X^2=\\frac{1}{n^2}\\sum_{ij} \\mathrm{cov}(X_i, X_j) =\n", + "\\frac{1}{n^2} \\sum_i \\mathrm{var}(X_i),\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "resulting in" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\n", + "
\n", + "\n", + "$$\n", + "\\begin{equation}\n", + "\\mathrm{err}_X^2\\approx \\frac{1}{n^2} \\sum_i \\mathrm{var}(x)= \\frac{1}{n}\\mathrm{var}(x)\n", + "\\label{eq:error_estimate_uncorrel} \\tag{23}\n", + "\\end{equation}\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "where in the second step we have used Eq. ([21](#eq:var_estimate_i_think)).\n", + "The error of the sample is then just its standard deviation divided by\n", + "the square root of the number of measurements the sample contains.\n", + "This is a very useful formula which is easy to compute. It acts as a\n", + "first approximation to the error, but in numerical experiments, we\n", + "cannot overlook the always present correlations.\n", + "\n", + "\n", + "\n", + "## Statistics, computations\n", + "For computational purposes one usually splits up the estimate of\n", + "$\\mathrm{err}_X^2$, given by Eq. ([22](#eq:error_estimate)), into two\n", + "parts" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "\\mathrm{err}_X^2 = \\frac{1}{n}\\mathrm{var}(x) + \\frac{1}{n}(\\mathrm{cov}(x)-\\mathrm{var}(x)),\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "which equals" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\n", + "
\n", + "\n", + "$$\n", + "\\begin{equation}\n", + "\\frac{1}{n^2}\\sum_{k=1}^n (x_k - \\bar x_n)^2 +\\frac{2}{n^2}\\sum_{k\n", + "
\n", + "\n", + "$$\n", + "\\begin{equation} \n", + "=\\frac{\\tau}{n}\\cdot\\mathrm{var}(x)\n", + "\\label{eq:error_estimate_corr_time} \\tag{25}\n", + "\\end{equation}\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "and we see that $\\mathrm{err}_X$ can be expressed in terms the\n", + "uncorrelated sample variance times a correction factor $\\tau$ which\n", + "accounts for the correlation between measurements. We call this\n", + "correction factor the *autocorrelation time*:" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\n", + "
\n", + "\n", + "$$\n", + "\\begin{equation}\n", + "\\tau = 1+2\\sum_{d=1}^{n-1}\\kappa_d\n", + "\\label{eq:autocorrelation_time} \\tag{26}\n", + "\\end{equation}\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Statistics, effective number of correlations\n", + "For a correlation free experiment, $\\tau$\n", + "equals 1. From the point of view of\n", + "eq. ([25](#eq:error_estimate_corr_time)) we can interpret a sequential\n", + "correlation as an effective reduction of the number of measurements by\n", + "a factor $\\tau$. The effective number of measurements becomes:" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "n_\\mathrm{eff} = \\frac{n}{\\tau}\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "To neglect the autocorrelation time $\\tau$ will always cause our\n", + "simple uncorrelated estimate of $\\mathrm{err}_X^2\\approx \\mathrm{var}(x)/n$ to\n", + "be less than the true sample error. The estimate of the error will be\n", + "too *good*. On the other hand, the calculation of the full\n", + "autocorrelation time poses an efficiency problem if the set of\n", + "measurements is very large.\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "## Linking the regression analysis with a statistical interpretation\n", + "\n", + "Finally, we are going to discuss several statistical properties which can be obtained in terms of analytical expressions. \n", + "The\n", + "advantage of doing linear regression is that we actually end up with\n", + "analytical expressions for several statistical quantities. \n", + "Standard least squares and Ridge regression allow us to\n", + "derive quantities like the variance and other expectation values in a\n", + "rather straightforward way.\n", + "\n", + "\n", + "It is assumed that $\\varepsilon_i\n", + "\\sim \\mathcal{N}(0, \\sigma^2)$ and the $\\varepsilon_{i}$ are\n", + "independent, i.e.:" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "\\begin{align*} \n", + "\\mbox{Cov}(\\varepsilon_{i_1},\n", + "\\varepsilon_{i_2}) & = \\left\\{ \\begin{array}{lcc} \\sigma^2 & \\mbox{if}\n", + "& i_1 = i_2, \\\\ 0 & \\mbox{if} & i_1 \\not= i_2. \\end{array} \\right.\n", + "\\end{align*}\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "The randomness of $\\varepsilon_i$ implies that\n", + "$\\mathbf{y}_i$ is also a random variable. In particular,\n", + "$\\mathbf{y}_i$ is normally distributed, because $\\varepsilon_i \\sim\n", + "\\mathcal{N}(0, \\sigma^2)$ and $\\mathbf{X}_{i,\\ast} \\, \\boldsymbol{\\beta}$ is a\n", + "non-random scalar. To specify the parameters of the distribution of\n", + "$\\mathbf{y}_i$ we need to calculate its first two moments. \n", + "\n", + "Recall that $\\boldsymbol{X}$ is a matrix of dimensionality $n\\times p$. The\n", + "notation above $\\mathbf{X}_{i,\\ast}$ means that we are looking at the\n", + "row number $i$ and perform a sum over all values $p$.\n", + "\n", + "\n", + "## Assumptions made\n", + "\n", + "The assumption we have made here can be summarized as (and this is going to useful when we discuss the bias-variance trade off)\n", + "that there exists a function $f(\\boldsymbol{x})$ and a normal distributed error $\\boldsymbol{\\varepsilon}\\sim \\mathcal{N}(0, \\sigma^2)$\n", + "which describes our data" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "\\boldsymbol{y} = f(\\boldsymbol{x})+\\boldsymbol{\\varepsilon}\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "We approximate this function with our model from the solution of the linear regression equations, that is our\n", + "function $f$ is approximated by $\\boldsymbol{\\tilde{y}}$ where we want to minimize $(\\boldsymbol{y}-\\boldsymbol{\\tilde{y}})^2$, our MSE, with" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "\\boldsymbol{\\tilde{y}} = \\boldsymbol{X}\\boldsymbol{\\beta}.\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Expectation value and variance\n", + "\n", + "We can calculate the expectation value of $\\boldsymbol{y}$ for a given element $i$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "\\begin{align*} \n", + "\\mathbb{E}(y_i) & =\n", + "\\mathbb{E}(\\mathbf{X}_{i, \\ast} \\, \\boldsymbol{\\beta}) + \\mathbb{E}(\\varepsilon_i)\n", + "\\, \\, \\, = \\, \\, \\, \\mathbf{X}_{i, \\ast} \\, \\beta, \n", + "\\end{align*}\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "while\n", + "its variance is" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "\\begin{align*} \\mbox{Var}(y_i) & = \\mathbb{E} \\{ [y_i\n", + "- \\mathbb{E}(y_i)]^2 \\} \\, \\, \\, = \\, \\, \\, \\mathbb{E} ( y_i^2 ) -\n", + "[\\mathbb{E}(y_i)]^2 \\\\ & = \\mathbb{E} [ ( \\mathbf{X}_{i, \\ast} \\,\n", + "\\beta + \\varepsilon_i )^2] - ( \\mathbf{X}_{i, \\ast} \\, \\boldsymbol{\\beta})^2 \\\\ &\n", + "= \\mathbb{E} [ ( \\mathbf{X}_{i, \\ast} \\, \\boldsymbol{\\beta})^2 + 2 \\varepsilon_i\n", + "\\mathbf{X}_{i, \\ast} \\, \\boldsymbol{\\beta} + \\varepsilon_i^2 ] - ( \\mathbf{X}_{i,\n", + "\\ast} \\, \\beta)^2 \\\\ & = ( \\mathbf{X}_{i, \\ast} \\, \\boldsymbol{\\beta})^2 + 2\n", + "\\mathbb{E}(\\varepsilon_i) \\mathbf{X}_{i, \\ast} \\, \\boldsymbol{\\beta} +\n", + "\\mathbb{E}(\\varepsilon_i^2 ) - ( \\mathbf{X}_{i, \\ast} \\, \\boldsymbol{\\beta})^2 \n", + "\\\\ & = \\mathbb{E}(\\varepsilon_i^2 ) \\, \\, \\, = \\, \\, \\,\n", + "\\mbox{Var}(\\varepsilon_i) \\, \\, \\, = \\, \\, \\, \\sigma^2. \n", + "\\end{align*}\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Hence, $y_i \\sim \\mathcal{N}( \\mathbf{X}_{i, \\ast} \\, \\boldsymbol{\\beta}, \\sigma^2)$, that is $\\boldsymbol{y}$ follows a normal distribution with \n", + "mean value $\\boldsymbol{X}\\boldsymbol{\\beta}$ and variance $\\sigma^2$ (not be confused with the singular values of the SVD). \n", + "\n", + "## Expectation value and variance for $\\boldsymbol{\\beta}$\n", + "\n", + "With the OLS expressions for the parameters $\\boldsymbol{\\beta}$ we can evaluate the expectation value" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "\\mathbb{E}(\\boldsymbol{\\beta}) = \\mathbb{E}[ (\\mathbf{X}^{\\top} \\mathbf{X})^{-1}\\mathbf{X}^{T} \\mathbf{Y}]=(\\mathbf{X}^{T} \\mathbf{X})^{-1}\\mathbf{X}^{T} \\mathbb{E}[ \\mathbf{Y}]=(\\mathbf{X}^{T} \\mathbf{X})^{-1} \\mathbf{X}^{T}\\mathbf{X}\\boldsymbol{\\beta}=\\boldsymbol{\\beta}.\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "This means that the estimator of the regression parameters is unbiased.\n", + "\n", + "We can also calculate the variance\n", + "\n", + "The variance of $\\boldsymbol{\\beta}$ is" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "\\begin{eqnarray*}\n", + "\\mbox{Var}(\\boldsymbol{\\beta}) & = & \\mathbb{E} \\{ [\\boldsymbol{\\beta} - \\mathbb{E}(\\boldsymbol{\\beta})] [\\boldsymbol{\\beta} - \\mathbb{E}(\\boldsymbol{\\beta})]^{T} \\}\n", + "\\\\\n", + "& = & \\mathbb{E} \\{ [(\\mathbf{X}^{T} \\mathbf{X})^{-1} \\, \\mathbf{X}^{T} \\mathbf{Y} - \\boldsymbol{\\beta}] \\, [(\\mathbf{X}^{T} \\mathbf{X})^{-1} \\, \\mathbf{X}^{T} \\mathbf{Y} - \\boldsymbol{\\beta}]^{T} \\}\n", + "\\\\\n", + "% & = & \\mathbb{E} \\{ [(\\mathbf{X}^{T} \\mathbf{X})^{-1} \\, \\mathbf{X}^{T} \\mathbf{Y}] \\, [(\\mathbf{X}^{T} \\mathbf{X})^{-1} \\, \\mathbf{X}^{T} \\mathbf{Y}]^{T} \\} - \\boldsymbol{\\beta} \\, \\boldsymbol{\\beta}^{T}\n", + "% \\\\\n", + "% & = & \\mathbb{E} \\{ (\\mathbf{X}^{T} \\mathbf{X})^{-1} \\, \\mathbf{X}^{T} \\mathbf{Y} \\, \\mathbf{Y}^{T} \\, \\mathbf{X} \\, (\\mathbf{X}^{T} \\mathbf{X})^{-1} \\} - \\boldsymbol{\\beta} \\, \\boldsymbol{\\beta}^{T}\n", + "% \\\\\n", + "& = & (\\mathbf{X}^{T} \\mathbf{X})^{-1} \\, \\mathbf{X}^{T} \\, \\mathbb{E} \\{ \\mathbf{Y} \\, \\mathbf{Y}^{T} \\} \\, \\mathbf{X} \\, (\\mathbf{X}^{T} \\mathbf{X})^{-1} - \\boldsymbol{\\beta} \\, \\boldsymbol{\\beta}^{T}\n", + "\\\\\n", + "& = & (\\mathbf{X}^{T} \\mathbf{X})^{-1} \\, \\mathbf{X}^{T} \\, \\{ \\mathbf{X} \\, \\boldsymbol{\\beta} \\, \\boldsymbol{\\beta}^{T} \\, \\mathbf{X}^{T} + \\sigma^2 \\} \\, \\mathbf{X} \\, (\\mathbf{X}^{T} \\mathbf{X})^{-1} - \\boldsymbol{\\beta} \\, \\boldsymbol{\\beta}^{T}\n", + "% \\\\\n", + "% & = & (\\mathbf{X}^T \\mathbf{X})^{-1} \\, \\mathbf{X}^T \\, \\mathbf{X} \\, \\boldsymbol{\\beta} \\, \\boldsymbol{\\beta}^T \\, \\mathbf{X}^T \\, \\mathbf{X} \\, (\\mathbf{X}^T % \\mathbf{X})^{-1}\n", + "% \\\\\n", + "% & & + \\, \\, \\sigma^2 \\, (\\mathbf{X}^T \\mathbf{X})^{-1} \\, \\mathbf{X}^T \\, \\mathbf{X} \\, (\\mathbf{X}^T \\mathbf{X})^{-1} - \\boldsymbol{\\beta} \\boldsymbol{\\beta}^T\n", + "\\\\\n", + "& = & \\boldsymbol{\\beta} \\, \\boldsymbol{\\beta}^{T} + \\sigma^2 \\, (\\mathbf{X}^{T} \\mathbf{X})^{-1} - \\boldsymbol{\\beta} \\, \\boldsymbol{\\beta}^{T}\n", + "\\, \\, \\, = \\, \\, \\, \\sigma^2 \\, (\\mathbf{X}^{T} \\mathbf{X})^{-1},\n", + "\\end{eqnarray*}\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "where we have used that $\\mathbb{E} (\\mathbf{Y} \\mathbf{Y}^{T}) =\n", + "\\mathbf{X} \\, \\boldsymbol{\\beta} \\, \\boldsymbol{\\beta}^{T} \\, \\mathbf{X}^{T} +\n", + "\\sigma^2 \\, \\mathbf{I}_{nn}$. From $\\mbox{Var}(\\boldsymbol{\\beta}) = \\sigma^2\n", + "\\, (\\mathbf{X}^{T} \\mathbf{X})^{-1}$, one obtains an estimate of the\n", + "variance of the estimate of the $j$-th regression coefficient:\n", + "$\\hat{\\sigma}^2 (\\hat{\\beta}_j ) = \\hat{\\sigma}^2 \\sqrt{\n", + "[(\\mathbf{X}^{T} \\mathbf{X})^{-1}]_{jj} }$. This may be used to\n", + "construct a confidence interval for the estimates.\n", + "\n", + "\n", + "In a similar way, we cna obtain analytical expressions for say the\n", + "expectation values of the parameters $\\boldsymbol{\\beta}$ and their variance\n", + "when we employ Ridge regression, and thereby a confidence interval. \n", + "\n", + "It is rather straightforward to show that" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "\\mathbb{E} \\big[ \\boldsymbol{\\beta}^{\\mathrm{Ridge}} \\big]=(\\mathbf{X}^{T} \\mathbf{X} + \\lambda \\mathbf{I}_{pp})^{-1} (\\mathbf{X}^{\\top} \\mathbf{X})\\boldsymbol{\\beta}^{\\mathrm{OLS}}.\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "We see clearly that \n", + "$\\mathbb{E} \\big[ \\boldsymbol{\\beta}^{\\mathrm{Ridge}} \\big] \\not= \\boldsymbol{\\beta}^{\\mathrm{OLS}}$ for any $\\lambda > 0$. We say then that the ridge estimator is biased.\n", + "\n", + "We can also compute the variance as" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "\\mbox{Var}[\\boldsymbol{\\beta}^{\\mathrm{Ridge}}]=\\sigma^2[ \\mathbf{X}^{T} \\mathbf{X} + \\lambda \\mathbf{I} ]^{-1} \\mathbf{X}^{T} \\mathbf{X} \\{ [ \\mathbf{X}^{\\top} \\mathbf{X} + \\lambda \\mathbf{I} ]^{-1}\\}^{T},\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "and it is easy to see that if the parameter $\\lambda$ goes to infinity then the variance of Ridge parameters $\\boldsymbol{\\beta}$ goes to zero. \n", + "\n", + "With this, we can compute the difference" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "\\mbox{Var}[\\boldsymbol{\\beta}^{\\mathrm{OLS}}]-\\mbox{Var}(\\boldsymbol{\\beta}^{\\mathrm{Ridge}})=\\sigma^2 [ \\mathbf{X}^{T} \\mathbf{X} + \\lambda \\mathbf{I} ]^{-1}[ 2\\lambda\\mathbf{I} + \\lambda^2 (\\mathbf{X}^{T} \\mathbf{X})^{-1} ] \\{ [ \\mathbf{X}^{T} \\mathbf{X} + \\lambda \\mathbf{I} ]^{-1}\\}^{T}.\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "The difference is non-negative definite since each component of the\n", + "matrix product is non-negative definite. \n", + "This means the variance we obtain with the standard OLS will always for $\\lambda > 0$ be larger than the variance of $\\boldsymbol{\\beta}$ obtained with the Ridge estimator. This has interesting consequences when we discuss the so-called bias-variance trade-off below. \n", + "\n", + "\n", + "## Cross-validation\n", + "\n", + "Instead of choosing the penalty parameter to balance model fit with\n", + "model complexity, cross-validation requires it (i.e. the penalty\n", + "parameter) to yield a model with good prediction\n", + "performance. Commonly, this performance is evaluated on novel\n", + "data. Novel data need not be easy to come by and one has to make do\n", + "with the data at hand.\n", + "\n", + "The setting of **original** and novel data is\n", + "then mimicked by sample splitting: the data set is divided into two\n", + "(groups of samples). One of these two data sets, called the \n", + "*training set*, plays the role of **original** data on which the model is\n", + "built. The second of these data sets, called the *test set*, plays the\n", + "role of the **novel** data and is used to evaluate the prediction\n", + "performance (often operationalized as the log-likelihood or the\n", + "prediction error or its square or the R2 score) of the model built on the training data set. This\n", + "procedure (model building and prediction evaluation on training and\n", + "test set, respectively) is done for a collection of possible penalty\n", + "parameter choices. The penalty parameter that yields the model with\n", + "the best prediction performance is to be preferred. The thus obtained\n", + "performance evaluation depends on the actual split of the data set. To\n", + "remove this dependence the data set is split many times into a\n", + "training and test set. For each split the model parameters are\n", + "estimated for all choices of $\\lambda$ using the training data and\n", + "estimated parameters are evaluated on the corresponding test set. The\n", + "penalty parameter that on average over the test sets performs best (in\n", + "some sense) is then selected.\n", + "\n", + "\n", + "## Computationally expensive\n", + "\n", + "The validation set approach is conceptually simple and is easy to implement. But it has two potential drawbacks:\n", + "\n", + "* The validation estimate of the test error rate can be highly variable, depending on precisely which observations are included in the training set and which observations are included in the validation set.\n", + "\n", + "* In the validation approach, only a subset of the observations, those that are included in the training set rather than in the validation set are used to fit the model. Since statistical methods tend to perform worse when trained on fewer observations, this suggests that the validation set error rate may tend to overestimate the test error rate for the model fit on the entire data set.\n", + "\n", + "\n", + "## Various steps in cross-validation\n", + "\n", + "When the repetitive splitting of the data set is done randomly,\n", + "samples may accidently end up in a fast majority of the splits in\n", + "either training or test set. Such samples may have an unbalanced\n", + "influence on either model building or prediction evaluation. To avoid\n", + "this $k$-fold cross-validation structures the data splitting. The\n", + "samples are divided into $k$ more or less equally sized exhaustive and\n", + "mutually exclusive subsets. In turn (at each split) one of these\n", + "subsets plays the role of the test set while the union of the\n", + "remaining subsets constitutes the training set. Such a splitting\n", + "warrants a balanced representation of each sample in both training and\n", + "test set over the splits. Still the division into the $k$ subsets\n", + "involves a degree of randomness. This may be fully excluded when\n", + "choosing $k=n$. This particular case is referred to as leave-one-out\n", + "cross-validation (LOOCV). \n", + "\n", + "\n", + "## How to set up the cross-validation for Ridge and/or Lasso\n", + "\n", + "* Define a range of interest for the penalty parameter.\n", + "\n", + "* Divide the data set into training and test set comprising samples $\\{1, \\ldots, n\\} \\setminus i$ and $\\{ i \\}$, respectively.\n", + "\n", + "* Fit the linear regression model by means of ridge estimation for each $\\lambda$ in the grid using the training set, and the corresponding estimate of the error variance $\\boldsymbol{\\sigma}_{-i}^2(\\lambda)$, as" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "\\begin{align*}\n", + "\\boldsymbol{\\beta}_{-i}(\\lambda) & = ( \\boldsymbol{X}_{-i, \\ast}^{T}\n", + "\\boldsymbol{X}_{-i, \\ast} + \\lambda \\boldsymbol{I}_{pp})^{-1}\n", + "\\boldsymbol{X}_{-i, \\ast}^{T} \\boldsymbol{y}_{-i}\n", + "\\end{align*}\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "* Evaluate the prediction performance of these models on the test set by $\\log\\{L[y_i, \\boldsymbol{X}_{i, \\ast}; \\boldsymbol{\\beta}_{-i}(\\lambda), \\boldsymbol{\\sigma}_{-i}^2(\\lambda)]\\}$. Or, by the prediction error $|y_i - \\boldsymbol{X}_{i, \\ast} \\boldsymbol{\\beta}_{-i}(\\lambda)|$, the relative error, the error squared or the R2 score function.\n", + "\n", + "* Repeat the first three steps such that each sample plays the role of the test set once.\n", + "\n", + "* Average the prediction performances of the test sets at each grid point of the penalty bias/parameter by computing the *cross-validated log-likelihood*. It is an estimate of the prediction performance of the model corresponding to this value of the penalty parameter on novel data. It is defined as" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "\\begin{align*}\n", + "\\frac{1}{n} \\sum_{i = 1}^n \\log\\{L[y_i, \\mathbf{X}_{i, \\ast}; \\boldsymbol{\\beta}_{-i}(\\lambda), \\boldsymbol{\\sigma}_{-i}^2(\\lambda)]\\}.\n", + "\\end{align*}\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "* The value of the penalty parameter that maximizes the cross-validated log-likelihood is the value of choice. Or we can use the MSE or the R2 score functions.\n", + "\n", + "## Resampling methods: Jackknife and Bootstrap\n", + "\n", + "Two famous\n", + "resampling methods are the **independent bootstrap** and **the jackknife**. \n", + "\n", + "The jackknife is a special case of the independent bootstrap. Still, the jackknife was made\n", + "popular prior to the independent bootstrap. And as the popularity of\n", + "the independent bootstrap soared, new variants, such as **the dependent bootstrap**.\n", + "\n", + "The Jackknife and independent bootstrap work for\n", + "independent, identically distributed random variables.\n", + "If these conditions are not\n", + "satisfied, the methods will fail. Yet, it should be said that if the data are\n", + "independent, identically distributed, and we only want to estimate the\n", + "variance of $\\overline{X}$ (which often is the case), then there is no\n", + "need for bootstrapping. \n", + "\n", + "## Resampling methods: Jackknife\n", + "\n", + "The Jackknife works by making many replicas of the estimator $\\widehat{\\theta}$. \n", + "The jackknife is a resampling method where we systematically leave out one observation from the vector of observed values $\\boldsymbol{x} = (x_1,x_2,\\cdots,X_n)$. \n", + "Let $\\boldsymbol{x}_i$ denote the vector" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "\\boldsymbol{x}_i = (x_1,x_2,\\cdots,x_{i-1},x_{i+1},\\cdots,x_n),\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "which equals the vector $\\boldsymbol{x}$ with the exception that observation\n", + "number $i$ is left out. Using this notation, define\n", + "$\\widehat{\\theta}_i$ to be the estimator\n", + "$\\widehat{\\theta}$ computed using $\\vec{X}_i$. \n", + "\n", + "\n", + "## Jackknife code example" + ] + }, + { + "cell_type": "code", + "execution_count": 19, + "metadata": { + "collapsed": false + }, + "outputs": [], + "source": [ + "from numpy import *\n", + "from numpy.random import randint, randn\n", + "from time import time\n", + "\n", + "def jackknife(data, stat):\n", + " n = len(data);t = zeros(n); inds = arange(n); t0 = time()\n", + " ## 'jackknifing' by leaving out an observation for each i \n", + " for i in range(n):\n", + " t[i] = stat(delete(data,i) )\n", + "\n", + " # analysis \n", + " print(\"Runtime: %g sec\" % (time()-t0)); print(\"Jackknife Statistics :\")\n", + " print(\"original bias std. error\")\n", + " print(\"%8g %14g %15g\" % (stat(data),(n-1)*mean(t)/n, (n*var(t))**.5))\n", + "\n", + " return t\n", + "\n", + "\n", + "# Returns mean of data samples \n", + "def stat(data):\n", + " return mean(data)\n", + "\n", + "\n", + "mu, sigma = 100, 15\n", + "datapoints = 10000\n", + "x = mu + sigma*random.randn(datapoints)\n", + "# jackknife returns the data sample \n", + "t = jackknife(x, stat)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Resampling methods: Bootstrap\n", + "Bootstrapping is a nonparametric approach to statistical inference\n", + "that substitutes computation for more traditional distributional\n", + "assumptions and asymptotic results. Bootstrapping offers a number of\n", + "advantages: \n", + "1. The bootstrap is quite general, although there are some cases in which it fails. \n", + "\n", + "2. Because it does not require distributional assumptions (such as normally distributed errors), the bootstrap can provide more accurate inferences when the data are not well behaved or when the sample size is small. \n", + "\n", + "3. It is possible to apply the bootstrap to statistics with sampling distributions that are difficult to derive, even asymptotically. \n", + "\n", + "4. It is relatively simple to apply the bootstrap to complex data-collection plans (such as stratified and clustered samples).\n", + "\n", + "\n", + "\n", + "\n", + "## Resampling methods: Bootstrap background\n", + "\n", + "Since $\\widehat{\\theta} = \\widehat{\\theta}(\\boldsymbol{X})$ is a function of random variables,\n", + "$\\widehat{\\theta}$ itself must be a random variable. Thus it has\n", + "a pdf, call this function $p(\\boldsymbol{t})$. The aim of the bootstrap is to\n", + "estimate $p(\\boldsymbol{t})$ by the relative frequency of\n", + "$\\widehat{\\theta}$. You can think of this as using a histogram\n", + "in the place of $p(\\boldsymbol{t})$. If the relative frequency closely\n", + "resembles $p(\\vec{t})$, then using numerics, it is straight forward to\n", + "estimate all the interesting parameters of $p(\\boldsymbol{t})$ using point\n", + "estimators. \n", + "\n", + "\n", + "## Resampling methods: More Bootstrap background\n", + "\n", + "In the case that $\\widehat{\\theta}$ has\n", + "more than one component, and the components are independent, we use the\n", + "same estimator on each component separately. If the probability\n", + "density function of $X_i$, $p(x)$, had been known, then it would have\n", + "been straight forward to do this by: \n", + "1. Drawing lots of numbers from $p(x)$, suppose we call one such set of numbers $(X_1^*, X_2^*, \\cdots, X_n^*)$. \n", + "\n", + "2. Then using these numbers, we could compute a replica of $\\widehat{\\theta}$ called $\\widehat{\\theta}^*$. \n", + "\n", + "By repeated use of (1) and (2), many\n", + "estimates of $\\widehat{\\theta}$ could have been obtained. The\n", + "idea is to use the relative frequency of $\\widehat{\\theta}^*$\n", + "(think of a histogram) as an estimate of $p(\\boldsymbol{t})$.\n", + "\n", + "## Resampling methods: Bootstrap approach\n", + "\n", + "But\n", + "unless there is enough information available about the process that\n", + "generated $X_1,X_2,\\cdots,X_n$, $p(x)$ is in general\n", + "unknown. Therefore, [Efron in 1979](https://projecteuclid.org/euclid.aos/1176344552) asked the\n", + "question: What if we replace $p(x)$ by the relative frequency\n", + "of the observation $X_i$; if we draw observations in accordance with\n", + "the relative frequency of the observations, will we obtain the same\n", + "result in some asymptotic sense? The answer is yes.\n", + "\n", + "\n", + "Instead of generating the histogram for the relative\n", + "frequency of the observation $X_i$, just draw the values\n", + "$(X_1^*,X_2^*,\\cdots,X_n^*)$ with replacement from the vector\n", + "$\\boldsymbol{X}$. \n", + "\n", + "## Resampling methods: Bootstrap steps\n", + "\n", + "The independent bootstrap works like this: \n", + "\n", + "1. Draw with replacement $n$ numbers for the observed variables $\\boldsymbol{x} = (x_1,x_2,\\cdots,x_n)$. \n", + "\n", + "2. Define a vector $\\boldsymbol{x}^*$ containing the values which were drawn from $\\boldsymbol{x}$. \n", + "\n", + "3. Using the vector $\\boldsymbol{x}^*$ compute $\\widehat{\\theta}^*$ by evaluating $\\widehat \\theta$ under the observations $\\boldsymbol{x}^*$. \n", + "\n", + "4. Repeat this process $k$ times. \n", + "\n", + "When you are done, you can draw a histogram of the relative frequency\n", + "of $\\widehat \\theta^*$. This is your estimate of the probability\n", + "distribution $p(t)$. Using this probability distribution you can\n", + "estimate any statistics thereof. In principle you never draw the\n", + "histogram of the relative frequency of $\\widehat{\\theta}^*$. Instead\n", + "you use the estimators corresponding to the statistic of interest. For\n", + "example, if you are interested in estimating the variance of $\\widehat\n", + "\\theta$, apply the etsimator $\\widehat \\sigma^2$ to the values\n", + "$\\widehat \\theta ^*$.\n", + "\n", + "\n", + "## Code example for the Bootstrap method\n", + "\n", + "The following code starts with a Gaussian distribution with mean value\n", + "$\\mu =100$ and variance $\\sigma=15$. We use this to generate the data\n", + "used in the bootstrap analysis. The bootstrap analysis returns a data\n", + "set after a given number of bootstrap operations (as many as we have\n", + "data points). This data set consists of estimated mean values for each\n", + "bootstrap operation. The histogram generated by the bootstrap method\n", + "shows that the distribution for these mean values is also a Gaussian,\n", + "centered around the mean value $\\mu=100$ but with standard deviation\n", + "$\\sigma/\\sqrt{n}$, where $n$ is the number of bootstrap samples (in\n", + "this case the same as the number of original data points). The value\n", + "of the standard deviation is what we expect from the central limit\n", + "theorem." + ] + }, + { + "cell_type": "code", + "execution_count": 20, + "metadata": { + "collapsed": false + }, + "outputs": [], + "source": [ + "from numpy import *\n", + "from numpy.random import randint, randn\n", + "from time import time\n", + "import matplotlib.mlab as mlab\n", + "import matplotlib.pyplot as plt\n", + "\n", + "# Returns mean of bootstrap samples \n", + "def stat(data):\n", + " return mean(data)\n", + "\n", + "# Bootstrap algorithm\n", + "def bootstrap(data, statistic, R):\n", + " t = zeros(R); n = len(data); inds = arange(n); t0 = time()\n", + " # non-parametric bootstrap \n", + " for i in range(R):\n", + " t[i] = statistic(data[randint(0,n,n)])\n", + "\n", + " # analysis \n", + " print(\"Runtime: %g sec\" % (time()-t0)); print(\"Bootstrap Statistics :\")\n", + " print(\"original bias std. error\")\n", + " print(\"%8g %8g %14g %15g\" % (statistic(data), std(data),mean(t),std(t)))\n", + " return t\n", + "\n", + "\n", + "mu, sigma = 100, 15\n", + "datapoints = 10000\n", + "x = mu + sigma*random.randn(datapoints)\n", + "# bootstrap returns the data sample \n", + "t = bootstrap(x, stat, datapoints)\n", + "# the histogram of the bootstrapped data \n", + "n, binsboot, patches = plt.hist(t, 50, normed=1, facecolor='red', alpha=0.75)\n", + "\n", + "# add a 'best fit' line \n", + "y = mlab.normpdf( binsboot, mean(t), std(t))\n", + "lt = plt.plot(binsboot, y, 'r--', linewidth=1)\n", + "plt.xlabel('Smarts')\n", + "plt.ylabel('Probability')\n", + "plt.axis([99.5, 100.6, 0, 3.0])\n", + "plt.grid(True)\n", + "\n", + "plt.show()" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Code Example for Cross-validation and $k$-fold Cross-validation\n", + "\n", + "The code here uses Ridge regression with cross-validation (CV) resampling and $k$-fold CV in order to fit a specific polynomial." + ] + }, + { + "cell_type": "code", + "execution_count": 21, + "metadata": { + "collapsed": false + }, + "outputs": [], + "source": [ + "import numpy as np\n", + "import matplotlib.pyplot as plt\n", + "from sklearn.model_selection import KFold\n", + "from sklearn.linear_model import Ridge\n", + "from sklearn.model_selection import cross_val_score\n", + "from sklearn.preprocessing import PolynomialFeatures\n", + "\n", + "# A seed just to ensure that the random numbers are the same for every run.\n", + "# Useful for eventual debugging.\n", + "np.random.seed(3155)\n", + "\n", + "# Generate the data.\n", + "nsamples = 100\n", + "x = np.random.randn(nsamples)\n", + "y = 3*x**2 + np.random.randn(nsamples)\n", + "\n", + "## Cross-validation on Ridge regression using KFold only\n", + "\n", + "# Decide degree on polynomial to fit\n", + "poly = PolynomialFeatures(degree = 6)\n", + "\n", + "# Decide which values of lambda to use\n", + "nlambdas = 500\n", + "lambdas = np.logspace(-3, 5, nlambdas)\n", + "\n", + "# Initialize a KFold instance\n", + "k = 5\n", + "kfold = KFold(n_splits = k)\n", + "\n", + "# Perform the cross-validation to estimate MSE\n", + "scores_KFold = np.zeros((nlambdas, k))\n", + "\n", + "i = 0\n", + "for lmb in lambdas:\n", + " ridge = Ridge(alpha = lmb)\n", + " j = 0\n", + " for train_inds, test_inds in kfold.split(x):\n", + " xtrain = x[train_inds]\n", + " ytrain = y[train_inds]\n", + "\n", + " xtest = x[test_inds]\n", + " ytest = y[test_inds]\n", + "\n", + " Xtrain = poly.fit_transform(xtrain[:, np.newaxis])\n", + " ridge.fit(Xtrain, ytrain[:, np.newaxis])\n", + "\n", + " Xtest = poly.fit_transform(xtest[:, np.newaxis])\n", + " ypred = ridge.predict(Xtest)\n", + "\n", + " scores_KFold[i,j] = np.sum((ypred - ytest[:, np.newaxis])**2)/np.size(ypred)\n", + "\n", + " j += 1\n", + " i += 1\n", + "\n", + "\n", + "estimated_mse_KFold = np.mean(scores_KFold, axis = 1)\n", + "\n", + "## Cross-validation using cross_val_score from sklearn along with KFold\n", + "\n", + "# kfold is an instance initialized above as:\n", + "# kfold = KFold(n_splits = k)\n", + "\n", + "estimated_mse_sklearn = np.zeros(nlambdas)\n", + "i = 0\n", + "for lmb in lambdas:\n", + " ridge = Ridge(alpha = lmb)\n", + "\n", + " X = poly.fit_transform(x[:, np.newaxis])\n", + " estimated_mse_folds = cross_val_score(ridge, X, y[:, np.newaxis], scoring='neg_mean_squared_error', cv=kfold)\n", + "\n", + " # cross_val_score return an array containing the estimated negative mse for every fold.\n", + " # we have to the the mean of every array in order to get an estimate of the mse of the model\n", + " estimated_mse_sklearn[i] = np.mean(-estimated_mse_folds)\n", + "\n", + " i += 1\n", + "\n", + "## Plot and compare the slightly different ways to perform cross-validation\n", + "\n", + "plt.figure()\n", + "\n", + "plt.plot(np.log10(lambdas), estimated_mse_sklearn, label = 'cross_val_score')\n", + "plt.plot(np.log10(lambdas), estimated_mse_KFold, 'r--', label = 'KFold')\n", + "\n", + "plt.xlabel('log10(lambda)')\n", + "plt.ylabel('mse')\n", + "\n", + "plt.legend()\n", + "\n", + "plt.show()" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## The bias-variance tradeoff\n", + "\n", + "\n", + "We will discuss the bias-variance tradeoff in the context of\n", + "continuous predictions such as regression. However, many of the\n", + "intuitions and ideas discussed here also carry over to classification\n", + "tasks. Consider a dataset $\\mathcal{L}$ consisting of the data\n", + "$\\mathbf{X}_\\mathcal{L}=\\{(y_j, \\boldsymbol{x}_j), j=0\\ldots n-1\\}$. \n", + "\n", + "Let us assume that the true data is generated from a noisy model" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "\\boldsymbol{y}=f(\\boldsymbol{x}) + \\boldsymbol{\\epsilon}\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "where $\\epsilon$ is normally distributed with mean zero and standard deviation $\\sigma^2$.\n", + "\n", + "In our derivation of the ordinary least squares method we defined then\n", + "an approximation to the function $f$ in terms of the parameters\n", + "$\\boldsymbol{\\beta}$ and the design matrix $\\boldsymbol{X}$ which embody our model,\n", + "that is $\\boldsymbol{\\tilde{y}}=\\boldsymbol{X}\\boldsymbol{\\beta}$. \n", + "\n", + "Thereafter we found the parameters $\\boldsymbol{\\beta}$ by optimizing the means squared error via the so-called cost function" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "C(\\boldsymbol{X},\\boldsymbol{\\beta}) =\\frac{1}{n}\\sum_{i=0}^{n-1}(y_i-\\tilde{y}_i)^2=\\mathbb{E}\\left[(\\boldsymbol{y}-\\boldsymbol{\\tilde{y}})^2\\right].\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "We can rewrite this as" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "\\mathbb{E}\\left[(\\boldsymbol{y}-\\boldsymbol{\\tilde{y}})^2\\right]=\\frac{1}{n}\\sum_i(f_i-\\mathbb{E}\\left[\\boldsymbol{\\tilde{y}}\\right])^2+\\frac{1}{n}\\sum_i(\\tilde{y}_i-\\mathbb{E}\\left[\\boldsymbol{\\tilde{y}}\\right])^2+\\sigma^2.\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "The three terms represent the square of the bias of the learning\n", + "method, which can be thought of as the error caused by the simplifying\n", + "assumptions built into the method. The second term represents the\n", + "variance of the chosen model and finally the last terms is variance of\n", + "the error $\\boldsymbol{\\epsilon}$.\n", + "\n", + "To derive this equation, we need to recall that the variance of $\\boldsymbol{y}$ and $\\boldsymbol{\\epsilon}$ are both equal to $\\sigma^2$. The mean value of $\\boldsymbol{\\epsilon}$ is by definition equal to zero. Furthermore, the function $f$ is not a stochastics variable, idem for $\\boldsymbol{\\tilde{y}}$.\n", + "We use a more compact notation in terms of the expectation value" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "\\mathbb{E}\\left[(\\boldsymbol{y}-\\boldsymbol{\\tilde{y}})^2\\right]=\\mathbb{E}\\left[(\\boldsymbol{f}+\\boldsymbol{\\epsilon}-\\boldsymbol{\\tilde{y}})^2\\right],\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "and adding and subtracting $\\mathbb{E}\\left[\\boldsymbol{\\tilde{y}}\\right]$ we get" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "\\mathbb{E}\\left[(\\boldsymbol{y}-\\boldsymbol{\\tilde{y}})^2\\right]=\\mathbb{E}\\left[(\\boldsymbol{f}+\\boldsymbol{\\epsilon}-\\boldsymbol{\\tilde{y}}+\\mathbb{E}\\left[\\boldsymbol{\\tilde{y}}\\right]-\\mathbb{E}\\left[\\boldsymbol{\\tilde{y}}\\right])^2\\right],\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "which, using the abovementioned expectation values can be rewritten as" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "\\mathbb{E}\\left[(\\boldsymbol{y}-\\boldsymbol{\\tilde{y}})^2\\right]=\\mathbb{E}\\left[(\\boldsymbol{y}-\\mathbb{E}\\left[\\boldsymbol{\\tilde{y}}\\right])^2\\right]+\\mathrm{Var}\\left[\\boldsymbol{\\tilde{y}}\\right]+\\sigma^2,\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "that is the rewriting in terms of the so-called bias, the variance of the model $\\boldsymbol{\\tilde{y}}$ and the variance of $\\boldsymbol{\\epsilon}$.\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "## Example code for Bias-Variance tradeoff" + ] + }, + { + "cell_type": "code", + "execution_count": 22, + "metadata": { + "collapsed": false + }, + "outputs": [], + "source": [ + "import matplotlib.pyplot as plt\n", + "import numpy as np\n", + "from sklearn.linear_model import LinearRegression, Ridge, Lasso\n", + "from sklearn.preprocessing import PolynomialFeatures\n", + "from sklearn.model_selection import train_test_split\n", + "from sklearn.pipeline import make_pipeline\n", + "from sklearn.utils import resample\n", + "\n", + "np.random.seed(2018)\n", + "\n", + "n = 500\n", + "n_boostraps = 100\n", + "degree = 18 # A quite high value, just to show.\n", + "noise = 0.1\n", + "\n", + "# Make data set.\n", + "x = np.linspace(-1, 3, n).reshape(-1, 1)\n", + "y = np.exp(-x**2) + 1.5 * np.exp(-(x-2)**2) + np.random.normal(0, 0.1, x.shape)\n", + "\n", + "# Hold out some test data that is never used in training.\n", + "x_train, x_test, y_train, y_test = train_test_split(x, y, test_size=0.2)\n", + "\n", + "# Combine x transformation and model into one operation.\n", + "# Not neccesary, but convenient.\n", + "model = make_pipeline(PolynomialFeatures(degree=degree), LinearRegression(fit_intercept=False))\n", + "\n", + "# The following (m x n_bootstraps) matrix holds the column vectors y_pred\n", + "# for each bootstrap iteration.\n", + "y_pred = np.empty((y_test.shape[0], n_boostraps))\n", + "for i in range(n_boostraps):\n", + " x_, y_ = resample(x_train, y_train)\n", + "\n", + " # Evaluate the new model on the same test data each time.\n", + " y_pred[:, i] = model.fit(x_, y_).predict(x_test).ravel()\n", + "\n", + "# Note: Expectations and variances taken w.r.t. different training\n", + "# data sets, hence the axis=1. Subsequent means are taken across the test data\n", + "# set in order to obtain a total value, but before this we have error/bias/variance\n", + "# calculated per data point in the test set.\n", + "# Note 2: The use of keepdims=True is important in the calculation of bias as this \n", + "# maintains the column vector form. Dropping this yields very unexpected results.\n", + "error = np.mean( np.mean((y_test - y_pred)**2, axis=1, keepdims=True) )\n", + "bias = np.mean( (y_test - np.mean(y_pred, axis=1, keepdims=True))**2 )\n", + "variance = np.mean( np.var(y_pred, axis=1, keepdims=True) )\n", + "print('Error:', error)\n", + "print('Bias^2:', bias)\n", + "print('Var:', variance)\n", + "print('{} >= {} + {} = {}'.format(error, bias, variance, bias+variance))\n", + "\n", + "plt.plot(x[::5, :], y[::5, :], label='f(x)')\n", + "plt.scatter(x_test, y_test, label='Data points')\n", + "plt.scatter(x_test, np.mean(y_pred, axis=1), label='Pred')\n", + "plt.legend()\n", + "plt.show()" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Understanding what happens" + ] + }, + { + "cell_type": "code", + "execution_count": 23, + "metadata": { + "collapsed": false + }, + "outputs": [], + "source": [ + "import matplotlib.pyplot as plt\n", + "import numpy as np\n", + "from sklearn.linear_model import LinearRegression, Ridge, Lasso\n", + "from sklearn.preprocessing import PolynomialFeatures\n", + "from sklearn.model_selection import train_test_split\n", + "from sklearn.pipeline import make_pipeline\n", + "from sklearn.utils import resample\n", + "\n", + "np.random.seed(2018)\n", + "\n", + "n = 40\n", + "n_boostraps = 100\n", + "maxdegree = 14\n", + "\n", + "\n", + "# Make data set.\n", + "x = np.linspace(-3, 3, n).reshape(-1, 1)\n", + "y = np.exp(-x**2) + 1.5 * np.exp(-(x-2)**2)+ np.random.normal(0, 0.1, x.shape)\n", + "error = np.zeros(maxdegree)\n", + "bias = np.zeros(maxdegree)\n", + "variance = np.zeros(maxdegree)\n", + "polydegree = np.zeros(maxdegree)\n", + "x_train, x_test, y_train, y_test = train_test_split(x, y, test_size=0.2)\n", + "\n", + "for degree in range(maxdegree):\n", + " model = make_pipeline(PolynomialFeatures(degree=degree), LinearRegression(fit_intercept=False))\n", + " y_pred = np.empty((y_test.shape[0], n_boostraps))\n", + " for i in range(n_boostraps):\n", + " x_, y_ = resample(x_train, y_train)\n", + " y_pred[:, i] = model.fit(x_, y_).predict(x_test).ravel()\n", + "\n", + " polydegree[degree] = degree\n", + " error[degree] = np.mean( np.mean((y_test - y_pred)**2, axis=1, keepdims=True) )\n", + " bias[degree] = np.mean( (y_test - np.mean(y_pred, axis=1, keepdims=True))**2 )\n", + " variance[degree] = np.mean( np.var(y_pred, axis=1, keepdims=True) )\n", + " print('Polynomial degree:', degree)\n", + " print('Error:', error[degree])\n", + " print('Bias^2:', bias[degree])\n", + " print('Var:', variance[degree])\n", + " print('{} >= {} + {} = {}'.format(error[degree], bias[degree], variance[degree], bias[degree]+variance[degree]))\n", + "\n", + "plt.plot(polydegree, np.log10(error), label='Error')\n", + "plt.plot(polydegree, bias, label='bias')\n", + "plt.plot(polydegree, variance, label='Variance')\n", + "plt.legend()\n", + "plt.show()" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\n", + "## Summing up\n", + "\n", + "\n", + "\n", + "\n", + "The bias-variance tradeoff summarizes the fundamental tension in\n", + "machine learning, particularly supervised learning, between the\n", + "complexity of a model and the amount of training data needed to train\n", + "it. Since data is often limited, in practice it is often useful to\n", + "use a less-complex model with higher bias, that is a model whose asymptotic\n", + "performance is worse than another model because it is easier to\n", + "train and less sensitive to sampling noise arising from having a\n", + "finite-sized training dataset (smaller variance). \n", + "\n", + "\n", + "\n", + "The above equations tell us that in\n", + "order to minimize the expected test error, we need to select a\n", + "statistical learning method that simultaneously achieves low variance\n", + "and low bias. Note that variance is inherently a nonnegative quantity,\n", + "and squared bias is also nonnegative. Hence, we see that the expected\n", + "test MSE can never lie below $Var(\\epsilon)$, the irreducible error.\n", + "\n", + "\n", + "What do we mean by the variance and bias of a statistical learning\n", + "method? The variance refers to the amount by which our model would change if we\n", + "estimated it using a different training data set. Since the training\n", + "data are used to fit the statistical learning method, different\n", + "training data sets will result in a different estimate. But ideally the\n", + "estimate for our model should not vary too much between training\n", + "sets. However, if a method has high variance then small changes in\n", + "the training data can result in large changes in the model. In general, more\n", + "flexible statistical methods have higher variance.\n", + "\n", + "\n", + "## Another Example rom Scikit-Learn's Repository" + ] + }, + { + "cell_type": "code", + "execution_count": 24, + "metadata": { + "collapsed": false + }, + "outputs": [], + "source": [ + "\"\"\"\n", + "============================\n", + "Underfitting vs. Overfitting\n", + "============================\n", + "\n", + "This example demonstrates the problems of underfitting and overfitting and\n", + "how we can use linear regression with polynomial features to approximate\n", + "nonlinear functions. The plot shows the function that we want to approximate,\n", + "which is a part of the cosine function. In addition, the samples from the\n", + "real function and the approximations of different models are displayed. The\n", + "models have polynomial features of different degrees. We can see that a\n", + "linear function (polynomial with degree 1) is not sufficient to fit the\n", + "training samples. This is called **underfitting**. A polynomial of degree 4\n", + "approximates the true function almost perfectly. However, for higher degrees\n", + "the model will **overfit** the training data, i.e. it learns the noise of the\n", + "training data.\n", + "We evaluate quantitatively **overfitting** / **underfitting** by using\n", + "cross-validation. We calculate the mean squared error (MSE) on the validation\n", + "set, the higher, the less likely the model generalizes correctly from the\n", + "training data.\n", + "\"\"\"\n", + "\n", + "print(__doc__)\n", + "\n", + "import numpy as np\n", + "import matplotlib.pyplot as plt\n", + "from sklearn.pipeline import Pipeline\n", + "from sklearn.preprocessing import PolynomialFeatures\n", + "from sklearn.linear_model import LinearRegression\n", + "from sklearn.model_selection import cross_val_score\n", + "\n", + "\n", + "def true_fun(X):\n", + " return np.cos(1.5 * np.pi * X)\n", + "\n", + "np.random.seed(0)\n", + "\n", + "n_samples = 30\n", + "degrees = [1, 4, 15]\n", + "\n", + "X = np.sort(np.random.rand(n_samples))\n", + "y = true_fun(X) + np.random.randn(n_samples) * 0.1\n", + "\n", + "plt.figure(figsize=(14, 5))\n", + "for i in range(len(degrees)):\n", + " ax = plt.subplot(1, len(degrees), i + 1)\n", + " plt.setp(ax, xticks=(), yticks=())\n", + "\n", + " polynomial_features = PolynomialFeatures(degree=degrees[i],\n", + " include_bias=False)\n", + " linear_regression = LinearRegression()\n", + " pipeline = Pipeline([(\"polynomial_features\", polynomial_features),\n", + " (\"linear_regression\", linear_regression)])\n", + " pipeline.fit(X[:, np.newaxis], y)\n", + "\n", + " # Evaluate the models using crossvalidation\n", + " scores = cross_val_score(pipeline, X[:, np.newaxis], y,\n", + " scoring=\"neg_mean_squared_error\", cv=10)\n", + "\n", + " X_test = np.linspace(0, 1, 100)\n", + " plt.plot(X_test, pipeline.predict(X_test[:, np.newaxis]), label=\"Model\")\n", + " plt.plot(X_test, true_fun(X_test), label=\"True function\")\n", + " plt.scatter(X, y, edgecolor='b', s=20, label=\"Samples\")\n", + " plt.xlabel(\"x\")\n", + " plt.ylabel(\"y\")\n", + " plt.xlim((0, 1))\n", + " plt.ylim((-2, 2))\n", + " plt.legend(loc=\"best\")\n", + " plt.title(\"Degree {}\\nMSE = {:.2e}(+/- {:.2e})\".format(\n", + " degrees[i], -scores.mean(), scores.std()))\n", + "plt.show()" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## The one-dimensional Ising model\n", + "\n", + "Let us bring back the Ising model again, but now with an additional\n", + "focus on Ridge and Lasso regression as well. We repeat some of the\n", + "basic parts of the Ising model and the setup of the training and test\n", + "data. The one-dimensional Ising model with nearest neighbor\n", + "interaction, no external field and a constant coupling constant $J$ is\n", + "given by" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\n", + "
\n", + "\n", + "$$\n", + "\\begin{equation}\n", + " H = -J \\sum_{k}^L s_k s_{k + 1},\n", + "\\label{_auto17} \\tag{27}\n", + "\\end{equation}\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "where $s_i \\in \\{-1, 1\\}$ and $s_{N + 1} = s_1$. The number of spins in the system is determined by $L$. For the one-dimensional system there is no phase transition.\n", + "\n", + "We will look at a system of $L = 40$ spins with a coupling constant of $J = 1$. To get enough training data we will generate 10000 states with their respective energies." + ] + }, + { + "cell_type": "code", + "execution_count": 25, + "metadata": { + "collapsed": false + }, + "outputs": [], + "source": [ + "import numpy as np\n", + "import matplotlib.pyplot as plt\n", + "from mpl_toolkits.axes_grid1 import make_axes_locatable\n", + "import seaborn as sns\n", + "import scipy.linalg as scl\n", + "from sklearn.model_selection import train_test_split\n", + "import sklearn.linear_model as skl\n", + "import tqdm\n", + "sns.set(color_codes=True)\n", + "cmap_args=dict(vmin=-1., vmax=1., cmap='seismic')\n", + "\n", + "L = 40\n", + "n = int(1e4)\n", + "\n", + "spins = np.random.choice([-1, 1], size=(n, L))\n", + "J = 1.0\n", + "\n", + "energies = np.zeros(n)\n", + "\n", + "for i in range(n):\n", + " energies[i] = - J * np.dot(spins[i], np.roll(spins[i], 1))" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "A more general form for the one-dimensional Ising model is" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\n", + "
\n", + "\n", + "$$\n", + "\\begin{equation}\n", + " H = - \\sum_j^L \\sum_k^L s_j s_k J_{jk}.\n", + "\\label{_auto18} \\tag{28}\n", + "\\end{equation}\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Here we allow for interactions beyond the nearest neighbors and a more\n", + "adaptive coupling matrix. This latter expression can be formulated as\n", + "a matrix-product on the form" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\n", + "
\n", + "\n", + "$$\n", + "\\begin{equation}\n", + " H = X J,\n", + "\\label{_auto19} \\tag{29}\n", + "\\end{equation}\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "where $X_{jk} = s_j s_k$ and $J$ is the matrix consisting of the\n", + "elements $-J_{jk}$. This form of writing the energy fits perfectly\n", + "with the form utilized in linear regression, viz." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\n", + "
\n", + "\n", + "$$\n", + "\\begin{equation}\n", + " \\boldsymbol{y} = \\boldsymbol{X}\\boldsymbol{\\beta} + \\boldsymbol{\\epsilon}.\n", + "\\label{_auto20} \\tag{30}\n", + "\\end{equation}\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "We organize the data as we did above" + ] + }, + { + "cell_type": "code", + "execution_count": 26, + "metadata": { + "collapsed": false + }, + "outputs": [], + "source": [ + "X = np.zeros((n, L ** 2))\n", + "for i in range(n):\n", + " X[i] = np.outer(spins[i], spins[i]).ravel()\n", + "y = energies\n", + "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.96)\n", + "\n", + "X_train_own = np.concatenate(\n", + " (np.ones(len(X_train))[:, np.newaxis], X_train),\n", + " axis=1\n", + ")\n", + "\n", + "X_test_own = np.concatenate(\n", + " (np.ones(len(X_test))[:, np.newaxis], X_test),\n", + " axis=1\n", + ")" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "We will do all fitting with **Scikit-Learn**," + ] + }, + { + "cell_type": "code", + "execution_count": 27, + "metadata": { + "collapsed": false + }, + "outputs": [], + "source": [ + "clf = skl.LinearRegression().fit(X_train, y_train)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "When extracting the $J$-matrix we make sure to remove the intercept" + ] + }, + { + "cell_type": "code", + "execution_count": 28, + "metadata": { + "collapsed": false + }, + "outputs": [], + "source": [ + "J_sk = clf.coef_.reshape(L, L)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "And then we plot the results" + ] + }, + { + "cell_type": "code", + "execution_count": 29, + "metadata": { + "collapsed": false + }, + "outputs": [], + "source": [ + "fig = plt.figure(figsize=(20, 14))\n", + "im = plt.imshow(J_sk, **cmap_args)\n", + "plt.title(\"LinearRegression from Scikit-learn\", fontsize=18)\n", + "plt.xticks(fontsize=18)\n", + "plt.yticks(fontsize=18)\n", + "cb = fig.colorbar(im)\n", + "cb.ax.set_yticklabels(cb.ax.get_yticklabels(), fontsize=18)\n", + "plt.show()" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "The results perfectly with our previous discussion where we used our own code.\n", + "\n", + "## Ridge regression\n", + "\n", + "Having explored the ordinary least squares we move on to ridge\n", + "regression. In ridge regression we include a **regularizer**. This\n", + "involves a new cost function which leads to a new estimate for the\n", + "weights $\\boldsymbol{\\beta}$. This results in a penalized regression problem. The\n", + "cost function is given by" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "1\n", + "6\n", + "8\n", + " \n", + "<\n", + "<\n", + "<\n", + "!\n", + "!\n", + "M\n", + "A\n", + "T\n", + "H\n", + "_\n", + "B\n", + "L\n", + "O\n", + "C\n", + "K" + ] + }, + { + "cell_type": "code", + "execution_count": 30, + "metadata": { + "collapsed": false + }, + "outputs": [], + "source": [ + "_lambda = 0.1\n", + "clf_ridge = skl.Ridge(alpha=_lambda).fit(X_train, y_train)\n", + "J_ridge_sk = clf_ridge.coef_.reshape(L, L)\n", + "fig = plt.figure(figsize=(20, 14))\n", + "im = plt.imshow(J_ridge_sk, **cmap_args)\n", + "plt.title(\"Ridge from Scikit-learn\", fontsize=18)\n", + "plt.xticks(fontsize=18)\n", + "plt.yticks(fontsize=18)\n", + "cb = fig.colorbar(im)\n", + "cb.ax.set_yticklabels(cb.ax.get_yticklabels(), fontsize=18)\n", + "\n", + "plt.show()" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## LASSO regression\n", + "\n", + "In the **Least Absolute Shrinkage and Selection Operator** (LASSO)-method we get a third cost function." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\n", + "
\n", + "\n", + "$$\n", + "\\begin{equation}\n", + " C(\\boldsymbol{X}, \\boldsymbol{\\beta}; \\lambda) = (\\boldsymbol{X}\\boldsymbol{\\beta} - \\boldsymbol{y})^T(\\boldsymbol{X}\\boldsymbol{\\beta} - \\boldsymbol{y}) + \\lambda \\sqrt{\\boldsymbol{\\beta}^T\\boldsymbol{\\beta}}.\n", + "\\label{_auto22} \\tag{32}\n", + "\\end{equation}\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Finding the extremal point of this cost function is not so straight-forward as in least squares and ridge. We will therefore rely solely on the function ``Lasso`` from **Scikit-Learn**." + ] + }, + { + "cell_type": "code", + "execution_count": 31, + "metadata": { + "collapsed": false + }, + "outputs": [], + "source": [ + "clf_lasso = skl.Lasso(alpha=_lambda).fit(X_train, y_train)\n", + "J_lasso_sk = clf_lasso.coef_.reshape(L, L)\n", + "fig = plt.figure(figsize=(20, 14))\n", + "im = plt.imshow(J_lasso_sk, **cmap_args)\n", + "plt.title(\"Lasso from Scikit-learn\", fontsize=18)\n", + "plt.xticks(fontsize=18)\n", + "plt.yticks(fontsize=18)\n", + "cb = fig.colorbar(im)\n", + "cb.ax.set_yticklabels(cb.ax.get_yticklabels(), fontsize=18)\n", + "\n", + "plt.show()" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "It is quite striking how LASSO breaks the symmetry of the coupling\n", + "constant as opposed to ridge and OLS. We get a sparse solution with\n", + "$J_{j, j + 1} = -1$.\n", + "\n", + "\n", + "\n", + "## Performance as function of the regularization parameter\n", + "\n", + "We see how the different models perform for a different set of values for $\\lambda$." + ] + }, + { + "cell_type": "code", + "execution_count": 32, + "metadata": { + "collapsed": false + }, + "outputs": [], + "source": [ + "lambdas = np.logspace(-4, 5, 10)\n", + "\n", + "train_errors = {\n", + " \"ols_sk\": np.zeros(lambdas.size),\n", + " \"ridge_sk\": np.zeros(lambdas.size),\n", + " \"lasso_sk\": np.zeros(lambdas.size)\n", + "}\n", + "\n", + "test_errors = {\n", + " \"ols_sk\": np.zeros(lambdas.size),\n", + " \"ridge_sk\": np.zeros(lambdas.size),\n", + " \"lasso_sk\": np.zeros(lambdas.size)\n", + "}\n", + "\n", + "plot_counter = 1\n", + "\n", + "fig = plt.figure(figsize=(32, 54))\n", + "\n", + "for i, _lambda in enumerate(tqdm.tqdm(lambdas)):\n", + " for key, method in zip(\n", + " [\"ols_sk\", \"ridge_sk\", \"lasso_sk\"],\n", + " [skl.LinearRegression(), skl.Ridge(alpha=_lambda), skl.Lasso(alpha=_lambda)]\n", + " ):\n", + " method = method.fit(X_train, y_train)\n", + "\n", + " train_errors[key][i] = method.score(X_train, y_train)\n", + " test_errors[key][i] = method.score(X_test, y_test)\n", + "\n", + " omega = method.coef_.reshape(L, L)\n", + "\n", + " plt.subplot(10, 5, plot_counter)\n", + " plt.imshow(omega, **cmap_args)\n", + " plt.title(r\"%s, $\\lambda = %.4f$\" % (key, _lambda))\n", + " plot_counter += 1\n", + "\n", + "plt.show()" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "We see that LASSO reaches a good solution for low\n", + "values of $\\lambda$, but will \"wither\" when we increase $\\lambda$ too\n", + "much. Ridge is more stable over a larger range of values for\n", + "$\\lambda$, but eventually also fades away.\n", + "\n", + "## Finding the optimal value of $\\lambda$\n", + "\n", + "To determine which value of $\\lambda$ is best we plot the accuracy of\n", + "the models when predicting the training and the testing set. We expect\n", + "the accuracy of the training set to be quite good, but if the accuracy\n", + "of the testing set is much lower this tells us that we might be\n", + "subject to an overfit model. The ideal scenario is an accuracy on the\n", + "testing set that is close to the accuracy of the training set." + ] + }, + { + "cell_type": "code", + "execution_count": 33, + "metadata": { + "collapsed": false + }, + "outputs": [], + "source": [ + "fig = plt.figure(figsize=(20, 14))\n", + "\n", + "colors = {\n", + " \"ols_sk\": \"r\",\n", + " \"ridge_sk\": \"y\",\n", + " \"lasso_sk\": \"c\"\n", + "}\n", + "\n", + "for key in train_errors:\n", + " plt.semilogx(\n", + " lambdas,\n", + " train_errors[key],\n", + " colors[key],\n", + " label=\"Train {0}\".format(key),\n", + " linewidth=4.0\n", + " )\n", + "\n", + "for key in test_errors:\n", + " plt.semilogx(\n", + " lambdas,\n", + " test_errors[key],\n", + " colors[key] + \"--\",\n", + " label=\"Test {0}\".format(key),\n", + " linewidth=4.0\n", + " )\n", + "plt.legend(loc=\"best\", fontsize=18)\n", + "plt.xlabel(r\"$\\lambda$\", fontsize=18)\n", + "plt.ylabel(r\"$R^2$\", fontsize=18)\n", + "plt.tick_params(labelsize=18)\n", + "plt.show()" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "From the above figure we can see that LASSO with $\\lambda = 10^{-2}$\n", + "achieves a very good accuracy on the test set. This by far surpasses the\n", + "other models for all values of $\\lambda$.\n", + "\n", + "\n", + "\n", + "## Further Exercises\n", + "\n", + "### Exercise 1\n", + "\n", + "We will generate our own dataset for a function $y(x)$ where $x \\in [0,1]$ and defined by random numbers computed with the uniform distribution. The function $y$ is a quadratic polynomial in $x$ with added stochastic noise according to the normal distribution $\\cal {N}(0,1)$.\n", + "The following simple Python instructions define our $x$ and $y$ values (with 100 data points)." + ] + }, + { + "cell_type": "code", + "execution_count": 34, + "metadata": { + "collapsed": false + }, + "outputs": [], + "source": [ + "x = np.random.rand(100,1)\n", + "y = 5*x*x+0.1*np.random.randn(100,1)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "1. Write your own code (following the examples above) for computing the parametrization of the data set fitting a second-order polynomial. \n", + "\n", + "2. Use thereafter **scikit-learn** (see again the examples in the regression slides) and compare with your own code. \n", + "\n", + "3. Using scikit-learn, compute also the mean square error, a risk metric corresponding to the expected value of the squared (quadratic) error defined as" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "MSE(\\hat{y},\\hat{\\tilde{y}}) = \\frac{1}{n}\n", + "\\sum_{i=0}^{n-1}(y_i-\\tilde{y}_i)^2,\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "and the $R^2$ score function.\n", + "If $\\tilde{\\hat{y}}_i$ is the predicted value of the $i-th$ sample and $y_i$ is the corresponding true value, then the score $R^2$ is defined as" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "R^2(\\hat{y}, \\tilde{\\hat{y}}) = 1 - \\frac{\\sum_{i=0}^{n - 1} (y_i - \\tilde{y}_i)^2}{\\sum_{i=0}^{n - 1} (y_i - \\bar{y})^2},\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "where we have defined the mean value of $\\hat{y}$ as" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "\\bar{y} = \\frac{1}{n} \\sum_{i=0}^{n - 1} y_i.\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "You can use the functionality included in scikit-learn. If you feel\n", + "for it, you can use your own program and define functions which\n", + "compute the above two functions. Discuss the meaning of these\n", + "results. Try also to vary the coefficient in front of the added\n", + "stochastic noise term and discuss the quality of the fits.\n", + "\n", + "\n", + "\n", + "\n", + "### Exercise 2, variance of the parameters $\\beta$ in linear regression\n", + "\n", + "Show that the variance of the parameters $\\beta$ in the linear regression method (chapter 3, equation (3.8) of [Trevor Hastie, Robert Tibshirani, Jerome H. Friedman, The Elements of Statistical Learning, Springer](https://www.springer.com/gp/book/9780387848570)) is given as" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "\\mathrm{Var}(\\hat{\\beta}) = \\left(\\hat{X}^T\\hat{X}\\right)^{-1}\\sigma^2,\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "with" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "\\sigma^2 = \\frac{1}{N-p-1}\\sum_{i=1}^{N} (y_i-\\tilde{y}_i)^2,\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "where we have assumed that we fit a function of degree $p-1$ (for example a polynomial in $x$). \n", + "\n", + "\n", + "\n", + "### Exercise 3\n", + "\n", + "This exercise is a continuation of exercise 1. We will\n", + "use the same function to generate our data set, still staying with a\n", + "simple function $y(x)$ which we want to fit using linear regression,\n", + "but now extending the analysis to include the Ridge and the Lasso\n", + "regression methods. You can use the code under the Regression as an example on how to use the Ridge and the Lasso methods.\n", + "\n", + "We will thus again generate our own dataset for a function $y(x)$ where \n", + "$x \\in [0,1]$ and defined by random numbers computed with the uniform\n", + "distribution. The function $y$ is a quadratic polynomial in $x$ with\n", + "added stochastic noise according to the normal distribution $\\cal{N}(0,1)$.\n", + "\n", + "The following simple Python instructions define our $x$ and $y$ values (with 100 data points)." + ] + }, + { + "cell_type": "code", + "execution_count": 35, + "metadata": { + "collapsed": false + }, + "outputs": [], + "source": [ + "x = np.random.rand(100,1)\n", + "y = 5*x*x+0.1*np.random.randn(100,1)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "1. Write your own code for the Ridge method and compute the parametrization for different values of $\\lambda$. Compare and analyze your results with those from exercise 1. Study the dependence on $\\lambda$ while also varying the strength of the noise in your expression for $y(x)$. \n", + "\n", + "2. Repeat the above but using the functionality of **scikit-learn**. Compare your code with the results from **scikit-learn**. Remember to run with the same random numbers for generating $x$ and $y$. \n", + "\n", + "3. Our next step is to study the variance of the parameters $\\beta_1$ and $\\beta_2$ (assuming that we are parametrizing our function with a second-order polynomial. We will use standard linear regression and the Ridge regression. You can now opt for either writing your own function that calculates the variance of these paramaters (recall that this is equal to the diagonal elements of the matrix $(\\hat{X}^T\\hat{X})+\\lambda\\hat{I})^{-1}$) or use the functionality of **scikit-learn** and compute their variances. Discuss the results of these variances as functions \n", + "\n", + "4. Repeat the previous step but add now the Lasso method. Discuss your results and compare with standard regression and the Ridge regression results.\n", + "\n", + "5. Try to implement the cross-validation as well. \n", + "\n", + "6. Finally, using **scikit-learn** or your own code, compute also the mean square error, a risk metric corresponding to the expected value of the squared (quadratic) error defined as" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "MSE(\\hat{y},\\hat{\\tilde{y}}) = \\frac{1}{n}\n", + "\\sum_{i=0}^{n-1}(y_i-\\tilde{y}_i)^2,\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "and the $R^2$ score function.\n", + "If $\\tilde{\\hat{y}}_i$ is the predicted value of the $i-th$ sample and $y_i$ is the corresponding true value, then the score $R^2$ is defined as" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "R^2(\\hat{y}, \\tilde{\\hat{y}}) = 1 - \\frac{\\sum_{i=0}^{n - 1} (y_i - \\tilde{y}_i)^2}{\\sum_{i=0}^{n - 1} (y_i - \\bar{y})^2},\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "where we have defined the mean value of $\\hat{y}$ as" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "\\bar{y} = \\frac{1}{n} \\sum_{i=0}^{n - 1} y_i.\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Discuss these quantities as functions of the variable $\\lambda$ in the Ridge and Lasso regression methods. \n", + "\n", + "### Exercise 4\n", + "\n", + "We will study how\n", + "to fit polynomials to a specific two-dimensional function called\n", + "[Franke's\n", + "function](http://www.dtic.mil/dtic/tr/fulltext/u2/a081688.pdf). This\n", + "is a function which has been widely used when testing various interpolation and fitting\n", + "algorithms. Furthermore, after having established the model and the\n", + "method, we will employ resamling techniques such as the cross-validation and/or\n", + "the bootstrap methods, in order to perform a proper assessment of our models.\n", + "\n", + "\n", + "The Franke function, which is a weighted sum of four exponentials reads as follows" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "\\begin{align*}\n", + "f(x,y) &= \\frac{3}{4}\\exp{\\left(-\\frac{(9x-2)^2}{4} - \\frac{(9y-2)^2}{4}\\right)}+\\frac{3}{4}\\exp{\\left(-\\frac{(9x+1)^2}{49}- \\frac{(9y+1)}{10}\\right)} \\\\\n", + "&+\\frac{1}{2}\\exp{\\left(-\\frac{(9x-7)^2}{4} - \\frac{(9y-3)^2}{4}\\right)} -\\frac{1}{5}\\exp{\\left(-(9x-4)^2 - (9y-7)^2\\right) }.\n", + "\\end{align*}\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "The function will be defined for $x,y\\in [0,1]$. Our first step will\n", + "be to perform an OLS regression analysis of this function, trying out\n", + "a polynomial fit with an $x$ and $y$ dependence of the form $[x, y,\n", + "x^2, y^2, xy, \\dots]$. We will also include cross-validation and\n", + "bootstrap as resampling techniques. As in homeworks 1 and 2, we\n", + "can use a uniform distribution to set up the arrays of values for $x$\n", + "and $y$, or as in the example below just a fix values for $x$ and $y$ with a given step size.\n", + "In this case we will have two predictors and need to fit a\n", + "function (for example a polynomial) of $x$ and $y$. Thereafter we will\n", + "repeat much of the same procedure using the the Ridge and\n", + "Lasso regression methods, introducing thus a dependence on the bias\n", + "(penalty) $\\lambda$.\n", + "\n", + "\n", + "The Python fucntion for the Franke function is included here (it performs also a three-dimensional plot of it)" + ] + }, + { + "cell_type": "code", + "execution_count": 36, + "metadata": { + "collapsed": false + }, + "outputs": [], + "source": [ + "from mpl_toolkits.mplot3d import Axes3D\n", + "import matplotlib.pyplot as plt\n", + "from matplotlib import cm\n", + "from matplotlib.ticker import LinearLocator, FormatStrFormatter\n", + "import numpy as np\n", + "from random import random, seed\n", + "\n", + "fig = plt.figure()\n", + "ax = fig.gca(projection='3d')\n", + "\n", + "# Make data.\n", + "x = np.arange(0, 1, 0.05)\n", + "y = np.arange(0, 1, 0.05)\n", + "x, y = np.meshgrid(x,y)\n", + "\n", + "\n", + "def FrankeFunction(x,y):\n", + " term1 = 0.75*np.exp(-(0.25*(9*x-2)**2) - 0.25*((9*y-2)**2))\n", + " term2 = 0.75*np.exp(-((9*x+1)**2)/49.0 - 0.1*(9*y+1))\n", + " term3 = 0.5*np.exp(-(9*x-7)**2/4.0 - 0.25*((9*y-3)**2))\n", + " term4 = -0.2*np.exp(-(9*x-4)**2 - (9*y-7)**2)\n", + " return term1 + term2 + term3 + term4\n", + "\n", + "\n", + "z = FrankeFunction(x, y)\n", + "\n", + "# Plot the surface.\n", + "surf = ax.plot_surface(x, y, z, cmap=cm.coolwarm,\n", + " linewidth=0, antialiased=False)\n", + "\n", + "# Customize the z axis.\n", + "ax.set_zlim(-0.10, 1.40)\n", + "ax.zaxis.set_major_locator(LinearLocator(10))\n", + "ax.zaxis.set_major_formatter(FormatStrFormatter('%.02f'))\n", + "\n", + "# Add a color bar which maps values to colors.\n", + "fig.colorbar(surf, shrink=0.5, aspect=5)\n", + "\n", + "plt.show()" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "We will thus again generate our own dataset for a function $\\mathrm{FrankeFunction}(x,y)$ where \n", + "$x,y \\in [0,1]$ could be defined by random numbers computed with the uniform\n", + "distribution. The function $f(x,y)$ is the Franke function. You should explore also the addition\n", + "an added stochastic noise to this function using the normal distribution $\\cal{N}(0,1)$.\n", + "\n", + "Write your own code (using either a matrix inversion or a singular value decomposition from e.g., **numpy** ) or use your code from exercises 1 and 3\n", + "and perform a standard least square regression analysis using polynomials in $x$ and $y$ up to fifth order. Find the confidence intervals of the parameters $\\beta$ by computing their variances, evaluate the Mean Squared error (MSE)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "MSE(\\hat{y},\\hat{\\tilde{y}}) = \\frac{1}{n}\n", + "\\sum_{i=0}^{n-1}(y_i-\\tilde{y}_i)^2,\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "and the $R^2$ score function.\n", + "If $\\tilde{\\hat{y}}_i$ is the predicted value of the $i-th$ sample and $y_i$ is the corresponding true value, then the score $R^2$ is defined as" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "R^2(\\hat{y}, \\tilde{\\hat{y}}) = 1 - \\frac{\\sum_{i=0}^{n - 1} (y_i - \\tilde{y}_i)^2}{\\sum_{i=0}^{n - 1} (y_i - \\bar{y})^2},\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "where we have defined the mean value of $\\hat{y}$ as" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "\\bar{y} = \\frac{1}{n} \\sum_{i=0}^{n - 1} y_i.\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Perform a resampling of the data where you split the data in training data and test data. Implement the $k$-fold cross-validation algorithm and/or the bootstrap algorithm\n", + "and evaluate again the MSE and the $R^2$ functions resulting from the test data. Evaluate also the bias and variance of the final models.\n", + "\n", + "\n", + "Write then your own code for the Ridge method, either using matrix\n", + "inversion or the singular value decomposition as done for standard OLS. Perform the same analysis as in the\n", + "previous exercise (for the same polynomials and include resampling\n", + "techniques) but now for different values of $\\lambda$. Compare and\n", + "analyze your results with those obtained with standard OLS. Study the\n", + "dependence on $\\lambda$ while also varying eventually the strength of\n", + "the noise in your expression for $\\mathrm{FrankeFunction}(x,y)$.\n", + "\n", + "Then perform the same studies but now with Lasso regression. Use the functionalities of\n", + "**scikit-learn**. Give a critical discussion of the three methods and a\n", + "judgement of which model fits the data best." + ] + } + ], + "metadata": {}, + "nbformat": 4, + "nbformat_minor": 2 +} diff --git a/doc/src/Regression/Regression.log b/doc/src/Regression/Regression.log new file mode 100644 index 000000000..e34c03458 --- /dev/null +++ b/doc/src/Regression/Regression.log @@ -0,0 +1,2482 @@ +This is pdfTeX, Version 3.14159265-2.6-1.40.16 (TeX Live 2015) (preloaded format=pdflatex 2015.5.24) 22 JUL 2019 20:11 +entering extended mode + \write18 enabled. + %&-line parsing enabled. +**Regression +(./Regression.tex +LaTeX2e <2015/01/01> +Babel <3.9l> and hyphenation patterns for 79 languages loaded. +(/usr/local/texlive/2015/texmf-dist/tex/latex/base/article.cls +Document Class: article 2014/09/29 v1.4h Standard LaTeX document class +(/usr/local/texlive/2015/texmf-dist/tex/latex/base/size10.clo +File: size10.clo 2014/09/29 v1.4h Standard LaTeX file (size option) +) +\c@part=\count79 +\c@section=\count80 +\c@subsection=\count81 +\c@subsubsection=\count82 +\c@paragraph=\count83 +\c@subparagraph=\count84 +\c@figure=\count85 +\c@table=\count86 +\abovecaptionskip=\skip41 +\belowcaptionskip=\skip42 +\bibindent=\dimen102 +) +(/usr/local/texlive/2015/texmf-dist/tex/latex/relsize/relsize.sty +Package: relsize 2013/03/29 ver 4.1 +) +(/usr/local/texlive/2015/texmf-dist/tex/latex/base/makeidx.sty +Package: makeidx 2014/09/29 v1.0m Standard LaTeX package +) +(/usr/local/texlive/2015/texmf-dist/tex/latex/graphics/color.sty +Package: color 2014/10/28 v1.1a Standard LaTeX Color (DPC) + +(/usr/local/texlive/2015/texmf-dist/tex/latex/latexconfig/color.cfg +File: color.cfg 2007/01/18 v1.5 color configuration of teTeX/TeXLive +) +Package color Info: Driver file: pdftex.def on input line 142. + +(/usr/local/texlive/2015/texmf-dist/tex/latex/pdftex-def/pdftex.def +File: pdftex.def 2011/05/27 v0.06d Graphics/color for pdfTeX + +(/usr/local/texlive/2015/texmf-dist/tex/generic/oberdiek/infwarerr.sty +Package: infwarerr 2010/04/08 v1.3 Providing info/warning/error messages (HO) +) +(/usr/local/texlive/2015/texmf-dist/tex/generic/oberdiek/ltxcmds.sty +Package: ltxcmds 2011/11/09 v1.22 LaTeX kernel commands for general use (HO) +) +\Gread@gobject=\count87 +)) +(/usr/local/texlive/2015/texmf-dist/tex/latex/setspace/setspace.sty +Package: setspace 2011/12/19 v6.7a set line spacing +) +(/usr/local/texlive/2015/texmf-dist/tex/latex/amsmath/amsmath.sty +Package: amsmath 2013/01/14 v2.14 AMS math features +\@mathmargin=\skip43 + +For additional information on amsmath, use the `?' option. +(/usr/local/texlive/2015/texmf-dist/tex/latex/amsmath/amstext.sty +Package: amstext 2000/06/29 v2.01 + +(/usr/local/texlive/2015/texmf-dist/tex/latex/amsmath/amsgen.sty +File: amsgen.sty 1999/11/30 v2.0 +\@emptytoks=\toks14 +\ex@=\dimen103 +)) +(/usr/local/texlive/2015/texmf-dist/tex/latex/amsmath/amsbsy.sty +Package: amsbsy 1999/11/29 v1.2d +\pmbraise@=\dimen104 +) +(/usr/local/texlive/2015/texmf-dist/tex/latex/amsmath/amsopn.sty +Package: amsopn 1999/12/14 v2.01 operator names +) +\inf@bad=\count88 +LaTeX Info: Redefining \frac on input line 210. +\uproot@=\count89 +\leftroot@=\count90 +LaTeX Info: Redefining \overline on input line 306. +\classnum@=\count91 +\DOTSCASE@=\count92 +LaTeX Info: Redefining \ldots on input line 378. +LaTeX Info: Redefining \dots on input line 381. +LaTeX Info: Redefining \cdots on input line 466. +\Mathstrutbox@=\box26 +\strutbox@=\box27 +\big@size=\dimen105 +LaTeX Font Info: Redeclaring font encoding OML on input line 566. +LaTeX Font Info: Redeclaring font encoding OMS on input line 567. +\macc@depth=\count93 +\c@MaxMatrixCols=\count94 +\dotsspace@=\muskip10 +\c@parentequation=\count95 +\dspbrk@lvl=\count96 +\tag@help=\toks15 +\row@=\count97 +\column@=\count98 +\maxfields@=\count99 +\andhelp@=\toks16 +\eqnshift@=\dimen106 +\alignsep@=\dimen107 +\tagshift@=\dimen108 +\tagwidth@=\dimen109 +\totwidth@=\dimen110 +\lineht@=\dimen111 +\@envbody=\toks17 +\multlinegap=\skip44 +\multlinetaggap=\skip45 +\mathdisplay@stack=\toks18 +LaTeX Info: Redefining \[ on input line 2665. +LaTeX Info: Redefining \] on input line 2666. +) +(/usr/local/texlive/2015/texmf-dist/tex/latex/amsfonts/amsfonts.sty +Package: amsfonts 2013/01/14 v3.01 Basic AMSFonts support +\symAMSa=\mathgroup4 +\symAMSb=\mathgroup5 +LaTeX Font Info: Overwriting math alphabet `\mathfrak' in version `bold' +(Font) U/euf/m/n --> U/euf/b/n on input line 106. +) +(/usr/local/texlive/2015/texmf-dist/tex/latex/amsfonts/amssymb.sty +Package: amssymb 2013/01/14 v3.01 AMS font symbols +) +(/usr/local/texlive/2015/texmf-dist/tex/latex/xcolor/xcolor.sty +Package: xcolor 2007/01/21 v2.11 LaTeX color extensions (UK) + +(/usr/local/texlive/2015/texmf-dist/tex/latex/latexconfig/color.cfg +File: color.cfg 2007/01/18 v1.5 color configuration of teTeX/TeXLive +) +Package xcolor Info: Driver file: pdftex.def on input line 225. + +(/usr/local/texlive/2015/texmf-dist/tex/latex/colortbl/colortbl.sty +Package: colortbl 2012/02/13 v1.0a Color table columns (DPC) + +(/usr/local/texlive/2015/texmf-dist/tex/latex/tools/array.sty +Package: array 2014/10/28 v2.4c Tabular extension package (FMi) +\col@sep=\dimen112 +\extrarowheight=\dimen113 +\NC@list=\toks19 +\extratabsurround=\skip46 +\backup@length=\skip47 +) +\everycr=\toks20 +\minrowclearance=\skip48 +) +LaTeX Info: Redefining \color on input line 702. +\rownum=\count100 +Package xcolor Info: Model `cmy' substituted by `cmy0' on input line 1337. +Package xcolor Info: Model `hsb' substituted by `rgb' on input line 1341. +Package xcolor Info: Model `RGB' extended on input line 1353. +Package xcolor Info: Model `HTML' substituted by `rgb' on input line 1355. +Package xcolor Info: Model `Hsb' substituted by `hsb' on input line 1356. +Package xcolor Info: Model `tHsb' substituted by `hsb' on input line 1357. +Package xcolor Info: Model `HSB' substituted by `hsb' on input line 1358. +Package xcolor Info: Model `Gray' substituted by `gray' on input line 1359. +Package xcolor Info: Model `wave' substituted by `hsb' on input line 1360. +) +(/usr/local/texlive/2015/texmf-dist/tex/latex/tools/bm.sty +Package: bm 2014/10/28 v1.1c Bold Symbol Support (DPC/FMi) +\symboldoperators=\mathgroup6 +\symboldletters=\mathgroup7 +\symboldsymbols=\mathgroup8 +LaTeX Font Info: Redeclaring math alphabet \mathbf on input line 141. +LaTeX Info: Redefining \bm on input line 207. +) +(/usr/local/texlive/2015/texmf-dist/tex/latex/ltablex/ltablex.sty +Package: ltablex 2014/08/13 v1.1 Modified tabularx + +(/usr/local/texlive/2015/texmf-dist/tex/latex/tools/longtable.sty +Package: longtable 2014/10/28 v4.11 Multi-page Table package (DPC) +\LTleft=\skip49 +\LTright=\skip50 +\LTpre=\skip51 +\LTpost=\skip52 +\LTchunksize=\count101 +\LTcapwidth=\dimen114 +\LT@head=\box28 +\LT@firsthead=\box29 +\LT@foot=\box30 +\LT@lastfoot=\box31 +\LT@cols=\count102 +\LT@rows=\count103 +\c@LT@tables=\count104 +\c@LT@chunks=\count105 +\LT@p@ftn=\toks21 +) +(/usr/local/texlive/2015/texmf-dist/tex/latex/tools/tabularx.sty +Package: tabularx 2014/10/28 v2.10 `tabularx' package (DPC) +\TX@col@width=\dimen115 +\TX@old@table=\dimen116 +\TX@old@col=\dimen117 +\TX@target=\dimen118 +\TX@delta=\dimen119 +\TX@cols=\count106 +\TX@ftn=\toks22 +)) +(/usr/local/texlive/2015/texmf-dist/tex/latex/microtype/microtype.sty +Package: microtype 2013/05/23 v2.5a Micro-typographical refinements (RS) + +(/usr/local/texlive/2015/texmf-dist/tex/latex/graphics/keyval.sty +Package: keyval 2014/10/28 v1.15 key=value parser (DPC) +\KV@toks@=\toks23 +) +\MT@toks=\toks24 +\MT@count=\count107 +LaTeX Info: Redefining \textls on input line 766. +\MT@outer@kern=\dimen120 +LaTeX Info: Redefining \textmicrotypecontext on input line 1285. +\MT@listname@count=\count108 + +(/usr/local/texlive/2015/texmf-dist/tex/latex/microtype/microtype-pdftex.def +File: microtype-pdftex.def 2013/05/23 v2.5a Definitions specific to pdftex (RS) + +LaTeX Info: Redefining \lsstyle on input line 915. +LaTeX Info: Redefining \lslig on input line 915. +\MT@outer@space=\skip53 +) +Package microtype Info: Loading configuration file microtype.cfg. + +(/usr/local/texlive/2015/texmf-dist/tex/latex/microtype/microtype.cfg +File: microtype.cfg 2013/05/23 v2.5a microtype main configuration file (RS) +)) +(/usr/local/texlive/2015/texmf-dist/tex/latex/graphics/graphicx.sty +Package: graphicx 2014/10/28 v1.0g Enhanced LaTeX Graphics (DPC,SPQR) + +(/usr/local/texlive/2015/texmf-dist/tex/latex/graphics/graphics.sty +Package: graphics 2014/10/28 v1.0p Standard LaTeX Graphics (DPC,SPQR) + +(/usr/local/texlive/2015/texmf-dist/tex/latex/graphics/trig.sty +Package: trig 1999/03/16 v1.09 sin cos tan (DPC) +) +(/usr/local/texlive/2015/texmf-dist/tex/latex/latexconfig/graphics.cfg +File: graphics.cfg 2010/04/23 v1.9 graphics configuration of TeX Live +) +Package graphics Info: Driver file: pdftex.def on input line 94. +) +\Gin@req@height=\dimen121 +\Gin@req@width=\dimen122 +) +(/usr/local/texlive/2015/texmf-dist/tex/latex/fancyvrb/fancyvrb.sty +Package: fancyvrb 2008/02/07 + +Style option: `fancyvrb' v2.7a, with DG/SPQR fixes, and firstline=lastline fix +<2008/02/07> (tvz) +\FV@CodeLineNo=\count109 +\FV@InFile=\read1 +\FV@TabBox=\box32 +\c@FancyVerbLine=\count110 +\FV@StepNumber=\count111 +\FV@OutFile=\write3 +) +(/usr/local/texlive/2015/texmf-dist/tex/latex/minted/minted.sty +Package: minted 2015/01/31 v2.0 Yet another Pygments shim for LaTeX + +(/usr/local/texlive/2015/texmf-dist/tex/latex/oberdiek/kvoptions.sty +Package: kvoptions 2011/06/30 v3.11 Key value format for package options (HO) + +(/usr/local/texlive/2015/texmf-dist/tex/generic/oberdiek/kvsetkeys.sty +Package: kvsetkeys 2012/04/25 v1.16 Key value parser (HO) + +(/usr/local/texlive/2015/texmf-dist/tex/generic/oberdiek/etexcmds.sty +Package: etexcmds 2011/02/16 v1.5 Avoid name clashes with e-TeX commands (HO) + +(/usr/local/texlive/2015/texmf-dist/tex/generic/oberdiek/ifluatex.sty +Package: ifluatex 2010/03/01 v1.3 Provides the ifluatex switch (HO) +Package ifluatex Info: LuaTeX not detected. +) +Package etexcmds Info: Could not find \expanded. +(etexcmds) That can mean that you are not using pdfTeX 1.50 or +(etexcmds) that some package has redefined \expanded. +(etexcmds) In the latter case, load this package earlier. +))) +(/usr/local/texlive/2015/texmf-dist/tex/latex/float/float.sty +Package: float 2001/11/08 v1.3d Float enhancements (AL) +\c@float@type=\count112 +\float@exts=\toks25 +\float@box=\box33 +\@float@everytoks=\toks26 +\@floatcapt=\box34 +) +(/usr/local/texlive/2015/texmf-dist/tex/latex/base/ifthen.sty +Package: ifthen 2014/09/29 v1.1c Standard LaTeX ifthen package (DPC) +) +(/usr/local/texlive/2015/texmf-dist/tex/latex/tools/calc.sty +Package: calc 2014/10/28 v4.3 Infix arithmetic (KKT,FJ) +\calc@Acount=\count113 +\calc@Bcount=\count114 +\calc@Adimen=\dimen123 +\calc@Bdimen=\dimen124 +\calc@Askip=\skip54 +\calc@Bskip=\skip55 +LaTeX Info: Redefining \setlength on input line 80. +LaTeX Info: Redefining \addtolength on input line 81. +\calc@Ccount=\count115 +\calc@Cskip=\skip56 +) +(/usr/local/texlive/2015/texmf-dist/tex/latex/ifplatform/ifplatform.sty +Package: ifplatform 2010/10/22 v0.4 Testing for the operating system + +(/usr/local/texlive/2015/texmf-dist/tex/generic/oberdiek/pdftexcmds.sty +Package: pdftexcmds 2011/11/29 v0.20 Utility functions of pdfTeX for LuaTeX (HO +) + +(/usr/local/texlive/2015/texmf-dist/tex/generic/oberdiek/ifpdf.sty +Package: ifpdf 2011/01/30 v2.3 Provides the ifpdf switch (HO) +Package ifpdf Info: pdfTeX in PDF mode is detected. +) +Package pdftexcmds Info: LuaTeX not detected. +Package pdftexcmds Info: \pdf@primitive is available. +Package pdftexcmds Info: \pdf@ifprimitive is available. +Package pdftexcmds Info: \pdfdraftmode found. +) +(/usr/local/texlive/2015/texmf-dist/tex/generic/oberdiek/catchfile.sty +Package: catchfile 2011/03/01 v1.6 Catch the contents of a file (HO) +) +runsystem(uname -s > "Regression.w18")...executed. + + +(./Regression.w18) +runsystem(rm -- "Regression.w18")...executed. + +) +(/usr/local/texlive/2015/texmf-dist/tex/latex/etoolbox/etoolbox.sty +Package: etoolbox 2015/05/04 v2.2 e-TeX tools for LaTeX (JAW) +\etb@tempcnta=\count116 +) +(/usr/local/texlive/2015/texmf-dist/tex/generic/xstring/xstring.sty +(/usr/local/texlive/2015/texmf-dist/tex/generic/xstring/xstring.tex +\@xs@message=\write4 +\integerpart=\count117 +\decimalpart=\count118 +) +Package: xstring 2013/10/13 v1.7c String manipulations (C Tellechea) +) +(/usr/local/texlive/2015/texmf-dist/tex/latex/lineno/lineno.sty +Package: lineno 2005/11/02 line numbers on paragraphs v4.41 +\linenopenalty=\count119 +\output=\toks27 +\linenoprevgraf=\count120 +\linenumbersep=\dimen125 +\linenumberwidth=\dimen126 +\c@linenumber=\count121 +\c@pagewiselinenumber=\count122 +\c@LN@truepage=\count123 +\c@internallinenumber=\count124 +\c@internallinenumbers=\count125 +\quotelinenumbersep=\dimen127 +\bframerule=\dimen128 +\bframesep=\dimen129 +\bframebox=\box35 +LaTeX Info: Redefining \\ on input line 3056. +) +\minted@appexistsfile=\read2 +\FV@BreakIndent=\dimen130 +\FV@BreakSymbolSepLeft=\dimen131 +\FV@BreakSymbolSepRight=\dimen132 +\FV@BreakSymbolIndentLeft=\dimen133 +\FV@BreakSymbolIndentRight=\dimen134 +\c@FancyVerbLineBreakLast=\count126 +\FV@LineBox=\box36 +\FV@LineIndentBox=\box37 +\minted@bgbox=\box38 +\minted@code=\write5 +\c@minted@FancyVerbLineTemp=\count127 +\@float@every@listing=\toks28 +\c@listing=\count128 +) +runsystem(mkdir -p "_minted-Regression")...executed. + +runsystem(which "pygmentize" && touch "Regression.aex")...executed. + +runsystem(rm "Regression.aex")...executed. + + +(./_minted-Regression/default.pygstyle) +(/usr/local/texlive/2015/texmf-dist/tex/latex/base/fontenc.sty +Package: fontenc 2005/09/27 v1.99g Standard LaTeX package + +(/usr/local/texlive/2015/texmf-dist/tex/latex/base/t1enc.def +File: t1enc.def 2005/09/27 v1.99g Standard LaTeX file +LaTeX Font Info: Redeclaring font encoding T1 on input line 48. +)) +(/usr/local/texlive/2015/texmf-dist/tex/latex/ucs/ucs.sty +Package: ucs 2013/05/11 v2.2 UCS: Unicode input support + +(/usr/local/texlive/2015/texmf-dist/tex/latex/ucs/data/uni-global.def +File: uni-global.def 2013/05/13 UCS: Unicode global data +) +\uc@secondtry=\count129 +\uc@combtoks=\toks29 +\uc@combtoksb=\toks30 +\uc@temptokena=\toks31 +) +(/usr/local/texlive/2015/texmf-dist/tex/latex/base/inputenc.sty +Package: inputenc 2015/03/17 v1.2c Input encoding file +\inpenc@prehook=\toks32 +\inpenc@posthook=\toks33 + +(/usr/local/texlive/2015/texmf-dist/tex/latex/ucs/utf8x.def +File: utf8x.def 2004/10/17 UCS: Input encoding UTF-8 +)) +(/usr/local/texlive/2015/texmf-dist/tex/latex/lm/lmodern.sty +Package: lmodern 2009/10/30 v1.6 Latin Modern Fonts +LaTeX Font Info: Overwriting symbol font `operators' in version `normal' +(Font) OT1/cmr/m/n --> OT1/lmr/m/n on input line 22. +LaTeX Font Info: Overwriting symbol font `letters' in version `normal' +(Font) OML/cmm/m/it --> OML/lmm/m/it on input line 23. +LaTeX Font Info: Overwriting symbol font `symbols' in version `normal' +(Font) OMS/cmsy/m/n --> OMS/lmsy/m/n on input line 24. +LaTeX Font Info: Overwriting symbol font `largesymbols' in version `normal' +(Font) OMX/cmex/m/n --> OMX/lmex/m/n on input line 25. +LaTeX Font Info: Overwriting symbol font `operators' in version `bold' +(Font) OT1/cmr/bx/n --> OT1/lmr/bx/n on input line 26. +LaTeX Font Info: Overwriting symbol font `letters' in version `bold' +(Font) OML/cmm/b/it --> OML/lmm/b/it on input line 27. +LaTeX Font Info: Overwriting symbol font `symbols' in version `bold' +(Font) OMS/cmsy/b/n --> OMS/lmsy/b/n on input line 28. +LaTeX Font Info: Overwriting symbol font `largesymbols' in version `bold' +(Font) OMX/cmex/m/n --> OMX/lmex/m/n on input line 29. +LaTeX Font Info: Overwriting math alphabet `\mathbf' in version `normal' +(Font) OT1/cmr/bx/n --> OT1/lmr/bx/n on input line 31. +LaTeX Font Info: Overwriting math alphabet `\mathsf' in version `normal' +(Font) OT1/cmss/m/n --> OT1/lmss/m/n on input line 32. +LaTeX Font Info: Overwriting math alphabet `\mathit' in version `normal' +(Font) OT1/cmr/m/it --> OT1/lmr/m/it on input line 33. +LaTeX Font Info: Overwriting math alphabet `\mathtt' in version `normal' +(Font) OT1/cmtt/m/n --> OT1/lmtt/m/n on input line 34. +LaTeX Font Info: Overwriting math alphabet `\mathbf' in version `bold' +(Font) OT1/cmr/bx/n --> OT1/lmr/bx/n on input line 35. +LaTeX Font Info: Overwriting math alphabet `\mathsf' in version `bold' +(Font) OT1/cmss/bx/n --> OT1/lmss/bx/n on input line 36. +LaTeX Font Info: Overwriting math alphabet `\mathit' in version `bold' +(Font) OT1/cmr/bx/it --> OT1/lmr/bx/it on input line 37. +LaTeX Font Info: Overwriting math alphabet `\mathtt' in version `bold' +(Font) OT1/cmtt/m/n --> OT1/lmtt/m/n on input line 38. +) +(/usr/local/texlive/2015/texmf-dist/tex/latex/hyperref/hyperref.sty +Package: hyperref 2012/11/06 v6.83m Hypertext links for LaTeX + +(/usr/local/texlive/2015/texmf-dist/tex/generic/oberdiek/hobsub-hyperref.sty +Package: hobsub-hyperref 2012/05/28 v1.13 Bundle oberdiek, subset hyperref (HO) + + +(/usr/local/texlive/2015/texmf-dist/tex/generic/oberdiek/hobsub-generic.sty +Package: hobsub-generic 2012/05/28 v1.13 Bundle oberdiek, subset generic (HO) +Package: hobsub 2012/05/28 v1.13 Construct package bundles (HO) +Package hobsub Info: Skipping package `infwarerr' (already loaded). +Package hobsub Info: Skipping package `ltxcmds' (already loaded). +Package hobsub Info: Skipping package `ifluatex' (already loaded). +Package: ifvtex 2010/03/01 v1.5 Detect VTeX and its facilities (HO) +Package ifvtex Info: VTeX not detected. +Package: intcalc 2007/09/27 v1.1 Expandable calculations with integers (HO) +Package hobsub Info: Skipping package `ifpdf' (already loaded). +Package hobsub Info: Skipping package `etexcmds' (already loaded). +Package hobsub Info: Skipping package `kvsetkeys' (already loaded). +Package: kvdefinekeys 2011/04/07 v1.3 Define keys (HO) +Package hobsub Info: Skipping package `pdftexcmds' (already loaded). +Package: pdfescape 2011/11/25 v1.13 Implements pdfTeX's escape features (HO) +Package: bigintcalc 2012/04/08 v1.3 Expandable calculations on big integers (HO +) +Package: bitset 2011/01/30 v1.1 Handle bit-vector datatype (HO) +Package: uniquecounter 2011/01/30 v1.2 Provide unlimited unique counter (HO) +) +Package hobsub Info: Skipping package `hobsub' (already loaded). +Package: letltxmacro 2010/09/02 v1.4 Let assignment for LaTeX macros (HO) +Package: hopatch 2012/05/28 v1.2 Wrapper for package hooks (HO) +Package: xcolor-patch 2011/01/30 xcolor patch +Package: atveryend 2011/06/30 v1.8 Hooks at the very end of document (HO) +Package: atbegshi 2011/10/05 v1.16 At begin shipout hook (HO) +Package: refcount 2011/10/16 v3.4 Data extraction from label references (HO) +Package: hycolor 2011/01/30 v1.7 Color options for hyperref/bookmark (HO) +) +(/usr/local/texlive/2015/texmf-dist/tex/generic/ifxetex/ifxetex.sty +Package: ifxetex 2010/09/12 v0.6 Provides ifxetex conditional +) +(/usr/local/texlive/2015/texmf-dist/tex/latex/oberdiek/auxhook.sty +Package: auxhook 2011/03/04 v1.3 Hooks for auxiliary files (HO) +) +\@linkdim=\dimen135 +\Hy@linkcounter=\count130 +\Hy@pagecounter=\count131 + +(/usr/local/texlive/2015/texmf-dist/tex/latex/hyperref/pd1enc.def +File: pd1enc.def 2012/11/06 v6.83m Hyperref: PDFDocEncoding definition (HO) +) +\Hy@SavedSpaceFactor=\count132 + +(/usr/local/texlive/2015/texmf-dist/tex/latex/latexconfig/hyperref.cfg +File: hyperref.cfg 2002/06/06 v1.2 hyperref configuration of TeXLive +) +Package hyperref Info: Option `final' set `true' on input line 4319. +Package hyperref Info: Hyper figures OFF on input line 4443. +Package hyperref Info: Link nesting OFF on input line 4448. +Package hyperref Info: Hyper index ON on input line 4451. +Package hyperref Info: Plain pages OFF on input line 4458. +Package hyperref Info: Backreferencing OFF on input line 4463. +Package hyperref Info: Implicit mode ON; LaTeX internals redefined. +Package hyperref Info: Bookmarks ON on input line 4688. +\c@Hy@tempcnt=\count133 + +(/usr/local/texlive/2015/texmf-dist/tex/latex/url/url.sty +\Urlmuskip=\muskip11 +Package: url 2013/09/16 ver 3.4 Verb mode for urls, etc. +) +LaTeX Info: Redefining \url on input line 5041. +\XeTeXLinkMargin=\dimen136 +\Fld@menulength=\count134 +\Field@Width=\dimen137 +\Fld@charsize=\dimen138 +Package hyperref Info: Hyper figures OFF on input line 6295. +Package hyperref Info: Link nesting OFF on input line 6300. +Package hyperref Info: Hyper index ON on input line 6303. +Package hyperref Info: backreferencing OFF on input line 6310. +Package hyperref Info: Link coloring OFF on input line 6315. +Package hyperref Info: Link coloring with OCG OFF on input line 6320. +Package hyperref Info: PDF/A mode OFF on input line 6325. +LaTeX Info: Redefining \ref on input line 6365. +LaTeX Info: Redefining \pageref on input line 6369. +\Hy@abspage=\count135 +\c@Item=\count136 +\c@Hfootnote=\count137 +) + +Package hyperref Message: Driver (autodetected): hpdftex. + +(/usr/local/texlive/2015/texmf-dist/tex/latex/hyperref/hpdftex.def +File: hpdftex.def 2012/11/06 v6.83m Hyperref driver for pdfTeX +\Fld@listcount=\count138 +\c@bookmark@seq@number=\count139 + +(/usr/local/texlive/2015/texmf-dist/tex/latex/oberdiek/rerunfilecheck.sty +Package: rerunfilecheck 2011/04/15 v1.7 Rerun checks for auxiliary files (HO) +Package uniquecounter Info: New unique counter `rerunfilecheck' on input line 2 +82. +) +\Hy@SectionHShift=\skip57 +) +Package hyperref Info: Option `breaklinks' set `true' on input line 48. +Package hyperref Info: Option `colorlinks' set `true' on input line 48. +Package hyperref Info: Option `pdfmenubar' set `true' on input line 48. +Package hyperref Info: Option `pdftoolbar' set `true' on input line 48. + +(/usr/local/texlive/2015/texmf-dist/tex/latex/fancyhdr/fancyhdr.sty +\fancy@headwidth=\skip58 +\f@ncyO@elh=\skip59 +\f@ncyO@erh=\skip60 +\f@ncyO@olh=\skip61 +\f@ncyO@orh=\skip62 +\f@ncyO@elf=\skip63 +\f@ncyO@erf=\skip64 +\f@ncyO@olf=\skip65 +\f@ncyO@orf=\skip66 +) + +Package Fancyhdr Warning: \fancyfoot's `E' option without twoside option is use +less on input line 57. + +(/usr/local/texlive/2015/texmf-dist/tex/latex/mdframed/mdframed.sty +Package: mdframed 2013/07/01 1.9b: mdframed + +(/usr/local/texlive/2015/texmf-dist/tex/latex/l3packages/xparse/xparse.sty +(/usr/local/texlive/2015/texmf-dist/tex/latex/l3kernel/expl3.sty +Package: expl3 2015/03/01 v5547 L3 programming layer (loader) + +(/usr/local/texlive/2015/texmf-dist/tex/latex/l3kernel/expl3-code.tex +Package: expl3 2015/03/01 v5547 L3 programming layer (code) +L3 Module: l3bootstrap 2015/02/28 v5542 L3 Bootstrap code +L3 Module: l3names 2015/02/24 v5535 L3 Namespace for primitives +L3 Module: l3basics 2015/01/27 v5500 L3 Basic definitions +L3 Module: l3expan 2014/11/27 v5472 L3 Argument expansion +L3 Module: l3tl 2015/01/27 v5500 L3 Token lists +L3 Module: l3str 2015/03/01 v5545 L3 Strings +L3 Module: l3seq 2014/08/23 v5354 L3 Sequences and stacks +L3 Module: l3int 2015/02/21 v5529 L3 Integers +\c_max_int=\count140 +\l_tmpa_int=\count141 +\l_tmpb_int=\count142 +\g_tmpa_int=\count143 +\g_tmpb_int=\count144 +L3 Module: l3quark 2014/08/23 v5354 L3 Quarks +L3 Module: l3prg 2014/08/23 v5354 L3 Control structures +\g__prg_map_int=\count145 +L3 Module: l3clist 2014/08/23 v5354 L3 Comma separated lists +L3 Module: l3token 2014/09/15 v5422 L3 Experimental token manipulation +L3 Module: l3prop 2014/08/23 v5354 L3 Property lists +L3 Module: l3msg 2015/02/26 v5537 L3 Messages +L3 Module: l3file 2014/08/24 v5369 L3 File and I/O operations +\l_iow_line_count_int=\count146 +\l__iow_target_count_int=\count147 +\l__iow_current_line_int=\count148 +\l__iow_current_word_int=\count149 +\l__iow_current_indentation_int=\count150 +L3 Module: l3skip 2014/08/23 v5354 L3 Dimensions and skips +\c_zero_dim=\dimen139 +\c_max_dim=\dimen140 +\l_tmpa_dim=\dimen141 +\l_tmpb_dim=\dimen142 +\g_tmpa_dim=\dimen143 +\g_tmpb_dim=\dimen144 +\c_zero_skip=\skip67 +\c_max_skip=\skip68 +\l_tmpa_skip=\skip69 +\l_tmpb_skip=\skip70 +\g_tmpa_skip=\skip71 +\g_tmpb_skip=\skip72 +\c_zero_muskip=\muskip12 +\c_max_muskip=\muskip13 +\l_tmpa_muskip=\muskip14 +\l_tmpb_muskip=\muskip15 +\g_tmpa_muskip=\muskip16 +\g_tmpb_muskip=\muskip17 +L3 Module: l3keys 2015/01/27 v5500 L3 Key-value interfaces +\g__keyval_level_int=\count151 +\l_keys_choice_int=\count152 +L3 Module: l3fp 2014/08/22 v5336 L3 Floating points +\c__fp_leading_shift_int=\count153 +\c__fp_middle_shift_int=\count154 +\c__fp_trailing_shift_int=\count155 +\c__fp_big_leading_shift_int=\count156 +\c__fp_big_middle_shift_int=\count157 +\c__fp_big_trailing_shift_int=\count158 +\c__fp_Bigg_leading_shift_int=\count159 +\c__fp_Bigg_middle_shift_int=\count160 +\c__fp_Bigg_trailing_shift_int=\count161 +L3 Module: l3box 2014/08/23 v5354 L3 Experimental boxes +\c_empty_box=\box39 +\l_tmpa_box=\box40 +\l_tmpb_box=\box41 +\g_tmpa_box=\box42 +\g_tmpb_box=\box43 +L3 Module: l3coffins 2014/08/23 v5354 L3 Coffin code layer +\l__coffin_internal_box=\box44 +\l__coffin_internal_dim=\dimen145 +\l__coffin_offset_x_dim=\dimen146 +\l__coffin_offset_y_dim=\dimen147 +\l__coffin_x_dim=\dimen148 +\l__coffin_y_dim=\dimen149 +\l__coffin_x_prime_dim=\dimen150 +\l__coffin_y_prime_dim=\dimen151 +\c_empty_coffin=\box45 +\l__coffin_aligned_coffin=\box46 +\l__coffin_aligned_internal_coffin=\box47 +\l_tmpa_coffin=\box48 +\l_tmpb_coffin=\box49 +\l__coffin_display_coffin=\box50 +\l__coffin_display_coord_coffin=\box51 +\l__coffin_display_pole_coffin=\box52 +\l__coffin_display_offset_dim=\dimen152 +\l__coffin_display_x_dim=\dimen153 +\l__coffin_display_y_dim=\dimen154 +L3 Module: l3color 2014/08/23 v5354 L3 Experimental color support +L3 Module: l3candidates 2015/03/01 v5544 L3 Experimental additions to l3kernel +\l__box_top_dim=\dimen155 +\l__box_bottom_dim=\dimen156 +\l__box_left_dim=\dimen157 +\l__box_right_dim=\dimen158 +\l__box_top_new_dim=\dimen159 +\l__box_bottom_new_dim=\dimen160 +\l__box_left_new_dim=\dimen161 +\l__box_right_new_dim=\dimen162 +\l__box_internal_box=\box53 +\l__coffin_bounding_shift_dim=\dimen163 +\l__coffin_left_corner_dim=\dimen164 +\l__coffin_right_corner_dim=\dimen165 +\l__coffin_bottom_corner_dim=\dimen166 +\l__coffin_top_corner_dim=\dimen167 +\l__coffin_scaled_total_height_dim=\dimen168 +\l__coffin_scaled_width_dim=\dimen169 +) +(/usr/local/texlive/2015/texmf-dist/tex/latex/l3kernel/l3unicode-data.def +File: l3unicode-data.def 2015/03/01 v5544 L3 Unicode data +) +(/usr/local/texlive/2015/texmf-dist/tex/latex/l3kernel/l3pdfmode.def +File: l3pdfmode.def 2015/03/01 v5544 L3 Experimental driver: PDF mode +\l__driver_color_stack_int=\count162 +)) +Package: xparse 2014/11/25 v5471 L3 Experimental document command parser +\l__xparse_current_arg_int=\count163 +\l__xparse_m_args_int=\count164 +\l__xparse_mandatory_args_int=\count165 +\l__xparse_processor_int=\count166 +\l__xparse_v_nesting_int=\count167 +) +(/usr/local/texlive/2015/texmf-dist/tex/latex/oberdiek/zref-abspage.sty +Package: zref-abspage 2012/04/04 v2.24 Module abspage for zref (HO) + +(/usr/local/texlive/2015/texmf-dist/tex/latex/oberdiek/zref-base.sty +Package: zref-base 2012/04/04 v2.24 Module base for zref (HO) +Package zref Info: New property list: main on input line 759. +Package zref Info: New property: default on input line 760. +Package zref Info: New property: page on input line 761. +) +\c@abspage=\count168 +Package zref Info: New property: abspage on input line 62. +) +(/usr/local/texlive/2015/texmf-dist/tex/latex/needspace/needspace.sty +Package: needspace 2010/09/12 v1.3d reserve vertical space +) +\mdf@templength=\skip73 +\c@mdf@globalstyle@cnt=\count169 +\mdf@skipabove@length=\skip74 +\mdf@skipbelow@length=\skip75 +\mdf@leftmargin@length=\skip76 +\mdf@rightmargin@length=\skip77 +\mdf@innerleftmargin@length=\skip78 +\mdf@innerrightmargin@length=\skip79 +\mdf@innertopmargin@length=\skip80 +\mdf@innerbottommargin@length=\skip81 +\mdf@splittopskip@length=\skip82 +\mdf@splitbottomskip@length=\skip83 +\mdf@outermargin@length=\skip84 +\mdf@innermargin@length=\skip85 +\mdf@linewidth@length=\skip86 +\mdf@innerlinewidth@length=\skip87 +\mdf@middlelinewidth@length=\skip88 +\mdf@outerlinewidth@length=\skip89 +\mdf@roundcorner@length=\skip90 +\mdf@footenotedistance@length=\skip91 +\mdf@userdefinedwidth@length=\skip92 +\mdf@needspace@length=\skip93 +\mdf@frametitleaboveskip@length=\skip94 +\mdf@frametitlebelowskip@length=\skip95 +\mdf@frametitlerulewidth@length=\skip96 +\mdf@frametitleleftmargin@length=\skip97 +\mdf@frametitlerightmargin@length=\skip98 +\mdf@shadowsize@length=\skip99 +\mdf@extratopheight@length=\skip100 +\mdf@subtitleabovelinewidth@length=\skip101 +\mdf@subtitlebelowlinewidth@length=\skip102 +\mdf@subtitleaboveskip@length=\skip103 +\mdf@subtitlebelowskip@length=\skip104 +\mdf@subtitleinneraboveskip@length=\skip105 +\mdf@subtitleinnerbelowskip@length=\skip106 +\mdf@subsubtitleabovelinewidth@length=\skip107 +\mdf@subsubtitlebelowlinewidth@length=\skip108 +\mdf@subsubtitleaboveskip@length=\skip109 +\mdf@subsubtitlebelowskip@length=\skip110 +\mdf@subsubtitleinneraboveskip@length=\skip111 +\mdf@subsubtitleinnerbelowskip@length=\skip112 + +(/usr/local/texlive/2015/texmf-dist/tex/latex/pgf/frontendlayer/tikz.sty +(/usr/local/texlive/2015/texmf-dist/tex/latex/pgf/basiclayer/pgf.sty +(/usr/local/texlive/2015/texmf-dist/tex/latex/pgf/utilities/pgfrcs.sty +(/usr/local/texlive/2015/texmf-dist/tex/generic/pgf/utilities/pgfutil-common.te +x +\pgfutil@everybye=\toks34 +\pgfutil@tempdima=\dimen170 +\pgfutil@tempdimb=\dimen171 + +(/usr/local/texlive/2015/texmf-dist/tex/generic/pgf/utilities/pgfutil-common-li +sts.tex)) +(/usr/local/texlive/2015/texmf-dist/tex/generic/pgf/utilities/pgfutil-latex.def +\pgfutil@abb=\box54 +(/usr/local/texlive/2015/texmf-dist/tex/latex/ms/everyshi.sty +Package: everyshi 2001/05/15 v3.00 EveryShipout Package (MS) +)) +(/usr/local/texlive/2015/texmf-dist/tex/generic/pgf/utilities/pgfrcs.code.tex +Package: pgfrcs 2013/12/20 v3.0.0 (rcs-revision 1.28) +)) +Package: pgf 2013/12/18 v3.0.0 (rcs-revision 1.14) +(/usr/local/texlive/2015/texmf-dist/tex/latex/pgf/basiclayer/pgfcore.sty +(/usr/local/texlive/2015/texmf-dist/tex/latex/pgf/systemlayer/pgfsys.sty +(/usr/local/texlive/2015/texmf-dist/tex/generic/pgf/systemlayer/pgfsys.code.tex +Package: pgfsys 2013/11/30 v3.0.0 (rcs-revision 1.47) +(/usr/local/texlive/2015/texmf-dist/tex/generic/pgf/utilities/pgfkeys.code.tex +\pgfkeys@pathtoks=\toks35 +\pgfkeys@temptoks=\toks36 + +(/usr/local/texlive/2015/texmf-dist/tex/generic/pgf/utilities/pgfkeysfiltered.c +ode.tex +\pgfkeys@tmptoks=\toks37 +)) +\pgf@x=\dimen172 +\pgf@y=\dimen173 +\pgf@xa=\dimen174 +\pgf@ya=\dimen175 +\pgf@xb=\dimen176 +\pgf@yb=\dimen177 +\pgf@xc=\dimen178 +\pgf@yc=\dimen179 +\w@pgf@writea=\write6 +\r@pgf@reada=\read3 +\c@pgf@counta=\count170 +\c@pgf@countb=\count171 +\c@pgf@countc=\count172 +\c@pgf@countd=\count173 +\t@pgf@toka=\toks38 +\t@pgf@tokb=\toks39 +\t@pgf@tokc=\toks40 + +(/usr/local/texlive/2015/texmf-dist/tex/generic/pgf/systemlayer/pgf.cfg +File: pgf.cfg 2008/05/14 (rcs-revision 1.7) +) +Driver file for pgf: pgfsys-pdftex.def + +(/usr/local/texlive/2015/texmf-dist/tex/generic/pgf/systemlayer/pgfsys-pdftex.d +ef +File: pgfsys-pdftex.def 2013/07/18 (rcs-revision 1.33) + +(/usr/local/texlive/2015/texmf-dist/tex/generic/pgf/systemlayer/pgfsys-common-p +df.def +File: pgfsys-common-pdf.def 2013/10/10 (rcs-revision 1.13) +))) +(/usr/local/texlive/2015/texmf-dist/tex/generic/pgf/systemlayer/pgfsyssoftpath. +code.tex +File: pgfsyssoftpath.code.tex 2013/09/09 (rcs-revision 1.9) +\pgfsyssoftpath@smallbuffer@items=\count174 +\pgfsyssoftpath@bigbuffer@items=\count175 +) +(/usr/local/texlive/2015/texmf-dist/tex/generic/pgf/systemlayer/pgfsysprotocol. +code.tex +File: pgfsysprotocol.code.tex 2006/10/16 (rcs-revision 1.4) +)) +(/usr/local/texlive/2015/texmf-dist/tex/generic/pgf/basiclayer/pgfcore.code.tex +Package: pgfcore 2010/04/11 v3.0.0 (rcs-revision 1.7) +(/usr/local/texlive/2015/texmf-dist/tex/generic/pgf/math/pgfmath.code.tex +(/usr/local/texlive/2015/texmf-dist/tex/generic/pgf/math/pgfmathcalc.code.tex +(/usr/local/texlive/2015/texmf-dist/tex/generic/pgf/math/pgfmathutil.code.tex) +(/usr/local/texlive/2015/texmf-dist/tex/generic/pgf/math/pgfmathparser.code.tex +\pgfmath@dimen=\dimen180 +\pgfmath@count=\count176 +\pgfmath@box=\box55 +\pgfmath@toks=\toks41 +\pgfmath@stack@operand=\toks42 +\pgfmath@stack@operation=\toks43 +) +(/usr/local/texlive/2015/texmf-dist/tex/generic/pgf/math/pgfmathfunctions.code. +tex +(/usr/local/texlive/2015/texmf-dist/tex/generic/pgf/math/pgfmathfunctions.basic +.code.tex) +(/usr/local/texlive/2015/texmf-dist/tex/generic/pgf/math/pgfmathfunctions.trigo +nometric.code.tex) +(/usr/local/texlive/2015/texmf-dist/tex/generic/pgf/math/pgfmathfunctions.rando +m.code.tex) +(/usr/local/texlive/2015/texmf-dist/tex/generic/pgf/math/pgfmathfunctions.compa +rison.code.tex) +(/usr/local/texlive/2015/texmf-dist/tex/generic/pgf/math/pgfmathfunctions.base. +code.tex) +(/usr/local/texlive/2015/texmf-dist/tex/generic/pgf/math/pgfmathfunctions.round +.code.tex) +(/usr/local/texlive/2015/texmf-dist/tex/generic/pgf/math/pgfmathfunctions.misc. +code.tex) +(/usr/local/texlive/2015/texmf-dist/tex/generic/pgf/math/pgfmathfunctions.integ +erarithmetics.code.tex))) +(/usr/local/texlive/2015/texmf-dist/tex/generic/pgf/math/pgfmathfloat.code.tex +\c@pgfmathroundto@lastzeros=\count177 +)) +(/usr/local/texlive/2015/texmf-dist/tex/generic/pgf/basiclayer/pgfcorepoints.co +de.tex +File: pgfcorepoints.code.tex 2013/10/07 (rcs-revision 1.27) +\pgf@picminx=\dimen181 +\pgf@picmaxx=\dimen182 +\pgf@picminy=\dimen183 +\pgf@picmaxy=\dimen184 +\pgf@pathminx=\dimen185 +\pgf@pathmaxx=\dimen186 +\pgf@pathminy=\dimen187 +\pgf@pathmaxy=\dimen188 +\pgf@xx=\dimen189 +\pgf@xy=\dimen190 +\pgf@yx=\dimen191 +\pgf@yy=\dimen192 +\pgf@zx=\dimen193 +\pgf@zy=\dimen194 +) +(/usr/local/texlive/2015/texmf-dist/tex/generic/pgf/basiclayer/pgfcorepathconst +ruct.code.tex +File: pgfcorepathconstruct.code.tex 2013/10/07 (rcs-revision 1.29) +\pgf@path@lastx=\dimen195 +\pgf@path@lasty=\dimen196 +) +(/usr/local/texlive/2015/texmf-dist/tex/generic/pgf/basiclayer/pgfcorepathusage +.code.tex +File: pgfcorepathusage.code.tex 2013/12/13 (rcs-revision 1.23) +\pgf@shorten@end@additional=\dimen197 +\pgf@shorten@start@additional=\dimen198 +) +(/usr/local/texlive/2015/texmf-dist/tex/generic/pgf/basiclayer/pgfcorescopes.co +de.tex +File: pgfcorescopes.code.tex 2013/10/09 (rcs-revision 1.44) +\pgfpic=\box56 +\pgf@hbox=\box57 +\pgf@layerbox@main=\box58 +\pgf@picture@serial@count=\count178 +) +(/usr/local/texlive/2015/texmf-dist/tex/generic/pgf/basiclayer/pgfcoregraphicst +ate.code.tex +File: pgfcoregraphicstate.code.tex 2013/09/19 (rcs-revision 1.11) +\pgflinewidth=\dimen199 +) +(/usr/local/texlive/2015/texmf-dist/tex/generic/pgf/basiclayer/pgfcoretransform +ations.code.tex +File: pgfcoretransformations.code.tex 2013/10/10 (rcs-revision 1.17) +\pgf@pt@x=\dimen200 +\pgf@pt@y=\dimen201 +\pgf@pt@temp=\dimen202 +) +(/usr/local/texlive/2015/texmf-dist/tex/generic/pgf/basiclayer/pgfcorequick.cod +e.tex +File: pgfcorequick.code.tex 2008/10/09 (rcs-revision 1.3) +) +(/usr/local/texlive/2015/texmf-dist/tex/generic/pgf/basiclayer/pgfcoreobjects.c +ode.tex +File: pgfcoreobjects.code.tex 2006/10/11 (rcs-revision 1.2) +) +(/usr/local/texlive/2015/texmf-dist/tex/generic/pgf/basiclayer/pgfcorepathproce +ssing.code.tex +File: pgfcorepathprocessing.code.tex 2013/09/09 (rcs-revision 1.9) +) +(/usr/local/texlive/2015/texmf-dist/tex/generic/pgf/basiclayer/pgfcorearrows.co +de.tex +File: pgfcorearrows.code.tex 2013/11/07 (rcs-revision 1.40) +\pgfarrowsep=\dimen203 +) +(/usr/local/texlive/2015/texmf-dist/tex/generic/pgf/basiclayer/pgfcoreshade.cod +e.tex +File: pgfcoreshade.code.tex 2013/07/15 (rcs-revision 1.15) +\pgf@max=\dimen204 +\pgf@sys@shading@range@num=\count179 +) +(/usr/local/texlive/2015/texmf-dist/tex/generic/pgf/basiclayer/pgfcoreimage.cod +e.tex +File: pgfcoreimage.code.tex 2013/07/15 (rcs-revision 1.18) + +(/usr/local/texlive/2015/texmf-dist/tex/generic/pgf/basiclayer/pgfcoreexternal. +code.tex +File: pgfcoreexternal.code.tex 2013/07/15 (rcs-revision 1.20) +\pgfexternal@startupbox=\box59 +)) +(/usr/local/texlive/2015/texmf-dist/tex/generic/pgf/basiclayer/pgfcorelayers.co +de.tex +File: pgfcorelayers.code.tex 2013/07/18 (rcs-revision 1.7) +) +(/usr/local/texlive/2015/texmf-dist/tex/generic/pgf/basiclayer/pgfcoretranspare +ncy.code.tex +File: pgfcoretransparency.code.tex 2013/09/30 (rcs-revision 1.5) +) +(/usr/local/texlive/2015/texmf-dist/tex/generic/pgf/basiclayer/pgfcorepatterns. +code.tex +File: pgfcorepatterns.code.tex 2013/11/07 (rcs-revision 1.5) +))) +(/usr/local/texlive/2015/texmf-dist/tex/generic/pgf/modules/pgfmoduleshapes.cod +e.tex +File: pgfmoduleshapes.code.tex 2013/10/31 (rcs-revision 1.34) +\pgfnodeparttextbox=\box60 +) +(/usr/local/texlive/2015/texmf-dist/tex/generic/pgf/modules/pgfmoduleplot.code. +tex +File: pgfmoduleplot.code.tex 2013/07/31 (rcs-revision 1.12) +) +(/usr/local/texlive/2015/texmf-dist/tex/latex/pgf/compatibility/pgfcomp-version +-0-65.sty +Package: pgfcomp-version-0-65 2007/07/03 v3.0.0 (rcs-revision 1.7) +\pgf@nodesepstart=\dimen205 +\pgf@nodesepend=\dimen206 +) +(/usr/local/texlive/2015/texmf-dist/tex/latex/pgf/compatibility/pgfcomp-version +-1-18.sty +Package: pgfcomp-version-1-18 2007/07/23 v3.0.0 (rcs-revision 1.1) +)) +(/usr/local/texlive/2015/texmf-dist/tex/latex/pgf/utilities/pgffor.sty +(/usr/local/texlive/2015/texmf-dist/tex/latex/pgf/utilities/pgfkeys.sty +(/usr/local/texlive/2015/texmf-dist/tex/generic/pgf/utilities/pgfkeys.code.tex) +) (/usr/local/texlive/2015/texmf-dist/tex/latex/pgf/math/pgfmath.sty +(/usr/local/texlive/2015/texmf-dist/tex/generic/pgf/math/pgfmath.code.tex)) +(/usr/local/texlive/2015/texmf-dist/tex/generic/pgf/utilities/pgffor.code.tex +Package: pgffor 2013/12/13 v3.0.0 (rcs-revision 1.25) + +(/usr/local/texlive/2015/texmf-dist/tex/generic/pgf/math/pgfmath.code.tex) +\pgffor@iter=\dimen207 +\pgffor@skip=\dimen208 +\pgffor@stack=\toks44 +\pgffor@toks=\toks45 +)) +(/usr/local/texlive/2015/texmf-dist/tex/generic/pgf/frontendlayer/tikz/tikz.cod +e.tex +Package: tikz 2013/12/13 v3.0.0 (rcs-revision 1.142) + +(/usr/local/texlive/2015/texmf-dist/tex/generic/pgf/libraries/pgflibraryplothan +dlers.code.tex +File: pgflibraryplothandlers.code.tex 2013/08/31 v3.0.0 (rcs-revision 1.20) +\pgf@plot@mark@count=\count180 +\pgfplotmarksize=\dimen209 +) +\tikz@lastx=\dimen210 +\tikz@lasty=\dimen211 +\tikz@lastxsaved=\dimen212 +\tikz@lastysaved=\dimen213 +\tikzleveldistance=\dimen214 +\tikzsiblingdistance=\dimen215 +\tikz@figbox=\box61 +\tikz@figbox@bg=\box62 +\tikz@tempbox=\box63 +\tikz@tempbox@bg=\box64 +\tikztreelevel=\count181 +\tikznumberofchildren=\count182 +\tikznumberofcurrentchild=\count183 +\tikz@fig@count=\count184 + +(/usr/local/texlive/2015/texmf-dist/tex/generic/pgf/modules/pgfmodulematrix.cod +e.tex +File: pgfmodulematrix.code.tex 2013/09/17 (rcs-revision 1.8) +\pgfmatrixcurrentrow=\count185 +\pgfmatrixcurrentcolumn=\count186 +\pgf@matrix@numberofcolumns=\count187 +) +\tikz@expandcount=\count188 + +(/usr/local/texlive/2015/texmf-dist/tex/generic/pgf/frontendlayer/tikz/librarie +s/tikzlibrarytopaths.code.tex +File: tikzlibrarytopaths.code.tex 2008/06/17 v3.0.0 (rcs-revision 1.2) +))) +(/usr/local/texlive/2015/texmf-dist/tex/latex/mdframed/md-frame-1.mdf +File: md-frame-1.mdf 2013/07/01\ 1.9b: md-frame-1 +) +\mdf@frametitlebox=\box65 +\mdf@footnotebox=\box66 +\mdf@splitbox@one=\box67 +\mdf@splitbox@two=\box68 +\mdf@splitbox@save=\box69 +\mdfsplitboxwidth=\skip113 +\mdfsplitboxtotalwidth=\skip114 +\mdfsplitboxheight=\skip115 +\mdfsplitboxdepth=\skip116 +\mdfsplitboxtotalheight=\skip117 +\mdfframetitleboxwidth=\skip118 +\mdfframetitleboxtotalwidth=\skip119 +\mdfframetitleboxheight=\skip120 +\mdfframetitleboxdepth=\skip121 +\mdfframetitleboxtotalheight=\skip122 +\mdffootnoteboxwidth=\skip123 +\mdffootnoteboxtotalwidth=\skip124 +\mdffootnoteboxheight=\skip125 +\mdffootnoteboxdepth=\skip126 +\mdffootnoteboxtotalheight=\skip127 +\mdftotallinewidth=\skip128 +\mdfboundingboxwidth=\skip129 +\mdfboundingboxtotalwidth=\skip130 +\mdfboundingboxheight=\skip131 +\mdfboundingboxdepth=\skip132 +\mdfboundingboxtotalheight=\skip133 +\mdf@freevspace@length=\skip134 +\mdf@horizontalwidthofbox@length=\skip135 +\mdf@verticalmarginwhole@length=\skip136 +\mdf@horizontalspaceofbox=\skip137 +\mdfsubtitleheight=\skip138 +\mdfsubsubtitleheight=\skip139 +\c@mdfcountframes=\count189 + +****** mdframed patching \endmdf@trivlist + +****** -- success****** + +................................................. +. LaTeX info: "xparse/define-command" +. +. Defining command \newmdtheoremenv with sig. 'O{} m o m o ' on line 601. +................................................. +................................................. +. LaTeX info: "xparse/define-command" +. +. Defining command \mdtheorem with sig. ' O{} m o m o ' on line 701. +................................................. +\mdf@envdepth=\count190 +\c@mdf@env@i=\count191 +\c@mdf@env@ii=\count192 +\c@mdf@zref@counter=\count193 +Package zref Info: New property: mdf@pagevalue on input line 895. +) +\@indexfile=\write7 +\openout7 = `Regression.idx'. + + +Writing index file Regression.idx +(/usr/local/texlive/2015/texmf-dist/tex/latex/idxlayout/idxlayout.sty +Package: idxlayout 2012/03/30 v0.4d Configurable index layout + +(/usr/local/texlive/2015/texmf-dist/tex/latex/tools/multicol.sty +Package: multicol 2015/03/31 v1.8m multicolumn formatting (FMi) +\c@tracingmulticols=\count194 +\mult@box=\box70 +\multicol@leftmargin=\dimen216 +\c@unbalance=\count195 +\c@collectmore=\count196 +\doublecol@number=\count197 +\multicoltolerance=\count198 +\multicolpretolerance=\count199 +\full@width=\dimen217 +\page@free=\dimen218 +\premulticols=\dimen219 +\postmulticols=\dimen220 +\multicolsep=\skip140 +\multicolbaselineskip=\skip141 +\partial@page=\box71 +\last@line=\box72 +\maxbalancingoverflow=\dimen221 +\mult@rightbox=\box73 +\mult@grightbox=\box74 +\mult@gfirstbox=\box75 +\mult@firstbox=\box76 +\@tempa=\box77 +\@tempa=\box78 +\@tempa=\box79 +\@tempa=\box80 +\@tempa=\box81 +\@tempa=\box82 +\@tempa=\box83 +\@tempa=\box84 +\@tempa=\box85 +\@tempa=\box86 +\@tempa=\box87 +\@tempa=\box88 +\@tempa=\box89 +\@tempa=\box90 +\@tempa=\box91 +\@tempa=\box92 +\@tempa=\box93 +\c@columnbadness=\count200 +\c@finalcolumnbadness=\count201 +\last@try=\dimen222 +\multicolovershoot=\dimen223 +\multicolundershoot=\dimen224 +\mult@nat@firstbox=\box94 +\colbreak@box=\box95 +\mc@col@check@num=\count202 +) +\c@idxcols=\count203 +\indexcolsep=\skip142 +\indexrule=\skip143 +\ila@indentunit=\skip144 +\ila@initsep=\skip145 +) +(/usr/local/texlive/2015/texmf-dist/tex/latex/tocbibind/tocbibind.sty +Package: tocbibind 2010/10/13 v1.5k extra ToC listings +Package tocbibind Info: The document has section divisions on input line 50. + + +Package tocbibind Note: Using section or other style headings. + +) (/usr/local/texlive/2015/texmf-dist/tex/latex/ms/ragged2e.sty +Package: ragged2e 2009/05/21 v2.1 ragged2e Package (MS) + +(/usr/local/texlive/2015/texmf-dist/tex/latex/ms/everysel.sty +Package: everysel 2011/10/28 v1.2 EverySelectfont Package (MS) +) +\CenteringLeftskip=\skip146 +\RaggedLeftLeftskip=\skip147 +\RaggedRightLeftskip=\skip148 +\CenteringRightskip=\skip149 +\RaggedLeftRightskip=\skip150 +\RaggedRightRightskip=\skip151 +\CenteringParfillskip=\skip152 +\RaggedLeftParfillskip=\skip153 +\RaggedRightParfillskip=\skip154 +\JustifyingParfillskip=\skip155 +\CenteringParindent=\skip156 +\RaggedLeftParindent=\skip157 +\RaggedRightParindent=\skip158 +\JustifyingParindent=\skip159 +) +(./Regression.aux) +\openout1 = `Regression.aux'. + +LaTeX Font Info: Checking defaults for OML/cmm/m/it on input line 98. +LaTeX Font Info: ... okay on input line 98. +LaTeX Font Info: Checking defaults for T1/cmr/m/n on input line 98. +LaTeX Font Info: ... okay on input line 98. +LaTeX Font Info: Checking defaults for OT1/cmr/m/n on input line 98. +LaTeX Font Info: ... okay on input line 98. +LaTeX Font Info: Checking defaults for OMS/cmsy/m/n on input line 98. +LaTeX Font Info: ... okay on input line 98. +LaTeX Font Info: Checking defaults for OMX/cmex/m/n on input line 98. +LaTeX Font Info: ... okay on input line 98. +LaTeX Font Info: Checking defaults for U/cmr/m/n on input line 98. +LaTeX Font Info: ... okay on input line 98. +LaTeX Font Info: Checking defaults for PD1/pdf/m/n on input line 98. +LaTeX Font Info: ... okay on input line 98. +LaTeX Font Info: Try loading font information for T1+lmr on input line 98. + (/usr/local/texlive/2015/texmf-dist/tex/latex/lm/t1lmr.fd +File: t1lmr.fd 2009/10/30 v1.6 Font defs for Latin Modern +) +(/usr/local/texlive/2015/texmf-dist/tex/context/base/supp-pdf.mkii +[Loading MPS to PDF converter (version 2006.09.02).] +\scratchcounter=\count204 +\scratchdimen=\dimen225 +\scratchbox=\box96 +\nofMPsegments=\count205 +\nofMParguments=\count206 +\everyMPshowfont=\toks46 +\MPscratchCnt=\count207 +\MPscratchDim=\dimen226 +\MPnumerator=\count208 +\makeMPintoPDFobject=\count209 +\everyMPtoPDFconversion=\toks47 +) +LaTeX Info: Redefining \microtypecontext on input line 98. +Package microtype Info: Generating PDF output. +Package microtype Info: Character protrusion enabled (level 2). +Package microtype Info: Using default protrusion set `alltext'. +Package microtype Info: Automatic font expansion enabled (level 2), +(microtype) stretch: 20, shrink: 20, step: 1, non-selected. +Package microtype Info: Using default expansion set `basictext'. +Package microtype Info: No adjustment of tracking. +Package microtype Info: No adjustment of interword spacing. +Package microtype Info: No adjustment of character kerning. + (/usr/local/texlive/2015/texmf-dist/tex/latex/microtype/mt-cmr.cfg +File: mt-cmr.cfg 2013/05/19 v2.2 microtype config. file: Computer Modern Roman +(RS) +) +(/usr/local/texlive/2015/texmf-dist/tex/latex/oberdiek/epstopdf-base.sty +Package: epstopdf-base 2010/02/09 v2.5 Base part for package epstopdf + +(/usr/local/texlive/2015/texmf-dist/tex/latex/oberdiek/grfext.sty +Package: grfext 2010/08/19 v1.1 Manage graphics extensions (HO) +) +Package grfext Info: Graphics extension search list: +(grfext) [.png,.pdf,.jpg,.mps,.jpeg,.jbig2,.jb2,.PNG,.PDF,.JPG,.JPE +G,.JBIG2,.JB2,.eps] +(grfext) \AppendGraphicsExtensions on input line 452. + +(/usr/local/texlive/2015/texmf-dist/tex/latex/latexconfig/epstopdf-sys.cfg +File: epstopdf-sys.cfg 2010/07/13 v1.3 Configuration of (r)epstopdf for TeX Liv +e +)) +(/usr/local/texlive/2015/texmf-dist/tex/latex/ucs/ucsencs.def +File: ucsencs.def 2011/01/21 Fixes to fontencodings LGR, T3 +) +\AtBeginShipoutBox=\box97 +Package hyperref Info: Link coloring ON on input line 98. + +(/usr/local/texlive/2015/texmf-dist/tex/latex/hyperref/nameref.sty +Package: nameref 2012/10/27 v2.43 Cross-referencing by name of section + +(/usr/local/texlive/2015/texmf-dist/tex/generic/oberdiek/gettitlestring.sty +Package: gettitlestring 2010/12/03 v1.4 Cleanup title references (HO) +) +\c@section@level=\count210 +) +LaTeX Info: Redefining \ref on input line 98. +LaTeX Info: Redefining \pageref on input line 98. +LaTeX Info: Redefining \nameref on input line 98. + +(./Regression.out) (./Regression.out) +\@outlinefile=\write8 +\openout8 = `Regression.out'. + + ABD: EveryShipout initializing macros +ABD: EverySelectfont initializing macros +LaTeX Info: Redefining \selectfont on input line 98. +LaTeX Font Info: Try loading font information for OT1+lmr on input line 124. + + +(/usr/local/texlive/2015/texmf-dist/tex/latex/lm/ot1lmr.fd +File: ot1lmr.fd 2009/10/30 v1.6 Font defs for Latin Modern +) +LaTeX Font Info: Try loading font information for OML+lmm on input line 124. + + +(/usr/local/texlive/2015/texmf-dist/tex/latex/lm/omllmm.fd +File: omllmm.fd 2009/10/30 v1.6 Font defs for Latin Modern +) +LaTeX Font Info: Try loading font information for OMS+lmsy on input line 124 +. + +(/usr/local/texlive/2015/texmf-dist/tex/latex/lm/omslmsy.fd +File: omslmsy.fd 2009/10/30 v1.6 Font defs for Latin Modern +) +LaTeX Font Info: Try loading font information for OMX+lmex on input line 124 +. + +(/usr/local/texlive/2015/texmf-dist/tex/latex/lm/omxlmex.fd +File: omxlmex.fd 2009/10/30 v1.6 Font defs for Latin Modern +) +LaTeX Font Info: External font `lmex10' loaded for size +(Font) <10> on input line 124. +LaTeX Font Info: External font `lmex10' loaded for size +(Font) <7> on input line 124. +LaTeX Font Info: External font `lmex10' loaded for size +(Font) <5> on input line 124. +LaTeX Font Info: Try loading font information for U+msa on input line 124. + +(/usr/local/texlive/2015/texmf-dist/tex/latex/amsfonts/umsa.fd +File: umsa.fd 2013/01/14 v3.01 AMS symbols A +) +(/usr/local/texlive/2015/texmf-dist/tex/latex/microtype/mt-msa.cfg +File: mt-msa.cfg 2006/02/04 v1.1 microtype config. file: AMS symbols (a) (RS) +) +LaTeX Font Info: Try loading font information for U+msb on input line 124. + +(/usr/local/texlive/2015/texmf-dist/tex/latex/amsfonts/umsb.fd +File: umsb.fd 2013/01/14 v3.01 AMS symbols B +) +(/usr/local/texlive/2015/texmf-dist/tex/latex/microtype/mt-msb.cfg +File: mt-msb.cfg 2005/06/01 v1.0 microtype config. file: AMS symbols (b) (RS) +) +LaTeX Font Info: External font `lmex10' loaded for size +(Font) <9> on input line 129. +LaTeX Font Info: External font `lmex10' loaded for size +(Font) <6> on input line 129. +LaTeX Font Info: Try loading font information for OMS+lmr on input line 149. + + +(/usr/local/texlive/2015/texmf-dist/tex/latex/lm/omslmr.fd +File: omslmr.fd 2009/10/30 v1.6 Font defs for Latin Modern +) +LaTeX Font Info: Font shape `OMS/lmr/m/n' in size <10> not available +(Font) Font shape `OMS/lmsy/m/n' tried instead on input line 149. +LaTeX Font Info: Font shape `OMS/lmr/m/n' in size <8> not available +(Font) Font shape `OMS/lmsy/m/n' tried instead on input line 181. + [1 + +{/usr/local/texlive/2015/texmf-var/fonts/map/pdftex/updmap/pdftex.map}] +Overfull \hbox (37.20496pt too wide) in paragraph at lines 188--189 +[]$\OML/lmm/m/it/10 p$ \T1/lmr/m/n/10 (-20) so-called ex-plana-tory (in-de-pen- +dent or pre-dic-tor) vari-ables $\OML/cmm/b/it/10 x[] \OT1/lmr/m/n/10 (-20) = [ +\OML/lmm/m/it/10 x[]; x[]; [] ; x[]\OT1/lmr/m/n/10 (-20) ]$ + [] + +[2] [3] [4] +\openout3 = `Regression.pyg'. + +runsystem(pygmentize -l python -f latex -F tokenmerge -P style=default -P comma +ndprefix=PYGdefault -P style=default -P commandprefix=PYGdefault -P mathescape= +True -o "_minted-Regression/51639A41B11E5FE7D4CD027BA48FB6C64B967B05670D715D94F +4392BD3176A49.pygtex" "Regression.pyg" )...executed. + + + +! Package minted Error: Missing Pygments output; \inputminted was +probably given a file that does not exist--otherwise, you may need +the outputdir package option, or may be using an incompatible build tool. + +See the minted package documentation for explanation. +Type H for immediate help. + ... + +l.481 \end{minted} + +? s +OK, entering \scrollmode... +[5] [6] +\openout3 = `Regression.pyg'. + +runsystem(pygmentize -l python -f latex -F tokenmerge -P style=default -P comma +ndprefix=PYGdefault -P style=default -P commandprefix=PYGdefault -P mathescape= +True -o "_minted-Regression/4C68C697EB414B135747B9ECA18C3B5B4B967B05670D715D94F +4392BD3176A49.pygtex" "Regression.pyg" )...executed. + + + +! Package minted Error: Missing Pygments output; \inputminted was +probably given a file that does not exist--otherwise, you may need +the outputdir package option, or may be using an incompatible build tool. + +See the minted package documentation for explanation. +Type H for immediate help. + ... + +l.637 \end{minted} + +This could be caused by using -output-directory or -aux-directory +without setting minted's outputdir, or by using a build tool that +changes paths in ways minted cannot detect. + +\openout3 = `Regression.pyg'. + +runsystem(pygmentize -l python -f latex -F tokenmerge -P style=default -P comma +ndprefix=PYGdefault -P style=default -P commandprefix=PYGdefault -P mathescape= +True -o "_minted-Regression/439DCAB036A38F6389948C8DF1465C424B967B05670D715D94F +4392BD3176A49.pygtex" "Regression.pyg" )...executed. + + +! Package minted Error: Missing Pygments output; \inputminted was +probably given a file that does not exist--otherwise, you may need +the outputdir package option, or may be using an incompatible build tool. + +See the minted package documentation for explanation. +Type H for immediate help. + ... + +l.642 \end{minted} + +This could be caused by using -output-directory or -aux-directory +without setting minted's outputdir, or by using a build tool that +changes paths in ways minted cannot detect. + +[7] +\openout3 = `Regression.pyg'. + +runsystem(pygmentize -l python -f latex -F tokenmerge -P style=default -P comma +ndprefix=PYGdefault -P style=default -P commandprefix=PYGdefault -P mathescape= +True -o "_minted-Regression/0A3C1B39ACFAF4BAB627404CB6156A144B967B05670D715D94F +4392BD3176A49.pygtex" "Regression.pyg" )...executed. + + + +! Package minted Error: Missing Pygments output; \inputminted was +probably given a file that does not exist--otherwise, you may need +the outputdir package option, or may be using an incompatible build tool. + +See the minted package documentation for explanation. +Type H for immediate help. + ... + +l.658 \end{minted} + +This could be caused by using -output-directory or -aux-directory +without setting minted's outputdir, or by using a build tool that +changes paths in ways minted cannot detect. + +! Missing $ inserted. + + $ +l.663 ...in connection with the functionality of _ + Scikit_Learn_ in the intro... +I've inserted a begin-math/end-math symbol since I think +you left one out. Proceed, with fingers crossed. + +\openout3 = `Regression.pyg'. + +runsystem(pygmentize -l python -f latex -F tokenmerge -P style=default -P comma +ndprefix=PYGdefault -P style=default -P commandprefix=PYGdefault -P mathescape= +True -o "_minted-Regression/F5688422A51FDCF3C06A3FF7EF8F9E6E4B967B05670D715D94F +4392BD3176A49.pygtex" "Regression.pyg" )...executed. + + +! Package minted Error: Missing Pygments output; \inputminted was +probably given a file that does not exist--otherwise, you may need +the outputdir package option, or may be using an incompatible build tool. + +See the minted package documentation for explanation. +Type H for immediate help. + ... + +l.668 \end{minted} + +This could be caused by using -output-directory or -aux-directory +without setting minted's outputdir, or by using a build tool that +changes paths in ways minted cannot detect. + +\openout3 = `Regression.pyg'. + +runsystem(pygmentize -l python -f latex -F tokenmerge -P style=default -P comma +ndprefix=PYGdefault -P style=default -P commandprefix=PYGdefault -P mathescape= +True -o "_minted-Regression/FE87C84792E25A9D98329B989214E79F4B967B05670D715D94F +4392BD3176A49.pygtex" "Regression.pyg" )...executed. + + +! Package minted Error: Missing Pygments output; \inputminted was +probably given a file that does not exist--otherwise, you may need +the outputdir package option, or may be using an incompatible build tool. + +See the minted package documentation for explanation. +Type H for immediate help. + ... + +l.672 \end{minted} + +This could be caused by using -output-directory or -aux-directory +without setting minted's outputdir, or by using a build tool that +changes paths in ways minted cannot detect. + +! Missing $ inserted. + + $ +l.673 + +I've inserted a begin-math/end-math symbol since I think +you left one out. Proceed, with fingers crossed. + + +Overfull \hbox (64.00754pt too wide) in paragraph at lines 663--673 +\T1/lmr/m/n/10 (-20) tion with the func-tion-al-ity of $[]\OML/lmm/m/it/10 ciki +t[]earn[]ntheintroductoryslides:Sincewearenotusing[]cikit \OMS/lmsy/m/n/10 ^^@ + [] + +\openout3 = `Regression.pyg'. + +runsystem(pygmentize -l python -f latex -F tokenmerge -P style=default -P comma +ndprefix=PYGdefault -P style=default -P commandprefix=PYGdefault -P mathescape= +True -o "_minted-Regression/F9FF5FEA9F501D7189D0F820FC8D1D3D4B967B05670D715D94F +4392BD3176A49.pygtex" "Regression.pyg" )...executed. + + +! Package minted Error: Missing Pygments output; \inputminted was +probably given a file that does not exist--otherwise, you may need +the outputdir package option, or may be using an incompatible build tool. + +See the minted package documentation for explanation. +Type H for immediate help. + ... + +l.681 \end{minted} + +This could be caused by using -output-directory or -aux-directory +without setting minted's outputdir, or by using a build tool that +changes paths in ways minted cannot detect. + +\openout3 = `Regression.pyg'. + +runsystem(pygmentize -l python -f latex -F tokenmerge -P style=default -P comma +ndprefix=PYGdefault -P style=default -P commandprefix=PYGdefault -P mathescape= +True -o "_minted-Regression/2702C77D52D3F655EDD2BB83D8F1240B4B967B05670D715D94F +4392BD3176A49.pygtex" "Regression.pyg" )...executed. + + +! Package minted Error: Missing Pygments output; \inputminted was +probably given a file that does not exist--otherwise, you may need +the outputdir package option, or may be using an incompatible build tool. + +See the minted package documentation for explanation. +Type H for immediate help. + ... + +l.687 \end{minted} + +This could be caused by using -output-directory or -aux-directory +without setting minted's outputdir, or by using a build tool that +changes paths in ways minted cannot detect. + +LaTeX Font Info: External font `lmex10' loaded for size +(Font) <12> on input line 694. +LaTeX Font Info: External font `lmex10' loaded for size +(Font) <8> on input line 694. + +Overfull \hbox (8.74574pt too wide) detected at line 728 +[] \OT1/lmr/m/n/10 = [] [] = 0\OML/lmm/m/it/10 ; + [] + + +Overfull \hbox (9.06128pt too wide) detected at line 732 +[] \OT1/lmr/m/n/10 = \OMS/lmsy/m/n/10 ^^@[] [] \OT1/lmr/m/n/10 = 0\OML/lmm/m/it +/10 ; + [] + +[8] [9] +\openout3 = `Regression.pyg'. + +runsystem(pygmentize -l python -f latex -F tokenmerge -P style=default -P comma +ndprefix=PYGdefault -P style=default -P commandprefix=PYGdefault -P mathescape= +True -o "_minted-Regression/E81889C7775BD173862927052713983D4B967B05670D715D94F +4392BD3176A49.pygtex" "Regression.pyg" )...executed. + + + +! Package minted Error: Missing Pygments output; \inputminted was +probably given a file that does not exist--otherwise, you may need +the outputdir package option, or may be using an incompatible build tool. + +See the minted package documentation for explanation. +Type H for immediate help. + ... + +l.968 \end{minted} + +This could be caused by using -output-directory or -aux-directory +without setting minted's outputdir, or by using a build tool that +changes paths in ways minted cannot detect. + +[10] +\openout3 = `Regression.pyg'. + +runsystem(pygmentize -l python -f latex -F tokenmerge -P style=default -P comma +ndprefix=PYGdefault -P style=default -P commandprefix=PYGdefault -P mathescape= +True -o "_minted-Regression/31D43D24D5F9D55728C4E563E0E557924B967B05670D715D94F +4392BD3176A49.pygtex" "Regression.pyg" )...executed. + + + +! Package minted Error: Missing Pygments output; \inputminted was +probably given a file that does not exist--otherwise, you may need +the outputdir package option, or may be using an incompatible build tool. + +See the minted package documentation for explanation. +Type H for immediate help. + ... + +l.1057 \end{minted} + +This could be caused by using -output-directory or -aux-directory +without setting minted's outputdir, or by using a build tool that +changes paths in ways minted cannot detect. + +\openout3 = `Regression.pyg'. + +runsystem(pygmentize -l python -f latex -F tokenmerge -P style=default -P comma +ndprefix=PYGdefault -P style=default -P commandprefix=PYGdefault -P mathescape= +True -o "_minted-Regression/F6068C18D7EFD701D08693F8BD52B4664B967B05670D715D94F +4392BD3176A49.pygtex" "Regression.pyg" )...executed. + + +! Package minted Error: Missing Pygments output; \inputminted was +probably given a file that does not exist--otherwise, you may need +the outputdir package option, or may be using an incompatible build tool. + +See the minted package documentation for explanation. +Type H for immediate help. + ... + +l.1125 \end{minted} + +This could be caused by using -output-directory or -aux-directory +without setting minted's outputdir, or by using a build tool that +changes paths in ways minted cannot detect. + +[11] +\openout3 = `Regression.pyg'. + +runsystem(pygmentize -l python -f latex -F tokenmerge -P style=default -P comma +ndprefix=PYGdefault -P style=default -P commandprefix=PYGdefault -P mathescape= +True -o "_minted-Regression/9CC2E80F086A0B9D080182D504A931C14B967B05670D715D94F +4392BD3176A49.pygtex" "Regression.pyg" )...executed. + + + +! Package minted Error: Missing Pygments output; \inputminted was +probably given a file that does not exist--otherwise, you may need +the outputdir package option, or may be using an incompatible build tool. + +See the minted package documentation for explanation. +Type H for immediate help. + ... + +l.1165 \end{minted} + +This could be caused by using -output-directory or -aux-directory +without setting minted's outputdir, or by using a build tool that +changes paths in ways minted cannot detect. + +\openout3 = `Regression.pyg'. + +runsystem(pygmentize -l python -f latex -F tokenmerge -P style=default -P comma +ndprefix=PYGdefault -P style=default -P commandprefix=PYGdefault -P mathescape= +True -o "_minted-Regression/4C6F7D24872AF7644FF40FC87BFFAB2B4B967B05670D715D94F +4392BD3176A49.pygtex" "Regression.pyg" )...executed. + + +! Package minted Error: Missing Pygments output; \inputminted was +probably given a file that does not exist--otherwise, you may need +the outputdir package option, or may be using an incompatible build tool. + +See the minted package documentation for explanation. +Type H for immediate help. + ... + +l.1197 \end{minted} + +This could be caused by using -output-directory or -aux-directory +without setting minted's outputdir, or by using a build tool that +changes paths in ways minted cannot detect. + +\openout3 = `Regression.pyg'. + +runsystem(pygmentize -l python -f latex -F tokenmerge -P style=default -P comma +ndprefix=PYGdefault -P style=default -P commandprefix=PYGdefault -P mathescape= +True -o "_minted-Regression/C8DBC01C383B0800E51E0923C05691E04B967B05670D715D94F +4392BD3176A49.pygtex" "Regression.pyg" )...executed. + + +! Package minted Error: Missing Pygments output; \inputminted was +probably given a file that does not exist--otherwise, you may need +the outputdir package option, or may be using an incompatible build tool. + +See the minted package documentation for explanation. +Type H for immediate help. + ... + +l.1203 \end{minted} + +This could be caused by using -output-directory or -aux-directory +without setting minted's outputdir, or by using a build tool that +changes paths in ways minted cannot detect. + +[12] +\openout3 = `Regression.pyg'. + +runsystem(pygmentize -l python -f latex -F tokenmerge -P style=default -P comma +ndprefix=PYGdefault -P style=default -P commandprefix=PYGdefault -P mathescape= +True -o "_minted-Regression/E06035EDB37DBFD5697CD9DD1C5CA4A34B967B05670D715D94F +4392BD3176A49.pygtex" "Regression.pyg" )...executed. + + + +! Package minted Error: Missing Pygments output; \inputminted was +probably given a file that does not exist--otherwise, you may need +the outputdir package option, or may be using an incompatible build tool. + +See the minted package documentation for explanation. +Type H for immediate help. + ... + +l.1244 \end{minted} + +This could be caused by using -output-directory or -aux-directory +without setting minted's outputdir, or by using a build tool that +changes paths in ways minted cannot detect. + +\openout3 = `Regression.pyg'. + +runsystem(pygmentize -l python -f latex -F tokenmerge -P style=default -P comma +ndprefix=PYGdefault -P style=default -P commandprefix=PYGdefault -P mathescape= +True -o "_minted-Regression/363FA9AE05CC4E390538E08524572CF34B967B05670D715D94F +4392BD3176A49.pygtex" "Regression.pyg" )...executed. + + +! Package minted Error: Missing Pygments output; \inputminted was +probably given a file that does not exist--otherwise, you may need +the outputdir package option, or may be using an incompatible build tool. + +See the minted package documentation for explanation. +Type H for immediate help. + ... + +l.1248 \end{minted} + +This could be caused by using -output-directory or -aux-directory +without setting minted's outputdir, or by using a build tool that +changes paths in ways minted cannot detect. + +\openout3 = `Regression.pyg'. + +runsystem(pygmentize -l python -f latex -F tokenmerge -P style=default -P comma +ndprefix=PYGdefault -P style=default -P commandprefix=PYGdefault -P mathescape= +True -o "_minted-Regression/367BFC34E39C7EA712AF1384CAA96F674B967B05670D715D94F +4392BD3176A49.pygtex" "Regression.pyg" )...executed. + + +! Package minted Error: Missing Pygments output; \inputminted was +probably given a file that does not exist--otherwise, you may need +the outputdir package option, or may be using an incompatible build tool. + +See the minted package documentation for explanation. +Type H for immediate help. + ... + +l.1254 \end{minted} + +This could be caused by using -output-directory or -aux-directory +without setting minted's outputdir, or by using a build tool that +changes paths in ways minted cannot detect. + +\openout3 = `Regression.pyg'. + +runsystem(pygmentize -l python -f latex -F tokenmerge -P style=default -P comma +ndprefix=PYGdefault -P style=default -P commandprefix=PYGdefault -P mathescape= +True -o "_minted-Regression/09054641A8F49E0FADA410EE6C9BC1BA4B967B05670D715D94F +4392BD3176A49.pygtex" "Regression.pyg" )...executed. + + +! Package minted Error: Missing Pygments output; \inputminted was +probably given a file that does not exist--otherwise, you may need +the outputdir package option, or may be using an incompatible build tool. + +See the minted package documentation for explanation. +Type H for immediate help. + ... + +l.1268 \end{minted} + +This could be caused by using -output-directory or -aux-directory +without setting minted's outputdir, or by using a build tool that +changes paths in ways minted cannot detect. + +[13] + +! LaTeX Error: Bad math environment delimiter. + +See the LaTeX manual or LaTeX Companion for explanation. +Type H for immediate help. + ... + +l.1353 \[ + +Your command was ignored. +Type I to replace it with another command, +or to continue without it. + +! Missing \endgroup inserted. + + \endgroup +l.1355 \] + +I've inserted something that you may have forgotten. +(See the above.) +With luck, this will get me unwedged. But if you +really didn't forget anything, try typing `2' now; then +my insertion and my current dilemma will both disappear. + +! Missing \endgroup inserted. + + \endgroup +l.1355 \] + +I've inserted something that you may have forgotten. +(See the above.) +With luck, this will get me unwedged. But if you +really didn't forget anything, try typing `2' now; then +my insertion and my current dilemma will both disappear. + + +Overfull \hbox (63.72496pt too wide) detected at line 1355 +\OT1/lmr/m/n/10 (\OML/lmm/m/it/10 ^^U[]; \OML/cmm/b/it/10 u[]\OT1/lmr/m/n/10 )\ +OML/lmm/m/it/10 ; [] ; \OT1/lmr/m/n/10 (\OML/lmm/m/it/10 ^^U[]; \OML/cmm/b/it/1 +0 u[]\OT1/lmr/m/n/10 )\OML/lmm/m/it/10 ; andtheeigenvaluesaregivenbythediagonal +matrix\OT1/cmr/bx/n/10 ^^F \OT1/lmr/m/n/10 = [](\OML/lmm/m/it/10 ^^U[]; [] ; ^^ +U[]\OT1/lmr/m/n/10 )\OML/lmm/m/it/10 : + [] + +[14] + +! LaTeX Error: \begin{bmatrix} on input line 1407 ended by \end{matrix}. + +See the LaTeX manual or LaTeX Companion for explanation. +Type H for immediate help. + ... + +l.1407 ...x} 14 & 2\\ 4 & 22\\ 16 & 13\end{matrix} + =\frac{1}{3}\begin{bmatrix... + +Your command was ignored. +Type I to replace it with another command, +or to continue without it. + +! Missing \right. inserted. + + \right . +l.1407 ...x} 14 & 2\\ 4 & 22\\ 16 & 13\end{matrix} + =\frac{1}{3}\begin{bmatrix... +I've inserted something that you may have forgotten. +(See the above.) +With luck, this will get me unwedged. But if you +really didn't forget anything, try typing `2' now; then +my insertion and my current dilemma will both disappear. + +[15] [16] [17] [18] [19] +(/usr/local/texlive/2015/texmf-dist/tex/latex/ucs/data/uni-32.def +File: uni-32.def 2013/05/13 UCS: Unicode data U+2000..U+20FF +) [20] +[21] +Overfull \hbox (12.18277pt too wide) in paragraph at lines 1886--1888 + [][] \T1/lmr/m/n/10 (-20) If $\OML/lmm/m/it/10 X[]$ \T1/lmr/m/n/10 (-20) and $ +\OML/lmm/m/it/10 X[]$ \T1/lmr/m/n/10 (-20) are in-de-pen-dent, we get $\OMS/lms +y/m/n/10 h\OML/lmm/m/it/10 x[]x[]\OMS/lmsy/m/n/10 i \OT1/lmr/m/n/10 (-20) = \OM +S/lmsy/m/n/10 h\OML/lmm/m/it/10 x[]\OMS/lmsy/m/n/10 ih\OML/lmm/m/it/10 x[]\OMS/ +lmsy/m/n/10 i$\T1/lmr/m/n/10 (-20) , re-sult-ing in $[]\OT1/lmr/m/n/10 (-20) (\ +OML/lmm/m/it/10 X[]; X[]\OT1/lmr/m/n/10 (-20) ) = + [] + +[22] [23] [24] [25] [26] [27] [28] [29] [30] [31] [32] +\openout3 = `Regression.pyg'. + +runsystem(pygmentize -l python -f latex -F tokenmerge -P style=default -P comma +ndprefix=PYGdefault -P style=default -P commandprefix=PYGdefault -P mathescape= +True -o "_minted-Regression/A7435FEC44C0E9C1E9660C02CDC74B3F4B967B05670D715D94F +4392BD3176A49.pygtex" "Regression.pyg" )...executed. + + + +! Package minted Error: Missing Pygments output; \inputminted was +probably given a file that does not exist--otherwise, you may need +the outputdir package option, or may be using an incompatible build tool. + +See the minted package documentation for explanation. +Type H for immediate help. + ... + +l.2638 \end{minted} + +This could be caused by using -output-directory or -aux-directory +without setting minted's outputdir, or by using a build tool that +changes paths in ways minted cannot detect. + +[33] +Overfull \hbox (23.70952pt too wide) in paragraph at lines 2724--2725 +[]\T1/lmr/m/n/10 (-20) Draw with re-place-ment $\OML/lmm/m/it/10 n$ \T1/lmr/m/n +/10 (-20) num-bers for the ob-served vari-ables $\OML/cmm/b/it/10 x \OT1/lmr/m/ +n/10 (-20) = (\OML/lmm/m/it/10 x[]; x[]; [] ; x[]\OT1/lmr/m/n/10 (-20) )$\T1/l +mr/m/n/10 (-20) . + [] + +[34] +\openout3 = `Regression.pyg'. + +runsystem(pygmentize -l python -f latex -F tokenmerge -P style=default -P comma +ndprefix=PYGdefault -P style=default -P commandprefix=PYGdefault -P mathescape= +True -o "_minted-Regression/16523CD54121DE1835E829B872835EA64B967B05670D715D94F +4392BD3176A49.pygtex" "Regression.pyg" )...executed. + + + +! Package minted Error: Missing Pygments output; \inputminted was +probably given a file that does not exist--otherwise, you may need +the outputdir package option, or may be using an incompatible build tool. + +See the minted package documentation for explanation. +Type H for immediate help. + ... + +l.2805 \end{minted} + +This could be caused by using -output-directory or -aux-directory +without setting minted's outputdir, or by using a build tool that +changes paths in ways minted cannot detect. + + +Overfull \hbox (15.6617pt too wide) in paragraph at lines 2809--2809 +[][]\T1/lmr/bx/n/12 Code Ex-am-ple for Cross-validation and $\OML/lmm/m/it/12 k +$\T1/lmr/bx/n/12 -fold Cross-validation + [] + +\openout3 = `Regression.pyg'. + +runsystem(pygmentize -l python -f latex -F tokenmerge -P style=default -P comma +ndprefix=PYGdefault -P style=default -P commandprefix=PYGdefault -P mathescape= +True -o "_minted-Regression/5A860BD7323291C7CF7221BB9B86BDC74B967B05670D715D94F +4392BD3176A49.pygtex" "Regression.pyg" )...executed. + + +! Package minted Error: Missing Pygments output; \inputminted was +probably given a file that does not exist--otherwise, you may need +the outputdir package option, or may be using an incompatible build tool. + +See the minted package documentation for explanation. +Type H for immediate help. + ... + +l.2903 \end{minted} + +This could be caused by using -output-directory or -aux-directory +without setting minted's outputdir, or by using a build tool that +changes paths in ways minted cannot detect. + +[35] +\openout3 = `Regression.pyg'. + +runsystem(pygmentize -l python -f latex -F tokenmerge -P style=default -P comma +ndprefix=PYGdefault -P style=default -P commandprefix=PYGdefault -P mathescape= +True -o "_minted-Regression/457F2C4C88437DDFAF88B1D28783CF034B967B05670D715D94F +4392BD3176A49.pygtex" "Regression.pyg" )...executed. + + + +! Package minted Error: Missing Pygments output; \inputminted was +probably given a file that does not exist--otherwise, you may need +the outputdir package option, or may be using an incompatible build tool. + +See the minted package documentation for explanation. +Type H for immediate help. + ... + +l.3022 \end{minted} + +This could be caused by using -output-directory or -aux-directory +without setting minted's outputdir, or by using a build tool that +changes paths in ways minted cannot detect. + +\openout3 = `Regression.pyg'. + +runsystem(pygmentize -l python -f latex -F tokenmerge -P style=default -P comma +ndprefix=PYGdefault -P style=default -P commandprefix=PYGdefault -P mathescape= +True -o "_minted-Regression/066BCC1A504ED48C98D58B2EC6A448284B967B05670D715D94F +4392BD3176A49.pygtex" "Regression.pyg" )...executed. + + +! Package minted Error: Missing Pygments output; \inputminted was +probably given a file that does not exist--otherwise, you may need +the outputdir package option, or may be using an incompatible build tool. + +See the minted package documentation for explanation. +Type H for immediate help. + ... + +l.3078 \end{minted} + +This could be caused by using -output-directory or -aux-directory +without setting minted's outputdir, or by using a build tool that +changes paths in ways minted cannot detect. + +[36] +\openout3 = `Regression.pyg'. + +runsystem(pygmentize -l python -f latex -F tokenmerge -P style=default -P comma +ndprefix=PYGdefault -P style=default -P commandprefix=PYGdefault -P mathescape= +True -o "_minted-Regression/5D2065327E88AF636A043C8526ED982B4B967B05670D715D94F +4392BD3176A49.pygtex" "Regression.pyg" )...executed. + + + +! Package minted Error: Missing Pygments output; \inputminted was +probably given a file that does not exist--otherwise, you may need +the outputdir package option, or may be using an incompatible build tool. + +See the minted package documentation for explanation. +Type H for immediate help. + ... + +l.3190 \end{minted} + +This could be caused by using -output-directory or -aux-directory +without setting minted's outputdir, or by using a build tool that +changes paths in ways minted cannot detect. + +\openout3 = `Regression.pyg'. + +runsystem(pygmentize -l python -f latex -F tokenmerge -P style=default -P comma +ndprefix=PYGdefault -P style=default -P commandprefix=PYGdefault -P mathescape= +True -o "_minted-Regression/B0F453236926744E43E08A3FC3D889CE4B967B05670D715D94F +4392BD3176A49.pygtex" "Regression.pyg" )...executed. + + +! Package minted Error: Missing Pygments output; \inputminted was +probably given a file that does not exist--otherwise, you may need +the outputdir package option, or may be using an incompatible build tool. + +See the minted package documentation for explanation. +Type H for immediate help. + ... + +l.3234 \end{minted} + +This could be caused by using -output-directory or -aux-directory +without setting minted's outputdir, or by using a build tool that +changes paths in ways minted cannot detect. + +\openout3 = `Regression.pyg'. + +runsystem(pygmentize -l python -f latex -F tokenmerge -P style=default -P comma +ndprefix=PYGdefault -P style=default -P commandprefix=PYGdefault -P mathescape= +True -o "_minted-Regression/BFFA2954B53E806314EE31A7A4153BE34B967B05670D715D94F +4392BD3176A49.pygtex" "Regression.pyg" )...executed. + + +! Package minted Error: Missing Pygments output; \inputminted was +probably given a file that does not exist--otherwise, you may need +the outputdir package option, or may be using an incompatible build tool. + +See the minted package documentation for explanation. +Type H for immediate help. + ... + +l.3272 \end{minted} + +This could be caused by using -output-directory or -aux-directory +without setting minted's outputdir, or by using a build tool that +changes paths in ways minted cannot detect. + +\openout3 = `Regression.pyg'. + +runsystem(pygmentize -l python -f latex -F tokenmerge -P style=default -P comma +ndprefix=PYGdefault -P style=default -P commandprefix=PYGdefault -P mathescape= +True -o "_minted-Regression/F0C6E7177EB3BF181C2BCDCF6C126D174B967B05670D715D94F +4392BD3176A49.pygtex" "Regression.pyg" )...executed. + + +! Package minted Error: Missing Pygments output; \inputminted was +probably given a file that does not exist--otherwise, you may need +the outputdir package option, or may be using an incompatible build tool. + +See the minted package documentation for explanation. +Type H for immediate help. + ... + +l.3278 \end{minted} + +This could be caused by using -output-directory or -aux-directory +without setting minted's outputdir, or by using a build tool that +changes paths in ways minted cannot detect. + +[37] +\openout3 = `Regression.pyg'. + +runsystem(pygmentize -l python -f latex -F tokenmerge -P style=default -P comma +ndprefix=PYGdefault -P style=default -P commandprefix=PYGdefault -P mathescape= +True -o "_minted-Regression/679F13D40F197D86A49DDBAE2064874A4B967B05670D715D94F +4392BD3176A49.pygtex" "Regression.pyg" )...executed. + + + +! Package minted Error: Missing Pygments output; \inputminted was +probably given a file that does not exist--otherwise, you may need +the outputdir package option, or may be using an incompatible build tool. + +See the minted package documentation for explanation. +Type H for immediate help. + ... + +l.3282 \end{minted} + +This could be caused by using -output-directory or -aux-directory +without setting minted's outputdir, or by using a build tool that +changes paths in ways minted cannot detect. + +\openout3 = `Regression.pyg'. + +runsystem(pygmentize -l python -f latex -F tokenmerge -P style=default -P comma +ndprefix=PYGdefault -P style=default -P commandprefix=PYGdefault -P mathescape= +True -o "_minted-Regression/ECE8CFF2AC201FE83D423B89AB24D84D4B967B05670D715D94F +4392BD3176A49.pygtex" "Regression.pyg" )...executed. + + +! Package minted Error: Missing Pygments output; \inputminted was +probably given a file that does not exist--otherwise, you may need +the outputdir package option, or may be using an incompatible build tool. + +See the minted package documentation for explanation. +Type H for immediate help. + ... + +l.3293 \end{minted} + +This could be caused by using -output-directory or -aux-directory +without setting minted's outputdir, or by using a build tool that +changes paths in ways minted cannot detect. + +\openout3 = `Regression.pyg'. + +runsystem(pygmentize -l python -f latex -F tokenmerge -P style=default -P comma +ndprefix=PYGdefault -P style=default -P commandprefix=PYGdefault -P mathescape= +True -o "_minted-Regression/90854007E426D557D1191356516A553B4B967B05670D715D94F +4392BD3176A49.pygtex" "Regression.pyg" )...executed. + + +! Package minted Error: Missing Pygments output; \inputminted was +probably given a file that does not exist--otherwise, you may need +the outputdir package option, or may be using an incompatible build tool. + +See the minted package documentation for explanation. +Type H for immediate help. + ... + +l.3321 \end{minted} + +This could be caused by using -output-directory or -aux-directory +without setting minted's outputdir, or by using a build tool that +changes paths in ways minted cannot detect. + + +Overfull \hbox (2.88571pt too wide) in paragraph at lines 3326--3327 +\T1/lmr/m/n/10 (-20) In the \T1/lmr/bx/n/10 Least Ab-so-lute Shrink-age and Se- +lec-tion Op-er-a-tor \T1/lmr/m/n/10 (-20) (LASSO)-method + [] + +\openout3 = `Regression.pyg'. + +runsystem(pygmentize -l python -f latex -F tokenmerge -P style=default -P comma +ndprefix=PYGdefault -P style=default -P commandprefix=PYGdefault -P mathescape= +True -o "_minted-Regression/D10574B3CE6341E3B1CE27A555FC959C4B967B05670D715D94F +4392BD3176A49.pygtex" "Regression.pyg" )...executed. + + +! Package minted Error: Missing Pygments output; \inputminted was +probably given a file that does not exist--otherwise, you may need +the outputdir package option, or may be using an incompatible build tool. + +See the minted package documentation for explanation. +Type H for immediate help. + ... + +l.3346 \end{minted} + +This could be caused by using -output-directory or -aux-directory +without setting minted's outputdir, or by using a build tool that +changes paths in ways minted cannot detect. + +\openout3 = `Regression.pyg'. + +runsystem(pygmentize -l python -f latex -F tokenmerge -P style=default -P comma +ndprefix=PYGdefault -P style=default -P commandprefix=PYGdefault -P mathescape= +True -o "_minted-Regression/207A7B5C5A05397AD9CC324488E0F0A34B967B05670D715D94F +4392BD3176A49.pygtex" "Regression.pyg" )...executed. + + +! Package minted Error: Missing Pygments output; \inputminted was +probably given a file that does not exist--otherwise, you may need +the outputdir package option, or may be using an incompatible build tool. + +See the minted package documentation for explanation. +Type H for immediate help. + ... + +l.3397 \end{minted} + +This could be caused by using -output-directory or -aux-directory +without setting minted's outputdir, or by using a build tool that +changes paths in ways minted cannot detect. + +\openout3 = `Regression.pyg'. + +runsystem(pygmentize -l python -f latex -F tokenmerge -P style=default -P comma +ndprefix=PYGdefault -P style=default -P commandprefix=PYGdefault -P mathescape= +True -o "_minted-Regression/0FE2839ED408ED62CECC4C3D9FE2DDE84B967B05670D715D94F +4392BD3176A49.pygtex" "Regression.pyg" )...executed. + + +! Package minted Error: Missing Pygments output; \inputminted was +probably given a file that does not exist--otherwise, you may need +the outputdir package option, or may be using an incompatible build tool. + +See the minted package documentation for explanation. +Type H for immediate help. + ... + +l.3446 \end{minted} + +This could be caused by using -output-directory or -aux-directory +without setting minted's outputdir, or by using a build tool that +changes paths in ways minted cannot detect. + +\openout3 = `Regression.pyg'. + +runsystem(pygmentize -l python -f latex -F tokenmerge -P style=default -P comma +ndprefix=PYGdefault -P style=default -P commandprefix=PYGdefault -P mathescape= +True -o "_minted-Regression/EAD510BC3ABCC6143355164B81A3C2ED4B967B05670D715D94F +4392BD3176A49.pygtex" "Regression.pyg" )...executed. + + +! Package minted Error: Missing Pygments output; \inputminted was +probably given a file that does not exist--otherwise, you may need +the outputdir package option, or may be using an incompatible build tool. + +See the minted package documentation for explanation. +Type H for immediate help. + ... + +l.3463 \end{minted} + +This could be caused by using -output-directory or -aux-directory +without setting minted's outputdir, or by using a build tool that +changes paths in ways minted cannot detect. + +[38] [39] +\openout3 = `Regression.pyg'. + +runsystem(pygmentize -l python -f latex -F tokenmerge -P style=default -P comma +ndprefix=PYGdefault -P style=default -P commandprefix=PYGdefault -P mathescape= +True -o "_minted-Regression/EAD510BC3ABCC6143355164B81A3C2ED4B967B05670D715D94F +4392BD3176A49.pygtex" "Regression.pyg" )...executed. + + + +! Package minted Error: Missing Pygments output; \inputminted was +probably given a file that does not exist--otherwise, you may need +the outputdir package option, or may be using an incompatible build tool. + +See the minted package documentation for explanation. +Type H for immediate help. + ... + +l.3526 \end{minted} + +This could be caused by using -output-directory or -aux-directory +without setting minted's outputdir, or by using a build tool that +changes paths in ways minted cannot detect. + +[40] +\openout3 = `Regression.pyg'. + +runsystem(pygmentize -l python -f latex -F tokenmerge -P style=default -P comma +ndprefix=PYGdefault -P style=default -P commandprefix=PYGdefault -P mathescape= +True -o "_minted-Regression/795F82986CD7DD6E2E776380A1AFB2624B967B05670D715D94F +4392BD3176A49.pygtex" "Regression.pyg" )...executed. + + + +! Package minted Error: Missing Pygments output; \inputminted was +probably given a file that does not exist--otherwise, you may need +the outputdir package option, or may be using an incompatible build tool. + +See the minted package documentation for explanation. +Type H for immediate help. + ... + +l.3630 \end{minted} + +This could be caused by using -output-directory or -aux-directory +without setting minted's outputdir, or by using a build tool that +changes paths in ways minted cannot detect. + + +Overfull \hbox (4.59245pt too wide) in paragraph at lines 3633--3637 +[]\T1/lmr/m/n/10 (-20) We will thus again gen-er-ate our own dataset for a func +-tion $[]\OT1/lmr/m/n/10 (-20) (\OML/lmm/m/it/10 x; y\OT1/lmr/m/n/10 (-20) )$ + [] + +[41] +runsystem(rm "Regression.pyg")...executed. + + [42] +Package atveryend Info: Empty hook `BeforeClearDocument' on input line 3672. +Package atveryend Info: Empty hook `AfterLastShipout' on input line 3672. + (./Regression.aux) +Package atveryend Info: Executing hook `AtVeryEndDocument' on input line 3672. + + + *File List* + article.cls 2014/09/29 v1.4h Standard LaTeX document class + size10.clo 2014/09/29 v1.4h Standard LaTeX file (size option) + relsize.sty 2013/03/29 ver 4.1 + makeidx.sty 2014/09/29 v1.0m Standard LaTeX package + color.sty 1999/02/16 + color.cfg 2007/01/18 v1.5 color configuration of teTeX/TeXLive + pdftex.def 2011/05/27 v0.06d Graphics/color for pdfTeX +infwarerr.sty 2010/04/08 v1.3 Providing info/warning/error messages (HO) + ltxcmds.sty 2011/11/09 v1.22 LaTeX kernel commands for general use (HO) +setspace.sty 2011/12/19 v6.7a set line spacing + amsmath.sty 2013/01/14 v2.14 AMS math features + amstext.sty 2000/06/29 v2.01 + amsgen.sty 1999/11/30 v2.0 + amsbsy.sty 1999/11/29 v1.2d + amsopn.sty 1999/12/14 v2.01 operator names +amsfonts.sty 2013/01/14 v3.01 Basic AMSFonts support + amssymb.sty 2013/01/14 v3.01 AMS font symbols + xcolor.sty 2007/01/21 v2.11 LaTeX color extensions (UK) + color.cfg 2007/01/18 v1.5 color configuration of teTeX/TeXLive +colortbl.sty 2012/02/13 v1.0a Color table columns (DPC) + array.sty 2014/10/28 v2.4c Tabular extension package (FMi) + bm.sty 2014/10/28 v1.1c Bold Symbol Support (DPC/FMi) + ltablex.sty 2014/08/13 v1.1 Modified tabularx +longtable.sty 2014/10/28 v4.11 Multi-page Table package (DPC) +tabularx.sty 2014/10/28 v2.10 `tabularx' package (DPC) +microtype.sty 2013/05/23 v2.5a Micro-typographical refinements (RS) + keyval.sty 2014/10/28 v1.15 key=value parser (DPC) +microtype-pdftex.def 2013/05/23 v2.5a Definitions specific to pdftex (RS) +microtype.cfg 2013/05/23 v2.5a microtype main configuration file (RS) +graphicx.sty 2014/10/28 v1.0g Enhanced LaTeX Graphics (DPC,SPQR) +graphics.sty 2014/10/28 v1.0p Standard LaTeX Graphics (DPC,SPQR) + trig.sty 1999/03/16 v1.09 sin cos tan (DPC) +graphics.cfg 2010/04/23 v1.9 graphics configuration of TeX Live +fancyvrb.sty 2008/02/07 + minted.sty 2015/01/31 v2.0 Yet another Pygments shim for LaTeX +kvoptions.sty 2011/06/30 v3.11 Key value format for package options (HO) +kvsetkeys.sty 2012/04/25 v1.16 Key value parser (HO) +etexcmds.sty 2011/02/16 v1.5 Avoid name clashes with e-TeX commands (HO) +ifluatex.sty 2010/03/01 v1.3 Provides the ifluatex switch (HO) + float.sty 2001/11/08 v1.3d Float enhancements (AL) + ifthen.sty 2014/09/29 v1.1c Standard LaTeX ifthen package (DPC) + calc.sty 2014/10/28 v4.3 Infix arithmetic (KKT,FJ) +ifplatform.sty 2010/10/22 v0.4 Testing for the operating system +pdftexcmds.sty 2011/11/29 v0.20 Utility functions of pdfTeX for LuaTeX (HO) + ifpdf.sty 2011/01/30 v2.3 Provides the ifpdf switch (HO) +catchfile.sty 2011/03/01 v1.6 Catch the contents of a file (HO) +Regression.w18 +etoolbox.sty 2015/05/04 v2.2 e-TeX tools for LaTeX (JAW) + xstring.sty 2013/10/13 v1.7c String manipulations (C Tellechea) + lineno.sty 2005/11/02 line numbers on paragraphs v4.41 +_minted-Regression/default.pygstyle + fontenc.sty + t1enc.def 2005/09/27 v1.99g Standard LaTeX file + ucs.sty 2013/05/11 v2.2 UCS: Unicode input support +uni-global.def 2013/05/13 UCS: Unicode global data +inputenc.sty 2015/03/17 v1.2c Input encoding file + utf8x.def 2004/10/17 UCS: Input encoding UTF-8 + lmodern.sty 2009/10/30 v1.6 Latin Modern Fonts +hyperref.sty 2012/11/06 v6.83m Hypertext links for LaTeX +hobsub-hyperref.sty 2012/05/28 v1.13 Bundle oberdiek, subset hyperref (HO) +hobsub-generic.sty 2012/05/28 v1.13 Bundle oberdiek, subset generic (HO) + hobsub.sty 2012/05/28 v1.13 Construct package bundles (HO) + ifvtex.sty 2010/03/01 v1.5 Detect VTeX and its facilities (HO) + intcalc.sty 2007/09/27 v1.1 Expandable calculations with integers (HO) +kvdefinekeys.sty 2011/04/07 v1.3 Define keys (HO) +pdfescape.sty 2011/11/25 v1.13 Implements pdfTeX's escape features (HO) +bigintcalc.sty 2012/04/08 v1.3 Expandable calculations on big integers (HO) + bitset.sty 2011/01/30 v1.1 Handle bit-vector datatype (HO) +uniquecounter.sty 2011/01/30 v1.2 Provide unlimited unique counter (HO) +letltxmacro.sty 2010/09/02 v1.4 Let assignment for LaTeX macros (HO) + hopatch.sty 2012/05/28 v1.2 Wrapper for package hooks (HO) +xcolor-patch.sty 2011/01/30 xcolor patch +atveryend.sty 2011/06/30 v1.8 Hooks at the very end of document (HO) +atbegshi.sty 2011/10/05 v1.16 At begin shipout hook (HO) +refcount.sty 2011/10/16 v3.4 Data extraction from label references (HO) + hycolor.sty 2011/01/30 v1.7 Color options for hyperref/bookmark (HO) + ifxetex.sty 2010/09/12 v0.6 Provides ifxetex conditional + auxhook.sty 2011/03/04 v1.3 Hooks for auxiliary files (HO) + pd1enc.def 2012/11/06 v6.83m Hyperref: PDFDocEncoding definition (HO) +hyperref.cfg 2002/06/06 v1.2 hyperref configuration of TeXLive + url.sty 2013/09/16 ver 3.4 Verb mode for urls, etc. + hpdftex.def 2012/11/06 v6.83m Hyperref driver for pdfTeX +rerunfilecheck.sty 2011/04/15 v1.7 Rerun checks for auxiliary files (HO) +fancyhdr.sty +mdframed.sty 2013/07/01 1.9b: mdframed + xparse.sty 2014/11/25 v5471 L3 Experimental document command parser + expl3.sty 2015/03/01 v5547 L3 programming layer (loader) +expl3-code.tex 2015/03/01 v5547 L3 programming layer +l3unicode-data.def 2015/03/01 v5544 L3 Unicode data +l3pdfmode.def 2015/03/01 v5544 L3 Experimental driver: PDF mode +zref-abspage.sty 2012/04/04 v2.24 Module abspage for zref (HO) +zref-base.sty 2012/04/04 v2.24 Module base for zref (HO) +needspace.sty 2010/09/12 v1.3d reserve vertical space + tikz.sty 2013/12/13 v3.0.0 (rcs-revision 1.142) + pgf.sty 2013/12/18 v3.0.0 (rcs-revision 1.14) + pgfrcs.sty 2013/12/20 v3.0.0 (rcs-revision 1.28) +everyshi.sty 2001/05/15 v3.00 EveryShipout Package (MS) + pgfrcs.code.tex + pgfcore.sty 2010/04/11 v3.0.0 (rcs-revision 1.7) + pgfsys.sty 2013/11/30 v3.0.0 (rcs-revision 1.47) + pgfsys.code.tex +pgfsyssoftpath.code.tex 2013/09/09 (rcs-revision 1.9) +pgfsysprotocol.code.tex 2006/10/16 (rcs-revision 1.4) + pgfcore.code.tex +pgfcomp-version-0-65.sty 2007/07/03 v3.0.0 (rcs-revision 1.7) +pgfcomp-version-1-18.sty 2007/07/23 v3.0.0 (rcs-revision 1.1) + pgffor.sty 2013/12/13 v3.0.0 (rcs-revision 1.25) + pgfkeys.sty + pgfkeys.code.tex + pgfmath.sty + pgfmath.code.tex + pgffor.code.tex + tikz.code.tex +md-frame-1.mdf 2013/07/01\ 1.9b: md-frame-1 +idxlayout.sty 2012/03/30 v0.4d Configurable index layout +multicol.sty 2015/03/31 v1.8m multicolumn formatting (FMi) +tocbibind.sty 2010/10/13 v1.5k extra ToC listings +ragged2e.sty 2009/05/21 v2.1 ragged2e Package (MS) +everysel.sty 2011/10/28 v1.2 EverySelectfont Package (MS) + t1lmr.fd 2009/10/30 v1.6 Font defs for Latin Modern +supp-pdf.mkii + mt-cmr.cfg 2013/05/19 v2.2 microtype config. file: Computer Modern Roman ( +RS) +epstopdf-base.sty 2010/02/09 v2.5 Base part for package epstopdf + grfext.sty 2010/08/19 v1.1 Manage graphics extensions (HO) +epstopdf-sys.cfg 2010/07/13 v1.3 Configuration of (r)epstopdf for TeX Live + ucsencs.def 2011/01/21 Fixes to fontencodings LGR, T3 + nameref.sty 2012/10/27 v2.43 Cross-referencing by name of section +gettitlestring.sty 2010/12/03 v1.4 Cleanup title references (HO) +Regression.out +Regression.out + ot1lmr.fd 2009/10/30 v1.6 Font defs for Latin Modern + omllmm.fd 2009/10/30 v1.6 Font defs for Latin Modern + omslmsy.fd 2009/10/30 v1.6 Font defs for Latin Modern + omxlmex.fd 2009/10/30 v1.6 Font defs for Latin Modern + umsa.fd 2013/01/14 v3.01 AMS symbols A + mt-msa.cfg 2006/02/04 v1.1 microtype config. file: AMS symbols (a) (RS) + umsb.fd 2013/01/14 v3.01 AMS symbols B + mt-msb.cfg 2005/06/01 v1.0 microtype config. file: AMS symbols (b) (RS) + omslmr.fd 2009/10/30 v1.6 Font defs for Latin Modern + uni-32.def 2013/05/13 UCS: Unicode data U+2000..U+20FF + *********** + +Package atveryend Info: Executing hook `AtEndAfterFileList' on input line 3672. + +Package rerunfilecheck Info: File `Regression.out' has not changed. +(rerunfilecheck) Checksum: D41D8CD98F00B204E9800998ECF8427E;0. + ) +Here is how much of TeX's memory you used: + 28066 strings out of 493089 + 535586 string characters out of 6134842 + 655626 words of memory out of 5000000 + 30575 multiletter control sequences out of 15000+600000 + 92488 words of font info for 170 fonts, out of 8000000 for 9000 + 1141 hyphenation exceptions out of 8191 + 67i,15n,64p,10395b,430s stack positions out of 5000i,500n,10000p,200000b,80000s +{/usr/local/texlive/2015/texmf-dist/fonts/enc/dvips/lm/lm-mathsy.enc}{/usr/lo +cal/texlive/2015/texmf-dist/fonts/enc/dvips/lm/lm-rm.enc}{/usr/local/texlive/20 +15/texmf-dist/fonts/enc/dvips/lm/lm-mathit.enc}{/usr/local/texlive/2015/texmf-d +ist/fonts/enc/dvips/lm/lm-mathex.enc}{/usr/local/texlive/2015/texmf-dist/fonts/ +enc/dvips/lm/lm-ec.enc} +Output written on Regression.pdf (42 pages, 459187 bytes). +PDF statistics: + 582 PDF objects out of 1000 (max. 8388607) + 508 compressed objects within 6 object streams + 247 named destinations out of 1000 (max. 500000) + 28685 words of extra memory for PDF output out of 29859 (max. 10000000) + diff --git a/doc/src/Regression/Regression.out b/doc/src/Regression/Regression.out new file mode 100644 index 000000000..e69de29bb diff --git a/doc/src/Regression/Regression.p.tex b/doc/src/Regression/Regression.p.tex new file mode 100644 index 000000000..9e331b7eb --- /dev/null +++ b/doc/src/Regression/Regression.p.tex @@ -0,0 +1,3703 @@ +%% +%% Automatically generated file from DocOnce source +%% (https://github.com/hplgit/doconce/) +%% +%% +% #ifdef PTEX2TEX_EXPLANATION +%% +%% The file follows the ptex2tex extended LaTeX format, see +%% ptex2tex: http://code.google.com/p/ptex2tex/ +%% +%% Run +%% ptex2tex myfile +%% or +%% doconce ptex2tex myfile +%% +%% to turn myfile.p.tex into an ordinary LaTeX file myfile.tex. +%% (The ptex2tex program: http://code.google.com/p/ptex2tex) +%% Many preprocess options can be added to ptex2tex or doconce ptex2tex +%% +%% ptex2tex -DMINTED myfile +%% doconce ptex2tex myfile envir=minted +%% +%% ptex2tex will typeset code environments according to a global or local +%% .ptex2tex.cfg configure file. doconce ptex2tex will typeset code +%% according to options on the command line (just type doconce ptex2tex to +%% see examples). If doconce ptex2tex has envir=minted, it enables the +%% minted style without needing -DMINTED. +% #endif + +% #define PREAMBLE + +% #ifdef PREAMBLE +%-------------------- begin preamble ---------------------- + +\documentclass[% +oneside, % oneside: electronic viewing, twoside: printing +final, % draft: marks overfull hboxes, figures with paths +10pt]{article} + +\listfiles % print all files needed to compile this document + +\usepackage{relsize,makeidx,color,setspace,amsmath,amsfonts,amssymb} +\usepackage[table]{xcolor} +\usepackage{bm,ltablex,microtype} + +\usepackage[pdftex]{graphicx} + +\usepackage{ptex2tex} +% #ifdef MINTED +\usepackage{minted} +\usemintedstyle{default} +% #endif + +\usepackage[T1]{fontenc} +%\usepackage[latin1]{inputenc} +\usepackage{ucs} +\usepackage[utf8x]{inputenc} + +\usepackage{lmodern} % Latin Modern fonts derived from Computer Modern + +% Hyperlinks in PDF: +\definecolor{linkcolor}{rgb}{0,0,0.4} +\usepackage{hyperref} +\hypersetup{ + breaklinks=true, + colorlinks=true, + linkcolor=linkcolor, + urlcolor=linkcolor, + citecolor=black, + filecolor=black, + %filecolor=blue, + pdfmenubar=true, + pdftoolbar=true, + bookmarksdepth=3 % Uncomment (and tweak) for PDF bookmarks with more levels than the TOC + } +%\hyperbaseurl{} % hyperlinks are relative to this root + +\setcounter{tocdepth}{2} % levels in table of contents + +% --- fancyhdr package for fancy headers --- +\usepackage{fancyhdr} +\fancyhf{} % sets both header and footer to nothing +\renewcommand{\headrulewidth}{0pt} +\fancyfoot[LE,RO]{\thepage} +% Ensure copyright on titlepage (article style) and chapter pages (book style) +\fancypagestyle{plain}{ + \fancyhf{} + \fancyfoot[C]{{\footnotesize \copyright\ 1999-2019, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license}} +% \renewcommand{\footrulewidth}{0mm} + \renewcommand{\headrulewidth}{0mm} +} +% Ensure copyright on titlepages with \thispagestyle{empty} +\fancypagestyle{empty}{ + \fancyhf{} + \fancyfoot[C]{{\footnotesize \copyright\ 1999-2019, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license}} + \renewcommand{\footrulewidth}{0mm} + \renewcommand{\headrulewidth}{0mm} +} + +\pagestyle{fancy} + + +\usepackage[framemethod=TikZ]{mdframed} + +% --- begin definitions of admonition environments --- + +% --- end of definitions of admonition environments --- + +% prevent orhpans and widows +\clubpenalty = 10000 +\widowpenalty = 10000 + +% --- end of standard preamble for documents --- + + +% insert custom LaTeX commands... + +\raggedbottom +\makeindex +\usepackage[totoc]{idxlayout} % for index in the toc +\usepackage[nottoc]{tocbibind} % for references/bibliography in the toc + +%-------------------- end preamble ---------------------- + +\begin{document} + +% matching end for #ifdef PREAMBLE +% #endif + +\newcommand{\exercisesection}[1]{\subsection*{#1}} + + +% ------------------- main content ---------------------- + + + +% ----------------- title ------------------------- + +\thispagestyle{empty} + +\begin{center} +{\LARGE\bf +\begin{spacing}{1.25} +Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis +\end{spacing} +} +\end{center} + +% ----------------- author(s) ------------------------- + +\begin{center} +{\bf Morten Hjorth-Jensen${}^{1, 2}$} \\ [0mm] +\end{center} + +\begin{center} +% List of all institutions: +\centerline{{\small ${}^1$Department of Physics, University of Oslo}} +\centerline{{\small ${}^2$Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University}} +\end{center} + +% ----------------- end author(s) ------------------------- + +% --- begin date --- +\begin{center} +Jul 22, 2019 +\end{center} +% --- end date --- + +\vspace{1cm} + + +% !split +\subsection{Why Linear Regression (aka Ordinary Least Squares and family)} + +Fitting a continuous function with linear parameterization in terms of the parameters $\bm{\beta}$. +\begin{itemize} +\item Method of choice for fitting a continuous function! + +\item Gives an excellent introduction to central Machine Learning features with \textbf{understandable pedagogical} links to other methods like \textbf{Neural Networks}, \textbf{Support Vector Machines} etc + +\item Analytical expression for the fitting parameters $\bm{\beta}$ + +\item Analytical expressions for statistical propertiers like mean values, variances, confidence intervals and more + +\item Analytical relation with probabilistic interpretations + +\item Easy to introduce basic concepts like bias-variance tradeoff, cross-validation, resampling and regularization techniques and many other ML topics + +\item Easy to code! And links well with classification problems and logistic regression and neural networks + +\item Allows for \textbf{easy} hands-on understanding of gradient descent methods + +\item and many more features +\end{itemize} + +\noindent +For more discussions of Ridge and Lasso regression, \href{{https://arxiv.org/abs/1509.09169}}{Wessel van Wieringen's} article is highly recommended. +Similarly, \href{{https://arxiv.org/abs/1803.08823}}{Mehta et al's article} is also recommended. + + +% !split +\subsection{Regression analysis, overarching aims} + +% --- begin paragraph admon --- +\paragraph{} + +Regression modeling deals with the description of the sampling distribution of a given random variable $y$ and how it varies as function of another variable or a set of such variables $\bm{x} =[x_0, x_1,\dots, x_{n-1}]^T$. +The first variable is called the \textbf{dependent}, the \textbf{outcome} or the \textbf{response} variable while the set of variables $\bm{x}$ is called the independent variable, or the predictor variable or the explanatory variable. + +A regression model aims at finding a likelihood function $p(\bm{y}\vert \bm{x})$, that is the conditional distribution for $\bm{y}$ with a given $\bm{x}$. The estimation of $p(\bm{y}\vert \bm{x})$ is made using a data set with +\begin{itemize} +\item $n$ cases $i = 0, 1, 2, \dots, n-1$ + +\item Response (target, dependent or outcome) variable $y_i$ with $i = 0, 1, 2, \dots, n-1$ + +\item $p$ so-called explanatory (independent or predictor) variables $\bm{x}_i=[x_{i0}, x_{i1}, \dots, x_{ip-1}]$ with $i = 0, 1, 2, \dots, n-1$ and explanatory variables running from $0$ to $p-1$. See below for more explicit examples. +\end{itemize} + +\noindent + The goal of the regression analysis is to extract/exploit relationship between $\bm{y}$ and $\bm{X}$ in or to infer causal dependencies, approximations to the likelihood functions, functional relationships and to make predictions, making fits and many other things. +% --- end paragraph admon --- + + + +% !split +\subsection{Regression analysis, overarching aims II} + +% --- begin paragraph admon --- +\paragraph{} + + +Consider an experiment in which $p$ characteristics of $n$ samples are +measured. The data from this experiment, for various explanatory variables $p$ are normally represented by a matrix +$\mathbf{X}$. + +The matrix $\mathbf{X}$ is called the \emph{design +matrix}. Additional information of the samples is available in the +form of $\bm{y}$ (also as above). The variable $\bm{y}$ is +generally referred to as the \emph{response variable}. The aim of +regression analysis is to explain $\bm{y}$ in terms of +$\bm{X}$ through a functional relationship like $y_i = +f(\mathbf{X}_{i,\ast})$. When no prior knowledge on the form of +$f(\cdot)$ is available, it is common to assume a linear relationship +between $\bm{X}$ and $\bm{y}$. This assumption gives rise to +the \emph{linear regression model} where $\bm{\beta} = [\beta_0, \ldots, +\beta_{p-1}]^{T}$ are the \emph{regression parameters}. + +Linear regression gives us a set of analytical equations for the parameters $\beta_j$. +% --- end paragraph admon --- + + + + + +% !split +\subsection{Examples} + +% --- begin paragraph admon --- +\paragraph{} +In order to understand the relation among the predictors $p$, the set of data $n$ and the target (outcome, output etc) $\bm{y}$, +consider the model we discussed for describing nuclear binding energies. + +There we assumed that we could parametrize the data using a polynomial approximation based on the liquid drop model. +Assuming +\[ +BE(A) = a_0+a_1A+a_2A^{2/3}+a_3A^{-1/3}+a_4A^{-1}, +\] +we have five predictors, that is the intercept, the $A$ dependent term, the $A^{2/3}$ term and the $A^{-1/3}$ and $A^{-1}$ terms. +This gives $p=0,1,2,3,4$. Furthermore we have $n$ entries for each predictor. It means that our design matrix is a +$p\times n$ matrix $\bm{X}$. + +Here the predictors are based on a model we have made. A popular data set which is widely encountered in ML applications is the +so-called \href{{https://www.sciencedirect.com/science/article/pii/S0957417407006719?via%3Dihub}}{credit card default data from Taiwan}. The data set contains data on $n=30000$ credit card holders with predictors like gender, marital status, age, profession, education, etc. In total there are $24$ such predictors or attributes leading to a design matrix of dimensionality $24 \times 30000$ +% --- end paragraph admon --- + + + + + + + +% !split +\subsection{General linear models} + +% --- begin paragraph admon --- +\paragraph{} +Before we proceed let us study a case from linear algebra where we aim at fitting a set of data $\bm{y}=[y_0,y_1,\dots,y_{n-1}]$. We could think of these data as a result of an experiment or a complicated numerical experiment. These data are functions of a series of variables $\bm{x}=[x_0,x_1,\dots,x_{n-1}]$, that is $y_i = y(x_i)$ with $i=0,1,2,\dots,n-1$. The variables $x_i$ could represent physical quantities like time, temperature, position etc. We assume that $y(x)$ is a smooth function. + +Since obtaining these data points may not be trivial, we want to use these data to fit a function which can allow us to make predictions for values of $y$ which are not in the present set. The perhaps simplest approach is to assume we can parametrize our function in terms of a polynomial of degree $n-1$ with $n$ points, that is +\[ +y=y(x) \rightarrow y(x_i)=\tilde{y}_i+\epsilon_i=\sum_{j=0}^{n-1} \beta_j x_i^j+\epsilon_i, +\] +where $\epsilon_i$ is the error in our approximation. +% --- end paragraph admon --- + + + + +% !split +\subsection{Rewriting the fitting procedure as a linear algebra problem} + +% --- begin paragraph admon --- +\paragraph{} +For every set of values $y_i,x_i$ we have thus the corresponding set of equations +\begin{align*} +y_0&=\beta_0+\beta_1x_0^1+\beta_2x_0^2+\dots+\beta_{n-1}x_0^{n-1}+\epsilon_0\\ +y_1&=\beta_0+\beta_1x_1^1+\beta_2x_1^2+\dots+\beta_{n-1}x_1^{n-1}+\epsilon_1\\ +y_2&=\beta_0+\beta_1x_2^1+\beta_2x_2^2+\dots+\beta_{n-1}x_2^{n-1}+\epsilon_2\\ +\dots & \dots \\ +y_{n-1}&=\beta_0+\beta_1x_{n-1}^1+\beta_2x_{n-1}^2+\dots+\beta_{n-1}x_{n-1}^{n-1}+\epsilon_{n-1}.\\ +\end{align*} +% --- end paragraph admon --- + + + + +% !split +\subsection{Rewriting the fitting procedure as a linear algebra problem, more details} + +% --- begin paragraph admon --- +\paragraph{} +Defining the vectors +\[ +\bm{y} = [y_0,y_1, y_2,\dots, y_{n-1}]^T, +\] +and +\[ +\bm{\beta} = [\beta_0,\beta_1, \beta_2,\dots, \beta_{n-1}]^T, +\] +and +\[ +\bm{\epsilon} = [\epsilon_0,\epsilon_1, \epsilon_2,\dots, \epsilon_{n-1}]^T, +\] +and the design matrix +\[ +\bm{X}= +\begin{bmatrix} +1& x_{0}^1 &x_{0}^2& \dots & \dots &x_{0}^{n-1}\\ +1& x_{1}^1 &x_{1}^2& \dots & \dots &x_{1}^{n-1}\\ +1& x_{2}^1 &x_{2}^2& \dots & \dots &x_{2}^{n-1}\\ +\dots& \dots &\dots& \dots & \dots &\dots\\ +1& x_{n-1}^1 &x_{n-1}^2& \dots & \dots &x_{n-1}^{n-1}\\ +\end{bmatrix} +\] +we can rewrite our equations as +\[ +\bm{y} = \bm{X}\bm{\beta}+\bm{\epsilon}. +\] +The above design matrix is called a \href{{https://en.wikipedia.org/wiki/Vandermonde_matrix}}{Vandermonde matrix}. +% --- end paragraph admon --- + + + + +% !split +\subsection{Generalizing the fitting procedure as a linear algebra problem} + +% --- begin paragraph admon --- +\paragraph{} + +We are obviously not limited to the above polynomial expansions. We +could replace the various powers of $x$ with elements of Fourier +series or instead of $x_i^j$ we could have $\cos{(j x_i)}$ or $\sin{(j +x_i)}$, or time series or other orthogonal functions. For every set +of values $y_i,x_i$ we can then generalize the equations to + +\begin{align*} +y_0&=\beta_0x_{00}+\beta_1x_{01}+\beta_2x_{02}+\dots+\beta_{n-1}x_{0n-1}+\epsilon_0\\ +y_1&=\beta_0x_{10}+\beta_1x_{11}+\beta_2x_{12}+\dots+\beta_{n-1}x_{1n-1}+\epsilon_1\\ +y_2&=\beta_0x_{20}+\beta_1x_{21}+\beta_2x_{22}+\dots+\beta_{n-1}x_{2n-1}+\epsilon_2\\ +\dots & \dots \\ +y_{i}&=\beta_0x_{i0}+\beta_1x_{i1}+\beta_2x_{i2}+\dots+\beta_{n-1}x_{in-1}+\epsilon_i\\ +\dots & \dots \\ +y_{n-1}&=\beta_0x_{n-1,0}+\beta_1x_{n-1,2}+\beta_2x_{n-1,2}+\dots+\beta_{n-1}x_{n-1,n-1}+\epsilon_{n-1}.\\ +\end{align*} + +\textbf{Note that we have $p=n$ here. The matrix is symmetric. This is generally not the case!} +% --- end paragraph admon --- + + + + +% !split +\subsection{Generalizing the fitting procedure as a linear algebra problem} + +% --- begin paragraph admon --- +\paragraph{} +We redefine in turn the matrix $\bm{X}$ as +\[ +\bm{X}= +\begin{bmatrix} +x_{00}& x_{01} &x_{02}& \dots & \dots &x_{0,n-1}\\ +x_{10}& x_{11} &x_{12}& \dots & \dots &x_{1,n-1}\\ +x_{20}& x_{21} &x_{22}& \dots & \dots &x_{2,n-1}\\ +\dots& \dots &\dots& \dots & \dots &\dots\\ +x_{n-1,0}& x_{n-1,1} &x_{n-1,2}& \dots & \dots &x_{n-1,n-1}\\ +\end{bmatrix} +\] +and without loss of generality we rewrite again our equations as +\[ +\bm{y} = \bm{X}\bm{\beta}+\bm{\epsilon}. +\] +The left-hand side of this equation is kwown. Our error vector $\bm{\epsilon}$ and the parameter vector $\bm{\beta}$ are our unknow quantities. How can we obtain the optimal set of $\beta_i$ values? +% --- end paragraph admon --- + + + + +% !split +\subsection{Optimizing our parameters} + +% --- begin paragraph admon --- +\paragraph{} +We have defined the matrix $\bm{X}$ via the equations +\begin{align*} +y_0&=\beta_0x_{00}+\beta_1x_{01}+\beta_2x_{02}+\dots+\beta_{n-1}x_{0n-1}+\epsilon_0\\ +y_1&=\beta_0x_{10}+\beta_1x_{11}+\beta_2x_{12}+\dots+\beta_{n-1}x_{1n-1}+\epsilon_1\\ +y_2&=\beta_0x_{20}+\beta_1x_{21}+\beta_2x_{22}+\dots+\beta_{n-1}x_{2n-1}+\epsilon_1\\ +\dots & \dots \\ +y_{i}&=\beta_0x_{i0}+\beta_1x_{i1}+\beta_2x_{i2}+\dots+\beta_{n-1}x_{in-1}+\epsilon_1\\ +\dots & \dots \\ +y_{n-1}&=\beta_0x_{n-1,0}+\beta_1x_{n-1,2}+\beta_2x_{n-1,2}+\dots+\beta_{n-1}x_{n-1,n-1}+\epsilon_{n-1}.\\ +\end{align*} + +As we noted above, we stayed with a system with the design matrix + $\bm{X}\in {\mathbb{R}}^{n\times n}$, that is we have $p=n$. For reasons to come later (algorithmic arguments) we will hereafter define +our matrix as $\bm{X}\in {\mathbb{R}}^{n\times p}$, with the predictors refering to the column numbers and the entries $n$ being the row elements. +% --- end paragraph admon --- + + + + +% !split +\subsection{Our model for the nuclear binding energies} + +In our \href{{https://compphysics.github.io/MachineLearningMSU/doc/pub/Introduction/html/Introduction.html}}{introductory notes} we looked at the so-called \href{{https://en.wikipedia.org/wiki/Semi-empirical_mass_formula}}{liguid drop model}. Let us remind ourselves about what we did by looking at the code. + +We restate the parts of the code we are most interested in. +\bpycod +# Common imports +import numpy as np +import pandas as pd +import matplotlib.pyplot as plt +from IPython.display import display +import os + +# Where to save the figures and data files +PROJECT_ROOT_DIR = "Results" +FIGURE_ID = "Results/FigureFiles" +DATA_ID = "DataFiles/" + +if not os.path.exists(PROJECT_ROOT_DIR): + os.mkdir(PROJECT_ROOT_DIR) + +if not os.path.exists(FIGURE_ID): + os.makedirs(FIGURE_ID) + +if not os.path.exists(DATA_ID): + os.makedirs(DATA_ID) + +def image_path(fig_id): + return os.path.join(FIGURE_ID, fig_id) + +def data_path(dat_id): + return os.path.join(DATA_ID, dat_id) + +def save_fig(fig_id): + plt.savefig(image_path(fig_id) + ".png", format='png') + +infile = open(data_path("MassEval2016.dat"),'r') + + +# Read the experimental data with Pandas +Masses = pd.read_fwf(infile, usecols=(2,3,4,6,11), + names=('N', 'Z', 'A', 'Element', 'Ebinding'), + widths=(1,3,5,5,5,1,3,4,1,13,11,11,9,1,2,11,9,1,3,1,12,11,1), + header=39, + index_col=False) + +# Extrapolated values are indicated by '#' in place of the decimal place, so +# the Ebinding column won't be numeric. Coerce to float and drop these entries. +Masses['Ebinding'] = pd.to_numeric(Masses['Ebinding'], errors='coerce') +Masses = Masses.dropna() +# Convert from keV to MeV. +Masses['Ebinding'] /= 1000 + +# Group the DataFrame by nucleon number, A. +Masses = Masses.groupby('A') +# Find the rows of the grouped DataFrame with the maximum binding energy. +Masses = Masses.apply(lambda t: t[t.Ebinding==t.Ebinding.max()]) +A = Masses['A'] +Z = Masses['Z'] +N = Masses['N'] +Element = Masses['Element'] +Energies = Masses['Ebinding'] + +# Now we set up the design matrix X +X = np.zeros((len(A),5)) +X[:,0] = 1 +X[:,1] = A +X[:,2] = A**(2.0/3.0) +X[:,3] = A**(-1.0/3.0) +X[:,4] = A**(-1.0) +# Then nice printout using pandas +DesignMatrix = pd.DataFrame(X) +DesignMatrix.index = A +DesignMatrix.columns = ['1', 'A', 'A^(2/3)', 'A^(-1/3)', '1/A'] +display(DesignMatrix) +\epycod + +With $\bm{\beta}\in {\mathbb{R}}^{p\times 1}$, it means that we will hereafter write our equations for the approximation as +\[ +\bm{\tilde{y}}= \bm{X}\bm{\beta}, +\] +throughout these lectures. + + +% !split +\subsection{Optimizing our parameters, more details} + +% --- begin paragraph admon --- +\paragraph{} +With the above we use the design matrix to define the approximation $\bm{\tilde{y}}$ via the unknown quantity $\bm{\beta}$ as +\[ +\bm{\tilde{y}}= \bm{X}\bm{\beta}, +\] +and in order to find the optimal parameters $\beta_i$ instead of solving the above linear algebra problem, we define a function which gives a measure of the spread between the values $y_i$ (which represent hopefully the exact values) and the parameterized values $\tilde{y}_i$, namely +\[ +C(\bm{\beta})=\frac{1}{n}\sum_{i=0}^{n-1}\left(y_i-\tilde{y}_i\right)^2=\frac{1}{n}\left\{\left(\bm{y}-\bm{\tilde{y}}\right)^T\left(\bm{y}-\bm{\tilde{y}}\right)\right\}, +\] +or using the matrix $\bm{X}$ and in a more compact matrix-vector notation as +\[ +C(\bm{\beta})=\frac{1}{n}\left\{\left(\bm{y}-\bm{X}^T\bm{\beta}\right)^T\left(\bm{y}-\bm{X}^T\bm{\beta}\right)\right\}. +\] +This function is one possible way to define the so-called cost function. + + + +It is also common to define +the function $Q$ as + +\[ +C(\bm{\beta})=\frac{1}{2n}\sum_{i=0}^{n-1}\left(y_i-\tilde{y}_i\right)^2, +\] +since when taking the first derivative with respect to the unknown parameters $\beta$, the factor of $2$ cancels out. +% --- end paragraph admon --- + + + + +% !split +\subsection{Interpretations and optimizing our parameters} + +% --- begin paragraph admon --- +\paragraph{} + +The function +\[ +C(\bm{\beta})=\frac{1}{n}\left\{\left(\bm{y}-\bm{X}\bm{\beta}\right)^T\left(\bm{y}-\bm{X}\bm{\beta}\right)\right\}, +\] +can be linked to the variance of the quantity $y_i$ if we interpret the latter as the mean value. +When linking below with the maximum likelihood approach below, we will indeed interpret $y_i$ as a mean value (see exercises) +\[ +y_{i}=\langle y_i \rangle = \beta_0x_{i,0}+\beta_1x_{i,1}+\beta_2x_{i,2}+\dots+\beta_{n-1}x_{i,n-1}+\epsilon_i, +\] + +where $\langle y_i \rangle$ is the mean value. Keep in mind also that +till now we have treated $y_i$ as the exact value. Normally, the +response (dependent or outcome) variable $y_i$ the outcome of a +numerical experiment or another type of experiment and is thus only an +approximation to the true value. It is then always accompanied by an +error estimate, often limited to a statistical error estimate given by +the standard deviation discussed earlier. In the discussion here we +will treat $y_i$ as our exact value for the response variable. + +In order to find the parameters $\beta_i$ we will then minimize the spread of $C(\bm{\beta})$, that is we are going to solve the problem +\[ +{\displaystyle \min_{\bm{\beta}\in +{\mathbb{R}}^{p}}}\frac{1}{n}\left\{\left(\bm{y}-\bm{X}\bm{\beta}\right)^T\left(\bm{y}-\bm{X}\bm{\beta}\right)\right\}. +\] +In practical terms it means we will require +\[ +\frac{\partial C(\bm{\beta})}{\partial \beta_j} = \frac{\partial }{\partial \beta_j}\left[ \frac{1}{n}\sum_{i=0}^{n-1}\left(y_i-\beta_0x_{i,0}-\beta_1x_{i,1}-\beta_2x_{i,2}-\dots-\beta_{n-1}x_{i,n-1}\right)^2\right]=0, +\] +which results in +\[ +\frac{\partial C(\bm{\beta})}{\partial \beta_j} = -\frac{2}{n}\left[ \sum_{i=0}^{n-1}x_{ij}\left(y_i-\beta_0x_{i,0}-\beta_1x_{i,1}-\beta_2x_{i,2}-\dots-\beta_{n-1}x_{i,n-1}\right)\right]=0, +\] +or in a matrix-vector form as +\[ +\frac{\partial C(\bm{\beta})}{\partial \bm{\beta}} = 0 = \bm{X}^T\left( \bm{y}-\bm{X}\bm{\beta}\right). +\] +% --- end paragraph admon --- + + + + +% !split +\subsection{Interpretations and optimizing our parameters} + +% --- begin paragraph admon --- +\paragraph{} +We can rewrite +\[ +\frac{\partial C(\bm{\beta})}{\partial \bm{\beta}} = 0 = \bm{X}^T\left( \bm{y}-\bm{X}\bm{\beta}\right), +\] +as +\[ +\bm{X}^T\bm{y} = \bm{X}^T\bm{X}\bm{\beta}, +\] +and if the matrix $\bm{X}^T\bm{X}$ is invertible we have the solution +\[ +\bm{\beta} =\left(\bm{X}^T\bm{X}\right)^{-1}\bm{X}^T\bm{y}. +\] + +We note also that since our design matrix is defined as $\bm{X}\in +{\mathbb{R}}^{n\times p}$, the product $\bm{X}^T\bm{X} \in +{\mathbb{R}}^{p\times p}$. In the above case we have that $p \ll n$, +in our case $p=5$ meaning that we end up with inverting a small +$5\times 5$ matrix. This is a rather common situation, in many cases we end up with low-dimensional +matrices to invert. The methods discussed here and for many other +supervised learning algorithms like classification with logistic +regression or support vector machines, exhibit dimensionalities which +allow for the usage of direct linear algebra methods such as \textbf{LU} decomposition or \textbf{Singular Value Decomposition} (SVD) for finding the inverse of the matrix +$\bm{X}^T\bm{X}$. +% --- end paragraph admon --- + + + +% !split +\subsection{Interpretations and optimizing our parameters} + +% --- begin paragraph admon --- +\paragraph{} +The residuals $\bm{\epsilon}$ are in turn given by +\[ +\bm{\epsilon} = \bm{y}-\bm{\tilde{y}} = \bm{y}-\bm{X}\bm{\beta}, +\] +and with +\[ +\bm{X}^T\left( \bm{y}-\bm{X}\bm{\beta}\right)= 0, +\] +we have +\[ +\bm{X}^T\bm{\epsilon}=\bm{X}^T\left( \bm{y}-\bm{X}\bm{\beta}\right)= 0, +\] +meaning that the solution for $\bm{\beta}$ is the one which minimizes the residuals. Later we will link this with the maximum likelihood approach. +% --- end paragraph admon --- + + + + +Let us now return to our nuclear binding energies and simply code the above equations. + +% !split +\subsection{Own code for Ordinary Least Squares} + +It is rather straightforward to implement the matrix inversion and obtain the parameters $\bm{\beta}$. After having defined the matrix $\bm{X}$ we simply need to +write +\bpycod +# matrix inversion to find beta +beta = np.linalg.inv(X.T.dot(X)).dot(X.T).dot(Energies) +# and then make the prediction +ytilde = X @ beta +\epycod +Alternatively, you can use the least squares functionality in \textbf{Numpy} as +\bpycod +fit = np.linalg.lstsq(X, Energies, rcond =None)[0] +ytildenp = np.dot(fit,X.T) +\epycod + +And finally we plot our fit with and compare with data +\bpycod +Masses['Eapprox'] = ytilde +# Generate a plot comparing the experimental with the fitted values values. +fig, ax = plt.subplots() +ax.set_xlabel(r'$A = N + Z$') +ax.set_ylabel(r'$E_\mathrm{bind}\,/\mathrm{MeV}$') +ax.plot(Masses['A'], Masses['Ebinding'], alpha=0.7, lw=2, + label='Ame2016') +ax.plot(Masses['A'], Masses['Eapprox'], alpha=0.7, lw=2, c='m', + label='Fit') +ax.legend() +save_fig("Masses2016OLS") +plt.show() +\epycod + +% !split +\subsection{Adding error analysis and training set up} + +We can easily test our fit by computing the $R2$ score that we discussed in connection with the functionality of _Scikit_Learn_ in the introductory slides. +Since we are not using _Scikit-Learn here we can define our own $R2$ function as +\bpycod +def R2(y_data, y_model): + return 1 - np.sum((y_data - y_model) ** 2) / np.sum((y_data - np.mean(y_model)) ** 2) +\epycod +and we would be using it as +\bpycod +print(R2(Energies,ytilde)) +\epycod + +We can easily add our \textbf{MSE} score as +\bpycod +def MSE(y_data,y_model): + n = np.size(y_model) + return np.sum((y_data-y_model)**2)/n + +print(MSE(Energies,ytilde)) +\epycod +and finally the relative error as +\bpycod +def RelativeError(y_data,y_model): + return abs((y_data-y_model)/y_data) +print(RelativeError(Energies, ytilde)) +\epycod + + + + + +% !split +\subsection{The $\chi^2$ function} + +% --- begin paragraph admon --- +\paragraph{} + +Normally, the response (dependent or outcome) variable $y_i$ is the +outcome of a numerical experiment or another type of experiment and is +thus only an approximation to the true value. It is then always +accompanied by an error estimate, often limited to a statistical error +estimate given by the standard deviation discussed earlier. In the +discussion here we will treat $y_i$ as our exact value for the +response variable. + +Introducing the standard deviation $\sigma_i$ for each measurement +$y_i$, we define now the $\chi^2$ function (omitting the $1/n$ term) +as + +\[ +\chi^2(\bm{\beta})=\frac{1}{n}\sum_{i=0}^{n-1}\frac{\left(y_i-\tilde{y}_i\right)^2}{\sigma_i^2}=\frac{1}{n}\left\{\left(\bm{y}-\bm{\tilde{y}}\right)^T\frac{1}{\bm{\Sigma^2}}\left(\bm{y}-\bm{\tilde{y}}\right)\right\}, +\] +where the matrix $\bm{\Sigma}$ is a diagonal matrix with $\sigma_i$ as matrix elements. +% --- end paragraph admon --- + + + +% !split +\subsection{The $\chi^2$ function} + +% --- begin paragraph admon --- +\paragraph{} + +In order to find the parameters $\beta_i$ we will then minimize the spread of $\chi^2(\bm{\beta})$ by requiring +\[ +\frac{\partial \chi^2(\bm{\beta})}{\partial \beta_j} = \frac{\partial }{\partial \beta_j}\left[ \frac{1}{n}\sum_{i=0}^{n-1}\left(\frac{y_i-\beta_0x_{i,0}-\beta_1x_{i,1}-\beta_2x_{i,2}-\dots-\beta_{n-1}x_{i,n-1}}{\sigma_i}\right)^2\right]=0, +\] +which results in +\[ +\frac{\partial \chi^2(\bm{\beta})}{\partial \beta_j} = -\frac{2}{n}\left[ \sum_{i=0}^{n-1}\frac{x_{ij}}{\sigma_i}\left(\frac{y_i-\beta_0x_{i,0}-\beta_1x_{i,1}-\beta_2x_{i,2}-\dots-\beta_{n-1}x_{i,n-1}}{\sigma_i}\right)\right]=0, +\] +or in a matrix-vector form as +\[ +\frac{\partial \chi^2(\bm{\beta})}{\partial \bm{\beta}} = 0 = \bm{A}^T\left( \bm{b}-\bm{A}\bm{\beta}\right). +\] +where we have defined the matrix $\bm{A} =\bm{X}/\bm{\Sigma}$ with matrix elements $a_{ij} = x_{ij}/\sigma_i$ and the vector $\bm{b}$ with elements $b_i = y_i/\sigma_i$. +% --- end paragraph admon --- + + + +% !split +\subsection{The $\chi^2$ function} + +% --- begin paragraph admon --- +\paragraph{} + +We can rewrite +\[ +\frac{\partial \chi^2(\bm{\beta})}{\partial \bm{\beta}} = 0 = \bm{A}^T\left( \bm{b}-\bm{A}\bm{\beta}\right), +\] +as +\[ +\bm{A}^T\bm{b} = \bm{A}^T\bm{A}\bm{\beta}, +\] +and if the matrix $\bm{A}^T\bm{A}$ is invertible we have the solution +\[ +\bm{\beta} =\left(\bm{A}^T\bm{A}\right)^{-1}\bm{A}^T\bm{b}. +\] +% --- end paragraph admon --- + + + +% !split +\subsection{The $\chi^2$ function} + +% --- begin paragraph admon --- +\paragraph{} + +If we then introduce the matrix +\[ +\bm{H} = \left(\bm{A}^T\bm{A}\right)^{-1}, +\] +we have then the following expression for the parameters $\beta_j$ (the matrix elements of $\bm{H}$ are $h_{ij}$) +\[ +\beta_j = \sum_{k=0}^{p-1}h_{jk}\sum_{i=0}^{n-1}\frac{y_i}{\sigma_i}\frac{x_{ik}}{\sigma_i} = \sum_{k=0}^{p-1}h_{jk}\sum_{i=0}^{n-1}b_ia_{ik} +\] +We state without proof the expression for the uncertainty in the parameters $\beta_j$ as (we leave this as an exercise) +\[ +\sigma^2(\beta_j) = \sum_{i=0}^{n-1}\sigma_i^2\left( \frac{\partial \beta_j}{\partial y_i}\right)^2, +\] +resulting in +\[ +\sigma^2(\beta_j) = \left(\sum_{k=0}^{p-1}h_{jk}\sum_{i=0}^{n-1}a_{ik}\right)\left(\sum_{l=0}^{p-1}h_{jl}\sum_{m=0}^{n-1}a_{ml}\right) = h_{jj}! +\] +% --- end paragraph admon --- + + + +% !split +\subsection{The $\chi^2$ function} + +% --- begin paragraph admon --- +\paragraph{} +The first step here is to approximate the function $y$ with a first-order polynomial, that is we write +\[ +y=y(x) \rightarrow y(x_i) \approx \beta_0+\beta_1 x_i. +\] +By computing the derivatives of $\chi^2$ with respect to $\beta_0$ and $\beta_1$ show that these are given by +\[ +\frac{\partial \chi^2(\bm{\beta})}{\partial \beta_0} = -2\left[ \frac{1}{n}\sum_{i=0}^{n-1}\left(\frac{y_i-\beta_0-\beta_1x_{i}}{\sigma_i^2}\right)\right]=0, +\] +and +\[ +\frac{\partial \chi^2(\bm{\beta})}{\partial \beta_1} = -\frac{2}{n}\left[ \sum_{i=0}^{n-1}x_i\left(\frac{y_i-\beta_0-\beta_1x_{i}}{\sigma_i^2}\right)\right]=0. +\] +% --- end paragraph admon --- + + + +% !split +\subsection{The $\chi^2$ function} + +% --- begin paragraph admon --- +\paragraph{} + +For a linear fit (a first-order polynomial) we don't need to invert a matrix!! +Defining +\[ +\gamma = \sum_{i=0}^{n-1}\frac{1}{\sigma_i^2}, +\] + +\[ +\gamma_x = \sum_{i=0}^{n-1}\frac{x_{i}}{\sigma_i^2}, +\] + +\[ +\gamma_y = \sum_{i=0}^{n-1}\left(\frac{y_i}{\sigma_i^2}\right), +\] + +\[ +\gamma_{xx} = \sum_{i=0}^{n-1}\frac{x_ix_{i}}{\sigma_i^2}, +\] + +\[ +\gamma_{xy} = \sum_{i=0}^{n-1}\frac{y_ix_{i}}{\sigma_i^2}, +\] + +we obtain + +\[ +\beta_0 = \frac{\gamma_{xx}\gamma_y-\gamma_x\gamma_y}{\gamma\gamma_{xx}-\gamma_x^2}, +\] + +\[ +\beta_1 = \frac{\gamma_{xy}\gamma-\gamma_x\gamma_y}{\gamma\gamma_{xx}-\gamma_x^2}. +\] + +This approach (different linear and non-linear regression) suffers +often from both being underdetermined and overdetermined in the +unknown coefficients $\beta_i$. A better approach is to use the +Singular Value Decomposition (SVD) method discussed below. Or using +Lasso and Ridge regression. See below. +% --- end paragraph admon --- + + + + +% !split +\subsection{Fitting an Equation of State for Dense Nuclear Matter} + +Before we continue, let us introduce yet another example. We are going to fit the +nuclear equation of state using results from many-body calculations. +The equation of state we have made available here, as function of +density, has been derived using modern nucleon-nucleon potentials with +\href{{https://www.sciencedirect.com/science/article/pii/S0370157399001106}}{the addition of three-body +forces}. This +time the file is presented as a standard \textbf{csv} file. + +The beginning of the Python code here is similar to what you have seen before, +with the same initializations and declarations. We use also \textbf{pandas} +again, rather extensively in order to organize our data. + +The difference now is that we use \textbf{Scikit-Learn's} regression tools +instead of our own matrix inversion implementation. Furthermore, we +sneak in \textbf{Ridge} regression (to be discussed below) which includes a +hyperparameter $\lambda$, also to be explained below. + +% !split +\subsection{The code} + +\bpycod +# Common imports +import os +import numpy as np +import pandas as pd +import matplotlib.pyplot as plt +import matplotlib.pyplot as plt +import sklearn.linear_model as skl +from sklearn.metrics import mean_squared_error, r2_score, mean_absolute_error + +# Where to save the figures and data files +PROJECT_ROOT_DIR = "Results" +FIGURE_ID = "Results/FigureFiles" +DATA_ID = "DataFiles/" + +if not os.path.exists(PROJECT_ROOT_DIR): + os.mkdir(PROJECT_ROOT_DIR) + +if not os.path.exists(FIGURE_ID): + os.makedirs(FIGURE_ID) + +if not os.path.exists(DATA_ID): + os.makedirs(DATA_ID) + +def image_path(fig_id): + return os.path.join(FIGURE_ID, fig_id) + +def data_path(dat_id): + return os.path.join(DATA_ID, dat_id) + +def save_fig(fig_id): + plt.savefig(image_path(fig_id) + ".png", format='png') + +infile = open(data_path("EoS.csv"),'r') + +# Read the EoS data as csv file and organize the data into two arrays with density and energies +EoS = pd.read_csv(infile, names=('Density', 'Energy')) +EoS['Energy'] = pd.to_numeric(EoS['Energy'], errors='coerce') +EoS = EoS.dropna() +Energies = EoS['Energy'] +Density = EoS['Density'] +# The design matrix now as function of various polytrops +X = np.zeros((len(Density),4)) +X[:,3] = Density**(4.0/3.0) +X[:,2] = Density +X[:,1] = Density**(2.0/3.0) +X[:,0] = 1 + +# We use now Scikit-Learn's linear regressor and ridge regressor +# OLS part +clf = skl.LinearRegression().fit(X, Energies) +ytilde = clf.predict(X) +EoS['Eols'] = ytilde +# The mean squared error +print("Mean squared error: %.2f" % mean_squared_error(Energies, ytilde)) +# Explained variance score: 1 is perfect prediction +print('Variance score: %.2f' % r2_score(Energies, ytilde)) +# Mean absolute error +print('Mean absolute error: %.2f' % mean_absolute_error(Energies, ytilde)) +print(clf.coef_, clf.intercept_) + +# The Ridge regression with a hyperparameter lambda = 0.1 +_lambda = 0.1 +clf_ridge = skl.Ridge(alpha=_lambda).fit(X, Energies) +yridge = clf_ridge.predict(X) +EoS['Eridge'] = yridge +# The mean squared error +print("Mean squared error: %.2f" % mean_squared_error(Energies, yridge)) +# Explained variance score: 1 is perfect prediction +print('Variance score: %.2f' % r2_score(Energies, yridge)) +# Mean absolute error +print('Mean absolute error: %.2f' % mean_absolute_error(Energies, yridge)) +print(clf_ridge.coef_, clf_ridge.intercept_) + +fig, ax = plt.subplots() +ax.set_xlabel(r'$\rho[\mathrm{fm}^{-3}]$') +ax.set_ylabel(r'Energy per particle') +ax.plot(EoS['Density'], EoS['Energy'], alpha=0.7, lw=2, + label='Theoretical data') +ax.plot(EoS['Density'], EoS['Eols'], alpha=0.7, lw=2, c='m', + label='OLS') +ax.plot(EoS['Density'], EoS['Eridge'], alpha=0.7, lw=2, c='g', + label='Ridge $\lambda = 0.1$') +ax.legend() +save_fig("EoSfitting") +plt.show() +\epycod + +The above simple polynomial in density $\rho$ gives an excellent fit +to the data. Can you give an interpretation of the various powers of $\rho$? + +We note also that there is a small deviation between the +standard OLS and the Ridge regression at higher densities. We discuss this in more detail +below. + + +% !split +\subsection{Splitting our Data in Training and Test data} + +It is normal in essentially all Machine Learning studies to split the +data in a training set and a test set (sometimes also an additional +validation set). \textbf{Scikit-Learn} has an own function for this. There +is no explicit recipe for how much data should be included as training +data and say test data. An accepted rule of thumb is to use +approximately $2/3$ to $4/5$ of the data as training data. We will +postpone a discussion of this splitting to the end of these notes and +our discussion of the so-called \textbf{bias-variance} tradeoff. Here we +limit ourselves to repeat the above equation of state fitting example +but now splitting the data into a training set and a test set. + +\bpycod +import os +import numpy as np +import pandas as pd +import matplotlib.pyplot as plt +from sklearn.model_selection import train_test_split +# Where to save the figures and data files +PROJECT_ROOT_DIR = "Results" +FIGURE_ID = "Results/FigureFiles" +DATA_ID = "DataFiles/" + +if not os.path.exists(PROJECT_ROOT_DIR): + os.mkdir(PROJECT_ROOT_DIR) + +if not os.path.exists(FIGURE_ID): + os.makedirs(FIGURE_ID) + +if not os.path.exists(DATA_ID): + os.makedirs(DATA_ID) + +def image_path(fig_id): + return os.path.join(FIGURE_ID, fig_id) + +def data_path(dat_id): + return os.path.join(DATA_ID, dat_id) + +def save_fig(fig_id): + plt.savefig(image_path(fig_id) + ".png", format='png') + +def R2(y_data, y_model): + return 1 - np.sum((y_data - y_model) ** 2) / np.sum((y_data - np.mean(y_model)) ** 2) +def MSE(y_data,y_model): + n = np.size(y_model) + return np.sum((y_data-y_model)**2)/n + +infile = open(data_path("EoS.csv"),'r') + +# Read the EoS data as csv file and organized into two arrays with density and energies +EoS = pd.read_csv(infile, names=('Density', 'Energy')) +EoS['Energy'] = pd.to_numeric(EoS['Energy'], errors='coerce') +EoS = EoS.dropna() +Energies = EoS['Energy'] +Density = EoS['Density'] +# The design matrix now as function of various polytrops +X = np.zeros((len(Density),5)) +X[:,0] = 1 +X[:,1] = Density**(2.0/3.0) +X[:,2] = Density +X[:,3] = Density**(4.0/3.0) +X[:,4] = Density**(5.0/3.0) +# We split the data in test and training data +X_train, X_test, y_train, y_test = train_test_split(X, Energies, test_size=0.2) +# matrix inversion to find beta +beta = np.linalg.inv(X_train.T.dot(X_train)).dot(X_train.T).dot(y_train) +# and then make the prediction +ytilde = X_train @ beta +print("Training R2") +print(R2(y_train,ytilde)) +print("Training MSE") +print(MSE(y_train,ytilde)) +ypredict = X_test @ beta +print("Test R2") +print(R2(y_test,ypredict)) +print("Test MSE") +print(MSE(y_test,ypredict)) +\epycod + + +% !split +\subsection{The singular value decomposition} + + +% --- begin paragraph admon --- +\paragraph{} + +The examples we have looked at so far are cases where we normally can +invert the matrix $\bm{X}^T\bm{X}$. Using a polynomial expansion as we +did both for the masses and the fitting of the equation of state, +leads to row vectors of the design matrix which are essentially +orthogonal due to the polynomial character of our model. This may +however not the be case in general and a standard matrix inversion +algorithm based on say LU decomposition may lead to singularities. We will see an example of this below when we try to fit +the coupling constant of the widely used Ising model. +There is however a way to partially circumvent this problem and also gain some insight about the ordinary least squares approach. + +This is given by the \textbf{Singular Value Decomposition} algorithm, perhaps +the most powerful linear algebra algorithm. Let us look at a +different example where we may have problems with the standard matrix +inversion algorithm. Thereafter we dive into the math of the SVD. +% --- end paragraph admon --- + + + +% !split +\subsection{The Ising model} + +The one-dimensional Ising model with nearest neighbor interaction, no +external field and a constant coupling constant $J$ is given by + +\begin{align} + H = -J \sum_{k}^L s_k s_{k + 1}, +\end{align} + +where $s_i \in \{-1, 1\}$ and $s_{N + 1} = s_1$. The number of spins +in the system is determined by $L$. For the one-dimensional system +there is no phase transition. + +We will look at a system of $L = 40$ spins with a coupling constant of +$J = 1$. To get enough training data we will generate 10000 states +with their respective energies. + + +\bpycod +import numpy as np +import matplotlib.pyplot as plt +from mpl_toolkits.axes_grid1 import make_axes_locatable +import seaborn as sns +import scipy.linalg as scl +from sklearn.model_selection import train_test_split +import tqdm +sns.set(color_codes=True) +cmap_args=dict(vmin=-1., vmax=1., cmap='seismic') + +L = 40 +n = int(1e4) + +spins = np.random.choice([-1, 1], size=(n, L)) +J = 1.0 + +energies = np.zeros(n) + +for i in range(n): + energies[i] = - J * np.dot(spins[i], np.roll(spins[i], 1)) +\epycod + +Here we use ordinary least squares +regression to predict the energy for the nearest neighbor +one-dimensional Ising model on a ring, i.e., the endpoints wrap +around. We will use linear regression to fit a value for +the coupling constant to achieve this. + +% !split +\subsection{Reformulating the problem to suit regression} + +A more general form for the one-dimensional Ising model is + +\begin{align} + H = - \sum_j^L \sum_k^L s_j s_k J_{jk}. +\end{align} + +Here we allow for interactions beyond the nearest neighbors and a state dependent +coupling constant. This latter expression can be formulated as +a matrix-product +\begin{align} + \bm{H} = \bm{X} J, +\end{align} + +where $X_{jk} = s_j s_k$ and $J$ is a matrix which consists of the +elements $-J_{jk}$. This form of writing the energy fits perfectly +with the form utilized in linear regression, that is + +\begin{align} + \bm{y} = \bm{X}\bm{\beta} + \bm{\epsilon}, +\end{align} + +We split the data in training and test data as discussed in the previous example + +\bpycod +X = np.zeros((n, L ** 2)) +for i in range(n): + X[i] = np.outer(spins[i], spins[i]).ravel() +y = energies +X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2) +\epycod + +% !split +\subsection{Linear regression} + +In the ordinary least squares method we choose the cost function + +\begin{align} + C(\bm{X}, \bm{\beta})= \frac{1}{n}\left\{(\bm{X}\bm{\beta} - \bm{y})^T(\bm{X}\bm{\beta} - \bm{y})\right\}. +\end{align} + +We then find the extremal point of $C$ by taking the derivative with respect to $\bm{\beta}$ as discussed above. +This yields the expression for $\bm{\beta}$ to be + +\[ + \bm{\beta} = \frac{\bm{X}^T \bm{y}}{\bm{X}^T \bm{X}}, +\] + +which immediately imposes some requirements on $\bm{X}$ as there must exist +an inverse of $\bm{X}^T \bm{X}$. If the expression we are modeling contains an +intercept, i.e., a constant term, we must make sure that the +first column of $\bm{X}$ consists of $1$. We do this here + +\bpycod +X_train_own = np.concatenate( + (np.ones(len(X_train))[:, np.newaxis], X_train), + axis=1 +) +X_test_own = np.concatenate( + (np.ones(len(X_test))[:, np.newaxis], X_test), + axis=1 +) +\epycod + +\bpycod +def ols_inv(x: np.ndarray, y: np.ndarray) -> np.ndarray: + return scl.inv(x.T @ x) @ (x.T @ y) +beta = ols_inv(X_train_own, y_train) +\epycod + + +% !split +\subsection{Singular Value decomposition} + +Doing the inversion directly turns out to be a bad idea since the matrix +$\bm{X}^T\bm{X}$ is singular. An alternative approach is to use the \textbf{singular +value decomposition}. Using the definition of the Moore-Penrose +pseudoinverse we can write the equation for $\bm{\beta}$ as + +\[ + \bm{\beta} = \bm{X}^{+}\bm{y}, +\] + +where the pseudoinverse of $\bm{X}$ is given by + +\[ + \bm{X}^{+} = \frac{\bm{X}^T}{\bm{X}^T\bm{X}}. +\] + +Using singular value decomposition we can decompose the matrix $\bm{X} = \bm{U}\bm{\Sigma} \bm{V}^T$, +where $\bm{U}$ and $\bm{V}$ are orthogonal(unitary) matrices and $\bm{\Sigma}$ contains the singular values (more details below). +where $X^{+} = V\Sigma^{+} U^T$. This reduces the equation for +$\omega$ to +\begin{align} + \bm{\beta} = \bm{V}\bm{\Sigma}^{+} \bm{U}^T \bm{y}. +\end{align} + +Note that solving this equation by actually doing the pseudoinverse +(which is what we will do) is not a good idea as this operation scales +as $\mathcal{O}(n^3)$, where $n$ is the number of elements in a +general matrix. Instead, doing $QR$-factorization and solving the +linear system as an equation would reduce this down to +$\mathcal{O}(n^2)$ operations. + + +\bpycod +def ols_svd(x: np.ndarray, y: np.ndarray) -> np.ndarray: + u, s, v = scl.svd(x) + return v.T @ scl.pinv(scl.diagsvd(s, u.shape[0], v.shape[0])) @ u.T @ y +\epycod + +\bpycod +beta = ols_svd(X_train_own,y_train) +\epycod + +When extracting the $J$-matrix we need to make sure that we remove the intercept, as is done here + +\bpycod +J = beta[1:].reshape(L, L) +\epycod + +A way of looking at the coefficients in $J$ is to plot the matrices as images. + + +\bpycod +fig = plt.figure(figsize=(20, 14)) +im = plt.imshow(J, **cmap_args) +plt.title("OLS", fontsize=18) +plt.xticks(fontsize=18) +plt.yticks(fontsize=18) +cb = fig.colorbar(im) +cb.ax.set_yticklabels(cb.ax.get_yticklabels(), fontsize=18) +plt.show() +\epycod +It is interesting to note that OLS +considers both $J_{j, j + 1} = -0.5$ and $J_{j, j - 1} = -0.5$ as +valid matrix elements for $J$. +In our discussion below on hyperparameters and Ridge and Lasso regression we will see that +this problem can be removed, partly and only with Lasso regression. + +In this case our matrix inversion was actually possible. The obvious question now is what is the mathematics behind the SVD? + + +% !split +\subsection{Linear Regression Problems} + +One of the typical problems we encounter with linear regression, in particular +when the matrix $\bm{X}$ (our so-called design matrix) is high-dimensional, +are problems with near singular or singular matrices. The column vectors of $\bm{X}$ +may be linearly dependent, normally referred to as super-collinearity. +This means that the matrix may be rank deficient and it is basically impossible to +to model the data using linear regression. As an example, consider the matrix +\begin{align*} +\mathbf{X} & = \left[ +\begin{array}{rrr} +1 & -1 & 2 +\\ +1 & 0 & 1 +\\ +1 & 2 & -1 +\\ +1 & 1 & 0 +\end{array} \right] +\end{align*} + +The columns of $\bm{X}$ are linearly dependent. We see this easily since the +the first column is the row-wise sum of the other two columns. The rank (more correct, +the column rank) of a matrix is the dimension of the space spanned by the +column vectors. Hence, the rank of $\mathbf{X}$ is equal to the number +of linearly independent columns. In this particular case the matrix has rank 2. + +Super-collinearity of an $(n \times p)$-dimensional design matrix $\mathbf{X}$ implies +that the inverse of the matrix $\bm{X}^T\bm{x}$ (the matrix we need to invert to solve the linear regression equations) is non-invertible. If we have a square matrix that does not have an inverse, we say this matrix singular. The example here demonstrates this +\begin{align*} +\bm{X} & = \left[ +\begin{array}{rr} +1 & -1 +\\ +1 & -1 +\end{array} \right]. +\end{align*} +We see easily that $\mbox{det}(\bm{X}) = x_{11} x_{22} - x_{12} x_{21} = 1 \times (-1) - 1 \times (-1) = 0$. Hence, $\mathbf{X}$ is singular and its inverse is undefined. +This is equivalent to saying that the matrix $\bm{X}$ has at least an eigenvalue which is zero. + + +% !split +\subsection{Fixing the singularity} + +If our design matrix $\bm{X}$ which enters the linear regression problem +\begin{align} +\bm{\beta} & = (\bm{X}^{T} \bm{X})^{-1} \bm{X}^{T} \bm{y}, +\end{align} +has linearly dependent column vectors, we will not be able to compute the inverse +of $\bm{X}^T\bm{X}$ and we cannot find the parameters (estimators) $\beta_i$. +The estimators are only well-defined if $(\bm{X}^{T}\bm{X})^{-1}$ exits. +This is more likely to happen when the matrix $\bm{X}$ is high-dimensional. In this case it is likely to encounter a situation where +the regression parameters $\beta_i$ cannot be estimated. + +A cheap \emph{ad hoc} approach is simply to add a small diagonal component to the matrix to invert, that is we change +\[ +\bm{X}^{T} \bm{X} \rightarrow \bm{X}^{T} \bm{X}+\lambda \bm{I}, +\] +where $\bm{I}$ is the identity matrix. When we discuss \textbf{Ridge} regression this is actually what we end up evaluating. The parameter $\lambda$ is called a hyperparameter. More about this later. + + + +% !split +\subsection{Basic math of the SVD} + + +From standard linear algebra we know that a square matrix $\bm{X}$ can be diagonalized if and only it is +a so-called \href{{https://en.wikipedia.org/wiki/Normal_matrix}}{normal matrix}, that is if $\bm{X}\in {\mathbb{R}}^{n\times n}$ +we have $\bm{X}\bm{X}^T=\bm{X}^T\bm{X}$ or if $\bm{X}\in {\mathbb{C}}^{n\times n}$ we have $\bm{X}\bm{X}^{\dagger}=\bm{X}^{\dagger}\bm{X}$. +The matrix has then a set of eigenpairs + +\[ +(\lambda_1,\bm{u}_1),\dots, (\lambda_n,\bm{u}_n), +and the eigenvalues are given by the diagonal matrix +\[ +\bm{\Sigma}=\mathrm{Diag}(\lambda_1, \dots,\lambda_n). +\] +The matrix $\bm{X}$ can be written in terms of an orthogonal/unitary transformation $\bm{U}$ +\[ +\bm{X} = \bm{U}\bm{\Sigma}\bm{V}^T, +\] +with $\bm{U}\bm{U}^T=\bm{I}$ or $\bm{U}\bm{U}^{\dagger}=\bm{I}$. + +Not all square matrices are diagonalizable. A matrix like the one discussed above +\[ +\bm{X} = \begin{bmatrix} +1& -1 \\ +1& -1\\ +\end{bmatrix} +\] +is not diagonalizable, it is a so-called \href{{https://en.wikipedia.org/wiki/Defective_matrix}}{defective matrix}. It is easy to see that the condition +$\bm{X}\bm{X}^T=\bm{X}^T\bm{X}$ is not fulfilled. + + +% !split +\subsection{The SVD, a Fantastic Algorithm} + + +However, and this is the strength of the SVD algorithm, any general +matrix $\bm{X}$ can be decomposed in terms of a diagonal matrix and +two orthogonal/unitary matrices. The \href{{https://en.wikipedia.org/wiki/Singular_value_decomposition}}{Singular Value Decompostion +(SVD) theorem} +states that a general $m\times n$ matrix $\bm{X}$ can be written in +terms of a diagonal matrix $\bm{\Sigma}$ of dimensionality $n\times n$ +and two orthognal matrices $\bm{U}$ and $\bm{V}$, where the first has +dimensionality $m \times m$ and the last dimensionality $n\times n$. +We have then + +\[ +\bm{X} = \bm{U}\bm{\Sigma}\bm{V}^T +\] + +As an example, the above defective matrix can be decomposed as + +\[ +\bm{X} = \frac{1}{\sqrt{2}}\begin{bmatrix} 1& 1 \\ 1& -1\\ \end{bmatrix} \begin{bmatrix} 2& 0 \\ 0& 0\\ \end{bmatrix} \frac{1}{\sqrt{2}}\begin{bmatrix} 1& -1 \\ 1& 1\\ \end{bmatrix}=\bm{U}\bm{\Sigma}\bm{V}^T, +\] + +with eigenvalues $\sigma_1=2$ and $\sigma_2=0$. +The SVD exits always! + + +% !split +\subsection{Another Example} + +Consider the following matrix which can be SVD decomposed as + +\[ +\bm{X} = \frac{1}{15}\begin{bmatrix} 14 & 2\\ 4 & 22\\ 16 & 13\end{matrix}=\frac{1}{3}\begin{bmatrix} 1& 2 & 2 \\ 2& -1 & 1\\ 2 & 1& -2\end{bmatrix} \begin{bmatrix} 2& 0 \\ 0& 1\\ 0 & 0\end{bmatrix}\frac{1}{5}\begin{bmatrix} 3& 4 \\ 4& -3\end{bmatrix}=\bm{U}\bm{\Sigma}\bm{V}^T. +\] + +This is a $3\times 2$ matrix which is decomposed in terms of a +$3\times 3$ matrix $\bm{U}$, and a $2\times 2$ matrix $\bm{V}$. It is easy to see +that $\bm{U}$ and $\bm{V}$ are orthogonal (how?). + +And the SVD +decomposition (singular values) gives eigenvalues +$\sigma_i\geq\sigma_{i+1}$ for all $i$ and for dimensions larger than $i=2$, the +eigenvalues (singular values) are zero. + +In the general case, where our design matrix $\bm{X}$ has dimension +$n\times p$, the matrix is thus decomposed into an $n\times n$ +orthogonal matrix $\bm{U}$, a $p\times p$ orthogonal matrix $\bm{V}$ +and a diagonal matrix $\bm{\Sigma}$ with $r=\mathrm{min}(n,p)$ +singular values $\sigma_i\lg 0$ on the main diagonal and zeros filling +the rest of the matrix. There are at most $p$ singular values +assuming that $n > p$. In our regression examples for the nuclear +masses and the equation of state this is indeed the case, while for +the Ising model we have $p > n$. These are often cases that lead to +near singular or singular matrices. + +The columns of $\bm{U}$ are called the left singular vectors while the columns of $\bm{V}$ are the right singular vectors. + +% !split +\subsection{Economy-size SVD} + +If we assume that $n > p$, then our matrix $\bm{U}$ has dimension $n +\times n$. The last $n-p$ columns of $\bm{U}$ become however +irrelevant in our calculations since they are multiplied with the +zeros in $\bm{\Sigma}$. + +The economy-size decomposition removes extra rows or columns of zeros +from the diagonal matrix of singular values, $\bm{\Sigma}$, along with the columns +in either $\bm{U}$ or $\bm{V}$ that multiply those zeros in the expression. +Removing these zeros and columns can improve execution time +and reduce storage requirements without compromising the accuracy of +the decomposition. + +If $n > p$, we keep only the first $p$ columns of $\bm{U}$ and $\bm{\Sigma}$ has dimension $p\times p$. +If $p > n$, then only the first $n$ columns of $\bm{V}$ are computed and $\bm{\Sigma}$ has dimension $n\times n$. +The $n=p$ case is obvious, we retain the full SVD. +In general the economy-size SVD leads to less FLOPS and still conserving the desired accuracy. + +% !split +\subsection{Mathematical Properties} + +There are several interesting mathematical properties which will be +relevant when we are going to discuss the differences between say +ordinary least squares (OLS) and \textbf{Ridge} regression. + +We have from OLS that the parameters of the linear approximation are given by +\[ +\bm{\tilde{y}} = \bm{X}\bm{\beta} = \bm{X}\left(\bm{X}^T\bm{X}\right)^{-1}\bm{X}^T\bm{y}. +\] + +The matrix to invert can be rewritten in terms of our SVD decomposition as + +\[ +\bm{X}^T\bm{X} = \bm{V}\bm{\Sigma}^T\bm{U}^T\bm{U}\bm{\Sigma}\bm{V}^T. +\] +Using the orthogonality properties of $\bm{U}$ we have + +\[ +\bm{X}^T\bm{X} = \bm{V}\bm{\Sigma}^T\bm{\Sigma}\bm{V}^T = \bm{V}\bm{D}\bm{V}^T, +\] +with $\bm{D}$ being a diagonal matrix with values along the diagonal given by the singular values squared. + +This means that +\[ +(\bm{X}^T\bm{X})\bm{V} = \bm{V}\bm{D}, +\] +that is the eigenvectors of $(\bm{X}^T\bm{X})$ are given by the columns of the right singular matrix of $\bm{X}$ and the eigenvalues are the squared singular values. It is easy to show (show this) that +\[ +(\bm{X}\bm{X}^T)\bm{U} = \bm{U}\bm{D}, +\] +that is, the eigenvectors of $(\bm{X}\bm{X})^T$ are the columns of the left singular matrix and the eigenvalues are the same. + +Going back to our OLS equation we have +\[ +\bm{X}\bm{\beta} = \bm{X}\left(\bm{V}\bm{D}\bm{V}^T \right)^{-1}\bm{X}^T\bm{y}=\bm{U\Sigma V^T}\left(\bm{V}\bm{D}\bm{V}^T \right)^{-1}(\bm{U\Sigma V^T})^T\bm{y}=\bm{U}\bm{U}^T\bm{y}. +\] +We will come back to this expression when we discuss Ridge regression. + + +% !split +\subsection{Ridge and LASSO Regression} + +Let us remind ourselves about the expression for the standard Mean Squared Error (MSE) which we used to define our cost function and the equations for the ordinary least squares (OLS) method, that is +our optimization problem is +\[ +{\displaystyle \min_{\bm{\beta}\in {\mathbb{R}}^{p}}}\frac{1}{n}\left\{\left(\bm{y}-\bm{X}\bm{\beta}\right)^T\left(\bm{y}-\bm{X}\bm{\beta}\right)\right\}. +\] +or we can state it as +\[ +{\displaystyle \min_{\bm{\beta}\in +{\mathbb{R}}^{p}}}\frac{1}{n}\sum_{i=0}^{n-1}\left(y_i-\tilde{y}_i\right)^2=\frac{1}{n}\vert\vert \bm{y}-\bm{X}\bm{\beta}\vert\vert_2^2, +\] +where we have used the definition of a norm-2 vector, that is +\[ +\vert\vert \bm{x}\vert\vert_2 = \sqrt{\sum_i x_i^2}. +\] + +By minimizing the above equation with respect to the parameters +$\bm{\beta}$ we could then obtain an analytical expression for the +parameters $\bm{\beta}$. We can add a regularization parameter $\lambda$ by +defining a new cost function to be optimized, that is + +\[ +{\displaystyle \min_{\bm{\beta}\in +{\mathbb{R}}^{p}}}\frac{1}{n}\vert\vert \bm{y}-\bm{X}\bm{\beta}\vert\vert_2^2+\lambda\vert\vert \bm{\beta}\vert\vert_2^2 +\] + +which leads to the Ridge regression minimization problem where we +require that $\vert\vert \bm{\beta}\vert\vert_2^2\le t$, where $t$ is +a finite number larger than zero. By defining + +\[ +C(\bm{X},\bm{\beta})=\frac{1}{n}\vert\vert \bm{y}-\bm{X}\bm{\beta}\vert\vert_2^2+\lambda\vert\vert \bm{\beta}\vert\vert_1, +\] + +we have a new optimization equation +\[ +{\displaystyle \min_{\bm{\beta}\in +{\mathbb{R}}^{p}}}\frac{1}{n}\vert\vert \bm{y}-\bm{X}\bm{\beta}\vert\vert_2^2+\lambda\vert\vert \bm{\beta}\vert\vert_1 +\] +which leads to Lasso regression. Lasso stands for least absolute shrinkage and selection operator. + +Here we have defined the norm-1 as +\[ +\vert\vert \bm{x}\vert\vert_1 = \sum_i \vert x_i\vert. +\] + + +% !split +\subsection{More on Ridge Regression} + +Using the matrix-vector expression for Ridge regression, + +\[ +C(\bm{X},\bm{\beta})=\frac{1}{n}\left\{(\bm{y}-\bm{X}\bm{\beta})^T(\bm{y}-\bm{X}\bm{\beta})\right\}+\lambda\bm{\beta}^T\bm{\beta}, +\] + +by taking the derivatives with respect to $\bm{\beta}$ we obtain then +a slightly modified matrix inversion problem which for finite values +of $\lambda$ does not suffer from singularity problems. We obtain + +\[ +\bm{\beta}^{\mathrm{Ridge}} = \left(\bm{X}^T\bm{X}+\lambda\bm{I}\right)^{-1}\bm{X}^T\bm{y}, +\] + +with $\bm{I}$ being a $p\times p$ identity matrix with the constraint that + +\[ +\sum_{i=0}^{p-1} \beta_i^2 \leq t, +\] + +with $t$ a finite positive number. + +We see that Ridge regression is nothing but the standard +OLS with a modified diagonal term added to $\bm{X}^T\bm{X}$. The +consequences, in particular for our discussion of the bias-variance +are rather interesting. + +Furthermore, if we use the result above in terms of the SVD decomposition (our analysis was done for the OLS method), we had +\[ +(\bm{X}\bm{X}^T)\bm{U} = \bm{U}\bm{D}. +\] + +We can analyse the OLS solutions in terms of the eigenvectors (the columns) of the right singular value matrix $\bm{U}$ as +\[ +\bm{X}\bm{\beta} = \bm{X}\left(\bm{V}\bm{D}\bm{V}^T \right)^{-1}\bm{X}^T\bm{y}=\bm{U\Sigma V^T}\left(\bm{V}\bm{D}\bm{V}^T \right)^{-1}(\bm{U\Sigma V^T})^T\bm{y}=\bm{U}\bm{U}^T\bm{y} +\] + + +For Ridge regression this becomes + +\[ +\bm{X}\bm{\beta}^{\mathrm{Ridge}} = \bm{U\Sigma V^T}\left(\bm{V}\bm{D}\bm{V}^T+\lambda\bm{I} \right)^{-1}(\bm{U\Sigma V^T})^T\bm{y}=\sum_{j=0}^{p-1}\bm{u}_j\bm{u}_j^T\frac{\sigma_j^2}{\sigma_j^2+\lambda}\bm{y}, +\] + +with the vectors $\bm{u}_j$ being the columns of $\bm{U}$. + +% !split +\subsection{Interpreting the Ridge results} + +Since $\lambda \geq 0$, it means that compared to OLS, we have + +\[ +\frac{\sigma_j^2}{\sigma_j^2+\lambda} \leq 1. +\] + +Ridge regression finds the coordinates of $\bm{y}$ with respect to the +orthonormal basis $\bm{U}$, it then shrinks the coordinates by +$\frac{\sigma_j^2}{\sigma_j^2+\lambda}$. Recall that the SVD has +eigenvalues ordered in a descending way, that is $\sigma_i \geq +\sigma_{i+1}$. + +For small eigenvalues $\sigma_i$ it means that their contributions become less important, a fact which can be used to reduce the number of degrees of freedom. +Actually, calculating the variance of $\bm{X}\bm{v}_j$ shows that this quantity is equal to $\sigma_j^2/n$. +With a parameter $\lambda$ we can thus shrink the role of specific parameters. + + +% !split +\subsection{More interpretations} + +For the sake of simplicity, let us assume that the design matrix is orthonormal, that is + +\[ +\bm{X}^T\bm{X}=(\bm{X}^T\bm{X})^{-1} =\bm{I}. +\] + +In this case the standard OLS results in +\[ +\bm{\beta}^{\mathrm{OLS}} = \bm{X}^T\bm{y}=\sum_{i=0}^{p-1}\bm{u}_j\bm{u}_j^T\bm{y}, +\] + +and + +\[ +\bm{\beta}^{\mathrm{Ridge}} = \left(\bm{I}+\lambda\bm{I}\right)^{-1}\bm{X}^T\bm{y}=\left(1+\lambda\right)^{-1}\bm{\beta}^{\mathrm{OLS}}, +\] + +that is the Ridge estimator scales the OLS estimator by the inverse of a factor $1+\lambda$, and +the Ridge estimator converges to zero when the hyperparameter goes to +infinity. + +We will come back to more interpreations after we have gone through some of the statistical analysis part. + +For more discussions of Ridge and Lasso regression, \href{{https://arxiv.org/abs/1509.09169}}{Wessel van Wieringen's} article is highly recommended. +Similarly, \href{{https://arxiv.org/abs/1803.08823}}{Mehta et al's article} is also recommended. + +% !split +\subsection{Where are we going?} + +Before we proceed, we need to rethink what we have been doing. In our +eager to fit the data, we have omitted several important elements in +our regression analysis. In what follows we will +\begin{enumerate} +\item look at statistical properties, including a discussion of mean values, variance and the so-called bias-variance tradeoff + +\item introduce resampling techniques like cross-validation, bootstrapping and jackknife and more +\end{enumerate} + +\noindent +This will allow us to link the standard linear algebra methods we have discussed above to a statistical interpretation of the methods. + + + + + +% !split +\subsection{Resampling methods} + +% --- begin paragraph admon --- +\paragraph{} +Resampling methods are an indispensable tool in modern +statistics. They involve repeatedly drawing samples from a training +set and refitting a model of interest on each sample in order to +obtain additional information about the fitted model. For example, in +order to estimate the variability of a linear regression fit, we can +repeatedly draw different samples from the training data, fit a linear +regression to each new sample, and then examine the extent to which +the resulting fits differ. Such an approach may allow us to obtain +information that would not be available from fitting the model only +once using the original training sample. +% --- end paragraph admon --- + + + +% !split +\subsection{Resampling approaches can be computationally expensive} + +% --- begin paragraph admon --- +\paragraph{} + +Resampling approaches can be computationally expensive, because they +involve fitting the same statistical method multiple times using +different subsets of the training data. However, due to recent +advances in computing power, the computational requirements of +resampling methods generally are not prohibitive. In this chapter, we +discuss two of the most commonly used resampling methods, +cross-validation and the bootstrap. Both methods are important tools +in the practical application of many statistical learning +procedures. For example, cross-validation can be used to estimate the +test error associated with a given statistical learning method in +order to evaluate its performance, or to select the appropriate level +of flexibility. The process of evaluating a model’s performance is +known as model assessment, whereas the process of selecting the proper +level of flexibility for a model is known as model selection. The +bootstrap is widely used. +% --- end paragraph admon --- + + + +% !split +\subsection{Why resampling methods ?} + +% --- begin paragraph admon --- +\paragraph{Statistical analysis.} + +\begin{itemize} +\item Our simulations can be treated as \emph{computer experiments}. This is particularly the case for Monte Carlo methods + +\item The results can be analysed with the same statistical tools as we would use analysing experimental data. + +\item As in all experiments, we are looking for expectation values and an estimate of how accurate they are, i.e., possible sources for errors. +\end{itemize} + +\noindent +% --- end paragraph admon --- + + + +% !split +\subsection{Statistical analysis} + +% --- begin paragraph admon --- +\paragraph{} + +\begin{itemize} +\item As in other experiments, many numerical experiments have two classes of errors: +\begin{itemize} + + \item Statistical errors + + \item Systematical errors + +\end{itemize} + +\noindent +\item Statistical errors can be estimated using standard tools from statistics + +\item Systematical errors are method specific and must be treated differently from case to case. +\end{itemize} + +\noindent +% --- end paragraph admon --- + + + +% !split +\subsection{Statistics} + +% --- begin paragraph admon --- +\paragraph{} +The \emph{probability distribution function (PDF)} is a function +$p(x)$ on the domain which, in the discrete case, gives us the +probability or relative frequency with which these values of $X$ occur: +\[ +p(x) = \mathrm{prob}(X=x) +\] +In the continuous case, the PDF does not directly depict the +actual probability. Instead we define the probability for the +stochastic variable to assume any value on an infinitesimal interval +around $x$ to be $p(x)dx$. The continuous function $p(x)$ then gives us +the \emph{density} of the probability rather than the probability +itself. The probability for a stochastic variable to assume any value +on a non-infinitesimal interval $[a,\,b]$ is then just the integral: +\[ +\mathrm{prob}(a\leq X\leq b) = \int_a^b p(x)dx +\] +Qualitatively speaking, a stochastic variable represents the values of +numbers chosen as if by chance from some specified PDF so that the +selection of a large set of these numbers reproduces this PDF. +% --- end paragraph admon --- + + + + +% !split +\subsection{Statistics, moments} + +% --- begin paragraph admon --- +\paragraph{} +A particularly useful class of special expectation values are the +\emph{moments}. The $n$-th moment of the PDF $p$ is defined as +follows: +\[ +\langle x^n\rangle \equiv \int\! x^n p(x)\,dx +\] +The zero-th moment $\langle 1\rangle$ is just the normalization condition of +$p$. The first moment, $\langle x\rangle$, is called the \emph{mean} of $p$ +and often denoted by the letter $\mu$: +\[ +\langle x\rangle = \mu \equiv \int\! x p(x)\,dx +\] +% --- end paragraph admon --- + + + +% !split +\subsection{Statistics, central moments} + +% --- begin paragraph admon --- +\paragraph{} +A special version of the moments is the set of \emph{central moments}, +the n-th central moment defined as: +\[ +\langle (x-\langle x \rangle )^n\rangle \equiv \int\! (x-\langle x\rangle)^n p(x)\,dx +\] +The zero-th and first central moments are both trivial, equal $1$ and +$0$, respectively. But the second central moment, known as the +\emph{variance} of $p$, is of particular interest. For the stochastic +variable $X$, the variance is denoted as $\sigma^2_X$ or $\mathrm{var}(X)$: +\begin{align} +\sigma^2_X\ \ =\ \ \mathrm{var}(X) & = \langle (x-\langle x\rangle)^2\rangle = +\int\! (x-\langle x\rangle)^2 p(x)\,dx\\ +& = \int\! \left(x^2 - 2 x \langle x\rangle^{2} + + \langle x\rangle^2\right)p(x)\,dx\\ +& = \langle x^2\rangle - 2 \langle x\rangle\langle x\rangle + \langle x\rangle^2\\ +& = \langle x^2\rangle - \langle x\rangle^2 +\end{align} +The square root of the variance, $\sigma =\sqrt{\langle (x-\langle x\rangle)^2\rangle}$ is called the \emph{standard deviation} of $p$. It is clearly just the RMS (root-mean-square) +value of the deviation of the PDF from its mean value, interpreted +qualitatively as the \emph{spread} of $p$ around its mean. +% --- end paragraph admon --- + + + +% !split +\subsection{Statistics, covariance} + +% --- begin paragraph admon --- +\paragraph{} +Another important quantity is the so called covariance, a variant of +the above defined variance. Consider again the set $\{X_i\}$ of $n$ +stochastic variables (not necessarily uncorrelated) with the +multivariate PDF $P(x_1,\dots,x_n)$. The \emph{covariance} of two +of the stochastic variables, $X_i$ and $X_j$, is defined as follows: +\begin{align} +\mathrm{cov}(X_i,\,X_j) &\equiv \langle (x_i-\langle x_i\rangle)(x_j-\langle x_j\rangle)\rangle +\nonumber\\ +&= +\int\!\cdots\!\int\!(x_i-\langle x_i \rangle)(x_j-\langle x_j \rangle)\, +P(x_1,\dots,x_n)\,dx_1\dots dx_n +\label{eq:def_covariance} +\end{align} +with +\[ +\langle x_i\rangle = +\int\!\cdots\!\int\!x_i\,P(x_1,\dots,x_n)\,dx_1\dots dx_n +\] +% --- end paragraph admon --- + + + +% !split +\subsection{Statistics, more covariance} + +% --- begin paragraph admon --- +\paragraph{} +If we consider the above covariance as a matrix $C_{ij}=\mathrm{cov}(X_i,\,X_j)$, then the diagonal elements are just the familiar +variances, $C_{ii} = \mathrm{cov}(X_i,\,X_i) = \mathrm{var}(X_i)$. It turns out that +all the off-diagonal elements are zero if the stochastic variables are +uncorrelated. This is easy to show, keeping in mind the linearity of +the expectation value. Consider the stochastic variables $X_i$ and +$X_j$, ($i\neq j$): +\begin{align} +\mathrm{cov}(X_i,\,X_j) &= \langle(x_i-\langle x_i\rangle)(x_j-\langle x_j\rangle)\rangle\\ +&=\langle x_i x_j - x_i\langle x_j\rangle - \langle x_i\rangle x_j + \langle x_i\rangle\langle x_j\rangle\rangle \\ +&=\langle x_i x_j\rangle - \langle x_i\langle x_j\rangle\rangle - \langle \langle x_i\rangle x_j\rangle + +\langle \langle x_i\rangle\langle x_j\rangle\rangle\\ +&=\langle x_i x_j\rangle - \langle x_i\rangle\langle x_j\rangle - \langle x_i\rangle\langle x_j\rangle + +\langle x_i\rangle\langle x_j\rangle\\ +&=\langle x_i x_j\rangle - \langle x_i\rangle\langle x_j\rangle +\end{align} +% --- end paragraph admon --- + + + + + +% !split +\subsection{Statistics, independent variables} + +% --- begin paragraph admon --- +\paragraph{} +If $X_i$ and $X_j$ are independent, we get +$\langle x_i x_j\rangle =\langle x_i\rangle\langle x_j\rangle$, resulting in $\mathrm{cov}(X_i, X_j) = 0\ \ (i\neq j)$. + +Also useful for us is the covariance of linear combinations of +stochastic variables. Let $\{X_i\}$ and $\{Y_i\}$ be two sets of +stochastic variables. Let also $\{a_i\}$ and $\{b_i\}$ be two sets of +scalars. Consider the linear combination: +\[ +U = \sum_i a_i X_i \qquad V = \sum_j b_j Y_j +\] +By the linearity of the expectation value +\[ +\mathrm{cov}(U, V) = \sum_{i,j}a_i b_j \mathrm{cov}(X_i, Y_j) +\] +% --- end paragraph admon --- + + + +% !split +\subsection{Statistics, more variance} + +% --- begin paragraph admon --- +\paragraph{} +Now, since the variance is just $\mathrm{var}(X_i) = \mathrm{cov}(X_i, X_i)$, we get +the variance of the linear combination $U = \sum_i a_i X_i$: +\begin{equation} +\mathrm{var}(U) = \sum_{i,j}a_i a_j \mathrm{cov}(X_i, X_j) +\label{eq:variance_linear_combination} +\end{equation} +And in the special case when the stochastic variables are +uncorrelated, the off-diagonal elements of the covariance are as we +know zero, resulting in: +\[ +\mathrm{var}(U) = \sum_i a_i^2 \mathrm{cov}(X_i, X_i) = \sum_i a_i^2 \mathrm{var}(X_i) +\] +\[ +\mathrm{var}(\sum_i a_i X_i) = \sum_i a_i^2 \mathrm{var}(X_i) +\] +which will become very useful in our study of the error in the mean +value of a set of measurements. +% --- end paragraph admon --- + + + +% !split +\subsection{Statistics and stochastic processes} + +% --- begin paragraph admon --- +\paragraph{} +A \emph{stochastic process} is a process that produces sequentially a +chain of values: +\[ +\{x_1, x_2,\dots\,x_k,\dots\}. +\] +We will call these +values our \emph{measurements} and the entire set as our measured +\emph{sample}. The action of measuring all the elements of a sample +we will call a stochastic \emph{experiment} since, operationally, +they are often associated with results of empirical observation of +some physical or mathematical phenomena; precisely an experiment. We +assume that these values are distributed according to some +PDF $p_X^{\phantom X}(x)$, where $X$ is just the formal symbol for the +stochastic variable whose PDF is $p_X^{\phantom X}(x)$. Instead of +trying to determine the full distribution $p$ we are often only +interested in finding the few lowest moments, like the mean +$\mu_X^{\phantom X}$ and the variance $\sigma_X^{\phantom X}$. +% --- end paragraph admon --- + + + + +% !split +\subsection{Statistics and sample variables} + +% --- begin paragraph admon --- +\paragraph{} +In practical situations a sample is always of finite size. Let that +size be $n$. The expectation value of a sample, the \emph{sample mean}, is then defined as follows: +\[ +\bar{x}_n \equiv \frac{1}{n}\sum_{k=1}^n x_k +\] +The \emph{sample variance} is: +\[ +\mathrm{var}(x) \equiv \frac{1}{n}\sum_{k=1}^n (x_k - \bar{x}_n)^2 +\] +its square root being the \emph{standard deviation of the sample}. The +\emph{sample covariance} is: +\[ +\mathrm{cov}(x)\equiv\frac{1}{n}\sum_{kl}(x_k - \bar{x}_n)(x_l - \bar{x}_n) +\] +% --- end paragraph admon --- + + + +% !split +\subsection{Statistics, sample variance and covariance} + +% --- begin paragraph admon --- +\paragraph{} +Note that the sample variance is the sample covariance without the +cross terms. In a similar manner as the covariance in Eq.~(\ref{eq:def_covariance}) is a measure of the correlation between +two stochastic variables, the above defined sample covariance is a +measure of the sequential correlation between succeeding measurements +of a sample. + +These quantities, being known experimental values, differ +significantly from and must not be confused with the similarly named +quantities for stochastic variables, mean $\mu_X$, variance $\mathrm{var}(X)$ +and covariance $\mathrm{cov}(X,Y)$. +% --- end paragraph admon --- + + + +% !split +\subsection{Statistics, law of large numbers} + +% --- begin paragraph admon --- +\paragraph{} +The law of large numbers +states that as the size of our sample grows to infinity, the sample +mean approaches the true mean $\mu_X^{\phantom X}$ of the chosen PDF: +\[ +\lim_{n\to\infty}\bar{x}_n = \mu_X^{\phantom X} +\] +The sample mean $\bar{x}_n$ works therefore as an estimate of the true +mean $\mu_X^{\phantom X}$. + +What we need to find out is how good an approximation $\bar{x}_n$ is to +$\mu_X^{\phantom X}$. In any stochastic measurement, an estimated +mean is of no use to us without a measure of its error. A quantity +that tells us how well we can reproduce it in another experiment. We +are therefore interested in the PDF of the sample mean itself. Its +standard deviation will be a measure of the spread of sample means, +and we will simply call it the \emph{error} of the sample mean, or +just sample error, and denote it by $\mathrm{err}_X^{\phantom X}$. In +practice, we will only be able to produce an \emph{estimate} of the +sample error since the exact value would require the knowledge of the +true PDFs behind, which we usually do not have. +% --- end paragraph admon --- + + + + +% !split +\subsection{Statistics, more on sample error} + +% --- begin paragraph admon --- +\paragraph{} +Let us first take a look at what happens to the sample error as the +size of the sample grows. In a sample, each of the measurements $x_i$ +can be associated with its own stochastic variable $X_i$. The +stochastic variable $\overline X_n$ for the sample mean $\bar{x}_n$ is +then just a linear combination, already familiar to us: +\[ +\overline X_n = \frac{1}{n}\sum_{i=1}^n X_i +\] +All the coefficients are just equal $1/n$. The PDF of $\overline X_n$, +denoted by $p_{\overline X_n}(x)$ is the desired PDF of the sample +means. +% --- end paragraph admon --- + + + +% !split +\subsection{Statistics} + +% --- begin paragraph admon --- +\paragraph{} +The probability density of obtaining a sample mean $\bar x_n$ +is the product of probabilities of obtaining arbitrary values $x_1, +x_2,\dots,x_n$ with the constraint that the mean of the set $\{x_i\}$ +is $\bar x_n$: +\[ +p_{\overline X_n}(x) = \int p_X^{\phantom X}(x_1)\cdots +\int p_X^{\phantom X}(x_n)\ +\delta\!\left(x - \frac{x_1+x_2+\dots+x_n}{n}\right)dx_n \cdots dx_1 +\] +And in particular we are interested in its variance $\mathrm{var}(\overline X_n)$. +% --- end paragraph admon --- + + + + + +% !split +\subsection{Statistics, central limit theorem} + +% --- begin paragraph admon --- +\paragraph{} +It is generally not possible to express $p_{\overline X_n}(x)$ in a +closed form given an arbitrary PDF $p_X^{\phantom X}$ and a number +$n$. But for the limit $n\to\infty$ it is possible to make an +approximation. The very important result is called \emph{the central limit theorem}. It tells us that as $n$ goes to infinity, +$p_{\overline X_n}(x)$ approaches a Gaussian distribution whose mean +and variance equal the true mean and variance, $\mu_{X}^{\phantom X}$ +and $\sigma_{X}^{2}$, respectively: +\begin{equation} +\lim_{n\to\infty} p_{\overline X_n}(x) = +\left(\frac{n}{2\pi\mathrm{var}(X)}\right)^{1/2} +e^{-\frac{n(x-\bar x_n)^2}{2\mathrm{var}(X)}} +\label{eq:central_limit_gaussian} +\end{equation} +% --- end paragraph admon --- + + + +% !split +\subsection{Statistics, more technicalities} + +% --- begin paragraph admon --- +\paragraph{} +The desired variance +$\mathrm{var}(\overline X_n)$, i.e.~the sample error squared +$\mathrm{err}_X^2$, is given by: +\begin{equation} +\mathrm{err}_X^2 = \mathrm{var}(\overline X_n) = \frac{1}{n^2} +\sum_{ij} \mathrm{cov}(X_i, X_j) +\label{eq:error_exact} +\end{equation} +We see now that in order to calculate the exact error of the sample +with the above expression, we would need the true means +$\mu_{X_i}^{\phantom X}$ of the stochastic variables $X_i$. To +calculate these requires that we know the true multivariate PDF of all +the $X_i$. But this PDF is unknown to us, we have only got the measurements of +one sample. The best we can do is to let the sample itself be an +estimate of the PDF of each of the $X_i$, estimating all properties of +$X_i$ through the measurements of the sample. +% --- end paragraph admon --- + + + + +% !split +\subsection{Statistics} + +% --- begin paragraph admon --- +\paragraph{} +Our estimate of $\mu_{X_i}^{\phantom X}$ is then the sample mean $\bar x$ +itself, in accordance with the the central limit theorem: +\[ +\mu_{X_i}^{\phantom X} = \langle x_i\rangle \approx \frac{1}{n}\sum_{k=1}^n x_k = \bar x +\] +Using $\bar x$ in place of $\mu_{X_i}^{\phantom X}$ we can give an +\emph{estimate} of the covariance in Eq.~(\ref{eq:error_exact}) +\[ +\mathrm{cov}(X_i, X_j) = \langle (x_i-\langle x_i\rangle)(x_j-\langle x_j\rangle)\rangle +\approx\langle (x_i - \bar x)(x_j - \bar{x})\rangle, +\] +resulting in +\[ +\frac{1}{n} \sum_{l}^n \left(\frac{1}{n}\sum_{k}^n (x_k -\bar x_n)(x_l - \bar x_n)\right)=\frac{1}{n}\frac{1}{n} \sum_{kl} (x_k -\bar x_n)(x_l - \bar x_n)=\frac{1}{n}\mathrm{cov}(x) +\] +% --- end paragraph admon --- + + + +% !split +\subsection{Statistics and sample variance} + +% --- begin paragraph admon --- +\paragraph{} +By the same procedure we can use the sample variance as an +estimate of the variance of any of the stochastic variables $X_i$ +\[ +\mathrm{var}(X_i)=\langle x_i - \langle x_i\rangle\rangle \approx \langle x_i - \bar x_n\rangle\nonumber, +\] +which is approximated as +\begin{equation} +\mathrm{var}(X_i)\approx \frac{1}{n}\sum_{k=1}^n (x_k - \bar x_n)=\mathrm{var}(x) +\label{eq:var_estimate_i_think} +\end{equation} + +Now we can calculate an estimate of the error +$\mathrm{err}_X^{\phantom X}$ of the sample mean $\bar x_n$: +\begin{align} +\mathrm{err}_X^2 +&=\frac{1}{n^2}\sum_{ij} \mathrm{cov}(X_i, X_j) \nonumber \\ +&\approx&\frac{1}{n^2}\sum_{ij}\frac{1}{n}\mathrm{cov}(x) =\frac{1}{n^2}n^2\frac{1}{n}\mathrm{cov}(x)\nonumber\\ +&=\frac{1}{n}\mathrm{cov}(x) +\label{eq:error_estimate} +\end{align} +which is nothing but the sample covariance divided by the number of +measurements in the sample. +% --- end paragraph admon --- + + + +% !split +\subsection{Statistics, uncorrelated results} + +% --- begin paragraph admon --- +\paragraph{} + +In the special case that the measurements of the sample are +uncorrelated (equivalently the stochastic variables $X_i$ are +uncorrelated) we have that the off-diagonal elements of the covariance +are zero. This gives the following estimate of the sample error: +\[ +\mathrm{err}_X^2=\frac{1}{n^2}\sum_{ij} \mathrm{cov}(X_i, X_j) = +\frac{1}{n^2} \sum_i \mathrm{var}(X_i), +\] +resulting in +\begin{equation} +\mathrm{err}_X^2\approx \frac{1}{n^2} \sum_i \mathrm{var}(x)= \frac{1}{n}\mathrm{var}(x) +\label{eq:error_estimate_uncorrel} +\end{equation} +where in the second step we have used Eq.~(\ref{eq:var_estimate_i_think}). +The error of the sample is then just its standard deviation divided by +the square root of the number of measurements the sample contains. +This is a very useful formula which is easy to compute. It acts as a +first approximation to the error, but in numerical experiments, we +cannot overlook the always present correlations. +% --- end paragraph admon --- + + + +% !split +\subsection{Statistics, computations} + +% --- begin paragraph admon --- +\paragraph{} +For computational purposes one usually splits up the estimate of +$\mathrm{err}_X^2$, given by Eq.~(\ref{eq:error_estimate}), into two +parts +\[ +\mathrm{err}_X^2 = \frac{1}{n}\mathrm{var}(x) + \frac{1}{n}(\mathrm{cov}(x)-\mathrm{var}(x)), +\] +which equals +\begin{equation} +\frac{1}{n^2}\sum_{k=1}^n (x_k - \bar x_n)^2 +\frac{2}{n^2}\sum_{k 0$. We say then that the ridge estimator is biased. + +We can also compute the variance as + +\[ +\mbox{Var}[\bm{\beta}^{\mathrm{Ridge}}]=\sigma^2[ \mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I} ]^{-1} \mathbf{X}^{T} \mathbf{X} \{ [ \mathbf{X}^{\top} \mathbf{X} + \lambda \mathbf{I} ]^{-1}\}^{T}, +\] +and it is easy to see that if the parameter $\lambda$ goes to infinity then the variance of Ridge parameters $\bm{\beta}$ goes to zero. + +With this, we can compute the difference + +\[ +\mbox{Var}[\bm{\beta}^{\mathrm{OLS}}]-\mbox{Var}(\bm{\beta}^{\mathrm{Ridge}})=\sigma^2 [ \mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I} ]^{-1}[ 2\lambda\mathbf{I} + \lambda^2 (\mathbf{X}^{T} \mathbf{X})^{-1} ] \{ [ \mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I} ]^{-1}\}^{T}. +\] +The difference is non-negative definite since each component of the +matrix product is non-negative definite. +This means the variance we obtain with the standard OLS will always for $\lambda > 0$ be larger than the variance of $\bm{\beta}$ obtained with the Ridge estimator. This has interesting consequences when we discuss the so-called bias-variance trade-off below. + +% !split +\subsection{Cross-validation} + +Instead of choosing the penalty parameter to balance model fit with +model complexity, cross-validation requires it (i.e.~the penalty +parameter) to yield a model with good prediction +performance. Commonly, this performance is evaluated on novel +data. Novel data need not be easy to come by and one has to make do +with the data at hand. + +The setting of \textbf{original} and novel data is +then mimicked by sample splitting: the data set is divided into two +(groups of samples). One of these two data sets, called the +\emph{training set}, plays the role of \textbf{original} data on which the model is +built. The second of these data sets, called the \emph{test set}, plays the +role of the \textbf{novel} data and is used to evaluate the prediction +performance (often operationalized as the log-likelihood or the +prediction error or its square or the R2 score) of the model built on the training data set. This +procedure (model building and prediction evaluation on training and +test set, respectively) is done for a collection of possible penalty +parameter choices. The penalty parameter that yields the model with +the best prediction performance is to be preferred. The thus obtained +performance evaluation depends on the actual split of the data set. To +remove this dependence the data set is split many times into a +training and test set. For each split the model parameters are +estimated for all choices of $\lambda$ using the training data and +estimated parameters are evaluated on the corresponding test set. The +penalty parameter that on average over the test sets performs best (in +some sense) is then selected. + + +% !split +\subsection{Computationally expensive} + +The validation set approach is conceptually simple and is easy to implement. But it has two potential drawbacks: + +\begin{itemize} +\item The validation estimate of the test error rate can be highly variable, depending on precisely which observations are included in the training set and which observations are included in the validation set. + +\item In the validation approach, only a subset of the observations, those that are included in the training set rather than in the validation set are used to fit the model. Since statistical methods tend to perform worse when trained on fewer observations, this suggests that the validation set error rate may tend to overestimate the test error rate for the model fit on the entire data set. +\end{itemize} + +\noindent +% !split +\subsection{Various steps in cross-validation} + +When the repetitive splitting of the data set is done randomly, +samples may accidently end up in a fast majority of the splits in +either training or test set. Such samples may have an unbalanced +influence on either model building or prediction evaluation. To avoid +this $k$-fold cross-validation structures the data splitting. The +samples are divided into $k$ more or less equally sized exhaustive and +mutually exclusive subsets. In turn (at each split) one of these +subsets plays the role of the test set while the union of the +remaining subsets constitutes the training set. Such a splitting +warrants a balanced representation of each sample in both training and +test set over the splits. Still the division into the $k$ subsets +involves a degree of randomness. This may be fully excluded when +choosing $k=n$. This particular case is referred to as leave-one-out +cross-validation (LOOCV). + +% !split +\subsection{How to set up the cross-validation for Ridge and/or Lasso} + +\begin{itemize} +\item Define a range of interest for the penalty parameter. + +\item Divide the data set into training and test set comprising samples $\{1, \ldots, n\} \setminus i$ and $\{ i \}$, respectively. + +\item Fit the linear regression model by means of ridge estimation for each $\lambda$ in the grid using the training set, and the corresponding estimate of the error variance $\bm{\sigma}_{-i}^2(\lambda)$, as +\end{itemize} + +\noindent +\begin{align*} +\bm{\beta}_{-i}(\lambda) & = ( \bm{X}_{-i, \ast}^{T} +\bm{X}_{-i, \ast} + \lambda \bm{I}_{pp})^{-1} +\bm{X}_{-i, \ast}^{T} \bm{y}_{-i} +\end{align*} + +\begin{itemize} +\item Evaluate the prediction performance of these models on the test set by $\log\{L[y_i, \bm{X}_{i, \ast}; \bm{\beta}_{-i}(\lambda), \bm{\sigma}_{-i}^2(\lambda)]\}$. Or, by the prediction error $|y_i - \bm{X}_{i, \ast} \bm{\beta}_{-i}(\lambda)|$, the relative error, the error squared or the R2 score function. + +\item Repeat the first three steps such that each sample plays the role of the test set once. + +\item Average the prediction performances of the test sets at each grid point of the penalty bias/parameter by computing the \emph{cross-validated log-likelihood}. It is an estimate of the prediction performance of the model corresponding to this value of the penalty parameter on novel data. It is defined as +\end{itemize} + +\noindent +\begin{align*} +\frac{1}{n} \sum_{i = 1}^n \log\{L[y_i, \mathbf{X}_{i, \ast}; \bm{\beta}_{-i}(\lambda), \bm{\sigma}_{-i}^2(\lambda)]\}. +\end{align*} + +\begin{itemize} +\item The value of the penalty parameter that maximizes the cross-validated log-likelihood is the value of choice. Or we can use the MSE or the R2 score functions. +\end{itemize} + +\noindent +% !split +\subsection{Resampling methods: Jackknife and Bootstrap} + +Two famous +resampling methods are the \textbf{independent bootstrap} and \textbf{the jackknife}. + +The jackknife is a special case of the independent bootstrap. Still, the jackknife was made +popular prior to the independent bootstrap. And as the popularity of +the independent bootstrap soared, new variants, such as \textbf{the dependent bootstrap}. + +The Jackknife and independent bootstrap work for +independent, identically distributed random variables. +If these conditions are not +satisfied, the methods will fail. Yet, it should be said that if the data are +independent, identically distributed, and we only want to estimate the +variance of $\overline{X}$ (which often is the case), then there is no +need for bootstrapping. + +% !split +\subsection{Resampling methods: Jackknife} + +The Jackknife works by making many replicas of the estimator $\widehat{\theta}$. +The jackknife is a resampling method where we systematically leave out one observation from the vector of observed values $\bm{x} = (x_1,x_2,\cdots,X_n)$. +Let $\bm{x}_i$ denote the vector +\[ +\bm{x}_i = (x_1,x_2,\cdots,x_{i-1},x_{i+1},\cdots,x_n), +\] + +which equals the vector $\bm{x}$ with the exception that observation +number $i$ is left out. Using this notation, define +$\widehat{\theta}_i$ to be the estimator +$\widehat{\theta}$ computed using $\vec{X}_i$. + + +% !split +\subsection{Jackknife code example} +\bpycod +from numpy import * +from numpy.random import randint, randn +from time import time + +def jackknife(data, stat): + n = len(data);t = zeros(n); inds = arange(n); t0 = time() + ## 'jackknifing' by leaving out an observation for each i + for i in range(n): + t[i] = stat(delete(data,i) ) + + # analysis + print("Runtime: %g sec" % (time()-t0)); print("Jackknife Statistics :") + print("original bias std. error") + print("%8g %14g %15g" % (stat(data),(n-1)*mean(t)/n, (n*var(t))**.5)) + + return t + + +# Returns mean of data samples +def stat(data): + return mean(data) + + +mu, sigma = 100, 15 +datapoints = 10000 +x = mu + sigma*random.randn(datapoints) +# jackknife returns the data sample +t = jackknife(x, stat) + +\epycod + + +% !split +\subsection{Resampling methods: Bootstrap} + +% --- begin paragraph admon --- +\paragraph{} +Bootstrapping is a nonparametric approach to statistical inference +that substitutes computation for more traditional distributional +assumptions and asymptotic results. Bootstrapping offers a number of +advantages: +\begin{enumerate} +\item The bootstrap is quite general, although there are some cases in which it fails. + +\item Because it does not require distributional assumptions (such as normally distributed errors), the bootstrap can provide more accurate inferences when the data are not well behaved or when the sample size is small. + +\item It is possible to apply the bootstrap to statistics with sampling distributions that are difficult to derive, even asymptotically. + +\item It is relatively simple to apply the bootstrap to complex data-collection plans (such as stratified and clustered samples). +\end{enumerate} + +\noindent +% --- end paragraph admon --- + + + + +% !split +\subsection{Resampling methods: Bootstrap background} + +Since $\widehat{\theta} = \widehat{\theta}(\bm{X})$ is a function of random variables, +$\widehat{\theta}$ itself must be a random variable. Thus it has +a pdf, call this function $p(\bm{t})$. The aim of the bootstrap is to +estimate $p(\bm{t})$ by the relative frequency of +$\widehat{\theta}$. You can think of this as using a histogram +in the place of $p(\bm{t})$. If the relative frequency closely +resembles $p(\vec{t})$, then using numerics, it is straight forward to +estimate all the interesting parameters of $p(\bm{t})$ using point +estimators. + + +% !split +\subsection{Resampling methods: More Bootstrap background} + +In the case that $\widehat{\theta}$ has +more than one component, and the components are independent, we use the +same estimator on each component separately. If the probability +density function of $X_i$, $p(x)$, had been known, then it would have +been straight forward to do this by: +\begin{enumerate} +\item Drawing lots of numbers from $p(x)$, suppose we call one such set of numbers $(X_1^*, X_2^*, \cdots, X_n^*)$. + +\item Then using these numbers, we could compute a replica of $\widehat{\theta}$ called $\widehat{\theta}^*$. +\end{enumerate} + +\noindent +By repeated use of (1) and (2), many +estimates of $\widehat{\theta}$ could have been obtained. The +idea is to use the relative frequency of $\widehat{\theta}^*$ +(think of a histogram) as an estimate of $p(\bm{t})$. + +% !split +\subsection{Resampling methods: Bootstrap approach} + +But +unless there is enough information available about the process that +generated $X_1,X_2,\cdots,X_n$, $p(x)$ is in general +unknown. Therefore, \href{{https://projecteuclid.org/euclid.aos/1176344552}}{Efron in 1979} asked the +question: What if we replace $p(x)$ by the relative frequency +of the observation $X_i$; if we draw observations in accordance with +the relative frequency of the observations, will we obtain the same +result in some asymptotic sense? The answer is yes. + + +Instead of generating the histogram for the relative +frequency of the observation $X_i$, just draw the values +$(X_1^*,X_2^*,\cdots,X_n^*)$ with replacement from the vector +$\bm{X}$. + +% !split +\subsection{Resampling methods: Bootstrap steps} + +The independent bootstrap works like this: + +\begin{enumerate} +\item Draw with replacement $n$ numbers for the observed variables $\bm{x} = (x_1,x_2,\cdots,x_n)$. + +\item Define a vector $\bm{x}^*$ containing the values which were drawn from $\bm{x}$. + +\item Using the vector $\bm{x}^*$ compute $\widehat{\theta}^*$ by evaluating $\widehat \theta$ under the observations $\bm{x}^*$. + +\item Repeat this process $k$ times. +\end{enumerate} + +\noindent +When you are done, you can draw a histogram of the relative frequency +of $\widehat \theta^*$. This is your estimate of the probability +distribution $p(t)$. Using this probability distribution you can +estimate any statistics thereof. In principle you never draw the +histogram of the relative frequency of $\widehat{\theta}^*$. Instead +you use the estimators corresponding to the statistic of interest. For +example, if you are interested in estimating the variance of $\widehat +\theta$, apply the etsimator $\widehat \sigma^2$ to the values +$\widehat \theta ^*$. + + +% !split +\subsection{Code example for the Bootstrap method} + +The following code starts with a Gaussian distribution with mean value +$\mu =100$ and variance $\sigma=15$. We use this to generate the data +used in the bootstrap analysis. The bootstrap analysis returns a data +set after a given number of bootstrap operations (as many as we have +data points). This data set consists of estimated mean values for each +bootstrap operation. The histogram generated by the bootstrap method +shows that the distribution for these mean values is also a Gaussian, +centered around the mean value $\mu=100$ but with standard deviation +$\sigma/\sqrt{n}$, where $n$ is the number of bootstrap samples (in +this case the same as the number of original data points). The value +of the standard deviation is what we expect from the central limit +theorem. + + +\bpycod +from numpy import * +from numpy.random import randint, randn +from time import time +import matplotlib.mlab as mlab +import matplotlib.pyplot as plt + +# Returns mean of bootstrap samples +def stat(data): + return mean(data) + +# Bootstrap algorithm +def bootstrap(data, statistic, R): + t = zeros(R); n = len(data); inds = arange(n); t0 = time() + # non-parametric bootstrap + for i in range(R): + t[i] = statistic(data[randint(0,n,n)]) + + # analysis + print("Runtime: %g sec" % (time()-t0)); print("Bootstrap Statistics :") + print("original bias std. error") + print("%8g %8g %14g %15g" % (statistic(data), std(data),mean(t),std(t))) + return t + + +mu, sigma = 100, 15 +datapoints = 10000 +x = mu + sigma*random.randn(datapoints) +# bootstrap returns the data sample +t = bootstrap(x, stat, datapoints) +# the histogram of the bootstrapped data +n, binsboot, patches = plt.hist(t, 50, normed=1, facecolor='red', alpha=0.75) + +# add a 'best fit' line +y = mlab.normpdf( binsboot, mean(t), std(t)) +lt = plt.plot(binsboot, y, 'r--', linewidth=1) +plt.xlabel('Smarts') +plt.ylabel('Probability') +plt.axis([99.5, 100.6, 0, 3.0]) +plt.grid(True) + +plt.show() + +\epycod + + +% !split +\subsection{Code Example for Cross-validation and $k$-fold Cross-validation} + +The code here uses Ridge regression with cross-validation (CV) resampling and $k$-fold CV in order to fit a specific polynomial. +\bpycod +import numpy as np +import matplotlib.pyplot as plt +from sklearn.model_selection import KFold +from sklearn.linear_model import Ridge +from sklearn.model_selection import cross_val_score +from sklearn.preprocessing import PolynomialFeatures + +# A seed just to ensure that the random numbers are the same for every run. +# Useful for eventual debugging. +np.random.seed(3155) + +# Generate the data. +nsamples = 100 +x = np.random.randn(nsamples) +y = 3*x**2 + np.random.randn(nsamples) + +## Cross-validation on Ridge regression using KFold only + +# Decide degree on polynomial to fit +poly = PolynomialFeatures(degree = 6) + +# Decide which values of lambda to use +nlambdas = 500 +lambdas = np.logspace(-3, 5, nlambdas) + +# Initialize a KFold instance +k = 5 +kfold = KFold(n_splits = k) + +# Perform the cross-validation to estimate MSE +scores_KFold = np.zeros((nlambdas, k)) + +i = 0 +for lmb in lambdas: + ridge = Ridge(alpha = lmb) + j = 0 + for train_inds, test_inds in kfold.split(x): + xtrain = x[train_inds] + ytrain = y[train_inds] + + xtest = x[test_inds] + ytest = y[test_inds] + + Xtrain = poly.fit_transform(xtrain[:, np.newaxis]) + ridge.fit(Xtrain, ytrain[:, np.newaxis]) + + Xtest = poly.fit_transform(xtest[:, np.newaxis]) + ypred = ridge.predict(Xtest) + + scores_KFold[i,j] = np.sum((ypred - ytest[:, np.newaxis])**2)/np.size(ypred) + + j += 1 + i += 1 + + +estimated_mse_KFold = np.mean(scores_KFold, axis = 1) + +## Cross-validation using cross_val_score from sklearn along with KFold + +# kfold is an instance initialized above as: +# kfold = KFold(n_splits = k) + +estimated_mse_sklearn = np.zeros(nlambdas) +i = 0 +for lmb in lambdas: + ridge = Ridge(alpha = lmb) + + X = poly.fit_transform(x[:, np.newaxis]) + estimated_mse_folds = cross_val_score(ridge, X, y[:, np.newaxis], scoring='neg_mean_squared_error', cv=kfold) + + # cross_val_score return an array containing the estimated negative mse for every fold. + # we have to the the mean of every array in order to get an estimate of the mse of the model + estimated_mse_sklearn[i] = np.mean(-estimated_mse_folds) + + i += 1 + +## Plot and compare the slightly different ways to perform cross-validation + +plt.figure() + +plt.plot(np.log10(lambdas), estimated_mse_sklearn, label = 'cross_val_score') +plt.plot(np.log10(lambdas), estimated_mse_KFold, 'r--', label = 'KFold') + +plt.xlabel('log10(lambda)') +plt.ylabel('mse') + +plt.legend() + +plt.show() + +\epycod + + +% !split +\subsection{The bias-variance tradeoff} + + +We will discuss the bias-variance tradeoff in the context of +continuous predictions such as regression. However, many of the +intuitions and ideas discussed here also carry over to classification +tasks. Consider a dataset $\mathcal{L}$ consisting of the data +$\mathbf{X}_\mathcal{L}=\{(y_j, \boldsymbol{x}_j), j=0\ldots n-1\}$. + +Let us assume that the true data is generated from a noisy model + +\[ +\bm{y}=f(\boldsymbol{x}) + \bm{\epsilon} +\] + +where $\epsilon$ is normally distributed with mean zero and standard deviation $\sigma^2$. + +In our derivation of the ordinary least squares method we defined then +an approximation to the function $f$ in terms of the parameters +$\bm{\beta}$ and the design matrix $\bm{X}$ which embody our model, +that is $\bm{\tilde{y}}=\bm{X}\bm{\beta}$. + +Thereafter we found the parameters $\bm{\beta}$ by optimizing the means squared error via the so-called cost function +\[ +C(\bm{X},\bm{\beta}) =\frac{1}{n}\sum_{i=0}^{n-1}(y_i-\tilde{y}_i)^2=\mathbb{E}\left[(\bm{y}-\bm{\tilde{y}})^2\right]. +\] + +We can rewrite this as +\[ +\mathbb{E}\left[(\bm{y}-\bm{\tilde{y}})^2\right]=\frac{1}{n}\sum_i(f_i-\mathbb{E}\left[\bm{\tilde{y}}\right])^2+\frac{1}{n}\sum_i(\tilde{y}_i-\mathbb{E}\left[\bm{\tilde{y}}\right])^2+\sigma^2. +\] + +The three terms represent the square of the bias of the learning +method, which can be thought of as the error caused by the simplifying +assumptions built into the method. The second term represents the +variance of the chosen model and finally the last terms is variance of +the error $\bm{\epsilon}$. + +To derive this equation, we need to recall that the variance of $\bm{y}$ and $\bm{\epsilon}$ are both equal to $\sigma^2$. The mean value of $\bm{\epsilon}$ is by definition equal to zero. Furthermore, the function $f$ is not a stochastics variable, idem for $\bm{\tilde{y}}$. +We use a more compact notation in terms of the expectation value +\[ +\mathbb{E}\left[(\bm{y}-\bm{\tilde{y}})^2\right]=\mathbb{E}\left[(\bm{f}+\bm{\epsilon}-\bm{\tilde{y}})^2\right], +\] +and adding and subtracting $\mathbb{E}\left[\bm{\tilde{y}}\right]$ we get +\[ +\mathbb{E}\left[(\bm{y}-\bm{\tilde{y}})^2\right]=\mathbb{E}\left[(\bm{f}+\bm{\epsilon}-\bm{\tilde{y}}+\mathbb{E}\left[\bm{\tilde{y}}\right]-\mathbb{E}\left[\bm{\tilde{y}}\right])^2\right], +\] +which, using the abovementioned expectation values can be rewritten as +\[ +\mathbb{E}\left[(\bm{y}-\bm{\tilde{y}})^2\right]=\mathbb{E}\left[(\bm{y}-\mathbb{E}\left[\bm{\tilde{y}}\right])^2\right]+\mathrm{Var}\left[\bm{\tilde{y}}\right]+\sigma^2, +\] +that is the rewriting in terms of the so-called bias, the variance of the model $\bm{\tilde{y}}$ and the variance of $\bm{\epsilon}$. + + + + + +% !split +\subsection{Example code for Bias-Variance tradeoff} +\bpycod +import matplotlib.pyplot as plt +import numpy as np +from sklearn.linear_model import LinearRegression, Ridge, Lasso +from sklearn.preprocessing import PolynomialFeatures +from sklearn.model_selection import train_test_split +from sklearn.pipeline import make_pipeline +from sklearn.utils import resample + +np.random.seed(2018) + +n = 500 +n_boostraps = 100 +degree = 18 # A quite high value, just to show. +noise = 0.1 + +# Make data set. +x = np.linspace(-1, 3, n).reshape(-1, 1) +y = np.exp(-x**2) + 1.5 * np.exp(-(x-2)**2) + np.random.normal(0, 0.1, x.shape) + +# Hold out some test data that is never used in training. +x_train, x_test, y_train, y_test = train_test_split(x, y, test_size=0.2) + +# Combine x transformation and model into one operation. +# Not neccesary, but convenient. +model = make_pipeline(PolynomialFeatures(degree=degree), LinearRegression(fit_intercept=False)) + +# The following (m x n_bootstraps) matrix holds the column vectors y_pred +# for each bootstrap iteration. +y_pred = np.empty((y_test.shape[0], n_boostraps)) +for i in range(n_boostraps): + x_, y_ = resample(x_train, y_train) + + # Evaluate the new model on the same test data each time. + y_pred[:, i] = model.fit(x_, y_).predict(x_test).ravel() + +# Note: Expectations and variances taken w.r.t. different training +# data sets, hence the axis=1. Subsequent means are taken across the test data +# set in order to obtain a total value, but before this we have error/bias/variance +# calculated per data point in the test set. +# Note 2: The use of keepdims=True is important in the calculation of bias as this +# maintains the column vector form. Dropping this yields very unexpected results. +error = np.mean( np.mean((y_test - y_pred)**2, axis=1, keepdims=True) ) +bias = np.mean( (y_test - np.mean(y_pred, axis=1, keepdims=True))**2 ) +variance = np.mean( np.var(y_pred, axis=1, keepdims=True) ) +print('Error:', error) +print('Bias^2:', bias) +print('Var:', variance) +print('{} >= {} + {} = {}'.format(error, bias, variance, bias+variance)) + +plt.plot(x[::5, :], y[::5, :], label='f(x)') +plt.scatter(x_test, y_test, label='Data points') +plt.scatter(x_test, np.mean(y_pred, axis=1), label='Pred') +plt.legend() +plt.show() + +\epycod + + +% !split +\subsection{Understanding what happens} +\bpycod +import matplotlib.pyplot as plt +import numpy as np +from sklearn.linear_model import LinearRegression, Ridge, Lasso +from sklearn.preprocessing import PolynomialFeatures +from sklearn.model_selection import train_test_split +from sklearn.pipeline import make_pipeline +from sklearn.utils import resample + +np.random.seed(2018) + +n = 40 +n_boostraps = 100 +maxdegree = 14 + + +# Make data set. +x = np.linspace(-3, 3, n).reshape(-1, 1) +y = np.exp(-x**2) + 1.5 * np.exp(-(x-2)**2)+ np.random.normal(0, 0.1, x.shape) +error = np.zeros(maxdegree) +bias = np.zeros(maxdegree) +variance = np.zeros(maxdegree) +polydegree = np.zeros(maxdegree) +x_train, x_test, y_train, y_test = train_test_split(x, y, test_size=0.2) + +for degree in range(maxdegree): + model = make_pipeline(PolynomialFeatures(degree=degree), LinearRegression(fit_intercept=False)) + y_pred = np.empty((y_test.shape[0], n_boostraps)) + for i in range(n_boostraps): + x_, y_ = resample(x_train, y_train) + y_pred[:, i] = model.fit(x_, y_).predict(x_test).ravel() + + polydegree[degree] = degree + error[degree] = np.mean( np.mean((y_test - y_pred)**2, axis=1, keepdims=True) ) + bias[degree] = np.mean( (y_test - np.mean(y_pred, axis=1, keepdims=True))**2 ) + variance[degree] = np.mean( np.var(y_pred, axis=1, keepdims=True) ) + print('Polynomial degree:', degree) + print('Error:', error[degree]) + print('Bias^2:', bias[degree]) + print('Var:', variance[degree]) + print('{} >= {} + {} = {}'.format(error[degree], bias[degree], variance[degree], bias[degree]+variance[degree])) + +plt.plot(polydegree, np.log10(error), label='Error') +plt.plot(polydegree, bias, label='bias') +plt.plot(polydegree, variance, label='Variance') +plt.legend() +plt.show() + + + + +\epycod + +% !split +\subsection{Summing up} + + + + +The bias-variance tradeoff summarizes the fundamental tension in +machine learning, particularly supervised learning, between the +complexity of a model and the amount of training data needed to train +it. Since data is often limited, in practice it is often useful to +use a less-complex model with higher bias, that is a model whose asymptotic +performance is worse than another model because it is easier to +train and less sensitive to sampling noise arising from having a +finite-sized training dataset (smaller variance). + + + +The above equations tell us that in +order to minimize the expected test error, we need to select a +statistical learning method that simultaneously achieves low variance +and low bias. Note that variance is inherently a nonnegative quantity, +and squared bias is also nonnegative. Hence, we see that the expected +test MSE can never lie below $Var(\epsilon)$, the irreducible error. + + +What do we mean by the variance and bias of a statistical learning +method? The variance refers to the amount by which our model would change if we +estimated it using a different training data set. Since the training +data are used to fit the statistical learning method, different +training data sets will result in a different estimate. But ideally the +estimate for our model should not vary too much between training +sets. However, if a method has high variance then small changes in +the training data can result in large changes in the model. In general, more +flexible statistical methods have higher variance. + + +% !split +\subsection{Another Example rom Scikit-Learn's Repository} +\bpycod +""" +============================ +Underfitting vs. Overfitting +============================ + +This example demonstrates the problems of underfitting and overfitting and +how we can use linear regression with polynomial features to approximate +nonlinear functions. The plot shows the function that we want to approximate, +which is a part of the cosine function. In addition, the samples from the +real function and the approximations of different models are displayed. The +models have polynomial features of different degrees. We can see that a +linear function (polynomial with degree 1) is not sufficient to fit the +training samples. This is called **underfitting**. A polynomial of degree 4 +approximates the true function almost perfectly. However, for higher degrees +the model will **overfit** the training data, i.e. it learns the noise of the +training data. +We evaluate quantitatively **overfitting** / **underfitting** by using +cross-validation. We calculate the mean squared error (MSE) on the validation +set, the higher, the less likely the model generalizes correctly from the +training data. +""" + +print(__doc__) + +import numpy as np +import matplotlib.pyplot as plt +from sklearn.pipeline import Pipeline +from sklearn.preprocessing import PolynomialFeatures +from sklearn.linear_model import LinearRegression +from sklearn.model_selection import cross_val_score + + +def true_fun(X): + return np.cos(1.5 * np.pi * X) + +np.random.seed(0) + +n_samples = 30 +degrees = [1, 4, 15] + +X = np.sort(np.random.rand(n_samples)) +y = true_fun(X) + np.random.randn(n_samples) * 0.1 + +plt.figure(figsize=(14, 5)) +for i in range(len(degrees)): + ax = plt.subplot(1, len(degrees), i + 1) + plt.setp(ax, xticks=(), yticks=()) + + polynomial_features = PolynomialFeatures(degree=degrees[i], + include_bias=False) + linear_regression = LinearRegression() + pipeline = Pipeline([("polynomial_features", polynomial_features), + ("linear_regression", linear_regression)]) + pipeline.fit(X[:, np.newaxis], y) + + # Evaluate the models using crossvalidation + scores = cross_val_score(pipeline, X[:, np.newaxis], y, + scoring="neg_mean_squared_error", cv=10) + + X_test = np.linspace(0, 1, 100) + plt.plot(X_test, pipeline.predict(X_test[:, np.newaxis]), label="Model") + plt.plot(X_test, true_fun(X_test), label="True function") + plt.scatter(X, y, edgecolor='b', s=20, label="Samples") + plt.xlabel("x") + plt.ylabel("y") + plt.xlim((0, 1)) + plt.ylim((-2, 2)) + plt.legend(loc="best") + plt.title("Degree {}\nMSE = {:.2e}(+/- {:.2e})".format( + degrees[i], -scores.mean(), scores.std())) +plt.show() +\epycod + + + +% !split +\subsection{The one-dimensional Ising model} + +Let us bring back the Ising model again, but now with an additional +focus on Ridge and Lasso regression as well. We repeat some of the +basic parts of the Ising model and the setup of the training and test +data. The one-dimensional Ising model with nearest neighbor +interaction, no external field and a constant coupling constant $J$ is +given by + +\begin{align} + H = -J \sum_{k}^L s_k s_{k + 1}, +\end{align} +where $s_i \in \{-1, 1\}$ and $s_{N + 1} = s_1$. The number of spins in the system is determined by $L$. For the one-dimensional system there is no phase transition. + +We will look at a system of $L = 40$ spins with a coupling constant of $J = 1$. To get enough training data we will generate 10000 states with their respective energies. + + +\bpycod +import numpy as np +import matplotlib.pyplot as plt +from mpl_toolkits.axes_grid1 import make_axes_locatable +import seaborn as sns +import scipy.linalg as scl +from sklearn.model_selection import train_test_split +import sklearn.linear_model as skl +import tqdm +sns.set(color_codes=True) +cmap_args=dict(vmin=-1., vmax=1., cmap='seismic') + +L = 40 +n = int(1e4) + +spins = np.random.choice([-1, 1], size=(n, L)) +J = 1.0 + +energies = np.zeros(n) + +for i in range(n): + energies[i] = - J * np.dot(spins[i], np.roll(spins[i], 1)) +\epycod + +A more general form for the one-dimensional Ising model is + +\begin{align} + H = - \sum_j^L \sum_k^L s_j s_k J_{jk}. +\end{align} + +Here we allow for interactions beyond the nearest neighbors and a more +adaptive coupling matrix. This latter expression can be formulated as +a matrix-product on the form +\begin{align} + H = X J, +\end{align} + +where $X_{jk} = s_j s_k$ and $J$ is the matrix consisting of the +elements $-J_{jk}$. This form of writing the energy fits perfectly +with the form utilized in linear regression, viz. +\begin{align} + \bm{y} = \bm{X}\bm{\beta} + \bm{\epsilon}. +\end{align} +We organize the data as we did above +\bpycod +X = np.zeros((n, L ** 2)) +for i in range(n): + X[i] = np.outer(spins[i], spins[i]).ravel() +y = energies +X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.96) + +X_train_own = np.concatenate( + (np.ones(len(X_train))[:, np.newaxis], X_train), + axis=1 +) + +X_test_own = np.concatenate( + (np.ones(len(X_test))[:, np.newaxis], X_test), + axis=1 +) +\epycod + +We will do all fitting with \textbf{Scikit-Learn}, + +\bpycod +clf = skl.LinearRegression().fit(X_train, y_train) +\epycod +When extracting the $J$-matrix we make sure to remove the intercept +\bpycod +J_sk = clf.coef_.reshape(L, L) +\epycod +And then we plot the results +\bpycod +fig = plt.figure(figsize=(20, 14)) +im = plt.imshow(J_sk, **cmap_args) +plt.title("LinearRegression from Scikit-learn", fontsize=18) +plt.xticks(fontsize=18) +plt.yticks(fontsize=18) +cb = fig.colorbar(im) +cb.ax.set_yticklabels(cb.ax.get_yticklabels(), fontsize=18) +plt.show() +\epycod +The results perfectly with our previous discussion where we used our own code. + +% !split +\subsection{Ridge regression} + +Having explored the ordinary least squares we move on to ridge +regression. In ridge regression we include a \textbf{regularizer}. This +involves a new cost function which leads to a new estimate for the +weights $\bm{\beta}$. This results in a penalized regression problem. The +cost function is given by + +\begin{align} + C(\bm{X}, \bm{\beta}; \lambda) = (\bm{X}\bm{\beta} - \bm{y})^T(\bm{X}\bm{\beta} - \bm{y}) + \lambda \bm{\beta}^T\bm{\beta}. +\end{align} +\bpycod +_lambda = 0.1 +clf_ridge = skl.Ridge(alpha=_lambda).fit(X_train, y_train) +J_ridge_sk = clf_ridge.coef_.reshape(L, L) +fig = plt.figure(figsize=(20, 14)) +im = plt.imshow(J_ridge_sk, **cmap_args) +plt.title("Ridge from Scikit-learn", fontsize=18) +plt.xticks(fontsize=18) +plt.yticks(fontsize=18) +cb = fig.colorbar(im) +cb.ax.set_yticklabels(cb.ax.get_yticklabels(), fontsize=18) + +plt.show() +\epycod + +% !split +\subsection{LASSO regression} + +In the \textbf{Least Absolute Shrinkage and Selection Operator} (LASSO)-method we get a third cost function. + +\begin{align} + C(\bm{X}, \bm{\beta}; \lambda) = (\bm{X}\bm{\beta} - \bm{y})^T(\bm{X}\bm{\beta} - \bm{y}) + \lambda \sqrt{\bm{\beta}^T\bm{\beta}}. +\end{align} + +Finding the extremal point of this cost function is not so straight-forward as in least squares and ridge. We will therefore rely solely on the function ``Lasso`` from \textbf{Scikit-Learn}. + +\bpycod +clf_lasso = skl.Lasso(alpha=_lambda).fit(X_train, y_train) +J_lasso_sk = clf_lasso.coef_.reshape(L, L) +fig = plt.figure(figsize=(20, 14)) +im = plt.imshow(J_lasso_sk, **cmap_args) +plt.title("Lasso from Scikit-learn", fontsize=18) +plt.xticks(fontsize=18) +plt.yticks(fontsize=18) +cb = fig.colorbar(im) +cb.ax.set_yticklabels(cb.ax.get_yticklabels(), fontsize=18) + +plt.show() +\epycod + +It is quite striking how LASSO breaks the symmetry of the coupling +constant as opposed to ridge and OLS. We get a sparse solution with +$J_{j, j + 1} = -1$. + + + +% !split +\subsection{Performance as function of the regularization parameter} + +We see how the different models perform for a different set of values for $\lambda$. + + +\bpycod +lambdas = np.logspace(-4, 5, 10) + +train_errors = { + "ols_sk": np.zeros(lambdas.size), + "ridge_sk": np.zeros(lambdas.size), + "lasso_sk": np.zeros(lambdas.size) +} + +test_errors = { + "ols_sk": np.zeros(lambdas.size), + "ridge_sk": np.zeros(lambdas.size), + "lasso_sk": np.zeros(lambdas.size) +} + +plot_counter = 1 + +fig = plt.figure(figsize=(32, 54)) + +for i, _lambda in enumerate(tqdm.tqdm(lambdas)): + for key, method in zip( + ["ols_sk", "ridge_sk", "lasso_sk"], + [skl.LinearRegression(), skl.Ridge(alpha=_lambda), skl.Lasso(alpha=_lambda)] + ): + method = method.fit(X_train, y_train) + + train_errors[key][i] = method.score(X_train, y_train) + test_errors[key][i] = method.score(X_test, y_test) + + omega = method.coef_.reshape(L, L) + + plt.subplot(10, 5, plot_counter) + plt.imshow(omega, **cmap_args) + plt.title(r"%s, $\lambda = %.4f$" % (key, _lambda)) + plot_counter += 1 + +plt.show() +\epycod + +We see that LASSO reaches a good solution for low +values of $\lambda$, but will "wither" when we increase $\lambda$ too +much. Ridge is more stable over a larger range of values for +$\lambda$, but eventually also fades away. + +% !split +\subsection{Finding the optimal value of $\lambda$} + +To determine which value of $\lambda$ is best we plot the accuracy of +the models when predicting the training and the testing set. We expect +the accuracy of the training set to be quite good, but if the accuracy +of the testing set is much lower this tells us that we might be +subject to an overfit model. The ideal scenario is an accuracy on the +testing set that is close to the accuracy of the training set. + + +\bpycod +fig = plt.figure(figsize=(20, 14)) + +colors = { + "ols_sk": "r", + "ridge_sk": "y", + "lasso_sk": "c" +} + +for key in train_errors: + plt.semilogx( + lambdas, + train_errors[key], + colors[key], + label="Train {0}".format(key), + linewidth=4.0 + ) + +for key in test_errors: + plt.semilogx( + lambdas, + test_errors[key], + colors[key] + "--", + label="Test {0}".format(key), + linewidth=4.0 + ) +plt.legend(loc="best", fontsize=18) +plt.xlabel(r"$\lambda$", fontsize=18) +plt.ylabel(r"$R^2$", fontsize=18) +plt.tick_params(labelsize=18) +plt.show() +\epycod + +From the above figure we can see that LASSO with $\lambda = 10^{-2}$ +achieves a very good accuracy on the test set. This by far surpasses the +other models for all values of $\lambda$. + + + +% !split +\subsection{Further Exercises} + +\paragraph{Exercise 1.} +We will generate our own dataset for a function $y(x)$ where $x \in [0,1]$ and defined by random numbers computed with the uniform distribution. The function $y$ is a quadratic polynomial in $x$ with added stochastic noise according to the normal distribution $\cal {N}(0,1)$. +The following simple Python instructions define our $x$ and $y$ values (with 100 data points). +\bpycod +x = np.random.rand(100,1) +y = 5*x*x+0.1*np.random.randn(100,1) +\epycod + +\begin{enumerate} +\item Write your own code (following the examples above) for computing the parametrization of the data set fitting a second-order polynomial. + +\item Use thereafter \textbf{scikit-learn} (see again the examples in the regression slides) and compare with your own code. + +\item Using scikit-learn, compute also the mean square error, a risk metric corresponding to the expected value of the squared (quadratic) error defined as +\end{enumerate} + +\noindent +\[ MSE(\hat{y},\hat{\tilde{y}}) = \frac{1}{n} +\sum_{i=0}^{n-1}(y_i-\tilde{y}_i)^2, +\] +and the $R^2$ score function. +If $\tilde{\hat{y}}_i$ is the predicted value of the $i-th$ sample and $y_i$ is the corresponding true value, then the score $R^2$ is defined as +\[ +R^2(\hat{y}, \tilde{\hat{y}}) = 1 - \frac{\sum_{i=0}^{n - 1} (y_i - \tilde{y}_i)^2}{\sum_{i=0}^{n - 1} (y_i - \bar{y})^2}, +\] +where we have defined the mean value of $\hat{y}$ as +\[ +\bar{y} = \frac{1}{n} \sum_{i=0}^{n - 1} y_i. +\] + +You can use the functionality included in scikit-learn. If you feel +for it, you can use your own program and define functions which +compute the above two functions. Discuss the meaning of these +results. Try also to vary the coefficient in front of the added +stochastic noise term and discuss the quality of the fits. + + + + +\paragraph{Exercise 2, variance of the parameters $\beta$ in linear regression.} +Show that the variance of the parameters $\beta$ in the linear regression method (chapter 3, equation (3.8) of \href{{https://www.springer.com/gp/book/9780387848570}}{Trevor Hastie, Robert Tibshirani, Jerome H. Friedman, The Elements of Statistical Learning, Springer}) is given as + +\[ +\mathrm{Var}(\hat{\beta}) = \left(\hat{X}^T\hat{X}\right)^{-1}\sigma^2, +\] +with +\[ +\sigma^2 = \frac{1}{N-p-1}\sum_{i=1}^{N} (y_i-\tilde{y}_i)^2, +\] +where we have assumed that we fit a function of degree $p-1$ (for example a polynomial in $x$). + + + +\paragraph{Exercise 3.} +This exercise is a continuation of exercise 1. We will +use the same function to generate our data set, still staying with a +simple function $y(x)$ which we want to fit using linear regression, +but now extending the analysis to include the Ridge and the Lasso +regression methods. You can use the code under the Regression as an example on how to use the Ridge and the Lasso methods. + +We will thus again generate our own dataset for a function $y(x)$ where +$x \in [0,1]$ and defined by random numbers computed with the uniform +distribution. The function $y$ is a quadratic polynomial in $x$ with +added stochastic noise according to the normal distribution $\cal{N}(0,1)$. + +The following simple Python instructions define our $x$ and $y$ values (with 100 data points). +\bpycod +x = np.random.rand(100,1) +y = 5*x*x+0.1*np.random.randn(100,1) +\epycod + +\begin{enumerate} +\item Write your own code for the Ridge method and compute the parametrization for different values of $\lambda$. Compare and analyze your results with those from exercise 1. Study the dependence on $\lambda$ while also varying the strength of the noise in your expression for $y(x)$. + +\item Repeat the above but using the functionality of \textbf{scikit-learn}. Compare your code with the results from \textbf{scikit-learn}. Remember to run with the same random numbers for generating $x$ and $y$. + +\item Our next step is to study the variance of the parameters $\beta_1$ and $\beta_2$ (assuming that we are parametrizing our function with a second-order polynomial. We will use standard linear regression and the Ridge regression. You can now opt for either writing your own function that calculates the variance of these paramaters (recall that this is equal to the diagonal elements of the matrix $(\hat{X}^T\hat{X})+\lambda\hat{I})^{-1}$) or use the functionality of \textbf{scikit-learn} and compute their variances. Discuss the results of these variances as functions + +\item Repeat the previous step but add now the Lasso method. Discuss your results and compare with standard regression and the Ridge regression results. + +\item Try to implement the cross-validation as well. + +\item Finally, using \textbf{scikit-learn} or your own code, compute also the mean square error, a risk metric corresponding to the expected value of the squared (quadratic) error defined as +\end{enumerate} + +\noindent +\[ MSE(\hat{y},\hat{\tilde{y}}) = \frac{1}{n} +\sum_{i=0}^{n-1}(y_i-\tilde{y}_i)^2, +\] +and the $R^2$ score function. +If $\tilde{\hat{y}}_i$ is the predicted value of the $i-th$ sample and $y_i$ is the corresponding true value, then the score $R^2$ is defined as +\[ +R^2(\hat{y}, \tilde{\hat{y}}) = 1 - \frac{\sum_{i=0}^{n - 1} (y_i - \tilde{y}_i)^2}{\sum_{i=0}^{n - 1} (y_i - \bar{y})^2}, +\] +where we have defined the mean value of $\hat{y}$ as +\[ +\bar{y} = \frac{1}{n} \sum_{i=0}^{n - 1} y_i. +\] +Discuss these quantities as functions of the variable $\lambda$ in the Ridge and Lasso regression methods. + +\paragraph{Exercise 4.} +We will study how +to fit polynomials to a specific two-dimensional function called +\href{{http://www.dtic.mil/dtic/tr/fulltext/u2/a081688.pdf}}{Franke's +function}. This +is a function which has been widely used when testing various interpolation and fitting +algorithms. Furthermore, after having established the model and the +method, we will employ resamling techniques such as the cross-validation and/or +the bootstrap methods, in order to perform a proper assessment of our models. + + +The Franke function, which is a weighted sum of four exponentials reads as follows +\begin{align*} +f(x,y) &= \frac{3}{4}\exp{\left(-\frac{(9x-2)^2}{4} - \frac{(9y-2)^2}{4}\right)}+\frac{3}{4}\exp{\left(-\frac{(9x+1)^2}{49}- \frac{(9y+1)}{10}\right)} \\ +&+\frac{1}{2}\exp{\left(-\frac{(9x-7)^2}{4} - \frac{(9y-3)^2}{4}\right)} -\frac{1}{5}\exp{\left(-(9x-4)^2 - (9y-7)^2\right) }. +\end{align*} + +The function will be defined for $x,y\in [0,1]$. Our first step will +be to perform an OLS regression analysis of this function, trying out +a polynomial fit with an $x$ and $y$ dependence of the form $[x, y, +x^2, y^2, xy, \dots]$. We will also include cross-validation and +bootstrap as resampling techniques. As in homeworks 1 and 2, we +can use a uniform distribution to set up the arrays of values for $x$ +and $y$, or as in the example below just a fix values for $x$ and $y$ with a given step size. +In this case we will have two predictors and need to fit a +function (for example a polynomial) of $x$ and $y$. Thereafter we will +repeat much of the same procedure using the the Ridge and +Lasso regression methods, introducing thus a dependence on the bias +(penalty) $\lambda$. + + +The Python fucntion for the Franke function is included here (it performs also a three-dimensional plot of it) +\bpycod +from mpl_toolkits.mplot3d import Axes3D +import matplotlib.pyplot as plt +from matplotlib import cm +from matplotlib.ticker import LinearLocator, FormatStrFormatter +import numpy as np +from random import random, seed + +fig = plt.figure() +ax = fig.gca(projection='3d') + +# Make data. +x = np.arange(0, 1, 0.05) +y = np.arange(0, 1, 0.05) +x, y = np.meshgrid(x,y) + + +def FrankeFunction(x,y): + term1 = 0.75*np.exp(-(0.25*(9*x-2)**2) - 0.25*((9*y-2)**2)) + term2 = 0.75*np.exp(-((9*x+1)**2)/49.0 - 0.1*(9*y+1)) + term3 = 0.5*np.exp(-(9*x-7)**2/4.0 - 0.25*((9*y-3)**2)) + term4 = -0.2*np.exp(-(9*x-4)**2 - (9*y-7)**2) + return term1 + term2 + term3 + term4 + + +z = FrankeFunction(x, y) + +# Plot the surface. +surf = ax.plot_surface(x, y, z, cmap=cm.coolwarm, + linewidth=0, antialiased=False) + +# Customize the z axis. +ax.set_zlim(-0.10, 1.40) +ax.zaxis.set_major_locator(LinearLocator(10)) +ax.zaxis.set_major_formatter(FormatStrFormatter('%.02f')) + +# Add a color bar which maps values to colors. +fig.colorbar(surf, shrink=0.5, aspect=5) + +plt.show() + +\epycod + + +We will thus again generate our own dataset for a function $\mathrm{FrankeFunction}(x,y)$ where +$x,y \in [0,1]$ could be defined by random numbers computed with the uniform +distribution. The function $f(x,y)$ is the Franke function. You should explore also the addition +an added stochastic noise to this function using the normal distribution $\cal{N}(0,1)$. + +Write your own code (using either a matrix inversion or a singular value decomposition from e.g., \textbf{numpy} ) or use your code from exercises 1 and 3 +and perform a standard least square regression analysis using polynomials in $x$ and $y$ up to fifth order. Find the confidence intervals of the parameters $\beta$ by computing their variances, evaluate the Mean Squared error (MSE) +\[ MSE(\hat{y},\hat{\tilde{y}}) = \frac{1}{n} +\sum_{i=0}^{n-1}(y_i-\tilde{y}_i)^2, +\] +and the $R^2$ score function. +If $\tilde{\hat{y}}_i$ is the predicted value of the $i-th$ sample and $y_i$ is the corresponding true value, then the score $R^2$ is defined as +\[ +R^2(\hat{y}, \tilde{\hat{y}}) = 1 - \frac{\sum_{i=0}^{n - 1} (y_i - \tilde{y}_i)^2}{\sum_{i=0}^{n - 1} (y_i - \bar{y})^2}, +\] +where we have defined the mean value of $\hat{y}$ as +\[ +\bar{y} = \frac{1}{n} \sum_{i=0}^{n - 1} y_i. +\] + +Perform a resampling of the data where you split the data in training data and test data. Implement the $k$-fold cross-validation algorithm and/or the bootstrap algorithm +and evaluate again the MSE and the $R^2$ functions resulting from the test data. Evaluate also the bias and variance of the final models. + + +Write then your own code for the Ridge method, either using matrix +inversion or the singular value decomposition as done for standard OLS. Perform the same analysis as in the +previous exercise (for the same polynomials and include resampling +techniques) but now for different values of $\lambda$. Compare and +analyze your results with those obtained with standard OLS. Study the +dependence on $\lambda$ while also varying eventually the strength of +the noise in your expression for $\mathrm{FrankeFunction}(x,y)$. + +Then perform the same studies but now with Lasso regression. Use the functionalities of +\textbf{scikit-learn}. Give a critical discussion of the three methods and a +judgement of which model fits the data best. + + +% ------------------- end of main content --------------- + +% #ifdef PREAMBLE +\end{document} +% #endif + diff --git a/doc/src/Regression/Regression.tex b/doc/src/Regression/Regression.tex new file mode 100644 index 000000000..35dc47bdb --- /dev/null +++ b/doc/src/Regression/Regression.tex @@ -0,0 +1,3673 @@ +%% +%% Automatically generated file from DocOnce source +%% (https://github.com/hplgit/doconce/) +%% +%% + + +%-------------------- begin preamble ---------------------- + +\documentclass[% +oneside, % oneside: electronic viewing, twoside: printing +final, % draft: marks overfull hboxes, figures with paths +10pt]{article} + +\listfiles % print all files needed to compile this document + +\usepackage{relsize,makeidx,color,setspace,amsmath,amsfonts,amssymb} +\usepackage[table]{xcolor} +\usepackage{bm,ltablex,microtype} + +\usepackage[pdftex]{graphicx} + +\usepackage{fancyvrb} % packages needed for verbatim environments +\usepackage{minted} +\usemintedstyle{default} + +\usepackage[T1]{fontenc} +%\usepackage[latin1]{inputenc} +\usepackage{ucs} +\usepackage[utf8x]{inputenc} + +\usepackage{lmodern} % Latin Modern fonts derived from Computer Modern + +% Hyperlinks in PDF: +\definecolor{linkcolor}{rgb}{0,0,0.4} +\usepackage{hyperref} +\hypersetup{ + breaklinks=true, + colorlinks=true, + linkcolor=linkcolor, + urlcolor=linkcolor, + citecolor=black, + filecolor=black, + %filecolor=blue, + pdfmenubar=true, + pdftoolbar=true, + bookmarksdepth=3 % Uncomment (and tweak) for PDF bookmarks with more levels than the TOC + } +%\hyperbaseurl{} % hyperlinks are relative to this root + +\setcounter{tocdepth}{2} % levels in table of contents + +% --- fancyhdr package for fancy headers --- +\usepackage{fancyhdr} +\fancyhf{} % sets both header and footer to nothing +\renewcommand{\headrulewidth}{0pt} +\fancyfoot[LE,RO]{\thepage} +% Ensure copyright on titlepage (article style) and chapter pages (book style) +\fancypagestyle{plain}{ + \fancyhf{} + \fancyfoot[C]{{\footnotesize \copyright\ 1999-2019, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license}} +% \renewcommand{\footrulewidth}{0mm} + \renewcommand{\headrulewidth}{0mm} +} +% Ensure copyright on titlepages with \thispagestyle{empty} +\fancypagestyle{empty}{ + \fancyhf{} + \fancyfoot[C]{{\footnotesize \copyright\ 1999-2019, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license}} + \renewcommand{\footrulewidth}{0mm} + \renewcommand{\headrulewidth}{0mm} +} + +\pagestyle{fancy} + + +\usepackage[framemethod=TikZ]{mdframed} + +% --- begin definitions of admonition environments --- + +% --- end of definitions of admonition environments --- + +% prevent orhpans and widows +\clubpenalty = 10000 +\widowpenalty = 10000 + +% --- end of standard preamble for documents --- + + +% insert custom LaTeX commands... + +\raggedbottom +\makeindex +\usepackage[totoc]{idxlayout} % for index in the toc +\usepackage[nottoc]{tocbibind} % for references/bibliography in the toc + +%-------------------- end preamble ---------------------- + +\begin{document} + +% matching end for #ifdef PREAMBLE + +\newcommand{\exercisesection}[1]{\subsection*{#1}} + + +% ------------------- main content ---------------------- + + + +% ----------------- title ------------------------- + +\thispagestyle{empty} + +\begin{center} +{\LARGE\bf +\begin{spacing}{1.25} +Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis +\end{spacing} +} +\end{center} + +% ----------------- author(s) ------------------------- + +\begin{center} +{\bf Morten Hjorth-Jensen${}^{1, 2}$} \\ [0mm] +\end{center} + +\begin{center} +% List of all institutions: +\centerline{{\small ${}^1$Department of Physics, University of Oslo}} +\centerline{{\small ${}^2$Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University}} +\end{center} + +% ----------------- end author(s) ------------------------- + +% --- begin date --- +\begin{center} +Jul 22, 2019 +\end{center} +% --- end date --- + +\vspace{1cm} + + +% !split +\subsection*{Why Linear Regression (aka Ordinary Least Squares and family)} + +Fitting a continuous function with linear parameterization in terms of the parameters $\bm{\beta}$. +\begin{itemize} +\item Method of choice for fitting a continuous function! + +\item Gives an excellent introduction to central Machine Learning features with \textbf{understandable pedagogical} links to other methods like \textbf{Neural Networks}, \textbf{Support Vector Machines} etc + +\item Analytical expression for the fitting parameters $\bm{\beta}$ + +\item Analytical expressions for statistical propertiers like mean values, variances, confidence intervals and more + +\item Analytical relation with probabilistic interpretations + +\item Easy to introduce basic concepts like bias-variance tradeoff, cross-validation, resampling and regularization techniques and many other ML topics + +\item Easy to code! And links well with classification problems and logistic regression and neural networks + +\item Allows for \textbf{easy} hands-on understanding of gradient descent methods + +\item and many more features +\end{itemize} + +\noindent +For more discussions of Ridge and Lasso regression, \href{{https://arxiv.org/abs/1509.09169}}{Wessel van Wieringen's} article is highly recommended. +Similarly, \href{{https://arxiv.org/abs/1803.08823}}{Mehta et al's article} is also recommended. + + +% !split +\subsection*{Regression analysis, overarching aims} + +% --- begin paragraph admon --- +\paragraph{} + +Regression modeling deals with the description of the sampling distribution of a given random variable $y$ and how it varies as function of another variable or a set of such variables $\bm{x} =[x_0, x_1,\dots, x_{n-1}]^T$. +The first variable is called the \textbf{dependent}, the \textbf{outcome} or the \textbf{response} variable while the set of variables $\bm{x}$ is called the independent variable, or the predictor variable or the explanatory variable. + +A regression model aims at finding a likelihood function $p(\bm{y}\vert \bm{x})$, that is the conditional distribution for $\bm{y}$ with a given $\bm{x}$. The estimation of $p(\bm{y}\vert \bm{x})$ is made using a data set with +\begin{itemize} +\item $n$ cases $i = 0, 1, 2, \dots, n-1$ + +\item Response (target, dependent or outcome) variable $y_i$ with $i = 0, 1, 2, \dots, n-1$ + +\item $p$ so-called explanatory (independent or predictor) variables $\bm{x}_i=[x_{i0}, x_{i1}, \dots, x_{ip-1}]$ with $i = 0, 1, 2, \dots, n-1$ and explanatory variables running from $0$ to $p-1$. See below for more explicit examples. +\end{itemize} + +\noindent + The goal of the regression analysis is to extract/exploit relationship between $\bm{y}$ and $\bm{X}$ in or to infer causal dependencies, approximations to the likelihood functions, functional relationships and to make predictions, making fits and many other things. +% --- end paragraph admon --- + + + +% !split +\subsection*{Regression analysis, overarching aims II} + +% --- begin paragraph admon --- +\paragraph{} + + +Consider an experiment in which $p$ characteristics of $n$ samples are +measured. The data from this experiment, for various explanatory variables $p$ are normally represented by a matrix +$\mathbf{X}$. + +The matrix $\mathbf{X}$ is called the \emph{design +matrix}. Additional information of the samples is available in the +form of $\bm{y}$ (also as above). The variable $\bm{y}$ is +generally referred to as the \emph{response variable}. The aim of +regression analysis is to explain $\bm{y}$ in terms of +$\bm{X}$ through a functional relationship like $y_i = +f(\mathbf{X}_{i,\ast})$. When no prior knowledge on the form of +$f(\cdot)$ is available, it is common to assume a linear relationship +between $\bm{X}$ and $\bm{y}$. This assumption gives rise to +the \emph{linear regression model} where $\bm{\beta} = [\beta_0, \ldots, +\beta_{p-1}]^{T}$ are the \emph{regression parameters}. + +Linear regression gives us a set of analytical equations for the parameters $\beta_j$. +% --- end paragraph admon --- + + + + + +% !split +\subsection*{Examples} + +% --- begin paragraph admon --- +\paragraph{} +In order to understand the relation among the predictors $p$, the set of data $n$ and the target (outcome, output etc) $\bm{y}$, +consider the model we discussed for describing nuclear binding energies. + +There we assumed that we could parametrize the data using a polynomial approximation based on the liquid drop model. +Assuming +\[ +BE(A) = a_0+a_1A+a_2A^{2/3}+a_3A^{-1/3}+a_4A^{-1}, +\] +we have five predictors, that is the intercept, the $A$ dependent term, the $A^{2/3}$ term and the $A^{-1/3}$ and $A^{-1}$ terms. +This gives $p=0,1,2,3,4$. Furthermore we have $n$ entries for each predictor. It means that our design matrix is a +$p\times n$ matrix $\bm{X}$. + +Here the predictors are based on a model we have made. A popular data set which is widely encountered in ML applications is the +so-called \href{{https://www.sciencedirect.com/science/article/pii/S0957417407006719?via%3Dihub}}{credit card default data from Taiwan}. The data set contains data on $n=30000$ credit card holders with predictors like gender, marital status, age, profession, education, etc. In total there are $24$ such predictors or attributes leading to a design matrix of dimensionality $24 \times 30000$ +% --- end paragraph admon --- + + + + + + + +% !split +\subsection*{General linear models} + +% --- begin paragraph admon --- +\paragraph{} +Before we proceed let us study a case from linear algebra where we aim at fitting a set of data $\bm{y}=[y_0,y_1,\dots,y_{n-1}]$. We could think of these data as a result of an experiment or a complicated numerical experiment. These data are functions of a series of variables $\bm{x}=[x_0,x_1,\dots,x_{n-1}]$, that is $y_i = y(x_i)$ with $i=0,1,2,\dots,n-1$. The variables $x_i$ could represent physical quantities like time, temperature, position etc. We assume that $y(x)$ is a smooth function. + +Since obtaining these data points may not be trivial, we want to use these data to fit a function which can allow us to make predictions for values of $y$ which are not in the present set. The perhaps simplest approach is to assume we can parametrize our function in terms of a polynomial of degree $n-1$ with $n$ points, that is +\[ +y=y(x) \rightarrow y(x_i)=\tilde{y}_i+\epsilon_i=\sum_{j=0}^{n-1} \beta_j x_i^j+\epsilon_i, +\] +where $\epsilon_i$ is the error in our approximation. +% --- end paragraph admon --- + + + + +% !split +\subsection*{Rewriting the fitting procedure as a linear algebra problem} + +% --- begin paragraph admon --- +\paragraph{} +For every set of values $y_i,x_i$ we have thus the corresponding set of equations +\begin{align*} +y_0&=\beta_0+\beta_1x_0^1+\beta_2x_0^2+\dots+\beta_{n-1}x_0^{n-1}+\epsilon_0\\ +y_1&=\beta_0+\beta_1x_1^1+\beta_2x_1^2+\dots+\beta_{n-1}x_1^{n-1}+\epsilon_1\\ +y_2&=\beta_0+\beta_1x_2^1+\beta_2x_2^2+\dots+\beta_{n-1}x_2^{n-1}+\epsilon_2\\ +\dots & \dots \\ +y_{n-1}&=\beta_0+\beta_1x_{n-1}^1+\beta_2x_{n-1}^2+\dots+\beta_{n-1}x_{n-1}^{n-1}+\epsilon_{n-1}.\\ +\end{align*} +% --- end paragraph admon --- + + + + +% !split +\subsection*{Rewriting the fitting procedure as a linear algebra problem, more details} + +% --- begin paragraph admon --- +\paragraph{} +Defining the vectors +\[ +\bm{y} = [y_0,y_1, y_2,\dots, y_{n-1}]^T, +\] +and +\[ +\bm{\beta} = [\beta_0,\beta_1, \beta_2,\dots, \beta_{n-1}]^T, +\] +and +\[ +\bm{\epsilon} = [\epsilon_0,\epsilon_1, \epsilon_2,\dots, \epsilon_{n-1}]^T, +\] +and the design matrix +\[ +\bm{X}= +\begin{bmatrix} +1& x_{0}^1 &x_{0}^2& \dots & \dots &x_{0}^{n-1}\\ +1& x_{1}^1 &x_{1}^2& \dots & \dots &x_{1}^{n-1}\\ +1& x_{2}^1 &x_{2}^2& \dots & \dots &x_{2}^{n-1}\\ +\dots& \dots &\dots& \dots & \dots &\dots\\ +1& x_{n-1}^1 &x_{n-1}^2& \dots & \dots &x_{n-1}^{n-1}\\ +\end{bmatrix} +\] +we can rewrite our equations as +\[ +\bm{y} = \bm{X}\bm{\beta}+\bm{\epsilon}. +\] +The above design matrix is called a \href{{https://en.wikipedia.org/wiki/Vandermonde_matrix}}{Vandermonde matrix}. +% --- end paragraph admon --- + + + + +% !split +\subsection*{Generalizing the fitting procedure as a linear algebra problem} + +% --- begin paragraph admon --- +\paragraph{} + +We are obviously not limited to the above polynomial expansions. We +could replace the various powers of $x$ with elements of Fourier +series or instead of $x_i^j$ we could have $\cos{(j x_i)}$ or $\sin{(j +x_i)}$, or time series or other orthogonal functions. For every set +of values $y_i,x_i$ we can then generalize the equations to + +\begin{align*} +y_0&=\beta_0x_{00}+\beta_1x_{01}+\beta_2x_{02}+\dots+\beta_{n-1}x_{0n-1}+\epsilon_0\\ +y_1&=\beta_0x_{10}+\beta_1x_{11}+\beta_2x_{12}+\dots+\beta_{n-1}x_{1n-1}+\epsilon_1\\ +y_2&=\beta_0x_{20}+\beta_1x_{21}+\beta_2x_{22}+\dots+\beta_{n-1}x_{2n-1}+\epsilon_2\\ +\dots & \dots \\ +y_{i}&=\beta_0x_{i0}+\beta_1x_{i1}+\beta_2x_{i2}+\dots+\beta_{n-1}x_{in-1}+\epsilon_i\\ +\dots & \dots \\ +y_{n-1}&=\beta_0x_{n-1,0}+\beta_1x_{n-1,2}+\beta_2x_{n-1,2}+\dots+\beta_{n-1}x_{n-1,n-1}+\epsilon_{n-1}.\\ +\end{align*} + +\textbf{Note that we have $p=n$ here. The matrix is symmetric. This is generally not the case!} +% --- end paragraph admon --- + + + + +% !split +\subsection*{Generalizing the fitting procedure as a linear algebra problem} + +% --- begin paragraph admon --- +\paragraph{} +We redefine in turn the matrix $\bm{X}$ as +\[ +\bm{X}= +\begin{bmatrix} +x_{00}& x_{01} &x_{02}& \dots & \dots &x_{0,n-1}\\ +x_{10}& x_{11} &x_{12}& \dots & \dots &x_{1,n-1}\\ +x_{20}& x_{21} &x_{22}& \dots & \dots &x_{2,n-1}\\ +\dots& \dots &\dots& \dots & \dots &\dots\\ +x_{n-1,0}& x_{n-1,1} &x_{n-1,2}& \dots & \dots &x_{n-1,n-1}\\ +\end{bmatrix} +\] +and without loss of generality we rewrite again our equations as +\[ +\bm{y} = \bm{X}\bm{\beta}+\bm{\epsilon}. +\] +The left-hand side of this equation is kwown. Our error vector $\bm{\epsilon}$ and the parameter vector $\bm{\beta}$ are our unknow quantities. How can we obtain the optimal set of $\beta_i$ values? +% --- end paragraph admon --- + + + + +% !split +\subsection*{Optimizing our parameters} + +% --- begin paragraph admon --- +\paragraph{} +We have defined the matrix $\bm{X}$ via the equations +\begin{align*} +y_0&=\beta_0x_{00}+\beta_1x_{01}+\beta_2x_{02}+\dots+\beta_{n-1}x_{0n-1}+\epsilon_0\\ +y_1&=\beta_0x_{10}+\beta_1x_{11}+\beta_2x_{12}+\dots+\beta_{n-1}x_{1n-1}+\epsilon_1\\ +y_2&=\beta_0x_{20}+\beta_1x_{21}+\beta_2x_{22}+\dots+\beta_{n-1}x_{2n-1}+\epsilon_1\\ +\dots & \dots \\ +y_{i}&=\beta_0x_{i0}+\beta_1x_{i1}+\beta_2x_{i2}+\dots+\beta_{n-1}x_{in-1}+\epsilon_1\\ +\dots & \dots \\ +y_{n-1}&=\beta_0x_{n-1,0}+\beta_1x_{n-1,2}+\beta_2x_{n-1,2}+\dots+\beta_{n-1}x_{n-1,n-1}+\epsilon_{n-1}.\\ +\end{align*} + +As we noted above, we stayed with a system with the design matrix + $\bm{X}\in {\mathbb{R}}^{n\times n}$, that is we have $p=n$. For reasons to come later (algorithmic arguments) we will hereafter define +our matrix as $\bm{X}\in {\mathbb{R}}^{n\times p}$, with the predictors refering to the column numbers and the entries $n$ being the row elements. +% --- end paragraph admon --- + + + + +% !split +\subsection*{Our model for the nuclear binding energies} + +In our \href{{https://compphysics.github.io/MachineLearningMSU/doc/pub/Introduction/html/Introduction.html}}{introductory notes} we looked at the so-called \href{{https://en.wikipedia.org/wiki/Semi-empirical_mass_formula}}{liguid drop model}. Let us remind ourselves about what we did by looking at the code. + +We restate the parts of the code we are most interested in. +\begin{minted}[fontsize=\fontsize{9pt}{9pt},linenos=false,mathescape,baselinestretch=1.0,fontfamily=tt,xleftmargin=7mm]{python} +# Common imports +import numpy as np +import pandas as pd +import matplotlib.pyplot as plt +from IPython.display import display +import os + +# Where to save the figures and data files +PROJECT_ROOT_DIR = "Results" +FIGURE_ID = "Results/FigureFiles" +DATA_ID = "DataFiles/" + +if not os.path.exists(PROJECT_ROOT_DIR): + os.mkdir(PROJECT_ROOT_DIR) + +if not os.path.exists(FIGURE_ID): + os.makedirs(FIGURE_ID) + +if not os.path.exists(DATA_ID): + os.makedirs(DATA_ID) + +def image_path(fig_id): + return os.path.join(FIGURE_ID, fig_id) + +def data_path(dat_id): + return os.path.join(DATA_ID, dat_id) + +def save_fig(fig_id): + plt.savefig(image_path(fig_id) + ".png", format='png') + +infile = open(data_path("MassEval2016.dat"),'r') + + +# Read the experimental data with Pandas +Masses = pd.read_fwf(infile, usecols=(2,3,4,6,11), + names=('N', 'Z', 'A', 'Element', 'Ebinding'), + widths=(1,3,5,5,5,1,3,4,1,13,11,11,9,1,2,11,9,1,3,1,12,11,1), + header=39, + index_col=False) + +# Extrapolated values are indicated by '#' in place of the decimal place, so +# the Ebinding column won't be numeric. Coerce to float and drop these entries. +Masses['Ebinding'] = pd.to_numeric(Masses['Ebinding'], errors='coerce') +Masses = Masses.dropna() +# Convert from keV to MeV. +Masses['Ebinding'] /= 1000 + +# Group the DataFrame by nucleon number, A. +Masses = Masses.groupby('A') +# Find the rows of the grouped DataFrame with the maximum binding energy. +Masses = Masses.apply(lambda t: t[t.Ebinding==t.Ebinding.max()]) +A = Masses['A'] +Z = Masses['Z'] +N = Masses['N'] +Element = Masses['Element'] +Energies = Masses['Ebinding'] + +# Now we set up the design matrix X +X = np.zeros((len(A),5)) +X[:,0] = 1 +X[:,1] = A +X[:,2] = A**(2.0/3.0) +X[:,3] = A**(-1.0/3.0) +X[:,4] = A**(-1.0) +# Then nice printout using pandas +DesignMatrix = pd.DataFrame(X) +DesignMatrix.index = A +DesignMatrix.columns = ['1', 'A', 'A^(2/3)', 'A^(-1/3)', '1/A'] +display(DesignMatrix) +\end{minted} + +With $\bm{\beta}\in {\mathbb{R}}^{p\times 1}$, it means that we will hereafter write our equations for the approximation as +\[ +\bm{\tilde{y}}= \bm{X}\bm{\beta}, +\] +throughout these lectures. + + +% !split +\subsection*{Optimizing our parameters, more details} + +% --- begin paragraph admon --- +\paragraph{} +With the above we use the design matrix to define the approximation $\bm{\tilde{y}}$ via the unknown quantity $\bm{\beta}$ as +\[ +\bm{\tilde{y}}= \bm{X}\bm{\beta}, +\] +and in order to find the optimal parameters $\beta_i$ instead of solving the above linear algebra problem, we define a function which gives a measure of the spread between the values $y_i$ (which represent hopefully the exact values) and the parameterized values $\tilde{y}_i$, namely +\[ +C(\bm{\beta})=\frac{1}{n}\sum_{i=0}^{n-1}\left(y_i-\tilde{y}_i\right)^2=\frac{1}{n}\left\{\left(\bm{y}-\bm{\tilde{y}}\right)^T\left(\bm{y}-\bm{\tilde{y}}\right)\right\}, +\] +or using the matrix $\bm{X}$ and in a more compact matrix-vector notation as +\[ +C(\bm{\beta})=\frac{1}{n}\left\{\left(\bm{y}-\bm{X}^T\bm{\beta}\right)^T\left(\bm{y}-\bm{X}^T\bm{\beta}\right)\right\}. +\] +This function is one possible way to define the so-called cost function. + + + +It is also common to define +the function $Q$ as + +\[ +C(\bm{\beta})=\frac{1}{2n}\sum_{i=0}^{n-1}\left(y_i-\tilde{y}_i\right)^2, +\] +since when taking the first derivative with respect to the unknown parameters $\beta$, the factor of $2$ cancels out. +% --- end paragraph admon --- + + + + +% !split +\subsection*{Interpretations and optimizing our parameters} + +% --- begin paragraph admon --- +\paragraph{} + +The function +\[ +C(\bm{\beta})=\frac{1}{n}\left\{\left(\bm{y}-\bm{X}\bm{\beta}\right)^T\left(\bm{y}-\bm{X}\bm{\beta}\right)\right\}, +\] +can be linked to the variance of the quantity $y_i$ if we interpret the latter as the mean value. +When linking below with the maximum likelihood approach below, we will indeed interpret $y_i$ as a mean value (see exercises) +\[ +y_{i}=\langle y_i \rangle = \beta_0x_{i,0}+\beta_1x_{i,1}+\beta_2x_{i,2}+\dots+\beta_{n-1}x_{i,n-1}+\epsilon_i, +\] + +where $\langle y_i \rangle$ is the mean value. Keep in mind also that +till now we have treated $y_i$ as the exact value. Normally, the +response (dependent or outcome) variable $y_i$ the outcome of a +numerical experiment or another type of experiment and is thus only an +approximation to the true value. It is then always accompanied by an +error estimate, often limited to a statistical error estimate given by +the standard deviation discussed earlier. In the discussion here we +will treat $y_i$ as our exact value for the response variable. + +In order to find the parameters $\beta_i$ we will then minimize the spread of $C(\bm{\beta})$, that is we are going to solve the problem +\[ +{\displaystyle \min_{\bm{\beta}\in +{\mathbb{R}}^{p}}}\frac{1}{n}\left\{\left(\bm{y}-\bm{X}\bm{\beta}\right)^T\left(\bm{y}-\bm{X}\bm{\beta}\right)\right\}. +\] +In practical terms it means we will require +\[ +\frac{\partial C(\bm{\beta})}{\partial \beta_j} = \frac{\partial }{\partial \beta_j}\left[ \frac{1}{n}\sum_{i=0}^{n-1}\left(y_i-\beta_0x_{i,0}-\beta_1x_{i,1}-\beta_2x_{i,2}-\dots-\beta_{n-1}x_{i,n-1}\right)^2\right]=0, +\] +which results in +\[ +\frac{\partial C(\bm{\beta})}{\partial \beta_j} = -\frac{2}{n}\left[ \sum_{i=0}^{n-1}x_{ij}\left(y_i-\beta_0x_{i,0}-\beta_1x_{i,1}-\beta_2x_{i,2}-\dots-\beta_{n-1}x_{i,n-1}\right)\right]=0, +\] +or in a matrix-vector form as +\[ +\frac{\partial C(\bm{\beta})}{\partial \bm{\beta}} = 0 = \bm{X}^T\left( \bm{y}-\bm{X}\bm{\beta}\right). +\] +% --- end paragraph admon --- + + + + +% !split +\subsection*{Interpretations and optimizing our parameters} + +% --- begin paragraph admon --- +\paragraph{} +We can rewrite +\[ +\frac{\partial C(\bm{\beta})}{\partial \bm{\beta}} = 0 = \bm{X}^T\left( \bm{y}-\bm{X}\bm{\beta}\right), +\] +as +\[ +\bm{X}^T\bm{y} = \bm{X}^T\bm{X}\bm{\beta}, +\] +and if the matrix $\bm{X}^T\bm{X}$ is invertible we have the solution +\[ +\bm{\beta} =\left(\bm{X}^T\bm{X}\right)^{-1}\bm{X}^T\bm{y}. +\] + +We note also that since our design matrix is defined as $\bm{X}\in +{\mathbb{R}}^{n\times p}$, the product $\bm{X}^T\bm{X} \in +{\mathbb{R}}^{p\times p}$. In the above case we have that $p \ll n$, +in our case $p=5$ meaning that we end up with inverting a small +$5\times 5$ matrix. This is a rather common situation, in many cases we end up with low-dimensional +matrices to invert. The methods discussed here and for many other +supervised learning algorithms like classification with logistic +regression or support vector machines, exhibit dimensionalities which +allow for the usage of direct linear algebra methods such as \textbf{LU} decomposition or \textbf{Singular Value Decomposition} (SVD) for finding the inverse of the matrix +$\bm{X}^T\bm{X}$. +% --- end paragraph admon --- + + + +% !split +\subsection*{Interpretations and optimizing our parameters} + +% --- begin paragraph admon --- +\paragraph{} +The residuals $\bm{\epsilon}$ are in turn given by +\[ +\bm{\epsilon} = \bm{y}-\bm{\tilde{y}} = \bm{y}-\bm{X}\bm{\beta}, +\] +and with +\[ +\bm{X}^T\left( \bm{y}-\bm{X}\bm{\beta}\right)= 0, +\] +we have +\[ +\bm{X}^T\bm{\epsilon}=\bm{X}^T\left( \bm{y}-\bm{X}\bm{\beta}\right)= 0, +\] +meaning that the solution for $\bm{\beta}$ is the one which minimizes the residuals. Later we will link this with the maximum likelihood approach. +% --- end paragraph admon --- + + + + +Let us now return to our nuclear binding energies and simply code the above equations. + +% !split +\subsection*{Own code for Ordinary Least Squares} + +It is rather straightforward to implement the matrix inversion and obtain the parameters $\bm{\beta}$. After having defined the matrix $\bm{X}$ we simply need to +write +\begin{minted}[fontsize=\fontsize{9pt}{9pt},linenos=false,mathescape,baselinestretch=1.0,fontfamily=tt,xleftmargin=7mm]{python} +# matrix inversion to find beta +beta = np.linalg.inv(X.T.dot(X)).dot(X.T).dot(Energies) +# and then make the prediction +ytilde = X @ beta +\end{minted} +Alternatively, you can use the least squares functionality in \textbf{Numpy} as +\begin{minted}[fontsize=\fontsize{9pt}{9pt},linenos=false,mathescape,baselinestretch=1.0,fontfamily=tt,xleftmargin=7mm]{python} +fit = np.linalg.lstsq(X, Energies, rcond =None)[0] +ytildenp = np.dot(fit,X.T) +\end{minted} + +And finally we plot our fit with and compare with data +\begin{minted}[fontsize=\fontsize{9pt}{9pt},linenos=false,mathescape,baselinestretch=1.0,fontfamily=tt,xleftmargin=7mm]{python} +Masses['Eapprox'] = ytilde +# Generate a plot comparing the experimental with the fitted values values. +fig, ax = plt.subplots() +ax.set_xlabel(r'$A = N + Z$') +ax.set_ylabel(r'$E_\mathrm{bind}\,/\mathrm{MeV}$') +ax.plot(Masses['A'], Masses['Ebinding'], alpha=0.7, lw=2, + label='Ame2016') +ax.plot(Masses['A'], Masses['Eapprox'], alpha=0.7, lw=2, c='m', + label='Fit') +ax.legend() +save_fig("Masses2016OLS") +plt.show() +\end{minted} + +% !split +\subsection*{Adding error analysis and training set up} + +We can easily test our fit by computing the $R2$ score that we discussed in connection with the functionality of _Scikit_Learn_ in the introductory slides. +Since we are not using _Scikit-Learn here we can define our own $R2$ function as +\begin{minted}[fontsize=\fontsize{9pt}{9pt},linenos=false,mathescape,baselinestretch=1.0,fontfamily=tt,xleftmargin=7mm]{python} +def R2(y_data, y_model): + return 1 - np.sum((y_data - y_model) ** 2) / np.sum((y_data - np.mean(y_model)) ** 2) +\end{minted} +and we would be using it as +\begin{minted}[fontsize=\fontsize{9pt}{9pt},linenos=false,mathescape,baselinestretch=1.0,fontfamily=tt,xleftmargin=7mm]{python} +print(R2(Energies,ytilde)) +\end{minted} + +We can easily add our \textbf{MSE} score as +\begin{minted}[fontsize=\fontsize{9pt}{9pt},linenos=false,mathescape,baselinestretch=1.0,fontfamily=tt,xleftmargin=7mm]{python} +def MSE(y_data,y_model): + n = np.size(y_model) + return np.sum((y_data-y_model)**2)/n + +print(MSE(Energies,ytilde)) +\end{minted} +and finally the relative error as +\begin{minted}[fontsize=\fontsize{9pt}{9pt},linenos=false,mathescape,baselinestretch=1.0,fontfamily=tt,xleftmargin=7mm]{python} +def RelativeError(y_data,y_model): + return abs((y_data-y_model)/y_data) +print(RelativeError(Energies, ytilde)) +\end{minted} + + + + + +% !split +\subsection*{The $\chi^2$ function} + +% --- begin paragraph admon --- +\paragraph{} + +Normally, the response (dependent or outcome) variable $y_i$ is the +outcome of a numerical experiment or another type of experiment and is +thus only an approximation to the true value. It is then always +accompanied by an error estimate, often limited to a statistical error +estimate given by the standard deviation discussed earlier. In the +discussion here we will treat $y_i$ as our exact value for the +response variable. + +Introducing the standard deviation $\sigma_i$ for each measurement +$y_i$, we define now the $\chi^2$ function (omitting the $1/n$ term) +as + +\[ +\chi^2(\bm{\beta})=\frac{1}{n}\sum_{i=0}^{n-1}\frac{\left(y_i-\tilde{y}_i\right)^2}{\sigma_i^2}=\frac{1}{n}\left\{\left(\bm{y}-\bm{\tilde{y}}\right)^T\frac{1}{\bm{\Sigma^2}}\left(\bm{y}-\bm{\tilde{y}}\right)\right\}, +\] +where the matrix $\bm{\Sigma}$ is a diagonal matrix with $\sigma_i$ as matrix elements. +% --- end paragraph admon --- + + + +% !split +\subsection*{The $\chi^2$ function} + +% --- begin paragraph admon --- +\paragraph{} + +In order to find the parameters $\beta_i$ we will then minimize the spread of $\chi^2(\bm{\beta})$ by requiring +\[ +\frac{\partial \chi^2(\bm{\beta})}{\partial \beta_j} = \frac{\partial }{\partial \beta_j}\left[ \frac{1}{n}\sum_{i=0}^{n-1}\left(\frac{y_i-\beta_0x_{i,0}-\beta_1x_{i,1}-\beta_2x_{i,2}-\dots-\beta_{n-1}x_{i,n-1}}{\sigma_i}\right)^2\right]=0, +\] +which results in +\[ +\frac{\partial \chi^2(\bm{\beta})}{\partial \beta_j} = -\frac{2}{n}\left[ \sum_{i=0}^{n-1}\frac{x_{ij}}{\sigma_i}\left(\frac{y_i-\beta_0x_{i,0}-\beta_1x_{i,1}-\beta_2x_{i,2}-\dots-\beta_{n-1}x_{i,n-1}}{\sigma_i}\right)\right]=0, +\] +or in a matrix-vector form as +\[ +\frac{\partial \chi^2(\bm{\beta})}{\partial \bm{\beta}} = 0 = \bm{A}^T\left( \bm{b}-\bm{A}\bm{\beta}\right). +\] +where we have defined the matrix $\bm{A} =\bm{X}/\bm{\Sigma}$ with matrix elements $a_{ij} = x_{ij}/\sigma_i$ and the vector $\bm{b}$ with elements $b_i = y_i/\sigma_i$. +% --- end paragraph admon --- + + + +% !split +\subsection*{The $\chi^2$ function} + +% --- begin paragraph admon --- +\paragraph{} + +We can rewrite +\[ +\frac{\partial \chi^2(\bm{\beta})}{\partial \bm{\beta}} = 0 = \bm{A}^T\left( \bm{b}-\bm{A}\bm{\beta}\right), +\] +as +\[ +\bm{A}^T\bm{b} = \bm{A}^T\bm{A}\bm{\beta}, +\] +and if the matrix $\bm{A}^T\bm{A}$ is invertible we have the solution +\[ +\bm{\beta} =\left(\bm{A}^T\bm{A}\right)^{-1}\bm{A}^T\bm{b}. +\] +% --- end paragraph admon --- + + + +% !split +\subsection*{The $\chi^2$ function} + +% --- begin paragraph admon --- +\paragraph{} + +If we then introduce the matrix +\[ +\bm{H} = \left(\bm{A}^T\bm{A}\right)^{-1}, +\] +we have then the following expression for the parameters $\beta_j$ (the matrix elements of $\bm{H}$ are $h_{ij}$) +\[ +\beta_j = \sum_{k=0}^{p-1}h_{jk}\sum_{i=0}^{n-1}\frac{y_i}{\sigma_i}\frac{x_{ik}}{\sigma_i} = \sum_{k=0}^{p-1}h_{jk}\sum_{i=0}^{n-1}b_ia_{ik} +\] +We state without proof the expression for the uncertainty in the parameters $\beta_j$ as (we leave this as an exercise) +\[ +\sigma^2(\beta_j) = \sum_{i=0}^{n-1}\sigma_i^2\left( \frac{\partial \beta_j}{\partial y_i}\right)^2, +\] +resulting in +\[ +\sigma^2(\beta_j) = \left(\sum_{k=0}^{p-1}h_{jk}\sum_{i=0}^{n-1}a_{ik}\right)\left(\sum_{l=0}^{p-1}h_{jl}\sum_{m=0}^{n-1}a_{ml}\right) = h_{jj}! +\] +% --- end paragraph admon --- + + + +% !split +\subsection*{The $\chi^2$ function} + +% --- begin paragraph admon --- +\paragraph{} +The first step here is to approximate the function $y$ with a first-order polynomial, that is we write +\[ +y=y(x) \rightarrow y(x_i) \approx \beta_0+\beta_1 x_i. +\] +By computing the derivatives of $\chi^2$ with respect to $\beta_0$ and $\beta_1$ show that these are given by +\[ +\frac{\partial \chi^2(\bm{\beta})}{\partial \beta_0} = -2\left[ \frac{1}{n}\sum_{i=0}^{n-1}\left(\frac{y_i-\beta_0-\beta_1x_{i}}{\sigma_i^2}\right)\right]=0, +\] +and +\[ +\frac{\partial \chi^2(\bm{\beta})}{\partial \beta_1} = -\frac{2}{n}\left[ \sum_{i=0}^{n-1}x_i\left(\frac{y_i-\beta_0-\beta_1x_{i}}{\sigma_i^2}\right)\right]=0. +\] +% --- end paragraph admon --- + + + +% !split +\subsection*{The $\chi^2$ function} + +% --- begin paragraph admon --- +\paragraph{} + +For a linear fit (a first-order polynomial) we don't need to invert a matrix!! +Defining +\[ +\gamma = \sum_{i=0}^{n-1}\frac{1}{\sigma_i^2}, +\] + +\[ +\gamma_x = \sum_{i=0}^{n-1}\frac{x_{i}}{\sigma_i^2}, +\] + +\[ +\gamma_y = \sum_{i=0}^{n-1}\left(\frac{y_i}{\sigma_i^2}\right), +\] + +\[ +\gamma_{xx} = \sum_{i=0}^{n-1}\frac{x_ix_{i}}{\sigma_i^2}, +\] + +\[ +\gamma_{xy} = \sum_{i=0}^{n-1}\frac{y_ix_{i}}{\sigma_i^2}, +\] + +we obtain + +\[ +\beta_0 = \frac{\gamma_{xx}\gamma_y-\gamma_x\gamma_y}{\gamma\gamma_{xx}-\gamma_x^2}, +\] + +\[ +\beta_1 = \frac{\gamma_{xy}\gamma-\gamma_x\gamma_y}{\gamma\gamma_{xx}-\gamma_x^2}. +\] + +This approach (different linear and non-linear regression) suffers +often from both being underdetermined and overdetermined in the +unknown coefficients $\beta_i$. A better approach is to use the +Singular Value Decomposition (SVD) method discussed below. Or using +Lasso and Ridge regression. See below. +% --- end paragraph admon --- + + + + +% !split +\subsection*{Fitting an Equation of State for Dense Nuclear Matter} + +Before we continue, let us introduce yet another example. We are going to fit the +nuclear equation of state using results from many-body calculations. +The equation of state we have made available here, as function of +density, has been derived using modern nucleon-nucleon potentials with +\href{{https://www.sciencedirect.com/science/article/pii/S0370157399001106}}{the addition of three-body +forces}. This +time the file is presented as a standard \textbf{csv} file. + +The beginning of the Python code here is similar to what you have seen before, +with the same initializations and declarations. We use also \textbf{pandas} +again, rather extensively in order to organize our data. + +The difference now is that we use \textbf{Scikit-Learn's} regression tools +instead of our own matrix inversion implementation. Furthermore, we +sneak in \textbf{Ridge} regression (to be discussed below) which includes a +hyperparameter $\lambda$, also to be explained below. + +% !split +\subsection*{The code} + +\begin{minted}[fontsize=\fontsize{9pt}{9pt},linenos=false,mathescape,baselinestretch=1.0,fontfamily=tt,xleftmargin=7mm]{python} +# Common imports +import os +import numpy as np +import pandas as pd +import matplotlib.pyplot as plt +import matplotlib.pyplot as plt +import sklearn.linear_model as skl +from sklearn.metrics import mean_squared_error, r2_score, mean_absolute_error + +# Where to save the figures and data files +PROJECT_ROOT_DIR = "Results" +FIGURE_ID = "Results/FigureFiles" +DATA_ID = "DataFiles/" + +if not os.path.exists(PROJECT_ROOT_DIR): + os.mkdir(PROJECT_ROOT_DIR) + +if not os.path.exists(FIGURE_ID): + os.makedirs(FIGURE_ID) + +if not os.path.exists(DATA_ID): + os.makedirs(DATA_ID) + +def image_path(fig_id): + return os.path.join(FIGURE_ID, fig_id) + +def data_path(dat_id): + return os.path.join(DATA_ID, dat_id) + +def save_fig(fig_id): + plt.savefig(image_path(fig_id) + ".png", format='png') + +infile = open(data_path("EoS.csv"),'r') + +# Read the EoS data as csv file and organize the data into two arrays with density and energies +EoS = pd.read_csv(infile, names=('Density', 'Energy')) +EoS['Energy'] = pd.to_numeric(EoS['Energy'], errors='coerce') +EoS = EoS.dropna() +Energies = EoS['Energy'] +Density = EoS['Density'] +# The design matrix now as function of various polytrops +X = np.zeros((len(Density),4)) +X[:,3] = Density**(4.0/3.0) +X[:,2] = Density +X[:,1] = Density**(2.0/3.0) +X[:,0] = 1 + +# We use now Scikit-Learn's linear regressor and ridge regressor +# OLS part +clf = skl.LinearRegression().fit(X, Energies) +ytilde = clf.predict(X) +EoS['Eols'] = ytilde +# The mean squared error +print("Mean squared error: %.2f" % mean_squared_error(Energies, ytilde)) +# Explained variance score: 1 is perfect prediction +print('Variance score: %.2f' % r2_score(Energies, ytilde)) +# Mean absolute error +print('Mean absolute error: %.2f' % mean_absolute_error(Energies, ytilde)) +print(clf.coef_, clf.intercept_) + +# The Ridge regression with a hyperparameter lambda = 0.1 +_lambda = 0.1 +clf_ridge = skl.Ridge(alpha=_lambda).fit(X, Energies) +yridge = clf_ridge.predict(X) +EoS['Eridge'] = yridge +# The mean squared error +print("Mean squared error: %.2f" % mean_squared_error(Energies, yridge)) +# Explained variance score: 1 is perfect prediction +print('Variance score: %.2f' % r2_score(Energies, yridge)) +# Mean absolute error +print('Mean absolute error: %.2f' % mean_absolute_error(Energies, yridge)) +print(clf_ridge.coef_, clf_ridge.intercept_) + +fig, ax = plt.subplots() +ax.set_xlabel(r'$\rho[\mathrm{fm}^{-3}]$') +ax.set_ylabel(r'Energy per particle') +ax.plot(EoS['Density'], EoS['Energy'], alpha=0.7, lw=2, + label='Theoretical data') +ax.plot(EoS['Density'], EoS['Eols'], alpha=0.7, lw=2, c='m', + label='OLS') +ax.plot(EoS['Density'], EoS['Eridge'], alpha=0.7, lw=2, c='g', + label='Ridge $\lambda = 0.1$') +ax.legend() +save_fig("EoSfitting") +plt.show() +\end{minted} + +The above simple polynomial in density $\rho$ gives an excellent fit +to the data. Can you give an interpretation of the various powers of $\rho$? + +We note also that there is a small deviation between the +standard OLS and the Ridge regression at higher densities. We discuss this in more detail +below. + + +% !split +\subsection*{Splitting our Data in Training and Test data} + +It is normal in essentially all Machine Learning studies to split the +data in a training set and a test set (sometimes also an additional +validation set). \textbf{Scikit-Learn} has an own function for this. There +is no explicit recipe for how much data should be included as training +data and say test data. An accepted rule of thumb is to use +approximately $2/3$ to $4/5$ of the data as training data. We will +postpone a discussion of this splitting to the end of these notes and +our discussion of the so-called \textbf{bias-variance} tradeoff. Here we +limit ourselves to repeat the above equation of state fitting example +but now splitting the data into a training set and a test set. + +\begin{minted}[fontsize=\fontsize{9pt}{9pt},linenos=false,mathescape,baselinestretch=1.0,fontfamily=tt,xleftmargin=7mm]{python} +import os +import numpy as np +import pandas as pd +import matplotlib.pyplot as plt +from sklearn.model_selection import train_test_split +# Where to save the figures and data files +PROJECT_ROOT_DIR = "Results" +FIGURE_ID = "Results/FigureFiles" +DATA_ID = "DataFiles/" + +if not os.path.exists(PROJECT_ROOT_DIR): + os.mkdir(PROJECT_ROOT_DIR) + +if not os.path.exists(FIGURE_ID): + os.makedirs(FIGURE_ID) + +if not os.path.exists(DATA_ID): + os.makedirs(DATA_ID) + +def image_path(fig_id): + return os.path.join(FIGURE_ID, fig_id) + +def data_path(dat_id): + return os.path.join(DATA_ID, dat_id) + +def save_fig(fig_id): + plt.savefig(image_path(fig_id) + ".png", format='png') + +def R2(y_data, y_model): + return 1 - np.sum((y_data - y_model) ** 2) / np.sum((y_data - np.mean(y_model)) ** 2) +def MSE(y_data,y_model): + n = np.size(y_model) + return np.sum((y_data-y_model)**2)/n + +infile = open(data_path("EoS.csv"),'r') + +# Read the EoS data as csv file and organized into two arrays with density and energies +EoS = pd.read_csv(infile, names=('Density', 'Energy')) +EoS['Energy'] = pd.to_numeric(EoS['Energy'], errors='coerce') +EoS = EoS.dropna() +Energies = EoS['Energy'] +Density = EoS['Density'] +# The design matrix now as function of various polytrops +X = np.zeros((len(Density),5)) +X[:,0] = 1 +X[:,1] = Density**(2.0/3.0) +X[:,2] = Density +X[:,3] = Density**(4.0/3.0) +X[:,4] = Density**(5.0/3.0) +# We split the data in test and training data +X_train, X_test, y_train, y_test = train_test_split(X, Energies, test_size=0.2) +# matrix inversion to find beta +beta = np.linalg.inv(X_train.T.dot(X_train)).dot(X_train.T).dot(y_train) +# and then make the prediction +ytilde = X_train @ beta +print("Training R2") +print(R2(y_train,ytilde)) +print("Training MSE") +print(MSE(y_train,ytilde)) +ypredict = X_test @ beta +print("Test R2") +print(R2(y_test,ypredict)) +print("Test MSE") +print(MSE(y_test,ypredict)) +\end{minted} + + +% !split +\subsection*{The singular value decomposition} + + +% --- begin paragraph admon --- +\paragraph{} + +The examples we have looked at so far are cases where we normally can +invert the matrix $\bm{X}^T\bm{X}$. Using a polynomial expansion as we +did both for the masses and the fitting of the equation of state, +leads to row vectors of the design matrix which are essentially +orthogonal due to the polynomial character of our model. This may +however not the be case in general and a standard matrix inversion +algorithm based on say LU decomposition may lead to singularities. We will see an example of this below when we try to fit +the coupling constant of the widely used Ising model. +There is however a way to partially circumvent this problem and also gain some insight about the ordinary least squares approach. + +This is given by the \textbf{Singular Value Decomposition} algorithm, perhaps +the most powerful linear algebra algorithm. Let us look at a +different example where we may have problems with the standard matrix +inversion algorithm. Thereafter we dive into the math of the SVD. +% --- end paragraph admon --- + + + +% !split +\subsection*{The Ising model} + +The one-dimensional Ising model with nearest neighbor interaction, no +external field and a constant coupling constant $J$ is given by + +\begin{align} + H = -J \sum_{k}^L s_k s_{k + 1}, +\end{align} + +where $s_i \in \{-1, 1\}$ and $s_{N + 1} = s_1$. The number of spins +in the system is determined by $L$. For the one-dimensional system +there is no phase transition. + +We will look at a system of $L = 40$ spins with a coupling constant of +$J = 1$. To get enough training data we will generate 10000 states +with their respective energies. + + +\begin{minted}[fontsize=\fontsize{9pt}{9pt},linenos=false,mathescape,baselinestretch=1.0,fontfamily=tt,xleftmargin=7mm]{python} +import numpy as np +import matplotlib.pyplot as plt +from mpl_toolkits.axes_grid1 import make_axes_locatable +import seaborn as sns +import scipy.linalg as scl +from sklearn.model_selection import train_test_split +import tqdm +sns.set(color_codes=True) +cmap_args=dict(vmin=-1., vmax=1., cmap='seismic') + +L = 40 +n = int(1e4) + +spins = np.random.choice([-1, 1], size=(n, L)) +J = 1.0 + +energies = np.zeros(n) + +for i in range(n): + energies[i] = - J * np.dot(spins[i], np.roll(spins[i], 1)) +\end{minted} + +Here we use ordinary least squares +regression to predict the energy for the nearest neighbor +one-dimensional Ising model on a ring, i.e., the endpoints wrap +around. We will use linear regression to fit a value for +the coupling constant to achieve this. + +% !split +\subsection*{Reformulating the problem to suit regression} + +A more general form for the one-dimensional Ising model is + +\begin{align} + H = - \sum_j^L \sum_k^L s_j s_k J_{jk}. +\end{align} + +Here we allow for interactions beyond the nearest neighbors and a state dependent +coupling constant. This latter expression can be formulated as +a matrix-product +\begin{align} + \bm{H} = \bm{X} J, +\end{align} + +where $X_{jk} = s_j s_k$ and $J$ is a matrix which consists of the +elements $-J_{jk}$. This form of writing the energy fits perfectly +with the form utilized in linear regression, that is + +\begin{align} + \bm{y} = \bm{X}\bm{\beta} + \bm{\epsilon}, +\end{align} + +We split the data in training and test data as discussed in the previous example + +\begin{minted}[fontsize=\fontsize{9pt}{9pt},linenos=false,mathescape,baselinestretch=1.0,fontfamily=tt,xleftmargin=7mm]{python} +X = np.zeros((n, L ** 2)) +for i in range(n): + X[i] = np.outer(spins[i], spins[i]).ravel() +y = energies +X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2) +\end{minted} + +% !split +\subsection*{Linear regression} + +In the ordinary least squares method we choose the cost function + +\begin{align} + C(\bm{X}, \bm{\beta})= \frac{1}{n}\left\{(\bm{X}\bm{\beta} - \bm{y})^T(\bm{X}\bm{\beta} - \bm{y})\right\}. +\end{align} + +We then find the extremal point of $C$ by taking the derivative with respect to $\bm{\beta}$ as discussed above. +This yields the expression for $\bm{\beta}$ to be + +\[ + \bm{\beta} = \frac{\bm{X}^T \bm{y}}{\bm{X}^T \bm{X}}, +\] + +which immediately imposes some requirements on $\bm{X}$ as there must exist +an inverse of $\bm{X}^T \bm{X}$. If the expression we are modeling contains an +intercept, i.e., a constant term, we must make sure that the +first column of $\bm{X}$ consists of $1$. We do this here + +\begin{minted}[fontsize=\fontsize{9pt}{9pt},linenos=false,mathescape,baselinestretch=1.0,fontfamily=tt,xleftmargin=7mm]{python} +X_train_own = np.concatenate( + (np.ones(len(X_train))[:, np.newaxis], X_train), + axis=1 +) +X_test_own = np.concatenate( + (np.ones(len(X_test))[:, np.newaxis], X_test), + axis=1 +) +\end{minted} + +\begin{minted}[fontsize=\fontsize{9pt}{9pt},linenos=false,mathescape,baselinestretch=1.0,fontfamily=tt,xleftmargin=7mm]{python} +def ols_inv(x: np.ndarray, y: np.ndarray) -> np.ndarray: + return scl.inv(x.T @ x) @ (x.T @ y) +beta = ols_inv(X_train_own, y_train) +\end{minted} + + +% !split +\subsection*{Singular Value decomposition} + +Doing the inversion directly turns out to be a bad idea since the matrix +$\bm{X}^T\bm{X}$ is singular. An alternative approach is to use the \textbf{singular +value decomposition}. Using the definition of the Moore-Penrose +pseudoinverse we can write the equation for $\bm{\beta}$ as + +\[ + \bm{\beta} = \bm{X}^{+}\bm{y}, +\] + +where the pseudoinverse of $\bm{X}$ is given by + +\[ + \bm{X}^{+} = \frac{\bm{X}^T}{\bm{X}^T\bm{X}}. +\] + +Using singular value decomposition we can decompose the matrix $\bm{X} = \bm{U}\bm{\Sigma} \bm{V}^T$, +where $\bm{U}$ and $\bm{V}$ are orthogonal(unitary) matrices and $\bm{\Sigma}$ contains the singular values (more details below). +where $X^{+} = V\Sigma^{+} U^T$. This reduces the equation for +$\omega$ to +\begin{align} + \bm{\beta} = \bm{V}\bm{\Sigma}^{+} \bm{U}^T \bm{y}. +\end{align} + +Note that solving this equation by actually doing the pseudoinverse +(which is what we will do) is not a good idea as this operation scales +as $\mathcal{O}(n^3)$, where $n$ is the number of elements in a +general matrix. Instead, doing $QR$-factorization and solving the +linear system as an equation would reduce this down to +$\mathcal{O}(n^2)$ operations. + + +\begin{minted}[fontsize=\fontsize{9pt}{9pt},linenos=false,mathescape,baselinestretch=1.0,fontfamily=tt,xleftmargin=7mm]{python} +def ols_svd(x: np.ndarray, y: np.ndarray) -> np.ndarray: + u, s, v = scl.svd(x) + return v.T @ scl.pinv(scl.diagsvd(s, u.shape[0], v.shape[0])) @ u.T @ y +\end{minted} + +\begin{minted}[fontsize=\fontsize{9pt}{9pt},linenos=false,mathescape,baselinestretch=1.0,fontfamily=tt,xleftmargin=7mm]{python} +beta = ols_svd(X_train_own,y_train) +\end{minted} + +When extracting the $J$-matrix we need to make sure that we remove the intercept, as is done here + +\begin{minted}[fontsize=\fontsize{9pt}{9pt},linenos=false,mathescape,baselinestretch=1.0,fontfamily=tt,xleftmargin=7mm]{python} +J = beta[1:].reshape(L, L) +\end{minted} + +A way of looking at the coefficients in $J$ is to plot the matrices as images. + + +\begin{minted}[fontsize=\fontsize{9pt}{9pt},linenos=false,mathescape,baselinestretch=1.0,fontfamily=tt,xleftmargin=7mm]{python} +fig = plt.figure(figsize=(20, 14)) +im = plt.imshow(J, **cmap_args) +plt.title("OLS", fontsize=18) +plt.xticks(fontsize=18) +plt.yticks(fontsize=18) +cb = fig.colorbar(im) +cb.ax.set_yticklabels(cb.ax.get_yticklabels(), fontsize=18) +plt.show() +\end{minted} +It is interesting to note that OLS +considers both $J_{j, j + 1} = -0.5$ and $J_{j, j - 1} = -0.5$ as +valid matrix elements for $J$. +In our discussion below on hyperparameters and Ridge and Lasso regression we will see that +this problem can be removed, partly and only with Lasso regression. + +In this case our matrix inversion was actually possible. The obvious question now is what is the mathematics behind the SVD? + + +% !split +\subsection*{Linear Regression Problems} + +One of the typical problems we encounter with linear regression, in particular +when the matrix $\bm{X}$ (our so-called design matrix) is high-dimensional, +are problems with near singular or singular matrices. The column vectors of $\bm{X}$ +may be linearly dependent, normally referred to as super-collinearity. +This means that the matrix may be rank deficient and it is basically impossible to +to model the data using linear regression. As an example, consider the matrix +\begin{align*} +\mathbf{X} & = \left[ +\begin{array}{rrr} +1 & -1 & 2 +\\ +1 & 0 & 1 +\\ +1 & 2 & -1 +\\ +1 & 1 & 0 +\end{array} \right] +\end{align*} + +The columns of $\bm{X}$ are linearly dependent. We see this easily since the +the first column is the row-wise sum of the other two columns. The rank (more correct, +the column rank) of a matrix is the dimension of the space spanned by the +column vectors. Hence, the rank of $\mathbf{X}$ is equal to the number +of linearly independent columns. In this particular case the matrix has rank 2. + +Super-collinearity of an $(n \times p)$-dimensional design matrix $\mathbf{X}$ implies +that the inverse of the matrix $\bm{X}^T\bm{x}$ (the matrix we need to invert to solve the linear regression equations) is non-invertible. If we have a square matrix that does not have an inverse, we say this matrix singular. The example here demonstrates this +\begin{align*} +\bm{X} & = \left[ +\begin{array}{rr} +1 & -1 +\\ +1 & -1 +\end{array} \right]. +\end{align*} +We see easily that $\mbox{det}(\bm{X}) = x_{11} x_{22} - x_{12} x_{21} = 1 \times (-1) - 1 \times (-1) = 0$. Hence, $\mathbf{X}$ is singular and its inverse is undefined. +This is equivalent to saying that the matrix $\bm{X}$ has at least an eigenvalue which is zero. + + +% !split +\subsection*{Fixing the singularity} + +If our design matrix $\bm{X}$ which enters the linear regression problem +\begin{align} +\bm{\beta} & = (\bm{X}^{T} \bm{X})^{-1} \bm{X}^{T} \bm{y}, +\end{align} +has linearly dependent column vectors, we will not be able to compute the inverse +of $\bm{X}^T\bm{X}$ and we cannot find the parameters (estimators) $\beta_i$. +The estimators are only well-defined if $(\bm{X}^{T}\bm{X})^{-1}$ exits. +This is more likely to happen when the matrix $\bm{X}$ is high-dimensional. In this case it is likely to encounter a situation where +the regression parameters $\beta_i$ cannot be estimated. + +A cheap \emph{ad hoc} approach is simply to add a small diagonal component to the matrix to invert, that is we change +\[ +\bm{X}^{T} \bm{X} \rightarrow \bm{X}^{T} \bm{X}+\lambda \bm{I}, +\] +where $\bm{I}$ is the identity matrix. When we discuss \textbf{Ridge} regression this is actually what we end up evaluating. The parameter $\lambda$ is called a hyperparameter. More about this later. + + + +% !split +\subsection*{Basic math of the SVD} + + +From standard linear algebra we know that a square matrix $\bm{X}$ can be diagonalized if and only it is +a so-called \href{{https://en.wikipedia.org/wiki/Normal_matrix}}{normal matrix}, that is if $\bm{X}\in {\mathbb{R}}^{n\times n}$ +we have $\bm{X}\bm{X}^T=\bm{X}^T\bm{X}$ or if $\bm{X}\in {\mathbb{C}}^{n\times n}$ we have $\bm{X}\bm{X}^{\dagger}=\bm{X}^{\dagger}\bm{X}$. +The matrix has then a set of eigenpairs + +\[ +(\lambda_1,\bm{u}_1),\dots, (\lambda_n,\bm{u}_n), +and the eigenvalues are given by the diagonal matrix +\[ +\bm{\Sigma}=\mathrm{Diag}(\lambda_1, \dots,\lambda_n). +\] +The matrix $\bm{X}$ can be written in terms of an orthogonal/unitary transformation $\bm{U}$ +\[ +\bm{X} = \bm{U}\bm{\Sigma}\bm{V}^T, +\] +with $\bm{U}\bm{U}^T=\bm{I}$ or $\bm{U}\bm{U}^{\dagger}=\bm{I}$. + +Not all square matrices are diagonalizable. A matrix like the one discussed above +\[ +\bm{X} = \begin{bmatrix} +1& -1 \\ +1& -1\\ +\end{bmatrix} +\] +is not diagonalizable, it is a so-called \href{{https://en.wikipedia.org/wiki/Defective_matrix}}{defective matrix}. It is easy to see that the condition +$\bm{X}\bm{X}^T=\bm{X}^T\bm{X}$ is not fulfilled. + + +% !split +\subsection*{The SVD, a Fantastic Algorithm} + + +However, and this is the strength of the SVD algorithm, any general +matrix $\bm{X}$ can be decomposed in terms of a diagonal matrix and +two orthogonal/unitary matrices. The \href{{https://en.wikipedia.org/wiki/Singular_value_decomposition}}{Singular Value Decompostion +(SVD) theorem} +states that a general $m\times n$ matrix $\bm{X}$ can be written in +terms of a diagonal matrix $\bm{\Sigma}$ of dimensionality $n\times n$ +and two orthognal matrices $\bm{U}$ and $\bm{V}$, where the first has +dimensionality $m \times m$ and the last dimensionality $n\times n$. +We have then + +\[ +\bm{X} = \bm{U}\bm{\Sigma}\bm{V}^T +\] + +As an example, the above defective matrix can be decomposed as + +\[ +\bm{X} = \frac{1}{\sqrt{2}}\begin{bmatrix} 1& 1 \\ 1& -1\\ \end{bmatrix} \begin{bmatrix} 2& 0 \\ 0& 0\\ \end{bmatrix} \frac{1}{\sqrt{2}}\begin{bmatrix} 1& -1 \\ 1& 1\\ \end{bmatrix}=\bm{U}\bm{\Sigma}\bm{V}^T, +\] + +with eigenvalues $\sigma_1=2$ and $\sigma_2=0$. +The SVD exits always! + + +% !split +\subsection*{Another Example} + +Consider the following matrix which can be SVD decomposed as + +\[ +\bm{X} = \frac{1}{15}\begin{bmatrix} 14 & 2\\ 4 & 22\\ 16 & 13\end{matrix}=\frac{1}{3}\begin{bmatrix} 1& 2 & 2 \\ 2& -1 & 1\\ 2 & 1& -2\end{bmatrix} \begin{bmatrix} 2& 0 \\ 0& 1\\ 0 & 0\end{bmatrix}\frac{1}{5}\begin{bmatrix} 3& 4 \\ 4& -3\end{bmatrix}=\bm{U}\bm{\Sigma}\bm{V}^T. +\] + +This is a $3\times 2$ matrix which is decomposed in terms of a +$3\times 3$ matrix $\bm{U}$, and a $2\times 2$ matrix $\bm{V}$. It is easy to see +that $\bm{U}$ and $\bm{V}$ are orthogonal (how?). + +And the SVD +decomposition (singular values) gives eigenvalues +$\sigma_i\geq\sigma_{i+1}$ for all $i$ and for dimensions larger than $i=2$, the +eigenvalues (singular values) are zero. + +In the general case, where our design matrix $\bm{X}$ has dimension +$n\times p$, the matrix is thus decomposed into an $n\times n$ +orthogonal matrix $\bm{U}$, a $p\times p$ orthogonal matrix $\bm{V}$ +and a diagonal matrix $\bm{\Sigma}$ with $r=\mathrm{min}(n,p)$ +singular values $\sigma_i\lg 0$ on the main diagonal and zeros filling +the rest of the matrix. There are at most $p$ singular values +assuming that $n > p$. In our regression examples for the nuclear +masses and the equation of state this is indeed the case, while for +the Ising model we have $p > n$. These are often cases that lead to +near singular or singular matrices. + +The columns of $\bm{U}$ are called the left singular vectors while the columns of $\bm{V}$ are the right singular vectors. + +% !split +\subsection*{Economy-size SVD} + +If we assume that $n > p$, then our matrix $\bm{U}$ has dimension $n +\times n$. The last $n-p$ columns of $\bm{U}$ become however +irrelevant in our calculations since they are multiplied with the +zeros in $\bm{\Sigma}$. + +The economy-size decomposition removes extra rows or columns of zeros +from the diagonal matrix of singular values, $\bm{\Sigma}$, along with the columns +in either $\bm{U}$ or $\bm{V}$ that multiply those zeros in the expression. +Removing these zeros and columns can improve execution time +and reduce storage requirements without compromising the accuracy of +the decomposition. + +If $n > p$, we keep only the first $p$ columns of $\bm{U}$ and $\bm{\Sigma}$ has dimension $p\times p$. +If $p > n$, then only the first $n$ columns of $\bm{V}$ are computed and $\bm{\Sigma}$ has dimension $n\times n$. +The $n=p$ case is obvious, we retain the full SVD. +In general the economy-size SVD leads to less FLOPS and still conserving the desired accuracy. + +% !split +\subsection*{Mathematical Properties} + +There are several interesting mathematical properties which will be +relevant when we are going to discuss the differences between say +ordinary least squares (OLS) and \textbf{Ridge} regression. + +We have from OLS that the parameters of the linear approximation are given by +\[ +\bm{\tilde{y}} = \bm{X}\bm{\beta} = \bm{X}\left(\bm{X}^T\bm{X}\right)^{-1}\bm{X}^T\bm{y}. +\] + +The matrix to invert can be rewritten in terms of our SVD decomposition as + +\[ +\bm{X}^T\bm{X} = \bm{V}\bm{\Sigma}^T\bm{U}^T\bm{U}\bm{\Sigma}\bm{V}^T. +\] +Using the orthogonality properties of $\bm{U}$ we have + +\[ +\bm{X}^T\bm{X} = \bm{V}\bm{\Sigma}^T\bm{\Sigma}\bm{V}^T = \bm{V}\bm{D}\bm{V}^T, +\] +with $\bm{D}$ being a diagonal matrix with values along the diagonal given by the singular values squared. + +This means that +\[ +(\bm{X}^T\bm{X})\bm{V} = \bm{V}\bm{D}, +\] +that is the eigenvectors of $(\bm{X}^T\bm{X})$ are given by the columns of the right singular matrix of $\bm{X}$ and the eigenvalues are the squared singular values. It is easy to show (show this) that +\[ +(\bm{X}\bm{X}^T)\bm{U} = \bm{U}\bm{D}, +\] +that is, the eigenvectors of $(\bm{X}\bm{X})^T$ are the columns of the left singular matrix and the eigenvalues are the same. + +Going back to our OLS equation we have +\[ +\bm{X}\bm{\beta} = \bm{X}\left(\bm{V}\bm{D}\bm{V}^T \right)^{-1}\bm{X}^T\bm{y}=\bm{U\Sigma V^T}\left(\bm{V}\bm{D}\bm{V}^T \right)^{-1}(\bm{U\Sigma V^T})^T\bm{y}=\bm{U}\bm{U}^T\bm{y}. +\] +We will come back to this expression when we discuss Ridge regression. + + +% !split +\subsection*{Ridge and LASSO Regression} + +Let us remind ourselves about the expression for the standard Mean Squared Error (MSE) which we used to define our cost function and the equations for the ordinary least squares (OLS) method, that is +our optimization problem is +\[ +{\displaystyle \min_{\bm{\beta}\in {\mathbb{R}}^{p}}}\frac{1}{n}\left\{\left(\bm{y}-\bm{X}\bm{\beta}\right)^T\left(\bm{y}-\bm{X}\bm{\beta}\right)\right\}. +\] +or we can state it as +\[ +{\displaystyle \min_{\bm{\beta}\in +{\mathbb{R}}^{p}}}\frac{1}{n}\sum_{i=0}^{n-1}\left(y_i-\tilde{y}_i\right)^2=\frac{1}{n}\vert\vert \bm{y}-\bm{X}\bm{\beta}\vert\vert_2^2, +\] +where we have used the definition of a norm-2 vector, that is +\[ +\vert\vert \bm{x}\vert\vert_2 = \sqrt{\sum_i x_i^2}. +\] + +By minimizing the above equation with respect to the parameters +$\bm{\beta}$ we could then obtain an analytical expression for the +parameters $\bm{\beta}$. We can add a regularization parameter $\lambda$ by +defining a new cost function to be optimized, that is + +\[ +{\displaystyle \min_{\bm{\beta}\in +{\mathbb{R}}^{p}}}\frac{1}{n}\vert\vert \bm{y}-\bm{X}\bm{\beta}\vert\vert_2^2+\lambda\vert\vert \bm{\beta}\vert\vert_2^2 +\] + +which leads to the Ridge regression minimization problem where we +require that $\vert\vert \bm{\beta}\vert\vert_2^2\le t$, where $t$ is +a finite number larger than zero. By defining + +\[ +C(\bm{X},\bm{\beta})=\frac{1}{n}\vert\vert \bm{y}-\bm{X}\bm{\beta}\vert\vert_2^2+\lambda\vert\vert \bm{\beta}\vert\vert_1, +\] + +we have a new optimization equation +\[ +{\displaystyle \min_{\bm{\beta}\in +{\mathbb{R}}^{p}}}\frac{1}{n}\vert\vert \bm{y}-\bm{X}\bm{\beta}\vert\vert_2^2+\lambda\vert\vert \bm{\beta}\vert\vert_1 +\] +which leads to Lasso regression. Lasso stands for least absolute shrinkage and selection operator. + +Here we have defined the norm-1 as +\[ +\vert\vert \bm{x}\vert\vert_1 = \sum_i \vert x_i\vert. +\] + + +% !split +\subsection*{More on Ridge Regression} + +Using the matrix-vector expression for Ridge regression, + +\[ +C(\bm{X},\bm{\beta})=\frac{1}{n}\left\{(\bm{y}-\bm{X}\bm{\beta})^T(\bm{y}-\bm{X}\bm{\beta})\right\}+\lambda\bm{\beta}^T\bm{\beta}, +\] + +by taking the derivatives with respect to $\bm{\beta}$ we obtain then +a slightly modified matrix inversion problem which for finite values +of $\lambda$ does not suffer from singularity problems. We obtain + +\[ +\bm{\beta}^{\mathrm{Ridge}} = \left(\bm{X}^T\bm{X}+\lambda\bm{I}\right)^{-1}\bm{X}^T\bm{y}, +\] + +with $\bm{I}$ being a $p\times p$ identity matrix with the constraint that + +\[ +\sum_{i=0}^{p-1} \beta_i^2 \leq t, +\] + +with $t$ a finite positive number. + +We see that Ridge regression is nothing but the standard +OLS with a modified diagonal term added to $\bm{X}^T\bm{X}$. The +consequences, in particular for our discussion of the bias-variance +are rather interesting. + +Furthermore, if we use the result above in terms of the SVD decomposition (our analysis was done for the OLS method), we had +\[ +(\bm{X}\bm{X}^T)\bm{U} = \bm{U}\bm{D}. +\] + +We can analyse the OLS solutions in terms of the eigenvectors (the columns) of the right singular value matrix $\bm{U}$ as +\[ +\bm{X}\bm{\beta} = \bm{X}\left(\bm{V}\bm{D}\bm{V}^T \right)^{-1}\bm{X}^T\bm{y}=\bm{U\Sigma V^T}\left(\bm{V}\bm{D}\bm{V}^T \right)^{-1}(\bm{U\Sigma V^T})^T\bm{y}=\bm{U}\bm{U}^T\bm{y} +\] + + +For Ridge regression this becomes + +\[ +\bm{X}\bm{\beta}^{\mathrm{Ridge}} = \bm{U\Sigma V^T}\left(\bm{V}\bm{D}\bm{V}^T+\lambda\bm{I} \right)^{-1}(\bm{U\Sigma V^T})^T\bm{y}=\sum_{j=0}^{p-1}\bm{u}_j\bm{u}_j^T\frac{\sigma_j^2}{\sigma_j^2+\lambda}\bm{y}, +\] + +with the vectors $\bm{u}_j$ being the columns of $\bm{U}$. + +% !split +\subsection*{Interpreting the Ridge results} + +Since $\lambda \geq 0$, it means that compared to OLS, we have + +\[ +\frac{\sigma_j^2}{\sigma_j^2+\lambda} \leq 1. +\] + +Ridge regression finds the coordinates of $\bm{y}$ with respect to the +orthonormal basis $\bm{U}$, it then shrinks the coordinates by +$\frac{\sigma_j^2}{\sigma_j^2+\lambda}$. Recall that the SVD has +eigenvalues ordered in a descending way, that is $\sigma_i \geq +\sigma_{i+1}$. + +For small eigenvalues $\sigma_i$ it means that their contributions become less important, a fact which can be used to reduce the number of degrees of freedom. +Actually, calculating the variance of $\bm{X}\bm{v}_j$ shows that this quantity is equal to $\sigma_j^2/n$. +With a parameter $\lambda$ we can thus shrink the role of specific parameters. + + +% !split +\subsection*{More interpretations} + +For the sake of simplicity, let us assume that the design matrix is orthonormal, that is + +\[ +\bm{X}^T\bm{X}=(\bm{X}^T\bm{X})^{-1} =\bm{I}. +\] + +In this case the standard OLS results in +\[ +\bm{\beta}^{\mathrm{OLS}} = \bm{X}^T\bm{y}=\sum_{i=0}^{p-1}\bm{u}_j\bm{u}_j^T\bm{y}, +\] + +and + +\[ +\bm{\beta}^{\mathrm{Ridge}} = \left(\bm{I}+\lambda\bm{I}\right)^{-1}\bm{X}^T\bm{y}=\left(1+\lambda\right)^{-1}\bm{\beta}^{\mathrm{OLS}}, +\] + +that is the Ridge estimator scales the OLS estimator by the inverse of a factor $1+\lambda$, and +the Ridge estimator converges to zero when the hyperparameter goes to +infinity. + +We will come back to more interpreations after we have gone through some of the statistical analysis part. + +For more discussions of Ridge and Lasso regression, \href{{https://arxiv.org/abs/1509.09169}}{Wessel van Wieringen's} article is highly recommended. +Similarly, \href{{https://arxiv.org/abs/1803.08823}}{Mehta et al's article} is also recommended. + +% !split +\subsection*{Where are we going?} + +Before we proceed, we need to rethink what we have been doing. In our +eager to fit the data, we have omitted several important elements in +our regression analysis. In what follows we will +\begin{enumerate} +\item look at statistical properties, including a discussion of mean values, variance and the so-called bias-variance tradeoff + +\item introduce resampling techniques like cross-validation, bootstrapping and jackknife and more +\end{enumerate} + +\noindent +This will allow us to link the standard linear algebra methods we have discussed above to a statistical interpretation of the methods. + + + + + +% !split +\subsection*{Resampling methods} + +% --- begin paragraph admon --- +\paragraph{} +Resampling methods are an indispensable tool in modern +statistics. They involve repeatedly drawing samples from a training +set and refitting a model of interest on each sample in order to +obtain additional information about the fitted model. For example, in +order to estimate the variability of a linear regression fit, we can +repeatedly draw different samples from the training data, fit a linear +regression to each new sample, and then examine the extent to which +the resulting fits differ. Such an approach may allow us to obtain +information that would not be available from fitting the model only +once using the original training sample. +% --- end paragraph admon --- + + + +% !split +\subsection*{Resampling approaches can be computationally expensive} + +% --- begin paragraph admon --- +\paragraph{} + +Resampling approaches can be computationally expensive, because they +involve fitting the same statistical method multiple times using +different subsets of the training data. However, due to recent +advances in computing power, the computational requirements of +resampling methods generally are not prohibitive. In this chapter, we +discuss two of the most commonly used resampling methods, +cross-validation and the bootstrap. Both methods are important tools +in the practical application of many statistical learning +procedures. For example, cross-validation can be used to estimate the +test error associated with a given statistical learning method in +order to evaluate its performance, or to select the appropriate level +of flexibility. The process of evaluating a model’s performance is +known as model assessment, whereas the process of selecting the proper +level of flexibility for a model is known as model selection. The +bootstrap is widely used. +% --- end paragraph admon --- + + + +% !split +\subsection*{Why resampling methods ?} + +% --- begin paragraph admon --- +\paragraph{Statistical analysis.} + +\begin{itemize} +\item Our simulations can be treated as \emph{computer experiments}. This is particularly the case for Monte Carlo methods + +\item The results can be analysed with the same statistical tools as we would use analysing experimental data. + +\item As in all experiments, we are looking for expectation values and an estimate of how accurate they are, i.e., possible sources for errors. +\end{itemize} + +\noindent +% --- end paragraph admon --- + + + +% !split +\subsection*{Statistical analysis} + +% --- begin paragraph admon --- +\paragraph{} + +\begin{itemize} +\item As in other experiments, many numerical experiments have two classes of errors: +\begin{itemize} + + \item Statistical errors + + \item Systematical errors + +\end{itemize} + +\noindent +\item Statistical errors can be estimated using standard tools from statistics + +\item Systematical errors are method specific and must be treated differently from case to case. +\end{itemize} + +\noindent +% --- end paragraph admon --- + + + +% !split +\subsection*{Statistics} + +% --- begin paragraph admon --- +\paragraph{} +The \emph{probability distribution function (PDF)} is a function +$p(x)$ on the domain which, in the discrete case, gives us the +probability or relative frequency with which these values of $X$ occur: +\[ +p(x) = \mathrm{prob}(X=x) +\] +In the continuous case, the PDF does not directly depict the +actual probability. Instead we define the probability for the +stochastic variable to assume any value on an infinitesimal interval +around $x$ to be $p(x)dx$. The continuous function $p(x)$ then gives us +the \emph{density} of the probability rather than the probability +itself. The probability for a stochastic variable to assume any value +on a non-infinitesimal interval $[a,\,b]$ is then just the integral: +\[ +\mathrm{prob}(a\leq X\leq b) = \int_a^b p(x)dx +\] +Qualitatively speaking, a stochastic variable represents the values of +numbers chosen as if by chance from some specified PDF so that the +selection of a large set of these numbers reproduces this PDF. +% --- end paragraph admon --- + + + + +% !split +\subsection*{Statistics, moments} + +% --- begin paragraph admon --- +\paragraph{} +A particularly useful class of special expectation values are the +\emph{moments}. The $n$-th moment of the PDF $p$ is defined as +follows: +\[ +\langle x^n\rangle \equiv \int\! x^n p(x)\,dx +\] +The zero-th moment $\langle 1\rangle$ is just the normalization condition of +$p$. The first moment, $\langle x\rangle$, is called the \emph{mean} of $p$ +and often denoted by the letter $\mu$: +\[ +\langle x\rangle = \mu \equiv \int\! x p(x)\,dx +\] +% --- end paragraph admon --- + + + +% !split +\subsection*{Statistics, central moments} + +% --- begin paragraph admon --- +\paragraph{} +A special version of the moments is the set of \emph{central moments}, +the n-th central moment defined as: +\[ +\langle (x-\langle x \rangle )^n\rangle \equiv \int\! (x-\langle x\rangle)^n p(x)\,dx +\] +The zero-th and first central moments are both trivial, equal $1$ and +$0$, respectively. But the second central moment, known as the +\emph{variance} of $p$, is of particular interest. For the stochastic +variable $X$, the variance is denoted as $\sigma^2_X$ or $\mathrm{var}(X)$: +\begin{align} +\sigma^2_X\ \ =\ \ \mathrm{var}(X) & = \langle (x-\langle x\rangle)^2\rangle = +\int\! (x-\langle x\rangle)^2 p(x)\,dx\\ +& = \int\! \left(x^2 - 2 x \langle x\rangle^{2} + + \langle x\rangle^2\right)p(x)\,dx\\ +& = \langle x^2\rangle - 2 \langle x\rangle\langle x\rangle + \langle x\rangle^2\\ +& = \langle x^2\rangle - \langle x\rangle^2 +\end{align} +The square root of the variance, $\sigma =\sqrt{\langle (x-\langle x\rangle)^2\rangle}$ is called the \emph{standard deviation} of $p$. It is clearly just the RMS (root-mean-square) +value of the deviation of the PDF from its mean value, interpreted +qualitatively as the \emph{spread} of $p$ around its mean. +% --- end paragraph admon --- + + + +% !split +\subsection*{Statistics, covariance} + +% --- begin paragraph admon --- +\paragraph{} +Another important quantity is the so called covariance, a variant of +the above defined variance. Consider again the set $\{X_i\}$ of $n$ +stochastic variables (not necessarily uncorrelated) with the +multivariate PDF $P(x_1,\dots,x_n)$. The \emph{covariance} of two +of the stochastic variables, $X_i$ and $X_j$, is defined as follows: +\begin{align} +\mathrm{cov}(X_i,\,X_j) &\equiv \langle (x_i-\langle x_i\rangle)(x_j-\langle x_j\rangle)\rangle +\nonumber\\ +&= +\int\!\cdots\!\int\!(x_i-\langle x_i \rangle)(x_j-\langle x_j \rangle)\, +P(x_1,\dots,x_n)\,dx_1\dots dx_n +\label{eq:def_covariance} +\end{align} +with +\[ +\langle x_i\rangle = +\int\!\cdots\!\int\!x_i\,P(x_1,\dots,x_n)\,dx_1\dots dx_n +\] +% --- end paragraph admon --- + + + +% !split +\subsection*{Statistics, more covariance} + +% --- begin paragraph admon --- +\paragraph{} +If we consider the above covariance as a matrix $C_{ij}=\mathrm{cov}(X_i,\,X_j)$, then the diagonal elements are just the familiar +variances, $C_{ii} = \mathrm{cov}(X_i,\,X_i) = \mathrm{var}(X_i)$. It turns out that +all the off-diagonal elements are zero if the stochastic variables are +uncorrelated. This is easy to show, keeping in mind the linearity of +the expectation value. Consider the stochastic variables $X_i$ and +$X_j$, ($i\neq j$): +\begin{align} +\mathrm{cov}(X_i,\,X_j) &= \langle(x_i-\langle x_i\rangle)(x_j-\langle x_j\rangle)\rangle\\ +&=\langle x_i x_j - x_i\langle x_j\rangle - \langle x_i\rangle x_j + \langle x_i\rangle\langle x_j\rangle\rangle \\ +&=\langle x_i x_j\rangle - \langle x_i\langle x_j\rangle\rangle - \langle \langle x_i\rangle x_j\rangle + +\langle \langle x_i\rangle\langle x_j\rangle\rangle\\ +&=\langle x_i x_j\rangle - \langle x_i\rangle\langle x_j\rangle - \langle x_i\rangle\langle x_j\rangle + +\langle x_i\rangle\langle x_j\rangle\\ +&=\langle x_i x_j\rangle - \langle x_i\rangle\langle x_j\rangle +\end{align} +% --- end paragraph admon --- + + + + + +% !split +\subsection*{Statistics, independent variables} + +% --- begin paragraph admon --- +\paragraph{} +If $X_i$ and $X_j$ are independent, we get +$\langle x_i x_j\rangle =\langle x_i\rangle\langle x_j\rangle$, resulting in $\mathrm{cov}(X_i, X_j) = 0\ \ (i\neq j)$. + +Also useful for us is the covariance of linear combinations of +stochastic variables. Let $\{X_i\}$ and $\{Y_i\}$ be two sets of +stochastic variables. Let also $\{a_i\}$ and $\{b_i\}$ be two sets of +scalars. Consider the linear combination: +\[ +U = \sum_i a_i X_i \qquad V = \sum_j b_j Y_j +\] +By the linearity of the expectation value +\[ +\mathrm{cov}(U, V) = \sum_{i,j}a_i b_j \mathrm{cov}(X_i, Y_j) +\] +% --- end paragraph admon --- + + + +% !split +\subsection*{Statistics, more variance} + +% --- begin paragraph admon --- +\paragraph{} +Now, since the variance is just $\mathrm{var}(X_i) = \mathrm{cov}(X_i, X_i)$, we get +the variance of the linear combination $U = \sum_i a_i X_i$: +\begin{equation} +\mathrm{var}(U) = \sum_{i,j}a_i a_j \mathrm{cov}(X_i, X_j) +\label{eq:variance_linear_combination} +\end{equation} +And in the special case when the stochastic variables are +uncorrelated, the off-diagonal elements of the covariance are as we +know zero, resulting in: +\[ +\mathrm{var}(U) = \sum_i a_i^2 \mathrm{cov}(X_i, X_i) = \sum_i a_i^2 \mathrm{var}(X_i) +\] +\[ +\mathrm{var}(\sum_i a_i X_i) = \sum_i a_i^2 \mathrm{var}(X_i) +\] +which will become very useful in our study of the error in the mean +value of a set of measurements. +% --- end paragraph admon --- + + + +% !split +\subsection*{Statistics and stochastic processes} + +% --- begin paragraph admon --- +\paragraph{} +A \emph{stochastic process} is a process that produces sequentially a +chain of values: +\[ +\{x_1, x_2,\dots\,x_k,\dots\}. +\] +We will call these +values our \emph{measurements} and the entire set as our measured +\emph{sample}. The action of measuring all the elements of a sample +we will call a stochastic \emph{experiment} since, operationally, +they are often associated with results of empirical observation of +some physical or mathematical phenomena; precisely an experiment. We +assume that these values are distributed according to some +PDF $p_X^{\phantom X}(x)$, where $X$ is just the formal symbol for the +stochastic variable whose PDF is $p_X^{\phantom X}(x)$. Instead of +trying to determine the full distribution $p$ we are often only +interested in finding the few lowest moments, like the mean +$\mu_X^{\phantom X}$ and the variance $\sigma_X^{\phantom X}$. +% --- end paragraph admon --- + + + + +% !split +\subsection*{Statistics and sample variables} + +% --- begin paragraph admon --- +\paragraph{} +In practical situations a sample is always of finite size. Let that +size be $n$. The expectation value of a sample, the \emph{sample mean}, is then defined as follows: +\[ +\bar{x}_n \equiv \frac{1}{n}\sum_{k=1}^n x_k +\] +The \emph{sample variance} is: +\[ +\mathrm{var}(x) \equiv \frac{1}{n}\sum_{k=1}^n (x_k - \bar{x}_n)^2 +\] +its square root being the \emph{standard deviation of the sample}. The +\emph{sample covariance} is: +\[ +\mathrm{cov}(x)\equiv\frac{1}{n}\sum_{kl}(x_k - \bar{x}_n)(x_l - \bar{x}_n) +\] +% --- end paragraph admon --- + + + +% !split +\subsection*{Statistics, sample variance and covariance} + +% --- begin paragraph admon --- +\paragraph{} +Note that the sample variance is the sample covariance without the +cross terms. In a similar manner as the covariance in Eq.~(\ref{eq:def_covariance}) is a measure of the correlation between +two stochastic variables, the above defined sample covariance is a +measure of the sequential correlation between succeeding measurements +of a sample. + +These quantities, being known experimental values, differ +significantly from and must not be confused with the similarly named +quantities for stochastic variables, mean $\mu_X$, variance $\mathrm{var}(X)$ +and covariance $\mathrm{cov}(X,Y)$. +% --- end paragraph admon --- + + + +% !split +\subsection*{Statistics, law of large numbers} + +% --- begin paragraph admon --- +\paragraph{} +The law of large numbers +states that as the size of our sample grows to infinity, the sample +mean approaches the true mean $\mu_X^{\phantom X}$ of the chosen PDF: +\[ +\lim_{n\to\infty}\bar{x}_n = \mu_X^{\phantom X} +\] +The sample mean $\bar{x}_n$ works therefore as an estimate of the true +mean $\mu_X^{\phantom X}$. + +What we need to find out is how good an approximation $\bar{x}_n$ is to +$\mu_X^{\phantom X}$. In any stochastic measurement, an estimated +mean is of no use to us without a measure of its error. A quantity +that tells us how well we can reproduce it in another experiment. We +are therefore interested in the PDF of the sample mean itself. Its +standard deviation will be a measure of the spread of sample means, +and we will simply call it the \emph{error} of the sample mean, or +just sample error, and denote it by $\mathrm{err}_X^{\phantom X}$. In +practice, we will only be able to produce an \emph{estimate} of the +sample error since the exact value would require the knowledge of the +true PDFs behind, which we usually do not have. +% --- end paragraph admon --- + + + + +% !split +\subsection*{Statistics, more on sample error} + +% --- begin paragraph admon --- +\paragraph{} +Let us first take a look at what happens to the sample error as the +size of the sample grows. In a sample, each of the measurements $x_i$ +can be associated with its own stochastic variable $X_i$. The +stochastic variable $\overline X_n$ for the sample mean $\bar{x}_n$ is +then just a linear combination, already familiar to us: +\[ +\overline X_n = \frac{1}{n}\sum_{i=1}^n X_i +\] +All the coefficients are just equal $1/n$. The PDF of $\overline X_n$, +denoted by $p_{\overline X_n}(x)$ is the desired PDF of the sample +means. +% --- end paragraph admon --- + + + +% !split +\subsection*{Statistics} + +% --- begin paragraph admon --- +\paragraph{} +The probability density of obtaining a sample mean $\bar x_n$ +is the product of probabilities of obtaining arbitrary values $x_1, +x_2,\dots,x_n$ with the constraint that the mean of the set $\{x_i\}$ +is $\bar x_n$: +\[ +p_{\overline X_n}(x) = \int p_X^{\phantom X}(x_1)\cdots +\int p_X^{\phantom X}(x_n)\ +\delta\!\left(x - \frac{x_1+x_2+\dots+x_n}{n}\right)dx_n \cdots dx_1 +\] +And in particular we are interested in its variance $\mathrm{var}(\overline X_n)$. +% --- end paragraph admon --- + + + + + +% !split +\subsection*{Statistics, central limit theorem} + +% --- begin paragraph admon --- +\paragraph{} +It is generally not possible to express $p_{\overline X_n}(x)$ in a +closed form given an arbitrary PDF $p_X^{\phantom X}$ and a number +$n$. But for the limit $n\to\infty$ it is possible to make an +approximation. The very important result is called \emph{the central limit theorem}. It tells us that as $n$ goes to infinity, +$p_{\overline X_n}(x)$ approaches a Gaussian distribution whose mean +and variance equal the true mean and variance, $\mu_{X}^{\phantom X}$ +and $\sigma_{X}^{2}$, respectively: +\begin{equation} +\lim_{n\to\infty} p_{\overline X_n}(x) = +\left(\frac{n}{2\pi\mathrm{var}(X)}\right)^{1/2} +e^{-\frac{n(x-\bar x_n)^2}{2\mathrm{var}(X)}} +\label{eq:central_limit_gaussian} +\end{equation} +% --- end paragraph admon --- + + + +% !split +\subsection*{Statistics, more technicalities} + +% --- begin paragraph admon --- +\paragraph{} +The desired variance +$\mathrm{var}(\overline X_n)$, i.e.~the sample error squared +$\mathrm{err}_X^2$, is given by: +\begin{equation} +\mathrm{err}_X^2 = \mathrm{var}(\overline X_n) = \frac{1}{n^2} +\sum_{ij} \mathrm{cov}(X_i, X_j) +\label{eq:error_exact} +\end{equation} +We see now that in order to calculate the exact error of the sample +with the above expression, we would need the true means +$\mu_{X_i}^{\phantom X}$ of the stochastic variables $X_i$. To +calculate these requires that we know the true multivariate PDF of all +the $X_i$. But this PDF is unknown to us, we have only got the measurements of +one sample. The best we can do is to let the sample itself be an +estimate of the PDF of each of the $X_i$, estimating all properties of +$X_i$ through the measurements of the sample. +% --- end paragraph admon --- + + + + +% !split +\subsection*{Statistics} + +% --- begin paragraph admon --- +\paragraph{} +Our estimate of $\mu_{X_i}^{\phantom X}$ is then the sample mean $\bar x$ +itself, in accordance with the the central limit theorem: +\[ +\mu_{X_i}^{\phantom X} = \langle x_i\rangle \approx \frac{1}{n}\sum_{k=1}^n x_k = \bar x +\] +Using $\bar x$ in place of $\mu_{X_i}^{\phantom X}$ we can give an +\emph{estimate} of the covariance in Eq.~(\ref{eq:error_exact}) +\[ +\mathrm{cov}(X_i, X_j) = \langle (x_i-\langle x_i\rangle)(x_j-\langle x_j\rangle)\rangle +\approx\langle (x_i - \bar x)(x_j - \bar{x})\rangle, +\] +resulting in +\[ +\frac{1}{n} \sum_{l}^n \left(\frac{1}{n}\sum_{k}^n (x_k -\bar x_n)(x_l - \bar x_n)\right)=\frac{1}{n}\frac{1}{n} \sum_{kl} (x_k -\bar x_n)(x_l - \bar x_n)=\frac{1}{n}\mathrm{cov}(x) +\] +% --- end paragraph admon --- + + + +% !split +\subsection*{Statistics and sample variance} + +% --- begin paragraph admon --- +\paragraph{} +By the same procedure we can use the sample variance as an +estimate of the variance of any of the stochastic variables $X_i$ +\[ +\mathrm{var}(X_i)=\langle x_i - \langle x_i\rangle\rangle \approx \langle x_i - \bar x_n\rangle\nonumber, +\] +which is approximated as +\begin{equation} +\mathrm{var}(X_i)\approx \frac{1}{n}\sum_{k=1}^n (x_k - \bar x_n)=\mathrm{var}(x) +\label{eq:var_estimate_i_think} +\end{equation} + +Now we can calculate an estimate of the error +$\mathrm{err}_X^{\phantom X}$ of the sample mean $\bar x_n$: +\begin{align} +\mathrm{err}_X^2 +&=\frac{1}{n^2}\sum_{ij} \mathrm{cov}(X_i, X_j) \nonumber \\ +&\approx&\frac{1}{n^2}\sum_{ij}\frac{1}{n}\mathrm{cov}(x) =\frac{1}{n^2}n^2\frac{1}{n}\mathrm{cov}(x)\nonumber\\ +&=\frac{1}{n}\mathrm{cov}(x) +\label{eq:error_estimate} +\end{align} +which is nothing but the sample covariance divided by the number of +measurements in the sample. +% --- end paragraph admon --- + + + +% !split +\subsection*{Statistics, uncorrelated results} + +% --- begin paragraph admon --- +\paragraph{} + +In the special case that the measurements of the sample are +uncorrelated (equivalently the stochastic variables $X_i$ are +uncorrelated) we have that the off-diagonal elements of the covariance +are zero. This gives the following estimate of the sample error: +\[ +\mathrm{err}_X^2=\frac{1}{n^2}\sum_{ij} \mathrm{cov}(X_i, X_j) = +\frac{1}{n^2} \sum_i \mathrm{var}(X_i), +\] +resulting in +\begin{equation} +\mathrm{err}_X^2\approx \frac{1}{n^2} \sum_i \mathrm{var}(x)= \frac{1}{n}\mathrm{var}(x) +\label{eq:error_estimate_uncorrel} +\end{equation} +where in the second step we have used Eq.~(\ref{eq:var_estimate_i_think}). +The error of the sample is then just its standard deviation divided by +the square root of the number of measurements the sample contains. +This is a very useful formula which is easy to compute. It acts as a +first approximation to the error, but in numerical experiments, we +cannot overlook the always present correlations. +% --- end paragraph admon --- + + + +% !split +\subsection*{Statistics, computations} + +% --- begin paragraph admon --- +\paragraph{} +For computational purposes one usually splits up the estimate of +$\mathrm{err}_X^2$, given by Eq.~(\ref{eq:error_estimate}), into two +parts +\[ +\mathrm{err}_X^2 = \frac{1}{n}\mathrm{var}(x) + \frac{1}{n}(\mathrm{cov}(x)-\mathrm{var}(x)), +\] +which equals +\begin{equation} +\frac{1}{n^2}\sum_{k=1}^n (x_k - \bar x_n)^2 +\frac{2}{n^2}\sum_{k 0$. We say then that the ridge estimator is biased. + +We can also compute the variance as + +\[ +\mbox{Var}[\bm{\beta}^{\mathrm{Ridge}}]=\sigma^2[ \mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I} ]^{-1} \mathbf{X}^{T} \mathbf{X} \{ [ \mathbf{X}^{\top} \mathbf{X} + \lambda \mathbf{I} ]^{-1}\}^{T}, +\] +and it is easy to see that if the parameter $\lambda$ goes to infinity then the variance of Ridge parameters $\bm{\beta}$ goes to zero. + +With this, we can compute the difference + +\[ +\mbox{Var}[\bm{\beta}^{\mathrm{OLS}}]-\mbox{Var}(\bm{\beta}^{\mathrm{Ridge}})=\sigma^2 [ \mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I} ]^{-1}[ 2\lambda\mathbf{I} + \lambda^2 (\mathbf{X}^{T} \mathbf{X})^{-1} ] \{ [ \mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I} ]^{-1}\}^{T}. +\] +The difference is non-negative definite since each component of the +matrix product is non-negative definite. +This means the variance we obtain with the standard OLS will always for $\lambda > 0$ be larger than the variance of $\bm{\beta}$ obtained with the Ridge estimator. This has interesting consequences when we discuss the so-called bias-variance trade-off below. + +% !split +\subsection*{Cross-validation} + +Instead of choosing the penalty parameter to balance model fit with +model complexity, cross-validation requires it (i.e.~the penalty +parameter) to yield a model with good prediction +performance. Commonly, this performance is evaluated on novel +data. Novel data need not be easy to come by and one has to make do +with the data at hand. + +The setting of \textbf{original} and novel data is +then mimicked by sample splitting: the data set is divided into two +(groups of samples). One of these two data sets, called the +\emph{training set}, plays the role of \textbf{original} data on which the model is +built. The second of these data sets, called the \emph{test set}, plays the +role of the \textbf{novel} data and is used to evaluate the prediction +performance (often operationalized as the log-likelihood or the +prediction error or its square or the R2 score) of the model built on the training data set. This +procedure (model building and prediction evaluation on training and +test set, respectively) is done for a collection of possible penalty +parameter choices. The penalty parameter that yields the model with +the best prediction performance is to be preferred. The thus obtained +performance evaluation depends on the actual split of the data set. To +remove this dependence the data set is split many times into a +training and test set. For each split the model parameters are +estimated for all choices of $\lambda$ using the training data and +estimated parameters are evaluated on the corresponding test set. The +penalty parameter that on average over the test sets performs best (in +some sense) is then selected. + + +% !split +\subsection*{Computationally expensive} + +The validation set approach is conceptually simple and is easy to implement. But it has two potential drawbacks: + +\begin{itemize} +\item The validation estimate of the test error rate can be highly variable, depending on precisely which observations are included in the training set and which observations are included in the validation set. + +\item In the validation approach, only a subset of the observations, those that are included in the training set rather than in the validation set are used to fit the model. Since statistical methods tend to perform worse when trained on fewer observations, this suggests that the validation set error rate may tend to overestimate the test error rate for the model fit on the entire data set. +\end{itemize} + +\noindent +% !split +\subsection*{Various steps in cross-validation} + +When the repetitive splitting of the data set is done randomly, +samples may accidently end up in a fast majority of the splits in +either training or test set. Such samples may have an unbalanced +influence on either model building or prediction evaluation. To avoid +this $k$-fold cross-validation structures the data splitting. The +samples are divided into $k$ more or less equally sized exhaustive and +mutually exclusive subsets. In turn (at each split) one of these +subsets plays the role of the test set while the union of the +remaining subsets constitutes the training set. Such a splitting +warrants a balanced representation of each sample in both training and +test set over the splits. Still the division into the $k$ subsets +involves a degree of randomness. This may be fully excluded when +choosing $k=n$. This particular case is referred to as leave-one-out +cross-validation (LOOCV). + +% !split +\subsection*{How to set up the cross-validation for Ridge and/or Lasso} + +\begin{itemize} +\item Define a range of interest for the penalty parameter. + +\item Divide the data set into training and test set comprising samples $\{1, \ldots, n\} \setminus i$ and $\{ i \}$, respectively. + +\item Fit the linear regression model by means of ridge estimation for each $\lambda$ in the grid using the training set, and the corresponding estimate of the error variance $\bm{\sigma}_{-i}^2(\lambda)$, as +\end{itemize} + +\noindent +\begin{align*} +\bm{\beta}_{-i}(\lambda) & = ( \bm{X}_{-i, \ast}^{T} +\bm{X}_{-i, \ast} + \lambda \bm{I}_{pp})^{-1} +\bm{X}_{-i, \ast}^{T} \bm{y}_{-i} +\end{align*} + +\begin{itemize} +\item Evaluate the prediction performance of these models on the test set by $\log\{L[y_i, \bm{X}_{i, \ast}; \bm{\beta}_{-i}(\lambda), \bm{\sigma}_{-i}^2(\lambda)]\}$. Or, by the prediction error $|y_i - \bm{X}_{i, \ast} \bm{\beta}_{-i}(\lambda)|$, the relative error, the error squared or the R2 score function. + +\item Repeat the first three steps such that each sample plays the role of the test set once. + +\item Average the prediction performances of the test sets at each grid point of the penalty bias/parameter by computing the \emph{cross-validated log-likelihood}. It is an estimate of the prediction performance of the model corresponding to this value of the penalty parameter on novel data. It is defined as +\end{itemize} + +\noindent +\begin{align*} +\frac{1}{n} \sum_{i = 1}^n \log\{L[y_i, \mathbf{X}_{i, \ast}; \bm{\beta}_{-i}(\lambda), \bm{\sigma}_{-i}^2(\lambda)]\}. +\end{align*} + +\begin{itemize} +\item The value of the penalty parameter that maximizes the cross-validated log-likelihood is the value of choice. Or we can use the MSE or the R2 score functions. +\end{itemize} + +\noindent +% !split +\subsection*{Resampling methods: Jackknife and Bootstrap} + +Two famous +resampling methods are the \textbf{independent bootstrap} and \textbf{the jackknife}. + +The jackknife is a special case of the independent bootstrap. Still, the jackknife was made +popular prior to the independent bootstrap. And as the popularity of +the independent bootstrap soared, new variants, such as \textbf{the dependent bootstrap}. + +The Jackknife and independent bootstrap work for +independent, identically distributed random variables. +If these conditions are not +satisfied, the methods will fail. Yet, it should be said that if the data are +independent, identically distributed, and we only want to estimate the +variance of $\overline{X}$ (which often is the case), then there is no +need for bootstrapping. + +% !split +\subsection*{Resampling methods: Jackknife} + +The Jackknife works by making many replicas of the estimator $\widehat{\theta}$. +The jackknife is a resampling method where we systematically leave out one observation from the vector of observed values $\bm{x} = (x_1,x_2,\cdots,X_n)$. +Let $\bm{x}_i$ denote the vector +\[ +\bm{x}_i = (x_1,x_2,\cdots,x_{i-1},x_{i+1},\cdots,x_n), +\] + +which equals the vector $\bm{x}$ with the exception that observation +number $i$ is left out. Using this notation, define +$\widehat{\theta}_i$ to be the estimator +$\widehat{\theta}$ computed using $\vec{X}_i$. + + +% !split +\subsection*{Jackknife code example} +\begin{minted}[fontsize=\fontsize{9pt}{9pt},linenos=false,mathescape,baselinestretch=1.0,fontfamily=tt,xleftmargin=7mm]{python} +from numpy import * +from numpy.random import randint, randn +from time import time + +def jackknife(data, stat): + n = len(data);t = zeros(n); inds = arange(n); t0 = time() + ## 'jackknifing' by leaving out an observation for each i + for i in range(n): + t[i] = stat(delete(data,i) ) + + # analysis + print("Runtime: %g sec" % (time()-t0)); print("Jackknife Statistics :") + print("original bias std. error") + print("%8g %14g %15g" % (stat(data),(n-1)*mean(t)/n, (n*var(t))**.5)) + + return t + + +# Returns mean of data samples +def stat(data): + return mean(data) + + +mu, sigma = 100, 15 +datapoints = 10000 +x = mu + sigma*random.randn(datapoints) +# jackknife returns the data sample +t = jackknife(x, stat) + +\end{minted} + + +% !split +\subsection*{Resampling methods: Bootstrap} + +% --- begin paragraph admon --- +\paragraph{} +Bootstrapping is a nonparametric approach to statistical inference +that substitutes computation for more traditional distributional +assumptions and asymptotic results. Bootstrapping offers a number of +advantages: +\begin{enumerate} +\item The bootstrap is quite general, although there are some cases in which it fails. + +\item Because it does not require distributional assumptions (such as normally distributed errors), the bootstrap can provide more accurate inferences when the data are not well behaved or when the sample size is small. + +\item It is possible to apply the bootstrap to statistics with sampling distributions that are difficult to derive, even asymptotically. + +\item It is relatively simple to apply the bootstrap to complex data-collection plans (such as stratified and clustered samples). +\end{enumerate} + +\noindent +% --- end paragraph admon --- + + + + +% !split +\subsection*{Resampling methods: Bootstrap background} + +Since $\widehat{\theta} = \widehat{\theta}(\bm{X})$ is a function of random variables, +$\widehat{\theta}$ itself must be a random variable. Thus it has +a pdf, call this function $p(\bm{t})$. The aim of the bootstrap is to +estimate $p(\bm{t})$ by the relative frequency of +$\widehat{\theta}$. You can think of this as using a histogram +in the place of $p(\bm{t})$. If the relative frequency closely +resembles $p(\vec{t})$, then using numerics, it is straight forward to +estimate all the interesting parameters of $p(\bm{t})$ using point +estimators. + + +% !split +\subsection*{Resampling methods: More Bootstrap background} + +In the case that $\widehat{\theta}$ has +more than one component, and the components are independent, we use the +same estimator on each component separately. If the probability +density function of $X_i$, $p(x)$, had been known, then it would have +been straight forward to do this by: +\begin{enumerate} +\item Drawing lots of numbers from $p(x)$, suppose we call one such set of numbers $(X_1^*, X_2^*, \cdots, X_n^*)$. + +\item Then using these numbers, we could compute a replica of $\widehat{\theta}$ called $\widehat{\theta}^*$. +\end{enumerate} + +\noindent +By repeated use of (1) and (2), many +estimates of $\widehat{\theta}$ could have been obtained. The +idea is to use the relative frequency of $\widehat{\theta}^*$ +(think of a histogram) as an estimate of $p(\bm{t})$. + +% !split +\subsection*{Resampling methods: Bootstrap approach} + +But +unless there is enough information available about the process that +generated $X_1,X_2,\cdots,X_n$, $p(x)$ is in general +unknown. Therefore, \href{{https://projecteuclid.org/euclid.aos/1176344552}}{Efron in 1979} asked the +question: What if we replace $p(x)$ by the relative frequency +of the observation $X_i$; if we draw observations in accordance with +the relative frequency of the observations, will we obtain the same +result in some asymptotic sense? The answer is yes. + + +Instead of generating the histogram for the relative +frequency of the observation $X_i$, just draw the values +$(X_1^*,X_2^*,\cdots,X_n^*)$ with replacement from the vector +$\bm{X}$. + +% !split +\subsection*{Resampling methods: Bootstrap steps} + +The independent bootstrap works like this: + +\begin{enumerate} +\item Draw with replacement $n$ numbers for the observed variables $\bm{x} = (x_1,x_2,\cdots,x_n)$. + +\item Define a vector $\bm{x}^*$ containing the values which were drawn from $\bm{x}$. + +\item Using the vector $\bm{x}^*$ compute $\widehat{\theta}^*$ by evaluating $\widehat \theta$ under the observations $\bm{x}^*$. + +\item Repeat this process $k$ times. +\end{enumerate} + +\noindent +When you are done, you can draw a histogram of the relative frequency +of $\widehat \theta^*$. This is your estimate of the probability +distribution $p(t)$. Using this probability distribution you can +estimate any statistics thereof. In principle you never draw the +histogram of the relative frequency of $\widehat{\theta}^*$. Instead +you use the estimators corresponding to the statistic of interest. For +example, if you are interested in estimating the variance of $\widehat +\theta$, apply the etsimator $\widehat \sigma^2$ to the values +$\widehat \theta ^*$. + + +% !split +\subsection*{Code example for the Bootstrap method} + +The following code starts with a Gaussian distribution with mean value +$\mu =100$ and variance $\sigma=15$. We use this to generate the data +used in the bootstrap analysis. The bootstrap analysis returns a data +set after a given number of bootstrap operations (as many as we have +data points). This data set consists of estimated mean values for each +bootstrap operation. The histogram generated by the bootstrap method +shows that the distribution for these mean values is also a Gaussian, +centered around the mean value $\mu=100$ but with standard deviation +$\sigma/\sqrt{n}$, where $n$ is the number of bootstrap samples (in +this case the same as the number of original data points). The value +of the standard deviation is what we expect from the central limit +theorem. + + +\begin{minted}[fontsize=\fontsize{9pt}{9pt},linenos=false,mathescape,baselinestretch=1.0,fontfamily=tt,xleftmargin=7mm]{python} +from numpy import * +from numpy.random import randint, randn +from time import time +import matplotlib.mlab as mlab +import matplotlib.pyplot as plt + +# Returns mean of bootstrap samples +def stat(data): + return mean(data) + +# Bootstrap algorithm +def bootstrap(data, statistic, R): + t = zeros(R); n = len(data); inds = arange(n); t0 = time() + # non-parametric bootstrap + for i in range(R): + t[i] = statistic(data[randint(0,n,n)]) + + # analysis + print("Runtime: %g sec" % (time()-t0)); print("Bootstrap Statistics :") + print("original bias std. error") + print("%8g %8g %14g %15g" % (statistic(data), std(data),mean(t),std(t))) + return t + + +mu, sigma = 100, 15 +datapoints = 10000 +x = mu + sigma*random.randn(datapoints) +# bootstrap returns the data sample +t = bootstrap(x, stat, datapoints) +# the histogram of the bootstrapped data +n, binsboot, patches = plt.hist(t, 50, normed=1, facecolor='red', alpha=0.75) + +# add a 'best fit' line +y = mlab.normpdf( binsboot, mean(t), std(t)) +lt = plt.plot(binsboot, y, 'r--', linewidth=1) +plt.xlabel('Smarts') +plt.ylabel('Probability') +plt.axis([99.5, 100.6, 0, 3.0]) +plt.grid(True) + +plt.show() + +\end{minted} + + +% !split +\subsection*{Code Example for Cross-validation and $k$-fold Cross-validation} + +The code here uses Ridge regression with cross-validation (CV) resampling and $k$-fold CV in order to fit a specific polynomial. +\begin{minted}[fontsize=\fontsize{9pt}{9pt},linenos=false,mathescape,baselinestretch=1.0,fontfamily=tt,xleftmargin=7mm]{python} +import numpy as np +import matplotlib.pyplot as plt +from sklearn.model_selection import KFold +from sklearn.linear_model import Ridge +from sklearn.model_selection import cross_val_score +from sklearn.preprocessing import PolynomialFeatures + +# A seed just to ensure that the random numbers are the same for every run. +# Useful for eventual debugging. +np.random.seed(3155) + +# Generate the data. +nsamples = 100 +x = np.random.randn(nsamples) +y = 3*x**2 + np.random.randn(nsamples) + +## Cross-validation on Ridge regression using KFold only + +# Decide degree on polynomial to fit +poly = PolynomialFeatures(degree = 6) + +# Decide which values of lambda to use +nlambdas = 500 +lambdas = np.logspace(-3, 5, nlambdas) + +# Initialize a KFold instance +k = 5 +kfold = KFold(n_splits = k) + +# Perform the cross-validation to estimate MSE +scores_KFold = np.zeros((nlambdas, k)) + +i = 0 +for lmb in lambdas: + ridge = Ridge(alpha = lmb) + j = 0 + for train_inds, test_inds in kfold.split(x): + xtrain = x[train_inds] + ytrain = y[train_inds] + + xtest = x[test_inds] + ytest = y[test_inds] + + Xtrain = poly.fit_transform(xtrain[:, np.newaxis]) + ridge.fit(Xtrain, ytrain[:, np.newaxis]) + + Xtest = poly.fit_transform(xtest[:, np.newaxis]) + ypred = ridge.predict(Xtest) + + scores_KFold[i,j] = np.sum((ypred - ytest[:, np.newaxis])**2)/np.size(ypred) + + j += 1 + i += 1 + + +estimated_mse_KFold = np.mean(scores_KFold, axis = 1) + +## Cross-validation using cross_val_score from sklearn along with KFold + +# kfold is an instance initialized above as: +# kfold = KFold(n_splits = k) + +estimated_mse_sklearn = np.zeros(nlambdas) +i = 0 +for lmb in lambdas: + ridge = Ridge(alpha = lmb) + + X = poly.fit_transform(x[:, np.newaxis]) + estimated_mse_folds = cross_val_score(ridge, X, y[:, np.newaxis], scoring='neg_mean_squared_error', cv=kfold) + + # cross_val_score return an array containing the estimated negative mse for every fold. + # we have to the the mean of every array in order to get an estimate of the mse of the model + estimated_mse_sklearn[i] = np.mean(-estimated_mse_folds) + + i += 1 + +## Plot and compare the slightly different ways to perform cross-validation + +plt.figure() + +plt.plot(np.log10(lambdas), estimated_mse_sklearn, label = 'cross_val_score') +plt.plot(np.log10(lambdas), estimated_mse_KFold, 'r--', label = 'KFold') + +plt.xlabel('log10(lambda)') +plt.ylabel('mse') + +plt.legend() + +plt.show() + +\end{minted} + + +% !split +\subsection*{The bias-variance tradeoff} + + +We will discuss the bias-variance tradeoff in the context of +continuous predictions such as regression. However, many of the +intuitions and ideas discussed here also carry over to classification +tasks. Consider a dataset $\mathcal{L}$ consisting of the data +$\mathbf{X}_\mathcal{L}=\{(y_j, \boldsymbol{x}_j), j=0\ldots n-1\}$. + +Let us assume that the true data is generated from a noisy model + +\[ +\bm{y}=f(\boldsymbol{x}) + \bm{\epsilon} +\] + +where $\epsilon$ is normally distributed with mean zero and standard deviation $\sigma^2$. + +In our derivation of the ordinary least squares method we defined then +an approximation to the function $f$ in terms of the parameters +$\bm{\beta}$ and the design matrix $\bm{X}$ which embody our model, +that is $\bm{\tilde{y}}=\bm{X}\bm{\beta}$. + +Thereafter we found the parameters $\bm{\beta}$ by optimizing the means squared error via the so-called cost function +\[ +C(\bm{X},\bm{\beta}) =\frac{1}{n}\sum_{i=0}^{n-1}(y_i-\tilde{y}_i)^2=\mathbb{E}\left[(\bm{y}-\bm{\tilde{y}})^2\right]. +\] + +We can rewrite this as +\[ +\mathbb{E}\left[(\bm{y}-\bm{\tilde{y}})^2\right]=\frac{1}{n}\sum_i(f_i-\mathbb{E}\left[\bm{\tilde{y}}\right])^2+\frac{1}{n}\sum_i(\tilde{y}_i-\mathbb{E}\left[\bm{\tilde{y}}\right])^2+\sigma^2. +\] + +The three terms represent the square of the bias of the learning +method, which can be thought of as the error caused by the simplifying +assumptions built into the method. The second term represents the +variance of the chosen model and finally the last terms is variance of +the error $\bm{\epsilon}$. + +To derive this equation, we need to recall that the variance of $\bm{y}$ and $\bm{\epsilon}$ are both equal to $\sigma^2$. The mean value of $\bm{\epsilon}$ is by definition equal to zero. Furthermore, the function $f$ is not a stochastics variable, idem for $\bm{\tilde{y}}$. +We use a more compact notation in terms of the expectation value +\[ +\mathbb{E}\left[(\bm{y}-\bm{\tilde{y}})^2\right]=\mathbb{E}\left[(\bm{f}+\bm{\epsilon}-\bm{\tilde{y}})^2\right], +\] +and adding and subtracting $\mathbb{E}\left[\bm{\tilde{y}}\right]$ we get +\[ +\mathbb{E}\left[(\bm{y}-\bm{\tilde{y}})^2\right]=\mathbb{E}\left[(\bm{f}+\bm{\epsilon}-\bm{\tilde{y}}+\mathbb{E}\left[\bm{\tilde{y}}\right]-\mathbb{E}\left[\bm{\tilde{y}}\right])^2\right], +\] +which, using the abovementioned expectation values can be rewritten as +\[ +\mathbb{E}\left[(\bm{y}-\bm{\tilde{y}})^2\right]=\mathbb{E}\left[(\bm{y}-\mathbb{E}\left[\bm{\tilde{y}}\right])^2\right]+\mathrm{Var}\left[\bm{\tilde{y}}\right]+\sigma^2, +\] +that is the rewriting in terms of the so-called bias, the variance of the model $\bm{\tilde{y}}$ and the variance of $\bm{\epsilon}$. + + + + + +% !split +\subsection*{Example code for Bias-Variance tradeoff} +\begin{minted}[fontsize=\fontsize{9pt}{9pt},linenos=false,mathescape,baselinestretch=1.0,fontfamily=tt,xleftmargin=7mm]{python} +import matplotlib.pyplot as plt +import numpy as np +from sklearn.linear_model import LinearRegression, Ridge, Lasso +from sklearn.preprocessing import PolynomialFeatures +from sklearn.model_selection import train_test_split +from sklearn.pipeline import make_pipeline +from sklearn.utils import resample + +np.random.seed(2018) + +n = 500 +n_boostraps = 100 +degree = 18 # A quite high value, just to show. +noise = 0.1 + +# Make data set. +x = np.linspace(-1, 3, n).reshape(-1, 1) +y = np.exp(-x**2) + 1.5 * np.exp(-(x-2)**2) + np.random.normal(0, 0.1, x.shape) + +# Hold out some test data that is never used in training. +x_train, x_test, y_train, y_test = train_test_split(x, y, test_size=0.2) + +# Combine x transformation and model into one operation. +# Not neccesary, but convenient. +model = make_pipeline(PolynomialFeatures(degree=degree), LinearRegression(fit_intercept=False)) + +# The following (m x n_bootstraps) matrix holds the column vectors y_pred +# for each bootstrap iteration. +y_pred = np.empty((y_test.shape[0], n_boostraps)) +for i in range(n_boostraps): + x_, y_ = resample(x_train, y_train) + + # Evaluate the new model on the same test data each time. + y_pred[:, i] = model.fit(x_, y_).predict(x_test).ravel() + +# Note: Expectations and variances taken w.r.t. different training +# data sets, hence the axis=1. Subsequent means are taken across the test data +# set in order to obtain a total value, but before this we have error/bias/variance +# calculated per data point in the test set. +# Note 2: The use of keepdims=True is important in the calculation of bias as this +# maintains the column vector form. Dropping this yields very unexpected results. +error = np.mean( np.mean((y_test - y_pred)**2, axis=1, keepdims=True) ) +bias = np.mean( (y_test - np.mean(y_pred, axis=1, keepdims=True))**2 ) +variance = np.mean( np.var(y_pred, axis=1, keepdims=True) ) +print('Error:', error) +print('Bias^2:', bias) +print('Var:', variance) +print('{} >= {} + {} = {}'.format(error, bias, variance, bias+variance)) + +plt.plot(x[::5, :], y[::5, :], label='f(x)') +plt.scatter(x_test, y_test, label='Data points') +plt.scatter(x_test, np.mean(y_pred, axis=1), label='Pred') +plt.legend() +plt.show() + +\end{minted} + + +% !split +\subsection*{Understanding what happens} +\begin{minted}[fontsize=\fontsize{9pt}{9pt},linenos=false,mathescape,baselinestretch=1.0,fontfamily=tt,xleftmargin=7mm]{python} +import matplotlib.pyplot as plt +import numpy as np +from sklearn.linear_model import LinearRegression, Ridge, Lasso +from sklearn.preprocessing import PolynomialFeatures +from sklearn.model_selection import train_test_split +from sklearn.pipeline import make_pipeline +from sklearn.utils import resample + +np.random.seed(2018) + +n = 40 +n_boostraps = 100 +maxdegree = 14 + + +# Make data set. +x = np.linspace(-3, 3, n).reshape(-1, 1) +y = np.exp(-x**2) + 1.5 * np.exp(-(x-2)**2)+ np.random.normal(0, 0.1, x.shape) +error = np.zeros(maxdegree) +bias = np.zeros(maxdegree) +variance = np.zeros(maxdegree) +polydegree = np.zeros(maxdegree) +x_train, x_test, y_train, y_test = train_test_split(x, y, test_size=0.2) + +for degree in range(maxdegree): + model = make_pipeline(PolynomialFeatures(degree=degree), LinearRegression(fit_intercept=False)) + y_pred = np.empty((y_test.shape[0], n_boostraps)) + for i in range(n_boostraps): + x_, y_ = resample(x_train, y_train) + y_pred[:, i] = model.fit(x_, y_).predict(x_test).ravel() + + polydegree[degree] = degree + error[degree] = np.mean( np.mean((y_test - y_pred)**2, axis=1, keepdims=True) ) + bias[degree] = np.mean( (y_test - np.mean(y_pred, axis=1, keepdims=True))**2 ) + variance[degree] = np.mean( np.var(y_pred, axis=1, keepdims=True) ) + print('Polynomial degree:', degree) + print('Error:', error[degree]) + print('Bias^2:', bias[degree]) + print('Var:', variance[degree]) + print('{} >= {} + {} = {}'.format(error[degree], bias[degree], variance[degree], bias[degree]+variance[degree])) + +plt.plot(polydegree, np.log10(error), label='Error') +plt.plot(polydegree, bias, label='bias') +plt.plot(polydegree, variance, label='Variance') +plt.legend() +plt.show() + + + + +\end{minted} + +% !split +\subsection*{Summing up} + + + + +The bias-variance tradeoff summarizes the fundamental tension in +machine learning, particularly supervised learning, between the +complexity of a model and the amount of training data needed to train +it. Since data is often limited, in practice it is often useful to +use a less-complex model with higher bias, that is a model whose asymptotic +performance is worse than another model because it is easier to +train and less sensitive to sampling noise arising from having a +finite-sized training dataset (smaller variance). + + + +The above equations tell us that in +order to minimize the expected test error, we need to select a +statistical learning method that simultaneously achieves low variance +and low bias. Note that variance is inherently a nonnegative quantity, +and squared bias is also nonnegative. Hence, we see that the expected +test MSE can never lie below $Var(\epsilon)$, the irreducible error. + + +What do we mean by the variance and bias of a statistical learning +method? The variance refers to the amount by which our model would change if we +estimated it using a different training data set. Since the training +data are used to fit the statistical learning method, different +training data sets will result in a different estimate. But ideally the +estimate for our model should not vary too much between training +sets. However, if a method has high variance then small changes in +the training data can result in large changes in the model. In general, more +flexible statistical methods have higher variance. + + +% !split +\subsection*{Another Example rom Scikit-Learn's Repository} +\begin{minted}[fontsize=\fontsize{9pt}{9pt},linenos=false,mathescape,baselinestretch=1.0,fontfamily=tt,xleftmargin=7mm]{python} +""" +============================ +Underfitting vs. Overfitting +============================ + +This example demonstrates the problems of underfitting and overfitting and +how we can use linear regression with polynomial features to approximate +nonlinear functions. The plot shows the function that we want to approximate, +which is a part of the cosine function. In addition, the samples from the +real function and the approximations of different models are displayed. The +models have polynomial features of different degrees. We can see that a +linear function (polynomial with degree 1) is not sufficient to fit the +training samples. This is called **underfitting**. A polynomial of degree 4 +approximates the true function almost perfectly. However, for higher degrees +the model will **overfit** the training data, i.e. it learns the noise of the +training data. +We evaluate quantitatively **overfitting** / **underfitting** by using +cross-validation. We calculate the mean squared error (MSE) on the validation +set, the higher, the less likely the model generalizes correctly from the +training data. +""" + +print(__doc__) + +import numpy as np +import matplotlib.pyplot as plt +from sklearn.pipeline import Pipeline +from sklearn.preprocessing import PolynomialFeatures +from sklearn.linear_model import LinearRegression +from sklearn.model_selection import cross_val_score + + +def true_fun(X): + return np.cos(1.5 * np.pi * X) + +np.random.seed(0) + +n_samples = 30 +degrees = [1, 4, 15] + +X = np.sort(np.random.rand(n_samples)) +y = true_fun(X) + np.random.randn(n_samples) * 0.1 + +plt.figure(figsize=(14, 5)) +for i in range(len(degrees)): + ax = plt.subplot(1, len(degrees), i + 1) + plt.setp(ax, xticks=(), yticks=()) + + polynomial_features = PolynomialFeatures(degree=degrees[i], + include_bias=False) + linear_regression = LinearRegression() + pipeline = Pipeline([("polynomial_features", polynomial_features), + ("linear_regression", linear_regression)]) + pipeline.fit(X[:, np.newaxis], y) + + # Evaluate the models using crossvalidation + scores = cross_val_score(pipeline, X[:, np.newaxis], y, + scoring="neg_mean_squared_error", cv=10) + + X_test = np.linspace(0, 1, 100) + plt.plot(X_test, pipeline.predict(X_test[:, np.newaxis]), label="Model") + plt.plot(X_test, true_fun(X_test), label="True function") + plt.scatter(X, y, edgecolor='b', s=20, label="Samples") + plt.xlabel("x") + plt.ylabel("y") + plt.xlim((0, 1)) + plt.ylim((-2, 2)) + plt.legend(loc="best") + plt.title("Degree {}\nMSE = {:.2e}(+/- {:.2e})".format( + degrees[i], -scores.mean(), scores.std())) +plt.show() +\end{minted} + + + +% !split +\subsection*{The one-dimensional Ising model} + +Let us bring back the Ising model again, but now with an additional +focus on Ridge and Lasso regression as well. We repeat some of the +basic parts of the Ising model and the setup of the training and test +data. The one-dimensional Ising model with nearest neighbor +interaction, no external field and a constant coupling constant $J$ is +given by + +\begin{align} + H = -J \sum_{k}^L s_k s_{k + 1}, +\end{align} +where $s_i \in \{-1, 1\}$ and $s_{N + 1} = s_1$. The number of spins in the system is determined by $L$. For the one-dimensional system there is no phase transition. + +We will look at a system of $L = 40$ spins with a coupling constant of $J = 1$. To get enough training data we will generate 10000 states with their respective energies. + + +\begin{minted}[fontsize=\fontsize{9pt}{9pt},linenos=false,mathescape,baselinestretch=1.0,fontfamily=tt,xleftmargin=7mm]{python} +import numpy as np +import matplotlib.pyplot as plt +from mpl_toolkits.axes_grid1 import make_axes_locatable +import seaborn as sns +import scipy.linalg as scl +from sklearn.model_selection import train_test_split +import sklearn.linear_model as skl +import tqdm +sns.set(color_codes=True) +cmap_args=dict(vmin=-1., vmax=1., cmap='seismic') + +L = 40 +n = int(1e4) + +spins = np.random.choice([-1, 1], size=(n, L)) +J = 1.0 + +energies = np.zeros(n) + +for i in range(n): + energies[i] = - J * np.dot(spins[i], np.roll(spins[i], 1)) +\end{minted} + +A more general form for the one-dimensional Ising model is + +\begin{align} + H = - \sum_j^L \sum_k^L s_j s_k J_{jk}. +\end{align} + +Here we allow for interactions beyond the nearest neighbors and a more +adaptive coupling matrix. This latter expression can be formulated as +a matrix-product on the form +\begin{align} + H = X J, +\end{align} + +where $X_{jk} = s_j s_k$ and $J$ is the matrix consisting of the +elements $-J_{jk}$. This form of writing the energy fits perfectly +with the form utilized in linear regression, viz. +\begin{align} + \bm{y} = \bm{X}\bm{\beta} + \bm{\epsilon}. +\end{align} +We organize the data as we did above +\begin{minted}[fontsize=\fontsize{9pt}{9pt},linenos=false,mathescape,baselinestretch=1.0,fontfamily=tt,xleftmargin=7mm]{python} +X = np.zeros((n, L ** 2)) +for i in range(n): + X[i] = np.outer(spins[i], spins[i]).ravel() +y = energies +X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.96) + +X_train_own = np.concatenate( + (np.ones(len(X_train))[:, np.newaxis], X_train), + axis=1 +) + +X_test_own = np.concatenate( + (np.ones(len(X_test))[:, np.newaxis], X_test), + axis=1 +) +\end{minted} + +We will do all fitting with \textbf{Scikit-Learn}, + +\begin{minted}[fontsize=\fontsize{9pt}{9pt},linenos=false,mathescape,baselinestretch=1.0,fontfamily=tt,xleftmargin=7mm]{python} +clf = skl.LinearRegression().fit(X_train, y_train) +\end{minted} +When extracting the $J$-matrix we make sure to remove the intercept +\begin{minted}[fontsize=\fontsize{9pt}{9pt},linenos=false,mathescape,baselinestretch=1.0,fontfamily=tt,xleftmargin=7mm]{python} +J_sk = clf.coef_.reshape(L, L) +\end{minted} +And then we plot the results +\begin{minted}[fontsize=\fontsize{9pt}{9pt},linenos=false,mathescape,baselinestretch=1.0,fontfamily=tt,xleftmargin=7mm]{python} +fig = plt.figure(figsize=(20, 14)) +im = plt.imshow(J_sk, **cmap_args) +plt.title("LinearRegression from Scikit-learn", fontsize=18) +plt.xticks(fontsize=18) +plt.yticks(fontsize=18) +cb = fig.colorbar(im) +cb.ax.set_yticklabels(cb.ax.get_yticklabels(), fontsize=18) +plt.show() +\end{minted} +The results perfectly with our previous discussion where we used our own code. + +% !split +\subsection*{Ridge regression} + +Having explored the ordinary least squares we move on to ridge +regression. In ridge regression we include a \textbf{regularizer}. This +involves a new cost function which leads to a new estimate for the +weights $\bm{\beta}$. This results in a penalized regression problem. The +cost function is given by + +\begin{align} + C(\bm{X}, \bm{\beta}; \lambda) = (\bm{X}\bm{\beta} - \bm{y})^T(\bm{X}\bm{\beta} - \bm{y}) + \lambda \bm{\beta}^T\bm{\beta}. +\end{align} +\begin{minted}[fontsize=\fontsize{9pt}{9pt},linenos=false,mathescape,baselinestretch=1.0,fontfamily=tt,xleftmargin=7mm]{python} +_lambda = 0.1 +clf_ridge = skl.Ridge(alpha=_lambda).fit(X_train, y_train) +J_ridge_sk = clf_ridge.coef_.reshape(L, L) +fig = plt.figure(figsize=(20, 14)) +im = plt.imshow(J_ridge_sk, **cmap_args) +plt.title("Ridge from Scikit-learn", fontsize=18) +plt.xticks(fontsize=18) +plt.yticks(fontsize=18) +cb = fig.colorbar(im) +cb.ax.set_yticklabels(cb.ax.get_yticklabels(), fontsize=18) + +plt.show() +\end{minted} + +% !split +\subsection*{LASSO regression} + +In the \textbf{Least Absolute Shrinkage and Selection Operator} (LASSO)-method we get a third cost function. + +\begin{align} + C(\bm{X}, \bm{\beta}; \lambda) = (\bm{X}\bm{\beta} - \bm{y})^T(\bm{X}\bm{\beta} - \bm{y}) + \lambda \sqrt{\bm{\beta}^T\bm{\beta}}. +\end{align} + +Finding the extremal point of this cost function is not so straight-forward as in least squares and ridge. We will therefore rely solely on the function ``Lasso`` from \textbf{Scikit-Learn}. + +\begin{minted}[fontsize=\fontsize{9pt}{9pt},linenos=false,mathescape,baselinestretch=1.0,fontfamily=tt,xleftmargin=7mm]{python} +clf_lasso = skl.Lasso(alpha=_lambda).fit(X_train, y_train) +J_lasso_sk = clf_lasso.coef_.reshape(L, L) +fig = plt.figure(figsize=(20, 14)) +im = plt.imshow(J_lasso_sk, **cmap_args) +plt.title("Lasso from Scikit-learn", fontsize=18) +plt.xticks(fontsize=18) +plt.yticks(fontsize=18) +cb = fig.colorbar(im) +cb.ax.set_yticklabels(cb.ax.get_yticklabels(), fontsize=18) + +plt.show() +\end{minted} + +It is quite striking how LASSO breaks the symmetry of the coupling +constant as opposed to ridge and OLS. We get a sparse solution with +$J_{j, j + 1} = -1$. + + + +% !split +\subsection*{Performance as function of the regularization parameter} + +We see how the different models perform for a different set of values for $\lambda$. + + +\begin{minted}[fontsize=\fontsize{9pt}{9pt},linenos=false,mathescape,baselinestretch=1.0,fontfamily=tt,xleftmargin=7mm]{python} +lambdas = np.logspace(-4, 5, 10) + +train_errors = { + "ols_sk": np.zeros(lambdas.size), + "ridge_sk": np.zeros(lambdas.size), + "lasso_sk": np.zeros(lambdas.size) +} + +test_errors = { + "ols_sk": np.zeros(lambdas.size), + "ridge_sk": np.zeros(lambdas.size), + "lasso_sk": np.zeros(lambdas.size) +} + +plot_counter = 1 + +fig = plt.figure(figsize=(32, 54)) + +for i, _lambda in enumerate(tqdm.tqdm(lambdas)): + for key, method in zip( + ["ols_sk", "ridge_sk", "lasso_sk"], + [skl.LinearRegression(), skl.Ridge(alpha=_lambda), skl.Lasso(alpha=_lambda)] + ): + method = method.fit(X_train, y_train) + + train_errors[key][i] = method.score(X_train, y_train) + test_errors[key][i] = method.score(X_test, y_test) + + omega = method.coef_.reshape(L, L) + + plt.subplot(10, 5, plot_counter) + plt.imshow(omega, **cmap_args) + plt.title(r"%s, $\lambda = %.4f$" % (key, _lambda)) + plot_counter += 1 + +plt.show() +\end{minted} + +We see that LASSO reaches a good solution for low +values of $\lambda$, but will "wither" when we increase $\lambda$ too +much. Ridge is more stable over a larger range of values for +$\lambda$, but eventually also fades away. + +% !split +\subsection*{Finding the optimal value of $\lambda$} + +To determine which value of $\lambda$ is best we plot the accuracy of +the models when predicting the training and the testing set. We expect +the accuracy of the training set to be quite good, but if the accuracy +of the testing set is much lower this tells us that we might be +subject to an overfit model. The ideal scenario is an accuracy on the +testing set that is close to the accuracy of the training set. + + +\begin{minted}[fontsize=\fontsize{9pt}{9pt},linenos=false,mathescape,baselinestretch=1.0,fontfamily=tt,xleftmargin=7mm]{python} +fig = plt.figure(figsize=(20, 14)) + +colors = { + "ols_sk": "r", + "ridge_sk": "y", + "lasso_sk": "c" +} + +for key in train_errors: + plt.semilogx( + lambdas, + train_errors[key], + colors[key], + label="Train {0}".format(key), + linewidth=4.0 + ) + +for key in test_errors: + plt.semilogx( + lambdas, + test_errors[key], + colors[key] + "--", + label="Test {0}".format(key), + linewidth=4.0 + ) +plt.legend(loc="best", fontsize=18) +plt.xlabel(r"$\lambda$", fontsize=18) +plt.ylabel(r"$R^2$", fontsize=18) +plt.tick_params(labelsize=18) +plt.show() +\end{minted} + +From the above figure we can see that LASSO with $\lambda = 10^{-2}$ +achieves a very good accuracy on the test set. This by far surpasses the +other models for all values of $\lambda$. + + + +% !split +\subsection*{Further Exercises} + +\paragraph{Exercise 1.} +We will generate our own dataset for a function $y(x)$ where $x \in [0,1]$ and defined by random numbers computed with the uniform distribution. The function $y$ is a quadratic polynomial in $x$ with added stochastic noise according to the normal distribution $\cal {N}(0,1)$. +The following simple Python instructions define our $x$ and $y$ values (with 100 data points). +\begin{minted}[fontsize=\fontsize{9pt}{9pt},linenos=false,mathescape,baselinestretch=1.0,fontfamily=tt,xleftmargin=7mm]{python} +x = np.random.rand(100,1) +y = 5*x*x+0.1*np.random.randn(100,1) +\end{minted} + +\begin{enumerate} +\item Write your own code (following the examples above) for computing the parametrization of the data set fitting a second-order polynomial. + +\item Use thereafter \textbf{scikit-learn} (see again the examples in the regression slides) and compare with your own code. + +\item Using scikit-learn, compute also the mean square error, a risk metric corresponding to the expected value of the squared (quadratic) error defined as +\end{enumerate} + +\noindent +\[ MSE(\hat{y},\hat{\tilde{y}}) = \frac{1}{n} +\sum_{i=0}^{n-1}(y_i-\tilde{y}_i)^2, +\] +and the $R^2$ score function. +If $\tilde{\hat{y}}_i$ is the predicted value of the $i-th$ sample and $y_i$ is the corresponding true value, then the score $R^2$ is defined as +\[ +R^2(\hat{y}, \tilde{\hat{y}}) = 1 - \frac{\sum_{i=0}^{n - 1} (y_i - \tilde{y}_i)^2}{\sum_{i=0}^{n - 1} (y_i - \bar{y})^2}, +\] +where we have defined the mean value of $\hat{y}$ as +\[ +\bar{y} = \frac{1}{n} \sum_{i=0}^{n - 1} y_i. +\] + +You can use the functionality included in scikit-learn. If you feel +for it, you can use your own program and define functions which +compute the above two functions. Discuss the meaning of these +results. Try also to vary the coefficient in front of the added +stochastic noise term and discuss the quality of the fits. + + + + +\paragraph{Exercise 2, variance of the parameters $\beta$ in linear regression.} +Show that the variance of the parameters $\beta$ in the linear regression method (chapter 3, equation (3.8) of \href{{https://www.springer.com/gp/book/9780387848570}}{Trevor Hastie, Robert Tibshirani, Jerome H. Friedman, The Elements of Statistical Learning, Springer}) is given as + +\[ +\mathrm{Var}(\hat{\beta}) = \left(\hat{X}^T\hat{X}\right)^{-1}\sigma^2, +\] +with +\[ +\sigma^2 = \frac{1}{N-p-1}\sum_{i=1}^{N} (y_i-\tilde{y}_i)^2, +\] +where we have assumed that we fit a function of degree $p-1$ (for example a polynomial in $x$). + + + +\paragraph{Exercise 3.} +This exercise is a continuation of exercise 1. We will +use the same function to generate our data set, still staying with a +simple function $y(x)$ which we want to fit using linear regression, +but now extending the analysis to include the Ridge and the Lasso +regression methods. You can use the code under the Regression as an example on how to use the Ridge and the Lasso methods. + +We will thus again generate our own dataset for a function $y(x)$ where +$x \in [0,1]$ and defined by random numbers computed with the uniform +distribution. The function $y$ is a quadratic polynomial in $x$ with +added stochastic noise according to the normal distribution $\cal{N}(0,1)$. + +The following simple Python instructions define our $x$ and $y$ values (with 100 data points). +\begin{minted}[fontsize=\fontsize{9pt}{9pt},linenos=false,mathescape,baselinestretch=1.0,fontfamily=tt,xleftmargin=7mm]{python} +x = np.random.rand(100,1) +y = 5*x*x+0.1*np.random.randn(100,1) +\end{minted} + +\begin{enumerate} +\item Write your own code for the Ridge method and compute the parametrization for different values of $\lambda$. Compare and analyze your results with those from exercise 1. Study the dependence on $\lambda$ while also varying the strength of the noise in your expression for $y(x)$. + +\item Repeat the above but using the functionality of \textbf{scikit-learn}. Compare your code with the results from \textbf{scikit-learn}. Remember to run with the same random numbers for generating $x$ and $y$. + +\item Our next step is to study the variance of the parameters $\beta_1$ and $\beta_2$ (assuming that we are parametrizing our function with a second-order polynomial. We will use standard linear regression and the Ridge regression. You can now opt for either writing your own function that calculates the variance of these paramaters (recall that this is equal to the diagonal elements of the matrix $(\hat{X}^T\hat{X})+\lambda\hat{I})^{-1}$) or use the functionality of \textbf{scikit-learn} and compute their variances. Discuss the results of these variances as functions + +\item Repeat the previous step but add now the Lasso method. Discuss your results and compare with standard regression and the Ridge regression results. + +\item Try to implement the cross-validation as well. + +\item Finally, using \textbf{scikit-learn} or your own code, compute also the mean square error, a risk metric corresponding to the expected value of the squared (quadratic) error defined as +\end{enumerate} + +\noindent +\[ MSE(\hat{y},\hat{\tilde{y}}) = \frac{1}{n} +\sum_{i=0}^{n-1}(y_i-\tilde{y}_i)^2, +\] +and the $R^2$ score function. +If $\tilde{\hat{y}}_i$ is the predicted value of the $i-th$ sample and $y_i$ is the corresponding true value, then the score $R^2$ is defined as +\[ +R^2(\hat{y}, \tilde{\hat{y}}) = 1 - \frac{\sum_{i=0}^{n - 1} (y_i - \tilde{y}_i)^2}{\sum_{i=0}^{n - 1} (y_i - \bar{y})^2}, +\] +where we have defined the mean value of $\hat{y}$ as +\[ +\bar{y} = \frac{1}{n} \sum_{i=0}^{n - 1} y_i. +\] +Discuss these quantities as functions of the variable $\lambda$ in the Ridge and Lasso regression methods. + +\paragraph{Exercise 4.} +We will study how +to fit polynomials to a specific two-dimensional function called +\href{{http://www.dtic.mil/dtic/tr/fulltext/u2/a081688.pdf}}{Franke's +function}. This +is a function which has been widely used when testing various interpolation and fitting +algorithms. Furthermore, after having established the model and the +method, we will employ resamling techniques such as the cross-validation and/or +the bootstrap methods, in order to perform a proper assessment of our models. + + +The Franke function, which is a weighted sum of four exponentials reads as follows +\begin{align*} +f(x,y) &= \frac{3}{4}\exp{\left(-\frac{(9x-2)^2}{4} - \frac{(9y-2)^2}{4}\right)}+\frac{3}{4}\exp{\left(-\frac{(9x+1)^2}{49}- \frac{(9y+1)}{10}\right)} \\ +&+\frac{1}{2}\exp{\left(-\frac{(9x-7)^2}{4} - \frac{(9y-3)^2}{4}\right)} -\frac{1}{5}\exp{\left(-(9x-4)^2 - (9y-7)^2\right) }. +\end{align*} + +The function will be defined for $x,y\in [0,1]$. Our first step will +be to perform an OLS regression analysis of this function, trying out +a polynomial fit with an $x$ and $y$ dependence of the form $[x, y, +x^2, y^2, xy, \dots]$. We will also include cross-validation and +bootstrap as resampling techniques. As in homeworks 1 and 2, we +can use a uniform distribution to set up the arrays of values for $x$ +and $y$, or as in the example below just a fix values for $x$ and $y$ with a given step size. +In this case we will have two predictors and need to fit a +function (for example a polynomial) of $x$ and $y$. Thereafter we will +repeat much of the same procedure using the the Ridge and +Lasso regression methods, introducing thus a dependence on the bias +(penalty) $\lambda$. + + +The Python fucntion for the Franke function is included here (it performs also a three-dimensional plot of it) +\begin{minted}[fontsize=\fontsize{9pt}{9pt},linenos=false,mathescape,baselinestretch=1.0,fontfamily=tt,xleftmargin=7mm]{python} +from mpl_toolkits.mplot3d import Axes3D +import matplotlib.pyplot as plt +from matplotlib import cm +from matplotlib.ticker import LinearLocator, FormatStrFormatter +import numpy as np +from random import random, seed + +fig = plt.figure() +ax = fig.gca(projection='3d') + +# Make data. +x = np.arange(0, 1, 0.05) +y = np.arange(0, 1, 0.05) +x, y = np.meshgrid(x,y) + + +def FrankeFunction(x,y): + term1 = 0.75*np.exp(-(0.25*(9*x-2)**2) - 0.25*((9*y-2)**2)) + term2 = 0.75*np.exp(-((9*x+1)**2)/49.0 - 0.1*(9*y+1)) + term3 = 0.5*np.exp(-(9*x-7)**2/4.0 - 0.25*((9*y-3)**2)) + term4 = -0.2*np.exp(-(9*x-4)**2 - (9*y-7)**2) + return term1 + term2 + term3 + term4 + + +z = FrankeFunction(x, y) + +# Plot the surface. +surf = ax.plot_surface(x, y, z, cmap=cm.coolwarm, + linewidth=0, antialiased=False) + +# Customize the z axis. +ax.set_zlim(-0.10, 1.40) +ax.zaxis.set_major_locator(LinearLocator(10)) +ax.zaxis.set_major_formatter(FormatStrFormatter('%.02f')) + +# Add a color bar which maps values to colors. +fig.colorbar(surf, shrink=0.5, aspect=5) + +plt.show() + +\end{minted} + + +We will thus again generate our own dataset for a function $\mathrm{FrankeFunction}(x,y)$ where +$x,y \in [0,1]$ could be defined by random numbers computed with the uniform +distribution. The function $f(x,y)$ is the Franke function. You should explore also the addition +an added stochastic noise to this function using the normal distribution $\cal{N}(0,1)$. + +Write your own code (using either a matrix inversion or a singular value decomposition from e.g., \textbf{numpy} ) or use your code from exercises 1 and 3 +and perform a standard least square regression analysis using polynomials in $x$ and $y$ up to fifth order. Find the confidence intervals of the parameters $\beta$ by computing their variances, evaluate the Mean Squared error (MSE) +\[ MSE(\hat{y},\hat{\tilde{y}}) = \frac{1}{n} +\sum_{i=0}^{n-1}(y_i-\tilde{y}_i)^2, +\] +and the $R^2$ score function. +If $\tilde{\hat{y}}_i$ is the predicted value of the $i-th$ sample and $y_i$ is the corresponding true value, then the score $R^2$ is defined as +\[ +R^2(\hat{y}, \tilde{\hat{y}}) = 1 - \frac{\sum_{i=0}^{n - 1} (y_i - \tilde{y}_i)^2}{\sum_{i=0}^{n - 1} (y_i - \bar{y})^2}, +\] +where we have defined the mean value of $\hat{y}$ as +\[ +\bar{y} = \frac{1}{n} \sum_{i=0}^{n - 1} y_i. +\] + +Perform a resampling of the data where you split the data in training data and test data. Implement the $k$-fold cross-validation algorithm and/or the bootstrap algorithm +and evaluate again the MSE and the $R^2$ functions resulting from the test data. Evaluate also the bias and variance of the final models. + + +Write then your own code for the Ridge method, either using matrix +inversion or the singular value decomposition as done for standard OLS. Perform the same analysis as in the +previous exercise (for the same polynomials and include resampling +techniques) but now for different values of $\lambda$. Compare and +analyze your results with those obtained with standard OLS. Study the +dependence on $\lambda$ while also varying eventually the strength of +the noise in your expression for $\mathrm{FrankeFunction}(x,y)$. + +Then perform the same studies but now with Lasso regression. Use the functionalities of +\textbf{scikit-learn}. Give a critical discussion of the three methods and a +judgement of which model fits the data best. + + +% ------------------- end of main content --------------- + +\end{document} + diff --git a/doc/src/Regression/Regression.tex.old~~ b/doc/src/Regression/Regression.tex.old~~ new file mode 100644 index 000000000..53761ed23 --- /dev/null +++ b/doc/src/Regression/Regression.tex.old~~ @@ -0,0 +1,3673 @@ +%% +%% Automatically generated file from DocOnce source +%% (https://github.com/hplgit/doconce/) +%% +%% + + +%-------------------- begin preamble ---------------------- + +\documentclass[% +oneside, % oneside: electronic viewing, twoside: printing +final, % draft: marks overfull hboxes, figures with paths +10pt]{article} + +\listfiles % print all files needed to compile this document + +\usepackage{relsize,makeidx,color,setspace,amsmath,amsfonts,amssymb} +\usepackage[table]{xcolor} +\usepackage{bm,ltablex,microtype} + +\usepackage[pdftex]{graphicx} + +\usepackage{fancyvrb} % packages needed for verbatim environments +\usepackage{minted} +\usemintedstyle{default} + +\usepackage[T1]{fontenc} +%\usepackage[latin1]{inputenc} +\usepackage{ucs} +\usepackage[utf8x]{inputenc} + +\usepackage{lmodern} % Latin Modern fonts derived from Computer Modern + +% Hyperlinks in PDF: +\definecolor{linkcolor}{rgb}{0,0,0.4} +\usepackage{hyperref} +\hypersetup{ + breaklinks=true, + colorlinks=true, + linkcolor=linkcolor, + urlcolor=linkcolor, + citecolor=black, + filecolor=black, + %filecolor=blue, + pdfmenubar=true, + pdftoolbar=true, + bookmarksdepth=3 % Uncomment (and tweak) for PDF bookmarks with more levels than the TOC + } +%\hyperbaseurl{} % hyperlinks are relative to this root + +\setcounter{tocdepth}{2} % levels in table of contents + +% --- fancyhdr package for fancy headers --- +\usepackage{fancyhdr} +\fancyhf{} % sets both header and footer to nothing +\renewcommand{\headrulewidth}{0pt} +\fancyfoot[LE,RO]{\thepage} +% Ensure copyright on titlepage (article style) and chapter pages (book style) +\fancypagestyle{plain}{ + \fancyhf{} + \fancyfoot[C]{{\footnotesize \copyright\ 1999-2019, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license}} +% \renewcommand{\footrulewidth}{0mm} + \renewcommand{\headrulewidth}{0mm} +} +% Ensure copyright on titlepages with \thispagestyle{empty} +\fancypagestyle{empty}{ + \fancyhf{} + \fancyfoot[C]{{\footnotesize \copyright\ 1999-2019, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license}} + \renewcommand{\footrulewidth}{0mm} + \renewcommand{\headrulewidth}{0mm} +} + +\pagestyle{fancy} + + +\usepackage[framemethod=TikZ]{mdframed} + +% --- begin definitions of admonition environments --- + +% --- end of definitions of admonition environments --- + +% prevent orhpans and widows +\clubpenalty = 10000 +\widowpenalty = 10000 + +% --- end of standard preamble for documents --- + + +% insert custom LaTeX commands... + +\raggedbottom +\makeindex +\usepackage[totoc]{idxlayout} % for index in the toc +\usepackage[nottoc]{tocbibind} % for references/bibliography in the toc + +%-------------------- end preamble ---------------------- + +\begin{document} + +% matching end for #ifdef PREAMBLE + +\newcommand{\exercisesection}[1]{\subsection*{#1}} + + +% ------------------- main content ---------------------- + + + +% ----------------- title ------------------------- + +\thispagestyle{empty} + +\begin{center} +{\LARGE\bf +\begin{spacing}{1.25} +Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis +\end{spacing} +} +\end{center} + +% ----------------- author(s) ------------------------- + +\begin{center} +{\bf Morten Hjorth-Jensen${}^{1, 2}$} \\ [0mm] +\end{center} + +\begin{center} +% List of all institutions: +\centerline{{\small ${}^1$Department of Physics, University of Oslo}} +\centerline{{\small ${}^2$Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University}} +\end{center} + +% ----------------- end author(s) ------------------------- + +% --- begin date --- +\begin{center} +Jul 22, 2019 +\end{center} +% --- end date --- + +\vspace{1cm} + + +% !split +\subsection{Why Linear Regression (aka Ordinary Least Squares and family)} + +Fitting a continuous function with linear parameterization in terms of the parameters $\bm{\beta}$. +\begin{itemize} +\item Method of choice for fitting a continuous function! + +\item Gives an excellent introduction to central Machine Learning features with \textbf{understandable pedagogical} links to other methods like \textbf{Neural Networks}, \textbf{Support Vector Machines} etc + +\item Analytical expression for the fitting parameters $\bm{\beta}$ + +\item Analytical expressions for statistical propertiers like mean values, variances, confidence intervals and more + +\item Analytical relation with probabilistic interpretations + +\item Easy to introduce basic concepts like bias-variance tradeoff, cross-validation, resampling and regularization techniques and many other ML topics + +\item Easy to code! And links well with classification problems and logistic regression and neural networks + +\item Allows for \textbf{easy} hands-on understanding of gradient descent methods + +\item and many more features +\end{itemize} + +\noindent +For more discussions of Ridge and Lasso regression, \href{{https://arxiv.org/abs/1509.09169}}{Wessel van Wieringen's} article is highly recommended. +Similarly, \href{{https://arxiv.org/abs/1803.08823}}{Mehta et al's article} is also recommended. + + +% !split +\subsection{Regression analysis, overarching aims} + +% --- begin paragraph admon --- +\paragraph{} + +Regression modeling deals with the description of the sampling distribution of a given random variable $y$ and how it varies as function of another variable or a set of such variables $\bm{x} =[x_0, x_1,\dots, x_{n-1}]^T$. +The first variable is called the \textbf{dependent}, the \textbf{outcome} or the \textbf{response} variable while the set of variables $\bm{x}$ is called the independent variable, or the predictor variable or the explanatory variable. + +A regression model aims at finding a likelihood function $p(\bm{y}\vert \bm{x})$, that is the conditional distribution for $\bm{y}$ with a given $\bm{x}$. The estimation of $p(\bm{y}\vert \bm{x})$ is made using a data set with +\begin{itemize} +\item $n$ cases $i = 0, 1, 2, \dots, n-1$ + +\item Response (target, dependent or outcome) variable $y_i$ with $i = 0, 1, 2, \dots, n-1$ + +\item $p$ so-called explanatory (independent or predictor) variables $\bm{x}_i=[x_{i0}, x_{i1}, \dots, x_{ip-1}]$ with $i = 0, 1, 2, \dots, n-1$ and explanatory variables running from $0$ to $p-1$. See below for more explicit examples. +\end{itemize} + +\noindent + The goal of the regression analysis is to extract/exploit relationship between $\bm{y}$ and $\bm{X}$ in or to infer causal dependencies, approximations to the likelihood functions, functional relationships and to make predictions, making fits and many other things. +% --- end paragraph admon --- + + + +% !split +\subsection{Regression analysis, overarching aims II} + +% --- begin paragraph admon --- +\paragraph{} + + +Consider an experiment in which $p$ characteristics of $n$ samples are +measured. The data from this experiment, for various explanatory variables $p$ are normally represented by a matrix +$\mathbf{X}$. + +The matrix $\mathbf{X}$ is called the \emph{design +matrix}. Additional information of the samples is available in the +form of $\bm{y}$ (also as above). The variable $\bm{y}$ is +generally referred to as the \emph{response variable}. The aim of +regression analysis is to explain $\bm{y}$ in terms of +$\bm{X}$ through a functional relationship like $y_i = +f(\mathbf{X}_{i,\ast})$. When no prior knowledge on the form of +$f(\cdot)$ is available, it is common to assume a linear relationship +between $\bm{X}$ and $\bm{y}$. This assumption gives rise to +the \emph{linear regression model} where $\bm{\beta} = [\beta_0, \ldots, +\beta_{p-1}]^{T}$ are the \emph{regression parameters}. + +Linear regression gives us a set of analytical equations for the parameters $\beta_j$. +% --- end paragraph admon --- + + + + + +% !split +\subsection{Examples} + +% --- begin paragraph admon --- +\paragraph{} +In order to understand the relation among the predictors $p$, the set of data $n$ and the target (outcome, output etc) $\bm{y}$, +consider the model we discussed for describing nuclear binding energies. + +There we assumed that we could parametrize the data using a polynomial approximation based on the liquid drop model. +Assuming +\[ +BE(A) = a_0+a_1A+a_2A^{2/3}+a_3A^{-1/3}+a_4A^{-1}, +\] +we have five predictors, that is the intercept, the $A$ dependent term, the $A^{2/3}$ term and the $A^{-1/3}$ and $A^{-1}$ terms. +This gives $p=0,1,2,3,4$. Furthermore we have $n$ entries for each predictor. It means that our design matrix is a +$p\times n$ matrix $\bm{X}$. + +Here the predictors are based on a model we have made. A popular data set which is widely encountered in ML applications is the +so-called \href{{https://www.sciencedirect.com/science/article/pii/S0957417407006719?via%3Dihub}}{credit card default data from Taiwan}. The data set contains data on $n=30000$ credit card holders with predictors like gender, marital status, age, profession, education, etc. In total there are $24$ such predictors or attributes leading to a design matrix of dimensionality $24 \times 30000$ +% --- end paragraph admon --- + + + + + + + +% !split +\subsection{General linear models} + +% --- begin paragraph admon --- +\paragraph{} +Before we proceed let us study a case from linear algebra where we aim at fitting a set of data $\bm{y}=[y_0,y_1,\dots,y_{n-1}]$. We could think of these data as a result of an experiment or a complicated numerical experiment. These data are functions of a series of variables $\bm{x}=[x_0,x_1,\dots,x_{n-1}]$, that is $y_i = y(x_i)$ with $i=0,1,2,\dots,n-1$. The variables $x_i$ could represent physical quantities like time, temperature, position etc. We assume that $y(x)$ is a smooth function. + +Since obtaining these data points may not be trivial, we want to use these data to fit a function which can allow us to make predictions for values of $y$ which are not in the present set. The perhaps simplest approach is to assume we can parametrize our function in terms of a polynomial of degree $n-1$ with $n$ points, that is +\[ +y=y(x) \rightarrow y(x_i)=\tilde{y}_i+\epsilon_i=\sum_{j=0}^{n-1} \beta_j x_i^j+\epsilon_i, +\] +where $\epsilon_i$ is the error in our approximation. +% --- end paragraph admon --- + + + + +% !split +\subsection{Rewriting the fitting procedure as a linear algebra problem} + +% --- begin paragraph admon --- +\paragraph{} +For every set of values $y_i,x_i$ we have thus the corresponding set of equations +\begin{align*} +y_0&=\beta_0+\beta_1x_0^1+\beta_2x_0^2+\dots+\beta_{n-1}x_0^{n-1}+\epsilon_0\\ +y_1&=\beta_0+\beta_1x_1^1+\beta_2x_1^2+\dots+\beta_{n-1}x_1^{n-1}+\epsilon_1\\ +y_2&=\beta_0+\beta_1x_2^1+\beta_2x_2^2+\dots+\beta_{n-1}x_2^{n-1}+\epsilon_2\\ +\dots & \dots \\ +y_{n-1}&=\beta_0+\beta_1x_{n-1}^1+\beta_2x_{n-1}^2+\dots+\beta_{n-1}x_{n-1}^{n-1}+\epsilon_{n-1}.\\ +\end{align*} +% --- end paragraph admon --- + + + + +% !split +\subsection{Rewriting the fitting procedure as a linear algebra problem, more details} + +% --- begin paragraph admon --- +\paragraph{} +Defining the vectors +\[ +\bm{y} = [y_0,y_1, y_2,\dots, y_{n-1}]^T, +\] +and +\[ +\bm{\beta} = [\beta_0,\beta_1, \beta_2,\dots, \beta_{n-1}]^T, +\] +and +\[ +\bm{\epsilon} = [\epsilon_0,\epsilon_1, \epsilon_2,\dots, \epsilon_{n-1}]^T, +\] +and the design matrix +\[ +\bm{X}= +\begin{bmatrix} +1& x_{0}^1 &x_{0}^2& \dots & \dots &x_{0}^{n-1}\\ +1& x_{1}^1 &x_{1}^2& \dots & \dots &x_{1}^{n-1}\\ +1& x_{2}^1 &x_{2}^2& \dots & \dots &x_{2}^{n-1}\\ +\dots& \dots &\dots& \dots & \dots &\dots\\ +1& x_{n-1}^1 &x_{n-1}^2& \dots & \dots &x_{n-1}^{n-1}\\ +\end{bmatrix} +\] +we can rewrite our equations as +\[ +\bm{y} = \bm{X}\bm{\beta}+\bm{\epsilon}. +\] +The above design matrix is called a \href{{https://en.wikipedia.org/wiki/Vandermonde_matrix}}{Vandermonde matrix}. +% --- end paragraph admon --- + + + + +% !split +\subsection{Generalizing the fitting procedure as a linear algebra problem} + +% --- begin paragraph admon --- +\paragraph{} + +We are obviously not limited to the above polynomial expansions. We +could replace the various powers of $x$ with elements of Fourier +series or instead of $x_i^j$ we could have $\cos{(j x_i)}$ or $\sin{(j +x_i)}$, or time series or other orthogonal functions. For every set +of values $y_i,x_i$ we can then generalize the equations to + +\begin{align*} +y_0&=\beta_0x_{00}+\beta_1x_{01}+\beta_2x_{02}+\dots+\beta_{n-1}x_{0n-1}+\epsilon_0\\ +y_1&=\beta_0x_{10}+\beta_1x_{11}+\beta_2x_{12}+\dots+\beta_{n-1}x_{1n-1}+\epsilon_1\\ +y_2&=\beta_0x_{20}+\beta_1x_{21}+\beta_2x_{22}+\dots+\beta_{n-1}x_{2n-1}+\epsilon_2\\ +\dots & \dots \\ +y_{i}&=\beta_0x_{i0}+\beta_1x_{i1}+\beta_2x_{i2}+\dots+\beta_{n-1}x_{in-1}+\epsilon_i\\ +\dots & \dots \\ +y_{n-1}&=\beta_0x_{n-1,0}+\beta_1x_{n-1,2}+\beta_2x_{n-1,2}+\dots+\beta_{n-1}x_{n-1,n-1}+\epsilon_{n-1}.\\ +\end{align*} + +\textbf{Note that we have $p=n$ here. The matrix is symmetric. This is generally not the case!} +% --- end paragraph admon --- + + + + +% !split +\subsection{Generalizing the fitting procedure as a linear algebra problem} + +% --- begin paragraph admon --- +\paragraph{} +We redefine in turn the matrix $\bm{X}$ as +\[ +\bm{X}= +\begin{bmatrix} +x_{00}& x_{01} &x_{02}& \dots & \dots &x_{0,n-1}\\ +x_{10}& x_{11} &x_{12}& \dots & \dots &x_{1,n-1}\\ +x_{20}& x_{21} &x_{22}& \dots & \dots &x_{2,n-1}\\ +\dots& \dots &\dots& \dots & \dots &\dots\\ +x_{n-1,0}& x_{n-1,1} &x_{n-1,2}& \dots & \dots &x_{n-1,n-1}\\ +\end{bmatrix} +\] +and without loss of generality we rewrite again our equations as +\[ +\bm{y} = \bm{X}\bm{\beta}+\bm{\epsilon}. +\] +The left-hand side of this equation is kwown. Our error vector $\bm{\epsilon}$ and the parameter vector $\bm{\beta}$ are our unknow quantities. How can we obtain the optimal set of $\beta_i$ values? +% --- end paragraph admon --- + + + + +% !split +\subsection{Optimizing our parameters} + +% --- begin paragraph admon --- +\paragraph{} +We have defined the matrix $\bm{X}$ via the equations +\begin{align*} +y_0&=\beta_0x_{00}+\beta_1x_{01}+\beta_2x_{02}+\dots+\beta_{n-1}x_{0n-1}+\epsilon_0\\ +y_1&=\beta_0x_{10}+\beta_1x_{11}+\beta_2x_{12}+\dots+\beta_{n-1}x_{1n-1}+\epsilon_1\\ +y_2&=\beta_0x_{20}+\beta_1x_{21}+\beta_2x_{22}+\dots+\beta_{n-1}x_{2n-1}+\epsilon_1\\ +\dots & \dots \\ +y_{i}&=\beta_0x_{i0}+\beta_1x_{i1}+\beta_2x_{i2}+\dots+\beta_{n-1}x_{in-1}+\epsilon_1\\ +\dots & \dots \\ +y_{n-1}&=\beta_0x_{n-1,0}+\beta_1x_{n-1,2}+\beta_2x_{n-1,2}+\dots+\beta_{n-1}x_{n-1,n-1}+\epsilon_{n-1}.\\ +\end{align*} + +As we noted above, we stayed with a system with the design matrix + $\bm{X}\in {\mathbb{R}}^{n\times n}$, that is we have $p=n$. For reasons to come later (algorithmic arguments) we will hereafter define +our matrix as $\bm{X}\in {\mathbb{R}}^{n\times p}$, with the predictors refering to the column numbers and the entries $n$ being the row elements. +% --- end paragraph admon --- + + + + +% !split +\subsection{Our model for the nuclear binding energies} + +In our \href{{https://compphysics.github.io/MachineLearningMSU/doc/pub/Introduction/html/Introduction.html}}{introductory notes} we looked at the so-called \href{{https://en.wikipedia.org/wiki/Semi-empirical_mass_formula}}{liguid drop model}. Let us remind ourselves about what we did by looking at the code. + +We restate the parts of the code we are most interested in. +\begin{minted}[fontsize=\fontsize{9pt}{9pt},linenos=false,mathescape,baselinestretch=1.0,fontfamily=tt,xleftmargin=7mm]{python} +# Common imports +import numpy as np +import pandas as pd +import matplotlib.pyplot as plt +from IPython.display import display +import os + +# Where to save the figures and data files +PROJECT_ROOT_DIR = "Results" +FIGURE_ID = "Results/FigureFiles" +DATA_ID = "DataFiles/" + +if not os.path.exists(PROJECT_ROOT_DIR): + os.mkdir(PROJECT_ROOT_DIR) + +if not os.path.exists(FIGURE_ID): + os.makedirs(FIGURE_ID) + +if not os.path.exists(DATA_ID): + os.makedirs(DATA_ID) + +def image_path(fig_id): + return os.path.join(FIGURE_ID, fig_id) + +def data_path(dat_id): + return os.path.join(DATA_ID, dat_id) + +def save_fig(fig_id): + plt.savefig(image_path(fig_id) + ".png", format='png') + +infile = open(data_path("MassEval2016.dat"),'r') + + +# Read the experimental data with Pandas +Masses = pd.read_fwf(infile, usecols=(2,3,4,6,11), + names=('N', 'Z', 'A', 'Element', 'Ebinding'), + widths=(1,3,5,5,5,1,3,4,1,13,11,11,9,1,2,11,9,1,3,1,12,11,1), + header=39, + index_col=False) + +# Extrapolated values are indicated by '#' in place of the decimal place, so +# the Ebinding column won't be numeric. Coerce to float and drop these entries. +Masses['Ebinding'] = pd.to_numeric(Masses['Ebinding'], errors='coerce') +Masses = Masses.dropna() +# Convert from keV to MeV. +Masses['Ebinding'] /= 1000 + +# Group the DataFrame by nucleon number, A. +Masses = Masses.groupby('A') +# Find the rows of the grouped DataFrame with the maximum binding energy. +Masses = Masses.apply(lambda t: t[t.Ebinding==t.Ebinding.max()]) +A = Masses['A'] +Z = Masses['Z'] +N = Masses['N'] +Element = Masses['Element'] +Energies = Masses['Ebinding'] + +# Now we set up the design matrix X +X = np.zeros((len(A),5)) +X[:,0] = 1 +X[:,1] = A +X[:,2] = A**(2.0/3.0) +X[:,3] = A**(-1.0/3.0) +X[:,4] = A**(-1.0) +# Then nice printout using pandas +DesignMatrix = pd.DataFrame(X) +DesignMatrix.index = A +DesignMatrix.columns = ['1', 'A', 'A^(2/3)', 'A^(-1/3)', '1/A'] +display(DesignMatrix) +\end{minted} + +With $\bm{\beta}\in {\mathbb{R}}^{p\times 1}$, it means that we will hereafter write our equations for the approximation as +\[ +\bm{\tilde{y}}= \bm{X}\bm{\beta}, +\] +throughout these lectures. + + +% !split +\subsection{Optimizing our parameters, more details} + +% --- begin paragraph admon --- +\paragraph{} +With the above we use the design matrix to define the approximation $\bm{\tilde{y}}$ via the unknown quantity $\bm{\beta}$ as +\[ +\bm{\tilde{y}}= \bm{X}\bm{\beta}, +\] +and in order to find the optimal parameters $\beta_i$ instead of solving the above linear algebra problem, we define a function which gives a measure of the spread between the values $y_i$ (which represent hopefully the exact values) and the parameterized values $\tilde{y}_i$, namely +\[ +C(\bm{\beta})=\frac{1}{n}\sum_{i=0}^{n-1}\left(y_i-\tilde{y}_i\right)^2=\frac{1}{n}\left\{\left(\bm{y}-\bm{\tilde{y}}\right)^T\left(\bm{y}-\bm{\tilde{y}}\right)\right\}, +\] +or using the matrix $\bm{X}$ and in a more compact matrix-vector notation as +\[ +C(\bm{\beta})=\frac{1}{n}\left\{\left(\bm{y}-\bm{X}^T\bm{\beta}\right)^T\left(\bm{y}-\bm{X}^T\bm{\beta}\right)\right\}. +\] +This function is one possible way to define the so-called cost function. + + + +It is also common to define +the function $Q$ as + +\[ +C(\bm{\beta})=\frac{1}{2n}\sum_{i=0}^{n-1}\left(y_i-\tilde{y}_i\right)^2, +\] +since when taking the first derivative with respect to the unknown parameters $\beta$, the factor of $2$ cancels out. +% --- end paragraph admon --- + + + + +% !split +\subsection{Interpretations and optimizing our parameters} + +% --- begin paragraph admon --- +\paragraph{} + +The function +\[ +C(\bm{\beta})=\frac{1}{n}\left\{\left(\bm{y}-\bm{X}\bm{\beta}\right)^T\left(\bm{y}-\bm{X}\bm{\beta}\right)\right\}, +\] +can be linked to the variance of the quantity $y_i$ if we interpret the latter as the mean value. +When linking below with the maximum likelihood approach below, we will indeed interpret $y_i$ as a mean value (see exercises) +\[ +y_{i}=\langle y_i \rangle = \beta_0x_{i,0}+\beta_1x_{i,1}+\beta_2x_{i,2}+\dots+\beta_{n-1}x_{i,n-1}+\epsilon_i, +\] + +where $\langle y_i \rangle$ is the mean value. Keep in mind also that +till now we have treated $y_i$ as the exact value. Normally, the +response (dependent or outcome) variable $y_i$ the outcome of a +numerical experiment or another type of experiment and is thus only an +approximation to the true value. It is then always accompanied by an +error estimate, often limited to a statistical error estimate given by +the standard deviation discussed earlier. In the discussion here we +will treat $y_i$ as our exact value for the response variable. + +In order to find the parameters $\beta_i$ we will then minimize the spread of $C(\bm{\beta})$, that is we are going to solve the problem +\[ +{\displaystyle \min_{\bm{\beta}\in +{\mathbb{R}}^{p}}}\frac{1}{n}\left\{\left(\bm{y}-\bm{X}\bm{\beta}\right)^T\left(\bm{y}-\bm{X}\bm{\beta}\right)\right\}. +\] +In practical terms it means we will require +\[ +\frac{\partial C(\bm{\beta})}{\partial \beta_j} = \frac{\partial }{\partial \beta_j}\left[ \frac{1}{n}\sum_{i=0}^{n-1}\left(y_i-\beta_0x_{i,0}-\beta_1x_{i,1}-\beta_2x_{i,2}-\dots-\beta_{n-1}x_{i,n-1}\right)^2\right]=0, +\] +which results in +\[ +\frac{\partial C(\bm{\beta})}{\partial \beta_j} = -\frac{2}{n}\left[ \sum_{i=0}^{n-1}x_{ij}\left(y_i-\beta_0x_{i,0}-\beta_1x_{i,1}-\beta_2x_{i,2}-\dots-\beta_{n-1}x_{i,n-1}\right)\right]=0, +\] +or in a matrix-vector form as +\[ +\frac{\partial C(\bm{\beta})}{\partial \bm{\beta}} = 0 = \bm{X}^T\left( \bm{y}-\bm{X}\bm{\beta}\right). +\] +% --- end paragraph admon --- + + + + +% !split +\subsection{Interpretations and optimizing our parameters} + +% --- begin paragraph admon --- +\paragraph{} +We can rewrite +\[ +\frac{\partial C(\bm{\beta})}{\partial \bm{\beta}} = 0 = \bm{X}^T\left( \bm{y}-\bm{X}\bm{\beta}\right), +\] +as +\[ +\bm{X}^T\bm{y} = \bm{X}^T\bm{X}\bm{\beta}, +\] +and if the matrix $\bm{X}^T\bm{X}$ is invertible we have the solution +\[ +\bm{\beta} =\left(\bm{X}^T\bm{X}\right)^{-1}\bm{X}^T\bm{y}. +\] + +We note also that since our design matrix is defined as $\bm{X}\in +{\mathbb{R}}^{n\times p}$, the product $\bm{X}^T\bm{X} \in +{\mathbb{R}}^{p\times p}$. In the above case we have that $p \ll n$, +in our case $p=5$ meaning that we end up with inverting a small +$5\times 5$ matrix. This is a rather common situation, in many cases we end up with low-dimensional +matrices to invert. The methods discussed here and for many other +supervised learning algorithms like classification with logistic +regression or support vector machines, exhibit dimensionalities which +allow for the usage of direct linear algebra methods such as \textbf{LU} decomposition or \textbf{Singular Value Decomposition} (SVD) for finding the inverse of the matrix +$\bm{X}^T\bm{X}$. +% --- end paragraph admon --- + + + +% !split +\subsection{Interpretations and optimizing our parameters} + +% --- begin paragraph admon --- +\paragraph{} +The residuals $\bm{\epsilon}$ are in turn given by +\[ +\bm{\epsilon} = \bm{y}-\bm{\tilde{y}} = \bm{y}-\bm{X}\bm{\beta}, +\] +and with +\[ +\bm{X}^T\left( \bm{y}-\bm{X}\bm{\beta}\right)= 0, +\] +we have +\[ +\bm{X}^T\bm{\epsilon}=\bm{X}^T\left( \bm{y}-\bm{X}\bm{\beta}\right)= 0, +\] +meaning that the solution for $\bm{\beta}$ is the one which minimizes the residuals. Later we will link this with the maximum likelihood approach. +% --- end paragraph admon --- + + + + +Let us now return to our nuclear binding energies and simply code the above equations. + +% !split +\subsection{Own code for Ordinary Least Squares} + +It is rather straightforward to implement the matrix inversion and obtain the parameters $\bm{\beta}$. After having defined the matrix $\bm{X}$ we simply need to +write +\begin{minted}[fontsize=\fontsize{9pt}{9pt},linenos=false,mathescape,baselinestretch=1.0,fontfamily=tt,xleftmargin=7mm]{python} +# matrix inversion to find beta +beta = np.linalg.inv(X.T.dot(X)).dot(X.T).dot(Energies) +# and then make the prediction +ytilde = X @ beta +\end{minted} +Alternatively, you can use the least squares functionality in \textbf{Numpy} as +\begin{minted}[fontsize=\fontsize{9pt}{9pt},linenos=false,mathescape,baselinestretch=1.0,fontfamily=tt,xleftmargin=7mm]{python} +fit = np.linalg.lstsq(X, Energies, rcond =None)[0] +ytildenp = np.dot(fit,X.T) +\end{minted} + +And finally we plot our fit with and compare with data +\begin{minted}[fontsize=\fontsize{9pt}{9pt},linenos=false,mathescape,baselinestretch=1.0,fontfamily=tt,xleftmargin=7mm]{python} +Masses['Eapprox'] = ytilde +# Generate a plot comparing the experimental with the fitted values values. +fig, ax = plt.subplots() +ax.set_xlabel(r'$A = N + Z$') +ax.set_ylabel(r'$E_\mathrm{bind}\,/\mathrm{MeV}$') +ax.plot(Masses['A'], Masses['Ebinding'], alpha=0.7, lw=2, + label='Ame2016') +ax.plot(Masses['A'], Masses['Eapprox'], alpha=0.7, lw=2, c='m', + label='Fit') +ax.legend() +save_fig("Masses2016OLS") +plt.show() +\end{minted} + +% !split +\subsection{Adding error analysis and training set up} + +We can easily test our fit by computing the $R2$ score that we discussed in connection with the functionality of _Scikit_Learn_ in the introductory slides. +Since we are not using _Scikit-Learn here we can define our own $R2$ function as +\begin{minted}[fontsize=\fontsize{9pt}{9pt},linenos=false,mathescape,baselinestretch=1.0,fontfamily=tt,xleftmargin=7mm]{python} +def R2(y_data, y_model): + return 1 - np.sum((y_data - y_model) ** 2) / np.sum((y_data - np.mean(y_model)) ** 2) +\end{minted} +and we would be using it as +\begin{minted}[fontsize=\fontsize{9pt}{9pt},linenos=false,mathescape,baselinestretch=1.0,fontfamily=tt,xleftmargin=7mm]{python} +print(R2(Energies,ytilde)) +\end{minted} + +We can easily add our \textbf{MSE} score as +\begin{minted}[fontsize=\fontsize{9pt}{9pt},linenos=false,mathescape,baselinestretch=1.0,fontfamily=tt,xleftmargin=7mm]{python} +def MSE(y_data,y_model): + n = np.size(y_model) + return np.sum((y_data-y_model)**2)/n + +print(MSE(Energies,ytilde)) +\end{minted} +and finally the relative error as +\begin{minted}[fontsize=\fontsize{9pt}{9pt},linenos=false,mathescape,baselinestretch=1.0,fontfamily=tt,xleftmargin=7mm]{python} +def RelativeError(y_data,y_model): + return abs((y_data-y_model)/y_data) +print(RelativeError(Energies, ytilde)) +\end{minted} + + + + + +% !split +\subsection{The $\chi^2$ function} + +% --- begin paragraph admon --- +\paragraph{} + +Normally, the response (dependent or outcome) variable $y_i$ is the +outcome of a numerical experiment or another type of experiment and is +thus only an approximation to the true value. It is then always +accompanied by an error estimate, often limited to a statistical error +estimate given by the standard deviation discussed earlier. In the +discussion here we will treat $y_i$ as our exact value for the +response variable. + +Introducing the standard deviation $\sigma_i$ for each measurement +$y_i$, we define now the $\chi^2$ function (omitting the $1/n$ term) +as + +\[ +\chi^2(\bm{\beta})=\frac{1}{n}\sum_{i=0}^{n-1}\frac{\left(y_i-\tilde{y}_i\right)^2}{\sigma_i^2}=\frac{1}{n}\left\{\left(\bm{y}-\bm{\tilde{y}}\right)^T\frac{1}{\bm{\Sigma^2}}\left(\bm{y}-\bm{\tilde{y}}\right)\right\}, +\] +where the matrix $\bm{\Sigma}$ is a diagonal matrix with $\sigma_i$ as matrix elements. +% --- end paragraph admon --- + + + +% !split +\subsection{The $\chi^2$ function} + +% --- begin paragraph admon --- +\paragraph{} + +In order to find the parameters $\beta_i$ we will then minimize the spread of $\chi^2(\bm{\beta})$ by requiring +\[ +\frac{\partial \chi^2(\bm{\beta})}{\partial \beta_j} = \frac{\partial }{\partial \beta_j}\left[ \frac{1}{n}\sum_{i=0}^{n-1}\left(\frac{y_i-\beta_0x_{i,0}-\beta_1x_{i,1}-\beta_2x_{i,2}-\dots-\beta_{n-1}x_{i,n-1}}{\sigma_i}\right)^2\right]=0, +\] +which results in +\[ +\frac{\partial \chi^2(\bm{\beta})}{\partial \beta_j} = -\frac{2}{n}\left[ \sum_{i=0}^{n-1}\frac{x_{ij}}{\sigma_i}\left(\frac{y_i-\beta_0x_{i,0}-\beta_1x_{i,1}-\beta_2x_{i,2}-\dots-\beta_{n-1}x_{i,n-1}}{\sigma_i}\right)\right]=0, +\] +or in a matrix-vector form as +\[ +\frac{\partial \chi^2(\bm{\beta})}{\partial \bm{\beta}} = 0 = \bm{A}^T\left( \bm{b}-\bm{A}\bm{\beta}\right). +\] +where we have defined the matrix $\bm{A} =\bm{X}/\bm{\Sigma}$ with matrix elements $a_{ij} = x_{ij}/\sigma_i$ and the vector $\bm{b}$ with elements $b_i = y_i/\sigma_i$. +% --- end paragraph admon --- + + + +% !split +\subsection{The $\chi^2$ function} + +% --- begin paragraph admon --- +\paragraph{} + +We can rewrite +\[ +\frac{\partial \chi^2(\bm{\beta})}{\partial \bm{\beta}} = 0 = \bm{A}^T\left( \bm{b}-\bm{A}\bm{\beta}\right), +\] +as +\[ +\bm{A}^T\bm{b} = \bm{A}^T\bm{A}\bm{\beta}, +\] +and if the matrix $\bm{A}^T\bm{A}$ is invertible we have the solution +\[ +\bm{\beta} =\left(\bm{A}^T\bm{A}\right)^{-1}\bm{A}^T\bm{b}. +\] +% --- end paragraph admon --- + + + +% !split +\subsection{The $\chi^2$ function} + +% --- begin paragraph admon --- +\paragraph{} + +If we then introduce the matrix +\[ +\bm{H} = \left(\bm{A}^T\bm{A}\right)^{-1}, +\] +we have then the following expression for the parameters $\beta_j$ (the matrix elements of $\bm{H}$ are $h_{ij}$) +\[ +\beta_j = \sum_{k=0}^{p-1}h_{jk}\sum_{i=0}^{n-1}\frac{y_i}{\sigma_i}\frac{x_{ik}}{\sigma_i} = \sum_{k=0}^{p-1}h_{jk}\sum_{i=0}^{n-1}b_ia_{ik} +\] +We state without proof the expression for the uncertainty in the parameters $\beta_j$ as (we leave this as an exercise) +\[ +\sigma^2(\beta_j) = \sum_{i=0}^{n-1}\sigma_i^2\left( \frac{\partial \beta_j}{\partial y_i}\right)^2, +\] +resulting in +\[ +\sigma^2(\beta_j) = \left(\sum_{k=0}^{p-1}h_{jk}\sum_{i=0}^{n-1}a_{ik}\right)\left(\sum_{l=0}^{p-1}h_{jl}\sum_{m=0}^{n-1}a_{ml}\right) = h_{jj}! +\] +% --- end paragraph admon --- + + + +% !split +\subsection{The $\chi^2$ function} + +% --- begin paragraph admon --- +\paragraph{} +The first step here is to approximate the function $y$ with a first-order polynomial, that is we write +\[ +y=y(x) \rightarrow y(x_i) \approx \beta_0+\beta_1 x_i. +\] +By computing the derivatives of $\chi^2$ with respect to $\beta_0$ and $\beta_1$ show that these are given by +\[ +\frac{\partial \chi^2(\bm{\beta})}{\partial \beta_0} = -2\left[ \frac{1}{n}\sum_{i=0}^{n-1}\left(\frac{y_i-\beta_0-\beta_1x_{i}}{\sigma_i^2}\right)\right]=0, +\] +and +\[ +\frac{\partial \chi^2(\bm{\beta})}{\partial \beta_1} = -\frac{2}{n}\left[ \sum_{i=0}^{n-1}x_i\left(\frac{y_i-\beta_0-\beta_1x_{i}}{\sigma_i^2}\right)\right]=0. +\] +% --- end paragraph admon --- + + + +% !split +\subsection{The $\chi^2$ function} + +% --- begin paragraph admon --- +\paragraph{} + +For a linear fit (a first-order polynomial) we don't need to invert a matrix!! +Defining +\[ +\gamma = \sum_{i=0}^{n-1}\frac{1}{\sigma_i^2}, +\] + +\[ +\gamma_x = \sum_{i=0}^{n-1}\frac{x_{i}}{\sigma_i^2}, +\] + +\[ +\gamma_y = \sum_{i=0}^{n-1}\left(\frac{y_i}{\sigma_i^2}\right), +\] + +\[ +\gamma_{xx} = \sum_{i=0}^{n-1}\frac{x_ix_{i}}{\sigma_i^2}, +\] + +\[ +\gamma_{xy} = \sum_{i=0}^{n-1}\frac{y_ix_{i}}{\sigma_i^2}, +\] + +we obtain + +\[ +\beta_0 = \frac{\gamma_{xx}\gamma_y-\gamma_x\gamma_y}{\gamma\gamma_{xx}-\gamma_x^2}, +\] + +\[ +\beta_1 = \frac{\gamma_{xy}\gamma-\gamma_x\gamma_y}{\gamma\gamma_{xx}-\gamma_x^2}. +\] + +This approach (different linear and non-linear regression) suffers +often from both being underdetermined and overdetermined in the +unknown coefficients $\beta_i$. A better approach is to use the +Singular Value Decomposition (SVD) method discussed below. Or using +Lasso and Ridge regression. See below. +% --- end paragraph admon --- + + + + +% !split +\subsection{Fitting an Equation of State for Dense Nuclear Matter} + +Before we continue, let us introduce yet another example. We are going to fit the +nuclear equation of state using results from many-body calculations. +The equation of state we have made available here, as function of +density, has been derived using modern nucleon-nucleon potentials with +\href{{https://www.sciencedirect.com/science/article/pii/S0370157399001106}}{the addition of three-body +forces}. This +time the file is presented as a standard \textbf{csv} file. + +The beginning of the Python code here is similar to what you have seen before, +with the same initializations and declarations. We use also \textbf{pandas} +again, rather extensively in order to organize our data. + +The difference now is that we use \textbf{Scikit-Learn's} regression tools +instead of our own matrix inversion implementation. Furthermore, we +sneak in \textbf{Ridge} regression (to be discussed below) which includes a +hyperparameter $\lambda$, also to be explained below. + +% !split +\subsection{The code} + +\begin{minted}[fontsize=\fontsize{9pt}{9pt},linenos=false,mathescape,baselinestretch=1.0,fontfamily=tt,xleftmargin=7mm]{python} +# Common imports +import os +import numpy as np +import pandas as pd +import matplotlib.pyplot as plt +import matplotlib.pyplot as plt +import sklearn.linear_model as skl +from sklearn.metrics import mean_squared_error, r2_score, mean_absolute_error + +# Where to save the figures and data files +PROJECT_ROOT_DIR = "Results" +FIGURE_ID = "Results/FigureFiles" +DATA_ID = "DataFiles/" + +if not os.path.exists(PROJECT_ROOT_DIR): + os.mkdir(PROJECT_ROOT_DIR) + +if not os.path.exists(FIGURE_ID): + os.makedirs(FIGURE_ID) + +if not os.path.exists(DATA_ID): + os.makedirs(DATA_ID) + +def image_path(fig_id): + return os.path.join(FIGURE_ID, fig_id) + +def data_path(dat_id): + return os.path.join(DATA_ID, dat_id) + +def save_fig(fig_id): + plt.savefig(image_path(fig_id) + ".png", format='png') + +infile = open(data_path("EoS.csv"),'r') + +# Read the EoS data as csv file and organize the data into two arrays with density and energies +EoS = pd.read_csv(infile, names=('Density', 'Energy')) +EoS['Energy'] = pd.to_numeric(EoS['Energy'], errors='coerce') +EoS = EoS.dropna() +Energies = EoS['Energy'] +Density = EoS['Density'] +# The design matrix now as function of various polytrops +X = np.zeros((len(Density),4)) +X[:,3] = Density**(4.0/3.0) +X[:,2] = Density +X[:,1] = Density**(2.0/3.0) +X[:,0] = 1 + +# We use now Scikit-Learn's linear regressor and ridge regressor +# OLS part +clf = skl.LinearRegression().fit(X, Energies) +ytilde = clf.predict(X) +EoS['Eols'] = ytilde +# The mean squared error +print("Mean squared error: %.2f" % mean_squared_error(Energies, ytilde)) +# Explained variance score: 1 is perfect prediction +print('Variance score: %.2f' % r2_score(Energies, ytilde)) +# Mean absolute error +print('Mean absolute error: %.2f' % mean_absolute_error(Energies, ytilde)) +print(clf.coef_, clf.intercept_) + +# The Ridge regression with a hyperparameter lambda = 0.1 +_lambda = 0.1 +clf_ridge = skl.Ridge(alpha=_lambda).fit(X, Energies) +yridge = clf_ridge.predict(X) +EoS['Eridge'] = yridge +# The mean squared error +print("Mean squared error: %.2f" % mean_squared_error(Energies, yridge)) +# Explained variance score: 1 is perfect prediction +print('Variance score: %.2f' % r2_score(Energies, yridge)) +# Mean absolute error +print('Mean absolute error: %.2f' % mean_absolute_error(Energies, yridge)) +print(clf_ridge.coef_, clf_ridge.intercept_) + +fig, ax = plt.subplots() +ax.set_xlabel(r'$\rho[\mathrm{fm}^{-3}]$') +ax.set_ylabel(r'Energy per particle') +ax.plot(EoS['Density'], EoS['Energy'], alpha=0.7, lw=2, + label='Theoretical data') +ax.plot(EoS['Density'], EoS['Eols'], alpha=0.7, lw=2, c='m', + label='OLS') +ax.plot(EoS['Density'], EoS['Eridge'], alpha=0.7, lw=2, c='g', + label='Ridge $\lambda = 0.1$') +ax.legend() +save_fig("EoSfitting") +plt.show() +\end{minted} + +The above simple polynomial in density $\rho$ gives an excellent fit +to the data. Can you give an interpretation of the various powers of $\rho$? + +We note also that there is a small deviation between the +standard OLS and the Ridge regression at higher densities. We discuss this in more detail +below. + + +% !split +\subsection{Splitting our Data in Training and Test data} + +It is normal in essentially all Machine Learning studies to split the +data in a training set and a test set (sometimes also an additional +validation set). \textbf{Scikit-Learn} has an own function for this. There +is no explicit recipe for how much data should be included as training +data and say test data. An accepted rule of thumb is to use +approximately $2/3$ to $4/5$ of the data as training data. We will +postpone a discussion of this splitting to the end of these notes and +our discussion of the so-called \textbf{bias-variance} tradeoff. Here we +limit ourselves to repeat the above equation of state fitting example +but now splitting the data into a training set and a test set. + +\begin{minted}[fontsize=\fontsize{9pt}{9pt},linenos=false,mathescape,baselinestretch=1.0,fontfamily=tt,xleftmargin=7mm]{python} +import os +import numpy as np +import pandas as pd +import matplotlib.pyplot as plt +from sklearn.model_selection import train_test_split +# Where to save the figures and data files +PROJECT_ROOT_DIR = "Results" +FIGURE_ID = "Results/FigureFiles" +DATA_ID = "DataFiles/" + +if not os.path.exists(PROJECT_ROOT_DIR): + os.mkdir(PROJECT_ROOT_DIR) + +if not os.path.exists(FIGURE_ID): + os.makedirs(FIGURE_ID) + +if not os.path.exists(DATA_ID): + os.makedirs(DATA_ID) + +def image_path(fig_id): + return os.path.join(FIGURE_ID, fig_id) + +def data_path(dat_id): + return os.path.join(DATA_ID, dat_id) + +def save_fig(fig_id): + plt.savefig(image_path(fig_id) + ".png", format='png') + +def R2(y_data, y_model): + return 1 - np.sum((y_data - y_model) ** 2) / np.sum((y_data - np.mean(y_model)) ** 2) +def MSE(y_data,y_model): + n = np.size(y_model) + return np.sum((y_data-y_model)**2)/n + +infile = open(data_path("EoS.csv"),'r') + +# Read the EoS data as csv file and organized into two arrays with density and energies +EoS = pd.read_csv(infile, names=('Density', 'Energy')) +EoS['Energy'] = pd.to_numeric(EoS['Energy'], errors='coerce') +EoS = EoS.dropna() +Energies = EoS['Energy'] +Density = EoS['Density'] +# The design matrix now as function of various polytrops +X = np.zeros((len(Density),5)) +X[:,0] = 1 +X[:,1] = Density**(2.0/3.0) +X[:,2] = Density +X[:,3] = Density**(4.0/3.0) +X[:,4] = Density**(5.0/3.0) +# We split the data in test and training data +X_train, X_test, y_train, y_test = train_test_split(X, Energies, test_size=0.2) +# matrix inversion to find beta +beta = np.linalg.inv(X_train.T.dot(X_train)).dot(X_train.T).dot(y_train) +# and then make the prediction +ytilde = X_train @ beta +print("Training R2") +print(R2(y_train,ytilde)) +print("Training MSE") +print(MSE(y_train,ytilde)) +ypredict = X_test @ beta +print("Test R2") +print(R2(y_test,ypredict)) +print("Test MSE") +print(MSE(y_test,ypredict)) +\end{minted} + + +% !split +\subsection{The singular value decomposition} + + +% --- begin paragraph admon --- +\paragraph{} + +The examples we have looked at so far are cases where we normally can +invert the matrix $\bm{X}^T\bm{X}$. Using a polynomial expansion as we +did both for the masses and the fitting of the equation of state, +leads to row vectors of the design matrix which are essentially +orthogonal due to the polynomial character of our model. This may +however not the be case in general and a standard matrix inversion +algorithm based on say LU decomposition may lead to singularities. We will see an example of this below when we try to fit +the coupling constant of the widely used Ising model. +There is however a way to partially circumvent this problem and also gain some insight about the ordinary least squares approach. + +This is given by the \textbf{Singular Value Decomposition} algorithm, perhaps +the most powerful linear algebra algorithm. Let us look at a +different example where we may have problems with the standard matrix +inversion algorithm. Thereafter we dive into the math of the SVD. +% --- end paragraph admon --- + + + +% !split +\subsection{The Ising model} + +The one-dimensional Ising model with nearest neighbor interaction, no +external field and a constant coupling constant $J$ is given by + +\begin{align} + H = -J \sum_{k}^L s_k s_{k + 1}, +\end{align} + +where $s_i \in \{-1, 1\}$ and $s_{N + 1} = s_1$. The number of spins +in the system is determined by $L$. For the one-dimensional system +there is no phase transition. + +We will look at a system of $L = 40$ spins with a coupling constant of +$J = 1$. To get enough training data we will generate 10000 states +with their respective energies. + + +\begin{minted}[fontsize=\fontsize{9pt}{9pt},linenos=false,mathescape,baselinestretch=1.0,fontfamily=tt,xleftmargin=7mm]{python} +import numpy as np +import matplotlib.pyplot as plt +from mpl_toolkits.axes_grid1 import make_axes_locatable +import seaborn as sns +import scipy.linalg as scl +from sklearn.model_selection import train_test_split +import tqdm +sns.set(color_codes=True) +cmap_args=dict(vmin=-1., vmax=1., cmap='seismic') + +L = 40 +n = int(1e4) + +spins = np.random.choice([-1, 1], size=(n, L)) +J = 1.0 + +energies = np.zeros(n) + +for i in range(n): + energies[i] = - J * np.dot(spins[i], np.roll(spins[i], 1)) +\end{minted} + +Here we use ordinary least squares +regression to predict the energy for the nearest neighbor +one-dimensional Ising model on a ring, i.e., the endpoints wrap +around. We will use linear regression to fit a value for +the coupling constant to achieve this. + +% !split +\subsection{Reformulating the problem to suit regression} + +A more general form for the one-dimensional Ising model is + +\begin{align} + H = - \sum_j^L \sum_k^L s_j s_k J_{jk}. +\end{align} + +Here we allow for interactions beyond the nearest neighbors and a state dependent +coupling constant. This latter expression can be formulated as +a matrix-product +\begin{align} + \bm{H} = \bm{X} J, +\end{align} + +where $X_{jk} = s_j s_k$ and $J$ is a matrix which consists of the +elements $-J_{jk}$. This form of writing the energy fits perfectly +with the form utilized in linear regression, that is + +\begin{align} + \bm{y} = \bm{X}\bm{\beta} + \bm{\epsilon}, +\end{align} + +We split the data in training and test data as discussed in the previous example + +\begin{minted}[fontsize=\fontsize{9pt}{9pt},linenos=false,mathescape,baselinestretch=1.0,fontfamily=tt,xleftmargin=7mm]{python} +X = np.zeros((n, L ** 2)) +for i in range(n): + X[i] = np.outer(spins[i], spins[i]).ravel() +y = energies +X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2) +\end{minted} + +% !split +\subsection{Linear regression} + +In the ordinary least squares method we choose the cost function + +\begin{align} + C(\bm{X}, \bm{\beta})= \frac{1}{n}\left\{(\bm{X}\bm{\beta} - \bm{y})^T(\bm{X}\bm{\beta} - \bm{y})\right\}. +\end{align} + +We then find the extremal point of $C$ by taking the derivative with respect to $\bm{\beta}$ as discussed above. +This yields the expression for $\bm{\beta}$ to be + +\[ + \bm{\beta} = \frac{\bm{X}^T \bm{y}}{\bm{X}^T \bm{X}}, +\] + +which immediately imposes some requirements on $\bm{X}$ as there must exist +an inverse of $\bm{X}^T \bm{X}$. If the expression we are modeling contains an +intercept, i.e., a constant term, we must make sure that the +first column of $\bm{X}$ consists of $1$. We do this here + +\begin{minted}[fontsize=\fontsize{9pt}{9pt},linenos=false,mathescape,baselinestretch=1.0,fontfamily=tt,xleftmargin=7mm]{python} +X_train_own = np.concatenate( + (np.ones(len(X_train))[:, np.newaxis], X_train), + axis=1 +) +X_test_own = np.concatenate( + (np.ones(len(X_test))[:, np.newaxis], X_test), + axis=1 +) +\end{minted} + +\begin{minted}[fontsize=\fontsize{9pt}{9pt},linenos=false,mathescape,baselinestretch=1.0,fontfamily=tt,xleftmargin=7mm]{python} +def ols_inv(x: np.ndarray, y: np.ndarray) -> np.ndarray: + return scl.inv(x.T @ x) @ (x.T @ y) +beta = ols_inv(X_train_own, y_train) +\end{minted} + + +% !split +\subsection{Singular Value decomposition} + +Doing the inversion directly turns out to be a bad idea since the matrix +$\bm{X}^T\bm{X}$ is singular. An alternative approach is to use the \textbf{singular +value decomposition}. Using the definition of the Moore-Penrose +pseudoinverse we can write the equation for $\bm{\beta}$ as + +\[ + \bm{\beta} = \bm{X}^{+}\bm{y}, +\] + +where the pseudoinverse of $\bm{X}$ is given by + +\[ + \bm{X}^{+} = \frac{\bm{X}^T}{\bm{X}^T\bm{X}}. +\] + +Using singular value decomposition we can decompose the matrix $\bm{X} = \bm{U}\bm{\Sigma} \bm{V}^T$, +where $\bm{U}$ and $\bm{V}$ are orthogonal(unitary) matrices and $\bm{\Sigma}$ contains the singular values (more details below). +where $X^{+} = V\Sigma^{+} U^T$. This reduces the equation for +$\omega$ to +\begin{align} + \bm{\beta} = \bm{V}\bm{\Sigma}^{+} \bm{U}^T \bm{y}. +\end{align} + +Note that solving this equation by actually doing the pseudoinverse +(which is what we will do) is not a good idea as this operation scales +as $\mathcal{O}(n^3)$, where $n$ is the number of elements in a +general matrix. Instead, doing $QR$-factorization and solving the +linear system as an equation would reduce this down to +$\mathcal{O}(n^2)$ operations. + + +\begin{minted}[fontsize=\fontsize{9pt}{9pt},linenos=false,mathescape,baselinestretch=1.0,fontfamily=tt,xleftmargin=7mm]{python} +def ols_svd(x: np.ndarray, y: np.ndarray) -> np.ndarray: + u, s, v = scl.svd(x) + return v.T @ scl.pinv(scl.diagsvd(s, u.shape[0], v.shape[0])) @ u.T @ y +\end{minted} + +\begin{minted}[fontsize=\fontsize{9pt}{9pt},linenos=false,mathescape,baselinestretch=1.0,fontfamily=tt,xleftmargin=7mm]{python} +beta = ols_svd(X_train_own,y_train) +\end{minted} + +When extracting the $J$-matrix we need to make sure that we remove the intercept, as is done here + +\begin{minted}[fontsize=\fontsize{9pt}{9pt},linenos=false,mathescape,baselinestretch=1.0,fontfamily=tt,xleftmargin=7mm]{python} +J = beta[1:].reshape(L, L) +\end{minted} + +A way of looking at the coefficients in $J$ is to plot the matrices as images. + + +\begin{minted}[fontsize=\fontsize{9pt}{9pt},linenos=false,mathescape,baselinestretch=1.0,fontfamily=tt,xleftmargin=7mm]{python} +fig = plt.figure(figsize=(20, 14)) +im = plt.imshow(J, **cmap_args) +plt.title("OLS", fontsize=18) +plt.xticks(fontsize=18) +plt.yticks(fontsize=18) +cb = fig.colorbar(im) +cb.ax.set_yticklabels(cb.ax.get_yticklabels(), fontsize=18) +plt.show() +\end{minted} +It is interesting to note that OLS +considers both $J_{j, j + 1} = -0.5$ and $J_{j, j - 1} = -0.5$ as +valid matrix elements for $J$. +In our discussion below on hyperparameters and Ridge and Lasso regression we will see that +this problem can be removed, partly and only with Lasso regression. + +In this case our matrix inversion was actually possible. The obvious question now is what is the mathematics behind the SVD? + + +% !split +\subsection{Linear Regression Problems} + +One of the typical problems we encounter with linear regression, in particular +when the matrix $\bm{X}$ (our so-called design matrix) is high-dimensional, +are problems with near singular or singular matrices. The column vectors of $\bm{X}$ +may be linearly dependent, normally referred to as super-collinearity. +This means that the matrix may be rank deficient and it is basically impossible to +to model the data using linear regression. As an example, consider the matrix +\begin{align*} +\mathbf{X} & = \left[ +\begin{array}{rrr} +1 & -1 & 2 +\\ +1 & 0 & 1 +\\ +1 & 2 & -1 +\\ +1 & 1 & 0 +\end{array} \right] +\end{align*} + +The columns of $\bm{X}$ are linearly dependent. We see this easily since the +the first column is the row-wise sum of the other two columns. The rank (more correct, +the column rank) of a matrix is the dimension of the space spanned by the +column vectors. Hence, the rank of $\mathbf{X}$ is equal to the number +of linearly independent columns. In this particular case the matrix has rank 2. + +Super-collinearity of an $(n \times p)$-dimensional design matrix $\mathbf{X}$ implies +that the inverse of the matrix $\bm{X}^T\bm{x}$ (the matrix we need to invert to solve the linear regression equations) is non-invertible. If we have a square matrix that does not have an inverse, we say this matrix singular. The example here demonstrates this +\begin{align*} +\bm{X} & = \left[ +\begin{array}{rr} +1 & -1 +\\ +1 & -1 +\end{array} \right]. +\end{align*} +We see easily that $\mbox{det}(\bm{X}) = x_{11} x_{22} - x_{12} x_{21} = 1 \times (-1) - 1 \times (-1) = 0$. Hence, $\mathbf{X}$ is singular and its inverse is undefined. +This is equivalent to saying that the matrix $\bm{X}$ has at least an eigenvalue which is zero. + + +% !split +\subsection{Fixing the singularity} + +If our design matrix $\bm{X}$ which enters the linear regression problem +\begin{align} +\bm{\beta} & = (\bm{X}^{T} \bm{X})^{-1} \bm{X}^{T} \bm{y}, +\end{align} +has linearly dependent column vectors, we will not be able to compute the inverse +of $\bm{X}^T\bm{X}$ and we cannot find the parameters (estimators) $\beta_i$. +The estimators are only well-defined if $(\bm{X}^{T}\bm{X})^{-1}$ exits. +This is more likely to happen when the matrix $\bm{X}$ is high-dimensional. In this case it is likely to encounter a situation where +the regression parameters $\beta_i$ cannot be estimated. + +A cheap \emph{ad hoc} approach is simply to add a small diagonal component to the matrix to invert, that is we change +\[ +\bm{X}^{T} \bm{X} \rightarrow \bm{X}^{T} \bm{X}+\lambda \bm{I}, +\] +where $\bm{I}$ is the identity matrix. When we discuss \textbf{Ridge} regression this is actually what we end up evaluating. The parameter $\lambda$ is called a hyperparameter. More about this later. + + + +% !split +\subsection{Basic math of the SVD} + + +From standard linear algebra we know that a square matrix $\bm{X}$ can be diagonalized if and only it is +a so-called \href{{https://en.wikipedia.org/wiki/Normal_matrix}}{normal matrix}, that is if $\bm{X}\in {\mathbb{R}}^{n\times n}$ +we have $\bm{X}\bm{X}^T=\bm{X}^T\bm{X}$ or if $\bm{X}\in {\mathbb{C}}^{n\times n}$ we have $\bm{X}\bm{X}^{\dagger}=\bm{X}^{\dagger}\bm{X}$. +The matrix has then a set of eigenpairs + +\[ +(\lambda_1,\bm{u}_1),\dots, (\lambda_n,\bm{u}_n), +and the eigenvalues are given by the diagonal matrix +\[ +\bm{\Sigma}=\mathrm{Diag}(\lambda_1, \dots,\lambda_n). +\] +The matrix $\bm{X}$ can be written in terms of an orthogonal/unitary transformation $\bm{U}$ +\[ +\bm{X} = \bm{U}\bm{\Sigma}\bm{V}^T, +\] +with $\bm{U}\bm{U}^T=\bm{I}$ or $\bm{U}\bm{U}^{\dagger}=\bm{I}$. + +Not all square matrices are diagonalizable. A matrix like the one discussed above +\[ +\bm{X} = \begin{bmatrix} +1& -1 \\ +1& -1\\ +\end{bmatrix} +\] +is not diagonalizable, it is a so-called \href{{https://en.wikipedia.org/wiki/Defective_matrix}}{defective matrix}. It is easy to see that the condition +$\bm{X}\bm{X}^T=\bm{X}^T\bm{X}$ is not fulfilled. + + +% !split +\subsection{The SVD, a Fantastic Algorithm} + + +However, and this is the strength of the SVD algorithm, any general +matrix $\bm{X}$ can be decomposed in terms of a diagonal matrix and +two orthogonal/unitary matrices. The \href{{https://en.wikipedia.org/wiki/Singular_value_decomposition}}{Singular Value Decompostion +(SVD) theorem} +states that a general $m\times n$ matrix $\bm{X}$ can be written in +terms of a diagonal matrix $\bm{\Sigma}$ of dimensionality $n\times n$ +and two orthognal matrices $\bm{U}$ and $\bm{V}$, where the first has +dimensionality $m \times m$ and the last dimensionality $n\times n$. +We have then + +\[ +\bm{X} = \bm{U}\bm{\Sigma}\bm{V}^T +\] + +As an example, the above defective matrix can be decomposed as + +\[ +\bm{X} = \frac{1}{\sqrt{2}}\begin{bmatrix} 1& 1 \\ 1& -1\\ \end{bmatrix} \begin{bmatrix} 2& 0 \\ 0& 0\\ \end{bmatrix} \frac{1}{\sqrt{2}}\begin{bmatrix} 1& -1 \\ 1& 1\\ \end{bmatrix}=\bm{U}\bm{\Sigma}\bm{V}^T, +\] + +with eigenvalues $\sigma_1=2$ and $\sigma_2=0$. +The SVD exits always! + + +% !split +\subsection{Another Example} + +Consider the following matrix which can be SVD decomposed as + +\[ +\bm{X} = \frac{1}{15}\begin{bmatrix} 14 & 2\\ 4 & 22\\ 16 & 13\end{matrix}=\frac{1}{3}\begin{bmatrix} 1& 2 & 2 \\ 2& -1 & 1\\ 2 & 1& -2\end{bmatrix} \begin{bmatrix} 2& 0 \\ 0& 1\\ 0 & 0\end{bmatrix}\frac{1}{5}\begin{bmatrix} 3& 4 \\ 4& -3\end{bmatrix}=\bm{U}\bm{\Sigma}\bm{V}^T. +\] + +This is a $3\times 2$ matrix which is decomposed in terms of a +$3\times 3$ matrix $\bm{U}$, and a $2\times 2$ matrix $\bm{V}$. It is easy to see +that $\bm{U}$ and $\bm{V}$ are orthogonal (how?). + +And the SVD +decomposition (singular values) gives eigenvalues +$\sigma_i\geq\sigma_{i+1}$ for all $i$ and for dimensions larger than $i=2$, the +eigenvalues (singular values) are zero. + +In the general case, where our design matrix $\bm{X}$ has dimension +$n\times p$, the matrix is thus decomposed into an $n\times n$ +orthogonal matrix $\bm{U}$, a $p\times p$ orthogonal matrix $\bm{V}$ +and a diagonal matrix $\bm{\Sigma}$ with $r=\mathrm{min}(n,p)$ +singular values $\sigma_i\lg 0$ on the main diagonal and zeros filling +the rest of the matrix. There are at most $p$ singular values +assuming that $n > p$. In our regression examples for the nuclear +masses and the equation of state this is indeed the case, while for +the Ising model we have $p > n$. These are often cases that lead to +near singular or singular matrices. + +The columns of $\bm{U}$ are called the left singular vectors while the columns of $\bm{V}$ are the right singular vectors. + +% !split +\subsection{Economy-size SVD} + +If we assume that $n > p$, then our matrix $\bm{U}$ has dimension $n +\times n$. The last $n-p$ columns of $\bm{U}$ become however +irrelevant in our calculations since they are multiplied with the +zeros in $\bm{\Sigma}$. + +The economy-size decomposition removes extra rows or columns of zeros +from the diagonal matrix of singular values, $\bm{\Sigma}$, along with the columns +in either $\bm{U}$ or $\bm{V}$ that multiply those zeros in the expression. +Removing these zeros and columns can improve execution time +and reduce storage requirements without compromising the accuracy of +the decomposition. + +If $n > p$, we keep only the first $p$ columns of $\bm{U}$ and $\bm{\Sigma}$ has dimension $p\times p$. +If $p > n$, then only the first $n$ columns of $\bm{V}$ are computed and $\bm{\Sigma}$ has dimension $n\times n$. +The $n=p$ case is obvious, we retain the full SVD. +In general the economy-size SVD leads to less FLOPS and still conserving the desired accuracy. + +% !split +\subsection{Mathematical Properties} + +There are several interesting mathematical properties which will be +relevant when we are going to discuss the differences between say +ordinary least squares (OLS) and \textbf{Ridge} regression. + +We have from OLS that the parameters of the linear approximation are given by +\[ +\bm{\tilde{y}} = \bm{X}\bm{\beta} = \bm{X}\left(\bm{X}^T\bm{X}\right)^{-1}\bm{X}^T\bm{y}. +\] + +The matrix to invert can be rewritten in terms of our SVD decomposition as + +\[ +\bm{X}^T\bm{X} = \bm{V}\bm{\Sigma}^T\bm{U}^T\bm{U}\bm{\Sigma}\bm{V}^T. +\] +Using the orthogonality properties of $\bm{U}$ we have + +\[ +\bm{X}^T\bm{X} = \bm{V}\bm{\Sigma}^T\bm{\Sigma}\bm{V}^T = \bm{V}\bm{D}\bm{V}^T, +\] +with $\bm{D}$ being a diagonal matrix with values along the diagonal given by the singular values squared. + +This means that +\[ +(\bm{X}^T\bm{X})\bm{V} = \bm{V}\bm{D}, +\] +that is the eigenvectors of $(\bm{X}^T\bm{X})$ are given by the columns of the right singular matrix of $\bm{X}$ and the eigenvalues are the squared singular values. It is easy to show (show this) that +\[ +(\bm{X}\bm{X}^T)\bm{U} = \bm{U}\bm{D}, +\] +that is, the eigenvectors of $(\bm{X}\bm{X})^T$ are the columns of the left singular matrix and the eigenvalues are the same. + +Going back to our OLS equation we have +\[ +\bm{X}\bm{\beta} = \bm{X}\left(\bm{V}\bm{D}\bm{V}^T \right)^{-1}\bm{X}^T\bm{y}=\bm{U\Sigma V^T}\left(\bm{V}\bm{D}\bm{V}^T \right)^{-1}(\bm{U\Sigma V^T})^T\bm{y}=\bm{U}\bm{U}^T\bm{y}. +\] +We will come back to this expression when we discuss Ridge regression. + + +% !split +\subsection{Ridge and LASSO Regression} + +Let us remind ourselves about the expression for the standard Mean Squared Error (MSE) which we used to define our cost function and the equations for the ordinary least squares (OLS) method, that is +our optimization problem is +\[ +{\displaystyle \min_{\bm{\beta}\in {\mathbb{R}}^{p}}}\frac{1}{n}\left\{\left(\bm{y}-\bm{X}\bm{\beta}\right)^T\left(\bm{y}-\bm{X}\bm{\beta}\right)\right\}. +\] +or we can state it as +\[ +{\displaystyle \min_{\bm{\beta}\in +{\mathbb{R}}^{p}}}\frac{1}{n}\sum_{i=0}^{n-1}\left(y_i-\tilde{y}_i\right)^2=\frac{1}{n}\vert\vert \bm{y}-\bm{X}\bm{\beta}\vert\vert_2^2, +\] +where we have used the definition of a norm-2 vector, that is +\[ +\vert\vert \bm{x}\vert\vert_2 = \sqrt{\sum_i x_i^2}. +\] + +By minimizing the above equation with respect to the parameters +$\bm{\beta}$ we could then obtain an analytical expression for the +parameters $\bm{\beta}$. We can add a regularization parameter $\lambda$ by +defining a new cost function to be optimized, that is + +\[ +{\displaystyle \min_{\bm{\beta}\in +{\mathbb{R}}^{p}}}\frac{1}{n}\vert\vert \bm{y}-\bm{X}\bm{\beta}\vert\vert_2^2+\lambda\vert\vert \bm{\beta}\vert\vert_2^2 +\] + +which leads to the Ridge regression minimization problem where we +require that $\vert\vert \bm{\beta}\vert\vert_2^2\le t$, where $t$ is +a finite number larger than zero. By defining + +\[ +C(\bm{X},\bm{\beta})=\frac{1}{n}\vert\vert \bm{y}-\bm{X}\bm{\beta}\vert\vert_2^2+\lambda\vert\vert \bm{\beta}\vert\vert_1, +\] + +we have a new optimization equation +\[ +{\displaystyle \min_{\bm{\beta}\in +{\mathbb{R}}^{p}}}\frac{1}{n}\vert\vert \bm{y}-\bm{X}\bm{\beta}\vert\vert_2^2+\lambda\vert\vert \bm{\beta}\vert\vert_1 +\] +which leads to Lasso regression. Lasso stands for least absolute shrinkage and selection operator. + +Here we have defined the norm-1 as +\[ +\vert\vert \bm{x}\vert\vert_1 = \sum_i \vert x_i\vert. +\] + + +% !split +\subsection{More on Ridge Regression} + +Using the matrix-vector expression for Ridge regression, + +\[ +C(\bm{X},\bm{\beta})=\frac{1}{n}\left\{(\bm{y}-\bm{X}\bm{\beta})^T(\bm{y}-\bm{X}\bm{\beta})\right\}+\lambda\bm{\beta}^T\bm{\beta}, +\] + +by taking the derivatives with respect to $\bm{\beta}$ we obtain then +a slightly modified matrix inversion problem which for finite values +of $\lambda$ does not suffer from singularity problems. We obtain + +\[ +\bm{\beta}^{\mathrm{Ridge}} = \left(\bm{X}^T\bm{X}+\lambda\bm{I}\right)^{-1}\bm{X}^T\bm{y}, +\] + +with $\bm{I}$ being a $p\times p$ identity matrix with the constraint that + +\[ +\sum_{i=0}^{p-1} \beta_i^2 \leq t, +\] + +with $t$ a finite positive number. + +We see that Ridge regression is nothing but the standard +OLS with a modified diagonal term added to $\bm{X}^T\bm{X}$. The +consequences, in particular for our discussion of the bias-variance +are rather interesting. + +Furthermore, if we use the result above in terms of the SVD decomposition (our analysis was done for the OLS method), we had +\[ +(\bm{X}\bm{X}^T)\bm{U} = \bm{U}\bm{D}. +\] + +We can analyse the OLS solutions in terms of the eigenvectors (the columns) of the right singular value matrix $\bm{U}$ as +\[ +\bm{X}\bm{\beta} = \bm{X}\left(\bm{V}\bm{D}\bm{V}^T \right)^{-1}\bm{X}^T\bm{y}=\bm{U\Sigma V^T}\left(\bm{V}\bm{D}\bm{V}^T \right)^{-1}(\bm{U\Sigma V^T})^T\bm{y}=\bm{U}\bm{U}^T\bm{y} +\] + + +For Ridge regression this becomes + +\[ +\bm{X}\bm{\beta}^{\mathrm{Ridge}} = \bm{U\Sigma V^T}\left(\bm{V}\bm{D}\bm{V}^T+\lambda\bm{I} \right)^{-1}(\bm{U\Sigma V^T})^T\bm{y}=\sum_{j=0}^{p-1}\bm{u}_j\bm{u}_j^T\frac{\sigma_j^2}{\sigma_j^2+\lambda}\bm{y}, +\] + +with the vectors $\bm{u}_j$ being the columns of $\bm{U}$. + +% !split +\subsection{Interpreting the Ridge results} + +Since $\lambda \geq 0$, it means that compared to OLS, we have + +\[ +\frac{\sigma_j^2}{\sigma_j^2+\lambda} \leq 1. +\] + +Ridge regression finds the coordinates of $\bm{y}$ with respect to the +orthonormal basis $\bm{U}$, it then shrinks the coordinates by +$\frac{\sigma_j^2}{\sigma_j^2+\lambda}$. Recall that the SVD has +eigenvalues ordered in a descending way, that is $\sigma_i \geq +\sigma_{i+1}$. + +For small eigenvalues $\sigma_i$ it means that their contributions become less important, a fact which can be used to reduce the number of degrees of freedom. +Actually, calculating the variance of $\bm{X}\bm{v}_j$ shows that this quantity is equal to $\sigma_j^2/n$. +With a parameter $\lambda$ we can thus shrink the role of specific parameters. + + +% !split +\subsection{More interpretations} + +For the sake of simplicity, let us assume that the design matrix is orthonormal, that is + +\[ +\bm{X}^T\bm{X}=(\bm{X}^T\bm{X})^{-1} =\bm{I}. +\] + +In this case the standard OLS results in +\[ +\bm{\beta}^{\mathrm{OLS}} = \bm{X}^T\bm{y}=\sum_{i=0}^{p-1}\bm{u}_j\bm{u}_j^T\bm{y}, +\] + +and + +\[ +\bm{\beta}^{\mathrm{Ridge}} = \left(\bm{I}+\lambda\bm{I}\right)^{-1}\bm{X}^T\bm{y}=\left(1+\lambda\right)^{-1}\bm{\beta}^{\mathrm{OLS}}, +\] + +that is the Ridge estimator scales the OLS estimator by the inverse of a factor $1+\lambda$, and +the Ridge estimator converges to zero when the hyperparameter goes to +infinity. + +We will come back to more interpreations after we have gone through some of the statistical analysis part. + +For more discussions of Ridge and Lasso regression, \href{{https://arxiv.org/abs/1509.09169}}{Wessel van Wieringen's} article is highly recommended. +Similarly, \href{{https://arxiv.org/abs/1803.08823}}{Mehta et al's article} is also recommended. + +% !split +\subsection{Where are we going?} + +Before we proceed, we need to rethink what we have been doing. In our +eager to fit the data, we have omitted several important elements in +our regression analysis. In what follows we will +\begin{enumerate} +\item look at statistical properties, including a discussion of mean values, variance and the so-called bias-variance tradeoff + +\item introduce resampling techniques like cross-validation, bootstrapping and jackknife and more +\end{enumerate} + +\noindent +This will allow us to link the standard linear algebra methods we have discussed above to a statistical interpretation of the methods. + + + + + +% !split +\subsection{Resampling methods} + +% --- begin paragraph admon --- +\paragraph{} +Resampling methods are an indispensable tool in modern +statistics. They involve repeatedly drawing samples from a training +set and refitting a model of interest on each sample in order to +obtain additional information about the fitted model. For example, in +order to estimate the variability of a linear regression fit, we can +repeatedly draw different samples from the training data, fit a linear +regression to each new sample, and then examine the extent to which +the resulting fits differ. Such an approach may allow us to obtain +information that would not be available from fitting the model only +once using the original training sample. +% --- end paragraph admon --- + + + +% !split +\subsection{Resampling approaches can be computationally expensive} + +% --- begin paragraph admon --- +\paragraph{} + +Resampling approaches can be computationally expensive, because they +involve fitting the same statistical method multiple times using +different subsets of the training data. However, due to recent +advances in computing power, the computational requirements of +resampling methods generally are not prohibitive. In this chapter, we +discuss two of the most commonly used resampling methods, +cross-validation and the bootstrap. Both methods are important tools +in the practical application of many statistical learning +procedures. For example, cross-validation can be used to estimate the +test error associated with a given statistical learning method in +order to evaluate its performance, or to select the appropriate level +of flexibility. The process of evaluating a model’s performance is +known as model assessment, whereas the process of selecting the proper +level of flexibility for a model is known as model selection. The +bootstrap is widely used. +% --- end paragraph admon --- + + + +% !split +\subsection{Why resampling methods ?} + +% --- begin paragraph admon --- +\paragraph{Statistical analysis.} + +\begin{itemize} +\item Our simulations can be treated as \emph{computer experiments}. This is particularly the case for Monte Carlo methods + +\item The results can be analysed with the same statistical tools as we would use analysing experimental data. + +\item As in all experiments, we are looking for expectation values and an estimate of how accurate they are, i.e., possible sources for errors. +\end{itemize} + +\noindent +% --- end paragraph admon --- + + + +% !split +\subsection{Statistical analysis} + +% --- begin paragraph admon --- +\paragraph{} + +\begin{itemize} +\item As in other experiments, many numerical experiments have two classes of errors: +\begin{itemize} + + \item Statistical errors + + \item Systematical errors + +\end{itemize} + +\noindent +\item Statistical errors can be estimated using standard tools from statistics + +\item Systematical errors are method specific and must be treated differently from case to case. +\end{itemize} + +\noindent +% --- end paragraph admon --- + + + +% !split +\subsection{Statistics} + +% --- begin paragraph admon --- +\paragraph{} +The \emph{probability distribution function (PDF)} is a function +$p(x)$ on the domain which, in the discrete case, gives us the +probability or relative frequency with which these values of $X$ occur: +\[ +p(x) = \mathrm{prob}(X=x) +\] +In the continuous case, the PDF does not directly depict the +actual probability. Instead we define the probability for the +stochastic variable to assume any value on an infinitesimal interval +around $x$ to be $p(x)dx$. The continuous function $p(x)$ then gives us +the \emph{density} of the probability rather than the probability +itself. The probability for a stochastic variable to assume any value +on a non-infinitesimal interval $[a,\,b]$ is then just the integral: +\[ +\mathrm{prob}(a\leq X\leq b) = \int_a^b p(x)dx +\] +Qualitatively speaking, a stochastic variable represents the values of +numbers chosen as if by chance from some specified PDF so that the +selection of a large set of these numbers reproduces this PDF. +% --- end paragraph admon --- + + + + +% !split +\subsection{Statistics, moments} + +% --- begin paragraph admon --- +\paragraph{} +A particularly useful class of special expectation values are the +\emph{moments}. The $n$-th moment of the PDF $p$ is defined as +follows: +\[ +\langle x^n\rangle \equiv \int\! x^n p(x)\,dx +\] +The zero-th moment $\langle 1\rangle$ is just the normalization condition of +$p$. The first moment, $\langle x\rangle$, is called the \emph{mean} of $p$ +and often denoted by the letter $\mu$: +\[ +\langle x\rangle = \mu \equiv \int\! x p(x)\,dx +\] +% --- end paragraph admon --- + + + +% !split +\subsection{Statistics, central moments} + +% --- begin paragraph admon --- +\paragraph{} +A special version of the moments is the set of \emph{central moments}, +the n-th central moment defined as: +\[ +\langle (x-\langle x \rangle )^n\rangle \equiv \int\! (x-\langle x\rangle)^n p(x)\,dx +\] +The zero-th and first central moments are both trivial, equal $1$ and +$0$, respectively. But the second central moment, known as the +\emph{variance} of $p$, is of particular interest. For the stochastic +variable $X$, the variance is denoted as $\sigma^2_X$ or $\mathrm{var}(X)$: +\begin{align} +\sigma^2_X\ \ =\ \ \mathrm{var}(X) & = \langle (x-\langle x\rangle)^2\rangle = +\int\! (x-\langle x\rangle)^2 p(x)\,dx\\ +& = \int\! \left(x^2 - 2 x \langle x\rangle^{2} + + \langle x\rangle^2\right)p(x)\,dx\\ +& = \langle x^2\rangle - 2 \langle x\rangle\langle x\rangle + \langle x\rangle^2\\ +& = \langle x^2\rangle - \langle x\rangle^2 +\end{align} +The square root of the variance, $\sigma =\sqrt{\langle (x-\langle x\rangle)^2\rangle}$ is called the \emph{standard deviation} of $p$. It is clearly just the RMS (root-mean-square) +value of the deviation of the PDF from its mean value, interpreted +qualitatively as the \emph{spread} of $p$ around its mean. +% --- end paragraph admon --- + + + +% !split +\subsection{Statistics, covariance} + +% --- begin paragraph admon --- +\paragraph{} +Another important quantity is the so called covariance, a variant of +the above defined variance. Consider again the set $\{X_i\}$ of $n$ +stochastic variables (not necessarily uncorrelated) with the +multivariate PDF $P(x_1,\dots,x_n)$. The \emph{covariance} of two +of the stochastic variables, $X_i$ and $X_j$, is defined as follows: +\begin{align} +\mathrm{cov}(X_i,\,X_j) &\equiv \langle (x_i-\langle x_i\rangle)(x_j-\langle x_j\rangle)\rangle +\nonumber\\ +&= +\int\!\cdots\!\int\!(x_i-\langle x_i \rangle)(x_j-\langle x_j \rangle)\, +P(x_1,\dots,x_n)\,dx_1\dots dx_n +\label{eq:def_covariance} +\end{align} +with +\[ +\langle x_i\rangle = +\int\!\cdots\!\int\!x_i\,P(x_1,\dots,x_n)\,dx_1\dots dx_n +\] +% --- end paragraph admon --- + + + +% !split +\subsection{Statistics, more covariance} + +% --- begin paragraph admon --- +\paragraph{} +If we consider the above covariance as a matrix $C_{ij}=\mathrm{cov}(X_i,\,X_j)$, then the diagonal elements are just the familiar +variances, $C_{ii} = \mathrm{cov}(X_i,\,X_i) = \mathrm{var}(X_i)$. It turns out that +all the off-diagonal elements are zero if the stochastic variables are +uncorrelated. This is easy to show, keeping in mind the linearity of +the expectation value. Consider the stochastic variables $X_i$ and +$X_j$, ($i\neq j$): +\begin{align} +\mathrm{cov}(X_i,\,X_j) &= \langle(x_i-\langle x_i\rangle)(x_j-\langle x_j\rangle)\rangle\\ +&=\langle x_i x_j - x_i\langle x_j\rangle - \langle x_i\rangle x_j + \langle x_i\rangle\langle x_j\rangle\rangle \\ +&=\langle x_i x_j\rangle - \langle x_i\langle x_j\rangle\rangle - \langle \langle x_i\rangle x_j\rangle + +\langle \langle x_i\rangle\langle x_j\rangle\rangle\\ +&=\langle x_i x_j\rangle - \langle x_i\rangle\langle x_j\rangle - \langle x_i\rangle\langle x_j\rangle + +\langle x_i\rangle\langle x_j\rangle\\ +&=\langle x_i x_j\rangle - \langle x_i\rangle\langle x_j\rangle +\end{align} +% --- end paragraph admon --- + + + + + +% !split +\subsection{Statistics, independent variables} + +% --- begin paragraph admon --- +\paragraph{} +If $X_i$ and $X_j$ are independent, we get +$\langle x_i x_j\rangle =\langle x_i\rangle\langle x_j\rangle$, resulting in $\mathrm{cov}(X_i, X_j) = 0\ \ (i\neq j)$. + +Also useful for us is the covariance of linear combinations of +stochastic variables. Let $\{X_i\}$ and $\{Y_i\}$ be two sets of +stochastic variables. Let also $\{a_i\}$ and $\{b_i\}$ be two sets of +scalars. Consider the linear combination: +\[ +U = \sum_i a_i X_i \qquad V = \sum_j b_j Y_j +\] +By the linearity of the expectation value +\[ +\mathrm{cov}(U, V) = \sum_{i,j}a_i b_j \mathrm{cov}(X_i, Y_j) +\] +% --- end paragraph admon --- + + + +% !split +\subsection{Statistics, more variance} + +% --- begin paragraph admon --- +\paragraph{} +Now, since the variance is just $\mathrm{var}(X_i) = \mathrm{cov}(X_i, X_i)$, we get +the variance of the linear combination $U = \sum_i a_i X_i$: +\begin{equation} +\mathrm{var}(U) = \sum_{i,j}a_i a_j \mathrm{cov}(X_i, X_j) +\label{eq:variance_linear_combination} +\end{equation} +And in the special case when the stochastic variables are +uncorrelated, the off-diagonal elements of the covariance are as we +know zero, resulting in: +\[ +\mathrm{var}(U) = \sum_i a_i^2 \mathrm{cov}(X_i, X_i) = \sum_i a_i^2 \mathrm{var}(X_i) +\] +\[ +\mathrm{var}(\sum_i a_i X_i) = \sum_i a_i^2 \mathrm{var}(X_i) +\] +which will become very useful in our study of the error in the mean +value of a set of measurements. +% --- end paragraph admon --- + + + +% !split +\subsection{Statistics and stochastic processes} + +% --- begin paragraph admon --- +\paragraph{} +A \emph{stochastic process} is a process that produces sequentially a +chain of values: +\[ +\{x_1, x_2,\dots\,x_k,\dots\}. +\] +We will call these +values our \emph{measurements} and the entire set as our measured +\emph{sample}. The action of measuring all the elements of a sample +we will call a stochastic \emph{experiment} since, operationally, +they are often associated with results of empirical observation of +some physical or mathematical phenomena; precisely an experiment. We +assume that these values are distributed according to some +PDF $p_X^{\phantom X}(x)$, where $X$ is just the formal symbol for the +stochastic variable whose PDF is $p_X^{\phantom X}(x)$. Instead of +trying to determine the full distribution $p$ we are often only +interested in finding the few lowest moments, like the mean +$\mu_X^{\phantom X}$ and the variance $\sigma_X^{\phantom X}$. +% --- end paragraph admon --- + + + + +% !split +\subsection{Statistics and sample variables} + +% --- begin paragraph admon --- +\paragraph{} +In practical situations a sample is always of finite size. Let that +size be $n$. The expectation value of a sample, the \emph{sample mean}, is then defined as follows: +\[ +\bar{x}_n \equiv \frac{1}{n}\sum_{k=1}^n x_k +\] +The \emph{sample variance} is: +\[ +\mathrm{var}(x) \equiv \frac{1}{n}\sum_{k=1}^n (x_k - \bar{x}_n)^2 +\] +its square root being the \emph{standard deviation of the sample}. The +\emph{sample covariance} is: +\[ +\mathrm{cov}(x)\equiv\frac{1}{n}\sum_{kl}(x_k - \bar{x}_n)(x_l - \bar{x}_n) +\] +% --- end paragraph admon --- + + + +% !split +\subsection{Statistics, sample variance and covariance} + +% --- begin paragraph admon --- +\paragraph{} +Note that the sample variance is the sample covariance without the +cross terms. In a similar manner as the covariance in Eq.~(\ref{eq:def_covariance}) is a measure of the correlation between +two stochastic variables, the above defined sample covariance is a +measure of the sequential correlation between succeeding measurements +of a sample. + +These quantities, being known experimental values, differ +significantly from and must not be confused with the similarly named +quantities for stochastic variables, mean $\mu_X$, variance $\mathrm{var}(X)$ +and covariance $\mathrm{cov}(X,Y)$. +% --- end paragraph admon --- + + + +% !split +\subsection{Statistics, law of large numbers} + +% --- begin paragraph admon --- +\paragraph{} +The law of large numbers +states that as the size of our sample grows to infinity, the sample +mean approaches the true mean $\mu_X^{\phantom X}$ of the chosen PDF: +\[ +\lim_{n\to\infty}\bar{x}_n = \mu_X^{\phantom X} +\] +The sample mean $\bar{x}_n$ works therefore as an estimate of the true +mean $\mu_X^{\phantom X}$. + +What we need to find out is how good an approximation $\bar{x}_n$ is to +$\mu_X^{\phantom X}$. In any stochastic measurement, an estimated +mean is of no use to us without a measure of its error. A quantity +that tells us how well we can reproduce it in another experiment. We +are therefore interested in the PDF of the sample mean itself. Its +standard deviation will be a measure of the spread of sample means, +and we will simply call it the \emph{error} of the sample mean, or +just sample error, and denote it by $\mathrm{err}_X^{\phantom X}$. In +practice, we will only be able to produce an \emph{estimate} of the +sample error since the exact value would require the knowledge of the +true PDFs behind, which we usually do not have. +% --- end paragraph admon --- + + + + +% !split +\subsection{Statistics, more on sample error} + +% --- begin paragraph admon --- +\paragraph{} +Let us first take a look at what happens to the sample error as the +size of the sample grows. In a sample, each of the measurements $x_i$ +can be associated with its own stochastic variable $X_i$. The +stochastic variable $\overline X_n$ for the sample mean $\bar{x}_n$ is +then just a linear combination, already familiar to us: +\[ +\overline X_n = \frac{1}{n}\sum_{i=1}^n X_i +\] +All the coefficients are just equal $1/n$. The PDF of $\overline X_n$, +denoted by $p_{\overline X_n}(x)$ is the desired PDF of the sample +means. +% --- end paragraph admon --- + + + +% !split +\subsection{Statistics} + +% --- begin paragraph admon --- +\paragraph{} +The probability density of obtaining a sample mean $\bar x_n$ +is the product of probabilities of obtaining arbitrary values $x_1, +x_2,\dots,x_n$ with the constraint that the mean of the set $\{x_i\}$ +is $\bar x_n$: +\[ +p_{\overline X_n}(x) = \int p_X^{\phantom X}(x_1)\cdots +\int p_X^{\phantom X}(x_n)\ +\delta\!\left(x - \frac{x_1+x_2+\dots+x_n}{n}\right)dx_n \cdots dx_1 +\] +And in particular we are interested in its variance $\mathrm{var}(\overline X_n)$. +% --- end paragraph admon --- + + + + + +% !split +\subsection{Statistics, central limit theorem} + +% --- begin paragraph admon --- +\paragraph{} +It is generally not possible to express $p_{\overline X_n}(x)$ in a +closed form given an arbitrary PDF $p_X^{\phantom X}$ and a number +$n$. But for the limit $n\to\infty$ it is possible to make an +approximation. The very important result is called \emph{the central limit theorem}. It tells us that as $n$ goes to infinity, +$p_{\overline X_n}(x)$ approaches a Gaussian distribution whose mean +and variance equal the true mean and variance, $\mu_{X}^{\phantom X}$ +and $\sigma_{X}^{2}$, respectively: +\begin{equation} +\lim_{n\to\infty} p_{\overline X_n}(x) = +\left(\frac{n}{2\pi\mathrm{var}(X)}\right)^{1/2} +e^{-\frac{n(x-\bar x_n)^2}{2\mathrm{var}(X)}} +\label{eq:central_limit_gaussian} +\end{equation} +% --- end paragraph admon --- + + + +% !split +\subsection{Statistics, more technicalities} + +% --- begin paragraph admon --- +\paragraph{} +The desired variance +$\mathrm{var}(\overline X_n)$, i.e.~the sample error squared +$\mathrm{err}_X^2$, is given by: +\begin{equation} +\mathrm{err}_X^2 = \mathrm{var}(\overline X_n) = \frac{1}{n^2} +\sum_{ij} \mathrm{cov}(X_i, X_j) +\label{eq:error_exact} +\end{equation} +We see now that in order to calculate the exact error of the sample +with the above expression, we would need the true means +$\mu_{X_i}^{\phantom X}$ of the stochastic variables $X_i$. To +calculate these requires that we know the true multivariate PDF of all +the $X_i$. But this PDF is unknown to us, we have only got the measurements of +one sample. The best we can do is to let the sample itself be an +estimate of the PDF of each of the $X_i$, estimating all properties of +$X_i$ through the measurements of the sample. +% --- end paragraph admon --- + + + + +% !split +\subsection{Statistics} + +% --- begin paragraph admon --- +\paragraph{} +Our estimate of $\mu_{X_i}^{\phantom X}$ is then the sample mean $\bar x$ +itself, in accordance with the the central limit theorem: +\[ +\mu_{X_i}^{\phantom X} = \langle x_i\rangle \approx \frac{1}{n}\sum_{k=1}^n x_k = \bar x +\] +Using $\bar x$ in place of $\mu_{X_i}^{\phantom X}$ we can give an +\emph{estimate} of the covariance in Eq.~(\ref{eq:error_exact}) +\[ +\mathrm{cov}(X_i, X_j) = \langle (x_i-\langle x_i\rangle)(x_j-\langle x_j\rangle)\rangle +\approx\langle (x_i - \bar x)(x_j - \bar{x})\rangle, +\] +resulting in +\[ +\frac{1}{n} \sum_{l}^n \left(\frac{1}{n}\sum_{k}^n (x_k -\bar x_n)(x_l - \bar x_n)\right)=\frac{1}{n}\frac{1}{n} \sum_{kl} (x_k -\bar x_n)(x_l - \bar x_n)=\frac{1}{n}\mathrm{cov}(x) +\] +% --- end paragraph admon --- + + + +% !split +\subsection{Statistics and sample variance} + +% --- begin paragraph admon --- +\paragraph{} +By the same procedure we can use the sample variance as an +estimate of the variance of any of the stochastic variables $X_i$ +\[ +\mathrm{var}(X_i)=\langle x_i - \langle x_i\rangle\rangle \approx \langle x_i - \bar x_n\rangle\nonumber, +\] +which is approximated as +\begin{equation} +\mathrm{var}(X_i)\approx \frac{1}{n}\sum_{k=1}^n (x_k - \bar x_n)=\mathrm{var}(x) +\label{eq:var_estimate_i_think} +\end{equation} + +Now we can calculate an estimate of the error +$\mathrm{err}_X^{\phantom X}$ of the sample mean $\bar x_n$: +\begin{align} +\mathrm{err}_X^2 +&=\frac{1}{n^2}\sum_{ij} \mathrm{cov}(X_i, X_j) \nonumber \\ +&\approx&\frac{1}{n^2}\sum_{ij}\frac{1}{n}\mathrm{cov}(x) =\frac{1}{n^2}n^2\frac{1}{n}\mathrm{cov}(x)\nonumber\\ +&=\frac{1}{n}\mathrm{cov}(x) +\label{eq:error_estimate} +\end{align} +which is nothing but the sample covariance divided by the number of +measurements in the sample. +% --- end paragraph admon --- + + + +% !split +\subsection{Statistics, uncorrelated results} + +% --- begin paragraph admon --- +\paragraph{} + +In the special case that the measurements of the sample are +uncorrelated (equivalently the stochastic variables $X_i$ are +uncorrelated) we have that the off-diagonal elements of the covariance +are zero. This gives the following estimate of the sample error: +\[ +\mathrm{err}_X^2=\frac{1}{n^2}\sum_{ij} \mathrm{cov}(X_i, X_j) = +\frac{1}{n^2} \sum_i \mathrm{var}(X_i), +\] +resulting in +\begin{equation} +\mathrm{err}_X^2\approx \frac{1}{n^2} \sum_i \mathrm{var}(x)= \frac{1}{n}\mathrm{var}(x) +\label{eq:error_estimate_uncorrel} +\end{equation} +where in the second step we have used Eq.~(\ref{eq:var_estimate_i_think}). +The error of the sample is then just its standard deviation divided by +the square root of the number of measurements the sample contains. +This is a very useful formula which is easy to compute. It acts as a +first approximation to the error, but in numerical experiments, we +cannot overlook the always present correlations. +% --- end paragraph admon --- + + + +% !split +\subsection{Statistics, computations} + +% --- begin paragraph admon --- +\paragraph{} +For computational purposes one usually splits up the estimate of +$\mathrm{err}_X^2$, given by Eq.~(\ref{eq:error_estimate}), into two +parts +\[ +\mathrm{err}_X^2 = \frac{1}{n}\mathrm{var}(x) + \frac{1}{n}(\mathrm{cov}(x)-\mathrm{var}(x)), +\] +which equals +\begin{equation} +\frac{1}{n^2}\sum_{k=1}^n (x_k - \bar x_n)^2 +\frac{2}{n^2}\sum_{k 0$. We say then that the ridge estimator is biased. + +We can also compute the variance as + +\[ +\mbox{Var}[\bm{\beta}^{\mathrm{Ridge}}]=\sigma^2[ \mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I} ]^{-1} \mathbf{X}^{T} \mathbf{X} \{ [ \mathbf{X}^{\top} \mathbf{X} + \lambda \mathbf{I} ]^{-1}\}^{T}, +\] +and it is easy to see that if the parameter $\lambda$ goes to infinity then the variance of Ridge parameters $\bm{\beta}$ goes to zero. + +With this, we can compute the difference + +\[ +\mbox{Var}[\bm{\beta}^{\mathrm{OLS}}]-\mbox{Var}(\bm{\beta}^{\mathrm{Ridge}})=\sigma^2 [ \mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I} ]^{-1}[ 2\lambda\mathbf{I} + \lambda^2 (\mathbf{X}^{T} \mathbf{X})^{-1} ] \{ [ \mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I} ]^{-1}\}^{T}. +\] +The difference is non-negative definite since each component of the +matrix product is non-negative definite. +This means the variance we obtain with the standard OLS will always for $\lambda > 0$ be larger than the variance of $\bm{\beta}$ obtained with the Ridge estimator. This has interesting consequences when we discuss the so-called bias-variance trade-off below. + +% !split +\subsection{Cross-validation} + +Instead of choosing the penalty parameter to balance model fit with +model complexity, cross-validation requires it (i.e.~the penalty +parameter) to yield a model with good prediction +performance. Commonly, this performance is evaluated on novel +data. Novel data need not be easy to come by and one has to make do +with the data at hand. + +The setting of \textbf{original} and novel data is +then mimicked by sample splitting: the data set is divided into two +(groups of samples). One of these two data sets, called the +\emph{training set}, plays the role of \textbf{original} data on which the model is +built. The second of these data sets, called the \emph{test set}, plays the +role of the \textbf{novel} data and is used to evaluate the prediction +performance (often operationalized as the log-likelihood or the +prediction error or its square or the R2 score) of the model built on the training data set. This +procedure (model building and prediction evaluation on training and +test set, respectively) is done for a collection of possible penalty +parameter choices. The penalty parameter that yields the model with +the best prediction performance is to be preferred. The thus obtained +performance evaluation depends on the actual split of the data set. To +remove this dependence the data set is split many times into a +training and test set. For each split the model parameters are +estimated for all choices of $\lambda$ using the training data and +estimated parameters are evaluated on the corresponding test set. The +penalty parameter that on average over the test sets performs best (in +some sense) is then selected. + + +% !split +\subsection{Computationally expensive} + +The validation set approach is conceptually simple and is easy to implement. But it has two potential drawbacks: + +\begin{itemize} +\item The validation estimate of the test error rate can be highly variable, depending on precisely which observations are included in the training set and which observations are included in the validation set. + +\item In the validation approach, only a subset of the observations, those that are included in the training set rather than in the validation set are used to fit the model. Since statistical methods tend to perform worse when trained on fewer observations, this suggests that the validation set error rate may tend to overestimate the test error rate for the model fit on the entire data set. +\end{itemize} + +\noindent +% !split +\subsection{Various steps in cross-validation} + +When the repetitive splitting of the data set is done randomly, +samples may accidently end up in a fast majority of the splits in +either training or test set. Such samples may have an unbalanced +influence on either model building or prediction evaluation. To avoid +this $k$-fold cross-validation structures the data splitting. The +samples are divided into $k$ more or less equally sized exhaustive and +mutually exclusive subsets. In turn (at each split) one of these +subsets plays the role of the test set while the union of the +remaining subsets constitutes the training set. Such a splitting +warrants a balanced representation of each sample in both training and +test set over the splits. Still the division into the $k$ subsets +involves a degree of randomness. This may be fully excluded when +choosing $k=n$. This particular case is referred to as leave-one-out +cross-validation (LOOCV). + +% !split +\subsection{How to set up the cross-validation for Ridge and/or Lasso} + +\begin{itemize} +\item Define a range of interest for the penalty parameter. + +\item Divide the data set into training and test set comprising samples $\{1, \ldots, n\} \setminus i$ and $\{ i \}$, respectively. + +\item Fit the linear regression model by means of ridge estimation for each $\lambda$ in the grid using the training set, and the corresponding estimate of the error variance $\bm{\sigma}_{-i}^2(\lambda)$, as +\end{itemize} + +\noindent +\begin{align*} +\bm{\beta}_{-i}(\lambda) & = ( \bm{X}_{-i, \ast}^{T} +\bm{X}_{-i, \ast} + \lambda \bm{I}_{pp})^{-1} +\bm{X}_{-i, \ast}^{T} \bm{y}_{-i} +\end{align*} + +\begin{itemize} +\item Evaluate the prediction performance of these models on the test set by $\log\{L[y_i, \bm{X}_{i, \ast}; \bm{\beta}_{-i}(\lambda), \bm{\sigma}_{-i}^2(\lambda)]\}$. Or, by the prediction error $|y_i - \bm{X}_{i, \ast} \bm{\beta}_{-i}(\lambda)|$, the relative error, the error squared or the R2 score function. + +\item Repeat the first three steps such that each sample plays the role of the test set once. + +\item Average the prediction performances of the test sets at each grid point of the penalty bias/parameter by computing the \emph{cross-validated log-likelihood}. It is an estimate of the prediction performance of the model corresponding to this value of the penalty parameter on novel data. It is defined as +\end{itemize} + +\noindent +\begin{align*} +\frac{1}{n} \sum_{i = 1}^n \log\{L[y_i, \mathbf{X}_{i, \ast}; \bm{\beta}_{-i}(\lambda), \bm{\sigma}_{-i}^2(\lambda)]\}. +\end{align*} + +\begin{itemize} +\item The value of the penalty parameter that maximizes the cross-validated log-likelihood is the value of choice. Or we can use the MSE or the R2 score functions. +\end{itemize} + +\noindent +% !split +\subsection{Resampling methods: Jackknife and Bootstrap} + +Two famous +resampling methods are the \textbf{independent bootstrap} and \textbf{the jackknife}. + +The jackknife is a special case of the independent bootstrap. Still, the jackknife was made +popular prior to the independent bootstrap. And as the popularity of +the independent bootstrap soared, new variants, such as \textbf{the dependent bootstrap}. + +The Jackknife and independent bootstrap work for +independent, identically distributed random variables. +If these conditions are not +satisfied, the methods will fail. Yet, it should be said that if the data are +independent, identically distributed, and we only want to estimate the +variance of $\overline{X}$ (which often is the case), then there is no +need for bootstrapping. + +% !split +\subsection{Resampling methods: Jackknife} + +The Jackknife works by making many replicas of the estimator $\widehat{\theta}$. +The jackknife is a resampling method where we systematically leave out one observation from the vector of observed values $\bm{x} = (x_1,x_2,\cdots,X_n)$. +Let $\bm{x}_i$ denote the vector +\[ +\bm{x}_i = (x_1,x_2,\cdots,x_{i-1},x_{i+1},\cdots,x_n), +\] + +which equals the vector $\bm{x}$ with the exception that observation +number $i$ is left out. Using this notation, define +$\widehat{\theta}_i$ to be the estimator +$\widehat{\theta}$ computed using $\vec{X}_i$. + + +% !split +\subsection{Jackknife code example} +\begin{minted}[fontsize=\fontsize{9pt}{9pt},linenos=false,mathescape,baselinestretch=1.0,fontfamily=tt,xleftmargin=7mm]{python} +from numpy import * +from numpy.random import randint, randn +from time import time + +def jackknife(data, stat): + n = len(data);t = zeros(n); inds = arange(n); t0 = time() + ## 'jackknifing' by leaving out an observation for each i + for i in range(n): + t[i] = stat(delete(data,i) ) + + # analysis + print("Runtime: %g sec" % (time()-t0)); print("Jackknife Statistics :") + print("original bias std. error") + print("%8g %14g %15g" % (stat(data),(n-1)*mean(t)/n, (n*var(t))**.5)) + + return t + + +# Returns mean of data samples +def stat(data): + return mean(data) + + +mu, sigma = 100, 15 +datapoints = 10000 +x = mu + sigma*random.randn(datapoints) +# jackknife returns the data sample +t = jackknife(x, stat) + +\end{minted} + + +% !split +\subsection{Resampling methods: Bootstrap} + +% --- begin paragraph admon --- +\paragraph{} +Bootstrapping is a nonparametric approach to statistical inference +that substitutes computation for more traditional distributional +assumptions and asymptotic results. Bootstrapping offers a number of +advantages: +\begin{enumerate} +\item The bootstrap is quite general, although there are some cases in which it fails. + +\item Because it does not require distributional assumptions (such as normally distributed errors), the bootstrap can provide more accurate inferences when the data are not well behaved or when the sample size is small. + +\item It is possible to apply the bootstrap to statistics with sampling distributions that are difficult to derive, even asymptotically. + +\item It is relatively simple to apply the bootstrap to complex data-collection plans (such as stratified and clustered samples). +\end{enumerate} + +\noindent +% --- end paragraph admon --- + + + + +% !split +\subsection{Resampling methods: Bootstrap background} + +Since $\widehat{\theta} = \widehat{\theta}(\bm{X})$ is a function of random variables, +$\widehat{\theta}$ itself must be a random variable. Thus it has +a pdf, call this function $p(\bm{t})$. The aim of the bootstrap is to +estimate $p(\bm{t})$ by the relative frequency of +$\widehat{\theta}$. You can think of this as using a histogram +in the place of $p(\bm{t})$. If the relative frequency closely +resembles $p(\vec{t})$, then using numerics, it is straight forward to +estimate all the interesting parameters of $p(\bm{t})$ using point +estimators. + + +% !split +\subsection{Resampling methods: More Bootstrap background} + +In the case that $\widehat{\theta}$ has +more than one component, and the components are independent, we use the +same estimator on each component separately. If the probability +density function of $X_i$, $p(x)$, had been known, then it would have +been straight forward to do this by: +\begin{enumerate} +\item Drawing lots of numbers from $p(x)$, suppose we call one such set of numbers $(X_1^*, X_2^*, \cdots, X_n^*)$. + +\item Then using these numbers, we could compute a replica of $\widehat{\theta}$ called $\widehat{\theta}^*$. +\end{enumerate} + +\noindent +By repeated use of (1) and (2), many +estimates of $\widehat{\theta}$ could have been obtained. The +idea is to use the relative frequency of $\widehat{\theta}^*$ +(think of a histogram) as an estimate of $p(\bm{t})$. + +% !split +\subsection{Resampling methods: Bootstrap approach} + +But +unless there is enough information available about the process that +generated $X_1,X_2,\cdots,X_n$, $p(x)$ is in general +unknown. Therefore, \href{{https://projecteuclid.org/euclid.aos/1176344552}}{Efron in 1979} asked the +question: What if we replace $p(x)$ by the relative frequency +of the observation $X_i$; if we draw observations in accordance with +the relative frequency of the observations, will we obtain the same +result in some asymptotic sense? The answer is yes. + + +Instead of generating the histogram for the relative +frequency of the observation $X_i$, just draw the values +$(X_1^*,X_2^*,\cdots,X_n^*)$ with replacement from the vector +$\bm{X}$. + +% !split +\subsection{Resampling methods: Bootstrap steps} + +The independent bootstrap works like this: + +\begin{enumerate} +\item Draw with replacement $n$ numbers for the observed variables $\bm{x} = (x_1,x_2,\cdots,x_n)$. + +\item Define a vector $\bm{x}^*$ containing the values which were drawn from $\bm{x}$. + +\item Using the vector $\bm{x}^*$ compute $\widehat{\theta}^*$ by evaluating $\widehat \theta$ under the observations $\bm{x}^*$. + +\item Repeat this process $k$ times. +\end{enumerate} + +\noindent +When you are done, you can draw a histogram of the relative frequency +of $\widehat \theta^*$. This is your estimate of the probability +distribution $p(t)$. Using this probability distribution you can +estimate any statistics thereof. In principle you never draw the +histogram of the relative frequency of $\widehat{\theta}^*$. Instead +you use the estimators corresponding to the statistic of interest. For +example, if you are interested in estimating the variance of $\widehat +\theta$, apply the etsimator $\widehat \sigma^2$ to the values +$\widehat \theta ^*$. + + +% !split +\subsection{Code example for the Bootstrap method} + +The following code starts with a Gaussian distribution with mean value +$\mu =100$ and variance $\sigma=15$. We use this to generate the data +used in the bootstrap analysis. The bootstrap analysis returns a data +set after a given number of bootstrap operations (as many as we have +data points). This data set consists of estimated mean values for each +bootstrap operation. The histogram generated by the bootstrap method +shows that the distribution for these mean values is also a Gaussian, +centered around the mean value $\mu=100$ but with standard deviation +$\sigma/\sqrt{n}$, where $n$ is the number of bootstrap samples (in +this case the same as the number of original data points). The value +of the standard deviation is what we expect from the central limit +theorem. + + +\begin{minted}[fontsize=\fontsize{9pt}{9pt},linenos=false,mathescape,baselinestretch=1.0,fontfamily=tt,xleftmargin=7mm]{python} +from numpy import * +from numpy.random import randint, randn +from time import time +import matplotlib.mlab as mlab +import matplotlib.pyplot as plt + +# Returns mean of bootstrap samples +def stat(data): + return mean(data) + +# Bootstrap algorithm +def bootstrap(data, statistic, R): + t = zeros(R); n = len(data); inds = arange(n); t0 = time() + # non-parametric bootstrap + for i in range(R): + t[i] = statistic(data[randint(0,n,n)]) + + # analysis + print("Runtime: %g sec" % (time()-t0)); print("Bootstrap Statistics :") + print("original bias std. error") + print("%8g %8g %14g %15g" % (statistic(data), std(data),mean(t),std(t))) + return t + + +mu, sigma = 100, 15 +datapoints = 10000 +x = mu + sigma*random.randn(datapoints) +# bootstrap returns the data sample +t = bootstrap(x, stat, datapoints) +# the histogram of the bootstrapped data +n, binsboot, patches = plt.hist(t, 50, normed=1, facecolor='red', alpha=0.75) + +# add a 'best fit' line +y = mlab.normpdf( binsboot, mean(t), std(t)) +lt = plt.plot(binsboot, y, 'r--', linewidth=1) +plt.xlabel('Smarts') +plt.ylabel('Probability') +plt.axis([99.5, 100.6, 0, 3.0]) +plt.grid(True) + +plt.show() + +\end{minted} + + +% !split +\subsection{Code Example for Cross-validation and $k$-fold Cross-validation} + +The code here uses Ridge regression with cross-validation (CV) resampling and $k$-fold CV in order to fit a specific polynomial. +\begin{minted}[fontsize=\fontsize{9pt}{9pt},linenos=false,mathescape,baselinestretch=1.0,fontfamily=tt,xleftmargin=7mm]{python} +import numpy as np +import matplotlib.pyplot as plt +from sklearn.model_selection import KFold +from sklearn.linear_model import Ridge +from sklearn.model_selection import cross_val_score +from sklearn.preprocessing import PolynomialFeatures + +# A seed just to ensure that the random numbers are the same for every run. +# Useful for eventual debugging. +np.random.seed(3155) + +# Generate the data. +nsamples = 100 +x = np.random.randn(nsamples) +y = 3*x**2 + np.random.randn(nsamples) + +## Cross-validation on Ridge regression using KFold only + +# Decide degree on polynomial to fit +poly = PolynomialFeatures(degree = 6) + +# Decide which values of lambda to use +nlambdas = 500 +lambdas = np.logspace(-3, 5, nlambdas) + +# Initialize a KFold instance +k = 5 +kfold = KFold(n_splits = k) + +# Perform the cross-validation to estimate MSE +scores_KFold = np.zeros((nlambdas, k)) + +i = 0 +for lmb in lambdas: + ridge = Ridge(alpha = lmb) + j = 0 + for train_inds, test_inds in kfold.split(x): + xtrain = x[train_inds] + ytrain = y[train_inds] + + xtest = x[test_inds] + ytest = y[test_inds] + + Xtrain = poly.fit_transform(xtrain[:, np.newaxis]) + ridge.fit(Xtrain, ytrain[:, np.newaxis]) + + Xtest = poly.fit_transform(xtest[:, np.newaxis]) + ypred = ridge.predict(Xtest) + + scores_KFold[i,j] = np.sum((ypred - ytest[:, np.newaxis])**2)/np.size(ypred) + + j += 1 + i += 1 + + +estimated_mse_KFold = np.mean(scores_KFold, axis = 1) + +## Cross-validation using cross_val_score from sklearn along with KFold + +# kfold is an instance initialized above as: +# kfold = KFold(n_splits = k) + +estimated_mse_sklearn = np.zeros(nlambdas) +i = 0 +for lmb in lambdas: + ridge = Ridge(alpha = lmb) + + X = poly.fit_transform(x[:, np.newaxis]) + estimated_mse_folds = cross_val_score(ridge, X, y[:, np.newaxis], scoring='neg_mean_squared_error', cv=kfold) + + # cross_val_score return an array containing the estimated negative mse for every fold. + # we have to the the mean of every array in order to get an estimate of the mse of the model + estimated_mse_sklearn[i] = np.mean(-estimated_mse_folds) + + i += 1 + +## Plot and compare the slightly different ways to perform cross-validation + +plt.figure() + +plt.plot(np.log10(lambdas), estimated_mse_sklearn, label = 'cross_val_score') +plt.plot(np.log10(lambdas), estimated_mse_KFold, 'r--', label = 'KFold') + +plt.xlabel('log10(lambda)') +plt.ylabel('mse') + +plt.legend() + +plt.show() + +\end{minted} + + +% !split +\subsection{The bias-variance tradeoff} + + +We will discuss the bias-variance tradeoff in the context of +continuous predictions such as regression. However, many of the +intuitions and ideas discussed here also carry over to classification +tasks. Consider a dataset $\mathcal{L}$ consisting of the data +$\mathbf{X}_\mathcal{L}=\{(y_j, \boldsymbol{x}_j), j=0\ldots n-1\}$. + +Let us assume that the true data is generated from a noisy model + +\[ +\bm{y}=f(\boldsymbol{x}) + \bm{\epsilon} +\] + +where $\epsilon$ is normally distributed with mean zero and standard deviation $\sigma^2$. + +In our derivation of the ordinary least squares method we defined then +an approximation to the function $f$ in terms of the parameters +$\bm{\beta}$ and the design matrix $\bm{X}$ which embody our model, +that is $\bm{\tilde{y}}=\bm{X}\bm{\beta}$. + +Thereafter we found the parameters $\bm{\beta}$ by optimizing the means squared error via the so-called cost function +\[ +C(\bm{X},\bm{\beta}) =\frac{1}{n}\sum_{i=0}^{n-1}(y_i-\tilde{y}_i)^2=\mathbb{E}\left[(\bm{y}-\bm{\tilde{y}})^2\right]. +\] + +We can rewrite this as +\[ +\mathbb{E}\left[(\bm{y}-\bm{\tilde{y}})^2\right]=\frac{1}{n}\sum_i(f_i-\mathbb{E}\left[\bm{\tilde{y}}\right])^2+\frac{1}{n}\sum_i(\tilde{y}_i-\mathbb{E}\left[\bm{\tilde{y}}\right])^2+\sigma^2. +\] + +The three terms represent the square of the bias of the learning +method, which can be thought of as the error caused by the simplifying +assumptions built into the method. The second term represents the +variance of the chosen model and finally the last terms is variance of +the error $\bm{\epsilon}$. + +To derive this equation, we need to recall that the variance of $\bm{y}$ and $\bm{\epsilon}$ are both equal to $\sigma^2$. The mean value of $\bm{\epsilon}$ is by definition equal to zero. Furthermore, the function $f$ is not a stochastics variable, idem for $\bm{\tilde{y}}$. +We use a more compact notation in terms of the expectation value +\[ +\mathbb{E}\left[(\bm{y}-\bm{\tilde{y}})^2\right]=\mathbb{E}\left[(\bm{f}+\bm{\epsilon}-\bm{\tilde{y}})^2\right], +\] +and adding and subtracting $\mathbb{E}\left[\bm{\tilde{y}}\right]$ we get +\[ +\mathbb{E}\left[(\bm{y}-\bm{\tilde{y}})^2\right]=\mathbb{E}\left[(\bm{f}+\bm{\epsilon}-\bm{\tilde{y}}+\mathbb{E}\left[\bm{\tilde{y}}\right]-\mathbb{E}\left[\bm{\tilde{y}}\right])^2\right], +\] +which, using the abovementioned expectation values can be rewritten as +\[ +\mathbb{E}\left[(\bm{y}-\bm{\tilde{y}})^2\right]=\mathbb{E}\left[(\bm{y}-\mathbb{E}\left[\bm{\tilde{y}}\right])^2\right]+\mathrm{Var}\left[\bm{\tilde{y}}\right]+\sigma^2, +\] +that is the rewriting in terms of the so-called bias, the variance of the model $\bm{\tilde{y}}$ and the variance of $\bm{\epsilon}$. + + + + + +% !split +\subsection{Example code for Bias-Variance tradeoff} +\begin{minted}[fontsize=\fontsize{9pt}{9pt},linenos=false,mathescape,baselinestretch=1.0,fontfamily=tt,xleftmargin=7mm]{python} +import matplotlib.pyplot as plt +import numpy as np +from sklearn.linear_model import LinearRegression, Ridge, Lasso +from sklearn.preprocessing import PolynomialFeatures +from sklearn.model_selection import train_test_split +from sklearn.pipeline import make_pipeline +from sklearn.utils import resample + +np.random.seed(2018) + +n = 500 +n_boostraps = 100 +degree = 18 # A quite high value, just to show. +noise = 0.1 + +# Make data set. +x = np.linspace(-1, 3, n).reshape(-1, 1) +y = np.exp(-x**2) + 1.5 * np.exp(-(x-2)**2) + np.random.normal(0, 0.1, x.shape) + +# Hold out some test data that is never used in training. +x_train, x_test, y_train, y_test = train_test_split(x, y, test_size=0.2) + +# Combine x transformation and model into one operation. +# Not neccesary, but convenient. +model = make_pipeline(PolynomialFeatures(degree=degree), LinearRegression(fit_intercept=False)) + +# The following (m x n_bootstraps) matrix holds the column vectors y_pred +# for each bootstrap iteration. +y_pred = np.empty((y_test.shape[0], n_boostraps)) +for i in range(n_boostraps): + x_, y_ = resample(x_train, y_train) + + # Evaluate the new model on the same test data each time. + y_pred[:, i] = model.fit(x_, y_).predict(x_test).ravel() + +# Note: Expectations and variances taken w.r.t. different training +# data sets, hence the axis=1. Subsequent means are taken across the test data +# set in order to obtain a total value, but before this we have error/bias/variance +# calculated per data point in the test set. +# Note 2: The use of keepdims=True is important in the calculation of bias as this +# maintains the column vector form. Dropping this yields very unexpected results. +error = np.mean( np.mean((y_test - y_pred)**2, axis=1, keepdims=True) ) +bias = np.mean( (y_test - np.mean(y_pred, axis=1, keepdims=True))**2 ) +variance = np.mean( np.var(y_pred, axis=1, keepdims=True) ) +print('Error:', error) +print('Bias^2:', bias) +print('Var:', variance) +print('{} >= {} + {} = {}'.format(error, bias, variance, bias+variance)) + +plt.plot(x[::5, :], y[::5, :], label='f(x)') +plt.scatter(x_test, y_test, label='Data points') +plt.scatter(x_test, np.mean(y_pred, axis=1), label='Pred') +plt.legend() +plt.show() + +\end{minted} + + +% !split +\subsection{Understanding what happens} +\begin{minted}[fontsize=\fontsize{9pt}{9pt},linenos=false,mathescape,baselinestretch=1.0,fontfamily=tt,xleftmargin=7mm]{python} +import matplotlib.pyplot as plt +import numpy as np +from sklearn.linear_model import LinearRegression, Ridge, Lasso +from sklearn.preprocessing import PolynomialFeatures +from sklearn.model_selection import train_test_split +from sklearn.pipeline import make_pipeline +from sklearn.utils import resample + +np.random.seed(2018) + +n = 40 +n_boostraps = 100 +maxdegree = 14 + + +# Make data set. +x = np.linspace(-3, 3, n).reshape(-1, 1) +y = np.exp(-x**2) + 1.5 * np.exp(-(x-2)**2)+ np.random.normal(0, 0.1, x.shape) +error = np.zeros(maxdegree) +bias = np.zeros(maxdegree) +variance = np.zeros(maxdegree) +polydegree = np.zeros(maxdegree) +x_train, x_test, y_train, y_test = train_test_split(x, y, test_size=0.2) + +for degree in range(maxdegree): + model = make_pipeline(PolynomialFeatures(degree=degree), LinearRegression(fit_intercept=False)) + y_pred = np.empty((y_test.shape[0], n_boostraps)) + for i in range(n_boostraps): + x_, y_ = resample(x_train, y_train) + y_pred[:, i] = model.fit(x_, y_).predict(x_test).ravel() + + polydegree[degree] = degree + error[degree] = np.mean( np.mean((y_test - y_pred)**2, axis=1, keepdims=True) ) + bias[degree] = np.mean( (y_test - np.mean(y_pred, axis=1, keepdims=True))**2 ) + variance[degree] = np.mean( np.var(y_pred, axis=1, keepdims=True) ) + print('Polynomial degree:', degree) + print('Error:', error[degree]) + print('Bias^2:', bias[degree]) + print('Var:', variance[degree]) + print('{} >= {} + {} = {}'.format(error[degree], bias[degree], variance[degree], bias[degree]+variance[degree])) + +plt.plot(polydegree, np.log10(error), label='Error') +plt.plot(polydegree, bias, label='bias') +plt.plot(polydegree, variance, label='Variance') +plt.legend() +plt.show() + + + + +\end{minted} + +% !split +\subsection{Summing up} + + + + +The bias-variance tradeoff summarizes the fundamental tension in +machine learning, particularly supervised learning, between the +complexity of a model and the amount of training data needed to train +it. Since data is often limited, in practice it is often useful to +use a less-complex model with higher bias, that is a model whose asymptotic +performance is worse than another model because it is easier to +train and less sensitive to sampling noise arising from having a +finite-sized training dataset (smaller variance). + + + +The above equations tell us that in +order to minimize the expected test error, we need to select a +statistical learning method that simultaneously achieves low variance +and low bias. Note that variance is inherently a nonnegative quantity, +and squared bias is also nonnegative. Hence, we see that the expected +test MSE can never lie below $Var(\epsilon)$, the irreducible error. + + +What do we mean by the variance and bias of a statistical learning +method? The variance refers to the amount by which our model would change if we +estimated it using a different training data set. Since the training +data are used to fit the statistical learning method, different +training data sets will result in a different estimate. But ideally the +estimate for our model should not vary too much between training +sets. However, if a method has high variance then small changes in +the training data can result in large changes in the model. In general, more +flexible statistical methods have higher variance. + + +% !split +\subsection{Another Example rom Scikit-Learn's Repository} +\begin{minted}[fontsize=\fontsize{9pt}{9pt},linenos=false,mathescape,baselinestretch=1.0,fontfamily=tt,xleftmargin=7mm]{python} +""" +============================ +Underfitting vs. Overfitting +============================ + +This example demonstrates the problems of underfitting and overfitting and +how we can use linear regression with polynomial features to approximate +nonlinear functions. The plot shows the function that we want to approximate, +which is a part of the cosine function. In addition, the samples from the +real function and the approximations of different models are displayed. The +models have polynomial features of different degrees. We can see that a +linear function (polynomial with degree 1) is not sufficient to fit the +training samples. This is called **underfitting**. A polynomial of degree 4 +approximates the true function almost perfectly. However, for higher degrees +the model will **overfit** the training data, i.e. it learns the noise of the +training data. +We evaluate quantitatively **overfitting** / **underfitting** by using +cross-validation. We calculate the mean squared error (MSE) on the validation +set, the higher, the less likely the model generalizes correctly from the +training data. +""" + +print(__doc__) + +import numpy as np +import matplotlib.pyplot as plt +from sklearn.pipeline import Pipeline +from sklearn.preprocessing import PolynomialFeatures +from sklearn.linear_model import LinearRegression +from sklearn.model_selection import cross_val_score + + +def true_fun(X): + return np.cos(1.5 * np.pi * X) + +np.random.seed(0) + +n_samples = 30 +degrees = [1, 4, 15] + +X = np.sort(np.random.rand(n_samples)) +y = true_fun(X) + np.random.randn(n_samples) * 0.1 + +plt.figure(figsize=(14, 5)) +for i in range(len(degrees)): + ax = plt.subplot(1, len(degrees), i + 1) + plt.setp(ax, xticks=(), yticks=()) + + polynomial_features = PolynomialFeatures(degree=degrees[i], + include_bias=False) + linear_regression = LinearRegression() + pipeline = Pipeline([("polynomial_features", polynomial_features), + ("linear_regression", linear_regression)]) + pipeline.fit(X[:, np.newaxis], y) + + # Evaluate the models using crossvalidation + scores = cross_val_score(pipeline, X[:, np.newaxis], y, + scoring="neg_mean_squared_error", cv=10) + + X_test = np.linspace(0, 1, 100) + plt.plot(X_test, pipeline.predict(X_test[:, np.newaxis]), label="Model") + plt.plot(X_test, true_fun(X_test), label="True function") + plt.scatter(X, y, edgecolor='b', s=20, label="Samples") + plt.xlabel("x") + plt.ylabel("y") + plt.xlim((0, 1)) + plt.ylim((-2, 2)) + plt.legend(loc="best") + plt.title("Degree {}\nMSE = {:.2e}(+/- {:.2e})".format( + degrees[i], -scores.mean(), scores.std())) +plt.show() +\end{minted} + + + +% !split +\subsection{The one-dimensional Ising model} + +Let us bring back the Ising model again, but now with an additional +focus on Ridge and Lasso regression as well. We repeat some of the +basic parts of the Ising model and the setup of the training and test +data. The one-dimensional Ising model with nearest neighbor +interaction, no external field and a constant coupling constant $J$ is +given by + +\begin{align} + H = -J \sum_{k}^L s_k s_{k + 1}, +\end{align} +where $s_i \in \{-1, 1\}$ and $s_{N + 1} = s_1$. The number of spins in the system is determined by $L$. For the one-dimensional system there is no phase transition. + +We will look at a system of $L = 40$ spins with a coupling constant of $J = 1$. To get enough training data we will generate 10000 states with their respective energies. + + +\begin{minted}[fontsize=\fontsize{9pt}{9pt},linenos=false,mathescape,baselinestretch=1.0,fontfamily=tt,xleftmargin=7mm]{python} +import numpy as np +import matplotlib.pyplot as plt +from mpl_toolkits.axes_grid1 import make_axes_locatable +import seaborn as sns +import scipy.linalg as scl +from sklearn.model_selection import train_test_split +import sklearn.linear_model as skl +import tqdm +sns.set(color_codes=True) +cmap_args=dict(vmin=-1., vmax=1., cmap='seismic') + +L = 40 +n = int(1e4) + +spins = np.random.choice([-1, 1], size=(n, L)) +J = 1.0 + +energies = np.zeros(n) + +for i in range(n): + energies[i] = - J * np.dot(spins[i], np.roll(spins[i], 1)) +\end{minted} + +A more general form for the one-dimensional Ising model is + +\begin{align} + H = - \sum_j^L \sum_k^L s_j s_k J_{jk}. +\end{align} + +Here we allow for interactions beyond the nearest neighbors and a more +adaptive coupling matrix. This latter expression can be formulated as +a matrix-product on the form +\begin{align} + H = X J, +\end{align} + +where $X_{jk} = s_j s_k$ and $J$ is the matrix consisting of the +elements $-J_{jk}$. This form of writing the energy fits perfectly +with the form utilized in linear regression, viz. +\begin{align} + \bm{y} = \bm{X}\bm{\beta} + \bm{\epsilon}. +\end{align} +We organize the data as we did above +\begin{minted}[fontsize=\fontsize{9pt}{9pt},linenos=false,mathescape,baselinestretch=1.0,fontfamily=tt,xleftmargin=7mm]{python} +X = np.zeros((n, L ** 2)) +for i in range(n): + X[i] = np.outer(spins[i], spins[i]).ravel() +y = energies +X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.96) + +X_train_own = np.concatenate( + (np.ones(len(X_train))[:, np.newaxis], X_train), + axis=1 +) + +X_test_own = np.concatenate( + (np.ones(len(X_test))[:, np.newaxis], X_test), + axis=1 +) +\end{minted} + +We will do all fitting with \textbf{Scikit-Learn}, + +\begin{minted}[fontsize=\fontsize{9pt}{9pt},linenos=false,mathescape,baselinestretch=1.0,fontfamily=tt,xleftmargin=7mm]{python} +clf = skl.LinearRegression().fit(X_train, y_train) +\end{minted} +When extracting the $J$-matrix we make sure to remove the intercept +\begin{minted}[fontsize=\fontsize{9pt}{9pt},linenos=false,mathescape,baselinestretch=1.0,fontfamily=tt,xleftmargin=7mm]{python} +J_sk = clf.coef_.reshape(L, L) +\end{minted} +And then we plot the results +\begin{minted}[fontsize=\fontsize{9pt}{9pt},linenos=false,mathescape,baselinestretch=1.0,fontfamily=tt,xleftmargin=7mm]{python} +fig = plt.figure(figsize=(20, 14)) +im = plt.imshow(J_sk, **cmap_args) +plt.title("LinearRegression from Scikit-learn", fontsize=18) +plt.xticks(fontsize=18) +plt.yticks(fontsize=18) +cb = fig.colorbar(im) +cb.ax.set_yticklabels(cb.ax.get_yticklabels(), fontsize=18) +plt.show() +\end{minted} +The results perfectly with our previous discussion where we used our own code. + +% !split +\subsection{Ridge regression} + +Having explored the ordinary least squares we move on to ridge +regression. In ridge regression we include a \textbf{regularizer}. This +involves a new cost function which leads to a new estimate for the +weights $\bm{\beta}$. This results in a penalized regression problem. The +cost function is given by + +\begin{align} + C(\bm{X}, \bm{\beta}; \lambda) = (\bm{X}\bm{\beta} - \bm{y})^T(\bm{X}\bm{\beta} - \bm{y}) + \lambda \bm{\beta}^T\bm{\beta}. +\end{align} +\begin{minted}[fontsize=\fontsize{9pt}{9pt},linenos=false,mathescape,baselinestretch=1.0,fontfamily=tt,xleftmargin=7mm]{python} +_lambda = 0.1 +clf_ridge = skl.Ridge(alpha=_lambda).fit(X_train, y_train) +J_ridge_sk = clf_ridge.coef_.reshape(L, L) +fig = plt.figure(figsize=(20, 14)) +im = plt.imshow(J_ridge_sk, **cmap_args) +plt.title("Ridge from Scikit-learn", fontsize=18) +plt.xticks(fontsize=18) +plt.yticks(fontsize=18) +cb = fig.colorbar(im) +cb.ax.set_yticklabels(cb.ax.get_yticklabels(), fontsize=18) + +plt.show() +\end{minted} + +% !split +\subsection{LASSO regression} + +In the \textbf{Least Absolute Shrinkage and Selection Operator} (LASSO)-method we get a third cost function. + +\begin{align} + C(\bm{X}, \bm{\beta}; \lambda) = (\bm{X}\bm{\beta} - \bm{y})^T(\bm{X}\bm{\beta} - \bm{y}) + \lambda \sqrt{\bm{\beta}^T\bm{\beta}}. +\end{align} + +Finding the extremal point of this cost function is not so straight-forward as in least squares and ridge. We will therefore rely solely on the function ``Lasso`` from \textbf{Scikit-Learn}. + +\begin{minted}[fontsize=\fontsize{9pt}{9pt},linenos=false,mathescape,baselinestretch=1.0,fontfamily=tt,xleftmargin=7mm]{python} +clf_lasso = skl.Lasso(alpha=_lambda).fit(X_train, y_train) +J_lasso_sk = clf_lasso.coef_.reshape(L, L) +fig = plt.figure(figsize=(20, 14)) +im = plt.imshow(J_lasso_sk, **cmap_args) +plt.title("Lasso from Scikit-learn", fontsize=18) +plt.xticks(fontsize=18) +plt.yticks(fontsize=18) +cb = fig.colorbar(im) +cb.ax.set_yticklabels(cb.ax.get_yticklabels(), fontsize=18) + +plt.show() +\end{minted} + +It is quite striking how LASSO breaks the symmetry of the coupling +constant as opposed to ridge and OLS. We get a sparse solution with +$J_{j, j + 1} = -1$. + + + +% !split +\subsection{Performance as function of the regularization parameter} + +We see how the different models perform for a different set of values for $\lambda$. + + +\begin{minted}[fontsize=\fontsize{9pt}{9pt},linenos=false,mathescape,baselinestretch=1.0,fontfamily=tt,xleftmargin=7mm]{python} +lambdas = np.logspace(-4, 5, 10) + +train_errors = { + "ols_sk": np.zeros(lambdas.size), + "ridge_sk": np.zeros(lambdas.size), + "lasso_sk": np.zeros(lambdas.size) +} + +test_errors = { + "ols_sk": np.zeros(lambdas.size), + "ridge_sk": np.zeros(lambdas.size), + "lasso_sk": np.zeros(lambdas.size) +} + +plot_counter = 1 + +fig = plt.figure(figsize=(32, 54)) + +for i, _lambda in enumerate(tqdm.tqdm(lambdas)): + for key, method in zip( + ["ols_sk", "ridge_sk", "lasso_sk"], + [skl.LinearRegression(), skl.Ridge(alpha=_lambda), skl.Lasso(alpha=_lambda)] + ): + method = method.fit(X_train, y_train) + + train_errors[key][i] = method.score(X_train, y_train) + test_errors[key][i] = method.score(X_test, y_test) + + omega = method.coef_.reshape(L, L) + + plt.subplot(10, 5, plot_counter) + plt.imshow(omega, **cmap_args) + plt.title(r"%s, $\lambda = %.4f$" % (key, _lambda)) + plot_counter += 1 + +plt.show() +\end{minted} + +We see that LASSO reaches a good solution for low +values of $\lambda$, but will "wither" when we increase $\lambda$ too +much. Ridge is more stable over a larger range of values for +$\lambda$, but eventually also fades away. + +% !split +\subsection{Finding the optimal value of $\lambda$} + +To determine which value of $\lambda$ is best we plot the accuracy of +the models when predicting the training and the testing set. We expect +the accuracy of the training set to be quite good, but if the accuracy +of the testing set is much lower this tells us that we might be +subject to an overfit model. The ideal scenario is an accuracy on the +testing set that is close to the accuracy of the training set. + + +\begin{minted}[fontsize=\fontsize{9pt}{9pt},linenos=false,mathescape,baselinestretch=1.0,fontfamily=tt,xleftmargin=7mm]{python} +fig = plt.figure(figsize=(20, 14)) + +colors = { + "ols_sk": "r", + "ridge_sk": "y", + "lasso_sk": "c" +} + +for key in train_errors: + plt.semilogx( + lambdas, + train_errors[key], + colors[key], + label="Train {0}".format(key), + linewidth=4.0 + ) + +for key in test_errors: + plt.semilogx( + lambdas, + test_errors[key], + colors[key] + "--", + label="Test {0}".format(key), + linewidth=4.0 + ) +plt.legend(loc="best", fontsize=18) +plt.xlabel(r"$\lambda$", fontsize=18) +plt.ylabel(r"$R^2$", fontsize=18) +plt.tick_params(labelsize=18) +plt.show() +\end{minted} + +From the above figure we can see that LASSO with $\lambda = 10^{-2}$ +achieves a very good accuracy on the test set. This by far surpasses the +other models for all values of $\lambda$. + + + +% !split +\subsection{Further Exercises} + +\paragraph{Exercise 1.} +We will generate our own dataset for a function $y(x)$ where $x \in [0,1]$ and defined by random numbers computed with the uniform distribution. The function $y$ is a quadratic polynomial in $x$ with added stochastic noise according to the normal distribution $\cal {N}(0,1)$. +The following simple Python instructions define our $x$ and $y$ values (with 100 data points). +\begin{minted}[fontsize=\fontsize{9pt}{9pt},linenos=false,mathescape,baselinestretch=1.0,fontfamily=tt,xleftmargin=7mm]{python} +x = np.random.rand(100,1) +y = 5*x*x+0.1*np.random.randn(100,1) +\end{minted} + +\begin{enumerate} +\item Write your own code (following the examples above) for computing the parametrization of the data set fitting a second-order polynomial. + +\item Use thereafter \textbf{scikit-learn} (see again the examples in the regression slides) and compare with your own code. + +\item Using scikit-learn, compute also the mean square error, a risk metric corresponding to the expected value of the squared (quadratic) error defined as +\end{enumerate} + +\noindent +\[ MSE(\hat{y},\hat{\tilde{y}}) = \frac{1}{n} +\sum_{i=0}^{n-1}(y_i-\tilde{y}_i)^2, +\] +and the $R^2$ score function. +If $\tilde{\hat{y}}_i$ is the predicted value of the $i-th$ sample and $y_i$ is the corresponding true value, then the score $R^2$ is defined as +\[ +R^2(\hat{y}, \tilde{\hat{y}}) = 1 - \frac{\sum_{i=0}^{n - 1} (y_i - \tilde{y}_i)^2}{\sum_{i=0}^{n - 1} (y_i - \bar{y})^2}, +\] +where we have defined the mean value of $\hat{y}$ as +\[ +\bar{y} = \frac{1}{n} \sum_{i=0}^{n - 1} y_i. +\] + +You can use the functionality included in scikit-learn. If you feel +for it, you can use your own program and define functions which +compute the above two functions. Discuss the meaning of these +results. Try also to vary the coefficient in front of the added +stochastic noise term and discuss the quality of the fits. + + + + +\paragraph{Exercise 2, variance of the parameters $\beta$ in linear regression.} +Show that the variance of the parameters $\beta$ in the linear regression method (chapter 3, equation (3.8) of \href{{https://www.springer.com/gp/book/9780387848570}}{Trevor Hastie, Robert Tibshirani, Jerome H. Friedman, The Elements of Statistical Learning, Springer}) is given as + +\[ +\mathrm{Var}(\hat{\beta}) = \left(\hat{X}^T\hat{X}\right)^{-1}\sigma^2, +\] +with +\[ +\sigma^2 = \frac{1}{N-p-1}\sum_{i=1}^{N} (y_i-\tilde{y}_i)^2, +\] +where we have assumed that we fit a function of degree $p-1$ (for example a polynomial in $x$). + + + +\paragraph{Exercise 3.} +This exercise is a continuation of exercise 1. We will +use the same function to generate our data set, still staying with a +simple function $y(x)$ which we want to fit using linear regression, +but now extending the analysis to include the Ridge and the Lasso +regression methods. You can use the code under the Regression as an example on how to use the Ridge and the Lasso methods. + +We will thus again generate our own dataset for a function $y(x)$ where +$x \in [0,1]$ and defined by random numbers computed with the uniform +distribution. The function $y$ is a quadratic polynomial in $x$ with +added stochastic noise according to the normal distribution $\cal{N}(0,1)$. + +The following simple Python instructions define our $x$ and $y$ values (with 100 data points). +\begin{minted}[fontsize=\fontsize{9pt}{9pt},linenos=false,mathescape,baselinestretch=1.0,fontfamily=tt,xleftmargin=7mm]{python} +x = np.random.rand(100,1) +y = 5*x*x+0.1*np.random.randn(100,1) +\end{minted} + +\begin{enumerate} +\item Write your own code for the Ridge method and compute the parametrization for different values of $\lambda$. Compare and analyze your results with those from exercise 1. Study the dependence on $\lambda$ while also varying the strength of the noise in your expression for $y(x)$. + +\item Repeat the above but using the functionality of \textbf{scikit-learn}. Compare your code with the results from \textbf{scikit-learn}. Remember to run with the same random numbers for generating $x$ and $y$. + +\item Our next step is to study the variance of the parameters $\beta_1$ and $\beta_2$ (assuming that we are parametrizing our function with a second-order polynomial. We will use standard linear regression and the Ridge regression. You can now opt for either writing your own function that calculates the variance of these paramaters (recall that this is equal to the diagonal elements of the matrix $(\hat{X}^T\hat{X})+\lambda\hat{I})^{-1}$) or use the functionality of \textbf{scikit-learn} and compute their variances. Discuss the results of these variances as functions + +\item Repeat the previous step but add now the Lasso method. Discuss your results and compare with standard regression and the Ridge regression results. + +\item Try to implement the cross-validation as well. + +\item Finally, using \textbf{scikit-learn} or your own code, compute also the mean square error, a risk metric corresponding to the expected value of the squared (quadratic) error defined as +\end{enumerate} + +\noindent +\[ MSE(\hat{y},\hat{\tilde{y}}) = \frac{1}{n} +\sum_{i=0}^{n-1}(y_i-\tilde{y}_i)^2, +\] +and the $R^2$ score function. +If $\tilde{\hat{y}}_i$ is the predicted value of the $i-th$ sample and $y_i$ is the corresponding true value, then the score $R^2$ is defined as +\[ +R^2(\hat{y}, \tilde{\hat{y}}) = 1 - \frac{\sum_{i=0}^{n - 1} (y_i - \tilde{y}_i)^2}{\sum_{i=0}^{n - 1} (y_i - \bar{y})^2}, +\] +where we have defined the mean value of $\hat{y}$ as +\[ +\bar{y} = \frac{1}{n} \sum_{i=0}^{n - 1} y_i. +\] +Discuss these quantities as functions of the variable $\lambda$ in the Ridge and Lasso regression methods. + +\paragraph{Exercise 4.} +We will study how +to fit polynomials to a specific two-dimensional function called +\href{{http://www.dtic.mil/dtic/tr/fulltext/u2/a081688.pdf}}{Franke's +function}. This +is a function which has been widely used when testing various interpolation and fitting +algorithms. Furthermore, after having established the model and the +method, we will employ resamling techniques such as the cross-validation and/or +the bootstrap methods, in order to perform a proper assessment of our models. + + +The Franke function, which is a weighted sum of four exponentials reads as follows +\begin{align*} +f(x,y) &= \frac{3}{4}\exp{\left(-\frac{(9x-2)^2}{4} - \frac{(9y-2)^2}{4}\right)}+\frac{3}{4}\exp{\left(-\frac{(9x+1)^2}{49}- \frac{(9y+1)}{10}\right)} \\ +&+\frac{1}{2}\exp{\left(-\frac{(9x-7)^2}{4} - \frac{(9y-3)^2}{4}\right)} -\frac{1}{5}\exp{\left(-(9x-4)^2 - (9y-7)^2\right) }. +\end{align*} + +The function will be defined for $x,y\in [0,1]$. Our first step will +be to perform an OLS regression analysis of this function, trying out +a polynomial fit with an $x$ and $y$ dependence of the form $[x, y, +x^2, y^2, xy, \dots]$. We will also include cross-validation and +bootstrap as resampling techniques. As in homeworks 1 and 2, we +can use a uniform distribution to set up the arrays of values for $x$ +and $y$, or as in the example below just a fix values for $x$ and $y$ with a given step size. +In this case we will have two predictors and need to fit a +function (for example a polynomial) of $x$ and $y$. Thereafter we will +repeat much of the same procedure using the the Ridge and +Lasso regression methods, introducing thus a dependence on the bias +(penalty) $\lambda$. + + +The Python fucntion for the Franke function is included here (it performs also a three-dimensional plot of it) +\begin{minted}[fontsize=\fontsize{9pt}{9pt},linenos=false,mathescape,baselinestretch=1.0,fontfamily=tt,xleftmargin=7mm]{python} +from mpl_toolkits.mplot3d import Axes3D +import matplotlib.pyplot as plt +from matplotlib import cm +from matplotlib.ticker import LinearLocator, FormatStrFormatter +import numpy as np +from random import random, seed + +fig = plt.figure() +ax = fig.gca(projection='3d') + +# Make data. +x = np.arange(0, 1, 0.05) +y = np.arange(0, 1, 0.05) +x, y = np.meshgrid(x,y) + + +def FrankeFunction(x,y): + term1 = 0.75*np.exp(-(0.25*(9*x-2)**2) - 0.25*((9*y-2)**2)) + term2 = 0.75*np.exp(-((9*x+1)**2)/49.0 - 0.1*(9*y+1)) + term3 = 0.5*np.exp(-(9*x-7)**2/4.0 - 0.25*((9*y-3)**2)) + term4 = -0.2*np.exp(-(9*x-4)**2 - (9*y-7)**2) + return term1 + term2 + term3 + term4 + + +z = FrankeFunction(x, y) + +# Plot the surface. +surf = ax.plot_surface(x, y, z, cmap=cm.coolwarm, + linewidth=0, antialiased=False) + +# Customize the z axis. +ax.set_zlim(-0.10, 1.40) +ax.zaxis.set_major_locator(LinearLocator(10)) +ax.zaxis.set_major_formatter(FormatStrFormatter('%.02f')) + +# Add a color bar which maps values to colors. +fig.colorbar(surf, shrink=0.5, aspect=5) + +plt.show() + +\end{minted} + + +We will thus again generate our own dataset for a function $\mathrm{FrankeFunction}(x,y)$ where +$x,y \in [0,1]$ could be defined by random numbers computed with the uniform +distribution. The function $f(x,y)$ is the Franke function. You should explore also the addition +an added stochastic noise to this function using the normal distribution $\cal{N}(0,1)$. + +Write your own code (using either a matrix inversion or a singular value decomposition from e.g., \textbf{numpy} ) or use your code from exercises 1 and 3 +and perform a standard least square regression analysis using polynomials in $x$ and $y$ up to fifth order. Find the confidence intervals of the parameters $\beta$ by computing their variances, evaluate the Mean Squared error (MSE) +\[ MSE(\hat{y},\hat{\tilde{y}}) = \frac{1}{n} +\sum_{i=0}^{n-1}(y_i-\tilde{y}_i)^2, +\] +and the $R^2$ score function. +If $\tilde{\hat{y}}_i$ is the predicted value of the $i-th$ sample and $y_i$ is the corresponding true value, then the score $R^2$ is defined as +\[ +R^2(\hat{y}, \tilde{\hat{y}}) = 1 - \frac{\sum_{i=0}^{n - 1} (y_i - \tilde{y}_i)^2}{\sum_{i=0}^{n - 1} (y_i - \bar{y})^2}, +\] +where we have defined the mean value of $\hat{y}$ as +\[ +\bar{y} = \frac{1}{n} \sum_{i=0}^{n - 1} y_i. +\] + +Perform a resampling of the data where you split the data in training data and test data. Implement the $k$-fold cross-validation algorithm and/or the bootstrap algorithm +and evaluate again the MSE and the $R^2$ functions resulting from the test data. Evaluate also the bias and variance of the final models. + + +Write then your own code for the Ridge method, either using matrix +inversion or the singular value decomposition as done for standard OLS. Perform the same analysis as in the +previous exercise (for the same polynomials and include resampling +techniques) but now for different values of $\lambda$. Compare and +analyze your results with those obtained with standard OLS. Study the +dependence on $\lambda$ while also varying eventually the strength of +the noise in your expression for $\mathrm{FrankeFunction}(x,y)$. + +Then perform the same studies but now with Lasso regression. Use the functionalities of +\textbf{scikit-learn}. Give a critical discussion of the three methods and a +judgement of which model fits the data best. + + +% ------------------- end of main content --------------- + +\end{document} + diff --git a/doc/src/Regression/_minted-Regression/default.pygstyle b/doc/src/Regression/_minted-Regression/default.pygstyle new file mode 100644 index 000000000..e69de29bb diff --git a/doc/src/Regression/ipynb-Regression-src.tar.gz b/doc/src/Regression/ipynb-Regression-src.tar.gz new file mode 100644 index 0000000000000000000000000000000000000000..69eb49f3d4dc04ecbf1c5bb13e7c188932f5bac9 GIT binary patch literal 211 zcmb2|=3q$tV;alA{Pz6WEG9#dw#4gkM|UYTq@9%Wt~dQZQla zUDw%@xc3!2zn8y+*X-_m)m#1D-C;fNk8W9e+WEN5@#hj{7v}^S{(IZZfD9hY-@&-K LZq|7Q4F(1PmuP2h literal 0 HcmV?d00001 diff --git a/doc/src/Regression/reveal.js/.gitignore b/doc/src/Regression/reveal.js/.gitignore new file mode 100644 index 000000000..a5df3133d --- /dev/null +++ b/doc/src/Regression/reveal.js/.gitignore @@ -0,0 +1,8 @@ +.DS_Store +.svn +log/*.log +tmp/** +node_modules/ +.sass-cache +css/reveal.min.css +js/reveal.min.js diff --git a/doc/src/Regression/reveal.js/.travis.yml b/doc/src/Regression/reveal.js/.travis.yml new file mode 100644 index 000000000..165d9ae9f --- /dev/null +++ b/doc/src/Regression/reveal.js/.travis.yml @@ -0,0 +1,5 @@ +language: node_js +node_js: + - 0.10 +before_script: + - npm install -g grunt-cli \ No newline at end of file diff --git a/doc/src/Regression/reveal.js/CONTRIBUTING.md b/doc/src/Regression/reveal.js/CONTRIBUTING.md new file mode 100644 index 000000000..c2091e88f --- /dev/null +++ b/doc/src/Regression/reveal.js/CONTRIBUTING.md @@ -0,0 +1,23 @@ +## Contributing + +Please keep the [issue tracker](http://github.com/hakimel/reveal.js/issues) limited to **bug reports**, **feature requests** and **pull requests**. + + +### Personal Support +If you have personal support or setup questions the best place to ask those are [StackOverflow](http://stackoverflow.com/questions/tagged/reveal.js). + + +### Bug Reports +When reporting a bug make sure to include information about which browser and operating system you are on as well as the necessary steps to reproduce the issue. If possible please include a link to a sample presentation where the bug can be tested. + + +### Pull Requests +- Should follow the coding style of the file you work in, most importantly: + - Tabs to indent + - Single-quoted strings +- Should be made towards the **dev branch** +- Should be submitted from a feature/topic branch (not your master) + + +### Plugins +Please do not submit plugins as pull requests. They should be maintained in their own separate repository. More information here: https://github.com/hakimel/reveal.js/wiki/Plugin-Guidelines diff --git a/doc/src/Regression/reveal.js/Gruntfile.js b/doc/src/Regression/reveal.js/Gruntfile.js new file mode 100644 index 000000000..b257e8f32 --- /dev/null +++ b/doc/src/Regression/reveal.js/Gruntfile.js @@ -0,0 +1,140 @@ +/* global module:false */ +module.exports = function(grunt) { + var port = grunt.option('port') || 8000; + // Project configuration + grunt.initConfig({ + pkg: grunt.file.readJSON('package.json'), + meta: { + banner: + '/*!\n' + + ' * reveal.js <%= pkg.version %> (<%= grunt.template.today("yyyy-mm-dd, HH:MM") %>)\n' + + ' * http://lab.hakim.se/reveal-js\n' + + ' * MIT licensed\n' + + ' *\n' + + ' * Copyright (C) 2014 Hakim El Hattab, http://hakim.se\n' + + ' */' + }, + + qunit: { + files: [ 'test/*.html' ] + }, + + uglify: { + options: { + banner: '<%= meta.banner %>\n' + }, + build: { + src: 'js/reveal.js', + dest: 'js/reveal.min.js' + } + }, + + cssmin: { + compress: { + files: { + 'css/reveal.min.css': [ 'css/reveal.css' ] + } + } + }, + + sass: { + main: { + files: { + 'css/theme/darkgray.css': 'css/theme/source/darkgray.scss', + 'css/theme/beigesmall.css': 'css/theme/source/beigesmall.scss', + 'css/theme/cbc.css': 'css/theme/source/cbc.scss', + 'css/theme/default.css': 'css/theme/source/default.scss', + 'css/theme/beige.css': 'css/theme/source/beige.scss', + 'css/theme/night.css': 'css/theme/source/night.scss', + 'css/theme/serif.css': 'css/theme/source/serif.scss', + 'css/theme/simple.css': 'css/theme/source/simple.scss', + 'css/theme/sky.css': 'css/theme/source/sky.scss', + 'css/theme/moon.css': 'css/theme/source/moon.scss', + 'css/theme/solarized.css': 'css/theme/source/solarized.scss', + 'css/theme/blood.css': 'css/theme/source/blood.scss' + } + } + }, + + jshint: { + options: { + curly: false, + eqeqeq: true, + immed: true, + latedef: true, + newcap: true, + noarg: true, + sub: true, + undef: true, + eqnull: true, + browser: true, + expr: true, + globals: { + head: false, + module: false, + console: false, + unescape: false + } + }, + files: [ 'Gruntfile.js', 'js/reveal.js' ] + }, + + connect: { + server: { + options: { + port: port, + base: '.' + } + } + }, + + zip: { + 'reveal-js-presentation.zip': [ + 'index.html', + 'css/**', + 'js/**', + 'lib/**', + 'images/**', + 'plugin/**' + ] + }, + + watch: { + main: { + files: [ 'Gruntfile.js', 'js/reveal.js', 'css/reveal.css' ], + tasks: 'default' + }, + theme: { + files: [ 'css/theme/source/*.scss', 'css/theme/template/*.scss' ], + tasks: 'themes' + } + } + + }); + + // Dependencies + grunt.loadNpmTasks( 'grunt-contrib-qunit' ); + grunt.loadNpmTasks( 'grunt-contrib-jshint' ); + grunt.loadNpmTasks( 'grunt-contrib-cssmin' ); + grunt.loadNpmTasks( 'grunt-contrib-uglify' ); + grunt.loadNpmTasks( 'grunt-contrib-watch' ); + grunt.loadNpmTasks( 'grunt-contrib-sass' ); + grunt.loadNpmTasks( 'grunt-contrib-connect' ); + grunt.loadNpmTasks( 'grunt-zip' ); + + // Default task + grunt.registerTask( 'default', [ 'jshint', 'cssmin', 'uglify', 'qunit' ] ); + + // Theme task + grunt.registerTask( 'themes', [ 'sass' ] ); + + // Package presentation to archive + grunt.registerTask( 'package', [ 'default', 'zip' ] ); + + // Serve presentation locally + grunt.registerTask( 'serve', [ 'connect', 'watch' ] ); + + // Run tests + grunt.registerTask( 'test', [ 'jshint', 'qunit' ] ); + +}; diff --git a/doc/src/Regression/reveal.js/LICENSE b/doc/src/Regression/reveal.js/LICENSE new file mode 100644 index 000000000..09623076f --- /dev/null +++ b/doc/src/Regression/reveal.js/LICENSE @@ -0,0 +1,19 @@ +Copyright (C) 2015 Hakim El Hattab, http://hakim.se + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in +all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN +THE SOFTWARE. \ No newline at end of file diff --git a/doc/src/Regression/reveal.js/README.md b/doc/src/Regression/reveal.js/README.md new file mode 100644 index 000000000..573b19597 --- /dev/null +++ b/doc/src/Regression/reveal.js/README.md @@ -0,0 +1,1052 @@ +# reveal.js [![Build Status](https://travis-ci.org/hakimel/reveal.js.svg?branch=master)](https://travis-ci.org/hakimel/reveal.js) + +A framework for easily creating beautiful presentations using HTML. [Check out the live demo](http://lab.hakim.se/reveal-js/). + +reveal.js comes with a broad range of features including [nested slides](https://github.com/hakimel/reveal.js#markup), [Markdown contents](https://github.com/hakimel/reveal.js#markdown), [PDF export](https://github.com/hakimel/reveal.js#pdf-export), [speaker notes](https://github.com/hakimel/reveal.js#speaker-notes) and a [JavaScript API](https://github.com/hakimel/reveal.js#api). It's best viewed in a modern browser but [fallbacks](https://github.com/hakimel/reveal.js/wiki/Browser-Support) are available to make sure your presentation can still be viewed elsewhere. + + +#### More reading: +- [Installation](#installation): Step-by-step instructions for getting reveal.js running on your computer. +- [Changelog](https://github.com/hakimel/reveal.js/releases): Up-to-date version history. +- [Examples](https://github.com/hakimel/reveal.js/wiki/Example-Presentations): Presentations created with reveal.js, add your own! +- [Browser Support](https://github.com/hakimel/reveal.js/wiki/Browser-Support): Explanation of browser support and fallbacks. +- [Plugins](https://github.com/hakimel/reveal.js/wiki/Plugins,-Tools-and-Hardware): A list of plugins that can be used to extend reveal.js. + +## Online Editor + +Presentations are written using HTML or Markdown but there's also an online editor for those of you who prefer a graphical interface. Give it a try at [http://slides.com](http://slides.com). + + +## Instructions + +### Markup + +Markup hierarchy needs to be ``
`` where the ``
`` represents one slide and can be repeated indefinitely. If you place multiple ``
``'s inside of another ``
`` they will be shown as vertical slides. The first of the vertical slides is the "root" of the others (at the top), and it will be included in the horizontal sequence. For example: + +```html +
+
+
Single Horizontal Slide
+
+
Vertical Slide 1
+
Vertical Slide 2
+
+
+
+``` + +### Markdown + +It's possible to write your slides using Markdown. To enable Markdown, add the ```data-markdown``` attribute to your ```
``` elements and wrap the contents in a ``` +
+``` + +#### External Markdown + +You can write your content as a separate file and have reveal.js load it at runtime. Note the separator arguments which determine how slides are delimited in the external file. The ```data-charset``` attribute is optional and specifies which charset to use when loading the external file. + +When used locally, this feature requires that reveal.js [runs from a local web server](#full-setup). + +```html +
+
+``` + +#### Element Attributes + +Special syntax (in html comment) is available for adding attributes to Markdown elements. This is useful for fragments, amongst other things. + +```html +
+ +
+``` + +#### Slide Attributes + +Special syntax (in html comment) is available for adding attributes to the slide `
` elements generated by your Markdown. + +```html +
+ +
+``` + + +### Configuration + +At the end of your page you need to initialize reveal by running the following code. Note that all config values are optional and will default as specified below. + +```javascript +Reveal.initialize({ + + // Display controls in the bottom right corner + controls: true, + + // Display a presentation progress bar + progress: true, + + // Display the page number of the current slide + slideNumber: false, + + // Push each slide change to the browser history + history: false, + + // Enable keyboard shortcuts for navigation + keyboard: true, + + // Enable the slide overview mode + overview: true, + + // Vertical centering of slides + center: true, + + // Enables touch navigation on devices with touch input + touch: true, + + // Loop the presentation + loop: false, + + // Change the presentation direction to be RTL + rtl: false, + + // Turns fragments on and off globally + fragments: true, + + // Flags if the presentation is running in an embedded mode, + // i.e. contained within a limited portion of the screen + embedded: false, + + // Flags if we should show a help overlay when the questionmark + // key is pressed + help: true, + + // Number of milliseconds between automatically proceeding to the + // next slide, disabled when set to 0, this value can be overwritten + // by using a data-autoslide attribute on your slides + autoSlide: 0, + + // Stop auto-sliding after user input + autoSlideStoppable: true, + + // Enable slide navigation via mouse wheel + mouseWheel: false, + + // Hides the address bar on mobile devices + hideAddressBar: true, + + // Opens links in an iframe preview overlay + previewLinks: false, + + // Transition style + transition: 'default', // none/fade/slide/convex/concave/zoom + + // Transition speed + transitionSpeed: 'default', // default/fast/slow + + // Transition style for full page slide backgrounds + backgroundTransition: 'default', // none/fade/slide/convex/concave/zoom + + // Number of slides away from the current that are visible + viewDistance: 3, + + // Parallax background image + parallaxBackgroundImage: '', // e.g. "'https://s3.amazonaws.com/hakim-static/reveal-js/reveal-parallax-1.jpg'" + + // Parallax background size + parallaxBackgroundSize: '', // CSS syntax, e.g. "2100px 900px" + + // Amount to move parallax background (horizontal and vertical) on slide change + // Number, e.g. 100 + parallaxBackgroundHorizontal: '', + parallaxBackgroundVertical: '' + +}); +``` + + +The configuration can be updated after initialization using the ```configure``` method: + +```javascript +// Turn autoSlide off +Reveal.configure({ autoSlide: 0 }); + +// Start auto-sliding every 5s +Reveal.configure({ autoSlide: 5000 }); +``` + + +### Dependencies + +Reveal.js doesn't _rely_ on any third party scripts to work but a few optional libraries are included by default. These libraries are loaded as dependencies in the order they appear, for example: + +```javascript +Reveal.initialize({ + dependencies: [ + // Cross-browser shim that fully implements classList - https://github.com/eligrey/classList.js/ + { src: 'lib/js/classList.js', condition: function() { return !document.body.classList; } }, + + // Interpret Markdown in
elements + { src: 'plugin/markdown/marked.js', condition: function() { return !!document.querySelector( '[data-markdown]' ); } }, + { src: 'plugin/markdown/markdown.js', condition: function() { return !!document.querySelector( '[data-markdown]' ); } }, + + // Syntax highlight for elements + { src: 'plugin/highlight/highlight.js', async: true, callback: function() { hljs.initHighlightingOnLoad(); } }, + + // Zoom in and out with Alt+click + { src: 'plugin/zoom-js/zoom.js', async: true }, + + // Speaker notes + { src: 'plugin/notes/notes.js', async: true }, + + // Remote control your reveal.js presentation using a touch device + { src: 'plugin/remotes/remotes.js', async: true }, + + // MathJax + { src: 'plugin/math/math.js', async: true } + ] +}); +``` + +You can add your own extensions using the same syntax. The following properties are available for each dependency object: +- **src**: Path to the script to load +- **async**: [optional] Flags if the script should load after reveal.js has started, defaults to false +- **callback**: [optional] Function to execute when the script has loaded +- **condition**: [optional] Function which must return true for the script to be loaded + + +### Ready Event + +A 'ready' event is fired when reveal.js has loaded all non-async dependencies and is ready to start navigating. To check if reveal.js is already 'ready' you can call `Reveal.isReady()`. + +```javascript +Reveal.addEventListener( 'ready', function( event ) { + // event.currentSlide, event.indexh, event.indexv +} ); +``` + + +### Presentation Size + +All presentations have a normal size, that is the resolution at which they are authored. The framework will automatically scale presentations uniformly based on this size to ensure that everything fits on any given display or viewport. + +See below for a list of configuration options related to sizing, including default values: + +```javascript +Reveal.initialize({ + + ... + + // The "normal" size of the presentation, aspect ratio will be preserved + // when the presentation is scaled to fit different resolutions. Can be + // specified using percentage units. + width: 960, + height: 700, + + // Factor of the display size that should remain empty around the content + margin: 0.1, + + // Bounds for smallest/largest possible scale to apply to content + minScale: 0.2, + maxScale: 1.5 + +}); +``` + + +### Auto-sliding + +Presentations can be configured to progress through slides automatically, without any user input. To enable this you will need to tell the framework how many milliseconds it should wait between slides: + +```javascript +// Slide every five seconds +Reveal.configure({ + autoSlide: 5000 +}); +``` +When this is turned on a control element will appear that enables users to pause and resume auto-sliding. Alternatively, sliding can be paused or resumed by pressing »a« on the keyboard. Sliding is paused automatically as soon as the user starts navigating. You can disable these controls by specifying ```autoSlideStoppable: false``` in your reveal.js config. + +You can also override the slide duration for individual slides and fragments by using the ```data-autoslide``` attribute: + +```html +
+

After 2 seconds the first fragment will be shown.

+

After 10 seconds the next fragment will be shown.

+

Now, the fragment is displayed for 2 seconds before the next slide is shown.

+
+``` + +Whenever the auto-slide mode is resumed or paused the ```autoslideresumed``` and ```autoslidepaused``` events are fired. + + +### Keyboard Bindings + +If you're unhappy with any of the default keyboard bindings you can override them using the ```keyboard``` config option: + +```javascript +Reveal.configure({ + keyboard: { + 13: 'next', // go to the next slide when the ENTER key is pressed + 27: function() {}, // do something custom when ESC is pressed + 32: null // don't do anything when SPACE is pressed (i.e. disable a reveal.js default binding) + } +}); +``` + +### Lazy Loading + +When working on presentation with a lot of media or iframe content it's important to load lazily. Lazy loading means that reveal.js will only load content for the few slides nearest to the current slide. The number of slides that are preloaded is determined by the `viewDistance` configuration option. + +To enable lazy loading all you need to do is change your "src" attributes to "data-src" as shown below. This is supported for image, video, audio and iframe elements. Lazy loaded iframes will also unload when the containing slide is no longer visible. + +```html +
+ + + +
+``` + + +### API + +The ``Reveal`` object exposes a JavaScript API for controlling navigation and reading state: + +```javascript +// Navigation +Reveal.slide( indexh, indexv, indexf ); +Reveal.left(); +Reveal.right(); +Reveal.up(); +Reveal.down(); +Reveal.prev(); +Reveal.next(); +Reveal.prevFragment(); +Reveal.nextFragment(); + +// Toggle presentation states, optionally pass true/false to force on/off +Reveal.toggleOverview(); +Reveal.togglePause(); +Reveal.toggleAutoSlide(); + +// Change a config value at runtime +Reveal.configure({ controls: true }); + +// Returns the present configuration options +Reveal.getConfig(); + +// Fetch the current scale of the presentation +Reveal.getScale(); + +// Retrieves the previous and current slide elements +Reveal.getPreviousSlide(); +Reveal.getCurrentSlide(); + +Reveal.getIndices(); // { h: 0, v: 0 } } +Reveal.getProgress(); // 0-1 +Reveal.getTotalSlides(); + +// State checks +Reveal.isFirstSlide(); +Reveal.isLastSlide(); +Reveal.isOverview(); +Reveal.isPaused(); +Reveal.isAutoSliding(); +``` + +### Slide Changed Event + +A 'slidechanged' event is fired each time the slide is changed (regardless of state). The event object holds the index values of the current slide as well as a reference to the previous and current slide HTML nodes. + +Some libraries, like MathJax (see [#226](https://github.com/hakimel/reveal.js/issues/226#issuecomment-10261609)), get confused by the transforms and display states of slides. Often times, this can be fixed by calling their update or render function from this callback. + +```javascript +Reveal.addEventListener( 'slidechanged', function( event ) { + // event.previousSlide, event.currentSlide, event.indexh, event.indexv +} ); +``` + +### Presentation State + +The presentation's current state can be fetched by using the `getState` method. A state object contains all of the information required to put the presentation back as it was when `getState` was first called. Sort of like a snapshot. It's a simple object that can easily be stringified and persisted or sent over the wire. + +```javascript +Reveal.slide( 1 ); +// we're on slide 1 + +var state = Reveal.getState(); + +Reveal.slide( 3 ); +// we're on slide 3 + +Reveal.setState( state ); +// we're back on slide 1 +``` + +### Slide States + +If you set ``data-state="somestate"`` on a slide ``
``, "somestate" will be applied as a class on the document element when that slide is opened. This allows you to apply broad style changes to the page based on the active slide. + +Furthermore you can also listen to these changes in state via JavaScript: + +```javascript +Reveal.addEventListener( 'somestate', function() { + // TODO: Sprinkle magic +}, false ); +``` + +### Slide Backgrounds + +Slides are contained within a limited portion of the screen by default to allow them to fit any display and scale uniformly. You can apply full page backgrounds outside of the slide area by adding a ```data-background``` attribute to your ```
``` elements. Four different types of backgrounds are supported: color, image, video and iframe. Below are a few examples. + +```html +
+

All CSS color formats are supported, like rgba() or hsl().

+
+
+

This slide will have a full-size background image.

+
+
+

This background image will be sized to 100px and repeated.

+
+
+

Video. Multiple sources can be defined using a comma separated list. Video will loop when the data-background-video-loop attribute is provided.

+
+
+

Embeds a web page as a background. Note that the page won't be interactive.

+
+``` + +Backgrounds transition using a fade animation by default. This can be changed to a linear sliding transition by passing ```backgroundTransition: 'slide'``` to the ```Reveal.initialize()``` call. Alternatively you can set ```data-background-transition``` on any section with a background to override that specific transition. + + +### Parallax Background + +If you want to use a parallax scrolling background, set the first two config properties below when initializing reveal.js (the other two are optional). + +```javascript +Reveal.initialize({ + + // Parallax background image + parallaxBackgroundImage: '', // e.g. "https://s3.amazonaws.com/hakim-static/reveal-js/reveal-parallax-1.jpg" + + // Parallax background size + parallaxBackgroundSize: '', // CSS syntax, e.g. "2100px 900px" - currently only pixels are supported (don't use % or auto) + + // Amount of pixels to move the parallax background per slide step, + // a value of 0 disables movement along the given axis + // These are optional, if they aren't specified they'll be calculated automatically + parallaxBackgroundHorizontal: 200, + parallaxBackgroundVertical: 50 + +}); +``` + +Make sure that the background size is much bigger than screen size to allow for some scrolling. [View example](http://lab.hakim.se/reveal-js/?parallaxBackgroundImage=https%3A%2F%2Fs3.amazonaws.com%2Fhakim-static%2Freveal-js%2Freveal-parallax-1.jpg¶llaxBackgroundSize=2100px%20900px). + + + +### Slide Transitions +The global presentation transition is set using the ```transition``` config value. You can override the global transition for a specific slide by using the ```data-transition``` attribute: + +```html +
+

This slide will override the presentation transition and zoom!

+
+ +
+

Choose from three transition speeds: default, fast or slow!

+
+``` + +You can also use different in and out transitions for the same slide: + +```html +
+ The train goes on … +
+
+ and on … +
+
+ and stops. +
+
+ (Passengers entering and leaving) +
+
+ And it starts again. +
+``` + + +Note that this does not work with the page and cube transitions. + + +### Internal links + +It's easy to link between slides. The first example below targets the index of another slide whereas the second targets a slide with an ID attribute (```
```): + +```html +Link +Link +``` + +You can also add relative navigation links, similar to the built in reveal.js controls, by appending one of the following classes on any element. Note that each element is automatically given an ```enabled``` class when it's a valid navigation route based on the current slide. + +```html + + + + + + +``` + + +### Fragments +Fragments are used to highlight individual elements on a slide. Every element with the class ```fragment``` will be stepped through before moving on to the next slide. Here's an example: http://lab.hakim.se/reveal-js/#/fragments + +The default fragment style is to start out invisible and fade in. This style can be changed by appending a different class to the fragment: + +```html +
+

grow

+

shrink

+

fade-out

+

visible only once

+

blue only once

+

highlight-red

+

highlight-green

+

highlight-blue

+
+``` + +Multiple fragments can be applied to the same element sequentially by wrapping it, this will fade in the text on the first step and fade it back out on the second. + +```html +
+ + I'll fade in, then out + +
+``` + +The display order of fragments can be controlled using the ```data-fragment-index``` attribute. + +```html +
+

Appears last

+

Appears first

+

Appears second

+
+``` + +### Fragment events + +When a slide fragment is either shown or hidden reveal.js will dispatch an event. + +Some libraries, like MathJax (see #505), get confused by the initially hidden fragment elements. Often times this can be fixed by calling their update or render function from this callback. + +```javascript +Reveal.addEventListener( 'fragmentshown', function( event ) { + // event.fragment = the fragment DOM element +} ); +Reveal.addEventListener( 'fragmenthidden', function( event ) { + // event.fragment = the fragment DOM element +} ); +``` + +### Code syntax highlighting + +By default, Reveal is configured with [highlight.js](http://softwaremaniacs.org/soft/highlight/en/) for code syntax highlighting. Below is an example with clojure code that will be syntax highlighted. When the `data-trim` attribute is present surrounding whitespace is automatically removed. + +```html +
+ 

+(def lazy-fib
+  (concat
+   [0 1]
+   ((fn rfib [a b]
+        (lazy-cons (+ a b) (rfib b (+ a b)))) 0 1)))
+
+
+``` + +### Slide number +If you would like to display the page number of the current slide you can do so using the ```slideNumber``` configuration value. + +```javascript +// Shows the slide number using default formatting +Reveal.configure({ slideNumber: true }); + +// Slide number formatting can be configured using these variables: +// h: current slide's horizontal index +// v: current slide's vertical index +// c: current slide index (flattened) +// t: total number of slides (flattened) +Reveal.configure({ slideNumber: 'c / t' }); + +``` + + +### Overview mode + +Press "Esc" or "o" keys to toggle the overview mode on and off. While you're in this mode, you can still navigate between slides, +as if you were at 1,000 feet above your presentation. The overview mode comes with a few API hooks: + +```javascript +Reveal.addEventListener( 'overviewshown', function( event ) { /* ... */ } ); +Reveal.addEventListener( 'overviewhidden', function( event ) { /* ... */ } ); + +// Toggle the overview mode programmatically +Reveal.toggleOverview(); +``` + +### Fullscreen mode +Just press »F« on your keyboard to show your presentation in fullscreen mode. Press the »ESC« key to exit fullscreen mode. + + +### Embedded media +Embedded HTML5 `

!~XH3U9QOpqReWi>D=+N~qaJ&)TFnBTA%YvcAwqcBNC} zNkoi7mumLWj$UhhYyY+h(}F}ymx>SVd_AUnLnTDvDsQy`(No}gLrxf|>)IFcDm@Dm z6iYCMPRU)B_a~Z(wuOId`&D+VmOQT&fRMls6@wSJ#G@=JZR%gsnmtSKDztjE_V%2@ znbG5o4=G29sNlzfdad}r+vOW!*{+FC>fHxsPVUmekKpzrlte9LKwGLH>P=~=%*B{P zen1BikmP*CECe?$I7Ta!5Z1{|B);(v7vsRnt}^GkfkNehl}TL6`IL3%&+hIXgnQv@ z1a1&s2Q4=kG{p)7JW~V(n<(@4yJaQiD#&0_#D%46vT1%0B8PMLlz1jPTAYu5Q8+uw z`%1AbMW>+U;+(i|Tv>QcFQpaO`LW~%x;)=q-UoZzD8lCw{w^#n53^5k9WN2!fo_}A z0Y(eP*Td#~c-E@`u4DCbhT#@n>Vb zjO|h(Wy9IGk6ofq$7@JAxGzWG5iNMN(<;0mI8;wY4C}w=a{&zA(9FiYB|Kf{g|SWp z!zMI(v3S<<*@I+{G<2Gc1bxjJsaRe4MnaimtHJh315J0AFR{e!kWoj$?|92V4XOd$?-JboUx1@s&w$uh?UH)sS2qA$ue8 zA_i3(z!_v539B_#RL&%dn-l9)*em9gME9bHkHpv1md3}lH|0llIj-B)p{J3kVEEf2 zT-u`ncEk}k!h|=nG}R2teJ0B*D)JgQ>|t`_0ll*Ybg|m)AH=?xub^>x<72Uv3wq#J z0y`xAD^_UF^isap0qnGMnT#S^n?H&Crob@UTmpv$=E@IN9OPgST2tBPt$un!bp^tTD0ujYSJqk zEfFVkxyUwipcKD(^kg|wZ#C;(%pU@l8Y6(-rL=ROi{WsxHHuOd^!qCoK8GwwUx>qi zOt$g7zj;arHkmrn=hHq~!2wuxq!dG%oim*q#Icf09)ILs&P8*?zGXx%PJ0BBdSCvs z+lM|so}%Ky6~9z10#if>ZYpVkeo7ftKeO9k^x*tz(=yrGh+J_S56~ijr)?^sp~1Ry z*1-hYFYl1P_Y;PMh9*(;19d|*0zX?UMDWwp9o4*w4j;b^8CiOXfy3miuvjjm7)=gJ zn6r+k?g-(n51&#lA1hk_4nd>jf&OrGwgm$3l=X^LnWy%@bAh-9w#(S1{z>JyU_Y$eaj8GMkm+gx6Z?1&l)#)yu_jPD;7epOE7C(ZEwt>syb&OuQD zaJ$okdU=`vEH^+ZjZ3`>Qa%mszfCNkm??o02D7sy^#Zn{&>G$N;V}CS9T7UBQ9D(r zTSnm38AYukH|A0ya?_x!$%$4Nu{)f-A0{j&BmXzRE^OnAz1N3u=HS{Et2fmuY&ftT zzj2{WULtD(F$7RRQQTqsAtPqquP%+;xeknVX&h-Cxoiz`N8UzV z@rAE=#=ypHByY7Eetf>aEqWj3j{~Ba5SKlLO_!!p=`dcrXE>yg3d|NS^Z3XLrh57_# zqCbZ9Qb94dt=}AYr?u7)`~wGQ3idL1Btr780p@BE!N$}pDXElbGUC~f)8Z9mYDpo& z2N^DpK7kc*kohWHuj4xM`4lP4q*@gzEFAYzd=~rnZdL1CQ_B`6$Q$yLh}u1NBA2On zAfm`!GVAM;s5AVqmT({>F>_jZ((kRT= zyc*ZU=IQgwTif;7Xbcb}a-2+#nqH2BJ!lMR6IlV9m2&y1JKbQ|O)kS@R`7>_BHYD7 zqJavng1>wPOJBv2nTSkH!E}VJZ6Y+vxD$$ezm8`c1)n_kbL>RdY8Ec7JRL;k@TLoG z6}<8^!niSyp5b59f8zE%TR>lHK2|{Oscl|0Yx=k?;^VTT`iN96u8vL;6*5DGqcAs< zKN3wl6kH>rE9y4*hCG&n=KF>daiOpsxj1);5+yRqhlLwrI2%*f%`o!Cp9h#lSH zLI+)ZQhanq(G_@~yk^cCy@@suW2XE+AwM{?ShwL!4&oUYY3W07h4?)x89>4*N?XAL$;B73+m~@O=(A;Qzn+i-9HMpf>?V&IMZkxFWpL(81pty7N8Bm zT&z0Sbl7HvfLb!>vCV$)2S?P$f%v|IJd|)a4h{%Q#G~~=3+!$5amUlu#Jwg%2ne@& zQ$4h&>8ihcBFr4uV%&I~#cI2j^(Ce)wb%f}`z`$N8dMw8_T}-(yZi7f^ZE5F5$me`|SX` z;=`JMC|v@%{DEJ|KFdKH;cV)!SgWK%hZy%RZ*yYuP@~CJF|1x5)AEKfY;*k2Mligb zg$E74S2M&lCNNM=7*1>Ek?#DpU5_Iv$PRb1sh~>BBW&U+4}@iq+J4JGl)%hH1Vg z-DMQ}qD>8Uj+7VjX2vvanUaaPcS6h#9Qzkc=1u!kR;FV_{-H@qmEJhNK6aVeLVyJB zZ%|f$4l^`K> zx1oHq#p28U*gn8FQ{gVo=H$)sLYFRv8=(VF5{{kE4Re%sldE;cgrgqT>Pi^!omtIl(~x?XvC=HbG4F-0WMY4oR*u z)`z25WIJK60SBhxPVZ5j&6Udf-aU3ffy4*t*XL#+c##!uHBL2BWEPD8dugj#_3z(a7UYAv5Xj>*%@%QML93LG zq?bvdk%7gMiIXE*2qDM_`iP(@Q%;ctv%1y-P%Iw-uE>j5MO=e03_$@RG*i6}rh|-} zhF=%>sPwItA=AHf?yU?YcsYB)5S6zx#w?;hz=TujF9+QQ6W zw2V+Om=_MuV9K$8pEBaE1Z?DRO->Hd1mlfj+_xok<&28;F&bnxhI6wrm_(n}& zT`OG5JWA}N-gnxI_^`V(?bkbHH~JoMGteSf1zb$Ch7PcGN+ORKIBI0g7gCtmueHb5oMM@#Pj3NizNx!Ftg=vqJp_Qwfg zO4g9D`+rKUg<`JU>ceO4PcjL3ox?X4?dzsY#?PI7FJRs^FxyhlJP*=qYJVy5?Y-V5 zstGYZVAU|a5&QzM(sG*E%(i2=k+8r)#pCs|(Dn2U)||OEt<#j134l1j`&t};d!Jvd z&nn$hctu{b7u*l65ZSvAvPrhwiib-M8k~Kk3nily0zTS=yv^ih;j)l4aI$5lorW%C zwf#wu@Jy*zwdsn{kC7cCzcOwGarhHp>N`Av&C{>+WyswI%gqt}Sg#J;2=XhoR29=@ z1}L0{52nx_U2ZjcLK-^@QWhf$mUj%r*b^hqbQf5a4ULGze#&n3mqa0SoLb`=E%RS+ zLPr6=U@t)=YTs55swAN%nweb(1O(mrl7hEvLiumo%5uv&VirN8%jaNCZEm`kmtdmvybYGH3B)D8J2gt-?c{ZlwZtv|tL)&i)c_S+d@OPUk zA_uLl`+Jp%hFgROnLBXxL2MH$nNC@z84D;%pTqigHqt&VZ%%8_dXNHqeZ5#83G{d^ znJ=`3H>w6g{m?b8?UxE9iQkRF_Wt#myK55~-?@6F6Y7vsEVNRq@W=J1#Q4b|53~+z zhD|%fx${>BBnHKR;410HG*l#_*uUSmkHT+dIq$XQon|Cep6Mo*2`zG+_?GOF)R&12 zdl7$xR;nG{hI?h2Kk?M?6ydGMIfAf#nWGu(%N~wpYXXkrDk0R1PjvpoHOB_wVwX3%CXHQaRgkoJtm(wXhqR(K9 zUZZOx;GEzf!TSWwu^O@1i&zY1%1iodM(4dq>|^86X%p3saWQwb>UXq^R8rBgr6puy z#TYDUpq=jD341^`zfb$GOvlp?ySK?y|H?Bn;gOb+6+kewuy?N-J&Bzn>&99i>b4tx zpNM)=1AQ#0kqw{w#WKk!*AWx?1sTnV5TdyT(8m^hR<&9`jgSEa9v;KOauY91Wd-^F zp%%oPL*@@p3@!lz`=EAN=a{40zt&Nl);Cu5Y|lOAnozA`bp1T*V3(fUhoh;HR~YnJ zS9?g;U%L$DU^ww&ZWR20BnN$tjxYCMxDAbU(6)SN>3_VQ4_cUJw{Ncw7WkLq;41N) zmZ+!vM5?B|5{6>(xAlB`65j(hJIXB}c>C2kgFiK-471{|Yt}hG0qHdFHk)f!nGtni z=Qx79A>{Cm6R}+-PbEcOmYepDqmOwqSq=jYB(6$A#LsudzM?Z^b1dCPqkxNl+>P!X zX6G5!G5h=b&0XZ#!(QaB8=&+rBWOX~z`|{NU1v+V_L(LMFQe*39=N*BXh{)E%>`di zC9SR`fbRHfZcQA6X%BKb>#!4YEziI%)TYR4-kdBQ3hv*LsMA~DvAP`X)4793x58O3 zd69JwYF-t5*jZWUTE{fZP+GjxvXN4{F%AJ%xIb7}NqW>dc?V8@ZGk)Li;`m?u z?b8`oHzvHI4Fy+aTq{?_5+WebdGf+za+Ml zFeNsEn+5fJ8#iGbDkD&xV9s4b>&ixpjh`StTIJ;{L2veM(fc7#{n8e|O~pi6Gf-R61p=PL{$GsULy#y@cm?3GZQHhO+w%r* zY}>YN+qUt>wr$%pnIx5}EV4-Nx~qHJb#LACpAVy7vcLrd(oRSd-db5S)vZ3rjy;yn zno#|qbM5Wby-Km3zDxBbnp?JV47V%HVhF&g%Q{xWOeP0_C;o@Y< zgr7=119Bpw3b_Ku^=WZ>X&U|VI#(F$msqdQV(G0@GE&ord&-d^V8>Ze4h(Zpe_kv% zk2_*kQYZ_zt*ZHZgCT`bo?5kJ0~ux_a(ws?xtcd^EMYJ3hZBROH?M>0Df@(Y6&zW- zWT$Rev@HwNT*=k0p_zx}%8ooA_4!5w6)nmfk<^o$-R_|r#uMaqE)q#j=CQNGdh$8o zUycFAr4@!fU~fr(0bB)?QJhlSzu82O%(tt^#B+*Hbt$_jmny{6j>SH?faHM%w~5~d zS6KITP^Sa^EiYTZbLB8tM%boAmclRtv-(aW*7n~*9tAka^IA+Fq2A=<5V$K8)T5>6 zb1#+$e)KI3Nb$g=xjf{@+Sg`*KEQ7STZYVi#~f5+-)@^P=X?p$_L%5(2cN9sARbyI zDzojC|K2G`d=0FYp3Vqv)P8o&Dk^defK>O4a@;!ANx~+)KMFk4@+lMoHQ_d}*bW zLLpy+X}_@pb)whMpLU2RiXiqAz)r&HBmYRSYY5qRcg5{)(D|FPp7nwVZl$%|Rccu{b26h7J5LVoer-J_THF)OS^1m)SRsQpomsPvUUI&;!Jy_I` zL?u9sf6M7Pi-l@yaw*q^p3OZU*Qmd6req7XW}mj%zQ5C%Truw%!vZgi|a*?Xo$ajx#$N)sj0A|N*U}BRKYn>6#PSzF5SrHS=Zt2&Q zb@A@cv}PoJi!bQ`Ry`1`TOT`EK>FD&b%SY0@MEzr5JD|@<%L^yM*bM_KKd(9Lcbl; zA)od=?$Abf0 zB!{CKFQ=wo-OIqD_oeX9g430xvyDoze#rKGj4JhsF3cngeW@k1gB`!($HWU8juyt! zTJ*yyhH{1s@oyxO)OKtJGP=#lidTfRSG!*3v|B6}%VjzmmPK+>C9g>4oYP9%IcC&x z`0j5knjNc7FzjV9x7{e3Xo9*iHkC##RZJWId;msf7TekhK{_%Y2G>+`wi@YBa(y?E!c`J)7qdsMG82=Pcv zZMs2Z5WLm9fhNy7abFWUACC}cUvz}bj3DdFQ%>J(RRo)A`PQyPUUEC!kzQINO19ex zo%u`YA^!~G_+a@+jQ^Xhm2YcFkMFJ6sn(GL6@}%cfcAU=#wq9E?MV zrH$Q8aYYG55-s#?)t^&VpEu;Rl6Uq3D;>AhQqoe{`3B72O$0Dg%rkr2Wpjq>V|v>n zGv4fe&-q*CAPsMWDY#1mo{wJqf$+&6Bsc&2$B67$iYNeQyiiJs#5!ylswwK@jr4-5 zJs~J?p;Njg#1=U7Bp(&_dID^b>>Z>nk;fSrPRLvHRH>>|2H+5(Ij=t2iWJ6+H4r23 z3cw@E0#bRLnZS-yN;v*7ZR^}pZeOFOZS+4Tyz_jCVxv#JtRkV*$>$=c1D@G#b<2=u zmZ%XvS547+&TsFmyE*MeXI8SKLYP`V!4)OF4sFxky~&rLT-sV6b0ZI?5(Y~y!34h3 zOOkfphlf}RtF-Qu0~D$PKvGh^Oco`IQ*Vj3KW{6{D0MO3WH<3g;BXwt2g_+QGW$K& zwt->}g0LCBY%|t9N_I_AqL&#Btjdy14tVELZMs&SQ!$?n=pN*T}V22qR zzx1Sdi|1t#7FN~DV52_$J9%1vrP6^Fp!65+c-=~9pG1ln;A|Ta-Ux?(ro0lK;j)`o zB^b0eBa4e$QP&>McNy5f_46YZ2{?AL9`q|+FU^ufonAWJw)f`ZMCVX(x7bl!*$9!5 zZ)#5_+&b!r!;0Tfg{!H3m_Lv=7&@8;g*Zy}rmV0nAC^5d(q{|4OwwoG5{9MAo>8zo z^m-^fP#4y;kSRK6=wmVMZ-P3jp(-z;BnPidqPYPSWNvIik8Z4ZVH4|!E^~)7urXQo zTUU4e`ntv=X7wNzZX!0@$Up(hj&kGIyq}@pLN-^FE5jS?0-y-cbvqQ}MKuZNO&jQH zq;&1tptM{Gd})Ayf`7OVPm5M?Sfg|;ayY4>U6ZFGF<|FDG2YS%H?u)gQ;%IvB2~>Z zQPQpI>W~O75WZg}e>Z!GR^7cBadq1!Y2|QS9#XSZ=5}qVsSt!B@w~GWT`>O_?C%S? ztOzVM|CCXi`c*~bJGK&5rM72#YCM-=zq0*nc)r7DB&hzG=G#({yaK8vUyCd~9$VUB z{79)1LdWD6dBRW+O!(JR(d5SDYg_WZY9sy;>ax_U(}{x0g?F zy_kZTu0->cX7PSq)I!RTS$SZgN8KaEisWj*@AuT8eTFRwm383HsXAJYpvkZ}U&#ks zl*i5EiN>Gm%H{cuSx>$NjocdB*#u4fI}z}s<8R2<={|cxxo+*@NZ4C2+%FCzkvEBb-uuc>Hfaq>I50BjMkeb0T{o z=c(ziwd-D}5>)|EkS#T4~A1UApXyR%J*a#6x`s;|d&df^&irpf3 zz0n{>PZNP&XMq)acG4*4IV2E+Ns&r(Vw|>jG7-aKo#-)Df(H!hSkn^27IOp7FtGE! zC}cDpIGZ@<_j8Dt-)=R-n$PuY_sX%!iPA7dW?i$eNOGnhHBfTsS9IeLOPbxNUCp+R z%2C?b4cE?X$26Mjce~qrA+mII@S9fnb`$oChHd-WtwUP``lf+_Egen_?Kxgt9~l0q ztGc)&L|-{`Zl4iK);~n~l(2##ZqZ|K{L3iJ1?)9B=|{g+Lv7syX2Nu6{Q?t!HwRMY zg37gJaA7b+Ydfv}p)F@{jC(?TkP$Xd(Ar4m%&v@sm!01pYBhs7ZB}WN*^v#adGqTA zf$&>xKtRv}!U4hA;*-MT@2lyD$czR10HaKM_!HzY1|xG=AM^7m_f(194j@!o{RC4z zqJf}{Le<8b2MbwSupaV0wQ4`>&XWlKisyVo?PynJ)-w()mpr`c#U3Ign9$t`vJAd* zvG)Q{QC2i#9}TEqE}l{x#>?zQb}O`<-)%-Ww%RTA2Sl+ zMnlbP=+b89swB`={{RZvyI^RIBgnE-Te64YIsD1sev0OaeE%(=+Yjryza*4jc4Lf! zuZ~hsVB*>mY-!XK64qRzkAY2m4i#H1oK&dg<3TMyEr&jW)F+(OE;)r)t4OMGp(5(e zEOB!jB+EPv=Ph1Eb(vMs)`nRRG~X18$?CsvKJKOW7$RPrL1setva7~UlVdT0LtA)Q_C(;zQxAp1bL*l*MGay}LURm;> zEi%NqROLqK!BHz?Ajs=5gFsR?p72zolt>{FBMui0t;%OObu2tjP2U2^_RQiK3Oc3C z&cSoCY)S?h?8Kn=AX}w2NOzF!3*gi0BV~CKKk?oC-C%_fQ?Vyk8JC_mzr!&#H=7kD z1raQTxwUgYV?XI9XkJ6xQ&m~>Rl)(m#??{hT_xs>Ehgte^s*tCa$uIStD|B>T!>Ai z`by&&IN99GEEqA{Xsw?r$rwckcIvW#f0O4=gUxH+R7?>C&1rLjaId9&AQ==hZ&MaL zmhD71?ZsqOVdmw9x@s)r+E|R$%=AM#D$AZfL~@BBB=;JSpg*ze?3;C8WUwK73yPxt zbb4Q^7KTta3ldkgWzy2Z9e$;yX4oHh4!g#ZTuvfi~(n9Dh{=5Y?;Zujv9_axJ?_3va%_G%Ak0oB{0Ny6AsbO7HZxRT*g}Nwov2A zG`vH@U;kifu>59j9!w07bF@cT`g>sI_HnY|4OK0UU^I2Nm5#vW;2w>TZ!alw*6S|+u-`pPSdoxHAKt0`ckjS*fASk zEK#vsDHp<3vdj`~8tN9oglWkF-tqjjLZ2K_eYaizYsJ4;oCd>`6F%RZPvC2~)w*0_ zxgQaoV|2YRZ@;mc09T`}Y+FdWvcaxKdhGkJA@dWq{ECpyI2{)(2FX}qO6>5VW(e8^ zRo@_e#9&s%V1y`|5{Biy={_zp7rC^=q3I&JuLvUkuEVM&aIL?qt$K*qE-4J`ZI<=O zcz<^(iPbQ&;?!JFGx2xv^c3ReBW>>B^YwF)Hyh%G%V4QSHk_J#%?j~n+&qo5NT*Zn z-(dKf1jM{X#OnO4=GD;)RW=1&Vw4fMtogcPi`XZ=MbSw9R|rHo-;`DdPp6lvy$F~)s~pUHRXn%YH&hk z+OITPQdunTv-WYmmvKJvQiqQNcAZ8ycoeLh%m9O4GbY=@x#p6{Yi`mTDMY3(~HsPBbK^*8rW z3|h!&?sx-rsi&dD#Kf@ZNy98g3+hw>znyl?I{t_#` z^3AX{xe)2Egw`gTb&z^70^mIuT}lz6UR@AiQCM=@4K7U_Hp_0|Qx(H##h zZrYvctoXI%S~J~_I?b{Wc=`9orTCxwwYdfepz~SLRJxe_Sk;K`8Aa(rbE)1>-fL8h zL$OLd3C0H9;38NF$9c}o@=tp-hTAqL_z0-UN~8l5ncEF*)pJav@GmoS5v~lKUa}DT z<+_n1%izzU)uin+ax^oFPQz%vo@lgP>!$~QkOT9F7%CAU&0>65fb8On(Xcp1oNb2E z@DI(yM8M-UyKFD&6g3svJ~IVY)o&e2T+PmtF)LGAg;OD1lAj7=B?CcZZ;c=*n#mA) za?To#)|u$87CU#EIKnrEd<)dk#F{vHQ~bOZ>Xg)gib@VAK}sG&`i(Tj>tqffZ5X50 zrczl7TJ0CRU2nz*C@Re|qe-^!?~k3BCua=SZv8;LSB^KGFq`LXVmE$Db7-Cb24QEkm1V~}l^V07XDO0<1s@RGG zL%R2PzLIS6a>cg$v1Rzp<1BjOlf4)I+aPq;XS1sGutIda20;HL3E%A8}bd?unNR;wNn#%o5uIGJke+fi!jT z_U*XkTtJkX7VsSoS^BDMQFVWH4~}XAzLR%&b!#I#}gLZWO?*jH|uO|=_L;w9+)&{DDDjbaT+4kbr29&t1E66d{kfB;y= zo)408%17c}U1m^UF_9kpZwYn5E?9YkCOT2-Zo&IjC4}L9N4^~a*9TqwB?G_k`i{$C zevgcn^{KcKMBMOIvxb{2}dJJ@_?~E*^RHVv6=QnlZ(J*5aqG?`ijggN2n$I{Tc5`R$2J+eC;_8x(Rt!J1@tU(DI~MD zAq*op{3~;Wy+q?30gW;Fe1fEz(Ybf~pYg-Q%~*d{;sXz~2-lxaM@!WuEk5~*UJHXM z!m=EsHkow5C-Pnz>YG5=s4$s(oc2sx?uh#<`8$`mVR-^lkMS?a3j!z=9YEqQA>wWU zQU%W&A$=safU7;$h)!B}DjqN*D*x)aU5WLc&l7#KU2pSO6{P9bTrK;LmL0?vx&3Kr z0Y@S>wsoNU`*9D~Vek0*ay`=bYh3tA$dZ(6(P1#|#rhOLk-_0FCiRl`qvtW{u7+=@ zl`*>G${3f0E^kGJ4)uj5$H5#9F${EF@3TgOOyK9V8Ifz-jvu812>rl&TVa&MvIN*M zPy4p+7~cV`aB&2)-Ny^?7|aCx$)1i+jkC81+)T86&jE9cLFLitb!Vy*PlUKqM(cmA zN!yhTKRA|;G9vkUoDpxH~dKpUH85*iyZS1xd*K|AG(f2WT?4rjqc^ zEv(6_|7y1W{TWc$MibFRN7iZi4n9OOho(xGxCe4+mA#k2nxT=Ys~3|;SO8#FapjbS zz59bt0ISiy4Ip@JLhpRPAIVAEim)$xBy6lyjTF@}483Wf3MR024crNdctk%++jmK) zx@)EL38nvfCySHXWO9nqr&oH}3tv-4p+JsuYQV+vQ8q*LSHU~vv;0fZwPj>HsQ#!} z7`xYvjW?-5XC2>}$d%g4OVe323j5=qxv!ROD2MgrqlX>6Om{oJU0R+Gfv0R(#}(V6 z4|JOo9`|Ffu;z<5o`>kLUC?zfc{q48oX>Xcc`UwI@?B`LGXnp9CDK2uGYA0hq&w+( z@^Z2V0gaP+wg!)8b{FySsyreyh+=A+Wvyg{ku%ECUOkI%5lA9p7f*Cg)y|HyaiM*34wKo|~iz!+JB85C-Sh6dX zNMa#DLyt!Xj)Z5QXpc8@9}t^=3E#Y+Op?s**;_H<%*l5eN^D_) zZadZAbZpV)w05Bndb8OsX@N;8P?V#kcs$s3e#45Xput z>}}@)Sy?cvZOex6yriPlE}hG((~ST%ZIG}~3G-AJUfXLlzZ=jYBEy|v`oXC|kg-73 zOOIeGZDlYcX>rj}+j<+E3-P}3;?%;~-hYleCOHtAW@-jI8q4nT@s|A{z3(<%6X26+ zmu3Z(4ixd0^t_r-?a}hoR=U_h`n-r9zG+_Q@j!oi^=|vVQ&B48WPg{#a>y)F6b`PxbS3WroaL!j@^-{%Z`vhp`ov=^?Cd`SA0 z?g6S@>dtfMvE2W`vm4_T#zE+4!pc?7Kes%`ZW)8KduyPr_1#5_ZQUGXFB#&QF1DN$ z(>p|({Ny@=t@dkz^%)xn&iu|%@=WS;cMy-gzRxA(5pGxrqL5$D;d z8TSFXlk}FVY>^3u2Is(NcyR0&J_Sz4l}A1ByEYMV>xyBz>KWd`oEis9T|W0!#k*YwaI3P_ z;pYY88#An{_F-;*ZJ6K+*vFkIisc2-r;Jc^H6`_~lz#|>;ZH7ywikru>E%B?4J6Ri z+p&L6XiAGK&FwxG&wcWV(I}#^rD5r+4xheM)D7sjfdzV1Tw81wB`GHKvxIa$Q~K9J zk*H?OFPg(Peon}*#NDM{Y(TZk%NY=JnwSz+33fQ87rwQHO^EHU)V*X_cr zbriI^8tjSmqE9Xt4kt(4+-jHcfK{@0a;QVonjh-)zgt_F;1w_+h$x|#=mDcywE$Am z>-y^D>9UgK_+l#oa=yvi#NRkU4S9%c%5Eo`GwpFXBAXt8#CP~}2q^2LfDWVhmr>l| zR=JYYGxB8``v_Ib_qsnHKSKlajHy)8 z&@7{*m4YgL;bOSyQ)t}}llP*b@)hY_>4H+SO}dkL<|02!(`Cdgv0nuTNFYJyT2zU` zL*{lRR90d{1E^Pjsfv=I_5o1pC4S9*R*i{)FWf=6E@ESv%BMFHzE?C{YS3%J#juT@ z*^?wu&%C$kq8=os?%rAFwRBx0FuDaB;j1T;5m3&mMlu)|yMG@3jKGEbl#{hEZKl^& z`c#htT6qaeswoy=;A4#)qjt!Ys40M7$w8zuyOhvOV^d@{wBqP;?~5;&~1F&cPK zJmd88wZ-4S{eHQmFZ`#hi+NY+%B9=6V(6sHq)zceG6yAF6=3jds^?2rh#*)iutwey zO7>YgARKwF76Qs^gizov+@pmwvK+M!2wd_O(EXXirlbVl&83XwKdJJbus!ZZoekcA zNT5+^&{<6VZZG$}4P@d;-I$bBTtBeoYC@U@xd+~>>Td?^$?kIG*u(2LUxB*`IjzEU zWP~dsuMGyKNZ7JeB1heu#u*L;IiXs2XTRI`0UjFq}B;MH7ulTUg(p5B3jhBO{QnwsoI&r9fIBnp%^# zOwt(>Ow#TpA1Ku_SCYMiMQ%z~I3Y&MJ*4>}W$V5xds7)6ecf*U0gl&AHY}r1U1fOD z!&bgi$(vi%3_~_YmAX}T#9)woHpwWDs+F4AKzi7u<(|Y8FX zNXOi+Erg#lnb2K+L%7Xef?=CfQh5R;yuK)Fy=t9#!R%ne1MD+$pYOcOWfyxkkn`x~ zK>wv`!+ECpN3}H^en|0fx3J?)kyqzKJxJJ44Y_Q1h23cyO-JgEbD6D1eH@+IDO9hs zzDq!{Umf$xQYEjY^-jx6a5-n?XqAOV9st_rq;*(@K_+wvM9a`7*JyJUw0ld~!npR|}U6#nR{m&1QVw^Q8pfza>oF$l!PP*65DCS|f9S7G2f;C$_-$^^pdM zz>TO8OkV}zh~*${O_4)DEtH=FajyJq`OHMbd+$4rjuKe8vB z3>HgSM-y&oz#g15CM^&tFbvSVp%IKUx$ULa3K)0eSE}1Q)=pwXYP`31z08BoD3$o0 zSF6jyjfG)+3z4D>i&1_E*~rHTaZVVeRSTqc(+saT|1c_&f6{R&ZZ{QTQ~i zCpWzTpuc17fw z+TICKtxfcskMSxClHI1NBjd`qWgOd>>iMHMPZTT(Usr$zf$uZp3doAJyN4x+XUw&h zY~%QjpQ()iW&7%~mBMNg!O_!w(uyG`T+znd9YNzySX#9qFq|_Z_u2&63m+~iNRybw z%^C(x#sIu|YAl64=7sw`KJSV>>W!@p&!<#d$|GvJ>I{5-9xHup^7Ioc`0JQvhoo!$ z^BHMKG~addxp?yEq56$wVPzTCM3rg1&6S^-blFdNxDrA*7J5{wvxGp+#@q~8FOU6r zUp`u-zGCoqzMfKdUAbYM9g4}hGWml&zfO%gIkY<@cqWZA<4m?GFSiGI1jO(@AyZQN zJ=g-Cnsje~%*k0#_#R+>LLMD90`7xPh@W_$Ktbrb>zrHiUb-+g&xM{BUv}U`3`YED zwLFIhTK3@of{ZED%0jNkoGq#}DVd51fA`H%M%q{Gz9p?@We3yVgIx-Cc?a6*C~`}k zm5F9Dbbr|Ou{9>LarjHOPtk`^lWiST<|aq2_Z)0$?R7d20|90W9>5Dl zo!B(NBZYztvPpq*Qa16#odjj#m=iCFne43xble&vFSP4PS)Qq2=EA}LCpKpO8!EfJ z_<#b3vyX8`NjXDtss!&RL)2|H9y6&Ck5L!3@-@((T#Zvjt~HzQ;F)XoA#iSw5R zj`r4{>nysmjyKY7>~Pe3Qb(@NsNlQL_38&h=nXTXx7f&Lycau&feT<*_G2E`$6U%w zABd}u3?p`K`725fAq$r+6C)c>C2;G`ZX84rPp|XTr%kWN@;E{nJH&<5?1qPnshJa0 z=c=2j{4b5!GbfXs&3N%LOk8?ThhK*GP#K4`nY;QM5oTLIQ&1hCjbl6`oWbS+eVli0 zJ#u{rHufmdo=3cNEnl~}!j5ZyJq2E}=B~QzqT%uWCaJypP4;xz)%h;tt0mWO3+e0q zr^dpw+qMcjPW5uY1_g+Ah%{&!uUB$Yjr+7vR}DqK?w+G;|)Zdm*6A>oavva+eax>{l?Z2mw5HkIm zhxfSBsfxBv8hh+YTe&;Q5$SBuN#Qh$qVkC{#v=}oWJXX$tSmokvwg2f_^Lm&2E>o! zT0?D6YJbwaE`8Rllv&C`9loU75%m($aaU>swtm>5N9w^$OXvL1{C&MK8?z19WN!8= zU^iL7gA{XEdrnzmon7X?%J&W85pXO)D4#g6q}>vDmMxa@iBQHR@pWlGG&bF1;^UWe zU#5^}ROb{q)1>!EdD*f5UV1dtGr|U9y)80B>Og~mPJ73%AdK9Tz@x7@;O}O>oPXgB z3U#(yu=5T%mQ;Fr zpqDKAnkKH(OPt%}6$5wbIBv#clf1yrd;*HlM?6l3q< z8$QYASh)tZ2>!fvr_^FL#7;pzG3m$cWE8CGLrvt$d~Hy*avT%8U1{*M&9s6qowTVn zG?JmkBHSqSKJSn$%V>%zDBn$2*Q#o`$+?~`4G;wq}?@3lt0EiO{H^+i4n$jm0 z>-_OiADz&J3Kecc_eNa#?&t!MYIcr#A*lOP*UUOW>oPv1L6A!Nv95m4jU@Ajks^Ot ztF45QhG3pL;%J~dF=bhtgobS&C!07@IVaDHw4%UEnh^@C=?69~kX+@R9KTgtxI}K% zaet)GbyJGM!h^!oIRp~|H%)n)AC=`$)n!=^VYM`CdzUeUT4a71G1RNm7#prO;^L0b z?SR`h<|p6MRX|dsTFT}gpMt$ch9>?Ba^yoSi;JoBR<4=XBvue%(U+8=`^=e+y~JFF ziie)$(>@r?8t3Arvg78hzWgJ%t2=Nya_`WgGe0@Y^Whk8(F&??FO>FWP zLuF)sp?U6^jy`q$`kFgEC6d+7lw4i4p^doWgAl@zcN#-*o4Ty$0F5|%P~gZNEFL|` zCEdJ9nH2SiOz%>7Mo;3oCKaSZ7M7<(KVTIPC0H~f%`MDPiiC->vr?06PtkKzP5v8t zD${I#yC*KPJ>&cU92#6FNe4uit@+{XPYOlB|qVYYl13C069#&BhYb%e8J0r4re zPnhXRkd*mNZ7l!Y@2A!hYAQ?dXcW}@%3=41Vamb^Dv)=pGG1XP(}ltHG*07n(Rcq#gu;9>t(~| z;Jcb}vgacaDCP<|!fg#&U$)=*!IP7%;QiDKc)R(%j?0H#_62l_X3a-Klf(*`(BQ?I z&6#>37hEc*r5+pBXAWfwQVu8N&YKq;V5*|;BfK*kcNXBxO|end#MYh4*5ptcKaK8XxpJe9< z&Zi@*!JN>deC!K%%0I2gA#OJ?tS(&>s_mW<^s!>(Am6-?I4I+x>kltJLJ+dU>YLHn zGg?ysQk@2QBcbk#x}k@^MqzvSO1A*#{xZ}z27Vp<0Cqv!l3*9qkZpQY=C<==Pe77VRO1O&_3*cF5XGY(E*5FLZd3hKk-9m*rmUXN(8exRHD_kyLFWb&JvDN9l ziOb}&mVI8N$rlo^cn+L22O+9Lt?fT4xSi_Z{77+L5$BoTKWj^l97q7*CBId~KQ@jM zl4UfBt!7b{PTmAlm+0^aN~B7G=*k{Nk!$y7<y0;5h9>WM(abuE^GOv6J$}Ja>58UJ|H*A`^W_G=le`S2JyQ@CUl;XvH6BNyEz-oe znjcS-_H`X21ZR$>hhgRrsJvt6mJ3-bU=zR3vCvW89r5d)%XYB{uh;7WvPSO{RsaGp z)SGwp_?6nNiJ#OvxPsbt+X=NYD56IRqKjU>0yEb!c)Q!{F5BfSr+$YUB)rZM)1&~u z!l$q1I#xuy+7po6OuWR4cPpBZ84~#rsoDNgpl!Ev`$Io#TIG_r>ksd6cQ(-9avM#$ zuNORIzLeyGy%-NQL!rl_qV6~fI7|OpZ2@9+{fwd4S=m}A^|{r6+V7Oy`rG9NhlQ_l zGWN3Xn2pO5ulkR z-Yui4mrDY5{A#Lj2?etIgd&wdtoQyqZVcGv1^;NkA5Yz{hMZN(tHb9`aIkw8v^#4L ze#%J|cGz;i|NIf#TQ!7pu;s?zcb(Cv1w8?NO)*#oo|>Y~xYQ=CA(dShA93rXf|^!UJg_udzAeqnQY5c0x{a45`)7jTU1hLop9Q>!`Z?^ zsT5n**k?nv5cYwRDE+gY93}Bl{+~7Mi~KR^)o9d8!yo8plV{6#ZZherseO!q9%T^N zu~Oljxs->Y=I34}6{O@=tXN(5=FLx}aixGXj8@-qmm+ySA*&esL3(;tX z)T4wi!F=Z3tSYZ!6SjusFAp=)WTJ?l_RS=$8FiU=T+JeVeW?cu)ex+LfxdPTfgQQ= zE>O_ONh69dor=AesTNUYS#1OU_;ABfH}ULl7?l*$asGl*AOip!g@k2yn1^FWjq&a-0l^1xVHh00 zxHj9Dt2#(q8}c0jF{NSDujDP&>_w5AaOI`y1D>URw!3S-|2236_VqNcE;IQ*2q zijtfuD$UuvK9YU>tw@J1L8Yamqg@@>sZN~GxCk2 zWBi(goY2^hf3*q&Elt^8I&dI=_^Jj!R@(I7AM|j%`9OVc>YMy(^~~O+s?Zv$E`=Y7{b#)-`Ji0;Z-)@GejT= z@YInMy&Q~H&jvG3sFZpLi0m#=^`rA$g^NuCy#R2Pa>y|ID26J9@1=q=!=bWGX@D0b z;9?(UP=vl#dcn%#e}sgk_(cu(-C5p&vPW#sszV>9R11bbu1&Ro>3*)JMsA%MCwcDs zNL}stD59d zYfqEN5_4mGx*)L#mo`Xtuo{1UHXK|YS^U-8=BlXI(lGm_)D$BK>0+YrVzVRTc-~n= z8X_xRfXZ=&b}55^&b0M~Q5=w0n<_N)T8tFpDirY#Ake&wCiuf-vnU6QP)`8 zyT@|$L6ZNxK}X&+2^UcAQFxsrxW&Y$|gnU`#jC9ndr%4BMUDVGVPDrTt1X87)yP_d!1(W{;E2x|C z6)-N?U9SW248N5@sS{vVpGGZo&6H8s-tCxRY*XYpQ zLd)Vsf+k$$KPqo?AGE{@nY>*l?o*ktobU8Z4>4 z%p;)N1_VIEZIFye>01yd27r@1+Ljuq^HTVw7HK#Ws!?V$Ff4k)&GrTrAyN7i2<8FS zDN#EAa2l6m)-zPzQ>Vl)LU*gk80RZgU*KHIQadWX?|Q{GDY^%wHwWDe*U|qh6mUy& zPUe_17D|*>e6=~g1rEO+P#Jm*66>L%1y~96{~6sVSxyENIyv?ikSyp$UT|tFX+8tc zaJT%LAZbZpJ{02<(vL6P-RpTM zKwDnSnH1u}V3j=f8Kkk@AKxL{;1=I5uncj~NS(y$p7esll};L2A@Xu~ zO`)B)r=(b14lHECYnmJjR{RaCUvLcRHxr9jP`b(%r3>n)^7g?+?b}>AKCo}k z70Or`AIr~!j(Y`_^4r^a;5~d))(ii*K0xGUvtKU+1vTYxPZ9rC!nc9rJ$2M)3hrk> zinG88qa*08op(h>ayU2Iy*2eQ-pGs$W>T=99zx4V6>nm|!{(pjFGnyV-F)%-RX-P@ zC{sWK$qmRwMnXYXevDkOeo`_#l+(1gD}*-zi-m-an#^sK;AM~E)h?`exJxQl2U^LP zGy5Sj#w$4@hJqa`PY`xy529cLc);g43IoNH{lW*hQMAn1Y-MW=hwcN8`Gb$9@G1U( z$ugY(r!2$9^gomg69FSDDVAhi%vpnvp^cL}d= zd~z~S05A?9!0-7+&d@I(TLA9VEbrU|+6Yhq`x6w!8V3gx1{P;~OCYYC6GGx4hseP2 z!o=3b;`p@`3>$bbSIV~`7!&X}mYBju2bXdT0363Ckf9iZNML(#Z^Htvu;B*~BESqx zjqDcxzDsz`tKvh6VUrXLlwaz-TJkj{tl(m&87^H+%V}&`aK8)p{) zGf^blb7XX5!AoTcPFlK_EW-^li-G<5T) zhi zJw5QdVGMFiIy|zxjA^t78h(ht^nni-p|G&>KyS|X95Q_T9;hvy6Oe}I65!Z1KEgjr z)T*j_N0Xc26EnB$?IR1v`f@AnJGO>akd2Pu<;Rkp5gaq~Grg-mkv0))baNV#g7lq# z?uIPLcSFWM2EYVh77oCPmC5jZ;hMhrldRk0!!*1sy0;J>c!9wU6n2Z_Q6Uz>^+&^6^&+ z)`nlzCcr-dWDqiU>=9Q%g`XeCneWQfw#4M*E`Qdz_g?Y$9ec3lV>O!}%r`wktUi9D z@+0fZYS^mp3Dau=n`K?wN-rn6W_P(Vsj%%ATADo)$|_+z@3phy`i0%U3hDN#G=x^h3ii`ba^nfeq=E}Ee)TsPqx87+%MU!U%#J9hWZM~0W9zb?r?H{ z0rxn%e6t&Q+g`r^YBO_J^sl)SUsis;OF!^oDZaZszvF3XtPakn`=VIPjdV1-M+O+Gk)rrOHg2Rwyq*+v)m;;367;9V@au= zE;~-8YrRxnmy~iO$xm*8U@|AH+%13n*3;3e{&vwg3(!q3Vp_(Cz=T=yH z1h5X68ld=qW*;b_Gu_xdCIM)#swoGyp#e*?pnLWvun-ZR)NgdHzrt zNz>!E>17bV17^H(Jh66U2=y*IO*Ai(Ry@&?t_>yG9Vc@rIouTS5n+S}l_J4Q%c=8c48GNmDPGKYmtU2#zXk{%TiCjIgC z(XUl?&Z-9bpNT`^>%%du8$)ykBi}mDzPLF3_K7%vqr(HxTA(k(Vy7a*m9}+uu%!fB8-Ns-~>gB)=0$V*fg}>I!<{zr)`)n zK7(KREncqOy=X&F28FMIbOJeY@AO4B`gcF}Mx?7FGa15qV=9?b%_mmv^U;T;<(b%x zD|lKPmekR9i+6KMZz4NFBLz`o#en%5-Z-F~8j&Zy?u0;Ol zxu?Of>^fGoY^dG&i*%t+z{YKh8 z@WQG%09%e)*waW_cVq0uYpc~l)w)^3;Ah#03wd+{gS_Lhf`knJZ)Y@B$Um47GNP3)2ui33b5x zm}@~A-{(teQ300k9i#z$F7xr5rVJc5S$*?3nlr}Y;PWmBvf-2@$2W7OLt?9VQfxS! z+p*v%PcHL1nc-Z7VR}d(yhJMAn=rwsd!MhZy|&Xc$ybBC?I}_cBxlrs9JO7Z9zuwY z>V)fw+5@xHf3S5A!MQMDgN9?@h;Q|$9w-eh1jt$wyBSJUxeVQd~hxhnw{2r~Hn6@aoJ3{UGOK*5c(vt9F zEP8-ggvK%&-cyTGIOw^!1hr;K4f5`NgWwWWp-ogwa)I zb7IobD!$EPgsv#>rVfC0E0fk}Y3q;Rk5(&A zCIe8oJfp1G*JUQXh&>-jP9f$iZRP&xJF}k$-6R;8EIlj!-T2$Rt$?g^^}cU@O!HU? zvhmJL1Dh7Wh>IbtiSSo+#WG1!)Lbl+6ya~^@dsVnt?^1jH1v^r_$r%*JS`_Ngbd7A zk%E7p+Hn@A@YC>lM@7hC=k2~xpByI*#;wlfX6Pwd|G~6+s#3d_pqlG&+eE-N8M$LK z=MUq697A2!gJK^-5^f=5Xs!XZwE20qen0ws;gqpXZ^ejq1S`{Y&? zW~@vw7>8Sv`sdTqV>F;b4jilviz4Gc=P51QoeyM4Pld%8sI=@~W72xi0xP^ifhuZD zwf@ZTi2eCR~zE8&%A`zE?mF_#i2T*c@K3~v}kL-P1^n~%LF!P z9GvPJ!?O2{+np>gF`JC;#;atR>fjHn&sr)OnK=a?BK2E?p!&Rx!M&2BacNmr63GMS z^)@)lX>?e}rSz~P2(iFFVEi0`NQ!wf3zt|02bc-+u0I>crsLFl(t|5zL8HZ8qCBVu z1HNmasXktd(`|^oxnywQRGSB~9H@RBwl6LvKX}`~d+!pL zSnEYeHWNHEqD_6HNh}5b93^4&d~5|WI-{Xw;UoWe6jWVVSUi0Of71ab2o%eu$Dlqb< zyrxF*Ja-IPB$9+iG)_2~p_vX|0v7w$j}_`pHPdEn=CZL)Qi&wW>_*Kq zi}0b{ijhA=lc^dEL-P$2&(cMAmJE^@7a?I5F&u_@>`_9KYtoN9NwRvj?8Rz9>rbMr zcjU+NPWo&CIq4$TA+xqVzaC~)q(A}<9;pY&`ddSW99GmJ*)WjnJ4BQDg&~H-9V!0u zdVPKxD(a?T(+f;+GxG1$Fx00Vd2|a37oJ$1F{aP^N1PwakQ1RtK}x5hnP_+m)tCPrzITDS%;Sfh zYA}|G8#~)0f(8E@vH2a`B5|6pzHE#zkDlK}q#OurJ&7ib-Di!RZi?HrMbN$PA=64x zV`E`O6+apuhA3_X(`Feq56+5K8!5E!>qP(Z#9Ciz3}IqamD=Kw#;PrD5JFb+C<}S(& zj8%mUlv2%K$R;U-{nwW=E_oF9c6n0@n1V45QbVcLv$l1b+}A2ag~bCaKYhAh;D9k$_ho{JJM*xCO` z)=G>h+=4dWD?>DE68-p-33J}d*US%|1Gd+o^iTo`xoYlJ4vvWCfAHp+u9n-RgiHkC z9W1Q8e%d!WWw%x2%IfG^Z67o4HD`*;7-~>hq#=qeYBgRnH?tJ>VrtE%_#l3~vVWPF z{nb!_W;*X68~-G4x!#0Dj?$VHMf@eCmiz)BKT=w+m=4s?&DpmAJ*|tO+1HE42o|Ux z#*|U`77dV49&QoY!+R8S#ug*!#ZBR2=&n@N5R=W?g0GBHI7-ToOpY)9Wc?LTm~ zM6}@mw-aIs>3I?BTEg6%H619;dG-xGF2p_uG<1kn{?YMmr>)-9Y)y{QV;L+JR|=de zbC!2bb)BsfL_Od-{1blTsk*P|`^L+5m=z9HghY=!lrYpTh*&C-Kf2~OSvXG^d?_%& zt>~oqwH&~jv&G;uXerf*?UYiK72Nf7Zz`fHl0RBIZ54fEMe6y*hNF+DgLM;MS=?JZ z-y|sQv=b^*UK`95`mF5x&JZt%lAaK&AIygGcCs~c;pp-HQNfSs|9V_qOtgZDmgJY- zxvbjXDM~xg9^`w=Q3!up)28>vC#;h>t888baZ8@y8lD#!x`wqD1)U;d0=FN?CMIqF zcPGWw4D~s4#wrf)*E5{ugS=O&qqw%=HmS~Vj-OGbR?nPSlS;0eg45CWkV#;7^UGV= zoq-cJf-DDi8GG>^z4kX1cNz82HkS4UfxO(igt#>%K4Rpna5$kp3(XEtUIT{BMDsiV zy2vJZd(i~jBkOP?X1&9EB&3%)RH@tG2SF`IjJ>_e3ul_H;p zTwdH-5UtsLc2zeqci7-O1J_h?v-#Zf4HwyB?+yPnBr2XfDLd?jBG&M5iNff2QTx6Ubz`!v*PP?~}ce7N5;Zm0#{Bvv8|C?x*NFRM`- z1NWTwxakozWgj<|=D?WMd5OwoqeIanvZ4V_%MyP|_o})~6UY6ST4AiG(|8)#LU|ev ztRMV*BQSRF_;`0$$=BQ_xf+kWY0lwmD-y6oi7>kneHX9xU%M(OjV4+#B$yCEssdwf z%d_vMafZ0;tx>k`hI=;qpRyqzl`Q0s=W(hu6+`Dh_SudzHW2MbxfDvicLXIdewgR= zA_!?0WyB#yR8_rC6)%zVL8R22$vL|;1J9t0My#`fTI?qe3ccT_8YhJEv4+9C2<@L~ z_#HFDT6YC@`27YSqw+RSWyED@evdU&X{C3hvtkl6c zZ@0)&BSw)2fp@wDm}hm_Yt5W&Wr9QmC#kqXlsWSx%BVMJssXkabH(8ydd~7bGDvXu za(MtDdf;1BSgb~pPM{q+n4jN$DKa7`5~2Zmm<(_gqwNAx^VDXK*Ds~{X;AxETT5L~ zx%$Ey0)QrjAF20Rb6;WsXS8bb4Lu5pzTJA&?opu`OwOfQ*-p3jQR)L3dge^B%}1|X z>=}?6ysLWR$DRvmokHrmza%)ZC0@W+70o=T{1a|D)YDH1J9R%ApTVxfpC+ksS-3AV zb56uyTTw`x6kM=lr!ifME?j?{|K34SE7R{Q?a0W*r&%Fk(vqwgT$=d1Ov8kv zRmo%dR)jNi#7BfWye1Lu*!wi}Uky{R?TxUO=pvzlY&?OAyzVn-INl=S;Q z<;Xw{X9-h2s^fYqBiO!G=dH9=;RNqAu4UrhS9;Ui{oV(3!C)qd3Mix^uxE+sQp@5I#8b6B}#o@b>{j43tOp>4dKuW zW3(S(e#`$oRs6Gx3Cnu#-@XnejIAVwMRK-3JI1D1$CfC9Q|0kqyNaIfXUUgyWxQp` z>FMJ82^0Z(PZ#G@=>3grQzZ+hGx-Xta;)uYoWnw6`q8E=6@3u39L_A;%#yjTiWl+z zJxO4_9YXX7N2p45YCKO{kw9nsY>z_dyJ^HQT+(G%T+g_t*pQJn#^rp(9BqwGzIOhuM9OvrKd4j73zuLS6kE1CrU6cLs@ZaArO4AgD88 zNqAWC4~x_x!8t=Bx}frAdC;HkM;zA1mAbx_*QpdKK^4IG$|D@ylA4q z2h7hZ13oOI`rL18wLu;Moj~VOZ`_4xv2rgP8@bQm>X7aPBVj82zDB`G*!kqLgR66> zBHg)4o2r);`Ioc6TkuxyF$c)L6;?SMInC@Vy*%oqy%2p7VNua=MQS+;6;0yVB7ekt z{QUyaPkLedF=3!00~o@ZM25&@Wp&rBA&of4r0&nBn>uJw;=OQDxbQ)Ud_sXDXo*fSSTiHqUJ5KkK!q zkJv8iE`7n&sg`e=Q_51S1fr7)(T{lY)x~6j!dz8?ai8+!O-tK2c2zA-RwK$2wmvp^ zh(;I;z>|zP^YNWc`hdY=HKz#C25F%!jd{z|AA(-)Ld=7kXUlEZhAr)@tb=CkfHd~x zO>XzSLs^NjaBt*J05K#Xe1{-R(mPxY=*_DZS)V?fIh#N>W-6q`2&ej-wovoKC&t$S zNpUYH6NT%KE*#E%s==5dM3zf*DHn-YF;a#3PLZ7UUyopxc}~7O&}35(MI|i)=_l6s zABzVvTx>|_)AF*=5zd{MDo;frnU0MEQXw#G%PblZdX=}L=LS9+Cw-|WW9cTc z;l)0TXoFP;VkNlpxEF6?%Jzo7w_3+F;R89Ruuv_%@6BOn{dn&@<;L;fZc}uh&l2kO zu4G#Zz6>P|UYB?&y$Paa%D>vli6zc6v2vP4Rt@G!@*7F7Nm=p2oYa+br_YtC%a@T4 ztNPTcQ~rg%r|iW{I?IUOE}#&}(JSuMv0VmYjZex)fRg9~Vvx>YM~UNZhu+VewOekw zG7SxZ53ImhS++>_4+I}|8+lZ1Y3lj$>I1G7#ztqWPI*$iKRk+TyGZZ3417M=>3<#N z<_)tU1(O41t4WhN-@UGPY)<4R5ANuvYEbnDWmmYbvQaiUCB>5bp%PL}(HLZ&G!@^R zo}ZR&!5@#jtfRoe3Xy+39OV!$4@!$X(;?n>WN3Njth#}kb|of!)jDyfm6Zvh_RUrl z30w1IMh7467LnSzw#UwZ8Avt_%r-f@$JRJ$veF=Z6t(m(V1Wj_N3Q8^tZ{`8X^xR= zr4oaobhX$OWWLPl)~dKO@DIhfDq~HoIgGO-Yzq=@ty})gu2~bo1vE)*LDz0)AmIh& zoCrm=wBYK30RHnpGGHS#zj%n(K#jBghubMLMyVvQt<^e_*^r&KFkl8c+Tw} z+c+o>R690wsh!O(+2DM(r&9+nj43=$DCy<884sHeis$7<$o4o}^4ezIw+UQ(@D(PH z>kWuJSyW@E^LHLICX3B zaM73Ge3zGNS#R^k(off{=AKTC2rUEPFY^-a-~` z&6YoKP4Q15kv%`VlarK^_O_5BG!!xjF|rQ`@A%I6fCr`gIZSCM)z;*@Kj8Qp&f&PR zo=(aOyLm1y+e1yDUI7}WmT)|;sxF^*`I8c3oGM{Bd3?;dRiMoLM(`R+mx`r@iq$X?(lATQ*SzlAf& zIyoKj{$zzP2@&?G!r&;Uvw>P7$XCNQgfE4Ifrt&Kh+8@So+x7}t&S`|y@ROu4kL^K ztdvRKlrk7z)>mvzSIvl)8F%T!K$mV4HrkWqT5d4$->(_8`1#5VXr%&_GQ|*b%nnzZ z`U|o2u3si%4|go~T6ZFz`OGdx&_6YCJCC7rj*Lbdt{Il!MMHnH;}Urwjj3?VD;V0h zcH}Hp$`$;|3O6tfW&r^>aEw|gEin%<@sOGZ;B1XjdC>;`$aio%y^B-RW;JIE$5wJX zFENu|K)%_Y{#GR}Zg42#YDWlt@Z6r3ZZip!(0y3MG+l1Is|KL0l?z+TYMGK43D(BJI$1P57_g9KUbN2}N+(;IdSaTkNZfMJi9} zDWb-IQ>+A zkZ=7GNiNopD^R6xm$U28pVRp|wsVF*hD({rL2Wm2R)&tB2~D)#Hr!OE{aYSUfO2hd ztMc&JB=?;V8}sa(KS|GS$56RQjNGg14lw9_P)l^Y3g02c4xq3w?&S%YjzVEDc;)hi`yCEe}s?BNXQ+YR`t;3?SO>pv7SmahM-=FiKCI z{Cw#*z=i$`-6^G6|9;$&-jQua>rHmIRCo&Nm4U5L^|3q_mWIq{LWSON#>Tx(gEMZ* zottZ@$rd$D$b)Euklg?nu7Z#D-~{Y(KW;*I;*i;d=|>T5Wxxb4ZhJYt@Fc8-)KKb7 zTATrGS+3r!DLB!P1&={`)Y*27xzlaZ-)*d{HvmXjyF5GvrI3PRXmU#f$|%O6ls((S zV1bwUqJa+pNiIGN+o_S)+R?o93>aLJDxv94)i}ST7dF`Y+v}VfNQ4q~A4b@<`roe=U;NVlg^2 zA`Rermk4*>uG_sLjY;{@Ie;Wtna()L;&mukUItM+uZVa3ySk*ng>{h7xa; zcbw}3VF&dhmzQtWP6Nk{LLI!E&&HnIOqOj|Rkoq8g>tAfoMF2q#`L-{1k5bH}`Bu#vD=!yPt zc8IPJuo}ah2S`#dTO?{mvWDSLR{w`AAQ4m9+*wMN*vpc3@(h%2Fq4I)Eue4nPM>av zbxFc1yu0GKW}}*8bJ!WLT9s-D92$L~FndfM1_<)y;SbCtPmTy?o@{;=RevWYGXKj7 zMyt*D1LwLd4P%^6ZPoIA*UXe5KW&v@K3j*jXS68`h4Zs<$qDDMoJyf(ID6%FZh>M% zM!cvH&`G7H`4bY<$*Uu7Ex{Kj%M$EAzI(D#9=Qhw)8+d$>aZQP5gfTLLnCnx*ETnV z^R)W;L>bHGXy+Zt*FB6Fgntq%%HWuZuW`2B3p2{`2LHEedYt=Os$7v%(ob@^Euy35 zWEJi4(M>Q@y>LOEWZ(*HlE2t?#%}a8X4FRD&z09E8}=M-9WBXQHu<+FSvza*483mH zQIimhY{_W9!qWRycnu}0(y^9RYuAkf@6r4w^Q~5XXSr>FLz#y2jPS$t&zz zS-k?OO^rI88u>kIDRZKg8@=($O-A1`6CpdWMd?wp6gCa$(o6Rhabi%*67R}25gC88 zZh@~2SBU4w)@Rb8fLXRNZ{fg`G~>VnAJ^)sb5qI67Oud}uBiH&Njn(|KNIiD_((VT zeGYr4Zk5y#p;yN7(!Wk;iO`aRB-Tp~6wlewuy>nNH+4x>De==ck(|Sgw?!-%rR{ct zZSm^A%gkrgLG&ZOA~QHmf5r}abHC_|N3B*{AR&ABTa*gfR|8QyEoG(M-g~Z|7JrTq z7D+u|+WzbH&Cyt980zbVhoCn*a0+(>D7mRd^^%rU8^gSW3IF+GGqmC+0kc#**f=#< zsk`a~G1V|hlc>J~{Me4}N^%y&#{1^IMlRGFLGv3E^jdJ--`j$Yd9pcGv`1O4V^vwGe{6Y7Few2>*v#6pNIhjoMUhQy0!I_@e%1Uvq zN)D;=qM5tw7$4mAtSx6!2Ab4pvuT%-F(@WUkrBIs7lJ3Ldh~$R+25#U4?prBPYdb7 zib}hanT;Ke3Rd@LA8cCD%l>wlF^k3ZbhBH*3`25>+5PI0-E;4 zcKa)=AyZ-Uhl!fG9GhLjzy`ma-!E2ZX=h>)4M3jWNt6141!_>*wLLwEm7PpZ^I(vi zOl2y)$vO_x$@aztYzgR&liLS9tTl;!*o5f<2r?#%;4Ba*@_R;HSkLOYGCQJ?BD>0V zK`4^IH$Ls6gkvxWHHf*!=dHSSE)F0ghDF)4kBD~?QNUAMGC$Nk`zziq+e4eY$1uxx zT-N3)o?DlpJ1vS$X>5c?Z_}0|3sciv&Pba0g38;5y$Rl$4;_CNy%}v%*+3BVV_(BTeV=;;Sxo}_5>D%Pe=zkcyh6%rwQs29L zZb4Oi3O299u17%9TW(^Ms!$c9Zy1sDHeb8Jq-R2TluUnM5VRE3p|F!U-|%WL52{*6#!ci3zfEHd#yhE^-H8~~rBWn_-o zmN%F_nvEWGtFlZD!Je2L~rLcXwpSF@F*=$p%+$wmyr0D3sK7CljPXT9hXjfRb*d9ry-%YC{ z5p(kf(LoV?vc+F@U}fu00uE@__>mnM40nkDLxsDySu%Xnnx{v?T?UR5X;lf;tmq<_ ziK+v8dY_seN(I^Fm0~>tc|ZQ^x`BwbYKd-tW2jl4%eh5JR$9CCY}1t+I9;}0Do`R| ziI!Qkh?F{Df=>ricySLG_Ai7MuwDo^E_Clet?F>uq_!f`ex&_6Fo5GrW%&{qxWx^V zAm*R;dm~b+NMey#(=vem15Pnq_MxT;uGos-UA;g$A`^~WA>ceZ0J^Go0U`7WKS!JWs5K^8m(YY&JlX86{zn1vRAla-;EhUp5T8WoGl}`@ z-pYU1_E(m71rl6geC8`{;4iTOHog z{tm4(O#6BTsF&tuSE8)oNYhsv3h;++82~3mK(?0?P5UO2qkD|oWgGk`K<#E02^Ui` zn2$+zZO((2x6;_F%SA*?eagu)OylFwJgO~Qz-vdgzx=o&u96fi$-Q^D zLa0~jKE4}^yV*W*yf{q1j<&?Q)7#2J=v>9?I}GM%ZuZgFoetu_RYtIK{x_T|+cj&m zS)({WK=c6W1faJswp2b;8SfL8uVy@{QdAsMjjA88H;j9vu~)Ibjr)6v{bmMOh?*pR zH0y9q&CI*okmZ_gl6iCtG=Iw%%i z4mo;%H{vEsb{PW z>f(BK@YCCl24TGT^ol+P7li72&u$0!<{ac$4*{(?MdL%8*LV)O98DFR_k;8$ufrMN zA4Fa3nb;89i&)1+D@d>y8j*S9dv}Anx;qtD-Sc%zP2^AciUzY=cpd`21gV&$P7ix^ zq*CM3!|&(YM#ZG{^d_O@f*mL8N|k6vT-nyhIY>Lf^$<&}?A?@*Ed`Tj5nRW-_&lwS zb`m_Hn<26pzN(+CM9dARz5J&c0;CCDOFiL(7auqjpW`z`Eaa*wv%>uO&x@6~Rj|}! z#;(buh;R`EG6m~N+GK_eAP6nObFl37X25C+wdrWQZ!39Denj-Jz%pH#J89__9?Mf3 zEO6h=V-d&`fGu9(D);1fmv$T#1fPe5s$kg)J-ihNZ*y?L)h?QvbKcxY!~N-dwRlQn z1{&|trIiKwPf}F&hZmC0A}rifF0OOt6%d}wF7SzMmK_* z)Cr5R3J%vr8~Ge0ZFcT_CTIJa>*vDsF4yW9LU_)>hGy_LDFf5NTo1Ely7sK=9F8oPs+{Dl z(+lg)D&P6T5-622lqGL-iiiPH>rK)Y(TjB$7pg~a&9WMUH4Sf@nhq>p1eHB=!5@}~ z`f?)$KV(fQqLkAbEwr}5MJ42Gpp-YIF?%ZN;@nOY&e}@;klEu*1fS^>K`C!))EXI_ z8ZU636`b7VzNuimle1hN+;neY#u{w`=$N09b20q1{VOa%zgjai&gK&6;fPg zWZGN*4F7AXs*-$Bu2qELrZL_vk#<=H6cq2TtUOhSZE~P^knDS$qP-E@o7zqdjAziS zaR_Z%<=m=}E|^e;+SpjWTpsj#yAp_1wU4xjRT9vyWD#pbz-fs0vhVj3(UiD#l+K`D zvNmcz;RM%G>+ny*NB9lOP`e2XT7Gr%i3dKfZ2uM`{aY(D{gl_DdXrlj$g9>CEbZYu zdd5Ci?Y!R)KW@#Y%|aS5bOq~w`1b%C=bUA&%0pfy;(IoOVSQC|g_W{LW&9Y0 z?~U7L{?xt!)rFPn9cmOSBvc?`84mZkSH|h_(>mI z32SP+8rP$o{JE?wT)~<`818$ohAu~gRYzLUVGp7a(Rjmc%9-V2Xj`i_J7FZ{thCQm z7nGj0#mB@?hLEl=O$)nTBDTY8vA{N0nN9nUUI<-mgw;Ju{RsPxS+BKQxVSF0;P=>! zc3fnxjbJm*ut)O)XZO`!oU567AyKX}jMcoSI?7RQY#(HtiW2D|=77*rQZeLwC&I{U zJw2{{t9l0`m)L!5_gJCLNYU-t;rPo@$xBDBiB}+&wBz8Nc&4+l@F*>oA8ogNjzU_L z`5BEQG2h8c8|1=Tr}Vj2&Ur8~_wTtNyAZb_d)ynge`akY$?A$oUOC0 zn}zrNTBKM+j2Fv}2f2j9fD=P3Q|WA{N?O)~V?9|Lk8Q}MYjtq|i-KiH=-#w6^Jx#L zDCHb?N3fH^+ zDmsZbl^?T87!nFV#@RRbev+=(f!l#Y9^;v|8uHx}Vqm1deq$T-vs>NA(H}!~35HQe zG4QlLCu(B7sNNSw!Nn&1Ybox@4UJmmT8%5o8b>(EFq7i4`qA>Q9O{zS>zw|G`M2IJ zv&$V!y8%5n@$cLf5yOBrkE|Efwx?Jz6foD+$U}!L#2iriy9#&6!6IZgLPOr-_TNd7 z(kV3-ui1MsX)9=*t5N=dtqp6*=}l}i{x7&8X_&BXTN98fjaBMEsg+!QU2-MT{L;=p z5_>h>w~9r1>y-vUs^3IJQY^;(1uIXUMJ zpH4Biz~eGb5QID%v8c*4I535VM}jCVEyuA8^&c+OE+#<*ozz@}zab5sECR>BDl+e)U~k;bpPq@=nxqZTR7^pUId zt1zAAPy~b9Ojk02jq+`PErlGDN!;#18-sTzyNeZWNIFC{oefmi-~=MA)U~c0N{f+k z6z5t!^2ysnG0`vb)p@l zUNOF}-5Uk8VVfeFQhu1z5AAf1>v<>PO(4{Pb-xbm&l=8 z%zx)=3S8sw7^Zhh+TB!7p4G)KHgJjvh5dG{Lvm7iRNc=@s%;~EeJF2p>m{sO?S5ms zl_vmHvHr3xf#%aIDWGx}HcumhmYKZ$Y3ki@HaIYhz|bOhaPfBH)DMjPzKun^N&!yj zU#5CBjG5bd>~exxlR{B3hfs})C|?ZdSy%y`&P#$iV$qsn)sv92I0I|vaPB`SpE{c{ z_f%NDp+BL{0;0g!Sxm|VS#oVx(ctBuMF66W@uuBe zxE(LkrBMErU~>`AVn~ATBZOC5#!TS}ZA2PvpjB62@hus7U9+nnYJ-R7Dpzsrml+Pn zmCKU6XV@t%JH0K@W{`CJRj<^s2%n7)oY&P1Y2CcAmoBTnE`|SI@H7Df-Zi=&KZQn6 zwcE7~`<@uRxH1mWzu<>Osj~X=y!Q?m7H<&ta!gR13rz9@QGRd!&*^pdrWxBD63e&G zMGj+@ZiPxsNmWg6$ay-80!vUI4dxzbVIcjEp^tEp+R|qudkte)I)xBeLMg$m07Do= zb-Qh2oyF8)v}m_vp*Qf$j^59M(r2An(DzA5TXO)gW1R)LbIxaZ{Jdw)|x3DTALv)^K)+?i0@6xZ){Px21o3Cd3P zZ_IoTpu`S1de*iOsi-425x>U^W?raVz^Xs6wbpNeFuBuFysG-FI)$2)yxE&-(e5^O z>Z3*7HL|;cX?3rkzD3tpx(n14{Vrw7_FpbB%Imx~`a(7P=AiN^nH_AJTVg4Q>t}k7 zITk)%gJc$)GbG;@!{X+M1^o5<1hnF685dE8sm`@hq0(sLeY=&( zNP4_?am)`1T#H<1s2tVvh>W zSJ%zK9tRPJ-W=6-fdgoO)!V%1B(HA~J@n)Gy@zlMZylw|!C`b{^s;u}`1qr5YF+53 z=x@dA51#LEO*QITjZO#3J?3RdrK)`w=pRMeuhE<)dWD5zi?p`Q)3Or%$~Vv~$DFS< zRNGTTX!~?NO~FN`ymnL0o{8XL?UJ9^J@VNXM)?NQXh#vv!Ww)rl{hFH$c(IGFK( zM&obdYrD&$^#kpL#t_}vQIy(bGsaekn$;h;A19xIJ1w$(OqdLXsQbMmC`jn4(8%xj zk-H@|Zoq4ieEr?%9jQKWG%NEb2g{~gTD~3CcjPK48VzOwu+~f&p|Wi~4*m;+eZ5u` zU5&wXN0?#H_xRUR3=7&uc2mqJKP}32)7+#J6k(*6z@pZy68wBMkAD*rVAPnz)H0R9 zMOPhC>a#uJ9J9ip^HiQs&Y?(pfDrmiYLhTu_ic-eM(saFx2Eq-OO9A*bmv_qOWULs z!@(frcU^55_JjgbvPBjaeyH!ErQobT(66Jf9g=G}0bObzyN4W5UFQlmfxyRVqM^)Fj+ezg^L z>-Ky|Xxur~s5{Dt}<|KGNb8f~Pb)q?G2JK3@N$+Yvj*pSXl z2hdZB!#@ND^#&`>uu90Mpb#<9P>Q_mnmKfqA-)vrw9(W0R3GKvC#Y7wVj+iXRhCCuE?4BU#$ia zt$W7KDubhZe=X)l~xg4G*!!xVEe7<>DV3iD7tn{ut zk7bZWK;t0Z=CXMMK9x#p!-~4bNg{Q$U?lS0I9cs!B^9I1XhRg}Fq{8UC7~K>_r^Pw zf-#e`)eXTY1$21@q!n*h*i6J3C!wU=c2@FF2^B;1uyZ>=|0B>(7iE27zJR(!rGZNi ztcH4IJVK+?`OCunuixIYm2&jDN-1%svrgvOv{z{yVJKFQ24>NtQ9{xv_~l(-AQ$gh zKOe!ju}&TY&IZ6ud+s6jXHPrARd|GTMBLQf--YcKJyIoRY&OZ`YqXN+a*xPLziJk; zIF&qG9eQN}(&-`mS$m9TMe~Fhl9TCvIh2hvHUeIXOl+1S`m=KD1mYEgSiWaF9Qj07 z7uFf07fqdl$ZV8dhNU5j(6=kqEg>vf%3=>uMh#zDY^<^PhP+Uj-Zz0gPrw&TZd18~ z&KC%g0~vBeom(QW;wCKN0&M!>inIjwVgzFH81~zgU9Oa(A>QIVG5>dbBYkh3-L=l<{8-DKTr#fpy)cw7@yIWMFFhr#4f&fJ9wg+_PCVz z9J)ci33mOfK?7hs^BSlCfIXz)+oR{8^^nQYw? za4%NUC7nzwso&)pnKx80^!;fXC2vCt$k?OrJ>iuaiyu!hc}13ape~P?l+?13Vguz1 zK0~;@6><=Mz)$4|Gt-8SiaYvw!DFhmx>$?|)>B)}=w# zc`z(Y+O&p!vQNy*eix*MLPF=B-%m(NZr6;y_)=(=z{G>M$aSfRI9c4p!M+KOYag%W zOYm}^!l^&Rv)M-<`0`}b4h#(8kq_OR&cat6o^blQtXA1r^&*W-C)|9C50pRF=ZgHV z0HK3|U3i1q3eWiBJb_Bxe7DvG6pLQOkQD_s-s zzvx$rhkpj7OM=VB3k`5wR?RF&Ev`v|_=~_emlyvR8-khnf3qQ2SeX9@gJ2It{!~J z*ROuzVu0D1ZR4m6W;dAiSD!1{(2g)@)+ytdF%)90NT_v`6f#*EDCq>GNTpUPQv;Fr z9hKR~=5F6SK|jViX6Kw&t({Zd(`?zJ_zrYAP*Dd^S7>(@IOsHW;AnZdIa9?dB0|nU zPH1ZQCNO@g9Saiw;7KW2)`)NmWMMXJ$v*3%@q8G6RU$7C^?XL`Cr8i-pyI+QGLW+X z_jss%tUh`uCg5=k)p*8mu%UTDgcC3n?2}LmVW#lczI|cnF|nvUT(Kb!_rwYTQzNEI z%E}fBJSMR&FJurYeJ-GwQJ_!Ei{1~)+bK8)@AISjs@ZK;q!^l7fD?F(L zE4#v=1)=qc+S zq)>*AJ47k`5uWB;Ux*nIp%nqOvFI>naL)t;W+MKTL19UtJ;K;}7yjpmBdTpbucf{4UBsoG-~+rD3T`EE*Y$aK?VjzQP`9Ln;JV5d%45vSvZ3 z_y{y6wxhp$v#6-b$bEZT<4T*NMT1X+UBCpgK*@fQ#1y|Gzih$naCn(i&=xk#VBLR_ z{DB+fpj1Gxtq|{j1yx=(==37n+Nd`m(S8Y2zIE`ggB}q+{II`DEMS#0qOwrJ1uSlR znee1BF`Ai+dh-lGIuh*ckeL<}~^wgyBV{GJyPZSlK#frAFUfLI=2fHI}T zAep5_T|R!f}wlf?$;;ehBXk*pBjDw(=D~{0`}BpooZ^ zFXDfdRsGS3OS$}-Ii>Whe}N<8%@Hpy!BKvv#3XQNND&a>xnl92vIK^(w8e{i7t|Af zsb31(e^YZwl4B4Nv|W0;Nr=S>BRR28FY%wU(4dL#@<-!1pXyMkg$29|x}mWyM8Y(U zeT$&DAIT5A`*!W%Ws<`~OMm{lE$LA1=R<##E%|Yjal88cYDz5hc0T8YU5~Cu%8|JT zv~s~cRsa4sK`|j?o>9TuqL7h={rr3%e@Ox*7|7mnTFc9e7(&TOVEa?k3T?-T(Evq8 zJ{0(AcK&)B`Cc(ul>hLntnvZ{2ITWD3JLaqyDyzTU8E;6v3PHemGFY z3FMb!Y7I(1nIU>lPQ0f7+OC@5Jj`g!6oa`Wjr29F*1j`CZo*@G=dsoE*uh|>=-+3` zTxcCyMy3(ccDTMSC-r|my2MO(Oej3|(@id0{u=#e z6EB%QaOpcfY%b-FX2!%wy2?GQCa=Z;Hl%o_K0!jr?D=ApuF$y_2phqx+)wR~jL`%Jtq zq{??Mm-`zmOIpw^+SR?I#{IO2x%C(2v1N&el8)v%=MlJ6C&#XI-r3>=Ky>NpRZAa80c9hCdUP=eCqnOS8rxt2!!pu(d@OLXB{0Niv z6vfA()d*_qvozdpO**O&&X z#M-S6_UYz6N~>dL%**jM6a0HG0@o$&Pb zyf0Ls{@imXqqg=^Ri6J-RM6jbI{}rb(EX->Kt%FoJup_(1r~qEZcX3u z06X$FBwhjvCSXICoW4oYob6X(RiA+9@wrO+6^v7U4=81}iMMaTo-Z&QA;CN6?MY{H zq=volclt-!tLXqJeV)OUIWZml^%Kr<)0%snSAIBY$4rL6*vF*_zYKKeH)KzGHDlG$ z$L&>0bPb|E6?K30j7_O@z9~V4+PY#kJCx9QjRVfxIk4dww`bZOuO6*e)o)yM%aKEN^OjIPcx^nz;2G*nH#6Y{cHd z`@){%nV<4)=}};r(Yy|bjMsdRY2DSy!g;9W*Xy3=`j(t2(L=Qc#OEKTQYap&^w=8% z8ZDFW)Zlm4r|VYt`ouZjm`rD=yENCstbhY^E$Pg?eFe6cM||>LP-0_{L9$MbPZotG zSW*|mYO7tC=&Uc6@nEDMy_0NM1qg_#3Lp*)N@PqoC|x3ucQL0NQvMy4D(5UL`D?qJ zk(|nm$41mhO%v*-0*To%!uQlZb^+6KNS8Z_*8eBt`UG#9R$K3lCSD9w?WBEY6oEgl z*VsqA#S2`dV@Haf=l-P(rdI!mZPV6!Y(*d`@)`CB-%P8?C`C<0am+1O=qY%Z;_aOEx@Sydybu6NqH?EdL zG?cs~a)QQk{I0%VuT?~|a^QRyA8AS3bYRe`b`VdR^6NT8g9%jb*a~v((;H(3=kFYT z6dMl_8f(eXm5<_gR9Thz*Z1iGttULWTZZ45cBuJzTs^_fEqUokuWR$w#})MqG{FH& z)+q0T?}pahm~d{6!>=sm%__i&h;Pi$@G{a$4iU~rTXeGdhHD{se~oOoW^%(-4*sjd z+M?qM-LI1seCJ>3byW-!as4VdG8&jI5@+C$_a~ZR`p#ovKKVv`A{*=y1TZic`j<*N zfcNn}3Rc(JvekU8JddD@3mkp`_l_$Rj|PhUbDA+mw?S+<1JuO*z6r{u4m_1!P6I1> z$vnccd_2YQK$4l(y$s8|)32o|#WD)|J9AwQY5dJ~iHV=EXg#w#{8|sF%^Idjb=$pF zx&xbKJH2UNE7Mrw5d)TvGS{8*t6^yH5<0V`rM(m9?s0aG9+RX$BF}i5nO0G!DTwKK zT>G<30BqILvQ?ghp-##{>1fXaych0<3BpFBWc^OwnFfuM#UstaYH`A8p~*Q=_s%}X zlwJ#o+|TjZx(R5P#0}4%YBl;N+pEqhf?ECNgKoa~={RXwIJBJhyh6uz0yZ&%ee9g_ zCg+x!_evMmt};<{nJP}^2t5oWhvuHKh&_jcj(LbKz*#kmLNPfs^gP+qjHyh1$rgtQ zg+B8j^nnY1{prVa_TPop59QBKvfNE|k0j*SBK>Vk3jFc_75kYR45Vqt^qFZGsJ2Bj zBDDEoaJzTkT?^rR^R-+q2Y4M5MoezTN>3iv$~rldGr-oI4)R#c)QKB9asT;|STH=d zb+=zvX>bRXz?&HnME{|qmP$L3`z$q%#U5^y)PZUn^ZOl4%w1D))++w>`QMvn8M7gd z=S;G?#;QELJawt>f~JZhzc1~}sQ2lOstcZ%`mHsnG&nwaTZMD|F@!s(iKFa+^MlSf zlQJ3_j=TKrowMWCF-I2K--t{nOa>hIYP9}z4ML6Fgny8aajA>&d8?8mJlAXYwR(&( zQ-05aaF5cGo%uN6XF5aDRvb&AuSQ3`Z*~KS``s#Of9Q7jKQED{-RT<5#ZnS}V*$Hc zh+(o=+z_O7^!bz?iUCe!jzR{xJW!>1l%M9Ln8>%Q(4bte_-FE?mN(`<#T0l^3H8Kt3*g!f%J9j}&(1E{&XVzeY4NXd+F9opO_k!n`^Mcter1v8t%$cDW^@a;&G)h&$Ru{FSD7!aFCgKmOx~TTRT)@+e$@5p__fh%X@3)aQ`9fd2V(oYB+?UZJL<5igU++Q zFwt4G?JZLo3*60Yx#V z7x2l#*a2_6C;IoS$j`Fgez(xZB0`D-Z`)6VwB+e4m^0PC5 zC9~XPtqLbfv27mi;LxE^ewC?@RZdO8jt;(2zx|H;#R|ALOh;yswy z52>72!fZ83LlPvFxlhCA+xLcmCIf!1&U+6}-@?+M2l0ATf)bVZUM>EAu@#z+%-z6f zHxS&xt>jkbiTNxCGl|o_fe)?ss;=kpnp{1hrZ|9ADG#|)bez#gLQs9Lcj zidf%shq`a8Tfgk^bf=og!A7RFz4PAXnr*u*$;O)Wm|Oa3Tjw6ow(g|6i4Rt^D6OvP zqI~J)r02qLHDkxexFVOL^BWe6Xn5X`^$=iqPy%L|Ia0;^rHmMuMreeuUNs{hp|E-! z+JecYIw3=E>j77E9oTxP#VEX#w*23o6d>f-*Usy(Q?Iqjm&i-q#g6Qh>mRo|#pGk# zzB|co%)Gm?R9|w$!+Q1yAoi|35B~&sGR}UZxxarlWXpm9p;kAU=kJ~T>%;YeS$@&`IXG?JVEh!VoIN>&Y_>82BP_MCak8-RPl++Z&hcm|)7>8J z+`2oO75?)Ry>+c-G=ng{Dfbv`;`U&`r7YffXG z2^m}^=*L@t*YVv6q&@8{;aK@TlHcGwBYuWd@SRh}w9<*T6jlF(JqE#DNwBL$r>Go} z(_w7n!J41dW#egr^CG8Pv~qHdPDPbtte`_B0) z2)TJCP10SZ7Vk3YX2g130Kn3MEhKD5#~GTq|=2e*vHXL)0%i|9Pt(WCC9w679!FXoX$>bU!^q{ zuTEpL&%ghKpzX)~6Bb7~`(qV8OO0duu;*OZ7qSiVZ02FxcQKnHkDU4(v=yITcb><1 z3UOuZGGXF$W~(uP`6=Nqn0r9kIbV)TF7=N0wR8$%h{g%NLtCFE8vr^T-*mLS29X2fU2Tt&|EYPlu89 z;8^N=u^#i+p~+8qht^p5R+^mTQ66_gC6mkd47_QB+v=@8;2<4ys0Kbk~b(v6(`;;tkR8RBy~-DbMnE2Y7#RC zX53?M_Eiq!&R%O2Ul!>zE#2Kq>rv3+Hdj~e1)wH=0b4fqvxX3ia@TndgSy;9M!Jk@ zTT>xsR99d)Kc-Jf&66XtK|~u7D{xiVMhX?Hy*Mlw!Dvqx5yj^6F znS>jD)Cj|-K+4GcjS5;!G)!$)>XXL^9M!#Mc>T{6_9wIctt6@FOH((Qi{3i1Xm`0P z;H0B%-eRx^1%wL3Gm7G4( zd~I30Vx;n*Zn(A2>qYRTYj_J?blz*(5AEx&26Qry^55 zpg-vcbSS8b?!V%fe}M|2y|EQEFYo_~SeOX^-_Gs-wfa}h&CdLv&D_lF>|Foj_(j#5 zQ`M7zLD|}Aj10BQPbP|0WAK!Ch4_G7q$+t~63cmVlI!0Eu-snMl*<7PRqS1^YOx?* zu(+IBOxbQhMz*1cVXGSblC7lU>GtE3=V>aBP%1?VaAFmt_Dip4VzeTAyf2^ zVauyhYeh~z7X$)DFbu0gGl3AQ0)bfIVng<;h9ChyV?wL<6H5|_)5^DsiSseBT=;_Y!d6fZag~tc638 zE^u}8D(I_5FeJ%$gBk{bY1RmiwiMP)5heuA%tol@<0e^Q19>CGO>CaIq)3ZJt57>& zlgAg1kz)JsA{78p%hLZERDqW1vy?J(5jGzaq#AYiPf%8^K~{V|v}hoWICZ~Zgrp&IS5P)q zkd+PY457b(0ew8Sqy<8;6Jrowau~&arbR<;slp!~2PZdh8x6tnMGQ0#fHD=Zwh+j*Fe9Fo+w4g4AuY(x(%ep?Vl&le!4L96A;d z2jTsQ$z&QDB*#M`vclKt9s^0zhcBZ3B0@}!hLJQzZoue`#0CfEh8uDq1|LoKU`8}{ zg61+M2Tq_Dix?ulO0EqINzkcK!aHDTh+j!No*X(#$JCF4jF|X{Mg*zGm5MP#Of*7v z+kZGSnACu~7uXGaw{tVw8m8w#cG*Sr(sfTXRb`PY5Kh8 zoDI{SW}AOhEwA!jOk#r$PfX`#pnh#zjpc*f4aBZ&YiH}+IQ2X_h&^cO@x>Q%85y0F zrnZ*c)l>{0uYF+=FxOMCJCC3r)a&&P)lWRyBD=ZVdO39{t>pHFqnvl1UzC10J$IVE z%&cSY{k6fqSm7vtGA=yKb~W29w_e11Fla-$-}8P~^zfwF>T+4n zJgj2XwquyNE$bTCX7#tBe2u#9lP*BhcfFMsE_cVreYLRP?sldod$;|XQ8#!S(`Hm- z?OAWz-8_QfFn>$dhzr&Qs@7vU+bSGgVo6?aU~BHEKax`DBLZ zT*i0zPSG&SxYG-FMYw9|o!{nh8#e$w{ygx@jVuSi>rlp!LphZxDJyZ_t}SXupCY(< zcXitNKckyc0)t^lgy22Qebkw{T9;+oKIxv;pC2cO{`PF6Tfjq?a^J3W(6$d>MrD54 z#uA4pV^?r7tO>3te$}n?Wp2JXy8c+PtD@q;jAy9s{!8?Jax;WK$oa3p?~ zwk*r@)2Y~G*D*g*j$a(#W~92Ypmzup)`2{D`t7zwRCz|7Orlr;4u+-1;+&ktN0+`b2(?K7GVr$uMbBl$r9w zY4k}a;%Ol#v|PD-zW{7YaHzAw@BPkdG*+7=*NwHAFn?h2C9kz=snNmny{9C*F>CUc zTojrW%>+6j<+pLuN`4*B8&|Ay{en&~4NCvfJ9gv}x-wMaB{bc))SYOxr^wta%INy}dn_BmPHyO6I}Xpwm^Qb8h|;S2 zVBwCRZ!yN!`HCl-7VVCJ*kAoD$HAHJLN!Y98RM}=ro-OPcVGyhm5re04*pqg=R5=d zb2p%Np+P3L3}H+^*|XYp@V=FP>d50Na|jNjz-og}>7|-?2hn50s6%t}c#ow)pxbDB zp>>>yUY%={pn|G0?|eD+Y4>4w7nojPr$U{ zt2sCOUu%dp`GXhM=(R2H(jS%EeZ^z>{N>s%X(ipi@_^9-IU~0=J?kCjVje;cGc-0pAh_>W3$0tEYOY!3AS?cKL9qi7+ zPR1)I9mjfgwc0w;h`AL+X*kawe@9F1E)FlLVw?Gc$v8Z1U-5j6L%VN?%!;yt?|b<0 z%ZFNP{NsnG0}%QUY}w|V+Gd#^;->XU)}?>FBiBuEl`AY0H!uU_x!Z`iBL4NE7;hNb zxr={KhN+b<1O6zRrfi!D499kDc$O_q`QBVyVpKg=^JLjoq%y386~ME?*=a1Cs_jS= zKf~#!D>0E>?IYzg_YL|Ha>sQM%^j~@*ED3GOkms7+b*B;eeuT}3@z2faUxGMN!GSi z<1q5gn|hBqm^W_ig*C925=>9(H29AHCI@t=Bj5z1Sp>h|ZBiI-yu2&X`QC9fq@L^W zpk|M=FzmQ#`78V3Y}@BP2R8p1SUHTddk9fi8K@VJ|E5O&_DeQ<+{fVivWCm*h!I{r z8#lo2B32G-uzpC>Z}ZtO8xxoL;Vhl5imeaxya0IK!4nJ<)^aPrv@ogb=qBec0p`s! z)O}z^!oOby1h|#3{|j>2{tI$B**N|Wcl% z6N|-7+D%KGSu!7&B^=9aIwpfF*F_H492rW~;?SW)Ai9sOzg;Mh z_tz*eBxgYly|~_1CTJjJ4){g@qzN=4EE2Y0*ec}UV>l|(7 zLNFHua}h8>m~5WE4Syk*2@VU)AmNJ!64tdn4c-j6w?t@n((t0t7@sv4DYSfE>a*fe;$1aj@##H}?0WmxuEs$9UPZ zhTg)X+19oO{P+w3XZvO5cGOz!>g}^ku=^(EU;^*QwVVdbUs= zeLO&5h8%-?^Wdib2PJ{bpx?z|VZR(SC)+>*(BXtuOFVomEGX#{M#LOwSuOYj4Y>nYazOfYB@sfRXSqZnFkv8AO+|DIkHDdEq4#gCDmB z-;|DdTxrnX&G=(REf^nPA_E!Yaqd+9eKu{OVUFk9Al>;%z`$m)5(4-%7+JV^!++E2 z;a=K|U2B@FdNZaD3U`1WBruvx>mK@Dvnf4BTA=sDDpr?zaGrADHXGYJZ@72r%PVbTyb%Op*W{tC5Y8m03I^pI)n^& zGysTJh@iC6Tr#`W1M;8$WgNRJ^E7AzuKD8IP6SN`DvXF8V|F2rMaaV34-g+G(u>!7 zElM**wKGq_hds_uo9)T8Nyq?nSj${F35Psf`|QwkalEpoK-Kqe+45Xf8rBqkP0p?_r2q=C0{WvKkehnB+$5F+X?CB_^D zE!4&?p?}RoGR8JA+B0DQquJvMAS+HeDrk>&GrV)J>5+rQGx~!81ftg=ZyiSLAi|0BJaPKnH1~%mwXge-cOH)@ZK3Y22 zABD4_XXgY#x-;96d~?U^=if%_S^8r56^zu{DcDNe##A`Q{Og?*GXq*jw;7F zgT0r=Io!c?p5$CL%~vmWngef}b?j6#cdo>#1Bd#W%maTOx zBVnAnieGQ%-vBw9Qw1C`{WjansstoyIgdh@Y2D(Q1DE^6Jr}Hl#UyxJdGGP5b~!n| z>(_bDcYTA2G&={T6?X97mL;q2ODyrfy@tH_+KCgZ%hiKtnX{p?_Ses9;3@=s) zvG2YRDD^AaXe}*LEP~))$ZfeiZVIl%#@tSlOUPZ!zt*KUnB&<+=_%*$aKARdvhh-V z&~{W%Sha=IHXSTkZ3^>k^AFMJaiX7DS9NUkX@}#Xv zEZNuYm*xDry!keL2PlmuAQSSQ@;2mbY|Qc6{S!TVeQU8s-I zZR*;{2UfYp9IhJDC!k^A&TQ)2myx&;4n_|xTp3*P{HYgiQTRFdGJ;4{-eXyz!Pw{) za>-CL=E(ZK?efrvpnGM#i8ZV zQL}jGn5(#oF(2VY)1d3eCS7-mW%502Y!W(=yB$B39W)%z6F;{lEP;qN%F(fUGE|RS zU2~MXa{aQMs|54*6Z^NG!Afe<)~0>oTTvxudCTI!?kNifb-%EKy`>aP0RA5*lBU6s8YVuWA%jBHi&sx^{#@;(I;=l<1prvpe18nqYK zzPgpl?MzqK5u5auBeL})WVrTwt8D#=wA4eL6Hc3t*&bn^^D@d-nFcHqkE&`(Z1lFe z1@ttg8ag9G}rkp420l7Q#;bJI2yF${J-(yT{E=V5-aXdb34${pB>9{NALNYq*B znb=YF<4SAG7rxI<;Vqfca=4<(0!df+m1M4w$V_X!I_<w_)6S#~zltRCU0 z?&>G0dcKo8ltxD6c6~fMyIyk%zTU6GQ3u`b8`Hu&=Wp>XIT5wkkEd3n+&G6to*3F! zX+EXSTKBOc^Jm)Y$TM{+HKTsZ@S_LhP)U%-cVBqO2(j8nO?o`U&yQb`H7&DMQ*UOa zLk}9?V$RT0UDUp(Thj^u+y=|MWWVb9HohI$X6fB6QuNh!G#-W98yt;QXBuDRKHwYa z&Xi7eMs~D)S_J$C+uM76Resl8KdIE;rpT;VPxD?lr?1=mNiEtWdAzjEc$1-96Jy6r z@8dl%t}x`$UPjF`qDe-zROr;uDrXDvThZ~WEPfAJOT#coVEGVLjxLxF!;mY} ze(q6+d7o@nWS$RFNY*#Ttnhu{B*$lOk`lk8JP1(y{6%5@jY8vjB!_{9Ya)B7x8Gzy z<;6gNyEQ>Ba|{I4Q7&BSXjEQlOuke~g3L&I{)r|Y zXNe{IlbQPKBhwwfQoC_>M4ITmp=Ocgx7NhcQ91K44E*#W9U=mf*4EdVFwxIuQ}j`_ zVC@NB`@zU7b6>HYkyd&t>O%~E#zRA55`F7eD;|M7!Mk9?rBB<>cXc@F_xw7I zrq2^bB-#6WivdBlPoD`x3l}~g`(rZxGPmEuP1yKx1(wsNkQ>JK$h9~CyI?4z`Kg(D zM>;x{Dlya>u4))6O^cdD_~l+pqRL32wXk!ZXpqywwZk`F8vO+fU>| zuB+S41Pz$Mu`oQ1;AXp9`X{coC_>L<=u&1$*|CgmwvOSP&)jWcx+3l*RX8pRj_w_5 z0^Wvgt5#Iu2)(9LDHodtO8zB6UvZ1HYu{mAgzZ%v%+uYu;3}1B%4QEyINIMI!?~|! zG8?#pm)Wn$SGC1(hNC=IbnKr#wwUC(2w652{IGlRd7H1$xIrQHv3z{WlVerpKm;cB zSe2&sCbgrL{X}fwqo@SoDV5K9yGbX!cF%*WrMc|(SC|+dRnY*n7x(Ch6pdffYtD4p zBBMC&#HgP^$o#EHxjve6^Kuynb}r!FD71;iENXMRe_X-z=t?@eeE3$)erhf(BKyJo z^LMc3%x-KJlIY=k#o+yZIW9w(+rVTiYxv?u#mvxOPw?}NQjXjyOd!Y*^w;N-N@a}jfN)-1kEw>Z`W-$xGdj`zC< zXf!}%L+X_mdG;mTu7N%5@?9K%z<8VaU6`^856Xblp!~^l#WfR3WM(g0$||C6mk!r- zTf%W8u`i>QqgnAu>qgA*WF2btP~Km&u7l6vjE+~9?XivdZfYk>gzqZsn`k^(90N7Q zPs7SbPr;-6ATdq=9-5xX$X)EScy6A`Lv9QH0baHr4F{)(!kXaeG^?Z113Jvr44U`O zm~#7#xSoxMiih4`(GSba^TQw*YCBW>`VMYeYnurI1uXm(Py0Vc_iH+wWJf-dy3}-R zV4ll(1kV-p+#mN6A;<#ftD~ODNg_ym^LmU|M$+Xw%_3yawtm~fZ^`3-ih30$>Lplf zBdO^lSvUQD>lhl?5WNptz&a+na^EYFmYLd68%dnRoQC2}zZh^{b!+6U>}BtqtG-*H zmC3Sl-JQGT>T}D{>(42*4cS*6oz7OhnbYlunZhJ*x9voENX8ks{`Dp8xC4MH+0^yR z#myYxF#clq_vk3)nRw`5ZU61t71aPILBFFBj*wC9W$hk95k>mX zFIl!h(2NoSQlWD_p|FTg;YS|SKN(!m%#sz{@~-*rd92`pUaDYWZ-W}UIXlv-2op)M zzPWh>jlO-epfnEF(6}6VXpv~I3AfT2357u?wcm9XzlBK&XVXj2f@qLMse&iNHCPry zbP}0{3Gn8#F5dZT1$GP5*IKy^O=ci@=Z%`lZ8vDm!NV}sv}x*9%N)+Z>N{>`QZmjf zc~u`ToEb9o1MDHlV)S2o8XW(%r@_wkf6YxXF?0Oy)&D&=#mvI^|0B5bR7F|A7uT)( zUGOJJth$wOG!;$@OcxT79jb&}NR&68s0|8`n8C>cOdP$dO;fNRW^CQePotSqS{6VY zom%KxEt!J>v@=gV*92&O!SM)~&`V>T&fO#%fZ z7!xwoaM7sjyNR-d!rpWkb|HLWK*9+fqJyrkFjWK-r&Nd$EG(F?06F;RpTNL(6G>oH zSut~?a8Pjag6_%4(%7=7HBv^a*hy`= zEy!7T{h)K{$}%N$+PH=x(4r5dyofR-L?rN=k|3l5sIsNMim*qGcojzn8%+s~pJ_u? zkEcaKO~r!)i?)rRQpV7b^a*8AX)r)BPr!*$nYlg~!om7$??CFE4$VLQ2(s%Dr$!$- z7aJ4hZGj>NsY=dqZnO}vYlOi11cWIsp_X%S0NGK&0w50`7Ut(Yut}zFC8Y;gPA6Xv zsW{>0C8c3zvx2xS$cH7%5OZXqFX?)7;UoG|68;cnp~^pVKSf9Zq*X zz*|6mG)eVYINQajVuL&oijJoHGNZrA&Gm!|mRw3&a>Up7cVHqf&73GGHOlf8l10|=Vg^_VUM5o0{h#38X0Qx# zlg}iLSQKhDZ)obmz*@rFP;pGOXL^VXP68Y3Q6S&|7+_hAlH0Xqv!gmQB5Uzv6H`n{ zP(=VYw%MLlgnl+GwoGV1eheec0YG90P2v3n3fkf~ClSVx8A}Qo48wu%+-p%3e z@s!BJVO*HH7&14Cq7Yj+fef=GuH8>w#>EQFX4%4!vDfMI%iweT@Z;a@YbE;Rv*5?D_2f9c z+lT}vTnGr1tCVE~Gk+z0P0~lCpM~S&K}6%-ia8(shGNS7AtZOMOIJ*!&YlIbFVc)D zU__A!K;6H(fME`ie5FpeOGVs$fJ_vk>NPZJ2*@nrykvs;@&Pu8RFTG0Z8#aNvy(UA zRd&;zI!EyVdcDSHsWpWav)aSe4Yj+A7e=&zR>-3eoJS(&dGGqk-1#RzhFJ1X#zPh^ zF?XPV0pb9<5~IxChxz;hZFo1K_GUnBv_Oi#S-E5dg4}2pwGkPA;0WUr{Dy*+j+|WzOZ8^XLj&=F3~ks=$vSnYv<_E zu-WuKi#(s$WvA6im8}Cqs?fvDn1ay2Atz`XB$F1~Fg*ab> z_Iy$hj%igXF?oT-(P+~6{Jc&xGrU;+Y)iN8eye;%vCJa9GL`GQekUjr_nFoN3{J0r zAsOCG;rn!@-T_S47?p@D;41yqqP5-K9eZo<@4Hb!T~uOLpxJPtCD=OB!Z|Z|xBaAm zzG7k9xLjSA#zoJ@w!>TyN=DyJP<4wR-IC;LD@c-C!>^H0v)+i|{54i@8F6( z7b2Pb>h~Sa(W@iiD^6^9GaHaWra{kU=j0N1B8!5M)6Oc!Uu~qcmfD8~%*EuUQR#J> zzki*D-)%!lP^jr7&vB6+es<^Au^u?yvM zYnZ5ypUyfDVU5emf8ZkF0Ju?kru55w==pYsAfR}o^{yj074AUSNuOoyr?C{Lu_l*# zJkQ9~`G;d9KwQHpUuPyRqQ=rdAv^D9i1$Y>YfJIAIeRE59(NGB!o+AY!cS7VA>N$ZjM$dP!k%sDqDMXF-~1M~ zhqUS!iXqrGl~!kwS3Kt0v+9CBj>FhT1;cNqc;4c+v+Q}^T2HrOrY~%EP-;DAogEHG zL5{WSD~8UU8n!jqBE}oxkf`hw}^wiLRwX{Bia_HnQDN1jF23)JyL&1yaUB&lzX$k$NOD?Pce^*>+X zCAT7ZJY-yNw74mJvWty>f;?D>3j5ufr}Xkm?yK%UL#=86ySuUU^m+{5xv@KpL-d7f z=LQQG-w?v^h3fwWUtIqMUo0%F|0hsl;^g|@@zVbXzBt$!|EGactO}HBj5m%#KCIqL zOM>!OGiABfMr)<51xd=h_`)R5ly$26a>?&VWQs%_y>gp08aiY;>9R5!;v%_d z2GtOn9Uior58o4?oebygH-VW4@8=!B-~N%A$;dEKL6Z;?RuaRzmfnq-wX6UL4_>I~ z5frAduL6tp3= z&kGtS(-tn83G5qIlM!$$aw^OeiU6$I7gUZY5zrpN$r^z$d)Yg)1!J=-oQ+kt0^5r^ z2@io-7gD=t)@2%KINih&DlN9E3oQZSObm=?CS(Y@2165ro(TdSP7bQ92?F+1Dyn3N zP6h%lnV(KdaU3lg&c)hZ1(z8Sy)hW*Nj$pciXL3Sxa0CW0a0^T3Jk}ZIKTras)X60 z0qh%@p4FCgbkPD~9+SQw+@%Rl5_*B?5dnqdkEoE~D2C8{dJj%_8Y39UpFUNhhyY?E z_+uRxRk%5G6PF(!PVA6frFSGC{pXK&Xb~)N5l-4Mj@QR1N53r4?5-~LnU>F`)5{pq zU`!Sm7(+98QBP@R3SgW^qA&h91n}JI1N-jle@&rPX5(TfkHv>&TJV@N_@mUfzEw|< zskLV?Jl|!BD6tESxz`aZ^!i&0LN1chvYQ6Yd}BqOkJdRp zMp8h$w^GI6Y+~H&{RPC|`OfkeVj;eyZt}`uQJneHm^c{vSXz0ve*ZSzP$zksLqwS zfOSZAGjT~N4XkH2N%H6SpLvJ{I_EEZL8M&&3gWQFHYl-5g4i+Ntf0{5t-YFL$pLjY zw}6AAa^@R}N8l>yDQj7Lgth4qV!B|2RH35dxS89JPK=#UlI)~-$tlxtADry5MW+QMd)!pGZqcIQ}; zn{W2#Gk!Qnd=py%0DqmSW|st!b^6w>lnoVTS5HBwTOGH@>5(P+E&)k&j)55E<1mQLpcPl-xs8KFu64Df&+(qxyZafnhM5TA>t&y(0w1*g^=R|O|(=Q z=S%wAqvZ9`hiZvqKX;vV`>sXWX5ziBPinsE={nW@e)9D8#^*V@$q#*TwpvdhN1FN< zuFdp|;A#7>_;!9gmBAf+?JLx8+N^gPPE1W_ih+(s#-RT;eymuLDy#}XV{2E*%4oTp zRPB2H+*hV_2UzSH7(Pa4rr@!w5gDz%Jhx$Nz1ehrzFq&SFM9HH?(4^S&-hrL^4!{& zbjd1xnbvUKOJYf>O#|31WTC$n{Ze@;u~3$aYJSo4r^^Lksn#ux(6;!F$V8SuOwpTJ z-J~FT?$9n7Ol2UD+&#M=!>e-+NhRY;J{z#7Ieno1Gm((c-Mw(QP4 zDcPezNghO(Kh;#5w?fiBIMYVQ$MKRZldoJX?TVNn!UGQfs`U! zD9Tp?z)e#UKLd30srBn`;pqLPdgXu^$Q9c9u819Z`Uf0u8H;%9ABwpN8`yVE1sy+V zRBWSr&p1>gw*^^S8Gl`Two31#2ZkOloIv+8*RP|7X>QCbagE7szSe9m`dg23np*Pr z!{rqsB9pN4*{FFR6fwkHPcd7csg_xJ8^6!n%;R~)bC(eihVEMRpBH|VYQrykE+T}L zJns!R3o!igwvf(HP^x=$9?j-RxrRlK${&!6u}6xb ztSa!pYow+@h6D4i&j(B!NqgEf4l+iRj`cCLPF`msg z_aw6d1T?aEl^aegc#8SmgM5v1fQ84*_*>jm3!W4-X&hJ1oX@Q@YoXi>{I6f~oz9yc zRUMoXK99u{7&Es0I?fW8r4goDB0($diw+e>vZ=9yD13!aa7u|jN0@RB!9;Fr1B}ytB};m53qBoB;!8o`~`h!yN;8Ldn^rX z9gO>mL~0g2OSPih*;e+t5SoYQ6388#eg+L|Z>s_sSB2Lz|qRJnfKMhgQ(sA2};7S>BNrUd$x$YP-u>@T}yO(XIfIEwg1)F7JE`dHeLb4obY( z8I#Po9AKa6e64EK(xX%gUoqsa0-+^Sk%yQ1L(e90cZKMu7V}A? zY!+Ex%YZ|DVRb0xFJX-4^%Y1Wgz$xC|QH z26u;GgKN-1gAVQlCkYlTxVyW1aCZnKxZ9im{O{a%&bw!=d)Mx+uB!gJ`|IlN)z!PI zYa=I1o5e>*Bj6dJsr1n1URCNs!3;<2Nc2FgO{6`nO^mmp2slN!ldN#7kx~toQ`Q0E zxn*~7 z4RwO25N{m}2?oGzQ**o(qzR5FgBy@yalJS=ux28p$}!zit!G3A(KNz2fzerFpm^F>*9Wl~x_2MNmZpE&P%cGsL+wPmjYIq5vm@@II88Lnjy~ zny5>j#t@!>XmAUZhISf34JPdlix5@+h7<{fXG`g>ZtNhlT$618G90H<7@P;_)>hEI5veB1@CN` zq6qHK(?bYNmZ;+1qWwxbfFn_Gk?rg5E^|jxnJJ%oFf)*pTR;TnYMr@VhDbITv`Pi&+feo?J*}ULz5eA}}6{4}JQ?WcJRKjO2 zX*s5a$5)$th5 zi*3WEP!0-2SGkNEj`A3xQcb_zF|V+`DNElmEQPDksW!6?J)Gr#psWR*k-4Y*|0;9Sk?9eO*o zATC8`{%ne0f6KAR?=F>TjZfV~1m3^qtS+s8@h+_0j;rq-T#Fx1JQ)&Hi9!AJ<7FfF z#H?_vKHMIYul`Z0&Tc!lGWz>N8K$Gw$>N|{EM=K0slUgLp|#qh=cw96>&ALk6=H~3 zuGx+$@xbKy6-CXgUch@H~ltp?$+kT_3o~D?Cr*SZVbJ%iiC=6 zqUNrI$Gg z+hjt>kEOi<>z_#U4e7->Hh#{4GDBY#rQt=TWa=78G@CfWk9U>Cp?{ zoYt4uXg3^##l2LXzJMH))Vl7kGC=}=1d={T^K&rWUyG1{S({;9F7S-;>y`b*6 z@Z+4e$kd|3X1WtUDdn?3;jzPU{n8MhOt+Ie{n1LR6_5AD#AtN~>D;gM@AVLcieEcm zS-fu1Mq@S$(ubCe5;eORyLR8b^VtO9LL;BJOp?tqe52~Ij+4yej17UT!etq38oOsB zb`p%%DH9z}@dxqEDee<4?S|4Qg(!uei0;HO$@{;?{8o4^ty{pJCc8a8ZQHqpt#6Jg zMyM963)GkWe~*7rVHyuoN~NUasx|XO@=qmroLitybQAN<)Nd0;ep<`UGSf(f9IpC3 z{i$5F-uaZwg&l@bGvC0M$%e_*nEmi$XVgN=W->^eQ%KP4wctzHgeaGPd8snydP7Oq z-t`=`ym0FXb);JgdLOPQC;jHa4LP%EMWjZ7+1;d$&hnYv@7tH{8ud|8-=#WGo= ztZ>z#QNECyyQVi&MagiDwnRvb4;KqI_tm5ly9ZABaNYLH;F4!P^}sEtczLLYx?YjD z`R-@$QmbZv#8|oTQ*#;Xm&1yin36dn!Na>%rMH>K3}n8fl||?Ix{v|xH}qL@DLP5>12fmDaI9yEiA~p&Cs6 zi%2YE9(LzE=t`VkQZPP9y-K-#mK0UZ-Q|v0dy3_$F3hvyf;^LXR=^6UXCFTv zvK5F)66uC763qgGI-f3q$g^kD*knVr;wv7FYKBA~M-UjPT|?sV%}7LoHvVL3Eh;E! z3d$jqBuNN8=!O$f;NH*1D@PDXh_$A}g`YyvtcK!!(jvD@I{hn zp;3ba<^tf#UxNVxR-{!>U6eEv^%0W~@Dm6uR&)3zv_ArfmjYi!>SErL;vQpgpfNxc z^|p|l4S*zo&9~A{aC@|CBr@NiQ&Ex_LamgtxUUK8x*8ZENPmNv#Uwh&7sK94@m_GUtG3Hg92Gaj zBRhzn9r4Y#u1EO*>~U641kO&n(R|tixWafr3{Q&{I9@t&RF6jQ${j^GvPO*qW9$>L zu=v-oU{VaVRA5Cfh${$Nuk$-IvH&gWJ>fLknwfhN!W(r&qkOTj+8>fRJ?~NL84YkO zPq85ZdI8AKzz|O&77Ps(LxVucRV(8|@#p*m*+iHME+W(e<#V_9f zgepxK9{bhBwz})n0QRb}Qn44}=MDFR#PfSEEn}m)xF7iIzk#R(ZPvp!!i!nP%FOD= zlQjijd_j~;Ia$kbd0>tKttO!ovU76Q!_sdo#IDV0I+x7`+7~u%Q^75CeqyP$cb~*N zPEe9$ct%;eMljDE_KjbfQ=g2D{2wpxT&+6Vd;G$Cgt0NeyOYjM_ZAOjB;1r`C>N&N zgn8dkNjy%EJzsX^tiF-*VVe;~2f1CfqX*uM^po;guK;}|*QdC=m6<<0xoIdL_93_F zuCN@^mGwYX9JOnkK4@nm-)Gq_=1K?!M<&Kq7_|u>{HXL$g%5o5Bi=#b$Ubz{Crcz} z;!wd`!WfZ`?n#;+Qk^y?uM)ez6KV9++FraTHuf^Zm@&nlG2b~w&7sA5)W?s6c)Huv zyx5uIt*S{OEGq0fU$y%Otl%C?1C*0kda6;PC5i~Hp zS6y5sJ7%&NS@)=Fh5WrdADw@LN*6FU!oCesvo09xJ}D^I4w%bCleylGE zAI7rq$Pt@ra}Q*)Xm0=s*v1*{zL>NbNn^8%6EHbp2(o5OGj7t8aF|wt362GILAJkq zI{uBh{A-#fRKvn=#(w%KiJ~R_T+R<92MiSzsZo?vJjo-wTPJ{L6iMwF7dK=Ng-Bz~ zXIKAu?51?(l2E7*3IniZc74Q#Crk6vMK4*Y4y$JFBFfu{PxP?%Lm3q9#F8(OS)gP; z{>hSWrl39ZIq%|Ewf(O?A#3cjSi(-t{Y!7X38rS2i6eBys$wgi({C(*PpY#E{{H8? z+g~B@1LQ#n+Bly$j{9TG^mW_tL~nlP?Fs+D_YwBJ+q|`~U3-&{H`1=rj=a0j<=7)U zAaAzg2v{N0L&Y zU%)D`rt%EvK+X*t4KdG1xnS1Z`N?%L&s~I}Fw6JZ)5+!WVQ8@nof&0GMAt9ks~VMa zi`L2#T^+^fcAwZs1^mRm%clKRX$K)Tn{PrrqkC7rK_ZAey1+&7xW_+136!n+aBpmqd8u`q%-oAN}w zhBzbt#4Wq6)Y-Tli>sgoXt_h@;^do+CaSqLjPaw{tJ+8ORkpnIf@Xu1nYzL0lIpt_ zk&08$?%ZQgu&0mhQM)l>L9-0^k>A^{_fOh9bZsoYHwH?Z?afBB8zZWxEJ*OJbgkL` z5;B?+Md`kRVKH~FHY1JZAzpO>jg7%7U(Z9)r0I1;IEcBd^_nBXs}0jaCVNB*V;bw} zs6T8~^YFO9sr}gwwU)`k?~z>Te9MG{UHt`wU44IN^f|~%?P(#fy8Q6Q7GdbJU7WDk zz5++z!_Iju^p9wfW@U>5IjCE4g`B#tAqKp{fX~)a&h|$hz2V3YkF?_>S$OS7jmAH? zODTa2>DLLMiyxm(u_S|nn$Rbnd2bp#xW^}nQL}yvSgbri{35J;K9akNmh%Wpo7EY9MSjke&K%QpSFjvK|ys@zu#GBb7jcZ3lL55GFb z_=BjpRZw?}yZM6KMZE9t7LV4mzESvG8>&pJ>my6B$ryxG@7nCDbaxF&-kg$L7l}eV+eJ$l16AG4seR9=s5e#lY>cI7G z4)}fn=+gu@4(j!D--U{d9oPp=cQ1Vf3xQx6&i(Oz^}=d`rr+U4OxT{y)u14((x&E> zeNW*RM(#(da#W>Z5;874?xt7PJu>0G*6tJ^@EzUBCqv`TyeD-(i!+C27nO+#dOck3 z+?li-`Q7Ih?J+aVnkidx6)FHYiNt@Bg07sx3n|xI{mx56NYcprJahzNY$L`wBWiI_ zZ?}%0&QgO20hAFRcr$-RGH~ZDqjc76tDLNND8d!v`RFm#^HOmXhZQfNwCBz|Sv@d{ za6wg8)tJ5mfcjk?`{=Hs(RZHVy*7FlaFlB1Cj#QXw-e27x7=my5myaSvwb1nNo};k z#QG*Dv55mB?WKMkc!-2}?PaI)qQK+LO}(i}=P*+mM~!p5Lq6f{BjyQ%)pM%dWB&zV zGN3NsMH16XgVNH*oOI1cG4f~3D3R(?_47P+PI3!`I#`kTKTe)ZaW-@(uE*$ewk2Ot zJdcEkRI_+g*Lur5#VrndKTAvqx5n|@h`WC<(k^*(Z|)oMK-73^hb#Zf4AYCJ> zsB5xDd=oB-V4Gd|9$&XXkc{<)$v(ke!sdWEwPX=IPU7#`)qKfbc})rJjd!)p&vB$Z z=Y(bK4M`EGYRT5Pf5si@PEZ=SJv2*2yNwq&t{lUnhSu~n*uEO#dn=qTM-_Z|3hHv5 zoI#H?)I}myAd%N(*<_fi0v#ia3H&G+xGHt<{xHAIX}uZJi>@e?FCmAZ6ckKv`$|5c zf(15H$Qw{DeK_*|X$hu4cbW2`Kuh=@PKQOuk^o08(MKUq*h>g?rY1bXB3w6&Tf~yv zZaAYRBkZlwC+r3B8&qykb!;wjWEQ;So|dp{vY`;%!C7Rt@^K@QcA&g!rD0BSO>%^z zRV{aI0yPcoV(jv!+9g@NzKp+WC*eVQzvmJ}^?H{r6>+;2GgmhD53L@Xbia}VYV+%c zT=%rqXJRvVC)%}LcPGfAX%K~Q!ys{FE@F*O&U*p+5m+=%fZr62`p6e(i+0~Q;+vXV z(n66Yw&lgv57~EVMKEg~oL6y5IsytFJG?{z1!=d2@3a`k%Mjhuhs)qx(#P8{d=vNU zbL`ydd~(rU8u&!GHpq(OC*GD#pBjLj!0&ycuuqIlK{Vjfu4f9wF=_fF55ljPnkI zAVKW25Lk4AN#;7mE)?&GSs zsNbw>8h&q(tsS}@5TH444eOI$7!9o4GS>hv;&~bMEYYmU{y9TtAQeTRED3TVeZdzD zrY7F*7o8Bn_jr+tC5(fMwAZ7}M2xneG_FcGGeZH3Pt3$!Y6))>9@)Pp9G<_i$JmYX zu_PkNmr20cl9P@Dqf)z6gq*mAl9vK~R~lW?u@r$%@B7Vp_f8@@PJtd>kEwS8mZD$b z;Z>@hgo4s1EVB@E*d8c2_<7ON>ECt_P#|$-j*!5M)V%GSA8yCG9c@M!#VI|I>`&^8 z+-w~r7{E2tLZ6`+>Bg#|V7_lcOr$U)by$(Ic!IM;NvT6jD_)QlqW)rjiRQE(b{+U| zQQMBykX7m!eW^bwVn2!+ePd!4Y3hSizpNyK)6-wuU#;6++Rqu+nTN7l#4cqig@Tt( z^u!ce6ftCq+o^u3)>2-Cen?ip_6RVvDe=ewRUDstr& zfx}m0x^~u&pA|$WvHsBOt&;>T582n4#4nbjatzwn&-2LQQ=&K|WKKr&M~Yk4&|1Pg z{YqlaJMwN}`ckQO&^|?>RJ2;Br!sNLVZgRU%F<#Tr%d$cuTO1KmagH0rNpelJ4wr$>t(7MJ$#64!rQZeusnyK55?P z3j-s&rVYg20=Cp}GqtragTP{vFntpUY@MQsriCuR!v8E-n_904EbOI}RE0B$wgAK^b$gW`F<>BaJ?#jT3@%Nr=934O~0TloPNPv?A z$j`yY$q8Emxw)7*IO#b!=wWtBj^_W9iKdH*laqxxOpV3F-qiwwT~$*`mrdHu-rm&2 z!2zby!k}qw;|hRHfBVA%(6(@Kg&i8e#RlXC^6+!=^00EU@&40Km_6*=6m8rs05E5G z7y))pCJ<{^3lo6*|J{z4jgt+?_?O^|g@gHDqnYvW{p+}9|3Am}{+|pRFXulS908ns z+<)_H{|hmhj07bGL?CYH$sLhvE!48+Yx<%XV|d%0+SxX(Fv~{<%-;^cvyBeEMdhjw zky}(kqYjp9^i+d%gaP(lgzxZjC}dS%ng95vP}*f``GF2$u4o{EEH1g-CKcWYFN%8~ z4>zWgP+NkzceH7`;deny(FMz`l>Nh>$LR$ZBx4J$xIJ6WyVnw&!69{%zh$Y#_snH? zvAG)G4+HS%{{mqo;@J=uZy4$L8>?`>2)-!3k=?5Il~3uw-^9c!RbEjY-=JqN?kx1? z&l>le!9_SWW`1l%t}kbIW=G=2d{ufk%=wgGl5u*thCf~U)(p|L7^c6hT`5!;L$lfj z+?MaMFYS+7N~my-{VwS>m&>mI)BVf&-#AWPbqh-jb~y)g3r~PP2Y}~2mmvnbhK;wy zUr`Kp9e_R%zy$!pGSwX&A+XH9Xj1jRwy<=BrT%&#{z`!W`a)7X+(00ogcOeypEMVb zw3IZD6elN$Uz(SbgHr-_2Z;dwXA{_Q6fGRAU=IwCi=XFz%dOI-DA;Rh5r*Cf+*H|M z)`m<=C@|LU&Ys&ds;=!CGE5+Rv{5j&-GbW+_*_a-+l55%ZlqV%6hS#DP%SqiHe7(C zD-b)pH{zf(_y>tuxAM>Iji;1E_M?@W`NPBKj0<2#y{?RG(E^7CdyZropbU#6E4+s8 zX2V`tP(1@S#4Cewlw>frE?hC)PovBI*?=bS+tc}v+2=Z6aUNaJf22-UuP?ubN>`$B z2lizn*FxPVS#qArJy?)OI$bIeZl!Nc*T2~oV^_*c%UY=rAd(t+LMCRc!ala=hD)%EaLY2jT zZD7g;9PEIgY+?!o88Oj=#M9#!b}uXnRQgUiC&K6e4SXYp8@IwPQVuwjF5LH+p$Z&) z&YIo%BWups(Sgn!@b9_^qk?3Wz!_5Ua*cDpQ+4ed?T|N2t8YTxLS?l%&+cCtM+a-e zSB_i4oAa1*+gHn@pZJ_aEs9EVNxqj}O=JDGc6ScZ2;0C8^Bui`&9{(kcbj+*pe@k@_VjIoDUlJ*~< zJW4%1iF)4AfD6O=FO0rnlB4--W0^~B;kFhf8Tt?K_BF$ScJ^El95%-LH zb4a?dR8fWl<0|Mhx!@(4$MHaN6O=s{ZnVVqY5ny7++| z?`s&{1C%>;WRCF<%dXa9zyx5DK8Mc62z@DK$5XoHPWMXGW5RTwF~?&BVKCqFj{gB* zf!SrVn0*oTItX!t_JFAA=PwVOz@BUz3V4|=pcBP6xWa&uN%nKwwwp{8t4=r)B*_gH z5)>M6lB1l8zhbY0*U9@dxI{Q#A9^h@6qkgK`vm_`@(;LPHv}CZN~ymAUxs^kP{c0& zk5nJ14B>sAY~rBgl!P6~Vf}*lQtfX9&v?Tc#@Qn!ml$03a~e!yP5%{X_;D$DO&+Vu zm2y>bQ!@V#pQWonf*?QdnNMdbEpY7z?>@^2`}K58KECv&?!0jGG8*eevJW*N*$~BN(Bqlsqb(b&> zt+z@iLkTIPk%nHLLyaQ|wS~@@61y)SjCS)*k7z2+)rFjUOQ3|t{%a-Q3D+MMOBni8 zI<9R)2~S#oYo!yOu$>nhMBYqz?YhR05H!)bC?I`#e?3G))8A_@`}jlg+wBheJv+`z z+vetH{qr7WhlmyYCc23+8k#cDhlpBy8@g%vykuGC&)U@gp5a_cfGYTJfh1K7{5WJw zk$xP)lyPYP$)@JzSx6OqPz-xV9G+0zhW@45;KQS*&Mgi$RXF2+f48_oOk5zIunG np.ndarray: + return scl.inv(x.T @ x) @ (x.T @ y) +beta = ols_inv(X_train_own, y_train) +\end{minted} + + +% !split +\subsection*{Singular Value decomposition} + +Doing the inversion directly turns out to be a bad idea since the matrix +$\bm{X}^T\bm{X}$ is singular. An alternative approach is to use the \textbf{singular +value decomposition}. Using the definition of the Moore-Penrose +pseudoinverse we can write the equation for $\bm{\beta}$ as + +\[ + \bm{\beta} = \bm{X}^{+}\bm{y}, +\] + +where the pseudoinverse of $\bm{X}$ is given by + +\[ + \bm{X}^{+} = \frac{\bm{X}^T}{\bm{X}^T\bm{X}}. +\] + +Using singular value decomposition we can decompose the matrix $\bm{X} = \bm{U}\bm{\Sigma} \bm{V}^T$, +where $\bm{U}$ and $\bm{V}$ are orthogonal(unitary) matrices and $\bm{\Sigma}$ contains the singular values (more details below). +where $X^{+} = V\Sigma^{+} U^T$. This reduces the equation for +$\omega$ to +\begin{align} + \bm{\beta} = \bm{V}\bm{\Sigma}^{+} \bm{U}^T \bm{y}. +\end{align} + +Note that solving this equation by actually doing the pseudoinverse +(which is what we will do) is not a good idea as this operation scales +as $\mathcal{O}(n^3)$, where $n$ is the number of elements in a +general matrix. Instead, doing $QR$-factorization and solving the +linear system as an equation would reduce this down to +$\mathcal{O}(n^2)$ operations. + + +\begin{minted}[fontsize=\fontsize{9pt}{9pt},linenos=false,mathescape,baselinestretch=1.0,fontfamily=tt,xleftmargin=7mm]{python} +def ols_svd(x: np.ndarray, y: np.ndarray) -> np.ndarray: + u, s, v = scl.svd(x) + return v.T @ scl.pinv(scl.diagsvd(s, u.shape[0], v.shape[0])) @ u.T @ y +\end{minted} + +\begin{minted}[fontsize=\fontsize{9pt}{9pt},linenos=false,mathescape,baselinestretch=1.0,fontfamily=tt,xleftmargin=7mm]{python} +beta = ols_svd(X_train_own,y_train) +\end{minted} + +When extracting the $J$-matrix we need to make sure that we remove the intercept, as is done here + +\begin{minted}[fontsize=\fontsize{9pt}{9pt},linenos=false,mathescape,baselinestretch=1.0,fontfamily=tt,xleftmargin=7mm]{python} +J = beta[1:].reshape(L, L) +\end{minted} + +A way of looking at the coefficients in $J$ is to plot the matrices as images. + + +\begin{minted}[fontsize=\fontsize{9pt}{9pt},linenos=false,mathescape,baselinestretch=1.0,fontfamily=tt,xleftmargin=7mm]{python} +fig = plt.figure(figsize=(20, 14)) +im = plt.imshow(J, **cmap_args) +plt.title("OLS", fontsize=18) +plt.xticks(fontsize=18) +plt.yticks(fontsize=18) +cb = fig.colorbar(im) +cb.ax.set_yticklabels(cb.ax.get_yticklabels(), fontsize=18) +plt.show() +\end{minted} +It is interesting to note that OLS +considers both $J_{j, j + 1} = -0.5$ and $J_{j, j - 1} = -0.5$ as +valid matrix elements for $J$. +In our discussion below on hyperparameters and Ridge and Lasso regression we will see that +this problem can be removed, partly and only with Lasso regression. + +In this case our matrix inversion was actually possible. The obvious question now is what is the mathematics behind the SVD? + + +% !split +\subsection*{Linear Regression Problems} + +One of the typical problems we encounter with linear regression, in particular +when the matrix $\bm{X}$ (our so-called design matrix) is high-dimensional, +are problems with near singular or singular matrices. The column vectors of $\bm{X}$ +may be linearly dependent, normally referred to as super-collinearity. +This means that the matrix may be rank deficient and it is basically impossible to +to model the data using linear regression. As an example, consider the matrix +\begin{align*} +\mathbf{X} & = \left[ +\begin{array}{rrr} +1 & -1 & 2 +\\ +1 & 0 & 1 +\\ +1 & 2 & -1 +\\ +1 & 1 & 0 +\end{array} \right] +\end{align*} + +The columns of $\bm{X}$ are linearly dependent. We see this easily since the +the first column is the row-wise sum of the other two columns. The rank (more correct, +the column rank) of a matrix is the dimension of the space spanned by the +column vectors. Hence, the rank of $\mathbf{X}$ is equal to the number +of linearly independent columns. In this particular case the matrix has rank 2. + +Super-collinearity of an $(n \times p)$-dimensional design matrix $\mathbf{X}$ implies +that the inverse of the matrix $\bm{X}^T\bm{x}$ (the matrix we need to invert to solve the linear regression equations) is non-invertible. If we have a square matrix that does not have an inverse, we say this matrix singular. The example here demonstrates this +\begin{align*} +\bm{X} & = \left[ +\begin{array}{rr} +1 & -1 +\\ +1 & -1 +\end{array} \right]. +\end{align*} +We see easily that $\mbox{det}(\bm{X}) = x_{11} x_{22} - x_{12} x_{21} = 1 \times (-1) - 1 \times (-1) = 0$. Hence, $\mathbf{X}$ is singular and its inverse is undefined. +This is equivalent to saying that the matrix $\bm{X}$ has at least an eigenvalue which is zero. + + +% !split +\subsection*{Fixing the singularity} + +If our design matrix $\bm{X}$ which enters the linear regression problem +\begin{align} +\bm{\beta} & = (\bm{X}^{T} \bm{X})^{-1} \bm{X}^{T} \bm{y}, +\end{align} +has linearly dependent column vectors, we will not be able to compute the inverse +of $\bm{X}^T\bm{X}$ and we cannot find the parameters (estimators) $\beta_i$. +The estimators are only well-defined if $(\bm{X}^{T}\bm{X})^{-1}$ exits. +This is more likely to happen when the matrix $\bm{X}$ is high-dimensional. In this case it is likely to encounter a situation where +the regression parameters $\beta_i$ cannot be estimated. + +A cheap \emph{ad hoc} approach is simply to add a small diagonal component to the matrix to invert, that is we change +\[ +\bm{X}^{T} \bm{X} \rightarrow \bm{X}^{T} \bm{X}+\lambda \bm{I}, +\] +where $\bm{I}$ is the identity matrix. When we discuss \textbf{Ridge} regression this is actually what we end up evaluating. The parameter $\lambda$ is called a hyperparameter. More about this later. + + + +% !split +\subsection*{Basic math of the SVD} + + +From standard linear algebra we know that a square matrix $\bm{X}$ can be diagonalized if and only it is +a so-called \href{{https://en.wikipedia.org/wiki/Normal_matrix}}{normal matrix}, that is if $\bm{X}\in {\mathbb{R}}^{n\times n}$ +we have $\bm{X}\bm{X}^T=\bm{X}^T\bm{X}$ or if $\bm{X}\in {\mathbb{C}}^{n\times n}$ we have $\bm{X}\bm{X}^{\dagger}=\bm{X}^{\dagger}\bm{X}$. +The matrix has then a set of eigenpairs + +\[ +(\lambda_1,\bm{u}_1),\dots, (\lambda_n,\bm{u}_n), +and the eigenvalues are given by the diagonal matrix +\[ +\bm{\Sigma}=\mathrm{Diag}(\lambda_1, \dots,\lambda_n). +\] +The matrix $\bm{X}$ can be written in terms of an orthogonal/unitary transformation $\bm{U}$ +\[ +\bm{X} = \bm{U}\bm{\Sigma}\bm{V}^T, +\] +with $\bm{U}\bm{U}^T=\bm{I}$ or $\bm{U}\bm{U}^{\dagger}=\bm{I}$. + +Not all square matrices are diagonalizable. A matrix like the one discussed above +\[ +\bm{X} = \begin{bmatrix} +1& -1 \\ +1& -1\\ +\end{bmatrix} +\] +is not diagonalizable, it is a so-called \href{{https://en.wikipedia.org/wiki/Defective_matrix}}{defective matrix}. It is easy to see that the condition +$\bm{X}\bm{X}^T=\bm{X}^T\bm{X}$ is not fulfilled. + + +% !split +\subsection*{The SVD, a Fantastic Algorithm} + + +However, and this is the strength of the SVD algorithm, any general +matrix $\bm{X}$ can be decomposed in terms of a diagonal matrix and +two orthogonal/unitary matrices. The \href{{https://en.wikipedia.org/wiki/Singular_value_decomposition}}{Singular Value Decompostion +(SVD) theorem} +states that a general $m\times n$ matrix $\bm{X}$ can be written in +terms of a diagonal matrix $\bm{\Sigma}$ of dimensionality $n\times n$ +and two orthognal matrices $\bm{U}$ and $\bm{V}$, where the first has +dimensionality $m \times m$ and the last dimensionality $n\times n$. +We have then + +\[ +\bm{X} = \bm{U}\bm{\Sigma}\bm{V}^T +\] + +As an example, the above defective matrix can be decomposed as + +\[ +\bm{X} = \frac{1}{\sqrt{2}}\begin{bmatrix} 1& 1 \\ 1& -1\\ \end{bmatrix} \begin{bmatrix} 2& 0 \\ 0& 0\\ \end{bmatrix} \frac{1}{\sqrt{2}}\begin{bmatrix} 1& -1 \\ 1& 1\\ \end{bmatrix}=\bm{U}\bm{\Sigma}\bm{V}^T, +\] + +with eigenvalues $\sigma_1=2$ and $\sigma_2=0$. +The SVD exits always! + + +% !split +\subsection*{Another Example} + +Consider the following matrix which can be SVD decomposed as + +\[ +\bm{X} = \frac{1}{15}\begin{bmatrix} 14 & 2\\ 4 & 22\\ 16 & 13\end{matrix}=\frac{1}{3}\begin{bmatrix} 1& 2 & 2 \\ 2& -1 & 1\\ 2 & 1& -2\end{bmatrix} \begin{bmatrix} 2& 0 \\ 0& 1\\ 0 & 0\end{bmatrix}\frac{1}{5}\begin{bmatrix} 3& 4 \\ 4& -3\end{bmatrix}=\bm{U}\bm{\Sigma}\bm{V}^T. +\] + +This is a $3\times 2$ matrix which is decomposed in terms of a +$3\times 3$ matrix $\bm{U}$, and a $2\times 2$ matrix $\bm{V}$. It is easy to see +that $\bm{U}$ and $\bm{V}$ are orthogonal (how?). + +And the SVD +decomposition (singular values) gives eigenvalues +$\sigma_i\geq\sigma_{i+1}$ for all $i$ and for dimensions larger than $i=2$, the +eigenvalues (singular values) are zero. + +In the general case, where our design matrix $\bm{X}$ has dimension +$n\times p$, the matrix is thus decomposed into an $n\times n$ +orthogonal matrix $\bm{U}$, a $p\times p$ orthogonal matrix $\bm{V}$ +and a diagonal matrix $\bm{\Sigma}$ with $r=\mathrm{min}(n,p)$ +singular values $\sigma_i\lg 0$ on the main diagonal and zeros filling +the rest of the matrix. There are at most $p$ singular values +assuming that $n > p$. In our regression examples for the nuclear +masses and the equation of state this is indeed the case, while for +the Ising model we have $p > n$. These are often cases that lead to +near singular or singular matrices. + +The columns of $\bm{U}$ are called the left singular vectors while the columns of $\bm{V}$ are the right singular vectors. + +% !split +\subsection*{Economy-size SVD} + +If we assume that $n > p$, then our matrix $\bm{U}$ has dimension $n +\times n$. The last $n-p$ columns of $\bm{U}$ become however +irrelevant in our calculations since they are multiplied with the +zeros in $\bm{\Sigma}$. + +The economy-size decomposition removes extra rows or columns of zeros +from the diagonal matrix of singular values, $\bm{\Sigma}$, along with the columns +in either $\bm{U}$ or $\bm{V}$ that multiply those zeros in the expression. +Removing these zeros and columns can improve execution time +and reduce storage requirements without compromising the accuracy of +the decomposition. + +If $n > p$, we keep only the first $p$ columns of $\bm{U}$ and $\bm{\Sigma}$ has dimension $p\times p$. +If $p > n$, then only the first $n$ columns of $\bm{V}$ are computed and $\bm{\Sigma}$ has dimension $n\times n$. +The $n=p$ case is obvious, we retain the full SVD. +In general the economy-size SVD leads to less FLOPS and still conserving the desired accuracy. + +% !split +\subsection*{Mathematical Properties} + +There are several interesting mathematical properties which will be +relevant when we are going to discuss the differences between say +ordinary least squares (OLS) and \textbf{Ridge} regression. + +We have from OLS that the parameters of the linear approximation are given by +\[ +\bm{\tilde{y}} = \bm{X}\bm{\beta} = \bm{X}\left(\bm{X}^T\bm{X}\right)^{-1}\bm{X}^T\bm{y}. +\] + +The matrix to invert can be rewritten in terms of our SVD decomposition as + +\[ +\bm{X}^T\bm{X} = \bm{V}\bm{\Sigma}^T\bm{U}^T\bm{U}\bm{\Sigma}\bm{V}^T. +\] +Using the orthogonality properties of $\bm{U}$ we have + +\[ +\bm{X}^T\bm{X} = \bm{V}\bm{\Sigma}^T\bm{\Sigma}\bm{V}^T = \bm{V}\bm{D}\bm{V}^T, +\] +with $\bm{D}$ being a diagonal matrix with values along the diagonal given by the singular values squared. + +This means that +\[ +(\bm{X}^T\bm{X})\bm{V} = \bm{V}\bm{D}, +\] +that is the eigenvectors of $(\bm{X}^T\bm{X})$ are given by the columns of the right singular matrix of $\bm{X}$ and the eigenvalues are the squared singular values. It is easy to show (show this) that +\[ +(\bm{X}\bm{X}^T)\bm{U} = \bm{U}\bm{D}, +\] +that is, the eigenvectors of $(\bm{X}\bm{X})^T$ are the columns of the left singular matrix and the eigenvalues are the same. + +Going back to our OLS equation we have +\[ +\bm{X}\bm{\beta} = \bm{X}\left(\bm{V}\bm{D}\bm{V}^T \right)^{-1}\bm{X}^T\bm{y}=\bm{U\Sigma V^T}\left(\bm{V}\bm{D}\bm{V}^T \right)^{-1}(\bm{U\Sigma V^T})^T\bm{y}=\bm{U}\bm{U}^T\bm{y}. +\] +We will come back to this expression when we discuss Ridge regression. + + +% !split +\subsection*{Ridge and LASSO Regression} + +Let us remind ourselves about the expression for the standard Mean Squared Error (MSE) which we used to define our cost function and the equations for the ordinary least squares (OLS) method, that is +our optimization problem is +\[ +{\displaystyle \min_{\bm{\beta}\in {\mathbb{R}}^{p}}}\frac{1}{n}\left\{\left(\bm{y}-\bm{X}\bm{\beta}\right)^T\left(\bm{y}-\bm{X}\bm{\beta}\right)\right\}. +\] +or we can state it as +\[ +{\displaystyle \min_{\bm{\beta}\in +{\mathbb{R}}^{p}}}\frac{1}{n}\sum_{i=0}^{n-1}\left(y_i-\tilde{y}_i\right)^2=\frac{1}{n}\vert\vert \bm{y}-\bm{X}\bm{\beta}\vert\vert_2^2, +\] +where we have used the definition of a norm-2 vector, that is +\[ +\vert\vert \bm{x}\vert\vert_2 = \sqrt{\sum_i x_i^2}. +\] + +By minimizing the above equation with respect to the parameters +$\bm{\beta}$ we could then obtain an analytical expression for the +parameters $\bm{\beta}$. We can add a regularization parameter $\lambda$ by +defining a new cost function to be optimized, that is + +\[ +{\displaystyle \min_{\bm{\beta}\in +{\mathbb{R}}^{p}}}\frac{1}{n}\vert\vert \bm{y}-\bm{X}\bm{\beta}\vert\vert_2^2+\lambda\vert\vert \bm{\beta}\vert\vert_2^2 +\] + +which leads to the Ridge regression minimization problem where we +require that $\vert\vert \bm{\beta}\vert\vert_2^2\le t$, where $t$ is +a finite number larger than zero. By defining + +\[ +C(\bm{X},\bm{\beta})=\frac{1}{n}\vert\vert \bm{y}-\bm{X}\bm{\beta}\vert\vert_2^2+\lambda\vert\vert \bm{\beta}\vert\vert_1, +\] + +we have a new optimization equation +\[ +{\displaystyle \min_{\bm{\beta}\in +{\mathbb{R}}^{p}}}\frac{1}{n}\vert\vert \bm{y}-\bm{X}\bm{\beta}\vert\vert_2^2+\lambda\vert\vert \bm{\beta}\vert\vert_1 +\] +which leads to Lasso regression. Lasso stands for least absolute shrinkage and selection operator. + +Here we have defined the norm-1 as +\[ +\vert\vert \bm{x}\vert\vert_1 = \sum_i \vert x_i\vert. +\] + + +% !split +\subsection*{More on Ridge Regression} + +Using the matrix-vector expression for Ridge regression, + +\[ +C(\bm{X},\bm{\beta})=\frac{1}{n}\left\{(\bm{y}-\bm{X}\bm{\beta})^T(\bm{y}-\bm{X}\bm{\beta})\right\}+\lambda\bm{\beta}^T\bm{\beta}, +\] + +by taking the derivatives with respect to $\bm{\beta}$ we obtain then +a slightly modified matrix inversion problem which for finite values +of $\lambda$ does not suffer from singularity problems. We obtain + +\[ +\bm{\beta}^{\mathrm{Ridge}} = \left(\bm{X}^T\bm{X}+\lambda\bm{I}\right)^{-1}\bm{X}^T\bm{y}, +\] + +with $\bm{I}$ being a $p\times p$ identity matrix with the constraint that + +\[ +\sum_{i=0}^{p-1} \beta_i^2 \leq t, +\] + +with $t$ a finite positive number. + +We see that Ridge regression is nothing but the standard +OLS with a modified diagonal term added to $\bm{X}^T\bm{X}$. The +consequences, in particular for our discussion of the bias-variance +are rather interesting. + +Furthermore, if we use the result above in terms of the SVD decomposition (our analysis was done for the OLS method), we had +\[ +(\bm{X}\bm{X}^T)\bm{U} = \bm{U}\bm{D}. +\] + +We can analyse the OLS solutions in terms of the eigenvectors (the columns) of the right singular value matrix $\bm{U}$ as +\[ +\bm{X}\bm{\beta} = \bm{X}\left(\bm{V}\bm{D}\bm{V}^T \right)^{-1}\bm{X}^T\bm{y}=\bm{U\Sigma V^T}\left(\bm{V}\bm{D}\bm{V}^T \right)^{-1}(\bm{U\Sigma V^T})^T\bm{y}=\bm{U}\bm{U}^T\bm{y} +\] + + +For Ridge regression this becomes + +\[ +\bm{X}\bm{\beta}^{\mathrm{Ridge}} = \bm{U\Sigma V^T}\left(\bm{V}\bm{D}\bm{V}^T+\lambda\bm{I} \right)^{-1}(\bm{U\Sigma V^T})^T\bm{y}=\sum_{j=0}^{p-1}\bm{u}_j\bm{u}_j^T\frac{\sigma_j^2}{\sigma_j^2+\lambda}\bm{y}, +\] + +with the vectors $\bm{u}_j$ being the columns of $\bm{U}$. + +% !split +\subsection*{Interpreting the Ridge results} + +Since $\lambda \geq 0$, it means that compared to OLS, we have + +\[ +\frac{\sigma_j^2}{\sigma_j^2+\lambda} \leq 1. +\] + +Ridge regression finds the coordinates of $\bm{y}$ with respect to the +orthonormal basis $\bm{U}$, it then shrinks the coordinates by +$\frac{\sigma_j^2}{\sigma_j^2+\lambda}$. Recall that the SVD has +eigenvalues ordered in a descending way, that is $\sigma_i \geq +\sigma_{i+1}$. + +For small eigenvalues $\sigma_i$ it means that their contributions become less important, a fact which can be used to reduce the number of degrees of freedom. +Actually, calculating the variance of $\bm{X}\bm{v}_j$ shows that this quantity is equal to $\sigma_j^2/n$. +With a parameter $\lambda$ we can thus shrink the role of specific parameters. + + +% !split +\subsection*{More interpretations} + +For the sake of simplicity, let us assume that the design matrix is orthonormal, that is + +\[ +\bm{X}^T\bm{X}=(\bm{X}^T\bm{X})^{-1} =\bm{I}. +\] + +In this case the standard OLS results in +\[ +\bm{\beta}^{\mathrm{OLS}} = \bm{X}^T\bm{y}=\sum_{i=0}^{p-1}\bm{u}_j\bm{u}_j^T\bm{y}, +\] + +and + +\[ +\bm{\beta}^{\mathrm{Ridge}} = \left(\bm{I}+\lambda\bm{I}\right)^{-1}\bm{X}^T\bm{y}=\left(1+\lambda\right)^{-1}\bm{\beta}^{\mathrm{OLS}}, +\] + +that is the Ridge estimator scales the OLS estimator by the inverse of a factor $1+\lambda$, and +the Ridge estimator converges to zero when the hyperparameter goes to +infinity. + +We will come back to more interpreations after we have gone through some of the statistical analysis part. + +For more discussions of Ridge and Lasso regression, \href{{https://arxiv.org/abs/1509.09169}}{Wessel van Wieringen's} article is highly recommended. +Similarly, \href{{https://arxiv.org/abs/1803.08823}}{Mehta et al's article} is also recommended. + +% !split +\subsection*{Where are we going?} + +Before we proceed, we need to rethink what we have been doing. In our +eager to fit the data, we have omitted several important elements in +our regression analysis. In what follows we will +\begin{enumerate} +\item look at statistical properties, including a discussion of mean values, variance and the so-called bias-variance tradeoff + +\item introduce resampling techniques like cross-validation, bootstrapping and jackknife and more +\end{enumerate} + +\noindent +This will allow us to link the standard linear algebra methods we have discussed above to a statistical interpretation of the methods. + + + + + +% !split +\subsection*{Resampling methods} + +% --- begin paragraph admon --- +\paragraph{} +Resampling methods are an indispensable tool in modern +statistics. They involve repeatedly drawing samples from a training +set and refitting a model of interest on each sample in order to +obtain additional information about the fitted model. For example, in +order to estimate the variability of a linear regression fit, we can +repeatedly draw different samples from the training data, fit a linear +regression to each new sample, and then examine the extent to which +the resulting fits differ. Such an approach may allow us to obtain +information that would not be available from fitting the model only +once using the original training sample. +% --- end paragraph admon --- + + + +% !split +\subsection*{Resampling approaches can be computationally expensive} + +% --- begin paragraph admon --- +\paragraph{} + +Resampling approaches can be computationally expensive, because they +involve fitting the same statistical method multiple times using +different subsets of the training data. However, due to recent +advances in computing power, the computational requirements of +resampling methods generally are not prohibitive. In this chapter, we +discuss two of the most commonly used resampling methods, +cross-validation and the bootstrap. Both methods are important tools +in the practical application of many statistical learning +procedures. For example, cross-validation can be used to estimate the +test error associated with a given statistical learning method in +order to evaluate its performance, or to select the appropriate level +of flexibility. The process of evaluating a model’s performance is +known as model assessment, whereas the process of selecting the proper +level of flexibility for a model is known as model selection. The +bootstrap is widely used. +% --- end paragraph admon --- + + + +% !split +\subsection*{Why resampling methods ?} + +% --- begin paragraph admon --- +\paragraph{Statistical analysis.} + +\begin{itemize} +\item Our simulations can be treated as \emph{computer experiments}. This is particularly the case for Monte Carlo methods + +\item The results can be analysed with the same statistical tools as we would use analysing experimental data. + +\item As in all experiments, we are looking for expectation values and an estimate of how accurate they are, i.e., possible sources for errors. +\end{itemize} + +\noindent +% --- end paragraph admon --- + + + +% !split +\subsection*{Statistical analysis} + +% --- begin paragraph admon --- +\paragraph{} + +\begin{itemize} +\item As in other experiments, many numerical experiments have two classes of errors: +\begin{itemize} + + \item Statistical errors + + \item Systematical errors + +\end{itemize} + +\noindent +\item Statistical errors can be estimated using standard tools from statistics + +\item Systematical errors are method specific and must be treated differently from case to case. +\end{itemize} + +\noindent +% --- end paragraph admon --- + + + +% !split +\subsection*{Statistics} + +% --- begin paragraph admon --- +\paragraph{} +The \emph{probability distribution function (PDF)} is a function +$p(x)$ on the domain which, in the discrete case, gives us the +probability or relative frequency with which these values of $X$ occur: +\[ +p(x) = \mathrm{prob}(X=x) +\] +In the continuous case, the PDF does not directly depict the +actual probability. Instead we define the probability for the +stochastic variable to assume any value on an infinitesimal interval +around $x$ to be $p(x)dx$. The continuous function $p(x)$ then gives us +the \emph{density} of the probability rather than the probability +itself. The probability for a stochastic variable to assume any value +on a non-infinitesimal interval $[a,\,b]$ is then just the integral: +\[ +\mathrm{prob}(a\leq X\leq b) = \int_a^b p(x)dx +\] +Qualitatively speaking, a stochastic variable represents the values of +numbers chosen as if by chance from some specified PDF so that the +selection of a large set of these numbers reproduces this PDF. +% --- end paragraph admon --- + + + + +% !split +\subsection*{Statistics, moments} + +% --- begin paragraph admon --- +\paragraph{} +A particularly useful class of special expectation values are the +\emph{moments}. The $n$-th moment of the PDF $p$ is defined as +follows: +\[ +\langle x^n\rangle \equiv \int\! x^n p(x)\,dx +\] +The zero-th moment $\langle 1\rangle$ is just the normalization condition of +$p$. The first moment, $\langle x\rangle$, is called the \emph{mean} of $p$ +and often denoted by the letter $\mu$: +\[ +\langle x\rangle = \mu \equiv \int\! x p(x)\,dx +\] +% --- end paragraph admon --- + + + +% !split +\subsection*{Statistics, central moments} + +% --- begin paragraph admon --- +\paragraph{} +A special version of the moments is the set of \emph{central moments}, +the n-th central moment defined as: +\[ +\langle (x-\langle x \rangle )^n\rangle \equiv \int\! (x-\langle x\rangle)^n p(x)\,dx +\] +The zero-th and first central moments are both trivial, equal $1$ and +$0$, respectively. But the second central moment, known as the +\emph{variance} of $p$, is of particular interest. For the stochastic +variable $X$, the variance is denoted as $\sigma^2_X$ or $\mathrm{var}(X)$: +\begin{align} +\sigma^2_X\ \ =\ \ \mathrm{var}(X) & = \langle (x-\langle x\rangle)^2\rangle = +\int\! (x-\langle x\rangle)^2 p(x)\,dx\\ +& = \int\! \left(x^2 - 2 x \langle x\rangle^{2} + + \langle x\rangle^2\right)p(x)\,dx\\ +& = \langle x^2\rangle - 2 \langle x\rangle\langle x\rangle + \langle x\rangle^2\\ +& = \langle x^2\rangle - \langle x\rangle^2 +\end{align} +The square root of the variance, $\sigma =\sqrt{\langle (x-\langle x\rangle)^2\rangle}$ is called the \emph{standard deviation} of $p$. It is clearly just the RMS (root-mean-square) +value of the deviation of the PDF from its mean value, interpreted +qualitatively as the \emph{spread} of $p$ around its mean. +% --- end paragraph admon --- + + + +% !split +\subsection*{Statistics, covariance} + +% --- begin paragraph admon --- +\paragraph{} +Another important quantity is the so called covariance, a variant of +the above defined variance. Consider again the set $\{X_i\}$ of $n$ +stochastic variables (not necessarily uncorrelated) with the +multivariate PDF $P(x_1,\dots,x_n)$. The \emph{covariance} of two +of the stochastic variables, $X_i$ and $X_j$, is defined as follows: +\begin{align} +\mathrm{cov}(X_i,\,X_j) &\equiv \langle (x_i-\langle x_i\rangle)(x_j-\langle x_j\rangle)\rangle +\nonumber\\ +&= +\int\!\cdots\!\int\!(x_i-\langle x_i \rangle)(x_j-\langle x_j \rangle)\, +P(x_1,\dots,x_n)\,dx_1\dots dx_n +\label{eq:def_covariance} +\end{align} +with +\[ +\langle x_i\rangle = +\int\!\cdots\!\int\!x_i\,P(x_1,\dots,x_n)\,dx_1\dots dx_n +\] +% --- end paragraph admon --- + + + +% !split +\subsection*{Statistics, more covariance} + +% --- begin paragraph admon --- +\paragraph{} +If we consider the above covariance as a matrix $C_{ij}=\mathrm{cov}(X_i,\,X_j)$, then the diagonal elements are just the familiar +variances, $C_{ii} = \mathrm{cov}(X_i,\,X_i) = \mathrm{var}(X_i)$. It turns out that +all the off-diagonal elements are zero if the stochastic variables are +uncorrelated. This is easy to show, keeping in mind the linearity of +the expectation value. Consider the stochastic variables $X_i$ and +$X_j$, ($i\neq j$): +\begin{align} +\mathrm{cov}(X_i,\,X_j) &= \langle(x_i-\langle x_i\rangle)(x_j-\langle x_j\rangle)\rangle\\ +&=\langle x_i x_j - x_i\langle x_j\rangle - \langle x_i\rangle x_j + \langle x_i\rangle\langle x_j\rangle\rangle \\ +&=\langle x_i x_j\rangle - \langle x_i\langle x_j\rangle\rangle - \langle \langle x_i\rangle x_j\rangle + +\langle \langle x_i\rangle\langle x_j\rangle\rangle\\ +&=\langle x_i x_j\rangle - \langle x_i\rangle\langle x_j\rangle - \langle x_i\rangle\langle x_j\rangle + +\langle x_i\rangle\langle x_j\rangle\\ +&=\langle x_i x_j\rangle - \langle x_i\rangle\langle x_j\rangle +\end{align} +% --- end paragraph admon --- + + + + + +% !split +\subsection*{Statistics, independent variables} + +% --- begin paragraph admon --- +\paragraph{} +If $X_i$ and $X_j$ are independent, we get +$\langle x_i x_j\rangle =\langle x_i\rangle\langle x_j\rangle$, resulting in $\mathrm{cov}(X_i, X_j) = 0\ \ (i\neq j)$. + +Also useful for us is the covariance of linear combinations of +stochastic variables. Let $\{X_i\}$ and $\{Y_i\}$ be two sets of +stochastic variables. Let also $\{a_i\}$ and $\{b_i\}$ be two sets of +scalars. Consider the linear combination: +\[ +U = \sum_i a_i X_i \qquad V = \sum_j b_j Y_j +\] +By the linearity of the expectation value +\[ +\mathrm{cov}(U, V) = \sum_{i,j}a_i b_j \mathrm{cov}(X_i, Y_j) +\] +% --- end paragraph admon --- + + + +% !split +\subsection*{Statistics, more variance} + +% --- begin paragraph admon --- +\paragraph{} +Now, since the variance is just $\mathrm{var}(X_i) = \mathrm{cov}(X_i, X_i)$, we get +the variance of the linear combination $U = \sum_i a_i X_i$: +\begin{equation} +\mathrm{var}(U) = \sum_{i,j}a_i a_j \mathrm{cov}(X_i, X_j) +\label{eq:variance_linear_combination} +\end{equation} +And in the special case when the stochastic variables are +uncorrelated, the off-diagonal elements of the covariance are as we +know zero, resulting in: +\[ +\mathrm{var}(U) = \sum_i a_i^2 \mathrm{cov}(X_i, X_i) = \sum_i a_i^2 \mathrm{var}(X_i) +\] +\[ +\mathrm{var}(\sum_i a_i X_i) = \sum_i a_i^2 \mathrm{var}(X_i) +\] +which will become very useful in our study of the error in the mean +value of a set of measurements. +% --- end paragraph admon --- + + + +% !split +\subsection*{Statistics and stochastic processes} + +% --- begin paragraph admon --- +\paragraph{} +A \emph{stochastic process} is a process that produces sequentially a +chain of values: +\[ +\{x_1, x_2,\dots\,x_k,\dots\}. +\] +We will call these +values our \emph{measurements} and the entire set as our measured +\emph{sample}. The action of measuring all the elements of a sample +we will call a stochastic \emph{experiment} since, operationally, +they are often associated with results of empirical observation of +some physical or mathematical phenomena; precisely an experiment. We +assume that these values are distributed according to some +PDF $p_X^{\phantom X}(x)$, where $X$ is just the formal symbol for the +stochastic variable whose PDF is $p_X^{\phantom X}(x)$. Instead of +trying to determine the full distribution $p$ we are often only +interested in finding the few lowest moments, like the mean +$\mu_X^{\phantom X}$ and the variance $\sigma_X^{\phantom X}$. +% --- end paragraph admon --- + + + + +% !split +\subsection*{Statistics and sample variables} + +% --- begin paragraph admon --- +\paragraph{} +In practical situations a sample is always of finite size. Let that +size be $n$. The expectation value of a sample, the \emph{sample mean}, is then defined as follows: +\[ +\bar{x}_n \equiv \frac{1}{n}\sum_{k=1}^n x_k +\] +The \emph{sample variance} is: +\[ +\mathrm{var}(x) \equiv \frac{1}{n}\sum_{k=1}^n (x_k - \bar{x}_n)^2 +\] +its square root being the \emph{standard deviation of the sample}. The +\emph{sample covariance} is: +\[ +\mathrm{cov}(x)\equiv\frac{1}{n}\sum_{kl}(x_k - \bar{x}_n)(x_l - \bar{x}_n) +\] +% --- end paragraph admon --- + + + +% !split +\subsection*{Statistics, sample variance and covariance} + +% --- begin paragraph admon --- +\paragraph{} +Note that the sample variance is the sample covariance without the +cross terms. In a similar manner as the covariance in Eq.~(\ref{eq:def_covariance}) is a measure of the correlation between +two stochastic variables, the above defined sample covariance is a +measure of the sequential correlation between succeeding measurements +of a sample. + +These quantities, being known experimental values, differ +significantly from and must not be confused with the similarly named +quantities for stochastic variables, mean $\mu_X$, variance $\mathrm{var}(X)$ +and covariance $\mathrm{cov}(X,Y)$. +% --- end paragraph admon --- + + + +% !split +\subsection*{Statistics, law of large numbers} + +% --- begin paragraph admon --- +\paragraph{} +The law of large numbers +states that as the size of our sample grows to infinity, the sample +mean approaches the true mean $\mu_X^{\phantom X}$ of the chosen PDF: +\[ +\lim_{n\to\infty}\bar{x}_n = \mu_X^{\phantom X} +\] +The sample mean $\bar{x}_n$ works therefore as an estimate of the true +mean $\mu_X^{\phantom X}$. + +What we need to find out is how good an approximation $\bar{x}_n$ is to +$\mu_X^{\phantom X}$. In any stochastic measurement, an estimated +mean is of no use to us without a measure of its error. A quantity +that tells us how well we can reproduce it in another experiment. We +are therefore interested in the PDF of the sample mean itself. Its +standard deviation will be a measure of the spread of sample means, +and we will simply call it the \emph{error} of the sample mean, or +just sample error, and denote it by $\mathrm{err}_X^{\phantom X}$. In +practice, we will only be able to produce an \emph{estimate} of the +sample error since the exact value would require the knowledge of the +true PDFs behind, which we usually do not have. +% --- end paragraph admon --- + + + + +% !split +\subsection*{Statistics, more on sample error} + +% --- begin paragraph admon --- +\paragraph{} +Let us first take a look at what happens to the sample error as the +size of the sample grows. In a sample, each of the measurements $x_i$ +can be associated with its own stochastic variable $X_i$. The +stochastic variable $\overline X_n$ for the sample mean $\bar{x}_n$ is +then just a linear combination, already familiar to us: +\[ +\overline X_n = \frac{1}{n}\sum_{i=1}^n X_i +\] +All the coefficients are just equal $1/n$. The PDF of $\overline X_n$, +denoted by $p_{\overline X_n}(x)$ is the desired PDF of the sample +means. +% --- end paragraph admon --- + + + +% !split +\subsection*{Statistics} + +% --- begin paragraph admon --- +\paragraph{} +The probability density of obtaining a sample mean $\bar x_n$ +is the product of probabilities of obtaining arbitrary values $x_1, +x_2,\dots,x_n$ with the constraint that the mean of the set $\{x_i\}$ +is $\bar x_n$: +\[ +p_{\overline X_n}(x) = \int p_X^{\phantom X}(x_1)\cdots +\int p_X^{\phantom X}(x_n)\ +\delta\!\left(x - \frac{x_1+x_2+\dots+x_n}{n}\right)dx_n \cdots dx_1 +\] +And in particular we are interested in its variance $\mathrm{var}(\overline X_n)$. +% --- end paragraph admon --- + + + + + +% !split +\subsection*{Statistics, central limit theorem} + +% --- begin paragraph admon --- +\paragraph{} +It is generally not possible to express $p_{\overline X_n}(x)$ in a +closed form given an arbitrary PDF $p_X^{\phantom X}$ and a number +$n$. But for the limit $n\to\infty$ it is possible to make an +approximation. The very important result is called \emph{the central limit theorem}. It tells us that as $n$ goes to infinity, +$p_{\overline X_n}(x)$ approaches a Gaussian distribution whose mean +and variance equal the true mean and variance, $\mu_{X}^{\phantom X}$ +and $\sigma_{X}^{2}$, respectively: +\begin{equation} +\lim_{n\to\infty} p_{\overline X_n}(x) = +\left(\frac{n}{2\pi\mathrm{var}(X)}\right)^{1/2} +e^{-\frac{n(x-\bar x_n)^2}{2\mathrm{var}(X)}} +\label{eq:central_limit_gaussian} +\end{equation} +% --- end paragraph admon --- + + + +% !split +\subsection*{Statistics, more technicalities} + +% --- begin paragraph admon --- +\paragraph{} +The desired variance +$\mathrm{var}(\overline X_n)$, i.e.~the sample error squared +$\mathrm{err}_X^2$, is given by: +\begin{equation} +\mathrm{err}_X^2 = \mathrm{var}(\overline X_n) = \frac{1}{n^2} +\sum_{ij} \mathrm{cov}(X_i, X_j) +\label{eq:error_exact} +\end{equation} +We see now that in order to calculate the exact error of the sample +with the above expression, we would need the true means +$\mu_{X_i}^{\phantom X}$ of the stochastic variables $X_i$. To +calculate these requires that we know the true multivariate PDF of all +the $X_i$. But this PDF is unknown to us, we have only got the measurements of +one sample. The best we can do is to let the sample itself be an +estimate of the PDF of each of the $X_i$, estimating all properties of +$X_i$ through the measurements of the sample. +% --- end paragraph admon --- + + + + +% !split +\subsection*{Statistics} + +% --- begin paragraph admon --- +\paragraph{} +Our estimate of $\mu_{X_i}^{\phantom X}$ is then the sample mean $\bar x$ +itself, in accordance with the the central limit theorem: +\[ +\mu_{X_i}^{\phantom X} = \langle x_i\rangle \approx \frac{1}{n}\sum_{k=1}^n x_k = \bar x +\] +Using $\bar x$ in place of $\mu_{X_i}^{\phantom X}$ we can give an +\emph{estimate} of the covariance in Eq.~(\ref{eq:error_exact}) +\[ +\mathrm{cov}(X_i, X_j) = \langle (x_i-\langle x_i\rangle)(x_j-\langle x_j\rangle)\rangle +\approx\langle (x_i - \bar x)(x_j - \bar{x})\rangle, +\] +resulting in +\[ +\frac{1}{n} \sum_{l}^n \left(\frac{1}{n}\sum_{k}^n (x_k -\bar x_n)(x_l - \bar x_n)\right)=\frac{1}{n}\frac{1}{n} \sum_{kl} (x_k -\bar x_n)(x_l - \bar x_n)=\frac{1}{n}\mathrm{cov}(x) +\] +% --- end paragraph admon --- + + + +% !split +\subsection*{Statistics and sample variance} + +% --- begin paragraph admon --- +\paragraph{} +By the same procedure we can use the sample variance as an +estimate of the variance of any of the stochastic variables $X_i$ +\[ +\mathrm{var}(X_i)=\langle x_i - \langle x_i\rangle\rangle \approx \langle x_i - \bar x_n\rangle\nonumber, +\] +which is approximated as +\begin{equation} +\mathrm{var}(X_i)\approx \frac{1}{n}\sum_{k=1}^n (x_k - \bar x_n)=\mathrm{var}(x) +\label{eq:var_estimate_i_think} +\end{equation} + +Now we can calculate an estimate of the error +$\mathrm{err}_X^{\phantom X}$ of the sample mean $\bar x_n$: +\begin{align} +\mathrm{err}_X^2 +&=\frac{1}{n^2}\sum_{ij} \mathrm{cov}(X_i, X_j) \nonumber \\ +&\approx&\frac{1}{n^2}\sum_{ij}\frac{1}{n}\mathrm{cov}(x) =\frac{1}{n^2}n^2\frac{1}{n}\mathrm{cov}(x)\nonumber\\ +&=\frac{1}{n}\mathrm{cov}(x) +\label{eq:error_estimate} +\end{align} +which is nothing but the sample covariance divided by the number of +measurements in the sample. +% --- end paragraph admon --- + + + +% !split +\subsection*{Statistics, uncorrelated results} + +% --- begin paragraph admon --- +\paragraph{} + +In the special case that the measurements of the sample are +uncorrelated (equivalently the stochastic variables $X_i$ are +uncorrelated) we have that the off-diagonal elements of the covariance +are zero. This gives the following estimate of the sample error: +\[ +\mathrm{err}_X^2=\frac{1}{n^2}\sum_{ij} \mathrm{cov}(X_i, X_j) = +\frac{1}{n^2} \sum_i \mathrm{var}(X_i), +\] +resulting in +\begin{equation} +\mathrm{err}_X^2\approx \frac{1}{n^2} \sum_i \mathrm{var}(x)= \frac{1}{n}\mathrm{var}(x) +\label{eq:error_estimate_uncorrel} +\end{equation} +where in the second step we have used Eq.~(\ref{eq:var_estimate_i_think}). +The error of the sample is then just its standard deviation divided by +the square root of the number of measurements the sample contains. +This is a very useful formula which is easy to compute. It acts as a +first approximation to the error, but in numerical experiments, we +cannot overlook the always present correlations. +% --- end paragraph admon --- + + + +% !split +\subsection*{Statistics, computations} + +% --- begin paragraph admon --- +\paragraph{} +For computational purposes one usually splits up the estimate of +$\mathrm{err}_X^2$, given by Eq.~(\ref{eq:error_estimate}), into two +parts +\[ +\mathrm{err}_X^2 = \frac{1}{n}\mathrm{var}(x) + \frac{1}{n}(\mathrm{cov}(x)-\mathrm{var}(x)), +\] +which equals +\begin{equation} +\frac{1}{n^2}\sum_{k=1}^n (x_k - \bar x_n)^2 +\frac{2}{n^2}\sum_{k 0$. We say then that the ridge estimator is biased. + +We can also compute the variance as + +\[ +\mbox{Var}[\bm{\beta}^{\mathrm{Ridge}}]=\sigma^2[ \mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I} ]^{-1} \mathbf{X}^{T} \mathbf{X} \{ [ \mathbf{X}^{\top} \mathbf{X} + \lambda \mathbf{I} ]^{-1}\}^{T}, +\] +and it is easy to see that if the parameter $\lambda$ goes to infinity then the variance of Ridge parameters $\bm{\beta}$ goes to zero. + +With this, we can compute the difference + +\[ +\mbox{Var}[\bm{\beta}^{\mathrm{OLS}}]-\mbox{Var}(\bm{\beta}^{\mathrm{Ridge}})=\sigma^2 [ \mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I} ]^{-1}[ 2\lambda\mathbf{I} + \lambda^2 (\mathbf{X}^{T} \mathbf{X})^{-1} ] \{ [ \mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I} ]^{-1}\}^{T}. +\] +The difference is non-negative definite since each component of the +matrix product is non-negative definite. +This means the variance we obtain with the standard OLS will always for $\lambda > 0$ be larger than the variance of $\bm{\beta}$ obtained with the Ridge estimator. This has interesting consequences when we discuss the so-called bias-variance trade-off below. + +% !split +\subsection*{Cross-validation} + +Instead of choosing the penalty parameter to balance model fit with +model complexity, cross-validation requires it (i.e.~the penalty +parameter) to yield a model with good prediction +performance. Commonly, this performance is evaluated on novel +data. Novel data need not be easy to come by and one has to make do +with the data at hand. + +The setting of \textbf{original} and novel data is +then mimicked by sample splitting: the data set is divided into two +(groups of samples). One of these two data sets, called the +\emph{training set}, plays the role of \textbf{original} data on which the model is +built. The second of these data sets, called the \emph{test set}, plays the +role of the \textbf{novel} data and is used to evaluate the prediction +performance (often operationalized as the log-likelihood or the +prediction error or its square or the R2 score) of the model built on the training data set. This +procedure (model building and prediction evaluation on training and +test set, respectively) is done for a collection of possible penalty +parameter choices. The penalty parameter that yields the model with +the best prediction performance is to be preferred. The thus obtained +performance evaluation depends on the actual split of the data set. To +remove this dependence the data set is split many times into a +training and test set. For each split the model parameters are +estimated for all choices of $\lambda$ using the training data and +estimated parameters are evaluated on the corresponding test set. The +penalty parameter that on average over the test sets performs best (in +some sense) is then selected. + + +% !split +\subsection*{Computationally expensive} + +The validation set approach is conceptually simple and is easy to implement. But it has two potential drawbacks: + +\begin{itemize} +\item The validation estimate of the test error rate can be highly variable, depending on precisely which observations are included in the training set and which observations are included in the validation set. + +\item In the validation approach, only a subset of the observations, those that are included in the training set rather than in the validation set are used to fit the model. Since statistical methods tend to perform worse when trained on fewer observations, this suggests that the validation set error rate may tend to overestimate the test error rate for the model fit on the entire data set. +\end{itemize} + +\noindent +% !split +\subsection*{Various steps in cross-validation} + +When the repetitive splitting of the data set is done randomly, +samples may accidently end up in a fast majority of the splits in +either training or test set. Such samples may have an unbalanced +influence on either model building or prediction evaluation. To avoid +this $k$-fold cross-validation structures the data splitting. The +samples are divided into $k$ more or less equally sized exhaustive and +mutually exclusive subsets. In turn (at each split) one of these +subsets plays the role of the test set while the union of the +remaining subsets constitutes the training set. Such a splitting +warrants a balanced representation of each sample in both training and +test set over the splits. Still the division into the $k$ subsets +involves a degree of randomness. This may be fully excluded when +choosing $k=n$. This particular case is referred to as leave-one-out +cross-validation (LOOCV). + +% !split +\subsection*{How to set up the cross-validation for Ridge and/or Lasso} + +\begin{itemize} +\item Define a range of interest for the penalty parameter. + +\item Divide the data set into training and test set comprising samples $\{1, \ldots, n\} \setminus i$ and $\{ i \}$, respectively. + +\item Fit the linear regression model by means of ridge estimation for each $\lambda$ in the grid using the training set, and the corresponding estimate of the error variance $\bm{\sigma}_{-i}^2(\lambda)$, as +\end{itemize} + +\noindent +\begin{align*} +\bm{\beta}_{-i}(\lambda) & = ( \bm{X}_{-i, \ast}^{T} +\bm{X}_{-i, \ast} + \lambda \bm{I}_{pp})^{-1} +\bm{X}_{-i, \ast}^{T} \bm{y}_{-i} +\end{align*} + +\begin{itemize} +\item Evaluate the prediction performance of these models on the test set by $\log\{L[y_i, \bm{X}_{i, \ast}; \bm{\beta}_{-i}(\lambda), \bm{\sigma}_{-i}^2(\lambda)]\}$. Or, by the prediction error $|y_i - \bm{X}_{i, \ast} \bm{\beta}_{-i}(\lambda)|$, the relative error, the error squared or the R2 score function. + +\item Repeat the first three steps such that each sample plays the role of the test set once. + +\item Average the prediction performances of the test sets at each grid point of the penalty bias/parameter by computing the \emph{cross-validated log-likelihood}. It is an estimate of the prediction performance of the model corresponding to this value of the penalty parameter on novel data. It is defined as +\end{itemize} + +\noindent +\begin{align*} +\frac{1}{n} \sum_{i = 1}^n \log\{L[y_i, \mathbf{X}_{i, \ast}; \bm{\beta}_{-i}(\lambda), \bm{\sigma}_{-i}^2(\lambda)]\}. +\end{align*} + +\begin{itemize} +\item The value of the penalty parameter that maximizes the cross-validated log-likelihood is the value of choice. Or we can use the MSE or the R2 score functions. +\end{itemize} + +\noindent +% !split +\subsection*{Resampling methods: Jackknife and Bootstrap} + +Two famous +resampling methods are the \textbf{independent bootstrap} and \textbf{the jackknife}. + +The jackknife is a special case of the independent bootstrap. Still, the jackknife was made +popular prior to the independent bootstrap. And as the popularity of +the independent bootstrap soared, new variants, such as \textbf{the dependent bootstrap}. + +The Jackknife and independent bootstrap work for +independent, identically distributed random variables. +If these conditions are not +satisfied, the methods will fail. Yet, it should be said that if the data are +independent, identically distributed, and we only want to estimate the +variance of $\overline{X}$ (which often is the case), then there is no +need for bootstrapping. + +% !split +\subsection*{Resampling methods: Jackknife} + +The Jackknife works by making many replicas of the estimator $\widehat{\theta}$. +The jackknife is a resampling method where we systematically leave out one observation from the vector of observed values $\bm{x} = (x_1,x_2,\cdots,X_n)$. +Let $\bm{x}_i$ denote the vector +\[ +\bm{x}_i = (x_1,x_2,\cdots,x_{i-1},x_{i+1},\cdots,x_n), +\] + +which equals the vector $\bm{x}$ with the exception that observation +number $i$ is left out. Using this notation, define +$\widehat{\theta}_i$ to be the estimator +$\widehat{\theta}$ computed using $\vec{X}_i$. + + +% !split +\subsection*{Jackknife code example} +\begin{minted}[fontsize=\fontsize{9pt}{9pt},linenos=false,mathescape,baselinestretch=1.0,fontfamily=tt,xleftmargin=7mm]{python} +from numpy import * +from numpy.random import randint, randn +from time import time + +def jackknife(data, stat): + n = len(data);t = zeros(n); inds = arange(n); t0 = time() + ## 'jackknifing' by leaving out an observation for each i + for i in range(n): + t[i] = stat(delete(data,i) ) + + # analysis + print("Runtime: %g sec" % (time()-t0)); print("Jackknife Statistics :") + print("original bias std. error") + print("%8g %14g %15g" % (stat(data),(n-1)*mean(t)/n, (n*var(t))**.5)) + + return t + + +# Returns mean of data samples +def stat(data): + return mean(data) + + +mu, sigma = 100, 15 +datapoints = 10000 +x = mu + sigma*random.randn(datapoints) +# jackknife returns the data sample +t = jackknife(x, stat) + +\end{minted} + + +% !split +\subsection*{Resampling methods: Bootstrap} + +% --- begin paragraph admon --- +\paragraph{} +Bootstrapping is a nonparametric approach to statistical inference +that substitutes computation for more traditional distributional +assumptions and asymptotic results. Bootstrapping offers a number of +advantages: +\begin{enumerate} +\item The bootstrap is quite general, although there are some cases in which it fails. + +\item Because it does not require distributional assumptions (such as normally distributed errors), the bootstrap can provide more accurate inferences when the data are not well behaved or when the sample size is small. + +\item It is possible to apply the bootstrap to statistics with sampling distributions that are difficult to derive, even asymptotically. + +\item It is relatively simple to apply the bootstrap to complex data-collection plans (such as stratified and clustered samples). +\end{enumerate} + +\noindent +% --- end paragraph admon --- + + + + +% !split +\subsection*{Resampling methods: Bootstrap background} + +Since $\widehat{\theta} = \widehat{\theta}(\bm{X})$ is a function of random variables, +$\widehat{\theta}$ itself must be a random variable. Thus it has +a pdf, call this function $p(\bm{t})$. The aim of the bootstrap is to +estimate $p(\bm{t})$ by the relative frequency of +$\widehat{\theta}$. You can think of this as using a histogram +in the place of $p(\bm{t})$. If the relative frequency closely +resembles $p(\vec{t})$, then using numerics, it is straight forward to +estimate all the interesting parameters of $p(\bm{t})$ using point +estimators. + + +% !split +\subsection*{Resampling methods: More Bootstrap background} + +In the case that $\widehat{\theta}$ has +more than one component, and the components are independent, we use the +same estimator on each component separately. If the probability +density function of $X_i$, $p(x)$, had been known, then it would have +been straight forward to do this by: +\begin{enumerate} +\item Drawing lots of numbers from $p(x)$, suppose we call one such set of numbers $(X_1^*, X_2^*, \cdots, X_n^*)$. + +\item Then using these numbers, we could compute a replica of $\widehat{\theta}$ called $\widehat{\theta}^*$. +\end{enumerate} + +\noindent +By repeated use of (1) and (2), many +estimates of $\widehat{\theta}$ could have been obtained. The +idea is to use the relative frequency of $\widehat{\theta}^*$ +(think of a histogram) as an estimate of $p(\bm{t})$. + +% !split +\subsection*{Resampling methods: Bootstrap approach} + +But +unless there is enough information available about the process that +generated $X_1,X_2,\cdots,X_n$, $p(x)$ is in general +unknown. Therefore, \href{{https://projecteuclid.org/euclid.aos/1176344552}}{Efron in 1979} asked the +question: What if we replace $p(x)$ by the relative frequency +of the observation $X_i$; if we draw observations in accordance with +the relative frequency of the observations, will we obtain the same +result in some asymptotic sense? The answer is yes. + + +Instead of generating the histogram for the relative +frequency of the observation $X_i$, just draw the values +$(X_1^*,X_2^*,\cdots,X_n^*)$ with replacement from the vector +$\bm{X}$. + +% !split +\subsection*{Resampling methods: Bootstrap steps} + +The independent bootstrap works like this: + +\begin{enumerate} +\item Draw with replacement $n$ numbers for the observed variables $\bm{x} = (x_1,x_2,\cdots,x_n)$. + +\item Define a vector $\bm{x}^*$ containing the values which were drawn from $\bm{x}$. + +\item Using the vector $\bm{x}^*$ compute $\widehat{\theta}^*$ by evaluating $\widehat \theta$ under the observations $\bm{x}^*$. + +\item Repeat this process $k$ times. +\end{enumerate} + +\noindent +When you are done, you can draw a histogram of the relative frequency +of $\widehat \theta^*$. This is your estimate of the probability +distribution $p(t)$. Using this probability distribution you can +estimate any statistics thereof. In principle you never draw the +histogram of the relative frequency of $\widehat{\theta}^*$. Instead +you use the estimators corresponding to the statistic of interest. For +example, if you are interested in estimating the variance of $\widehat +\theta$, apply the etsimator $\widehat \sigma^2$ to the values +$\widehat \theta ^*$. + + +% !split +\subsection*{Code example for the Bootstrap method} + +The following code starts with a Gaussian distribution with mean value +$\mu =100$ and variance $\sigma=15$. We use this to generate the data +used in the bootstrap analysis. The bootstrap analysis returns a data +set after a given number of bootstrap operations (as many as we have +data points). This data set consists of estimated mean values for each +bootstrap operation. The histogram generated by the bootstrap method +shows that the distribution for these mean values is also a Gaussian, +centered around the mean value $\mu=100$ but with standard deviation +$\sigma/\sqrt{n}$, where $n$ is the number of bootstrap samples (in +this case the same as the number of original data points). The value +of the standard deviation is what we expect from the central limit +theorem. + + +\begin{minted}[fontsize=\fontsize{9pt}{9pt},linenos=false,mathescape,baselinestretch=1.0,fontfamily=tt,xleftmargin=7mm]{python} +from numpy import * +from numpy.random import randint, randn +from time import time +import matplotlib.mlab as mlab +import matplotlib.pyplot as plt + +# Returns mean of bootstrap samples +def stat(data): + return mean(data) + +# Bootstrap algorithm +def bootstrap(data, statistic, R): + t = zeros(R); n = len(data); inds = arange(n); t0 = time() + # non-parametric bootstrap + for i in range(R): + t[i] = statistic(data[randint(0,n,n)]) + + # analysis + print("Runtime: %g sec" % (time()-t0)); print("Bootstrap Statistics :") + print("original bias std. error") + print("%8g %8g %14g %15g" % (statistic(data), std(data),mean(t),std(t))) + return t + + +mu, sigma = 100, 15 +datapoints = 10000 +x = mu + sigma*random.randn(datapoints) +# bootstrap returns the data sample +t = bootstrap(x, stat, datapoints) +# the histogram of the bootstrapped data +n, binsboot, patches = plt.hist(t, 50, normed=1, facecolor='red', alpha=0.75) + +# add a 'best fit' line +y = mlab.normpdf( binsboot, mean(t), std(t)) +lt = plt.plot(binsboot, y, 'r--', linewidth=1) +plt.xlabel('Smarts') +plt.ylabel('Probability') +plt.axis([99.5, 100.6, 0, 3.0]) +plt.grid(True) + +plt.show() + +\end{minted} + + +% !split +\subsection*{Code Example for Cross-validation and $k$-fold Cross-validation} + +The code here uses Ridge regression with cross-validation (CV) resampling and $k$-fold CV in order to fit a specific polynomial. +\begin{minted}[fontsize=\fontsize{9pt}{9pt},linenos=false,mathescape,baselinestretch=1.0,fontfamily=tt,xleftmargin=7mm]{python} +import numpy as np +import matplotlib.pyplot as plt +from sklearn.model_selection import KFold +from sklearn.linear_model import Ridge +from sklearn.model_selection import cross_val_score +from sklearn.preprocessing import PolynomialFeatures + +# A seed just to ensure that the random numbers are the same for every run. +# Useful for eventual debugging. +np.random.seed(3155) + +# Generate the data. +nsamples = 100 +x = np.random.randn(nsamples) +y = 3*x**2 + np.random.randn(nsamples) + +## Cross-validation on Ridge regression using KFold only + +# Decide degree on polynomial to fit +poly = PolynomialFeatures(degree = 6) + +# Decide which values of lambda to use +nlambdas = 500 +lambdas = np.logspace(-3, 5, nlambdas) + +# Initialize a KFold instance +k = 5 +kfold = KFold(n_splits = k) + +# Perform the cross-validation to estimate MSE +scores_KFold = np.zeros((nlambdas, k)) + +i = 0 +for lmb in lambdas: + ridge = Ridge(alpha = lmb) + j = 0 + for train_inds, test_inds in kfold.split(x): + xtrain = x[train_inds] + ytrain = y[train_inds] + + xtest = x[test_inds] + ytest = y[test_inds] + + Xtrain = poly.fit_transform(xtrain[:, np.newaxis]) + ridge.fit(Xtrain, ytrain[:, np.newaxis]) + + Xtest = poly.fit_transform(xtest[:, np.newaxis]) + ypred = ridge.predict(Xtest) + + scores_KFold[i,j] = np.sum((ypred - ytest[:, np.newaxis])**2)/np.size(ypred) + + j += 1 + i += 1 + + +estimated_mse_KFold = np.mean(scores_KFold, axis = 1) + +## Cross-validation using cross_val_score from sklearn along with KFold + +# kfold is an instance initialized above as: +# kfold = KFold(n_splits = k) + +estimated_mse_sklearn = np.zeros(nlambdas) +i = 0 +for lmb in lambdas: + ridge = Ridge(alpha = lmb) + + X = poly.fit_transform(x[:, np.newaxis]) + estimated_mse_folds = cross_val_score(ridge, X, y[:, np.newaxis], scoring='neg_mean_squared_error', cv=kfold) + + # cross_val_score return an array containing the estimated negative mse for every fold. + # we have to the the mean of every array in order to get an estimate of the mse of the model + estimated_mse_sklearn[i] = np.mean(-estimated_mse_folds) + + i += 1 + +## Plot and compare the slightly different ways to perform cross-validation + +plt.figure() + +plt.plot(np.log10(lambdas), estimated_mse_sklearn, label = 'cross_val_score') +plt.plot(np.log10(lambdas), estimated_mse_KFold, 'r--', label = 'KFold') + +plt.xlabel('log10(lambda)') +plt.ylabel('mse') + +plt.legend() + +plt.show() + +\end{minted} + + +% !split +\subsection*{The bias-variance tradeoff} + + +We will discuss the bias-variance tradeoff in the context of +continuous predictions such as regression. However, many of the +intuitions and ideas discussed here also carry over to classification +tasks. Consider a dataset $\mathcal{L}$ consisting of the data +$\mathbf{X}_\mathcal{L}=\{(y_j, \boldsymbol{x}_j), j=0\ldots n-1\}$. + +Let us assume that the true data is generated from a noisy model + +\[ +\bm{y}=f(\boldsymbol{x}) + \bm{\epsilon} +\] + +where $\epsilon$ is normally distributed with mean zero and standard deviation $\sigma^2$. + +In our derivation of the ordinary least squares method we defined then +an approximation to the function $f$ in terms of the parameters +$\bm{\beta}$ and the design matrix $\bm{X}$ which embody our model, +that is $\bm{\tilde{y}}=\bm{X}\bm{\beta}$. + +Thereafter we found the parameters $\bm{\beta}$ by optimizing the means squared error via the so-called cost function +\[ +C(\bm{X},\bm{\beta}) =\frac{1}{n}\sum_{i=0}^{n-1}(y_i-\tilde{y}_i)^2=\mathbb{E}\left[(\bm{y}-\bm{\tilde{y}})^2\right]. +\] + +We can rewrite this as +\[ +\mathbb{E}\left[(\bm{y}-\bm{\tilde{y}})^2\right]=\frac{1}{n}\sum_i(f_i-\mathbb{E}\left[\bm{\tilde{y}}\right])^2+\frac{1}{n}\sum_i(\tilde{y}_i-\mathbb{E}\left[\bm{\tilde{y}}\right])^2+\sigma^2. +\] + +The three terms represent the square of the bias of the learning +method, which can be thought of as the error caused by the simplifying +assumptions built into the method. The second term represents the +variance of the chosen model and finally the last terms is variance of +the error $\bm{\epsilon}$. + +To derive this equation, we need to recall that the variance of $\bm{y}$ and $\bm{\epsilon}$ are both equal to $\sigma^2$. The mean value of $\bm{\epsilon}$ is by definition equal to zero. Furthermore, the function $f$ is not a stochastics variable, idem for $\bm{\tilde{y}}$. +We use a more compact notation in terms of the expectation value +\[ +\mathbb{E}\left[(\bm{y}-\bm{\tilde{y}})^2\right]=\mathbb{E}\left[(\bm{f}+\bm{\epsilon}-\bm{\tilde{y}})^2\right], +\] +and adding and subtracting $\mathbb{E}\left[\bm{\tilde{y}}\right]$ we get +\[ +\mathbb{E}\left[(\bm{y}-\bm{\tilde{y}})^2\right]=\mathbb{E}\left[(\bm{f}+\bm{\epsilon}-\bm{\tilde{y}}+\mathbb{E}\left[\bm{\tilde{y}}\right]-\mathbb{E}\left[\bm{\tilde{y}}\right])^2\right], +\] +which, using the abovementioned expectation values can be rewritten as +\[ +\mathbb{E}\left[(\bm{y}-\bm{\tilde{y}})^2\right]=\mathbb{E}\left[(\bm{y}-\mathbb{E}\left[\bm{\tilde{y}}\right])^2\right]+\mathrm{Var}\left[\bm{\tilde{y}}\right]+\sigma^2, +\] +that is the rewriting in terms of the so-called bias, the variance of the model $\bm{\tilde{y}}$ and the variance of $\bm{\epsilon}$. + + + + + +% !split +\subsection*{Example code for Bias-Variance tradeoff} +\begin{minted}[fontsize=\fontsize{9pt}{9pt},linenos=false,mathescape,baselinestretch=1.0,fontfamily=tt,xleftmargin=7mm]{python} +import matplotlib.pyplot as plt +import numpy as np +from sklearn.linear_model import LinearRegression, Ridge, Lasso +from sklearn.preprocessing import PolynomialFeatures +from sklearn.model_selection import train_test_split +from sklearn.pipeline import make_pipeline +from sklearn.utils import resample + +np.random.seed(2018) + +n = 500 +n_boostraps = 100 +degree = 18 # A quite high value, just to show. +noise = 0.1 + +# Make data set. +x = np.linspace(-1, 3, n).reshape(-1, 1) +y = np.exp(-x**2) + 1.5 * np.exp(-(x-2)**2) + np.random.normal(0, 0.1, x.shape) + +# Hold out some test data that is never used in training. +x_train, x_test, y_train, y_test = train_test_split(x, y, test_size=0.2) + +# Combine x transformation and model into one operation. +# Not neccesary, but convenient. +model = make_pipeline(PolynomialFeatures(degree=degree), LinearRegression(fit_intercept=False)) + +# The following (m x n_bootstraps) matrix holds the column vectors y_pred +# for each bootstrap iteration. +y_pred = np.empty((y_test.shape[0], n_boostraps)) +for i in range(n_boostraps): + x_, y_ = resample(x_train, y_train) + + # Evaluate the new model on the same test data each time. + y_pred[:, i] = model.fit(x_, y_).predict(x_test).ravel() + +# Note: Expectations and variances taken w.r.t. different training +# data sets, hence the axis=1. Subsequent means are taken across the test data +# set in order to obtain a total value, but before this we have error/bias/variance +# calculated per data point in the test set. +# Note 2: The use of keepdims=True is important in the calculation of bias as this +# maintains the column vector form. Dropping this yields very unexpected results. +error = np.mean( np.mean((y_test - y_pred)**2, axis=1, keepdims=True) ) +bias = np.mean( (y_test - np.mean(y_pred, axis=1, keepdims=True))**2 ) +variance = np.mean( np.var(y_pred, axis=1, keepdims=True) ) +print('Error:', error) +print('Bias^2:', bias) +print('Var:', variance) +print('{} >= {} + {} = {}'.format(error, bias, variance, bias+variance)) + +plt.plot(x[::5, :], y[::5, :], label='f(x)') +plt.scatter(x_test, y_test, label='Data points') +plt.scatter(x_test, np.mean(y_pred, axis=1), label='Pred') +plt.legend() +plt.show() + +\end{minted} + + +% !split +\subsection*{Understanding what happens} +\begin{minted}[fontsize=\fontsize{9pt}{9pt},linenos=false,mathescape,baselinestretch=1.0,fontfamily=tt,xleftmargin=7mm]{python} +import matplotlib.pyplot as plt +import numpy as np +from sklearn.linear_model import LinearRegression, Ridge, Lasso +from sklearn.preprocessing import PolynomialFeatures +from sklearn.model_selection import train_test_split +from sklearn.pipeline import make_pipeline +from sklearn.utils import resample + +np.random.seed(2018) + +n = 40 +n_boostraps = 100 +maxdegree = 14 + + +# Make data set. +x = np.linspace(-3, 3, n).reshape(-1, 1) +y = np.exp(-x**2) + 1.5 * np.exp(-(x-2)**2)+ np.random.normal(0, 0.1, x.shape) +error = np.zeros(maxdegree) +bias = np.zeros(maxdegree) +variance = np.zeros(maxdegree) +polydegree = np.zeros(maxdegree) +x_train, x_test, y_train, y_test = train_test_split(x, y, test_size=0.2) + +for degree in range(maxdegree): + model = make_pipeline(PolynomialFeatures(degree=degree), LinearRegression(fit_intercept=False)) + y_pred = np.empty((y_test.shape[0], n_boostraps)) + for i in range(n_boostraps): + x_, y_ = resample(x_train, y_train) + y_pred[:, i] = model.fit(x_, y_).predict(x_test).ravel() + + polydegree[degree] = degree + error[degree] = np.mean( np.mean((y_test - y_pred)**2, axis=1, keepdims=True) ) + bias[degree] = np.mean( (y_test - np.mean(y_pred, axis=1, keepdims=True))**2 ) + variance[degree] = np.mean( np.var(y_pred, axis=1, keepdims=True) ) + print('Polynomial degree:', degree) + print('Error:', error[degree]) + print('Bias^2:', bias[degree]) + print('Var:', variance[degree]) + print('{} >= {} + {} = {}'.format(error[degree], bias[degree], variance[degree], bias[degree]+variance[degree])) + +plt.plot(polydegree, np.log10(error), label='Error') +plt.plot(polydegree, bias, label='bias') +plt.plot(polydegree, variance, label='Variance') +plt.legend() +plt.show() + + + + +\end{minted} + +% !split +\subsection*{Summing up} + + + + +The bias-variance tradeoff summarizes the fundamental tension in +machine learning, particularly supervised learning, between the +complexity of a model and the amount of training data needed to train +it. Since data is often limited, in practice it is often useful to +use a less-complex model with higher bias, that is a model whose asymptotic +performance is worse than another model because it is easier to +train and less sensitive to sampling noise arising from having a +finite-sized training dataset (smaller variance). + + + +The above equations tell us that in +order to minimize the expected test error, we need to select a +statistical learning method that simultaneously achieves low variance +and low bias. Note that variance is inherently a nonnegative quantity, +and squared bias is also nonnegative. Hence, we see that the expected +test MSE can never lie below $Var(\epsilon)$, the irreducible error. + + +What do we mean by the variance and bias of a statistical learning +method? The variance refers to the amount by which our model would change if we +estimated it using a different training data set. Since the training +data are used to fit the statistical learning method, different +training data sets will result in a different estimate. But ideally the +estimate for our model should not vary too much between training +sets. However, if a method has high variance then small changes in +the training data can result in large changes in the model. In general, more +flexible statistical methods have higher variance. + + +% !split +\subsection*{Another Example rom Scikit-Learn's Repository} +\begin{minted}[fontsize=\fontsize{9pt}{9pt},linenos=false,mathescape,baselinestretch=1.0,fontfamily=tt,xleftmargin=7mm]{python} +""" +============================ +Underfitting vs. Overfitting +============================ + +This example demonstrates the problems of underfitting and overfitting and +how we can use linear regression with polynomial features to approximate +nonlinear functions. The plot shows the function that we want to approximate, +which is a part of the cosine function. In addition, the samples from the +real function and the approximations of different models are displayed. The +models have polynomial features of different degrees. We can see that a +linear function (polynomial with degree 1) is not sufficient to fit the +training samples. This is called **underfitting**. A polynomial of degree 4 +approximates the true function almost perfectly. However, for higher degrees +the model will **overfit** the training data, i.e. it learns the noise of the +training data. +We evaluate quantitatively **overfitting** / **underfitting** by using +cross-validation. We calculate the mean squared error (MSE) on the validation +set, the higher, the less likely the model generalizes correctly from the +training data. +""" + +print(__doc__) + +import numpy as np +import matplotlib.pyplot as plt +from sklearn.pipeline import Pipeline +from sklearn.preprocessing import PolynomialFeatures +from sklearn.linear_model import LinearRegression +from sklearn.model_selection import cross_val_score + + +def true_fun(X): + return np.cos(1.5 * np.pi * X) + +np.random.seed(0) + +n_samples = 30 +degrees = [1, 4, 15] + +X = np.sort(np.random.rand(n_samples)) +y = true_fun(X) + np.random.randn(n_samples) * 0.1 + +plt.figure(figsize=(14, 5)) +for i in range(len(degrees)): + ax = plt.subplot(1, len(degrees), i + 1) + plt.setp(ax, xticks=(), yticks=()) + + polynomial_features = PolynomialFeatures(degree=degrees[i], + include_bias=False) + linear_regression = LinearRegression() + pipeline = Pipeline([("polynomial_features", polynomial_features), + ("linear_regression", linear_regression)]) + pipeline.fit(X[:, np.newaxis], y) + + # Evaluate the models using crossvalidation + scores = cross_val_score(pipeline, X[:, np.newaxis], y, + scoring="neg_mean_squared_error", cv=10) + + X_test = np.linspace(0, 1, 100) + plt.plot(X_test, pipeline.predict(X_test[:, np.newaxis]), label="Model") + plt.plot(X_test, true_fun(X_test), label="True function") + plt.scatter(X, y, edgecolor='b', s=20, label="Samples") + plt.xlabel("x") + plt.ylabel("y") + plt.xlim((0, 1)) + plt.ylim((-2, 2)) + plt.legend(loc="best") + plt.title("Degree {}\nMSE = {:.2e}(+/- {:.2e})".format( + degrees[i], -scores.mean(), scores.std())) +plt.show() +\end{minted} + + + +% !split +\subsection*{The one-dimensional Ising model} + +Let us bring back the Ising model again, but now with an additional +focus on Ridge and Lasso regression as well. We repeat some of the +basic parts of the Ising model and the setup of the training and test +data. The one-dimensional Ising model with nearest neighbor +interaction, no external field and a constant coupling constant $J$ is +given by + +\begin{align} + H = -J \sum_{k}^L s_k s_{k + 1}, +\end{align} +where $s_i \in \{-1, 1\}$ and $s_{N + 1} = s_1$. The number of spins in the system is determined by $L$. For the one-dimensional system there is no phase transition. + +We will look at a system of $L = 40$ spins with a coupling constant of $J = 1$. To get enough training data we will generate 10000 states with their respective energies. + + +\begin{minted}[fontsize=\fontsize{9pt}{9pt},linenos=false,mathescape,baselinestretch=1.0,fontfamily=tt,xleftmargin=7mm]{python} +import numpy as np +import matplotlib.pyplot as plt +from mpl_toolkits.axes_grid1 import make_axes_locatable +import seaborn as sns +import scipy.linalg as scl +from sklearn.model_selection import train_test_split +import sklearn.linear_model as skl +import tqdm +sns.set(color_codes=True) +cmap_args=dict(vmin=-1., vmax=1., cmap='seismic') + +L = 40 +n = int(1e4) + +spins = np.random.choice([-1, 1], size=(n, L)) +J = 1.0 + +energies = np.zeros(n) + +for i in range(n): + energies[i] = - J * np.dot(spins[i], np.roll(spins[i], 1)) +\end{minted} + +A more general form for the one-dimensional Ising model is + +\begin{align} + H = - \sum_j^L \sum_k^L s_j s_k J_{jk}. +\end{align} + +Here we allow for interactions beyond the nearest neighbors and a more +adaptive coupling matrix. This latter expression can be formulated as +a matrix-product on the form +\begin{align} + H = X J, +\end{align} + +where $X_{jk} = s_j s_k$ and $J$ is the matrix consisting of the +elements $-J_{jk}$. This form of writing the energy fits perfectly +with the form utilized in linear regression, viz. +\begin{align} + \bm{y} = \bm{X}\bm{\beta} + \bm{\epsilon}. +\end{align} +We organize the data as we did above +\begin{minted}[fontsize=\fontsize{9pt}{9pt},linenos=false,mathescape,baselinestretch=1.0,fontfamily=tt,xleftmargin=7mm]{python} +X = np.zeros((n, L ** 2)) +for i in range(n): + X[i] = np.outer(spins[i], spins[i]).ravel() +y = energies +X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.96) + +X_train_own = np.concatenate( + (np.ones(len(X_train))[:, np.newaxis], X_train), + axis=1 +) + +X_test_own = np.concatenate( + (np.ones(len(X_test))[:, np.newaxis], X_test), + axis=1 +) +\end{minted} + +We will do all fitting with \textbf{Scikit-Learn}, + +\begin{minted}[fontsize=\fontsize{9pt}{9pt},linenos=false,mathescape,baselinestretch=1.0,fontfamily=tt,xleftmargin=7mm]{python} +clf = skl.LinearRegression().fit(X_train, y_train) +\end{minted} +When extracting the $J$-matrix we make sure to remove the intercept +\begin{minted}[fontsize=\fontsize{9pt}{9pt},linenos=false,mathescape,baselinestretch=1.0,fontfamily=tt,xleftmargin=7mm]{python} +J_sk = clf.coef_.reshape(L, L) +\end{minted} +And then we plot the results +\begin{minted}[fontsize=\fontsize{9pt}{9pt},linenos=false,mathescape,baselinestretch=1.0,fontfamily=tt,xleftmargin=7mm]{python} +fig = plt.figure(figsize=(20, 14)) +im = plt.imshow(J_sk, **cmap_args) +plt.title("LinearRegression from Scikit-learn", fontsize=18) +plt.xticks(fontsize=18) +plt.yticks(fontsize=18) +cb = fig.colorbar(im) +cb.ax.set_yticklabels(cb.ax.get_yticklabels(), fontsize=18) +plt.show() +\end{minted} +The results perfectly with our previous discussion where we used our own code. + +% !split +\subsection*{Ridge regression} + +Having explored the ordinary least squares we move on to ridge +regression. In ridge regression we include a \textbf{regularizer}. This +involves a new cost function which leads to a new estimate for the +weights $\bm{\beta}$. This results in a penalized regression problem. The +cost function is given by + +\begin{align} + C(\bm{X}, \bm{\beta}; \lambda) = (\bm{X}\bm{\beta} - \bm{y})^T(\bm{X}\bm{\beta} - \bm{y}) + \lambda \bm{\beta}^T\bm{\beta}. +\end{align} +\begin{minted}[fontsize=\fontsize{9pt}{9pt},linenos=false,mathescape,baselinestretch=1.0,fontfamily=tt,xleftmargin=7mm]{python} +_lambda = 0.1 +clf_ridge = skl.Ridge(alpha=_lambda).fit(X_train, y_train) +J_ridge_sk = clf_ridge.coef_.reshape(L, L) +fig = plt.figure(figsize=(20, 14)) +im = plt.imshow(J_ridge_sk, **cmap_args) +plt.title("Ridge from Scikit-learn", fontsize=18) +plt.xticks(fontsize=18) +plt.yticks(fontsize=18) +cb = fig.colorbar(im) +cb.ax.set_yticklabels(cb.ax.get_yticklabels(), fontsize=18) + +plt.show() +\end{minted} + +% !split +\subsection*{LASSO regression} + +In the \textbf{Least Absolute Shrinkage and Selection Operator} (LASSO)-method we get a third cost function. + +\begin{align} + C(\bm{X}, \bm{\beta}; \lambda) = (\bm{X}\bm{\beta} - \bm{y})^T(\bm{X}\bm{\beta} - \bm{y}) + \lambda \sqrt{\bm{\beta}^T\bm{\beta}}. +\end{align} + +Finding the extremal point of this cost function is not so straight-forward as in least squares and ridge. We will therefore rely solely on the function ``Lasso`` from \textbf{Scikit-Learn}. + +\begin{minted}[fontsize=\fontsize{9pt}{9pt},linenos=false,mathescape,baselinestretch=1.0,fontfamily=tt,xleftmargin=7mm]{python} +clf_lasso = skl.Lasso(alpha=_lambda).fit(X_train, y_train) +J_lasso_sk = clf_lasso.coef_.reshape(L, L) +fig = plt.figure(figsize=(20, 14)) +im = plt.imshow(J_lasso_sk, **cmap_args) +plt.title("Lasso from Scikit-learn", fontsize=18) +plt.xticks(fontsize=18) +plt.yticks(fontsize=18) +cb = fig.colorbar(im) +cb.ax.set_yticklabels(cb.ax.get_yticklabels(), fontsize=18) + +plt.show() +\end{minted} + +It is quite striking how LASSO breaks the symmetry of the coupling +constant as opposed to ridge and OLS. We get a sparse solution with +$J_{j, j + 1} = -1$. + + + +% !split +\subsection*{Performance as function of the regularization parameter} + +We see how the different models perform for a different set of values for $\lambda$. + + +\begin{minted}[fontsize=\fontsize{9pt}{9pt},linenos=false,mathescape,baselinestretch=1.0,fontfamily=tt,xleftmargin=7mm]{python} +lambdas = np.logspace(-4, 5, 10) + +train_errors = { + "ols_sk": np.zeros(lambdas.size), + "ridge_sk": np.zeros(lambdas.size), + "lasso_sk": np.zeros(lambdas.size) +} + +test_errors = { + "ols_sk": np.zeros(lambdas.size), + "ridge_sk": np.zeros(lambdas.size), + "lasso_sk": np.zeros(lambdas.size) +} + +plot_counter = 1 + +fig = plt.figure(figsize=(32, 54)) + +for i, _lambda in enumerate(tqdm.tqdm(lambdas)): + for key, method in zip( + ["ols_sk", "ridge_sk", "lasso_sk"], + [skl.LinearRegression(), skl.Ridge(alpha=_lambda), skl.Lasso(alpha=_lambda)] + ): + method = method.fit(X_train, y_train) + + train_errors[key][i] = method.score(X_train, y_train) + test_errors[key][i] = method.score(X_test, y_test) + + omega = method.coef_.reshape(L, L) + + plt.subplot(10, 5, plot_counter) + plt.imshow(omega, **cmap_args) + plt.title(r"%s, $\lambda = %.4f$" % (key, _lambda)) + plot_counter += 1 + +plt.show() +\end{minted} + +We see that LASSO reaches a good solution for low +values of $\lambda$, but will "wither" when we increase $\lambda$ too +much. Ridge is more stable over a larger range of values for +$\lambda$, but eventually also fades away. + +% !split +\subsection*{Finding the optimal value of $\lambda$} + +To determine which value of $\lambda$ is best we plot the accuracy of +the models when predicting the training and the testing set. We expect +the accuracy of the training set to be quite good, but if the accuracy +of the testing set is much lower this tells us that we might be +subject to an overfit model. The ideal scenario is an accuracy on the +testing set that is close to the accuracy of the training set. + + +\begin{minted}[fontsize=\fontsize{9pt}{9pt},linenos=false,mathescape,baselinestretch=1.0,fontfamily=tt,xleftmargin=7mm]{python} +fig = plt.figure(figsize=(20, 14)) + +colors = { + "ols_sk": "r", + "ridge_sk": "y", + "lasso_sk": "c" +} + +for key in train_errors: + plt.semilogx( + lambdas, + train_errors[key], + colors[key], + label="Train {0}".format(key), + linewidth=4.0 + ) + +for key in test_errors: + plt.semilogx( + lambdas, + test_errors[key], + colors[key] + "--", + label="Test {0}".format(key), + linewidth=4.0 + ) +plt.legend(loc="best", fontsize=18) +plt.xlabel(r"$\lambda$", fontsize=18) +plt.ylabel(r"$R^2$", fontsize=18) +plt.tick_params(labelsize=18) +plt.show() +\end{minted} + +From the above figure we can see that LASSO with $\lambda = 10^{-2}$ +achieves a very good accuracy on the test set. This by far surpasses the +other models for all values of $\lambda$. + + + +% !split +\subsection*{Further Exercises} + +\paragraph{Exercise 1.} +We will generate our own dataset for a function $y(x)$ where $x \in [0,1]$ and defined by random numbers computed with the uniform distribution. The function $y$ is a quadratic polynomial in $x$ with added stochastic noise according to the normal distribution $\cal {N}(0,1)$. +The following simple Python instructions define our $x$ and $y$ values (with 100 data points). +\begin{minted}[fontsize=\fontsize{9pt}{9pt},linenos=false,mathescape,baselinestretch=1.0,fontfamily=tt,xleftmargin=7mm]{python} +x = np.random.rand(100,1) +y = 5*x*x+0.1*np.random.randn(100,1) +\end{minted} + +\begin{enumerate} +\item Write your own code (following the examples above) for computing the parametrization of the data set fitting a second-order polynomial. + +\item Use thereafter \textbf{scikit-learn} (see again the examples in the regression slides) and compare with your own code. + +\item Using scikit-learn, compute also the mean square error, a risk metric corresponding to the expected value of the squared (quadratic) error defined as +\end{enumerate} + +\noindent +\[ MSE(\hat{y},\hat{\tilde{y}}) = \frac{1}{n} +\sum_{i=0}^{n-1}(y_i-\tilde{y}_i)^2, +\] +and the $R^2$ score function. +If $\tilde{\hat{y}}_i$ is the predicted value of the $i-th$ sample and $y_i$ is the corresponding true value, then the score $R^2$ is defined as +\[ +R^2(\hat{y}, \tilde{\hat{y}}) = 1 - \frac{\sum_{i=0}^{n - 1} (y_i - \tilde{y}_i)^2}{\sum_{i=0}^{n - 1} (y_i - \bar{y})^2}, +\] +where we have defined the mean value of $\hat{y}$ as +\[ +\bar{y} = \frac{1}{n} \sum_{i=0}^{n - 1} y_i. +\] + +You can use the functionality included in scikit-learn. If you feel +for it, you can use your own program and define functions which +compute the above two functions. Discuss the meaning of these +results. Try also to vary the coefficient in front of the added +stochastic noise term and discuss the quality of the fits. + + + + +\paragraph{Exercise 2, variance of the parameters $\beta$ in linear regression.} +Show that the variance of the parameters $\beta$ in the linear regression method (chapter 3, equation (3.8) of \href{{https://www.springer.com/gp/book/9780387848570}}{Trevor Hastie, Robert Tibshirani, Jerome H. Friedman, The Elements of Statistical Learning, Springer}) is given as + +\[ +\mathrm{Var}(\hat{\beta}) = \left(\hat{X}^T\hat{X}\right)^{-1}\sigma^2, +\] +with +\[ +\sigma^2 = \frac{1}{N-p-1}\sum_{i=1}^{N} (y_i-\tilde{y}_i)^2, +\] +where we have assumed that we fit a function of degree $p-1$ (for example a polynomial in $x$). + + + +\paragraph{Exercise 3.} +This exercise is a continuation of exercise 1. We will +use the same function to generate our data set, still staying with a +simple function $y(x)$ which we want to fit using linear regression, +but now extending the analysis to include the Ridge and the Lasso +regression methods. You can use the code under the Regression as an example on how to use the Ridge and the Lasso methods. + +We will thus again generate our own dataset for a function $y(x)$ where +$x \in [0,1]$ and defined by random numbers computed with the uniform +distribution. The function $y$ is a quadratic polynomial in $x$ with +added stochastic noise according to the normal distribution $\cal{N}(0,1)$. + +The following simple Python instructions define our $x$ and $y$ values (with 100 data points). +\begin{minted}[fontsize=\fontsize{9pt}{9pt},linenos=false,mathescape,baselinestretch=1.0,fontfamily=tt,xleftmargin=7mm]{python} +x = np.random.rand(100,1) +y = 5*x*x+0.1*np.random.randn(100,1) +\end{minted} + +\begin{enumerate} +\item Write your own code for the Ridge method and compute the parametrization for different values of $\lambda$. Compare and analyze your results with those from exercise 1. Study the dependence on $\lambda$ while also varying the strength of the noise in your expression for $y(x)$. + +\item Repeat the above but using the functionality of \textbf{scikit-learn}. Compare your code with the results from \textbf{scikit-learn}. Remember to run with the same random numbers for generating $x$ and $y$. + +\item Our next step is to study the variance of the parameters $\beta_1$ and $\beta_2$ (assuming that we are parametrizing our function with a second-order polynomial. We will use standard linear regression and the Ridge regression. You can now opt for either writing your own function that calculates the variance of these paramaters (recall that this is equal to the diagonal elements of the matrix $(\hat{X}^T\hat{X})+\lambda\hat{I})^{-1}$) or use the functionality of \textbf{scikit-learn} and compute their variances. Discuss the results of these variances as functions + +\item Repeat the previous step but add now the Lasso method. Discuss your results and compare with standard regression and the Ridge regression results. + +\item Try to implement the cross-validation as well. + +\item Finally, using \textbf{scikit-learn} or your own code, compute also the mean square error, a risk metric corresponding to the expected value of the squared (quadratic) error defined as +\end{enumerate} + +\noindent +\[ MSE(\hat{y},\hat{\tilde{y}}) = \frac{1}{n} +\sum_{i=0}^{n-1}(y_i-\tilde{y}_i)^2, +\] +and the $R^2$ score function. +If $\tilde{\hat{y}}_i$ is the predicted value of the $i-th$ sample and $y_i$ is the corresponding true value, then the score $R^2$ is defined as +\[ +R^2(\hat{y}, \tilde{\hat{y}}) = 1 - \frac{\sum_{i=0}^{n - 1} (y_i - \tilde{y}_i)^2}{\sum_{i=0}^{n - 1} (y_i - \bar{y})^2}, +\] +where we have defined the mean value of $\hat{y}$ as +\[ +\bar{y} = \frac{1}{n} \sum_{i=0}^{n - 1} y_i. +\] +Discuss these quantities as functions of the variable $\lambda$ in the Ridge and Lasso regression methods. + +\paragraph{Exercise 4.} +We will study how +to fit polynomials to a specific two-dimensional function called +\href{{http://www.dtic.mil/dtic/tr/fulltext/u2/a081688.pdf}}{Franke's +function}. This +is a function which has been widely used when testing various interpolation and fitting +algorithms. Furthermore, after having established the model and the +method, we will employ resamling techniques such as the cross-validation and/or +the bootstrap methods, in order to perform a proper assessment of our models. + + +The Franke function, which is a weighted sum of four exponentials reads as follows +\begin{align*} +f(x,y) &= \frac{3}{4}\exp{\left(-\frac{(9x-2)^2}{4} - \frac{(9y-2)^2}{4}\right)}+\frac{3}{4}\exp{\left(-\frac{(9x+1)^2}{49}- \frac{(9y+1)}{10}\right)} \\ +&+\frac{1}{2}\exp{\left(-\frac{(9x-7)^2}{4} - \frac{(9y-3)^2}{4}\right)} -\frac{1}{5}\exp{\left(-(9x-4)^2 - (9y-7)^2\right) }. +\end{align*} + +The function will be defined for $x,y\in [0,1]$. Our first step will +be to perform an OLS regression analysis of this function, trying out +a polynomial fit with an $x$ and $y$ dependence of the form $[x, y, +x^2, y^2, xy, \dots]$. We will also include cross-validation and +bootstrap as resampling techniques. As in homeworks 1 and 2, we +can use a uniform distribution to set up the arrays of values for $x$ +and $y$, or as in the example below just a fix values for $x$ and $y$ with a given step size. +In this case we will have two predictors and need to fit a +function (for example a polynomial) of $x$ and $y$. Thereafter we will +repeat much of the same procedure using the the Ridge and +Lasso regression methods, introducing thus a dependence on the bias +(penalty) $\lambda$. + + +The Python fucntion for the Franke function is included here (it performs also a three-dimensional plot of it) +\begin{minted}[fontsize=\fontsize{9pt}{9pt},linenos=false,mathescape,baselinestretch=1.0,fontfamily=tt,xleftmargin=7mm]{python} +from mpl_toolkits.mplot3d import Axes3D +import matplotlib.pyplot as plt +from matplotlib import cm +from matplotlib.ticker import LinearLocator, FormatStrFormatter +import numpy as np +from random import random, seed + +fig = plt.figure() +ax = fig.gca(projection='3d') + +# Make data. +x = np.arange(0, 1, 0.05) +y = np.arange(0, 1, 0.05) +x, y = np.meshgrid(x,y) + + +def FrankeFunction(x,y): + term1 = 0.75*np.exp(-(0.25*(9*x-2)**2) - 0.25*((9*y-2)**2)) + term2 = 0.75*np.exp(-((9*x+1)**2)/49.0 - 0.1*(9*y+1)) + term3 = 0.5*np.exp(-(9*x-7)**2/4.0 - 0.25*((9*y-3)**2)) + term4 = -0.2*np.exp(-(9*x-4)**2 - (9*y-7)**2) + return term1 + term2 + term3 + term4 + + +z = FrankeFunction(x, y) + +# Plot the surface. +surf = ax.plot_surface(x, y, z, cmap=cm.coolwarm, + linewidth=0, antialiased=False) + +# Customize the z axis. +ax.set_zlim(-0.10, 1.40) +ax.zaxis.set_major_locator(LinearLocator(10)) +ax.zaxis.set_major_formatter(FormatStrFormatter('%.02f')) + +# Add a color bar which maps values to colors. +fig.colorbar(surf, shrink=0.5, aspect=5) + +plt.show() + +\end{minted} + + +We will thus again generate our own dataset for a function $\mathrm{FrankeFunction}(x,y)$ where +$x,y \in [0,1]$ could be defined by random numbers computed with the uniform +distribution. The function $f(x,y)$ is the Franke function. You should explore also the addition +an added stochastic noise to this function using the normal distribution $\cal{N}(0,1)$. + +Write your own code (using either a matrix inversion or a singular value decomposition from e.g., \textbf{numpy} ) or use your code from exercises 1 and 3 +and perform a standard least square regression analysis using polynomials in $x$ and $y$ up to fifth order. Find the confidence intervals of the parameters $\beta$ by computing their variances, evaluate the Mean Squared error (MSE) +\[ MSE(\hat{y},\hat{\tilde{y}}) = \frac{1}{n} +\sum_{i=0}^{n-1}(y_i-\tilde{y}_i)^2, +\] +and the $R^2$ score function. +If $\tilde{\hat{y}}_i$ is the predicted value of the $i-th$ sample and $y_i$ is the corresponding true value, then the score $R^2$ is defined as +\[ +R^2(\hat{y}, \tilde{\hat{y}}) = 1 - \frac{\sum_{i=0}^{n - 1} (y_i - \tilde{y}_i)^2}{\sum_{i=0}^{n - 1} (y_i - \bar{y})^2}, +\] +where we have defined the mean value of $\hat{y}$ as +\[ +\bar{y} = \frac{1}{n} \sum_{i=0}^{n - 1} y_i. +\] + +Perform a resampling of the data where you split the data in training data and test data. Implement the $k$-fold cross-validation algorithm and/or the bootstrap algorithm +and evaluate again the MSE and the $R^2$ functions resulting from the test data. Evaluate also the bias and variance of the final models. + + +Write then your own code for the Ridge method, either using matrix +inversion or the singular value decomposition as done for standard OLS. Perform the same analysis as in the +previous exercise (for the same polynomials and include resampling +techniques) but now for different values of $\lambda$. Compare and +analyze your results with those obtained with standard OLS. Study the +dependence on $\lambda$ while also varying eventually the strength of +the noise in your expression for $\mathrm{FrankeFunction}(x,y)$. + +Then perform the same studies but now with Lasso regression. Use the functionalities of +\textbf{scikit-learn}. Give a critical discussion of the three methods and a +judgement of which model fits the data best. + + +% ------------------- end of main content --------------- + +\end{document} + diff --git a/doc/src/Regression/Regression-reveal.html b/doc/src/Regression/Regression-reveal.html new file mode 100644 index 000000000..e55e0387b --- /dev/null +++ b/doc/src/Regression/Regression-reveal.html @@ -0,0 +1,4558 @@ + + + + + + + +Data Analysis and Machine Learning: Linear Regression and more Advanced Regression Analysis + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +