diff --git a/doc/LectureNotes/_build/.doctrees/environment.pickle b/doc/LectureNotes/_build/.doctrees/environment.pickle index 2159d6ab1..3e6742268 100644 Binary files a/doc/LectureNotes/_build/.doctrees/environment.pickle and b/doc/LectureNotes/_build/.doctrees/environment.pickle differ diff --git a/doc/LectureNotes/_build/.doctrees/week37.doctree b/doc/LectureNotes/_build/.doctrees/week37.doctree index 4f4067067..a18017cac 100644 Binary files a/doc/LectureNotes/_build/.doctrees/week37.doctree and b/doc/LectureNotes/_build/.doctrees/week37.doctree differ diff --git a/doc/LectureNotes/_build/html/_sources/week37.ipynb b/doc/LectureNotes/_build/html/_sources/week37.ipynb index acd6b4e52..6ec95c3cd 100644 --- a/doc/LectureNotes/_build/html/_sources/week37.ipynb +++ b/doc/LectureNotes/_build/html/_sources/week37.ipynb @@ -2,7 +2,7 @@ "cells": [ { "cell_type": "markdown", - "id": "c543a71a", + "id": "4f3beaf1", "metadata": { "editable": true }, @@ -14,7 +14,7 @@ }, { "cell_type": "markdown", - "id": "4ff6b050", + "id": "46fe6a26", "metadata": { "editable": true }, @@ -29,7 +29,7 @@ }, { "cell_type": "markdown", - "id": "a7079d3e", + "id": "f7c7fee9", "metadata": { "editable": true }, @@ -52,7 +52,7 @@ }, { "cell_type": "markdown", - "id": "db3af8cc", + "id": "29a5df4a", "metadata": { "editable": true }, @@ -69,7 +69,7 @@ }, { "cell_type": "markdown", - "id": "79d6d5be", + "id": "d1cabb8b", "metadata": { "editable": true }, @@ -79,7 +79,7 @@ }, { "cell_type": "markdown", - "id": "fa427fc3", + "id": "c1499668", "metadata": { "editable": true }, @@ -103,20 +103,21 @@ { "cell_type": "code", "execution_count": 1, - "id": "1185629e", + "id": "8f93b870", "metadata": { "collapsed": false, "editable": true }, "outputs": [], "source": [ + "import numpy as np\n", "x = 2*np.random.rand(m,1)\n", "y = 4+3*x+np.random.randn(m,1)" ] }, { "cell_type": "markdown", - "id": "42dfe293", + "id": "cafa88bc", "metadata": { "editable": true }, @@ -127,7 +128,7 @@ }, { "cell_type": "markdown", - "id": "a726cc6e", + "id": "d46f9bbc", "metadata": { "editable": true }, @@ -139,7 +140,7 @@ }, { "cell_type": "markdown", - "id": "991c8919", + "id": "c6860f97", "metadata": { "editable": true }, @@ -149,7 +150,7 @@ }, { "cell_type": "markdown", - "id": "ba895122", + "id": "5961000b", "metadata": { "editable": true }, @@ -161,7 +162,7 @@ }, { "cell_type": "markdown", - "id": "5c2b9317", + "id": "602cdea2", "metadata": { "editable": true }, @@ -175,7 +176,7 @@ }, { "cell_type": "markdown", - "id": "264eef67", + "id": "2a6ab00e", "metadata": { "editable": true }, @@ -191,17 +192,17 @@ }, { "cell_type": "markdown", - "id": "a451a9ce", + "id": "f9fb9a51", "metadata": { "editable": true }, "source": [ - "The cost/loss/risk function is given by (" + "The cost/loss/risk function is given by" ] }, { "cell_type": "markdown", - "id": "e7fa569b", + "id": "a2f3adf9", "metadata": { "editable": true }, @@ -213,7 +214,7 @@ }, { "cell_type": "markdown", - "id": "a2655bc6", + "id": "e4cffe8e", "metadata": { "editable": true }, @@ -223,7 +224,7 @@ }, { "cell_type": "markdown", - "id": "277ace2e", + "id": "3ebf7860", "metadata": { "editable": true }, @@ -235,7 +236,7 @@ }, { "cell_type": "markdown", - "id": "75bbf03c", + "id": "f8cdd774", "metadata": { "editable": true }, @@ -249,7 +250,7 @@ }, { "cell_type": "markdown", - "id": "d5bb2db2", + "id": "67a028fe", "metadata": { "editable": true }, @@ -259,7 +260,7 @@ }, { "cell_type": "markdown", - "id": "53430d47", + "id": "28b34be8", "metadata": { "editable": true }, @@ -270,7 +271,7 @@ }, { "cell_type": "markdown", - "id": "89dce328", + "id": "7068c946", "metadata": { "editable": true }, @@ -285,7 +286,7 @@ }, { "cell_type": "markdown", - "id": "542b8278", + "id": "289d7cf9", "metadata": { "editable": true }, @@ -295,37 +296,37 @@ }, { "cell_type": "markdown", - "id": "2b94baca", + "id": "3bacfa14", "metadata": { "editable": true }, "source": [ "## Simple program\n", "\n", - "We can now write a program that minimizes $C(\\theta)$ using the gradient descent method with a constant learning rate $\\gamma$ according to" + "We can now write a program that minimizes $C(\\theta)$ using the gradient descent method with a constant learning rate $\\eta$ according to" ] }, { "cell_type": "markdown", - "id": "bfc6b93f", + "id": "16c159ac", "metadata": { "editable": true }, "source": [ "$$\n", - "\\theta_{k+1} = \\theta_k - \\gamma \\nabla_\\theta C(\\theta_k), \\ k=0,1,\\cdots\n", + "\\theta_{k+1} = \\theta_k - \\eta \\nabla_\\theta C(\\theta_k), \\ k=0,1,\\cdots\n", "$$" ] }, { "cell_type": "markdown", - "id": "703de92e", + "id": "cf04e4b0", "metadata": { "editable": true }, "source": [ "We can use the expression we computed for the gradient and let use a\n", - "$\\theta_0$ be chosen randomly and let $\\gamma = 0.001$. Stop iterating\n", + "$\\theta_0$ be chosen randomly and let $\\eta = 0.001$. Stop iterating\n", "when $||\\nabla_\\theta C(\\theta_k) || \\leq \\epsilon = 10^{-8}$. **Note that the code below does not include the latter stop criterion**.\n", "\n", "And finally we can compare our solution for $\\theta$ with the analytic result given by \n", @@ -334,7 +335,7 @@ }, { "cell_type": "markdown", - "id": "f843aae1", + "id": "0d9644f3", "metadata": { "editable": true }, @@ -347,7 +348,7 @@ { "cell_type": "code", "execution_count": 2, - "id": "b76443c9", + "id": "33a70e79", "metadata": { "collapsed": false, "editable": true @@ -406,7 +407,7 @@ }, { "cell_type": "markdown", - "id": "7b75f4ec", + "id": "85b9190b", "metadata": { "editable": true }, @@ -418,7 +419,7 @@ }, { "cell_type": "markdown", - "id": "47ecdee7", + "id": "c6bf4720", "metadata": { "editable": true }, @@ -430,7 +431,7 @@ }, { "cell_type": "markdown", - "id": "ff25299b", + "id": "8f766543", "metadata": { "editable": true }, @@ -440,7 +441,7 @@ }, { "cell_type": "markdown", - "id": "21a88a84", + "id": "31c687f3", "metadata": { "editable": true }, @@ -454,7 +455,7 @@ }, { "cell_type": "markdown", - "id": "67491c15", + "id": "ff33ba55", "metadata": { "editable": true }, @@ -464,7 +465,7 @@ }, { "cell_type": "markdown", - "id": "f1ac13f1", + "id": "02d68d7d", "metadata": { "editable": true }, @@ -476,7 +477,7 @@ }, { "cell_type": "markdown", - "id": "67d83802", + "id": "aee5a999", "metadata": { "editable": true }, @@ -487,7 +488,7 @@ }, { "cell_type": "markdown", - "id": "75935d70", + "id": "d4c6e860", "metadata": { "editable": true }, @@ -502,7 +503,7 @@ }, { "cell_type": "markdown", - "id": "4707a65d", + "id": "d0b1ab36", "metadata": { "editable": true }, @@ -516,7 +517,7 @@ }, { "cell_type": "markdown", - "id": "ab16b163", + "id": "feffa94a", "metadata": { "editable": true }, @@ -527,7 +528,7 @@ { "cell_type": "code", "execution_count": 3, - "id": "fe7ee1f7", + "id": "17b7aebb", "metadata": { "collapsed": false, "editable": true @@ -588,7 +589,7 @@ }, { "cell_type": "markdown", - "id": "823fdaf7", + "id": "845d5551", "metadata": { "editable": true }, @@ -610,20 +611,33 @@ }, { "cell_type": "markdown", - "id": "21ea3f2a", + "id": "55818d27", "metadata": { "editable": true }, "source": [ - "## Improving gradient descent with momentum\n", + "## Momentum based GD\n", "\n", - "We discuss here some simple examples where we introduce what is called 'memory'about previous steps, or what is normally called momentum gradient descent. The mathematics is explained below in connection with Stochastic gradient descent." + "We discuss here some simple examples where we introduce what is called\n", + "'memory'about previous steps, or what is normally called momentum\n", + "gradient descent.\n", + "For the mathematical details, see whiteboad notes from lecture on September 8, 2025." + ] + }, + { + "cell_type": "markdown", + "id": "bdfa4705", + "metadata": { + "editable": true + }, + "source": [ + "## Improving gradient descent with momentum" ] }, { "cell_type": "code", "execution_count": 4, - "id": "5b729a7e", + "id": "b13373bb", "metadata": { "collapsed": false, "editable": true @@ -689,7 +703,7 @@ }, { "cell_type": "markdown", - "id": "209fc69c", + "id": "01f07f3c", "metadata": { "editable": true }, @@ -700,7 +714,7 @@ { "cell_type": "code", "execution_count": 5, - "id": "984c4202", + "id": "dd296802", "metadata": { "collapsed": false, "editable": true @@ -774,7 +788,7 @@ }, { "cell_type": "markdown", - "id": "8577b238", + "id": "2b995d39", "metadata": { "editable": true }, @@ -793,7 +807,7 @@ }, { "cell_type": "markdown", - "id": "c11d0874", + "id": "ec3098ab", "metadata": { "editable": true }, @@ -814,7 +828,7 @@ }, { "cell_type": "markdown", - "id": "64ad52d0", + "id": "ae3c5061", "metadata": { "editable": true }, @@ -830,7 +844,7 @@ }, { "cell_type": "markdown", - "id": "5dd4c467", + "id": "54028ccc", "metadata": { "editable": true }, @@ -844,7 +858,7 @@ }, { "cell_type": "markdown", - "id": "42904823", + "id": "17875f98", "metadata": { "editable": true }, @@ -872,7 +886,7 @@ }, { "cell_type": "markdown", - "id": "49dfaf2e", + "id": "40862db0", "metadata": { "editable": true }, @@ -904,7 +918,7 @@ }, { "cell_type": "markdown", - "id": "284b8663", + "id": "075431a2", "metadata": { "editable": true }, @@ -921,20 +935,20 @@ }, { "cell_type": "markdown", - "id": "5e8cc1b8", + "id": "65dcd718", "metadata": { "editable": true }, "source": [ "$$\n", - "C(\\mathbf{\\beta}) = \\sum_{i=1}^n c_i(\\mathbf{x}_i,\n", - "\\mathbf{\\beta}).\n", + "C(\\mathbf{\\theta}) = \\sum_{i=1}^n c_i(\\mathbf{x}_i,\n", + "\\mathbf{\\theta}).\n", "$$" ] }, { "cell_type": "markdown", - "id": "510505a0", + "id": "ccf20bf2", "metadata": { "editable": true }, @@ -947,20 +961,20 @@ }, { "cell_type": "markdown", - "id": "32fc501f", + "id": "7cf7a667", "metadata": { "editable": true }, "source": [ "$$\n", - "\\nabla_\\beta C(\\mathbf{\\beta}) = \\sum_i^n \\nabla_\\beta c_i(\\mathbf{x}_i,\n", - "\\mathbf{\\beta}).\n", + "\\nabla_\\theta C(\\mathbf{\\theta}) = \\sum_i^n \\nabla_\\theta c_i(\\mathbf{x}_i,\n", + "\\mathbf{\\theta}).\n", "$$" ] }, { "cell_type": "markdown", - "id": "9dcac7ef", + "id": "21eeb773", "metadata": { "editable": true }, @@ -974,7 +988,7 @@ }, { "cell_type": "markdown", - "id": "c9464120", + "id": "562bbf59", "metadata": { "editable": true }, @@ -996,22 +1010,22 @@ }, { "cell_type": "markdown", - "id": "59bc847c", + "id": "1b9e6fb3", "metadata": { "editable": true }, "source": [ "$$\n", - "\\nabla_{\\beta}\n", - "C(\\mathbf{\\beta}) = \\sum_{i=1}^n \\nabla_\\beta c_i(\\mathbf{x}_i,\n", - "\\mathbf{\\beta}) \\rightarrow \\sum_{i \\in B_k}^n \\nabla_\\beta\n", - "c_i(\\mathbf{x}_i, \\mathbf{\\beta}).\n", + "\\nabla_{\\theta}\n", + "C(\\mathbf{\\theta}) = \\sum_{i=1}^n \\nabla_\\theta c_i(\\mathbf{x}_i,\n", + "\\mathbf{\\theta}) \\rightarrow \\sum_{i \\in B_k}^n \\nabla_\\theta\n", + "c_i(\\mathbf{x}_i, \\mathbf{\\theta}).\n", "$$" ] }, { "cell_type": "markdown", - "id": "ba220299", + "id": "4cd95b23", "metadata": { "editable": true }, @@ -1023,20 +1037,20 @@ }, { "cell_type": "markdown", - "id": "8608ba34", + "id": "ea4324e1", "metadata": { "editable": true }, "source": [ "$$\n", - "\\beta_{j+1} = \\beta_j - \\gamma_j \\sum_{i \\in B_k}^n \\nabla_\\beta c_i(\\mathbf{x}_i,\n", - "\\mathbf{\\beta})\n", + "\\theta_{j+1} = \\theta_j - \\eta_j \\sum_{i \\in B_k}^n \\nabla_\\theta c_i(\\mathbf{x}_i,\n", + "\\mathbf{\\theta})\n", "$$" ] }, { "cell_type": "markdown", - "id": "0faa1782", + "id": "5287666c", "metadata": { "editable": true }, @@ -1050,7 +1064,7 @@ }, { "cell_type": "markdown", - "id": "97eddb68", + "id": "596acab8", "metadata": { "editable": true }, @@ -1061,7 +1075,7 @@ { "cell_type": "code", "execution_count": 6, - "id": "309da17e", + "id": "3b4d8b03", "metadata": { "collapsed": false, "editable": true @@ -1086,7 +1100,7 @@ }, { "cell_type": "markdown", - "id": "7aea4344", + "id": "f4f09885", "metadata": { "editable": true }, @@ -1102,7 +1116,7 @@ }, { "cell_type": "markdown", - "id": "52f8d549", + "id": "156e8953", "metadata": { "editable": true }, @@ -1117,20 +1131,20 @@ "that we are close to a local/global minimum. However, we could also\n", "evaluate the cost function at this point, store the result and\n", "continue the search. If the test kicks in at a later stage we can\n", - "compare the values of the cost function and keep the $\\beta$ that\n", + "compare the values of the cost function and keep the $\\theta$ that\n", "gave the lowest value." ] }, { "cell_type": "markdown", - "id": "dcd04de7", + "id": "7076d164", "metadata": { "editable": true }, "source": [ "## Slightly different approach\n", "\n", - "Another approach is to let the step length $\\gamma_j$ depend on the\n", + "Another approach is to let the step length $\\eta_j$ depend on the\n", "number of epochs in such a way that it becomes very small after a\n", "reasonable time such that we do not move at all. Such approaches are\n", "also called scaling. There are many such ways to [scale the learning\n", @@ -1143,26 +1157,26 @@ }, { "cell_type": "markdown", - "id": "4ecf0b16", + "id": "d822d1d4", "metadata": { "editable": true }, "source": [ "## Time decay rate\n", "\n", - "As an example, let $e = 0,1,2,3,\\cdots$ denote the current epoch and let $t_0, t_1 > 0$ be two fixed numbers. Furthermore, let $t = e \\cdot m + i$ where $m$ is the number of minibatches and $i=0,\\cdots,m-1$. Then the function $$\\gamma_j(t; t_0, t_1) = \\frac{t_0}{t+t_1} $$ goes to zero as the number of epochs gets large. I.e. we start with a step length $\\gamma_j (0; t_0, t_1) = t_0/t_1$ which decays in *time* $t$.\n", + "As an example, let $e = 0,1,2,3,\\cdots$ denote the current epoch and let $t_0, t_1 > 0$ be two fixed numbers. Furthermore, let $t = e \\cdot m + i$ where $m$ is the number of minibatches and $i=0,\\cdots,m-1$. Then the function $$\\eta_j(t; t_0, t_1) = \\frac{t_0}{t+t_1} $$ goes to zero as the number of epochs gets large. I.e. we start with a step length $\\eta_j (0; t_0, t_1) = t_0/t_1$ which decays in *time* $t$.\n", "\n", - "In this way we can fix the number of epochs, compute $\\beta$ and\n", + "In this way we can fix the number of epochs, compute $\\theta$ and\n", "evaluate the cost function at the end. Repeating the computation will\n", "give a different result since the scheme is random by design. Then we\n", - "pick the final $\\beta$ that gives the lowest value of the cost\n", + "pick the final $\\theta$ that gives the lowest value of the cost\n", "function." ] }, { "cell_type": "code", "execution_count": 7, - "id": "71eb89db", + "id": "c8987f85", "metadata": { "collapsed": false, "editable": true @@ -1181,23 +1195,23 @@ "t0 = 1.0\n", "t1 = 10\n", "\n", - "gamma_j = t0/t1\n", + "eta_j = t0/t1\n", "j = 0\n", "for epoch in range(1,n_epochs+1):\n", " for i in range(m):\n", " k = np.random.randint(m) #Pick the k-th minibatch at random\n", " #Compute the gradient using the data in minibatch Bk\n", - " #Compute new suggestion for beta\n", + " #Compute new suggestion for theta\n", " t = epoch*m+i\n", - " gamma_j = step_length(t,t0,t1)\n", + " eta_j = step_length(t,t0,t1)\n", " j += 1\n", "\n", - "print(\"gamma_j after %d epochs: %g\" % (n_epochs,gamma_j))" + "print(\"eta_j after %d epochs: %g\" % (n_epochs,eta_j))" ] }, { "cell_type": "markdown", - "id": "10ce3fc7", + "id": "4c72814e", "metadata": { "editable": true }, @@ -1210,7 +1224,7 @@ { "cell_type": "code", "execution_count": 8, - "id": "f2d402a8", + "id": "7b243058", "metadata": { "collapsed": false, "editable": true @@ -1287,7 +1301,7 @@ }, { "cell_type": "markdown", - "id": "ffd1e92d", + "id": "e008b37c", "metadata": { "editable": true }, @@ -1302,267 +1316,7 @@ }, { "cell_type": "markdown", - "id": "4b72d3f9", - "metadata": { - "editable": true - }, - "source": [ - "## Momentum based GD\n", - "\n", - "The stochastic gradient descent (SGD) is almost always used with a\n", - "*momentum* or inertia term that serves as a memory of the direction we\n", - "are moving in parameter space. This is typically implemented as\n", - "follows" - ] - }, - { - "cell_type": "markdown", - "id": "1b60ec02", - "metadata": { - "editable": true - }, - "source": [ - "$$\n", - "\\mathbf{v}_{t}=\\gamma \\mathbf{v}_{t-1}+\\eta_{t}\\nabla_\\theta E(\\boldsymbol{\\theta}_t) \\nonumber\n", - "$$" - ] - }, - { - "cell_type": "markdown", - "id": "e1fa24ba", - "metadata": { - "editable": true - }, - "source": [ - "\n", - "
\n", - "\n", - "$$\n", - "\\begin{equation} \n", - "\\boldsymbol{\\theta}_{t+1}= \\boldsymbol{\\theta}_t -\\mathbf{v}_{t},\n", - "\\label{_auto1} \\tag{1}\n", - "\\end{equation}\n", - "$$" - ] - }, - { - "cell_type": "markdown", - "id": "7a1fc9fc", - "metadata": { - "editable": true - }, - "source": [ - "where we have introduced a momentum parameter $\\gamma$, with\n", - "$0\\le\\gamma\\le 1$, and for brevity we dropped the explicit notation to\n", - "indicate the gradient is to be taken over a different mini-batch at\n", - "each step. We call this algorithm gradient descent with momentum\n", - "(GDM). From these equations, it is clear that $\\mathbf{v}_t$ is a\n", - "running average of recently encountered gradients and\n", - "$(1-\\gamma)^{-1}$ sets the characteristic time scale for the memory\n", - "used in the averaging procedure. Consistent with this, when\n", - "$\\gamma=0$, this just reduces down to ordinary SGD as discussed\n", - "earlier. An equivalent way of writing the updates is" - ] - }, - { - "cell_type": "markdown", - "id": "23abda5d", - "metadata": { - "editable": true - }, - "source": [ - "$$\n", - "\\Delta \\boldsymbol{\\theta}_{t+1} = \\gamma \\Delta \\boldsymbol{\\theta}_t -\\ \\eta_{t}\\nabla_\\theta E(\\boldsymbol{\\theta}_t),\n", - "$$" - ] - }, - { - "cell_type": "markdown", - "id": "0f0067a9", - "metadata": { - "editable": true - }, - "source": [ - "where we have defined $\\Delta \\boldsymbol{\\theta}_{t}= \\boldsymbol{\\theta}_t-\\boldsymbol{\\theta}_{t-1}$." - ] - }, - { - "cell_type": "markdown", - "id": "b86ec959", - "metadata": { - "editable": true - }, - "source": [ - "## More on momentum based approaches\n", - "\n", - "Let us try to get more intuition from these equations. It is helpful\n", - "to consider a simple physical analogy with a particle of mass $m$\n", - "moving in a viscous medium with drag coefficient $\\mu$ and potential\n", - "$E(\\mathbf{w})$. If we denote the particle's position by $\\mathbf{w}$,\n", - "then its motion is described by" - ] - }, - { - "cell_type": "markdown", - "id": "7685ff97", - "metadata": { - "editable": true - }, - "source": [ - "$$\n", - "m {d^2 \\mathbf{w} \\over dt^2} + \\mu {d \\mathbf{w} \\over dt }= -\\nabla_w E(\\mathbf{w}).\n", - "$$" - ] - }, - { - "cell_type": "markdown", - "id": "28203c1e", - "metadata": { - "editable": true - }, - "source": [ - "We can discretize this equation in the usual way to get" - ] - }, - { - "cell_type": "markdown", - "id": "1495b23f", - "metadata": { - "editable": true - }, - "source": [ - "$$\n", - "m { \\mathbf{w}_{t+\\Delta t}-2 \\mathbf{w}_{t} +\\mathbf{w}_{t-\\Delta t} \\over (\\Delta t)^2}+\\mu {\\mathbf{w}_{t+\\Delta t}- \\mathbf{w}_{t} \\over \\Delta t} = -\\nabla_w E(\\mathbf{w}).\n", - "$$" - ] - }, - { - "cell_type": "markdown", - "id": "15bbc6e6", - "metadata": { - "editable": true - }, - "source": [ - "Rearranging this equation, we can rewrite this as" - ] - }, - { - "cell_type": "markdown", - "id": "ffeffc05", - "metadata": { - "editable": true - }, - "source": [ - "$$\n", - "\\Delta \\mathbf{w}_{t +\\Delta t}= - { (\\Delta t)^2 \\over m +\\mu \\Delta t} \\nabla_w E(\\mathbf{w})+ {m \\over m +\\mu \\Delta t} \\Delta \\mathbf{w}_t.\n", - "$$" - ] - }, - { - "cell_type": "markdown", - "id": "166cfafc", - "metadata": { - "editable": true - }, - "source": [ - "## Momentum parameter\n", - "\n", - "Notice that this equation is identical to previous one if we identify\n", - "the position of the particle, $\\mathbf{w}$, with the parameters\n", - "$\\boldsymbol{\\theta}$. This allows us to identify the momentum\n", - "parameter and learning rate with the mass of the particle and the\n", - "viscous drag as:" - ] - }, - { - "cell_type": "markdown", - "id": "486fadf0", - "metadata": { - "editable": true - }, - "source": [ - "$$\n", - "\\gamma= {m \\over m +\\mu \\Delta t }, \\qquad \\eta = {(\\Delta t)^2 \\over m +\\mu \\Delta t}.\n", - "$$" - ] - }, - { - "cell_type": "markdown", - "id": "cdb0769d", - "metadata": { - "editable": true - }, - "source": [ - "Thus, as the name suggests, the momentum parameter is proportional to\n", - "the mass of the particle and effectively provides inertia.\n", - "Furthermore, in the large viscosity/small learning rate limit, our\n", - "memory time scales as $(1-\\gamma)^{-1} \\approx m/(\\mu \\Delta t)$.\n", - "\n", - "Why is momentum useful? SGD momentum helps the gradient descent\n", - "algorithm gain speed in directions with persistent but small gradients\n", - "even in the presence of stochasticity, while suppressing oscillations\n", - "in high-curvature directions. This becomes especially important in\n", - "situations where the landscape is shallow and flat in some directions\n", - "and narrow and steep in others. It has been argued that first-order\n", - "methods (with appropriate initial conditions) can perform comparable\n", - "to more expensive second order methods, especially in the context of\n", - "complex deep learning models.\n", - "\n", - "These beneficial properties of momentum can sometimes become even more\n", - "pronounced by using a slight modification of the classical momentum\n", - "algorithm called Nesterov Accelerated Gradient (NAG).\n", - "\n", - "In the NAG algorithm, rather than calculating the gradient at the\n", - "current parameters, $\\nabla_\\theta E(\\boldsymbol{\\theta}_t)$, one\n", - "calculates the gradient at the expected value of the parameters given\n", - "our current momentum, $\\nabla_\\theta E(\\boldsymbol{\\theta}_t +\\gamma\n", - "\\mathbf{v}_{t-1})$. This yields the NAG update rule" - ] - }, - { - "cell_type": "markdown", - "id": "e65569bb", - "metadata": { - "editable": true - }, - "source": [ - "$$\n", - "\\mathbf{v}_{t}=\\gamma \\mathbf{v}_{t-1}+\\eta_{t}\\nabla_\\theta E(\\boldsymbol{\\theta}_t +\\gamma \\mathbf{v}_{t-1}) \\nonumber\n", - "$$" - ] - }, - { - "cell_type": "markdown", - "id": "9d72d316", - "metadata": { - "editable": true - }, - "source": [ - "\n", - "
\n", - "\n", - "$$\n", - "\\begin{equation} \n", - "\\boldsymbol{\\theta}_{t+1}= \\boldsymbol{\\theta}_t -\\mathbf{v}_{t}.\n", - "\\label{_auto2} \\tag{2}\n", - "\\end{equation}\n", - "$$" - ] - }, - { - "cell_type": "markdown", - "id": "b7b9811e", - "metadata": { - "editable": true - }, - "source": [ - "One of the major advantages of NAG is that it allows for the use of a\n", - "larger learning rate than GDM for the same choice of $\\gamma$." - ] - }, - { - "cell_type": "markdown", - "id": "d8f3c523", + "id": "b921e0a8", "metadata": { "editable": true }, @@ -1593,7 +1347,7 @@ }, { "cell_type": "markdown", - "id": "22d6be74", + "id": "0ce23497", "metadata": { "editable": true }, @@ -1615,14 +1369,14 @@ }, { "cell_type": "markdown", - "id": "091ace5a", + "id": "072e6d0d", "metadata": { "editable": true }, "source": [ "## Motivation for Adaptive Step Sizes\n", "\n", - "1. Instead of a fixed global $\\eta$, use an \\textbf{adaptive learning rate} for each parameter that depends on the history of gradients.\n", + "1. Instead of a fixed global $\\eta$, use an **adaptive learning rate** for each parameter that depends on the history of gradients.\n", "\n", "2. Parameters that have large accumulated gradient magnitude should get smaller steps (they've been changing a lot), whereas parameters with small or infrequent gradients can have larger relative steps.\n", "\n", @@ -1635,457 +1389,7 @@ }, { "cell_type": "markdown", - "id": "5c05fb8b", - "metadata": { - "editable": true - }, - "source": [ - "## Derivation of the AdaGrad Algorithm\n", - "\n", - "**Accumulating Gradient History.**\n", - "\n", - "1. AdaGrad maintains a running sum of squared gradients for each parameter (coordinate)\n", - "\n", - "2. Let $g_t = \\nabla \\ell_{i_t}(x_t)$ be the gradient at step $t$ (or a subgradient for nondifferentiable cases).\n", - "\n", - "3. Initialize $h_0 = 0$ (an all-zero vector in $\\mathbb{R}^d$).\n", - "\n", - "4. At each iteration $t$, update the accumulation:" - ] - }, - { - "cell_type": "markdown", - "id": "21fb8bd0", - "metadata": { - "editable": true - }, - "source": [ - "$$\n", - "h_t \\;=\\; h_{t-1} + g_t \\circ g_t,\n", - "$$" - ] - }, - { - "cell_type": "markdown", - "id": "f71dfafa", - "metadata": { - "editable": true - }, - "source": [ - "1. Here $g_t \\circ g_t$ denotes element-wise square of the gradient vector. $h_t^{(j)} = h_{t-1}^{(j)} + (g_{t,j})^2$ for each parameter $j$.\n", - "\n", - "2. We can view $H_t = \\mathrm{diag}(h_t)$ as a diagonal matrix of past squared gradients. Initially $H_0 = 0$." - ] - }, - { - "cell_type": "markdown", - "id": "64869e2e", - "metadata": { - "editable": true - }, - "source": [ - "## AdaGrad Update Rule Derivation\n", - "\n", - "We scale the gradient by the inverse square root of the accumulated matrix $H_t$. The AdaGrad update at step $t$ is:" - ] - }, - { - "cell_type": "markdown", - "id": "c2d8d6b1", - "metadata": { - "editable": true - }, - "source": [ - "$$\n", - "x_{t+1} \\;=\\; x_t \\;-\\; \\eta\\, H_t^{-1/2}\\, g_t,\n", - "$$" - ] - }, - { - "cell_type": "markdown", - "id": "2171198a", - "metadata": { - "editable": true - }, - "source": [ - "where $H_t^{-1/2}$ is the diagonal matrix with entries $(h_{t}^{(1)})^{-1/2}, \\dots, (h_{t}^{(d)})^{-1/2}$\n", - "In coordinates, this means each parameter $j$ has an individual step size:" - ] - }, - { - "cell_type": "markdown", - "id": "353e21d3", - "metadata": { - "editable": true - }, - "source": [ - "$$\n", - "x_{t+1,j} =x_{t,j} -\\frac{\\eta}{\\sqrt{h_{t,j}}}g_{t,j}.\n", - "$$" - ] - }, - { - "cell_type": "markdown", - "id": "b59e6add", - "metadata": { - "editable": true - }, - "source": [ - "In practice we add a small constant $\\epsilon$ in the denominator for numerical stability to avoid division by zero:" - ] - }, - { - "cell_type": "markdown", - "id": "707caefe", - "metadata": { - "editable": true - }, - "source": [ - "$$\n", - "x_{t+1,j}= x_{t,j}-\\frac{\\eta}{\\sqrt{\\epsilon + h_{t,j}}}g_{t,j}.\n", - "$$" - ] - }, - { - "cell_type": "markdown", - "id": "62dafce4", - "metadata": { - "editable": true - }, - "source": [ - "Equivalently, the effective learning rate for parameter $j$ at time $t$ is $\\displaystyle \\alpha_{t,j} = \\frac{\\eta}{\\sqrt{\\epsilon + h_{t,j}}}$. This decreases over time as $h_{t,j}$ grows." - ] - }, - { - "cell_type": "markdown", - "id": "01a2ba61", - "metadata": { - "editable": true - }, - "source": [ - "## AdaGrad Properties\n", - "1. AdaGrad automatically tunes the step size for each parameter. Parameters with more **volatile or large gradients** get smaller steps, and those with **small or infrequent gradients** get relatively larger steps\n", - "\n", - "2. No manual schedule needed: The accumulation $h_t$ keeps increasing (or stays the same if gradient is zero), so step sizes $\\eta/\\sqrt{h_t}$ are non-increasing. This has a similar effect to a learning rate schedule, but individualized per coordinate.\n", - "\n", - "3. Sparse data benefit: For very sparse features, $h_{t,j}$ grows slowly, so that feature’s parameter retains a higher learning rate for longer, allowing it to make significant updates when it does get a gradient signal\n", - "\n", - "4. Convergence: In convex optimization, AdaGrad can be shown to achieve a sub-linear convergence rate (e.g. $O(1/\\sqrt{T})$ regret bound) comparable to the best fixed learning rate tuned for the problem\n", - "\n", - "It effectively reduces the need to tune $\\eta$ by hand.\n", - "1. Limitations: Because $h_t$ accumulates without bound, AdaGrad’s learning rates can become extremely small over long training, potentially slowing progress. (Later variants like RMSProp, AdaDelta, Adam address this by modifying the accumulation rule.)" - ] - }, - { - "cell_type": "markdown", - "id": "b4bc88e0", - "metadata": { - "editable": true - }, - "source": [ - "## RMSProp: Adaptive Learning Rates\n", - "Addresses AdaGrad’s diminishing learning rate issue.\n", - "Uses a decaying average of squared gradients (instead of a cumulative sum):" - ] - }, - { - "cell_type": "markdown", - "id": "4933fddb", - "metadata": { - "editable": true - }, - "source": [ - "$$\n", - "v_t = \\beta_2\\, v_{t-1} + (1-\\beta_2)\\, (\\nabla L(w_t))^2,\n", - "$$" - ] - }, - { - "cell_type": "markdown", - "id": "0dd8edf7", - "metadata": { - "editable": true - }, - "source": [ - "with $\\beta_2$ typically $0.9$ (or $0.99$).\n", - "1. Update: $w_{t+1} = w_t - \\frac{\\alpha}{\\sqrt{v_t + \\epsilon}} \\nabla L(w_t)$.\n", - "\n", - "2. Recent gradients have more weight, so $v_t$ adapts to the current landscape.\n", - "\n", - "3. Avoids AdaGrad’s “infinite memory” problem – learning rate does not continuously decay to zero.\n", - "\n", - "RMSProp was first proposed in lecture notes by Geoff Hinton, 2012 - unpublished.)" - ] - }, - { - "cell_type": "markdown", - "id": "1ce98f28", - "metadata": { - "editable": true - }, - "source": [ - "## Adam Optimizer\n", - "\n", - "Why Combine Momentum and RMSProp? Motivation for Adam: Adaptive Moment Estimation (Adam) was introduced by Kingma an Ba (2014) to combine the benefits of momentum and RMSProp.\n", - "\n", - "1. Fast convergence by smoothing gradients (accelerates in long-term gradient direction).\n", - "\n", - "2. Adaptive rates (RMSProp): Per-dimension learning rate scaling for stability (handles different feature scales, sparse gradients).\n", - "\n", - "3. Adam uses both: maintains moving averages of both first moment (gradients) and second moment (squared gradients)\n", - "\n", - "4. Additionally, includes a mechanism to correct the bias in these moving averages (crucial in early iterations)\n", - "\n", - "**Result**: Adam is robust, achieves faster convergence with less tuning, and often outperforms SGD (with momentum) in practice." - ] - }, - { - "cell_type": "markdown", - "id": "d7d646bc", - "metadata": { - "editable": true - }, - "source": [ - "## RMS prop\n", - "\n", - "In RMS prop, in addition to keeping a running average of the first\n", - "moment of the gradient, we also keep track of the second moment\n", - "denoted by $\\mathbf{s}_t=\\mathbb{E}[\\mathbf{g}_t^2]$. The update rule\n", - "for RMS prop is given by" - ] - }, - { - "cell_type": "markdown", - "id": "3a212292", - "metadata": { - "editable": true - }, - "source": [ - "\n", - "
\n", - "\n", - "$$\n", - "\\begin{equation}\n", - "\\mathbf{g}_t = \\nabla_\\theta E(\\boldsymbol{\\theta}) \n", - "\\label{_auto3} \\tag{3}\n", - "\\end{equation}\n", - "$$" - ] - }, - { - "cell_type": "markdown", - "id": "88c0ec96", - "metadata": { - "editable": true - }, - "source": [ - "$$\n", - "\\mathbf{s}_t =\\beta \\mathbf{s}_{t-1} +(1-\\beta)\\mathbf{g}_t^2 \\nonumber\n", - "$$" - ] - }, - { - "cell_type": "markdown", - "id": "9cfb794f", - "metadata": { - "editable": true - }, - "source": [ - "$$\n", - "\\boldsymbol{\\theta}_{t+1}=\\boldsymbol{\\theta}_t - \\eta_t { \\mathbf{g}_t \\over \\sqrt{\\mathbf{s}_t +\\epsilon}}, \\nonumber\n", - "$$" - ] - }, - { - "cell_type": "markdown", - "id": "aaa4936c", - "metadata": { - "editable": true - }, - "source": [ - "where $\\beta$ controls the averaging time of the second moment and is\n", - "typically taken to be about $\\beta=0.9$, $\\eta_t$ is a learning rate\n", - "typically chosen to be $10^{-3}$, and $\\epsilon\\sim 10^{-8} $ is a\n", - "small regularization constant to prevent divergences. Multiplication\n", - "and division by vectors is understood as an element-wise operation. It\n", - "is clear from this formula that the learning rate is reduced in\n", - "directions where the norm of the gradient is consistently large. This\n", - "greatly speeds up the convergence by allowing us to use a larger\n", - "learning rate for flat directions." - ] - }, - { - "cell_type": "markdown", - "id": "5493a6cd", - "metadata": { - "editable": true - }, - "source": [ - "## [ADAM optimizer](https://arxiv.org/abs/1412.6980)\n", - "\n", - "A related algorithm is the ADAM optimizer. In\n", - "[ADAM](https://arxiv.org/abs/1412.6980), we keep a running average of\n", - "both the first and second moment of the gradient and use this\n", - "information to adaptively change the learning rate for different\n", - "parameters. The method isefficient when working with large\n", - "problems involving lots data and/or parameters. It is a combination of the\n", - "gradient descent with momentum algorithm and the RMSprop algorithm\n", - "discussed above.\n", - "\n", - "In addition to keeping a running average of the first and\n", - "second moments of the gradient\n", - "(i.e. $\\mathbf{m}_t=\\mathbb{E}[\\mathbf{g}_t]$ and\n", - "$\\mathbf{s}_t=\\mathbb{E}[\\mathbf{g}^2_t]$, respectively), ADAM\n", - "performs an additional bias correction to account for the fact that we\n", - "are estimating the first two moments of the gradient using a running\n", - "average (denoted by the hats in the update rule below). The update\n", - "rule for ADAM is given by (where multiplication and division are once\n", - "again understood to be element-wise operations below)" - ] - }, - { - "cell_type": "markdown", - "id": "35795070", - "metadata": { - "editable": true - }, - "source": [ - "\n", - "
\n", - "\n", - "$$\n", - "\\begin{equation}\n", - "\\mathbf{g}_t = \\nabla_\\theta E(\\boldsymbol{\\theta}) \n", - "\\label{_auto4} \\tag{4}\n", - "\\end{equation}\n", - "$$" - ] - }, - { - "cell_type": "markdown", - "id": "1e67e6dd", - "metadata": { - "editable": true - }, - "source": [ - "$$\n", - "\\mathbf{m}_t = \\beta_1 \\mathbf{m}_{t-1} + (1-\\beta_1) \\mathbf{g}_t \\nonumber\n", - "$$" - ] - }, - { - "cell_type": "markdown", - "id": "39d582bb", - "metadata": { - "editable": true - }, - "source": [ - "$$\n", - "\\mathbf{s}_t =\\beta_2 \\mathbf{s}_{t-1} +(1-\\beta_2)\\mathbf{g}_t^2 \\nonumber\n", - "$$" - ] - }, - { - "cell_type": "markdown", - "id": "4a7e2423", - "metadata": { - "editable": true - }, - "source": [ - "$$\n", - "\\boldsymbol{\\mathbf{m}}_t={\\mathbf{m}_t \\over 1-\\beta_1^t} \\nonumber\n", - "$$" - ] - }, - { - "cell_type": "markdown", - "id": "53da77c2", - "metadata": { - "editable": true - }, - "source": [ - "$$\n", - "\\boldsymbol{\\mathbf{s}}_t ={\\mathbf{s}_t \\over1-\\beta_2^t} \\nonumber\n", - "$$" - ] - }, - { - "cell_type": "markdown", - "id": "a4c7df45", - "metadata": { - "editable": true - }, - "source": [ - "$$\n", - "\\boldsymbol{\\theta}_{t+1}=\\boldsymbol{\\theta}_t - \\eta_t { \\boldsymbol{\\mathbf{m}}_t \\over \\sqrt{\\boldsymbol{\\mathbf{s}}_t} +\\epsilon}, \\nonumber\n", - "$$" - ] - }, - { - "cell_type": "markdown", - "id": "f645a0aa", - "metadata": { - "editable": true - }, - "source": [ - "\n", - "
\n", - "\n", - "$$\n", - "\\begin{equation} \n", - "\\label{_auto5} \\tag{5}\n", - "\\end{equation}\n", - "$$" - ] - }, - { - "cell_type": "markdown", - "id": "505619bb", - "metadata": { - "editable": true - }, - "source": [ - "where $\\beta_1$ and $\\beta_2$ set the memory lifetime of the first and\n", - "second moment and are typically taken to be $0.9$ and $0.99$\n", - "respectively, and $\\eta$ and $\\epsilon$ are identical to RMSprop.\n", - "\n", - "Like in RMSprop, the effective step size of a parameter depends on the\n", - "magnitude of its gradient squared. To understand this better, let us\n", - "rewrite this expression in terms of the variance\n", - "$\\boldsymbol{\\sigma}_t^2 = \\boldsymbol{\\mathbf{s}}_t -\n", - "(\\boldsymbol{\\mathbf{m}}_t)^2$. Consider a single parameter $\\theta_t$. The\n", - "update rule for this parameter is given by" - ] - }, - { - "cell_type": "markdown", - "id": "6cb4c9a9", - "metadata": { - "editable": true - }, - "source": [ - "$$\n", - "\\Delta \\theta_{t+1}= -\\eta_t { \\boldsymbol{m}_t \\over \\sqrt{\\sigma_t^2 + m_t^2 }+\\epsilon}.\n", - "$$" - ] - }, - { - "cell_type": "markdown", - "id": "39627950", - "metadata": { - "editable": true - }, - "source": [ - "## Algorithms and codes for Adagrad, RMSprop and Adam\n", - "\n", - "The algorithms we have implemented are well described in the text by [Goodfellow, Bengio and Courville, chapter 8](https://www.deeplearningbook.org/contents/optimization.html).\n", - "\n", - "The codes which implement these algorithms are discussed after our presentation of automatic differentiation." - ] - }, - { - "cell_type": "markdown", - "id": "f240057d", + "id": "05c46307", "metadata": { "editable": true }, @@ -2101,7 +1405,193 @@ }, { "cell_type": "markdown", - "id": "428ee09d", + "id": "183480ab", + "metadata": { + "editable": true + }, + "source": [ + "## Derivation of the AdaGrad Algorithm\n", + "\n", + "**Accumulating Gradient History.**\n", + "\n", + "1. AdaGrad maintains a running sum of squared gradients for each parameter (coordinate)\n", + "\n", + "2. Let $g_t = \\nabla C_{i_t}(x_t)$ be the gradient at step $t$ (or a subgradient for nondifferentiable cases).\n", + "\n", + "3. Initialize $r_0 = 0$ (an all-zero vector in $\\mathbb{R}^d$).\n", + "\n", + "4. At each iteration $t$, update the accumulation:" + ] + }, + { + "cell_type": "markdown", + "id": "0bffad7b", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "r_t = r_{t-1} + g_t \\circ g_t,\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "289693d6", + "metadata": { + "editable": true + }, + "source": [ + "1. Here $g_t \\circ g_t$ denotes element-wise square of the gradient vector. $g_t^{(j)} = g_{t-1}^{(j)} + (g_{t,j})^2$ for each parameter $j$.\n", + "\n", + "2. We can view $H_t = \\mathrm{diag}(r_t)$ as a diagonal matrix of past squared gradients. Initially $H_0 = 0$." + ] + }, + { + "cell_type": "markdown", + "id": "98e5e657", + "metadata": { + "editable": true + }, + "source": [ + "## AdaGrad Update Rule Derivation\n", + "\n", + "We scale the gradient by the inverse square root of the accumulated matrix $H_t$. The AdaGrad update at step $t$ is:" + ] + }, + { + "cell_type": "markdown", + "id": "d2b6b9cd", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\theta_{t+1} =\\theta_t - \\eta H_t^{-1/2} g_t,\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "795888a2", + "metadata": { + "editable": true + }, + "source": [ + "where $H_t^{-1/2}$ is the diagonal matrix with entries $(r_{t}^{(1)})^{-1/2}, \\dots, (r_{t}^{(d)})^{-1/2}$\n", + "In coordinates, this means each parameter $j$ has an individual step size:" + ] + }, + { + "cell_type": "markdown", + "id": "283c4520", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\theta_{t+1,j} =\\theta_{t,j} -\\frac{\\eta}{\\sqrt{r_{t,j}}}g_{t,j}.\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "0958350b", + "metadata": { + "editable": true + }, + "source": [ + "In practice we add a small constant $\\epsilon$ in the denominator for numerical stability to avoid division by zero:" + ] + }, + { + "cell_type": "markdown", + "id": "f806f43a", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\theta_{t+1,j}= \\theta_{t,j}-\\frac{\\eta}{\\sqrt{\\epsilon + r_{t,j}}}g_{t,j}.\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "2c7d2de6", + "metadata": { + "editable": true + }, + "source": [ + "Equivalently, the effective learning rate for parameter $j$ at time $t$ is $\\displaystyle \\alpha_{t,j} = \\frac{\\eta}{\\sqrt{\\epsilon + r_{t,j}}}$. This decreases over time as $r_{t,j}$ grows." + ] + }, + { + "cell_type": "markdown", + "id": "afabce90", + "metadata": { + "editable": true + }, + "source": [ + "## AdaGrad Properties\n", + "\n", + "1. AdaGrad automatically tunes the step size for each parameter. Parameters with more *volatile or large gradients* get smaller steps, and those with *small or infrequent gradients* get relatively larger steps\n", + "\n", + "2. No manual schedule needed: The accumulation $r_t$ keeps increasing (or stays the same if gradient is zero), so step sizes $\\eta/\\sqrt{r_t}$ are non-increasing. This has a similar effect to a learning rate schedule, but individualized per coordinate.\n", + "\n", + "3. Sparse data benefit: For very sparse features, $r_{t,j}$ grows slowly, so that feature’s parameter retains a higher learning rate for longer, allowing it to make significant updates when it does get a gradient signal\n", + "\n", + "4. Convergence: In convex optimization, AdaGrad can be shown to achieve a sub-linear convergence rate comparable to the best fixed learning rate tuned for the problem\n", + "\n", + "It effectively reduces the need to tune $\\eta$ by hand.\n", + "1. Limitations: Because $r_t$ accumulates without bound, AdaGrad’s learning rates can become extremely small over long training, potentially slowing progress. (Later variants like RMSProp, AdaDelta, Adam address this by modifying the accumulation rule.)" + ] + }, + { + "cell_type": "markdown", + "id": "489bb0db", + "metadata": { + "editable": true + }, + "source": [ + "## RMSProp: Adaptive Learning Rates\n", + "\n", + "Addresses AdaGrad’s diminishing learning rate issue.\n", + "Uses a decaying average of squared gradients (instead of a cumulative sum):" + ] + }, + { + "cell_type": "markdown", + "id": "b5808edc", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "v_t = \\rho v_{t-1} + (1-\\rho)(\\nabla C(\\theta_t))^2,\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "c36f305f", + "metadata": { + "editable": true + }, + "source": [ + "with $\\rho$ typically $0.9$ (or $0.99$).\n", + "1. Update: $\\theta_{t+1} = \\theta_t - \\frac{\\eta}{\\sqrt{v_t + \\epsilon}} \\nabla C(\\theta_t)$.\n", + "\n", + "2. Recent gradients have more weight, so $v_t$ adapts to the current landscape.\n", + "\n", + "3. Avoids AdaGrad’s “infinite memory” problem – learning rate does not continuously decay to zero.\n", + "\n", + "RMSProp was first proposed in lecture notes by Geoff Hinton, 2012 - unpublished.)" + ] + }, + { + "cell_type": "markdown", + "id": "6e4679b3", "metadata": { "editable": true }, @@ -2117,7 +1607,251 @@ }, { "cell_type": "markdown", - "id": "9ce3cc3e", + "id": "d58547fa", + "metadata": { + "editable": true + }, + "source": [ + "## Adam Optimizer\n", + "\n", + "Why combine Momentum and RMSProp? Motivation for Adam: Adaptive Moment Estimation (Adam) was introduced by Kingma an Ba (2014) to combine the benefits of momentum and RMSProp.\n", + "\n", + "1. Fast convergence by smoothing gradients (accelerates in long-term gradient direction).\n", + "\n", + "2. Adaptive rates (RMSProp): Per-dimension learning rate scaling for stability (handles different feature scales, sparse gradients).\n", + "\n", + "3. Adam uses both: maintains moving averages of both first moment (gradients) and second moment (squared gradients)\n", + "\n", + "4. Additionally, includes a mechanism to correct the bias in these moving averages (crucial in early iterations)\n", + "\n", + "**Result**: Adam is robust, achieves faster convergence with less tuning, and often outperforms SGD (with momentum) in practice." + ] + }, + { + "cell_type": "markdown", + "id": "6989903c", + "metadata": { + "editable": true + }, + "source": [ + "## [ADAM optimizer](https://arxiv.org/abs/1412.6980)\n", + "\n", + "In [ADAM](https://arxiv.org/abs/1412.6980), we keep a running average of\n", + "both the first and second moment of the gradient and use this\n", + "information to adaptively change the learning rate for different\n", + "parameters. The method is efficient when working with large\n", + "problems involving lots data and/or parameters. It is a combination of the\n", + "gradient descent with momentum algorithm and the RMSprop algorithm\n", + "discussed above." + ] + }, + { + "cell_type": "markdown", + "id": "a58fbe05", + "metadata": { + "editable": true + }, + "source": [ + "## Why Combine Momentum and RMSProp?\n", + "\n", + "1. Momentum: Fast convergence by smoothing gradients (accelerates in long-term gradient direction).\n", + "\n", + "2. Adaptive rates (RMSProp): Per-dimension learning rate scaling for stability (handles different feature scales, sparse gradients).\n", + "\n", + "3. Adam uses both: maintains moving averages of both first moment (gradients) and second moment (squared gradients)\n", + "\n", + "4. Additionally, includes a mechanism to correct the bias in these moving averages (crucial in early iterations)\n", + "\n", + "Result: Adam is robust, achieves faster convergence with less tuning, and often outperforms SGD (with momentum) in practice" + ] + }, + { + "cell_type": "markdown", + "id": "98700cff", + "metadata": { + "editable": true + }, + "source": [ + "## Adam: Exponential Moving Averages (Moments)\n", + "Adam maintains two moving averages at each time step $t$ for each parameter $w$:\n", + "**First moment (mean) $m_t$.**\n", + "\n", + "The Momentum term" + ] + }, + { + "cell_type": "markdown", + "id": "e55261d4", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "m_t = \\beta_1m_{t-1} + (1-\\beta_1)\\, \\nabla C(\\theta_t),\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "64676960", + "metadata": { + "editable": true + }, + "source": [ + "**Second moment (uncentered variance) $v_t$.**\n", + "\n", + "The RMS term" + ] + }, + { + "cell_type": "markdown", + "id": "1e3007bd", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "v_t = \\beta_2v_{t-1} + (1-\\beta_2)(\\nabla C(\\theta_t))^2,\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "5df06a58", + "metadata": { + "editable": true + }, + "source": [ + "with typical $\\beta_1 = 0.9$, $\\beta_2 = 0.999$. Initialize $m_0 = 0$, $v_0 = 0$.\n", + "\n", + " These are **biased** estimators of the true first and second moment of the gradients, especially at the start (since $m_0,v_0$ are zero)" + ] + }, + { + "cell_type": "markdown", + "id": "034cb12a", + "metadata": { + "editable": true + }, + "source": [ + "## Adam: Bias Correction\n", + "To counteract initialization bias in $m_t, v_t$, Adam computes bias-corrected estimates" + ] + }, + { + "cell_type": "markdown", + "id": "429a18be", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\hat{m}_t = \\frac{m_t}{1 - \\beta_1^t}, \\qquad \\hat{v}_t = \\frac{v_t}{1 - \\beta_2^t}.\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "1672b4fd", + "metadata": { + "editable": true + }, + "source": [ + "* When $t$ is small, $1-\\beta_i^t \\approx 0$, so $\\hat{m}_t, \\hat{v}_t$ significantly larger than raw $m_t, v_t$, compensating for the initial zero bias.\n", + "\n", + "* As $t$ increases, $1-\\beta_i^t \\to 1$, and $\\hat{m}_t, \\hat{v}_t$ converge to $m_t, v_t$.\n", + "\n", + "* Bias correction is important for Adam’s stability in early iterations" + ] + }, + { + "cell_type": "markdown", + "id": "dba6d945", + "metadata": { + "editable": true + }, + "source": [ + "## Adam: Update Rule Derivation\n", + "Finally, Adam updates parameters using the bias-corrected moments:" + ] + }, + { + "cell_type": "markdown", + "id": "13db1c10", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\theta_{t+1} =\\theta_t -\\frac{\\alpha}{\\sqrt{\\hat{v}_t} + \\epsilon}\\hat{m}_t,\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "e30b02ab", + "metadata": { + "editable": true + }, + "source": [ + "where $\\epsilon$ is a small constant (e.g. $10^{-8}$) to prevent division by zero.\n", + "Breaking it down:\n", + "1. Compute gradient $\\nabla C(\\theta_t)$.\n", + "\n", + "2. Update first moment $m_t$ and second moment $v_t$ (exponential moving averages).\n", + "\n", + "3. Bias-correct: $\\hat{m}_t = m_t/(1-\\beta_1^t)$, $\\; \\hat{v}_t = v_t/(1-\\beta_2^t)$.\n", + "\n", + "4. Compute step: $\\Delta \\theta_t = \\frac{\\hat{m}_t}{\\sqrt{\\hat{v}_t} + \\epsilon}$.\n", + "\n", + "5. Update parameters: $\\theta_{t+1} = \\theta_t - \\alpha\\, \\Delta \\theta_t$.\n", + "\n", + "This is the Adam update rule as given in the original paper." + ] + }, + { + "cell_type": "markdown", + "id": "5ac8ae68", + "metadata": { + "editable": true + }, + "source": [ + "## Adam vs. AdaGrad and RMSProp\n", + "\n", + "1. AdaGrad: Uses per-coordinate scaling like Adam, but no momentum. Tends to slow down too much due to cumulative history (no forgetting)\n", + "\n", + "2. RMSProp: Uses moving average of squared gradients (like Adam’s $v_t$) to maintain adaptive learning rates, but does not include momentum or bias-correction.\n", + "\n", + "3. Adam: Effectively RMSProp + Momentum + Bias-correction\n", + "\n", + " * Momentum ($m_t$) provides acceleration and smoother convergence.\n", + "\n", + " * Adaptive $v_t$ scaling moderates the step size per dimension.\n", + "\n", + " * Bias correction (absent in AdaGrad/RMSProp) ensures robust estimates early on.\n", + "\n", + "In practice, Adam often yields faster convergence and better tuning stability than RMSProp or AdaGrad alone" + ] + }, + { + "cell_type": "markdown", + "id": "86abec40", + "metadata": { + "editable": true + }, + "source": [ + "## Adaptivity Across Dimensions\n", + "\n", + "1. Adam adapts the step size \\emph{per coordinate}: parameters with larger gradient variance get smaller effective steps, those with smaller or sparse gradients get larger steps.\n", + "\n", + "2. This per-dimension adaptivity is inherited from AdaGrad/RMSProp and helps handle ill-conditioned or sparse problems.\n", + "\n", + "3. Meanwhile, momentum (first moment) allows Adam to continue making progress even if gradients become small or noisy, by leveraging accumulated direction." + ] + }, + { + "cell_type": "markdown", + "id": "448ff439", "metadata": { "editable": true }, @@ -2133,7 +1867,21 @@ }, { "cell_type": "markdown", - "id": "193b3778", + "id": "86183049", + "metadata": { + "editable": true + }, + "source": [ + "## Algorithms and codes for Adagrad, RMSprop and Adam\n", + "\n", + "The algorithms we have implemented are well described in the text by [Goodfellow, Bengio and Courville, chapter 8](https://www.deeplearningbook.org/contents/optimization.html).\n", + "\n", + "The codes which implement these algorithms are discussed below here." + ] + }, + { + "cell_type": "markdown", + "id": "78a7d51b", "metadata": { "editable": true }, @@ -2151,7 +1899,7 @@ }, { "cell_type": "markdown", - "id": "8d7187f5", + "id": "0b0e62b6", "metadata": { "editable": true }, @@ -2172,7 +1920,7 @@ { "cell_type": "code", "execution_count": 9, - "id": "68d35b76", + "id": "d19005f0", "metadata": { "collapsed": false, "editable": true @@ -2186,8 +1934,8 @@ "import matplotlib.pyplot as plt\n", "from autograd import grad\n", "\n", - "def CostOLS(beta):\n", - " return (1.0/n)*np.sum((y-X @ beta)**2)\n", + "def CostOLS(theta):\n", + " return (1.0/n)*np.sum((y-X @ theta)**2)\n", "\n", "n = 100\n", "x = 2*np.random.rand(n,1)\n", @@ -2232,7 +1980,7 @@ }, { "cell_type": "markdown", - "id": "f39b471c", + "id": "4f7fa973", "metadata": { "editable": true }, @@ -2243,7 +1991,7 @@ { "cell_type": "code", "execution_count": 10, - "id": "265059ef", + "id": "34cb0de4", "metadata": { "collapsed": false, "editable": true @@ -2257,8 +2005,8 @@ "import matplotlib.pyplot as plt\n", "from autograd import grad\n", "\n", - "def CostOLS(beta):\n", - " return (1.0/n)*np.sum((y-X @ beta)**2)\n", + "def CostOLS(theta):\n", + " return (1.0/n)*np.sum((y-X @ theta)**2)\n", "\n", "n = 100\n", "x = 2*np.random.rand(n,1)\n", @@ -2307,7 +2055,7 @@ }, { "cell_type": "markdown", - "id": "18183fb0", + "id": "4cb35cee", "metadata": { "editable": true }, @@ -2322,7 +2070,7 @@ { "cell_type": "code", "execution_count": 11, - "id": "4d9d93b3", + "id": "74934aab", "metadata": { "collapsed": false, "editable": true @@ -2406,7 +2154,7 @@ }, { "cell_type": "markdown", - "id": "1652a471", + "id": "d8ee3d35", "metadata": { "editable": true }, @@ -2417,7 +2165,7 @@ { "cell_type": "code", "execution_count": 12, - "id": "6092712c", + "id": "63184f2b", "metadata": { "collapsed": false, "editable": true @@ -2495,7 +2243,7 @@ }, { "cell_type": "markdown", - "id": "4d2f230d", + "id": "c44ce795", "metadata": { "editable": true }, @@ -2508,7 +2256,7 @@ { "cell_type": "code", "execution_count": 13, - "id": "b46cedcc", + "id": "74e118d6", "metadata": { "collapsed": false, "editable": true @@ -2521,8 +2269,8 @@ "import autograd.numpy as np\n", "from autograd import grad\n", "\n", - "def CostOLS(beta):\n", - " return (1.0/n)*np.sum((y-X @ beta)**2)\n", + "def CostOLS(theta):\n", + " return (1.0/n)*np.sum((y-X @ theta)**2)\n", "\n", "n = 100\n", "x = 2*np.random.rand(n,1)\n", @@ -2530,29 +2278,29 @@ "\n", "X = np.c_[np.ones((n,1)), x, x*x]\n", "XT_X = X.T @ X\n", - "beta_linreg = np.linalg.pinv(XT_X) @ (X.T @ y)\n", + "theta_linreg = np.linalg.pinv(XT_X) @ (X.T @ y)\n", "print(\"Own inversion\")\n", - "print(beta_linreg)\n", + "print(theta_linreg)\n", "# Hessian matrix\n", "H = (2.0/n)* XT_X\n", - "# Note that here the Hessian does not depend on the parameters beta\n", + "# Note that here the Hessian does not depend on the parameters theta\n", "invH = np.linalg.pinv(H)\n", - "beta = np.random.randn(3,1)\n", + "theta = np.random.randn(3,1)\n", "Niterations = 5\n", "# define the gradient\n", "training_gradient = grad(CostOLS)\n", "\n", "for iter in range(Niterations):\n", - " gradients = training_gradient(beta)\n", - " beta -= invH @ gradients\n", + " gradients = training_gradient(theta)\n", + " theta -= invH @ gradients\n", " print(iter,gradients[0],gradients[1])\n", - "print(\"beta from own Newton code\")\n", - "print(beta)" + "print(\"theta from own Newton code\")\n", + "print(theta)" ] }, { "cell_type": "markdown", - "id": "f048d5c7", + "id": "3789172e", "metadata": { "editable": true }, @@ -2563,7 +2311,7 @@ { "cell_type": "code", "execution_count": 14, - "id": "d8bdd180", + "id": "5152d211", "metadata": { "collapsed": false, "editable": true @@ -2622,7 +2370,7 @@ }, { "cell_type": "markdown", - "id": "145ce394", + "id": "34e87757", "metadata": { "editable": true }, @@ -2632,7 +2380,7 @@ }, { "cell_type": "markdown", - "id": "ec0ee110", + "id": "713154cb", "metadata": { "editable": true }, @@ -2643,7 +2391,7 @@ { "cell_type": "code", "execution_count": 15, - "id": "b6e8e3e4", + "id": "3e863305", "metadata": { "collapsed": false, "editable": true @@ -2708,7 +2456,7 @@ }, { "cell_type": "markdown", - "id": "79d8c0ae", + "id": "a5fa4647", "metadata": { "editable": true }, @@ -2719,7 +2467,7 @@ { "cell_type": "code", "execution_count": 16, - "id": "c16ec5ac", + "id": "0b318139", "metadata": { "collapsed": false, "editable": true @@ -2760,9 +2508,9 @@ "\n", "# Value for learning rate\n", "eta = 0.01\n", - "# Value for parameters beta1 and beta2, see https://arxiv.org/abs/1412.6980\n", - "beta1 = 0.9\n", - "beta2 = 0.999\n", + "# Value for parameters theta1 and theta2, see https://arxiv.org/abs/1412.6980\n", + "theta1 = 0.9\n", + "theta2 = 0.999\n", "# Including AdaGrad parameter to avoid possible division by zero\n", "delta = 1e-7\n", "iter = 0\n", @@ -2776,10 +2524,10 @@ " yi = y[random_index:random_index+M]\n", " gradients = (1.0/M)*training_gradient(yi, xi, theta)\n", " # Computing moments first\n", - " first_moment = beta1*first_moment + (1-beta1)*gradients\n", - " second_moment = beta2*second_moment+(1-beta2)*gradients*gradients\n", - " first_term = first_moment/(1.0-beta1**iter)\n", - " second_term = second_moment/(1.0-beta2**iter)\n", + " first_moment = theta1*first_moment + (1-theta1)*gradients\n", + " second_moment = theta2*second_moment+(1-theta2)*gradients*gradients\n", + " first_term = first_moment/(1.0-theta1**iter)\n", + " second_term = second_moment/(1.0-theta2**iter)\n", "\t# Scaling with rho the new and the previous results\n", " update = eta*first_term/(np.sqrt(second_term)+delta)\n", " theta -= update\n", @@ -2789,7 +2537,7 @@ }, { "cell_type": "markdown", - "id": "7c581f8b", + "id": "967dd1d1", "metadata": { "editable": true }, @@ -2801,12 +2549,12 @@ "2. Work on project 1\n", "\n", "\n", - " * For more discussions of Ridge regression and calculation of averages, [Wessel van Wieringen's](https://arxiv.org/abs/1509.09169) article is highly recommended." + "For more discussions of Ridge regression and calculation of averages, [Wessel van Wieringen's](https://arxiv.org/abs/1509.09169) article is highly recommended." ] }, { "cell_type": "markdown", - "id": "d8a82e01", + "id": "ad137d49", "metadata": { "editable": true }, @@ -2834,7 +2582,7 @@ { "cell_type": "code", "execution_count": 17, - "id": "3fa174a9", + "id": "8a6c2cb4", "metadata": { "collapsed": false, "editable": true @@ -2854,7 +2602,7 @@ }, { "cell_type": "markdown", - "id": "b02de7a6", + "id": "76bf3c81", "metadata": { "editable": true }, @@ -2870,7 +2618,7 @@ }, { "cell_type": "markdown", - "id": "cd3f85ef", + "id": "818c6d58", "metadata": { "editable": true }, @@ -2890,7 +2638,7 @@ }, { "cell_type": "markdown", - "id": "1e0198a0", + "id": "58481e8f", "metadata": { "editable": true }, @@ -2917,7 +2665,7 @@ }, { "cell_type": "markdown", - "id": "04da94ca", + "id": "4e94a32c", "metadata": { "editable": true }, @@ -2930,7 +2678,7 @@ }, { "cell_type": "markdown", - "id": "aa1ad257", + "id": "3a93bb3e", "metadata": { "editable": true }, @@ -2942,7 +2690,7 @@ }, { "cell_type": "markdown", - "id": "6587016d", + "id": "c603fb31", "metadata": { "editable": true }, @@ -2957,7 +2705,7 @@ { "cell_type": "code", "execution_count": 18, - "id": "2e0350b4", + "id": "363ae494", "metadata": { "collapsed": false, "editable": true @@ -2984,7 +2732,7 @@ }, { "cell_type": "markdown", - "id": "8179b037", + "id": "429b71ce", "metadata": { "editable": true }, @@ -2998,7 +2746,7 @@ }, { "cell_type": "markdown", - "id": "3ebed257", + "id": "ec3c7657", "metadata": { "editable": true }, @@ -3010,7 +2758,7 @@ }, { "cell_type": "markdown", - "id": "51b239a4", + "id": "c6c2fd1d", "metadata": { "editable": true }, @@ -3027,7 +2775,7 @@ }, { "cell_type": "markdown", - "id": "295790f3", + "id": "1141934f", "metadata": { "editable": true }, @@ -3039,7 +2787,7 @@ }, { "cell_type": "markdown", - "id": "b68338d6", + "id": "efc84979", "metadata": { "editable": true }, @@ -3049,7 +2797,7 @@ }, { "cell_type": "markdown", - "id": "c015cb3b", + "id": "6be29872", "metadata": { "editable": true }, @@ -3061,7 +2809,7 @@ }, { "cell_type": "markdown", - "id": "5e4917fa", + "id": "8f128adc", "metadata": { "editable": true }, @@ -3071,7 +2819,7 @@ }, { "cell_type": "markdown", - "id": "150fedf5", + "id": "967b009a", "metadata": { "editable": true }, @@ -3083,7 +2831,7 @@ }, { "cell_type": "markdown", - "id": "e6d782b1", + "id": "181e0b18", "metadata": { "editable": true }, @@ -3094,7 +2842,7 @@ }, { "cell_type": "markdown", - "id": "a4065dd3", + "id": "6fe29ace", "metadata": { "editable": true }, @@ -3106,7 +2854,7 @@ }, { "cell_type": "markdown", - "id": "06324d79", + "id": "795a350c", "metadata": { "editable": true }, @@ -3116,7 +2864,7 @@ }, { "cell_type": "markdown", - "id": "9e3e6b35", + "id": "9a0ebe42", "metadata": { "editable": true }, @@ -3128,7 +2876,7 @@ }, { "cell_type": "markdown", - "id": "cd9c3754", + "id": "800ad4c7", "metadata": { "editable": true }, @@ -3138,7 +2886,7 @@ }, { "cell_type": "markdown", - "id": "8361829b", + "id": "4ed44194", "metadata": { "editable": true }, @@ -3150,7 +2898,7 @@ }, { "cell_type": "markdown", - "id": "8f5f46fd", + "id": "f6fc455e", "metadata": { "editable": true }, @@ -3160,7 +2908,7 @@ }, { "cell_type": "markdown", - "id": "cf210a6e", + "id": "1d434e5d", "metadata": { "editable": true }, @@ -3172,7 +2920,7 @@ }, { "cell_type": "markdown", - "id": "d3af09fc", + "id": "011d7385", "metadata": { "editable": true }, @@ -3182,7 +2930,7 @@ }, { "cell_type": "markdown", - "id": "ec91f8c1", + "id": "9bb92755", "metadata": { "editable": true }, @@ -3194,7 +2942,7 @@ }, { "cell_type": "markdown", - "id": "0c9f35e6", + "id": "7287a085", "metadata": { "editable": true }, @@ -3204,7 +2952,7 @@ }, { "cell_type": "markdown", - "id": "de80fa4b", + "id": "1ebd1481", "metadata": { "editable": true }, @@ -3216,7 +2964,7 @@ }, { "cell_type": "markdown", - "id": "34c99b80", + "id": "01da7453", "metadata": { "editable": true }, @@ -3226,7 +2974,7 @@ }, { "cell_type": "markdown", - "id": "758ade4d", + "id": "2719429f", "metadata": { "editable": true }, @@ -3238,7 +2986,7 @@ }, { "cell_type": "markdown", - "id": "ff0f4725", + "id": "1b7a1119", "metadata": { "editable": true }, @@ -3248,7 +2996,7 @@ }, { "cell_type": "markdown", - "id": "997b829b", + "id": "5455e569", "metadata": { "editable": true }, @@ -3260,7 +3008,7 @@ }, { "cell_type": "markdown", - "id": "621e7d85", + "id": "82719ade", "metadata": { "editable": true }, @@ -3272,7 +3020,7 @@ }, { "cell_type": "markdown", - "id": "5530e459", + "id": "d87ce00b", "metadata": { "editable": true }, @@ -3284,7 +3032,7 @@ }, { "cell_type": "markdown", - "id": "fd9bace8", + "id": "aad7612a", "metadata": { "editable": true }, @@ -3294,7 +3042,7 @@ }, { "cell_type": "markdown", - "id": "d949be75", + "id": "3c7ca158", "metadata": { "editable": true }, @@ -3306,7 +3054,7 @@ }, { "cell_type": "markdown", - "id": "4ad52617", + "id": "b5c113ba", "metadata": { "editable": true }, @@ -3319,7 +3067,7 @@ }, { "cell_type": "markdown", - "id": "41cabc72", + "id": "aad866d0", "metadata": { "editable": true }, @@ -3331,7 +3079,7 @@ }, { "cell_type": "markdown", - "id": "1fd416a6", + "id": "a4e7459d", "metadata": { "editable": true }, @@ -3345,7 +3093,7 @@ { "cell_type": "code", "execution_count": 19, - "id": "f13cacb4", + "id": "2af50ed1", "metadata": { "collapsed": false, "editable": true @@ -3442,7 +3190,7 @@ }, { "cell_type": "markdown", - "id": "ff4b28ed", + "id": "e468f434", "metadata": { "editable": true }, @@ -3463,7 +3211,7 @@ }, { "cell_type": "markdown", - "id": "03b4dc3c", + "id": "0daaeb51", "metadata": { "editable": true }, @@ -3475,7 +3223,7 @@ }, { "cell_type": "markdown", - "id": "6cff2148", + "id": "29f1ae2a", "metadata": { "editable": true }, @@ -3485,7 +3233,7 @@ }, { "cell_type": "markdown", - "id": "51d823c4", + "id": "133f65de", "metadata": { "editable": true }, @@ -3497,7 +3245,7 @@ }, { "cell_type": "markdown", - "id": "5ab270f9", + "id": "b5f4350c", "metadata": { "editable": true }, @@ -3507,7 +3255,7 @@ }, { "cell_type": "markdown", - "id": "2b2e1492", + "id": "1f6fc172", "metadata": { "editable": true }, @@ -3519,7 +3267,7 @@ }, { "cell_type": "markdown", - "id": "063be947", + "id": "c5afa388", "metadata": { "editable": true }, @@ -3537,7 +3285,7 @@ { "cell_type": "code", "execution_count": 20, - "id": "5e841a10", + "id": "5718b4ec", "metadata": { "collapsed": false, "editable": true @@ -3613,7 +3361,7 @@ }, { "cell_type": "markdown", - "id": "40e54083", + "id": "5e7c8319", "metadata": { "editable": true }, @@ -3627,7 +3375,7 @@ { "cell_type": "code", "execution_count": 21, - "id": "edd7dd42", + "id": "3b7f797a", "metadata": { "collapsed": false, "editable": true @@ -3716,7 +3464,7 @@ }, { "cell_type": "markdown", - "id": "f9cc046b", + "id": "5e9da20b", "metadata": { "editable": true }, diff --git a/doc/LectureNotes/_build/html/searchindex.js b/doc/LectureNotes/_build/html/searchindex.js index 58a74b5a1..d558009df 100644 --- a/doc/LectureNotes/_build/html/searchindex.js +++ b/doc/LectureNotes/_build/html/searchindex.js @@ -1 +1 @@ -Search.setIndex({"alltitles": {"1a)": [[18, "a"]], "3a)": [[18, "id1"]], "3b)": [[18, "b"]], "4a)": [[18, "id2"]], "4b)": [[18, "id3"]], "A Classification Tree": [[9, "a-classification-tree"]], "A Frequentist approach to data analysis": [[0, "a-frequentist-approach-to-data-analysis"], [27, "a-frequentist-approach-to-data-analysis"]], "A better approach": [[8, "a-better-approach"]], "A first summary": [[27, "a-first-summary"]], "A quick Reminder on Lagrangian Multipliers": [[8, "a-quick-reminder-on-lagrangian-multipliers"]], "A simple example": [[4, "a-simple-example"]], "A soft classifier": [[8, "a-soft-classifier"]], "A top-down perspective on Neural networks": [[1, "a-top-down-perspective-on-neural-networks"]], "ADAM algorithm, taken from Goodfellow et al": [[30, "adam-algorithm-taken-from-goodfellow-et-al"]], "ADAM optimizer": [[13, "adam-optimizer"], [30, "id2"]], "Accuracy": [[30, "accuracy"]], "Activation functions": [[12, "activation-functions"]], "AdaGrad Properties": [[30, "adagrad-properties"]], "AdaGrad Update Rule Derivation": [[30, "adagrad-update-rule-derivation"]], "AdaGrad algorithm, taken from Goodfellow et al": [[30, "adagrad-algorithm-taken-from-goodfellow-et-al"]], "Adam Optimizer": [[30, "adam-optimizer"]], "Adaptive boosting: AdaBoost, Basic Algorithm": [[10, "adaptive-boosting-adaboost-basic-algorithm"]], "Adding error analysis and training set up": [[27, "adding-error-analysis-and-training-set-up"], [28, "adding-error-analysis-and-training-set-up"]], "Adjust hyperparameters": [[1, "adjust-hyperparameters"]], "Algorithms and codes for Adagrad, RMSprop and Adam": [[30, "algorithms-and-codes-for-adagrad-rmsprop-and-adam"]], "Algorithms for Setting up Decision Trees": [[9, "algorithms-for-setting-up-decision-trees"]], "An Overview of Ensemble Methods": [[10, "an-overview-of-ensemble-methods"]], "An extrapolation example": [[4, "an-extrapolation-example"]], "An optimization/minimization problem": [[27, "an-optimization-minimization-problem"]], "And finally \\boldsymbol{X}\\boldsymbol{X}^T": [[28, "and-finally-boldsymbol-x-boldsymbol-x-t"]], "And finally ADAM": [[30, "and-finally-adam"]], "And what about using neural networks?": [[27, "and-what-about-using-neural-networks"]], "Another Example, now with a polynomial fit": [[29, "another-example-now-with-a-polynomial-fit"]], "Another example, the moons again": [[9, "another-example-the-moons-again"]], "Applied Data Analysis and Machine Learning": [[20, null]], "Autocorrelation function": [[24, "autocorrelation-function"]], "Automatic differentiation": [[13, "automatic-differentiation"]], "Back to Ridge and LASSO Regression": [[28, "back-to-ridge-and-lasso-regression"], [29, "back-to-ridge-and-lasso-regression"]], "Back to the Cancer Data": [[11, "back-to-the-cancer-data"]], "Background literature": [[22, "background-literature"]], "Bagging": [[10, "bagging"]], "Bagging Examples": [[10, "bagging-examples"]], "Basic Matrix Features": [[21, "basic-matrix-features"]], "Basic ideas of the Principal Component Analysis (PCA)": [[11, null]], "Basic math of the SVD": [[5, "basic-math-of-the-svd"], [28, "basic-math-of-the-svd"], [29, "basic-math-of-the-svd"]], "Basics": [[7, "basics"]], "Basics of a tree": [[9, "basics-of-a-tree"]], "Batch Normalization": [[1, "batch-normalization"]], "Batches and mini-batches": [[30, "batches-and-mini-batches"]], "Bayes\u2019 Theorem and Ridge and Lasso Regression": [[5, "bayes-theorem-and-ridge-and-lasso-regression"]], "Boosting, a Bird\u2019s Eye View": [[10, "boosting-a-bird-s-eye-view"]], "Bootstrap": [[6, "bootstrap"]], "Bringing it together, first back propagation equation": [[12, "bringing-it-together-first-back-propagation-equation"]], "Building a Feed Forward Neural Network": [[1, null]], "Building a tree, regression": [[9, "building-a-tree-regression"]], "Building neural networks in Tensorflow and Keras": [[1, "building-neural-networks-in-tensorflow-and-keras"]], "But none of these can compete with Newton\u2019s method": [[30, "but-none-of-these-can-compete-with-newton-s-method"]], "CNNs in more detail, building convolutional neural networks in Tensorflow and Keras": [[3, "cnns-in-more-detail-building-convolutional-neural-networks-in-tensorflow-and-keras"]], "Cancer Data again now with Decision Trees and other Methods": [[9, "cancer-data-again-now-with-decision-trees-and-other-methods"]], "Challenge: Choosing a Fixed Learning Rate": [[30, "challenge-choosing-a-fixed-learning-rate"]], "Choose cost function and optimizer": [[1, "choose-cost-function-and-optimizer"]], "Classical PCA Theorem": [[11, "classical-pca-theorem"]], "Clustering and Unsupervised Learning": [[14, null]], "Code for SVD and Inversion of Matrices": [[5, "code-for-svd-and-inversion-of-matrices"]], "Code with a Number of Minibatches which varies": [[30, "code-with-a-number-of-minibatches-which-varies"]], "Codes and Approaches": [[14, "codes-and-approaches"]], "Codes for the SVD": [[5, "codes-for-the-svd"], [28, "codes-for-the-svd"], [29, "codes-for-the-svd"]], "Coding Setup and Linear Regression": [[15, "coding-setup-and-linear-regression"]], "Collect and pre-process data": [[1, "collect-and-pre-process-data"]], "Communication channels": [[27, "communication-channels"]], "Compare Bagging on Trees with Random Forests": [[10, "compare-bagging-on-trees-with-random-forests"]], "Comparing with a numerical scheme": [[2, "comparing-with-a-numerical-scheme"]], "Comparison with OLS": [[29, "comparison-with-ols"]], "Computation of gradients": [[30, "computation-of-gradients"]], "Computing the Gini index": [[9, "computing-the-gini-index"]], "Conditions on convex functions": [[29, "conditions-on-convex-functions"]], "Conjugate gradient method": [[13, "conjugate-gradient-method"]], "Convergence rates": [[30, "convergence-rates"]], "Convex function": [[29, "convex-function"]], "Convex functions": [[13, "convex-functions"], [29, "convex-functions"]], "Convolution Examples: Polynomial multiplication": [[3, "convolution-examples-polynomial-multiplication"]], "Convolution Examples: Principle of Superposition and Periodic Forces (Fourier Transforms)": [[3, "convolution-examples-principle-of-superposition-and-periodic-forces-fourier-transforms"]], "Convolutional Neural Network": [[12, "convolutional-neural-network"]], "Convolutional Neural Networks": [[3, null]], "Correlation Function and Design/Feature Matrix": [[28, "correlation-function-and-design-feature-matrix"]], "Correlation Matrix": [[11, "correlation-matrix"], [28, "correlation-matrix"]], "Correlation Matrix with Pandas": [[28, "correlation-matrix-with-pandas"]], "Course Format": [[27, "course-format"]], "Course setting": [[23, null]], "Covariance Matrix Examples": [[28, "covariance-matrix-examples"]], "Covariance and Correlation Matrix": [[28, "covariance-and-correlation-matrix"]], "Cross-validation": [[6, "cross-validation"]], "Deadlines for projects (tentative)": [[27, "deadlines-for-projects-tentative"]], "Decision trees, overarching aims": [[9, null]], "Deep learning methods": [[27, "deep-learning-methods"]], "Define model and architecture": [[1, "define-model-and-architecture"]], "Defining the cost function": [[1, "defining-the-cost-function"]], "Deliverables": [[15, "deliverables"], [16, "deliverables"]], "Derivation of the AdaGrad Algorithm": [[30, "derivation-of-the-adagrad-algorithm"]], "Derivatives and the chain rule": [[12, "derivatives-and-the-chain-rule"]], "Derivatives, example 1": [[28, "derivatives-example-1"]], "Deriving OLS from a probability distribution": [[5, "deriving-ols-from-a-probability-distribution"]], "Deriving and Implementing Ordinary Least Squares": [[16, "deriving-and-implementing-ordinary-least-squares"]], "Deriving and Implementing Ridge Regression": [[17, "deriving-and-implementing-ridge-regression"]], "Deriving the Lasso Regression Equations": [[28, "deriving-the-lasso-regression-equations"], [29, "deriving-the-lasso-regression-equations"], [29, "id6"]], "Deriving the Ridge Regression Equations": [[28, "deriving-the-ridge-regression-equations"], [29, "deriving-the-ridge-regression-equations"], [29, "id3"]], "Deriving the back propagation code for a multilayer perceptron model": [[12, "deriving-the-back-propagation-code-for-a-multilayer-perceptron-model"]], "Developing a code for doing neural networks with back propagation": [[1, "developing-a-code-for-doing-neural-networks-with-back-propagation"]], "Diagonalize the sample covariance matrix to obtain the principal components": [[11, "diagonalize-the-sample-covariance-matrix-to-obtain-the-principal-components"]], "Different kernels and Mercer\u2019s theorem": [[8, "different-kernels-and-mercer-s-theorem"]], "Disadvantages": [[9, "disadvantages"]], "Discriminative Modeling": [[27, "discriminative-modeling"]], "Domains and probabilities": [[24, "domains-and-probabilities"]], "Dropout": [[1, "dropout"]], "Economy-size SVD": [[28, "economy-size-svd"], [29, "economy-size-svd"]], "Elements of Probability Theory and Statistical Data Analysis": [[24, null]], "Ensemble Methods: From a Single Tree to Many Trees and Extreme Boosting, Meet the Jungle of Methods": [[10, null]], "Entropy and the ID3 algorithm": [[9, "entropy-and-the-id3-algorithm"]], "Essential elements of ML": [[27, "essential-elements-of-ml"]], "Evaluate model performance on test data": [[1, "evaluate-model-performance-on-test-data"]], "Example 2": [[28, "example-2"]], "Example 3": [[28, "example-3"]], "Example 4": [[28, "example-4"]], "Example Matrix": [[28, "example-matrix"], [29, "example-matrix"]], "Example of discriminative modeling, taken from Generative Deep Learning by David Foster": [[27, "example-of-discriminative-modeling-taken-from-generative-deep-learning-by-david-foster"]], "Example of generative modeling, taken from Generative Deep Learning by David Foster": [[27, "example-of-generative-modeling-taken-from-generative-deep-learning-by-david-foster"]], "Example of own Standard scaling": [[28, "example-of-own-standard-scaling"]], "Example relevant for the exercises": [[28, "example-relevant-for-the-exercises"]], "Example: Exponential decay": [[2, "example-exponential-decay"]], "Example: Population growth": [[2, "example-population-growth"]], "Example: The diffusion equation": [[2, "example-the-diffusion-equation"]], "Example: binary classification problem": [[1, "example-binary-classification-problem"]], "Examples": [[27, "examples"]], "Examples of likelihood functions used in logistic regression and neural networks": [[7, "examples-of-likelihood-functions-used-in-logistic-regression-and-neural-networks"]], "Exercise 1 - Choice of model and degrees of freedom": [[17, "exercise-1-choice-of-model-and-degrees-of-freedom"]], "Exercise 1 - Finding the derivative of Matrix-Vector expressions": [[16, "exercise-1-finding-the-derivative-of-matrix-vector-expressions"]], "Exercise 1 - Github Setup": [[15, "exercise-1-github-setup"]], "Exercise 1, scale your data": [[18, "exercise-1-scale-your-data"]], "Exercise 1: Expectation values for ordinary least squares expressions": [[19, "exercise-1-expectation-values-for-ordinary-least-squares-expressions"]], "Exercise 1: Setting up various Python environments": [[0, "exercise-1-setting-up-various-python-environments"]], "Exercise 2 - Deriving the expression for OLS": [[16, "exercise-2-deriving-the-expression-for-ols"]], "Exercise 2 - Deriving the expression for Ridge Regression": [[17, "exercise-2-deriving-the-expression-for-ridge-regression"]], "Exercise 2 - Setting up a Github repository": [[15, "exercise-2-setting-up-a-github-repository"]], "Exercise 2, calculate the gradients": [[18, "exercise-2-calculate-the-gradients"]], "Exercise 2: Expectation values for Ridge regression": [[19, "exercise-2-expectation-values-for-ridge-regression"]], "Exercise 2: making your own data and exploring scikit-learn": [[0, "exercise-2-making-your-own-data-and-exploring-scikit-learn"]], "Exercise 3 - Creating feature matrix and implementing OLS using the analytical expression": [[16, "exercise-3-creating-feature-matrix-and-implementing-ols-using-the-analytical-expression"]], "Exercise 3 - Fitting an OLS model to data": [[15, "exercise-3-fitting-an-ols-model-to-data"]], "Exercise 3 - Scaling data": [[17, "exercise-3-scaling-data"]], "Exercise 3 - Setting up a Python virtual environment": [[15, "exercise-3-setting-up-a-python-virtual-environment"]], "Exercise 3, using the analytical formulae for OLS and Ridge regression to find the optimal paramters \\boldsymbol{\\theta}": [[18, "exercise-3-using-the-analytical-formulae-for-ols-and-ridge-regression-to-find-the-optimal-paramters-boldsymbol-theta"]], "Exercise 3: Deriving the expression for the Bias-Variance Trade-off": [[19, "exercise-3-deriving-the-expression-for-the-bias-variance-trade-off"]], "Exercise 3: Normalizing our data": [[0, "exercise-3-normalizing-our-data"]], "Exercise 4 - Fitting a polynomial": [[16, "exercise-4-fitting-a-polynomial"]], "Exercise 4 - Implementing Ridge Regression": [[17, "exercise-4-implementing-ridge-regression"]], "Exercise 4 - Testing multiple hyperparameters": [[17, "exercise-4-testing-multiple-hyperparameters"]], "Exercise 4 - The train-test split": [[15, "exercise-4-the-train-test-split"]], "Exercise 4, Implementing the simplest form for gradient descent": [[18, "exercise-4-implementing-the-simplest-form-for-gradient-descent"]], "Exercise 4: Adding Ridge Regression": [[0, "exercise-4-adding-ridge-regression"]], "Exercise 4: Computing the Bias and Variance": [[19, "exercise-4-computing-the-bias-and-variance"]], "Exercise 5 - Comparing your code with sklearn": [[16, "exercise-5-comparing-your-code-with-sklearn"]], "Exercise 5, Ridge regression and a new Synthetic Dataset": [[18, "exercise-5-ridge-regression-and-a-new-synthetic-dataset"]], "Exercise 5: Analytical exercises": [[0, "exercise-5-analytical-exercises"]], "Exercise: Cross-validation as resampling techniques, adding more complexity": [[6, "exercise-cross-validation-as-resampling-techniques-adding-more-complexity"]], "Exercise: Analysis of real data": [[6, "exercise-analysis-of-real-data"]], "Exercise: Bias-variance trade-off and resampling techniques": [[6, "exercise-bias-variance-trade-off-and-resampling-techniques"]], "Exercise: Lasso Regression on the Franke function with resampling": [[6, "exercise-lasso-regression-on-the-franke-function-with-resampling"]], "Exercise: Ordinary Least Square (OLS) on the Franke function": [[6, "exercise-ordinary-least-square-ols-on-the-franke-function"]], "Exercise: Ridge Regression on the Franke function with resampling": [[6, "exercise-ridge-regression-on-the-franke-function-with-resampling"]], "Exercises": [[0, "exercises"]], "Exercises and Projects": [[6, "exercises-and-projects"]], "Exercises week 34": [[15, null]], "Exercises week 35": [[16, null]], "Exercises week 36": [[17, null]], "Exercises week 37": [[18, null], [19, null]], "Expectation values": [[24, "expectation-values"]], "Extending to more than one variable": [[29, "extending-to-more-than-one-variable"]], "Extremely useful tools, strongly recommended": [[27, "extremely-useful-tools-strongly-recommended"]], "Feed-forward neural networks": [[12, "feed-forward-neural-networks"]], "Feed-forward pass": [[1, "feed-forward-pass"]], "Final back propagating equation": [[12, "final-back-propagating-equation"]], "Fine-tuning neural network hyperparameters": [[1, "fine-tuning-neural-network-hyperparameters"]], "Fitting an Equation of State for Dense Nuclear Matter": [[0, "fitting-an-equation-of-state-for-dense-nuclear-matter"]], "Fixing the singularity": [[28, "fixing-the-singularity"], [29, "fixing-the-singularity"]], "Format for electronic delivery of report and programs": [[22, "format-for-electronic-delivery-of-report-and-programs"]], "Frequently used scaling functions": [[28, "frequently-used-scaling-functions"], [30, "frequently-used-scaling-functions"]], "From OLS to Ridge and Lasso": [[29, "from-ols-to-ridge-and-lasso"]], "From one to many layers, the universal approximation theorem": [[12, "from-one-to-many-layers-the-universal-approximation-theorem"]], "Functionality in Scikit-Learn": [[28, "functionality-in-scikit-learn"], [30, "functionality-in-scikit-learn"]], "Further Dimensionality Remarks": [[3, "further-dimensionality-remarks"]], "Further properties (important for our analyses later)": [[5, "further-properties-important-for-our-analyses-later"], [28, "further-properties-important-for-our-analyses-later"], [29, "further-properties-important-for-our-analyses-later"]], "Gaussian Elimination": [[21, "gaussian-elimination"]], "General Features": [[9, "general-features"]], "General linear models and linear algebra": [[27, "general-linear-models-and-linear-algebra"]], "Generalizing the fitting procedure as a linear algebra problem": [[27, "generalizing-the-fitting-procedure-as-a-linear-algebra-problem"], [27, "id1"]], "Generative Adversarial Networks": [[4, "generative-adversarial-networks"]], "Generative Models": [[4, "generative-models"]], "Generative Versus Discriminative Modeling": [[27, "generative-versus-discriminative-modeling"]], "Geometric Interpretation and link with Singular Value Decomposition": [[11, "geometric-interpretation-and-link-with-singular-value-decomposition"]], "Gradient Boosting, Classification Example": [[10, "gradient-boosting-classification-example"]], "Gradient Boosting, Examples of Regression": [[10, "gradient-boosting-examples-of-regression"]], "Gradient Clipping": [[1, "gradient-clipping"]], "Gradient Descent Example": [[29, "id1"], [30, "id1"]], "Gradient boosting: Basics with Steepest Descent/Functional Gradient Descent": [[10, "gradient-boosting-basics-with-steepest-descent-functional-gradient-descent"]], "Gradient descent": [[2, "gradient-descent"]], "Gradient descent and Ridge": [[29, "gradient-descent-and-ridge"], [30, "gradient-descent-and-ridge"]], "Gradient descent and revisiting Ordinary Least Squares from last week": [[30, "gradient-descent-and-revisiting-ordinary-least-squares-from-last-week"]], "Gradient descent example": [[29, "gradient-descent-example"], [30, "gradient-descent-example"]], "Grading": [[25, "grading"], [25, "id2"], [27, "grading"]], "How to take derivatives of Matrix-Vector expressions": [[16, "how-to-take-derivatives-of-matrix-vector-expressions"]], "Hyperplanes and all that": [[8, "hyperplanes-and-all-that"]], "Important Matrix and vector handling packages": [[21, "important-matrix-and-vector-handling-packages"]], "Important technicalities: More on Rescaling data": [[28, "important-technicalities-more-on-rescaling-data"]], "Improving gradient descent with momentum": [[30, "improving-gradient-descent-with-momentum"]], "Improving performance": [[1, "improving-performance"]], "In summary": [[25, "in-summary"]], "Including Stochastic Gradient Descent with Autograd": [[13, "including-stochastic-gradient-descent-with-autograd"], [30, "including-stochastic-gradient-descent-with-autograd"]], "Incremental PCA": [[11, "incremental-pca"]], "Installing R, C++, cython or Julia": [[27, "installing-r-c-cython-or-julia"]], "Installing R, C++, cython, Numba etc": [[27, "installing-r-c-cython-numba-etc"]], "Instructor information": [[25, "instructor-information"]], "Interpretations and optimizing our parameters": [[27, "interpretations-and-optimizing-our-parameters"], [27, "id2"], [27, "id3"], [28, "interpretations-and-optimizing-our-parameters"], [28, "id1"], [28, "id2"]], "Interpreting the Ridge results": [[28, "interpreting-the-ridge-results"], [29, "interpreting-the-ridge-results"], [29, "id4"]], "Introducing JAX": [[13, "introducing-jax"]], "Introducing the Covariance and Correlation functions": [[11, "introducing-the-covariance-and-correlation-functions"], [28, "introducing-the-covariance-and-correlation-functions"]], "Introduction": [[0, "introduction"], [6, "introduction"], [20, "introduction"], [21, "introduction"]], "Introduction to numerical projects": [[22, "introduction-to-numerical-projects"]], "Iterative Fitting, Classification and AdaBoost": [[10, "iterative-fitting-classification-and-adaboost"]], "Iterative Fitting, Regression and Squared-error Cost Function": [[10, "iterative-fitting-regression-and-squared-error-cost-function"]], "Kernel PCA": [[11, "kernel-pca"]], "Kernels and non-linearity": [[8, "kernels-and-non-linearity"]], "LU Decomposition, the inverse of a matrix": [[21, "lu-decomposition-the-inverse-of-a-matrix"]], "Lasso Regression": [[29, "lasso-regression"]], "Lasso case": [[29, "lasso-case"]], "Layers": [[1, "layers"]], "Layers used to build CNNs": [[3, "layers-used-to-build-cnns"]], "Learning goals": [[15, "learning-goals"], [16, "learning-goals"], [17, "learning-goals"], [18, "learning-goals"], [19, "learning-goals"]], "Learning outcomes": [[20, "learning-outcomes"], [27, "learning-outcomes"]], "Lectures and ComputerLab": [[27, "lectures-and-computerlab"]], "Limitations of supervised learning with deep networks": [[1, "limitations-of-supervised-learning-with-deep-networks"]], "Linear Algebra, Handling of Arrays and more Python Features": [[21, null]], "Linear Regression": [[0, null]], "Linear Regression Problems": [[28, "linear-regression-problems"], [29, "linear-regression-problems"]], "Linear Regression and the SVD": [[29, "linear-regression-and-the-svd"]], "Linear Regression, basic elements": [[0, "linear-regression-basic-elements"]], "Linking Bayes\u2019 Theorem with Ridge and Lasso Regression": [[5, "linking-bayes-theorem-with-ridge-and-lasso-regression"]], "Linking the regression analysis with a statistical interpretation": [[5, "linking-the-regression-analysis-with-a-statistical-interpretation"]], "Linking with the SVD": [[5, "linking-with-the-svd"], [28, "linking-with-the-svd"]], "Links to relevant courses at the University of Oslo": [[26, "links-to-relevant-courses-at-the-university-of-oslo"]], "Logistic Regression": [[7, null], [7, "id1"]], "MNIST and GANs": [[4, "mnist-and-gans"]], "Machine Learning": [[27, "machine-learning"]], "Machine learning": [[20, "machine-learning"]], "Main textbooks": [[27, "main-textbooks"]], "Making a tree": [[9, "making-a-tree"]], "Making your own Bootstrap: Changing the Level of the Decision Tree": [[10, "making-your-own-bootstrap-changing-the-level-of-the-decision-tree"]], "Making your own test-train splitting": [[28, "making-your-own-test-train-splitting"]], "Material for exercises week 35": [[28, "material-for-exercises-week-35"]], "Material for lab sessions sessions Tuesday and Wednesday": [[29, "material-for-lab-sessions-sessions-tuesday-and-wednesday"]], "Material for lecture Monday September 2": [[29, "material-for-lecture-monday-september-2"]], "Material for lecture Monday September 8": [[30, "material-for-lecture-monday-september-8"]], "Material for the lab sessions": [[30, "material-for-the-lab-sessions"]], "Mathematical Interpretation of Ordinary Least Squares": [[5, "mathematical-interpretation-of-ordinary-least-squares"], [28, "mathematical-interpretation-of-ordinary-least-squares"], [29, "mathematical-interpretation-of-ordinary-least-squares"]], "Mathematical optimization of convex functions": [[8, "mathematical-optimization-of-convex-functions"]], "Mathematics of CNNs": [[3, "mathematics-of-cnns"]], "Mathematics of the SVD and implications": [[5, "mathematics-of-the-svd-and-implications"], [28, "mathematics-of-the-svd-and-implications"], [29, "mathematics-of-the-svd-and-implications"]], "Matrices in Python": [[27, "matrices-in-python"]], "Matrix multiplication": [[1, "matrix-multiplication"]], "Matrix-vector notation and activation": [[12, "matrix-vector-notation-and-activation"]], "Meet the covariance!": [[24, "meet-the-covariance"]], "Meet the Covariance Matrix": [[5, "meet-the-covariance-matrix"], [28, "meet-the-covariance-matrix"]], "Meet the Hessian Matrix": [[28, "meet-the-hessian-matrix"]], "Meet the Pandas": [[27, "meet-the-pandas"]], "Min-Max Scaling": [[28, "min-max-scaling"]], "Momentum based GD": [[13, "momentum-based-gd"], [30, "momentum-based-gd"]], "Momentum parameter": [[30, "momentum-parameter"]], "More complicated Example: The Ising model": [[6, "more-complicated-example-the-ising-model"]], "More interpretations": [[28, "more-interpretations"], [29, "more-interpretations"], [29, "id5"]], "More on Dimensionalities": [[3, "more-on-dimensionalities"]], "More on Rescaling data": [[6, "more-on-rescaling-data"]], "More on Steepest descent": [[29, "more-on-steepest-descent"]], "More on convex functions": [[29, "more-on-convex-functions"]], "More on momentum based approaches": [[30, "more-on-momentum-based-approaches"]], "More preprocessing": [[28, "more-preprocessing"], [30, "more-preprocessing"]], "Motivation for Adaptive Step Sizes": [[30, "motivation-for-adaptive-step-sizes"]], "Multilayer perceptrons": [[12, "multilayer-perceptrons"]], "Network requirements": [[2, "network-requirements"]], "Neural Networks vs CNNs": [[3, "neural-networks-vs-cnns"]], "Neural networks": [[12, null]], "Note about SVD Calculations": [[28, "note-about-svd-calculations"], [29, "note-about-svd-calculations"]], "Note on Scikit-Learn": [[29, "note-on-scikit-learn"]], "Numerical experiments and the covariance, central limit theorem": [[24, "numerical-experiments-and-the-covariance-central-limit-theorem"]], "Numpy and arrays": [[21, "numpy-and-arrays"], [27, "numpy-and-arrays"]], "Numpy examples and Important Matrix and vector handling packages": [[27, "numpy-examples-and-important-matrix-and-vector-handling-packages"]], "Optimization and gradient descent, the central part of any Machine Learning algortithm": [[29, "optimization-and-gradient-descent-the-central-part-of-any-machine-learning-algortithm"]], "Optimization, the central part of any Machine Learning algortithm": [[13, null]], "Optimizing our parameters": [[27, "optimizing-our-parameters"]], "Optimizing our parameters, more details": [[27, "optimizing-our-parameters-more-details"]], "Optimizing the cost function": [[1, "optimizing-the-cost-function"]], "Organizing our data": [[0, "organizing-our-data"], [27, "organizing-our-data"]], "Other Matrix and Vector Operations": [[21, "other-matrix-and-vector-operations"]], "Other Types of Recurrent Neural Networks": [[4, "other-types-of-recurrent-neural-networks"]], "Other courses on Data science and Machine Learning at UiO": [[27, "other-courses-on-data-science-and-machine-learning-at-uio"]], "Other courses on Data science and Machine Learning at UiO, contn": [[27, "other-courses-on-data-science-and-machine-learning-at-uio-contn"]], "Other popular texts": [[27, "other-popular-texts"]], "Other techniques": [[11, "other-techniques"]], "Other types of networks": [[12, "other-types-of-networks"]], "Other ways of visualizing the trees": [[9, "other-ways-of-visualizing-the-trees"]], "Our model for the nuclear binding energies": [[27, "our-model-for-the-nuclear-binding-energies"]], "Overview of first week": [[27, "overview-of-first-week"]], "Overview video on Stochastic Gradient Descent (SGD)": [[30, "overview-video-on-stochastic-gradient-descent-sgd"]], "Own code for Ordinary Least Squares": [[27, "own-code-for-ordinary-least-squares"], [28, "own-code-for-ordinary-least-squares"]], "PCA and scikit-learn": [[11, "pca-and-scikit-learn"]], "Pandas AI": [[27, "pandas-ai"]], "Part a : Ordinary Least Square (OLS) for the Runge function": [[22, "part-a-ordinary-least-square-ols-for-the-runge-function"]], "Part b: Adding Ridge regression for the Runge function": [[22, "part-b-adding-ridge-regression-for-the-runge-function"]], "Part c: Writing your own gradient descent code": [[22, "part-c-writing-your-own-gradient-descent-code"]], "Part d: Including momentum and more advanced ways to update the learning the rate": [[22, "part-d-including-momentum-and-more-advanced-ways-to-update-the-learning-the-rate"]], "Part e: Writing our own code for Lasso regression": [[22, "part-e-writing-our-own-code-for-lasso-regression"]], "Part f: Stochastic gradient descent": [[22, "part-f-stochastic-gradient-descent"]], "Part g: Bias-variance trade-off and resampling techniques": [[22, "part-g-bias-variance-trade-off-and-resampling-techniques"]], "Part h): Cross-validation as resampling techniques, adding more complexity": [[22, "part-h-cross-validation-as-resampling-techniques-adding-more-complexity"]], "Partial Differential Equations": [[2, "partial-differential-equations"]], "Plans for week 35": [[28, "plans-for-week-35"]], "Plans for week 36": [[29, "plans-for-week-36"]], "Plans for week 37, lecture Monday": [[30, "plans-for-week-37-lecture-monday"]], "Practical tips": [[13, "practical-tips"], [30, "practical-tips"]], "Practicalities": [[25, "practicalities"], [25, "id1"]], "Preamble: Note on writing reports, using reference material, AI and other tools": [[22, "preamble-note-on-writing-reports-using-reference-material-ai-and-other-tools"]], "Predicting New Points With A Trained Recurrent Neural Network": [[4, "predicting-new-points-with-a-trained-recurrent-neural-network"]], "Preprocessing our data": [[28, "preprocessing-our-data"]], "Prerequisites": [[27, "prerequisites"]], "Prerequisites and background": [[20, "prerequisites-and-background"]], "Prerequisites: Collect and pre-process data": [[3, "prerequisites-collect-and-pre-process-data"]], "Probability Distribution Functions": [[24, "probability-distribution-functions"]], "Program example for gradient descent with Ridge Regression": [[29, "program-example-for-gradient-descent-with-ridge-regression"], [30, "program-example-for-gradient-descent-with-ridge-regression"]], "Program for stochastic gradient": [[13, "program-for-stochastic-gradient"]], "Project 1 on Machine Learning, deadline October 6 (midnight), 2025": [[22, null]], "Properties of PDFs": [[24, "properties-of-pdfs"]], "Pros and cons": [[30, "pros-and-cons"]], "Pros and cons of trees, pros": [[9, "pros-and-cons-of-trees-pros"]], "Python installers": [[20, "python-installers"], [27, "python-installers"]], "RMS prop": [[13, "rms-prop"], [30, "rms-prop"]], "RMSProp algorithm, taken from Goodfellow et al": [[30, "rmsprop-algorithm-taken-from-goodfellow-et-al"]], "RMSProp: Adaptive Learning Rates": [[30, "rmsprop-adaptive-learning-rates"]], "RMSprop for adaptive learning rate with Stochastic Gradient Descent": [[30, "rmsprop-for-adaptive-learning-rate-with-stochastic-gradient-descent"]], "Random Numbers": [[24, "random-numbers"]], "Random forests": [[10, "random-forests"]], "Randomized PCA": [[11, "randomized-pca"]], "Reading material": [[27, "reading-material"]], "Reading recommendations:": [[28, "reading-recommendations"]], "Reading suggestions week 34": [[27, "reading-suggestions-week-34"]], "Readings and Videos:": [[30, "readings-and-videos"]], "Recurrent neural networks": [[12, "recurrent-neural-networks"]], "Recurrent neural networks: Overarching view": [[4, null]], "Reducing the number of degrees of freedom, overarching view": [[0, "reducing-the-number-of-degrees-of-freedom-overarching-view"], [28, "reducing-the-number-of-degrees-of-freedom-overarching-view"]], "Reformulating the problem": [[2, "reformulating-the-problem"]], "Regression Case": [[10, "regression-case"]], "Regression analysis and resampling methods": [[22, "regression-analysis-and-resampling-methods"]], "Regression analysis, overarching aims": [[27, "regression-analysis-overarching-aims"]], "Regression analysis, overarching aims II": [[27, "regression-analysis-overarching-aims-ii"]], "Regularization": [[1, "regularization"]], "Reminder from last week": [[28, "reminder-from-last-week"]], "Reminder on Newton-Raphson\u2019s method": [[29, "reminder-on-newton-raphson-s-method"]], "Reminder on Statistics": [[6, "reminder-on-statistics"]], "Reminder on different scaling methods": [[30, "reminder-on-different-scaling-methods"]], "Replace or not": [[13, "replace-or-not"], [30, "replace-or-not"]], "Required Technologies": [[20, "required-technologies"]], "Resampling Methods": [[6, null]], "Resampling and the Bias-Variance Trade-off": [[19, "resampling-and-the-bias-variance-trade-off"]], "Resampling methods": [[6, "id1"]], "Residual Error": [[28, "residual-error"], [29, "residual-error"]], "Resources on differential equations and deep learning": [[2, "resources-on-differential-equations-and-deep-learning"]], "Revisiting Ordinary Least Squares": [[29, "revisiting-ordinary-least-squares"]], "Revisiting our Linear Regression Solvers": [[13, "revisiting-our-linear-regression-solvers"]], "Rewriting the Covariance and/or Correlation Matrix": [[28, "rewriting-the-covariance-and-or-correlation-matrix"]], "Rewriting the fitting procedure as a linear algebra problem": [[27, "rewriting-the-fitting-procedure-as-a-linear-algebra-problem"]], "Rewriting the fitting procedure as a linear algebra problem, more details": [[27, "rewriting-the-fitting-procedure-as-a-linear-algebra-problem-more-details"]], "Ridge Regression": [[29, "ridge-regression"]], "Ridge and LASSO Regression": [[28, "ridge-and-lasso-regression"], [29, "ridge-and-lasso-regression"], [29, "id2"]], "Ridge and Lasso Regression": [[5, null], [5, "id1"]], "SGD example": [[30, "sgd-example"]], "SVD analysis": [[29, "svd-analysis"]], "Same code but now with momentum gradient descent": [[13, "same-code-but-now-with-momentum-gradient-descent"], [30, "same-code-but-now-with-momentum-gradient-descent"], [30, "id3"], [30, "id4"]], "Schedule first week": [[27, "schedule-first-week"]], "Schematic Regression Procedure": [[9, "schematic-regression-procedure"]], "Second moment of the gradient": [[30, "second-moment-of-the-gradient"]], "September 15-19": [[19, "september-15-19"]], "Setting up the Back propagation algorithm": [[12, "setting-up-the-back-propagation-algorithm"]], "Setting up the Matrix to be inverted": [[28, "setting-up-the-matrix-to-be-inverted"], [29, "setting-up-the-matrix-to-be-inverted"]], "Setting up the network using Autograd; The full program": [[2, "setting-up-the-network-using-autograd-the-full-program"]], "Similar (second order function now) problem but now with AdaGrad": [[13, "similar-second-order-function-now-problem-but-now-with-adagrad"], [30, "similar-second-order-function-now-problem-but-now-with-adagrad"]], "Simple Python Code to read in Data and perform Classification": [[9, "simple-python-code-to-read-in-data-and-perform-classification"]], "Simple case": [[28, "simple-case"], [29, "simple-case"]], "Simple code for solving the above problem": [[29, "simple-code-for-solving-the-above-problem"]], "Simple example code": [[30, "simple-example-code"]], "Simple example to illustrate Ordinary Least Squares, Ridge and Lasso Regression": [[29, "simple-example-to-illustrate-ordinary-least-squares-ridge-and-lasso-regression"]], "Simple geometric interpretation": [[29, "simple-geometric-interpretation"]], "Simple linear regression model using scikit-learn": [[0, "simple-linear-regression-model-using-scikit-learn"], [27, "simple-linear-regression-model-using-scikit-learn"]], "Simple one-dimensional second-order polynomial": [[18, "simple-one-dimensional-second-order-polynomial"]], "Simple program": [[29, "simple-program"], [30, "simple-program"]], "Slightly different approach": [[30, "slightly-different-approach"]], "Sneaking in automatic differentiation using Autograd": [[30, "sneaking-in-automatic-differentiation-using-autograd"]], "Software and needed installations": [[22, "software-and-needed-installations"], [27, "software-and-needed-installations"]], "Solving Differential Equations with Deep Learning": [[2, null]], "Solving the one dimensional Poisson equation": [[2, "solving-the-one-dimensional-poisson-equation"]], "Solving the wave equation with Neural Networks": [[2, "solving-the-wave-equation-with-neural-networks"]], "Some famous Matrices": [[21, "some-famous-matrices"]], "Some simple problems": [[13, "some-simple-problems"], [29, "some-simple-problems"]], "Some useful matrix and vector expressions": [[28, "some-useful-matrix-and-vector-expressions"]], "Splitting our Data in Training and Test data": [[0, "splitting-our-data-in-training-and-test-data"], [28, "splitting-our-data-in-training-and-test-data"]], "Standard steepest descent": [[13, "standard-steepest-descent"]], "Statistical analysis and optimization of data": [[20, "statistical-analysis-and-optimization-of-data"], [27, "statistical-analysis-and-optimization-of-data"]], "Steepest descent": [[13, "steepest-descent"], [29, "steepest-descent"]], "Stochastic Gradient Descent": [[30, "stochastic-gradient-descent"]], "Stochastic Gradient Descent (SGD)": [[13, "stochastic-gradient-descent-sgd"], [30, "stochastic-gradient-descent-sgd"]], "Stochastic variables and the main concepts, the discrete case": [[24, "stochastic-variables-and-the-main-concepts-the-discrete-case"]], "Support Vector Machines, overarching aims": [[8, null]], "Systematic reduction": [[3, "systematic-reduction"]], "Teachers": [[27, "teachers"]], "Teachers and Grading": [[25, null]], "Teaching Assistants Fall semester 2023": [[25, "teaching-assistants-fall-semester-2023"]], "Tentative deadllines for projects": [[25, "tentative-deadllines-for-projects"]], "Testing the Means Squared Error as function of Complexity": [[0, "testing-the-means-squared-error-as-function-of-complexity"], [28, "testing-the-means-squared-error-as-function-of-complexity"]], "Textbooks": [[26, null]], "The Algorithm before theorem": [[11, "the-algorithm-before-theorem"]], "The Breast Cancer Data, now with Keras": [[1, "the-breast-cancer-data-now-with-keras"]], "The CART algorithm for Classification": [[9, "the-cart-algorithm-for-classification"]], "The CART algorithm for Regression": [[9, "the-cart-algorithm-for-regression"]], "The CIFAR01 data set": [[3, "the-cifar01-data-set"]], "The Hessian matrix": [[29, "the-hessian-matrix"], [30, "the-hessian-matrix"]], "The Hessian matrix for Ridge Regression": [[29, "the-hessian-matrix-for-ridge-regression"], [30, "the-hessian-matrix-for-ridge-regression"]], "The Jacobian": [[28, "the-jacobian"]], "The MNIST dataset again": [[3, "the-mnist-dataset-again"]], "The OLS case": [[29, "the-ols-case"]], "The RELU function family": [[1, "the-relu-function-family"]], "The Ridge case": [[29, "the-ridge-case"]], "The SVD, a Fantastic Algorithm": [[28, "the-svd-a-fantastic-algorithm"], [29, "the-svd-a-fantastic-algorithm"]], "The Softmax function": [[1, "the-softmax-function"]], "The \\chi^2 function": [[0, "the-chi-2-function"], [27, "the-chi-2-function"], [27, "id4"], [27, "id5"], [27, "id6"], [27, "id7"], [27, "id8"]], "The bias-variance tradeoff": [[6, "the-bias-variance-tradeoff"]], "The code for solving the ODE": [[2, "the-code-for-solving-the-ode"]], "The complete code with a simple data set": [[28, "the-complete-code-with-a-simple-data-set"]], "The cost/loss function": [[28, "the-cost-loss-function"]], "The course has two central parts": [[20, "the-course-has-two-central-parts"]], "The derivative of the cost/loss function": [[29, "the-derivative-of-the-cost-loss-function"], [30, "the-derivative-of-the-cost-loss-function"]], "The equations": [[29, "the-equations"]], "The equations for ordinary least squares": [[28, "the-equations-for-ordinary-least-squares"]], "The first Case": [[29, "the-first-case"]], "The gradient step": [[30, "the-gradient-step"]], "The ideal": [[29, "the-ideal"]], "The logistic function": [[7, "the-logistic-function"]], "The mean squared error and its derivative": [[28, "the-mean-squared-error-and-its-derivative"]], "The moons example": [[8, "the-moons-example"]], "The multilayer perceptron (MLP)": [[12, "the-multilayer-perceptron-mlp"]], "The network with one input layer, specified number of hidden layers, and one output layer": [[2, "the-network-with-one-input-layer-specified-number-of-hidden-layers-and-one-output-layer"]], "The plethora of machine learning algorithms/methods": [[27, "the-plethora-of-machine-learning-algorithms-methods"]], "The sensitiveness of the gradient descent": [[29, "the-sensitiveness-of-the-gradient-descent"]], "The singular value decomposition": [[5, "the-singular-value-decomposition"], [28, "the-singular-value-decomposition"], [29, "the-singular-value-decomposition"]], "The two-dimensional case": [[8, "the-two-dimensional-case"]], "Time decay rate": [[30, "time-decay-rate"]], "To our real data: nuclear binding energies. Brief reminder on masses and binding energies": [[27, "to-our-real-data-nuclear-binding-energies-brief-reminder-on-masses-and-binding-energies"]], "Topics covered in this course: Statistical analysis and optimization of data": [[27, "topics-covered-in-this-course-statistical-analysis-and-optimization-of-data"]], "Towards the PCA theorem": [[11, "towards-the-pca-theorem"]], "Train and test datasets": [[1, "train-and-test-datasets"]], "Two-dimensional Objects": [[3, "two-dimensional-objects"]], "Type of problem": [[2, "type-of-problem"]], "Types of Machine Learning": [[27, "types-of-machine-learning"]], "Useful Python libraries": [[20, "useful-python-libraries"], [27, "useful-python-libraries"]], "Using Autograd": [[13, "using-autograd"]], "Using forward Euler to solve the ODE": [[2, "using-forward-euler-to-solve-the-ode"]], "Using gradient descent methods, limitations": [[13, "using-gradient-descent-methods-limitations"], [29, "using-gradient-descent-methods-limitations"], [30, "using-gradient-descent-methods-limitations"]], "Visualization": [[1, "visualization"], [1, "id1"]], "Visualizing the Tree, Classification": [[9, "visualizing-the-tree-classification"]], "Week 34: Introduction to the course, Logistics and Practicalities": [[27, null]], "Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression": [[28, null]], "Week 36: Linear Regression and Gradient descent": [[29, null]], "Week 37: Statistical interpretations and Resampling Methods": [[30, null]], "What Is Generative Modeling?": [[27, "what-is-generative-modeling"]], "What does it mean?": [[28, "what-does-it-mean"], [29, "what-does-it-mean"]], "What is Machine Learning?": [[0, "what-is-machine-learning"]], "What is a good model?": [[0, "what-is-a-good-model"], [27, "what-is-a-good-model"]], "What is a good model? Can we define it?": [[27, "what-is-a-good-model-can-we-define-it"]], "When do we stop?": [[30, "when-do-we-stop"]], "Which activation function should I use?": [[1, "which-activation-function-should-i-use"]], "Why Linear Regression (aka Ordinary Least Squares and family)": [[27, "why-linear-regression-aka-ordinary-least-squares-and-family"]], "Wisconsin Cancer Data": [[7, "wisconsin-cancer-data"]], "With Lasso Regression": [[29, "with-lasso-regression"]], "Writing Our First Generative Adversarial Network": [[4, "writing-our-first-generative-adversarial-network"]], "Writing our own PCA code": [[11, "writing-our-own-pca-code"]], "Writing the Cost Function": [[29, "writing-the-cost-function"]], "XGBoost: Extreme Gradient Boosting": [[10, "xgboost-extreme-gradient-boosting"]], "Yet another Example": [[29, "yet-another-example"]], "a) Expression for Ridge regression": [[17, "a-expression-for-ridge-regression"]], "scikit-learn implementation": [[1, "scikit-learn-implementation"]]}, "docnames": ["chapter1", "chapter10", "chapter11", "chapter12", "chapter13", "chapter2", "chapter3", "chapter4", "chapter5", "chapter6", "chapter7", "chapter8", "chapter9", "chapteroptimization", "clustering", "exercisesweek34", "exercisesweek35", "exercisesweek36", "exercisesweek37", "exercisesweek38", "intro", "linalg", "project1", "schedule", "statistics", "teachers", "textbooks", "week34", "week35", "week36", "week37"], "envversion": {"sphinx": 62, "sphinx.domains.c": 3, "sphinx.domains.changeset": 1, "sphinx.domains.citation": 1, "sphinx.domains.cpp": 9, "sphinx.domains.index": 1, "sphinx.domains.javascript": 3, "sphinx.domains.math": 2, "sphinx.domains.python": 4, "sphinx.domains.rst": 2, "sphinx.domains.std": 2, "sphinx.ext.intersphinx": 1}, "filenames": ["chapter1.ipynb", "chapter10.ipynb", "chapter11.ipynb", "chapter12.ipynb", "chapter13.ipynb", "chapter2.ipynb", "chapter3.ipynb", "chapter4.ipynb", "chapter5.ipynb", "chapter6.ipynb", "chapter7.ipynb", "chapter8.ipynb", "chapter9.ipynb", "chapteroptimization.ipynb", "clustering.ipynb", "exercisesweek34.ipynb", "exercisesweek35.ipynb", "exercisesweek36.ipynb", "exercisesweek37.ipynb", "exercisesweek38.ipynb", "intro.md", "linalg.ipynb", "project1.ipynb", "schedule.md", "statistics.ipynb", "teachers.md", "textbooks.md", "week34.ipynb", "week35.ipynb", "week36.ipynb", "week37.ipynb"], "indexentries": {}, "objects": {}, "objnames": {}, "objtypes": {}, "terms": {"": [0, 1, 2, 3, 4, 5, 6, 7, 9, 11, 12, 13, 15, 16, 17, 19, 20, 21, 22, 24, 25, 27, 28], "0": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 21, 22, 24, 25, 27, 28, 29, 30], "00": [0, 1, 5, 11, 27, 28], "000": [1, 3], "00000000e": [], "001": [2, 8, 13, 29, 30], "004": 5, "004113634617443131": 28, "004113634617443139": 28, "00411363461744314": 28, "004113634617443147": 28, "00727646693": [0, 27], "0086649156": [0, 27], "01": [0, 1, 2, 5, 9, 11, 13, 17, 26, 27, 28, 30], "0110": 24, "01719003e": [], "02": [0, 4, 7, 12, 27], "02334824": [], "02857": 4, "02f": 6, "03077640549": 4, "03097597e": [], "031": 5, "04": 11, "0458": 9, "05": [4, 6], "062292565": 4, "062435": [], "06730814": [], "07": [], "0713": [0, 27], "07285": 3, "08": 24, "08078025e": [], "08336233266": 4, "08376632": 28, "083766322923899": 28, "0837663229239043": 28, "0917": 9, "0n": [0, 27], "0x113e21950": 17, "1": [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 21, 23, 24, 25, 26, 27, 29, 30], "10": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 16, 17, 18, 19, 21, 23, 24, 25, 27, 28, 29, 30], "100": [0, 1, 2, 3, 4, 5, 6, 8, 9, 10, 11, 13, 14, 15, 16, 17, 18, 19, 21, 24, 25, 27, 28, 29, 30], "1000": [0, 1, 2, 4, 5, 8, 11, 13, 14, 18, 19, 20, 24, 27, 29, 30], "10000": [2, 5, 6, 10, 11, 13, 24], "100000": 8, "10001": 10, "1001": 24, "1002": 24, "1003": 24, "1005": 24, "1009": 24, "101": 16, "1011": 24, "1013": 24, "1013904243": 24, "1015": 24, "102": 16, "1023": 24, "1024": 3, "1026": 24, "1027": 24, "103": 1, "1030": 24, "1037": 24, "1038": 24, "1040": 24, "1047": 24, "107": 16, "108": [], "10th": 9, "10x": [0, 27], "11": [0, 2, 5, 6, 7, 8, 9, 10, 11, 12, 13, 16, 21, 22, 24, 26, 27, 28, 29, 30], "110": [], "1100": 24, "1101": 24, "111": [1, 7, 12], "112": 16, "11340253": [], "11590451": [], "116": 16, "117": 16, "118": 16, "12": [0, 1, 2, 3, 4, 5, 6, 8, 9, 11, 12, 18, 21, 24, 26, 27, 28, 29, 30], "120": 3, "121": [8, 9, 10, 16], "1215pm": [25, 27], "122": [8, 9, 10], "124": [0, 27], "125": 16, "127": [4, 16], "128": [3, 4, 13, 30], "129": 16, "1298": 9, "12pm": [25, 27], "13": [0, 2, 9, 12, 21, 24, 27], "131": 16, "133": 7, "135": 16, "136": 16, "14": [0, 2, 4, 6, 8, 9, 10, 12, 21, 24, 26, 28], "141": 16, "1412": 30, "143": 16, "1446729567": 4, "149": 16, "14g": 6, "15": [0, 2, 4, 6, 7, 8, 9, 12, 13, 22, 24, 27, 29, 30], "150": [4, 8], "152": 16, "153760": [], "156": 16, "157": [], "158": [], "159": 16, "15g": 6, "15pm": 27, "16": [1, 2, 3, 4, 5, 8, 9, 10, 24, 27, 29], "160": 16, "1603": 3, "161": 16, "162": 16, "16231451": 4, "163": 16, "16384": 3, "164": 16, "167": 16, "17": [1, 2, 8, 24], "172": 16, "173": 16, "176": 16, "178": 16, "179": 16, "1797": 1, "18": [2, 6, 7, 8, 9, 10, 24, 27], "1807": 4, "18392847": [], "19": [2, 24, 27], "1940": [], "1943": 12, "19569961": 28, "1970": [21, 27], "1973": 9, "1979": 6, "1_1": 12, "1_2": 12, "1_3": 12, "1cm": [0, 8, 10, 24, 27], "1d": [1, 2, 3], "1e": [2, 4, 13, 14, 30], "1e10": 14, "1e4": 6, "1f": 1, "1k": 21, "1n": [0, 27], "1x": [0, 27], "2": [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 20, 21, 22, 24, 26, 30], "20": [0, 1, 2, 6, 7, 8, 16, 17, 24, 25, 27, 28, 29, 30], "200": [0, 2, 3, 4, 8, 9, 10], "2000": [0, 28], "2004": [13, 29], "2006": 26, "20072279": [], "2008": 27, "2010": 1, "2011": 1, "2012": 30, "2014": [4, 30], "2015": 1, "2016": [0, 27], "2018": [0, 6, 28], "2021": [6, 14, 28, 30], "2022": 27, "2025": [18, 27, 28, 29, 30], "21": [0, 1, 5, 7, 9, 12, 21, 27, 28, 29], "2116753732": 4, "215pm": [25, 27], "2167072": [], "22": [0, 1, 5, 12, 13, 21, 27, 28, 29], "221": 8, "225": 4, "22948497": [], "23": [1, 12, 21], "24": [0, 1, 21, 27], "25": [2, 3, 4, 5, 6, 8, 9, 11, 28], "250": [2, 4, 7, 9], "25000": [], "250154": [], "253775": [], "255": 3, "256": 4, "25x": 22, "26": [], "26303845": [], "264": [], "265": [], "265109911": 4, "266": [], "269": [], "27": 1, "270": [], "278": [29, 30], "27n_": 24, "28": [1, 3, 4], "283": [29, 30], "2830637392": 4, "2861": 24, "2873": 9, "2882": 24, "2886": 24, "2890": [0, 27], "2892": 24, "29": 28, "2915": 24, "2931": 27, "29364655": [], "294399745619595": [], "296247": [], "2968": 27, "2980": 27, "298273": [], "298375": [], "2990": 27, "2_": 12, "2_1": 12, "2_2": 12, "2_3": 12, "2_i": 12, "2_m": [6, 24], "2_t": [13, 30], "2_x": 24, "2a": 17, "2b": 24, "2c8f433990d1": 30, "2cm": 8, "2d": [1, 3, 11, 12, 20, 27], "2e": 6, "2f": [0, 7, 9, 10, 11, 12, 27], "2g": 2, "2g_i": 2, "2k": 3, "2m": 6, "2mvizaqfst8": 28, "2n": [0, 2, 3, 27, 28], "2nd": 9, "2p": 24, "2pt": 4, "2x": [0, 3, 8, 13, 27], "2x_ix_jy_iy_j": 8, "2x_j": 8, "2y_i": 10, "2y_j": 8, "3": [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 20, 21, 22, 23, 24, 25, 27, 29, 30], "30": [0, 1, 4, 6, 7, 10, 13, 25, 30], "30000": [0, 27], "3072": 3, "31": [12, 21, 24], "315": [6, 28, 30], "3155": [0, 5, 6, 28, 29, 30], "32": [3, 4, 6, 12, 13, 21, 24, 30], "3200": 1, "3250": 1, "3297": [], "33": [12, 21, 25], "3303": [], "3310": [], "332331": [], "333": 7, "3331": [], "3337": [], "34": 21, "3436": [0, 27], "3437": [0, 27], "35": [0, 6, 22, 27, 29, 30], "3581341341": 4, "359": [5, 29], "36": [0, 5, 6, 18, 22, 24], "37": [22, 29], "370782966": 4, "38": [22, 24], "39": [0, 25, 27], "3d": [2, 3, 4, 6, 13, 16], "3f": [1, 3, 9], "3n": 21, "3x": [2, 8], "3x_i": 2, "3y": 8, "4": [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 21, 22, 24, 27, 29, 30], "40": [1, 6, 25, 27], "400": 4, "4000": 27, "4050": [26, 27], "41": 21, "4155": [2, 15], "41589548": [], "42": [1, 4, 8, 9, 10, 21], "43": [0, 7, 21], "4310": 27, "436462435": 4, "44": [0, 21, 29, 30], "45": [25, 27], "46": [25, 27], "462": 7, "47": [25, 27], "479465113": 4, "47958494": [], "48": [], "48257387": [25, 27], "49": [5, 6, 11], "49152": 3, "4940954": [0, 27], "4990": 24, "4992": 24, "4997": 24, "4c4c7f": [9, 10], "4d": 3, "4f": 6, "4pm": [25, 27], "4y": 8, "4y_i": 10, "5": [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 17, 19, 21, 22, 24, 27, 28, 29, 30], "50": [1, 2, 3, 4, 6, 7, 8, 10, 13, 27, 28, 30], "500": [1, 3, 4, 6, 9, 10, 13, 30], "5018": 24, "506": [], "507d50": [9, 10], "50j": 13, "50x10": 1, "51": 10, "510": 1, "512132": [], "5177783846": 4, "53": 9, "54": [6, 24], "5411205": [], "54894451": [], "55": 1, "56": 1, "56536": [0, 27], "569": 1, "57": [0, 8, 25, 27], "571": [5, 29], "58": [10, 25, 27], "591317992": 4, "5cm": 24, "5f": [8, 30], "5x": [8, 18], "5y": 8, "6": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 12, 13, 18, 21, 24, 25, 27, 28, 29, 30], "60": [1, 3], "60000": 4, "6019067271": 4, "606439": [], "625": 7, "63": 1, "64": [1, 3, 4, 13, 21, 27, 30], "64x50": 1, "65": [1, 8, 9], "6887363571": 4, "69": [16, 24], "69069n_": 24, "691": [], "6980": 30, "6n_": 24, "7": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 11, 12, 13, 21, 22, 24, 26, 27, 28, 30], "70": [1, 7], "70653767": 4, "71": 1, "724": 3, "73": [], "7304881": [], "75": [5, 6, 8, 11], "76": [25, 27], "765": 7, "77": [25, 27], "7718": 9, "7782028952": 4, "77893972": [], "78": [], "7d7d58": [9, 10], "8": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 12, 13, 14, 18, 19, 21, 24, 25, 27, 29], "80": [0, 1, 5, 8, 17, 28], "800": [4, 7], "81": 1, "815am": [25, 27], "85": 1, "8702784034": 4, "88": 27, "8f": 6, "8g": 6, "8n": 21, "8x8": 1, "9": [0, 1, 2, 4, 5, 6, 7, 8, 9, 11, 12, 13, 21, 24, 27, 30], "90": 1, "9040": 9, "91": [25, 27], "92": [25, 27], "93": 16, "931": [0, 27], "933": [5, 29], "937": 24, "938": 24, "939": [0, 24, 27], "94": 24, "95": [1, 11], "954": 24, "955820c21e8b": 4, "96": 6, "960": 24, "961": 24, "962": 24, "9649652536": 4, "96611194e": [], "9780387310732": 26, "9780387848570": 26, "9781098134174": 27, "9781492032632": 26, "9781801819312": 27, "97898392": 28, "98": [0, 1, 16], "985": 24, "986": 24, "989": 24, "9898ff": [9, 10], "99": [13, 16, 30], "991": 24, "992": 24, "993": 24, "996": 5, "999": [9, 24, 30], "9x": 6, "9y": 6, "A": [2, 3, 5, 6, 7, 10, 11, 12, 13, 15, 16, 19, 20, 21, 23, 24, 25, 26, 28, 29, 30], "AND": 2, "And": [0, 3, 4, 5, 6, 9, 13, 20, 22, 24, 29], "As": [0, 1, 2, 3, 4, 5, 6, 8, 10, 12, 13, 15, 16, 21, 22, 24, 27, 28, 29, 30], "At": [0, 4, 6, 13, 27, 30], "BE": [0, 27], "Be": [2, 18, 20, 27], "Being": 13, "But": [0, 1, 2, 3, 5, 6, 9, 10, 16, 24, 28], "By": [0, 3, 5, 6, 12, 13, 17, 21, 27, 28, 29, 30], "For": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 18, 19, 20, 21, 22, 24, 26, 27, 28, 29, 30], "IF": [6, 28, 30], "IN": 26, "If": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 15, 16, 18, 20, 21, 22, 24, 27, 28, 29, 30], "In": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 18, 20, 21, 22, 24, 26, 27, 28, 29, 30], "Ising": [5, 12, 28, 29], "It": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 18, 20, 21, 22, 24, 27, 28, 29, 30], "Its": [1, 2, 4, 11], "No": [6, 9, 27, 28, 30], "Not": [0, 1, 5, 6, 28, 29, 30], "OR": 24, "Of": 24, "On": [0, 3, 22, 24, 25, 26, 27], "One": [0, 1, 3, 4, 5, 6, 7, 8, 11, 12, 13, 17, 24, 28, 29, 30], "Or": [0, 1, 6, 27], "Such": [0, 6, 12, 16, 24, 30], "That": [0, 5, 7, 10, 11, 12, 14, 22, 24, 27], "The": [4, 10, 13, 14, 16, 17, 18, 19, 21, 22, 23, 24, 25, 26], "Then": [0, 1, 6, 8, 9, 10, 11, 12, 13, 14, 15, 16, 18, 21, 27, 29, 30], "There": [0, 3, 4, 5, 6, 8, 9, 11, 12, 14, 15, 21, 22, 24, 25, 27, 28, 29, 30], "These": [0, 3, 4, 5, 8, 9, 10, 11, 12, 13, 14, 17, 18, 21, 22, 24, 25, 27, 28, 29, 30], "To": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 15, 16, 21, 24, 28, 29, 30], "With": [0, 5, 6, 8, 9, 10, 11, 12, 14, 16, 19, 21, 22, 24, 27, 28], "_": [0, 1, 2, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 16, 17, 18, 19, 21, 22, 27, 28, 29, 30], "_0": [5, 8, 10, 11, 13, 28, 29], "_1": [2, 5, 6, 8, 10, 11, 12, 13, 14, 21, 28, 29, 30], "_2": [2, 5, 8, 11, 12, 13, 21, 28, 30], "_3": 21, "_4": 21, "_9": [13, 30], "__class__": 10, "__doc__": 6, "__future__": [8, 9], "__init__": 1, "__main__": 2, "__name__": [2, 10], "_auto1": [2, 3, 4, 5, 6, 7, 12, 13, 21, 24, 28, 29, 30], "_auto10": [6, 12], "_auto11": 6, "_auto12": 6, "_auto2": [2, 3, 4, 5, 6, 12, 13, 21, 24, 30], "_auto3": [3, 4, 5, 6, 12, 13, 21, 30], "_auto4": [4, 6, 12, 13, 21, 30], "_auto5": [4, 6, 12, 13, 21, 30], "_auto6": [4, 6, 12, 21], "_auto7": [4, 6, 12, 21], "_auto8": [6, 12], "_auto9": [6, 12], "_build": [0, 20, 22, 26, 27], "_c": 1, "_center": [], "_compon": 11, "_depth": 9, "_export": [15, 16], "_fraction": 9, "_i": [0, 1, 2, 5, 6, 7, 8, 11, 12, 13, 19, 22, 27, 28, 29, 30], "_j": [0, 1, 2, 3, 5, 6, 8, 13, 19, 22, 28, 29, 30], "_k": [13, 29, 30], "_l": 12, "_lambda": 6, "_leaf": 9, "_m": 10, "_multilayer_perceptron": [], "_n": [2, 5, 8, 11, 13, 28, 29, 30], "_node": 9, "_norm": [], "_p": [5, 8, 28, 29], "_ratio": 11, "_sampl": 9, "_split": [6, 9, 22], "_t": [13, 30], "_test": [6, 22], "_varianc": 11, "_weight": 9, "a0": 3, "a0faa0": [9, 10], "a1": [0, 27], "a2": [0, 27], "a3": [0, 27], "a4": [0, 27], "a_": [0, 1, 16, 21, 27, 28], "a_0": [0, 27], "a_1a": [0, 27], "a_2a": [0, 27], "a_3": [0, 27], "a_3a": [0, 27], "a_4": [0, 27], "a_4a": [0, 27], "a_h": 1, "a_i": [0, 1, 2, 12, 27], "a_j": [1, 12], "a_k": [0, 1, 12], "aaron": 26, "ab": [0, 2, 5, 13, 14, 27, 28, 30], "ab_channel": 20, "abandon": 1, "abid": 24, "abil": [0, 10], "abl": [0, 1, 4, 5, 6, 7, 10, 12, 13, 16, 18, 19, 22, 28, 29, 30], "about": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 15, 16, 20, 21, 22, 25, 30], "abov": [0, 1, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 21, 24, 26, 27, 28, 30], "abovement": [6, 22, 27], "abscissa": [13, 29], "absolut": [0, 2, 5, 6, 13, 27, 28, 29], "absorb": [28, 29], "abstract": 1, "acceler": [13, 30], "accept": [0, 3, 6, 9, 22, 28], "access": [3, 11, 24, 27], "accid": [4, 6], "accompani": [0, 27, 28], "accomplish": [8, 9, 13, 30], "accord": [0, 1, 2, 5, 6, 9, 12, 13, 14, 24, 27, 29, 30], "accordingli": 11, "account": [0, 3, 5, 13, 15, 16, 24, 27, 30], "accumul": [12, 13, 24, 30], "accur": [0, 3, 4, 6, 10, 13, 30], "accuraci": [0, 1, 3, 4, 5, 6, 7, 9, 10, 11, 12, 27, 28, 29], "accuracy_scor": [0, 1, 10, 27], "accuracy_score_numpi": 1, "achiev": [0, 1, 5, 6, 8, 12, 21, 27, 30], "aco": 24, "acquaint": 20, "acquir": [1, 20, 27], "acr": [], "across": [1, 3, 6, 9, 17, 20, 27], "act": [1, 3, 21], "action": 24, "activ": [0, 2, 3, 4, 9, 15, 23, 25, 27, 30], "actual": [0, 1, 4, 5, 6, 8, 11, 15, 16, 18, 21, 24, 27, 28, 29, 30], "ad": [1, 3, 4, 5, 8, 13, 15, 16, 21, 29, 30], "ada_clf": 10, "adaboostclassifi": 10, "adadelta": [13, 30], "adagrad": 22, "adam": [1, 3, 4, 22, 27], "adapt": [4, 6, 13, 17, 26, 29], "add": [0, 1, 2, 3, 4, 5, 6, 8, 10, 11, 12, 15, 16, 17, 18, 24, 25, 27, 28, 29, 30], "add_subplot": [1, 7, 12, 14], "addendum": 5, "addit": [0, 2, 3, 5, 6, 7, 8, 9, 10, 12, 13, 15, 20, 21, 22, 24, 25, 26, 27, 28, 30], "addition": [12, 13, 29, 30], "address": [1, 9, 11, 13, 27, 30], "adjac": [3, 12], "adjoint": [5, 28], "adjust": [0, 5, 12, 13, 29, 30], "admir": [0, 27], "advanc": [4, 6, 12, 26, 27, 30], "advantag": [1, 3, 5, 6, 10, 13, 21, 29, 30], "adversari": 27, "afecionado": 27, "affect": [3, 15], "affin": [0, 3, 8, 11, 28], "afford": 3, "aficionado": 27, "aforement": 14, "african": [], "after": [0, 1, 2, 4, 5, 6, 9, 11, 12, 13, 15, 16, 17, 18, 19, 20, 21, 22, 24, 27, 28, 29, 30], "afterward": [0, 27], "ag": [0, 7, 27, 28], "ag_0": 2, "again": [0, 1, 4, 5, 6, 7, 8, 10, 11, 12, 13, 22, 24, 27, 28, 29, 30], "against": [1, 4, 7, 10], "agegroup": 7, "agegroupmean": 7, "aggreg": [9, 10], "agorithm": 10, "agre": [5, 6, 24, 28, 29, 30], "agreement": [13, 30], "ahead": 9, "ai": [0, 26], "aid": 11, "aim": [0, 1, 4, 6, 7, 11, 14, 16, 17, 19, 20, 21, 22, 28], "ainv": 5, "airplan": 3, "aka": 5, "al": [0, 2, 4, 16, 17, 26, 27, 28, 29], "alarm": [5, 7], "aldo": 28, "algebra": [0, 3, 5, 13, 20, 28, 29], "algorithm": [0, 1, 2, 4, 5, 6, 7, 8, 13, 14, 16, 20, 21, 22, 24, 26], "align": [0, 2, 5, 6, 7, 8, 13, 24, 27, 28, 29], "all": [0, 1, 2, 3, 4, 5, 6, 7, 9, 10, 11, 12, 13, 14, 15, 18, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30], "allevi": [1, 13, 29], "alloc": [3, 21], "allow": [0, 1, 2, 3, 5, 6, 8, 10, 13, 15, 20, 21, 22, 27, 28, 29, 30], "almost": [0, 1, 6, 8, 11, 13, 24, 29, 30], "alon": [2, 9], "along": [2, 3, 4, 5, 6, 9, 10, 11, 15, 20, 21, 27, 28, 29], "alpha": [0, 1, 2, 3, 4, 6, 7, 8, 9, 10, 13, 14, 24, 27, 28, 29, 30], "alpha_": [10, 30], "alpha_0": 3, "alpha_1": 3, "alpha_2": 3, "alpha_i": [3, 13], "alpha_k": 13, "alpha_m": 10, "alpha_n": 3, "alpha_opt": 13, "alreadi": [2, 3, 4, 5, 6, 10, 12, 15, 20, 21, 24, 27, 28, 29], "also": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 18, 19, 20, 21, 22, 24, 27, 28, 29, 30], "alter": 1, "altern": [0, 1, 4, 5, 6, 8, 9, 11, 13, 15, 18, 21, 22, 27, 28, 30], "although": [0, 1, 5, 6, 8, 10, 13, 16, 27, 30], "alwai": [0, 3, 5, 6, 12, 13, 16, 22, 24, 27, 28, 29, 30], "am": 4, "ame2016": [0, 27], "american": [], "among": [0, 3, 5, 9, 10, 12, 21, 27, 28], "amongst": 5, "amount": [0, 1, 3, 4, 6, 8, 10, 14, 20], "an": [1, 2, 3, 5, 6, 7, 8, 9, 11, 12, 13, 14, 16, 17, 18, 19, 20, 21, 22, 24, 25, 26, 28, 29, 30], "an_": 24, "anaconda": [0, 1, 20, 22, 27], "analogi": [13, 30], "analys": 6, "analysi": [1, 3, 4, 7, 14, 19, 21, 26], "analyt": [2, 3, 5, 6, 7, 12, 13, 17, 20, 22, 27, 28, 29, 30], "analyz": [0, 1, 3, 4, 5, 6, 16, 22, 24, 28, 29], "andrew": 1, "angl": [0, 3, 9, 28, 30], "anharmon": 3, "ani": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 12, 14, 15, 16, 19, 24, 27, 28, 30], "anim": [4, 12], "ann": 12, "annot": [0, 1, 3, 7, 8, 27], "announc": 27, "anonym": 18, "anoth": [0, 1, 3, 4, 5, 6, 7, 8, 10, 11, 12, 13, 15, 21, 22, 24, 27, 28, 30], "ansatz": [0, 18, 27], "answer": [0, 1, 3, 5, 6, 21, 22, 25, 27], "antialias": [2, 6], "anticip": 4, "anymor": [1, 8], "anyon": [4, 8, 15], "anyth": [1, 15, 16, 24], "anytim": [25, 27], "apach": 1, "apart": [11, 13, 29], "api": [1, 20, 27], "appar": 2, "appear": [0, 1, 3, 13, 21, 24], "append": [1, 3, 4, 8, 9, 13, 19, 27, 30], "appendix": 22, "appli": [0, 1, 3, 4, 6, 7, 8, 9, 10, 11, 12, 13, 18, 22, 24, 26, 27, 28, 30], "applic": [0, 1, 3, 4, 5, 6, 7, 9, 12, 13, 16, 21, 24, 26, 27, 28, 29, 30], "apply_gradi": 4, "approach": [1, 2, 4, 5, 6, 9, 10, 11, 12, 13, 15, 16, 18, 20, 22, 24, 26, 28, 29], "approch": 22, "appropri": [2, 6, 9, 12, 13, 17, 20, 24, 30], "approv": 27, "approx": [0, 2, 3, 6, 10, 11, 13, 18, 22, 24, 27, 29, 30], "approxim": [0, 1, 2, 3, 4, 5, 6, 7, 10, 11, 13, 19, 22, 24, 27, 28, 29, 30], "apt": [0, 20, 22, 27], "aq": 24, "ar": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30], "aragorn": 27, "arang": [1, 3, 4, 6, 7, 9, 10, 12, 13, 27, 30], "arbitrari": [1, 4, 6, 8, 12, 13, 24, 29], "arbitrarili": [0, 1, 11, 27], "arc": 6, "architectur": [3, 4, 12], "archiv": 22, "area": [0, 3, 6, 26, 27], "argmax": [1, 11], "argmin": [4, 10, 14], "argsort": 11, "argu": [1, 13, 30], "argument": [0, 2, 3, 5, 11, 12, 13, 17, 27, 28, 30], "aris": [0, 6, 12, 13, 24, 27, 29], "arithmet": [0, 13, 21, 27], "arm": [6, 28, 30], "armadillo": 21, "around": [0, 1, 4, 5, 6, 11, 18, 22, 24, 27], "arrai": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 12, 13, 14, 16, 18, 20, 22, 24, 28, 29, 30], "arrang": [3, 27], "arraybox": 13, "arriv": [0, 6, 9, 11, 21, 24, 27], "arrow": 12, "arrowprop": 8, "art": [0, 1, 20], "articl": [0, 3, 4, 6, 10, 19, 27, 28, 29, 30], "artifici": [0, 2, 7, 12, 26, 27], "artificialneuron": 12, "arug": 13, "arxiv": [3, 4, 30], "asarrai": [0, 6, 9, 28, 30], "asid": 28, "ask": [5, 6, 11, 12, 15, 22], "aspect": [0, 6, 20, 27, 28], "assembl": 3, "assembli": [0, 27], "assert": 4, "assess": [0, 6, 22, 27, 28], "assici": 4, "assign": [0, 7, 8, 9, 12, 13, 14, 15, 23, 25, 26, 27], "associ": [0, 6, 9, 12, 14, 24, 27], "assum": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 17, 19, 21, 22, 24, 27, 28, 29, 30], "assumpt": [0, 3, 5, 6, 9, 11, 24, 27, 28], "ast": [0, 5, 6, 27], "astyp": [4, 9, 10], "asymmetri": [0, 27], "asymptot": [4, 6], "atom": [0, 27], "attempt": [0, 4, 6, 7, 8, 10, 27, 28, 30], "attend": 27, "attent": [0, 21, 27], "attract": [0, 10, 27], "attribut": [0, 9, 27], "audi": [0, 27], "audio": [3, 4], "august": [27, 28], "aurelien": [0, 26, 27], "austfjel": 6, "auth": 15, "authent": 15, "author": [0, 1, 10, 24], "authour": 27, "auto": [9, 10, 24], "auto_exampl": [22, 28], "autocor": 24, "autocorrelation_tim": 24, "autocorrelform": 24, "autocovari": 24, "autoencod": [4, 20, 27], "autoencond": 20, "autograd": [20, 27], "autom": [0, 20, 26, 27], "automac": 21, "automag": 27, "automat": [0, 1, 2, 3, 4, 11, 16, 20, 21, 27], "automobil": 3, "autonom": 4, "avail": [0, 1, 4, 6, 10, 11, 20, 21, 22, 23, 25, 26, 27], "averag": [0, 1, 3, 6, 9, 10, 13, 14, 24, 25, 27, 28, 30], "avoid": [0, 4, 5, 6, 9, 11, 13, 18, 21, 28, 30], "awai": [2, 3, 6, 28, 30], "awar": [2, 10], "award": [25, 27], "ax": [0, 1, 2, 3, 4, 6, 7, 8, 9, 10, 11, 12, 13, 14, 21, 22, 27], "axes3d": [2, 6, 13, 29, 30], "axes_grid1": 6, "axhlin": 8, "axi": [0, 1, 2, 3, 4, 6, 7, 8, 9, 10, 11, 12, 13, 14, 18, 21, 24, 27, 28, 29, 30], "axiom": 5, "axvlin": [4, 8], "axvspan": 4, "b": [0, 1, 3, 4, 5, 6, 8, 9, 10, 12, 13, 14, 15, 16, 17, 19, 24, 25, 27, 28, 29, 30], "b1": 8, "b2": 8, "b3": 8, "b_": [0, 1, 21], "b_0": 0, "b_1": [0, 2, 12, 13, 30], "b_2": [0, 13], "b_5": [13, 30], "b_group": 9, "b_i": [0, 1, 2, 12, 27], "b_ia_": [0, 27], "b_ia_i": 0, "b_index": 9, "b_j": [1, 12], "b_k": [0, 1, 12, 13, 30], "b_m": 12, "b_score": 9, "b_valu": 9, "ba": 30, "babcock": 27, "bachelor": [23, 25], "back": [0, 3, 4, 5, 6, 8, 9, 10, 15, 16, 21, 24, 27, 30], "backbon": 21, "backend": [1, 4], "background": [26, 27], "backpropag": 1, "backtrack": 9, "backup": 21, "backward": [1, 2, 4, 12, 21], "bad": [6, 17, 28], "badli": 24, "bag": [9, 20, 27], "bag_clf": 10, "baggin": 27, "baggingboot": 10, "baggingclassifi": 10, "baggingtre": 10, "balanc": [6, 30], "ballpark": 18, "band": 21, "bandwidth": 21, "bar": [0, 6, 11, 22, 27], "barber": 26, "bare": [4, 10], "base": [0, 1, 3, 4, 5, 7, 8, 9, 10, 14, 15, 16, 17, 20, 24, 25, 26, 27, 28, 29], "basi": [5, 7, 8, 10, 11, 12, 13, 21, 28, 29], "basic": [6, 8, 12, 13, 14, 15, 20, 22, 24, 27], "batch": [3, 4, 11, 12, 13, 29], "batch_shap": 4, "batch_siz": [1, 3, 4], "batchnorm": 4, "bay": 7, "bayesian": [5, 20, 26, 27], "becaus": [0, 1, 2, 3, 4, 5, 6, 8, 9, 12, 13, 14, 27, 28, 29, 30], "becom": [0, 1, 2, 5, 6, 7, 9, 12, 13, 24, 27, 28, 29, 30], "been": [0, 1, 2, 3, 4, 5, 6, 11, 12, 13, 20, 21, 22, 27, 28, 30], "befor": [0, 1, 2, 3, 4, 5, 6, 7, 8, 12, 13, 14, 16, 17, 18, 19, 21, 22, 24, 27, 28, 30], "beforehand": [0, 24, 27], "begin": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 11, 12, 13, 14, 15, 21, 24, 25, 27, 28, 29, 30], "behav": [1, 6, 13, 29], "behavior": [0, 1, 13, 27, 29, 30], "behaviour": [12, 30], "behind": [0, 1, 6, 8, 13, 27, 29], "being": [0, 1, 2, 3, 4, 5, 7, 8, 10, 11, 12, 13, 17, 24, 27, 28, 29, 30], "believ": [9, 21], "belong": [7, 8, 9, 13, 14, 29], "below": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 15, 18, 21, 22, 24, 27, 28, 29, 30], "benchmark": 10, "benefici": [1, 13, 30], "benefit": [0, 1, 4, 11, 13, 20, 27, 29, 30], "bengio": [1, 26, 27, 28, 30], "benign": [1, 7], "besid": [4, 5, 29], "bessel": [5, 28], "best": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 12, 13, 15, 16, 18, 25, 27, 28, 29, 30], "beta": [1, 3, 10, 11, 13, 16, 17, 19, 27, 28, 29, 30], "beta1": 30, "beta2": 30, "beta_": [3, 13, 17, 28, 30], "beta_0": [1, 3, 13, 28], "beta_1": [1, 3, 10, 13, 28, 30], "beta_1x_i": 13, "beta_2": [3, 13, 30], "beta_3": 3, "beta_i": 3, "beta_j": [13, 28, 30], "beta_k": 13, "beta_linreg": [13, 30], "beta_m": 10, "beta_mg_m": 10, "beta_n": 3, "better": [0, 1, 2, 3, 4, 6, 9, 10, 11, 12, 13, 27, 28, 30], "between": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 11, 12, 13, 14, 15, 16, 17, 18, 19, 22, 24, 27, 28, 29, 30], "beyond": [0, 1, 5, 6, 8, 13, 27, 28, 29, 30], "bf": [13, 14, 19, 21, 24, 29], "bg": 27, "bgd": [13, 30], "bia": [0, 1, 2, 3, 5, 8, 9, 10, 12, 13, 27, 28, 29, 30], "bias": [1, 2, 3, 5, 6, 9, 12, 19], "bibliographi": 22, "big": [0, 1, 2, 5, 6, 14, 19], "bigger": [1, 6, 28], "bigr": 12, "bike": 9, "bilbo": 27, "billion": [3, 12, 20], "bin": [7, 24], "binari": [0, 3, 5, 7, 9, 10, 12, 27], "binarycrossentropi": 4, "bind": 0, "binomi": [20, 24, 27], "binsboot": 6, "bioinformat": 0, "biolog": [1, 12], "bios1100": [20, 27], "bird": [0, 3], "birth": 27, "bishop": [26, 27], "bit": [1, 4, 21, 24, 27], "bitwis": 24, "bivari": 2, "bk": [13, 30], "bla": [21, 27], "black": [8, 9, 14], "block": [6, 10, 20, 21, 24, 27], "blog": 27, "blogpost": 4, "blue": [0, 3], "bm": [], "bmatrix": [0, 1, 3, 5, 7, 8, 11, 13, 21, 27, 28, 29, 30], "bmi": 1, "bodi": [0, 1, 4, 12], "bold": 1, "boldfac": [0, 5, 16, 28, 29], "boldsymbol": [0, 1, 2, 3, 5, 6, 7, 8, 10, 11, 13, 14, 16, 17, 19, 22, 27, 29, 30], "boltzmann": [12, 20, 27], "book": [17, 22, 26, 27, 28], "book1": 26, "boolean": [4, 17], "boost": [1, 9, 20, 27], "boostrap": 10, "bootstrap": [1, 13, 19, 20, 22, 27, 30], "borrow": 27, "boston_dataset": [], "bot": 8, "both": [0, 1, 4, 5, 6, 8, 9, 10, 13, 14, 15, 16, 17, 20, 21, 22, 24, 25, 27, 28, 29, 30], "bottl": 7, "bound": [8, 12, 30], "boundari": [2, 4, 8, 11, 12], "box": [4, 9], "boyd": [8, 13, 29], "bracket": [4, 24], "brain": [1, 7, 12], "branch": [9, 27], "break": [0, 4, 6, 11, 14, 27], "breast": [5, 7, 11], "breviti": [13, 30], "brew": [0, 20, 22, 27], "brg": 8, "brief": [22, 28], "briefli": [0, 16, 27], "bring": [0, 5, 6, 10, 28, 30], "britt": [25, 27], "broad": 0, "broadli": 27, "brought": [13, 20, 27], "brownle": 4, "browser": [15, 27], "brute": [3, 5, 11, 28], "buffer_s": 4, "bui": 4, "build": [0, 4, 5, 6, 10, 16, 21, 24, 27], "built": [1, 3, 4, 6], "bunch": 11, "busi": [], "byte": [21, 27], "c": [0, 1, 2, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 19, 20, 21, 23, 24, 25, 26, 28, 29, 30], "c1": [8, 11], "c2": [8, 11], "c_": [8, 9, 10, 13, 24, 29, 30], "c_0": 24, "c_1": 12, "c_2": 12, "c_3": 12, "c_4": 12, "c_i": [12, 13, 30], "c_k": 24, "ca": [1, 27], "cach": 10, "cal": [0, 8, 10, 12, 13, 29, 30], "calcul": [0, 1, 2, 4, 5, 6, 8, 9, 10, 11, 12, 13, 14, 16, 19, 21, 24, 27, 30], "california": 22, "call": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 18, 19, 20, 21, 22, 24, 25, 27, 28, 29, 30], "calor": [0, 28], "cambridg": [13, 26, 29], "can": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 24, 25, 26, 28, 29], "cancel": [0, 13, 27, 28], "cancer": [5, 10], "cancerpd": 7, "candid": [8, 9, 10, 30], "cannot": [0, 1, 4, 5, 6, 7, 8, 9, 22, 24, 28, 29], "canopi": [0, 20, 22, 27], "canva": [15, 16, 22, 27], "cap": 5, "capabl": [0, 1, 8, 13, 20, 27], "capac": [2, 25], "capita": [], "caption": 22, "captur": [4, 11, 12, 27], "car": [3, 4], "card": [0, 7, 27], "cardin": 1, "care": [11, 15], "carefulli": [13, 30], "carlo": [0, 6, 20, 24, 26, 27], "carri": [2, 6, 7, 22], "cart": 10, "case": [0, 1, 2, 3, 4, 5, 6, 7, 11, 12, 13, 14, 15, 16, 20, 21, 22, 27, 30], "casella": 26, "cast": 1, "cat": [3, 4], "catch": 0, "categor": [0, 1, 3, 9, 11, 27], "categori": [0, 1, 3, 7, 10, 12, 14, 27], "categorical_crossentropi": [1, 3], "caus": [0, 5, 6, 24, 27, 28, 29, 30], "causal": 0, "causat": [0, 27], "cax": 1, "cb": [6, 27], "cbar": 1, "cc": [0, 1, 5, 13, 27, 28, 29], "ccc": [5, 12, 29], "cdf": 24, "cdot": [0, 2, 6, 12, 13, 14, 21, 24, 27, 29, 30], "celebr": [13, 29], "cell": 4, "center": [0, 1, 6, 7, 8, 9, 11, 14, 18, 22, 24, 27, 28, 30], "central": [0, 3, 5, 6, 8, 16, 21, 27, 28], "centroid": [14, 24], "centroid_differ": 14, "centuri": 3, "certain": [0, 3, 6, 7, 9, 24, 27, 28], "cg": 13, "cha": [], "chain": [0, 1, 13, 20, 24, 27], "challeng": 15, "chanc": [1, 5, 13, 24, 30], "chang": [0, 1, 2, 3, 4, 5, 6, 8, 9, 11, 12, 13, 14, 15, 16, 19, 21, 22, 24, 27, 28, 29, 30], "channel": 3, "chapter": [0, 6, 10, 11, 16, 17, 19, 21, 22, 26, 27, 28, 29, 30], "chapter3": [0, 22], "charact": [0, 3, 5, 27, 28, 29], "character": [8, 9, 10, 12, 24], "characterist": [0, 1, 3, 10, 13, 27, 30], "charg": [0, 27], "charl": [], "chase": 4, "chatgpt": [15, 22], "chd": 7, "chddata": 7, "cheap": [5, 28, 29], "cheaper": [1, 13, 30], "check": [1, 3, 4, 5, 11, 13, 15, 16, 19, 21, 27, 30], "checkmark": 3, "checkpoint": 4, "checkpoint_dir": 4, "checkpoint_prefix": 4, "chen": 10, "cheng": 28, "chiaramont": 2, "childcar": 16, "children": 16, "choic": [0, 1, 2, 3, 4, 6, 9, 12, 13, 14, 21, 27, 28, 29, 30], "choleski": [5, 21, 28, 29], "choos": [2, 3, 6, 9, 10, 11, 13, 14, 15, 18, 19, 22, 29], "chosen": [0, 1, 2, 6, 8, 9, 10, 13, 16, 24, 27, 29, 30], "chosen_datapoint": 1, "christian": 26, "christoph": [26, 27], "cifar": 3, "cifar10": 3, "circ": [1, 12, 30], "circl": [0, 8, 12, 28, 30], "circuit": 3, "circumfer": 9, "circumv": [1, 5, 13, 28, 29, 30], "cite": 22, "ckpt": 4, "clariti": 24, "class": [0, 1, 3, 4, 6, 7, 8, 9, 11, 12, 13, 24, 27], "class_nam": [3, 9], "class_val": 9, "class_valu": 9, "classic": [7, 9, 13, 30], "classif": [0, 3, 5, 6, 7, 8, 11, 12, 20, 22, 26, 27, 28], "classifi": [0, 1, 4, 7, 9, 10, 11, 27], "classificaton": 1, "classifii": 10, "clean": 1, "clear": [1, 5, 10, 12, 13, 30], "clearli": [0, 3, 5, 6, 7, 8, 24, 28, 29], "clever": [1, 10], "clf": [0, 6, 8, 9, 10, 27, 28], "clf3": 0, "clf_lasso": 6, "clf_ridg": 6, "cli": 15, "clip": [3, 24], "clone": [15, 25], "close": [0, 1, 2, 4, 6, 8, 9, 11, 12, 13, 14, 18, 24, 26, 27, 29, 30], "closer": [3, 5, 13, 28, 29], "closest": [8, 11, 13, 14], "closur": [20, 27], "cloud": [20, 27], "cluster": [0, 1, 4, 6, 11, 20, 27], "cluster_label": 14, "cm": [1, 2, 3, 6, 8, 13, 29, 30], "cmap": [0, 1, 2, 3, 4, 6, 8, 9, 10, 27], "cmap_arg": 6, "cmd": [9, 15], "cn_": 24, "cnn": 12, "cnn_kera": 3, "cntk": [20, 27], "co": [0, 2, 3, 6, 9, 13, 27], "code": [0, 3, 4, 6, 7, 8, 18, 20, 21, 24, 26], "coef": [0, 27], "coef0": 8, "coef_": [0, 5, 6, 8, 9, 13, 16, 27, 28, 29, 30], "coeff": 5, "coeffici": [0, 3, 5, 6, 7, 8, 9, 13, 18, 21, 27, 28, 30], "coerc": [0, 6, 27], "coin": [10, 24], "coin_toss": 10, "col": [0, 11, 27, 28], "colab": [20, 27], "cold": 9, "colinear": [], "collabor": 22, "collaps": 8, "collect": [2, 6, 10, 11, 17, 20, 24, 26, 27], "collinear": [5, 28, 29], "color": [0, 3, 4, 6, 8, 9, 10, 24, 30], "color_channel": 3, "color_cod": 6, "colorbar": [1, 6], "colsample_bytre": 10, "colsaobject": 10, "column": [0, 1, 2, 5, 6, 7, 8, 9, 11, 12, 16, 17, 18, 21, 27, 28, 29, 30], "columntransform": 9, "com": [4, 6, 15, 16, 20, 22, 26, 27, 29, 30], "combin": [1, 2, 5, 6, 7, 10, 15, 18, 24, 30], "come": [0, 1, 3, 4, 5, 12, 13, 14, 15, 27, 28, 29, 30], "command": [0, 1, 15], "comment": [0, 4, 5, 6, 22], "commerci": [0, 20, 22, 27], "commit": 15, "commod": [0, 27], "common": [0, 1, 3, 5, 6, 7, 9, 11, 13, 14, 16, 22, 24, 27, 28, 29, 30], "commonli": [0, 1, 4, 6, 7, 9, 13, 14, 28, 30], "commun": [0, 12, 15, 22], "commut": 3, "commutatitav": 3, "compact": [0, 1, 3, 5, 6, 7, 9, 11, 12, 13, 14, 27, 28], "compair": 0, "compar": [0, 3, 4, 5, 6, 11, 13, 18, 21, 22, 27, 28, 29, 30], "comparison": [2, 4, 13], "compat": 7, "compet": 0, "competit": 10, "compil": [0, 1, 3, 4, 13, 20, 21, 27], "complet": [0, 2, 3, 4, 9, 12, 15, 16, 17, 18, 19, 27], "completenn": 12, "complex": [1, 5, 8, 9, 11, 12, 13, 16, 19, 27, 29, 30], "complic": [0, 1, 9, 13, 22, 27, 29, 30], "compoment": 28, "compon": [0, 1, 3, 4, 5, 6, 7, 9, 14, 16, 20, 27, 28, 29], "components_": 11, "compos": [9, 12, 13, 14, 20, 27], "compphys": [0, 6, 16, 20, 22, 23, 25, 26, 27, 28, 29], "compress": [0, 27, 28], "compris": 6, "compromis": [5, 28, 29], "compulsori": [20, 27], "comput": [0, 1, 2, 3, 4, 5, 6, 7, 8, 10, 11, 12, 13, 15, 16, 17, 18, 20, 21, 22, 23, 24, 26, 27, 28, 29], "computation": [0, 3, 6, 9, 13, 24, 27, 29, 30], "computationalscienceuio": 27, "computerlab": 22, "concaten": [2, 4, 6, 14], "concav": [1, 13, 28, 29], "concentr": 10, "concept": [0, 2, 20, 27, 28], "conceptu": [12, 13, 29], "concern": [0, 1, 4, 7, 27, 29], "concic": 27, "conclud": [0, 5, 13], "conclus": 1, "cond": 2, "conda": [0, 1, 20, 22, 27], "condis": 28, "condit": [0, 2, 4, 5, 6, 8, 9, 11, 13, 24, 27, 28, 30], "conduct": 20, "condwav": 2, "confid": [0, 5, 6, 7, 8, 19, 27, 28], "configur": 3, "confirm": [5, 12], "confus": [5, 6, 7, 10, 21, 28], "confusion_matrix": 9, "congruenti": 24, "conjug": [4, 8], "conjugaci": 13, "conjunct": 3, "connect": [0, 1, 3, 4, 9, 11, 12, 13, 21, 27, 28, 29, 30], "consequ": [5, 6, 8, 10, 12, 13, 28, 29, 30], "conserv": [5, 14, 28, 29], "consid": [0, 1, 2, 3, 5, 6, 7, 8, 9, 10, 12, 13, 16, 21, 22, 24, 27, 28, 29, 30], "consider": [0, 1, 5, 13, 27, 28, 29], "consist": [1, 2, 3, 4, 6, 12, 13, 22, 24, 28, 29, 30], "constant": [0, 2, 4, 5, 6, 8, 12, 13, 16, 18, 24, 27, 28, 29, 30], "constitu": [0, 27], "constitut": [2, 6], "constrain": [1, 3, 5, 7, 11, 29], "constraint": [5, 6, 8, 13, 28, 29, 30], "construct": [0, 1, 2, 3, 5, 6, 7, 8, 9, 10, 11, 21, 24, 27, 28], "contact": [0, 27], "contain": [0, 2, 3, 4, 5, 6, 7, 8, 9, 11, 12, 13, 15, 18, 19, 21, 22, 24, 26, 27, 28, 29, 30], "contemporari": 27, "content": [1, 15, 20, 21, 27, 29, 30], "context": [6, 10, 13, 22, 29, 30], "contigu": 21, "contin": 19, "continu": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 12, 13, 19, 21, 22, 24, 27, 28, 29, 30], "contour": [9, 10, 13], "contourf": [8, 9, 10], "contrast": [1, 4, 9, 10, 12, 27, 30], "contribut": [0, 3, 5, 13, 18, 24, 27, 28, 29, 30], "contributor": [0, 22], "control": [0, 1, 3, 9, 13, 15, 20, 27, 30], "conv": [3, 4], "conv2d": [3, 4], "conv2dtranspos": 4, "convei": 27, "conveni": [5, 6, 12, 13, 21, 22, 27, 29, 30], "convent": [12, 28], "converg": [1, 2, 4, 5, 8, 13, 14, 18, 28, 29], "convergencewarn": [], "convers": 30, "convert": [0, 1, 4, 5, 9, 11, 13, 21, 27, 28, 29], "converttomatrix": 4, "convex": [4, 5, 7, 28, 30], "convinc": [13, 29], "convolut": [1, 4, 20, 27], "cool": [4, 9], "coolwarm": 6, "coordin": [5, 12, 14, 28, 29, 30], "coorel": [], "copi": [0, 1, 14, 15, 28], "core": 10, "corel": 27, "coronari": 7, "corr": [5, 7, 11, 28], "correalt": [11, 20], "correct": [0, 1, 2, 3, 4, 5, 7, 13, 15, 21, 24, 27, 28, 29, 30], "correctli": [1, 2, 6, 7, 10, 18, 19, 22], "correl": [0, 1, 3, 5, 6, 7, 10, 12, 13, 20, 24, 27, 29, 30], "correlation_matrix": [5, 7, 11, 28], "correspond": [0, 3, 5, 6, 8, 9, 11, 12, 20, 21, 22, 24, 27, 28, 29], "cortex": 12, "cosin": [3, 6], "cost": [0, 2, 3, 5, 6, 7, 8, 9, 12, 13, 16, 17, 18, 19, 22, 27], "cost_deep_grad": 2, "cost_funct": 2, "cost_function_deep": 2, "cost_function_deep_grad": 2, "cost_function_grad": 2, "cost_grad": 2, "cost_histori": [], "cost_ol": [], "cost_ridg": [], "cost_sum": 2, "costol": [13, 30], "could": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 15, 16, 17, 18, 21, 22, 24, 27, 28, 29, 30], "coulomb": [0, 27], "count": [0, 9, 15, 23, 24, 25, 27], "counterpart": 27, "countor": 13, "coupl": [4, 5, 6], "cours": [0, 1, 3, 5, 11, 15, 16, 17, 22, 25, 28], "coursework": 15, "courvil": [26, 27, 28, 30], "cov": [5, 6, 11, 21, 24, 27, 28], "cov_xi": [5, 11, 28], "cov_xx": [5, 11, 28], "cov_yi": [5, 11, 28], "covari": [0, 7, 20, 21, 27, 29], "covariance_matrix": [5, 11, 14], "cover": [0, 5, 20, 25, 26, 28, 29], "covert": [0, 27], "covxi": 24, "covxx": 24, "covxz": 24, "covyi": 24, "covyz": 24, "covzz": 24, "cpu": 1, "craft": 3, "creat": [1, 3, 4, 5, 9, 10, 11, 12, 15, 18, 19, 20, 27, 30], "create_biases_and_weight": 1, "create_convolutional_neural_network_kera": 3, "create_neural_network_kera": 1, "create_x": [5, 11], "credit": [0, 7, 25, 27], "crim": [], "crime": [], "criteria": [0, 4, 9, 10, 14, 24, 27], "criterion": [9, 10, 13, 18, 29, 30], "critic": [6, 22, 28], "critiqu": 22, "cross": [0, 1, 3, 7, 9, 10, 13, 15, 20, 24, 27, 28, 29, 30], "cross_entropi": 4, "cross_val_scor": 6, "cross_valid": [7, 10], "crossvalid": 6, "crucial": [1, 24, 30], "cs231": 3, "csr_matrix": [21, 27], "csv": [0, 4, 6, 7, 9], "ctnk": 1, "cubic": 0, "cumbersom": 5, "cumsum": [10, 11, 27], "cumul": [7, 10, 24, 30], "cumulative_heads_ratio": 10, "cup": 5, "current": [1, 2, 3, 4, 13, 14, 15, 16, 26, 29, 30], "curs": [0, 28], "curv": [6, 7, 10, 12, 22], "curvatur": [13, 29, 30], "custom": [6, 14], "custom_cmap": [9, 10], "custom_cmap2": [9, 10], "cutpoint": 9, "cv": [6, 7, 10], "cvxbook": [13, 29], "cvxopt": [5, 8, 28], "cycl": [1, 12], "d": [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 13, 14, 15, 16, 17, 19, 21, 24, 25, 27, 28, 29, 30], "d2_g_t": 2, "d_f": [13, 29], "d_g_t": 2, "d_net_out": 2, "da": 3, "dagger": [5, 21, 28, 29], "dai": [1, 9, 20], "damp": 3, "darget": 9, "darkr": 24, "dat": [0, 27], "dat_id": [0, 6, 7, 9, 27], "data": [2, 4, 5, 8, 10, 12, 13, 14, 16, 19, 21, 22, 26, 29, 30], "data1": 14, "data2": 14, "data3": 14, "data4": 14, "data_id": [0, 6, 7, 9, 27], "data_indic": 1, "data_panda": 27, "data_path": [0, 6, 7, 9, 27], "databas": 1, "datafil": [0, 6, 7, 9, 27], "datafram": [0, 4, 5, 7, 9, 11, 27, 28], "datapoint": [1, 5, 6, 7, 11, 13, 16, 29, 30], "datasci": [15, 16], "dataset": [0, 4, 6, 7, 8, 9, 10, 11, 13, 14, 16, 22, 27, 29, 30], "datatyp": 4, "date": [15, 18, 22, 27, 28, 29, 30], "daughter": 10, "david": 26, "dbh": 1, "dbo": 1, "dcomposit": 21, "ddot": 2, "dead": 1, "deadlin": 15, "deal": [0, 1, 3, 5, 6, 8, 11, 13, 14, 19, 21, 24, 27, 28, 29, 30], "dealt": 0, "debt": 7, "debug": [0, 5, 6, 28, 29, 30], "decad": [0, 3, 30], "decai": [0, 13, 24, 27], "decemb": [25, 27], "decent": 10, "decid": [0, 2, 3, 5, 6, 9, 18, 28, 29, 30], "decim": [0, 27], "decis": [0, 1, 8, 11, 20, 26, 27], "decision_funct": 8, "decision_tre": 9, "decisiontreeclassifi": [9, 10], "decisiontreeregressor": [0, 9, 10], "declar": [0, 4, 21, 27], "decompos": [5, 6, 21, 28, 29], "decomposit": [0, 6, 12, 27], "decompost": [5, 28, 29], "deconvolut": 3, "decorrel": [10, 13, 30], "decreas": [1, 2, 4, 5, 6, 10, 11, 13, 19, 29, 30], "deduc": [0, 27], "deep": [3, 7, 12, 13, 20, 26, 28, 29, 30], "deep_neural_network": 2, "deep_param": 2, "deep_tree_clf": [9, 10], "deep_tree_clf1": 9, "deep_tree_clf2": 9, "deepen": [5, 20, 27], "deeper": [0, 3, 4, 27], "deeplearningbook": [26, 27, 29, 30], "deer": 3, "def": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 13, 14, 16, 17, 24, 27, 28, 29, 30], "def_covari": 24, "default": [0, 1, 2, 4, 6, 7, 21, 27, 28], "default_tim": 4, "defect": [5, 28, 29], "defici": [5, 28, 29], "defin": [0, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 16, 17, 18, 19, 21, 22, 24, 28, 29, 30], "definit": [1, 2, 5, 6, 7, 8, 10, 11, 12, 13, 21, 24, 28, 29, 30], "defint": 24, "degre": [3, 5, 6, 8, 9, 10, 11, 15, 16, 19, 22, 24, 27, 29, 30], "deisenroth": 28, "del": 1, "delet": [6, 15], "delimit": 4, "deliv": [15, 23, 27], "delta": [0, 2, 3, 6, 8, 12, 13, 14, 27, 30], "delta_": [1, 21], "delta_0": 3, "delta_1": 3, "delta_2": 3, "delta_3": 3, "delta_4": 3, "delta_5": 3, "delta_h": [0, 1, 27], "delta_j": [3, 12], "delta_k": 12, "delta_l": [1, 3], "delta_momentum": [13, 30], "delta_n": [0, 3, 27], "delug": 20, "delv": 0, "demand": [13, 29], "demonstr": [0, 3, 5, 6, 7, 11, 12, 20, 27, 28, 29], "den": 4, "denomin": [1, 5, 30], "denot": [1, 2, 6, 7, 13, 24, 29, 30], "dens": [1, 3, 4], "densiti": [0, 2, 6, 24], "depart": [25, 27, 28, 29, 30], "depend": [0, 1, 2, 4, 5, 6, 7, 8, 11, 12, 13, 15, 16, 20, 21, 22, 24, 27, 28, 29, 30], "depict": 24, "deploy": [0, 20, 22, 27], "depth": [0, 3, 9, 10, 21], "deriv": [0, 1, 2, 6, 7, 8, 10, 11, 13, 18, 20, 22, 27], "derivati": 13, "derivative_fn": 13, "descend": [5, 9, 11, 28, 29], "descent": [0, 1, 3, 7, 8, 12, 27, 28], "describ": [0, 2, 4, 5, 6, 8, 10, 11, 12, 13, 19, 21, 22, 27, 30], "descript": [0, 8, 9, 22, 27], "design": [0, 1, 3, 4, 5, 6, 7, 10, 11, 12, 13, 17, 18, 22, 27, 29, 30], "designmatrix": [0, 27], "desir": [0, 2, 4, 5, 13, 14, 27, 28, 29, 30], "desktop": 15, "despit": [1, 12], "destroi": 21, "det": [5, 21, 28, 29], "detail": [0, 6, 11, 13, 14, 18, 21, 22, 28, 29], "detect": [3, 8, 12], "determin": [0, 2, 3, 4, 5, 6, 8, 9, 10, 11, 12, 13, 18, 21, 24, 27, 28, 29, 30], "determinist": [7, 13, 24, 29, 30], "dev": [1, 22], "develop": [0, 3, 5, 8, 10, 11, 12, 20, 21, 22, 27, 28], "deviat": [0, 1, 2, 4, 5, 6, 17, 18, 19, 22, 24, 27, 28, 30], "devis": 12, "df": [4, 8, 11, 13, 27], "df1": 27, "di": [], "diag": [5, 8, 28, 29, 30], "diagnost": [1, 10], "diagon": [0, 5, 7, 13, 18, 19, 21, 24, 27, 28, 29, 30], "diagonaliz": [5, 28, 29], "diagram": 10, "diagsvd": 6, "dice": [6, 24], "dict": [6, 8], "dictionari": [], "did": [0, 1, 5, 6, 7, 10, 11, 14, 16, 22, 27], "die": 1, "diff": 2, "diff1": 2, "diff2": 2, "diff_ag": 2, "diffeent": 8, "differ": [0, 1, 2, 3, 4, 5, 6, 7, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 24, 26, 27, 28, 29], "differenti": [0, 3, 16, 20, 21, 27, 28, 29], "difficult": [0, 1, 6, 10, 13, 24, 27, 30], "difficulti": [0, 1, 13, 27, 29], "diffonedim": 2, "digit": [0, 1, 3, 4, 6, 25, 27], "dilemma": [13, 30], "dilut": 1, "dim": [4, 11, 14, 21], "dimens": [0, 1, 2, 3, 4, 5, 8, 11, 14, 16, 21, 27, 28, 29, 30], "dimension": [0, 4, 5, 6, 9, 11, 13, 14, 19, 20, 21, 22, 27, 28, 29, 30], "dimensionless": [0, 3, 27], "diment": 21, "diminish": 30, "dimnsion": 4, "diod": 3, "direct": [0, 1, 2, 4, 11, 12, 13, 14, 27, 28, 29, 30], "directli": [1, 4, 5, 6, 18, 24, 28, 29], "disadvantag": [0, 27, 30], "disappear": [3, 6], "disc_loss": 4, "disc_tap": 4, "discard": [6, 11], "disciplin": [0, 3, 12], "disclaim": 24, "discord": 27, "discourag": [13, 15, 29], "discov": [0, 27], "discover": 5, "discret": [1, 3, 5, 7, 13, 30], "discrimin": [4, 7, 10, 11], "discriminator_loss": 4, "discriminator_loss_list": 4, "discriminator_model": 4, "discriminator_optim": 4, "discuss": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 16, 18, 19, 20, 21, 22, 24, 26, 27, 28, 29, 30], "diseas": 7, "disguis": [6, 28, 30], "disord": [1, 7], "displai": [0, 1, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 14, 22, 24, 27, 28], "displaystyl": [0, 5, 17, 27, 28, 29, 30], "disregard": [0, 27], "dissimilar": [11, 14], "dist": 14, "distanc": [8, 9, 11, 14, 24], "distance_list": 9, "distinct": [3, 7, 8, 9, 10, 14], "distinctli": 8, "distinguish": [0, 4, 7, 8, 24, 27], "distplot": [], "distribut": [0, 1, 4, 6, 7, 10, 11, 13, 14, 18, 19, 20, 21, 22, 27, 28, 29, 30], "distrubut": [0, 20, 22, 27], "dive": [0, 8, 21, 27], "diverg": [1, 13, 29, 30], "divid": [0, 1, 3, 5, 6, 7, 8, 9, 11, 12, 18, 24, 27, 28, 30], "divis": [6, 8, 9, 13, 18, 21, 24, 30], "dna": 7, "dnn": [0, 1, 2, 4, 12, 27], "dnn1": 4, "dnn2_gru2": 4, "dnn_kera": 1, "dnn_model": 1, "dnn_numpi": 1, "dnn_scikit": [0, 1, 27], "do": [0, 2, 3, 4, 5, 6, 8, 9, 10, 11, 12, 13, 14, 15, 16, 18, 19, 21, 22, 27, 28, 29], "doc": [0, 15, 16, 20, 22, 23, 25, 26, 27], "document": [4, 13, 15], "doe": [0, 1, 2, 3, 4, 5, 6, 8, 10, 11, 12, 13, 15, 16, 17, 18, 21, 22, 24, 27, 30], "doesn": [3, 9, 12, 27], "dog": [1, 3, 4], "domain": [5, 8, 13, 22, 29], "domin": [0, 27], "don": [0, 1, 3, 5, 6, 8, 11, 13, 15, 16, 20, 22, 27, 28, 30], "done": [0, 2, 3, 4, 5, 6, 9, 10, 11, 13, 16, 21, 22, 27, 28, 29, 30], "dot": [0, 2, 3, 5, 6, 7, 8, 9, 10, 11, 12, 13, 18, 21, 22, 24, 27, 28, 29, 30], "doubl": [3, 4, 16, 21, 27], "doubli": 1, "doubt": 22, "down": [0, 3, 6, 9, 11, 12, 13, 29, 30], "download": [0, 1, 3, 5, 6, 15, 21, 26, 27], "downsampl": 3, "dozen": 1, "dq": 6, "drag": [13, 30], "dramat": 11, "drastic": 4, "draw": [4, 6, 10, 13, 29], "drawback": [0, 1, 3, 13, 28, 29, 30], "drawn": [1, 4, 6, 7, 11, 24, 27], "drive": [3, 4], "driven": 3, "drop": [0, 1, 5, 6, 11, 13, 24, 27, 28, 29, 30], "dropna": [0, 6, 27], "dropout": 4, "dt": [2, 3, 13, 24, 30], "dtype": [0, 1, 3, 4, 14, 21, 27], "dub": [0, 27], "due": [1, 2, 5, 6, 8, 10, 12, 13, 18, 25, 27, 28, 29, 30], "dummi": [], "dure": [0, 1, 3, 4, 8, 9, 11, 20, 22, 27, 30], "dwell": [], "dwh": 1, "dwo": 1, "dx": [2, 3, 8, 24], "dx_1": 24, "dx_1p": 6, "dx_2p": 6, "dx_mp": 6, "dx_n": 24, "dxp": 6, "dy": [1, 8, 24], "dynam": 4, "dz": 8, "e": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 24, 25, 27, 28, 29, 30], "e_": [0, 2, 27], "each": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 18, 20, 21, 23, 24, 25, 27, 28, 29, 30], "eapprox": [0, 27], "earli": [1, 13, 30], "earlier": [0, 5, 7, 8, 9, 11, 12, 13, 27, 28, 30], "earthexplor": 6, "eas": [6, 9, 14], "easi": [0, 5, 6, 7, 8, 9, 10, 11, 12, 13, 15, 20, 21, 27, 28, 29, 30], "easier": [5, 6, 8, 9, 13, 15, 22, 24, 27, 28, 29], "easiest": [13, 18], "easili": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 21, 22, 27, 28, 29, 30], "eastern": [25, 27], "ebind": [0, 27], "eblock": 9, "econometr": 27, "economi": 5, "ecosystem": [20, 27], "ect": 23, "edg": 3, "edgecolor": 6, "editor": 15, "edu": [13, 22, 29], "educ": [0, 22, 27], "eff": 24, "effect": [1, 4, 10, 13, 16, 17, 18, 24, 30], "effic": 1, "effici": [0, 3, 10, 13, 20, 21, 24, 27, 30], "efron": 6, "egrad": 13, "eig": [5, 11, 13, 21, 24, 27, 28, 29, 30], "eigen": 24, "eigenpair": [5, 11, 28, 29], "eigenvalu": [0, 5, 8, 11, 13, 21, 27, 28, 29, 30], "eigenvector": [5, 11, 13, 28, 29], "eight": [21, 27], "eigval": [21, 24, 27], "eigvalu": [11, 13, 29, 30], "eigvec": [21, 24, 27], "eigvector": [11, 13, 29, 30], "eir": [25, 27], "eispack": [21, 27], "either": [1, 5, 6, 7, 8, 9, 10, 11, 13, 18, 22, 24, 27, 28, 29, 30], "eivind": 25, "eivinsto": 25, "ekstr\u00f8m": 4, "elabor": 24, "elarn": 3, "electr": [0, 3, 12, 27], "electron": 27, "eleg": 11, "element": [1, 2, 3, 4, 5, 6, 7, 8, 11, 12, 13, 19, 20, 21, 22, 26, 28, 30], "elementari": [10, 13, 21], "elementwis": [3, 13], "elementwise_grad": [2, 13], "elessar": 27, "elif": 14, "elim": 21, "elimin": [3, 8], "elin": [25, 27], "ell_": 30, "ellipsi": 16, "els": [1, 3, 4, 7, 9, 12, 13, 16, 21], "elu": 1, "elus": [0, 27], "email": [23, 25, 27], "embed": [0, 11, 28], "embodi": [6, 22], "emit": 24, "emner": 26, "emphas": [0, 10, 20, 27], "emphasi": [0, 20, 26, 27], "empir": [1, 11, 24], "emploi": [0, 1, 5, 6, 11, 13, 24, 27, 28, 29], "employ": 0, "empti": [6, 10, 15], "emul": 12, "en": [20, 22, 26], "enabl": 11, "enbodi": 6, "encod": [0, 3, 5, 9, 11, 14, 27, 28, 29], "encompass": [0, 22, 24], "encount": [0, 1, 5, 7, 13, 15, 22, 24, 27, 28, 29, 30], "encourag": [15, 22], "end": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 21, 24, 25, 27, 28, 29, 30], "endpoint": [3, 6], "energi": [0, 4, 6], "enforc": 12, "eng": 26, "engin": [0, 1, 3, 4, 20, 27], "english": 22, "enocurag": 22, "enorm": 3, "enough": [0, 6, 13, 27, 29, 30], "ensembl": [1, 9, 27], "ensur": [0, 1, 2, 3, 5, 6, 11, 13, 18, 24, 28, 29, 30], "entail": 27, "enter": [5, 6, 28, 29, 30], "enthought": [0, 20, 22, 27], "entir": [1, 3, 7, 9, 20, 24, 27, 30], "entiti": [9, 12, 21, 27], "entri": [0, 5, 8, 11, 12, 21, 27, 28, 30], "entropi": [1, 3, 7, 10, 13, 27, 29, 30], "enumer": [0, 1, 2, 3, 4, 6, 8, 27, 28, 30], "env": 24, "environ": [2, 20, 22, 27], "environemnt": 15, "eo": [0, 6], "eol": 0, "eosfit": 0, "epoch": [0, 1, 3, 4, 12, 13, 27, 30], "epsilon": [0, 5, 6, 7, 13, 22, 27, 28, 29, 30], "epsilon_": [0, 27], "epsilon_0": [0, 27], "epsilon_1": [0, 27], "epsilon_2": [0, 27], "epsilon_i": [0, 27, 28], "eq": [3, 13, 14, 21, 24, 29], "eqnarrai": [3, 5, 6], "equal": [0, 1, 2, 3, 4, 5, 6, 8, 9, 11, 12, 13, 14, 16, 18, 21, 22, 24, 27, 28, 29, 30], "equat": [1, 3, 4, 5, 6, 7, 8, 9, 10, 11, 13, 14, 17, 19, 21, 24, 27, 30], "equilibrium": [2, 12], "equiv": [3, 13, 21, 24, 29, 30], "equival": [0, 1, 5, 7, 8, 11, 13, 20, 21, 27, 28, 29, 30], "erf": 24, "eriador": 27, "err": [0, 10], "err_": 6, "err_sqr": 2, "errat": [13, 29, 30], "erron": 2, "error": [1, 2, 4, 5, 6, 7, 9, 11, 12, 13, 15, 16, 17, 18, 19, 20, 21, 22, 24, 30], "error_estimate_corr_tim": 24, "error_hidden": 1, "error_output": 1, "escap": [13, 29, 30], "especi": [1, 3, 9, 12, 13, 15, 18, 22, 30], "essenti": [0, 5, 6, 9, 10, 12, 14, 15, 22, 24, 28, 29], "establish": [0, 6, 10, 11, 16, 22], "estim": [0, 1, 5, 6, 7, 10, 11, 13, 19, 20, 24, 27, 28, 29, 30], "estimated_mse_fold": 6, "estimated_mse_kfold": 6, "estimated_mse_sklearn": 6, "et": [0, 2, 4, 16, 17, 26, 27, 28, 29], "eta": [0, 1, 3, 8, 12, 13, 18, 27, 29, 30], "eta0": [8, 13], "eta_": [13, 30], "eta_t": [13, 30], "eta_v": [0, 1, 3, 27], "etc": [0, 1, 3, 5, 7, 8, 9, 11, 12, 13, 14, 20, 21, 22, 24, 28, 29, 30], "ethic": 20, "euclidean": [0, 14, 28, 30], "evalu": [0, 2, 3, 4, 5, 6, 9, 13, 15, 16, 17, 19, 22, 24, 27, 28, 29, 30], "evalut": [13, 22], "even": [0, 1, 3, 4, 5, 6, 8, 9, 10, 11, 12, 13, 14, 20, 21, 24, 27, 28, 29, 30], "evenli": 4, "event": [5, 7, 10, 24], "eventu": [0, 5, 6, 11, 12, 13, 22, 25, 28, 29, 30], "everi": [0, 1, 2, 3, 4, 5, 6, 9, 10, 11, 12, 13, 14, 15, 20, 24, 25, 27, 28, 29, 30], "everyth": [4, 12, 16, 18], "everywher": [4, 13, 29], "evolv": 0, "exact": [0, 5, 11, 12, 13, 21, 24, 27, 28], "exactli": [0, 3, 4, 6, 12, 18, 20, 28, 30], "exam": 27, "examin": 6, "exampl": [0, 5, 11, 12, 13, 15, 16, 18, 20, 21, 22, 24, 26], "exce": [1, 12, 13, 30], "excel": [0, 1, 4, 5, 10, 22, 27, 28], "except": [3, 4, 6, 8, 9, 21], "excess": [0, 27], "excit": 0, "exclud": [1, 6, 12, 28, 30], "exclus": [0, 1, 3, 6, 24, 27], "execut": [2, 5, 13, 15, 28, 29], "exemplifi": [13, 30], "exercic": [25, 27], "exercis": [5, 20, 22, 23, 25, 27, 29, 30], "exhaust": 6, "exhibit": [0, 5, 6, 8, 27, 28], "exist": [0, 1, 2, 3, 5, 6, 7, 8, 9, 13, 19, 21, 22, 27, 29], "exit": [5, 21, 28, 29], "exp": [0, 1, 2, 5, 6, 7, 8, 10, 11, 12, 13, 16, 17, 19, 24, 28, 29, 30], "exp_term": 1, "expand": [5, 7, 11, 13, 29], "expans": [0, 3, 5, 8, 10, 12, 13, 27, 28, 29], "expect": [0, 1, 5, 6, 7, 11, 12, 13, 15, 18, 20, 22, 27, 28, 30], "expectation_value_of_h_wrt_p": 24, "expens": [6, 10, 13, 16, 29, 30], "experi": [0, 1, 6, 8, 13, 15, 20, 22, 27, 28, 29], "experiment": [0, 4, 6, 9, 24, 27], "expert": [1, 9], "explain": [0, 6, 9, 10, 11, 13, 16, 19, 22, 27, 29, 30], "explained_variance_ratio_": 11, "explanatori": [0, 27], "explicit": [0, 3, 6, 13, 21, 22, 27, 28, 29, 30], "explicitli": [0, 4], "explod": 1, "exploit": [0, 3, 12, 13, 27], "explor": [1, 4, 6, 8, 13, 18, 20, 22, 27, 29, 30], "expon": 1, "exponenti": [0, 1, 5, 6, 10, 13, 24, 27, 29, 30], "export": [9, 15, 16], "export_graphviz": 9, "export_text": 9, "exporttext": 9, "expos": 20, "express": [0, 2, 3, 5, 6, 7, 10, 12, 13, 18, 21, 22, 24, 27, 29, 30], "exptmean": 24, "exptvari": 24, "extend": [0, 2, 7, 11, 13, 20, 27, 30], "extens": [0, 12, 15, 20, 27], "extent": [0, 1, 6, 26], "extern": [3, 6, 9], "extra": [1, 3, 5, 15, 25, 27, 28, 29], "extract": [0, 3, 5, 6, 7, 8, 11, 13, 16, 17, 21, 27, 28], "extrapol": [0, 27], "extrem": [0, 1, 4, 5, 6, 7, 8, 9, 13, 15, 16, 21, 28, 29, 30], "extremum": [13, 29], "extrins": 11, "ey": [0, 5, 6, 13, 14, 18, 21, 27, 28, 29, 30], "f": [0, 1, 2, 3, 4, 5, 6, 7, 8, 10, 12, 13, 14, 15, 16, 17, 18, 19, 21, 24, 25, 27, 28, 29, 30], "f1": 13, "f11": [0, 27], "f12": [0, 27], "f13": [0, 27], "f1_grad": 13, "f1d": 13, "f2": 13, "f2_grad_x1": 13, "f2_grad_x1_analyt": 13, "f2_grad_x2": 13, "f2_grad_x2_analyt": 13, "f3": 13, "f3_grad": 13, "f3_grad_analyt": 13, "f4": 13, "f4_grad": 13, "f4_grad_analyt": 13, "f5": 13, "f5_grad": 13, "f6": 13, "f6_for": 13, "f6_for_grad": 13, "f6_grad_analyt": 13, "f6_while": 13, "f6_while_grad": 13, "f7": 13, "f7_grad": 13, "f7_grad_analyt": 13, "f8": 13, "f8_grad": 13, "f9": [0, 13, 27], "f9_altern": 13, "f9_alternative_grad": 13, "f9_grad": 13, "f_": 10, "f_0": [3, 10], "f_1": [10, 13, 29], "f_2": [12, 13, 29], "f_3": 12, "f_d": 24, "f_grad": 13, "f_grad_analyt": 13, "f_i": [0, 6, 12, 16], "f_m": [3, 10], "f_n": 3, "f_vec": 2, "face": [13, 27, 29], "facecolor": [6, 8, 24], "facil": [0, 20], "facilit": 12, "fact": [0, 1, 3, 5, 9, 11, 12, 13, 27, 28, 29, 30], "factor": [0, 1, 3, 5, 6, 9, 10, 11, 13, 21, 24, 27, 28, 29], "factori": 13, "fade": 6, "fafab0": [9, 10], "fail": [0, 6, 13, 25, 27, 29], "failur": 7, "fairli": [1, 2, 18, 24, 30], "faisal": [16, 28], "fake": 4, "fake_loss": 4, "fake_output": 4, "fall": [8, 9, 23], "fals": [0, 1, 2, 3, 4, 5, 6, 7, 9, 10, 14, 16, 17, 21, 27, 28, 29, 30], "famili": [0, 7, 8, 24, 28, 30], "familiar": [0, 3, 5, 6, 8, 15, 20, 21, 22, 24, 27], "famou": [6, 12], "far": [0, 3, 4, 5, 6, 8, 11, 12, 13, 14, 16, 27, 28, 29], "fashion": [0, 9, 10, 27], "fast": [1, 3, 6, 10, 12, 13, 20, 24, 27, 29, 30], "faster": [1, 11, 13, 30], "fastest": [13, 21, 29], "favor": 7, "favorit": 24, "fc": 3, "featur": [0, 1, 3, 5, 6, 7, 8, 10, 11, 12, 13, 15, 17, 18, 19, 20, 24, 27, 29, 30], "feature_nam": [1, 7, 9], "feautur": 9, "fed": 1, "feed": [0, 2, 3, 11, 20, 27], "feed_forward": 1, "feed_forward_out": 1, "feed_forward_train": 1, "feedback": [4, 27], "feeddorward": 4, "feedforward": [1, 4, 12], "feel": [0, 5, 6, 11, 13, 15, 16, 18, 20, 22, 25, 27], "feet": [], "felt": 22, "fetch": [6, 15], "few": [1, 3, 4, 5, 9, 17, 18, 24, 27], "fewer": [0, 9, 11, 27], "ffnn": [1, 12], "field": [0, 3, 6, 12, 20], "fifth": [0, 6, 27], "fig": [0, 1, 2, 3, 4, 6, 7, 12, 13, 14, 22, 27], "fig_id": [0, 6, 7, 9, 27], "figaxi": 24, "figsiz": [0, 1, 2, 3, 4, 6, 7, 8, 9, 10, 27], "figur": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 12, 13, 14, 16, 20, 22, 27, 28, 29, 30], "figure_id": [0, 6, 7, 9, 27], "figurefil": [0, 6, 7, 9, 27], "file": [0, 4, 5, 6, 7, 9, 15, 22, 27], "file_prefix": 4, "filenam": 27, "fill": [5, 9, 18, 28, 29], "filter": [3, 4], "final": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 13, 14, 18, 19, 22, 23, 24, 25, 27, 29], "financ": 0, "find": [0, 1, 2, 3, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 20, 22, 24, 27, 28, 29, 30], "fine": [0, 14], "finish": 2, "finit": [3, 5, 6, 12, 13, 17, 24, 28, 29], "finnicki": 15, "first": [0, 1, 2, 3, 5, 6, 7, 8, 9, 10, 11, 13, 14, 15, 16, 18, 21, 22, 24, 25, 26, 28, 30], "first_moment": 30, "first_term": 30, "firsteigvector": 11, "fit": [1, 3, 4, 5, 6, 7, 8, 9, 11, 12, 13, 17, 18, 19, 22, 24, 28, 30], "fit_beta": 28, "fit_intercept": [0, 5, 6, 16, 28, 29, 30], "fit_mod": 9, "fit_theta": [6, 30], "fit_transform": [0, 6, 8, 9, 11, 15, 19], "fiti": [0, 27], "five": [0, 9, 27, 28], "fix": [0, 3, 4, 6, 10, 11, 12, 13, 22, 27], "flag": 4, "flat": [12, 13, 29, 30], "flatten": [1, 3, 4, 5, 21], "flexibl": [1, 6, 8, 10, 12, 27, 30], "flip": [25, 27], "float": [0, 3, 4, 5, 9, 11, 13, 14, 21, 27, 28, 29], "float32": [4, 9], "float64": [4, 21, 27], "flop": [5, 21, 28, 29], "flow": [1, 4, 12], "fluctuat": 5, "fly": 11, "fm": 0, "fmax": 3, "fmesh": 13, "fn": 7, "focu": [0, 3, 4, 5, 6, 15, 20, 22, 26, 27, 28, 29, 30], "focus": [1, 6, 7, 21, 28, 30], "fold": [6, 9, 22], "folder": [0, 4, 6, 15, 22, 27], "follow": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 19, 20, 21, 22, 24, 25, 26, 27, 28, 29, 30], "font": [7, 24, 27], "fontdict": 24, "fontsiz": [1, 6, 8, 9, 10, 24], "fontweight": 1, "footprint": 3, "foral": [8, 28], "forc": [0, 5, 6, 10, 11, 28, 29, 30], "forcast": 4, "forecast": [4, 12], "forest": [0, 1, 9, 20, 27], "forget": 11, "form": [0, 3, 4, 5, 6, 7, 8, 9, 11, 12, 13, 15, 16, 20, 21, 22, 24, 27, 28, 29, 30], "formal": [3, 4, 14, 18, 24], "format": [0, 1, 3, 4, 6, 7, 8, 9, 10, 11, 20, 24, 26], "format_data": 4, "formatstrformatt": [6, 13, 29, 30], "formul": [4, 6, 11, 14], "formula": [3, 13, 24, 29, 30], "forth": [4, 12], "fortran": [0, 20, 21, 27], "fortran2003": [20, 27], "fortran2008": 22, "fortran90": 24, "fortun": [0, 11, 28], "forward": [0, 3, 6, 20, 21, 27], "found": [1, 2, 4, 5, 6, 12, 13, 19, 22, 27, 28, 30], "foundat": [20, 27], "four": [4, 5, 6, 8, 12, 21, 23, 25, 27, 29], "fourier": [0, 27], "fourierdef1": 3, "fourierdef2": 3, "fourierseriessign": 3, "fourth": [12, 27, 28], "fp": 7, "frac": [0, 1, 2, 3, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 16, 17, 19, 21, 22, 24, 27, 28, 29, 30], "fraction": 9, "frame": 7, "framework": [1, 8, 10, 24], "frank": [5, 11], "frankefunct": [5, 6, 11], "fredli": [25, 27], "free": [0, 6, 11, 13, 15, 16, 18, 20, 21, 22, 24, 25, 26, 27], "freecodecamp": 20, "freedom": [5, 29], "freeli": [0, 22], "freez": 15, "frequenc": [3, 6, 7, 24], "frequent": [0, 8, 9, 13, 29], "frequentist": 20, "fresh": 10, "fridai": [15, 25, 27], "friedman": [6, 19, 22, 26, 27], "friendli": 4, "fro": 22, "frodo": 27, "frog": 3, "from": [0, 1, 2, 3, 4, 6, 7, 8, 9, 11, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 24, 25, 26], "from_cod": 9, "from_logit": [3, 4], "from_tensor_slic": 4, "front": [0, 4, 5, 27, 28, 29], "frustrat": 15, "fulfil": [2, 5, 12, 28, 29], "full": [1, 3, 5, 7, 9, 10, 13, 24, 27, 28, 29, 30], "full_matric": [5, 28, 29], "fulli": [3, 6, 12, 24], "fun": [20, 27], "func": 2, "function": [2, 3, 4, 5, 9, 14, 15, 16, 17, 18, 19, 20, 21], "functionali": 11, "fundament": [0, 6, 20, 27], "funtion": 2, "further": [2, 7, 9, 19, 27], "furthermor": [0, 3, 5, 6, 7, 11, 12, 13, 20, 22, 27, 28, 29, 30], "futur": [0, 4, 8, 9, 27], "fy": [15, 22, 23, 25, 26, 27], "fys4155": 22, "fys5419": [26, 27], "fys5429": [26, 27], "f\u00f8470": [25, 27], "g": [0, 1, 2, 3, 4, 6, 8, 9, 10, 11, 13, 15, 18, 24, 27, 28, 29, 30], "g0": 2, "g_": [2, 9, 10, 30], "g_0": 2, "g_1": [2, 10], "g_2": [2, 10], "g_analyt": 2, "g_dnn_ag": 2, "g_euler": 2, "g_i": 2, "g_m": [3, 10], "g_n": 3, "g_re": 2, "g_t": [2, 30], "g_t_d2t": 2, "g_t_d2x": 2, "g_t_dt": 2, "g_t_hessian": 2, "g_t_hessian_func": 2, "g_t_jacobian": 2, "g_t_jacobian_func": 2, "g_trial": 2, "g_trial_deep": 2, "g_vec": 2, "gain": [1, 5, 7, 9, 10, 13, 28, 29, 30], "galleri": [0, 27], "game": 4, "gamge": 27, "gamma": [0, 2, 8, 9, 10, 11, 13, 27, 29, 30], "gamma1": 8, "gamma2": 8, "gamma_": [0, 27], "gamma_0": 10, "gamma_1": 10, "gamma_1x": 10, "gamma_i": [0, 8, 24, 27], "gamma_j": [13, 30], "gamma_k": [13, 29], "gamma_m": 10, "gamma_x": [0, 27], "gap": 8, "gate": [4, 12], "gather": [0, 1, 12, 28], "gaug": 12, "gaussbacksub": 21, "gaussian": [4, 5, 6, 8, 14, 18, 24, 27], "gaussian_point": 14, "gaussian_rbf": 8, "gave": [13, 30], "gavra": 27, "gbc": 27, "gca": [2, 6, 8, 13], "gd": [1, 29], "gd_clf": 10, "gdclassiffiercgain": 10, "gdclassiffierconfus": 10, "gdclassiffierroc": 10, "gdm": [13, 30], "gdregress": 10, "ge": [1, 5, 7, 24, 28, 29], "gen_loss": 4, "gen_tap": 4, "gender": [0, 27], "genener": 4, "gener": [0, 1, 2, 3, 5, 6, 8, 10, 11, 12, 13, 14, 15, 16, 18, 21, 22, 24, 26, 28, 29, 30], "generaliz": 16, "generallay": 12, "generate_and_save_imag": 4, "generate_imag": 4, "generate_latent_point": 4, "generate_simple_clustering_dataset": 14, "generated_imag": 4, "generator_loss": 4, "generator_loss_list": 4, "generator_model": 4, "generator_optim": 4, "genom": 20, "geodes": 11, "geoff": 30, "geometr": [0, 13, 27], "geometri": 5, "georg": 26, "geotif": 6, "geq": [2, 5, 8, 9, 13, 28, 29, 30], "geron": [0, 26, 27], "get": [0, 1, 2, 3, 4, 5, 6, 7, 9, 10, 11, 13, 15, 19, 20, 21, 22, 24, 25, 27, 28, 29, 30], "get_dummi": 9, "get_paramet": 2, "get_split": 9, "get_yaxi": 8, "get_yticklabel": 6, "gh": 15, "gibb": [20, 27], "gif": 4, "gini": 10, "gini_index": 9, "ginvers": 13, "git": [0, 15, 20, 27], "giter": [13, 30], "github": [0, 20, 22, 23, 25, 26, 27, 28], "gitignor": 15, "gitlab": [0, 15, 20, 22, 27], "give": [0, 1, 2, 3, 5, 6, 7, 8, 9, 10, 12, 13, 14, 18, 19, 20, 22, 24, 27, 28, 29, 30], "given": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 17, 19, 21, 24, 27, 28, 29, 30], "global": [6, 7, 13, 29, 30], "glorot": 1, "gnew": 13, "go": [0, 1, 3, 5, 6, 8, 9, 11, 12, 13, 15, 16, 18, 27, 28, 29], "goal": [0, 7, 9, 27], "goe": [0, 1, 2, 5, 6, 13, 14, 15, 19, 21, 27, 28, 29, 30], "golden": 13, "gone": [5, 28, 29], "gong": 1, "good": [1, 3, 4, 5, 6, 9, 10, 11, 13, 15, 18, 20, 24, 26, 28, 29, 30], "goodfellow": [4, 26, 27, 28, 29], "googl": [1, 4, 20, 27], "got": [1, 6, 22], "gotten": 27, "gov": 6, "govern": 27, "gp": 26, "gpu": [1, 13, 20, 27], "grad": [2, 13, 30], "grad_analyt": 13, "grad_ol": 18, "grad_ridg": 18, "grade": [22, 23], "gradient": [0, 3, 4, 7, 8, 9, 12, 20, 27, 28], "gradient_desc": 30, "gradientboostingclassifi": 10, "gradientboostingregressor": 10, "gradients_of_discrimin": 4, "gradients_of_gener": 4, "gradienttap": 4, "gradual": [1, 14], "grai": [4, 6], "graph": [1, 9, 11, 12, 13, 16, 29], "graph_from_dot_data": 9, "graphic": [0, 1, 9, 15, 27], "grasp": 0, "gray_r": [1, 3], "grayscal": 3, "great": [5, 13, 15, 29, 30], "greater": [1, 7, 24, 28], "greatli": [13, 30], "greedi": 9, "green": [0, 3, 9, 24], "grei": 4, "grid": [1, 3, 6, 7, 8, 12, 24, 28, 30], "grossli": [13, 29], "ground": [0, 27], "group": [0, 6, 7, 9, 14, 15, 20, 22, 23, 25, 27], "groupbi": [0, 27], "grow": [1, 3, 9, 10, 30], "growth": [0, 27], "gru": 4, "guarante": [0, 4, 13, 24, 27, 28, 29, 30], "guess": [1, 4, 10, 13, 14, 29, 30], "guestrin": 10, "gui": 15, "guid": 1, "h": [0, 1, 5, 6, 8, 13, 15, 19, 24, 25, 26, 27, 28, 29, 30], "h1": 2, "h_": [0, 13, 27, 29, 30], "h_0": 30, "h_1": [2, 13, 29], "h_2": [2, 13, 29], "h_m": 10, "h_t": 30, "ha": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 17, 18, 19, 21, 22, 24, 27, 28, 29, 30], "haanen": [25, 27], "habit": [0, 28], "had": [0, 1, 6, 7, 13, 27, 29, 30], "hadamard": [1, 12, 13, 30], "half": [1, 8, 9], "halv": 10, "hand": [0, 1, 2, 3, 5, 11, 12, 13, 20, 21, 22, 24, 25, 26, 27, 28, 29, 30], "handi": [3, 22], "handl": [0, 1, 2, 5, 9, 11, 15, 18, 20, 28, 29, 30], "handle_unknown": 9, "handsid": 12, "handwrit": 12, "handwritten": [1, 5], "happen": [1, 2, 3, 4, 5, 6, 10, 13, 24, 28, 29, 30], "hard": [1, 7, 8, 10, 13, 29, 30], "hardcopi": [20, 27], "harder": [0, 1, 28], "harmon": 3, "hasn": [], "hassl": [0, 20, 27], "hast": [20, 27], "hasti": [0, 6, 16, 17, 19, 22, 26, 27, 28], "hat": [0, 1, 5, 6, 7, 9, 10, 11, 12, 13, 16, 17, 18, 19, 21, 28, 29, 30], "have": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 24, 25, 27, 28, 29, 30], "haven": 1, "he": 7, "head": [4, 10, 24], "header": [0, 27], "heads_proba": 10, "health": [0, 28], "hear": [0, 13, 27, 30], "heart": [0, 7, 27], "heatmap": [0, 1, 3, 7, 17, 27], "heavili": 0, "heavisid": 1, "height": [1, 3, 6, 28], "held": [13, 30], "help": [0, 1, 4, 12, 13, 15, 16, 22, 27, 30], "helper": [4, 14], "henc": [0, 5, 6, 8, 9, 10, 12, 13, 27, 28, 29, 30], "henrik": [25, 27], "her": 7, "here": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 17, 18, 19, 20, 21, 22, 24, 27, 28, 29, 30], "hereaft": [0, 8, 12, 27], "hermitian": 21, "hessenberg": 21, "hessian": [0, 2, 5, 13], "heterogen": [9, 10], "hi": 7, "hidden": [1, 3, 4, 12], "hidden_bia": 1, "hidden_bias_gradi": 1, "hidden_layer_s": [0, 1, 27], "hidden_neuron": 4, "hidden_weight": 1, "hidden_weights_gradi": 1, "hierarch": [5, 28, 29], "high": [0, 1, 2, 3, 4, 5, 6, 9, 10, 11, 13, 14, 20, 21, 22, 27, 28, 29, 30], "higher": [0, 1, 3, 5, 6, 8, 13, 18, 22, 27, 28, 29, 30], "highest": [1, 2], "highli": [0, 3, 4, 10, 19, 20, 21, 26, 27, 28, 29, 30], "highwai": [], "hing": 8, "hint": [13, 15, 16, 28, 29], "hinton": 30, "hip": 20, "hire": 0, "hist": [4, 6, 7, 24], "histogram": [6, 7, 24], "histor": [7, 11], "histori": [3, 4, 12, 15, 30], "hitherto": 5, "hjorth": [25, 27, 28, 29, 30], "hobbi": 24, "hoc": [5, 28, 29], "hoff": 26, "hold": [1, 3, 6, 13, 14, 29, 30], "holder": [0, 27], "home": [], "homepag": [22, 27], "homework": [6, 13, 29, 30], "homogen": [1, 3, 9, 10, 13, 30], "honchar": 2, "hopefulli": [0, 11, 15, 24, 27, 30], "horizont": 11, "horlyk": [25, 27], "hors": [3, 7, 27], "hot": [1, 9], "hour": [1, 20, 23, 24, 25, 27], "how": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 17, 18, 19, 20, 21, 22, 24, 27, 28, 29, 30], "howev": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 20, 21, 22, 24, 27, 28, 29, 30], "hspace": [0, 4, 8, 10, 24, 27], "hstack": 1, "htf": 27, "html": [0, 16, 20, 22, 23, 25, 26, 27, 28, 29, 30], "http": [0, 3, 4, 6, 13, 15, 16, 20, 21, 22, 23, 25, 26, 27, 28, 29, 30], "huang": [0, 27], "huber": [0, 27], "huge": [1, 3, 4, 20], "human": [0, 1, 3, 6, 9, 12, 28], "humid": 9, "hundr": 1, "hungri": 1, "hybrid": 23, "hydrogen": [0, 27], "hyperbol": [1, 4, 12], "hyperparam": 8, "hyperparamet": [3, 4, 5, 6, 9, 13, 18, 22, 28, 29, 30], "hyperplan": 11, "h\u00f8rlyk": [25, 27], "i": [2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 28, 29, 30], "i0": [0, 27], "i1": [0, 6, 8, 12, 27, 28, 30], "i2": [0, 8, 12, 27], "i3": [0, 12, 27], "i5": [0, 27], "i_": [13, 29, 30], "i_1": [5, 6], "i_2": [5, 6], "i_t": 30, "ian": 26, "ic": [1, 22], "id": [7, 13, 29, 30], "ida": [25, 27], "idea": [0, 1, 2, 3, 4, 6, 9, 10, 12, 13, 21, 22, 28, 29, 30], "ideal": [0, 2, 6, 8, 13, 24, 27, 30], "idem": 6, "ident": [5, 6, 12, 13, 17, 18, 21, 28, 29, 30], "identifi": [0, 1, 7, 9, 11, 12, 13, 14, 27, 28, 30], "ieor": 24, "ifi": 26, "ifs": [20, 27], "ignor": [0, 1, 3, 9, 15, 28, 30], "ii": [21, 24], "iii": [21, 27], "ij": [0, 1, 3, 6, 8, 12, 14, 16, 21, 24, 27, 28, 30], "ik": [0, 21, 27, 28], "illustr": [5, 7, 10, 12, 13, 14, 20, 27], "ilsvrc": 30, "im": 6, "imag": [1, 3, 4, 6, 9, 11, 12, 14, 26, 27], "image_at_epoch_": 4, "image_batch": 4, "image_height": 3, "image_path": [0, 6, 7, 9, 27], "image_width": 3, "imageio": 6, "images_from_seed_imag": 4, "imagin": 1, "immedi": [0, 3, 4, 6, 20, 27], "implement": [0, 2, 3, 4, 5, 6, 8, 9, 10, 11, 12, 13, 14, 22, 24, 27, 28, 29, 30], "impli": [3, 5, 6, 7, 13, 21, 28, 29, 30], "implicit": 3, "implicitli": [11, 24], "import": [0, 1, 2, 3, 4, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 22, 24, 30], "importantli": 3, "impos": [0, 6, 11, 12, 27], "imposs": [0, 5, 27, 28, 29], "impress": [0, 12, 27], "improv": [0, 4, 5, 9, 10, 11, 13, 15, 22, 28, 29], "impur": 9, "imread": 6, "imshow": [1, 3, 4, 6], "in3050": [26, 27], "in3310": 27, "in4080": [26, 27], "in4300": [26, 27], "in4310": 26, "in5400": 3, "in5550": 26, "in_out_neuron": 4, "inaccur": [13, 29], "inact": 12, "inadequ": [0, 27], "inappropri": 30, "inch": [6, 28], "includ": [0, 1, 2, 3, 4, 5, 6, 7, 11, 12, 15, 16, 17, 18, 20, 24, 25, 26, 27, 28, 29], "include_bia": [6, 9], "incom": [12, 16], "incorrect": 1, "incoveni": 8, "increas": [0, 1, 3, 4, 5, 6, 9, 12, 13, 19, 22, 24, 27, 28, 30], "increasingli": 24, "increment": 30, "ind": 6, "inde": [0, 2, 4, 5, 6, 13, 27, 28, 29], "indefinit": 4, "independ": [0, 5, 6, 7, 8, 12, 13, 24, 27, 28, 29], "index": [0, 1, 3, 4, 10, 14, 20, 21, 22, 24, 26, 27], "index_col": [0, 27], "indic": [0, 1, 3, 4, 5, 6, 9, 10, 11, 13, 16, 22, 27, 28, 30], "indispens": 6, "individu": [1, 6, 7, 10, 12, 24, 27, 28, 30], "indu": [], "indx": 21, "indx1": 2, "indx2": 2, "indx3": 2, "ineffici": [3, 13], "inequ": [8, 13], "inequaltii": 29, "inertia": [13, 30], "inf1000": [20, 27], "inf1100": [20, 27], "inf1100l": [20, 27], "inf1110": [20, 27], "inf3000": 27, "infeas": 9, "infer": [0, 1, 4, 6, 26, 27], "inferenc": 1, "infil": [0, 6, 7, 9, 27], "infin": [5, 6, 7, 11, 19, 28, 29], "infinit": [3, 30], "infinitesim": 24, "influenc": [6, 10, 18], "influenti": 1, "info": 27, "inform": [0, 1, 3, 4, 6, 9, 11, 12, 13, 14, 21, 22, 26, 27, 29, 30], "inforom": 15, "infrequ": 30, "infti": [3, 6, 13, 24, 29], "ingeni": [13, 29, 30], "ingredi": [0, 9, 27], "inher": 6, "inherit": [21, 27], "initi": [0, 1, 2, 6, 10, 13, 14, 18, 21, 24, 27, 29, 30], "inject": 14, "inlin": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 21, 24, 27, 28, 29, 30], "inner": [0, 13, 28], "inp": 4, "inplac": 13, "input": [0, 1, 3, 4, 5, 6, 7, 8, 12, 13, 14, 16, 22, 24, 27, 28, 29, 30], "input_dim": 1, "input_shap": [3, 4], "inputs": 1, "inputs_shuffl": [0, 1, 28], "insert": [3, 5, 6, 8, 10, 24, 28, 29], "insid": [4, 7], "insight": [0, 1, 5, 20, 27, 28, 29], "insist": [6, 13, 28, 30], "inspir": [0, 1, 12, 22, 27], "instabl": 2, "instal": [0, 1, 5, 6, 9, 15], "instanc": [0, 1, 2, 4, 6, 9, 11, 13, 16, 27, 28, 29], "instanti": 10, "instead": [0, 1, 2, 3, 4, 5, 6, 8, 9, 11, 13, 14, 17, 21, 24, 27, 28, 30], "institut": 1, "instruct": [0, 1, 15], "int": [0, 1, 2, 3, 4, 5, 6, 11, 13, 14, 21, 24, 28, 30], "int32": 10, "int_": [3, 6, 24], "int_0": 24, "int_a": 24, "intak": [0, 28], "integ": [1, 2, 13, 14, 21, 24, 27], "integer_vector": 1, "integr": [3, 6, 24, 27], "intellig": [0, 14, 26, 27], "intend": 10, "intens": [1, 18], "intention": 14, "interact": [0, 6, 9, 12, 20, 22, 27], "intercept": [0, 6, 8, 11, 13, 16, 17, 18, 27, 28, 29, 30], "intercept_": [0, 6, 8, 9, 13, 27, 28, 30], "interchang": [5, 12, 21], "interconnect": 1, "interesit": 19, "interest": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 12, 20, 22, 24, 27, 28, 29], "interfac": [0, 1, 15, 21, 28], "interior": [0, 9, 27], "intermedi": [21, 28], "intern": [1, 10, 12], "interpol": [1, 3, 4, 6, 12], "interpr": [5, 28, 29], "interpret": [0, 1, 6, 9, 10, 12, 13, 15, 16, 19, 21, 22, 24], "interv": [0, 3, 5, 6, 7, 13, 19, 24, 27, 28, 29], "intial": [13, 29], "intract": [0, 4, 28], "intrins": [3, 11, 21, 24, 27], "intro": [20, 26, 27], "introduc": [0, 1, 5, 6, 8, 10, 12, 21, 22, 24, 27, 29, 30], "introduct": [1, 2, 4, 13, 26, 28, 29, 30], "introductori": [0, 4, 21, 26, 27, 28], "intuit": [0, 5, 6, 8, 12, 13, 22, 27, 30], "inv": [0, 5, 13, 17, 27, 28, 29, 30], "invalu": [0, 13, 20, 27, 29], "invari": 1, "invd": 5, "inver": 8, "invers": [0, 3, 6, 13, 27, 28, 29, 30], "inverse_transform": 8, "invert": [0, 5, 7, 10, 13, 16, 18, 27, 30], "invh": [13, 30], "invok": 8, "involv": [0, 2, 6, 7, 11, 12, 27, 28, 30], "io": [0, 20, 22, 23, 25, 26, 27, 28], "ip": [0, 8, 24, 27], "ipca": 11, "ipynb": [20, 27], "ipython": [0, 5, 7, 9, 11, 14, 20, 22, 27, 28], "iq": 6, "iri": [8, 9], "irreduc": 6, "irrelev": [5, 28, 29], "irrespect": [0, 27], "irvin": 22, "iseffici": 30, "isn": 5, "isnul": [], "isomap": 11, "issu": [1, 9, 15, 21, 30], "it_arrai": 13, "item": [0, 13, 27], "items": [21, 27], "iter": [1, 2, 4, 6, 8, 13, 14, 18, 22, 24, 29, 30], "its": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 20, 21, 22, 24, 27, 29, 30], "itself": [5, 6, 12, 19, 22, 24, 27, 28], "j": [0, 1, 2, 3, 4, 5, 6, 8, 9, 11, 12, 13, 14, 15, 16, 21, 22, 24, 26, 27, 28, 29, 30], "j1": 21, "j_": 6, "j_lasso_sk": 6, "j_ridge_sk": 6, "j_sk": 6, "jackknif": [6, 20, 27], "jacobian": [2, 13, 29], "jason": 4, "jax": [20, 27, 30], "jensen": [25, 27, 28, 29, 30], "jerom": [19, 22, 26], "ji": [12, 21], "jit": 13, "jj": [0, 5, 6, 27], "jk": [0, 1, 6, 12, 21, 27], "jl": [0, 27], "jm": 21, "jnp": 13, "job": [2, 8, 10, 15], "join": [0, 4, 6, 7, 9, 27], "joint": [4, 5], "judg": [13, 29], "judgement": 6, "julia": [20, 21, 22], "jump": 24, "junk": 4, "jupit": 27, "jupyt": [0, 15, 16, 20, 22, 26, 27], "just": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 20, 24, 27, 28, 29, 30], "justif": 0, "justifi": [3, 10], "k": [0, 1, 3, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 20, 21, 22, 24, 25, 27, 28, 29, 30], "k0": 7, "k1": 7, "kaggl": [6, 22], "kappa_d": 24, "karl": [25, 27], "karush": 8, "katrin": [25, 27], "keep": [0, 1, 4, 5, 6, 11, 13, 14, 15, 18, 21, 22, 27, 28, 29, 30], "keepdim": [1, 6, 10, 21], "kei": [1, 3, 6, 12], "kept": [4, 6, 14], "kera": [0, 4, 20, 22, 27], "kernel": [0, 1, 3, 20, 27, 28], "kernel_regular": [1, 3], "kernel_s": 4, "kernelpca": 11, "kev": [0, 27], "kevin": [26, 27], "keyword": [18, 21, 27], "kfold": 6, "kg": 1, "ki": 21, "kick": [1, 13, 30], "kiener": 2, "kilomet": [6, 28], "kind": [0, 2, 3, 4, 8, 12, 13, 14, 27, 28], "kingma": 30, "kj": [6, 12, 21, 28, 30], "kjm": [20, 27], "kkt": 8, "kl": 24, "km": [12, 27], "kmean": 14, "kmeanspoint": 14, "kn_k": 14, "know": [0, 1, 2, 5, 6, 8, 13, 15, 16, 17, 19, 20, 27, 28, 29], "knowledg": [0, 20, 27], "known": [1, 3, 4, 5, 6, 7, 8, 9, 12, 18, 21, 22, 24, 26, 28, 30], "kondev": [0, 27], "kp": 24, "kpca": 11, "kroneck": 14, "kuhn": 8, "kvalsund": [25, 27], "kwown": [0, 27], "l": [0, 1, 2, 3, 5, 6, 7, 8, 10, 11, 12, 13, 21, 22, 24, 27, 29, 30], "l0": 7, "l1": [0, 1, 3, 7, 27], "l1_l2": [1, 3], "l1regl": 5, "l2": [1, 3], "l_": 21, "l_1": 7, "l_2": [7, 13, 29, 30], "l_j": 12, "la": 13, "la_i": 12, "la_k": 12, "lab": [20, 22, 27], "label": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 12, 13, 14, 15, 20, 21, 22, 24, 27, 28, 29, 30], "labelencod": [7, 10], "labels": [6, 8, 9], "labels_shuffl": [0, 1, 28], "laboratori": 23, "lack": [0, 27], "lagari": 2, "lagrang": [8, 11], "lam": 18, "lambda": [0, 1, 2, 3, 5, 6, 7, 8, 10, 12, 13, 17, 18, 19, 22, 24, 27, 28, 29, 30], "lambda_": 11, "lambda_0": 11, "lambda_1": [5, 8, 11, 28, 29], "lambda_2": [8, 11], "lambda_i": [8, 11], "lambda_iy_i": 8, "lambda_jy_iy_j": 8, "lambda_k": 8, "lambda_n": [5, 8, 28, 29], "lamda": 1, "land": 8, "landmark": 8, "landscap": [13, 18, 29, 30], "langl": [0, 6, 11, 24, 27, 28], "languag": [0, 1, 4, 8, 20, 21, 22, 26, 27], "lapack": [21, 27], "laplac": 5, "laptop": [15, 20], "larg": [0, 1, 2, 4, 5, 6, 8, 9, 10, 11, 13, 18, 20, 21, 22, 24, 26, 27, 28, 29, 30], "larger": [0, 3, 5, 6, 8, 10, 11, 13, 17, 24, 27, 28, 29, 30], "largest": [4, 8, 11], "lasso": [0, 7, 20, 27, 30], "lasso_sk": 6, "last": [0, 1, 3, 4, 5, 6, 7, 8, 12, 16, 17, 19, 21, 22, 24, 25, 27, 29], "latent": 4, "latent_dim": 4, "latent_point": 4, "latent_space_value_rang": 4, "later": [0, 1, 4, 7, 8, 12, 13, 14, 15, 20, 22, 27, 30], "latest": [4, 15, 20], "latest_checkpoint": 4, "latex": 27, "latter": [0, 3, 6, 7, 8, 11, 13, 21, 24, 27, 28, 29, 30], "lattic": 12, "law": 0, "layer": [0, 4, 13, 27, 30], "lbfg": [7, 9, 10], "lcc": [5, 6], "lda": 11, "ldot": [0, 6, 11, 22, 27], "le": [5, 7, 10, 13, 17, 24, 28, 29, 30], "lead": [0, 1, 3, 5, 6, 7, 8, 9, 10, 11, 12, 13, 16, 17, 21, 24, 27, 28, 29, 30], "leaf": 9, "leaki": 1, "leakyrelu": 4, "lear": [13, 29], "learn": [3, 4, 5, 6, 7, 8, 9, 10, 12, 21, 25, 26], "learnabl": 3, "learner": 10, "learnig": 27, "learning_r": [8, 10], "learning_rate_init": [0, 1, 27], "learning_schedul": [13, 30], "learnt": 22, "least": [0, 7, 8, 10, 11, 17, 18, 20, 21, 24], "leat": [13, 30], "leav": [0, 1, 3, 5, 6, 9, 11, 27, 29], "lectur": [0, 1, 5, 10, 11, 12, 13, 20, 21, 22, 23, 25, 26, 28], "lecturenot": [0, 20, 22, 26, 27], "left": [0, 1, 2, 3, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 16, 17, 19, 21, 22, 24, 27, 28, 29, 30], "leftarrow": [8, 12], "legend": [0, 2, 3, 4, 5, 6, 7, 8, 9, 10, 13, 15, 27, 28, 29, 30], "leinonen": 27, "len": [0, 1, 2, 3, 4, 5, 6, 8, 9, 10, 11, 12, 16, 17, 21, 27, 28, 29, 30], "length": [0, 1, 3, 4, 8, 9, 13, 16, 20, 27, 28, 29, 30], "length_of_sequ": 4, "leq": [0, 5, 7, 8, 13, 14, 24, 27, 28, 29, 30], "less": [0, 1, 3, 4, 5, 6, 8, 9, 13, 20, 24, 27, 28, 29, 30], "lessen": 1, "let": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 19, 21, 24, 27, 28, 29, 30], "letter": [0, 16, 21, 24, 27, 28], "level": [0, 1, 5, 6, 9, 20, 21, 22, 23, 25, 27], "li": [8, 11], "lib": [], "liberti": 30, "liblinear": 10, "librari": [0, 1, 2, 3, 4, 5, 6, 9, 10, 11, 21, 22, 24, 26, 28, 29, 30], "licens": [0, 1, 20, 22, 27], "lie": [0, 6, 11, 24, 27, 28], "life": [0, 1, 8, 12, 27], "lifetim": [13, 30], "like": [0, 1, 2, 3, 4, 5, 6, 7, 9, 10, 11, 12, 13, 15, 16, 20, 21, 22, 24, 27, 28, 29, 30], "likelihood": [0, 1, 5, 9, 27, 28], "lim_": 24, "limit": [0, 5, 6, 8, 12, 21, 22, 27, 28], "lin_clf": 8, "lin_model": [], "lin_reg": 9, "linalg": [0, 2, 5, 6, 8, 11, 13, 17, 21, 24, 27, 28, 29, 30], "line": [0, 3, 6, 8, 11, 13, 15, 16, 27, 29, 30], "line1": 8, "line2": 8, "line3": 8, "line_model": 15, "line_ms": 15, "line_predict": 15, "linear": [1, 3, 5, 6, 7, 9, 10, 11, 12, 16, 17, 18, 19, 20, 22, 24, 30], "linear_model": [0, 5, 6, 7, 8, 9, 10, 11, 13, 15, 16, 19, 27, 28, 29, 30], "linear_regress": 6, "linearli": [5, 28, 29], "linearloc": [6, 13, 29, 30], "linearregress": [0, 6, 7, 9, 15, 16, 19, 27, 28, 30], "linearsvc": 8, "lineat": 29, "liner": [1, 3], "linerar": 10, "linewidth": [0, 2, 4, 6, 8, 9, 10], "link": [0, 4, 9, 12, 15, 20, 22, 23, 25, 27], "linlag": 5, "linpack": [21, 27], "linreg": [0, 27], "linspac": [0, 2, 3, 4, 6, 8, 9, 10, 13, 16, 17, 19, 21, 24, 27, 28, 30], "linu": 4, "linux": [0, 1, 20, 22, 27], "liquid": [0, 27], "list": [1, 2, 3, 4, 9, 15, 20, 22, 27, 30], "listedcolormap": [9, 10], "literatur": [1, 7, 14, 26], "littl": [1, 3, 9, 12, 30], "live": [8, 16], "ll": [0, 18, 24, 27, 28], "lle": [0, 28], "lloyd": [4, 14], "lmb": [0, 2, 5, 6, 28, 29, 30], "lmbd": [0, 1, 3, 27], "lmbd_val": [0, 1, 3, 27], "lmbda": [13, 29, 30], "ln": [1, 13, 29], "load": [1, 4, 6, 7, 9, 10], "load_boston": [], "load_breast_canc": [1, 7, 9, 10, 11], "load_data": [3, 4], "load_digit": [1, 3], "load_iri": [8, 9], "loc": [3, 6, 7, 8, 9, 10, 27], "local": [0, 1, 3, 7, 12, 13, 15, 28, 29, 30], "locat": [2, 3, 8, 15], "log": [0, 1, 2, 4, 5, 6, 7, 9, 10, 11, 13, 15, 21, 22, 27], "log10": [0, 5, 6, 28, 29, 30], "log_": [0, 27], "log_clf": 10, "logarithm": [0, 5, 7, 17, 21, 27], "logbook": 22, "logic": [0, 1, 9, 27], "login": 15, "logist": [0, 1, 2, 8, 9, 10, 11, 12, 13, 20, 28, 29, 30], "logisticregress": [7, 9, 10, 11], "logit": 7, "logreg": [7, 9, 10, 11], "logspac": [0, 1, 3, 5, 6, 27, 28, 29, 30], "long": [0, 1, 3, 4, 12, 13, 27, 29, 30], "longer": [2, 3, 8, 10, 14, 21, 24, 27, 30], "loocv": 6, "look": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 13, 15, 16, 19, 21, 22, 24, 27, 28, 29, 30], "loop": [1, 4, 6, 10, 12, 14, 16, 17, 18, 20, 21, 27], "lose": 1, "loss": [0, 1, 3, 4, 5, 6, 7, 8, 10, 11, 13, 18, 21, 22, 27], "loss_fil": 4, "lossfil": 4, "lost": 4, "lot": [1, 4, 6, 16, 30], "low": [0, 6, 9, 10, 11, 22, 27, 28], "lower": [0, 1, 3, 6, 9, 10, 16, 21, 28], "lowercas": [21, 27], "lowest": [9, 13, 24, 30], "lr": [1, 3, 4, 10], "lstat": [], "lstm": 4, "lstm_2layer": 4, "lstsq": [0, 27, 28], "lt": 6, "lu": [0, 5, 27, 28, 29], "lubksb": 21, "luckili": 2, "ludcmp": 21, "lux": 21, "lvert": 1, "lw": [0, 27], "m": [0, 1, 2, 3, 5, 6, 8, 9, 10, 11, 12, 13, 15, 21, 24, 25, 26, 27, 28, 29, 30], "m_": [9, 12], "m_1": 14, "m_h": [0, 27], "m_k": 14, "m_l": 12, "m_n": [0, 27], "m_p": [0, 27], "m_t": [13, 30], "ma": 11, "machin": [1, 3, 4, 5, 6, 7, 9, 10, 11, 12, 15, 16, 21, 26, 28], "machinelearn": [0, 6, 16, 20, 22, 23, 25, 26, 27, 28, 29], "mackai": 26, "made": [0, 1, 3, 4, 5, 6, 7, 9, 11, 12, 22, 27, 28], "mae": [0, 27], "magic": 4, "magnitud": [1, 6, 7, 13, 28, 30], "mai": [0, 1, 2, 3, 5, 6, 7, 8, 9, 11, 12, 13, 19, 20, 21, 22, 24, 27, 28, 29, 30], "mail": [23, 25], "main": [0, 1, 3, 4, 5, 6, 7, 9, 21, 22, 26, 28, 29, 30], "mainli": [0, 5, 6, 7, 9, 27, 28], "maintain": [6, 30], "major": [1, 6, 9, 10, 13, 21, 27, 29, 30], "make": [1, 2, 3, 4, 5, 6, 7, 8, 11, 12, 13, 15, 16, 18, 20, 21, 22, 24, 26, 27, 29, 30], "make_axes_locat": 6, "make_moon": [8, 9, 10], "make_pipelin": [0, 6, 10, 28], "makedir": [0, 6, 7, 9, 27], "malcondit": 21, "malign": [1, 7, 9], "mammographi": 5, "manag": [0, 2, 3, 15, 20, 22, 27], "mandatori": [25, 27], "mani": [0, 1, 3, 4, 5, 6, 7, 8, 9, 11, 13, 14, 15, 16, 17, 18, 20, 21, 22, 24, 26, 27, 28, 29, 30], "manifold": 11, "manner": 3, "manual": [6, 28, 30], "map": [0, 1, 2, 6, 7, 8, 11, 12, 14, 24, 27], "marc": 28, "margin": [0, 5, 8], "marit": [0, 27], "mark": 27, "marker": [7, 21, 27], "markov": [20, 27], "marsaglia": 24, "mass": [0, 1, 5, 13, 28, 29, 30], "massag": [0, 27], "masses2016": [0, 27], "masses2016ol": [0, 27], "masses2016tre": 0, "masseval2016": [0, 27], "master": [23, 25], "mat": [20, 27], "mat1100": [20, 27], "mat1110": [20, 27], "mat1120": [20, 27], "match": [1, 4, 5, 13, 14, 15, 28, 29, 30], "materi": [4, 5, 7, 13, 15, 21, 23, 25], "math": [3, 7, 12, 13, 21, 24, 26, 27, 30], "mathbb": [0, 4, 5, 6, 7, 8, 11, 12, 13, 14, 17, 19, 21, 22, 24, 27, 28, 29, 30], "mathbf": [0, 5, 6, 7, 8, 13, 19, 21, 22, 27, 28, 29, 30], "mathcal": [1, 5, 6, 7, 13, 22], "matheemat": 3, "mathemat": [0, 6, 11, 12, 13, 20, 21, 24, 26, 27, 30], "mathemati": 27, "mathrm": [0, 1, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 17, 18, 19, 22, 24, 27, 28, 29, 30], "matmul": [1, 2, 5], "matnat": 26, "matplotlib": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 17, 19, 20, 21, 22, 24, 27, 28, 29, 30], "matric": [0, 1, 3, 4, 6, 7, 8, 11, 13, 16, 17, 20, 28, 29], "matrix": [0, 2, 3, 4, 6, 7, 8, 10, 13, 17, 18, 19, 22, 24], "matshow": 1, "matter": [2, 3, 13, 28, 29, 30], "max": [0, 1, 2, 3, 4, 9, 10, 12, 13, 25, 27, 29, 30], "max_depth": [0, 9, 10], "max_diff": 2, "max_diff1": 2, "max_diff2": 2, "max_it": [0, 1, 8, 13, 27], "max_iter": 14, "max_leaf_nod": 10, "max_sampl": 10, "maxdegre": [0, 6, 10, 28], "maxdepth": 10, "maxim": [1, 4, 5, 7, 8, 11], "maximum": [0, 2, 3, 5, 7, 8, 9, 10, 13, 14, 27, 28, 29, 30], "maxpolydegre": [5, 6, 28, 29, 30], "maxpooling2d": 3, "mbox": [5, 6, 28, 29], "mcculloch": 12, "md": 11, "mdoel": 4, "mean": [1, 2, 3, 4, 5, 6, 7, 9, 10, 11, 12, 13, 14, 15, 17, 18, 19, 20, 21, 22, 24, 27, 30], "mean_absolute_error": [0, 27], "mean_divisor": 14, "mean_i": 24, "mean_matrix": 14, "mean_squared_error": [0, 4, 6, 7, 10, 15, 19, 27, 28], "mean_squared_log_error": [0, 27], "mean_vector": 14, "mean_x": 24, "meaning": [0, 4, 7, 27], "meansquarederror": [0, 27], "meant": [3, 7, 10, 13], "measur": [0, 1, 2, 5, 6, 9, 11, 12, 14, 16, 18, 19, 22, 24, 27, 28, 30], "mechan": [0, 4, 24, 27, 30], "median": [0, 27, 28, 30], "medicin": 12, "medium": [4, 8, 13, 30], "medv": [], "meet": [0, 25], "mehta": [0, 27, 28, 29], "memori": [3, 4, 11, 12, 13, 18, 21, 30], "mention": [0, 12, 13, 22, 24, 27, 29, 30], "mere": [0, 22], "meshgrid": [2, 5, 6, 8, 9, 10, 11], "mess": 15, "messag": [5, 13], "messi": 2, "met": [0, 3, 8, 28], "meteorolog": 9, "meter": [6, 28], "method": [0, 1, 2, 3, 4, 5, 7, 8, 11, 12, 14, 15, 16, 17, 18, 19, 20, 21, 24, 26, 28], "metion": 6, "metric": [0, 1, 3, 6, 7, 9, 10, 14, 15, 19, 27, 28], "metropoli": [20, 27], "mev": [0, 24, 27], "mgd": [13, 30], "mglearn": [20, 27], "mgrid": 13, "mhjensen": [], "mi": 10, "mia": [25, 27], "microsoft": 26, "mid": 1, "midel": 4, "midnight": 15, "midpoint": 9, "might": [0, 1, 2, 4, 6, 9, 13, 15, 17, 18, 28, 29, 30], "migth": 17, "mild": 9, "millimet": [6, 28], "million": [0, 27, 28, 30], "mimic": 12, "min": [0, 2, 5, 8, 9, 29], "min_": [0, 2, 5, 14, 17, 27, 28, 29], "min_samples_leaf": 9, "mind": [0, 6, 13, 15, 18, 27, 28, 29, 30], "mindboard": 4, "mine": [20, 27], "mini": [1, 11, 12, 13, 29], "minibatch": [1, 11, 13], "minibathc": [13, 30], "miniforge3": [], "minim": [0, 1, 2, 3, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 16, 17, 28, 29, 30], "minima": [0, 1, 7, 13, 27, 29, 30], "minimum": [0, 1, 2, 6, 8, 9, 11, 13, 28, 29, 30], "minmaxscal": [0, 28, 30], "minor": 24, "minst": 1, "minu": 7, "mirjalili": 27, "mirror": 9, "misc": 6, "misclassif": [8, 9, 10], "misclassifi": [8, 10], "miser": 0, "mismatch": 1, "miss": [7, 10], "mistak": 4, "mit": 26, "mix": [1, 2, 27], "mixtur": [13, 30], "mk": [9, 21], "mkdir": [0, 6, 7, 9, 27], "ml": [0, 1, 10, 13, 21, 22, 28, 29, 30], "mlab": 24, "mle": [5, 7], "mlp": 1, "mlpclassifi": 1, "mlpregressor": [0, 27], "mm": 21, "mml": 28, "mn": [12, 24], "mnist": [1, 11], "mod": 24, "mode": [23, 25, 27], "model": [2, 3, 5, 7, 8, 9, 10, 11, 13, 14, 16, 18, 19, 20, 22, 24, 26, 28, 29, 30], "model_select": [0, 1, 3, 5, 6, 7, 9, 10, 11, 15, 16, 17, 19, 27, 28, 29, 30], "moder": 10, "modern": [0, 6, 7, 20, 27], "modif": [2, 12, 13, 30], "modifi": [0, 1, 3, 5, 7, 8, 10, 12, 13, 27, 28, 29, 30], "modul": [0, 16, 21, 27], "modular": 24, "modulo": 24, "moe": [11, 28], "moment": [5, 6, 13, 24], "mondai": [25, 27], "monitor": [13, 30], "monoton": [5, 12, 24], "mont": [0, 6, 20, 24, 26, 27], "montli": 16, "moor": [5, 6], "more": [0, 1, 2, 4, 5, 7, 8, 9, 10, 11, 12, 13, 14, 16, 17, 19, 20, 24], "moreov": [0, 3], "morten": [25, 27, 28, 29, 30], "mortenhj": 27, "most": [0, 1, 3, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 20, 22, 24, 27, 28, 29, 30], "mostli": [1, 11, 18], "motion": [0, 13, 30], "motiv": [1, 4], "move": [0, 4, 5, 6, 7, 9, 12, 13, 14, 15, 16, 22, 24, 28, 29, 30], "mpl": [7, 27], "mpl_toolkit": [2, 6, 13, 29, 30], "mplot3d": [2, 6, 13, 29, 30], "mplregressor": 1, "mse": [0, 4, 5, 6, 9, 10, 15, 16, 17, 19, 22, 27, 28, 29, 30], "mse_simpletre": 10, "mselassopredict": [5, 29], "mselassotrain": [5, 29], "mseownridgepredict": [6, 28, 29, 30], "msepredict": [5, 29], "mseridgepredict": [0, 5, 6, 28, 29, 30], "msetrain": [5, 29], "msle": [0, 27], "mt": [7, 12], "mu": [0, 6, 11, 13, 24, 27, 30], "mu0": 24, "mu1": 24, "mu2": 24, "mu_": [6, 24, 28, 30], "mu_i": [6, 28, 30], "mu_n": 11, "mu_x": 24, "much": [0, 1, 2, 3, 4, 5, 6, 8, 9, 10, 11, 12, 13, 15, 21, 22, 24, 27, 28, 29, 30], "multi": [0, 1, 3, 7, 20, 27], "multiclass": [1, 7], "multidimension": [11, 12, 27], "multilay": 1, "multinomi": 7, "multipl": [2, 4, 5, 6, 7, 12, 13, 15, 24, 28, 29, 30], "multipli": [3, 5, 6, 11, 13, 18, 21, 24, 28, 29, 30], "multiplum": 8, "multivari": [0, 2, 10, 11, 20, 24, 27], "multivariate_norm": [11, 14], "multpli": 16, "murphi": [11, 26, 27], "must": [1, 2, 5, 6, 8, 10, 12, 13, 14, 15, 24, 28, 29, 30], "mutat": 7, "mutual": [1, 3, 6, 13], "mx_": 24, "my": 27, "myenv": [], "myriad": [0, 20, 27], "mz1": 24, "mz2": 24, "m\u00f8svatn": 6, "n": [0, 1, 2, 3, 4, 5, 6, 7, 8, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 21, 22, 24, 27, 28, 29, 30], "n1": 21, "n2": 21, "n_": [1, 2, 3, 8, 12, 24], "n_0": [12, 24], "n_boostrap": [6, 10], "n_bootstrap": 6, "n_categori": [1, 3], "n_cluster": 14, "n_compon": 11, "n_epoch": [13, 30], "n_estim": 10, "n_examples_to_gener": 4, "n_featur": [1, 18], "n_filter": 3, "n_hidden": 2, "n_hidden_neuron": [0, 1, 27], "n_i": 24, "n_input": [0, 1, 3, 28], "n_instanc": 9, "n_iter": 30, "n_job": 10, "n_k": 14, "n_l": [12, 24], "n_layer": 1, "n_m": 9, "n_neuron": 1, "n_neurons_connect": 3, "n_neurons_layer1": 1, "n_neurons_layer2": 1, "n_point": 14, "n_sampl": [6, 8, 9, 10, 14, 18], "n_split": 6, "n_step": 4, "n_t": 2, "n_x": 2, "nabla": [1, 13, 29, 30], "nabla_": [2, 13, 29, 30], "nabla_w": [13, 30], "nag": [13, 30], "naimi": [0, 27], "naiv": 7, "naive_kmean": 14, "name": [0, 1, 3, 4, 5, 6, 7, 8, 9, 10, 12, 13, 14, 15, 18, 20, 21, 22, 24, 25, 27, 28, 29, 30], "narrow": [13, 30], "nation": [1, 5], "nativ": [20, 27], "natur": [0, 1, 4, 8, 9, 12, 13, 22, 24, 26, 27, 29, 30], "navier": 12, "navig": 15, "nb": 24, "nb_": 21, "nbconvert": 27, "nd": 14, "ndarrai": 6, "ne": [9, 10, 21, 24, 28, 29], "nearest": [1, 3, 6, 11], "nearli": [13, 29], "neat": 27, "neccesari": 6, "necess": 2, "necessari": [0, 1, 3, 4, 8, 14, 18, 27], "necessarili": [0, 4, 11, 24, 27], "necesserali": 5, "neck": 7, "need": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 18, 19, 21, 24, 28, 29, 30], "neg": [0, 1, 3, 5, 6, 7, 10, 13, 21, 24, 27, 29], "neg_mean_squared_error": 6, "neglect": [24, 30], "neglig": 24, "neighbor": [3, 6, 11], "neither": [4, 13], "neq": [13, 14, 24, 29], "nervou": 12, "nest": [9, 12], "nesterov": [13, 30], "net": [2, 4, 12], "netlib": [21, 27], "network": [0, 9, 13, 20, 26, 28, 30], "neural": [0, 13, 20, 26, 28, 30], "neural_network": [0, 1, 2, 27], "neuralnetwork": 1, "neuron": [1, 2, 3, 4, 12], "neutral": [0, 27], "neutron": [0, 27], "never": [1, 4, 6, 9, 24], "new": [0, 1, 2, 3, 5, 6, 7, 8, 9, 10, 11, 13, 14, 15, 17, 21, 27, 28, 29, 30], "new_chang": [13, 30], "new_hobbit": 27, "newaxi": [0, 3, 6, 9], "newli": [0, 27], "newton": [1, 7, 8, 13, 24], "next": [0, 1, 2, 3, 4, 5, 6, 8, 9, 13, 14, 15, 16, 27, 28, 29], "next_guess": 13, "next_input": 4, "ng": 1, "ni": 14, "nice": [0, 1, 5, 11, 27, 28, 29], "nicer": [18, 30], "niter": [13, 29, 30], "nitric": [], "nlambda": [0, 5, 6, 28, 29, 30], "nlp": 26, "nm": 24, "nm_n": [0, 27], "nmse": 6, "nn": [2, 5, 6, 12, 21, 27], "nn_model": 1, "nnmin": 2, "node": [1, 3, 9, 10, 12], "nois": [0, 4, 5, 6, 8, 9, 10, 13, 18, 19, 22, 27, 28, 29, 30], "noise_dimens": 4, "noisi": [1, 6, 22], "non": [0, 1, 3, 5, 6, 7, 9, 10, 11, 12, 13, 14, 18, 21, 24, 27, 28, 29, 30], "nondifferenti": 30, "none": [0, 1, 2, 4, 5, 9, 10, 13, 24, 27, 28], "nonlinear": [3, 6, 8, 9, 11, 12], "nonneg": [6, 9, 13, 29], "nonparametr": 6, "nonsens": 24, "nonsingular": 21, "nonumb": [3, 7, 8, 13, 21, 30], "nor": [1, 4, 13], "norm": [0, 1, 5, 6, 8, 11, 13, 18, 27, 28, 29, 30], "normal": [3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 16, 17, 18, 19, 20, 21, 22, 24, 27, 28, 29, 30], "normali": [21, 27], "norwai": [6, 22, 27, 29, 30], "notat": [0, 2, 5, 6, 13, 14, 24, 27, 28, 29, 30], "note": [0, 1, 2, 3, 4, 5, 6, 7, 8, 11, 12, 13, 14, 15, 16, 18, 19, 20, 21, 24, 26, 27, 30], "notebook": [0, 1, 3, 9, 15, 16, 20, 22, 27], "noth": [1, 2, 5, 8, 12, 14, 24, 28, 29], "notic": [4, 5, 12, 13, 21, 24, 27, 30], "notion": 3, "novel": [3, 6, 10, 27], "novemb": [1, 25, 27], "now": [0, 2, 4, 5, 6, 7, 8, 10, 11, 12, 14, 15, 16, 19, 20, 21, 22, 24, 27, 28], "nowadai": [0, 1, 3, 9, 20, 27], "nox": [], "np": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 13, 14, 15, 16, 17, 18, 19, 21, 24, 27, 28, 29, 30], "npr": 2, "nsampl": 6, "nt": 2, "nu": 24, "nuclear": [5, 28, 29], "nuclei": [0, 24, 27], "nucleon": [0, 27], "nucleu": [0, 27], "num": 4, "num_coordin": 2, "num_hidden_neuron": 2, "num_it": [2, 18], "num_neuron": 2, "num_neurons_hidden": 2, "num_point": 2, "num_tre": 10, "num_valu": 2, "number": [1, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 16, 17, 18, 19, 21, 22, 23, 25, 27, 29], "numberid": 7, "numberparamet": 3, "numer": [0, 5, 6, 9, 10, 11, 12, 13, 20, 21, 26, 27, 28, 29, 30], "numpi": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 22, 24, 28, 29, 30], "nunmpi": [5, 28], "nve_frngahw": 29, "nx": 2, "ny": 24, "o": [0, 1, 4, 5, 6, 7, 8, 9, 11, 21, 25, 26, 27, 28, 29, 30], "obei": [6, 11, 13, 28, 30], "object": [0, 1, 4, 8, 10, 15, 19, 21, 27, 30], "obliqu": [5, 28, 29], "observ": [0, 1, 3, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 24, 27, 29, 30], "obtain": [0, 1, 5, 6, 7, 8, 9, 10, 12, 13, 14, 17, 21, 22, 24, 27, 28, 29, 30], "obviou": [5, 6, 11, 24, 28, 29], "obviouli": 27, "obvious": [0, 4, 5, 6, 21, 27], "oc": [28, 29], "occupi": [], "occur": [0, 6, 8, 9, 21, 24, 27], "octob": [25, 27], "od": 0, "odd": [0, 3, 7, 27, 28, 30], "odenum": 2, "odesi": 2, "oen": 0, "off": [1, 3, 4, 5, 9, 13, 24], "offer": [6, 11, 20, 21, 23, 25, 27], "offic": [25, 27], "offici": [23, 27], "often": [0, 1, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 18, 20, 21, 22, 24, 27, 28, 29, 30], "ofter": [21, 27], "ol": [0, 13, 17, 19, 28, 30], "old": [1, 5, 10, 13, 15, 18], "ols_paramet": 16, "ols_sk": 6, "ols_svd": 6, "olsbeta": 29, "olstheta": [0, 5], "omega": [2, 3, 6], "omega_0": 3, "omit": [0, 5, 27, 28, 29], "onc": [1, 6, 9, 11, 13, 30], "one": [0, 1, 3, 4, 5, 6, 7, 8, 9, 10, 11, 13, 14, 15, 19, 20, 21, 22, 24, 25, 27, 28, 30], "onehot": 1, "onehot_vector": 1, "onehotencod": 9, "ones": [0, 2, 5, 6, 8, 9, 10, 11, 13, 16, 18, 21, 22, 27, 28, 29, 30], "ones_lik": 4, "ong": 28, "onl": 3, "onli": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 17, 18, 21, 22, 24, 27, 28, 29, 30], "onlin": [11, 15, 23], "onto": [5, 11, 28, 29], "open": [0, 1, 4, 6, 7, 9, 15, 20, 22, 23, 25, 27], "oper": [0, 1, 3, 5, 6, 10, 11, 12, 13, 15, 16, 20, 24, 27, 28, 29, 30], "operation": 24, "oplu": 24, "opmiz": [13, 30], "opportun": 0, "oppos": [6, 13], "opposit": [1, 5, 8, 28, 29], "opt": [1, 5, 22, 27, 29], "optim": [0, 2, 3, 4, 5, 6, 7, 9, 10, 11, 14, 16, 17, 19, 22], "optimis": [1, 3], "option": [0, 1, 3, 5, 6, 8, 11, 15, 18, 21, 28, 30], "optmiz": [1, 8, 13, 28], "oral": 27, "orang": 0, "order": [0, 1, 2, 3, 5, 6, 7, 8, 9, 10, 11, 12, 15, 19, 21, 22, 24, 27, 28, 29], "ordinari": [0, 2, 3, 7, 11, 13, 17, 18, 20], "oreilli": [26, 27], "org": [0, 3, 4, 16, 20, 21, 22, 26, 27, 28, 29, 30], "organ": [6, 7, 10, 21], "orient": [1, 5, 24, 28, 29], "origin": [0, 3, 5, 6, 8, 11, 12, 13, 15, 21, 27, 28, 29], "orthogn": [5, 28, 29], "orthogon": [0, 5, 6, 8, 11, 13, 21, 27, 28, 29], "orthonorm": [5, 28, 29], "os": [25, 27], "oscar": 1, "oscil": [3, 13, 30], "oskar": 27, "oskarlei": 27, "osl": 18, "oslo": [0, 20, 22, 23, 25, 27, 28, 29, 30], "osx": [0, 20, 22, 27], "other": [0, 1, 2, 3, 5, 6, 7, 8, 10, 13, 14, 16, 20, 23, 24, 25, 26, 28, 29, 30], "otherwis": [0, 1, 4, 7, 13, 21, 27, 30], "ouput": [5, 7, 12], "our": [1, 2, 3, 6, 7, 8, 9, 10, 12, 14, 15, 16, 17, 18, 19, 20, 21, 24, 30], "ourmodel": 0, "ourselv": [0, 5, 6, 8, 11, 13, 27, 28, 29], "out": [0, 1, 2, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 15, 16, 20, 21, 22, 24, 27, 28, 30], "out_fil": 9, "outcom": [0, 7, 9, 10, 12, 24, 28], "outdoor": 9, "outer": [6, 12, 13], "outfil": 4, "outlier": [0, 8, 27, 28, 30], "outlin": [6, 10, 11], "outlook": 9, "outperform": [10, 30], "output": [0, 1, 3, 4, 5, 6, 7, 8, 9, 10, 12, 13, 21, 22, 24, 27, 28, 29, 30], "output_bia": 1, "output_bias_gradi": 1, "output_shap": 4, "output_weight": 1, "output_weights_gradi": 1, "outputlayer1": 12, "outputlayer2": 12, "outsid": 4, "over": [0, 1, 3, 4, 5, 6, 9, 10, 12, 13, 15, 16, 21, 22, 27, 28, 29, 30], "over1": [13, 30], "overal": [1, 10, 30], "overcast": 9, "overcom": [12, 13], "overdetermin": [0, 27], "overfit": [0, 1, 3, 6, 9, 10, 13, 30], "overflow": 5, "overhead": 12, "overlap": [3, 7, 8, 9], "overlin": [0, 5, 6, 9, 10, 11, 14, 21, 27, 28, 30], "overshoot": 30, "overst": 0, "overtrain": 4, "overview": 3, "own": [4, 5, 6, 8, 12, 13, 16, 18, 20, 21, 29, 30], "owner": [], "ownmsepredict": 0, "ownmsetrain": 0, "ownridgebeta": 28, "ownridgetheta": [0, 6, 28, 29, 30], "ownypredictridg": 0, "ownytilderidg": 0, "oxid": [], "p": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 13, 14, 15, 16, 17, 18, 19, 21, 24, 27, 28, 29, 30], "p0": 2, "p1": 2, "p_": [2, 4, 8, 9], "p_hidden": 2, "p_i": [5, 24], "p_j": 24, "p_n": 24, "p_output": 2, "p_x": 24, "pack": [0, 27], "packag": [0, 1, 3, 4, 5, 8, 11, 13, 15, 20, 22, 24, 28, 29, 30], "packtpub": 27, "packtpublish": 27, "pad": [3, 4], "page": [0, 20, 27, 29, 30], "pai": [0, 1, 9, 13, 15, 30], "pair": [0, 2, 3, 9, 20, 24, 27], "paltform": 15, "panda": [0, 4, 5, 6, 7, 9, 11, 20, 22, 29, 30], "panel": 27, "paper": 1, "paradigm": [0, 27], "parallel": [10, 13, 20, 21, 27], "param": 2, "paramat": 2, "paramet": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 12, 13, 16, 17, 18, 19, 22, 24, 29], "parameter": [0, 6, 10, 27, 28], "parametr": [0, 6, 27, 28], "paramt": [3, 5], "part": [0, 1, 3, 5, 6, 10, 17, 19, 21, 23, 24, 25, 27, 28], "partial": [0, 1, 5, 6, 7, 8, 10, 11, 12, 13, 16, 24, 27, 28, 29, 30], "particip": [15, 20, 23, 25, 27], "particl": [0, 4, 13, 24, 27, 30], "particular": [0, 1, 2, 3, 5, 6, 9, 10, 11, 12, 13, 16, 22, 24, 26, 27, 28, 29, 30], "particularli": [5, 6, 8, 11, 13, 24, 28, 29, 30], "partit": [1, 4, 9], "partli": [6, 27], "partner": 15, "pass": [2, 3, 12, 14], "password": 22, "past": [10, 24, 30], "patch": [6, 24], "path": [0, 4, 6, 7, 9, 20, 27], "pathcollect": 17, "patient": 7, "patter": 4, "pattern": [0, 3, 4, 12, 26, 27], "pauli": [0, 27], "pc": [11, 15, 20], "pca": [0, 7, 20, 27, 28], "pd": [0, 4, 5, 6, 7, 9, 11, 27, 28, 29, 30], "pde": 2, "pdf": [0, 3, 4, 5, 6, 9, 15, 16, 22, 26, 27], "pedagog": [0, 27, 28], "penal": [6, 18, 28, 30], "penalti": [6, 13, 18, 22, 28, 30], "penros": [5, 6], "pentagon": [13, 29], "peopl": [1, 9, 13, 20, 30], "per": [0, 1, 6, 23, 25, 27, 30], "percentag": [10, 11, 25], "perceptron": [0, 1, 7, 27], "peregrin": 27, "perfect": [0, 1, 13, 27, 30], "perfectli": [4, 6], "perform": [0, 2, 3, 4, 5, 6, 8, 10, 11, 12, 13, 14, 16, 18, 19, 20, 21, 22, 24, 27, 28, 29, 30], "performac": 4, "perhap": [0, 5, 13, 27, 28, 29], "perimet": 1, "period": [1, 4, 24], "permiss": 15, "permut": 11, "persist": [13, 30], "person": [5, 6, 7, 16, 23, 25, 27, 28], "perspect": 26, "pertin": [12, 27], "petal": [8, 9], "peter": [26, 28], "phantom": 24, "phase": [6, 12], "phenomena": 24, "phi": 8, "phi_k": 8, "philosophi": 13, "phone": [25, 27], "photo": [4, 27], "php": 22, "phrase": [0, 27], "physic": [0, 1, 4, 7, 12, 13, 24, 25, 26, 27, 28, 29, 30], "pi": [2, 3, 5, 6, 7, 9, 12, 13, 24], "pick": [1, 9, 10, 11, 13, 14, 30], "pickl": 1, "pictur": [0, 27], "pie": [20, 27], "piec": [11, 14], "pillow": [0, 20, 22, 27], "pinv": [5, 6, 13, 22, 28, 29, 30], "pip": [0, 1, 15, 20, 22, 27], "pip3": [0, 1, 22, 27], "pipelin": [0, 6, 8, 10, 28], "pippin": 27, "pit": 4, "pitfal": [6, 28], "pitt": 12, "pixel": [1, 3, 4, 27], "pixel_height": [1, 3], "pixel_width": [1, 3], "place": [0, 4, 6, 8, 13, 15, 21, 22, 27, 29], "plai": [0, 3, 4, 5, 6, 8, 11, 18, 20, 22, 27, 28, 29], "plain": [8, 10, 12, 13, 14, 29, 30], "plan": [6, 9, 25, 26, 27], "plane": [8, 9], "plateau": [5, 29], "platform": [20, 27], "plausibl": 12, "pleas": [13, 22, 25, 27], "plenti": 1, "plethora": [3, 12], "plot": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 16, 17, 19, 20, 21, 22, 24, 27, 28, 29, 30], "plot_all_sc": [22, 28], "plot_confusion_matrix": [7, 10], "plot_count": 6, "plot_cumulative_gain": [7, 10], "plot_data": 1, "plot_dataset": 8, "plot_decision_boundari": [9, 10], "plot_import": 10, "plot_max": 4, "plot_min": 4, "plot_model": 4, "plot_numb": 4, "plot_predict": 8, "plot_regression_predict": 9, "plot_result": 4, "plot_roc": [7, 10], "plot_surfac": [2, 6, 13], "plot_train": 9, "plot_tre": [9, 10], "plt": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 17, 19, 21, 24, 27, 28, 29, 30], "plu": [0, 3, 5, 7, 18, 27, 28], "pm": 8, "pmatrix": 2, "pml": 26, "pn": 3, "png": [0, 4, 6, 7, 9, 27], "point": [0, 1, 2, 3, 5, 6, 7, 8, 9, 10, 11, 13, 14, 18, 19, 21, 22, 24, 25, 27, 28, 29, 30], "point_1": 4, "point_2": 4, "poisson": [20, 24, 27], "poli": [6, 8], "poly100_kernel_svm_clf": 8, "poly3": 0, "poly3_plot": 0, "poly_featur": [8, 9, 15], "poly_features10": 9, "poly_fit": 9, "poly_fit10": 9, "poly_kernel_svm_clf": 8, "poly_model": 15, "poly_ms": 15, "poly_predict": 15, "polydegre": [0, 5, 6, 10, 28], "polygon": [13, 29], "polym": 12, "polymi": 22, "polynomi": [0, 5, 6, 7, 8, 9, 10, 11, 15, 17, 19, 22, 27, 28], "polynomial_featur": [6, 15, 16, 17], "polynomial_svm_clf": 8, "polynomialfeatur": [0, 6, 8, 9, 15, 16, 19, 28], "polytrop": [0, 6], "pool": 3, "pool_siz": 3, "poor": [1, 13, 29, 30], "poorli": [0, 28], "popul": [0, 5, 27, 28], "popular": [0, 1, 3, 6, 7, 8, 9, 11, 12, 15, 20, 21, 22, 24, 28], "popularli": [0, 27], "portabl": 10, "portion": [11, 13, 30], "pose": [0, 4, 5, 6, 11, 24, 27], "posit": [0, 1, 2, 3, 5, 7, 8, 10, 11, 13, 14, 21, 24, 27, 28, 29, 30], "possibl": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 20, 21, 22, 24, 25, 27, 28, 29, 30], "possibli": [6, 8, 13, 22], "posterior": 5, "postpon": [0, 28], "postscript": 22, "postul": 5, "potenti": [0, 3, 5, 6, 12, 13, 28, 30], "pott": 12, "power": [0, 1, 5, 6, 8, 9, 12, 13, 27, 28, 29, 30], "pp": [5, 6, 19], "practic": [0, 5, 6, 7, 8, 16, 18, 22, 24, 28], "practition": [0, 1, 3, 27], "pre": 27, "preced": [1, 11, 12, 24], "preceed": 4, "preceq": 8, "precis": [0, 2, 5, 11, 13, 21, 22, 24, 27, 28], "pred": 6, "predicit": 0, "predict": [0, 1, 5, 6, 7, 8, 9, 10, 15, 16, 17, 19, 20, 22, 26, 27, 28, 29, 30], "predict_prob": 1, "predict_proba": [7, 10], "predictor": [0, 5, 6, 7, 9, 10, 11, 27, 28, 30], "prefer": [0, 1, 6, 8, 9, 11, 13, 15, 20, 22, 27], "prepar": [0, 6, 21, 22, 27, 28], "preprocess": [0, 4, 6, 7, 8, 9, 10, 11, 15, 16, 17, 18, 19, 22], "prerequisit": 0, "prescript": 22, "presenc": [13, 30], "present": [0, 5, 6, 7, 9, 12, 13, 21, 22, 24, 27, 28, 29, 30], "preserv": [3, 11, 21], "press": [13, 15, 26, 29], "pretrain": [1, 4], "pretti": [0, 4, 8, 9, 20, 22, 27], "prev_centroid": 14, "prevent": [13, 24, 30], "previou": [0, 1, 2, 3, 4, 5, 6, 8, 10, 11, 12, 13, 15, 16, 21, 22, 24, 28, 29, 30], "previous": [2, 3, 9, 10, 24], "price": [0, 4, 9, 13, 30], "primal": 8, "primari": [0, 7, 27], "prime": 24, "princip": [0, 5, 7, 20, 27, 28, 29], "principl": [0, 6, 7, 8, 14, 27], "print": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 13, 14, 18, 21, 24, 27, 28, 29, 30], "print_funct": [8, 9], "printout": [0, 27], "prior": [0, 5, 6, 27], "privat": 0, "prob": [1, 24], "probabilist": [0, 26, 27, 28], "probabl": [0, 1, 3, 4, 6, 7, 10, 13, 20, 27, 28, 30], "problem": [0, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 17, 19, 20, 21, 22, 24], "probml": 26, "proce": [0, 5, 6, 7, 8, 9, 10, 11, 13, 21, 27, 28], "procedur": [2, 4, 5, 6, 8, 10, 11, 13, 28, 29, 30], "proceed": 21, "process": [0, 2, 4, 6, 9, 10, 12, 13, 20, 21, 22, 24, 26, 27, 29, 30], "prod": 26, "prod_": [1, 5, 7], "produc": [0, 3, 4, 5, 6, 9, 10, 11, 12, 13, 18, 20, 21, 24, 27, 28], "product": [0, 1, 3, 5, 6, 7, 8, 12, 13, 16, 17, 20, 21, 27, 28, 30], "profess": [0, 27], "program": [0, 1, 4, 5, 6, 8, 12, 14, 15, 20, 21, 23, 24, 25, 27, 28], "programm": 21, "progress": [1, 4, 14, 30], "prohibit": 6, "project": [0, 1, 2, 3, 5, 11, 13, 15, 19, 20, 23, 28, 29, 30], "project_root_dir": [0, 6, 7, 9, 27], "promin": 12, "promis": 8, "promot": [25, 27], "prone": [9, 15], "pronounc": [13, 20, 27, 30], "proof": [0, 11, 12, 13, 27, 29], "propag": [2, 3, 13, 30], "proper": [0, 2, 6, 7], "properli": [1, 6, 8, 10, 13, 18, 22, 30], "properti": [0, 1, 3, 12, 13, 16, 21, 27], "proport": [0, 1, 5, 9, 11, 13, 24, 27, 28, 30], "propos": [1, 4, 6, 10, 22, 27, 30], "propto": [5, 13, 29, 30], "proton": [0, 27], "prove": [3, 13, 29, 30], "provid": [0, 1, 3, 4, 5, 6, 8, 9, 10, 12, 13, 20, 21, 22, 24, 27, 28, 29, 30], "proxi": [1, 13, 30], "prune": 9, "pseudo": [21, 24, 30], "pseudocod": 22, "pseudoinv": 5, "pseudoinvers": [5, 6, 22], "pseudorandom": [6, 24], "psychologi": [0, 27], "pt": 13, "public": [0, 15, 20, 27], "pull": 15, "punish": [0, 1, 27], "pure": [3, 9, 24], "purest": 9, "puriti": 9, "purpos": [0, 3, 10, 12, 14, 27], "push": 15, "put": 1, "py": 5, "pycod": 27, "pydata": 20, "pydot": 9, "pyhton2": 27, "pylab": [7, 27], "pypi": 20, "pyplot": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 17, 19, 21, 24, 27, 28, 29, 30], "pythagora": 5, "python": [1, 2, 3, 5, 6, 8, 11, 12, 13, 14, 18, 22, 24, 28, 30], "python2": [0, 22], "python3": [0, 20, 22, 27], "pytorch": [0, 20, 22, 27], "q": [5, 6, 8, 11, 24], "qp": 8, "qquad": [2, 11, 13, 21, 30], "qr": [5, 6, 21, 28, 29], "quad": [1, 13, 21], "quadrat": [0, 8, 9, 13, 27], "qualit": [4, 9, 22, 24], "qualiti": [0, 9, 20, 27, 28], "quantifi": 1, "quantil": 10, "quantit": [0, 6, 9, 22, 27], "quantiti": [0, 2, 5, 6, 7, 9, 10, 11, 12, 14, 16, 21, 24, 27, 28, 29, 30], "quantum": [4, 12, 26, 27], "quartil": [0, 28, 30], "quench": 5, "queri": 9, "question": [0, 5, 6, 9, 11, 12, 13, 22, 25, 27, 28, 30], "qugan": 4, "quick": [4, 24], "quickli": [1, 3, 9, 11, 13, 29], "quit": [1, 5, 6, 9, 10, 12, 15, 28, 29], "quot": 4, "r": [0, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 17, 20, 21, 22, 24, 28, 29, 30], "r2": [0, 5, 6, 27, 28, 29], "r2_score": [0, 27], "r2score": [0, 27], "r_1": 9, "r_2": 9, "r_j": 9, "r_m": 9, "rad": [], "rade": [], "radial": [8, 12], "radioact": 24, "radiu": [0, 1, 28, 30], "rain": 9, "ramp": 1, "ran0": 24, "ran1": 24, "ran2": 24, "ran3": 24, "rand": [0, 4, 5, 6, 9, 10, 13, 15, 19, 21, 27, 28, 29, 30], "randint": [6, 9, 13, 30], "randn": [0, 1, 2, 5, 6, 9, 11, 13, 15, 18, 27, 28, 29, 30], "random": [0, 1, 2, 3, 4, 5, 6, 8, 9, 13, 14, 15, 16, 17, 18, 19, 20, 21, 27, 28, 29, 30], "random_forest_model": 10, "random_index": [13, 30], "random_indic": [1, 3], "random_st": [7, 8, 9, 10, 11], "randomforestclassifi": 10, "randomli": [1, 6, 9, 13, 14, 18, 29, 30], "rang": [0, 1, 2, 3, 4, 5, 6, 7, 9, 10, 11, 12, 13, 14, 18, 19, 21, 24, 27, 28, 29, 30], "rangl": [0, 6, 11, 24, 27, 28], "rangle_x": 24, "rank": [5, 28, 29], "rankdir": 4, "raphson": [1, 8, 13], "rapidli": 0, "rare": [1, 13, 30], "raschka": [27, 28], "rasckha": 27, "rashcka": [29, 30], "rate": [1, 2, 3, 4, 8, 9, 10, 12, 13, 18, 29], "rather": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 21, 24, 27, 28, 29, 30], "ratio": [4, 7, 9, 10, 11], "rational": [0, 27], "ravel": [5, 6, 7, 8, 9, 10, 11, 13, 21], "raw": 3, "rbf": [8, 11, 12], "rbf_kernel_svm_clf": 8, "rbf_pca": 11, "rc": 24, "rcond": [0, 27, 28], "rcparam": [1, 3, 7, 8, 9, 10, 24, 27], "re": [2, 4, 13, 15, 29], "reach": [1, 4, 5, 6, 9, 10, 12, 13, 14, 29, 30], "read": [0, 2, 3, 4, 5, 6, 7, 8, 11, 12, 16, 17, 21, 22, 24, 26, 29], "read_csv": [0, 6, 7, 9], "read_fwf": [0, 27], "reader": [0, 6, 21, 24, 27, 28, 30], "readi": [0, 1, 5, 6, 8, 10, 11, 12, 21, 27], "readili": 1, "readm": 15, "readthedoc": 20, "real": [0, 1, 4, 7, 10, 11, 12, 16, 18, 19, 21, 28], "real_loss": 4, "real_output": 4, "realist": [8, 27], "realiti": 24, "realiz": [1, 12], "realli": [0, 1, 27], "rearrang": [13, 30], "reason": [0, 1, 3, 4, 10, 13, 26, 27, 29, 30], "reassign": 1, "recal": [5, 6, 9, 10, 11, 12, 21, 24, 27, 28, 29, 30], "recast": 3, "receiv": [1, 3, 10, 12, 24], "recent": [0, 6, 13, 26, 30], "recept": [3, 12], "receptive_field": 3, "recip": [0, 6, 7, 21, 22, 27, 28], "reciproc": 5, "recogn": [0, 4, 5, 10, 27], "recognit": [0, 1, 3, 12, 26, 27], "recommen": 27, "recommend": [0, 2, 3, 4, 5, 6, 8, 13, 15, 19, 20, 21, 22, 26, 29, 30], "reconsid": 9, "reconstruct": 11, "record": [10, 22, 23, 25, 27], "recreat": 15, "rectangl": [9, 13, 29], "rectangular": [5, 28, 29], "rectifi": [1, 3, 12], "recur": [0, 20, 27], "recurr": [0, 1, 20, 27], "recurs": [9, 20, 21, 27], "red": [0, 3, 4, 6, 8, 9, 30], "redefin": [0, 10, 27, 28, 29], "redefinit": 29, "reduc": [1, 3, 5, 6, 9, 10, 11, 13, 27, 29, 30], "reduct": [0, 10, 11, 20, 24, 27, 28], "reegress": 22, "refer": [0, 1, 2, 3, 5, 6, 11, 12, 13, 14, 21, 26, 27, 28, 29, 30], "referenc": 2, "refin": 12, "refit": 6, "reflect": [0, 1, 4, 5, 22, 24, 27], "refresh": [20, 27], "refreshprogrammingskil": 27, "reg": [10, 11], "regard": [1, 9, 13], "regardless": [12, 16], "region": [3, 4, 6, 9, 12, 22], "regist": [6, 24], "reglasso": [5, 29], "regr_1": [0, 9], "regr_2": [0, 9], "regr_3": [0, 9], "regress": [1, 8, 11, 12, 16, 20, 21], "regressor": [0, 7, 10], "regret": 30, "regridg": [0, 5, 6, 28, 29, 30], "regular": [0, 3, 4, 5, 6, 7, 9, 13, 17, 18, 25, 27, 28, 29, 30], "regularli": 15, "reilli": [0, 26, 27], "reinforc": [0, 8, 20, 27], "reiter": 1, "reject": 7, "rel": [0, 4, 6, 7, 9, 12, 13, 24, 27, 28, 30], "relat": [0, 1, 3, 4, 5, 11, 13, 14, 19, 21, 24, 27, 28, 29, 30], "relationship": [0, 4, 9, 18, 27], "relativeerror": [0, 27, 28], "releas": [1, 20, 27], "relev": [0, 1, 5, 7, 11, 20, 22, 24, 27, 29], "reli": [0, 6, 8], "reliabilti": 22, "reliabl": [7, 24], "relu": [3, 4, 27], "remain": [1, 2, 4, 6, 12, 21, 24, 28, 30], "remaind": 24, "reman": 2, "remark": 1, "rememb": [0, 8, 13, 21, 22, 27], "remind": [0, 5, 11, 13, 21, 24], "remot": 15, "remov": [4, 5, 6, 18, 28, 29, 30], "renam": 15, "render": [0, 27, 28], "reorder": [5, 7, 28, 29], "reorgan": [0, 27], "repeat": [0, 1, 3, 4, 5, 6, 9, 10, 11, 13, 14, 21, 22, 24, 27, 28, 29, 30], "repeated": 27, "repeatedli": [0, 6, 10, 13], "repet": 3, "repetit": [6, 27, 28], "rephras": [13, 29], "replac": [0, 1, 3, 4, 5, 6, 10, 12, 14, 20, 22, 27, 28, 29], "replica": 6, "repo": [15, 22], "report": [27, 30], "repositori": [4, 22, 27], "repres": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 12, 13, 22, 24, 27, 28, 29, 30], "represent": [0, 1, 3, 6, 24, 27], "representd": 3, "reproduc": [0, 5, 6, 9, 12, 15, 16, 18, 20, 24, 27, 28], "repuls": [0, 27], "request": [0, 13, 30], "requir": [0, 1, 3, 4, 5, 6, 8, 9, 11, 12, 13, 15, 17, 18, 21, 27, 28, 29, 30], "res1": 2, "res2": 2, "res3": 2, "res_analyt": 2, "res_analytical1": 2, "res_analytical2": 2, "res_analytical3": 2, "resaml": 6, "resampl": [0, 7, 10, 20, 27, 28], "rescal": [0, 11, 12, 30], "rescu": 5, "reseach": 6, "research": [0, 4, 13, 20, 26, 27], "resembl": [6, 24], "reserv": [1, 5, 6, 24], "reshap": [0, 1, 2, 3, 4, 6, 8, 9, 10, 21, 27, 28], "residenti": [], "residu": [0, 5, 13, 27], "resiz": [5, 28, 29], "resourc": 27, "respect": [0, 1, 2, 3, 5, 6, 7, 8, 10, 11, 12, 13, 14, 16, 17, 18, 22, 24, 27, 28, 29, 30], "respond": 12, "respons": [0, 7, 9, 12, 27, 28], "rest": [0, 5, 18, 28, 29, 30], "restat": [0, 12, 27], "restor": 4, "restored_discrimin": 4, "restored_gener": 4, "restrict": [0, 3, 9, 12, 27], "result": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 20, 21, 22, 24, 27, 30], "retail": [], "retain": [5, 6, 28, 29, 30], "return": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 11, 13, 14, 16, 17, 21, 24, 27, 28, 29, 30], "return_data": 14, "return_sequ": 4, "return_x_i": 9, "reus": [1, 3, 6, 19, 22], "reveal": [0, 12, 27], "revers": [1, 21], "review": [20, 21], "revisit": 14, "revolut": 27, "reward": [0, 4, 27], "rewrit": [0, 3, 5, 6, 7, 8, 10, 11, 12, 13, 16, 19, 21, 22, 24, 29, 30], "rewritten": [2, 6, 8, 10, 24], "rewrot": 13, "rf": 10, "rgb": 3, "rgoj5yh7evk": 20, "rh": 6, "rho": [0, 10, 13, 30], "rho_1": 10, "rho_2": 10, "rho_m": 10, "rich": [0, 27], "ride": 9, "rideclass": 9, "ridedata": 9, "ridg": [7, 11, 13, 20, 27], "ridge_paramet": 17, "ridge_sk": 6, "ridgebeta": 29, "ridgetheta": 5, "right": [0, 1, 2, 3, 5, 6, 7, 8, 9, 10, 12, 13, 14, 16, 17, 19, 21, 22, 24, 27, 28, 29, 30], "right_sid": 2, "rightarrow": [0, 1, 5, 6, 8, 11, 12, 13, 24, 27, 28, 29, 30], "rigor": [0, 27, 28, 29], "ring": 6, "rise": [0, 27], "risk": [0, 13, 27, 29, 30], "rival": 4, "river": [], "rlm": 27, "rm": 24, "rmse": [], "rmsporp": [13, 30], "rmsprop": [1, 3, 4, 13, 22], "rnd_clf": 10, "rng": 24, "rnn": [4, 12], "rnn1": 4, "rnn2": 4, "rnn_2layer": 4, "rnn_input": 4, "rnn_output": 4, "rnn_train": 4, "rntrick1": 24, "rntrick2": 24, "rntrick3": 24, "rntrick4": 24, "ro": [0, 13, 27, 29, 30], "robert": [19, 22, 26], "robust": [0, 27, 30], "robustscal": [0, 28, 30], "roc": [7, 10], "role": [0, 2, 5, 6, 8, 18, 20, 22, 27, 28, 29, 30], "roll": 6, "room": [0, 25, 27], "root": [0, 5, 9, 13, 15, 24, 28, 29, 30], "rot": 27, "rotat": [1, 8, 9, 10], "rotation_matrix": 9, "roughli": [1, 3, 18], "round": [7, 9, 13], "routin": [13, 21, 27, 29], "row": [0, 1, 2, 5, 6, 9, 11, 16, 21, 27, 28, 29], "rr": [5, 28, 29], "rrr": [5, 28, 29], "rudg": [], "rug": [13, 29, 30], "rule": [0, 1, 5, 6, 13, 22, 27, 28, 29], "run": [0, 1, 2, 4, 5, 6, 8, 9, 11, 13, 15, 20, 22, 27, 28, 29, 30], "runtim": [1, 6, 14, 15], "rust": [0, 20, 21, 27], "rvert": 1, "rvert_2": 1, "s_": [3, 6], "s_1": 6, "s_i": [6, 7], "s_j": 6, "s_k": 6, "s_phenomenon": 22, "saddl": [13, 29, 30], "safeguard": [18, 30], "sai": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 19, 21, 22, 24, 27, 28, 29], "said": [6, 9, 13, 29], "sake": [0, 5, 7, 11, 27, 28, 29], "sale": [0, 27], "sam": 27, "same": [0, 1, 2, 3, 4, 5, 6, 8, 9, 11, 12, 14, 15, 16, 18, 21, 22, 24, 27, 28, 29], "samm": 10, "sampl": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 13, 14, 18, 19, 20, 21, 22, 24, 27, 28, 30], "sample_vari": 14, "sampleexptvari": 24, "samwis": 27, "sastri": 11, "satisfactori": [0, 27], "satisfi": [1, 2, 3, 6, 8, 13, 21, 24, 29], "satur": [1, 6], "save": [0, 4, 6, 7, 9, 13, 27, 30], "save_fig": [0, 6, 7, 9, 10, 27], "savefig": [0, 4, 6, 7, 9, 24, 27], "savetxt": 4, "saw": [5, 28], "scalabl": 10, "scalar": [2, 5, 6, 10, 28], "scale": [0, 1, 3, 5, 6, 7, 8, 9, 10, 11, 12, 13, 20, 21, 22, 25, 27, 29], "scale_mean": 4, "scale_std": 4, "scaler": [0, 7, 8, 9, 10, 11, 17, 22, 28], "scan": [5, 7], "scari": 5, "scatter": [0, 1, 6, 7, 8, 9, 14, 15, 17, 27, 28, 30], "scenario": [6, 13, 29, 30], "schedul": [13, 30], "scheme": [1, 13, 29, 30], "schrage": 24, "sch\u00f8yen": [6, 28, 30], "scienc": [0, 1, 10, 12, 13, 20, 23, 24, 25, 26, 29], "scientif": [0, 20, 22, 27], "scientist": [0, 27], "scikit": [3, 5, 6, 8, 9, 10, 13, 15, 16, 20, 21, 22, 26], "scikit_learn": 0, "scikitlearn": 27, "scikitplot": [7, 10], "scipi": [0, 3, 5, 6, 13, 20, 21, 22, 27, 28, 29], "scl": 6, "scm": 15, "score": [0, 1, 3, 6, 7, 9, 10, 11, 15, 16, 22, 25, 27, 28, 30], "scores_kfold": 6, "scratch": [1, 13, 16], "sdg": [13, 30], "sdv4f4s2sb8": [29, 30], "seaborn": [0, 1, 3, 6, 7, 27], "seamless": [0, 20, 22, 27], "search": [0, 1, 3, 5, 9, 13, 15, 27, 29, 30], "sebastian": 27, "sebastianraschka": 27, "sec": 6, "second": [0, 2, 3, 4, 5, 6, 7, 8, 9, 11, 12, 14, 15, 16, 20, 21, 24, 25, 27, 28, 29], "second_mo": 30, "second_term": 30, "secondeigvector": 11, "secondli": 12, "section": [4, 11, 16, 21, 22, 24, 28, 30], "sector": 0, "see": [0, 1, 2, 3, 4, 5, 6, 7, 8, 10, 11, 12, 13, 15, 16, 18, 19, 20, 21, 22, 24, 27, 28, 29, 30], "seed": [0, 1, 2, 3, 4, 5, 6, 8, 9, 11, 13, 14, 18, 24, 27, 28, 29, 30], "seed_imag": 4, "seek": [1, 2, 8], "seem": [1, 3, 4, 30], "seemingli": [0, 27], "seen": [0, 1, 3, 5, 10, 12, 24], "segment": [13, 29], "seismic": 6, "seldomli": [0, 27], "select": [1, 5, 6, 8, 9, 10, 11, 15, 22, 23, 24, 25, 26, 27, 28, 29], "selevet": 15, "self": [1, 5, 28], "sell": 4, "semest": [7, 23], "semi": [8, 13, 29, 30], "semilogx": 6, "send": [5, 12, 13, 25, 27], "senior": [23, 25], "sens": [0, 4, 6, 8, 27], "sensibl": 3, "sensit": [0, 5, 6, 9, 13, 27, 28, 30], "sent": 2, "sentenc": [4, 12], "separ": [0, 1, 2, 4, 6, 8, 9, 12, 14, 18, 20, 22, 24, 27, 30], "septemb": [18, 22, 27], "sequenc": [3, 4, 7, 9, 10, 12, 13, 20, 21, 24, 27, 29], "sequenti": [1, 3, 4, 10, 12, 24], "seri": [0, 1, 2, 3, 4, 5, 6, 10, 11, 12, 13, 21, 27, 28, 29], "serif": [7, 24, 27], "serv": [0, 1, 2, 3, 5, 7, 13, 26, 27, 28, 29, 30], "servic": 22, "session": [1, 15, 22, 23, 25, 27], "set": [1, 4, 5, 6, 7, 8, 10, 11, 13, 14, 16, 17, 18, 20, 21, 22, 24, 25, 30], "set_major_formatt": 6, "set_major_loc": 6, "set_tick": [1, 8], "set_ticklabel": 1, "set_titl": [0, 1, 2, 3, 7, 12, 14, 27], "set_xlabel": [0, 1, 2, 3, 7, 12, 27], "set_xlim": [7, 12], "set_xticklabel": 1, "set_ylabel": [0, 1, 2, 3, 7, 27], "set_ylim": [7, 12], "set_ytick": 7, "set_yticklabel": [1, 6], "set_zlim": 6, "seth": 4, "setminu": 6, "setosa": [8, 9], "setosa_or_versicolor": 8, "setp": 6, "setup": [1, 4, 6, 8, 20, 27, 28, 29], "sever": [0, 3, 5, 6, 7, 8, 9, 11, 12, 13, 16, 20, 21, 22, 24, 27, 28, 29, 30], "sgd": [1, 3, 29], "sgd_clf": 8, "sgdclassifi": 8, "sgdreg": 13, "sgdregressor": 13, "sgn": [5, 28, 29], "shallow": [13, 30], "shape": [0, 1, 3, 4, 5, 6, 7, 8, 9, 10, 11, 13, 14, 15, 16, 18, 21, 27, 28, 29, 30], "share": [1, 3, 15, 27], "shareabl": 15, "she": 7, "shift": [1, 6, 12, 15, 18, 24, 28, 30], "ship": 3, "shire": 27, "short": [4, 5, 22], "shortcom": [13, 29, 30], "shorten": 4, "shorter": 24, "shorthand": 27, "shortli": [21, 27], "should": [0, 2, 3, 5, 6, 8, 9, 11, 12, 15, 18, 21, 22, 24, 27, 28, 30], "show": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 19, 21, 22, 24, 27, 28, 29, 30], "show_shap": 4, "shown": [0, 4, 5, 8, 12, 13, 21, 28, 29, 30], "shrink": [3, 5, 6, 8, 11, 28, 29], "shrinkag": [5, 6, 28, 29], "shrunk": 11, "shuffl": [0, 1, 4, 6, 13, 28, 30], "side": [0, 2, 5, 8, 12, 13, 21, 22, 27, 29], "sigh": [20, 27], "sigma": [0, 1, 5, 6, 7, 10, 11, 12, 13, 19, 21, 22, 24, 27, 28, 29, 30], "sigma0": 24, "sigma1": 24, "sigma2": 24, "sigma_": [5, 21, 27, 28, 29], "sigma_0": [5, 28, 29], "sigma_1": [5, 28, 29], "sigma_2": [5, 28, 29], "sigma_fn": [7, 12], "sigma_i": [0, 5, 27, 28, 29], "sigma_j": [5, 28, 29], "sigma_m": [6, 24], "sigma_n": [11, 24], "sigma_t": [13, 30], "sigma_x": 24, "sigmoid": [1, 2, 4, 7, 8, 10, 12], "sigmundson": [6, 28, 30], "sign": [1, 2, 7, 8, 10, 24, 25], "signal": [1, 3, 10, 12, 30], "signifi": 4, "signific": [1, 30], "significantli": [1, 13, 18, 24, 29, 30], "sim": [4, 5, 6, 13, 19, 24, 30], "similar": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 14, 18, 20, 21, 22, 27, 29], "similarli": [0, 1, 3, 5, 8, 10, 13, 24, 27, 28, 29, 30], "simpl": [1, 2, 3, 5, 6, 7, 8, 10, 11, 12, 14, 16, 17, 20, 21, 24], "simplepredict": 10, "simpler": [0, 1, 5, 6, 7, 13, 16, 20, 22, 27, 29, 30], "simplernn": 4, "simplest": [0, 1, 3, 4, 9, 10, 12, 14, 22, 27], "simpletre": 10, "simpli": [0, 1, 2, 4, 5, 6, 8, 9, 10, 11, 12, 20, 21, 22, 24, 27, 28, 29, 30], "simplic": [2, 5, 6, 7, 8, 9, 10, 11, 12, 14, 28, 29, 30], "simplicti": [5, 28, 29], "simplifi": [0, 6, 9, 18, 20, 22, 27, 28, 30], "simplist": [3, 6, 24], "simul": [6, 18], "simultan": 6, "sin": [0, 1, 2, 3, 4, 9, 12, 13, 21, 27], "sinc": [0, 1, 2, 3, 5, 6, 7, 8, 9, 10, 11, 13, 16, 18, 21, 22, 24, 26, 27, 28, 29, 30], "sine": [3, 12], "singl": [0, 1, 2, 3, 5, 6, 7, 8, 9, 12, 13, 18, 19, 21, 24, 27, 28, 29, 30], "singular": [0, 6, 13, 21, 27], "sinusoid": 3, "site": [0, 22, 23, 28], "situat": [0, 4, 5, 7, 13, 24, 27, 28, 29, 30], "six": [3, 24], "size": [0, 1, 2, 3, 4, 5, 6, 8, 9, 10, 11, 13, 18, 21, 22, 24, 27], "sizesp": 30, "sketch": 10, "ski": 9, "skill": 0, "skip": 11, "skl": [0, 6, 27, 28, 30], "sklearn": [0, 1, 3, 5, 6, 7, 8, 9, 10, 11, 13, 14, 15, 17, 19, 27, 28, 29, 30], "skplt": [7, 10], "sl": [6, 28, 30], "slack": 8, "slice": [2, 21, 27], "slide": [0, 3, 16, 22, 24, 27, 28, 29], "slight": [6, 13, 30], "slightli": [1, 2, 3, 5, 6, 7, 10, 24, 28, 29], "slope": [8, 11, 12], "slow": [0, 2, 8, 13, 18, 28, 29, 30], "slower": [5, 21, 27, 28, 29, 30], "slowest": 21, "slowli": [12, 30], "slp": 1, "small": [0, 1, 2, 3, 5, 6, 8, 9, 10, 11, 12, 13, 18, 20, 21, 24, 27, 28, 29, 30], "smaller": [0, 1, 2, 5, 6, 8, 9, 11, 13, 24, 27, 28, 29, 30], "smallest": [0, 4, 14, 27], "smallest_row_index": 14, "smooth": [0, 3, 6, 13, 22, 27, 29, 30], "sn": [0, 1, 3, 6, 7, 27], "sne": 11, "so": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 18, 19, 20, 21, 22, 24, 25, 27, 28, 29, 30], "soar": 6, "social": 0, "soft": [1, 7, 10, 12], "soften": 8, "softmax": [3, 7], "softwar": [0, 8, 20, 21], "sol": 8, "sole": [0, 6, 27], "solid": [0, 7], "solut": [0, 1, 2, 3, 5, 6, 8, 10, 11, 13, 18, 21, 22, 24, 27, 28, 29, 30], "solution_ev": 30, "soluton": 2, "solv": [0, 1, 3, 5, 6, 8, 10, 11, 12, 13, 16, 21, 22, 27, 28], "solve_expdec": 2, "solve_ode_deep_neural_network": 2, "solve_ode_neural_network": 2, "solve_pde_deep_neural_network": 2, "solveod": 2, "solveode_popul": 2, "solver": [2, 7, 8, 9, 10, 21, 27], "some": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 14, 15, 16, 18, 19, 22, 24, 27, 30], "some_model": [6, 28, 30], "somehow": 4, "someon": 16, "someth": [0, 1, 3, 4, 7, 9, 11, 15, 19, 22, 24, 27, 28], "sometim": [0, 1, 11, 12, 13, 14, 28, 30], "soon": [21, 25, 28], "sophist": [0, 27], "sopt": 13, "sort": [5, 6, 9, 11, 24], "sound": [3, 5], "sourc": [0, 1, 3, 6, 20, 21, 22, 24, 27], "space": [0, 1, 4, 5, 8, 9, 11, 12, 13, 14, 24, 28, 29, 30], "span": [0, 3, 5, 9, 11, 21, 27, 28, 29], "spare": 1, "spars": [3, 6, 18, 21, 27, 30], "sparse_mtx": [21, 27], "sparsecategoricalcrossentropi": 3, "sparsiti": [10, 18], "spatial": [1, 2, 3, 12], "speak": 24, "special": [6, 7, 10, 12, 13, 21, 24, 27, 28, 29, 30], "specif": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 11, 12, 15, 16, 20, 21, 22, 24, 26, 27, 28, 29], "specifi": [0, 3, 5, 6, 7, 9, 11, 13, 14, 24, 27, 29, 30], "specifici": [0, 10, 27], "spectacular": 3, "spectral": 1, "speech": [0, 1, 3, 4, 12], "speed": [1, 2, 4, 13, 30], "spend": [16, 24], "spent": 22, "sphere": [0, 28, 30], "spin": 6, "spite": 0, "spline": 8, "split": [1, 3, 4, 5, 6, 8, 9, 10, 11, 14, 16, 17, 22, 24, 27, 29, 30], "splite": 0, "splitter": [1, 10], "spontan": 24, "spot": 3, "spread": [0, 11, 24, 27, 28], "springer": [19, 22, 26, 27], "spuriou": [13, 30], "sqquar": 29, "sqrsignal": 3, "sqrt": [3, 4, 5, 6, 8, 10, 11, 13, 24, 28, 29, 30], "squar": [1, 2, 3, 4, 7, 8, 9, 11, 13, 14, 15, 17, 18, 20, 21, 24], "squarederror": 10, "squaredeuclidean": 14, "squash": 12, "srtm": 6, "srtm_data_norway_1": 6, "stabil": [5, 22, 30], "stabl": [0, 4, 5, 6, 9, 16, 20, 22, 27, 28, 29], "stack": [3, 4], "stage": [5, 13, 15, 22, 30], "stai": [0, 2, 4, 5, 11, 27, 28, 30], "stand": [0, 5, 9, 12, 27, 28, 29], "standard": [0, 1, 4, 5, 6, 7, 8, 10, 12, 17, 18, 21, 22, 24, 27, 29, 30], "standardscal": [0, 6, 7, 8, 9, 10, 11, 17, 28, 30], "stanford": [13, 29], "start": [0, 1, 2, 3, 4, 5, 6, 8, 9, 10, 11, 12, 13, 14, 15, 16, 18, 21, 24, 25, 27, 28, 29, 30], "start_tim": 14, "stat": 6, "state": [1, 2, 4, 5, 6, 7, 8, 10, 11, 12, 13, 20, 24, 27, 28, 29], "statement": [0, 7, 21, 27], "stationari": [29, 30], "statist": [0, 1, 3, 4, 7, 9, 10, 11, 12, 13, 14, 19, 21, 22, 26, 28, 29], "statu": [0, 7, 15, 27], "stavang": 6, "std": [0, 4, 6, 18, 27, 28, 30], "steep": [13, 29, 30], "steepest": 30, "step": [0, 1, 2, 4, 6, 7, 9, 10, 11, 12, 13, 14, 15, 18, 21, 22, 27, 29], "step_fn": [7, 12], "step_length": [13, 30], "step_siz": 30, "steps_list": 9, "stereo": 3, "still": [0, 2, 3, 5, 6, 11, 13, 24, 28, 29, 30], "stimuli": 12, "stk": [26, 27], "stk2100": [26, 27], "stk3155": [15, 22, 23, 25], "stk4021": [26, 27], "stk4051": [26, 27], "stk4155": [23, 25], "stk5000": 26, "stochast": [0, 1, 5, 6, 8, 11, 12, 29], "stock": 4, "stoke": 12, "stone": [0, 7], "stop": [1, 4, 9, 13, 14, 18, 29], "storag": [5, 28, 29], "store": [0, 1, 2, 3, 6, 11, 13, 24, 27, 30], "storehaug": [25, 27], "str": [1, 3, 4], "straight": [0, 6, 8, 13, 27, 29], "straightforward": [0, 2, 3, 5, 6, 8, 9, 10, 13, 21, 27, 28, 29], "strategi": [0, 1, 9, 27], "stratifi": 6, "strength": [0, 5, 14, 28, 29], "stretch": 11, "strict": [8, 13, 29], "strictli": [8, 13, 29], "stride": [4, 21], "strike": 6, "string": 1, "stroke": 7, "strong": [3, 6, 9, 10, 12, 21, 24], "strongli": [0, 8, 15, 20, 21], "stronli": [], "structur": [0, 1, 2, 3, 6, 9, 10, 12, 20, 27], "stuck": [1, 13, 29, 30], "student": [0, 15, 22, 23, 25, 26, 27], "studi": [0, 3, 4, 5, 6, 7, 8, 11, 12, 13, 20, 22, 26, 27, 28, 29, 30], "studier": 26, "style": [7, 9, 21, 27], "st\u00f8land": 25, "sub": [9, 12, 30], "subdivid": [0, 21, 27], "subfield": 0, "subgradi": 30, "subject": [6, 8, 24], "submit": 27, "subplot": [0, 1, 3, 4, 6, 7, 8, 9, 10, 14, 27], "subplots_adjust": [8, 24], "subprogram": [21, 27], "subract": [0, 28], "subroutin": [0, 27], "subscript": 1, "subsequ": [1, 4, 5, 6, 12, 21, 24, 28, 29], "subset": [1, 6, 9, 12, 13, 20, 27, 29, 30], "subspac": [0, 8, 11, 28], "substanti": [9, 10], "substep": 11, "substitut": [3, 6, 12, 16, 21], "subsubset": 9, "subtask": 6, "subtl": 1, "subtract": [0, 4, 5, 6, 11, 13, 18, 21, 22, 24, 28, 30], "subtre": 9, "succeed": [0, 4, 27], "success": [3, 7, 9, 13, 24], "successfulli": [4, 9], "sudo": [0, 20, 22, 27], "suffer": [0, 1, 2, 5, 10, 27, 28, 29], "suffici": [1, 6, 8, 11, 13, 29], "suggest": [1, 13, 22, 26, 29, 30], "suit": [8, 12], "suitabl": [0, 15, 19, 24, 28], "sum": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 19, 21, 24, 27, 28, 29, 30], "sum_": [0, 1, 2, 3, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 16, 19, 21, 22, 24, 27, 28, 29, 30], "sum_i": [0, 2, 5, 6, 8, 13, 19, 22, 28, 29, 30], "sum_j": [6, 18, 30], "sum_ja_": 0, "sum_k": [6, 8, 12, 21], "sum_logist": 13, "sum_m": 3, "sum_n": 3, "sum_nx_": 3, "summar": [5, 6, 9], "summari": [1, 3, 4, 10, 23, 29], "summat": [0, 3, 16, 28, 29], "sunni": 9, "super": [5, 28, 29], "superfici": 3, "superscript": [1, 12], "supervis": [0, 5, 6, 7, 9, 12, 20, 27, 28, 29], "supplement": [7, 22], "support": [0, 1, 9, 10, 11, 13, 20, 27, 28], "suppos": [0, 5, 6, 7, 8, 10, 11, 12, 13, 21, 27, 28, 29, 30], "suppress": [5, 13, 29, 30], "sure": [0, 1, 4, 6, 16, 22], "surf": 6, "surfac": [0, 6, 27], "surpass": 6, "surpris": [0, 27], "surround": [3, 20], "survei": [0, 5, 6, 27, 28], "svc": [8, 9, 10], "svd": [0, 6, 11, 27], "svdinv": 5, "svm": [8, 9, 10, 11], "svm_clf": [8, 10], "swath": [5, 28, 29], "switch": 0, "sy": [13, 29, 30], "symbol": [1, 5, 11, 13, 20, 24, 27, 28, 29], "symmeteri": 1, "symmetr": [0, 5, 8, 11, 12, 13, 21, 27, 28], "symmetri": 6, "sympi": [0, 20, 22, 27], "synonim": 24, "syntax": 13, "system": [0, 1, 3, 4, 6, 7, 9, 10, 12, 13, 15, 20, 21, 22, 27, 29], "systemat": [4, 6], "t": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 24, 25, 27, 29, 30], "t0": [3, 6, 13, 30], "t1": [2, 13, 30], "t2": 2, "t3": 2, "t_": 2, "t_0": [2, 9, 13, 30], "t_1": [13, 30], "t_b": 10, "t_i": [1, 2, 5, 12, 28, 29], "t_j": 12, "t_k": 9, "tabl": [9, 22, 24, 25, 27], "tabul": [0, 27], "tabular": 27, "tackl": 4, "tag": [2, 3, 4, 5, 6, 7, 12, 13, 14, 21, 24, 28, 29, 30], "taht": [0, 27], "tail": 24, "tailor": [2, 8, 11, 27], "taiwan": [0, 27], "take": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 17, 20, 21, 24, 27, 28, 29, 30], "taken": [0, 1, 3, 6, 10, 13, 21], "tan": 3, "tangent": [1, 4, 12, 13, 29], "tanh": [1, 4, 7, 8, 12], "target": [0, 1, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 15, 16, 18, 19, 27, 28, 29, 30], "target_nam": 9, "task": [0, 1, 3, 6, 9, 11, 12, 14, 22, 27], "tau": [3, 5, 24], "taught": 27, "tax": [], "taylor": [2, 13, 29], "taylornr": [13, 29], "tc": 8, "teach": [15, 23, 27], "team": 1, "teaser": 0, "technic": [0, 5, 6, 13, 22, 29, 30], "techniqu": [0, 1, 8, 10, 13, 20, 24, 26, 27, 28, 30], "technologi": [0, 1], "tell": [0, 4, 6, 10, 11, 13, 16, 24, 30], "temp": 1, "temp1": 1, "temp2": 1, "temperatur": [0, 9, 27], "templat": 18, "temporarili": 1, "ten": [3, 27], "tend": [3, 5, 6, 8, 9, 10, 12, 13, 14, 28, 30], "tendenc": [0, 27], "tension": 6, "tensor": 3, "tensorflow": [0, 2, 4, 8, 14, 20, 21, 22, 26, 27, 28], "term": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 16, 18, 19, 22, 24, 27, 28, 29, 30], "term1": [5, 6, 11], "term2": [5, 6, 11], "term3": [5, 6, 11], "term4": [5, 6, 11], "termin": [0, 4, 5, 9, 10, 13, 15, 28, 29, 30], "terminarl": 15, "terrain": 6, "terrain1": 6, "test": [3, 4, 5, 6, 7, 8, 9, 10, 13, 16, 21, 22, 24, 27, 29, 30], "test_acc": 3, "test_accuraci": [1, 3], "test_error": 6, "test_imag": [3, 4], "test_ind": 6, "test_input": 4, "test_label": [3, 4], "test_loss": 3, "test_pr": 1, "test_predict": 1, "test_rnn": 4, "test_scor": [7, 10], "test_siz": [0, 1, 3, 5, 6, 10, 15, 17, 28, 29, 30], "test_split": 9, "testerror": [0, 6, 28], "testi": 4, "testpredict": 4, "testx": 4, "text": [0, 1, 2, 4, 5, 8, 9, 11, 13, 15, 18, 21, 24, 26, 28, 29, 30], "textbf": 30, "textbook": [16, 22, 28, 29], "textual": 9, "textur": 1, "tf": [1, 3, 4, 13, 14, 29], "th": [0, 1, 2, 5, 6, 7, 9, 12, 13, 14, 21, 22, 24, 27, 28, 30], "than": [0, 1, 2, 3, 4, 5, 6, 7, 9, 10, 11, 12, 13, 17, 20, 24, 27, 28, 30], "thank": [4, 6, 28, 30], "theano": [1, 20, 27], "thei": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 11, 12, 13, 15, 16, 18, 21, 22, 24, 27, 28, 29, 30], "them": [0, 1, 3, 4, 6, 8, 9, 10, 11, 12, 13, 18, 19, 21, 22, 27, 28], "theme": [0, 15, 27], "themselv": [0, 22, 24, 27], "thenc": 6, "theorem": [2, 6, 7, 28, 29], "theoret": [0, 4, 10], "theori": [0, 1, 3, 8, 9, 12, 13, 19, 20, 22, 26, 27], "thereaft": [0, 5, 6, 11, 12, 21, 22, 27], "therebi": [0, 5, 7, 11, 22, 27, 28, 29], "therefor": [0, 1, 2, 3, 4, 6, 7, 8, 11, 13, 19, 24, 27, 28, 29, 30], "therein": 11, "thereof": [0, 6, 13, 27, 30], "theta": [0, 1, 4, 5, 6, 7, 13, 16, 22, 24, 27, 28, 29, 30], "theta_": [0, 1, 6, 7, 13, 27, 28, 29, 30], "theta_0": [0, 5, 6, 7, 16, 27, 28, 29, 30], "theta_0x_": [0, 27, 28], "theta_1": [0, 5, 6, 7, 27, 28, 29, 30], "theta_1x_": [0, 27, 28], "theta_1x_0": [0, 27], "theta_1x_1": [0, 7, 27], "theta_1x_2": [0, 27], "theta_1x_i": [7, 28, 29, 30], "theta_2": [0, 27, 28], "theta_2x_": [0, 27, 28], "theta_2x_0": [0, 27], "theta_2x_1": [0, 27], "theta_2x_2": [0, 7, 27], "theta_2x_i": 28, "theta_3x_i": 28, "theta_4x_i": 28, "theta_closed_form": 18, "theta_closed_formol": 18, "theta_closed_formridg": 18, "theta_gdol": 18, "theta_gdridg": 18, "theta_i": [0, 1, 5, 27, 28, 29], "theta_j": [0, 5, 6, 18, 27, 28, 30], "theta_k": [29, 30], "theta_linreg": [13, 29, 30], "theta_ol": 18, "theta_p": 7, "theta_px_p": 7, "theta_ridg": 18, "theta_t": [13, 30], "theta_tru": 18, "thetavalu": 5, "thi": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 26, 28, 29, 30], "thing": [0, 1, 2, 4, 5, 7, 9, 15, 16, 18, 24, 27], "think": [0, 1, 3, 4, 6, 9, 12, 13, 14, 24, 27, 28, 29, 30], "third": [0, 3, 6, 13, 25, 27, 29, 30], "thirti": 7, "thorughout": 27, "those": [0, 3, 5, 6, 8, 9, 10, 11, 21, 22, 27, 28, 29, 30], "though": [1, 2, 3, 4, 13, 16, 17, 21, 24, 30], "thought": [6, 14, 22, 24], "thousand": [0, 1, 22, 28, 30], "three": [0, 1, 3, 5, 6, 8, 9, 12, 21, 22, 23, 24, 25, 27, 28, 29], "threshold": [1, 3, 9, 10, 11, 12, 13, 30], "through": [0, 1, 2, 3, 4, 5, 6, 8, 11, 12, 13, 14, 15, 20, 21, 22, 24, 27, 28, 29, 30], "throughout": [0, 4, 5, 14, 15, 20, 21, 24, 27], "throw": [3, 6, 24], "thu": [0, 1, 2, 5, 6, 7, 8, 10, 11, 12, 13, 25, 27, 28, 29, 30], "thumb": [0, 6, 22, 28], "thursdai": [], "tibshirani": [6, 19, 22, 26, 27], "tick_param": 6, "ticker": [6, 13, 24, 29, 30], "tif": 6, "tight_layout": [1, 7], "tightli": 11, "tild": [0, 5, 6, 7, 11, 19, 22, 24, 27, 28, 29, 30], "till": [0, 4, 7, 8, 9, 10, 12, 21, 27, 28], "time": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 20, 21, 22, 24, 27, 28, 29], "timeit": 4, "timer": 4, "tini": 1, "tip": 3, "titl": [0, 1, 2, 3, 4, 6, 7, 8, 9, 10, 13, 15, 24, 27, 29, 30], "tmp": 13, "tn": [2, 3, 7], "to_categor": [1, 3, 4], "to_categorical_numpi": 1, "to_numer": [0, 6, 27], "todai": 3, "togeth": [0, 3, 6, 8, 11, 13, 20, 27], "toi": 14, "told": 13, "toler": [2, 14], "tolist": 4, "tomographi": 12, "too": [0, 2, 4, 5, 6, 9, 11, 13, 17, 18, 24, 26, 28, 29, 30], "took": [8, 27], "tool": [0, 1, 3, 6, 13, 15, 20, 28], "toolbox": 8, "top": [0, 3, 5, 6, 9, 10, 19, 20, 27], "topic": [0, 5, 6, 7, 8, 20, 22, 28, 29], "topolog": [3, 12], "topologi": [1, 12], "torkjellsdatt": [25, 27], "toss": [10, 24], "total": [0, 1, 2, 3, 4, 6, 7, 8, 10, 11, 12, 13, 14, 21, 24, 25, 27, 28, 29, 30], "total_loss": 4, "totalclustervari": 14, "totalscatt": 14, "toward": [1, 2, 7, 12, 13, 15, 29], "towardsdatasci": 30, "town": [], "tp": [4, 7], "tpng": 9, "tpu": [13, 20, 27], "tqdm": 6, "tr": [], "track": [3, 13, 14, 15, 21, 28, 29, 30], "tract": [], "tractabl": [0, 27, 28], "trade": [5, 9], "tradeoff": [0, 5, 19, 22, 27, 28, 29], "tradit": [0, 1, 4, 6, 27], "train": [2, 3, 5, 6, 8, 9, 10, 11, 12, 13, 16, 17, 22, 29, 30], "train_accuraci": [0, 1, 3, 27], "train_dataset": 4, "train_end": [0, 1, 28], "train_error": 6, "train_imag": [3, 4], "train_ind": 6, "train_label": [3, 4], "train_pr": 1, "train_siz": [0, 1, 3, 28], "train_step": 4, "train_test_split": [0, 1, 3, 5, 6, 7, 9, 10, 11, 15, 16, 17, 19, 27, 28, 29, 30], "train_test_split_numpi": [0, 1, 28], "trainable_vari": 4, "trained_model": [6, 28, 30], "trainerror": [0, 28], "traini": 4, "training_checkpoint": 4, "training_dataset": 4, "training_gradi": [13, 30], "trainingerror": 6, "trainpredict": 4, "trainscor": 4, "trainx": 4, "trait": [0, 27], "trajectori": 4, "transfer": [9, 27], "transform": [0, 5, 6, 7, 8, 9, 10, 11, 12, 13, 17, 20, 21, 27, 28, 29, 30], "transit": [6, 12], "translat": [1, 4, 6, 10, 27, 28, 30], "transpos": [1, 5, 11, 21, 28, 29], "travers": [0, 5], "treat": [0, 1, 3, 6, 12, 13, 18, 24, 27, 28, 29, 30], "tree": [0, 1, 20, 27], "tree_clf": [9, 10], "tree_clf_": 9, "tree_clf_sr": 9, "tree_reg": 9, "tree_reg1": 9, "tree_reg2": 9, "trend": 24, "treue": 7, "trevor": [19, 22, 26], "tri": [2, 3, 4, 9, 13, 16], "triain": 0, "trial": [0, 2, 4, 6, 13, 24, 27, 29, 30], "triangl": [13, 29], "triangular": 21, "trick": [3, 4, 8, 11, 13, 24, 30], "trickier": 24, "tridiagon": 21, "trillion": 20, "trivial": [0, 1, 5, 11, 24, 27, 29], "troubl": [0, 8, 12, 15, 28, 30], "truck": 3, "true": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 12, 13, 14, 16, 17, 18, 19, 21, 22, 24, 27, 28, 29, 30], "true_beta": 28, "true_fun": 6, "true_theta": [6, 30], "truli": 27, "try": [0, 1, 2, 4, 5, 6, 7, 8, 9, 10, 11, 13, 14, 15, 18, 20, 21, 22, 24, 27, 28, 29, 30], "tucker": 8, "tuesdai": [25, 27], "tumor": [7, 9], "tumour": 7, "tunabl": 1, "tune": [4, 9, 13, 21, 27, 30], "turn": [0, 1, 5, 6, 7, 8, 9, 10, 11, 12, 13, 21, 22, 24, 27, 28, 29, 30], "tutori": [1, 4], "tv": 2, "tveito": 2, "tweak": [1, 4, 10, 24], "twice": [13, 29], "twist": 11, "two": [0, 1, 2, 4, 5, 6, 7, 9, 10, 11, 12, 13, 15, 17, 21, 22, 23, 24, 26, 27, 28, 29, 30], "tx": [13, 29, 30], "tx_1": [13, 29], "txt": [4, 15], "ty": [13, 29], "type": [0, 1, 3, 6, 8, 10, 13, 21, 24, 28, 29, 30], "typic": [0, 1, 2, 3, 4, 5, 7, 9, 10, 12, 13, 15, 16, 24, 27, 28, 29, 30], "typo": 22, "u": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 21, 22, 24, 26, 27, 28, 29, 30], "u_": 21, "u_i": 12, "u_m": 10, "ua": [0, 27], "ubuntu": [0, 20, 22, 27], "uci": 22, "uio": [15, 22, 25, 26], "un": 14, "unabl": 15, "unari": [21, 27], "unbalanc": [6, 9], "unbias": [0, 5, 6, 27], "uncent": [6, 28, 30], "uncertainti": [0, 5, 27], "uncertitud": 24, "unchang": [1, 3], "uncorrel": [10, 24], "undefin": [5, 28, 29], "under": [0, 1, 5, 6, 10, 13, 20, 22, 27, 28, 29], "underdetermin": [0, 27], "underfit": [1, 6], "underflowproblem": 5, "undergo": 5, "undergradu": [23, 25], "underli": [0, 1, 9, 13, 18, 24, 27, 30], "underset": [4, 14], "understand": [0, 1, 3, 5, 6, 10, 13, 14, 15, 20, 27, 28, 29, 30], "understood": [8, 13, 30], "undesir": 8, "undetermin": [5, 8], "undo": 4, "unexpect": 6, "unexpected": 24, "unexplain": 18, "unfair": [6, 28], "unfortun": [1, 8, 9, 10], "unicode_liter": [8, 9], "uniform": [0, 1, 5, 6, 11, 13, 22, 24, 27, 29, 30], "uniformli": [13, 24, 29, 30], "unifrompdf": 24, "unimport": [13, 29], "union": [5, 6], "uniqu": [0, 2, 6, 13, 14, 21, 27], "unique_cluster_label": 14, "unit": [0, 1, 3, 4, 5, 10, 12, 18, 24, 27, 28, 29, 30], "unitari": [5, 6, 21, 28, 29], "unitarili": [21, 27], "uniti": 24, "univari": 24, "univers": [0, 1, 2, 13, 20, 22, 23, 25, 27, 28, 29, 30], "unix": 1, "unknow": [0, 21, 27], "unknown": [0, 1, 3, 4, 5, 6, 8, 10, 13, 21, 22, 27, 28, 29, 30], "unknowwn": 12, "unlabel": 1, "unless": [0, 3, 6, 11, 13, 22, 27, 29], "unlik": [1, 3, 8, 13, 29, 30], "unnecessarili": 9, "unord": 3, "unpublish": 30, "unravel": 1, "unrol": [3, 11], "unseen": [0, 7, 9, 15], "unstabl": 1, "unsupervis": [0, 1, 4, 12, 20, 27], "unsymmetr": [21, 27], "until": [1, 2, 4, 9, 12, 13, 14, 29], "untouch": 0, "unusu": 12, "up": [1, 3, 4, 5, 6, 8, 10, 11, 13, 14, 16, 18, 19, 20, 21, 22, 24, 25, 30], "updat": [1, 2, 10, 12, 13, 14, 15, 18], "uploa": 27, "upload": [15, 20, 22, 26], "upon": [0, 1, 6, 7, 11, 21], "upper": [0, 8, 9, 16, 21, 28], "uppercas": [21, 27], "upsampl": 4, "upscal": 4, "url": [27, 28], "us": [4, 5, 6, 8, 9, 10, 11, 12, 14, 15, 17, 19, 21, 24, 26], "usag": [0, 8, 20, 27, 28, 30], "usd": [], "usd10000": [], "use_bia": 4, "usecol": [0, 27], "useless": 1, "user": [0, 1, 2, 4, 6, 7, 15, 20, 21, 22, 27, 28], "usernam": [15, 22], "usetex": 24, "usg": 6, "usr": 24, "usual": [0, 3, 4, 7, 12, 13, 14, 27, 30], "ut": 5, "util": [1, 3, 4, 6, 7, 10, 14, 19, 27], "ux": 21, "v": [2, 4, 5, 6, 11, 13, 15, 20, 28, 29, 30], "v0": 24, "v1": 24, "v2": 24, "v_": 30, "v_0": 11, "v_t": 30, "va": 1, "vahid": 27, "val": 13, "val_accuraci": 3, "val_loss": 4, "vale": 2, "valid": [0, 1, 4, 7, 9, 10, 13, 20, 24, 27, 28, 30], "validation_data": 3, "validation_split": 4, "valu": [0, 1, 2, 3, 4, 6, 7, 8, 9, 10, 12, 13, 14, 16, 17, 18, 20, 21, 22, 27, 30], "valuat": 9, "valy": 4, "van": [0, 19, 22, 27, 28, 29, 30], "vandenbergh": [8, 13, 29], "vandermond": [0, 27], "vanilla": [0, 6, 11, 14, 28, 30], "vanish": [1, 4, 13, 24, 29], "var": [5, 6, 10, 11, 19, 22, 24, 28], "var_x": 24, "varabl": 8, "varepsilon": [5, 6, 19], "varepsilon_": [5, 6], "varepsilon_i": [5, 6], "vari": [0, 1, 3, 5, 6, 10, 27], "variabl": [0, 1, 2, 5, 6, 7, 8, 10, 11, 12, 13, 14, 21, 27, 28, 30], "varianc": [0, 1, 5, 7, 9, 10, 11, 13, 14, 18, 20, 21, 24, 27, 28, 29, 30], "variance_i": [5, 11, 28], "variance_x": [5, 11, 28], "variant": [0, 1, 6, 8, 12, 13, 27, 28, 29, 30], "variat": [3, 4, 11, 27], "varieti": [0, 3, 12, 20, 22, 27], "variou": [1, 3, 5, 6, 7, 8, 9, 11, 12, 13, 16, 19, 20, 21, 22, 24, 27, 28, 29, 30], "varydimens": 4, "vastli": 3, "vaue": 1, "vault": 0, "vdot": [2, 13, 29, 30], "ve": [22, 30], "vec": 6, "vector": [0, 1, 2, 3, 4, 5, 6, 7, 9, 10, 11, 13, 14, 17, 18, 20, 29, 30], "vector_mean": 14, "ventur": [0, 8, 20, 27], "venv": 15, "verbos": [1, 3, 4], "veri": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 16, 18, 22, 24, 26, 27, 28, 29, 30], "verifi": [3, 11, 21, 27], "versatil": [8, 27], "versicolor": [8, 9], "version": [0, 3, 10, 13, 14, 15, 20, 21, 22, 24, 27], "versu": 1, "vert": [0, 1, 5, 6, 7, 8, 9, 11, 13, 16, 17, 27, 28, 29, 30], "vert_1": [5, 6, 28, 29, 30], "vert_2": [5, 6, 11, 17, 28, 29, 30], "via": [0, 5, 6, 7, 8, 9, 10, 11, 12, 19, 20, 21, 22, 23, 24, 25, 27, 28, 29, 30], "vidal": 11, "video": [0, 1, 12, 20, 23, 25, 27, 28, 29], "view": [1, 3, 5, 6, 12, 13, 24, 26, 27, 29, 30], "violat": 8, "virginica": 9, "viridi": [0, 1, 2, 3, 27], "virtual": 1, "viscos": [13, 30], "viscou": [13, 30], "visibl": 15, "vision": [0, 3], "visual": [0, 3, 11, 12, 18, 20, 27, 28], "visualis": 1, "visualstudio": [15, 16], "viz": [6, 8, 24], "vmap": 13, "vmax": [1, 6], "vmh0zpt0tli": 30, "vmin": [1, 6], "voic": 3, "volatil": 30, "volum": [0, 3, 27], "vote": [10, 27], "voting_clf": 10, "votingclassifi": 10, "votingsimpl": 10, "vstack": [5, 11, 21, 24, 27, 28], "vt": [5, 28, 29], "w": [0, 1, 2, 3, 4, 5, 6, 7, 8, 10, 11, 12, 13, 14, 21, 24, 27, 28, 29, 30], "w1": 8, "w2": [8, 11], "w3": 8, "w_": [1, 12, 30], "w_1": [8, 21], "w_1x_": 8, "w_1x_1": 8, "w_2": [8, 21], "w_2x_": 8, "w_2x_2": 8, "w_3": 21, "w_4": 21, "w_hidden": 2, "w_i": [1, 2, 10], "w_ix_i": 12, "w_j": 21, "w_m": 21, "w_output": 2, "w_px_": 8, "w_px_p": 8, "w_t": 30, "wa": [1, 3, 4, 5, 6, 7, 10, 11, 12, 14, 17, 21, 27, 28, 30], "wai": [0, 1, 2, 3, 4, 5, 6, 7, 8, 10, 11, 12, 13, 14, 15, 18, 19, 21, 24, 27, 28, 29, 30], "walk": 9, "walker": 24, "wang": [0, 27], "want": [0, 1, 2, 3, 4, 5, 6, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 20, 22, 24, 27, 28, 29, 30], "warn": 4, "warrant": 6, "wast": [3, 30], "watch": [20, 29, 30], "wave": 3, "wavelet": 8, "we": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 24, 25, 26, 28, 29], "weak": [9, 10, 14], "weather": [1, 12], "web": [20, 23, 25, 27], "webpag": 27, "websit": [6, 21, 22, 23, 27], "wedg": [8, 24], "wednesdai": [25, 27], "wee": 11, "week": [0, 5, 6, 7, 22, 23, 25], "weekli": [15, 16, 20, 22, 23, 25, 26, 27], "weight": [1, 2, 3, 6, 7, 9, 10, 12, 13, 18, 24, 30], "weigth": 2, "welcom": [8, 15, 20], "well": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 12, 13, 15, 16, 20, 21, 22, 24, 26, 27, 28, 29, 30], "went": 8, "were": [0, 1, 3, 4, 5, 6, 7, 8, 10, 11, 12, 14, 24, 27], "wessel": [0, 19, 22, 27, 28, 29, 30], "what": [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 19, 20, 21, 22, 24, 30], "whatev": 3, "when": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 18, 19, 21, 22, 24, 27, 28, 29], "whenev": [13, 15, 24, 30], "where": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 20, 21, 22, 24, 25, 27, 28, 29, 30], "wherea": [6, 24, 30], "wherefrom": 22, "wherein": [1, 12], "whether": [0, 3, 5, 7, 9, 22, 24, 27], "which": [0, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 27, 28, 29], "whichev": [1, 3], "while": [0, 1, 3, 4, 5, 6, 7, 8, 9, 11, 12, 13, 15, 16, 19, 24, 27, 28, 29, 30], "white": 9, "whiteboard": [28, 29], "who": [0, 15], "whole": [1, 3, 4, 5, 9, 11, 13, 30], "whose": [0, 6, 10, 24, 28], "whow": [11, 28], "why": [0, 1, 3, 6, 13, 15, 16, 17, 19, 22, 28, 29, 30], "wide": [0, 1, 3, 6, 7, 12, 20, 21, 22, 27], "widehat": 6, "width": [0, 3, 8, 9, 27], "wieringen": [0, 19, 22, 27, 28, 29, 30], "wiki": 22, "wikipedia": 22, "win": 10, "wind": 9, "wing": [25, 27], "winther": 2, "wiothout": 6, "wiscons": 7, "wisconsin": 10, "wisdom": [6, 28, 30], "wise": [1, 5, 12, 13, 28, 29, 30], "wish": [0, 2, 5, 7, 8, 11, 13, 14, 18, 21, 22, 27, 28, 29, 30], "with_std": [0, 28], "wither": 6, "within": [0, 2, 3, 4, 7, 9, 12, 13, 14, 24, 26, 27, 29], "withinclust": 14, "without": [0, 1, 5, 6, 8, 9, 11, 12, 13, 15, 18, 22, 27, 28, 29, 30], "won": [0, 15, 27], "wonder": 8, "word": [0, 1, 3, 4, 5, 6, 7, 14, 24, 27, 28, 29, 30], "work": [0, 1, 4, 6, 7, 8, 9, 13, 15, 16, 18, 20, 22, 23, 24, 25, 27, 28, 30], "workabl": 30, "workshop": 27, "world": [0, 8, 16, 28], "worldwid": [0, 27], "worri": 15, "wors": [0, 1, 3, 4, 6, 27], "worth": 9, "worthi": 22, "would": [0, 1, 3, 5, 6, 7, 8, 9, 10, 11, 12, 13, 16, 18, 21, 22, 24, 27, 28, 29, 30], "wrap": [6, 21, 27], "write": [0, 1, 2, 3, 5, 6, 7, 8, 12, 13, 15, 16, 18, 21, 27, 28, 30], "written": [0, 2, 3, 5, 11, 12, 13, 16, 20, 21, 22, 24, 27, 28, 29, 30], "wrong": [1, 8, 15], "wrongli": 10, "wrote": [5, 11, 28], "wrt": [10, 13, 30], "wth": [10, 13, 30], "www": [20, 21, 22, 26, 27, 29, 30], "wx_1": 8, "x": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 21, 22, 24, 27, 29, 30], "x0": 8, "x1": [4, 8, 9, 10, 13], "x1_exampl": 8, "x1d": 8, "x2": [8, 9, 10, 13], "x2d": [8, 11], "x2d_train": 11, "x2dsl": 11, "x3": 8, "x_": [0, 2, 3, 5, 6, 8, 10, 11, 13, 14, 21, 24, 27, 28, 29, 30], "x_0": [0, 5, 11, 18, 21, 27, 28], "x_1": [0, 2, 5, 6, 7, 8, 9, 10, 11, 13, 18, 21, 24, 27, 28, 29, 30], "x_2": [0, 2, 5, 6, 7, 8, 9, 10, 11, 13, 21, 24, 27, 28, 29], "x_3": [8, 21, 24], "x_4": 21, "x_6": 18, "x_center": 11, "x_data": 1, "x_data_ful": 1, "x_hidden": 2, "x_i": [0, 1, 2, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 21, 24, 27, 28, 29, 30], "x_input": 2, "x_ix_": [0, 27], "x_iy_i": 8, "x_j": [0, 2, 8, 9, 12, 16, 24, 28, 30], "x_jy_j": 8, "x_k": [12, 14, 21, 24, 28], "x_l": 24, "x_m": [6, 12, 21, 24], "x_mean": [18, 30], "x_n": [0, 2, 3, 6, 8, 11, 12, 13, 21, 24, 27, 29], "x_new": [9, 10], "x_norm": [18, 30], "x_offset": [6, 28, 30], "x_output": 2, "x_p": [3, 7, 9], "x_poli": 9, "x_poly10": 9, "x_pred": 4, "x_prev": 2, "x_reduc": 11, "x_sampl": 19, "x_scale": 8, "x_small": 13, "x_std": [18, 30], "x_t": 30, "x_test": [0, 1, 3, 5, 6, 7, 9, 10, 11, 15, 16, 17, 19, 28, 29, 30], "x_test_": 17, "x_test_own": 6, "x_test_scal": [0, 6, 7, 9, 10, 11, 28, 30], "x_tot": 4, "x_train": [0, 1, 3, 4, 5, 6, 7, 9, 10, 11, 15, 16, 17, 19, 27, 28, 29, 30], "x_train_": 17, "x_train_mean": [6, 28, 30], "x_train_own": 6, "x_train_scal": [0, 6, 7, 9, 10, 11, 28, 30], "x_val": 1, "xarrai": [20, 27], "xavier": 1, "xbnew": [13, 29, 30], "xcode": [0, 20, 22, 27], "xdclassiffierconfus": 10, "xdclassiffierroc": 10, "xg_clf": 10, "xgb": 10, "xgbclassifi": 10, "xgboost": 9, "xgboot": 10, "xgbregressor": 10, "xgparam": 10, "xgtree": 10, "xi": [8, 13, 30], "xi_": 8, "xi_1": 8, "xi_i": 8, "xk": 8, "xla": [13, 20, 27], "xlabel": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 13, 24, 27, 28, 29, 30], "xlim": [6, 10], "xm": 9, "xmesh": 13, "xnew": [0, 13, 27, 29, 30], "xp": 24, "xpanda": [0, 28], "xpd": [5, 11, 28], "xplot": 0, "xscale": [0, 28], "xsr": 9, "xt_x": [13, 29, 30], "xtest": 6, "xtick": [3, 6, 8, 9], "xtrain": 6, "xu": [0, 27], "xx": [0, 21, 27], "xy": [0, 6, 8, 21, 27], "xytext": 8, "xz": [21, 27], "y": [0, 1, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 21, 22, 24, 27, 28, 29, 30], "y1": 4, "y2": 4, "y3": 4, "y_": [0, 1, 5, 6, 10, 11, 21, 27, 28], "y_0": [0, 5, 11, 21, 27, 28], "y_1": [0, 5, 8, 9, 11, 13, 21, 27, 28, 29, 30], "y_1y_1": 8, "y_1y_1k": 8, "y_1y_2": 8, "y_1y_2k": 8, "y_1y_n": 8, "y_1y_nk": 8, "y_2": [0, 5, 8, 9, 11, 21, 27, 28], "y_2y_1": 8, "y_2y_1k": 8, "y_2y_2": 8, "y_2y_2k": 8, "y_3": [0, 9, 21], "y_4": 21, "y_center": [18, 30], "y_data": [0, 1, 5, 6, 27, 28, 29, 30], "y_data_ful": 1, "y_decis": 8, "y_fit": [0, 28], "y_i": [0, 1, 5, 6, 7, 8, 9, 10, 11, 12, 13, 19, 21, 22, 27, 28, 29, 30], "y_if_": 10, "y_ix_": [0, 27], "y_ix_i": [7, 8, 13, 28, 29, 30], "y_iy_jk": 8, "y_j": [6, 8, 12, 22], "y_k": 12, "y_m": 21, "y_mean": [18, 30], "y_model": [0, 4, 5, 6, 27, 28, 29, 30], "y_n": [8, 13, 29, 30], "y_ny_1": 8, "y_ny_1k": 8, "y_ny_2": 8, "y_ny_2k": 8, "y_ny_n": 8, "y_ny_nk": 8, "y_offset": [6, 17, 28, 30], "y_plot": 9, "y_pred": [0, 1, 4, 6, 7, 8, 9, 10, 28, 30], "y_pred1": 9, "y_pred2": 9, "y_pred_rf": 10, "y_pred_tre": 10, "y_proba": [7, 10], "y_sampl": 19, "y_scaler": [6, 28, 30], "y_test": [0, 1, 3, 4, 5, 6, 7, 9, 10, 11, 15, 16, 17, 19, 28, 29, 30], "y_test_onehot": 1, "y_test_predict": [], "y_tot": 4, "y_train": [0, 1, 3, 4, 5, 6, 7, 9, 10, 11, 15, 16, 17, 19, 27, 28, 29, 30], "y_train_mean": [6, 28, 30], "y_train_onehot": 1, "y_train_predict": [], "y_train_scal": [6, 28, 30], "y_val": 1, "ye": [3, 6, 7], "year": [0, 20, 27], "yet": [0, 1, 6, 8, 11, 13, 27], "yi": [13, 30], "yield": [0, 2, 5, 6, 8, 10, 12, 13, 14, 21, 24, 27, 29, 30], "yk": 8, "ylabel": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 13, 24, 27, 28, 29, 30], "ylim": [3, 6], "ym": 9, "ymesh": 13, "yn": 0, "yo": [8, 9, 10], "yoshua": [1, 26], "you": [0, 1, 3, 4, 5, 6, 8, 9, 10, 11, 13, 15, 16, 17, 18, 19, 20, 21, 22, 24, 25, 26, 27, 28, 29, 30], "young": 0, "your": [1, 2, 4, 5, 6, 8, 11, 13, 15, 17, 19, 20, 21, 27, 29, 30], "your_model_object": 16, "yourself": [11, 13, 27, 29], "youtu": [28, 29], "youtub": [20, 29, 30], "ypred": 6, "ypredict": [0, 13, 27, 28, 29, 30], "ypredict2": [13, 29, 30], "ypredictlasso": [5, 29], "ypredictol": [0, 5, 29], "ypredictown": [6, 28, 30], "ypredictownridg": [6, 28, 29, 30], "ypredictridg": [0, 5, 6, 28, 29, 30], "ypredictskl": [6, 28, 30], "ytest": 6, "ytick": [3, 6, 8, 9], "ytild": [0, 6, 27, 28], "ytildelasso": [5, 29], "ytildenp": [0, 27, 28], "ytildeol": [0, 5, 29], "ytildeownridg": [6, 28, 29, 30], "ytilderidg": [5, 6, 28, 29, 30], "ytrain": 6, "yuxi": 27, "yx": [21, 27], "yy": [21, 27], "yz": [21, 27], "z": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 11, 12, 13, 21, 24, 27, 28], "z_": [1, 2, 12, 21, 27], "z_0": [21, 27], "z_1": [21, 27], "z_2": [21, 27], "z_c": 1, "z_h": 1, "z_hidden": 2, "z_i": [1, 12], "z_j": [1, 12], "z_k": [12, 28], "z_m": 1, "z_mod": 9, "z_o": 1, "z_output": 2, "zaman": 24, "zaxi": 6, "zero": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 16, 17, 18, 19, 21, 22, 24, 27, 28, 29, 30], "zeros_lik": 4, "zeroth": 28, "zfill": 4, "zip": [4, 6], "zm_h": [0, 27], "zn": [], "zone": [], "zoom": 27, "zx": [21, 27], "zy": [21, 27], "zz": [21, 27], "\u00f8yvind": [6, 28, 30]}, "titles": ["3. Linear Regression", "14. Building a Feed Forward Neural Network", "15. Solving Differential Equations with Deep Learning", "16. Convolutional Neural Networks", "17. Recurrent neural networks: Overarching view", "4. Ridge and Lasso Regression", "5. Resampling Methods", "6. Logistic Regression", "8. Support Vector Machines, overarching aims", "9. Decision trees, overarching aims", "10. Ensemble Methods: From a Single Tree to Many Trees and Extreme Boosting, Meet the Jungle of Methods", "11. Basic ideas of the Principal Component Analysis (PCA)", "13. Neural networks", "7. Optimization, the central part of any Machine Learning algortithm", "12. Clustering and Unsupervised Learning", "Exercises week 34", "Exercises week 35", "Exercises week 36", "Exercises week 37", "Exercises week 37", "Applied Data Analysis and Machine Learning", "2. Linear Algebra, Handling of Arrays and more Python Features", "Project 1 on Machine Learning, deadline October 6 (midnight), 2025", "Course setting", "1. Elements of Probability Theory and Statistical Data Analysis", "Teachers and Grading", "Textbooks", "Week 34: Introduction to the course, Logistics and Practicalities", "Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression", "Week 36: Linear Regression and Gradient descent", "Week 37: Statistical interpretations and Resampling Methods"], "titleterms": {"": [8, 10, 29, 30], "1": [0, 15, 16, 17, 18, 19, 22, 28], "15": 19, "19": 19, "1a": 18, "2": [0, 15, 16, 17, 18, 19, 27, 28, 29], "2023": 25, "2025": 22, "2a": [], "2b": [], "3": [0, 15, 16, 17, 18, 19, 28], "34": [15, 27], "35": [16, 28], "36": [17, 29], "37": [18, 19, 30], "3a": 18, "3b": 18, "4": [0, 15, 16, 17, 18, 19, 28], "4a": 18, "4b": 18, "5": [0, 16, 18], "6": 22, "8": 30, "A": [0, 1, 4, 8, 9, 27], "And": [27, 28, 30], "But": 30, "In": 25, "Ising": 6, "The": [0, 1, 2, 3, 5, 6, 7, 8, 9, 11, 12, 15, 20, 27, 28, 29, 30], "To": 27, "With": [4, 29], "about": [27, 28, 29], "abov": 29, "accuraci": 30, "activ": [1, 12], "ad": [0, 6, 22, 27, 28], "adaboost": 10, "adagrad": [13, 30], "adam": [13, 30], "adapt": [10, 30], "adjust": 1, "advanc": 22, "adversari": 4, "again": [3, 9], "ai": [22, 27], "aim": [8, 9, 27], "aka": 27, "al": 30, "algebra": [21, 27], "algorithm": [9, 10, 11, 12, 27, 28, 29, 30], "algortithm": [13, 29], "all": 8, "an": [0, 4, 10, 15, 27], "analys": [5, 28, 29], "analysi": [0, 5, 6, 11, 20, 22, 24, 27, 28, 29], "analyt": [0, 16, 18], "ani": [13, 29], "anoth": [9, 29], "appli": 20, "approach": [0, 8, 14, 27, 30], "approxim": 12, "architectur": 1, "arrai": [21, 27], "assist": 25, "autocorrel": 24, "autograd": [2, 13, 30], "automat": [13, 30], "b": 22, "back": [1, 11, 12, 28, 29], "background": [20, 22], "bag": 10, "base": [13, 30], "basic": [0, 5, 7, 9, 10, 11, 21, 28, 29], "batch": [1, 30], "bay": 5, "befor": 11, "better": 8, "bia": [6, 19, 22], "binari": 1, "bind": 27, "bird": 10, "boldsymbol": [18, 28], "boost": 10, "bootstrap": [6, 10], "boston": [], "breast": 1, "brief": 27, "bring": 12, "build": [1, 3, 9], "c": [22, 27], "calcul": [18, 28, 29], "can": [27, 30], "cancer": [1, 7, 9, 11], "cart": 9, "case": [8, 10, 24, 28, 29], "central": [13, 20, 24, 29], "chain": 12, "challeng": 30, "chang": 10, "channel": 27, "chi": [0, 27], "choic": 17, "choos": [1, 30], "cifar01": 3, "classic": 11, "classif": [1, 9, 10], "classifi": 8, "clip": 1, "cluster": 14, "cnn": 3, "code": [1, 2, 5, 9, 11, 12, 13, 14, 15, 16, 22, 27, 28, 29, 30], "collect": [1, 3], "commun": 27, "compar": [2, 10, 16], "comparison": 29, "compet": 30, "complet": 28, "complex": [0, 6, 22, 28], "complic": 6, "compon": 11, "comput": [9, 19, 30], "computerlab": 27, "con": [9, 30], "concept": 24, "condit": 29, "conjug": 13, "contn": 27, "converg": 30, "convex": [8, 13, 29], "convolut": [3, 12], "correl": [11, 28], "correspond": [], "cost": [1, 10, 28, 29, 30], "cours": [20, 23, 26, 27], "covari": [5, 11, 24, 28], "cover": 27, "creat": 16, "cross": [6, 22], "cython": 27, "d": 22, "data": [0, 1, 3, 6, 7, 9, 11, 15, 17, 18, 20, 24, 27, 28], "dataset": [1, 3, 18], "david": 27, "deadlin": [22, 27], "deadllin": 25, "decai": [2, 30], "decis": [9, 10], "decomposit": [5, 11, 21, 28, 29], "deeep": [], "deep": [1, 2, 27], "defin": [1, 27], "degre": [0, 17, 28], "deliver": [15, 16], "deliveri": 22, "dens": 0, "deriv": [5, 12, 16, 17, 19, 28, 29, 30], "descent": [2, 10, 13, 18, 22, 29, 30], "design": 28, "detail": [3, 27], "develop": 1, "diagon": 11, "differ": [8, 30], "differenti": [2, 13, 30], "diffus": 2, "dimension": [2, 3, 8, 18], "disadvantag": 9, "discret": 24, "discrimin": 27, "distribut": [5, 24], "do": [1, 30], "doe": [28, 29], "domain": 24, "down": 1, "dropout": 1, "e": 22, "economi": [28, 29], "electron": 22, "element": [0, 24, 27], "elimin": 21, "energi": 27, "ensembl": 10, "entropi": 9, "environ": [0, 15], "equat": [0, 2, 12, 28, 29], "error": [0, 10, 27, 28, 29], "essenti": 27, "et": 30, "etc": 27, "euler": 2, "evalu": 1, "exampl": [1, 2, 3, 4, 6, 7, 8, 9, 10, 27, 28, 29, 30], "exercis": [0, 6, 15, 16, 17, 18, 19, 28], "expect": [19, 24], "experi": 24, "explor": 0, "exponenti": 2, "express": [16, 17, 19, 28], "extend": 29, "extrapol": 4, "extrem": [10, 27], "ey": 10, "f": 22, "fall": 25, "famili": [1, 27], "famou": 21, "fantast": [28, 29], "featur": [9, 16, 21, 28], "feed": [1, 12], "final": [12, 28, 30], "find": [16, 18], "fine": 1, "first": [4, 12, 27, 29], "fit": [0, 10, 15, 16, 27, 29], "fix": [28, 29, 30], "forc": 3, "forest": 10, "form": 18, "format": [22, 27], "formula": 18, "forward": [1, 2, 12], "foster": 27, "fourier": 3, "frank": 6, "freedom": [0, 17, 28], "frequent": [28, 30], "frequentist": [0, 27], "from": [5, 10, 12, 27, 28, 29, 30], "full": 2, "function": [0, 1, 6, 7, 8, 10, 11, 12, 13, 22, 24, 27, 28, 29, 30], "further": [3, 5, 28, 29], "g": 22, "gan": 4, "gaussian": 21, "gd": [13, 30], "gener": [4, 9, 27], "geometr": [11, 29], "gini": 9, "github": 15, "goal": [15, 16, 17, 18, 19], "good": [0, 27], "goodfellow": 30, "grade": [25, 27], "gradient": [1, 2, 10, 13, 18, 22, 29, 30], "growth": 2, "h": 22, "ha": 20, "handl": [21, 27], "hessian": [28, 29, 30], "hidden": 2, "hous": [], "how": 16, "hyperparamet": [1, 17], "hyperplan": 8, "i": [0, 1, 27], "id3": 9, "idea": 11, "ideal": 29, "ii": 27, "illustr": 29, "implement": [1, 16, 17, 18], "implic": [5, 28, 29], "import": [5, 21, 27, 28, 29], "improv": [1, 30], "includ": [13, 22, 30], "increment": 11, "index": 9, "inform": 25, "input": 2, "instal": [20, 22, 27], "instructor": 25, "interpret": [5, 11, 27, 28, 29, 30], "introduc": [11, 13, 28], "introduct": [0, 6, 20, 21, 22, 27], "invers": [5, 21], "invert": [28, 29], "iter": 10, "its": 28, "jacobian": 28, "jax": 13, "julia": 27, "jungl": 10, "kera": [1, 3], "kernel": [8, 11], "lab": [29, 30], "lagrangian": 8, "lasso": [5, 6, 22, 28, 29], "last": [28, 30], "later": [5, 28, 29], "layer": [1, 2, 3, 12], "learn": [0, 1, 2, 11, 13, 14, 15, 16, 17, 18, 19, 20, 22, 27, 28, 29, 30], "least": [5, 6, 16, 19, 22, 27, 28, 29, 30], "lectur": [27, 29, 30], "level": 10, "librari": [20, 27], "likelihood": 7, "limit": [1, 13, 24, 29, 30], "linear": [0, 8, 13, 15, 21, 27, 28, 29], "link": [5, 11, 26, 28], "literatur": 22, "logist": [7, 27], "loss": [28, 29, 30], "lu": 21, "machin": [0, 8, 13, 20, 22, 27, 29], "main": [24, 27], "make": [0, 9, 10, 28], "mani": [10, 12], "mass": 27, "materi": [22, 27, 28, 29, 30], "math": [5, 28, 29], "mathemat": [3, 5, 8, 28, 29], "matric": [5, 21, 27], "matrix": [1, 5, 11, 12, 16, 21, 27, 28, 29, 30], "matter": 0, "max": 28, "mean": [0, 28, 29], "meet": [5, 10, 24, 27, 28], "mercer": 8, "method": [6, 9, 10, 13, 22, 27, 29, 30], "midnight": 22, "min": 28, "mini": 30, "minibatch": 30, "minim": 27, "ml": 27, "mlp": 12, "mnist": [3, 4], "model": [0, 1, 4, 6, 12, 15, 17, 27], "moment": 30, "momentum": [13, 22, 30], "mondai": [29, 30], "moon": [8, 9], "more": [3, 6, 21, 22, 27, 28, 29, 30], "motiv": 30, "multilay": 12, "multipl": [1, 3, 17], "multipli": 8, "need": [22, 27], "network": [1, 2, 3, 4, 7, 12, 27], "neural": [1, 2, 3, 4, 7, 12, 27], "new": [4, 18], "newton": [29, 30], "non": 8, "none": 30, "normal": [0, 1], "notat": 12, "note": [22, 28, 29], "now": [1, 9, 13, 29, 30], "nuclear": [0, 27], "numba": 27, "number": [0, 2, 24, 28, 30], "numer": [2, 22, 24], "numpi": [21, 27], "object": 3, "obtain": 11, "octob": 22, "od": 2, "off": [6, 19, 22], "ol": [5, 6, 15, 16, 18, 22, 29], "one": [2, 12, 18, 29], "oper": 21, "optim": [1, 8, 13, 18, 20, 27, 28, 29, 30], "order": [13, 18, 30], "ordinari": [5, 6, 16, 19, 22, 27, 28, 29, 30], "organ": [0, 27], "oslo": 26, "other": [4, 9, 11, 12, 21, 22, 27], "our": [0, 4, 5, 11, 13, 22, 27, 28, 29], "outcom": [20, 27], "output": 2, "overarch": [0, 4, 8, 9, 27, 28], "overview": [10, 27, 30], "own": [0, 10, 11, 22, 27, 28], "packag": [21, 27], "panda": [27, 28], "paramet": [27, 28, 30], "paramt": 18, "part": [13, 20, 22, 29], "partial": 2, "pass": 1, "pca": 11, "pdf": 24, "perceptron": 12, "perform": [1, 9], "period": 3, "perspect": 1, "plan": [28, 29, 30], "plethora": 27, "point": 4, "poisson": 2, "polynomi": [3, 16, 18, 29], "popul": 2, "popular": 27, "practic": [13, 25, 27, 30], "pre": [1, 3], "preambl": 22, "predict": 4, "preprocess": [28, 30], "prerequisit": [3, 20, 27], "princip": 11, "principl": 3, "pro": [9, 30], "probabl": [5, 24], "problem": [1, 2, 13, 27, 28, 29, 30], "procedur": [9, 27], "process": [1, 3], "program": [2, 13, 22, 29, 30], "project": [6, 22, 25, 27], "prop": [13, 30], "propag": [1, 12], "properti": [5, 24, 28, 29, 30], "python": [0, 9, 15, 20, 21, 27], "quick": 8, "r": 27, "random": [10, 11, 24], "raphson": 29, "rate": [22, 30], "read": [9, 27, 28, 30], "real": [6, 27], "recommend": [27, 28], "recurr": [4, 12], "reduc": [0, 28], "reduct": 3, "refer": 22, "reformul": 2, "regress": [0, 5, 6, 7, 9, 10, 13, 15, 17, 18, 19, 22, 27, 28, 29, 30], "regular": 1, "relat": [], "relev": [26, 28], "relu": 1, "remark": 3, "remind": [6, 8, 27, 28, 29, 30], "replac": [13, 30], "report": 22, "repositori": 15, "requir": [2, 20], "resampl": [6, 19, 22, 30], "rescal": [6, 28], "residu": [28, 29], "resourc": 2, "result": [28, 29], "revisit": [13, 29, 30], "rewrit": [27, 28], "ridg": [0, 5, 6, 17, 18, 19, 22, 28, 29, 30], "rm": [13, 30], "rmsprop": 30, "rule": [12, 30], "rung": 22, "same": [13, 30], "sampl": 11, "scale": [17, 18, 28, 30], "schedul": 27, "schemat": 9, "scheme": 2, "scienc": 27, "scikit": [0, 1, 11, 27, 28, 29, 30], "second": [13, 18, 30], "semest": 25, "sensit": 29, "septemb": [19, 29, 30], "session": [29, 30], "set": [0, 2, 3, 9, 12, 15, 23, 27, 28, 29], "setup": 15, "sgd": [13, 30], "should": 1, "similar": [13, 30], "simpl": [0, 4, 9, 13, 18, 27, 28, 29, 30], "simplest": 18, "singl": 10, "singular": [5, 11, 28, 29], "size": [28, 29, 30], "sklearn": 16, "slightli": 30, "sneak": 30, "soft": 8, "softmax": 1, "softwar": [22, 27], "solv": [2, 29], "solver": 13, "some": [13, 21, 28, 29], "specifi": 2, "split": [0, 15, 28], "squar": [0, 5, 6, 10, 16, 19, 22, 27, 28, 29, 30], "standard": [13, 28], "state": 0, "statist": [5, 6, 20, 24, 27, 30], "steepest": [10, 13, 29], "step": 30, "stochast": [13, 22, 24, 30], "stop": 30, "strongli": 27, "suggest": 27, "summari": [25, 27], "superposit": 3, "supervis": 1, "support": 8, "svd": [5, 28, 29], "synthet": 18, "systemat": 3, "t": 28, "take": 16, "taken": [27, 30], "teach": 25, "teacher": [25, 27], "technic": 28, "techniqu": [6, 11, 22], "technologi": 20, "tensorflow": [1, 3], "tent": [25, 27], "test": [0, 1, 15, 17, 28], "text": 27, "textbook": [26, 27], "than": 29, "theorem": [5, 8, 11, 12, 24], "theori": 24, "theta": 18, "thi": 27, "time": 30, "tip": [13, 30], "togeth": 12, "tool": [22, 27], "top": 1, "topic": 27, "toward": 11, "trade": [6, 19, 22], "tradeoff": 6, "train": [0, 1, 4, 15, 27, 28], "transform": 3, "tree": [9, 10], "tuesdai": 29, "tune": 1, "two": [3, 8, 20], "type": [2, 4, 12, 27], "uio": 27, "univers": [12, 26], "unsupervis": 14, "up": [0, 2, 9, 12, 15, 27, 28, 29], "updat": [22, 30], "us": [0, 1, 2, 3, 7, 13, 16, 18, 20, 22, 27, 28, 29, 30], "v": 3, "valid": [6, 22], "valu": [5, 11, 19, 24, 28, 29], "vari": 30, "variabl": [24, 29], "varianc": [6, 19, 22], "variou": 0, "vector": [8, 12, 16, 21, 27, 28], "versu": 27, "video": 30, "view": [0, 4, 10, 28], "virtual": 15, "visual": [1, 9], "wai": [9, 22], "wave": 2, "we": [27, 30], "wednesdai": 29, "week": [15, 16, 17, 18, 19, 27, 28, 29, 30], "weekli": [], "what": [0, 27, 28, 29], "when": 30, "which": [1, 30], "why": 27, "wisconsin": 7, "write": [4, 11, 22, 29], "x": 28, "xgboost": 10, "yet": 29, "your": [0, 10, 16, 18, 22, 28]}}) \ No newline at end of file +Search.setIndex({"alltitles": {"1a)": [[18, "a"]], "3a)": [[18, "id1"]], "3b)": [[18, "b"]], "4a)": [[18, "id2"]], "4b)": [[18, "id3"]], "A Classification Tree": [[9, "a-classification-tree"]], "A Frequentist approach to data analysis": [[0, "a-frequentist-approach-to-data-analysis"], [27, "a-frequentist-approach-to-data-analysis"]], "A better approach": [[8, "a-better-approach"]], "A first summary": [[27, "a-first-summary"]], "A quick Reminder on Lagrangian Multipliers": [[8, "a-quick-reminder-on-lagrangian-multipliers"]], "A simple example": [[4, "a-simple-example"]], "A soft classifier": [[8, "a-soft-classifier"]], "A top-down perspective on Neural networks": [[1, "a-top-down-perspective-on-neural-networks"]], "ADAM algorithm, taken from Goodfellow et al": [[30, "adam-algorithm-taken-from-goodfellow-et-al"]], "ADAM optimizer": [[13, "adam-optimizer"], [30, "id2"]], "Accuracy": [[30, "accuracy"]], "Activation functions": [[12, "activation-functions"]], "AdaGrad Properties": [[30, "adagrad-properties"]], "AdaGrad Update Rule Derivation": [[30, "adagrad-update-rule-derivation"]], "AdaGrad algorithm, taken from Goodfellow et al": [[30, "adagrad-algorithm-taken-from-goodfellow-et-al"]], "Adam Optimizer": [[30, "adam-optimizer"]], "Adam vs. AdaGrad and RMSProp": [[30, "adam-vs-adagrad-and-rmsprop"]], "Adam: Bias Correction": [[30, "adam-bias-correction"]], "Adam: Exponential Moving Averages (Moments)": [[30, "adam-exponential-moving-averages-moments"]], "Adam: Update Rule Derivation": [[30, "adam-update-rule-derivation"]], "Adaptive boosting: AdaBoost, Basic Algorithm": [[10, "adaptive-boosting-adaboost-basic-algorithm"]], "Adaptivity Across Dimensions": [[30, "adaptivity-across-dimensions"]], "Adding error analysis and training set up": [[27, "adding-error-analysis-and-training-set-up"], [28, "adding-error-analysis-and-training-set-up"]], "Adjust hyperparameters": [[1, "adjust-hyperparameters"]], "Algorithms and codes for Adagrad, RMSprop and Adam": [[30, "algorithms-and-codes-for-adagrad-rmsprop-and-adam"]], "Algorithms for Setting up Decision Trees": [[9, "algorithms-for-setting-up-decision-trees"]], "An Overview of Ensemble Methods": [[10, "an-overview-of-ensemble-methods"]], "An extrapolation example": [[4, "an-extrapolation-example"]], "An optimization/minimization problem": [[27, "an-optimization-minimization-problem"]], "And finally \\boldsymbol{X}\\boldsymbol{X}^T": [[28, "and-finally-boldsymbol-x-boldsymbol-x-t"]], "And finally ADAM": [[30, "and-finally-adam"]], "And what about using neural networks?": [[27, "and-what-about-using-neural-networks"]], "Another Example, now with a polynomial fit": [[29, "another-example-now-with-a-polynomial-fit"]], "Another example, the moons again": [[9, "another-example-the-moons-again"]], "Applied Data Analysis and Machine Learning": [[20, null]], "Autocorrelation function": [[24, "autocorrelation-function"]], "Automatic differentiation": [[13, "automatic-differentiation"]], "Back to Ridge and LASSO Regression": [[28, "back-to-ridge-and-lasso-regression"], [29, "back-to-ridge-and-lasso-regression"]], "Back to the Cancer Data": [[11, "back-to-the-cancer-data"]], "Background literature": [[22, "background-literature"]], "Bagging": [[10, "bagging"]], "Bagging Examples": [[10, "bagging-examples"]], "Basic Matrix Features": [[21, "basic-matrix-features"]], "Basic ideas of the Principal Component Analysis (PCA)": [[11, null]], "Basic math of the SVD": [[5, "basic-math-of-the-svd"], [28, "basic-math-of-the-svd"], [29, "basic-math-of-the-svd"]], "Basics": [[7, "basics"]], "Basics of a tree": [[9, "basics-of-a-tree"]], "Batch Normalization": [[1, "batch-normalization"]], "Batches and mini-batches": [[30, "batches-and-mini-batches"]], "Bayes\u2019 Theorem and Ridge and Lasso Regression": [[5, "bayes-theorem-and-ridge-and-lasso-regression"]], "Boosting, a Bird\u2019s Eye View": [[10, "boosting-a-bird-s-eye-view"]], "Bootstrap": [[6, "bootstrap"]], "Bringing it together, first back propagation equation": [[12, "bringing-it-together-first-back-propagation-equation"]], "Building a Feed Forward Neural Network": [[1, null]], "Building a tree, regression": [[9, "building-a-tree-regression"]], "Building neural networks in Tensorflow and Keras": [[1, "building-neural-networks-in-tensorflow-and-keras"]], "But none of these can compete with Newton\u2019s method": [[30, "but-none-of-these-can-compete-with-newton-s-method"]], "CNNs in more detail, building convolutional neural networks in Tensorflow and Keras": [[3, "cnns-in-more-detail-building-convolutional-neural-networks-in-tensorflow-and-keras"]], "Cancer Data again now with Decision Trees and other Methods": [[9, "cancer-data-again-now-with-decision-trees-and-other-methods"]], "Challenge: Choosing a Fixed Learning Rate": [[30, "challenge-choosing-a-fixed-learning-rate"]], "Choose cost function and optimizer": [[1, "choose-cost-function-and-optimizer"]], "Classical PCA Theorem": [[11, "classical-pca-theorem"]], "Clustering and Unsupervised Learning": [[14, null]], "Code for SVD and Inversion of Matrices": [[5, "code-for-svd-and-inversion-of-matrices"]], "Code with a Number of Minibatches which varies": [[30, "code-with-a-number-of-minibatches-which-varies"]], "Codes and Approaches": [[14, "codes-and-approaches"]], "Codes for the SVD": [[5, "codes-for-the-svd"], [28, "codes-for-the-svd"], [29, "codes-for-the-svd"]], "Coding Setup and Linear Regression": [[15, "coding-setup-and-linear-regression"]], "Collect and pre-process data": [[1, "collect-and-pre-process-data"]], "Communication channels": [[27, "communication-channels"]], "Compare Bagging on Trees with Random Forests": [[10, "compare-bagging-on-trees-with-random-forests"]], "Comparing with a numerical scheme": [[2, "comparing-with-a-numerical-scheme"]], "Comparison with OLS": [[29, "comparison-with-ols"]], "Computation of gradients": [[30, "computation-of-gradients"]], "Computing the Gini index": [[9, "computing-the-gini-index"]], "Conditions on convex functions": [[29, "conditions-on-convex-functions"]], "Conjugate gradient method": [[13, "conjugate-gradient-method"]], "Convergence rates": [[30, "convergence-rates"]], "Convex function": [[29, "convex-function"]], "Convex functions": [[13, "convex-functions"], [29, "convex-functions"]], "Convolution Examples: Polynomial multiplication": [[3, "convolution-examples-polynomial-multiplication"]], "Convolution Examples: Principle of Superposition and Periodic Forces (Fourier Transforms)": [[3, "convolution-examples-principle-of-superposition-and-periodic-forces-fourier-transforms"]], "Convolutional Neural Network": [[12, "convolutional-neural-network"]], "Convolutional Neural Networks": [[3, null]], "Correlation Function and Design/Feature Matrix": [[28, "correlation-function-and-design-feature-matrix"]], "Correlation Matrix": [[11, "correlation-matrix"], [28, "correlation-matrix"]], "Correlation Matrix with Pandas": [[28, "correlation-matrix-with-pandas"]], "Course Format": [[27, "course-format"]], "Course setting": [[23, null]], "Covariance Matrix Examples": [[28, "covariance-matrix-examples"]], "Covariance and Correlation Matrix": [[28, "covariance-and-correlation-matrix"]], "Cross-validation": [[6, "cross-validation"]], "Deadlines for projects (tentative)": [[27, "deadlines-for-projects-tentative"]], "Decision trees, overarching aims": [[9, null]], "Deep learning methods": [[27, "deep-learning-methods"]], "Define model and architecture": [[1, "define-model-and-architecture"]], "Defining the cost function": [[1, "defining-the-cost-function"]], "Deliverables": [[15, "deliverables"], [16, "deliverables"]], "Derivation of the AdaGrad Algorithm": [[30, "derivation-of-the-adagrad-algorithm"]], "Derivatives and the chain rule": [[12, "derivatives-and-the-chain-rule"]], "Derivatives, example 1": [[28, "derivatives-example-1"]], "Deriving OLS from a probability distribution": [[5, "deriving-ols-from-a-probability-distribution"]], "Deriving and Implementing Ordinary Least Squares": [[16, "deriving-and-implementing-ordinary-least-squares"]], "Deriving and Implementing Ridge Regression": [[17, "deriving-and-implementing-ridge-regression"]], "Deriving the Lasso Regression Equations": [[28, "deriving-the-lasso-regression-equations"], [29, "deriving-the-lasso-regression-equations"], [29, "id6"]], "Deriving the Ridge Regression Equations": [[28, "deriving-the-ridge-regression-equations"], [29, "deriving-the-ridge-regression-equations"], [29, "id3"]], "Deriving the back propagation code for a multilayer perceptron model": [[12, "deriving-the-back-propagation-code-for-a-multilayer-perceptron-model"]], "Developing a code for doing neural networks with back propagation": [[1, "developing-a-code-for-doing-neural-networks-with-back-propagation"]], "Diagonalize the sample covariance matrix to obtain the principal components": [[11, "diagonalize-the-sample-covariance-matrix-to-obtain-the-principal-components"]], "Different kernels and Mercer\u2019s theorem": [[8, "different-kernels-and-mercer-s-theorem"]], "Disadvantages": [[9, "disadvantages"]], "Discriminative Modeling": [[27, "discriminative-modeling"]], "Domains and probabilities": [[24, "domains-and-probabilities"]], "Dropout": [[1, "dropout"]], "Economy-size SVD": [[28, "economy-size-svd"], [29, "economy-size-svd"]], "Elements of Probability Theory and Statistical Data Analysis": [[24, null]], "Ensemble Methods: From a Single Tree to Many Trees and Extreme Boosting, Meet the Jungle of Methods": [[10, null]], "Entropy and the ID3 algorithm": [[9, "entropy-and-the-id3-algorithm"]], "Essential elements of ML": [[27, "essential-elements-of-ml"]], "Evaluate model performance on test data": [[1, "evaluate-model-performance-on-test-data"]], "Example 2": [[28, "example-2"]], "Example 3": [[28, "example-3"]], "Example 4": [[28, "example-4"]], "Example Matrix": [[28, "example-matrix"], [29, "example-matrix"]], "Example of discriminative modeling, taken from Generative Deep Learning by David Foster": [[27, "example-of-discriminative-modeling-taken-from-generative-deep-learning-by-david-foster"]], "Example of generative modeling, taken from Generative Deep Learning by David Foster": [[27, "example-of-generative-modeling-taken-from-generative-deep-learning-by-david-foster"]], "Example of own Standard scaling": [[28, "example-of-own-standard-scaling"]], "Example relevant for the exercises": [[28, "example-relevant-for-the-exercises"]], "Example: Exponential decay": [[2, "example-exponential-decay"]], "Example: Population growth": [[2, "example-population-growth"]], "Example: The diffusion equation": [[2, "example-the-diffusion-equation"]], "Example: binary classification problem": [[1, "example-binary-classification-problem"]], "Examples": [[27, "examples"]], "Examples of likelihood functions used in logistic regression and neural networks": [[7, "examples-of-likelihood-functions-used-in-logistic-regression-and-neural-networks"]], "Exercise 1 - Choice of model and degrees of freedom": [[17, "exercise-1-choice-of-model-and-degrees-of-freedom"]], "Exercise 1 - Finding the derivative of Matrix-Vector expressions": [[16, "exercise-1-finding-the-derivative-of-matrix-vector-expressions"]], "Exercise 1 - Github Setup": [[15, "exercise-1-github-setup"]], "Exercise 1, scale your data": [[18, "exercise-1-scale-your-data"]], "Exercise 1: Expectation values for ordinary least squares expressions": [[19, "exercise-1-expectation-values-for-ordinary-least-squares-expressions"]], "Exercise 1: Setting up various Python environments": [[0, "exercise-1-setting-up-various-python-environments"]], "Exercise 2 - Deriving the expression for OLS": [[16, "exercise-2-deriving-the-expression-for-ols"]], "Exercise 2 - Deriving the expression for Ridge Regression": [[17, "exercise-2-deriving-the-expression-for-ridge-regression"]], "Exercise 2 - Setting up a Github repository": [[15, "exercise-2-setting-up-a-github-repository"]], "Exercise 2, calculate the gradients": [[18, "exercise-2-calculate-the-gradients"]], "Exercise 2: Expectation values for Ridge regression": [[19, "exercise-2-expectation-values-for-ridge-regression"]], "Exercise 2: making your own data and exploring scikit-learn": [[0, "exercise-2-making-your-own-data-and-exploring-scikit-learn"]], "Exercise 3 - Creating feature matrix and implementing OLS using the analytical expression": [[16, "exercise-3-creating-feature-matrix-and-implementing-ols-using-the-analytical-expression"]], "Exercise 3 - Fitting an OLS model to data": [[15, "exercise-3-fitting-an-ols-model-to-data"]], "Exercise 3 - Scaling data": [[17, "exercise-3-scaling-data"]], "Exercise 3 - Setting up a Python virtual environment": [[15, "exercise-3-setting-up-a-python-virtual-environment"]], "Exercise 3, using the analytical formulae for OLS and Ridge regression to find the optimal paramters \\boldsymbol{\\theta}": [[18, "exercise-3-using-the-analytical-formulae-for-ols-and-ridge-regression-to-find-the-optimal-paramters-boldsymbol-theta"]], "Exercise 3: Deriving the expression for the Bias-Variance Trade-off": [[19, "exercise-3-deriving-the-expression-for-the-bias-variance-trade-off"]], "Exercise 3: Normalizing our data": [[0, "exercise-3-normalizing-our-data"]], "Exercise 4 - Fitting a polynomial": [[16, "exercise-4-fitting-a-polynomial"]], "Exercise 4 - Implementing Ridge Regression": [[17, "exercise-4-implementing-ridge-regression"]], "Exercise 4 - Testing multiple hyperparameters": [[17, "exercise-4-testing-multiple-hyperparameters"]], "Exercise 4 - The train-test split": [[15, "exercise-4-the-train-test-split"]], "Exercise 4, Implementing the simplest form for gradient descent": [[18, "exercise-4-implementing-the-simplest-form-for-gradient-descent"]], "Exercise 4: Adding Ridge Regression": [[0, "exercise-4-adding-ridge-regression"]], "Exercise 4: Computing the Bias and Variance": [[19, "exercise-4-computing-the-bias-and-variance"]], "Exercise 5 - Comparing your code with sklearn": [[16, "exercise-5-comparing-your-code-with-sklearn"]], "Exercise 5, Ridge regression and a new Synthetic Dataset": [[18, "exercise-5-ridge-regression-and-a-new-synthetic-dataset"]], "Exercise 5: Analytical exercises": [[0, "exercise-5-analytical-exercises"]], "Exercise: Cross-validation as resampling techniques, adding more complexity": [[6, "exercise-cross-validation-as-resampling-techniques-adding-more-complexity"]], "Exercise: Analysis of real data": [[6, "exercise-analysis-of-real-data"]], "Exercise: Bias-variance trade-off and resampling techniques": [[6, "exercise-bias-variance-trade-off-and-resampling-techniques"]], "Exercise: Lasso Regression on the Franke function with resampling": [[6, "exercise-lasso-regression-on-the-franke-function-with-resampling"]], "Exercise: Ordinary Least Square (OLS) on the Franke function": [[6, "exercise-ordinary-least-square-ols-on-the-franke-function"]], "Exercise: Ridge Regression on the Franke function with resampling": [[6, "exercise-ridge-regression-on-the-franke-function-with-resampling"]], "Exercises": [[0, "exercises"]], "Exercises and Projects": [[6, "exercises-and-projects"]], "Exercises week 34": [[15, null]], "Exercises week 35": [[16, null]], "Exercises week 36": [[17, null]], "Exercises week 37": [[18, null], [19, null]], "Expectation values": [[24, "expectation-values"]], "Extending to more than one variable": [[29, "extending-to-more-than-one-variable"]], "Extremely useful tools, strongly recommended": [[27, "extremely-useful-tools-strongly-recommended"]], "Feed-forward neural networks": [[12, "feed-forward-neural-networks"]], "Feed-forward pass": [[1, "feed-forward-pass"]], "Final back propagating equation": [[12, "final-back-propagating-equation"]], "Fine-tuning neural network hyperparameters": [[1, "fine-tuning-neural-network-hyperparameters"]], "Fitting an Equation of State for Dense Nuclear Matter": [[0, "fitting-an-equation-of-state-for-dense-nuclear-matter"]], "Fixing the singularity": [[28, "fixing-the-singularity"], [29, "fixing-the-singularity"]], "Format for electronic delivery of report and programs": [[22, "format-for-electronic-delivery-of-report-and-programs"]], "Frequently used scaling functions": [[28, "frequently-used-scaling-functions"], [30, "frequently-used-scaling-functions"]], "From OLS to Ridge and Lasso": [[29, "from-ols-to-ridge-and-lasso"]], "From one to many layers, the universal approximation theorem": [[12, "from-one-to-many-layers-the-universal-approximation-theorem"]], "Functionality in Scikit-Learn": [[28, "functionality-in-scikit-learn"], [30, "functionality-in-scikit-learn"]], "Further Dimensionality Remarks": [[3, "further-dimensionality-remarks"]], "Further properties (important for our analyses later)": [[5, "further-properties-important-for-our-analyses-later"], [28, "further-properties-important-for-our-analyses-later"], [29, "further-properties-important-for-our-analyses-later"]], "Gaussian Elimination": [[21, "gaussian-elimination"]], "General Features": [[9, "general-features"]], "General linear models and linear algebra": [[27, "general-linear-models-and-linear-algebra"]], "Generalizing the fitting procedure as a linear algebra problem": [[27, "generalizing-the-fitting-procedure-as-a-linear-algebra-problem"], [27, "id1"]], "Generative Adversarial Networks": [[4, "generative-adversarial-networks"]], "Generative Models": [[4, "generative-models"]], "Generative Versus Discriminative Modeling": [[27, "generative-versus-discriminative-modeling"]], "Geometric Interpretation and link with Singular Value Decomposition": [[11, "geometric-interpretation-and-link-with-singular-value-decomposition"]], "Gradient Boosting, Classification Example": [[10, "gradient-boosting-classification-example"]], "Gradient Boosting, Examples of Regression": [[10, "gradient-boosting-examples-of-regression"]], "Gradient Clipping": [[1, "gradient-clipping"]], "Gradient Descent Example": [[29, "id1"], [30, "id1"]], "Gradient boosting: Basics with Steepest Descent/Functional Gradient Descent": [[10, "gradient-boosting-basics-with-steepest-descent-functional-gradient-descent"]], "Gradient descent": [[2, "gradient-descent"]], "Gradient descent and Ridge": [[29, "gradient-descent-and-ridge"], [30, "gradient-descent-and-ridge"]], "Gradient descent and revisiting Ordinary Least Squares from last week": [[30, "gradient-descent-and-revisiting-ordinary-least-squares-from-last-week"]], "Gradient descent example": [[29, "gradient-descent-example"], [30, "gradient-descent-example"]], "Grading": [[25, "grading"], [25, "id2"], [27, "grading"]], "How to take derivatives of Matrix-Vector expressions": [[16, "how-to-take-derivatives-of-matrix-vector-expressions"]], "Hyperplanes and all that": [[8, "hyperplanes-and-all-that"]], "Important Matrix and vector handling packages": [[21, "important-matrix-and-vector-handling-packages"]], "Important technicalities: More on Rescaling data": [[28, "important-technicalities-more-on-rescaling-data"]], "Improving gradient descent with momentum": [[30, "improving-gradient-descent-with-momentum"]], "Improving performance": [[1, "improving-performance"]], "In summary": [[25, "in-summary"]], "Including Stochastic Gradient Descent with Autograd": [[13, "including-stochastic-gradient-descent-with-autograd"], [30, "including-stochastic-gradient-descent-with-autograd"]], "Incremental PCA": [[11, "incremental-pca"]], "Installing R, C++, cython or Julia": [[27, "installing-r-c-cython-or-julia"]], "Installing R, C++, cython, Numba etc": [[27, "installing-r-c-cython-numba-etc"]], "Instructor information": [[25, "instructor-information"]], "Interpretations and optimizing our parameters": [[27, "interpretations-and-optimizing-our-parameters"], [27, "id2"], [27, "id3"], [28, "interpretations-and-optimizing-our-parameters"], [28, "id1"], [28, "id2"]], "Interpreting the Ridge results": [[28, "interpreting-the-ridge-results"], [29, "interpreting-the-ridge-results"], [29, "id4"]], "Introducing JAX": [[13, "introducing-jax"]], "Introducing the Covariance and Correlation functions": [[11, "introducing-the-covariance-and-correlation-functions"], [28, "introducing-the-covariance-and-correlation-functions"]], "Introduction": [[0, "introduction"], [6, "introduction"], [20, "introduction"], [21, "introduction"]], "Introduction to numerical projects": [[22, "introduction-to-numerical-projects"]], "Iterative Fitting, Classification and AdaBoost": [[10, "iterative-fitting-classification-and-adaboost"]], "Iterative Fitting, Regression and Squared-error Cost Function": [[10, "iterative-fitting-regression-and-squared-error-cost-function"]], "Kernel PCA": [[11, "kernel-pca"]], "Kernels and non-linearity": [[8, "kernels-and-non-linearity"]], "LU Decomposition, the inverse of a matrix": [[21, "lu-decomposition-the-inverse-of-a-matrix"]], "Lasso Regression": [[29, "lasso-regression"]], "Lasso case": [[29, "lasso-case"]], "Layers": [[1, "layers"]], "Layers used to build CNNs": [[3, "layers-used-to-build-cnns"]], "Learning goals": [[15, "learning-goals"], [16, "learning-goals"], [17, "learning-goals"], [18, "learning-goals"], [19, "learning-goals"]], "Learning outcomes": [[20, "learning-outcomes"], [27, "learning-outcomes"]], "Lectures and ComputerLab": [[27, "lectures-and-computerlab"]], "Limitations of supervised learning with deep networks": [[1, "limitations-of-supervised-learning-with-deep-networks"]], "Linear Algebra, Handling of Arrays and more Python Features": [[21, null]], "Linear Regression": [[0, null]], "Linear Regression Problems": [[28, "linear-regression-problems"], [29, "linear-regression-problems"]], "Linear Regression and the SVD": [[29, "linear-regression-and-the-svd"]], "Linear Regression, basic elements": [[0, "linear-regression-basic-elements"]], "Linking Bayes\u2019 Theorem with Ridge and Lasso Regression": [[5, "linking-bayes-theorem-with-ridge-and-lasso-regression"]], "Linking the regression analysis with a statistical interpretation": [[5, "linking-the-regression-analysis-with-a-statistical-interpretation"]], "Linking with the SVD": [[5, "linking-with-the-svd"], [28, "linking-with-the-svd"]], "Links to relevant courses at the University of Oslo": [[26, "links-to-relevant-courses-at-the-university-of-oslo"]], "Logistic Regression": [[7, null], [7, "id1"]], "MNIST and GANs": [[4, "mnist-and-gans"]], "Machine Learning": [[27, "machine-learning"]], "Machine learning": [[20, "machine-learning"]], "Main textbooks": [[27, "main-textbooks"]], "Making a tree": [[9, "making-a-tree"]], "Making your own Bootstrap: Changing the Level of the Decision Tree": [[10, "making-your-own-bootstrap-changing-the-level-of-the-decision-tree"]], "Making your own test-train splitting": [[28, "making-your-own-test-train-splitting"]], "Material for exercises week 35": [[28, "material-for-exercises-week-35"]], "Material for lab sessions sessions Tuesday and Wednesday": [[29, "material-for-lab-sessions-sessions-tuesday-and-wednesday"]], "Material for lecture Monday September 2": [[29, "material-for-lecture-monday-september-2"]], "Material for lecture Monday September 8": [[30, "material-for-lecture-monday-september-8"]], "Material for the lab sessions": [[30, "material-for-the-lab-sessions"]], "Mathematical Interpretation of Ordinary Least Squares": [[5, "mathematical-interpretation-of-ordinary-least-squares"], [28, "mathematical-interpretation-of-ordinary-least-squares"], [29, "mathematical-interpretation-of-ordinary-least-squares"]], "Mathematical optimization of convex functions": [[8, "mathematical-optimization-of-convex-functions"]], "Mathematics of CNNs": [[3, "mathematics-of-cnns"]], "Mathematics of the SVD and implications": [[5, "mathematics-of-the-svd-and-implications"], [28, "mathematics-of-the-svd-and-implications"], [29, "mathematics-of-the-svd-and-implications"]], "Matrices in Python": [[27, "matrices-in-python"]], "Matrix multiplication": [[1, "matrix-multiplication"]], "Matrix-vector notation and activation": [[12, "matrix-vector-notation-and-activation"]], "Meet the covariance!": [[24, "meet-the-covariance"]], "Meet the Covariance Matrix": [[5, "meet-the-covariance-matrix"], [28, "meet-the-covariance-matrix"]], "Meet the Hessian Matrix": [[28, "meet-the-hessian-matrix"]], "Meet the Pandas": [[27, "meet-the-pandas"]], "Min-Max Scaling": [[28, "min-max-scaling"]], "Momentum based GD": [[13, "momentum-based-gd"], [30, "momentum-based-gd"]], "More complicated Example: The Ising model": [[6, "more-complicated-example-the-ising-model"]], "More interpretations": [[28, "more-interpretations"], [29, "more-interpretations"], [29, "id5"]], "More on Dimensionalities": [[3, "more-on-dimensionalities"]], "More on Rescaling data": [[6, "more-on-rescaling-data"]], "More on Steepest descent": [[29, "more-on-steepest-descent"]], "More on convex functions": [[29, "more-on-convex-functions"]], "More preprocessing": [[28, "more-preprocessing"], [30, "more-preprocessing"]], "Motivation for Adaptive Step Sizes": [[30, "motivation-for-adaptive-step-sizes"]], "Multilayer perceptrons": [[12, "multilayer-perceptrons"]], "Network requirements": [[2, "network-requirements"]], "Neural Networks vs CNNs": [[3, "neural-networks-vs-cnns"]], "Neural networks": [[12, null]], "Note about SVD Calculations": [[28, "note-about-svd-calculations"], [29, "note-about-svd-calculations"]], "Note on Scikit-Learn": [[29, "note-on-scikit-learn"]], "Numerical experiments and the covariance, central limit theorem": [[24, "numerical-experiments-and-the-covariance-central-limit-theorem"]], "Numpy and arrays": [[21, "numpy-and-arrays"], [27, "numpy-and-arrays"]], "Numpy examples and Important Matrix and vector handling packages": [[27, "numpy-examples-and-important-matrix-and-vector-handling-packages"]], "Optimization and gradient descent, the central part of any Machine Learning algortithm": [[29, "optimization-and-gradient-descent-the-central-part-of-any-machine-learning-algortithm"]], "Optimization, the central part of any Machine Learning algortithm": [[13, null]], "Optimizing our parameters": [[27, "optimizing-our-parameters"]], "Optimizing our parameters, more details": [[27, "optimizing-our-parameters-more-details"]], "Optimizing the cost function": [[1, "optimizing-the-cost-function"]], "Organizing our data": [[0, "organizing-our-data"], [27, "organizing-our-data"]], "Other Matrix and Vector Operations": [[21, "other-matrix-and-vector-operations"]], "Other Types of Recurrent Neural Networks": [[4, "other-types-of-recurrent-neural-networks"]], "Other courses on Data science and Machine Learning at UiO": [[27, "other-courses-on-data-science-and-machine-learning-at-uio"]], "Other courses on Data science and Machine Learning at UiO, contn": [[27, "other-courses-on-data-science-and-machine-learning-at-uio-contn"]], "Other popular texts": [[27, "other-popular-texts"]], "Other techniques": [[11, "other-techniques"]], "Other types of networks": [[12, "other-types-of-networks"]], "Other ways of visualizing the trees": [[9, "other-ways-of-visualizing-the-trees"]], "Our model for the nuclear binding energies": [[27, "our-model-for-the-nuclear-binding-energies"]], "Overview of first week": [[27, "overview-of-first-week"]], "Overview video on Stochastic Gradient Descent (SGD)": [[30, "overview-video-on-stochastic-gradient-descent-sgd"]], "Own code for Ordinary Least Squares": [[27, "own-code-for-ordinary-least-squares"], [28, "own-code-for-ordinary-least-squares"]], "PCA and scikit-learn": [[11, "pca-and-scikit-learn"]], "Pandas AI": [[27, "pandas-ai"]], "Part a : Ordinary Least Square (OLS) for the Runge function": [[22, "part-a-ordinary-least-square-ols-for-the-runge-function"]], "Part b: Adding Ridge regression for the Runge function": [[22, "part-b-adding-ridge-regression-for-the-runge-function"]], "Part c: Writing your own gradient descent code": [[22, "part-c-writing-your-own-gradient-descent-code"]], "Part d: Including momentum and more advanced ways to update the learning the rate": [[22, "part-d-including-momentum-and-more-advanced-ways-to-update-the-learning-the-rate"]], "Part e: Writing our own code for Lasso regression": [[22, "part-e-writing-our-own-code-for-lasso-regression"]], "Part f: Stochastic gradient descent": [[22, "part-f-stochastic-gradient-descent"]], "Part g: Bias-variance trade-off and resampling techniques": [[22, "part-g-bias-variance-trade-off-and-resampling-techniques"]], "Part h): Cross-validation as resampling techniques, adding more complexity": [[22, "part-h-cross-validation-as-resampling-techniques-adding-more-complexity"]], "Partial Differential Equations": [[2, "partial-differential-equations"]], "Plans for week 35": [[28, "plans-for-week-35"]], "Plans for week 36": [[29, "plans-for-week-36"]], "Plans for week 37, lecture Monday": [[30, "plans-for-week-37-lecture-monday"]], "Practical tips": [[13, "practical-tips"], [30, "practical-tips"]], "Practicalities": [[25, "practicalities"], [25, "id1"]], "Preamble: Note on writing reports, using reference material, AI and other tools": [[22, "preamble-note-on-writing-reports-using-reference-material-ai-and-other-tools"]], "Predicting New Points With A Trained Recurrent Neural Network": [[4, "predicting-new-points-with-a-trained-recurrent-neural-network"]], "Preprocessing our data": [[28, "preprocessing-our-data"]], "Prerequisites": [[27, "prerequisites"]], "Prerequisites and background": [[20, "prerequisites-and-background"]], "Prerequisites: Collect and pre-process data": [[3, "prerequisites-collect-and-pre-process-data"]], "Probability Distribution Functions": [[24, "probability-distribution-functions"]], "Program example for gradient descent with Ridge Regression": [[29, "program-example-for-gradient-descent-with-ridge-regression"], [30, "program-example-for-gradient-descent-with-ridge-regression"]], "Program for stochastic gradient": [[13, "program-for-stochastic-gradient"]], "Project 1 on Machine Learning, deadline October 6 (midnight), 2025": [[22, null]], "Properties of PDFs": [[24, "properties-of-pdfs"]], "Pros and cons": [[30, "pros-and-cons"]], "Pros and cons of trees, pros": [[9, "pros-and-cons-of-trees-pros"]], "Python installers": [[20, "python-installers"], [27, "python-installers"]], "RMS prop": [[13, "rms-prop"]], "RMSProp algorithm, taken from Goodfellow et al": [[30, "rmsprop-algorithm-taken-from-goodfellow-et-al"]], "RMSProp: Adaptive Learning Rates": [[30, "rmsprop-adaptive-learning-rates"]], "RMSprop for adaptive learning rate with Stochastic Gradient Descent": [[30, "rmsprop-for-adaptive-learning-rate-with-stochastic-gradient-descent"]], "Random Numbers": [[24, "random-numbers"]], "Random forests": [[10, "random-forests"]], "Randomized PCA": [[11, "randomized-pca"]], "Reading material": [[27, "reading-material"]], "Reading recommendations:": [[28, "reading-recommendations"]], "Reading suggestions week 34": [[27, "reading-suggestions-week-34"]], "Readings and Videos:": [[30, "readings-and-videos"]], "Recurrent neural networks": [[12, "recurrent-neural-networks"]], "Recurrent neural networks: Overarching view": [[4, null]], "Reducing the number of degrees of freedom, overarching view": [[0, "reducing-the-number-of-degrees-of-freedom-overarching-view"], [28, "reducing-the-number-of-degrees-of-freedom-overarching-view"]], "Reformulating the problem": [[2, "reformulating-the-problem"]], "Regression Case": [[10, "regression-case"]], "Regression analysis and resampling methods": [[22, "regression-analysis-and-resampling-methods"]], "Regression analysis, overarching aims": [[27, "regression-analysis-overarching-aims"]], "Regression analysis, overarching aims II": [[27, "regression-analysis-overarching-aims-ii"]], "Regularization": [[1, "regularization"]], "Reminder from last week": [[28, "reminder-from-last-week"]], "Reminder on Newton-Raphson\u2019s method": [[29, "reminder-on-newton-raphson-s-method"]], "Reminder on Statistics": [[6, "reminder-on-statistics"]], "Reminder on different scaling methods": [[30, "reminder-on-different-scaling-methods"]], "Replace or not": [[13, "replace-or-not"], [30, "replace-or-not"]], "Required Technologies": [[20, "required-technologies"]], "Resampling Methods": [[6, null]], "Resampling and the Bias-Variance Trade-off": [[19, "resampling-and-the-bias-variance-trade-off"]], "Resampling methods": [[6, "id1"]], "Residual Error": [[28, "residual-error"], [29, "residual-error"]], "Resources on differential equations and deep learning": [[2, "resources-on-differential-equations-and-deep-learning"]], "Revisiting Ordinary Least Squares": [[29, "revisiting-ordinary-least-squares"]], "Revisiting our Linear Regression Solvers": [[13, "revisiting-our-linear-regression-solvers"]], "Rewriting the Covariance and/or Correlation Matrix": [[28, "rewriting-the-covariance-and-or-correlation-matrix"]], "Rewriting the fitting procedure as a linear algebra problem": [[27, "rewriting-the-fitting-procedure-as-a-linear-algebra-problem"]], "Rewriting the fitting procedure as a linear algebra problem, more details": [[27, "rewriting-the-fitting-procedure-as-a-linear-algebra-problem-more-details"]], "Ridge Regression": [[29, "ridge-regression"]], "Ridge and LASSO Regression": [[28, "ridge-and-lasso-regression"], [29, "ridge-and-lasso-regression"], [29, "id2"]], "Ridge and Lasso Regression": [[5, null], [5, "id1"]], "SGD example": [[30, "sgd-example"]], "SVD analysis": [[29, "svd-analysis"]], "Same code but now with momentum gradient descent": [[13, "same-code-but-now-with-momentum-gradient-descent"], [30, "same-code-but-now-with-momentum-gradient-descent"], [30, "id3"], [30, "id4"]], "Schedule first week": [[27, "schedule-first-week"]], "Schematic Regression Procedure": [[9, "schematic-regression-procedure"]], "Second moment of the gradient": [[30, "second-moment-of-the-gradient"]], "September 15-19": [[19, "september-15-19"]], "Setting up the Back propagation algorithm": [[12, "setting-up-the-back-propagation-algorithm"]], "Setting up the Matrix to be inverted": [[28, "setting-up-the-matrix-to-be-inverted"], [29, "setting-up-the-matrix-to-be-inverted"]], "Setting up the network using Autograd; The full program": [[2, "setting-up-the-network-using-autograd-the-full-program"]], "Similar (second order function now) problem but now with AdaGrad": [[13, "similar-second-order-function-now-problem-but-now-with-adagrad"], [30, "similar-second-order-function-now-problem-but-now-with-adagrad"]], "Simple Python Code to read in Data and perform Classification": [[9, "simple-python-code-to-read-in-data-and-perform-classification"]], "Simple case": [[28, "simple-case"], [29, "simple-case"]], "Simple code for solving the above problem": [[29, "simple-code-for-solving-the-above-problem"]], "Simple example code": [[30, "simple-example-code"]], "Simple example to illustrate Ordinary Least Squares, Ridge and Lasso Regression": [[29, "simple-example-to-illustrate-ordinary-least-squares-ridge-and-lasso-regression"]], "Simple geometric interpretation": [[29, "simple-geometric-interpretation"]], "Simple linear regression model using scikit-learn": [[0, "simple-linear-regression-model-using-scikit-learn"], [27, "simple-linear-regression-model-using-scikit-learn"]], "Simple one-dimensional second-order polynomial": [[18, "simple-one-dimensional-second-order-polynomial"]], "Simple program": [[29, "simple-program"], [30, "simple-program"]], "Slightly different approach": [[30, "slightly-different-approach"]], "Sneaking in automatic differentiation using Autograd": [[30, "sneaking-in-automatic-differentiation-using-autograd"]], "Software and needed installations": [[22, "software-and-needed-installations"], [27, "software-and-needed-installations"]], "Solving Differential Equations with Deep Learning": [[2, null]], "Solving the one dimensional Poisson equation": [[2, "solving-the-one-dimensional-poisson-equation"]], "Solving the wave equation with Neural Networks": [[2, "solving-the-wave-equation-with-neural-networks"]], "Some famous Matrices": [[21, "some-famous-matrices"]], "Some simple problems": [[13, "some-simple-problems"], [29, "some-simple-problems"]], "Some useful matrix and vector expressions": [[28, "some-useful-matrix-and-vector-expressions"]], "Splitting our Data in Training and Test data": [[0, "splitting-our-data-in-training-and-test-data"], [28, "splitting-our-data-in-training-and-test-data"]], "Standard steepest descent": [[13, "standard-steepest-descent"]], "Statistical analysis and optimization of data": [[20, "statistical-analysis-and-optimization-of-data"], [27, "statistical-analysis-and-optimization-of-data"]], "Steepest descent": [[13, "steepest-descent"], [29, "steepest-descent"]], "Stochastic Gradient Descent": [[30, "stochastic-gradient-descent"]], "Stochastic Gradient Descent (SGD)": [[13, "stochastic-gradient-descent-sgd"], [30, "stochastic-gradient-descent-sgd"]], "Stochastic variables and the main concepts, the discrete case": [[24, "stochastic-variables-and-the-main-concepts-the-discrete-case"]], "Support Vector Machines, overarching aims": [[8, null]], "Systematic reduction": [[3, "systematic-reduction"]], "Teachers": [[27, "teachers"]], "Teachers and Grading": [[25, null]], "Teaching Assistants Fall semester 2023": [[25, "teaching-assistants-fall-semester-2023"]], "Tentative deadllines for projects": [[25, "tentative-deadllines-for-projects"]], "Testing the Means Squared Error as function of Complexity": [[0, "testing-the-means-squared-error-as-function-of-complexity"], [28, "testing-the-means-squared-error-as-function-of-complexity"]], "Textbooks": [[26, null]], "The Algorithm before theorem": [[11, "the-algorithm-before-theorem"]], "The Breast Cancer Data, now with Keras": [[1, "the-breast-cancer-data-now-with-keras"]], "The CART algorithm for Classification": [[9, "the-cart-algorithm-for-classification"]], "The CART algorithm for Regression": [[9, "the-cart-algorithm-for-regression"]], "The CIFAR01 data set": [[3, "the-cifar01-data-set"]], "The Hessian matrix": [[29, "the-hessian-matrix"], [30, "the-hessian-matrix"]], "The Hessian matrix for Ridge Regression": [[29, "the-hessian-matrix-for-ridge-regression"], [30, "the-hessian-matrix-for-ridge-regression"]], "The Jacobian": [[28, "the-jacobian"]], "The MNIST dataset again": [[3, "the-mnist-dataset-again"]], "The OLS case": [[29, "the-ols-case"]], "The RELU function family": [[1, "the-relu-function-family"]], "The Ridge case": [[29, "the-ridge-case"]], "The SVD, a Fantastic Algorithm": [[28, "the-svd-a-fantastic-algorithm"], [29, "the-svd-a-fantastic-algorithm"]], "The Softmax function": [[1, "the-softmax-function"]], "The \\chi^2 function": [[0, "the-chi-2-function"], [27, "the-chi-2-function"], [27, "id4"], [27, "id5"], [27, "id6"], [27, "id7"], [27, "id8"]], "The bias-variance tradeoff": [[6, "the-bias-variance-tradeoff"]], "The code for solving the ODE": [[2, "the-code-for-solving-the-ode"]], "The complete code with a simple data set": [[28, "the-complete-code-with-a-simple-data-set"]], "The cost/loss function": [[28, "the-cost-loss-function"]], "The course has two central parts": [[20, "the-course-has-two-central-parts"]], "The derivative of the cost/loss function": [[29, "the-derivative-of-the-cost-loss-function"], [30, "the-derivative-of-the-cost-loss-function"]], "The equations": [[29, "the-equations"]], "The equations for ordinary least squares": [[28, "the-equations-for-ordinary-least-squares"]], "The first Case": [[29, "the-first-case"]], "The gradient step": [[30, "the-gradient-step"]], "The ideal": [[29, "the-ideal"]], "The logistic function": [[7, "the-logistic-function"]], "The mean squared error and its derivative": [[28, "the-mean-squared-error-and-its-derivative"]], "The moons example": [[8, "the-moons-example"]], "The multilayer perceptron (MLP)": [[12, "the-multilayer-perceptron-mlp"]], "The network with one input layer, specified number of hidden layers, and one output layer": [[2, "the-network-with-one-input-layer-specified-number-of-hidden-layers-and-one-output-layer"]], "The plethora of machine learning algorithms/methods": [[27, "the-plethora-of-machine-learning-algorithms-methods"]], "The sensitiveness of the gradient descent": [[29, "the-sensitiveness-of-the-gradient-descent"]], "The singular value decomposition": [[5, "the-singular-value-decomposition"], [28, "the-singular-value-decomposition"], [29, "the-singular-value-decomposition"]], "The two-dimensional case": [[8, "the-two-dimensional-case"]], "Time decay rate": [[30, "time-decay-rate"]], "To our real data: nuclear binding energies. Brief reminder on masses and binding energies": [[27, "to-our-real-data-nuclear-binding-energies-brief-reminder-on-masses-and-binding-energies"]], "Topics covered in this course: Statistical analysis and optimization of data": [[27, "topics-covered-in-this-course-statistical-analysis-and-optimization-of-data"]], "Towards the PCA theorem": [[11, "towards-the-pca-theorem"]], "Train and test datasets": [[1, "train-and-test-datasets"]], "Two-dimensional Objects": [[3, "two-dimensional-objects"]], "Type of problem": [[2, "type-of-problem"]], "Types of Machine Learning": [[27, "types-of-machine-learning"]], "Useful Python libraries": [[20, "useful-python-libraries"], [27, "useful-python-libraries"]], "Using Autograd": [[13, "using-autograd"]], "Using forward Euler to solve the ODE": [[2, "using-forward-euler-to-solve-the-ode"]], "Using gradient descent methods, limitations": [[13, "using-gradient-descent-methods-limitations"], [29, "using-gradient-descent-methods-limitations"], [30, "using-gradient-descent-methods-limitations"]], "Visualization": [[1, "visualization"], [1, "id1"]], "Visualizing the Tree, Classification": [[9, "visualizing-the-tree-classification"]], "Week 34: Introduction to the course, Logistics and Practicalities": [[27, null]], "Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression": [[28, null]], "Week 36: Linear Regression and Gradient descent": [[29, null]], "Week 37: Statistical interpretations and Resampling Methods": [[30, null]], "What Is Generative Modeling?": [[27, "what-is-generative-modeling"]], "What does it mean?": [[28, "what-does-it-mean"], [29, "what-does-it-mean"]], "What is Machine Learning?": [[0, "what-is-machine-learning"]], "What is a good model?": [[0, "what-is-a-good-model"], [27, "what-is-a-good-model"]], "What is a good model? Can we define it?": [[27, "what-is-a-good-model-can-we-define-it"]], "When do we stop?": [[30, "when-do-we-stop"]], "Which activation function should I use?": [[1, "which-activation-function-should-i-use"]], "Why Combine Momentum and RMSProp?": [[30, "why-combine-momentum-and-rmsprop"]], "Why Linear Regression (aka Ordinary Least Squares and family)": [[27, "why-linear-regression-aka-ordinary-least-squares-and-family"]], "Wisconsin Cancer Data": [[7, "wisconsin-cancer-data"]], "With Lasso Regression": [[29, "with-lasso-regression"]], "Writing Our First Generative Adversarial Network": [[4, "writing-our-first-generative-adversarial-network"]], "Writing our own PCA code": [[11, "writing-our-own-pca-code"]], "Writing the Cost Function": [[29, "writing-the-cost-function"]], "XGBoost: Extreme Gradient Boosting": [[10, "xgboost-extreme-gradient-boosting"]], "Yet another Example": [[29, "yet-another-example"]], "a) Expression for Ridge regression": [[17, "a-expression-for-ridge-regression"]], "scikit-learn implementation": [[1, "scikit-learn-implementation"]]}, "docnames": ["chapter1", "chapter10", "chapter11", "chapter12", "chapter13", "chapter2", "chapter3", "chapter4", "chapter5", "chapter6", "chapter7", "chapter8", "chapter9", "chapteroptimization", "clustering", "exercisesweek34", "exercisesweek35", "exercisesweek36", "exercisesweek37", "exercisesweek38", "intro", "linalg", "project1", "schedule", "statistics", "teachers", "textbooks", "week34", "week35", "week36", "week37"], "envversion": {"sphinx": 62, "sphinx.domains.c": 3, "sphinx.domains.changeset": 1, "sphinx.domains.citation": 1, "sphinx.domains.cpp": 9, "sphinx.domains.index": 1, "sphinx.domains.javascript": 3, "sphinx.domains.math": 2, "sphinx.domains.python": 4, "sphinx.domains.rst": 2, "sphinx.domains.std": 2, "sphinx.ext.intersphinx": 1}, "filenames": ["chapter1.ipynb", "chapter10.ipynb", "chapter11.ipynb", "chapter12.ipynb", "chapter13.ipynb", "chapter2.ipynb", "chapter3.ipynb", "chapter4.ipynb", "chapter5.ipynb", "chapter6.ipynb", "chapter7.ipynb", "chapter8.ipynb", "chapter9.ipynb", "chapteroptimization.ipynb", "clustering.ipynb", "exercisesweek34.ipynb", "exercisesweek35.ipynb", "exercisesweek36.ipynb", "exercisesweek37.ipynb", "exercisesweek38.ipynb", "intro.md", "linalg.ipynb", "project1.ipynb", "schedule.md", "statistics.ipynb", "teachers.md", "textbooks.md", "week34.ipynb", "week35.ipynb", "week36.ipynb", "week37.ipynb"], "indexentries": {}, "objects": {}, "objnames": {}, "objtypes": {}, "terms": {"": [0, 1, 2, 3, 4, 5, 6, 7, 9, 11, 12, 13, 15, 16, 17, 19, 20, 21, 22, 24, 25, 27, 28], "0": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 21, 22, 24, 25, 27, 28, 29, 30], "00": [0, 1, 5, 11, 27, 28], "000": [1, 3], "00000000e": [], "001": [2, 8, 13, 29, 30], "004": 5, "004113634617443131": 28, "004113634617443139": 28, "00411363461744314": 28, "004113634617443147": 28, "00727646693": [0, 27], "0086649156": [0, 27], "01": [0, 1, 2, 5, 9, 11, 13, 17, 26, 27, 28, 30], "0110": 24, "01719003e": [], "02": [0, 4, 7, 12, 27], "02334824": [], "02857": 4, "02f": 6, "03077640549": 4, "03097597e": [], "031": 5, "04": 11, "0458": 9, "05": [4, 6], "062292565": 4, "062435": [], "06730814": [], "07": [], "0713": [0, 27], "07285": 3, "08": 24, "08078025e": [], "08336233266": 4, "08376632": 28, "083766322923899": 28, "0837663229239043": 28, "0917": 9, "0n": [0, 27], "0x113e21950": 17, "1": [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 21, 23, 24, 25, 26, 27, 29, 30], "10": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 16, 17, 18, 19, 21, 23, 24, 25, 27, 28, 29, 30], "100": [0, 1, 2, 3, 4, 5, 6, 8, 9, 10, 11, 13, 14, 15, 16, 17, 18, 19, 21, 24, 25, 27, 28, 29, 30], "1000": [0, 1, 2, 4, 5, 8, 11, 13, 14, 18, 19, 20, 24, 27, 29, 30], "10000": [2, 5, 6, 10, 11, 13, 24], "100000": 8, "10001": 10, "1001": 24, "1002": 24, "1003": 24, "1005": 24, "1009": 24, "101": 16, "1011": 24, "1013": 24, "1013904243": 24, "1015": 24, "102": 16, "1023": 24, "1024": 3, "1026": 24, "1027": 24, "103": 1, "1030": 24, "1037": 24, "1038": 24, "1040": 24, "1047": 24, "107": 16, "108": [], "10th": 9, "10x": [0, 27], "11": [0, 2, 5, 6, 7, 8, 9, 10, 11, 12, 13, 16, 21, 22, 24, 26, 27, 28, 29, 30], "110": [], "1100": 24, "1101": 24, "111": [1, 7, 12], "112": 16, "11340253": [], "11590451": [], "116": 16, "117": 16, "118": 16, "12": [0, 1, 2, 3, 4, 5, 6, 8, 9, 11, 12, 18, 21, 24, 26, 27, 28, 29, 30], "120": 3, "121": [8, 9, 10, 16], "1215pm": [25, 27], "122": [8, 9, 10], "124": [0, 27], "125": 16, "127": [4, 16], "128": [3, 4, 13, 30], "129": 16, "1298": 9, "12pm": [25, 27], "13": [0, 2, 9, 12, 21, 24, 27], "131": 16, "133": 7, "135": 16, "136": 16, "14": [0, 2, 4, 6, 8, 9, 10, 12, 21, 24, 26, 28], "141": 16, "1412": 30, "143": 16, "1446729567": 4, "149": 16, "14g": 6, "15": [0, 2, 4, 6, 7, 8, 9, 12, 13, 22, 24, 27, 29, 30], "150": [4, 8], "152": 16, "153760": [], "156": 16, "157": [], "158": [], "159": 16, "15g": 6, "15pm": 27, "16": [1, 2, 3, 4, 5, 8, 9, 10, 24, 27, 29], "160": 16, "1603": 3, "161": 16, "162": 16, "16231451": 4, "163": 16, "16384": 3, "164": 16, "167": 16, "17": [1, 2, 8, 24], "172": 16, "173": 16, "176": 16, "178": 16, "179": 16, "1797": 1, "18": [2, 6, 7, 8, 9, 10, 24, 27], "1807": 4, "18392847": [], "19": [2, 24, 27], "1940": [], "1943": 12, "19569961": 28, "1970": [21, 27], "1973": 9, "1979": 6, "1_1": 12, "1_2": 12, "1_3": 12, "1cm": [0, 8, 10, 24, 27], "1d": [1, 2, 3], "1e": [2, 4, 13, 14, 30], "1e10": 14, "1e4": 6, "1f": 1, "1k": 21, "1n": [0, 27], "1x": [0, 27], "2": [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 20, 21, 22, 24, 26, 30], "20": [0, 1, 2, 6, 7, 8, 16, 17, 24, 25, 27, 28, 29, 30], "200": [0, 2, 3, 4, 8, 9, 10], "2000": [0, 28], "2004": [13, 29], "2006": 26, "20072279": [], "2008": 27, "2010": 1, "2011": 1, "2012": 30, "2014": [4, 30], "2015": 1, "2016": [0, 27], "2018": [0, 6, 28], "2021": [6, 14, 28, 30], "2022": 27, "2025": [18, 27, 28, 29, 30], "21": [0, 1, 5, 7, 9, 12, 21, 27, 28, 29], "2116753732": 4, "215pm": [25, 27], "2167072": [], "22": [0, 1, 5, 12, 13, 21, 27, 28, 29], "221": 8, "225": 4, "22948497": [], "23": [1, 12, 21], "24": [0, 1, 21, 27], "25": [2, 3, 4, 5, 6, 8, 9, 11, 28], "250": [2, 4, 7, 9], "25000": [], "250154": [], "253775": [], "255": 3, "256": 4, "25x": 22, "26": [], "26303845": [], "264": [], "265": [], "265109911": 4, "266": [], "269": [], "27": 1, "270": [], "278": [29, 30], "27n_": 24, "28": [1, 3, 4], "283": [29, 30], "2830637392": 4, "2861": 24, "2873": 9, "2882": 24, "2886": 24, "2890": [0, 27], "2892": 24, "29": 28, "2915": 24, "2931": 27, "29364655": [], "294399745619595": [], "296247": [], "2968": 27, "2980": 27, "298273": [], "298375": [], "2990": 27, "2_": 12, "2_1": 12, "2_2": 12, "2_3": 12, "2_i": 12, "2_m": [6, 24], "2_t": 13, "2_x": 24, "2a": 17, "2b": 24, "2c8f433990d1": 30, "2cm": 8, "2d": [1, 3, 11, 12, 20, 27], "2e": 6, "2f": [0, 7, 9, 10, 11, 12, 27], "2g": 2, "2g_i": 2, "2k": 3, "2m": 6, "2mvizaqfst8": 28, "2n": [0, 2, 3, 27, 28], "2nd": 9, "2p": 24, "2pt": 4, "2x": [0, 3, 8, 13, 27], "2x_ix_jy_iy_j": 8, "2x_j": 8, "2y_i": 10, "2y_j": 8, "3": [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 20, 21, 22, 23, 24, 25, 27, 29, 30], "30": [0, 1, 4, 6, 7, 10, 13, 25, 30], "30000": [0, 27], "3072": 3, "31": [12, 21, 24], "315": [6, 28, 30], "3155": [0, 5, 6, 28, 29, 30], "32": [3, 4, 6, 12, 13, 21, 24, 30], "3200": 1, "3250": 1, "3297": [], "33": [12, 21, 25], "3303": [], "3310": [], "332331": [], "333": 7, "3331": [], "3337": [], "34": 21, "3436": [0, 27], "3437": [0, 27], "35": [0, 6, 22, 27, 29, 30], "3581341341": 4, "359": [5, 29], "36": [0, 5, 6, 18, 22, 24], "37": [22, 29], "370782966": 4, "38": [22, 24], "39": [0, 25, 27], "3d": [2, 3, 4, 6, 13, 16], "3f": [1, 3, 9], "3n": 21, "3x": [2, 8], "3x_i": 2, "3y": 8, "4": [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 21, 22, 24, 27, 29, 30], "40": [1, 6, 25, 27], "400": 4, "4000": 27, "4050": [26, 27], "41": 21, "4155": [2, 15], "41589548": [], "42": [1, 4, 8, 9, 10, 21], "43": [0, 7, 21], "4310": 27, "436462435": 4, "44": [0, 21, 29, 30], "45": [25, 27], "46": [25, 27], "462": 7, "47": [25, 27], "479465113": 4, "47958494": [], "48": [], "48257387": [25, 27], "49": [5, 6, 11], "49152": 3, "4940954": [0, 27], "4990": 24, "4992": 24, "4997": 24, "4c4c7f": [9, 10], "4d": 3, "4f": 6, "4pm": [25, 27], "4y": 8, "4y_i": 10, "5": [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 17, 19, 21, 22, 24, 27, 28, 29, 30], "50": [1, 2, 3, 4, 6, 7, 8, 10, 13, 27, 28, 30], "500": [1, 3, 4, 6, 9, 10, 13, 30], "5018": 24, "506": [], "507d50": [9, 10], "50j": 13, "50x10": 1, "51": 10, "510": 1, "512132": [], "5177783846": 4, "53": 9, "54": [6, 24], "5411205": [], "54894451": [], "55": 1, "56": 1, "56536": [0, 27], "569": 1, "57": [0, 8, 25, 27], "571": [5, 29], "58": [10, 25, 27], "591317992": 4, "5cm": 24, "5f": [8, 30], "5x": [8, 18], "5y": 8, "6": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 12, 13, 18, 21, 24, 25, 27, 28, 29, 30], "60": [1, 3], "60000": 4, "6019067271": 4, "606439": [], "625": 7, "63": 1, "64": [1, 3, 4, 13, 21, 27, 30], "64x50": 1, "65": [1, 8, 9], "6887363571": 4, "69": [16, 24], "69069n_": 24, "691": [], "6980": 30, "6n_": 24, "7": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 11, 12, 13, 21, 22, 24, 26, 27, 28, 30], "70": [1, 7], "70653767": 4, "71": 1, "724": 3, "73": [], "7304881": [], "75": [5, 6, 8, 11], "76": [25, 27], "765": 7, "77": [25, 27], "7718": 9, "7782028952": 4, "77893972": [], "78": [], "7d7d58": [9, 10], "8": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 12, 13, 14, 18, 19, 21, 24, 25, 27, 29], "80": [0, 1, 5, 8, 17, 28], "800": [4, 7], "81": 1, "815am": [25, 27], "85": 1, "8702784034": 4, "88": 27, "8f": 6, "8g": 6, "8n": 21, "8x8": 1, "9": [0, 1, 2, 4, 5, 6, 7, 8, 9, 11, 12, 13, 21, 24, 27, 30], "90": 1, "9040": 9, "91": [25, 27], "92": [25, 27], "93": 16, "931": [0, 27], "933": [5, 29], "937": 24, "938": 24, "939": [0, 24, 27], "94": 24, "95": [1, 11], "954": 24, "955820c21e8b": 4, "96": 6, "960": 24, "961": 24, "962": 24, "9649652536": 4, "96611194e": [], "9780387310732": 26, "9780387848570": 26, "9781098134174": 27, "9781492032632": 26, "9781801819312": 27, "97898392": 28, "98": [0, 1, 16], "985": 24, "986": 24, "989": 24, "9898ff": [9, 10], "99": [13, 16, 30], "991": 24, "992": 24, "993": 24, "996": 5, "999": [9, 24, 30], "9x": 6, "9y": 6, "A": [2, 3, 5, 6, 7, 10, 11, 12, 13, 15, 16, 19, 20, 21, 23, 24, 25, 26, 28, 29, 30], "AND": 2, "And": [0, 3, 4, 5, 6, 9, 13, 20, 22, 24, 29], "As": [0, 1, 2, 3, 4, 5, 6, 8, 10, 12, 13, 15, 16, 21, 22, 24, 27, 28, 29, 30], "At": [0, 4, 6, 13, 27, 30], "BE": [0, 27], "Be": [2, 18, 20, 27], "Being": 13, "But": [0, 1, 2, 3, 5, 6, 9, 10, 16, 24, 28], "By": [0, 3, 5, 6, 12, 13, 17, 21, 27, 28, 29, 30], "For": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 18, 19, 20, 21, 22, 24, 26, 27, 28, 29, 30], "IF": [6, 28, 30], "IN": 26, "If": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 15, 16, 18, 20, 21, 22, 24, 27, 28, 29, 30], "In": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 18, 20, 21, 22, 24, 26, 27, 28, 29, 30], "Ising": [5, 12, 28, 29], "It": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 18, 20, 21, 22, 24, 27, 28, 29, 30], "Its": [1, 2, 4, 11], "No": [6, 9, 27, 28, 30], "Not": [0, 1, 5, 6, 28, 29, 30], "OR": 24, "Of": 24, "On": [0, 3, 22, 24, 25, 26, 27], "One": [0, 1, 3, 4, 5, 6, 7, 8, 11, 12, 13, 17, 24, 28, 29, 30], "Or": [0, 1, 6, 27], "Such": [0, 6, 12, 16, 24, 30], "That": [0, 5, 7, 10, 11, 12, 14, 22, 24, 27], "The": [4, 10, 13, 14, 16, 17, 18, 19, 21, 22, 23, 24, 25, 26], "Then": [0, 1, 6, 8, 9, 10, 11, 12, 13, 14, 15, 16, 18, 21, 27, 29, 30], "There": [0, 3, 4, 5, 6, 8, 9, 11, 12, 14, 15, 21, 22, 24, 25, 27, 28, 29, 30], "These": [0, 3, 4, 5, 8, 9, 10, 11, 12, 13, 14, 17, 18, 21, 22, 24, 25, 27, 28, 29, 30], "To": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 15, 16, 21, 24, 28, 29, 30], "With": [0, 5, 6, 8, 9, 10, 11, 12, 14, 16, 19, 21, 22, 24, 27, 28], "_": [0, 1, 2, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 16, 17, 18, 19, 21, 22, 27, 28, 29, 30], "_0": [5, 8, 10, 11, 13, 28, 29], "_1": [2, 5, 6, 8, 10, 11, 12, 13, 14, 21, 28, 29, 30], "_2": [2, 5, 8, 11, 12, 13, 21, 28, 30], "_3": 21, "_4": 21, "_9": [13, 30], "__class__": 10, "__doc__": 6, "__future__": [8, 9], "__init__": 1, "__main__": 2, "__name__": [2, 10], "_auto1": [2, 3, 4, 5, 6, 7, 12, 13, 21, 24, 28, 29], "_auto10": [6, 12], "_auto11": 6, "_auto12": 6, "_auto2": [2, 3, 4, 5, 6, 12, 13, 21, 24], "_auto3": [3, 4, 5, 6, 12, 13, 21], "_auto4": [4, 6, 12, 13, 21], "_auto5": [4, 6, 12, 13, 21], "_auto6": [4, 6, 12, 21], "_auto7": [4, 6, 12, 21], "_auto8": [6, 12], "_auto9": [6, 12], "_build": [0, 20, 22, 26, 27], "_c": 1, "_center": [], "_compon": 11, "_depth": 9, "_export": [15, 16], "_fraction": 9, "_i": [0, 1, 2, 5, 6, 7, 8, 11, 12, 13, 19, 22, 27, 28, 29, 30], "_j": [0, 1, 2, 3, 5, 6, 8, 13, 19, 22, 28, 29, 30], "_k": [13, 29, 30], "_l": 12, "_lambda": 6, "_leaf": 9, "_m": 10, "_multilayer_perceptron": [], "_n": [2, 5, 8, 11, 13, 28, 29, 30], "_node": 9, "_norm": [], "_p": [5, 8, 28, 29], "_ratio": 11, "_sampl": 9, "_split": [6, 9, 22], "_t": [13, 30], "_test": [6, 22], "_varianc": 11, "_weight": 9, "a0": 3, "a0faa0": [9, 10], "a1": [0, 27], "a2": [0, 27], "a3": [0, 27], "a4": [0, 27], "a_": [0, 1, 16, 21, 27, 28], "a_0": [0, 27], "a_1a": [0, 27], "a_2a": [0, 27], "a_3": [0, 27], "a_3a": [0, 27], "a_4": [0, 27], "a_4a": [0, 27], "a_h": 1, "a_i": [0, 1, 2, 12, 27], "a_j": [1, 12], "a_k": [0, 1, 12], "aaron": 26, "ab": [0, 2, 5, 13, 14, 27, 28, 30], "ab_channel": 20, "abandon": 1, "abid": 24, "abil": [0, 10], "abl": [0, 1, 4, 5, 6, 7, 10, 12, 13, 16, 18, 19, 22, 28, 29, 30], "about": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 15, 16, 20, 21, 22, 25, 30], "abov": [0, 1, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 21, 24, 26, 27, 28, 30], "abovement": [6, 22, 27], "abscissa": [13, 29], "absent": 30, "absolut": [0, 2, 5, 6, 13, 27, 28, 29], "absorb": [28, 29], "abstract": 1, "acceler": [13, 30], "accept": [0, 3, 6, 9, 22, 28], "access": [3, 11, 24, 27], "accid": [4, 6], "accompani": [0, 27, 28], "accomplish": [8, 9, 13, 30], "accord": [0, 1, 2, 5, 6, 9, 12, 13, 14, 24, 27, 29, 30], "accordingli": 11, "account": [0, 3, 5, 13, 15, 16, 24, 27], "accumul": [12, 13, 24, 30], "accur": [0, 3, 4, 6, 10, 13, 30], "accuraci": [0, 1, 3, 4, 5, 6, 7, 9, 10, 11, 12, 27, 28, 29], "accuracy_scor": [0, 1, 10, 27], "accuracy_score_numpi": 1, "achiev": [0, 1, 5, 6, 8, 12, 21, 27, 30], "aco": 24, "acquaint": 20, "acquir": [1, 20, 27], "acr": [], "across": [1, 3, 6, 9, 17, 20, 27], "act": [1, 3, 21], "action": 24, "activ": [0, 2, 3, 4, 9, 15, 23, 25, 27, 30], "actual": [0, 1, 4, 5, 6, 8, 11, 15, 16, 18, 21, 24, 27, 28, 29, 30], "ad": [1, 3, 4, 5, 8, 13, 15, 16, 21, 29, 30], "ada_clf": 10, "adaboostclassifi": 10, "adadelta": [13, 30], "adagrad": 22, "adam": [1, 3, 4, 22, 27], "adapt": [4, 6, 13, 17, 26, 29], "add": [0, 1, 2, 3, 4, 5, 6, 8, 10, 11, 12, 15, 16, 17, 18, 24, 25, 27, 28, 29, 30], "add_subplot": [1, 7, 12, 14], "addendum": 5, "addit": [0, 2, 3, 5, 6, 7, 8, 9, 10, 12, 13, 15, 20, 21, 22, 24, 25, 26, 27, 28], "addition": [12, 13, 29, 30], "address": [1, 9, 11, 13, 27, 30], "adjac": [3, 12], "adjoint": [5, 28], "adjust": [0, 5, 12, 13, 29, 30], "admir": [0, 27], "advanc": [4, 6, 12, 26, 27, 30], "advantag": [1, 3, 5, 6, 10, 13, 21, 29, 30], "adversari": 27, "afecionado": 27, "affect": [3, 15], "affin": [0, 3, 8, 11, 28], "afford": 3, "aficionado": 27, "aforement": 14, "african": [], "after": [0, 1, 2, 4, 5, 6, 9, 11, 12, 13, 15, 16, 17, 18, 19, 20, 21, 22, 24, 27, 28, 29, 30], "afterward": [0, 27], "ag": [0, 7, 27, 28], "ag_0": 2, "again": [0, 1, 4, 5, 6, 7, 8, 10, 11, 12, 13, 22, 24, 27, 28, 29], "against": [1, 4, 7, 10], "agegroup": 7, "agegroupmean": 7, "aggreg": [9, 10], "agorithm": 10, "agre": [5, 6, 24, 28, 29, 30], "agreement": [13, 30], "ahead": 9, "ai": [0, 26], "aid": 11, "aim": [0, 1, 4, 6, 7, 11, 14, 16, 17, 19, 20, 21, 22, 28], "ainv": 5, "airplan": 3, "aka": 5, "al": [0, 2, 4, 16, 17, 26, 27, 28, 29], "alarm": [5, 7], "aldo": 28, "algebra": [0, 3, 5, 13, 20, 28, 29], "algorithm": [0, 1, 2, 4, 5, 6, 7, 8, 13, 14, 16, 20, 21, 22, 24, 26], "align": [0, 2, 5, 6, 7, 8, 13, 24, 27, 28, 29], "all": [0, 1, 2, 3, 4, 5, 6, 7, 9, 10, 11, 12, 13, 14, 15, 18, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30], "allevi": [1, 13, 29], "alloc": [3, 21], "allow": [0, 1, 2, 3, 5, 6, 8, 10, 13, 15, 20, 21, 22, 27, 28, 29, 30], "almost": [0, 1, 6, 8, 11, 13, 24, 29, 30], "alon": [2, 9, 30], "along": [2, 3, 4, 5, 6, 9, 10, 11, 15, 20, 21, 27, 28, 29], "alpha": [0, 1, 2, 3, 4, 6, 7, 8, 9, 10, 13, 14, 24, 27, 28, 29, 30], "alpha_": [10, 30], "alpha_0": 3, "alpha_1": 3, "alpha_2": 3, "alpha_i": [3, 13], "alpha_k": 13, "alpha_m": 10, "alpha_n": 3, "alpha_opt": 13, "alreadi": [2, 3, 4, 5, 6, 10, 12, 15, 20, 21, 24, 27, 28, 29], "also": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 18, 19, 20, 21, 22, 24, 27, 28, 29, 30], "alter": 1, "altern": [0, 1, 4, 5, 6, 8, 9, 11, 13, 15, 18, 21, 22, 27, 28, 30], "although": [0, 1, 5, 6, 8, 10, 13, 16, 27, 30], "alwai": [0, 3, 5, 6, 12, 13, 16, 22, 24, 27, 28, 29, 30], "am": 4, "ame2016": [0, 27], "american": [], "among": [0, 3, 5, 9, 10, 12, 21, 27, 28], "amongst": 5, "amount": [0, 1, 3, 4, 6, 8, 10, 14, 20], "an": [1, 2, 3, 5, 6, 7, 8, 9, 11, 12, 13, 14, 16, 17, 18, 19, 20, 21, 22, 24, 25, 26, 28, 29, 30], "an_": 24, "anaconda": [0, 1, 20, 22, 27], "analogi": 13, "analys": 6, "analysi": [1, 3, 4, 7, 14, 19, 21, 26], "analyt": [2, 3, 5, 6, 7, 12, 13, 17, 20, 22, 27, 28, 29, 30], "analyz": [0, 1, 3, 4, 5, 6, 16, 22, 24, 28, 29], "andrew": 1, "angl": [0, 3, 9, 28, 30], "anharmon": 3, "ani": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 12, 14, 15, 16, 19, 24, 27, 28, 30], "anim": [4, 12], "ann": 12, "annot": [0, 1, 3, 7, 8, 27], "announc": 27, "anonym": 18, "anoth": [0, 1, 3, 4, 5, 6, 7, 8, 10, 11, 12, 13, 15, 21, 22, 24, 27, 28, 30], "ansatz": [0, 18, 27], "answer": [0, 1, 3, 5, 6, 21, 22, 25, 27], "antialias": [2, 6], "anticip": 4, "anymor": [1, 8], "anyon": [4, 8, 15], "anyth": [1, 15, 16, 24], "anytim": [25, 27], "apach": 1, "apart": [11, 13, 29], "api": [1, 20, 27], "appar": 2, "appear": [0, 1, 3, 13, 21, 24], "append": [1, 3, 4, 8, 9, 13, 19, 27, 30], "appendix": 22, "appli": [0, 1, 3, 4, 6, 7, 8, 9, 10, 11, 12, 13, 18, 22, 24, 26, 27, 28, 30], "applic": [0, 1, 3, 4, 5, 6, 7, 9, 12, 13, 16, 21, 24, 26, 27, 28, 29, 30], "apply_gradi": 4, "approach": [1, 2, 4, 5, 6, 9, 10, 11, 12, 13, 15, 16, 18, 20, 22, 24, 26, 28, 29], "approch": 22, "appropri": [2, 6, 9, 12, 13, 17, 20, 24], "approv": 27, "approx": [0, 2, 3, 6, 10, 11, 13, 18, 22, 24, 27, 29, 30], "approxim": [0, 1, 2, 3, 4, 5, 6, 7, 10, 11, 13, 19, 22, 24, 27, 28, 29, 30], "apt": [0, 20, 22, 27], "aq": 24, "ar": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30], "aragorn": 27, "arang": [1, 3, 4, 6, 7, 9, 10, 12, 13, 27, 30], "arbitrari": [1, 4, 6, 8, 12, 13, 24, 29], "arbitrarili": [0, 1, 11, 27], "arc": 6, "architectur": [3, 4, 12], "archiv": 22, "area": [0, 3, 6, 26, 27], "argmax": [1, 11], "argmin": [4, 10, 14], "argsort": 11, "argu": [1, 13], "argument": [0, 2, 3, 5, 11, 12, 13, 17, 27, 28, 30], "aris": [0, 6, 12, 13, 24, 27, 29], "arithmet": [0, 13, 21, 27], "arm": [6, 28, 30], "armadillo": 21, "around": [0, 1, 4, 5, 6, 11, 18, 22, 24, 27], "arrai": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 12, 13, 14, 16, 18, 20, 22, 24, 28, 29, 30], "arrang": [3, 27], "arraybox": 13, "arriv": [0, 6, 9, 11, 21, 24, 27], "arrow": 12, "arrowprop": 8, "art": [0, 1, 20], "articl": [0, 3, 4, 6, 10, 19, 27, 28, 29, 30], "artifici": [0, 2, 7, 12, 26, 27], "artificialneuron": 12, "arug": 13, "arxiv": [3, 4, 30], "asarrai": [0, 6, 9, 28, 30], "asid": 28, "ask": [5, 6, 11, 12, 15, 22], "aspect": [0, 6, 20, 27, 28], "assembl": 3, "assembli": [0, 27], "assert": 4, "assess": [0, 6, 22, 27, 28], "assici": 4, "assign": [0, 7, 8, 9, 12, 13, 14, 15, 23, 25, 26, 27], "associ": [0, 6, 9, 12, 14, 24, 27], "assum": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 17, 19, 21, 22, 24, 27, 28, 29, 30], "assumpt": [0, 3, 5, 6, 9, 11, 24, 27, 28], "ast": [0, 5, 6, 27], "astyp": [4, 9, 10], "asymmetri": [0, 27], "asymptot": [4, 6], "atom": [0, 27], "attempt": [0, 4, 6, 7, 8, 10, 27, 28, 30], "attend": 27, "attent": [0, 21, 27], "attract": [0, 10, 27], "attribut": [0, 9, 27], "audi": [0, 27], "audio": [3, 4], "august": [27, 28], "aurelien": [0, 26, 27], "austfjel": 6, "auth": 15, "authent": 15, "author": [0, 1, 10, 24], "authour": 27, "auto": [9, 10, 24], "auto_exampl": [22, 28], "autocor": 24, "autocorrelation_tim": 24, "autocorrelform": 24, "autocovari": 24, "autoencod": [4, 20, 27], "autoencond": 20, "autograd": [20, 27], "autom": [0, 20, 26, 27], "automac": 21, "automag": 27, "automat": [0, 1, 2, 3, 4, 11, 16, 20, 21, 27], "automobil": 3, "autonom": 4, "avail": [0, 1, 4, 6, 10, 11, 20, 21, 22, 23, 25, 26, 27], "averag": [0, 1, 3, 6, 9, 10, 13, 14, 24, 25, 27, 28], "avoid": [0, 4, 5, 6, 9, 11, 13, 18, 21, 28, 30], "awai": [2, 3, 6, 28, 30], "awar": [2, 10], "award": [25, 27], "ax": [0, 1, 2, 3, 4, 6, 7, 8, 9, 10, 11, 12, 13, 14, 21, 22, 27], "axes3d": [2, 6, 13, 29, 30], "axes_grid1": 6, "axhlin": 8, "axi": [0, 1, 2, 3, 4, 6, 7, 8, 9, 10, 11, 12, 13, 14, 18, 21, 24, 27, 28, 29, 30], "axiom": 5, "axvlin": [4, 8], "axvspan": 4, "b": [0, 1, 3, 4, 5, 6, 8, 9, 10, 12, 13, 14, 15, 16, 17, 19, 24, 25, 27, 28, 29, 30], "b1": 8, "b2": 8, "b3": 8, "b_": [0, 1, 21], "b_0": 0, "b_1": [0, 2, 12, 13, 30], "b_2": [0, 13], "b_5": [13, 30], "b_group": 9, "b_i": [0, 1, 2, 12, 27], "b_ia_": [0, 27], "b_ia_i": 0, "b_index": 9, "b_j": [1, 12], "b_k": [0, 1, 12, 13, 30], "b_m": 12, "b_score": 9, "b_valu": 9, "ba": 30, "babcock": 27, "bachelor": [23, 25], "back": [0, 3, 4, 5, 6, 8, 9, 10, 15, 16, 21, 24, 27, 30], "backbon": 21, "backend": [1, 4], "background": [26, 27], "backpropag": 1, "backtrack": 9, "backup": 21, "backward": [1, 2, 4, 12, 21], "bad": [6, 17, 28], "badli": 24, "bag": [9, 20, 27], "bag_clf": 10, "baggin": 27, "baggingboot": 10, "baggingclassifi": 10, "baggingtre": 10, "balanc": [6, 30], "ballpark": 18, "band": 21, "bandwidth": 21, "bar": [0, 6, 11, 22, 27], "barber": 26, "bare": [4, 10], "base": [0, 1, 3, 4, 5, 7, 8, 9, 10, 14, 15, 16, 17, 20, 24, 25, 26, 27, 28, 29], "basi": [5, 7, 8, 10, 11, 12, 13, 21, 28, 29], "basic": [6, 8, 12, 13, 14, 15, 20, 22, 24, 27], "batch": [3, 4, 11, 12, 13, 29], "batch_shap": 4, "batch_siz": [1, 3, 4], "batchnorm": 4, "bay": 7, "bayesian": [5, 20, 26, 27], "becaus": [0, 1, 2, 3, 4, 5, 6, 8, 9, 12, 13, 14, 27, 28, 29, 30], "becom": [0, 1, 2, 5, 6, 7, 9, 12, 13, 24, 27, 28, 29, 30], "been": [0, 1, 2, 3, 4, 5, 6, 11, 12, 13, 20, 21, 22, 27, 28, 30], "befor": [0, 1, 2, 3, 4, 5, 6, 7, 8, 12, 13, 14, 16, 17, 18, 19, 21, 22, 24, 27, 28, 30], "beforehand": [0, 24, 27], "begin": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 11, 12, 13, 14, 15, 21, 24, 25, 27, 28, 29, 30], "behav": [1, 6, 13, 29], "behavior": [0, 1, 13, 27, 29, 30], "behaviour": [12, 30], "behind": [0, 1, 6, 8, 13, 27, 29], "being": [0, 1, 2, 3, 4, 5, 7, 8, 10, 11, 12, 13, 17, 24, 27, 28, 29, 30], "believ": [9, 21], "belong": [7, 8, 9, 13, 14, 29], "below": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 15, 18, 21, 22, 24, 27, 28, 29, 30], "benchmark": 10, "benefici": [1, 13], "benefit": [0, 1, 4, 11, 13, 20, 27, 29, 30], "bengio": [1, 26, 27, 28, 30], "benign": [1, 7], "besid": [4, 5, 29], "bessel": [5, 28], "best": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 12, 13, 15, 16, 18, 25, 27, 28, 29, 30], "beta": [1, 3, 10, 11, 13, 16, 17, 19, 27, 28, 29], "beta1": [], "beta2": [], "beta_": [3, 13, 17, 28], "beta_0": [1, 3, 13, 28], "beta_1": [1, 3, 10, 13, 28, 30], "beta_1m_": 30, "beta_1x_i": 13, "beta_2": [3, 13, 30], "beta_2v_": 30, "beta_3": 3, "beta_i": [3, 30], "beta_j": [13, 28], "beta_k": 13, "beta_linreg": 13, "beta_m": 10, "beta_mg_m": 10, "beta_n": 3, "better": [0, 1, 2, 3, 4, 6, 9, 10, 11, 12, 13, 27, 28, 30], "between": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 11, 12, 13, 14, 15, 16, 17, 18, 19, 22, 24, 27, 28, 29, 30], "beyond": [0, 1, 5, 6, 8, 13, 27, 28, 29, 30], "bf": [13, 14, 19, 21, 24, 29], "bg": 27, "bgd": [13, 30], "bia": [0, 1, 2, 3, 5, 8, 9, 10, 12, 13, 27, 28, 29], "bias": [1, 2, 3, 5, 6, 9, 12, 19, 30], "bibliographi": 22, "big": [0, 1, 2, 5, 6, 14, 19], "bigger": [1, 6, 28], "bigr": 12, "bike": 9, "bilbo": 27, "billion": [3, 12, 20], "bin": [7, 24], "binari": [0, 3, 5, 7, 9, 10, 12, 27], "binarycrossentropi": 4, "bind": 0, "binomi": [20, 24, 27], "binsboot": 6, "bioinformat": 0, "biolog": [1, 12], "bios1100": [20, 27], "bird": [0, 3], "birth": 27, "bishop": [26, 27], "bit": [1, 4, 21, 24, 27], "bitwis": 24, "bivari": 2, "bk": [13, 30], "bla": [21, 27], "black": [8, 9, 14], "block": [6, 10, 20, 21, 24, 27], "blog": 27, "blogpost": 4, "blue": [0, 3], "bm": [], "bmatrix": [0, 1, 3, 5, 7, 8, 11, 13, 21, 27, 28, 29, 30], "bmi": 1, "bodi": [0, 1, 4, 12], "bold": 1, "boldfac": [0, 5, 16, 28, 29], "boldsymbol": [0, 1, 2, 3, 5, 6, 7, 8, 10, 11, 13, 14, 16, 17, 19, 22, 27, 29, 30], "boltzmann": [12, 20, 27], "book": [17, 22, 26, 27, 28], "book1": 26, "boolean": [4, 17], "boost": [1, 9, 20, 27], "boostrap": 10, "bootstrap": [1, 13, 19, 20, 22, 27, 30], "borrow": 27, "boston_dataset": [], "bot": 8, "both": [0, 1, 4, 5, 6, 8, 9, 10, 13, 14, 15, 16, 17, 20, 21, 22, 24, 25, 27, 28, 29, 30], "bottl": 7, "bound": [8, 12, 30], "boundari": [2, 4, 8, 11, 12], "box": [4, 9], "boyd": [8, 13, 29], "bracket": [4, 24], "brain": [1, 7, 12], "branch": [9, 27], "break": [0, 4, 6, 11, 14, 27, 30], "breast": [5, 7, 11], "breviti": 13, "brew": [0, 20, 22, 27], "brg": 8, "brief": [22, 28], "briefli": [0, 16, 27], "bring": [0, 5, 6, 10, 28, 30], "britt": [25, 27], "broad": 0, "broadli": 27, "brought": [13, 20, 27], "brownle": 4, "browser": [15, 27], "brute": [3, 5, 11, 28], "buffer_s": 4, "bui": 4, "build": [0, 4, 5, 6, 10, 16, 21, 24, 27], "built": [1, 3, 4, 6], "bunch": 11, "busi": [], "byte": [21, 27], "c": [0, 1, 2, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 19, 20, 21, 23, 24, 25, 26, 28, 29, 30], "c1": [8, 11], "c2": [8, 11], "c_": [8, 9, 10, 13, 24, 29, 30], "c_0": 24, "c_1": 12, "c_2": 12, "c_3": 12, "c_4": 12, "c_i": [12, 13, 30], "c_k": 24, "ca": [1, 27], "cach": 10, "cal": [0, 8, 10, 12, 13, 29, 30], "calcul": [0, 1, 2, 4, 5, 6, 8, 9, 10, 11, 12, 13, 14, 16, 19, 21, 24, 27, 30], "california": 22, "call": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 18, 19, 20, 21, 22, 24, 25, 27, 28, 29, 30], "calor": [0, 28], "cambridg": [13, 26, 29], "can": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 24, 25, 26, 28, 29], "cancel": [0, 13, 27, 28], "cancer": [5, 10], "cancerpd": 7, "candid": [8, 9, 10, 30], "cannot": [0, 1, 4, 5, 6, 7, 8, 9, 22, 24, 28, 29], "canopi": [0, 20, 22, 27], "canva": [15, 16, 22, 27], "cap": 5, "capabl": [0, 1, 8, 13, 20, 27], "capac": [2, 25], "capita": [], "caption": 22, "captur": [4, 11, 12, 27], "car": [3, 4], "card": [0, 7, 27], "cardin": 1, "care": [11, 15], "carefulli": [13, 30], "carlo": [0, 6, 20, 24, 26, 27], "carri": [2, 6, 7, 22], "cart": 10, "case": [0, 1, 2, 3, 4, 5, 6, 7, 11, 12, 13, 14, 15, 16, 20, 21, 22, 27, 30], "casella": 26, "cast": 1, "cat": [3, 4], "catch": 0, "categor": [0, 1, 3, 9, 11, 27], "categori": [0, 1, 3, 7, 10, 12, 14, 27], "categorical_crossentropi": [1, 3], "caus": [0, 5, 6, 24, 27, 28, 29, 30], "causal": 0, "causat": [0, 27], "cax": 1, "cb": [6, 27], "cbar": 1, "cc": [0, 1, 5, 13, 27, 28, 29], "ccc": [5, 12, 29], "cdf": 24, "cdot": [0, 2, 6, 12, 13, 14, 21, 24, 27, 29, 30], "celebr": [13, 29], "cell": 4, "center": [0, 1, 6, 7, 8, 9, 11, 14, 18, 22, 24, 27, 28, 30], "central": [0, 3, 5, 6, 8, 16, 21, 27, 28], "centroid": [14, 24], "centroid_differ": 14, "centuri": 3, "certain": [0, 3, 6, 7, 9, 24, 27, 28], "cg": 13, "cha": [], "chain": [0, 1, 13, 20, 24, 27], "challeng": 15, "chanc": [1, 5, 13, 24, 30], "chang": [0, 1, 2, 3, 4, 5, 6, 8, 9, 11, 12, 13, 14, 15, 16, 19, 21, 22, 24, 27, 28, 29, 30], "channel": 3, "chapter": [0, 6, 10, 11, 16, 17, 19, 21, 22, 26, 27, 28, 29, 30], "chapter3": [0, 22], "charact": [0, 3, 5, 27, 28, 29], "character": [8, 9, 10, 12, 24], "characterist": [0, 1, 3, 10, 13, 27], "charg": [0, 27], "charl": [], "chase": 4, "chatgpt": [15, 22], "chd": 7, "chddata": 7, "cheap": [5, 28, 29], "cheaper": [1, 13, 30], "check": [1, 3, 4, 5, 11, 13, 15, 16, 19, 21, 27, 30], "checkmark": 3, "checkpoint": 4, "checkpoint_dir": 4, "checkpoint_prefix": 4, "chen": 10, "cheng": 28, "chiaramont": 2, "childcar": 16, "children": 16, "choic": [0, 1, 2, 3, 4, 6, 9, 12, 13, 14, 21, 27, 28, 29, 30], "choleski": [5, 21, 28, 29], "choos": [2, 3, 6, 9, 10, 11, 13, 14, 15, 18, 19, 22, 29], "chosen": [0, 1, 2, 6, 8, 9, 10, 13, 16, 24, 27, 29, 30], "chosen_datapoint": 1, "christian": 26, "christoph": [26, 27], "cifar": 3, "cifar10": 3, "circ": [1, 12, 30], "circl": [0, 8, 12, 28, 30], "circuit": 3, "circumfer": 9, "circumv": [1, 5, 13, 28, 29, 30], "cite": 22, "ckpt": 4, "clariti": 24, "class": [0, 1, 3, 4, 6, 7, 8, 9, 11, 12, 13, 24, 27], "class_nam": [3, 9], "class_val": 9, "class_valu": 9, "classic": [7, 9, 13], "classif": [0, 3, 5, 6, 7, 8, 11, 12, 20, 22, 26, 27, 28], "classifi": [0, 1, 4, 7, 9, 10, 11, 27], "classificaton": 1, "classifii": 10, "clean": 1, "clear": [1, 5, 10, 12, 13, 30], "clearli": [0, 3, 5, 6, 7, 8, 24, 28, 29], "clever": [1, 10], "clf": [0, 6, 8, 9, 10, 27, 28], "clf3": 0, "clf_lasso": 6, "clf_ridg": 6, "cli": 15, "clip": [3, 24], "clone": [15, 25], "close": [0, 1, 2, 4, 6, 8, 9, 11, 12, 13, 14, 18, 24, 26, 27, 29, 30], "closer": [3, 5, 13, 28, 29], "closest": [8, 11, 13, 14], "closur": [20, 27], "cloud": [20, 27], "cluster": [0, 1, 4, 6, 11, 20, 27], "cluster_label": 14, "cm": [1, 2, 3, 6, 8, 13, 29, 30], "cmap": [0, 1, 2, 3, 4, 6, 8, 9, 10, 27], "cmap_arg": 6, "cmd": [9, 15], "cn_": 24, "cnn": 12, "cnn_kera": 3, "cntk": [20, 27], "co": [0, 2, 3, 6, 9, 13, 27], "code": [0, 3, 4, 6, 7, 8, 18, 20, 21, 24, 26], "coef": [0, 27], "coef0": 8, "coef_": [0, 5, 6, 8, 9, 13, 16, 27, 28, 29, 30], "coeff": 5, "coeffici": [0, 3, 5, 6, 7, 8, 9, 13, 18, 21, 27, 28, 30], "coerc": [0, 6, 27], "coin": [10, 24], "coin_toss": 10, "col": [0, 11, 27, 28], "colab": [20, 27], "cold": 9, "colinear": [], "collabor": 22, "collaps": 8, "collect": [2, 6, 10, 11, 17, 20, 24, 26, 27], "collinear": [5, 28, 29], "color": [0, 3, 4, 6, 8, 9, 10, 24, 30], "color_channel": 3, "color_cod": 6, "colorbar": [1, 6], "colsample_bytre": 10, "colsaobject": 10, "column": [0, 1, 2, 5, 6, 7, 8, 9, 11, 12, 16, 17, 18, 21, 27, 28, 29, 30], "columntransform": 9, "com": [4, 6, 15, 16, 20, 22, 26, 27, 29, 30], "combin": [1, 2, 5, 6, 7, 10, 15, 18, 24], "come": [0, 1, 3, 4, 5, 12, 13, 14, 15, 27, 28, 29, 30], "command": [0, 1, 15], "comment": [0, 4, 5, 6, 22], "commerci": [0, 20, 22, 27], "commit": 15, "commod": [0, 27], "common": [0, 1, 3, 5, 6, 7, 9, 11, 13, 14, 16, 22, 24, 27, 28, 29, 30], "commonli": [0, 1, 4, 6, 7, 9, 13, 14, 28, 30], "commun": [0, 12, 15, 22], "commut": 3, "commutatitav": 3, "compact": [0, 1, 3, 5, 6, 7, 9, 11, 12, 13, 14, 27, 28], "compair": 0, "compar": [0, 3, 4, 5, 6, 11, 13, 18, 21, 22, 27, 28, 29, 30], "comparison": [2, 4, 13], "compat": 7, "compens": 30, "compet": 0, "competit": 10, "compil": [0, 1, 3, 4, 13, 20, 21, 27], "complet": [0, 2, 3, 4, 9, 12, 15, 16, 17, 18, 19, 27], "completenn": 12, "complex": [1, 5, 8, 9, 11, 12, 13, 16, 19, 27, 29], "complic": [0, 1, 9, 13, 22, 27, 29, 30], "compoment": 28, "compon": [0, 1, 3, 4, 5, 6, 7, 9, 14, 16, 20, 27, 28, 29], "components_": 11, "compos": [9, 12, 13, 14, 20, 27], "compphys": [0, 6, 16, 20, 22, 23, 25, 26, 27, 28, 29], "compress": [0, 27, 28], "compris": 6, "compromis": [5, 28, 29], "compulsori": [20, 27], "comput": [0, 1, 2, 3, 4, 5, 6, 7, 8, 10, 11, 12, 13, 15, 16, 17, 18, 20, 21, 22, 23, 24, 26, 27, 28, 29], "computation": [0, 3, 6, 9, 13, 24, 27, 29, 30], "computationalscienceuio": 27, "computerlab": 22, "concaten": [2, 4, 6, 14], "concav": [1, 13, 28, 29], "concentr": 10, "concept": [0, 2, 20, 27, 28], "conceptu": [12, 13, 29], "concern": [0, 1, 4, 7, 27, 29], "concic": 27, "conclud": [0, 5, 13], "conclus": 1, "cond": 2, "conda": [0, 1, 20, 22, 27], "condis": 28, "condit": [0, 2, 4, 5, 6, 8, 9, 11, 13, 24, 27, 28, 30], "conduct": 20, "condwav": 2, "confid": [0, 5, 6, 7, 8, 19, 27, 28], "configur": 3, "confirm": [5, 12], "confus": [5, 6, 7, 10, 21, 28], "confusion_matrix": 9, "congruenti": 24, "conjug": [4, 8], "conjugaci": 13, "conjunct": 3, "connect": [0, 1, 3, 4, 9, 11, 12, 13, 21, 27, 28, 29], "consequ": [5, 6, 8, 10, 12, 13, 28, 29, 30], "conserv": [5, 14, 28, 29], "consid": [0, 1, 2, 3, 5, 6, 7, 8, 9, 10, 12, 13, 16, 21, 22, 24, 27, 28, 29, 30], "consider": [0, 1, 5, 13, 27, 28, 29], "consist": [1, 2, 3, 4, 6, 12, 13, 22, 24, 28, 29], "constant": [0, 2, 4, 5, 6, 8, 12, 13, 16, 18, 24, 27, 28, 29, 30], "constitu": [0, 27], "constitut": [2, 6], "constrain": [1, 3, 5, 7, 11, 29], "constraint": [5, 6, 8, 13, 28, 29, 30], "construct": [0, 1, 2, 3, 5, 6, 7, 8, 9, 10, 11, 21, 24, 27, 28], "contact": [0, 27], "contain": [0, 2, 3, 4, 5, 6, 7, 8, 9, 11, 12, 13, 15, 18, 19, 21, 22, 24, 26, 27, 28, 29, 30], "contemporari": 27, "content": [1, 15, 20, 21, 27, 29, 30], "context": [6, 10, 13, 22, 29, 30], "contigu": 21, "contin": 19, "continu": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 12, 13, 19, 21, 22, 24, 27, 28, 29, 30], "contour": [9, 10, 13], "contourf": [8, 9, 10], "contrast": [1, 4, 9, 10, 12, 27, 30], "contribut": [0, 3, 5, 13, 18, 24, 27, 28, 29, 30], "contributor": [0, 22], "control": [0, 1, 3, 9, 13, 15, 20, 27], "conv": [3, 4], "conv2d": [3, 4], "conv2dtranspos": 4, "convei": 27, "conveni": [5, 6, 12, 13, 21, 22, 27, 29, 30], "convent": [12, 28], "converg": [1, 2, 4, 5, 8, 13, 14, 18, 28, 29], "convergencewarn": [], "convers": 30, "convert": [0, 1, 4, 5, 9, 11, 13, 21, 27, 28, 29], "converttomatrix": 4, "convex": [4, 5, 7, 28, 30], "convinc": [13, 29], "convolut": [1, 4, 20, 27], "cool": [4, 9], "coolwarm": 6, "coordin": [5, 12, 14, 28, 29, 30], "coorel": [], "copi": [0, 1, 14, 15, 28], "core": 10, "corel": 27, "coronari": 7, "corr": [5, 7, 11, 28], "correalt": [11, 20], "correct": [0, 1, 2, 3, 4, 5, 7, 13, 15, 21, 24, 27, 28, 29], "correctli": [1, 2, 6, 7, 10, 18, 19, 22], "correl": [0, 1, 3, 5, 6, 7, 10, 12, 13, 20, 24, 27, 29, 30], "correlation_matrix": [5, 7, 11, 28], "correspond": [0, 3, 5, 6, 8, 9, 11, 12, 20, 21, 22, 24, 27, 28, 29], "cortex": 12, "cosin": [3, 6], "cost": [0, 2, 3, 5, 6, 7, 8, 9, 12, 13, 16, 17, 18, 19, 22, 27], "cost_deep_grad": 2, "cost_funct": 2, "cost_function_deep": 2, "cost_function_deep_grad": 2, "cost_function_grad": 2, "cost_grad": 2, "cost_histori": [], "cost_ol": [], "cost_ridg": [], "cost_sum": 2, "costol": [13, 30], "could": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 15, 16, 17, 18, 21, 22, 24, 27, 28, 29, 30], "coulomb": [0, 27], "count": [0, 9, 15, 23, 24, 25, 27], "counteract": 30, "counterpart": 27, "countor": 13, "coupl": [4, 5, 6], "cours": [0, 1, 3, 5, 11, 15, 16, 17, 22, 25, 28], "coursework": 15, "courvil": [26, 27, 28, 30], "cov": [5, 6, 11, 21, 24, 27, 28], "cov_xi": [5, 11, 28], "cov_xx": [5, 11, 28], "cov_yi": [5, 11, 28], "covari": [0, 7, 20, 21, 27, 29], "covariance_matrix": [5, 11, 14], "cover": [0, 5, 20, 25, 26, 28, 29], "covert": [0, 27], "covxi": 24, "covxx": 24, "covxz": 24, "covyi": 24, "covyz": 24, "covzz": 24, "cpu": 1, "craft": 3, "creat": [1, 3, 4, 5, 9, 10, 11, 12, 15, 18, 19, 20, 27, 30], "create_biases_and_weight": 1, "create_convolutional_neural_network_kera": 3, "create_neural_network_kera": 1, "create_x": [5, 11], "credit": [0, 7, 25, 27], "crim": [], "crime": [], "criteria": [0, 4, 9, 10, 14, 24, 27], "criterion": [9, 10, 13, 18, 29, 30], "critic": [6, 22, 28], "critiqu": 22, "cross": [0, 1, 3, 7, 9, 10, 13, 15, 20, 24, 27, 28, 29, 30], "cross_entropi": 4, "cross_val_scor": 6, "cross_valid": [7, 10], "crossvalid": 6, "crucial": [1, 24, 30], "cs231": 3, "csr_matrix": [21, 27], "csv": [0, 4, 6, 7, 9], "ctnk": 1, "cubic": 0, "cumbersom": 5, "cumsum": [10, 11, 27], "cumul": [7, 10, 24, 30], "cumulative_heads_ratio": 10, "cup": 5, "current": [1, 2, 3, 4, 13, 14, 15, 16, 26, 29, 30], "curs": [0, 28], "curv": [6, 7, 10, 12, 22], "curvatur": [13, 29, 30], "custom": [6, 14], "custom_cmap": [9, 10], "custom_cmap2": [9, 10], "cutpoint": 9, "cv": [6, 7, 10], "cvxbook": [13, 29], "cvxopt": [5, 8, 28], "cycl": [1, 12], "d": [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 13, 14, 15, 16, 17, 19, 21, 24, 25, 27, 28, 29, 30], "d2_g_t": 2, "d_f": [13, 29], "d_g_t": 2, "d_net_out": 2, "da": 3, "dagger": [5, 21, 28, 29], "dai": [1, 9, 20], "damp": 3, "darget": 9, "darkr": 24, "dat": [0, 27], "dat_id": [0, 6, 7, 9, 27], "data": [2, 4, 5, 8, 10, 12, 13, 14, 16, 19, 21, 22, 26, 29, 30], "data1": 14, "data2": 14, "data3": 14, "data4": 14, "data_id": [0, 6, 7, 9, 27], "data_indic": 1, "data_panda": 27, "data_path": [0, 6, 7, 9, 27], "databas": 1, "datafil": [0, 6, 7, 9, 27], "datafram": [0, 4, 5, 7, 9, 11, 27, 28], "datapoint": [1, 5, 6, 7, 11, 13, 16, 29, 30], "datasci": [15, 16], "dataset": [0, 4, 6, 7, 8, 9, 10, 11, 13, 14, 16, 22, 27, 29, 30], "datatyp": 4, "date": [15, 18, 22, 27, 28, 29, 30], "daughter": 10, "david": 26, "dbh": 1, "dbo": 1, "dcomposit": 21, "ddot": 2, "dead": 1, "deadlin": 15, "deal": [0, 1, 3, 5, 6, 8, 11, 13, 14, 19, 21, 24, 27, 28, 29, 30], "dealt": 0, "debt": 7, "debug": [0, 5, 6, 28, 29, 30], "decad": [0, 3, 30], "decai": [0, 13, 24, 27], "decemb": [25, 27], "decent": 10, "decid": [0, 2, 3, 5, 6, 9, 18, 28, 29, 30], "decim": [0, 27], "decis": [0, 1, 8, 11, 20, 26, 27], "decision_funct": 8, "decision_tre": 9, "decisiontreeclassifi": [9, 10], "decisiontreeregressor": [0, 9, 10], "declar": [0, 4, 21, 27], "decompos": [5, 6, 21, 28, 29], "decomposit": [0, 6, 12, 27], "decompost": [5, 28, 29], "deconvolut": 3, "decorrel": [10, 13, 30], "decreas": [1, 2, 4, 5, 6, 10, 11, 13, 19, 29, 30], "deduc": [0, 27], "deep": [3, 7, 12, 13, 20, 26, 28, 29, 30], "deep_neural_network": 2, "deep_param": 2, "deep_tree_clf": [9, 10], "deep_tree_clf1": 9, "deep_tree_clf2": 9, "deepen": [5, 20, 27], "deeper": [0, 3, 4, 27], "deeplearningbook": [26, 27, 29, 30], "deer": 3, "def": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 13, 14, 16, 17, 24, 27, 28, 29, 30], "def_covari": 24, "default": [0, 1, 2, 4, 6, 7, 21, 27, 28], "default_tim": 4, "defect": [5, 28, 29], "defici": [5, 28, 29], "defin": [0, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 16, 17, 18, 19, 21, 22, 24, 28, 29, 30], "definit": [1, 2, 5, 6, 7, 8, 10, 11, 12, 13, 21, 24, 28, 29, 30], "defint": 24, "degre": [3, 5, 6, 8, 9, 10, 11, 15, 16, 19, 22, 24, 27, 29, 30], "deisenroth": 28, "del": 1, "delet": [6, 15], "delimit": 4, "deliv": [15, 23, 27], "delta": [0, 2, 3, 6, 8, 12, 13, 14, 27, 30], "delta_": [1, 21], "delta_0": 3, "delta_1": 3, "delta_2": 3, "delta_3": 3, "delta_4": 3, "delta_5": 3, "delta_h": [0, 1, 27], "delta_j": [3, 12], "delta_k": 12, "delta_l": [1, 3], "delta_momentum": [13, 30], "delta_n": [0, 3, 27], "delug": 20, "delv": 0, "demand": [13, 29], "demonstr": [0, 3, 5, 6, 7, 11, 12, 20, 27, 28, 29], "den": 4, "denomin": [1, 5, 30], "denot": [1, 2, 6, 7, 13, 24, 29, 30], "dens": [1, 3, 4], "densiti": [0, 2, 6, 24], "depart": [25, 27, 28, 29, 30], "depend": [0, 1, 2, 4, 5, 6, 7, 8, 11, 12, 13, 15, 16, 20, 21, 22, 24, 27, 28, 29, 30], "depict": 24, "deploy": [0, 20, 22, 27], "depth": [0, 3, 9, 10, 21], "deriv": [0, 1, 2, 6, 7, 8, 10, 11, 13, 18, 20, 22, 27], "derivati": 13, "derivative_fn": 13, "descend": [5, 9, 11, 28, 29], "descent": [0, 1, 3, 7, 8, 12, 27, 28], "describ": [0, 2, 4, 5, 6, 8, 10, 11, 12, 13, 19, 21, 22, 27, 30], "descript": [0, 8, 9, 22, 27], "design": [0, 1, 3, 4, 5, 6, 7, 10, 11, 12, 13, 17, 18, 22, 27, 29, 30], "designmatrix": [0, 27], "desir": [0, 2, 4, 5, 13, 14, 27, 28, 29, 30], "desktop": 15, "despit": [1, 12], "destroi": 21, "det": [5, 21, 28, 29], "detail": [0, 6, 11, 13, 14, 18, 21, 22, 28, 29, 30], "detect": [3, 8, 12], "determin": [0, 2, 3, 4, 5, 6, 8, 9, 10, 11, 12, 13, 18, 21, 24, 27, 28, 29, 30], "determinist": [7, 13, 24, 29, 30], "dev": [1, 22], "develop": [0, 3, 5, 8, 10, 11, 12, 20, 21, 22, 27, 28], "deviat": [0, 1, 2, 4, 5, 6, 17, 18, 19, 22, 24, 27, 28, 30], "devis": 12, "df": [4, 8, 11, 13, 27], "df1": 27, "di": [], "diag": [5, 8, 28, 29, 30], "diagnost": [1, 10], "diagon": [0, 5, 7, 13, 18, 19, 21, 24, 27, 28, 29, 30], "diagonaliz": [5, 28, 29], "diagram": 10, "diagsvd": 6, "dice": [6, 24], "dict": [6, 8], "dictionari": [], "did": [0, 1, 5, 6, 7, 10, 11, 14, 16, 22, 27], "die": 1, "diff": 2, "diff1": 2, "diff2": 2, "diff_ag": 2, "diffeent": 8, "differ": [0, 1, 2, 3, 4, 5, 6, 7, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 24, 26, 27, 28, 29], "differenti": [0, 3, 16, 20, 21, 27, 28, 29], "difficult": [0, 1, 6, 10, 13, 24, 27, 30], "difficulti": [0, 1, 13, 27, 29], "diffonedim": 2, "digit": [0, 1, 3, 4, 6, 25, 27], "dilemma": [13, 30], "dilut": 1, "dim": [4, 11, 14, 21], "dimens": [0, 1, 2, 3, 4, 5, 8, 11, 14, 16, 21, 27, 28, 29], "dimension": [0, 4, 5, 6, 9, 11, 13, 14, 19, 20, 21, 22, 27, 28, 29, 30], "dimensionless": [0, 3, 27], "diment": 21, "diminish": 30, "dimnsion": 4, "diod": 3, "direct": [0, 1, 2, 4, 11, 12, 13, 14, 27, 28, 29, 30], "directli": [1, 4, 5, 6, 18, 24, 28, 29], "disadvantag": [0, 27, 30], "disappear": [3, 6], "disc_loss": 4, "disc_tap": 4, "discard": [6, 11], "disciplin": [0, 3, 12], "disclaim": 24, "discord": 27, "discourag": [13, 15, 29], "discov": [0, 27], "discover": 5, "discret": [1, 3, 5, 7, 13], "discrimin": [4, 7, 10, 11], "discriminator_loss": 4, "discriminator_loss_list": 4, "discriminator_model": 4, "discriminator_optim": 4, "discuss": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 16, 18, 19, 20, 21, 22, 24, 26, 27, 28, 29, 30], "diseas": 7, "disguis": [6, 28, 30], "disord": [1, 7], "displai": [0, 1, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 14, 22, 24, 27, 28], "displaystyl": [0, 5, 17, 27, 28, 29, 30], "disregard": [0, 27], "dissimilar": [11, 14], "dist": 14, "distanc": [8, 9, 11, 14, 24], "distance_list": 9, "distinct": [3, 7, 8, 9, 10, 14], "distinctli": 8, "distinguish": [0, 4, 7, 8, 24, 27], "distplot": [], "distribut": [0, 1, 4, 6, 7, 10, 11, 13, 14, 18, 19, 20, 21, 22, 27, 28, 29, 30], "distrubut": [0, 20, 22, 27], "dive": [0, 8, 21, 27], "diverg": [1, 13, 29, 30], "divid": [0, 1, 3, 5, 6, 7, 8, 9, 11, 12, 18, 24, 27, 28, 30], "divis": [6, 8, 9, 13, 18, 21, 24, 30], "dna": 7, "dnn": [0, 1, 2, 4, 12, 27], "dnn1": 4, "dnn2_gru2": 4, "dnn_kera": 1, "dnn_model": 1, "dnn_numpi": 1, "dnn_scikit": [0, 1, 27], "do": [0, 2, 3, 4, 5, 6, 8, 9, 10, 11, 12, 13, 14, 15, 16, 18, 19, 21, 22, 27, 28, 29], "doc": [0, 15, 16, 20, 22, 23, 25, 26, 27], "document": [4, 13, 15], "doe": [0, 1, 2, 3, 4, 5, 6, 8, 10, 11, 12, 13, 15, 16, 17, 18, 21, 22, 24, 27, 30], "doesn": [3, 9, 12, 27], "dog": [1, 3, 4], "domain": [5, 8, 13, 22, 29], "domin": [0, 27], "don": [0, 1, 3, 5, 6, 8, 11, 13, 15, 16, 20, 22, 27, 28, 30], "done": [0, 2, 3, 4, 5, 6, 9, 10, 11, 13, 16, 21, 22, 27, 28, 29, 30], "dot": [0, 2, 3, 5, 6, 7, 8, 9, 10, 11, 12, 13, 18, 21, 22, 24, 27, 28, 29, 30], "doubl": [3, 4, 16, 21, 27], "doubli": 1, "doubt": 22, "down": [0, 3, 6, 9, 11, 12, 13, 29, 30], "download": [0, 1, 3, 5, 6, 15, 21, 26, 27], "downsampl": 3, "dozen": 1, "dq": 6, "drag": 13, "dramat": 11, "drastic": 4, "draw": [4, 6, 10, 13, 29], "drawback": [0, 1, 3, 13, 28, 29, 30], "drawn": [1, 4, 6, 7, 11, 24, 27], "drive": [3, 4], "driven": 3, "drop": [0, 1, 5, 6, 11, 13, 24, 27, 28, 29], "dropna": [0, 6, 27], "dropout": 4, "dt": [2, 3, 13, 24], "dtype": [0, 1, 3, 4, 14, 21, 27], "dub": [0, 27], "due": [1, 2, 5, 6, 8, 10, 12, 13, 18, 25, 27, 28, 29, 30], "dummi": [], "dure": [0, 1, 3, 4, 8, 9, 11, 20, 22, 27, 30], "dwell": [], "dwh": 1, "dwo": 1, "dx": [2, 3, 8, 24], "dx_1": 24, "dx_1p": 6, "dx_2p": 6, "dx_mp": 6, "dx_n": 24, "dxp": 6, "dy": [1, 8, 24], "dynam": 4, "dz": 8, "e": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 24, 25, 27, 28, 29, 30], "e_": [0, 2, 27], "each": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 18, 20, 21, 23, 24, 25, 27, 28, 29, 30], "eapprox": [0, 27], "earli": [1, 13, 30], "earlier": [0, 5, 7, 8, 9, 11, 12, 13, 27, 28], "earthexplor": 6, "eas": [6, 9, 14], "easi": [0, 5, 6, 7, 8, 9, 10, 11, 12, 13, 15, 20, 21, 27, 28, 29, 30], "easier": [5, 6, 8, 9, 13, 15, 22, 24, 27, 28, 29], "easiest": [13, 18], "easili": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 21, 22, 27, 28, 29, 30], "eastern": [25, 27], "ebind": [0, 27], "eblock": 9, "econometr": 27, "economi": 5, "ecosystem": [20, 27], "ect": 23, "edg": 3, "edgecolor": 6, "editor": 15, "edu": [13, 22, 29], "educ": [0, 22, 27], "eff": 24, "effect": [1, 4, 10, 13, 16, 17, 18, 24, 30], "effic": 1, "effici": [0, 3, 10, 13, 20, 21, 24, 27, 30], "efron": 6, "egrad": 13, "eig": [5, 11, 13, 21, 24, 27, 28, 29, 30], "eigen": 24, "eigenpair": [5, 11, 28, 29], "eigenvalu": [0, 5, 8, 11, 13, 21, 27, 28, 29, 30], "eigenvector": [5, 11, 13, 28, 29], "eight": [21, 27], "eigval": [21, 24, 27], "eigvalu": [11, 13, 29, 30], "eigvec": [21, 24, 27], "eigvector": [11, 13, 29, 30], "eir": [25, 27], "eispack": [21, 27], "either": [1, 5, 6, 7, 8, 9, 10, 11, 13, 18, 22, 24, 27, 28, 29, 30], "eivind": 25, "eivinsto": 25, "ekstr\u00f8m": 4, "elabor": 24, "elarn": 3, "electr": [0, 3, 12, 27], "electron": 27, "eleg": 11, "element": [1, 2, 3, 4, 5, 6, 7, 8, 11, 12, 13, 19, 20, 21, 22, 26, 28, 30], "elementari": [10, 13, 21], "elementwis": [3, 13], "elementwise_grad": [2, 13], "elessar": 27, "elif": 14, "elim": 21, "elimin": [3, 8], "elin": [25, 27], "ell_": [], "ellipsi": 16, "els": [1, 3, 4, 7, 9, 12, 13, 16, 21], "elu": 1, "elus": [0, 27], "email": [23, 25, 27], "embed": [0, 11, 28], "embodi": [6, 22], "emit": 24, "emner": 26, "emph": 30, "emphas": [0, 10, 20, 27], "emphasi": [0, 20, 26, 27], "empir": [1, 11, 24], "emploi": [0, 1, 5, 6, 11, 13, 24, 27, 28, 29], "employ": 0, "empti": [6, 10, 15], "emul": 12, "en": [20, 22, 26], "enabl": 11, "enbodi": 6, "encod": [0, 3, 5, 9, 11, 14, 27, 28, 29], "encompass": [0, 22, 24], "encount": [0, 1, 5, 7, 13, 15, 22, 24, 27, 28, 29], "encourag": [15, 22], "end": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 21, 24, 25, 27, 28, 29, 30], "endpoint": [3, 6], "energi": [0, 4, 6], "enforc": 12, "eng": 26, "engin": [0, 1, 3, 4, 20, 27], "english": 22, "enocurag": 22, "enorm": 3, "enough": [0, 6, 13, 27, 29, 30], "ensembl": [1, 9, 27], "ensur": [0, 1, 2, 3, 5, 6, 11, 13, 18, 24, 28, 29, 30], "entail": 27, "enter": [5, 6, 28, 29, 30], "enthought": [0, 20, 22, 27], "entir": [1, 3, 7, 9, 20, 24, 27, 30], "entiti": [9, 12, 21, 27], "entri": [0, 5, 8, 11, 12, 21, 27, 28, 30], "entropi": [1, 3, 7, 10, 13, 27, 29, 30], "enumer": [0, 1, 2, 3, 4, 6, 8, 27, 28, 30], "env": 24, "environ": [2, 20, 22, 27], "environemnt": 15, "eo": [0, 6], "eol": 0, "eosfit": 0, "epoch": [0, 1, 3, 4, 12, 13, 27, 30], "epsilon": [0, 5, 6, 7, 13, 22, 27, 28, 29, 30], "epsilon_": [0, 27], "epsilon_0": [0, 27], "epsilon_1": [0, 27], "epsilon_2": [0, 27], "epsilon_i": [0, 27, 28], "eq": [3, 13, 14, 21, 24, 29], "eqnarrai": [3, 5, 6], "equal": [0, 1, 2, 3, 4, 5, 6, 8, 9, 11, 12, 13, 14, 16, 18, 21, 22, 24, 27, 28, 29, 30], "equat": [1, 3, 4, 5, 6, 7, 8, 9, 10, 11, 13, 14, 17, 19, 21, 24, 27, 30], "equilibrium": [2, 12], "equiv": [3, 13, 21, 24, 29, 30], "equival": [0, 1, 5, 7, 8, 11, 13, 20, 21, 27, 28, 29, 30], "erf": 24, "eriador": 27, "err": [0, 10], "err_": 6, "err_sqr": 2, "errat": [13, 29, 30], "erron": 2, "error": [1, 2, 4, 5, 6, 7, 9, 11, 12, 13, 15, 16, 17, 18, 19, 20, 21, 22, 24, 30], "error_estimate_corr_tim": 24, "error_hidden": 1, "error_output": 1, "escap": [13, 29, 30], "especi": [1, 3, 9, 12, 13, 15, 18, 22, 30], "essenti": [0, 5, 6, 9, 10, 12, 14, 15, 22, 24, 28, 29], "establish": [0, 6, 10, 11, 16, 22], "estim": [0, 1, 5, 6, 7, 10, 11, 13, 19, 20, 24, 27, 28, 29, 30], "estimated_mse_fold": 6, "estimated_mse_kfold": 6, "estimated_mse_sklearn": 6, "et": [0, 2, 4, 16, 17, 26, 27, 28, 29], "eta": [0, 1, 3, 8, 12, 13, 18, 27, 29, 30], "eta0": [8, 13], "eta_": 13, "eta_j": 30, "eta_t": [13, 30], "eta_v": [0, 1, 3, 27], "etc": [0, 1, 3, 5, 7, 8, 9, 11, 12, 13, 14, 20, 21, 22, 24, 28, 29, 30], "ethic": 20, "euclidean": [0, 14, 28, 30], "evalu": [0, 2, 3, 4, 5, 6, 9, 13, 15, 16, 17, 19, 22, 24, 27, 28, 29, 30], "evalut": [13, 22], "even": [0, 1, 3, 4, 5, 6, 8, 9, 10, 11, 12, 13, 14, 20, 21, 24, 27, 28, 29, 30], "evenli": 4, "event": [5, 7, 10, 24], "eventu": [0, 5, 6, 11, 12, 13, 22, 25, 28, 29, 30], "everi": [0, 1, 2, 3, 4, 5, 6, 9, 10, 11, 12, 13, 14, 15, 20, 24, 25, 27, 28, 29, 30], "everyth": [4, 12, 16, 18], "everywher": [4, 13, 29], "evolv": 0, "exact": [0, 5, 11, 12, 13, 21, 24, 27, 28], "exactli": [0, 3, 4, 6, 12, 18, 20, 28, 30], "exam": 27, "examin": 6, "exampl": [0, 5, 11, 12, 13, 15, 16, 18, 20, 21, 22, 24, 26], "exce": [1, 12, 13, 30], "excel": [0, 1, 4, 5, 10, 22, 27, 28], "except": [3, 4, 6, 8, 9, 21], "excess": [0, 27], "excit": 0, "exclud": [1, 6, 12, 28, 30], "exclus": [0, 1, 3, 6, 24, 27], "execut": [2, 5, 13, 15, 28, 29], "exemplifi": [13, 30], "exercic": [25, 27], "exercis": [5, 20, 22, 23, 25, 27, 29, 30], "exhaust": 6, "exhibit": [0, 5, 6, 8, 27, 28], "exist": [0, 1, 2, 3, 5, 6, 7, 8, 9, 13, 19, 21, 22, 27, 29], "exit": [5, 21, 28, 29], "exp": [0, 1, 2, 5, 6, 7, 8, 10, 11, 12, 13, 16, 17, 19, 24, 28, 29, 30], "exp_term": 1, "expand": [5, 7, 11, 13, 29], "expans": [0, 3, 5, 8, 10, 12, 13, 27, 28, 29], "expect": [0, 1, 5, 6, 7, 11, 12, 13, 15, 18, 20, 22, 27, 28, 30], "expectation_value_of_h_wrt_p": 24, "expens": [6, 10, 13, 16, 29, 30], "experi": [0, 1, 6, 8, 13, 15, 20, 22, 27, 28, 29], "experiment": [0, 4, 6, 9, 24, 27], "expert": [1, 9], "explain": [0, 6, 9, 10, 11, 13, 16, 19, 22, 27, 29], "explained_variance_ratio_": 11, "explanatori": [0, 27], "explicit": [0, 3, 6, 13, 21, 22, 27, 28, 29, 30], "explicitli": [0, 4], "explod": 1, "exploit": [0, 3, 12, 13, 27], "explor": [1, 4, 6, 8, 13, 18, 20, 22, 27, 29, 30], "expon": 1, "exponenti": [0, 1, 5, 6, 10, 13, 24, 27, 29], "export": [9, 15, 16], "export_graphviz": 9, "export_text": 9, "exporttext": 9, "expos": 20, "express": [0, 2, 3, 5, 6, 7, 10, 12, 13, 18, 21, 22, 24, 27, 29, 30], "exptmean": 24, "exptvari": 24, "extend": [0, 2, 7, 11, 13, 20, 27, 30], "extens": [0, 12, 15, 20, 27], "extent": [0, 1, 6, 26], "extern": [3, 6, 9], "extra": [1, 3, 5, 15, 25, 27, 28, 29], "extract": [0, 3, 5, 6, 7, 8, 11, 13, 16, 17, 21, 27, 28], "extrapol": [0, 27], "extrem": [0, 1, 4, 5, 6, 7, 8, 9, 13, 15, 16, 21, 28, 29, 30], "extremum": [13, 29], "extrins": 11, "ey": [0, 5, 6, 13, 14, 18, 21, 27, 28, 29, 30], "f": [0, 1, 2, 3, 4, 5, 6, 7, 8, 10, 12, 13, 14, 15, 16, 17, 18, 19, 21, 24, 25, 27, 28, 29, 30], "f1": 13, "f11": [0, 27], "f12": [0, 27], "f13": [0, 27], "f1_grad": 13, "f1d": 13, "f2": 13, "f2_grad_x1": 13, "f2_grad_x1_analyt": 13, "f2_grad_x2": 13, "f2_grad_x2_analyt": 13, "f3": 13, "f3_grad": 13, "f3_grad_analyt": 13, "f4": 13, "f4_grad": 13, "f4_grad_analyt": 13, "f5": 13, "f5_grad": 13, "f6": 13, "f6_for": 13, "f6_for_grad": 13, "f6_grad_analyt": 13, "f6_while": 13, "f6_while_grad": 13, "f7": 13, "f7_grad": 13, "f7_grad_analyt": 13, "f8": 13, "f8_grad": 13, "f9": [0, 13, 27], "f9_altern": 13, "f9_alternative_grad": 13, "f9_grad": 13, "f_": 10, "f_0": [3, 10], "f_1": [10, 13, 29], "f_2": [12, 13, 29], "f_3": 12, "f_d": 24, "f_grad": 13, "f_grad_analyt": 13, "f_i": [0, 6, 12, 16], "f_m": [3, 10], "f_n": 3, "f_vec": 2, "face": [13, 27, 29], "facecolor": [6, 8, 24], "facil": [0, 20], "facilit": 12, "fact": [0, 1, 3, 5, 9, 11, 12, 13, 27, 28, 29], "factor": [0, 1, 3, 5, 6, 9, 10, 11, 13, 21, 24, 27, 28, 29], "factori": 13, "fade": 6, "fafab0": [9, 10], "fail": [0, 6, 13, 25, 27, 29], "failur": 7, "fairli": [1, 2, 18, 24, 30], "faisal": [16, 28], "fake": 4, "fake_loss": 4, "fake_output": 4, "fall": [8, 9, 23], "fals": [0, 1, 2, 3, 4, 5, 6, 7, 9, 10, 14, 16, 17, 21, 27, 28, 29, 30], "famili": [0, 7, 8, 24, 28, 30], "familiar": [0, 3, 5, 6, 8, 15, 20, 21, 22, 24, 27], "famou": [6, 12], "far": [0, 3, 4, 5, 6, 8, 11, 12, 13, 14, 16, 27, 28, 29], "fashion": [0, 9, 10, 27], "fast": [1, 3, 6, 10, 12, 13, 20, 24, 27, 29, 30], "faster": [1, 11, 13, 30], "fastest": [13, 21, 29], "favor": 7, "favorit": 24, "fc": 3, "featur": [0, 1, 3, 5, 6, 7, 8, 10, 11, 12, 13, 15, 17, 18, 19, 20, 24, 27, 29, 30], "feature_nam": [1, 7, 9], "feautur": 9, "fed": 1, "feed": [0, 2, 3, 11, 20, 27], "feed_forward": 1, "feed_forward_out": 1, "feed_forward_train": 1, "feedback": [4, 27], "feeddorward": 4, "feedforward": [1, 4, 12], "feel": [0, 5, 6, 11, 13, 15, 16, 18, 20, 22, 25, 27], "feet": [], "felt": 22, "fetch": [6, 15], "few": [1, 3, 4, 5, 9, 17, 18, 24, 27], "fewer": [0, 9, 11, 27], "ffnn": [1, 12], "field": [0, 3, 6, 12, 20], "fifth": [0, 6, 27], "fig": [0, 1, 2, 3, 4, 6, 7, 12, 13, 14, 22, 27], "fig_id": [0, 6, 7, 9, 27], "figaxi": 24, "figsiz": [0, 1, 2, 3, 4, 6, 7, 8, 9, 10, 27], "figur": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 12, 13, 14, 16, 20, 22, 27, 28, 29, 30], "figure_id": [0, 6, 7, 9, 27], "figurefil": [0, 6, 7, 9, 27], "file": [0, 4, 5, 6, 7, 9, 15, 22, 27], "file_prefix": 4, "filenam": 27, "fill": [5, 9, 18, 28, 29], "filter": [3, 4], "final": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 13, 14, 18, 19, 22, 23, 24, 25, 27, 29], "financ": 0, "find": [0, 1, 2, 3, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 20, 22, 24, 27, 28, 29, 30], "fine": [0, 14], "finish": 2, "finit": [3, 5, 6, 12, 13, 17, 24, 28, 29], "finnicki": 15, "first": [0, 1, 2, 3, 5, 6, 7, 8, 9, 10, 11, 13, 14, 15, 16, 18, 21, 22, 24, 25, 26, 28, 30], "first_moment": 30, "first_term": 30, "firsteigvector": 11, "fit": [1, 3, 4, 5, 6, 7, 8, 9, 11, 12, 13, 17, 18, 19, 22, 24, 28, 30], "fit_beta": 28, "fit_intercept": [0, 5, 6, 16, 28, 29, 30], "fit_mod": 9, "fit_theta": [6, 30], "fit_transform": [0, 6, 8, 9, 11, 15, 19], "fiti": [0, 27], "five": [0, 9, 27, 28], "fix": [0, 3, 4, 6, 10, 11, 12, 13, 22, 27], "flag": 4, "flat": [12, 13, 29, 30], "flatten": [1, 3, 4, 5, 21], "flexibl": [1, 6, 8, 10, 12, 27, 30], "flip": [25, 27], "float": [0, 3, 4, 5, 9, 11, 13, 14, 21, 27, 28, 29], "float32": [4, 9], "float64": [4, 21, 27], "flop": [5, 21, 28, 29], "flow": [1, 4, 12], "fluctuat": 5, "fly": 11, "fm": 0, "fmax": 3, "fmesh": 13, "fn": 7, "focu": [0, 3, 4, 5, 6, 15, 20, 22, 26, 27, 28, 29, 30], "focus": [1, 6, 7, 21, 28, 30], "fold": [6, 9, 22], "folder": [0, 4, 6, 15, 22, 27], "follow": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 19, 20, 21, 22, 24, 25, 26, 27, 28, 29, 30], "font": [7, 24, 27], "fontdict": 24, "fontsiz": [1, 6, 8, 9, 10, 24], "fontweight": 1, "footprint": 3, "foral": [8, 28], "forc": [0, 5, 6, 10, 11, 28, 29, 30], "forcast": 4, "forecast": [4, 12], "forest": [0, 1, 9, 20, 27], "forget": [11, 30], "form": [0, 3, 4, 5, 6, 7, 8, 9, 11, 12, 13, 15, 16, 20, 21, 22, 24, 27, 28, 29, 30], "formal": [3, 4, 14, 18, 24], "format": [0, 1, 3, 4, 6, 7, 8, 9, 10, 11, 20, 24, 26], "format_data": 4, "formatstrformatt": [6, 13, 29, 30], "formul": [4, 6, 11, 14], "formula": [3, 13, 24, 29], "forth": [4, 12], "fortran": [0, 20, 21, 27], "fortran2003": [20, 27], "fortran2008": 22, "fortran90": 24, "fortun": [0, 11, 28], "forward": [0, 3, 6, 20, 21, 27], "found": [1, 2, 4, 5, 6, 12, 13, 19, 22, 27, 28, 30], "foundat": [20, 27], "four": [4, 5, 6, 8, 12, 21, 23, 25, 27, 29], "fourier": [0, 27], "fourierdef1": 3, "fourierdef2": 3, "fourierseriessign": 3, "fourth": [12, 27, 28], "fp": 7, "frac": [0, 1, 2, 3, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 16, 17, 19, 21, 22, 24, 27, 28, 29, 30], "fraction": 9, "frame": 7, "framework": [1, 8, 10, 24], "frank": [5, 11], "frankefunct": [5, 6, 11], "fredli": [25, 27], "free": [0, 6, 11, 13, 15, 16, 18, 20, 21, 22, 24, 25, 26, 27], "freecodecamp": 20, "freedom": [5, 29], "freeli": [0, 22], "freez": 15, "frequenc": [3, 6, 7, 24], "frequent": [0, 8, 9, 13, 29], "frequentist": 20, "fresh": 10, "fridai": [15, 25, 27], "friedman": [6, 19, 22, 26, 27], "friendli": 4, "fro": 22, "frodo": 27, "frog": 3, "from": [0, 1, 2, 3, 4, 6, 7, 8, 9, 11, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 24, 25, 26], "from_cod": 9, "from_logit": [3, 4], "from_tensor_slic": 4, "front": [0, 4, 5, 27, 28, 29], "frustrat": 15, "fulfil": [2, 5, 12, 28, 29], "full": [1, 3, 5, 7, 9, 10, 13, 24, 27, 28, 29, 30], "full_matric": [5, 28, 29], "fulli": [3, 6, 12, 24], "fun": [20, 27], "func": 2, "function": [2, 3, 4, 5, 9, 14, 15, 16, 17, 18, 19, 20, 21], "functionali": 11, "fundament": [0, 6, 20, 27], "funtion": 2, "further": [2, 7, 9, 19, 27], "furthermor": [0, 3, 5, 6, 7, 11, 12, 13, 20, 22, 27, 28, 29, 30], "futur": [0, 4, 8, 9, 27], "fy": [15, 22, 23, 25, 26, 27], "fys4155": 22, "fys5419": [26, 27], "fys5429": [26, 27], "f\u00f8470": [25, 27], "g": [0, 1, 2, 3, 4, 6, 8, 9, 10, 11, 13, 15, 18, 24, 27, 28, 29, 30], "g0": 2, "g_": [2, 9, 10, 30], "g_0": 2, "g_1": [2, 10], "g_2": [2, 10], "g_analyt": 2, "g_dnn_ag": 2, "g_euler": 2, "g_i": 2, "g_m": [3, 10], "g_n": 3, "g_re": 2, "g_t": [2, 30], "g_t_d2t": 2, "g_t_d2x": 2, "g_t_dt": 2, "g_t_hessian": 2, "g_t_hessian_func": 2, "g_t_jacobian": 2, "g_t_jacobian_func": 2, "g_trial": 2, "g_trial_deep": 2, "g_vec": 2, "gain": [1, 5, 7, 9, 10, 13, 28, 29], "galleri": [0, 27], "game": 4, "gamge": 27, "gamma": [0, 2, 8, 9, 10, 11, 13, 27, 29], "gamma1": 8, "gamma2": 8, "gamma_": [0, 27], "gamma_0": 10, "gamma_1": 10, "gamma_1x": 10, "gamma_i": [0, 8, 24, 27], "gamma_j": 13, "gamma_k": [13, 29], "gamma_m": 10, "gamma_x": [0, 27], "gap": 8, "gate": [4, 12], "gather": [0, 1, 12, 28], "gaug": 12, "gaussbacksub": 21, "gaussian": [4, 5, 6, 8, 14, 18, 24, 27], "gaussian_point": 14, "gaussian_rbf": 8, "gave": [13, 30], "gavra": 27, "gbc": 27, "gca": [2, 6, 8, 13], "gd": [1, 29], "gd_clf": 10, "gdclassiffiercgain": 10, "gdclassiffierconfus": 10, "gdclassiffierroc": 10, "gdm": 13, "gdregress": 10, "ge": [1, 5, 7, 24, 28, 29], "gen_loss": 4, "gen_tap": 4, "gender": [0, 27], "genener": 4, "gener": [0, 1, 2, 3, 5, 6, 8, 10, 11, 12, 13, 14, 15, 16, 18, 21, 22, 24, 26, 28, 29, 30], "generaliz": 16, "generallay": 12, "generate_and_save_imag": 4, "generate_imag": 4, "generate_latent_point": 4, "generate_simple_clustering_dataset": 14, "generated_imag": 4, "generator_loss": 4, "generator_loss_list": 4, "generator_model": 4, "generator_optim": 4, "genom": 20, "geodes": 11, "geoff": 30, "geometr": [0, 13, 27], "geometri": 5, "georg": 26, "geotif": 6, "geq": [2, 5, 8, 9, 13, 28, 29, 30], "geron": [0, 26, 27], "get": [0, 1, 2, 3, 4, 5, 6, 7, 9, 10, 11, 13, 15, 19, 20, 21, 22, 24, 25, 27, 28, 29, 30], "get_dummi": 9, "get_paramet": 2, "get_split": 9, "get_yaxi": 8, "get_yticklabel": 6, "gh": 15, "gibb": [20, 27], "gif": 4, "gini": 10, "gini_index": 9, "ginvers": 13, "git": [0, 15, 20, 27], "giter": [13, 30], "github": [0, 20, 22, 23, 25, 26, 27, 28], "gitignor": 15, "gitlab": [0, 15, 20, 22, 27], "give": [0, 1, 2, 3, 5, 6, 7, 8, 9, 10, 12, 13, 14, 18, 19, 20, 22, 24, 27, 28, 29, 30], "given": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 17, 19, 21, 24, 27, 28, 29, 30], "global": [6, 7, 13, 29, 30], "glorot": 1, "gnew": 13, "go": [0, 1, 3, 5, 6, 8, 9, 11, 12, 13, 15, 16, 18, 27, 28, 29], "goal": [0, 7, 9, 27], "goe": [0, 1, 2, 5, 6, 13, 14, 15, 19, 21, 27, 28, 29, 30], "golden": 13, "gone": [5, 28, 29], "gong": 1, "good": [1, 3, 4, 5, 6, 9, 10, 11, 13, 15, 18, 20, 24, 26, 28, 29, 30], "goodfellow": [4, 26, 27, 28, 29], "googl": [1, 4, 20, 27], "got": [1, 6, 22], "gotten": 27, "gov": 6, "govern": 27, "gp": 26, "gpu": [1, 13, 20, 27], "grad": [2, 13, 30], "grad_analyt": 13, "grad_ol": 18, "grad_ridg": 18, "grade": [22, 23], "gradient": [0, 3, 4, 7, 8, 9, 12, 20, 27, 28], "gradient_desc": 30, "gradientboostingclassifi": 10, "gradientboostingregressor": 10, "gradients_of_discrimin": 4, "gradients_of_gener": 4, "gradienttap": 4, "gradual": [1, 14], "grai": [4, 6], "graph": [1, 9, 11, 12, 13, 16, 29], "graph_from_dot_data": 9, "graphic": [0, 1, 9, 15, 27], "grasp": 0, "gray_r": [1, 3], "grayscal": 3, "great": [5, 13, 15, 29, 30], "greater": [1, 7, 24, 28], "greatli": 13, "greedi": 9, "green": [0, 3, 9, 24], "grei": 4, "grid": [1, 3, 6, 7, 8, 12, 24, 28, 30], "grossli": [13, 29], "ground": [0, 27], "group": [0, 6, 7, 9, 14, 15, 20, 22, 23, 25, 27], "groupbi": [0, 27], "grow": [1, 3, 9, 10, 30], "growth": [0, 27], "gru": 4, "guarante": [0, 4, 13, 24, 27, 28, 29, 30], "guess": [1, 4, 10, 13, 14, 29, 30], "guestrin": 10, "gui": 15, "guid": 1, "h": [0, 1, 5, 6, 8, 13, 15, 19, 24, 25, 26, 27, 28, 29, 30], "h1": 2, "h_": [0, 13, 27, 29, 30], "h_0": 30, "h_1": [2, 13, 29], "h_2": [2, 13, 29], "h_m": 10, "h_t": 30, "ha": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 17, 18, 19, 21, 22, 24, 27, 28, 29, 30], "haanen": [25, 27], "habit": [0, 28], "had": [0, 1, 6, 7, 13, 27, 29, 30], "hadamard": [1, 12, 13, 30], "half": [1, 8, 9], "halv": 10, "hand": [0, 1, 2, 3, 5, 11, 12, 13, 20, 21, 22, 24, 25, 26, 27, 28, 29, 30], "handi": [3, 22], "handl": [0, 1, 2, 5, 9, 11, 15, 18, 20, 28, 29, 30], "handle_unknown": 9, "handsid": 12, "handwrit": 12, "handwritten": [1, 5], "happen": [1, 2, 3, 4, 5, 6, 10, 13, 24, 28, 29, 30], "hard": [1, 7, 8, 10, 13, 29, 30], "hardcopi": [20, 27], "harder": [0, 1, 28], "harmon": 3, "hasn": [], "hassl": [0, 20, 27], "hast": [20, 27], "hasti": [0, 6, 16, 17, 19, 22, 26, 27, 28], "hat": [0, 1, 5, 6, 7, 9, 10, 11, 12, 13, 16, 17, 18, 19, 21, 28, 29, 30], "have": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 24, 25, 27, 28, 29, 30], "haven": 1, "he": 7, "head": [4, 10, 24], "header": [0, 27], "heads_proba": 10, "health": [0, 28], "hear": [0, 13, 27, 30], "heart": [0, 7, 27], "heatmap": [0, 1, 3, 7, 17, 27], "heavili": 0, "heavisid": 1, "height": [1, 3, 6, 28], "held": [13, 30], "help": [0, 1, 4, 12, 13, 15, 16, 22, 27, 30], "helper": [4, 14], "henc": [0, 5, 6, 8, 9, 10, 12, 13, 27, 28, 29, 30], "henrik": [25, 27], "her": 7, "here": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 17, 18, 19, 20, 21, 22, 24, 27, 28, 29, 30], "hereaft": [0, 8, 12, 27], "hermitian": 21, "hessenberg": 21, "hessian": [0, 2, 5, 13], "heterogen": [9, 10], "hi": 7, "hidden": [1, 3, 4, 12], "hidden_bia": 1, "hidden_bias_gradi": 1, "hidden_layer_s": [0, 1, 27], "hidden_neuron": 4, "hidden_weight": 1, "hidden_weights_gradi": 1, "hierarch": [5, 28, 29], "high": [0, 1, 2, 3, 4, 5, 6, 9, 10, 11, 13, 14, 20, 21, 22, 27, 28, 29, 30], "higher": [0, 1, 3, 5, 6, 8, 13, 18, 22, 27, 28, 29, 30], "highest": [1, 2], "highli": [0, 3, 4, 10, 19, 20, 21, 26, 27, 28, 29, 30], "highwai": [], "hing": 8, "hint": [13, 15, 16, 28, 29], "hinton": 30, "hip": 20, "hire": 0, "hist": [4, 6, 7, 24], "histogram": [6, 7, 24], "histor": [7, 11], "histori": [3, 4, 12, 15, 30], "hitherto": 5, "hjorth": [25, 27, 28, 29, 30], "hobbi": 24, "hoc": [5, 28, 29], "hoff": 26, "hold": [1, 3, 6, 13, 14, 29, 30], "holder": [0, 27], "home": [], "homepag": [22, 27], "homework": [6, 13, 29, 30], "homogen": [1, 3, 9, 10, 13, 30], "honchar": 2, "hopefulli": [0, 11, 15, 24, 27, 30], "horizont": 11, "horlyk": [25, 27], "hors": [3, 7, 27], "hot": [1, 9], "hour": [1, 20, 23, 24, 25, 27], "how": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 17, 18, 19, 20, 21, 22, 24, 27, 28, 29, 30], "howev": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 20, 21, 22, 24, 27, 28, 29, 30], "hspace": [0, 4, 8, 10, 24, 27], "hstack": 1, "htf": 27, "html": [0, 16, 20, 22, 23, 25, 26, 27, 28, 29, 30], "http": [0, 3, 4, 6, 13, 15, 16, 20, 21, 22, 23, 25, 26, 27, 28, 29, 30], "huang": [0, 27], "huber": [0, 27], "huge": [1, 3, 4, 20], "human": [0, 1, 3, 6, 9, 12, 28], "humid": 9, "hundr": 1, "hungri": 1, "hybrid": 23, "hydrogen": [0, 27], "hyperbol": [1, 4, 12], "hyperparam": 8, "hyperparamet": [3, 4, 5, 6, 9, 13, 18, 22, 28, 29, 30], "hyperplan": 11, "h\u00f8rlyk": [25, 27], "i": [2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 28, 29, 30], "i0": [0, 27], "i1": [0, 6, 8, 12, 27, 28, 30], "i2": [0, 8, 12, 27], "i3": [0, 12, 27], "i5": [0, 27], "i_": [13, 29, 30], "i_1": [5, 6], "i_2": [5, 6], "i_t": 30, "ian": 26, "ic": [1, 22], "id": [7, 13, 29, 30], "ida": [25, 27], "idea": [0, 1, 2, 3, 4, 6, 9, 10, 12, 13, 21, 22, 28, 29, 30], "ideal": [0, 2, 6, 8, 13, 24, 27, 30], "idem": 6, "ident": [5, 6, 12, 13, 17, 18, 21, 28, 29], "identifi": [0, 1, 7, 9, 11, 12, 13, 14, 27, 28], "ieor": 24, "ifi": 26, "ifs": [20, 27], "ignor": [0, 1, 3, 9, 15, 28, 30], "ii": [21, 24], "iii": [21, 27], "ij": [0, 1, 3, 6, 8, 12, 14, 16, 21, 24, 27, 28, 30], "ik": [0, 21, 27, 28], "ill": 30, "illustr": [5, 7, 10, 12, 13, 14, 20, 27], "ilsvrc": 30, "im": 6, "imag": [1, 3, 4, 6, 9, 11, 12, 14, 26, 27], "image_at_epoch_": 4, "image_batch": 4, "image_height": 3, "image_path": [0, 6, 7, 9, 27], "image_width": 3, "imageio": 6, "images_from_seed_imag": 4, "imagin": 1, "immedi": [0, 3, 4, 6, 20, 27], "implement": [0, 2, 3, 4, 5, 6, 8, 9, 10, 11, 12, 13, 14, 22, 24, 27, 28, 29, 30], "impli": [3, 5, 6, 7, 13, 21, 28, 29, 30], "implicit": 3, "implicitli": [11, 24], "import": [0, 1, 2, 3, 4, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 22, 24, 30], "importantli": 3, "impos": [0, 6, 11, 12, 27], "imposs": [0, 5, 27, 28, 29], "impress": [0, 12, 27], "improv": [0, 4, 5, 9, 10, 11, 13, 15, 22, 28, 29], "impur": 9, "imread": 6, "imshow": [1, 3, 4, 6], "in3050": [26, 27], "in3310": 27, "in4080": [26, 27], "in4300": [26, 27], "in4310": 26, "in5400": 3, "in5550": 26, "in_out_neuron": 4, "inaccur": [13, 29], "inact": 12, "inadequ": [0, 27], "inappropri": 30, "inch": [6, 28], "includ": [0, 1, 2, 3, 4, 5, 6, 7, 11, 12, 15, 16, 17, 18, 20, 24, 25, 26, 27, 28, 29], "include_bia": [6, 9], "incom": [12, 16], "incorrect": 1, "incoveni": 8, "increas": [0, 1, 3, 4, 5, 6, 9, 12, 13, 19, 22, 24, 27, 28, 30], "increasingli": 24, "increment": 30, "ind": 6, "inde": [0, 2, 4, 5, 6, 13, 27, 28, 29], "indefinit": 4, "independ": [0, 5, 6, 7, 8, 12, 13, 24, 27, 28, 29], "index": [0, 1, 3, 4, 10, 14, 20, 21, 22, 24, 26, 27], "index_col": [0, 27], "indic": [0, 1, 3, 4, 5, 6, 9, 10, 11, 13, 16, 22, 27, 28], "indispens": 6, "individu": [1, 6, 7, 10, 12, 24, 27, 28, 30], "indu": [], "indx": 21, "indx1": 2, "indx2": 2, "indx3": 2, "ineffici": [3, 13], "inequ": [8, 13], "inequaltii": 29, "inertia": 13, "inf1000": [20, 27], "inf1100": [20, 27], "inf1100l": [20, 27], "inf1110": [20, 27], "inf3000": 27, "infeas": 9, "infer": [0, 1, 4, 6, 26, 27], "inferenc": 1, "infil": [0, 6, 7, 9, 27], "infin": [5, 6, 7, 11, 19, 28, 29], "infinit": [3, 30], "infinitesim": 24, "influenc": [6, 10, 18], "influenti": 1, "info": 27, "inform": [0, 1, 3, 4, 6, 9, 11, 12, 13, 14, 21, 22, 26, 27, 29, 30], "inforom": 15, "infrequ": 30, "infti": [3, 6, 13, 24, 29], "ingeni": [13, 29, 30], "ingredi": [0, 9, 27], "inher": 6, "inherit": [21, 27, 30], "initi": [0, 1, 2, 6, 10, 13, 14, 18, 21, 24, 27, 29, 30], "inject": 14, "inlin": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 21, 24, 27, 28, 29, 30], "inner": [0, 13, 28], "inp": 4, "inplac": 13, "input": [0, 1, 3, 4, 5, 6, 7, 8, 12, 13, 14, 16, 22, 24, 27, 28, 29, 30], "input_dim": 1, "input_shap": [3, 4], "inputs": 1, "inputs_shuffl": [0, 1, 28], "insert": [3, 5, 6, 8, 10, 24, 28, 29], "insid": [4, 7], "insight": [0, 1, 5, 20, 27, 28, 29], "insist": [6, 13, 28, 30], "inspir": [0, 1, 12, 22, 27], "instabl": 2, "instal": [0, 1, 5, 6, 9, 15], "instanc": [0, 1, 2, 4, 6, 9, 11, 13, 16, 27, 28, 29], "instanti": 10, "instead": [0, 1, 2, 3, 4, 5, 6, 8, 9, 11, 13, 14, 17, 21, 24, 27, 28, 30], "institut": 1, "instruct": [0, 1, 15], "int": [0, 1, 2, 3, 4, 5, 6, 11, 13, 14, 21, 24, 28, 30], "int32": 10, "int_": [3, 6, 24], "int_0": 24, "int_a": 24, "intak": [0, 28], "integ": [1, 2, 13, 14, 21, 24, 27], "integer_vector": 1, "integr": [3, 6, 24, 27], "intellig": [0, 14, 26, 27], "intend": 10, "intens": [1, 18], "intention": 14, "interact": [0, 6, 9, 12, 20, 22, 27], "intercept": [0, 6, 8, 11, 13, 16, 17, 18, 27, 28, 29, 30], "intercept_": [0, 6, 8, 9, 13, 27, 28, 30], "interchang": [5, 12, 21], "interconnect": 1, "interesit": 19, "interest": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 12, 20, 22, 24, 27, 28, 29], "interfac": [0, 1, 15, 21, 28], "interior": [0, 9, 27], "intermedi": [21, 28], "intern": [1, 10, 12], "interpol": [1, 3, 4, 6, 12], "interpr": [5, 28, 29], "interpret": [0, 1, 6, 9, 10, 12, 13, 15, 16, 19, 21, 22, 24], "interv": [0, 3, 5, 6, 7, 13, 19, 24, 27, 28, 29], "intial": [13, 29], "intract": [0, 4, 28], "intrins": [3, 11, 21, 24, 27], "intro": [20, 26, 27], "introduc": [0, 1, 5, 6, 8, 10, 12, 21, 22, 24, 27, 29, 30], "introduct": [1, 2, 4, 13, 26, 28, 29, 30], "introductori": [0, 4, 21, 26, 27, 28], "intuit": [0, 5, 6, 8, 12, 13, 22, 27, 30], "inv": [0, 5, 13, 17, 27, 28, 29, 30], "invalu": [0, 13, 20, 27, 29], "invari": 1, "invd": 5, "inver": 8, "invers": [0, 3, 6, 13, 27, 28, 29, 30], "inverse_transform": 8, "invert": [0, 5, 7, 10, 13, 16, 18, 27, 30], "invh": [13, 30], "invok": 8, "involv": [0, 2, 6, 7, 11, 12, 27, 28, 30], "io": [0, 20, 22, 23, 25, 26, 27, 28], "ip": [0, 8, 24, 27], "ipca": 11, "ipynb": [20, 27], "ipython": [0, 5, 7, 9, 11, 14, 20, 22, 27, 28], "iq": 6, "iri": [8, 9], "irreduc": 6, "irrelev": [5, 28, 29], "irrespect": [0, 27], "irvin": 22, "iseffici": [], "isn": 5, "isnul": [], "isomap": 11, "issu": [1, 9, 15, 21, 30], "it_arrai": 13, "item": [0, 13, 27], "items": [21, 27], "iter": [1, 2, 4, 6, 8, 13, 14, 18, 22, 24, 29, 30], "its": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 20, 21, 22, 24, 27, 29, 30], "itself": [5, 6, 12, 19, 22, 24, 27, 28], "j": [0, 1, 2, 3, 4, 5, 6, 8, 9, 11, 12, 13, 14, 15, 16, 21, 22, 24, 26, 27, 28, 29, 30], "j1": 21, "j_": 6, "j_lasso_sk": 6, "j_ridge_sk": 6, "j_sk": 6, "jackknif": [6, 20, 27], "jacobian": [2, 13, 29], "jason": 4, "jax": [20, 27, 30], "jensen": [25, 27, 28, 29, 30], "jerom": [19, 22, 26], "ji": [12, 21], "jit": 13, "jj": [0, 5, 6, 27], "jk": [0, 1, 6, 12, 21, 27], "jl": [0, 27], "jm": 21, "jnp": 13, "job": [2, 8, 10, 15], "join": [0, 4, 6, 7, 9, 27], "joint": [4, 5], "judg": [13, 29], "judgement": 6, "julia": [20, 21, 22], "jump": 24, "junk": 4, "jupit": 27, "jupyt": [0, 15, 16, 20, 22, 26, 27], "just": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 20, 24, 27, 28, 29, 30], "justif": 0, "justifi": [3, 10], "k": [0, 1, 3, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 20, 21, 22, 24, 25, 27, 28, 29, 30], "k0": 7, "k1": 7, "kaggl": [6, 22], "kappa_d": 24, "karl": [25, 27], "karush": 8, "katrin": [25, 27], "keep": [0, 1, 4, 5, 6, 11, 13, 14, 15, 18, 21, 22, 27, 28, 29, 30], "keepdim": [1, 6, 10, 21], "kei": [1, 3, 6, 12], "kept": [4, 6, 14], "kera": [0, 4, 20, 22, 27], "kernel": [0, 1, 3, 20, 27, 28], "kernel_regular": [1, 3], "kernel_s": 4, "kernelpca": 11, "kev": [0, 27], "kevin": [26, 27], "keyword": [18, 21, 27], "kfold": 6, "kg": 1, "ki": 21, "kick": [1, 13, 30], "kiener": 2, "kilomet": [6, 28], "kind": [0, 2, 3, 4, 8, 12, 13, 14, 27, 28], "kingma": 30, "kj": [6, 12, 21, 28, 30], "kjm": [20, 27], "kkt": 8, "kl": 24, "km": [12, 27], "kmean": 14, "kmeanspoint": 14, "kn_k": 14, "know": [0, 1, 2, 5, 6, 8, 13, 15, 16, 17, 19, 20, 27, 28, 29], "knowledg": [0, 20, 27], "known": [1, 3, 4, 5, 6, 7, 8, 9, 12, 18, 21, 22, 24, 26, 28, 30], "kondev": [0, 27], "kp": 24, "kpca": 11, "kroneck": 14, "kuhn": 8, "kvalsund": [25, 27], "kwown": [0, 27], "l": [0, 1, 2, 3, 5, 6, 7, 8, 10, 11, 12, 13, 21, 22, 24, 27, 29], "l0": 7, "l1": [0, 1, 3, 7, 27], "l1_l2": [1, 3], "l1regl": 5, "l2": [1, 3], "l_": 21, "l_1": 7, "l_2": [7, 13, 29, 30], "l_j": 12, "la": 13, "la_i": 12, "la_k": 12, "lab": [20, 22, 27], "label": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 12, 13, 14, 15, 20, 21, 22, 24, 27, 28, 29, 30], "labelencod": [7, 10], "labels": [6, 8, 9], "labels_shuffl": [0, 1, 28], "laboratori": 23, "lack": [0, 27], "lagari": 2, "lagrang": [8, 11], "lam": 18, "lambda": [0, 1, 2, 3, 5, 6, 7, 8, 10, 12, 13, 17, 18, 19, 22, 24, 27, 28, 29, 30], "lambda_": 11, "lambda_0": 11, "lambda_1": [5, 8, 11, 28, 29], "lambda_2": [8, 11], "lambda_i": [8, 11], "lambda_iy_i": 8, "lambda_jy_iy_j": 8, "lambda_k": 8, "lambda_n": [5, 8, 28, 29], "lamda": 1, "land": 8, "landmark": 8, "landscap": [13, 18, 29, 30], "langl": [0, 6, 11, 24, 27, 28], "languag": [0, 1, 4, 8, 20, 21, 22, 26, 27], "lapack": [21, 27], "laplac": 5, "laptop": [15, 20], "larg": [0, 1, 2, 4, 5, 6, 8, 9, 10, 11, 13, 18, 20, 21, 22, 24, 26, 27, 28, 29, 30], "larger": [0, 3, 5, 6, 8, 10, 11, 13, 17, 24, 27, 28, 29, 30], "largest": [4, 8, 11], "lasso": [0, 7, 20, 27, 30], "lasso_sk": 6, "last": [0, 1, 3, 4, 5, 6, 7, 8, 12, 16, 17, 19, 21, 22, 24, 25, 27, 29], "latent": 4, "latent_dim": 4, "latent_point": 4, "latent_space_value_rang": 4, "later": [0, 1, 4, 7, 8, 12, 13, 14, 15, 20, 22, 27, 30], "latest": [4, 15, 20], "latest_checkpoint": 4, "latex": 27, "latter": [0, 3, 6, 7, 8, 11, 13, 21, 24, 27, 28, 29, 30], "lattic": 12, "law": 0, "layer": [0, 4, 13, 27, 30], "lbfg": [7, 9, 10], "lcc": [5, 6], "lda": 11, "ldot": [0, 6, 11, 22, 27], "le": [5, 7, 10, 13, 17, 24, 28, 29], "lead": [0, 1, 3, 5, 6, 7, 8, 9, 10, 11, 12, 13, 16, 17, 21, 24, 27, 28, 29, 30], "leaf": 9, "leaki": 1, "leakyrelu": 4, "lear": [13, 29], "learn": [3, 4, 5, 6, 7, 8, 9, 10, 12, 21, 25, 26], "learnabl": 3, "learner": 10, "learnig": 27, "learning_r": [8, 10], "learning_rate_init": [0, 1, 27], "learning_schedul": [13, 30], "learnt": 22, "least": [0, 7, 8, 10, 11, 17, 18, 20, 21, 24], "leat": [13, 30], "leav": [0, 1, 3, 5, 6, 9, 11, 27, 29], "lectur": [0, 1, 5, 10, 11, 12, 13, 20, 21, 22, 23, 25, 26, 28], "lecturenot": [0, 20, 22, 26, 27], "left": [0, 1, 2, 3, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 16, 17, 19, 21, 22, 24, 27, 28, 29, 30], "leftarrow": [8, 12], "legend": [0, 2, 3, 4, 5, 6, 7, 8, 9, 10, 13, 15, 27, 28, 29, 30], "leinonen": 27, "len": [0, 1, 2, 3, 4, 5, 6, 8, 9, 10, 11, 12, 16, 17, 21, 27, 28, 29, 30], "length": [0, 1, 3, 4, 8, 9, 13, 16, 20, 27, 28, 29, 30], "length_of_sequ": 4, "leq": [0, 5, 7, 8, 13, 14, 24, 27, 28, 29, 30], "less": [0, 1, 3, 4, 5, 6, 8, 9, 13, 20, 24, 27, 28, 29, 30], "lessen": 1, "let": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 19, 21, 24, 27, 28, 29, 30], "letter": [0, 16, 21, 24, 27, 28], "level": [0, 1, 5, 6, 9, 20, 21, 22, 23, 25, 27], "leverag": 30, "li": [8, 11], "lib": [], "liberti": 30, "liblinear": 10, "librari": [0, 1, 2, 3, 4, 5, 6, 9, 10, 11, 21, 22, 24, 26, 28, 29, 30], "licens": [0, 1, 20, 22, 27], "lie": [0, 6, 11, 24, 27, 28], "life": [0, 1, 8, 12, 27], "lifetim": 13, "like": [0, 1, 2, 3, 4, 5, 6, 7, 9, 10, 11, 12, 13, 15, 16, 20, 21, 22, 24, 27, 28, 29, 30], "likelihood": [0, 1, 5, 9, 27, 28], "lim_": 24, "limit": [0, 5, 6, 8, 12, 21, 22, 27, 28], "lin_clf": 8, "lin_model": [], "lin_reg": 9, "linalg": [0, 2, 5, 6, 8, 11, 13, 17, 21, 24, 27, 28, 29, 30], "line": [0, 3, 6, 8, 11, 13, 15, 16, 27, 29, 30], "line1": 8, "line2": 8, "line3": 8, "line_model": 15, "line_ms": 15, "line_predict": 15, "linear": [1, 3, 5, 6, 7, 9, 10, 11, 12, 16, 17, 18, 19, 20, 22, 24, 30], "linear_model": [0, 5, 6, 7, 8, 9, 10, 11, 13, 15, 16, 19, 27, 28, 29, 30], "linear_regress": 6, "linearli": [5, 28, 29], "linearloc": [6, 13, 29, 30], "linearregress": [0, 6, 7, 9, 15, 16, 19, 27, 28, 30], "linearsvc": 8, "lineat": 29, "liner": [1, 3], "linerar": 10, "linewidth": [0, 2, 4, 6, 8, 9, 10], "link": [0, 4, 9, 12, 15, 20, 22, 23, 25, 27], "linlag": 5, "linpack": [21, 27], "linreg": [0, 27], "linspac": [0, 2, 3, 4, 6, 8, 9, 10, 13, 16, 17, 19, 21, 24, 27, 28, 30], "linu": 4, "linux": [0, 1, 20, 22, 27], "liquid": [0, 27], "list": [1, 2, 3, 4, 9, 15, 20, 22, 27, 30], "listedcolormap": [9, 10], "literatur": [1, 7, 14, 26], "littl": [1, 3, 9, 12, 30], "live": [8, 16], "ll": [0, 18, 24, 27, 28], "lle": [0, 28], "lloyd": [4, 14], "lmb": [0, 2, 5, 6, 28, 29, 30], "lmbd": [0, 1, 3, 27], "lmbd_val": [0, 1, 3, 27], "lmbda": [13, 29, 30], "ln": [1, 13, 29], "load": [1, 4, 6, 7, 9, 10], "load_boston": [], "load_breast_canc": [1, 7, 9, 10, 11], "load_data": [3, 4], "load_digit": [1, 3], "load_iri": [8, 9], "loc": [3, 6, 7, 8, 9, 10, 27], "local": [0, 1, 3, 7, 12, 13, 15, 28, 29, 30], "locat": [2, 3, 8, 15], "log": [0, 1, 2, 4, 5, 6, 7, 9, 10, 11, 13, 15, 21, 22, 27], "log10": [0, 5, 6, 28, 29, 30], "log_": [0, 27], "log_clf": 10, "logarithm": [0, 5, 7, 17, 21, 27], "logbook": 22, "logic": [0, 1, 9, 27], "login": 15, "logist": [0, 1, 2, 8, 9, 10, 11, 12, 13, 20, 28, 29, 30], "logisticregress": [7, 9, 10, 11], "logit": 7, "logreg": [7, 9, 10, 11], "logspac": [0, 1, 3, 5, 6, 27, 28, 29, 30], "long": [0, 1, 3, 4, 12, 13, 27, 29, 30], "longer": [2, 3, 8, 10, 14, 21, 24, 27, 30], "loocv": 6, "look": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 13, 15, 16, 19, 21, 22, 24, 27, 28, 29, 30], "loop": [1, 4, 6, 10, 12, 14, 16, 17, 18, 20, 21, 27], "lose": 1, "loss": [0, 1, 3, 4, 5, 6, 7, 8, 10, 11, 13, 18, 21, 22, 27], "loss_fil": 4, "lossfil": 4, "lost": 4, "lot": [1, 4, 6, 16, 30], "low": [0, 6, 9, 10, 11, 22, 27, 28], "lower": [0, 1, 3, 6, 9, 10, 16, 21, 28], "lowercas": [21, 27], "lowest": [9, 13, 24, 30], "lr": [1, 3, 4, 10], "lstat": [], "lstm": 4, "lstm_2layer": 4, "lstsq": [0, 27, 28], "lt": 6, "lu": [0, 5, 27, 28, 29], "lubksb": 21, "luckili": 2, "ludcmp": 21, "lux": 21, "lvert": 1, "lw": [0, 27], "m": [0, 1, 2, 3, 5, 6, 8, 9, 10, 11, 12, 13, 15, 21, 24, 25, 26, 27, 28, 29, 30], "m_": [9, 12], "m_0": 30, "m_1": 14, "m_h": [0, 27], "m_k": 14, "m_l": 12, "m_n": [0, 27], "m_p": [0, 27], "m_t": [13, 30], "ma": 11, "machin": [1, 3, 4, 5, 6, 7, 9, 10, 11, 12, 15, 16, 21, 26, 28], "machinelearn": [0, 6, 16, 20, 22, 23, 25, 26, 27, 28, 29], "mackai": 26, "made": [0, 1, 3, 4, 5, 6, 7, 9, 11, 12, 22, 27, 28], "mae": [0, 27], "magic": 4, "magnitud": [1, 6, 7, 13, 28, 30], "mai": [0, 1, 2, 3, 5, 6, 7, 8, 9, 11, 12, 13, 19, 20, 21, 22, 24, 27, 28, 29, 30], "mail": [23, 25], "main": [0, 1, 3, 4, 5, 6, 7, 9, 21, 22, 26, 28, 29, 30], "mainli": [0, 5, 6, 7, 9, 27, 28], "maintain": [6, 30], "major": [1, 6, 9, 10, 13, 21, 27, 29, 30], "make": [1, 2, 3, 4, 5, 6, 7, 8, 11, 12, 13, 15, 16, 18, 20, 21, 22, 24, 26, 27, 29, 30], "make_axes_locat": 6, "make_moon": [8, 9, 10], "make_pipelin": [0, 6, 10, 28], "makedir": [0, 6, 7, 9, 27], "malcondit": 21, "malign": [1, 7, 9], "mammographi": 5, "manag": [0, 2, 3, 15, 20, 22, 27], "mandatori": [25, 27], "mani": [0, 1, 3, 4, 5, 6, 7, 8, 9, 11, 13, 14, 15, 16, 17, 18, 20, 21, 22, 24, 26, 27, 28, 29, 30], "manifold": 11, "manner": 3, "manual": [6, 28, 30], "map": [0, 1, 2, 6, 7, 8, 11, 12, 14, 24, 27], "marc": 28, "margin": [0, 5, 8], "marit": [0, 27], "mark": 27, "marker": [7, 21, 27], "markov": [20, 27], "marsaglia": 24, "mass": [0, 1, 5, 13, 28, 29], "massag": [0, 27], "masses2016": [0, 27], "masses2016ol": [0, 27], "masses2016tre": 0, "masseval2016": [0, 27], "master": [23, 25], "mat": [20, 27], "mat1100": [20, 27], "mat1110": [20, 27], "mat1120": [20, 27], "match": [1, 4, 5, 13, 14, 15, 28, 29, 30], "materi": [4, 5, 7, 13, 15, 21, 23, 25], "math": [3, 7, 12, 13, 21, 24, 26, 27, 30], "mathbb": [0, 4, 5, 6, 7, 8, 11, 12, 13, 14, 17, 19, 21, 22, 24, 27, 28, 29, 30], "mathbf": [0, 5, 6, 7, 8, 13, 19, 21, 22, 27, 28, 29, 30], "mathcal": [1, 5, 6, 7, 13, 22], "matheemat": 3, "mathemat": [0, 6, 11, 12, 13, 20, 21, 24, 26, 27, 30], "mathemati": 27, "mathrm": [0, 1, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 17, 18, 19, 22, 24, 27, 28, 29, 30], "matmul": [1, 2, 5], "matnat": 26, "matplotlib": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 17, 19, 20, 21, 22, 24, 27, 28, 29, 30], "matric": [0, 1, 3, 4, 6, 7, 8, 11, 13, 16, 17, 20, 28, 29], "matrix": [0, 2, 3, 4, 6, 7, 8, 10, 13, 17, 18, 19, 22, 24], "matshow": 1, "matter": [2, 3, 13, 28, 29, 30], "max": [0, 1, 2, 3, 4, 9, 10, 12, 13, 25, 27, 29, 30], "max_depth": [0, 9, 10], "max_diff": 2, "max_diff1": 2, "max_diff2": 2, "max_it": [0, 1, 8, 13, 27], "max_iter": 14, "max_leaf_nod": 10, "max_sampl": 10, "maxdegre": [0, 6, 10, 28], "maxdepth": 10, "maxim": [1, 4, 5, 7, 8, 11], "maximum": [0, 2, 3, 5, 7, 8, 9, 10, 13, 14, 27, 28, 29, 30], "maxpolydegre": [5, 6, 28, 29, 30], "maxpooling2d": 3, "mbox": [5, 6, 28, 29], "mcculloch": 12, "md": 11, "mdoel": 4, "mean": [1, 2, 3, 4, 5, 6, 7, 9, 10, 11, 12, 13, 14, 15, 17, 18, 19, 20, 21, 22, 24, 27, 30], "mean_absolute_error": [0, 27], "mean_divisor": 14, "mean_i": 24, "mean_matrix": 14, "mean_squared_error": [0, 4, 6, 7, 10, 15, 19, 27, 28], "mean_squared_log_error": [0, 27], "mean_vector": 14, "mean_x": 24, "meaning": [0, 4, 7, 27], "meansquarederror": [0, 27], "meant": [3, 7, 10, 13], "meanwhil": 30, "measur": [0, 1, 2, 5, 6, 9, 11, 12, 14, 16, 18, 19, 22, 24, 27, 28, 30], "mechan": [0, 4, 24, 27, 30], "median": [0, 27, 28, 30], "medicin": 12, "medium": [4, 8, 13], "medv": [], "meet": [0, 25], "mehta": [0, 27, 28, 29], "memori": [3, 4, 11, 12, 13, 18, 21, 30], "mention": [0, 12, 13, 22, 24, 27, 29, 30], "mere": [0, 22], "meshgrid": [2, 5, 6, 8, 9, 10, 11], "mess": 15, "messag": [5, 13], "messi": 2, "met": [0, 3, 8, 28], "meteorolog": 9, "meter": [6, 28], "method": [0, 1, 2, 3, 4, 5, 7, 8, 11, 12, 14, 15, 16, 17, 18, 19, 20, 21, 24, 26, 28], "metion": 6, "metric": [0, 1, 3, 6, 7, 9, 10, 14, 15, 19, 27, 28], "metropoli": [20, 27], "mev": [0, 24, 27], "mgd": [13, 30], "mglearn": [20, 27], "mgrid": 13, "mhjensen": [], "mi": 10, "mia": [25, 27], "microsoft": 26, "mid": 1, "midel": 4, "midnight": 15, "midpoint": 9, "might": [0, 1, 2, 4, 6, 9, 13, 15, 17, 18, 28, 29, 30], "migth": 17, "mild": 9, "millimet": [6, 28], "million": [0, 27, 28, 30], "mimic": 12, "min": [0, 2, 5, 8, 9, 29], "min_": [0, 2, 5, 14, 17, 27, 28, 29], "min_samples_leaf": 9, "mind": [0, 6, 13, 15, 18, 27, 28, 29, 30], "mindboard": 4, "mine": [20, 27], "mini": [1, 11, 12, 13, 29], "minibatch": [1, 11, 13], "minibathc": [13, 30], "miniforge3": [], "minim": [0, 1, 2, 3, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 16, 17, 28, 29, 30], "minima": [0, 1, 7, 13, 27, 29, 30], "minimum": [0, 1, 2, 6, 8, 9, 11, 13, 28, 29, 30], "minmaxscal": [0, 28, 30], "minor": 24, "minst": 1, "minu": 7, "mirjalili": 27, "mirror": 9, "misc": 6, "misclassif": [8, 9, 10], "misclassifi": [8, 10], "miser": 0, "mismatch": 1, "miss": [7, 10], "mistak": 4, "mit": 26, "mix": [1, 2, 27], "mixtur": [13, 30], "mk": [9, 21], "mkdir": [0, 6, 7, 9, 27], "ml": [0, 1, 10, 13, 21, 22, 28, 29, 30], "mlab": 24, "mle": [5, 7], "mlp": 1, "mlpclassifi": 1, "mlpregressor": [0, 27], "mm": 21, "mml": 28, "mn": [12, 24], "mnist": [1, 11], "mod": 24, "mode": [23, 25, 27], "model": [2, 3, 5, 7, 8, 9, 10, 11, 13, 14, 16, 18, 19, 20, 22, 24, 26, 28, 29, 30], "model_select": [0, 1, 3, 5, 6, 7, 9, 10, 11, 15, 16, 17, 19, 27, 28, 29, 30], "moder": [10, 30], "modern": [0, 6, 7, 20, 27], "modif": [2, 12, 13], "modifi": [0, 1, 3, 5, 7, 8, 10, 12, 13, 27, 28, 29, 30], "modul": [0, 16, 21, 27], "modular": 24, "modulo": 24, "moe": [11, 28], "moment": [5, 6, 13, 24], "mondai": [25, 27], "monitor": [13, 30], "monoton": [5, 12, 24], "mont": [0, 6, 20, 24, 26, 27], "montli": 16, "moor": [5, 6], "more": [0, 1, 2, 4, 5, 7, 8, 9, 10, 11, 12, 13, 14, 16, 17, 19, 20, 24], "moreov": [0, 3], "morten": [25, 27, 28, 29, 30], "mortenhj": 27, "most": [0, 1, 3, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 20, 22, 24, 27, 28, 29, 30], "mostli": [1, 11, 18], "motion": [0, 13], "motiv": [1, 4], "move": [0, 4, 5, 6, 7, 9, 12, 13, 14, 15, 16, 22, 24, 28, 29], "mpl": [7, 27], "mpl_toolkit": [2, 6, 13, 29, 30], "mplot3d": [2, 6, 13, 29, 30], "mplregressor": 1, "mse": [0, 4, 5, 6, 9, 10, 15, 16, 17, 19, 22, 27, 28, 29, 30], "mse_simpletre": 10, "mselassopredict": [5, 29], "mselassotrain": [5, 29], "mseownridgepredict": [6, 28, 29, 30], "msepredict": [5, 29], "mseridgepredict": [0, 5, 6, 28, 29, 30], "msetrain": [5, 29], "msle": [0, 27], "mt": [7, 12], "mu": [0, 6, 11, 13, 24, 27], "mu0": 24, "mu1": 24, "mu2": 24, "mu_": [6, 24, 28, 30], "mu_i": [6, 28, 30], "mu_n": 11, "mu_x": 24, "much": [0, 1, 2, 3, 4, 5, 6, 8, 9, 10, 11, 12, 13, 15, 21, 22, 24, 27, 28, 29, 30], "multi": [0, 1, 3, 7, 20, 27], "multiclass": [1, 7], "multidimension": [11, 12, 27], "multilay": 1, "multinomi": 7, "multipl": [2, 4, 5, 6, 7, 12, 13, 15, 24, 28, 29], "multipli": [3, 5, 6, 11, 13, 18, 21, 24, 28, 29, 30], "multiplum": 8, "multivari": [0, 2, 10, 11, 20, 24, 27], "multivariate_norm": [11, 14], "multpli": 16, "murphi": [11, 26, 27], "must": [1, 2, 5, 6, 8, 10, 12, 13, 14, 15, 24, 28, 29, 30], "mutat": 7, "mutual": [1, 3, 6, 13], "mx_": 24, "my": 27, "myenv": [], "myriad": [0, 20, 27], "mz1": 24, "mz2": 24, "m\u00f8svatn": 6, "n": [0, 1, 2, 3, 4, 5, 6, 7, 8, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 21, 22, 24, 27, 28, 29, 30], "n1": 21, "n2": 21, "n_": [1, 2, 3, 8, 12, 24], "n_0": [12, 24], "n_boostrap": [6, 10], "n_bootstrap": 6, "n_categori": [1, 3], "n_cluster": 14, "n_compon": 11, "n_epoch": [13, 30], "n_estim": 10, "n_examples_to_gener": 4, "n_featur": [1, 18], "n_filter": 3, "n_hidden": 2, "n_hidden_neuron": [0, 1, 27], "n_i": 24, "n_input": [0, 1, 3, 28], "n_instanc": 9, "n_iter": 30, "n_job": 10, "n_k": 14, "n_l": [12, 24], "n_layer": 1, "n_m": 9, "n_neuron": 1, "n_neurons_connect": 3, "n_neurons_layer1": 1, "n_neurons_layer2": 1, "n_point": 14, "n_sampl": [6, 8, 9, 10, 14, 18], "n_split": 6, "n_step": 4, "n_t": 2, "n_x": 2, "nabla": [1, 13, 29, 30], "nabla_": [2, 13, 29, 30], "nabla_w": 13, "nag": 13, "naimi": [0, 27], "naiv": 7, "naive_kmean": 14, "name": [0, 1, 3, 4, 5, 6, 7, 8, 9, 10, 12, 13, 14, 15, 18, 20, 21, 22, 24, 25, 27, 28, 29], "narrow": [13, 30], "nation": [1, 5], "nativ": [20, 27], "natur": [0, 1, 4, 8, 9, 12, 13, 22, 24, 26, 27, 29, 30], "navier": 12, "navig": 15, "nb": 24, "nb_": 21, "nbconvert": 27, "nd": 14, "ndarrai": 6, "ne": [9, 10, 21, 24, 28, 29], "nearest": [1, 3, 6, 11], "nearli": [13, 29], "neat": 27, "neccesari": 6, "necess": 2, "necessari": [0, 1, 3, 4, 8, 14, 18, 27], "necessarili": [0, 4, 11, 24, 27], "necesserali": 5, "neck": 7, "need": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 18, 19, 21, 24, 28, 29, 30], "neg": [0, 1, 3, 5, 6, 7, 10, 13, 21, 24, 27, 29], "neg_mean_squared_error": 6, "neglect": [24, 30], "neglig": 24, "neighbor": [3, 6, 11], "neither": [4, 13], "neq": [13, 14, 24, 29], "nervou": 12, "nest": [9, 12], "nesterov": 13, "net": [2, 4, 12], "netlib": [21, 27], "network": [0, 9, 13, 20, 26, 28, 30], "neural": [0, 13, 20, 26, 28, 30], "neural_network": [0, 1, 2, 27], "neuralnetwork": 1, "neuron": [1, 2, 3, 4, 12], "neutral": [0, 27], "neutron": [0, 27], "never": [1, 4, 6, 9, 24], "new": [0, 1, 2, 3, 5, 6, 7, 8, 9, 10, 11, 13, 14, 15, 17, 21, 27, 28, 29, 30], "new_chang": [13, 30], "new_hobbit": 27, "newaxi": [0, 3, 6, 9], "newli": [0, 27], "newton": [1, 7, 8, 13, 24], "next": [0, 1, 2, 3, 4, 5, 6, 8, 9, 13, 14, 15, 16, 27, 28, 29], "next_guess": 13, "next_input": 4, "ng": 1, "ni": 14, "nice": [0, 1, 5, 11, 27, 28, 29], "nicer": [18, 30], "niter": [13, 29, 30], "nitric": [], "nlambda": [0, 5, 6, 28, 29, 30], "nlp": 26, "nm": 24, "nm_n": [0, 27], "nmse": 6, "nn": [2, 5, 6, 12, 21, 27], "nn_model": 1, "nnmin": 2, "node": [1, 3, 9, 10, 12], "nois": [0, 4, 5, 6, 8, 9, 10, 13, 18, 19, 22, 27, 28, 29, 30], "noise_dimens": 4, "noisi": [1, 6, 22, 30], "non": [0, 1, 3, 5, 6, 7, 9, 10, 11, 12, 13, 14, 18, 21, 24, 27, 28, 29, 30], "nondifferenti": 30, "none": [0, 1, 2, 4, 5, 9, 10, 13, 24, 27, 28], "nonlinear": [3, 6, 8, 9, 11, 12], "nonneg": [6, 9, 13, 29], "nonparametr": 6, "nonsens": 24, "nonsingular": 21, "nonumb": [3, 7, 8, 13, 21], "nor": [1, 4, 13], "norm": [0, 1, 5, 6, 8, 11, 13, 18, 27, 28, 29, 30], "normal": [3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 16, 17, 18, 19, 20, 21, 22, 24, 27, 28, 29, 30], "normali": [21, 27], "norwai": [6, 22, 27, 29, 30], "notat": [0, 2, 5, 6, 13, 14, 24, 27, 28, 29], "note": [0, 1, 2, 3, 4, 5, 6, 7, 8, 11, 12, 13, 14, 15, 16, 18, 19, 20, 21, 24, 26, 27, 30], "notebook": [0, 1, 3, 9, 15, 16, 20, 22, 27], "noth": [1, 2, 5, 8, 12, 14, 24, 28, 29], "notic": [4, 5, 12, 13, 21, 24, 27], "notion": 3, "novel": [3, 6, 10, 27], "novemb": [1, 25, 27], "now": [0, 2, 4, 5, 6, 7, 8, 10, 11, 12, 14, 15, 16, 19, 20, 21, 22, 24, 27, 28], "nowadai": [0, 1, 3, 9, 20, 27], "nox": [], "np": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 13, 14, 15, 16, 17, 18, 19, 21, 24, 27, 28, 29, 30], "npr": 2, "nsampl": 6, "nt": 2, "nu": 24, "nuclear": [5, 28, 29], "nuclei": [0, 24, 27], "nucleon": [0, 27], "nucleu": [0, 27], "num": 4, "num_coordin": 2, "num_hidden_neuron": 2, "num_it": [2, 18], "num_neuron": 2, "num_neurons_hidden": 2, "num_point": 2, "num_tre": 10, "num_valu": 2, "number": [1, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 16, 17, 18, 19, 21, 22, 23, 25, 27, 29], "numberid": 7, "numberparamet": 3, "numer": [0, 5, 6, 9, 10, 11, 12, 13, 20, 21, 26, 27, 28, 29, 30], "numpi": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 22, 24, 28, 29, 30], "nunmpi": [5, 28], "nve_frngahw": 29, "nx": 2, "ny": 24, "o": [0, 1, 4, 5, 6, 7, 8, 9, 11, 21, 25, 26, 27, 28, 29], "obei": [6, 11, 13, 28, 30], "object": [0, 1, 4, 8, 10, 15, 19, 21, 27, 30], "obliqu": [5, 28, 29], "observ": [0, 1, 3, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 24, 27, 29, 30], "obtain": [0, 1, 5, 6, 7, 8, 9, 10, 12, 13, 14, 17, 21, 22, 24, 27, 28, 29, 30], "obviou": [5, 6, 11, 24, 28, 29], "obviouli": 27, "obvious": [0, 4, 5, 6, 21, 27], "oc": [28, 29], "occupi": [], "occur": [0, 6, 8, 9, 21, 24, 27], "octob": [25, 27], "od": 0, "odd": [0, 3, 7, 27, 28, 30], "odenum": 2, "odesi": 2, "oen": 0, "off": [1, 3, 4, 5, 9, 13, 24], "offer": [6, 11, 20, 21, 23, 25, 27], "offic": [25, 27], "offici": [23, 27], "often": [0, 1, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 18, 20, 21, 22, 24, 27, 28, 29, 30], "ofter": [21, 27], "ol": [0, 13, 17, 19, 28, 30], "old": [1, 5, 10, 13, 15, 18], "ols_paramet": 16, "ols_sk": 6, "ols_svd": 6, "olsbeta": 29, "olstheta": [0, 5], "omega": [2, 3, 6], "omega_0": 3, "omit": [0, 5, 27, 28, 29], "onc": [1, 6, 9, 11, 13], "one": [0, 1, 3, 4, 5, 6, 7, 8, 9, 10, 11, 13, 14, 15, 19, 20, 21, 22, 24, 25, 27, 28, 30], "onehot": 1, "onehot_vector": 1, "onehotencod": 9, "ones": [0, 2, 5, 6, 8, 9, 10, 11, 13, 16, 18, 21, 22, 27, 28, 29, 30], "ones_lik": 4, "ong": 28, "onl": 3, "onli": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 17, 18, 21, 22, 24, 27, 28, 29, 30], "onlin": [11, 15, 23], "onto": [5, 11, 28, 29], "open": [0, 1, 4, 6, 7, 9, 15, 20, 22, 23, 25, 27], "oper": [0, 1, 3, 5, 6, 10, 11, 12, 13, 15, 16, 20, 24, 27, 28, 29, 30], "operation": 24, "oplu": 24, "opmiz": [13, 30], "opportun": 0, "oppos": [6, 13], "opposit": [1, 5, 8, 28, 29], "opt": [1, 5, 22, 27, 29], "optim": [0, 2, 3, 4, 5, 6, 7, 9, 10, 11, 14, 16, 17, 19, 22], "optimis": [1, 3], "option": [0, 1, 3, 5, 6, 8, 11, 15, 18, 21, 28, 30], "optmiz": [1, 8, 13, 28], "oral": 27, "orang": 0, "order": [0, 1, 2, 3, 5, 6, 7, 8, 9, 10, 11, 12, 15, 19, 21, 22, 24, 27, 28, 29], "ordinari": [0, 2, 3, 7, 11, 13, 17, 18, 20], "oreilli": [26, 27], "org": [0, 3, 4, 16, 20, 21, 22, 26, 27, 28, 29, 30], "organ": [6, 7, 10, 21], "orient": [1, 5, 24, 28, 29], "origin": [0, 3, 5, 6, 8, 11, 12, 13, 15, 21, 27, 28, 29, 30], "orthogn": [5, 28, 29], "orthogon": [0, 5, 6, 8, 11, 13, 21, 27, 28, 29], "orthonorm": [5, 28, 29], "os": [25, 27], "oscar": 1, "oscil": [3, 13, 30], "oskar": 27, "oskarlei": 27, "osl": 18, "oslo": [0, 20, 22, 23, 25, 27, 28, 29, 30], "osx": [0, 20, 22, 27], "other": [0, 1, 2, 3, 5, 6, 7, 8, 10, 13, 14, 16, 20, 23, 24, 25, 26, 28, 29, 30], "otherwis": [0, 1, 4, 7, 13, 21, 27, 30], "ouput": [5, 7, 12], "our": [1, 2, 3, 6, 7, 8, 9, 10, 12, 14, 15, 16, 17, 18, 19, 20, 21, 24, 30], "ourmodel": 0, "ourselv": [0, 5, 6, 8, 11, 13, 27, 28, 29], "out": [0, 1, 2, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 15, 16, 20, 21, 22, 24, 27, 28, 30], "out_fil": 9, "outcom": [0, 7, 9, 10, 12, 24, 28], "outdoor": 9, "outer": [6, 12, 13], "outfil": 4, "outlier": [0, 8, 27, 28, 30], "outlin": [6, 10, 11], "outlook": 9, "outperform": [10, 30], "output": [0, 1, 3, 4, 5, 6, 7, 8, 9, 10, 12, 13, 21, 22, 24, 27, 28, 29, 30], "output_bia": 1, "output_bias_gradi": 1, "output_shap": 4, "output_weight": 1, "output_weights_gradi": 1, "outputlayer1": 12, "outputlayer2": 12, "outsid": 4, "over": [0, 1, 3, 4, 5, 6, 9, 10, 12, 13, 15, 16, 21, 22, 27, 28, 29, 30], "over1": 13, "overal": [1, 10, 30], "overcast": 9, "overcom": [12, 13], "overdetermin": [0, 27], "overfit": [0, 1, 3, 6, 9, 10, 13, 30], "overflow": 5, "overhead": 12, "overlap": [3, 7, 8, 9], "overlin": [0, 5, 6, 9, 10, 11, 14, 21, 27, 28, 30], "overshoot": 30, "overst": 0, "overtrain": 4, "overview": 3, "own": [4, 5, 6, 8, 12, 13, 16, 18, 20, 21, 29, 30], "owner": [], "ownmsepredict": 0, "ownmsetrain": 0, "ownridgebeta": 28, "ownridgetheta": [0, 6, 28, 29, 30], "ownypredictridg": 0, "ownytilderidg": 0, "oxid": [], "p": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 13, 14, 15, 16, 17, 18, 19, 21, 24, 27, 28, 29, 30], "p0": 2, "p1": 2, "p_": [2, 4, 8, 9], "p_hidden": 2, "p_i": [5, 24], "p_j": 24, "p_n": 24, "p_output": 2, "p_x": 24, "pack": [0, 27], "packag": [0, 1, 3, 4, 5, 8, 11, 13, 15, 20, 22, 24, 28, 29, 30], "packtpub": 27, "packtpublish": 27, "pad": [3, 4], "page": [0, 20, 27, 29, 30], "pai": [0, 1, 9, 13, 15, 30], "pair": [0, 2, 3, 9, 20, 24, 27], "paltform": 15, "panda": [0, 4, 5, 6, 7, 9, 11, 20, 22, 29, 30], "panel": 27, "paper": [1, 30], "paradigm": [0, 27], "parallel": [10, 13, 20, 21, 27], "param": 2, "paramat": 2, "paramet": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 12, 13, 16, 17, 18, 19, 22, 24, 29, 30], "parameter": [0, 6, 10, 27, 28], "parametr": [0, 6, 27, 28], "paramt": [3, 5], "part": [0, 1, 3, 5, 6, 10, 17, 19, 21, 23, 24, 25, 27, 28], "partial": [0, 1, 5, 6, 7, 8, 10, 11, 12, 13, 16, 24, 27, 28, 29, 30], "particip": [15, 20, 23, 25, 27], "particl": [0, 4, 13, 24, 27], "particular": [0, 1, 2, 3, 5, 6, 9, 10, 11, 12, 13, 16, 22, 24, 26, 27, 28, 29, 30], "particularli": [5, 6, 8, 11, 13, 24, 28, 29, 30], "partit": [1, 4, 9], "partli": [6, 27], "partner": 15, "pass": [2, 3, 12, 14], "password": 22, "past": [10, 24, 30], "patch": [6, 24], "path": [0, 4, 6, 7, 9, 20, 27], "pathcollect": 17, "patient": 7, "patter": 4, "pattern": [0, 3, 4, 12, 26, 27], "pauli": [0, 27], "pc": [11, 15, 20], "pca": [0, 7, 20, 27, 28], "pd": [0, 4, 5, 6, 7, 9, 11, 27, 28, 29, 30], "pde": 2, "pdf": [0, 3, 4, 5, 6, 9, 15, 16, 22, 26, 27], "pedagog": [0, 27, 28], "penal": [6, 18, 28, 30], "penalti": [6, 13, 18, 22, 28, 30], "penros": [5, 6], "pentagon": [13, 29], "peopl": [1, 9, 13, 20, 30], "per": [0, 1, 6, 23, 25, 27, 30], "percentag": [10, 11, 25], "perceptron": [0, 1, 7, 27], "peregrin": 27, "perfect": [0, 1, 13, 27, 30], "perfectli": [4, 6], "perform": [0, 2, 3, 4, 5, 6, 8, 10, 11, 12, 13, 14, 16, 18, 19, 20, 21, 22, 24, 27, 28, 29, 30], "performac": 4, "perhap": [0, 5, 13, 27, 28, 29], "perimet": 1, "period": [1, 4, 24], "permiss": 15, "permut": 11, "persist": 13, "person": [5, 6, 7, 16, 23, 25, 27, 28], "perspect": 26, "pertin": [12, 27], "petal": [8, 9], "peter": [26, 28], "phantom": 24, "phase": [6, 12], "phenomena": 24, "phi": 8, "phi_k": 8, "philosophi": 13, "phone": [25, 27], "photo": [4, 27], "php": 22, "phrase": [0, 27], "physic": [0, 1, 4, 7, 12, 13, 24, 25, 26, 27, 28, 29, 30], "pi": [2, 3, 5, 6, 7, 9, 12, 13, 24], "pick": [1, 9, 10, 11, 13, 14, 30], "pickl": 1, "pictur": [0, 27], "pie": [20, 27], "piec": [11, 14], "pillow": [0, 20, 22, 27], "pinv": [5, 6, 13, 22, 28, 29, 30], "pip": [0, 1, 15, 20, 22, 27], "pip3": [0, 1, 22, 27], "pipelin": [0, 6, 8, 10, 28], "pippin": 27, "pit": 4, "pitfal": [6, 28], "pitt": 12, "pixel": [1, 3, 4, 27], "pixel_height": [1, 3], "pixel_width": [1, 3], "place": [0, 4, 6, 8, 13, 15, 21, 22, 27, 29], "plai": [0, 3, 4, 5, 6, 8, 11, 18, 20, 22, 27, 28, 29], "plain": [8, 10, 12, 13, 14, 29, 30], "plan": [6, 9, 25, 26, 27], "plane": [8, 9], "plateau": [5, 29], "platform": [20, 27], "plausibl": 12, "pleas": [13, 22, 25, 27], "plenti": 1, "plethora": [3, 12], "plot": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 16, 17, 19, 20, 21, 22, 24, 27, 28, 29, 30], "plot_all_sc": [22, 28], "plot_confusion_matrix": [7, 10], "plot_count": 6, "plot_cumulative_gain": [7, 10], "plot_data": 1, "plot_dataset": 8, "plot_decision_boundari": [9, 10], "plot_import": 10, "plot_max": 4, "plot_min": 4, "plot_model": 4, "plot_numb": 4, "plot_predict": 8, "plot_regression_predict": 9, "plot_result": 4, "plot_roc": [7, 10], "plot_surfac": [2, 6, 13], "plot_train": 9, "plot_tre": [9, 10], "plt": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 17, 19, 21, 24, 27, 28, 29, 30], "plu": [0, 3, 5, 7, 18, 27, 28], "pm": 8, "pmatrix": 2, "pml": 26, "pn": 3, "png": [0, 4, 6, 7, 9, 27], "point": [0, 1, 2, 3, 5, 6, 7, 8, 9, 10, 11, 13, 14, 18, 19, 21, 22, 24, 25, 27, 28, 29, 30], "point_1": 4, "point_2": 4, "poisson": [20, 24, 27], "poli": [6, 8], "poly100_kernel_svm_clf": 8, "poly3": 0, "poly3_plot": 0, "poly_featur": [8, 9, 15], "poly_features10": 9, "poly_fit": 9, "poly_fit10": 9, "poly_kernel_svm_clf": 8, "poly_model": 15, "poly_ms": 15, "poly_predict": 15, "polydegre": [0, 5, 6, 10, 28], "polygon": [13, 29], "polym": 12, "polymi": 22, "polynomi": [0, 5, 6, 7, 8, 9, 10, 11, 15, 17, 19, 22, 27, 28], "polynomial_featur": [6, 15, 16, 17], "polynomial_svm_clf": 8, "polynomialfeatur": [0, 6, 8, 9, 15, 16, 19, 28], "polytrop": [0, 6], "pool": 3, "pool_siz": 3, "poor": [1, 13, 29, 30], "poorli": [0, 28], "popul": [0, 5, 27, 28], "popular": [0, 1, 3, 6, 7, 8, 9, 11, 12, 15, 20, 21, 22, 24, 28], "popularli": [0, 27], "portabl": 10, "portion": [11, 13, 30], "pose": [0, 4, 5, 6, 11, 24, 27], "posit": [0, 1, 2, 3, 5, 7, 8, 10, 11, 13, 14, 21, 24, 27, 28, 29, 30], "possibl": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 20, 21, 22, 24, 25, 27, 28, 29, 30], "possibli": [6, 8, 13, 22], "posterior": 5, "postpon": [0, 28], "postscript": 22, "postul": 5, "potenti": [0, 3, 5, 6, 12, 13, 28, 30], "pott": 12, "power": [0, 1, 5, 6, 8, 9, 12, 13, 27, 28, 29, 30], "pp": [5, 6, 19], "practic": [0, 5, 6, 7, 8, 16, 18, 22, 24, 28], "practition": [0, 1, 3, 27], "pre": 27, "preced": [1, 11, 12, 24], "preceed": 4, "preceq": 8, "precis": [0, 2, 5, 11, 13, 21, 22, 24, 27, 28], "pred": 6, "predicit": 0, "predict": [0, 1, 5, 6, 7, 8, 9, 10, 15, 16, 17, 19, 20, 22, 26, 27, 28, 29, 30], "predict_prob": 1, "predict_proba": [7, 10], "predictor": [0, 5, 6, 7, 9, 10, 11, 27, 28, 30], "prefer": [0, 1, 6, 8, 9, 11, 13, 15, 20, 22, 27], "prepar": [0, 6, 21, 22, 27, 28], "preprocess": [0, 4, 6, 7, 8, 9, 10, 11, 15, 16, 17, 18, 19, 22], "prerequisit": 0, "prescript": 22, "presenc": 13, "present": [0, 5, 6, 7, 9, 12, 13, 21, 22, 24, 27, 28, 29, 30], "preserv": [3, 11, 21], "press": [13, 15, 26, 29], "pretrain": [1, 4], "pretti": [0, 4, 8, 9, 20, 22, 27], "prev_centroid": 14, "prevent": [13, 24, 30], "previou": [0, 1, 2, 3, 4, 5, 6, 8, 10, 11, 12, 13, 15, 16, 21, 22, 24, 28, 29, 30], "previous": [2, 3, 9, 10, 24], "price": [0, 4, 9, 13, 30], "primal": 8, "primari": [0, 7, 27], "prime": 24, "princip": [0, 5, 7, 20, 27, 28, 29], "principl": [0, 6, 7, 8, 14, 27], "print": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 13, 14, 18, 21, 24, 27, 28, 29, 30], "print_funct": [8, 9], "printout": [0, 27], "prior": [0, 5, 6, 27], "privat": 0, "prob": [1, 24], "probabilist": [0, 26, 27, 28], "probabl": [0, 1, 3, 4, 6, 7, 10, 13, 20, 27, 28, 30], "problem": [0, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 17, 19, 20, 21, 22, 24], "probml": 26, "proce": [0, 5, 6, 7, 8, 9, 10, 11, 13, 21, 27, 28], "procedur": [2, 4, 5, 6, 8, 10, 11, 13, 28, 29, 30], "proceed": 21, "process": [0, 2, 4, 6, 9, 10, 12, 13, 20, 21, 22, 24, 26, 27, 29, 30], "prod": 26, "prod_": [1, 5, 7], "produc": [0, 3, 4, 5, 6, 9, 10, 11, 12, 13, 18, 20, 21, 24, 27, 28], "product": [0, 1, 3, 5, 6, 7, 8, 12, 13, 16, 17, 20, 21, 27, 28, 30], "profess": [0, 27], "program": [0, 1, 4, 5, 6, 8, 12, 14, 15, 20, 21, 23, 24, 25, 27, 28], "programm": 21, "progress": [1, 4, 14, 30], "prohibit": 6, "project": [0, 1, 2, 3, 5, 11, 13, 15, 19, 20, 23, 28, 29, 30], "project_root_dir": [0, 6, 7, 9, 27], "promin": 12, "promis": 8, "promot": [25, 27], "prone": [9, 15], "pronounc": [13, 20, 27, 30], "proof": [0, 11, 12, 13, 27, 29], "prop": 30, "propag": [2, 3, 13, 30], "proper": [0, 2, 6, 7], "properli": [1, 6, 8, 10, 13, 18, 22, 30], "properti": [0, 1, 3, 12, 13, 16, 21, 27], "proport": [0, 1, 5, 9, 11, 13, 24, 27, 28], "propos": [1, 4, 6, 10, 22, 27, 30], "propto": [5, 13, 29, 30], "proton": [0, 27], "prove": [3, 13, 29, 30], "provid": [0, 1, 3, 4, 5, 6, 8, 9, 10, 12, 13, 20, 21, 22, 24, 27, 28, 29, 30], "proxi": [1, 13, 30], "prune": 9, "pseudo": [21, 24, 30], "pseudocod": 22, "pseudoinv": 5, "pseudoinvers": [5, 6, 22], "pseudorandom": [6, 24], "psychologi": [0, 27], "pt": 13, "public": [0, 15, 20, 27], "pull": 15, "punish": [0, 1, 27], "pure": [3, 9, 24], "purest": 9, "puriti": 9, "purpos": [0, 3, 10, 12, 14, 27], "push": 15, "put": 1, "py": 5, "pycod": 27, "pydata": 20, "pydot": 9, "pyhton2": 27, "pylab": [7, 27], "pypi": 20, "pyplot": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 17, 19, 21, 24, 27, 28, 29, 30], "pythagora": 5, "python": [1, 2, 3, 5, 6, 8, 11, 12, 13, 14, 18, 22, 24, 28, 30], "python2": [0, 22], "python3": [0, 20, 22, 27], "pytorch": [0, 20, 22, 27], "q": [5, 6, 8, 11, 24], "qp": 8, "qquad": [2, 11, 13, 21, 30], "qr": [5, 6, 21, 28, 29], "quad": [1, 13, 21], "quadrat": [0, 8, 9, 13, 27], "qualit": [4, 9, 22, 24], "qualiti": [0, 9, 20, 27, 28], "quantifi": 1, "quantil": 10, "quantit": [0, 6, 9, 22, 27], "quantiti": [0, 2, 5, 6, 7, 9, 10, 11, 12, 14, 16, 21, 24, 27, 28, 29, 30], "quantum": [4, 12, 26, 27], "quartil": [0, 28, 30], "quench": 5, "queri": 9, "question": [0, 5, 6, 9, 11, 12, 13, 22, 25, 27, 28, 30], "qugan": 4, "quick": [4, 24], "quickli": [1, 3, 9, 11, 13, 29], "quit": [1, 5, 6, 9, 10, 12, 15, 28, 29], "quot": 4, "r": [0, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 17, 20, 21, 22, 24, 28, 29, 30], "r2": [0, 5, 6, 27, 28, 29], "r2_score": [0, 27], "r2score": [0, 27], "r_": 30, "r_0": 30, "r_1": 9, "r_2": 9, "r_j": 9, "r_m": 9, "r_t": 30, "rad": [], "rade": [], "radial": [8, 12], "radioact": 24, "radiu": [0, 1, 28, 30], "rain": 9, "ramp": 1, "ran0": 24, "ran1": 24, "ran2": 24, "ran3": 24, "rand": [0, 4, 5, 6, 9, 10, 13, 15, 19, 21, 27, 28, 29, 30], "randint": [6, 9, 13, 30], "randn": [0, 1, 2, 5, 6, 9, 11, 13, 15, 18, 27, 28, 29, 30], "random": [0, 1, 2, 3, 4, 5, 6, 8, 9, 13, 14, 15, 16, 17, 18, 19, 20, 21, 27, 28, 29, 30], "random_forest_model": 10, "random_index": [13, 30], "random_indic": [1, 3], "random_st": [7, 8, 9, 10, 11], "randomforestclassifi": 10, "randomli": [1, 6, 9, 13, 14, 18, 29, 30], "rang": [0, 1, 2, 3, 4, 5, 6, 7, 9, 10, 11, 12, 13, 14, 18, 19, 21, 24, 27, 28, 29, 30], "rangl": [0, 6, 11, 24, 27, 28], "rangle_x": 24, "rank": [5, 28, 29], "rankdir": 4, "raphson": [1, 8, 13], "rapidli": 0, "rare": [1, 13, 30], "raschka": [27, 28], "rasckha": 27, "rashcka": [29, 30], "rate": [1, 2, 3, 4, 8, 9, 10, 12, 13, 18, 29], "rather": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 21, 24, 27, 28, 29], "ratio": [4, 7, 9, 10, 11], "rational": [0, 27], "ravel": [5, 6, 7, 8, 9, 10, 11, 13, 21], "raw": [3, 30], "rbf": [8, 11, 12], "rbf_kernel_svm_clf": 8, "rbf_pca": 11, "rc": 24, "rcond": [0, 27, 28], "rcparam": [1, 3, 7, 8, 9, 10, 24, 27], "re": [2, 4, 13, 15, 29], "reach": [1, 4, 5, 6, 9, 10, 12, 13, 14, 29, 30], "read": [0, 2, 3, 4, 5, 6, 7, 8, 11, 12, 16, 17, 21, 22, 24, 26, 29], "read_csv": [0, 6, 7, 9], "read_fwf": [0, 27], "reader": [0, 6, 21, 24, 27, 28, 30], "readi": [0, 1, 5, 6, 8, 10, 11, 12, 21, 27], "readili": 1, "readm": 15, "readthedoc": 20, "real": [0, 1, 4, 7, 10, 11, 12, 16, 18, 19, 21, 28], "real_loss": 4, "real_output": 4, "realist": [8, 27], "realiti": 24, "realiz": [1, 12], "realli": [0, 1, 27], "rearrang": 13, "reason": [0, 1, 3, 4, 10, 13, 26, 27, 29, 30], "reassign": 1, "recal": [5, 6, 9, 10, 11, 12, 21, 24, 27, 28, 29, 30], "recast": 3, "receiv": [1, 3, 10, 12, 24], "recent": [0, 6, 13, 26, 30], "recept": [3, 12], "receptive_field": 3, "recip": [0, 6, 7, 21, 22, 27, 28], "reciproc": 5, "recogn": [0, 4, 5, 10, 27], "recognit": [0, 1, 3, 12, 26, 27], "recommen": 27, "recommend": [0, 2, 3, 4, 5, 6, 8, 13, 15, 19, 20, 21, 22, 26, 29, 30], "reconsid": 9, "reconstruct": 11, "record": [10, 22, 23, 25, 27], "recreat": 15, "rectangl": [9, 13, 29], "rectangular": [5, 28, 29], "rectifi": [1, 3, 12], "recur": [0, 20, 27], "recurr": [0, 1, 20, 27], "recurs": [9, 20, 21, 27], "red": [0, 3, 4, 6, 8, 9, 30], "redefin": [0, 10, 27, 28, 29], "redefinit": 29, "reduc": [1, 3, 5, 6, 9, 10, 11, 13, 27, 29, 30], "reduct": [0, 10, 11, 20, 24, 27, 28], "reegress": 22, "refer": [0, 1, 2, 3, 5, 6, 11, 12, 13, 14, 21, 26, 27, 28, 29, 30], "referenc": 2, "refin": 12, "refit": 6, "reflect": [0, 1, 4, 5, 22, 24, 27], "refresh": [20, 27], "refreshprogrammingskil": 27, "reg": [10, 11], "regard": [1, 9, 13], "regardless": [12, 16], "region": [3, 4, 6, 9, 12, 22], "regist": [6, 24], "reglasso": [5, 29], "regr_1": [0, 9], "regr_2": [0, 9], "regr_3": [0, 9], "regress": [1, 8, 11, 12, 16, 20, 21], "regressor": [0, 7, 10], "regret": [], "regridg": [0, 5, 6, 28, 29, 30], "regular": [0, 3, 4, 5, 6, 7, 9, 13, 17, 18, 25, 27, 28, 29, 30], "regularli": 15, "reilli": [0, 26, 27], "reinforc": [0, 8, 20, 27], "reiter": 1, "reject": 7, "rel": [0, 4, 6, 7, 9, 12, 13, 24, 27, 28, 30], "relat": [0, 1, 3, 4, 5, 11, 13, 14, 19, 21, 24, 27, 28, 29], "relationship": [0, 4, 9, 18, 27], "relativeerror": [0, 27, 28], "releas": [1, 20, 27], "relev": [0, 1, 5, 7, 11, 20, 22, 24, 27, 29], "reli": [0, 6, 8], "reliabilti": 22, "reliabl": [7, 24], "relu": [3, 4, 27], "remain": [1, 2, 4, 6, 12, 21, 24, 28, 30], "remaind": 24, "reman": 2, "remark": 1, "rememb": [0, 8, 13, 21, 22, 27], "remind": [0, 5, 11, 13, 21, 24], "remot": 15, "remov": [4, 5, 6, 18, 28, 29, 30], "renam": 15, "render": [0, 27, 28], "reorder": [5, 7, 28, 29], "reorgan": [0, 27], "repeat": [0, 1, 3, 4, 5, 6, 9, 10, 11, 13, 14, 21, 22, 24, 27, 28, 29, 30], "repeated": 27, "repeatedli": [0, 6, 10, 13], "repet": 3, "repetit": [6, 27, 28], "rephras": [13, 29], "replac": [0, 1, 3, 4, 5, 6, 10, 12, 14, 20, 22, 27, 28, 29], "replica": 6, "repo": [15, 22], "report": [27, 30], "repositori": [4, 22, 27], "repres": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 12, 13, 22, 24, 27, 28, 29, 30], "represent": [0, 1, 3, 6, 24, 27], "representd": 3, "reproduc": [0, 5, 6, 9, 12, 15, 16, 18, 20, 24, 27, 28], "repuls": [0, 27], "request": [0, 13, 30], "requir": [0, 1, 3, 4, 5, 6, 8, 9, 11, 12, 13, 15, 17, 18, 21, 27, 28, 29, 30], "res1": 2, "res2": 2, "res3": 2, "res_analyt": 2, "res_analytical1": 2, "res_analytical2": 2, "res_analytical3": 2, "resaml": 6, "resampl": [0, 7, 10, 20, 27, 28], "rescal": [0, 11, 12, 30], "rescu": 5, "reseach": 6, "research": [0, 4, 13, 20, 26, 27], "resembl": [6, 24], "reserv": [1, 5, 6, 24], "reshap": [0, 1, 2, 3, 4, 6, 8, 9, 10, 21, 27, 28], "residenti": [], "residu": [0, 5, 13, 27], "resiz": [5, 28, 29], "resourc": 27, "respect": [0, 1, 2, 3, 5, 6, 7, 8, 10, 11, 12, 13, 14, 16, 17, 18, 22, 24, 27, 28, 29, 30], "respond": 12, "respons": [0, 7, 9, 12, 27, 28], "rest": [0, 5, 18, 28, 29, 30], "restat": [0, 12, 27], "restor": 4, "restored_discrimin": 4, "restored_gener": 4, "restrict": [0, 3, 9, 12, 27], "result": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 20, 21, 22, 24, 27, 30], "retail": [], "retain": [5, 6, 28, 29, 30], "return": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 11, 13, 14, 16, 17, 21, 24, 27, 28, 29, 30], "return_data": 14, "return_sequ": 4, "return_x_i": 9, "reus": [1, 3, 6, 19, 22], "reveal": [0, 12, 27], "revers": [1, 21], "review": [20, 21], "revisit": 14, "revolut": 27, "reward": [0, 4, 27], "rewrit": [0, 3, 5, 6, 7, 8, 10, 11, 12, 13, 16, 19, 21, 22, 24, 29, 30], "rewritten": [2, 6, 8, 10, 24], "rewrot": 13, "rf": 10, "rgb": 3, "rgoj5yh7evk": 20, "rh": 6, "rho": [0, 10, 13, 30], "rho_1": 10, "rho_2": 10, "rho_m": 10, "rich": [0, 27], "ride": 9, "rideclass": 9, "ridedata": 9, "ridg": [7, 11, 13, 20, 27], "ridge_paramet": 17, "ridge_sk": 6, "ridgebeta": 29, "ridgetheta": 5, "right": [0, 1, 2, 3, 5, 6, 7, 8, 9, 10, 12, 13, 14, 16, 17, 19, 21, 22, 24, 27, 28, 29, 30], "right_sid": 2, "rightarrow": [0, 1, 5, 6, 8, 11, 12, 13, 24, 27, 28, 29, 30], "rigor": [0, 27, 28, 29], "ring": 6, "rise": [0, 27], "risk": [0, 13, 27, 29, 30], "rival": 4, "river": [], "rlm": 27, "rm": [24, 30], "rmse": [], "rmsporp": [13, 30], "rmsprop": [1, 3, 4, 13, 22], "rnd_clf": 10, "rng": 24, "rnn": [4, 12], "rnn1": 4, "rnn2": 4, "rnn_2layer": 4, "rnn_input": 4, "rnn_output": 4, "rnn_train": 4, "rntrick1": 24, "rntrick2": 24, "rntrick3": 24, "rntrick4": 24, "ro": [0, 13, 27, 29, 30], "robert": [19, 22, 26], "robust": [0, 27, 30], "robustscal": [0, 28, 30], "roc": [7, 10], "role": [0, 2, 5, 6, 8, 18, 20, 22, 27, 28, 29, 30], "roll": 6, "room": [0, 25, 27], "root": [0, 5, 9, 13, 15, 24, 28, 29, 30], "rot": 27, "rotat": [1, 8, 9, 10], "rotation_matrix": 9, "roughli": [1, 3, 18], "round": [7, 9, 13], "routin": [13, 21, 27, 29], "row": [0, 1, 2, 5, 6, 9, 11, 16, 21, 27, 28, 29], "rr": [5, 28, 29], "rrr": [5, 28, 29], "rudg": [], "rug": [13, 29, 30], "rule": [0, 1, 5, 6, 13, 22, 27, 28, 29], "run": [0, 1, 2, 4, 5, 6, 8, 9, 11, 13, 15, 20, 22, 27, 28, 29, 30], "runtim": [1, 6, 14, 15], "rust": [0, 20, 21, 27], "rvert": 1, "rvert_2": 1, "s_": [3, 6], "s_1": 6, "s_i": [6, 7], "s_j": 6, "s_k": 6, "s_phenomenon": 22, "saddl": [13, 29, 30], "safeguard": [18, 30], "sai": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 19, 21, 22, 24, 27, 28, 29], "said": [6, 9, 13, 29], "sake": [0, 5, 7, 11, 27, 28, 29], "sale": [0, 27], "sam": 27, "same": [0, 1, 2, 3, 4, 5, 6, 8, 9, 11, 12, 14, 15, 16, 18, 21, 22, 24, 27, 28, 29], "samm": 10, "sampl": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 13, 14, 18, 19, 20, 21, 22, 24, 27, 28, 30], "sample_vari": 14, "sampleexptvari": 24, "samwis": 27, "sastri": 11, "satisfactori": [0, 27], "satisfi": [1, 2, 3, 6, 8, 13, 21, 24, 29], "satur": [1, 6], "save": [0, 4, 6, 7, 9, 13, 27, 30], "save_fig": [0, 6, 7, 9, 10, 27], "savefig": [0, 4, 6, 7, 9, 24, 27], "savetxt": 4, "saw": [5, 28], "scalabl": 10, "scalar": [2, 5, 6, 10, 28], "scale": [0, 1, 3, 5, 6, 7, 8, 9, 10, 11, 12, 13, 20, 21, 22, 25, 27, 29], "scale_mean": 4, "scale_std": 4, "scaler": [0, 7, 8, 9, 10, 11, 17, 22, 28], "scan": [5, 7], "scari": 5, "scatter": [0, 1, 6, 7, 8, 9, 14, 15, 17, 27, 28, 30], "scenario": [6, 13, 29, 30], "schedul": [13, 30], "scheme": [1, 13, 29, 30], "schrage": 24, "sch\u00f8yen": [6, 28, 30], "scienc": [0, 1, 10, 12, 13, 20, 23, 24, 25, 26, 29], "scientif": [0, 20, 22, 27], "scientist": [0, 27], "scikit": [3, 5, 6, 8, 9, 10, 13, 15, 16, 20, 21, 22, 26], "scikit_learn": 0, "scikitlearn": 27, "scikitplot": [7, 10], "scipi": [0, 3, 5, 6, 13, 20, 21, 22, 27, 28, 29], "scl": 6, "scm": 15, "score": [0, 1, 3, 6, 7, 9, 10, 11, 15, 16, 22, 25, 27, 28, 30], "scores_kfold": 6, "scratch": [1, 13, 16], "sdg": [13, 30], "sdv4f4s2sb8": [29, 30], "seaborn": [0, 1, 3, 6, 7, 27], "seamless": [0, 20, 22, 27], "search": [0, 1, 3, 5, 9, 13, 15, 27, 29, 30], "sebastian": 27, "sebastianraschka": 27, "sec": 6, "second": [0, 2, 3, 4, 5, 6, 7, 8, 9, 11, 12, 14, 15, 16, 20, 21, 24, 25, 27, 28, 29], "second_mo": 30, "second_term": 30, "secondeigvector": 11, "secondli": 12, "section": [4, 11, 16, 21, 22, 24, 28, 30], "sector": 0, "see": [0, 1, 2, 3, 4, 5, 6, 7, 8, 10, 11, 12, 13, 15, 16, 18, 19, 20, 21, 22, 24, 27, 28, 29, 30], "seed": [0, 1, 2, 3, 4, 5, 6, 8, 9, 11, 13, 14, 18, 24, 27, 28, 29, 30], "seed_imag": 4, "seek": [1, 2, 8], "seem": [1, 3, 4, 30], "seemingli": [0, 27], "seen": [0, 1, 3, 5, 10, 12, 24], "segment": [13, 29], "seismic": 6, "seldomli": [0, 27], "select": [1, 5, 6, 8, 9, 10, 11, 15, 22, 23, 24, 25, 26, 27, 28, 29], "selevet": 15, "self": [1, 5, 28], "sell": 4, "semest": [7, 23], "semi": [8, 13, 29, 30], "semilogx": 6, "send": [5, 12, 13, 25, 27], "senior": [23, 25], "sens": [0, 4, 6, 8, 27], "sensibl": 3, "sensit": [0, 5, 6, 9, 13, 27, 28, 30], "sent": 2, "sentenc": [4, 12], "separ": [0, 1, 2, 4, 6, 8, 9, 12, 14, 18, 20, 22, 24, 27, 30], "septemb": [18, 22, 27], "sequenc": [3, 4, 7, 9, 10, 12, 13, 20, 21, 24, 27, 29], "sequenti": [1, 3, 4, 10, 12, 24], "seri": [0, 1, 2, 3, 4, 5, 6, 10, 11, 12, 13, 21, 27, 28, 29], "serif": [7, 24, 27], "serv": [0, 1, 2, 3, 5, 7, 13, 26, 27, 28, 29, 30], "servic": 22, "session": [1, 15, 22, 23, 25, 27], "set": [1, 4, 5, 6, 7, 8, 10, 11, 13, 14, 16, 17, 18, 20, 21, 22, 24, 25, 30], "set_major_formatt": 6, "set_major_loc": 6, "set_tick": [1, 8], "set_ticklabel": 1, "set_titl": [0, 1, 2, 3, 7, 12, 14, 27], "set_xlabel": [0, 1, 2, 3, 7, 12, 27], "set_xlim": [7, 12], "set_xticklabel": 1, "set_ylabel": [0, 1, 2, 3, 7, 27], "set_ylim": [7, 12], "set_ytick": 7, "set_yticklabel": [1, 6], "set_zlim": 6, "seth": 4, "setminu": 6, "setosa": [8, 9], "setosa_or_versicolor": 8, "setp": 6, "setup": [1, 4, 6, 8, 20, 27, 28, 29], "sever": [0, 3, 5, 6, 7, 8, 9, 11, 12, 13, 16, 20, 21, 22, 24, 27, 28, 29, 30], "sgd": [1, 3, 29], "sgd_clf": 8, "sgdclassifi": 8, "sgdreg": 13, "sgdregressor": 13, "sgn": [5, 28, 29], "shallow": [13, 30], "shape": [0, 1, 3, 4, 5, 6, 7, 8, 9, 10, 11, 13, 14, 15, 16, 18, 21, 27, 28, 29, 30], "share": [1, 3, 15, 27], "shareabl": 15, "she": 7, "shift": [1, 6, 12, 15, 18, 24, 28, 30], "ship": 3, "shire": 27, "short": [4, 5, 22], "shortcom": [13, 29, 30], "shorten": 4, "shorter": 24, "shorthand": 27, "shortli": [21, 27], "should": [0, 2, 3, 5, 6, 8, 9, 11, 12, 15, 18, 21, 22, 24, 27, 28, 30], "show": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 19, 21, 22, 24, 27, 28, 29, 30], "show_shap": 4, "shown": [0, 4, 5, 8, 12, 13, 21, 28, 29, 30], "shrink": [3, 5, 6, 8, 11, 28, 29], "shrinkag": [5, 6, 28, 29], "shrunk": 11, "shuffl": [0, 1, 4, 6, 13, 28, 30], "side": [0, 2, 5, 8, 12, 13, 21, 22, 27, 29], "sigh": [20, 27], "sigma": [0, 1, 5, 6, 7, 10, 11, 12, 13, 19, 21, 22, 24, 27, 28, 29, 30], "sigma0": 24, "sigma1": 24, "sigma2": 24, "sigma_": [5, 21, 27, 28, 29], "sigma_0": [5, 28, 29], "sigma_1": [5, 28, 29], "sigma_2": [5, 28, 29], "sigma_fn": [7, 12], "sigma_i": [0, 5, 27, 28, 29], "sigma_j": [5, 28, 29], "sigma_m": [6, 24], "sigma_n": [11, 24], "sigma_t": 13, "sigma_x": 24, "sigmoid": [1, 2, 4, 7, 8, 10, 12], "sigmundson": [6, 28, 30], "sign": [1, 2, 7, 8, 10, 24, 25], "signal": [1, 3, 10, 12, 30], "signifi": 4, "signific": [1, 30], "significantli": [1, 13, 18, 24, 29, 30], "sim": [4, 5, 6, 13, 19, 24], "similar": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 14, 18, 20, 21, 22, 27, 29], "similarli": [0, 1, 3, 5, 8, 10, 13, 24, 27, 28, 29, 30], "simpl": [1, 2, 3, 5, 6, 7, 8, 10, 11, 12, 14, 16, 17, 20, 21, 24], "simplepredict": 10, "simpler": [0, 1, 5, 6, 7, 13, 16, 20, 22, 27, 29, 30], "simplernn": 4, "simplest": [0, 1, 3, 4, 9, 10, 12, 14, 22, 27], "simpletre": 10, "simpli": [0, 1, 2, 4, 5, 6, 8, 9, 10, 11, 12, 20, 21, 22, 24, 27, 28, 29, 30], "simplic": [2, 5, 6, 7, 8, 9, 10, 11, 12, 14, 28, 29, 30], "simplicti": [5, 28, 29], "simplifi": [0, 6, 9, 18, 20, 22, 27, 28, 30], "simplist": [3, 6, 24], "simul": [6, 18], "simultan": 6, "sin": [0, 1, 2, 3, 4, 9, 12, 13, 21, 27], "sinc": [0, 1, 2, 3, 5, 6, 7, 8, 9, 10, 11, 13, 16, 18, 21, 22, 24, 26, 27, 28, 29, 30], "sine": [3, 12], "singl": [0, 1, 2, 3, 5, 6, 7, 8, 9, 12, 13, 18, 19, 21, 24, 27, 28, 29, 30], "singular": [0, 6, 13, 21, 27], "sinusoid": 3, "site": [0, 22, 23, 28], "situat": [0, 4, 5, 7, 13, 24, 27, 28, 29, 30], "six": [3, 24], "size": [0, 1, 2, 3, 4, 5, 6, 8, 9, 10, 11, 13, 18, 21, 22, 24, 27], "sizesp": 30, "sketch": 10, "ski": 9, "skill": 0, "skip": 11, "skl": [0, 6, 27, 28, 30], "sklearn": [0, 1, 3, 5, 6, 7, 8, 9, 10, 11, 13, 14, 15, 17, 19, 27, 28, 29, 30], "skplt": [7, 10], "sl": [6, 28, 30], "slack": 8, "slice": [2, 21, 27], "slide": [0, 3, 16, 22, 24, 27, 28, 29], "slight": [6, 13], "slightli": [1, 2, 3, 5, 6, 7, 10, 24, 28, 29], "slope": [8, 11, 12], "slow": [0, 2, 8, 13, 18, 28, 29, 30], "slower": [5, 21, 27, 28, 29, 30], "slowest": 21, "slowli": [12, 30], "slp": 1, "small": [0, 1, 2, 3, 5, 6, 8, 9, 10, 11, 12, 13, 18, 20, 21, 24, 27, 28, 29, 30], "smaller": [0, 1, 2, 5, 6, 8, 9, 11, 13, 24, 27, 28, 29, 30], "smallest": [0, 4, 14, 27], "smallest_row_index": 14, "smooth": [0, 3, 6, 13, 22, 27, 29, 30], "smoother": 30, "sn": [0, 1, 3, 6, 7, 27], "sne": 11, "so": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 18, 19, 20, 21, 22, 24, 25, 27, 28, 29, 30], "soar": 6, "social": 0, "soft": [1, 7, 10, 12], "soften": 8, "softmax": [3, 7], "softwar": [0, 8, 20, 21], "sol": 8, "sole": [0, 6, 27], "solid": [0, 7], "solut": [0, 1, 2, 3, 5, 6, 8, 10, 11, 13, 18, 21, 22, 24, 27, 28, 29, 30], "solution_ev": 30, "soluton": 2, "solv": [0, 1, 3, 5, 6, 8, 10, 11, 12, 13, 16, 21, 22, 27, 28], "solve_expdec": 2, "solve_ode_deep_neural_network": 2, "solve_ode_neural_network": 2, "solve_pde_deep_neural_network": 2, "solveod": 2, "solveode_popul": 2, "solver": [2, 7, 8, 9, 10, 21, 27], "some": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 14, 15, 16, 18, 19, 22, 24, 27, 30], "some_model": [6, 28, 30], "somehow": 4, "someon": 16, "someth": [0, 1, 3, 4, 7, 9, 11, 15, 19, 22, 24, 27, 28], "sometim": [0, 1, 11, 12, 13, 14, 28, 30], "soon": [21, 25, 28], "sophist": [0, 27], "sopt": 13, "sort": [5, 6, 9, 11, 24], "sound": [3, 5], "sourc": [0, 1, 3, 6, 20, 21, 22, 24, 27], "space": [0, 1, 4, 5, 8, 9, 11, 12, 13, 14, 24, 28, 29, 30], "span": [0, 3, 5, 9, 11, 21, 27, 28, 29], "spare": 1, "spars": [3, 6, 18, 21, 27, 30], "sparse_mtx": [21, 27], "sparsecategoricalcrossentropi": 3, "sparsiti": [10, 18], "spatial": [1, 2, 3, 12], "speak": 24, "special": [6, 7, 10, 12, 13, 21, 24, 27, 28, 29, 30], "specif": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 11, 12, 15, 16, 20, 21, 22, 24, 26, 27, 28, 29], "specifi": [0, 3, 5, 6, 7, 9, 11, 13, 14, 24, 27, 29, 30], "specifici": [0, 10, 27], "spectacular": 3, "spectral": 1, "speech": [0, 1, 3, 4, 12], "speed": [1, 2, 4, 13, 30], "spend": [16, 24], "spent": 22, "sphere": [0, 28, 30], "spin": 6, "spite": 0, "spline": 8, "split": [1, 3, 4, 5, 6, 8, 9, 10, 11, 14, 16, 17, 22, 24, 27, 29, 30], "splite": 0, "splitter": [1, 10], "spontan": 24, "spot": 3, "spread": [0, 11, 24, 27, 28], "springer": [19, 22, 26, 27], "spuriou": [13, 30], "sqquar": 29, "sqrsignal": 3, "sqrt": [3, 4, 5, 6, 8, 10, 11, 13, 24, 28, 29, 30], "squar": [1, 2, 3, 4, 7, 8, 9, 11, 13, 14, 15, 17, 18, 20, 21, 24], "squarederror": 10, "squaredeuclidean": 14, "squash": 12, "srtm": 6, "srtm_data_norway_1": 6, "stabil": [5, 22, 30], "stabl": [0, 4, 5, 6, 9, 16, 20, 22, 27, 28, 29], "stack": [3, 4], "stage": [5, 13, 15, 22, 30], "stai": [0, 2, 4, 5, 11, 27, 28, 30], "stand": [0, 5, 9, 12, 27, 28, 29], "standard": [0, 1, 4, 5, 6, 7, 8, 10, 12, 17, 18, 21, 22, 24, 27, 29, 30], "standardscal": [0, 6, 7, 8, 9, 10, 11, 17, 28, 30], "stanford": [13, 29], "start": [0, 1, 2, 3, 4, 5, 6, 8, 9, 10, 11, 12, 13, 14, 15, 16, 18, 21, 24, 25, 27, 28, 29, 30], "start_tim": 14, "stat": 6, "state": [1, 2, 4, 5, 6, 7, 8, 10, 11, 12, 13, 20, 24, 27, 28, 29], "statement": [0, 7, 21, 27], "stationari": [29, 30], "statist": [0, 1, 3, 4, 7, 9, 10, 11, 12, 13, 14, 19, 21, 22, 26, 28, 29], "statu": [0, 7, 15, 27], "stavang": 6, "std": [0, 4, 6, 18, 27, 28, 30], "steep": [13, 29, 30], "steepest": 30, "step": [0, 1, 2, 4, 6, 7, 9, 10, 11, 12, 13, 14, 15, 18, 21, 22, 27, 29], "step_fn": [7, 12], "step_length": [13, 30], "step_siz": 30, "steps_list": 9, "stereo": 3, "still": [0, 2, 3, 5, 6, 11, 13, 24, 28, 29, 30], "stimuli": 12, "stk": [26, 27], "stk2100": [26, 27], "stk3155": [15, 22, 23, 25], "stk4021": [26, 27], "stk4051": [26, 27], "stk4155": [23, 25], "stk5000": 26, "stochast": [0, 1, 5, 6, 8, 11, 12, 29], "stock": 4, "stoke": 12, "stone": [0, 7], "stop": [1, 4, 9, 13, 14, 18, 29], "storag": [5, 28, 29], "store": [0, 1, 2, 3, 6, 11, 13, 24, 27, 30], "storehaug": [25, 27], "str": [1, 3, 4], "straight": [0, 6, 8, 13, 27, 29], "straightforward": [0, 2, 3, 5, 6, 8, 9, 10, 13, 21, 27, 28, 29], "strategi": [0, 1, 9, 27], "stratifi": 6, "strength": [0, 5, 14, 28, 29], "stretch": 11, "strict": [8, 13, 29], "strictli": [8, 13, 29], "stride": [4, 21], "strike": 6, "string": 1, "stroke": 7, "strong": [3, 6, 9, 10, 12, 21, 24], "strongli": [0, 8, 15, 20, 21], "stronli": [], "structur": [0, 1, 2, 3, 6, 9, 10, 12, 20, 27], "stuck": [1, 13, 29, 30], "student": [0, 15, 22, 23, 25, 26, 27], "studi": [0, 3, 4, 5, 6, 7, 8, 11, 12, 13, 20, 22, 26, 27, 28, 29, 30], "studier": 26, "style": [7, 9, 21, 27], "st\u00f8land": 25, "sub": [9, 12, 30], "subdivid": [0, 21, 27], "subfield": 0, "subgradi": 30, "subject": [6, 8, 24], "submit": 27, "subplot": [0, 1, 3, 4, 6, 7, 8, 9, 10, 14, 27], "subplots_adjust": [8, 24], "subprogram": [21, 27], "subract": [0, 28], "subroutin": [0, 27], "subscript": 1, "subsequ": [1, 4, 5, 6, 12, 21, 24, 28, 29], "subset": [1, 6, 9, 12, 13, 20, 27, 29, 30], "subspac": [0, 8, 11, 28], "substanti": [9, 10], "substep": 11, "substitut": [3, 6, 12, 16, 21], "subsubset": 9, "subtask": 6, "subtl": 1, "subtract": [0, 4, 5, 6, 11, 13, 18, 21, 22, 24, 28, 30], "subtre": 9, "succeed": [0, 4, 27], "success": [3, 7, 9, 13, 24], "successfulli": [4, 9], "sudo": [0, 20, 22, 27], "suffer": [0, 1, 2, 5, 10, 27, 28, 29], "suffici": [1, 6, 8, 11, 13, 29], "suggest": [1, 13, 22, 26, 29, 30], "suit": [8, 12], "suitabl": [0, 15, 19, 24, 28], "sum": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 19, 21, 24, 27, 28, 29, 30], "sum_": [0, 1, 2, 3, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 16, 19, 21, 22, 24, 27, 28, 29, 30], "sum_i": [0, 2, 5, 6, 8, 13, 19, 22, 28, 29, 30], "sum_j": [6, 18, 30], "sum_ja_": 0, "sum_k": [6, 8, 12, 21], "sum_logist": 13, "sum_m": 3, "sum_n": 3, "sum_nx_": 3, "summar": [5, 6, 9], "summari": [1, 3, 4, 10, 23, 29], "summat": [0, 3, 16, 28, 29], "sunni": 9, "super": [5, 28, 29], "superfici": 3, "superscript": [1, 12], "supervis": [0, 5, 6, 7, 9, 12, 20, 27, 28, 29], "supplement": [7, 22], "support": [0, 1, 9, 10, 11, 13, 20, 27, 28], "suppos": [0, 5, 6, 7, 8, 10, 11, 12, 13, 21, 27, 28, 29, 30], "suppress": [5, 13, 29], "sure": [0, 1, 4, 6, 16, 22], "surf": 6, "surfac": [0, 6, 27], "surpass": 6, "surpris": [0, 27], "surround": [3, 20], "survei": [0, 5, 6, 27, 28], "svc": [8, 9, 10], "svd": [0, 6, 11, 27], "svdinv": 5, "svm": [8, 9, 10, 11], "svm_clf": [8, 10], "swath": [5, 28, 29], "switch": 0, "sy": [13, 29, 30], "symbol": [1, 5, 11, 13, 20, 24, 27, 28, 29], "symmeteri": 1, "symmetr": [0, 5, 8, 11, 12, 13, 21, 27, 28], "symmetri": 6, "sympi": [0, 20, 22, 27], "synonim": 24, "syntax": 13, "system": [0, 1, 3, 4, 6, 7, 9, 10, 12, 13, 15, 20, 21, 22, 27, 29], "systemat": [4, 6], "t": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 24, 25, 27, 29, 30], "t0": [3, 6, 13, 30], "t1": [2, 13, 30], "t2": 2, "t3": 2, "t_": 2, "t_0": [2, 9, 13, 30], "t_1": [13, 30], "t_b": 10, "t_i": [1, 2, 5, 12, 28, 29], "t_j": 12, "t_k": 9, "tabl": [9, 22, 24, 25, 27], "tabul": [0, 27], "tabular": 27, "tackl": 4, "tag": [2, 3, 4, 5, 6, 7, 12, 13, 14, 21, 24, 28, 29], "taht": [0, 27], "tail": 24, "tailor": [2, 8, 11, 27], "taiwan": [0, 27], "take": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 17, 20, 21, 24, 27, 28, 29, 30], "taken": [0, 1, 3, 6, 10, 13, 21], "tan": 3, "tangent": [1, 4, 12, 13, 29], "tanh": [1, 4, 7, 8, 12], "target": [0, 1, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 15, 16, 18, 19, 27, 28, 29, 30], "target_nam": 9, "task": [0, 1, 3, 6, 9, 11, 12, 14, 22, 27], "tau": [3, 5, 24], "taught": 27, "tax": [], "taylor": [2, 13, 29], "taylornr": [13, 29], "tc": 8, "teach": [15, 23, 27], "team": 1, "teaser": 0, "technic": [0, 5, 6, 13, 22, 29, 30], "techniqu": [0, 1, 8, 10, 13, 20, 24, 26, 27, 28, 30], "technologi": [0, 1], "tell": [0, 4, 6, 10, 11, 13, 16, 24, 30], "temp": 1, "temp1": 1, "temp2": 1, "temperatur": [0, 9, 27], "templat": 18, "temporarili": 1, "ten": [3, 27], "tend": [3, 5, 6, 8, 9, 10, 12, 13, 14, 28, 30], "tendenc": [0, 27], "tension": 6, "tensor": 3, "tensorflow": [0, 2, 4, 8, 14, 20, 21, 22, 26, 27, 28], "term": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 16, 18, 19, 22, 24, 27, 28, 29, 30], "term1": [5, 6, 11], "term2": [5, 6, 11], "term3": [5, 6, 11], "term4": [5, 6, 11], "termin": [0, 4, 5, 9, 10, 13, 15, 28, 29, 30], "terminarl": 15, "terrain": 6, "terrain1": 6, "test": [3, 4, 5, 6, 7, 8, 9, 10, 13, 16, 21, 22, 24, 27, 29, 30], "test_acc": 3, "test_accuraci": [1, 3], "test_error": 6, "test_imag": [3, 4], "test_ind": 6, "test_input": 4, "test_label": [3, 4], "test_loss": 3, "test_pr": 1, "test_predict": 1, "test_rnn": 4, "test_scor": [7, 10], "test_siz": [0, 1, 3, 5, 6, 10, 15, 17, 28, 29, 30], "test_split": 9, "testerror": [0, 6, 28], "testi": 4, "testpredict": 4, "testx": 4, "text": [0, 1, 2, 4, 5, 8, 9, 11, 13, 15, 18, 21, 24, 26, 28, 29, 30], "textbf": [], "textbook": [16, 22, 28, 29], "textual": 9, "textur": 1, "tf": [1, 3, 4, 13, 14, 29], "th": [0, 1, 2, 5, 6, 7, 9, 12, 13, 14, 21, 22, 24, 27, 28, 30], "than": [0, 1, 2, 3, 4, 5, 6, 7, 9, 10, 11, 12, 13, 17, 20, 24, 27, 28, 30], "thank": [4, 6, 28, 30], "theano": [1, 20, 27], "thei": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 11, 12, 13, 15, 16, 18, 21, 22, 24, 27, 28, 29, 30], "them": [0, 1, 3, 4, 6, 8, 9, 10, 11, 12, 13, 18, 19, 21, 22, 27, 28], "theme": [0, 15, 27], "themselv": [0, 22, 24, 27], "thenc": 6, "theorem": [2, 6, 7, 28, 29], "theoret": [0, 4, 10], "theori": [0, 1, 3, 8, 9, 12, 13, 19, 20, 22, 26, 27], "thereaft": [0, 5, 6, 11, 12, 21, 22, 27], "therebi": [0, 5, 7, 11, 22, 27, 28, 29], "therefor": [0, 1, 2, 3, 4, 6, 7, 8, 11, 13, 19, 24, 27, 28, 29, 30], "therein": 11, "thereof": [0, 6, 13, 27, 30], "theta": [0, 1, 4, 5, 6, 7, 13, 16, 22, 24, 27, 28, 29, 30], "theta1": 30, "theta2": 30, "theta_": [0, 1, 6, 7, 13, 27, 28, 29, 30], "theta_0": [0, 5, 6, 7, 16, 27, 28, 29, 30], "theta_0x_": [0, 27, 28], "theta_1": [0, 5, 6, 7, 27, 28, 29, 30], "theta_1x_": [0, 27, 28], "theta_1x_0": [0, 27], "theta_1x_1": [0, 7, 27], "theta_1x_2": [0, 27], "theta_1x_i": [7, 28, 29, 30], "theta_2": [0, 27, 28], "theta_2x_": [0, 27, 28], "theta_2x_0": [0, 27], "theta_2x_1": [0, 27], "theta_2x_2": [0, 7, 27], "theta_2x_i": 28, "theta_3x_i": 28, "theta_4x_i": 28, "theta_closed_form": 18, "theta_closed_formol": 18, "theta_closed_formridg": 18, "theta_gdol": 18, "theta_gdridg": 18, "theta_i": [0, 1, 5, 27, 28, 29], "theta_j": [0, 5, 6, 18, 27, 28, 30], "theta_k": [29, 30], "theta_linreg": [13, 29, 30], "theta_ol": 18, "theta_p": 7, "theta_px_p": 7, "theta_ridg": 18, "theta_t": [13, 30], "theta_tru": 18, "thetavalu": 5, "thi": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 26, 28, 29, 30], "thing": [0, 1, 2, 4, 5, 7, 9, 15, 16, 18, 24, 27], "think": [0, 1, 3, 4, 6, 9, 12, 13, 14, 24, 27, 28, 29, 30], "third": [0, 3, 6, 13, 25, 27, 29, 30], "thirti": 7, "thorughout": 27, "those": [0, 3, 5, 6, 8, 9, 10, 11, 21, 22, 27, 28, 29, 30], "though": [1, 2, 3, 4, 13, 16, 17, 21, 24, 30], "thought": [6, 14, 22, 24], "thousand": [0, 1, 22, 28, 30], "three": [0, 1, 3, 5, 6, 8, 9, 12, 21, 22, 23, 24, 25, 27, 28, 29], "threshold": [1, 3, 9, 10, 11, 12, 13, 30], "through": [0, 1, 2, 3, 4, 5, 6, 8, 11, 12, 13, 14, 15, 20, 21, 22, 24, 27, 28, 29, 30], "throughout": [0, 4, 5, 14, 15, 20, 21, 24, 27], "throw": [3, 6, 24], "thu": [0, 1, 2, 5, 6, 7, 8, 10, 11, 12, 13, 25, 27, 28, 29, 30], "thumb": [0, 6, 22, 28], "thursdai": [], "tibshirani": [6, 19, 22, 26, 27], "tick_param": 6, "ticker": [6, 13, 24, 29, 30], "tif": 6, "tight_layout": [1, 7], "tightli": 11, "tild": [0, 5, 6, 7, 11, 19, 22, 24, 27, 28, 29, 30], "till": [0, 4, 7, 8, 9, 10, 12, 21, 27, 28], "time": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 20, 21, 22, 24, 27, 28, 29], "timeit": 4, "timer": 4, "tini": 1, "tip": 3, "titl": [0, 1, 2, 3, 4, 6, 7, 8, 9, 10, 13, 15, 24, 27, 29, 30], "tmp": 13, "tn": [2, 3, 7], "to_categor": [1, 3, 4], "to_categorical_numpi": 1, "to_numer": [0, 6, 27], "todai": 3, "togeth": [0, 3, 6, 8, 11, 13, 20, 27], "toi": 14, "told": 13, "toler": [2, 14], "tolist": 4, "tomographi": 12, "too": [0, 2, 4, 5, 6, 9, 11, 13, 17, 18, 24, 26, 28, 29, 30], "took": [8, 27], "tool": [0, 1, 3, 6, 13, 15, 20, 28], "toolbox": 8, "top": [0, 3, 5, 6, 9, 10, 19, 20, 27], "topic": [0, 5, 6, 7, 8, 20, 22, 28, 29], "topolog": [3, 12], "topologi": [1, 12], "torkjellsdatt": [25, 27], "toss": [10, 24], "total": [0, 1, 2, 3, 4, 6, 7, 8, 10, 11, 12, 13, 14, 21, 24, 25, 27, 28, 29, 30], "total_loss": 4, "totalclustervari": 14, "totalscatt": 14, "toward": [1, 2, 7, 12, 13, 15, 29], "towardsdatasci": 30, "town": [], "tp": [4, 7], "tpng": 9, "tpu": [13, 20, 27], "tqdm": 6, "tr": [], "track": [3, 13, 14, 15, 21, 28, 29, 30], "tract": [], "tractabl": [0, 27, 28], "trade": [5, 9], "tradeoff": [0, 5, 19, 22, 27, 28, 29], "tradit": [0, 1, 4, 6, 27], "train": [2, 3, 5, 6, 8, 9, 10, 11, 12, 13, 16, 17, 22, 29, 30], "train_accuraci": [0, 1, 3, 27], "train_dataset": 4, "train_end": [0, 1, 28], "train_error": 6, "train_imag": [3, 4], "train_ind": 6, "train_label": [3, 4], "train_pr": 1, "train_siz": [0, 1, 3, 28], "train_step": 4, "train_test_split": [0, 1, 3, 5, 6, 7, 9, 10, 11, 15, 16, 17, 19, 27, 28, 29, 30], "train_test_split_numpi": [0, 1, 28], "trainable_vari": 4, "trained_model": [6, 28, 30], "trainerror": [0, 28], "traini": 4, "training_checkpoint": 4, "training_dataset": 4, "training_gradi": [13, 30], "trainingerror": 6, "trainpredict": 4, "trainscor": 4, "trainx": 4, "trait": [0, 27], "trajectori": 4, "transfer": [9, 27], "transform": [0, 5, 6, 7, 8, 9, 10, 11, 12, 13, 17, 20, 21, 27, 28, 29, 30], "transit": [6, 12], "translat": [1, 4, 6, 10, 27, 28, 30], "transpos": [1, 5, 11, 21, 28, 29], "travers": [0, 5], "treat": [0, 1, 3, 6, 12, 13, 18, 24, 27, 28, 29, 30], "tree": [0, 1, 20, 27], "tree_clf": [9, 10], "tree_clf_": 9, "tree_clf_sr": 9, "tree_reg": 9, "tree_reg1": 9, "tree_reg2": 9, "trend": 24, "treue": 7, "trevor": [19, 22, 26], "tri": [2, 3, 4, 9, 13, 16], "triain": 0, "trial": [0, 2, 4, 6, 13, 24, 27, 29, 30], "triangl": [13, 29], "triangular": 21, "trick": [3, 4, 8, 11, 13, 24, 30], "trickier": 24, "tridiagon": 21, "trillion": 20, "trivial": [0, 1, 5, 11, 24, 27, 29], "troubl": [0, 8, 12, 15, 28, 30], "truck": 3, "true": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 12, 13, 14, 16, 17, 18, 19, 21, 22, 24, 27, 28, 29, 30], "true_beta": 28, "true_fun": 6, "true_theta": [6, 30], "truli": 27, "try": [0, 1, 2, 4, 5, 6, 7, 8, 9, 10, 11, 13, 14, 15, 18, 20, 21, 22, 24, 27, 28, 29, 30], "tucker": 8, "tuesdai": [25, 27], "tumor": [7, 9], "tumour": 7, "tunabl": 1, "tune": [4, 9, 13, 21, 27, 30], "turn": [0, 1, 5, 6, 7, 8, 9, 10, 11, 12, 13, 21, 22, 24, 27, 28, 29, 30], "tutori": [1, 4], "tv": 2, "tveito": 2, "tweak": [1, 4, 10, 24], "twice": [13, 29], "twist": 11, "two": [0, 1, 2, 4, 5, 6, 7, 9, 10, 11, 12, 13, 15, 17, 21, 22, 23, 24, 26, 27, 28, 29, 30], "tx": [13, 29, 30], "tx_1": [13, 29], "txt": [4, 15], "ty": [13, 29], "type": [0, 1, 3, 6, 8, 10, 13, 21, 24, 28, 29, 30], "typic": [0, 1, 2, 3, 4, 5, 7, 9, 10, 12, 13, 15, 16, 24, 27, 28, 29, 30], "typo": 22, "u": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 21, 22, 24, 26, 27, 28, 29, 30], "u_": 21, "u_i": 12, "u_m": 10, "ua": [0, 27], "ubuntu": [0, 20, 22, 27], "uci": 22, "uio": [15, 22, 25, 26], "un": 14, "unabl": 15, "unari": [21, 27], "unbalanc": [6, 9], "unbias": [0, 5, 6, 27], "uncent": [6, 28, 30], "uncertainti": [0, 5, 27], "uncertitud": 24, "unchang": [1, 3], "uncorrel": [10, 24], "undefin": [5, 28, 29], "under": [0, 1, 5, 6, 10, 13, 20, 22, 27, 28, 29], "underdetermin": [0, 27], "underfit": [1, 6], "underflowproblem": 5, "undergo": 5, "undergradu": [23, 25], "underli": [0, 1, 9, 13, 18, 24, 27, 30], "underset": [4, 14], "understand": [0, 1, 3, 5, 6, 10, 13, 14, 15, 20, 27, 28, 29, 30], "understood": [8, 13], "undesir": 8, "undetermin": [5, 8], "undo": 4, "unexpect": 6, "unexpected": 24, "unexplain": 18, "unfair": [6, 28], "unfortun": [1, 8, 9, 10], "unicode_liter": [8, 9], "uniform": [0, 1, 5, 6, 11, 13, 22, 24, 27, 29, 30], "uniformli": [13, 24, 29, 30], "unifrompdf": 24, "unimport": [13, 29], "union": [5, 6], "uniqu": [0, 2, 6, 13, 14, 21, 27], "unique_cluster_label": 14, "unit": [0, 1, 3, 4, 5, 10, 12, 18, 24, 27, 28, 29, 30], "unitari": [5, 6, 21, 28, 29], "unitarili": [21, 27], "uniti": 24, "univari": 24, "univers": [0, 1, 2, 13, 20, 22, 23, 25, 27, 28, 29, 30], "unix": 1, "unknow": [0, 21, 27], "unknown": [0, 1, 3, 4, 5, 6, 8, 10, 13, 21, 22, 27, 28, 29, 30], "unknowwn": 12, "unlabel": 1, "unless": [0, 3, 6, 11, 13, 22, 27, 29], "unlik": [1, 3, 8, 13, 29, 30], "unnecessarili": 9, "unord": 3, "unpublish": 30, "unravel": 1, "unrol": [3, 11], "unseen": [0, 7, 9, 15], "unstabl": 1, "unsupervis": [0, 1, 4, 12, 20, 27], "unsymmetr": [21, 27], "until": [1, 2, 4, 9, 12, 13, 14, 29], "untouch": 0, "unusu": 12, "up": [1, 3, 4, 5, 6, 8, 10, 11, 13, 14, 16, 18, 19, 20, 21, 22, 24, 25, 30], "updat": [1, 2, 10, 12, 13, 14, 15, 18], "uploa": 27, "upload": [15, 20, 22, 26], "upon": [0, 1, 6, 7, 11, 21], "upper": [0, 8, 9, 16, 21, 28], "uppercas": [21, 27], "upsampl": 4, "upscal": 4, "url": [27, 28], "us": [4, 5, 6, 8, 9, 10, 11, 12, 14, 15, 17, 19, 21, 24, 26], "usag": [0, 8, 20, 27, 28, 30], "usd": [], "usd10000": [], "use_bia": 4, "usecol": [0, 27], "useless": 1, "user": [0, 1, 2, 4, 6, 7, 15, 20, 21, 22, 27, 28], "usernam": [15, 22], "usetex": 24, "usg": 6, "usr": 24, "usual": [0, 3, 4, 7, 12, 13, 14, 27, 30], "ut": 5, "util": [1, 3, 4, 6, 7, 10, 14, 19, 27], "ux": 21, "v": [2, 4, 5, 6, 11, 13, 15, 20, 28, 29], "v0": 24, "v1": 24, "v2": 24, "v_": 30, "v_0": [11, 30], "v_t": 30, "va": 1, "vahid": 27, "val": 13, "val_accuraci": 3, "val_loss": 4, "vale": 2, "valid": [0, 1, 4, 7, 9, 10, 13, 20, 24, 27, 28, 30], "validation_data": 3, "validation_split": 4, "valu": [0, 1, 2, 3, 4, 6, 7, 8, 9, 10, 12, 13, 14, 16, 17, 18, 20, 21, 22, 27, 30], "valuat": 9, "valy": 4, "van": [0, 19, 22, 27, 28, 29, 30], "vandenbergh": [8, 13, 29], "vandermond": [0, 27], "vanilla": [0, 6, 11, 14, 28, 30], "vanish": [1, 4, 13, 24, 29], "var": [5, 6, 10, 11, 19, 22, 24, 28], "var_x": 24, "varabl": 8, "varepsilon": [5, 6, 19], "varepsilon_": [5, 6], "varepsilon_i": [5, 6], "vari": [0, 1, 3, 5, 6, 10, 27], "variabl": [0, 1, 2, 5, 6, 7, 8, 10, 11, 12, 13, 14, 21, 27, 28, 30], "varianc": [0, 1, 5, 7, 9, 10, 11, 13, 14, 18, 20, 21, 24, 27, 28, 29, 30], "variance_i": [5, 11, 28], "variance_x": [5, 11, 28], "variant": [0, 1, 6, 8, 12, 13, 27, 28, 29, 30], "variat": [3, 4, 11, 27], "varieti": [0, 3, 12, 20, 22, 27], "variou": [1, 3, 5, 6, 7, 8, 9, 11, 12, 13, 16, 19, 20, 21, 22, 24, 27, 28, 29, 30], "varydimens": 4, "vastli": 3, "vaue": 1, "vault": 0, "vdot": [2, 13, 29, 30], "ve": [22, 30], "vec": 6, "vector": [0, 1, 2, 3, 4, 5, 6, 7, 9, 10, 11, 13, 14, 17, 18, 20, 29, 30], "vector_mean": 14, "ventur": [0, 8, 20, 27], "venv": 15, "verbos": [1, 3, 4], "veri": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 16, 18, 22, 24, 26, 27, 28, 29, 30], "verifi": [3, 11, 21, 27], "versatil": [8, 27], "versicolor": [8, 9], "version": [0, 3, 10, 13, 14, 15, 20, 21, 22, 24, 27], "versu": 1, "vert": [0, 1, 5, 6, 7, 8, 9, 11, 13, 16, 17, 27, 28, 29, 30], "vert_1": [5, 6, 28, 29, 30], "vert_2": [5, 6, 11, 17, 28, 29, 30], "via": [0, 5, 6, 7, 8, 9, 10, 11, 12, 19, 20, 21, 22, 23, 24, 25, 27, 28, 29, 30], "vidal": 11, "video": [0, 1, 12, 20, 23, 25, 27, 28, 29], "view": [1, 3, 5, 6, 12, 13, 24, 26, 27, 29, 30], "violat": 8, "virginica": 9, "viridi": [0, 1, 2, 3, 27], "virtual": 1, "viscos": 13, "viscou": 13, "visibl": 15, "vision": [0, 3], "visual": [0, 3, 11, 12, 18, 20, 27, 28], "visualis": 1, "visualstudio": [15, 16], "viz": [6, 8, 24], "vmap": 13, "vmax": [1, 6], "vmh0zpt0tli": 30, "vmin": [1, 6], "voic": 3, "volatil": 30, "volum": [0, 3, 27], "vote": [10, 27], "voting_clf": 10, "votingclassifi": 10, "votingsimpl": 10, "vstack": [5, 11, 21, 24, 27, 28], "vt": [5, 28, 29], "w": [0, 1, 2, 3, 4, 5, 6, 7, 8, 10, 11, 12, 13, 14, 21, 24, 27, 28, 29, 30], "w1": 8, "w2": [8, 11], "w3": 8, "w_": [1, 12], "w_1": [8, 21], "w_1x_": 8, "w_1x_1": 8, "w_2": [8, 21], "w_2x_": 8, "w_2x_2": 8, "w_3": 21, "w_4": 21, "w_hidden": 2, "w_i": [1, 2, 10], "w_ix_i": 12, "w_j": 21, "w_m": 21, "w_output": 2, "w_px_": 8, "w_px_p": 8, "w_t": [], "wa": [1, 3, 4, 5, 6, 7, 10, 11, 12, 14, 17, 21, 27, 28, 30], "wai": [0, 1, 2, 3, 4, 5, 6, 7, 8, 10, 11, 12, 13, 14, 15, 18, 19, 21, 24, 27, 28, 29, 30], "walk": 9, "walker": 24, "wang": [0, 27], "want": [0, 1, 2, 3, 4, 5, 6, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 20, 22, 24, 27, 28, 29, 30], "warn": 4, "warrant": 6, "wast": [3, 30], "watch": [20, 29, 30], "wave": 3, "wavelet": 8, "we": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 24, 25, 26, 28, 29], "weak": [9, 10, 14], "weather": [1, 12], "web": [20, 23, 25, 27], "webpag": 27, "websit": [6, 21, 22, 23, 27], "wedg": [8, 24], "wednesdai": [25, 27], "wee": 11, "week": [0, 5, 6, 7, 22, 23, 25], "weekli": [15, 16, 20, 22, 23, 25, 26, 27], "weight": [1, 2, 3, 6, 7, 9, 10, 12, 13, 18, 24, 30], "weigth": 2, "welcom": [8, 15, 20], "well": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 12, 13, 15, 16, 20, 21, 22, 24, 26, 27, 28, 29, 30], "went": 8, "were": [0, 1, 3, 4, 5, 6, 7, 8, 10, 11, 12, 14, 24, 27], "wessel": [0, 19, 22, 27, 28, 29, 30], "what": [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 19, 20, 21, 22, 24, 30], "whatev": 3, "when": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 18, 19, 21, 22, 24, 27, 28, 29], "whenev": [13, 15, 24, 30], "where": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 20, 21, 22, 24, 25, 27, 28, 29, 30], "wherea": [6, 24, 30], "wherefrom": 22, "wherein": [1, 12], "whether": [0, 3, 5, 7, 9, 22, 24, 27], "which": [0, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 27, 28, 29], "whichev": [1, 3], "while": [0, 1, 3, 4, 5, 6, 7, 8, 9, 11, 12, 13, 15, 16, 19, 24, 27, 28, 29], "white": 9, "whiteboad": 30, "whiteboard": [28, 29], "who": [0, 15], "whole": [1, 3, 4, 5, 9, 11, 13, 30], "whose": [0, 6, 10, 24, 28], "whow": [11, 28], "why": [0, 1, 3, 6, 13, 15, 16, 17, 19, 22, 28, 29], "wide": [0, 1, 3, 6, 7, 12, 20, 21, 22, 27], "widehat": 6, "width": [0, 3, 8, 9, 27], "wieringen": [0, 19, 22, 27, 28, 29, 30], "wiki": 22, "wikipedia": 22, "win": 10, "wind": 9, "wing": [25, 27], "winther": 2, "wiothout": 6, "wiscons": 7, "wisconsin": 10, "wisdom": [6, 28, 30], "wise": [1, 5, 12, 13, 28, 29, 30], "wish": [0, 2, 5, 7, 8, 11, 13, 14, 18, 21, 22, 27, 28, 29, 30], "with_std": [0, 28], "wither": 6, "within": [0, 2, 3, 4, 7, 9, 12, 13, 14, 24, 26, 27, 29], "withinclust": 14, "without": [0, 1, 5, 6, 8, 9, 11, 12, 13, 15, 18, 22, 27, 28, 29, 30], "won": [0, 15, 27], "wonder": 8, "word": [0, 1, 3, 4, 5, 6, 7, 14, 24, 27, 28, 29, 30], "work": [0, 1, 4, 6, 7, 8, 9, 13, 15, 16, 18, 20, 22, 23, 24, 25, 27, 28, 30], "workabl": 30, "workshop": 27, "world": [0, 8, 16, 28], "worldwid": [0, 27], "worri": 15, "wors": [0, 1, 3, 4, 6, 27], "worth": 9, "worthi": 22, "would": [0, 1, 3, 5, 6, 7, 8, 9, 10, 11, 12, 13, 16, 18, 21, 22, 24, 27, 28, 29, 30], "wrap": [6, 21, 27], "write": [0, 1, 2, 3, 5, 6, 7, 8, 12, 13, 15, 16, 18, 21, 27, 28, 30], "written": [0, 2, 3, 5, 11, 12, 13, 16, 20, 21, 22, 24, 27, 28, 29, 30], "wrong": [1, 8, 15], "wrongli": 10, "wrote": [5, 11, 28], "wrt": [10, 13, 30], "wth": [10, 13, 30], "www": [20, 21, 22, 26, 27, 29, 30], "wx_1": 8, "x": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 21, 22, 24, 27, 29, 30], "x0": 8, "x1": [4, 8, 9, 10, 13], "x1_exampl": 8, "x1d": 8, "x2": [8, 9, 10, 13], "x2d": [8, 11], "x2d_train": 11, "x2dsl": 11, "x3": 8, "x_": [0, 2, 3, 5, 6, 8, 10, 11, 13, 14, 21, 24, 27, 28, 29, 30], "x_0": [0, 5, 11, 18, 21, 27, 28], "x_1": [0, 2, 5, 6, 7, 8, 9, 10, 11, 13, 18, 21, 24, 27, 28, 29, 30], "x_2": [0, 2, 5, 6, 7, 8, 9, 10, 11, 13, 21, 24, 27, 28, 29], "x_3": [8, 21, 24], "x_4": 21, "x_6": 18, "x_center": 11, "x_data": 1, "x_data_ful": 1, "x_hidden": 2, "x_i": [0, 1, 2, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 21, 24, 27, 28, 29, 30], "x_input": 2, "x_ix_": [0, 27], "x_iy_i": 8, "x_j": [0, 2, 8, 9, 12, 16, 24, 28, 30], "x_jy_j": 8, "x_k": [12, 14, 21, 24, 28], "x_l": 24, "x_m": [6, 12, 21, 24], "x_mean": [18, 30], "x_n": [0, 2, 3, 6, 8, 11, 12, 13, 21, 24, 27, 29], "x_new": [9, 10], "x_norm": [18, 30], "x_offset": [6, 28, 30], "x_output": 2, "x_p": [3, 7, 9], "x_poli": 9, "x_poly10": 9, "x_pred": 4, "x_prev": 2, "x_reduc": 11, "x_sampl": 19, "x_scale": 8, "x_small": 13, "x_std": [18, 30], "x_t": 30, "x_test": [0, 1, 3, 5, 6, 7, 9, 10, 11, 15, 16, 17, 19, 28, 29, 30], "x_test_": 17, "x_test_own": 6, "x_test_scal": [0, 6, 7, 9, 10, 11, 28, 30], "x_tot": 4, "x_train": [0, 1, 3, 4, 5, 6, 7, 9, 10, 11, 15, 16, 17, 19, 27, 28, 29, 30], "x_train_": 17, "x_train_mean": [6, 28, 30], "x_train_own": 6, "x_train_scal": [0, 6, 7, 9, 10, 11, 28, 30], "x_val": 1, "xarrai": [20, 27], "xavier": 1, "xbnew": [13, 29, 30], "xcode": [0, 20, 22, 27], "xdclassiffierconfus": 10, "xdclassiffierroc": 10, "xg_clf": 10, "xgb": 10, "xgbclassifi": 10, "xgboost": 9, "xgboot": 10, "xgbregressor": 10, "xgparam": 10, "xgtree": 10, "xi": [8, 13, 30], "xi_": 8, "xi_1": 8, "xi_i": 8, "xk": 8, "xla": [13, 20, 27], "xlabel": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 13, 24, 27, 28, 29, 30], "xlim": [6, 10], "xm": 9, "xmesh": 13, "xnew": [0, 13, 27, 29, 30], "xp": 24, "xpanda": [0, 28], "xpd": [5, 11, 28], "xplot": 0, "xscale": [0, 28], "xsr": 9, "xt_x": [13, 29, 30], "xtest": 6, "xtick": [3, 6, 8, 9], "xtrain": 6, "xu": [0, 27], "xx": [0, 21, 27], "xy": [0, 6, 8, 21, 27], "xytext": 8, "xz": [21, 27], "y": [0, 1, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 21, 22, 24, 27, 28, 29, 30], "y1": 4, "y2": 4, "y3": 4, "y_": [0, 1, 5, 6, 10, 11, 21, 27, 28], "y_0": [0, 5, 11, 21, 27, 28], "y_1": [0, 5, 8, 9, 11, 13, 21, 27, 28, 29, 30], "y_1y_1": 8, "y_1y_1k": 8, "y_1y_2": 8, "y_1y_2k": 8, "y_1y_n": 8, "y_1y_nk": 8, "y_2": [0, 5, 8, 9, 11, 21, 27, 28], "y_2y_1": 8, "y_2y_1k": 8, "y_2y_2": 8, "y_2y_2k": 8, "y_3": [0, 9, 21], "y_4": 21, "y_center": [18, 30], "y_data": [0, 1, 5, 6, 27, 28, 29, 30], "y_data_ful": 1, "y_decis": 8, "y_fit": [0, 28], "y_i": [0, 1, 5, 6, 7, 8, 9, 10, 11, 12, 13, 19, 21, 22, 27, 28, 29, 30], "y_if_": 10, "y_ix_": [0, 27], "y_ix_i": [7, 8, 13, 28, 29, 30], "y_iy_jk": 8, "y_j": [6, 8, 12, 22], "y_k": 12, "y_m": 21, "y_mean": [18, 30], "y_model": [0, 4, 5, 6, 27, 28, 29, 30], "y_n": [8, 13, 29, 30], "y_ny_1": 8, "y_ny_1k": 8, "y_ny_2": 8, "y_ny_2k": 8, "y_ny_n": 8, "y_ny_nk": 8, "y_offset": [6, 17, 28, 30], "y_plot": 9, "y_pred": [0, 1, 4, 6, 7, 8, 9, 10, 28, 30], "y_pred1": 9, "y_pred2": 9, "y_pred_rf": 10, "y_pred_tre": 10, "y_proba": [7, 10], "y_sampl": 19, "y_scaler": [6, 28, 30], "y_test": [0, 1, 3, 4, 5, 6, 7, 9, 10, 11, 15, 16, 17, 19, 28, 29, 30], "y_test_onehot": 1, "y_test_predict": [], "y_tot": 4, "y_train": [0, 1, 3, 4, 5, 6, 7, 9, 10, 11, 15, 16, 17, 19, 27, 28, 29, 30], "y_train_mean": [6, 28, 30], "y_train_onehot": 1, "y_train_predict": [], "y_train_scal": [6, 28, 30], "y_val": 1, "ye": [3, 6, 7], "year": [0, 20, 27], "yet": [0, 1, 6, 8, 11, 13, 27], "yi": [13, 30], "yield": [0, 2, 5, 6, 8, 10, 12, 13, 14, 21, 24, 27, 29, 30], "yk": 8, "ylabel": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 13, 24, 27, 28, 29, 30], "ylim": [3, 6], "ym": 9, "ymesh": 13, "yn": 0, "yo": [8, 9, 10], "yoshua": [1, 26], "you": [0, 1, 3, 4, 5, 6, 8, 9, 10, 11, 13, 15, 16, 17, 18, 19, 20, 21, 22, 24, 25, 26, 27, 28, 29, 30], "young": 0, "your": [1, 2, 4, 5, 6, 8, 11, 13, 15, 17, 19, 20, 21, 27, 29, 30], "your_model_object": 16, "yourself": [11, 13, 27, 29], "youtu": [28, 29], "youtub": [20, 29, 30], "ypred": 6, "ypredict": [0, 13, 27, 28, 29, 30], "ypredict2": [13, 29, 30], "ypredictlasso": [5, 29], "ypredictol": [0, 5, 29], "ypredictown": [6, 28, 30], "ypredictownridg": [6, 28, 29, 30], "ypredictridg": [0, 5, 6, 28, 29, 30], "ypredictskl": [6, 28, 30], "ytest": 6, "ytick": [3, 6, 8, 9], "ytild": [0, 6, 27, 28], "ytildelasso": [5, 29], "ytildenp": [0, 27, 28], "ytildeol": [0, 5, 29], "ytildeownridg": [6, 28, 29, 30], "ytilderidg": [5, 6, 28, 29, 30], "ytrain": 6, "yuxi": 27, "yx": [21, 27], "yy": [21, 27], "yz": [21, 27], "z": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 11, 12, 13, 21, 24, 27, 28], "z_": [1, 2, 12, 21, 27], "z_0": [21, 27], "z_1": [21, 27], "z_2": [21, 27], "z_c": 1, "z_h": 1, "z_hidden": 2, "z_i": [1, 12], "z_j": [1, 12], "z_k": [12, 28], "z_m": 1, "z_mod": 9, "z_o": 1, "z_output": 2, "zaman": 24, "zaxi": 6, "zero": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 16, 17, 18, 19, 21, 22, 24, 27, 28, 29, 30], "zeros_lik": 4, "zeroth": 28, "zfill": 4, "zip": [4, 6], "zm_h": [0, 27], "zn": [], "zone": [], "zoom": 27, "zx": [21, 27], "zy": [21, 27], "zz": [21, 27], "\u00f8yvind": [6, 28, 30]}, "titles": ["3. Linear Regression", "14. Building a Feed Forward Neural Network", "15. Solving Differential Equations with Deep Learning", "16. Convolutional Neural Networks", "17. Recurrent neural networks: Overarching view", "4. Ridge and Lasso Regression", "5. Resampling Methods", "6. Logistic Regression", "8. Support Vector Machines, overarching aims", "9. Decision trees, overarching aims", "10. Ensemble Methods: From a Single Tree to Many Trees and Extreme Boosting, Meet the Jungle of Methods", "11. Basic ideas of the Principal Component Analysis (PCA)", "13. Neural networks", "7. Optimization, the central part of any Machine Learning algortithm", "12. Clustering and Unsupervised Learning", "Exercises week 34", "Exercises week 35", "Exercises week 36", "Exercises week 37", "Exercises week 37", "Applied Data Analysis and Machine Learning", "2. Linear Algebra, Handling of Arrays and more Python Features", "Project 1 on Machine Learning, deadline October 6 (midnight), 2025", "Course setting", "1. Elements of Probability Theory and Statistical Data Analysis", "Teachers and Grading", "Textbooks", "Week 34: Introduction to the course, Logistics and Practicalities", "Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression", "Week 36: Linear Regression and Gradient descent", "Week 37: Statistical interpretations and Resampling Methods"], "titleterms": {"": [8, 10, 29, 30], "1": [0, 15, 16, 17, 18, 19, 22, 28], "15": 19, "19": 19, "1a": 18, "2": [0, 15, 16, 17, 18, 19, 27, 28, 29], "2023": 25, "2025": 22, "2a": [], "2b": [], "3": [0, 15, 16, 17, 18, 19, 28], "34": [15, 27], "35": [16, 28], "36": [17, 29], "37": [18, 19, 30], "3a": 18, "3b": 18, "4": [0, 15, 16, 17, 18, 19, 28], "4a": 18, "4b": 18, "5": [0, 16, 18], "6": 22, "8": 30, "A": [0, 1, 4, 8, 9, 27], "And": [27, 28, 30], "But": 30, "In": 25, "Ising": 6, "The": [0, 1, 2, 3, 5, 6, 7, 8, 9, 11, 12, 15, 20, 27, 28, 29, 30], "To": 27, "With": [4, 29], "about": [27, 28, 29], "abov": 29, "accuraci": 30, "across": 30, "activ": [1, 12], "ad": [0, 6, 22, 27, 28], "adaboost": 10, "adagrad": [13, 30], "adam": [13, 30], "adapt": [10, 30], "adjust": 1, "advanc": 22, "adversari": 4, "again": [3, 9], "ai": [22, 27], "aim": [8, 9, 27], "aka": 27, "al": 30, "algebra": [21, 27], "algorithm": [9, 10, 11, 12, 27, 28, 29, 30], "algortithm": [13, 29], "all": 8, "an": [0, 4, 10, 15, 27], "analys": [5, 28, 29], "analysi": [0, 5, 6, 11, 20, 22, 24, 27, 28, 29], "analyt": [0, 16, 18], "ani": [13, 29], "anoth": [9, 29], "appli": 20, "approach": [0, 8, 14, 27, 30], "approxim": 12, "architectur": 1, "arrai": [21, 27], "assist": 25, "autocorrel": 24, "autograd": [2, 13, 30], "automat": [13, 30], "averag": 30, "b": 22, "back": [1, 11, 12, 28, 29], "background": [20, 22], "bag": 10, "base": [13, 30], "basic": [0, 5, 7, 9, 10, 11, 21, 28, 29], "batch": [1, 30], "bay": 5, "befor": 11, "better": 8, "bia": [6, 19, 22, 30], "binari": 1, "bind": 27, "bird": 10, "boldsymbol": [18, 28], "boost": 10, "bootstrap": [6, 10], "boston": [], "breast": 1, "brief": 27, "bring": 12, "build": [1, 3, 9], "c": [22, 27], "calcul": [18, 28, 29], "can": [27, 30], "cancer": [1, 7, 9, 11], "cart": 9, "case": [8, 10, 24, 28, 29], "central": [13, 20, 24, 29], "chain": 12, "challeng": 30, "chang": 10, "channel": 27, "chi": [0, 27], "choic": 17, "choos": [1, 30], "cifar01": 3, "classic": 11, "classif": [1, 9, 10], "classifi": 8, "clip": 1, "cluster": 14, "cnn": 3, "code": [1, 2, 5, 9, 11, 12, 13, 14, 15, 16, 22, 27, 28, 29, 30], "collect": [1, 3], "combin": 30, "commun": 27, "compar": [2, 10, 16], "comparison": 29, "compet": 30, "complet": 28, "complex": [0, 6, 22, 28], "complic": 6, "compon": 11, "comput": [9, 19, 30], "computerlab": 27, "con": [9, 30], "concept": 24, "condit": 29, "conjug": 13, "contn": 27, "converg": 30, "convex": [8, 13, 29], "convolut": [3, 12], "correct": 30, "correl": [11, 28], "correspond": [], "cost": [1, 10, 28, 29, 30], "cours": [20, 23, 26, 27], "covari": [5, 11, 24, 28], "cover": 27, "creat": 16, "cross": [6, 22], "cython": 27, "d": 22, "data": [0, 1, 3, 6, 7, 9, 11, 15, 17, 18, 20, 24, 27, 28], "dataset": [1, 3, 18], "david": 27, "deadlin": [22, 27], "deadllin": 25, "decai": [2, 30], "decis": [9, 10], "decomposit": [5, 11, 21, 28, 29], "deeep": [], "deep": [1, 2, 27], "defin": [1, 27], "degre": [0, 17, 28], "deliver": [15, 16], "deliveri": 22, "dens": 0, "deriv": [5, 12, 16, 17, 19, 28, 29, 30], "descent": [2, 10, 13, 18, 22, 29, 30], "design": 28, "detail": [3, 27], "develop": 1, "diagon": 11, "differ": [8, 30], "differenti": [2, 13, 30], "diffus": 2, "dimens": 30, "dimension": [2, 3, 8, 18], "disadvantag": 9, "discret": 24, "discrimin": 27, "distribut": [5, 24], "do": [1, 30], "doe": [28, 29], "domain": 24, "down": 1, "dropout": 1, "e": 22, "economi": [28, 29], "electron": 22, "element": [0, 24, 27], "elimin": 21, "energi": 27, "ensembl": 10, "entropi": 9, "environ": [0, 15], "equat": [0, 2, 12, 28, 29], "error": [0, 10, 27, 28, 29], "essenti": 27, "et": 30, "etc": 27, "euler": 2, "evalu": 1, "exampl": [1, 2, 3, 4, 6, 7, 8, 9, 10, 27, 28, 29, 30], "exercis": [0, 6, 15, 16, 17, 18, 19, 28], "expect": [19, 24], "experi": 24, "explor": 0, "exponenti": [2, 30], "express": [16, 17, 19, 28], "extend": 29, "extrapol": 4, "extrem": [10, 27], "ey": 10, "f": 22, "fall": 25, "famili": [1, 27], "famou": 21, "fantast": [28, 29], "featur": [9, 16, 21, 28], "feed": [1, 12], "final": [12, 28, 30], "find": [16, 18], "fine": 1, "first": [4, 12, 27, 29], "fit": [0, 10, 15, 16, 27, 29], "fix": [28, 29, 30], "forc": 3, "forest": 10, "form": 18, "format": [22, 27], "formula": 18, "forward": [1, 2, 12], "foster": 27, "fourier": 3, "frank": 6, "freedom": [0, 17, 28], "frequent": [28, 30], "frequentist": [0, 27], "from": [5, 10, 12, 27, 28, 29, 30], "full": 2, "function": [0, 1, 6, 7, 8, 10, 11, 12, 13, 22, 24, 27, 28, 29, 30], "further": [3, 5, 28, 29], "g": 22, "gan": 4, "gaussian": 21, "gd": [13, 30], "gener": [4, 9, 27], "geometr": [11, 29], "gini": 9, "github": 15, "goal": [15, 16, 17, 18, 19], "good": [0, 27], "goodfellow": 30, "grade": [25, 27], "gradient": [1, 2, 10, 13, 18, 22, 29, 30], "growth": 2, "h": 22, "ha": 20, "handl": [21, 27], "hessian": [28, 29, 30], "hidden": 2, "hous": [], "how": 16, "hyperparamet": [1, 17], "hyperplan": 8, "i": [0, 1, 27], "id3": 9, "idea": 11, "ideal": 29, "ii": 27, "illustr": 29, "implement": [1, 16, 17, 18], "implic": [5, 28, 29], "import": [5, 21, 27, 28, 29], "improv": [1, 30], "includ": [13, 22, 30], "increment": 11, "index": 9, "inform": 25, "input": 2, "instal": [20, 22, 27], "instructor": 25, "interpret": [5, 11, 27, 28, 29, 30], "introduc": [11, 13, 28], "introduct": [0, 6, 20, 21, 22, 27], "invers": [5, 21], "invert": [28, 29], "iter": 10, "its": 28, "jacobian": 28, "jax": 13, "julia": 27, "jungl": 10, "kera": [1, 3], "kernel": [8, 11], "lab": [29, 30], "lagrangian": 8, "lasso": [5, 6, 22, 28, 29], "last": [28, 30], "later": [5, 28, 29], "layer": [1, 2, 3, 12], "learn": [0, 1, 2, 11, 13, 14, 15, 16, 17, 18, 19, 20, 22, 27, 28, 29, 30], "least": [5, 6, 16, 19, 22, 27, 28, 29, 30], "lectur": [27, 29, 30], "level": 10, "librari": [20, 27], "likelihood": 7, "limit": [1, 13, 24, 29, 30], "linear": [0, 8, 13, 15, 21, 27, 28, 29], "link": [5, 11, 26, 28], "literatur": 22, "logist": [7, 27], "loss": [28, 29, 30], "lu": 21, "machin": [0, 8, 13, 20, 22, 27, 29], "main": [24, 27], "make": [0, 9, 10, 28], "mani": [10, 12], "mass": 27, "materi": [22, 27, 28, 29, 30], "math": [5, 28, 29], "mathemat": [3, 5, 8, 28, 29], "matric": [5, 21, 27], "matrix": [1, 5, 11, 12, 16, 21, 27, 28, 29, 30], "matter": 0, "max": 28, "mean": [0, 28, 29], "meet": [5, 10, 24, 27, 28], "mercer": 8, "method": [6, 9, 10, 13, 22, 27, 29, 30], "midnight": 22, "min": 28, "mini": 30, "minibatch": 30, "minim": 27, "ml": 27, "mlp": 12, "mnist": [3, 4], "model": [0, 1, 4, 6, 12, 15, 17, 27], "moment": 30, "momentum": [13, 22, 30], "mondai": [29, 30], "moon": [8, 9], "more": [3, 6, 21, 22, 27, 28, 29, 30], "motiv": 30, "move": 30, "multilay": 12, "multipl": [1, 3, 17], "multipli": 8, "need": [22, 27], "network": [1, 2, 3, 4, 7, 12, 27], "neural": [1, 2, 3, 4, 7, 12, 27], "new": [4, 18], "newton": [29, 30], "non": 8, "none": 30, "normal": [0, 1], "notat": 12, "note": [22, 28, 29], "now": [1, 9, 13, 29, 30], "nuclear": [0, 27], "numba": 27, "number": [0, 2, 24, 28, 30], "numer": [2, 22, 24], "numpi": [21, 27], "object": 3, "obtain": 11, "octob": 22, "od": 2, "off": [6, 19, 22], "ol": [5, 6, 15, 16, 18, 22, 29], "one": [2, 12, 18, 29], "oper": 21, "optim": [1, 8, 13, 18, 20, 27, 28, 29, 30], "order": [13, 18, 30], "ordinari": [5, 6, 16, 19, 22, 27, 28, 29, 30], "organ": [0, 27], "oslo": 26, "other": [4, 9, 11, 12, 21, 22, 27], "our": [0, 4, 5, 11, 13, 22, 27, 28, 29], "outcom": [20, 27], "output": 2, "overarch": [0, 4, 8, 9, 27, 28], "overview": [10, 27, 30], "own": [0, 10, 11, 22, 27, 28], "packag": [21, 27], "panda": [27, 28], "paramet": [27, 28], "paramt": 18, "part": [13, 20, 22, 29], "partial": 2, "pass": 1, "pca": 11, "pdf": 24, "perceptron": 12, "perform": [1, 9], "period": 3, "perspect": 1, "plan": [28, 29, 30], "plethora": 27, "point": 4, "poisson": 2, "polynomi": [3, 16, 18, 29], "popul": 2, "popular": 27, "practic": [13, 25, 27, 30], "pre": [1, 3], "preambl": 22, "predict": 4, "preprocess": [28, 30], "prerequisit": [3, 20, 27], "princip": 11, "principl": 3, "pro": [9, 30], "probabl": [5, 24], "problem": [1, 2, 13, 27, 28, 29, 30], "procedur": [9, 27], "process": [1, 3], "program": [2, 13, 22, 29, 30], "project": [6, 22, 25, 27], "prop": 13, "propag": [1, 12], "properti": [5, 24, 28, 29, 30], "python": [0, 9, 15, 20, 21, 27], "quick": 8, "r": 27, "random": [10, 11, 24], "raphson": 29, "rate": [22, 30], "read": [9, 27, 28, 30], "real": [6, 27], "recommend": [27, 28], "recurr": [4, 12], "reduc": [0, 28], "reduct": 3, "refer": 22, "reformul": 2, "regress": [0, 5, 6, 7, 9, 10, 13, 15, 17, 18, 19, 22, 27, 28, 29, 30], "regular": 1, "relat": [], "relev": [26, 28], "relu": 1, "remark": 3, "remind": [6, 8, 27, 28, 29, 30], "replac": [13, 30], "report": 22, "repositori": 15, "requir": [2, 20], "resampl": [6, 19, 22, 30], "rescal": [6, 28], "residu": [28, 29], "resourc": 2, "result": [28, 29], "revisit": [13, 29, 30], "rewrit": [27, 28], "ridg": [0, 5, 6, 17, 18, 19, 22, 28, 29, 30], "rm": 13, "rmsprop": 30, "rule": [12, 30], "rung": 22, "same": [13, 30], "sampl": 11, "scale": [17, 18, 28, 30], "schedul": 27, "schemat": 9, "scheme": 2, "scienc": 27, "scikit": [0, 1, 11, 27, 28, 29, 30], "second": [13, 18, 30], "semest": 25, "sensit": 29, "septemb": [19, 29, 30], "session": [29, 30], "set": [0, 2, 3, 9, 12, 15, 23, 27, 28, 29], "setup": 15, "sgd": [13, 30], "should": 1, "similar": [13, 30], "simpl": [0, 4, 9, 13, 18, 27, 28, 29, 30], "simplest": 18, "singl": 10, "singular": [5, 11, 28, 29], "size": [28, 29, 30], "sklearn": 16, "slightli": 30, "sneak": 30, "soft": 8, "softmax": 1, "softwar": [22, 27], "solv": [2, 29], "solver": 13, "some": [13, 21, 28, 29], "specifi": 2, "split": [0, 15, 28], "squar": [0, 5, 6, 10, 16, 19, 22, 27, 28, 29, 30], "standard": [13, 28], "state": 0, "statist": [5, 6, 20, 24, 27, 30], "steepest": [10, 13, 29], "step": 30, "stochast": [13, 22, 24, 30], "stop": 30, "strongli": 27, "suggest": 27, "summari": [25, 27], "superposit": 3, "supervis": 1, "support": 8, "svd": [5, 28, 29], "synthet": 18, "systemat": 3, "t": 28, "take": 16, "taken": [27, 30], "teach": 25, "teacher": [25, 27], "technic": 28, "techniqu": [6, 11, 22], "technologi": 20, "tensorflow": [1, 3], "tent": [25, 27], "test": [0, 1, 15, 17, 28], "text": 27, "textbook": [26, 27], "than": 29, "theorem": [5, 8, 11, 12, 24], "theori": 24, "theta": 18, "thi": 27, "time": 30, "tip": [13, 30], "togeth": 12, "tool": [22, 27], "top": 1, "topic": 27, "toward": 11, "trade": [6, 19, 22], "tradeoff": 6, "train": [0, 1, 4, 15, 27, 28], "transform": 3, "tree": [9, 10], "tuesdai": 29, "tune": 1, "two": [3, 8, 20], "type": [2, 4, 12, 27], "uio": 27, "univers": [12, 26], "unsupervis": 14, "up": [0, 2, 9, 12, 15, 27, 28, 29], "updat": [22, 30], "us": [0, 1, 2, 3, 7, 13, 16, 18, 20, 22, 27, 28, 29, 30], "v": [3, 30], "valid": [6, 22], "valu": [5, 11, 19, 24, 28, 29], "vari": 30, "variabl": [24, 29], "varianc": [6, 19, 22], "variou": 0, "vector": [8, 12, 16, 21, 27, 28], "versu": 27, "video": 30, "view": [0, 4, 10, 28], "virtual": 15, "visual": [1, 9], "wai": [9, 22], "wave": 2, "we": [27, 30], "wednesdai": 29, "week": [15, 16, 17, 18, 19, 27, 28, 29, 30], "weekli": [], "what": [0, 27, 28, 29], "when": 30, "which": [1, 30], "why": [27, 30], "wisconsin": 7, "write": [4, 11, 22, 29], "x": 28, "xgboost": 10, "yet": 29, "your": [0, 10, 16, 18, 22, 28]}}) \ No newline at end of file diff --git a/doc/LectureNotes/_build/html/week37.html b/doc/LectureNotes/_build/html/week37.html index fca68dada..06a32ce6e 100644 --- a/doc/LectureNotes/_build/html/week37.html +++ b/doc/LectureNotes/_build/html/week37.html @@ -397,6 +397,7 @@ document.write(`
  • The Hessian matrix for Ridge Regression
  • Program example for gradient descent with Ridge Regression
  • Using gradient descent methods, limitations
    • +
  • Momentum based GD
  • Improving gradient descent with momentum
  • Same code but now with momentum gradient descent
  • Overview video on Stochastic Gradient Descent (SGD)
    • @@ -415,23 +416,25 @@ document.write(`
  • Time decay rate
  • Code with a Number of Minibatches which varies
  • Replace or not
    • -
  • Momentum based GD
    • -
  • More on momentum based approaches
    • -
  • Momentum parameter
  • Second moment of the gradient
  • Challenge: Choosing a Fixed Learning Rate
  • Motivation for Adaptive Step Sizes
    • +
  • AdaGrad algorithm, taken from Goodfellow et al
  • Derivation of the AdaGrad Algorithm
  • AdaGrad Update Rule Derivation
  • AdaGrad Properties
  • RMSProp: Adaptive Learning Rates
    • -
  • Adam Optimizer
    • -
  • RMS prop
    • -
  • ADAM optimizer
    • -
  • Algorithms and codes for Adagrad, RMSprop and Adam
    • -
  • AdaGrad algorithm, taken from Goodfellow et al
  • RMSProp algorithm, taken from Goodfellow et al
    • +
  • Adam Optimizer
    • +
  • ADAM optimizer
    • +
  • Why Combine Momentum and RMSProp?
    • +
  • Adam: Exponential Moving Averages (Moments)
    • +
  • Adam: Bias Correction
    • +
  • Adam: Update Rule Derivation
    • +
  • Adam vs. AdaGrad and RMSProp
    • +
  • Adaptivity Across Dimensions
  • ADAM algorithm, taken from Goodfellow et al
    • +
  • Algorithms and codes for Adagrad, RMSprop and Adam
  • Practical tips
  • Sneaking in automatic differentiation using Autograd
  • Same code but now with momentum gradient descent
    • @@ -501,7 +504,8 @@ desirable properties such as:

    We revisit an example similar to what we had in the first homework set. We have a function of the type

    -
    x = 2*np.random.rand(m,1)
+
    import numpy as np
+x = 2*np.random.rand(m,1)
 y = 4+3*x+np.random.randn(m,1)
 
    
 
    
@@ -531,7 +535,7 @@ X \equiv \begin{bmatrix}
 1 & x_{100} &  \\
 \end{bmatrix}.
 \end{split}\]
    -

    The cost/loss/risk function is given by (

    +

    The cost/loss/risk function is given by

    \[ C(\theta) = \frac{1}{n}||X\theta-\mathbf{y}||_{2}^{2} = \frac{1}{n}\sum_{i=1}^{100}\left[ (\theta_0 + \theta_1 x_i)^2 - 2 y_i (\theta_0 + \theta_1 x_i) + y_i^2\right] @@ -563,13 +567,13 @@ C(\theta) = \frac{1}{n}||X\theta-\mathbf{y}||_{2}^{2} = \frac{1}{n}\sum_{i=1}^{1

    Simple program#

    -

    We can now write a program that minimizes \(C(\theta)\) using the gradient descent method with a constant learning rate \(\gamma\) according to

    +

    We can now write a program that minimizes \(C(\theta)\) using the gradient descent method with a constant learning rate \(\eta\) according to

    \[ -\theta_{k+1} = \theta_k - \gamma \nabla_\theta C(\theta_k), \ k=0,1,\cdots +\theta_{k+1} = \theta_k - \eta \nabla_\theta C(\theta_k), \ k=0,1,\cdots \]

    We can use the expression we computed for the gradient and let use a -\(\theta_0\) be chosen randomly and let \(\gamma = 0.001\). Stop iterating +\(\theta_0\) be chosen randomly and let \(\eta = 0.001\). Stop iterating when \(||\nabla_\theta C(\theta_k) || \leq \epsilon = 10^{-8}\). Note that the code below does not include the latter stop criterion.

    And finally we can compare our solution for \(\theta\) with the analytic result given by \(\theta= (X^TX)^{-1} X^T \mathbf{y}\).

    @@ -738,9 +742,15 @@ plt.show()

  • GD can take exponential time to escape saddle points, even with random initialization. As we mentioned, GD is extremely sensitive to initial condition since it determines the particular local minimum GD would eventually reach. However, even with a good initialization scheme, through the introduction of randomness, GD can still take exponential time to escape saddle points.

    • +
    +

    Momentum based GD#

    +

    We discuss here some simple examples where we introduce what is called +‘memory’about previous steps, or what is normally called momentum +gradient descent. +For the mathematical details, see whiteboad notes from lecture on September 8, 2025.

    +

    Improving gradient descent with momentum#

    -

    We discuss here some simple examples where we introduce what is called ‘memory’about previous steps, or what is normally called momentum gradient descent. The mathematics is explained below in connection with Stochastic gradient descent.

    from numpy import asarray
@@ -961,8 +971,8 @@ function, which we want to minimize, can almost always be written as a
 sum over \(n\) data points \(\{\mathbf{x}_i\}_{i=1}^n\),
    
 
    
 \[
-C(\mathbf{\beta}) = \sum_{i=1}^n c_i(\mathbf{x}_i,
-\mathbf{\beta}).
+C(\mathbf{\theta}) = \sum_{i=1}^n c_i(\mathbf{x}_i,
+\mathbf{\theta}).
 \]
    @@ -971,8 +981,8 @@ C(\mathbf{\beta}) = \sum_{i=1}^n c_i(\mathbf{x}_i, computed as a sum over \(i\)-gradients

    \[ -\nabla_\beta C(\mathbf{\beta}) = \sum_i^n \nabla_\beta c_i(\mathbf{x}_i, -\mathbf{\beta}). +\nabla_\theta C(\mathbf{\theta}) = \sum_i^n \nabla_\theta c_i(\mathbf{x}_i, +\mathbf{\theta}). \]

    Stochasticity/randomness is introduced by only taking the gradient on a subset of the data called minibatches. If there are \(n\) @@ -995,10 +1005,10 @@ all data points with a sum over the data points in one the minibatches picked at random in each gradient descent step

    \[ -\nabla_{\beta} -C(\mathbf{\beta}) = \sum_{i=1}^n \nabla_\beta c_i(\mathbf{x}_i, -\mathbf{\beta}) \rightarrow \sum_{i \in B_k}^n \nabla_\beta -c_i(\mathbf{x}_i, \mathbf{\beta}). +\nabla_{\theta} +C(\mathbf{\theta}) = \sum_{i=1}^n \nabla_\theta c_i(\mathbf{x}_i, +\mathbf{\theta}) \rightarrow \sum_{i \in B_k}^n \nabla_\theta +c_i(\mathbf{x}_i, \mathbf{\theta}). \]
    @@ -1006,8 +1016,8 @@ c_i(\mathbf{x}_i, \mathbf{\beta}).

    Thus a gradient descent step now looks like

    \[ -\beta_{j+1} = \beta_j - \gamma_j \sum_{i \in B_k}^n \nabla_\beta c_i(\mathbf{x}_i, -\mathbf{\beta}) +\theta_{j+1} = \theta_j - \eta_j \sum_{i \in B_k}^n \nabla_\theta c_i(\mathbf{x}_i, +\mathbf{\theta}) \]

    where \(k\) is picked at random with equal probability from \([1,n/M]\). An iteration over the number of @@ -1055,12 +1065,12 @@ is zero is valid also for local minima, so this would only tell us that we are close to a local/global minimum. However, we could also evaluate the cost function at this point, store the result and continue the search. If the test kicks in at a later stage we can -compare the values of the cost function and keep the \(\beta\) that +compare the values of the cost function and keep the \(\theta\) that gave the lowest value.

    Slightly different approach#

    -

    Another approach is to let the step length \(\gamma_j\) depend on the +

    Another approach is to let the step length \(\eta_j\) depend on the number of epochs in such a way that it becomes very small after a reasonable time such that we do not move at all. Such approaches are also called scaling. There are many such ways to scale the learning @@ -1072,11 +1082,11 @@ for a discussion of different scaling functions for the learning rate.

    Time decay rate#

    -

    As an example, let \(e = 0,1,2,3,\cdots\) denote the current epoch and let \(t_0, t_1 > 0\) be two fixed numbers. Furthermore, let \(t = e \cdot m + i\) where \(m\) is the number of minibatches and \(i=0,\cdots,m-1\). Then the function $\(\gamma_j(t; t_0, t_1) = \frac{t_0}{t+t_1} \)\( goes to zero as the number of epochs gets large. I.e. we start with a step length \)\gamma_j (0; t_0, t_1) = t_0/t_1\( which decays in *time* \)t$.

    -

    In this way we can fix the number of epochs, compute \(\beta\) and +

    As an example, let \(e = 0,1,2,3,\cdots\) denote the current epoch and let \(t_0, t_1 > 0\) be two fixed numbers. Furthermore, let \(t = e \cdot m + i\) where \(m\) is the number of minibatches and \(i=0,\cdots,m-1\). Then the function $\(\eta_j(t; t_0, t_1) = \frac{t_0}{t+t_1} \)\( goes to zero as the number of epochs gets large. I.e. we start with a step length \)\eta_j (0; t_0, t_1) = t_0/t_1\( which decays in *time* \)t$.

    +

    In this way we can fix the number of epochs, compute \(\theta\) and evaluate the cost function at the end. Repeating the computation will give a different result since the scheme is random by design. Then we -pick the final \(\beta\) that gives the lowest value of the cost +pick the final \(\theta\) that gives the lowest value of the cost function.

    @@ -1092,18 +1102,18 @@ n_epochs = 500 #number of epochs t0 = 1.0 t1 = 10 -gamma_j = t0/t1 +eta_j = t0/t1 j = 0 for epoch in range(1,n_epochs+1): for i in range(m): k = np.random.randint(m) #Pick the k-th minibatch at random #Compute the gradient using the data in minibatch Bk - #Compute new suggestion for beta + #Compute new suggestion for theta t = epoch*m+i - gamma_j = step_length(t,t0,t1) + eta_j = step_length(t,t0,t1) j += 1 -print("gamma_j after %d epochs: %g" % (n_epochs,gamma_j)) +print("eta_j after %d epochs: %g" % (n_epochs,eta_j))
    @@ -1192,111 +1202,6 @@ mini-batches. The discussion here may be useful.

    -
    -

    Momentum based GD#

    -

    The stochastic gradient descent (SGD) is almost always used with a -momentum or inertia term that serves as a memory of the direction we -are moving in parameter space. This is typically implemented as -follows

    -
    -\[ -\mathbf{v}_{t}=\gamma \mathbf{v}_{t-1}+\eta_{t}\nabla_\theta E(\boldsymbol{\theta}_t) \nonumber -\]
    - -
    -
    -\[ -\begin{equation} -\boldsymbol{\theta}_{t+1}= \boldsymbol{\theta}_t -\mathbf{v}_{t}, -\label{_auto1} \tag{1} -\end{equation} -\]
    -

    where we have introduced a momentum parameter \(\gamma\), with -\(0\le\gamma\le 1\), and for brevity we dropped the explicit notation to -indicate the gradient is to be taken over a different mini-batch at -each step. We call this algorithm gradient descent with momentum -(GDM). From these equations, it is clear that \(\mathbf{v}_t\) is a -running average of recently encountered gradients and -\((1-\gamma)^{-1}\) sets the characteristic time scale for the memory -used in the averaging procedure. Consistent with this, when -\(\gamma=0\), this just reduces down to ordinary SGD as discussed -earlier. An equivalent way of writing the updates is

    -
    -\[ -\Delta \boldsymbol{\theta}_{t+1} = \gamma \Delta \boldsymbol{\theta}_t -\ \eta_{t}\nabla_\theta E(\boldsymbol{\theta}_t), -\]
    -

    where we have defined \(\Delta \boldsymbol{\theta}_{t}= \boldsymbol{\theta}_t-\boldsymbol{\theta}_{t-1}\).

    -
    -
    -

    More on momentum based approaches#

    -

    Let us try to get more intuition from these equations. It is helpful -to consider a simple physical analogy with a particle of mass \(m\) -moving in a viscous medium with drag coefficient \(\mu\) and potential -\(E(\mathbf{w})\). If we denote the particle’s position by \(\mathbf{w}\), -then its motion is described by

    -
    -\[ -m {d^2 \mathbf{w} \over dt^2} + \mu {d \mathbf{w} \over dt }= -\nabla_w E(\mathbf{w}). -\]
    -

    We can discretize this equation in the usual way to get

    -
    -\[ -m { \mathbf{w}_{t+\Delta t}-2 \mathbf{w}_{t} +\mathbf{w}_{t-\Delta t} \over (\Delta t)^2}+\mu {\mathbf{w}_{t+\Delta t}- \mathbf{w}_{t} \over \Delta t} = -\nabla_w E(\mathbf{w}). -\]
    -

    Rearranging this equation, we can rewrite this as

    -
    -\[ -\Delta \mathbf{w}_{t +\Delta t}= - { (\Delta t)^2 \over m +\mu \Delta t} \nabla_w E(\mathbf{w})+ {m \over m +\mu \Delta t} \Delta \mathbf{w}_t. -\]
    -
    -
    -

    Momentum parameter#

    -

    Notice that this equation is identical to previous one if we identify -the position of the particle, \(\mathbf{w}\), with the parameters -\(\boldsymbol{\theta}\). This allows us to identify the momentum -parameter and learning rate with the mass of the particle and the -viscous drag as:

    -
    -\[ -\gamma= {m \over m +\mu \Delta t }, \qquad \eta = {(\Delta t)^2 \over m +\mu \Delta t}. -\]
    -

    Thus, as the name suggests, the momentum parameter is proportional to -the mass of the particle and effectively provides inertia. -Furthermore, in the large viscosity/small learning rate limit, our -memory time scales as \((1-\gamma)^{-1} \approx m/(\mu \Delta t)\).

    -

    Why is momentum useful? SGD momentum helps the gradient descent -algorithm gain speed in directions with persistent but small gradients -even in the presence of stochasticity, while suppressing oscillations -in high-curvature directions. This becomes especially important in -situations where the landscape is shallow and flat in some directions -and narrow and steep in others. It has been argued that first-order -methods (with appropriate initial conditions) can perform comparable -to more expensive second order methods, especially in the context of -complex deep learning models.

    -

    These beneficial properties of momentum can sometimes become even more -pronounced by using a slight modification of the classical momentum -algorithm called Nesterov Accelerated Gradient (NAG).

    -

    In the NAG algorithm, rather than calculating the gradient at the -current parameters, \(\nabla_\theta E(\boldsymbol{\theta}_t)\), one -calculates the gradient at the expected value of the parameters given -our current momentum, \(\nabla_\theta E(\boldsymbol{\theta}_t +\gamma -\mathbf{v}_{t-1})\). This yields the NAG update rule

    -
    -\[ -\mathbf{v}_{t}=\gamma \mathbf{v}_{t-1}+\eta_{t}\nabla_\theta E(\boldsymbol{\theta}_t +\gamma \mathbf{v}_{t-1}) \nonumber -\]
    - -
    -
    -\[ -\begin{equation} -\boldsymbol{\theta}_{t+1}= \boldsymbol{\theta}_t -\mathbf{v}_{t}. -\label{_auto2} \tag{2} -\end{equation} -\]
    -

    One of the major advantages of NAG is that it allows for the use of a -larger learning rate than GDM for the same choice of \(\gamma\).

    -

    Second moment of the gradient#

    In stochastic gradient descent, with and without momentum, we still @@ -1337,29 +1242,35 @@ For a function with steep directions and flat directions, a single global

    Motivation for Adaptive Step Sizes#

      -

    1. Instead of a fixed global \(\eta\), use an \textbf{adaptive learning rate} for each parameter that depends on the history of gradients.

      2. +

    2. Instead of a fixed global \(\eta\), use an adaptive learning rate for each parameter that depends on the history of gradients.

    3. Parameters that have large accumulated gradient magnitude should get smaller steps (they’ve been changing a lot), whereas parameters with small or infrequent gradients can have larger relative steps.

    4. This is especially useful for sparse features: Rarely active features accumulate little gradient, so their learning rate remains comparatively high, ensuring they are not neglected

    5. Conversely, frequently active features accumulate large gradient sums, and their learning rate automatically decreases, preventing too-large updates

    6. Several algorithms implement this idea (AdaGrad, RMSProp, AdaDelta, Adam, etc.). We will derive AdaGrad, one of the first adaptive methods.

    +
    +

    AdaGrad algorithm, taken from Goodfellow et al#

    + + +

    Figure 1:

    +

    Derivation of the AdaGrad Algorithm#

    Accumulating Gradient History.

    1. AdaGrad maintains a running sum of squared gradients for each parameter (coordinate)

      2. -

    2. Let \(g_t = \nabla \ell_{i_t}(x_t)\) be the gradient at step \(t\) (or a subgradient for nondifferentiable cases).

      3. -

    3. Initialize \(h_0 = 0\) (an all-zero vector in \(\mathbb{R}^d\)).

      4. +

    4. Let \(g_t = \nabla C_{i_t}(x_t)\) be the gradient at step \(t\) (or a subgradient for nondifferentiable cases).

      5. +

    5. Initialize \(r_0 = 0\) (an all-zero vector in \(\mathbb{R}^d\)).

    6. At each iteration \(t\), update the accumulation:

    \[ -h_t \;=\; h_{t-1} + g_t \circ g_t, +r_t = r_{t-1} + g_t \circ g_t, \]
      -

    1. Here \(g_t \circ g_t\) denotes element-wise square of the gradient vector. \(h_t^{(j)} = h_{t-1}^{(j)} + (g_{t,j})^2\) for each parameter \(j\).

      2. -

    2. We can view \(H_t = \mathrm{diag}(h_t)\) as a diagonal matrix of past squared gradients. Initially \(H_0 = 0\).

      3. +

    3. Here \(g_t \circ g_t\) denotes element-wise square of the gradient vector. \(g_t^{(j)} = g_{t-1}^{(j)} + (g_{t,j})^2\) for each parameter \(j\).

      4. +

    4. We can view \(H_t = \mathrm{diag}(r_t)\) as a diagonal matrix of past squared gradients. Initially \(H_0 = 0\).

    @@ -1367,32 +1278,32 @@ h_t \;=\; h_{t-1} + g_t \circ g_t,

    We scale the gradient by the inverse square root of the accumulated matrix \(H_t\). The AdaGrad update at step \(t\) is:

    \[ -x_{t+1} \;=\; x_t \;-\; \eta\, H_t^{-1/2}\, g_t, +\theta_{t+1} =\theta_t - \eta H_t^{-1/2} g_t, \]
    -

    where \(H_t^{-1/2}\) is the diagonal matrix with entries \((h_{t}^{(1)})^{-1/2}, \dots, (h_{t}^{(d)})^{-1/2}\) +

    where \(H_t^{-1/2}\) is the diagonal matrix with entries \((r_{t}^{(1)})^{-1/2}, \dots, (r_{t}^{(d)})^{-1/2}\) In coordinates, this means each parameter \(j\) has an individual step size:

    \[ -x_{t+1,j} =x_{t,j} -\frac{\eta}{\sqrt{h_{t,j}}}g_{t,j}. +\theta_{t+1,j} =\theta_{t,j} -\frac{\eta}{\sqrt{r_{t,j}}}g_{t,j}. \]

    In practice we add a small constant \(\epsilon\) in the denominator for numerical stability to avoid division by zero:

    \[ -x_{t+1,j}= x_{t,j}-\frac{\eta}{\sqrt{\epsilon + h_{t,j}}}g_{t,j}. +\theta_{t+1,j}= \theta_{t,j}-\frac{\eta}{\sqrt{\epsilon + r_{t,j}}}g_{t,j}. \]
    -

    Equivalently, the effective learning rate for parameter \(j\) at time \(t\) is \(\displaystyle \alpha_{t,j} = \frac{\eta}{\sqrt{\epsilon + h_{t,j}}}\). This decreases over time as \(h_{t,j}\) grows.

    +

    Equivalently, the effective learning rate for parameter \(j\) at time \(t\) is \(\displaystyle \alpha_{t,j} = \frac{\eta}{\sqrt{\epsilon + r_{t,j}}}\). This decreases over time as \(r_{t,j}\) grows.

    AdaGrad Properties#

      -

    1. AdaGrad automatically tunes the step size for each parameter. Parameters with more volatile or large gradients get smaller steps, and those with small or infrequent gradients get relatively larger steps

      2. -

    2. No manual schedule needed: The accumulation \(h_t\) keeps increasing (or stays the same if gradient is zero), so step sizes \(\eta/\sqrt{h_t}\) are non-increasing. This has a similar effect to a learning rate schedule, but individualized per coordinate.

      3. -

    3. Sparse data benefit: For very sparse features, \(h_{t,j}\) grows slowly, so that feature’s parameter retains a higher learning rate for longer, allowing it to make significant updates when it does get a gradient signal

      4. -

    4. Convergence: In convex optimization, AdaGrad can be shown to achieve a sub-linear convergence rate (e.g. \(O(1/\sqrt{T})\) regret bound) comparable to the best fixed learning rate tuned for the problem

      5. +

    5. AdaGrad automatically tunes the step size for each parameter. Parameters with more volatile or large gradients get smaller steps, and those with small or infrequent gradients get relatively larger steps

      6. +

    6. No manual schedule needed: The accumulation \(r_t\) keeps increasing (or stays the same if gradient is zero), so step sizes \(\eta/\sqrt{r_t}\) are non-increasing. This has a similar effect to a learning rate schedule, but individualized per coordinate.

      7. +

    7. Sparse data benefit: For very sparse features, \(r_{t,j}\) grows slowly, so that feature’s parameter retains a higher learning rate for longer, allowing it to make significant updates when it does get a gradient signal

      8. +

    8. Convergence: In convex optimization, AdaGrad can be shown to achieve a sub-linear convergence rate comparable to the best fixed learning rate tuned for the problem

    It effectively reduces the need to tune \(\eta\) by hand.

      -

    1. Limitations: Because \(h_t\) accumulates without bound, AdaGrad’s learning rates can become extremely small over long training, potentially slowing progress. (Later variants like RMSProp, AdaDelta, Adam address this by modifying the accumulation rule.)

      2. +

    2. Limitations: Because \(r_t\) accumulates without bound, AdaGrad’s learning rates can become extremely small over long training, potentially slowing progress. (Later variants like RMSProp, AdaDelta, Adam address this by modifying the accumulation rule.)

    @@ -1401,19 +1312,25 @@ x_{t+1,j}= x_{t,j}-\frac{\eta}{\sqrt{\epsilon + h_{t,j}}}g_{t,j}. Uses a decaying average of squared gradients (instead of a cumulative sum):

    \[ -v_t = \beta_2\, v_{t-1} + (1-\beta_2)\, (\nabla L(w_t))^2, +v_t = \rho v_{t-1} + (1-\rho)(\nabla C(\theta_t))^2, \]
    -

    with \(\beta_2\) typically \(0.9\) (or \(0.99\)).

    +

    with \(\rho\) typically \(0.9\) (or \(0.99\)).

      -

    1. Update: \(w_{t+1} = w_t - \frac{\alpha}{\sqrt{v_t + \epsilon}} \nabla L(w_t)\).

      2. +

    2. Update: \(\theta_{t+1} = \theta_t - \frac{\eta}{\sqrt{v_t + \epsilon}} \nabla C(\theta_t)\).

    3. Recent gradients have more weight, so \(v_t\) adapts to the current landscape.

    4. Avoids AdaGrad’s “infinite memory” problem – learning rate does not continuously decay to zero.

    RMSProp was first proposed in lecture notes by Geoff Hinton, 2012 - unpublished.)

    +
    +

    RMSProp algorithm, taken from Goodfellow et al#

    + + +

    Figure 1:

    +

    Adam Optimizer#

    -

    Why Combine Momentum and RMSProp? Motivation for Adam: Adaptive Moment Estimation (Adam) was introduced by Kingma an Ba (2014) to combine the benefits of momentum and RMSProp.

    +

    Why combine Momentum and RMSProp? Motivation for Adam: Adaptive Moment Estimation (Adam) was introduced by Kingma an Ba (2014) to combine the benefits of momentum and RMSProp.

    1. Fast convergence by smoothing gradients (accelerates in long-term gradient direction).

    2. Adaptive rates (RMSProp): Per-dimension learning rate scaling for stability (handles different feature scales, sparse gradients).

      3. @@ -1422,132 +1339,108 @@ v_t = \beta_2\, v_{t-1} + (1-\beta_2)\, (\nabla L(w_t))^2,

    Result: Adam is robust, achieves faster convergence with less tuning, and often outperforms SGD (with momentum) in practice.

    -
    -

    RMS prop#

    -

    In RMS prop, in addition to keeping a running average of the first -moment of the gradient, we also keep track of the second moment -denoted by \(\mathbf{s}_t=\mathbb{E}[\mathbf{g}_t^2]\). The update rule -for RMS prop is given by

    - -
    -
    -\[ -\begin{equation} -\mathbf{g}_t = \nabla_\theta E(\boldsymbol{\theta}) -\label{_auto3} \tag{3} -\end{equation} -\]
    -
    -\[ -\mathbf{s}_t =\beta \mathbf{s}_{t-1} +(1-\beta)\mathbf{g}_t^2 \nonumber -\]
    -
    -\[ -\boldsymbol{\theta}_{t+1}=\boldsymbol{\theta}_t - \eta_t { \mathbf{g}_t \over \sqrt{\mathbf{s}_t +\epsilon}}, \nonumber -\]
    -

    where \(\beta\) controls the averaging time of the second moment and is -typically taken to be about \(\beta=0.9\), \(\eta_t\) is a learning rate -typically chosen to be \(10^{-3}\), and \(\epsilon\sim 10^{-8} \) is a -small regularization constant to prevent divergences. Multiplication -and division by vectors is understood as an element-wise operation. It -is clear from this formula that the learning rate is reduced in -directions where the norm of the gradient is consistently large. This -greatly speeds up the convergence by allowing us to use a larger -learning rate for flat directions.

    -

    ADAM optimizer#

    -

    A related algorithm is the ADAM optimizer. In -ADAM, we keep a running average of +

    In ADAM, we keep a running average of both the first and second moment of the gradient and use this information to adaptively change the learning rate for different -parameters. The method isefficient when working with large +parameters. The method is efficient when working with large problems involving lots data and/or parameters. It is a combination of the gradient descent with momentum algorithm and the RMSprop algorithm discussed above.

    -

    In addition to keeping a running average of the first and -second moments of the gradient -(i.e. \(\mathbf{m}_t=\mathbb{E}[\mathbf{g}_t]\) and -\(\mathbf{s}_t=\mathbb{E}[\mathbf{g}^2_t]\), respectively), ADAM -performs an additional bias correction to account for the fact that we -are estimating the first two moments of the gradient using a running -average (denoted by the hats in the update rule below). The update -rule for ADAM is given by (where multiplication and division are once -again understood to be element-wise operations below)

    - -
    -
    -\[ -\begin{equation} -\mathbf{g}_t = \nabla_\theta E(\boldsymbol{\theta}) -\label{_auto4} \tag{4} -\end{equation} -\]
    -
    -\[ -\mathbf{m}_t = \beta_1 \mathbf{m}_{t-1} + (1-\beta_1) \mathbf{g}_t \nonumber -\]
    -
    -\[ -\mathbf{s}_t =\beta_2 \mathbf{s}_{t-1} +(1-\beta_2)\mathbf{g}_t^2 \nonumber -\]
    -
    -\[ -\boldsymbol{\mathbf{m}}_t={\mathbf{m}_t \over 1-\beta_1^t} \nonumber -\]
    -
    -\[ -\boldsymbol{\mathbf{s}}_t ={\mathbf{s}_t \over1-\beta_2^t} \nonumber -\]
    -
    -\[ -\boldsymbol{\theta}_{t+1}=\boldsymbol{\theta}_t - \eta_t { \boldsymbol{\mathbf{m}}_t \over \sqrt{\boldsymbol{\mathbf{s}}_t} +\epsilon}, \nonumber -\]
    - -
    -
    -\[ -\begin{equation} -\label{_auto5} \tag{5} -\end{equation} -\]
    -

    where \(\beta_1\) and \(\beta_2\) set the memory lifetime of the first and -second moment and are typically taken to be \(0.9\) and \(0.99\) -respectively, and \(\eta\) and \(\epsilon\) are identical to RMSprop.

    -

    Like in RMSprop, the effective step size of a parameter depends on the -magnitude of its gradient squared. To understand this better, let us -rewrite this expression in terms of the variance -\(\boldsymbol{\sigma}_t^2 = \boldsymbol{\mathbf{s}}_t - -(\boldsymbol{\mathbf{m}}_t)^2\). Consider a single parameter \(\theta_t\). The -update rule for this parameter is given by

    -
    -\[ -\Delta \theta_{t+1}= -\eta_t { \boldsymbol{m}_t \over \sqrt{\sigma_t^2 + m_t^2 }+\epsilon}. -\]
    -
    -

    Algorithms and codes for Adagrad, RMSprop and Adam#

    -

    The algorithms we have implemented are well described in the text by Goodfellow, Bengio and Courville, chapter 8.

    -

    The codes which implement these algorithms are discussed after our presentation of automatic differentiation.

    +
    +

    Why Combine Momentum and RMSProp?#

    +
      +

    1. Momentum: Fast convergence by smoothing gradients (accelerates in long-term gradient direction).

      2. +

    2. Adaptive rates (RMSProp): Per-dimension learning rate scaling for stability (handles different feature scales, sparse gradients).

      3. +

    3. Adam uses both: maintains moving averages of both first moment (gradients) and second moment (squared gradients)

      4. +

    4. Additionally, includes a mechanism to correct the bias in these moving averages (crucial in early iterations)

      5. +
    +

    Result: Adam is robust, achieves faster convergence with less tuning, and often outperforms SGD (with momentum) in practice

    +
    +
    +

    Adam: Exponential Moving Averages (Moments)#

    +

    Adam maintains two moving averages at each time step \(t\) for each parameter \(w\): +First moment (mean) \(m_t\).

    +

    The Momentum term

    +
    +\[ +m_t = \beta_1m_{t-1} + (1-\beta_1)\, \nabla C(\theta_t), +\]
    +

    Second moment (uncentered variance) \(v_t\).

    +

    The RMS term

    +
    +\[ +v_t = \beta_2v_{t-1} + (1-\beta_2)(\nabla C(\theta_t))^2, +\]
    +

    with typical \(\beta_1 = 0.9\), \(\beta_2 = 0.999\). Initialize \(m_0 = 0\), \(v_0 = 0\).

    +

    These are biased estimators of the true first and second moment of the gradients, especially at the start (since \(m_0,v_0\) are zero)

    +
    +
    +

    Adam: Bias Correction#

    +

    To counteract initialization bias in \(m_t, v_t\), Adam computes bias-corrected estimates

    +
    +\[ +\hat{m}_t = \frac{m_t}{1 - \beta_1^t}, \qquad \hat{v}_t = \frac{v_t}{1 - \beta_2^t}. +\]
    +
      +

    • When \(t\) is small, \(1-\beta_i^t \approx 0\), so \(\hat{m}_t, \hat{v}_t\) significantly larger than raw \(m_t, v_t\), compensating for the initial zero bias.

      • +

    • As \(t\) increases, \(1-\beta_i^t \to 1\), and \(\hat{m}_t, \hat{v}_t\) converge to \(m_t, v_t\).

      • +

    • Bias correction is important for Adam’s stability in early iterations

      • +
    +
    +
    +

    Adam: Update Rule Derivation#

    +

    Finally, Adam updates parameters using the bias-corrected moments:

    +
    +\[ +\theta_{t+1} =\theta_t -\frac{\alpha}{\sqrt{\hat{v}_t} + \epsilon}\hat{m}_t, +\]
    +

    where \(\epsilon\) is a small constant (e.g. \(10^{-8}\)) to prevent division by zero. +Breaking it down:

    +
      +

    1. Compute gradient \(\nabla C(\theta_t)\).

      2. +

    2. Update first moment \(m_t\) and second moment \(v_t\) (exponential moving averages).

      3. +

    3. Bias-correct: \(\hat{m}_t = m_t/(1-\beta_1^t)\), \(\; \hat{v}_t = v_t/(1-\beta_2^t)\).

      4. +

    4. Compute step: \(\Delta \theta_t = \frac{\hat{m}_t}{\sqrt{\hat{v}_t} + \epsilon}\).

      5. +

    5. Update parameters: \(\theta_{t+1} = \theta_t - \alpha\, \Delta \theta_t\).

      6. +
    +

    This is the Adam update rule as given in the original paper.

    +
    +
    +

    Adam vs. AdaGrad and RMSProp#

    +
      +

    1. AdaGrad: Uses per-coordinate scaling like Adam, but no momentum. Tends to slow down too much due to cumulative history (no forgetting)

      2. +

    2. RMSProp: Uses moving average of squared gradients (like Adam’s \(v_t\)) to maintain adaptive learning rates, but does not include momentum or bias-correction.

      3. +

    3. Adam: Effectively RMSProp + Momentum + Bias-correction

      4. +
    +
      +

    • Momentum (\(m_t\)) provides acceleration and smoother convergence.

      • +

    • Adaptive \(v_t\) scaling moderates the step size per dimension.

      • +

    • Bias correction (absent in AdaGrad/RMSProp) ensures robust estimates early on.

      • +
    +

    In practice, Adam often yields faster convergence and better tuning stability than RMSProp or AdaGrad alone

    +
    +
    +

    Adaptivity Across Dimensions#

    +
      +

    1. Adam adapts the step size \emph{per coordinate}: parameters with larger gradient variance get smaller effective steps, those with smaller or sparse gradients get larger steps.

      2. +

    2. This per-dimension adaptivity is inherited from AdaGrad/RMSProp and helps handle ill-conditioned or sparse problems.

      3. +

    3. Meanwhile, momentum (first moment) allows Adam to continue making progress even if gradients become small or noisy, by leveraging accumulated direction.

      4. +
    -
    -

    AdaGrad algorithm, taken from Goodfellow et al#

    - - -

    Figure 1:

    -
    -
    -

    RMSProp algorithm, taken from Goodfellow et al#

    - - -

    Figure 1:

    -

    ADAM algorithm, taken from Goodfellow et al#

    Figure 1:

    +
    +

    Algorithms and codes for Adagrad, RMSprop and Adam#

    +

    The algorithms we have implemented are well described in the text by Goodfellow, Bengio and Courville, chapter 8.

    +

    The codes which implement these algorithms are discussed below here.

    +

    Practical tips#

      @@ -1576,8 +1469,8 @@ import autograd.numpy as np import matplotlib.pyplot as plt from autograd import grad -def CostOLS(beta): - return (1.0/n)*np.sum((y-X @ beta)**2) +def CostOLS(theta): + return (1.0/n)*np.sum((y-X @ theta)**2) n = 100 x = 2*np.random.rand(n,1) @@ -1634,8 +1527,8 @@ import autograd.numpy as np import matplotlib.pyplot as plt from autograd import grad -def CostOLS(beta): - return (1.0/n)*np.sum((y-X @ beta)**2) +def CostOLS(theta): + return (1.0/n)*np.sum((y-X @ theta)**2) n = 100 x = 2*np.random.rand(n,1) @@ -1857,8 +1750,8 @@ import numpy as np import autograd.numpy as np from autograd import grad -def CostOLS(beta): - return (1.0/n)*np.sum((y-X @ beta)**2) +def CostOLS(theta): + return (1.0/n)*np.sum((y-X @ theta)**2) n = 100 x = 2*np.random.rand(n,1) @@ -1866,24 +1759,24 @@ y = 4+3*x+5*x*x X = np.c_[np.ones((n,1)), x, x*x] XT_X = X.T @ X -beta_linreg = np.linalg.pinv(XT_X) @ (X.T @ y) +theta_linreg = np.linalg.pinv(XT_X) @ (X.T @ y) print("Own inversion") -print(beta_linreg) +print(theta_linreg) # Hessian matrix H = (2.0/n)* XT_X -# Note that here the Hessian does not depend on the parameters beta +# Note that here the Hessian does not depend on the parameters theta invH = np.linalg.pinv(H) -beta = np.random.randn(3,1) +theta = np.random.randn(3,1) Niterations = 5 # define the gradient training_gradient = grad(CostOLS) for iter in range(Niterations): - gradients = training_gradient(beta) - beta -= invH @ gradients + gradients = training_gradient(theta) + theta -= invH @ gradients print(iter,gradients[0],gradients[1]) -print("beta from own Newton code") -print(beta) +print("theta from own Newton code") +print(theta)
    @@ -2048,9 +1941,9 @@ theta = np.random.randn(3,1) # Value for learning rate eta = 0.01 -# Value for parameters beta1 and beta2, see https://arxiv.org/abs/1412.6980 -beta1 = 0.9 -beta2 = 0.999 +# Value for parameters theta1 and theta2, see https://arxiv.org/abs/1412.6980 +theta1 = 0.9 +theta2 = 0.999 # Including AdaGrad parameter to avoid possible division by zero delta = 1e-7 iter = 0 @@ -2064,10 +1957,10 @@ for epoch in range(n_epochs): yi = y[random_index:random_index+M] gradients = (1.0/M)*training_gradient(yi, xi, theta) # Computing moments first - first_moment = beta1*first_moment + (1-beta1)*gradients - second_moment = beta2*second_moment+(1-beta2)*gradients*gradients - first_term = first_moment/(1.0-beta1**iter) - second_term = second_moment/(1.0-beta2**iter) + first_moment = theta1*first_moment + (1-theta1)*gradients + second_moment = theta2*second_moment+(1-theta2)*gradients*gradients + first_term = first_moment/(1.0-theta1**iter) + second_term = second_moment/(1.0-theta2**iter) # Scaling with rho the new and the previous results update = eta*first_term/(np.sqrt(second_term)+delta) theta -= update @@ -2085,9 +1978,7 @@ print(theta)

  • Work on project 1

    • - +

    For more discussions of Ridge regression and calculation of averages, Wessel van Wieringen’s article is highly recommended.

    Reminder on different scaling methods#

    @@ -2650,6 +2541,7 @@ centered matrix and/or vector that enter the fitting procedure.

  • The Hessian matrix for Ridge Regression
  • Program example for gradient descent with Ridge Regression
  • Using gradient descent methods, limitations
    • +
  • Momentum based GD
  • Improving gradient descent with momentum
  • Same code but now with momentum gradient descent
  • Overview video on Stochastic Gradient Descent (SGD)
    • @@ -2668,23 +2560,25 @@ centered matrix and/or vector that enter the fitting procedure.

  • Time decay rate
  • Code with a Number of Minibatches which varies
  • Replace or not
    • -
  • Momentum based GD
    • -
  • More on momentum based approaches
    • -
  • Momentum parameter
  • Second moment of the gradient
  • Challenge: Choosing a Fixed Learning Rate
  • Motivation for Adaptive Step Sizes
    • +
  • AdaGrad algorithm, taken from Goodfellow et al
  • Derivation of the AdaGrad Algorithm
  • AdaGrad Update Rule Derivation
  • AdaGrad Properties
  • RMSProp: Adaptive Learning Rates
    • -
  • Adam Optimizer
    • -
  • RMS prop
    • -
  • ADAM optimizer
    • -
  • Algorithms and codes for Adagrad, RMSprop and Adam
    • -
  • AdaGrad algorithm, taken from Goodfellow et al
  • RMSProp algorithm, taken from Goodfellow et al
    • +
  • Adam Optimizer
    • +
  • ADAM optimizer
    • +
  • Why Combine Momentum and RMSProp?
    • +
  • Adam: Exponential Moving Averages (Moments)
    • +
  • Adam: Bias Correction
    • +
  • Adam: Update Rule Derivation
    • +
  • Adam vs. AdaGrad and RMSProp
    • +
  • Adaptivity Across Dimensions
  • ADAM algorithm, taken from Goodfellow et al
    • +
  • Algorithms and codes for Adagrad, RMSprop and Adam
  • Practical tips
  • Sneaking in automatic differentiation using Autograd
  • Same code but now with momentum gradient descent
    • diff --git a/doc/LectureNotes/_build/jupyter_execute/week37.ipynb b/doc/LectureNotes/_build/jupyter_execute/week37.ipynb index 7d4bf87cb..63e30ded3 100644 --- a/doc/LectureNotes/_build/jupyter_execute/week37.ipynb +++ b/doc/LectureNotes/_build/jupyter_execute/week37.ipynb @@ -2,7 +2,7 @@ "cells": [ { "cell_type": "markdown", - "id": "c543a71a", + "id": "4f3beaf1", "metadata": { "editable": true }, @@ -14,7 +14,7 @@ }, { "cell_type": "markdown", - "id": "4ff6b050", + "id": "46fe6a26", "metadata": { "editable": true }, @@ -29,7 +29,7 @@ }, { "cell_type": "markdown", - "id": "a7079d3e", + "id": "f7c7fee9", "metadata": { "editable": true }, @@ -52,7 +52,7 @@ }, { "cell_type": "markdown", - "id": "db3af8cc", + "id": "29a5df4a", "metadata": { "editable": true }, @@ -69,7 +69,7 @@ }, { "cell_type": "markdown", - "id": "79d6d5be", + "id": "d1cabb8b", "metadata": { "editable": true }, @@ -79,7 +79,7 @@ }, { "cell_type": "markdown", - "id": "fa427fc3", + "id": "c1499668", "metadata": { "editable": true }, @@ -103,20 +103,21 @@ { "cell_type": "code", "execution_count": 1, - "id": "1185629e", + "id": "8f93b870", "metadata": { "collapsed": false, "editable": true }, "outputs": [], "source": [ + "import numpy as np\n", "x = 2*np.random.rand(m,1)\n", "y = 4+3*x+np.random.randn(m,1)" ] }, { "cell_type": "markdown", - "id": "42dfe293", + "id": "cafa88bc", "metadata": { "editable": true }, @@ -127,7 +128,7 @@ }, { "cell_type": "markdown", - "id": "a726cc6e", + "id": "d46f9bbc", "metadata": { "editable": true }, @@ -139,7 +140,7 @@ }, { "cell_type": "markdown", - "id": "991c8919", + "id": "c6860f97", "metadata": { "editable": true }, @@ -149,7 +150,7 @@ }, { "cell_type": "markdown", - "id": "ba895122", + "id": "5961000b", "metadata": { "editable": true }, @@ -161,7 +162,7 @@ }, { "cell_type": "markdown", - "id": "5c2b9317", + "id": "602cdea2", "metadata": { "editable": true }, @@ -175,7 +176,7 @@ }, { "cell_type": "markdown", - "id": "264eef67", + "id": "2a6ab00e", "metadata": { "editable": true }, @@ -191,17 +192,17 @@ }, { "cell_type": "markdown", - "id": "a451a9ce", + "id": "f9fb9a51", "metadata": { "editable": true }, "source": [ - "The cost/loss/risk function is given by (" + "The cost/loss/risk function is given by" ] }, { "cell_type": "markdown", - "id": "e7fa569b", + "id": "a2f3adf9", "metadata": { "editable": true }, @@ -213,7 +214,7 @@ }, { "cell_type": "markdown", - "id": "a2655bc6", + "id": "e4cffe8e", "metadata": { "editable": true }, @@ -223,7 +224,7 @@ }, { "cell_type": "markdown", - "id": "277ace2e", + "id": "3ebf7860", "metadata": { "editable": true }, @@ -235,7 +236,7 @@ }, { "cell_type": "markdown", - "id": "75bbf03c", + "id": "f8cdd774", "metadata": { "editable": true }, @@ -249,7 +250,7 @@ }, { "cell_type": "markdown", - "id": "d5bb2db2", + "id": "67a028fe", "metadata": { "editable": true }, @@ -259,7 +260,7 @@ }, { "cell_type": "markdown", - "id": "53430d47", + "id": "28b34be8", "metadata": { "editable": true }, @@ -270,7 +271,7 @@ }, { "cell_type": "markdown", - "id": "89dce328", + "id": "7068c946", "metadata": { "editable": true }, @@ -285,7 +286,7 @@ }, { "cell_type": "markdown", - "id": "542b8278", + "id": "289d7cf9", "metadata": { "editable": true }, @@ -295,37 +296,37 @@ }, { "cell_type": "markdown", - "id": "2b94baca", + "id": "3bacfa14", "metadata": { "editable": true }, "source": [ "## Simple program\n", "\n", - "We can now write a program that minimizes $C(\\theta)$ using the gradient descent method with a constant learning rate $\\gamma$ according to" + "We can now write a program that minimizes $C(\\theta)$ using the gradient descent method with a constant learning rate $\\eta$ according to" ] }, { "cell_type": "markdown", - "id": "bfc6b93f", + "id": "16c159ac", "metadata": { "editable": true }, "source": [ "$$\n", - "\\theta_{k+1} = \\theta_k - \\gamma \\nabla_\\theta C(\\theta_k), \\ k=0,1,\\cdots\n", + "\\theta_{k+1} = \\theta_k - \\eta \\nabla_\\theta C(\\theta_k), \\ k=0,1,\\cdots\n", "$$" ] }, { "cell_type": "markdown", - "id": "703de92e", + "id": "cf04e4b0", "metadata": { "editable": true }, "source": [ "We can use the expression we computed for the gradient and let use a\n", - "$\\theta_0$ be chosen randomly and let $\\gamma = 0.001$. Stop iterating\n", + "$\\theta_0$ be chosen randomly and let $\\eta = 0.001$. Stop iterating\n", "when $||\\nabla_\\theta C(\\theta_k) || \\leq \\epsilon = 10^{-8}$. **Note that the code below does not include the latter stop criterion**.\n", "\n", "And finally we can compare our solution for $\\theta$ with the analytic result given by \n", @@ -334,7 +335,7 @@ }, { "cell_type": "markdown", - "id": "f843aae1", + "id": "0d9644f3", "metadata": { "editable": true }, @@ -347,7 +348,7 @@ { "cell_type": "code", "execution_count": 2, - "id": "b76443c9", + "id": "33a70e79", "metadata": { "collapsed": false, "editable": true @@ -406,7 +407,7 @@ }, { "cell_type": "markdown", - "id": "7b75f4ec", + "id": "85b9190b", "metadata": { "editable": true }, @@ -418,7 +419,7 @@ }, { "cell_type": "markdown", - "id": "47ecdee7", + "id": "c6bf4720", "metadata": { "editable": true }, @@ -430,7 +431,7 @@ }, { "cell_type": "markdown", - "id": "ff25299b", + "id": "8f766543", "metadata": { "editable": true }, @@ -440,7 +441,7 @@ }, { "cell_type": "markdown", - "id": "21a88a84", + "id": "31c687f3", "metadata": { "editable": true }, @@ -454,7 +455,7 @@ }, { "cell_type": "markdown", - "id": "67491c15", + "id": "ff33ba55", "metadata": { "editable": true }, @@ -464,7 +465,7 @@ }, { "cell_type": "markdown", - "id": "f1ac13f1", + "id": "02d68d7d", "metadata": { "editable": true }, @@ -476,7 +477,7 @@ }, { "cell_type": "markdown", - "id": "67d83802", + "id": "aee5a999", "metadata": { "editable": true }, @@ -487,7 +488,7 @@ }, { "cell_type": "markdown", - "id": "75935d70", + "id": "d4c6e860", "metadata": { "editable": true }, @@ -502,7 +503,7 @@ }, { "cell_type": "markdown", - "id": "4707a65d", + "id": "d0b1ab36", "metadata": { "editable": true }, @@ -516,7 +517,7 @@ }, { "cell_type": "markdown", - "id": "ab16b163", + "id": "feffa94a", "metadata": { "editable": true }, @@ -527,7 +528,7 @@ { "cell_type": "code", "execution_count": 3, - "id": "fe7ee1f7", + "id": "17b7aebb", "metadata": { "collapsed": false, "editable": true @@ -588,7 +589,7 @@ }, { "cell_type": "markdown", - "id": "823fdaf7", + "id": "845d5551", "metadata": { "editable": true }, @@ -610,20 +611,33 @@ }, { "cell_type": "markdown", - "id": "21ea3f2a", + "id": "55818d27", "metadata": { "editable": true }, "source": [ - "## Improving gradient descent with momentum\n", + "## Momentum based GD\n", "\n", - "We discuss here some simple examples where we introduce what is called 'memory'about previous steps, or what is normally called momentum gradient descent. The mathematics is explained below in connection with Stochastic gradient descent." + "We discuss here some simple examples where we introduce what is called\n", + "'memory'about previous steps, or what is normally called momentum\n", + "gradient descent.\n", + "For the mathematical details, see whiteboad notes from lecture on September 8, 2025." + ] + }, + { + "cell_type": "markdown", + "id": "bdfa4705", + "metadata": { + "editable": true + }, + "source": [ + "## Improving gradient descent with momentum" ] }, { "cell_type": "code", "execution_count": 4, - "id": "5b729a7e", + "id": "b13373bb", "metadata": { "collapsed": false, "editable": true @@ -689,7 +703,7 @@ }, { "cell_type": "markdown", - "id": "209fc69c", + "id": "01f07f3c", "metadata": { "editable": true }, @@ -700,7 +714,7 @@ { "cell_type": "code", "execution_count": 5, - "id": "984c4202", + "id": "dd296802", "metadata": { "collapsed": false, "editable": true @@ -774,7 +788,7 @@ }, { "cell_type": "markdown", - "id": "8577b238", + "id": "2b995d39", "metadata": { "editable": true }, @@ -793,7 +807,7 @@ }, { "cell_type": "markdown", - "id": "c11d0874", + "id": "ec3098ab", "metadata": { "editable": true }, @@ -814,7 +828,7 @@ }, { "cell_type": "markdown", - "id": "64ad52d0", + "id": "ae3c5061", "metadata": { "editable": true }, @@ -830,7 +844,7 @@ }, { "cell_type": "markdown", - "id": "5dd4c467", + "id": "54028ccc", "metadata": { "editable": true }, @@ -844,7 +858,7 @@ }, { "cell_type": "markdown", - "id": "42904823", + "id": "17875f98", "metadata": { "editable": true }, @@ -872,7 +886,7 @@ }, { "cell_type": "markdown", - "id": "49dfaf2e", + "id": "40862db0", "metadata": { "editable": true }, @@ -904,7 +918,7 @@ }, { "cell_type": "markdown", - "id": "284b8663", + "id": "075431a2", "metadata": { "editable": true }, @@ -921,20 +935,20 @@ }, { "cell_type": "markdown", - "id": "5e8cc1b8", + "id": "65dcd718", "metadata": { "editable": true }, "source": [ "$$\n", - "C(\\mathbf{\\beta}) = \\sum_{i=1}^n c_i(\\mathbf{x}_i,\n", - "\\mathbf{\\beta}).\n", + "C(\\mathbf{\\theta}) = \\sum_{i=1}^n c_i(\\mathbf{x}_i,\n", + "\\mathbf{\\theta}).\n", "$$" ] }, { "cell_type": "markdown", - "id": "510505a0", + "id": "ccf20bf2", "metadata": { "editable": true }, @@ -947,20 +961,20 @@ }, { "cell_type": "markdown", - "id": "32fc501f", + "id": "7cf7a667", "metadata": { "editable": true }, "source": [ "$$\n", - "\\nabla_\\beta C(\\mathbf{\\beta}) = \\sum_i^n \\nabla_\\beta c_i(\\mathbf{x}_i,\n", - "\\mathbf{\\beta}).\n", + "\\nabla_\\theta C(\\mathbf{\\theta}) = \\sum_i^n \\nabla_\\theta c_i(\\mathbf{x}_i,\n", + "\\mathbf{\\theta}).\n", "$$" ] }, { "cell_type": "markdown", - "id": "9dcac7ef", + "id": "21eeb773", "metadata": { "editable": true }, @@ -974,7 +988,7 @@ }, { "cell_type": "markdown", - "id": "c9464120", + "id": "562bbf59", "metadata": { "editable": true }, @@ -996,22 +1010,22 @@ }, { "cell_type": "markdown", - "id": "59bc847c", + "id": "1b9e6fb3", "metadata": { "editable": true }, "source": [ "$$\n", - "\\nabla_{\\beta}\n", - "C(\\mathbf{\\beta}) = \\sum_{i=1}^n \\nabla_\\beta c_i(\\mathbf{x}_i,\n", - "\\mathbf{\\beta}) \\rightarrow \\sum_{i \\in B_k}^n \\nabla_\\beta\n", - "c_i(\\mathbf{x}_i, \\mathbf{\\beta}).\n", + "\\nabla_{\\theta}\n", + "C(\\mathbf{\\theta}) = \\sum_{i=1}^n \\nabla_\\theta c_i(\\mathbf{x}_i,\n", + "\\mathbf{\\theta}) \\rightarrow \\sum_{i \\in B_k}^n \\nabla_\\theta\n", + "c_i(\\mathbf{x}_i, \\mathbf{\\theta}).\n", "$$" ] }, { "cell_type": "markdown", - "id": "ba220299", + "id": "4cd95b23", "metadata": { "editable": true }, @@ -1023,20 +1037,20 @@ }, { "cell_type": "markdown", - "id": "8608ba34", + "id": "ea4324e1", "metadata": { "editable": true }, "source": [ "$$\n", - "\\beta_{j+1} = \\beta_j - \\gamma_j \\sum_{i \\in B_k}^n \\nabla_\\beta c_i(\\mathbf{x}_i,\n", - "\\mathbf{\\beta})\n", + "\\theta_{j+1} = \\theta_j - \\eta_j \\sum_{i \\in B_k}^n \\nabla_\\theta c_i(\\mathbf{x}_i,\n", + "\\mathbf{\\theta})\n", "$$" ] }, { "cell_type": "markdown", - "id": "0faa1782", + "id": "5287666c", "metadata": { "editable": true }, @@ -1050,7 +1064,7 @@ }, { "cell_type": "markdown", - "id": "97eddb68", + "id": "596acab8", "metadata": { "editable": true }, @@ -1061,7 +1075,7 @@ { "cell_type": "code", "execution_count": 6, - "id": "309da17e", + "id": "3b4d8b03", "metadata": { "collapsed": false, "editable": true @@ -1086,7 +1100,7 @@ }, { "cell_type": "markdown", - "id": "7aea4344", + "id": "f4f09885", "metadata": { "editable": true }, @@ -1102,7 +1116,7 @@ }, { "cell_type": "markdown", - "id": "52f8d549", + "id": "156e8953", "metadata": { "editable": true }, @@ -1117,20 +1131,20 @@ "that we are close to a local/global minimum. However, we could also\n", "evaluate the cost function at this point, store the result and\n", "continue the search. If the test kicks in at a later stage we can\n", - "compare the values of the cost function and keep the $\\beta$ that\n", + "compare the values of the cost function and keep the $\\theta$ that\n", "gave the lowest value." ] }, { "cell_type": "markdown", - "id": "dcd04de7", + "id": "7076d164", "metadata": { "editable": true }, "source": [ "## Slightly different approach\n", "\n", - "Another approach is to let the step length $\\gamma_j$ depend on the\n", + "Another approach is to let the step length $\\eta_j$ depend on the\n", "number of epochs in such a way that it becomes very small after a\n", "reasonable time such that we do not move at all. Such approaches are\n", "also called scaling. There are many such ways to [scale the learning\n", @@ -1143,26 +1157,26 @@ }, { "cell_type": "markdown", - "id": "4ecf0b16", + "id": "d822d1d4", "metadata": { "editable": true }, "source": [ "## Time decay rate\n", "\n", - "As an example, let $e = 0,1,2,3,\\cdots$ denote the current epoch and let $t_0, t_1 > 0$ be two fixed numbers. Furthermore, let $t = e \\cdot m + i$ where $m$ is the number of minibatches and $i=0,\\cdots,m-1$. Then the function $$\\gamma_j(t; t_0, t_1) = \\frac{t_0}{t+t_1} $$ goes to zero as the number of epochs gets large. I.e. we start with a step length $\\gamma_j (0; t_0, t_1) = t_0/t_1$ which decays in *time* $t$.\n", + "As an example, let $e = 0,1,2,3,\\cdots$ denote the current epoch and let $t_0, t_1 > 0$ be two fixed numbers. Furthermore, let $t = e \\cdot m + i$ where $m$ is the number of minibatches and $i=0,\\cdots,m-1$. Then the function $$\\eta_j(t; t_0, t_1) = \\frac{t_0}{t+t_1} $$ goes to zero as the number of epochs gets large. I.e. we start with a step length $\\eta_j (0; t_0, t_1) = t_0/t_1$ which decays in *time* $t$.\n", "\n", - "In this way we can fix the number of epochs, compute $\\beta$ and\n", + "In this way we can fix the number of epochs, compute $\\theta$ and\n", "evaluate the cost function at the end. Repeating the computation will\n", "give a different result since the scheme is random by design. Then we\n", - "pick the final $\\beta$ that gives the lowest value of the cost\n", + "pick the final $\\theta$ that gives the lowest value of the cost\n", "function." ] }, { "cell_type": "code", "execution_count": 7, - "id": "71eb89db", + "id": "c8987f85", "metadata": { "collapsed": false, "editable": true @@ -1181,23 +1195,23 @@ "t0 = 1.0\n", "t1 = 10\n", "\n", - "gamma_j = t0/t1\n", + "eta_j = t0/t1\n", "j = 0\n", "for epoch in range(1,n_epochs+1):\n", " for i in range(m):\n", " k = np.random.randint(m) #Pick the k-th minibatch at random\n", " #Compute the gradient using the data in minibatch Bk\n", - " #Compute new suggestion for beta\n", + " #Compute new suggestion for theta\n", " t = epoch*m+i\n", - " gamma_j = step_length(t,t0,t1)\n", + " eta_j = step_length(t,t0,t1)\n", " j += 1\n", "\n", - "print(\"gamma_j after %d epochs: %g\" % (n_epochs,gamma_j))" + "print(\"eta_j after %d epochs: %g\" % (n_epochs,eta_j))" ] }, { "cell_type": "markdown", - "id": "10ce3fc7", + "id": "4c72814e", "metadata": { "editable": true }, @@ -1210,7 +1224,7 @@ { "cell_type": "code", "execution_count": 8, - "id": "f2d402a8", + "id": "7b243058", "metadata": { "collapsed": false, "editable": true @@ -1287,7 +1301,7 @@ }, { "cell_type": "markdown", - "id": "ffd1e92d", + "id": "e008b37c", "metadata": { "editable": true }, @@ -1302,267 +1316,7 @@ }, { "cell_type": "markdown", - "id": "4b72d3f9", - "metadata": { - "editable": true - }, - "source": [ - "## Momentum based GD\n", - "\n", - "The stochastic gradient descent (SGD) is almost always used with a\n", - "*momentum* or inertia term that serves as a memory of the direction we\n", - "are moving in parameter space. This is typically implemented as\n", - "follows" - ] - }, - { - "cell_type": "markdown", - "id": "1b60ec02", - "metadata": { - "editable": true - }, - "source": [ - "$$\n", - "\\mathbf{v}_{t}=\\gamma \\mathbf{v}_{t-1}+\\eta_{t}\\nabla_\\theta E(\\boldsymbol{\\theta}_t) \\nonumber\n", - "$$" - ] - }, - { - "cell_type": "markdown", - "id": "e1fa24ba", - "metadata": { - "editable": true - }, - "source": [ - "\n", - "
    \n", - "\n", - "$$\n", - "\\begin{equation} \n", - "\\boldsymbol{\\theta}_{t+1}= \\boldsymbol{\\theta}_t -\\mathbf{v}_{t},\n", - "\\label{_auto1} \\tag{1}\n", - "\\end{equation}\n", - "$$" - ] - }, - { - "cell_type": "markdown", - "id": "7a1fc9fc", - "metadata": { - "editable": true - }, - "source": [ - "where we have introduced a momentum parameter $\\gamma$, with\n", - "$0\\le\\gamma\\le 1$, and for brevity we dropped the explicit notation to\n", - "indicate the gradient is to be taken over a different mini-batch at\n", - "each step. We call this algorithm gradient descent with momentum\n", - "(GDM). From these equations, it is clear that $\\mathbf{v}_t$ is a\n", - "running average of recently encountered gradients and\n", - "$(1-\\gamma)^{-1}$ sets the characteristic time scale for the memory\n", - "used in the averaging procedure. Consistent with this, when\n", - "$\\gamma=0$, this just reduces down to ordinary SGD as discussed\n", - "earlier. An equivalent way of writing the updates is" - ] - }, - { - "cell_type": "markdown", - "id": "23abda5d", - "metadata": { - "editable": true - }, - "source": [ - "$$\n", - "\\Delta \\boldsymbol{\\theta}_{t+1} = \\gamma \\Delta \\boldsymbol{\\theta}_t -\\ \\eta_{t}\\nabla_\\theta E(\\boldsymbol{\\theta}_t),\n", - "$$" - ] - }, - { - "cell_type": "markdown", - "id": "0f0067a9", - "metadata": { - "editable": true - }, - "source": [ - "where we have defined $\\Delta \\boldsymbol{\\theta}_{t}= \\boldsymbol{\\theta}_t-\\boldsymbol{\\theta}_{t-1}$." - ] - }, - { - "cell_type": "markdown", - "id": "b86ec959", - "metadata": { - "editable": true - }, - "source": [ - "## More on momentum based approaches\n", - "\n", - "Let us try to get more intuition from these equations. It is helpful\n", - "to consider a simple physical analogy with a particle of mass $m$\n", - "moving in a viscous medium with drag coefficient $\\mu$ and potential\n", - "$E(\\mathbf{w})$. If we denote the particle's position by $\\mathbf{w}$,\n", - "then its motion is described by" - ] - }, - { - "cell_type": "markdown", - "id": "7685ff97", - "metadata": { - "editable": true - }, - "source": [ - "$$\n", - "m {d^2 \\mathbf{w} \\over dt^2} + \\mu {d \\mathbf{w} \\over dt }= -\\nabla_w E(\\mathbf{w}).\n", - "$$" - ] - }, - { - "cell_type": "markdown", - "id": "28203c1e", - "metadata": { - "editable": true - }, - "source": [ - "We can discretize this equation in the usual way to get" - ] - }, - { - "cell_type": "markdown", - "id": "1495b23f", - "metadata": { - "editable": true - }, - "source": [ - "$$\n", - "m { \\mathbf{w}_{t+\\Delta t}-2 \\mathbf{w}_{t} +\\mathbf{w}_{t-\\Delta t} \\over (\\Delta t)^2}+\\mu {\\mathbf{w}_{t+\\Delta t}- \\mathbf{w}_{t} \\over \\Delta t} = -\\nabla_w E(\\mathbf{w}).\n", - "$$" - ] - }, - { - "cell_type": "markdown", - "id": "15bbc6e6", - "metadata": { - "editable": true - }, - "source": [ - "Rearranging this equation, we can rewrite this as" - ] - }, - { - "cell_type": "markdown", - "id": "ffeffc05", - "metadata": { - "editable": true - }, - "source": [ - "$$\n", - "\\Delta \\mathbf{w}_{t +\\Delta t}= - { (\\Delta t)^2 \\over m +\\mu \\Delta t} \\nabla_w E(\\mathbf{w})+ {m \\over m +\\mu \\Delta t} \\Delta \\mathbf{w}_t.\n", - "$$" - ] - }, - { - "cell_type": "markdown", - "id": "166cfafc", - "metadata": { - "editable": true - }, - "source": [ - "## Momentum parameter\n", - "\n", - "Notice that this equation is identical to previous one if we identify\n", - "the position of the particle, $\\mathbf{w}$, with the parameters\n", - "$\\boldsymbol{\\theta}$. This allows us to identify the momentum\n", - "parameter and learning rate with the mass of the particle and the\n", - "viscous drag as:" - ] - }, - { - "cell_type": "markdown", - "id": "486fadf0", - "metadata": { - "editable": true - }, - "source": [ - "$$\n", - "\\gamma= {m \\over m +\\mu \\Delta t }, \\qquad \\eta = {(\\Delta t)^2 \\over m +\\mu \\Delta t}.\n", - "$$" - ] - }, - { - "cell_type": "markdown", - "id": "cdb0769d", - "metadata": { - "editable": true - }, - "source": [ - "Thus, as the name suggests, the momentum parameter is proportional to\n", - "the mass of the particle and effectively provides inertia.\n", - "Furthermore, in the large viscosity/small learning rate limit, our\n", - "memory time scales as $(1-\\gamma)^{-1} \\approx m/(\\mu \\Delta t)$.\n", - "\n", - "Why is momentum useful? SGD momentum helps the gradient descent\n", - "algorithm gain speed in directions with persistent but small gradients\n", - "even in the presence of stochasticity, while suppressing oscillations\n", - "in high-curvature directions. This becomes especially important in\n", - "situations where the landscape is shallow and flat in some directions\n", - "and narrow and steep in others. It has been argued that first-order\n", - "methods (with appropriate initial conditions) can perform comparable\n", - "to more expensive second order methods, especially in the context of\n", - "complex deep learning models.\n", - "\n", - "These beneficial properties of momentum can sometimes become even more\n", - "pronounced by using a slight modification of the classical momentum\n", - "algorithm called Nesterov Accelerated Gradient (NAG).\n", - "\n", - "In the NAG algorithm, rather than calculating the gradient at the\n", - "current parameters, $\\nabla_\\theta E(\\boldsymbol{\\theta}_t)$, one\n", - "calculates the gradient at the expected value of the parameters given\n", - "our current momentum, $\\nabla_\\theta E(\\boldsymbol{\\theta}_t +\\gamma\n", - "\\mathbf{v}_{t-1})$. This yields the NAG update rule" - ] - }, - { - "cell_type": "markdown", - "id": "e65569bb", - "metadata": { - "editable": true - }, - "source": [ - "$$\n", - "\\mathbf{v}_{t}=\\gamma \\mathbf{v}_{t-1}+\\eta_{t}\\nabla_\\theta E(\\boldsymbol{\\theta}_t +\\gamma \\mathbf{v}_{t-1}) \\nonumber\n", - "$$" - ] - }, - { - "cell_type": "markdown", - "id": "9d72d316", - "metadata": { - "editable": true - }, - "source": [ - "\n", - "
    \n", - "\n", - "$$\n", - "\\begin{equation} \n", - "\\boldsymbol{\\theta}_{t+1}= \\boldsymbol{\\theta}_t -\\mathbf{v}_{t}.\n", - "\\label{_auto2} \\tag{2}\n", - "\\end{equation}\n", - "$$" - ] - }, - { - "cell_type": "markdown", - "id": "b7b9811e", - "metadata": { - "editable": true - }, - "source": [ - "One of the major advantages of NAG is that it allows for the use of a\n", - "larger learning rate than GDM for the same choice of $\\gamma$." - ] - }, - { - "cell_type": "markdown", - "id": "d8f3c523", + "id": "b921e0a8", "metadata": { "editable": true }, @@ -1593,7 +1347,7 @@ }, { "cell_type": "markdown", - "id": "22d6be74", + "id": "0ce23497", "metadata": { "editable": true }, @@ -1615,14 +1369,14 @@ }, { "cell_type": "markdown", - "id": "091ace5a", + "id": "072e6d0d", "metadata": { "editable": true }, "source": [ "## Motivation for Adaptive Step Sizes\n", "\n", - "1. Instead of a fixed global $\\eta$, use an \\textbf{adaptive learning rate} for each parameter that depends on the history of gradients.\n", + "1. Instead of a fixed global $\\eta$, use an **adaptive learning rate** for each parameter that depends on the history of gradients.\n", "\n", "2. Parameters that have large accumulated gradient magnitude should get smaller steps (they've been changing a lot), whereas parameters with small or infrequent gradients can have larger relative steps.\n", "\n", @@ -1635,457 +1389,7 @@ }, { "cell_type": "markdown", - "id": "5c05fb8b", - "metadata": { - "editable": true - }, - "source": [ - "## Derivation of the AdaGrad Algorithm\n", - "\n", - "**Accumulating Gradient History.**\n", - "\n", - "1. AdaGrad maintains a running sum of squared gradients for each parameter (coordinate)\n", - "\n", - "2. Let $g_t = \\nabla \\ell_{i_t}(x_t)$ be the gradient at step $t$ (or a subgradient for nondifferentiable cases).\n", - "\n", - "3. Initialize $h_0 = 0$ (an all-zero vector in $\\mathbb{R}^d$).\n", - "\n", - "4. At each iteration $t$, update the accumulation:" - ] - }, - { - "cell_type": "markdown", - "id": "21fb8bd0", - "metadata": { - "editable": true - }, - "source": [ - "$$\n", - "h_t \\;=\\; h_{t-1} + g_t \\circ g_t,\n", - "$$" - ] - }, - { - "cell_type": "markdown", - "id": "f71dfafa", - "metadata": { - "editable": true - }, - "source": [ - "1. Here $g_t \\circ g_t$ denotes element-wise square of the gradient vector. $h_t^{(j)} = h_{t-1}^{(j)} + (g_{t,j})^2$ for each parameter $j$.\n", - "\n", - "2. We can view $H_t = \\mathrm{diag}(h_t)$ as a diagonal matrix of past squared gradients. Initially $H_0 = 0$." - ] - }, - { - "cell_type": "markdown", - "id": "64869e2e", - "metadata": { - "editable": true - }, - "source": [ - "## AdaGrad Update Rule Derivation\n", - "\n", - "We scale the gradient by the inverse square root of the accumulated matrix $H_t$. The AdaGrad update at step $t$ is:" - ] - }, - { - "cell_type": "markdown", - "id": "c2d8d6b1", - "metadata": { - "editable": true - }, - "source": [ - "$$\n", - "x_{t+1} \\;=\\; x_t \\;-\\; \\eta\\, H_t^{-1/2}\\, g_t,\n", - "$$" - ] - }, - { - "cell_type": "markdown", - "id": "2171198a", - "metadata": { - "editable": true - }, - "source": [ - "where $H_t^{-1/2}$ is the diagonal matrix with entries $(h_{t}^{(1)})^{-1/2}, \\dots, (h_{t}^{(d)})^{-1/2}$\n", - "In coordinates, this means each parameter $j$ has an individual step size:" - ] - }, - { - "cell_type": "markdown", - "id": "353e21d3", - "metadata": { - "editable": true - }, - "source": [ - "$$\n", - "x_{t+1,j} =x_{t,j} -\\frac{\\eta}{\\sqrt{h_{t,j}}}g_{t,j}.\n", - "$$" - ] - }, - { - "cell_type": "markdown", - "id": "b59e6add", - "metadata": { - "editable": true - }, - "source": [ - "In practice we add a small constant $\\epsilon$ in the denominator for numerical stability to avoid division by zero:" - ] - }, - { - "cell_type": "markdown", - "id": "707caefe", - "metadata": { - "editable": true - }, - "source": [ - "$$\n", - "x_{t+1,j}= x_{t,j}-\\frac{\\eta}{\\sqrt{\\epsilon + h_{t,j}}}g_{t,j}.\n", - "$$" - ] - }, - { - "cell_type": "markdown", - "id": "62dafce4", - "metadata": { - "editable": true - }, - "source": [ - "Equivalently, the effective learning rate for parameter $j$ at time $t$ is $\\displaystyle \\alpha_{t,j} = \\frac{\\eta}{\\sqrt{\\epsilon + h_{t,j}}}$. This decreases over time as $h_{t,j}$ grows." - ] - }, - { - "cell_type": "markdown", - "id": "01a2ba61", - "metadata": { - "editable": true - }, - "source": [ - "## AdaGrad Properties\n", - "1. AdaGrad automatically tunes the step size for each parameter. Parameters with more **volatile or large gradients** get smaller steps, and those with **small or infrequent gradients** get relatively larger steps\n", - "\n", - "2. No manual schedule needed: The accumulation $h_t$ keeps increasing (or stays the same if gradient is zero), so step sizes $\\eta/\\sqrt{h_t}$ are non-increasing. This has a similar effect to a learning rate schedule, but individualized per coordinate.\n", - "\n", - "3. Sparse data benefit: For very sparse features, $h_{t,j}$ grows slowly, so that feature’s parameter retains a higher learning rate for longer, allowing it to make significant updates when it does get a gradient signal\n", - "\n", - "4. Convergence: In convex optimization, AdaGrad can be shown to achieve a sub-linear convergence rate (e.g. $O(1/\\sqrt{T})$ regret bound) comparable to the best fixed learning rate tuned for the problem\n", - "\n", - "It effectively reduces the need to tune $\\eta$ by hand.\n", - "1. Limitations: Because $h_t$ accumulates without bound, AdaGrad’s learning rates can become extremely small over long training, potentially slowing progress. (Later variants like RMSProp, AdaDelta, Adam address this by modifying the accumulation rule.)" - ] - }, - { - "cell_type": "markdown", - "id": "b4bc88e0", - "metadata": { - "editable": true - }, - "source": [ - "## RMSProp: Adaptive Learning Rates\n", - "Addresses AdaGrad’s diminishing learning rate issue.\n", - "Uses a decaying average of squared gradients (instead of a cumulative sum):" - ] - }, - { - "cell_type": "markdown", - "id": "4933fddb", - "metadata": { - "editable": true - }, - "source": [ - "$$\n", - "v_t = \\beta_2\\, v_{t-1} + (1-\\beta_2)\\, (\\nabla L(w_t))^2,\n", - "$$" - ] - }, - { - "cell_type": "markdown", - "id": "0dd8edf7", - "metadata": { - "editable": true - }, - "source": [ - "with $\\beta_2$ typically $0.9$ (or $0.99$).\n", - "1. Update: $w_{t+1} = w_t - \\frac{\\alpha}{\\sqrt{v_t + \\epsilon}} \\nabla L(w_t)$.\n", - "\n", - "2. Recent gradients have more weight, so $v_t$ adapts to the current landscape.\n", - "\n", - "3. Avoids AdaGrad’s “infinite memory” problem – learning rate does not continuously decay to zero.\n", - "\n", - "RMSProp was first proposed in lecture notes by Geoff Hinton, 2012 - unpublished.)" - ] - }, - { - "cell_type": "markdown", - "id": "1ce98f28", - "metadata": { - "editable": true - }, - "source": [ - "## Adam Optimizer\n", - "\n", - "Why Combine Momentum and RMSProp? Motivation for Adam: Adaptive Moment Estimation (Adam) was introduced by Kingma an Ba (2014) to combine the benefits of momentum and RMSProp.\n", - "\n", - "1. Fast convergence by smoothing gradients (accelerates in long-term gradient direction).\n", - "\n", - "2. Adaptive rates (RMSProp): Per-dimension learning rate scaling for stability (handles different feature scales, sparse gradients).\n", - "\n", - "3. Adam uses both: maintains moving averages of both first moment (gradients) and second moment (squared gradients)\n", - "\n", - "4. Additionally, includes a mechanism to correct the bias in these moving averages (crucial in early iterations)\n", - "\n", - "**Result**: Adam is robust, achieves faster convergence with less tuning, and often outperforms SGD (with momentum) in practice." - ] - }, - { - "cell_type": "markdown", - "id": "d7d646bc", - "metadata": { - "editable": true - }, - "source": [ - "## RMS prop\n", - "\n", - "In RMS prop, in addition to keeping a running average of the first\n", - "moment of the gradient, we also keep track of the second moment\n", - "denoted by $\\mathbf{s}_t=\\mathbb{E}[\\mathbf{g}_t^2]$. The update rule\n", - "for RMS prop is given by" - ] - }, - { - "cell_type": "markdown", - "id": "3a212292", - "metadata": { - "editable": true - }, - "source": [ - "\n", - "
    \n", - "\n", - "$$\n", - "\\begin{equation}\n", - "\\mathbf{g}_t = \\nabla_\\theta E(\\boldsymbol{\\theta}) \n", - "\\label{_auto3} \\tag{3}\n", - "\\end{equation}\n", - "$$" - ] - }, - { - "cell_type": "markdown", - "id": "88c0ec96", - "metadata": { - "editable": true - }, - "source": [ - "$$\n", - "\\mathbf{s}_t =\\beta \\mathbf{s}_{t-1} +(1-\\beta)\\mathbf{g}_t^2 \\nonumber\n", - "$$" - ] - }, - { - "cell_type": "markdown", - "id": "9cfb794f", - "metadata": { - "editable": true - }, - "source": [ - "$$\n", - "\\boldsymbol{\\theta}_{t+1}=\\boldsymbol{\\theta}_t - \\eta_t { \\mathbf{g}_t \\over \\sqrt{\\mathbf{s}_t +\\epsilon}}, \\nonumber\n", - "$$" - ] - }, - { - "cell_type": "markdown", - "id": "aaa4936c", - "metadata": { - "editable": true - }, - "source": [ - "where $\\beta$ controls the averaging time of the second moment and is\n", - "typically taken to be about $\\beta=0.9$, $\\eta_t$ is a learning rate\n", - "typically chosen to be $10^{-3}$, and $\\epsilon\\sim 10^{-8} $ is a\n", - "small regularization constant to prevent divergences. Multiplication\n", - "and division by vectors is understood as an element-wise operation. It\n", - "is clear from this formula that the learning rate is reduced in\n", - "directions where the norm of the gradient is consistently large. This\n", - "greatly speeds up the convergence by allowing us to use a larger\n", - "learning rate for flat directions." - ] - }, - { - "cell_type": "markdown", - "id": "5493a6cd", - "metadata": { - "editable": true - }, - "source": [ - "## [ADAM optimizer](https://arxiv.org/abs/1412.6980)\n", - "\n", - "A related algorithm is the ADAM optimizer. In\n", - "[ADAM](https://arxiv.org/abs/1412.6980), we keep a running average of\n", - "both the first and second moment of the gradient and use this\n", - "information to adaptively change the learning rate for different\n", - "parameters. The method isefficient when working with large\n", - "problems involving lots data and/or parameters. It is a combination of the\n", - "gradient descent with momentum algorithm and the RMSprop algorithm\n", - "discussed above.\n", - "\n", - "In addition to keeping a running average of the first and\n", - "second moments of the gradient\n", - "(i.e. $\\mathbf{m}_t=\\mathbb{E}[\\mathbf{g}_t]$ and\n", - "$\\mathbf{s}_t=\\mathbb{E}[\\mathbf{g}^2_t]$, respectively), ADAM\n", - "performs an additional bias correction to account for the fact that we\n", - "are estimating the first two moments of the gradient using a running\n", - "average (denoted by the hats in the update rule below). The update\n", - "rule for ADAM is given by (where multiplication and division are once\n", - "again understood to be element-wise operations below)" - ] - }, - { - "cell_type": "markdown", - "id": "35795070", - "metadata": { - "editable": true - }, - "source": [ - "\n", - "
    \n", - "\n", - "$$\n", - "\\begin{equation}\n", - "\\mathbf{g}_t = \\nabla_\\theta E(\\boldsymbol{\\theta}) \n", - "\\label{_auto4} \\tag{4}\n", - "\\end{equation}\n", - "$$" - ] - }, - { - "cell_type": "markdown", - "id": "1e67e6dd", - "metadata": { - "editable": true - }, - "source": [ - "$$\n", - "\\mathbf{m}_t = \\beta_1 \\mathbf{m}_{t-1} + (1-\\beta_1) \\mathbf{g}_t \\nonumber\n", - "$$" - ] - }, - { - "cell_type": "markdown", - "id": "39d582bb", - "metadata": { - "editable": true - }, - "source": [ - "$$\n", - "\\mathbf{s}_t =\\beta_2 \\mathbf{s}_{t-1} +(1-\\beta_2)\\mathbf{g}_t^2 \\nonumber\n", - "$$" - ] - }, - { - "cell_type": "markdown", - "id": "4a7e2423", - "metadata": { - "editable": true - }, - "source": [ - "$$\n", - "\\boldsymbol{\\mathbf{m}}_t={\\mathbf{m}_t \\over 1-\\beta_1^t} \\nonumber\n", - "$$" - ] - }, - { - "cell_type": "markdown", - "id": "53da77c2", - "metadata": { - "editable": true - }, - "source": [ - "$$\n", - "\\boldsymbol{\\mathbf{s}}_t ={\\mathbf{s}_t \\over1-\\beta_2^t} \\nonumber\n", - "$$" - ] - }, - { - "cell_type": "markdown", - "id": "a4c7df45", - "metadata": { - "editable": true - }, - "source": [ - "$$\n", - "\\boldsymbol{\\theta}_{t+1}=\\boldsymbol{\\theta}_t - \\eta_t { \\boldsymbol{\\mathbf{m}}_t \\over \\sqrt{\\boldsymbol{\\mathbf{s}}_t} +\\epsilon}, \\nonumber\n", - "$$" - ] - }, - { - "cell_type": "markdown", - "id": "f645a0aa", - "metadata": { - "editable": true - }, - "source": [ - "\n", - "
    \n", - "\n", - "$$\n", - "\\begin{equation} \n", - "\\label{_auto5} \\tag{5}\n", - "\\end{equation}\n", - "$$" - ] - }, - { - "cell_type": "markdown", - "id": "505619bb", - "metadata": { - "editable": true - }, - "source": [ - "where $\\beta_1$ and $\\beta_2$ set the memory lifetime of the first and\n", - "second moment and are typically taken to be $0.9$ and $0.99$\n", - "respectively, and $\\eta$ and $\\epsilon$ are identical to RMSprop.\n", - "\n", - "Like in RMSprop, the effective step size of a parameter depends on the\n", - "magnitude of its gradient squared. To understand this better, let us\n", - "rewrite this expression in terms of the variance\n", - "$\\boldsymbol{\\sigma}_t^2 = \\boldsymbol{\\mathbf{s}}_t -\n", - "(\\boldsymbol{\\mathbf{m}}_t)^2$. Consider a single parameter $\\theta_t$. The\n", - "update rule for this parameter is given by" - ] - }, - { - "cell_type": "markdown", - "id": "6cb4c9a9", - "metadata": { - "editable": true - }, - "source": [ - "$$\n", - "\\Delta \\theta_{t+1}= -\\eta_t { \\boldsymbol{m}_t \\over \\sqrt{\\sigma_t^2 + m_t^2 }+\\epsilon}.\n", - "$$" - ] - }, - { - "cell_type": "markdown", - "id": "39627950", - "metadata": { - "editable": true - }, - "source": [ - "## Algorithms and codes for Adagrad, RMSprop and Adam\n", - "\n", - "The algorithms we have implemented are well described in the text by [Goodfellow, Bengio and Courville, chapter 8](https://www.deeplearningbook.org/contents/optimization.html).\n", - "\n", - "The codes which implement these algorithms are discussed after our presentation of automatic differentiation." - ] - }, - { - "cell_type": "markdown", - "id": "f240057d", + "id": "05c46307", "metadata": { "editable": true }, @@ -2101,7 +1405,193 @@ }, { "cell_type": "markdown", - "id": "428ee09d", + "id": "183480ab", + "metadata": { + "editable": true + }, + "source": [ + "## Derivation of the AdaGrad Algorithm\n", + "\n", + "**Accumulating Gradient History.**\n", + "\n", + "1. AdaGrad maintains a running sum of squared gradients for each parameter (coordinate)\n", + "\n", + "2. Let $g_t = \\nabla C_{i_t}(x_t)$ be the gradient at step $t$ (or a subgradient for nondifferentiable cases).\n", + "\n", + "3. Initialize $r_0 = 0$ (an all-zero vector in $\\mathbb{R}^d$).\n", + "\n", + "4. At each iteration $t$, update the accumulation:" + ] + }, + { + "cell_type": "markdown", + "id": "0bffad7b", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "r_t = r_{t-1} + g_t \\circ g_t,\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "289693d6", + "metadata": { + "editable": true + }, + "source": [ + "1. Here $g_t \\circ g_t$ denotes element-wise square of the gradient vector. $g_t^{(j)} = g_{t-1}^{(j)} + (g_{t,j})^2$ for each parameter $j$.\n", + "\n", + "2. We can view $H_t = \\mathrm{diag}(r_t)$ as a diagonal matrix of past squared gradients. Initially $H_0 = 0$." + ] + }, + { + "cell_type": "markdown", + "id": "98e5e657", + "metadata": { + "editable": true + }, + "source": [ + "## AdaGrad Update Rule Derivation\n", + "\n", + "We scale the gradient by the inverse square root of the accumulated matrix $H_t$. The AdaGrad update at step $t$ is:" + ] + }, + { + "cell_type": "markdown", + "id": "d2b6b9cd", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\theta_{t+1} =\\theta_t - \\eta H_t^{-1/2} g_t,\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "795888a2", + "metadata": { + "editable": true + }, + "source": [ + "where $H_t^{-1/2}$ is the diagonal matrix with entries $(r_{t}^{(1)})^{-1/2}, \\dots, (r_{t}^{(d)})^{-1/2}$\n", + "In coordinates, this means each parameter $j$ has an individual step size:" + ] + }, + { + "cell_type": "markdown", + "id": "283c4520", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\theta_{t+1,j} =\\theta_{t,j} -\\frac{\\eta}{\\sqrt{r_{t,j}}}g_{t,j}.\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "0958350b", + "metadata": { + "editable": true + }, + "source": [ + "In practice we add a small constant $\\epsilon$ in the denominator for numerical stability to avoid division by zero:" + ] + }, + { + "cell_type": "markdown", + "id": "f806f43a", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\theta_{t+1,j}= \\theta_{t,j}-\\frac{\\eta}{\\sqrt{\\epsilon + r_{t,j}}}g_{t,j}.\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "2c7d2de6", + "metadata": { + "editable": true + }, + "source": [ + "Equivalently, the effective learning rate for parameter $j$ at time $t$ is $\\displaystyle \\alpha_{t,j} = \\frac{\\eta}{\\sqrt{\\epsilon + r_{t,j}}}$. This decreases over time as $r_{t,j}$ grows." + ] + }, + { + "cell_type": "markdown", + "id": "afabce90", + "metadata": { + "editable": true + }, + "source": [ + "## AdaGrad Properties\n", + "\n", + "1. AdaGrad automatically tunes the step size for each parameter. Parameters with more *volatile or large gradients* get smaller steps, and those with *small or infrequent gradients* get relatively larger steps\n", + "\n", + "2. No manual schedule needed: The accumulation $r_t$ keeps increasing (or stays the same if gradient is zero), so step sizes $\\eta/\\sqrt{r_t}$ are non-increasing. This has a similar effect to a learning rate schedule, but individualized per coordinate.\n", + "\n", + "3. Sparse data benefit: For very sparse features, $r_{t,j}$ grows slowly, so that feature’s parameter retains a higher learning rate for longer, allowing it to make significant updates when it does get a gradient signal\n", + "\n", + "4. Convergence: In convex optimization, AdaGrad can be shown to achieve a sub-linear convergence rate comparable to the best fixed learning rate tuned for the problem\n", + "\n", + "It effectively reduces the need to tune $\\eta$ by hand.\n", + "1. Limitations: Because $r_t$ accumulates without bound, AdaGrad’s learning rates can become extremely small over long training, potentially slowing progress. (Later variants like RMSProp, AdaDelta, Adam address this by modifying the accumulation rule.)" + ] + }, + { + "cell_type": "markdown", + "id": "489bb0db", + "metadata": { + "editable": true + }, + "source": [ + "## RMSProp: Adaptive Learning Rates\n", + "\n", + "Addresses AdaGrad’s diminishing learning rate issue.\n", + "Uses a decaying average of squared gradients (instead of a cumulative sum):" + ] + }, + { + "cell_type": "markdown", + "id": "b5808edc", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "v_t = \\rho v_{t-1} + (1-\\rho)(\\nabla C(\\theta_t))^2,\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "c36f305f", + "metadata": { + "editable": true + }, + "source": [ + "with $\\rho$ typically $0.9$ (or $0.99$).\n", + "1. Update: $\\theta_{t+1} = \\theta_t - \\frac{\\eta}{\\sqrt{v_t + \\epsilon}} \\nabla C(\\theta_t)$.\n", + "\n", + "2. Recent gradients have more weight, so $v_t$ adapts to the current landscape.\n", + "\n", + "3. Avoids AdaGrad’s “infinite memory” problem – learning rate does not continuously decay to zero.\n", + "\n", + "RMSProp was first proposed in lecture notes by Geoff Hinton, 2012 - unpublished.)" + ] + }, + { + "cell_type": "markdown", + "id": "6e4679b3", "metadata": { "editable": true }, @@ -2117,7 +1607,251 @@ }, { "cell_type": "markdown", - "id": "9ce3cc3e", + "id": "d58547fa", + "metadata": { + "editable": true + }, + "source": [ + "## Adam Optimizer\n", + "\n", + "Why combine Momentum and RMSProp? Motivation for Adam: Adaptive Moment Estimation (Adam) was introduced by Kingma an Ba (2014) to combine the benefits of momentum and RMSProp.\n", + "\n", + "1. Fast convergence by smoothing gradients (accelerates in long-term gradient direction).\n", + "\n", + "2. Adaptive rates (RMSProp): Per-dimension learning rate scaling for stability (handles different feature scales, sparse gradients).\n", + "\n", + "3. Adam uses both: maintains moving averages of both first moment (gradients) and second moment (squared gradients)\n", + "\n", + "4. Additionally, includes a mechanism to correct the bias in these moving averages (crucial in early iterations)\n", + "\n", + "**Result**: Adam is robust, achieves faster convergence with less tuning, and often outperforms SGD (with momentum) in practice." + ] + }, + { + "cell_type": "markdown", + "id": "6989903c", + "metadata": { + "editable": true + }, + "source": [ + "## [ADAM optimizer](https://arxiv.org/abs/1412.6980)\n", + "\n", + "In [ADAM](https://arxiv.org/abs/1412.6980), we keep a running average of\n", + "both the first and second moment of the gradient and use this\n", + "information to adaptively change the learning rate for different\n", + "parameters. The method is efficient when working with large\n", + "problems involving lots data and/or parameters. It is a combination of the\n", + "gradient descent with momentum algorithm and the RMSprop algorithm\n", + "discussed above." + ] + }, + { + "cell_type": "markdown", + "id": "a58fbe05", + "metadata": { + "editable": true + }, + "source": [ + "## Why Combine Momentum and RMSProp?\n", + "\n", + "1. Momentum: Fast convergence by smoothing gradients (accelerates in long-term gradient direction).\n", + "\n", + "2. Adaptive rates (RMSProp): Per-dimension learning rate scaling for stability (handles different feature scales, sparse gradients).\n", + "\n", + "3. Adam uses both: maintains moving averages of both first moment (gradients) and second moment (squared gradients)\n", + "\n", + "4. Additionally, includes a mechanism to correct the bias in these moving averages (crucial in early iterations)\n", + "\n", + "Result: Adam is robust, achieves faster convergence with less tuning, and often outperforms SGD (with momentum) in practice" + ] + }, + { + "cell_type": "markdown", + "id": "98700cff", + "metadata": { + "editable": true + }, + "source": [ + "## Adam: Exponential Moving Averages (Moments)\n", + "Adam maintains two moving averages at each time step $t$ for each parameter $w$:\n", + "**First moment (mean) $m_t$.**\n", + "\n", + "The Momentum term" + ] + }, + { + "cell_type": "markdown", + "id": "e55261d4", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "m_t = \\beta_1m_{t-1} + (1-\\beta_1)\\, \\nabla C(\\theta_t),\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "64676960", + "metadata": { + "editable": true + }, + "source": [ + "**Second moment (uncentered variance) $v_t$.**\n", + "\n", + "The RMS term" + ] + }, + { + "cell_type": "markdown", + "id": "1e3007bd", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "v_t = \\beta_2v_{t-1} + (1-\\beta_2)(\\nabla C(\\theta_t))^2,\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "5df06a58", + "metadata": { + "editable": true + }, + "source": [ + "with typical $\\beta_1 = 0.9$, $\\beta_2 = 0.999$. Initialize $m_0 = 0$, $v_0 = 0$.\n", + "\n", + " These are **biased** estimators of the true first and second moment of the gradients, especially at the start (since $m_0,v_0$ are zero)" + ] + }, + { + "cell_type": "markdown", + "id": "034cb12a", + "metadata": { + "editable": true + }, + "source": [ + "## Adam: Bias Correction\n", + "To counteract initialization bias in $m_t, v_t$, Adam computes bias-corrected estimates" + ] + }, + { + "cell_type": "markdown", + "id": "429a18be", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\hat{m}_t = \\frac{m_t}{1 - \\beta_1^t}, \\qquad \\hat{v}_t = \\frac{v_t}{1 - \\beta_2^t}.\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "1672b4fd", + "metadata": { + "editable": true + }, + "source": [ + "* When $t$ is small, $1-\\beta_i^t \\approx 0$, so $\\hat{m}_t, \\hat{v}_t$ significantly larger than raw $m_t, v_t$, compensating for the initial zero bias.\n", + "\n", + "* As $t$ increases, $1-\\beta_i^t \\to 1$, and $\\hat{m}_t, \\hat{v}_t$ converge to $m_t, v_t$.\n", + "\n", + "* Bias correction is important for Adam’s stability in early iterations" + ] + }, + { + "cell_type": "markdown", + "id": "dba6d945", + "metadata": { + "editable": true + }, + "source": [ + "## Adam: Update Rule Derivation\n", + "Finally, Adam updates parameters using the bias-corrected moments:" + ] + }, + { + "cell_type": "markdown", + "id": "13db1c10", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\theta_{t+1} =\\theta_t -\\frac{\\alpha}{\\sqrt{\\hat{v}_t} + \\epsilon}\\hat{m}_t,\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "e30b02ab", + "metadata": { + "editable": true + }, + "source": [ + "where $\\epsilon$ is a small constant (e.g. $10^{-8}$) to prevent division by zero.\n", + "Breaking it down:\n", + "1. Compute gradient $\\nabla C(\\theta_t)$.\n", + "\n", + "2. Update first moment $m_t$ and second moment $v_t$ (exponential moving averages).\n", + "\n", + "3. Bias-correct: $\\hat{m}_t = m_t/(1-\\beta_1^t)$, $\\; \\hat{v}_t = v_t/(1-\\beta_2^t)$.\n", + "\n", + "4. Compute step: $\\Delta \\theta_t = \\frac{\\hat{m}_t}{\\sqrt{\\hat{v}_t} + \\epsilon}$.\n", + "\n", + "5. Update parameters: $\\theta_{t+1} = \\theta_t - \\alpha\\, \\Delta \\theta_t$.\n", + "\n", + "This is the Adam update rule as given in the original paper." + ] + }, + { + "cell_type": "markdown", + "id": "5ac8ae68", + "metadata": { + "editable": true + }, + "source": [ + "## Adam vs. AdaGrad and RMSProp\n", + "\n", + "1. AdaGrad: Uses per-coordinate scaling like Adam, but no momentum. Tends to slow down too much due to cumulative history (no forgetting)\n", + "\n", + "2. RMSProp: Uses moving average of squared gradients (like Adam’s $v_t$) to maintain adaptive learning rates, but does not include momentum or bias-correction.\n", + "\n", + "3. Adam: Effectively RMSProp + Momentum + Bias-correction\n", + "\n", + " * Momentum ($m_t$) provides acceleration and smoother convergence.\n", + "\n", + " * Adaptive $v_t$ scaling moderates the step size per dimension.\n", + "\n", + " * Bias correction (absent in AdaGrad/RMSProp) ensures robust estimates early on.\n", + "\n", + "In practice, Adam often yields faster convergence and better tuning stability than RMSProp or AdaGrad alone" + ] + }, + { + "cell_type": "markdown", + "id": "86abec40", + "metadata": { + "editable": true + }, + "source": [ + "## Adaptivity Across Dimensions\n", + "\n", + "1. Adam adapts the step size \\emph{per coordinate}: parameters with larger gradient variance get smaller effective steps, those with smaller or sparse gradients get larger steps.\n", + "\n", + "2. This per-dimension adaptivity is inherited from AdaGrad/RMSProp and helps handle ill-conditioned or sparse problems.\n", + "\n", + "3. Meanwhile, momentum (first moment) allows Adam to continue making progress even if gradients become small or noisy, by leveraging accumulated direction." + ] + }, + { + "cell_type": "markdown", + "id": "448ff439", "metadata": { "editable": true }, @@ -2133,7 +1867,21 @@ }, { "cell_type": "markdown", - "id": "193b3778", + "id": "86183049", + "metadata": { + "editable": true + }, + "source": [ + "## Algorithms and codes for Adagrad, RMSprop and Adam\n", + "\n", + "The algorithms we have implemented are well described in the text by [Goodfellow, Bengio and Courville, chapter 8](https://www.deeplearningbook.org/contents/optimization.html).\n", + "\n", + "The codes which implement these algorithms are discussed below here." + ] + }, + { + "cell_type": "markdown", + "id": "78a7d51b", "metadata": { "editable": true }, @@ -2151,7 +1899,7 @@ }, { "cell_type": "markdown", - "id": "8d7187f5", + "id": "0b0e62b6", "metadata": { "editable": true }, @@ -2172,7 +1920,7 @@ { "cell_type": "code", "execution_count": 9, - "id": "68d35b76", + "id": "d19005f0", "metadata": { "collapsed": false, "editable": true @@ -2186,8 +1934,8 @@ "import matplotlib.pyplot as plt\n", "from autograd import grad\n", "\n", - "def CostOLS(beta):\n", - " return (1.0/n)*np.sum((y-X @ beta)**2)\n", + "def CostOLS(theta):\n", + " return (1.0/n)*np.sum((y-X @ theta)**2)\n", "\n", "n = 100\n", "x = 2*np.random.rand(n,1)\n", @@ -2232,7 +1980,7 @@ }, { "cell_type": "markdown", - "id": "f39b471c", + "id": "4f7fa973", "metadata": { "editable": true }, @@ -2243,7 +1991,7 @@ { "cell_type": "code", "execution_count": 10, - "id": "265059ef", + "id": "34cb0de4", "metadata": { "collapsed": false, "editable": true @@ -2257,8 +2005,8 @@ "import matplotlib.pyplot as plt\n", "from autograd import grad\n", "\n", - "def CostOLS(beta):\n", - " return (1.0/n)*np.sum((y-X @ beta)**2)\n", + "def CostOLS(theta):\n", + " return (1.0/n)*np.sum((y-X @ theta)**2)\n", "\n", "n = 100\n", "x = 2*np.random.rand(n,1)\n", @@ -2307,7 +2055,7 @@ }, { "cell_type": "markdown", - "id": "18183fb0", + "id": "4cb35cee", "metadata": { "editable": true }, @@ -2322,7 +2070,7 @@ { "cell_type": "code", "execution_count": 11, - "id": "4d9d93b3", + "id": "74934aab", "metadata": { "collapsed": false, "editable": true @@ -2406,7 +2154,7 @@ }, { "cell_type": "markdown", - "id": "1652a471", + "id": "d8ee3d35", "metadata": { "editable": true }, @@ -2417,7 +2165,7 @@ { "cell_type": "code", "execution_count": 12, - "id": "6092712c", + "id": "63184f2b", "metadata": { "collapsed": false, "editable": true @@ -2495,7 +2243,7 @@ }, { "cell_type": "markdown", - "id": "4d2f230d", + "id": "c44ce795", "metadata": { "editable": true }, @@ -2508,7 +2256,7 @@ { "cell_type": "code", "execution_count": 13, - "id": "b46cedcc", + "id": "74e118d6", "metadata": { "collapsed": false, "editable": true @@ -2521,8 +2269,8 @@ "import autograd.numpy as np\n", "from autograd import grad\n", "\n", - "def CostOLS(beta):\n", - " return (1.0/n)*np.sum((y-X @ beta)**2)\n", + "def CostOLS(theta):\n", + " return (1.0/n)*np.sum((y-X @ theta)**2)\n", "\n", "n = 100\n", "x = 2*np.random.rand(n,1)\n", @@ -2530,29 +2278,29 @@ "\n", "X = np.c_[np.ones((n,1)), x, x*x]\n", "XT_X = X.T @ X\n", - "beta_linreg = np.linalg.pinv(XT_X) @ (X.T @ y)\n", + "theta_linreg = np.linalg.pinv(XT_X) @ (X.T @ y)\n", "print(\"Own inversion\")\n", - "print(beta_linreg)\n", + "print(theta_linreg)\n", "# Hessian matrix\n", "H = (2.0/n)* XT_X\n", - "# Note that here the Hessian does not depend on the parameters beta\n", + "# Note that here the Hessian does not depend on the parameters theta\n", "invH = np.linalg.pinv(H)\n", - "beta = np.random.randn(3,1)\n", + "theta = np.random.randn(3,1)\n", "Niterations = 5\n", "# define the gradient\n", "training_gradient = grad(CostOLS)\n", "\n", "for iter in range(Niterations):\n", - " gradients = training_gradient(beta)\n", - " beta -= invH @ gradients\n", + " gradients = training_gradient(theta)\n", + " theta -= invH @ gradients\n", " print(iter,gradients[0],gradients[1])\n", - "print(\"beta from own Newton code\")\n", - "print(beta)" + "print(\"theta from own Newton code\")\n", + "print(theta)" ] }, { "cell_type": "markdown", - "id": "f048d5c7", + "id": "3789172e", "metadata": { "editable": true }, @@ -2563,7 +2311,7 @@ { "cell_type": "code", "execution_count": 14, - "id": "d8bdd180", + "id": "5152d211", "metadata": { "collapsed": false, "editable": true @@ -2622,7 +2370,7 @@ }, { "cell_type": "markdown", - "id": "145ce394", + "id": "34e87757", "metadata": { "editable": true }, @@ -2632,7 +2380,7 @@ }, { "cell_type": "markdown", - "id": "ec0ee110", + "id": "713154cb", "metadata": { "editable": true }, @@ -2643,7 +2391,7 @@ { "cell_type": "code", "execution_count": 15, - "id": "b6e8e3e4", + "id": "3e863305", "metadata": { "collapsed": false, "editable": true @@ -2708,7 +2456,7 @@ }, { "cell_type": "markdown", - "id": "79d8c0ae", + "id": "a5fa4647", "metadata": { "editable": true }, @@ -2719,7 +2467,7 @@ { "cell_type": "code", "execution_count": 16, - "id": "c16ec5ac", + "id": "0b318139", "metadata": { "collapsed": false, "editable": true @@ -2760,9 +2508,9 @@ "\n", "# Value for learning rate\n", "eta = 0.01\n", - "# Value for parameters beta1 and beta2, see https://arxiv.org/abs/1412.6980\n", - "beta1 = 0.9\n", - "beta2 = 0.999\n", + "# Value for parameters theta1 and theta2, see https://arxiv.org/abs/1412.6980\n", + "theta1 = 0.9\n", + "theta2 = 0.999\n", "# Including AdaGrad parameter to avoid possible division by zero\n", "delta = 1e-7\n", "iter = 0\n", @@ -2776,10 +2524,10 @@ " yi = y[random_index:random_index+M]\n", " gradients = (1.0/M)*training_gradient(yi, xi, theta)\n", " # Computing moments first\n", - " first_moment = beta1*first_moment + (1-beta1)*gradients\n", - " second_moment = beta2*second_moment+(1-beta2)*gradients*gradients\n", - " first_term = first_moment/(1.0-beta1**iter)\n", - " second_term = second_moment/(1.0-beta2**iter)\n", + " first_moment = theta1*first_moment + (1-theta1)*gradients\n", + " second_moment = theta2*second_moment+(1-theta2)*gradients*gradients\n", + " first_term = first_moment/(1.0-theta1**iter)\n", + " second_term = second_moment/(1.0-theta2**iter)\n", "\t# Scaling with rho the new and the previous results\n", " update = eta*first_term/(np.sqrt(second_term)+delta)\n", " theta -= update\n", @@ -2789,7 +2537,7 @@ }, { "cell_type": "markdown", - "id": "7c581f8b", + "id": "967dd1d1", "metadata": { "editable": true }, @@ -2801,12 +2549,12 @@ "2. Work on project 1\n", "\n", "\n", - " * For more discussions of Ridge regression and calculation of averages, [Wessel van Wieringen's](https://arxiv.org/abs/1509.09169) article is highly recommended." + "For more discussions of Ridge regression and calculation of averages, [Wessel van Wieringen's](https://arxiv.org/abs/1509.09169) article is highly recommended." ] }, { "cell_type": "markdown", - "id": "d8a82e01", + "id": "ad137d49", "metadata": { "editable": true }, @@ -2834,7 +2582,7 @@ { "cell_type": "code", "execution_count": 17, - "id": "3fa174a9", + "id": "8a6c2cb4", "metadata": { "collapsed": false, "editable": true @@ -2854,7 +2602,7 @@ }, { "cell_type": "markdown", - "id": "b02de7a6", + "id": "76bf3c81", "metadata": { "editable": true }, @@ -2870,7 +2618,7 @@ }, { "cell_type": "markdown", - "id": "cd3f85ef", + "id": "818c6d58", "metadata": { "editable": true }, @@ -2890,7 +2638,7 @@ }, { "cell_type": "markdown", - "id": "1e0198a0", + "id": "58481e8f", "metadata": { "editable": true }, @@ -2917,7 +2665,7 @@ }, { "cell_type": "markdown", - "id": "04da94ca", + "id": "4e94a32c", "metadata": { "editable": true }, @@ -2930,7 +2678,7 @@ }, { "cell_type": "markdown", - "id": "aa1ad257", + "id": "3a93bb3e", "metadata": { "editable": true }, @@ -2942,7 +2690,7 @@ }, { "cell_type": "markdown", - "id": "6587016d", + "id": "c603fb31", "metadata": { "editable": true }, @@ -2957,7 +2705,7 @@ { "cell_type": "code", "execution_count": 18, - "id": "2e0350b4", + "id": "363ae494", "metadata": { "collapsed": false, "editable": true @@ -2984,7 +2732,7 @@ }, { "cell_type": "markdown", - "id": "8179b037", + "id": "429b71ce", "metadata": { "editable": true }, @@ -2998,7 +2746,7 @@ }, { "cell_type": "markdown", - "id": "3ebed257", + "id": "ec3c7657", "metadata": { "editable": true }, @@ -3010,7 +2758,7 @@ }, { "cell_type": "markdown", - "id": "51b239a4", + "id": "c6c2fd1d", "metadata": { "editable": true }, @@ -3027,7 +2775,7 @@ }, { "cell_type": "markdown", - "id": "295790f3", + "id": "1141934f", "metadata": { "editable": true }, @@ -3039,7 +2787,7 @@ }, { "cell_type": "markdown", - "id": "b68338d6", + "id": "efc84979", "metadata": { "editable": true }, @@ -3049,7 +2797,7 @@ }, { "cell_type": "markdown", - "id": "c015cb3b", + "id": "6be29872", "metadata": { "editable": true }, @@ -3061,7 +2809,7 @@ }, { "cell_type": "markdown", - "id": "5e4917fa", + "id": "8f128adc", "metadata": { "editable": true }, @@ -3071,7 +2819,7 @@ }, { "cell_type": "markdown", - "id": "150fedf5", + "id": "967b009a", "metadata": { "editable": true }, @@ -3083,7 +2831,7 @@ }, { "cell_type": "markdown", - "id": "e6d782b1", + "id": "181e0b18", "metadata": { "editable": true }, @@ -3094,7 +2842,7 @@ }, { "cell_type": "markdown", - "id": "a4065dd3", + "id": "6fe29ace", "metadata": { "editable": true }, @@ -3106,7 +2854,7 @@ }, { "cell_type": "markdown", - "id": "06324d79", + "id": "795a350c", "metadata": { "editable": true }, @@ -3116,7 +2864,7 @@ }, { "cell_type": "markdown", - "id": "9e3e6b35", + "id": "9a0ebe42", "metadata": { "editable": true }, @@ -3128,7 +2876,7 @@ }, { "cell_type": "markdown", - "id": "cd9c3754", + "id": "800ad4c7", "metadata": { "editable": true }, @@ -3138,7 +2886,7 @@ }, { "cell_type": "markdown", - "id": "8361829b", + "id": "4ed44194", "metadata": { "editable": true }, @@ -3150,7 +2898,7 @@ }, { "cell_type": "markdown", - "id": "8f5f46fd", + "id": "f6fc455e", "metadata": { "editable": true }, @@ -3160,7 +2908,7 @@ }, { "cell_type": "markdown", - "id": "cf210a6e", + "id": "1d434e5d", "metadata": { "editable": true }, @@ -3172,7 +2920,7 @@ }, { "cell_type": "markdown", - "id": "d3af09fc", + "id": "011d7385", "metadata": { "editable": true }, @@ -3182,7 +2930,7 @@ }, { "cell_type": "markdown", - "id": "ec91f8c1", + "id": "9bb92755", "metadata": { "editable": true }, @@ -3194,7 +2942,7 @@ }, { "cell_type": "markdown", - "id": "0c9f35e6", + "id": "7287a085", "metadata": { "editable": true }, @@ -3204,7 +2952,7 @@ }, { "cell_type": "markdown", - "id": "de80fa4b", + "id": "1ebd1481", "metadata": { "editable": true }, @@ -3216,7 +2964,7 @@ }, { "cell_type": "markdown", - "id": "34c99b80", + "id": "01da7453", "metadata": { "editable": true }, @@ -3226,7 +2974,7 @@ }, { "cell_type": "markdown", - "id": "758ade4d", + "id": "2719429f", "metadata": { "editable": true }, @@ -3238,7 +2986,7 @@ }, { "cell_type": "markdown", - "id": "ff0f4725", + "id": "1b7a1119", "metadata": { "editable": true }, @@ -3248,7 +2996,7 @@ }, { "cell_type": "markdown", - "id": "997b829b", + "id": "5455e569", "metadata": { "editable": true }, @@ -3260,7 +3008,7 @@ }, { "cell_type": "markdown", - "id": "621e7d85", + "id": "82719ade", "metadata": { "editable": true }, @@ -3272,7 +3020,7 @@ }, { "cell_type": "markdown", - "id": "5530e459", + "id": "d87ce00b", "metadata": { "editable": true }, @@ -3284,7 +3032,7 @@ }, { "cell_type": "markdown", - "id": "fd9bace8", + "id": "aad7612a", "metadata": { "editable": true }, @@ -3294,7 +3042,7 @@ }, { "cell_type": "markdown", - "id": "d949be75", + "id": "3c7ca158", "metadata": { "editable": true }, @@ -3306,7 +3054,7 @@ }, { "cell_type": "markdown", - "id": "4ad52617", + "id": "b5c113ba", "metadata": { "editable": true }, @@ -3319,7 +3067,7 @@ }, { "cell_type": "markdown", - "id": "41cabc72", + "id": "aad866d0", "metadata": { "editable": true }, @@ -3331,7 +3079,7 @@ }, { "cell_type": "markdown", - "id": "1fd416a6", + "id": "a4e7459d", "metadata": { "editable": true }, @@ -3345,7 +3093,7 @@ { "cell_type": "code", "execution_count": 19, - "id": "f13cacb4", + "id": "2af50ed1", "metadata": { "collapsed": false, "editable": true @@ -3442,7 +3190,7 @@ }, { "cell_type": "markdown", - "id": "ff4b28ed", + "id": "e468f434", "metadata": { "editable": true }, @@ -3463,7 +3211,7 @@ }, { "cell_type": "markdown", - "id": "03b4dc3c", + "id": "0daaeb51", "metadata": { "editable": true }, @@ -3475,7 +3223,7 @@ }, { "cell_type": "markdown", - "id": "6cff2148", + "id": "29f1ae2a", "metadata": { "editable": true }, @@ -3485,7 +3233,7 @@ }, { "cell_type": "markdown", - "id": "51d823c4", + "id": "133f65de", "metadata": { "editable": true }, @@ -3497,7 +3245,7 @@ }, { "cell_type": "markdown", - "id": "5ab270f9", + "id": "b5f4350c", "metadata": { "editable": true }, @@ -3507,7 +3255,7 @@ }, { "cell_type": "markdown", - "id": "2b2e1492", + "id": "1f6fc172", "metadata": { "editable": true }, @@ -3519,7 +3267,7 @@ }, { "cell_type": "markdown", - "id": "063be947", + "id": "c5afa388", "metadata": { "editable": true }, @@ -3537,7 +3285,7 @@ { "cell_type": "code", "execution_count": 20, - "id": "5e841a10", + "id": "5718b4ec", "metadata": { "collapsed": false, "editable": true @@ -3613,7 +3361,7 @@ }, { "cell_type": "markdown", - "id": "40e54083", + "id": "5e7c8319", "metadata": { "editable": true }, @@ -3627,7 +3375,7 @@ { "cell_type": "code", "execution_count": 21, - "id": "edd7dd42", + "id": "3b7f797a", "metadata": { "collapsed": false, "editable": true @@ -3716,7 +3464,7 @@ }, { "cell_type": "markdown", - "id": "f9cc046b", + "id": "5e9da20b", "metadata": { "editable": true }, diff --git a/doc/LectureNotes/week37.ipynb b/doc/LectureNotes/week37.ipynb index acd6b4e52..6ec95c3cd 100644 --- a/doc/LectureNotes/week37.ipynb +++ b/doc/LectureNotes/week37.ipynb @@ -2,7 +2,7 @@ "cells": [ { "cell_type": "markdown", - "id": "c543a71a", + "id": "4f3beaf1", "metadata": { "editable": true }, @@ -14,7 +14,7 @@ }, { "cell_type": "markdown", - "id": "4ff6b050", + "id": "46fe6a26", "metadata": { "editable": true }, @@ -29,7 +29,7 @@ }, { "cell_type": "markdown", - "id": "a7079d3e", + "id": "f7c7fee9", "metadata": { "editable": true }, @@ -52,7 +52,7 @@ }, { "cell_type": "markdown", - "id": "db3af8cc", + "id": "29a5df4a", "metadata": { "editable": true }, @@ -69,7 +69,7 @@ }, { "cell_type": "markdown", - "id": "79d6d5be", + "id": "d1cabb8b", "metadata": { "editable": true }, @@ -79,7 +79,7 @@ }, { "cell_type": "markdown", - "id": "fa427fc3", + "id": "c1499668", "metadata": { "editable": true }, @@ -103,20 +103,21 @@ { "cell_type": "code", "execution_count": 1, - "id": "1185629e", + "id": "8f93b870", "metadata": { "collapsed": false, "editable": true }, "outputs": [], "source": [ + "import numpy as np\n", "x = 2*np.random.rand(m,1)\n", "y = 4+3*x+np.random.randn(m,1)" ] }, { "cell_type": "markdown", - "id": "42dfe293", + "id": "cafa88bc", "metadata": { "editable": true }, @@ -127,7 +128,7 @@ }, { "cell_type": "markdown", - "id": "a726cc6e", + "id": "d46f9bbc", "metadata": { "editable": true }, @@ -139,7 +140,7 @@ }, { "cell_type": "markdown", - "id": "991c8919", + "id": "c6860f97", "metadata": { "editable": true }, @@ -149,7 +150,7 @@ }, { "cell_type": "markdown", - "id": "ba895122", + "id": "5961000b", "metadata": { "editable": true }, @@ -161,7 +162,7 @@ }, { "cell_type": "markdown", - "id": "5c2b9317", + "id": "602cdea2", "metadata": { "editable": true }, @@ -175,7 +176,7 @@ }, { "cell_type": "markdown", - "id": "264eef67", + "id": "2a6ab00e", "metadata": { "editable": true }, @@ -191,17 +192,17 @@ }, { "cell_type": "markdown", - "id": "a451a9ce", + "id": "f9fb9a51", "metadata": { "editable": true }, "source": [ - "The cost/loss/risk function is given by (" + "The cost/loss/risk function is given by" ] }, { "cell_type": "markdown", - "id": "e7fa569b", + "id": "a2f3adf9", "metadata": { "editable": true }, @@ -213,7 +214,7 @@ }, { "cell_type": "markdown", - "id": "a2655bc6", + "id": "e4cffe8e", "metadata": { "editable": true }, @@ -223,7 +224,7 @@ }, { "cell_type": "markdown", - "id": "277ace2e", + "id": "3ebf7860", "metadata": { "editable": true }, @@ -235,7 +236,7 @@ }, { "cell_type": "markdown", - "id": "75bbf03c", + "id": "f8cdd774", "metadata": { "editable": true }, @@ -249,7 +250,7 @@ }, { "cell_type": "markdown", - "id": "d5bb2db2", + "id": "67a028fe", "metadata": { "editable": true }, @@ -259,7 +260,7 @@ }, { "cell_type": "markdown", - "id": "53430d47", + "id": "28b34be8", "metadata": { "editable": true }, @@ -270,7 +271,7 @@ }, { "cell_type": "markdown", - "id": "89dce328", + "id": "7068c946", "metadata": { "editable": true }, @@ -285,7 +286,7 @@ }, { "cell_type": "markdown", - "id": "542b8278", + "id": "289d7cf9", "metadata": { "editable": true }, @@ -295,37 +296,37 @@ }, { "cell_type": "markdown", - "id": "2b94baca", + "id": "3bacfa14", "metadata": { "editable": true }, "source": [ "## Simple program\n", "\n", - "We can now write a program that minimizes $C(\\theta)$ using the gradient descent method with a constant learning rate $\\gamma$ according to" + "We can now write a program that minimizes $C(\\theta)$ using the gradient descent method with a constant learning rate $\\eta$ according to" ] }, { "cell_type": "markdown", - "id": "bfc6b93f", + "id": "16c159ac", "metadata": { "editable": true }, "source": [ "$$\n", - "\\theta_{k+1} = \\theta_k - \\gamma \\nabla_\\theta C(\\theta_k), \\ k=0,1,\\cdots\n", + "\\theta_{k+1} = \\theta_k - \\eta \\nabla_\\theta C(\\theta_k), \\ k=0,1,\\cdots\n", "$$" ] }, { "cell_type": "markdown", - "id": "703de92e", + "id": "cf04e4b0", "metadata": { "editable": true }, "source": [ "We can use the expression we computed for the gradient and let use a\n", - "$\\theta_0$ be chosen randomly and let $\\gamma = 0.001$. Stop iterating\n", + "$\\theta_0$ be chosen randomly and let $\\eta = 0.001$. Stop iterating\n", "when $||\\nabla_\\theta C(\\theta_k) || \\leq \\epsilon = 10^{-8}$. **Note that the code below does not include the latter stop criterion**.\n", "\n", "And finally we can compare our solution for $\\theta$ with the analytic result given by \n", @@ -334,7 +335,7 @@ }, { "cell_type": "markdown", - "id": "f843aae1", + "id": "0d9644f3", "metadata": { "editable": true }, @@ -347,7 +348,7 @@ { "cell_type": "code", "execution_count": 2, - "id": "b76443c9", + "id": "33a70e79", "metadata": { "collapsed": false, "editable": true @@ -406,7 +407,7 @@ }, { "cell_type": "markdown", - "id": "7b75f4ec", + "id": "85b9190b", "metadata": { "editable": true }, @@ -418,7 +419,7 @@ }, { "cell_type": "markdown", - "id": "47ecdee7", + "id": "c6bf4720", "metadata": { "editable": true }, @@ -430,7 +431,7 @@ }, { "cell_type": "markdown", - "id": "ff25299b", + "id": "8f766543", "metadata": { "editable": true }, @@ -440,7 +441,7 @@ }, { "cell_type": "markdown", - "id": "21a88a84", + "id": "31c687f3", "metadata": { "editable": true }, @@ -454,7 +455,7 @@ }, { "cell_type": "markdown", - "id": "67491c15", + "id": "ff33ba55", "metadata": { "editable": true }, @@ -464,7 +465,7 @@ }, { "cell_type": "markdown", - "id": "f1ac13f1", + "id": "02d68d7d", "metadata": { "editable": true }, @@ -476,7 +477,7 @@ }, { "cell_type": "markdown", - "id": "67d83802", + "id": "aee5a999", "metadata": { "editable": true }, @@ -487,7 +488,7 @@ }, { "cell_type": "markdown", - "id": "75935d70", + "id": "d4c6e860", "metadata": { "editable": true }, @@ -502,7 +503,7 @@ }, { "cell_type": "markdown", - "id": "4707a65d", + "id": "d0b1ab36", "metadata": { "editable": true }, @@ -516,7 +517,7 @@ }, { "cell_type": "markdown", - "id": "ab16b163", + "id": "feffa94a", "metadata": { "editable": true }, @@ -527,7 +528,7 @@ { "cell_type": "code", "execution_count": 3, - "id": "fe7ee1f7", + "id": "17b7aebb", "metadata": { "collapsed": false, "editable": true @@ -588,7 +589,7 @@ }, { "cell_type": "markdown", - "id": "823fdaf7", + "id": "845d5551", "metadata": { "editable": true }, @@ -610,20 +611,33 @@ }, { "cell_type": "markdown", - "id": "21ea3f2a", + "id": "55818d27", "metadata": { "editable": true }, "source": [ - "## Improving gradient descent with momentum\n", + "## Momentum based GD\n", "\n", - "We discuss here some simple examples where we introduce what is called 'memory'about previous steps, or what is normally called momentum gradient descent. The mathematics is explained below in connection with Stochastic gradient descent." + "We discuss here some simple examples where we introduce what is called\n", + "'memory'about previous steps, or what is normally called momentum\n", + "gradient descent.\n", + "For the mathematical details, see whiteboad notes from lecture on September 8, 2025." + ] + }, + { + "cell_type": "markdown", + "id": "bdfa4705", + "metadata": { + "editable": true + }, + "source": [ + "## Improving gradient descent with momentum" ] }, { "cell_type": "code", "execution_count": 4, - "id": "5b729a7e", + "id": "b13373bb", "metadata": { "collapsed": false, "editable": true @@ -689,7 +703,7 @@ }, { "cell_type": "markdown", - "id": "209fc69c", + "id": "01f07f3c", "metadata": { "editable": true }, @@ -700,7 +714,7 @@ { "cell_type": "code", "execution_count": 5, - "id": "984c4202", + "id": "dd296802", "metadata": { "collapsed": false, "editable": true @@ -774,7 +788,7 @@ }, { "cell_type": "markdown", - "id": "8577b238", + "id": "2b995d39", "metadata": { "editable": true }, @@ -793,7 +807,7 @@ }, { "cell_type": "markdown", - "id": "c11d0874", + "id": "ec3098ab", "metadata": { "editable": true }, @@ -814,7 +828,7 @@ }, { "cell_type": "markdown", - "id": "64ad52d0", + "id": "ae3c5061", "metadata": { "editable": true }, @@ -830,7 +844,7 @@ }, { "cell_type": "markdown", - "id": "5dd4c467", + "id": "54028ccc", "metadata": { "editable": true }, @@ -844,7 +858,7 @@ }, { "cell_type": "markdown", - "id": "42904823", + "id": "17875f98", "metadata": { "editable": true }, @@ -872,7 +886,7 @@ }, { "cell_type": "markdown", - "id": "49dfaf2e", + "id": "40862db0", "metadata": { "editable": true }, @@ -904,7 +918,7 @@ }, { "cell_type": "markdown", - "id": "284b8663", + "id": "075431a2", "metadata": { "editable": true }, @@ -921,20 +935,20 @@ }, { "cell_type": "markdown", - "id": "5e8cc1b8", + "id": "65dcd718", "metadata": { "editable": true }, "source": [ "$$\n", - "C(\\mathbf{\\beta}) = \\sum_{i=1}^n c_i(\\mathbf{x}_i,\n", - "\\mathbf{\\beta}).\n", + "C(\\mathbf{\\theta}) = \\sum_{i=1}^n c_i(\\mathbf{x}_i,\n", + "\\mathbf{\\theta}).\n", "$$" ] }, { "cell_type": "markdown", - "id": "510505a0", + "id": "ccf20bf2", "metadata": { "editable": true }, @@ -947,20 +961,20 @@ }, { "cell_type": "markdown", - "id": "32fc501f", + "id": "7cf7a667", "metadata": { "editable": true }, "source": [ "$$\n", - "\\nabla_\\beta C(\\mathbf{\\beta}) = \\sum_i^n \\nabla_\\beta c_i(\\mathbf{x}_i,\n", - "\\mathbf{\\beta}).\n", + "\\nabla_\\theta C(\\mathbf{\\theta}) = \\sum_i^n \\nabla_\\theta c_i(\\mathbf{x}_i,\n", + "\\mathbf{\\theta}).\n", "$$" ] }, { "cell_type": "markdown", - "id": "9dcac7ef", + "id": "21eeb773", "metadata": { "editable": true }, @@ -974,7 +988,7 @@ }, { "cell_type": "markdown", - "id": "c9464120", + "id": "562bbf59", "metadata": { "editable": true }, @@ -996,22 +1010,22 @@ }, { "cell_type": "markdown", - "id": "59bc847c", + "id": "1b9e6fb3", "metadata": { "editable": true }, "source": [ "$$\n", - "\\nabla_{\\beta}\n", - "C(\\mathbf{\\beta}) = \\sum_{i=1}^n \\nabla_\\beta c_i(\\mathbf{x}_i,\n", - "\\mathbf{\\beta}) \\rightarrow \\sum_{i \\in B_k}^n \\nabla_\\beta\n", - "c_i(\\mathbf{x}_i, \\mathbf{\\beta}).\n", + "\\nabla_{\\theta}\n", + "C(\\mathbf{\\theta}) = \\sum_{i=1}^n \\nabla_\\theta c_i(\\mathbf{x}_i,\n", + "\\mathbf{\\theta}) \\rightarrow \\sum_{i \\in B_k}^n \\nabla_\\theta\n", + "c_i(\\mathbf{x}_i, \\mathbf{\\theta}).\n", "$$" ] }, { "cell_type": "markdown", - "id": "ba220299", + "id": "4cd95b23", "metadata": { "editable": true }, @@ -1023,20 +1037,20 @@ }, { "cell_type": "markdown", - "id": "8608ba34", + "id": "ea4324e1", "metadata": { "editable": true }, "source": [ "$$\n", - "\\beta_{j+1} = \\beta_j - \\gamma_j \\sum_{i \\in B_k}^n \\nabla_\\beta c_i(\\mathbf{x}_i,\n", - "\\mathbf{\\beta})\n", + "\\theta_{j+1} = \\theta_j - \\eta_j \\sum_{i \\in B_k}^n \\nabla_\\theta c_i(\\mathbf{x}_i,\n", + "\\mathbf{\\theta})\n", "$$" ] }, { "cell_type": "markdown", - "id": "0faa1782", + "id": "5287666c", "metadata": { "editable": true }, @@ -1050,7 +1064,7 @@ }, { "cell_type": "markdown", - "id": "97eddb68", + "id": "596acab8", "metadata": { "editable": true }, @@ -1061,7 +1075,7 @@ { "cell_type": "code", "execution_count": 6, - "id": "309da17e", + "id": "3b4d8b03", "metadata": { "collapsed": false, "editable": true @@ -1086,7 +1100,7 @@ }, { "cell_type": "markdown", - "id": "7aea4344", + "id": "f4f09885", "metadata": { "editable": true }, @@ -1102,7 +1116,7 @@ }, { "cell_type": "markdown", - "id": "52f8d549", + "id": "156e8953", "metadata": { "editable": true }, @@ -1117,20 +1131,20 @@ "that we are close to a local/global minimum. However, we could also\n", "evaluate the cost function at this point, store the result and\n", "continue the search. If the test kicks in at a later stage we can\n", - "compare the values of the cost function and keep the $\\beta$ that\n", + "compare the values of the cost function and keep the $\\theta$ that\n", "gave the lowest value." ] }, { "cell_type": "markdown", - "id": "dcd04de7", + "id": "7076d164", "metadata": { "editable": true }, "source": [ "## Slightly different approach\n", "\n", - "Another approach is to let the step length $\\gamma_j$ depend on the\n", + "Another approach is to let the step length $\\eta_j$ depend on the\n", "number of epochs in such a way that it becomes very small after a\n", "reasonable time such that we do not move at all. Such approaches are\n", "also called scaling. There are many such ways to [scale the learning\n", @@ -1143,26 +1157,26 @@ }, { "cell_type": "markdown", - "id": "4ecf0b16", + "id": "d822d1d4", "metadata": { "editable": true }, "source": [ "## Time decay rate\n", "\n", - "As an example, let $e = 0,1,2,3,\\cdots$ denote the current epoch and let $t_0, t_1 > 0$ be two fixed numbers. Furthermore, let $t = e \\cdot m + i$ where $m$ is the number of minibatches and $i=0,\\cdots,m-1$. Then the function $$\\gamma_j(t; t_0, t_1) = \\frac{t_0}{t+t_1} $$ goes to zero as the number of epochs gets large. I.e. we start with a step length $\\gamma_j (0; t_0, t_1) = t_0/t_1$ which decays in *time* $t$.\n", + "As an example, let $e = 0,1,2,3,\\cdots$ denote the current epoch and let $t_0, t_1 > 0$ be two fixed numbers. Furthermore, let $t = e \\cdot m + i$ where $m$ is the number of minibatches and $i=0,\\cdots,m-1$. Then the function $$\\eta_j(t; t_0, t_1) = \\frac{t_0}{t+t_1} $$ goes to zero as the number of epochs gets large. I.e. we start with a step length $\\eta_j (0; t_0, t_1) = t_0/t_1$ which decays in *time* $t$.\n", "\n", - "In this way we can fix the number of epochs, compute $\\beta$ and\n", + "In this way we can fix the number of epochs, compute $\\theta$ and\n", "evaluate the cost function at the end. Repeating the computation will\n", "give a different result since the scheme is random by design. Then we\n", - "pick the final $\\beta$ that gives the lowest value of the cost\n", + "pick the final $\\theta$ that gives the lowest value of the cost\n", "function." ] }, { "cell_type": "code", "execution_count": 7, - "id": "71eb89db", + "id": "c8987f85", "metadata": { "collapsed": false, "editable": true @@ -1181,23 +1195,23 @@ "t0 = 1.0\n", "t1 = 10\n", "\n", - "gamma_j = t0/t1\n", + "eta_j = t0/t1\n", "j = 0\n", "for epoch in range(1,n_epochs+1):\n", " for i in range(m):\n", " k = np.random.randint(m) #Pick the k-th minibatch at random\n", " #Compute the gradient using the data in minibatch Bk\n", - " #Compute new suggestion for beta\n", + " #Compute new suggestion for theta\n", " t = epoch*m+i\n", - " gamma_j = step_length(t,t0,t1)\n", + " eta_j = step_length(t,t0,t1)\n", " j += 1\n", "\n", - "print(\"gamma_j after %d epochs: %g\" % (n_epochs,gamma_j))" + "print(\"eta_j after %d epochs: %g\" % (n_epochs,eta_j))" ] }, { "cell_type": "markdown", - "id": "10ce3fc7", + "id": "4c72814e", "metadata": { "editable": true }, @@ -1210,7 +1224,7 @@ { "cell_type": "code", "execution_count": 8, - "id": "f2d402a8", + "id": "7b243058", "metadata": { "collapsed": false, "editable": true @@ -1287,7 +1301,7 @@ }, { "cell_type": "markdown", - "id": "ffd1e92d", + "id": "e008b37c", "metadata": { "editable": true }, @@ -1302,267 +1316,7 @@ }, { "cell_type": "markdown", - "id": "4b72d3f9", - "metadata": { - "editable": true - }, - "source": [ - "## Momentum based GD\n", - "\n", - "The stochastic gradient descent (SGD) is almost always used with a\n", - "*momentum* or inertia term that serves as a memory of the direction we\n", - "are moving in parameter space. This is typically implemented as\n", - "follows" - ] - }, - { - "cell_type": "markdown", - "id": "1b60ec02", - "metadata": { - "editable": true - }, - "source": [ - "$$\n", - "\\mathbf{v}_{t}=\\gamma \\mathbf{v}_{t-1}+\\eta_{t}\\nabla_\\theta E(\\boldsymbol{\\theta}_t) \\nonumber\n", - "$$" - ] - }, - { - "cell_type": "markdown", - "id": "e1fa24ba", - "metadata": { - "editable": true - }, - "source": [ - "\n", - "
    \n", - "\n", - "$$\n", - "\\begin{equation} \n", - "\\boldsymbol{\\theta}_{t+1}= \\boldsymbol{\\theta}_t -\\mathbf{v}_{t},\n", - "\\label{_auto1} \\tag{1}\n", - "\\end{equation}\n", - "$$" - ] - }, - { - "cell_type": "markdown", - "id": "7a1fc9fc", - "metadata": { - "editable": true - }, - "source": [ - "where we have introduced a momentum parameter $\\gamma$, with\n", - "$0\\le\\gamma\\le 1$, and for brevity we dropped the explicit notation to\n", - "indicate the gradient is to be taken over a different mini-batch at\n", - "each step. We call this algorithm gradient descent with momentum\n", - "(GDM). From these equations, it is clear that $\\mathbf{v}_t$ is a\n", - "running average of recently encountered gradients and\n", - "$(1-\\gamma)^{-1}$ sets the characteristic time scale for the memory\n", - "used in the averaging procedure. Consistent with this, when\n", - "$\\gamma=0$, this just reduces down to ordinary SGD as discussed\n", - "earlier. An equivalent way of writing the updates is" - ] - }, - { - "cell_type": "markdown", - "id": "23abda5d", - "metadata": { - "editable": true - }, - "source": [ - "$$\n", - "\\Delta \\boldsymbol{\\theta}_{t+1} = \\gamma \\Delta \\boldsymbol{\\theta}_t -\\ \\eta_{t}\\nabla_\\theta E(\\boldsymbol{\\theta}_t),\n", - "$$" - ] - }, - { - "cell_type": "markdown", - "id": "0f0067a9", - "metadata": { - "editable": true - }, - "source": [ - "where we have defined $\\Delta \\boldsymbol{\\theta}_{t}= \\boldsymbol{\\theta}_t-\\boldsymbol{\\theta}_{t-1}$." - ] - }, - { - "cell_type": "markdown", - "id": "b86ec959", - "metadata": { - "editable": true - }, - "source": [ - "## More on momentum based approaches\n", - "\n", - "Let us try to get more intuition from these equations. It is helpful\n", - "to consider a simple physical analogy with a particle of mass $m$\n", - "moving in a viscous medium with drag coefficient $\\mu$ and potential\n", - "$E(\\mathbf{w})$. If we denote the particle's position by $\\mathbf{w}$,\n", - "then its motion is described by" - ] - }, - { - "cell_type": "markdown", - "id": "7685ff97", - "metadata": { - "editable": true - }, - "source": [ - "$$\n", - "m {d^2 \\mathbf{w} \\over dt^2} + \\mu {d \\mathbf{w} \\over dt }= -\\nabla_w E(\\mathbf{w}).\n", - "$$" - ] - }, - { - "cell_type": "markdown", - "id": "28203c1e", - "metadata": { - "editable": true - }, - "source": [ - "We can discretize this equation in the usual way to get" - ] - }, - { - "cell_type": "markdown", - "id": "1495b23f", - "metadata": { - "editable": true - }, - "source": [ - "$$\n", - "m { \\mathbf{w}_{t+\\Delta t}-2 \\mathbf{w}_{t} +\\mathbf{w}_{t-\\Delta t} \\over (\\Delta t)^2}+\\mu {\\mathbf{w}_{t+\\Delta t}- \\mathbf{w}_{t} \\over \\Delta t} = -\\nabla_w E(\\mathbf{w}).\n", - "$$" - ] - }, - { - "cell_type": "markdown", - "id": "15bbc6e6", - "metadata": { - "editable": true - }, - "source": [ - "Rearranging this equation, we can rewrite this as" - ] - }, - { - "cell_type": "markdown", - "id": "ffeffc05", - "metadata": { - "editable": true - }, - "source": [ - "$$\n", - "\\Delta \\mathbf{w}_{t +\\Delta t}= - { (\\Delta t)^2 \\over m +\\mu \\Delta t} \\nabla_w E(\\mathbf{w})+ {m \\over m +\\mu \\Delta t} \\Delta \\mathbf{w}_t.\n", - "$$" - ] - }, - { - "cell_type": "markdown", - "id": "166cfafc", - "metadata": { - "editable": true - }, - "source": [ - "## Momentum parameter\n", - "\n", - "Notice that this equation is identical to previous one if we identify\n", - "the position of the particle, $\\mathbf{w}$, with the parameters\n", - "$\\boldsymbol{\\theta}$. This allows us to identify the momentum\n", - "parameter and learning rate with the mass of the particle and the\n", - "viscous drag as:" - ] - }, - { - "cell_type": "markdown", - "id": "486fadf0", - "metadata": { - "editable": true - }, - "source": [ - "$$\n", - "\\gamma= {m \\over m +\\mu \\Delta t }, \\qquad \\eta = {(\\Delta t)^2 \\over m +\\mu \\Delta t}.\n", - "$$" - ] - }, - { - "cell_type": "markdown", - "id": "cdb0769d", - "metadata": { - "editable": true - }, - "source": [ - "Thus, as the name suggests, the momentum parameter is proportional to\n", - "the mass of the particle and effectively provides inertia.\n", - "Furthermore, in the large viscosity/small learning rate limit, our\n", - "memory time scales as $(1-\\gamma)^{-1} \\approx m/(\\mu \\Delta t)$.\n", - "\n", - "Why is momentum useful? SGD momentum helps the gradient descent\n", - "algorithm gain speed in directions with persistent but small gradients\n", - "even in the presence of stochasticity, while suppressing oscillations\n", - "in high-curvature directions. This becomes especially important in\n", - "situations where the landscape is shallow and flat in some directions\n", - "and narrow and steep in others. It has been argued that first-order\n", - "methods (with appropriate initial conditions) can perform comparable\n", - "to more expensive second order methods, especially in the context of\n", - "complex deep learning models.\n", - "\n", - "These beneficial properties of momentum can sometimes become even more\n", - "pronounced by using a slight modification of the classical momentum\n", - "algorithm called Nesterov Accelerated Gradient (NAG).\n", - "\n", - "In the NAG algorithm, rather than calculating the gradient at the\n", - "current parameters, $\\nabla_\\theta E(\\boldsymbol{\\theta}_t)$, one\n", - "calculates the gradient at the expected value of the parameters given\n", - "our current momentum, $\\nabla_\\theta E(\\boldsymbol{\\theta}_t +\\gamma\n", - "\\mathbf{v}_{t-1})$. This yields the NAG update rule" - ] - }, - { - "cell_type": "markdown", - "id": "e65569bb", - "metadata": { - "editable": true - }, - "source": [ - "$$\n", - "\\mathbf{v}_{t}=\\gamma \\mathbf{v}_{t-1}+\\eta_{t}\\nabla_\\theta E(\\boldsymbol{\\theta}_t +\\gamma \\mathbf{v}_{t-1}) \\nonumber\n", - "$$" - ] - }, - { - "cell_type": "markdown", - "id": "9d72d316", - "metadata": { - "editable": true - }, - "source": [ - "\n", - "
    \n", - "\n", - "$$\n", - "\\begin{equation} \n", - "\\boldsymbol{\\theta}_{t+1}= \\boldsymbol{\\theta}_t -\\mathbf{v}_{t}.\n", - "\\label{_auto2} \\tag{2}\n", - "\\end{equation}\n", - "$$" - ] - }, - { - "cell_type": "markdown", - "id": "b7b9811e", - "metadata": { - "editable": true - }, - "source": [ - "One of the major advantages of NAG is that it allows for the use of a\n", - "larger learning rate than GDM for the same choice of $\\gamma$." - ] - }, - { - "cell_type": "markdown", - "id": "d8f3c523", + "id": "b921e0a8", "metadata": { "editable": true }, @@ -1593,7 +1347,7 @@ }, { "cell_type": "markdown", - "id": "22d6be74", + "id": "0ce23497", "metadata": { "editable": true }, @@ -1615,14 +1369,14 @@ }, { "cell_type": "markdown", - "id": "091ace5a", + "id": "072e6d0d", "metadata": { "editable": true }, "source": [ "## Motivation for Adaptive Step Sizes\n", "\n", - "1. Instead of a fixed global $\\eta$, use an \\textbf{adaptive learning rate} for each parameter that depends on the history of gradients.\n", + "1. Instead of a fixed global $\\eta$, use an **adaptive learning rate** for each parameter that depends on the history of gradients.\n", "\n", "2. Parameters that have large accumulated gradient magnitude should get smaller steps (they've been changing a lot), whereas parameters with small or infrequent gradients can have larger relative steps.\n", "\n", @@ -1635,457 +1389,7 @@ }, { "cell_type": "markdown", - "id": "5c05fb8b", - "metadata": { - "editable": true - }, - "source": [ - "## Derivation of the AdaGrad Algorithm\n", - "\n", - "**Accumulating Gradient History.**\n", - "\n", - "1. AdaGrad maintains a running sum of squared gradients for each parameter (coordinate)\n", - "\n", - "2. Let $g_t = \\nabla \\ell_{i_t}(x_t)$ be the gradient at step $t$ (or a subgradient for nondifferentiable cases).\n", - "\n", - "3. Initialize $h_0 = 0$ (an all-zero vector in $\\mathbb{R}^d$).\n", - "\n", - "4. At each iteration $t$, update the accumulation:" - ] - }, - { - "cell_type": "markdown", - "id": "21fb8bd0", - "metadata": { - "editable": true - }, - "source": [ - "$$\n", - "h_t \\;=\\; h_{t-1} + g_t \\circ g_t,\n", - "$$" - ] - }, - { - "cell_type": "markdown", - "id": "f71dfafa", - "metadata": { - "editable": true - }, - "source": [ - "1. Here $g_t \\circ g_t$ denotes element-wise square of the gradient vector. $h_t^{(j)} = h_{t-1}^{(j)} + (g_{t,j})^2$ for each parameter $j$.\n", - "\n", - "2. We can view $H_t = \\mathrm{diag}(h_t)$ as a diagonal matrix of past squared gradients. Initially $H_0 = 0$." - ] - }, - { - "cell_type": "markdown", - "id": "64869e2e", - "metadata": { - "editable": true - }, - "source": [ - "## AdaGrad Update Rule Derivation\n", - "\n", - "We scale the gradient by the inverse square root of the accumulated matrix $H_t$. The AdaGrad update at step $t$ is:" - ] - }, - { - "cell_type": "markdown", - "id": "c2d8d6b1", - "metadata": { - "editable": true - }, - "source": [ - "$$\n", - "x_{t+1} \\;=\\; x_t \\;-\\; \\eta\\, H_t^{-1/2}\\, g_t,\n", - "$$" - ] - }, - { - "cell_type": "markdown", - "id": "2171198a", - "metadata": { - "editable": true - }, - "source": [ - "where $H_t^{-1/2}$ is the diagonal matrix with entries $(h_{t}^{(1)})^{-1/2}, \\dots, (h_{t}^{(d)})^{-1/2}$\n", - "In coordinates, this means each parameter $j$ has an individual step size:" - ] - }, - { - "cell_type": "markdown", - "id": "353e21d3", - "metadata": { - "editable": true - }, - "source": [ - "$$\n", - "x_{t+1,j} =x_{t,j} -\\frac{\\eta}{\\sqrt{h_{t,j}}}g_{t,j}.\n", - "$$" - ] - }, - { - "cell_type": "markdown", - "id": "b59e6add", - "metadata": { - "editable": true - }, - "source": [ - "In practice we add a small constant $\\epsilon$ in the denominator for numerical stability to avoid division by zero:" - ] - }, - { - "cell_type": "markdown", - "id": "707caefe", - "metadata": { - "editable": true - }, - "source": [ - "$$\n", - "x_{t+1,j}= x_{t,j}-\\frac{\\eta}{\\sqrt{\\epsilon + h_{t,j}}}g_{t,j}.\n", - "$$" - ] - }, - { - "cell_type": "markdown", - "id": "62dafce4", - "metadata": { - "editable": true - }, - "source": [ - "Equivalently, the effective learning rate for parameter $j$ at time $t$ is $\\displaystyle \\alpha_{t,j} = \\frac{\\eta}{\\sqrt{\\epsilon + h_{t,j}}}$. This decreases over time as $h_{t,j}$ grows." - ] - }, - { - "cell_type": "markdown", - "id": "01a2ba61", - "metadata": { - "editable": true - }, - "source": [ - "## AdaGrad Properties\n", - "1. AdaGrad automatically tunes the step size for each parameter. Parameters with more **volatile or large gradients** get smaller steps, and those with **small or infrequent gradients** get relatively larger steps\n", - "\n", - "2. No manual schedule needed: The accumulation $h_t$ keeps increasing (or stays the same if gradient is zero), so step sizes $\\eta/\\sqrt{h_t}$ are non-increasing. This has a similar effect to a learning rate schedule, but individualized per coordinate.\n", - "\n", - "3. Sparse data benefit: For very sparse features, $h_{t,j}$ grows slowly, so that feature’s parameter retains a higher learning rate for longer, allowing it to make significant updates when it does get a gradient signal\n", - "\n", - "4. Convergence: In convex optimization, AdaGrad can be shown to achieve a sub-linear convergence rate (e.g. $O(1/\\sqrt{T})$ regret bound) comparable to the best fixed learning rate tuned for the problem\n", - "\n", - "It effectively reduces the need to tune $\\eta$ by hand.\n", - "1. Limitations: Because $h_t$ accumulates without bound, AdaGrad’s learning rates can become extremely small over long training, potentially slowing progress. (Later variants like RMSProp, AdaDelta, Adam address this by modifying the accumulation rule.)" - ] - }, - { - "cell_type": "markdown", - "id": "b4bc88e0", - "metadata": { - "editable": true - }, - "source": [ - "## RMSProp: Adaptive Learning Rates\n", - "Addresses AdaGrad’s diminishing learning rate issue.\n", - "Uses a decaying average of squared gradients (instead of a cumulative sum):" - ] - }, - { - "cell_type": "markdown", - "id": "4933fddb", - "metadata": { - "editable": true - }, - "source": [ - "$$\n", - "v_t = \\beta_2\\, v_{t-1} + (1-\\beta_2)\\, (\\nabla L(w_t))^2,\n", - "$$" - ] - }, - { - "cell_type": "markdown", - "id": "0dd8edf7", - "metadata": { - "editable": true - }, - "source": [ - "with $\\beta_2$ typically $0.9$ (or $0.99$).\n", - "1. Update: $w_{t+1} = w_t - \\frac{\\alpha}{\\sqrt{v_t + \\epsilon}} \\nabla L(w_t)$.\n", - "\n", - "2. Recent gradients have more weight, so $v_t$ adapts to the current landscape.\n", - "\n", - "3. Avoids AdaGrad’s “infinite memory” problem – learning rate does not continuously decay to zero.\n", - "\n", - "RMSProp was first proposed in lecture notes by Geoff Hinton, 2012 - unpublished.)" - ] - }, - { - "cell_type": "markdown", - "id": "1ce98f28", - "metadata": { - "editable": true - }, - "source": [ - "## Adam Optimizer\n", - "\n", - "Why Combine Momentum and RMSProp? Motivation for Adam: Adaptive Moment Estimation (Adam) was introduced by Kingma an Ba (2014) to combine the benefits of momentum and RMSProp.\n", - "\n", - "1. Fast convergence by smoothing gradients (accelerates in long-term gradient direction).\n", - "\n", - "2. Adaptive rates (RMSProp): Per-dimension learning rate scaling for stability (handles different feature scales, sparse gradients).\n", - "\n", - "3. Adam uses both: maintains moving averages of both first moment (gradients) and second moment (squared gradients)\n", - "\n", - "4. Additionally, includes a mechanism to correct the bias in these moving averages (crucial in early iterations)\n", - "\n", - "**Result**: Adam is robust, achieves faster convergence with less tuning, and often outperforms SGD (with momentum) in practice." - ] - }, - { - "cell_type": "markdown", - "id": "d7d646bc", - "metadata": { - "editable": true - }, - "source": [ - "## RMS prop\n", - "\n", - "In RMS prop, in addition to keeping a running average of the first\n", - "moment of the gradient, we also keep track of the second moment\n", - "denoted by $\\mathbf{s}_t=\\mathbb{E}[\\mathbf{g}_t^2]$. The update rule\n", - "for RMS prop is given by" - ] - }, - { - "cell_type": "markdown", - "id": "3a212292", - "metadata": { - "editable": true - }, - "source": [ - "\n", - "
    \n", - "\n", - "$$\n", - "\\begin{equation}\n", - "\\mathbf{g}_t = \\nabla_\\theta E(\\boldsymbol{\\theta}) \n", - "\\label{_auto3} \\tag{3}\n", - "\\end{equation}\n", - "$$" - ] - }, - { - "cell_type": "markdown", - "id": "88c0ec96", - "metadata": { - "editable": true - }, - "source": [ - "$$\n", - "\\mathbf{s}_t =\\beta \\mathbf{s}_{t-1} +(1-\\beta)\\mathbf{g}_t^2 \\nonumber\n", - "$$" - ] - }, - { - "cell_type": "markdown", - "id": "9cfb794f", - "metadata": { - "editable": true - }, - "source": [ - "$$\n", - "\\boldsymbol{\\theta}_{t+1}=\\boldsymbol{\\theta}_t - \\eta_t { \\mathbf{g}_t \\over \\sqrt{\\mathbf{s}_t +\\epsilon}}, \\nonumber\n", - "$$" - ] - }, - { - "cell_type": "markdown", - "id": "aaa4936c", - "metadata": { - "editable": true - }, - "source": [ - "where $\\beta$ controls the averaging time of the second moment and is\n", - "typically taken to be about $\\beta=0.9$, $\\eta_t$ is a learning rate\n", - "typically chosen to be $10^{-3}$, and $\\epsilon\\sim 10^{-8} $ is a\n", - "small regularization constant to prevent divergences. Multiplication\n", - "and division by vectors is understood as an element-wise operation. It\n", - "is clear from this formula that the learning rate is reduced in\n", - "directions where the norm of the gradient is consistently large. This\n", - "greatly speeds up the convergence by allowing us to use a larger\n", - "learning rate for flat directions." - ] - }, - { - "cell_type": "markdown", - "id": "5493a6cd", - "metadata": { - "editable": true - }, - "source": [ - "## [ADAM optimizer](https://arxiv.org/abs/1412.6980)\n", - "\n", - "A related algorithm is the ADAM optimizer. In\n", - "[ADAM](https://arxiv.org/abs/1412.6980), we keep a running average of\n", - "both the first and second moment of the gradient and use this\n", - "information to adaptively change the learning rate for different\n", - "parameters. The method isefficient when working with large\n", - "problems involving lots data and/or parameters. It is a combination of the\n", - "gradient descent with momentum algorithm and the RMSprop algorithm\n", - "discussed above.\n", - "\n", - "In addition to keeping a running average of the first and\n", - "second moments of the gradient\n", - "(i.e. $\\mathbf{m}_t=\\mathbb{E}[\\mathbf{g}_t]$ and\n", - "$\\mathbf{s}_t=\\mathbb{E}[\\mathbf{g}^2_t]$, respectively), ADAM\n", - "performs an additional bias correction to account for the fact that we\n", - "are estimating the first two moments of the gradient using a running\n", - "average (denoted by the hats in the update rule below). The update\n", - "rule for ADAM is given by (where multiplication and division are once\n", - "again understood to be element-wise operations below)" - ] - }, - { - "cell_type": "markdown", - "id": "35795070", - "metadata": { - "editable": true - }, - "source": [ - "\n", - "
    \n", - "\n", - "$$\n", - "\\begin{equation}\n", - "\\mathbf{g}_t = \\nabla_\\theta E(\\boldsymbol{\\theta}) \n", - "\\label{_auto4} \\tag{4}\n", - "\\end{equation}\n", - "$$" - ] - }, - { - "cell_type": "markdown", - "id": "1e67e6dd", - "metadata": { - "editable": true - }, - "source": [ - "$$\n", - "\\mathbf{m}_t = \\beta_1 \\mathbf{m}_{t-1} + (1-\\beta_1) \\mathbf{g}_t \\nonumber\n", - "$$" - ] - }, - { - "cell_type": "markdown", - "id": "39d582bb", - "metadata": { - "editable": true - }, - "source": [ - "$$\n", - "\\mathbf{s}_t =\\beta_2 \\mathbf{s}_{t-1} +(1-\\beta_2)\\mathbf{g}_t^2 \\nonumber\n", - "$$" - ] - }, - { - "cell_type": "markdown", - "id": "4a7e2423", - "metadata": { - "editable": true - }, - "source": [ - "$$\n", - "\\boldsymbol{\\mathbf{m}}_t={\\mathbf{m}_t \\over 1-\\beta_1^t} \\nonumber\n", - "$$" - ] - }, - { - "cell_type": "markdown", - "id": "53da77c2", - "metadata": { - "editable": true - }, - "source": [ - "$$\n", - "\\boldsymbol{\\mathbf{s}}_t ={\\mathbf{s}_t \\over1-\\beta_2^t} \\nonumber\n", - "$$" - ] - }, - { - "cell_type": "markdown", - "id": "a4c7df45", - "metadata": { - "editable": true - }, - "source": [ - "$$\n", - "\\boldsymbol{\\theta}_{t+1}=\\boldsymbol{\\theta}_t - \\eta_t { \\boldsymbol{\\mathbf{m}}_t \\over \\sqrt{\\boldsymbol{\\mathbf{s}}_t} +\\epsilon}, \\nonumber\n", - "$$" - ] - }, - { - "cell_type": "markdown", - "id": "f645a0aa", - "metadata": { - "editable": true - }, - "source": [ - "\n", - "
    \n", - "\n", - "$$\n", - "\\begin{equation} \n", - "\\label{_auto5} \\tag{5}\n", - "\\end{equation}\n", - "$$" - ] - }, - { - "cell_type": "markdown", - "id": "505619bb", - "metadata": { - "editable": true - }, - "source": [ - "where $\\beta_1$ and $\\beta_2$ set the memory lifetime of the first and\n", - "second moment and are typically taken to be $0.9$ and $0.99$\n", - "respectively, and $\\eta$ and $\\epsilon$ are identical to RMSprop.\n", - "\n", - "Like in RMSprop, the effective step size of a parameter depends on the\n", - "magnitude of its gradient squared. To understand this better, let us\n", - "rewrite this expression in terms of the variance\n", - "$\\boldsymbol{\\sigma}_t^2 = \\boldsymbol{\\mathbf{s}}_t -\n", - "(\\boldsymbol{\\mathbf{m}}_t)^2$. Consider a single parameter $\\theta_t$. The\n", - "update rule for this parameter is given by" - ] - }, - { - "cell_type": "markdown", - "id": "6cb4c9a9", - "metadata": { - "editable": true - }, - "source": [ - "$$\n", - "\\Delta \\theta_{t+1}= -\\eta_t { \\boldsymbol{m}_t \\over \\sqrt{\\sigma_t^2 + m_t^2 }+\\epsilon}.\n", - "$$" - ] - }, - { - "cell_type": "markdown", - "id": "39627950", - "metadata": { - "editable": true - }, - "source": [ - "## Algorithms and codes for Adagrad, RMSprop and Adam\n", - "\n", - "The algorithms we have implemented are well described in the text by [Goodfellow, Bengio and Courville, chapter 8](https://www.deeplearningbook.org/contents/optimization.html).\n", - "\n", - "The codes which implement these algorithms are discussed after our presentation of automatic differentiation." - ] - }, - { - "cell_type": "markdown", - "id": "f240057d", + "id": "05c46307", "metadata": { "editable": true }, @@ -2101,7 +1405,193 @@ }, { "cell_type": "markdown", - "id": "428ee09d", + "id": "183480ab", + "metadata": { + "editable": true + }, + "source": [ + "## Derivation of the AdaGrad Algorithm\n", + "\n", + "**Accumulating Gradient History.**\n", + "\n", + "1. AdaGrad maintains a running sum of squared gradients for each parameter (coordinate)\n", + "\n", + "2. Let $g_t = \\nabla C_{i_t}(x_t)$ be the gradient at step $t$ (or a subgradient for nondifferentiable cases).\n", + "\n", + "3. Initialize $r_0 = 0$ (an all-zero vector in $\\mathbb{R}^d$).\n", + "\n", + "4. At each iteration $t$, update the accumulation:" + ] + }, + { + "cell_type": "markdown", + "id": "0bffad7b", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "r_t = r_{t-1} + g_t \\circ g_t,\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "289693d6", + "metadata": { + "editable": true + }, + "source": [ + "1. Here $g_t \\circ g_t$ denotes element-wise square of the gradient vector. $g_t^{(j)} = g_{t-1}^{(j)} + (g_{t,j})^2$ for each parameter $j$.\n", + "\n", + "2. We can view $H_t = \\mathrm{diag}(r_t)$ as a diagonal matrix of past squared gradients. Initially $H_0 = 0$." + ] + }, + { + "cell_type": "markdown", + "id": "98e5e657", + "metadata": { + "editable": true + }, + "source": [ + "## AdaGrad Update Rule Derivation\n", + "\n", + "We scale the gradient by the inverse square root of the accumulated matrix $H_t$. The AdaGrad update at step $t$ is:" + ] + }, + { + "cell_type": "markdown", + "id": "d2b6b9cd", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\theta_{t+1} =\\theta_t - \\eta H_t^{-1/2} g_t,\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "795888a2", + "metadata": { + "editable": true + }, + "source": [ + "where $H_t^{-1/2}$ is the diagonal matrix with entries $(r_{t}^{(1)})^{-1/2}, \\dots, (r_{t}^{(d)})^{-1/2}$\n", + "In coordinates, this means each parameter $j$ has an individual step size:" + ] + }, + { + "cell_type": "markdown", + "id": "283c4520", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\theta_{t+1,j} =\\theta_{t,j} -\\frac{\\eta}{\\sqrt{r_{t,j}}}g_{t,j}.\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "0958350b", + "metadata": { + "editable": true + }, + "source": [ + "In practice we add a small constant $\\epsilon$ in the denominator for numerical stability to avoid division by zero:" + ] + }, + { + "cell_type": "markdown", + "id": "f806f43a", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\theta_{t+1,j}= \\theta_{t,j}-\\frac{\\eta}{\\sqrt{\\epsilon + r_{t,j}}}g_{t,j}.\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "2c7d2de6", + "metadata": { + "editable": true + }, + "source": [ + "Equivalently, the effective learning rate for parameter $j$ at time $t$ is $\\displaystyle \\alpha_{t,j} = \\frac{\\eta}{\\sqrt{\\epsilon + r_{t,j}}}$. This decreases over time as $r_{t,j}$ grows." + ] + }, + { + "cell_type": "markdown", + "id": "afabce90", + "metadata": { + "editable": true + }, + "source": [ + "## AdaGrad Properties\n", + "\n", + "1. AdaGrad automatically tunes the step size for each parameter. Parameters with more *volatile or large gradients* get smaller steps, and those with *small or infrequent gradients* get relatively larger steps\n", + "\n", + "2. No manual schedule needed: The accumulation $r_t$ keeps increasing (or stays the same if gradient is zero), so step sizes $\\eta/\\sqrt{r_t}$ are non-increasing. This has a similar effect to a learning rate schedule, but individualized per coordinate.\n", + "\n", + "3. Sparse data benefit: For very sparse features, $r_{t,j}$ grows slowly, so that feature’s parameter retains a higher learning rate for longer, allowing it to make significant updates when it does get a gradient signal\n", + "\n", + "4. Convergence: In convex optimization, AdaGrad can be shown to achieve a sub-linear convergence rate comparable to the best fixed learning rate tuned for the problem\n", + "\n", + "It effectively reduces the need to tune $\\eta$ by hand.\n", + "1. Limitations: Because $r_t$ accumulates without bound, AdaGrad’s learning rates can become extremely small over long training, potentially slowing progress. (Later variants like RMSProp, AdaDelta, Adam address this by modifying the accumulation rule.)" + ] + }, + { + "cell_type": "markdown", + "id": "489bb0db", + "metadata": { + "editable": true + }, + "source": [ + "## RMSProp: Adaptive Learning Rates\n", + "\n", + "Addresses AdaGrad’s diminishing learning rate issue.\n", + "Uses a decaying average of squared gradients (instead of a cumulative sum):" + ] + }, + { + "cell_type": "markdown", + "id": "b5808edc", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "v_t = \\rho v_{t-1} + (1-\\rho)(\\nabla C(\\theta_t))^2,\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "c36f305f", + "metadata": { + "editable": true + }, + "source": [ + "with $\\rho$ typically $0.9$ (or $0.99$).\n", + "1. Update: $\\theta_{t+1} = \\theta_t - \\frac{\\eta}{\\sqrt{v_t + \\epsilon}} \\nabla C(\\theta_t)$.\n", + "\n", + "2. Recent gradients have more weight, so $v_t$ adapts to the current landscape.\n", + "\n", + "3. Avoids AdaGrad’s “infinite memory” problem – learning rate does not continuously decay to zero.\n", + "\n", + "RMSProp was first proposed in lecture notes by Geoff Hinton, 2012 - unpublished.)" + ] + }, + { + "cell_type": "markdown", + "id": "6e4679b3", "metadata": { "editable": true }, @@ -2117,7 +1607,251 @@ }, { "cell_type": "markdown", - "id": "9ce3cc3e", + "id": "d58547fa", + "metadata": { + "editable": true + }, + "source": [ + "## Adam Optimizer\n", + "\n", + "Why combine Momentum and RMSProp? Motivation for Adam: Adaptive Moment Estimation (Adam) was introduced by Kingma an Ba (2014) to combine the benefits of momentum and RMSProp.\n", + "\n", + "1. Fast convergence by smoothing gradients (accelerates in long-term gradient direction).\n", + "\n", + "2. Adaptive rates (RMSProp): Per-dimension learning rate scaling for stability (handles different feature scales, sparse gradients).\n", + "\n", + "3. Adam uses both: maintains moving averages of both first moment (gradients) and second moment (squared gradients)\n", + "\n", + "4. Additionally, includes a mechanism to correct the bias in these moving averages (crucial in early iterations)\n", + "\n", + "**Result**: Adam is robust, achieves faster convergence with less tuning, and often outperforms SGD (with momentum) in practice." + ] + }, + { + "cell_type": "markdown", + "id": "6989903c", + "metadata": { + "editable": true + }, + "source": [ + "## [ADAM optimizer](https://arxiv.org/abs/1412.6980)\n", + "\n", + "In [ADAM](https://arxiv.org/abs/1412.6980), we keep a running average of\n", + "both the first and second moment of the gradient and use this\n", + "information to adaptively change the learning rate for different\n", + "parameters. The method is efficient when working with large\n", + "problems involving lots data and/or parameters. It is a combination of the\n", + "gradient descent with momentum algorithm and the RMSprop algorithm\n", + "discussed above." + ] + }, + { + "cell_type": "markdown", + "id": "a58fbe05", + "metadata": { + "editable": true + }, + "source": [ + "## Why Combine Momentum and RMSProp?\n", + "\n", + "1. Momentum: Fast convergence by smoothing gradients (accelerates in long-term gradient direction).\n", + "\n", + "2. Adaptive rates (RMSProp): Per-dimension learning rate scaling for stability (handles different feature scales, sparse gradients).\n", + "\n", + "3. Adam uses both: maintains moving averages of both first moment (gradients) and second moment (squared gradients)\n", + "\n", + "4. Additionally, includes a mechanism to correct the bias in these moving averages (crucial in early iterations)\n", + "\n", + "Result: Adam is robust, achieves faster convergence with less tuning, and often outperforms SGD (with momentum) in practice" + ] + }, + { + "cell_type": "markdown", + "id": "98700cff", + "metadata": { + "editable": true + }, + "source": [ + "## Adam: Exponential Moving Averages (Moments)\n", + "Adam maintains two moving averages at each time step $t$ for each parameter $w$:\n", + "**First moment (mean) $m_t$.**\n", + "\n", + "The Momentum term" + ] + }, + { + "cell_type": "markdown", + "id": "e55261d4", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "m_t = \\beta_1m_{t-1} + (1-\\beta_1)\\, \\nabla C(\\theta_t),\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "64676960", + "metadata": { + "editable": true + }, + "source": [ + "**Second moment (uncentered variance) $v_t$.**\n", + "\n", + "The RMS term" + ] + }, + { + "cell_type": "markdown", + "id": "1e3007bd", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "v_t = \\beta_2v_{t-1} + (1-\\beta_2)(\\nabla C(\\theta_t))^2,\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "5df06a58", + "metadata": { + "editable": true + }, + "source": [ + "with typical $\\beta_1 = 0.9$, $\\beta_2 = 0.999$. Initialize $m_0 = 0$, $v_0 = 0$.\n", + "\n", + " These are **biased** estimators of the true first and second moment of the gradients, especially at the start (since $m_0,v_0$ are zero)" + ] + }, + { + "cell_type": "markdown", + "id": "034cb12a", + "metadata": { + "editable": true + }, + "source": [ + "## Adam: Bias Correction\n", + "To counteract initialization bias in $m_t, v_t$, Adam computes bias-corrected estimates" + ] + }, + { + "cell_type": "markdown", + "id": "429a18be", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\hat{m}_t = \\frac{m_t}{1 - \\beta_1^t}, \\qquad \\hat{v}_t = \\frac{v_t}{1 - \\beta_2^t}.\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "1672b4fd", + "metadata": { + "editable": true + }, + "source": [ + "* When $t$ is small, $1-\\beta_i^t \\approx 0$, so $\\hat{m}_t, \\hat{v}_t$ significantly larger than raw $m_t, v_t$, compensating for the initial zero bias.\n", + "\n", + "* As $t$ increases, $1-\\beta_i^t \\to 1$, and $\\hat{m}_t, \\hat{v}_t$ converge to $m_t, v_t$.\n", + "\n", + "* Bias correction is important for Adam’s stability in early iterations" + ] + }, + { + "cell_type": "markdown", + "id": "dba6d945", + "metadata": { + "editable": true + }, + "source": [ + "## Adam: Update Rule Derivation\n", + "Finally, Adam updates parameters using the bias-corrected moments:" + ] + }, + { + "cell_type": "markdown", + "id": "13db1c10", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\theta_{t+1} =\\theta_t -\\frac{\\alpha}{\\sqrt{\\hat{v}_t} + \\epsilon}\\hat{m}_t,\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "e30b02ab", + "metadata": { + "editable": true + }, + "source": [ + "where $\\epsilon$ is a small constant (e.g. $10^{-8}$) to prevent division by zero.\n", + "Breaking it down:\n", + "1. Compute gradient $\\nabla C(\\theta_t)$.\n", + "\n", + "2. Update first moment $m_t$ and second moment $v_t$ (exponential moving averages).\n", + "\n", + "3. Bias-correct: $\\hat{m}_t = m_t/(1-\\beta_1^t)$, $\\; \\hat{v}_t = v_t/(1-\\beta_2^t)$.\n", + "\n", + "4. Compute step: $\\Delta \\theta_t = \\frac{\\hat{m}_t}{\\sqrt{\\hat{v}_t} + \\epsilon}$.\n", + "\n", + "5. Update parameters: $\\theta_{t+1} = \\theta_t - \\alpha\\, \\Delta \\theta_t$.\n", + "\n", + "This is the Adam update rule as given in the original paper." + ] + }, + { + "cell_type": "markdown", + "id": "5ac8ae68", + "metadata": { + "editable": true + }, + "source": [ + "## Adam vs. AdaGrad and RMSProp\n", + "\n", + "1. AdaGrad: Uses per-coordinate scaling like Adam, but no momentum. Tends to slow down too much due to cumulative history (no forgetting)\n", + "\n", + "2. RMSProp: Uses moving average of squared gradients (like Adam’s $v_t$) to maintain adaptive learning rates, but does not include momentum or bias-correction.\n", + "\n", + "3. Adam: Effectively RMSProp + Momentum + Bias-correction\n", + "\n", + " * Momentum ($m_t$) provides acceleration and smoother convergence.\n", + "\n", + " * Adaptive $v_t$ scaling moderates the step size per dimension.\n", + "\n", + " * Bias correction (absent in AdaGrad/RMSProp) ensures robust estimates early on.\n", + "\n", + "In practice, Adam often yields faster convergence and better tuning stability than RMSProp or AdaGrad alone" + ] + }, + { + "cell_type": "markdown", + "id": "86abec40", + "metadata": { + "editable": true + }, + "source": [ + "## Adaptivity Across Dimensions\n", + "\n", + "1. Adam adapts the step size \\emph{per coordinate}: parameters with larger gradient variance get smaller effective steps, those with smaller or sparse gradients get larger steps.\n", + "\n", + "2. This per-dimension adaptivity is inherited from AdaGrad/RMSProp and helps handle ill-conditioned or sparse problems.\n", + "\n", + "3. Meanwhile, momentum (first moment) allows Adam to continue making progress even if gradients become small or noisy, by leveraging accumulated direction." + ] + }, + { + "cell_type": "markdown", + "id": "448ff439", "metadata": { "editable": true }, @@ -2133,7 +1867,21 @@ }, { "cell_type": "markdown", - "id": "193b3778", + "id": "86183049", + "metadata": { + "editable": true + }, + "source": [ + "## Algorithms and codes for Adagrad, RMSprop and Adam\n", + "\n", + "The algorithms we have implemented are well described in the text by [Goodfellow, Bengio and Courville, chapter 8](https://www.deeplearningbook.org/contents/optimization.html).\n", + "\n", + "The codes which implement these algorithms are discussed below here." + ] + }, + { + "cell_type": "markdown", + "id": "78a7d51b", "metadata": { "editable": true }, @@ -2151,7 +1899,7 @@ }, { "cell_type": "markdown", - "id": "8d7187f5", + "id": "0b0e62b6", "metadata": { "editable": true }, @@ -2172,7 +1920,7 @@ { "cell_type": "code", "execution_count": 9, - "id": "68d35b76", + "id": "d19005f0", "metadata": { "collapsed": false, "editable": true @@ -2186,8 +1934,8 @@ "import matplotlib.pyplot as plt\n", "from autograd import grad\n", "\n", - "def CostOLS(beta):\n", - " return (1.0/n)*np.sum((y-X @ beta)**2)\n", + "def CostOLS(theta):\n", + " return (1.0/n)*np.sum((y-X @ theta)**2)\n", "\n", "n = 100\n", "x = 2*np.random.rand(n,1)\n", @@ -2232,7 +1980,7 @@ }, { "cell_type": "markdown", - "id": "f39b471c", + "id": "4f7fa973", "metadata": { "editable": true }, @@ -2243,7 +1991,7 @@ { "cell_type": "code", "execution_count": 10, - "id": "265059ef", + "id": "34cb0de4", "metadata": { "collapsed": false, "editable": true @@ -2257,8 +2005,8 @@ "import matplotlib.pyplot as plt\n", "from autograd import grad\n", "\n", - "def CostOLS(beta):\n", - " return (1.0/n)*np.sum((y-X @ beta)**2)\n", + "def CostOLS(theta):\n", + " return (1.0/n)*np.sum((y-X @ theta)**2)\n", "\n", "n = 100\n", "x = 2*np.random.rand(n,1)\n", @@ -2307,7 +2055,7 @@ }, { "cell_type": "markdown", - "id": "18183fb0", + "id": "4cb35cee", "metadata": { "editable": true }, @@ -2322,7 +2070,7 @@ { "cell_type": "code", "execution_count": 11, - "id": "4d9d93b3", + "id": "74934aab", "metadata": { "collapsed": false, "editable": true @@ -2406,7 +2154,7 @@ }, { "cell_type": "markdown", - "id": "1652a471", + "id": "d8ee3d35", "metadata": { "editable": true }, @@ -2417,7 +2165,7 @@ { "cell_type": "code", "execution_count": 12, - "id": "6092712c", + "id": "63184f2b", "metadata": { "collapsed": false, "editable": true @@ -2495,7 +2243,7 @@ }, { "cell_type": "markdown", - "id": "4d2f230d", + "id": "c44ce795", "metadata": { "editable": true }, @@ -2508,7 +2256,7 @@ { "cell_type": "code", "execution_count": 13, - "id": "b46cedcc", + "id": "74e118d6", "metadata": { "collapsed": false, "editable": true @@ -2521,8 +2269,8 @@ "import autograd.numpy as np\n", "from autograd import grad\n", "\n", - "def CostOLS(beta):\n", - " return (1.0/n)*np.sum((y-X @ beta)**2)\n", + "def CostOLS(theta):\n", + " return (1.0/n)*np.sum((y-X @ theta)**2)\n", "\n", "n = 100\n", "x = 2*np.random.rand(n,1)\n", @@ -2530,29 +2278,29 @@ "\n", "X = np.c_[np.ones((n,1)), x, x*x]\n", "XT_X = X.T @ X\n", - "beta_linreg = np.linalg.pinv(XT_X) @ (X.T @ y)\n", + "theta_linreg = np.linalg.pinv(XT_X) @ (X.T @ y)\n", "print(\"Own inversion\")\n", - "print(beta_linreg)\n", + "print(theta_linreg)\n", "# Hessian matrix\n", "H = (2.0/n)* XT_X\n", - "# Note that here the Hessian does not depend on the parameters beta\n", + "# Note that here the Hessian does not depend on the parameters theta\n", "invH = np.linalg.pinv(H)\n", - "beta = np.random.randn(3,1)\n", + "theta = np.random.randn(3,1)\n", "Niterations = 5\n", "# define the gradient\n", "training_gradient = grad(CostOLS)\n", "\n", "for iter in range(Niterations):\n", - " gradients = training_gradient(beta)\n", - " beta -= invH @ gradients\n", + " gradients = training_gradient(theta)\n", + " theta -= invH @ gradients\n", " print(iter,gradients[0],gradients[1])\n", - "print(\"beta from own Newton code\")\n", - "print(beta)" + "print(\"theta from own Newton code\")\n", + "print(theta)" ] }, { "cell_type": "markdown", - "id": "f048d5c7", + "id": "3789172e", "metadata": { "editable": true }, @@ -2563,7 +2311,7 @@ { "cell_type": "code", "execution_count": 14, - "id": "d8bdd180", + "id": "5152d211", "metadata": { "collapsed": false, "editable": true @@ -2622,7 +2370,7 @@ }, { "cell_type": "markdown", - "id": "145ce394", + "id": "34e87757", "metadata": { "editable": true }, @@ -2632,7 +2380,7 @@ }, { "cell_type": "markdown", - "id": "ec0ee110", + "id": "713154cb", "metadata": { "editable": true }, @@ -2643,7 +2391,7 @@ { "cell_type": "code", "execution_count": 15, - "id": "b6e8e3e4", + "id": "3e863305", "metadata": { "collapsed": false, "editable": true @@ -2708,7 +2456,7 @@ }, { "cell_type": "markdown", - "id": "79d8c0ae", + "id": "a5fa4647", "metadata": { "editable": true }, @@ -2719,7 +2467,7 @@ { "cell_type": "code", "execution_count": 16, - "id": "c16ec5ac", + "id": "0b318139", "metadata": { "collapsed": false, "editable": true @@ -2760,9 +2508,9 @@ "\n", "# Value for learning rate\n", "eta = 0.01\n", - "# Value for parameters beta1 and beta2, see https://arxiv.org/abs/1412.6980\n", - "beta1 = 0.9\n", - "beta2 = 0.999\n", + "# Value for parameters theta1 and theta2, see https://arxiv.org/abs/1412.6980\n", + "theta1 = 0.9\n", + "theta2 = 0.999\n", "# Including AdaGrad parameter to avoid possible division by zero\n", "delta = 1e-7\n", "iter = 0\n", @@ -2776,10 +2524,10 @@ " yi = y[random_index:random_index+M]\n", " gradients = (1.0/M)*training_gradient(yi, xi, theta)\n", " # Computing moments first\n", - " first_moment = beta1*first_moment + (1-beta1)*gradients\n", - " second_moment = beta2*second_moment+(1-beta2)*gradients*gradients\n", - " first_term = first_moment/(1.0-beta1**iter)\n", - " second_term = second_moment/(1.0-beta2**iter)\n", + " first_moment = theta1*first_moment + (1-theta1)*gradients\n", + " second_moment = theta2*second_moment+(1-theta2)*gradients*gradients\n", + " first_term = first_moment/(1.0-theta1**iter)\n", + " second_term = second_moment/(1.0-theta2**iter)\n", "\t# Scaling with rho the new and the previous results\n", " update = eta*first_term/(np.sqrt(second_term)+delta)\n", " theta -= update\n", @@ -2789,7 +2537,7 @@ }, { "cell_type": "markdown", - "id": "7c581f8b", + "id": "967dd1d1", "metadata": { "editable": true }, @@ -2801,12 +2549,12 @@ "2. Work on project 1\n", "\n", "\n", - " * For more discussions of Ridge regression and calculation of averages, [Wessel van Wieringen's](https://arxiv.org/abs/1509.09169) article is highly recommended." + "For more discussions of Ridge regression and calculation of averages, [Wessel van Wieringen's](https://arxiv.org/abs/1509.09169) article is highly recommended." ] }, { "cell_type": "markdown", - "id": "d8a82e01", + "id": "ad137d49", "metadata": { "editable": true }, @@ -2834,7 +2582,7 @@ { "cell_type": "code", "execution_count": 17, - "id": "3fa174a9", + "id": "8a6c2cb4", "metadata": { "collapsed": false, "editable": true @@ -2854,7 +2602,7 @@ }, { "cell_type": "markdown", - "id": "b02de7a6", + "id": "76bf3c81", "metadata": { "editable": true }, @@ -2870,7 +2618,7 @@ }, { "cell_type": "markdown", - "id": "cd3f85ef", + "id": "818c6d58", "metadata": { "editable": true }, @@ -2890,7 +2638,7 @@ }, { "cell_type": "markdown", - "id": "1e0198a0", + "id": "58481e8f", "metadata": { "editable": true }, @@ -2917,7 +2665,7 @@ }, { "cell_type": "markdown", - "id": "04da94ca", + "id": "4e94a32c", "metadata": { "editable": true }, @@ -2930,7 +2678,7 @@ }, { "cell_type": "markdown", - "id": "aa1ad257", + "id": "3a93bb3e", "metadata": { "editable": true }, @@ -2942,7 +2690,7 @@ }, { "cell_type": "markdown", - "id": "6587016d", + "id": "c603fb31", "metadata": { "editable": true }, @@ -2957,7 +2705,7 @@ { "cell_type": "code", "execution_count": 18, - "id": "2e0350b4", + "id": "363ae494", "metadata": { "collapsed": false, "editable": true @@ -2984,7 +2732,7 @@ }, { "cell_type": "markdown", - "id": "8179b037", + "id": "429b71ce", "metadata": { "editable": true }, @@ -2998,7 +2746,7 @@ }, { "cell_type": "markdown", - "id": "3ebed257", + "id": "ec3c7657", "metadata": { "editable": true }, @@ -3010,7 +2758,7 @@ }, { "cell_type": "markdown", - "id": "51b239a4", + "id": "c6c2fd1d", "metadata": { "editable": true }, @@ -3027,7 +2775,7 @@ }, { "cell_type": "markdown", - "id": "295790f3", + "id": "1141934f", "metadata": { "editable": true }, @@ -3039,7 +2787,7 @@ }, { "cell_type": "markdown", - "id": "b68338d6", + "id": "efc84979", "metadata": { "editable": true }, @@ -3049,7 +2797,7 @@ }, { "cell_type": "markdown", - "id": "c015cb3b", + "id": "6be29872", "metadata": { "editable": true }, @@ -3061,7 +2809,7 @@ }, { "cell_type": "markdown", - "id": "5e4917fa", + "id": "8f128adc", "metadata": { "editable": true }, @@ -3071,7 +2819,7 @@ }, { "cell_type": "markdown", - "id": "150fedf5", + "id": "967b009a", "metadata": { "editable": true }, @@ -3083,7 +2831,7 @@ }, { "cell_type": "markdown", - "id": "e6d782b1", + "id": "181e0b18", "metadata": { "editable": true }, @@ -3094,7 +2842,7 @@ }, { "cell_type": "markdown", - "id": "a4065dd3", + "id": "6fe29ace", "metadata": { "editable": true }, @@ -3106,7 +2854,7 @@ }, { "cell_type": "markdown", - "id": "06324d79", + "id": "795a350c", "metadata": { "editable": true }, @@ -3116,7 +2864,7 @@ }, { "cell_type": "markdown", - "id": "9e3e6b35", + "id": "9a0ebe42", "metadata": { "editable": true }, @@ -3128,7 +2876,7 @@ }, { "cell_type": "markdown", - "id": "cd9c3754", + "id": "800ad4c7", "metadata": { "editable": true }, @@ -3138,7 +2886,7 @@ }, { "cell_type": "markdown", - "id": "8361829b", + "id": "4ed44194", "metadata": { "editable": true }, @@ -3150,7 +2898,7 @@ }, { "cell_type": "markdown", - "id": "8f5f46fd", + "id": "f6fc455e", "metadata": { "editable": true }, @@ -3160,7 +2908,7 @@ }, { "cell_type": "markdown", - "id": "cf210a6e", + "id": "1d434e5d", "metadata": { "editable": true }, @@ -3172,7 +2920,7 @@ }, { "cell_type": "markdown", - "id": "d3af09fc", + "id": "011d7385", "metadata": { "editable": true }, @@ -3182,7 +2930,7 @@ }, { "cell_type": "markdown", - "id": "ec91f8c1", + "id": "9bb92755", "metadata": { "editable": true }, @@ -3194,7 +2942,7 @@ }, { "cell_type": "markdown", - "id": "0c9f35e6", + "id": "7287a085", "metadata": { "editable": true }, @@ -3204,7 +2952,7 @@ }, { "cell_type": "markdown", - "id": "de80fa4b", + "id": "1ebd1481", "metadata": { "editable": true }, @@ -3216,7 +2964,7 @@ }, { "cell_type": "markdown", - "id": "34c99b80", + "id": "01da7453", "metadata": { "editable": true }, @@ -3226,7 +2974,7 @@ }, { "cell_type": "markdown", - "id": "758ade4d", + "id": "2719429f", "metadata": { "editable": true }, @@ -3238,7 +2986,7 @@ }, { "cell_type": "markdown", - "id": "ff0f4725", + "id": "1b7a1119", "metadata": { "editable": true }, @@ -3248,7 +2996,7 @@ }, { "cell_type": "markdown", - "id": "997b829b", + "id": "5455e569", "metadata": { "editable": true }, @@ -3260,7 +3008,7 @@ }, { "cell_type": "markdown", - "id": "621e7d85", + "id": "82719ade", "metadata": { "editable": true }, @@ -3272,7 +3020,7 @@ }, { "cell_type": "markdown", - "id": "5530e459", + "id": "d87ce00b", "metadata": { "editable": true }, @@ -3284,7 +3032,7 @@ }, { "cell_type": "markdown", - "id": "fd9bace8", + "id": "aad7612a", "metadata": { "editable": true }, @@ -3294,7 +3042,7 @@ }, { "cell_type": "markdown", - "id": "d949be75", + "id": "3c7ca158", "metadata": { "editable": true }, @@ -3306,7 +3054,7 @@ }, { "cell_type": "markdown", - "id": "4ad52617", + "id": "b5c113ba", "metadata": { "editable": true }, @@ -3319,7 +3067,7 @@ }, { "cell_type": "markdown", - "id": "41cabc72", + "id": "aad866d0", "metadata": { "editable": true }, @@ -3331,7 +3079,7 @@ }, { "cell_type": "markdown", - "id": "1fd416a6", + "id": "a4e7459d", "metadata": { "editable": true }, @@ -3345,7 +3093,7 @@ { "cell_type": "code", "execution_count": 19, - "id": "f13cacb4", + "id": "2af50ed1", "metadata": { "collapsed": false, "editable": true @@ -3442,7 +3190,7 @@ }, { "cell_type": "markdown", - "id": "ff4b28ed", + "id": "e468f434", "metadata": { "editable": true }, @@ -3463,7 +3211,7 @@ }, { "cell_type": "markdown", - "id": "03b4dc3c", + "id": "0daaeb51", "metadata": { "editable": true }, @@ -3475,7 +3223,7 @@ }, { "cell_type": "markdown", - "id": "6cff2148", + "id": "29f1ae2a", "metadata": { "editable": true }, @@ -3485,7 +3233,7 @@ }, { "cell_type": "markdown", - "id": "51d823c4", + "id": "133f65de", "metadata": { "editable": true }, @@ -3497,7 +3245,7 @@ }, { "cell_type": "markdown", - "id": "5ab270f9", + "id": "b5f4350c", "metadata": { "editable": true }, @@ -3507,7 +3255,7 @@ }, { "cell_type": "markdown", - "id": "2b2e1492", + "id": "1f6fc172", "metadata": { "editable": true }, @@ -3519,7 +3267,7 @@ }, { "cell_type": "markdown", - "id": "063be947", + "id": "c5afa388", "metadata": { "editable": true }, @@ -3537,7 +3285,7 @@ { "cell_type": "code", "execution_count": 20, - "id": "5e841a10", + "id": "5718b4ec", "metadata": { "collapsed": false, "editable": true @@ -3613,7 +3361,7 @@ }, { "cell_type": "markdown", - "id": "40e54083", + "id": "5e7c8319", "metadata": { "editable": true }, @@ -3627,7 +3375,7 @@ { "cell_type": "code", "execution_count": 21, - "id": "edd7dd42", + "id": "3b7f797a", "metadata": { "collapsed": false, "editable": true @@ -3716,7 +3464,7 @@ }, { "cell_type": "markdown", - "id": "f9cc046b", + "id": "5e9da20b", "metadata": { "editable": true },