diff --git a/doc/pub/week38/ipynb/week38.ipynb b/doc/pub/week38/ipynb/week38.ipynb
index 2edb4f733..d30ad7a21 100644
--- a/doc/pub/week38/ipynb/week38.ipynb
+++ b/doc/pub/week38/ipynb/week38.ipynb
@@ -3,9 +3,7 @@
{
"cell_type": "markdown",
"id": "a4e5b8f2",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"\n",
@@ -15,9 +13,7 @@
{
"cell_type": "markdown",
"id": "93b0d36c",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"# Week 38: Logistic Regression and Optimization\n",
"**Morten Hjorth-Jensen**, Department of Physics and Center for Computing in Science Education, University of Oslo and Department of Physics and Astronomy and Facility for Rare Isotope Beams, Michigan State University\n",
@@ -28,9 +24,7 @@
{
"cell_type": "markdown",
"id": "42899962",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Plans for week 38, lecture Monday September 16\n",
"\n",
@@ -46,9 +40,7 @@
{
"cell_type": "markdown",
"id": "4175fb49",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Suggested reading and videos\n",
" * Readings and Videos:\n",
@@ -69,9 +61,7 @@
{
"cell_type": "markdown",
"id": "cd73082e",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Plans for the lab sessions\n",
"\n",
@@ -89,9 +79,7 @@
{
"cell_type": "markdown",
"id": "84c47adf",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Material for lecture Thursday September 21"
]
@@ -99,9 +87,7 @@
{
"cell_type": "markdown",
"id": "1299fcbc",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Logistic Regression\n",
"\n",
@@ -121,9 +107,7 @@
{
"cell_type": "markdown",
"id": "e70ead5f",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Classification problems\n",
"\n",
@@ -147,9 +131,7 @@
{
"cell_type": "markdown",
"id": "fa1b67be",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Optimization and Deep learning\n",
"\n",
@@ -171,9 +153,7 @@
{
"cell_type": "markdown",
"id": "7c54a4ff",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Basics\n",
"\n",
@@ -196,9 +176,7 @@
{
"cell_type": "markdown",
"id": "838d316e",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"y_i = \\begin{bmatrix} 0 & \\mathrm{no}\\\\ 1 & \\mathrm{yes} \\end{bmatrix}.\n",
@@ -208,9 +186,7 @@
{
"cell_type": "markdown",
"id": "285bf3a2",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Linear classifier\n",
"\n",
@@ -226,9 +202,7 @@
{
"cell_type": "markdown",
"id": "6b03bb50",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"\n",
"\n",
@@ -244,9 +218,7 @@
{
"cell_type": "markdown",
"id": "f153c89c",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"where $\\boldsymbol{y}$ is a vector representing the possible outcomes, $\\boldsymbol{X}$ is our\n",
"$n\\times p$ design matrix and $\\boldsymbol{\\beta}$ represents our estimators/predictors."
@@ -255,9 +227,7 @@
{
"cell_type": "markdown",
"id": "487ff8fe",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Some selected properties\n",
"\n",
@@ -282,9 +252,7 @@
{
"cell_type": "markdown",
"id": "e64f34ad",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Simple example\n",
"\n",
@@ -295,10 +263,7 @@
"cell_type": "code",
"execution_count": 1,
"id": "eb389ab6",
- "metadata": {
- "collapsed": false,
- "editable": true
- },
+ "metadata": {},
"outputs": [],
"source": [
"%matplotlib inline\n",
@@ -362,9 +327,7 @@
{
"cell_type": "markdown",
"id": "d0f9afe8",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Plotting the mean value for each group\n",
"\n",
@@ -375,10 +338,7 @@
"cell_type": "code",
"execution_count": 2,
"id": "499179b1",
- "metadata": {
- "collapsed": false,
- "editable": true
- },
+ "metadata": {},
"outputs": [],
"source": [
"agegroupmean = np.array([0.1, 0.133, 0.250, 0.333, 0.462, 0.625, 0.765, 0.800])\n",
@@ -394,9 +354,7 @@
{
"cell_type": "markdown",
"id": "cf814d3f",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"We are now trying to find a function $f(y\\vert x)$, that is a function which gives us an expected value for the output $y$ with a given input $x$.\n",
"In standard linear regression with a linear dependence on $x$, we would write this in terms of our model"
@@ -405,9 +363,7 @@
{
"cell_type": "markdown",
"id": "28658b61",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"f(y_i\\vert x_i)=\\beta_0+\\beta_1 x_i.\n",
@@ -417,9 +373,7 @@
{
"cell_type": "markdown",
"id": "dec92579",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"This expression implies however that $f(y_i\\vert x_i)$ could take any\n",
"value from minus infinity to plus infinity. If we however let\n",
@@ -436,9 +390,7 @@
{
"cell_type": "markdown",
"id": "92ad450b",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## The logistic function\n",
"\n",
@@ -458,9 +410,7 @@
{
"cell_type": "markdown",
"id": "35cf0753",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"p(t) = \\frac{1}{1+\\mathrm \\exp{-t}}=\\frac{\\exp{t}}{1+\\mathrm \\exp{t}}.\n",
@@ -470,9 +420,7 @@
{
"cell_type": "markdown",
"id": "fd212f79",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"Note that $1-p(t)= p(-t)$."
]
@@ -480,9 +428,7 @@
{
"cell_type": "markdown",
"id": "34d6199f",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Examples of likelihood functions used in logistic regression and nueral networks\n",
"\n",
@@ -493,10 +439,7 @@
"cell_type": "code",
"execution_count": 3,
"id": "174df351",
- "metadata": {
- "collapsed": false,
- "editable": true
- },
+ "metadata": {},
"outputs": [],
"source": [
"\"\"\"The sigmoid function (or the logistic curve) is a\n",
@@ -558,9 +501,7 @@
{
"cell_type": "markdown",
"id": "89fdf010",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Two parameters\n",
"\n",
@@ -570,9 +511,7 @@
{
"cell_type": "markdown",
"id": "6216d0ba",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"\\begin{align*}\n",
@@ -585,9 +524,7 @@
{
"cell_type": "markdown",
"id": "daa87f9a",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"where $\\boldsymbol{\\beta}$ are the weights we wish to extract from data, in our case $\\beta_0$ and $\\beta_1$. \n",
"\n",
@@ -597,9 +534,7 @@
{
"cell_type": "markdown",
"id": "85088741",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"p(y_i=0\\vert x_i, \\boldsymbol{\\beta}) = 1-p(y_i=1\\vert x_i, \\boldsymbol{\\beta}).\n",
@@ -609,9 +544,7 @@
{
"cell_type": "markdown",
"id": "035508f1",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Maximum likelihood\n",
"\n",
@@ -626,9 +559,7 @@
{
"cell_type": "markdown",
"id": "8972c037",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"\\begin{align*}\n",
@@ -640,9 +571,7 @@
{
"cell_type": "markdown",
"id": "3eac3b65",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"from which we obtain the log-likelihood and our **cost/loss** function"
]
@@ -650,9 +579,7 @@
{
"cell_type": "markdown",
"id": "133efea3",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"\\mathcal{C}(\\boldsymbol{\\beta}) = \\sum_{i=1}^n \\left( y_i\\log{p(y_i=1|x_i,\\boldsymbol{\\beta})} + (1-y_i)\\log\\left[1-p(y_i=1|x_i,\\boldsymbol{\\beta}))\\right]\\right).\n",
@@ -662,9 +589,7 @@
{
"cell_type": "markdown",
"id": "566aabdb",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## The cost function rewritten\n",
"\n",
@@ -674,9 +599,7 @@
{
"cell_type": "markdown",
"id": "4cf34420",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"\\mathcal{C}(\\boldsymbol{\\beta}) = \\sum_{i=1}^n \\left(y_i(\\beta_0+\\beta_1x_i) -\\log{(1+\\exp{(\\beta_0+\\beta_1x_i)})}\\right).\n",
@@ -686,9 +609,7 @@
{
"cell_type": "markdown",
"id": "90ccf024",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"The maximum likelihood estimator is defined as the set of parameters that maximize the log-likelihood where we maximize with respect to $\\beta$.\n",
"Since the cost (error) function is just the negative log-likelihood, for logistic regression we have that"
@@ -697,9 +618,7 @@
{
"cell_type": "markdown",
"id": "824ccda5",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"\\mathcal{C}(\\boldsymbol{\\beta})=-\\sum_{i=1}^n \\left(y_i(\\beta_0+\\beta_1x_i) -\\log{(1+\\exp{(\\beta_0+\\beta_1x_i)})}\\right).\n",
@@ -709,9 +628,7 @@
{
"cell_type": "markdown",
"id": "bbfb14d3",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"This equation is known in statistics as the **cross entropy**. Finally, we note that just as in linear regression, \n",
"in practice we often supplement the cross-entropy with additional regularization terms, usually $L_1$ and $L_2$ regularization as we did for Ridge and Lasso regression."
@@ -720,9 +637,7 @@
{
"cell_type": "markdown",
"id": "bfc7e1be",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Minimizing the cross entropy\n",
"\n",
@@ -736,9 +651,7 @@
{
"cell_type": "markdown",
"id": "bae6c599",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"\\frac{\\partial \\mathcal{C}(\\boldsymbol{\\beta})}{\\partial \\beta_0} = -\\sum_{i=1}^n \\left(y_i -\\frac{\\exp{(\\beta_0+\\beta_1x_i)}}{1+\\exp{(\\beta_0+\\beta_1x_i)}}\\right),\n",
@@ -748,9 +661,7 @@
{
"cell_type": "markdown",
"id": "550a7b2b",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"and"
]
@@ -758,9 +669,7 @@
{
"cell_type": "markdown",
"id": "5b072ab8",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"\\frac{\\partial \\mathcal{C}(\\boldsymbol{\\beta})}{\\partial \\beta_1} = -\\sum_{i=1}^n \\left(y_ix_i -x_i\\frac{\\exp{(\\beta_0+\\beta_1x_i)}}{1+\\exp{(\\beta_0+\\beta_1x_i)}}\\right).\n",
@@ -770,9 +679,7 @@
{
"cell_type": "markdown",
"id": "9ed08b1d",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## A more compact expression\n",
"\n",
@@ -785,9 +692,7 @@
{
"cell_type": "markdown",
"id": "11b32bd3",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"\\frac{\\partial \\mathcal{C}(\\boldsymbol{\\beta})}{\\partial \\boldsymbol{\\beta}} = -\\boldsymbol{X}^T\\left(\\boldsymbol{y}-\\boldsymbol{p}\\right).\n",
@@ -797,9 +702,7 @@
{
"cell_type": "markdown",
"id": "47dec6ef",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"If we in addition define a diagonal matrix $\\boldsymbol{W}$ with elements \n",
"$p(y_i\\vert x_i,\\boldsymbol{\\beta})(1-p(y_i\\vert x_i,\\boldsymbol{\\beta})$, we can obtain a compact expression of the second derivative as"
@@ -808,9 +711,7 @@
{
"cell_type": "markdown",
"id": "43e12319",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"\\frac{\\partial^2 \\mathcal{C}(\\boldsymbol{\\beta})}{\\partial \\boldsymbol{\\beta}\\partial \\boldsymbol{\\beta}^T} = \\boldsymbol{X}^T\\boldsymbol{W}\\boldsymbol{X}.\n",
@@ -820,9 +721,7 @@
{
"cell_type": "markdown",
"id": "a0c3198a",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Extending to more predictors\n",
"\n",
@@ -832,9 +731,7 @@
{
"cell_type": "markdown",
"id": "fb7b011d",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"\\log{ \\frac{p(\\boldsymbol{\\beta}\\boldsymbol{x})}{1-p(\\boldsymbol{\\beta}\\boldsymbol{x})}} = \\beta_0+\\beta_1x_1+\\beta_2x_2+\\dots+\\beta_px_p.\n",
@@ -844,9 +741,7 @@
{
"cell_type": "markdown",
"id": "acfaf754",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"Here we defined $\\boldsymbol{x}=[1,x_1,x_2,\\dots,x_p]$ and $\\boldsymbol{\\beta}=[\\beta_0, \\beta_1, \\dots, \\beta_p]$ leading to"
]
@@ -854,9 +749,7 @@
{
"cell_type": "markdown",
"id": "33e112ba",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"p(\\boldsymbol{\\beta}\\boldsymbol{x})=\\frac{ \\exp{(\\beta_0+\\beta_1x_1+\\beta_2x_2+\\dots+\\beta_px_p)}}{1+\\exp{(\\beta_0+\\beta_1x_1+\\beta_2x_2+\\dots+\\beta_px_p)}}.\n",
@@ -866,9 +759,7 @@
{
"cell_type": "markdown",
"id": "75173b91",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Including more classes\n",
"\n",
@@ -880,9 +771,7 @@
{
"cell_type": "markdown",
"id": "60ea4955",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"\\log{\\frac{p(C=1\\vert x)}{p(K\\vert x)}} = \\beta_{10}+\\beta_{11}x_1,\n",
@@ -892,9 +781,7 @@
{
"cell_type": "markdown",
"id": "2ac0c3fe",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"and"
]
@@ -902,9 +789,7 @@
{
"cell_type": "markdown",
"id": "c5633fe2",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"\\log{\\frac{p(C=2\\vert x)}{p(K\\vert x)}} = \\beta_{20}+\\beta_{21}x_1,\n",
@@ -914,9 +799,7 @@
{
"cell_type": "markdown",
"id": "af63c7fc",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"and so on till the class $C=K-1$ class"
]
@@ -924,9 +807,7 @@
{
"cell_type": "markdown",
"id": "68713442",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"\\log{\\frac{p(C=K-1\\vert x)}{p(K\\vert x)}} = \\beta_{(K-1)0}+\\beta_{(K-1)1}x_1,\n",
@@ -936,9 +817,7 @@
{
"cell_type": "markdown",
"id": "ae89a4ec",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"and the model is specified in term of $K-1$ so-called log-odds or\n",
"**logit** transformations."
@@ -947,9 +826,7 @@
{
"cell_type": "markdown",
"id": "dc86b762",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## More classes\n",
"\n",
@@ -970,9 +847,7 @@
{
"cell_type": "markdown",
"id": "4109330d",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"p(C=k\\vert \\mathbf {x} )=\\frac{\\exp{(\\beta_{k0}+\\beta_{k1}x_1)}}{1+\\sum_{l=1}^{K-1}\\exp{(\\beta_{l0}+\\beta_{l1}x_1)}}.\n",
@@ -982,9 +857,7 @@
{
"cell_type": "markdown",
"id": "dcf3f4c4",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"It is easy to extend to more predictors. The final class is"
]
@@ -992,9 +865,7 @@
{
"cell_type": "markdown",
"id": "148d45e3",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"p(C=K\\vert \\mathbf {x} )=\\frac{1}{1+\\sum_{l=1}^{K-1}\\exp{(\\beta_{l0}+\\beta_{l1}x_1)}},\n",
@@ -1004,9 +875,7 @@
{
"cell_type": "markdown",
"id": "5c082186",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"and they sum to one. Our earlier discussions were all specialized to\n",
"the case with two classes only. It is easy to see from the above that\n",
@@ -1021,9 +890,7 @@
{
"cell_type": "markdown",
"id": "d5068fec",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Friday September 23"
]
@@ -1031,9 +898,7 @@
{
"cell_type": "markdown",
"id": "d33abe73",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Searching for Optimal Regularization Parameters $\\lambda$\n",
"\n",
@@ -1050,10 +915,7 @@
"cell_type": "code",
"execution_count": 4,
"id": "d445125f",
- "metadata": {
- "collapsed": false,
- "editable": true
- },
+ "metadata": {},
"outputs": [],
"source": [
"import numpy as np\n",
@@ -1105,9 +967,7 @@
{
"cell_type": "markdown",
"id": "3ad4a826",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"Here we have performed a rather data greedy calculation as function of the regularization parameter $\\lambda$. There is no resampling here. The latter can easily be added by employing the function **RidgeCV** instead of just calling the **Ridge** function. For **RidgeCV** we need to pass the array of $\\lambda$ values.\n",
"By inspecting the figure we can in turn determine which is the optimal regularization parameter.\n",
@@ -1117,9 +977,7 @@
{
"cell_type": "markdown",
"id": "ff84b2e7",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Grid Search\n",
"\n",
@@ -1132,10 +990,7 @@
"cell_type": "code",
"execution_count": 5,
"id": "4b834234",
- "metadata": {
- "collapsed": false,
- "editable": true
- },
+ "metadata": {},
"outputs": [],
"source": [
"import numpy as np\n",
@@ -1185,9 +1040,7 @@
{
"cell_type": "markdown",
"id": "7514764b",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"By default the grid search function includes cross validation with\n",
"five folds. The [Scikit-Learn\n",
@@ -1200,9 +1053,7 @@
{
"cell_type": "markdown",
"id": "7991ea9f",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Randomized Grid Search\n",
"\n",
@@ -1220,10 +1071,7 @@
"cell_type": "code",
"execution_count": 6,
"id": "ae4f99a1",
- "metadata": {
- "collapsed": false,
- "editable": true
- },
+ "metadata": {},
"outputs": [],
"source": [
"import numpy as np\n",
@@ -1274,9 +1122,7 @@
{
"cell_type": "markdown",
"id": "91278700",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Wisconsin Cancer Data\n",
"\n",
@@ -1289,10 +1135,7 @@
"cell_type": "code",
"execution_count": 7,
"id": "8f006bb8",
- "metadata": {
- "collapsed": false,
- "editable": true
- },
+ "metadata": {},
"outputs": [],
"source": [
"import matplotlib.pyplot as plt\n",
@@ -1316,9 +1159,7 @@
{
"cell_type": "markdown",
"id": "015f8534",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Using the correlation matrix\n",
"\n",
@@ -1330,10 +1171,7 @@
"cell_type": "code",
"execution_count": 8,
"id": "fa6d3421",
- "metadata": {
- "collapsed": false,
- "editable": true
- },
+ "metadata": {},
"outputs": [],
"source": [
"import matplotlib.pyplot as plt\n",
@@ -1375,9 +1213,7 @@
{
"cell_type": "markdown",
"id": "49ff3f87",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Discussing the correlation data\n",
"\n",
@@ -1400,10 +1236,7 @@
"cell_type": "code",
"execution_count": 9,
"id": "3e800820",
- "metadata": {
- "collapsed": false,
- "editable": true
- },
+ "metadata": {},
"outputs": [],
"source": [
"cancerpd = pd.DataFrame(cancer.data, columns=cancer.feature_names)"
@@ -1412,9 +1245,7 @@
{
"cell_type": "markdown",
"id": "9f831f22",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"and then"
]
@@ -1423,10 +1254,7 @@
"cell_type": "code",
"execution_count": 10,
"id": "e663ae38",
- "metadata": {
- "collapsed": false,
- "editable": true
- },
+ "metadata": {},
"outputs": [],
"source": [
"correlation_matrix = cancerpd.corr().round(1)"
@@ -1435,9 +1263,7 @@
{
"cell_type": "markdown",
"id": "737193a2",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"Diagonalizing this matrix we can in turn say something about which\n",
"features are of relevance and which are not. This leads us to\n",
@@ -1448,9 +1274,7 @@
{
"cell_type": "markdown",
"id": "25104113",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Other measures in classification studies: Cancer Data again"
]
@@ -1459,10 +1283,7 @@
"cell_type": "code",
"execution_count": 11,
"id": "6783b76e",
- "metadata": {
- "collapsed": false,
- "editable": true
- },
+ "metadata": {},
"outputs": [],
"source": [
"import matplotlib.pyplot as plt\n",
@@ -1502,9 +1323,7 @@
{
"cell_type": "markdown",
"id": "bdb4928a",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Optimization, the central part of any Machine Learning algortithm\n",
"\n",
@@ -1523,9 +1342,7 @@
{
"cell_type": "markdown",
"id": "8b73ff85",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Revisiting our Logistic Regression case\n",
"\n",
@@ -1540,9 +1357,7 @@
{
"cell_type": "markdown",
"id": "2b771369",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"\\begin{align*}\n",
@@ -1555,9 +1370,7 @@
{
"cell_type": "markdown",
"id": "10c01ae1",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"where $\\boldsymbol{\\beta}$ are the weights we wish to extract from data, in our case $\\beta_0$ and $\\beta_1$."
]
@@ -1565,9 +1378,7 @@
{
"cell_type": "markdown",
"id": "b205858e",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## The equations to solve\n",
"\n",
@@ -1581,9 +1392,7 @@
{
"cell_type": "markdown",
"id": "5f48f306",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"\\frac{\\partial \\mathcal{C}(\\boldsymbol{\\beta})}{\\partial \\boldsymbol{\\beta}} = -\\boldsymbol{X}^T\\left(\\boldsymbol{y}-\\boldsymbol{p}\\right).\n",
@@ -1593,9 +1402,7 @@
{
"cell_type": "markdown",
"id": "deff0dec",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"If we in addition define a diagonal matrix $\\boldsymbol{W}$ with elements \n",
"$p(y_i\\vert x_i,\\boldsymbol{\\beta})(1-p(y_i\\vert x_i,\\boldsymbol{\\beta})$, we can obtain a compact expression of the second derivative as"
@@ -1604,9 +1411,7 @@
{
"cell_type": "markdown",
"id": "4a3eee34",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"\\frac{\\partial^2 \\mathcal{C}(\\boldsymbol{\\beta})}{\\partial \\boldsymbol{\\beta}\\partial \\boldsymbol{\\beta}^T} = \\boldsymbol{X}^T\\boldsymbol{W}\\boldsymbol{X}.\n",
@@ -1616,9 +1421,7 @@
{
"cell_type": "markdown",
"id": "ac2953d0",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"This defines what is called the Hessian matrix."
]
@@ -1626,9 +1429,7 @@
{
"cell_type": "markdown",
"id": "fcacb96c",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Solving using Newton-Raphson's method\n",
"\n",
@@ -1640,9 +1441,7 @@
{
"cell_type": "markdown",
"id": "d8946939",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"\\boldsymbol{\\beta}^{\\mathrm{new}} = \\boldsymbol{\\beta}^{\\mathrm{old}}-\\left(\\frac{\\partial^2 \\mathcal{C}(\\boldsymbol{\\beta})}{\\partial \\boldsymbol{\\beta}\\partial \\boldsymbol{\\beta}^T}\\right)^{-1}_{\\boldsymbol{\\beta}^{\\mathrm{old}}}\\times \\left(\\frac{\\partial \\mathcal{C}(\\boldsymbol{\\beta})}{\\partial \\boldsymbol{\\beta}}\\right)_{\\boldsymbol{\\beta}^{\\mathrm{old}}},\n",
@@ -1652,9 +1451,7 @@
{
"cell_type": "markdown",
"id": "1cb3f7cd",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"or in matrix form as"
]
@@ -1662,9 +1459,7 @@
{
"cell_type": "markdown",
"id": "5707329b",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"\\boldsymbol{\\beta}^{\\mathrm{new}} = \\boldsymbol{\\beta}^{\\mathrm{old}}-\\left(\\boldsymbol{X}^T\\boldsymbol{W}\\boldsymbol{X} \\right)^{-1}\\times \\left(-\\boldsymbol{X}^T(\\boldsymbol{y}-\\boldsymbol{p}) \\right)_{\\boldsymbol{\\beta}^{\\mathrm{old}}}.\n",
@@ -1674,9 +1469,7 @@
{
"cell_type": "markdown",
"id": "996511e9",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"The right-hand side is computed with the old values of $\\beta$. \n",
"\n",
@@ -1686,9 +1479,7 @@
{
"cell_type": "markdown",
"id": "a35fcbf1",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Brief reminder on Newton-Raphson's method\n",
"\n",
@@ -1706,9 +1497,7 @@
{
"cell_type": "markdown",
"id": "080ea026",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## The equations\n",
"\n",
@@ -1722,9 +1511,7 @@
{
"cell_type": "markdown",
"id": "49636667",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"\n",
"\n",
@@ -1738,9 +1525,7 @@
{
"cell_type": "markdown",
"id": "3bd16905",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"For small enough values of the function and for well-behaved\n",
"functions, the terms beyond linear are unimportant, hence we obtain"
@@ -1749,9 +1534,7 @@
{
"cell_type": "markdown",
"id": "68cc347e",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"f(x)+(s-x)f'(x)\\approx 0,\n",
@@ -1761,9 +1544,7 @@
{
"cell_type": "markdown",
"id": "3ba3e954",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"yielding"
]
@@ -1771,9 +1552,7 @@
{
"cell_type": "markdown",
"id": "18a71133",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"s\\approx x-\\frac{f(x)}{f'(x)}.\n",
@@ -1783,9 +1562,7 @@
{
"cell_type": "markdown",
"id": "cebe1598",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"Having in mind an iterative procedure, it is natural to start iterating with"
]
@@ -1793,9 +1570,7 @@
{
"cell_type": "markdown",
"id": "262fc2e7",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"x_{n+1}=x_n-\\frac{f(x_n)}{f'(x_n)}.\n",
@@ -1805,9 +1580,7 @@
{
"cell_type": "markdown",
"id": "d8ac2615",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Simple geometric interpretation\n",
"\n",
@@ -1827,9 +1600,7 @@
{
"cell_type": "markdown",
"id": "1637b673",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Extending to more than one variable\n",
"\n",
@@ -1840,9 +1611,7 @@
{
"cell_type": "markdown",
"id": "efb3a153",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"\\begin{array}{cc} f_1(x_1,x_2) &=0\\\\\n",
@@ -1853,9 +1622,7 @@
{
"cell_type": "markdown",
"id": "cd1d0b0b",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"which we Taylor expand to obtain"
]
@@ -1863,9 +1630,7 @@
{
"cell_type": "markdown",
"id": "1409701b",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"\\begin{array}{cc} 0=f_1(x_1+h_1,x_2+h_2)=&f_1(x_1,x_2)+h_1\n",
@@ -1881,9 +1646,7 @@
{
"cell_type": "markdown",
"id": "969cdccb",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"Defining the Jacobian matrix ${\\bf \\boldsymbol{J}}$ we have"
]
@@ -1891,9 +1654,7 @@
{
"cell_type": "markdown",
"id": "1d26f373",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"{\\bf \\boldsymbol{J}}=\\left( \\begin{array}{cc}\n",
@@ -1906,9 +1667,7 @@
{
"cell_type": "markdown",
"id": "b6fa2c0e",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"we can rephrase Newton's method as"
]
@@ -1916,9 +1675,7 @@
{
"cell_type": "markdown",
"id": "c26cca81",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"\\left(\\begin{array}{c} x_1^{n+1} \\\\ x_2^{n+1} \\end{array} \\right)=\n",
@@ -1930,9 +1687,7 @@
{
"cell_type": "markdown",
"id": "e6813f1b",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"where we have defined"
]
@@ -1940,9 +1695,7 @@
{
"cell_type": "markdown",
"id": "f53cce6d",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"\\left(\\begin{array}{c} h_1^{n} \\\\ h_2^{n} \\end{array} \\right)=\n",
@@ -1954,9 +1707,7 @@
{
"cell_type": "markdown",
"id": "47a097ab",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"We need thus to compute the inverse of the Jacobian matrix and it\n",
"is to understand that difficulties may\n",
@@ -1969,9 +1720,7 @@
{
"cell_type": "markdown",
"id": "682046e4",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Steepest descent\n",
"\n",
@@ -1986,9 +1735,7 @@
{
"cell_type": "markdown",
"id": "91f0fc21",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"\\mathbf{x}_{k+1} = \\mathbf{x}_k - \\gamma_k \\nabla F(\\mathbf{x}_k),\n",
@@ -1998,9 +1745,7 @@
{
"cell_type": "markdown",
"id": "7e415ba2",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"with $\\gamma_k > 0$.\n",
"\n",
@@ -2012,9 +1757,7 @@
{
"cell_type": "markdown",
"id": "d67a88be",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## More on Steepest descent\n",
"\n",
@@ -2027,9 +1770,7 @@
{
"cell_type": "markdown",
"id": "ee068302",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"\\mathbf{x}_{k+1} = \\mathbf{x}_k - \\gamma_k \\nabla F(\\mathbf{x}_k), \\ \\ k \\geq 0.\n",
@@ -2039,9 +1780,7 @@
{
"cell_type": "markdown",
"id": "d63dc655",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"The parameter $\\gamma_k$ is often referred to as the step length or\n",
"the learning rate within the context of Machine Learning."
@@ -2050,9 +1789,7 @@
{
"cell_type": "markdown",
"id": "fc1fc57d",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## The ideal\n",
"\n",
@@ -2078,9 +1815,7 @@
{
"cell_type": "markdown",
"id": "b97eed70",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## The sensitiveness of the gradient descent\n",
"\n",
@@ -2100,9 +1835,7 @@
{
"cell_type": "markdown",
"id": "c35084e9",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Convex functions\n",
"\n",
@@ -2122,9 +1855,7 @@
{
"cell_type": "markdown",
"id": "a11b1893",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Convex function\n",
"\n",
@@ -2134,9 +1865,7 @@
{
"cell_type": "markdown",
"id": "f54f8d27",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Conditions on convex functions\n",
"\n",
@@ -2171,9 +1900,7 @@
{
"cell_type": "markdown",
"id": "a8d7420c",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## More on convex functions\n",
"\n",
@@ -2199,9 +1926,7 @@
{
"cell_type": "markdown",
"id": "877d661b",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Some simple problems\n",
"\n",
@@ -2229,9 +1954,7 @@
{
"cell_type": "markdown",
"id": "74823208",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Revisiting our first homework\n",
"\n",
@@ -2253,10 +1976,7 @@
"cell_type": "code",
"execution_count": 12,
"id": "2bafe07c",
- "metadata": {
- "collapsed": false,
- "editable": true
- },
+ "metadata": {},
"outputs": [],
"source": [
"x = 2*np.random.rand(m,1)\n",
@@ -2266,9 +1986,7 @@
{
"cell_type": "markdown",
"id": "a18bdeb4",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"with $x_i \\in [0,1] $ is chosen randomly using a uniform distribution. Additionally we have a stochastic noise chosen according to a normal distribution $\\cal {N}(0,1)$. \n",
"The linear regression model is given by"
@@ -2277,9 +1995,7 @@
{
"cell_type": "markdown",
"id": "16415b8c",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"h_\\beta(x) = \\boldsymbol{y} = \\beta_0 + \\beta_1 x,\n",
@@ -2289,9 +2005,7 @@
{
"cell_type": "markdown",
"id": "dc273c04",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"such that"
]
@@ -2299,9 +2013,7 @@
{
"cell_type": "markdown",
"id": "4332c289",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"\\boldsymbol{y}_i = \\beta_0 + \\beta_1 x_i.\n",
@@ -2311,9 +2023,7 @@
{
"cell_type": "markdown",
"id": "e96ba3ef",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Gradient descent example\n",
"\n",
@@ -2325,9 +2035,7 @@
{
"cell_type": "markdown",
"id": "deb14fe7",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"X \\equiv \\begin{bmatrix}\n",
@@ -2341,9 +2049,7 @@
{
"cell_type": "markdown",
"id": "e468ef7d",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"The cost/loss/risk function is given by ("
]
@@ -2351,9 +2057,7 @@
{
"cell_type": "markdown",
"id": "97154cec",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"C(\\beta) = \\frac{1}{n}||X\\beta-\\mathbf{y}||_{2}^{2} = \\frac{1}{n}\\sum_{i=1}^{100}\\left[ (\\beta_0 + \\beta_1 x_i)^2 - 2 y_i (\\beta_0 + \\beta_1 x_i) + y_i^2\\right]\n",
@@ -2363,9 +2067,7 @@
{
"cell_type": "markdown",
"id": "6e30b09a",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"and we want to find $\\beta$ such that $C(\\beta)$ is minimized."
]
@@ -2373,9 +2075,7 @@
{
"cell_type": "markdown",
"id": "d5dc7377",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## The derivative of the cost/loss function\n",
"\n",
@@ -2385,9 +2085,7 @@
{
"cell_type": "markdown",
"id": "49d2621f",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"\\nabla_{\\beta} C(\\beta) = \\frac{2}{n}\\begin{bmatrix} \\sum_{i=1}^{100} \\left(\\beta_0+\\beta_1x_i-y_i\\right) \\\\\n",
@@ -2399,9 +2097,7 @@
{
"cell_type": "markdown",
"id": "f2950e6b",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"where $X$ is the design matrix defined above."
]
@@ -2409,9 +2105,7 @@
{
"cell_type": "markdown",
"id": "00be7184",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## The Hessian matrix\n",
"The Hessian matrix of $C(\\beta)$ is given by"
@@ -2420,9 +2114,7 @@
{
"cell_type": "markdown",
"id": "97b443f2",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"\\boldsymbol{H} \\equiv \\begin{bmatrix}\n",
@@ -2435,9 +2127,7 @@
{
"cell_type": "markdown",
"id": "0676ebaf",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"This result implies that $C(\\beta)$ is a convex function since the matrix $X^T X$ always is positive semi-definite."
]
@@ -2445,9 +2135,7 @@
{
"cell_type": "markdown",
"id": "4ad98c2d",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Simple program\n",
"\n",
@@ -2457,9 +2145,7 @@
{
"cell_type": "markdown",
"id": "7405f7a9",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"\\beta_{k+1} = \\beta_k - \\gamma \\nabla_\\beta C(\\beta_k), \\ k=0,1,\\cdots\n",
@@ -2469,9 +2155,7 @@
{
"cell_type": "markdown",
"id": "ca70147e",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"We can use the expression we computed for the gradient and let use a\n",
"$\\beta_0$ be chosen randomly and let $\\gamma = 0.001$. Stop iterating\n",
@@ -2484,9 +2168,7 @@
{
"cell_type": "markdown",
"id": "d07fdcda",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Gradient Descent Example\n",
"\n",
@@ -2497,10 +2179,7 @@
"cell_type": "code",
"execution_count": 13,
"id": "2483d42a",
- "metadata": {
- "collapsed": false,
- "editable": true
- },
+ "metadata": {},
"outputs": [],
"source": [
"\n",
@@ -2554,9 +2233,7 @@
{
"cell_type": "markdown",
"id": "6071973b",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## And a corresponding example using **scikit-learn**"
]
@@ -2565,10 +2242,7 @@
"cell_type": "code",
"execution_count": 14,
"id": "f31a67da",
- "metadata": {
- "collapsed": false,
- "editable": true
- },
+ "metadata": {},
"outputs": [],
"source": [
"# Importing various packages\n",
@@ -2592,9 +2266,7 @@
{
"cell_type": "markdown",
"id": "c6ae27ed",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Gradient descent and Ridge\n",
"\n",
@@ -2604,9 +2276,7 @@
{
"cell_type": "markdown",
"id": "55efcb99",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"C_{\\text{ridge}}(\\beta) = \\frac{1}{n}||X\\beta -\\mathbf{y}||^2 + \\lambda ||\\beta||^2, \\ \\lambda \\geq 0.\n",
@@ -2616,9 +2286,7 @@
{
"cell_type": "markdown",
"id": "aedf6cb2",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"In order to minimize $C_{\\text{ridge}}(\\beta)$ using GD we adjust the gradient as follows"
]
@@ -2626,9 +2294,7 @@
{
"cell_type": "markdown",
"id": "7bb7c811",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"\\nabla_\\beta C_{\\text{ridge}}(\\beta) = \\frac{2}{n}\\begin{bmatrix} \\sum_{i=1}^{100} \\left(\\beta_0+\\beta_1x_i-y_i\\right) \\\\\n",
@@ -2640,9 +2306,7 @@
{
"cell_type": "markdown",
"id": "42227947",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"We can easily extend our program to minimize $C_{\\text{ridge}}(\\beta)$ using gradient descent and compare with the analytical solution given by"
]
@@ -2650,9 +2314,7 @@
{
"cell_type": "markdown",
"id": "9c164deb",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"\\beta_{\\text{ridge}} = \\left(X^T X + n\\lambda I_{2 \\times 2} \\right)^{-1} X^T \\mathbf{y}.\n",
@@ -2662,9 +2324,7 @@
{
"cell_type": "markdown",
"id": "eb38ae48",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## The Hessian matrix for Ridge Regression\n",
"The Hessian matrix of Ridge Regression for our simple example is given by"
@@ -2673,9 +2333,7 @@
{
"cell_type": "markdown",
"id": "a6fcf56c",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"\\boldsymbol{H} \\equiv \\begin{bmatrix}\n",
@@ -2688,9 +2346,7 @@
{
"cell_type": "markdown",
"id": "3a2b5ae4",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"This implies that the Hessian matrix is positive definite, hence the stationary point is a\n",
"minimum.\n",
@@ -2702,9 +2358,7 @@
{
"cell_type": "markdown",
"id": "1a7eb7e6",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Program example for gradient descent with Ridge Regression"
]
@@ -2713,10 +2367,7 @@
"cell_type": "code",
"execution_count": 15,
"id": "ccb7a9b1",
- "metadata": {
- "collapsed": false,
- "editable": true
- },
+ "metadata": {},
"outputs": [],
"source": [
"from random import random, seed\n",
@@ -2774,9 +2425,7 @@
{
"cell_type": "markdown",
"id": "95a71108",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Using gradient descent methods, limitations\n",
"\n",
@@ -2796,9 +2445,7 @@
{
"cell_type": "markdown",
"id": "33f25532",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Challenge yourself the coming weekend\n",
"\n",
@@ -2808,9 +2455,7 @@
{
"cell_type": "markdown",
"id": "13ee68fe",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Lab session: Material from last week and relevant for the first project"
]
@@ -2818,9 +2463,7 @@
{
"cell_type": "markdown",
"id": "e3e607f1",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Various steps in cross-validation\n",
"\n",
@@ -2843,9 +2486,7 @@
{
"cell_type": "markdown",
"id": "42b264a0",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## How to set up the cross-validation for Ridge and/or Lasso\n",
"\n",
@@ -2859,9 +2500,7 @@
{
"cell_type": "markdown",
"id": "ff85b5e7",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"\\begin{align*}\n",
@@ -2875,9 +2514,7 @@
{
"cell_type": "markdown",
"id": "5d0533bb",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"* Evaluate the prediction performance of these models on the test set by $C[y_i, \\boldsymbol{X}_{i, \\ast}; \\boldsymbol{\\beta}_{-i}(\\lambda), \\boldsymbol{\\sigma}_{-i}^2(\\lambda)]$. Or, by the prediction error $|y_i - \\boldsymbol{X}_{i, \\ast} \\boldsymbol{\\beta}_{-i}(\\lambda)|$, the relative error, the error squared or the R2 score function.\n",
"\n",
@@ -2889,9 +2526,7 @@
{
"cell_type": "markdown",
"id": "3ab3921a",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Cross-validation in brief\n",
"\n",
@@ -2917,9 +2552,7 @@
{
"cell_type": "markdown",
"id": "0595f2f0",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Code Example for Cross-validation and $k$-fold Cross-validation\n",
"\n",
@@ -2930,10 +2563,7 @@
"cell_type": "code",
"execution_count": 16,
"id": "2902bb22",
- "metadata": {
- "collapsed": false,
- "editable": true
- },
+ "metadata": {},
"outputs": [],
"source": [
"import numpy as np\n",
@@ -3028,7 +2658,25 @@
]
}
],
- "metadata": {},
+ "metadata": {
+ "kernelspec": {
+ "display_name": "Python 3 (ipykernel)",
+ "language": "python",
+ "name": "python3"
+ },
+ "language_info": {
+ "codemirror_mode": {
+ "name": "ipython",
+ "version": 3
+ },
+ "file_extension": ".py",
+ "mimetype": "text/x-python",
+ "name": "python",
+ "nbconvert_exporter": "python",
+ "pygments_lexer": "ipython3",
+ "version": "3.9.18"
+ }
+ },
"nbformat": 4,
"nbformat_minor": 5
}