Many of these shortcomings can be alleviated by introducing
randomness. One such method is that of Stochastic Gradient Descent
-(SGD), see below.
+(SGD), to be discussed next week.
-
-Revisiting our Logistic Regression case
-
-In our discussion on Logistic Regression we studied the
-case of
-two classes, with \( y_i \) either
-\( 0 \) or \( 1 \). Furthermore we assumed also that we have only two
-parameters \( \beta \) in our fitting, that is we
-defined probabilities
-
-
-
-$$
-\begin{align*}
-p(y_i=1|x_i,\boldsymbol{\beta}) &= \frac{\exp{(\beta_0+\beta_1x_i)}}{1+\exp{(\beta_0+\beta_1x_i)}},\nonumber\\
-p(y_i=0|x_i,\boldsymbol{\beta}) &= 1 - p(y_i=1|x_i,\boldsymbol{\beta}),
-\end{align*}
-$$
-
-
-
where \( \boldsymbol{\beta} \) are the weights we wish to extract from data, in our case \( \beta_0 \) and \( \beta_1 \).
-
-
-
-The equations to solve
-
-Our compact equations used a definition of a vector \( \boldsymbol{y} \) with \( n \)
-elements \( y_i \), an \( n\times p \) matrix \( \boldsymbol{X} \) which contains the
-\( x_i \) values and a vector \( \boldsymbol{p} \) of fitted probabilities
-\( p(y_i\vert x_i,\boldsymbol{\beta}) \). We rewrote in a more compact form
-the first derivative of the cost function as
-
-
-
-$$
-\frac{\partial \mathcal{C}(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}} = -\boldsymbol{X}^T\left(\boldsymbol{y}-\boldsymbol{p}\right).
-$$
-
-
-
If we in addition define a diagonal matrix \( \boldsymbol{W} \) with elements
-\( p(y_i\vert x_i,\boldsymbol{\beta})(1-p(y_i\vert x_i,\boldsymbol{\beta}) \), we can obtain a compact expression of the second derivative as
-
-
-
-$$
-\frac{\partial^2 \mathcal{C}(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}\partial \boldsymbol{\beta}^T} = \boldsymbol{X}^T\boldsymbol{W}\boldsymbol{X}.
-$$
-
-
-
This defines what is called the Hessian matrix.
-
-
-
-Solving using Newton-Raphson's method
-
-If we can set up these equations, Newton-Raphson's iterative method is normally the method of choice. It requires however that we can compute in an efficient way the matrices that define the first and second derivatives.
-
-Our iterative scheme is then given by
-
-
-$$
-\boldsymbol{\beta}^{\mathrm{new}} = \boldsymbol{\beta}^{\mathrm{old}}-\left(\frac{\partial^2 \mathcal{C}(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}\partial \boldsymbol{\beta}^T}\right)^{-1}_{\boldsymbol{\beta}^{\mathrm{old}}}\times \left(\frac{\partial \mathcal{C}(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}}\right)_{\boldsymbol{\beta}^{\mathrm{old}}},
-$$
-
-
-
or in matrix form as
-
-
-$$
-\boldsymbol{\beta}^{\mathrm{new}} = \boldsymbol{\beta}^{\mathrm{old}}-\left(\boldsymbol{X}^T\boldsymbol{W}\boldsymbol{X} \right)^{-1}\times \left(-\boldsymbol{X}^T(\boldsymbol{y}-\boldsymbol{p}) \right)_{\boldsymbol{\beta}^{\mathrm{old}}}.
-$$
-
-
-
The right-hand side is computed with the old values of \( \beta \).
-
-If we can compute these matrices, in particular the Hessian, the above is often the easiest method to implement.
-
-
Convex functions
diff --git a/doc/pub/week38/html/week38-solarized.html b/doc/pub/week38/html/week38-solarized.html
index c72f6ae20..4f2a2b95c 100644
--- a/doc/pub/week38/html/week38-solarized.html
+++ b/doc/pub/week38/html/week38-solarized.html
@@ -179,15 +179,6 @@ div.toc p,a {
2,
None,
'the-sensitiveness-of-the-gradient-descent'),
- ('Revisiting our Logistic Regression case',
- 2,
- None,
- 'revisiting-our-logistic-regression-case'),
- ('The equations to solve', 2, None, 'the-equations-to-solve'),
- ("Solving using Newton-Raphson's method",
- 2,
- None,
- 'solving-using-newton-raphson-s-method'),
('Convex functions', 2, None, 'convex-functions'),
('Convex function', 2, None, 'convex-function'),
('Conditions on convex functions',
@@ -1821,74 +1812,9 @@ large we can experience erratic behavior.
Many of these shortcomings can be alleviated by introducing
randomness. One such method is that of Stochastic Gradient Descent
-(SGD), see below.
+(SGD), to be discussed next week.
-
-Revisiting our Logistic Regression case
-
-In our discussion on Logistic Regression we studied the
-case of
-two classes, with \( y_i \) either
-\( 0 \) or \( 1 \). Furthermore we assumed also that we have only two
-parameters \( \beta \) in our fitting, that is we
-defined probabilities
-
-
-$$
-\begin{align*}
-p(y_i=1|x_i,\boldsymbol{\beta}) &= \frac{\exp{(\beta_0+\beta_1x_i)}}{1+\exp{(\beta_0+\beta_1x_i)}},\nonumber\\
-p(y_i=0|x_i,\boldsymbol{\beta}) &= 1 - p(y_i=1|x_i,\boldsymbol{\beta}),
-\end{align*}
-$$
-
-where \( \boldsymbol{\beta} \) are the weights we wish to extract from data, in our case \( \beta_0 \) and \( \beta_1 \).
-
-
-The equations to solve
-
-Our compact equations used a definition of a vector \( \boldsymbol{y} \) with \( n \)
-elements \( y_i \), an \( n\times p \) matrix \( \boldsymbol{X} \) which contains the
-\( x_i \) values and a vector \( \boldsymbol{p} \) of fitted probabilities
-\( p(y_i\vert x_i,\boldsymbol{\beta}) \). We rewrote in a more compact form
-the first derivative of the cost function as
-
-
-$$
-\frac{\partial \mathcal{C}(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}} = -\boldsymbol{X}^T\left(\boldsymbol{y}-\boldsymbol{p}\right).
-$$
-
-If we in addition define a diagonal matrix \( \boldsymbol{W} \) with elements
-\( p(y_i\vert x_i,\boldsymbol{\beta})(1-p(y_i\vert x_i,\boldsymbol{\beta}) \), we can obtain a compact expression of the second derivative as
-
-
-$$
-\frac{\partial^2 \mathcal{C}(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}\partial \boldsymbol{\beta}^T} = \boldsymbol{X}^T\boldsymbol{W}\boldsymbol{X}.
-$$
-
-This defines what is called the Hessian matrix.
-
-
-Solving using Newton-Raphson's method
-
-If we can set up these equations, Newton-Raphson's iterative method is normally the method of choice. It requires however that we can compute in an efficient way the matrices that define the first and second derivatives.
-
-Our iterative scheme is then given by
-
-$$
-\boldsymbol{\beta}^{\mathrm{new}} = \boldsymbol{\beta}^{\mathrm{old}}-\left(\frac{\partial^2 \mathcal{C}(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}\partial \boldsymbol{\beta}^T}\right)^{-1}_{\boldsymbol{\beta}^{\mathrm{old}}}\times \left(\frac{\partial \mathcal{C}(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}}\right)_{\boldsymbol{\beta}^{\mathrm{old}}},
-$$
-
-or in matrix form as
-
-$$
-\boldsymbol{\beta}^{\mathrm{new}} = \boldsymbol{\beta}^{\mathrm{old}}-\left(\boldsymbol{X}^T\boldsymbol{W}\boldsymbol{X} \right)^{-1}\times \left(-\boldsymbol{X}^T(\boldsymbol{y}-\boldsymbol{p}) \right)_{\boldsymbol{\beta}^{\mathrm{old}}}.
-$$
-
-The right-hand side is computed with the old values of \( \beta \).
-
-If we can compute these matrices, in particular the Hessian, the above is often the easiest method to implement.
-
Convex functions
diff --git a/doc/pub/week38/html/week38.html b/doc/pub/week38/html/week38.html
index d78efc01a..ec6f6a458 100644
--- a/doc/pub/week38/html/week38.html
+++ b/doc/pub/week38/html/week38.html
@@ -256,15 +256,6 @@ div.toc p,a {
2,
None,
'the-sensitiveness-of-the-gradient-descent'),
- ('Revisiting our Logistic Regression case',
- 2,
- None,
- 'revisiting-our-logistic-regression-case'),
- ('The equations to solve', 2, None, 'the-equations-to-solve'),
- ("Solving using Newton-Raphson's method",
- 2,
- None,
- 'solving-using-newton-raphson-s-method'),
('Convex functions', 2, None, 'convex-functions'),
('Convex function', 2, None, 'convex-function'),
('Conditions on convex functions',
@@ -1898,74 +1889,9 @@ large we can experience erratic behavior.
Many of these shortcomings can be alleviated by introducing
randomness. One such method is that of Stochastic Gradient Descent
-(SGD), see below.
+(SGD), to be discussed next week.
-
-Revisiting our Logistic Regression case
-
-In our discussion on Logistic Regression we studied the
-case of
-two classes, with \( y_i \) either
-\( 0 \) or \( 1 \). Furthermore we assumed also that we have only two
-parameters \( \beta \) in our fitting, that is we
-defined probabilities
-
-
-$$
-\begin{align*}
-p(y_i=1|x_i,\boldsymbol{\beta}) &= \frac{\exp{(\beta_0+\beta_1x_i)}}{1+\exp{(\beta_0+\beta_1x_i)}},\nonumber\\
-p(y_i=0|x_i,\boldsymbol{\beta}) &= 1 - p(y_i=1|x_i,\boldsymbol{\beta}),
-\end{align*}
-$$
-
-where \( \boldsymbol{\beta} \) are the weights we wish to extract from data, in our case \( \beta_0 \) and \( \beta_1 \).
-
-
-The equations to solve
-
-Our compact equations used a definition of a vector \( \boldsymbol{y} \) with \( n \)
-elements \( y_i \), an \( n\times p \) matrix \( \boldsymbol{X} \) which contains the
-\( x_i \) values and a vector \( \boldsymbol{p} \) of fitted probabilities
-\( p(y_i\vert x_i,\boldsymbol{\beta}) \). We rewrote in a more compact form
-the first derivative of the cost function as
-
-
-$$
-\frac{\partial \mathcal{C}(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}} = -\boldsymbol{X}^T\left(\boldsymbol{y}-\boldsymbol{p}\right).
-$$
-
-If we in addition define a diagonal matrix \( \boldsymbol{W} \) with elements
-\( p(y_i\vert x_i,\boldsymbol{\beta})(1-p(y_i\vert x_i,\boldsymbol{\beta}) \), we can obtain a compact expression of the second derivative as
-
-
-$$
-\frac{\partial^2 \mathcal{C}(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}\partial \boldsymbol{\beta}^T} = \boldsymbol{X}^T\boldsymbol{W}\boldsymbol{X}.
-$$
-
-This defines what is called the Hessian matrix.
-
-
-Solving using Newton-Raphson's method
-
-If we can set up these equations, Newton-Raphson's iterative method is normally the method of choice. It requires however that we can compute in an efficient way the matrices that define the first and second derivatives.
-
-Our iterative scheme is then given by
-
-$$
-\boldsymbol{\beta}^{\mathrm{new}} = \boldsymbol{\beta}^{\mathrm{old}}-\left(\frac{\partial^2 \mathcal{C}(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}\partial \boldsymbol{\beta}^T}\right)^{-1}_{\boldsymbol{\beta}^{\mathrm{old}}}\times \left(\frac{\partial \mathcal{C}(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}}\right)_{\boldsymbol{\beta}^{\mathrm{old}}},
-$$
-
-or in matrix form as
-
-$$
-\boldsymbol{\beta}^{\mathrm{new}} = \boldsymbol{\beta}^{\mathrm{old}}-\left(\boldsymbol{X}^T\boldsymbol{W}\boldsymbol{X} \right)^{-1}\times \left(-\boldsymbol{X}^T(\boldsymbol{y}-\boldsymbol{p}) \right)_{\boldsymbol{\beta}^{\mathrm{old}}}.
-$$
-
-The right-hand side is computed with the old values of \( \beta \).
-
-If we can compute these matrices, in particular the Hessian, the above is often the easiest method to implement.
-
Convex functions
diff --git a/doc/pub/week38/ipynb/ipynb-week38-src.tar.gz b/doc/pub/week38/ipynb/ipynb-week38-src.tar.gz
index 4408c6d9b..51effdeb0 100644
Binary files a/doc/pub/week38/ipynb/ipynb-week38-src.tar.gz and b/doc/pub/week38/ipynb/ipynb-week38-src.tar.gz differ
diff --git a/doc/pub/week38/ipynb/week38.ipynb b/doc/pub/week38/ipynb/week38.ipynb
index b0aca7109..8e33a53e5 100644
--- a/doc/pub/week38/ipynb/week38.ipynb
+++ b/doc/pub/week38/ipynb/week38.ipynb
@@ -2,7 +2,7 @@
"cells": [
{
"cell_type": "markdown",
- "id": "29ef836c",
+ "id": "e1437fb7",
"metadata": {
"editable": true
},
@@ -14,7 +14,7 @@
},
{
"cell_type": "markdown",
- "id": "04e43ab6",
+ "id": "7edae964",
"metadata": {
"editable": true
},
@@ -27,7 +27,7 @@
},
{
"cell_type": "markdown",
- "id": "874182e4",
+ "id": "b080984b",
"metadata": {
"editable": true
},
@@ -53,7 +53,7 @@
},
{
"cell_type": "markdown",
- "id": "d0b402a3",
+ "id": "8766162e",
"metadata": {
"editable": true
},
@@ -66,7 +66,7 @@
},
{
"cell_type": "markdown",
- "id": "b45b9ff7",
+ "id": "a0f26a88",
"metadata": {
"editable": true
},
@@ -78,7 +78,7 @@
},
{
"cell_type": "markdown",
- "id": "529e022b",
+ "id": "2ab8174d",
"metadata": {
"editable": true
},
@@ -88,7 +88,7 @@
},
{
"cell_type": "markdown",
- "id": "d0a946a9",
+ "id": "7310c495",
"metadata": {
"editable": true
},
@@ -101,7 +101,7 @@
},
{
"cell_type": "markdown",
- "id": "1e389406",
+ "id": "cf23c167",
"metadata": {
"editable": true
},
@@ -111,7 +111,7 @@
},
{
"cell_type": "markdown",
- "id": "6beb7daa",
+ "id": "99d4c9b7",
"metadata": {
"editable": true
},
@@ -123,7 +123,7 @@
},
{
"cell_type": "markdown",
- "id": "d2adcc96",
+ "id": "bd386f6d",
"metadata": {
"editable": true
},
@@ -136,7 +136,7 @@
},
{
"cell_type": "markdown",
- "id": "ea18f58f",
+ "id": "182d5d41",
"metadata": {
"editable": true
},
@@ -149,7 +149,7 @@
},
{
"cell_type": "markdown",
- "id": "1d1a3360",
+ "id": "0d3682e5",
"metadata": {
"editable": true
},
@@ -161,7 +161,7 @@
},
{
"cell_type": "markdown",
- "id": "ea825d7d",
+ "id": "25248610",
"metadata": {
"editable": true
},
@@ -173,7 +173,7 @@
},
{
"cell_type": "markdown",
- "id": "5be1e9a8",
+ "id": "da238137",
"metadata": {
"editable": true
},
@@ -183,7 +183,7 @@
},
{
"cell_type": "markdown",
- "id": "e5ab7fb3",
+ "id": "f9bb791c",
"metadata": {
"editable": true
},
@@ -196,7 +196,7 @@
},
{
"cell_type": "markdown",
- "id": "51eca8f5",
+ "id": "01cb254e",
"metadata": {
"editable": true
},
@@ -208,7 +208,7 @@
},
{
"cell_type": "markdown",
- "id": "bef46cda",
+ "id": "b6ab9d2d",
"metadata": {
"editable": true
},
@@ -220,7 +220,7 @@
},
{
"cell_type": "markdown",
- "id": "cb40ab90",
+ "id": "a5fd2fb1",
"metadata": {
"editable": true
},
@@ -245,7 +245,7 @@
},
{
"cell_type": "markdown",
- "id": "85735d28",
+ "id": "567db724",
"metadata": {
"editable": true
},
@@ -261,7 +261,7 @@
},
{
"cell_type": "markdown",
- "id": "8b201ff0",
+ "id": "db50c5d3",
"metadata": {
"editable": true
},
@@ -277,7 +277,7 @@
},
{
"cell_type": "markdown",
- "id": "ba6fb049",
+ "id": "77644e97",
"metadata": {
"editable": true
},
@@ -291,7 +291,7 @@
},
{
"cell_type": "markdown",
- "id": "32493fc3",
+ "id": "222c145f",
"metadata": {
"editable": true
},
@@ -319,7 +319,7 @@
},
{
"cell_type": "markdown",
- "id": "942d2d71",
+ "id": "1906df63",
"metadata": {
"editable": true
},
@@ -332,7 +332,7 @@
{
"cell_type": "code",
"execution_count": 1,
- "id": "0822cca6",
+ "id": "c152fd7e",
"metadata": {
"collapsed": false,
"editable": true
@@ -434,7 +434,7 @@
},
{
"cell_type": "markdown",
- "id": "5e6943d5",
+ "id": "5d7ccc04",
"metadata": {
"editable": true
},
@@ -456,7 +456,7 @@
},
{
"cell_type": "markdown",
- "id": "c6f01005",
+ "id": "b84fbc4f",
"metadata": {
"editable": true
},
@@ -482,7 +482,7 @@
},
{
"cell_type": "markdown",
- "id": "65485ae8",
+ "id": "495a8bd0",
"metadata": {
"editable": true
},
@@ -506,7 +506,7 @@
},
{
"cell_type": "markdown",
- "id": "77da4608",
+ "id": "48b58d37",
"metadata": {
"editable": true
},
@@ -531,7 +531,7 @@
},
{
"cell_type": "markdown",
- "id": "395292f8",
+ "id": "cf4ea236",
"metadata": {
"editable": true
},
@@ -543,7 +543,7 @@
},
{
"cell_type": "markdown",
- "id": "783b7c46",
+ "id": "eeb9a563",
"metadata": {
"editable": true
},
@@ -561,7 +561,7 @@
},
{
"cell_type": "markdown",
- "id": "1781e6fd",
+ "id": "5d676ec0",
"metadata": {
"editable": true
},
@@ -579,7 +579,7 @@
},
{
"cell_type": "markdown",
- "id": "e92a522b",
+ "id": "4f1c8890",
"metadata": {
"editable": true
},
@@ -590,7 +590,7 @@
},
{
"cell_type": "markdown",
- "id": "38f13d47",
+ "id": "95d3b55e",
"metadata": {
"editable": true
},
@@ -617,7 +617,7 @@
},
{
"cell_type": "markdown",
- "id": "f0cc1c19",
+ "id": "a58cb406",
"metadata": {
"editable": true
},
@@ -630,7 +630,7 @@
{
"cell_type": "code",
"execution_count": 2,
- "id": "8b9a2f9b",
+ "id": "cccbdeef",
"metadata": {
"collapsed": false,
"editable": true
@@ -695,7 +695,7 @@
},
{
"cell_type": "markdown",
- "id": "9e40a566",
+ "id": "eda7b9f2",
"metadata": {
"editable": true
},
@@ -708,7 +708,7 @@
{
"cell_type": "code",
"execution_count": 3,
- "id": "73b773ef",
+ "id": "c657b457",
"metadata": {
"collapsed": false,
"editable": true
@@ -727,7 +727,7 @@
},
{
"cell_type": "markdown",
- "id": "5f333675",
+ "id": "c027575d",
"metadata": {
"editable": true
},
@@ -738,7 +738,7 @@
},
{
"cell_type": "markdown",
- "id": "1912adab",
+ "id": "02de7a3d",
"metadata": {
"editable": true
},
@@ -750,7 +750,7 @@
},
{
"cell_type": "markdown",
- "id": "2789f0aa",
+ "id": "64969b5a",
"metadata": {
"editable": true
},
@@ -769,7 +769,7 @@
},
{
"cell_type": "markdown",
- "id": "077ff376",
+ "id": "c8870523",
"metadata": {
"editable": true
},
@@ -791,7 +791,7 @@
},
{
"cell_type": "markdown",
- "id": "ea6a9677",
+ "id": "17cebf36",
"metadata": {
"editable": true
},
@@ -803,7 +803,7 @@
},
{
"cell_type": "markdown",
- "id": "d5682b86",
+ "id": "ef1c3060",
"metadata": {
"editable": true
},
@@ -813,7 +813,7 @@
},
{
"cell_type": "markdown",
- "id": "e7cbcd65",
+ "id": "65471b2a",
"metadata": {
"editable": true
},
@@ -826,7 +826,7 @@
{
"cell_type": "code",
"execution_count": 4,
- "id": "35f06e8b",
+ "id": "e9a4591d",
"metadata": {
"collapsed": false,
"editable": true
@@ -891,7 +891,7 @@
},
{
"cell_type": "markdown",
- "id": "0d60f84f",
+ "id": "b988efeb",
"metadata": {
"editable": true
},
@@ -903,7 +903,7 @@
},
{
"cell_type": "markdown",
- "id": "b59afebf",
+ "id": "cfb6a943",
"metadata": {
"editable": true
},
@@ -918,7 +918,7 @@
},
{
"cell_type": "markdown",
- "id": "8c20e9af",
+ "id": "2361aee9",
"metadata": {
"editable": true
},
@@ -930,7 +930,7 @@
},
{
"cell_type": "markdown",
- "id": "6e678787",
+ "id": "847fd47b",
"metadata": {
"editable": true
},
@@ -942,7 +942,7 @@
},
{
"cell_type": "markdown",
- "id": "87315bfa",
+ "id": "be4b11d1",
"metadata": {
"editable": true
},
@@ -959,7 +959,7 @@
},
{
"cell_type": "markdown",
- "id": "b35af430",
+ "id": "0c649e6a",
"metadata": {
"editable": true
},
@@ -973,7 +973,7 @@
},
{
"cell_type": "markdown",
- "id": "95642c01",
+ "id": "a2382368",
"metadata": {
"editable": true
},
@@ -983,7 +983,7 @@
},
{
"cell_type": "markdown",
- "id": "216ec0e8",
+ "id": "8e56dda5",
"metadata": {
"editable": true
},
@@ -995,7 +995,7 @@
},
{
"cell_type": "markdown",
- "id": "17a272ea",
+ "id": "c77515e4",
"metadata": {
"editable": true
},
@@ -1007,7 +1007,7 @@
},
{
"cell_type": "markdown",
- "id": "4a947d2e",
+ "id": "d0d27dc3",
"metadata": {
"editable": true
},
@@ -1019,7 +1019,7 @@
},
{
"cell_type": "markdown",
- "id": "8af36a82",
+ "id": "4955b6b1",
"metadata": {
"editable": true
},
@@ -1030,7 +1030,7 @@
},
{
"cell_type": "markdown",
- "id": "9ee722ed",
+ "id": "c3e91e52",
"metadata": {
"editable": true
},
@@ -1042,7 +1042,7 @@
},
{
"cell_type": "markdown",
- "id": "ec4a3ac0",
+ "id": "3b3d97b6",
"metadata": {
"editable": true
},
@@ -1053,7 +1053,7 @@
},
{
"cell_type": "markdown",
- "id": "6bb4ca7d",
+ "id": "47890d23",
"metadata": {
"editable": true
},
@@ -1069,7 +1069,7 @@
},
{
"cell_type": "markdown",
- "id": "1a91b11b",
+ "id": "71d34eed",
"metadata": {
"editable": true
},
@@ -1081,7 +1081,7 @@
},
{
"cell_type": "markdown",
- "id": "6d4ad1b7",
+ "id": "243773d3",
"metadata": {
"editable": true
},
@@ -1091,7 +1091,7 @@
},
{
"cell_type": "markdown",
- "id": "dccd3d6f",
+ "id": "81fd7337",
"metadata": {
"editable": true
},
@@ -1103,7 +1103,7 @@
},
{
"cell_type": "markdown",
- "id": "be548919",
+ "id": "38fbde35",
"metadata": {
"editable": true
},
@@ -1118,7 +1118,7 @@
},
{
"cell_type": "markdown",
- "id": "59939b49",
+ "id": "bd7644e6",
"metadata": {
"editable": true
},
@@ -1130,7 +1130,7 @@
},
{
"cell_type": "markdown",
- "id": "06d249a9",
+ "id": "0fcc74e2",
"metadata": {
"editable": true
},
@@ -1141,7 +1141,7 @@
},
{
"cell_type": "markdown",
- "id": "d98943cd",
+ "id": "7b5aa78f",
"metadata": {
"editable": true
},
@@ -1153,7 +1153,7 @@
},
{
"cell_type": "markdown",
- "id": "ca77eee0",
+ "id": "45ee534e",
"metadata": {
"editable": true
},
@@ -1165,7 +1165,7 @@
},
{
"cell_type": "markdown",
- "id": "555522ce",
+ "id": "f62d8583",
"metadata": {
"editable": true
},
@@ -1177,7 +1177,7 @@
},
{
"cell_type": "markdown",
- "id": "fa6aa07e",
+ "id": "837cc6c1",
"metadata": {
"editable": true
},
@@ -1187,7 +1187,7 @@
},
{
"cell_type": "markdown",
- "id": "e7a47c69",
+ "id": "75b9139f",
"metadata": {
"editable": true
},
@@ -1199,7 +1199,7 @@
},
{
"cell_type": "markdown",
- "id": "1152b511",
+ "id": "372ee8ea",
"metadata": {
"editable": true
},
@@ -1213,7 +1213,7 @@
},
{
"cell_type": "markdown",
- "id": "e21201d4",
+ "id": "b171dd51",
"metadata": {
"editable": true
},
@@ -1225,7 +1225,7 @@
},
{
"cell_type": "markdown",
- "id": "bf0b1ce2",
+ "id": "7816121e",
"metadata": {
"editable": true
},
@@ -1235,7 +1235,7 @@
},
{
"cell_type": "markdown",
- "id": "bc15aa1e",
+ "id": "b58d0bc3",
"metadata": {
"editable": true
},
@@ -1247,7 +1247,7 @@
},
{
"cell_type": "markdown",
- "id": "fe389fa1",
+ "id": "0d75989b",
"metadata": {
"editable": true
},
@@ -1257,7 +1257,7 @@
},
{
"cell_type": "markdown",
- "id": "e4e1cbfd",
+ "id": "af883d63",
"metadata": {
"editable": true
},
@@ -1269,7 +1269,7 @@
},
{
"cell_type": "markdown",
- "id": "23152f67",
+ "id": "6727eebf",
"metadata": {
"editable": true
},
@@ -1280,7 +1280,7 @@
},
{
"cell_type": "markdown",
- "id": "0b904be3",
+ "id": "c8713334",
"metadata": {
"editable": true
},
@@ -1303,7 +1303,7 @@
},
{
"cell_type": "markdown",
- "id": "2d759e0f",
+ "id": "7819ea46",
"metadata": {
"editable": true
},
@@ -1315,7 +1315,7 @@
},
{
"cell_type": "markdown",
- "id": "fab6c23c",
+ "id": "278c2527",
"metadata": {
"editable": true
},
@@ -1325,7 +1325,7 @@
},
{
"cell_type": "markdown",
- "id": "aa3f9525",
+ "id": "376c873f",
"metadata": {
"editable": true
},
@@ -1337,7 +1337,7 @@
},
{
"cell_type": "markdown",
- "id": "cae6b286",
+ "id": "389cdb39",
"metadata": {
"editable": true
},
@@ -1354,7 +1354,7 @@
},
{
"cell_type": "markdown",
- "id": "4281725f",
+ "id": "5a3810ce",
"metadata": {
"editable": true
},
@@ -1364,7 +1364,7 @@
},
{
"cell_type": "markdown",
- "id": "33e897bb",
+ "id": "f1e08704",
"metadata": {
"editable": true
},
@@ -1383,7 +1383,7 @@
{
"cell_type": "code",
"execution_count": 5,
- "id": "9afb8352",
+ "id": "d2e8e94e",
"metadata": {
"collapsed": false,
"editable": true
@@ -1438,7 +1438,7 @@
},
{
"cell_type": "markdown",
- "id": "e98a4468",
+ "id": "d9054061",
"metadata": {
"editable": true
},
@@ -1450,7 +1450,7 @@
},
{
"cell_type": "markdown",
- "id": "c9f83bc2",
+ "id": "e8d8872f",
"metadata": {
"editable": true
},
@@ -1465,7 +1465,7 @@
{
"cell_type": "code",
"execution_count": 6,
- "id": "db311e69",
+ "id": "212e1c6b",
"metadata": {
"collapsed": false,
"editable": true
@@ -1518,7 +1518,7 @@
},
{
"cell_type": "markdown",
- "id": "ceec4fb1",
+ "id": "91b3b83c",
"metadata": {
"editable": true
},
@@ -1533,7 +1533,7 @@
},
{
"cell_type": "markdown",
- "id": "ad0c9dbb",
+ "id": "7e0ab91a",
"metadata": {
"editable": true
},
@@ -1553,7 +1553,7 @@
{
"cell_type": "code",
"execution_count": 7,
- "id": "11b6cfab",
+ "id": "92c30256",
"metadata": {
"collapsed": false,
"editable": true
@@ -1607,7 +1607,7 @@
},
{
"cell_type": "markdown",
- "id": "69430481",
+ "id": "fb8e4f5a",
"metadata": {
"editable": true
},
@@ -1622,7 +1622,7 @@
{
"cell_type": "code",
"execution_count": 8,
- "id": "74ca9036",
+ "id": "d040e14f",
"metadata": {
"collapsed": false,
"editable": true
@@ -1649,7 +1649,7 @@
},
{
"cell_type": "markdown",
- "id": "f8487661",
+ "id": "47009c95",
"metadata": {
"editable": true
},
@@ -1663,7 +1663,7 @@
{
"cell_type": "code",
"execution_count": 9,
- "id": "e6d68d3e",
+ "id": "40128247",
"metadata": {
"collapsed": false,
"editable": true
@@ -1708,7 +1708,7 @@
},
{
"cell_type": "markdown",
- "id": "49a15360",
+ "id": "dc60a6ef",
"metadata": {
"editable": true
},
@@ -1733,7 +1733,7 @@
{
"cell_type": "code",
"execution_count": 10,
- "id": "3870dc95",
+ "id": "47ec15b8",
"metadata": {
"collapsed": false,
"editable": true
@@ -1745,7 +1745,7 @@
},
{
"cell_type": "markdown",
- "id": "55b78595",
+ "id": "eb8b4198",
"metadata": {
"editable": true
},
@@ -1756,7 +1756,7 @@
{
"cell_type": "code",
"execution_count": 11,
- "id": "e0913cac",
+ "id": "58486f42",
"metadata": {
"collapsed": false,
"editable": true
@@ -1768,7 +1768,7 @@
},
{
"cell_type": "markdown",
- "id": "7b21be8d",
+ "id": "07834b9b",
"metadata": {
"editable": true
},
@@ -1781,7 +1781,7 @@
},
{
"cell_type": "markdown",
- "id": "b34571be",
+ "id": "9908e2e3",
"metadata": {
"editable": true
},
@@ -1792,7 +1792,7 @@
{
"cell_type": "code",
"execution_count": 12,
- "id": "93d889a8",
+ "id": "645bfbca",
"metadata": {
"collapsed": false,
"editable": true
@@ -1835,7 +1835,7 @@
},
{
"cell_type": "markdown",
- "id": "438e33e3",
+ "id": "f528c325",
"metadata": {
"editable": true
},
@@ -1856,7 +1856,7 @@
},
{
"cell_type": "markdown",
- "id": "252cd985",
+ "id": "ff30edd5",
"metadata": {
"editable": true
},
@@ -1873,7 +1873,7 @@
},
{
"cell_type": "markdown",
- "id": "c8eade61",
+ "id": "847724bd",
"metadata": {
"editable": true
},
@@ -1888,7 +1888,7 @@
},
{
"cell_type": "markdown",
- "id": "dea75e60",
+ "id": "f1969cb5",
"metadata": {
"editable": true
},
@@ -1898,7 +1898,7 @@
},
{
"cell_type": "markdown",
- "id": "aed7e5af",
+ "id": "e163f74c",
"metadata": {
"editable": true
},
@@ -1914,7 +1914,7 @@
},
{
"cell_type": "markdown",
- "id": "0f185516",
+ "id": "9b49b101",
"metadata": {
"editable": true
},
@@ -1926,7 +1926,7 @@
},
{
"cell_type": "markdown",
- "id": "720d1af1",
+ "id": "f19697ae",
"metadata": {
"editable": true
},
@@ -1937,7 +1937,7 @@
},
{
"cell_type": "markdown",
- "id": "e3203d75",
+ "id": "3f641f13",
"metadata": {
"editable": true
},
@@ -1949,7 +1949,7 @@
},
{
"cell_type": "markdown",
- "id": "a33dd1f0",
+ "id": "d82f5c8e",
"metadata": {
"editable": true
},
@@ -1959,7 +1959,7 @@
},
{
"cell_type": "markdown",
- "id": "e39fa431",
+ "id": "f8d50532",
"metadata": {
"editable": true
},
@@ -1973,7 +1973,7 @@
},
{
"cell_type": "markdown",
- "id": "b38d6957",
+ "id": "81e002c2",
"metadata": {
"editable": true
},
@@ -1985,7 +1985,7 @@
},
{
"cell_type": "markdown",
- "id": "19a4fce0",
+ "id": "9ec52629",
"metadata": {
"editable": true
},
@@ -1995,7 +1995,7 @@
},
{
"cell_type": "markdown",
- "id": "45d09cf0",
+ "id": "d0b02495",
"metadata": {
"editable": true
},
@@ -2007,7 +2007,7 @@
},
{
"cell_type": "markdown",
- "id": "e986bb71",
+ "id": "b1475d07",
"metadata": {
"editable": true
},
@@ -2019,7 +2019,7 @@
},
{
"cell_type": "markdown",
- "id": "af0c244c",
+ "id": "7297eb74",
"metadata": {
"editable": true
},
@@ -2039,7 +2039,7 @@
},
{
"cell_type": "markdown",
- "id": "9a061289",
+ "id": "5ca9494b",
"metadata": {
"editable": true
},
@@ -2055,7 +2055,7 @@
},
{
"cell_type": "markdown",
- "id": "afb03ff5",
+ "id": "ec42264c",
"metadata": {
"editable": true
},
@@ -2071,7 +2071,7 @@
},
{
"cell_type": "markdown",
- "id": "304b5ea7",
+ "id": "0d0c3ae9",
"metadata": {
"editable": true
},
@@ -2082,7 +2082,7 @@
},
{
"cell_type": "markdown",
- "id": "578c8540",
+ "id": "df03a2fd",
"metadata": {
"editable": true
},
@@ -2094,7 +2094,7 @@
},
{
"cell_type": "markdown",
- "id": "e8750d8f",
+ "id": "f6b5cd5a",
"metadata": {
"editable": true
},
@@ -2104,7 +2104,7 @@
},
{
"cell_type": "markdown",
- "id": "c306bf6a",
+ "id": "251ef281",
"metadata": {
"editable": true
},
@@ -2116,7 +2116,7 @@
},
{
"cell_type": "markdown",
- "id": "b247f287",
+ "id": "84013f15",
"metadata": {
"editable": true
},
@@ -2126,7 +2126,7 @@
},
{
"cell_type": "markdown",
- "id": "40851e5b",
+ "id": "0498425e",
"metadata": {
"editable": true
},
@@ -2138,7 +2138,7 @@
},
{
"cell_type": "markdown",
- "id": "d01734da",
+ "id": "096433ab",
"metadata": {
"editable": true
},
@@ -2160,7 +2160,7 @@
},
{
"cell_type": "markdown",
- "id": "1475257c",
+ "id": "b0851d8d",
"metadata": {
"editable": true
},
@@ -2173,7 +2173,7 @@
},
{
"cell_type": "markdown",
- "id": "bf260ca5",
+ "id": "010769ef",
"metadata": {
"editable": true
},
@@ -2186,7 +2186,7 @@
},
{
"cell_type": "markdown",
- "id": "1a6a0163",
+ "id": "d853f614",
"metadata": {
"editable": true
},
@@ -2196,7 +2196,7 @@
},
{
"cell_type": "markdown",
- "id": "75428d23",
+ "id": "4bac06cd",
"metadata": {
"editable": true
},
@@ -2214,7 +2214,7 @@
},
{
"cell_type": "markdown",
- "id": "3422e608",
+ "id": "85987829",
"metadata": {
"editable": true
},
@@ -2224,7 +2224,7 @@
},
{
"cell_type": "markdown",
- "id": "cea80cf2",
+ "id": "e0708642",
"metadata": {
"editable": true
},
@@ -2239,7 +2239,7 @@
},
{
"cell_type": "markdown",
- "id": "1d9bd9a3",
+ "id": "8c47428d",
"metadata": {
"editable": true
},
@@ -2249,7 +2249,7 @@
},
{
"cell_type": "markdown",
- "id": "60b2e14c",
+ "id": "5b1a1b9b",
"metadata": {
"editable": true
},
@@ -2263,7 +2263,7 @@
},
{
"cell_type": "markdown",
- "id": "2a422ee3",
+ "id": "688f39ed",
"metadata": {
"editable": true
},
@@ -2273,7 +2273,7 @@
},
{
"cell_type": "markdown",
- "id": "c44dd1ba",
+ "id": "0540c80a",
"metadata": {
"editable": true
},
@@ -2287,7 +2287,7 @@
},
{
"cell_type": "markdown",
- "id": "67b8a115",
+ "id": "8749a5d9",
"metadata": {
"editable": true
},
@@ -2302,7 +2302,7 @@
},
{
"cell_type": "markdown",
- "id": "09510bcc",
+ "id": "10468ee8",
"metadata": {
"editable": true
},
@@ -2319,7 +2319,7 @@
},
{
"cell_type": "markdown",
- "id": "50c80755",
+ "id": "fd5204db",
"metadata": {
"editable": true
},
@@ -2331,7 +2331,7 @@
},
{
"cell_type": "markdown",
- "id": "1fbf7076",
+ "id": "3643016f",
"metadata": {
"editable": true
},
@@ -2345,7 +2345,7 @@
},
{
"cell_type": "markdown",
- "id": "2d22e42e",
+ "id": "3f3e0600",
"metadata": {
"editable": true
},
@@ -2360,7 +2360,7 @@
},
{
"cell_type": "markdown",
- "id": "107c0c93",
+ "id": "51927cc7",
"metadata": {
"editable": true
},
@@ -2372,7 +2372,7 @@
},
{
"cell_type": "markdown",
- "id": "a94c6b7d",
+ "id": "62148b2a",
"metadata": {
"editable": true
},
@@ -2383,7 +2383,7 @@
},
{
"cell_type": "markdown",
- "id": "415e7088",
+ "id": "0aaedf94",
"metadata": {
"editable": true
},
@@ -2411,7 +2411,7 @@
},
{
"cell_type": "markdown",
- "id": "1fcd9773",
+ "id": "13569ae6",
"metadata": {
"editable": true
},
@@ -2428,175 +2428,12 @@
"\n",
"Many of these shortcomings can be alleviated by introducing\n",
"randomness. One such method is that of Stochastic Gradient Descent\n",
- "(SGD), see below."
+ "(SGD), to be discussed next week."
]
},
{
"cell_type": "markdown",
- "id": "7bc29e67",
- "metadata": {
- "editable": true
- },
- "source": [
- "## Revisiting our Logistic Regression case\n",
- "\n",
- "In our discussion on Logistic Regression we studied the \n",
- "case of\n",
- "two classes, with $y_i$ either\n",
- "$0$ or $1$. Furthermore we assumed also that we have only two\n",
- "parameters $\\beta$ in our fitting, that is we\n",
- "defined probabilities"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "2591623a",
- "metadata": {
- "editable": true
- },
- "source": [
- "$$\n",
- "\\begin{align*}\n",
- "p(y_i=1|x_i,\\boldsymbol{\\beta}) &= \\frac{\\exp{(\\beta_0+\\beta_1x_i)}}{1+\\exp{(\\beta_0+\\beta_1x_i)}},\\nonumber\\\\\n",
- "p(y_i=0|x_i,\\boldsymbol{\\beta}) &= 1 - p(y_i=1|x_i,\\boldsymbol{\\beta}),\n",
- "\\end{align*}\n",
- "$$"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "74941000",
- "metadata": {
- "editable": true
- },
- "source": [
- "where $\\boldsymbol{\\beta}$ are the weights we wish to extract from data, in our case $\\beta_0$ and $\\beta_1$."
- ]
- },
- {
- "cell_type": "markdown",
- "id": "e8a83c98",
- "metadata": {
- "editable": true
- },
- "source": [
- "## The equations to solve\n",
- "\n",
- "Our compact equations used a definition of a vector $\\boldsymbol{y}$ with $n$\n",
- "elements $y_i$, an $n\\times p$ matrix $\\boldsymbol{X}$ which contains the\n",
- "$x_i$ values and a vector $\\boldsymbol{p}$ of fitted probabilities\n",
- "$p(y_i\\vert x_i,\\boldsymbol{\\beta})$. We rewrote in a more compact form\n",
- "the first derivative of the cost function as"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "0a28a0fc",
- "metadata": {
- "editable": true
- },
- "source": [
- "$$\n",
- "\\frac{\\partial \\mathcal{C}(\\boldsymbol{\\beta})}{\\partial \\boldsymbol{\\beta}} = -\\boldsymbol{X}^T\\left(\\boldsymbol{y}-\\boldsymbol{p}\\right).\n",
- "$$"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "1186db52",
- "metadata": {
- "editable": true
- },
- "source": [
- "If we in addition define a diagonal matrix $\\boldsymbol{W}$ with elements \n",
- "$p(y_i\\vert x_i,\\boldsymbol{\\beta})(1-p(y_i\\vert x_i,\\boldsymbol{\\beta})$, we can obtain a compact expression of the second derivative as"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "86f167a3",
- "metadata": {
- "editable": true
- },
- "source": [
- "$$\n",
- "\\frac{\\partial^2 \\mathcal{C}(\\boldsymbol{\\beta})}{\\partial \\boldsymbol{\\beta}\\partial \\boldsymbol{\\beta}^T} = \\boldsymbol{X}^T\\boldsymbol{W}\\boldsymbol{X}.\n",
- "$$"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "559c083b",
- "metadata": {
- "editable": true
- },
- "source": [
- "This defines what is called the Hessian matrix."
- ]
- },
- {
- "cell_type": "markdown",
- "id": "527ecec2",
- "metadata": {
- "editable": true
- },
- "source": [
- "## Solving using Newton-Raphson's method\n",
- "\n",
- "If we can set up these equations, Newton-Raphson's iterative method is normally the method of choice. It requires however that we can compute in an efficient way the matrices that define the first and second derivatives. \n",
- "\n",
- "Our iterative scheme is then given by"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "2b43706a",
- "metadata": {
- "editable": true
- },
- "source": [
- "$$\n",
- "\\boldsymbol{\\beta}^{\\mathrm{new}} = \\boldsymbol{\\beta}^{\\mathrm{old}}-\\left(\\frac{\\partial^2 \\mathcal{C}(\\boldsymbol{\\beta})}{\\partial \\boldsymbol{\\beta}\\partial \\boldsymbol{\\beta}^T}\\right)^{-1}_{\\boldsymbol{\\beta}^{\\mathrm{old}}}\\times \\left(\\frac{\\partial \\mathcal{C}(\\boldsymbol{\\beta})}{\\partial \\boldsymbol{\\beta}}\\right)_{\\boldsymbol{\\beta}^{\\mathrm{old}}},\n",
- "$$"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "3f9cc07b",
- "metadata": {
- "editable": true
- },
- "source": [
- "or in matrix form as"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "efb2648e",
- "metadata": {
- "editable": true
- },
- "source": [
- "$$\n",
- "\\boldsymbol{\\beta}^{\\mathrm{new}} = \\boldsymbol{\\beta}^{\\mathrm{old}}-\\left(\\boldsymbol{X}^T\\boldsymbol{W}\\boldsymbol{X} \\right)^{-1}\\times \\left(-\\boldsymbol{X}^T(\\boldsymbol{y}-\\boldsymbol{p}) \\right)_{\\boldsymbol{\\beta}^{\\mathrm{old}}}.\n",
- "$$"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "e95e3abc",
- "metadata": {
- "editable": true
- },
- "source": [
- "The right-hand side is computed with the old values of $\\beta$. \n",
- "\n",
- "If we can compute these matrices, in particular the Hessian, the above is often the easiest method to implement."
- ]
- },
- {
- "cell_type": "markdown",
- "id": "57b7e63c",
+ "id": "bfdc53c0",
"metadata": {
"editable": true
},
@@ -2618,7 +2455,7 @@
},
{
"cell_type": "markdown",
- "id": "b9f8d543",
+ "id": "a73c982b",
"metadata": {
"editable": true
},
@@ -2630,7 +2467,7 @@
},
{
"cell_type": "markdown",
- "id": "a6e51c0c",
+ "id": "a56a2ae9",
"metadata": {
"editable": true
},
@@ -2667,7 +2504,7 @@
},
{
"cell_type": "markdown",
- "id": "3a1c6ae4",
+ "id": "3090b99d",
"metadata": {
"editable": true
},
@@ -2695,7 +2532,7 @@
},
{
"cell_type": "markdown",
- "id": "e6acadbb",
+ "id": "093b11b8",
"metadata": {
"editable": true
},
@@ -2725,7 +2562,7 @@
},
{
"cell_type": "markdown",
- "id": "0fa71cae",
+ "id": "6f15fc92",
"metadata": {
"editable": true
},
@@ -2749,7 +2586,7 @@
{
"cell_type": "code",
"execution_count": 13,
- "id": "3657e008",
+ "id": "ce786ea2",
"metadata": {
"collapsed": false,
"editable": true
@@ -2762,7 +2599,7 @@
},
{
"cell_type": "markdown",
- "id": "5cf5087d",
+ "id": "02fce565",
"metadata": {
"editable": true
},
@@ -2773,7 +2610,7 @@
},
{
"cell_type": "markdown",
- "id": "665338d9",
+ "id": "f2ae83a6",
"metadata": {
"editable": true
},
@@ -2785,7 +2622,7 @@
},
{
"cell_type": "markdown",
- "id": "12d24ce4",
+ "id": "9016e945",
"metadata": {
"editable": true
},
@@ -2795,7 +2632,7 @@
},
{
"cell_type": "markdown",
- "id": "e1ae1307",
+ "id": "9195a113",
"metadata": {
"editable": true
},
@@ -2807,7 +2644,7 @@
},
{
"cell_type": "markdown",
- "id": "6342baf4",
+ "id": "e66e1135",
"metadata": {
"editable": true
},
@@ -2821,7 +2658,7 @@
},
{
"cell_type": "markdown",
- "id": "7c27ddb3",
+ "id": "4e5fe9a8",
"metadata": {
"editable": true
},
@@ -2837,7 +2674,7 @@
},
{
"cell_type": "markdown",
- "id": "4f73b206",
+ "id": "4178bf2b",
"metadata": {
"editable": true
},
@@ -2847,7 +2684,7 @@
},
{
"cell_type": "markdown",
- "id": "16fbc3dd",
+ "id": "91dbd291",
"metadata": {
"editable": true
},
@@ -2859,7 +2696,7 @@
},
{
"cell_type": "markdown",
- "id": "82dca579",
+ "id": "b36bcd33",
"metadata": {
"editable": true
},
@@ -2869,7 +2706,7 @@
},
{
"cell_type": "markdown",
- "id": "87f38e83",
+ "id": "dcfc85c6",
"metadata": {
"editable": true
},
@@ -2881,7 +2718,7 @@
},
{
"cell_type": "markdown",
- "id": "d8f707ab",
+ "id": "cd6314b3",
"metadata": {
"editable": true
},
@@ -2895,7 +2732,7 @@
},
{
"cell_type": "markdown",
- "id": "51de9ac3",
+ "id": "54f6c9ae",
"metadata": {
"editable": true
},
@@ -2905,7 +2742,7 @@
},
{
"cell_type": "markdown",
- "id": "0bd452e8",
+ "id": "9b2c6838",
"metadata": {
"editable": true
},
@@ -2916,7 +2753,7 @@
},
{
"cell_type": "markdown",
- "id": "43da0408",
+ "id": "46b26877",
"metadata": {
"editable": true
},
@@ -2931,7 +2768,7 @@
},
{
"cell_type": "markdown",
- "id": "98adc1ed",
+ "id": "4972c71f",
"metadata": {
"editable": true
},
@@ -2941,7 +2778,7 @@
},
{
"cell_type": "markdown",
- "id": "c13bccab",
+ "id": "f8ed6356",
"metadata": {
"editable": true
},
@@ -2953,7 +2790,7 @@
},
{
"cell_type": "markdown",
- "id": "459b1374",
+ "id": "643c173d",
"metadata": {
"editable": true
},
@@ -2965,7 +2802,7 @@
},
{
"cell_type": "markdown",
- "id": "75dd7666",
+ "id": "c5629d6e",
"metadata": {
"editable": true
},
@@ -2980,7 +2817,7 @@
},
{
"cell_type": "markdown",
- "id": "778e9848",
+ "id": "ec12f593",
"metadata": {
"editable": true
},
@@ -2993,7 +2830,7 @@
{
"cell_type": "code",
"execution_count": 14,
- "id": "52f85a1c",
+ "id": "27555cc9",
"metadata": {
"collapsed": false,
"editable": true
@@ -3050,7 +2887,7 @@
},
{
"cell_type": "markdown",
- "id": "fc43d3cf",
+ "id": "b48c2f03",
"metadata": {
"editable": true
},
@@ -3061,7 +2898,7 @@
{
"cell_type": "code",
"execution_count": 15,
- "id": "2e5876fe",
+ "id": "54641cf5",
"metadata": {
"collapsed": false,
"editable": true
@@ -3088,7 +2925,7 @@
},
{
"cell_type": "markdown",
- "id": "3a778be9",
+ "id": "1ab2de11",
"metadata": {
"editable": true
},
@@ -3100,7 +2937,7 @@
},
{
"cell_type": "markdown",
- "id": "c8ea3390",
+ "id": "ebf9e606",
"metadata": {
"editable": true
},
@@ -3112,7 +2949,7 @@
},
{
"cell_type": "markdown",
- "id": "f469189c",
+ "id": "27d8b0b2",
"metadata": {
"editable": true
},
@@ -3122,7 +2959,7 @@
},
{
"cell_type": "markdown",
- "id": "0d2af466",
+ "id": "afa59358",
"metadata": {
"editable": true
},
@@ -3136,7 +2973,7 @@
},
{
"cell_type": "markdown",
- "id": "19c91458",
+ "id": "c8b5852f",
"metadata": {
"editable": true
},
@@ -3146,7 +2983,7 @@
},
{
"cell_type": "markdown",
- "id": "36048e3f",
+ "id": "8e13c9e7",
"metadata": {
"editable": true
},
@@ -3158,7 +2995,7 @@
},
{
"cell_type": "markdown",
- "id": "83d71583",
+ "id": "4e94f377",
"metadata": {
"editable": true
},
@@ -3169,7 +3006,7 @@
},
{
"cell_type": "markdown",
- "id": "2c5fe2ce",
+ "id": "2b2beaeb",
"metadata": {
"editable": true
},
@@ -3184,7 +3021,7 @@
},
{
"cell_type": "markdown",
- "id": "f7972c71",
+ "id": "36f7c79f",
"metadata": {
"editable": true
},
@@ -3198,7 +3035,7 @@
},
{
"cell_type": "markdown",
- "id": "5c018560",
+ "id": "fb66e23f",
"metadata": {
"editable": true
},
@@ -3209,7 +3046,7 @@
{
"cell_type": "code",
"execution_count": 16,
- "id": "e1570e58",
+ "id": "aeaf35cf",
"metadata": {
"collapsed": false,
"editable": true
@@ -3270,7 +3107,7 @@
},
{
"cell_type": "markdown",
- "id": "e1a6b8fd",
+ "id": "41ef6f59",
"metadata": {
"editable": true
},
@@ -3292,7 +3129,7 @@
},
{
"cell_type": "markdown",
- "id": "38ac0b06",
+ "id": "14c608fa",
"metadata": {
"editable": true
},
diff --git a/doc/src/week38/week38.do.txt b/doc/src/week38/week38.do.txt
index 65e4ae189..5cea91e1f 100644
--- a/doc/src/week38/week38.do.txt
+++ b/doc/src/week38/week38.do.txt
@@ -1355,82 +1355,7 @@ large we can experience erratic behavior.
Many of these shortcomings can be alleviated by introducing
randomness. One such method is that of Stochastic Gradient Descent
-(SGD), see below.
-
-
-
-
-
-!split
-===== Revisiting our Logistic Regression case =====
-
-In our discussion on Logistic Regression we studied the
-case of
-two classes, with $y_i$ either
-$0$ or $1$. Furthermore we assumed also that we have only two
-parameters $\beta$ in our fitting, that is we
-defined probabilities
-
-!bt
-\begin{align*}
-p(y_i=1|x_i,\bm{\beta}) &= \frac{\exp{(\beta_0+\beta_1x_i)}}{1+\exp{(\beta_0+\beta_1x_i)}},\nonumber\\
-p(y_i=0|x_i,\bm{\beta}) &= 1 - p(y_i=1|x_i,\bm{\beta}),
-\end{align*}
-!et
-where $\bm{\beta}$ are the weights we wish to extract from data, in our case $\beta_0$ and $\beta_1$.
-
-!split
-===== The equations to solve =====
-
-Our compact equations used a definition of a vector $\bm{y}$ with $n$
-elements $y_i$, an $n\times p$ matrix $\bm{X}$ which contains the
-$x_i$ values and a vector $\bm{p}$ of fitted probabilities
-$p(y_i\vert x_i,\bm{\beta})$. We rewrote in a more compact form
-the first derivative of the cost function as
-
-!bt
-\[
-\frac{\partial \mathcal{C}(\bm{\beta})}{\partial \bm{\beta}} = -\bm{X}^T\left(\bm{y}-\bm{p}\right).
-\]
-!et
-
-If we in addition define a diagonal matrix $\bm{W}$ with elements
-$p(y_i\vert x_i,\bm{\beta})(1-p(y_i\vert x_i,\bm{\beta})$, we can obtain a compact expression of the second derivative as
-
-!bt
-\[
-\frac{\partial^2 \mathcal{C}(\bm{\beta})}{\partial \bm{\beta}\partial \bm{\beta}^T} = \bm{X}^T\bm{W}\bm{X}.
-\]
-!et
-This defines what is called the Hessian matrix.
-
-
-
-
-
-!split
-===== Solving using Newton-Raphson's method =====
-
-If we can set up these equations, Newton-Raphson's iterative method is normally the method of choice. It requires however that we can compute in an efficient way the matrices that define the first and second derivatives.
-
-Our iterative scheme is then given by
-
-!bt
-\[
-\bm{\beta}^{\mathrm{new}} = \bm{\beta}^{\mathrm{old}}-\left(\frac{\partial^2 \mathcal{C}(\bm{\beta})}{\partial \bm{\beta}\partial \bm{\beta}^T}\right)^{-1}_{\bm{\beta}^{\mathrm{old}}}\times \left(\frac{\partial \mathcal{C}(\bm{\beta})}{\partial \bm{\beta}}\right)_{\bm{\beta}^{\mathrm{old}}},
-\]
-!et
-or in matrix form as
-
-!bt
-\[
-\bm{\beta}^{\mathrm{new}} = \bm{\beta}^{\mathrm{old}}-\left(\bm{X}^T\bm{W}\bm{X} \right)^{-1}\times \left(-\bm{X}^T(\bm{y}-\bm{p}) \right)_{\bm{\beta}^{\mathrm{old}}}.
-\]
-!et
-The right-hand side is computed with the old values of $\beta$.
-
-If we can compute these matrices, in particular the Hessian, the above is often the easiest method to implement.
-
+(SGD), to be discussed next week.
!split