From 8566eec36d0b169e9040d2dd4f129b95296d4f56 Mon Sep 17 00:00:00 2001 From: mhjensen Date: Fri, 25 Sep 2020 06:00:41 +0200 Subject: [PATCH] replacing hats with boldface --- doc/pub/week39/html/._week39-bs034.html | 14 +- doc/pub/week39/html/._week39-bs035.html | 12 +- doc/pub/week39/html/._week39-bs036.html | 4 +- doc/pub/week39/html/._week39-bs037.html | 6 +- doc/pub/week39/html/._week39-bs038.html | 2 +- doc/pub/week39/html/._week39-bs039.html | 14 +- doc/pub/week39/html/week39-reveal.html | 54 ++--- doc/pub/week39/html/week39-solarized.html | 52 ++--- doc/pub/week39/html/week39.html | 52 ++--- doc/pub/week39/ipynb/ipynb-week39-src.tar.gz | Bin 192 -> 192 bytes doc/pub/week39/ipynb/week39.ipynb | 213 +++++++++++-------- doc/src/week39/week39.do.txt | 52 ++--- 12 files changed, 260 insertions(+), 215 deletions(-) diff --git a/doc/pub/week39/html/._week39-bs034.html b/doc/pub/week39/html/._week39-bs034.html index 1e05ee655..b3ebfda6e 100644 --- a/doc/pub/week39/html/._week39-bs034.html +++ b/doc/pub/week39/html/._week39-bs034.html @@ -273,12 +273,16 @@ desirable properties such as:
  • The cost function is convex which guarantees that gradient descent converges for small enough learning rates
    • -We revisit the example from homework set 1 where we had -$$ -y_i = 5x_i^2 + 0.1\xi_i, \ i=1,\cdots,100 -$$ +We revisitan example similar to what we had in the first homework set. We had a function of the type -with \( x_i \in [0,1] \) chosen randomly with a uniform distribution. Additionally \( \xi_i \) represents stochastic noise chosen according to a normal distribution \( \cal {N}(0,1) \). +

    + + +

    x = 2*np.random.rand(m,1)
+y = 4+3*x+np.random.randn(m,1)
+
    +

    +with \( x_i \in [0,1] \) is chosen randomly using a uniform distribution. Additionally we have a stochastic noise chosen according to a normal distribution \( \cal {N}(0,1) \). The linear regression model is given by $$ h_\beta(x) = \boldsymbol{y} = \beta_0 + \beta_1 x, diff --git a/doc/pub/week39/html/._week39-bs035.html b/doc/pub/week39/html/._week39-bs035.html index e112a8dac..489ec73b6 100644 --- a/doc/pub/week39/html/._week39-bs035.html +++ b/doc/pub/week39/html/._week39-bs035.html @@ -265,18 +265,18 @@ MathJax.Hub.Config({ Let \( \mathbf{y} = (y_1,\cdots,y_n)^T \), \( \mathbf{\boldsymbol{y}} = (\boldsymbol{y}_1,\cdots,\boldsymbol{y}_n)^T \) and \( \beta = (\beta_0, \beta_1)^T \)

    -It is convenient to write \( \mathbf{\boldsymbol{y}} = X\beta \) where \( X \in \mathbb{R}^{100 \times 2} \) is the design matrix given by +It is convenient to write \( \mathbf{\boldsymbol{y}} = X\beta \) where \( X \in \mathbb{R}^{100 \times 2} \) is the design matrix given by (we keep the intercept here) $$ X \equiv \begin{bmatrix} -1 &; x_1 \\ -\vdots &; \vdots \\ -1 &; x_{100} &; \\ +1 & x_1 \\ +\vdots & \vdots \\ +1 & x_{100} & \\ \end{bmatrix}. $$ -The loss function is given by +The cost/loss/risk function is given by ( $$ -C(\beta) = ||X\beta-\mathbf{y}||^2 = ||X\beta||^2 - 2 \mathbf{y}^T X\beta + ||\mathbf{y}||^2 = \sum_{i=1}^{100} (\beta_0 + \beta_1 x_i)^2 - 2 y_i (\beta_0 + \beta_1 x_i) + y_i^2 +C(\beta) = \frac{1}{n}||X\beta-\mathbf{y}||_{2}^{2} = \frac{1}{n}\sum_{i=1}^{100}\left[ (\beta_0 + \beta_1 x_i)^2 - 2 y_i (\beta_0 + \beta_1 x_i) + y_i^2\right] $$ and we want to find \( \beta \) such that \( C(\beta) \) is minimized. diff --git a/doc/pub/week39/html/._week39-bs036.html b/doc/pub/week39/html/._week39-bs036.html index 57ee1ffe1..488d8f83b 100644 --- a/doc/pub/week39/html/._week39-bs036.html +++ b/doc/pub/week39/html/._week39-bs036.html @@ -264,9 +264,9 @@ MathJax.Hub.Config({

    Computing \( \partial C(\beta) / \partial \beta_0 \) and \( \partial C(\beta) / \partial \beta_1 \) we can show that the gradient can be written as $$ -\nabla_{\beta} C(\beta) = (\partial C(\beta) / \partial \beta_0, \partial C(\beta) / \partial \beta_1)^T = 2\begin{bmatrix} \sum_{i=1}^{100} \left(\beta_0+\beta_1x_i-y_i\right) \\ +\nabla_{\beta} C(\beta) = \frac{2}{n}\begin{bmatrix} \sum_{i=1}^{100} \left(\beta_0+\beta_1x_i-y_i\right) \\ \sum_{i=1}^{100}\left( x_i (\beta_0+\beta_1x_i)-y_ix_i\right) \\ -\end{bmatrix} = 2X^T(X\beta - \mathbf{y}), +\end{bmatrix} = \frac{2}{n}X^T(X\beta - \mathbf{y}), $$ where \( X \) is the design matrix defined above. diff --git a/doc/pub/week39/html/._week39-bs037.html b/doc/pub/week39/html/._week39-bs037.html index 112fed356..e2668e2cf 100644 --- a/doc/pub/week39/html/._week39-bs037.html +++ b/doc/pub/week39/html/._week39-bs037.html @@ -263,9 +263,9 @@ MathJax.Hub.Config({ The Hessian matrix of \( C(\beta) \) is given by $$ \boldsymbol{H} \equiv \begin{bmatrix} -\frac{\partial^2 C(\beta)}{\partial \beta_0^2} &; \frac{\partial^2 C(\beta)}{\partial \beta_0 \partial \beta_1} \\ -\frac{\partial^2 C(\beta)}{\partial \beta_0 \partial \beta_1} &; \frac{\partial^2 C(\beta)}{\partial \beta_1^2} &; \\ -\end{bmatrix} = 2X^T X. +\frac{\partial^2 C(\beta)}{\partial \beta_0^2} & \frac{\partial^2 C(\beta)}{\partial \beta_0 \partial \beta_1} \\ +\frac{\partial^2 C(\beta)}{\partial \beta_0 \partial \beta_1} & \frac{\partial^2 C(\beta)}{\partial \beta_1^2} & \\ +\end{bmatrix} = \frac{2}{n}X^T X. $$ This result implies that \( C(\beta) \) is a convex function since the matrix \( X^T X \) always is positive semi-definite. diff --git a/doc/pub/week39/html/._week39-bs038.html b/doc/pub/week39/html/._week39-bs038.html index fd8a6e3bb..4490148cf 100644 --- a/doc/pub/week39/html/._week39-bs038.html +++ b/doc/pub/week39/html/._week39-bs038.html @@ -270,7 +270,7 @@ $$

    We can use the expression we computed for the gradient and let use a \( \beta_0 \) be chosen randomly and let \( \gamma = 0.001 \). Stop iterating -when \( ||\nabla_\beta C(\beta_k) || \leq \epsilon = 10^{-8} \). +when \( ||\nabla_\beta C(\beta_k) || \leq \epsilon = 10^{-8} \). Note that the code below does not include the latter stop criterion.

    And finally we can compare our solution for \( \beta \) with the analytic result given by diff --git a/doc/pub/week39/html/._week39-bs039.html b/doc/pub/week39/html/._week39-bs039.html index e7497eb6e..43aaac74f 100644 --- a/doc/pub/week39/html/._week39-bs039.html +++ b/doc/pub/week39/html/._week39-bs039.html @@ -276,12 +276,12 @@ Here our simple example import sys # the number of datapoints -m = 100 -x = 2*np.random.rand(m,1) -y = 4+3*x+np.random.randn(m,1) +n = 100 +x = 2*np.random.rand(n,1) +y = 4+3*x+np.random.randn(n,1) -xb = np.c_[np.ones((m,1)), x] -beta_linreg = np.linalg.inv(xb.T.dot(xb)).dot(xb.T).dot(y) +X = np.c_[np.ones((n,1)), x] +beta_linreg = np.linalg.inv(X.T @ X) @ X.T @ y print(beta_linreg) beta = np.random.randn(2,1) @@ -289,8 +289,8 @@ eta = 0.1= 1000 for iter in range(Niterations): - gradients = 2.0/m*xb.T.dot(xb.dot(beta)-y) - beta -= eta*gradients + gradient = (2.0/n)*X.T @ (X @ beta-y) + beta -= eta*gradient print(beta) xnew = np.array([[0],[2]]) diff --git a/doc/pub/week39/html/week39-reveal.html b/doc/pub/week39/html/week39-reveal.html index 6ae6497c6..4befb5adc 100644 --- a/doc/pub/week39/html/week39-reveal.html +++ b/doc/pub/week39/html/week39-reveal.html @@ -1165,14 +1165,16 @@ desirable properties such as:

    -We revisit the example from homework set 1 where we had -

     
    -$$ -y_i = 5x_i^2 + 0.1\xi_i, \ i=1,\cdots,100 -$$ -

     
    +We revisitan example similar to what we had in the first homework set. We had a function of the type -with \( x_i \in [0,1] \) chosen randomly with a uniform distribution. Additionally \( \xi_i \) represents stochastic noise chosen according to a normal distribution \( \cal {N}(0,1) \). +

    + + +

    x = 2*np.random.rand(m,1)
+y = 4+3*x+np.random.randn(m,1)
+
    +

    +with \( x_i \in [0,1] \) is chosen randomly using a uniform distribution. Additionally we have a stochastic noise chosen according to a normal distribution \( \cal {N}(0,1) \). The linear regression model is given by

     
    $$ @@ -1196,21 +1198,21 @@ $$ Let \( \mathbf{y} = (y_1,\cdots,y_n)^T \), \( \mathbf{\boldsymbol{y}} = (\boldsymbol{y}_1,\cdots,\boldsymbol{y}_n)^T \) and \( \beta = (\beta_0, \beta_1)^T \)

    -It is convenient to write \( \mathbf{\boldsymbol{y}} = X\beta \) where \( X \in \mathbb{R}^{100 \times 2} \) is the design matrix given by +It is convenient to write \( \mathbf{\boldsymbol{y}} = X\beta \) where \( X \in \mathbb{R}^{100 \times 2} \) is the design matrix given by (we keep the intercept here)

     
    $$ X \equiv \begin{bmatrix} -1 &; x_1 \\ -\vdots &; \vdots \\ -1 &; x_{100} &; \\ +1 & x_1 \\ +\vdots & \vdots \\ +1 & x_{100} & \\ \end{bmatrix}. $$

     
    -The loss function is given by +The cost/loss/risk function is given by (

     
    $$ -C(\beta) = ||X\beta-\mathbf{y}||^2 = ||X\beta||^2 - 2 \mathbf{y}^T X\beta + ||\mathbf{y}||^2 = \sum_{i=1}^{100} (\beta_0 + \beta_1 x_i)^2 - 2 y_i (\beta_0 + \beta_1 x_i) + y_i^2 +C(\beta) = \frac{1}{n}||X\beta-\mathbf{y}||_{2}^{2} = \frac{1}{n}\sum_{i=1}^{100}\left[ (\beta_0 + \beta_1 x_i)^2 - 2 y_i (\beta_0 + \beta_1 x_i) + y_i^2\right] $$

     
    @@ -1225,9 +1227,9 @@ and we want to find \( \beta \) such that \( C(\beta) \) is minimized. Computing \( \partial C(\beta) / \partial \beta_0 \) and \( \partial C(\beta) / \partial \beta_1 \) we can show that the gradient can be written as

     
    $$ -\nabla_{\beta} C(\beta) = (\partial C(\beta) / \partial \beta_0, \partial C(\beta) / \partial \beta_1)^T = 2\begin{bmatrix} \sum_{i=1}^{100} \left(\beta_0+\beta_1x_i-y_i\right) \\ +\nabla_{\beta} C(\beta) = \frac{2}{n}\begin{bmatrix} \sum_{i=1}^{100} \left(\beta_0+\beta_1x_i-y_i\right) \\ \sum_{i=1}^{100}\left( x_i (\beta_0+\beta_1x_i)-y_ix_i\right) \\ -\end{bmatrix} = 2X^T(X\beta - \mathbf{y}), +\end{bmatrix} = \frac{2}{n}X^T(X\beta - \mathbf{y}), $$

     
    @@ -1241,9 +1243,9 @@ The Hessian matrix of \( C(\beta) \) is given by

     
    $$ \boldsymbol{H} \equiv \begin{bmatrix} -\frac{\partial^2 C(\beta)}{\partial \beta_0^2} &; \frac{\partial^2 C(\beta)}{\partial \beta_0 \partial \beta_1} \\ -\frac{\partial^2 C(\beta)}{\partial \beta_0 \partial \beta_1} &; \frac{\partial^2 C(\beta)}{\partial \beta_1^2} &; \\ -\end{bmatrix} = 2X^T X. +\frac{\partial^2 C(\beta)}{\partial \beta_0^2} & \frac{\partial^2 C(\beta)}{\partial \beta_0 \partial \beta_1} \\ +\frac{\partial^2 C(\beta)}{\partial \beta_0 \partial \beta_1} & \frac{\partial^2 C(\beta)}{\partial \beta_1^2} & \\ +\end{bmatrix} = \frac{2}{n}X^T X. $$

     
    @@ -1265,7 +1267,7 @@ $$

    We can use the expression we computed for the gradient and let use a \( \beta_0 \) be chosen randomly and let \( \gamma = 0.001 \). Stop iterating -when \( ||\nabla_\beta C(\beta_k) || \leq \epsilon = 10^{-8} \). +when \( ||\nabla_\beta C(\beta_k) || \leq \epsilon = 10^{-8} \). Note that the code below does not include the latter stop criterion.

    And finally we can compare our solution for \( \beta \) with the analytic result given by @@ -1291,12 +1293,12 @@ Here our simple example import sys # the number of datapoints -m = 100 -x = 2*np.random.rand(m,1) -y = 4+3*x+np.random.randn(m,1) +n = 100 +x = 2*np.random.rand(n,1) +y = 4+3*x+np.random.randn(n,1) -xb = np.c_[np.ones((m,1)), x] -beta_linreg = np.linalg.inv(xb.T.dot(xb)).dot(xb.T).dot(y) +X = np.c_[np.ones((n,1)), x] +beta_linreg = np.linalg.inv(X.T @ X) @ X.T @ y print(beta_linreg) beta = np.random.randn(2,1) @@ -1304,8 +1306,8 @@ eta = 0.1 Niterations = 1000 for iter in range(Niterations): - gradients = 2.0/m*xb.T.dot(xb.dot(beta)-y) - beta -= eta*gradients + gradient = (2.0/n)*X.T @ (X @ beta-y) + beta -= eta*gradient print(beta) xnew = np.array([[0],[2]]) diff --git a/doc/pub/week39/html/week39-solarized.html b/doc/pub/week39/html/week39-solarized.html index ee2540461..edace6844 100644 --- a/doc/pub/week39/html/week39-solarized.html +++ b/doc/pub/week39/html/week39-solarized.html @@ -1125,12 +1125,16 @@ desirable properties such as:

  • The cost function is convex which guarantees that gradient descent converges for small enough learning rates
    • -We revisit the example from homework set 1 where we had -$$ -y_i = 5x_i^2 + 0.1\xi_i, \ i=1,\cdots,100 -$$ +We revisitan example similar to what we had in the first homework set. We had a function of the type -with \( x_i \in [0,1] \) chosen randomly with a uniform distribution. Additionally \( \xi_i \) represents stochastic noise chosen according to a normal distribution \( \cal {N}(0,1) \). +

    + + +

    x = 2*np.random.rand(m,1)
+y = 4+3*x+np.random.randn(m,1)
+
    +

    +with \( x_i \in [0,1] \) is chosen randomly using a uniform distribution. Additionally we have a stochastic noise chosen according to a normal distribution \( \cal {N}(0,1) \). The linear regression model is given by $$ h_\beta(x) = \boldsymbol{y} = \beta_0 + \beta_1 x, @@ -1150,18 +1154,18 @@ $$ Let \( \mathbf{y} = (y_1,\cdots,y_n)^T \), \( \mathbf{\boldsymbol{y}} = (\boldsymbol{y}_1,\cdots,\boldsymbol{y}_n)^T \) and \( \beta = (\beta_0, \beta_1)^T \)

    -It is convenient to write \( \mathbf{\boldsymbol{y}} = X\beta \) where \( X \in \mathbb{R}^{100 \times 2} \) is the design matrix given by +It is convenient to write \( \mathbf{\boldsymbol{y}} = X\beta \) where \( X \in \mathbb{R}^{100 \times 2} \) is the design matrix given by (we keep the intercept here) $$ X \equiv \begin{bmatrix} -1 &; x_1 \\ -\vdots &; \vdots \\ -1 &; x_{100} &; \\ +1 & x_1 \\ +\vdots & \vdots \\ +1 & x_{100} & \\ \end{bmatrix}. $$ -The loss function is given by +The cost/loss/risk function is given by ( $$ -C(\beta) = ||X\beta-\mathbf{y}||^2 = ||X\beta||^2 - 2 \mathbf{y}^T X\beta + ||\mathbf{y}||^2 = \sum_{i=1}^{100} (\beta_0 + \beta_1 x_i)^2 - 2 y_i (\beta_0 + \beta_1 x_i) + y_i^2 +C(\beta) = \frac{1}{n}||X\beta-\mathbf{y}||_{2}^{2} = \frac{1}{n}\sum_{i=1}^{100}\left[ (\beta_0 + \beta_1 x_i)^2 - 2 y_i (\beta_0 + \beta_1 x_i) + y_i^2\right] $$ and we want to find \( \beta \) such that \( C(\beta) \) is minimized. @@ -1174,9 +1178,9 @@ and we want to find \( \beta \) such that \( C(\beta) \) is minimized.

    Computing \( \partial C(\beta) / \partial \beta_0 \) and \( \partial C(\beta) / \partial \beta_1 \) we can show that the gradient can be written as $$ -\nabla_{\beta} C(\beta) = (\partial C(\beta) / \partial \beta_0, \partial C(\beta) / \partial \beta_1)^T = 2\begin{bmatrix} \sum_{i=1}^{100} \left(\beta_0+\beta_1x_i-y_i\right) \\ +\nabla_{\beta} C(\beta) = \frac{2}{n}\begin{bmatrix} \sum_{i=1}^{100} \left(\beta_0+\beta_1x_i-y_i\right) \\ \sum_{i=1}^{100}\left( x_i (\beta_0+\beta_1x_i)-y_ix_i\right) \\ -\end{bmatrix} = 2X^T(X\beta - \mathbf{y}), +\end{bmatrix} = \frac{2}{n}X^T(X\beta - \mathbf{y}), $$ where \( X \) is the design matrix defined above. @@ -1188,9 +1192,9 @@ where \( X \) is the design matrix defined above. The Hessian matrix of \( C(\beta) \) is given by $$ \boldsymbol{H} \equiv \begin{bmatrix} -\frac{\partial^2 C(\beta)}{\partial \beta_0^2} &; \frac{\partial^2 C(\beta)}{\partial \beta_0 \partial \beta_1} \\ -\frac{\partial^2 C(\beta)}{\partial \beta_0 \partial \beta_1} &; \frac{\partial^2 C(\beta)}{\partial \beta_1^2} &; \\ -\end{bmatrix} = 2X^T X. +\frac{\partial^2 C(\beta)}{\partial \beta_0^2} & \frac{\partial^2 C(\beta)}{\partial \beta_0 \partial \beta_1} \\ +\frac{\partial^2 C(\beta)}{\partial \beta_0 \partial \beta_1} & \frac{\partial^2 C(\beta)}{\partial \beta_1^2} & \\ +\end{bmatrix} = \frac{2}{n}X^T X. $$ This result implies that \( C(\beta) \) is a convex function since the matrix \( X^T X \) always is positive semi-definite. @@ -1209,7 +1213,7 @@ $$

    We can use the expression we computed for the gradient and let use a \( \beta_0 \) be chosen randomly and let \( \gamma = 0.001 \). Stop iterating -when \( ||\nabla_\beta C(\beta_k) || \leq \epsilon = 10^{-8} \). +when \( ||\nabla_\beta C(\beta_k) || \leq \epsilon = 10^{-8} \). Note that the code below does not include the latter stop criterion.

    And finally we can compare our solution for \( \beta \) with the analytic result given by @@ -1235,12 +1239,12 @@ Here our simple example import sys # the number of datapoints -m = 100 -x = 2*np.random.rand(m,1) -y = 4+3*x+np.random.randn(m,1) +n = 100 +x = 2*np.random.rand(n,1) +y = 4+3*x+np.random.randn(n,1) -xb = np.c_[np.ones((m,1)), x] -beta_linreg = np.linalg.inv(xb.T.dot(xb)).dot(xb.T).dot(y) +X = np.c_[np.ones((n,1)), x] +beta_linreg = np.linalg.inv(X.T @ X) @ X.T @ y print(beta_linreg) beta = np.random.randn(2,1) @@ -1248,8 +1252,8 @@ eta = 0.1 Niterations = 1000 for iter in range(Niterations): - gradients = 2.0/m*xb.T.dot(xb.dot(beta)-y) - beta -= eta*gradients + gradient = (2.0/n)*X.T @ (X @ beta-y) + beta -= eta*gradient print(beta) xnew = np.array([[0],[2]]) diff --git a/doc/pub/week39/html/week39.html b/doc/pub/week39/html/week39.html index 6ff562ef6..05c7372cb 100644 --- a/doc/pub/week39/html/week39.html +++ b/doc/pub/week39/html/week39.html @@ -1130,12 +1130,16 @@ desirable properties such as:

  • The cost function is convex which guarantees that gradient descent converges for small enough learning rates
    • -We revisit the example from homework set 1 where we had -$$ -y_i = 5x_i^2 + 0.1\xi_i, \ i=1,\cdots,100 -$$ +We revisitan example similar to what we had in the first homework set. We had a function of the type -with \( x_i \in [0,1] \) chosen randomly with a uniform distribution. Additionally \( \xi_i \) represents stochastic noise chosen according to a normal distribution \( \cal {N}(0,1) \). +

    + + +

    x = 2*np.random.rand(m,1)
+y = 4+3*x+np.random.randn(m,1)
+
    +

    +with \( x_i \in [0,1] \) is chosen randomly using a uniform distribution. Additionally we have a stochastic noise chosen according to a normal distribution \( \cal {N}(0,1) \). The linear regression model is given by $$ h_\beta(x) = \boldsymbol{y} = \beta_0 + \beta_1 x, @@ -1155,18 +1159,18 @@ $$ Let \( \mathbf{y} = (y_1,\cdots,y_n)^T \), \( \mathbf{\boldsymbol{y}} = (\boldsymbol{y}_1,\cdots,\boldsymbol{y}_n)^T \) and \( \beta = (\beta_0, \beta_1)^T \)

    -It is convenient to write \( \mathbf{\boldsymbol{y}} = X\beta \) where \( X \in \mathbb{R}^{100 \times 2} \) is the design matrix given by +It is convenient to write \( \mathbf{\boldsymbol{y}} = X\beta \) where \( X \in \mathbb{R}^{100 \times 2} \) is the design matrix given by (we keep the intercept here) $$ X \equiv \begin{bmatrix} -1 &; x_1 \\ -\vdots &; \vdots \\ -1 &; x_{100} &; \\ +1 & x_1 \\ +\vdots & \vdots \\ +1 & x_{100} & \\ \end{bmatrix}. $$ -The loss function is given by +The cost/loss/risk function is given by ( $$ -C(\beta) = ||X\beta-\mathbf{y}||^2 = ||X\beta||^2 - 2 \mathbf{y}^T X\beta + ||\mathbf{y}||^2 = \sum_{i=1}^{100} (\beta_0 + \beta_1 x_i)^2 - 2 y_i (\beta_0 + \beta_1 x_i) + y_i^2 +C(\beta) = \frac{1}{n}||X\beta-\mathbf{y}||_{2}^{2} = \frac{1}{n}\sum_{i=1}^{100}\left[ (\beta_0 + \beta_1 x_i)^2 - 2 y_i (\beta_0 + \beta_1 x_i) + y_i^2\right] $$ and we want to find \( \beta \) such that \( C(\beta) \) is minimized. @@ -1179,9 +1183,9 @@ and we want to find \( \beta \) such that \( C(\beta) \) is minimized.

    Computing \( \partial C(\beta) / \partial \beta_0 \) and \( \partial C(\beta) / \partial \beta_1 \) we can show that the gradient can be written as $$ -\nabla_{\beta} C(\beta) = (\partial C(\beta) / \partial \beta_0, \partial C(\beta) / \partial \beta_1)^T = 2\begin{bmatrix} \sum_{i=1}^{100} \left(\beta_0+\beta_1x_i-y_i\right) \\ +\nabla_{\beta} C(\beta) = \frac{2}{n}\begin{bmatrix} \sum_{i=1}^{100} \left(\beta_0+\beta_1x_i-y_i\right) \\ \sum_{i=1}^{100}\left( x_i (\beta_0+\beta_1x_i)-y_ix_i\right) \\ -\end{bmatrix} = 2X^T(X\beta - \mathbf{y}), +\end{bmatrix} = \frac{2}{n}X^T(X\beta - \mathbf{y}), $$ where \( X \) is the design matrix defined above. @@ -1193,9 +1197,9 @@ where \( X \) is the design matrix defined above. The Hessian matrix of \( C(\beta) \) is given by $$ \boldsymbol{H} \equiv \begin{bmatrix} -\frac{\partial^2 C(\beta)}{\partial \beta_0^2} &; \frac{\partial^2 C(\beta)}{\partial \beta_0 \partial \beta_1} \\ -\frac{\partial^2 C(\beta)}{\partial \beta_0 \partial \beta_1} &; \frac{\partial^2 C(\beta)}{\partial \beta_1^2} &; \\ -\end{bmatrix} = 2X^T X. +\frac{\partial^2 C(\beta)}{\partial \beta_0^2} & \frac{\partial^2 C(\beta)}{\partial \beta_0 \partial \beta_1} \\ +\frac{\partial^2 C(\beta)}{\partial \beta_0 \partial \beta_1} & \frac{\partial^2 C(\beta)}{\partial \beta_1^2} & \\ +\end{bmatrix} = \frac{2}{n}X^T X. $$ This result implies that \( C(\beta) \) is a convex function since the matrix \( X^T X \) always is positive semi-definite. @@ -1214,7 +1218,7 @@ $$

    We can use the expression we computed for the gradient and let use a \( \beta_0 \) be chosen randomly and let \( \gamma = 0.001 \). Stop iterating -when \( ||\nabla_\beta C(\beta_k) || \leq \epsilon = 10^{-8} \). +when \( ||\nabla_\beta C(\beta_k) || \leq \epsilon = 10^{-8} \). Note that the code below does not include the latter stop criterion.

    And finally we can compare our solution for \( \beta \) with the analytic result given by @@ -1240,12 +1244,12 @@ Here our simple example import sys # the number of datapoints -m = 100 -x = 2*np.random.rand(m,1) -y = 4+3*x+np.random.randn(m,1) +n = 100 +x = 2*np.random.rand(n,1) +y = 4+3*x+np.random.randn(n,1) -xb = np.c_[np.ones((m,1)), x] -beta_linreg = np.linalg.inv(xb.T.dot(xb)).dot(xb.T).dot(y) +X = np.c_[np.ones((n,1)), x] +beta_linreg = np.linalg.inv(X.T @ X) @ X.T @ y print(beta_linreg) beta = np.random.randn(2,1) @@ -1253,8 +1257,8 @@ eta = 0.1= 1000 for iter in range(Niterations): - gradients = 2.0/m*xb.T.dot(xb.dot(beta)-y) - beta -= eta*gradients + gradient = (2.0/n)*X.T @ (X @ beta-y) + beta -= eta*gradient print(beta) xnew = np.array([[0],[2]]) diff --git a/doc/pub/week39/ipynb/ipynb-week39-src.tar.gz b/doc/pub/week39/ipynb/ipynb-week39-src.tar.gz index 09b10088679ff91d763e7c7a6b4846f61988dbf8..3d99f0b446b67102d0a5400c58bb07cd39ebdd00 100644 GIT binary patch delta 35 rcmX@Wcz}^jzMF#~F?%9gFh|k60EH=r^H?V)hI4oxnQ(`JgMk46!VC(t delta 35 rcmX@Wcz}^jzMF$#W#UA(V2(*sf)%D1&SROF7|!ANHflEm2Ll5D$wLa$ diff --git a/doc/pub/week39/ipynb/week39.ipynb b/doc/pub/week39/ipynb/week39.ipynb index c5771bd12..b2b5ef1b4 100644 --- a/doc/pub/week39/ipynb/week39.ipynb +++ b/doc/pub/week39/ipynb/week39.ipynb @@ -782,7 +782,9 @@ { "cell_type": "code", "execution_count": 1, - "metadata": {}, + "metadata": { + "collapsed": false + }, "outputs": [], "source": [ "%matplotlib inline\n", @@ -819,7 +821,9 @@ { "cell_type": "code", "execution_count": 2, - "metadata": {}, + "metadata": { + "collapsed": false + }, "outputs": [], "source": [ "pt.axis(\"equal\")\n", @@ -837,7 +841,9 @@ { "cell_type": "code", "execution_count": 3, - "metadata": {}, + "metadata": { + "collapsed": false + }, "outputs": [], "source": [ "x = guesses[-1]\n", @@ -854,7 +860,9 @@ { "cell_type": "code", "execution_count": 4, - "metadata": {}, + "metadata": { + "collapsed": false + }, "outputs": [], "source": [ "def f1d(alpha):\n", @@ -876,7 +884,9 @@ { "cell_type": "code", "execution_count": 5, - "metadata": {}, + "metadata": { + "collapsed": false + }, "outputs": [], "source": [ "pt.axis(\"equal\")\n", @@ -1246,23 +1256,26 @@ "\n", "3. The cost function is convex which guarantees that gradient descent converges for small enough learning rates\n", "\n", - "We revisit the example from homework set 1 where we had" + "We revisitan example similar to what we had in the first homework set. We had a function of the type" + ] + }, + { + "cell_type": "code", + "execution_count": 6, + "metadata": { + "collapsed": false + }, + "outputs": [], + "source": [ + "x = 2*np.random.rand(m,1)\n", + "y = 4+3*x+np.random.randn(m,1)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ - "$$\n", - "y_i = 5x_i^2 + 0.1\\xi_i, \\ i=1,\\cdots,100\n", - "$$" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "with $x_i \\in [0,1] $ chosen randomly with a uniform distribution. Additionally $\\xi_i$ represents stochastic noise chosen according to a normal distribution $\\cal {N}(0,1)$. \n", + "with $x_i \\in [0,1] $ is chosen randomly using a uniform distribution. Additionally we have a stochastic noise chosen according to a normal distribution $\\cal {N}(0,1)$. \n", "The linear regression model is given by" ] }, @@ -1300,7 +1313,7 @@ "\n", "Let $\\mathbf{y} = (y_1,\\cdots,y_n)^T$, $\\mathbf{\\boldsymbol{y}} = (\\boldsymbol{y}_1,\\cdots,\\boldsymbol{y}_n)^T$ and $\\beta = (\\beta_0, \\beta_1)^T$\n", "\n", - "It is convenient to write $\\mathbf{\\boldsymbol{y}} = X\\beta$ where $X \\in \\mathbb{R}^{100 \\times 2} $ is the design matrix given by" + "It is convenient to write $\\mathbf{\\boldsymbol{y}} = X\\beta$ where $X \\in \\mathbb{R}^{100 \\times 2} $ is the design matrix given by (we keep the intercept here)" ] }, { @@ -1309,9 +1322,9 @@ "source": [ "$$\n", "X \\equiv \\begin{bmatrix}\n", - "1 &; x_1 \\\\\n", - "\\vdots &; \\vdots \\\\\n", - "1 &; x_{100} &; \\\\\n", + "1 & x_1 \\\\\n", + "\\vdots & \\vdots \\\\\n", + "1 & x_{100} & \\\\\n", "\\end{bmatrix}.\n", "$$" ] @@ -1320,7 +1333,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "The loss function is given by" + "The cost/loss/risk function is given by (" ] }, { @@ -1328,7 +1341,7 @@ "metadata": {}, "source": [ "$$\n", - "C(\\beta) = ||X\\beta-\\mathbf{y}||^2 = ||X\\beta||^2 - 2 \\mathbf{y}^T X\\beta + ||\\mathbf{y}||^2 = \\sum_{i=1}^{100} (\\beta_0 + \\beta_1 x_i)^2 - 2 y_i (\\beta_0 + \\beta_1 x_i) + y_i^2\n", + "C(\\beta) = \\frac{1}{n}||X\\beta-\\mathbf{y}||_{2}^{2} = \\frac{1}{n}\\sum_{i=1}^{100}\\left[ (\\beta_0 + \\beta_1 x_i)^2 - 2 y_i (\\beta_0 + \\beta_1 x_i) + y_i^2\\right]\n", "$$" ] }, @@ -1348,9 +1361,9 @@ "metadata": {}, "source": [ "$$\n", - "\\nabla_{\\beta} C(\\beta) = (\\partial C(\\beta) / \\partial \\beta_0, \\partial C(\\beta) / \\partial \\beta_1)^T = 2\\begin{bmatrix} \\sum_{i=1}^{100} \\left(\\beta_0+\\beta_1x_i-y_i\\right) \\\\\n", + "\\nabla_{\\beta} C(\\beta) = \\frac{2}{n}\\begin{bmatrix} \\sum_{i=1}^{100} \\left(\\beta_0+\\beta_1x_i-y_i\\right) \\\\\n", "\\sum_{i=1}^{100}\\left( x_i (\\beta_0+\\beta_1x_i)-y_ix_i\\right) \\\\\n", - "\\end{bmatrix} = 2X^T(X\\beta - \\mathbf{y}),\n", + "\\end{bmatrix} = \\frac{2}{n}X^T(X\\beta - \\mathbf{y}),\n", "$$" ] }, @@ -1370,9 +1383,9 @@ "source": [ "$$\n", "\\boldsymbol{H} \\equiv \\begin{bmatrix}\n", - "\\frac{\\partial^2 C(\\beta)}{\\partial \\beta_0^2} &; \\frac{\\partial^2 C(\\beta)}{\\partial \\beta_0 \\partial \\beta_1} \\\\\n", - "\\frac{\\partial^2 C(\\beta)}{\\partial \\beta_0 \\partial \\beta_1} &; \\frac{\\partial^2 C(\\beta)}{\\partial \\beta_1^2} &; \\\\\n", - "\\end{bmatrix} = 2X^T X.\n", + "\\frac{\\partial^2 C(\\beta)}{\\partial \\beta_0^2} & \\frac{\\partial^2 C(\\beta)}{\\partial \\beta_0 \\partial \\beta_1} \\\\\n", + "\\frac{\\partial^2 C(\\beta)}{\\partial \\beta_0 \\partial \\beta_1} & \\frac{\\partial^2 C(\\beta)}{\\partial \\beta_1^2} & \\\\\n", + "\\end{bmatrix} = \\frac{2}{n}X^T X.\n", "$$" ] }, @@ -1405,7 +1418,7 @@ "source": [ "We can use the expression we computed for the gradient and let use a\n", "$\\beta_0$ be chosen randomly and let $\\gamma = 0.001$. Stop iterating\n", - "when $||\\nabla_\\beta C(\\beta_k) || \\leq \\epsilon = 10^{-8}$. \n", + "when $||\\nabla_\\beta C(\\beta_k) || \\leq \\epsilon = 10^{-8}$. **Note that the code below does not include the latter stop criterion**.\n", "\n", "And finally we can compare our solution for $\\beta$ with the analytic result given by \n", "$\\beta= (X^TX)^{-1} X^T \\mathbf{y}$.\n", @@ -1417,8 +1430,10 @@ }, { "cell_type": "code", - "execution_count": 6, - "metadata": {}, + "execution_count": 7, + "metadata": { + "collapsed": false + }, "outputs": [], "source": [ "\n", @@ -1432,12 +1447,12 @@ "import sys\n", "\n", "# the number of datapoints\n", - "m = 100\n", - "x = 2*np.random.rand(m,1)\n", - "y = 4+3*x+np.random.randn(m,1)\n", + "n = 100\n", + "x = 2*np.random.rand(n,1)\n", + "y = 4+3*x+np.random.randn(n,1)\n", "\n", - "xb = np.c_[np.ones((m,1)), x]\n", - "beta_linreg = np.linalg.inv(xb.T.dot(xb)).dot(xb.T).dot(y)\n", + "X = np.c_[np.ones((n,1)), x]\n", + "beta_linreg = np.linalg.inv(X.T @ X) @ X.T @ y\n", "print(beta_linreg)\n", "beta = np.random.randn(2,1)\n", "\n", @@ -1445,8 +1460,8 @@ "Niterations = 1000\n", "\n", "for iter in range(Niterations):\n", - " gradients = 2.0/m*xb.T.dot(xb.dot(beta)-y)\n", - " beta -= eta*gradients\n", + " gradient = (2.0/n)*X.T @ (X @ beta-y)\n", + " beta -= eta*gradient\n", "\n", "print(beta)\n", "xnew = np.array([[0],[2]])\n", @@ -1472,8 +1487,10 @@ }, { "cell_type": "code", - "execution_count": 7, - "metadata": {}, + "execution_count": 8, + "metadata": { + "collapsed": false + }, "outputs": [], "source": [ "# Importing various packages\n", @@ -1555,8 +1572,10 @@ }, { "cell_type": "code", - "execution_count": 8, - "metadata": {}, + "execution_count": 9, + "metadata": { + "collapsed": false + }, "outputs": [], "source": [ "from random import random, seed\n", @@ -1736,8 +1755,10 @@ }, { "cell_type": "code", - "execution_count": 9, - "metadata": {}, + "execution_count": 10, + "metadata": { + "collapsed": false + }, "outputs": [], "source": [ "import numpy as np \n", @@ -1798,8 +1819,10 @@ }, { "cell_type": "code", - "execution_count": 10, - "metadata": {}, + "execution_count": 11, + "metadata": { + "collapsed": false + }, "outputs": [], "source": [ "import numpy as np \n", @@ -1837,8 +1860,10 @@ }, { "cell_type": "code", - "execution_count": 11, - "metadata": {}, + "execution_count": 12, + "metadata": { + "collapsed": false + }, "outputs": [], "source": [ "# Importing various packages\n", @@ -2390,8 +2415,10 @@ }, { "cell_type": "code", - "execution_count": 12, - "metadata": {}, + "execution_count": 13, + "metadata": { + "collapsed": false + }, "outputs": [], "source": [ "import autograd.numpy as np\n", @@ -2446,8 +2473,10 @@ }, { "cell_type": "code", - "execution_count": 13, - "metadata": {}, + "execution_count": 14, + "metadata": { + "collapsed": false + }, "outputs": [], "source": [ "import autograd.numpy as np\n", @@ -2482,8 +2511,10 @@ }, { "cell_type": "code", - "execution_count": 14, - "metadata": {}, + "execution_count": 15, + "metadata": { + "collapsed": false + }, "outputs": [], "source": [ "import autograd.numpy as np\n", @@ -2533,8 +2564,10 @@ }, { "cell_type": "code", - "execution_count": 15, - "metadata": {}, + "execution_count": 16, + "metadata": { + "collapsed": false + }, "outputs": [], "source": [ "import autograd.numpy as np\n", @@ -2573,8 +2606,10 @@ }, { "cell_type": "code", - "execution_count": 16, - "metadata": {}, + "execution_count": 17, + "metadata": { + "collapsed": false + }, "outputs": [], "source": [ "import autograd.numpy as np\n", @@ -2605,8 +2640,10 @@ }, { "cell_type": "code", - "execution_count": 17, - "metadata": {}, + "execution_count": 18, + "metadata": { + "collapsed": false + }, "outputs": [], "source": [ "import autograd.numpy as np\n", @@ -2637,7 +2674,7 @@ "metadata": {}, "source": [ "1\n", - "7\n", + "8\n", " \n", "<\n", "<\n", @@ -2665,8 +2702,10 @@ }, { "cell_type": "code", - "execution_count": 18, - "metadata": {}, + "execution_count": 19, + "metadata": { + "collapsed": false + }, "outputs": [], "source": [ "import autograd.numpy as np\n", @@ -2689,8 +2728,10 @@ }, { "cell_type": "code", - "execution_count": 19, - "metadata": {}, + "execution_count": 20, + "metadata": { + "collapsed": false + }, "outputs": [], "source": [ "import autograd.numpy as np\n", @@ -2736,8 +2777,10 @@ }, { "cell_type": "code", - "execution_count": 20, - "metadata": {}, + "execution_count": 21, + "metadata": { + "collapsed": false + }, "outputs": [], "source": [ "import autograd.numpy as np\n", @@ -2764,8 +2807,10 @@ }, { "cell_type": "code", - "execution_count": 21, - "metadata": {}, + "execution_count": 22, + "metadata": { + "collapsed": false + }, "outputs": [], "source": [ "import autograd.numpy as np\n", @@ -2792,8 +2837,10 @@ }, { "cell_type": "code", - "execution_count": 22, - "metadata": {}, + "execution_count": 23, + "metadata": { + "collapsed": false + }, "outputs": [], "source": [ "import autograd.numpy as np\n", @@ -2822,8 +2869,10 @@ }, { "cell_type": "code", - "execution_count": 23, - "metadata": {}, + "execution_count": 24, + "metadata": { + "collapsed": false + }, "outputs": [], "source": [ "a += b\n", @@ -2833,25 +2882,7 @@ ] } ], - "metadata": { - "kernelspec": { - "display_name": "Python 3", - "language": "python", - "name": "python3" - }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.6.8" - } - }, + "metadata": {}, "nbformat": 4, "nbformat_minor": 4 } diff --git a/doc/src/week39/week39.do.txt b/doc/src/week39/week39.do.txt index d39a61535..6ed5b157c 100644 --- a/doc/src/week39/week39.do.txt +++ b/doc/src/week39/week39.do.txt @@ -766,13 +766,13 @@ o An analytical solution (recall homework set 1). o The gradient can be computed analytically. o The cost function is convex which guarantees that gradient descent converges for small enough learning rates -We revisit the example from homework set 1 where we had -!bt -\[ -y_i = 5x_i^2 + 0.1\xi_i, \ i=1,\cdots,100 -\] -!et -with $x_i \in [0,1] $ chosen randomly with a uniform distribution. Additionally $\xi_i$ represents stochastic noise chosen according to a normal distribution $\cal {N}(0,1)$. +We revisitan example similar to what we had in the first homework set. We had a function of the type + +!bc pycod +x = 2*np.random.rand(m,1) +y = 4+3*x+np.random.randn(m,1) +!ec +with $x_i \in [0,1] $ is chosen randomly using a uniform distribution. Additionally we have a stochastic noise chosen according to a normal distribution $\cal {N}(0,1)$. The linear regression model is given by !bt \[ @@ -791,20 +791,20 @@ such that Let $\mathbf{y} = (y_1,\cdots,y_n)^T$, $\mathbf{\bm{y}} = (\bm{y}_1,\cdots,\bm{y}_n)^T$ and $\beta = (\beta_0, \beta_1)^T$ -It is convenient to write $\mathbf{\bm{y}} = X\beta$ where $X \in \mathbb{R}^{100 \times 2} $ is the design matrix given by +It is convenient to write $\mathbf{\bm{y}} = X\beta$ where $X \in \mathbb{R}^{100 \times 2} $ is the design matrix given by (we keep the intercept here) !bt \[ X \equiv \begin{bmatrix} -1 &; x_1 \\ -\vdots &; \vdots \\ -1 &; x_{100} &; \\ +1 & x_1 \\ +\vdots & \vdots \\ +1 & x_{100} & \\ \end{bmatrix}. \] !et -The loss function is given by +The cost/loss/risk function is given by ( !bt \[ -C(\beta) = ||X\beta-\mathbf{y}||^2 = ||X\beta||^2 - 2 \mathbf{y}^T X\beta + ||\mathbf{y}||^2 = \sum_{i=1}^{100} (\beta_0 + \beta_1 x_i)^2 - 2 y_i (\beta_0 + \beta_1 x_i) + y_i^2 +C(\beta) = \frac{1}{n}||X\beta-\mathbf{y}||_{2}^{2} = \frac{1}{n}\sum_{i=1}^{100}\left[ (\beta_0 + \beta_1 x_i)^2 - 2 y_i (\beta_0 + \beta_1 x_i) + y_i^2\right] \] !et and we want to find $\beta$ such that $C(\beta)$ is minimized. @@ -815,9 +815,9 @@ and we want to find $\beta$ such that $C(\beta)$ is minimized. Computing $\partial C(\beta) / \partial \beta_0$ and $\partial C(\beta) / \partial \beta_1$ we can show that the gradient can be written as !bt \[ -\nabla_{\beta} C(\beta) = (\partial C(\beta) / \partial \beta_0, \partial C(\beta) / \partial \beta_1)^T = 2\begin{bmatrix} \sum_{i=1}^{100} \left(\beta_0+\beta_1x_i-y_i\right) \\ +\nabla_{\beta} C(\beta) = \frac{2}{n}\begin{bmatrix} \sum_{i=1}^{100} \left(\beta_0+\beta_1x_i-y_i\right) \\ \sum_{i=1}^{100}\left( x_i (\beta_0+\beta_1x_i)-y_ix_i\right) \\ -\end{bmatrix} = 2X^T(X\beta - \mathbf{y}), +\end{bmatrix} = \frac{2}{n}X^T(X\beta - \mathbf{y}), \] !et where $X$ is the design matrix defined above. @@ -828,9 +828,9 @@ The Hessian matrix of $C(\beta)$ is given by !bt \[ \bm{H} \equiv \begin{bmatrix} -\frac{\partial^2 C(\beta)}{\partial \beta_0^2} &; \frac{\partial^2 C(\beta)}{\partial \beta_0 \partial \beta_1} \\ -\frac{\partial^2 C(\beta)}{\partial \beta_0 \partial \beta_1} &; \frac{\partial^2 C(\beta)}{\partial \beta_1^2} &; \\ -\end{bmatrix} = 2X^T X. +\frac{\partial^2 C(\beta)}{\partial \beta_0^2} & \frac{\partial^2 C(\beta)}{\partial \beta_0 \partial \beta_1} \\ +\frac{\partial^2 C(\beta)}{\partial \beta_0 \partial \beta_1} & \frac{\partial^2 C(\beta)}{\partial \beta_1^2} & \\ +\end{bmatrix} = \frac{2}{n}X^T X. \] !et This result implies that $C(\beta)$ is a convex function since the matrix $X^T X$ always is positive semi-definite. @@ -850,7 +850,7 @@ We can now write a program that minimizes $C(\beta)$ using the gradient descent We can use the expression we computed for the gradient and let use a $\beta_0$ be chosen randomly and let $\gamma = 0.001$. Stop iterating -when $||\nabla_\beta C(\beta_k) || \leq \epsilon = 10^{-8}$. +when $||\nabla_\beta C(\beta_k) || \leq \epsilon = 10^{-8}$. _Note that the code below does not include the latter stop criterion_. And finally we can compare our solution for $\beta$ with the analytic result given by $\beta= (X^TX)^{-1} X^T \mathbf{y}$. @@ -871,12 +871,12 @@ from matplotlib.ticker import LinearLocator, FormatStrFormatter import sys # the number of datapoints -m = 100 -x = 2*np.random.rand(m,1) -y = 4+3*x+np.random.randn(m,1) +n = 100 +x = 2*np.random.rand(n,1) +y = 4+3*x+np.random.randn(n,1) -xb = np.c_[np.ones((m,1)), x] -beta_linreg = np.linalg.inv(xb.T.dot(xb)).dot(xb.T).dot(y) +X = np.c_[np.ones((n,1)), x] +beta_linreg = np.linalg.inv(X.T @ X) @ X.T @ y print(beta_linreg) beta = np.random.randn(2,1) @@ -884,8 +884,8 @@ eta = 0.1 Niterations = 1000 for iter in range(Niterations): - gradients = 2.0/m*xb.T.dot(xb.dot(beta)-y) - beta -= eta*gradients + gradient = (2.0/n)*X.T @ (X @ beta-y) + beta -= eta*gradient print(beta) xnew = np.array([[0],[2]])