lars/FYS-STK4155
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

replacing hats with boldface

Browse Source
This commit is contained in:
mhjensen
2020-09-25 06:00:41 +02:00
parent 352a68b1cc
commit 8566eec36d
12 changed files with 260 additions and 215 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/pub/week39/html/._week39-bs034.html
+9 -5
View File
@@ -273,12 +273,16 @@ desirable properties such as:
<li> The cost function is convex which guarantees that gradient descent converges for small enough learning rates</li>
</ol>
We revisit the example from homework set 1 where we had
$$
y_i = 5x_i^2 + 0.1\xi_i, \ i=1,\cdots,100
$$
We revisitan example similar to what we had in the first homework set. We had a function of the type
with \( x_i \in [0,1] \) chosen randomly with a uniform distribution. Additionally \( \xi_i \) represents stochastic noise chosen according to a normal distribution \( \cal {N}(0,1) \).
<p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span>x <span style="color: #666666">=</span> <span style="color: #666666">2*</span>np<span style="color: #666666">.</span>random<span style="color: #666666">.</span>rand(m,<span style="color: #666666">1</span>)
y <span style="color: #666666">=</span> <span style="color: #666666">4+3*</span>x<span style="color: #666666">+</span>np<span style="color: #666666">.</span>random<span style="color: #666666">.</span>randn(m,<span style="color: #666666">1</span>)
</pre></div>
<p>
with \( x_i \in [0,1] \) is chosen randomly using a uniform distribution. Additionally we have a stochastic noise chosen according to a normal distribution \( \cal {N}(0,1) \).
The linear regression model is given by
$$
h_\beta(x) = \boldsymbol{y} = \beta_0 + \beta_1 x,
doc/pub/week39/html/._week39-bs035.html
+6 -6
View File
@@ -265,18 +265,18 @@ MathJax.Hub.Config({
Let \( \mathbf{y} = (y_1,\cdots,y_n)^T \), \( \mathbf{\boldsymbol{y}} = (\boldsymbol{y}_1,\cdots,\boldsymbol{y}_n)^T \) and \( \beta = (\beta_0, \beta_1)^T \)
<p>
It is convenient to write \( \mathbf{\boldsymbol{y}} = X\beta \) where \( X \in \mathbb{R}^{100 \times 2} \) is the design matrix given by
It is convenient to write \( \mathbf{\boldsymbol{y}} = X\beta \) where \( X \in \mathbb{R}^{100 \times 2} \) is the design matrix given by (we keep the intercept here)
$$
X \equiv \begin{bmatrix}
1 &; x_1 \\
\vdots &; \vdots \\
1 &; x_{100} &; \\
1 & x_1 \\
\vdots & \vdots \\
1 & x_{100} & \\
\end{bmatrix}.
$$
The loss function is given by
The cost/loss/risk function is given by (
$$
C(\beta) = ||X\beta-\mathbf{y}||^2 = ||X\beta||^2 - 2 \mathbf{y}^T X\beta + ||\mathbf{y}||^2 = \sum_{i=1}^{100} (\beta_0 + \beta_1 x_i)^2 - 2 y_i (\beta_0 + \beta_1 x_i) + y_i^2
C(\beta) = \frac{1}{n}||X\beta-\mathbf{y}||_{2}^{2} = \frac{1}{n}\sum_{i=1}^{100}\left[ (\beta_0 + \beta_1 x_i)^2 - 2 y_i (\beta_0 + \beta_1 x_i) + y_i^2\right]
$$
and we want to find \( \beta \) such that \( C(\beta) \) is minimized.
doc/pub/week39/html/._week39-bs036.html
+2 -2
View File
@@ -264,9 +264,9 @@ MathJax.Hub.Config({
<p>
Computing \( \partial C(\beta) / \partial \beta_0 \) and \( \partial C(\beta) / \partial \beta_1 \) we can show that the gradient can be written as
$$
\nabla_{\beta} C(\beta) = (\partial C(\beta) / \partial \beta_0, \partial C(\beta) / \partial \beta_1)^T = 2\begin{bmatrix} \sum_{i=1}^{100} \left(\beta_0+\beta_1x_i-y_i\right) \\
\nabla_{\beta} C(\beta) = \frac{2}{n}\begin{bmatrix} \sum_{i=1}^{100} \left(\beta_0+\beta_1x_i-y_i\right) \\
\sum_{i=1}^{100}\left( x_i (\beta_0+\beta_1x_i)-y_ix_i\right) \\
\end{bmatrix} = 2X^T(X\beta - \mathbf{y}),
\end{bmatrix} = \frac{2}{n}X^T(X\beta - \mathbf{y}),
$$
where \( X \) is the design matrix defined above.
doc/pub/week39/html/._week39-bs037.html
+3 -3
View File
@@ -263,9 +263,9 @@ MathJax.Hub.Config({
The Hessian matrix of \( C(\beta) \) is given by
$$
\boldsymbol{H} \equiv \begin{bmatrix}
\frac{\partial^2 C(\beta)}{\partial \beta_0^2} &; \frac{\partial^2 C(\beta)}{\partial \beta_0 \partial \beta_1} \\
\frac{\partial^2 C(\beta)}{\partial \beta_0 \partial \beta_1} &; \frac{\partial^2 C(\beta)}{\partial \beta_1^2} &; \\
\end{bmatrix} = 2X^T X.
\frac{\partial^2 C(\beta)}{\partial \beta_0^2} & \frac{\partial^2 C(\beta)}{\partial \beta_0 \partial \beta_1} \\
\frac{\partial^2 C(\beta)}{\partial \beta_0 \partial \beta_1} & \frac{\partial^2 C(\beta)}{\partial \beta_1^2} & \\
\end{bmatrix} = \frac{2}{n}X^T X.
$$
This result implies that \( C(\beta) \) is a convex function since the matrix \( X^T X \) always is positive semi-definite.
doc/pub/week39/html/._week39-bs038.html
+1 -1
View File
@@ -270,7 +270,7 @@ $$
<p>
We can use the expression we computed for the gradient and let use a
\( \beta_0 \) be chosen randomly and let \( \gamma = 0.001 \). Stop iterating
when \( ||\nabla_\beta C(\beta_k) || \leq \epsilon = 10^{-8} \).
when \( ||\nabla_\beta C(\beta_k) || \leq \epsilon = 10^{-8} \). <b>Note that the code below does not include the latter stop criterion</b>.
<p>
And finally we can compare our solution for \( \beta \) with the analytic result given by
doc/pub/week39/html/._week39-bs039.html
+7 -7
View File
@@ -276,12 +276,12 @@ Here our simple example
<span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">sys</span>
<span style="color: #408080; font-style: italic"># the number of datapoints</span>
m <span style="color: #666666">=</span> <span style="color: #666666">100</span>
x <span style="color: #666666">=</span> <span style="color: #666666">2*</span>np<span style="color: #666666">.</span>random<span style="color: #666666">.</span>rand(m,<span style="color: #666666">1</span>)
y <span style="color: #666666">=</span> <span style="color: #666666">4+3*</span>x<span style="color: #666666">+</span>np<span style="color: #666666">.</span>random<span style="color: #666666">.</span>randn(m,<span style="color: #666666">1</span>)
n <span style="color: #666666">=</span> <span style="color: #666666">100</span>
x <span style="color: #666666">=</span> <span style="color: #666666">2*</span>np<span style="color: #666666">.</span>random<span style="color: #666666">.</span>rand(n,<span style="color: #666666">1</span>)
y <span style="color: #666666">=</span> <span style="color: #666666">4+3*</span>x<span style="color: #666666">+</span>np<span style="color: #666666">.</span>random<span style="color: #666666">.</span>randn(n,<span style="color: #666666">1</span>)
xb <span style="color: #666666">=</span> np<span style="color: #666666">.</span>c_[np<span style="color: #666666">.</span>ones((m,<span style="color: #666666">1</span>)), x]
beta_linreg <span style="color: #666666">=</span> np<span style="color: #666666">.</span>linalg<span style="color: #666666">.</span>inv(xb<span style="color: #666666">.</span>T<span style="color: #666666">.</span>dot(xb))<span style="color: #666666">.</span>dot(xb<span style="color: #666666">.</span>T)<span style="color: #666666">.</span>dot(y)
X <span style="color: #666666">=</span> np<span style="color: #666666">.</span>c_[np<span style="color: #666666">.</span>ones((n,<span style="color: #666666">1</span>)), x]
beta_linreg <span style="color: #666666">=</span> np<span style="color: #666666">.</span>linalg<span style="color: #666666">.</span>inv(X<span style="color: #666666">.</span>T <span style="color: #666666">@</span> X) <span style="color: #666666">@</span> X<span style="color: #666666">.</span>T <span style="color: #666666">@</span> y
<span style="color: #008000">print</span>(beta_linreg)
beta <span style="color: #666666">=</span> np<span style="color: #666666">.</span>random<span style="color: #666666">.</span>randn(<span style="color: #666666">2</span>,<span style="color: #666666">1</span>)
@@ -289,8 +289,8 @@ eta <span style="color: #666666">=</span> <span style="color: #666666">0.1</span
Niterations <span style="color: #666666">=</span> <span style="color: #666666">1000</span>
<span style="color: #008000; font-weight: bold">for</span> <span style="color: #008000">iter</span> <span style="color: #AA22FF; font-weight: bold">in</span> <span style="color: #008000">range</span>(Niterations):
gradients <span style="color: #666666">=</span> <span style="color: #666666">2.0/</span>m<span style="color: #666666">*</span>xb<span style="color: #666666">.</span>T<span style="color: #666666">.</span>dot(xb<span style="color: #666666">.</span>dot(beta)<span style="color: #666666">-</span>y)
beta <span style="color: #666666">-=</span> eta<span style="color: #666666">*</span>gradients
gradient <span style="color: #666666">=</span> (<span style="color: #666666">2.0/</span>n)<span style="color: #666666">*</span>X<span style="color: #666666">.</span>T <span style="color: #666666">@</span> (X <span style="color: #666666">@</span> beta<span style="color: #666666">-</span>y)
beta <span style="color: #666666">-=</span> eta<span style="color: #666666">*</span>gradient
<span style="color: #008000">print</span>(beta)
xnew <span style="color: #666666">=</span> np<span style="color: #666666">.</span>array([[<span style="color: #666666">0</span>],[<span style="color: #666666">2</span>]])
doc/pub/week39/html/week39-reveal.html
+28 -26
View File
@@ -1165,14 +1165,16 @@ desirable properties such as:
</ol>
<p>
We revisit the example from homework set 1 where we had
<p>&nbsp;<br>
$$
y_i = 5x_i^2 + 0.1\xi_i, \ i=1,\cdots,100
$$
<p>&nbsp;<br>
We revisitan example similar to what we had in the first homework set. We had a function of the type
with \( x_i \in [0,1] \) chosen randomly with a uniform distribution. Additionally \( \xi_i \) represents stochastic noise chosen according to a normal distribution \( \cal {N}(0,1) \).
<p>
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
<div class="highlight" style="background: #eeeedd"><pre style="font-size: 80%; line-height: 125%"><span></span>x = <span style="color: #B452CD">2</span>*np.random.rand(m,<span style="color: #B452CD">1</span>)
y = <span style="color: #B452CD">4</span>+<span style="color: #B452CD">3</span>*x+np.random.randn(m,<span style="color: #B452CD">1</span>)
</pre></div>
<p>
with \( x_i \in [0,1] \) is chosen randomly using a uniform distribution. Additionally we have a stochastic noise chosen according to a normal distribution \( \cal {N}(0,1) \).
The linear regression model is given by
<p>&nbsp;<br>
$$
@@ -1196,21 +1198,21 @@ $$
Let \( \mathbf{y} = (y_1,\cdots,y_n)^T \), \( \mathbf{\boldsymbol{y}} = (\boldsymbol{y}_1,\cdots,\boldsymbol{y}_n)^T \) and \( \beta = (\beta_0, \beta_1)^T \)
<p>
It is convenient to write \( \mathbf{\boldsymbol{y}} = X\beta \) where \( X \in \mathbb{R}^{100 \times 2} \) is the design matrix given by
It is convenient to write \( \mathbf{\boldsymbol{y}} = X\beta \) where \( X \in \mathbb{R}^{100 \times 2} \) is the design matrix given by (we keep the intercept here)
<p>&nbsp;<br>
$$
X \equiv \begin{bmatrix}
1 &; x_1 \\
\vdots &; \vdots \\
1 &; x_{100} &; \\
1 & x_1 \\
\vdots & \vdots \\
1 & x_{100} & \\
\end{bmatrix}.
$$
<p>&nbsp;<br>
The loss function is given by
The cost/loss/risk function is given by (
<p>&nbsp;<br>
$$
C(\beta) = ||X\beta-\mathbf{y}||^2 = ||X\beta||^2 - 2 \mathbf{y}^T X\beta + ||\mathbf{y}||^2 = \sum_{i=1}^{100} (\beta_0 + \beta_1 x_i)^2 - 2 y_i (\beta_0 + \beta_1 x_i) + y_i^2
C(\beta) = \frac{1}{n}||X\beta-\mathbf{y}||_{2}^{2} = \frac{1}{n}\sum_{i=1}^{100}\left[ (\beta_0 + \beta_1 x_i)^2 - 2 y_i (\beta_0 + \beta_1 x_i) + y_i^2\right]
$$
<p>&nbsp;<br>
@@ -1225,9 +1227,9 @@ and we want to find \( \beta \) such that \( C(\beta) \) is minimized.
Computing \( \partial C(\beta) / \partial \beta_0 \) and \( \partial C(\beta) / \partial \beta_1 \) we can show that the gradient can be written as
<p>&nbsp;<br>
$$
\nabla_{\beta} C(\beta) = (\partial C(\beta) / \partial \beta_0, \partial C(\beta) / \partial \beta_1)^T = 2\begin{bmatrix} \sum_{i=1}^{100} \left(\beta_0+\beta_1x_i-y_i\right) \\
\nabla_{\beta} C(\beta) = \frac{2}{n}\begin{bmatrix} \sum_{i=1}^{100} \left(\beta_0+\beta_1x_i-y_i\right) \\
\sum_{i=1}^{100}\left( x_i (\beta_0+\beta_1x_i)-y_ix_i\right) \\
\end{bmatrix} = 2X^T(X\beta - \mathbf{y}),
\end{bmatrix} = \frac{2}{n}X^T(X\beta - \mathbf{y}),
$$
<p>&nbsp;<br>
@@ -1241,9 +1243,9 @@ The Hessian matrix of \( C(\beta) \) is given by
<p>&nbsp;<br>
$$
\boldsymbol{H} \equiv \begin{bmatrix}
\frac{\partial^2 C(\beta)}{\partial \beta_0^2} &; \frac{\partial^2 C(\beta)}{\partial \beta_0 \partial \beta_1} \\
\frac{\partial^2 C(\beta)}{\partial \beta_0 \partial \beta_1} &; \frac{\partial^2 C(\beta)}{\partial \beta_1^2} &; \\
\end{bmatrix} = 2X^T X.
\frac{\partial^2 C(\beta)}{\partial \beta_0^2} & \frac{\partial^2 C(\beta)}{\partial \beta_0 \partial \beta_1} \\
\frac{\partial^2 C(\beta)}{\partial \beta_0 \partial \beta_1} & \frac{\partial^2 C(\beta)}{\partial \beta_1^2} & \\
\end{bmatrix} = \frac{2}{n}X^T X.
$$
<p>&nbsp;<br>
@@ -1265,7 +1267,7 @@ $$
<p>
We can use the expression we computed for the gradient and let use a
\( \beta_0 \) be chosen randomly and let \( \gamma = 0.001 \). Stop iterating
when \( ||\nabla_\beta C(\beta_k) || \leq \epsilon = 10^{-8} \).
when \( ||\nabla_\beta C(\beta_k) || \leq \epsilon = 10^{-8} \). <b>Note that the code below does not include the latter stop criterion</b>.
<p>
And finally we can compare our solution for \( \beta \) with the analytic result given by
@@ -1291,12 +1293,12 @@ Here our simple example
<span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">sys</span>
<span style="color: #228B22"># the number of datapoints</span>
m = <span style="color: #B452CD">100</span>
x = <span style="color: #B452CD">2</span>*np.random.rand(m,<span style="color: #B452CD">1</span>)
y = <span style="color: #B452CD">4</span>+<span style="color: #B452CD">3</span>*x+np.random.randn(m,<span style="color: #B452CD">1</span>)
n = <span style="color: #B452CD">100</span>
x = <span style="color: #B452CD">2</span>*np.random.rand(n,<span style="color: #B452CD">1</span>)
y = <span style="color: #B452CD">4</span>+<span style="color: #B452CD">3</span>*x+np.random.randn(n,<span style="color: #B452CD">1</span>)
xb = np.c_[np.ones((m,<span style="color: #B452CD">1</span>)), x]
beta_linreg = np.linalg.inv(xb.T.dot(xb)).dot(xb.T).dot(y)
X = np.c_[np.ones((n,<span style="color: #B452CD">1</span>)), x]
beta_linreg = np.linalg.inv(X.T @ X) @ X.T @ y
<span style="color: #658b00">print</span>(beta_linreg)
beta = np.random.randn(<span style="color: #B452CD">2</span>,<span style="color: #B452CD">1</span>)
@@ -1304,8 +1306,8 @@ eta = <span style="color: #B452CD">0.1</span>
Niterations = <span style="color: #B452CD">1000</span>
<span style="color: #8B008B; font-weight: bold">for</span> <span style="color: #658b00">iter</span> <span style="color: #8B008B">in</span> <span style="color: #658b00">range</span>(Niterations):
gradients = <span style="color: #B452CD">2.0</span>/m*xb.T.dot(xb.dot(beta)-y)
beta -= eta*gradients
gradient = (<span style="color: #B452CD">2.0</span>/n)*X.T @ (X @ beta-y)
beta -= eta*gradient
<span style="color: #658b00">print</span>(beta)
xnew = np.array([[<span style="color: #B452CD">0</span>],[<span style="color: #B452CD">2</span>]])
doc/pub/week39/html/week39-solarized.html
+28 -24
View File
@@ -1125,12 +1125,16 @@ desirable properties such as:
<li> The cost function is convex which guarantees that gradient descent converges for small enough learning rates</li>
</ol>
We revisit the example from homework set 1 where we had
$$
y_i = 5x_i^2 + 0.1\xi_i, \ i=1,\cdots,100
$$
We revisitan example similar to what we had in the first homework set. We had a function of the type
with \( x_i \in [0,1] \) chosen randomly with a uniform distribution. Additionally \( \xi_i \) represents stochastic noise chosen according to a normal distribution \( \cal {N}(0,1) \).
<p>
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
<div class="highlight" style="background: #eeeedd"><pre style="line-height: 125%"><span></span>x = <span style="color: #B452CD">2</span>*np.random.rand(m,<span style="color: #B452CD">1</span>)
y = <span style="color: #B452CD">4</span>+<span style="color: #B452CD">3</span>*x+np.random.randn(m,<span style="color: #B452CD">1</span>)
</pre></div>
<p>
with \( x_i \in [0,1] \) is chosen randomly using a uniform distribution. Additionally we have a stochastic noise chosen according to a normal distribution \( \cal {N}(0,1) \).
The linear regression model is given by
$$
h_\beta(x) = \boldsymbol{y} = \beta_0 + \beta_1 x,
@@ -1150,18 +1154,18 @@ $$
Let \( \mathbf{y} = (y_1,\cdots,y_n)^T \), \( \mathbf{\boldsymbol{y}} = (\boldsymbol{y}_1,\cdots,\boldsymbol{y}_n)^T \) and \( \beta = (\beta_0, \beta_1)^T \)
<p>
It is convenient to write \( \mathbf{\boldsymbol{y}} = X\beta \) where \( X \in \mathbb{R}^{100 \times 2} \) is the design matrix given by
It is convenient to write \( \mathbf{\boldsymbol{y}} = X\beta \) where \( X \in \mathbb{R}^{100 \times 2} \) is the design matrix given by (we keep the intercept here)
$$
X \equiv \begin{bmatrix}
1 &; x_1 \\
\vdots &; \vdots \\
1 &; x_{100} &; \\
1 & x_1 \\
\vdots & \vdots \\
1 & x_{100} & \\
\end{bmatrix}.
$$
The loss function is given by
The cost/loss/risk function is given by (
$$
C(\beta) = ||X\beta-\mathbf{y}||^2 = ||X\beta||^2 - 2 \mathbf{y}^T X\beta + ||\mathbf{y}||^2 = \sum_{i=1}^{100} (\beta_0 + \beta_1 x_i)^2 - 2 y_i (\beta_0 + \beta_1 x_i) + y_i^2
C(\beta) = \frac{1}{n}||X\beta-\mathbf{y}||_{2}^{2} = \frac{1}{n}\sum_{i=1}^{100}\left[ (\beta_0 + \beta_1 x_i)^2 - 2 y_i (\beta_0 + \beta_1 x_i) + y_i^2\right]
$$
and we want to find \( \beta \) such that \( C(\beta) \) is minimized.
@@ -1174,9 +1178,9 @@ and we want to find \( \beta \) such that \( C(\beta) \) is minimized.
<p>
Computing \( \partial C(\beta) / \partial \beta_0 \) and \( \partial C(\beta) / \partial \beta_1 \) we can show that the gradient can be written as
$$
\nabla_{\beta} C(\beta) = (\partial C(\beta) / \partial \beta_0, \partial C(\beta) / \partial \beta_1)^T = 2\begin{bmatrix} \sum_{i=1}^{100} \left(\beta_0+\beta_1x_i-y_i\right) \\
\nabla_{\beta} C(\beta) = \frac{2}{n}\begin{bmatrix} \sum_{i=1}^{100} \left(\beta_0+\beta_1x_i-y_i\right) \\
\sum_{i=1}^{100}\left( x_i (\beta_0+\beta_1x_i)-y_ix_i\right) \\
\end{bmatrix} = 2X^T(X\beta - \mathbf{y}),
\end{bmatrix} = \frac{2}{n}X^T(X\beta - \mathbf{y}),
$$
where \( X \) is the design matrix defined above.
@@ -1188,9 +1192,9 @@ where \( X \) is the design matrix defined above.
The Hessian matrix of \( C(\beta) \) is given by
$$
\boldsymbol{H} \equiv \begin{bmatrix}
\frac{\partial^2 C(\beta)}{\partial \beta_0^2} &; \frac{\partial^2 C(\beta)}{\partial \beta_0 \partial \beta_1} \\
\frac{\partial^2 C(\beta)}{\partial \beta_0 \partial \beta_1} &; \frac{\partial^2 C(\beta)}{\partial \beta_1^2} &; \\
\end{bmatrix} = 2X^T X.
\frac{\partial^2 C(\beta)}{\partial \beta_0^2} & \frac{\partial^2 C(\beta)}{\partial \beta_0 \partial \beta_1} \\
\frac{\partial^2 C(\beta)}{\partial \beta_0 \partial \beta_1} & \frac{\partial^2 C(\beta)}{\partial \beta_1^2} & \\
\end{bmatrix} = \frac{2}{n}X^T X.
$$
This result implies that \( C(\beta) \) is a convex function since the matrix \( X^T X \) always is positive semi-definite.
@@ -1209,7 +1213,7 @@ $$
<p>
We can use the expression we computed for the gradient and let use a
\( \beta_0 \) be chosen randomly and let \( \gamma = 0.001 \). Stop iterating
when \( ||\nabla_\beta C(\beta_k) || \leq \epsilon = 10^{-8} \).
when \( ||\nabla_\beta C(\beta_k) || \leq \epsilon = 10^{-8} \). <b>Note that the code below does not include the latter stop criterion</b>.
<p>
And finally we can compare our solution for \( \beta \) with the analytic result given by
@@ -1235,12 +1239,12 @@ Here our simple example
<span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">sys</span>
<span style="color: #228B22"># the number of datapoints</span>
m = <span style="color: #B452CD">100</span>
x = <span style="color: #B452CD">2</span>*np.random.rand(m,<span style="color: #B452CD">1</span>)
y = <span style="color: #B452CD">4</span>+<span style="color: #B452CD">3</span>*x+np.random.randn(m,<span style="color: #B452CD">1</span>)
n = <span style="color: #B452CD">100</span>
x = <span style="color: #B452CD">2</span>*np.random.rand(n,<span style="color: #B452CD">1</span>)
y = <span style="color: #B452CD">4</span>+<span style="color: #B452CD">3</span>*x+np.random.randn(n,<span style="color: #B452CD">1</span>)
xb = np.c_[np.ones((m,<span style="color: #B452CD">1</span>)), x]
beta_linreg = np.linalg.inv(xb.T.dot(xb)).dot(xb.T).dot(y)
X = np.c_[np.ones((n,<span style="color: #B452CD">1</span>)), x]
beta_linreg = np.linalg.inv(X.T @ X) @ X.T @ y
<span style="color: #658b00">print</span>(beta_linreg)
beta = np.random.randn(<span style="color: #B452CD">2</span>,<span style="color: #B452CD">1</span>)
@@ -1248,8 +1252,8 @@ eta = <span style="color: #B452CD">0.1</span>
Niterations = <span style="color: #B452CD">1000</span>
<span style="color: #8B008B; font-weight: bold">for</span> <span style="color: #658b00">iter</span> <span style="color: #8B008B">in</span> <span style="color: #658b00">range</span>(Niterations):
gradients = <span style="color: #B452CD">2.0</span>/m*xb.T.dot(xb.dot(beta)-y)
beta -= eta*gradients
gradient = (<span style="color: #B452CD">2.0</span>/n)*X.T @ (X @ beta-y)
beta -= eta*gradient
<span style="color: #658b00">print</span>(beta)
xnew = np.array([[<span style="color: #B452CD">0</span>],[<span style="color: #B452CD">2</span>]])
doc/pub/week39/html/week39.html
+28 -24
View File
@@ -1130,12 +1130,16 @@ desirable properties such as:
<li> The cost function is convex which guarantees that gradient descent converges for small enough learning rates</li>
</ol>
We revisit the example from homework set 1 where we had
$$
y_i = 5x_i^2 + 0.1\xi_i, \ i=1,\cdots,100
$$
We revisitan example similar to what we had in the first homework set. We had a function of the type
with \( x_i \in [0,1] \) chosen randomly with a uniform distribution. Additionally \( \xi_i \) represents stochastic noise chosen according to a normal distribution \( \cal {N}(0,1) \).
<p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span>x <span style="color: #666666">=</span> <span style="color: #666666">2*</span>np<span style="color: #666666">.</span>random<span style="color: #666666">.</span>rand(m,<span style="color: #666666">1</span>)
y <span style="color: #666666">=</span> <span style="color: #666666">4+3*</span>x<span style="color: #666666">+</span>np<span style="color: #666666">.</span>random<span style="color: #666666">.</span>randn(m,<span style="color: #666666">1</span>)
</pre></div>
<p>
with \( x_i \in [0,1] \) is chosen randomly using a uniform distribution. Additionally we have a stochastic noise chosen according to a normal distribution \( \cal {N}(0,1) \).
The linear regression model is given by
$$
h_\beta(x) = \boldsymbol{y} = \beta_0 + \beta_1 x,
@@ -1155,18 +1159,18 @@ $$
Let \( \mathbf{y} = (y_1,\cdots,y_n)^T \), \( \mathbf{\boldsymbol{y}} = (\boldsymbol{y}_1,\cdots,\boldsymbol{y}_n)^T \) and \( \beta = (\beta_0, \beta_1)^T \)
<p>
It is convenient to write \( \mathbf{\boldsymbol{y}} = X\beta \) where \( X \in \mathbb{R}^{100 \times 2} \) is the design matrix given by
It is convenient to write \( \mathbf{\boldsymbol{y}} = X\beta \) where \( X \in \mathbb{R}^{100 \times 2} \) is the design matrix given by (we keep the intercept here)
$$
X \equiv \begin{bmatrix}
1 &; x_1 \\
\vdots &; \vdots \\
1 &; x_{100} &; \\
1 & x_1 \\
\vdots & \vdots \\
1 & x_{100} & \\
\end{bmatrix}.
$$
The loss function is given by
The cost/loss/risk function is given by (
$$
C(\beta) = ||X\beta-\mathbf{y}||^2 = ||X\beta||^2 - 2 \mathbf{y}^T X\beta + ||\mathbf{y}||^2 = \sum_{i=1}^{100} (\beta_0 + \beta_1 x_i)^2 - 2 y_i (\beta_0 + \beta_1 x_i) + y_i^2
C(\beta) = \frac{1}{n}||X\beta-\mathbf{y}||_{2}^{2} = \frac{1}{n}\sum_{i=1}^{100}\left[ (\beta_0 + \beta_1 x_i)^2 - 2 y_i (\beta_0 + \beta_1 x_i) + y_i^2\right]
$$
and we want to find \( \beta \) such that \( C(\beta) \) is minimized.
@@ -1179,9 +1183,9 @@ and we want to find \( \beta \) such that \( C(\beta) \) is minimized.
<p>
Computing \( \partial C(\beta) / \partial \beta_0 \) and \( \partial C(\beta) / \partial \beta_1 \) we can show that the gradient can be written as
$$
\nabla_{\beta} C(\beta) = (\partial C(\beta) / \partial \beta_0, \partial C(\beta) / \partial \beta_1)^T = 2\begin{bmatrix} \sum_{i=1}^{100} \left(\beta_0+\beta_1x_i-y_i\right) \\
\nabla_{\beta} C(\beta) = \frac{2}{n}\begin{bmatrix} \sum_{i=1}^{100} \left(\beta_0+\beta_1x_i-y_i\right) \\
\sum_{i=1}^{100}\left( x_i (\beta_0+\beta_1x_i)-y_ix_i\right) \\
\end{bmatrix} = 2X^T(X\beta - \mathbf{y}),
\end{bmatrix} = \frac{2}{n}X^T(X\beta - \mathbf{y}),
$$
where \( X \) is the design matrix defined above.
@@ -1193,9 +1197,9 @@ where \( X \) is the design matrix defined above.
The Hessian matrix of \( C(\beta) \) is given by
$$
\boldsymbol{H} \equiv \begin{bmatrix}
\frac{\partial^2 C(\beta)}{\partial \beta_0^2} &; \frac{\partial^2 C(\beta)}{\partial \beta_0 \partial \beta_1} \\
\frac{\partial^2 C(\beta)}{\partial \beta_0 \partial \beta_1} &; \frac{\partial^2 C(\beta)}{\partial \beta_1^2} &; \\
\end{bmatrix} = 2X^T X.
\frac{\partial^2 C(\beta)}{\partial \beta_0^2} & \frac{\partial^2 C(\beta)}{\partial \beta_0 \partial \beta_1} \\
\frac{\partial^2 C(\beta)}{\partial \beta_0 \partial \beta_1} & \frac{\partial^2 C(\beta)}{\partial \beta_1^2} & \\
\end{bmatrix} = \frac{2}{n}X^T X.
$$
This result implies that \( C(\beta) \) is a convex function since the matrix \( X^T X \) always is positive semi-definite.
@@ -1214,7 +1218,7 @@ $$
<p>
We can use the expression we computed for the gradient and let use a
\( \beta_0 \) be chosen randomly and let \( \gamma = 0.001 \). Stop iterating
when \( ||\nabla_\beta C(\beta_k) || \leq \epsilon = 10^{-8} \).
when \( ||\nabla_\beta C(\beta_k) || \leq \epsilon = 10^{-8} \). <b>Note that the code below does not include the latter stop criterion</b>.
<p>
And finally we can compare our solution for \( \beta \) with the analytic result given by
@@ -1240,12 +1244,12 @@ Here our simple example
<span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">sys</span>
<span style="color: #408080; font-style: italic"># the number of datapoints</span>
m <span style="color: #666666">=</span> <span style="color: #666666">100</span>
x <span style="color: #666666">=</span> <span style="color: #666666">2*</span>np<span style="color: #666666">.</span>random<span style="color: #666666">.</span>rand(m,<span style="color: #666666">1</span>)
y <span style="color: #666666">=</span> <span style="color: #666666">4+3*</span>x<span style="color: #666666">+</span>np<span style="color: #666666">.</span>random<span style="color: #666666">.</span>randn(m,<span style="color: #666666">1</span>)
n <span style="color: #666666">=</span> <span style="color: #666666">100</span>
x <span style="color: #666666">=</span> <span style="color: #666666">2*</span>np<span style="color: #666666">.</span>random<span style="color: #666666">.</span>rand(n,<span style="color: #666666">1</span>)
y <span style="color: #666666">=</span> <span style="color: #666666">4+3*</span>x<span style="color: #666666">+</span>np<span style="color: #666666">.</span>random<span style="color: #666666">.</span>randn(n,<span style="color: #666666">1</span>)
xb <span style="color: #666666">=</span> np<span style="color: #666666">.</span>c_[np<span style="color: #666666">.</span>ones((m,<span style="color: #666666">1</span>)), x]
beta_linreg <span style="color: #666666">=</span> np<span style="color: #666666">.</span>linalg<span style="color: #666666">.</span>inv(xb<span style="color: #666666">.</span>T<span style="color: #666666">.</span>dot(xb))<span style="color: #666666">.</span>dot(xb<span style="color: #666666">.</span>T)<span style="color: #666666">.</span>dot(y)
X <span style="color: #666666">=</span> np<span style="color: #666666">.</span>c_[np<span style="color: #666666">.</span>ones((n,<span style="color: #666666">1</span>)), x]
beta_linreg <span style="color: #666666">=</span> np<span style="color: #666666">.</span>linalg<span style="color: #666666">.</span>inv(X<span style="color: #666666">.</span>T <span style="color: #666666">@</span> X) <span style="color: #666666">@</span> X<span style="color: #666666">.</span>T <span style="color: #666666">@</span> y
<span style="color: #008000">print</span>(beta_linreg)
beta <span style="color: #666666">=</span> np<span style="color: #666666">.</span>random<span style="color: #666666">.</span>randn(<span style="color: #666666">2</span>,<span style="color: #666666">1</span>)
@@ -1253,8 +1257,8 @@ eta <span style="color: #666666">=</span> <span style="color: #666666">0.1</span
Niterations <span style="color: #666666">=</span> <span style="color: #666666">1000</span>
<span style="color: #008000; font-weight: bold">for</span> <span style="color: #008000">iter</span> <span style="color: #AA22FF; font-weight: bold">in</span> <span style="color: #008000">range</span>(Niterations):
gradients <span style="color: #666666">=</span> <span style="color: #666666">2.0/</span>m<span style="color: #666666">*</span>xb<span style="color: #666666">.</span>T<span style="color: #666666">.</span>dot(xb<span style="color: #666666">.</span>dot(beta)<span style="color: #666666">-</span>y)
beta <span style="color: #666666">-=</span> eta<span style="color: #666666">*</span>gradients
gradient <span style="color: #666666">=</span> (<span style="color: #666666">2.0/</span>n)<span style="color: #666666">*</span>X<span style="color: #666666">.</span>T <span style="color: #666666">@</span> (X <span style="color: #666666">@</span> beta<span style="color: #666666">-</span>y)
beta <span style="color: #666666">-=</span> eta<span style="color: #666666">*</span>gradient
<span style="color: #008000">print</span>(beta)
xnew <span style="color: #666666">=</span> np<span style="color: #666666">.</span>array([[<span style="color: #666666">0</span>],[<span style="color: #666666">2</span>]])
doc/pub/week39/ipynb/ipynb-week39-src.tar.gz
BIN
View File
Binary file not shown.
doc/pub/week39/ipynb/week39.ipynb
+122 -91
View File
@@ -782,7 +782,9 @@
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"%matplotlib inline\n",
@@ -819,7 +821,9 @@
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"pt.axis(\"equal\")\n",
@@ -837,7 +841,9 @@
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"x = guesses[-1]\n",
@@ -854,7 +860,9 @@
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"def f1d(alpha):\n",
@@ -876,7 +884,9 @@
{
"cell_type": "code",
"execution_count": 5,
"metadata": {},
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"pt.axis(\"equal\")\n",
@@ -1246,23 +1256,26 @@
"\n",
"3. The cost function is convex which guarantees that gradient descent converges for small enough learning rates\n",
"\n",
"We revisit the example from homework set 1 where we had"
"We revisitan example similar to what we had in the first homework set. We had a function of the type"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"x = 2*np.random.rand(m,1)\n",
"y = 4+3*x+np.random.randn(m,1)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\n",
"y_i = 5x_i^2 + 0.1\\xi_i, \\ i=1,\\cdots,100\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"with $x_i \\in [0,1] $ chosen randomly with a uniform distribution. Additionally $\\xi_i$ represents stochastic noise chosen according to a normal distribution $\\cal {N}(0,1)$. \n",
"with $x_i \\in [0,1] $ is chosen randomly using a uniform distribution. Additionally we have a stochastic noise chosen according to a normal distribution $\\cal {N}(0,1)$. \n",
"The linear regression model is given by"
]
},
@@ -1300,7 +1313,7 @@
"\n",
"Let $\\mathbf{y} = (y_1,\\cdots,y_n)^T$, $\\mathbf{\\boldsymbol{y}} = (\\boldsymbol{y}_1,\\cdots,\\boldsymbol{y}_n)^T$ and $\\beta = (\\beta_0, \\beta_1)^T$\n",
"\n",
"It is convenient to write $\\mathbf{\\boldsymbol{y}} = X\\beta$ where $X \\in \\mathbb{R}^{100 \\times 2} $ is the design matrix given by"
"It is convenient to write $\\mathbf{\\boldsymbol{y}} = X\\beta$ where $X \\in \\mathbb{R}^{100 \\times 2} $ is the design matrix given by (we keep the intercept here)"
]
},
{
@@ -1309,9 +1322,9 @@
"source": [
"$$\n",
"X \\equiv \\begin{bmatrix}\n",
"1 &; x_1 \\\\\n",
"\\vdots &; \\vdots \\\\\n",
"1 &; x_{100} &; \\\\\n",
"1 & x_1 \\\\\n",
"\\vdots & \\vdots \\\\\n",
"1 & x_{100} & \\\\\n",
"\\end{bmatrix}.\n",
"$$"
]
@@ -1320,7 +1333,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"The loss function is given by"
"The cost/loss/risk function is given by ("
]
},
{
@@ -1328,7 +1341,7 @@
"metadata": {},
"source": [
"$$\n",
"C(\\beta) = ||X\\beta-\\mathbf{y}||^2 = ||X\\beta||^2 - 2 \\mathbf{y}^T X\\beta + ||\\mathbf{y}||^2 = \\sum_{i=1}^{100} (\\beta_0 + \\beta_1 x_i)^2 - 2 y_i (\\beta_0 + \\beta_1 x_i) + y_i^2\n",
"C(\\beta) = \\frac{1}{n}||X\\beta-\\mathbf{y}||_{2}^{2} = \\frac{1}{n}\\sum_{i=1}^{100}\\left[ (\\beta_0 + \\beta_1 x_i)^2 - 2 y_i (\\beta_0 + \\beta_1 x_i) + y_i^2\\right]\n",
"$$"
]
},
@@ -1348,9 +1361,9 @@
"metadata": {},
"source": [
"$$\n",
"\\nabla_{\\beta} C(\\beta) = (\\partial C(\\beta) / \\partial \\beta_0, \\partial C(\\beta) / \\partial \\beta_1)^T = 2\\begin{bmatrix} \\sum_{i=1}^{100} \\left(\\beta_0+\\beta_1x_i-y_i\\right) \\\\\n",
"\\nabla_{\\beta} C(\\beta) = \\frac{2}{n}\\begin{bmatrix} \\sum_{i=1}^{100} \\left(\\beta_0+\\beta_1x_i-y_i\\right) \\\\\n",
"\\sum_{i=1}^{100}\\left( x_i (\\beta_0+\\beta_1x_i)-y_ix_i\\right) \\\\\n",
"\\end{bmatrix} = 2X^T(X\\beta - \\mathbf{y}),\n",
"\\end{bmatrix} = \\frac{2}{n}X^T(X\\beta - \\mathbf{y}),\n",
"$$"
]
},
@@ -1370,9 +1383,9 @@
"source": [
"$$\n",
"\\boldsymbol{H} \\equiv \\begin{bmatrix}\n",
"\\frac{\\partial^2 C(\\beta)}{\\partial \\beta_0^2} &; \\frac{\\partial^2 C(\\beta)}{\\partial \\beta_0 \\partial \\beta_1} \\\\\n",
"\\frac{\\partial^2 C(\\beta)}{\\partial \\beta_0 \\partial \\beta_1} &; \\frac{\\partial^2 C(\\beta)}{\\partial \\beta_1^2} &; \\\\\n",
"\\end{bmatrix} = 2X^T X.\n",
"\\frac{\\partial^2 C(\\beta)}{\\partial \\beta_0^2} & \\frac{\\partial^2 C(\\beta)}{\\partial \\beta_0 \\partial \\beta_1} \\\\\n",
"\\frac{\\partial^2 C(\\beta)}{\\partial \\beta_0 \\partial \\beta_1} & \\frac{\\partial^2 C(\\beta)}{\\partial \\beta_1^2} & \\\\\n",
"\\end{bmatrix} = \\frac{2}{n}X^T X.\n",
"$$"
]
},
@@ -1405,7 +1418,7 @@
"source": [
"We can use the expression we computed for the gradient and let use a\n",
"$\\beta_0$ be chosen randomly and let $\\gamma = 0.001$. Stop iterating\n",
"when $||\\nabla_\\beta C(\\beta_k) || \\leq \\epsilon = 10^{-8}$. \n",
"when $||\\nabla_\\beta C(\\beta_k) || \\leq \\epsilon = 10^{-8}$. **Note that the code below does not include the latter stop criterion**.\n",
"\n",
"And finally we can compare our solution for $\\beta$ with the analytic result given by \n",
"$\\beta= (X^TX)^{-1} X^T \\mathbf{y}$.\n",
@@ -1417,8 +1430,10 @@
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {},
"execution_count": 7,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"\n",
@@ -1432,12 +1447,12 @@
"import sys\n",
"\n",
"# the number of datapoints\n",
"m = 100\n",
"x = 2*np.random.rand(m,1)\n",
"y = 4+3*x+np.random.randn(m,1)\n",
"n = 100\n",
"x = 2*np.random.rand(n,1)\n",
"y = 4+3*x+np.random.randn(n,1)\n",
"\n",
"xb = np.c_[np.ones((m,1)), x]\n",
"beta_linreg = np.linalg.inv(xb.T.dot(xb)).dot(xb.T).dot(y)\n",
"X = np.c_[np.ones((n,1)), x]\n",
"beta_linreg = np.linalg.inv(X.T @ X) @ X.T @ y\n",
"print(beta_linreg)\n",
"beta = np.random.randn(2,1)\n",
"\n",
@@ -1445,8 +1460,8 @@
"Niterations = 1000\n",
"\n",
"for iter in range(Niterations):\n",
" gradients = 2.0/m*xb.T.dot(xb.dot(beta)-y)\n",
" beta -= eta*gradients\n",
" gradient = (2.0/n)*X.T @ (X @ beta-y)\n",
" beta -= eta*gradient\n",
"\n",
"print(beta)\n",
"xnew = np.array([[0],[2]])\n",
@@ -1472,8 +1487,10 @@
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {},
"execution_count": 8,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"# Importing various packages\n",
@@ -1555,8 +1572,10 @@
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {},
"execution_count": 9,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"from random import random, seed\n",
@@ -1736,8 +1755,10 @@
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {},
"execution_count": 10,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"import numpy as np \n",
@@ -1798,8 +1819,10 @@
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {},
"execution_count": 11,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"import numpy as np \n",
@@ -1837,8 +1860,10 @@
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {},
"execution_count": 12,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"# Importing various packages\n",
@@ -2390,8 +2415,10 @@
},
{
"cell_type": "code",
"execution_count": 12,
"metadata": {},
"execution_count": 13,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"import autograd.numpy as np\n",
@@ -2446,8 +2473,10 @@
},
{
"cell_type": "code",
"execution_count": 13,
"metadata": {},
"execution_count": 14,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"import autograd.numpy as np\n",
@@ -2482,8 +2511,10 @@
},
{
"cell_type": "code",
"execution_count": 14,
"metadata": {},
"execution_count": 15,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"import autograd.numpy as np\n",
@@ -2533,8 +2564,10 @@
},
{
"cell_type": "code",
"execution_count": 15,
"metadata": {},
"execution_count": 16,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"import autograd.numpy as np\n",
@@ -2573,8 +2606,10 @@
},
{
"cell_type": "code",
"execution_count": 16,
"metadata": {},
"execution_count": 17,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"import autograd.numpy as np\n",
@@ -2605,8 +2640,10 @@
},
{
"cell_type": "code",
"execution_count": 17,
"metadata": {},
"execution_count": 18,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"import autograd.numpy as np\n",
@@ -2637,7 +2674,7 @@
"metadata": {},
"source": [
"1\n",
"7\n",
"8\n",
" \n",
"<\n",
"<\n",
@@ -2665,8 +2702,10 @@
},
{
"cell_type": "code",
"execution_count": 18,
"metadata": {},
"execution_count": 19,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"import autograd.numpy as np\n",
@@ -2689,8 +2728,10 @@
},
{
"cell_type": "code",
"execution_count": 19,
"metadata": {},
"execution_count": 20,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"import autograd.numpy as np\n",
@@ -2736,8 +2777,10 @@
},
{
"cell_type": "code",
"execution_count": 20,
"metadata": {},
"execution_count": 21,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"import autograd.numpy as np\n",
@@ -2764,8 +2807,10 @@
},
{
"cell_type": "code",
"execution_count": 21,
"metadata": {},
"execution_count": 22,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"import autograd.numpy as np\n",
@@ -2792,8 +2837,10 @@
},
{
"cell_type": "code",
"execution_count": 22,
"metadata": {},
"execution_count": 23,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"import autograd.numpy as np\n",
@@ -2822,8 +2869,10 @@
},
{
"cell_type": "code",
"execution_count": 23,
"metadata": {},
"execution_count": 24,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"a += b\n",
@@ -2833,25 +2882,7 @@
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.6.8"
}
},
"metadata": {},
"nbformat": 4,
"nbformat_minor": 4
}
doc/src/week39/week39.do.txt
+26 -26
View File
@@ -766,13 +766,13 @@ o An analytical solution (recall homework set 1).
o The gradient can be computed analytically.
o The cost function is convex which guarantees that gradient descent converges for small enough learning rates
We revisit the example from homework set 1 where we had
!bt
\[
y_i = 5x_i^2 + 0.1\xi_i, \ i=1,\cdots,100
\]
!et
with $x_i \in [0,1] $ chosen randomly with a uniform distribution. Additionally $\xi_i$ represents stochastic noise chosen according to a normal distribution $\cal {N}(0,1)$.
We revisitan example similar to what we had in the first homework set. We had a function of the type
!bc pycod
x = 2*np.random.rand(m,1)
y = 4+3*x+np.random.randn(m,1)
!ec
with $x_i \in [0,1] $ is chosen randomly using a uniform distribution. Additionally we have a stochastic noise chosen according to a normal distribution $\cal {N}(0,1)$.
The linear regression model is given by
!bt
\[
@@ -791,20 +791,20 @@ such that
Let $\mathbf{y} = (y_1,\cdots,y_n)^T$, $\mathbf{\bm{y}} = (\bm{y}_1,\cdots,\bm{y}_n)^T$ and $\beta = (\beta_0, \beta_1)^T$
It is convenient to write $\mathbf{\bm{y}} = X\beta$ where $X \in \mathbb{R}^{100 \times 2} $ is the design matrix given by
It is convenient to write $\mathbf{\bm{y}} = X\beta$ where $X \in \mathbb{R}^{100 \times 2} $ is the design matrix given by (we keep the intercept here)
!bt
\[
X \equiv \begin{bmatrix}
1 &; x_1 \\
\vdots &; \vdots \\
1 &; x_{100} &; \\
1 & x_1 \\
\vdots & \vdots \\
1 & x_{100} & \\
\end{bmatrix}.
\]
!et
The loss function is given by
The cost/loss/risk function is given by (
!bt
\[
C(\beta) = ||X\beta-\mathbf{y}||^2 = ||X\beta||^2 - 2 \mathbf{y}^T X\beta + ||\mathbf{y}||^2 = \sum_{i=1}^{100} (\beta_0 + \beta_1 x_i)^2 - 2 y_i (\beta_0 + \beta_1 x_i) + y_i^2
C(\beta) = \frac{1}{n}||X\beta-\mathbf{y}||_{2}^{2} = \frac{1}{n}\sum_{i=1}^{100}\left[ (\beta_0 + \beta_1 x_i)^2 - 2 y_i (\beta_0 + \beta_1 x_i) + y_i^2\right]
\]
!et
and we want to find $\beta$ such that $C(\beta)$ is minimized.
@@ -815,9 +815,9 @@ and we want to find $\beta$ such that $C(\beta)$ is minimized.
Computing $\partial C(\beta) / \partial \beta_0$ and $\partial C(\beta) / \partial \beta_1$ we can show that the gradient can be written as
!bt
\[
\nabla_{\beta} C(\beta) = (\partial C(\beta) / \partial \beta_0, \partial C(\beta) / \partial \beta_1)^T = 2\begin{bmatrix} \sum_{i=1}^{100} \left(\beta_0+\beta_1x_i-y_i\right) \\
\nabla_{\beta} C(\beta) = \frac{2}{n}\begin{bmatrix} \sum_{i=1}^{100} \left(\beta_0+\beta_1x_i-y_i\right) \\
\sum_{i=1}^{100}\left( x_i (\beta_0+\beta_1x_i)-y_ix_i\right) \\
\end{bmatrix} = 2X^T(X\beta - \mathbf{y}),
\end{bmatrix} = \frac{2}{n}X^T(X\beta - \mathbf{y}),
\]
!et
where $X$ is the design matrix defined above.
@@ -828,9 +828,9 @@ The Hessian matrix of $C(\beta)$ is given by
!bt
\[
\bm{H} \equiv \begin{bmatrix}
\frac{\partial^2 C(\beta)}{\partial \beta_0^2} &; \frac{\partial^2 C(\beta)}{\partial \beta_0 \partial \beta_1} \\
\frac{\partial^2 C(\beta)}{\partial \beta_0 \partial \beta_1} &; \frac{\partial^2 C(\beta)}{\partial \beta_1^2} &; \\
\end{bmatrix} = 2X^T X.
\frac{\partial^2 C(\beta)}{\partial \beta_0^2} & \frac{\partial^2 C(\beta)}{\partial \beta_0 \partial \beta_1} \\
\frac{\partial^2 C(\beta)}{\partial \beta_0 \partial \beta_1} & \frac{\partial^2 C(\beta)}{\partial \beta_1^2} & \\
\end{bmatrix} = \frac{2}{n}X^T X.
\]
!et
This result implies that $C(\beta)$ is a convex function since the matrix $X^T X$ always is positive semi-definite.
@@ -850,7 +850,7 @@ We can now write a program that minimizes $C(\beta)$ using the gradient descent
We can use the expression we computed for the gradient and let use a
$\beta_0$ be chosen randomly and let $\gamma = 0.001$. Stop iterating
when $||\nabla_\beta C(\beta_k) || \leq \epsilon = 10^{-8}$.
when $||\nabla_\beta C(\beta_k) || \leq \epsilon = 10^{-8}$. _Note that the code below does not include the latter stop criterion_.
And finally we can compare our solution for $\beta$ with the analytic result given by
$\beta= (X^TX)^{-1} X^T \mathbf{y}$.
@@ -871,12 +871,12 @@ from matplotlib.ticker import LinearLocator, FormatStrFormatter
import sys
# the number of datapoints
m = 100
x = 2*np.random.rand(m,1)
y = 4+3*x+np.random.randn(m,1)
n = 100
x = 2*np.random.rand(n,1)
y = 4+3*x+np.random.randn(n,1)
xb = np.c_[np.ones((m,1)), x]
beta_linreg = np.linalg.inv(xb.T.dot(xb)).dot(xb.T).dot(y)
X = np.c_[np.ones((n,1)), x]
beta_linreg = np.linalg.inv(X.T @ X) @ X.T @ y
print(beta_linreg)
beta = np.random.randn(2,1)
@@ -884,8 +884,8 @@ eta = 0.1
Niterations = 1000
for iter in range(Niterations):
gradients = 2.0/m*xb.T.dot(xb.dot(beta)-y)
beta -= eta*gradients
gradient = (2.0/n)*X.T @ (X @ beta-y)
beta -= eta*gradient
print(beta)
xnew = np.array([[0],[2]])