lars/FYS-STK4155
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

small update on codes

Browse Source
This commit is contained in:
Morten Hjorth-Jensen
2021-09-23 10:45:55 +02:00
parent ee68e6816a
commit ad3594d65e
9 changed files with 197 additions and 82 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/pub/week38/html/._week38-bs014.html
+17
View File
@@ -527,6 +527,23 @@ intercept. Not including the intercept in the fit, means that the
regularization term does not include \( \beta_0 \). For different values
of \( \lambda \), this may lead to differeing MSE values.
<p>
To remind the reader, the regularization term, with the intercept in Ridge regression is given by
$$
\lambda \vert\vert \boldsymbol{\beta} \vert\vert_2^2 = \lambda \sum_{j=0}^{p-1}\beta_j^2,
$$
but when we take out the intercept, this equation becomes
$$
\lambda \vert\vert \boldsymbol{\beta} \vert\vert_2^2 = \lambda \sum_{j=1}^{p-1}\beta_j^2.
$$
<p>
For Lasso regression we have
$$
\lambda \vert\vert \boldsymbol{\beta} \vert\vert_1 = \lambda \sum_{j=1}^{p-1}\vert\beta_j\vert.
$$
<p>
<p>
<!-- navigation buttons at the bottom of the page -->
doc/pub/week38/html/._week38-bs015.html
+6 -5
View File
@@ -421,7 +421,7 @@ MathJax.Hub.Config({
<h2 id="code-examples" class="anchor">Code Examples </h2>
<p>
Armed with this wisdom, we attempt first simply set the intercept equal to <b>False</b> in our implementation of Ridge regression for yet another vanilla data set.
Armed with this wisdom, we attempt first to simply set the intercept equal to <b>False</b> in our implementation of Ridge regression for our well-known vanilla data set.
<p>
@@ -456,10 +456,10 @@ X_train, X_test, y_train, y_test <span style="color: #666666">=</span> train_tes
p <span style="color: #666666">=</span> Maxpolydegree
I <span style="color: #666666">=</span> np<span style="color: #666666">.</span>eye(p,p)
<span style="color: #408080; font-style: italic"># Decide which values of lambda to use</span>
nlambdas <span style="color: #666666">=</span> <span style="color: #666666">4</span>
nlambdas <span style="color: #666666">=</span> <span style="color: #666666">6</span>
MSEOwnRidgePredict <span style="color: #666666">=</span> np<span style="color: #666666">.</span>zeros(nlambdas)
MSERidgePredict <span style="color: #666666">=</span> np<span style="color: #666666">.</span>zeros(nlambdas)
lambdas <span style="color: #666666">=</span> np<span style="color: #666666">.</span>logspace(<span style="color: #666666">-4</span>, <span style="color: #666666">4</span>, nlambdas)
lambdas <span style="color: #666666">=</span> np<span style="color: #666666">.</span>logspace(<span style="color: #666666">-4</span>, <span style="color: #666666">2</span>, nlambdas)
<span style="color: #008000; font-weight: bold">for</span> i <span style="color: #AA22FF; font-weight: bold">in</span> <span style="color: #008000">range</span>(nlambdas):
lmb <span style="color: #666666">=</span> lambdas[i]
OwnRidgeBeta <span style="color: #666666">=</span> np<span style="color: #666666">.</span>linalg<span style="color: #666666">.</span>pinv(X_train<span style="color: #666666">.</span>T <span style="color: #666666">@</span> X_train<span style="color: #666666">+</span>lmb<span style="color: #666666">*</span>I) <span style="color: #666666">@</span> X_train<span style="color: #666666">.</span>T <span style="color: #666666">@</span> y_train
@@ -494,8 +494,9 @@ plt<span style="color: #666666">.</span>show()
</pre></div>
<p>
The results here agree when we force <b>Scikit-Learn</b>'s Ridge function to include the first column in our design matrix.
We see that the results agree very well. What happens if we do not include the intercept in our fit?
Let us see how we can change this code by zero centering.
We see that the results agree very well. Here we have thus explicitely included the intercept column in the design matrix.
What happens if we do not include the intercept in our fit?
Let us see how we can change this code by zero centering (thanks to Stian Bilek for inpouts here).
<p>
<p>
doc/pub/week38/html/._week38-bs016.html
+3 -8
View File
@@ -449,29 +449,24 @@ X <span style="color: #666666">=</span> np<span style="color: #666666">.</span>z
<span style="color: #408080; font-style: italic"># We split the data in test and training data</span>
X_train, X_test, y_train, y_test <span style="color: #666666">=</span> train_test_split(X, y, test_size<span style="color: #666666">=0.2</span>)
<span style="color: #408080; font-style: italic">#For our own implementation, we will need to deal with the intercept by centering the design matrix and the target variable</span>
X_train_mean <span style="color: #666666">=</span> np<span style="color: #666666">.</span>mean(X_train,axis<span style="color: #666666">=0</span>)
<span style="color: #408080; font-style: italic">#Center by removing mean from each feature</span>
X_train_scaled <span style="color: #666666">=</span> X_train <span style="color: #666666">-</span> X_train_mean
X_test_scaled <span style="color: #666666">=</span> X_test <span style="color: #666666">-</span> X_train_mean
<span style="color: #408080; font-style: italic">#The model intercept (called y_scaler) is given by the mean of target variable (IF X is centered)</span>
<span style="color: #408080; font-style: italic">#The model intercept (called y_scaler) is given by the mean of the target variable (IF X is centered)</span>
<span style="color: #408080; font-style: italic">#Remove the intercept from the training data.</span>
y_scaler <span style="color: #666666">=</span> np<span style="color: #666666">.</span>mean(y_train)
y_train_scaled <span style="color: #666666">=</span> y_train <span style="color: #666666">-</span> y_scaler
p <span style="color: #666666">=</span> Maxpolydegree<span style="color: #666666">-1</span>
I <span style="color: #666666">=</span> np<span style="color: #666666">.</span>eye(p,p)
<span style="color: #408080; font-style: italic"># Decide which values of lambda to use</span>
nlambdas <span style="color: #666666">=</span> <span style="color: #666666">4</span>
nlambdas <span style="color: #666666">=</span> <span style="color: #666666">6</span>
MSEOwnRidgePredict <span style="color: #666666">=</span> np<span style="color: #666666">.</span>zeros(nlambdas)
MSERidgePredict <span style="color: #666666">=</span> np<span style="color: #666666">.</span>zeros(nlambdas)
lambdas <span style="color: #666666">=</span> np<span style="color: #666666">.</span>logspace(<span style="color: #666666">-4</span>, <span style="color: #666666">1</span>, nlambdas)
lambdas <span style="color: #666666">=</span> np<span style="color: #666666">.</span>logspace(<span style="color: #666666">-4</span>, <span style="color: #666666">2</span>, nlambdas)
<span style="color: #008000; font-weight: bold">for</span> i <span style="color: #AA22FF; font-weight: bold">in</span> <span style="color: #008000">range</span>(nlambdas):
lmb <span style="color: #666666">=</span> lambdas[i]
OwnRidgeBeta <span style="color: #666666">=</span> np<span style="color: #666666">.</span>linalg<span style="color: #666666">.</span>pinv(X_train_scaled<span style="color: #666666">.</span>T <span style="color: #666666">@</span> X_train_scaled<span style="color: #666666">+</span>lmb<span style="color: #666666">*</span>I) <span style="color: #666666">@</span> X_train_scaled<span style="color: #666666">.</span>T <span style="color: #666666">@</span> (y_train_scaled)
doc/pub/week38/html/week38-reveal.html
+32 -13
View File
@@ -779,6 +779,29 @@ meaning that the MSE can be penalized by the value of the
intercept. Not including the intercept in the fit, means that the
regularization term does not include \( \beta_0 \). For different values
of \( \lambda \), this may lead to differeing MSE values.
<p>
To remind the reader, the regularization term, with the intercept in Ridge regression is given by
<p>&nbsp;<br>
$$
\lambda \vert\vert \boldsymbol{\beta} \vert\vert_2^2 = \lambda \sum_{j=0}^{p-1}\beta_j^2,
$$
<p>&nbsp;<br>
but when we take out the intercept, this equation becomes
<p>&nbsp;<br>
$$
\lambda \vert\vert \boldsymbol{\beta} \vert\vert_2^2 = \lambda \sum_{j=1}^{p-1}\beta_j^2.
$$
<p>&nbsp;<br>
<p>
For Lasso regression we have
<p>&nbsp;<br>
$$
\lambda \vert\vert \boldsymbol{\beta} \vert\vert_1 = \lambda \sum_{j=1}^{p-1}\vert\beta_j\vert.
$$
<p>&nbsp;<br>
</section>
@@ -786,7 +809,7 @@ of \( \lambda \), this may lead to differeing MSE values.
<h2 id="code-examples">Code Examples </h2>
<p>
Armed with this wisdom, we attempt first simply set the intercept equal to <b>False</b> in our implementation of Ridge regression for yet another vanilla data set.
Armed with this wisdom, we attempt first to simply set the intercept equal to <b>False</b> in our implementation of Ridge regression for our well-known vanilla data set.
<p>
@@ -821,10 +844,10 @@ X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=<span style=
p = Maxpolydegree
I = np.eye(p,p)
<span style="color: #228B22"># Decide which values of lambda to use</span>
nlambdas = <span style="color: #B452CD">4</span>
nlambdas = <span style="color: #B452CD">6</span>
MSEOwnRidgePredict = np.zeros(nlambdas)
MSERidgePredict = np.zeros(nlambdas)
lambdas = np.logspace(-<span style="color: #B452CD">4</span>, <span style="color: #B452CD">4</span>, nlambdas)
lambdas = np.logspace(-<span style="color: #B452CD">4</span>, <span style="color: #B452CD">2</span>, nlambdas)
<span style="color: #8B008B; font-weight: bold">for</span> i <span style="color: #8B008B">in</span> <span style="color: #658b00">range</span>(nlambdas):
lmb = lambdas[i]
OwnRidgeBeta = np.linalg.pinv(X_train.T @ X_train+lmb*I) @ X_train.T @ y_train
@@ -859,8 +882,9 @@ plt.show()
</pre></div>
<p>
The results here agree when we force <b>Scikit-Learn</b>'s Ridge function to include the first column in our design matrix.
We see that the results agree very well. What happens if we do not include the intercept in our fit?
Let us see how we can change this code by zero centering.
We see that the results agree very well. Here we have thus explicitely included the intercept column in the design matrix.
What happens if we do not include the intercept in our fit?
Let us see how we can change this code by zero centering (thanks to Stian Bilek for inpouts here).
</section>
@@ -896,29 +920,24 @@ X = np.zeros((n,Maxpolydegree-<span style="color: #B452CD">1</span>))
<span style="color: #228B22"># We split the data in test and training data</span>
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=<span style="color: #B452CD">0.2</span>)
<span style="color: #228B22">#For our own implementation, we will need to deal with the intercept by centering the design matrix and the target variable</span>
X_train_mean = np.mean(X_train,axis=<span style="color: #B452CD">0</span>)
<span style="color: #228B22">#Center by removing mean from each feature</span>
X_train_scaled = X_train - X_train_mean
X_test_scaled = X_test - X_train_mean
<span style="color: #228B22">#The model intercept (called y_scaler) is given by the mean of target variable (IF X is centered)</span>
<span style="color: #228B22">#The model intercept (called y_scaler) is given by the mean of the target variable (IF X is centered)</span>
<span style="color: #228B22">#Remove the intercept from the training data.</span>
y_scaler = np.mean(y_train)
y_train_scaled = y_train - y_scaler
p = Maxpolydegree-<span style="color: #B452CD">1</span>
I = np.eye(p,p)
<span style="color: #228B22"># Decide which values of lambda to use</span>
nlambdas = <span style="color: #B452CD">4</span>
nlambdas = <span style="color: #B452CD">6</span>
MSEOwnRidgePredict = np.zeros(nlambdas)
MSERidgePredict = np.zeros(nlambdas)
lambdas = np.logspace(-<span style="color: #B452CD">4</span>, <span style="color: #B452CD">1</span>, nlambdas)
lambdas = np.logspace(-<span style="color: #B452CD">4</span>, <span style="color: #B452CD">2</span>, nlambdas)
<span style="color: #8B008B; font-weight: bold">for</span> i <span style="color: #8B008B">in</span> <span style="color: #658b00">range</span>(nlambdas):
lmb = lambdas[i]
OwnRidgeBeta = np.linalg.pinv(X_train_scaled.T @ X_train_scaled+lmb*I) @ X_train_scaled.T @ (y_train_scaled)
doc/pub/week38/html/week38-solarized.html
+26 -13
View File
@@ -914,13 +914,30 @@ intercept. Not including the intercept in the fit, means that the
regularization term does not include \( \beta_0 \). For different values
of \( \lambda \), this may lead to differeing MSE values.
<p>
To remind the reader, the regularization term, with the intercept in Ridge regression is given by
$$
\lambda \vert\vert \boldsymbol{\beta} \vert\vert_2^2 = \lambda \sum_{j=0}^{p-1}\beta_j^2,
$$
but when we take out the intercept, this equation becomes
$$
\lambda \vert\vert \boldsymbol{\beta} \vert\vert_2^2 = \lambda \sum_{j=1}^{p-1}\beta_j^2.
$$
<p>
For Lasso regression we have
$$
\lambda \vert\vert \boldsymbol{\beta} \vert\vert_1 = \lambda \sum_{j=1}^{p-1}\vert\beta_j\vert.
$$
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="code-examples">Code Examples </h2>
<p>
Armed with this wisdom, we attempt first simply set the intercept equal to <b>False</b> in our implementation of Ridge regression for yet another vanilla data set.
Armed with this wisdom, we attempt first to simply set the intercept equal to <b>False</b> in our implementation of Ridge regression for our well-known vanilla data set.
<p>
@@ -955,10 +972,10 @@ X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=<span style=
p = Maxpolydegree
I = np.eye(p,p)
<span style="color: #228B22"># Decide which values of lambda to use</span>
nlambdas = <span style="color: #B452CD">4</span>
nlambdas = <span style="color: #B452CD">6</span>
MSEOwnRidgePredict = np.zeros(nlambdas)
MSERidgePredict = np.zeros(nlambdas)
lambdas = np.logspace(-<span style="color: #B452CD">4</span>, <span style="color: #B452CD">4</span>, nlambdas)
lambdas = np.logspace(-<span style="color: #B452CD">4</span>, <span style="color: #B452CD">2</span>, nlambdas)
<span style="color: #8B008B; font-weight: bold">for</span> i <span style="color: #8B008B">in</span> <span style="color: #658b00">range</span>(nlambdas):
lmb = lambdas[i]
OwnRidgeBeta = np.linalg.pinv(X_train.T @ X_train+lmb*I) @ X_train.T @ y_train
@@ -993,8 +1010,9 @@ plt.show()
</pre></div>
<p>
The results here agree when we force <b>Scikit-Learn</b>'s Ridge function to include the first column in our design matrix.
We see that the results agree very well. What happens if we do not include the intercept in our fit?
Let us see how we can change this code by zero centering.
We see that the results agree very well. Here we have thus explicitely included the intercept column in the design matrix.
What happens if we do not include the intercept in our fit?
Let us see how we can change this code by zero centering (thanks to Stian Bilek for inpouts here).
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
@@ -1030,29 +1048,24 @@ X = np.zeros((n,Maxpolydegree-<span style="color: #B452CD">1</span>))
<span style="color: #228B22"># We split the data in test and training data</span>
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=<span style="color: #B452CD">0.2</span>)
<span style="color: #228B22">#For our own implementation, we will need to deal with the intercept by centering the design matrix and the target variable</span>
X_train_mean = np.mean(X_train,axis=<span style="color: #B452CD">0</span>)
<span style="color: #228B22">#Center by removing mean from each feature</span>
X_train_scaled = X_train - X_train_mean
X_test_scaled = X_test - X_train_mean
<span style="color: #228B22">#The model intercept (called y_scaler) is given by the mean of target variable (IF X is centered)</span>
<span style="color: #228B22">#The model intercept (called y_scaler) is given by the mean of the target variable (IF X is centered)</span>
<span style="color: #228B22">#Remove the intercept from the training data.</span>
y_scaler = np.mean(y_train)
y_train_scaled = y_train - y_scaler
p = Maxpolydegree-<span style="color: #B452CD">1</span>
I = np.eye(p,p)
<span style="color: #228B22"># Decide which values of lambda to use</span>
nlambdas = <span style="color: #B452CD">4</span>
nlambdas = <span style="color: #B452CD">6</span>
MSEOwnRidgePredict = np.zeros(nlambdas)
MSERidgePredict = np.zeros(nlambdas)
lambdas = np.logspace(-<span style="color: #B452CD">4</span>, <span style="color: #B452CD">1</span>, nlambdas)
lambdas = np.logspace(-<span style="color: #B452CD">4</span>, <span style="color: #B452CD">2</span>, nlambdas)
<span style="color: #8B008B; font-weight: bold">for</span> i <span style="color: #8B008B">in</span> <span style="color: #658b00">range</span>(nlambdas):
lmb = lambdas[i]
OwnRidgeBeta = np.linalg.pinv(X_train_scaled.T @ X_train_scaled+lmb*I) @ X_train_scaled.T @ (y_train_scaled)
doc/pub/week38/html/week38.html
+26 -13
View File
@@ -919,13 +919,30 @@ intercept. Not including the intercept in the fit, means that the
regularization term does not include \( \beta_0 \). For different values
of \( \lambda \), this may lead to differeing MSE values.
<p>
To remind the reader, the regularization term, with the intercept in Ridge regression is given by
$$
\lambda \vert\vert \boldsymbol{\beta} \vert\vert_2^2 = \lambda \sum_{j=0}^{p-1}\beta_j^2,
$$
but when we take out the intercept, this equation becomes
$$
\lambda \vert\vert \boldsymbol{\beta} \vert\vert_2^2 = \lambda \sum_{j=1}^{p-1}\beta_j^2.
$$
<p>
For Lasso regression we have
$$
\lambda \vert\vert \boldsymbol{\beta} \vert\vert_1 = \lambda \sum_{j=1}^{p-1}\vert\beta_j\vert.
$$
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="code-examples">Code Examples </h2>
<p>
Armed with this wisdom, we attempt first simply set the intercept equal to <b>False</b> in our implementation of Ridge regression for yet another vanilla data set.
Armed with this wisdom, we attempt first to simply set the intercept equal to <b>False</b> in our implementation of Ridge regression for our well-known vanilla data set.
<p>
@@ -960,10 +977,10 @@ X_train, X_test, y_train, y_test <span style="color: #666666">=</span> train_tes
p <span style="color: #666666">=</span> Maxpolydegree
I <span style="color: #666666">=</span> np<span style="color: #666666">.</span>eye(p,p)
<span style="color: #408080; font-style: italic"># Decide which values of lambda to use</span>
nlambdas <span style="color: #666666">=</span> <span style="color: #666666">4</span>
nlambdas <span style="color: #666666">=</span> <span style="color: #666666">6</span>
MSEOwnRidgePredict <span style="color: #666666">=</span> np<span style="color: #666666">.</span>zeros(nlambdas)
MSERidgePredict <span style="color: #666666">=</span> np<span style="color: #666666">.</span>zeros(nlambdas)
lambdas <span style="color: #666666">=</span> np<span style="color: #666666">.</span>logspace(<span style="color: #666666">-4</span>, <span style="color: #666666">4</span>, nlambdas)
lambdas <span style="color: #666666">=</span> np<span style="color: #666666">.</span>logspace(<span style="color: #666666">-4</span>, <span style="color: #666666">2</span>, nlambdas)
<span style="color: #008000; font-weight: bold">for</span> i <span style="color: #AA22FF; font-weight: bold">in</span> <span style="color: #008000">range</span>(nlambdas):
lmb <span style="color: #666666">=</span> lambdas[i]
OwnRidgeBeta <span style="color: #666666">=</span> np<span style="color: #666666">.</span>linalg<span style="color: #666666">.</span>pinv(X_train<span style="color: #666666">.</span>T <span style="color: #666666">@</span> X_train<span style="color: #666666">+</span>lmb<span style="color: #666666">*</span>I) <span style="color: #666666">@</span> X_train<span style="color: #666666">.</span>T <span style="color: #666666">@</span> y_train
@@ -998,8 +1015,9 @@ plt<span style="color: #666666">.</span>show()
</pre></div>
<p>
The results here agree when we force <b>Scikit-Learn</b>'s Ridge function to include the first column in our design matrix.
We see that the results agree very well. What happens if we do not include the intercept in our fit?
Let us see how we can change this code by zero centering.
We see that the results agree very well. Here we have thus explicitely included the intercept column in the design matrix.
What happens if we do not include the intercept in our fit?
Let us see how we can change this code by zero centering (thanks to Stian Bilek for inpouts here).
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
@@ -1035,29 +1053,24 @@ X <span style="color: #666666">=</span> np<span style="color: #666666">.</span>z
<span style="color: #408080; font-style: italic"># We split the data in test and training data</span>
X_train, X_test, y_train, y_test <span style="color: #666666">=</span> train_test_split(X, y, test_size<span style="color: #666666">=0.2</span>)
<span style="color: #408080; font-style: italic">#For our own implementation, we will need to deal with the intercept by centering the design matrix and the target variable</span>
X_train_mean <span style="color: #666666">=</span> np<span style="color: #666666">.</span>mean(X_train,axis<span style="color: #666666">=0</span>)
<span style="color: #408080; font-style: italic">#Center by removing mean from each feature</span>
X_train_scaled <span style="color: #666666">=</span> X_train <span style="color: #666666">-</span> X_train_mean
X_test_scaled <span style="color: #666666">=</span> X_test <span style="color: #666666">-</span> X_train_mean
<span style="color: #408080; font-style: italic">#The model intercept (called y_scaler) is given by the mean of target variable (IF X is centered)</span>
<span style="color: #408080; font-style: italic">#The model intercept (called y_scaler) is given by the mean of the target variable (IF X is centered)</span>
<span style="color: #408080; font-style: italic">#Remove the intercept from the training data.</span>
y_scaler <span style="color: #666666">=</span> np<span style="color: #666666">.</span>mean(y_train)
y_train_scaled <span style="color: #666666">=</span> y_train <span style="color: #666666">-</span> y_scaler
p <span style="color: #666666">=</span> Maxpolydegree<span style="color: #666666">-1</span>
I <span style="color: #666666">=</span> np<span style="color: #666666">.</span>eye(p,p)
<span style="color: #408080; font-style: italic"># Decide which values of lambda to use</span>
nlambdas <span style="color: #666666">=</span> <span style="color: #666666">4</span>
nlambdas <span style="color: #666666">=</span> <span style="color: #666666">6</span>
MSEOwnRidgePredict <span style="color: #666666">=</span> np<span style="color: #666666">.</span>zeros(nlambdas)
MSERidgePredict <span style="color: #666666">=</span> np<span style="color: #666666">.</span>zeros(nlambdas)
lambdas <span style="color: #666666">=</span> np<span style="color: #666666">.</span>logspace(<span style="color: #666666">-4</span>, <span style="color: #666666">1</span>, nlambdas)
lambdas <span style="color: #666666">=</span> np<span style="color: #666666">.</span>logspace(<span style="color: #666666">-4</span>, <span style="color: #666666">2</span>, nlambdas)
<span style="color: #008000; font-weight: bold">for</span> i <span style="color: #AA22FF; font-weight: bold">in</span> <span style="color: #008000">range</span>(nlambdas):
lmb <span style="color: #666666">=</span> lambdas[i]
OwnRidgeBeta <span style="color: #666666">=</span> np<span style="color: #666666">.</span>linalg<span style="color: #666666">.</span>pinv(X_train_scaled<span style="color: #666666">.</span>T <span style="color: #666666">@</span> X_train_scaled<span style="color: #666666">+</span>lmb<span style="color: #666666">*</span>I) <span style="color: #666666">@</span> X_train_scaled<span style="color: #666666">.</span>T <span style="color: #666666">@</span> (y_train_scaled)
doc/pub/week38/ipynb/ipynb-week38-src.tar.gz
BIN
View File
Binary file not shown.
doc/pub/week38/ipynb/week38.ipynb
+59 -17
View File
@@ -845,11 +845,57 @@
"regularization term does not include $\\beta_0$. For different values\n",
"of $\\lambda$, this may lead to differeing MSE values.\n",
"\n",
"\n",
"\n",
"To remind the reader, the regularization term, with the intercept in Ridge regression is given by"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\n",
"\\lambda \\vert\\vert \\boldsymbol{\\beta} \\vert\\vert_2^2 = \\lambda \\sum_{j=0}^{p-1}\\beta_j^2,\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"but when we take out the intercept, this equation becomes"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\n",
"\\lambda \\vert\\vert \\boldsymbol{\\beta} \\vert\\vert_2^2 = \\lambda \\sum_{j=1}^{p-1}\\beta_j^2.\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"For Lasso regression we have"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\n",
"\\lambda \\vert\\vert \\boldsymbol{\\beta} \\vert\\vert_1 = \\lambda \\sum_{j=1}^{p-1}\\vert\\beta_j\\vert.\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Code Examples\n",
"\n",
"Armed with this wisdom, we attempt first simply set the intercept equal to **False** in our implementation of Ridge regression for yet another vanilla data set."
"Armed with this wisdom, we attempt first to simply set the intercept equal to **False** in our implementation of Ridge regression for our well-known vanilla data set."
]
},
{
@@ -891,10 +937,10 @@
"p = Maxpolydegree\n",
"I = np.eye(p,p)\n",
"# Decide which values of lambda to use\n",
"nlambdas = 4\n",
"nlambdas = 6\n",
"MSEOwnRidgePredict = np.zeros(nlambdas)\n",
"MSERidgePredict = np.zeros(nlambdas)\n",
"lambdas = np.logspace(-4, 4, nlambdas)\n",
"lambdas = np.logspace(-4, 2, nlambdas)\n",
"for i in range(nlambdas):\n",
" lmb = lambdas[i]\n",
" OwnRidgeBeta = np.linalg.pinv(X_train.T @ X_train+lmb*I) @ X_train.T @ y_train\n",
@@ -933,8 +979,9 @@
"metadata": {},
"source": [
"The results here agree when we force **Scikit-Learn**'s Ridge function to include the first column in our design matrix.\n",
"We see that the results agree very well. What happens if we do not include the intercept in our fit?\n",
"Let us see how we can change this code by zero centering.\n",
"We see that the results agree very well. Here we have thus explicitely included the intercept column in the design matrix.\n",
"What happens if we do not include the intercept in our fit?\n",
"Let us see how we can change this code by zero centering (thanks to Stian Bilek for inpouts here).\n",
"\n",
"## Taking out the mean"
]
@@ -975,29 +1022,24 @@
"# We split the data in test and training data\n",
"X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"#For our own implementation, we will need to deal with the intercept by centering the design matrix and the target variable\n",
"X_train_mean = np.mean(X_train,axis=0)\n",
"#Center by removing mean from each feature\n",
"X_train_scaled = X_train - X_train_mean \n",
"X_test_scaled = X_test - X_train_mean\n",
"#The model intercept (called y_scaler) is given by the mean of target variable (IF X is centered)\n",
"#The model intercept (called y_scaler) is given by the mean of the target variable (IF X is centered)\n",
"#Remove the intercept from the training data.\n",
"y_scaler = np.mean(y_train) \n",
"y_train_scaled = y_train - y_scaler \n",
"\n",
"\n",
"p = Maxpolydegree-1\n",
"I = np.eye(p,p)\n",
"# Decide which values of lambda to use\n",
"nlambdas = 4\n",
"nlambdas = 6\n",
"MSEOwnRidgePredict = np.zeros(nlambdas)\n",
"MSERidgePredict = np.zeros(nlambdas)\n",
"\n",
"lambdas = np.logspace(-4, 1, nlambdas)\n",
"lambdas = np.logspace(-4, 2, nlambdas)\n",
"for i in range(nlambdas):\n",
" lmb = lambdas[i]\n",
" OwnRidgeBeta = np.linalg.pinv(X_train_scaled.T @ X_train_scaled+lmb*I) @ X_train_scaled.T @ (y_train_scaled)\n",
@@ -1709,8 +1751,8 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"3\n",
"7\n",
"4\n",
"0\n",
" \n",
"<\n",
"<\n",
doc/src/week38/week38.do.txt
+28 -13
View File
@@ -559,12 +559,31 @@ intercept. Not including the intercept in the fit, means that the
regularization term does not include $\beta_0$. For different values
of $\lambda$, this may lead to differeing MSE values.
To remind the reader, the regularization term, with the intercept in Ridge regression is given by
!bt
\[
\lambda \vert\vert \bm{\beta} \vert\vert_2^2 = \lambda \sum_{j=0}^{p-1}\beta_j^2,
\]
!et
but when we take out the intercept, this equation becomes
!bt
\[
\lambda \vert\vert \bm{\beta} \vert\vert_2^2 = \lambda \sum_{j=1}^{p-1}\beta_j^2.
\]
!et
For Lasso regression we have
!bt
\[
\lambda \vert\vert \bm{\beta} \vert\vert_1 = \lambda \sum_{j=1}^{p-1}\vert\beta_j\vert.
\]
!et
!split
===== Code Examples =====
Armed with this wisdom, we attempt first simply set the intercept equal to _False_ in our implementation of Ridge regression for yet another vanilla data set.
Armed with this wisdom, we attempt first to simply set the intercept equal to _False_ in our implementation of Ridge regression for our well-known vanilla data set.
!bc pycod
import numpy as np
@@ -597,10 +616,10 @@ X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)
p = Maxpolydegree
I = np.eye(p,p)
# Decide which values of lambda to use
nlambdas = 4
nlambdas = 6
MSEOwnRidgePredict = np.zeros(nlambdas)
MSERidgePredict = np.zeros(nlambdas)
lambdas = np.logspace(-4, 4, nlambdas)
lambdas = np.logspace(-4, 2, nlambdas)
for i in range(nlambdas):
lmb = lambdas[i]
OwnRidgeBeta = np.linalg.pinv(X_train.T @ X_train+lmb*I) @ X_train.T @ y_train
@@ -636,8 +655,9 @@ plt.show()
!ec
The results here agree when we force _Scikit-Learn_'s Ridge function to include the first column in our design matrix.
We see that the results agree very well. What happens if we do not include the intercept in our fit?
Let us see how we can change this code by zero centering.
We see that the results agree very well. Here we have thus explicitely included the intercept column in the design matrix.
What happens if we do not include the intercept in our fit?
Let us see how we can change this code by zero centering (thanks to Stian Bilek for inpouts here).
!split
===== Taking out the mean =====
@@ -669,29 +689,24 @@ for degree in range(1,Maxpolydegree): #No intercept column
# We split the data in test and training data
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)
#For our own implementation, we will need to deal with the intercept by centering the design matrix and the target variable
X_train_mean = np.mean(X_train,axis=0)
#Center by removing mean from each feature
X_train_scaled = X_train - X_train_mean
X_test_scaled = X_test - X_train_mean
#The model intercept (called y_scaler) is given by the mean of target variable (IF X is centered)
#The model intercept (called y_scaler) is given by the mean of the target variable (IF X is centered)
#Remove the intercept from the training data.
y_scaler = np.mean(y_train)
y_train_scaled = y_train - y_scaler
p = Maxpolydegree-1
I = np.eye(p,p)
# Decide which values of lambda to use
nlambdas = 4
nlambdas = 6
MSEOwnRidgePredict = np.zeros(nlambdas)
MSERidgePredict = np.zeros(nlambdas)
lambdas = np.logspace(-4, 1, nlambdas)
lambdas = np.logspace(-4, 2, nlambdas)
for i in range(nlambdas):
lmb = lambdas[i]
OwnRidgeBeta = np.linalg.pinv(X_train_scaled.T @ X_train_scaled+lmb*I) @ X_train_scaled.T @ (y_train_scaled)