lars/FYS-STK4155
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

update week 36

Browse Source
This commit is contained in:
Morten Hjorth-Jensen
2025-09-03 13:40:18 +02:00
parent 1a1618aa74
commit abd1d73a8f
15 changed files with 668 additions and 119 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/LectureNotes/_build/.doctrees/environment.pickle
BIN
View File
Binary file not shown.
doc/LectureNotes/_build/.doctrees/exercisesweek37.doctree
BIN
View File
Binary file not shown.
doc/LectureNotes/_build/html/_sources/exercisesweek37.ipynb
+30 -29
View File
@@ -2,7 +2,7 @@
"cells": [
{
"cell_type": "markdown",
"id": "1b638450",
"id": "1b941c35",
"metadata": {
"editable": true
},
@@ -14,7 +14,7 @@
},
{
"cell_type": "markdown",
"id": "60060788",
"id": "dc05b096",
"metadata": {
"editable": true
},
@@ -27,7 +27,7 @@
},
{
"cell_type": "markdown",
"id": "7cdd88e4",
"id": "2cf07405",
"metadata": {
"editable": true
},
@@ -37,7 +37,7 @@
"After having completed these exercises you will have:\n",
"1. Your own code for the implementation of the simplest gradient descent approach applied to ordinary least squares (OLS) and Ridge regression\n",
"\n",
"2. Be able to compare the analytical expressions for OLS and Rudge regression with the gradient descent approach\n",
"2. Be able to compare the analytical expressions for OLS and Ridge regression with the gradient descent approach\n",
"\n",
"3. Explore the role of the learning rate in the gradient descent approach and the hyperparameter $\\lambda$ in Ridge regression\n",
"\n",
@@ -46,7 +46,7 @@
},
{
"cell_type": "markdown",
"id": "0328fa2a",
"id": "3c139edb",
"metadata": {
"editable": true
},
@@ -58,7 +58,7 @@
},
{
"cell_type": "markdown",
"id": "ac760265",
"id": "aad4cfac",
"metadata": {
"editable": true
},
@@ -70,7 +70,7 @@
},
{
"cell_type": "markdown",
"id": "8d4b0753",
"id": "6682282f",
"metadata": {
"editable": true
},
@@ -83,7 +83,7 @@
},
{
"cell_type": "markdown",
"id": "4517d311",
"id": "89e2f4c4",
"metadata": {
"editable": true
},
@@ -99,7 +99,7 @@
},
{
"cell_type": "markdown",
"id": "da1834a9",
"id": "b06d4e53",
"metadata": {
"editable": true
},
@@ -120,7 +120,7 @@
{
"cell_type": "code",
"execution_count": 1,
"id": "590b2fb0",
"id": "63796480",
"metadata": {
"collapsed": false,
"editable": true
@@ -140,7 +140,7 @@
},
{
"cell_type": "markdown",
"id": "24dd92fc",
"id": "80748600",
"metadata": {
"editable": true
},
@@ -156,7 +156,7 @@
},
{
"cell_type": "markdown",
"id": "4ba80e16",
"id": "92751e5f",
"metadata": {
"editable": true
},
@@ -168,18 +168,18 @@
},
{
"cell_type": "markdown",
"id": "be65f56f",
"id": "aedfbd7a",
"metadata": {
"editable": true
},
"source": [
"## Exercise 3, use the analytical formulae for OLS and Ridge regression to find the optimal paramters $\\boldsymbol{\\theta}$"
"## Exercise 3, using the analytical formulae for OLS and Ridge regression to find the optimal paramters $\\boldsymbol{\\theta}$"
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "c265677e",
"id": "5d1288fa",
"metadata": {
"collapsed": false,
"editable": true
@@ -200,7 +200,7 @@
},
{
"cell_type": "markdown",
"id": "b989efb9",
"id": "628f5e89",
"metadata": {
"editable": true
},
@@ -214,7 +214,7 @@
},
{
"cell_type": "markdown",
"id": "17b08af7",
"id": "f115ba4e",
"metadata": {
"editable": true
},
@@ -226,7 +226,7 @@
},
{
"cell_type": "markdown",
"id": "6acd4708",
"id": "a9b5189c",
"metadata": {
"editable": true
},
@@ -238,7 +238,7 @@
},
{
"cell_type": "markdown",
"id": "50579422",
"id": "a3969ff6",
"metadata": {
"editable": true
},
@@ -258,7 +258,7 @@
{
"cell_type": "code",
"execution_count": 3,
"id": "c57cc917",
"id": "34d87303",
"metadata": {
"collapsed": false,
"editable": true
@@ -277,13 +277,14 @@
"cost_history = np.zeros(num_iters)\n",
"\n",
"# Gradient descent loop\n",
"m = n_samples # number of examples\n",
"m = n_samples # number of data points\n",
"for t in range(num_iters):\n",
" # Compute prediction error\n",
" error = X_norm.dot(theta) - y_centered \n",
" # Compute cost for OLS and Ridge (MSE + regularization for Ridge) for monitoring\n",
" cost_OLS = ?\n",
" cost_Ridge = ?\n",
" # You could add a history for both methods (optional)\n",
" cost_history[t] = ?\n",
" # Compute gradients for OSL and Ridge\n",
" grad_OLS = ?\n",
@@ -301,7 +302,7 @@
},
{
"cell_type": "markdown",
"id": "e2654903",
"id": "989f70bb",
"metadata": {
"editable": true
},
@@ -313,19 +314,19 @@
},
{
"cell_type": "markdown",
"id": "7e7e7de6",
"id": "370b2dad",
"metadata": {
"editable": true
},
"source": [
"### 4b)\n",
"\n",
"Try to add a stopping parameter as function of the number iterations. How would you define a stopping criterion?"
"Try to add a stopping parameter as function of the number iterations and the difference between the new and old $\\theta$ values. How would you define a stopping criterion?"
]
},
{
"cell_type": "markdown",
"id": "75542708",
"id": "ef197cd7",
"metadata": {
"editable": true
},
@@ -351,7 +352,7 @@
{
"cell_type": "code",
"execution_count": 4,
"id": "06077986",
"id": "4ccc2f65",
"metadata": {
"collapsed": false,
"editable": true
@@ -380,7 +381,7 @@
},
{
"cell_type": "markdown",
"id": "86d46505",
"id": "00e279ef",
"metadata": {
"editable": true
},
@@ -394,7 +395,7 @@
},
{
"cell_type": "markdown",
"id": "f1d2f848",
"id": "c910b3f4",
"metadata": {
"editable": true
},
@@ -406,7 +407,7 @@
},
{
"cell_type": "markdown",
"id": "f1c91e84",
"id": "89e6e040",
"metadata": {
"editable": true
},
doc/LectureNotes/_build/html/exercisesweek37.html
+8 -7
View File
@@ -389,7 +389,7 @@ document.write(`
</ul>
</li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#exercise-2-calculate-the-gradients">Exercise 2, calculate the gradients</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#exercise-3-use-the-analytical-formulae-for-ols-and-ridge-regression-to-find-the-optimal-paramters-boldsymbol-theta">Exercise 3, use the analytical formulae for OLS and Ridge regression to find the optimal paramters <span class="math notranslate nohighlight">\(\boldsymbol{\theta}\)</span></a><ul class="nav section-nav flex-column">
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#exercise-3-using-the-analytical-formulae-for-ols-and-ridge-regression-to-find-the-optimal-paramters-boldsymbol-theta">Exercise 3, using the analytical formulae for OLS and Ridge regression to find the optimal paramters <span class="math notranslate nohighlight">\(\boldsymbol{\theta}\)</span></a><ul class="nav section-nav flex-column">
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#id1">3a)</a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#b">3b)</a></li>
</ul>
@@ -422,7 +422,7 @@ doconce format html exercisesweek37.do.txt -->
<p>After having completed these exercises you will have:</p>
<ol class="arabic simple">
<li><p>Your own code for the implementation of the simplest gradient descent approach applied to ordinary least squares (OLS) and Ridge regression</p></li>
<li><p>Be able to compare the analytical expressions for OLS and Rudge regression with the gradient descent approach</p></li>
<li><p>Be able to compare the analytical expressions for OLS and Ridge regression with the gradient descent approach</p></li>
<li><p>Explore the role of the learning rate in the gradient descent approach and the hyperparameter <span class="math notranslate nohighlight">\(\lambda\)</span> in Ridge regression</p></li>
<li><p>Scale the data properly</p></li>
</ol>
@@ -482,8 +482,8 @@ same scale).</p>
<h2>Exercise 2, calculate the gradients<a class="headerlink" href="#exercise-2-calculate-the-gradients" title="Link to this heading">#</a></h2>
<p>Find the gradients for OLS and Ridge regression using the mean-squared error as cost/loss function.</p>
</section>
<section id="exercise-3-use-the-analytical-formulae-for-ols-and-ridge-regression-to-find-the-optimal-paramters-boldsymbol-theta">
<h2>Exercise 3, use the analytical formulae for OLS and Ridge regression to find the optimal paramters <span class="math notranslate nohighlight">\(\boldsymbol{\theta}\)</span><a class="headerlink" href="#exercise-3-use-the-analytical-formulae-for-ols-and-ridge-regression-to-find-the-optimal-paramters-boldsymbol-theta" title="Link to this heading">#</a></h2>
<section id="exercise-3-using-the-analytical-formulae-for-ols-and-ridge-regression-to-find-the-optimal-paramters-boldsymbol-theta">
<h2>Exercise 3, using the analytical formulae for OLS and Ridge regression to find the optimal paramters <span class="math notranslate nohighlight">\(\boldsymbol{\theta}\)</span><a class="headerlink" href="#exercise-3-using-the-analytical-formulae-for-ols-and-ridge-regression-to-find-the-optimal-paramters-boldsymbol-theta" title="Link to this heading">#</a></h2>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span># Set regularization parameter, either a single value or a vector of values
@@ -537,13 +537,14 @@ theta = np.zeros(n_features)
cost_history = np.zeros(num_iters)
# Gradient descent loop
m = n_samples # number of examples
m = n_samples # number of data points
for t in range(num_iters):
# Compute prediction error
error = X_norm.dot(theta) - y_centered
# Compute cost for OLS and Ridge (MSE + regularization for Ridge) for monitoring
cost_OLS = ?
cost_Ridge = ?
# You could add a history for both methods (optional)
cost_history[t] = ?
# Compute gradients for OSL and Ridge
grad_OLS = ?
@@ -567,7 +568,7 @@ print(&quot;Gradient Descent Ridge coefficients:&quot;, theta_gdRidge)
</section>
<section id="id3">
<h3>4b)<a class="headerlink" href="#id3" title="Link to this heading">#</a></h3>
<p>Try to add a stopping parameter as function of the number iterations. How would you define a stopping criterion?</p>
<p>Try to add a stopping parameter as function of the number iterations and the difference between the new and old <span class="math notranslate nohighlight">\(\theta\)</span> values. How would you define a stopping criterion?</p>
</section>
</section>
<section id="exercise-5-ridge-regression-and-a-new-synthetic-dataset">
@@ -697,7 +698,7 @@ should be in the same ballpark. Which method (OLS or Ridge) gives the best resu
</ul>
</li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#exercise-2-calculate-the-gradients">Exercise 2, calculate the gradients</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#exercise-3-use-the-analytical-formulae-for-ols-and-ridge-regression-to-find-the-optimal-paramters-boldsymbol-theta">Exercise 3, use the analytical formulae for OLS and Ridge regression to find the optimal paramters <span class="math notranslate nohighlight">\(\boldsymbol{\theta}\)</span></a><ul class="nav section-nav flex-column">
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#exercise-3-using-the-analytical-formulae-for-ols-and-ridge-regression-to-find-the-optimal-paramters-boldsymbol-theta">Exercise 3, using the analytical formulae for OLS and Ridge regression to find the optimal paramters <span class="math notranslate nohighlight">\(\boldsymbol{\theta}\)</span></a><ul class="nav section-nav flex-column">
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#id1">3a)</a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#b">3b)</a></li>
</ul>
doc/LectureNotes/_build/html/searchindex.js
+1 -1
View File
File diff suppressed because one or more lines are too long
doc/LectureNotes/_build/jupyter_execute/exercisesweek37.ipynb
+30 -29
View File
@@ -2,7 +2,7 @@
"cells": [
{
"cell_type": "markdown",
"id": "1b638450",
"id": "1b941c35",
"metadata": {
"editable": true
},
@@ -14,7 +14,7 @@
},
{
"cell_type": "markdown",
"id": "60060788",
"id": "dc05b096",
"metadata": {
"editable": true
},
@@ -27,7 +27,7 @@
},
{
"cell_type": "markdown",
"id": "7cdd88e4",
"id": "2cf07405",
"metadata": {
"editable": true
},
@@ -37,7 +37,7 @@
"After having completed these exercises you will have:\n",
"1. Your own code for the implementation of the simplest gradient descent approach applied to ordinary least squares (OLS) and Ridge regression\n",
"\n",
"2. Be able to compare the analytical expressions for OLS and Rudge regression with the gradient descent approach\n",
"2. Be able to compare the analytical expressions for OLS and Ridge regression with the gradient descent approach\n",
"\n",
"3. Explore the role of the learning rate in the gradient descent approach and the hyperparameter $\\lambda$ in Ridge regression\n",
"\n",
@@ -46,7 +46,7 @@
},
{
"cell_type": "markdown",
"id": "0328fa2a",
"id": "3c139edb",
"metadata": {
"editable": true
},
@@ -58,7 +58,7 @@
},
{
"cell_type": "markdown",
"id": "ac760265",
"id": "aad4cfac",
"metadata": {
"editable": true
},
@@ -70,7 +70,7 @@
},
{
"cell_type": "markdown",
"id": "8d4b0753",
"id": "6682282f",
"metadata": {
"editable": true
},
@@ -83,7 +83,7 @@
},
{
"cell_type": "markdown",
"id": "4517d311",
"id": "89e2f4c4",
"metadata": {
"editable": true
},
@@ -99,7 +99,7 @@
},
{
"cell_type": "markdown",
"id": "da1834a9",
"id": "b06d4e53",
"metadata": {
"editable": true
},
@@ -120,7 +120,7 @@
{
"cell_type": "code",
"execution_count": 1,
"id": "590b2fb0",
"id": "63796480",
"metadata": {
"collapsed": false,
"editable": true
@@ -140,7 +140,7 @@
},
{
"cell_type": "markdown",
"id": "24dd92fc",
"id": "80748600",
"metadata": {
"editable": true
},
@@ -156,7 +156,7 @@
},
{
"cell_type": "markdown",
"id": "4ba80e16",
"id": "92751e5f",
"metadata": {
"editable": true
},
@@ -168,18 +168,18 @@
},
{
"cell_type": "markdown",
"id": "be65f56f",
"id": "aedfbd7a",
"metadata": {
"editable": true
},
"source": [
"## Exercise 3, use the analytical formulae for OLS and Ridge regression to find the optimal paramters $\\boldsymbol{\\theta}$"
"## Exercise 3, using the analytical formulae for OLS and Ridge regression to find the optimal paramters $\\boldsymbol{\\theta}$"
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "c265677e",
"id": "5d1288fa",
"metadata": {
"collapsed": false,
"editable": true
@@ -200,7 +200,7 @@
},
{
"cell_type": "markdown",
"id": "b989efb9",
"id": "628f5e89",
"metadata": {
"editable": true
},
@@ -214,7 +214,7 @@
},
{
"cell_type": "markdown",
"id": "17b08af7",
"id": "f115ba4e",
"metadata": {
"editable": true
},
@@ -226,7 +226,7 @@
},
{
"cell_type": "markdown",
"id": "6acd4708",
"id": "a9b5189c",
"metadata": {
"editable": true
},
@@ -238,7 +238,7 @@
},
{
"cell_type": "markdown",
"id": "50579422",
"id": "a3969ff6",
"metadata": {
"editable": true
},
@@ -258,7 +258,7 @@
{
"cell_type": "code",
"execution_count": 3,
"id": "c57cc917",
"id": "34d87303",
"metadata": {
"collapsed": false,
"editable": true
@@ -277,13 +277,14 @@
"cost_history = np.zeros(num_iters)\n",
"\n",
"# Gradient descent loop\n",
"m = n_samples # number of examples\n",
"m = n_samples # number of data points\n",
"for t in range(num_iters):\n",
" # Compute prediction error\n",
" error = X_norm.dot(theta) - y_centered \n",
" # Compute cost for OLS and Ridge (MSE + regularization for Ridge) for monitoring\n",
" cost_OLS = ?\n",
" cost_Ridge = ?\n",
" # You could add a history for both methods (optional)\n",
" cost_history[t] = ?\n",
" # Compute gradients for OSL and Ridge\n",
" grad_OLS = ?\n",
@@ -301,7 +302,7 @@
},
{
"cell_type": "markdown",
"id": "e2654903",
"id": "989f70bb",
"metadata": {
"editable": true
},
@@ -313,19 +314,19 @@
},
{
"cell_type": "markdown",
"id": "7e7e7de6",
"id": "370b2dad",
"metadata": {
"editable": true
},
"source": [
"### 4b)\n",
"\n",
"Try to add a stopping parameter as function of the number iterations. How would you define a stopping criterion?"
"Try to add a stopping parameter as function of the number iterations and the difference between the new and old $\\theta$ values. How would you define a stopping criterion?"
]
},
{
"cell_type": "markdown",
"id": "75542708",
"id": "ef197cd7",
"metadata": {
"editable": true
},
@@ -351,7 +352,7 @@
{
"cell_type": "code",
"execution_count": 4,
"id": "06077986",
"id": "4ccc2f65",
"metadata": {
"collapsed": false,
"editable": true
@@ -380,7 +381,7 @@
},
{
"cell_type": "markdown",
"id": "86d46505",
"id": "00e279ef",
"metadata": {
"editable": true
},
@@ -394,7 +395,7 @@
},
{
"cell_type": "markdown",
"id": "f1d2f848",
"id": "c910b3f4",
"metadata": {
"editable": true
},
@@ -406,7 +407,7 @@
},
{
"cell_type": "markdown",
"id": "f1c91e84",
"id": "89e6e040",
"metadata": {
"editable": true
},
doc/LectureNotes/exercisesweek37.ipynb
+30 -29
View File
@@ -2,7 +2,7 @@
"cells": [
{
"cell_type": "markdown",
"id": "1b638450",
"id": "1b941c35",
"metadata": {
"editable": true
},
@@ -14,7 +14,7 @@
},
{
"cell_type": "markdown",
"id": "60060788",
"id": "dc05b096",
"metadata": {
"editable": true
},
@@ -27,7 +27,7 @@
},
{
"cell_type": "markdown",
"id": "7cdd88e4",
"id": "2cf07405",
"metadata": {
"editable": true
},
@@ -37,7 +37,7 @@
"After having completed these exercises you will have:\n",
"1. Your own code for the implementation of the simplest gradient descent approach applied to ordinary least squares (OLS) and Ridge regression\n",
"\n",
"2. Be able to compare the analytical expressions for OLS and Rudge regression with the gradient descent approach\n",
"2. Be able to compare the analytical expressions for OLS and Ridge regression with the gradient descent approach\n",
"\n",
"3. Explore the role of the learning rate in the gradient descent approach and the hyperparameter $\\lambda$ in Ridge regression\n",
"\n",
@@ -46,7 +46,7 @@
},
{
"cell_type": "markdown",
"id": "0328fa2a",
"id": "3c139edb",
"metadata": {
"editable": true
},
@@ -58,7 +58,7 @@
},
{
"cell_type": "markdown",
"id": "ac760265",
"id": "aad4cfac",
"metadata": {
"editable": true
},
@@ -70,7 +70,7 @@
},
{
"cell_type": "markdown",
"id": "8d4b0753",
"id": "6682282f",
"metadata": {
"editable": true
},
@@ -83,7 +83,7 @@
},
{
"cell_type": "markdown",
"id": "4517d311",
"id": "89e2f4c4",
"metadata": {
"editable": true
},
@@ -99,7 +99,7 @@
},
{
"cell_type": "markdown",
"id": "da1834a9",
"id": "b06d4e53",
"metadata": {
"editable": true
},
@@ -120,7 +120,7 @@
{
"cell_type": "code",
"execution_count": 1,
"id": "590b2fb0",
"id": "63796480",
"metadata": {
"collapsed": false,
"editable": true
@@ -140,7 +140,7 @@
},
{
"cell_type": "markdown",
"id": "24dd92fc",
"id": "80748600",
"metadata": {
"editable": true
},
@@ -156,7 +156,7 @@
},
{
"cell_type": "markdown",
"id": "4ba80e16",
"id": "92751e5f",
"metadata": {
"editable": true
},
@@ -168,18 +168,18 @@
},
{
"cell_type": "markdown",
"id": "be65f56f",
"id": "aedfbd7a",
"metadata": {
"editable": true
},
"source": [
"## Exercise 3, use the analytical formulae for OLS and Ridge regression to find the optimal paramters $\\boldsymbol{\\theta}$"
"## Exercise 3, using the analytical formulae for OLS and Ridge regression to find the optimal paramters $\\boldsymbol{\\theta}$"
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "c265677e",
"id": "5d1288fa",
"metadata": {
"collapsed": false,
"editable": true
@@ -200,7 +200,7 @@
},
{
"cell_type": "markdown",
"id": "b989efb9",
"id": "628f5e89",
"metadata": {
"editable": true
},
@@ -214,7 +214,7 @@
},
{
"cell_type": "markdown",
"id": "17b08af7",
"id": "f115ba4e",
"metadata": {
"editable": true
},
@@ -226,7 +226,7 @@
},
{
"cell_type": "markdown",
"id": "6acd4708",
"id": "a9b5189c",
"metadata": {
"editable": true
},
@@ -238,7 +238,7 @@
},
{
"cell_type": "markdown",
"id": "50579422",
"id": "a3969ff6",
"metadata": {
"editable": true
},
@@ -258,7 +258,7 @@
{
"cell_type": "code",
"execution_count": 3,
"id": "c57cc917",
"id": "34d87303",
"metadata": {
"collapsed": false,
"editable": true
@@ -277,13 +277,14 @@
"cost_history = np.zeros(num_iters)\n",
"\n",
"# Gradient descent loop\n",
"m = n_samples # number of examples\n",
"m = n_samples # number of data points\n",
"for t in range(num_iters):\n",
" # Compute prediction error\n",
" error = X_norm.dot(theta) - y_centered \n",
" # Compute cost for OLS and Ridge (MSE + regularization for Ridge) for monitoring\n",
" cost_OLS = ?\n",
" cost_Ridge = ?\n",
" # You could add a history for both methods (optional)\n",
" cost_history[t] = ?\n",
" # Compute gradients for OSL and Ridge\n",
" grad_OLS = ?\n",
@@ -301,7 +302,7 @@
},
{
"cell_type": "markdown",
"id": "e2654903",
"id": "989f70bb",
"metadata": {
"editable": true
},
@@ -313,19 +314,19 @@
},
{
"cell_type": "markdown",
"id": "7e7e7de6",
"id": "370b2dad",
"metadata": {
"editable": true
},
"source": [
"### 4b)\n",
"\n",
"Try to add a stopping parameter as function of the number iterations. How would you define a stopping criterion?"
"Try to add a stopping parameter as function of the number iterations and the difference between the new and old $\\theta$ values. How would you define a stopping criterion?"
]
},
{
"cell_type": "markdown",
"id": "75542708",
"id": "ef197cd7",
"metadata": {
"editable": true
},
@@ -351,7 +352,7 @@
{
"cell_type": "code",
"execution_count": 4,
"id": "06077986",
"id": "4ccc2f65",
"metadata": {
"collapsed": false,
"editable": true
@@ -380,7 +381,7 @@
},
{
"cell_type": "markdown",
"id": "86d46505",
"id": "00e279ef",
"metadata": {
"editable": true
},
@@ -394,7 +395,7 @@
},
{
"cell_type": "markdown",
"id": "f1d2f848",
"id": "c910b3f4",
"metadata": {
"editable": true
},
@@ -406,7 +407,7 @@
},
{
"cell_type": "markdown",
"id": "f1c91e84",
"id": "89e6e040",
"metadata": {
"editable": true
},
doc/Programs/Regression/ooreg.py
+17 -17
View File
@@ -30,8 +30,8 @@ class LinearRegression:
return X_bias @ self.weights
class RidgeRegression:
def __init__(self, alpha=1.0):
self.alpha = alpha
def __init__(self, theta=1.0):
self.theta = theta
self.weights = None
def fit(self, X, y):
@@ -39,15 +39,15 @@ class RidgeRegression:
n = X_bias.shape[1]
I = np.eye(n)
I[0, 0] = 0
self.weights = np.linalg.inv(X_bias.T @ X_bias + self.alpha * I) @ X_bias.T @ y
self.weights = np.linalg.pinv(X_bias.T @ X_bias + self.theta * I) @ X_bias.T @ y
def predict(self, X):
X_bias = np.c_[np.ones((X.shape[0], 1)), X]
return X_bias @ self.weights
class LassoRegression:
def __init__(self, alpha=1.0, max_iter=1000, tol=1e-4):
self.alpha = alpha
def __init__(self, theta=1.0, max_iter=1000, tol=1e-4):
self.theta = theta
self.max_iter = max_iter
self.tol = tol
self.weights = None
@@ -65,10 +65,10 @@ class LassoRegression:
if j == 0:
self.weights[j] = rho / np.sum(X_bias[:, j] ** 2)
else:
if rho < -self.alpha / 2:
self.weights[j] = (rho + self.alpha / 2) / np.sum(X_bias[:, j] ** 2)
elif rho > self.alpha / 2:
self.weights[j] = (rho - self.alpha / 2) / np.sum(X_bias[:, j] ** 2)
if rho < -self.theta / 2:
self.weights[j] = (rho + self.theta / 2) / np.sum(X_bias[:, j] ** 2)
elif rho > self.theta / 2:
self.weights[j] = (rho - self.theta / 2) / np.sum(X_bias[:, j] ** 2)
else:
self.weights[j] = 0
if np.linalg.norm(self.weights - weights_old, ord=1) < self.tol:
@@ -79,11 +79,11 @@ class LassoRegression:
return X_bias @ self.weights
class KernelRidgeRegression:
def __init__(self, alpha=1.0, gamma=0.1):
self.alpha = alpha
def __init__(self, theta=1.0, gamma=0.1):
self.theta = theta
self.gamma = gamma
self.X_train = None
self.alpha_vec = None
self.theta_vec = None
def _rbf_kernel(self, X1, X2):
dists = np.sum((X1[:, np.newaxis] - X2[np.newaxis, :]) ** 2, axis=2)
@@ -93,11 +93,11 @@ class KernelRidgeRegression:
self.X_train = X
K = self._rbf_kernel(X, X)
n = K.shape[0]
self.alpha_vec = np.linalg.inv(K + self.alpha * np.eye(n)) @ y
self.theta_vec = np.linalg.pinv(K + self.theta * np.eye(n)) @ y
def predict(self, X):
K = self._rbf_kernel(X, self.X_train)
return K @ self.alpha_vec
return K @ self.theta_vec
if __name__ == "__main__":
np.random.seed(42)
@@ -106,9 +106,9 @@ if __name__ == "__main__":
models = {
"linear": LinearRegression(),
"ridge": RidgeRegression(alpha=1.0),
"lasso": LassoRegression(alpha=0.1),
"kernel_ridge": KernelRidgeRegression(alpha=1.0, gamma=5.0)
"ridge": RidgeRegression(theta=1.0),
"lasso": LassoRegression(theta=0.1),
"kernel_ridge": KernelRidgeRegression(theta=1.0, gamma=5.0)
}
for name, model in models.items():
doc/Programs/Regression/predictions_kernel_ridge.csv
+101
View File
@@ -0,0 +1,101 @@
X,True Y,Predicted Y
0.749080237694725,6.29076424720326,5.9632111096332805
1.9014286128198323,9.554782163226564,8.923791804579743
1.4639878836228102,8.43784403913618,8.14487876291345
1.1973169683940732,6.598166447881773,7.23065576409039
0.31203728088487304,4.826275898735863,5.010606946926668
0.3119890406724053,5.114523407773089,5.010493117819951
0.11616722433639892,5.087448695379955,4.368853673783048
1.7323522915498704,8.937921765512787,9.20139481417348
1.2022300234864176,7.202443269012659,7.2453112461884785
1.416145155592091,7.997556944984005,7.952005287248253
0.041168988591604894,4.581208024625852,3.9887394849785283
1.9398197043239886,9.983834667801808,8.66111601751442
1.6648852816008435,8.72977574291901,9.010778402281698
0.4246782213565523,5.530668380626334,5.252160449642787
0.36364993441420124,5.139488577916623,5.125983179085797
0.36680901970686763,5.584749554387048,5.132710704422761
0.6084844859190754,5.474426910818551,5.6348527679735305
1.0495128632644757,6.9847075164945425,6.792075642513141
0.8638900372842315,6.395616035286616,6.260416621744536
0.5824582803960838,5.015617367122192,5.577965701090422
1.223705789444759,7.8191775068665645,7.309638532769508
0.27898772130408367,4.967490800002196,4.929153964118509
0.5842892970704363,5.75542461953254,5.581930369469573
0.7327236865873834,6.080877493074577,5.923091591995984
0.9121399684340719,6.0287345342770084,6.393531167034059
1.5703519227860272,8.500733106975401,8.615662374441502
0.39934756431671947,5.026685434686774,5.200558539662605
1.0284688768272232,6.68426799587086,6.729856570260571
1.184829137724085,7.47384455733925,7.193476949149691
0.09290082543999545,4.480727904727256,4.259978265008839
1.2150897038028767,8.588362062013896,7.28377082725751
0.34104824737458306,5.110433648539669,5.076867610258187
0.13010318597055903,4.519084753273059,4.430205969594416
1.8977710745066665,9.656090265636918,8.94469169179684
1.9312640661491187,8.834406590797835,8.72648057222058
1.6167946962329223,8.837127150974158,8.820376511597601
0.6092275383467414,5.857797720010738,5.636493949472784
0.19534422801276774,5.817653740280946,4.681074504439231
1.3684660530243138,8.009217676682379,7.7753720741564525
0.8803049874792026,6.791688633604414,6.305181146720942
0.24407646968955765,4.714873524216051,4.834020940213563
0.9903538202225404,6.386722441857855,6.617888670619095
0.06877704223043679,4.777742533948821,4.1384784372420516
1.8186408041575641,9.83188892881608,9.225386516840524
0.5175599632000338,5.948195863121625,5.440833966306462
1.325044568707964,7.520439978726522,7.6266956607154
0.6234221521788219,6.5716636120045155,5.668030433844864
1.0401360423556216,6.419482595670725,6.764327510889418
1.0934205586865593,7.573690222959813,6.922278413331732
0.3697089110510541,6.204354546058151,5.138860558343526
1.9391692555291171,9.322239604022007,8.666221853700872
1.550265646722229,8.3676480753653,8.524809424502802
1.8789978831283782,9.686819331928955,9.040701774048289
1.7896547008552977,9.117226275507795,9.251443032721058
1.1957999576221703,6.812068157333445,7.2261341487986055
1.8437484700462337,9.565526897541716,9.17090433765779
0.176985004103839,3.9998031554484643,4.616244794067541
0.3919657248382904,5.412693389832461,5.185359884543399
0.09045457782107613,3.811651616346327,4.248057873282361
0.6506606615265287,6.726949187088356,5.729543847640402
0.777354579378964,5.940437091968773,6.033835735521897
0.5426980635477918,5.467063432540538,5.493207852633572
1.6574750183038587,9.37918366359641,8.98364977129139
0.7135066533871786,5.525087801944558,5.876627090660308
0.5618690193747615,5.799337025426349,5.533763227561337
1.085392166316497,7.909747876090704,6.898456333861031
0.2818484499495253,4.0418037325679625,4.93650387688113
1.6043939615080793,8.90549881379039,8.767000512389243
0.14910128735954165,4.5772452592028365,4.5092940448450864
1.9737738732010346,10.31223305549176,8.363935752423266
1.5444895385933148,8.014993260340905,8.498661435867785
0.3974313630683448,4.5320657826628965,5.1966219660726445
0.011044234247204798,4.294103485550063,3.812779002299245
1.6309228569096683,9.041260907345597,8.879431789427978
1.4137146876952342,8.36639048825864,7.94262841477836
1.4580143360819746,8.547267112994412,8.119954571086216
1.5425406933718915,8.28760971932643,8.489844914245644
0.14808930346818072,4.560394758985043,4.505211113881285
0.7169314570885452,6.297330607914976,5.884855474594401
0.23173811905025943,4.338038648137594,4.797714038583459
1.726206851751187,10.11150781082594,9.189123852131019
1.2465962536551158,7.976705221421241,7.378888841756547
0.6617960497052984,5.389736400514571,5.755074415423479
0.12711670057204727,4.709626906033057,4.417298683437371
0.6219646434313244,5.378553095180313,5.664775998213837
0.6503666440534941,6.344642234031708,5.72887277792006
1.4592123566761281,8.956934859532087,8.124934745358393
1.2751149427104262,7.415003668955423,7.4666792325570395
1.774425485152653,9.80496452008012,9.250473324765727
0.9444298503238986,7.039680014439945,6.485039717959085
0.2391884918766034,5.128595555627054,4.819819059143873
1.42648957444599,9.227865214664945,7.992374640716333
1.5215700972337949,8.442016233699949,8.395392421300997
1.1225543951389925,6.9907951032382325,7.008708925719992
1.541934359909122,8.181045864914603,8.48710270019971
0.9875911927287815,6.554868435703625,6.6098235186289145
1.0454656587639881,7.0978461215849125,6.780094908540351
0.8550820367170993,6.73582209755962,6.2366273241183805
0.05083825348819038,4.290860160129581,4.042465502982899
0.2157828539866089,5.060940186477839,4.748416398789303
1 X True Y Predicted Y
2 0.749080237694725 6.29076424720326 5.9632111096332805
3 1.9014286128198323 9.554782163226564 8.923791804579743
4 1.4639878836228102 8.43784403913618 8.14487876291345
5 1.1973169683940732 6.598166447881773 7.23065576409039
6 0.31203728088487304 4.826275898735863 5.010606946926668
7 0.3119890406724053 5.114523407773089 5.010493117819951
8 0.11616722433639892 5.087448695379955 4.368853673783048
9 1.7323522915498704 8.937921765512787 9.20139481417348
10 1.2022300234864176 7.202443269012659 7.2453112461884785
11 1.416145155592091 7.997556944984005 7.952005287248253
12 0.041168988591604894 4.581208024625852 3.9887394849785283
13 1.9398197043239886 9.983834667801808 8.66111601751442
14 1.6648852816008435 8.72977574291901 9.010778402281698
15 0.4246782213565523 5.530668380626334 5.252160449642787
16 0.36364993441420124 5.139488577916623 5.125983179085797
17 0.36680901970686763 5.584749554387048 5.132710704422761
18 0.6084844859190754 5.474426910818551 5.6348527679735305
19 1.0495128632644757 6.9847075164945425 6.792075642513141
20 0.8638900372842315 6.395616035286616 6.260416621744536
21 0.5824582803960838 5.015617367122192 5.577965701090422
22 1.223705789444759 7.8191775068665645 7.309638532769508
23 0.27898772130408367 4.967490800002196 4.929153964118509
24 0.5842892970704363 5.75542461953254 5.581930369469573
25 0.7327236865873834 6.080877493074577 5.923091591995984
26 0.9121399684340719 6.0287345342770084 6.393531167034059
27 1.5703519227860272 8.500733106975401 8.615662374441502
28 0.39934756431671947 5.026685434686774 5.200558539662605
29 1.0284688768272232 6.68426799587086 6.729856570260571
30 1.184829137724085 7.47384455733925 7.193476949149691
31 0.09290082543999545 4.480727904727256 4.259978265008839
32 1.2150897038028767 8.588362062013896 7.28377082725751
33 0.34104824737458306 5.110433648539669 5.076867610258187
34 0.13010318597055903 4.519084753273059 4.430205969594416
35 1.8977710745066665 9.656090265636918 8.94469169179684
36 1.9312640661491187 8.834406590797835 8.72648057222058
37 1.6167946962329223 8.837127150974158 8.820376511597601
38 0.6092275383467414 5.857797720010738 5.636493949472784
39 0.19534422801276774 5.817653740280946 4.681074504439231
40 1.3684660530243138 8.009217676682379 7.7753720741564525
41 0.8803049874792026 6.791688633604414 6.305181146720942
42 0.24407646968955765 4.714873524216051 4.834020940213563
43 0.9903538202225404 6.386722441857855 6.617888670619095
44 0.06877704223043679 4.777742533948821 4.1384784372420516
45 1.8186408041575641 9.83188892881608 9.225386516840524
46 0.5175599632000338 5.948195863121625 5.440833966306462
47 1.325044568707964 7.520439978726522 7.6266956607154
48 0.6234221521788219 6.5716636120045155 5.668030433844864
49 1.0401360423556216 6.419482595670725 6.764327510889418
50 1.0934205586865593 7.573690222959813 6.922278413331732
51 0.3697089110510541 6.204354546058151 5.138860558343526
52 1.9391692555291171 9.322239604022007 8.666221853700872
53 1.550265646722229 8.3676480753653 8.524809424502802
54 1.8789978831283782 9.686819331928955 9.040701774048289
55 1.7896547008552977 9.117226275507795 9.251443032721058
56 1.1957999576221703 6.812068157333445 7.2261341487986055
57 1.8437484700462337 9.565526897541716 9.17090433765779
58 0.176985004103839 3.9998031554484643 4.616244794067541
59 0.3919657248382904 5.412693389832461 5.185359884543399
60 0.09045457782107613 3.811651616346327 4.248057873282361
61 0.6506606615265287 6.726949187088356 5.729543847640402
62 0.777354579378964 5.940437091968773 6.033835735521897
63 0.5426980635477918 5.467063432540538 5.493207852633572
64 1.6574750183038587 9.37918366359641 8.98364977129139
65 0.7135066533871786 5.525087801944558 5.876627090660308
66 0.5618690193747615 5.799337025426349 5.533763227561337
67 1.085392166316497 7.909747876090704 6.898456333861031
68 0.2818484499495253 4.0418037325679625 4.93650387688113
69 1.6043939615080793 8.90549881379039 8.767000512389243
70 0.14910128735954165 4.5772452592028365 4.5092940448450864
71 1.9737738732010346 10.31223305549176 8.363935752423266
72 1.5444895385933148 8.014993260340905 8.498661435867785
73 0.3974313630683448 4.5320657826628965 5.1966219660726445
74 0.011044234247204798 4.294103485550063 3.812779002299245
75 1.6309228569096683 9.041260907345597 8.879431789427978
76 1.4137146876952342 8.36639048825864 7.94262841477836
77 1.4580143360819746 8.547267112994412 8.119954571086216
78 1.5425406933718915 8.28760971932643 8.489844914245644
79 0.14808930346818072 4.560394758985043 4.505211113881285
80 0.7169314570885452 6.297330607914976 5.884855474594401
81 0.23173811905025943 4.338038648137594 4.797714038583459
82 1.726206851751187 10.11150781082594 9.189123852131019
83 1.2465962536551158 7.976705221421241 7.378888841756547
84 0.6617960497052984 5.389736400514571 5.755074415423479
85 0.12711670057204727 4.709626906033057 4.417298683437371
86 0.6219646434313244 5.378553095180313 5.664775998213837
87 0.6503666440534941 6.344642234031708 5.72887277792006
88 1.4592123566761281 8.956934859532087 8.124934745358393
89 1.2751149427104262 7.415003668955423 7.4666792325570395
90 1.774425485152653 9.80496452008012 9.250473324765727
91 0.9444298503238986 7.039680014439945 6.485039717959085
92 0.2391884918766034 5.128595555627054 4.819819059143873
93 1.42648957444599 9.227865214664945 7.992374640716333
94 1.5215700972337949 8.442016233699949 8.395392421300997
95 1.1225543951389925 6.9907951032382325 7.008708925719992
96 1.541934359909122 8.181045864914603 8.48710270019971
97 0.9875911927287815 6.554868435703625 6.6098235186289145
98 1.0454656587639881 7.0978461215849125 6.780094908540351
99 0.8550820367170993 6.73582209755962 6.2366273241183805
100 0.05083825348819038 4.290860160129581 4.042465502982899
101 0.2157828539866089 5.060940186477839 4.748416398789303
doc/Programs/Regression/predictions_lasso.csv
+101
View File
@@ -0,0 +1,101 @@
X,True Y,Predicted Y
0.749080237694725,6.29076424720326,6.269012008684799
1.9014286128198323,9.554782163226564,9.591851942859844
1.4639878836228102,8.43784403913618,8.33047512731766
1.1973169683940732,6.598166447881773,7.5615195011088305
0.31203728088487304,4.826275898735863,5.008782184800115
0.3119890406724053,5.114523407773089,5.0086430823308605
0.11616722433639892,5.087448695379955,4.443983500775647
1.7323522915498704,8.937921765512787,9.104314003494723
1.2022300234864176,7.202443269012659,7.575686480480734
1.416145155592091,7.997556944984005,8.192518819405924
0.041168988591604894,4.581208024625852,4.22772326376319
1.9398197043239886,9.983834667801808,9.702554100922704
1.6648852816008435,8.72977574291901,8.909770339671123
0.4246782213565523,5.530668380626334,5.3335865784208325
0.36364993441420124,5.139488577916623,5.157609214852874
0.36680901970686763,5.584749554387048,5.166718556257376
0.6084844859190754,5.474426910818551,5.863598863962162
1.0495128632644757,6.9847075164945425,7.135320798041875
0.8638900372842315,6.395616035286616,6.600070389245476
0.5824582803960838,5.015617367122192,5.788551320256071
1.223705789444759,7.8191775068665645,7.637612660344203
0.27898772130408367,4.967490800002196,4.913482535186757
0.5842892970704363,5.75542461953254,5.793831125787987
0.7327236865873834,6.080877493074577,6.221847277620767
0.9121399684340719,6.0287345342770084,6.739200882685431
1.5703519227860272,8.500733106975401,8.637179839643329
0.39934756431671947,5.026685434686774,5.260544674958513
1.0284688768272232,6.68426799587086,7.074639670687183
1.184829137724085,7.47384455733925,7.525510371112055
0.09290082543999545,4.480727904727256,4.376893963404605
1.2150897038028767,8.588362062013896,7.612767852936018
0.34104824737458306,5.110433648539669,5.092436399152226
0.13010318597055903,4.519084753273059,4.484168370940532
1.8977710745066665,9.656090265636918,9.581305293400263
1.9312640661491187,8.834406590797835,9.677883596061946
1.6167946962329223,8.837127150974158,8.771099325793887
0.6092275383467414,5.857797720010738,5.865741483618796
0.19534422801276774,5.817653740280946,4.672293372480547
1.3684660530243138,8.009217676682379,8.055034331678296
0.8803049874792026,6.791688633604414,6.647403516277849
0.24407646968955765,4.714873524216051,4.812814626487545
0.9903538202225404,6.386722441857855,6.964733469651725
0.06877704223043679,4.777742533948821,4.307332126079673
1.8186408041575641,9.83188892881608,9.353130179004394
0.5175599632000338,5.948195863121625,5.6014145787161
1.325044568707964,7.520439978726522,7.92982684650971
0.6234221521788219,6.5716636120045155,5.906672187136568
1.0401360423556216,6.419482595670725,7.108282381844027
1.0934205586865593,7.573690222959813,7.261930291243317
0.3697089110510541,6.204354546058151,5.175080502150851
1.9391692555291171,9.322239604022007,9.700678507328448
1.550265646722229,8.3676480753653,8.57926030621542
1.8789978831283782,9.686819331928955,9.527172089110445
1.7896547008552977,9.117226275507795,9.26954765858956
1.1957999576221703,6.812068157333445,7.55714514342129
1.8437484700462337,9.565526897541716,9.425529079085056
0.176985004103839,3.9998031554484643,4.619353858977587
0.3919657248382904,5.412693389832461,5.239258862841209
0.09045457782107613,3.811651616346327,4.369840116276003
0.6506606615265287,6.726949187088356,5.9852154545734955
0.777354579378964,5.940437091968773,6.350542137670922
0.5426980635477918,5.467063432540538,5.6739012377056275
1.6574750183038587,9.37918366359641,8.888402566402645
0.7135066533871786,5.525087801944558,6.166434238723501
0.5618690193747615,5.799337025426349,5.7291814107620125
1.085392166316497,7.909747876090704,7.238780119515455
0.2818484499495253,4.0418037325679625,4.921731553963635
1.6043939615080793,8.90549881379039,8.73534134023475
0.14910128735954165,4.5772452592028365,4.5389501117157245
1.9737738732010346,10.31223305549176,9.800462225507097
1.5444895385933148,8.014993260340905,8.562604680912191
0.3974313630683448,4.5320657826628965,5.255019236499572
0.011044234247204798,4.294103485550063,4.140857400189669
1.6309228569096683,9.041260907345597,8.811838409135795
1.4137146876952342,8.36639048825864,8.185510473700113
1.4580143360819746,8.547267112994412,8.31325017803509
1.5425406933718915,8.28760971932643,8.556985112327675
0.14808930346818072,4.560394758985043,4.536032018056252
0.7169314570885452,6.297330607914976,6.176309789165768
0.23173811905025943,4.338038648137594,4.777236527788301
1.726206851751187,10.11150781082594,9.086593396413626
1.2465962536551158,7.976705221421241,7.703618175891288
0.6617960497052984,5.389736400514571,6.017324765717306
0.12711670057204727,4.709626906033057,4.475556727830498
0.6219646434313244,5.378553095180313,5.9024694057770954
0.6503666440534941,6.344642234031708,5.984367644118051
1.4592123566761281,8.956934859532087,8.316704715532733
1.2751149427104262,7.415003668955423,7.785852889723395
1.774425485152653,9.80496452008012,9.22563364161755
0.9444298503238986,7.039680014439945,6.832309973069372
0.2391884918766034,5.128595555627054,4.798719958395109
1.42648957444599,9.227865214664945,8.222347340715581
1.5215700972337949,8.442016233699949,8.49651560866429
1.1225543951389925,6.9907951032382325,7.34593880575991
1.541934359909122,8.181045864914603,8.555236726954806
0.9875911927287815,6.554868435703625,6.956767329239963
1.0454656587639881,7.0978461215849125,7.123650531457317
0.8550820367170993,6.73582209755962,6.574672187923306
0.05083825348819038,4.290860160129581,4.255604953212358
0.2157828539866089,5.060940186477839,4.731228920142662
1 X True Y Predicted Y
2 0.749080237694725 6.29076424720326 6.269012008684799
3 1.9014286128198323 9.554782163226564 9.591851942859844
4 1.4639878836228102 8.43784403913618 8.33047512731766
5 1.1973169683940732 6.598166447881773 7.5615195011088305
6 0.31203728088487304 4.826275898735863 5.008782184800115
7 0.3119890406724053 5.114523407773089 5.0086430823308605
8 0.11616722433639892 5.087448695379955 4.443983500775647
9 1.7323522915498704 8.937921765512787 9.104314003494723
10 1.2022300234864176 7.202443269012659 7.575686480480734
11 1.416145155592091 7.997556944984005 8.192518819405924
12 0.041168988591604894 4.581208024625852 4.22772326376319
13 1.9398197043239886 9.983834667801808 9.702554100922704
14 1.6648852816008435 8.72977574291901 8.909770339671123
15 0.4246782213565523 5.530668380626334 5.3335865784208325
16 0.36364993441420124 5.139488577916623 5.157609214852874
17 0.36680901970686763 5.584749554387048 5.166718556257376
18 0.6084844859190754 5.474426910818551 5.863598863962162
19 1.0495128632644757 6.9847075164945425 7.135320798041875
20 0.8638900372842315 6.395616035286616 6.600070389245476
21 0.5824582803960838 5.015617367122192 5.788551320256071
22 1.223705789444759 7.8191775068665645 7.637612660344203
23 0.27898772130408367 4.967490800002196 4.913482535186757
24 0.5842892970704363 5.75542461953254 5.793831125787987
25 0.7327236865873834 6.080877493074577 6.221847277620767
26 0.9121399684340719 6.0287345342770084 6.739200882685431
27 1.5703519227860272 8.500733106975401 8.637179839643329
28 0.39934756431671947 5.026685434686774 5.260544674958513
29 1.0284688768272232 6.68426799587086 7.074639670687183
30 1.184829137724085 7.47384455733925 7.525510371112055
31 0.09290082543999545 4.480727904727256 4.376893963404605
32 1.2150897038028767 8.588362062013896 7.612767852936018
33 0.34104824737458306 5.110433648539669 5.092436399152226
34 0.13010318597055903 4.519084753273059 4.484168370940532
35 1.8977710745066665 9.656090265636918 9.581305293400263
36 1.9312640661491187 8.834406590797835 9.677883596061946
37 1.6167946962329223 8.837127150974158 8.771099325793887
38 0.6092275383467414 5.857797720010738 5.865741483618796
39 0.19534422801276774 5.817653740280946 4.672293372480547
40 1.3684660530243138 8.009217676682379 8.055034331678296
41 0.8803049874792026 6.791688633604414 6.647403516277849
42 0.24407646968955765 4.714873524216051 4.812814626487545
43 0.9903538202225404 6.386722441857855 6.964733469651725
44 0.06877704223043679 4.777742533948821 4.307332126079673
45 1.8186408041575641 9.83188892881608 9.353130179004394
46 0.5175599632000338 5.948195863121625 5.6014145787161
47 1.325044568707964 7.520439978726522 7.92982684650971
48 0.6234221521788219 6.5716636120045155 5.906672187136568
49 1.0401360423556216 6.419482595670725 7.108282381844027
50 1.0934205586865593 7.573690222959813 7.261930291243317
51 0.3697089110510541 6.204354546058151 5.175080502150851
52 1.9391692555291171 9.322239604022007 9.700678507328448
53 1.550265646722229 8.3676480753653 8.57926030621542
54 1.8789978831283782 9.686819331928955 9.527172089110445
55 1.7896547008552977 9.117226275507795 9.26954765858956
56 1.1957999576221703 6.812068157333445 7.55714514342129
57 1.8437484700462337 9.565526897541716 9.425529079085056
58 0.176985004103839 3.9998031554484643 4.619353858977587
59 0.3919657248382904 5.412693389832461 5.239258862841209
60 0.09045457782107613 3.811651616346327 4.369840116276003
61 0.6506606615265287 6.726949187088356 5.9852154545734955
62 0.777354579378964 5.940437091968773 6.350542137670922
63 0.5426980635477918 5.467063432540538 5.6739012377056275
64 1.6574750183038587 9.37918366359641 8.888402566402645
65 0.7135066533871786 5.525087801944558 6.166434238723501
66 0.5618690193747615 5.799337025426349 5.7291814107620125
67 1.085392166316497 7.909747876090704 7.238780119515455
68 0.2818484499495253 4.0418037325679625 4.921731553963635
69 1.6043939615080793 8.90549881379039 8.73534134023475
70 0.14910128735954165 4.5772452592028365 4.5389501117157245
71 1.9737738732010346 10.31223305549176 9.800462225507097
72 1.5444895385933148 8.014993260340905 8.562604680912191
73 0.3974313630683448 4.5320657826628965 5.255019236499572
74 0.011044234247204798 4.294103485550063 4.140857400189669
75 1.6309228569096683 9.041260907345597 8.811838409135795
76 1.4137146876952342 8.36639048825864 8.185510473700113
77 1.4580143360819746 8.547267112994412 8.31325017803509
78 1.5425406933718915 8.28760971932643 8.556985112327675
79 0.14808930346818072 4.560394758985043 4.536032018056252
80 0.7169314570885452 6.297330607914976 6.176309789165768
81 0.23173811905025943 4.338038648137594 4.777236527788301
82 1.726206851751187 10.11150781082594 9.086593396413626
83 1.2465962536551158 7.976705221421241 7.703618175891288
84 0.6617960497052984 5.389736400514571 6.017324765717306
85 0.12711670057204727 4.709626906033057 4.475556727830498
86 0.6219646434313244 5.378553095180313 5.9024694057770954
87 0.6503666440534941 6.344642234031708 5.984367644118051
88 1.4592123566761281 8.956934859532087 8.316704715532733
89 1.2751149427104262 7.415003668955423 7.785852889723395
90 1.774425485152653 9.80496452008012 9.22563364161755
91 0.9444298503238986 7.039680014439945 6.832309973069372
92 0.2391884918766034 5.128595555627054 4.798719958395109
93 1.42648957444599 9.227865214664945 8.222347340715581
94 1.5215700972337949 8.442016233699949 8.49651560866429
95 1.1225543951389925 6.9907951032382325 7.34593880575991
96 1.541934359909122 8.181045864914603 8.555236726954806
97 0.9875911927287815 6.554868435703625 6.956767329239963
98 1.0454656587639881 7.0978461215849125 7.123650531457317
99 0.8550820367170993 6.73582209755962 6.574672187923306
100 0.05083825348819038 4.290860160129581 4.255604953212358
101 0.2157828539866089 5.060940186477839 4.731228920142662
doc/Programs/Regression/predictions_linear.csv
+101
View File
@@ -0,0 +1,101 @@
X,True Y,Predicted Y
0.749080237694725,6.29076424720326,6.268687032292801
1.9014286128198323,9.554782163226564,9.593277424867807
1.4639878836228102,8.43784403913618,8.331236121211234
1.1973169683940732,6.598166447881773,7.561875412343666
0.31203728088487304,4.826275898735863,5.007793324524211
0.3119890406724053,5.114523407773089,5.007654148776349
0.11616722433639892,5.087448695379955,4.442697106877805
1.7323522915498704,8.937921765512787,9.105482652523017
1.2022300234864176,7.202443269012659,7.576049854821989
1.416145155592091,7.997556944984005,8.19320713848434
0.041168988591604894,4.581208024625852,4.226322944862653
1.9398197043239886,9.983834667801808,9.704037900371866
1.6648852816008435,8.72977574291901,8.910836503898087
0.4246782213565523,5.530668380626334,5.33276882376254
0.36364993441420124,5.139488577916623,5.156698756043805
0.36680901970686763,5.584749554387048,5.165812896211863
0.6084844859190754,5.474426910818551,5.8630603175942735
1.0495128632644757,6.9847075164945425,7.13545218955424
0.8638900372842315,6.395616035286616,6.599919813045667
0.5824582803960838,5.015617367122192,5.787973239151067
1.223705789444759,7.8191775068665645,7.6380086571421115
0.27898772130408367,4.967490800002196,4.912443471447707
0.5842892970704363,5.75542461953254,5.793255826062803
0.7327236865873834,6.080877493074577,6.2214974550425834
0.9121399684340719,6.0287345342770084,6.739123599856881
1.5703519227860272,8.500733106975401,8.638102404316905
0.39934756431671947,5.026685434686774,5.259688442126128
1.0284688768272232,6.68426799587086,7.07473909563143
1.184829137724085,7.47384455733925,7.5258473128854275
0.09290082543999545,4.480727904727256,4.375572227014628
1.2150897038028767,8.588362062013896,7.6131507615916485
0.34104824737458306,5.110433648539669,5.091491607572109
0.13010318597055903,4.519084753273059,4.482903146266885
1.8977710745066665,9.656090265636918,9.582725219476702
1.9312640661491187,8.834406590797835,9.679354399190695
1.6167946962329223,8.837127150974158,8.772092438701536
0.6092275383467414,5.857797720010738,5.865204065974123
0.19534422801276774,5.817653740280946,4.671127251283357
1.3684660530243138,8.009217676682379,8.055650224494487
0.8803049874792026,6.791688633604414,6.647277874974528
0.24407646968955765,4.714873524216051,4.811722531308558
0.9903538202225404,6.386722441857855,6.96477499646166
0.06877704223043679,4.777742533948821,4.305973744800121
1.8186408041575641,9.83188892881608,9.354429903369777
0.5175599632000338,5.948195863121625,5.600737914745937
1.325044568707964,7.520439978726522,7.930376780538088
0.6234221521788219,6.5716636120045155,5.90615633161803
1.0401360423556216,6.419482595670725,7.108399529630033
1.0934205586865593,7.573690222959813,7.262128380115554
0.3697089110510541,6.204354546058151,5.174179247144017
1.9391692555291171,9.322239604022007,9.702161318722625
1.550265646722229,8.3676480753653,8.580152359117196
1.8789978831283782,9.686819331928955,9.528563498037686
1.7896547008552977,9.117226275507795,9.270803352027228
1.1957999576221703,6.812068157333445,7.557498750262502
1.8437484700462337,9.565526897541716,9.426866942892994
0.176985004103839,3.9998031554484643,4.618159849462592
0.3919657248382904,5.412693389832461,5.238391416730617
0.09045457782107613,3.811651616346327,4.3685146639483925
0.6506606615265287,6.726949187088356,5.984740975324946
0.777354579378964,5.940437091968773,6.350260111015283
0.5426980635477918,5.467063432540538,5.673262759409053
1.6574750183038587,9.37918366359641,8.889457474174606
0.7135066533871786,5.525087801944558,6.1660552247845155
0.5618690193747615,5.799337025426349,5.728572053833063
1.085392166316497,7.909747876090704,7.238966012972516
0.2818484499495253,4.0418037325679625,4.920696835773723
1.6043939615080793,8.90549881379039,8.736315615982791
0.14910128735954165,4.5772452592028365,4.537713745837627
1.9737738732010346,10.31223305549176,9.801997602553284
1.5444895385933148,8.014993260340905,8.563487959699113
0.3974313630683448,4.5320657826628965,5.2541600928889505
0.011044234247204798,4.294103485550063,4.139411320709756
1.6309228569096683,9.041260907345597,8.81285298322486
1.4137146876952342,8.36639048825864,8.186195100810858
1.4580143360819746,8.547267112994412,8.314002097896282
1.5425406933718915,8.28760971932643,8.557865430748999
0.14808930346818072,4.560394758985043,4.5347941149384265
0.7169314570885452,6.297330607914976,6.175935977626105
0.23173811905025943,4.338038648137594,4.776125690213364
1.726206851751187,10.11150781082594,9.08775271029905
1.2465962536551158,7.976705221421241,7.704048944123097
0.6617960497052984,5.389736400514571,6.0168672015217
0.12711670057204727,4.709626906033057,4.474286966578708
0.6219646434313244,5.378553095180313,5.901951336250637
0.6503666440534941,6.344642234031708,5.9838927182464445
1.4592123566761281,8.956934859532087,8.31745845523006
1.2751149427104262,7.415003668955423,7.786326978863959
1.774425485152653,9.80496452008012,9.226866201331834
0.9444298503238986,7.039680014439945,6.832281739726384
0.2391884918766034,5.128595555627054,4.797620438202987
1.42648957444599,9.227865214664945,8.223051373336244
1.5215700972337949,8.442016233699949,8.497364071999986
1.1225543951389925,6.9907951032382325,7.346181149971801
1.541934359909122,8.181045864914603,8.556116124333911
0.9875911927287815,6.554868435703625,6.9568046595199196
1.0454656587639881,7.0978461215849125,7.123775775121281
0.8550820367170993,6.73582209755962,6.5745082320555825
0.05083825348819038,4.290860160129581,4.254219322271057
0.2157828539866089,5.060940186477839,4.730093845949392
1 X True Y Predicted Y
2 0.749080237694725 6.29076424720326 6.268687032292801
3 1.9014286128198323 9.554782163226564 9.593277424867807
4 1.4639878836228102 8.43784403913618 8.331236121211234
5 1.1973169683940732 6.598166447881773 7.561875412343666
6 0.31203728088487304 4.826275898735863 5.007793324524211
7 0.3119890406724053 5.114523407773089 5.007654148776349
8 0.11616722433639892 5.087448695379955 4.442697106877805
9 1.7323522915498704 8.937921765512787 9.105482652523017
10 1.2022300234864176 7.202443269012659 7.576049854821989
11 1.416145155592091 7.997556944984005 8.19320713848434
12 0.041168988591604894 4.581208024625852 4.226322944862653
13 1.9398197043239886 9.983834667801808 9.704037900371866
14 1.6648852816008435 8.72977574291901 8.910836503898087
15 0.4246782213565523 5.530668380626334 5.33276882376254
16 0.36364993441420124 5.139488577916623 5.156698756043805
17 0.36680901970686763 5.584749554387048 5.165812896211863
18 0.6084844859190754 5.474426910818551 5.8630603175942735
19 1.0495128632644757 6.9847075164945425 7.13545218955424
20 0.8638900372842315 6.395616035286616 6.599919813045667
21 0.5824582803960838 5.015617367122192 5.787973239151067
22 1.223705789444759 7.8191775068665645 7.6380086571421115
23 0.27898772130408367 4.967490800002196 4.912443471447707
24 0.5842892970704363 5.75542461953254 5.793255826062803
25 0.7327236865873834 6.080877493074577 6.2214974550425834
26 0.9121399684340719 6.0287345342770084 6.739123599856881
27 1.5703519227860272 8.500733106975401 8.638102404316905
28 0.39934756431671947 5.026685434686774 5.259688442126128
29 1.0284688768272232 6.68426799587086 7.07473909563143
30 1.184829137724085 7.47384455733925 7.5258473128854275
31 0.09290082543999545 4.480727904727256 4.375572227014628
32 1.2150897038028767 8.588362062013896 7.6131507615916485
33 0.34104824737458306 5.110433648539669 5.091491607572109
34 0.13010318597055903 4.519084753273059 4.482903146266885
35 1.8977710745066665 9.656090265636918 9.582725219476702
36 1.9312640661491187 8.834406590797835 9.679354399190695
37 1.6167946962329223 8.837127150974158 8.772092438701536
38 0.6092275383467414 5.857797720010738 5.865204065974123
39 0.19534422801276774 5.817653740280946 4.671127251283357
40 1.3684660530243138 8.009217676682379 8.055650224494487
41 0.8803049874792026 6.791688633604414 6.647277874974528
42 0.24407646968955765 4.714873524216051 4.811722531308558
43 0.9903538202225404 6.386722441857855 6.96477499646166
44 0.06877704223043679 4.777742533948821 4.305973744800121
45 1.8186408041575641 9.83188892881608 9.354429903369777
46 0.5175599632000338 5.948195863121625 5.600737914745937
47 1.325044568707964 7.520439978726522 7.930376780538088
48 0.6234221521788219 6.5716636120045155 5.90615633161803
49 1.0401360423556216 6.419482595670725 7.108399529630033
50 1.0934205586865593 7.573690222959813 7.262128380115554
51 0.3697089110510541 6.204354546058151 5.174179247144017
52 1.9391692555291171 9.322239604022007 9.702161318722625
53 1.550265646722229 8.3676480753653 8.580152359117196
54 1.8789978831283782 9.686819331928955 9.528563498037686
55 1.7896547008552977 9.117226275507795 9.270803352027228
56 1.1957999576221703 6.812068157333445 7.557498750262502
57 1.8437484700462337 9.565526897541716 9.426866942892994
58 0.176985004103839 3.9998031554484643 4.618159849462592
59 0.3919657248382904 5.412693389832461 5.238391416730617
60 0.09045457782107613 3.811651616346327 4.3685146639483925
61 0.6506606615265287 6.726949187088356 5.984740975324946
62 0.777354579378964 5.940437091968773 6.350260111015283
63 0.5426980635477918 5.467063432540538 5.673262759409053
64 1.6574750183038587 9.37918366359641 8.889457474174606
65 0.7135066533871786 5.525087801944558 6.1660552247845155
66 0.5618690193747615 5.799337025426349 5.728572053833063
67 1.085392166316497 7.909747876090704 7.238966012972516
68 0.2818484499495253 4.0418037325679625 4.920696835773723
69 1.6043939615080793 8.90549881379039 8.736315615982791
70 0.14910128735954165 4.5772452592028365 4.537713745837627
71 1.9737738732010346 10.31223305549176 9.801997602553284
72 1.5444895385933148 8.014993260340905 8.563487959699113
73 0.3974313630683448 4.5320657826628965 5.2541600928889505
74 0.011044234247204798 4.294103485550063 4.139411320709756
75 1.6309228569096683 9.041260907345597 8.81285298322486
76 1.4137146876952342 8.36639048825864 8.186195100810858
77 1.4580143360819746 8.547267112994412 8.314002097896282
78 1.5425406933718915 8.28760971932643 8.557865430748999
79 0.14808930346818072 4.560394758985043 4.5347941149384265
80 0.7169314570885452 6.297330607914976 6.175935977626105
81 0.23173811905025943 4.338038648137594 4.776125690213364
82 1.726206851751187 10.11150781082594 9.08775271029905
83 1.2465962536551158 7.976705221421241 7.704048944123097
84 0.6617960497052984 5.389736400514571 6.0168672015217
85 0.12711670057204727 4.709626906033057 4.474286966578708
86 0.6219646434313244 5.378553095180313 5.901951336250637
87 0.6503666440534941 6.344642234031708 5.9838927182464445
88 1.4592123566761281 8.956934859532087 8.31745845523006
89 1.2751149427104262 7.415003668955423 7.786326978863959
90 1.774425485152653 9.80496452008012 9.226866201331834
91 0.9444298503238986 7.039680014439945 6.832281739726384
92 0.2391884918766034 5.128595555627054 4.797620438202987
93 1.42648957444599 9.227865214664945 8.223051373336244
94 1.5215700972337949 8.442016233699949 8.497364071999986
95 1.1225543951389925 6.9907951032382325 7.346181149971801
96 1.541934359909122 8.181045864914603 8.556116124333911
97 0.9875911927287815 6.554868435703625 6.9568046595199196
98 1.0454656587639881 7.0978461215849125 7.123775775121281
99 0.8550820367170993 6.73582209755962 6.5745082320555825
100 0.05083825348819038 4.290860160129581 4.254219322271057
101 0.2157828539866089 5.060940186477839 4.730093845949392
doc/Programs/Regression/predictions_ridge.csv
+101
View File
@@ -0,0 +1,101 @@
X,True Y,Predicted Y
0.749080237694725,6.29076424720326,6.283996846226401
1.9014286128198323,9.554782163226564,9.51635530652161
1.4639878836228102,8.43784403913618,8.289325987593479
1.1973169683940732,6.598166447881773,7.541309148622632
0.31203728088487304,4.826275898735863,5.058083286185681
0.3119890406724053,5.114523407773089,5.057947971499375
0.11616722433639892,5.087448695379955,4.508664162533752
1.7323522915498704,8.937921765512787,9.042093104838704
1.2022300234864176,7.202443269012659,7.555090358833399
1.416145155592091,7.997556944984005,8.155126252468317
0.041168988591604894,4.581208024625852,4.29829272708084
1.9398197043239886,9.983834667801808,9.624043026747671
1.6648852816008435,8.72977574291901,8.852846896741365
0.4246782213565523,5.530668380626334,5.3740432031419685
0.36364993441420124,5.139488577916623,5.202857731913257
0.36680901970686763,5.584749554387048,5.211719024464163
0.6084844859190754,5.474426910818551,5.88962316762053
1.0495128632644757,6.9847075164945425,7.126715905819518
0.8638900372842315,6.395616035286616,6.6060404530372745
0.5824582803960838,5.015617367122192,5.8166191807814895
1.223705789444759,7.8191775068665645,7.61533027870939
0.27898772130408367,4.967490800002196,4.965378661589498
0.5842892970704363,5.75542461953254,5.821755216346812
0.7327236865873834,6.080877493074577,6.23811641848915
0.9121399684340719,6.0287345342770084,6.7413824004270255
1.5703519227860272,8.500733106975401,8.587679080531254
0.39934756431671947,5.026685434686774,5.3029902426354285
1.0284688768272232,6.68426799587086,7.067687135469227
1.184829137724085,7.47384455733925,7.506280553114361
0.09290082543999545,4.480727904727256,4.443401484218066
1.2150897038028767,8.588362062013896,7.591161999462984
0.34104824737458306,5.110433648539669,5.139459582623965
0.13010318597055903,4.519084753273059,4.547754792123642
1.8977710745066665,9.656090265636918,9.506095844048613
1.9312640661491187,8.834406590797835,9.60004430376187
1.6167946962329223,8.837127150974158,8.717951917210828
0.6092275383467414,5.857797720010738,5.891707443394121
0.19534422801276774,5.817653740280946,4.730757119159261
1.3684660530243138,8.009217676682379,8.02138548978663
0.8803049874792026,6.791688633604414,6.652084691293156
0.24407646968955765,4.714873524216051,4.8674519564787655
0.9903538202225404,6.386722441857855,6.960773698230769
0.06877704223043679,4.777742533948821,4.375733827079852
1.8186408041575641,9.83188892881608,9.284133975268611
0.5175599632000338,5.948195863121625,5.634578203264773
1.325044568707964,7.520439978726522,7.899587424880006
0.6234221521788219,6.5716636120045155,5.931523597172935
1.0401360423556216,6.419482595670725,7.100413750108909
1.0934205586865593,7.573690222959813,7.249877801851374
0.3697089110510541,6.204354546058151,5.219853273205825
1.9391692555291171,9.322239604022007,9.622218505873187
1.550265646722229,8.3676480753653,8.531336705454278
1.8789978831283782,9.686819331928955,9.453436695749156
1.7896547008552977,9.117226275507795,9.202827420529989
1.1957999576221703,6.812068157333445,7.537053905409062
1.8437484700462337,9.565526897541716,9.35456144148055
0.176985004103839,3.9998031554484643,4.679259157251658
0.3919657248382904,5.412693389832461,5.282284046657106
0.09045457782107613,3.811651616346327,4.436539714498356
0.6506606615265287,6.726949187088356,6.0079281186149265
0.777354579378964,5.940437091968773,6.3633068964899815
0.5426980635477918,5.467063432540538,5.705091038696162
1.6574750183038587,9.37918366359641,8.832060971427396
0.7135066533871786,5.525087801944558,6.184212285687546
0.5618690193747615,5.799337025426349,5.758865923615424
1.085392166316497,7.909747876090704,7.227358013079057
0.2818484499495253,4.0418037325679625,4.973403058238383
1.6043939615080793,8.90549881379039,8.683167627436156
0.14910128735954165,4.5772452592028365,4.601044817154099
1.9737738732010346,10.31223305549176,9.71928509711558
1.5444895385933148,8.014993260340905,8.51513461554718
0.3974313630683448,4.5320657826628965,5.297615262767782
0.011044234247204798,4.294103485550063,4.2137922351537656
1.6309228569096683,9.041260907345597,8.75758166866245
1.4137146876952342,8.36639048825864,8.148308745160229
1.4580143360819746,8.547267112994412,8.272570076491943
1.5425406933718915,8.28760971932643,8.509668068732644
0.14808930346818072,4.560394758985043,4.598206183661369
0.7169314570885452,6.297330607914976,6.19381892327903
0.23173811905025943,4.338038648137594,4.832842655215426
1.726206851751187,10.11150781082594,9.024855032781414
1.2465962536551158,7.976705221421241,7.679538453292357
0.6617960497052984,5.389736400514571,6.039163087974266
0.12711670057204727,4.709626906033057,4.539377645461393
0.6219646434313244,5.378553095180313,5.9274352582360095
0.6503666440534941,6.344642234031708,6.007103394176594
1.4592123566761281,8.956934859532087,8.275930546369551
1.2751149427104262,7.415003668955423,7.759533902441538
1.774425485152653,9.80496452008012,9.160109189409335
0.9444298503238986,7.039680014439945,6.831956115079231
0.2391884918766034,5.128595555627054,4.853741088499521
1.42648957444599,9.227865214664945,8.184142538273042
1.5215700972337949,8.442016233699949,8.450845159525999
1.1225543951389925,6.9907951032382325,7.331598750802819
1.541934359909122,8.181045864914603,8.507967292178913
0.9875911927287815,6.554868435703625,6.953024477122937
1.0454656587639881,7.0978461215849125,7.115363422499293
0.8550820367170993,6.73582209755962,6.581333848883506
0.05083825348819038,4.290860160129581,4.3254151935744645
0.2157828539866089,5.060940186477839,4.788087842236978
1 X True Y Predicted Y
2 0.749080237694725 6.29076424720326 6.283996846226401
3 1.9014286128198323 9.554782163226564 9.51635530652161
4 1.4639878836228102 8.43784403913618 8.289325987593479
5 1.1973169683940732 6.598166447881773 7.541309148622632
6 0.31203728088487304 4.826275898735863 5.058083286185681
7 0.3119890406724053 5.114523407773089 5.057947971499375
8 0.11616722433639892 5.087448695379955 4.508664162533752
9 1.7323522915498704 8.937921765512787 9.042093104838704
10 1.2022300234864176 7.202443269012659 7.555090358833399
11 1.416145155592091 7.997556944984005 8.155126252468317
12 0.041168988591604894 4.581208024625852 4.29829272708084
13 1.9398197043239886 9.983834667801808 9.624043026747671
14 1.6648852816008435 8.72977574291901 8.852846896741365
15 0.4246782213565523 5.530668380626334 5.3740432031419685
16 0.36364993441420124 5.139488577916623 5.202857731913257
17 0.36680901970686763 5.584749554387048 5.211719024464163
18 0.6084844859190754 5.474426910818551 5.88962316762053
19 1.0495128632644757 6.9847075164945425 7.126715905819518
20 0.8638900372842315 6.395616035286616 6.6060404530372745
21 0.5824582803960838 5.015617367122192 5.8166191807814895
22 1.223705789444759 7.8191775068665645 7.61533027870939
23 0.27898772130408367 4.967490800002196 4.965378661589498
24 0.5842892970704363 5.75542461953254 5.821755216346812
25 0.7327236865873834 6.080877493074577 6.23811641848915
26 0.9121399684340719 6.0287345342770084 6.7413824004270255
27 1.5703519227860272 8.500733106975401 8.587679080531254
28 0.39934756431671947 5.026685434686774 5.3029902426354285
29 1.0284688768272232 6.68426799587086 7.067687135469227
30 1.184829137724085 7.47384455733925 7.506280553114361
31 0.09290082543999545 4.480727904727256 4.443401484218066
32 1.2150897038028767 8.588362062013896 7.591161999462984
33 0.34104824737458306 5.110433648539669 5.139459582623965
34 0.13010318597055903 4.519084753273059 4.547754792123642
35 1.8977710745066665 9.656090265636918 9.506095844048613
36 1.9312640661491187 8.834406590797835 9.60004430376187
37 1.6167946962329223 8.837127150974158 8.717951917210828
38 0.6092275383467414 5.857797720010738 5.891707443394121
39 0.19534422801276774 5.817653740280946 4.730757119159261
40 1.3684660530243138 8.009217676682379 8.02138548978663
41 0.8803049874792026 6.791688633604414 6.652084691293156
42 0.24407646968955765 4.714873524216051 4.8674519564787655
43 0.9903538202225404 6.386722441857855 6.960773698230769
44 0.06877704223043679 4.777742533948821 4.375733827079852
45 1.8186408041575641 9.83188892881608 9.284133975268611
46 0.5175599632000338 5.948195863121625 5.634578203264773
47 1.325044568707964 7.520439978726522 7.899587424880006
48 0.6234221521788219 6.5716636120045155 5.931523597172935
49 1.0401360423556216 6.419482595670725 7.100413750108909
50 1.0934205586865593 7.573690222959813 7.249877801851374
51 0.3697089110510541 6.204354546058151 5.219853273205825
52 1.9391692555291171 9.322239604022007 9.622218505873187
53 1.550265646722229 8.3676480753653 8.531336705454278
54 1.8789978831283782 9.686819331928955 9.453436695749156
55 1.7896547008552977 9.117226275507795 9.202827420529989
56 1.1957999576221703 6.812068157333445 7.537053905409062
57 1.8437484700462337 9.565526897541716 9.35456144148055
58 0.176985004103839 3.9998031554484643 4.679259157251658
59 0.3919657248382904 5.412693389832461 5.282284046657106
60 0.09045457782107613 3.811651616346327 4.436539714498356
61 0.6506606615265287 6.726949187088356 6.0079281186149265
62 0.777354579378964 5.940437091968773 6.3633068964899815
63 0.5426980635477918 5.467063432540538 5.705091038696162
64 1.6574750183038587 9.37918366359641 8.832060971427396
65 0.7135066533871786 5.525087801944558 6.184212285687546
66 0.5618690193747615 5.799337025426349 5.758865923615424
67 1.085392166316497 7.909747876090704 7.227358013079057
68 0.2818484499495253 4.0418037325679625 4.973403058238383
69 1.6043939615080793 8.90549881379039 8.683167627436156
70 0.14910128735954165 4.5772452592028365 4.601044817154099
71 1.9737738732010346 10.31223305549176 9.71928509711558
72 1.5444895385933148 8.014993260340905 8.51513461554718
73 0.3974313630683448 4.5320657826628965 5.297615262767782
74 0.011044234247204798 4.294103485550063 4.2137922351537656
75 1.6309228569096683 9.041260907345597 8.75758166866245
76 1.4137146876952342 8.36639048825864 8.148308745160229
77 1.4580143360819746 8.547267112994412 8.272570076491943
78 1.5425406933718915 8.28760971932643 8.509668068732644
79 0.14808930346818072 4.560394758985043 4.598206183661369
80 0.7169314570885452 6.297330607914976 6.19381892327903
81 0.23173811905025943 4.338038648137594 4.832842655215426
82 1.726206851751187 10.11150781082594 9.024855032781414
83 1.2465962536551158 7.976705221421241 7.679538453292357
84 0.6617960497052984 5.389736400514571 6.039163087974266
85 0.12711670057204727 4.709626906033057 4.539377645461393
86 0.6219646434313244 5.378553095180313 5.9274352582360095
87 0.6503666440534941 6.344642234031708 6.007103394176594
88 1.4592123566761281 8.956934859532087 8.275930546369551
89 1.2751149427104262 7.415003668955423 7.759533902441538
90 1.774425485152653 9.80496452008012 9.160109189409335
91 0.9444298503238986 7.039680014439945 6.831956115079231
92 0.2391884918766034 5.128595555627054 4.853741088499521
93 1.42648957444599 9.227865214664945 8.184142538273042
94 1.5215700972337949 8.442016233699949 8.450845159525999
95 1.1225543951389925 6.9907951032382325 7.331598750802819
96 1.541934359909122 8.181045864914603 8.507967292178913
97 0.9875911927287815 6.554868435703625 6.953024477122937
98 1.0454656587639881 7.0978461215849125 7.115363422499293
99 0.8550820367170993 6.73582209755962 6.581333848883506
100 0.05083825348819038 4.290860160129581 4.3254151935744645
101 0.2157828539866089 5.060940186477839 4.788087842236978
doc/pub/week36/ipynb/week36.ipynb
+4 -3
View File
@@ -3296,10 +3296,11 @@
"cell_type": "markdown",
"id": "d6b1ae12",
"metadata": {
"editable": true
"editable": true,
"jp-MarkdownHeadingCollapsed": true
},
"source": [
"## Material for lab sessions sessions Tuesday and Wednesday\n",
"## Material for lab sessions Tuesday and Wednesday\n",
"\n",
"The material here contains a summary of the lecture on Monday and discussion of SVD, Ridge and Lasso regression with examples"
]
@@ -5129,7 +5130,7 @@
},
{
"cell_type": "code",
"execution_count": 5,
"execution_count": 3,
"id": "92ca291e",
"metadata": {
"collapsed": false,
doc/src/week35/programs/lasso.py
+139
View File
@@ -0,0 +1,139 @@
import numpy as np
# Seed for reproducibility
np.random.seed(0)
# Dimensions of the synthetic dataset
N = 100 # number of samples (observations)
p = 10 # number of features
# True sparse coefficients (only a few non-zero)
w_true = np.array([5, -3, 0, 0, 2, 0, 0, 0, 0, 0], dtype=float)
# For example, feature 0 has coefficient 5, feature 1 has -3, feature 4 has 2, rest are 0.
# Generate feature matrix X from a normal distribution
X = np.random.randn(N, p)
# Generate target values: linear combination of X with w_true + noise
noise = np.random.randn(N) * 1.0 # noise with standard deviation 1.0
y = X.dot(w_true) + noise
# Standardize features (zero mean, unit variance for each column)
X_mean = X.mean(axis=0)
X_std = X.std(axis=0)
X_std[X_std == 0] = 1.0 # avoid division by zero if any constant feature
X_norm = (X - X_mean) / X_std
# Center the target to zero mean
y_mean = y.mean()
y_centered = y - y_mean
def soft_threshold(rho, lam):
"""Soft thresholding operator: S(rho, lam) = sign(rho)*max(|rho|-lam, 0)."""
if rho < -lam:
return rho + lam
elif rho > lam:
return rho - lam
else:
return 0.0
def lasso_coordinate_descent(X, y, alpha, max_iter=1000, tol=1e-6):
"""
Perform LASSO regression using coordinate descent.
X : array of shape (n_samples, n_features), assumed to be standardized.
y : array of shape (n_samples,), assumed centered.
alpha : regularization strength (L1 penalty coefficient).
max_iter : maximum number of coordinate descent iterations (full cycles).
tol : tolerance for convergence (stop if max coef change < tol).
"""
n_samples, n_features = X.shape
w = np.zeros(n_features) # initialize weights to zero
for it in range(max_iter):
w_old = w.copy()
# Loop over each feature coordinate
for j in range(n_features):
# Compute rho_j = x_j^T (y - X w + w_j * x_j)
# (This is the contribution of feature j to the residual)
X_j = X[:, j]
# temporarily exclude feature j's effect
residual = y - X.dot(w) + w[j] * X_j
rho_j = X_j.dot(residual)
# Soft thresholding update for w_j
w[j] = soft_threshold(rho_j, alpha) / (X_j.dot(X_j))
# Check convergence: if all updates are very small, break
if np.max(np.abs(w - w_old)) < tol:
break
return w
alpha = 50.0 # regularization strength
w_learned = lasso_coordinate_descent(X_norm, y_centered, alpha)
print("True coefficients:", w_true)
print("Learned coefficients:", w_learned)
# Plot y vs a relevant feature (0) and an irrelevant feature (2)
fig, axes = plt.subplots(1, 2, figsize=(10, 4))
axes[0].scatter(X[:, 0], y, color='blue', alpha=0.6)
axes[0].set_title("Feature 0 (Relevant) vs Target")
axes[0].set_xlabel("Feature 0 values")
axes[0].set_ylabel("Target (y)")
axes[1].scatter(X[:, 2], y, color='red', alpha=0.6)
axes[1].set_title("Feature 2 (Irrelevant) vs Target")
axes[1].set_xlabel("Feature 2 values")
axes[1].set_ylabel("Target (y)")
plt.tight_layout()
plt.show()
# Track cost history during coordinate descent for plotting
def lasso_with_cost_history(X, y, alpha, max_iter=1000):
n_samples, n_features = X.shape
w = np.zeros(n_features)
cost_history = []
# initial cost
cost_history.append(0.5 * np.sum((y - X.dot(w))**2) + alpha * np.sum(np.abs(w)))
for it in range(max_iter):
w_old = w.copy()
for j in range(n_features):
X_j = X[:, j]
residual = y - X.dot(w) + w[j] * X_j
rho_j = X_j.dot(residual)
w[j] = soft_threshold(rho_j, alpha) / (X_j.dot(X_j))
# compute cost after this iteration
cost = 0.5 * np.sum((y - X.dot(w))**2) + alpha * np.sum(np.abs(w))
cost_history.append(cost)
if np.max(np.abs(w - w_old)) < 1e-6:
break
return w, cost_history
# Run coordinate descent and get cost history
w_fit, cost_history = lasso_with_cost_history(X_norm, y_centered, alpha=50.0)
# Plot cost vs iteration
plt.figure(figsize=(6,4))
plt.plot(cost_history, marker='o', color='purple')
plt.title("LASSO Cost Decrease over Iterations")
plt.xlabel("Iteration")
plt.ylabel("Cost function value")
plt.grid(True)
plt.show()
# Compare true vs learned coefficients
import numpy as np
import matplotlib.pyplot as plt
indices = np.arange(p)
width = 0.4
plt.figure(figsize=(6,4))
plt.bar(indices - width/2, w_true, width=width, label='True Coefficient')
plt.bar(indices + width/2, w_fit, width=width, label='Learned Coefficient')
plt.xlabel("Feature index")
plt.ylabel("Coefficient value")
plt.title("True vs Learned Coefficients")
plt.legend()
plt.show()
doc/src/week37/exercisesweek37.do.txt
+5 -4
View File
@@ -7,7 +7,7 @@ DATE: September 8-12, 2025
After having completed these exercises you will have:
o Your own code for the implementation of the simplest gradient descent approach applied to ordinary least squares (OLS) and Ridge regression
o Be able to compare the analytical expressions for OLS and Rudge regression with the gradient descent approach
o Be able to compare the analytical expressions for OLS and Ridge regression with the gradient descent approach
o Explore the role of the learning rate in the gradient descent approach and the hyperparameter $\lambda$ in Ridge regression
o Scale the data properly
@@ -72,7 +72,7 @@ same scale).
Find the gradients for OLS and Ridge regression using the mean-squared error as cost/loss function.
===== Exercise 3, use the analytical formulae for OLS and Ridge regression to find the optimal paramters $\bm{\theta}$ =====
===== Exercise 3, using the analytical formulae for OLS and Ridge regression to find the optimal paramters $\bm{\theta}$ =====
!bc pycod
# Set regularization parameter, either a single value or a vector of values
@@ -124,13 +124,14 @@ theta = np.zeros(n_features)
cost_history = np.zeros(num_iters)
# Gradient descent loop
m = n_samples # number of examples
m = n_samples # number of data points
for t in range(num_iters):
# Compute prediction error
error = X_norm.dot(theta) - y_centered
# Compute cost for OLS and Ridge (MSE + regularization for Ridge) for monitoring
cost_OLS = ?
cost_Ridge = ?
# You could add a history for both methods (optional)
cost_history[t] = ?
# Compute gradients for OSL and Ridge
grad_OLS = ?
@@ -150,7 +151,7 @@ print("Gradient Descent Ridge coefficients:", theta_gdRidge)
Discuss the results as function of the learning rate parameters and the number of iterations.
=== 4b) ===
Try to add a stopping parameter as function of the number iterations. How would you define a stopping criterion?
Try to add a stopping parameter as function of the number iterations and the difference between the new and old $\theta$ values. How would you define a stopping criterion?
===== Exercise 5, Ridge regression and a new Synthetic Dataset =====