lars/FYS-STK4155
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

change

Browse Source
This commit is contained in:
Morten Hjorth-Jensen
2025-08-25 09:58:17 +02:00
parent 9f4218ad1a
commit a0460cde92
9 changed files with 910 additions and 2341 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/BookChapters/chapter1.dlog
+10
View File
@@ -97,3 +97,13 @@ found info about 5 exercises
*** warning: latex envir \begin{bmatrix} does not work well in Markdown. Stick to \[ ... \], equation, equation*, align, or align* environments in math environments.
output in chapter1.ipynb
Translating doconce text in chapter1.do.txt to ipynb
*** replacing \bm{...} by \boldsymbol{...} (\bm is not supported by MathJax)
found info about 5 exercises
*** warning: latex envir \begin{bmatrix} does not work well in Markdown. Stick to \[ ... \], equation, equation*, align, or align* environments in math environments.
*** warning: latex envir \begin{bmatrix} does not work well in Markdown. Stick to \[ ... \], equation, equation*, align, or align* environments in math environments.
*** warning: latex envir \begin{bmatrix} does not work well in Markdown. Stick to \[ ... \], equation, equation*, align, or align* environments in math environments.
output in chapter1.ipynb
doc/BookChapters/chapter1.do.txt
+7 -161
View File
@@ -262,10 +262,10 @@ y = 2x+N(0,1),
where $N(0,1)$ represents random numbers generated by the normal
distribution. From _Scikit-Learn_ we import then the
_LinearRegression_ functionality and make a prediction $\tilde{y} =
\alpha + \beta x$ using the function _fit(x,y)_. We call the set of
\alpha + \theta x$ using the function _fit(x,y)_. We call the set of
data $(\bm{x},\bm{y})$ for our training data. The Python package
_scikit-learn_ has also a functionality which extracts the above
fitting parameters $\alpha$ and $\beta$ (see below). Later we will
fitting parameters $\alpha$ and $\theta$ (see below). Later we will
distinguish between training data and test data.
For plotting we use the Python package
@@ -351,7 +351,7 @@ dimensionless.
Minimizing the cost function is a central aspect of
our discussions to come. Finding its minima as function of the model
parameters ($\alpha$ and $\beta$ in our case) will be a recurring
parameters ($\alpha$ and $\theta$ in our case) will be a recurring
theme in these series of lectures. Essentially all machine learning
algorithms we will discuss center around the minimization of the
chosen cost function. This depends in turn on our specific
@@ -407,7 +407,7 @@ different training data sets and study (graphically) the value of the
relative error.
As mentioned above, _Scikit-Learn_ has an impressive functionality.
We can for example extract the values of $\alpha$ and $\beta$ and
We can for example extract the values of $\alpha$ and $\theta$ and
their error estimates, or the variance and standard deviation and many
other properties from the statistical data analysis.
@@ -425,7 +425,7 @@ linreg = LinearRegression()
linreg.fit(x,y)
ypredict = linreg.predict(x)
print('The intercept alpha: \n', linreg.intercept_)
print('Coefficient beta : \n', linreg.coef_)
print('Coefficient theta : \n', linreg.coef_)
# The mean squared error
print("Mean squared error: %.2f" % mean_squared_error(y, ypredict))
# Explained variance score: 1 is perfect prediction
@@ -443,8 +443,8 @@ plt.title(r'Linear Regression fit ')
plt.show()
!ec
The function _coef_ gives us the parameter $\beta$ of our fit while _intercept_ yields
$\alpha$. Depending on the constant in front of the normal distribution, we get values near or far from $alpha =2$ and $\beta =5$. Try to play around with different parameters in front of the normal distribution. The function _meansquarederror_ gives us the mean square error, a risk metric corresponding to the expected value of the squared (quadratic) error or loss defined as
The function _coef_ gives us the parameter $\theta$ of our fit while _intercept_ yields
$\alpha$. Depending on the constant in front of the normal distribution, we get values near or far from $alpha =2$ and $\theta =5$. Try to play around with different parameters in front of the normal distribution. The function _meansquarederror_ gives us the mean square error, a risk metric corresponding to the expected value of the squared (quadratic) error or loss defined as
!bt
\[ MSE(\bm{y},\bm{\tilde{y}}) = \frac{1}{n}
\sum_{i=0}^{n-1}(y_i-\tilde{y}_i)^2,
@@ -1815,160 +1815,6 @@ print(MSE(y_test,ypredict))
!ec
===== The Boston housing data example =====
The Boston housing
data set was originally a part of UCI Machine Learning Repository
and has been removed now. The data set is now included in _Scikit-Learn_'s
library. There are 506 samples and 13 feature (predictor) variables
in this data set. The objective is to predict the value of prices of
the house using the features (predictors) listed here.
The features/predictors are
o CRIM: Per capita crime rate by town
o ZN: Proportion of residential land zoned for lots over 25000 square feet
o INDUS: Proportion of non-retail business acres per town
o CHAS: Charles River dummy variable (= 1 if tract bounds river; 0 otherwise)
o NOX: Nitric oxide concentration (parts per 10 million)
o RM: Average number of rooms per dwelling
o AGE: Proportion of owner-occupied units built prior to 1940
o DIS: Weighted distances to five Boston employment centers
o RAD: Index of accessibility to radial highways
o TAX: Full-value property tax rate per USD10000
o B: $1000(Bk - 0.63)^2$, where $Bk$ is the proportion of [people of African American descent] by town
o LSTAT: Percentage of lower status of the population
o MEDV: Median value of owner-occupied homes in USD 1000s
!split
===== Housing data, the code =====
We start by importing the libraries
!bc pycod
import numpy as np
import matplotlib.pyplot as plt
import pandas as pd
import seaborn as sns
!ec
and load the Boston Housing DataSet from _Scikit-Learn_
!bc pycod
from sklearn.datasets import load_boston
boston_dataset = load_boston()
# boston_dataset is a dictionary
# let's check what it contains
boston_dataset.keys()
!ec
Then we invoke Pandas
!bc pycod
boston = pd.DataFrame(boston_dataset.data, columns=boston_dataset.feature_names)
boston.head()
boston['MEDV'] = boston_dataset.target
!ec
and preprocess the data
!bc pycod
# check for missing values in all the columns
boston.isnull().sum()
!ec
We can then visualize the data
!bc pycod
# set the size of the figure
sns.set(rc={'figure.figsize':(11.7,8.27)})
# plot a histogram showing the distribution of the target values
sns.distplot(boston['MEDV'], bins=30)
plt.show()
!ec
It is now useful to look at the correlation matrix
!bc pycod
# compute the pair wise correlation for all columns
correlation_matrix = boston.corr().round(2)
# use the heatmap function from seaborn to plot the correlation matrix
# annot = True to print the values inside the square
sns.heatmap(data=correlation_matrix, annot=True)
!ec
From the above coorelation plot we can see that _MEDV_ is strongly correlated to _LSTAT_ and _RM_. We see also that _RAD_ and _TAX_ are stronly correlated, but we don't include this in our features together to avoid multi-colinearity
!bc pycod
plt.figure(figsize=(20, 5))
features = ['LSTAT', 'RM']
target = boston['MEDV']
for i, col in enumerate(features):
plt.subplot(1, len(features) , i+1)
x = boston[col]
y = target
plt.scatter(x, y, marker='o')
plt.title(col)
plt.xlabel(col)
plt.ylabel('MEDV')
!ec
Now we start training our model
!bc pycod
X = pd.DataFrame(np.c_[boston['LSTAT'], boston['RM']], columns = ['LSTAT','RM'])
Y = boston['MEDV']
!ec
We split the data into training and test sets
!bc pycod
from sklearn.model_selection import train_test_split
# splits the training and test data set in 80% : 20%
# assign random_state to any value.This ensures consistency.
X_train, X_test, Y_train, Y_test = train_test_split(X, Y, test_size = 0.2, random_state=5)
print(X_train.shape)
print(X_test.shape)
print(Y_train.shape)
print(Y_test.shape)
!ec
Then we use the linear regression functionality from _Scikit-Learn_
!bc pycod
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_squared_error, r2_score
lin_model = LinearRegression()
lin_model.fit(X_train, Y_train)
# model evaluation for training set
y_train_predict = lin_model.predict(X_train)
rmse = (np.sqrt(mean_squared_error(Y_train, y_train_predict)))
r2 = r2_score(Y_train, y_train_predict)
print("The model performance for training set")
print("--------------------------------------")
print('RMSE is {}'.format(rmse))
print('R2 score is {}'.format(r2))
print("\n")
# model evaluation for testing set
y_test_predict = lin_model.predict(X_test)
# root mean square error of the model
rmse = (np.sqrt(mean_squared_error(Y_test, y_test_predict)))
# r-squared score of the model
r2 = r2_score(Y_test, y_test_predict)
print("The model performance for testing set")
print("--------------------------------------")
print('RMSE is {}'.format(rmse))
print('R2 score is {}'.format(r2))
!ec
!bc pycod
# plotting the y_test vs y_pred
# ideally should have been a straight line
plt.scatter(Y_test, y_test_predict)
plt.show()
!ec
===== Reducing the number of degrees of freedom, overarching view =====
doc/LectureNotes/_build/.doctrees/chapter1.doctree
BIN
View File
Binary file not shown.
doc/LectureNotes/_build/.doctrees/environment.pickle
BIN
View File
Binary file not shown.
doc/LectureNotes/_build/html/_sources/chapter1.ipynb
+287 -651
View File
File diff suppressed because it is too large Load Diff
doc/LectureNotes/_build/html/chapter1.html
+31 -226
View File
@@ -391,16 +391,14 @@ document.write(`
</ul>
</li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#splitting-our-data-in-training-and-test-data">3.5. Splitting our Data in Training and Test data</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#the-boston-housing-data-example">3.6. The Boston housing data example</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#housing-data-the-code">3.7. Housing data, the code</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#reducing-the-number-of-degrees-of-freedom-overarching-view">3.8. Reducing the number of degrees of freedom, overarching view</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#testing-the-means-squared-error-as-function-of-complexity">3.9. Testing the Means Squared Error as function of Complexity</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#exercises">3.10. Exercises</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#exercise-1-setting-up-various-python-environments">3.11. Exercise 1: Setting up various Python environments</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#exercise-2-making-your-own-data-and-exploring-scikit-learn">3.12. Exercise 2: making your own data and exploring scikit-learn</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#exercise-3-normalizing-our-data">3.13. Exercise 3: Normalizing our data</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#exercise-4-adding-ridge-regression">3.14. Exercise 4: Adding Ridge Regression</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#exercise-5-analytical-exercises">3.15. Exercise 5: Analytical exercises</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#reducing-the-number-of-degrees-of-freedom-overarching-view">3.6. Reducing the number of degrees of freedom, overarching view</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#testing-the-means-squared-error-as-function-of-complexity">3.7. Testing the Means Squared Error as function of Complexity</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#exercises">3.8. Exercises</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#exercise-1-setting-up-various-python-environments">3.9. Exercise 1: Setting up various Python environments</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#exercise-2-making-your-own-data-and-exploring-scikit-learn">3.10. Exercise 2: making your own data and exploring scikit-learn</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#exercise-3-normalizing-our-data">3.11. Exercise 3: Normalizing our data</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#exercise-4-adding-ridge-regression">3.12. Exercise 4: Adding Ridge Regression</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#exercise-5-analytical-exercises">3.13. Exercise 5: Analytical exercises</a></li>
</ul>
</nav>
</div>
@@ -621,10 +619,10 @@ y = 2x+N(0,1),
<p>where <span class="math notranslate nohighlight">\(N(0,1)\)</span> represents random numbers generated by the normal
distribution. From <strong>Scikit-Learn</strong> we import then the
<strong>LinearRegression</strong> functionality and make a prediction <span class="math notranslate nohighlight">\(\tilde{y} =
\alpha + \beta x\)</span> using the function <strong>fit(x,y)</strong>. We call the set of
\alpha + \theta x\)</span> using the function <strong>fit(x,y)</strong>. We call the set of
data <span class="math notranslate nohighlight">\((\boldsymbol{x},\boldsymbol{y})\)</span> for our training data. The Python package
<strong>scikit-learn</strong> has also a functionality which extracts the above
fitting parameters <span class="math notranslate nohighlight">\(\alpha\)</span> and <span class="math notranslate nohighlight">\(\beta\)</span> (see below). Later we will
fitting parameters <span class="math notranslate nohighlight">\(\alpha\)</span> and <span class="math notranslate nohighlight">\(\theta\)</span> (see below). Later we will
distinguish between training data and test data.</p>
<p>For plotting we use the Python package
<a class="reference external" href="https://matplotlib.org/">matplotlib</a> which produces publication
@@ -704,7 +702,7 @@ however the aim of scaling the equations and make the cost function
dimensionless.</p>
<p>Minimizing the cost function is a central aspect of
our discussions to come. Finding its minima as function of the model
parameters (<span class="math notranslate nohighlight">\(\alpha\)</span> and <span class="math notranslate nohighlight">\(\beta\)</span> in our case) will be a recurring
parameters (<span class="math notranslate nohighlight">\(\alpha\)</span> and <span class="math notranslate nohighlight">\(\theta\)</span> in our case) will be a recurring
theme in these series of lectures. Essentially all machine learning
algorithms we will discuss center around the minimization of the
chosen cost function. This depends in turn on our specific
@@ -757,7 +755,7 @@ have a small or larger relative error. Try to play around with
different training data sets and study (graphically) the value of the
relative error.</p>
<p>As mentioned above, <strong>Scikit-Learn</strong> has an impressive functionality.
We can for example extract the values of <span class="math notranslate nohighlight">\(\alpha\)</span> and <span class="math notranslate nohighlight">\(\beta\)</span> and
We can for example extract the values of <span class="math notranslate nohighlight">\(\alpha\)</span> and <span class="math notranslate nohighlight">\(\theta\)</span> and
their error estimates, or the variance and standard deviation and many
other properties from the statistical data analysis.</p>
<p>Here we show an
@@ -775,7 +773,7 @@ linreg = LinearRegression()
linreg.fit(x,y)
ypredict = linreg.predict(x)
print(&#39;The intercept alpha: \n&#39;, linreg.intercept_)
print(&#39;Coefficient beta : \n&#39;, linreg.coef_)
print(&#39;Coefficient theta : \n&#39;, linreg.coef_)
# The mean squared error
print(&quot;Mean squared error: %.2f&quot; % mean_squared_error(y, ypredict))
# Explained variance score: 1 is perfect prediction
@@ -795,8 +793,8 @@ plt.show()
</div>
</div>
</div>
<p>The function <strong>coef</strong> gives us the parameter <span class="math notranslate nohighlight">\(\beta\)</span> of our fit while <strong>intercept</strong> yields
<span class="math notranslate nohighlight">\(\alpha\)</span>. Depending on the constant in front of the normal distribution, we get values near or far from <span class="math notranslate nohighlight">\(alpha =2\)</span> and <span class="math notranslate nohighlight">\(\beta =5\)</span>. Try to play around with different parameters in front of the normal distribution. The function <strong>meansquarederror</strong> gives us the mean square error, a risk metric corresponding to the expected value of the squared (quadratic) error or loss defined as</p>
<p>The function <strong>coef</strong> gives us the parameter <span class="math notranslate nohighlight">\(\theta\)</span> of our fit while <strong>intercept</strong> yields
<span class="math notranslate nohighlight">\(\alpha\)</span>. Depending on the constant in front of the normal distribution, we get values near or far from <span class="math notranslate nohighlight">\(alpha =2\)</span> and <span class="math notranslate nohighlight">\(\theta =5\)</span>. Try to play around with different parameters in front of the normal distribution. The function <strong>meansquarederror</strong> gives us the mean square error, a risk metric corresponding to the expected value of the squared (quadratic) error or loss defined as</p>
<div class="math notranslate nohighlight">
\[
MSE(\boldsymbol{y},\boldsymbol{\tilde{y}}) = \frac{1}{n}
@@ -2038,199 +2036,8 @@ print(MSE(y_test,ypredict))
</div>
</div>
</section>
<section id="the-boston-housing-data-example">
<h2><span class="section-number">3.6. </span>The Boston housing data example<a class="headerlink" href="#the-boston-housing-data-example" title="Link to this heading">#</a></h2>
<p>The Boston housing<br />
data set was originally a part of UCI Machine Learning Repository
and has been removed now. The data set is now included in <strong>Scikit-Learn</strong>’s
library. There are 506 samples and 13 feature (predictor) variables
in this data set. The objective is to predict the value of prices of
the house using the features (predictors) listed here.</p>
<p>The features/predictors are</p>
<ol class="arabic simple">
<li><p>CRIM: Per capita crime rate by town</p></li>
<li><p>ZN: Proportion of residential land zoned for lots over 25000 square feet</p></li>
<li><p>INDUS: Proportion of non-retail business acres per town</p></li>
<li><p>CHAS: Charles River dummy variable (= 1 if tract bounds river; 0 otherwise)</p></li>
<li><p>NOX: Nitric oxide concentration (parts per 10 million)</p></li>
<li><p>RM: Average number of rooms per dwelling</p></li>
<li><p>AGE: Proportion of owner-occupied units built prior to 1940</p></li>
<li><p>DIS: Weighted distances to five Boston employment centers</p></li>
<li><p>RAD: Index of accessibility to radial highways</p></li>
<li><p>TAX: Full-value property tax rate per USD10000</p></li>
<li><p>B: <span class="math notranslate nohighlight">\(1000(Bk - 0.63)^2\)</span>, where <span class="math notranslate nohighlight">\(Bk\)</span> is the proportion of [people of African American descent] by town</p></li>
<li><p>LSTAT: Percentage of lower status of the population</p></li>
<li><p>MEDV: Median value of owner-occupied homes in USD 1000s</p></li>
</ol>
</section>
<section id="housing-data-the-code">
<h2><span class="section-number">3.7. </span>Housing data, the code<a class="headerlink" href="#housing-data-the-code" title="Link to this heading">#</a></h2>
<p>We start by importing the libraries</p>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span>import numpy as np
import matplotlib.pyplot as plt
import pandas as pd
import seaborn as sns
</pre></div>
</div>
</div>
</div>
<p>and load the Boston Housing DataSet from <strong>Scikit-Learn</strong></p>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span>from sklearn.datasets import load_boston
boston_dataset = load_boston()
# boston_dataset is a dictionary
# let&#39;s check what it contains
boston_dataset.keys()
</pre></div>
</div>
</div>
</div>
<p>Then we invoke Pandas</p>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span>boston = pd.DataFrame(boston_dataset.data, columns=boston_dataset.feature_names)
boston.head()
boston[&#39;MEDV&#39;] = boston_dataset.target
</pre></div>
</div>
</div>
</div>
<p>and preprocess the data</p>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span># check for missing values in all the columns
boston.isnull().sum()
</pre></div>
</div>
</div>
</div>
<p>We can then visualize the data</p>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span># set the size of the figure
sns.set(rc={&#39;figure.figsize&#39;:(11.7,8.27)})
# plot a histogram showing the distribution of the target values
sns.distplot(boston[&#39;MEDV&#39;], bins=30)
plt.show()
</pre></div>
</div>
</div>
</div>
<p>It is now useful to look at the correlation matrix</p>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span># compute the pair wise correlation for all columns
correlation_matrix = boston.corr().round(2)
# use the heatmap function from seaborn to plot the correlation matrix
# annot = True to print the values inside the square
sns.heatmap(data=correlation_matrix, annot=True)
</pre></div>
</div>
</div>
</div>
<p>From the above coorelation plot we can see that <strong>MEDV</strong> is strongly correlated to <strong>LSTAT</strong> and <strong>RM</strong>. We see also that <strong>RAD</strong> and <strong>TAX</strong> are stronly correlated, but we don’t include this in our features together to avoid multi-colinearity</p>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span>plt.figure(figsize=(20, 5))
features = [&#39;LSTAT&#39;, &#39;RM&#39;]
target = boston[&#39;MEDV&#39;]
for i, col in enumerate(features):
plt.subplot(1, len(features) , i+1)
x = boston[col]
y = target
plt.scatter(x, y, marker=&#39;o&#39;)
plt.title(col)
plt.xlabel(col)
plt.ylabel(&#39;MEDV&#39;)
</pre></div>
</div>
</div>
</div>
<p>Now we start training our model</p>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span>X = pd.DataFrame(np.c_[boston[&#39;LSTAT&#39;], boston[&#39;RM&#39;]], columns = [&#39;LSTAT&#39;,&#39;RM&#39;])
Y = boston[&#39;MEDV&#39;]
</pre></div>
</div>
</div>
</div>
<p>We split the data into training and test sets</p>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span>from sklearn.model_selection import train_test_split
# splits the training and test data set in 80% : 20%
# assign random_state to any value.This ensures consistency.
X_train, X_test, Y_train, Y_test = train_test_split(X, Y, test_size = 0.2, random_state=5)
print(X_train.shape)
print(X_test.shape)
print(Y_train.shape)
print(Y_test.shape)
</pre></div>
</div>
</div>
</div>
<p>Then we use the linear regression functionality from <strong>Scikit-Learn</strong></p>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span>from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_squared_error, r2_score
lin_model = LinearRegression()
lin_model.fit(X_train, Y_train)
# model evaluation for training set
y_train_predict = lin_model.predict(X_train)
rmse = (np.sqrt(mean_squared_error(Y_train, y_train_predict)))
r2 = r2_score(Y_train, y_train_predict)
print(&quot;The model performance for training set&quot;)
print(&quot;--------------------------------------&quot;)
print(&#39;RMSE is {}&#39;.format(rmse))
print(&#39;R2 score is {}&#39;.format(r2))
print(&quot;\n&quot;)
# model evaluation for testing set
y_test_predict = lin_model.predict(X_test)
# root mean square error of the model
rmse = (np.sqrt(mean_squared_error(Y_test, y_test_predict)))
# r-squared score of the model
r2 = r2_score(Y_test, y_test_predict)
print(&quot;The model performance for testing set&quot;)
print(&quot;--------------------------------------&quot;)
print(&#39;RMSE is {}&#39;.format(rmse))
print(&#39;R2 score is {}&#39;.format(r2))
</pre></div>
</div>
</div>
</div>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span># plotting the y_test vs y_pred
# ideally should have been a straight line
plt.scatter(Y_test, y_test_predict)
plt.show()
</pre></div>
</div>
</div>
</div>
</section>
<section id="reducing-the-number-of-degrees-of-freedom-overarching-view">
<h2><span class="section-number">3.8. </span>Reducing the number of degrees of freedom, overarching view<a class="headerlink" href="#reducing-the-number-of-degrees-of-freedom-overarching-view" title="Link to this heading">#</a></h2>
<h2><span class="section-number">3.6. </span>Reducing the number of degrees of freedom, overarching view<a class="headerlink" href="#reducing-the-number-of-degrees-of-freedom-overarching-view" title="Link to this heading">#</a></h2>
<p>Many Machine Learning problems involve thousands or even millions of
features for each training instance. Not only does this make training
extremely slow, it can also make it much harder to find a good
@@ -2343,7 +2150,7 @@ x_j^{(i)} \rightarrow (b-a)\frac{x_j^{(i)} - \min(x_j)}{\max(x_j) - \min(x_j)} -
<p>where <span class="math notranslate nohighlight">\(\min(x_j)\)</span> and <span class="math notranslate nohighlight">\(\max(x_j)\)</span> return the minimum and maximum value of <span class="math notranslate nohighlight">\(x_j\)</span> over the data set, respectively.</p>
</section>
<section id="testing-the-means-squared-error-as-function-of-complexity">
<h2><span class="section-number">3.9. </span>Testing the Means Squared Error as function of Complexity<a class="headerlink" href="#testing-the-means-squared-error-as-function-of-complexity" title="Link to this heading">#</a></h2>
<h2><span class="section-number">3.7. </span>Testing the Means Squared Error as function of Complexity<a class="headerlink" href="#testing-the-means-squared-error-as-function-of-complexity" title="Link to this heading">#</a></h2>
<p>Before we proceed with a more detailed analysis of the so-called
Bias-Variance tradeoff, we present here an example of the relation
between model complexity and the mean squared error for the triaining
@@ -2394,10 +2201,10 @@ plt.show()
</div>
</section>
<section id="exercises">
<h2><span class="section-number">3.10. </span>Exercises<a class="headerlink" href="#exercises" title="Link to this heading">#</a></h2>
<h2><span class="section-number">3.8. </span>Exercises<a class="headerlink" href="#exercises" title="Link to this heading">#</a></h2>
</section>
<section id="exercise-1-setting-up-various-python-environments">
<h2><span class="section-number">3.11. </span>Exercise 1: Setting up various Python environments<a class="headerlink" href="#exercise-1-setting-up-various-python-environments" title="Link to this heading">#</a></h2>
<h2><span class="section-number">3.9. </span>Exercise 1: Setting up various Python environments<a class="headerlink" href="#exercise-1-setting-up-various-python-environments" title="Link to this heading">#</a></h2>
<p>The first exercise here is of a mere technical art. We want you to have</p>
<ul class="simple">
<li><p>git as a version control software and to establish a user account on a provider like GitHub. Other providers like GitLab etc are equally fine. You can also use the University of Oslo <a class="reference external" href="https://www.uio.no/tjenester/it/maskin/filer/versjonskontroll/github.html">GitHub facilities</a>.</p></li>
@@ -2453,7 +2260,7 @@ license.</p>
<p>We recommend using <strong>Anaconda</strong> if you are not too familiar with setting paths in a terminal environment.</p>
</section>
<section id="exercise-2-making-your-own-data-and-exploring-scikit-learn">
<h2><span class="section-number">3.12. </span>Exercise 2: making your own data and exploring scikit-learn<a class="headerlink" href="#exercise-2-making-your-own-data-and-exploring-scikit-learn" title="Link to this heading">#</a></h2>
<h2><span class="section-number">3.10. </span>Exercise 2: making your own data and exploring scikit-learn<a class="headerlink" href="#exercise-2-making-your-own-data-and-exploring-scikit-learn" title="Link to this heading">#</a></h2>
<p>We will generate our own dataset for a function <span class="math notranslate nohighlight">\(y(x)\)</span> where <span class="math notranslate nohighlight">\(x \in [0,1]\)</span> and defined by random numbers computed with the uniform distribution. The function <span class="math notranslate nohighlight">\(y\)</span> is a quadratic polynomial in <span class="math notranslate nohighlight">\(x\)</span> with added stochastic noise according to the normal distribution <span class="math notranslate nohighlight">\(\cal {N}(0,1)\)</span>.
The following simple Python instructions define our <span class="math notranslate nohighlight">\(x\)</span> and <span class="math notranslate nohighlight">\(y\)</span> values (with 100 data points).</p>
<div class="cell docutils container">
@@ -2538,7 +2345,7 @@ print(MSE(y_test,ypredict))
</div>
<!-- --- end solution of exercise --- --></section>
<section id="exercise-3-normalizing-our-data">
<h2><span class="section-number">3.13. </span>Exercise 3: Normalizing our data<a class="headerlink" href="#exercise-3-normalizing-our-data" title="Link to this heading">#</a></h2>
<h2><span class="section-number">3.11. </span>Exercise 3: Normalizing our data<a class="headerlink" href="#exercise-3-normalizing-our-data" title="Link to this heading">#</a></h2>
<p>A much used approach before starting to train the data is to preprocess our
data. Normally the data may need a rescaling and/or may be sensitive
to extreme values. Scaling the data renders our inputs much more
@@ -2658,7 +2465,7 @@ Perform an ordinary least squares and compute the means squared error and the <s
Add now a model which allows you to make polynomials up to degree <span class="math notranslate nohighlight">\(15\)</span>. Perform a standard OLS fitting of the training data and compute the MSE and <span class="math notranslate nohighlight">\(R2\)</span> for the training and test data and plot both test and training data MSE and <span class="math notranslate nohighlight">\(R2\)</span> as functions of the polynomial degree. Compare what you see with Figure 2.11 of Hastie et al. Comment your results. For which polynomial degree do you find an optimal MSE (smallest value)?</p>
</section>
<section id="exercise-4-adding-ridge-regression">
<h2><span class="section-number">3.14. </span>Exercise 4: Adding Ridge Regression<a class="headerlink" href="#exercise-4-adding-ridge-regression" title="Link to this heading">#</a></h2>
<h2><span class="section-number">3.12. </span>Exercise 4: Adding Ridge Regression<a class="headerlink" href="#exercise-4-adding-ridge-regression" title="Link to this heading">#</a></h2>
<p>This exercise is a continuation of exercise 2. We will use the same function to
generate our data set, still staying with a simple function <span class="math notranslate nohighlight">\(y(x)\)</span>
which we want to fit using linear regression, but now extending the
@@ -2793,7 +2600,7 @@ plt.show()
</div>
<!-- --- end solution of exercise --- --></section>
<section id="exercise-5-analytical-exercises">
<h2><span class="section-number">3.15. </span>Exercise 5: Analytical exercises<a class="headerlink" href="#exercise-5-analytical-exercises" title="Link to this heading">#</a></h2>
<h2><span class="section-number">3.13. </span>Exercise 5: Analytical exercises<a class="headerlink" href="#exercise-5-analytical-exercises" title="Link to this heading">#</a></h2>
<p>In this exercise we derive the expressions for various derivatives of
products of vectors and matrices. Such derivatives are central to the
optimization of various cost functions. Although we will often use
@@ -2956,16 +2763,14 @@ f_i =\sum_{j=0}^{n-1}a_{ij}x_j,
</ul>
</li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#splitting-our-data-in-training-and-test-data">3.5. Splitting our Data in Training and Test data</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#the-boston-housing-data-example">3.6. The Boston housing data example</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#housing-data-the-code">3.7. Housing data, the code</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#reducing-the-number-of-degrees-of-freedom-overarching-view">3.8. Reducing the number of degrees of freedom, overarching view</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#testing-the-means-squared-error-as-function-of-complexity">3.9. Testing the Means Squared Error as function of Complexity</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#exercises">3.10. Exercises</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#exercise-1-setting-up-various-python-environments">3.11. Exercise 1: Setting up various Python environments</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#exercise-2-making-your-own-data-and-exploring-scikit-learn">3.12. Exercise 2: making your own data and exploring scikit-learn</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#exercise-3-normalizing-our-data">3.13. Exercise 3: Normalizing our data</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#exercise-4-adding-ridge-regression">3.14. Exercise 4: Adding Ridge Regression</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#exercise-5-analytical-exercises">3.15. Exercise 5: Analytical exercises</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#reducing-the-number-of-degrees-of-freedom-overarching-view">3.6. Reducing the number of degrees of freedom, overarching view</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#testing-the-means-squared-error-as-function-of-complexity">3.7. Testing the Means Squared Error as function of Complexity</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#exercises">3.8. Exercises</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#exercise-1-setting-up-various-python-environments">3.9. Exercise 1: Setting up various Python environments</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#exercise-2-making-your-own-data-and-exploring-scikit-learn">3.10. Exercise 2: making your own data and exploring scikit-learn</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#exercise-3-normalizing-our-data">3.11. Exercise 3: Normalizing our data</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#exercise-4-adding-ridge-regression">3.12. Exercise 4: Adding Ridge Regression</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#exercise-5-analytical-exercises">3.13. Exercise 5: Analytical exercises</a></li>
</ul>
</nav></div>
doc/LectureNotes/_build/html/searchindex.js
+1 -1
View File
File diff suppressed because one or more lines are too long
doc/LectureNotes/_build/jupyter_execute/chapter1.ipynb
+287 -651
View File
File diff suppressed because it is too large Load Diff
doc/LectureNotes/chapter1.ipynb
+287 -651
View File
File diff suppressed because it is too large Load Diff