-The first exercise here is of a mere technical art. We want you to have
-
-
-
git as a version control software and to establish a user account on a provider like GitHub. Other providers like GitLab etc are equally fine. You can also use the University of Oslo GitHub facilities.
-
Install various Python packages
-
-
-We will make extensive use of Python as programming language and its
-myriad of available libraries. You will find
-IPython/Jupyter notebooks invaluable in your work. You can run R
-codes in the Jupyter/IPython notebooks, with the immediate benefit of
-visualizing your data. You can also use compiled languages like C++,
-Rust, Fortran etc if you prefer. The focus in these lectures will be
-on Python.
-
-
-If you have Python installed (we recommend Python3) and you feel
-pretty familiar with installing different packages, we recommend that
-you install the following Python packages via pip as
-
-
-For OSX users we recommend, after having installed Xcode, to
-install brew. Brew allows for a seamless installation of additional
-software via for example
-
-
-
brew install python3
-
-
-For Linux users, with its variety of distributions like for example the widely popular Ubuntu distribution,
-you can use pip as well and simply install Python as
-
-
-
sudo apt-get install python3 (or python for Python2.7)
-
-
-If you don't want to perform these operations separately and venture
-into the hassle of exploring how to set up dependencies and paths, we
-recommend two widely used distrubutions which set up all relevant
-dependencies for Python, namely
-
-
-
-which is an open source
-distribution of the Python and R programming languages for large-scale
-data processing, predictive analytics, and scientific computing, that
-aims to simplify package management and deployment. Package versions
-are managed by the package management system conda.
-
-
-
-is a Python
-distribution for scientific and analytic computing distribution and
-analysis environment, available for free and under a commercial
-license.
-
-
Write a Python code which reads the in the above mentioned file.
-
Use for example pandas to order your data and find out how many data points there are.
-
Set thereafter up the design matrix with dimensionality \( n\times p \) where \( p=4 \) and where you have defined a polynomial of degree \( p-1=3 \). Print the matrix and check that the numbers are correct.
-
-
-We recommend looking at the examples in the regression slides.
-
-
importos
-importnumpyasnp
-importpandasaspd
-importmatplotlib.pyplotasplt
-fromsklearn.model_selectionimport train_test_split
-# Where to save the figures and data files
-PROJECT_ROOT_DIR ="Results"
-FIGURE_ID ="Results/FigureFiles"
-DATA_ID ="DataFiles/"
-
-ifnot os.path.exists(PROJECT_ROOT_DIR):
- os.mkdir(PROJECT_ROOT_DIR)
-
-ifnot os.path.exists(FIGURE_ID):
- os.makedirs(FIGURE_ID)
-
-ifnot os.path.exists(DATA_ID):
- os.makedirs(DATA_ID)
-
-defimage_path(fig_id):
- return os.path.join(FIGURE_ID, fig_id)
-
-defdata_path(dat_id):
- return os.path.join(DATA_ID, dat_id)
-
-defsave_fig(fig_id):
- plt.savefig(image_path(fig_id) +".png", format='png')
-
-defR2(y_data, y_model):
- return1- np.sum((y_data - y_model) **2) / np.sum((y_data - np.mean(y_data)) **2)
-defMSE(y_data,y_model):
- n = np.size(y_model)
- return np.sum((y_data-y_model)**2)/n
-
-infile =open(data_path("EoS.csv"),'r')
-
-# Read the EoS data as csv file and organized into two arrays with density and energies
-EoS = pd.read_csv(infile, names=('Density', 'Energy'))
-EoS['Energy'] = pd.to_numeric(EoS['Energy'], errors='coerce')
-EoS = EoS.dropna()
-Energies = EoS['Energy']
-Density = EoS['Density']
-# The design matrix now as function of various polytrops
-X = np.zeros((len(Density),5))
-X[:,0] =1
-X[:,1] = Density**(2.0/3.0)
-X[:,2] = Density
-X[:,3] = Density**(4.0/3.0)
-X[:,4] = Density**(5.0/3.0)
-# We split the data in test and training data
-X_train, X_test, y_train, y_test = train_test_split(X, Energies, test_size=0.2)
-# matrix inversion to find beta
-beta = np.linalg.inv(X_train.T @ X_train) @ X_train.T @ y_train
-# and then make the prediction
-ytilde = X_train @ beta
-print("Training R2")
-print(R2(y_train,ytilde))
-print("Training MSE")
-print(MSE(y_train,ytilde))
-ypredict = X_test @ beta
-print("Test R2")
-print(R2(y_test,ypredict))
-print("Test MSE")
-print(MSE(y_test,ypredict))
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
Exercise 2: making your own data and exploring scikit-learn
-
-
-We will generate our own dataset for a function \( y(x) \) where \( x \in [0,1] \) and defined by random numbers computed with the uniform distribution. The function \( y \) is a quadratic polynomial in \( x \) with added stochastic noise according to the normal distribution \( \cal {N}(0,1) \).
-The following simple Python instructions define our \( x \) and \( y \) values (with 100 data points).
-
-
-
-
x = np.random.rand(100,1)
-y =2.0+5*x*x+0.1*np.random.randn(100,1)
-
-
-
Write your own code (following the examples under the regression slides) for computing the parametrization of the data set fitting a second-order polynomial.
-
Use thereafter scikit-learn (see again the examples in the regression slides) and compare with your own code.
-
Using scikit-learn, compute also the mean square error, a risk metric corresponding to the expected value of the squared (quadratic) error defined as
-
-
-$$ MSE(\hat{y},\hat{\tilde{y}}) = \frac{1}{n}
-\sum_{i=0}^{n-1}(y_i-\tilde{y}_i)^2,
-$$
-
-and the \( R^2 \) score function.
-If \( \tilde{\hat{y}}_i \) is the predicted value of the \( i-th \) sample and \( y_i \) is the corresponding true value, then the score \( R^2 \) is defined as
-$$
-R^2(\hat{y}, \tilde{\hat{y}}) = 1 - \frac{\sum_{i=0}^{n - 1} (y_i - \tilde{y}_i)^2}{\sum_{i=0}^{n - 1} (y_i - \bar{y})^2},
-$$
-
-where we have defined the mean value of \( \hat{y} \) as
-$$
-\bar{y} = \frac{1}{n} \sum_{i=0}^{n - 1} y_i.
-$$
-
-You can use the functionality included in scikit-learn. If you feel for it, you can use your own program and define functions which compute the above two functions.
-Discuss the meaning of these results. Try also to vary the coefficient in front of the added stochastic noise term and discuss the quality of the fits.
-
-
-The assumption we have made is
-that there exists a function \( f(\boldsymbol{x}) \) and a normal distributed error \( \boldsymbol{\varepsilon}\sim \mathcal{N}(0, \sigma^2) \)
-which describes our data
-$$
-\boldsymbol{y} = f(\boldsymbol{x})+\boldsymbol{\varepsilon}
-$$
-
-
-We then approximate this function with our model from the solution of the linear regression equations (ordinary least squares OLS), that is our
-function \( f \) is approximated by \( \boldsymbol{\tilde{y}} \) where we minimized \( (\boldsymbol{y}-\boldsymbol{\tilde{y}})^2 \), with
-$$
-\boldsymbol{\tilde{y}} = \boldsymbol{X}\boldsymbol{\beta}.
-$$
-
-The matrix \( \boldsymbol{X} \) is the so-called design matrix.
-
-
-a)
-Show that the expectation value of \( \boldsymbol{y} \) for a given element \( i \)
-$$
-\begin{align*}
-\mathbb{E}(y_i) & =\mathbf{X}_{i, \ast} \, \beta,
-\end{align*}
-$$
-
-and that
-its variance is
-$$
-\begin{align*} \mbox{Var}(y_i) & = \sigma^2.
-\end{align*}
-$$
-
-Hence, \( y_i \sim \mathcal{N}( \mathbf{X}_{i, \ast} \, \boldsymbol{\beta}, \sigma^2) \), that is \( \boldsymbol{y} \) follows a normal distribution with
-mean value \( \boldsymbol{X}\boldsymbol{\beta} \) and variance \( \sigma^2 \).
-
-
-
-$$
-\mathbb{E}(\boldsymbol{\beta}) = \mathbb{E}[ (\mathbf{X}^{\top} \mathbf{X})^{-1}\mathbf{X}^{T} \mathbf{Y}]=(\mathbf{X}^{T} \mathbf{X})^{-1}\mathbf{X}^{T} \mathbb{E}[ \mathbf{Y}]=(\mathbf{X}^{T} \mathbf{X})^{-1} \mathbf{X}^{T}\mathbf{X}\boldsymbol{\beta}=\boldsymbol{\beta}.
-$$
-
-This means that the estimator of the regression parameters is unbiased.
-
-
-
-
-
-
-
-
-
-
-c)
-Show finally that the variance of \( \boldsymbol{\beta} \) is
-$$
-\begin{eqnarray*}
-\mbox{Var}(\boldsymbol{\beta}) & = & \sigma^2 \, (\mathbf{X}^{T} \mathbf{X})^{-1}.
-\end{eqnarray*}
-$$
-
-
-where we have used that \( \mathbb{E} (\mathbf{Y} \mathbf{Y}^{T}) =
-\mathbf{X} \, \boldsymbol{\beta} \, \boldsymbol{\beta}^{T} \, \mathbf{X}^{T} +
-\sigma^2 \, \mathbf{I}_{nn} \). From \( \mbox{Var}(\boldsymbol{\beta}) = \sigma^2
-\, (\mathbf{X}^{T} \mathbf{X})^{-1} \), one obtains an estimate of the
-variance of the estimate of the \( j \)-th regression coefficient:
-\( \boldsymbol{\sigma}^2 (\hat{\beta}_j ) = \boldsymbol{\sigma}^2 \sqrt{
-[(\mathbf{X}^{T} \mathbf{X})^{-1}]_{jj} } \). This may be used to
-construct a confidence interval for the estimates.
-
-
-In a similar way, we can obtain analytical expressions for say the
-expectation values of the parameters \( \boldsymbol{\beta} \) and their variance
-when we employ Ridge regression, allowing us again to define a confidence interval.
-
-
-The first exercise here is of a mere technical art. We want you to have
-
-
-
git as a version control software and to establish a user account on a provider like GitHub. Other providers like GitLab etc are equally fine. You can also use the University of Oslo GitHub facilities.
-
Install various Python packages
-
-
-We will make extensive use of Python as programming language and its
-myriad of available libraries. You will find
-IPython/Jupyter notebooks invaluable in your work. You can run R
-codes in the Jupyter/IPython notebooks, with the immediate benefit of
-visualizing your data. You can also use compiled languages like C++,
-Rust, Fortran etc if you prefer. The focus in these lectures will be
-on Python.
-
-
-If you have Python installed (we recommend Python3) and you feel
-pretty familiar with installing different packages, we recommend that
-you install the following Python packages via pip as
-
-
-For OSX users we recommend, after having installed Xcode, to
-install brew. Brew allows for a seamless installation of additional
-software via for example
-
-
-
brew install python3
-
-
-For Linux users, with its variety of distributions like for example the widely popular Ubuntu distribution,
-you can use pip as well and simply install Python as
-
-
-
sudo apt-get install python3 (or python for Python2.7)
-
-
-If you don't want to perform these operations separately and venture
-into the hassle of exploring how to set up dependencies and paths, we
-recommend two widely used distrubutions which set up all relevant
-dependencies for Python, namely
-
-
-
-which is an open source
-distribution of the Python and R programming languages for large-scale
-data processing, predictive analytics, and scientific computing, that
-aims to simplify package management and deployment. Package versions
-are managed by the package management system conda.
-
-
-
-is a Python
-distribution for scientific and analytic computing distribution and
-analysis environment, available for free and under a commercial
-license.
-
-
Write a Python code which reads the in the above mentioned file.
-
Use for example pandas to order your data and find out how many data points there are.
-
Set thereafter up the design matrix with dimensionality \( n\times p \) where \( p=4 \) and where you have defined a polynomial of degree \( p-1=3 \). Print the matrix and check that the numbers are correct.
-
-
-We recommend looking at the examples in the regression slides.
-
-
-
-Solution.
-
-
-
-
importos
-importnumpyasnp
-importpandasaspd
-importmatplotlib.pyplotasplt
-fromsklearn.model_selectionimport train_test_split
-# Where to save the figures and data files
-PROJECT_ROOT_DIR ="Results"
-FIGURE_ID ="Results/FigureFiles"
-DATA_ID ="DataFiles/"
-
-ifnot os.path.exists(PROJECT_ROOT_DIR):
- os.mkdir(PROJECT_ROOT_DIR)
-
-ifnot os.path.exists(FIGURE_ID):
- os.makedirs(FIGURE_ID)
-
-ifnot os.path.exists(DATA_ID):
- os.makedirs(DATA_ID)
-
-defimage_path(fig_id):
- return os.path.join(FIGURE_ID, fig_id)
-
-defdata_path(dat_id):
- return os.path.join(DATA_ID, dat_id)
-
-defsave_fig(fig_id):
- plt.savefig(image_path(fig_id) +".png", format='png')
-
-defR2(y_data, y_model):
- return1- np.sum((y_data - y_model) **2) / np.sum((y_data - np.mean(y_data)) **2)
-defMSE(y_data,y_model):
- n = np.size(y_model)
- return np.sum((y_data-y_model)**2)/n
-
-infile =open(data_path("EoS.csv"),'r')
-
-# Read the EoS data as csv file and organized into two arrays with density and energies
-EoS = pd.read_csv(infile, names=('Density', 'Energy'))
-EoS['Energy'] = pd.to_numeric(EoS['Energy'], errors='coerce')
-EoS = EoS.dropna()
-Energies = EoS['Energy']
-Density = EoS['Density']
-# The design matrix now as function of various polytrops
-X = np.zeros((len(Density),5))
-X[:,0] =1
-X[:,1] = Density**(2.0/3.0)
-X[:,2] = Density
-X[:,3] = Density**(4.0/3.0)
-X[:,4] = Density**(5.0/3.0)
-# We split the data in test and training data
-X_train, X_test, y_train, y_test = train_test_split(X, Energies, test_size=0.2)
-# matrix inversion to find beta
-beta = np.linalg.inv(X_train.T @ X_train) @ X_train.T @ y_train
-# and then make the prediction
-ytilde = X_train @ beta
-print("Training R2")
-print(R2(y_train,ytilde))
-print("Training MSE")
-print(MSE(y_train,ytilde))
-ypredict = X_test @ beta
-print("Test R2")
-print(R2(y_test,ypredict))
-print("Test MSE")
-print(MSE(y_test,ypredict))
-
-
-
-
-
-
-
-
-
-
-
Exercise 2: making your own data and exploring scikit-learn
-
-
-We will generate our own dataset for a function \( y(x) \) where \( x \in [0,1] \) and defined by random numbers computed with the uniform distribution. The function \( y \) is a quadratic polynomial in \( x \) with added stochastic noise according to the normal distribution \( \cal {N}(0,1) \).
-The following simple Python instructions define our \( x \) and \( y \) values (with 100 data points).
-
-
-
-
x = np.random.rand(100,1)
-y =2.0+5*x*x+0.1*np.random.randn(100,1)
-
-
-
Write your own code (following the examples under the regression slides) for computing the parametrization of the data set fitting a second-order polynomial.
-
Use thereafter scikit-learn (see again the examples in the regression slides) and compare with your own code.
-
Using scikit-learn, compute also the mean square error, a risk metric corresponding to the expected value of the squared (quadratic) error defined as
-
-
-$$ MSE(\hat{y},\hat{\tilde{y}}) = \frac{1}{n}
-\sum_{i=0}^{n-1}(y_i-\tilde{y}_i)^2,
-$$
-
-and the \( R^2 \) score function.
-If \( \tilde{\hat{y}}_i \) is the predicted value of the \( i-th \) sample and \( y_i \) is the corresponding true value, then the score \( R^2 \) is defined as
-$$
-R^2(\hat{y}, \tilde{\hat{y}}) = 1 - \frac{\sum_{i=0}^{n - 1} (y_i - \tilde{y}_i)^2}{\sum_{i=0}^{n - 1} (y_i - \bar{y})^2},
-$$
-
-where we have defined the mean value of \( \hat{y} \) as
-$$
-\bar{y} = \frac{1}{n} \sum_{i=0}^{n - 1} y_i.
-$$
-
-You can use the functionality included in scikit-learn. If you feel for it, you can use your own program and define functions which compute the above two functions.
-Discuss the meaning of these results. Try also to vary the coefficient in front of the added stochastic noise term and discuss the quality of the fits.
-
-
-
-Solution.
-The code here is an example of where we define our own design matrix and fit parameters \( \beta \).
-
-
-
-
importos
-importnumpyasnp
-importpandasaspd
-importmatplotlib.pyplotasplt
-fromsklearn.model_selectionimport train_test_split
-
-defsave_fig(fig_id):
- plt.savefig(image_path(fig_id) +".png", format='png')
-
-defR2(y_data, y_model):
- return1- np.sum((y_data - y_model) **2) / np.sum((y_data - np.mean(y_data)) **2)
-defMSE(y_data,y_model):
- n = np.size(y_model)
- return np.sum((y_data-y_model)**2)/n
-
-x = np.random.rand(100)
-y =2.0+5*x*x+0.1*np.random.randn(100)
-
-
-# The design matrix now as function of a given polynomial
-X = np.zeros((len(x),3))
-X[:,0] =1.0
-X[:,1] = x
-X[:,2] = x**2
-# We split the data in test and training data
-X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)
-# matrix inversion to find beta
-beta = np.linalg.inv(X_train.T @ X_train) @ X_train.T @ y_train
-print(beta)
-# and then make the prediction
-ytilde = X_train @ beta
-print("Training R2")
-print(R2(y_train,ytilde))
-print("Training MSE")
-print(MSE(y_train,ytilde))
-ypredict = X_test @ beta
-print("Test R2")
-print(R2(y_test,ypredict))
-print("Test MSE")
-print(MSE(y_test,ypredict))
-
-
-
-
-
-
-
-
-
-
-
Exercise 3: mean values and variances in linear regression
-The assumption we have made is
-that there exists a function \( f(\boldsymbol{x}) \) and a normal distributed error \( \boldsymbol{\varepsilon}\sim \mathcal{N}(0, \sigma^2) \)
-which describes our data
-$$
-\boldsymbol{y} = f(\boldsymbol{x})+\boldsymbol{\varepsilon}
-$$
-
-
-We then approximate this function with our model from the solution of the linear regression equations (ordinary least squares OLS), that is our
-function \( f \) is approximated by \( \boldsymbol{\tilde{y}} \) where we minimized \( (\boldsymbol{y}-\boldsymbol{\tilde{y}})^2 \), with
-$$
-\boldsymbol{\tilde{y}} = \boldsymbol{X}\boldsymbol{\beta}.
-$$
-
-The matrix \( \boldsymbol{X} \) is the so-called design matrix.
-
-
-a)
-Show that the expectation value of \( \boldsymbol{y} \) for a given element \( i \)
-$$
-\begin{align*}
-\mathbb{E}(y_i) & =\mathbf{X}_{i, \ast} \, \beta,
-\end{align*}
-$$
-
-and that
-its variance is
-$$
-\begin{align*} \mbox{Var}(y_i) & = \sigma^2.
-\end{align*}
-$$
-
-Hence, \( y_i \sim \mathcal{N}( \mathbf{X}_{i, \ast} \, \boldsymbol{\beta}, \sigma^2) \), that is \( \boldsymbol{y} \) follows a normal distribution with
-mean value \( \boldsymbol{X}\boldsymbol{\beta} \) and variance \( \sigma^2 \).
-
-
-
-Solution.
-We can calculate the expectation value of \( \boldsymbol{y} \) for a given element \( i \)
-$$
-\begin{align*}
-\mathbb{E}(y_i) & =
-\mathbb{E}(\mathbf{X}_{i, \ast} \, \boldsymbol{\beta}) + \mathbb{E}(\varepsilon_i)
-\, \, \, = \, \, \, \mathbf{X}_{i, \ast} \, \beta,
-\end{align*}
-$$
-
-while
-its variance is
-$$
-\begin{align*} \mbox{Var}(y_i) & = \mathbb{E} \{ [y_i
-- \mathbb{E}(y_i)]^2 \} \, \, \, = \, \, \, \mathbb{E} ( y_i^2 ) -
-[\mathbb{E}(y_i)]^2 \\ & = \mathbb{E} [ ( \mathbf{X}_{i, \ast} \,
-\beta + \varepsilon_i )^2] - ( \mathbf{X}_{i, \ast} \, \boldsymbol{\beta})^2 \\ &
-= \mathbb{E} [ ( \mathbf{X}_{i, \ast} \, \boldsymbol{\beta})^2 + 2 \varepsilon_i
-\mathbf{X}_{i, \ast} \, \boldsymbol{\beta} + \varepsilon_i^2 ] - ( \mathbf{X}_{i,
-\ast} \, \beta)^2 \\ & = ( \mathbf{X}_{i, \ast} \, \boldsymbol{\beta})^2 + 2
-\mathbb{E}(\varepsilon_i) \mathbf{X}_{i, \ast} \, \boldsymbol{\beta} +
-\mathbb{E}(\varepsilon_i^2 ) - ( \mathbf{X}_{i, \ast} \, \boldsymbol{\beta})^2
-\\ & = \mathbb{E}(\varepsilon_i^2 ) \, \, \, = \, \, \,
-\mbox{Var}(\varepsilon_i) \, \, \, = \, \, \, \sigma^2.
-\end{align*}
-$$
-
-Hence, \( y_i \sim \mathcal{N}( \mathbf{X}_{i, \ast} \, \boldsymbol{\beta}, \sigma^2) \), that is \( \boldsymbol{y} \) follows a normal distribution with
-mean value \( \boldsymbol{X}\boldsymbol{\beta} \) and variance \( \sigma^2 \) (not be confused with the singular values of the SVD).
-
-
-
-
-
-b)
-With the OLS expressions for the parameters \( \boldsymbol{\beta} \) show that
-$$
-\mathbb{E}(\boldsymbol{\beta}) = \boldsymbol{\beta}.
-$$
-
-
-
-Solution.
-$$
-\mathbb{E}(\boldsymbol{\beta}) = \mathbb{E}[ (\mathbf{X}^{\top} \mathbf{X})^{-1}\mathbf{X}^{T} \mathbf{Y}]=(\mathbf{X}^{T} \mathbf{X})^{-1}\mathbf{X}^{T} \mathbb{E}[ \mathbf{Y}]=(\mathbf{X}^{T} \mathbf{X})^{-1} \mathbf{X}^{T}\mathbf{X}\boldsymbol{\beta}=\boldsymbol{\beta}.
-$$
-
-This means that the estimator of the regression parameters is unbiased.
-
-
-
-
-
-c)
-Show finally that the variance of \( \boldsymbol{\beta} \) is
-$$
-\begin{eqnarray*}
-\mbox{Var}(\boldsymbol{\beta}) & = & \sigma^2 \, (\mathbf{X}^{T} \mathbf{X})^{-1}.
-\end{eqnarray*}
-$$
-
-
-where we have used that \( \mathbb{E} (\mathbf{Y} \mathbf{Y}^{T}) =
-\mathbf{X} \, \boldsymbol{\beta} \, \boldsymbol{\beta}^{T} \, \mathbf{X}^{T} +
-\sigma^2 \, \mathbf{I}_{nn} \). From \( \mbox{Var}(\boldsymbol{\beta}) = \sigma^2
-\, (\mathbf{X}^{T} \mathbf{X})^{-1} \), one obtains an estimate of the
-variance of the estimate of the \( j \)-th regression coefficient:
-\( \boldsymbol{\sigma}^2 (\hat{\beta}_j ) = \boldsymbol{\sigma}^2 \sqrt{
-[(\mathbf{X}^{T} \mathbf{X})^{-1}]_{jj} } \). This may be used to
-construct a confidence interval for the estimates.
-
-
-In a similar way, we can obtain analytical expressions for say the
-expectation values of the parameters \( \boldsymbol{\beta} \) and their variance
-when we employ Ridge regression, allowing us again to define a confidence interval.
-
-
-
-
-
-
-
-
diff --git a/doc/src/Projects/2020/Exercises/hw1.p.tex b/doc/src/Projects/2020/Exercises/hw1.p.tex
deleted file mode 100644
index 00f94739c..000000000
--- a/doc/src/Projects/2020/Exercises/hw1.p.tex
+++ /dev/null
@@ -1,593 +0,0 @@
-%%
-%% Automatically generated file from DocOnce source
-%% (https://github.com/hplgit/doconce/)
-%%
-%%
-% #ifdef PTEX2TEX_EXPLANATION
-%%
-%% The file follows the ptex2tex extended LaTeX format, see
-%% ptex2tex: http://code.google.com/p/ptex2tex/
-%%
-%% Run
-%% ptex2tex myfile
-%% or
-%% doconce ptex2tex myfile
-%%
-%% to turn myfile.p.tex into an ordinary LaTeX file myfile.tex.
-%% (The ptex2tex program: http://code.google.com/p/ptex2tex)
-%% Many preprocess options can be added to ptex2tex or doconce ptex2tex
-%%
-%% ptex2tex -DMINTED myfile
-%% doconce ptex2tex myfile envir=minted
-%%
-%% ptex2tex will typeset code environments according to a global or local
-%% .ptex2tex.cfg configure file. doconce ptex2tex will typeset code
-%% according to options on the command line (just type doconce ptex2tex to
-%% see examples). If doconce ptex2tex has envir=minted, it enables the
-%% minted style without needing -DMINTED.
-% #endif
-
-% #define PREAMBLE
-
-% #ifdef PREAMBLE
-%-------------------- begin preamble ----------------------
-
-\documentclass[%
-oneside, % oneside: electronic viewing, twoside: printing
-final, % draft: marks overfull hboxes, figures with paths
-10pt]{article}
-
-\listfiles % print all files needed to compile this document
-
-\usepackage{relsize,makeidx,color,setspace,amsmath,amsfonts,amssymb}
-\usepackage[table]{xcolor}
-\usepackage{bm,ltablex,microtype}
-
-\usepackage[pdftex]{graphicx}
-
-\usepackage{ptex2tex}
-% #ifdef MINTED
-\usepackage{minted}
-\usemintedstyle{default}
-% #endif
-
-\usepackage[T1]{fontenc}
-%\usepackage[latin1]{inputenc}
-\usepackage{ucs}
-\usepackage[utf8x]{inputenc}
-
-\usepackage{lmodern} % Latin Modern fonts derived from Computer Modern
-
-% Hyperlinks in PDF:
-\definecolor{linkcolor}{rgb}{0,0,0.4}
-\usepackage{hyperref}
-\hypersetup{
- breaklinks=true,
- colorlinks=true,
- linkcolor=linkcolor,
- urlcolor=linkcolor,
- citecolor=black,
- filecolor=black,
- %filecolor=blue,
- pdfmenubar=true,
- pdftoolbar=true,
- bookmarksdepth=3 % Uncomment (and tweak) for PDF bookmarks with more levels than the TOC
- }
-%\hyperbaseurl{} % hyperlinks are relative to this root
-
-\setcounter{tocdepth}{2} % levels in table of contents
-
-% --- fancyhdr package for fancy headers ---
-\usepackage{fancyhdr}
-\fancyhf{} % sets both header and footer to nothing
-\renewcommand{\headrulewidth}{0pt}
-\fancyfoot[LE,RO]{\thepage}
-% Ensure copyright on titlepage (article style) and chapter pages (book style)
-\fancypagestyle{plain}{
- \fancyhf{}
- \fancyfoot[C]{{\footnotesize \copyright\ 1999-2020, "Data Analysis and Machine Learning FYS-STK3155/FYS4155":"http://www.uio.no/studier/emner/matnat/fys/FYS3155/index-eng.html". Released under CC Attribution-NonCommercial 4.0 license}}
-% \renewcommand{\footrulewidth}{0mm}
- \renewcommand{\headrulewidth}{0mm}
-}
-% Ensure copyright on titlepages with \thispagestyle{empty}
-\fancypagestyle{empty}{
- \fancyhf{}
- \fancyfoot[C]{{\footnotesize \copyright\ 1999-2020, "Data Analysis and Machine Learning FYS-STK3155/FYS4155":"http://www.uio.no/studier/emner/matnat/fys/FYS3155/index-eng.html". Released under CC Attribution-NonCommercial 4.0 license}}
- \renewcommand{\footrulewidth}{0mm}
- \renewcommand{\headrulewidth}{0mm}
-}
-
-\pagestyle{fancy}
-
-
-% prevent orhpans and widows
-\clubpenalty = 10000
-\widowpenalty = 10000
-
-\newenvironment{doconceexercise}{}{}
-\newcounter{doconceexercisecounter}
-
-
-% ------ header in subexercises ------
-%\newcommand{\subex}[1]{\paragraph{#1}}
-%\newcommand{\subex}[1]{\par\vspace{1.7mm}\noindent{\bf #1}\ \ }
-\makeatletter
-% 1.5ex is the spacing above the header, 0.5em the spacing after subex title
-\newcommand\subex{\@startsection{paragraph}{4}{\z@}%
- {1.5ex\@plus1ex \@minus.2ex}%
- {-0.5em}%
- {\normalfont\normalsize\bfseries}}
-\makeatother
-
-
-% --- end of standard preamble for documents ---
-
-
-% insert custom LaTeX commands...
-
-\raggedbottom
-\makeindex
-\usepackage[totoc]{idxlayout} % for index in the toc
-\usepackage[nottoc]{tocbibind} % for references/bibliography in the toc
-
-%-------------------- end preamble ----------------------
-
-\begin{document}
-
-% matching end for #ifdef PREAMBLE
-% #endif
-
-\newcommand{\exercisesection}[1]{\subsection*{#1}}
-
-
-% ------------------- main content ----------------------
-
-
-
-% ----------------- title -------------------------
-
-\thispagestyle{empty}
-
-\begin{center}
-{\LARGE\bf
-\begin{spacing}{1.25}
-Homework 1 Fall Semester 2020
-\end{spacing}
-}
-\end{center}
-
-% ----------------- author(s) -------------------------
-
-\begin{center}
-{\bf \href{{http://www.uio.no/studier/emner/matnat/fys/FYS3155/index-eng.html}}{Data Analysis and Machine Learning FYS-STK3155/FYS4155}}
-\end{center}
-
- \begin{center}
-% List of all institutions:
-\centerline{{\small Department of Physics, University of Oslo, Norway}}
-\end{center}
-
-% ----------------- end author(s) -------------------------
-
-% --- begin date ---
-\begin{center}
-Aug 19, 2020
-\end{center}
-% --- end date ---
-
-\vspace{1cm}
-
-
-\subsection{Exercise, Setting up various Python environments}
-
-The first exercise here is of a mere technical art. We want you to have
-\begin{itemize}
-\item git as a version control software and to establish a user account on a provider like GitHub. Other providers like GitLab etc are equally fine. You can also use the University of Oslo \href{{https://www.uio.no/tjenester/it/maskin/filer/versjonskontroll/github.html}}{GitHub facilities}.
-
-\item Install various Python packages
-\end{itemize}
-
-\noindent
-We will make extensive use of Python as programming language and its
-myriad of available libraries. You will find
-IPython/Jupyter notebooks invaluable in your work. You can run \textbf{R}
-codes in the Jupyter/IPython notebooks, with the immediate benefit of
-visualizing your data. You can also use compiled languages like C++,
-Rust, Fortran etc if you prefer. The focus in these lectures will be
-on Python.
-
-If you have Python installed (we recommend Python3) and you feel
-pretty familiar with installing different packages, we recommend that
-you install the following Python packages via \textbf{pip} as
-
-\begin{enumerate}
-\item pip install numpy scipy matplotlib ipython scikit-learn sympy pandas pillow
-\end{enumerate}
-
-\noindent
-For \textbf{Tensorflow}, we recommend following the instructions in the text of
-\href{{http://shop.oreilly.com/product/0636920052289.do}}{Aurelien Geron, Hands‑On Machine Learning with Scikit‑Learn and TensorFlow, O'Reilly}
-
-We will come back to \textbf{tensorflow} later.
-
-For Python3, replace \textbf{pip} with \textbf{pip3}.
-
-For OSX users we recommend, after having installed Xcode, to
-install \textbf{brew}. Brew allows for a seamless installation of additional
-software via for example
-
-\begin{enumerate}
-\item brew install python3
-\end{enumerate}
-
-\noindent
-For Linux users, with its variety of distributions like for example the widely popular Ubuntu distribution,
-you can use \textbf{pip} as well and simply install Python as
-
-\begin{enumerate}
-\item sudo apt-get install python3 (or python for Python2.7)
-\end{enumerate}
-
-\noindent
-If you don't want to perform these operations separately and venture
-into the hassle of exploring how to set up dependencies and paths, we
-recommend two widely used distrubutions which set up all relevant
-dependencies for Python, namely
-
-\begin{itemize}
-\item \href{{https://docs.anaconda.com/}}{Anaconda},
-\end{itemize}
-
-\noindent
-which is an open source
-distribution of the Python and R programming languages for large-scale
-data processing, predictive analytics, and scientific computing, that
-aims to simplify package management and deployment. Package versions
-are managed by the package management system \textbf{conda}.
-
-\begin{itemize}
-\item \href{{https://www.enthought.com/product/canopy/}}{Enthought canopy}
-\end{itemize}
-
-\noindent
-is a Python
-distribution for scientific and analytic computing distribution and
-analysis environment, available for free and under a commercial
-license.
-
-We recommend using \textbf{Anaconda}.
-
-
-
-
-% --- begin exercise ---
-\begin{doconceexercise}
-\refstepcounter{doconceexercisecounter}
-
-\exercisesection{Exercise \thedoconceexercisecounter: Our first Python encounter}
-
-
-This exercise has as its aim to write a small program which reads in data from a \textbf{csv} file on the equation of state for dense nuclear matter. The file is localized at \href{{https://github.com/mhjensen/MachineLearningMSU-FRIB2020/blob/master/doc/pub/Regression/ipynb/datafiles/EoS.csv}}{\nolinkurl{https://github.com/mhjensen/MachineLearningMSU-FRIB2020/blob/master/doc/pub/Regression/ipynb/datafiles/EoS.csv}}. Thereafter you will have to set up the design matrix $\bm{X}$ for the $n$
-datapoints and a polynomial of degree $3$. The steps are:
-\begin{itemize}
-\item Write a Python code which reads the in the above mentioned file.
-
-\item Use for example \textbf{pandas} to order your data and find out how many data points there are.
-
-\item Set thereafter up the design matrix with dimensionality $n\times p$ where $p=4$ and where you have defined a polynomial of degree $p-1=3$. Print the matrix and check that the numbers are correct.
-\end{itemize}
-
-\noindent
-We recommend looking at the examples in the \href{{https://compphysics.github.io/MachineLearning/doc/pub/Regression/html/Regression-bs.html}}{regression slides}.
-
-
-% --- begin solution of exercise ---
-\paragraph{Solution.}
-\bpycod
-import os
-import numpy as np
-import pandas as pd
-import matplotlib.pyplot as plt
-from sklearn.model_selection import train_test_split
-# Where to save the figures and data files
-PROJECT_ROOT_DIR = "Results"
-FIGURE_ID = "Results/FigureFiles"
-DATA_ID = "DataFiles/"
-
-if not os.path.exists(PROJECT_ROOT_DIR):
- os.mkdir(PROJECT_ROOT_DIR)
-
-if not os.path.exists(FIGURE_ID):
- os.makedirs(FIGURE_ID)
-
-if not os.path.exists(DATA_ID):
- os.makedirs(DATA_ID)
-
-def image_path(fig_id):
- return os.path.join(FIGURE_ID, fig_id)
-
-def data_path(dat_id):
- return os.path.join(DATA_ID, dat_id)
-
-def save_fig(fig_id):
- plt.savefig(image_path(fig_id) + ".png", format='png')
-
-def R2(y_data, y_model):
- return 1 - np.sum((y_data - y_model) ** 2) / np.sum((y_data - np.mean(y_data)) ** 2)
-def MSE(y_data,y_model):
- n = np.size(y_model)
- return np.sum((y_data-y_model)**2)/n
-
-infile = open(data_path("EoS.csv"),'r')
-
-# Read the EoS data as csv file and organized into two arrays with density and energies
-EoS = pd.read_csv(infile, names=('Density', 'Energy'))
-EoS['Energy'] = pd.to_numeric(EoS['Energy'], errors='coerce')
-EoS = EoS.dropna()
-Energies = EoS['Energy']
-Density = EoS['Density']
-# The design matrix now as function of various polytrops
-X = np.zeros((len(Density),5))
-X[:,0] = 1
-X[:,1] = Density**(2.0/3.0)
-X[:,2] = Density
-X[:,3] = Density**(4.0/3.0)
-X[:,4] = Density**(5.0/3.0)
-# We split the data in test and training data
-X_train, X_test, y_train, y_test = train_test_split(X, Energies, test_size=0.2)
-# matrix inversion to find beta
-beta = np.linalg.inv(X_train.T @ X_train) @ X_train.T @ y_train
-# and then make the prediction
-ytilde = X_train @ beta
-print("Training R2")
-print(R2(y_train,ytilde))
-print("Training MSE")
-print(MSE(y_train,ytilde))
-ypredict = X_test @ beta
-print("Test R2")
-print(R2(y_test,ypredict))
-print("Test MSE")
-print(MSE(y_test,ypredict))
-\epycod
-
-% --- end solution of exercise ---
-
-\end{doconceexercise}
-% --- end exercise ---
-
-
-
-
-% --- begin exercise ---
-\begin{doconceexercise}
-\refstepcounter{doconceexercisecounter}
-
-\exercisesection{Exercise \thedoconceexercisecounter: making your own data and exploring scikit-learn}
-
-
-We will generate our own dataset for a function $y(x)$ where $x \in [0,1]$ and defined by random numbers computed with the uniform distribution. The function $y$ is a quadratic polynomial in $x$ with added stochastic noise according to the normal distribution $\cal {N}(0,1)$.
-The following simple Python instructions define our $x$ and $y$ values (with 100 data points).
-\bpycod
-x = np.random.rand(100,1)
-y = 2.0+5*x*x+0.1*np.random.randn(100,1)
-\epycod
-
-\begin{enumerate}
-\item Write your own code (following the examples under the \href{{https://compphysics.github.io/MachineLearningECT/doc/pub/Day1/html/Day1-bs.html}}{regression slides}) for computing the parametrization of the data set fitting a second-order polynomial.
-
-\item Use thereafter \textbf{scikit-learn} (see again the examples in the regression slides) and compare with your own code.
-
-\item Using scikit-learn, compute also the mean square error, a risk metric corresponding to the expected value of the squared (quadratic) error defined as
-\end{enumerate}
-
-\noindent
-\[ MSE(\hat{y},\hat{\tilde{y}}) = \frac{1}{n}
-\sum_{i=0}^{n-1}(y_i-\tilde{y}_i)^2,
-\]
-and the $R^2$ score function.
-If $\tilde{\hat{y}}_i$ is the predicted value of the $i-th$ sample and $y_i$ is the corresponding true value, then the score $R^2$ is defined as
-\[
-R^2(\hat{y}, \tilde{\hat{y}}) = 1 - \frac{\sum_{i=0}^{n - 1} (y_i - \tilde{y}_i)^2}{\sum_{i=0}^{n - 1} (y_i - \bar{y})^2},
-\]
-where we have defined the mean value of $\hat{y}$ as
-\[
-\bar{y} = \frac{1}{n} \sum_{i=0}^{n - 1} y_i.
-\]
-You can use the functionality included in scikit-learn. If you feel for it, you can use your own program and define functions which compute the above two functions.
-Discuss the meaning of these results. Try also to vary the coefficient in front of the added stochastic noise term and discuss the quality of the fits.
-
-
-% --- begin solution of exercise ---
-\paragraph{Solution.}
-The code here is an example of where we define our own design matrix and fit parameters $\beta$.
-\bpycod
-import os
-import numpy as np
-import pandas as pd
-import matplotlib.pyplot as plt
-from sklearn.model_selection import train_test_split
-
-def save_fig(fig_id):
- plt.savefig(image_path(fig_id) + ".png", format='png')
-
-def R2(y_data, y_model):
- return 1 - np.sum((y_data - y_model) ** 2) / np.sum((y_data - np.mean(y_data)) ** 2)
-def MSE(y_data,y_model):
- n = np.size(y_model)
- return np.sum((y_data-y_model)**2)/n
-
-x = np.random.rand(100)
-y = 2.0+5*x*x+0.1*np.random.randn(100)
-
-
-# The design matrix now as function of a given polynomial
-X = np.zeros((len(x),3))
-X[:,0] = 1.0
-X[:,1] = x
-X[:,2] = x**2
-# We split the data in test and training data
-X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)
-# matrix inversion to find beta
-beta = np.linalg.inv(X_train.T @ X_train) @ X_train.T @ y_train
-print(beta)
-# and then make the prediction
-ytilde = X_train @ beta
-print("Training R2")
-print(R2(y_train,ytilde))
-print("Training MSE")
-print(MSE(y_train,ytilde))
-ypredict = X_test @ beta
-print("Test R2")
-print(R2(y_test,ypredict))
-print("Test MSE")
-print(MSE(y_test,ypredict))
-\epycod
-
-% --- end solution of exercise ---
-
-\end{doconceexercise}
-% --- end exercise ---
-
-
-
-
-% --- begin exercise ---
-\begin{doconceexercise}
-\refstepcounter{doconceexercisecounter}
-
-\exercisesection{Exercise \thedoconceexercisecounter: mean values and variances in linear regression}
-
-
-This exercise deals with various mean values ad variances in linear regression method (here it may be useful to look up chapter 3, equation (3.8) of \href{{https://www.springer.com/gp/book/9780387848570}}{Trevor Hastie, Robert Tibshirani, Jerome H. Friedman, The Elements of Statistical Learning, Springer}).
-
-The assumption we have made is
-that there exists a function $f(\bm{x})$ and a normal distributed error $\bm{\varepsilon}\sim \mathcal{N}(0, \sigma^2)$
-which describes our data
-\[
-\bm{y} = f(\bm{x})+\bm{\varepsilon}
-\]
-
-We then approximate this function with our model from the solution of the linear regression equations (ordinary least squares OLS), that is our
-function $f$ is approximated by $\bm{\tilde{y}}$ where we minimized $(\bm{y}-\bm{\tilde{y}})^2$, with
-\[
-\bm{\tilde{y}} = \bm{X}\bm{\beta}.
-\]
-The matrix $\bm{X}$ is the so-called design matrix.
-
-
-\subex{a)}
-Show that the expectation value of $\bm{y}$ for a given element $i$
-\begin{align*}
-\mathbb{E}(y_i) & =\mathbf{X}_{i, \ast} \, \beta,
-\end{align*}
-and that
-its variance is
-\begin{align*} \mbox{Var}(y_i) & = \sigma^2.
-\end{align*}
-Hence, $y_i \sim \mathcal{N}( \mathbf{X}_{i, \ast} \, \bm{\beta}, \sigma^2)$, that is $\bm{y}$ follows a normal distribution with
-mean value $\bm{X}\bm{\beta}$ and variance $\sigma^2$.
-
-
-% --- begin solution of exercise ---
-\paragraph{Solution.}
-We can calculate the expectation value of $\bm{y}$ for a given element $i$
-\begin{align*}
-\mathbb{E}(y_i) & =
-\mathbb{E}(\mathbf{X}_{i, \ast} \, \bm{\beta}) + \mathbb{E}(\varepsilon_i)
-\, \, \, = \, \, \, \mathbf{X}_{i, \ast} \, \beta,
-\end{align*}
-while
-its variance is
-\begin{align*} \mbox{Var}(y_i) & = \mathbb{E} \{ [y_i
-- \mathbb{E}(y_i)]^2 \} \, \, \, = \, \, \, \mathbb{E} ( y_i^2 ) -
-[\mathbb{E}(y_i)]^2 \\ & = \mathbb{E} [ ( \mathbf{X}_{i, \ast} \,
-\beta + \varepsilon_i )^2] - ( \mathbf{X}_{i, \ast} \, \bm{\beta})^2 \\ &
-= \mathbb{E} [ ( \mathbf{X}_{i, \ast} \, \bm{\beta})^2 + 2 \varepsilon_i
-\mathbf{X}_{i, \ast} \, \bm{\beta} + \varepsilon_i^2 ] - ( \mathbf{X}_{i,
-\ast} \, \beta)^2 \\ & = ( \mathbf{X}_{i, \ast} \, \bm{\beta})^2 + 2
-\mathbb{E}(\varepsilon_i) \mathbf{X}_{i, \ast} \, \bm{\beta} +
-\mathbb{E}(\varepsilon_i^2 ) - ( \mathbf{X}_{i, \ast} \, \bm{\beta})^2
-\\ & = \mathbb{E}(\varepsilon_i^2 ) \, \, \, = \, \, \,
-\mbox{Var}(\varepsilon_i) \, \, \, = \, \, \, \sigma^2.
-\end{align*}
-Hence, $y_i \sim \mathcal{N}( \mathbf{X}_{i, \ast} \, \bm{\beta}, \sigma^2)$, that is $\bm{y}$ follows a normal distribution with
-mean value $\bm{X}\bm{\beta}$ and variance $\sigma^2$ (not be confused with the singular values of the SVD).
-
-% --- end solution of exercise ---
-
-\subex{b)}
-With the OLS expressions for the parameters $\bm{\beta}$ show that
-\[
-\mathbb{E}(\bm{\beta}) = \bm{\beta}.
-\]
-
-
-% --- begin solution of exercise ---
-\paragraph{Solution.}
-\[
-\mathbb{E}(\bm{\beta}) = \mathbb{E}[ (\mathbf{X}^{\top} \mathbf{X})^{-1}\mathbf{X}^{T} \mathbf{Y}]=(\mathbf{X}^{T} \mathbf{X})^{-1}\mathbf{X}^{T} \mathbb{E}[ \mathbf{Y}]=(\mathbf{X}^{T} \mathbf{X})^{-1} \mathbf{X}^{T}\mathbf{X}\bm{\beta}=\bm{\beta}.
-\]
-This means that the estimator of the regression parameters is unbiased.
-
-% --- end solution of exercise ---
-
-\subex{c)}
-Show finally that the variance of $\bm{\beta}$ is
-\begin{eqnarray*}
-\mbox{Var}(\bm{\beta}) & = & \sigma^2 \, (\mathbf{X}^{T} \mathbf{X})^{-1}.
-\end{eqnarray*}
-
-
-% --- begin solution of exercise ---
-\paragraph{Solution.}
-The variance of $\bm{\beta}$ is
-\begin{eqnarray*}
-\mbox{Var}(\bm{\beta}) & = & \mathbb{E} \{ [\bm{\beta} - \mathbb{E}(\bm{\beta})] [\bm{\beta} - \mathbb{E}(\bm{\beta})]^{T} \}
-\\
-& = & \mathbb{E} \{ [(\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{Y} - \bm{\beta}] \, [(\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{Y} - \bm{\beta}]^{T} \}
-\\
-% & = & \mathbb{E} \{ [(\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{Y}] \, [(\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{Y}]^{T} \} - \bm{\beta} \, \bm{\beta}^{T}
-% \\
-% & = & \mathbb{E} \{ (\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{Y} \, \mathbf{Y}^{T} \, \mathbf{X} \, (\mathbf{X}^{T} \mathbf{X})^{-1} \} - \bm{\beta} \, \bm{\beta}^{T}
-% \\
-& = & (\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \, \mathbb{E} \{ \mathbf{Y} \, \mathbf{Y}^{T} \} \, \mathbf{X} \, (\mathbf{X}^{T} \mathbf{X})^{-1} - \bm{\beta} \, \bm{\beta}^{T}
-\\
-& = & (\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \, \{ \mathbf{X} \, \bm{\beta} \, \bm{\beta}^{T} \, \mathbf{X}^{T} + \sigma^2 \} \, \mathbf{X} \, (\mathbf{X}^{T} \mathbf{X})^{-1} - \bm{\beta} \, \bm{\beta}^{T}
-% \\
-% & = & (\mathbf{X}^T \mathbf{X})^{-1} \, \mathbf{X}^T \, \mathbf{X} \, \bm{\beta} \, \bm{\beta}^T \, \mathbf{X}^T \, \mathbf{X} \, (\mathbf{X}^T % \mathbf{X})^{-1}
-% \\
-% & & + \, \, \sigma^2 \, (\mathbf{X}^T \mathbf{X})^{-1} \, \mathbf{X}^T \, \mathbf{X} \, (\mathbf{X}^T \mathbf{X})^{-1} - \bm{\beta} \bm{\beta}^T
-\\
-& = & \bm{\beta} \, \bm{\beta}^{T} + \sigma^2 \, (\mathbf{X}^{T} \mathbf{X})^{-1} - \bm{\beta} \, \bm{\beta}^{T}
-\, \, \, = \, \, \, \sigma^2 \, (\mathbf{X}^{T} \mathbf{X})^{-1},
-\end{eqnarray*}
-
-where we have used that $\mathbb{E} (\mathbf{Y} \mathbf{Y}^{T}) =
-\mathbf{X} \, \bm{\beta} \, \bm{\beta}^{T} \, \mathbf{X}^{T} +
-\sigma^2 \, \mathbf{I}_{nn}$. From $\mbox{Var}(\bm{\beta}) = \sigma^2
-\, (\mathbf{X}^{T} \mathbf{X})^{-1}$, one obtains an estimate of the
-variance of the estimate of the $j$-th regression coefficient:
-$\bm{\sigma}^2 (\hat{\beta}_j ) = \bm{\sigma}^2 \sqrt{
-[(\mathbf{X}^{T} \mathbf{X})^{-1}]_{jj} }$. This may be used to
-construct a confidence interval for the estimates.
-
-
-In a similar way, we can obtain analytical expressions for say the
-expectation values of the parameters $\bm{\beta}$ and their variance
-when we employ Ridge regression, allowing us again to define a confidence interval.
-
-% --- end solution of exercise ---
-
-
-
-
-\end{doconceexercise}
-% --- end exercise ---
-
-
-% ------------------- end of main content ---------------
-
-% #ifdef PREAMBLE
-\end{document}
-% #endif
-
diff --git a/doc/src/Projects/2020/Exercises/hw1.pdf b/doc/src/Projects/2020/Exercises/hw1.pdf
deleted file mode 100644
index 061218b90..000000000
Binary files a/doc/src/Projects/2020/Exercises/hw1.pdf and /dev/null differ
diff --git a/doc/src/Projects/2020/Exercises/hw1.tex b/doc/src/Projects/2020/Exercises/hw1.tex
deleted file mode 100644
index c8c1def76..000000000
--- a/doc/src/Projects/2020/Exercises/hw1.tex
+++ /dev/null
@@ -1,561 +0,0 @@
-%%
-%% Automatically generated file from DocOnce source
-%% (https://github.com/hplgit/doconce/)
-%%
-%%
-
-
-%-------------------- begin preamble ----------------------
-
-\documentclass[%
-oneside, % oneside: electronic viewing, twoside: printing
-final, % draft: marks overfull hboxes, figures with paths
-10pt]{article}
-
-\listfiles % print all files needed to compile this document
-
-\usepackage{relsize,makeidx,color,setspace,amsmath,amsfonts,amssymb}
-\usepackage[table]{xcolor}
-\usepackage{bm,ltablex,microtype}
-
-\usepackage[pdftex]{graphicx}
-
-\usepackage{fancyvrb} % packages needed for verbatim environments
-
-\usepackage[T1]{fontenc}
-%\usepackage[latin1]{inputenc}
-\usepackage{ucs}
-\usepackage[utf8x]{inputenc}
-
-\usepackage{lmodern} % Latin Modern fonts derived from Computer Modern
-
-% Hyperlinks in PDF:
-\definecolor{linkcolor}{rgb}{0,0,0.4}
-\usepackage{hyperref}
-\hypersetup{
- breaklinks=true,
- colorlinks=true,
- linkcolor=linkcolor,
- urlcolor=linkcolor,
- citecolor=black,
- filecolor=black,
- %filecolor=blue,
- pdfmenubar=true,
- pdftoolbar=true,
- bookmarksdepth=3 % Uncomment (and tweak) for PDF bookmarks with more levels than the TOC
- }
-%\hyperbaseurl{} % hyperlinks are relative to this root
-
-\setcounter{tocdepth}{2} % levels in table of contents
-
-% --- fancyhdr package for fancy headers ---
-\usepackage{fancyhdr}
-\fancyhf{} % sets both header and footer to nothing
-\renewcommand{\headrulewidth}{0pt}
-\fancyfoot[LE,RO]{\thepage}
-% Ensure copyright on titlepage (article style) and chapter pages (book style)
-\fancypagestyle{plain}{
- \fancyhf{}
- \fancyfoot[C]{{\footnotesize \copyright\ 1999-2020, "Data Analysis and Machine Learning FYS-STK3155/FYS4155":"http://www.uio.no/studier/emner/matnat/fys/FYS3155/index-eng.html". Released under CC Attribution-NonCommercial 4.0 license}}
-% \renewcommand{\footrulewidth}{0mm}
- \renewcommand{\headrulewidth}{0mm}
-}
-% Ensure copyright on titlepages with \thispagestyle{empty}
-\fancypagestyle{empty}{
- \fancyhf{}
- \fancyfoot[C]{{\footnotesize \copyright\ 1999-2020, "Data Analysis and Machine Learning FYS-STK3155/FYS4155":"http://www.uio.no/studier/emner/matnat/fys/FYS3155/index-eng.html". Released under CC Attribution-NonCommercial 4.0 license}}
- \renewcommand{\footrulewidth}{0mm}
- \renewcommand{\headrulewidth}{0mm}
-}
-
-\pagestyle{fancy}
-
-
-% prevent orhpans and widows
-\clubpenalty = 10000
-\widowpenalty = 10000
-
-\newenvironment{doconceexercise}{}{}
-\newcounter{doconceexercisecounter}
-
-
-% ------ header in subexercises ------
-%\newcommand{\subex}[1]{\paragraph{#1}}
-%\newcommand{\subex}[1]{\par\vspace{1.7mm}\noindent{\bf #1}\ \ }
-\makeatletter
-% 1.5ex is the spacing above the header, 0.5em the spacing after subex title
-\newcommand\subex{\@startsection*{paragraph}{4}{\z@}%
- {1.5ex\@plus1ex \@minus.2ex}%
- {-0.5em}%
- {\normalfont\normalsize\bfseries}}
-\makeatother
-
-
-% --- end of standard preamble for documents ---
-
-
-% insert custom LaTeX commands...
-
-\raggedbottom
-\makeindex
-\usepackage[totoc]{idxlayout} % for index in the toc
-\usepackage[nottoc]{tocbibind} % for references/bibliography in the toc
-
-%-------------------- end preamble ----------------------
-
-\begin{document}
-
-% matching end for #ifdef PREAMBLE
-
-\newcommand{\exercisesection}[1]{\subsection*{#1}}
-
-
-% ------------------- main content ----------------------
-
-
-
-% ----------------- title -------------------------
-
-\thispagestyle{empty}
-
-\begin{center}
-{\LARGE\bf
-\begin{spacing}{1.25}
-Homework 1 Fall Semester 2020
-\end{spacing}
-}
-\end{center}
-
-% ----------------- author(s) -------------------------
-
-\begin{center}
-{\bf \href{{http://www.uio.no/studier/emner/matnat/fys/FYS3155/index-eng.html}}{Data Analysis and Machine Learning FYS-STK3155/FYS4155}}
-\end{center}
-
- \begin{center}
-% List of all institutions:
-\centerline{{\small Department of Physics, University of Oslo, Norway}}
-\end{center}
-
-% ----------------- end author(s) -------------------------
-
-% --- begin date ---
-\begin{center}
-Aug 19, 2020
-\end{center}
-% --- end date ---
-
-\vspace{1cm}
-
-
-\subsection*{Exercise, Setting up various Python environments}
-
-The first exercise here is of a mere technical art. We want you to have
-\begin{itemize}
-\item git as a version control software and to establish a user account on a provider like GitHub. Other providers like GitLab etc are equally fine. You can also use the University of Oslo \href{{https://www.uio.no/tjenester/it/maskin/filer/versjonskontroll/github.html}}{GitHub facilities}.
-
-\item Install various Python packages
-\end{itemize}
-
-\noindent
-We will make extensive use of Python as programming language and its
-myriad of available libraries. You will find
-IPython/Jupyter notebooks invaluable in your work. You can run \textbf{R}
-codes in the Jupyter/IPython notebooks, with the immediate benefit of
-visualizing your data. You can also use compiled languages like C++,
-Rust, Fortran etc if you prefer. The focus in these lectures will be
-on Python.
-
-If you have Python installed (we recommend Python3) and you feel
-pretty familiar with installing different packages, we recommend that
-you install the following Python packages via \textbf{pip} as
-
-\begin{enumerate}
-\item pip install numpy scipy matplotlib ipython scikit-learn sympy pandas pillow
-\end{enumerate}
-
-\noindent
-For \textbf{Tensorflow}, we recommend following the instructions in the text of
-\href{{http://shop.oreilly.com/product/0636920052289.do}}{Aurelien Geron, Hands‑On Machine Learning with Scikit‑Learn and TensorFlow, O'Reilly}
-
-We will come back to \textbf{tensorflow} later.
-
-For Python3, replace \textbf{pip} with \textbf{pip3}.
-
-For OSX users we recommend, after having installed Xcode, to
-install \textbf{brew}. Brew allows for a seamless installation of additional
-software via for example
-
-\begin{enumerate}
-\item brew install python3
-\end{enumerate}
-
-\noindent
-For Linux users, with its variety of distributions like for example the widely popular Ubuntu distribution,
-you can use \textbf{pip} as well and simply install Python as
-
-\begin{enumerate}
-\item sudo apt-get install python3 (or python for Python2.7)
-\end{enumerate}
-
-\noindent
-If you don't want to perform these operations separately and venture
-into the hassle of exploring how to set up dependencies and paths, we
-recommend two widely used distrubutions which set up all relevant
-dependencies for Python, namely
-
-\begin{itemize}
-\item \href{{https://docs.anaconda.com/}}{Anaconda},
-\end{itemize}
-
-\noindent
-which is an open source
-distribution of the Python and R programming languages for large-scale
-data processing, predictive analytics, and scientific computing, that
-aims to simplify package management and deployment. Package versions
-are managed by the package management system \textbf{conda}.
-
-\begin{itemize}
-\item \href{{https://www.enthought.com/product/canopy/}}{Enthought canopy}
-\end{itemize}
-
-\noindent
-is a Python
-distribution for scientific and analytic computing distribution and
-analysis environment, available for free and under a commercial
-license.
-
-We recommend using \textbf{Anaconda}.
-
-
-
-
-% --- begin exercise ---
-\begin{doconceexercise}
-\refstepcounter{doconceexercisecounter}
-
-\exercisesection*{Exercise \thedoconceexercisecounter: Our first Python encounter}
-
-
-This exercise has as its aim to write a small program which reads in data from a \textbf{csv} file on the equation of state for dense nuclear matter. The file is localized at \href{{https://github.com/mhjensen/MachineLearningMSU-FRIB2020/blob/master/doc/pub/Regression/ipynb/datafiles/EoS.csv}}{\nolinkurl{https://github.com/mhjensen/MachineLearningMSU-FRIB2020/blob/master/doc/pub/Regression/ipynb/datafiles/EoS.csv}}. Thereafter you will have to set up the design matrix $\bm{X}$ for the $n$
-datapoints and a polynomial of degree $3$. The steps are:
-\begin{itemize}
-\item Write a Python code which reads the in the above mentioned file.
-
-\item Use for example \textbf{pandas} to order your data and find out how many data points there are.
-
-\item Set thereafter up the design matrix with dimensionality $n\times p$ where $p=4$ and where you have defined a polynomial of degree $p-1=3$. Print the matrix and check that the numbers are correct.
-\end{itemize}
-
-\noindent
-We recommend looking at the examples in the \href{{https://compphysics.github.io/MachineLearning/doc/pub/Regression/html/Regression-bs.html}}{regression slides}.
-
-
-% --- begin solution of exercise ---
-\paragraph{Solution.}
-\begin{print}
-import os
-import numpy as np
-import pandas as pd
-import matplotlib.pyplot as plt
-from sklearn.model_selection import train_test_split
-# Where to save the figures and data files
-PROJECT_ROOT_DIR = "Results"
-FIGURE_ID = "Results/FigureFiles"
-DATA_ID = "DataFiles/"
-
-if not os.path.exists(PROJECT_ROOT_DIR):
- os.mkdir(PROJECT_ROOT_DIR)
-
-if not os.path.exists(FIGURE_ID):
- os.makedirs(FIGURE_ID)
-
-if not os.path.exists(DATA_ID):
- os.makedirs(DATA_ID)
-
-def image_path(fig_id):
- return os.path.join(FIGURE_ID, fig_id)
-
-def data_path(dat_id):
- return os.path.join(DATA_ID, dat_id)
-
-def save_fig(fig_id):
- plt.savefig(image_path(fig_id) + ".png", format='png')
-
-def R2(y_data, y_model):
- return 1 - np.sum((y_data - y_model) ** 2) / np.sum((y_data - np.mean(y_data)) ** 2)
-def MSE(y_data,y_model):
- n = np.size(y_model)
- return np.sum((y_data-y_model)**2)/n
-
-infile = open(data_path("EoS.csv"),'r')
-
-# Read the EoS data as csv file and organized into two arrays with density and energies
-EoS = pd.read_csv(infile, names=('Density', 'Energy'))
-EoS['Energy'] = pd.to_numeric(EoS['Energy'], errors='coerce')
-EoS = EoS.dropna()
-Energies = EoS['Energy']
-Density = EoS['Density']
-# The design matrix now as function of various polytrops
-X = np.zeros((len(Density),5))
-X[:,0] = 1
-X[:,1] = Density**(2.0/3.0)
-X[:,2] = Density
-X[:,3] = Density**(4.0/3.0)
-X[:,4] = Density**(5.0/3.0)
-# We split the data in test and training data
-X_train, X_test, y_train, y_test = train_test_split(X, Energies, test_size=0.2)
-# matrix inversion to find beta
-beta = np.linalg.inv(X_train.T @ X_train) @ X_train.T @ y_train
-# and then make the prediction
-ytilde = X_train @ beta
-print("Training R2")
-print(R2(y_train,ytilde))
-print("Training MSE")
-print(MSE(y_train,ytilde))
-ypredict = X_test @ beta
-print("Test R2")
-print(R2(y_test,ypredict))
-print("Test MSE")
-print(MSE(y_test,ypredict))
-\end{print}
-
-% --- end solution of exercise ---
-
-\end{doconceexercise}
-% --- end exercise ---
-
-
-
-
-% --- begin exercise ---
-\begin{doconceexercise}
-\refstepcounter{doconceexercisecounter}
-
-\exercisesection*{Exercise \thedoconceexercisecounter: making your own data and exploring scikit-learn}
-
-
-We will generate our own dataset for a function $y(x)$ where $x \in [0,1]$ and defined by random numbers computed with the uniform distribution. The function $y$ is a quadratic polynomial in $x$ with added stochastic noise according to the normal distribution $\cal {N}(0,1)$.
-The following simple Python instructions define our $x$ and $y$ values (with 100 data points).
-\begin{print}
-x = np.random.rand(100,1)
-y = 2.0+5*x*x+0.1*np.random.randn(100,1)
-\end{print}
-
-\begin{enumerate}
-\item Write your own code (following the examples under the \href{{https://compphysics.github.io/MachineLearningECT/doc/pub/Day1/html/Day1-bs.html}}{regression slides}) for computing the parametrization of the data set fitting a second-order polynomial.
-
-\item Use thereafter \textbf{scikit-learn} (see again the examples in the regression slides) and compare with your own code.
-
-\item Using scikit-learn, compute also the mean square error, a risk metric corresponding to the expected value of the squared (quadratic) error defined as
-\end{enumerate}
-
-\noindent
-\[ MSE(\hat{y},\hat{\tilde{y}}) = \frac{1}{n}
-\sum_{i=0}^{n-1}(y_i-\tilde{y}_i)^2,
-\]
-and the $R^2$ score function.
-If $\tilde{\hat{y}}_i$ is the predicted value of the $i-th$ sample and $y_i$ is the corresponding true value, then the score $R^2$ is defined as
-\[
-R^2(\hat{y}, \tilde{\hat{y}}) = 1 - \frac{\sum_{i=0}^{n - 1} (y_i - \tilde{y}_i)^2}{\sum_{i=0}^{n - 1} (y_i - \bar{y})^2},
-\]
-where we have defined the mean value of $\hat{y}$ as
-\[
-\bar{y} = \frac{1}{n} \sum_{i=0}^{n - 1} y_i.
-\]
-You can use the functionality included in scikit-learn. If you feel for it, you can use your own program and define functions which compute the above two functions.
-Discuss the meaning of these results. Try also to vary the coefficient in front of the added stochastic noise term and discuss the quality of the fits.
-
-
-% --- begin solution of exercise ---
-\paragraph{Solution.}
-The code here is an example of where we define our own design matrix and fit parameters $\beta$.
-\begin{print}
-import os
-import numpy as np
-import pandas as pd
-import matplotlib.pyplot as plt
-from sklearn.model_selection import train_test_split
-
-def save_fig(fig_id):
- plt.savefig(image_path(fig_id) + ".png", format='png')
-
-def R2(y_data, y_model):
- return 1 - np.sum((y_data - y_model) ** 2) / np.sum((y_data - np.mean(y_data)) ** 2)
-def MSE(y_data,y_model):
- n = np.size(y_model)
- return np.sum((y_data-y_model)**2)/n
-
-x = np.random.rand(100)
-y = 2.0+5*x*x+0.1*np.random.randn(100)
-
-
-# The design matrix now as function of a given polynomial
-X = np.zeros((len(x),3))
-X[:,0] = 1.0
-X[:,1] = x
-X[:,2] = x**2
-# We split the data in test and training data
-X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)
-# matrix inversion to find beta
-beta = np.linalg.inv(X_train.T @ X_train) @ X_train.T @ y_train
-print(beta)
-# and then make the prediction
-ytilde = X_train @ beta
-print("Training R2")
-print(R2(y_train,ytilde))
-print("Training MSE")
-print(MSE(y_train,ytilde))
-ypredict = X_test @ beta
-print("Test R2")
-print(R2(y_test,ypredict))
-print("Test MSE")
-print(MSE(y_test,ypredict))
-\end{print}
-
-% --- end solution of exercise ---
-
-\end{doconceexercise}
-% --- end exercise ---
-
-
-
-
-% --- begin exercise ---
-\begin{doconceexercise}
-\refstepcounter{doconceexercisecounter}
-
-\exercisesection*{Exercise \thedoconceexercisecounter: mean values and variances in linear regression}
-
-
-This exercise deals with various mean values ad variances in linear regression method (here it may be useful to look up chapter 3, equation (3.8) of \href{{https://www.springer.com/gp/book/9780387848570}}{Trevor Hastie, Robert Tibshirani, Jerome H. Friedman, The Elements of Statistical Learning, Springer}).
-
-The assumption we have made is
-that there exists a function $f(\bm{x})$ and a normal distributed error $\bm{\varepsilon}\sim \mathcal{N}(0, \sigma^2)$
-which describes our data
-\[
-\bm{y} = f(\bm{x})+\bm{\varepsilon}
-\]
-
-We then approximate this function with our model from the solution of the linear regression equations (ordinary least squares OLS), that is our
-function $f$ is approximated by $\bm{\tilde{y}}$ where we minimized $(\bm{y}-\bm{\tilde{y}})^2$, with
-\[
-\bm{\tilde{y}} = \bm{X}\bm{\beta}.
-\]
-The matrix $\bm{X}$ is the so-called design matrix.
-
-
-\subex{a)}
-Show that the expectation value of $\bm{y}$ for a given element $i$
-\begin{align*}
-\mathbb{E}(y_i) & =\mathbf{X}_{i, \ast} \, \beta,
-\end{align*}
-and that
-its variance is
-\begin{align*} \mbox{Var}(y_i) & = \sigma^2.
-\end{align*}
-Hence, $y_i \sim \mathcal{N}( \mathbf{X}_{i, \ast} \, \bm{\beta}, \sigma^2)$, that is $\bm{y}$ follows a normal distribution with
-mean value $\bm{X}\bm{\beta}$ and variance $\sigma^2$.
-
-
-% --- begin solution of exercise ---
-\paragraph{Solution.}
-We can calculate the expectation value of $\bm{y}$ for a given element $i$
-\begin{align*}
-\mathbb{E}(y_i) & =
-\mathbb{E}(\mathbf{X}_{i, \ast} \, \bm{\beta}) + \mathbb{E}(\varepsilon_i)
-\, \, \, = \, \, \, \mathbf{X}_{i, \ast} \, \beta,
-\end{align*}
-while
-its variance is
-\begin{align*} \mbox{Var}(y_i) & = \mathbb{E} \{ [y_i
-- \mathbb{E}(y_i)]^2 \} \, \, \, = \, \, \, \mathbb{E} ( y_i^2 ) -
-[\mathbb{E}(y_i)]^2 \\ & = \mathbb{E} [ ( \mathbf{X}_{i, \ast} \,
-\beta + \varepsilon_i )^2] - ( \mathbf{X}_{i, \ast} \, \bm{\beta})^2 \\ &
-= \mathbb{E} [ ( \mathbf{X}_{i, \ast} \, \bm{\beta})^2 + 2 \varepsilon_i
-\mathbf{X}_{i, \ast} \, \bm{\beta} + \varepsilon_i^2 ] - ( \mathbf{X}_{i,
-\ast} \, \beta)^2 \\ & = ( \mathbf{X}_{i, \ast} \, \bm{\beta})^2 + 2
-\mathbb{E}(\varepsilon_i) \mathbf{X}_{i, \ast} \, \bm{\beta} +
-\mathbb{E}(\varepsilon_i^2 ) - ( \mathbf{X}_{i, \ast} \, \bm{\beta})^2
-\\ & = \mathbb{E}(\varepsilon_i^2 ) \, \, \, = \, \, \,
-\mbox{Var}(\varepsilon_i) \, \, \, = \, \, \, \sigma^2.
-\end{align*}
-Hence, $y_i \sim \mathcal{N}( \mathbf{X}_{i, \ast} \, \bm{\beta}, \sigma^2)$, that is $\bm{y}$ follows a normal distribution with
-mean value $\bm{X}\bm{\beta}$ and variance $\sigma^2$ (not be confused with the singular values of the SVD).
-
-% --- end solution of exercise ---
-
-\subex{b)}
-With the OLS expressions for the parameters $\bm{\beta}$ show that
-\[
-\mathbb{E}(\bm{\beta}) = \bm{\beta}.
-\]
-
-
-% --- begin solution of exercise ---
-\paragraph{Solution.}
-\[
-\mathbb{E}(\bm{\beta}) = \mathbb{E}[ (\mathbf{X}^{\top} \mathbf{X})^{-1}\mathbf{X}^{T} \mathbf{Y}]=(\mathbf{X}^{T} \mathbf{X})^{-1}\mathbf{X}^{T} \mathbb{E}[ \mathbf{Y}]=(\mathbf{X}^{T} \mathbf{X})^{-1} \mathbf{X}^{T}\mathbf{X}\bm{\beta}=\bm{\beta}.
-\]
-This means that the estimator of the regression parameters is unbiased.
-
-% --- end solution of exercise ---
-
-\subex{c)}
-Show finally that the variance of $\bm{\beta}$ is
-\begin{eqnarray*}
-\mbox{Var}(\bm{\beta}) & = & \sigma^2 \, (\mathbf{X}^{T} \mathbf{X})^{-1}.
-\end{eqnarray*}
-
-
-% --- begin solution of exercise ---
-\paragraph{Solution.}
-The variance of $\bm{\beta}$ is
-\begin{eqnarray*}
-\mbox{Var}(\bm{\beta}) & = & \mathbb{E} \{ [\bm{\beta} - \mathbb{E}(\bm{\beta})] [\bm{\beta} - \mathbb{E}(\bm{\beta})]^{T} \}
-\\
-& = & \mathbb{E} \{ [(\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{Y} - \bm{\beta}] \, [(\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{Y} - \bm{\beta}]^{T} \}
-\\
-% & = & \mathbb{E} \{ [(\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{Y}] \, [(\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{Y}]^{T} \} - \bm{\beta} \, \bm{\beta}^{T}
-% \\
-% & = & \mathbb{E} \{ (\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{Y} \, \mathbf{Y}^{T} \, \mathbf{X} \, (\mathbf{X}^{T} \mathbf{X})^{-1} \} - \bm{\beta} \, \bm{\beta}^{T}
-% \\
-& = & (\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \, \mathbb{E} \{ \mathbf{Y} \, \mathbf{Y}^{T} \} \, \mathbf{X} \, (\mathbf{X}^{T} \mathbf{X})^{-1} - \bm{\beta} \, \bm{\beta}^{T}
-\\
-& = & (\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \, \{ \mathbf{X} \, \bm{\beta} \, \bm{\beta}^{T} \, \mathbf{X}^{T} + \sigma^2 \} \, \mathbf{X} \, (\mathbf{X}^{T} \mathbf{X})^{-1} - \bm{\beta} \, \bm{\beta}^{T}
-% \\
-% & = & (\mathbf{X}^T \mathbf{X})^{-1} \, \mathbf{X}^T \, \mathbf{X} \, \bm{\beta} \, \bm{\beta}^T \, \mathbf{X}^T \, \mathbf{X} \, (\mathbf{X}^T % \mathbf{X})^{-1}
-% \\
-% & & + \, \, \sigma^2 \, (\mathbf{X}^T \mathbf{X})^{-1} \, \mathbf{X}^T \, \mathbf{X} \, (\mathbf{X}^T \mathbf{X})^{-1} - \bm{\beta} \bm{\beta}^T
-\\
-& = & \bm{\beta} \, \bm{\beta}^{T} + \sigma^2 \, (\mathbf{X}^{T} \mathbf{X})^{-1} - \bm{\beta} \, \bm{\beta}^{T}
-\, \, \, = \, \, \, \sigma^2 \, (\mathbf{X}^{T} \mathbf{X})^{-1},
-\end{eqnarray*}
-
-where we have used that $\mathbb{E} (\mathbf{Y} \mathbf{Y}^{T}) =
-\mathbf{X} \, \bm{\beta} \, \bm{\beta}^{T} \, \mathbf{X}^{T} +
-\sigma^2 \, \mathbf{I}_{nn}$. From $\mbox{Var}(\bm{\beta}) = \sigma^2
-\, (\mathbf{X}^{T} \mathbf{X})^{-1}$, one obtains an estimate of the
-variance of the estimate of the $j$-th regression coefficient:
-$\bm{\sigma}^2 (\hat{\beta}_j ) = \bm{\sigma}^2 \sqrt{
-[(\mathbf{X}^{T} \mathbf{X})^{-1}]_{jj} }$. This may be used to
-construct a confidence interval for the estimates.
-
-
-In a similar way, we can obtain analytical expressions for say the
-expectation values of the parameters $\bm{\beta}$ and their variance
-when we employ Ridge regression, allowing us again to define a confidence interval.
-
-% --- end solution of exercise ---
-
-
-
-
-\end{doconceexercise}
-% --- end exercise ---
-
-
-% ------------------- end of main content ---------------
-
-\end{document}
-
diff --git a/doc/src/Projects/2020/Exercises/ipynb-hw1-src.tar.gz b/doc/src/Projects/2020/Exercises/ipynb-hw1-src.tar.gz
deleted file mode 100644
index 398dcd682..000000000
Binary files a/doc/src/Projects/2020/Exercises/ipynb-hw1-src.tar.gz and /dev/null differ