+This exercise is a continuation of exercise 2 from homework 1. We will
+use the same function to generate our data set, still staying with a
+simple function \( y(x) \) which we want to fit using linear regression,
+but now extending the analysis to include the Ridge and the Lasso
+regression methods. You can use the code under the Regression as an example on how to use the Ridge and the Lasso methods, see the regression slides).
+
+
+We will thus again generate our own dataset for a function \( y(x) \) where
+\( x \in [0,1] \) and defined by random numbers computed with the uniform
+distribution. The function \( y \) is a quadratic polynomial in \( x \) with
+added stochastic noise according to the normal distribution \( \cal{N}(0,1) \).
+
+
+The following simple Python instructions define our \( x \) and \( y \) values (with 100 data points).
+
+
+
+
x = np.random.rand(100,1)
+y =5*x*x+0.1*np.random.randn(100,1)
+
+
+
Write your own code for the Ridge method (see chapter 3.4 of Hastie et al., equations (3.43) and (3.44)) and compute the parametrization for different values of \( \lambda \).
+
+
+Compare and analyze your results with those from exercise 2. Study the dependence on \( \lambda \) while also varying the strength of the noise in your expression for \( y(x) \).
+
+
+
Repeat the above but using the functionality of scikit-learn. Compare your code with the results from scikit-learn. Remember to run with the same random numbers for generating \( x \) and \( y \).
+
Our next step is to study the variance of the parameters \( \beta_1 \) and \( \beta_2 \) (assuming that we are parametrizing our function with a second-order polynomial. We will use standard linear regression and the Ridge regression.
+
+
+You can now opt for either writing your own function that calculates the variance of these paramaters (recall that this is equal to the diagonal elements of the matrix \( (\hat{X}^T\hat{X})^{-1}+\lambda\hat{I} \)) or use the functionality of scikit-learn and computetheir variances. Discuss the results of these variances as functions of \( \lambda \). In particular, try to link your discussion with the discussion in Hastie et al. and their figure 3.11.
+
+
+
Repeat the previous step but add now the Lasso method, see equation (3.53) of Hastie et al.. Discuss your results and compare with standard regression and the Ridge regression results. You can write your own code or use the functionality of scikit-learn.
+
Finally, using scikit-learn or your own code, compute also the mean square error, a risk metric corresponding to the expected value of the squared (quadratic) error defined as
+
+
+$$ MSE(\hat{y},\hat{\tilde{y}}) = \frac{1}{n}
+\sum_{i=0}^{n-1}(y_i-\tilde{y}_i)^2,
+$$
+
+and the \( R^2 \) score function.
+If \( \tilde{\hat{y}}_i \) is the predicted value of the \( i-th \) sample and \( y_i \) is the corresponding true value, then the score \( R^2 \) is defined as
+$$
+R^2(\hat{y}, \tilde{\hat{y}}) = 1 - \frac{\sum_{i=0}^{n - 1} (y_i - \tilde{y}_i)^2}{\sum_{i=0}^{n - 1} (y_i - \bar{y})^2},
+$$
+
+where we have defined the mean value of \( \hat{y} \) as
+$$
+\bar{y} = \frac{1}{n} \sum_{i=0}^{n - 1} y_i.
+$$
+
+Discuss these quantities as functions of the variable \( \lambda \) in the Ridge and Lasso regression methods.
+
+
Exercise 5
+
+
+Using the singular value decomposition, show that the variance of the direction vector
+\( \hat{z}_i=\hat{X}\hat{v}_i \) is equal to (equation (3.49) of Hastie et al.)
+$$
+\mathrm{Var}(\hat{z}_i)=\frac{d_i^2}{N},
+$$
+
+where \( d_i \) are the singular values of the matrix \( \hat{X} \). Give an interprepation of these results, in particular in connection with the variance of the coefficients you obtained in the previous exercise.
+
+
+This exercise is a continuation of exercise 2 from homework 1. We will
+use the same function to generate our data set, still staying with a
+simple function \( y(x) \) which we want to fit using linear regression,
+but now extending the analysis to include the Ridge and the Lasso
+regression methods. You can use the code under the Regression as an example on how to use the Ridge and the Lasso methods, see the regression slides).
+
+
+We will thus again generate our own dataset for a function \( y(x) \) where
+\( x \in [0,1] \) and defined by random numbers computed with the uniform
+distribution. The function \( y \) is a quadratic polynomial in \( x \) with
+added stochastic noise according to the normal distribution \( \cal{N}(0,1) \).
+
+
+The following simple Python instructions define our \( x \) and \( y \) values (with 100 data points).
+
+
+
+
x = np.random.rand(100,1)
+y =5*x*x+0.1*np.random.randn(100,1)
+
+
+
Write your own code for the Ridge method (see chapter 3.4 of Hastie et al., equations (3.43) and (3.44)) and compute the parametrization for different values of \( \lambda \).
+
+
+Compare and analyze your results with those from exercise 2. Study the dependence on \( \lambda \) while also varying the strength of the noise in your expression for \( y(x) \).
+
+
+
Repeat the above but using the functionality of scikit-learn. Compare your code with the results from scikit-learn. Remember to run with the same random numbers for generating \( x \) and \( y \).
+
Our next step is to study the variance of the parameters \( \beta_1 \) and \( \beta_2 \) (assuming that we are parametrizing our function with a second-order polynomial. We will use standard linear regression and the Ridge regression.
+
+
+You can now opt for either writing your own function that calculates the variance of these paramaters (recall that this is equal to the diagonal elements of the matrix \( (\hat{X}^T\hat{X})^{-1}+\lambda\hat{I} \)) or use the functionality of scikit-learn and computetheir variances. Discuss the results of these variances as functions of \( \lambda \). In particular, try to link your discussion with the discussion in Hastie et al. and their figure 3.11.
+
+
+
Repeat the previous step but add now the Lasso method, see equation (3.53) of Hastie et al.. Discuss your results and compare with standard regression and the Ridge regression results. You can write your own code or use the functionality of scikit-learn.
+
Finally, using scikit-learn or your own code, compute also the mean square error, a risk metric corresponding to the expected value of the squared (quadratic) error defined as
+
+
+$$ MSE(\hat{y},\hat{\tilde{y}}) = \frac{1}{n}
+\sum_{i=0}^{n-1}(y_i-\tilde{y}_i)^2,
+$$
+
+and the \( R^2 \) score function.
+If \( \tilde{\hat{y}}_i \) is the predicted value of the \( i-th \) sample and \( y_i \) is the corresponding true value, then the score \( R^2 \) is defined as
+$$
+R^2(\hat{y}, \tilde{\hat{y}}) = 1 - \frac{\sum_{i=0}^{n - 1} (y_i - \tilde{y}_i)^2}{\sum_{i=0}^{n - 1} (y_i - \bar{y})^2},
+$$
+
+where we have defined the mean value of \( \hat{y} \) as
+$$
+\bar{y} = \frac{1}{n} \sum_{i=0}^{n - 1} y_i.
+$$
+
+Discuss these quantities as functions of the variable \( \lambda \) in the Ridge and Lasso regression methods.
+
+
Exercise 5
+
+
+Using the singular value decomposition, show that the variance of the direction vector
+\( \hat{z}_i=\hat{X}\hat{v}_i \) is equal to (equation (3.49) of Hastie et al.)
+$$
+\mathrm{Var}(\hat{z}_i)=\frac{d_i^2}{N},
+$$
+
+where \( d_i \) are the singular values of the matrix \( \hat{X} \). Give an interprepation of these results, in particular in connection with the variance of the coefficients you obtained in the previous exercise.
+
+
+
+
+
+
+
+
+
+
+
diff --git a/doc/Projects/2018/hw2/ipynb/ipynb-hw2-src.tar.gz b/doc/Projects/2018/hw2/ipynb/ipynb-hw2-src.tar.gz
new file mode 100644
index 000000000..4b1da116e
Binary files /dev/null and b/doc/Projects/2018/hw2/ipynb/ipynb-hw2-src.tar.gz differ
diff --git a/doc/Projects/2018/hw2/pdf/hw2.p.tex b/doc/Projects/2018/hw2/pdf/hw2.p.tex
new file mode 100644
index 000000000..7824027f3
--- /dev/null
+++ b/doc/Projects/2018/hw2/pdf/hw2.p.tex
@@ -0,0 +1,235 @@
+%%
+%% Automatically generated file from DocOnce source
+%% (https://github.com/hplgit/doconce/)
+%%
+%%
+% #ifdef PTEX2TEX_EXPLANATION
+%%
+%% The file follows the ptex2tex extended LaTeX format, see
+%% ptex2tex: http://code.google.com/p/ptex2tex/
+%%
+%% Run
+%% ptex2tex myfile
+%% or
+%% doconce ptex2tex myfile
+%%
+%% to turn myfile.p.tex into an ordinary LaTeX file myfile.tex.
+%% (The ptex2tex program: http://code.google.com/p/ptex2tex)
+%% Many preprocess options can be added to ptex2tex or doconce ptex2tex
+%%
+%% ptex2tex -DMINTED myfile
+%% doconce ptex2tex myfile envir=minted
+%%
+%% ptex2tex will typeset code environments according to a global or local
+%% .ptex2tex.cfg configure file. doconce ptex2tex will typeset code
+%% according to options on the command line (just type doconce ptex2tex to
+%% see examples). If doconce ptex2tex has envir=minted, it enables the
+%% minted style without needing -DMINTED.
+% #endif
+
+% #define PREAMBLE
+
+% #ifdef PREAMBLE
+%-------------------- begin preamble ----------------------
+
+\documentclass[%
+oneside, % oneside: electronic viewing, twoside: printing
+final, % draft: marks overfull hboxes, figures with paths
+10pt]{article}
+
+\listfiles % print all files needed to compile this document
+
+\usepackage{relsize,makeidx,color,setspace,amsmath,amsfonts,amssymb}
+\usepackage[table]{xcolor}
+\usepackage{bm,ltablex,microtype}
+
+\usepackage[pdftex]{graphicx}
+
+\usepackage{ptex2tex}
+% #ifdef MINTED
+\usepackage{minted}
+\usemintedstyle{default}
+% #endif
+
+\usepackage[T1]{fontenc}
+%\usepackage[latin1]{inputenc}
+\usepackage{ucs}
+\usepackage[utf8x]{inputenc}
+
+\usepackage{lmodern} % Latin Modern fonts derived from Computer Modern
+
+% Hyperlinks in PDF:
+\definecolor{linkcolor}{rgb}{0,0,0.4}
+\usepackage{hyperref}
+\hypersetup{
+ breaklinks=true,
+ colorlinks=true,
+ linkcolor=linkcolor,
+ urlcolor=linkcolor,
+ citecolor=black,
+ filecolor=black,
+ %filecolor=blue,
+ pdfmenubar=true,
+ pdftoolbar=true,
+ bookmarksdepth=3 % Uncomment (and tweak) for PDF bookmarks with more levels than the TOC
+ }
+%\hyperbaseurl{} % hyperlinks are relative to this root
+
+\setcounter{tocdepth}{2} % levels in table of contents
+
+% --- fancyhdr package for fancy headers ---
+\usepackage{fancyhdr}
+\fancyhf{} % sets both header and footer to nothing
+\renewcommand{\headrulewidth}{0pt}
+\fancyfoot[LE,RO]{\thepage}
+% Ensure copyright on titlepage (article style) and chapter pages (book style)
+\fancypagestyle{plain}{
+ \fancyhf{}
+ \fancyfoot[C]{{\footnotesize \copyright\ 1999-2018, "Data Analysis and Machine Learning FYS-STK3155/FYS4155":"http://www.uio.no/studier/emner/matnat/fys/FYS3155/index-eng.html". Released under CC Attribution-NonCommercial 4.0 license}}
+% \renewcommand{\footrulewidth}{0mm}
+ \renewcommand{\headrulewidth}{0mm}
+}
+% Ensure copyright on titlepages with \thispagestyle{empty}
+\fancypagestyle{empty}{
+ \fancyhf{}
+ \fancyfoot[C]{{\footnotesize \copyright\ 1999-2018, "Data Analysis and Machine Learning FYS-STK3155/FYS4155":"http://www.uio.no/studier/emner/matnat/fys/FYS3155/index-eng.html". Released under CC Attribution-NonCommercial 4.0 license}}
+ \renewcommand{\footrulewidth}{0mm}
+ \renewcommand{\headrulewidth}{0mm}
+}
+
+\pagestyle{fancy}
+
+
+% prevent orhpans and widows
+\clubpenalty = 10000
+\widowpenalty = 10000
+
+% --- end of standard preamble for documents ---
+
+
+% insert custom LaTeX commands...
+
+\raggedbottom
+\makeindex
+\usepackage[totoc]{idxlayout} % for index in the toc
+\usepackage[nottoc]{tocbibind} % for references/bibliography in the toc
+
+%-------------------- end preamble ----------------------
+
+\begin{document}
+
+% matching end for #ifdef PREAMBLE
+% #endif
+
+\newcommand{\exercisesection}[1]{\subsection*{#1}}
+
+
+% ------------------- main content ----------------------
+
+
+
+% ----------------- title -------------------------
+
+\thispagestyle{empty}
+
+\begin{center}
+{\LARGE\bf
+\begin{spacing}{1.25}
+Homework 2
+\end{spacing}
+}
+\end{center}
+
+% ----------------- author(s) -------------------------
+
+\begin{center}
+{\bf \href{{http://www.uio.no/studier/emner/matnat/fys/FYS3155/index-eng.html}}{Data Analysis and Machine Learning FYS-STK3155/FYS4155}}
+\end{center}
+
+ \begin{center}
+% List of all institutions:
+\centerline{{\small Department of Physics, University of Oslo, Norway}}
+\end{center}
+
+% ----------------- end author(s) -------------------------
+
+% --- begin date ---
+\begin{center}
+Sep 3, 2018
+\end{center}
+% --- end date ---
+
+\vspace{1cm}
+
+
+\subsection{Exercise 4}
+
+This exercise is a continuation of exercise 2 from homework 1. We will
+use the same function to generate our data set, still staying with a
+simple function $y(x)$ which we want to fit using linear regression,
+but now extending the analysis to include the Ridge and the Lasso
+regression methods. You can use the code under the Regression as an example on how to use the Ridge and the Lasso methods, see the \href{{https://compphysics.github.io/MachineLearning/doc/pub/Regression/html/Regression-bs.html}}{regression slides}).
+
+We will thus again generate our own dataset for a function $y(x)$ where
+$x \in [0,1]$ and defined by random numbers computed with the uniform
+distribution. The function $y$ is a quadratic polynomial in $x$ with
+added stochastic noise according to the normal distribution $\cal{N}(0,1)$.
+
+The following simple Python instructions define our $x$ and $y$ values (with 100 data points).
+\bpycod
+x = np.random.rand(100,1)
+y = 5*x*x+0.1*np.random.randn(100,1)
+\epycod
+
+\begin{enumerate}
+\item Write your own code for the Ridge method (see chapter 3.4 of Hastie \emph{et al.}, equations (3.43) and (3.44)) and compute the parametrization for different values of $\lambda$.
+\end{enumerate}
+
+\noindent
+Compare and analyze your results with those from exercise 2. Study the dependence on $\lambda$ while also varying the strength of the noise in your expression for $y(x)$.
+
+\begin{enumerate}
+\item Repeat the above but using the functionality of \textbf{scikit-learn}. Compare your code with the results from \textbf{scikit-learn}. Remember to run with the same random numbers for generating $x$ and $y$.
+
+\item Our next step is to study the variance of the parameters $\beta_1$ and $\beta_2$ (assuming that we are parametrizing our function with a second-order polynomial. We will use standard linear regression and the Ridge regression.
+\end{enumerate}
+
+\noindent
+You can now opt for either writing your own function that calculates the variance of these paramaters (recall that this is equal to the diagonal elements of the matrix $(\hat{X}^T\hat{X})^{-1}+\lambda\hat{I}$) or use the functionality of \textbf{scikit-learn} and computetheir variances. Discuss the results of these variances as functions of $\lambda$. In particular, try to link your discussion with the discussion in Hastie \emph{et al.} and their figure 3.11.
+
+\begin{enumerate}
+\item Repeat the previous step but add now the Lasso method, see equation (3.53) of Hastie \emph{et al.}. Discuss your results and compare with standard regression and the Ridge regression results. You can write your own code or use the functionality of \textbf{scikit-learn}.
+
+\item Finally, using \textbf{scikit-learn} or your own code, compute also the mean square error, a risk metric corresponding to the expected value of the squared (quadratic) error defined as
+\end{enumerate}
+
+\noindent
+\[ MSE(\hat{y},\hat{\tilde{y}}) = \frac{1}{n}
+\sum_{i=0}^{n-1}(y_i-\tilde{y}_i)^2,
+\]
+and the $R^2$ score function.
+If $\tilde{\hat{y}}_i$ is the predicted value of the $i-th$ sample and $y_i$ is the corresponding true value, then the score $R^2$ is defined as
+\[
+R^2(\hat{y}, \tilde{\hat{y}}) = 1 - \frac{\sum_{i=0}^{n - 1} (y_i - \tilde{y}_i)^2}{\sum_{i=0}^{n - 1} (y_i - \bar{y})^2},
+\]
+where we have defined the mean value of $\hat{y}$ as
+\[
+\bar{y} = \frac{1}{n} \sum_{i=0}^{n - 1} y_i.
+\]
+Discuss these quantities as functions of the variable $\lambda$ in the Ridge and Lasso regression methods.
+
+\subsection{Exercise 5}
+
+Using the singular value decomposition, show that the variance of the direction vector
+$\hat{z}_i=\hat{X}\hat{v}_i$ is equal to (equation (3.49) of Hastie \emph{et al.})
+\[
+\mathrm{Var}(\hat{z}_i)=\frac{d_i^2}{N},
+\]
+where $d_i$ are the singular values of the matrix $\hat{X}$. Give an interprepation of these results, in particular in connection with the variance of the coefficients you obtained in the previous exercise.
+
+% ------------------- end of main content ---------------
+
+% #ifdef PREAMBLE
+\end{document}
+% #endif
+
diff --git a/doc/Projects/2018/hw2/pdf/hw2.pdf b/doc/Projects/2018/hw2/pdf/hw2.pdf
new file mode 100644
index 000000000..d5d7c1f38
Binary files /dev/null and b/doc/Projects/2018/hw2/pdf/hw2.pdf differ
diff --git a/doc/Projects/2018/hw2/pdf/hw2.tex b/doc/Projects/2018/hw2/pdf/hw2.tex
new file mode 100644
index 000000000..84344cbca
--- /dev/null
+++ b/doc/Projects/2018/hw2/pdf/hw2.tex
@@ -0,0 +1,203 @@
+%%
+%% Automatically generated file from DocOnce source
+%% (https://github.com/hplgit/doconce/)
+%%
+%%
+
+
+%-------------------- begin preamble ----------------------
+
+\documentclass[%
+oneside, % oneside: electronic viewing, twoside: printing
+final, % draft: marks overfull hboxes, figures with paths
+10pt]{article}
+
+\listfiles % print all files needed to compile this document
+
+\usepackage{relsize,makeidx,color,setspace,amsmath,amsfonts,amssymb}
+\usepackage[table]{xcolor}
+\usepackage{bm,ltablex,microtype}
+
+\usepackage[pdftex]{graphicx}
+
+\usepackage{fancyvrb} % packages needed for verbatim environments
+
+\usepackage[T1]{fontenc}
+%\usepackage[latin1]{inputenc}
+\usepackage{ucs}
+\usepackage[utf8x]{inputenc}
+
+\usepackage{lmodern} % Latin Modern fonts derived from Computer Modern
+
+% Hyperlinks in PDF:
+\definecolor{linkcolor}{rgb}{0,0,0.4}
+\usepackage{hyperref}
+\hypersetup{
+ breaklinks=true,
+ colorlinks=true,
+ linkcolor=linkcolor,
+ urlcolor=linkcolor,
+ citecolor=black,
+ filecolor=black,
+ %filecolor=blue,
+ pdfmenubar=true,
+ pdftoolbar=true,
+ bookmarksdepth=3 % Uncomment (and tweak) for PDF bookmarks with more levels than the TOC
+ }
+%\hyperbaseurl{} % hyperlinks are relative to this root
+
+\setcounter{tocdepth}{2} % levels in table of contents
+
+% --- fancyhdr package for fancy headers ---
+\usepackage{fancyhdr}
+\fancyhf{} % sets both header and footer to nothing
+\renewcommand{\headrulewidth}{0pt}
+\fancyfoot[LE,RO]{\thepage}
+% Ensure copyright on titlepage (article style) and chapter pages (book style)
+\fancypagestyle{plain}{
+ \fancyhf{}
+ \fancyfoot[C]{{\footnotesize \copyright\ 1999-2018, "Data Analysis and Machine Learning FYS-STK3155/FYS4155":"http://www.uio.no/studier/emner/matnat/fys/FYS3155/index-eng.html". Released under CC Attribution-NonCommercial 4.0 license}}
+% \renewcommand{\footrulewidth}{0mm}
+ \renewcommand{\headrulewidth}{0mm}
+}
+% Ensure copyright on titlepages with \thispagestyle{empty}
+\fancypagestyle{empty}{
+ \fancyhf{}
+ \fancyfoot[C]{{\footnotesize \copyright\ 1999-2018, "Data Analysis and Machine Learning FYS-STK3155/FYS4155":"http://www.uio.no/studier/emner/matnat/fys/FYS3155/index-eng.html". Released under CC Attribution-NonCommercial 4.0 license}}
+ \renewcommand{\footrulewidth}{0mm}
+ \renewcommand{\headrulewidth}{0mm}
+}
+
+\pagestyle{fancy}
+
+
+% prevent orhpans and widows
+\clubpenalty = 10000
+\widowpenalty = 10000
+
+% --- end of standard preamble for documents ---
+
+
+% insert custom LaTeX commands...
+
+\raggedbottom
+\makeindex
+\usepackage[totoc]{idxlayout} % for index in the toc
+\usepackage[nottoc]{tocbibind} % for references/bibliography in the toc
+
+%-------------------- end preamble ----------------------
+
+\begin{document}
+
+% matching end for #ifdef PREAMBLE
+
+\newcommand{\exercisesection}[1]{\subsection*{#1}}
+
+
+% ------------------- main content ----------------------
+
+
+
+% ----------------- title -------------------------
+
+\thispagestyle{empty}
+
+\begin{center}
+{\LARGE\bf
+\begin{spacing}{1.25}
+Homework 2
+\end{spacing}
+}
+\end{center}
+
+% ----------------- author(s) -------------------------
+
+\begin{center}
+{\bf \href{{http://www.uio.no/studier/emner/matnat/fys/FYS3155/index-eng.html}}{Data Analysis and Machine Learning FYS-STK3155/FYS4155}}
+\end{center}
+
+ \begin{center}
+% List of all institutions:
+\centerline{{\small Department of Physics, University of Oslo, Norway}}
+\end{center}
+
+% ----------------- end author(s) -------------------------
+
+% --- begin date ---
+\begin{center}
+Sep 3, 2018
+\end{center}
+% --- end date ---
+
+\vspace{1cm}
+
+
+\subsection*{Exercise 4}
+
+This exercise is a continuation of exercise 2 from homework 1. We will
+use the same function to generate our data set, still staying with a
+simple function $y(x)$ which we want to fit using linear regression,
+but now extending the analysis to include the Ridge and the Lasso
+regression methods. You can use the code under the Regression as an example on how to use the Ridge and the Lasso methods, see the \href{{https://compphysics.github.io/MachineLearning/doc/pub/Regression/html/Regression-bs.html}}{regression slides}).
+
+We will thus again generate our own dataset for a function $y(x)$ where
+$x \in [0,1]$ and defined by random numbers computed with the uniform
+distribution. The function $y$ is a quadratic polynomial in $x$ with
+added stochastic noise according to the normal distribution $\cal{N}(0,1)$.
+
+The following simple Python instructions define our $x$ and $y$ values (with 100 data points).
+\begin{print}
+x = np.random.rand(100,1)
+y = 5*x*x+0.1*np.random.randn(100,1)
+\end{print}
+
+\begin{enumerate}
+\item Write your own code for the Ridge method (see chapter 3.4 of Hastie \emph{et al.}, equations (3.43) and (3.44)) and compute the parametrization for different values of $\lambda$.
+\end{enumerate}
+
+\noindent
+Compare and analyze your results with those from exercise 2. Study the dependence on $\lambda$ while also varying the strength of the noise in your expression for $y(x)$.
+
+\begin{enumerate}
+\item Repeat the above but using the functionality of \textbf{scikit-learn}. Compare your code with the results from \textbf{scikit-learn}. Remember to run with the same random numbers for generating $x$ and $y$.
+
+\item Our next step is to study the variance of the parameters $\beta_1$ and $\beta_2$ (assuming that we are parametrizing our function with a second-order polynomial. We will use standard linear regression and the Ridge regression.
+\end{enumerate}
+
+\noindent
+You can now opt for either writing your own function that calculates the variance of these paramaters (recall that this is equal to the diagonal elements of the matrix $(\hat{X}^T\hat{X})^{-1}+\lambda\hat{I}$) or use the functionality of \textbf{scikit-learn} and computetheir variances. Discuss the results of these variances as functions of $\lambda$. In particular, try to link your discussion with the discussion in Hastie \emph{et al.} and their figure 3.11.
+
+\begin{enumerate}
+\item Repeat the previous step but add now the Lasso method, see equation (3.53) of Hastie \emph{et al.}. Discuss your results and compare with standard regression and the Ridge regression results. You can write your own code or use the functionality of \textbf{scikit-learn}.
+
+\item Finally, using \textbf{scikit-learn} or your own code, compute also the mean square error, a risk metric corresponding to the expected value of the squared (quadratic) error defined as
+\end{enumerate}
+
+\noindent
+\[ MSE(\hat{y},\hat{\tilde{y}}) = \frac{1}{n}
+\sum_{i=0}^{n-1}(y_i-\tilde{y}_i)^2,
+\]
+and the $R^2$ score function.
+If $\tilde{\hat{y}}_i$ is the predicted value of the $i-th$ sample and $y_i$ is the corresponding true value, then the score $R^2$ is defined as
+\[
+R^2(\hat{y}, \tilde{\hat{y}}) = 1 - \frac{\sum_{i=0}^{n - 1} (y_i - \tilde{y}_i)^2}{\sum_{i=0}^{n - 1} (y_i - \bar{y})^2},
+\]
+where we have defined the mean value of $\hat{y}$ as
+\[
+\bar{y} = \frac{1}{n} \sum_{i=0}^{n - 1} y_i.
+\]
+Discuss these quantities as functions of the variable $\lambda$ in the Ridge and Lasso regression methods.
+
+\subsection*{Exercise 5}
+
+Using the singular value decomposition, show that the variance of the direction vector
+$\hat{z}_i=\hat{X}\hat{v}_i$ is equal to (equation (3.49) of Hastie \emph{et al.})
+\[
+\mathrm{Var}(\hat{z}_i)=\frac{d_i^2}{N},
+\]
+where $d_i$ are the singular values of the matrix $\hat{X}$. Give an interprepation of these results, in particular in connection with the variance of the coefficients you obtained in the previous exercise.
+
+% ------------------- end of main content ---------------
+
+\end{document}
+
diff --git a/doc/src/Projects/2018/Exercises/hw2.do.txt b/doc/src/Projects/2018/Exercises/hw2.do.txt
index bd2355fab..48412eeb1 100644
--- a/doc/src/Projects/2018/Exercises/hw2.do.txt
+++ b/doc/src/Projects/2018/Exercises/hw2.do.txt
@@ -1,12 +1,65 @@
-TITLE: Homework 1
+TITLE: Homework 2
AUTHOR: "Data Analysis and Machine Learning FYS-STK3155/FYS4155":"http://www.uio.no/studier/emner/matnat/fys/FYS3155/index-eng.html" {copyright, 1999-present|CC BY-NC} at Department of Physics, University of Oslo, Norway
DATE:Today
-===== Exercise 1 =====
+===== Exercise 4 =====
-Make sure you have installed all necessary t
+This exercise is a continuation of exercise 2 from homework 1. We will
+use the same function to generate our data set, still staying with a
+simple function $y(x)$ which we want to fit using linear regression,
+but now extending the analysis to include the Ridge and the Lasso
+regression methods. You can use the code under the Regression as an example on how to use the Ridge and the Lasso methods, see the "regression slides":"https://compphysics.github.io/MachineLearning/doc/pub/Regression/html/Regression-bs.html").
-===== Exercise 2 =====
+We will thus again generate our own dataset for a function $y(x)$ where
+$x \in [0,1]$ and defined by random numbers computed with the uniform
+distribution. The function $y$ is a quadratic polynomial in $x$ with
+added stochastic noise according to the normal distribution $\cal{N}(0,1)$.
-===== Exercise 3 =====
+The following simple Python instructions define our $x$ and $y$ values (with 100 data points).
+!bc pycod
+x = np.random.rand(100,1)
+y = 5*x*x+0.1*np.random.randn(100,1)
+!ec
+
+o Write your own code for the Ridge method (see chapter 3.4 of Hastie *et al.*, equations (3.43) and (3.44)) and compute the parametrization for different values of $\lambda$.
+Compare and analyze your results with those from exercise 2. Study the dependence on $\lambda$ while also varying the strength of the noise in your expression for $y(x)$.
+
+o Repeat the above but using the functionality of _scikit-learn_. Compare your code with the results from _scikit-learn_. Remember to run with the same random numbers for generating $x$ and $y$.
+
+o Our next step is to study the variance of the parameters $\beta_1$ and $\beta_2$ (assuming that we are parametrizing our function with a second-order polynomial. We will use standard linear regression and the Ridge regression.
+You can now opt for either writing your own function that calculates the variance of these paramaters (recall that this is equal to the diagonal elements of the matrix $(\hat{X}^T\hat{X})^{-1}+\lambda\hat{I}$) or use the functionality of _scikit-learn_ and computetheir variances. Discuss the results of these variances as functions of $\lambda$. In particular, try to link your discussion with the discussion in Hastie *et al.* and their figure 3.11.
+
+o Repeat the previous step but add now the Lasso method, see equation (3.53) of Hastie *et al.*. Discuss your results and compare with standard regression and the Ridge regression results. You can write your own code or use the functionality of _scikit-learn_.
+
+o Finally, using _scikit-learn_ or your own code, compute also the mean square error, a risk metric corresponding to the expected value of the squared (quadratic) error defined as
+!bt
+\[ MSE(\hat{y},\hat{\tilde{y}}) = \frac{1}{n}
+\sum_{i=0}^{n-1}(y_i-\tilde{y}_i)^2,
+\]
+!et
+and the $R^2$ score function.
+If $\tilde{\hat{y}}_i$ is the predicted value of the $i-th$ sample and $y_i$ is the corresponding true value, then the score $R^2$ is defined as
+!bt
+\[
+R^2(\hat{y}, \tilde{\hat{y}}) = 1 - \frac{\sum_{i=0}^{n - 1} (y_i - \tilde{y}_i)^2}{\sum_{i=0}^{n - 1} (y_i - \bar{y})^2},
+\]
+!et
+where we have defined the mean value of $\hat{y}$ as
+!bt
+\[
+\bar{y} = \frac{1}{n} \sum_{i=0}^{n - 1} y_i.
+\]
+!et
+Discuss these quantities as functions of the variable $\lambda$ in the Ridge and Lasso regression methods.
+
+===== Exercise 5 =====
+
+Using the singular value decomposition, show that the variance of the direction vector
+$\hat{z}_i=\hat{X}\hat{v}_i$ is equal to (equation (3.49) of Hastie *et al.*)
+!bt
+\[
+\mathrm{Var}(\hat{z}_i)=\frac{d_i^2}{N},
+\]
+!et
+where $d_i$ are the singular values of the matrix $\hat{X}$. Give an interprepation of these results, in particular in connection with the variance of the coefficients you obtained in the previous exercise.
diff --git a/doc/web/course.do.txt b/doc/web/course.do.txt
index d98c3a3ac..68763e1ff 100644
--- a/doc/web/course.do.txt
+++ b/doc/web/course.do.txt
@@ -96,6 +96,14 @@ ${text_types(ch)}
* "Plain html":"http://compphysics.github.io/MachineLearning/doc/Projects/2018/hw1/html/hw1.html"
* "Bootstrap slide style, easy for reading on mobile devices": "http://compphysics.github.io/MachineLearning/doc/Projects/2018/hw1/html/hw1-bs.html"
+=== Second homework set, week 36 ===
+ * LaTeX and PDF:
+ * "LaTex file":"http://compphysics.github.io/MachineLearning/doc/Projects/2018/hw2/pdf/hw2.tex"
+ * "PDF file":"http://compphysics.github.io/MachineLearning/doc/Projects/2018/hw2/pdf/hw2.pdf"
+ * HTML:
+ * "Plain html":"http://compphysics.github.io/MachineLearning/doc/Projects/2018/hw2/html/hw2.html"
+ * "Bootstrap slide style, easy for reading on mobile devices": "http://compphysics.github.io/MachineLearning/doc/Projects/2018/hw2/html/hw2-bs.html"
+
=== Project 1, Deadline October 1 ===
diff --git a/doc/web/course.html b/doc/web/course.html
index f17cafa09..b207da6d6 100644
--- a/doc/web/course.html
+++ b/doc/web/course.html
@@ -99,25 +99,26 @@ div { text-align: justify; text-justify: inter-word; }
('Teach yourself C++', 2, None, '___sec13'),
('Projects and Exercises Fall 2018', 2, None, '___sec14'),
('First homework set, week 35', 3, None, '___sec15'),
- ('Project 1, Deadline October 1', 3, None, '___sec16'),
- ('Project 2, Deadline November 5', 3, None, '___sec17'),
- ('Project 3, Deadline November 30', 3, None, '___sec18'),
- ('Course content', 3, None, '___sec19'),
- ('Learning outcomes', 2, None, '___sec20'),
- ('Prerequisites', 2, None, '___sec21'),
- ('The course has two central parts', 2, None, '___sec22'),
+ ('Second homework set, week 36', 3, None, '___sec16'),
+ ('Project 1, Deadline October 1', 3, None, '___sec17'),
+ ('Project 2, Deadline November 5', 3, None, '___sec18'),
+ ('Project 3, Deadline November 30', 3, None, '___sec19'),
+ ('Course content', 3, None, '___sec20'),
+ ('Learning outcomes', 2, None, '___sec21'),
+ ('Prerequisites', 2, None, '___sec22'),
+ ('The course has two central parts', 2, None, '___sec23'),
('Statistical analysis and optimization of data',
3,
None,
- '___sec23'),
- ('Machine learning', 3, None, '___sec24'),
- ('Recommended textbooks', 2, None, '___sec25'),
+ '___sec24'),
+ ('Machine learning', 3, None, '___sec25'),
+ ('Recommended textbooks', 2, None, '___sec26'),
('"Other '
'textbooks":"https://github.com/CompPhysics/MachineLearning/tree/master/doc/Textbooks"',
2,
None,
- '___sec26'),
- ('Teaching schedule Fall 2018', 2, None, '___sec27')]}
+ '___sec27'),
+ ('Teaching schedule Fall 2018', 2, None, '___sec28')]}
end of tocinfo -->
@@ -577,7 +578,26 @@ formulas in HTML or ipython notebook files.
-
@@ -596,7 +616,7 @@ formulas in HTML or ipython notebook files.
-
Project 2, Deadline November 5
+
Project 2, Deadline November 5
LaTeX and PDF:
@@ -615,7 +635,7 @@ formulas in HTML or ipython notebook files.
-
Project 3, Deadline November 30
+
Project 3, Deadline November 30
LaTeX and PDF:
@@ -634,7 +654,7 @@ formulas in HTML or ipython notebook files.
-
Course content
+
Course content
Probability theory and statistical methods play a central role in science. Nowadays we are
@@ -653,7 +673,7 @@ tools of probability theory, the aim of this course is to expose you to central
This course covers thus topics like Monte Carlo methods and Markov chains, Bayesian statistics, error estimates, various linear methods, optimization of data and error analysis and central algorithms in machine learning.
The course has several numerical projects and numerical exercises that are meant to illustrate the theory.
-
Learning outcomes
+
Learning outcomes
The course introduces a variety of central algorithms and methods
@@ -670,19 +690,19 @@ essential for studies of data analysis and machine learning. The course is proje
Work on numerical projects to illustrate the theory. The projects play a central role and students are expected to know modern programming languages like Python or C++.
-
Prerequisites
+
Prerequisites
Basic knowledge in programming and numerics. Required courses are the equivalents to the University of Oslo mathematics courses MAT1100, MAT1110, MAT1120 and at least one of the corresponding computing and programming courses INF1000/INF1110 or MAT-INF1100/MAT-INF1100L/BIOS1100/KJM-INF1xxx.
-
The course has two central parts
+
The course has two central parts
Statistical analysis and optimization of data
Machine learning
-
Statistical analysis and optimization of data
+
Statistical analysis and optimization of data
The following topics will be covered
@@ -699,7 +719,7 @@ The following topics will be covered
Practical optimization using Singular-value decomposition and least squares for parameterizing data.
-
Machine learning
+
Machine learning
The following topics will be covered
@@ -715,14 +735,14 @@ The following topics will be covered
All the above topics will be supported by examples, hands-on exercises and project work.
-