Corrected exercise 4 and changed strange error
This commit is contained in:
mhjensen
@@ -145,10 +145,10 @@ y <span style="color: #666666">=</span> <span style="color: #666666">5*</span>x<
|
||||
</pre></div>
|
||||
<ol>
|
||||
<li> Write your own code for the Ridge method (see chapter 3.4 of Hastie <em>et al.</em>, equations (3.43) and (3.44)) and compute the parametrization for different values of \( \lambda \). Compare and analyze your results with those from exercise 2. Study the dependence on \( \lambda \) while also varying the strength of the noise in your expression for \( y(x) \).</li>
|
||||
<li> Repeat the above but using the functionality of <b>scikit-learn</b>. Compare your code with the results from <b>scikit-learn</b>. Remember to run with the same random numbers for generating \( x \) and \( y \).</li>
|
||||
<li> Our next step is to study the variance of the parameters \( \beta_1 \) and \( \beta_2 \) (assuming that we are parametrizing our function with a second-order polynomial. We will use standard linear regression and the Ridge regression. You can now opt for either writing your own function that calculates the variance of these paramaters (recall that this is equal to the diagonal elements of the matrix \( (\hat{X}^T\hat{X})+\lambda\hat{I})^{-1} \)) or use the functionality of <b>scikit-learn</b> and compute their variances. Discuss the results of these variances as functions of \( \lambda \). In particular, try to link your discussion with the discussion in Hastie <em>et al.</em> and their figure 3.11.</li>
|
||||
<li> Repeat the previous step but add now the Lasso method, see equation (3.53) of Hastie <em>et al.</em>. Discuss your results and compare with standard regression and the Ridge regression results. You can write your own code or use the functionality of <b>scikit-learn</b>. We recommend the last since we have not yet discussed how to solve the Lasso equations numerically.</li>
|
||||
<li> Finally, using <b>scikit-learn</b> or your own code, compute also the mean square error, a risk metric corresponding to the expected value of the squared (quadratic) error defined as</li>
|
||||
<li> Repeat the above but using the functionality of <b>Scikit-Learn</b>. Compare your code with the results from <b>Scikit-Learn</b>. Remember to run with the same random numbers for generating \( x \) and \( y \).</li>
|
||||
<li> Our next step is to study the variance of the parameters \( \beta_1 \) and \( \beta_2 \) (assuming that we are parameterizing our function with a second-order polynomial). We will use standard linear regression and the Ridge regression. You can now opt for either writing your own function or using <b>Scikit-Learn</b> to find the parameters \( \beta \). From your results calculate the variance of these paramaters (recall that this is equal to the diagonal elements of the matrix \( (\hat{X}^T\hat{X})+\lambda\hat{I})^{-1} \)). Discuss the results of these variances as functions of \( \lambda \). In particular, try to link your discussion with the discussion in Hastie <em>et al.</em> and their figure 3.11.</li>
|
||||
<li> Repeat the previous step but add now the Lasso method, see equation (3.53) of Hastie <em>et al.</em>. Discuss your results and compare with standard regression and the Ridge regression results. You can write your own code or use the functionality of <b>scikit-learn</b>. We recommend the latter since we have not yet discussed how to solve the Lasso equations numerically.</li>
|
||||
<li> Finally, using <b>Scikit-Learn</b> or your own code, compute also the mean square error, a risk metric corresponding to the expected value of the squared (quadratic) error defined as</li>
|
||||
</ol>
|
||||
|
||||
$$ MSE(\hat{y},\hat{\tilde{y}}) = \frac{1}{n}
|
||||
@@ -171,7 +171,7 @@ Discuss these quantities as functions of the variable \( \lambda \) in the Ridge
|
||||
<h2 id="___sec1" class="anchor">Exercise 5 </h2>
|
||||
|
||||
<p>
|
||||
The theory behind this exercise will be covered during the lectures of week 36. It requires reading chapter three of Hastie et al, in particular the derivations preceeding equation (3.49) as the well as the material in the Regression slides the singular value decomposition.
|
||||
The theory behind this exercise will be covered during the lectures of week 36. It requires reading chapter three of Hastie et al, in particular the derivations preceeding equation (3.49) as the well as the material in the Regression slides that deal with the singular value decomposition.
|
||||
|
||||
<p>
|
||||
Using the singular value decomposition, show that the variance of the direction vector
|
||||
|
||||
@@ -145,10 +145,10 @@ y <span style="color: #666666">=</span> <span style="color: #666666">5*</span>x<
|
||||
</pre></div>
|
||||
<ol>
|
||||
<li> Write your own code for the Ridge method (see chapter 3.4 of Hastie <em>et al.</em>, equations (3.43) and (3.44)) and compute the parametrization for different values of \( \lambda \). Compare and analyze your results with those from exercise 2. Study the dependence on \( \lambda \) while also varying the strength of the noise in your expression for \( y(x) \).</li>
|
||||
<li> Repeat the above but using the functionality of <b>scikit-learn</b>. Compare your code with the results from <b>scikit-learn</b>. Remember to run with the same random numbers for generating \( x \) and \( y \).</li>
|
||||
<li> Our next step is to study the variance of the parameters \( \beta_1 \) and \( \beta_2 \) (assuming that we are parametrizing our function with a second-order polynomial. We will use standard linear regression and the Ridge regression. You can now opt for either writing your own function that calculates the variance of these paramaters (recall that this is equal to the diagonal elements of the matrix \( (\hat{X}^T\hat{X})+\lambda\hat{I})^{-1} \)) or use the functionality of <b>scikit-learn</b> and compute their variances. Discuss the results of these variances as functions of \( \lambda \). In particular, try to link your discussion with the discussion in Hastie <em>et al.</em> and their figure 3.11.</li>
|
||||
<li> Repeat the previous step but add now the Lasso method, see equation (3.53) of Hastie <em>et al.</em>. Discuss your results and compare with standard regression and the Ridge regression results. You can write your own code or use the functionality of <b>scikit-learn</b>. We recommend the last since we have not yet discussed how to solve the Lasso equations numerically.</li>
|
||||
<li> Finally, using <b>scikit-learn</b> or your own code, compute also the mean square error, a risk metric corresponding to the expected value of the squared (quadratic) error defined as</li>
|
||||
<li> Repeat the above but using the functionality of <b>Scikit-Learn</b>. Compare your code with the results from <b>Scikit-Learn</b>. Remember to run with the same random numbers for generating \( x \) and \( y \).</li>
|
||||
<li> Our next step is to study the variance of the parameters \( \beta_1 \) and \( \beta_2 \) (assuming that we are parameterizing our function with a second-order polynomial). We will use standard linear regression and the Ridge regression. You can now opt for either writing your own function or using <b>Scikit-Learn</b> to find the parameters \( \beta \). From your results calculate the variance of these paramaters (recall that this is equal to the diagonal elements of the matrix \( (\hat{X}^T\hat{X})+\lambda\hat{I})^{-1} \)). Discuss the results of these variances as functions of \( \lambda \). In particular, try to link your discussion with the discussion in Hastie <em>et al.</em> and their figure 3.11.</li>
|
||||
<li> Repeat the previous step but add now the Lasso method, see equation (3.53) of Hastie <em>et al.</em>. Discuss your results and compare with standard regression and the Ridge regression results. You can write your own code or use the functionality of <b>scikit-learn</b>. We recommend the latter since we have not yet discussed how to solve the Lasso equations numerically.</li>
|
||||
<li> Finally, using <b>Scikit-Learn</b> or your own code, compute also the mean square error, a risk metric corresponding to the expected value of the squared (quadratic) error defined as</li>
|
||||
</ol>
|
||||
|
||||
$$ MSE(\hat{y},\hat{\tilde{y}}) = \frac{1}{n}
|
||||
@@ -171,7 +171,7 @@ Discuss these quantities as functions of the variable \( \lambda \) in the Ridge
|
||||
<h2 id="___sec1" class="anchor">Exercise 5 </h2>
|
||||
|
||||
<p>
|
||||
The theory behind this exercise will be covered during the lectures of week 36. It requires reading chapter three of Hastie et al, in particular the derivations preceeding equation (3.49) as the well as the material in the Regression slides the singular value decomposition.
|
||||
The theory behind this exercise will be covered during the lectures of week 36. It requires reading chapter three of Hastie et al, in particular the derivations preceeding equation (3.49) as the well as the material in the Regression slides that deal with the singular value decomposition.
|
||||
|
||||
<p>
|
||||
Using the singular value decomposition, show that the variance of the direction vector
|
||||
|
||||
@@ -110,10 +110,10 @@ y <span style="color: #666666">=</span> <span style="color: #666666">5*</span>x<
|
||||
</pre></div>
|
||||
<ol>
|
||||
<li> Write your own code for the Ridge method (see chapter 3.4 of Hastie <em>et al.</em>, equations (3.43) and (3.44)) and compute the parametrization for different values of \( \lambda \). Compare and analyze your results with those from exercise 2. Study the dependence on \( \lambda \) while also varying the strength of the noise in your expression for \( y(x) \).</li>
|
||||
<li> Repeat the above but using the functionality of <b>scikit-learn</b>. Compare your code with the results from <b>scikit-learn</b>. Remember to run with the same random numbers for generating \( x \) and \( y \).</li>
|
||||
<li> Our next step is to study the variance of the parameters \( \beta_1 \) and \( \beta_2 \) (assuming that we are parametrizing our function with a second-order polynomial. We will use standard linear regression and the Ridge regression. You can now opt for either writing your own function that calculates the variance of these paramaters (recall that this is equal to the diagonal elements of the matrix \( (\hat{X}^T\hat{X})+\lambda\hat{I})^{-1} \)) or use the functionality of <b>scikit-learn</b> and compute their variances. Discuss the results of these variances as functions of \( \lambda \). In particular, try to link your discussion with the discussion in Hastie <em>et al.</em> and their figure 3.11.</li>
|
||||
<li> Repeat the previous step but add now the Lasso method, see equation (3.53) of Hastie <em>et al.</em>. Discuss your results and compare with standard regression and the Ridge regression results. You can write your own code or use the functionality of <b>scikit-learn</b>. We recommend the last since we have not yet discussed how to solve the Lasso equations numerically.</li>
|
||||
<li> Finally, using <b>scikit-learn</b> or your own code, compute also the mean square error, a risk metric corresponding to the expected value of the squared (quadratic) error defined as</li>
|
||||
<li> Repeat the above but using the functionality of <b>Scikit-Learn</b>. Compare your code with the results from <b>Scikit-Learn</b>. Remember to run with the same random numbers for generating \( x \) and \( y \).</li>
|
||||
<li> Our next step is to study the variance of the parameters \( \beta_1 \) and \( \beta_2 \) (assuming that we are parameterizing our function with a second-order polynomial). We will use standard linear regression and the Ridge regression. You can now opt for either writing your own function or using <b>Scikit-Learn</b> to find the parameters \( \beta \). From your results calculate the variance of these paramaters (recall that this is equal to the diagonal elements of the matrix \( (\hat{X}^T\hat{X})+\lambda\hat{I})^{-1} \)). Discuss the results of these variances as functions of \( \lambda \). In particular, try to link your discussion with the discussion in Hastie <em>et al.</em> and their figure 3.11.</li>
|
||||
<li> Repeat the previous step but add now the Lasso method, see equation (3.53) of Hastie <em>et al.</em>. Discuss your results and compare with standard regression and the Ridge regression results. You can write your own code or use the functionality of <b>scikit-learn</b>. We recommend the latter since we have not yet discussed how to solve the Lasso equations numerically.</li>
|
||||
<li> Finally, using <b>Scikit-Learn</b> or your own code, compute also the mean square error, a risk metric corresponding to the expected value of the squared (quadratic) error defined as</li>
|
||||
</ol>
|
||||
|
||||
$$ MSE(\hat{y},\hat{\tilde{y}}) = \frac{1}{n}
|
||||
@@ -136,7 +136,7 @@ Discuss these quantities as functions of the variable \( \lambda \) in the Ridge
|
||||
<h2 id="___sec1">Exercise 5 </h2>
|
||||
|
||||
<p>
|
||||
The theory behind this exercise will be covered during the lectures of week 36. It requires reading chapter three of Hastie et al, in particular the derivations preceeding equation (3.49) as the well as the material in the Regression slides the singular value decomposition.
|
||||
The theory behind this exercise will be covered during the lectures of week 36. It requires reading chapter three of Hastie et al, in particular the derivations preceeding equation (3.49) as the well as the material in the Regression slides that deal with the singular value decomposition.
|
||||
|
||||
<p>
|
||||
Using the singular value decomposition, show that the variance of the direction vector
|
||||
|
||||
Binary file not shown.
@@ -184,13 +184,13 @@ y = 5*x*x+0.1*np.random.randn(100,1)
|
||||
\begin{enumerate}
|
||||
\item Write your own code for the Ridge method (see chapter 3.4 of Hastie \emph{et al.}, equations (3.43) and (3.44)) and compute the parametrization for different values of $\lambda$. Compare and analyze your results with those from exercise 2. Study the dependence on $\lambda$ while also varying the strength of the noise in your expression for $y(x)$.
|
||||
|
||||
\item Repeat the above but using the functionality of \textbf{scikit-learn}. Compare your code with the results from \textbf{scikit-learn}. Remember to run with the same random numbers for generating $x$ and $y$.
|
||||
\item Repeat the above but using the functionality of \textbf{Scikit-Learn}. Compare your code with the results from \textbf{Scikit-Learn}. Remember to run with the same random numbers for generating $x$ and $y$.
|
||||
|
||||
\item Our next step is to study the variance of the parameters $\beta_1$ and $\beta_2$ (assuming that we are parametrizing our function with a second-order polynomial. We will use standard linear regression and the Ridge regression. You can now opt for either writing your own function that calculates the variance of these paramaters (recall that this is equal to the diagonal elements of the matrix $(\hat{X}^T\hat{X})+\lambda\hat{I})^{-1}$) or use the functionality of \textbf{scikit-learn} and compute their variances. Discuss the results of these variances as functions of $\lambda$. In particular, try to link your discussion with the discussion in Hastie \emph{et al.} and their figure 3.11.
|
||||
\item Our next step is to study the variance of the parameters $\beta_1$ and $\beta_2$ (assuming that we are parameterizing our function with a second-order polynomial). We will use standard linear regression and the Ridge regression. You can now opt for either writing your own function or using \textbf{Scikit-Learn} to find the parameters $\beta$. From your results calculate the variance of these paramaters (recall that this is equal to the diagonal elements of the matrix $(\hat{X}^T\hat{X})+\lambda\hat{I})^{-1}$). Discuss the results of these variances as functions of $\lambda$. In particular, try to link your discussion with the discussion in Hastie \emph{et al.} and their figure 3.11.
|
||||
|
||||
\item Repeat the previous step but add now the Lasso method, see equation (3.53) of Hastie \emph{et al.}. Discuss your results and compare with standard regression and the Ridge regression results. You can write your own code or use the functionality of \textbf{scikit-learn}. We recommend the last since we have not yet discussed how to solve the Lasso equations numerically.
|
||||
\item Repeat the previous step but add now the Lasso method, see equation (3.53) of Hastie \emph{et al.}. Discuss your results and compare with standard regression and the Ridge regression results. You can write your own code or use the functionality of \textbf{scikit-learn}. We recommend the latter since we have not yet discussed how to solve the Lasso equations numerically.
|
||||
|
||||
\item Finally, using \textbf{scikit-learn} or your own code, compute also the mean square error, a risk metric corresponding to the expected value of the squared (quadratic) error defined as
|
||||
\item Finally, using \textbf{Scikit-Learn} or your own code, compute also the mean square error, a risk metric corresponding to the expected value of the squared (quadratic) error defined as
|
||||
\end{enumerate}
|
||||
|
||||
\noindent
|
||||
@@ -210,7 +210,7 @@ Discuss these quantities as functions of the variable $\lambda$ in the Ridge and
|
||||
|
||||
\subsection{Exercise 5}
|
||||
|
||||
The theory behind this exercise will be covered during the lectures of week 36. It requires reading chapter three of Hastie et al, in particular the derivations preceeding equation (3.49) as the well as the material in the Regression slides the singular value decomposition.
|
||||
The theory behind this exercise will be covered during the lectures of week 36. It requires reading chapter three of Hastie et al, in particular the derivations preceeding equation (3.49) as the well as the material in the Regression slides that deal with the singular value decomposition.
|
||||
|
||||
Using the singular value decomposition, show that the variance of the direction vector
|
||||
$\hat{z}_i=\hat{X}\hat{v}_i=\hat{u}_1d_1$ is equal to (equation (3.49) of Hastie \emph{et al.})
|
||||
|
||||
Binary file not shown.
@@ -143,7 +143,7 @@ regression methods. You can use the code under the Regression as an example on h
|
||||
We will thus again generate our own dataset for a function $y(x)$ where
|
||||
$x \in [0,1]$ and defined by random numbers computed with the uniform
|
||||
distribution. The function $y$ is a quadratic polynomial in $x$ with
|
||||
added stochastic noise according to the normal distribution $\cal{N}(0,1)$.
|
||||
added stochastic noise according to the normal distribution $N(\mu=0,\sigma^2=1)$.
|
||||
|
||||
The following simple Python instructions define our $x$ and $y$ values (with 100 data points).
|
||||
\begin{verbatim}
|
||||
@@ -154,13 +154,13 @@ y = 5*x*x+0.1*np.random.randn(100,1)
|
||||
\begin{enumerate}
|
||||
\item Write your own code for the Ridge method (see chapter 3.4 of Hastie \emph{et al.}, equations (3.43) and (3.44)) and compute the parametrization for different values of $\lambda$. Compare and analyze your results with those from exercise 2. Study the dependence on $\lambda$ while also varying the strength of the noise in your expression for $y(x)$.
|
||||
|
||||
\item Repeat the above but using the functionality of \textbf{scikit-learn}. Compare your code with the results from \textbf{scikit-learn}. Remember to run with the same random numbers for generating $x$ and $y$.
|
||||
\item Repeat the above but using the functionality of \textbf{Scikit-Learn}. Compare your code with the results from \textbf{Scikit-Learn}. Remember to run with the same random numbers for generating $x$ and $y$.
|
||||
|
||||
\item Our next step is to study the variance of the parameters $\beta_1$ and $\beta_2$ (assuming that we are parametrizing our function with a second-order polynomial. We will use standard linear regression and the Ridge regression. You can now opt for either writing your own function that calculates the variance of these paramaters (recall that this is equal to the diagonal elements of the matrix $(\hat{X}^T\hat{X})+\lambda\hat{I})^{-1}$) or use the functionality of \textbf{scikit-learn} and compute their variances. Discuss the results of these variances as functions of $\lambda$. In particular, try to link your discussion with the discussion in Hastie \emph{et al.} and their figure 3.11.
|
||||
\item Our next step is to study the variance of the parameters $\beta_1$ and $\beta_2$ (assuming that we are parameterizing our function with a second-order polynomial). We will use standard linear regression and the Ridge regression. You can now opt for either writing your own function or using \textbf{Scikit-Learn} to find the parameters $\beta$. From your results calculate the variance of these paramaters (recall that this is equal to the diagonal elements of the matrix $(\hat{X}^T\hat{X})+\lambda\hat{I})^{-1}$). Discuss the results of these variances as functions of $\lambda$. In particular, try to link your discussion with the discussion in Hastie \emph{et al.} and their figure 3.11.
|
||||
|
||||
\item Repeat the previous step but add now the Lasso method, see equation (3.53) of Hastie \emph{et al.}. Discuss your results and compare with standard regression and the Ridge regression results. You can write your own code or use the functionality of \textbf{scikit-learn}. We recommend the last since we have not yet discussed how to solve the Lasso equations numerically.
|
||||
\item Repeat the previous step but add now the Lasso method, see equation (3.53) of Hastie \emph{et al.}. Discuss your results and compare with standard regression and the Ridge regression results. You can write your own code or use the functionality of \textbf{scikit-learn}. We recommend the latter since we have not yet discussed how to solve the Lasso equations numerically.
|
||||
|
||||
\item Finally, using \textbf{scikit-learn} or your own code, compute also the mean square error, a risk metric corresponding to the expected value of the squared (quadratic) error defined as
|
||||
\item Finally, using \textbf{Scikit-Learn} or your own code, compute also the mean square error, a risk metric corresponding to the expected value of the squared (quadratic) error defined as
|
||||
\end{enumerate}
|
||||
|
||||
\noindent
|
||||
@@ -180,7 +180,7 @@ Discuss these quantities as functions of the variable $\lambda$ in the Ridge and
|
||||
|
||||
\subsection*{Exercise 5}
|
||||
|
||||
The theory behind this exercise will be covered during the lectures of week 36. It requires reading chapter three of Hastie et al, in particular the derivations preceeding equation (3.49) as the well as the material in the Regression slides the singular value decomposition.
|
||||
The theory behind this exercise will be covered during the lectures of week 36. It requires reading chapter three of Hastie et al, in particular the derivations preceeding equation (3.49) as the well as the material in the Regression slides that deal with the singular value decomposition.
|
||||
|
||||
Using the singular value decomposition, show that the variance of the direction vector
|
||||
$\hat{z}_i=\hat{X}\hat{v}_i=\hat{u}_1d_1$ is equal to (equation (3.49) of Hastie \emph{et al.})
|
||||
|
||||
@@ -146,21 +146,21 @@ distribution. The function $y$ is a quadratic polynomial in $x$ with
|
||||
added stochastic noise according to the normal distribution $\cal{N}(0,1)$.
|
||||
|
||||
The following simple Python instructions define our $x$ and $y$ values (with 100 data points).
|
||||
\begin{print}
|
||||
\begin{verbatim}
|
||||
x = np.random.rand(100,1)
|
||||
y = 5*x*x+0.1*np.random.randn(100,1)
|
||||
\end{print}
|
||||
\end{verbatim}
|
||||
|
||||
\begin{enumerate}
|
||||
\item Write your own code for the Ridge method (see chapter 3.4 of Hastie \emph{et al.}, equations (3.43) and (3.44)) and compute the parametrization for different values of $\lambda$. Compare and analyze your results with those from exercise 2. Study the dependence on $\lambda$ while also varying the strength of the noise in your expression for $y(x)$.
|
||||
|
||||
\item Repeat the above but using the functionality of \textbf{scikit-learn}. Compare your code with the results from \textbf{scikit-learn}. Remember to run with the same random numbers for generating $x$ and $y$.
|
||||
\item Repeat the above but using the functionality of \textbf{Scikit-Learn}. Compare your code with the results from \textbf{Scikit-Learn}. Remember to run with the same random numbers for generating $x$ and $y$.
|
||||
|
||||
\item Our next step is to study the variance of the parameters $\beta_1$ and $\beta_2$ (assuming that we are parametrizing our function with a second-order polynomial. We will use standard linear regression and the Ridge regression. You can now opt for either writing your own function that calculates the variance of these paramaters (recall that this is equal to the diagonal elements of the matrix $(\hat{X}^T\hat{X})+\lambda\hat{I})^{-1}$) or use the functionality of \textbf{scikit-learn} and compute their variances. Discuss the results of these variances as functions of $\lambda$. In particular, try to link your discussion with the discussion in Hastie \emph{et al.} and their figure 3.11.
|
||||
\item Our next step is to study the variance of the parameters $\beta_1$ and $\beta_2$ (assuming that we are parameterizing our function with a second-order polynomial). We will use standard linear regression and the Ridge regression. You can now opt for either writing your own function or using \textbf{Scikit-Learn} to find the parameters $\beta$. From your results calculate the variance of these paramaters (recall that this is equal to the diagonal elements of the matrix $(\hat{X}^T\hat{X})+\lambda\hat{I})^{-1}$). Discuss the results of these variances as functions of $\lambda$. In particular, try to link your discussion with the discussion in Hastie \emph{et al.} and their figure 3.11.
|
||||
|
||||
\item Repeat the previous step but add now the Lasso method, see equation (3.53) of Hastie \emph{et al.}. Discuss your results and compare with standard regression and the Ridge regression results. You can write your own code or use the functionality of \textbf{scikit-learn}. We recommend the last since we have not yet discussed how to solve the Lasso equations numerically.
|
||||
\item Repeat the previous step but add now the Lasso method, see equation (3.53) of Hastie \emph{et al.}. Discuss your results and compare with standard regression and the Ridge regression results. You can write your own code or use the functionality of \textbf{scikit-learn}. We recommend the latter since we have not yet discussed how to solve the Lasso equations numerically.
|
||||
|
||||
\item Finally, using \textbf{scikit-learn} or your own code, compute also the mean square error, a risk metric corresponding to the expected value of the squared (quadratic) error defined as
|
||||
\item Finally, using \textbf{Scikit-Learn} or your own code, compute also the mean square error, a risk metric corresponding to the expected value of the squared (quadratic) error defined as
|
||||
\end{enumerate}
|
||||
|
||||
\noindent
|
||||
@@ -180,7 +180,7 @@ Discuss these quantities as functions of the variable $\lambda$ in the Ridge and
|
||||
|
||||
\subsection*{Exercise 5}
|
||||
|
||||
The theory behind this exercise will be covered during the lectures of week 36. It requires reading chapter three of Hastie et al, in particular the derivations preceeding equation (3.49) as the well as the material in the Regression slides the singular value decomposition.
|
||||
The theory behind this exercise will be covered during the lectures of week 36. It requires reading chapter three of Hastie et al, in particular the derivations preceeding equation (3.49) as the well as the material in the Regression slides that deal with the singular value decomposition.
|
||||
|
||||
Using the singular value decomposition, show that the variance of the direction vector
|
||||
$\hat{z}_i=\hat{X}\hat{v}_i=\hat{u}_1d_1$ is equal to (equation (3.49) of Hastie \emph{et al.})
|
||||
|
||||
@@ -24,13 +24,13 @@ y = 5*x*x+0.1*np.random.randn(100,1)
|
||||
|
||||
o Write your own code for the Ridge method (see chapter 3.4 of Hastie *et al.*, equations (3.43) and (3.44)) and compute the parametrization for different values of $\lambda$. Compare and analyze your results with those from exercise 2. Study the dependence on $\lambda$ while also varying the strength of the noise in your expression for $y(x)$.
|
||||
|
||||
o Repeat the above but using the functionality of _scikit-learn_. Compare your code with the results from _scikit-learn_. Remember to run with the same random numbers for generating $x$ and $y$.
|
||||
o Repeat the above but using the functionality of _Scikit-Learn_. Compare your code with the results from _Scikit-Learn_. Remember to run with the same random numbers for generating $x$ and $y$.
|
||||
|
||||
o Our next step is to study the variance of the parameters $\beta_1$ and $\beta_2$ (assuming that we are parametrizing our function with a second-order polynomial. We will use standard linear regression and the Ridge regression. You can now opt for either writing your own function that calculates the variance of these paramaters (recall that this is equal to the diagonal elements of the matrix $(\hat{X}^T\hat{X})+\lambda\hat{I})^{-1}$) or use the functionality of _scikit-learn_ and compute their variances. Discuss the results of these variances as functions of $\lambda$. In particular, try to link your discussion with the discussion in Hastie *et al.* and their figure 3.11.
|
||||
o Our next step is to study the variance of the parameters $\beta_1$ and $\beta_2$ (assuming that we are parameterizing our function with a second-order polynomial). We will use standard linear regression and the Ridge regression. You can now opt for either writing your own function or using _Scikit-Learn_ to find the parameters $\beta$. From your results calculate the variance of these paramaters (recall that this is equal to the diagonal elements of the matrix $(\hat{X}^T\hat{X})+\lambda\hat{I})^{-1}$). Discuss the results of these variances as functions of $\lambda$. In particular, try to link your discussion with the discussion in Hastie *et al.* and their figure 3.11.
|
||||
|
||||
o Repeat the previous step but add now the Lasso method, see equation (3.53) of Hastie *et al.*. Discuss your results and compare with standard regression and the Ridge regression results. You can write your own code or use the functionality of _scikit-learn_. We recommend the last since we have not yet discussed how to solve the Lasso equations numerically.
|
||||
o Repeat the previous step but add now the Lasso method, see equation (3.53) of Hastie *et al.*. Discuss your results and compare with standard regression and the Ridge regression results. You can write your own code or use the functionality of _scikit-learn_. We recommend the latter since we have not yet discussed how to solve the Lasso equations numerically.
|
||||
|
||||
o Finally, using _scikit-learn_ or your own code, compute also the mean square error, a risk metric corresponding to the expected value of the squared (quadratic) error defined as
|
||||
o Finally, using _Scikit-Learn_ or your own code, compute also the mean square error, a risk metric corresponding to the expected value of the squared (quadratic) error defined as
|
||||
!bt
|
||||
\[ MSE(\hat{y},\hat{\tilde{y}}) = \frac{1}{n}
|
||||
\sum_{i=0}^{n-1}(y_i-\tilde{y}_i)^2,
|
||||
@@ -53,7 +53,7 @@ Discuss these quantities as functions of the variable $\lambda$ in the Ridge and
|
||||
|
||||
===== Exercise 5 =====
|
||||
|
||||
The theory behind this exercise will be covered during the lectures of week 36. It requires reading chapter three of Hastie et al, in particular the derivations preceeding equation (3.49) as the well as the material in the Regression slides the singular value decomposition.
|
||||
The theory behind this exercise will be covered during the lectures of week 36. It requires reading chapter three of Hastie et al, in particular the derivations preceeding equation (3.49) as the well as the material in the Regression slides that deal with the singular value decomposition.
|
||||
|
||||
Using the singular value decomposition, show that the variance of the direction vector
|
||||
$\hat{z}_i=\hat{X}\hat{v}_i=\hat{u}_1d_1$ is equal to (equation (3.49) of Hastie *et al.*)
|
||||
|
||||
Reference in New Issue
Block a user