@@ -212,7 +251,7 @@ MathJax.Hub.Config({
-
@@ -248,13 +287,13 @@ MathJax.Hub.Config({
@@ -148,149 +148,33 @@ MathJax.Hub.Config({
-
-Video of Lecture and handwritten notes - -
-More material will be added here, see handwritten notes also. +
-Before we proceed, we need to rethink what we have been doing. In our -eager to fit the data, we have omitted several important elements in -our regression analysis. In what follows we will - -
- -This will allow us to link the standard linear algebra methods we have discussed above to a statistical interpretation of the methods. -
-Resampling methods are an indispensable tool in modern -statistics. They involve repeatedly drawing samples from a training -set and refitting a model of interest on each sample in order to -obtain additional information about the fitted model. For example, in -order to estimate the variability of a linear regression fit, we can -repeatedly draw different samples from the training data, fit a linear -regression to each new sample, and then examine the extent to which -the resulting fits differ. Such an approach may allow us to obtain -information that would not be available from fitting the model only -once using the original training sample. - -
-Two resampling methods are often used in Machine Learning analyses, - -
- -In addition there are several other methods such as the Jackknife and the Blocking methods. We will discuss in particular -cross-validation and the bootstrap method. - - -
-Resampling approaches can be computationally expensive, because they -involve fitting the same statistical method multiple times using -different subsets of the training data. However, due to recent -advances in computing power, the computational requirements of -resampling methods generally are not prohibitive. In this chapter, we -discuss two of the most commonly used resampling methods, -cross-validation and the bootstrap. Both methods are important tools -in the practical application of many statistical learning -procedures. For example, cross-validation can be used to estimate the -test error associated with a given statistical learning method in -order to evaluate its performance, or to select the appropriate level -of flexibility. The process of evaluating a model’s performance is -known as model assessment, whereas the process of selecting the proper -level of flexibility for a model is known as model selection. The -bootstrap is widely used. - - -
The
@@ -329,7 +213,7 @@ row number \( i \) and perform a sum over all values \( p \).
The assumption we have made here can be summarized as (and this is going to be useful when we discuss the bias-variance trade off)
@@ -353,7 +237,7 @@ $$
We can calculate the expectation value of \( \boldsymbol{y} \) for a given element \( i \)
@@ -392,7 +276,7 @@ mean value \( \boldsymbol{X}\boldsymbol{\beta} \) and variance \( \sigma^2 \) (n
With the OLS expressions for the parameters \( \boldsymbol{\beta} \) we can evaluate the expectation value
@@ -487,7 +371,122 @@ This means the variance we obtain with the standard OLS will always for \( \lamb
+Before we proceed, we need to rethink what we have been doing. In our
+eager to fit the data, we have omitted several important elements in
+our regression analysis. In what follows we will
+
+
+
+This will allow us to link the standard linear algebra methods we have discussed above to a statistical interpretation of the methods.
+
+Resampling methods are an indispensable tool in modern
+statistics. They involve repeatedly drawing samples from a training
+set and refitting a model of interest on each sample in order to
+obtain additional information about the fitted model. For example, in
+order to estimate the variability of a linear regression fit, we can
+repeatedly draw different samples from the training data, fit a linear
+regression to each new sample, and then examine the extent to which
+the resulting fits differ. Such an approach may allow us to obtain
+information that would not be available from fitting the model only
+once using the original training sample.
+
+
+Two resampling methods are often used in Machine Learning analyses,
+
+
+
+In addition there are several other methods such as the Jackknife and the Blocking methods. We will discuss in particular
+cross-validation and the bootstrap method.
+
+
+
+Resampling approaches can be computationally expensive, because they
+involve fitting the same statistical method multiple times using
+different subsets of the training data. However, due to recent
+advances in computing power, the computational requirements of
+resampling methods generally are not prohibitive. In this chapter, we
+discuss two of the most commonly used resampling methods,
+cross-validation and the bootstrap. Both methods are important tools
+in the practical application of many statistical learning
+procedures. For example, cross-validation can be used to estimate the
+test error associated with a given statistical learning method in
+order to evaluate its performance, or to select the appropriate level
+of flexibility. The process of evaluating a model’s performance is
+known as model assessment, whereas the process of selecting the proper
+level of flexibility for a model is known as model selection. The
+bootstrap is widely used.
+
+
+
With all these analytical equations for both the OLS and Ridge
@@ -517,7 +516,7 @@ training error reaches a saturation.
Two famous
@@ -540,7 +539,7 @@ need for bootstrapping.
The Jackknife works by making many replicas of the estimator \( \widehat{\theta} \).
@@ -561,11 +560,11 @@ number \( i \) is left out. Using this notation, define
-
@@ -620,7 +619,7 @@ advantages:
Since \( \widehat{\theta} = \widehat{\theta}(\boldsymbol{X}) \) is a function of random variables,
@@ -636,7 +635,7 @@ estimators.
In the case that \( \widehat{\theta} \) has
@@ -659,7 +658,7 @@ idea is to use the relative frequency of \( \widehat{\theta}^* \)
But
@@ -680,7 +679,7 @@ frequency of the observation \( X_i \), just draw the values
The independent bootstrap works like this:
@@ -706,7 +705,7 @@ example, if you are interested in estimating the variance of \( \widehat
The following code starts with a Gaussian distribution with mean value
@@ -725,7 +724,7 @@ theorem.
-
When the repetitive splitting of the data set is done randomly,
@@ -792,7 +791,7 @@ cross-validation (LOOCV).
For the various values of \( k \)
@@ -848,14 +847,14 @@ For the various values of \( k \)
The code here uses Ridge regression with cross-validation (CV) resampling and \( k \)-fold CV in order to fit a specific polynomial.
-
We will discuss the bias-variance tradeoff in the context of
@@ -1027,11 +1026,11 @@ that is the rewriting in terms of the so-called bias, the variance of the model
-
-
The bias-variance tradeoff summarizes the fundamental tension in
@@ -1182,11 +1181,11 @@ You may also find this recent Another Example from Scikit-Learn's Repository
+
-
-
-
-
-Video of Lecture and handwritten notes
-
-
-More material will be added here, see handwritten notes also.
-
@@ -168,149 +207,26 @@ MathJax.Hub.Config({
-
-Video of Lecture and handwritten notes
-
-
-More material will be added here, see handwritten notes also.
-
-
-
-Before we proceed, we need to rethink what we have been doing. In our
-eager to fit the data, we have omitted several important elements in
-our regression analysis. In what follows we will
-
-
-
-Resampling methods are an indispensable tool in modern
-statistics. They involve repeatedly drawing samples from a training
-set and refitting a model of interest on each sample in order to
-obtain additional information about the fitted model. For example, in
-order to estimate the variability of a linear regression fit, we can
-repeatedly draw different samples from the training data, fit a linear
-regression to each new sample, and then examine the extent to which
-the resulting fits differ. Such an approach may allow us to obtain
-information that would not be available from fitting the model only
-once using the original training sample.
-
-
-Two resampling methods are often used in Machine Learning analyses,
-
-
-
-
-
-Resampling approaches can be computationally expensive, because they
-involve fitting the same statistical method multiple times using
-different subsets of the training data. However, due to recent
-advances in computing power, the computational requirements of
-resampling methods generally are not prohibitive. In this chapter, we
-discuss two of the most commonly used resampling methods,
-cross-validation and the bootstrap. Both methods are important tools
-in the practical application of many statistical learning
-procedures. For example, cross-validation can be used to estimate the
-test error associated with a given statistical learning method in
-order to evaluate its performance, or to select the appropriate level
-of flexibility. The process of evaluating a model’s performance is
-known as model assessment, whereas the process of selecting the proper
-level of flexibility for a model is known as model selection. The
-bootstrap is widely used.
-
-
-
-
-
-
-
-
-
-
The
@@ -347,7 +263,7 @@ row number \( i \) and perform a sum over all values \( p \).
The assumption we have made here can be summarized as (and this is going to be useful when we discuss the bias-variance trade off)
@@ -367,7 +283,7 @@ $$
We can calculate the expectation value of \( \boldsymbol{y} \) for a given element \( i \)
@@ -402,7 +318,7 @@ mean value \( \boldsymbol{X}\boldsymbol{\beta} \) and variance \( \sigma^2 \) (n
With the OLS expressions for the parameters \( \boldsymbol{\beta} \) we can evaluate the expectation value
@@ -487,7 +403,129 @@ This means the variance we obtain with the standard OLS will always for \( \lamb
+
+Before we proceed, we need to rethink what we have been doing. In our
+eager to fit the data, we have omitted several important elements in
+our regression analysis. In what follows we will
+
+
+
+Resampling methods are an indispensable tool in modern
+statistics. They involve repeatedly drawing samples from a training
+set and refitting a model of interest on each sample in order to
+obtain additional information about the fitted model. For example, in
+order to estimate the variability of a linear regression fit, we can
+repeatedly draw different samples from the training data, fit a linear
+regression to each new sample, and then examine the extent to which
+the resulting fits differ. Such an approach may allow us to obtain
+information that would not be available from fitting the model only
+once using the original training sample.
+
+
+Two resampling methods are often used in Machine Learning analyses,
+
+
+
+
+
+Resampling approaches can be computationally expensive, because they
+involve fitting the same statistical method multiple times using
+different subsets of the training data. However, due to recent
+advances in computing power, the computational requirements of
+resampling methods generally are not prohibitive. In this chapter, we
+discuss two of the most commonly used resampling methods,
+cross-validation and the bootstrap. Both methods are important tools
+in the practical application of many statistical learning
+procedures. For example, cross-validation can be used to estimate the
+test error associated with a given statistical learning method in
+order to evaluate its performance, or to select the appropriate level
+of flexibility. The process of evaluating a model’s performance is
+known as model assessment, whereas the process of selecting the proper
+level of flexibility for a model is known as model selection. The
+bootstrap is widely used.
+
+
+
+
+
+
+
+
+
+
With all these analytical equations for both the OLS and Ridge
@@ -516,7 +554,7 @@ training error reaches a saturation.
Two famous
@@ -539,7 +577,7 @@ need for bootstrapping.
The Jackknife works by making many replicas of the estimator \( \widehat{\theta} \).
@@ -558,11 +596,11 @@ number \( i \) is left out. Using this notation, define
-
@@ -615,7 +653,7 @@ advantages:
Since \( \widehat{\theta} = \widehat{\theta}(\boldsymbol{X}) \) is a function of random variables,
@@ -631,7 +669,7 @@ estimators.
In the case that \( \widehat{\theta} \) has
@@ -653,7 +691,7 @@ idea is to use the relative frequency of \( \widehat{\theta}^* \)
But
@@ -674,7 +712,7 @@ frequency of the observation \( X_i \), just draw the values
The independent bootstrap works like this:
@@ -699,7 +737,7 @@ example, if you are interested in estimating the variance of \( \widehat
The following code starts with a Gaussian distribution with mean value
@@ -718,7 +756,7 @@ theorem.
-
-
When the repetitive splitting of the data set is done randomly,
@@ -784,7 +822,7 @@ cross-validation (LOOCV).
-
For the various values of \( k \)
@@ -838,14 +876,14 @@ For the various values of \( k \)
The code here uses Ridge regression with cross-validation (CV) resampling and \( k \)-fold CV in order to fit a specific polynomial.
-
We will discuss the bias-variance tradeoff in the context of
@@ -1004,11 +1042,11 @@ that is the rewriting in terms of the so-called bias, the variance of the model
-
-
-
The bias-variance tradeoff summarizes the fundamental tension in
@@ -1157,11 +1195,11 @@ You may also find this recent Another Example from Scikit-Learn's Repository
+
-
-
-
-
-
-
-Video of Lecture and handwritten notes
-
-
-More material will be added here, see handwritten notes also.
-
@@ -173,149 +212,26 @@ MathJax.Hub.Config({
-
-Video of Lecture and handwritten notes
-
-
-More material will be added here, see handwritten notes also.
-
-
-
-Before we proceed, we need to rethink what we have been doing. In our
-eager to fit the data, we have omitted several important elements in
-our regression analysis. In what follows we will
-
-
-
-Resampling methods are an indispensable tool in modern
-statistics. They involve repeatedly drawing samples from a training
-set and refitting a model of interest on each sample in order to
-obtain additional information about the fitted model. For example, in
-order to estimate the variability of a linear regression fit, we can
-repeatedly draw different samples from the training data, fit a linear
-regression to each new sample, and then examine the extent to which
-the resulting fits differ. Such an approach may allow us to obtain
-information that would not be available from fitting the model only
-once using the original training sample.
-
-
-Two resampling methods are often used in Machine Learning analyses,
-
-
-
-
-
-Resampling approaches can be computationally expensive, because they
-involve fitting the same statistical method multiple times using
-different subsets of the training data. However, due to recent
-advances in computing power, the computational requirements of
-resampling methods generally are not prohibitive. In this chapter, we
-discuss two of the most commonly used resampling methods,
-cross-validation and the bootstrap. Both methods are important tools
-in the practical application of many statistical learning
-procedures. For example, cross-validation can be used to estimate the
-test error associated with a given statistical learning method in
-order to evaluate its performance, or to select the appropriate level
-of flexibility. The process of evaluating a model’s performance is
-known as model assessment, whereas the process of selecting the proper
-level of flexibility for a model is known as model selection. The
-bootstrap is widely used.
-
-
-
-
-
-
-
-
-
-
The
@@ -352,7 +268,7 @@ row number \( i \) and perform a sum over all values \( p \).
The assumption we have made here can be summarized as (and this is going to be useful when we discuss the bias-variance trade off)
@@ -372,7 +288,7 @@ $$
We can calculate the expectation value of \( \boldsymbol{y} \) for a given element \( i \)
@@ -407,7 +323,7 @@ mean value \( \boldsymbol{X}\boldsymbol{\beta} \) and variance \( \sigma^2 \) (n
With the OLS expressions for the parameters \( \boldsymbol{\beta} \) we can evaluate the expectation value
@@ -492,7 +408,129 @@ This means the variance we obtain with the standard OLS will always for \( \lamb
+
+Before we proceed, we need to rethink what we have been doing. In our
+eager to fit the data, we have omitted several important elements in
+our regression analysis. In what follows we will
+
+
+
+Resampling methods are an indispensable tool in modern
+statistics. They involve repeatedly drawing samples from a training
+set and refitting a model of interest on each sample in order to
+obtain additional information about the fitted model. For example, in
+order to estimate the variability of a linear regression fit, we can
+repeatedly draw different samples from the training data, fit a linear
+regression to each new sample, and then examine the extent to which
+the resulting fits differ. Such an approach may allow us to obtain
+information that would not be available from fitting the model only
+once using the original training sample.
+
+
+Two resampling methods are often used in Machine Learning analyses,
+
+
+
+
+
+Resampling approaches can be computationally expensive, because they
+involve fitting the same statistical method multiple times using
+different subsets of the training data. However, due to recent
+advances in computing power, the computational requirements of
+resampling methods generally are not prohibitive. In this chapter, we
+discuss two of the most commonly used resampling methods,
+cross-validation and the bootstrap. Both methods are important tools
+in the practical application of many statistical learning
+procedures. For example, cross-validation can be used to estimate the
+test error associated with a given statistical learning method in
+order to evaluate its performance, or to select the appropriate level
+of flexibility. The process of evaluating a model’s performance is
+known as model assessment, whereas the process of selecting the proper
+level of flexibility for a model is known as model selection. The
+bootstrap is widely used.
+
+
+
+
+
+
+
+
+
+
With all these analytical equations for both the OLS and Ridge
@@ -521,7 +559,7 @@ training error reaches a saturation.
Two famous
@@ -544,7 +582,7 @@ need for bootstrapping.
The Jackknife works by making many replicas of the estimator \( \widehat{\theta} \).
@@ -563,11 +601,11 @@ number \( i \) is left out. Using this notation, define
-
@@ -620,7 +658,7 @@ advantages:
Since \( \widehat{\theta} = \widehat{\theta}(\boldsymbol{X}) \) is a function of random variables,
@@ -636,7 +674,7 @@ estimators.
In the case that \( \widehat{\theta} \) has
@@ -658,7 +696,7 @@ idea is to use the relative frequency of \( \widehat{\theta}^* \)
But
@@ -679,7 +717,7 @@ frequency of the observation \( X_i \), just draw the values
The independent bootstrap works like this:
@@ -704,7 +742,7 @@ example, if you are interested in estimating the variance of \( \widehat
The following code starts with a Gaussian distribution with mean value
@@ -723,7 +761,7 @@ theorem.
-
-
When the repetitive splitting of the data set is done randomly,
@@ -789,7 +827,7 @@ cross-validation (LOOCV).
-
For the various values of \( k \)
@@ -843,14 +881,14 @@ For the various values of \( k \)
The code here uses Ridge regression with cross-validation (CV) resampling and \( k \)-fold CV in order to fit a specific polynomial.
-
We will discuss the bias-variance tradeoff in the context of
@@ -1009,11 +1047,11 @@ that is the rewriting in terms of the so-called bias, the variance of the model
-
-
-
The bias-variance tradeoff summarizes the fundamental tension in
@@ -1162,11 +1200,11 @@ You may also find this recent Another Example from Scikit-Learn's Repository
+
-
-
-
-
-
-
-Video of Lecture and handwritten notes
-
-
-More material will be added here, see handwritten notes also.
-
Assumptions made
+Assumptions made
Expectation value and variance
+Expectation value and variance
Expectation value and variance for \( \boldsymbol{\beta} \)
+Expectation value and variance for \( \boldsymbol{\beta} \)
Resampling methods
+Friday September 10
+Why resampling methods
+
+
+
+Resampling methods
+
+
+Resampling approaches can be computationally expensive
+Why resampling methods ?
+
+
+Statistical analysis
+
+
+
+
+
+Resampling methods
Resampling methods: Jackknife and Bootstrap
+Resampling methods: Jackknife and Bootstrap
Resampling methods: Jackknife
+Resampling methods: Jackknife
Jackknife code example
+Jackknife code example
from numpy import *
+
from numpy import *
from numpy.random import randint, randn
from time import time
@@ -576,9 +575,9 @@ number \( i \) is left out. Using this notation, define
t[i] = stat(delete(data,i) )
# analysis
- print("Runtime: %g sec" % (time()-t0)); print("Jackknife Statistics :")
- print("original bias std. error")
- print("%8g %14g %15g" % (stat(data),(n-1)*mean(t)/n, (n*var(t))**.5))
+ print("Runtime: %g sec" % (time()-t0)); print("Jackknife Statistics :")
+ print("original bias std. error")
+ print("%8g %14g %15g" % (stat(data),(n-1)*mean(t)/n, (n*var(t))**.5))
return t
@@ -598,7 +597,7 @@ t = jackknife(x, stat)
Resampling methods: Bootstrap
+Resampling methods: Bootstrap
Resampling methods: Bootstrap background
+Resampling methods: Bootstrap background
Resampling methods: More Bootstrap background
+Resampling methods: More Bootstrap background
Resampling methods: Bootstrap approach
+Resampling methods: Bootstrap approach
Resampling methods: Bootstrap steps
+Resampling methods: Bootstrap steps
Code example for the Bootstrap method
+Code example for the Bootstrap method
from numpy import *
+
from numpy import *
from numpy.random import randint, randn
from time import time
import matplotlib.mlab as mlab
@@ -743,9 +742,9 @@ theorem.
t[i] = statistic(data[randint(0,n,n)])
# analysis
- print("Runtime: %g sec" % (time()-t0)); print("Bootstrap Statistics :")
- print("original bias std. error")
- print("%8g %8g %14g %15g" % (statistic(data), std(data),mean(t),std(t)))
+ print("Runtime: %g sec" % (time()-t0)); print("Bootstrap Statistics :")
+ print("original bias std. error")
+ print("%8g %8g %14g %15g" % (statistic(data), std(data),mean(t),std(t)))
return t
@@ -763,7 +762,7 @@ lt = plt.plot(binsboot, y, 'r--', li
plt.xlabel('Smarts')
plt.ylabel('Probability')
plt.axis([99.5, 100.6, 0, 3.0])
-plt.grid(True)
+plt.grid(True)
plt.show()
Various steps in cross-validation
+Various steps in cross-validation
How to set up the cross-validation for Ridge and/or Lasso
+How to set up the cross-validation for Ridge and/or Lasso
Cross-validation in brief
+Cross-validation in brief
Code Example for Cross-validation and \( k \)-fold Cross-validation
+Code Example for Cross-validation and \( k \)-fold Cross-validation
import numpy as np
+
import numpy as np
import matplotlib.pyplot as plt
from sklearn.model_selection import KFold
from sklearn.linear_model import Ridge
@@ -949,7 +948,7 @@ plt.show()
The bias-variance tradeoff
+The bias-variance tradeoff
Example code for Bias-Variance tradeoff
+Example code for Bias-Variance tradeoff
import matplotlib.pyplot as plt
+
import matplotlib.pyplot as plt
import numpy as np
from sklearn.linear_model import LinearRegression, Ridge, Lasso
from sklearn.preprocessing import PolynomialFeatures
@@ -1055,7 +1054,7 @@ x_train, x_test, y_train, y_test = train_test_split(x, y, test_size=# Combine x transformation and model into one operation.
# Not neccesary, but convenient.
-model = make_pipeline(PolynomialFeatures(degree=degree), LinearRegression(fit_intercept=False))
+model = make_pipeline(PolynomialFeatures(degree=degree), LinearRegression(fit_intercept=False))
# The following (m x n_bootstraps) matrix holds the column vectors y_pred
# for each bootstrap iteration.
@@ -1072,13 +1071,13 @@ y_pred = np.empty((y_test.shape[0], n_boostr
# calculated per data point in the test set.
# Note 2: The use of keepdims=True is important in the calculation of bias as this
# maintains the column vector form. Dropping this yields very unexpected results.
-error = np.mean( np.mean((y_test - y_pred)**2, axis=1, keepdims=True) )
-bias = np.mean( (y_test - np.mean(y_pred, axis=1, keepdims=True))**2 )
-variance = np.mean( np.var(y_pred, axis=1, keepdims=True) )
-print('Error:', error)
-print('Bias^2:', bias)
-print('Var:', variance)
-print('{} >= {} + {} = {}'.format(error, bias, variance, bias+variance))
+error = np.mean( np.mean((y_test - y_pred)**2, axis=1, keepdims=True) )
+bias = np.mean( (y_test - np.mean(y_pred, axis=1, keepdims=True))**2 )
+variance = np.mean( np.var(y_pred, axis=1, keepdims=True) )
+print('Error:', error)
+print('Bias^2:', bias)
+print('Var:', variance)
+print('{} >= {} + {} = {}'.format(error, bias, variance, bias+variance))
plt.plot(x[::5, :], y[::5, :], label='f(x)')
plt.scatter(x_test, y_test, label='Data points')
@@ -1090,11 +1089,11 @@ plt.show()
Understanding what happens
+Understanding what happens
import matplotlib.pyplot as plt
+
import matplotlib.pyplot as plt
import numpy as np
from sklearn.linear_model import LinearRegression, Ridge, Lasso
from sklearn.preprocessing import PolynomialFeatures
@@ -1119,21 +1118,21 @@ polydegree = np.zeros(maxdegree)
x_train, x_test, y_train, y_test = train_test_split(x, y, test_size=0.2)
for degree in range(maxdegree):
- model = make_pipeline(PolynomialFeatures(degree=degree), LinearRegression(fit_intercept=False))
+ model = make_pipeline(PolynomialFeatures(degree=degree), LinearRegression(fit_intercept=False))
y_pred = np.empty((y_test.shape[0], n_boostraps))
for i in range(n_boostraps):
x_, y_ = resample(x_train, y_train)
y_pred[:, i] = model.fit(x_, y_).predict(x_test).ravel()
polydegree[degree] = degree
- error[degree] = np.mean( np.mean((y_test - y_pred)**2, axis=1, keepdims=True) )
- bias[degree] = np.mean( (y_test - np.mean(y_pred, axis=1, keepdims=True))**2 )
- variance[degree] = np.mean( np.var(y_pred, axis=1, keepdims=True) )
- print('Polynomial degree:', degree)
- print('Error:', error[degree])
- print('Bias^2:', bias[degree])
- print('Var:', variance[degree])
- print('{} >= {} + {} = {}'.format(error[degree], bias[degree], variance[degree], bias[degree]+variance[degree]))
+ error[degree] = np.mean( np.mean((y_test - y_pred)**2, axis=1, keepdims=True) )
+ bias[degree] = np.mean( (y_test - np.mean(y_pred, axis=1, keepdims=True))**2 )
+ variance[degree] = np.mean( np.var(y_pred, axis=1, keepdims=True) )
+ print('Polynomial degree:', degree)
+ print('Error:', error[degree])
+ print('Bias^2:', bias[degree])
+ print('Var:', variance[degree])
+ print('{} >= {} + {} = {}'.format(error[degree], bias[degree], variance[degree], bias[degree]+variance[degree]))
plt.plot(polydegree, error, label='Error')
plt.plot(polydegree, bias, label='bias')
@@ -1145,7 +1144,7 @@ plt.show()
Summing up
+Summing up
Another Example from Scikit-Learn's Repository
"""
+
"""
============================
Underfitting vs. Overfitting
============================
@@ -1208,7 +1207,7 @@ You may also find this recent training data.
"""
-print(__doc__)
+print(__doc__)
import numpy as np
import matplotlib.pyplot as plt
@@ -1235,7 +1234,7 @@ plt.figure(figsize=(14, False)
+ include_bias=False)
linear_regression = LinearRegression()
pipeline = Pipeline([("polynomial_features", polynomial_features),
("linear_regression", linear_regression)])
@@ -1262,12 +1261,12 @@ plt.show()
More examples on bootstrap and cross-validation and errors
+More examples on bootstrap and cross-validation and errors
# Common imports
+
# Common imports
import os
import numpy as np
import pandas as pd
@@ -1297,7 +1296,7 @@ DATA_ID = "DataFiles/"
return os.path.join(DATA_ID, dat_id)
def save_fig(fig_id):
- plt.savefig(image_path(fig_id) + ".png", format='png')
+ plt.savefig(image_path(fig_id) + ".png", format='png')
infile = open(data_path("EoS.csv"),'r')
@@ -1327,7 +1326,7 @@ trials = 100
trainingerror[polydegree] = 0.0
for samples in range(trials):
x_train, x_test, y_train, y_test = train_test_split(X, Energies, test_size=0.2)
- model = LinearRegression(fit_intercept=True).fit(x_train, y_train)
+ model = LinearRegression(fit_intercept=True).fit(x_train, y_train)
ypred = model.predict(x_train)
ytilde = model.predict(x_test)
testerror[polydegree] += mean_squared_error(y_test, ytilde)
@@ -1335,9 +1334,9 @@ trials = 100
testerror[polydegree] /= trials
trainingerror[polydegree] /= trials
- print("Degree of polynomial: %3d"% polynomial[polydegree])
- print("Mean squared error on training data: %.8f" % trainingerror[polydegree])
- print("Mean squared error on test data: %.8f" % testerror[polydegree])
+ print("Degree of polynomial: %3d"% polynomial[polydegree])
+ print("Mean squared error on training data: %.8f" % trainingerror[polydegree])
+ print("Mean squared error on test data: %.8f" % testerror[polydegree])
plt.plot(polynomial, np.log10(trainingerror), label='Training Error')
plt.plot(polynomial, np.log10(testerror), label='Test Error')
@@ -1350,12 +1349,12 @@ plt.show()
The same example but now with cross-validation
+The same example but now with cross-validation
# Common imports
+
# Common imports
import os
import numpy as np
import pandas as pd
@@ -1387,7 +1386,7 @@ DATA_ID = "DataFiles/"
return os.path.join(DATA_ID, dat_id)
def save_fig(fig_id):
- plt.savefig(image_path(fig_id) + ".png", format='png')
+ plt.savefig(image_path(fig_id) + ".png", format='png')
infile = open(data_path("EoS.csv"),'r')
@@ -1427,11 +1426,11 @@ plt.show()
Cross-validation with Ridge
+Cross-validation with Ridge
import numpy as np
+
import numpy as np
import matplotlib.pyplot as plt
from sklearn.model_selection import KFold
from sklearn.linear_model import Ridge
@@ -1470,17 +1469,6 @@ plt.show()
-
Friday September 4
-
-Week 36: Resampling techniques and Ordinary Least Square
Week 36: Statistical interpretation of Linear Regression and Resampling techniques
Sep 16, 2020
Sep 6, 2021
-Plans for week 36
+Plans for week 36
-
-Thursday September 3
-
-
-
-Why resampling methods
-
-
-
-
-This will allow us to link the standard linear algebra methods we have discussed above to a statistical interpretation of the methods.
-
-
-
-Resampling methods
-
-
-
-In addition there are several other methods such as the Jackknife and the Blocking methods. We will discuss in particular
-cross-validation and the bootstrap method.
-
-
-
-
-Resampling approaches can be computationally expensive
-
-
-Why resampling methods ?
-
-
-
-
-Statistical analysis
-
-
-
-
-
-Thursday September 9
Linking the regression analysis with a statistical interpretation
+Linking the regression analysis with a statistical interpretation
-Assumptions made
+Assumptions made
-Expectation value and variance
+Expectation value and variance
-Expectation value and variance for \( \boldsymbol{\beta} \)
+Expectation value and variance for \( \boldsymbol{\beta} \)
-Resampling methods
+Friday September 10
+
+
+
+Why resampling methods
+
+
+
+
+This will allow us to link the standard linear algebra methods we have discussed above to a statistical interpretation of the methods.
+
+
+
+Resampling methods
+
+
+
+In addition there are several other methods such as the Jackknife and the Blocking methods. We will discuss in particular
+cross-validation and the bootstrap method.
+
+
+
+
+Resampling approaches can be computationally expensive
+
+
+Why resampling methods ?
+
+
+
+
+Statistical analysis
+
+
+
+
+
+
+
+Resampling methods
-Resampling methods: Jackknife and Bootstrap
+Resampling methods: Jackknife and Bootstrap
-Resampling methods: Jackknife
+Resampling methods: Jackknife
-Jackknife code example
+Jackknife code example
from numpy import *
+
from numpy import *
from numpy.random import randint, randn
from time import time
@@ -573,9 +611,9 @@ number \( i \) is left out. Using this notation, define
t[i] = stat(delete(data,i) )
# analysis
- print("Runtime: %g sec" % (time()-t0)); print("Jackknife Statistics :")
- print("original bias std. error")
- print("%8g %14g %15g" % (stat(data),(n-1)*mean(t)/n, (n*var(t))**.5))
+ print("Runtime: %g sec" % (time()-t0)); print("Jackknife Statistics :")
+ print("original bias std. error")
+ print("%8g %14g %15g" % (stat(data),(n-1)*mean(t)/n, (n*var(t))**.5))
return t
@@ -594,7 +632,7 @@ t = jackknife(x, stat)
-Resampling methods: Bootstrap
+Resampling methods: Bootstrap
-Resampling methods: Bootstrap background
+Resampling methods: Bootstrap background
-Resampling methods: More Bootstrap background
+Resampling methods: More Bootstrap background
-Resampling methods: Bootstrap approach
+Resampling methods: Bootstrap approach
-Resampling methods: Bootstrap steps
+Resampling methods: Bootstrap steps
-Code example for the Bootstrap method
+Code example for the Bootstrap method
from numpy import *
+
from numpy import *
from numpy.random import randint, randn
from time import time
import matplotlib.mlab as mlab
@@ -736,9 +774,9 @@ theorem.
t[i] = statistic(data[randint(0,n,n)])
# analysis
- print("Runtime: %g sec" % (time()-t0)); print("Bootstrap Statistics :")
- print("original bias std. error")
- print("%8g %8g %14g %15g" % (statistic(data), std(data),mean(t),std(t)))
+ print("Runtime: %g sec" % (time()-t0)); print("Bootstrap Statistics :")
+ print("original bias std. error")
+ print("%8g %8g %14g %15g" % (statistic(data), std(data),mean(t),std(t)))
return t
@@ -756,14 +794,14 @@ lt = plt.plot(binsboot, y, 'r--', li
plt.xlabel('Smarts')
plt.ylabel('Probability')
plt.axis([99.5, 100.6, 0, 3.0])
-plt.grid(True)
+plt.grid(True)
plt.show()
Various steps in cross-validation
+Various steps in cross-validation
How to set up the cross-validation for Ridge and/or Lasso
+How to set up the cross-validation for Ridge and/or Lasso
-Cross-validation in brief
+Cross-validation in brief
-Code Example for Cross-validation and \( k \)-fold Cross-validation
+Code Example for Cross-validation and \( k \)-fold Cross-validation
import numpy as np
+
import numpy as np
import matplotlib.pyplot as plt
from sklearn.model_selection import KFold
from sklearn.linear_model import Ridge
@@ -938,7 +976,7 @@ plt.show()
-The bias-variance tradeoff
+The bias-variance tradeoff
-Example code for Bias-Variance tradeoff
+Example code for Bias-Variance tradeoff
import matplotlib.pyplot as plt
+
import matplotlib.pyplot as plt
import numpy as np
from sklearn.linear_model import LinearRegression, Ridge, Lasso
from sklearn.preprocessing import PolynomialFeatures
@@ -1032,7 +1070,7 @@ x_train, x_test, y_train, y_test = train_test_split(x, y, test_size=# Combine x transformation and model into one operation.
# Not neccesary, but convenient.
-model = make_pipeline(PolynomialFeatures(degree=degree), LinearRegression(fit_intercept=False))
+model = make_pipeline(PolynomialFeatures(degree=degree), LinearRegression(fit_intercept=False))
# The following (m x n_bootstraps) matrix holds the column vectors y_pred
# for each bootstrap iteration.
@@ -1049,13 +1087,13 @@ y_pred = np.empty((y_test.shape[0], n_boostr
# calculated per data point in the test set.
# Note 2: The use of keepdims=True is important in the calculation of bias as this
# maintains the column vector form. Dropping this yields very unexpected results.
-error = np.mean( np.mean((y_test - y_pred)**2, axis=1, keepdims=True) )
-bias = np.mean( (y_test - np.mean(y_pred, axis=1, keepdims=True))**2 )
-variance = np.mean( np.var(y_pred, axis=1, keepdims=True) )
-print('Error:', error)
-print('Bias^2:', bias)
-print('Var:', variance)
-print('{} >= {} + {} = {}'.format(error, bias, variance, bias+variance))
+error = np.mean( np.mean((y_test - y_pred)**2, axis=1, keepdims=True) )
+bias = np.mean( (y_test - np.mean(y_pred, axis=1, keepdims=True))**2 )
+variance = np.mean( np.var(y_pred, axis=1, keepdims=True) )
+print('Error:', error)
+print('Bias^2:', bias)
+print('Var:', variance)
+print('{} >= {} + {} = {}'.format(error, bias, variance, bias+variance))
plt.plot(x[::5, :], y[::5, :], label='f(x)')
plt.scatter(x_test, y_test, label='Data points')
@@ -1066,11 +1104,11 @@ plt.show()
-Understanding what happens
+Understanding what happens
import matplotlib.pyplot as plt
+
import matplotlib.pyplot as plt
import numpy as np
from sklearn.linear_model import LinearRegression, Ridge, Lasso
from sklearn.preprocessing import PolynomialFeatures
@@ -1095,21 +1133,21 @@ polydegree = np.zeros(maxdegree)
x_train, x_test, y_train, y_test = train_test_split(x, y, test_size=0.2)
for degree in range(maxdegree):
- model = make_pipeline(PolynomialFeatures(degree=degree), LinearRegression(fit_intercept=False))
+ model = make_pipeline(PolynomialFeatures(degree=degree), LinearRegression(fit_intercept=False))
y_pred = np.empty((y_test.shape[0], n_boostraps))
for i in range(n_boostraps):
x_, y_ = resample(x_train, y_train)
y_pred[:, i] = model.fit(x_, y_).predict(x_test).ravel()
polydegree[degree] = degree
- error[degree] = np.mean( np.mean((y_test - y_pred)**2, axis=1, keepdims=True) )
- bias[degree] = np.mean( (y_test - np.mean(y_pred, axis=1, keepdims=True))**2 )
- variance[degree] = np.mean( np.var(y_pred, axis=1, keepdims=True) )
- print('Polynomial degree:', degree)
- print('Error:', error[degree])
- print('Bias^2:', bias[degree])
- print('Var:', variance[degree])
- print('{} >= {} + {} = {}'.format(error[degree], bias[degree], variance[degree], bias[degree]+variance[degree]))
+ error[degree] = np.mean( np.mean((y_test - y_pred)**2, axis=1, keepdims=True) )
+ bias[degree] = np.mean( (y_test - np.mean(y_pred, axis=1, keepdims=True))**2 )
+ variance[degree] = np.mean( np.var(y_pred, axis=1, keepdims=True) )
+ print('Polynomial degree:', degree)
+ print('Error:', error[degree])
+ print('Bias^2:', bias[degree])
+ print('Var:', variance[degree])
+ print('{} >= {} + {} = {}'.format(error[degree], bias[degree], variance[degree], bias[degree]+variance[degree]))
plt.plot(polydegree, error, label='Error')
plt.plot(polydegree, bias, label='bias')
@@ -1120,7 +1158,7 @@ plt.show()
Summing up
+Summing up
Another Example from Scikit-Learn's Repository
"""
+
"""
============================
Underfitting vs. Overfitting
============================
@@ -1183,7 +1221,7 @@ You may also find this recent training data.
"""
-print(__doc__)
+print(__doc__)
import numpy as np
import matplotlib.pyplot as plt
@@ -1210,7 +1248,7 @@ plt.figure(figsize=(14, False)
+ include_bias=False)
linear_regression = LinearRegression()
pipeline = Pipeline([("polynomial_features", polynomial_features),
("linear_regression", linear_regression)])
@@ -1236,12 +1274,12 @@ plt.show()
-More examples on bootstrap and cross-validation and errors
+More examples on bootstrap and cross-validation and errors
# Common imports
+
# Common imports
import os
import numpy as np
import pandas as pd
@@ -1271,7 +1309,7 @@ DATA_ID = "DataFiles/"
return os.path.join(DATA_ID, dat_id)
def save_fig(fig_id):
- plt.savefig(image_path(fig_id) + ".png", format='png')
+ plt.savefig(image_path(fig_id) + ".png", format='png')
infile = open(data_path("EoS.csv"),'r')
@@ -1301,7 +1339,7 @@ trials = 100
trainingerror[polydegree] = 0.0
for samples in range(trials):
x_train, x_test, y_train, y_test = train_test_split(X, Energies, test_size=0.2)
- model = LinearRegression(fit_intercept=True).fit(x_train, y_train)
+ model = LinearRegression(fit_intercept=True).fit(x_train, y_train)
ypred = model.predict(x_train)
ytilde = model.predict(x_test)
testerror[polydegree] += mean_squared_error(y_test, ytilde)
@@ -1309,9 +1347,9 @@ trials = 100
testerror[polydegree] /= trials
trainingerror[polydegree] /= trials
- print("Degree of polynomial: %3d"% polynomial[polydegree])
- print("Mean squared error on training data: %.8f" % trainingerror[polydegree])
- print("Mean squared error on test data: %.8f" % testerror[polydegree])
+ print("Degree of polynomial: %3d"% polynomial[polydegree])
+ print("Mean squared error on training data: %.8f" % trainingerror[polydegree])
+ print("Mean squared error on test data: %.8f" % testerror[polydegree])
plt.plot(polynomial, np.log10(trainingerror), label='Training Error')
plt.plot(polynomial, np.log10(testerror), label='Test Error')
@@ -1323,12 +1361,12 @@ plt.show()
The same example but now with cross-validation
+The same example but now with cross-validation
# Common imports
+
# Common imports
import os
import numpy as np
import pandas as pd
@@ -1360,7 +1398,7 @@ DATA_ID = "DataFiles/"
return os.path.join(DATA_ID, dat_id)
def save_fig(fig_id):
- plt.savefig(image_path(fig_id) + ".png", format='png')
+ plt.savefig(image_path(fig_id) + ".png", format='png')
infile = open(data_path("EoS.csv"),'r')
@@ -1399,11 +1437,11 @@ plt.show()
-Cross-validation with Ridge
+Cross-validation with Ridge
import numpy as np
+
import numpy as np
import matplotlib.pyplot as plt
from sklearn.model_selection import KFold
from sklearn.linear_model import Ridge
@@ -1439,24 +1477,13 @@ plt.ylabel('MSE')
plt.legend()
plt.show()
-
-Friday September 4
-
-Week 36: Resampling techniques and Ordinary Least Square
Week 36: Statistical interpretation of Linear Regression and Resampling techniques
Sep 16, 2020
Sep 6, 2021
-Plans for week 36
+Plans for week 36
-
-Thursday September 3
-
-
-
-Why resampling methods
-
-
-
-
-This will allow us to link the standard linear algebra methods we have discussed above to a statistical interpretation of the methods.
-
-
-
-Resampling methods
-
-
-
-In addition there are several other methods such as the Jackknife and the Blocking methods. We will discuss in particular
-cross-validation and the bootstrap method.
-
-
-
-
-Resampling approaches can be computationally expensive
-
-
-Why resampling methods ?
-
-
-
-
-Statistical analysis
-
-
-
-
-
-Thursday September 9
Linking the regression analysis with a statistical interpretation
+Linking the regression analysis with a statistical interpretation
-Assumptions made
+Assumptions made
-Expectation value and variance
+Expectation value and variance
-Expectation value and variance for \( \boldsymbol{\beta} \)
+Expectation value and variance for \( \boldsymbol{\beta} \)
-Resampling methods
+Friday September 10
+
+
+
+Why resampling methods
+
+
+
+
+This will allow us to link the standard linear algebra methods we have discussed above to a statistical interpretation of the methods.
+
+
+
+Resampling methods
+
+
+
+In addition there are several other methods such as the Jackknife and the Blocking methods. We will discuss in particular
+cross-validation and the bootstrap method.
+
+
+
+
+Resampling approaches can be computationally expensive
+
+
+Why resampling methods ?
+
+
+
+
+Statistical analysis
+
+
+
+
+
+
+
+Resampling methods
-Resampling methods: Jackknife and Bootstrap
+Resampling methods: Jackknife and Bootstrap
-Resampling methods: Jackknife
+Resampling methods: Jackknife
-Jackknife code example
+Jackknife code example
from numpy import *
+
from numpy import *
from numpy.random import randint, randn
from time import time
@@ -578,9 +616,9 @@ number \( i \) is left out. Using this notation, define
t[i] = stat(delete(data,i) )
# analysis
- print("Runtime: %g sec" % (time()-t0)); print("Jackknife Statistics :")
- print("original bias std. error")
- print("%8g %14g %15g" % (stat(data),(n-1)*mean(t)/n, (n*var(t))**.5))
+ print("Runtime: %g sec" % (time()-t0)); print("Jackknife Statistics :")
+ print("original bias std. error")
+ print("%8g %14g %15g" % (stat(data),(n-1)*mean(t)/n, (n*var(t))**.5))
return t
@@ -599,7 +637,7 @@ t = jackknife(x, stat)
-Resampling methods: Bootstrap
+Resampling methods: Bootstrap
-Resampling methods: Bootstrap background
+Resampling methods: Bootstrap background
-Resampling methods: More Bootstrap background
+Resampling methods: More Bootstrap background
-Resampling methods: Bootstrap approach
+Resampling methods: Bootstrap approach
-Resampling methods: Bootstrap steps
+Resampling methods: Bootstrap steps
-Code example for the Bootstrap method
+Code example for the Bootstrap method
from numpy import *
+
from numpy import *
from numpy.random import randint, randn
from time import time
import matplotlib.mlab as mlab
@@ -741,9 +779,9 @@ theorem.
t[i] = statistic(data[randint(0,n,n)])
# analysis
- print("Runtime: %g sec" % (time()-t0)); print("Bootstrap Statistics :")
- print("original bias std. error")
- print("%8g %8g %14g %15g" % (statistic(data), std(data),mean(t),std(t)))
+ print("Runtime: %g sec" % (time()-t0)); print("Bootstrap Statistics :")
+ print("original bias std. error")
+ print("%8g %8g %14g %15g" % (statistic(data), std(data),mean(t),std(t)))
return t
@@ -761,14 +799,14 @@ lt = plt..xlabel('Smarts')
plt.ylabel('Probability')
plt.axis([99.5, 100.6, 0, 3.0])
-plt.grid(True)
+plt.grid(True)
plt.show()
Various steps in cross-validation
+Various steps in cross-validation
How to set up the cross-validation for Ridge and/or Lasso
+How to set up the cross-validation for Ridge and/or Lasso
-Cross-validation in brief
+Cross-validation in brief
-Code Example for Cross-validation and \( k \)-fold Cross-validation
+Code Example for Cross-validation and \( k \)-fold Cross-validation
import numpy as np
+
import numpy as np
import matplotlib.pyplot as plt
from sklearn.model_selection import KFold
from sklearn.linear_model import Ridge
@@ -943,7 +981,7 @@ plt.show()
-The bias-variance tradeoff
+The bias-variance tradeoff
-Example code for Bias-Variance tradeoff
+Example code for Bias-Variance tradeoff
import matplotlib.pyplot as plt
+
import matplotlib.pyplot as plt
import numpy as np
from sklearn.linear_model import LinearRegression, Ridge, Lasso
from sklearn.preprocessing import PolynomialFeatures
@@ -1037,7 +1075,7 @@ x_train, x_test, y_train, y_test = train_tes
# Combine x transformation and model into one operation.
# Not neccesary, but convenient.
-model = make_pipeline(PolynomialFeatures(degree=degree), LinearRegression(fit_intercept=False))
+model = make_pipeline(PolynomialFeatures(degree=degree), LinearRegression(fit_intercept=False))
# The following (m x n_bootstraps) matrix holds the column vectors y_pred
# for each bootstrap iteration.
@@ -1054,13 +1092,13 @@ y_pred = np.# calculated per data point in the test set.
# Note 2: The use of keepdims=True is important in the calculation of bias as this
# maintains the column vector form. Dropping this yields very unexpected results.
-error = np.mean( np.mean((y_test - y_pred)**2, axis=1, keepdims=True) )
-bias = np.mean( (y_test - np.mean(y_pred, axis=1, keepdims=True))**2 )
-variance = np.mean( np.var(y_pred, axis=1, keepdims=True) )
-print('Error:', error)
-print('Bias^2:', bias)
-print('Var:', variance)
-print('{} >= {} + {} = {}'.format(error, bias, variance, bias+variance))
+error = np.mean( np.mean((y_test - y_pred)**2, axis=1, keepdims=True) )
+bias = np.mean( (y_test - np.mean(y_pred, axis=1, keepdims=True))**2 )
+variance = np.mean( np.var(y_pred, axis=1, keepdims=True) )
+print('Error:', error)
+print('Bias^2:', bias)
+print('Var:', variance)
+print('{} >= {} + {} = {}'.format(error, bias, variance, bias+variance))
plt.plot(x[::5, :], y[::5, :], label='f(x)')
plt.scatter(x_test, y_test, label='Data points')
@@ -1071,11 +1109,11 @@ plt.show()
-Understanding what happens
+Understanding what happens
import matplotlib.pyplot as plt
+
import matplotlib.pyplot as plt
import numpy as np
from sklearn.linear_model import LinearRegression, Ridge, Lasso
from sklearn.preprocessing import PolynomialFeatures
@@ -1100,21 +1138,21 @@ polydegree = np
x_train, x_test, y_train, y_test = train_test_split(x, y, test_size=0.2)
for degree in range(maxdegree):
- model = make_pipeline(PolynomialFeatures(degree=degree), LinearRegression(fit_intercept=False))
+ model = make_pipeline(PolynomialFeatures(degree=degree), LinearRegression(fit_intercept=False))
y_pred = np.empty((y_test.shape[0], n_boostraps))
for i in range(n_boostraps):
x_, y_ = resample(x_train, y_train)
y_pred[:, i] = model.fit(x_, y_).predict(x_test).ravel()
polydegree[degree] = degree
- error[degree] = np.mean( np.mean((y_test - y_pred)**2, axis=1, keepdims=True) )
- bias[degree] = np.mean( (y_test - np.mean(y_pred, axis=1, keepdims=True))**2 )
- variance[degree] = np.mean( np.var(y_pred, axis=1, keepdims=True) )
- print('Polynomial degree:', degree)
- print('Error:', error[degree])
- print('Bias^2:', bias[degree])
- print('Var:', variance[degree])
- print('{} >= {} + {} = {}'.format(error[degree], bias[degree], variance[degree], bias[degree]+variance[degree]))
+ error[degree] = np.mean( np.mean((y_test - y_pred)**2, axis=1, keepdims=True) )
+ bias[degree] = np.mean( (y_test - np.mean(y_pred, axis=1, keepdims=True))**2 )
+ variance[degree] = np.mean( np.var(y_pred, axis=1, keepdims=True) )
+ print('Polynomial degree:', degree)
+ print('Error:', error[degree])
+ print('Bias^2:', bias[degree])
+ print('Var:', variance[degree])
+ print('{} >= {} + {} = {}'.format(error[degree], bias[degree], variance[degree], bias[degree]+variance[degree]))
plt.plot(polydegree, error, label='Error')
plt.plot(polydegree, bias, label='bias')
@@ -1125,7 +1163,7 @@ plt.show()
Summing up
+Summing up
Another Example from Scikit-Learn's Repository
"""
+
"""
============================
Underfitting vs. Overfitting
============================
@@ -1188,7 +1226,7 @@ You may also find this recent training data.
"""
-print(__doc__)
+print(__doc__)
import numpy as np
import matplotlib.pyplot as plt
@@ -1215,7 +1253,7 @@ plt.figure(figsize.setp(ax, xticks=(), yticks=())
polynomial_features = PolynomialFeatures(degree=degrees[i],
- include_bias=False)
+ include_bias=False)
linear_regression = LinearRegression()
pipeline = Pipeline([("polynomial_features", polynomial_features),
("linear_regression", linear_regression)])
@@ -1234,19 +1272,19 @@ plt.figure(figsize.xlim((0, 1))
plt.ylim((-2, 2))
plt.legend(loc="best")
- plt.title("Degree {}\nMSE = {:.2e}(+/- {:.2e})".format(
+ plt.title("Degree {}\nMSE = {:.2e}(+/- {:.2e})".format(
degrees[i], -scores.mean(), scores.std()))
plt.show()
-More examples on bootstrap and cross-validation and errors
+More examples on bootstrap and cross-validation and errors
# Common imports
+
# Common imports
import os
import numpy as np
import pandas as pd
@@ -1276,7 +1314,7 @@ DATA_ID = "
return os.path.join(DATA_ID, dat_id)
def save_fig(fig_id):
- plt.savefig(image_path(fig_id) + ".png", format='png')
+ plt.savefig(image_path(fig_id) + ".png", format='png')
infile = open(data_path("EoS.csv"),'r')
@@ -1306,7 +1344,7 @@ trials = 100= 0.0
for samples in range(trials):
x_train, x_test, y_train, y_test = train_test_split(X, Energies, test_size=0.2)
- model = LinearRegression(fit_intercept=True).fit(x_train, y_train)
+ model = LinearRegression(fit_intercept=True).fit(x_train, y_train)
ypred = model.predict(x_train)
ytilde = model.predict(x_test)
testerror[polydegree] += mean_squared_error(y_test, ytilde)
@@ -1314,9 +1352,9 @@ trials = 100/= trials
trainingerror[polydegree] /= trials
- print("Degree of polynomial: %3d"% polynomial[polydegree])
- print("Mean squared error on training data: %.8f" % trainingerror[polydegree])
- print("Mean squared error on test data: %.8f" % testerror[polydegree])
+ print("Degree of polynomial: %3d"% polynomial[polydegree])
+ print("Mean squared error on training data: %.8f" % trainingerror[polydegree])
+ print("Mean squared error on test data: %.8f" % testerror[polydegree])
plt.plot(polynomial, np.log10(trainingerror), label='Training Error')
plt.plot(polynomial, np.log10(testerror), label='Test Error')
@@ -1328,12 +1366,12 @@ plt.show()
The same example but now with cross-validation
+The same example but now with cross-validation
# Common imports
+
# Common imports
import os
import numpy as np
import pandas as pd
@@ -1365,7 +1403,7 @@ DATA_ID = "
return os.path.join(DATA_ID, dat_id)
def save_fig(fig_id):
- plt.savefig(image_path(fig_id) + ".png", format='png')
+ plt.savefig(image_path(fig_id) + ".png", format='png')
infile = open(data_path("EoS.csv"),'r')
@@ -1404,11 +1442,11 @@ plt.show()
-Cross-validation with Ridge
+Cross-validation with Ridge
import numpy as np
+
import numpy as np
import matplotlib.pyplot as plt
from sklearn.model_selection import KFold
from sklearn.linear_model import Ridge
@@ -1444,24 +1482,13 @@ plt.ylabel('
plt.legend()
plt.show()
-
-Friday September 4
-
-