lars/FYS-STK4155
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

updating week 36

Browse Source
This commit is contained in:
Morten Hjorth-Jensen
2021-09-06 22:28:54 +02:00
parent ed47ea2760
commit 331bb78d0a
7 changed files with 1110 additions and 1032 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/src/week36/week36.do.txt
+100 -105
View File
@@ -1,4 +1,4 @@
TITLE: Week 36: Resampling techniques and Ordinary Least Square
TITLE: Week 36: Statistical interpretation of Linear Regression and Resampling techniques
AUTHOR: Morten Hjorth-Jensen {copyright, 1999-present|CC BY-NC} at Department of Physics, University of Oslo & Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University
DATE: today
@@ -6,103 +6,12 @@ DATE: today
!split
===== Plans for week 36 =====
* Thursday: Statistics, probability theory and resampling methods
* Friday: Resampling methods and motivation for Ridge Regression
* Thursday: Statistics, probability theory and linear regression
* Friday: Resampling methods
!split
===== Thursday September 3 =====
"Video of Lecture":"https://www.uio.no/studier/emner/matnat/fys/FYS-STK4155/h20/forelesningsvideoer/LectureSept3.mp4?vrtx=view-as-webpage" and "handwritten notes":"https://github.com/CompPhysics/MachineLearning/blob/master/doc/HandWrittenNotes/NotesSeptember3.pdf"
More material will be added here, see handwritten notes also.
!split
===== Why resampling methods =====
Before we proceed, we need to rethink what we have been doing. In our
eager to fit the data, we have omitted several important elements in
our regression analysis. In what follows we will
o look at statistical properties, including a discussion of mean values, variance and the so-called bias-variance tradeoff
o introduce resampling techniques like cross-validation, bootstrapping and jackknife and more
This will allow us to link the standard linear algebra methods we have discussed above to a statistical interpretation of the methods.
!split
===== Resampling methods =====
!bblock
Resampling methods are an indispensable tool in modern
statistics. They involve repeatedly drawing samples from a training
set and refitting a model of interest on each sample in order to
obtain additional information about the fitted model. For example, in
order to estimate the variability of a linear regression fit, we can
repeatedly draw different samples from the training data, fit a linear
regression to each new sample, and then examine the extent to which
the resulting fits differ. Such an approach may allow us to obtain
information that would not be available from fitting the model only
once using the original training sample.
Two resampling methods are often used in Machine Learning analyses,
o The _bootstrap method_
o and _Cross-Validation_
In addition there are several other methods such as the Jackknife and the Blocking methods. We will discuss in particular
cross-validation and the bootstrap method.
!eblock
!split
===== Resampling approaches can be computationally expensive =====
!bblock
Resampling approaches can be computationally expensive, because they
involve fitting the same statistical method multiple times using
different subsets of the training data. However, due to recent
advances in computing power, the computational requirements of
resampling methods generally are not prohibitive. In this chapter, we
discuss two of the most commonly used resampling methods,
cross-validation and the bootstrap. Both methods are important tools
in the practical application of many statistical learning
procedures. For example, cross-validation can be used to estimate the
test error associated with a given statistical learning method in
order to evaluate its performance, or to select the appropriate level
of flexibility. The process of evaluating a model’s performance is
known as model assessment, whereas the process of selecting the proper
level of flexibility for a model is known as model selection. The
bootstrap is widely used.
!eblock
!split
===== Why resampling methods ? =====
!bblock Statistical analysis
* Our simulations can be treated as *computer experiments*. This is particularly the case for Monte Carlo methods
* The results can be analysed with the same statistical tools as we would use analysing experimental data.
* As in all experiments, we are looking for expectation values and an estimate of how accurate they are, i.e., possible sources for errors.
!eblock
!split
===== Statistical analysis =====
!bblock
* As in other experiments, many numerical experiments have two classes of errors:
* Statistical errors
* Systematical errors
* Statistical errors can be estimated using standard tools from statistics
* Systematical errors are method specific and must be treated differently from case to case.
!eblock
===== Thursday September 9 =====
!split
@@ -270,6 +179,102 @@ matrix product is non-negative definite.
This means the variance we obtain with the standard OLS will always for $\lambda > 0$ be larger than the variance of $\bm{\beta}$ obtained with the Ridge estimator. This has interesting consequences when we discuss the so-called bias-variance trade-off below.
!split
===== Friday September 10 =====
!split
===== Why resampling methods =====
Before we proceed, we need to rethink what we have been doing. In our
eager to fit the data, we have omitted several important elements in
our regression analysis. In what follows we will
o look at statistical properties, including a discussion of mean values, variance and the so-called bias-variance tradeoff
o introduce resampling techniques like cross-validation, bootstrapping and jackknife and more
This will allow us to link the standard linear algebra methods we have discussed above to a statistical interpretation of the methods.
!split
===== Resampling methods =====
!bblock
Resampling methods are an indispensable tool in modern
statistics. They involve repeatedly drawing samples from a training
set and refitting a model of interest on each sample in order to
obtain additional information about the fitted model. For example, in
order to estimate the variability of a linear regression fit, we can
repeatedly draw different samples from the training data, fit a linear
regression to each new sample, and then examine the extent to which
the resulting fits differ. Such an approach may allow us to obtain
information that would not be available from fitting the model only
once using the original training sample.
Two resampling methods are often used in Machine Learning analyses,
o The _bootstrap method_
o and _Cross-Validation_
In addition there are several other methods such as the Jackknife and the Blocking methods. We will discuss in particular
cross-validation and the bootstrap method.
!eblock
!split
===== Resampling approaches can be computationally expensive =====
!bblock
Resampling approaches can be computationally expensive, because they
involve fitting the same statistical method multiple times using
different subsets of the training data. However, due to recent
advances in computing power, the computational requirements of
resampling methods generally are not prohibitive. In this chapter, we
discuss two of the most commonly used resampling methods,
cross-validation and the bootstrap. Both methods are important tools
in the practical application of many statistical learning
procedures. For example, cross-validation can be used to estimate the
test error associated with a given statistical learning method in
order to evaluate its performance, or to select the appropriate level
of flexibility. The process of evaluating a model’s performance is
known as model assessment, whereas the process of selecting the proper
level of flexibility for a model is known as model selection. The
bootstrap is widely used.
!eblock
!split
===== Why resampling methods ? =====
!bblock Statistical analysis
* Our simulations can be treated as *computer experiments*. This is particularly the case for Monte Carlo methods
* The results can be analysed with the same statistical tools as we would use analysing experimental data.
* As in all experiments, we are looking for expectation values and an estimate of how accurate they are, i.e., possible sources for errors.
!eblock
!split
===== Statistical analysis =====
!bblock
* As in other experiments, many numerical experiments have two classes of errors:
* Statistical errors
* Systematical errors
* Statistical errors can be estimated using standard tools from statistics
* Systematical errors are method specific and must be treated differently from case to case.
!eblock
!split
===== Resampling methods =====
@@ -1174,13 +1179,3 @@ plt.show()
!ec
!split
===== Friday September 4 =====
"Video of Lecture":"https://www.uio.no/studier/emner/matnat/fys/FYS-STK4155/h20/forelesningsvideoer/LectureSeptember4.mp4?vrtx=view-as-webpage" and "handwritten notes":"https://github.com/CompPhysics/MachineLearning/blob/master/doc/HandWrittenNotes/NotesSeptember4.pdf"
More material will be added here, see handwritten notes also.