Fitting vs. predicting when data is in the model class
We start by considering the case
\( f(x)=2x \).
Then the data is clearly generated by a model that is contained within
all three model classes we are using to make predictions (linear
models, third order polynomials, and tenth order polynomials).
Run the code for the following cases:
- For \( f(x)=2x \) , \( Ntrain=10 \) and \( \sigma =0 \) (noiseless case), train the three classes of models (linear, third-order polynomial, and tenth order polynomial) for a training set when \( x \in [0,1] \) . Make graphs comparing fits for different order of polynomials. Which model fits the data the best?
- Do you think that the data that has the least error on the training set will also make the best predictions? Why or why not? Can you try to discuss and formalize your intuition? What can go right and what can go wrong?
- Check your answer by seeing how well your fits predict newly generated test data (including on data outside the range you fit on, for example \( x \in [0,1.2] \) ) using the code below. How well do you do on points in the range of x where you trained the model? How about points outside the original training data set?
- Repeat the above for \( f(x)=2x \) , \( Ntrain=10 \) , and \( \sigma=1 \) . What changes?
Repeat the exercises above for \( f(x)=2x \) , \( Ntrain=100 \) , and \( \sigma=1 \) . What changes?
Summarize what you have learned about the relationship between model complexity (number of parameters), goodness of fit on training data, and the ability to predict well.