mhjensen
299 lines
10 KiB
TeX
299 lines
10 KiB
TeX
%%
|
||
%% Automatically generated file from DocOnce source
|
||
%% (https://github.com/hplgit/doconce/)
|
||
%%
|
||
%%
|
||
|
||
|
||
%-------------------- begin preamble ----------------------
|
||
|
||
\documentclass[%
|
||
oneside, % oneside: electronic viewing, twoside: printing
|
||
final, % draft: marks overfull hboxes, figures with paths
|
||
10pt]{article}
|
||
|
||
\listfiles % print all files needed to compile this document
|
||
|
||
\usepackage{relsize,makeidx,color,setspace,amsmath,amsfonts,amssymb}
|
||
\usepackage[table]{xcolor}
|
||
\usepackage{bm,ltablex,microtype}
|
||
|
||
\usepackage[pdftex]{graphicx}
|
||
|
||
\usepackage{fancyvrb} % packages needed for verbatim environments
|
||
|
||
\usepackage[T1]{fontenc}
|
||
%\usepackage[latin1]{inputenc}
|
||
\usepackage{ucs}
|
||
\usepackage[utf8x]{inputenc}
|
||
|
||
\usepackage{lmodern} % Latin Modern fonts derived from Computer Modern
|
||
|
||
% Hyperlinks in PDF:
|
||
\definecolor{linkcolor}{rgb}{0,0,0.4}
|
||
\usepackage{hyperref}
|
||
\hypersetup{
|
||
breaklinks=true,
|
||
colorlinks=true,
|
||
linkcolor=linkcolor,
|
||
urlcolor=linkcolor,
|
||
citecolor=black,
|
||
filecolor=black,
|
||
%filecolor=blue,
|
||
pdfmenubar=true,
|
||
pdftoolbar=true,
|
||
bookmarksdepth=3 % Uncomment (and tweak) for PDF bookmarks with more levels than the TOC
|
||
}
|
||
%\hyperbaseurl{} % hyperlinks are relative to this root
|
||
|
||
\setcounter{tocdepth}{2} % levels in table of contents
|
||
|
||
% --- fancyhdr package for fancy headers ---
|
||
\usepackage{fancyhdr}
|
||
\fancyhf{} % sets both header and footer to nothing
|
||
\renewcommand{\headrulewidth}{0pt}
|
||
\fancyfoot[LE,RO]{\thepage}
|
||
% Ensure copyright on titlepage (article style) and chapter pages (book style)
|
||
\fancypagestyle{plain}{
|
||
\fancyhf{}
|
||
\fancyfoot[C]{{\footnotesize \copyright\ 1999-2019, "Data Analysis and Machine Learning FYS-STK3155/FYS4155":"http://www.uio.no/studier/emner/matnat/fys/FYS3155/index-eng.html". Released under CC Attribution-NonCommercial 4.0 license}}
|
||
% \renewcommand{\footrulewidth}{0mm}
|
||
\renewcommand{\headrulewidth}{0mm}
|
||
}
|
||
% Ensure copyright on titlepages with \thispagestyle{empty}
|
||
\fancypagestyle{empty}{
|
||
\fancyhf{}
|
||
\fancyfoot[C]{{\footnotesize \copyright\ 1999-2019, "Data Analysis and Machine Learning FYS-STK3155/FYS4155":"http://www.uio.no/studier/emner/matnat/fys/FYS3155/index-eng.html". Released under CC Attribution-NonCommercial 4.0 license}}
|
||
\renewcommand{\footrulewidth}{0mm}
|
||
\renewcommand{\headrulewidth}{0mm}
|
||
}
|
||
|
||
\pagestyle{fancy}
|
||
|
||
|
||
% prevent orhpans and widows
|
||
\clubpenalty = 10000
|
||
\widowpenalty = 10000
|
||
|
||
% --- end of standard preamble for documents ---
|
||
|
||
|
||
% insert custom LaTeX commands...
|
||
|
||
\raggedbottom
|
||
\makeindex
|
||
\usepackage[totoc]{idxlayout} % for index in the toc
|
||
\usepackage[nottoc]{tocbibind} % for references/bibliography in the toc
|
||
|
||
%-------------------- end preamble ----------------------
|
||
|
||
\begin{document}
|
||
|
||
% matching end for #ifdef PREAMBLE
|
||
|
||
\newcommand{\exercisesection}[1]{\subsection*{#1}}
|
||
|
||
|
||
% ------------------- main content ----------------------
|
||
|
||
|
||
|
||
% ----------------- title -------------------------
|
||
|
||
\thispagestyle{empty}
|
||
|
||
\begin{center}
|
||
{\LARGE\bf
|
||
\begin{spacing}{1.25}
|
||
Homework 1 Fall Semester 2019
|
||
\end{spacing}
|
||
}
|
||
\end{center}
|
||
|
||
% ----------------- author(s) -------------------------
|
||
|
||
\begin{center}
|
||
{\bf \href{{http://www.uio.no/studier/emner/matnat/fys/FYS3155/index-eng.html}}{Data Analysis and Machine Learning FYS-STK3155/FYS4155}}
|
||
\end{center}
|
||
|
||
\begin{center}
|
||
% List of all institutions:
|
||
\centerline{{\small Department of Physics, University of Oslo, Norway}}
|
||
\end{center}
|
||
|
||
% ----------------- end author(s) -------------------------
|
||
|
||
% --- begin date ---
|
||
\begin{center}
|
||
Aug 20, 2019
|
||
\end{center}
|
||
% --- end date ---
|
||
|
||
\vspace{1cm}
|
||
|
||
|
||
\subsection*{Exercise 1}
|
||
|
||
The first exercise here is of a mere technical art. We want you to have
|
||
\begin{itemize}
|
||
\item git as a version control software and to establish a user account on a provider like GitHub. Other providers like GitLab etc are equally fine. You can also use the University of Oslo \href{{https://www.uio.no/tjenester/it/maskin/filer/versjonskontroll/github.html}}{GitHub facilities}.
|
||
|
||
\item Install various Python packages
|
||
\end{itemize}
|
||
|
||
\noindent
|
||
We will make extensive use of Python as programming language and its
|
||
myriad of available libraries. You will find
|
||
IPython/Jupyter notebooks invaluable in your work. You can run \textbf{R}
|
||
codes in the Jupyter/IPython notebooks, with the immediate benefit of
|
||
visualizing your data. You can also use compiled languages like C++,
|
||
Rust, Fortran etc if you prefer. The focus in these lectures will be
|
||
on Python.
|
||
|
||
If you have Python installed (we recommend Python3) and you feel
|
||
pretty familiar with installing different packages, we recommend that
|
||
you install the following Python packages via \textbf{pip} as
|
||
|
||
\begin{enumerate}
|
||
\item pip install numpy scipy matplotlib ipython scikit-learn sympy pandas pillow
|
||
\end{enumerate}
|
||
|
||
\noindent
|
||
For \textbf{Tensorflow}, we recommend following the instructions in the text of
|
||
\href{{http://shop.oreilly.com/product/0636920052289.do}}{Aurelien Geron, Hands‑On Machine Learning with Scikit‑Learn and TensorFlow, O'Reilly}
|
||
|
||
We will come back to \textbf{tensorflow} later.
|
||
|
||
For Python3, replace \textbf{pip} with \textbf{pip3}.
|
||
|
||
For OSX users we recommend, after having installed Xcode, to
|
||
install \textbf{brew}. Brew allows for a seamless installation of additional
|
||
software via for example
|
||
|
||
\begin{enumerate}
|
||
\item brew install python3
|
||
\end{enumerate}
|
||
|
||
\noindent
|
||
For Linux users, with its variety of distributions like for example the widely popular Ubuntu distribution,
|
||
you can use \textbf{pip} as well and simply install Python as
|
||
|
||
\begin{enumerate}
|
||
\item sudo apt-get install python3 (or python for pyhton2.7)
|
||
\end{enumerate}
|
||
|
||
\noindent
|
||
If you don't want to perform these operations separately and venture
|
||
into the hassle of exploring how to set up dependencies and paths, we
|
||
recommend two widely used distrubutions which set up all relevant
|
||
dependencies for Python, namely
|
||
|
||
\begin{itemize}
|
||
\item \href{{https://docs.anaconda.com/}}{Anaconda},
|
||
\end{itemize}
|
||
|
||
\noindent
|
||
which is an open source
|
||
distribution of the Python and R programming languages for large-scale
|
||
data processing, predictive analytics, and scientific computing, that
|
||
aims to simplify package management and deployment. Package versions
|
||
are managed by the package management system \textbf{conda}.
|
||
|
||
\begin{itemize}
|
||
\item \href{{https://www.enthought.com/product/canopy/}}{Enthought canopy}
|
||
\end{itemize}
|
||
|
||
\noindent
|
||
is a Python
|
||
distribution for scientific and analytic computing distribution and
|
||
analysis environment, available for free and under a commercial
|
||
license.
|
||
|
||
We recommend using \textbf{Anaconda}.
|
||
|
||
\subsection*{Exercise 2}
|
||
|
||
We will generate our own dataset for a function $y(x)$ where $x \in [0,1]$ and defined by random numbers computed with the uniform distribution. The function $y$ is a quadratic polynomial in $x$ with added stochastic noise according to the normal distribution $\cal {N}(0,1)$.
|
||
The following simple Python instructions define our $x$ and $y$ values (with 100 data points).
|
||
\begin{verbatim}
|
||
x = np.random.rand(100,1)
|
||
y = 5*x*x+0.1*np.random.randn(100,1)
|
||
\end{verbatim}
|
||
|
||
\begin{enumerate}
|
||
\item Write your own code (following the examples under the \href{{https://compphysics.github.io/MachineLearning/doc/pub/Regression/html/Regression-bs.html}}{regression slides}) for computing the parametrization of the data set fitting a second-order polynomial.
|
||
|
||
\item Use thereafter \textbf{scikit-learn} (see again the examples in the regression slides) and compare with your own code.
|
||
|
||
\item Using scikit-learn, compute also the mean square error, a risk metric corresponding to the expected value of the squared (quadratic) error defined as
|
||
\end{enumerate}
|
||
|
||
\noindent
|
||
\[ MSE(\hat{y},\hat{\tilde{y}}) = \frac{1}{n}
|
||
\sum_{i=0}^{n-1}(y_i-\tilde{y}_i)^2,
|
||
\]
|
||
and the $R^2$ score function.
|
||
If $\tilde{\hat{y}}_i$ is the predicted value of the $i-th$ sample and $y_i$ is the corresponding true value, then the score $R^2$ is defined as
|
||
\[
|
||
R^2(\hat{y}, \tilde{\hat{y}}) = 1 - \frac{\sum_{i=0}^{n - 1} (y_i - \tilde{y}_i)^2}{\sum_{i=0}^{n - 1} (y_i - \bar{y})^2},
|
||
\]
|
||
where we have defined the mean value of $\hat{y}$ as
|
||
\[
|
||
\bar{y} = \frac{1}{n} \sum_{i=0}^{n - 1} y_i.
|
||
\]
|
||
You can use the functionality included in scikit-learn. If you feel for it, you can use your own program and define functions which compute the above two functions.
|
||
Discuss the meaning of these results. Try also to vary the coefficient in front of the added stochastic noise term and discuss the quality of the fits.
|
||
|
||
|
||
|
||
|
||
\subsection*{Exercise 3, mean values and variances in linear regression}
|
||
|
||
|
||
This exercise deals with various mean values ad variances in linear regression method (here it may be useful to look up chapter 3, equation (3.8) of \href{{https://www.springer.com/gp/book/9780387848570}}{Trevor Hastie, Robert Tibshirani, Jerome H. Friedman, The Elements of Statistical Learning, Springer}).
|
||
|
||
The assumption we have made is
|
||
that there exists a function $f(\bm{x})$ and a normal distributed error $\bm{\varepsilon}\sim \mathcal{N}(0, \sigma^2)$
|
||
which describes our data
|
||
\[
|
||
\bm{y} = f(\bm{x})+\bm{\varepsilon}
|
||
\]
|
||
|
||
We then approximate this function with our model from the solution of the linear regression equations (ordinary least squares OLS), that is our
|
||
function $f$ is approximated by $\bm{\tilde{y}}$ where we minimized $(\bm{y}-\bm{\tilde{y}})^2$, with
|
||
\[
|
||
\bm{\tilde{y}} = \bm{X}\bm{\beta}.
|
||
\]
|
||
The matrix $\bm{X}$ is the so-called design matrix.
|
||
|
||
|
||
Show that the expectation value of $\bm{y}$ for a given element $i$
|
||
\begin{align*}
|
||
\mathbb{E}(y_i) & =\mathbf{X}_{i, \ast} \, \beta,
|
||
\end{align*}
|
||
and that
|
||
its variance is
|
||
\begin{align*} \mbox{Var}(y_i) & = \sigma^2.
|
||
\end{align*}
|
||
Hence, $y_i \sim \mathcal{N}( \mathbf{X}_{i, \ast} \, \bm{\beta}, \sigma^2)$, that is $\bm{y}$ follows a normal distribution with
|
||
mean value $\bm{X}\bm{\beta}$ and variance $\sigma^2$.
|
||
|
||
|
||
With the OLS expressions for the parameters $\bm{\beta}$ show that
|
||
\[
|
||
\mathbb{E}(\bm{\beta}) = \bm{\beta}.
|
||
\]
|
||
This means that the estimator of the regression parameters is unbiased.
|
||
|
||
Show finally that the variance of $\bm{\beta}$ is
|
||
\begin{eqnarray*}
|
||
\mbox{Var}(\bm{\beta}) & = & \sigma^2 \, (\mathbf{X}^{T} \mathbf{X})^{-1}.
|
||
\end{eqnarray*}
|
||
|
||
|
||
|
||
% ------------------- end of main content ---------------
|
||
|
||
\end{document}
|
||
|