added files

doc/src/LectureNotes/_build/html/.buildinfo

@@ -1,4 +1,4 @@
 # Sphinx build info version 1
 # This file hashes the configuration used when building these files. When it is not found, a full rebuild will be done.
-config: fdb4c65fe6fdd143bcbae649c1ee1a55
+config: 29159f964fb7395f0e429ef40b5eaa8d
 tags: 645f666f9bcd5a90fca523b33c5a78b7

doc/src/LectureNotes/_build/html/_sources/chapter5.ipynb

@@ -0,0 +1,583 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Logistic Regression\n",
+    "\n",
+    "## Introduction\n",
+    "In linear regression our main interest was centered on learning the\n",
+    "coefficients of a functional fit (say a polynomial) in order to be\n",
+    "able to predict the response of a continuous variable on some unseen\n",
+    "data. The fit to the continuous variable $y_i$ is based on some\n",
+    "independent variables $\\hat{x}_i$. Linear regression resulted in\n",
+    "analytical expressions for standard ordinary Least Squares or Ridge\n",
+    "regression (in terms of matrices to invert) for several quantities,\n",
+    "ranging from the variance and thereby the confidence intervals of the\n",
+    "parameters $\\hat{\\beta}$ to the mean squared error. If we can invert\n",
+    "the product of the design matrices, linear regression gives then a\n",
+    "simple recipe for fitting our data.\n",
+    "\n",
+    "\n",
+    "Classification problems, however, are concerned with outcomes taking\n",
+    "the form of discrete variables (i.e. categories). We may for example,\n",
+    "on the basis of DNA sequencing for a number of patients, like to find\n",
+    "out which mutations are important for a certain disease; or based on\n",
+    "scans of various patients' brains, figure out if there is a tumor or\n",
+    "not; or given a specific physical system, we'd like to identify its\n",
+    "state, say whether it is an ordered or disordered system (typical\n",
+    "situation in solid state physics); or classify the status of a\n",
+    "patient, whether she/he has a stroke or not and many other similar\n",
+    "situations.\n",
+    "\n",
+    "The most common situation we encounter when we apply logistic\n",
+    "regression is that of two possible outcomes, normally denoted as a\n",
+    "binary outcome, true or false, positive or negative, success or\n",
+    "failure etc.\n",
+    "\n",
+    "Logistic regression will also serve as our stepping stone towards\n",
+    "neural network algorithms and supervised deep learning. For logistic\n",
+    "learning, the minimization of the cost function leads to a non-linear\n",
+    "equation in the parameters $\\hat{\\beta}$. The optimization of the\n",
+    "problem calls therefore for minimization algorithms. This forms the\n",
+    "bottle neck of all machine learning algorithms, namely how to find\n",
+    "reliable minima of a multi-variable function. This leads us to the\n",
+    "family of gradient descent methods. The latter are the working horses\n",
+    "of basically all modern machine learning algorithms.\n",
+    "\n",
+    "We note also that many of the topics discussed here on logistic \n",
+    "regression are also commonly used in modern supervised Deep Learning\n",
+    "models, as we will see later.\n",
+    "\n",
+    "\n",
+    "\n",
+    "## Basics\n",
+    "\n",
+    "We consider the case where the dependent variables, also called the\n",
+    "responses or the outcomes, $y_i$ are discrete and only take values\n",
+    "from $k=0,\\dots,K-1$ (i.e. $K$ classes).\n",
+    "\n",
+    "The goal is to predict the\n",
+    "output classes from the design matrix $\\hat{X}\\in\\mathbb{R}^{n\\times p}$\n",
+    "made of $n$ samples, each of which carries $p$ features or predictors. The\n",
+    "primary goal is to identify the classes to which new unseen samples\n",
+    "belong.\n",
+    "\n",
+    "Let us specialize to the case of two classes only, with outputs\n",
+    "$y_i=0$ and $y_i=1$. Our outcomes could represent the status of a\n",
+    "credit card user that could default or not on her/his credit card\n",
+    "debt. That is"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "$$\n",
+    "y_i = \\begin{bmatrix} 0 & \\mathrm{no}\\\\  1 & \\mathrm{yes} \\end{bmatrix}.\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Before moving to the logistic model, let us try to use our linear\n",
+    "regression model to classify these two outcomes. We could for example\n",
+    "fit a linear model to the default case if $y_i > 0.5$ and the no\n",
+    "default case $y_i \\leq 0.5$.\n",
+    "\n",
+    "We would then have our \n",
+    "weighted linear combination, namely"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "<!-- Equation labels as ordinary links -->\n",
+    "<div id=\"_auto1\"></div>\n",
+    "\n",
+    "$$\n",
+    "\\begin{equation}\n",
+    "\\hat{y} = \\hat{X}^T\\hat{\\beta} +  \\hat{\\epsilon},\n",
+    "\\label{_auto1} \\tag{1}\n",
+    "\\end{equation}\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "where $\\hat{y}$ is a vector representing the possible outcomes, $\\hat{X}$ is our\n",
+    "$n\\times p$ design matrix and $\\hat{\\beta}$ represents our estimators/predictors.\n",
+    "\n",
+    "\n",
+    "The main problem with our function is that it takes values on the\n",
+    "entire real axis. In the case of logistic regression, however, the\n",
+    "labels $y_i$ are discrete variables. A typical example is the credit\n",
+    "card data discussed below here, where we can set the state of\n",
+    "defaulting the debt to $y_i=1$ and not to $y_i=0$ for one the persons\n",
+    "in the data set (see the full example below).\n",
+    "\n",
+    "One simple way to get a discrete output is to have sign\n",
+    "functions that map the output of a linear regressor to values $\\{0,1\\}$,\n",
+    "$f(s_i)=sign(s_i)=1$ if $s_i\\ge 0$ and 0 if otherwise. \n",
+    "We will encounter this model in our first demonstration of neural networks. Historically it is called the \"perceptron\" model in the machine learning\n",
+    "literature. This model is extremely simple. However, in many cases it is more\n",
+    "favorable to use a ``soft\" classifier that outputs\n",
+    "the probability of a given category. This leads us to the logistic function.\n",
+    "\n",
+    "\n",
+    "\n",
+    "## The logistic function\n",
+    "\n",
+    "The perceptron is an example of a ``hard classification\" model. We\n",
+    "will encounter this model when we discuss neural networks as\n",
+    "well. Each datapoint is deterministically assigned to a category (i.e\n",
+    "$y_i=0$ or $y_i=1$). In many cases, it is favorable to have a \"soft\"\n",
+    "classifier that outputs the probability of a given category rather\n",
+    "than a single value. For example, given $x_i$, the classifier\n",
+    "outputs the probability of being in a category $k$.  Logistic regression\n",
+    "is the most common example of a so-called soft classifier. In logistic\n",
+    "regression, the probability that a data point $x_i$\n",
+    "belongs to a category $y_i=\\{0,1\\}$ is given by the so-called logit function (or Sigmoid) which is meant to represent the likelihood for a given event,"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "$$\n",
+    "p(t) = \\frac{1}{1+\\mathrm \\exp{-t}}=\\frac{\\exp{t}}{1+\\mathrm \\exp{t}}.\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Note that $1-p(t)= p(-t)$.\n",
+    "\n",
+    "\n",
+    "The following code plots the logistic function, the step function and other functions we will encounter from here and on."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "metadata": {
+    "collapsed": false
+   },
+   "outputs": [],
+   "source": [
+    "%matplotlib inline\n",
+    "\n",
+    "\"\"\"The sigmoid function (or the logistic curve) is a\n",
+    "function that takes any real number, z, and outputs a number (0,1).\n",
+    "It is useful in neural networks for assigning weights on a relative scale.\n",
+    "The value z is the weighted sum of parameters involved in the learning algorithm.\"\"\"\n",
+    "\n",
+    "import numpy\n",
+    "import matplotlib.pyplot as plt\n",
+    "import math as mt\n",
+    "\n",
+    "z = numpy.arange(-5, 5, .1)\n",
+    "sigma_fn = numpy.vectorize(lambda z: 1/(1+numpy.exp(-z)))\n",
+    "sigma = sigma_fn(z)\n",
+    "\n",
+    "fig = plt.figure()\n",
+    "ax = fig.add_subplot(111)\n",
+    "ax.plot(z, sigma)\n",
+    "ax.set_ylim([-0.1, 1.1])\n",
+    "ax.set_xlim([-5,5])\n",
+    "ax.grid(True)\n",
+    "ax.set_xlabel('z')\n",
+    "ax.set_title('sigmoid function')\n",
+    "\n",
+    "plt.show()\n",
+    "\n",
+    "\"\"\"Step Function\"\"\"\n",
+    "z = numpy.arange(-5, 5, .02)\n",
+    "step_fn = numpy.vectorize(lambda z: 1.0 if z >= 0.0 else 0.0)\n",
+    "step = step_fn(z)\n",
+    "\n",
+    "fig = plt.figure()\n",
+    "ax = fig.add_subplot(111)\n",
+    "ax.plot(z, step)\n",
+    "ax.set_ylim([-0.5, 1.5])\n",
+    "ax.set_xlim([-5,5])\n",
+    "ax.grid(True)\n",
+    "ax.set_xlabel('z')\n",
+    "ax.set_title('step function')\n",
+    "\n",
+    "plt.show()\n",
+    "\n",
+    "\"\"\"tanh Function\"\"\"\n",
+    "z = numpy.arange(-2*mt.pi, 2*mt.pi, 0.1)\n",
+    "t = numpy.tanh(z)\n",
+    "\n",
+    "fig = plt.figure()\n",
+    "ax = fig.add_subplot(111)\n",
+    "ax.plot(z, t)\n",
+    "ax.set_ylim([-1.0, 1.0])\n",
+    "ax.set_xlim([-2*mt.pi,2*mt.pi])\n",
+    "ax.grid(True)\n",
+    "ax.set_xlabel('z')\n",
+    "ax.set_title('tanh function')\n",
+    "\n",
+    "plt.show()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Two parameters\n",
+    "\n",
+    "We assume now that we have two classes with $y_i$ either $0$ or $1$. Furthermore we assume also that we have only two parameters $\\beta$ in our fitting of the Sigmoid function, that is we define probabilities"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "$$\n",
+    "\\begin{align*}\n",
+    "p(y_i=1|x_i,\\hat{\\beta}) &= \\frac{\\exp{(\\beta_0+\\beta_1x_i)}}{1+\\exp{(\\beta_0+\\beta_1x_i)}},\\nonumber\\\\\n",
+    "p(y_i=0|x_i,\\hat{\\beta}) &= 1 - p(y_i=1|x_i,\\hat{\\beta}),\n",
+    "\\end{align*}\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "where $\\hat{\\beta}$ are the weights we wish to extract from data, in our case $\\beta_0$ and $\\beta_1$. \n",
+    "\n",
+    "Note that we used"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "$$\n",
+    "p(y_i=0\\vert x_i, \\hat{\\beta}) = 1-p(y_i=1\\vert x_i, \\hat{\\beta}).\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Maximum likelihood\n",
+    "\n",
+    "In order to define the total likelihood for all possible outcomes from a  \n",
+    "dataset $\\mathcal{D}=\\{(y_i,x_i)\\}$, with the binary labels\n",
+    "$y_i\\in\\{0,1\\}$ and where the data points are drawn independently, we use the so-called [Maximum Likelihood Estimation](https://en.wikipedia.org/wiki/Maximum_likelihood_estimation) (MLE) principle. \n",
+    "We aim thus at maximizing \n",
+    "the probability of seeing the observed data. We can then approximate the \n",
+    "likelihood in terms of the product of the individual probabilities of a specific outcome $y_i$, that is"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "$$\n",
+    "\\begin{align*}\n",
+    "P(\\mathcal{D}|\\hat{\\beta})& = \\prod_{i=1}^n \\left[p(y_i=1|x_i,\\hat{\\beta})\\right]^{y_i}\\left[1-p(y_i=1|x_i,\\hat{\\beta}))\\right]^{1-y_i}\\nonumber \\\\\n",
+    "\\end{align*}\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "from which we obtain the log-likelihood and our **cost/loss** function"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "$$\n",
+    "\\mathcal{C}(\\hat{\\beta}) = \\sum_{i=1}^n \\left( y_i\\log{p(y_i=1|x_i,\\hat{\\beta})} + (1-y_i)\\log\\left[1-p(y_i=1|x_i,\\hat{\\beta}))\\right]\\right).\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Reordering the logarithms, we can rewrite the **cost/loss** function as"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "$$\n",
+    "\\mathcal{C}(\\hat{\\beta}) = \\sum_{i=1}^n  \\left(y_i(\\beta_0+\\beta_1x_i) -\\log{(1+\\exp{(\\beta_0+\\beta_1x_i)})}\\right).\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "The maximum likelihood estimator is defined as the set of parameters that maximize the log-likelihood where we maximize with respect to $\\beta$.\n",
+    "Since the cost (error) function is just the negative log-likelihood, for logistic regression we have that"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "$$\n",
+    "\\mathcal{C}(\\hat{\\beta})=-\\sum_{i=1}^n  \\left(y_i(\\beta_0+\\beta_1x_i) -\\log{(1+\\exp{(\\beta_0+\\beta_1x_i)})}\\right).\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "This equation is known in statistics as the **cross entropy**. Finally, we note that just as in linear regression, \n",
+    "in practice we often supplement the cross-entropy with additional regularization terms, usually $L_1$ and $L_2$ regularization as we did for Ridge and Lasso regression.\n",
+    "\n",
+    "\n",
+    "The cross entropy is a convex function of the weights $\\hat{\\beta}$ and,\n",
+    "therefore, any local minimizer is a global minimizer. \n",
+    "\n",
+    "\n",
+    "Minimizing this\n",
+    "cost function with respect to the two parameters $\\beta_0$ and $\\beta_1$ we obtain"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "$$\n",
+    "\\frac{\\partial \\mathcal{C}(\\hat{\\beta})}{\\partial \\beta_0} = -\\sum_{i=1}^n  \\left(y_i -\\frac{\\exp{(\\beta_0+\\beta_1x_i)}}{1+\\exp{(\\beta_0+\\beta_1x_i)}}\\right),\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "and"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "$$\n",
+    "\\frac{\\partial \\mathcal{C}(\\hat{\\beta})}{\\partial \\beta_1} = -\\sum_{i=1}^n  \\left(y_ix_i -x_i\\frac{\\exp{(\\beta_0+\\beta_1x_i)}}{1+\\exp{(\\beta_0+\\beta_1x_i)}}\\right).\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Let us now define a vector $\\hat{y}$ with $n$ elements $y_i$, an\n",
+    "$n\\times p$ matrix $\\hat{X}$ which contains the $x_i$ values and a\n",
+    "vector $\\hat{p}$ of fitted probabilities $p(y_i\\vert x_i,\\hat{\\beta})$. We can rewrite in a more compact form the first\n",
+    "derivative of cost function as"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "$$\n",
+    "\\frac{\\partial \\mathcal{C}(\\hat{\\beta})}{\\partial \\hat{\\beta}} = -\\hat{X}^T\\left(\\hat{y}-\\hat{p}\\right).\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "If we in addition define a diagonal matrix $\\hat{W}$ with elements \n",
+    "$p(y_i\\vert x_i,\\hat{\\beta})(1-p(y_i\\vert x_i,\\hat{\\beta})$, we can obtain a compact expression of the second derivative as"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "$$\n",
+    "\\frac{\\partial^2 \\mathcal{C}(\\hat{\\beta})}{\\partial \\hat{\\beta}\\partial \\hat{\\beta}^T} = \\hat{X}^T\\hat{W}\\hat{X}.\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Within a binary classification problem, we can easily expand our model to include multiple predictors. Our ratio between likelihoods is then with $p$ predictors"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "$$\n",
+    "\\log{ \\frac{p(\\hat{\\beta}\\hat{x})}{1-p(\\hat{\\beta}\\hat{x})}} = \\beta_0+\\beta_1x_1+\\beta_2x_2+\\dots+\\beta_px_p.\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Here we defined $\\hat{x}=[1,x_1,x_2,\\dots,x_p]$ and $\\hat{\\beta}=[\\beta_0, \\beta_1, \\dots, \\beta_p]$ leading to"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "$$\n",
+    "p(\\hat{\\beta}\\hat{x})=\\frac{ \\exp{(\\beta_0+\\beta_1x_1+\\beta_2x_2+\\dots+\\beta_px_p)}}{1+\\exp{(\\beta_0+\\beta_1x_1+\\beta_2x_2+\\dots+\\beta_px_p)}}.\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Including more classes\n",
+    "\n",
+    "Till now we have mainly focused on two classes, the so-called binary\n",
+    "system. Suppose we wish to extend to $K$ classes.  Let us for the sake\n",
+    "of simplicity assume we have only two predictors. We have then\n",
+    "following model"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "1\n",
+    "5\n",
+    " \n",
+    "<\n",
+    "<\n",
+    "<\n",
+    "!\n",
+    "!\n",
+    "M\n",
+    "A\n",
+    "T\n",
+    "H\n",
+    "_\n",
+    "B\n",
+    "L\n",
+    "O\n",
+    "C\n",
+    "K"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "$$\n",
+    "\\log{\\frac{p(C=2\\vert x)}{p(K\\vert x)}} = \\beta_{20}+\\beta_{21}x_1,\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "and so on till the class $C=K-1$ class"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "$$\n",
+    "\\log{\\frac{p(C=K-1\\vert x)}{p(K\\vert x)}} = \\beta_{(K-1)0}+\\beta_{(K-1)1}x_1,\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "and the model is specified in term of $K-1$ so-called log-odds or\n",
+    "**logit** transformations.\n",
+    "\n",
+    "\n",
+    "\n",
+    "In our discussion of neural networks we will encounter the above again\n",
+    "in terms of a slightly modified function, the so-called **Softmax** function.\n",
+    "\n",
+    "The softmax function is used in various multiclass classification\n",
+    "methods, such as multinomial logistic regression (also known as\n",
+    "softmax regression), multiclass linear discriminant analysis, naive\n",
+    "Bayes classifiers, and artificial neural networks.  Specifically, in\n",
+    "multinomial logistic regression and linear discriminant analysis, the\n",
+    "input to the function is the result of $K$ distinct linear functions,\n",
+    "and the predicted probability for the $k$-th class given a sample\n",
+    "vector $\\hat{x}$ and a weighting vector $\\hat{\\beta}$ is (with two\n",
+    "predictors):"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "$$\n",
+    "p(C=k\\vert \\mathbf {x} )=\\frac{\\exp{(\\beta_{k0}+\\beta_{k1}x_1)}}{1+\\sum_{l=1}^{K-1}\\exp{(\\beta_{l0}+\\beta_{l1}x_1)}}.\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "It is easy to extend to more predictors. The final class is"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "$$\n",
+    "p(C=K\\vert \\mathbf {x} )=\\frac{1}{1+\\sum_{l=1}^{K-1}\\exp{(\\beta_{l0}+\\beta_{l1}x_1)}},\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "and they sum to one. Our earlier discussions were all specialized to\n",
+    "the case with two classes only. It is easy to see from the above that\n",
+    "what we derived earlier is compatible with these equations.\n",
+    "\n",
+    "To find the optimal parameters we would typically use a gradient\n",
+    "descent method.  Newton's method and gradient descent methods are\n",
+    "discussed in the material on [optimization\n",
+    "methods](https://compphysics.github.io/MachineLearning/doc/pub/Splines/html/Splines-bs.html)."
+   ]
+  }
+ ],
+ "metadata": {},
+ "nbformat": 4,
+ "nbformat_minor": 4
+}

doc/src/LectureNotes/_build/html/_sources/testbook/chapter1.ipynb

@@ -0,0 +1,224 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Introduction\n",
+    "\n",
+    "Classical mechanics is a topic which has been taught intensively over\n",
+    "several centuries. It is, with its many variants and ways of\n",
+    "presenting the educational material, normally the first **real** physics\n",
+    "course many of us meet and it lays the foundation for further physics\n",
+    "studies. Many of the equations and ways of reasoning about the\n",
+    "underlying laws of motion and pertinent forces, shape our approaches and understanding\n",
+    "of the scientific method and discourse, as well as the way we develop our insights\n",
+    "and deeper understanding about physical systems.  \n",
+    "\n",
+    "There is a wealth of\n",
+    "well-tested (from both a physics point of view and a pedagogical\n",
+    "standpoint) exercises and problems which can be solved\n",
+    "analytically. However, many of these problems represent idealized and\n",
+    "less realistic situations.  The large majority of these problems are\n",
+    "solved by paper and pencil and are traditionally aimed\n",
+    "at what we normally refer to as continuous models from which we may find an analytical solution.  As a consequence,\n",
+    "when teaching mechanics, it implies that we can seldomly venture beyond an idealized case\n",
+    "in order to develop our understandings and insights about the\n",
+    "underlying forces and laws of motion.\n",
+    "\n",
+    "\n",
+    "On the other hand, numerical algorithms call for approximate discrete\n",
+    "models and much of the development of methods for continuous models\n",
+    "are nowadays being replaced by methods for discrete models in science and\n",
+    "industry, simply because **much larger classes of problems can be addressed** with discrete models, often  by simpler and more\n",
+    "generic methodologies.\n",
+    "\n",
+    "As we will see below, when properly scaling the equations at hand,\n",
+    "discrete models open up for more advanced abstractions and the possibility to\n",
+    "study  real life systems, with the added bonus that we can explore and\n",
+    "deepen our basic understanding of various physical systems\n",
+    "\n",
+    "Analytical solutions are as important as before. In addition, such\n",
+    "solutions provide us with invaluable benchmarks and tests for our\n",
+    "discrete models. Such benchmarks, as we will see below, allow us \n",
+    "to discuss possible sources of errors and their behaviors.  And\n",
+    "finally, since most of our models are based on various algorithms from\n",
+    "numerical mathematics, we have a unique oppotunity to gain a deeper\n",
+    "understanding of the mathematical approaches we are using.\n",
+    "\n",
+    "\n",
+    "\n",
+    "With computing and data science as important elements in essentially\n",
+    "all aspects of a modern society, we could  then try to define Computing as\n",
+    "**solving scientific problems using all possible tools, including\n",
+    "symbolic computing, computers and numerical algorithms, and analytical\n",
+    "paper and pencil solutions**. \n",
+    "Computing provides us with the tools to develope our own understanding of the scientific method by enhancing algorithmic thinking.\n",
+    "\n",
+    "\n",
+    "The way we will teach this course reflects\n",
+    "this definition of computing. The course contains both classical paper\n",
+    "and pencil exercises as well as  computational projects and exercises. The\n",
+    "hope is that this will allow you to explore the physics of systems\n",
+    "governed by the degrees of freedom of classical mechanics at a deeper\n",
+    "level, and that these insights about the scientific method will help\n",
+    "you to develop a better understanding of how the underlying forces and\n",
+    "equations of motion and how they impact a given system. Furthermore, by introducing various numerical methods\n",
+    "via computational projects and exercises, we aim at developing your competences and skills about these topics.\n",
+    "\n",
+    "\n",
+    "These competences will enable you to\n",
+    "\n",
+    "* understand how algorithms are used to solve mathematical problems,\n",
+    "\n",
+    "* derive, verify, and implement algorithms,\n",
+    "\n",
+    "* understand what can go wrong with algorithms,\n",
+    "\n",
+    "* use these algorithms to construct reproducible scientific outcomes and to engage in science in ethical ways, and\n",
+    "\n",
+    "* think algorithmically for the purposes of gaining deeper insights about scientific problems.\n",
+    "\n",
+    "All these elements are central for maturing and gaining a better understanding of the modern scientific process *per se*.\n",
+    "\n",
+    "The power of the scientific method lies in identifying a given problem\n",
+    "as a special case of an abstract class of problems, identifying\n",
+    "general solution methods for this class of problems, and applying a\n",
+    "general method to the specific problem (applying means, in the case of\n",
+    "computing, calculations by pen and paper, symbolic computing, or\n",
+    "numerical computing by ready-made and/or self-written software). This\n",
+    "generic view on problems and methods is particularly important for\n",
+    "understanding how to apply available, generic software to solve a\n",
+    "particular problem.\n",
+    "\n",
+    "*However, verification of algorithms and understanding their limitations requires much of the classical knowledge about continuous models.*\n",
+    "\n",
+    "\n",
+    "\n",
+    "## A well-known examples to illustrate many of the above concepts\n",
+    "\n",
+    "Before we venture into a reminder on Python and mechanics relevant applications, let us briefly outline some of the\n",
+    "abovementioned topics using an example many of you may have seen before in for example CMSE201. \n",
+    "A simple algorithm for integration is the Trapezoidal rule. \n",
+    "Integration of a function $f(x)$ by the Trapezoidal Rule is given by following algorithm for an interval $x \\in [a,b]$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "$$\n",
+    "\\int_a^b(f(x) dx = \\frac{1}{2}\\left [f(a)+2f(a+h)+\\dots+2f(b-h)+f(b)\\right] +O(h^2),\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "where $h$ is the so-called stepsize defined by the number of integration points $N$ as $h=(b-a)/(n)$.\n",
+    "Python offers an  extremely versatile programming  environment, allowing for\n",
+    "the inclusion of analytical studies in a numerical program. Here we show an\n",
+    "example code with the **trapezoidal rule**. We use also **SymPy** to evaluate the exact value of the integral and compute the absolute error\n",
+    "with respect to the numerically evaluated one of the integral\n",
+    "$\\int_0^1 dx x^2 = 1/3$.\n",
+    "The following code for  the trapezoidal rule allows you  to plot the relative error by comparing with the exact result. By increasing to $10^8$ points one arrives at a region where numerical errors start to accumulate."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "metadata": {
+    "collapsed": false
+   },
+   "outputs": [],
+   "source": [
+    "%matplotlib inline\n",
+    "\n",
+    "from math import log10\n",
+    "import numpy as np\n",
+    "from sympy import Symbol, integrate\n",
+    "import matplotlib.pyplot as plt\n",
+    "# function for the trapezoidal rule\n",
+    "def Trapez(a,b,f,n):\n",
+    "   h = (b-a)/float(n)\n",
+    "   s = 0\n",
+    "   x = a\n",
+    "   for i in range(1,n,1):\n",
+    "       x = x+h\n",
+    "       s = s+ f(x)\n",
+    "   s = 0.5*(f(a)+f(b)) +s\n",
+    "   return h*s\n",
+    "#  function to compute pi\n",
+    "def function(x):\n",
+    "    return x*x\n",
+    "# define integration limits\n",
+    "a = 0.0;  b = 1.0;\n",
+    "# find result from sympy\n",
+    "# define x as a symbol to be used by sympy\n",
+    "x = Symbol('x')\n",
+    "exact = integrate(function(x), (x, a, b))\n",
+    "# set up the arrays for plotting the relative error\n",
+    "n = np.zeros(9); y = np.zeros(9);\n",
+    "# find the relative error as function of integration points\n",
+    "for i in range(1, 8, 1):\n",
+    "    npts = 10**i\n",
+    "    result = Trapez(a,b,function,npts)\n",
+    "    RelativeError = abs((exact-result)/exact)\n",
+    "    n[i] = log10(npts); y[i] = log10(RelativeError);\n",
+    "plt.plot(n,y, 'ro')\n",
+    "plt.xlabel('n')\n",
+    "plt.ylabel('Relative error')\n",
+    "plt.show()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "This example shows the potential of combining numerical algorithms with symbolic calculations, allowing us to \n",
+    "\n",
+    "* Validate and verify  their  algorithms. \n",
+    "\n",
+    "* Including concepts like unit testing, one has the possibility to test and test several or all parts of the code.\n",
+    "\n",
+    "* Validation and verification are then included *naturally* and one can develop a better attitude to what is meant with an ethically sound scientific approach.\n",
+    "\n",
+    "* The above example allows the student to also test the mathematical error of the algorithm for the trapezoidal rule by changing the number of integration points. The students get **trained from day one to think error analysis**. \n",
+    "\n",
+    "* With a Jupyter notebook you can keep exploring similar examples and turn them in as your own notebooks. \n",
+    "\n",
+    "In this process we can easily bake in\n",
+    "1. How to structure a code in terms of functions\n",
+    "\n",
+    "2. How to make a module\n",
+    "\n",
+    "3. How to read input data flexibly from the command line\n",
+    "\n",
+    "4. How to create graphical/web user interfaces\n",
+    "\n",
+    "5. How to write unit tests (test functions or doctests)\n",
+    "\n",
+    "6. How to refactor code in terms of classes (instead of functions only)\n",
+    "\n",
+    "7. How to conduct and automate large-scale numerical experiments\n",
+    "\n",
+    "8. How to write scientific reports in various formats (LaTeX, HTML)\n",
+    "\n",
+    "The conventions and techniques outlined here will save you a lot of time when you incrementally extend software over time from simpler to more complicated problems. In particular, you will benefit from many good habits:\n",
+    "1. New code is added in a modular fashion to a library (modules)\n",
+    "\n",
+    "2. Programs are run through convenient user interfaces\n",
+    "\n",
+    "3. It takes one quick command to let all your code undergo heavy testing \n",
+    "\n",
+    "4. Tedious manual work with running programs is automated,\n",
+    "\n",
+    "5. Your scientific investigations are reproducible, scientific reports with top quality typesetting are produced both for paper and electronic devices."
+   ]
+  }
+ ],
+ "metadata": {},
+ "nbformat": 4,
+ "nbformat_minor": 4
+}