diff --git a/doc/HandWrittenNotes/2022/NotesOct132022.pdf b/doc/HandWrittenNotes/2022/NotesOct132022.pdf
new file mode 100644
index 000000000..bace8dee9
Binary files /dev/null and b/doc/HandWrittenNotes/2022/NotesOct132022.pdf differ
diff --git a/doc/pub/week41/ipynb/week41.ipynb b/doc/pub/week41/ipynb/week41.ipynb
index 9da8ff5c1..da86b2233 100644
--- a/doc/pub/week41/ipynb/week41.ipynb
+++ b/doc/pub/week41/ipynb/week41.ipynb
@@ -3,9 +3,7 @@
{
"cell_type": "markdown",
"id": "975262ec",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"\n",
@@ -15,14 +13,12 @@
{
"cell_type": "markdown",
"id": "b0eae28d",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"# Week 41 Constructing a Neural Network code, Tensor flow and start Convolutional Neural Networks\n",
"**Morten Hjorth-Jensen**, Department of Physics, University of Oslo and Department of Physics and Astronomy and Facility for Rare Isotope Beams, Michigan State University\n",
"\n",
- "Date: **Oct 15, 2022**\n",
+ "Date: **Oct 13, 2022**\n",
"\n",
"Copyright 1999-2022, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license"
]
@@ -30,9 +26,7 @@
{
"cell_type": "markdown",
"id": "4f5a6dcb",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Plan for week 41\n",
"\n",
@@ -47,9 +41,7 @@
{
"cell_type": "markdown",
"id": "50f2936a",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Videos on Neural Networks\n",
"\n",
@@ -61,9 +53,7 @@
{
"cell_type": "markdown",
"id": "b827ad7e",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Review of the back propagation algorithm\n",
"\n",
@@ -76,9 +66,7 @@
{
"cell_type": "markdown",
"id": "806dffca",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Setting up the Back propagation algorithm\n",
"\n",
@@ -99,9 +87,7 @@
{
"cell_type": "markdown",
"id": "859b5d78",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"\\delta_j^L = f'(z_j^L)\\frac{\\partial {\\cal C}}{\\partial (a_j^L)}.\n",
@@ -111,9 +97,7 @@
{
"cell_type": "markdown",
"id": "753d8335",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"Then we compute the back propagate error for each $l=L-1,L-2,\\dots,2$ as"
]
@@ -121,9 +105,7 @@
{
"cell_type": "markdown",
"id": "9f6d7e0b",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"\\delta_j^l = \\sum_k \\delta_k^{l+1}w_{kj}^{l+1}f'(z_j^l).\n",
@@ -133,9 +115,7 @@
{
"cell_type": "markdown",
"id": "e0d5108f",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"Finally, we update the weights and the biases using gradient descent for each $l=L-1,L-2,\\dots,2$ and update the weights and biases according to the rules"
]
@@ -143,9 +123,7 @@
{
"cell_type": "markdown",
"id": "4e72fbea",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"w_{jk}^l\\leftarrow = w_{jk}^l- \\eta \\delta_j^la_k^{l-1},\n",
@@ -155,9 +133,7 @@
{
"cell_type": "markdown",
"id": "0602f4b2",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"b_j^l \\leftarrow b_j^l-\\eta \\frac{\\partial {\\cal C}}{\\partial b_j^l}=b_j^l-\\eta \\delta_j^l,\n",
@@ -167,9 +143,7 @@
{
"cell_type": "markdown",
"id": "7bf854a6",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"The parameter $\\eta$ is the learning parameter discussed in connection with the gradient descent methods.\n",
"Here it is convenient to use stochastic gradient descent (see the examples below) with mini-batches with an outer loop that steps through multiple epochs of training."
@@ -178,9 +152,7 @@
{
"cell_type": "markdown",
"id": "7c507f5b",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Setting up a Multi-layer perceptron model for classification\n",
"\n",
@@ -206,9 +178,7 @@
{
"cell_type": "markdown",
"id": "fd1e66c8",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"P(y = 0 \\mid \\boldsymbol{x}, \\boldsymbol{\\theta}) = \\frac{1}{1 + \\exp{(- \\boldsymbol{x}})} ,\n",
@@ -218,9 +188,7 @@
{
"cell_type": "markdown",
"id": "14a03523",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"and"
]
@@ -228,9 +196,7 @@
{
"cell_type": "markdown",
"id": "803c6861",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"P(y = 1 \\mid \\boldsymbol{x}, \\boldsymbol{\\theta}) = 1 - P(y = 0 \\mid \\boldsymbol{x}, \\boldsymbol{\\theta}) ,\n",
@@ -240,9 +206,7 @@
{
"cell_type": "markdown",
"id": "26a7a40f",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"where $y \\in \\{0, 1\\}$ and $\\boldsymbol{\\theta}$ represents the weights and biases\n",
"of our network."
@@ -251,9 +215,7 @@
{
"cell_type": "markdown",
"id": "33f03037",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Defining the cost function\n",
"\n",
@@ -263,9 +225,7 @@
{
"cell_type": "markdown",
"id": "83904654",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"\\mathcal{C}(\\boldsymbol{\\theta}) = - \\ln P(\\mathcal{D} \\mid \\boldsymbol{\\theta}) = - \\sum_{i=1}^n\n",
@@ -276,9 +236,7 @@
{
"cell_type": "markdown",
"id": "c612a009",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"This last equality means that we can interpret our *cost* function as a sum over the *loss* function\n",
"for each point in the dataset $\\mathcal{L}_i(\\boldsymbol{\\theta})$. \n",
@@ -301,9 +259,7 @@
{
"cell_type": "markdown",
"id": "4073d973",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"P(y_{ic} = 1 \\mid \\boldsymbol{x}_i, \\boldsymbol{\\theta}) = \\frac{\\exp{((\\boldsymbol{a}_i^{hidden})^T \\boldsymbol{w}_c)}}\n",
@@ -314,9 +270,7 @@
{
"cell_type": "markdown",
"id": "d077f76c",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"which reduces to the logistic function in the binary case. \n",
"The likelihood of this $C$-class classifier\n",
@@ -326,9 +280,7 @@
{
"cell_type": "markdown",
"id": "3354ad8c",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"P(\\mathcal{D} \\mid \\boldsymbol{\\theta}) = \\prod_{i=1}^n \\prod_{c=0}^{C-1} [P(y_{ic} = 1)]^{y_{ic}} .\n",
@@ -338,9 +290,7 @@
{
"cell_type": "markdown",
"id": "96a8ea69",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"Again we take the negative log-likelihood to define our cost function:"
]
@@ -348,9 +298,7 @@
{
"cell_type": "markdown",
"id": "3886730c",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"\\mathcal{C}(\\boldsymbol{\\theta}) = - \\log{P(\\mathcal{D} \\mid \\boldsymbol{\\theta})}.\n",
@@ -360,9 +308,7 @@
{
"cell_type": "markdown",
"id": "1ff35269",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"See the logistic regression lectures for a full definition of the cost function.\n",
"\n",
@@ -372,9 +318,7 @@
{
"cell_type": "markdown",
"id": "8648d883",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Example: binary classification problem\n",
"\n",
@@ -384,9 +328,7 @@
{
"cell_type": "markdown",
"id": "e21bbc09",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"\\mathcal{C}(\\boldsymbol{\\beta}) = - \\sum_{i=1}^n \\left(y_i\\log{p(y_i \\vert x_i,\\boldsymbol{\\beta})}+(1-y_i)\\log{1-p(y_i \\vert x_i,\\boldsymbol{\\beta})}\\right),\n",
@@ -396,9 +338,7 @@
{
"cell_type": "markdown",
"id": "3994187a",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"where we had defined the logistic (sigmoid) function"
]
@@ -406,9 +346,7 @@
{
"cell_type": "markdown",
"id": "9c5ae70c",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"p(y_i =1\\vert x_i,\\boldsymbol{\\beta})=\\frac{\\exp{(\\beta_0+\\beta_1 x_i)}}{1+\\exp{(\\beta_0+\\beta_1 x_i)}},\n",
@@ -418,9 +356,7 @@
{
"cell_type": "markdown",
"id": "b8cb3324",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"and"
]
@@ -428,9 +364,7 @@
{
"cell_type": "markdown",
"id": "eb9d0b98",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"p(y_i =0\\vert x_i,\\boldsymbol{\\beta})=1-p(y_i =1\\vert x_i,\\boldsymbol{\\beta}).\n",
@@ -440,9 +374,7 @@
{
"cell_type": "markdown",
"id": "e7e4f6cc",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"The parameters $\\boldsymbol{\\beta}$ were defined using a minimization method like gradient descent or Newton-Raphson's method. \n",
"\n",
@@ -453,9 +385,7 @@
{
"cell_type": "markdown",
"id": "60a0dea9",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"a_i^l = y_i = \\frac{\\exp{(z_i^l)}}{1+\\exp{(z_i^l)}},\n",
@@ -465,9 +395,7 @@
{
"cell_type": "markdown",
"id": "ff9153db",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"with"
]
@@ -475,9 +403,7 @@
{
"cell_type": "markdown",
"id": "937e0b88",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"z_i^l = \\sum_{j}w_{ij}^l a_j^{l-1}+b_i^l,\n",
@@ -487,9 +413,7 @@
{
"cell_type": "markdown",
"id": "d512ac32",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"where the superscript $l-1$ indicates that these are the outputs from layer $l-1$.\n",
"Our cost function at the final layer $l=L$ is now"
@@ -498,9 +422,7 @@
{
"cell_type": "markdown",
"id": "0116ab23",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"\\mathcal{C}(\\boldsymbol{W}) = - \\sum_{i=1}^n \\left(t_i\\log{a_i^L}+(1-t_i)\\log{(1-a_i^L)}\\right),\n",
@@ -510,9 +432,7 @@
{
"cell_type": "markdown",
"id": "68f7ca43",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"where we have defined the targets $t_i$. The derivatives of the cost function with respect to the output $a_i^L$ are then easily calculated and we get"
]
@@ -520,9 +440,7 @@
{
"cell_type": "markdown",
"id": "44352675",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"\\frac{\\partial \\mathcal{C}(\\boldsymbol{W})}{\\partial a_i^L} = \\frac{a_i^L-t_i}{a_i^L(1-a_i^L)}.\n",
@@ -532,9 +450,7 @@
{
"cell_type": "markdown",
"id": "c8b434c3",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"In case we use another activation function than the logistic one, we need to evaluate other derivatives."
]
@@ -542,9 +458,7 @@
{
"cell_type": "markdown",
"id": "28c8b238",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## The Softmax function\n",
"In case we employ the more general case given by the Softmax equation, we need to evaluate the derivative of the activation function with respect to the activation $z_i^l$, that is we need"
@@ -553,9 +467,7 @@
{
"cell_type": "markdown",
"id": "634a0119",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"\\frac{\\partial f(z_i^l)}{\\partial w_{jk}^l} =\n",
@@ -566,9 +478,7 @@
{
"cell_type": "markdown",
"id": "fe735cbf",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"For the Softmax function we have"
]
@@ -576,9 +486,7 @@
{
"cell_type": "markdown",
"id": "9e7c13c4",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"f(z_i^l) = \\frac{\\exp{(z_i^l)}}{\\sum_{m=1}^K\\exp{(z_m^l)}}.\n",
@@ -588,9 +496,7 @@
{
"cell_type": "markdown",
"id": "92f3fd45",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"Its derivative with respect to $z_j^l$ gives"
]
@@ -598,9 +504,7 @@
{
"cell_type": "markdown",
"id": "b471e4a9",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"\\frac{\\partial f(z_i^l)}{\\partial z_j^l}= f(z_i^l)\\left(\\delta_{ij}-f(z_j^l)\\right),\n",
@@ -610,9 +514,7 @@
{
"cell_type": "markdown",
"id": "80487840",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"which in case of the simply binary model reduces to having $i=j$."
]
@@ -620,9 +522,7 @@
{
"cell_type": "markdown",
"id": "f6d114e4",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Developing a code for doing neural networks with back propagation\n",
"\n",
@@ -644,9 +544,7 @@
{
"cell_type": "markdown",
"id": "03e211c1",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Collect and pre-process data\n",
"\n",
@@ -694,10 +592,7 @@
"cell_type": "code",
"execution_count": 1,
"id": "5f127181",
- "metadata": {
- "collapsed": false,
- "editable": true
- },
+ "metadata": {},
"outputs": [],
"source": [
"%matplotlib inline\n",
@@ -749,9 +644,7 @@
{
"cell_type": "markdown",
"id": "025321f0",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Train and test datasets\n",
"\n",
@@ -770,10 +663,7 @@
"cell_type": "code",
"execution_count": 2,
"id": "edd98947",
- "metadata": {
- "collapsed": false,
- "editable": true
- },
+ "metadata": {},
"outputs": [],
"source": [
"from sklearn.model_selection import train_test_split\n",
@@ -808,9 +698,7 @@
{
"cell_type": "markdown",
"id": "1bad86a8",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Define model and architecture\n",
"\n",
@@ -852,9 +740,7 @@
{
"cell_type": "markdown",
"id": "37094b97",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Layers\n",
"\n",
@@ -892,9 +778,7 @@
{
"cell_type": "markdown",
"id": "fb1d2b51",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Weights and biases\n",
"\n",
@@ -913,10 +797,7 @@
"cell_type": "code",
"execution_count": 3,
"id": "aa2cabf9",
- "metadata": {
- "collapsed": false,
- "editable": true
- },
+ "metadata": {},
"outputs": [],
"source": [
"# building our neural network\n",
@@ -939,9 +820,7 @@
{
"cell_type": "markdown",
"id": "3bea3936",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Feed-forward pass\n",
"\n",
@@ -967,9 +846,7 @@
{
"cell_type": "markdown",
"id": "af85b0df",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Matrix multiplications\n",
"\n",
@@ -1004,10 +881,7 @@
"cell_type": "code",
"execution_count": 4,
"id": "16a29f53",
- "metadata": {
- "collapsed": false,
- "editable": true
- },
+ "metadata": {},
"outputs": [],
"source": [
"# setup the feed-forward pass, subscript h = hidden layer\n",
@@ -1050,9 +924,7 @@
{
"cell_type": "markdown",
"id": "084ed7ba",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Choose cost function and optimizer\n",
"\n",
@@ -1081,9 +953,7 @@
{
"cell_type": "markdown",
"id": "d70e21d8",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Optimizing the cost function\n",
"\n",
@@ -1119,9 +989,7 @@
{
"cell_type": "markdown",
"id": "23940327",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Regularization\n",
"\n",
@@ -1153,9 +1021,7 @@
{
"cell_type": "markdown",
"id": "a7beed07",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Matrix multiplication\n",
"\n",
@@ -1194,10 +1060,7 @@
"cell_type": "code",
"execution_count": 5,
"id": "36ca3c90",
- "metadata": {
- "collapsed": false,
- "editable": true
- },
+ "metadata": {},
"outputs": [],
"source": [
"# to categorical turns our integer vector into a onehot representation\n",
@@ -1273,9 +1136,7 @@
{
"cell_type": "markdown",
"id": "7e0eac4c",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Improving performance\n",
"\n",
@@ -1294,9 +1155,7 @@
{
"cell_type": "markdown",
"id": "57e98ccc",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Full object-oriented implementation\n",
"\n",
@@ -1308,10 +1167,7 @@
"cell_type": "code",
"execution_count": 6,
"id": "158f9158",
- "metadata": {
- "collapsed": false,
- "editable": true
- },
+ "metadata": {},
"outputs": [],
"source": [
"class NeuralNetwork:\n",
@@ -1418,9 +1274,7 @@
{
"cell_type": "markdown",
"id": "b8d26ba8",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Evaluate model performance on test data\n",
"\n",
@@ -1437,10 +1291,7 @@
"cell_type": "code",
"execution_count": 7,
"id": "e32c4a43",
- "metadata": {
- "collapsed": false,
- "editable": true
- },
+ "metadata": {},
"outputs": [],
"source": [
"epochs = 100\n",
@@ -1464,9 +1315,7 @@
{
"cell_type": "markdown",
"id": "aef3f023",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Adjust hyperparameters\n",
"\n",
@@ -1478,10 +1327,7 @@
"cell_type": "code",
"execution_count": 8,
"id": "f0748a05",
- "metadata": {
- "collapsed": false,
- "editable": true
- },
+ "metadata": {},
"outputs": [],
"source": [
"eta_vals = np.logspace(-5, 1, 7)\n",
@@ -1509,9 +1355,7 @@
{
"cell_type": "markdown",
"id": "8c7200d4",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Visualization"
]
@@ -1520,10 +1364,7 @@
"cell_type": "code",
"execution_count": 9,
"id": "78b7dc25",
- "metadata": {
- "collapsed": false,
- "editable": true
- },
+ "metadata": {},
"outputs": [],
"source": [
"# visual representation of grid search\n",
@@ -1564,9 +1405,7 @@
{
"cell_type": "markdown",
"id": "af09471a",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## scikit-learn implementation\n",
"\n",
@@ -1587,10 +1426,7 @@
"cell_type": "code",
"execution_count": 10,
"id": "481e8a94",
- "metadata": {
- "collapsed": false,
- "editable": true
- },
+ "metadata": {},
"outputs": [],
"source": [
"from sklearn.neural_network import MLPClassifier\n",
@@ -1614,9 +1450,7 @@
{
"cell_type": "markdown",
"id": "b08a0123",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Visualization"
]
@@ -1625,10 +1459,7 @@
"cell_type": "code",
"execution_count": 11,
"id": "2177962e",
- "metadata": {
- "collapsed": false,
- "editable": true
- },
+ "metadata": {},
"outputs": [],
"source": [
"# optional\n",
@@ -1670,9 +1501,7 @@
{
"cell_type": "markdown",
"id": "b8f24e48",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Testing our code for the XOR, OR and AND gates\n",
"\n",
@@ -1697,9 +1526,7 @@
{
"cell_type": "markdown",
"id": "383d7ed1",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## The AND and XOR Gates\n",
"\n",
@@ -1735,9 +1562,7 @@
{
"cell_type": "markdown",
"id": "97373e7e",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Representing the Data Sets\n",
"\n",
@@ -1747,9 +1572,7 @@
{
"cell_type": "markdown",
"id": "bcff4010",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"\\boldsymbol{X}=\\begin{bmatrix} 0 & 0 \\\\\n",
@@ -1762,9 +1585,7 @@
{
"cell_type": "markdown",
"id": "00f83b12",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"while the vector of outputs is $\\boldsymbol{y}^T=[0,1,1,0]$ for the XOR gate, $\\boldsymbol{y}^T=[0,0,0,1]$ for the AND gate and $\\boldsymbol{y}^T=[0,1,1,1]$ for the OR gate."
]
@@ -1772,9 +1593,7 @@
{
"cell_type": "markdown",
"id": "cf1d8a68",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Setting up the Neural Network\n",
"\n",
@@ -1785,10 +1604,7 @@
"cell_type": "code",
"execution_count": 12,
"id": "4dd40dc9",
- "metadata": {
- "collapsed": false,
- "editable": true
- },
+ "metadata": {},
"outputs": [],
"source": [
"\"\"\"\n",
@@ -1861,9 +1677,7 @@
{
"cell_type": "markdown",
"id": "7d57a51a",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"Not an impressive result, but this was our first forward pass with randomly assigned weights. Let us now add the full network with the back-propagation algorithm discussed above."
]
@@ -1871,9 +1685,7 @@
{
"cell_type": "markdown",
"id": "339de45f",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## The Code using Scikit-Learn"
]
@@ -1882,10 +1694,7 @@
"cell_type": "code",
"execution_count": 13,
"id": "a70e4f4f",
- "metadata": {
- "collapsed": false,
- "editable": true
- },
+ "metadata": {},
"outputs": [],
"source": [
"# import necessary packages\n",
@@ -1950,9 +1759,7 @@
{
"cell_type": "markdown",
"id": "d963bb7a",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Building neural networks in Tensorflow and Keras\n",
"\n",
@@ -1968,9 +1775,7 @@
{
"cell_type": "markdown",
"id": "dd4b59e7",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Tensorflow\n",
"\n",
@@ -2003,10 +1808,7 @@
"cell_type": "code",
"execution_count": 14,
"id": "19e6be62",
- "metadata": {
- "collapsed": false,
- "editable": true
- },
+ "metadata": {},
"outputs": [],
"source": [
"pip3 install tensorflow"
@@ -2015,9 +1817,7 @@
{
"cell_type": "markdown",
"id": "e19a1aa7",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"and/or if you use **anaconda**, just write (or install from the graphical user interface)\n",
"(current release of CPU-only TensorFlow)"
@@ -2027,10 +1827,7 @@
"cell_type": "code",
"execution_count": 15,
"id": "ac145627",
- "metadata": {
- "collapsed": false,
- "editable": true
- },
+ "metadata": {},
"outputs": [],
"source": [
"conda create -n tf tensorflow\n",
@@ -2040,9 +1837,7 @@
{
"cell_type": "markdown",
"id": "41d1f0b2",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"To install the current release of GPU TensorFlow"
]
@@ -2051,10 +1846,7 @@
"cell_type": "code",
"execution_count": 16,
"id": "a988d71f",
- "metadata": {
- "collapsed": false,
- "editable": true
- },
+ "metadata": {},
"outputs": [],
"source": [
"conda create -n tf-gpu tensorflow-gpu\n",
@@ -2064,9 +1856,7 @@
{
"cell_type": "markdown",
"id": "02a10b42",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Using Keras\n",
"\n",
@@ -2079,10 +1869,7 @@
"cell_type": "code",
"execution_count": 17,
"id": "a00076bd",
- "metadata": {
- "collapsed": false,
- "editable": true
- },
+ "metadata": {},
"outputs": [],
"source": [
"conda install keras"
@@ -2091,9 +1878,7 @@
{
"cell_type": "markdown",
"id": "efc69dbe",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"You can look up the [instructions here](https://keras.io/) for more information.\n",
"\n",
@@ -2103,9 +1888,7 @@
{
"cell_type": "markdown",
"id": "6d57b48c",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Collect and pre-process data\n",
"\n",
@@ -2116,10 +1899,7 @@
"cell_type": "code",
"execution_count": 18,
"id": "94ee347e",
- "metadata": {
- "collapsed": false,
- "editable": true
- },
+ "metadata": {},
"outputs": [],
"source": [
"# import necessary packages\n",
@@ -2171,10 +1951,7 @@
"cell_type": "code",
"execution_count": 19,
"id": "912152f3",
- "metadata": {
- "collapsed": false,
- "editable": true
- },
+ "metadata": {},
"outputs": [],
"source": [
"from tensorflow.keras.layers import Input\n",
@@ -2200,10 +1977,7 @@
"cell_type": "code",
"execution_count": 20,
"id": "366b21d0",
- "metadata": {
- "collapsed": false,
- "editable": true
- },
+ "metadata": {},
"outputs": [],
"source": [
"\n",
@@ -2230,10 +2004,7 @@
"cell_type": "code",
"execution_count": 21,
"id": "7a1b52ce",
- "metadata": {
- "collapsed": false,
- "editable": true
- },
+ "metadata": {},
"outputs": [],
"source": [
"DNN_keras = np.zeros((len(eta_vals), len(lmbd_vals)), dtype=object)\n",
@@ -2257,10 +2028,7 @@
"cell_type": "code",
"execution_count": 22,
"id": "c9fe1912",
- "metadata": {
- "collapsed": false,
- "editable": true
- },
+ "metadata": {},
"outputs": [],
"source": [
"# optional\n",
@@ -2299,9 +2067,7 @@
{
"cell_type": "markdown",
"id": "212fcfaf",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## The Breast Cancer Data, now with Keras"
]
@@ -2310,10 +2076,7 @@
"cell_type": "code",
"execution_count": 23,
"id": "bece21bd",
- "metadata": {
- "collapsed": false,
- "editable": true
- },
+ "metadata": {},
"outputs": [],
"source": [
"\n",
@@ -2487,9 +2250,7 @@
{
"cell_type": "markdown",
"id": "e02319a1",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## The Mathematics of Neural Networks\n",
"\n",
@@ -2506,9 +2267,7 @@
{
"cell_type": "markdown",
"id": "1db82be9",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Fine-tuning neural network hyperparameters\n",
"\n",
@@ -2534,9 +2293,7 @@
{
"cell_type": "markdown",
"id": "36de6d0b",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Hidden layers\n",
"\n",
@@ -2557,9 +2314,7 @@
{
"cell_type": "markdown",
"id": "87d78436",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Which activation function should I use?\n",
"\n",
@@ -2588,9 +2343,7 @@
{
"cell_type": "markdown",
"id": "dda74efd",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Is the Logistic activation function (Sigmoid) our choice?\n",
"\n",
@@ -2620,9 +2373,7 @@
{
"cell_type": "markdown",
"id": "88908334",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## The derivative of the Logistic funtion\n",
"\n",
@@ -2658,9 +2409,7 @@
{
"cell_type": "markdown",
"id": "35364ee3",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## The RELU function family\n",
"\n",
@@ -2683,9 +2432,7 @@
{
"cell_type": "markdown",
"id": "aceb9e0f",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"ELU(z) = \\left\\{\\begin{array}{cc} \\alpha\\left( \\exp{(z)}-1\\right) & z < 0,\\\\ z & z \\ge 0.\\end{array}\\right.\n",
@@ -2695,9 +2442,7 @@
{
"cell_type": "markdown",
"id": "feb5106f",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Which activation function should we use?\n",
"\n",
@@ -2718,9 +2463,7 @@
{
"cell_type": "markdown",
"id": "c9c02da1",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## More on activation functions, output layers\n",
"\n",
@@ -2738,9 +2481,7 @@
{
"cell_type": "markdown",
"id": "23a19072",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Batch Normalization\n",
"\n",
@@ -2760,9 +2501,7 @@
{
"cell_type": "markdown",
"id": "a5b1214e",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Dropout\n",
"\n",
@@ -2778,9 +2517,7 @@
{
"cell_type": "markdown",
"id": "fc042ae9",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Gradient Clipping\n",
"\n",
@@ -2797,9 +2534,7 @@
{
"cell_type": "markdown",
"id": "959db45e",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## A very nice website on Neural Networks\n",
"\n",
@@ -2809,9 +2544,7 @@
{
"cell_type": "markdown",
"id": "125ce15c",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## A top-down perspective on Neural networks\n",
"\n",
@@ -2854,9 +2587,7 @@
{
"cell_type": "markdown",
"id": "7f4dbb26",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Limitations of supervised learning with deep networks\n",
"\n",
@@ -2884,9 +2615,7 @@
{
"cell_type": "markdown",
"id": "04ac900c",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Overarching Views, a personal note\n",
"\n",
@@ -2914,9 +2643,7 @@
{
"cell_type": "markdown",
"id": "dd5dc10e",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Using Automatic differentiation\n",
"\n",
@@ -2927,9 +2654,7 @@
{
"cell_type": "markdown",
"id": "a5bf7078",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Solving ODEs with Deep Learning\n",
"\n",
@@ -2950,9 +2675,7 @@
{
"cell_type": "markdown",
"id": "82fcf813",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Ordinary Differential Equations\n",
"\n",
@@ -2964,9 +2687,7 @@
{
"cell_type": "markdown",
"id": "65ebf025",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"\n",
"\n",
@@ -2981,9 +2702,7 @@
{
"cell_type": "markdown",
"id": "6cc0bb55",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"where $g(x)$ is the function to find, and $g^{(n)}(x)$ is the $n$-th derivative of $g(x)$.\n",
"\n",
@@ -2997,9 +2716,7 @@
{
"cell_type": "markdown",
"id": "851bc49a",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## The trial solution\n",
"\n",
@@ -3009,9 +2726,7 @@
{
"cell_type": "markdown",
"id": "e6038436",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"\n",
"\n",
@@ -3027,9 +2742,7 @@
{
"cell_type": "markdown",
"id": "f3716b02",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"where $h_1(x)$ is a function that makes $g_t(x)$ satisfy a given set\n",
"of conditions, $N(x,P)$ a neural network with weights and biases\n",
@@ -3048,9 +2761,7 @@
{
"cell_type": "markdown",
"id": "0cf0d338",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Minimization process\n",
"\n",
@@ -3065,9 +2776,7 @@
{
"cell_type": "markdown",
"id": "16e3dbe0",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"C\\left(x, P\\right) = \\big(f\\left(x, \\, g(x), \\, g'(x), \\, g''(x), \\, \\dots \\, , \\, g^{(n)}(x)\\right)\\big)^2\n",
@@ -3077,9 +2786,7 @@
{
"cell_type": "markdown",
"id": "d6033d6b",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"If $N$ inputs are given as a vector $\\boldsymbol{x}$ with elements $x_i$ for $i = 1,\\dots,N$,\n",
"the cost function becomes"
@@ -3088,9 +2795,7 @@
{
"cell_type": "markdown",
"id": "518c5c94",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"\n",
"\n",
@@ -3105,9 +2810,7 @@
{
"cell_type": "markdown",
"id": "3edf1469",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"The neural net should then find the parameters $P$ that minimizes the cost function in\n",
"([3](#cost)) for a set of $N$ training samples $x_i$."
@@ -3116,9 +2819,7 @@
{
"cell_type": "markdown",
"id": "114d4727",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Minimizing the cost function using gradient descent and automatic differentiation\n",
"\n",
@@ -3132,9 +2833,7 @@
{
"cell_type": "markdown",
"id": "67d1893f",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Example: Exponential decay\n",
"\n",
@@ -3144,9 +2843,7 @@
{
"cell_type": "markdown",
"id": "58166a75",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"\n",
"\n",
@@ -3161,9 +2858,7 @@
{
"cell_type": "markdown",
"id": "5c6e35d9",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"with $g(0) = g_0$ for some chosen initial value $g_0$.\n",
"\n",
@@ -3173,9 +2868,7 @@
{
"cell_type": "markdown",
"id": "75205e6f",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"\n",
"\n",
@@ -3191,9 +2884,7 @@
{
"cell_type": "markdown",
"id": "8a5a5f24",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"Having an analytical solution at hand, it is possible to use it to compare how well a neural network finds a solution of ([4](#solve_expdec))."
]
@@ -3201,9 +2892,7 @@
{
"cell_type": "markdown",
"id": "64b1719c",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## The function to solve for\n",
"\n",
@@ -3213,9 +2902,7 @@
{
"cell_type": "markdown",
"id": "c1fefb6f",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"\n",
"\n",
@@ -3230,9 +2917,7 @@
{
"cell_type": "markdown",
"id": "a8cc57e4",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"where $g(0) = g_0$ with $\\gamma$ and $g_0$ being some chosen values.\n",
"\n",
@@ -3242,9 +2927,7 @@
{
"cell_type": "markdown",
"id": "9b500328",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## The trial solution\n",
"To begin with, a trial solution $g_t(t)$ must be chosen. A general trial solution for ordinary differential equations could be"
@@ -3253,9 +2936,7 @@
{
"cell_type": "markdown",
"id": "9b67eee5",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"g_t(x, P) = h_1(x) + h_2(x, N(x, P))\n",
@@ -3265,9 +2946,7 @@
{
"cell_type": "markdown",
"id": "cce6a4e3",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"with $h_1(x)$ ensuring that $g_t(x)$ satisfies some conditions and $h_2(x,N(x, P))$ an expression involving $x$ and the output from the neural network $N(x,P)$ with $P $ being the collection of the weights and biases for each layer. For now, it is assumed that the network consists of one input layer, one hidden layer, and one output layer."
]
@@ -3275,9 +2954,7 @@
{
"cell_type": "markdown",
"id": "b9537f52",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Setup of Network\n",
"\n",
@@ -3295,9 +2972,7 @@
{
"cell_type": "markdown",
"id": "ca4e8f06",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"\n",
"\n",
@@ -3312,9 +2987,7 @@
{
"cell_type": "markdown",
"id": "0652089f",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Reformulating the problem\n",
"\n",
@@ -3331,9 +3004,7 @@
{
"cell_type": "markdown",
"id": "41cf6e19",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"g_t(x, P) = g_0 + x \\cdot N(x, P)\n",
@@ -3343,9 +3014,7 @@
{
"cell_type": "markdown",
"id": "49418fef",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"has been chosen such that it already solves the condition $g(0) = g_0$. What remains, is to find $P$ such that"
]
@@ -3353,9 +3022,7 @@
{
"cell_type": "markdown",
"id": "b7e00c55",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"\n",
"\n",
@@ -3370,9 +3037,7 @@
{
"cell_type": "markdown",
"id": "f4d3b6ed",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"is fulfilled as *best as possible*."
]
@@ -3380,9 +3045,7 @@
{
"cell_type": "markdown",
"id": "b6ada977",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## More technicalities\n",
"\n",
@@ -3396,9 +3059,7 @@
{
"cell_type": "markdown",
"id": "27ecbfef",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"\\min_{P}\\Big\\{ \\big(g_t'(x, P) - ( -\\gamma g_t(x, P) \\big)^2 \\Big\\}\n",
@@ -3408,9 +3069,7 @@
{
"cell_type": "markdown",
"id": "fd2fa044",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"(the notation $\\min_{P}\\{ f(x, P) \\}$ means that we desire to find $P$ that yields the minimum of $f(x, P)$)\n",
"\n",
@@ -3420,9 +3079,7 @@
{
"cell_type": "markdown",
"id": "9fd99313",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"\\min_{P_{\\text{hidden} }, \\ P_{\\text{output} }}\\Big\\{ \\big(g_t'(x, \\{ P_{\\text{hidden} }, P_{\\text{output} }\\}) - ( -\\gamma g_t(x, \\{ P_{\\text{hidden} }, P_{\\text{output} }\\}) \\big)^2 \\Big\\}\n",
@@ -3432,9 +3089,7 @@
{
"cell_type": "markdown",
"id": "3967be3c",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"for an input value $x$."
]
@@ -3442,9 +3097,7 @@
{
"cell_type": "markdown",
"id": "470d79c9",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## More details\n",
"\n",
@@ -3454,9 +3107,7 @@
{
"cell_type": "markdown",
"id": "afb9fa7a",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"\n",
"\n",
@@ -3471,9 +3122,7 @@
{
"cell_type": "markdown",
"id": "23c4c38a",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"Letting $\\boldsymbol{x}$ be a vector with elements $x_i$ and $C(\\boldsymbol{x}, P) = \\frac{1}{N} \\sum_i \\big(g_t'(x_i, P) - ( -\\gamma g_t(x_i, P) \\big)^2$ denote the cost function, the minimization problem that our network must solve, becomes"
]
@@ -3481,9 +3130,7 @@
{
"cell_type": "markdown",
"id": "acfa5e70",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"\\min_{P} C(\\boldsymbol{x}, P)\n",
@@ -3493,9 +3140,7 @@
{
"cell_type": "markdown",
"id": "6ca05848",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"In terms of $P_{\\text{hidden} }$ and $P_{\\text{output} }$, this could also be expressed as\n",
"\n",
@@ -3507,9 +3152,7 @@
{
"cell_type": "markdown",
"id": "11b8633a",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## A possible implementation of a neural network\n",
"\n",
@@ -3523,9 +3166,7 @@
{
"cell_type": "markdown",
"id": "82de0646",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Technicalities\n",
"\n",
@@ -3535,9 +3176,7 @@
{
"cell_type": "markdown",
"id": "5b117280",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"\\begin{aligned}\n",
@@ -3557,9 +3196,7 @@
{
"cell_type": "markdown",
"id": "9e23d93c",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Final technicalities I\n",
"\n",
@@ -3569,9 +3206,7 @@
{
"cell_type": "markdown",
"id": "c66c9d7a",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"\\begin{aligned}\n",
@@ -3592,9 +3227,7 @@
{
"cell_type": "markdown",
"id": "4d55f1d5",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Final technicalities II\n",
"\n",
@@ -3608,9 +3241,7 @@
{
"cell_type": "markdown",
"id": "81ccb597",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"f(z) = \\frac{1}{1 + \\exp{(-z)}}\n",
@@ -3620,9 +3251,7 @@
{
"cell_type": "markdown",
"id": "351e7bd0",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"It is possible to use other activations functions for the hidden layer also.\n",
"\n",
@@ -3644,9 +3273,7 @@
{
"cell_type": "markdown",
"id": "393e6171",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Final technicalities III\n",
"\n",
@@ -3656,9 +3283,7 @@
{
"cell_type": "markdown",
"id": "71b2569a",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"\\begin{aligned}\n",
@@ -3677,9 +3302,7 @@
{
"cell_type": "markdown",
"id": "5bc38371",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Final technicalities IV\n",
"\n",
@@ -3689,9 +3312,7 @@
{
"cell_type": "markdown",
"id": "e506906c",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"\\boldsymbol{z}_{1}^{\\text{output}} =\n",
@@ -3708,9 +3329,7 @@
{
"cell_type": "markdown",
"id": "15884d16",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"In this case we seek a continuous range of values since we are approximating a function. This means that after computing $\\boldsymbol{z}_{1}^{\\text{output}}$ the neural network has finished its feed forward step, and $\\boldsymbol{z}_{1}^{\\text{output}}$ is the final output of the network."
]
@@ -3718,9 +3337,7 @@
{
"cell_type": "markdown",
"id": "659ce37c",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Back propagation\n",
"\n",
@@ -3732,9 +3349,7 @@
{
"cell_type": "markdown",
"id": "81ad3b8d",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"C(\\boldsymbol{x}, P) = \\frac{1}{N} \\sum_i \\big(g_t'(x_i, P) - ( -\\gamma g_t(x_i, P) \\big)^2\n",
@@ -3744,9 +3359,7 @@
{
"cell_type": "markdown",
"id": "c80275f0",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"In order to minimize the cost function, an optimization method must be chosen.\n",
"\n",
@@ -3756,9 +3369,7 @@
{
"cell_type": "markdown",
"id": "277ed95a",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Gradient descent\n",
"\n",
@@ -3773,9 +3384,7 @@
{
"cell_type": "markdown",
"id": "b33a94dc",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"\\boldsymbol{\\omega}_{\\text{new} } = \\boldsymbol{\\omega} - \\lambda \\nabla_{\\boldsymbol{\\omega}} C(\\boldsymbol{x}, \\boldsymbol{\\omega})\n",
@@ -3785,9 +3394,7 @@
{
"cell_type": "markdown",
"id": "fc5ad11b",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"for a number of iterations or until $ \\big|\\big| \\boldsymbol{\\omega}_{\\text{new} } - \\boldsymbol{\\omega} \\big|\\big|$ becomes smaller than some given tolerance.\n",
"\n",
@@ -3807,9 +3414,7 @@
{
"cell_type": "markdown",
"id": "f1556cb6",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"\\begin{aligned}\n",
@@ -3822,9 +3427,7 @@
{
"cell_type": "markdown",
"id": "26238df7",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## The code for solving the ODE"
]
@@ -3833,10 +3436,7 @@
"cell_type": "code",
"execution_count": 24,
"id": "014617fe",
- "metadata": {
- "collapsed": false,
- "editable": true
- },
+ "metadata": {},
"outputs": [],
"source": [
"import autograd.numpy as np\n",
@@ -3988,9 +3588,7 @@
{
"cell_type": "markdown",
"id": "9d738f75",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## The network with one input layer, specified number of hidden layers, and one output layer\n",
"\n",
@@ -4003,10 +3601,7 @@
"cell_type": "code",
"execution_count": 25,
"id": "df58a772",
- "metadata": {
- "collapsed": false,
- "editable": true
- },
+ "metadata": {},
"outputs": [],
"source": [
"import autograd.numpy as np\n",
@@ -4172,9 +3767,7 @@
{
"cell_type": "markdown",
"id": "e9ec5a90",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Example: Population growth\n",
"\n",
@@ -4185,9 +3778,7 @@
{
"cell_type": "markdown",
"id": "2f4cc21b",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"\n",
"\n",
@@ -4202,9 +3793,7 @@
{
"cell_type": "markdown",
"id": "10d741aa",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"where $g(t)$ is the population density at time $t$, $\\alpha > 0$ the growth rate and $A > 0$ is the maximum population number in the environment.\n",
"Also, at $t = 0$ the population has the size $g(0) = g_0$, where $g_0$ is some chosen constant.\n",
@@ -4218,9 +3807,7 @@
{
"cell_type": "markdown",
"id": "9372f482",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Setting up the problem\n",
"\n",
@@ -4231,9 +3818,7 @@
{
"cell_type": "markdown",
"id": "4e5c33ed",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"\n",
"\n",
@@ -4248,9 +3833,7 @@
{
"cell_type": "markdown",
"id": "8fb325b6",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"where $g(0) = g_0$.\n",
"\n",
@@ -4260,9 +3843,7 @@
{
"cell_type": "markdown",
"id": "5cebac25",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## The trial solution\n",
"\n",
@@ -4287,9 +3868,7 @@
{
"cell_type": "markdown",
"id": "339fb0dc",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## The program using Autograd\n",
"\n",
@@ -4300,10 +3879,7 @@
"cell_type": "code",
"execution_count": 26,
"id": "73730937",
- "metadata": {
- "collapsed": false,
- "editable": true
- },
+ "metadata": {},
"outputs": [],
"source": [
"import autograd.numpy as np\n",
@@ -4474,9 +4050,7 @@
{
"cell_type": "markdown",
"id": "d6359e47",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Using forward Euler to solve the ODE\n",
"\n",
@@ -4494,9 +4068,7 @@
{
"cell_type": "markdown",
"id": "f5ada791",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"\\begin{aligned}\n",
@@ -4509,9 +4081,7 @@
{
"cell_type": "markdown",
"id": "6da1597b",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"along with the condition that $g(0) = g_0$.\n",
"\n",
@@ -4523,9 +4093,7 @@
{
"cell_type": "markdown",
"id": "13fa9fde",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"\\begin{aligned}\n",
@@ -4539,9 +4107,7 @@
{
"cell_type": "markdown",
"id": "1ba345dc",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"Now, if $g_i = g(t_i)$ then"
]
@@ -4549,9 +4115,7 @@
{
"cell_type": "markdown",
"id": "f263a5dc",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"\n",
"\n",
@@ -4571,9 +4135,7 @@
{
"cell_type": "markdown",
"id": "5ebd4428",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"for $i \\geq 1$ and $g_0 = g(t_0) = g(0) = g_0$.\n",
"\n",
@@ -4585,10 +4147,7 @@
"cell_type": "code",
"execution_count": 27,
"id": "95a32158",
- "metadata": {
- "collapsed": false,
- "editable": true
- },
+ "metadata": {},
"outputs": [],
"source": [
"# Assume that all function definitions from the example program using Autograd\n",
@@ -4661,9 +4220,7 @@
{
"cell_type": "markdown",
"id": "de8093cf",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Example: Solving the one dimensional Poisson equation\n",
"\n",
@@ -4673,9 +4230,7 @@
{
"cell_type": "markdown",
"id": "905ceb1e",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"\n",
"\n",
@@ -4690,9 +4245,7 @@
{
"cell_type": "markdown",
"id": "e3d5f8a0",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"where $f(x)$ is a given function for $x \\in (0,1)$.\n",
"\n",
@@ -4702,9 +4255,7 @@
{
"cell_type": "markdown",
"id": "52c161b4",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"\\begin{align*}\n",
@@ -4717,9 +4268,7 @@
{
"cell_type": "markdown",
"id": "7777097f",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"This equation can be solved numerically using programs where e.g Autograd and TensorFlow are used.\n",
"The results from the networks can then be compared to the analytical solution.\n",
@@ -4729,9 +4278,7 @@
{
"cell_type": "markdown",
"id": "4bbb70db",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## The specific equation to solve for\n",
"\n",
@@ -4741,9 +4288,7 @@
{
"cell_type": "markdown",
"id": "48006843",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"-g''(x) = f(x),\\qquad x \\in (0,1)\n",
@@ -4753,9 +4298,7 @@
{
"cell_type": "markdown",
"id": "8638c014",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"where $f(x)$ is a given function, along with the chosen conditions"
]
@@ -4763,9 +4306,7 @@
{
"cell_type": "markdown",
"id": "27c57849",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"\n",
"\n",
@@ -4780,9 +4321,7 @@
{
"cell_type": "markdown",
"id": "c391bc33",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"In this example, we consider the case when $f(x) = (3x + x^2)\\exp(x)$.\n",
"\n",
@@ -4792,9 +4331,7 @@
{
"cell_type": "markdown",
"id": "a3819b93",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"g_t(x) = x \\cdot (1-x) \\cdot N(P,x)\n",
@@ -4804,9 +4341,7 @@
{
"cell_type": "markdown",
"id": "7b1c2870",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"The analytical solution for this problem is"
]
@@ -4814,9 +4349,7 @@
{
"cell_type": "markdown",
"id": "c77e16fc",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"g(x) = x(1 - x)\\exp(x)\n",
@@ -4826,9 +4359,7 @@
{
"cell_type": "markdown",
"id": "fe153456",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Solving the equation using Autograd"
]
@@ -4837,10 +4368,7 @@
"cell_type": "code",
"execution_count": 28,
"id": "1ad3d48d",
- "metadata": {
- "collapsed": false,
- "editable": true
- },
+ "metadata": {},
"outputs": [],
"source": [
"import autograd.numpy as np\n",
@@ -4998,9 +4526,7 @@
{
"cell_type": "markdown",
"id": "c40a7203",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Comparing with a numerical scheme\n",
"\n",
@@ -5020,9 +4546,7 @@
{
"cell_type": "markdown",
"id": "c6fa6ab2",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"\n",
"\n",
@@ -5037,9 +4561,7 @@
{
"cell_type": "markdown",
"id": "f194e6eb",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"If $x_i = i \\Delta x = x_{i-1} + \\Delta x$ and $g_i = g(x_i)$ for $i = 1,\\dots N_x - 2$ with $N_x$ being the number of values for $x$, ([15](#approx)) becomes"
]
@@ -5047,9 +4569,7 @@
{
"cell_type": "markdown",
"id": "7f05f9b0",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"\\begin{aligned}\n",
@@ -5062,9 +4582,7 @@
{
"cell_type": "markdown",
"id": "770d5cf3",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"Since we know from our problem that"
]
@@ -5072,9 +4590,7 @@
{
"cell_type": "markdown",
"id": "805cd4c8",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"\\begin{aligned}\n",
@@ -5087,9 +4603,7 @@
{
"cell_type": "markdown",
"id": "cdf4faec",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"along with the conditions $g(0) = g(1) = 0$,\n",
"the following scheme can be used to find an approximate solution for $g(x)$ numerically:"
@@ -5098,9 +4612,7 @@
{
"cell_type": "markdown",
"id": "507b9b84",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"\n",
"\n",
@@ -5118,9 +4630,7 @@
{
"cell_type": "markdown",
"id": "67b2c134",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"for $i = 1, \\dots, N_x - 2$ where $g_0 = g_{N_x - 1} = 0$ and $f(x_i) = (3x_i + x_i^2)\\exp(x_i)$, which is given for our specific problem.\n",
"\n",
@@ -5130,9 +4640,7 @@
{
"cell_type": "markdown",
"id": "30036fd5",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"\\begin{aligned}\n",
@@ -5167,9 +4675,7 @@
{
"cell_type": "markdown",
"id": "374c2536",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"which makes it possible to solve for the vector $\\boldsymbol{g}$."
]
@@ -5177,9 +4683,7 @@
{
"cell_type": "markdown",
"id": "1391a6dd",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Setting up the code\n",
"\n",
@@ -5190,10 +4694,7 @@
"cell_type": "code",
"execution_count": 29,
"id": "e7f38d7e",
- "metadata": {
- "collapsed": false,
- "editable": true
- },
+ "metadata": {},
"outputs": [],
"source": [
"import autograd.numpy as np\n",
@@ -5391,9 +4892,7 @@
{
"cell_type": "markdown",
"id": "9fc31e49",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Partial Differential Equations\n",
"\n",
@@ -5408,9 +4907,7 @@
{
"cell_type": "markdown",
"id": "c0928905",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"\n",
"\n",
@@ -5425,9 +4922,7 @@
{
"cell_type": "markdown",
"id": "7bae33b7",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"where $f$ is an expression involving all kinds of possible mixed derivatives of $g(x_1,\\dots,x_N)$ up to an order $n$. In order for the solution to be unique, some additional conditions must also be given."
]
@@ -5435,9 +4930,7 @@
{
"cell_type": "markdown",
"id": "b43539b4",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Type of problem\n",
"\n",
@@ -5450,9 +4943,7 @@
{
"cell_type": "markdown",
"id": "6fc1ca67",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"\\begin{align*}\n",
@@ -5464,9 +4955,7 @@
{
"cell_type": "markdown",
"id": "d35a7f13",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"where $h_1(x_1,\\dots,x_N)$ is a function that ensures $g_t(x_1,\\dots,x_N)$ satisfies some given conditions.\n",
"The neural network $N(x_1,\\dots,x_N,P)$ has weights and biases described by $P$ and $h_2(x_1,\\dots,x_N,N(x_1,\\dots,x_N,P))$ is an expression using the output from the neural network in some way.\n",
@@ -5477,9 +4966,7 @@
{
"cell_type": "markdown",
"id": "4cce797b",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Network requirements\n",
"\n",
@@ -5497,9 +4984,7 @@
{
"cell_type": "markdown",
"id": "e399151e",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"C\\left(x_1, \\dots, x_N, P\\right) = \\left( f\\left(x_1, \\, \\dots \\, , x_N, \\frac{\\partial g(x_1,\\dots,x_N) }{\\partial x_1}, \\dots , \\frac{\\partial g(x_1,\\dots,x_N) }{\\partial x_N}, \\frac{\\partial g(x_1,\\dots,x_N) }{\\partial x_1\\partial x_2}, \\, \\dots \\, , \\frac{\\partial^n g(x_1,\\dots,x_N) }{\\partial x_N^n} \\right) \\right)^2\n",
@@ -5509,9 +4994,7 @@
{
"cell_type": "markdown",
"id": "67e4d279",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## More details\n",
"\n",
@@ -5521,9 +5004,7 @@
{
"cell_type": "markdown",
"id": "fcaa5c92",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"C\\left(\\boldsymbol{x}, P\\right) = f\\left( \\left( \\boldsymbol{x}, \\frac{\\partial g(\\boldsymbol{x}) }{\\partial x_1}, \\dots , \\frac{\\partial g(\\boldsymbol{x}) }{\\partial x_N}, \\frac{\\partial g(\\boldsymbol{x}) }{\\partial x_1\\partial x_2}, \\, \\dots \\, , \\frac{\\partial^n g(\\boldsymbol{x}) }{\\partial x_N^n} \\right) \\right)^2\n",
@@ -5533,9 +5014,7 @@
{
"cell_type": "markdown",
"id": "7681fcbc",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"If we also have $M$ different sets of values for $x_1, \\dots, x_N$, that is $\\boldsymbol{x}_i = \\big(x_1^{(i)}, \\dots, x_N^{(i)}\\big)$ for $i = 1,\\dots,M$ being the rows in matrix $X$, the cost function can be generalized into"
]
@@ -5543,9 +5022,7 @@
{
"cell_type": "markdown",
"id": "2657db4f",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"C\\left(X, P \\right) = \\sum_{i=1}^M f\\left( \\left( \\boldsymbol{x}_i, \\frac{\\partial g(\\boldsymbol{x}_i) }{\\partial x_1}, \\dots , \\frac{\\partial g(\\boldsymbol{x}_i) }{\\partial x_N}, \\frac{\\partial g(\\boldsymbol{x}_i) }{\\partial x_1\\partial x_2}, \\, \\dots \\, , \\frac{\\partial^n g(\\boldsymbol{x}_i) }{\\partial x_N^n} \\right) \\right)^2.\n",
@@ -5555,9 +5032,7 @@
{
"cell_type": "markdown",
"id": "7e50704f",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Example: The diffusion equation\n",
"\n",
@@ -5567,9 +5042,7 @@
{
"cell_type": "markdown",
"id": "b8ecd408",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"\\frac{\\partial g(x,t)}{\\partial t} = \\frac{\\partial^2 g(x,t)}{\\partial x^2}\n",
@@ -5579,9 +5052,7 @@
{
"cell_type": "markdown",
"id": "4088543f",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"where a possible choice of conditions are"
]
@@ -5589,9 +5060,7 @@
{
"cell_type": "markdown",
"id": "ced127ba",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"\\begin{align*}\n",
@@ -5605,9 +5074,7 @@
{
"cell_type": "markdown",
"id": "c9f0f219",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"with $u(x)$ being some given function."
]
@@ -5615,9 +5082,7 @@
{
"cell_type": "markdown",
"id": "55a10797",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Defining the problem\n",
"\n",
@@ -5627,9 +5092,7 @@
{
"cell_type": "markdown",
"id": "718ef592",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"\n",
"\n",
@@ -5644,9 +5107,7 @@
{
"cell_type": "markdown",
"id": "e9c1ebd6",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"and"
]
@@ -5654,9 +5115,7 @@
{
"cell_type": "markdown",
"id": "f900ccf7",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"\\begin{align*}\n",
@@ -5670,9 +5129,7 @@
{
"cell_type": "markdown",
"id": "83307dd1",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"with $u(x) = \\sin(\\pi x)$.\n",
"\n",
@@ -5684,9 +5141,7 @@
{
"cell_type": "markdown",
"id": "11a91a53",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Setting up the network using Autograd\n",
"\n",
@@ -5703,10 +5158,7 @@
"cell_type": "code",
"execution_count": 30,
"id": "c6626e17",
- "metadata": {
- "collapsed": false,
- "editable": true
- },
+ "metadata": {},
"outputs": [],
"source": [
"def sigmoid(z):\n",
@@ -5758,9 +5210,7 @@
{
"cell_type": "markdown",
"id": "9bc068c4",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Setting up the network using Autograd; The trial solution\n",
"\n",
@@ -5788,9 +5238,7 @@
{
"cell_type": "markdown",
"id": "72dcd8e2",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Why the jacobian?\n",
"\n",
@@ -5817,10 +5265,7 @@
"cell_type": "code",
"execution_count": 31,
"id": "17e14051",
- "metadata": {
- "collapsed": false,
- "editable": true
- },
+ "metadata": {},
"outputs": [],
"source": [
"# Set up the trial function:\n",
@@ -5864,9 +5309,7 @@
{
"cell_type": "markdown",
"id": "503a302f",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Setting up the network using Autograd; The full program\n",
"\n",
@@ -5890,10 +5333,7 @@
"cell_type": "code",
"execution_count": 32,
"id": "02343bb2",
- "metadata": {
- "collapsed": false,
- "editable": true
- },
+ "metadata": {},
"outputs": [],
"source": [
"import autograd.numpy as np\n",
@@ -6124,9 +5564,7 @@
{
"cell_type": "markdown",
"id": "61fa3ee3",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Example: Solving the wave equation with Neural Networks\n",
"\n",
@@ -6136,9 +5574,7 @@
{
"cell_type": "markdown",
"id": "f08ec1b3",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"\\frac{\\partial^2 g(x,t)}{\\partial t^2} = c^2\\frac{\\partial^2 g(x,t)}{\\partial x^2}\n",
@@ -6148,9 +5584,7 @@
{
"cell_type": "markdown",
"id": "aca9df35",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"with $c$ being the specified wave speed.\n",
"\n",
@@ -6160,9 +5594,7 @@
{
"cell_type": "markdown",
"id": "c854909b",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"$$\n",
"\\begin{align*}\n",
@@ -6177,9 +5609,7 @@
{
"cell_type": "markdown",
"id": "a4d855d2",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"where $\\frac{\\partial g(x,t)}{\\partial t} \\Big |_{t = 0}$ means the derivative of $g(x,t)$ with respect to $t$ is evaluated at $t = 0$, and $u(x)$ and $v(x)$ being given functions."
]
@@ -6187,9 +5617,7 @@
{
"cell_type": "markdown",
"id": "7bcd28e2",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## The problem to solve for\n",
"\n",
@@ -6199,9 +5627,7 @@
{
"cell_type": "markdown",
"id": "2c689212",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"\n",
"\n",
@@ -6216,9 +5642,7 @@
{
"cell_type": "markdown",
"id": "0bbaf576",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"where $c$ is the given wave speed.\n",
"The chosen conditions for this equation are"
@@ -6227,9 +5651,7 @@
{
"cell_type": "markdown",
"id": "6763059e",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"\n",
"\n",
@@ -6247,9 +5669,7 @@
{
"cell_type": "markdown",
"id": "1e65f7b9",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"In this example, let $c = 1$ and $u(x) = \\sin(\\pi x)$ and $v(x) = -\\pi\\sin(\\pi x)$."
]
@@ -6257,9 +5677,7 @@
{
"cell_type": "markdown",
"id": "e3d81595",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## The trial solution\n",
"Setting up the network is done in similar matter as for the example of solving the diffusion equation.\n",
@@ -6283,9 +5701,7 @@
{
"cell_type": "markdown",
"id": "a75130a3",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## The analytical solution\n",
"\n",
@@ -6299,9 +5715,7 @@
{
"cell_type": "markdown",
"id": "fae3ec5d",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Solving the wave equation - the full program using Autograd"
]
@@ -6310,10 +5724,7 @@
"cell_type": "code",
"execution_count": 33,
"id": "c2a73012",
- "metadata": {
- "collapsed": false,
- "editable": true
- },
+ "metadata": {},
"outputs": [],
"source": [
"import autograd.numpy as np\n",
@@ -6541,9 +5952,7 @@
{
"cell_type": "markdown",
"id": "b4f99f14",
- "metadata": {
- "editable": true
- },
+ "metadata": {},
"source": [
"## Resources on differential equations and deep learning\n",
"\n",
@@ -6557,7 +5966,25 @@
]
}
],
- "metadata": {},
+ "metadata": {
+ "kernelspec": {
+ "display_name": "Python 3 (ipykernel)",
+ "language": "python",
+ "name": "python3"
+ },
+ "language_info": {
+ "codemirror_mode": {
+ "name": "ipython",
+ "version": 3
+ },
+ "file_extension": ".py",
+ "mimetype": "text/x-python",
+ "name": "python",
+ "nbconvert_exporter": "python",
+ "pygments_lexer": "ipython3",
+ "version": "3.8.12"
+ }
+ },
"nbformat": 4,
"nbformat_minor": 5
}