diff --git a/doc/LectureNotes/_build/.doctrees/environment.pickle b/doc/LectureNotes/_build/.doctrees/environment.pickle index b898f62c8..8d3a58e34 100644 Binary files a/doc/LectureNotes/_build/.doctrees/environment.pickle and b/doc/LectureNotes/_build/.doctrees/environment.pickle differ diff --git a/doc/LectureNotes/_build/.doctrees/intro.doctree b/doc/LectureNotes/_build/.doctrees/intro.doctree index b4325ce79..f3aa1e84d 100644 Binary files a/doc/LectureNotes/_build/.doctrees/intro.doctree and b/doc/LectureNotes/_build/.doctrees/intro.doctree differ diff --git a/doc/LectureNotes/_build/.doctrees/project2.doctree b/doc/LectureNotes/_build/.doctrees/project2.doctree new file mode 100644 index 000000000..99edac8ac Binary files /dev/null and b/doc/LectureNotes/_build/.doctrees/project2.doctree differ diff --git a/doc/LectureNotes/_build/.doctrees/week42.doctree b/doc/LectureNotes/_build/.doctrees/week42.doctree new file mode 100644 index 000000000..2d658f359 Binary files /dev/null and b/doc/LectureNotes/_build/.doctrees/week42.doctree differ diff --git a/doc/LectureNotes/_build/html/_sources/project2.ipynb b/doc/LectureNotes/_build/html/_sources/project2.ipynb new file mode 100644 index 000000000..b22fbdd58 --- /dev/null +++ b/doc/LectureNotes/_build/html/_sources/project2.ipynb @@ -0,0 +1,554 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "id": "061af572", + "metadata": { + "editable": true + }, + "source": [ + "\n", + "" + ] + }, + { + "cell_type": "markdown", + "id": "22578683", + "metadata": { + "editable": true + }, + "source": [ + "# Project 2 on Machine Learning, deadline November 10 (Midnight)\n", + "**[Data Analysis and Machine Learning FYS-STK3155/FYS4155](http://www.uio.no/studier/emner/matnat/fys/FYS3155/index-eng.html)**, University of Oslo, Norway\n", + "\n", + "Date: **October 14, 2025**" + ] + }, + { + "cell_type": "markdown", + "id": "61fb162f", + "metadata": { + "editable": true + }, + "source": [ + "## Deliverables\n", + "\n", + "First, join a group in canvas with your group partners. Pick an avaliable group for Project 2 in the **People** page.\n", + "\n", + "In canvas, deliver as a group and include:\n", + "\n", + "* A PDF of your report which follows the guidelines covered below and in the week 39 exercises. Additional requirements include:\n", + "\n", + " * It should be around 5000 words, use the word counter in Overleaf for this. This often corresponds to 10-12 pages. References and appendices are excluded from the word count\n", + "\n", + " * It should include around 10-15 figures. You can include more figures in appendices and/or as supplemental material in your repository.\n", + "\n", + "* A comment linking to your github repository (or folder in one of your github repositories) for this project. The repository must include\n", + "\n", + "A PDF file of the report\n", + " * A folder named Code, where you put python files for your functions and notebooks for reproducing your results. Remember to use a seed for generating random data and for train-test splits when generating final results.\n", + "\n", + " * A README file with the name of the group members\n", + "\n", + " * a short description of the project\n", + "\n", + " * a description of how to install the required packages to run your code from a requirements.txt file or similar (such as a plain text description) names and descriptions of the various notebooks in the Code folder and the results they produce" + ] + }, + { + "cell_type": "markdown", + "id": "104c69e1", + "metadata": { + "editable": true + }, + "source": [ + "### Preamble: Note on writing reports, using reference material, AI and other tools\n", + "\n", + "We want you to answer the three different projects by handing in\n", + "reports written like a standard scientific/technical report. The links\n", + "at\n", + "https://github.com/CompPhysics/MachineLearning/tree/master/doc/Projects\n", + "contain more information. There you can find examples of previous\n", + "reports, the projects themselves, how we grade reports etc. How to\n", + "write reports will also be discussed during the various lab\n", + "sessions. Please do ask us if you are in doubt.\n", + "\n", + "When using codes and material from other sources, you should refer to\n", + "these in the bibliography of your report, indicating wherefrom you for\n", + "example got the code, whether this is from the lecture notes,\n", + "softwares like Scikit-Learn, TensorFlow, PyTorch or other\n", + "sources. These sources should always be cited correctly. How to cite\n", + "some of the libraries is often indicated from their corresponding\n", + "GitHub sites or websites, see for example how to cite Scikit-Learn at\n", + "https://scikit-learn.org/dev/about.html.\n", + "\n", + "We enocurage you to use tools like ChatGPT or similar in writing the\n", + "report. If you use for example ChatGPT, please do cite it properly and\n", + "include (if possible) your questions and answers as an addition to the\n", + "report. This can be uploaded to for example your website,\n", + "GitHub/GitLab or similar as supplemental material.\n", + "\n", + "If you would like to study other data sets, feel free to propose other\n", + "sets. What we have proposed here are mere suggestions from our\n", + "side. If you opt for another data set, consider using a set which has\n", + "been studied in the scientific literature. This makes it easier for\n", + "you to compare and analyze your results. Comparing with existing\n", + "results from the scientific literature is also an essential element of\n", + "the scientific discussion. The University of California at Irvine with\n", + "its Machine Learning repository at\n", + "https://archive.ics.uci.edu/ml/index.php is an excellent site to look\n", + "up for examples and inspiration. Kaggle.com is an equally interesting\n", + "site. Feel free to explore these sites." + ] + }, + { + "cell_type": "markdown", + "id": "0d2c42e3", + "metadata": { + "editable": true + }, + "source": [ + "## Classification and Regression, writing our own neural network code\n", + "\n", + "The main aim of this project is to study both classification and\n", + "regression problems by developing our own \n", + "feed-forward neural network (FFNN) code. The exercises from week 41 and 42 (see and ) as well as the lecture material from the same weeks (see and ) should contain enough information for you to get started with writing your own code.\n", + "\n", + "We will also reuse our codes on gradient descent methods from project 1.\n", + "\n", + "The data sets that we propose here are (the default sets)\n", + "\n", + "* Regression (fitting a continuous function). In this part you will need to bring back your results from project 1 and compare these with what you get from your Neural Network code to be developed here. The data sets could be\n", + "\n", + " * The simple one-dimensional function Runge function from project 1, that is $f(x) = \\frac{1}{1+25x^2}$. We recommend using a simpler function when developing your neural network code for regression problems. Feel however free to discuss and study other functions, such as the the two-dimensional Runge function $f(x,y)=\\left[(10x - 5)^2 + (10y - 5)^2 + 1 \\right]^{-1}$, or even more complicated two-dimensional functions (see the supplementary material of for an extensive list of two-dimensional functions). \n", + "\n", + "* Classification.\n", + "\n", + " * We will consider a multiclass classification problem given by the full MNIST data set. The full data set is at .\n", + "\n", + "We will start with a regression problem and we will reuse our codes on gradient descent methods from project 1." + ] + }, + { + "cell_type": "markdown", + "id": "d8baab67", + "metadata": { + "editable": true + }, + "source": [ + "### Part a): Analytical warm-up\n", + "\n", + "When using our gradient machinery from project 1, we will need the expressions for the cost/loss functions and their respective\n", + "gradients. The functions whose gradients we need are:\n", + "1. The mean-squared error (MSE) with and without the $L_1$ and $L_2$ norms (regression problems)\n", + "\n", + "2. The binary cross entropy (aka log loss) for binary classification problems with and without $L_1$ and $L_2$ norms\n", + "\n", + "3. The multiclass cross entropy cost/loss function (aka Softmax cross entropy or just Softmax loss function)\n", + "\n", + "Set up these three cost/loss functions and their respective derivatives and explain the various terms. In this project you will however only use the MSE and the Softmax cross entropy.\n", + "\n", + "We will test three activation functions for our neural network setup, these are the \n", + "1. The Sigmoid (aka **logit**) function,\n", + "\n", + "2. the RELU function and\n", + "\n", + "3. the Leaky RELU function\n", + "\n", + "Set up their expressions and their first derivatives.\n", + "You may consult the lecture notes (with codes and more) from week 42 at ." + ] + }, + { + "cell_type": "markdown", + "id": "87e7ed71", + "metadata": { + "editable": true + }, + "source": [ + "### Reminder about the gradient machinery from project 1\n", + "\n", + "In the setup of a neural network code you will need your gradient descent codes from\n", + "project 1. For neural networks we will recommend using stochastic\n", + "gradient descent with either the RMSprop or the ADAM algorithms for\n", + "updating the learning rates. But you should feel free to try plain gradient descent as well.\n", + "\n", + "We recommend reading chapter 8 on optimization from the textbook of\n", + "Goodfellow, Bengio and Courville at\n", + ". This chapter contains many\n", + "useful insights and discussions on the optimization part of machine\n", + "learning. A useful reference on the back progagation algorithm is\n", + "Nielsen's book at . \n", + "\n", + "You will find the Python [Seaborn\n", + "package](https://seaborn.pydata.org/generated/seaborn.heatmap.html)\n", + "useful when plotting the results as function of the learning rate\n", + "$\\eta$ and the hyper-parameter $\\lambda$ ." + ] + }, + { + "cell_type": "markdown", + "id": "5a26b6ad", + "metadata": { + "editable": true + }, + "source": [ + "### Part b): Writing your own Neural Network code\n", + "\n", + "Your aim now, and this is the central part of this project, is to\n", + "write your own FFNN code implementing the back\n", + "propagation algorithm discussed in the lecture slides from week 41 at and week 42 at .\n", + "\n", + "We will focus on a regression problem first, using the one-dimensional Runge function" + ] + }, + { + "cell_type": "markdown", + "id": "096fe6c4", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "f(x) = \\frac{1}{1+25x^2},\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "fd986596", + "metadata": { + "editable": true + }, + "source": [ + "from project 1.\n", + "\n", + "Use only the mean-squared error as cost function (no regularization terms) and \n", + "write an FFNN code for a regression problem with a flexible number of hidden\n", + "layers and nodes using only the Sigmoid function as activation function for\n", + "the hidden layers. Initialize the weights using a normal\n", + "distribution. How would you initialize the biases? And which\n", + "activation function would you select for the final output layer?\n", + "And how would you set up your design/feature matrix? Hint: does it have to represent a polynomial approximation as you did in project 1? \n", + "\n", + "Train your network and compare the results with those from your OLS\n", + "regression code from project 1 using the one-dimensional Runge\n", + "function. When comparing your neural network code with the OLS\n", + "results from project 1, use the same data sets which gave you the best\n", + "MSE score. Moreover, use the polynomial order from project 1 that gave you the\n", + "best result. Compare these results with your neural network with one\n", + "and two hidden layers using $50$ and $100$ hidden nodes, respectively.\n", + "\n", + "Comment your results and give a critical discussion of the results\n", + "obtained with the OLS code from project 1 and your own neural network\n", + "code. Make an analysis of the learning rates employed to find the\n", + "optimal MSE score. Test both stochastic gradient descent\n", + "with RMSprop and ADAM and plain gradient descent with different\n", + "learning rates.\n", + "\n", + "You should, as you did in project 1, scale your data." + ] + }, + { + "cell_type": "markdown", + "id": "e853d4b6", + "metadata": { + "editable": true + }, + "source": [ + "### Part c): Testing against other software libraries\n", + "\n", + "You should test your results against a similar code using **Scikit-Learn** (see the examples in the above lecture notes from weeks 41 and 42) or **tensorflow/keras** or **Pytorch** (for Pytorch, see Raschka et al.'s text chapters 12 and 13). \n", + "\n", + "Furthermore, you should also test that your derivatives are correctly\n", + "calculated using automatic differentiation, using for example the\n", + "**Autograd** library or the **JAX** library. It is optional to implement\n", + "these libraries for the present project. In this project they serve as\n", + "useful tests of our derivatives." + ] + }, + { + "cell_type": "markdown", + "id": "fc2d413b", + "metadata": { + "editable": true + }, + "source": [ + "### Part d): Testing different activation functions and depths of the neural network\n", + "\n", + "You should also test different activation functions for the hidden\n", + "layers. Try out the Sigmoid, the RELU and the Leaky RELU functions and\n", + "discuss your results. Test your results as functions of the number of hidden layers and nodes. Do you see signs of overfitting?\n", + "It is optional in this project to perform a bias-variance trade-off analysis." + ] + }, + { + "cell_type": "markdown", + "id": "e6821051", + "metadata": { + "editable": true + }, + "source": [ + "### Part e): Testing different norms\n", + "\n", + "Finally, still using the one-dimensional Runge function, add now the\n", + "hyperparameters $\\lambda$ with the $L_2$ and $L_1$ norms. Find the\n", + "optimal results for the hyperparameters $\\lambda$ and the learning\n", + "rates $\\eta$ and neural network architecture and compare the $L_2$ results with Ridge regression from\n", + "project 1 and the $L_1$ results with the Lasso calculations of project 1.\n", + "Use again the same data sets and the best results from project 1 in your comparisons." + ] + }, + { + "cell_type": "markdown", + "id": "cba72d68", + "metadata": { + "editable": true + }, + "source": [ + "### Part f): Classification analysis using neural networks\n", + "\n", + "With a well-written code it should now be easy to change the\n", + "activation function for the output layer.\n", + "\n", + "Here we will change the cost function for our neural network code\n", + "developed in parts b), d) and e) in order to perform a classification\n", + "analysis. The classification problem we will study is the multiclass\n", + "MNIST problem, see the description of the full data set at\n", + ". We will use the Softmax cross entropy function discussed in a). \n", + "The MNIST data set discussed in the lecture notes from week 42 is a downscaled variant of the full dataset. \n", + "\n", + "Feel free to suggest other data sets. If you find the classic MNIST data set somewhat limited, feel free to try the \n", + "MNIST-Fashion data set at for example .\n", + "\n", + "To set up the data set, the following python programs may be useful" + ] + }, + { + "cell_type": "code", + "execution_count": 1, + "id": "e16fb528", + "metadata": { + "collapsed": false, + "editable": true + }, + "outputs": [], + "source": [ + "from sklearn.datasets import fetch_openml\n", + "\n", + "# Fetch the MNIST dataset\n", + "mnist = fetch_openml('mnist_784', version=1, as_frame=False, parser='auto')\n", + "\n", + "# Extract data (features) and target (labels)\n", + "X = mnist.data\n", + "y = mnist.target" + ] + }, + { + "cell_type": "markdown", + "id": "73599f42", + "metadata": { + "editable": true + }, + "source": [ + "You should consider scaling the data. The Pixel values in MNIST range from 0 to 255. Scaling them to a 0-1 range can improve the performance of some models. That is, you could implement the following scaling" + ] + }, + { + "cell_type": "code", + "execution_count": 2, + "id": "f1a639ef", + "metadata": { + "collapsed": false, + "editable": true + }, + "outputs": [], + "source": [ + "X = X / 255.0" + ] + }, + { + "cell_type": "markdown", + "id": "90fb7b41", + "metadata": { + "editable": true + }, + "source": [ + "And then perform the standard train-test splitting" + ] + }, + { + "cell_type": "code", + "execution_count": 3, + "id": "424af629", + "metadata": { + "collapsed": false, + "editable": true + }, + "outputs": [], + "source": [ + "from sklearn.model_selection import train_test_split\n", + "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)" + ] + }, + { + "cell_type": "markdown", + "id": "3c006080", + "metadata": { + "editable": true + }, + "source": [ + "To measure the performance of our classification problem we will use the\n", + "so-called *accuracy* score. The accuracy is as you would expect just\n", + "the number of correctly guessed targets $t_i$ divided by the total\n", + "number of targets, that is" + ] + }, + { + "cell_type": "markdown", + "id": "a18ddd54", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\text{Accuracy} = \\frac{\\sum_{i=1}^n I(t_i = y_i)}{n} ,\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "1a1afaf9", + "metadata": { + "editable": true + }, + "source": [ + "where $I$ is the indicator function, $1$ if $t_i = y_i$ and $0$\n", + "otherwise if we have a binary classification problem. Here $t_i$\n", + "represents the target and $y_i$ the outputs of your FFNN code and $n$ is simply the number of targets $t_i$.\n", + "\n", + "Discuss your results and give a critical analysis of the various parameters, including hyper-parameters like the learning rates and the regularization parameter $\\lambda$, various activation functions, number of hidden layers and nodes and activation functions. \n", + "\n", + "Again, we strongly recommend that you compare your own neural Network\n", + "code for classification and pertinent results against a similar code using **Scikit-Learn** or **tensorflow/keras** or **pytorch**.\n", + "\n", + "If you have time, you can use the functionality of **scikit-learn** and compare your neural network results with those from Logistic regression. This is optional.\n", + "The weblink here compares logistic regression and FFNN using the so-called MNIST data set. You may find several useful hints and ideas from this article. Your neural network code can implement the equivalent of logistic regression by simply setting the number of hidden layers to zero. \n", + "\n", + "If you wish to compare with say Logisti Regression from **scikit-learn**, the following code uses the above data set" + ] + }, + { + "cell_type": "code", + "execution_count": 4, + "id": "3c37cbaf", + "metadata": { + "collapsed": false, + "editable": true + }, + "outputs": [], + "source": [ + "from sklearn.linear_model import LogisticRegression\n", + "# Initialize the model\n", + "model = LogisticRegression(solver='saga', multi_class='multinomial', max_iter=1000, random_state=42)\n", + "# Train the model\n", + "model.fit(X_train, y_train)\n", + "from sklearn.metrics import accuracy_score\n", + "# Make predictions on the test set\n", + "y_pred = model.predict(X_test)\n", + "# Calculate accuracy\n", + "accuracy = accuracy_score(y_test, y_pred)\n", + "print(f\"Model Accuracy: {accuracy:.4f}\")" + ] + }, + { + "cell_type": "markdown", + "id": "106b9303", + "metadata": { + "editable": true + }, + "source": [ + "### Part g) Critical evaluation of the various algorithms\n", + "\n", + "After all these glorious calculations, you should now summarize the\n", + "various algorithms and come with a critical evaluation of their pros\n", + "and cons. Which algorithm works best for the regression case and which\n", + "is best for the classification case. These codes can also be part of\n", + "your final project 3, but now applied to other data sets." + ] + }, + { + "cell_type": "markdown", + "id": "55da0d7f", + "metadata": { + "editable": true + }, + "source": [ + "## Background literature\n", + "\n", + "1. The text of Michael Nielsen is highly recommended, see Nielsen's book at . It is an excellent read.\n", + "\n", + "2. Goodfellow, Bengio and Courville, Deep Learning at . Here we recommend chapters 6, 7 and 8\n", + "\n", + "3. Raschka et al. at . Here we recommend chapters 11, 12 and 13." + ] + }, + { + "cell_type": "markdown", + "id": "d3731e2c", + "metadata": { + "editable": true + }, + "source": [ + "## Introduction to numerical projects\n", + "\n", + "Here follows a brief recipe and recommendation on how to write a report for each\n", + "project.\n", + "\n", + " * Give a short description of the nature of the problem and the eventual numerical methods you have used.\n", + "\n", + " * Describe the algorithm you have used and/or developed. Here you may find it convenient to use pseudocoding. In many cases you can describe the algorithm in the program itself.\n", + "\n", + " * Include the source code of your program. Comment your program properly.\n", + "\n", + " * If possible, try to find analytic solutions, or known limits in order to test your program when developing the code.\n", + "\n", + " * Include your results either in figure form or in a table. Remember to label your results. All tables and figures should have relevant captions and labels on the axes.\n", + "\n", + " * Try to evaluate the reliabilty and numerical stability/precision of your results. If possible, include a qualitative and/or quantitative discussion of the numerical stability, eventual loss of precision etc.\n", + "\n", + " * Try to give an interpretation of you results in your answers to the problems.\n", + "\n", + " * Critique: if possible include your comments and reflections about the exercise, whether you felt you learnt something, ideas for improvements and other thoughts you've made when solving the exercise. We wish to keep this course at the interactive level and your comments can help us improve it.\n", + "\n", + " * Try to establish a practice where you log your work at the computerlab. You may find such a logbook very handy at later stages in your work, especially when you don't properly remember what a previous test version of your program did. Here you could also record the time spent on solving the exercise, various algorithms you may have tested or other topics which you feel worthy of mentioning." + ] + }, + { + "cell_type": "markdown", + "id": "6c7c5340", + "metadata": { + "editable": true + }, + "source": [ + "## Format for electronic delivery of report and programs\n", + "\n", + "The preferred format for the report is a PDF file. You can also use DOC or postscript formats or as an ipython notebook file. As programming language we prefer that you choose between C/C++, Fortran2008 or Python. The following prescription should be followed when preparing the report:\n", + "\n", + " * Use Canvas to hand in your projects, log in at with your normal UiO username and password.\n", + "\n", + " * Upload **only** the report file or the link to your GitHub/GitLab or similar typo of repos! For the source code file(s) you have developed please provide us with your link to your GitHub/GitLab or similar domain. The report file should include all of your discussions and a list of the codes you have developed. Do not include library files which are available at the course homepage, unless you have made specific changes to them.\n", + "\n", + " * In your GitHub/GitLab or similar repository, please include a folder which contains selected results. These can be in the form of output from your code for a selected set of runs and input parameters.\n", + "\n", + "Finally, \n", + "we encourage you to collaborate. Optimal working groups consist of \n", + "2-3 students. You can then hand in a common report." + ] + } + ], + "metadata": {}, + "nbformat": 4, + "nbformat_minor": 5 +} diff --git a/doc/LectureNotes/_build/html/_sources/week42.ipynb b/doc/LectureNotes/_build/html/_sources/week42.ipynb new file mode 100644 index 000000000..af01abbf4 --- /dev/null +++ b/doc/LectureNotes/_build/html/_sources/week42.ipynb @@ -0,0 +1,5950 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "id": "7dcf8613", + "metadata": { + "editable": true + }, + "source": [ + "\n", + "" + ] + }, + { + "cell_type": "markdown", + "id": "98019924", + "metadata": { + "editable": true + }, + "source": [ + "# Week 42 Constructing a Neural Network code with examples\n", + "**Morten Hjorth-Jensen**, Department of Physics, University of Oslo, Norway\n", + "\n", + "Date: **October 13-17, 2025**" + ] + }, + { + "cell_type": "markdown", + "id": "ab0e660a", + "metadata": { + "editable": true + }, + "source": [ + "## Lecture October 13, 2025\n", + "1. Building our own Feed-forward Neural Network and discussion of project 2\n", + "\n", + "2. Project 2 is available at " + ] + }, + { + "cell_type": "markdown", + "id": "bcea021f", + "metadata": { + "editable": true + }, + "source": [ + "## Readings and videos\n", + "1. These lecture notes\n", + "\n", + "\n", + "\n", + "2. For a more in depth discussion on neural networks we recommend Goodfellow et al chapters 6 and 7. For the optimization part, see chapter 8. \n", + "\n", + "3. Neural Networks demystified at \n", + "\n", + "4. Building Neural Networks from scratch at \n", + "\n", + "5. Video on Neural Networks at \n", + "\n", + "6. Video on the back propagation algorithm at \n", + "\n", + "I also recommend Michael Nielsen's intuitive approach to the neural networks and the universal approximation theorem, see the slides at ." + ] + }, + { + "cell_type": "markdown", + "id": "fd77cd6b", + "metadata": { + "editable": true + }, + "source": [ + "## Material for the lab sessions on Tuesday and Wednesday\n", + "1. Exercises on writing a code for neural networks, back propagation part, see exercises for week 42 at \n", + "\n", + "2. Discussion of project 2" + ] + }, + { + "cell_type": "markdown", + "id": "7d93059a", + "metadata": { + "editable": true + }, + "source": [ + "## Lecture material: Writing a code which implements a feed-forward neural network\n", + "\n", + "Last week we discussed the basics of neural networks and deep learning\n", + "and the basics of automatic differentiation. We looked also at\n", + "examples on how compute the parameters of a simple network with scalar\n", + "inputs and ouputs and no or just one hidden layers.\n", + "\n", + "We ended our discussions with the derivation of the equations for a\n", + "neural network with one hidden layers and two input variables and two\n", + "hidden nodes but only one output node. We did almost finish the derivation of the back propagation algorithm." + ] + }, + { + "cell_type": "markdown", + "id": "5156bf60", + "metadata": { + "editable": true + }, + "source": [ + "## Mathematics of deep learning\n", + "\n", + "**Two recent books online.**\n", + "\n", + "1. [The Modern Mathematics of Deep Learning, by Julius Berner, Philipp Grohs, Gitta Kutyniok, Philipp Petersen](https://arxiv.org/abs/2105.04026), published as [Mathematical Aspects of Deep Learning, pp. 1-111. Cambridge University Press, 2022](https://doi.org/10.1017/9781009025096.002)\n", + "\n", + "2. [Mathematical Introduction to Deep Learning: Methods, Implementations, and Theory, Arnulf Jentzen, Benno Kuckuck, Philippe von Wurstemberger](https://doi.org/10.48550/arXiv.2310.20360)" + ] + }, + { + "cell_type": "markdown", + "id": "4f5d5b27", + "metadata": { + "editable": true + }, + "source": [ + "## Reminder on books with hands-on material and codes\n", + "* [Sebastian Rashcka et al, Machine learning with Sickit-Learn and PyTorch](https://sebastianraschka.com/blog/2022/ml-pytorch-book.html)" + ] + }, + { + "cell_type": "markdown", + "id": "0a490014", + "metadata": { + "editable": true + }, + "source": [ + "## Reading recommendations\n", + "\n", + "1. Rashkca et al., chapter 11, jupyter-notebook sent separately, from [GitHub](https://github.com/rasbt/machine-learning-book)\n", + "\n", + "2. Goodfellow et al, chapter 6 and 7 contain most of the neural network background." + ] + }, + { + "cell_type": "markdown", + "id": "9fa74383", + "metadata": { + "editable": true + }, + "source": [ + "## Reminder from last week: First network example, simple percepetron with one input\n", + "\n", + "As yet another example we define now a simple perceptron model with\n", + "all quantities given by scalars. We consider only one input variable\n", + "$x$ and one target value $y$. We define an activation function\n", + "$\\sigma_1$ which takes as input" + ] + }, + { + "cell_type": "markdown", + "id": "417ce076", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "z_1 = w_1x+b_1,\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "07ed6afb", + "metadata": { + "editable": true + }, + "source": [ + "where $w_1$ is the weight and $b_1$ is the bias. These are the\n", + "parameters we want to optimize. The output is $a_1=\\sigma(z_1)$ (see\n", + "graph from whiteboard notes). This output is then fed into the\n", + "**cost/loss** function, which we here for the sake of simplicity just\n", + "define as the squared error" + ] + }, + { + "cell_type": "markdown", + "id": "3b926536", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "C(x;w_1,b_1)=\\frac{1}{2}(a_1-y)^2.\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "0e1bd556", + "metadata": { + "editable": true + }, + "source": [ + "## Layout of a simple neural network with no hidden layer\n", + "\n", + "\n", + "\n", + "\n", + "

Figure 1:

\n", + "" + ] + }, + { + "cell_type": "markdown", + "id": "edebc935", + "metadata": { + "editable": true + }, + "source": [ + "## Optimizing the parameters\n", + "\n", + "In setting up the feed forward and back propagation parts of the\n", + "algorithm, we need now the derivative of the various variables we want\n", + "to train.\n", + "\n", + "We need" + ] + }, + { + "cell_type": "markdown", + "id": "3e2794ac", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\frac{\\partial C}{\\partial w_1} \\hspace{0.1cm}\\mathrm{and}\\hspace{0.1cm}\\frac{\\partial C}{\\partial b_1}.\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "54f91430", + "metadata": { + "editable": true + }, + "source": [ + "Using the chain rule we find" + ] + }, + { + "cell_type": "markdown", + "id": "f1e0a166", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\frac{\\partial C}{\\partial w_1}=\\frac{\\partial C}{\\partial a_1}\\frac{\\partial a_1}{\\partial z_1}\\frac{\\partial z_1}{\\partial w_1}=(a_1-y)\\sigma_1'x,\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "ad62ed90", + "metadata": { + "editable": true + }, + "source": [ + "and" + ] + }, + { + "cell_type": "markdown", + "id": "57dac362", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\frac{\\partial C}{\\partial b_1}=\\frac{\\partial C}{\\partial a_1}\\frac{\\partial a_1}{\\partial z_1}\\frac{\\partial z_1}{\\partial b_1}=(a_1-y)\\sigma_1',\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "a4dc4050", + "metadata": { + "editable": true + }, + "source": [ + "which we later will just define as" + ] + }, + { + "cell_type": "markdown", + "id": "b223c83d", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\frac{\\partial C}{\\partial a_1}\\frac{\\partial a_1}{\\partial z_1}=\\delta_1.\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "a1c9b5e7", + "metadata": { + "editable": true + }, + "source": [ + "## Adding a hidden layer\n", + "\n", + "We change our simple model to (see graph)\n", + "a network with just one hidden layer but with scalar variables only.\n", + "\n", + "Our output variable changes to $a_2$ and $a_1$ is now the output from the hidden node and $a_0=x$.\n", + "We have then" + ] + }, + { + "cell_type": "markdown", + "id": "9c3b03c0", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "z_1 = w_1a_0+b_1 \\hspace{0.1cm} \\wedge a_1 = \\sigma_1(z_1),\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "186485f2", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "z_2 = w_2a_1+b_2 \\hspace{0.1cm} \\wedge a_2 = \\sigma_2(z_2),\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "c2f1b30b", + "metadata": { + "editable": true + }, + "source": [ + "and the cost function" + ] + }, + { + "cell_type": "markdown", + "id": "13142fad", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "C(x;\\boldsymbol{\\Theta})=\\frac{1}{2}(a_2-y)^2,\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "b441f95a", + "metadata": { + "editable": true + }, + "source": [ + "with $\\boldsymbol{\\Theta}=[w_1,w_2,b_1,b_2]$." + ] + }, + { + "cell_type": "markdown", + "id": "46e57fd2", + "metadata": { + "editable": true + }, + "source": [ + "## Layout of a simple neural network with one hidden layer\n", + "\n", + "\n", + "\n", + "\n", + "

Figure 1:

\n", + "" + ] + }, + { + "cell_type": "markdown", + "id": "50158555", + "metadata": { + "editable": true + }, + "source": [ + "## The derivatives\n", + "\n", + "The derivatives are now, using the chain rule again" + ] + }, + { + "cell_type": "markdown", + "id": "4174ea9e", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\frac{\\partial C}{\\partial w_2}=\\frac{\\partial C}{\\partial a_2}\\frac{\\partial a_2}{\\partial z_2}\\frac{\\partial z_2}{\\partial w_2}=(a_2-y)\\sigma_2'a_1=\\delta_2a_1,\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "f7a29d74", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\frac{\\partial C}{\\partial b_2}=\\frac{\\partial C}{\\partial a_2}\\frac{\\partial a_2}{\\partial z_2}\\frac{\\partial z_2}{\\partial b_2}=(a_2-y)\\sigma_2'=\\delta_2,\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "cb1a387b", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\frac{\\partial C}{\\partial w_1}=\\frac{\\partial C}{\\partial a_2}\\frac{\\partial a_2}{\\partial z_2}\\frac{\\partial z_2}{\\partial a_1}\\frac{\\partial a_1}{\\partial z_1}\\frac{\\partial z_1}{\\partial w_1}=(a_2-y)\\sigma_2'a_1\\sigma_1'a_0,\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "125bcb29", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\frac{\\partial C}{\\partial b_1}=\\frac{\\partial C}{\\partial a_2}\\frac{\\partial a_2}{\\partial z_2}\\frac{\\partial z_2}{\\partial a_1}\\frac{\\partial a_1}{\\partial z_1}\\frac{\\partial z_1}{\\partial b_1}=(a_2-y)\\sigma_2'\\sigma_1'=\\delta_1.\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "76272ae9", + "metadata": { + "editable": true + }, + "source": [ + "Can you generalize this to more than one hidden layer?" + ] + }, + { + "cell_type": "markdown", + "id": "7e0d1157", + "metadata": { + "editable": true + }, + "source": [ + "## Important observations\n", + "\n", + "From the above equations we see that the derivatives of the activation\n", + "functions play a central role. If they vanish, the training may\n", + "stop. This is called the vanishing gradient problem, see discussions below. If they become\n", + "large, the parameters $w_i$ and $b_i$ may simply go to infinity. This\n", + "is referenced as the exploding gradient problem." + ] + }, + { + "cell_type": "markdown", + "id": "53dfc3bd", + "metadata": { + "editable": true + }, + "source": [ + "## The training\n", + "\n", + "The training of the parameters is done through various gradient descent approximations with" + ] + }, + { + "cell_type": "markdown", + "id": "126420bb", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "w_{i}\\leftarrow w_{i}- \\eta \\delta_i a_{i-1},\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "2114b4ba", + "metadata": { + "editable": true + }, + "source": [ + "and" + ] + }, + { + "cell_type": "markdown", + "id": "7d3549bb", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "b_i \\leftarrow b_i-\\eta \\delta_i,\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "80265b39", + "metadata": { + "editable": true + }, + "source": [ + "with $\\eta$ is the learning rate.\n", + "\n", + "One iteration consists of one feed forward step and one back-propagation step. Each back-propagation step does one update of the parameters $\\boldsymbol{\\Theta}$.\n", + "\n", + "For the first hidden layer $a_{i-1}=a_0=x$ for this simple model." + ] + }, + { + "cell_type": "markdown", + "id": "df2f78ac", + "metadata": { + "editable": true + }, + "source": [ + "## Code example\n", + "\n", + "The code here implements the above model with one hidden layer and\n", + "scalar variables for the same function we studied in the previous\n", + "example. The code is however set up so that we can add multiple\n", + "inputs $x$ and target values $y$. Note also that we have the\n", + "possibility of defining a feature matrix $\\boldsymbol{X}$ with more than just\n", + "one column for the input values. This will turn useful in our next example. We have also defined matrices and vectors for all of our operations although it is not necessary here." + ] + }, + { + "cell_type": "code", + "execution_count": 1, + "id": "0ec652bf", + "metadata": { + "collapsed": false, + "editable": true + }, + "outputs": [], + "source": [ + "import numpy as np\n", + "# We use the Sigmoid function as activation function\n", + "def sigmoid(z):\n", + " return 1.0/(1.0+np.exp(-z))\n", + "\n", + "def forwardpropagation(x):\n", + " # weighted sum of inputs to the hidden layer\n", + " z_1 = np.matmul(x, w_1) + b_1\n", + " # activation in the hidden layer\n", + " a_1 = sigmoid(z_1)\n", + " # weighted sum of inputs to the output layer\n", + " z_2 = np.matmul(a_1, w_2) + b_2\n", + " a_2 = z_2\n", + " return a_1, a_2\n", + "\n", + "def backpropagation(x, y):\n", + " a_1, a_2 = forwardpropagation(x)\n", + " # parameter delta for the output layer, note that a_2=z_2 and its derivative wrt z_2 is just 1\n", + " delta_2 = a_2 - y\n", + " print(0.5*((a_2-y)**2))\n", + " # delta for the hidden layer\n", + " delta_1 = np.matmul(delta_2, w_2.T) * a_1 * (1 - a_1)\n", + " # gradients for the output layer\n", + " output_weights_gradient = np.matmul(a_1.T, delta_2)\n", + " output_bias_gradient = np.sum(delta_2, axis=0)\n", + " # gradient for the hidden layer\n", + " hidden_weights_gradient = np.matmul(x.T, delta_1)\n", + " hidden_bias_gradient = np.sum(delta_1, axis=0)\n", + " return output_weights_gradient, output_bias_gradient, hidden_weights_gradient, hidden_bias_gradient\n", + "\n", + "\n", + "# ensure the same random numbers appear every time\n", + "np.random.seed(0)\n", + "# Input variable\n", + "x = np.array([4.0],dtype=np.float64)\n", + "# Target values\n", + "y = 2*x+1.0 \n", + "\n", + "# Defining the neural network, only scalars here\n", + "n_inputs = x.shape\n", + "n_features = 1\n", + "n_hidden_neurons = 1\n", + "n_outputs = 1\n", + "\n", + "# Initialize the network\n", + "# weights and bias in the hidden layer\n", + "w_1 = np.random.randn(n_features, n_hidden_neurons)\n", + "b_1 = np.zeros(n_hidden_neurons) + 0.01\n", + "\n", + "# weights and bias in the output layer\n", + "w_2 = np.random.randn(n_hidden_neurons, n_outputs)\n", + "b_2 = np.zeros(n_outputs) + 0.01\n", + "\n", + "eta = 0.1\n", + "for i in range(50):\n", + " # calculate gradients\n", + " derivW2, derivB2, derivW1, derivB1 = backpropagation(x, y)\n", + " # update weights and biases\n", + " w_2 -= eta * derivW2\n", + " b_2 -= eta * derivB2\n", + " w_1 -= eta * derivW1\n", + " b_1 -= eta * derivB1" + ] + }, + { + "cell_type": "markdown", + "id": "c382e2f2", + "metadata": { + "editable": true + }, + "source": [ + "We see that after some few iterations (the results do depend on the learning rate however), we get an error which is rather small." + ] + }, + { + "cell_type": "markdown", + "id": "1bec2330", + "metadata": { + "editable": true + }, + "source": [ + "## Simple neural network and the back propagation equations\n", + "\n", + "Let us now try to increase our level of ambition and attempt at setting \n", + "up the equations for a neural network with two input nodes, one hidden\n", + "layer with two hidden nodes and one output layer with one output node/neuron only (see graph)..\n", + "\n", + "We need to define the following parameters and variables with the input layer (layer $(0)$) \n", + "where we label the nodes $x_1$ and $x_2$" + ] + }, + { + "cell_type": "markdown", + "id": "165fb3f3", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "x_1 = a_1^{(0)} \\wedge x_2 = a_2^{(0)}.\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "7ae8de36", + "metadata": { + "editable": true + }, + "source": [ + "The hidden layer (layer $(1)$) has nodes which yield the outputs $a_1^{(1)}$ and $a_2^{(1)}$) with weight $\\boldsymbol{w}$ and bias $\\boldsymbol{b}$ parameters" + ] + }, + { + "cell_type": "markdown", + "id": "9a2729ab", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "w_{ij}^{(1)}=\\left\\{w_{11}^{(1)},w_{12}^{(1)},w_{21}^{(1)},w_{22}^{(1)}\\right\\} \\wedge b^{(1)}=\\left\\{b_1^{(1)},b_2^{(1)}\\right\\}.\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "372f0fac", + "metadata": { + "editable": true + }, + "source": [ + "## Layout of a simple neural network with two input nodes, one hidden layer with two hidden noeds and one output node\n", + "\n", + "\n", + "\n", + "\n", + "

Figure 1:

\n", + "" + ] + }, + { + "cell_type": "markdown", + "id": "d4f432cc", + "metadata": { + "editable": true + }, + "source": [ + "## The ouput layer\n", + "\n", + "We have the ouput layer given by layer label $(2)$ with output $a^{(2)}$ and weights and biases to be determined given by the variables" + ] + }, + { + "cell_type": "markdown", + "id": "064af49b", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "w_{i}^{(2)}=\\left\\{w_{1}^{(2)},w_{2}^{(2)}\\right\\} \\wedge b^{(2)}.\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "91115cc6", + "metadata": { + "editable": true + }, + "source": [ + "Our output is $\\tilde{y}=a^{(2)}$ and we define a generic cost function $C(a^{(2)},y;\\boldsymbol{\\Theta})$ where $y$ is the target value (a scalar here).\n", + "The parameters we need to optimize are given by" + ] + }, + { + "cell_type": "markdown", + "id": "c38d1990", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\boldsymbol{\\Theta}=\\left\\{w_{11}^{(1)},w_{12}^{(1)},w_{21}^{(1)},w_{22}^{(1)},w_{1}^{(2)},w_{2}^{(2)},b_1^{(1)},b_2^{(1)},b^{(2)}\\right\\}.\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "43de152e", + "metadata": { + "editable": true + }, + "source": [ + "## Compact expressions\n", + "\n", + "We can define the inputs to the activation functions for the various layers in terms of various matrix-vector multiplications and vector additions.\n", + "The inputs to the first hidden layer are" + ] + }, + { + "cell_type": "markdown", + "id": "f935912b", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\begin{bmatrix}z_1^{(1)} \\\\ z_2^{(1)} \\end{bmatrix}=\\left(\\begin{bmatrix}w_{11}^{(1)} & w_{12}^{(1)}\\\\ w_{21}^{(1)} &w_{22}^{(1)} \\end{bmatrix}\\right)^{T}\\begin{bmatrix}a_1^{(0)} \\\\ a_2^{(0)} \\end{bmatrix}+\\begin{bmatrix}b_1^{(1)} \\\\ b_2^{(1)} \\end{bmatrix},\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "15b380de", + "metadata": { + "editable": true + }, + "source": [ + "with outputs" + ] + }, + { + "cell_type": "markdown", + "id": "302da250", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\begin{bmatrix}a_1^{(1)} \\\\ a_2^{(1)} \\end{bmatrix}=\\begin{bmatrix}\\sigma^{(1)}(z_1^{(1)}) \\\\ \\sigma^{(1)}(z_2^{(1)}) \\end{bmatrix}.\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "3faf23cf", + "metadata": { + "editable": true + }, + "source": [ + "## Output layer\n", + "\n", + "For the final output layer we have the inputs to the final activation function" + ] + }, + { + "cell_type": "markdown", + "id": "4fe83376", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "z^{(2)} = w_{1}^{(2)}a_1^{(1)} +w_{2}^{(2)}a_2^{(1)}+b^{(2)},\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "9bee26b1", + "metadata": { + "editable": true + }, + "source": [ + "resulting in the output" + ] + }, + { + "cell_type": "markdown", + "id": "9affa08f", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "a^{(2)}=\\sigma^{(2)}(z^{(2)}).\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "1866378d", + "metadata": { + "editable": true + }, + "source": [ + "## Explicit derivatives\n", + "\n", + "In total we have nine parameters which we need to train. Using the\n", + "chain rule (or just the back-propagation algorithm) we can find all\n", + "derivatives. Since we will use automatic differentiation in reverse\n", + "mode, we start with the derivatives of the cost function with respect\n", + "to the parameters of the output layer, namely" + ] + }, + { + "cell_type": "markdown", + "id": "e6d27f9d", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\frac{\\partial C}{\\partial w_{i}^{(2)}}=\\frac{\\partial C}{\\partial a^{(2)}}\\frac{\\partial a^{(2)}}{\\partial z^{(2)}}\\frac{\\partial z^{(2)}}{\\partial w_{i}^{(2)}}=\\delta^{(2)}a_i^{(1)},\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "b33dac74", + "metadata": { + "editable": true + }, + "source": [ + "with" + ] + }, + { + "cell_type": "markdown", + "id": "27c367b4", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\delta^{(2)}=\\frac{\\partial C}{\\partial a^{(2)}}\\frac{\\partial a^{(2)}}{\\partial z^{(2)}}\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "18898e14", + "metadata": { + "editable": true + }, + "source": [ + "and finally" + ] + }, + { + "cell_type": "markdown", + "id": "31fcc7f9", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\frac{\\partial C}{\\partial b^{(2)}}=\\frac{\\partial C}{\\partial a^{(2)}}\\frac{\\partial a^{(2)}}{\\partial z^{(2)}}\\frac{\\partial z^{(2)}}{\\partial b^{(2)}}=\\delta^{(2)}.\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "76219fdd", + "metadata": { + "editable": true + }, + "source": [ + "## Derivatives of the hidden layer\n", + "\n", + "Using the chain rule we have the following expressions for say one of the weight parameters (it is easy to generalize to the other weight parameters)" + ] + }, + { + "cell_type": "markdown", + "id": "2cbccf94", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\frac{\\partial C}{\\partial w_{11}^{(1)}}=\\frac{\\partial C}{\\partial a^{(2)}}\\frac{\\partial a^{(2)}}{\\partial z^{(2)}}\n", + "\\frac{\\partial z^{(2)}}{\\partial z_1^{(1)}}\\frac{\\partial z_1^{(1)}}{\\partial w_{11}^{(1)}}= \\delta^{(2)}\\frac{\\partial z^{(2)}}{\\partial z_1^{(1)}}\\frac{\\partial z_1^{(1)}}{\\partial w_{11}^{(1)}},\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "ccdb071b", + "metadata": { + "editable": true + }, + "source": [ + "which, noting that" + ] + }, + { + "cell_type": "markdown", + "id": "55e4bed1", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "z^{(2)} =w_1^{(2)}a_1^{(1)}+w_2^{(2)}a_2^{(1)}+b^{(2)},\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "c00fd151", + "metadata": { + "editable": true + }, + "source": [ + "allows us to rewrite" + ] + }, + { + "cell_type": "markdown", + "id": "d49ae60d", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\frac{\\partial z^{(2)}}{\\partial z_1^{(1)}}\\frac{\\partial z_1^{(1)}}{\\partial w_{11}^{(1)}}=w_1^{(2)}\\frac{\\partial a_1^{(1)}}{\\partial z_1^{(1)}}a_1^{(1)}.\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "920db2dd", + "metadata": { + "editable": true + }, + "source": [ + "## Final expression\n", + "Defining" + ] + }, + { + "cell_type": "markdown", + "id": "62806f12", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\delta_1^{(1)}=w_1^{(2)}\\frac{\\partial a_1^{(1)}}{\\partial z_1^{(1)}}\\delta^{(2)},\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "f1d3e7b6", + "metadata": { + "editable": true + }, + "source": [ + "we have" + ] + }, + { + "cell_type": "markdown", + "id": "51fd0b81", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\frac{\\partial C}{\\partial w_{11}^{(1)}}=\\delta_1^{(1)}a_1^{(1)}.\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "ba9dd4ad", + "metadata": { + "editable": true + }, + "source": [ + "Similarly, we obtain" + ] + }, + { + "cell_type": "markdown", + "id": "91137440", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\frac{\\partial C}{\\partial w_{12}^{(1)}}=\\delta_1^{(1)}a_2^{(1)}.\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "8e2ecb8c", + "metadata": { + "editable": true + }, + "source": [ + "## Completing the list\n", + "\n", + "Similarly, we find" + ] + }, + { + "cell_type": "markdown", + "id": "8d91daf9", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\frac{\\partial C}{\\partial w_{21}^{(1)}}=\\delta_2^{(1)}a_1^{(1)},\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "8450f964", + "metadata": { + "editable": true + }, + "source": [ + "and" + ] + }, + { + "cell_type": "markdown", + "id": "1e662a52", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\frac{\\partial C}{\\partial w_{22}^{(1)}}=\\delta_2^{(1)}a_2^{(1)},\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "e90d34b5", + "metadata": { + "editable": true + }, + "source": [ + "where we have defined" + ] + }, + { + "cell_type": "markdown", + "id": "2d4b194d", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\delta_2^{(1)}=w_2^{(2)}\\frac{\\partial a_2^{(1)}}{\\partial z_2^{(1)}}\\delta^{(2)}.\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "8990fe0b", + "metadata": { + "editable": true + }, + "source": [ + "## Final expressions for the biases of the hidden layer\n", + "\n", + "For the sake of completeness, we list the derivatives of the biases, which are" + ] + }, + { + "cell_type": "markdown", + "id": "37e9e142", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\frac{\\partial C}{\\partial b_{1}^{(1)}}=\\delta_1^{(1)},\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "5f66c64a", + "metadata": { + "editable": true + }, + "source": [ + "and" + ] + }, + { + "cell_type": "markdown", + "id": "03e836ad", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\frac{\\partial C}{\\partial b_{2}^{(1)}}=\\delta_2^{(1)}.\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "1ea8a6ff", + "metadata": { + "editable": true + }, + "source": [ + "As we will see below, these expressions can be generalized in a more compact form." + ] + }, + { + "cell_type": "markdown", + "id": "84537cc9", + "metadata": { + "editable": true + }, + "source": [ + "## Gradient expressions\n", + "\n", + "For this specific model, with just one output node and two hidden\n", + "nodes, the gradient descent equations take the following form for output layer" + ] + }, + { + "cell_type": "markdown", + "id": "2bad6141", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "w_{i}^{(2)}\\leftarrow w_{i}^{(2)}- \\eta \\delta^{(2)} a_{i}^{(1)},\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "5150b413", + "metadata": { + "editable": true + }, + "source": [ + "and" + ] + }, + { + "cell_type": "markdown", + "id": "4166af55", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "b^{(2)} \\leftarrow b^{(2)}-\\eta \\delta^{(2)},\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "e5101f70", + "metadata": { + "editable": true + }, + "source": [ + "and" + ] + }, + { + "cell_type": "markdown", + "id": "0c662571", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "w_{ij}^{(1)}\\leftarrow w_{ij}^{(1)}- \\eta \\delta_{i}^{(1)} a_{j}^{(0)},\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "a7e1af99", + "metadata": { + "editable": true + }, + "source": [ + "and" + ] + }, + { + "cell_type": "markdown", + "id": "5a73623e", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "b_{i}^{(1)} \\leftarrow b_{i}^{(1)}-\\eta \\delta_{i}^{(1)},\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "b1d5327b", + "metadata": { + "editable": true + }, + "source": [ + "where $\\eta$ is the learning rate." + ] + }, + { + "cell_type": "markdown", + "id": "5cc5921c", + "metadata": { + "editable": true + }, + "source": [ + "## Setting up the equations for a neural network\n", + "\n", + "The questions we want to ask are how do changes in the biases and the\n", + "weights in our network change the cost function and how can we use the\n", + "final output to modify the weights and biases?\n", + "\n", + "To derive these equations let us start with a plain regression problem\n", + "and define our cost function as" + ] + }, + { + "cell_type": "markdown", + "id": "21633b14", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "{\\cal C}(\\boldsymbol{\\Theta}) = \\frac{1}{2}\\sum_{i=1}^n\\left(y_i - \\tilde{y}_i\\right)^2,\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "171782c4", + "metadata": { + "editable": true + }, + "source": [ + "where the $y_i$s are our $n$ targets (the values we want to\n", + "reproduce), while the outputs of the network after having propagated\n", + "all inputs $\\boldsymbol{x}$ are given by $\\boldsymbol{\\tilde{y}}_i$." + ] + }, + { + "cell_type": "markdown", + "id": "830e0cc3", + "metadata": { + "editable": true + }, + "source": [ + "## Layout of a neural network with three hidden layers (last layer = $l=L=4$, first layer $l=0$)\n", + "\n", + "\n", + "\n", + "\n", + "

Figure 1:

\n", + "" + ] + }, + { + "cell_type": "markdown", + "id": "3de97402", + "metadata": { + "editable": true + }, + "source": [ + "## Definitions\n", + "\n", + "With our definition of the targets $\\boldsymbol{y}$, the outputs of the\n", + "network $\\boldsymbol{\\tilde{y}}$ and the inputs $\\boldsymbol{x}$ we\n", + "define now the activation $z_j^l$ of node/neuron/unit $j$ of the\n", + "$l$-th layer as a function of the bias, the weights which add up from\n", + "the previous layer $l-1$ and the forward passes/outputs\n", + "$\\boldsymbol{a}^{l-1}$ from the previous layer as" + ] + }, + { + "cell_type": "markdown", + "id": "f4a20e55", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "z_j^l = \\sum_{i=1}^{M_{l-1}}w_{ij}^la_i^{l-1}+b_j^l,\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "e373e6cc", + "metadata": { + "editable": true + }, + "source": [ + "where $b_k^l$ are the biases from layer $l$. Here $M_{l-1}$\n", + "represents the total number of nodes/neurons/units of layer $l-1$. The\n", + "figure in the whiteboard notes illustrates this equation. We can rewrite this in a more\n", + "compact form as the matrix-vector products we discussed earlier," + ] + }, + { + "cell_type": "markdown", + "id": "a9fa6a69", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\boldsymbol{z}^l = \\left(\\boldsymbol{W}^l\\right)^T\\boldsymbol{a}^{l-1}+\\boldsymbol{b}^l.\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "f1730e5a", + "metadata": { + "editable": true + }, + "source": [ + "## Inputs to the activation function\n", + "\n", + "With the activation values $\\boldsymbol{z}^l$ we can in turn define the\n", + "output of layer $l$ as $\\boldsymbol{a}^l = \\sigma(\\boldsymbol{z}^l)$ where $\\sigma$ is our\n", + "activation function. In the examples here we will use the sigmoid\n", + "function discussed in our logistic regression lectures. We will also use the same activation function $\\sigma$ for all layers\n", + "and their nodes. It means we have" + ] + }, + { + "cell_type": "markdown", + "id": "59d7d98a", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "a_j^l = \\sigma(z_j^l) = \\frac{1}{1+\\exp{-(z_j^l)}}.\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "5a3d0931", + "metadata": { + "editable": true + }, + "source": [ + "## Layout of input to first hidden layer $l=1$ from input layer $l=0$\n", + "\n", + "\n", + "\n", + "\n", + "

Figure 1:

\n", + "" + ] + }, + { + "cell_type": "markdown", + "id": "c56d749e", + "metadata": { + "editable": true + }, + "source": [ + "## Derivatives and the chain rule\n", + "\n", + "From the definition of the input variable to the activation function, that is $z_j^l$ we have" + ] + }, + { + "cell_type": "markdown", + "id": "f3dd37e7", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\frac{\\partial z_j^l}{\\partial w_{ij}^l} = a_i^{l-1},\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "7654c47c", + "metadata": { + "editable": true + }, + "source": [ + "and" + ] + }, + { + "cell_type": "markdown", + "id": "cb1aa2a9", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\frac{\\partial z_j^l}{\\partial a_i^{l-1}} = w_{ji}^l.\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "11607454", + "metadata": { + "editable": true + }, + "source": [ + "With our definition of the activation function we have that (note that this function depends only on $z_j^l$)" + ] + }, + { + "cell_type": "markdown", + "id": "29a881cd", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\frac{\\partial a_j^l}{\\partial z_j^{l}} = a_j^l(1-a_j^l)=\\sigma(z_j^l)(1-\\sigma(z_j^l)).\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "1244c442", + "metadata": { + "editable": true + }, + "source": [ + "## Derivative of the cost function\n", + "\n", + "With these definitions we can now compute the derivative of the cost function in terms of the weights.\n", + "\n", + "Let us specialize to the output layer $l=L$. Our cost function is" + ] + }, + { + "cell_type": "markdown", + "id": "7d43f636", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "{\\cal C}(\\boldsymbol{\\Theta}^L) = \\frac{1}{2}\\sum_{i=1}^n\\left(y_i - \\tilde{y}_i\\right)^2=\\frac{1}{2}\\sum_{i=1}^n\\left(a_i^L - y_i\\right)^2,\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "c23708ab", + "metadata": { + "editable": true + }, + "source": [ + "The derivative of this function with respect to the weights is" + ] + }, + { + "cell_type": "markdown", + "id": "3a3d49b9", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\frac{\\partial{\\cal C}(\\boldsymbol{\\Theta}^L)}{\\partial w_{ij}^L} = \\left(a_j^L - y_j\\right)\\frac{\\partial a_j^L}{\\partial w_{ij}^{L}},\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "31b8b489", + "metadata": { + "editable": true + }, + "source": [ + "The last partial derivative can easily be computed and reads (by applying the chain rule)" + ] + }, + { + "cell_type": "markdown", + "id": "2cb11e87", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\frac{\\partial a_j^L}{\\partial w_{ij}^{L}} = \\frac{\\partial a_j^L}{\\partial z_{j}^{L}}\\frac{\\partial z_j^L}{\\partial w_{ij}^{L}}=a_j^L(1-a_j^L)a_i^{L-1}.\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "85b6783b", + "metadata": { + "editable": true + }, + "source": [ + "## The back propagation equations for a neural network\n", + "\n", + "We have thus" + ] + }, + { + "cell_type": "markdown", + "id": "1d9102e8", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\frac{\\partial{\\cal C}((\\boldsymbol{\\Theta}^L)}{\\partial w_{ij}^L} = \\left(a_j^L - y_j\\right)a_j^L(1-a_j^L)a_i^{L-1},\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "92547c80", + "metadata": { + "editable": true + }, + "source": [ + "Defining" + ] + }, + { + "cell_type": "markdown", + "id": "e06d7f47", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\delta_j^L = a_j^L(1-a_j^L)\\left(a_j^L - y_j\\right) = \\sigma'(z_j^L)\\frac{\\partial {\\cal C}}{\\partial (a_j^L)},\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "a15b2c15", + "metadata": { + "editable": true + }, + "source": [ + "and using the Hadamard product of two vectors we can write this as" + ] + }, + { + "cell_type": "markdown", + "id": "69a96bb6", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\boldsymbol{\\delta}^L = \\sigma'(\\boldsymbol{z}^L)\\circ\\frac{\\partial {\\cal C}}{\\partial (\\boldsymbol{a}^L)}.\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "6954d50b", + "metadata": { + "editable": true + }, + "source": [ + "## Analyzing the last results\n", + "\n", + "This is an important expression. The second term on the right handside\n", + "measures how fast the cost function is changing as a function of the $j$th\n", + "output activation. If, for example, the cost function doesn't depend\n", + "much on a particular output node $j$, then $\\delta_j^L$ will be small,\n", + "which is what we would expect. The first term on the right, measures\n", + "how fast the activation function $f$ is changing at a given activation\n", + "value $z_j^L$." + ] + }, + { + "cell_type": "markdown", + "id": "a8d2b720", + "metadata": { + "editable": true + }, + "source": [ + "## More considerations\n", + "\n", + "Notice that everything in the above equations is easily computed. In\n", + "particular, we compute $z_j^L$ while computing the behaviour of the\n", + "network, and it is only a small additional overhead to compute\n", + "$\\sigma'(z^L_j)$. The exact form of the derivative with respect to the\n", + "output depends on the form of the cost function.\n", + "However, provided the cost function is known there should be little\n", + "trouble in calculating" + ] + }, + { + "cell_type": "markdown", + "id": "b40feaea", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\frac{\\partial {\\cal C}}{\\partial (a_j^L)}\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "04d0916c", + "metadata": { + "editable": true + }, + "source": [ + "With the definition of $\\delta_j^L$ we have a more compact definition of the derivative of the cost function in terms of the weights, namely" + ] + }, + { + "cell_type": "markdown", + "id": "4067515f", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\frac{\\partial{\\cal C}}{\\partial w_{ij}^L} = \\delta_j^La_i^{L-1}.\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "572f5043", + "metadata": { + "editable": true + }, + "source": [ + "## Derivatives in terms of $z_j^L$\n", + "\n", + "It is also easy to see that our previous equation can be written as" + ] + }, + { + "cell_type": "markdown", + "id": "4272e56c", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\delta_j^L =\\frac{\\partial {\\cal C}}{\\partial z_j^L}= \\frac{\\partial {\\cal C}}{\\partial a_j^L}\\frac{\\partial a_j^L}{\\partial z_j^L},\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "3ffcc3a4", + "metadata": { + "editable": true + }, + "source": [ + "which can also be interpreted as the partial derivative of the cost function with respect to the biases $b_j^L$, namely" + ] + }, + { + "cell_type": "markdown", + "id": "87dc5557", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\delta_j^L = \\frac{\\partial {\\cal C}}{\\partial b_j^L}\\frac{\\partial b_j^L}{\\partial z_j^L}=\\frac{\\partial {\\cal C}}{\\partial b_j^L},\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "36f9508b", + "metadata": { + "editable": true + }, + "source": [ + "That is, the error $\\delta_j^L$ is exactly equal to the rate of change of the cost function as a function of the bias." + ] + }, + { + "cell_type": "markdown", + "id": "5e7f7ab6", + "metadata": { + "editable": true + }, + "source": [ + "## Bringing it together\n", + "\n", + "We have now three equations that are essential for the computations of the derivatives of the cost function at the output layer. These equations are needed to start the algorithm and they are" + ] + }, + { + "cell_type": "markdown", + "id": "b5caac62", + "metadata": { + "editable": true + }, + "source": [ + "\n", + "
\n", + "\n", + "$$\n", + "\\begin{equation}\n", + "\\frac{\\partial{\\cal C}(\\boldsymbol{W^L})}{\\partial w_{ij}^L} = \\delta_j^La_i^{L-1},\n", + "\\label{_auto1} \\tag{1}\n", + "\\end{equation}\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "beffe853", + "metadata": { + "editable": true + }, + "source": [ + "and" + ] + }, + { + "cell_type": "markdown", + "id": "9ff991bb", + "metadata": { + "editable": true + }, + "source": [ + "\n", + "
\n", + "\n", + "$$\n", + "\\begin{equation}\n", + "\\delta_j^L = \\sigma'(z_j^L)\\frac{\\partial {\\cal C}}{\\partial (a_j^L)},\n", + "\\label{_auto2} \\tag{2}\n", + "\\end{equation}\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "5c75d805", + "metadata": { + "editable": true + }, + "source": [ + "and" + ] + }, + { + "cell_type": "markdown", + "id": "df242f2d", + "metadata": { + "editable": true + }, + "source": [ + "\n", + "
\n", + "\n", + "$$\n", + "\\begin{equation}\n", + "\\delta_j^L = \\frac{\\partial {\\cal C}}{\\partial b_j^L},\n", + "\\label{_auto3} \\tag{3}\n", + "\\end{equation}\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "18a65420", + "metadata": { + "editable": true + }, + "source": [ + "## Final back propagating equation\n", + "\n", + "We have that (replacing $L$ with a general layer $l$)" + ] + }, + { + "cell_type": "markdown", + "id": "04cb64d1", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\delta_j^l =\\frac{\\partial {\\cal C}}{\\partial z_j^l}.\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "c5320606", + "metadata": { + "editable": true + }, + "source": [ + "We want to express this in terms of the equations for layer $l+1$." + ] + }, + { + "cell_type": "markdown", + "id": "58b67295", + "metadata": { + "editable": true + }, + "source": [ + "## Using the chain rule and summing over all $k$ entries\n", + "\n", + "We obtain" + ] + }, + { + "cell_type": "markdown", + "id": "3b7c74a4", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\delta_j^l =\\sum_k \\frac{\\partial {\\cal C}}{\\partial z_k^{l+1}}\\frac{\\partial z_k^{l+1}}{\\partial z_j^{l}}=\\sum_k \\delta_k^{l+1}\\frac{\\partial z_k^{l+1}}{\\partial z_j^{l}},\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "76ff59e0", + "metadata": { + "editable": true + }, + "source": [ + "and recalling that" + ] + }, + { + "cell_type": "markdown", + "id": "60a7be03", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "z_j^{l+1} = \\sum_{i=1}^{M_{l}}w_{ij}^{l+1}a_i^{l}+b_j^{l+1},\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "27739d6d", + "metadata": { + "editable": true + }, + "source": [ + "with $M_l$ being the number of nodes in layer $l$, we obtain" + ] + }, + { + "cell_type": "markdown", + "id": "1efdb6f5", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\delta_j^l =\\sum_k \\delta_k^{l+1}w_{kj}^{l+1}\\sigma'(z_j^l),\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "b0b04f61", + "metadata": { + "editable": true + }, + "source": [ + "This is our final equation.\n", + "\n", + "We are now ready to set up the algorithm for back propagation and learning the weights and biases." + ] + }, + { + "cell_type": "markdown", + "id": "e6c4318c", + "metadata": { + "editable": true + }, + "source": [ + "## Setting up the back propagation algorithm and algorithm for a feed forward NN, initalizations\n", + "\n", + "**The architecture (our model).**\n", + "\n", + "1. Set up your inputs and outputs (scalars, vectors, matrices or higher-order arrays)\n", + "\n", + "2. Define the number of hidden layers and hidden nodes\n", + "\n", + "3. Define activation functions for hidden layers and output layers\n", + "\n", + "4. Define optimizer (plan learning rate, momentum, ADAgrad, RMSprop, ADAM etc) and array of initial learning rates\n", + "\n", + "5. Define cost function and possible regularization terms with hyperparameters\n", + "\n", + "6. Initialize weights and biases\n", + "\n", + "7. Fix number of iterations for the feed forward part and back propagation part" + ] + }, + { + "cell_type": "markdown", + "id": "4e58634a", + "metadata": { + "editable": true + }, + "source": [ + "## Setting up the back propagation algorithm, part 1\n", + "\n", + "The four equations provide us with a way of computing the gradients of the cost function. Let us write this out in the form of an algorithm.\n", + "\n", + "**First**, we set up the input data $\\boldsymbol{x}$ and the activations\n", + "$\\boldsymbol{z}_1$ of the input layer and compute the activation function and\n", + "the pertinent outputs $\\boldsymbol{a}^1$.\n", + "\n", + "**Secondly**, we perform then the feed forward till we reach the output\n", + "layer and compute all $\\boldsymbol{z}_l$ of the input layer and compute the\n", + "activation function and the pertinent outputs $\\boldsymbol{a}^l$ for\n", + "$l=1,2,3,\\dots,L$.\n", + "\n", + "**Notation**: The first hidden layer has $l=1$ as label and the final output layer has $l=L$." + ] + }, + { + "cell_type": "markdown", + "id": "690eb424", + "metadata": { + "editable": true + }, + "source": [ + "## Setting up the back propagation algorithm, part 2\n", + "\n", + "Thereafter we compute the ouput error $\\boldsymbol{\\delta}^L$ by computing all" + ] + }, + { + "cell_type": "markdown", + "id": "5f475d02", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\delta_j^L = \\sigma'(z_j^L)\\frac{\\partial {\\cal C}}{\\partial (a_j^L)}.\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "016ce549", + "metadata": { + "editable": true + }, + "source": [ + "Then we compute the back propagate error for each $l=L-1,L-2,\\dots,1$ as" + ] + }, + { + "cell_type": "markdown", + "id": "51f9ed82", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\delta_j^l = \\sum_k \\delta_k^{l+1}w_{kj}^{l+1}\\sigma'(z_j^l).\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "a8996176", + "metadata": { + "editable": true + }, + "source": [ + "## Setting up the Back propagation algorithm, part 3\n", + "\n", + "Finally, we update the weights and the biases using gradient descent\n", + "for each $l=L-1,L-2,\\dots,1$ (the first hidden layer) and update the weights and biases\n", + "according to the rules" + ] + }, + { + "cell_type": "markdown", + "id": "aaf186a8", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "w_{ij}^l\\leftarrow = w_{ij}^l- \\eta \\delta_j^la_i^{l-1},\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "3b8a0a79", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "b_j^l \\leftarrow b_j^l-\\eta \\frac{\\partial {\\cal C}}{\\partial b_j^l}=b_j^l-\\eta \\delta_j^l,\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "299cc7bb", + "metadata": { + "editable": true + }, + "source": [ + "with $\\eta$ being the learning rate." + ] + }, + { + "cell_type": "markdown", + "id": "3b10a1a0", + "metadata": { + "editable": true + }, + "source": [ + "## Updating the gradients\n", + "\n", + "With the back propagate error for each $l=L-1,L-2,\\dots,1$ as" + ] + }, + { + "cell_type": "markdown", + "id": "0f8769f8", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\delta_j^l = \\sum_k \\delta_k^{l+1}w_{kj}^{l+1}\\sigma'(z_j^l),\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "03860e2f", + "metadata": { + "editable": true + }, + "source": [ + "we update the weights and the biases using gradient descent for each $l=L-1,L-2,\\dots,1$ and update the weights and biases according to the rules" + ] + }, + { + "cell_type": "markdown", + "id": "00cea1da", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "w_{ij}^l\\leftarrow = w_{ij}^l- \\eta \\delta_j^la_i^{l-1},\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "fe5bbd2a", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "b_j^l \\leftarrow b_j^l-\\eta \\frac{\\partial {\\cal C}}{\\partial b_j^l}=b_j^l-\\eta \\delta_j^l,\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "e791fb56", + "metadata": { + "editable": true + }, + "source": [ + "## Activation functions\n", + "\n", + "A property that characterizes a neural network, other than its\n", + "connectivity, is the choice of activation function(s). The following\n", + "restrictions are imposed on an activation function for an FFNN to\n", + "fulfill the universal approximation theorem\n", + "\n", + " * Non-constant\n", + "\n", + " * Bounded\n", + "\n", + " * Monotonically-increasing\n", + "\n", + " * Continuous" + ] + }, + { + "cell_type": "markdown", + "id": "14ed4124", + "metadata": { + "editable": true + }, + "source": [ + "### Activation functions, Logistic and Hyperbolic ones\n", + "\n", + "The second requirement excludes all linear functions. Furthermore, in\n", + "a MLP with only linear activation functions, each layer simply\n", + "performs a linear transformation of its inputs.\n", + "\n", + "Regardless of the number of layers, the output of the NN will be\n", + "nothing but a linear function of the inputs. Thus we need to introduce\n", + "some kind of non-linearity to the NN to be able to fit non-linear\n", + "functions Typical examples are the logistic *Sigmoid*" + ] + }, + { + "cell_type": "markdown", + "id": "d6b31eb8", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\sigma(x) = \\frac{1}{1 + e^{-x}},\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "2be10af5", + "metadata": { + "editable": true + }, + "source": [ + "and the *hyperbolic tangent* function" + ] + }, + { + "cell_type": "markdown", + "id": "f45b9a99", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\sigma(x) = \\tanh(x)\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "865d802b", + "metadata": { + "editable": true + }, + "source": [ + "## Relevance\n", + "\n", + "The *sigmoid* function are more biologically plausible because the\n", + "output of inactive neurons are zero. Such activation function are\n", + "called *one-sided*. However, it has been shown that the hyperbolic\n", + "tangent performs better than the sigmoid for training MLPs. has\n", + "become the most popular for *deep neural networks*" + ] + }, + { + "cell_type": "code", + "execution_count": 2, + "id": "37773874", + "metadata": { + "collapsed": false, + "editable": true + }, + "outputs": [], + "source": [ + "%matplotlib inline\n", + "\n", + "\"\"\"The sigmoid function (or the logistic curve) is a \n", + "function that takes any real number, z, and outputs a number (0,1).\n", + "It is useful in neural networks for assigning weights on a relative scale.\n", + "The value z is the weighted sum of parameters involved in the learning algorithm.\"\"\"\n", + "\n", + "import numpy\n", + "import matplotlib.pyplot as plt\n", + "import math as mt\n", + "\n", + "z = numpy.arange(-5, 5, .1)\n", + "sigma_fn = numpy.vectorize(lambda z: 1/(1+numpy.exp(-z)))\n", + "sigma = sigma_fn(z)\n", + "\n", + "fig = plt.figure()\n", + "ax = fig.add_subplot(111)\n", + "ax.plot(z, sigma)\n", + "ax.set_ylim([-0.1, 1.1])\n", + "ax.set_xlim([-5,5])\n", + "ax.grid(True)\n", + "ax.set_xlabel('z')\n", + "ax.set_title('sigmoid function')\n", + "\n", + "plt.show()\n", + "\n", + "\"\"\"Step Function\"\"\"\n", + "z = numpy.arange(-5, 5, .02)\n", + "step_fn = numpy.vectorize(lambda z: 1.0 if z >= 0.0 else 0.0)\n", + "step = step_fn(z)\n", + "\n", + "fig = plt.figure()\n", + "ax = fig.add_subplot(111)\n", + "ax.plot(z, step)\n", + "ax.set_ylim([-0.5, 1.5])\n", + "ax.set_xlim([-5,5])\n", + "ax.grid(True)\n", + "ax.set_xlabel('z')\n", + "ax.set_title('step function')\n", + "\n", + "plt.show()\n", + "\n", + "\"\"\"Sine Function\"\"\"\n", + "z = numpy.arange(-2*mt.pi, 2*mt.pi, 0.1)\n", + "t = numpy.sin(z)\n", + "\n", + "fig = plt.figure()\n", + "ax = fig.add_subplot(111)\n", + "ax.plot(z, t)\n", + "ax.set_ylim([-1.0, 1.0])\n", + "ax.set_xlim([-2*mt.pi,2*mt.pi])\n", + "ax.grid(True)\n", + "ax.set_xlabel('z')\n", + "ax.set_title('sine function')\n", + "\n", + "plt.show()\n", + "\n", + "\"\"\"Plots a graph of the squashing function used by a rectified linear\n", + "unit\"\"\"\n", + "z = numpy.arange(-2, 2, .1)\n", + "zero = numpy.zeros(len(z))\n", + "y = numpy.max([zero, z], axis=0)\n", + "\n", + "fig = plt.figure()\n", + "ax = fig.add_subplot(111)\n", + "ax.plot(z, y)\n", + "ax.set_ylim([-2.0, 2.0])\n", + "ax.set_xlim([-2.0, 2.0])\n", + "ax.grid(True)\n", + "ax.set_xlabel('z')\n", + "ax.set_title('Rectified linear unit')\n", + "\n", + "plt.show()" + ] + }, + { + "cell_type": "markdown", + "id": "9cccb9ca", + "metadata": { + "editable": true + }, + "source": [ + "## Vanishing gradients\n", + "\n", + "The Back propagation algorithm we derived above works by going from\n", + "the output layer to the input layer, propagating the error gradient on\n", + "the way. Once the algorithm has computed the gradient of the cost\n", + "function with regards to each parameter in the network, it uses these\n", + "gradients to update each parameter with a Gradient Descent (GD) step.\n", + "\n", + "Unfortunately for us, the gradients often get smaller and smaller as\n", + "the algorithm progresses down to the first hidden layers. As a result,\n", + "the GD update leaves the lower layer connection weights virtually\n", + "unchanged, and training never converges to a good solution. This is\n", + "known in the literature as **the vanishing gradients problem**." + ] + }, + { + "cell_type": "markdown", + "id": "a42f7226", + "metadata": { + "editable": true + }, + "source": [ + "## Exploding gradients\n", + "\n", + "In other cases, the opposite can happen, namely the the gradients can\n", + "grow bigger and bigger. The result is that many of the layers get\n", + "large updates of the weights the algorithm diverges. This is the\n", + "**exploding gradients problem**, which is mostly encountered in\n", + "recurrent neural networks. More generally, deep neural networks suffer\n", + "from unstable gradients, different layers may learn at widely\n", + "different speeds" + ] + }, + { + "cell_type": "markdown", + "id": "0f57d23f", + "metadata": { + "editable": true + }, + "source": [ + "## Is the Logistic activation function (Sigmoid) our choice?\n", + "\n", + "Although this unfortunate behavior has been empirically observed for\n", + "quite a while (it was one of the reasons why deep neural networks were\n", + "mostly abandoned for a long time), it is only around 2010 that\n", + "significant progress was made in understanding it.\n", + "\n", + "A paper titled [Understanding the Difficulty of Training Deep\n", + "Feedforward Neural Networks by Xavier Glorot and Yoshua Bengio](http://proceedings.mlr.press/v9/glorot10a.html) found that\n", + "the problems with the popular logistic\n", + "sigmoid activation function and the weight initialization technique\n", + "that was most popular at the time, namely random initialization using\n", + "a normal distribution with a mean of 0 and a standard deviation of\n", + "1." + ] + }, + { + "cell_type": "markdown", + "id": "ad459faf", + "metadata": { + "editable": true + }, + "source": [ + "## Logistic function as the root of problems\n", + "\n", + "They showed that with this activation function and this\n", + "initialization scheme, the variance of the outputs of each layer is\n", + "much greater than the variance of its inputs. Going forward in the\n", + "network, the variance keeps increasing after each layer until the\n", + "activation function saturates at the top layers. This is actually made\n", + "worse by the fact that the logistic function has a mean of 0.5, not 0\n", + "(the hyperbolic tangent function has a mean of 0 and behaves slightly\n", + "better than the logistic function in deep networks)." + ] + }, + { + "cell_type": "markdown", + "id": "1e307870", + "metadata": { + "editable": true + }, + "source": [ + "## The derivative of the Logistic funtion\n", + "\n", + "Looking at the logistic activation function, when inputs become large\n", + "(negative or positive), the function saturates at 0 or 1, with a\n", + "derivative extremely close to 0. Thus when backpropagation kicks in,\n", + "it has virtually no gradient to propagate back through the network,\n", + "and what little gradient exists keeps getting diluted as\n", + "backpropagation progresses down through the top layers, so there is\n", + "really nothing left for the lower layers.\n", + "\n", + "In their paper, Glorot and Bengio propose a way to significantly\n", + "alleviate this problem. We need the signal to flow properly in both\n", + "directions: in the forward direction when making predictions, and in\n", + "the reverse direction when backpropagating gradients. We don’t want\n", + "the signal to die out, nor do we want it to explode and saturate. For\n", + "the signal to flow properly, the authors argue that we need the\n", + "variance of the outputs of each layer to be equal to the variance of\n", + "its inputs, and we also need the gradients to have equal variance\n", + "before and after flowing through a layer in the reverse direction." + ] + }, + { + "cell_type": "markdown", + "id": "06fe5be6", + "metadata": { + "editable": true + }, + "source": [ + "## Insights from the paper by Glorot and Bengio\n", + "\n", + "One of the insights in the 2010 paper by Glorot and Bengio was that\n", + "the vanishing/exploding gradients problems were in part due to a poor\n", + "choice of activation function. Until then most people had assumed that\n", + "if Nature had chosen to use roughly sigmoid activation functions in\n", + "biological neurons, they must be an excellent choice. But it turns out\n", + "that other activation functions behave much better in deep neural\n", + "networks, in particular the ReLU activation function, mostly because\n", + "it does not saturate for positive values (and also because it is quite\n", + "fast to compute)." + ] + }, + { + "cell_type": "markdown", + "id": "736441f3", + "metadata": { + "editable": true + }, + "source": [ + "## The RELU function family\n", + "\n", + "The ReLU activation function suffers from a problem known as the dying\n", + "ReLUs: during training, some neurons effectively die, meaning they\n", + "stop outputting anything other than 0.\n", + "\n", + "In some cases, you may find that half of your network’s neurons are\n", + "dead, especially if you used a large learning rate. During training,\n", + "if a neuron’s weights get updated such that the weighted sum of the\n", + "neuron’s inputs is negative, it will start outputting 0. When this\n", + "happen, the neuron is unlikely to come back to life since the gradient\n", + "of the ReLU function is 0 when its input is negative." + ] + }, + { + "cell_type": "markdown", + "id": "9ac6f21d", + "metadata": { + "editable": true + }, + "source": [ + "## ELU function\n", + "\n", + "To solve this problem, nowadays practitioners use a variant of the\n", + "ReLU function, such as the leaky ReLU discussed above or the so-called\n", + "exponential linear unit (ELU) function" + ] + }, + { + "cell_type": "markdown", + "id": "80ebd77c", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "ELU(z) = \\left\\{\\begin{array}{cc} \\alpha\\left( \\exp{(z)}-1\\right) & z < 0,\\\\ z & z \\ge 0.\\end{array}\\right.\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "2566a903", + "metadata": { + "editable": true + }, + "source": [ + "## Which activation function should we use?\n", + "\n", + "In general it seems that the ELU activation function is better than\n", + "the leaky ReLU function (and its variants), which is better than\n", + "ReLU. ReLU performs better than $\\tanh$ which in turn performs better\n", + "than the logistic function.\n", + "\n", + "If runtime performance is an issue, then you may opt for the leaky\n", + "ReLU function over the ELU function If you don’t want to tweak yet\n", + "another hyperparameter, you may just use the default $\\alpha$ of\n", + "$0.01$ for the leaky ReLU, and $1$ for ELU. If you have spare time and\n", + "computing power, you can use cross-validation or bootstrap to evaluate\n", + "other activation functions." + ] + }, + { + "cell_type": "markdown", + "id": "382f5fe1", + "metadata": { + "editable": true + }, + "source": [ + "## More on activation functions, output layers\n", + "\n", + "In most cases you can use the ReLU activation function in the hidden\n", + "layers (or one of its variants).\n", + "\n", + "It is a bit faster to compute than other activation functions, and the\n", + "gradient descent optimization does in general not get stuck.\n", + "\n", + "**For the output layer:**\n", + "\n", + "* For classification the softmax activation function is generally a good choice for classification tasks (when the classes are mutually exclusive).\n", + "\n", + "* For regression tasks, you can simply use no activation function at all." + ] + }, + { + "cell_type": "markdown", + "id": "fed07226", + "metadata": { + "editable": true + }, + "source": [ + "## Fine-tuning neural network hyperparameters\n", + "\n", + "The flexibility of neural networks is also one of their main\n", + "drawbacks: there are many hyperparameters to tweak. Not only can you\n", + "use any imaginable network topology (how neurons/nodes are\n", + "interconnected), but even in a simple FFNN you can change the number\n", + "of layers, the number of neurons per layer, the type of activation\n", + "function to use in each layer, the weight initialization logic, the\n", + "stochastic gradient optmized and much more. How do you know what\n", + "combination of hyperparameters is the best for your task?\n", + "\n", + "* You can use grid search with cross-validation to find the right hyperparameters.\n", + "\n", + "However,since there are many hyperparameters to tune, and since\n", + "training a neural network on a large dataset takes a lot of time, you\n", + "will only be able to explore a tiny part of the hyperparameter space.\n", + "\n", + "* You can use randomized search.\n", + "\n", + "* Or use tools like [Oscar](http://oscar.calldesk.ai/), which implements more complex algorithms to help you find a good set of hyperparameters quickly." + ] + }, + { + "cell_type": "markdown", + "id": "9ddc2586", + "metadata": { + "editable": true + }, + "source": [ + "## Hidden layers\n", + "\n", + "For many problems you can start with just one or two hidden layers and\n", + "it will work just fine. For the MNIST data set discussed below you can easily get a\n", + "high accuracy using just one hidden layer with a few hundred neurons.\n", + "You can reach for this data set above 98% accuracy using two hidden\n", + "layers with the same total amount of neurons, in roughly the same\n", + "amount of training time.\n", + "\n", + "For more complex problems, you can gradually ramp up the number of\n", + "hidden layers, until you start overfitting the training set. Very\n", + "complex tasks, such as large image classification or speech\n", + "recognition, typically require networks with dozens of layers and they\n", + "need a huge amount of training data. However, you will rarely have to\n", + "train such networks from scratch: it is much more common to reuse\n", + "parts of a pretrained state-of-the-art network that performs a similar\n", + "task." + ] + }, + { + "cell_type": "markdown", + "id": "d327b303", + "metadata": { + "editable": true + }, + "source": [ + "## Batch Normalization\n", + "\n", + "Batch Normalization aims to address the vanishing/exploding gradients\n", + "problems, and more generally the problem that the distribution of each\n", + "layer’s inputs changes during training, as the parameters of the\n", + "previous layers change.\n", + "\n", + "The technique consists of adding an operation in the model just before\n", + "the activation function of each layer, simply zero-centering and\n", + "normalizing the inputs, then scaling and shifting the result using two\n", + "new parameters per layer (one for scaling, the other for shifting). In\n", + "other words, this operation lets the model learn the optimal scale and\n", + "mean of the inputs for each layer. In order to zero-center and\n", + "normalize the inputs, the algorithm needs to estimate the inputs’ mean\n", + "and standard deviation. It does so by evaluating the mean and standard\n", + "deviation of the inputs over the current mini-batch, from this the\n", + "name batch normalization." + ] + }, + { + "cell_type": "markdown", + "id": "a3f4a57d", + "metadata": { + "editable": true + }, + "source": [ + "## Dropout\n", + "\n", + "It is a fairly simple algorithm: at every training step, every neuron\n", + "(including the input neurons but excluding the output neurons) has a\n", + "probability $p$ of being temporarily dropped out, meaning it will be\n", + "entirely ignored during this training step, but it may be active\n", + "during the next step.\n", + "\n", + "The hyperparameter $p$ is called the dropout rate, and it is typically\n", + "set to 50%. After training, the neurons are not dropped anymore. It\n", + "is viewed as one of the most popular regularization techniques." + ] + }, + { + "cell_type": "markdown", + "id": "da0c8dbe", + "metadata": { + "editable": true + }, + "source": [ + "## Gradient Clipping\n", + "\n", + "A popular technique to lessen the exploding gradients problem is to\n", + "simply clip the gradients during backpropagation so that they never\n", + "exceed some threshold (this is mostly useful for recurrent neural\n", + "networks).\n", + "\n", + "This technique is called Gradient Clipping.\n", + "\n", + "In general however, Batch\n", + "Normalization is preferred." + ] + }, + { + "cell_type": "markdown", + "id": "60e3da23", + "metadata": { + "editable": true + }, + "source": [ + "## A top-down perspective on Neural networks\n", + "\n", + "The first thing we would like to do is divide the data into two or\n", + "three parts. A training set, a validation or dev (development) set,\n", + "and a test set. The test set is the data on which we want to make\n", + "predictions. The dev set is a subset of the training data we use to\n", + "check how well we are doing out-of-sample, after training the model on\n", + "the training dataset. We use the validation error as a proxy for the\n", + "test error in order to make tweaks to our model. It is crucial that we\n", + "do not use any of the test data to train the algorithm. This is a\n", + "cardinal sin in ML. Then:\n", + "\n", + "1. Estimate optimal error rate\n", + "\n", + "2. Minimize underfitting (bias) on training data set.\n", + "\n", + "3. Make sure you are not overfitting." + ] + }, + { + "cell_type": "markdown", + "id": "17f8f5ed", + "metadata": { + "editable": true + }, + "source": [ + "## More top-down perspectives\n", + "\n", + "If the validation and test sets are drawn from the same distributions,\n", + "then a good performance on the validation set should lead to similarly\n", + "good performance on the test set. \n", + "\n", + "However, sometimes\n", + "the training data and test data differ in subtle ways because, for\n", + "example, they are collected using slightly different methods, or\n", + "because it is cheaper to collect data in one way versus another. In\n", + "this case, there can be a mismatch between the training and test\n", + "data. This can lead to the neural network overfitting these small\n", + "differences between the test and training sets, and a poor performance\n", + "on the test set despite having a good performance on the validation\n", + "set. To rectify this, Andrew Ng suggests making two validation or dev\n", + "sets, one constructed from the training data and one constructed from\n", + "the test data. The difference between the performance of the algorithm\n", + "on these two validation sets quantifies the train-test mismatch. This\n", + "can serve as another important diagnostic when using DNNs for\n", + "supervised learning." + ] + }, + { + "cell_type": "markdown", + "id": "06ba3896", + "metadata": { + "editable": true + }, + "source": [ + "## Limitations of supervised learning with deep networks\n", + "\n", + "Like all statistical methods, supervised learning using neural\n", + "networks has important limitations. This is especially important when\n", + "one seeks to apply these methods, especially to physics problems. Like\n", + "all tools, DNNs are not a universal solution. Often, the same or\n", + "better performance on a task can be achieved by using a few\n", + "hand-engineered features (or even a collection of random\n", + "features)." + ] + }, + { + "cell_type": "markdown", + "id": "72bc99e6", + "metadata": { + "editable": true + }, + "source": [ + "## Limitations of NNs\n", + "\n", + "Here we list some of the important limitations of supervised neural network based models. \n", + "\n", + "* **Need labeled data**. All supervised learning methods, DNNs for supervised learning require labeled data. Often, labeled data is harder to acquire than unlabeled data (e.g. one must pay for human experts to label images).\n", + "\n", + "* **Supervised neural networks are extremely data intensive.** DNNs are data hungry. They perform best when data is plentiful. This is doubly so for supervised methods where the data must also be labeled. The utility of DNNs is extremely limited if data is hard to acquire or the datasets are small (hundreds to a few thousand samples). In this case, the performance of other methods that utilize hand-engineered features can exceed that of DNNs." + ] + }, + { + "cell_type": "markdown", + "id": "d5f452a2", + "metadata": { + "editable": true + }, + "source": [ + "## Homogeneous data\n", + "\n", + "* **Homogeneous data.** Almost all DNNs deal with homogeneous data of one type. It is very hard to design architectures that mix and match data types (i.e. some continuous variables, some discrete variables, some time series). In applications beyond images, video, and language, this is often what is required. In contrast, ensemble models like random forests or gradient-boosted trees have no difficulty handling mixed data types." + ] + }, + { + "cell_type": "markdown", + "id": "63889ee1", + "metadata": { + "editable": true + }, + "source": [ + "## More limitations\n", + "\n", + "* **Many problems are not about prediction.** In natural science we are often interested in learning something about the underlying distribution that generates the data. In this case, it is often difficult to cast these ideas in a supervised learning setting. While the problems are related, it is possible to make good predictions with a *wrong* model. The model might or might not be useful for understanding the underlying science.\n", + "\n", + "Some of these remarks are particular to DNNs, others are shared by all supervised learning methods. This motivates the use of unsupervised methods which in part circumvent these problems." + ] + }, + { + "cell_type": "markdown", + "id": "5f47ef0f", + "metadata": { + "editable": true + }, + "source": [ + "## Setting up a Multi-layer perceptron model for classification\n", + "\n", + "We are now gong to develop an example based on the MNIST data\n", + "base. This is a classification problem and we need to use our\n", + "cross-entropy function we discussed in connection with logistic\n", + "regression. The cross-entropy defines our cost function for the\n", + "classificaton problems with neural networks.\n", + "\n", + "In binary classification with two classes $(0, 1)$ we define the\n", + "logistic/sigmoid function as the probability that a particular input\n", + "is in class $0$ or $1$. This is possible because the logistic\n", + "function takes any input from the real numbers and inputs a number\n", + "between 0 and 1, and can therefore be interpreted as a probability. It\n", + "also has other nice properties, such as a derivative that is simple to\n", + "calculate.\n", + "\n", + "For an input $\\boldsymbol{a}$ from the hidden layer, the probability that the input $\\boldsymbol{x}$\n", + "is in class 0 or 1 is just. We let $\\theta$ represent the unknown weights and biases to be adjusted by our equations). The variable $x$\n", + "represents our activation values $z$. We have" + ] + }, + { + "cell_type": "markdown", + "id": "b08a7b7e", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "P(y = 0 \\mid \\boldsymbol{x}, \\boldsymbol{\\theta}) = \\frac{1}{1 + \\exp{(- \\boldsymbol{x}})} ,\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "10220190", + "metadata": { + "editable": true + }, + "source": [ + "and" + ] + }, + { + "cell_type": "markdown", + "id": "0fb4d5ad", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "P(y = 1 \\mid \\boldsymbol{x}, \\boldsymbol{\\theta}) = 1 - P(y = 0 \\mid \\boldsymbol{x}, \\boldsymbol{\\theta}) ,\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "f8aa1bae", + "metadata": { + "editable": true + }, + "source": [ + "where $y \\in \\{0, 1\\}$ and $\\boldsymbol{\\theta}$ represents the weights and biases\n", + "of our network." + ] + }, + { + "cell_type": "markdown", + "id": "17de8e22", + "metadata": { + "editable": true + }, + "source": [ + "## Defining the cost function\n", + "\n", + "Our cost function is given as (see the Logistic regression lectures)" + ] + }, + { + "cell_type": "markdown", + "id": "ca8f4f94", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\mathcal{C}(\\boldsymbol{\\theta}) = - \\ln P(\\mathcal{D} \\mid \\boldsymbol{\\theta}) = - \\sum_{i=1}^n\n", + "y_i \\ln[P(y_i = 0)] + (1 - y_i) \\ln [1 - P(y_i = 0)] = \\sum_{i=1}^n \\mathcal{L}_i(\\boldsymbol{\\theta}) .\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "ca998f7f", + "metadata": { + "editable": true + }, + "source": [ + "This last equality means that we can interpret our *cost* function as a sum over the *loss* function\n", + "for each point in the dataset $\\mathcal{L}_i(\\boldsymbol{\\theta})$. \n", + "The negative sign is just so that we can think about our algorithm as minimizing a positive number, rather\n", + "than maximizing a negative number. \n", + "\n", + "In *multiclass* classification it is common to treat each integer label as a so called *one-hot* vector: \n", + "\n", + "$y = 5 \\quad \\rightarrow \\quad \\boldsymbol{y} = (0, 0, 0, 0, 0, 1, 0, 0, 0, 0) ,$ and\n", + "\n", + "$y = 1 \\quad \\rightarrow \\quad \\boldsymbol{y} = (0, 1, 0, 0, 0, 0, 0, 0, 0, 0) ,$ \n", + "\n", + "i.e. a binary bit string of length $C$, where $C = 10$ is the number of classes in the MNIST dataset (numbers from $0$ to $9$).. \n", + "\n", + "If $\\boldsymbol{x}_i$ is the $i$-th input (image), $y_{ic}$ refers to the $c$-th component of the $i$-th\n", + "output vector $\\boldsymbol{y}_i$. \n", + "The probability of $\\boldsymbol{x}_i$ being in class $c$ will be given by the softmax function:" + ] + }, + { + "cell_type": "markdown", + "id": "525b0573", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "P(y_{ic} = 1 \\mid \\boldsymbol{x}_i, \\boldsymbol{\\theta}) = \\frac{\\exp{((\\boldsymbol{a}_i^{hidden})^T \\boldsymbol{w}_c)}}\n", + "{\\sum_{c'=0}^{C-1} \\exp{((\\boldsymbol{a}_i^{hidden})^T \\boldsymbol{w}_{c'})}} ,\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "a9279995", + "metadata": { + "editable": true + }, + "source": [ + "which reduces to the logistic function in the binary case. \n", + "The likelihood of this $C$-class classifier\n", + "is now given as:" + ] + }, + { + "cell_type": "markdown", + "id": "e6cef751", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "P(\\mathcal{D} \\mid \\boldsymbol{\\theta}) = \\prod_{i=1}^n \\prod_{c=0}^{C-1} [P(y_{ic} = 1)]^{y_{ic}} .\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "0ec11e1d", + "metadata": { + "editable": true + }, + "source": [ + "Again we take the negative log-likelihood to define our cost function:" + ] + }, + { + "cell_type": "markdown", + "id": "e5d6a067", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\mathcal{C}(\\boldsymbol{\\theta}) = - \\log{P(\\mathcal{D} \\mid \\boldsymbol{\\theta})}.\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "add6e597", + "metadata": { + "editable": true + }, + "source": [ + "See the logistic regression lectures for a full definition of the cost function.\n", + "\n", + "The back propagation equations need now only a small change, namely the definition of a new cost function. We are thus ready to use the same equations as before!" + ] + }, + { + "cell_type": "markdown", + "id": "dc7ca42d", + "metadata": { + "editable": true + }, + "source": [ + "## Example: binary classification problem\n", + "\n", + "As an example of the above, relevant for project 2 as well, let us consider a binary class. As discussed in our logistic regression lectures, we defined a cost function in terms of the parameters $\\beta$ as" + ] + }, + { + "cell_type": "markdown", + "id": "80b56aa5", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\mathcal{C}(\\boldsymbol{\\beta}) = - \\sum_{i=1}^n \\left(y_i\\log{p(y_i \\vert x_i,\\boldsymbol{\\beta})}+(1-y_i)\\log{1-p(y_i \\vert x_i,\\boldsymbol{\\beta})}\\right),\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "72a87573", + "metadata": { + "editable": true + }, + "source": [ + "where we had defined the logistic (sigmoid) function" + ] + }, + { + "cell_type": "markdown", + "id": "ff8418ea", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "p(y_i =1\\vert x_i,\\boldsymbol{\\beta})=\\frac{\\exp{(\\beta_0+\\beta_1 x_i)}}{1+\\exp{(\\beta_0+\\beta_1 x_i)}},\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "dd978909", + "metadata": { + "editable": true + }, + "source": [ + "and" + ] + }, + { + "cell_type": "markdown", + "id": "ce5a0650", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "p(y_i =0\\vert x_i,\\boldsymbol{\\beta})=1-p(y_i =1\\vert x_i,\\boldsymbol{\\beta}).\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "dce8478b", + "metadata": { + "editable": true + }, + "source": [ + "The parameters $\\boldsymbol{\\beta}$ were defined using a minimization method like gradient descent or Newton-Raphson's method. \n", + "\n", + "Now we replace $x_i$ with the activation $z_i^l$ for a given layer $l$ and the outputs as $y_i=a_i^l=f(z_i^l)$, with $z_i^l$ now being a function of the weights $w_{ij}^l$ and biases $b_i^l$. \n", + "We have then" + ] + }, + { + "cell_type": "markdown", + "id": "92f7282e", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "a_i^l = y_i = \\frac{\\exp{(z_i^l)}}{1+\\exp{(z_i^l)}},\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "e83bc7a2", + "metadata": { + "editable": true + }, + "source": [ + "with" + ] + }, + { + "cell_type": "markdown", + "id": "04f9d7f2", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "z_i^l = \\sum_{j}w_{ij}^l a_j^{l-1}+b_i^l,\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "c70ab677", + "metadata": { + "editable": true + }, + "source": [ + "where the superscript $l-1$ indicates that these are the outputs from layer $l-1$.\n", + "Our cost function at the final layer $l=L$ is now" + ] + }, + { + "cell_type": "markdown", + "id": "801a6bd2", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\mathcal{C}(\\boldsymbol{W}) = - \\sum_{i=1}^n \\left(t_i\\log{a_i^L}+(1-t_i)\\log{(1-a_i^L)}\\right),\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "38eeee4b", + "metadata": { + "editable": true + }, + "source": [ + "where we have defined the targets $t_i$. The derivatives of the cost function with respect to the output $a_i^L$ are then easily calculated and we get" + ] + }, + { + "cell_type": "markdown", + "id": "d70ddf11", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\frac{\\partial \\mathcal{C}(\\boldsymbol{W})}{\\partial a_i^L} = \\frac{a_i^L-t_i}{a_i^L(1-a_i^L)}.\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "324ee5e0", + "metadata": { + "editable": true + }, + "source": [ + "In case we use another activation function than the logistic one, we need to evaluate other derivatives." + ] + }, + { + "cell_type": "markdown", + "id": "3366df27", + "metadata": { + "editable": true + }, + "source": [ + "## The Softmax function\n", + "In case we employ the more general case given by the Softmax equation, we need to evaluate the derivative of the activation function with respect to the activation $z_i^l$, that is we need" + ] + }, + { + "cell_type": "markdown", + "id": "c9a2bd62", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\frac{\\partial f(z_i^l)}{\\partial w_{jk}^l} =\n", + "\\frac{\\partial f(z_i^l)}{\\partial z_j^l} \\frac{\\partial z_j^l}{\\partial w_{jk}^l}= \\frac{\\partial f(z_i^l)}{\\partial z_j^l}a_k^{l-1}.\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "401183f6", + "metadata": { + "editable": true + }, + "source": [ + "For the Softmax function we have" + ] + }, + { + "cell_type": "markdown", + "id": "d4e6a79e", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "f(z_i^l) = \\frac{\\exp{(z_i^l)}}{\\sum_{m=1}^K\\exp{(z_m^l)}}.\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "61cfab52", + "metadata": { + "editable": true + }, + "source": [ + "Its derivative with respect to $z_j^l$ gives" + ] + }, + { + "cell_type": "markdown", + "id": "fc5cf891", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\frac{\\partial f(z_i^l)}{\\partial z_j^l}= f(z_i^l)\\left(\\delta_{ij}-f(z_j^l)\\right),\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "2e7f3950", + "metadata": { + "editable": true + }, + "source": [ + "which in case of the simply binary model reduces to having $i=j$." + ] + }, + { + "cell_type": "markdown", + "id": "144969f6", + "metadata": { + "editable": true + }, + "source": [ + "## Developing a code for doing neural networks with back propagation\n", + "\n", + "One can identify a set of key steps when using neural networks to solve supervised learning problems: \n", + "\n", + "1. Collect and pre-process data \n", + "\n", + "2. Define model and architecture \n", + "\n", + "3. Choose cost function and optimizer \n", + "\n", + "4. Train the model \n", + "\n", + "5. Evaluate model performance on test data \n", + "\n", + "6. Adjust hyperparameters (if necessary, network architecture)" + ] + }, + { + "cell_type": "markdown", + "id": "b6e503bd", + "metadata": { + "editable": true + }, + "source": [ + "## Collect and pre-process data\n", + "\n", + "Here we will be using the MNIST dataset, which is readily available through the **scikit-learn**\n", + "package. You may also find it for example [here](http://yann.lecun.com/exdb/mnist/). \n", + "The *MNIST* (Modified National Institute of Standards and Technology) database is a large database\n", + "of handwritten digits that is commonly used for training various image processing systems. \n", + "The MNIST dataset consists of 70 000 images of size $28\\times 28$ pixels, each labeled from 0 to 9. \n", + "The scikit-learn dataset we will use consists of a selection of 1797 images of size $8\\times 8$ collected and processed from this database. \n", + "\n", + "To feed data into a feed-forward neural network we need to represent\n", + "the inputs as a design/feature matrix $X = (n_{inputs}, n_{features})$. Each\n", + "row represents an *input*, in this case a handwritten digit, and\n", + "each column represents a *feature*, in this case a pixel. The\n", + "correct answers, also known as *labels* or *targets* are\n", + "represented as a 1D array of integers \n", + "$Y = (n_{inputs}) = (5, 3, 1, 8,...)$.\n", + "\n", + "As an example, say we want to build a neural network using supervised learning to predict Body-Mass Index (BMI) from\n", + "measurements of height (in m) \n", + "and weight (in kg). If we have measurements of 5 people the design/feature matrix could be for example: \n", + "\n", + "$$ X = \\begin{bmatrix}\n", + "1.85 & 81\\\\\n", + "1.71 & 65\\\\\n", + "1.95 & 103\\\\\n", + "1.55 & 42\\\\\n", + "1.63 & 56\n", + "\\end{bmatrix} ,$$ \n", + "\n", + "and the targets would be: \n", + "\n", + "$$ Y = (23.7, 22.2, 27.1, 17.5, 21.1) $$ \n", + "\n", + "Since each input image is a 2D matrix, we need to flatten the image\n", + "(i.e. \"unravel\" the 2D matrix into a 1D array) to turn the data into a\n", + "design/feature matrix. This means we lose all spatial information in the\n", + "image, such as locality and translational invariance. More complicated\n", + "architectures such as Convolutional Neural Networks can take advantage\n", + "of such information, and are most commonly applied when analyzing\n", + "images." + ] + }, + { + "cell_type": "code", + "execution_count": 3, + "id": "54230ef9", + "metadata": { + "collapsed": false, + "editable": true + }, + "outputs": [], + "source": [ + "# import necessary packages\n", + "import numpy as np\n", + "import matplotlib.pyplot as plt\n", + "from sklearn import datasets\n", + "\n", + "\n", + "# ensure the same random numbers appear every time\n", + "np.random.seed(0)\n", + "\n", + "# display images in notebook\n", + "%matplotlib inline\n", + "plt.rcParams['figure.figsize'] = (12,12)\n", + "\n", + "\n", + "# download MNIST dataset\n", + "digits = datasets.load_digits()\n", + "\n", + "# define inputs and labels\n", + "inputs = digits.images\n", + "labels = digits.target\n", + "\n", + "print(\"inputs = (n_inputs, pixel_width, pixel_height) = \" + str(inputs.shape))\n", + "print(\"labels = (n_inputs) = \" + str(labels.shape))\n", + "\n", + "\n", + "# flatten the image\n", + "# the value -1 means dimension is inferred from the remaining dimensions: 8x8 = 64\n", + "n_inputs = len(inputs)\n", + "inputs = inputs.reshape(n_inputs, -1)\n", + "print(\"X = (n_inputs, n_features) = \" + str(inputs.shape))\n", + "\n", + "\n", + "# choose some random images to display\n", + "indices = np.arange(n_inputs)\n", + "random_indices = np.random.choice(indices, size=5)\n", + "\n", + "for i, image in enumerate(digits.images[random_indices]):\n", + " plt.subplot(1, 5, i+1)\n", + " plt.axis('off')\n", + " plt.imshow(image, cmap=plt.cm.gray_r, interpolation='nearest')\n", + " plt.title(\"Label: %d\" % digits.target[random_indices[i]])\n", + "plt.show()" + ] + }, + { + "cell_type": "markdown", + "id": "6738e7b4", + "metadata": { + "editable": true + }, + "source": [ + "## Train and test datasets\n", + "\n", + "Performing analysis before partitioning the dataset is a major error, that can lead to incorrect conclusions. \n", + "\n", + "We will reserve $80 \\%$ of our dataset for training and $20 \\%$ for testing. \n", + "\n", + "It is important that the train and test datasets are drawn randomly from our dataset, to ensure\n", + "no bias in the sampling. \n", + "Say you are taking measurements of weather data to predict the weather in the coming 5 days.\n", + "You don't want to train your model on measurements taken from the hours 00.00 to 12.00, and then test it on data\n", + "collected from 12.00 to 24.00." + ] + }, + { + "cell_type": "code", + "execution_count": 4, + "id": "f369aa9d", + "metadata": { + "collapsed": false, + "editable": true + }, + "outputs": [], + "source": [ + "from sklearn.model_selection import train_test_split\n", + "\n", + "# one-liner from scikit-learn library\n", + "train_size = 0.8\n", + "test_size = 1 - train_size\n", + "X_train, X_test, Y_train, Y_test = train_test_split(inputs, labels, train_size=train_size,\n", + " test_size=test_size)\n", + "\n", + "# equivalently in numpy\n", + "def train_test_split_numpy(inputs, labels, train_size, test_size):\n", + " n_inputs = len(inputs)\n", + " inputs_shuffled = inputs.copy()\n", + " labels_shuffled = labels.copy()\n", + " \n", + " np.random.shuffle(inputs_shuffled)\n", + " np.random.shuffle(labels_shuffled)\n", + " \n", + " train_end = int(n_inputs*train_size)\n", + " X_train, X_test = inputs_shuffled[:train_end], inputs_shuffled[train_end:]\n", + " Y_train, Y_test = labels_shuffled[:train_end], labels_shuffled[train_end:]\n", + " \n", + " return X_train, X_test, Y_train, Y_test\n", + "\n", + "#X_train, X_test, Y_train, Y_test = train_test_split_numpy(inputs, labels, train_size, test_size)\n", + "\n", + "print(\"Number of training images: \" + str(len(X_train)))\n", + "print(\"Number of test images: \" + str(len(X_test)))" + ] + }, + { + "cell_type": "markdown", + "id": "f8bf8d16", + "metadata": { + "editable": true + }, + "source": [ + "## Define model and architecture\n", + "\n", + "Our simple feed-forward neural network will consist of an *input* layer, a single *hidden* layer and an *output* layer. The activation $y$ of each neuron is a weighted sum of inputs, passed through an activation function. In case of the simple perceptron model we have \n", + "\n", + "$$ z = \\sum_{i=1}^n w_i a_i ,$$\n", + "\n", + "$$ y = f(z) ,$$\n", + "\n", + "where $f$ is the activation function, $a_i$ represents input from neuron $i$ in the preceding layer\n", + "and $w_i$ is the weight to input $i$. \n", + "The activation of the neurons in the input layer is just the features (e.g. a pixel value). \n", + "\n", + "The simplest activation function for a neuron is the *Heaviside* function:\n", + "\n", + "$$ f(z) = \n", + "\\begin{cases}\n", + "1, & z > 0\\\\\n", + "0, & \\text{otherwise}\n", + "\\end{cases}\n", + "$$\n", + "\n", + "A feed-forward neural network with this activation is known as a *perceptron*. \n", + "For a binary classifier (i.e. two classes, 0 or 1, dog or not-dog) we can also use this in our output layer. \n", + "This activation can be generalized to $k$ classes (using e.g. the *one-against-all* strategy), \n", + "and we call these architectures *multiclass perceptrons*. \n", + "\n", + "However, it is now common to use the terms Single Layer Perceptron (SLP) (1 hidden layer) and \n", + "Multilayer Perceptron (MLP) (2 or more hidden layers) to refer to feed-forward neural networks with any activation function. \n", + "\n", + "Typical choices for activation functions include the sigmoid function, hyperbolic tangent, and Rectified Linear Unit (ReLU). \n", + "We will be using the sigmoid function $\\sigma(x)$: \n", + "\n", + "$$ f(x) = \\sigma(x) = \\frac{1}{1 + e^{-x}} ,$$\n", + "\n", + "which is inspired by probability theory (see logistic regression) and was most commonly used until about 2011. See the discussion below concerning other activation functions." + ] + }, + { + "cell_type": "markdown", + "id": "112b8cc3", + "metadata": { + "editable": true + }, + "source": [ + "## Layers\n", + "\n", + "* Input \n", + "\n", + "Since each input image has 8x8 = 64 pixels or features, we have an input layer of 64 neurons. \n", + "\n", + "* Hidden layer\n", + "\n", + "We will use 50 neurons in the hidden layer receiving input from the neurons in the input layer. \n", + "Since each neuron in the hidden layer is connected to the 64 inputs we have 64x50 = 3200 weights to the hidden layer. \n", + "\n", + "* Output\n", + "\n", + "If we were building a binary classifier, it would be sufficient with a single neuron in the output layer,\n", + "which could output 0 or 1 according to the Heaviside function. This would be an example of a *hard* classifier, meaning it outputs the class of the input directly. However, if we are dealing with noisy data it is often beneficial to use a *soft* classifier, which outputs the probability of being in class 0 or 1. \n", + "\n", + "For a soft binary classifier, we could use a single neuron and interpret the output as either being the probability of being in class 0 or the probability of being in class 1. Alternatively we could use 2 neurons, and interpret each neuron as the probability of being in each class. \n", + "\n", + "Since we are doing multiclass classification, with 10 categories, it is natural to use 10 neurons in the output layer. We number the neurons $j = 0,1,...,9$. The activation of each output neuron $j$ will be according to the *softmax* function: \n", + "\n", + "$$ P(\\text{class $j$} \\mid \\text{input $\\boldsymbol{a}$}) = \\frac{\\exp{(\\boldsymbol{a}^T \\boldsymbol{w}_j)}}\n", + "{\\sum_{c=0}^{9} \\exp{(\\boldsymbol{a}^T \\boldsymbol{w}_c)}} ,$$ \n", + "\n", + "i.e. each neuron $j$ outputs the probability of being in class $j$ given an input from the hidden layer $\\boldsymbol{a}$, with $\\boldsymbol{w}_j$ the weights of neuron $j$ to the inputs. \n", + "The denominator is a normalization factor to ensure the outputs (probabilities) sum up to 1. \n", + "The exponent is just the weighted sum of inputs as before: \n", + "\n", + "$$ z_j = \\sum_{i=1}^n w_ {ij} a_i+b_j.$$ \n", + "\n", + "Since each neuron in the output layer is connected to the 50 inputs from the hidden layer we have 50x10 = 500\n", + "weights to the output layer." + ] + }, + { + "cell_type": "markdown", + "id": "e4166103", + "metadata": { + "editable": true + }, + "source": [ + "## Weights and biases\n", + "\n", + "Typically weights are initialized with small values distributed around zero, drawn from a uniform\n", + "or normal distribution. Setting all weights to zero means all neurons give the same output, making the network useless. \n", + "\n", + "Adding a bias value to the weighted sum of inputs allows the neural network to represent a greater range\n", + "of values. Without it, any input with the value 0 will be mapped to zero (before being passed through the activation). The bias unit has an output of 1, and a weight to each neuron $j$, $b_j$: \n", + "\n", + "$$ z_j = \\sum_{i=1}^n w_ {ij} a_i + b_j.$$ \n", + "\n", + "The bias weights $\\boldsymbol{b}$ are often initialized to zero, but a small value like $0.01$ ensures all neurons have some output which can be backpropagated in the first training cycle." + ] + }, + { + "cell_type": "code", + "execution_count": 5, + "id": "36399cb5", + "metadata": { + "collapsed": false, + "editable": true + }, + "outputs": [], + "source": [ + "# building our neural network\n", + "\n", + "n_inputs, n_features = X_train.shape\n", + "n_hidden_neurons = 50\n", + "n_categories = 10\n", + "\n", + "# we make the weights normally distributed using numpy.random.randn\n", + "\n", + "# weights and bias in the hidden layer\n", + "hidden_weights = np.random.randn(n_features, n_hidden_neurons)\n", + "hidden_bias = np.zeros(n_hidden_neurons) + 0.01\n", + "\n", + "# weights and bias in the output layer\n", + "output_weights = np.random.randn(n_hidden_neurons, n_categories)\n", + "output_bias = np.zeros(n_categories) + 0.01" + ] + }, + { + "cell_type": "markdown", + "id": "5eaf095b", + "metadata": { + "editable": true + }, + "source": [ + "## Feed-forward pass\n", + "\n", + "Denote $F$ the number of features, $H$ the number of hidden neurons and $C$ the number of categories. \n", + "For each input image we calculate a weighted sum of input features (pixel values) to each neuron $j$ in the hidden layer $l$: \n", + "\n", + "$$ z_{j}^{l} = \\sum_{i=1}^{F} w_{ij}^{l} x_i + b_{j}^{l},$$\n", + "\n", + "this is then passed through our activation function \n", + "\n", + "$$ a_{j}^{l} = f(z_{j}^{l}) .$$ \n", + "\n", + "We calculate a weighted sum of inputs (activations in the hidden layer) to each neuron $j$ in the output layer: \n", + "\n", + "$$ z_{j}^{L} = \\sum_{i=1}^{H} w_{ij}^{L} a_{i}^{l} + b_{j}^{L}.$$ \n", + "\n", + "Finally we calculate the output of neuron $j$ in the output layer using the softmax function: \n", + "\n", + "$$ a_{j}^{L} = \\frac{\\exp{(z_j^{L})}}\n", + "{\\sum_{c=0}^{C-1} \\exp{(z_c^{L})}} .$$" + ] + }, + { + "cell_type": "markdown", + "id": "91eec6b8", + "metadata": { + "editable": true + }, + "source": [ + "## Matrix multiplications\n", + "\n", + "Since our data has the dimensions $X = (n_{inputs}, n_{features})$ and our weights to the hidden\n", + "layer have the dimensions \n", + "$W_{hidden} = (n_{features}, n_{hidden})$,\n", + "we can easily feed the network all our training data in one go by taking the matrix product \n", + "\n", + "$$ X W^{h} = (n_{inputs}, n_{hidden}),$$ \n", + "\n", + "and obtain a matrix that holds the weighted sum of inputs to the hidden layer\n", + "for each input image and each hidden neuron. \n", + "We also add the bias to obtain a matrix of weighted sums to the hidden layer $Z^{h}$: \n", + "\n", + "$$ \\boldsymbol{z}^{l} = \\boldsymbol{X} \\boldsymbol{W}^{l} + \\boldsymbol{b}^{l} ,$$\n", + "\n", + "meaning the same bias (1D array with size equal number of hidden neurons) is added to each input image. \n", + "This is then passed through the activation: \n", + "\n", + "$$ \\boldsymbol{a}^{l} = f(\\boldsymbol{z}^l) .$$ \n", + "\n", + "This is fed to the output layer: \n", + "\n", + "$$ \\boldsymbol{z}^{L} = \\boldsymbol{a}^{L} \\boldsymbol{W}^{L} + \\boldsymbol{b}^{L} .$$\n", + "\n", + "Finally we receive our output values for each image and each category by passing it through the softmax function: \n", + "\n", + "$$ output = softmax (\\boldsymbol{z}^{L}) = (n_{inputs}, n_{categories}) .$$" + ] + }, + { + "cell_type": "code", + "execution_count": 6, + "id": "ef497434", + "metadata": { + "collapsed": false, + "editable": true + }, + "outputs": [], + "source": [ + "# setup the feed-forward pass, subscript h = hidden layer\n", + "\n", + "def sigmoid(x):\n", + " return 1/(1 + np.exp(-x))\n", + "\n", + "def feed_forward(X):\n", + " # weighted sum of inputs to the hidden layer\n", + " z_h = np.matmul(X, hidden_weights) + hidden_bias\n", + " # activation in the hidden layer\n", + " a_h = sigmoid(z_h)\n", + " \n", + " # weighted sum of inputs to the output layer\n", + " z_o = np.matmul(a_h, output_weights) + output_bias\n", + " # softmax output\n", + " # axis 0 holds each input and axis 1 the probabilities of each category\n", + " exp_term = np.exp(z_o)\n", + " probabilities = exp_term / np.sum(exp_term, axis=1, keepdims=True)\n", + " \n", + " return probabilities\n", + "\n", + "probabilities = feed_forward(X_train)\n", + "print(\"probabilities = (n_inputs, n_categories) = \" + str(probabilities.shape))\n", + "print(\"probability that image 0 is in category 0,1,2,...,9 = \\n\" + str(probabilities[0]))\n", + "print(\"probabilities sum up to: \" + str(probabilities[0].sum()))\n", + "print()\n", + "\n", + "# we obtain a prediction by taking the class with the highest likelihood\n", + "def predict(X):\n", + " probabilities = feed_forward(X)\n", + " return np.argmax(probabilities, axis=1)\n", + "\n", + "predictions = predict(X_train)\n", + "print(\"predictions = (n_inputs) = \" + str(predictions.shape))\n", + "print(\"prediction for image 0: \" + str(predictions[0]))\n", + "print(\"correct label for image 0: \" + str(Y_train[0]))" + ] + }, + { + "cell_type": "markdown", + "id": "ab88238a", + "metadata": { + "editable": true + }, + "source": [ + "## Choose cost function and optimizer\n", + "\n", + "To measure how well our neural network is doing we need to introduce a cost function. \n", + "We will call the function that gives the error of a single sample output the *loss* function, and the function\n", + "that gives the total error of our network across all samples the *cost* function.\n", + "A typical choice for multiclass classification is the *cross-entropy* loss, also known as the negative log likelihood. \n", + "\n", + "In *multiclass* classification it is common to treat each integer label as a so called *one-hot* vector: \n", + "\n", + "$$ y = 5 \\quad \\rightarrow \\quad \\boldsymbol{y} = (0, 0, 0, 0, 0, 1, 0, 0, 0, 0) ,$$ \n", + "\n", + "$$ y = 1 \\quad \\rightarrow \\quad \\boldsymbol{y} = (0, 1, 0, 0, 0, 0, 0, 0, 0, 0) ,$$ \n", + "\n", + "i.e. a binary bit string of length $C$, where $C = 10$ is the number of classes in the MNIST dataset. \n", + "\n", + "Let $y_{ic}$ denote the $c$-th component of the $i$-th one-hot vector. \n", + "We define the cost function $\\mathcal{C}$ as a sum over the cross-entropy loss for each point $\\boldsymbol{x}_i$ in the dataset.\n", + "\n", + "In the one-hot representation only one of the terms in the loss function is non-zero, namely the\n", + "probability of the correct category $c'$ \n", + "(i.e. the category $c'$ such that $y_{ic'} = 1$). This means that the cross entropy loss only punishes you for how wrong\n", + "you got the correct label. The probability of category $c$ is given by the softmax function. The vector $\\boldsymbol{\\theta}$ represents the parameters of our network, i.e. all the weights and biases." + ] + }, + { + "cell_type": "markdown", + "id": "506145ff", + "metadata": { + "editable": true + }, + "source": [ + "## Optimizing the cost function\n", + "\n", + "The network is trained by finding the weights and biases that minimize the cost function. One of the most widely used classes of methods is *gradient descent* and its generalizations. The idea behind gradient descent\n", + "is simply to adjust the weights in the direction where the gradient of the cost function is large and negative. This ensures we flow toward a *local* minimum of the cost function. \n", + "Each parameter $\\theta$ is iteratively adjusted according to the rule \n", + "\n", + "$$ \\theta_{i+1} = \\theta_i - \\eta \\nabla \\mathcal{C}(\\theta_i) ,$$\n", + "\n", + "where $\\eta$ is known as the *learning rate*, which controls how big a step we take towards the minimum. \n", + "This update can be repeated for any number of iterations, or until we are satisfied with the result. \n", + "\n", + "A simple and effective improvement is a variant called *Batch Gradient Descent*. \n", + "Instead of calculating the gradient on the whole dataset, we calculate an approximation of the gradient\n", + "on a subset of the data called a *minibatch*. \n", + "If there are $N$ data points and we have a minibatch size of $M$, the total number of batches\n", + "is $N/M$. \n", + "We denote each minibatch $B_k$, with $k = 1, 2,...,N/M$. The gradient then becomes: \n", + "\n", + "$$ \\nabla \\mathcal{C}(\\theta) = \\frac{1}{N} \\sum_{i=1}^N \\nabla \\mathcal{L}_i(\\theta) \\quad \\rightarrow \\quad\n", + "\\frac{1}{M} \\sum_{i \\in B_k} \\nabla \\mathcal{L}_i(\\theta) ,$$\n", + "\n", + "i.e. instead of averaging the loss over the entire dataset, we average over a minibatch. \n", + "\n", + "This has two important benefits: \n", + "1. Introducing stochasticity decreases the chance that the algorithm becomes stuck in a local minima. \n", + "\n", + "2. It significantly speeds up the calculation, since we do not have to use the entire dataset to calculate the gradient. \n", + "\n", + "The various optmization methods, with codes and algorithms, are discussed in our lectures on [Gradient descent approaches](https://compphysics.github.io/MachineLearning/doc/pub/Splines/html/Splines-bs.html)." + ] + }, + { + "cell_type": "markdown", + "id": "e8138025", + "metadata": { + "editable": true + }, + "source": [ + "## Regularization\n", + "\n", + "It is common to add an extra term to the cost function, proportional\n", + "to the size of the weights. This is equivalent to constraining the\n", + "size of the weights, so that they do not grow out of control.\n", + "Constraining the size of the weights means that the weights cannot\n", + "grow arbitrarily large to fit the training data, and in this way\n", + "reduces *overfitting*.\n", + "\n", + "We will measure the size of the weights using the so called *L2-norm*, meaning our cost function becomes: \n", + "\n", + "$$ \\mathcal{C}(\\theta) = \\frac{1}{N} \\sum_{i=1}^N \\mathcal{L}_i(\\theta) \\quad \\rightarrow \\quad\n", + "\\frac{1}{N} \\sum_{i=1}^N \\mathcal{L}_i(\\theta) + \\lambda \\lvert \\lvert \\boldsymbol{w} \\rvert \\rvert_2^2 \n", + "= \\frac{1}{N} \\sum_{i=1}^N \\mathcal{L}(\\theta) + \\lambda \\sum_{ij} w_{ij}^2,$$ \n", + "\n", + "i.e. we sum up all the weights squared. The factor $\\lambda$ is known as a regularization parameter.\n", + "\n", + "In order to train the model, we need to calculate the derivative of\n", + "the cost function with respect to every bias and weight in the\n", + "network. In total our network has $(64 + 1)\\times 50=3250$ weights in\n", + "the hidden layer and $(50 + 1)\\times 10=510$ weights to the output\n", + "layer ($+1$ for the bias), and the gradient must be calculated for\n", + "every parameter. We use the *backpropagation* algorithm discussed\n", + "above. This is a clever use of the chain rule that allows us to\n", + "calculate the gradient efficently." + ] + }, + { + "cell_type": "markdown", + "id": "25becf76", + "metadata": { + "editable": true + }, + "source": [ + "## Matrix multiplication\n", + "\n", + "To more efficently train our network these equations are implemented using matrix operations. \n", + "The error in the output layer is calculated simply as, with $\\boldsymbol{t}$ being our targets, \n", + "\n", + "$$ \\delta_L = \\boldsymbol{t} - \\boldsymbol{y} = (n_{inputs}, n_{categories}) .$$ \n", + "\n", + "The gradient for the output weights is calculated as \n", + "\n", + "$$ \\nabla W_{L} = \\boldsymbol{a}^T \\delta_L = (n_{hidden}, n_{categories}) ,$$\n", + "\n", + "where $\\boldsymbol{a} = (n_{inputs}, n_{hidden})$. This simply means that we are summing up the gradients for each input. \n", + "Since we are going backwards we have to transpose the activation matrix. \n", + "\n", + "The gradient with respect to the output bias is then \n", + "\n", + "$$ \\nabla \\boldsymbol{b}_{L} = \\sum_{i=1}^{n_{inputs}} \\delta_L = (n_{categories}) .$$ \n", + "\n", + "The error in the hidden layer is \n", + "\n", + "$$ \\Delta_h = \\delta_L W_{L}^T \\circ f'(z_{h}) = \\delta_L W_{L}^T \\circ a_{h} \\circ (1 - a_{h}) = (n_{inputs}, n_{hidden}) ,$$ \n", + "\n", + "where $f'(a_{h})$ is the derivative of the activation in the hidden layer. The matrix products mean\n", + "that we are summing up the products for each neuron in the output layer. The symbol $\\circ$ denotes\n", + "the *Hadamard product*, meaning element-wise multiplication. \n", + "\n", + "This again gives us the gradients in the hidden layer: \n", + "\n", + "$$ \\nabla W_{h} = X^T \\delta_h = (n_{features}, n_{hidden}) ,$$ \n", + "\n", + "$$ \\nabla b_{h} = \\sum_{i=1}^{n_{inputs}} \\delta_h = (n_{hidden}) .$$" + ] + }, + { + "cell_type": "code", + "execution_count": 7, + "id": "657c604c", + "metadata": { + "collapsed": false, + "editable": true + }, + "outputs": [], + "source": [ + "# to categorical turns our integer vector into a onehot representation\n", + "from sklearn.metrics import accuracy_score\n", + "\n", + "# one-hot in numpy\n", + "def to_categorical_numpy(integer_vector):\n", + " n_inputs = len(integer_vector)\n", + " n_categories = np.max(integer_vector) + 1\n", + " onehot_vector = np.zeros((n_inputs, n_categories))\n", + " onehot_vector[range(n_inputs), integer_vector] = 1\n", + " \n", + " return onehot_vector\n", + "\n", + "#Y_train_onehot, Y_test_onehot = to_categorical(Y_train), to_categorical(Y_test)\n", + "Y_train_onehot, Y_test_onehot = to_categorical_numpy(Y_train), to_categorical_numpy(Y_test)\n", + "\n", + "def feed_forward_train(X):\n", + " # weighted sum of inputs to the hidden layer\n", + " z_h = np.matmul(X, hidden_weights) + hidden_bias\n", + " # activation in the hidden layer\n", + " a_h = sigmoid(z_h)\n", + " \n", + " # weighted sum of inputs to the output layer\n", + " z_o = np.matmul(a_h, output_weights) + output_bias\n", + " # softmax output\n", + " # axis 0 holds each input and axis 1 the probabilities of each category\n", + " exp_term = np.exp(z_o)\n", + " probabilities = exp_term / np.sum(exp_term, axis=1, keepdims=True)\n", + " \n", + " # for backpropagation need activations in hidden and output layers\n", + " return a_h, probabilities\n", + "\n", + "def backpropagation(X, Y):\n", + " a_h, probabilities = feed_forward_train(X)\n", + " \n", + " # error in the output layer\n", + " error_output = probabilities - Y\n", + " # error in the hidden layer\n", + " error_hidden = np.matmul(error_output, output_weights.T) * a_h * (1 - a_h)\n", + " \n", + " # gradients for the output layer\n", + " output_weights_gradient = np.matmul(a_h.T, error_output)\n", + " output_bias_gradient = np.sum(error_output, axis=0)\n", + " \n", + " # gradient for the hidden layer\n", + " hidden_weights_gradient = np.matmul(X.T, error_hidden)\n", + " hidden_bias_gradient = np.sum(error_hidden, axis=0)\n", + "\n", + " return output_weights_gradient, output_bias_gradient, hidden_weights_gradient, hidden_bias_gradient\n", + "\n", + "print(\"Old accuracy on training data: \" + str(accuracy_score(predict(X_train), Y_train)))\n", + "\n", + "eta = 0.01\n", + "lmbd = 0.01\n", + "for i in range(1000):\n", + " # calculate gradients\n", + " dWo, dBo, dWh, dBh = backpropagation(X_train, Y_train_onehot)\n", + " \n", + " # regularization term gradients\n", + " dWo += lmbd * output_weights\n", + " dWh += lmbd * hidden_weights\n", + " \n", + " # update weights and biases\n", + " output_weights -= eta * dWo\n", + " output_bias -= eta * dBo\n", + " hidden_weights -= eta * dWh\n", + " hidden_bias -= eta * dBh\n", + "\n", + "print(\"New accuracy on training data: \" + str(accuracy_score(predict(X_train), Y_train)))" + ] + }, + { + "cell_type": "markdown", + "id": "3c595805", + "metadata": { + "editable": true + }, + "source": [ + "## Improving performance\n", + "\n", + "As we can see the network does not seem to be learning at all. It seems to be just guessing the label for each image. \n", + "In order to obtain a network that does something useful, we will have to do a bit more work. \n", + "\n", + "The choice of *hyperparameters* such as learning rate and regularization parameter is hugely influential for the performance of the network. Typically a *grid-search* is performed, wherein we test different hyperparameters separated by orders of magnitude. For example we could test the learning rates $\\eta = 10^{-6}, 10^{-5},...,10^{-1}$ with different regularization parameters $\\lambda = 10^{-6},...,10^{-0}$. \n", + "\n", + "Next, we haven't implemented minibatching yet, which introduces stochasticity and is though to act as an important regularizer on the weights. We call a feed-forward + backward pass with a minibatch an *iteration*, and a full training period\n", + "going through the entire dataset ($n/M$ batches) an *epoch*.\n", + "\n", + "If this does not improve network performance, you may want to consider altering the network architecture, adding more neurons or hidden layers. \n", + "Andrew Ng goes through some of these considerations in this [video](https://youtu.be/F1ka6a13S9I). You can find a summary of the video [here](https://kevinzakka.github.io/2016/09/26/applying-deep-learning/)." + ] + }, + { + "cell_type": "markdown", + "id": "17d5f534", + "metadata": { + "editable": true + }, + "source": [ + "## Full object-oriented implementation\n", + "\n", + "It is very natural to think of the network as an object, with specific instances of the network\n", + "being realizations of this object with different hyperparameters. An implementation using Python classes provides a clean structure and interface, and the full implementation of our neural network is given below." + ] + }, + { + "cell_type": "code", + "execution_count": 8, + "id": "5aeaa0cd", + "metadata": { + "collapsed": false, + "editable": true + }, + "outputs": [], + "source": [ + "class NeuralNetwork:\n", + " def __init__(\n", + " self,\n", + " X_data,\n", + " Y_data,\n", + " n_hidden_neurons=50,\n", + " n_categories=10,\n", + " epochs=10,\n", + " batch_size=100,\n", + " eta=0.1,\n", + " lmbd=0.0):\n", + "\n", + " self.X_data_full = X_data\n", + " self.Y_data_full = Y_data\n", + "\n", + " self.n_inputs = X_data.shape[0]\n", + " self.n_features = X_data.shape[1]\n", + " self.n_hidden_neurons = n_hidden_neurons\n", + " self.n_categories = n_categories\n", + "\n", + " self.epochs = epochs\n", + " self.batch_size = batch_size\n", + " self.iterations = self.n_inputs // self.batch_size\n", + " self.eta = eta\n", + " self.lmbd = lmbd\n", + "\n", + " self.create_biases_and_weights()\n", + "\n", + " def create_biases_and_weights(self):\n", + " self.hidden_weights = np.random.randn(self.n_features, self.n_hidden_neurons)\n", + " self.hidden_bias = np.zeros(self.n_hidden_neurons) + 0.01\n", + "\n", + " self.output_weights = np.random.randn(self.n_hidden_neurons, self.n_categories)\n", + " self.output_bias = np.zeros(self.n_categories) + 0.01\n", + "\n", + " def feed_forward(self):\n", + " # feed-forward for training\n", + " self.z_h = np.matmul(self.X_data, self.hidden_weights) + self.hidden_bias\n", + " self.a_h = sigmoid(self.z_h)\n", + "\n", + " self.z_o = np.matmul(self.a_h, self.output_weights) + self.output_bias\n", + "\n", + " exp_term = np.exp(self.z_o)\n", + " self.probabilities = exp_term / np.sum(exp_term, axis=1, keepdims=True)\n", + "\n", + " def feed_forward_out(self, X):\n", + " # feed-forward for output\n", + " z_h = np.matmul(X, self.hidden_weights) + self.hidden_bias\n", + " a_h = sigmoid(z_h)\n", + "\n", + " z_o = np.matmul(a_h, self.output_weights) + self.output_bias\n", + " \n", + " exp_term = np.exp(z_o)\n", + " probabilities = exp_term / np.sum(exp_term, axis=1, keepdims=True)\n", + " return probabilities\n", + "\n", + " def backpropagation(self):\n", + " error_output = self.probabilities - self.Y_data\n", + " error_hidden = np.matmul(error_output, self.output_weights.T) * self.a_h * (1 - self.a_h)\n", + "\n", + " self.output_weights_gradient = np.matmul(self.a_h.T, error_output)\n", + " self.output_bias_gradient = np.sum(error_output, axis=0)\n", + "\n", + " self.hidden_weights_gradient = np.matmul(self.X_data.T, error_hidden)\n", + " self.hidden_bias_gradient = np.sum(error_hidden, axis=0)\n", + "\n", + " if self.lmbd > 0.0:\n", + " self.output_weights_gradient += self.lmbd * self.output_weights\n", + " self.hidden_weights_gradient += self.lmbd * self.hidden_weights\n", + "\n", + " self.output_weights -= self.eta * self.output_weights_gradient\n", + " self.output_bias -= self.eta * self.output_bias_gradient\n", + " self.hidden_weights -= self.eta * self.hidden_weights_gradient\n", + " self.hidden_bias -= self.eta * self.hidden_bias_gradient\n", + "\n", + " def predict(self, X):\n", + " probabilities = self.feed_forward_out(X)\n", + " return np.argmax(probabilities, axis=1)\n", + "\n", + " def predict_probabilities(self, X):\n", + " probabilities = self.feed_forward_out(X)\n", + " return probabilities\n", + "\n", + " def train(self):\n", + " data_indices = np.arange(self.n_inputs)\n", + "\n", + " for i in range(self.epochs):\n", + " for j in range(self.iterations):\n", + " # pick datapoints with replacement\n", + " chosen_datapoints = np.random.choice(\n", + " data_indices, size=self.batch_size, replace=False\n", + " )\n", + "\n", + " # minibatch training data\n", + " self.X_data = self.X_data_full[chosen_datapoints]\n", + " self.Y_data = self.Y_data_full[chosen_datapoints]\n", + "\n", + " self.feed_forward()\n", + " self.backpropagation()" + ] + }, + { + "cell_type": "markdown", + "id": "77f30e1e", + "metadata": { + "editable": true + }, + "source": [ + "## Evaluate model performance on test data\n", + "\n", + "To measure the performance of our network we evaluate how well it does it data it has never seen before, i.e. the test data. \n", + "We measure the performance of the network using the *accuracy* score. \n", + "The accuracy is as you would expect just the number of images correctly labeled divided by the total number of images. A perfect classifier will have an accuracy score of $1$. \n", + "\n", + "$$ \\text{Accuracy} = \\frac{\\sum_{i=1}^n I(\\tilde{y}_i = y_i)}{n} ,$$ \n", + "\n", + "where $I$ is the indicator function, $1$ if $\\tilde{y}_i = y_i$ and $0$ otherwise." + ] + }, + { + "cell_type": "code", + "execution_count": 9, + "id": "c1ab41e1", + "metadata": { + "collapsed": false, + "editable": true + }, + "outputs": [], + "source": [ + "epochs = 100\n", + "batch_size = 100\n", + "\n", + "dnn = NeuralNetwork(X_train, Y_train_onehot, eta=eta, lmbd=lmbd, epochs=epochs, batch_size=batch_size,\n", + " n_hidden_neurons=n_hidden_neurons, n_categories=n_categories)\n", + "dnn.train()\n", + "test_predict = dnn.predict(X_test)\n", + "\n", + "# accuracy score from scikit library\n", + "print(\"Accuracy score on test set: \", accuracy_score(Y_test, test_predict))\n", + "\n", + "# equivalent in numpy\n", + "def accuracy_score_numpy(Y_test, Y_pred):\n", + " return np.sum(Y_test == Y_pred) / len(Y_test)\n", + "\n", + "#print(\"Accuracy score on test set: \", accuracy_score_numpy(Y_test, test_predict))" + ] + }, + { + "cell_type": "markdown", + "id": "1b2b35c3", + "metadata": { + "editable": true + }, + "source": [ + "## Adjust hyperparameters\n", + "\n", + "We now perform a grid search to find the optimal hyperparameters for the network. \n", + "Note that we are only using 1 layer with 50 neurons, and human performance is estimated to be around $98\\%$ ($2\\%$ error rate)." + ] + }, + { + "cell_type": "code", + "execution_count": 10, + "id": "ac2ab4a6", + "metadata": { + "collapsed": false, + "editable": true + }, + "outputs": [], + "source": [ + "eta_vals = np.logspace(-5, 1, 7)\n", + "lmbd_vals = np.logspace(-5, 1, 7)\n", + "# store the models for later use\n", + "DNN_numpy = np.zeros((len(eta_vals), len(lmbd_vals)), dtype=object)\n", + "\n", + "# grid search\n", + "for i, eta in enumerate(eta_vals):\n", + " for j, lmbd in enumerate(lmbd_vals):\n", + " dnn = NeuralNetwork(X_train, Y_train_onehot, eta=eta, lmbd=lmbd, epochs=epochs, batch_size=batch_size,\n", + " n_hidden_neurons=n_hidden_neurons, n_categories=n_categories)\n", + " dnn.train()\n", + " \n", + " DNN_numpy[i][j] = dnn\n", + " \n", + " test_predict = dnn.predict(X_test)\n", + " \n", + " print(\"Learning rate = \", eta)\n", + " print(\"Lambda = \", lmbd)\n", + " print(\"Accuracy score on test set: \", accuracy_score(Y_test, test_predict))\n", + " print()" + ] + }, + { + "cell_type": "markdown", + "id": "f3e5dfec", + "metadata": { + "editable": true + }, + "source": [ + "## Visualization" + ] + }, + { + "cell_type": "code", + "execution_count": 11, + "id": "6d875555", + "metadata": { + "collapsed": false, + "editable": true + }, + "outputs": [], + "source": [ + "# visual representation of grid search\n", + "# uses seaborn heatmap, you can also do this with matplotlib imshow\n", + "import seaborn as sns\n", + "\n", + "sns.set()\n", + "\n", + "train_accuracy = np.zeros((len(eta_vals), len(lmbd_vals)))\n", + "test_accuracy = np.zeros((len(eta_vals), len(lmbd_vals)))\n", + "\n", + "for i in range(len(eta_vals)):\n", + " for j in range(len(lmbd_vals)):\n", + " dnn = DNN_numpy[i][j]\n", + " \n", + " train_pred = dnn.predict(X_train) \n", + " test_pred = dnn.predict(X_test)\n", + "\n", + " train_accuracy[i][j] = accuracy_score(Y_train, train_pred)\n", + " test_accuracy[i][j] = accuracy_score(Y_test, test_pred)\n", + "\n", + " \n", + "fig, ax = plt.subplots(figsize = (10, 10))\n", + "sns.heatmap(train_accuracy, annot=True, ax=ax, cmap=\"viridis\")\n", + "ax.set_title(\"Training Accuracy\")\n", + "ax.set_ylabel(\"$\\eta$\")\n", + "ax.set_xlabel(\"$\\lambda$\")\n", + "plt.show()\n", + "\n", + "fig, ax = plt.subplots(figsize = (10, 10))\n", + "sns.heatmap(test_accuracy, annot=True, ax=ax, cmap=\"viridis\")\n", + "ax.set_title(\"Test Accuracy\")\n", + "ax.set_ylabel(\"$\\eta$\")\n", + "ax.set_xlabel(\"$\\lambda$\")\n", + "plt.show()" + ] + }, + { + "cell_type": "markdown", + "id": "81c742e3", + "metadata": { + "editable": true + }, + "source": [ + "## scikit-learn implementation\n", + "\n", + "**scikit-learn** focuses more\n", + "on traditional machine learning methods, such as regression,\n", + "clustering, decision trees, etc. As such, it has only two types of\n", + "neural networks: Multi Layer Perceptron outputting continuous values,\n", + "*MPLRegressor*, and Multi Layer Perceptron outputting labels,\n", + "*MLPClassifier*. We will see how simple it is to use these classes.\n", + "\n", + "**scikit-learn** implements a few improvements from our neural network,\n", + "such as early stopping, a varying learning rate, different\n", + "optimization methods, etc. We would therefore expect a better\n", + "performance overall." + ] + }, + { + "cell_type": "code", + "execution_count": 12, + "id": "d21828bb", + "metadata": { + "collapsed": false, + "editable": true + }, + "outputs": [], + "source": [ + "from sklearn.neural_network import MLPClassifier\n", + "# store models for later use\n", + "DNN_scikit = np.zeros((len(eta_vals), len(lmbd_vals)), dtype=object)\n", + "\n", + "for i, eta in enumerate(eta_vals):\n", + " for j, lmbd in enumerate(lmbd_vals):\n", + " dnn = MLPClassifier(hidden_layer_sizes=(n_hidden_neurons), activation='logistic',\n", + " alpha=lmbd, learning_rate_init=eta, max_iter=epochs)\n", + " dnn.fit(X_train, Y_train)\n", + " \n", + " DNN_scikit[i][j] = dnn\n", + " \n", + " print(\"Learning rate = \", eta)\n", + " print(\"Lambda = \", lmbd)\n", + " print(\"Accuracy score on test set: \", dnn.score(X_test, Y_test))\n", + " print()" + ] + }, + { + "cell_type": "markdown", + "id": "4b40579d", + "metadata": { + "editable": true + }, + "source": [ + "## Visualization" + ] + }, + { + "cell_type": "code", + "execution_count": 13, + "id": "bfc97198", + "metadata": { + "collapsed": false, + "editable": true + }, + "outputs": [], + "source": [ + "# optional\n", + "# visual representation of grid search\n", + "# uses seaborn heatmap, could probably do this in matplotlib\n", + "import seaborn as sns\n", + "\n", + "sns.set()\n", + "\n", + "train_accuracy = np.zeros((len(eta_vals), len(lmbd_vals)))\n", + "test_accuracy = np.zeros((len(eta_vals), len(lmbd_vals)))\n", + "\n", + "for i in range(len(eta_vals)):\n", + " for j in range(len(lmbd_vals)):\n", + " dnn = DNN_scikit[i][j]\n", + " \n", + " train_pred = dnn.predict(X_train) \n", + " test_pred = dnn.predict(X_test)\n", + "\n", + " train_accuracy[i][j] = accuracy_score(Y_train, train_pred)\n", + " test_accuracy[i][j] = accuracy_score(Y_test, test_pred)\n", + "\n", + " \n", + "fig, ax = plt.subplots(figsize = (10, 10))\n", + "sns.heatmap(train_accuracy, annot=True, ax=ax, cmap=\"viridis\")\n", + "ax.set_title(\"Training Accuracy\")\n", + "ax.set_ylabel(\"$\\eta$\")\n", + "ax.set_xlabel(\"$\\lambda$\")\n", + "plt.show()\n", + "\n", + "fig, ax = plt.subplots(figsize = (10, 10))\n", + "sns.heatmap(test_accuracy, annot=True, ax=ax, cmap=\"viridis\")\n", + "ax.set_title(\"Test Accuracy\")\n", + "ax.set_ylabel(\"$\\eta$\")\n", + "ax.set_xlabel(\"$\\lambda$\")\n", + "plt.show()" + ] + }, + { + "cell_type": "markdown", + "id": "550680d5", + "metadata": { + "editable": true + }, + "source": [ + "## Building neural networks in Tensorflow and Keras\n", + "\n", + "Now we want to build on the experience gained from our neural network implementation in NumPy and scikit-learn\n", + "and use it to construct a neural network in Tensorflow. Once we have constructed a neural network in NumPy\n", + "and Tensorflow, building one in Keras is really quite trivial, though the performance may suffer. \n", + "\n", + "In our previous example we used only one hidden layer, and in this we will use two. From this it should be quite\n", + "clear how to build one using an arbitrary number of hidden layers, using data structures such as Python lists or\n", + "NumPy arrays." + ] + }, + { + "cell_type": "markdown", + "id": "bbade60e", + "metadata": { + "editable": true + }, + "source": [ + "## Tensorflow\n", + "\n", + "Tensorflow is an open source library machine learning library\n", + "developed by the Google Brain team for internal use. It was released\n", + "under the Apache 2.0 open source license in November 9, 2015.\n", + "\n", + "Tensorflow is a computational framework that allows you to construct\n", + "machine learning models at different levels of abstraction, from\n", + "high-level, object-oriented APIs like Keras, down to the C++ kernels\n", + "that Tensorflow is built upon. The higher levels of abstraction are\n", + "simpler to use, but less flexible, and our choice of implementation\n", + "should reflect the problems we are trying to solve.\n", + "\n", + "[Tensorflow uses](https://www.tensorflow.org/guide/graphs) so-called graphs to represent your computation\n", + "in terms of the dependencies between individual operations, such that you first build a Tensorflow *graph*\n", + "to represent your model, and then create a Tensorflow *session* to run the graph.\n", + "\n", + "In this guide we will analyze the same data as we did in our NumPy and\n", + "scikit-learn tutorial, gathered from the MNIST database of images. We\n", + "will give an introduction to the lower level Python Application\n", + "Program Interfaces (APIs), and see how we use them to build our graph.\n", + "Then we will build (effectively) the same graph in Keras, to see just\n", + "how simple solving a machine learning problem can be.\n", + "\n", + "To install tensorflow on Unix/Linux systems, use pip as" + ] + }, + { + "cell_type": "code", + "execution_count": 14, + "id": "e5505ed3", + "metadata": { + "collapsed": false, + "editable": true + }, + "outputs": [], + "source": [ + "pip3 install tensorflow" + ] + }, + { + "cell_type": "markdown", + "id": "98d4797a", + "metadata": { + "editable": true + }, + "source": [ + "and/or if you use **anaconda**, just write (or install from the graphical user interface)\n", + "(current release of CPU-only TensorFlow)" + ] + }, + { + "cell_type": "code", + "execution_count": 15, + "id": "95b05730", + "metadata": { + "collapsed": false, + "editable": true + }, + "outputs": [], + "source": [ + "conda create -n tf tensorflow\n", + "conda activate tf" + ] + }, + { + "cell_type": "markdown", + "id": "94170f18", + "metadata": { + "editable": true + }, + "source": [ + "To install the current release of GPU TensorFlow" + ] + }, + { + "cell_type": "code", + "execution_count": 16, + "id": "65ae6e69", + "metadata": { + "collapsed": false, + "editable": true + }, + "outputs": [], + "source": [ + "conda create -n tf-gpu tensorflow-gpu\n", + "conda activate tf-gpu" + ] + }, + { + "cell_type": "markdown", + "id": "df39c5c6", + "metadata": { + "editable": true + }, + "source": [ + "## Using Keras\n", + "\n", + "Keras is a high level [neural network](https://en.wikipedia.org/wiki/Application_programming_interface)\n", + "that supports Tensorflow, CTNK and Theano as backends. \n", + "If you have Anaconda installed you may run the following command" + ] + }, + { + "cell_type": "code", + "execution_count": 17, + "id": "d0d627f6", + "metadata": { + "collapsed": false, + "editable": true + }, + "outputs": [], + "source": [ + "conda install keras" + ] + }, + { + "cell_type": "markdown", + "id": "4124b5af", + "metadata": { + "editable": true + }, + "source": [ + "You can look up the [instructions here](https://keras.io/) for more information.\n", + "\n", + "We will to a large extent use **keras** in this course." + ] + }, + { + "cell_type": "markdown", + "id": "22dc977c", + "metadata": { + "editable": true + }, + "source": [ + "## Collect and pre-process data\n", + "\n", + "Let us look again at the MINST data set." + ] + }, + { + "cell_type": "code", + "execution_count": 18, + "id": "7df2e4d4", + "metadata": { + "collapsed": false, + "editable": true + }, + "outputs": [], + "source": [ + "# import necessary packages\n", + "import numpy as np\n", + "import matplotlib.pyplot as plt\n", + "import tensorflow as tf\n", + "from sklearn import datasets\n", + "\n", + "\n", + "# ensure the same random numbers appear every time\n", + "np.random.seed(0)\n", + "\n", + "# display images in notebook\n", + "%matplotlib inline\n", + "plt.rcParams['figure.figsize'] = (12,12)\n", + "\n", + "\n", + "# download MNIST dataset\n", + "digits = datasets.load_digits()\n", + "\n", + "# define inputs and labels\n", + "inputs = digits.images\n", + "labels = digits.target\n", + "\n", + "print(\"inputs = (n_inputs, pixel_width, pixel_height) = \" + str(inputs.shape))\n", + "print(\"labels = (n_inputs) = \" + str(labels.shape))\n", + "\n", + "\n", + "# flatten the image\n", + "# the value -1 means dimension is inferred from the remaining dimensions: 8x8 = 64\n", + "n_inputs = len(inputs)\n", + "inputs = inputs.reshape(n_inputs, -1)\n", + "print(\"X = (n_inputs, n_features) = \" + str(inputs.shape))\n", + "\n", + "\n", + "# choose some random images to display\n", + "indices = np.arange(n_inputs)\n", + "random_indices = np.random.choice(indices, size=5)\n", + "\n", + "for i, image in enumerate(digits.images[random_indices]):\n", + " plt.subplot(1, 5, i+1)\n", + " plt.axis('off')\n", + " plt.imshow(image, cmap=plt.cm.gray_r, interpolation='nearest')\n", + " plt.title(\"Label: %d\" % digits.target[random_indices[i]])\n", + "plt.show()" + ] + }, + { + "cell_type": "code", + "execution_count": 19, + "id": "4f5bff6f", + "metadata": { + "collapsed": false, + "editable": true + }, + "outputs": [], + "source": [ + "from tensorflow.keras.layers import Input\n", + "from tensorflow.keras.models import Sequential #This allows appending layers to existing models\n", + "from tensorflow.keras.layers import Dense #This allows defining the characteristics of a particular layer\n", + "from tensorflow.keras import optimizers #This allows using whichever optimiser we want (sgd,adam,RMSprop)\n", + "from tensorflow.keras import regularizers #This allows using whichever regularizer we want (l1,l2,l1_l2)\n", + "from tensorflow.keras.utils import to_categorical #This allows using categorical cross entropy as the cost function\n", + "\n", + "from sklearn.model_selection import train_test_split\n", + "\n", + "# one-hot representation of labels\n", + "labels = to_categorical(labels)\n", + "\n", + "# split into train and test data\n", + "train_size = 0.8\n", + "test_size = 1 - train_size\n", + "X_train, X_test, Y_train, Y_test = train_test_split(inputs, labels, train_size=train_size,\n", + " test_size=test_size)" + ] + }, + { + "cell_type": "code", + "execution_count": 20, + "id": "3ae517ea", + "metadata": { + "collapsed": false, + "editable": true + }, + "outputs": [], + "source": [ + "\n", + "epochs = 100\n", + "batch_size = 100\n", + "n_neurons_layer1 = 100\n", + "n_neurons_layer2 = 50\n", + "n_categories = 10\n", + "eta_vals = np.logspace(-5, 1, 7)\n", + "lmbd_vals = np.logspace(-5, 1, 7)\n", + "def create_neural_network_keras(n_neurons_layer1, n_neurons_layer2, n_categories, eta, lmbd):\n", + " model = Sequential()\n", + " model.add(Dense(n_neurons_layer1, activation='sigmoid', kernel_regularizer=regularizers.l2(lmbd)))\n", + " model.add(Dense(n_neurons_layer2, activation='sigmoid', kernel_regularizer=regularizers.l2(lmbd)))\n", + " model.add(Dense(n_categories, activation='softmax'))\n", + " \n", + " sgd = optimizers.SGD(lr=eta)\n", + " model.compile(loss='categorical_crossentropy', optimizer=sgd, metrics=['accuracy'])\n", + " \n", + " return model" + ] + }, + { + "cell_type": "code", + "execution_count": 21, + "id": "c880b209", + "metadata": { + "collapsed": false, + "editable": true + }, + "outputs": [], + "source": [ + "DNN_keras = np.zeros((len(eta_vals), len(lmbd_vals)), dtype=object)\n", + " \n", + "for i, eta in enumerate(eta_vals):\n", + " for j, lmbd in enumerate(lmbd_vals):\n", + " DNN = create_neural_network_keras(n_neurons_layer1, n_neurons_layer2, n_categories,\n", + " eta=eta, lmbd=lmbd)\n", + " DNN.fit(X_train, Y_train, epochs=epochs, batch_size=batch_size, verbose=0)\n", + " scores = DNN.evaluate(X_test, Y_test)\n", + " \n", + " DNN_keras[i][j] = DNN\n", + " \n", + " print(\"Learning rate = \", eta)\n", + " print(\"Lambda = \", lmbd)\n", + " print(\"Test accuracy: %.3f\" % scores[1])\n", + " print()" + ] + }, + { + "cell_type": "code", + "execution_count": 22, + "id": "cf8cea73", + "metadata": { + "collapsed": false, + "editable": true + }, + "outputs": [], + "source": [ + "# optional\n", + "# visual representation of grid search\n", + "# uses seaborn heatmap, could probably do this in matplotlib\n", + "import seaborn as sns\n", + "\n", + "sns.set()\n", + "\n", + "train_accuracy = np.zeros((len(eta_vals), len(lmbd_vals)))\n", + "test_accuracy = np.zeros((len(eta_vals), len(lmbd_vals)))\n", + "\n", + "for i in range(len(eta_vals)):\n", + " for j in range(len(lmbd_vals)):\n", + " DNN = DNN_keras[i][j]\n", + "\n", + " train_accuracy[i][j] = DNN.evaluate(X_train, Y_train)[1]\n", + " test_accuracy[i][j] = DNN.evaluate(X_test, Y_test)[1]\n", + "\n", + " \n", + "fig, ax = plt.subplots(figsize = (10, 10))\n", + "sns.heatmap(train_accuracy, annot=True, ax=ax, cmap=\"viridis\")\n", + "ax.set_title(\"Training Accuracy\")\n", + "ax.set_ylabel(\"$\\eta$\")\n", + "ax.set_xlabel(\"$\\lambda$\")\n", + "plt.show()\n", + "\n", + "fig, ax = plt.subplots(figsize = (10, 10))\n", + "sns.heatmap(test_accuracy, annot=True, ax=ax, cmap=\"viridis\")\n", + "ax.set_title(\"Test Accuracy\")\n", + "ax.set_ylabel(\"$\\eta$\")\n", + "ax.set_xlabel(\"$\\lambda$\")\n", + "plt.show()" + ] + }, + { + "cell_type": "markdown", + "id": "3024761c", + "metadata": { + "editable": true + }, + "source": [ + "## Building a neural network code\n", + "\n", + "Here we present a flexible object oriented codebase\n", + "for a feed forward neural network, along with a demonstration of how\n", + "to use it. Before we get into the details of the neural network, we\n", + "will first present some implementations of various schedulers, cost\n", + "functions and activation functions that can be used together with the\n", + "neural network.\n", + "\n", + "The codes here were developed by Eric Reber and Gregor Kajda during spring 2023." + ] + }, + { + "cell_type": "markdown", + "id": "53987675", + "metadata": { + "editable": true + }, + "source": [ + "### Learning rate methods\n", + "\n", + "The code below shows object oriented implementations of the Constant,\n", + "Momentum, Adagrad, AdagradMomentum, RMS prop and Adam schedulers. All\n", + "of the classes belong to the shared abstract Scheduler class, and\n", + "share the update_change() and reset() methods allowing for any of the\n", + "schedulers to be seamlessly used during the training stage, as will\n", + "later be shown in the fit() method of the neural\n", + "network. Update_change() only has one parameter, the gradient\n", + "($δ^l_ja^{l−1}_k$), and returns the change which will be subtracted\n", + "from the weights. The reset() function takes no parameters, and resets\n", + "the desired variables. For Constant and Momentum, reset does nothing." + ] + }, + { + "cell_type": "code", + "execution_count": 23, + "id": "5ed03631", + "metadata": { + "collapsed": false, + "editable": true + }, + "outputs": [], + "source": [ + "import autograd.numpy as np\n", + "\n", + "class Scheduler:\n", + " \"\"\"\n", + " Abstract class for Schedulers\n", + " \"\"\"\n", + "\n", + " def __init__(self, eta):\n", + " self.eta = eta\n", + "\n", + " # should be overwritten\n", + " def update_change(self, gradient):\n", + " raise NotImplementedError\n", + "\n", + " # overwritten if needed\n", + " def reset(self):\n", + " pass\n", + "\n", + "\n", + "class Constant(Scheduler):\n", + " def __init__(self, eta):\n", + " super().__init__(eta)\n", + "\n", + " def update_change(self, gradient):\n", + " return self.eta * gradient\n", + " \n", + " def reset(self):\n", + " pass\n", + "\n", + "\n", + "class Momentum(Scheduler):\n", + " def __init__(self, eta: float, momentum: float):\n", + " super().__init__(eta)\n", + " self.momentum = momentum\n", + " self.change = 0\n", + "\n", + " def update_change(self, gradient):\n", + " self.change = self.momentum * self.change + self.eta * gradient\n", + " return self.change\n", + "\n", + " def reset(self):\n", + " pass\n", + "\n", + "\n", + "class Adagrad(Scheduler):\n", + " def __init__(self, eta):\n", + " super().__init__(eta)\n", + " self.G_t = None\n", + "\n", + " def update_change(self, gradient):\n", + " delta = 1e-8 # avoid division ny zero\n", + "\n", + " if self.G_t is None:\n", + " self.G_t = np.zeros((gradient.shape[0], gradient.shape[0]))\n", + "\n", + " self.G_t += gradient @ gradient.T\n", + "\n", + " G_t_inverse = 1 / (\n", + " delta + np.sqrt(np.reshape(np.diagonal(self.G_t), (self.G_t.shape[0], 1)))\n", + " )\n", + " return self.eta * gradient * G_t_inverse\n", + "\n", + " def reset(self):\n", + " self.G_t = None\n", + "\n", + "\n", + "class AdagradMomentum(Scheduler):\n", + " def __init__(self, eta, momentum):\n", + " super().__init__(eta)\n", + " self.G_t = None\n", + " self.momentum = momentum\n", + " self.change = 0\n", + "\n", + " def update_change(self, gradient):\n", + " delta = 1e-8 # avoid division ny zero\n", + "\n", + " if self.G_t is None:\n", + " self.G_t = np.zeros((gradient.shape[0], gradient.shape[0]))\n", + "\n", + " self.G_t += gradient @ gradient.T\n", + "\n", + " G_t_inverse = 1 / (\n", + " delta + np.sqrt(np.reshape(np.diagonal(self.G_t), (self.G_t.shape[0], 1)))\n", + " )\n", + " self.change = self.change * self.momentum + self.eta * gradient * G_t_inverse\n", + " return self.change\n", + "\n", + " def reset(self):\n", + " self.G_t = None\n", + "\n", + "\n", + "class RMS_prop(Scheduler):\n", + " def __init__(self, eta, rho):\n", + " super().__init__(eta)\n", + " self.rho = rho\n", + " self.second = 0.0\n", + "\n", + " def update_change(self, gradient):\n", + " delta = 1e-8 # avoid division ny zero\n", + " self.second = self.rho * self.second + (1 - self.rho) * gradient * gradient\n", + " return self.eta * gradient / (np.sqrt(self.second + delta))\n", + "\n", + " def reset(self):\n", + " self.second = 0.0\n", + "\n", + "\n", + "class Adam(Scheduler):\n", + " def __init__(self, eta, rho, rho2):\n", + " super().__init__(eta)\n", + " self.rho = rho\n", + " self.rho2 = rho2\n", + " self.moment = 0\n", + " self.second = 0\n", + " self.n_epochs = 1\n", + "\n", + " def update_change(self, gradient):\n", + " delta = 1e-8 # avoid division ny zero\n", + "\n", + " self.moment = self.rho * self.moment + (1 - self.rho) * gradient\n", + " self.second = self.rho2 * self.second + (1 - self.rho2) * gradient * gradient\n", + "\n", + " moment_corrected = self.moment / (1 - self.rho**self.n_epochs)\n", + " second_corrected = self.second / (1 - self.rho2**self.n_epochs)\n", + "\n", + " return self.eta * moment_corrected / (np.sqrt(second_corrected + delta))\n", + "\n", + " def reset(self):\n", + " self.n_epochs += 1\n", + " self.moment = 0\n", + " self.second = 0" + ] + }, + { + "cell_type": "markdown", + "id": "8b98c385", + "metadata": { + "editable": true + }, + "source": [ + "### Usage of the above learning rate schedulers\n", + "\n", + "To initalize a scheduler, simply create the object and pass in the\n", + "necessary parameters such as the learning rate and the momentum as\n", + "shown below. As the Scheduler class is an abstract class it should not\n", + "called directly, and will raise an error upon usage." + ] + }, + { + "cell_type": "code", + "execution_count": 24, + "id": "54298abd", + "metadata": { + "collapsed": false, + "editable": true + }, + "outputs": [], + "source": [ + "momentum_scheduler = Momentum(eta=1e-3, momentum=0.9)\n", + "adam_scheduler = Adam(eta=1e-3, rho=0.9, rho2=0.999)" + ] + }, + { + "cell_type": "markdown", + "id": "c03dc1b1", + "metadata": { + "editable": true + }, + "source": [ + "Here is a small example for how a segment of code using schedulers\n", + "could look. Switching out the schedulers is simple." + ] + }, + { + "cell_type": "code", + "execution_count": 25, + "id": "5a2f7871", + "metadata": { + "collapsed": false, + "editable": true + }, + "outputs": [], + "source": [ + "weights = np.ones((3,3))\n", + "print(f\"Before scheduler:\\n{weights=}\")\n", + "\n", + "epochs = 10\n", + "for e in range(epochs):\n", + " gradient = np.random.rand(3, 3)\n", + " change = adam_scheduler.update_change(gradient)\n", + " weights = weights - change\n", + " adam_scheduler.reset()\n", + "\n", + "print(f\"\\nAfter scheduler:\\n{weights=}\")" + ] + }, + { + "cell_type": "markdown", + "id": "07fa559a", + "metadata": { + "editable": true + }, + "source": [ + "### Cost functions\n", + "\n", + "Here we discuss cost functions that can be used when creating the\n", + "neural network. Every cost function takes the target vector as its\n", + "parameter, and returns a function valued only at $x$ such that it may\n", + "easily be differentiated." + ] + }, + { + "cell_type": "code", + "execution_count": 26, + "id": "718450a2", + "metadata": { + "collapsed": false, + "editable": true + }, + "outputs": [], + "source": [ + "import autograd.numpy as np\n", + "\n", + "def CostOLS(target):\n", + " \n", + " def func(X):\n", + " return (1.0 / target.shape[0]) * np.sum((target - X) ** 2)\n", + "\n", + " return func\n", + "\n", + "\n", + "def CostLogReg(target):\n", + "\n", + " def func(X):\n", + " \n", + " return -(1.0 / target.shape[0]) * np.sum(\n", + " (target * np.log(X + 10e-10)) + ((1 - target) * np.log(1 - X + 10e-10))\n", + " )\n", + "\n", + " return func\n", + "\n", + "\n", + "def CostCrossEntropy(target):\n", + " \n", + " def func(X):\n", + " return -(1.0 / target.size) * np.sum(target * np.log(X + 10e-10))\n", + "\n", + " return func" + ] + }, + { + "cell_type": "markdown", + "id": "0db28b52", + "metadata": { + "editable": true + }, + "source": [ + "Below we give a short example of how these cost function may be used\n", + "to obtain results if you wish to test them out on your own using\n", + "AutoGrad's automatics differentiation." + ] + }, + { + "cell_type": "code", + "execution_count": 27, + "id": "1479de5a", + "metadata": { + "collapsed": false, + "editable": true + }, + "outputs": [], + "source": [ + "from autograd import grad\n", + "\n", + "target = np.array([[1, 2, 3]]).T\n", + "a = np.array([[4, 5, 6]]).T\n", + "\n", + "cost_func = CostCrossEntropy\n", + "cost_func_derivative = grad(cost_func(target))\n", + "\n", + "valued_at_a = cost_func_derivative(a)\n", + "print(f\"Derivative of cost function {cost_func.__name__} valued at a:\\n{valued_at_a}\")" + ] + }, + { + "cell_type": "markdown", + "id": "d42e1ccf", + "metadata": { + "editable": true + }, + "source": [ + "### Activation functions\n", + "\n", + "Finally, before we look at the neural network, we will look at the\n", + "activation functions which can be specified between the hidden layers\n", + "and as the output function. Each function can be valued for any given\n", + "vector or matrix X, and can be differentiated via derivate()." + ] + }, + { + "cell_type": "code", + "execution_count": 28, + "id": "2c2a8d32", + "metadata": { + "collapsed": false, + "editable": true + }, + "outputs": [], + "source": [ + "import autograd.numpy as np\n", + "from autograd import elementwise_grad\n", + "\n", + "def identity(X):\n", + " return X\n", + "\n", + "\n", + "def sigmoid(X):\n", + " try:\n", + " return 1.0 / (1 + np.exp(-X))\n", + " except FloatingPointError:\n", + " return np.where(X > np.zeros(X.shape), np.ones(X.shape), np.zeros(X.shape))\n", + "\n", + "\n", + "def softmax(X):\n", + " X = X - np.max(X, axis=-1, keepdims=True)\n", + " delta = 10e-10\n", + " return np.exp(X) / (np.sum(np.exp(X), axis=-1, keepdims=True) + delta)\n", + "\n", + "\n", + "def RELU(X):\n", + " return np.where(X > np.zeros(X.shape), X, np.zeros(X.shape))\n", + "\n", + "\n", + "def LRELU(X):\n", + " delta = 10e-4\n", + " return np.where(X > np.zeros(X.shape), X, delta * X)\n", + "\n", + "\n", + "def derivate(func):\n", + " if func.__name__ == \"RELU\":\n", + "\n", + " def func(X):\n", + " return np.where(X > 0, 1, 0)\n", + "\n", + " return func\n", + "\n", + " elif func.__name__ == \"LRELU\":\n", + "\n", + " def func(X):\n", + " delta = 10e-4\n", + " return np.where(X > 0, 1, delta)\n", + "\n", + " return func\n", + "\n", + " else:\n", + " return elementwise_grad(func)" + ] + }, + { + "cell_type": "markdown", + "id": "65e7caab", + "metadata": { + "editable": true + }, + "source": [ + "Below follows a short demonstration of how to use an activation\n", + "function. The derivative of the activation function will be important\n", + "when calculating the output delta term during backpropagation. Note\n", + "that derivate() can also be used for cost functions for a more\n", + "generalized approach." + ] + }, + { + "cell_type": "code", + "execution_count": 29, + "id": "75e9399d", + "metadata": { + "collapsed": false, + "editable": true + }, + "outputs": [], + "source": [ + "z = np.array([[4, 5, 6]]).T\n", + "print(f\"Input to activation function:\\n{z}\")\n", + "\n", + "act_func = sigmoid\n", + "a = act_func(z)\n", + "print(f\"\\nOutput from {act_func.__name__} activation function:\\n{a}\")\n", + "\n", + "act_func_derivative = derivate(act_func)\n", + "valued_at_z = act_func_derivative(a)\n", + "print(f\"\\nDerivative of {act_func.__name__} activation function valued at z:\\n{valued_at_z}\")" + ] + }, + { + "cell_type": "markdown", + "id": "0b0b82e8", + "metadata": { + "editable": true + }, + "source": [ + "### The Neural Network\n", + "\n", + "Now that we have gotten a good understanding of the implementation of\n", + "some important components, we can take a look at an object oriented\n", + "implementation of a feed forward neural network. The feed forward\n", + "neural network has been implemented as a class named FFNN, which can\n", + "be initiated as a regressor or classifier dependant on the choice of\n", + "cost function. The FFNN can have any number of input nodes, hidden\n", + "layers with any amount of hidden nodes, and any amount of output nodes\n", + "meaning it can perform multiclass classification as well as binary\n", + "classification and regression problems. Although there is a lot of\n", + "code present, it makes for an easy to use and generalizeable interface\n", + "for creating many types of neural networks as will be demonstrated\n", + "below." + ] + }, + { + "cell_type": "code", + "execution_count": 30, + "id": "5351bfd6", + "metadata": { + "collapsed": false, + "editable": true + }, + "outputs": [], + "source": [ + "import math\n", + "import autograd.numpy as np\n", + "import sys\n", + "import warnings\n", + "from autograd import grad, elementwise_grad\n", + "from random import random, seed\n", + "from copy import deepcopy, copy\n", + "from typing import Tuple, Callable\n", + "from sklearn.utils import resample\n", + "\n", + "warnings.simplefilter(\"error\")\n", + "\n", + "\n", + "class FFNN:\n", + " \"\"\"\n", + " Description:\n", + " ------------\n", + " Feed Forward Neural Network with interface enabling flexible design of a\n", + " nerual networks architecture and the specification of activation function\n", + " in the hidden layers and output layer respectively. This model can be used\n", + " for both regression and classification problems, depending on the output function.\n", + "\n", + " Attributes:\n", + " ------------\n", + " I dimensions (tuple[int]): A list of positive integers, which specifies the\n", + " number of nodes in each of the networks layers. The first integer in the array\n", + " defines the number of nodes in the input layer, the second integer defines number\n", + " of nodes in the first hidden layer and so on until the last number, which\n", + " specifies the number of nodes in the output layer.\n", + " II hidden_func (Callable): The activation function for the hidden layers\n", + " III output_func (Callable): The activation function for the output layer\n", + " IV cost_func (Callable): Our cost function\n", + " V seed (int): Sets random seed, makes results reproducible\n", + " \"\"\"\n", + "\n", + " def __init__(\n", + " self,\n", + " dimensions: tuple[int],\n", + " hidden_func: Callable = sigmoid,\n", + " output_func: Callable = lambda x: x,\n", + " cost_func: Callable = CostOLS,\n", + " seed: int = None,\n", + " ):\n", + " self.dimensions = dimensions\n", + " self.hidden_func = hidden_func\n", + " self.output_func = output_func\n", + " self.cost_func = cost_func\n", + " self.seed = seed\n", + " self.weights = list()\n", + " self.schedulers_weight = list()\n", + " self.schedulers_bias = list()\n", + " self.a_matrices = list()\n", + " self.z_matrices = list()\n", + " self.classification = None\n", + "\n", + " self.reset_weights()\n", + " self._set_classification()\n", + "\n", + " def fit(\n", + " self,\n", + " X: np.ndarray,\n", + " t: np.ndarray,\n", + " scheduler: Scheduler,\n", + " batches: int = 1,\n", + " epochs: int = 100,\n", + " lam: float = 0,\n", + " X_val: np.ndarray = None,\n", + " t_val: np.ndarray = None,\n", + " ):\n", + " \"\"\"\n", + " Description:\n", + " ------------\n", + " This function performs the training the neural network by performing the feedforward and backpropagation\n", + " algorithm to update the networks weights.\n", + "\n", + " Parameters:\n", + " ------------\n", + " I X (np.ndarray) : training data\n", + " II t (np.ndarray) : target data\n", + " III scheduler (Scheduler) : specified scheduler (algorithm for optimization of gradient descent)\n", + " IV scheduler_args (list[int]) : list of all arguments necessary for scheduler\n", + "\n", + " Optional Parameters:\n", + " ------------\n", + " V batches (int) : number of batches the datasets are split into, default equal to 1\n", + " VI epochs (int) : number of iterations used to train the network, default equal to 100\n", + " VII lam (float) : regularization hyperparameter lambda\n", + " VIII X_val (np.ndarray) : validation set\n", + " IX t_val (np.ndarray) : validation target set\n", + "\n", + " Returns:\n", + " ------------\n", + " I scores (dict) : A dictionary containing the performance metrics of the model.\n", + " The number of the metrics depends on the parameters passed to the fit-function.\n", + "\n", + " \"\"\"\n", + "\n", + " # setup \n", + " if self.seed is not None:\n", + " np.random.seed(self.seed)\n", + "\n", + " val_set = False\n", + " if X_val is not None and t_val is not None:\n", + " val_set = True\n", + "\n", + " # creating arrays for score metrics\n", + " train_errors = np.empty(epochs)\n", + " train_errors.fill(np.nan)\n", + " val_errors = np.empty(epochs)\n", + " val_errors.fill(np.nan)\n", + "\n", + " train_accs = np.empty(epochs)\n", + " train_accs.fill(np.nan)\n", + " val_accs = np.empty(epochs)\n", + " val_accs.fill(np.nan)\n", + "\n", + " self.schedulers_weight = list()\n", + " self.schedulers_bias = list()\n", + "\n", + " batch_size = X.shape[0] // batches\n", + "\n", + " X, t = resample(X, t)\n", + "\n", + " # this function returns a function valued only at X\n", + " cost_function_train = self.cost_func(t)\n", + " if val_set:\n", + " cost_function_val = self.cost_func(t_val)\n", + "\n", + " # create schedulers for each weight matrix\n", + " for i in range(len(self.weights)):\n", + " self.schedulers_weight.append(copy(scheduler))\n", + " self.schedulers_bias.append(copy(scheduler))\n", + "\n", + " print(f\"{scheduler.__class__.__name__}: Eta={scheduler.eta}, Lambda={lam}\")\n", + "\n", + " try:\n", + " for e in range(epochs):\n", + " for i in range(batches):\n", + " # allows for minibatch gradient descent\n", + " if i == batches - 1:\n", + " # If the for loop has reached the last batch, take all thats left\n", + " X_batch = X[i * batch_size :, :]\n", + " t_batch = t[i * batch_size :, :]\n", + " else:\n", + " X_batch = X[i * batch_size : (i + 1) * batch_size, :]\n", + " t_batch = t[i * batch_size : (i + 1) * batch_size, :]\n", + "\n", + " self._feedforward(X_batch)\n", + " self._backpropagate(X_batch, t_batch, lam)\n", + "\n", + " # reset schedulers for each epoch (some schedulers pass in this call)\n", + " for scheduler in self.schedulers_weight:\n", + " scheduler.reset()\n", + "\n", + " for scheduler in self.schedulers_bias:\n", + " scheduler.reset()\n", + "\n", + " # computing performance metrics\n", + " pred_train = self.predict(X)\n", + " train_error = cost_function_train(pred_train)\n", + "\n", + " train_errors[e] = train_error\n", + " if val_set:\n", + " \n", + " pred_val = self.predict(X_val)\n", + " val_error = cost_function_val(pred_val)\n", + " val_errors[e] = val_error\n", + "\n", + " if self.classification:\n", + " train_acc = self._accuracy(self.predict(X), t)\n", + " train_accs[e] = train_acc\n", + " if val_set:\n", + " val_acc = self._accuracy(pred_val, t_val)\n", + " val_accs[e] = val_acc\n", + "\n", + " # printing progress bar\n", + " progression = e / epochs\n", + " print_length = self._progress_bar(\n", + " progression,\n", + " train_error=train_errors[e],\n", + " train_acc=train_accs[e],\n", + " val_error=val_errors[e],\n", + " val_acc=val_accs[e],\n", + " )\n", + " except KeyboardInterrupt:\n", + " # allows for stopping training at any point and seeing the result\n", + " pass\n", + "\n", + " # visualization of training progression (similiar to tensorflow progression bar)\n", + " sys.stdout.write(\"\\r\" + \" \" * print_length)\n", + " sys.stdout.flush()\n", + " self._progress_bar(\n", + " 1,\n", + " train_error=train_errors[e],\n", + " train_acc=train_accs[e],\n", + " val_error=val_errors[e],\n", + " val_acc=val_accs[e],\n", + " )\n", + " sys.stdout.write(\"\")\n", + "\n", + " # return performance metrics for the entire run\n", + " scores = dict()\n", + "\n", + " scores[\"train_errors\"] = train_errors\n", + "\n", + " if val_set:\n", + " scores[\"val_errors\"] = val_errors\n", + "\n", + " if self.classification:\n", + " scores[\"train_accs\"] = train_accs\n", + "\n", + " if val_set:\n", + " scores[\"val_accs\"] = val_accs\n", + "\n", + " return scores\n", + "\n", + " def predict(self, X: np.ndarray, *, threshold=0.5):\n", + " \"\"\"\n", + " Description:\n", + " ------------\n", + " Performs prediction after training of the network has been finished.\n", + "\n", + " Parameters:\n", + " ------------\n", + " I X (np.ndarray): The design matrix, with n rows of p features each\n", + "\n", + " Optional Parameters:\n", + " ------------\n", + " II threshold (float) : sets minimal value for a prediction to be predicted as the positive class\n", + " in classification problems\n", + "\n", + " Returns:\n", + " ------------\n", + " I z (np.ndarray): A prediction vector (row) for each row in our design matrix\n", + " This vector is thresholded if regression=False, meaning that classification results\n", + " in a vector of 1s and 0s, while regressions in an array of decimal numbers\n", + "\n", + " \"\"\"\n", + "\n", + " predict = self._feedforward(X)\n", + "\n", + " if self.classification:\n", + " return np.where(predict > threshold, 1, 0)\n", + " else:\n", + " return predict\n", + "\n", + " def reset_weights(self):\n", + " \"\"\"\n", + " Description:\n", + " ------------\n", + " Resets/Reinitializes the weights in order to train the network for a new problem.\n", + "\n", + " \"\"\"\n", + " if self.seed is not None:\n", + " np.random.seed(self.seed)\n", + "\n", + " self.weights = list()\n", + " for i in range(len(self.dimensions) - 1):\n", + " weight_array = np.random.randn(\n", + " self.dimensions[i] + 1, self.dimensions[i + 1]\n", + " )\n", + " weight_array[0, :] = np.random.randn(self.dimensions[i + 1]) * 0.01\n", + "\n", + " self.weights.append(weight_array)\n", + "\n", + " def _feedforward(self, X: np.ndarray):\n", + " \"\"\"\n", + " Description:\n", + " ------------\n", + " Calculates the activation of each layer starting at the input and ending at the output.\n", + " Each following activation is calculated from a weighted sum of each of the preceeding\n", + " activations (except in the case of the input layer).\n", + "\n", + " Parameters:\n", + " ------------\n", + " I X (np.ndarray): The design matrix, with n rows of p features each\n", + "\n", + " Returns:\n", + " ------------\n", + " I z (np.ndarray): A prediction vector (row) for each row in our design matrix\n", + " \"\"\"\n", + "\n", + " # reset matrices\n", + " self.a_matrices = list()\n", + " self.z_matrices = list()\n", + "\n", + " # if X is just a vector, make it into a matrix\n", + " if len(X.shape) == 1:\n", + " X = X.reshape((1, X.shape[0]))\n", + "\n", + " # Add a coloumn of zeros as the first coloumn of the design matrix, in order\n", + " # to add bias to our data\n", + " bias = np.ones((X.shape[0], 1)) * 0.01\n", + " X = np.hstack([bias, X])\n", + "\n", + " # a^0, the nodes in the input layer (one a^0 for each row in X - where the\n", + " # exponent indicates layer number).\n", + " a = X\n", + " self.a_matrices.append(a)\n", + " self.z_matrices.append(a)\n", + "\n", + " # The feed forward algorithm\n", + " for i in range(len(self.weights)):\n", + " if i < len(self.weights) - 1:\n", + " z = a @ self.weights[i]\n", + " self.z_matrices.append(z)\n", + " a = self.hidden_func(z)\n", + " # bias column again added to the data here\n", + " bias = np.ones((a.shape[0], 1)) * 0.01\n", + " a = np.hstack([bias, a])\n", + " self.a_matrices.append(a)\n", + " else:\n", + " try:\n", + " # a^L, the nodes in our output layers\n", + " z = a @ self.weights[i]\n", + " a = self.output_func(z)\n", + " self.a_matrices.append(a)\n", + " self.z_matrices.append(z)\n", + " except Exception as OverflowError:\n", + " print(\n", + " \"OverflowError in fit() in FFNN\\nHOW TO DEBUG ERROR: Consider lowering your learning rate or scheduler specific parameters such as momentum, or check if your input values need scaling\"\n", + " )\n", + "\n", + " # this will be a^L\n", + " return a\n", + "\n", + " def _backpropagate(self, X, t, lam):\n", + " \"\"\"\n", + " Description:\n", + " ------------\n", + " Performs the backpropagation algorithm. In other words, this method\n", + " calculates the gradient of all the layers starting at the\n", + " output layer, and moving from right to left accumulates the gradient until\n", + " the input layer is reached. Each layers respective weights are updated while\n", + " the algorithm propagates backwards from the output layer (auto-differentation in reverse mode).\n", + "\n", + " Parameters:\n", + " ------------\n", + " I X (np.ndarray): The design matrix, with n rows of p features each.\n", + " II t (np.ndarray): The target vector, with n rows of p targets.\n", + " III lam (float32): regularization parameter used to punish the weights in case of overfitting\n", + "\n", + " Returns:\n", + " ------------\n", + " No return value.\n", + "\n", + " \"\"\"\n", + " out_derivative = derivate(self.output_func)\n", + " hidden_derivative = derivate(self.hidden_func)\n", + "\n", + " for i in range(len(self.weights) - 1, -1, -1):\n", + " # delta terms for output\n", + " if i == len(self.weights) - 1:\n", + " # for multi-class classification\n", + " if (\n", + " self.output_func.__name__ == \"softmax\"\n", + " ):\n", + " delta_matrix = self.a_matrices[i + 1] - t\n", + " # for single class classification\n", + " else:\n", + " cost_func_derivative = grad(self.cost_func(t))\n", + " delta_matrix = out_derivative(\n", + " self.z_matrices[i + 1]\n", + " ) * cost_func_derivative(self.a_matrices[i + 1])\n", + "\n", + " # delta terms for hidden layer\n", + " else:\n", + " delta_matrix = (\n", + " self.weights[i + 1][1:, :] @ delta_matrix.T\n", + " ).T * hidden_derivative(self.z_matrices[i + 1])\n", + "\n", + " # calculate gradient\n", + " gradient_weights = self.a_matrices[i][:, 1:].T @ delta_matrix\n", + " gradient_bias = np.sum(delta_matrix, axis=0).reshape(\n", + " 1, delta_matrix.shape[1]\n", + " )\n", + "\n", + " # regularization term\n", + " gradient_weights += self.weights[i][1:, :] * lam\n", + "\n", + " # use scheduler\n", + " update_matrix = np.vstack(\n", + " [\n", + " self.schedulers_bias[i].update_change(gradient_bias),\n", + " self.schedulers_weight[i].update_change(gradient_weights),\n", + " ]\n", + " )\n", + "\n", + " # update weights and bias\n", + " self.weights[i] -= update_matrix\n", + "\n", + " def _accuracy(self, prediction: np.ndarray, target: np.ndarray):\n", + " \"\"\"\n", + " Description:\n", + " ------------\n", + " Calculates accuracy of given prediction to target\n", + "\n", + " Parameters:\n", + " ------------\n", + " I prediction (np.ndarray): vector of predicitons output network\n", + " (1s and 0s in case of classification, and real numbers in case of regression)\n", + " II target (np.ndarray): vector of true values (What the network ideally should predict)\n", + "\n", + " Returns:\n", + " ------------\n", + " A floating point number representing the percentage of correctly classified instances.\n", + " \"\"\"\n", + " assert prediction.size == target.size\n", + " return np.average((target == prediction))\n", + " def _set_classification(self):\n", + " \"\"\"\n", + " Description:\n", + " ------------\n", + " Decides if FFNN acts as classifier (True) og regressor (False),\n", + " sets self.classification during init()\n", + " \"\"\"\n", + " self.classification = False\n", + " if (\n", + " self.cost_func.__name__ == \"CostLogReg\"\n", + " or self.cost_func.__name__ == \"CostCrossEntropy\"\n", + " ):\n", + " self.classification = True\n", + "\n", + " def _progress_bar(self, progression, **kwargs):\n", + " \"\"\"\n", + " Description:\n", + " ------------\n", + " Displays progress of training\n", + " \"\"\"\n", + " print_length = 40\n", + " num_equals = int(progression * print_length)\n", + " num_not = print_length - num_equals\n", + " arrow = \">\" if num_equals > 0 else \"\"\n", + " bar = \"[\" + \"=\" * (num_equals - 1) + arrow + \"-\" * num_not + \"]\"\n", + " perc_print = self._format(progression * 100, decimals=5)\n", + " line = f\" {bar} {perc_print}% \"\n", + "\n", + " for key in kwargs:\n", + " if not np.isnan(kwargs[key]):\n", + " value = self._format(kwargs[key], decimals=4)\n", + " line += f\"| {key}: {value} \"\n", + " sys.stdout.write(\"\\r\" + line)\n", + " sys.stdout.flush()\n", + " return len(line)\n", + "\n", + " def _format(self, value, decimals=4):\n", + " \"\"\"\n", + " Description:\n", + " ------------\n", + " Formats decimal numbers for progress bar\n", + " \"\"\"\n", + " if value > 0:\n", + " v = value\n", + " elif value < 0:\n", + " v = -10 * value\n", + " else:\n", + " v = 1\n", + " n = 1 + math.floor(math.log10(v))\n", + " if n >= decimals - 1:\n", + " return str(round(value))\n", + " return f\"{value:.{decimals-n-1}f}\"" + ] + }, + { + "cell_type": "markdown", + "id": "2d63ec5b", + "metadata": { + "editable": true + }, + "source": [ + "Before we make a model, we will quickly generate a dataset we can use\n", + "for our linear regression problem as shown below" + ] + }, + { + "cell_type": "code", + "execution_count": 31, + "id": "85a6b185", + "metadata": { + "collapsed": false, + "editable": true + }, + "outputs": [], + "source": [ + "import autograd.numpy as np\n", + "from sklearn.model_selection import train_test_split\n", + "\n", + "def SkrankeFunction(x, y):\n", + " return np.ravel(0 + 1*x + 2*y + 3*x**2 + 4*x*y + 5*y**2)\n", + "\n", + "def create_X(x, y, n):\n", + " if len(x.shape) > 1:\n", + " x = np.ravel(x)\n", + " y = np.ravel(y)\n", + "\n", + " N = len(x)\n", + " l = int((n + 1) * (n + 2) / 2) # Number of elements in beta\n", + " X = np.ones((N, l))\n", + "\n", + " for i in range(1, n + 1):\n", + " q = int((i) * (i + 1) / 2)\n", + " for k in range(i + 1):\n", + " X[:, q + k] = (x ** (i - k)) * (y**k)\n", + "\n", + " return X\n", + "\n", + "step=0.5\n", + "x = np.arange(0, 1, step)\n", + "y = np.arange(0, 1, step)\n", + "x, y = np.meshgrid(x, y)\n", + "target = SkrankeFunction(x, y)\n", + "target = target.reshape(target.shape[0], 1)\n", + "\n", + "poly_degree=3\n", + "X = create_X(x, y, poly_degree)\n", + "\n", + "X_train, X_test, t_train, t_test = train_test_split(X, target)" + ] + }, + { + "cell_type": "markdown", + "id": "f8842209", + "metadata": { + "editable": true + }, + "source": [ + "Now that we have our dataset ready for the regression, we can create\n", + "our regressor. Note that with the seed parameter, we can make sure our\n", + "results stay the same every time we run the neural network. For\n", + "inititialization, we simply specify the dimensions (we wish the amount\n", + "of input nodes to be equal to the datapoints, and the output to\n", + "predict one value)." + ] + }, + { + "cell_type": "code", + "execution_count": 32, + "id": "b7b14d8f", + "metadata": { + "collapsed": false, + "editable": true + }, + "outputs": [], + "source": [ + "input_nodes = X_train.shape[1]\n", + "output_nodes = 1\n", + "\n", + "linear_regression = FFNN((input_nodes, output_nodes), output_func=identity, cost_func=CostOLS, seed=2023)" + ] + }, + { + "cell_type": "markdown", + "id": "3e35cc67", + "metadata": { + "editable": true + }, + "source": [ + "We then fit our model with our training data using the scheduler of our choice." + ] + }, + { + "cell_type": "code", + "execution_count": 33, + "id": "c2b3e7a3", + "metadata": { + "collapsed": false, + "editable": true + }, + "outputs": [], + "source": [ + "linear_regression.reset_weights() # reset weights such that previous runs or reruns don't affect the weights\n", + "\n", + "scheduler = Constant(eta=1e-3)\n", + "scores = linear_regression.fit(X_train, t_train, scheduler)" + ] + }, + { + "cell_type": "markdown", + "id": "9d0eb8b2", + "metadata": { + "editable": true + }, + "source": [ + "Due to the progress bar we can see the MSE (train_error) throughout\n", + "the FFNN's training. Note that the fit() function has some optional\n", + "parameters with defualt arguments. For example, the regularization\n", + "hyperparameter can be left ignored if not needed, and equally the FFNN\n", + "will by default run for 100 epochs. These can easily be changed, such\n", + "as for example:" + ] + }, + { + "cell_type": "code", + "execution_count": 34, + "id": "19fd68fe", + "metadata": { + "collapsed": false, + "editable": true + }, + "outputs": [], + "source": [ + "linear_regression.reset_weights() # reset weights such that previous runs or reruns don't affect the weights\n", + "\n", + "scores = linear_regression.fit(X_train, t_train, scheduler, lam=1e-4, epochs=1000)" + ] + }, + { + "cell_type": "markdown", + "id": "ecbc4c88", + "metadata": { + "editable": true + }, + "source": [ + "We see that given more epochs to train on, the regressor reaches a lower MSE.\n", + "\n", + "Let us then switch to a binary classification. We use a binary\n", + "classification dataset, and follow a similar setup to the regression\n", + "case." + ] + }, + { + "cell_type": "code", + "execution_count": 35, + "id": "c62f0877", + "metadata": { + "collapsed": false, + "editable": true + }, + "outputs": [], + "source": [ + "from sklearn.datasets import load_breast_cancer\n", + "from sklearn.preprocessing import MinMaxScaler\n", + "\n", + "wisconsin = load_breast_cancer()\n", + "X = wisconsin.data\n", + "target = wisconsin.target\n", + "target = target.reshape(target.shape[0], 1)\n", + "\n", + "X_train, X_val, t_train, t_val = train_test_split(X, target)\n", + "\n", + "scaler = MinMaxScaler()\n", + "scaler.fit(X_train)\n", + "X_train = scaler.transform(X_train)\n", + "X_val = scaler.transform(X_val)" + ] + }, + { + "cell_type": "code", + "execution_count": 36, + "id": "5b955b7a", + "metadata": { + "collapsed": false, + "editable": true + }, + "outputs": [], + "source": [ + "input_nodes = X_train.shape[1]\n", + "output_nodes = 1\n", + "\n", + "logistic_regression = FFNN((input_nodes, output_nodes), output_func=sigmoid, cost_func=CostLogReg, seed=2023)" + ] + }, + { + "cell_type": "markdown", + "id": "e4c3e9cc", + "metadata": { + "editable": true + }, + "source": [ + "We will now make use of our validation data by passing it into our fit function as a keyword argument" + ] + }, + { + "cell_type": "code", + "execution_count": 37, + "id": "4e994594", + "metadata": { + "collapsed": false, + "editable": true + }, + "outputs": [], + "source": [ + "logistic_regression.reset_weights() # reset weights such that previous runs or reruns don't affect the weights\n", + "\n", + "scheduler = Adam(eta=1e-3, rho=0.9, rho2=0.999)\n", + "scores = logistic_regression.fit(X_train, t_train, scheduler, epochs=1000, X_val=X_val, t_val=t_val)" + ] + }, + { + "cell_type": "markdown", + "id": "690debd0", + "metadata": { + "editable": true + }, + "source": [ + "Finally, we will create a neural network with 2 hidden layers with activation functions." + ] + }, + { + "cell_type": "code", + "execution_count": 38, + "id": "ea04ae66", + "metadata": { + "collapsed": false, + "editable": true + }, + "outputs": [], + "source": [ + "input_nodes = X_train.shape[1]\n", + "hidden_nodes1 = 100\n", + "hidden_nodes2 = 30\n", + "output_nodes = 1\n", + "\n", + "dims = (input_nodes, hidden_nodes1, hidden_nodes2, output_nodes)\n", + "\n", + "neural_network = FFNN(dims, hidden_func=RELU, output_func=sigmoid, cost_func=CostLogReg, seed=2023)" + ] + }, + { + "cell_type": "code", + "execution_count": 39, + "id": "cd892db0", + "metadata": { + "collapsed": false, + "editable": true + }, + "outputs": [], + "source": [ + "neural_network.reset_weights() # reset weights such that previous runs or reruns don't affect the weights\n", + "\n", + "scheduler = Adam(eta=1e-4, rho=0.9, rho2=0.999)\n", + "scores = neural_network.fit(X_train, t_train, scheduler, epochs=1000, X_val=X_val, t_val=t_val)" + ] + }, + { + "cell_type": "markdown", + "id": "c85e6aa4", + "metadata": { + "editable": true + }, + "source": [ + "### Multiclass classification\n", + "\n", + "Finally, we will demonstrate the use case of multiclass classification\n", + "using our FFNN with the famous MNIST dataset, which contain images of\n", + "digits between the range of 0 to 9." + ] + }, + { + "cell_type": "code", + "execution_count": 40, + "id": "1dc44b4d", + "metadata": { + "collapsed": false, + "editable": true + }, + "outputs": [], + "source": [ + "from sklearn.datasets import load_digits\n", + "\n", + "def onehot(target: np.ndarray):\n", + " onehot = np.zeros((target.size, target.max() + 1))\n", + " onehot[np.arange(target.size), target] = 1\n", + " return onehot\n", + "\n", + "digits = load_digits()\n", + "\n", + "X = digits.data\n", + "target = digits.target\n", + "target = onehot(target)\n", + "\n", + "input_nodes = 64\n", + "hidden_nodes1 = 100\n", + "hidden_nodes2 = 30\n", + "output_nodes = 10\n", + "\n", + "dims = (input_nodes, hidden_nodes1, hidden_nodes2, output_nodes)\n", + "\n", + "multiclass = FFNN(dims, hidden_func=LRELU, output_func=softmax, cost_func=CostCrossEntropy)\n", + "\n", + "multiclass.reset_weights() # reset weights such that previous runs or reruns don't affect the weights\n", + "\n", + "scheduler = Adam(eta=1e-4, rho=0.9, rho2=0.999)\n", + "scores = multiclass.fit(X, target, scheduler, epochs=1000)" + ] + }, + { + "cell_type": "markdown", + "id": "785a6539", + "metadata": { + "editable": true + }, + "source": [ + "## Testing the XOR gate and other gates\n", + "\n", + "Let us now use our code to test the XOR gate." + ] + }, + { + "cell_type": "code", + "execution_count": 41, + "id": "f2137088", + "metadata": { + "collapsed": false, + "editable": true + }, + "outputs": [], + "source": [ + "X = np.array([ [0, 0], [0, 1], [1, 0],[1, 1]],dtype=np.float64)\n", + "\n", + "# The XOR gate\n", + "yXOR = np.array( [[ 0], [1] ,[1], [0]])\n", + "\n", + "input_nodes = X.shape[1]\n", + "output_nodes = 1\n", + "\n", + "logistic_regression = FFNN((input_nodes, output_nodes), output_func=sigmoid, cost_func=CostLogReg, seed=2023)\n", + "logistic_regression.reset_weights() # reset weights such that previous runs or reruns don't affect the weights\n", + "scheduler = Adam(eta=1e-1, rho=0.9, rho2=0.999)\n", + "scores = logistic_regression.fit(X, yXOR, scheduler, epochs=1000)" + ] + }, + { + "cell_type": "markdown", + "id": "6caa78c7", + "metadata": { + "editable": true + }, + "source": [ + "Not bad, but the results depend strongly on the learning reate. Try different learning rates." + ] + } + ], + "metadata": {}, + "nbformat": 4, + "nbformat_minor": 5 +} diff --git a/doc/LectureNotes/_build/html/chapter1.html b/doc/LectureNotes/_build/html/chapter1.html index f265a0abe..97d9d2466 100644 --- a/doc/LectureNotes/_build/html/chapter1.html +++ b/doc/LectureNotes/_build/html/chapter1.html @@ -246,6 +246,7 @@ +
  • Week 42 Constructing a Neural Network code with examples
  • Exercises week 42
    • @@ -260,6 +261,7 @@

    Projects

    diff --git a/doc/LectureNotes/_build/html/chapter10.html b/doc/LectureNotes/_build/html/chapter10.html index 452834030..53807ad97 100644 --- a/doc/LectureNotes/_build/html/chapter10.html +++ b/doc/LectureNotes/_build/html/chapter10.html @@ -246,6 +246,7 @@ +
  • Week 42 Constructing a Neural Network code with examples
  • Exercises week 42
    • @@ -260,6 +261,7 @@

    Projects

    diff --git a/doc/LectureNotes/_build/html/chapter11.html b/doc/LectureNotes/_build/html/chapter11.html index 937fce8b1..69a64ae3d 100644 --- a/doc/LectureNotes/_build/html/chapter11.html +++ b/doc/LectureNotes/_build/html/chapter11.html @@ -246,6 +246,7 @@ +
  • Week 42 Constructing a Neural Network code with examples
  • Exercises week 42
    • @@ -260,6 +261,7 @@

    Projects

    diff --git a/doc/LectureNotes/_build/html/chapter12.html b/doc/LectureNotes/_build/html/chapter12.html index ebb87ec13..41ff3d453 100644 --- a/doc/LectureNotes/_build/html/chapter12.html +++ b/doc/LectureNotes/_build/html/chapter12.html @@ -246,6 +246,7 @@ +
  • Week 42 Constructing a Neural Network code with examples
  • Exercises week 42
    • @@ -260,6 +261,7 @@

    Projects

    diff --git a/doc/LectureNotes/_build/html/chapter13.html b/doc/LectureNotes/_build/html/chapter13.html index 2a2b52c52..8e8b9c05d 100644 --- a/doc/LectureNotes/_build/html/chapter13.html +++ b/doc/LectureNotes/_build/html/chapter13.html @@ -246,6 +246,7 @@ +
  • Week 42 Constructing a Neural Network code with examples
  • Exercises week 42
    • @@ -260,6 +261,7 @@

    Projects

    diff --git a/doc/LectureNotes/_build/html/chapter2.html b/doc/LectureNotes/_build/html/chapter2.html index 091cbab60..e60ca7d8a 100644 --- a/doc/LectureNotes/_build/html/chapter2.html +++ b/doc/LectureNotes/_build/html/chapter2.html @@ -246,6 +246,7 @@ +
  • Week 42 Constructing a Neural Network code with examples
  • Exercises week 42
    • @@ -260,6 +261,7 @@

    Projects

    diff --git a/doc/LectureNotes/_build/html/chapter3.html b/doc/LectureNotes/_build/html/chapter3.html index 04f78027c..e11cf5a8e 100644 --- a/doc/LectureNotes/_build/html/chapter3.html +++ b/doc/LectureNotes/_build/html/chapter3.html @@ -246,6 +246,7 @@ +
  • Week 42 Constructing a Neural Network code with examples
  • Exercises week 42
    • @@ -260,6 +261,7 @@

    Projects

    diff --git a/doc/LectureNotes/_build/html/chapter4.html b/doc/LectureNotes/_build/html/chapter4.html index c0853e479..d0286adc3 100644 --- a/doc/LectureNotes/_build/html/chapter4.html +++ b/doc/LectureNotes/_build/html/chapter4.html @@ -246,6 +246,7 @@ +
  • Week 42 Constructing a Neural Network code with examples
  • Exercises week 42
    • @@ -260,6 +261,7 @@

    Projects

    diff --git a/doc/LectureNotes/_build/html/chapter5.html b/doc/LectureNotes/_build/html/chapter5.html index 09c8b66fa..15f2f236c 100644 --- a/doc/LectureNotes/_build/html/chapter5.html +++ b/doc/LectureNotes/_build/html/chapter5.html @@ -246,6 +246,7 @@ +
  • Week 42 Constructing a Neural Network code with examples
  • Exercises week 42
    • @@ -260,6 +261,7 @@

    Projects

    diff --git a/doc/LectureNotes/_build/html/chapter6.html b/doc/LectureNotes/_build/html/chapter6.html index 7114fe17f..574aced21 100644 --- a/doc/LectureNotes/_build/html/chapter6.html +++ b/doc/LectureNotes/_build/html/chapter6.html @@ -246,6 +246,7 @@ +
  • Week 42 Constructing a Neural Network code with examples
  • Exercises week 42
    • @@ -260,6 +261,7 @@

    Projects

    diff --git a/doc/LectureNotes/_build/html/chapter7.html b/doc/LectureNotes/_build/html/chapter7.html index bc3798908..838b87b0d 100644 --- a/doc/LectureNotes/_build/html/chapter7.html +++ b/doc/LectureNotes/_build/html/chapter7.html @@ -246,6 +246,7 @@ +
  • Week 42 Constructing a Neural Network code with examples
  • Exercises week 42
    • @@ -260,6 +261,7 @@

    Projects

    diff --git a/doc/LectureNotes/_build/html/chapter8.html b/doc/LectureNotes/_build/html/chapter8.html index 3933ee295..e2b080557 100644 --- a/doc/LectureNotes/_build/html/chapter8.html +++ b/doc/LectureNotes/_build/html/chapter8.html @@ -246,6 +246,7 @@ +
  • Week 42 Constructing a Neural Network code with examples
  • Exercises week 42
    • @@ -260,6 +261,7 @@

    Projects

    diff --git a/doc/LectureNotes/_build/html/chapter9.html b/doc/LectureNotes/_build/html/chapter9.html index 1a08651be..bbedc0976 100644 --- a/doc/LectureNotes/_build/html/chapter9.html +++ b/doc/LectureNotes/_build/html/chapter9.html @@ -246,6 +246,7 @@ +
  • Week 42 Constructing a Neural Network code with examples
  • Exercises week 42
    • @@ -260,6 +261,7 @@

    Projects

    diff --git a/doc/LectureNotes/_build/html/chapteroptimization.html b/doc/LectureNotes/_build/html/chapteroptimization.html index 88af96a20..be9bab5ae 100644 --- a/doc/LectureNotes/_build/html/chapteroptimization.html +++ b/doc/LectureNotes/_build/html/chapteroptimization.html @@ -246,6 +246,7 @@ +
  • Week 42 Constructing a Neural Network code with examples
  • Exercises week 42
    • @@ -260,6 +261,7 @@

    Projects

    diff --git a/doc/LectureNotes/_build/html/clustering.html b/doc/LectureNotes/_build/html/clustering.html index 37253966f..a97db82ce 100644 --- a/doc/LectureNotes/_build/html/clustering.html +++ b/doc/LectureNotes/_build/html/clustering.html @@ -246,6 +246,7 @@ +
  • Week 42 Constructing a Neural Network code with examples
  • Exercises week 42
    • @@ -260,6 +261,7 @@

    Projects

    diff --git a/doc/LectureNotes/_build/html/exercisesweek34.html b/doc/LectureNotes/_build/html/exercisesweek34.html index bf086da9d..f7510cb77 100644 --- a/doc/LectureNotes/_build/html/exercisesweek34.html +++ b/doc/LectureNotes/_build/html/exercisesweek34.html @@ -244,6 +244,7 @@ +
  • Week 42 Constructing a Neural Network code with examples
  • Exercises week 42
    • @@ -258,6 +259,7 @@

    Projects

    diff --git a/doc/LectureNotes/_build/html/exercisesweek35.html b/doc/LectureNotes/_build/html/exercisesweek35.html index 6b5a6faab..e97a42850 100644 --- a/doc/LectureNotes/_build/html/exercisesweek35.html +++ b/doc/LectureNotes/_build/html/exercisesweek35.html @@ -246,6 +246,7 @@ +
  • Week 42 Constructing a Neural Network code with examples
  • Exercises week 42
    • @@ -260,6 +261,7 @@

    Projects

    diff --git a/doc/LectureNotes/_build/html/exercisesweek36.html b/doc/LectureNotes/_build/html/exercisesweek36.html index b31027538..525d1d80e 100644 --- a/doc/LectureNotes/_build/html/exercisesweek36.html +++ b/doc/LectureNotes/_build/html/exercisesweek36.html @@ -246,6 +246,7 @@ +
  • Week 42 Constructing a Neural Network code with examples
  • Exercises week 42
    • @@ -260,6 +261,7 @@

    Projects

    diff --git a/doc/LectureNotes/_build/html/exercisesweek37.html b/doc/LectureNotes/_build/html/exercisesweek37.html index b7c5dbcb2..883cc18cb 100644 --- a/doc/LectureNotes/_build/html/exercisesweek37.html +++ b/doc/LectureNotes/_build/html/exercisesweek37.html @@ -246,6 +246,7 @@ +
  • Week 42 Constructing a Neural Network code with examples
  • Exercises week 42
    • @@ -260,6 +261,7 @@

    Projects

    diff --git a/doc/LectureNotes/_build/html/exercisesweek38.html b/doc/LectureNotes/_build/html/exercisesweek38.html index ff082eb39..595ca59a0 100644 --- a/doc/LectureNotes/_build/html/exercisesweek38.html +++ b/doc/LectureNotes/_build/html/exercisesweek38.html @@ -246,6 +246,7 @@ +
  • Week 42 Constructing a Neural Network code with examples
  • Exercises week 42
    • @@ -260,6 +261,7 @@

    Projects

    diff --git a/doc/LectureNotes/_build/html/exercisesweek39.html b/doc/LectureNotes/_build/html/exercisesweek39.html index 68917699b..db14cd186 100644 --- a/doc/LectureNotes/_build/html/exercisesweek39.html +++ b/doc/LectureNotes/_build/html/exercisesweek39.html @@ -244,6 +244,7 @@ +
  • Week 42 Constructing a Neural Network code with examples
  • Exercises week 42
    • @@ -258,6 +259,7 @@

    Projects

    diff --git a/doc/LectureNotes/_build/html/exercisesweek41.html b/doc/LectureNotes/_build/html/exercisesweek41.html index a40cd4c0b..936c9478a 100644 --- a/doc/LectureNotes/_build/html/exercisesweek41.html +++ b/doc/LectureNotes/_build/html/exercisesweek41.html @@ -62,7 +62,7 @@ - + @@ -246,6 +246,7 @@ +
  • Week 42 Constructing a Neural Network code with examples
  • Exercises week 42
    • @@ -260,6 +261,7 @@

    Projects

    @@ -875,11 +877,11 @@ doconce format html exercisesweek41.do.txt -->

    next

    -

    Exercises week 42

    +

    Week 42 Constructing a Neural Network code with examples

     diff --git a/doc/LectureNotes/_build/html/exercisesweek42.html b/doc/LectureNotes/_build/html/exercisesweek42.html index f3861a2ce..41c2b8ac6 100644 --- a/doc/LectureNotes/_build/html/exercisesweek42.html +++ b/doc/LectureNotes/_build/html/exercisesweek42.html @@ -63,7 +63,7 @@ - + @@ -246,6 +246,7 @@ +
  • Week 42 Constructing a Neural Network code with examples
  • Exercises week 42
    • @@ -260,6 +261,7 @@

    Projects

    @@ -921,12 +923,12 @@ document.write(`

    previous

    -

    Exercises week 41

    +

    Week 42 Constructing a Neural Network code with examples

     Week 42 Constructing a Neural Network code with examples
  • Exercises week 42
    • @@ -257,6 +258,7 @@

    Projects

    diff --git a/doc/LectureNotes/_build/html/intro.html b/doc/LectureNotes/_build/html/intro.html index bb53943cf..2dbadfcc7 100644 --- a/doc/LectureNotes/_build/html/intro.html +++ b/doc/LectureNotes/_build/html/intro.html @@ -247,6 +247,7 @@ +
  • Week 42 Constructing a Neural Network code with examples
  • Exercises week 42
    • @@ -261,6 +262,7 @@

    Projects

    diff --git a/doc/LectureNotes/_build/html/linalg.html b/doc/LectureNotes/_build/html/linalg.html index 739caafcf..3370e0f90 100644 --- a/doc/LectureNotes/_build/html/linalg.html +++ b/doc/LectureNotes/_build/html/linalg.html @@ -246,6 +246,7 @@ +
  • Week 42 Constructing a Neural Network code with examples
  • Exercises week 42
    • @@ -260,6 +261,7 @@

    Projects

    diff --git a/doc/LectureNotes/_build/html/objects.inv b/doc/LectureNotes/_build/html/objects.inv index 86927a493..926574896 100644 Binary files a/doc/LectureNotes/_build/html/objects.inv and b/doc/LectureNotes/_build/html/objects.inv differ diff --git a/doc/LectureNotes/_build/html/project1.html b/doc/LectureNotes/_build/html/project1.html index 104430406..f6468fcdc 100644 --- a/doc/LectureNotes/_build/html/project1.html +++ b/doc/LectureNotes/_build/html/project1.html @@ -62,6 +62,7 @@ + @@ -245,6 +246,7 @@ +
  • Week 42 Constructing a Neural Network code with examples
  • Exercises week 42
    • @@ -259,6 +261,7 @@

    Projects

    @@ -813,6 +816,15 @@ of code developers and contributors keeps increasing.

    Exercises week 42

    + +
    +

    next

    +

    Project 2 on Machine Learning, deadline November 10 (Midnight)

    +
    + +     diff --git a/doc/LectureNotes/_build/html/project2.html b/doc/LectureNotes/_build/html/project2.html new file mode 100644 index 000000000..658681044 --- /dev/null +++ b/doc/LectureNotes/_build/html/project2.html @@ -0,0 +1,867 @@ + + + + + + + + + + + Project 2 on Machine Learning, deadline November 10 (Midnight) — Applied Data Analysis and Machine Learning + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
    + + + + + + + + + + +
    +
    +
    +
    + + + Ctrl+K +
    +
    + +
    + +
    + + +
    +
    + + +
    +
    + + + +
    + + + +
    + + + + +
    + +
    +
    + + + + + +
    +
    + +
    +
    +
    + + +
    +
    + + + + +
    + +
    + + + +
    + +
    +
    + +
    +
    + +
    + +
    + +
    + + +
    + +
    + +
    + + + + + +
    + + +
    + + + + + + + + + + + + + +
    + +
    + +
    +
    + + + +
    +

    Project 2 on Machine Learning, deadline November 10 (Midnight)

    + +
    + +
    +
    + + + + +
    + + +
    +

    Project 2 on Machine Learning, deadline November 10 (Midnight)#

    +

    Data Analysis and Machine Learning FYS-STK3155/FYS4155, University of Oslo, Norway

    +

    Date: October 14, 2025

    +
    +

    Deliverables#

    +

    First, join a group in canvas with your group partners. Pick an avaliable group for Project 2 in the People page.

    +

    In canvas, deliver as a group and include:

    +
      +

    • A PDF of your report which follows the guidelines covered below and in the week 39 exercises. Additional requirements include:

      +
        +

      • It should be around 5000 words, use the word counter in Overleaf for this. This often corresponds to 10-12 pages. References and appendices are excluded from the word count

        • +

      • It should include around 10-15 figures. You can include more figures in appendices and/or as supplemental material in your repository.

        • +
      +
      • +

    • A comment linking to your github repository (or folder in one of your github repositories) for this project. The repository must include

      • +
    +

    A PDF file of the report

    +
      +

    • A folder named Code, where you put python files for your functions and notebooks for reproducing your results. Remember to use a seed for generating random data and for train-test splits when generating final results.

      • +

    • A README file with the name of the group members

      • +

    • a short description of the project

      • +

    • a description of how to install the required packages to run your code from a requirements.txt file or similar (such as a plain text description) names and descriptions of the various notebooks in the Code folder and the results they produce

      • +
    +
    +

    Preamble: Note on writing reports, using reference material, AI and other tools#

    +

    We want you to answer the three different projects by handing in +reports written like a standard scientific/technical report. The links +at +CompPhysics/MachineLearning +contain more information. There you can find examples of previous +reports, the projects themselves, how we grade reports etc. How to +write reports will also be discussed during the various lab +sessions. Please do ask us if you are in doubt.

    +

    When using codes and material from other sources, you should refer to +these in the bibliography of your report, indicating wherefrom you for +example got the code, whether this is from the lecture notes, +softwares like Scikit-Learn, TensorFlow, PyTorch or other +sources. These sources should always be cited correctly. How to cite +some of the libraries is often indicated from their corresponding +GitHub sites or websites, see for example how to cite Scikit-Learn at +https://scikit-learn.org/dev/about.html.

    +

    We enocurage you to use tools like ChatGPT or similar in writing the +report. If you use for example ChatGPT, please do cite it properly and +include (if possible) your questions and answers as an addition to the +report. This can be uploaded to for example your website, +GitHub/GitLab or similar as supplemental material.

    +

    If you would like to study other data sets, feel free to propose other +sets. What we have proposed here are mere suggestions from our +side. If you opt for another data set, consider using a set which has +been studied in the scientific literature. This makes it easier for +you to compare and analyze your results. Comparing with existing +results from the scientific literature is also an essential element of +the scientific discussion. The University of California at Irvine with +its Machine Learning repository at +https://archive.ics.uci.edu/ml/index.php is an excellent site to look +up for examples and inspiration. Kaggle.com is an equally interesting +site. Feel free to explore these sites.

    +
    +
    +
    +

    Classification and Regression, writing our own neural network code#

    +

    The main aim of this project is to study both classification and +regression problems by developing our own +feed-forward neural network (FFNN) code. The exercises from week 41 and 42 (see https://compphysics.github.io/MachineLearning/doc/LectureNotes/_build/html/exercisesweek41.html and https://compphysics.github.io/MachineLearning/doc/LectureNotes/_build/html/exercisesweek42.html) as well as the lecture material from the same weeks (see https://compphysics.github.io/MachineLearning/doc/LectureNotes/_build/html/week41.html and https://compphysics.github.io/MachineLearning/doc/LectureNotes/_build/html/week42.html) should contain enough information for you to get started with writing your own code.

    +

    We will also reuse our codes on gradient descent methods from project 1.

    +

    The data sets that we propose here are (the default sets)

    +
      +

    • Regression (fitting a continuous function). In this part you will need to bring back your results from project 1 and compare these with what you get from your Neural Network code to be developed here. The data sets could be

      +
        +

      • The simple one-dimensional function Runge function from project 1, that is \(f(x) = \frac{1}{1+25x^2}\). We recommend using a simpler function when developing your neural network code for regression problems. Feel however free to discuss and study other functions, such as the the two-dimensional Runge function \(f(x,y)=\left[(10x - 5)^2 + (10y - 5)^2 + 1 \right]^{-1}\), or even more complicated two-dimensional functions (see the supplementary material of https://www.nature.com/articles/s41467-025-61362-4 for an extensive list of two-dimensional functions).

        • +
      +
      • +

    • Classification.

      • +

    • We will consider a multiclass classification problem given by the full MNIST data set. The full data set is at https://www.kaggle.com/datasets/hojjatk/mnist-dataset.

      • +
    +

    We will start with a regression problem and we will reuse our codes on gradient descent methods from project 1.

    +
    +

    Part a): Analytical warm-up#

    +

    When using our gradient machinery from project 1, we will need the expressions for the cost/loss functions and their respective +gradients. The functions whose gradients we need are:

    +
      +

    1. The mean-squared error (MSE) with and without the \(L_1\) and \(L_2\) norms (regression problems)

      2. +

    2. The binary cross entropy (aka log loss) for binary classification problems with and without \(L_1\) and \(L_2\) norms

      3. +

    3. The multiclass cross entropy cost/loss function (aka Softmax cross entropy or just Softmax loss function)

      4. +
    +

    Set up these three cost/loss functions and their respective derivatives and explain the various terms. In this project you will however only use the MSE and the Softmax cross entropy.

    +

    We will test three activation functions for our neural network setup, these are the

    +
      +

    1. The Sigmoid (aka logit) function,

      2. +

    2. the RELU function and

      3. +

    3. the Leaky RELU function

      4. +
    +

    Set up their expressions and their first derivatives. +You may consult the lecture notes (with codes and more) from week 42 at https://compphysics.github.io/MachineLearning/doc/LectureNotes/_build/html/week42.html.

    +
    +
    +

    Reminder about the gradient machinery from project 1#

    +

    In the setup of a neural network code you will need your gradient descent codes from +project 1. For neural networks we will recommend using stochastic +gradient descent with either the RMSprop or the ADAM algorithms for +updating the learning rates. But you should feel free to try plain gradient descent as well.

    +

    We recommend reading chapter 8 on optimization from the textbook of +Goodfellow, Bengio and Courville at +https://www.deeplearningbook.org/. This chapter contains many +useful insights and discussions on the optimization part of machine +learning. A useful reference on the back progagation algorithm is +Nielsen’s book at http://neuralnetworksanddeeplearning.com/.

    +

    You will find the Python Seaborn +package +useful when plotting the results as function of the learning rate +\(\eta\) and the hyper-parameter \(\lambda\) .

    +
    +
    +

    Part b): Writing your own Neural Network code#

    +

    Your aim now, and this is the central part of this project, is to +write your own FFNN code implementing the back +propagation algorithm discussed in the lecture slides from week 41 at https://compphysics.github.io/MachineLearning/doc/LectureNotes/_build/html/week41.html and week 42 at https://compphysics.github.io/MachineLearning/doc/LectureNotes/_build/html/week42.html.

    +

    We will focus on a regression problem first, using the one-dimensional Runge function

    +
    +\[ +f(x) = \frac{1}{1+25x^2}, +\]
    +

    from project 1.

    +

    Use only the mean-squared error as cost function (no regularization terms) and +write an FFNN code for a regression problem with a flexible number of hidden +layers and nodes using only the Sigmoid function as activation function for +the hidden layers. Initialize the weights using a normal +distribution. How would you initialize the biases? And which +activation function would you select for the final output layer? +And how would you set up your design/feature matrix? Hint: does it have to represent a polynomial approximation as you did in project 1?

    +

    Train your network and compare the results with those from your OLS +regression code from project 1 using the one-dimensional Runge +function. When comparing your neural network code with the OLS +results from project 1, use the same data sets which gave you the best +MSE score. Moreover, use the polynomial order from project 1 that gave you the +best result. Compare these results with your neural network with one +and two hidden layers using \(50\) and \(100\) hidden nodes, respectively.

    +

    Comment your results and give a critical discussion of the results +obtained with the OLS code from project 1 and your own neural network +code. Make an analysis of the learning rates employed to find the +optimal MSE score. Test both stochastic gradient descent +with RMSprop and ADAM and plain gradient descent with different +learning rates.

    +

    You should, as you did in project 1, scale your data.

    +
    +
    +

    Part c): Testing against other software libraries#

    +

    You should test your results against a similar code using Scikit-Learn (see the examples in the above lecture notes from weeks 41 and 42) or tensorflow/keras or Pytorch (for Pytorch, see Raschka et al.’s text chapters 12 and 13).

    +

    Furthermore, you should also test that your derivatives are correctly +calculated using automatic differentiation, using for example the +Autograd library or the JAX library. It is optional to implement +these libraries for the present project. In this project they serve as +useful tests of our derivatives.

    +
    +
    +

    Part d): Testing different activation functions and depths of the neural network#

    +

    You should also test different activation functions for the hidden +layers. Try out the Sigmoid, the RELU and the Leaky RELU functions and +discuss your results. Test your results as functions of the number of hidden layers and nodes. Do you see signs of overfitting? +It is optional in this project to perform a bias-variance trade-off analysis.

    +
    +
    +

    Part e): Testing different norms#

    +

    Finally, still using the one-dimensional Runge function, add now the +hyperparameters \(\lambda\) with the \(L_2\) and \(L_1\) norms. Find the +optimal results for the hyperparameters \(\lambda\) and the learning +rates \(\eta\) and neural network architecture and compare the \(L_2\) results with Ridge regression from +project 1 and the \(L_1\) results with the Lasso calculations of project 1. +Use again the same data sets and the best results from project 1 in your comparisons.

    +
    +
    +

    Part f): Classification analysis using neural networks#

    +

    With a well-written code it should now be easy to change the +activation function for the output layer.

    +

    Here we will change the cost function for our neural network code +developed in parts b), d) and e) in order to perform a classification +analysis. The classification problem we will study is the multiclass +MNIST problem, see the description of the full data set at +https://www.kaggle.com/datasets/hojjatk/mnist-dataset. We will use the Softmax cross entropy function discussed in a). +The MNIST data set discussed in the lecture notes from week 42 is a downscaled variant of the full dataset.

    +

    Feel free to suggest other data sets. If you find the classic MNIST data set somewhat limited, feel free to try the
    +MNIST-Fashion data set at for example https://www.kaggle.com/datasets/zalando-research/fashionmnist.

    +

    To set up the data set, the following python programs may be useful

    +
    +
    +
    from sklearn.datasets import fetch_openml
+
+# Fetch the MNIST dataset
+mnist = fetch_openml('mnist_784', version=1, as_frame=False, parser='auto')
+
+# Extract data (features) and target (labels)
+X = mnist.data
+y = mnist.target
+
    +
    +
    +
    +

    You should consider scaling the data. The Pixel values in MNIST range from 0 to 255. Scaling them to a 0-1 range can improve the performance of some models. That is, you could implement the following scaling

    +
    +
    +
    X = X / 255.0
+
    +
    +
    +
    +

    And then perform the standard train-test splitting

    +
    +
    +
    from sklearn.model_selection import train_test_split
+X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
+
    +
    +
    +
    +

    To measure the performance of our classification problem we will use the +so-called accuracy score. The accuracy is as you would expect just +the number of correctly guessed targets \(t_i\) divided by the total +number of targets, that is

    +
    +\[ +\text{Accuracy} = \frac{\sum_{i=1}^n I(t_i = y_i)}{n} , +\]
    +

    where \(I\) is the indicator function, \(1\) if \(t_i = y_i\) and \(0\) +otherwise if we have a binary classification problem. Here \(t_i\) +represents the target and \(y_i\) the outputs of your FFNN code and \(n\) is simply the number of targets \(t_i\).

    +

    Discuss your results and give a critical analysis of the various parameters, including hyper-parameters like the learning rates and the regularization parameter \(\lambda\), various activation functions, number of hidden layers and nodes and activation functions.

    +

    Again, we strongly recommend that you compare your own neural Network +code for classification and pertinent results against a similar code using Scikit-Learn or tensorflow/keras or pytorch.

    +

    If you have time, you can use the functionality of scikit-learn and compare your neural network results with those from Logistic regression. This is optional. +The weblink here https://medium.com/ai-in-plain-english/comparison-between-logistic-regression-and-neural-networks-in-classifying-digits-dc5e85cd93c3compares logistic regression and FFNN using the so-called MNIST data set. You may find several useful hints and ideas from this article. Your neural network code can implement the equivalent of logistic regression by simply setting the number of hidden layers to zero.

    +

    If you wish to compare with say Logisti Regression from scikit-learn, the following code uses the above data set

    +
    +
    +
    from sklearn.linear_model import LogisticRegression
+# Initialize the model
+model = LogisticRegression(solver='saga', multi_class='multinomial', max_iter=1000, random_state=42)
+# Train the model
+model.fit(X_train, y_train)
+from sklearn.metrics import accuracy_score
+# Make predictions on the test set
+y_pred = model.predict(X_test)
+# Calculate accuracy
+accuracy = accuracy_score(y_test, y_pred)
+print(f"Model Accuracy: {accuracy:.4f}")
+
    +
    +
    +
    +
    +
    +

    Part g) Critical evaluation of the various algorithms#

    +

    After all these glorious calculations, you should now summarize the +various algorithms and come with a critical evaluation of their pros +and cons. Which algorithm works best for the regression case and which +is best for the classification case. These codes can also be part of +your final project 3, but now applied to other data sets.

    +
    +
    +
    +

    Background literature#

    +
      +

    1. The text of Michael Nielsen is highly recommended, see Nielsen’s book at http://neuralnetworksanddeeplearning.com/. It is an excellent read.

      2. +

    2. Goodfellow, Bengio and Courville, Deep Learning at https://www.deeplearningbook.org/. Here we recommend chapters 6, 7 and 8

      3. +

    3. Raschka et al. at https://sebastianraschka.com/blog/2022/ml-pytorch-book.html. Here we recommend chapters 11, 12 and 13.

      4. +
    +
    +
    +

    Introduction to numerical projects#

    +

    Here follows a brief recipe and recommendation on how to write a report for each +project.

    +
      +

    • Give a short description of the nature of the problem and the eventual numerical methods you have used.

      • +

    • Describe the algorithm you have used and/or developed. Here you may find it convenient to use pseudocoding. In many cases you can describe the algorithm in the program itself.

      • +

    • Include the source code of your program. Comment your program properly.

      • +

    • If possible, try to find analytic solutions, or known limits in order to test your program when developing the code.

      • +

    • Include your results either in figure form or in a table. Remember to label your results. All tables and figures should have relevant captions and labels on the axes.

      • +

    • Try to evaluate the reliabilty and numerical stability/precision of your results. If possible, include a qualitative and/or quantitative discussion of the numerical stability, eventual loss of precision etc.

      • +

    • Try to give an interpretation of you results in your answers to the problems.

      • +

    • Critique: if possible include your comments and reflections about the exercise, whether you felt you learnt something, ideas for improvements and other thoughts you’ve made when solving the exercise. We wish to keep this course at the interactive level and your comments can help us improve it.

      • +

    • Try to establish a practice where you log your work at the computerlab. You may find such a logbook very handy at later stages in your work, especially when you don’t properly remember what a previous test version of your program did. Here you could also record the time spent on solving the exercise, various algorithms you may have tested or other topics which you feel worthy of mentioning.

      • +
    +
    +
    +

    Format for electronic delivery of report and programs#

    +

    The preferred format for the report is a PDF file. You can also use DOC or postscript formats or as an ipython notebook file. As programming language we prefer that you choose between C/C++, Fortran2008 or Python. The following prescription should be followed when preparing the report:

    +
      +

    • Use Canvas to hand in your projects, log in at https://www.uio.no/english/services/it/education/canvas/ with your normal UiO username and password.

      • +

    • Upload only the report file or the link to your GitHub/GitLab or similar typo of repos! For the source code file(s) you have developed please provide us with your link to your GitHub/GitLab or similar domain. The report file should include all of your discussions and a list of the codes you have developed. Do not include library files which are available at the course homepage, unless you have made specific changes to them.

      • +

    • In your GitHub/GitLab or similar repository, please include a folder which contains selected results. These can be in the form of output from your code for a selected set of runs and input parameters.

      • +
    +

    Finally, +we encourage you to collaborate. Optimal working groups consist of +2-3 students. You can then hand in a common report.

    +
    +
    + + + + +
    + + + + + + + + +
    + + + +
    + + +
    +
    + Contents +
    +
    + +
    + + +
    +
    + +
    + +
    + +

    +By Morten Hjorth-Jensen +

    + +
    + +
    + + +

    + + © Copyright 2023. +
    + +

    + +
    + +
    + +
    + +
    + +
    + +
    +
    + + +
    +
    +
    + + + + + +
    +
    + + \ No newline at end of file diff --git a/doc/LectureNotes/_build/html/schedule.html b/doc/LectureNotes/_build/html/schedule.html index 190d8eea9..c872cbd2e 100644 --- a/doc/LectureNotes/_build/html/schedule.html +++ b/doc/LectureNotes/_build/html/schedule.html @@ -244,6 +244,7 @@ +
  • Week 42 Constructing a Neural Network code with examples
  • Exercises week 42
    • @@ -258,6 +259,7 @@

    Projects

    diff --git a/doc/LectureNotes/_build/html/search.html b/doc/LectureNotes/_build/html/search.html index 8a659d6e4..f38a6cbc9 100644 --- a/doc/LectureNotes/_build/html/search.html +++ b/doc/LectureNotes/_build/html/search.html @@ -245,6 +245,7 @@ +
  • Week 42 Constructing a Neural Network code with examples
  • Exercises week 42
    • @@ -259,6 +260,7 @@

    Projects

    diff --git a/doc/LectureNotes/_build/html/searchindex.js b/doc/LectureNotes/_build/html/searchindex.js index a935eeddf..444471419 100644 --- a/doc/LectureNotes/_build/html/searchindex.js +++ b/doc/LectureNotes/_build/html/searchindex.js @@ -1 +1 @@ -Search.setIndex({"alltitles": {"1a)": [[18, "a"]], "3a)": [[18, "id1"]], "3b)": [[18, "b"]], "4a)": [[18, "id2"]], "4b)": [[18, "id3"]], "A Classification Tree": [[9, "a-classification-tree"]], "A Frequentist approach to data analysis": [[0, "a-frequentist-approach-to-data-analysis"], [30, "a-frequentist-approach-to-data-analysis"]], "A better approach": [[8, "a-better-approach"]], "A first summary": [[30, "a-first-summary"]], "A more compact expression": [[35, "a-more-compact-expression"], [36, "a-more-compact-expression"]], "A new Cost Function": [[34, "a-new-cost-function"]], "A quick Reminder on Lagrangian Multipliers": [[8, "a-quick-reminder-on-lagrangian-multipliers"]], "A simple example": [[4, "a-simple-example"]], "A soft classifier": [[8, "a-soft-classifier"]], "A top-down perspective on Neural networks": [[1, "a-top-down-perspective-on-neural-networks"]], "A way to Read the Bias-Variance Tradeoff": [[34, "a-way-to-read-the-bias-variance-tradeoff"], [35, "a-way-to-read-the-bias-variance-tradeoff"]], "ADAM algorithm, taken from Goodfellow et al": [[33, "adam-algorithm-taken-from-goodfellow-et-al"]], "ADAM optimizer": [[13, "adam-optimizer"], [33, "id2"]], "Accuracy": [[33, "accuracy"]], "Activation functions": [[12, "activation-functions"], [36, "activation-functions"]], "Activation functions, Logistic and Hyperbolic ones": [[36, "activation-functions-logistic-and-hyperbolic-ones"]], "AdaGrad Properties": [[33, "adagrad-properties"]], "AdaGrad Update Rule Derivation": [[33, "adagrad-update-rule-derivation"]], "AdaGrad algorithm, taken from Goodfellow et al": [[33, "adagrad-algorithm-taken-from-goodfellow-et-al"]], "Adam Optimizer": [[33, "adam-optimizer"]], "Adam vs. AdaGrad and RMSProp": [[33, "adam-vs-adagrad-and-rmsprop"]], "Adam: Bias Correction": [[33, "adam-bias-correction"]], "Adam: Exponential Moving Averages (Moments)": [[33, "adam-exponential-moving-averages-moments"]], "Adam: Update Rule Derivation": [[33, "adam-update-rule-derivation"]], "Adaptive boosting: AdaBoost, Basic Algorithm": [[10, "adaptive-boosting-adaboost-basic-algorithm"]], "Adaptivity Across Dimensions": [[33, "adaptivity-across-dimensions"]], "Adding Neural Networks": [[36, "adding-neural-networks"]], "Adding a hidden layer": [[37, "adding-a-hidden-layer"]], "Adding error analysis and training set up": [[30, "adding-error-analysis-and-training-set-up"], [31, "adding-error-analysis-and-training-set-up"]], "Adjust hyperparameters": [[1, "adjust-hyperparameters"]], "Algorithms and codes for Adagrad, RMSprop and Adam": [[33, "algorithms-and-codes-for-adagrad-rmsprop-and-adam"]], "Algorithms for Setting up Decision Trees": [[9, "algorithms-for-setting-up-decision-trees"]], "An Overview of Ensemble Methods": [[10, "an-overview-of-ensemble-methods"]], "An extrapolation example": [[4, "an-extrapolation-example"]], "An optimization/minimization problem": [[30, "an-optimization-minimization-problem"]], "Analyzing the last results": [[37, "analyzing-the-last-results"]], "And finally \\boldsymbol{X}\\boldsymbol{X}^T": [[31, "and-finally-boldsymbol-x-boldsymbol-x-t"]], "And finally ADAM": [[33, "and-finally-adam"]], "And what about using neural networks?": [[30, "and-what-about-using-neural-networks"]], "Another Example from Scikit-Learn\u2019s Repository": [[34, "another-example-from-scikit-learn-s-repository"], [35, "another-example-from-scikit-learn-s-repository"]], "Another Example, now with a polynomial fit": [[32, "another-example-now-with-a-polynomial-fit"]], "Another example, the moons again": [[9, "another-example-the-moons-again"]], "Applied Data Analysis and Machine Learning": [[23, null]], "Artificial neurons": [[36, "artificial-neurons"], [37, "artificial-neurons"]], "Assumptions made": [[34, "assumptions-made"]], "Autocorrelation function": [[27, "autocorrelation-function"]], "Automatic differentiation": [[13, "automatic-differentiation"], [37, "automatic-differentiation"]], "Automatic differentiation through examples": [[37, "automatic-differentiation-through-examples"]], "Back to Ridge and LASSO Regression": [[31, "back-to-ridge-and-lasso-regression"], [32, "back-to-ridge-and-lasso-regression"]], "Back to the Cancer Data": [[11, "back-to-the-cancer-data"]], "Background literature": [[25, "background-literature"]], "Bagging": [[10, "bagging"]], "Bagging Examples": [[10, "bagging-examples"]], "Basic Matrix Features": [[24, "basic-matrix-features"]], "Basic ideas of the Principal Component Analysis (PCA)": [[11, null]], "Basic math of the SVD": [[5, "basic-math-of-the-svd"], [31, "basic-math-of-the-svd"], [32, "basic-math-of-the-svd"]], "Basics": [[7, "basics"], [35, "basics"], [36, "basics"]], "Basics of a tree": [[9, "basics-of-a-tree"]], "Basics of an NN": [[37, "basics-of-an-nn"]], "Batch Normalization": [[1, "batch-normalization"]], "Batches and mini-batches": [[33, "batches-and-mini-batches"]], "Bayes\u2019 Theorem and Ridge and Lasso Regression": [[5, "bayes-theorem-and-ridge-and-lasso-regression"]], "Boosting, a Bird\u2019s Eye View": [[10, "boosting-a-bird-s-eye-view"]], "Bootstrap": [[6, "bootstrap"]], "Bringing it together": [[37, "bringing-it-together"]], "Bringing it together, first back propagation equation": [[12, "bringing-it-together-first-back-propagation-equation"]], "Building a Feed Forward Neural Network": [[1, null]], "Building a tree, regression": [[9, "building-a-tree-regression"]], "Building neural networks in Tensorflow and Keras": [[1, "building-neural-networks-in-tensorflow-and-keras"]], "But none of these can compete with Newton\u2019s method": [[33, "but-none-of-these-can-compete-with-newton-s-method"]], "CNNs in more detail, building convolutional neural networks in Tensorflow and Keras": [[3, "cnns-in-more-detail-building-convolutional-neural-networks-in-tensorflow-and-keras"]], "Cancer Data again now with Decision Trees and other Methods": [[9, "cancer-data-again-now-with-decision-trees-and-other-methods"]], "Chain rule": [[37, "chain-rule"]], "Chain rule, forward and reverse modes": [[37, "chain-rule-forward-and-reverse-modes"]], "Challenge: Choosing a Fixed Learning Rate": [[33, "challenge-choosing-a-fixed-learning-rate"]], "Choose cost function and optimizer": [[1, "choose-cost-function-and-optimizer"]], "Class of functions we can approximate": [[37, "class-of-functions-we-can-approximate"]], "Classical PCA Theorem": [[11, "classical-pca-theorem"]], "Classification problems": [[35, "classification-problems"], [36, "classification-problems"]], "Clustering and Unsupervised Learning": [[14, null]], "Code Example for Cross-validation and k-fold Cross-validation": [[34, "code-example-for-cross-validation-and-k-fold-cross-validation"], [35, "code-example-for-cross-validation-and-k-fold-cross-validation"]], "Code example": [[37, "code-example"]], "Code example for the Bootstrap method": [[34, "code-example-for-the-bootstrap-method"]], "Code for SVD and Inversion of Matrices": [[5, "code-for-svd-and-inversion-of-matrices"]], "Code with a Number of Minibatches which varies": [[33, "code-with-a-number-of-minibatches-which-varies"]], "Codes and Approaches": [[14, "codes-and-approaches"]], "Codes for the SVD": [[5, "codes-for-the-svd"], [31, "codes-for-the-svd"], [32, "codes-for-the-svd"]], "Coding Setup and Linear Regression": [[15, "coding-setup-and-linear-regression"]], "Collect and pre-process data": [[1, "collect-and-pre-process-data"]], "Communication channels": [[30, "communication-channels"]], "Compact expressions": [[37, "compact-expressions"]], "Compare Bagging on Trees with Random Forests": [[10, "compare-bagging-on-trees-with-random-forests"]], "Comparing with a numerical scheme": [[2, "comparing-with-a-numerical-scheme"]], "Comparison with OLS": [[32, "comparison-with-ols"]], "Completing the list": [[37, "completing-the-list"]], "Computation of gradients": [[33, "computation-of-gradients"]], "Computing the Gini index": [[9, "computing-the-gini-index"]], "Conditions on convex functions": [[32, "conditions-on-convex-functions"]], "Confidence Intervals": [[34, "confidence-intervals"]], "Conjugate gradient method": [[13, "conjugate-gradient-method"]], "Convergence rates": [[33, "convergence-rates"]], "Convex function": [[32, "convex-function"]], "Convex functions": [[13, "convex-functions"], [32, "convex-functions"]], "Convolution Examples: Polynomial multiplication": [[3, "convolution-examples-polynomial-multiplication"]], "Convolution Examples: Principle of Superposition and Periodic Forces (Fourier Transforms)": [[3, "convolution-examples-principle-of-superposition-and-periodic-forces-fourier-transforms"]], "Convolutional Neural Network": [[12, "convolutional-neural-network"], [36, "convolutional-neural-network"], [37, "convolutional-neural-network"]], "Convolutional Neural Networks": [[3, null]], "Correlation Function and Design/Feature Matrix": [[31, "correlation-function-and-design-feature-matrix"]], "Correlation Matrix": [[11, "correlation-matrix"], [31, "correlation-matrix"]], "Correlation Matrix with Pandas": [[31, "correlation-matrix-with-pandas"]], "Counting the number of floating point operations": [[37, "counting-the-number-of-floating-point-operations"]], "Course Format": [[30, "course-format"]], "Course setting": [[26, null]], "Covariance Matrix Examples": [[31, "covariance-matrix-examples"]], "Covariance and Correlation Matrix": [[31, "covariance-and-correlation-matrix"]], "Cross-validation": [[6, "cross-validation"]], "Cross-validation in brief": [[34, "cross-validation-in-brief"], [35, "cross-validation-in-brief"]], "Deadlines for projects (tentative)": [[30, "deadlines-for-projects-tentative"]], "Decision trees, overarching aims": [[9, null]], "Deep Neural Networks": [[33, "deep-neural-networks"]], "Deep learning methods": [[30, "deep-learning-methods"]], "Define model and architecture": [[1, "define-model-and-architecture"]], "Defining intermediate operations": [[37, "defining-intermediate-operations"]], "Defining the cost function": [[1, "defining-the-cost-function"]], "Definitions": [[19, "definitions"], [37, "definitions"]], "Deliverables": [[15, "deliverables"], [16, "deliverables"], [19, "deliverables"], [20, "deliverables"], [25, "deliverables"]], "Derivation of the AdaGrad Algorithm": [[33, "derivation-of-the-adagrad-algorithm"]], "Derivative of the cost function": [[37, "derivative-of-the-cost-function"]], "Derivatives and the chain rule": [[12, "derivatives-and-the-chain-rule"], [37, "derivatives-and-the-chain-rule"]], "Derivatives in terms of z_j^L": [[37, "derivatives-in-terms-of-z-j-l"]], "Derivatives of the hidden layer": [[37, "derivatives-of-the-hidden-layer"]], "Derivatives, example 1": [[31, "derivatives-example-1"]], "Deriving OLS from a probability distribution": [[5, "deriving-ols-from-a-probability-distribution"], [34, "deriving-ols-from-a-probability-distribution"]], "Deriving and Implementing Ordinary Least Squares": [[16, "deriving-and-implementing-ordinary-least-squares"]], "Deriving and Implementing Ridge Regression": [[17, "deriving-and-implementing-ridge-regression"]], "Deriving the Lasso Regression Equations": [[31, "deriving-the-lasso-regression-equations"], [32, "deriving-the-lasso-regression-equations"], [32, "id6"]], "Deriving the Ridge Regression Equations": [[31, "deriving-the-ridge-regression-equations"], [32, "deriving-the-ridge-regression-equations"], [32, "id3"]], "Deriving the back propagation code for a multilayer perceptron model": [[12, "deriving-the-back-propagation-code-for-a-multilayer-perceptron-model"]], "Developing a code for doing neural networks with back propagation": [[1, "developing-a-code-for-doing-neural-networks-with-back-propagation"]], "Diagonalize the sample covariance matrix to obtain the principal components": [[11, "diagonalize-the-sample-covariance-matrix-to-obtain-the-principal-components"]], "Different kernels and Mercer\u2019s theorem": [[8, "different-kernels-and-mercer-s-theorem"]], "Disadvantages": [[9, "disadvantages"]], "Discriminative Modeling": [[30, "discriminative-modeling"]], "Discussing the correlation data": [[36, "discussing-the-correlation-data"]], "Does Logistic Regression do a better Job?": [[36, "does-logistic-regression-do-a-better-job"]], "Domains and probabilities": [[27, "domains-and-probabilities"]], "Dropout": [[1, "dropout"]], "Economy-size SVD": [[31, "economy-size-svd"], [32, "economy-size-svd"]], "Elements of Probability Theory and Statistical Data Analysis": [[27, null]], "Empirical Evidence: Convergence Time and Memory in Practice": [[33, "empirical-evidence-convergence-time-and-memory-in-practice"]], "Ensemble Methods: From a Single Tree to Many Trees and Extreme Boosting, Meet the Jungle of Methods": [[10, null]], "Entropy and the ID3 algorithm": [[9, "entropy-and-the-id3-algorithm"]], "Essential elements of ML": [[30, "essential-elements-of-ml"]], "Evaluate model performance on test data": [[1, "evaluate-model-performance-on-test-data"]], "Example 2": [[31, "example-2"]], "Example 3": [[31, "example-3"]], "Example 4": [[31, "example-4"]], "Example Matrix": [[31, "example-matrix"], [32, "example-matrix"]], "Example code for Bias-Variance tradeoff": [[34, "example-code-for-bias-variance-tradeoff"]], "Example code for Logistic Regression": [[35, "example-code-for-logistic-regression"], [36, "example-code-for-logistic-regression"]], "Example of discriminative modeling, taken from Generative Deep Learning by David Foster": [[30, "example-of-discriminative-modeling-taken-from-generative-deep-learning-by-david-foster"]], "Example of generative modeling, taken from Generative Deep Learning by David Foster": [[30, "example-of-generative-modeling-taken-from-generative-deep-learning-by-david-foster"]], "Example of own Standard scaling": [[31, "example-of-own-standard-scaling"]], "Example relevant for the exercises": [[31, "example-relevant-for-the-exercises"]], "Example: Exponential decay": [[2, "example-exponential-decay"]], "Example: Population growth": [[2, "example-population-growth"]], "Example: The diffusion equation": [[2, "example-the-diffusion-equation"]], "Example: binary classification problem": [[1, "example-binary-classification-problem"]], "Examples": [[30, "examples"]], "Examples of XOR, OR and AND gates": [[36, "examples-of-xor-or-and-and-gates"]], "Examples of likelihood functions used in logistic regression and neural networks": [[7, "examples-of-likelihood-functions-used-in-logistic-regression-and-neural-networks"]], "Examples of likelihood functions used in logistic regression and nueral networks": [[35, "examples-of-likelihood-functions-used-in-logistic-regression-and-nueral-networks"]], "Exercise 1": [[21, "exercise-1"]], "Exercise 1 - Choice of model and degrees of freedom": [[17, "exercise-1-choice-of-model-and-degrees-of-freedom"]], "Exercise 1 - Finding the derivative of Matrix-Vector expressions": [[16, "exercise-1-finding-the-derivative-of-matrix-vector-expressions"]], "Exercise 1 - Github Setup": [[15, "exercise-1-github-setup"]], "Exercise 1 - Understand the feed forward pass": [[22, "exercise-1-understand-the-feed-forward-pass"]], "Exercise 1, scale your data": [[18, "exercise-1-scale-your-data"]], "Exercise 1: Creating the report document": [[20, "exercise-1-creating-the-report-document"]], "Exercise 1: Expectation values for ordinary least squares expressions": [[19, "exercise-1-expectation-values-for-ordinary-least-squares-expressions"]], "Exercise 1: Including more data": [[37, "exercise-1-including-more-data"]], "Exercise 1: Setting up various Python environments": [[0, "exercise-1-setting-up-various-python-environments"]], "Exercise 2": [[21, "exercise-2"]], "Exercise 2 - Deriving the expression for OLS": [[16, "exercise-2-deriving-the-expression-for-ols"]], "Exercise 2 - Deriving the expression for Ridge Regression": [[17, "exercise-2-deriving-the-expression-for-ridge-regression"]], "Exercise 2 - Gradient with one layer using autograd": [[22, "exercise-2-gradient-with-one-layer-using-autograd"]], "Exercise 2 - Setting up a Github repository": [[15, "exercise-2-setting-up-a-github-repository"]], "Exercise 2, calculate the gradients": [[18, "exercise-2-calculate-the-gradients"]], "Exercise 2: Adding good figures": [[20, "exercise-2-adding-good-figures"]], "Exercise 2: Expectation values for Ridge regression": [[19, "exercise-2-expectation-values-for-ridge-regression"]], "Exercise 2: Extended program": [[37, "exercise-2-extended-program"]], "Exercise 2: making your own data and exploring scikit-learn": [[0, "exercise-2-making-your-own-data-and-exploring-scikit-learn"]], "Exercise 3": [[21, "exercise-3"]], "Exercise 3 - Creating feature matrix and implementing OLS using the analytical expression": [[16, "exercise-3-creating-feature-matrix-and-implementing-ols-using-the-analytical-expression"]], "Exercise 3 - Fitting an OLS model to data": [[15, "exercise-3-fitting-an-ols-model-to-data"]], "Exercise 3 - Gradient with one layer writing backpropagation by hand": [[22, "exercise-3-gradient-with-one-layer-writing-backpropagation-by-hand"]], "Exercise 3 - Scaling data": [[17, "exercise-3-scaling-data"]], "Exercise 3 - Setting up a Python virtual environment": [[15, "exercise-3-setting-up-a-python-virtual-environment"]], "Exercise 3, using the analytical formulae for OLS and Ridge regression to find the optimal paramters \\boldsymbol{\\theta}": [[18, "exercise-3-using-the-analytical-formulae-for-ols-and-ridge-regression-to-find-the-optimal-paramters-boldsymbol-theta"]], "Exercise 3: Deriving the expression for the Bias-Variance Trade-off": [[19, "exercise-3-deriving-the-expression-for-the-bias-variance-trade-off"]], "Exercise 3: Normalizing our data": [[0, "exercise-3-normalizing-our-data"]], "Exercise 3: Writing an abstract and introduction": [[20, "exercise-3-writing-an-abstract-and-introduction"]], "Exercise 4 - Custom activation for each layer": [[21, "exercise-4-custom-activation-for-each-layer"]], "Exercise 4 - Fitting a polynomial": [[16, "exercise-4-fitting-a-polynomial"]], "Exercise 4 - Gradient with two layers writing backpropagation by hand": [[22, "exercise-4-gradient-with-two-layers-writing-backpropagation-by-hand"]], "Exercise 4 - Implementing Ridge Regression": [[17, "exercise-4-implementing-ridge-regression"]], "Exercise 4 - Testing multiple hyperparameters": [[17, "exercise-4-testing-multiple-hyperparameters"]], "Exercise 4 - The train-test split": [[15, "exercise-4-the-train-test-split"]], "Exercise 4, Implementing the simplest form for gradient descent": [[18, "exercise-4-implementing-the-simplest-form-for-gradient-descent"]], "Exercise 4: Adding Ridge Regression": [[0, "exercise-4-adding-ridge-regression"]], "Exercise 4: Computing the Bias and Variance": [[19, "exercise-4-computing-the-bias-and-variance"]], "Exercise 4: Making the code available and presentable": [[20, "exercise-4-making-the-code-available-and-presentable"]], "Exercise 5 - Comparing your code with sklearn": [[16, "exercise-5-comparing-your-code-with-sklearn"]], "Exercise 5 - Gradient with any number of layers writing backpropagation by hand": [[22, "exercise-5-gradient-with-any-number-of-layers-writing-backpropagation-by-hand"]], "Exercise 5 - Processing multiple inputs at once": [[21, "exercise-5-processing-multiple-inputs-at-once"]], "Exercise 5, Ridge regression and a new Synthetic Dataset": [[18, "exercise-5-ridge-regression-and-a-new-synthetic-dataset"]], "Exercise 5: Analytical exercises": [[0, "exercise-5-analytical-exercises"]], "Exercise 5: Interpretation of scaling and metrics": [[19, "exercise-5-interpretation-of-scaling-and-metrics"]], "Exercise 5: Referencing": [[20, "exercise-5-referencing"]], "Exercise 6 - Batched inputs": [[22, "exercise-6-batched-inputs"]], "Exercise 6 - Predicting on real data": [[21, "exercise-6-predicting-on-real-data"]], "Exercise 7 - Training": [[22, "exercise-7-training"]], "Exercise 7 - Training on real data (Optional)": [[21, "exercise-7-training-on-real-data-optional"]], "Exercise 8 (Optional) - Object orientation": [[22, "exercise-8-optional-object-orientation"]], "Exercise: Cross-validation as resampling techniques, adding more complexity": [[6, "exercise-cross-validation-as-resampling-techniques-adding-more-complexity"]], "Exercise: Analysis of real data": [[6, "exercise-analysis-of-real-data"]], "Exercise: Bias-variance trade-off and resampling techniques": [[6, "exercise-bias-variance-trade-off-and-resampling-techniques"]], "Exercise: Lasso Regression on the Franke function with resampling": [[6, "exercise-lasso-regression-on-the-franke-function-with-resampling"]], "Exercise: Ordinary Least Square (OLS) on the Franke function": [[6, "exercise-ordinary-least-square-ols-on-the-franke-function"]], "Exercise: Ridge Regression on the Franke function with resampling": [[6, "exercise-ridge-regression-on-the-franke-function-with-resampling"]], "Exercises": [[0, "exercises"]], "Exercises and Projects": [[6, "exercises-and-projects"]], "Exercises week 34": [[15, null]], "Exercises week 35": [[16, null]], "Exercises week 36": [[17, null]], "Exercises week 37": [[18, null]], "Exercises week 38": [[19, null]], "Exercises week 39": [[20, null]], "Exercises week 41": [[21, null]], "Exercises week 42": [[22, null]], "Expectation value and variance": [[34, "expectation-value-and-variance"]], "Expectation value and variance for \\boldsymbol{\\theta}": [[34, "expectation-value-and-variance-for-boldsymbol-theta"]], "Expectation values": [[27, "expectation-values"]], "Explicit derivatives": [[37, "explicit-derivatives"]], "Extending to more predictors": [[35, "extending-to-more-predictors"], [36, "extending-to-more-predictors"]], "Extending to more than one variable": [[32, "extending-to-more-than-one-variable"]], "Extremely useful tools, strongly recommended": [[30, "extremely-useful-tools-strongly-recommended"]], "Feed-forward neural networks": [[12, "feed-forward-neural-networks"], [36, "feed-forward-neural-networks"], [37, "feed-forward-neural-networks"]], "Feed-forward pass": [[1, "feed-forward-pass"]], "Final back propagating equation": [[12, "final-back-propagating-equation"], [37, "final-back-propagating-equation"]], "Final derivatives": [[37, "final-derivatives"]], "Final expression": [[37, "final-expression"]], "Final expressions for the biases of the hidden layer": [[37, "final-expressions-for-the-biases-of-the-hidden-layer"]], "Finding the Limit": [[34, "finding-the-limit"]], "Fine-tuning neural network hyperparameters": [[1, "fine-tuning-neural-network-hyperparameters"]], "First network example, simple percepetron with one input": [[37, "first-network-example-simple-percepetron-with-one-input"]], "Fitting an Equation of State for Dense Nuclear Matter": [[0, "fitting-an-equation-of-state-for-dense-nuclear-matter"]], "Fixing the singularity": [[31, "fixing-the-singularity"], [32, "fixing-the-singularity"]], "Format for electronic delivery of report and programs": [[25, "format-for-electronic-delivery-of-report-and-programs"]], "Forward and reverse modes": [[37, "forward-and-reverse-modes"]], "Frequently used scaling functions": [[31, "frequently-used-scaling-functions"], [33, "frequently-used-scaling-functions"]], "From OLS to Ridge and Lasso": [[32, "from-ols-to-ridge-and-lasso"]], "From one to many layers, the universal approximation theorem": [[12, "from-one-to-many-layers-the-universal-approximation-theorem"]], "Functionality in Scikit-Learn": [[31, "functionality-in-scikit-learn"], [33, "functionality-in-scikit-learn"]], "Further Dimensionality Remarks": [[3, "further-dimensionality-remarks"]], "Further properties (important for our analyses later)": [[5, "further-properties-important-for-our-analyses-later"], [31, "further-properties-important-for-our-analyses-later"], [32, "further-properties-important-for-our-analyses-later"]], "Gaussian Elimination": [[24, "gaussian-elimination"]], "General Features": [[9, "general-features"]], "General linear models and linear algebra": [[30, "general-linear-models-and-linear-algebra"]], "Generalizing the fitting procedure as a linear algebra problem": [[30, "generalizing-the-fitting-procedure-as-a-linear-algebra-problem"], [30, "id1"]], "Generative Adversarial Networks": [[4, "generative-adversarial-networks"]], "Generative Models": [[4, "generative-models"]], "Generative Versus Discriminative Modeling": [[30, "generative-versus-discriminative-modeling"]], "Geometric Interpretation and link with Singular Value Decomposition": [[11, "geometric-interpretation-and-link-with-singular-value-decomposition"]], "Getting serious, the back propagation equations for a neural network": [[37, "getting-serious-the-back-propagation-equations-for-a-neural-network"]], "Getting started with project 1": [[20, "getting-started-with-project-1"]], "Gradient Boosting, Classification Example": [[10, "gradient-boosting-classification-example"]], "Gradient Boosting, Examples of Regression": [[10, "gradient-boosting-examples-of-regression"]], "Gradient Clipping": [[1, "gradient-clipping"]], "Gradient Descent Example": [[32, "id1"], [33, "id1"]], "Gradient boosting: Basics with Steepest Descent/Functional Gradient Descent": [[10, "gradient-boosting-basics-with-steepest-descent-functional-gradient-descent"]], "Gradient descent": [[2, "gradient-descent"]], "Gradient descent and Ridge": [[32, "gradient-descent-and-ridge"], [33, "gradient-descent-and-ridge"]], "Gradient descent and revisiting Ordinary Least Squares from last week": [[33, "gradient-descent-and-revisiting-ordinary-least-squares-from-last-week"]], "Gradient descent example": [[32, "gradient-descent-example"], [33, "gradient-descent-example"]], "Gradient expressions": [[37, "gradient-expressions"]], "Grading": [[28, "grading"], [28, "id2"], [30, "grading"]], "How to take derivatives of Matrix-Vector expressions": [[16, "how-to-take-derivatives-of-matrix-vector-expressions"]], "Hyperplanes and all that": [[8, "hyperplanes-and-all-that"]], "Identifying Terms": [[34, "identifying-terms"]], "Illustration of a single perceptron model and a multi-perceptron model": [[36, "illustration-of-a-single-perceptron-model-and-a-multi-perceptron-model"], [37, "illustration-of-a-single-perceptron-model-and-a-multi-perceptron-model"]], "Important Matrix and vector handling packages": [[24, "important-matrix-and-vector-handling-packages"]], "Important observations": [[37, "important-observations"]], "Important technicalities: More on Rescaling data": [[31, "important-technicalities-more-on-rescaling-data"]], "Improving gradient descent with momentum": [[33, "improving-gradient-descent-with-momentum"]], "Improving performance": [[1, "improving-performance"]], "In general not this simple": [[37, "in-general-not-this-simple"]], "In summary": [[28, "in-summary"]], "Including Stochastic Gradient Descent with Autograd": [[13, "including-stochastic-gradient-descent-with-autograd"], [33, "including-stochastic-gradient-descent-with-autograd"]], "Including more classes": [[35, "including-more-classes"], [36, "including-more-classes"]], "Incremental PCA": [[11, "incremental-pca"]], "Independent and Identically Distributed (iid)": [[34, "independent-and-identically-distributed-iid"]], "Inputs to the activation function": [[37, "inputs-to-the-activation-function"]], "Installing R, C++, cython or Julia": [[30, "installing-r-c-cython-or-julia"]], "Installing R, C++, cython, Numba etc": [[30, "installing-r-c-cython-numba-etc"]], "Instructor information": [[28, "instructor-information"]], "Interpretations and optimizing our parameters": [[30, "interpretations-and-optimizing-our-parameters"], [30, "id2"], [30, "id3"], [31, "interpretations-and-optimizing-our-parameters"], [31, "id1"], [31, "id2"]], "Interpreting the Ridge results": [[31, "interpreting-the-ridge-results"], [32, "interpreting-the-ridge-results"], [32, "id4"]], "Introducing JAX": [[13, "introducing-jax"]], "Introducing the Covariance and Correlation functions": [[11, "introducing-the-covariance-and-correlation-functions"], [31, "introducing-the-covariance-and-correlation-functions"]], "Introduction": [[0, "introduction"], [6, "introduction"], [23, "introduction"], [24, "introduction"]], "Introduction to Neural networks": [[36, "introduction-to-neural-networks"], [37, "introduction-to-neural-networks"]], "Introduction to numerical projects": [[25, "introduction-to-numerical-projects"]], "Iterative Fitting, Classification and AdaBoost": [[10, "iterative-fitting-classification-and-adaboost"]], "Iterative Fitting, Regression and Squared-error Cost Function": [[10, "iterative-fitting-regression-and-squared-error-cost-function"]], "Kernel PCA": [[11, "kernel-pca"]], "Kernels and non-linearity": [[8, "kernels-and-non-linearity"]], "LU Decomposition, the inverse of a matrix": [[24, "lu-decomposition-the-inverse-of-a-matrix"]], "Lab sessions Tuesday and Wednesday": [[36, "lab-sessions-tuesday-and-wednesday"]], "Lab sessions on Tuesday and Wednesday": [[37, "lab-sessions-on-tuesday-and-wednesday"]], "Lab sessions week 39": [[35, "lab-sessions-week-39"]], "Lasso Regression": [[32, "lasso-regression"]], "Lasso case": [[32, "lasso-case"]], "Layers": [[1, "layers"]], "Layers used to build CNNs": [[3, "layers-used-to-build-cnns"]], "Layout of a neural network with three hidden layers": [[37, "layout-of-a-neural-network-with-three-hidden-layers"]], "Layout of a simple neural network with no hidden layer": [[37, "layout-of-a-simple-neural-network-with-no-hidden-layer"]], "Layout of a simple neural network with one hidden layer": [[37, "layout-of-a-simple-neural-network-with-one-hidden-layer"]], "Layout of a simple neural network with two input nodes, one hidden layer and one output node": [[37, "layout-of-a-simple-neural-network-with-two-input-nodes-one-hidden-layer-and-one-output-node"]], "Learning goals": [[15, "learning-goals"], [16, "learning-goals"], [17, "learning-goals"], [18, "learning-goals"], [19, "learning-goals"], [20, "learning-goals"]], "Learning outcomes": [[23, "learning-outcomes"], [30, "learning-outcomes"]], "Lecture Monday October 6": [[37, "lecture-monday-october-6"]], "Lecture Monday September 29, 2025": [[36, "lecture-monday-september-29-2025"]], "Lecture material": [[35, "lecture-material"]], "Lectures and ComputerLab": [[30, "lectures-and-computerlab"]], "Limitations of supervised learning with deep networks": [[1, "limitations-of-supervised-learning-with-deep-networks"]], "Linear Algebra, Handling of Arrays and more Python Features": [[24, null]], "Linear Regression": [[0, null]], "Linear Regression Problems": [[31, "linear-regression-problems"], [32, "linear-regression-problems"]], "Linear Regression and the SVD": [[32, "linear-regression-and-the-svd"]], "Linear Regression, basic elements": [[0, "linear-regression-basic-elements"]], "Linear classifier": [[35, "linear-classifier"]], "Linking Bayes\u2019 Theorem with Ridge and Lasso Regression": [[5, "linking-bayes-theorem-with-ridge-and-lasso-regression"]], "Linking the regression analysis with a statistical interpretation": [[5, "linking-the-regression-analysis-with-a-statistical-interpretation"], [34, "linking-the-regression-analysis-with-a-statistical-interpretation"]], "Linking with the SVD": [[5, "linking-with-the-svd"], [31, "linking-with-the-svd"]], "Links to relevant courses at the University of Oslo": [[29, "links-to-relevant-courses-at-the-university-of-oslo"]], "Logistic Regression": [[7, null], [7, "id1"], [35, "logistic-regression"]], "Logistic Regression, from last week": [[36, "logistic-regression-from-last-week"]], "MNIST and GANs": [[4, "mnist-and-gans"]], "Machine Learning": [[30, "machine-learning"]], "Machine learning": [[23, "machine-learning"]], "Main textbooks": [[30, "main-textbooks"]], "Making a tree": [[9, "making-a-tree"]], "Making your own Bootstrap: Changing the Level of the Decision Tree": [[10, "making-your-own-bootstrap-changing-the-level-of-the-decision-tree"]], "Making your own test-train splitting": [[31, "making-your-own-test-train-splitting"]], "Material for exercises week 35": [[31, "material-for-exercises-week-35"]], "Material for lab sessions sessions Tuesday and Wednesday": [[32, "material-for-lab-sessions-sessions-tuesday-and-wednesday"]], "Material for lecture Monday September 2": [[32, "material-for-lecture-monday-september-2"]], "Material for lecture Monday September 8": [[33, "material-for-lecture-monday-september-8"]], "Material for the lab sessions": [[33, "material-for-the-lab-sessions"], [34, "material-for-the-lab-sessions"]], "Material for the lecture on Monday October 6, 2025": [[37, "material-for-the-lecture-on-monday-october-6-2025"]], "Mathematical Interpretation of Ordinary Least Squares": [[5, "mathematical-interpretation-of-ordinary-least-squares"], [31, "mathematical-interpretation-of-ordinary-least-squares"], [32, "mathematical-interpretation-of-ordinary-least-squares"]], "Mathematical model": [[36, "mathematical-model"], [36, "id1"], [36, "id2"], [36, "id3"], [36, "id4"]], "Mathematical optimization of convex functions": [[8, "mathematical-optimization-of-convex-functions"]], "Mathematics of CNNs": [[3, "mathematics-of-cnns"]], "Mathematics of deep learning": [[37, "mathematics-of-deep-learning"]], "Mathematics of deep learning and neural networks": [[37, "mathematics-of-deep-learning-and-neural-networks"]], "Mathematics of the SVD and implications": [[5, "mathematics-of-the-svd-and-implications"], [31, "mathematics-of-the-svd-and-implications"], [32, "mathematics-of-the-svd-and-implications"]], "Matrices in Python": [[30, "matrices-in-python"]], "Matrix multiplication": [[1, "matrix-multiplication"]], "Matrix-vector notation": [[36, "matrix-vector-notation"]], "Matrix-vector notation and activation": [[12, "matrix-vector-notation-and-activation"], [36, "matrix-vector-notation-and-activation"]], "Maximum Likelihood Estimation (MLE)": [[34, "maximum-likelihood-estimation-mle"]], "Maximum likelihood": [[35, "maximum-likelihood"], [36, "maximum-likelihood"]], "Meet the covariance!": [[27, "meet-the-covariance"]], "Meet the Covariance Matrix": [[5, "meet-the-covariance-matrix"], [31, "meet-the-covariance-matrix"]], "Meet the Hessian Matrix": [[31, "meet-the-hessian-matrix"]], "Meet the Pandas": [[30, "meet-the-pandas"]], "Memory Usage and Scalability": [[33, "memory-usage-and-scalability"]], "Memory constraints": [[33, "memory-constraints"]], "Min-Max Scaling": [[31, "min-max-scaling"]], "Minimizing the cross entropy": [[35, "minimizing-the-cross-entropy"], [36, "minimizing-the-cross-entropy"]], "Momentum based GD": [[13, "momentum-based-gd"], [33, "momentum-based-gd"]], "More classes": [[35, "more-classes"], [36, "more-classes"]], "More complicated Example: The Ising model": [[6, "more-complicated-example-the-ising-model"]], "More complicated function": [[37, "more-complicated-function"]], "More considerations": [[37, "more-considerations"]], "More examples on bootstrap and cross-validation and errors": [[34, "more-examples-on-bootstrap-and-cross-validation-and-errors"], [35, "more-examples-on-bootstrap-and-cross-validation-and-errors"]], "More interpretations": [[31, "more-interpretations"], [32, "more-interpretations"], [32, "id5"]], "More on Dimensionalities": [[3, "more-on-dimensionalities"]], "More on Rescaling data": [[6, "more-on-rescaling-data"]], "More on Steepest descent": [[32, "more-on-steepest-descent"]], "More on convex functions": [[32, "more-on-convex-functions"]], "More on the general approximation theorem": [[37, "more-on-the-general-approximation-theorem"]], "More preprocessing": [[31, "more-preprocessing"], [33, "more-preprocessing"]], "Motivation for Adaptive Step Sizes": [[33, "motivation-for-adaptive-step-sizes"]], "Multilayer perceptrons": [[12, "multilayer-perceptrons"], [36, "multilayer-perceptrons"], [37, "multilayer-perceptrons"]], "Multivariable functions": [[37, "multivariable-functions"]], "Network requirements": [[2, "network-requirements"]], "Neural Networks vs CNNs": [[3, "neural-networks-vs-cnns"]], "Neural network types": [[36, "neural-network-types"], [37, "neural-network-types"]], "Neural networks": [[12, null]], "New expression for the derivative": [[37, "new-expression-for-the-derivative"]], "Non-Convex Problems": [[33, "non-convex-problems"]], "Note about SVD Calculations": [[31, "note-about-svd-calculations"], [32, "note-about-svd-calculations"]], "Note on Scikit-Learn": [[32, "note-on-scikit-learn"]], "Numerical experiments and the covariance, central limit theorem": [[27, "numerical-experiments-and-the-covariance-central-limit-theorem"]], "Numpy and arrays": [[24, "numpy-and-arrays"], [30, "numpy-and-arrays"]], "Numpy examples and Important Matrix and vector handling packages": [[30, "numpy-examples-and-important-matrix-and-vector-handling-packages"]], "Optimization and Deep learning": [[35, "optimization-and-deep-learning"], [36, "optimization-and-deep-learning"]], "Optimization and gradient descent, the central part of any Machine Learning algortithm": [[32, "optimization-and-gradient-descent-the-central-part-of-any-machine-learning-algortithm"]], "Optimization, the central part of any Machine Learning algortithm": [[13, null], [35, "optimization-the-central-part-of-any-machine-learning-algortithm"], [36, "optimization-the-central-part-of-any-machine-learning-algortithm"]], "Optimizing our parameters": [[30, "optimizing-our-parameters"]], "Optimizing our parameters, more details": [[30, "optimizing-our-parameters-more-details"]], "Optimizing the cost function": [[1, "optimizing-the-cost-function"]], "Optimizing the parameters": [[37, "optimizing-the-parameters"]], "Organizing our data": [[0, "organizing-our-data"], [30, "organizing-our-data"]], "Other Matrix and Vector Operations": [[24, "other-matrix-and-vector-operations"]], "Other Types of Recurrent Neural Networks": [[4, "other-types-of-recurrent-neural-networks"]], "Other courses on Data science and Machine Learning at UiO": [[30, "other-courses-on-data-science-and-machine-learning-at-uio"]], "Other courses on Data science and Machine Learning at UiO, contn": [[30, "other-courses-on-data-science-and-machine-learning-at-uio-contn"]], "Other ingredients of a neural network": [[37, "other-ingredients-of-a-neural-network"]], "Other measures in classification studies": [[36, "other-measures-in-classification-studies"]], "Other parameters": [[37, "other-parameters"]], "Other popular texts": [[30, "other-popular-texts"]], "Other techniques": [[11, "other-techniques"]], "Other types of networks": [[12, "other-types-of-networks"], [36, "other-types-of-networks"], [37, "other-types-of-networks"]], "Other ways of visualizing the trees": [[9, "other-ways-of-visualizing-the-trees"]], "Our model for the nuclear binding energies": [[30, "our-model-for-the-nuclear-binding-energies"]], "Output layer": [[37, "output-layer"]], "Overarching aims of the exercises this week": [[21, "overarching-aims-of-the-exercises-this-week"], [22, "overarching-aims-of-the-exercises-this-week"]], "Overarching view of a neural network": [[37, "overarching-view-of-a-neural-network"]], "Overview of first week": [[30, "overview-of-first-week"]], "Overview video on Stochastic Gradient Descent (SGD)": [[33, "overview-video-on-stochastic-gradient-descent-sgd"]], "Own code for Ordinary Least Squares": [[30, "own-code-for-ordinary-least-squares"], [31, "own-code-for-ordinary-least-squares"]], "PCA and scikit-learn": [[11, "pca-and-scikit-learn"]], "Pandas AI": [[30, "pandas-ai"]], "Parameters of neural networks": [[37, "parameters-of-neural-networks"]], "Part a : Ordinary Least Square (OLS) for the Runge function": [[25, "part-a-ordinary-least-square-ols-for-the-runge-function"]], "Part b: Adding Ridge regression for the Runge function": [[25, "part-b-adding-ridge-regression-for-the-runge-function"]], "Part c: Writing your own gradient descent code": [[25, "part-c-writing-your-own-gradient-descent-code"]], "Part d: Including momentum and more advanced ways to update the learning the rate": [[25, "part-d-including-momentum-and-more-advanced-ways-to-update-the-learning-the-rate"]], "Part e: Writing our own code for Lasso regression": [[25, "part-e-writing-our-own-code-for-lasso-regression"]], "Part f: Stochastic gradient descent": [[25, "part-f-stochastic-gradient-descent"]], "Part g: Bias-variance trade-off and resampling techniques": [[25, "part-g-bias-variance-trade-off-and-resampling-techniques"]], "Part h): Cross-validation as resampling techniques, adding more complexity": [[25, "part-h-cross-validation-as-resampling-techniques-adding-more-complexity"]], "Partial Differential Equations": [[2, "partial-differential-equations"]], "Plan for week 39, September 22-26, 2025": [[35, "plan-for-week-39-september-22-26-2025"]], "Plan for week 41, October 6-10": [[37, "plan-for-week-41-october-6-10"]], "Plans for week 35": [[31, "plans-for-week-35"]], "Plans for week 36": [[32, "plans-for-week-36"]], "Plans for week 37, lecture Monday": [[33, "plans-for-week-37-lecture-monday"]], "Plans for week 38, lecture Monday September 15": [[34, "plans-for-week-38-lecture-monday-september-15"]], "Plotting the Histogram": [[34, "plotting-the-histogram"]], "Plotting the mean value for each group": [[35, "plotting-the-mean-value-for-each-group"]], "Practical tips": [[13, "practical-tips"], [33, "practical-tips"]], "Practicalities": [[28, "practicalities"], [28, "id1"]], "Preamble: Note on writing reports, using reference material, AI and other tools": [[25, "preamble-note-on-writing-reports-using-reference-material-ai-and-other-tools"]], "Predicting New Points With A Trained Recurrent Neural Network": [[4, "predicting-new-points-with-a-trained-recurrent-neural-network"]], "Preprocessing our data": [[31, "preprocessing-our-data"]], "Prerequisites": [[30, "prerequisites"]], "Prerequisites and background": [[23, "prerequisites-and-background"]], "Prerequisites: Collect and pre-process data": [[3, "prerequisites-collect-and-pre-process-data"]], "Probability Distribution Functions": [[27, "probability-distribution-functions"]], "Program example for gradient descent with Ridge Regression": [[32, "program-example-for-gradient-descent-with-ridge-regression"], [33, "program-example-for-gradient-descent-with-ridge-regression"]], "Program for stochastic gradient": [[13, "program-for-stochastic-gradient"]], "Project 1 on Machine Learning, deadline October 6 (midnight), 2025": [[25, null]], "Properties of PDFs": [[27, "properties-of-pdfs"]], "Pros and cons": [[33, "pros-and-cons"]], "Pros and cons of trees, pros": [[9, "pros-and-cons-of-trees-pros"]], "Python installers": [[23, "python-installers"], [30, "python-installers"]], "RMS prop": [[13, "rms-prop"]], "RMSProp algorithm, taken from Goodfellow et al": [[33, "rmsprop-algorithm-taken-from-goodfellow-et-al"]], "RMSProp: Adaptive Learning Rates": [[33, "rmsprop-adaptive-learning-rates"]], "RMSprop for adaptive learning rate with Stochastic Gradient Descent": [[33, "rmsprop-for-adaptive-learning-rate-with-stochastic-gradient-descent"]], "Random Numbers": [[27, "random-numbers"]], "Random forests": [[10, "random-forests"]], "Randomized PCA": [[11, "randomized-pca"]], "Reading material": [[30, "reading-material"]], "Reading recommendations:": [[31, "reading-recommendations"]], "Reading suggestions week 34": [[30, "reading-suggestions-week-34"]], "Readings and Videos": [[34, "readings-and-videos"]], "Readings and Videos, logistic regression": [[35, "readings-and-videos-logistic-regression"]], "Readings and Videos, resampling methods": [[35, "readings-and-videos-resampling-methods"]], "Readings and Videos:": [[33, "readings-and-videos"], [37, "readings-and-videos"]], "Recurrent neural networks": [[12, "recurrent-neural-networks"], [36, "recurrent-neural-networks"], [37, "recurrent-neural-networks"]], "Recurrent neural networks: Overarching view": [[4, null]], "Reducing the number of degrees of freedom, overarching view": [[0, "reducing-the-number-of-degrees-of-freedom-overarching-view"], [31, "reducing-the-number-of-degrees-of-freedom-overarching-view"]], "Reducing the number of operations": [[37, "reducing-the-number-of-operations"]], "Reformulating the problem": [[2, "reformulating-the-problem"]], "Regression Case": [[10, "regression-case"]], "Regression analysis and resampling methods": [[25, "regression-analysis-and-resampling-methods"]], "Regression analysis, overarching aims": [[30, "regression-analysis-overarching-aims"]], "Regression analysis, overarching aims II": [[30, "regression-analysis-overarching-aims-ii"]], "Regularization": [[1, "regularization"]], "Relevance": [[36, "relevance"]], "Reminder from last week": [[31, "reminder-from-last-week"]], "Reminder on Newton-Raphson\u2019s method": [[32, "reminder-on-newton-raphson-s-method"]], "Reminder on Statistics": [[6, "reminder-on-statistics"]], "Reminder on books with hands-on material and codes": [[37, "reminder-on-books-with-hands-on-material-and-codes"]], "Reminder on different scaling methods": [[33, "reminder-on-different-scaling-methods"]], "Reminder on the chain rule and gradients": [[37, "reminder-on-the-chain-rule-and-gradients"]], "Replace or not": [[13, "replace-or-not"], [33, "replace-or-not"]], "Required Technologies": [[23, "required-technologies"]], "Resampling Methods": [[6, null]], "Resampling and the Bias-Variance Trade-off": [[19, "resampling-and-the-bias-variance-trade-off"]], "Resampling approaches can be computationally expensive": [[34, "resampling-approaches-can-be-computationally-expensive"], [35, "resampling-approaches-can-be-computationally-expensive"]], "Resampling methods": [[6, "id1"], [34, "resampling-methods"], [34, "id2"], [35, "resampling-methods"], [35, "id1"]], "Resampling methods: Bootstrap": [[34, "resampling-methods-bootstrap"], [35, "resampling-methods-bootstrap"]], "Resampling methods: Bootstrap approach": [[34, "resampling-methods-bootstrap-approach"]], "Resampling methods: Bootstrap background": [[34, "resampling-methods-bootstrap-background"]], "Resampling methods: Bootstrap steps": [[34, "resampling-methods-bootstrap-steps"]], "Resampling methods: More Bootstrap background": [[34, "resampling-methods-more-bootstrap-background"]], "Residual Error": [[31, "residual-error"], [32, "residual-error"]], "Resources on differential equations and deep learning": [[2, "resources-on-differential-equations-and-deep-learning"]], "Revisiting Ordinary Least Squares": [[32, "revisiting-ordinary-least-squares"]], "Revisiting our Linear Regression Solvers": [[13, "revisiting-our-linear-regression-solvers"]], "Revisiting our Logistic Regression case": [[35, "revisiting-our-logistic-regression-case"], [36, "revisiting-our-logistic-regression-case"]], "Rewriting the Covariance and/or Correlation Matrix": [[31, "rewriting-the-covariance-and-or-correlation-matrix"]], "Rewriting the \\delta-function": [[34, "rewriting-the-delta-function"]], "Rewriting the fitting procedure as a linear algebra problem": [[30, "rewriting-the-fitting-procedure-as-a-linear-algebra-problem"]], "Rewriting the fitting procedure as a linear algebra problem, more details": [[30, "rewriting-the-fitting-procedure-as-a-linear-algebra-problem-more-details"]], "Ridge Regression": [[32, "ridge-regression"]], "Ridge and LASSO Regression": [[31, "ridge-and-lasso-regression"], [32, "ridge-and-lasso-regression"], [32, "id2"]], "Ridge and Lasso Regression": [[5, null], [5, "id1"]], "SGD example": [[33, "sgd-example"]], "SGD vs Full-Batch GD: Convergence Speed and Memory Comparison": [[33, "sgd-vs-full-batch-gd-convergence-speed-and-memory-comparison"]], "SVD analysis": [[32, "svd-analysis"]], "Same code but now with momentum gradient descent": [[13, "same-code-but-now-with-momentum-gradient-descent"], [33, "same-code-but-now-with-momentum-gradient-descent"], [33, "id3"], [33, "id4"]], "Schedule first week": [[30, "schedule-first-week"]], "Schematic Regression Procedure": [[9, "schematic-regression-procedure"]], "Second moment of the gradient": [[33, "second-moment-of-the-gradient"]], "September 15-19": [[19, "september-15-19"]], "Setting up the Back propagation algorithm": [[12, "setting-up-the-back-propagation-algorithm"]], "Setting up the Back propagation algorithm, part 3": [[37, "setting-up-the-back-propagation-algorithm-part-3"]], "Setting up the Matrix to be inverted": [[31, "setting-up-the-matrix-to-be-inverted"], [32, "setting-up-the-matrix-to-be-inverted"]], "Setting up the back propagation algorithm": [[37, "setting-up-the-back-propagation-algorithm"]], "Setting up the back propagation algorithm, part 2": [[37, "setting-up-the-back-propagation-algorithm-part-2"]], "Setting up the equations for a neural network": [[37, "setting-up-the-equations-for-a-neural-network"]], "Setting up the network using Autograd; The full program": [[2, "setting-up-the-network-using-autograd-the-full-program"]], "Similar (second order function now) problem but now with AdaGrad": [[13, "similar-second-order-function-now-problem-but-now-with-adagrad"], [33, "similar-second-order-function-now-problem-but-now-with-adagrad"]], "Simple Python Code to read in Data and perform Classification": [[9, "simple-python-code-to-read-in-data-and-perform-classification"]], "Simple case": [[31, "simple-case"], [32, "simple-case"]], "Simple code for solving the above problem": [[32, "simple-code-for-solving-the-above-problem"]], "Simple example": [[35, "simple-example"], [37, "simple-example"]], "Simple example code": [[33, "simple-example-code"]], "Simple example to illustrate Ordinary Least Squares, Ridge and Lasso Regression": [[32, "simple-example-to-illustrate-ordinary-least-squares-ridge-and-lasso-regression"]], "Simple geometric interpretation": [[32, "simple-geometric-interpretation"]], "Simple linear regression model using scikit-learn": [[0, "simple-linear-regression-model-using-scikit-learn"], [30, "simple-linear-regression-model-using-scikit-learn"]], "Simple neural network and the back propagation equations": [[37, "simple-neural-network-and-the-back-propagation-equations"]], "Simple one-dimensional second-order polynomial": [[18, "simple-one-dimensional-second-order-polynomial"]], "Simple program": [[32, "simple-program"], [33, "simple-program"]], "Simpler examples first, and automatic differentiation": [[37, "simpler-examples-first-and-automatic-differentiation"]], "Slightly different approach": [[33, "slightly-different-approach"]], "Smarter way of evaluating the above function": [[37, "smarter-way-of-evaluating-the-above-function"]], "Sneaking in automatic differentiation using Autograd": [[33, "sneaking-in-automatic-differentiation-using-autograd"]], "Software and needed installations": [[25, "software-and-needed-installations"], [30, "software-and-needed-installations"]], "Solving Differential Equations with Deep Learning": [[2, null]], "Solving the one dimensional Poisson equation": [[2, "solving-the-one-dimensional-poisson-equation"]], "Solving the wave equation with Neural Networks": [[2, "solving-the-wave-equation-with-neural-networks"]], "Solving using Newton-Raphson\u2019s method": [[35, "solving-using-newton-raphson-s-method"], [36, "solving-using-newton-raphson-s-method"]], "Some famous Matrices": [[24, "some-famous-matrices"]], "Some parallels from real analysis": [[37, "some-parallels-from-real-analysis"]], "Some selected properties": [[35, "some-selected-properties"]], "Some simple problems": [[13, "some-simple-problems"], [32, "some-simple-problems"]], "Some useful matrix and vector expressions": [[31, "some-useful-matrix-and-vector-expressions"]], "Splitting our Data in Training and Test data": [[0, "splitting-our-data-in-training-and-test-data"], [31, "splitting-our-data-in-training-and-test-data"]], "Standard Approach based on the Normal Distribution": [[34, "standard-approach-based-on-the-normal-distribution"]], "Standard steepest descent": [[13, "standard-steepest-descent"]], "Statistical analysis": [[34, "statistical-analysis"], [35, "statistical-analysis"]], "Statistical analysis and optimization of data": [[23, "statistical-analysis-and-optimization-of-data"], [30, "statistical-analysis-and-optimization-of-data"]], "Steepest descent": [[13, "steepest-descent"], [32, "steepest-descent"]], "Stochastic Gradient Descent": [[33, "stochastic-gradient-descent"]], "Stochastic Gradient Descent (SGD)": [[13, "stochastic-gradient-descent-sgd"], [33, "stochastic-gradient-descent-sgd"]], "Stochastic variables and the main concepts, the discrete case": [[27, "stochastic-variables-and-the-main-concepts-the-discrete-case"]], "Strongly Convex Case": [[33, "strongly-convex-case"]], "Suggested readings and videos": [[36, "suggested-readings-and-videos"]], "Summing up": [[34, "summing-up"], [35, "summing-up"]], "Support Vector Machines, overarching aims": [[8, null]], "Synthetic data generation": [[35, "synthetic-data-generation"], [36, "synthetic-data-generation"]], "Systematic reduction": [[3, "systematic-reduction"]], "Teachers": [[30, "teachers"]], "Teachers and Grading": [[28, null]], "Teaching Assistants Fall semester 2023": [[28, "teaching-assistants-fall-semester-2023"]], "Tentative deadllines for projects": [[28, "tentative-deadllines-for-projects"]], "Testing the Means Squared Error as function of Complexity": [[0, "testing-the-means-squared-error-as-function-of-complexity"], [31, "testing-the-means-squared-error-as-function-of-complexity"]], "Textbooks": [[29, null]], "The Algorithm before theorem": [[11, "the-algorithm-before-theorem"]], "The Breast Cancer Data, now with Keras": [[1, "the-breast-cancer-data-now-with-keras"]], "The CART algorithm for Classification": [[9, "the-cart-algorithm-for-classification"]], "The CART algorithm for Regression": [[9, "the-cart-algorithm-for-regression"]], "The CIFAR01 data set": [[3, "the-cifar01-data-set"]], "The Central Limit Theorem": [[34, "the-central-limit-theorem"]], "The Hessian matrix": [[32, "the-hessian-matrix"], [33, "the-hessian-matrix"]], "The Hessian matrix for Ridge Regression": [[32, "the-hessian-matrix-for-ridge-regression"], [33, "the-hessian-matrix-for-ridge-regression"]], "The Jacobian": [[31, "the-jacobian"]], "The MNIST dataset again": [[3, "the-mnist-dataset-again"]], "The OLS case": [[32, "the-ols-case"]], "The RELU function family": [[1, "the-relu-function-family"]], "The Ridge case": [[32, "the-ridge-case"]], "The SVD, a Fantastic Algorithm": [[31, "the-svd-a-fantastic-algorithm"], [32, "the-svd-a-fantastic-algorithm"]], "The Softmax function": [[1, "the-softmax-function"]], "The \\chi^2 function": [[0, "the-chi-2-function"], [30, "the-chi-2-function"], [30, "id4"], [30, "id5"], [30, "id6"], [30, "id7"], [30, "id8"]], "The approximation theorem in words": [[37, "the-approximation-theorem-in-words"]], "The bias-variance tradeoff": [[6, "the-bias-variance-tradeoff"], [34, "the-bias-variance-tradeoff"], [35, "the-bias-variance-tradeoff"]], "The code for solving the ODE": [[2, "the-code-for-solving-the-ode"]], "The complete code with a simple data set": [[31, "the-complete-code-with-a-simple-data-set"]], "The cost function rewritten": [[35, "the-cost-function-rewritten"], [36, "the-cost-function-rewritten"]], "The cost/loss function": [[31, "the-cost-loss-function"]], "The course has two central parts": [[23, "the-course-has-two-central-parts"]], "The derivative of the cost/loss function": [[32, "the-derivative-of-the-cost-loss-function"], [33, "the-derivative-of-the-cost-loss-function"]], "The derivatives": [[37, "the-derivatives"]], "The equations": [[32, "the-equations"]], "The equations for ordinary least squares": [[31, "the-equations-for-ordinary-least-squares"]], "The equations to solve": [[35, "the-equations-to-solve"], [36, "the-equations-to-solve"]], "The first Case": [[32, "the-first-case"]], "The gradient step": [[33, "the-gradient-step"]], "The ideal": [[32, "the-ideal"]], "The logistic function": [[7, "the-logistic-function"], [35, "the-logistic-function"]], "The mean squared error and its derivative": [[31, "the-mean-squared-error-and-its-derivative"]], "The moons example": [[8, "the-moons-example"]], "The multilayer perceptron (MLP)": [[12, "the-multilayer-perceptron-mlp"]], "The network with one input layer, specified number of hidden layers, and one output layer": [[2, "the-network-with-one-input-layer-specified-number-of-hidden-layers-and-one-output-layer"]], "The optimization problem": [[37, "the-optimization-problem"]], "The ouput layer": [[37, "the-ouput-layer"]], "The plethora of machine learning algorithms/methods": [[30, "the-plethora-of-machine-learning-algorithms-methods"]], "The same example but now with cross-validation": [[34, "the-same-example-but-now-with-cross-validation"], [35, "the-same-example-but-now-with-cross-validation"]], "The sensitiveness of the gradient descent": [[32, "the-sensitiveness-of-the-gradient-descent"]], "The singular value decomposition": [[5, "the-singular-value-decomposition"], [31, "the-singular-value-decomposition"], [32, "the-singular-value-decomposition"]], "The training": [[37, "the-training"]], "The two-dimensional case": [[8, "the-two-dimensional-case"]], "Theoretical Convergence Speed and convex optimization": [[33, "theoretical-convergence-speed-and-convex-optimization"]], "Time decay rate": [[33, "time-decay-rate"]], "To our real data: nuclear binding energies. Brief reminder on masses and binding energies": [[30, "to-our-real-data-nuclear-binding-energies-brief-reminder-on-masses-and-binding-energies"]], "Topics covered in this course: Statistical analysis and optimization of data": [[30, "topics-covered-in-this-course-statistical-analysis-and-optimization-of-data"]], "Towards the PCA theorem": [[11, "towards-the-pca-theorem"]], "Train and test datasets": [[1, "train-and-test-datasets"]], "Two parameters": [[35, "two-parameters"], [36, "two-parameters"]], "Two-dimensional Objects": [[3, "two-dimensional-objects"]], "Type of problem": [[2, "type-of-problem"]], "Types of Machine Learning": [[30, "types-of-machine-learning"]], "Understanding what happens": [[34, "understanding-what-happens"], [35, "understanding-what-happens"]], "Universal approximation theorem": [[37, "universal-approximation-theorem"]], "Updating the gradients": [[37, "updating-the-gradients"]], "Use the books!": [[19, "use-the-books"]], "Useful Python libraries": [[23, "useful-python-libraries"], [30, "useful-python-libraries"]], "Using Autograd": [[13, "using-autograd"]], "Using Scikit-learn": [[36, "using-scikit-learn"]], "Using forward Euler to solve the ODE": [[2, "using-forward-euler-to-solve-the-ode"]], "Using gradient descent methods, limitations": [[13, "using-gradient-descent-methods-limitations"], [32, "using-gradient-descent-methods-limitations"], [33, "using-gradient-descent-methods-limitations"]], "Using the chain rule and summing over all k entries": [[37, "using-the-chain-rule-and-summing-over-all-k-entries"]], "Using the correlation matrix": [[36, "using-the-correlation-matrix"]], "Various steps in cross-validation": [[34, "various-steps-in-cross-validation"], [35, "various-steps-in-cross-validation"]], "Visualization": [[1, "visualization"], [1, "id1"]], "Visualizing the Tree, Classification": [[9, "visualizing-the-tree-classification"]], "Week 34: Introduction to the course, Logistics and Practicalities": [[30, null]], "Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression": [[31, null]], "Week 36: Linear Regression and Gradient descent": [[32, null]], "Week 37: Gradient descent methods": [[33, null]], "Week 38: Statistical analysis, bias-variance tradeoff and resampling methods": [[34, null]], "Week 39: Resampling methods and logistic regression": [[35, null]], "Week 40: Gradient descent methods (continued) and start Neural networks": [[36, null]], "Week 41 Neural networks and constructing a neural network code": [[37, null]], "What Is Generative Modeling?": [[30, "what-is-generative-modeling"]], "What does it mean?": [[31, "what-does-it-mean"], [32, "what-does-it-mean"]], "What is Machine Learning?": [[0, "what-is-machine-learning"]], "What is a good model?": [[0, "what-is-a-good-model"], [30, "what-is-a-good-model"]], "What is a good model? Can we define it?": [[30, "what-is-a-good-model-can-we-define-it"]], "When do we stop?": [[33, "when-do-we-stop"]], "Which activation function should I use?": [[1, "which-activation-function-should-i-use"]], "Why Combine Momentum and RMSProp?": [[33, "why-combine-momentum-and-rmsprop"]], "Why Linear Regression (aka Ordinary Least Squares and family)": [[30, "why-linear-regression-aka-ordinary-least-squares-and-family"]], "Why multilayer perceptrons?": [[36, "why-multilayer-perceptrons"], [37, "why-multilayer-perceptrons"]], "Why resampling methods": [[34, "why-resampling-methods"]], "Why resampling methods ?": [[34, "id1"], [35, "why-resampling-methods"]], "Wisconsin Cancer Data": [[7, "wisconsin-cancer-data"]], "With Lasso Regression": [[32, "with-lasso-regression"]], "Wrapping it up": [[34, "wrapping-it-up"]], "Writing Our First Generative Adversarial Network": [[4, "writing-our-first-generative-adversarial-network"]], "Writing our own PCA code": [[11, "writing-our-own-pca-code"]], "Writing the Cost Function": [[32, "writing-the-cost-function"]], "XGBoost: Extreme Gradient Boosting": [[10, "xgboost-extreme-gradient-boosting"]], "Yet another Example": [[32, "yet-another-example"]], "a) Expression for Ridge regression": [[17, "a-expression-for-ridge-regression"]], "scikit-learn implementation": [[1, "scikit-learn-implementation"]]}, "docnames": ["chapter1", "chapter10", "chapter11", "chapter12", "chapter13", "chapter2", "chapter3", "chapter4", "chapter5", "chapter6", "chapter7", "chapter8", "chapter9", "chapteroptimization", "clustering", "exercisesweek34", "exercisesweek35", "exercisesweek36", "exercisesweek37", "exercisesweek38", "exercisesweek39", "exercisesweek41", "exercisesweek42", "intro", "linalg", "project1", "schedule", "statistics", "teachers", "textbooks", "week34", "week35", "week36", "week37", "week38", "week39", "week40", "week41"], "envversion": {"sphinx": 62, "sphinx.domains.c": 3, "sphinx.domains.changeset": 1, "sphinx.domains.citation": 1, "sphinx.domains.cpp": 9, "sphinx.domains.index": 1, "sphinx.domains.javascript": 3, "sphinx.domains.math": 2, "sphinx.domains.python": 4, "sphinx.domains.rst": 2, "sphinx.domains.std": 2, "sphinx.ext.intersphinx": 1}, "filenames": ["chapter1.ipynb", "chapter10.ipynb", "chapter11.ipynb", "chapter12.ipynb", "chapter13.ipynb", "chapter2.ipynb", "chapter3.ipynb", "chapter4.ipynb", "chapter5.ipynb", "chapter6.ipynb", "chapter7.ipynb", "chapter8.ipynb", "chapter9.ipynb", "chapteroptimization.ipynb", "clustering.ipynb", "exercisesweek34.ipynb", "exercisesweek35.ipynb", "exercisesweek36.ipynb", "exercisesweek37.ipynb", "exercisesweek38.ipynb", "exercisesweek39.ipynb", "exercisesweek41.ipynb", "exercisesweek42.ipynb", "intro.md", "linalg.ipynb", "project1.ipynb", "schedule.md", "statistics.ipynb", "teachers.md", "textbooks.md", "week34.ipynb", "week35.ipynb", "week36.ipynb", "week37.ipynb", "week38.ipynb", "week39.ipynb", "week40.ipynb", "week41.ipynb"], "indexentries": {}, "objects": {}, "objnames": {}, "objtypes": {}, "terms": {"": [0, 1, 2, 3, 4, 5, 6, 7, 9, 11, 12, 13, 15, 16, 17, 19, 21, 22, 23, 24, 25, 27, 28, 30, 31, 37], "0": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 21, 22, 24, 25, 27, 28, 30, 31, 32, 33, 34, 35, 36, 37], "00": [0, 1, 5, 11, 30, 31, 37], "000": [1, 3], "000000": [], "00000000e": [], "001": [2, 8, 13, 21, 32, 33], "004": 5, "004113634617443131": 31, "004113634617443139": 31, "00411363461744314": 31, "004113634617443147": 31, "005b82": [], "00622f": [], "00727646693": [0, 30], "0072b2": [], "00749c": [], "0076268": 21, "008561": [], "0086649156": [0, 30], "00e0e0": [], "01": [0, 1, 2, 5, 9, 11, 13, 17, 29, 30, 31, 33, 35, 36, 37], "010726": [], "0110": 27, "01719003e": [], "02": [0, 4, 7, 12, 30, 35, 36], "02334824": [], "023b95": [], "024c1a": [], "02857": 4, "02f": 6, "03077640549": 4, "03097597e": [], "031": 5, "04": 11, "0458": 9, "05": [4, 6], "0550ae": [], "05767": 37, "062292565": 4, "062435": [], "06730814": [], "07": [], "0713": [0, 30], "07285": 3, "08": 27, "08078025e": [], "080808": [], "08336233266": 4, "08376632": 31, "083766322923899": 31, "0837663229239043": 31, "0917": 9, "0969da4a": [], "0d1117": [], "0n": [0, 30], "0x113e21950": 17, "1": [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 24, 26, 27, 28, 29, 30, 32, 33, 34, 35, 36], "10": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 16, 17, 18, 19, 21, 22, 24, 25, 26, 27, 28, 30, 31, 32, 33, 34, 35, 36], "100": [0, 1, 2, 3, 4, 5, 6, 8, 9, 10, 11, 13, 14, 15, 16, 17, 18, 19, 21, 24, 27, 28, 30, 31, 32, 33, 34, 35, 36, 37], "1000": [0, 1, 2, 4, 5, 8, 11, 13, 14, 18, 19, 21, 23, 27, 30, 32, 33, 35, 36], "10000": [2, 5, 6, 10, 11, 13, 27, 34], "100000": 8, "10001": 10, "1001": 27, "1002": 27, "1003": 27, "1005": 27, "1007": [34, 35], "1009": 27, "101": 16, "1011": 27, "1013": 27, "1013904243": 27, "1015": 27, "102": 16, "1023": 27, "1024": 3, "1026": 27, "1027": 27, "103": 1, "1030": 27, "1037": 27, "1038": 27, "1040": 27, "1047": 27, "107": 16, "108": [], "10th": 9, "10x": [0, 30], "11": [0, 2, 5, 6, 7, 8, 9, 10, 11, 12, 13, 16, 24, 25, 27, 29, 30, 31, 32, 33, 34, 35, 36, 37], "110": [], "1100": 27, "1101": 27, "111": [1, 7, 12, 35, 36, 37], "112": 16, "11340253": [], "11590451": [], "116": 16, "116329": [], "116633": [], "117": 16, "118": 16, "12": [0, 1, 2, 3, 4, 5, 6, 8, 9, 11, 12, 18, 21, 24, 25, 27, 29, 30, 31, 32, 33, 34, 36], "120": 3, "121": [8, 9, 10, 16], "1215pm": [28, 30], "122": [8, 9, 10], "124": [0, 30], "125": 16, "127": [4, 16], "128": [3, 4, 13, 33], "129": 16, "1298": 9, "12pm": [28, 30], "13": [0, 2, 9, 12, 22, 24, 27, 30, 36], "131": 16, "133": [7, 35], "135": 16, "136": 16, "14": [0, 2, 4, 6, 8, 9, 10, 12, 24, 27, 29, 31, 34, 35], "141": 16, "1412": 33, "141414": [], "143": 16, "1446729567": 4, "149": 16, "14g": [6, 34], "15": [0, 2, 4, 6, 7, 8, 9, 12, 13, 25, 27, 30, 32, 33, 35, 36], "150": [4, 8, 21, 35, 36], "1502": 37, "152": 16, "153760": [], "156": 16, "157": [], "158": [], "159": 16, "15g": [6, 34], "15pm": 30, "16": [1, 2, 3, 4, 5, 8, 9, 10, 21, 27, 30, 32, 34], "160": 16, "1603": 3, "161": 16, "162": 16, "16231451": 4, "163": 16, "16384": 3, "164": 16, "167": 16, "17": [1, 2, 8, 22, 27], "172": 16, "173": 16, "175": [34, 35], "176": 16, "178": 16, "179": 16, "1797": 1, "18": [2, 6, 7, 8, 9, 10, 27, 30, 34, 35], "1807": 4, "181036": [], "18392847": [], "18c1c4": [], "19": [2, 27, 30, 34], "192": [34, 35], "1940": [], "1943": [12, 36, 37], "19569961": 31, "19680801": [], "1970": [24, 30], "1973": 9, "1979": [6, 34], "1989": 37, "1991": 37, "1_1": [12, 36], "1_2": [12, 36], "1_3": [12, 36], "1cm": [0, 8, 10, 27, 30, 37], "1d": [1, 2, 3, 35, 36], "1e": [2, 4, 13, 14, 33, 35, 36], "1e10": 14, "1e1e1": [], "1e4": 6, "1f": 1, "1ffvbn0xlhv": 22, "1k": 24, "1n": [0, 30], "1x": [0, 30], "1zkibvqf": 21, "2": [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 23, 24, 25, 27, 29, 33, 34, 35, 36], "20": [0, 1, 2, 6, 7, 8, 16, 17, 27, 28, 30, 31, 32, 33, 34, 35, 36, 37], "200": [0, 2, 3, 4, 8, 9, 10, 35, 36], "2000": [0, 31], "2001": [], "2004": [13, 32], "2006": 29, "2007": [], "20072279": [], "2008": [30, 33], "2009": [], "2010": 1, "2011": [1, 33], "2012": 33, "2013": [], "2014": [4, 33], "2015": 1, "2016": [0, 30], "2018": [0, 6, 31, 34, 35], "2019": [], "2020": [], "2021": [6, 14, 31, 33], "2022": [30, 37], "2024": [21, 34], "2025": [18, 21, 22, 30, 31, 32, 33, 34], "21": [0, 1, 5, 7, 9, 12, 24, 30, 31, 32, 35, 36, 37], "2116753732": 4, "215pm": [28, 30], "2167072": [], "22": [0, 1, 5, 12, 13, 24, 30, 31, 32, 36], "221": 8, "225": 4, "22948497": [], "23": [1, 12, 24, 36], "24": [0, 1, 24, 30], "242424": [], "24292f": [], "25": [2, 3, 4, 5, 6, 8, 9, 11, 31], "250": [2, 4, 7, 9, 35], "25000": [], "250154": [], "252124": [], "253775": [], "255": 3, "256": [4, 33], "25x": 25, "26": [], "26303845": [], "264": [], "265": [], "265109911": 4, "266": [], "269": [], "27": 1, "270": [], "278": [32, 33], "27n_": 27, "28": [1, 3, 4], "283": [32, 33], "2830637392": 4, "2861": 27, "2873": 9, "2882": 27, "2886": 27, "2890": [0, 30], "2892": 27, "29": 31, "2915": 27, "2931": 30, "29364655": [], "294399745619595": [], "296247": [], "2968": 30, "2980": [21, 30], "298273": [], "298375": [], "2990": 30, "2_": [12, 36], "2_1": [12, 36], "2_2": [12, 36], "2_3": [12, 36], "2_i": [12, 36], "2_m": [6, 27, 34], "2_t": 13, "2_x": 27, "2a": 17, "2a1968": [], "2b": 27, "2b2b2b": [], "2c8f433990d1": 33, "2cm": 8, "2d": [1, 3, 11, 12, 23, 30, 35, 36, 37], "2e": [6, 34, 35], "2f": [0, 7, 9, 10, 11, 12, 30, 35, 36], "2g": 2, "2g_i": 2, "2k": 3, "2m": [6, 34], "2mvizaqfst8": 31, "2n": [0, 2, 3, 30, 31], "2nd": 9, "2p": [27, 37], "2pt": 4, "2x": [0, 3, 8, 13, 30, 37], "2x_ix_jy_iy_j": 8, "2x_j": 8, "2xb": 37, "2y_i": 10, "2y_j": 8, "3": [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 23, 24, 25, 26, 27, 28, 30, 32, 33, 34, 35, 36], "30": [0, 1, 4, 6, 7, 10, 13, 28, 33, 34, 35, 36], "300": [35, 36], "30000": [0, 30], "3072": 3, "31": [12, 24, 27, 36], "315": [6, 31, 33], "3155": [0, 5, 6, 31, 32, 33, 34, 35], "32": [3, 4, 6, 12, 13, 24, 27, 33, 36], "3200": 1, "3250": 1, "3297": [], "33": [12, 24, 28, 36], "3303": [], "3310": [], "332331": [], "333": [7, 35], "3331": [], "3337": [], "34": 24, "3436": [0, 30], "3437": [0, 30], "35": [0, 6, 25, 30, 32, 33], "3581341341": 4, "359": [5, 32], "36": [0, 5, 6, 18, 25, 27], "37": [25, 32, 34, 35], "370782966": 4, "38": [25, 27], "387": [34, 35], "39": [0, 25, 28, 30], "3d": [2, 3, 4, 6, 13, 16, 34, 35], "3d73a9": [], "3f": [1, 3, 9], "3n": 24, "3x": [2, 8], "3x_0x_1": 37, "3x_i": 2, "3y": 8, "4": [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 24, 25, 27, 30, 32, 33, 34, 35, 36, 37], "40": [1, 6, 28, 30, 34, 35], "400": 4, "4000": 30, "40008b9a5380fcacce3976bf7c08af5b": 33, "4050": [29, 30], "41": 24, "4155": [2, 15], "41589548": [], "42": [1, 4, 8, 9, 10, 24, 35, 36, 37], "43": [0, 7, 24], "4310": 30, "436462435": 4, "437a6b": [], "44": [0, 24, 32, 33], "45": [28, 30], "46": [28, 30], "462": [7, 35], "47": [28, 30], "473d18": [], "479465113": 4, "47958494": [], "48": [], "48257387": [28, 30], "49": [5, 6, 11], "49152": 3, "4940954": [0, 30], "4990": 27, "4992": 27, "4997": 27, "4c4b4be8": [], "4c4c7f": [9, 10], "4d": 3, "4f": [6, 35, 36], "4pm": [28, 30], "4y": 8, "4y_i": 10, "5": [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 17, 24, 25, 27, 30, 31, 32, 33, 34, 35, 36, 37], "50": [1, 2, 3, 4, 6, 7, 8, 10, 13, 30, 31, 33, 34, 36, 37], "500": [1, 3, 4, 6, 9, 10, 13, 33, 34, 35], "5000": 25, "5018": 27, "506": [], "507d50": [9, 10], "50j": 13, "50x10": 1, "51": 10, "510": 1, "512132": [], "515151": [], "5177783846": 4, "52": 35, "53": [9, 35], "5391cf": [], "54": [6, 27], "5411205": [], "54894451": [], "55": 1, "56": 1, "56469864": 21, "56536": [0, 30], "569": 1, "57": [0, 8, 28, 30], "571": [5, 32], "576": 34, "58": [10, 28, 30], "58a6ff70": [], "591317992": 4, "5ca7e4": [], "5cm": 27, "5f": [8, 33], "5x": [8, 18], "5y": 8, "6": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 12, 13, 18, 24, 27, 28, 30, 31, 32, 33, 34, 35, 36], "60": [1, 3], "60000": 4, "6019067271": 4, "60610368": 21, "606439": [], "622cbc": [], "625": [7, 35], "63": 1, "64": [1, 3, 4, 13, 24, 30, 33], "64x50": 1, "65": [1, 8, 9], "66666691": [], "66707b": [], "66ccee": [], "66e9ec": [], "6730c5": [], "6887363571": 4, "69": [16, 27], "69069n_": 27, "691": [], "6980": 33, "6e7681": [], "6e7781": [], "6f98b3": [], "6n_": 27, "7": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 11, 12, 13, 24, 25, 27, 29, 30, 31, 33, 34, 35, 36, 37], "70": [1, 7, 35], "702c00": [], "70653767": 4, "71": 1, "724": 3, "72f088": [], "73": [], "7304881": [], "737373": [], "75": [5, 6, 8, 11, 34], "76": [28, 30, 35], "765": [7, 35], "77": [28, 30], "7718": 9, "7782028952": 4, "77893972": [], "78": [], "797979": [], "7998f2": [], "79c0ff": [], "7d7d58": [9, 10], "7ee787": [], "7f4707": [], "8": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 12, 13, 14, 18, 19, 21, 24, 27, 28, 30, 32, 35, 36], "80": [0, 1, 5, 8, 17, 31], "800": [4, 7, 35], "8045e5": [], "81": 1, "815am": [28, 30], "81b19b": [], "8250df": [], "84858": [34, 35], "85": 1, "8702784034": 4, "8786ac": [], "88": 30, "8a4600": [], "8b949e": [], "8c8c8c": [], "8f": [6, 34, 35], "8g": [6, 34], "8n": 24, "8x8": 1, "9": [0, 1, 2, 4, 5, 6, 7, 8, 9, 11, 12, 13, 24, 27, 30, 33, 35, 36], "90": 1, "9040": 9, "91": [28, 30], "912583": [], "91cbff": [], "92": [28, 30], "93": 16, "931": [0, 30], "933": [5, 32], "937": 27, "938": 27, "939": [0, 27, 30], "94": 27, "95": [1, 11, 34], "953800": [], "954": 27, "955820c21e8b": 4, "9579870417283": 21, "96": [6, 34], "960": 27, "961": 27, "962": 27, "9649652536": 4, "96611194e": [], "974eb7": [], "978": [34, 35], "9780387310732": 29, "9780387848570": 29, "9781098134174": 30, "9781492032632": 29, "9781801819312": 30, "97898392": 31, "98": [0, 1, 16], "985": 27, "986": 27, "98661b": [], "989": 27, "9898ff": [9, 10], "99": [13, 16, 33, 34], "991": 27, "992": 27, "993": 27, "996": 5, "996b00": [], "999": [9, 27, 33], "999999": [], "9e86c8": [], "9e8741": [], "9f4e55": [], "9x": 6, "9y": 6, "A": [2, 3, 5, 6, 7, 10, 11, 12, 13, 15, 16, 19, 20, 23, 24, 25, 26, 27, 28, 29, 31, 32, 33, 37], "AND": 2, "AS": [], "AT": [], "And": [0, 3, 4, 5, 6, 9, 13, 20, 22, 23, 25, 27, 32], "As": [0, 1, 2, 3, 4, 5, 6, 8, 10, 12, 13, 15, 16, 24, 25, 27, 30, 31, 32, 33, 34, 35, 36, 37], "At": [0, 4, 6, 13, 20, 30, 33], "BE": [0, 30], "BUT": [], "BY": [], "Be": [2, 18, 23, 30], "Being": 13, "But": [0, 1, 2, 3, 5, 6, 9, 10, 16, 21, 27, 31, 34, 35], "By": [0, 3, 5, 6, 12, 13, 17, 19, 24, 30, 31, 32, 33, 34, 36], "FOR": [], "For": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 18, 19, 21, 22, 23, 24, 25, 27, 29, 30, 31, 32, 33, 34, 35, 36, 37], "IF": [6, 31, 33], "IN": 29, "If": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 15, 16, 18, 21, 22, 23, 24, 25, 27, 30, 31, 32, 33, 34, 35, 36, 37], "In": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 18, 19, 21, 22, 23, 24, 25, 27, 29, 30, 31, 32, 33, 34, 35, 36], "Ising": [5, 12, 31, 32, 36, 37], "It": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 18, 20, 21, 22, 23, 24, 25, 27, 30, 31, 32, 33, 34, 35, 36, 37], "Its": [1, 2, 4, 11], "NO": [], "NOT": [], "No": [6, 9, 30, 31, 33, 36], "Not": [0, 1, 5, 6, 31, 32, 33, 34, 36], "OF": [], "ON": [], "OR": 27, "Of": 27, "On": [0, 3, 25, 27, 28, 29, 30, 33, 34], "One": [0, 1, 3, 4, 5, 6, 7, 8, 11, 12, 13, 17, 27, 31, 32, 33, 34, 35, 36, 37], "Or": [0, 1, 6, 30], "SUCH": [], "Such": [0, 6, 12, 16, 27, 33, 34, 35, 36, 37], "THE": [], "TO": [], "That": [0, 5, 7, 10, 11, 12, 14, 25, 27, 30, 34, 35, 36, 37], "The": [4, 10, 13, 14, 16, 17, 18, 19, 20, 21, 22, 24, 25, 26, 27, 28, 29], "Then": [0, 1, 6, 8, 9, 10, 11, 12, 13, 14, 15, 16, 18, 19, 20, 21, 24, 30, 32, 33, 34, 37], "There": [0, 3, 4, 5, 6, 8, 9, 11, 12, 14, 15, 24, 25, 27, 28, 30, 31, 32, 33, 36, 37], "These": [0, 3, 4, 5, 8, 9, 10, 11, 12, 13, 14, 17, 18, 22, 24, 25, 27, 28, 30, 31, 32, 33, 37], "To": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 15, 16, 20, 21, 22, 24, 27, 31, 32, 33, 34, 35, 36, 37], "WITH": [], "Will": [35, 36], "With": [0, 5, 6, 8, 9, 10, 11, 12, 14, 16, 19, 21, 24, 25, 27, 30, 31, 34, 35, 36, 37], "_": [0, 1, 2, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 16, 17, 18, 19, 21, 24, 25, 30, 31, 32, 33, 34, 35, 36], "_0": [5, 8, 10, 11, 13, 31, 32], "_1": [2, 5, 6, 8, 10, 11, 12, 13, 14, 24, 31, 32, 33, 37], "_2": [2, 5, 8, 11, 12, 13, 24, 31, 33, 36], "_3": 24, "_4": 24, "_9": [13, 33], "__array_finalize__": [], "__class__": 10, "__doc__": [6, 34, 35], "__future__": [8, 9, 37], "__getattribute__": [], "__import__": [], "__init__": [1, 22, 35, 36], "__main__": 2, "__name__": [2, 10], "__new__": [], "__path__": [], "_add_intercept": [35, 36], "_auto1": [2, 3, 4, 5, 6, 7, 12, 13, 24, 27, 31, 32, 35, 36, 37], "_auto10": [6, 12], "_auto11": 6, "_auto12": 6, "_auto2": [2, 3, 4, 5, 6, 12, 13, 24, 27, 36, 37], "_auto3": [3, 4, 5, 6, 12, 13, 24, 36, 37], "_auto4": [4, 6, 12, 13, 24, 36], "_auto5": [4, 6, 12, 13, 24, 36], "_auto6": [4, 6, 12, 24, 36], "_auto7": [4, 6, 12, 24, 36], "_auto8": [6, 12], "_auto9": [6, 12], "_build": [0, 23, 25, 29, 30], "_c": 1, "_center": [], "_compile_transl": [], "_compon": 11, "_da": 22, "_data": [], "_depth": 9, "_export": [15, 16, 19], "_feed_forward_sav": 22, "_fraction": 9, "_i": [0, 1, 2, 5, 6, 7, 8, 11, 12, 13, 19, 25, 30, 31, 32, 33, 34, 35, 36, 37], "_j": [0, 1, 2, 3, 5, 6, 8, 13, 19, 25, 31, 32, 33, 34, 35], "_k": [13, 32, 33], "_l": [12, 36, 37], "_lambda": 6, "_leaf": 9, "_m": 10, "_mask": [], "_multilayer_perceptron": [], "_n": [2, 5, 8, 11, 13, 31, 32, 33], "_node": 9, "_norm": [], "_p": [5, 8, 31, 32], "_parse_numpydoc_see_also_sect": [], "_pydevd_bundl": [], "_ratio": 11, "_sampl": 9, "_sigmoid": [35, 36], "_softmax": [35, 36], "_split": [6, 9, 25], "_t": [13, 33], "_test": [6, 25], "_varianc": 11, "_weight": 9, "a0": 3, "a0111f": [], "a0faa0": [9, 10], "a1": [0, 21, 22, 30], "a11": [], "a12236": [], "a2": [0, 21, 22, 30], "a25e53": [], "a2bffc": [], "a3": [0, 30], "a4": [0, 30], "a5d6ff": [], "a_": [0, 1, 16, 24, 30, 31, 37], "a_0": [0, 30, 37], "a_1": 37, "a_1a": [0, 30], "a_2": 37, "a_2a": [0, 30], "a_3": [0, 30], "a_3a": [0, 30], "a_4": [0, 30], "a_4a": [0, 30], "a_h": 1, "a_i": [0, 1, 2, 12, 30, 37], "a_j": [1, 12, 37], "a_k": [0, 1, 12, 37], "aa": [], "aaa": [], "aaron": 29, "ab": [0, 2, 5, 13, 14, 30, 31, 33, 37], "ab6369": [], "ab_channel": [23, 36, 37], "abandon": 1, "abe338": [], "abid": 27, "abil": [0, 10], "abl": [0, 1, 4, 5, 6, 7, 10, 12, 13, 16, 18, 20, 21, 25, 31, 32, 33, 35, 36, 37], "about": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 15, 16, 19, 20, 22, 23, 24, 25, 28, 33, 34, 35, 36], "abov": [0, 1, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 21, 22, 24, 27, 29, 30, 31, 33, 34, 35, 36], "abovement": [6, 25, 30, 34, 35], "abscissa": [13, 32], "absent": 33, "absolut": [0, 2, 5, 6, 13, 30, 31, 32, 34, 35], "absorb": [31, 32], "abstract": [1, 33, 35], "abund": 33, "ac": [], "acc_bin": [35, 36], "acc_multi": [35, 36], "acceler": [13, 33], "accept": [0, 3, 6, 9, 21, 25, 31, 33], "access": [3, 11, 27, 30, 33], "accid": [4, 6, 34, 35], "accompani": [0, 30, 31], "accomplish": [8, 9, 13, 33], "accord": [0, 1, 2, 5, 6, 9, 12, 13, 14, 27, 30, 32, 33, 34, 36, 37], "accordingli": 11, "account": [0, 3, 5, 13, 15, 16, 20, 27, 30, 33], "accumul": [12, 13, 27, 33, 36, 37], "accur": [0, 3, 4, 6, 10, 13, 33, 34, 35], "accuraci": [0, 1, 3, 4, 5, 6, 7, 9, 10, 11, 12, 21, 30, 31, 32, 35, 36, 37], "accuracy_scor": [0, 1, 10, 21, 22, 30, 35, 36], "accuracy_score_numpi": 1, "acheiv": 21, "achiev": [0, 1, 5, 6, 8, 12, 24, 30, 33, 34, 35, 36, 37], "aco": 27, "acquaint": 23, "acquir": [1, 23, 30], "acr": [], "across": [1, 3, 6, 9, 17, 23, 30, 34], "act": [1, 3, 24, 33], "actic": 21, "action": 27, "activ": [0, 2, 3, 4, 9, 15, 22, 26, 28, 30, 33], "activation_d": 22, "activation_func": [21, 22], "activest": [], "actual": [0, 1, 4, 5, 6, 8, 11, 15, 16, 18, 21, 24, 27, 30, 31, 32, 33, 34], "ad": [1, 3, 4, 5, 8, 13, 15, 16, 24, 32, 33, 34, 35], "ada_clf": 10, "adaboostclassifi": 10, "adadelta": [13, 33], "adagrad": [25, 34, 37], "adam": [1, 3, 4, 21, 25, 30, 34, 37], "adap": 37, "adapt": [4, 6, 13, 17, 29, 32, 34, 35, 37], "add": [0, 1, 2, 3, 4, 5, 6, 8, 10, 11, 12, 15, 16, 17, 18, 20, 21, 27, 28, 30, 31, 32, 33, 34, 35, 36, 37], "add6ff": [], "add_": [], "add_subplot": [1, 7, 12, 14, 35, 36], "addendum": 5, "addeventlisten": [], "addit": [0, 2, 3, 5, 6, 7, 8, 9, 10, 12, 13, 15, 21, 23, 24, 25, 27, 28, 29, 30, 31, 34, 35, 36, 37], "addition": [12, 13, 32, 33, 36, 37], "address": [1, 9, 11, 13, 30, 33], "adjac": [3, 12, 36, 37], "adjoint": [5, 31], "adjust": [0, 5, 12, 13, 32, 33, 36], "admir": [0, 30], "advanc": [4, 6, 12, 29, 30, 33, 34, 35, 36, 37], "advantag": [1, 3, 5, 6, 10, 13, 19, 24, 32, 33, 34, 35], "adversari": 30, "advis": [], "afecionado": 30, "affect": [3, 15, 19], "affin": [0, 3, 8, 11, 31, 37], "afford": 3, "aficionado": 30, "aforement": 14, "african": [], "after": [0, 1, 2, 4, 5, 6, 9, 11, 12, 13, 15, 16, 17, 18, 19, 20, 21, 23, 24, 25, 27, 30, 31, 32, 33, 34, 37], "afterward": [0, 30], "ag": [0, 7, 30, 31, 35], "ag_0": 2, "again": [0, 1, 4, 5, 6, 7, 8, 10, 11, 12, 13, 25, 27, 30, 31, 32, 34, 35, 36, 37], "against": [1, 4, 7, 10, 35], "agegroup": [7, 35], "agegroupmean": [7, 35], "aggreg": [9, 10, 33], "agorithm": 10, "agre": [5, 6, 27, 31, 32, 33, 34], "agreement": [13, 33], "ahead": 9, "ai": [0, 29], "aid": [11, 20, 33], "aim": [0, 1, 4, 6, 7, 11, 14, 16, 17, 19, 20, 23, 24, 25, 31, 34, 35, 36, 37], "ainv": 5, "airplan": 3, "aka": 5, "al": [0, 2, 4, 16, 17, 20, 29, 30, 31, 32, 34, 35, 36, 37], "alarm": [5, 7], "aldo": 31, "algebra": [0, 3, 5, 13, 23, 31, 32, 34], "algorithm": [0, 1, 2, 4, 5, 6, 7, 8, 13, 14, 16, 23, 24, 25, 27, 29, 34, 35, 36], "align": [0, 2, 5, 6, 7, 8, 13, 27, 30, 31, 32, 34, 35, 36], "all": [0, 1, 2, 3, 4, 5, 6, 7, 9, 10, 11, 12, 13, 14, 15, 18, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36], "allclos": 21, "allevi": [1, 13, 32], "alloc": [3, 24], "allow": [0, 1, 2, 3, 5, 6, 8, 10, 13, 15, 23, 24, 25, 30, 31, 32, 33, 34, 35, 36, 37], "almost": [0, 1, 6, 8, 11, 13, 27, 32, 33, 34, 35, 36], "alon": [2, 9, 33], "along": [2, 3, 4, 5, 6, 9, 10, 11, 15, 20, 21, 22, 23, 24, 30, 31, 32, 34, 35], "alpha": [0, 1, 2, 3, 4, 6, 7, 8, 9, 10, 13, 14, 27, 30, 31, 32, 33, 34, 35, 36], "alpha_": [10, 33], "alpha_0": 3, "alpha_1": 3, "alpha_2": 3, "alpha_i": [3, 13], "alpha_k": 13, "alpha_m": 10, "alpha_n": 3, "alpha_opt": 13, "alreadi": [2, 3, 4, 5, 6, 10, 12, 15, 22, 23, 24, 27, 30, 31, 32, 35, 36, 37], "also": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 18, 19, 21, 22, 23, 24, 25, 27, 30, 31, 32, 33, 34, 35, 36, 37], "alter": 1, "altern": [0, 1, 4, 5, 6, 8, 9, 11, 13, 15, 18, 24, 25, 30, 31, 33, 34, 35], "although": [0, 1, 5, 6, 8, 10, 13, 16, 19, 20, 30, 33, 34, 35, 37], "alwai": [0, 3, 5, 6, 12, 13, 16, 19, 21, 22, 25, 27, 30, 31, 32, 33, 34, 36, 37], "am": 4, "ambit": 37, "ame2016": [0, 30], "american": [], "amjith": [], "among": [0, 3, 5, 9, 10, 12, 24, 30, 31, 36, 37], "amongst": [5, 34], "amount": [0, 1, 3, 4, 6, 8, 10, 14, 23, 34, 35, 37], "an": [1, 2, 3, 5, 6, 7, 8, 9, 11, 12, 13, 14, 16, 17, 18, 19, 21, 22, 23, 24, 25, 27, 28, 29, 31, 32, 33, 34, 35, 36], "an_": 27, "anaconda": [0, 1, 23, 25, 30], "analogi": 13, "analys": [6, 34, 35], "analysi": [1, 3, 4, 7, 14, 19, 24, 29, 33, 36], "analyt": [2, 3, 5, 6, 7, 12, 13, 17, 22, 23, 25, 30, 31, 32, 33, 34, 35, 36, 37], "analyz": [0, 1, 3, 4, 5, 6, 16, 25, 27, 31, 32, 33], "andrew": 1, "angl": [0, 3, 9, 31, 33], "anharmon": 3, "ani": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 12, 14, 15, 16, 19, 21, 27, 30, 31, 33, 34, 37], "anim": [4, 12, 36, 37], "ann": [12, 36, 37], "annot": [0, 1, 3, 7, 8, 30, 36], "announc": 30, "anom": [], "anomali": [], "anonym": 18, "anoth": [0, 1, 3, 4, 5, 6, 7, 8, 10, 11, 12, 13, 15, 24, 25, 27, 30, 31, 33, 37], "ansatz": [0, 18, 30], "answer": [0, 1, 3, 5, 6, 19, 22, 24, 25, 28, 30, 34], "antialias": [2, 6], "anticip": 4, "anymor": [1, 8], "anyon": [4, 8, 15], "anyth": [1, 15, 16, 21, 22, 27], "anytim": [28, 30], "anywai": [], "apach": 1, "apart": [11, 13, 32, 33], "api": [1, 23, 30], "appar": 2, "appear": [0, 1, 3, 13, 24, 27, 37], "append": [1, 3, 4, 8, 9, 13, 19, 21, 22, 30, 33, 35, 36], "appendic": 25, "appendix": 25, "appli": [0, 1, 3, 4, 6, 7, 8, 9, 10, 11, 12, 13, 18, 25, 27, 29, 30, 31, 33, 34, 35, 36, 37], "applic": [0, 1, 3, 4, 5, 6, 7, 9, 12, 13, 16, 24, 27, 29, 30, 31, 32, 33, 34, 35, 36, 37], "apply_gradi": 4, "approach": [1, 2, 4, 5, 6, 9, 10, 11, 12, 13, 15, 16, 18, 21, 23, 25, 27, 29, 31, 32, 37], "approch": 25, "appropri": [2, 6, 9, 12, 13, 17, 23, 27, 33, 34, 35, 36], "approv": 30, "approx": [0, 2, 3, 6, 10, 11, 13, 18, 25, 27, 30, 32, 33, 34], "approxim": [0, 1, 2, 3, 4, 5, 6, 7, 10, 11, 13, 19, 25, 27, 30, 31, 32, 33, 34, 35, 36], "apt": [0, 23, 25, 30], "aq": 27, "ar": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37], "aragorn": 30, "arang": [1, 3, 4, 6, 7, 9, 10, 12, 13, 30, 33, 35, 36], "arbitrari": [1, 4, 6, 8, 12, 13, 27, 32, 34, 36, 37], "arbitrarili": [0, 1, 11, 30, 33], "arc": 6, "architectur": [3, 4, 12, 37], "archiv": 25, "area": [0, 3, 6, 29, 30], "argmax": [1, 11, 21, 35, 36], "argmin": [4, 10, 14], "argsort": 11, "argu": [1, 13], "arguement": 19, "argument": [0, 2, 3, 5, 11, 12, 13, 17, 21, 30, 31, 33, 34, 36, 37], "aris": [0, 6, 12, 13, 27, 30, 32, 34, 35], "arithmet": [0, 13, 24, 30], "arm": [6, 31, 33], "armadillo": 24, "armin": [], "arnulf": 37, "around": [0, 1, 4, 5, 6, 11, 18, 21, 22, 25, 27, 30, 34, 35, 36, 37], "arrai": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 12, 13, 14, 16, 18, 21, 23, 25, 27, 31, 32, 33, 34, 35, 36, 37], "arrang": [3, 30], "array_equ": [35, 36], "arraybox": 13, "arriv": [0, 6, 9, 11, 19, 24, 27, 30, 34], "arrow": [12, 36, 37], "arrowprop": 8, "art": [0, 1, 23], "articl": [0, 3, 4, 6, 10, 19, 30, 31, 32, 33, 34, 35], "artifici": [0, 2, 7, 12, 29, 30, 35], "artificialneuron": [12, 36, 37], "arug": 13, "arxiv": [3, 4, 33, 37], "asarrai": [0, 6, 9, 31, 33], "asid": 31, "ask": [5, 6, 11, 12, 15, 19, 25, 34, 37], "aspect": [0, 6, 23, 30, 31, 37], "assembl": 3, "assembli": [0, 30], "assert": 4, "assess": [0, 6, 25, 30, 31, 34, 35], "asset": [], "assici": 4, "assign": [0, 7, 8, 9, 12, 13, 14, 15, 26, 28, 29, 30, 35, 36], "associ": [0, 6, 9, 12, 14, 27, 30, 34, 35, 36, 37], "assum": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 17, 19, 24, 25, 27, 30, 31, 32, 33, 34, 35, 36, 37], "assumpt": [0, 3, 5, 6, 9, 11, 27, 30, 31, 35], "ast": [0, 5, 6, 30, 34], "astyp": [4, 9, 10, 35, 36], "asymmetri": [0, 30], "asymptot": [4, 6, 33, 34, 35], "atom": [0, 30], "attain": 33, "attempt": [0, 4, 6, 7, 8, 10, 30, 31, 33, 35, 37], "attend": 30, "attent": [0, 24, 30], "attract": [0, 10, 30], "attribut": [0, 9, 22, 30], "audi": [0, 30], "audio": [3, 4], "august": [30, 31], "aurelien": [0, 29, 30], "austfjel": 6, "auth": 15, "authent": 15, "author": [0, 1, 10, 27], "authour": 30, "auto": [9, 10, 27], "auto_exampl": [21, 25, 31], "autocor": 27, "autocorrelation_tim": 27, "autocorrelform": 27, "autocovari": 27, "autoencod": [4, 23, 30], "autoencond": 23, "autograd": [21, 23, 30, 37], "autograd_compliant_predict": 22, "autograd_gradi": 22, "autograd_one_lay": 22, "autom": [0, 23, 29, 30], "automac": 24, "automag": 30, "automat": [0, 1, 2, 3, 4, 11, 16, 21, 22, 23, 24, 30, 36], "automobil": 3, "autonom": 4, "avail": [0, 1, 4, 6, 10, 11, 23, 24, 25, 26, 28, 29, 30, 34, 35], "avali": [20, 25], "averag": [0, 1, 3, 6, 9, 10, 13, 14, 27, 28, 30, 31, 34, 35], "avoid": [0, 4, 5, 6, 9, 11, 13, 18, 21, 24, 31, 33, 34, 35], "awai": [2, 3, 6, 31, 33, 37], "awar": [2, 10], "award": [28, 30], "ax": [0, 1, 2, 3, 4, 6, 7, 8, 9, 10, 11, 12, 13, 14, 20, 21, 24, 25, 30, 34, 35, 36], "axes3d": [2, 6, 13, 32, 33], "axes_grid1": 6, "axhlin": 8, "axi": [0, 1, 2, 3, 4, 6, 7, 8, 9, 10, 11, 12, 13, 14, 18, 21, 24, 27, 30, 31, 32, 33, 34, 35, 36, 37], "axiom": 5, "axvlin": [4, 8], "axvspan": 4, "b": [0, 1, 3, 4, 5, 6, 8, 9, 10, 12, 13, 14, 15, 16, 17, 19, 20, 21, 22, 27, 28, 30, 31, 32, 33, 34, 35, 36, 37], "b1": [8, 21, 22], "b19db4": [], "b1bac4": [], "b2": [8, 21, 22], "b3": 8, "b35900": [], "b89784": [], "b_": [0, 1, 24, 37], "b_0": [0, 37], "b_1": [0, 2, 12, 13, 33, 36, 37], "b_2": [0, 13, 37], "b_5": [13, 33], "b_g": [21, 22], "b_group": 9, "b_i": [0, 1, 2, 12, 30, 36, 37], "b_ia_": [0, 30], "b_ia_i": 0, "b_index": 9, "b_j": [1, 12, 36, 37], "b_k": [0, 1, 12, 13, 33, 36, 37], "b_m": [12, 36], "b_score": 9, "b_valu": 9, "ba": 33, "babcock": 30, "bach": 33, "bachelor": [26, 28], "back": [0, 3, 4, 5, 6, 8, 9, 10, 15, 16, 21, 24, 27, 30, 33], "backbon": 24, "backend": [1, 4], "background": [29, 30], "backprogag": 22, "backpropag": [1, 21, 33, 37], "backpropog": 22, "backslash": [], "backtrack": 9, "backup": 24, "backward": [1, 2, 4, 12, 22, 24, 33, 37], "bad": [6, 17, 31], "badli": 27, "bag": [9, 23, 30], "bag_clf": 10, "baggin": 30, "baggingboot": 10, "baggingclassifi": 10, "baggingtre": 10, "bailei": [], "balanc": [6, 33, 34, 35], "ballpark": 18, "band": 24, "bandwidth": 24, "banner": [], "bar": [0, 6, 11, 25, 30], "barber": 29, "bare": [4, 10], "base": [0, 1, 3, 4, 5, 7, 8, 9, 10, 14, 15, 16, 17, 23, 27, 28, 29, 30, 31, 32, 35, 36, 37], "basi": [5, 7, 8, 10, 11, 12, 13, 24, 31, 32, 35, 36, 37], "basic": [6, 8, 12, 13, 14, 15, 23, 25, 27, 30, 34], "basin": 33, "batch": [3, 4, 11, 12, 13, 21, 32, 35, 36], "batch_shap": 4, "batch_siz": [1, 3, 4], "batchnorm": 4, "bay": [7, 35, 36], "baydin": 37, "bayesian": [5, 23, 29, 30], "bbbbbb": [], "beauti": [], "becam": [], "becaus": [0, 1, 2, 3, 4, 5, 6, 8, 9, 12, 13, 14, 30, 31, 32, 33, 34, 35, 36], "becom": [0, 1, 2, 5, 6, 7, 9, 12, 13, 19, 27, 30, 31, 32, 33, 34, 35, 36, 37], "been": [0, 1, 2, 3, 4, 5, 6, 11, 12, 13, 19, 20, 23, 24, 25, 30, 31, 33, 34, 36, 37], "befor": [0, 1, 2, 3, 4, 5, 6, 7, 8, 12, 13, 14, 16, 17, 18, 19, 20, 21, 22, 24, 25, 27, 30, 31, 33, 34, 35, 36, 37], "beforehand": [0, 27, 30], "began": [], "begin": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 11, 12, 13, 14, 15, 22, 24, 27, 28, 30, 31, 32, 33, 34, 35, 36, 37], "behav": [1, 6, 13, 32, 34, 35], "behavior": [0, 1, 13, 30, 32, 33], "behaviour": [12, 33, 36, 37], "behind": [0, 1, 6, 8, 13, 30, 32], "being": [0, 1, 2, 3, 4, 5, 7, 8, 10, 11, 12, 13, 17, 20, 27, 30, 31, 32, 33, 35, 36, 37], "believ": [9, 24], "belong": [7, 8, 9, 13, 14, 32, 35, 36], "below": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 15, 18, 21, 22, 24, 25, 27, 30, 31, 32, 33, 34, 35, 36, 37], "benchmark": 10, "benefici": [1, 13], "benefit": [0, 1, 4, 11, 13, 23, 30, 32, 33], "bengio": [1, 29, 30, 31, 33], "benign": [1, 7, 36], "benno": 37, "berner": 37, "besid": [4, 5, 32], "bessel": [5, 31, 34], "best": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 12, 13, 15, 16, 18, 21, 28, 30, 31, 32, 33, 34, 35, 36], "beta": [1, 3, 10, 11, 13, 16, 17, 19, 30, 31, 32], "beta1": [], "beta2": [], "beta_": [3, 13, 17, 31], "beta_0": [1, 3, 13, 31], "beta_1": [1, 3, 10, 13, 31, 33], "beta_1m_": 33, "beta_1x_i": 13, "beta_2": [3, 13, 33], "beta_2v_": 33, "beta_3": 3, "beta_i": [3, 33], "beta_j": [13, 31], "beta_k": 13, "beta_linreg": 13, "beta_m": 10, "beta_mg_m": 10, "beta_n": 3, "better": [0, 1, 2, 3, 4, 6, 9, 10, 11, 12, 13, 19, 20, 22, 30, 31, 33, 34], "between": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 11, 12, 13, 14, 15, 16, 17, 18, 19, 22, 25, 27, 30, 31, 32, 33, 34, 35, 36, 37], "beyond": [0, 1, 5, 6, 8, 13, 30, 31, 32, 33], "bf": [13, 14, 24, 27, 32], "bf5400": [], "bg": 30, "bgd": [13, 33], "bia": [0, 1, 2, 3, 5, 8, 9, 10, 12, 13, 20, 21, 22, 30, 31, 32, 36, 37], "bias": [1, 2, 3, 5, 6, 9, 12, 19, 21, 22, 33, 34, 36], "bib": [], "bibliographi": 25, "bibtex": [], "big": [0, 1, 2, 5, 6, 14, 19, 33, 34], "bigger": [1, 6, 31], "bigr": [12, 36], "bike": 9, "bilbo": 30, "billion": [3, 12, 23, 33, 36, 37], "bin": [7, 27, 36], "binari": [0, 3, 5, 7, 9, 10, 12, 30, 35, 36], "binary_cross_entropi": [35, 36], "binary_result": [35, 36], "binarycrossentropi": 4, "bind": 0, "binomi": [23, 27, 30], "binsboot": [6, 34], "bioinformat": 0, "biolog": [1, 12, 36, 37], "bios1100": [23, 30], "bird": [0, 3], "birth": 30, "bishop": [29, 30], "bit": [1, 4, 19, 21, 24, 27, 30], "bitwis": 27, "bivari": 2, "bk": [13, 33], "bla": [24, 30], "black": [8, 9, 14], "blame": [], "block": [6, 10, 23, 24, 27, 30, 34, 35], "blockquot": [], "blog": 30, "blogpost": 4, "blue": [0, 3], "bm": [], "bmatrix": [0, 1, 3, 5, 7, 8, 11, 13, 24, 30, 31, 32, 33, 35, 36, 37], "bmi": 1, "bodi": [0, 1, 4, 12, 36, 37], "bold": 1, "boldfac": [0, 5, 16, 31, 32], "boldsymbol": [0, 1, 2, 3, 5, 6, 7, 8, 10, 11, 13, 14, 16, 17, 19, 25, 30, 32, 33, 35, 36, 37], "boltzmann": [12, 23, 30, 36, 37], "book": [17, 25, 29, 30, 31, 34, 35], "book1": 29, "bool": [], "boolean": [4, 17], "boost": [1, 9, 23, 30], "boostrap": 10, "bootstrap": [1, 13, 19, 23, 25, 30, 33], "born": 33, "borrow": 30, "boston_dataset": [], "bot": 8, "both": [0, 1, 4, 5, 6, 8, 9, 10, 13, 14, 15, 16, 17, 19, 23, 24, 25, 27, 28, 30, 31, 32, 33, 34, 35, 36], "bottl": [7, 35, 36], "bottou": 33, "bound": [8, 12, 33, 36, 37], "boundari": [2, 4, 8, 11, 12], "bousquet": 33, "bower": [], "box": [4, 9, 21, 22], "boyd": [8, 13, 32], "bracket": [4, 27], "brain": [1, 7, 12, 35, 36, 37], "branch": [9, 30], "break": [0, 4, 6, 11, 14, 30, 33], "breast": [5, 7, 11, 36], "breviti": 13, "brew": [0, 23, 25, 30], "brg": 8, "brian": [], "brief": [25, 31], "briefli": [0, 16, 19, 30, 34], "bring": [0, 5, 6, 10, 31, 33], "britt": [28, 30], "broad": 0, "broadcast": 21, "broadli": 30, "brought": [13, 23, 30], "brownle": 4, "browser": [15, 30], "brute": [3, 5, 11, 31, 37], "bsd": [], "budget": 33, "buffer_s": 4, "bug": [], "bugfix": [], "bui": 4, "build": [0, 4, 5, 6, 10, 16, 22, 24, 27, 30, 34, 35, 36, 37], "built": [1, 3, 4, 6, 34, 35], "bunch": 11, "bundl": [], "busi": [], "bxe2t": [36, 37], "byte": [24, 30], "c": [0, 1, 2, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 19, 20, 21, 22, 23, 24, 26, 27, 28, 29, 31, 32, 33, 34, 35, 36, 37], "c1": [8, 11], "c2": [8, 11], "c4a2f5": [], "c5e478": [], "c9d1d9": [], "c_": [8, 9, 10, 13, 27, 32, 33], "c_0": 27, "c_1": [12, 36], "c_2": [12, 36], "c_3": [12, 36], "c_4": [12, 36], "c_i": [12, 13, 33, 36], "c_k": 27, "ca": [1, 30], "caab6d": [], "cach": 10, "cal": [0, 8, 10, 12, 13, 32, 33, 37], "calcul": [0, 1, 2, 4, 5, 6, 8, 9, 10, 11, 12, 13, 14, 16, 19, 22, 24, 27, 30, 33, 34, 35, 36, 37], "california": 25, "call": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 18, 19, 21, 23, 24, 25, 27, 28, 30, 31, 32, 33, 34, 35, 36, 37], "calor": [0, 31], "caltech": [], "cambridg": [13, 29, 32, 37], "can": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 27, 28, 29, 31, 32, 36], "cancel": [0, 13, 30, 31], "cancer": [5, 10, 36], "cancerpd": [7, 36], "candid": [8, 9, 10, 33], "cannot": [0, 1, 4, 5, 6, 7, 8, 9, 25, 27, 31, 32, 33, 36], "canopi": [0, 23, 25, 30], "canva": [15, 16, 19, 20, 25, 30], "cap": 5, "capabl": [0, 1, 8, 13, 23, 30], "capac": [2, 28], "capita": [], "caption": [20, 25], "captur": [4, 11, 12, 30, 36, 37], "car": [3, 4], "card": [0, 7, 30, 35, 36], "cardin": 1, "care": [11, 15, 19, 22, 33], "carefulli": [13, 33], "carlo": [0, 6, 23, 27, 29, 30, 34, 35], "carri": [2, 6, 7, 25, 34, 35, 36], "cart": 10, "case": [0, 1, 2, 3, 4, 5, 6, 7, 11, 12, 13, 14, 15, 16, 23, 24, 25, 30, 34, 37], "casella": 29, "cast": 1, "cat": [3, 4], "catch": 0, "categor": [0, 1, 3, 9, 11, 30, 35, 36], "categori": [0, 1, 3, 7, 10, 12, 14, 30, 35, 36, 37], "categorical_cross_entropi": [35, 36], "categorical_crossentropi": [1, 3], "caus": [0, 5, 6, 27, 30, 31, 32, 33, 34, 35], "causal": 0, "causat": [0, 30], "cax": 1, "cb": [6, 30], "cbar": 1, "cc": [0, 1, 5, 13, 30, 31, 32, 33, 37], "cc398b": [], "ccbb44": [], "ccc": [5, 12, 32, 36], "cdf": 27, "cdot": [0, 2, 6, 12, 13, 14, 24, 27, 30, 32, 33, 34, 36], "celebr": [13, 32], "cell": [4, 21, 22], "center": [0, 1, 6, 7, 8, 9, 11, 14, 18, 25, 27, 30, 31, 33, 34, 35, 36], "central": [0, 3, 5, 6, 8, 16, 20, 24, 30, 31, 37], "centroid": [14, 27], "centroid_differ": 14, "centuri": 3, "certain": [0, 3, 6, 7, 9, 21, 27, 30, 31, 34, 35, 36], "certainti": 34, "cf": [], "cf222e": [], "cffi": [], "cg": 13, "cha": [], "chain": [0, 1, 13, 22, 23, 27, 30], "challeng": [15, 37], "chanc": [1, 5, 13, 27, 33], "chang": [0, 1, 2, 3, 4, 5, 6, 8, 9, 11, 12, 13, 14, 15, 16, 19, 21, 22, 24, 25, 27, 30, 31, 32, 33, 34, 35, 36, 37], "changelog": [], "channel": 3, "chap4": 37, "chapter": [0, 6, 10, 11, 16, 17, 19, 24, 25, 29, 30, 31, 32, 33, 34, 35, 36, 37], "chapter3": [0, 25], "charact": [0, 3, 5, 30, 31, 32], "character": [8, 9, 10, 12, 27, 36], "characterist": [0, 1, 3, 10, 13, 30], "charg": [0, 30], "charl": [], "charset": [], "chase": 4, "chatgpt": [15, 25], "chd": [7, 35], "chddata": [7, 35], "cheap": [5, 31, 32, 33], "cheaper": [1, 13, 33], "check": [1, 3, 4, 5, 11, 13, 15, 16, 19, 21, 22, 24, 30, 33, 35, 36], "checkmark": 3, "checkpoint": 4, "checkpoint_dir": 4, "checkpoint_prefix": 4, "chen": 10, "cheng": 31, "chiaramont": 2, "childcar": 16, "children": 16, "choic": [0, 1, 2, 3, 4, 6, 9, 12, 13, 14, 20, 24, 30, 31, 32, 33, 34, 35, 36], "choleski": [5, 24, 31, 32], "choos": [2, 3, 6, 9, 10, 11, 13, 14, 15, 18, 19, 21, 25, 32, 34, 35, 36], "chosen": [0, 1, 2, 6, 8, 9, 10, 13, 16, 27, 30, 32, 33, 34, 35], "chosen_datapoint": 1, "christian": 29, "christoph": [29, 30], "chunk": 33, "cifar": 3, "cifar10": 3, "circ": [1, 12, 33, 37], "circl": [0, 8, 12, 31, 33, 36, 37], "circuit": 3, "circumfer": 9, "circumv": [1, 5, 13, 31, 32, 33], "citat": [], "cite": [20, 25], "ckpt": 4, "cl": [35, 36], "claim": [], "clarifi": 21, "clariti": 27, "class": [0, 1, 3, 4, 6, 7, 8, 9, 11, 12, 13, 21, 22, 27, 30, 34], "class0": [35, 36], "class1": [35, 36], "class_nam": [3, 9], "class_to_index": [35, 36], "class_val": 9, "class_valu": 9, "classic": [7, 9, 13, 36], "classif": [0, 3, 5, 6, 7, 8, 11, 12, 21, 23, 25, 29, 30, 31, 34], "classifi": [0, 1, 4, 7, 9, 10, 11, 30, 36], "classificaton": 1, "classifii": 10, "claus": [], "clean": 1, "clear": [1, 5, 10, 12, 13, 33], "clearli": [0, 3, 5, 6, 7, 8, 27, 31, 32, 34, 35, 36], "clever": [1, 10], "clf": [0, 6, 8, 9, 10, 30, 31], "clf3": 0, "clf_lasso": 6, "clf_ridg": 6, "cli": 15, "click": [], "clip": [3, 27, 33, 35, 36], "clock": 33, "clone": [15, 28], "close": [0, 1, 2, 4, 6, 8, 9, 11, 12, 13, 14, 18, 27, 29, 30, 32, 33, 34, 36, 37], "closer": [3, 5, 13, 31, 32, 33], "closest": [8, 11, 13, 14], "closur": [23, 30], "cloud": [23, 30], "cluster": [0, 1, 4, 6, 11, 23, 30, 34, 35, 36], "cluster_label": 14, "cm": [1, 2, 3, 6, 8, 13, 32, 33], "cmap": [0, 1, 2, 3, 4, 6, 8, 9, 10, 30], "cmap_arg": 6, "cmd": [9, 15], "cn_": 27, "cnn": [12, 36, 37], "cnn_kera": 3, "cntk": [23, 30], "co": [0, 2, 3, 6, 9, 13, 30, 34, 35], "code": [0, 3, 4, 6, 7, 8, 18, 19, 21, 22, 23, 24, 27, 29], "codec": [], "coef": [0, 30], "coef0": 8, "coef_": [0, 5, 6, 8, 9, 13, 16, 30, 31, 32, 33], "coeff": 5, "coeffici": [0, 3, 5, 6, 7, 8, 9, 13, 18, 24, 30, 31, 33, 34, 35, 36], "coerc": [0, 6, 30, 34, 35], "coin": [10, 27], "coin_toss": 10, "col": [0, 11, 30, 31], "colab": [21, 22, 23, 30], "cold": 9, "colinear": [], "collabor": [20, 25], "collaps": 8, "collect": [2, 6, 10, 11, 17, 23, 27, 29, 30, 34, 35, 37], "collinear": [5, 31, 32], "color": [0, 3, 4, 6, 8, 9, 10, 27, 33], "color_channel": 3, "color_cod": 6, "colorbar": [1, 6, 20], "colsample_bytre": 10, "colsaobject": 10, "column": [0, 1, 2, 5, 6, 7, 8, 9, 11, 12, 16, 17, 18, 19, 24, 30, 31, 32, 33, 34, 35, 36, 37], "columntransform": 9, "com": [4, 6, 15, 16, 19, 20, 21, 22, 23, 25, 29, 30, 32, 33, 34, 35, 36, 37], "combin": [1, 2, 5, 6, 7, 10, 15, 18, 22, 27, 34, 35], "come": [0, 1, 3, 4, 5, 12, 13, 14, 15, 30, 31, 32, 33, 36, 37], "comfort": [], "command": [0, 1, 15], "comment": [0, 4, 5, 6, 20, 25], "commerci": [0, 23, 25, 30], "commit": 15, "commod": [0, 30], "common": [0, 1, 3, 5, 6, 7, 9, 11, 13, 14, 16, 25, 27, 30, 31, 32, 33, 34, 35, 36, 37], "commonli": [0, 1, 4, 6, 7, 9, 13, 14, 31, 33, 34, 35, 36], "commonmark": [], "commun": [0, 12, 15, 25, 36, 37], "commut": 3, "commutatitav": 3, "compact": [0, 1, 3, 5, 6, 7, 9, 11, 12, 13, 14, 21, 30, 31, 34], "compair": 0, "compar": [0, 3, 4, 5, 6, 11, 13, 18, 24, 25, 30, 31, 32, 33, 34, 35, 37], "comparison": [2, 4, 13], "compat": [7, 35, 36], "compens": 33, "compet": 0, "competit": 10, "compil": [0, 1, 3, 4, 13, 23, 24, 30], "compl": 21, "complet": [0, 2, 3, 4, 9, 12, 15, 16, 17, 18, 19, 20, 21, 22, 30, 36], "completenn": [12, 36], "complex": [1, 5, 8, 9, 11, 12, 13, 16, 19, 30, 32, 33, 34, 35], "complianc": [], "complic": [0, 1, 9, 13, 25, 30, 32, 33, 34, 35], "compoment": 31, "compon": [0, 1, 3, 4, 5, 6, 7, 9, 14, 16, 23, 30, 31, 32, 34, 36, 37], "components_": 11, "compos": [9, 12, 13, 14, 23, 30, 36, 37], "compphys": [0, 6, 16, 20, 23, 25, 26, 28, 29, 30, 31, 32, 35, 36], "compress": [0, 30, 31], "compris": 6, "compromis": [5, 31, 32], "compulsori": [23, 30], "comput": [0, 1, 2, 3, 4, 5, 6, 7, 8, 10, 11, 12, 13, 15, 16, 17, 18, 21, 22, 23, 24, 25, 26, 27, 29, 30, 31, 32, 34, 35, 36, 37], "computation": [0, 3, 6, 9, 13, 27, 30, 32, 33, 37], "computationalscienceuio": 30, "compute_gradi": 22, "computerlab": 25, "concaten": [2, 4, 6, 14, 35, 36], "concav": [1, 13, 31, 32], "concentr": 10, "concept": [0, 2, 23, 30, 31], "conceptu": [12, 13, 32, 36, 37], "concern": [0, 1, 4, 7, 30, 32, 35, 36], "concic": 30, "conclud": [0, 5, 13, 33], "conclus": 1, "cond": 2, "conda": [0, 1, 23, 25, 30], "condis": 31, "condit": [0, 2, 4, 5, 6, 8, 9, 11, 13, 27, 30, 31, 33, 34], "conduct": 23, "condwav": 2, "confid": [0, 5, 6, 7, 8, 19, 30, 31, 35, 36], "configur": 3, "confirm": [5, 12, 21, 36], "conform": [], "confus": [5, 6, 7, 10, 24, 31, 34], "confusion_matrix": 9, "congruenti": 27, "conjug": [4, 8], "conjugaci": 13, "conjunct": 3, "connect": [0, 1, 3, 4, 9, 11, 12, 13, 24, 30, 31, 32, 36, 37], "consensu": 33, "consequ": [5, 6, 8, 10, 12, 13, 31, 32, 33, 34], "consequenti": [], "conserv": [5, 14, 31, 32], "consid": [0, 1, 2, 3, 5, 6, 7, 8, 9, 10, 12, 13, 16, 19, 24, 25, 27, 30, 31, 32, 33, 34, 35, 36, 37], "consider": [0, 1, 5, 13, 30, 31, 32, 34], "consist": [1, 2, 3, 4, 6, 12, 13, 25, 27, 31, 32, 34, 35, 36, 37], "consol": [], "const": [], "constant": [0, 2, 4, 5, 6, 8, 12, 13, 16, 18, 27, 30, 31, 32, 33, 36, 37], "constitu": [0, 30], "constitut": [2, 6, 34, 35], "constrain": [1, 3, 5, 7, 11, 32, 35], "constraint": [5, 6, 8, 13, 31, 32, 34], "construct": [0, 1, 2, 3, 5, 6, 7, 8, 9, 10, 11, 24, 27, 30, 31, 34, 36], "constructor": [], "consum": 33, "contact": [0, 30], "contain": [0, 2, 3, 4, 5, 6, 7, 8, 9, 11, 12, 13, 15, 18, 19, 21, 24, 25, 27, 29, 30, 31, 32, 33, 34, 35, 36, 37], "contemporari": 30, "content": [1, 15, 20, 23, 24, 30, 32, 33], "context": [6, 10, 13, 22, 25, 32, 33, 34, 35, 37], "contigu": 24, "contin": 19, "continu": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 12, 13, 19, 24, 25, 27, 30, 31, 32, 33, 34, 35, 37], "contour": [9, 10, 13], "contourf": [8, 9, 10], "contract": [], "contrast": [1, 4, 9, 10, 12, 30, 33, 36, 37], "contribut": [0, 3, 5, 13, 18, 27, 30, 31, 32, 33], "contributor": [0, 25], "control": [0, 1, 3, 9, 13, 15, 23, 30], "conv": [3, 4], "conv2d": [3, 4], "conv2dtranspos": 4, "convei": 30, "conveni": [5, 6, 12, 13, 24, 25, 30, 32, 33, 34, 36], "convent": [12, 31], "converg": [1, 2, 4, 5, 8, 13, 14, 18, 31, 32, 37], "convergencewarn": [], "convers": [20, 33], "convert": [0, 1, 4, 5, 9, 11, 13, 24, 30, 31, 32, 35, 36], "converttomatrix": 4, "convex": [4, 5, 7, 31, 35, 36], "convinc": [13, 32], "convolut": [1, 4, 23, 30], "cool": [4, 9], "coolwarm": 6, "coordin": [5, 12, 14, 31, 32, 33, 36], "coorel": [], "copi": [0, 1, 14, 15, 31, 35, 36], "copyright": [], "core": 10, "corel": 30, "coronari": [7, 35], "corr": [5, 7, 11, 31, 36], "correalt": [11, 23], "correct": [0, 1, 2, 3, 4, 5, 7, 13, 15, 19, 20, 21, 22, 24, 27, 30, 31, 32, 34, 35, 36], "correctli": [1, 2, 6, 7, 10, 18, 19, 21, 22, 25, 34, 35], "correl": [0, 1, 3, 5, 6, 7, 10, 12, 13, 23, 27, 30, 32, 33, 34, 37], "correlation_matrix": [5, 7, 11, 31, 36], "correspond": [0, 3, 5, 6, 8, 9, 11, 12, 23, 24, 25, 27, 30, 31, 32, 34, 36, 37], "cortex": [12, 36, 37], "cosin": [3, 6, 34, 35], "cost": [0, 2, 3, 5, 6, 7, 8, 9, 12, 13, 16, 17, 18, 19, 21, 22, 25, 30], "cost_autograd": 22, "cost_deep_grad": 2, "cost_der": 22, "cost_fun": 22, "cost_funct": 2, "cost_function_deep": 2, "cost_function_deep_grad": 2, "cost_function_grad": 2, "cost_grad": [2, 22], "cost_histori": [], "cost_ol": [], "cost_one_lay": 22, "cost_ridg": [], "cost_sum": 2, "cost_two_lay": 22, "costli": 33, "costol": [13, 33], "could": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 15, 16, 17, 18, 24, 25, 27, 30, 31, 32, 33, 34, 35, 36, 37], "coulomb": [0, 30], "count": [0, 9, 15, 25, 26, 27, 28, 30], "counter": 25, "counteract": 33, "counterpart": 30, "countor": 13, "coupl": [4, 5, 6, 21, 34], "cours": [0, 1, 3, 5, 11, 15, 16, 17, 19, 20, 21, 25, 28, 31, 34, 35], "coursework": 15, "courvil": [29, 30, 31, 33], "cov": [5, 6, 11, 24, 27, 30, 31, 34], "cov_xi": [5, 11, 31], "cov_xx": [5, 11, 31], "cov_yi": [5, 11, 31], "covari": [0, 7, 23, 24, 30, 32, 36], "covariance_matrix": [5, 11, 14], "cover": [0, 5, 23, 25, 28, 29, 31, 32, 34], "covert": [0, 30], "covxi": 27, "covxx": 27, "covxz": 27, "covyi": 27, "covyz": 27, "covzz": 27, "cpu": 1, "cqofi41lfdw": 37, "craft": 3, "crash": 33, "creat": [1, 3, 4, 5, 9, 10, 11, 12, 15, 18, 19, 21, 22, 23, 30, 33, 35, 36, 37], "create_biases_and_weight": 1, "create_convolutional_neural_network_kera": 3, "create_lay": [21, 22], "create_layers_batch": 21, "create_neural_network_kera": 1, "create_x": [5, 11], "creation": [], "credit": [0, 7, 28, 30, 35, 36], "crim": [], "crime": [], "criteria": [0, 4, 9, 10, 14, 27, 30], "criterion": [9, 10, 13, 18, 32, 33, 37], "critic": [6, 25, 31], "critiqu": 25, "cross": [0, 1, 3, 7, 9, 10, 13, 15, 21, 22, 23, 27, 30, 31, 32, 33], "cross_entropi": [4, 21], "cross_val_scor": [6, 34, 35], "cross_valid": [7, 10, 36], "crossvalid": [6, 34, 35], "crucial": [1, 27, 33], "cs231": 3, "csr_matrix": [24, 30], "css": [], "csv": [0, 4, 6, 7, 9, 34, 35, 36], "ctnk": 1, "cube": 37, "cubic": 0, "culprit": [], "cumbersom": [5, 34], "cumprod": [], "cumsum": [10, 11, 30], "cumul": [7, 10, 27, 33], "cumulative_heads_ratio": 10, "cup": 5, "current": [1, 2, 3, 4, 13, 14, 15, 16, 29, 32, 33, 35, 36], "curs": [0, 31], "curv": [6, 7, 10, 12, 25, 35, 36], "curvatur": [13, 32, 33], "custom": [6, 14], "custom_cmap": [9, 10], "custom_cmap2": [9, 10], "custom_lin": [], "cutpoint": 9, "cv": [6, 7, 10, 34, 35, 36], "cvxbook": [13, 32], "cvxopt": [5, 8, 31], "cybenko": 37, "cycl": [1, 12, 36, 37], "cycler": [], "d": [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 13, 14, 15, 16, 17, 19, 20, 21, 22, 24, 27, 28, 30, 31, 32, 33, 34, 35, 36, 37], "d1": [], "d166a3": [], "d2": [], "d2_g_t": 2, "d2a8ff": [], "d4d0ab": [], "d71835": [], "d9dee3": [], "d_f": [13, 32], "d_g_t": 2, "d_net_out": 2, "da": [3, 22, 37], "da_1": 22, "dagger": [5, 24, 31, 32], "dai": [1, 9, 23], "damag": [], "damp": 3, "darget": 9, "darkr": 27, "dat": [0, 30], "dat_id": [0, 6, 7, 9, 30, 34, 35], "data": [2, 4, 5, 8, 10, 12, 13, 14, 16, 19, 20, 22, 24, 25, 29, 32, 33, 34], "data1": 14, "data2": 14, "data3": 14, "data4": 14, "data_id": [0, 6, 7, 9, 30, 34, 35], "data_indic": 1, "data_panda": 30, "data_path": [0, 6, 7, 9, 30, 34, 35], "databas": 1, "datafil": [0, 6, 7, 9, 30, 34, 35], "datafram": [0, 4, 5, 7, 9, 11, 30, 31, 36], "datapoint": [1, 5, 6, 7, 11, 13, 16, 32, 33, 34, 35], "datasci": [15, 16, 19], "dataset": [0, 4, 6, 7, 8, 9, 10, 11, 13, 14, 16, 21, 22, 25, 30, 32, 33, 34, 35, 36], "datatyp": 4, "date": [15, 18, 21, 22, 25, 30, 31, 32, 33, 34, 35, 36, 37], "daughter": 10, "davi": [], "david": 29, "davison": [34, 35], "db": [22, 37], "db_1": 22, "dbb7ff": [], "dbh": 1, "dbo": 1, "dc": 22, "dc_da": 22, "dc_da1": 22, "dc_da2": 22, "dc_db": 22, "dc_db1": 22, "dc_db2": 22, "dc_dw": 22, "dc_dw1": 22, "dc_dw2": 22, "dc_dz": 22, "dc_dz1": 22, "dc_dz2": 22, "dcc6e0": [], "dcomposit": 24, "ddot": 2, "de": 33, "dead": 1, "deadlin": [15, 20, 21, 22], "deal": [0, 1, 3, 5, 6, 8, 11, 13, 14, 19, 24, 27, 30, 31, 32, 33, 37], "dealt": 0, "debt": [7, 35, 36], "debug": [0, 5, 6, 31, 32, 33, 34, 35], "debugg": [], "decad": [0, 3, 33], "decai": [0, 13, 27, 30], "decemb": [28, 30], "decent": 10, "decid": [0, 2, 3, 5, 6, 9, 18, 31, 32, 33, 34, 35], "decim": [0, 30], "decis": [0, 1, 8, 11, 23, 29, 30], "decision_funct": 8, "decision_tre": 9, "decisiontreeclassifi": [9, 10], "decisiontreeregressor": [0, 9, 10], "declar": [0, 4, 20, 24, 30], "declare_namespac": [], "decompos": [5, 6, 24, 31, 32, 37], "decomposit": [0, 6, 12, 30, 36, 37], "decompost": [5, 31, 32], "deconvolut": 3, "decorrel": [10, 13, 33], "decreas": [1, 2, 4, 5, 6, 10, 11, 13, 19, 32, 33, 34, 35], "dedic": 20, "deduc": [0, 30], "deep": [3, 7, 12, 13, 23, 29, 31, 32], "deep_neural_network": 2, "deep_param": 2, "deep_tree_clf": [9, 10], "deep_tree_clf1": 9, "deep_tree_clf2": 9, "deepen": [5, 23, 30], "deeper": [0, 3, 4, 30], "deeplearningbook": [29, 30, 32, 33], "deer": 3, "def": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 13, 14, 16, 17, 21, 22, 27, 30, 31, 32, 33, 34, 35, 36, 37], "def_covari": 27, "default": [0, 1, 2, 4, 6, 7, 24, 30, 31, 35, 36], "default_tim": 4, "defect": [5, 31, 32], "defici": [5, 31, 32], "defin": [0, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 16, 17, 18, 19, 21, 22, 24, 25, 27, 31, 32, 33, 34, 35, 36], "definit": [1, 2, 5, 6, 7, 8, 10, 11, 12, 13, 24, 27, 31, 32, 33, 34, 35, 36], "defint": 27, "degre": [3, 5, 6, 8, 9, 10, 11, 15, 16, 19, 20, 25, 27, 30, 32, 33, 34, 35], "deisenroth": 31, "del": 1, "delet": [6, 15], "delimit": 4, "deliv": [15, 25, 26, 30], "delta": [0, 2, 3, 6, 8, 12, 13, 14, 30, 33, 37], "delta_": [1, 24, 37], "delta_0": [3, 37], "delta_1": [3, 37], "delta_2": [3, 37], "delta_2a_1": 37, "delta_3": 3, "delta_4": 3, "delta_5": 3, "delta_h": [0, 1, 30], "delta_i": 37, "delta_j": [3, 12, 37], "delta_k": [12, 37], "delta_l": [1, 3], "delta_momentum": [13, 33], "delta_n": [0, 3, 30], "delug": 23, "delv": 0, "demand": [13, 32], "demonstr": [0, 3, 5, 6, 7, 11, 12, 19, 23, 30, 31, 32, 33, 34, 35, 36], "demystifi": [36, 37], "den": 4, "denomin": [1, 5, 33], "denot": [1, 2, 6, 7, 13, 27, 32, 33, 35, 36], "dens": [1, 3, 4], "densiti": [0, 2, 6, 27, 34, 35], "depart": [28, 30, 31, 32, 33, 34, 35, 36, 37], "depend": [0, 1, 2, 4, 5, 6, 7, 8, 11, 12, 13, 15, 16, 23, 24, 25, 27, 30, 31, 32, 33, 35, 36, 37], "depict": 27, "deploy": [0, 23, 25, 30], "depth": [0, 3, 9, 10, 24, 34], "der": [], "deriv": [0, 1, 2, 6, 7, 8, 10, 11, 13, 18, 22, 23, 25, 30, 35, 36], "derivati": 13, "derivative_fn": 13, "derivb1": 37, "derivb2": 37, "derivw1": 37, "derivw2": 37, "descend": [5, 9, 11, 31, 32], "descent": [0, 1, 3, 7, 8, 12, 22, 30, 31, 35, 37], "describ": [0, 2, 4, 5, 6, 8, 10, 11, 12, 13, 19, 20, 24, 25, 30, 33, 34, 36, 37], "descript": [0, 8, 9, 20, 25, 30], "design": [0, 1, 3, 4, 5, 6, 7, 10, 11, 12, 13, 17, 18, 25, 30, 32, 33, 34, 35, 36, 37], "designmatrix": [0, 30], "desir": [0, 2, 4, 5, 13, 14, 30, 31, 32, 33], "desktop": 15, "despit": [1, 12, 33, 36], "destroi": 24, "det": [5, 24, 31, 32], "detail": [0, 6, 11, 13, 14, 18, 21, 22, 24, 25, 31, 32, 33], "detect": [3, 8, 12, 36, 37], "determin": [0, 2, 3, 4, 5, 6, 8, 9, 10, 11, 12, 13, 18, 24, 27, 30, 31, 32, 33, 34, 35, 36, 37], "determinist": [7, 13, 27, 32, 33, 35, 37], "deternin": 37, "dev": [1, 25], "develop": [0, 3, 5, 8, 10, 11, 12, 23, 24, 25, 30, 31, 36, 37], "deviat": [0, 1, 2, 4, 5, 6, 17, 18, 19, 25, 27, 30, 31, 33, 34, 35], "devis": [12, 36, 37], "df": [4, 8, 11, 13, 30, 37], "df1": 30, "di": [], "diag": [5, 8, 31, 32, 33], "diagnost": [1, 10], "diagon": [0, 5, 7, 13, 18, 19, 24, 27, 30, 31, 32, 33, 35, 36], "diagonaliz": [5, 31, 32], "diagram": 10, "diagsvd": 6, "dice": [6, 27, 34], "dict": [6, 8], "dictionari": [], "did": [0, 1, 5, 6, 7, 10, 11, 14, 16, 22, 25, 30, 34, 35, 36], "die": 1, "diff": [2, 37], "diff1": 2, "diff2": 2, "diff_ag": 2, "diffeent": 8, "differ": [0, 1, 2, 3, 4, 5, 6, 7, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 21, 22, 23, 24, 25, 27, 29, 30, 31, 32, 34, 35, 36, 37], "different": 37, "differenti": [0, 3, 16, 21, 22, 23, 24, 30, 31, 32, 36], "difficult": [0, 1, 6, 10, 13, 27, 30, 33, 34, 35], "difficulti": [0, 1, 13, 30, 32, 33], "diffonedim": 2, "digit": [0, 1, 3, 4, 6, 28, 30], "digress": 37, "dilemma": [13, 33], "dilut": 1, "dim": [4, 11, 14, 24], "dimens": [0, 1, 2, 3, 4, 5, 8, 11, 14, 16, 24, 30, 31, 32, 37], "dimension": [0, 4, 5, 6, 9, 11, 13, 14, 19, 23, 24, 25, 30, 31, 32, 33, 34], "dimensionless": [0, 3, 30], "diment": 24, "diminish": 33, "dimnsion": 4, "diod": 3, "direct": [0, 1, 2, 4, 11, 12, 13, 14, 30, 31, 32, 33, 36, 37], "directli": [1, 4, 5, 6, 18, 22, 27, 31, 32], "directori": [], "disadvantag": [0, 30, 33], "disappear": [3, 6, 34], "disc_loss": 4, "disc_tap": 4, "discard": [6, 11, 33, 34, 35], "disciplin": [0, 3, 12, 36, 37], "disclaim": 27, "discontinu": 37, "discord": [21, 30], "discourag": [13, 15, 32], "discov": [0, 30], "discover": 5, "discret": [1, 3, 5, 7, 13, 35, 36], "discrimin": [4, 7, 10, 11, 35, 36], "discriminator_loss": 4, "discriminator_loss_list": 4, "discriminator_model": 4, "discriminator_optim": 4, "discuss": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 16, 18, 19, 20, 23, 24, 25, 27, 29, 30, 31, 32, 33, 34, 35, 37], "diseas": [7, 35, 36], "disguis": [6, 31, 33], "disk": 33, "disord": [1, 7, 35, 36], "dispai": [36, 37], "displai": [0, 1, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 14, 25, 27, 30, 31, 33, 34, 35, 36, 37], "displaystyl": [0, 5, 17, 30, 31, 32, 33], "disregard": [0, 30], "dissimilar": [11, 14], "dist": 14, "distanc": [8, 9, 11, 14, 27], "distance_list": 9, "distinct": [3, 7, 8, 9, 10, 14, 35, 36], "distinctli": 8, "distinguish": [0, 4, 7, 8, 27, 30, 36], "distplot": [], "distribut": [0, 1, 4, 6, 7, 10, 11, 13, 14, 18, 19, 21, 23, 24, 25, 30, 31, 32, 33, 35], "distrubut": [0, 23, 25, 30], "div": [], "dive": [0, 8, 24, 30], "diverg": [1, 13, 32, 33], "divid": [0, 1, 3, 5, 6, 7, 8, 9, 11, 12, 18, 19, 27, 30, 31, 33, 34, 35, 36, 37], "divis": [6, 8, 9, 13, 18, 24, 27, 33, 34, 35, 37], "dl": [], "dm": [], "dna": [7, 35, 36], "dnn": [0, 1, 2, 4, 12, 30, 36, 37], "dnn1": 4, "dnn2_gru2": 4, "dnn_kera": 1, "dnn_model": 1, "dnn_numpi": 1, "dnn_scikit": [0, 1, 30], "do": [0, 2, 3, 4, 5, 6, 8, 9, 10, 11, 12, 13, 14, 15, 16, 18, 19, 20, 21, 22, 24, 25, 30, 31, 32, 34, 35, 37], "doc": [0, 15, 16, 19, 23, 25, 26, 28, 29, 30], "document": [4, 13, 15], "docutil": [], "doe": [0, 1, 2, 3, 4, 5, 6, 8, 10, 11, 12, 13, 15, 16, 17, 18, 19, 21, 22, 24, 25, 27, 30, 33, 34, 35, 37], "doesn": [3, 9, 12, 30, 33, 37], "dog": [1, 3, 4], "dollar": [], "domain": [5, 8, 13, 25, 32, 34], "domcontentload": [], "domin": [0, 30], "don": [0, 1, 3, 5, 6, 8, 11, 13, 15, 16, 21, 23, 25, 30, 31, 33], "done": [0, 2, 3, 4, 5, 6, 9, 10, 11, 13, 16, 20, 22, 24, 25, 30, 31, 32, 33, 34, 35, 37], "dot": [0, 2, 3, 5, 6, 7, 8, 9, 10, 11, 12, 13, 18, 24, 25, 27, 30, 31, 32, 33, 34, 35, 36, 37], "doubl": [3, 4, 16, 24, 30], "doubli": 1, "doubt": 25, "down": [0, 3, 6, 9, 11, 12, 13, 32, 33, 36], "download": [0, 1, 3, 5, 6, 15, 20, 24, 29, 30], "downsampl": 3, "dozen": 1, "dq": [6, 34], "draft": 20, "drag": 13, "dragon": [], "dramat": 11, "drastic": 4, "draw": [4, 6, 10, 13, 32, 34, 35], "drawback": [0, 1, 3, 13, 31, 32, 33], "drawn": [1, 4, 6, 7, 11, 27, 30, 34, 35, 36], "drive": [3, 4, 21, 22], "driven": 3, "drop": [0, 1, 5, 6, 11, 13, 27, 30, 31, 32, 34], "dropna": [0, 6, 30, 34, 35], "dropout": 4, "dt": [2, 3, 13, 27, 37], "dtype": [0, 1, 3, 4, 14, 24, 30, 35, 36, 37], "dual": [], "dub": [0, 30], "duboi": [], "due": [1, 2, 5, 6, 8, 10, 12, 13, 18, 28, 30, 31, 32, 33, 34, 35, 36, 37], "dugard": [], "dummi": [], "dure": [0, 1, 3, 4, 8, 9, 11, 20, 23, 25, 30, 33, 34, 35, 36], "dw": 22, "dw_1": 22, "dwell": [], "dwh": 1, "dwo": 1, "dx": [2, 3, 8, 27, 37], "dx_1": 27, "dx_1p": [6, 34], "dx_2p": [6, 34], "dx_mp": [6, 34], "dx_n": 27, "dxp": [6, 34], "dy": [1, 8, 27], "dynam": 4, "dz": [8, 22], "dz_1": 22, "dz_2": 22, "e": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 27, 28, 30, 31, 32, 33, 34, 35, 36, 37], "e1e1e1": [], "e_": [0, 2, 30], "e_z": 21, "each": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 18, 22, 23, 24, 26, 27, 28, 30, 31, 32, 33, 34, 36, 37], "eager": 34, "eapprox": [0, 30], "earli": [1, 13, 33], "earlier": [0, 5, 7, 8, 9, 11, 12, 13, 19, 20, 21, 22, 30, 31, 35, 36, 37], "earthexplor": 6, "eas": [6, 9, 14, 34], "easi": [0, 5, 6, 7, 8, 9, 10, 11, 12, 13, 15, 21, 22, 23, 24, 30, 31, 32, 33, 34, 35, 36, 37], "easier": [5, 6, 8, 9, 13, 15, 20, 21, 22, 25, 27, 30, 31, 32, 34, 35], "easiest": [13, 18, 35, 36], "easili": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 24, 25, 30, 31, 32, 33, 34, 35, 36, 37], "eastern": [28, 30], "ebind": [0, 30], "eblock": 9, "ec8e2c": [], "econom": [], "econometr": 30, "economi": 5, "ecosystem": [23, 30], "ect": 26, "edg": 3, "edgecolor": [6, 34, 35], "edit": [21, 22], "editor": [15, 20], "edu": [13, 25, 32], "educ": [0, 25, 30, 34], "ee6677": [], "eff": 27, "effect": [1, 4, 10, 13, 16, 17, 18, 27, 33], "effic": 1, "effici": [0, 3, 10, 13, 21, 22, 23, 24, 27, 30, 33, 35, 36, 37], "effort": 19, "efron": [6, 34, 35], "egrad": 13, "eig": [5, 11, 13, 24, 27, 30, 31, 32, 33], "eigen": 27, "eigenpair": [5, 11, 31, 32], "eigenvalu": [0, 5, 8, 11, 13, 24, 30, 31, 32, 33], "eigenvector": [5, 11, 13, 31, 32], "eight": [24, 30], "eigval": [24, 27, 30], "eigvalu": [11, 13, 32, 33], "eigvec": [24, 27, 30], "eigvector": [11, 13, 32, 33], "eir": [28, 30], "eispack": [24, 30], "either": [1, 5, 6, 7, 8, 9, 10, 11, 13, 18, 19, 25, 27, 30, 31, 32, 33, 34, 35, 36, 37], "eivind": 28, "eivinsto": 28, "ekstr\u00f8m": 4, "elabor": 27, "elarn": 3, "electr": [0, 3, 12, 30, 36, 37], "electron": 30, "eleg": 11, "element": [1, 2, 3, 4, 5, 6, 7, 8, 11, 12, 13, 19, 20, 21, 23, 24, 25, 29, 31, 33, 34, 35, 36, 37], "elementari": [10, 13, 24, 37], "elementwis": [3, 13], "elementwise_grad": [2, 13, 22], "elessar": 30, "elif": 14, "elim": 24, "elimin": [3, 8], "elin": [28, 30], "ell_": [], "ellipsi": 16, "els": [1, 3, 4, 7, 9, 12, 13, 16, 22, 24, 35, 36], "elu": 1, "elus": [0, 30], "em": [], "email": [20, 21, 26, 28, 30], "emb": [], "embark": 37, "embed": [0, 11, 31], "embodi": [6, 25, 34, 35], "emit": 27, "emner": 29, "emph": 33, "emphas": [0, 10, 23, 30], "emphasi": [0, 23, 29, 30], "empir": [1, 11, 27], "emploi": [0, 1, 5, 6, 11, 13, 27, 30, 31, 32, 34], "employ": 0, "empti": [6, 10, 15, 34, 35], "emul": [12, 36, 37], "en": [23, 25, 29], "enabl": [11, 33], "enbodi": [6, 34], "encod": [0, 3, 5, 9, 11, 14, 30, 31, 32, 35, 36], "encompass": [0, 25, 27], "encount": [0, 1, 5, 7, 13, 15, 21, 25, 27, 30, 31, 32, 33, 35, 36], "encourag": [15, 25], "end": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 20, 22, 24, 27, 28, 30, 31, 32, 33, 34, 35, 36, 37], "endblock": [], "endfor": [], "endif": [], "endors": [], "endpoint": [3, 6], "energi": [0, 4, 6, 34, 35], "enforc": [12, 36, 37], "eng": 29, "engin": [0, 1, 3, 4, 23, 30], "english": 25, "enjoi": 33, "enocurag": 25, "enorm": 3, "enough": [0, 6, 13, 30, 32, 33, 34], "ensembl": [1, 9, 30], "ensur": [0, 1, 2, 3, 5, 6, 11, 13, 18, 27, 31, 32, 33, 34, 35, 37], "entail": 30, "enter": [5, 6, 31, 32, 33], "enthought": [0, 23, 25, 30], "entir": [1, 3, 7, 9, 21, 23, 27, 30, 33, 35], "entireti": [], "entiti": [9, 12, 24, 30], "entri": [0, 5, 8, 11, 12, 24, 30, 31, 33, 34], "entropi": [1, 3, 7, 10, 13, 21, 22, 30, 32, 33], "enumer": [0, 1, 2, 3, 4, 6, 8, 21, 30, 31, 33, 35, 36], "env": 27, "environ": [2, 21, 22, 23, 25, 30], "environemnt": 15, "eo": [0, 6, 34, 35], "eol": 0, "eosfit": 0, "epoch": [0, 1, 3, 4, 12, 13, 21, 30, 33, 35, 36], "eppstein": [], "epsilon": [0, 5, 6, 7, 13, 25, 30, 31, 32, 33, 34, 35, 36, 37], "epsilon_": [0, 30], "epsilon_0": [0, 30], "epsilon_1": [0, 30], "epsilon_2": [0, 30], "epsilon_i": [0, 30, 31], "eq": [3, 13, 14, 24, 27, 32], "eqnarrai": [3, 5, 6, 34], "equal": [0, 1, 2, 3, 4, 5, 6, 8, 9, 11, 12, 13, 14, 16, 18, 24, 25, 27, 30, 31, 32, 33, 34, 35, 37], "equat": [1, 3, 4, 5, 6, 7, 8, 9, 10, 11, 13, 14, 17, 19, 24, 27, 30, 33, 34], "equilibrium": [2, 12, 36, 37], "equiv": [3, 13, 24, 27, 32, 33], "equival": [0, 1, 5, 7, 8, 11, 13, 23, 24, 30, 31, 32, 33, 34], "equivel": [19, 21, 22], "eras": [], "erf": 27, "eriador": 30, "eric": [], "err": [0, 10], "err_": [6, 34, 35], "err_sqr": 2, "errat": [13, 32, 33], "erron": 2, "error": [1, 2, 4, 5, 6, 7, 9, 11, 12, 13, 15, 16, 17, 18, 19, 21, 23, 24, 25, 27, 33, 36, 37], "error_estimate_corr_tim": 27, "error_hidden": 1, "error_output": 1, "escap": [13, 32, 33], "escapehtml": [], "especi": [1, 3, 9, 12, 13, 15, 18, 25, 33, 36, 37], "essenti": [0, 5, 6, 9, 10, 12, 14, 15, 25, 27, 31, 32, 33, 36, 37], "establish": [0, 6, 10, 11, 16, 25], "estim": [0, 1, 5, 6, 7, 10, 11, 13, 23, 27, 30, 31, 32, 33, 35, 36], "estimated_mse_fold": [6, 34, 35], "estimated_mse_kfold": [6, 34, 35], "estimated_mse_sklearn": [6, 34, 35], "et": [0, 2, 4, 16, 17, 20, 29, 30, 31, 32, 34, 35, 36, 37], "eta": [0, 1, 3, 8, 12, 13, 18, 30, 32, 33, 37], "eta0": [8, 13], "eta_": 13, "eta_j": 33, "eta_t": [13, 33], "eta_v": [0, 1, 3, 30], "etc": [0, 1, 3, 5, 7, 8, 9, 11, 12, 13, 14, 23, 24, 25, 27, 31, 32, 33, 35, 36], "ethic": 23, "etsim": 34, "euclidean": [0, 14, 31, 33], "euler": [], "evalu": [0, 2, 3, 4, 5, 6, 9, 13, 15, 16, 17, 19, 21, 25, 27, 30, 31, 32, 33, 34, 35, 36], "evalut": [13, 25], "even": [0, 1, 3, 4, 5, 6, 8, 9, 10, 11, 12, 13, 14, 22, 23, 24, 27, 30, 31, 32, 33, 34, 35, 36, 37], "evenli": 4, "event": [5, 7, 10, 27, 34, 35], "eventu": [0, 5, 6, 11, 12, 13, 25, 28, 31, 32, 33, 34, 35, 36, 37], "everi": [0, 1, 2, 3, 4, 5, 6, 9, 10, 11, 12, 13, 14, 15, 21, 23, 27, 28, 30, 31, 32, 33, 34, 35, 36, 37], "everyth": [4, 12, 16, 18, 21, 37], "everywher": [4, 13, 32], "evolv": 0, "exact": [0, 5, 11, 12, 13, 24, 27, 30, 31, 33, 37], "exactli": [0, 3, 4, 6, 12, 18, 23, 31, 33, 34, 36, 37], "exam": 30, "examin": [6, 34, 35], "exampl": [0, 5, 11, 12, 13, 15, 16, 18, 20, 23, 24, 25, 27, 29], "exce": [1, 12, 13, 33, 36, 37], "exceed": 33, "excel": [0, 1, 4, 5, 10, 20, 25, 30, 31], "except": [3, 4, 6, 8, 9, 24], "excess": [0, 30], "exchang": 33, "excit": 0, "exclud": [1, 6, 12, 25, 31, 33, 34, 35, 36], "exclus": [0, 1, 3, 6, 27, 30, 34, 35], "execut": [2, 5, 13, 15, 31, 32, 33], "exemplari": [], "exemplifi": [13, 33], "exercic": [28, 30], "exercis": [5, 23, 25, 26, 28, 30, 32, 33, 34, 35, 36], "exhaust": [6, 33, 34, 35], "exhibit": [0, 5, 6, 8, 30, 31, 34], "exist": [0, 1, 2, 3, 5, 6, 7, 8, 9, 13, 19, 24, 25, 30, 32, 33, 34, 35], "exit": [5, 24, 31, 32], "exp": [0, 1, 2, 5, 6, 7, 8, 10, 11, 12, 13, 16, 17, 19, 21, 22, 27, 31, 32, 33, 34, 35, 36, 37], "exp_term": 1, "exp_z": [35, 36], "expand": [5, 7, 11, 13, 32, 35, 36], "expans": [0, 3, 5, 8, 10, 12, 13, 30, 31, 32, 37], "expect": [0, 1, 5, 6, 7, 11, 12, 13, 15, 18, 23, 25, 30, 31, 33, 35, 37], "expectation_value_of_h_wrt_p": 27, "expens": [6, 10, 13, 16, 32, 33], "experi": [0, 1, 6, 8, 13, 15, 23, 25, 30, 31, 32, 33, 34, 35], "experiment": [0, 4, 6, 9, 27, 30, 34, 35], "expert": [1, 9], "explain": [0, 6, 9, 10, 11, 13, 16, 19, 25, 30, 32, 35, 36], "explained_variance_ratio_": 11, "explan": [], "explanatori": [0, 30], "explicit": [0, 3, 6, 13, 24, 25, 30, 31, 32, 33], "explicitli": [0, 4, 21], "explod": [1, 37], "exploit": [0, 3, 12, 13, 30, 33, 36, 37], "explor": [1, 4, 6, 8, 13, 18, 23, 25, 30, 32, 33], "expon": 1, "exponenti": [0, 1, 5, 6, 10, 13, 27, 30, 32, 37], "export": [9, 15, 16, 19, 20, 35, 36], "export_graphviz": 9, "export_text": 9, "exporttext": 9, "expos": 23, "expr": 37, "express": [0, 2, 3, 5, 6, 7, 10, 12, 13, 18, 22, 24, 25, 27, 30, 32, 33, 34], "exptmean": 27, "exptvari": 27, "extend": [0, 2, 7, 11, 13, 23, 30, 33], "extend_path": [], "extens": [0, 12, 15, 23, 30, 36, 37], "extent": [0, 1, 6, 29, 34, 35], "extern": [3, 6, 9], "extra": [1, 3, 5, 15, 28, 30, 31, 32], "extract": [0, 3, 5, 6, 7, 8, 11, 13, 16, 17, 24, 30, 31, 35, 36, 37], "extrapol": [0, 30], "extrem": [0, 1, 4, 5, 6, 7, 8, 9, 13, 15, 16, 24, 31, 32, 33, 35], "extremum": [13, 32], "extrins": 11, "ey": [0, 5, 6, 13, 14, 18, 24, 30, 31, 32, 33], "f": [0, 1, 2, 3, 4, 5, 6, 7, 8, 10, 12, 13, 14, 15, 16, 17, 18, 19, 22, 24, 27, 28, 30, 31, 32, 33, 34, 35, 36, 37], "f1": 13, "f11": [0, 30], "f12": [0, 30], "f13": [0, 30], "f1_grad": 13, "f1d": 13, "f2": 13, "f26196": [], "f2_grad_x1": 13, "f2_grad_x1_analyt": 13, "f2_grad_x2": 13, "f2_grad_x2_analyt": 13, "f2f2f2": [], "f3": 13, "f3_grad": 13, "f3_grad_analyt": 13, "f4": 13, "f4_grad": 13, "f4_grad_analyt": 13, "f5": 13, "f5_grad": 13, "f5a394": [], "f5ab35": [], "f5f5f5": [], "f6": 13, "f6_for": 13, "f6_for_grad": 13, "f6_grad_analyt": 13, "f6_while": 13, "f6_while_grad": 13, "f7": 13, "f78c6c": [], "f7_grad": 13, "f7_grad_analyt": 13, "f8": 13, "f8_grad": 13, "f8f8f2": [], "f9": [0, 13, 30], "f9_altern": 13, "f9_alternative_grad": 13, "f9_grad": 13, "f_": 10, "f_0": [3, 10], "f_1": [10, 13, 32], "f_2": [12, 13, 32, 36], "f_3": [12, 36], "f_d": 27, "f_grad": 13, "f_grad_analyt": 13, "f_i": [0, 6, 12, 16, 34, 35, 36], "f_m": [3, 10], "f_n": 3, "f_vec": 2, "face": [13, 30, 32], "facecolor": [6, 8, 27, 34], "facil": [0, 23], "facilit": [12, 36, 37], "fact": [0, 1, 3, 5, 9, 11, 12, 13, 22, 30, 31, 32, 33], "facto": 33, "factor": [0, 1, 3, 5, 6, 9, 10, 11, 13, 24, 27, 30, 31, 32], "factori": 13, "fad000": [], "fade": 6, "fae4c2": [], "fafab0": [9, 10], "fail": [0, 6, 13, 28, 30, 32, 34, 35, 37], "failur": [7, 35, 36], "fairli": [1, 2, 18, 27, 33], "faisal": [16, 31], "fake": 4, "fake_loss": 4, "fake_output": 4, "fall": [8, 9, 26], "fals": [0, 1, 2, 3, 4, 5, 6, 7, 9, 10, 14, 16, 17, 24, 30, 31, 32, 33, 34, 35, 36], "famili": [0, 7, 8, 27, 31, 33, 35, 36, 37], "familiar": [0, 3, 5, 6, 8, 15, 23, 24, 25, 27, 30, 34, 37], "famou": [6, 12], "far": [0, 3, 4, 5, 6, 8, 11, 12, 13, 14, 16, 20, 21, 22, 30, 31, 32, 33, 36, 37], "fashion": [0, 9, 10, 30, 33], "fast": [1, 3, 6, 10, 12, 13, 23, 27, 30, 32, 33, 34, 35, 37], "faster": [1, 11, 13, 21, 33], "fastest": [13, 24, 32], "fatal": [], "favor": [7, 33, 35], "favorit": 27, "fc": 3, "fcfcfc": [], "fdac54": [], "fdf2e2": [], "featur": [0, 1, 3, 5, 6, 7, 8, 10, 11, 12, 13, 15, 17, 18, 19, 21, 23, 27, 30, 32, 33, 34, 35, 36, 37], "feature_nam": [1, 7, 9, 21, 36], "feautur": 9, "fed": [1, 37], "feed": [0, 2, 3, 11, 21, 23, 30], "feed_forward": [1, 21, 22], "feed_forward_all_relu": 21, "feed_forward_batch": 21, "feed_forward_one_lay": 22, "feed_forward_out": 1, "feed_forward_sav": 22, "feed_forward_train": 1, "feed_forward_two_lay": 22, "feedback": [4, 20, 30], "feeddorward": 4, "feedforward": [1, 4, 12], "feel": [0, 5, 6, 11, 13, 15, 16, 18, 21, 22, 23, 25, 28, 30, 37], "feet": [], "fefef": [], "fefeff": [], "felt": 25, "fenc": [], "fernando": [], "fetch": [6, 15], "few": [1, 3, 4, 5, 9, 17, 18, 19, 22, 27, 30, 37], "fewer": [0, 9, 11, 19, 30, 33], "ff7b72": [], "ff9492": [], "ffa07a": [], "ffa657": [], "ffb757": [], "ffd700": [], "ffd900": [], "ffd9002e": [], "ffffff": [], "ffnn": [1, 12, 36, 37], "fi": [], "field": [0, 3, 6, 12, 19, 23, 36, 37], "fieldmask": [], "fifteen": 37, "fifth": [0, 6, 30], "fig": [0, 1, 2, 3, 4, 6, 7, 12, 13, 14, 25, 30, 35, 36], "fig_id": [0, 6, 7, 9, 30, 34, 35], "figaxi": 27, "figsiz": [0, 1, 2, 3, 4, 6, 7, 8, 9, 10, 30, 34, 35, 36], "figur": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 12, 13, 14, 16, 23, 25, 30, 31, 32, 33, 34, 35, 36, 37], "figure_id": [0, 6, 7, 9, 30, 34, 35], "figurefil": [0, 6, 7, 9, 30, 34, 35], "file": [0, 4, 5, 6, 7, 9, 15, 20, 21, 22, 25, 30, 34, 35], "file_prefix": 4, "filenam": 30, "fill": [5, 9, 18, 31, 32], "fill_valu": [], "filter": [3, 4], "final": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 13, 14, 18, 20, 21, 22, 25, 26, 27, 28, 30, 32, 34, 35, 36], "financ": 0, "find": [0, 1, 2, 3, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 21, 22, 23, 25, 27, 30, 31, 32, 33, 35, 36, 37], "fine": [0, 14], "finish": [2, 20, 21], "finit": [3, 5, 6, 12, 13, 17, 27, 31, 32, 34, 35, 36, 37], "finnicki": 15, "fire": [], "first": [0, 1, 2, 3, 5, 6, 7, 8, 9, 10, 11, 13, 14, 15, 16, 18, 19, 21, 22, 24, 25, 27, 28, 29, 31, 33, 34, 35, 36], "first_moment": 33, "first_term": 33, "firsteigvector": 11, "fit": [1, 3, 4, 5, 6, 7, 8, 9, 11, 12, 13, 17, 18, 19, 22, 25, 27, 31, 33, 34, 35, 36, 37], "fit_beta": 31, "fit_intercept": [0, 5, 6, 16, 31, 32, 33, 34, 35, 36], "fit_mod": 9, "fit_theta": [6, 33], "fit_transform": [0, 6, 8, 9, 11, 15, 19, 34, 35], "fiti": [0, 30], "five": [0, 9, 30, 31, 37], "fix": [0, 3, 4, 6, 10, 11, 12, 13, 25, 30, 34, 35, 36], "flag": 4, "flat": [12, 13, 32, 33], "flatten": [1, 3, 4, 5, 24], "flavor": [], "flexibl": [1, 6, 8, 10, 12, 30, 33, 34, 35, 36], "flip": [21, 28, 30], "float": [0, 3, 4, 5, 9, 11, 13, 14, 24, 30, 31, 32], "float32": [4, 9], "float64": [4, 24, 30, 36, 37], "flop": [5, 24, 31, 32], "flow": [1, 4, 12, 36, 37], "flower": 21, "fluctuat": [5, 33], "fly": 11, "fm": 0, "fmax": 3, "fmesh": 13, "fn": 7, "focu": [0, 3, 4, 5, 6, 15, 23, 25, 29, 30, 31, 32, 33, 34, 35], "focus": [1, 6, 7, 24, 31, 33, 35, 36], "fold": [6, 9, 25], "folder": [0, 4, 6, 15, 20, 25, 30], "follow": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 19, 20, 21, 22, 23, 24, 25, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37], "font": [7, 20, 27, 30, 35], "fontdict": 27, "fontsiz": [1, 6, 8, 9, 10, 27], "fontweight": 1, "footprint": [3, 33], "foral": [8, 31, 37], "forc": [0, 5, 6, 10, 11, 31, 32, 33, 37], "forcast": 4, "forcier": [], "forecast": [4, 12, 36, 37], "forest": [0, 1, 9, 23, 30], "forget": [11, 33], "form": [0, 3, 4, 5, 6, 7, 8, 9, 11, 12, 13, 15, 16, 23, 24, 25, 27, 30, 31, 32, 33, 34, 35, 36, 37], "formal": [3, 4, 14, 18, 27, 37], "format": [0, 1, 3, 4, 6, 7, 8, 9, 10, 11, 20, 23, 27, 29, 34, 35, 36], "format_data": 4, "formatstrformatt": [6, 13, 32, 33], "formatt": [], "formul": [4, 6, 11, 14], "formula": [3, 13, 27, 32, 37], "forth": [4, 12, 22, 36], "fortran": [0, 23, 24, 30], "fortran2003": [23, 30], "fortran2008": 25, "fortran90": 27, "fortun": [0, 11, 31], "forward": [0, 3, 6, 21, 23, 24, 30, 33, 34], "forwardpropag": 37, "found": [1, 2, 4, 5, 6, 12, 13, 19, 20, 21, 22, 25, 30, 31, 33, 34, 35, 36, 37], "foundat": [23, 30], "four": [4, 5, 6, 8, 12, 21, 24, 26, 28, 30, 32, 36, 37], "fourier": [0, 30, 37], "fourierdef1": 3, "fourierdef2": 3, "fourierseriessign": 3, "fourth": [12, 30, 31], "fp": 7, "frac": [0, 1, 2, 3, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 16, 17, 19, 21, 22, 24, 25, 27, 30, 31, 32, 33, 34, 35, 36, 37], "fraction": [9, 35, 36], "frame": [7, 33, 36], "framework": [1, 8, 10, 27], "frank": [5, 11], "frankefunct": [5, 6, 11], "fredli": [21, 28, 30], "free": [0, 6, 11, 13, 15, 16, 18, 21, 22, 23, 24, 25, 27, 28, 29, 30, 37], "freecodecamp": 23, "freedom": [5, 32], "freeli": [0, 25], "freez": 15, "frequenc": [3, 6, 7, 27, 34, 36], "frequent": [0, 8, 9, 13, 32], "frequentist": 23, "fresh": 10, "fridai": [15, 21, 22, 28, 30], "friedman": [6, 19, 25, 29, 30], "friendli": 4, "fro": 25, "frodo": 30, "frog": 3, "from": [0, 1, 2, 3, 4, 6, 7, 8, 9, 11, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 27, 28, 29], "from_cod": 9, "from_logit": [3, 4], "from_tensor_slic": 4, "front": [0, 4, 5, 30, 31, 32], "frustrat": 15, "fulfil": [2, 5, 12, 31, 32, 36], "full": [1, 3, 5, 7, 9, 10, 13, 21, 27, 30, 31, 32, 35], "full_matric": [5, 31, 32], "fulli": [3, 6, 12, 27, 34, 35, 36, 37], "fullnam": [], "fun": [23, 30], "func": [2, 21], "function": [2, 3, 4, 5, 9, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24], "functionali": 11, "fundament": [0, 6, 23, 30, 34, 35], "funtion": 2, "furnish": [], "furthemor": 37, "further": [2, 7, 9, 19, 30, 37], "furthermor": [0, 3, 5, 6, 7, 11, 12, 13, 23, 25, 30, 31, 32, 33, 34, 35, 36], "furthest": 22, "futur": [0, 4, 8, 9, 30], "fy": [15, 21, 25, 26, 28, 29, 30], "fys4155": 25, "fys5419": [29, 30], "fys5429": [29, 30], "f\u00f8470": [28, 30], "g": [0, 1, 2, 3, 4, 6, 8, 9, 10, 11, 13, 15, 18, 19, 27, 30, 31, 32, 33, 34, 35, 36], "g0": 2, "g_": [2, 9, 10, 33], "g_0": 2, "g_1": [2, 10], "g_2": [2, 10], "g_3": 37, "g_analyt": 2, "g_dnn_ag": 2, "g_euler": 2, "g_i": [2, 37], "g_j": 37, "g_m": [3, 10], "g_n": 3, "g_re": 2, "g_t": [2, 33], "g_t_d2t": 2, "g_t_d2x": 2, "g_t_dt": 2, "g_t_hessian": 2, "g_t_hessian_func": 2, "g_t_jacobian": 2, "g_t_jacobian_func": 2, "g_trial": 2, "g_trial_deep": 2, "g_vec": 2, "gain": [1, 5, 7, 9, 10, 13, 31, 32], "galleri": [0, 30], "game": 4, "gamge": 30, "gamma": [0, 2, 8, 9, 10, 11, 13, 30, 32], "gamma1": 8, "gamma2": 8, "gamma_": [0, 30], "gamma_0": 10, "gamma_1": 10, "gamma_1x": 10, "gamma_i": [0, 8, 27, 30], "gamma_j": 13, "gamma_k": [13, 32], "gamma_m": 10, "gamma_x": [0, 30], "gap": [8, 33], "gate": [4, 12, 37], "gather": [0, 1, 12, 31, 36, 37], "gaug": [12, 36, 37], "gaussbacksub": 24, "gaussian": [4, 5, 6, 8, 14, 18, 27, 30, 34, 35, 36], "gaussian_point": 14, "gaussian_rbf": 8, "gave": [13, 33], "gavra": 30, "gbc": 30, "gca": [2, 6, 8, 13], "gd": [1, 32, 37], "gd_clf": 10, "gdclassiffiercgain": 10, "gdclassiffierconfus": 10, "gdclassiffierroc": 10, "gdm": 13, "gdregress": 10, "ge": [1, 5, 7, 27, 31, 32, 35], "gen_loss": 4, "gen_tap": 4, "gender": [0, 30], "genener": 4, "gener": [0, 1, 2, 3, 5, 6, 8, 10, 11, 12, 13, 14, 15, 16, 18, 20, 21, 22, 24, 25, 27, 29, 31, 32, 33, 34], "generaliz": 16, "generallay": [12, 36], "generate_and_save_imag": 4, "generate_binary_data": [35, 36], "generate_imag": 4, "generate_latent_point": 4, "generate_multiclass_data": [35, 36], "generate_simple_clustering_dataset": 14, "generated_imag": 4, "generator_loss": 4, "generator_loss_list": 4, "generator_model": 4, "generator_optim": 4, "genom": 23, "geodes": 11, "geoff": 33, "geometr": [0, 13, 30, 33], "geometri": 5, "georg": 29, "geotif": 6, "geq": [2, 5, 8, 9, 13, 31, 32, 33], "gerard": [], "geron": [0, 29, 30], "get": [0, 1, 2, 3, 4, 5, 6, 7, 9, 10, 11, 13, 15, 19, 21, 22, 23, 24, 25, 27, 28, 30, 31, 32, 33, 34, 35], "get_dummi": 9, "get_paramet": 2, "get_split": 9, "get_yaxi": 8, "get_yticklabel": 6, "getmask": [], "gh": 15, "giant": 33, "gibb": [23, 30], "gif": 4, "gini": 10, "gini_index": 9, "ginvers": 13, "git": [0, 15, 23, 30], "gitcdn": [], "giter": [13, 33], "github": [0, 20, 23, 25, 26, 28, 29, 30, 31, 37], "gitignor": 15, "gitlab": [0, 15, 23, 25, 30], "gitta": 37, "give": [0, 1, 2, 3, 5, 6, 7, 8, 9, 10, 12, 13, 14, 18, 19, 23, 25, 27, 30, 31, 32, 33, 34, 35, 36, 37], "given": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 17, 19, 21, 24, 27, 30, 31, 32, 33, 34, 35, 36, 37], "glkfgrjhtlnplbx4": 21, "global": [6, 7, 13, 32, 33, 35, 36], "glorot": 1, "gmail": [], "gnew": 13, "go": [0, 1, 3, 5, 6, 8, 9, 11, 12, 13, 15, 16, 18, 21, 30, 31, 32, 34, 37], "goal": [0, 7, 9, 30, 35, 36], "goe": [0, 1, 2, 5, 6, 13, 14, 15, 19, 24, 30, 31, 32, 33, 34], "goessner": [], "golden": 13, "gone": [5, 31, 32], "gong": 1, "good": [1, 3, 4, 5, 6, 9, 10, 11, 13, 15, 18, 21, 23, 27, 29, 31, 32, 33, 35, 37], "goodfellow": [4, 29, 30, 31, 32, 35, 36, 37], "googl": [1, 4, 21, 22, 23, 30], "got": [1, 6, 21, 22, 25], "gotten": 30, "gov": 6, "govern": 30, "gp": 29, "gpu": [1, 13, 23, 30, 33], "grad": [2, 13, 21, 22, 33], "grad_analyt": 13, "grad_ol": 18, "grad_ridg": 18, "grad_two_lay": 22, "grade": [25, 26], "gradient": [0, 3, 4, 7, 8, 9, 12, 21, 23, 30, 31, 35], "gradient_desc": 33, "gradient_func": 21, "gradientboostingclassifi": 10, "gradientboostingregressor": 10, "gradients_of_discrimin": 4, "gradients_of_gener": 4, "gradienttap": 4, "gradual": [1, 14], "grai": [4, 6], "granger": [], "grant": [], "graph": [1, 9, 11, 12, 13, 16, 20, 32, 33, 36, 37], "graph_from_dot_data": 9, "graphic": [0, 1, 9, 15, 30], "grasp": 0, "gray_r": [1, 3], "grayscal": 3, "great": [5, 13, 15, 21, 22, 32, 33, 37], "greater": [1, 7, 27, 31, 36], "greatli": 13, "greedi": 9, "green": [0, 3, 9, 27], "grei": 4, "grid": [1, 3, 6, 7, 8, 12, 27, 31, 33, 34, 35, 36], "groh": 37, "grossli": [13, 32], "ground": [0, 30], "group": [0, 6, 7, 9, 14, 15, 20, 23, 25, 26, 28, 30, 34], "groupbi": [0, 30], "grow": [1, 3, 9, 10, 33], "growth": [0, 30], "gru": 4, "guarante": [0, 4, 13, 27, 30, 31, 32, 33], "guess": [1, 4, 10, 13, 14, 32, 33], "guestrin": 10, "gui": 15, "guid": [1, 21], "guidelin": [20, 25, 35, 36], "g\u00f6ssner": [], "h": [0, 1, 5, 6, 8, 13, 15, 19, 21, 27, 28, 29, 30, 31, 32, 33], "h1": 2, "h_": [0, 13, 30, 32, 33], "h_0": 33, "h_1": [2, 13, 32], "h_2": [2, 13, 32], "h_m": 10, "h_t": 33, "ha": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 17, 18, 19, 20, 21, 22, 24, 25, 27, 30, 31, 32, 33, 34, 35, 36, 37], "haanen": [28, 30], "habit": [0, 31], "had": [0, 1, 6, 7, 13, 30, 32, 33, 34, 35], "hadamard": [1, 12, 13, 33, 37], "half": [1, 8, 9, 35, 36, 37], "halv": 10, "hand": [0, 1, 2, 3, 5, 11, 12, 13, 23, 24, 25, 27, 28, 29, 30, 31, 32, 33, 35, 36], "handi": [3, 25], "handl": [0, 1, 2, 5, 9, 11, 15, 18, 22, 23, 31, 32, 33], "handle_unknown": 9, "handsid": [12, 37], "handwrit": [12, 36, 37], "handwritten": [1, 5], "happen": [1, 2, 3, 4, 5, 6, 10, 13, 27, 31, 32, 33, 36], "hard": [1, 7, 8, 10, 13, 21, 22, 32, 33, 35, 37], "hardcopi": [23, 30], "harder": [0, 1, 19, 21, 31], "harmon": 3, "hash": 33, "hasn": [], "hassl": [0, 23, 30], "hast": [23, 30], "hasti": [0, 6, 16, 17, 19, 20, 25, 29, 30, 31, 34, 35], "hat": [0, 1, 5, 6, 7, 9, 10, 11, 12, 13, 16, 17, 18, 19, 24, 31, 32, 33, 34, 36, 37], "hauser": [], "have": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 27, 28, 30, 31, 32, 33, 34, 35, 36, 37], "have_sys_un_h": [], "haven": [1, 22], "he": [7, 35, 36], "head": [4, 10, 27], "header": [0, 30], "heads_proba": 10, "health": [0, 31], "hear": [0, 13, 30, 33], "heart": [0, 7, 30, 35], "heatmap": [0, 1, 3, 7, 17, 20, 30, 36], "heavi": 33, "heavili": 0, "heavisid": 1, "height": [1, 3, 6, 31], "held": [13, 33], "help": [0, 1, 4, 12, 13, 15, 16, 25, 30, 33, 34, 36, 37], "helper": [4, 14, 35, 36], "henc": [0, 5, 6, 8, 9, 10, 12, 13, 30, 31, 32, 33, 34, 35, 36], "henrik": [28, 30], "her": [7, 35, 36], "here": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 17, 18, 19, 21, 22, 23, 24, 25, 27, 30, 31, 32, 33, 34, 35, 36, 37], "hereaft": [0, 8, 12, 30], "herebi": [], "hermitian": 24, "hessenberg": 24, "hessian": [0, 2, 5, 13, 35, 36], "heterogen": [9, 10], "hex": [], "hi": [7, 35, 36], "hidden": [1, 3, 4, 12, 21, 36], "hidden_bia": 1, "hidden_bias_gradi": [1, 37], "hidden_layer_s": [0, 1, 30], "hidden_neuron": 4, "hidden_weight": 1, "hidden_weights_gradi": [1, 37], "hierarch": [5, 31, 32], "high": [0, 1, 2, 3, 4, 5, 6, 9, 10, 11, 13, 14, 21, 23, 24, 25, 30, 31, 32, 33, 34, 35], "higher": [0, 1, 3, 5, 6, 8, 13, 18, 25, 30, 31, 32, 33, 34, 35], "highest": [1, 2, 35, 36], "highli": [0, 3, 4, 10, 19, 23, 24, 29, 30, 31, 32, 33], "highlight": [], "highwai": [], "hing": 8, "hint": [13, 15, 16, 21, 22, 31, 32], "hinton": 33, "hip": 23, "hire": 0, "hist": [4, 6, 7, 27, 34, 36], "histogram": [6, 7, 27, 36], "histor": [7, 11, 35], "histori": [3, 4, 12, 15, 33, 36, 37], "hitherto": 5, "hjorth": [28, 30, 31, 32, 33, 34, 35, 36, 37], "hobbi": 27, "hoc": [5, 31, 32], "hoff": 29, "hold": [1, 3, 6, 13, 14, 32, 33, 34], "holder": [0, 30], "holdgraf_evidence_2014": [], "home": [], "homepag": [25, 30], "homework": [6, 13, 32, 33], "homogen": [1, 3, 9, 10, 13, 33], "honchar": 2, "hopefulli": [0, 11, 15, 19, 27, 30, 33], "horizont": 11, "horlyk": [28, 30], "hornik": 37, "hors": [3, 7, 30, 35, 36], "hot": [1, 9, 35, 36], "hour": [1, 23, 26, 27, 28, 30, 33, 34], "how": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 17, 18, 19, 20, 21, 22, 23, 24, 25, 27, 30, 31, 32, 33, 34, 35, 36, 37], "howev": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 21, 23, 24, 25, 27, 30, 31, 32, 33, 34, 35, 36, 37], "href": [], "hspace": [0, 4, 8, 10, 27, 30, 37], "hstack": 1, "htf": 30, "html": [0, 16, 20, 21, 23, 25, 26, 28, 29, 30, 31, 32, 33, 37], "http": [0, 3, 4, 6, 13, 15, 16, 19, 20, 21, 22, 23, 24, 25, 26, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37], "huang": [0, 30], "huber": [0, 30], "huge": [1, 3, 4, 23, 33], "human": [0, 1, 3, 6, 9, 12, 31, 36, 37], "humid": 9, "hundr": 1, "hungri": 1, "hybrid": 26, "hydrogen": [0, 30], "hyperbol": [1, 4, 12], "hyperparam": 8, "hyperparamat": 37, "hyperparamet": [3, 4, 5, 6, 9, 13, 18, 25, 31, 32, 33, 37], "hyperplan": 11, "h\u00f8rlyk": [28, 30], "i": [2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 31, 32, 33, 34, 35, 36, 37], "i0": [0, 30], "i1": [0, 6, 8, 12, 30, 31, 33, 36], "i2": [0, 8, 12, 30, 36], "i3": [0, 12, 30, 36], "i5": [0, 30], "i_": [13, 32, 33], "i_1": [5, 6, 34], "i_2": [5, 6, 34], "i_siz": [21, 22], "i_t": 33, "ian": 29, "iayaan2": 21, "ic": [1, 25], "id": [7, 13, 32, 33, 35], "ida": [28, 30], "idea": [0, 1, 2, 3, 4, 6, 9, 10, 12, 13, 20, 24, 25, 31, 32, 33, 34, 35, 36, 37], "ideal": [0, 2, 6, 8, 13, 27, 30, 33, 34, 35, 36], "idem": [6, 34, 35], "ident": [5, 6, 12, 13, 17, 18, 24, 31, 32, 36], "identical": 34, "identifi": [0, 1, 7, 9, 11, 12, 13, 14, 30, 31, 35, 36], "idx": [35, 36], "ieor": 27, "ifi": 29, "ifs": [23, 30], "ignor": [0, 1, 3, 9, 15, 31, 33], "ii": [24, 27], "iii": [24, 30], "ij": [0, 1, 3, 6, 8, 12, 14, 16, 24, 27, 30, 31, 33, 36, 37], "ik": [0, 24, 30, 31], "iki": [], "ilg3ggewq5u": 37, "ill": 33, "illinoi": [], "illustr": [5, 7, 10, 12, 13, 14, 20, 23, 30, 35], "ilsvrc": 33, "im": 6, "imag": [1, 3, 4, 6, 9, 11, 12, 14, 29, 30, 36, 37], "image_at_epoch_": 4, "image_batch": 4, "image_height": 3, "image_path": [0, 6, 7, 9, 30, 34, 35], "image_width": 3, "imageio": 6, "imagenet": 33, "images_from_seed_imag": 4, "imagin": 1, "immedi": [0, 3, 4, 6, 23, 30, 33], "implement": [0, 2, 3, 4, 5, 6, 8, 9, 10, 11, 12, 13, 14, 19, 20, 21, 22, 25, 27, 30, 31, 32, 33, 35, 36, 37], "impli": [3, 5, 6, 7, 13, 24, 31, 32, 33, 34, 35], "implicit": [3, 33], "implicitli": [11, 27], "import": [0, 1, 2, 3, 4, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 21, 22, 25, 27, 33, 34, 35, 36], "importantli": 3, "importerror": [], "impos": [0, 6, 11, 12, 30, 36], "imposs": [0, 5, 30, 31, 32], "impract": 33, "impress": [0, 12, 30, 36, 37], "improv": [0, 4, 5, 9, 10, 11, 13, 15, 21, 25, 31, 32], "impur": 9, "imread": 6, "imshow": [1, 3, 4, 6], "in3050": [29, 30], "in3310": 30, "in4080": [29, 30], "in4300": [29, 30], "in4310": 29, "in5400": 3, "in5550": 29, "in_out_neuron": 4, "inaccur": [13, 32], "inact": [12, 36, 37], "inadequ": [0, 30], "inappropri": 33, "inch": [6, 31], "incident": [], "includ": [0, 1, 2, 3, 4, 5, 6, 7, 11, 12, 15, 16, 17, 18, 19, 20, 21, 22, 23, 27, 28, 29, 30, 31, 32, 34], "include_bia": [6, 9, 34, 35], "incom": [12, 16, 36, 37], "incorrect": 1, "incoveni": 8, "increas": [0, 1, 3, 4, 5, 6, 9, 12, 13, 19, 25, 27, 30, 31, 33, 34, 35, 36, 37], "increasingli": 27, "increment": 33, "ind": 6, "inde": [0, 2, 4, 5, 6, 13, 30, 31, 32, 37], "indefinit": 4, "independ": [0, 5, 6, 7, 8, 12, 13, 27, 30, 31, 32, 33, 35, 36], "index": [0, 1, 3, 4, 10, 14, 23, 24, 25, 27, 29, 30], "index_col": [0, 30], "indic": [0, 1, 3, 4, 5, 6, 9, 10, 11, 13, 16, 25, 30, 31, 37], "indirect": [], "indispens": [6, 34, 35], "individu": [1, 6, 7, 10, 12, 27, 30, 31, 33, 34, 35, 36, 37], "indu": [], "indx": 24, "indx1": 2, "indx2": 2, "indx3": 2, "ineffici": [3, 13], "inequ": [8, 13], "inequaltii": 32, "inertia": 13, "inexperi": [], "inf": [], "inf1000": [23, 30], "inf1100": [23, 30], "inf1100l": [23, 30], "inf1110": [23, 30], "inf3000": 30, "infeas": [9, 33], "infer": [0, 1, 4, 6, 29, 30, 34, 35], "inferenc": 1, "infil": [0, 6, 7, 9, 30, 34, 35], "infin": [5, 6, 7, 11, 19, 31, 32, 34, 35, 37], "infinit": [3, 33], "infinitesim": 27, "influenc": [6, 10, 18, 34, 35], "influenti": 1, "info": 30, "inform": [0, 1, 3, 4, 6, 9, 11, 12, 13, 14, 24, 25, 29, 30, 32, 33, 34, 35, 36, 37], "inforom": 15, "infrequ": 33, "infti": [3, 6, 13, 27, 32, 34, 37], "ingeni": [13, 32, 33], "ingredi": [0, 9, 30], "inher": [6, 33, 34, 35], "inherit": [24, 30, 33], "init": [], "initi": [0, 1, 2, 6, 10, 13, 14, 18, 24, 27, 30, 32, 33, 34, 35, 36, 37], "inject": 14, "inlin": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 24, 27, 30, 31, 32, 33, 34, 35, 36], "inner": [0, 13, 31], "innerhtml": [], "inp": 4, "inplac": 13, "inpput": 37, "input": [0, 1, 3, 4, 5, 6, 7, 8, 12, 13, 14, 16, 25, 27, 30, 31, 32, 33, 34, 35, 36], "input_dim": 1, "input_s": 21, "input_shap": [3, 4], "inputs": 1, "inputs_shuffl": [0, 1, 31], "inquiri": 20, "insert": [3, 5, 6, 8, 10, 27, 31, 32, 34], "insid": [4, 7, 21, 36], "insight": [0, 1, 5, 23, 30, 31, 32, 34, 35, 37], "insist": [6, 13, 31, 33], "inspir": [0, 1, 12, 25, 30, 36, 37], "instabl": 2, "instal": [0, 1, 5, 6, 9, 15, 20], "instanc": [0, 1, 2, 4, 6, 9, 11, 13, 16, 30, 31, 32, 33, 34, 35], "instanti": 10, "instead": [0, 1, 2, 3, 4, 5, 6, 8, 9, 11, 13, 14, 17, 20, 21, 22, 24, 27, 30, 31, 33, 34], "institut": 1, "instruct": [0, 1, 15], "int": [0, 1, 2, 3, 4, 5, 6, 11, 13, 14, 24, 27, 31, 33, 34, 35, 36], "int32": 10, "int_": [3, 6, 27, 34, 37], "int_0": 27, "int_a": 27, "intak": [0, 31], "integ": [1, 2, 13, 14, 24, 27, 30, 35, 36], "integer_vector": 1, "integr": [3, 6, 27, 30, 34], "intellig": [0, 14, 29, 30], "intend": 10, "intens": [1, 18], "intention": 14, "interact": [0, 6, 9, 12, 23, 25, 30, 36, 37], "intercept": [0, 6, 8, 11, 13, 16, 17, 18, 19, 30, 31, 32, 33, 34, 35, 36], "intercept_": [0, 6, 8, 9, 13, 30, 31, 33], "interchang": [5, 12, 24, 36, 37], "interconnect": 1, "interesit": [], "interest": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 12, 19, 23, 25, 27, 30, 31, 32, 34, 35, 36, 37], "interfac": [0, 1, 15, 24, 31], "interior": [0, 9, 30], "intermedi": [24, 31, 33], "intermediari": [21, 22], "intermeti": 22, "intermetidari": 22, "intern": [1, 10, 12, 22, 35, 36, 37], "internation": [], "interpol": [1, 3, 4, 6, 12, 36, 37], "interpr": [5, 31, 32], "interpret": [0, 1, 6, 9, 10, 12, 13, 15, 16, 21, 24, 25, 27, 37], "interrupt": [], "interv": [0, 3, 5, 6, 7, 13, 19, 27, 30, 31, 32, 35, 36], "intial": [13, 32], "intract": [0, 4, 31], "intrins": [3, 11, 24, 27, 30], "intro": [23, 29, 30], "introduc": [0, 1, 5, 6, 8, 10, 12, 24, 25, 27, 30, 32, 33, 34, 36, 37], "introduct": [1, 2, 4, 13, 29, 31, 32, 33, 35], "introductori": [0, 4, 24, 29, 30, 31], "intuit": [0, 5, 6, 8, 12, 13, 25, 30, 33, 34, 35, 36, 37], "inv": [0, 5, 13, 17, 30, 31, 32, 33], "invalid": [], "invalu": [0, 13, 23, 30, 32], "invari": 1, "invd": 5, "inver": [8, 36], "invers": [0, 3, 6, 13, 30, 31, 32, 33], "inverse_transform": 8, "invert": [0, 5, 7, 10, 13, 16, 18, 30, 33, 35, 36], "investig": [], "invh": [13, 33], "invok": 8, "involv": [0, 2, 6, 7, 11, 12, 30, 31, 33, 34, 35, 36, 37], "io": [0, 23, 25, 26, 28, 29, 30, 31], "ion": [], "ip": [0, 8, 27, 30], "ipca": 11, "ipynb": [23, 30], "ipython": [0, 5, 7, 9, 11, 14, 23, 25, 30, 31, 35], "iq": [6, 34], "iri": [8, 9, 21], "irreduc": [6, 34, 35], "irrelev": [5, 31, 32], "irrespect": [0, 30], "irvin": 25, "isaac": [], "isaacmus": [], "iseffici": [], "isn": 5, "isnul": [], "isolo": 22, "isomap": 11, "issu": [1, 9, 15, 24, 33], "it_arrai": 13, "item": [0, 13, 30], "items": [24, 30], "iter": [1, 2, 4, 6, 8, 13, 14, 18, 25, 27, 32, 33, 34, 35, 36, 37], "its": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 20, 21, 23, 24, 25, 27, 30, 32, 33, 34, 35, 36, 37], "itself": [5, 6, 12, 25, 27, 30, 31, 34, 37], "j": [0, 1, 2, 3, 4, 5, 6, 8, 9, 11, 12, 13, 14, 15, 16, 24, 25, 27, 29, 30, 31, 32, 33, 34, 35, 36, 37], "j1": 24, "j_": 6, "j_41hld6ttu": 34, "j_lasso_sk": 6, "j_ridge_sk": 6, "j_sk": 6, "jackknif": [6, 23, 30, 34, 35], "jacobian": [2, 13, 32], "janko": [], "jason": 4, "javascript": [], "jax": [23, 30, 33, 37], "jeff": [], "jensen": [28, 30, 31, 32, 33, 34, 35, 36, 37], "jentzen": 37, "jerom": [19, 25, 29], "jhauser": [], "ji": [12, 24, 37], "jit": 13, "jj": [0, 5, 6, 30, 34], "jk": [0, 1, 6, 12, 24, 30, 36, 37], "jl": [0, 30], "jm": 24, "jnp": 13, "job": [2, 8, 10, 15], "join": [0, 4, 6, 7, 9, 25, 30, 34, 35], "joint": [4, 5], "jonathan": [], "json": [], "judg": [13, 32, 35, 36], "judgement": 6, "julia": [23, 24, 25], "juliu": 37, "jump": [27, 33], "junk": 4, "jupit": 30, "jupyt": [0, 15, 16, 19, 23, 25, 29, 30, 34, 37], "jupyterbook": [], "jupytext": [], "just": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 20, 21, 22, 23, 27, 30, 31, 32, 33, 34, 35, 36, 37], "justif": 0, "justifi": [3, 10], "k": [0, 1, 3, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 21, 23, 24, 25, 27, 28, 30, 31, 32, 33, 36], "k0": [7, 35, 36], "k1": [7, 35, 36], "kaggl": [6, 25], "kappa_d": 27, "karl": [28, 30], "karush": 8, "katex": [], "katrin": [28, 30], "keep": [0, 1, 4, 5, 6, 11, 13, 14, 15, 18, 21, 22, 24, 25, 30, 31, 32, 33, 34, 35], "keepdim": [1, 6, 10, 24, 34, 35, 36], "kei": [1, 3, 6, 12, 33, 36], "kellei": [], "kenneth": [], "kept": [4, 6, 14, 34, 35], "kera": [0, 4, 23, 25, 30], "kernel": [0, 1, 3, 23, 30, 31], "kernel_regular": [1, 3], "kernel_s": 4, "kernelpca": 11, "kev": [0, 30], "kevin": [29, 30], "kevinsheppard": [], "keyword": [18, 24, 30], "kfold": [6, 34, 35], "kg": 1, "ki": 24, "kick": [1, 13, 33], "kiener": 2, "kilomet": [6, 31], "kim": [], "kind": [0, 2, 3, 4, 8, 12, 13, 14, 30, 31, 36, 37], "kingma": 33, "kj": [6, 12, 24, 31, 33, 37], "kjm": [23, 30], "kkt": 8, "kl": 27, "km": [12, 30, 36], "kmean": 14, "kmeanspoint": 14, "kn_k": 14, "know": [0, 1, 2, 5, 6, 8, 13, 15, 16, 17, 19, 20, 23, 30, 31, 32], "knowledg": [0, 23, 30], "known": [1, 3, 4, 5, 6, 7, 8, 9, 12, 18, 24, 25, 27, 29, 31, 33, 34, 35, 36, 37], "kondev": [0, 30], "kp": 27, "kpca": 11, "kramdown": [], "kroneck": 14, "kt": [], "kuckuck": 37, "kuhn": 8, "kutyniok": 37, "kvalsund": [28, 30], "kwown": [0, 30], "l": [0, 1, 2, 3, 5, 6, 7, 8, 10, 11, 12, 13, 22, 24, 25, 27, 30, 32, 33, 35, 36], "l0": [7, 35, 36], "l1": [0, 1, 3, 7, 30, 35, 36], "l1_l2": [1, 3], "l1regl": 5, "l2": [1, 3], "l_": [24, 33], "l_1": [7, 35, 36, 37], "l_2": [7, 13, 32, 33, 35, 36, 37], "l_i": 33, "l_j": [12, 37], "la": 13, "la_": [], "la_i": [12, 37], "la_k": [12, 37], "lab": [20, 23, 25, 30], "label": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 12, 13, 14, 15, 20, 23, 24, 25, 27, 30, 31, 32, 33, 34, 35, 36, 37], "labelencod": [7, 10, 36], "labels": [6, 8, 9], "labels_shuffl": [0, 1, 31], "laboratori": 26, "lack": [0, 30, 33], "lagari": 2, "lagrang": [8, 11], "lam": 18, "lambda": [0, 1, 2, 3, 5, 6, 7, 8, 10, 12, 13, 17, 18, 19, 20, 25, 27, 30, 31, 32, 33, 34, 35, 36], "lambda_": 11, "lambda_0": 11, "lambda_1": [5, 8, 11, 31, 32], "lambda_2": [8, 11], "lambda_i": [8, 11], "lambda_iy_i": 8, "lambda_jy_iy_j": 8, "lambda_k": 8, "lambda_n": [5, 8, 31, 32], "lamda": 1, "land": 8, "landmark": 8, "landscap": [13, 18, 32, 33], "langl": [0, 6, 11, 27, 30, 31], "languag": [0, 1, 4, 8, 23, 24, 25, 29, 30], "lapack": [24, 30], "laplac": 5, "laptop": [15, 23], "larg": [0, 1, 2, 4, 5, 6, 8, 9, 10, 11, 13, 18, 23, 24, 25, 27, 29, 30, 31, 32, 33, 34, 35, 37], "larger": [0, 3, 5, 6, 8, 10, 11, 13, 17, 22, 27, 30, 31, 32, 33, 34], "largest": [4, 8, 11], "lasso": [0, 7, 23, 30, 33, 34, 35, 36], "lasso_sk": 6, "last": [0, 1, 3, 4, 5, 6, 7, 8, 12, 16, 17, 19, 21, 22, 24, 25, 27, 28, 30, 32, 34, 35], "latent": 4, "latent_dim": 4, "latent_point": 4, "latent_space_value_rang": 4, "later": [0, 1, 4, 7, 8, 12, 13, 14, 15, 19, 21, 22, 23, 25, 30, 33, 35, 36, 37], "latest": [4, 15, 23], "latest_checkpoint": 4, "latex": [20, 30], "latexcodec": [], "latrpygrtttbnjr3znuhl": 22, "latter": [0, 3, 6, 7, 8, 11, 13, 24, 27, 30, 31, 32, 33, 34, 35, 36, 37], "lattic": [12, 36, 37], "law": 0, "layer": [0, 4, 13, 30, 33, 36], "layer_grad": 22, "layer_input": 22, "layer_output_s": [21, 22], "layers_grad": 21, "lbfg": [7, 9, 10, 36], "lc_messag": [], "lcc": [5, 6, 34], "lda": 11, "ldot": [0, 6, 11, 25, 30, 34, 35], "le": [5, 7, 10, 13, 17, 27, 31, 32, 33, 35], "lead": [0, 1, 3, 5, 6, 7, 8, 9, 10, 11, 12, 13, 16, 17, 21, 22, 24, 27, 30, 31, 32, 33, 34, 35, 36, 37], "leaf": 9, "leaki": 1, "leakyrelu": 4, "lear": [13, 32], "learn": [3, 4, 5, 6, 7, 8, 9, 10, 12, 21, 24, 28, 29], "learnabl": 3, "learner": 10, "learnig": 30, "learning_r": [8, 10, 21], "learning_rate_init": [0, 1, 30], "learning_schedul": [13, 33], "learnt": 25, "least": [0, 7, 8, 10, 11, 17, 18, 23, 24, 27, 34, 35, 36], "leat": [13, 33], "leav": [0, 1, 3, 5, 6, 9, 11, 21, 30, 32, 34, 35], "lectur": [0, 1, 5, 10, 11, 12, 13, 23, 24, 25, 26, 28, 29, 31], "lecturenot": [0, 23, 25, 29, 30], "left": [0, 1, 2, 3, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 16, 17, 19, 24, 25, 27, 30, 31, 32, 33, 34, 35, 36, 37], "leftarrow": [8, 12, 37], "legend": [0, 2, 3, 4, 5, 6, 7, 8, 9, 10, 13, 15, 21, 30, 31, 32, 33, 34, 35, 36], "legend_el": 21, "leinonen": 30, "len": [0, 1, 2, 3, 4, 5, 6, 8, 9, 10, 11, 12, 16, 17, 21, 22, 24, 30, 31, 32, 33, 34, 35, 36], "length": [0, 1, 3, 4, 8, 9, 13, 16, 21, 23, 30, 31, 32, 33], "length_of_sequ": 4, "leq": [0, 5, 7, 8, 13, 14, 27, 30, 31, 32, 33, 35], "less": [0, 1, 3, 4, 5, 6, 8, 9, 13, 23, 27, 30, 31, 32, 33, 34, 35], "lessen": 1, "let": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 19, 22, 24, 27, 30, 31, 32, 33, 34, 35, 36, 37], "letter": [0, 16, 24, 27, 30, 31], "level": [0, 1, 5, 6, 9, 23, 24, 25, 26, 28, 30, 33, 34, 35, 37], "leverag": 33, "lexer": [], "li": [8, 11], "liabil": [], "liabl": [], "lib": [], "liberti": 33, "liblinear": 10, "librari": [0, 1, 2, 3, 4, 5, 6, 9, 10, 11, 24, 25, 27, 29, 31, 32, 33], "licenc": [], "licens": [0, 1, 23, 25, 30], "lie": [0, 6, 11, 27, 30, 31, 34, 35], "life": [0, 1, 8, 12, 30, 36, 37], "lifetim": 13, "light": [], "like": [0, 1, 2, 3, 4, 5, 6, 7, 9, 10, 11, 12, 13, 15, 16, 20, 21, 22, 23, 24, 25, 27, 30, 31, 32, 33, 34, 35, 36, 37], "likelihood": [0, 1, 5, 9, 30, 31], "lim_": 27, "limit": [0, 5, 6, 8, 12, 24, 25, 30, 31, 35, 36, 37], "lin_clf": 8, "lin_model": [], "lin_reg": 9, "linalg": [0, 2, 5, 6, 8, 11, 13, 17, 24, 27, 30, 31, 32, 33, 36], "line": [0, 3, 6, 8, 11, 13, 15, 16, 20, 21, 30, 32, 33, 34, 37], "line1": 8, "line2": 8, "line2d": [], "line3": 8, "line_model": 15, "line_ms": 15, "line_predict": 15, "linear": [1, 3, 5, 6, 7, 9, 10, 11, 12, 16, 17, 18, 19, 21, 23, 25, 27, 33, 34, 36, 37], "linear_model": [0, 5, 6, 7, 8, 9, 10, 11, 13, 15, 16, 19, 30, 31, 32, 33, 34, 35, 36], "linear_regress": [6, 34, 35], "linearli": [5, 31, 32, 33], "linearloc": [6, 13, 32, 33], "linearregress": [0, 6, 7, 9, 15, 16, 19, 30, 31, 33, 34, 35], "linearsvc": 8, "lineat": 32, "liner": [1, 3], "linerar": 10, "linewidth": [0, 2, 4, 6, 8, 9, 10, 34], "link": [0, 4, 9, 12, 15, 20, 21, 23, 25, 26, 28, 30, 35, 37], "linlag": 5, "linpack": [24, 30], "linreg": [0, 30], "linspac": [0, 2, 3, 4, 6, 8, 9, 10, 13, 16, 17, 19, 24, 27, 30, 31, 33, 34, 35], "linu": 4, "linux": [0, 1, 23, 25, 30], "liquid": [0, 30], "list": [1, 2, 3, 4, 9, 15, 21, 22, 23, 25, 30, 33, 36], "listedcolormap": [9, 10], "literatur": [1, 7, 14, 29, 34, 35], "littl": [1, 3, 9, 12, 22, 33, 37], "live": [8, 16], "ll": [0, 18, 27, 30, 31], "lle": [0, 31], "llm": 20, "lloyd": [4, 14], "lmb": [0, 2, 5, 6, 31, 32, 33, 34, 35], "lmbd": [0, 1, 3, 30], "lmbd_val": [0, 1, 3, 30], "lmbda": [13, 32, 33], "ln": [1, 13, 32], "load": [1, 4, 6, 7, 9, 10, 33, 36], "load_boston": [], "load_breast_canc": [1, 7, 9, 10, 11, 36], "load_data": [3, 4], "load_digit": [1, 3], "load_iri": [8, 9, 21], "loc": [3, 6, 7, 8, 9, 10, 21, 30, 34, 35, 36], "local": [0, 1, 3, 7, 12, 13, 15, 21, 22, 31, 32, 33, 35, 36, 37], "locat": [2, 3, 8, 15], "log": [0, 1, 2, 4, 5, 6, 7, 9, 10, 11, 13, 15, 20, 21, 24, 25, 30, 33, 34, 35, 36], "log10": [0, 5, 6, 31, 32, 33, 34, 35], "log_": [0, 30], "log_clf": 10, "logarithm": [0, 5, 7, 17, 24, 30, 34, 35, 36], "logbook": 25, "logic": [0, 1, 9, 30], "logical_or": [], "login": 15, "logist": [0, 1, 2, 8, 9, 10, 11, 12, 13, 23, 31, 32, 33, 37], "logisticregress": [7, 9, 10, 11, 35, 36], "logit": [7, 35, 36], "logreg": [7, 9, 10, 11, 36], "logspac": [0, 1, 3, 5, 6, 30, 31, 32, 33, 34, 35], "long": [0, 1, 3, 4, 12, 13, 21, 30, 32, 33, 36, 37], "longer": [2, 3, 8, 10, 14, 24, 27, 30, 33], "loocv": [6, 34, 35], "look": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 13, 15, 16, 19, 20, 24, 25, 27, 30, 31, 32, 33, 34, 35], "loop": [1, 4, 6, 10, 12, 14, 16, 17, 18, 22, 23, 24, 30, 33, 34, 35], "lose": 1, "loss": [0, 1, 3, 4, 5, 6, 7, 8, 10, 11, 13, 18, 21, 24, 25, 30, 34, 35, 36, 37], "loss_bin": [35, 36], "loss_fil": 4, "loss_multi": [35, 36], "loss_vec": [35, 36], "lossfil": 4, "lost": 4, "lot": [1, 4, 6, 16, 19, 20, 33, 34], "low": [0, 6, 9, 10, 11, 25, 30, 31, 34, 35], "lower": [0, 1, 3, 6, 9, 10, 16, 21, 24, 31, 33], "lowercas": [24, 30], "lowest": [9, 13, 27, 33], "lr": [1, 3, 4, 10, 35, 36], "lstat": [], "lstm": 4, "lstm_2layer": 4, "lstsq": [0, 30, 31], "lt": [6, 34], "lu": [0, 5, 30, 31, 32], "lubksb": 24, "luckili": 2, "ludcmp": 24, "lux": 24, "lvert": 1, "lw": [0, 30], "m": [0, 1, 2, 3, 5, 6, 8, 9, 10, 11, 12, 13, 15, 24, 27, 28, 29, 30, 31, 32, 33, 34, 36, 37], "m_": [9, 12, 37], "m_0": 33, "m_1": 14, "m_h": [0, 30], "m_k": 14, "m_l": [12, 37], "m_n": [0, 30], "m_p": [0, 30], "m_t": [13, 33], "ma": 11, "machin": [1, 3, 4, 5, 6, 7, 9, 10, 11, 12, 15, 16, 24, 29, 31, 33, 34, 37], "machinelearn": [0, 6, 16, 20, 23, 25, 26, 28, 29, 30, 31, 32, 35, 36], "machineri": [], "mackai": 29, "macro": [], "made": [0, 1, 3, 4, 5, 6, 7, 9, 11, 12, 25, 30, 31, 33, 35, 36, 37], "mae": [0, 30], "magic": 4, "magnitud": [1, 6, 7, 13, 21, 31, 33, 36, 37], "mai": [0, 1, 2, 3, 5, 6, 7, 8, 9, 11, 12, 13, 19, 23, 24, 25, 27, 30, 31, 32, 33, 34, 35, 36, 37], "mail": [26, 28], "main": [0, 1, 3, 4, 5, 6, 7, 9, 24, 25, 29, 31, 32, 33, 35, 36], "mainli": [0, 5, 6, 7, 9, 30, 31, 34, 35, 36], "maintain": [6, 33, 34], "major": [1, 6, 9, 10, 13, 24, 30, 32, 33, 34, 35], "make": [1, 2, 3, 4, 5, 6, 7, 8, 11, 12, 13, 15, 16, 18, 19, 21, 22, 23, 24, 25, 27, 29, 30, 32, 33, 34, 35, 36, 37], "make_axes_locat": 6, "make_classif": 36, "make_moon": [8, 9, 10], "make_pipelin": [0, 6, 10, 31, 34, 35], "makedir": [0, 6, 7, 9, 30, 34, 35], "malcondit": 24, "malign": [1, 7, 9, 36], "mammographi": 5, "manag": [0, 2, 3, 15, 23, 25, 30, 33], "mandatori": [28, 30], "mani": [0, 1, 3, 4, 5, 6, 7, 8, 9, 11, 13, 14, 15, 16, 17, 18, 19, 21, 22, 23, 24, 25, 27, 29, 30, 31, 32, 33, 34, 35, 36, 37], "manifold": 11, "manner": 3, "manual": [6, 21, 22, 31, 33], "map": [0, 1, 2, 6, 7, 8, 11, 12, 14, 27, 30, 35, 36], "marc": 31, "marchant": [], "margin": [0, 5, 8], "marit": [0, 30], "mark": 30, "markdownfil": [], "markdownit": [], "markdownitdeflist": [], "markedli": [], "marker": [7, 24, 30, 35], "markov": [23, 30], "markup": [], "marsaglia": 27, "mask_or": [], "masked_arrai": [], "maskedrecord": [], "mass": [0, 1, 5, 13, 31, 32], "massag": [0, 30], "masses2016": [0, 30], "masses2016ol": [0, 30], "masses2016tre": 0, "masseval2016": [0, 30], "master": [26, 28], "mat": [23, 30], "mat1100": [23, 30], "mat1110": [23, 30], "mat1120": [23, 30], "match": [1, 4, 5, 13, 14, 15, 31, 32, 33], "materi": [4, 5, 7, 13, 15, 24, 26, 28, 36], "math": [3, 7, 12, 13, 24, 27, 29, 30, 33, 35, 36], "mathbb": [0, 4, 5, 6, 7, 8, 11, 12, 13, 14, 17, 19, 24, 25, 27, 30, 31, 32, 33, 34, 35, 36, 37], "mathbf": [0, 5, 6, 7, 8, 13, 19, 24, 25, 30, 31, 32, 33, 34, 35, 36, 37], "mathcal": [1, 5, 6, 7, 13, 25, 34, 35, 36], "matheemat": 3, "mathemat": [0, 6, 11, 12, 13, 21, 23, 24, 27, 29, 30, 33], "mathemati": 30, "mathrm": [0, 1, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 17, 18, 19, 25, 27, 30, 31, 32, 33, 34, 35, 36, 37], "matmul": [1, 2, 5, 37], "matnat": 29, "matplotlib": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 17, 19, 21, 22, 23, 24, 25, 27, 30, 31, 32, 33, 34, 35, 36], "matplotlibrc": [], "matric": [0, 1, 3, 4, 6, 7, 8, 11, 13, 16, 17, 23, 31, 32, 35, 36, 37], "matrix": [0, 2, 3, 4, 6, 7, 8, 10, 13, 17, 18, 19, 21, 25, 27, 34, 35, 37], "matshow": 1, "matter": [2, 3, 13, 31, 32, 33, 37], "matthia": [], "max": [0, 1, 2, 3, 4, 9, 10, 12, 13, 21, 28, 30, 32, 33, 35, 36, 37], "max_depth": [0, 9, 10], "max_diff": 2, "max_diff1": 2, "max_diff2": 2, "max_it": [0, 1, 8, 13, 30, 36], "max_iter": 14, "max_leaf_nod": 10, "max_sampl": 10, "maxdegre": [0, 6, 10, 31, 34, 35], "maxdepth": 10, "maxim": [1, 4, 5, 7, 8, 11, 34, 35, 36], "maximum": [0, 2, 3, 5, 7, 8, 9, 10, 13, 14, 30, 31, 32, 33], "maxpolydegre": [5, 6, 31, 32, 33, 34, 35], "maxpooling2d": 3, "mbox": [5, 6, 31, 32, 34], "mcculloch": [12, 36, 37], "md": 11, "mdoel": 4, "me": [], "mean": [1, 2, 3, 4, 5, 6, 7, 9, 10, 11, 12, 13, 14, 15, 17, 18, 19, 22, 23, 24, 25, 27, 30, 33, 34, 36, 37], "mean0": [35, 36], "mean1": [35, 36], "mean_absolute_error": [0, 30], "mean_divisor": 14, "mean_i": 27, "mean_matrix": 14, "mean_squared_error": [0, 4, 6, 7, 10, 15, 19, 30, 31, 34, 35], "mean_squared_log_error": [0, 30], "mean_vector": 14, "mean_x": 27, "meaning": [0, 4, 7, 30, 35], "meansquarederror": [0, 30], "meant": [3, 7, 10, 13, 35, 37], "meanwhil": 33, "measur": [0, 1, 2, 5, 6, 9, 11, 12, 14, 16, 18, 25, 27, 30, 31, 33, 34, 35, 37], "mechan": [0, 4, 27, 30, 33], "median": [0, 30, 31, 33], "medicin": [12, 36, 37], "medium": [4, 8, 13, 33], "medv": [], "meet": [0, 28], "mehta": [0, 30, 31, 32], "member": [20, 25], "memori": [3, 4, 11, 12, 13, 18, 24, 36, 37], "mentat": [], "mention": [0, 12, 13, 25, 27, 30, 32, 33, 36, 37], "merchant": [], "mere": [0, 25], "merg": [], "meshgrid": [2, 5, 6, 8, 9, 10, 11], "mess": 15, "messag": [5, 13], "messi": 2, "messier": 22, "met": [0, 3, 8, 31], "meta": [], "meteorolog": 9, "meter": [6, 31], "method": [0, 1, 2, 3, 4, 5, 7, 8, 11, 12, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 27, 29, 31, 37], "metion": 6, "metric": [0, 1, 3, 6, 7, 9, 10, 14, 15, 21, 22, 30, 31, 34, 35, 36], "metropoli": [23, 30], "mev": [0, 27, 30], "mgd": [13, 33], "mglearn": [23, 30], "mgrid": 13, "mhjensen": [], "mi": 10, "mia": [28, 30], "michael": 37, "microsoft": 29, "mid": 1, "midel": 4, "midnight": [15, 21, 22], "midpoint": 9, "might": [0, 1, 2, 4, 6, 9, 13, 15, 17, 18, 22, 31, 32, 33], "migth": 17, "mild": 9, "millimet": [6, 31], "million": [0, 30, 31, 33], "mimic": [12, 36, 37], "min": [0, 2, 5, 8, 9, 32], "min_": [0, 2, 5, 14, 17, 30, 31, 32], "min_samples_leaf": 9, "mind": [0, 6, 13, 15, 18, 21, 30, 31, 32, 33, 34], "mindboard": 4, "mine": [23, 30], "mini": [1, 11, 12, 13, 32], "minibatch": [1, 11, 13], "minibathc": [13, 33], "miniforge3": [], "minim": [0, 1, 2, 3, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 16, 17, 31, 32, 33, 34], "minima": [0, 1, 7, 13, 30, 32, 33, 35, 36], "minimum": [0, 1, 2, 6, 8, 9, 11, 13, 31, 32, 33, 34, 35, 36], "minmaxscal": [0, 31, 33], "minor": 27, "minst": 1, "minu": [7, 35], "mirjalili": 30, "mirror": 9, "misc": 6, "misclassif": [8, 9, 10], "misclassifi": [8, 10], "miser": 0, "mismatch": 1, "miss": [7, 10], "mistak": [4, 19], "mit": 29, "mitig": 33, "mix": [1, 2, 30], "mixtur": [13, 33], "mk": [9, 24], "mkdir": [0, 6, 7, 9, 30, 34, 35], "ml": [0, 1, 10, 13, 24, 25, 31, 32, 33], "mlab": 27, "mle": [5, 7, 35, 36], "mlp": [1, 36, 37], "mlpclassifi": [1, 36], "mlpregressor": [0, 30], "mm": 24, "mml": 31, "mn": [12, 27, 36], "mnist": [1, 11], "mo": [], "mod": 27, "mode": [26, 28, 30, 35, 36], "model": [2, 3, 5, 7, 8, 9, 10, 11, 13, 14, 16, 18, 19, 20, 21, 23, 25, 27, 29, 31, 32, 33, 34, 35], "model_bin": [35, 36], "model_multi": [35, 36], "model_select": [0, 1, 3, 5, 6, 7, 9, 10, 11, 15, 16, 17, 19, 30, 31, 32, 33, 34, 35, 36], "moder": [10, 33], "modern": [0, 6, 7, 23, 30, 33, 34, 35, 36, 37], "modest": 33, "modif": [2, 12, 13], "modifi": [0, 1, 3, 5, 7, 8, 10, 12, 13, 30, 31, 32, 33, 35, 36, 37], "modul": [0, 16, 24, 30], "modular": 27, "modulo": 27, "moe": [11, 31], "moment": [5, 6, 13, 27, 34], "momentum": [22, 37], "mondai": [28, 30, 35], "monitor": [13, 33], "monoton": [5, 12, 27, 34, 36, 37], "mont": [0, 6, 23, 27, 29, 30, 34, 35], "montli": 16, "moor": [5, 6], "more": [0, 1, 2, 4, 5, 7, 8, 9, 10, 11, 12, 13, 14, 16, 17, 19, 21, 22, 23, 27], "moreov": [0, 3], "morten": [28, 30, 31, 32, 33, 34, 35, 36, 37], "mortenhj": 30, "most": [0, 1, 3, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 21, 22, 23, 25, 27, 30, 31, 32, 33, 34, 35, 36, 37], "mostli": [1, 11, 18, 33], "motion": [0, 13], "motiv": [1, 4, 37], "moulin": 33, "move": [0, 4, 5, 6, 7, 9, 12, 13, 14, 15, 16, 21, 22, 25, 27, 31, 32, 34, 35, 36, 37], "mpl": [7, 30, 35], "mpl_toolkit": [2, 6, 13, 32, 33], "mplot3d": [2, 6, 13, 32, 33], "mplregressor": 1, "mr_": [], "mrecord": [], "ms3tv8fvar": 36, "mse": [0, 4, 5, 6, 9, 10, 15, 16, 17, 19, 20, 22, 25, 30, 31, 32, 33, 34, 35], "mse_der": 22, "mse_simpletre": 10, "mselassopredict": [5, 32], "mselassotrain": [5, 32], "mseownridgepredict": [6, 31, 32, 33], "msepredict": [5, 32], "mseridgepredict": [0, 5, 6, 31, 32, 33], "msetrain": [5, 32], "msg": [], "msle": [0, 30], "mt": [7, 12, 35, 36], "mu": [0, 6, 11, 13, 27, 30, 33, 34], "mu0": 27, "mu1": 27, "mu2": 27, "mu_": [6, 27, 31, 33, 34], "mu_i": [6, 31, 33], "mu_n": 11, "mu_x": 27, "much": [0, 1, 2, 3, 4, 5, 6, 8, 9, 10, 11, 12, 13, 15, 20, 21, 22, 24, 25, 27, 30, 31, 32, 33, 34, 35, 37], "multi": [0, 1, 3, 7, 23, 30, 35], "multi_class": [35, 36], "multiclass": [1, 7, 35, 36], "multiclass_result": [35, 36], "multidimension": [11, 12, 30, 36, 37], "multilay": 1, "multinomi": [7, 35, 36], "multipl": [2, 4, 5, 6, 7, 12, 13, 15, 22, 27, 31, 32, 33, 34, 35, 36, 37], "multipli": [3, 5, 6, 11, 13, 18, 22, 24, 27, 31, 32, 33], "multiplum": 8, "multivari": [0, 2, 10, 11, 23, 27, 30], "multivariate_norm": [11, 14], "multpli": 16, "murphi": [11, 29, 30], "muse": [], "must": [1, 2, 5, 6, 8, 10, 12, 13, 14, 15, 20, 22, 25, 27, 31, 32, 33, 34, 35, 36, 37], "mutat": [7, 35, 36], "mutual": [1, 3, 6, 13, 34, 35], "mx_": 27, "my": 30, "myenv": [], "myriad": [0, 23, 30], "myself": [], "mz1": 27, "mz2": 27, "m\u00f8svatn": 6, "n": [0, 1, 2, 3, 4, 5, 6, 7, 8, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 24, 25, 27, 30, 31, 32, 33, 34, 35, 36, 37], "n0": [35, 36], "n1": [24, 35, 36], "n2": 24, "n8grai": [], "n_": [1, 2, 3, 8, 12, 27, 36], "n_0": [12, 27, 36], "n_boostrap": [6, 10, 34, 35], "n_bootstrap": [6, 34], "n_categori": [1, 3], "n_class": [35, 36], "n_cluster": 14, "n_compon": 11, "n_epoch": [13, 33], "n_estim": 10, "n_examples_to_gener": 4, "n_featur": [1, 18, 35, 36, 37], "n_filter": 3, "n_hidden": 2, "n_hidden_neuron": [0, 1, 30, 37], "n_i": 27, "n_input": [0, 1, 3, 31, 37], "n_instanc": 9, "n_iter": 33, "n_job": 10, "n_k": 14, "n_l": [12, 27, 36], "n_layer": 1, "n_m": 9, "n_neuron": 1, "n_neurons_connect": 3, "n_neurons_layer1": 1, "n_neurons_layer2": 1, "n_output": 37, "n_point": 14, "n_sampl": [6, 8, 9, 10, 14, 18, 34, 35, 36], "n_split": [6, 34, 35], "n_step": 4, "n_t": 2, "n_x": 2, "nabla": [1, 13, 32, 33], "nabla_": [2, 13, 32, 33], "nabla_w": 13, "nag": 13, "naimi": [0, 30], "naiv": [7, 35, 36], "naive_kmean": 14, "name": [0, 1, 3, 4, 5, 6, 7, 8, 9, 10, 12, 13, 14, 15, 18, 20, 21, 23, 24, 25, 27, 28, 30, 31, 32, 34, 35, 36, 37], "namespac": [], "nan": [], "narrow": [13, 33], "nathaniel": [], "nation": [1, 5], "nativ": [23, 30], "natur": [0, 1, 4, 8, 9, 12, 13, 25, 27, 29, 30, 32, 33, 36, 37], "navier": [12, 36, 37], "navig": [15, 33], "nb": 27, "nb_": 24, "nbconvert": 30, "nd": 14, "ndarrai": 6, "ne": [9, 10, 24, 27, 31, 32], "nearest": [1, 3, 6, 11], "nearli": [13, 32], "neat": 30, "neccesari": [6, 34], "necess": 2, "necessari": [0, 1, 3, 4, 8, 14, 18, 30, 37], "necessarili": [0, 4, 11, 27, 30], "necesserali": 5, "neck": [7, 35, 36], "need": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 18, 19, 20, 21, 22, 24, 27, 31, 32, 33, 34, 35, 36, 37], "neg": [0, 1, 3, 5, 6, 7, 10, 13, 24, 27, 30, 32, 34, 35, 36], "neg_mean_squared_error": [6, 34, 35], "neglect": [27, 33], "neglig": 27, "neighbor": [3, 6, 11], "neither": [4, 13, 33], "neq": [13, 14, 27, 32], "nervou": [12, 36, 37], "nest": [9, 12, 36], "nesterov": 13, "net": [2, 4, 12, 36, 37], "netlib": [24, 30], "network": [0, 9, 13, 21, 22, 23, 29, 31], "network_input_s": [21, 22], "neural": [0, 13, 21, 22, 23, 29, 31, 35], "neural_network": [0, 1, 2, 30, 36], "neuralnetwork": [1, 22], "neuralnetworksanddeeplearn": 37, "neuron": [1, 2, 3, 4, 12], "neutral": [0, 30], "neutron": [0, 30], "never": [1, 4, 6, 9, 27, 34, 35], "new": [0, 1, 2, 3, 5, 6, 7, 8, 9, 10, 11, 13, 14, 15, 17, 20, 22, 24, 30, 31, 32, 33, 35, 36], "new_chang": [13, 33], "new_hobbit": 30, "new_ma": [], "newaxi": [0, 3, 6, 9, 21, 34, 35], "newli": [0, 30], "newlin": [35, 36], "newton": [1, 7, 8, 13, 27, 37], "next": [0, 1, 2, 3, 4, 5, 6, 8, 9, 13, 14, 15, 16, 21, 22, 30, 31, 32, 33, 34, 36, 37], "next_guess": 13, "next_input": 4, "ng": 1, "ni": 14, "nice": [0, 1, 5, 11, 22, 30, 31, 32], "nicer": [18, 33], "nielsen": 37, "nine": 37, "nip": 33, "niter": [13, 32, 33], "nitric": [], "nlambda": [0, 5, 6, 31, 32, 33, 34, 35], "nlp": 29, "nm": 27, "nm_n": [0, 30], "nmse": [6, 34, 35], "nn": [2, 5, 6, 12, 24, 30, 34, 36], "nn_model": 1, "nnmin": 2, "node": [1, 3, 9, 10, 12, 21, 36], "nois": [0, 4, 5, 6, 8, 9, 10, 13, 18, 19, 25, 30, 31, 32, 33, 34, 35], "noise_dimens": 4, "noisi": [1, 6, 25, 33, 34, 35], "nomask": [], "non": [0, 1, 3, 5, 6, 7, 9, 10, 11, 12, 13, 14, 18, 21, 24, 27, 30, 31, 32, 34, 35, 36, 37], "nondifferenti": 33, "none": [0, 1, 2, 4, 5, 9, 10, 13, 27, 30, 31, 35, 36, 37], "noninfring": [], "nonlinear": [3, 6, 8, 9, 11, 12, 34, 35, 36, 37], "nonneg": [6, 9, 13, 32, 34, 35], "nonparametr": 6, "nonsens": 27, "nonsingular": 24, "nonumb": [3, 7, 8, 13, 24, 35, 36], "nor": [1, 4, 13, 22, 33, 37], "norm": [0, 1, 5, 6, 8, 11, 13, 18, 30, 31, 32, 33, 34, 37], "normal": [3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 16, 17, 18, 19, 21, 23, 24, 25, 27, 30, 31, 32, 33, 35, 36, 37], "normali": [24, 30], "norwai": [6, 25, 30, 32, 33, 34, 36, 37], "notabl": [], "notat": [0, 2, 5, 6, 13, 14, 27, 30, 31, 32, 34, 35, 37], "note": [0, 1, 2, 3, 4, 5, 6, 7, 8, 11, 12, 13, 14, 15, 16, 18, 22, 23, 24, 27, 29, 30, 33, 34, 35, 36, 37], "notebook": [0, 1, 3, 9, 15, 16, 19, 20, 21, 22, 23, 25, 30, 34, 37], "noteworthi": 33, "noth": [1, 2, 5, 8, 12, 14, 27, 31, 32, 36], "notic": [4, 5, 12, 13, 22, 24, 27, 30, 37], "notion": 3, "novel": [3, 6, 10, 30], "novemb": [1, 28, 30], "now": [0, 2, 4, 5, 6, 7, 8, 10, 11, 12, 14, 15, 16, 19, 21, 22, 23, 24, 25, 27, 30, 31, 36, 37], "nowadai": [0, 1, 3, 9, 23, 30], "nox": [], "np": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 13, 14, 15, 16, 17, 18, 19, 21, 22, 24, 27, 30, 31, 32, 33, 34, 35, 36, 37], "npm": [], "npr": 2, "nsampl": [6, 34, 35], "nt": 2, "nu": 27, "nuclear": [5, 31, 32], "nuclei": [0, 27, 30], "nucleon": [0, 30], "nucleu": [0, 30], "num": 4, "num_coordin": 2, "num_hidden_neuron": 2, "num_it": [2, 18], "num_neuron": 2, "num_neurons_hidden": 2, "num_point": 2, "num_tre": 10, "num_valu": 2, "number": [1, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 16, 17, 18, 19, 21, 24, 25, 26, 28, 30, 32, 34, 35, 36], "numberid": [7, 35], "numberparamet": 3, "numer": [0, 5, 6, 9, 10, 11, 12, 13, 21, 23, 24, 29, 30, 31, 32, 33, 34, 35, 36, 37], "numpi": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 21, 22, 23, 25, 27, 31, 32, 33, 34, 35, 36, 37], "numpydocstr": [], "nunmpi": [5, 31], "nve_frngahw": 32, "nx": 2, "ny": 27, "o": [0, 1, 4, 5, 6, 7, 8, 9, 11, 24, 28, 29, 30, 31, 32, 33, 34, 35, 36], "obei": [6, 11, 13, 31, 33], "object": [0, 1, 4, 8, 10, 15, 19, 24, 30, 33, 37], "obliqu": [5, 31, 32], "observ": [0, 1, 3, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 27, 30, 32, 33, 34, 35, 36], "obtain": [0, 1, 5, 6, 7, 8, 9, 10, 12, 13, 14, 17, 24, 25, 27, 30, 31, 32, 33, 34, 35, 36, 37], "obviou": [5, 6, 11, 27, 31, 32], "obviouli": 30, "obvious": [0, 4, 5, 6, 24, 30, 34], "oc": [31, 32], "occupi": [], "occur": [0, 6, 8, 9, 24, 27, 30], "octob": [21, 22, 28, 30, 36], "od": 0, "odd": [0, 3, 7, 30, 31, 33, 35, 36], "odenum": 2, "odesi": 2, "oen": 0, "off": [1, 3, 4, 5, 9, 13, 20, 27, 33, 34], "offer": [6, 11, 23, 24, 26, 28, 30, 34, 35], "offic": [28, 30], "offici": [26, 30], "offlin": [21, 22], "often": [0, 1, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 18, 19, 21, 23, 24, 25, 27, 30, 31, 32, 33, 34, 35, 36, 37], "ofter": [24, 30], "ol": [0, 13, 17, 19, 31, 33, 35], "old": [1, 5, 10, 13, 15, 18, 35, 36], "old_ma": [], "oliph": [], "ols_paramet": 16, "ols_sk": 6, "ols_svd": 6, "olsbeta": 32, "olstheta": [0, 5], "omega": [2, 3, 6], "omega_0": 3, "omit": [0, 5, 30, 31, 32, 34], "onc": [1, 6, 9, 11, 13, 20, 34, 35], "one": [0, 1, 3, 4, 5, 6, 7, 8, 9, 10, 11, 13, 14, 15, 19, 20, 21, 23, 24, 25, 27, 28, 30, 31, 33, 34, 35, 36], "one_hot": [35, 36], "one_hot_predict": 21, "onehot": 1, "onehot_vector": 1, "onehotencod": 9, "ones": [0, 2, 5, 6, 8, 9, 10, 11, 13, 16, 18, 21, 22, 24, 25, 30, 31, 32, 33, 34, 35, 37], "ones_lik": 4, "ong": 31, "onl": 3, "onli": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 17, 18, 19, 20, 21, 22, 24, 25, 27, 30, 31, 32, 33, 34, 35, 36, 37], "onlin": [11, 15, 20, 26, 33, 37], "onto": [5, 11, 31, 32], "open": [0, 1, 4, 6, 7, 9, 15, 23, 25, 26, 28, 30, 34, 35, 36], "oper": [0, 1, 3, 5, 6, 10, 11, 12, 13, 15, 16, 21, 22, 23, 27, 30, 31, 32, 33, 34, 36], "operation": 27, "oplu": 27, "opmiz": [13, 33], "opportun": 0, "oppos": [6, 13], "opposit": [1, 5, 8, 31, 32], "opt": [1, 5, 25, 30, 32], "optim": [0, 2, 3, 4, 5, 6, 7, 9, 10, 11, 14, 16, 17, 19, 21, 22, 25, 34], "optimis": [1, 3], "option": [0, 1, 3, 5, 6, 8, 11, 15, 18, 24, 31, 33, 34], "optmiz": [1, 8, 13, 31], "oral": 30, "orang": 0, "order": [0, 1, 2, 3, 5, 6, 7, 8, 9, 10, 11, 12, 15, 19, 21, 24, 25, 27, 30, 31, 32, 34, 35, 36, 37], "ordinari": [0, 2, 3, 7, 11, 13, 17, 18, 23, 34, 35, 36], "oreilli": [29, 30], "org": [0, 3, 4, 16, 20, 21, 23, 24, 25, 29, 30, 31, 32, 33, 37], "organ": [6, 7, 10, 24, 34, 35], "orgin": 37, "orient": [1, 5, 27, 31, 32], "origin": [0, 3, 5, 6, 8, 11, 12, 13, 15, 24, 30, 31, 32, 33, 34, 35, 36], "orthogn": [5, 31, 32], "orthogon": [0, 5, 6, 8, 11, 13, 24, 30, 31, 32], "orthonorm": [5, 31, 32], "os": [28, 30], "oscar": 1, "oscil": [3, 13, 33], "oskar": 30, "oskarlei": 30, "osl": 18, "oslo": [0, 23, 25, 26, 28, 30, 31, 32, 33, 34, 35, 36, 37], "osx": [0, 23, 25, 30], "other": [0, 1, 2, 3, 5, 6, 7, 8, 10, 13, 14, 16, 19, 21, 22, 23, 26, 27, 28, 29, 31, 32, 33, 34, 35], "otherwis": [0, 1, 4, 7, 13, 24, 30, 33, 35, 36], "ouput": [5, 7, 12, 34, 35], "our": [1, 2, 3, 6, 7, 8, 9, 10, 12, 14, 15, 16, 17, 18, 19, 21, 23, 24, 27, 33, 34, 37], "ourmodel": 0, "ourselv": [0, 5, 6, 8, 11, 13, 30, 31, 32, 34], "out": [0, 1, 2, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 15, 16, 21, 22, 23, 24, 25, 27, 30, 31, 33, 34, 35, 36, 37], "out_fil": 9, "outcom": [0, 7, 9, 10, 12, 27, 31, 35, 36], "outdoor": 9, "outer": [6, 12, 13], "outfil": 4, "outlier": [0, 8, 30, 31, 33], "outlin": [6, 10, 11, 34, 35], "outlook": 9, "outperform": [10, 33], "output": [0, 1, 3, 4, 5, 6, 7, 8, 9, 10, 12, 13, 19, 21, 22, 24, 25, 27, 30, 31, 32, 33, 34, 35, 36], "output_bia": 1, "output_bias_gradi": [1, 37], "output_shap": 4, "output_weight": 1, "output_weights_gradi": [1, 37], "outputlayer1": [12, 36], "outputlayer2": [12, 36], "outsid": [4, 22], "over": [0, 1, 3, 4, 5, 6, 9, 10, 12, 13, 15, 16, 19, 22, 24, 25, 30, 31, 32, 33, 34, 35], "over1": 13, "overal": [1, 10, 33], "overcast": 9, "overcom": [12, 13, 36, 37], "overdetermin": [0, 30], "overfit": [0, 1, 3, 6, 9, 10, 13, 33, 34, 35], "overflow": [5, 33, 34], "overhead": [12, 37], "overlap": [3, 7, 8, 9, 36], "overleaf": [20, 25], "overlin": [0, 5, 6, 9, 10, 11, 14, 24, 30, 31, 33], "overshoot": 33, "overst": 0, "overtrain": 4, "overview": [3, 20], "own": [4, 5, 6, 8, 12, 13, 16, 18, 22, 23, 24, 32, 33, 34, 37], "owner": [], "ownmsepredict": 0, "ownmsetrain": 0, "ownridgebeta": 31, "ownridgetheta": [0, 6, 31, 32, 33], "ownypredictridg": 0, "ownytilderidg": 0, "ox": [], "oxid": [], "p": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 13, 14, 15, 16, 17, 18, 19, 24, 27, 30, 31, 32, 33, 34, 35, 36], "p0": 2, "p1": 2, "p_": [2, 4, 8, 9], "p_hidden": 2, "p_i": [5, 27], "p_j": 27, "p_n": 27, "p_output": 2, "p_x": 27, "pa": 37, "pack": [0, 30], "packag": [0, 1, 3, 4, 5, 8, 11, 13, 15, 20, 22, 23, 25, 27, 31, 32, 33], "packtpub": 30, "packtpublish": 30, "pad": [3, 4], "page": [0, 23, 25, 30, 32, 33, 34, 35], "pai": [0, 1, 9, 13, 15, 33], "pair": [0, 2, 3, 9, 23, 27, 30], "paltform": 15, "panda": [0, 4, 5, 6, 7, 9, 11, 23, 25, 32, 33, 34, 35, 36], "pandoc": [], "panel": 30, "paper": [1, 33], "paper_fil": 33, "paradigm": [0, 30], "paragraph": 20, "parallel": [10, 13, 23, 24, 30], "param": 2, "paramat": 2, "paramet": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 12, 13, 16, 17, 18, 19, 21, 22, 25, 27, 32, 33, 34], "parameter": [0, 6, 10, 30, 31], "parametr": [0, 6, 30, 31, 34, 35], "paramt": [3, 5, 34, 37], "parent": 37, "parser": [], "part": [0, 1, 3, 5, 6, 10, 17, 19, 20, 21, 22, 24, 26, 27, 28, 30, 31, 34], "partial": [0, 1, 5, 6, 7, 8, 10, 11, 12, 13, 16, 21, 27, 30, 31, 32, 33, 35, 36, 37], "particip": [15, 23, 26, 28, 30], "particl": [0, 4, 13, 27, 30], "particular": [0, 1, 2, 3, 5, 6, 9, 10, 11, 12, 13, 16, 25, 27, 29, 30, 31, 32, 33, 34, 35, 36, 37], "particularli": [5, 6, 8, 11, 13, 27, 31, 32, 33, 34, 35], "partit": [1, 4, 9], "partli": [6, 30], "partner": [15, 25], "pass": [2, 3, 12, 14, 21, 33, 37], "password": 25, "past": [10, 27, 33], "patch": [6, 27, 34], "path": [0, 4, 6, 7, 9, 23, 30, 33, 34, 35], "pathcollect": 17, "patholog": [], "patient": [7, 35, 36], "patter": 4, "pattern": [0, 3, 4, 12, 29, 30, 33, 36, 37], "paul": [], "pauli": [0, 30], "pav": [], "pc": [11, 15, 23], "pca": [0, 7, 23, 30, 31, 36], "pd": [0, 4, 5, 6, 7, 9, 11, 30, 31, 32, 33, 34, 35, 36], "pde": 2, "pdf": [0, 3, 4, 5, 6, 9, 15, 16, 19, 20, 25, 29, 30, 34], "pedagog": [0, 30, 31], "penal": [6, 18, 31, 33], "penalti": [6, 13, 18, 25, 31, 33], "penros": [5, 6], "pentagon": [13, 32], "peopl": [1, 9, 13, 23, 25, 33], "per": [0, 1, 6, 21, 26, 28, 30, 33, 34, 35, 36], "percentag": [10, 11, 28], "perceptron": [0, 1, 7, 30, 35], "peregrin": 30, "perez": [], "perfect": [0, 1, 13, 30, 33], "perfectli": [4, 6, 34, 35], "perform": [0, 2, 3, 4, 5, 6, 8, 10, 11, 12, 13, 14, 16, 18, 19, 21, 22, 23, 24, 25, 27, 30, 31, 32, 33, 34, 35, 36, 37], "performac": 4, "perhap": [0, 5, 13, 30, 31, 32, 33], "perimet": 1, "period": [1, 4, 27], "permiss": 15, "permit": [], "permut": 11, "persist": 13, "person": [5, 6, 7, 16, 20, 26, 28, 30, 31, 35], "perspect": 29, "pertin": [12, 30, 37], "petal": [8, 9], "peter": [29, 31], "petersen": 37, "phantom": 27, "phase": [6, 12, 36, 37], "phenomena": 27, "phenomenon": 33, "phi": 8, "phi_k": 8, "philipp": 37, "philosophi": 13, "phone": [28, 30], "photo": [4, 30], "php": 25, "phrase": [0, 30], "physic": [0, 1, 4, 7, 12, 13, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37], "pi": [2, 3, 5, 6, 7, 9, 12, 13, 27, 34, 35, 36], "pick": [1, 9, 10, 11, 13, 14, 25, 33], "pickl": 1, "pictur": [0, 30], "pie": [23, 30], "piec": [11, 14, 21], "pierr": [], "pillow": [0, 23, 25, 30], "pinv": [5, 6, 13, 25, 31, 32, 33, 36], "pip": [0, 1, 15, 23, 25, 30], "pip3": [0, 1, 25, 30], "pipelin": [0, 6, 8, 10, 31, 34, 35], "pippin": 30, "pit": 4, "pitfal": [6, 31], "pitt": [12, 36, 37], "pixel": [1, 3, 4, 30], "pixel_height": [1, 3], "pixel_width": [1, 3], "pkg_resourc": [], "pkgutil": [], "place": [0, 4, 6, 8, 13, 15, 24, 25, 30, 32, 34], "plai": [0, 3, 4, 5, 6, 8, 11, 18, 22, 23, 25, 30, 31, 32, 34, 35, 37], "plain": [8, 10, 12, 13, 14, 25, 32, 33, 37], "plan": [6, 9, 28, 29, 30], "plane": [8, 9], "plateau": [5, 32, 33], "platform": [23, 30], "plausibl": [12, 36], "pleas": [13, 25, 28, 30], "plenti": 1, "plethora": [3, 12, 36, 37], "pliahhy2ibx9hdharr6b7xevztgzra1p": [36, 37], "plot": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 16, 17, 19, 20, 21, 23, 24, 25, 27, 30, 31, 32, 33, 36], "plot_all_sc": [25, 31], "plot_confusion_matrix": [7, 10, 36], "plot_count": 6, "plot_cumulative_gain": [7, 10, 36], "plot_data": 1, "plot_dataset": 8, "plot_decision_boundari": [9, 10], "plot_import": 10, "plot_iris_dataset": 21, "plot_max": 4, "plot_min": 4, "plot_model": 4, "plot_numb": 4, "plot_predict": 8, "plot_regression_predict": 9, "plot_result": 4, "plot_roc": [7, 10, 36], "plot_surfac": [2, 6, 13], "plot_train": 9, "plot_tre": [9, 10], "plqvvvaa0qudcjd5baw2dxe6of2tius3v3": [36, 37], "plt": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 17, 19, 21, 22, 24, 27, 30, 31, 32, 33, 34, 35, 36], "plu": [0, 3, 5, 7, 18, 30, 31, 35], "plugin": [], "pm": [8, 34], "pmatrix": 2, "pml": 29, "pn": 3, "png": [0, 4, 6, 7, 9, 30, 34, 35], "point": [0, 1, 2, 3, 5, 6, 7, 8, 9, 10, 11, 13, 14, 18, 19, 20, 24, 25, 27, 28, 30, 31, 32, 33, 34, 35, 36], "point_1": 4, "point_2": 4, "poisson": [23, 27, 30], "poli": [6, 8, 34, 35], "poly100_kernel_svm_clf": 8, "poly3": 0, "poly3_plot": 0, "poly_featur": [8, 9, 15], "poly_features10": 9, "poly_fit": 9, "poly_fit10": 9, "poly_kernel_svm_clf": 8, "poly_model": 15, "poly_ms": 15, "poly_predict": 15, "polydegre": [0, 5, 6, 10, 31, 34, 35], "polygon": [13, 32], "polym": [12, 36, 37], "polymi": 25, "polynomi": [0, 5, 6, 7, 8, 9, 10, 11, 15, 17, 19, 20, 25, 30, 31, 33, 34, 35, 36, 37], "polynomial_featur": [6, 15, 16, 17, 34, 35], "polynomial_svm_clf": 8, "polynomialfeatur": [0, 6, 8, 9, 15, 16, 19, 31, 34, 35], "polytrop": [0, 6, 34, 35], "pool": 3, "pool_siz": 3, "poor": [1, 13, 32, 33], "poorli": [0, 31], "popul": [0, 5, 30, 31], "popular": [0, 1, 3, 6, 7, 8, 9, 11, 12, 15, 23, 24, 25, 27, 31, 35, 36], "popularli": [0, 30], "portabl": 10, "portion": [11, 13, 33], "pose": [0, 4, 5, 6, 11, 27, 30, 34], "posit": [0, 1, 2, 3, 5, 7, 8, 10, 11, 13, 14, 21, 24, 27, 30, 31, 32, 33, 35, 36], "possibl": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 21, 23, 24, 25, 27, 28, 30, 31, 32, 33, 34, 35, 36, 37], "possibli": [6, 8, 13, 25], "post": [], "posterior": 5, "postpon": [0, 31], "postscript": 25, "postul": 5, "potenti": [0, 3, 5, 6, 12, 13, 31, 33, 34, 36, 37], "pott": [12, 36, 37], "power": [0, 1, 5, 6, 8, 9, 12, 13, 30, 31, 32, 33, 34, 35, 36, 37], "pp": [5, 6, 19, 34, 37], "practic": [0, 5, 6, 7, 8, 16, 18, 19, 21, 25, 27, 31, 34, 35, 36], "practition": [0, 1, 3, 30, 33], "pre": 30, "preambl": [], "precalcul": 37, "preced": [1, 11, 12, 27, 36], "preceed": 4, "preceq": 8, "precis": [0, 2, 5, 11, 13, 24, 25, 27, 30, 31, 33, 34, 37], "pred": [6, 34, 35, 36], "predicit": 0, "predict": [0, 1, 5, 6, 7, 8, 9, 10, 15, 16, 17, 19, 22, 23, 25, 29, 30, 31, 32, 33, 34, 35, 36], "predict_prob": [1, 35, 36], "predict_proba": [7, 10, 36], "predictedlabel": [35, 36], "predictor": [0, 5, 6, 7, 9, 10, 11, 30, 31, 33], "prefer": [0, 1, 6, 8, 9, 11, 13, 15, 20, 23, 25, 30], "prefil": [], "prepar": [0, 6, 24, 25, 30, 31], "preprocess": [0, 4, 6, 7, 8, 9, 10, 11, 15, 16, 17, 18, 19, 25, 34, 35, 36], "prerequisit": 0, "prescript": 25, "presenc": 13, "present": [0, 5, 6, 7, 9, 12, 13, 24, 25, 27, 30, 31, 32, 33, 36, 37], "preserv": [3, 11, 24], "press": [13, 15, 29, 32, 37], "pretrain": [1, 4], "pretti": [0, 4, 8, 9, 21, 23, 25, 30], "prettier": [], "prev_centroid": 14, "prevent": [13, 27, 33], "previou": [0, 1, 2, 3, 4, 5, 6, 8, 10, 11, 12, 13, 15, 16, 21, 22, 24, 25, 27, 31, 32, 33, 36, 37], "previous": [2, 3, 9, 10, 27], "price": [0, 4, 9, 13, 33], "primal": 8, "primari": [0, 7, 30, 35, 36], "prime": 27, "princip": [0, 5, 7, 23, 30, 31, 32, 36], "principl": [0, 6, 7, 8, 14, 30, 34, 35, 36], "print": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 13, 14, 18, 21, 22, 24, 27, 30, 31, 32, 33, 34, 35, 36, 37], "print_funct": [8, 9], "printout": [0, 30], "prior": [0, 5, 6, 30], "privat": 0, "prob": [1, 27, 35, 36], "probabilist": [0, 29, 30, 31], "probabl": [0, 1, 3, 4, 6, 7, 10, 13, 21, 23, 30, 31, 33, 35, 36], "problem": [0, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 17, 23, 24, 25, 27, 34], "probml": 29, "proce": [0, 5, 6, 7, 8, 9, 10, 11, 13, 24, 30, 31, 34, 37], "procedur": [2, 4, 5, 6, 8, 10, 11, 13, 31, 32, 33, 34, 35], "proceed": 24, "process": [0, 2, 4, 6, 9, 10, 12, 13, 23, 24, 25, 27, 29, 30, 32, 33, 34, 35, 36, 37], "procur": [], "prod": 29, "prod_": [1, 5, 7, 34, 35, 36], "produc": [0, 3, 4, 5, 6, 9, 10, 11, 12, 13, 18, 20, 23, 24, 25, 27, 30, 31, 34, 36, 37], "product": [0, 1, 3, 5, 6, 7, 8, 12, 13, 16, 17, 23, 24, 30, 31, 33, 34, 35, 36, 37], "profess": [0, 30], "profit": [], "program": [0, 1, 4, 5, 6, 8, 12, 14, 15, 23, 24, 26, 27, 28, 30, 31, 36], "programm": 24, "progress": [1, 4, 14, 33, 35, 36], "prohibit": [6, 34, 35], "project": [0, 1, 2, 3, 5, 11, 13, 15, 19, 22, 23, 26, 31, 32, 33, 34, 35, 36], "project_root_dir": [0, 6, 7, 9, 30, 34, 35], "promin": [12, 36, 37], "promis": 8, "promot": [28, 30], "prompt": 20, "prone": [9, 15, 21, 37], "pronounc": [13, 23, 30, 33], "proof": [0, 11, 12, 13, 30, 32, 34, 35, 37], "prop": 33, "prop_cycl": [], "propag": [2, 3, 13, 21, 22, 33], "proper": [0, 2, 6, 7, 20, 34, 35], "properli": [1, 6, 8, 10, 13, 18, 20, 25, 33], "properti": [0, 1, 3, 12, 13, 16, 24, 30, 34, 36], "propgag": 37, "proport": [0, 1, 5, 9, 11, 13, 27, 30, 31], "propos": [1, 4, 6, 10, 25, 30, 33], "propto": [5, 13, 32, 33], "proton": [0, 30], "prove": [3, 13, 32, 33], "provid": [0, 1, 3, 4, 5, 6, 8, 9, 10, 12, 13, 20, 21, 22, 23, 24, 25, 27, 30, 31, 32, 33, 34, 35, 37], "proxi": [1, 13, 33], "prune": 9, "pseudo": [24, 27, 33], "pseudocod": 25, "pseudoinv": 5, "pseudoinvers": [5, 6, 25], "pseudorandom": [6, 27, 34], "psychologi": [0, 30], "pt": 13, "public": [0, 15, 23, 30], "publish": 37, "pull": 15, "punish": [0, 1, 30], "pure": [3, 9, 27], "purest": 9, "puriti": 9, "purpos": [0, 3, 10, 12, 14, 21, 30, 36, 37], "push": 15, "put": [1, 20, 25, 33], "putmask": [], "py": 5, "pybtex": [], "pycod": 30, "pydata": 23, "pydevd_extension_api": [], "pydevd_plugin": [], "pydevd_plugin_plugin_nam": [], "pydot": 9, "pygment": [], "pyhton2": 30, "pylab": [7, 30, 35], "pypi": 23, "pyplot": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 17, 19, 21, 22, 24, 27, 30, 31, 32, 33, 34, 35, 36], "pythagora": 5, "python": [1, 2, 3, 5, 6, 8, 11, 12, 13, 14, 18, 20, 21, 22, 25, 27, 31, 33, 37], "python2": [0, 25], "python3": [0, 23, 25, 30], "pythonpath": [], "pytorch": [0, 23, 25, 30, 37], "pyzmq": [], "q": [5, 6, 8, 11, 27, 34], "qp": 8, "qquad": [2, 11, 13, 24, 33], "qr": [5, 6, 24, 31, 32], "quad": [1, 13, 24], "quadrat": [0, 8, 9, 13, 30], "qualit": [4, 9, 25, 27], "qualiti": [0, 9, 23, 30, 31, 37], "quantifi": 1, "quantil": 10, "quantit": [0, 6, 9, 25, 30, 34, 35], "quantiti": [0, 2, 5, 6, 7, 9, 10, 11, 12, 14, 16, 24, 27, 30, 31, 32, 33, 34, 35, 36, 37], "quantum": [4, 12, 29, 30, 36, 37], "quartil": [0, 31, 33], "quasi": 37, "quench": 5, "queri": 9, "question": [0, 5, 6, 9, 11, 12, 13, 25, 28, 30, 31, 33, 34, 37], "qugan": 4, "quick": [4, 27], "quicker": 33, "quickli": [1, 3, 9, 11, 13, 32, 33], "quit": [1, 5, 6, 9, 10, 12, 15, 22, 31, 32, 34, 35, 36], "quot": 4, "r": [0, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 17, 23, 24, 25, 27, 31, 32, 33, 34, 35, 36, 37], "r2": [0, 5, 6, 19, 30, 31, 32], "r2_score": [0, 30], "r2score": [0, 30], "r_": 33, "r_0": 33, "r_1": 9, "r_2": 9, "r_j": 9, "r_m": 9, "r_t": 33, "rad": [], "rade": [], "radial": [8, 12, 36, 37], "radioact": 27, "radiu": [0, 1, 31, 33], "radziej": [], "ragan": [], "rain": 9, "rais": [], "ram": 33, "ramanujam": [], "ramp": 1, "ran0": 27, "ran1": 27, "ran2": 27, "ran3": 27, "rand": [0, 4, 5, 6, 9, 10, 13, 15, 19, 21, 22, 24, 30, 31, 32, 33, 34, 35], "randint": [6, 9, 13, 33, 34], "randn": [0, 1, 2, 5, 6, 9, 11, 13, 15, 18, 21, 22, 30, 31, 32, 33, 34, 35, 36, 37], "random": [0, 1, 2, 3, 4, 5, 6, 8, 9, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 30, 31, 32, 33, 34, 35, 36, 37], "random_forest_model": 10, "random_index": [13, 33], "random_indic": [1, 3], "random_st": [7, 8, 9, 10, 11, 35, 36], "randomforestclassifi": 10, "randomli": [1, 6, 9, 13, 14, 18, 32, 33, 34, 35], "randomst": [35, 36], "rang": [0, 1, 2, 3, 4, 5, 6, 7, 9, 10, 11, 12, 13, 14, 18, 19, 21, 22, 24, 27, 30, 31, 32, 33, 34, 35, 36, 37], "rangl": [0, 6, 11, 27, 30, 31], "rangle_x": 27, "rank": [5, 31, 32], "rankdir": 4, "raphson": [1, 8, 13], "rapidli": [0, 33], "rare": [1, 13, 33], "raschka": [30, 31, 34, 35, 36], "rasckha": 30, "rashcka": [32, 33, 37], "rashkca": 37, "rate": [1, 2, 3, 4, 8, 9, 10, 12, 13, 18, 32, 34, 35, 36, 37], "rather": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 24, 27, 30, 31, 32, 34, 35, 37], "ratio": [4, 7, 9, 10, 11, 35, 36], "rational": [0, 30], "ravel": [5, 6, 7, 8, 9, 10, 11, 13, 24, 34, 35, 36], "raw": [3, 33], "rbf": [8, 11, 12, 36, 37], "rbf_kernel_svm_clf": 8, "rbf_pca": 11, "rc": 27, "rcond": [0, 30, 31], "rcparam": [1, 3, 7, 8, 9, 10, 27, 30, 35], "re": [2, 4, 13, 15, 32], "reach": [1, 4, 5, 6, 9, 10, 12, 13, 14, 32, 33, 34, 35, 37], "react": [], "read": [0, 2, 3, 4, 5, 6, 7, 8, 11, 12, 16, 17, 19, 20, 24, 25, 27, 29, 32], "read_csv": [0, 6, 7, 9, 34, 35], "read_fwf": [0, 30], "reader": [0, 6, 20, 24, 27, 30, 31, 33], "readi": [0, 1, 5, 6, 8, 10, 11, 12, 24, 30, 37], "readili": 1, "readm": [15, 20, 25], "readthedoc": 23, "real": [0, 1, 4, 7, 10, 11, 12, 16, 18, 19, 24, 31, 34, 35, 36], "real_loss": 4, "real_output": 4, "realist": [8, 30], "realiti": 27, "realiz": [1, 12, 36], "realli": [0, 1, 30], "rearrang": 13, "reason": [0, 1, 3, 4, 10, 13, 29, 30, 32, 33], "reassign": 1, "recal": [5, 6, 9, 10, 11, 12, 22, 24, 27, 30, 31, 32, 33, 34, 35, 37], "recarrai": [], "recast": 3, "receiv": [1, 3, 10, 12, 27, 36, 37], "recent": [0, 6, 13, 29, 33, 34, 35, 37], "recept": [3, 12, 36, 37], "receptive_field": 3, "recip": [0, 6, 7, 24, 25, 30, 31, 35, 36], "reciproc": 5, "recogn": [0, 4, 5, 10, 30, 34], "recognit": [0, 1, 3, 12, 29, 30, 36, 37], "recommen": 30, "recommend": [0, 2, 3, 4, 5, 6, 8, 13, 15, 19, 20, 21, 22, 23, 24, 25, 29, 32, 33, 34, 35, 36, 37], "reconsid": 9, "reconstruct": 11, "record": [10, 25, 26, 28, 30, 35, 36], "recreat": [15, 21], "rectangl": [9, 13, 32], "rectangular": [5, 31, 32], "rectifi": [1, 3, 12, 36], "recur": [0, 23, 30], "recurr": [0, 1, 23, 30], "recurs": [9, 23, 24, 30], "red": [0, 3, 4, 6, 8, 9, 33, 34], "redefin": [0, 10, 30, 31, 32], "redefinit": 32, "redistribut": [], "reduc": [1, 3, 5, 6, 9, 10, 11, 13, 21, 30, 32, 33, 34], "reduct": [0, 10, 11, 23, 27, 30, 31], "reegress": 25, "ref": 20, "refer": [0, 1, 2, 3, 5, 6, 11, 12, 13, 14, 20, 24, 29, 30, 31, 32, 33, 34, 35, 36, 37], "referansestil": 20, "referenc": [2, 37], "refin": [12, 36, 37], "refit": [6, 34, 35], "reflect": [0, 1, 4, 5, 25, 27, 30], "refresh": [23, 30], "refreshprogrammingskil": 30, "reg": [10, 11], "regard": [1, 9, 13], "regardless": [12, 16, 36], "regexp": [], "reggi": [], "regim": 33, "region": [3, 4, 6, 9, 12, 25, 33, 36, 37], "regist": [6, 27], "reglasso": [5, 32], "regr_1": [0, 9], "regr_2": [0, 9], "regr_3": [0, 9], "regress": [1, 8, 11, 12, 16, 20, 23, 24, 37], "regressor": [0, 7, 10, 35], "regret": [], "regridg": [0, 5, 6, 31, 32, 33], "regular": [0, 3, 4, 5, 6, 7, 9, 13, 17, 18, 28, 30, 31, 32, 33, 34, 35, 36], "regularli": 15, "reilli": [0, 29, 30], "reinforc": [0, 8, 23, 30], "reiter": 1, "reitz": [], "reject": 7, "rel": [0, 4, 6, 7, 9, 12, 13, 21, 27, 30, 31, 33, 34, 35, 36], "relat": [0, 1, 3, 4, 5, 11, 13, 14, 19, 24, 27, 30, 31, 32, 34, 37], "relationship": [0, 4, 9, 18, 30], "relativeerror": [0, 30, 31], "releas": [1, 23, 30], "relev": [0, 1, 5, 7, 11, 23, 25, 27, 30, 32, 33], "reli": [0, 6, 8, 33], "reliabilti": 25, "reliabl": [7, 27, 35, 36], "relu": [3, 4, 21, 22, 30], "relu_d": 22, "remain": [1, 2, 4, 6, 12, 24, 27, 31, 33, 34, 35, 36, 37], "remaind": 27, "reman": 2, "remark": 1, "rememb": [0, 8, 13, 20, 21, 22, 24, 25, 30, 33], "remind": [0, 5, 11, 13, 19, 24, 27, 34], "remot": 15, "remov": [4, 5, 6, 18, 31, 32, 33], "renam": 15, "render": [0, 30, 31], "reorder": [5, 7, 31, 32, 35, 36], "reorgan": [0, 30], "repeat": [0, 1, 3, 4, 5, 6, 9, 10, 11, 13, 14, 24, 25, 27, 30, 31, 32, 33, 34, 35, 37], "repeated": 30, "repeatedli": [0, 6, 10, 13, 34, 35], "repet": 3, "repetit": [6, 30, 31, 34, 35], "rephras": [13, 32], "replac": [0, 1, 3, 4, 5, 6, 10, 12, 14, 23, 25, 30, 31, 32, 34, 35, 37], "replica": [6, 34], "repo": [15, 25], "report": [30, 33, 35, 36], "repositori": [4, 20, 25, 30], "reposotori": [], "repres": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 12, 13, 25, 27, 30, 31, 32, 33, 34, 35, 36, 37], "represent": [0, 1, 3, 6, 27, 30, 34, 35], "representd": 3, "reproduc": [0, 5, 6, 9, 12, 15, 16, 18, 20, 23, 25, 27, 30, 31, 37], "repuls": [0, 30], "request": [0, 13, 33], "requir": [0, 1, 3, 4, 5, 6, 8, 9, 11, 12, 13, 15, 17, 18, 19, 20, 24, 25, 30, 31, 32, 33, 34, 35, 36, 37], "res1": 2, "res2": 2, "res3": 2, "res_analyt": 2, "res_analytical1": 2, "res_analytical2": 2, "res_analytical3": 2, "resaml": 6, "resampl": [0, 7, 10, 23, 30, 31], "rescal": [0, 11, 12, 33, 36], "rescu": 5, "reseach": 6, "research": [0, 4, 13, 21, 22, 23, 29, 30, 33], "resembl": [6, 27, 34], "reserv": [1, 5, 6, 27, 34, 35], "reshap": [0, 1, 2, 3, 4, 6, 8, 9, 10, 24, 30, 31, 34, 35], "resid": 33, "residenti": [], "residu": [0, 5, 13, 30], "resiz": [5, 31, 32], "resnet": 33, "resort": 33, "resourc": [30, 33], "respect": [0, 1, 2, 3, 5, 6, 7, 8, 10, 11, 12, 13, 14, 16, 17, 18, 21, 25, 27, 30, 31, 32, 33, 34, 35, 36, 37], "respond": [12, 36, 37], "respons": [0, 7, 9, 12, 30, 31, 35, 36, 37], "rest": [0, 5, 18, 21, 22, 31, 32, 33], "restat": [0, 12, 30], "restor": 4, "restored_discrimin": 4, "restored_gener": 4, "restrict": [0, 3, 9, 12, 30, 36, 37], "result": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 27, 30, 33, 34, 35, 36], "retail": [], "retain": [5, 6, 31, 32, 33, 34, 35], "rethink": 34, "return": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 11, 13, 14, 16, 17, 21, 22, 24, 27, 30, 31, 32, 33, 34, 35, 36, 37], "return_data": 14, "return_sequ": 4, "return_x_i": 9, "reus": [1, 3, 6, 19, 20, 22, 25, 37], "reveal": [0, 12, 30, 36, 37], "revers": [1, 22, 24], "review": [23, 24], "revis": [], "revisit": 14, "revolut": 30, "reward": [0, 4, 30], "rewrit": [0, 3, 5, 6, 7, 8, 10, 11, 12, 13, 16, 19, 24, 25, 27, 32, 33, 35, 36, 37], "rewritten": [2, 6, 8, 10, 27, 34], "rewrot": [13, 35, 36], "rf": 10, "rgb": 3, "rgoj5yh7evk": 23, "rh": [6, 34], "rho": [0, 10, 13, 33], "rho_1": 10, "rho_2": 10, "rho_m": 10, "rich": [0, 30], "rid": [], "ride": 9, "rideclass": 9, "ridedata": 9, "ridg": [7, 11, 13, 20, 23, 30, 34, 35, 36], "ridge_paramet": 17, "ridge_sk": 6, "ridgebeta": 32, "ridgetheta": 5, "right": [0, 1, 2, 3, 5, 6, 7, 8, 9, 10, 12, 13, 14, 16, 17, 19, 21, 22, 24, 25, 27, 30, 31, 32, 33, 34, 35, 36, 37], "right_sid": 2, "rightarrow": [0, 1, 5, 6, 8, 11, 12, 13, 27, 30, 31, 32, 33, 34, 36, 37], "rigor": [0, 30, 31, 32], "ring": 6, "rise": [0, 30], "risk": [0, 13, 30, 32, 33], "rival": 4, "river": [], "rlm": 30, "rm": [27, 33], "rmse": [], "rmsporp": [13, 33], "rmsprop": [1, 3, 4, 13, 25, 34, 37], "rnd_clf": 10, "rng": [27, 35, 36], "rnn": [4, 12, 36, 37], "rnn1": 4, "rnn2": 4, "rnn_2layer": 4, "rnn_input": 4, "rnn_output": 4, "rnn_train": 4, "rntrick1": 27, "rntrick2": 27, "rntrick3": 27, "rntrick4": 27, "ro": [0, 13, 30, 32, 33], "robert": [19, 25, 29], "robust": [0, 30, 33], "robustscal": [0, 31, 33], "roc": [7, 10], "role": [0, 2, 5, 6, 8, 18, 23, 25, 30, 31, 32, 33, 34, 35, 37], "roll": 6, "ronach": [], "room": [0, 28, 30], "root": [0, 5, 9, 13, 15, 27, 31, 32, 33, 37], "root_directori": [], "rot": 30, "rotat": [1, 8, 9, 10], "rotation_matrix": 9, "roughli": [1, 3, 18], "round": [7, 9, 13, 36], "routin": [13, 24, 30, 32], "row": [0, 1, 2, 5, 6, 9, 11, 16, 21, 24, 30, 31, 32, 34], "rr": [5, 31, 32], "rrr": [5, 31, 32], "rubric": [], "rudg": [], "rug": [13, 32, 33], "rule": [0, 1, 5, 6, 13, 22, 25, 30, 31, 32, 36], "run": [0, 1, 2, 4, 5, 6, 8, 9, 11, 13, 15, 20, 21, 22, 23, 25, 30, 31, 32, 33, 34, 35], "runtim": [1, 6, 14, 15], "rust": [0, 23, 24, 30], "rvert": 1, "rvert_2": 1, "s_": [3, 6], "s_1": 6, "s_i": [6, 7, 35], "s_j": 6, "s_k": 6, "s_phenomenon": 25, "saddl": [13, 32, 33], "safeguard": [18, 33], "sai": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 19, 24, 25, 27, 30, 31, 32, 33, 34, 35, 36, 37], "said": [6, 9, 13, 32], "sake": [0, 5, 7, 11, 30, 31, 32, 35, 36, 37], "sale": [0, 30], "sam": 30, "same": [0, 1, 2, 3, 4, 5, 6, 8, 9, 11, 12, 14, 15, 16, 18, 20, 21, 22, 24, 25, 27, 30, 31, 32, 36, 37], "samm": 10, "sampl": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 13, 14, 18, 19, 23, 24, 25, 27, 30, 31, 33, 34, 35, 36], "sample_vari": 14, "sampleexptvari": 27, "samples_per_class": [35, 36], "samwis": 30, "sandbox": [], "sandboxmod": [21, 22], "sasha": [], "sastri": 11, "satisfactori": [0, 30], "satisfi": [1, 2, 3, 6, 8, 13, 24, 27, 32, 34], "satur": [1, 6, 34, 35], "save": [0, 4, 6, 7, 9, 13, 20, 22, 30, 33, 34, 35], "save_fig": [0, 6, 7, 9, 10, 30, 34, 35], "savefig": [0, 4, 6, 7, 9, 27, 30, 34, 35], "savetxt": 4, "saw": [5, 31], "scalabl": 10, "scalar": [2, 5, 6, 10, 31, 34, 37], "scale": [0, 1, 3, 5, 6, 7, 8, 9, 10, 11, 12, 13, 22, 23, 24, 25, 28, 30, 32, 35, 36], "scale_mean": 4, "scale_std": 4, "scaler": [0, 7, 8, 9, 10, 11, 17, 25, 31], "scan": [5, 7, 35, 36], "scari": 5, "scatter": [0, 1, 6, 7, 8, 9, 14, 15, 17, 21, 30, 31, 33, 34, 35], "scenario": [6, 13, 32, 33], "schedul": [13, 33], "scheme": [1, 13, 32, 33, 35, 36], "schrage": 27, "sch\u00f8yen": [6, 31, 33], "scienc": [0, 1, 10, 12, 13, 23, 26, 27, 28, 29, 32, 34, 35, 36, 37], "scientif": [0, 20, 23, 25, 30, 35, 36], "scientist": [0, 30], "scikit": [3, 5, 6, 8, 9, 10, 13, 15, 16, 20, 21, 23, 24, 25, 29], "scikit_learn": [0, 36], "scikitlearn": 30, "scikitplot": [7, 10, 36], "scipi": [0, 3, 5, 6, 13, 23, 24, 25, 30, 31, 32, 34], "scl": 6, "scm": 15, "score": [0, 1, 3, 6, 7, 9, 10, 11, 15, 16, 19, 21, 25, 28, 30, 31, 33, 34, 35, 36], "scores_kfold": [6, 34, 35], "scratch": [1, 13, 16, 36, 37], "script": [], "sdg": [13, 33], "sdv4f4s2sb8": [32, 33], "seaborn": [0, 1, 3, 6, 7, 30, 36], "seamless": [0, 23, 25, 30], "search": [0, 1, 3, 5, 9, 13, 15, 30, 32, 33], "sebastian": [30, 37], "sebastianraschka": 30, "sec": 6, "second": [0, 2, 3, 4, 5, 6, 7, 8, 9, 11, 12, 14, 15, 16, 20, 21, 22, 23, 24, 27, 28, 30, 31, 32, 34, 35, 36, 37], "second_mo": 33, "second_term": 33, "secondari": 33, "secondeigvector": 11, "secondli": [12, 37], "section": [4, 11, 16, 20, 24, 25, 27, 31, 33, 35], "sector": 0, "see": [0, 1, 2, 3, 4, 5, 6, 7, 8, 10, 11, 12, 13, 15, 16, 18, 19, 20, 21, 22, 23, 24, 25, 27, 30, 31, 32, 33, 34, 35, 36, 37], "seed": [0, 1, 2, 3, 4, 5, 6, 8, 9, 11, 13, 14, 18, 20, 21, 25, 27, 30, 31, 32, 33, 34, 35, 37], "seed_imag": 4, "seek": [1, 2, 8], "seem": [1, 3, 4, 33], "seemingli": [0, 30], "seen": [0, 1, 3, 5, 10, 12, 27], "segment": [13, 32], "seismic": 6, "seldomli": [0, 30], "select": [1, 5, 6, 8, 9, 10, 11, 15, 20, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34], "selevet": 15, "self": [1, 5, 22, 31, 35, 36], "sell": 4, "semest": [7, 26, 36], "semi": [8, 13, 32, 33], "semilogx": 6, "send": [5, 12, 13, 21, 22, 28, 30, 36, 37], "senior": [26, 28], "sens": [0, 4, 6, 8, 21, 30, 34], "sensibl": [3, 21], "sensit": [0, 5, 6, 9, 13, 30, 31, 33, 34, 35], "sent": [2, 21, 37], "sentdex": [36, 37], "sentenc": [4, 12, 36, 37], "separ": [0, 1, 2, 4, 6, 8, 9, 12, 14, 18, 21, 22, 23, 25, 27, 30, 33, 34, 36, 37], "septemb": [18, 25, 30], "sequenc": [3, 4, 7, 9, 10, 12, 13, 23, 24, 27, 30, 32, 35, 36, 37], "sequenti": [1, 3, 4, 10, 12, 27, 36, 37], "seri": [0, 1, 2, 3, 4, 5, 6, 10, 11, 12, 13, 24, 30, 31, 32, 34, 36, 37], "serif": [7, 27, 30, 35], "serv": [0, 1, 2, 3, 5, 7, 13, 29, 30, 31, 32, 33, 35, 36], "servic": 25, "session": [1, 15, 20, 25, 26, 28, 30], "set": [1, 4, 5, 6, 7, 8, 10, 11, 13, 14, 16, 17, 18, 21, 22, 23, 24, 25, 27, 28, 33, 34, 35, 36], "set_major_formatt": 6, "set_major_loc": 6, "set_tick": [1, 8], "set_ticklabel": 1, "set_titl": [0, 1, 2, 3, 7, 12, 14, 30, 35, 36], "set_xlabel": [0, 1, 2, 3, 7, 12, 30, 35, 36], "set_xlim": [7, 12, 35, 36], "set_xticklabel": 1, "set_ylabel": [0, 1, 2, 3, 7, 30, 36], "set_ylim": [7, 12, 35, 36], "set_ytick": [7, 36], "set_yticklabel": [1, 6], "set_zlim": 6, "seth": 4, "setminu": 6, "setosa": [8, 9], "setosa_or_versicolor": 8, "setp": [6, 34, 35], "setup": [1, 4, 6, 8, 22, 23, 30, 31, 32, 37], "sever": [0, 3, 5, 6, 7, 8, 9, 11, 12, 13, 16, 23, 24, 25, 27, 30, 31, 32, 33, 34, 35, 36, 37], "sgd": [1, 3, 32], "sgd_clf": 8, "sgdclassifi": 8, "sgdreg": 13, "sgdregressor": 13, "sgn": [5, 31, 32], "shall": [], "shallow": [13, 33], "shape": [0, 1, 3, 4, 5, 6, 7, 8, 9, 10, 11, 13, 14, 15, 16, 18, 21, 22, 24, 30, 31, 32, 33, 34, 35, 36, 37], "share": [1, 3, 15, 30], "share_mask": [], "shareabl": 15, "she": [7, 35, 36], "sheppard": [], "shibukawa": [], "shift": [1, 6, 12, 15, 18, 27, 31, 33, 36], "ship": 3, "shire": 30, "short": [4, 5, 20, 25], "shortcom": [13, 32, 33], "shorten": 4, "shorter": 27, "shorthand": [30, 34], "shortli": [24, 30], "should": [0, 2, 3, 5, 6, 8, 9, 11, 12, 15, 18, 19, 20, 21, 22, 24, 25, 27, 30, 31, 33, 34, 35, 37], "shouldn": [], "show": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 19, 20, 24, 25, 27, 30, 31, 32, 33, 34, 35, 36, 37], "show_shap": 4, "shown": [0, 4, 5, 8, 12, 13, 24, 31, 32, 33, 36, 37], "shrink": [3, 5, 6, 8, 11, 31, 32, 33], "shrinkag": [5, 6, 31, 32], "shrunk": 11, "shuffl": [0, 1, 4, 6, 13, 31, 33, 34, 35], "sickit": 37, "side": [0, 2, 5, 8, 12, 13, 24, 25, 30, 32, 35, 36], "sigh": [23, 30], "sigma": [0, 1, 5, 6, 7, 10, 11, 12, 13, 19, 24, 25, 27, 30, 31, 32, 33, 34, 35, 36, 37], "sigma0": 27, "sigma1": 27, "sigma2": 27, "sigma_": [5, 24, 30, 31, 32, 34], "sigma_0": [5, 31, 32], "sigma_1": [5, 31, 32, 37], "sigma_2": [5, 31, 32, 37], "sigma_fn": [7, 12, 35, 36], "sigma_i": [0, 5, 30, 31, 32], "sigma_j": [5, 31, 32], "sigma_m": [6, 27, 34], "sigma_n": [11, 27], "sigma_t": 13, "sigma_x": 27, "sigmoid": [1, 2, 4, 7, 8, 10, 12, 21, 22, 35, 36, 37], "sigmoid_autograd": 22, "sigmoid_d": 22, "sigmundson": [6, 31, 33], "sign": [1, 2, 7, 8, 10, 27, 28, 35], "signal": [1, 3, 10, 12, 33, 36, 37], "signifi": 4, "signific": [1, 33], "significantli": [1, 13, 18, 27, 32, 33], "sim": [4, 5, 6, 13, 19, 27, 34], "similar": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 14, 18, 23, 24, 25, 30, 32, 34, 35, 36, 37], "similarli": [0, 1, 3, 5, 8, 10, 13, 27, 30, 31, 32, 33, 37], "simpl": [1, 2, 3, 5, 6, 7, 8, 10, 11, 12, 14, 16, 17, 22, 23, 24, 27, 34, 36], "simple_plot": [], "simplepredict": 10, "simpler": [0, 1, 5, 6, 7, 13, 16, 23, 25, 30, 32, 33], "simplernn": 4, "simplest": [0, 1, 3, 4, 9, 10, 12, 14, 25, 30, 36, 37], "simpletre": 10, "simpli": [0, 1, 2, 4, 5, 6, 8, 9, 10, 11, 12, 23, 24, 25, 27, 30, 31, 32, 33, 34, 35, 36, 37], "simplic": [2, 5, 6, 7, 8, 9, 10, 11, 12, 14, 31, 32, 33, 35, 36, 37], "simplicti": [5, 31, 32], "simplif": 37, "simplifi": [0, 6, 9, 18, 22, 23, 25, 30, 31, 33, 34, 35, 37], "simplist": [3, 6, 27, 34], "simul": [6, 18, 33, 34, 35], "simultan": [6, 33, 34, 35], "sin": [0, 1, 2, 3, 4, 9, 12, 13, 24, 30, 36], "sinc": [0, 1, 2, 3, 5, 6, 7, 8, 9, 10, 11, 13, 16, 18, 21, 22, 24, 25, 27, 29, 30, 31, 32, 33, 34, 35, 36, 37], "sine": [3, 12, 36], "singl": [0, 1, 2, 3, 5, 6, 7, 8, 9, 12, 13, 18, 19, 21, 22, 24, 27, 30, 31, 32, 33, 34, 35], "singular": [0, 6, 13, 24, 30, 34], "sinusoid": 3, "site": [0, 25, 26, 31], "situat": [0, 4, 5, 7, 13, 27, 30, 31, 32, 33, 35, 36], "six": [3, 27, 37], "size": [0, 1, 2, 3, 4, 5, 6, 8, 9, 10, 11, 13, 18, 20, 21, 24, 25, 27, 30, 34, 35, 36, 37], "sizesp": 33, "skeleton": 22, "sketch": 10, "ski": 9, "skill": 0, "skip": 11, "skl": [0, 6, 30, 31, 33], "sklearn": [0, 1, 3, 5, 6, 7, 8, 9, 10, 11, 13, 14, 15, 17, 19, 20, 21, 22, 30, 31, 32, 33, 34, 35, 36], "skplt": [7, 10, 36], "sl": [6, 31, 33], "slack": 8, "slender": [], "slice": [2, 24, 30], "slide": [0, 3, 16, 25, 27, 30, 31, 32, 37], "slight": [6, 13, 34, 35], "slightli": [1, 2, 3, 5, 6, 7, 10, 27, 31, 32, 34, 35, 36, 37], "slope": [8, 11, 12, 36], "slow": [0, 2, 8, 13, 18, 31, 32, 33], "slower": [5, 24, 30, 31, 32, 33], "slowest": 24, "slowli": [12, 33], "slp": 1, "small": [0, 1, 2, 3, 5, 6, 8, 9, 10, 11, 12, 13, 18, 21, 22, 23, 24, 27, 30, 31, 32, 33, 34, 35, 36, 37], "smaller": [0, 1, 2, 5, 6, 8, 9, 11, 13, 21, 27, 30, 31, 32, 33, 34, 35], "smallest": [0, 4, 14, 30], "smallest_row_index": 14, "smodin": [], "smooth": [0, 3, 6, 13, 25, 30, 32, 33], "smoother": 33, "sn": [0, 1, 3, 6, 7, 30, 36], "sne": 11, "so": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 18, 19, 20, 21, 22, 23, 24, 25, 27, 28, 30, 31, 32, 33, 34, 35, 36, 37], "soar": 6, "social": 0, "soft": [1, 7, 10, 12, 35, 36, 37], "soften": 8, "softmax": [3, 7, 21, 22, 35, 36], "softmax_vec": 21, "softwar": [0, 8, 23, 24, 37], "sokogskriv": 20, "sol": 8, "sol1": 21, "sole": [0, 6, 30], "solid": [0, 7, 35, 36], "solut": [0, 1, 2, 3, 5, 6, 8, 10, 11, 13, 18, 21, 24, 25, 27, 30, 31, 32, 33, 34], "solution_ev": 33, "soluton": 2, "solv": [0, 1, 3, 5, 6, 8, 10, 11, 12, 13, 16, 24, 25, 30, 31, 37], "solve_expdec": 2, "solve_ode_deep_neural_network": 2, "solve_ode_neural_network": 2, "solve_pde_deep_neural_network": 2, "solveod": 2, "solveode_popul": 2, "solver": [2, 7, 8, 9, 10, 24, 30, 36], "some": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 14, 15, 16, 18, 19, 21, 22, 25, 27, 30, 33, 34, 36], "some_model": [6, 31, 33], "somehow": 4, "someon": 16, "someth": [0, 1, 3, 4, 7, 9, 11, 15, 19, 20, 25, 27, 30, 31, 36], "sometim": [0, 1, 11, 12, 13, 14, 19, 31, 33, 36, 37], "somewhat": 36, "soon": [24, 28, 31], "sophist": [0, 30], "sopt": 13, "sort": [5, 6, 9, 11, 27, 34, 35], "sound": [3, 5], "sourc": [0, 1, 3, 6, 23, 24, 25, 27, 30, 33, 34, 35], "source1": 22, "source2": 22, "space": [0, 1, 4, 5, 8, 9, 11, 12, 13, 14, 27, 31, 32, 33, 35, 36, 37], "span": [0, 3, 5, 9, 11, 24, 30, 31, 32], "spare": 1, "spars": [3, 6, 18, 24, 30, 33], "sparse_mtx": [24, 30], "sparsecategoricalcrossentropi": 3, "sparsiti": [10, 18], "spatial": [1, 2, 3, 12, 36, 37], "speak": 27, "special": [6, 7, 10, 12, 13, 24, 27, 30, 31, 32, 33, 35, 36, 37], "specif": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 11, 12, 15, 16, 23, 24, 25, 27, 29, 30, 31, 32, 34, 35, 36, 37], "specifi": [0, 3, 5, 6, 7, 9, 11, 13, 14, 27, 30, 32, 33, 34, 35, 36], "specifici": [0, 10, 30], "spectacular": 3, "spectral": 1, "speech": [0, 1, 3, 4, 12, 36, 37], "speed": [1, 2, 4, 13], "spend": [16, 27, 33], "spent": 25, "sphere": [0, 31, 33], "sphinx": [], "sphinx_book_them": [], "sphinxcontrib": [], "spike": 33, "spin": 6, "spite": 0, "spitzer": [], "spline": 8, "split": [1, 3, 4, 5, 6, 8, 9, 10, 11, 14, 16, 17, 20, 21, 22, 25, 27, 30, 32, 33, 34, 35], "splite": 0, "splitter": [1, 10], "spoiler": [], "spontan": 27, "spot": 3, "spread": [0, 11, 27, 30, 31, 35, 36], "springer": [19, 25, 29, 30, 34, 35], "spuriou": [13, 33], "sqquar": 32, "sqrsignal": 3, "sqrt": [3, 4, 5, 6, 8, 10, 11, 13, 27, 31, 32, 33, 34, 37], "squar": [1, 2, 3, 4, 7, 8, 9, 11, 13, 14, 15, 17, 18, 23, 24, 27, 34, 35, 36, 37], "squarederror": 10, "squaredeuclidean": 14, "squash": [12, 36], "src": [], "srtm": 6, "srtm_data_norway_1": 6, "sso": 20, "stabil": [5, 25, 33, 35, 36], "stabl": [0, 4, 5, 6, 9, 16, 20, 23, 25, 30, 31, 32, 33], "stack": [3, 4], "stage": [5, 13, 15, 25, 33, 37], "stagnat": 33, "stai": [0, 2, 4, 5, 11, 30, 31, 33], "stand": [0, 5, 9, 12, 30, 31, 32, 36], "standard": [0, 1, 4, 5, 6, 7, 8, 10, 12, 17, 18, 19, 24, 25, 27, 30, 32, 33, 35, 36, 37], "standardscal": [0, 6, 7, 8, 9, 10, 11, 17, 31, 33], "standpoint": 33, "stanford": [13, 32], "start": [0, 1, 2, 3, 4, 5, 6, 8, 9, 10, 11, 12, 13, 14, 15, 16, 18, 21, 22, 24, 27, 28, 30, 31, 32, 33, 34, 35, 37], "start_tim": 14, "starter": [], "stat": [6, 34], "state": [1, 2, 4, 5, 6, 7, 8, 10, 11, 12, 13, 23, 27, 30, 31, 32, 34, 35, 36, 37], "statement": [0, 7, 24, 30, 36], "static": [], "stationari": [32, 33], "statist": [0, 1, 3, 4, 7, 9, 10, 11, 12, 13, 14, 19, 24, 25, 29, 31, 32, 33, 36, 37], "statu": [0, 7, 15, 30, 35, 36], "stavang": 6, "stb": [], "std": [0, 4, 6, 18, 30, 31, 33, 34, 35], "steep": [13, 32, 33], "steepest": 33, "stefan": [], "step": [0, 1, 2, 4, 6, 7, 9, 10, 11, 12, 13, 14, 15, 18, 22, 24, 25, 30, 32, 36, 37], "step_fn": [7, 12, 35, 36], "step_length": [13, 33], "step_siz": 33, "steps_list": 9, "stereo": 3, "sticki": [], "still": [0, 2, 3, 5, 6, 11, 13, 21, 22, 27, 31, 32, 33, 34, 35, 37], "stimuli": [12, 36, 37], "stk": [29, 30], "stk2100": [29, 30], "stk3155": [15, 25, 26, 28], "stk4021": [29, 30], "stk4051": [29, 30], "stk4155": [26, 28], "stk5000": 29, "stochast": [0, 1, 5, 6, 8, 11, 12, 22, 32, 34, 35, 37], "stock": 4, "stoke": [12, 36, 37], "stone": [0, 7, 35, 36, 37], "stop": [1, 4, 9, 13, 14, 18, 32, 37], "storag": [5, 31, 32], "store": [0, 1, 2, 3, 6, 11, 13, 22, 27, 30, 33], "storehaug": [28, 30], "stori": [], "str": [1, 3, 4], "straight": [0, 6, 8, 13, 30, 32, 34], "straightforward": [0, 2, 3, 5, 6, 8, 9, 10, 13, 24, 30, 31, 32, 34], "strategi": [0, 1, 9, 30], "stratifi": [6, 34, 35], "stream": 33, "strength": [0, 5, 14, 31, 32], "stretch": 11, "strict": [8, 13, 32], "strictli": [8, 13, 32], "stride": [4, 24], "strike": 6, "string": 1, "stroke": [7, 35, 36], "strong": [3, 6, 9, 10, 12, 24, 27, 33, 34, 36, 37], "strongli": [0, 8, 15, 20, 22, 23, 24], "stronli": [], "structur": [0, 1, 2, 3, 6, 9, 10, 12, 22, 23, 30, 34, 35, 36], "stuck": [1, 13, 32, 33], "student": [0, 15, 25, 26, 28, 29, 30], "studi": [0, 3, 4, 5, 6, 7, 8, 11, 12, 13, 23, 25, 29, 30, 31, 32, 33, 35, 37], "studier": 29, "stuff": [21, 22], "style": [7, 9, 20, 24, 30], "stylesheet": [], "st\u00f8land": 28, "sub": [9, 12, 33, 36, 37], "subarrai": [], "subclass": [], "subdivid": [0, 24, 30], "subfield": 0, "subgradi": 33, "subject": [6, 8, 27], "sublicens": [], "sublinear": 33, "submit": 30, "subplot": [0, 1, 3, 4, 6, 7, 8, 9, 10, 14, 21, 30, 34, 35, 36], "subplots_adjust": [8, 27], "subprogram": [24, 30], "subproject": [], "subract": [0, 31], "subroutin": [0, 30], "subscript": 1, "subsequ": [1, 4, 5, 6, 12, 24, 27, 31, 32, 34, 36, 37], "subset": [1, 6, 9, 12, 13, 23, 30, 32, 33, 34, 35, 36, 37], "subspac": [0, 8, 11, 31], "substanti": [9, 10, 33], "substep": 11, "substitut": [3, 6, 12, 16, 24, 34, 35, 36], "subsubset": 9, "subtask": 6, "subtl": 1, "subtract": [0, 4, 5, 6, 11, 13, 18, 19, 24, 25, 27, 31, 33, 34, 35], "subtre": 9, "succeed": [0, 4, 30], "success": [3, 7, 9, 13, 27, 35, 36], "successfulli": [4, 9], "succinctli": 33, "sudo": [0, 23, 25, 30], "suffer": [0, 1, 2, 5, 10, 30, 31, 32], "suffici": [1, 6, 8, 11, 13, 32, 34, 35], "suggest": [1, 13, 25, 29, 32, 33], "suit": [8, 12, 36, 37], "suitabl": [0, 15, 19, 27, 31, 33], "sum": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 19, 21, 24, 27, 30, 31, 32, 33, 36], "sum_": [0, 1, 2, 3, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 16, 19, 24, 25, 27, 30, 31, 32, 33, 34, 35, 36, 37], "sum_i": [0, 2, 5, 6, 8, 13, 19, 25, 31, 32, 33, 34, 35], "sum_j": [6, 18, 33], "sum_ja_": 0, "sum_k": [6, 8, 12, 24, 37], "sum_logist": 13, "sum_m": 3, "sum_n": 3, "sum_nx_": 3, "summar": [5, 6, 9, 34, 35], "summari": [1, 3, 4, 10, 26, 32, 33], "summat": [0, 3, 16, 31, 32], "sunni": 9, "super": [5, 31, 32, 33], "superfici": 3, "superscript": [1, 12, 36, 37], "supervis": [0, 5, 6, 7, 9, 12, 23, 30, 31, 32, 34, 35, 36, 37], "supplement": [7, 25, 35, 36], "suppli": [], "support": [0, 1, 9, 10, 11, 13, 20, 21, 23, 30, 31, 33, 35, 36, 37], "suppos": [0, 5, 6, 7, 8, 10, 11, 12, 13, 24, 30, 31, 32, 33, 34, 35, 36, 37], "suppress": [5, 13, 32], "sure": [0, 1, 4, 6, 16, 20, 21, 22, 25], "surf": 6, "surfac": [0, 6, 30, 33], "surpass": 6, "surpris": [0, 30], "surround": [3, 23], "survei": [0, 5, 6, 30, 31], "svc": [8, 9, 10], "svd": [0, 6, 11, 30, 34], "svdinv": 5, "svm": [8, 9, 10, 11], "svm_clf": [8, 10], "svn": [], "swap": 21, "swath": [5, 31, 32], "switch": 0, "sy": [13, 32, 33], "symbol": [1, 5, 11, 13, 23, 27, 30, 31, 32, 37], "symmeteri": 1, "symmetr": [0, 5, 8, 11, 12, 13, 24, 30, 31, 36, 37], "symmetri": 6, "sympi": [0, 23, 25, 30, 37], "synonim": 27, "syntax": 13, "system": [0, 1, 3, 4, 6, 7, 9, 10, 12, 13, 15, 23, 24, 25, 30, 32, 33, 35, 36, 37], "systemat": [4, 6, 34, 35], "t": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 21, 22, 23, 24, 25, 27, 28, 30, 32, 33, 34, 35, 36, 37], "t0": [3, 6, 13, 33], "t1": [2, 13, 33], "t2": 2, "t3": 2, "t9jjwsmsd1o": 34, "t_": 2, "t_0": [2, 9, 13, 33], "t_1": [13, 33], "t_b": 10, "t_i": [1, 2, 5, 12, 31, 32], "t_j": 12, "t_k": 9, "tabl": [9, 25, 27, 28, 30, 36], "tabul": [0, 30], "tabular": 30, "tackl": 4, "tag": [2, 3, 4, 5, 6, 7, 12, 13, 14, 24, 27, 31, 32, 35, 36, 37], "tagrget": 37, "taht": [0, 30], "tail": 27, "tailor": [2, 8, 11, 30, 37], "taiwan": [0, 30], "take": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 17, 19, 21, 22, 23, 24, 27, 30, 31, 32, 33, 34, 35, 36, 37], "taken": [0, 1, 3, 6, 10, 13, 21, 24, 34], "tan": 3, "tangent": [1, 4, 12, 13, 32, 36], "tanh": [1, 4, 7, 8, 12, 35, 36], "target": [0, 1, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 15, 16, 18, 19, 21, 22, 30, 31, 32, 33, 34, 35, 36, 37], "target_nam": [9, 21], "task": [0, 1, 3, 6, 9, 11, 12, 14, 21, 25, 30, 33, 34, 35, 36, 37], "tau": [3, 5, 27], "taught": 30, "tax": [], "taylor": [2, 13, 32, 37], "taylornr": [13, 32], "tc": 8, "teach": [15, 26, 30, 34], "team": 1, "teaser": 0, "technic": [0, 5, 6, 13, 25, 32, 33, 34], "techniqu": [0, 1, 8, 10, 13, 23, 27, 29, 30, 31, 33, 34, 35], "technologi": [0, 1], "tell": [0, 4, 6, 10, 11, 13, 16, 27, 33, 34, 35], "temp": 1, "temp1": 1, "temp2": 1, "temperatur": [0, 9, 30], "templat": [18, 20], "temporari": [], "temporarili": 1, "ten": [3, 30, 37], "tend": [3, 5, 6, 8, 9, 10, 12, 13, 14, 31, 33, 34, 35], "tendenc": [0, 30], "tension": [6, 34, 35], "tensor": 3, "tensorflow": [0, 2, 4, 8, 14, 23, 24, 25, 29, 30, 31], "term": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 16, 18, 19, 22, 25, 27, 30, 31, 32, 33, 35, 36], "term1": [5, 6, 11], "term2": [5, 6, 11], "term3": [5, 6, 11], "term4": [5, 6, 11], "termin": [0, 4, 5, 9, 10, 13, 15, 31, 32, 33], "terminarl": 15, "terrain": 6, "terrain1": 6, "test": [3, 4, 5, 6, 7, 8, 9, 10, 13, 16, 19, 20, 21, 24, 25, 27, 30, 32, 33, 34, 35, 36], "test_acc": 3, "test_accuraci": [1, 3], "test_error": 6, "test_imag": [3, 4], "test_ind": [6, 34, 35], "test_input": 4, "test_label": [3, 4], "test_loss": 3, "test_pr": 1, "test_predict": 1, "test_rnn": 4, "test_scor": [7, 10, 36], "test_siz": [0, 1, 3, 5, 6, 10, 15, 17, 31, 32, 33, 34, 35], "test_split": 9, "testerror": [0, 6, 31, 34, 35], "testi": 4, "testpredict": 4, "testx": 4, "tex": [], "text": [0, 1, 2, 4, 5, 8, 9, 11, 13, 15, 18, 20, 24, 25, 27, 29, 31, 32, 33, 34, 35], "textbf": [], "textbook": [16, 25, 31, 32, 34, 35], "textual": 9, "textur": 1, "tf": [1, 3, 4, 13, 14, 32], "th": [0, 1, 2, 5, 6, 7, 9, 12, 13, 14, 24, 25, 27, 30, 31, 33, 34, 35, 36, 37], "than": [0, 1, 2, 3, 4, 5, 6, 7, 9, 10, 11, 12, 13, 17, 21, 23, 27, 30, 31, 33, 34, 35, 36, 37], "thank": [4, 6, 31, 33], "theano": [1, 23, 30], "thei": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 11, 12, 13, 15, 16, 18, 20, 22, 24, 25, 27, 30, 31, 32, 33, 34, 35, 36, 37], "them": [0, 1, 3, 4, 6, 8, 9, 10, 11, 12, 13, 18, 21, 24, 25, 30, 31, 36, 37], "theme": [0, 15, 30], "themselv": [0, 25, 27, 30, 33], "thenc": [6, 34, 35], "theorem": [2, 6, 7, 31, 32, 35, 36], "theoret": [0, 4, 10], "theori": [0, 1, 3, 8, 9, 12, 13, 19, 23, 25, 29, 30, 33, 36, 37], "thereaft": [0, 5, 6, 11, 12, 24, 25, 30, 34, 35, 37], "therebi": [0, 5, 7, 11, 25, 30, 31, 32, 35, 36, 37], "therefor": [0, 1, 2, 3, 4, 6, 7, 8, 11, 13, 19, 27, 30, 31, 32, 33, 34, 35, 36], "therein": 11, "thereof": [0, 6, 13, 30, 33, 34], "theta": [0, 1, 4, 5, 6, 7, 13, 16, 25, 27, 30, 31, 32, 33, 35, 36, 37], "theta1": 33, "theta2": 33, "theta_": [0, 1, 6, 7, 13, 30, 31, 32, 33, 35, 36], "theta_0": [0, 5, 6, 7, 16, 30, 31, 32, 33, 35, 36], "theta_0x_": [0, 30, 31], "theta_1": [0, 5, 6, 7, 30, 31, 32, 33, 35, 36], "theta_1x_": [0, 30, 31], "theta_1x_0": [0, 30], "theta_1x_1": [0, 7, 30, 35, 36], "theta_1x_2": [0, 30], "theta_1x_i": [7, 31, 32, 33, 35, 36], "theta_2": [0, 30, 31], "theta_2x_": [0, 30, 31], "theta_2x_0": [0, 30], "theta_2x_1": [0, 30], "theta_2x_2": [0, 7, 30, 35, 36], "theta_2x_i": 31, "theta_3x_i": 31, "theta_4x_i": 31, "theta_closed_form": 18, "theta_closed_formol": 18, "theta_closed_formridg": 18, "theta_gdol": 18, "theta_gdridg": 18, "theta_i": [0, 1, 5, 30, 31, 32], "theta_j": [0, 5, 6, 18, 30, 31, 33], "theta_k": [32, 33], "theta_linreg": [13, 32, 33], "theta_ol": 18, "theta_p": [7, 35, 36], "theta_px_p": [7, 35, 36], "theta_ridg": 18, "theta_t": [13, 33], "theta_tru": 18, "thetaand": 36, "thetaith": 33, "thetaor": 36, "thetavalu": 5, "thetaxor": 36, "thi": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 23, 24, 25, 26, 27, 29, 31, 32, 33, 34, 35, 36], "thing": [0, 1, 2, 4, 5, 7, 9, 15, 16, 18, 21, 22, 27, 30, 34, 36], "think": [0, 1, 3, 4, 6, 9, 12, 13, 14, 27, 30, 31, 32, 33, 34, 36], "third": [0, 3, 6, 13, 28, 30, 32, 33], "thirti": [7, 36], "thorughout": 30, "those": [0, 3, 5, 6, 8, 9, 10, 11, 24, 25, 30, 31, 32, 33, 34, 35, 37], "though": [1, 2, 3, 4, 13, 16, 17, 19, 21, 22, 24, 27, 33], "thought": [6, 14, 25, 27, 34, 35], "thousand": [0, 1, 25, 31, 33], "three": [0, 1, 3, 5, 6, 8, 9, 12, 21, 24, 25, 26, 27, 28, 30, 31, 32, 34, 35, 36], "threshold": [1, 3, 9, 10, 11, 12, 13, 33, 35, 36, 37], "through": [0, 1, 2, 3, 4, 5, 6, 8, 11, 12, 13, 14, 15, 21, 22, 23, 24, 25, 27, 30, 31, 32, 33, 34, 36], "throughout": [0, 4, 5, 14, 15, 23, 24, 27, 30], "throw": [3, 6, 27, 34], "thu": [0, 1, 2, 5, 6, 7, 8, 10, 11, 12, 13, 28, 30, 31, 32, 33, 34, 35, 36, 37], "thumb": [0, 6, 25, 31], "thursdai": [], "tibshirani": [6, 19, 25, 29, 30, 34, 35], "tick_param": 6, "ticker": [6, 13, 27, 32, 33], "tif": 6, "tight_layout": [1, 7, 36], "tightli": 11, "tild": [0, 5, 6, 7, 11, 19, 25, 27, 30, 31, 32, 33, 34, 35, 37], "till": [0, 4, 7, 8, 9, 10, 12, 24, 30, 31, 35, 36, 37], "time": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 20, 21, 22, 23, 24, 25, 27, 30, 31, 32, 34, 35, 36, 37], "timeit": 4, "timer": 4, "timeseri": [], "tini": [1, 33], "tip": 3, "titl": [0, 1, 2, 3, 4, 6, 7, 8, 9, 10, 13, 15, 20, 21, 27, 30, 32, 33, 34, 35], "tm": [], "tmp": 13, "tn": [2, 3, 7], "to_categor": [1, 3, 4], "to_categorical_numpi": 1, "to_numer": [0, 6, 30, 34, 35], "todai": 3, "togeth": [0, 3, 6, 8, 11, 13, 22, 23, 30], "toi": 14, "token": [], "told": 13, "toler": [2, 14], "tolist": 4, "tomographi": [12, 36, 37], "too": [0, 2, 4, 5, 6, 9, 11, 13, 17, 18, 27, 29, 31, 32, 33, 34, 35], "took": [8, 30], "tool": [0, 1, 3, 6, 13, 15, 23, 31, 34, 35], "toolbox": 8, "top": [0, 3, 5, 6, 9, 10, 19, 23, 30, 34], "topic": [0, 5, 6, 7, 8, 23, 25, 31, 32, 34, 35, 36, 37], "topolog": [3, 12, 36, 37], "topologi": [1, 12], "torkjellsdatt": [28, 30], "tort": [], "toss": [10, 27], "total": [0, 1, 2, 3, 4, 6, 7, 8, 10, 11, 12, 13, 14, 24, 27, 28, 30, 31, 32, 33, 34, 35, 36, 37], "total_loss": 4, "totalclustervari": 14, "totalscatt": 14, "toward": [1, 2, 7, 12, 13, 15, 32, 35, 36], "towardsdatasci": 33, "town": [], "tp": [4, 7], "tpng": 9, "tpu": [13, 23, 30], "tqdm": 6, "tr": [], "track": [3, 13, 14, 15, 22, 24, 31, 32, 33], "tract": [], "tractabl": [0, 30, 31], "trade": [5, 9, 20, 33, 34], "tradeoff": [0, 5, 19, 25, 30, 31, 32], "tradit": [0, 1, 4, 6, 30, 34, 35], "train": [2, 3, 5, 6, 8, 9, 10, 11, 12, 13, 16, 17, 20, 25, 32, 33, 34, 35, 36], "train_accuraci": [0, 1, 3, 30], "train_dataset": 4, "train_end": [0, 1, 31], "train_error": 6, "train_imag": [3, 4], "train_ind": [6, 34, 35], "train_label": [3, 4], "train_network": 21, "train_pr": 1, "train_siz": [0, 1, 3, 31], "train_step": 4, "train_test_split": [0, 1, 3, 5, 6, 7, 9, 10, 11, 15, 16, 17, 19, 30, 31, 32, 33, 34, 35, 36], "train_test_split_numpi": [0, 1, 31], "trainable_vari": 4, "trained_model": [6, 31, 33], "trainerror": [0, 31], "traini": 4, "training_checkpoint": 4, "training_dataset": 4, "training_gradi": [13, 33], "trainingerror": [6, 34, 35], "trainpredict": 4, "trainscor": 4, "trainx": 4, "trait": [0, 30], "trajectori": [4, 33], "transfer": [9, 30], "transform": [0, 5, 6, 7, 8, 9, 10, 11, 12, 13, 17, 21, 23, 24, 30, 31, 32, 33, 34, 35, 36, 37], "transit": [6, 12, 36, 37], "translat": [1, 4, 6, 10, 30, 31, 33], "transpos": [1, 5, 11, 21, 24, 31, 32], "travers": [0, 5], "travi": [], "treat": [0, 1, 3, 6, 12, 13, 18, 21, 27, 30, 31, 32, 33, 34, 35, 36, 37], "tree": [0, 1, 23, 30], "tree_clf": [9, 10], "tree_clf_": 9, "tree_clf_sr": 9, "tree_reg": 9, "tree_reg1": 9, "tree_reg2": 9, "trend": 27, "treue": 7, "trevor": [19, 25, 29], "tri": [2, 3, 4, 9, 13, 16, 33], "triain": 0, "trial": [0, 2, 4, 6, 13, 27, 30, 32, 33, 34, 35], "triangl": [13, 32], "triangular": 24, "trick": [3, 4, 8, 11, 13, 27, 33], "tricki": 22, "trickier": 27, "tridiagon": 24, "trillion": 23, "trim": [], "trivial": [0, 1, 5, 11, 27, 30, 32], "troffa": [], "troubl": [0, 8, 12, 15, 21, 22, 31, 33, 37], "truck": 3, "true": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 12, 13, 14, 16, 17, 18, 19, 21, 22, 24, 25, 27, 30, 31, 32, 33, 34, 35, 36], "true_beta": 31, "true_fun": [6, 34, 35], "true_theta": [6, 33], "truelabel": [35, 36], "truli": 30, "truncat": 37, "try": [0, 1, 2, 4, 5, 6, 7, 8, 9, 10, 11, 13, 14, 15, 18, 21, 22, 23, 24, 25, 27, 30, 31, 32, 33, 35, 36, 37], "tr\u00f6ger": [], "tucker": 8, "tuesdai": [28, 30, 35], "tumor": [7, 9, 35, 36], "tumour": [7, 36], "tunabl": 1, "tune": [4, 9, 13, 24, 30, 33], "tupl": 21, "turn": [0, 1, 5, 6, 7, 8, 9, 10, 11, 12, 13, 24, 25, 27, 30, 31, 32, 33, 34, 35, 36, 37], "tutori": [1, 4], "tv": 2, "tveito": 2, "tvw1zdmznwm": 36, "tweak": [1, 4, 10, 27], "twice": [13, 32], "twist": 11, "two": [0, 1, 2, 4, 5, 6, 7, 9, 10, 11, 12, 13, 15, 17, 21, 24, 25, 26, 27, 29, 30, 31, 32, 33, 34], "tx": [13, 32, 33, 36], "tx_1": [13, 32], "txt": [4, 15, 20, 25], "ty": [13, 32], "type": [0, 1, 3, 6, 8, 10, 13, 21, 24, 27, 31, 32, 33, 34], "typeset": 20, "typic": [0, 1, 2, 3, 4, 5, 7, 9, 10, 12, 13, 15, 16, 20, 27, 30, 31, 32, 33, 34, 35, 36, 37], "typo": 25, "u": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 21, 24, 25, 27, 29, 30, 31, 32, 33, 34, 35, 36, 37], "u_": 24, "u_i": [12, 36], "u_m": 10, "ua": [0, 30], "ubuntu": [0, 23, 25, 30], "uci": 25, "ufunc": [], "uio": [15, 20, 21, 25, 28, 29], "uk": [], "un": 14, "unabl": [15, 21], "unari": [24, 30], "unbalanc": [6, 9, 34, 35], "unbias": [0, 5, 6, 30, 34], "uncent": [6, 31, 33], "uncertainti": [0, 5, 30], "uncertitud": 27, "unchang": [1, 3], "uncom": [], "uncorrel": [10, 27], "undefin": [5, 31, 32], "under": [0, 1, 5, 6, 10, 13, 23, 25, 30, 31, 32, 33, 34], "underdetermin": [0, 30], "underfit": [1, 6, 34, 35], "underflowproblem": [5, 34], "undergo": [5, 21], "undergradu": [26, 28], "underli": [0, 1, 9, 13, 18, 27, 30, 33], "underlin": [], "underscor": [], "underset": [4, 14], "understand": [0, 1, 3, 5, 6, 10, 13, 14, 15, 19, 20, 21, 23, 30, 31, 32, 33, 37], "understood": [8, 13], "underwai": [], "undesir": 8, "undetermin": [5, 8, 34], "undo": 4, "unexpect": [6, 34], "unexpected": 27, "unexplain": 18, "unfair": [6, 31], "unfortun": [1, 8, 9, 10], "unicode_liter": [8, 9], "uniform": [0, 1, 5, 6, 11, 13, 25, 27, 30, 32, 33, 35, 36], "uniformli": [13, 27, 32, 33], "unifrompdf": 27, "unimport": [13, 32], "union": [5, 6, 34, 35], "uniqu": [0, 2, 6, 13, 14, 24, 30, 34, 35, 36], "unique_class": [35, 36], "unique_cluster_label": 14, "unit": [0, 1, 3, 4, 5, 10, 12, 18, 27, 30, 31, 32, 33, 36, 37], "unitari": [5, 6, 24, 31, 32], "unitarili": [24, 30], "uniti": 27, "univari": 27, "univers": [0, 1, 2, 13, 23, 25, 26, 28, 30, 31, 32, 33, 34, 35, 36], "unix": 1, "unknow": [0, 24, 30], "unknown": [0, 1, 3, 4, 5, 6, 8, 10, 13, 19, 24, 25, 30, 31, 32, 33, 34, 35, 37], "unknowwn": 12, "unlabel": 1, "unless": [0, 3, 6, 11, 13, 25, 30, 32, 34, 37], "unlik": [1, 3, 8, 13, 32, 33], "unnecessarili": 9, "unord": 3, "unpickl": [], "unpleas": [], "unpublish": 33, "unravel": 1, "unrol": [3, 11], "unscal": 19, "unseen": [0, 7, 9, 15, 35, 36], "unstabl": 1, "unsupervis": [0, 1, 4, 12, 23, 30, 36, 37], "unsymmetr": [24, 30], "until": [1, 2, 4, 9, 12, 13, 14, 21, 32, 33, 36], "untouch": 0, "unusu": [12, 36, 37], "up": [1, 3, 4, 5, 6, 8, 10, 11, 13, 14, 16, 18, 19, 20, 21, 22, 23, 24, 25, 27, 28, 33, 36], "updat": [1, 2, 10, 12, 13, 14, 15, 18, 19, 21, 22, 34, 35, 36], "update_weight": 22, "uploa": 30, "upload": [15, 20, 23, 25, 29], "upon": [0, 1, 6, 7, 11, 24, 37], "upper": [0, 8, 9, 16, 24, 31], "uppercas": [24, 30], "upsampl": 4, "upscal": 4, "uptad": 37, "upward": [], "url": [30, 31, 36], "us": [4, 5, 6, 8, 9, 10, 11, 12, 14, 15, 17, 20, 21, 24, 27, 29, 34], "usag": [0, 8, 23, 30, 31, 37], "usd": [], "usd10000": [], "use_bia": 4, "usecol": [0, 30], "useless": 1, "user": [0, 1, 2, 4, 6, 7, 15, 23, 24, 25, 30, 31, 35, 36], "usernam": [15, 25], "usetex": 27, "usg": 6, "usr": 27, "usual": [0, 3, 4, 7, 12, 13, 14, 30, 33, 35, 36, 37], "ut": 5, "utf": [], "util": [1, 3, 4, 6, 7, 10, 14, 19, 30, 34, 35], "ux": 24, "v": [2, 4, 5, 6, 11, 13, 15, 23, 31, 32, 34, 35, 36, 37], "v0": 27, "v1": 27, "v2": 27, "v5": [], "v8xr": [36, 37], "v_": 33, "v_0": [11, 33], "v_t": 33, "va": 1, "vahid": 30, "val": 13, "val_accuraci": 3, "val_loss": 4, "vale": 2, "valid": [0, 1, 4, 7, 9, 10, 13, 23, 27, 30, 31, 33, 36], "validation_data": 3, "validation_split": 4, "valu": [0, 1, 2, 3, 4, 6, 7, 8, 9, 10, 12, 13, 14, 16, 17, 18, 20, 21, 22, 23, 24, 25, 30, 33, 36, 37], "valuat": 9, "valueerror": [], "valy": 4, "van": [0, 19, 25, 30, 31, 32, 33], "vandenbergh": [8, 13, 32], "vandermond": [0, 30], "vanilla": [0, 6, 11, 14, 31, 33], "vanish": [1, 4, 13, 27, 32, 37], "var": [5, 6, 10, 11, 19, 25, 27, 31, 34, 35], "var_x": 27, "varabl": 8, "varepsilon": [5, 6, 19, 34], "varepsilon_": [5, 6, 34], "varepsilon_i": [5, 6, 34], "vari": [0, 1, 3, 5, 6, 10, 21, 30, 34, 35, 37], "variabl": [0, 1, 2, 5, 6, 7, 8, 10, 11, 12, 13, 14, 21, 24, 30, 31, 33, 34, 35, 36, 37], "varianc": [0, 1, 5, 7, 9, 10, 11, 13, 14, 18, 20, 23, 24, 27, 30, 31, 32, 33, 36], "variance_i": [5, 11, 31], "variance_x": [5, 11, 31], "variant": [0, 1, 6, 8, 12, 13, 30, 31, 32, 33, 36, 37], "variat": [3, 4, 11, 30], "varieti": [0, 3, 12, 23, 25, 30, 36, 37], "variou": [1, 3, 5, 6, 7, 8, 9, 11, 12, 13, 16, 19, 20, 23, 24, 25, 27, 30, 31, 32, 33, 36, 37], "varydimens": 4, "vast": 33, "vastli": 3, "vaue": 1, "vault": 0, "vdot": [2, 13, 32, 33], "ve": [25, 33], "vec": [6, 34], "vector": [0, 1, 2, 3, 4, 5, 6, 7, 9, 10, 11, 13, 14, 17, 18, 21, 22, 23, 32, 33, 34, 35, 37], "vector_mean": 14, "ventur": [0, 8, 23, 30], "venv": 15, "verbos": [1, 3, 4, 35, 36], "veri": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 16, 18, 21, 22, 25, 27, 29, 30, 31, 32, 33, 34, 35], "verifi": [3, 11, 24, 30], "versatil": [8, 30], "versicolor": [8, 9], "version": [0, 3, 10, 13, 14, 15, 21, 22, 23, 24, 25, 27, 30], "versu": [1, 33], "vert": [0, 1, 5, 6, 7, 8, 9, 11, 13, 16, 17, 30, 31, 32, 33, 34, 35, 36, 37], "vert_1": [5, 6, 31, 32, 33], "vert_2": [5, 6, 11, 17, 31, 32, 33, 34], "via": [0, 5, 6, 7, 8, 9, 10, 11, 12, 19, 23, 24, 25, 26, 27, 28, 30, 31, 32, 33, 34, 35, 36, 37], "vidal": 11, "video": [0, 1, 12, 23, 26, 28, 30, 31, 32], "view": [1, 3, 5, 6, 12, 13, 27, 29, 30, 32, 33, 34, 36], "violat": 8, "virginica": 9, "viridi": [0, 1, 2, 3, 30], "virtanen": [], "virtual": [1, 33], "viscos": 13, "viscou": 13, "visibl": 15, "vision": [0, 3], "visit": 33, "visual": [0, 3, 11, 12, 18, 23, 30, 31, 36, 37], "visualis": 1, "visualstudio": [15, 16, 19], "viz": [6, 8, 27], "vmap": 13, "vmax": [1, 6], "vmh0zpt0tli": 33, "vmin": [1, 6], "voic": 3, "volatil": 33, "volum": [0, 3, 30], "von": 37, "vote": [10, 30], "voting_clf": 10, "votingclassifi": 10, "votingsimpl": 10, "vscode": [21, 22], "vstack": [5, 11, 24, 27, 30, 31, 35, 36], "vt": [5, 31, 32], "w": [0, 1, 2, 3, 4, 5, 6, 7, 8, 10, 11, 12, 13, 14, 21, 22, 24, 27, 30, 31, 32, 33, 34, 35, 36, 37], "w1": [8, 21, 22], "w2": [8, 11, 21, 22], "w3": 8, "w_": [1, 12, 36, 37], "w_0": 37, "w_1": [8, 24, 37], "w_1a_0": 37, "w_1x": 37, "w_1x_": 8, "w_1x_1": 8, "w_2": [8, 24, 37], "w_2a_1": 37, "w_2x_": 8, "w_2x_2": 8, "w_3": 24, "w_4": 24, "w_g": [21, 22], "w_hidden": 2, "w_i": [1, 2, 10, 37], "w_ix_i": [12, 36, 37], "w_j": 24, "w_m": 24, "w_output": 2, "w_px_": 8, "w_px_p": 8, "w_t": [], "wa": [1, 3, 4, 5, 6, 7, 10, 11, 12, 14, 17, 19, 21, 24, 30, 31, 33, 34, 35, 36, 37], "wai": [0, 1, 2, 3, 4, 5, 6, 7, 8, 10, 11, 12, 13, 14, 15, 18, 19, 21, 22, 24, 27, 30, 31, 32, 33, 36], "walk": 9, "walker": 27, "wall": 33, "walt": [], "wang": [0, 30], "want": [0, 1, 2, 3, 4, 5, 6, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 20, 21, 22, 23, 25, 27, 30, 31, 32, 33, 34, 35, 37], "warn": 4, "warrant": [6, 34, 35], "warranti": [], "wast": [3, 33], "watch": [23, 32, 33, 34, 36, 37], "wave": 3, "wavelet": 8, "wcag": [], "we": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 27, 28, 29, 31, 32, 34, 35, 36], "weak": [9, 10, 14], "weaker": 33, "weather": [1, 12, 36, 37], "web": [23, 26, 28, 30], "webpag": 30, "websit": [6, 24, 25, 26, 30], "wedg": [8, 27, 37], "wednesdai": [28, 30, 35], "wee": 11, "week": [0, 5, 6, 7, 25, 26, 28], "weekli": [15, 16, 23, 25, 26, 28, 29, 30, 36], "weierstrass": 37, "weight": [1, 2, 3, 6, 7, 9, 10, 12, 13, 18, 21, 22, 27, 33, 35, 36, 37], "weigth": [2, 22], "welchlab": [36, 37], "welcom": [8, 15, 23], "well": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 12, 13, 15, 16, 20, 21, 22, 23, 24, 25, 27, 29, 30, 31, 32, 33, 34, 35, 36, 37], "went": 8, "were": [0, 1, 3, 4, 5, 6, 7, 8, 10, 11, 12, 14, 27, 30, 33, 34, 35, 36, 37], "wessel": [0, 19, 25, 30, 31, 32, 33], "wg_nf1awssi": 37, "what": [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 19, 20, 21, 22, 23, 24, 25, 27, 33, 36, 37], "whatev": [3, 21], "when": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 18, 19, 21, 22, 24, 25, 27, 30, 31, 32, 34, 35, 36, 37], "whenev": [13, 15, 27, 33, 37], "where": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 20, 21, 22, 23, 24, 25, 27, 28, 30, 31, 32, 33, 34, 35, 36, 37], "wherea": [6, 27, 33, 34, 35], "wherefrom": 25, "wherein": [1, 12, 36, 37], "whether": [0, 3, 5, 7, 9, 25, 27, 30, 35, 36], "which": [0, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 30, 31, 32, 34, 35, 36, 37], "whichev": [1, 3], "while": [0, 1, 3, 4, 5, 6, 7, 8, 9, 11, 12, 13, 15, 16, 19, 20, 21, 22, 27, 30, 31, 32, 33, 34, 35, 36, 37], "white": 9, "whiteboad": 33, "whiteboard": [31, 32, 33, 34, 35, 36, 37], "who": [0, 15], "whole": [1, 3, 4, 5, 9, 11, 13, 21, 33], "whom": [], "whose": [0, 6, 10, 27, 31, 34, 35], "whow": [11, 31], "why": [0, 1, 3, 6, 13, 15, 16, 17, 19, 21, 25, 31, 32], "wide": [0, 1, 3, 6, 7, 12, 23, 24, 25, 30, 34, 35, 36, 37], "widehat": [6, 34], "width": [0, 3, 8, 9, 21, 30], "wieringen": [0, 19, 25, 30, 31, 32, 33], "wiki": 25, "wikipedia": 25, "win": [10, 33], "wind": 9, "window": [], "wing": [28, 30], "winther": 2, "wiothout": 6, "wiscons": 7, "wisconsin": [10, 36], "wisdom": [6, 31, 33], "wise": [1, 5, 12, 13, 21, 31, 32, 33, 36], "wish": [0, 2, 5, 7, 8, 11, 13, 14, 18, 24, 25, 30, 31, 32, 33, 35, 36, 37], "with_std": [0, 31], "wither": 6, "within": [0, 2, 3, 4, 7, 9, 12, 13, 14, 27, 29, 30, 32, 35, 36], "withinclust": 14, "without": [0, 1, 5, 6, 8, 9, 11, 12, 13, 15, 18, 25, 30, 31, 32, 33, 34, 35, 36, 37], "wo5dmep_bbi": [36, 37], "won": [0, 15, 30, 37], "wonder": 8, "word": [0, 1, 3, 4, 5, 6, 7, 14, 19, 25, 27, 30, 31, 32, 33], "work": [0, 1, 4, 6, 7, 8, 9, 13, 15, 16, 18, 19, 20, 21, 22, 23, 25, 26, 27, 28, 30, 31, 33, 34, 35, 36, 37], "workabl": 33, "workaround": [], "workhors": 33, "workload": 33, "workshop": 30, "world": [0, 8, 16, 31], "worldwid": [0, 30], "worri": 15, "wors": [0, 1, 3, 4, 6, 30, 33, 34, 35], "worth": [9, 19, 21], "worthi": 25, "would": [0, 1, 3, 5, 6, 7, 8, 9, 10, 11, 12, 13, 16, 18, 20, 22, 24, 25, 27, 30, 31, 32, 33, 34, 35, 36, 37], "wouldn": [], "wrap": [6, 24, 30], "wrapper": [21, 22], "write": [0, 1, 2, 3, 5, 6, 7, 8, 12, 13, 15, 16, 18, 21, 24, 30, 31, 33, 34, 35, 36, 37], "writer": [35, 36], "writerow": [35, 36], "written": [0, 2, 3, 5, 11, 12, 13, 16, 23, 24, 25, 27, 30, 31, 32, 33, 37], "wrong": [1, 8, 15, 19], "wrongli": 10, "wrote": [5, 11, 31], "wrt": [10, 13, 21, 22, 33, 37], "wth": [10, 13, 33], "wurstemberg": 37, "www": [20, 23, 24, 25, 29, 30, 32, 33, 34, 36, 37], "wx_1": 8, "x": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 21, 22, 24, 25, 27, 30, 32, 33, 34, 35, 36, 37], "x0": [8, 35, 36], "x1": [4, 8, 9, 10, 13, 35, 36], "x1_exampl": 8, "x1d": 8, "x2": [8, 9, 10, 13], "x2d": [8, 11], "x2d_train": 11, "x2dsl": 11, "x3": 8, "x_": [0, 2, 3, 5, 6, 8, 10, 11, 13, 14, 24, 27, 30, 31, 32, 33, 34, 35, 37], "x_0": [0, 5, 11, 18, 24, 30, 31, 34, 37], "x_1": [0, 2, 5, 6, 7, 8, 9, 10, 11, 13, 18, 24, 27, 30, 31, 32, 33, 34, 35, 36, 37], "x_2": [0, 2, 5, 6, 7, 8, 9, 10, 11, 13, 24, 27, 30, 31, 32, 34, 35, 36, 37], "x_3": [8, 24, 27, 37], "x_4": [24, 37], "x_5": 37, "x_6": 18, "x_bin": [35, 36], "x_center": 11, "x_data": 1, "x_data_ful": 1, "x_hidden": 2, "x_i": [0, 1, 2, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 24, 27, 30, 31, 32, 33, 34, 35, 36, 37], "x_input": 2, "x_ix_": [0, 30], "x_iy_i": 8, "x_j": [0, 2, 8, 9, 12, 16, 27, 31, 33, 36, 37], "x_jy_j": 8, "x_k": [12, 14, 24, 27, 31, 36], "x_l": [27, 37], "x_m": [6, 12, 24, 27, 34, 36], "x_mean": [18, 33], "x_multi": [35, 36], "x_n": [0, 2, 3, 6, 8, 11, 12, 13, 24, 27, 30, 32, 34, 36, 37], "x_new": [9, 10], "x_norm": [18, 33], "x_offset": [6, 31, 33], "x_output": 2, "x_p": [3, 7, 9, 35, 36], "x_poli": 9, "x_poly10": 9, "x_pred": 4, "x_prev": 2, "x_reduc": 11, "x_sampl": [], "x_scale": 8, "x_small": 13, "x_std": [18, 33], "x_t": 33, "x_test": [0, 1, 3, 5, 6, 7, 9, 10, 11, 15, 16, 17, 19, 31, 32, 33, 34, 35, 36], "x_test_": 17, "x_test_own": 6, "x_test_scal": [0, 6, 7, 9, 10, 11, 31, 33], "x_tot": 4, "x_train": [0, 1, 3, 4, 5, 6, 7, 9, 10, 11, 15, 16, 17, 19, 30, 31, 32, 33, 34, 35, 36], "x_train_": 17, "x_train_mean": [6, 31, 33], "x_train_own": 6, "x_train_r": 19, "x_train_scal": [0, 6, 7, 9, 10, 11, 31, 33], "x_val": 1, "xarrai": [23, 30], "xavier": 1, "xbnew": [13, 32, 33], "xcode": [0, 23, 25, 30], "xdclassiffierconfus": 10, "xdclassiffierroc": 10, "xg_clf": 10, "xgb": 10, "xgbclassifi": 10, "xgboost": 9, "xgboot": 10, "xgbregressor": 10, "xgparam": 10, "xgtree": 10, "xi": [8, 13, 33, 35, 36], "xi_": 8, "xi_1": 8, "xi_i": 8, "xinv": 36, "xk": 8, "xla": [13, 23, 30], "xlabel": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 13, 21, 27, 30, 31, 32, 33, 34, 35], "xlim": [6, 10, 34, 35], "xm": 9, "xmesh": 13, "xnew": [0, 13, 30, 32, 33], "xp": 27, "xpanda": [0, 31], "xpd": [5, 11, 31], "xplot": 0, "xscale": [0, 31], "xsr": 9, "xt_x": [13, 32, 33], "xtest": [6, 34, 35], "xtick": [3, 6, 8, 9, 34, 35], "xtrain": [6, 34, 35], "xu": [0, 30], "xx": [0, 24, 30], "xy": [0, 6, 8, 24, 30], "xytext": 8, "xyz": [], "xz": [24, 30], "y": [0, 1, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 24, 25, 27, 30, 31, 32, 33, 34, 35, 36, 37], "y1": 4, "y2": 4, "y3": 4, "y_": [0, 1, 5, 6, 10, 11, 24, 30, 31, 34, 35], "y_0": [0, 5, 11, 24, 30, 31, 34], "y_1": [0, 5, 8, 9, 11, 13, 24, 30, 31, 32, 33, 34], "y_1y_1": 8, "y_1y_1k": 8, "y_1y_2": 8, "y_1y_2k": 8, "y_1y_n": 8, "y_1y_nk": 8, "y_2": [0, 5, 8, 9, 11, 24, 30, 31], "y_2y_1": 8, "y_2y_1k": 8, "y_2y_2": 8, "y_2y_2k": 8, "y_3": [0, 9, 24], "y_4": 24, "y_bin": [35, 36], "y_binari": [35, 36], "y_center": [18, 33], "y_data": [0, 1, 5, 6, 30, 31, 32, 33], "y_data_ful": 1, "y_decis": 8, "y_fit": [0, 31], "y_i": [0, 1, 5, 6, 7, 8, 9, 10, 11, 12, 13, 19, 24, 25, 30, 31, 32, 33, 34, 35, 36, 37], "y_if_": 10, "y_indic": [35, 36], "y_ix_": [0, 30], "y_ix_i": [7, 8, 13, 31, 32, 33, 35, 36], "y_iy_jk": 8, "y_j": [6, 8, 12, 25, 34, 35, 36, 37], "y_k": [12, 36], "y_m": 24, "y_mean": [18, 33], "y_model": [0, 4, 5, 6, 30, 31, 32, 33], "y_multi": [35, 36], "y_n": [8, 13, 32, 33], "y_ny_1": 8, "y_ny_1k": 8, "y_ny_2": 8, "y_ny_2k": 8, "y_ny_n": 8, "y_ny_nk": 8, "y_offset": [6, 17, 31, 33], "y_onehot": [35, 36], "y_plot": 9, "y_pred": [0, 1, 4, 6, 7, 8, 9, 10, 31, 33, 34, 35, 36], "y_pred1": 9, "y_pred2": 9, "y_pred_bin": [35, 36], "y_pred_multi": [35, 36], "y_pred_rf": 10, "y_pred_tre": 10, "y_prob": [35, 36], "y_prob_bin": [35, 36], "y_prob_multi": [35, 36], "y_proba": [7, 10, 36], "y_sampl": [], "y_scaler": [6, 31, 33], "y_test": [0, 1, 3, 4, 5, 6, 7, 9, 10, 11, 15, 16, 17, 19, 31, 32, 33, 34, 35, 36], "y_test_onehot": 1, "y_test_predict": [], "y_tot": 4, "y_train": [0, 1, 3, 4, 5, 6, 7, 9, 10, 11, 15, 16, 17, 19, 30, 31, 32, 33, 34, 35, 36], "y_train_mean": [6, 31, 33], "y_train_onehot": 1, "y_train_predict": [], "y_train_r": 19, "y_train_scal": [6, 31, 33], "y_true": [35, 36], "y_val": 1, "yand": 36, "ye": [3, 6, 7, 34, 35, 36], "year": [0, 23, 30], "yet": [0, 1, 6, 8, 11, 13, 20, 21, 30, 35, 37], "yi": [13, 33, 35, 36], "yield": [0, 2, 5, 6, 8, 10, 12, 13, 14, 24, 27, 30, 32, 33, 34, 36, 37], "yk": 8, "ylabel": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 13, 21, 27, 30, 31, 32, 33, 34, 35], "ylim": [3, 6, 34, 35], "ym": 9, "ymesh": 13, "yn": 0, "yo": [8, 9, 10], "yor": 36, "yoshiki": [], "yoshua": [1, 29], "you": [0, 1, 3, 4, 5, 6, 8, 9, 10, 11, 13, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 27, 28, 29, 30, 31, 32, 33, 34, 35, 37], "young": 0, "your": [1, 2, 4, 5, 6, 8, 11, 13, 15, 17, 19, 20, 21, 22, 23, 24, 30, 32, 33, 34, 35, 36, 37], "your_model_object": 16, "yourself": [11, 13, 30, 32], "youtu": [31, 32, 34, 36], "youtub": [23, 32, 33, 34, 36, 37], "ypred": [6, 34, 35], "ypredict": [0, 13, 30, 31, 32, 33], "ypredict2": [13, 32, 33], "ypredictlasso": [5, 32], "ypredictol": [0, 5, 32], "ypredictown": [6, 31, 33], "ypredictownridg": [6, 31, 32, 33], "ypredictridg": [0, 5, 6, 31, 32, 33], "ypredictskl": [6, 31, 33], "ytest": [6, 34, 35], "ytick": [3, 6, 8, 9, 34, 35], "ytild": [0, 6, 30, 31, 34, 35], "ytildelasso": [5, 32], "ytildenp": [0, 30, 31], "ytildeol": [0, 5, 32], "ytildeownridg": [6, 31, 32, 33], "ytilderidg": [5, 6, 31, 32, 33], "ytrain": [6, 34, 35], "yuxi": 30, "yx": [24, 30], "yxor": 36, "yy": [24, 30], "yz": [24, 30], "z": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 11, 12, 13, 21, 22, 24, 27, 30, 31, 34, 35, 36, 37], "z1": [21, 22], "z2": [21, 22], "z_": [1, 2, 12, 24, 30, 37], "z_0": [24, 30, 37], "z_1": [24, 30, 37], "z_2": [22, 24, 30, 37], "z_c": 1, "z_h": 1, "z_hidden": 2, "z_i": [1, 12, 36], "z_j": [1, 12], "z_k": [12, 31, 37], "z_m": 1, "z_mod": 9, "z_o": 1, "z_output": 2, "za": [], "zaman": 27, "zaxi": 6, "zero": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 16, 17, 18, 19, 21, 24, 25, 27, 30, 31, 32, 33, 34, 35, 36, 37], "zeros_lik": [4, 35, 36], "zeroth": 31, "zfill": 4, "zip": [4, 6, 21, 22, 35, 36], "zm_h": [0, 30], "zn": [], "zone": [], "zoom": 30, "zscout": [], "zx": [24, 30], "zy": [24, 30], "zz": [24, 30], "\u00f8yvind": [6, 31, 33]}, "titles": ["3. Linear Regression", "14. Building a Feed Forward Neural Network", "15. Solving Differential Equations with Deep Learning", "16. Convolutional Neural Networks", "17. Recurrent neural networks: Overarching view", "4. Ridge and Lasso Regression", "5. Resampling Methods", "6. Logistic Regression", "8. Support Vector Machines, overarching aims", "9. Decision trees, overarching aims", "10. Ensemble Methods: From a Single Tree to Many Trees and Extreme Boosting, Meet the Jungle of Methods", "11. Basic ideas of the Principal Component Analysis (PCA)", "13. Neural networks", "7. Optimization, the central part of any Machine Learning algortithm", "12. Clustering and Unsupervised Learning", "Exercises week 34", "Exercises week 35", "Exercises week 36", "Exercises week 37", "Exercises week 38", "Exercises week 39", "Exercises week 41", "Exercises week 42", "Applied Data Analysis and Machine Learning", "2. Linear Algebra, Handling of Arrays and more Python Features", "Project 1 on Machine Learning, deadline October 6 (midnight), 2025", "Course setting", "1. Elements of Probability Theory and Statistical Data Analysis", "Teachers and Grading", "Textbooks", "Week 34: Introduction to the course, Logistics and Practicalities", "Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression", "Week 36: Linear Regression and Gradient descent", "Week 37: Gradient descent methods", "Week 38: Statistical analysis, bias-variance tradeoff and resampling methods", "Week 39: Resampling methods and logistic regression", "Week 40: Gradient descent methods (continued) and start Neural networks", "Week 41 Neural networks and constructing a neural network code"], "titleterms": {"": [8, 10, 32, 33, 34, 35, 36], "0": [], "04": [], "05": [], "06": [], "07": [], "1": [0, 15, 16, 17, 18, 19, 20, 21, 22, 25, 31, 37], "10": 37, "11": [], "15": [19, 34], "19": 19, "1a": 18, "2": [0, 15, 16, 17, 18, 19, 20, 21, 22, 30, 31, 32, 37], "20": [], "2017": [], "2018": [], "2019": [], "2023": 28, "2025": [25, 35, 36, 37], "22": 35, "26": 35, "27": [], "29": 36, "2a": [], "2b": [], "3": [0, 15, 16, 17, 18, 19, 20, 21, 22, 31, 37], "34": [15, 30], "35": [16, 31], "36": [17, 32], "37": [18, 33], "38": [19, 34], "39": [20, 35], "3a": 18, "3b": 18, "4": [0, 15, 16, 17, 18, 19, 20, 21, 22, 31], "40": 36, "41": [21, 37], "42": 22, "4a": 18, "4b": 18, "5": [0, 16, 18, 19, 20, 21, 22], "6": [21, 22, 25, 37], "7": [21, 22], "8": [22, 33], "A": [0, 1, 4, 8, 9, 30, 34, 35, 36], "AND": 36, "And": [30, 31, 33], "But": 33, "In": [28, 37], "Ising": 6, "OR": 36, "The": [0, 1, 2, 3, 5, 6, 7, 8, 9, 11, 12, 15, 23, 30, 31, 32, 33, 34, 35, 36, 37], "To": 30, "With": [4, 32], "a11i": [], "about": [30, 31, 32], "abov": [32, 37], "abstract": 20, "accuraci": 33, "across": 33, "activ": [1, 12, 21, 36, 37], "ad": [0, 6, 20, 25, 30, 31, 36, 37], "adaboost": 10, "adagrad": [13, 33], "adam": [13, 33], "adapt": [10, 33], "add": [], "adjust": 1, "advanc": 25, "adversari": 4, "again": [3, 9], "ai": [25, 30], "aim": [8, 9, 21, 22, 30], "aka": 30, "al": 33, "algebra": [24, 30], "algorithm": [9, 10, 11, 12, 30, 31, 32, 33, 37], "algortithm": [13, 32, 35, 36], "all": [8, 37], "an": [0, 4, 10, 15, 20, 30, 37], "analys": [5, 31, 32], "analysi": [0, 5, 6, 11, 23, 25, 27, 30, 31, 32, 34, 35, 37], "analyt": [0, 16, 18], "analyz": 37, "ani": [13, 22, 32, 35, 36], "anoth": [9, 32, 34, 35], "api": [], "appli": 23, "approach": [0, 8, 14, 30, 33, 34, 35], "approxim": [12, 37], "architectur": 1, "arrai": [24, 30], "artifici": [36, 37], "assist": 28, "assumpt": 34, "august": [], "author": [], "autocorrel": 27, "autograd": [2, 13, 22, 33], "automat": [13, 33, 37], "avail": 20, "avali": [], "averag": 33, "b": 25, "back": [1, 11, 12, 31, 32, 37], "background": [23, 25, 34], "backpropag": 22, "bag": 10, "base": [13, 33, 34], "basic": [0, 5, 7, 9, 10, 11, 24, 31, 32, 35, 36, 37], "batch": [1, 22, 33], "bay": 5, "befor": 11, "beta": [], "better": [8, 36], "bia": [6, 19, 25, 33, 34, 35], "bias": 37, "binari": 1, "bind": 30, "bird": 10, "blind": [], "block": [], "boldsymbol": [18, 31, 34], "book": [19, 37], "boost": 10, "bootstrap": [6, 10, 34, 35], "boston": [], "breast": 1, "brief": [30, 34, 35], "bring": [12, 37], "browser": [], "bsd": [], "build": [1, 3, 9], "c": [25, 30], "calcul": [18, 31, 32], "can": [30, 33, 34, 35, 37], "cancer": [1, 7, 9, 11], "cart": 9, "case": [8, 10, 27, 31, 32, 33, 35, 36], "cdn": [], "cell": [], "central": [13, 23, 27, 32, 34, 35, 36], "chain": [12, 37], "challeng": 33, "chang": 10, "changelog": [], "channel": 30, "chi": [0, 30], "choic": 17, "choos": [1, 33], "cifar01": 3, "citat": [], "class": [35, 36, 37], "classic": 11, "classif": [1, 9, 10, 35, 36], "classifi": [8, 35], "claus": [], "clip": 1, "cluster": 14, "cnn": 3, "code": [1, 2, 5, 9, 11, 12, 13, 14, 15, 16, 20, 25, 30, 31, 32, 33, 34, 35, 36, 37], "collect": [1, 3], "color": [], "colorblind": [], "combin": 33, "commun": 30, "compact": [35, 36, 37], "compar": [2, 10, 16], "comparison": [32, 33], "compet": 33, "compil": [], "complet": [31, 37], "complex": [0, 6, 25, 31], "complic": [6, 37], "compon": 11, "comput": [9, 19, 33], "computation": [34, 35], "computerlab": 30, "con": [9, 33], "concept": 27, "condit": 32, "confid": 34, "conjug": 13, "consider": 37, "constraint": 33, "construct": 37, "contain": [], "content": [], "continu": 36, "contn": 30, "contrast": [], "contributor": [], "converg": 33, "convex": [8, 13, 32, 33], "convolut": [3, 12, 36, 37], "copyright": [], "core": [], "correct": 33, "correl": [11, 31, 36], "correspond": [], "cost": [1, 10, 31, 32, 33, 34, 35, 36, 37], "count": 37, "cours": [23, 26, 29, 30], "covari": [5, 11, 27, 31], "cover": 30, "creat": [16, 20], "creator": [], "cross": [6, 25, 34, 35, 36], "custom": 21, "cython": 30, "d": 25, "dark": [], "data": [0, 1, 3, 6, 7, 9, 11, 15, 17, 18, 21, 23, 27, 30, 31, 35, 36, 37], "dataset": [1, 3, 18], "david": 30, "deadlin": [25, 30], "deadllin": 28, "decai": [2, 33], "decis": [9, 10], "decomposit": [5, 11, 24, 31, 32], "deeep": [], "deep": [1, 2, 30, 33, 35, 36, 37], "defin": [1, 30, 37], "definit": [19, 37], "deflist": [], "degre": [0, 17, 31], "deliver": [15, 16, 19, 20, 25], "deliveri": 25, "delta": 34, "dens": 0, "depend": [], "deriv": [5, 12, 16, 17, 19, 31, 32, 33, 34, 37], "descent": [2, 10, 13, 18, 25, 32, 33, 36], "design": 31, "detail": [3, 30], "develop": 1, "diagon": 11, "differ": [8, 33], "differenti": [2, 13, 33, 37], "diffus": 2, "dimens": 33, "dimension": [2, 3, 8, 18], "direct": [], "disadvantag": 9, "discret": 27, "discrimin": 30, "discuss": 36, "distribut": [5, 27, 34], "do": [1, 33, 36], "document": 20, "doe": [31, 32, 36], "domain": 27, "down": 1, "dropout": 1, "e": 25, "each": [21, 35], "economi": [31, 32], "electron": 25, "element": [0, 27, 30], "elimin": 24, "empir": 33, "energi": 30, "ensembl": 10, "entri": 37, "entropi": [9, 35, 36], "environ": [0, 15], "equat": [0, 2, 12, 31, 32, 35, 36, 37], "error": [0, 10, 30, 31, 32, 34, 35], "essenti": 30, "estim": 34, "et": 33, "etc": 30, "euler": 2, "evalu": [1, 37], "evid": 33, "exampl": [1, 2, 3, 4, 6, 7, 8, 9, 10, 30, 31, 32, 33, 34, 35, 36, 37], "exercis": [0, 6, 15, 16, 17, 18, 19, 20, 21, 22, 31, 37], "expect": [19, 27, 34], "expens": [34, 35], "experi": 27, "explicit": 37, "explor": 0, "exponenti": [2, 33], "express": [16, 17, 19, 31, 35, 36, 37], "extend": [32, 35, 36, 37], "extrapol": 4, "extrem": [10, 30], "ey": 10, "f": 25, "fall": 28, "famili": [1, 30], "famou": 24, "fantast": [31, 32], "faq": [], "featur": [9, 16, 24, 31], "februari": [], "feed": [1, 12, 22, 36, 37], "figur": 20, "file": [], "fill": [], "final": [12, 31, 33, 37], "find": [16, 18, 34], "fine": 1, "first": [4, 12, 30, 32, 37], "fit": [0, 10, 15, 16, 30, 32], "fix": [31, 32, 33], "float": 37, "fold": [34, 35], "forc": 3, "forest": 10, "form": 18, "format": [25, 30], "formula": 18, "forward": [1, 2, 12, 22, 36, 37], "foster": 30, "fourier": 3, "frank": 6, "freedom": [0, 17, 31], "frequent": [31, 33], "frequentist": [0, 30], "from": [5, 10, 12, 30, 31, 32, 33, 34, 35, 36, 37], "full": [2, 33], "function": [0, 1, 6, 7, 8, 10, 11, 12, 13, 25, 27, 30, 31, 32, 33, 34, 35, 36, 37], "further": [3, 5, 31, 32], "g": 25, "gan": 4, "gate": 36, "gaussian": 24, "gd": [13, 33], "gener": [4, 9, 30, 35, 36, 37], "geometr": [11, 32], "get": [20, 37], "gini": 9, "github": 15, "goal": [15, 16, 17, 18, 19, 20], "good": [0, 20, 30], "goodfellow": 33, "gotthard": [], "grade": [28, 30], "gradient": [1, 2, 10, 13, 18, 22, 25, 32, 33, 36, 37], "greativ": [], "group": 35, "growth": 2, "guid": [], "h": 25, "ha": 23, "hand": [22, 37], "handl": [24, 30], "happen": [34, 35], "hessian": [31, 32, 33], "hidden": [2, 37], "high": [], "histogram": 34, "histori": [], "hous": [], "how": 16, "hyperbol": 36, "hyperparamet": [1, 17], "hyperplan": 8, "i": [0, 1, 30], "id3": 9, "idea": 11, "ideal": 32, "ident": 34, "identifi": 34, "ii": 30, "iid": 34, "illustr": [32, 36, 37], "implement": [1, 16, 17, 18], "implic": [5, 31, 32], "import": [5, 24, 30, 31, 32, 37], "improv": [1, 33], "includ": [13, 25, 33, 35, 36, 37], "incorpor": [], "increment": 11, "independ": 34, "index": 9, "inform": 28, "ingredi": 37, "input": [2, 21, 22, 37], "instal": [23, 25, 30], "instructor": 28, "intermedi": 37, "interpret": [5, 11, 19, 30, 31, 32, 34], "interv": 34, "introduc": [11, 13, 31], "introduct": [0, 6, 20, 23, 24, 25, 30, 36, 37], "invers": [5, 24], "invert": [31, 32], "ipython": [], "iter": 10, "its": 31, "j": [], "jacobian": 31, "januari": [], "jax": 13, "job": 36, "julia": 30, "jungl": 10, "jupyt": [], "k": [34, 35, 37], "kera": [1, 3], "kernel": [8, 11], "l": 37, "lab": [32, 33, 34, 35, 36, 37], "lagrangian": 8, "lasso": [5, 6, 25, 31, 32], "last": [31, 33, 36, 37], "later": [5, 31, 32], "layer": [1, 2, 3, 12, 21, 22, 37], "layout": 37, "learn": [0, 1, 2, 11, 13, 14, 15, 16, 17, 18, 19, 20, 23, 25, 30, 31, 32, 33, 34, 35, 36, 37], "least": [5, 6, 16, 19, 25, 30, 31, 32, 33], "lectur": [30, 32, 33, 34, 35, 36, 37], "level": 10, "librari": [23, 30], "licens": [], "light": [], "likelihood": [7, 34, 35, 36], "limit": [1, 13, 27, 32, 33, 34], "linear": [0, 8, 13, 15, 24, 30, 31, 32, 35], "link": [5, 11, 29, 31, 34], "list": 37, "literatur": 25, "logist": [7, 30, 35, 36], "loss": [31, 32, 33], "lu": 24, "ma": [], "machin": [0, 8, 13, 23, 25, 30, 32, 35, 36], "made": 34, "main": [27, 30], "make": [0, 9, 10, 20, 31], "mani": [10, 12], "markdown": [], "mask": [], "maskedarrai": [], "mass": 30, "materi": [25, 30, 31, 32, 33, 34, 35, 37], "math": [5, 31, 32], "mathemat": [3, 5, 8, 31, 32, 36, 37], "matplotlib": [], "matric": [5, 24, 30], "matrix": [1, 5, 11, 12, 16, 24, 30, 31, 32, 33, 36], "matter": 0, "max": 31, "maximum": [34, 35, 36], "me": [], "mean": [0, 31, 32, 35], "measur": 36, "meet": [5, 10, 27, 30, 31], "memori": 33, "mercer": 8, "metadata": [], "method": [6, 9, 10, 13, 25, 30, 32, 33, 34, 35, 36], "metric": 19, "midnight": 25, "min": 31, "mini": 33, "minibatch": 33, "minim": [30, 35, 36], "mit": [], "ml": 30, "mle": 34, "mlp": 12, "mnist": [3, 4], "mode": 37, "model": [0, 1, 4, 6, 12, 15, 17, 30, 36, 37], "moment": 33, "momentum": [13, 25, 33], "mondai": [32, 33, 34, 36, 37], "moon": [8, 9], "more": [3, 6, 24, 25, 30, 31, 32, 33, 34, 35, 36, 37], "motiv": 33, "move": 33, "multi": [36, 37], "multilay": [12, 36, 37], "multipl": [1, 3, 17, 21], "multipli": 8, "multivari": 37, "myst": [], "ncsa": [], "need": [25, 30], "network": [1, 2, 3, 4, 7, 12, 30, 33, 35, 36, 37], "neural": [1, 2, 3, 4, 7, 12, 30, 33, 36, 37], "neuron": [36, 37], "new": [4, 18, 34, 37], "newton": [32, 33, 35, 36], "nn": 37, "node": 37, "non": [8, 33], "none": 33, "normal": [0, 1, 34], "notat": [12, 36], "note": [25, 31, 32], "notebook": [], "novemb": [], "now": [1, 9, 13, 32, 33, 34, 35], "nuclear": [0, 30], "nueral": 35, "numba": 30, "number": [0, 2, 22, 27, 31, 33, 37], "numer": [2, 25, 27], "numpi": [24, 30], "object": [3, 22], "observ": 37, "obtain": 11, "octob": [25, 37], "od": 2, "off": [6, 19, 25], "ol": [5, 6, 15, 16, 18, 25, 32, 34], "onc": 21, "one": [2, 12, 18, 22, 32, 37], "ones": 36, "open": [], "oper": [24, 37], "optim": [1, 8, 13, 18, 23, 30, 31, 32, 33, 35, 36, 37], "option": [21, 22], "order": [13, 18, 33], "ordinari": [5, 6, 16, 19, 25, 30, 31, 32, 33], "organ": [0, 30], "orient": 22, "oslo": 29, "other": [4, 9, 11, 12, 24, 25, 30, 36, 37], "ouput": 37, "our": [0, 4, 5, 11, 13, 25, 30, 31, 32, 35, 36], "outcom": [23, 30], "output": [2, 37], "over": 37, "overarch": [0, 4, 8, 9, 21, 22, 30, 31, 37], "overview": [10, 30, 33], "own": [0, 10, 11, 25, 30, 31], "packag": [24, 30], "panda": [30, 31], "parallel": 37, "paramet": [30, 31, 35, 36, 37], "paramt": 18, "part": [13, 23, 25, 32, 35, 36, 37], "partial": 2, "pass": [1, 22], "pca": 11, "pdf": 27, "percepetron": 37, "perceptron": [12, 36, 37], "perform": [1, 9], "period": 3, "perspect": 1, "pitaya": [], "plan": [31, 32, 33, 34, 35, 37], "plethora": 30, "plot": [34, 35], "point": [4, 37], "poisson": 2, "polici": [], "polynomi": [3, 16, 18, 32], "popul": 2, "popular": 30, "practic": [13, 28, 30, 33], "pre": [1, 3], "preambl": 25, "predict": [4, 21], "predictor": [35, 36], "preprocess": [31, 33], "prerequisit": [3, 23, 30], "present": 20, "princip": 11, "principl": 3, "pro": [9, 33], "probabl": [5, 27, 34], "problem": [1, 2, 13, 30, 31, 32, 33, 35, 36, 37], "procedur": [9, 30], "process": [1, 3, 21], "program": [2, 13, 25, 32, 33, 37], "project": [6, 20, 25, 28, 30], "prop": 13, "propag": [1, 12, 37], "properti": [5, 27, 31, 32, 33, 35], "python": [0, 9, 15, 23, 24, 30], "quick": 8, "quickli": [], "r": 30, "random": [10, 11, 27], "raphson": [32, 35, 36], "rate": [25, 33], "read": [9, 30, 31, 33, 34, 35, 36, 37], "real": [6, 21, 30, 37], "recommend": [30, 31], "record": [], "recurr": [4, 12, 36, 37], "reduc": [0, 31, 37], "reduct": 3, "refer": 25, "referenc": 20, "reformul": 2, "regress": [0, 5, 6, 7, 9, 10, 13, 15, 17, 18, 19, 25, 30, 31, 32, 33, 34, 35, 36], "regular": 1, "relat": [], "relev": [29, 31, 36], "relu": 1, "remark": 3, "remind": [6, 8, 30, 31, 32, 33, 37], "replac": [13, 33], "report": [20, 25], "repositori": [15, 34, 35], "requir": [2, 23], "resampl": [6, 19, 25, 34, 35], "rescal": [6, 31], "residu": [31, 32], "resourc": 2, "result": [31, 32, 37], "revers": 37, "revis": [], "revisit": [13, 32, 33, 35, 36], "rewrit": [30, 31, 34], "rewritten": [35, 36], "ridg": [0, 5, 6, 17, 18, 19, 25, 31, 32, 33], "rm": 13, "rmsprop": 33, "role": [], "rule": [12, 33, 37], "rung": 25, "same": [13, 33, 34, 35], "sampl": 11, "scalabl": 33, "scale": [17, 18, 19, 31, 33], "schedul": 30, "schemat": 9, "scheme": 2, "scienc": 30, "scikit": [0, 1, 11, 30, 31, 32, 33, 34, 35, 36], "second": [13, 18, 33], "select": 35, "semest": 28, "sensit": 32, "septemb": [19, 32, 33, 34, 35, 36], "seriou": 37, "session": [32, 33, 34, 35, 36, 37], "set": [0, 2, 3, 9, 12, 15, 26, 30, 31, 32, 37], "setup": 15, "sgd": [13, 33], "should": 1, "show": [], "similar": [13, 33], "simpl": [0, 4, 9, 13, 18, 30, 31, 32, 33, 35, 37], "simpler": 37, "simplest": 18, "singl": [10, 36, 37], "singular": [5, 11, 31, 32], "size": [31, 32, 33], "sklearn": 16, "slightli": 33, "smarter": 37, "smoothi": [], "sneak": 33, "soft": 8, "softmax": 1, "softwar": [25, 30], "solv": [2, 32, 35, 36], "solver": 13, "some": [13, 24, 31, 32, 35, 37], "sourc": [], "specifi": 2, "speed": 33, "sphinx": [], "split": [0, 15, 31], "squar": [0, 5, 6, 10, 16, 19, 25, 30, 31, 32, 33], "standard": [13, 31, 34], "start": [20, 36], "state": 0, "statist": [5, 6, 23, 27, 30, 34, 35], "steepest": [10, 13, 32], "step": [33, 34, 35], "stochast": [13, 25, 27, 33], "stop": 33, "strongli": [30, 33], "structur": [], "studi": 36, "suggest": [30, 36], "sum": [34, 35, 37], "summari": [28, 30], "superposit": 3, "supervis": 1, "support": 8, "svd": [5, 31, 32], "synthet": [18, 35, 36], "systemat": 3, "t": 31, "take": 16, "taken": [30, 33], "teach": 28, "teacher": [28, 30], "team": [], "technic": 31, "techniqu": [6, 11, 25], "technologi": 23, "tensorflow": [1, 3], "tent": [28, 30], "term": [34, 37], "test": [0, 1, 15, 17, 31], "texmath": [], "text": 30, "textbook": [29, 30], "than": 32, "thank": [], "theorem": [5, 8, 11, 12, 27, 34, 37], "theoret": 33, "theori": 27, "theta": [18, 34], "thi": [21, 22, 30, 37], "three": 37, "through": 37, "time": 33, "tip": [13, 33], "todo": [], "togeth": [12, 37], "tool": [25, 30], "top": 1, "topic": 30, "toward": 11, "trade": [6, 19, 25], "tradeoff": [6, 34, 35], "train": [0, 1, 4, 15, 21, 22, 30, 31, 37], "transform": 3, "translat": [], "tree": [9, 10], "tuesdai": [32, 36, 37], "tune": 1, "two": [3, 8, 22, 23, 35, 36, 37], "type": [2, 4, 12, 30, 36, 37], "uio": 30, "understand": [22, 34, 35], "univers": [12, 29, 37], "unsupervis": 14, "up": [0, 2, 9, 12, 15, 30, 31, 32, 34, 35, 37], "updat": [25, 33, 37], "us": [0, 1, 2, 3, 7, 13, 16, 18, 19, 22, 23, 25, 30, 31, 32, 33, 35, 36, 37], "usag": 33, "v": [3, 33], "valid": [6, 25, 34, 35], "valu": [5, 11, 19, 27, 31, 32, 34, 35], "vari": 33, "variabl": [27, 32], "varianc": [6, 19, 25, 34, 35], "variou": [0, 34, 35], "vector": [8, 12, 16, 24, 30, 31, 36], "versu": 30, "video": [33, 34, 35, 36, 37], "view": [0, 4, 10, 31, 37], "virtual": 15, "visual": [1, 9], "wai": [9, 25, 34, 35, 37], "wave": 2, "we": [30, 33, 37], "wednesdai": [32, 36, 37], "week": [15, 16, 17, 18, 19, 20, 21, 22, 30, 31, 32, 33, 34, 35, 36, 37], "weekli": [], "welcom": [], "what": [0, 30, 31, 32, 34, 35], "when": 33, "which": [1, 33], "why": [30, 33, 34, 35, 36, 37], "wisconsin": 7, "word": 37, "workflow": [], "wrap": 34, "write": [4, 11, 20, 22, 25, 32], "x": 31, "xgboost": 10, "xor": 36, "yaml": [], "yet": 32, "your": [0, 10, 16, 18, 25, 31], "z_j": 37}}) \ No newline at end of file +Search.setIndex({"alltitles": {"1a)": [[18, "a"]], "3a)": [[18, "id1"]], "3b)": [[18, "b"]], "4a)": [[18, "id2"]], "4b)": [[18, "id3"]], "A Classification Tree": [[9, "a-classification-tree"]], "A Frequentist approach to data analysis": [[0, "a-frequentist-approach-to-data-analysis"], [31, "a-frequentist-approach-to-data-analysis"]], "A better approach": [[8, "a-better-approach"]], "A first summary": [[31, "a-first-summary"]], "A more compact expression": [[36, "a-more-compact-expression"], [37, "a-more-compact-expression"]], "A new Cost Function": [[35, "a-new-cost-function"]], "A quick Reminder on Lagrangian Multipliers": [[8, "a-quick-reminder-on-lagrangian-multipliers"]], "A simple example": [[4, "a-simple-example"]], "A soft classifier": [[8, "a-soft-classifier"]], "A top-down perspective on Neural networks": [[1, "a-top-down-perspective-on-neural-networks"], [39, "a-top-down-perspective-on-neural-networks"]], "A way to Read the Bias-Variance Tradeoff": [[35, "a-way-to-read-the-bias-variance-tradeoff"], [36, "a-way-to-read-the-bias-variance-tradeoff"]], "ADAM algorithm, taken from Goodfellow et al": [[34, "adam-algorithm-taken-from-goodfellow-et-al"]], "ADAM optimizer": [[13, "adam-optimizer"], [34, "id2"]], "Accuracy": [[34, "accuracy"]], "Activation functions": [[12, "activation-functions"], [37, "activation-functions"], [39, "activation-functions"], [39, "id3"]], "Activation functions, Logistic and Hyperbolic ones": [[37, "activation-functions-logistic-and-hyperbolic-ones"], [39, "activation-functions-logistic-and-hyperbolic-ones"]], "AdaGrad Properties": [[34, "adagrad-properties"]], "AdaGrad Update Rule Derivation": [[34, "adagrad-update-rule-derivation"]], "AdaGrad algorithm, taken from Goodfellow et al": [[34, "adagrad-algorithm-taken-from-goodfellow-et-al"]], "Adam Optimizer": [[34, "adam-optimizer"]], "Adam vs. AdaGrad and RMSProp": [[34, "adam-vs-adagrad-and-rmsprop"]], "Adam: Bias Correction": [[34, "adam-bias-correction"]], "Adam: Exponential Moving Averages (Moments)": [[34, "adam-exponential-moving-averages-moments"]], "Adam: Update Rule Derivation": [[34, "adam-update-rule-derivation"]], "Adaptive boosting: AdaBoost, Basic Algorithm": [[10, "adaptive-boosting-adaboost-basic-algorithm"]], "Adaptivity Across Dimensions": [[34, "adaptivity-across-dimensions"]], "Adding Neural Networks": [[37, "adding-neural-networks"]], "Adding a hidden layer": [[38, "adding-a-hidden-layer"], [39, "adding-a-hidden-layer"]], "Adding error analysis and training set up": [[31, "adding-error-analysis-and-training-set-up"], [32, "adding-error-analysis-and-training-set-up"]], "Adjust hyperparameters": [[1, "adjust-hyperparameters"], [39, "adjust-hyperparameters"]], "Algorithms and codes for Adagrad, RMSprop and Adam": [[34, "algorithms-and-codes-for-adagrad-rmsprop-and-adam"]], "Algorithms for Setting up Decision Trees": [[9, "algorithms-for-setting-up-decision-trees"]], "An Overview of Ensemble Methods": [[10, "an-overview-of-ensemble-methods"]], "An extrapolation example": [[4, "an-extrapolation-example"]], "An optimization/minimization problem": [[31, "an-optimization-minimization-problem"]], "Analyzing the last results": [[38, "analyzing-the-last-results"], [39, "analyzing-the-last-results"]], "And finally \\boldsymbol{X}\\boldsymbol{X}^T": [[32, "and-finally-boldsymbol-x-boldsymbol-x-t"]], "And finally ADAM": [[34, "and-finally-adam"]], "And what about using neural networks?": [[31, "and-what-about-using-neural-networks"]], "Another Example from Scikit-Learn\u2019s Repository": [[35, "another-example-from-scikit-learn-s-repository"], [36, "another-example-from-scikit-learn-s-repository"]], "Another Example, now with a polynomial fit": [[33, "another-example-now-with-a-polynomial-fit"]], "Another example, the moons again": [[9, "another-example-the-moons-again"]], "Applied Data Analysis and Machine Learning": [[23, null]], "Artificial neurons": [[37, "artificial-neurons"], [38, "artificial-neurons"]], "Assumptions made": [[35, "assumptions-made"]], "Autocorrelation function": [[28, "autocorrelation-function"]], "Automatic differentiation": [[13, "automatic-differentiation"], [38, "automatic-differentiation"]], "Automatic differentiation through examples": [[38, "automatic-differentiation-through-examples"]], "Back to Ridge and LASSO Regression": [[32, "back-to-ridge-and-lasso-regression"], [33, "back-to-ridge-and-lasso-regression"]], "Back to the Cancer Data": [[11, "back-to-the-cancer-data"]], "Background literature": [[25, "background-literature"], [26, "background-literature"]], "Bagging": [[10, "bagging"]], "Bagging Examples": [[10, "bagging-examples"]], "Basic Matrix Features": [[24, "basic-matrix-features"]], "Basic ideas of the Principal Component Analysis (PCA)": [[11, null]], "Basic math of the SVD": [[5, "basic-math-of-the-svd"], [32, "basic-math-of-the-svd"], [33, "basic-math-of-the-svd"]], "Basics": [[7, "basics"], [36, "basics"], [37, "basics"]], "Basics of a tree": [[9, "basics-of-a-tree"]], "Basics of an NN": [[38, "basics-of-an-nn"]], "Batch Normalization": [[1, "batch-normalization"], [39, "batch-normalization"]], "Batches and mini-batches": [[34, "batches-and-mini-batches"]], "Bayes\u2019 Theorem and Ridge and Lasso Regression": [[5, "bayes-theorem-and-ridge-and-lasso-regression"]], "Boosting, a Bird\u2019s Eye View": [[10, "boosting-a-bird-s-eye-view"]], "Bootstrap": [[6, "bootstrap"]], "Bringing it together": [[38, "bringing-it-together"], [39, "bringing-it-together"]], "Bringing it together, first back propagation equation": [[12, "bringing-it-together-first-back-propagation-equation"]], "Building a Feed Forward Neural Network": [[1, null]], "Building a neural network code": [[39, "building-a-neural-network-code"]], "Building a tree, regression": [[9, "building-a-tree-regression"]], "Building neural networks in Tensorflow and Keras": [[1, "building-neural-networks-in-tensorflow-and-keras"], [39, "building-neural-networks-in-tensorflow-and-keras"]], "But none of these can compete with Newton\u2019s method": [[34, "but-none-of-these-can-compete-with-newton-s-method"]], "CNNs in more detail, building convolutional neural networks in Tensorflow and Keras": [[3, "cnns-in-more-detail-building-convolutional-neural-networks-in-tensorflow-and-keras"]], "Cancer Data again now with Decision Trees and other Methods": [[9, "cancer-data-again-now-with-decision-trees-and-other-methods"]], "Chain rule": [[38, "chain-rule"]], "Chain rule, forward and reverse modes": [[38, "chain-rule-forward-and-reverse-modes"]], "Challenge: Choosing a Fixed Learning Rate": [[34, "challenge-choosing-a-fixed-learning-rate"]], "Choose cost function and optimizer": [[1, "choose-cost-function-and-optimizer"], [39, "choose-cost-function-and-optimizer"]], "Class of functions we can approximate": [[38, "class-of-functions-we-can-approximate"]], "Classical PCA Theorem": [[11, "classical-pca-theorem"]], "Classification and Regression, writing our own neural network code": [[26, "classification-and-regression-writing-our-own-neural-network-code"]], "Classification problems": [[36, "classification-problems"], [37, "classification-problems"]], "Clustering and Unsupervised Learning": [[14, null]], "Code Example for Cross-validation and k-fold Cross-validation": [[35, "code-example-for-cross-validation-and-k-fold-cross-validation"], [36, "code-example-for-cross-validation-and-k-fold-cross-validation"]], "Code example": [[38, "code-example"], [39, "code-example"]], "Code example for the Bootstrap method": [[35, "code-example-for-the-bootstrap-method"]], "Code for SVD and Inversion of Matrices": [[5, "code-for-svd-and-inversion-of-matrices"]], "Code with a Number of Minibatches which varies": [[34, "code-with-a-number-of-minibatches-which-varies"]], "Codes and Approaches": [[14, "codes-and-approaches"]], "Codes for the SVD": [[5, "codes-for-the-svd"], [32, "codes-for-the-svd"], [33, "codes-for-the-svd"]], "Coding Setup and Linear Regression": [[15, "coding-setup-and-linear-regression"]], "Collect and pre-process data": [[1, "collect-and-pre-process-data"], [39, "collect-and-pre-process-data"], [39, "id2"]], "Communication channels": [[31, "communication-channels"]], "Compact expressions": [[38, "compact-expressions"], [39, "compact-expressions"]], "Compare Bagging on Trees with Random Forests": [[10, "compare-bagging-on-trees-with-random-forests"]], "Comparing with a numerical scheme": [[2, "comparing-with-a-numerical-scheme"]], "Comparison with OLS": [[33, "comparison-with-ols"]], "Completing the list": [[38, "completing-the-list"], [39, "completing-the-list"]], "Computation of gradients": [[34, "computation-of-gradients"]], "Computing the Gini index": [[9, "computing-the-gini-index"]], "Conditions on convex functions": [[33, "conditions-on-convex-functions"]], "Confidence Intervals": [[35, "confidence-intervals"]], "Conjugate gradient method": [[13, "conjugate-gradient-method"]], "Convergence rates": [[34, "convergence-rates"]], "Convex function": [[33, "convex-function"]], "Convex functions": [[13, "convex-functions"], [33, "convex-functions"]], "Convolution Examples: Polynomial multiplication": [[3, "convolution-examples-polynomial-multiplication"]], "Convolution Examples: Principle of Superposition and Periodic Forces (Fourier Transforms)": [[3, "convolution-examples-principle-of-superposition-and-periodic-forces-fourier-transforms"]], "Convolutional Neural Network": [[12, "convolutional-neural-network"], [37, "convolutional-neural-network"], [38, "convolutional-neural-network"]], "Convolutional Neural Networks": [[3, null]], "Correlation Function and Design/Feature Matrix": [[32, "correlation-function-and-design-feature-matrix"]], "Correlation Matrix": [[11, "correlation-matrix"], [32, "correlation-matrix"]], "Correlation Matrix with Pandas": [[32, "correlation-matrix-with-pandas"]], "Cost functions": [[39, "cost-functions"]], "Counting the number of floating point operations": [[38, "counting-the-number-of-floating-point-operations"]], "Course Format": [[31, "course-format"]], "Course setting": [[27, null]], "Covariance Matrix Examples": [[32, "covariance-matrix-examples"]], "Covariance and Correlation Matrix": [[32, "covariance-and-correlation-matrix"]], "Cross-validation": [[6, "cross-validation"]], "Cross-validation in brief": [[35, "cross-validation-in-brief"], [36, "cross-validation-in-brief"]], "Deadlines for projects (tentative)": [[31, "deadlines-for-projects-tentative"]], "Decision trees, overarching aims": [[9, null]], "Deep Neural Networks": [[34, "deep-neural-networks"]], "Deep learning methods": [[31, "deep-learning-methods"]], "Define model and architecture": [[1, "define-model-and-architecture"], [39, "define-model-and-architecture"]], "Defining intermediate operations": [[38, "defining-intermediate-operations"]], "Defining the cost function": [[1, "defining-the-cost-function"], [39, "defining-the-cost-function"]], "Definitions": [[19, "definitions"], [38, "definitions"], [39, "definitions"]], "Deliverables": [[15, "deliverables"], [16, "deliverables"], [19, "deliverables"], [20, "deliverables"], [25, "deliverables"], [26, "deliverables"]], "Derivation of the AdaGrad Algorithm": [[34, "derivation-of-the-adagrad-algorithm"]], "Derivative of the cost function": [[38, "derivative-of-the-cost-function"], [39, "derivative-of-the-cost-function"]], "Derivatives and the chain rule": [[12, "derivatives-and-the-chain-rule"], [38, "derivatives-and-the-chain-rule"], [39, "derivatives-and-the-chain-rule"]], "Derivatives in terms of z_j^L": [[38, "derivatives-in-terms-of-z-j-l"], [39, "derivatives-in-terms-of-z-j-l"]], "Derivatives of the hidden layer": [[38, "derivatives-of-the-hidden-layer"], [39, "derivatives-of-the-hidden-layer"]], "Derivatives, example 1": [[32, "derivatives-example-1"]], "Deriving OLS from a probability distribution": [[5, "deriving-ols-from-a-probability-distribution"], [35, "deriving-ols-from-a-probability-distribution"]], "Deriving and Implementing Ordinary Least Squares": [[16, "deriving-and-implementing-ordinary-least-squares"]], "Deriving and Implementing Ridge Regression": [[17, "deriving-and-implementing-ridge-regression"]], "Deriving the Lasso Regression Equations": [[32, "deriving-the-lasso-regression-equations"], [33, "deriving-the-lasso-regression-equations"], [33, "id6"]], "Deriving the Ridge Regression Equations": [[32, "deriving-the-ridge-regression-equations"], [33, "deriving-the-ridge-regression-equations"], [33, "id3"]], "Deriving the back propagation code for a multilayer perceptron model": [[12, "deriving-the-back-propagation-code-for-a-multilayer-perceptron-model"]], "Developing a code for doing neural networks with back propagation": [[1, "developing-a-code-for-doing-neural-networks-with-back-propagation"], [39, "developing-a-code-for-doing-neural-networks-with-back-propagation"]], "Diagonalize the sample covariance matrix to obtain the principal components": [[11, "diagonalize-the-sample-covariance-matrix-to-obtain-the-principal-components"]], "Different kernels and Mercer\u2019s theorem": [[8, "different-kernels-and-mercer-s-theorem"]], "Disadvantages": [[9, "disadvantages"]], "Discriminative Modeling": [[31, "discriminative-modeling"]], "Discussing the correlation data": [[37, "discussing-the-correlation-data"]], "Does Logistic Regression do a better Job?": [[37, "does-logistic-regression-do-a-better-job"]], "Domains and probabilities": [[28, "domains-and-probabilities"]], "Dropout": [[1, "dropout"], [39, "dropout"]], "ELU function": [[39, "elu-function"]], "Economy-size SVD": [[32, "economy-size-svd"], [33, "economy-size-svd"]], "Elements of Probability Theory and Statistical Data Analysis": [[28, null]], "Empirical Evidence: Convergence Time and Memory in Practice": [[34, "empirical-evidence-convergence-time-and-memory-in-practice"]], "Ensemble Methods: From a Single Tree to Many Trees and Extreme Boosting, Meet the Jungle of Methods": [[10, null]], "Entropy and the ID3 algorithm": [[9, "entropy-and-the-id3-algorithm"]], "Essential elements of ML": [[31, "essential-elements-of-ml"]], "Evaluate model performance on test data": [[1, "evaluate-model-performance-on-test-data"], [39, "evaluate-model-performance-on-test-data"]], "Example 2": [[32, "example-2"]], "Example 3": [[32, "example-3"]], "Example 4": [[32, "example-4"]], "Example Matrix": [[32, "example-matrix"], [33, "example-matrix"]], "Example code for Bias-Variance tradeoff": [[35, "example-code-for-bias-variance-tradeoff"]], "Example code for Logistic Regression": [[36, "example-code-for-logistic-regression"], [37, "example-code-for-logistic-regression"]], "Example of discriminative modeling, taken from Generative Deep Learning by David Foster": [[31, "example-of-discriminative-modeling-taken-from-generative-deep-learning-by-david-foster"]], "Example of generative modeling, taken from Generative Deep Learning by David Foster": [[31, "example-of-generative-modeling-taken-from-generative-deep-learning-by-david-foster"]], "Example of own Standard scaling": [[32, "example-of-own-standard-scaling"]], "Example relevant for the exercises": [[32, "example-relevant-for-the-exercises"]], "Example: Exponential decay": [[2, "example-exponential-decay"]], "Example: Population growth": [[2, "example-population-growth"]], "Example: The diffusion equation": [[2, "example-the-diffusion-equation"]], "Example: binary classification problem": [[1, "example-binary-classification-problem"], [39, "example-binary-classification-problem"]], "Examples": [[31, "examples"]], "Examples of XOR, OR and AND gates": [[37, "examples-of-xor-or-and-and-gates"]], "Examples of likelihood functions used in logistic regression and neural networks": [[7, "examples-of-likelihood-functions-used-in-logistic-regression-and-neural-networks"]], "Examples of likelihood functions used in logistic regression and nueral networks": [[36, "examples-of-likelihood-functions-used-in-logistic-regression-and-nueral-networks"]], "Exercise 1": [[21, "exercise-1"]], "Exercise 1 - Choice of model and degrees of freedom": [[17, "exercise-1-choice-of-model-and-degrees-of-freedom"]], "Exercise 1 - Finding the derivative of Matrix-Vector expressions": [[16, "exercise-1-finding-the-derivative-of-matrix-vector-expressions"]], "Exercise 1 - Github Setup": [[15, "exercise-1-github-setup"]], "Exercise 1 - Understand the feed forward pass": [[22, "exercise-1-understand-the-feed-forward-pass"]], "Exercise 1, scale your data": [[18, "exercise-1-scale-your-data"]], "Exercise 1: Creating the report document": [[20, "exercise-1-creating-the-report-document"]], "Exercise 1: Expectation values for ordinary least squares expressions": [[19, "exercise-1-expectation-values-for-ordinary-least-squares-expressions"]], "Exercise 1: Including more data": [[38, "exercise-1-including-more-data"]], "Exercise 1: Setting up various Python environments": [[0, "exercise-1-setting-up-various-python-environments"]], "Exercise 2": [[21, "exercise-2"]], "Exercise 2 - Deriving the expression for OLS": [[16, "exercise-2-deriving-the-expression-for-ols"]], "Exercise 2 - Deriving the expression for Ridge Regression": [[17, "exercise-2-deriving-the-expression-for-ridge-regression"]], "Exercise 2 - Gradient with one layer using autograd": [[22, "exercise-2-gradient-with-one-layer-using-autograd"]], "Exercise 2 - Setting up a Github repository": [[15, "exercise-2-setting-up-a-github-repository"]], "Exercise 2, calculate the gradients": [[18, "exercise-2-calculate-the-gradients"]], "Exercise 2: Adding good figures": [[20, "exercise-2-adding-good-figures"]], "Exercise 2: Expectation values for Ridge regression": [[19, "exercise-2-expectation-values-for-ridge-regression"]], "Exercise 2: Extended program": [[38, "exercise-2-extended-program"]], "Exercise 2: making your own data and exploring scikit-learn": [[0, "exercise-2-making-your-own-data-and-exploring-scikit-learn"]], "Exercise 3": [[21, "exercise-3"]], "Exercise 3 - Creating feature matrix and implementing OLS using the analytical expression": [[16, "exercise-3-creating-feature-matrix-and-implementing-ols-using-the-analytical-expression"]], "Exercise 3 - Fitting an OLS model to data": [[15, "exercise-3-fitting-an-ols-model-to-data"]], "Exercise 3 - Gradient with one layer writing backpropagation by hand": [[22, "exercise-3-gradient-with-one-layer-writing-backpropagation-by-hand"]], "Exercise 3 - Scaling data": [[17, "exercise-3-scaling-data"]], "Exercise 3 - Setting up a Python virtual environment": [[15, "exercise-3-setting-up-a-python-virtual-environment"]], "Exercise 3, using the analytical formulae for OLS and Ridge regression to find the optimal paramters \\boldsymbol{\\theta}": [[18, "exercise-3-using-the-analytical-formulae-for-ols-and-ridge-regression-to-find-the-optimal-paramters-boldsymbol-theta"]], "Exercise 3: Deriving the expression for the Bias-Variance Trade-off": [[19, "exercise-3-deriving-the-expression-for-the-bias-variance-trade-off"]], "Exercise 3: Normalizing our data": [[0, "exercise-3-normalizing-our-data"]], "Exercise 3: Writing an abstract and introduction": [[20, "exercise-3-writing-an-abstract-and-introduction"]], "Exercise 4 - Custom activation for each layer": [[21, "exercise-4-custom-activation-for-each-layer"]], "Exercise 4 - Fitting a polynomial": [[16, "exercise-4-fitting-a-polynomial"]], "Exercise 4 - Gradient with two layers writing backpropagation by hand": [[22, "exercise-4-gradient-with-two-layers-writing-backpropagation-by-hand"]], "Exercise 4 - Implementing Ridge Regression": [[17, "exercise-4-implementing-ridge-regression"]], "Exercise 4 - Testing multiple hyperparameters": [[17, "exercise-4-testing-multiple-hyperparameters"]], "Exercise 4 - The train-test split": [[15, "exercise-4-the-train-test-split"]], "Exercise 4, Implementing the simplest form for gradient descent": [[18, "exercise-4-implementing-the-simplest-form-for-gradient-descent"]], "Exercise 4: Adding Ridge Regression": [[0, "exercise-4-adding-ridge-regression"]], "Exercise 4: Computing the Bias and Variance": [[19, "exercise-4-computing-the-bias-and-variance"]], "Exercise 4: Making the code available and presentable": [[20, "exercise-4-making-the-code-available-and-presentable"]], "Exercise 5 - Comparing your code with sklearn": [[16, "exercise-5-comparing-your-code-with-sklearn"]], "Exercise 5 - Gradient with any number of layers writing backpropagation by hand": [[22, "exercise-5-gradient-with-any-number-of-layers-writing-backpropagation-by-hand"]], "Exercise 5 - Processing multiple inputs at once": [[21, "exercise-5-processing-multiple-inputs-at-once"]], "Exercise 5, Ridge regression and a new Synthetic Dataset": [[18, "exercise-5-ridge-regression-and-a-new-synthetic-dataset"]], "Exercise 5: Analytical exercises": [[0, "exercise-5-analytical-exercises"]], "Exercise 5: Interpretation of scaling and metrics": [[19, "exercise-5-interpretation-of-scaling-and-metrics"]], "Exercise 5: Referencing": [[20, "exercise-5-referencing"]], "Exercise 6 - Batched inputs": [[22, "exercise-6-batched-inputs"]], "Exercise 6 - Predicting on real data": [[21, "exercise-6-predicting-on-real-data"]], "Exercise 7 - Training": [[22, "exercise-7-training"]], "Exercise 7 - Training on real data (Optional)": [[21, "exercise-7-training-on-real-data-optional"]], "Exercise 8 (Optional) - Object orientation": [[22, "exercise-8-optional-object-orientation"]], "Exercise: Cross-validation as resampling techniques, adding more complexity": [[6, "exercise-cross-validation-as-resampling-techniques-adding-more-complexity"]], "Exercise: Analysis of real data": [[6, "exercise-analysis-of-real-data"]], "Exercise: Bias-variance trade-off and resampling techniques": [[6, "exercise-bias-variance-trade-off-and-resampling-techniques"]], "Exercise: Lasso Regression on the Franke function with resampling": [[6, "exercise-lasso-regression-on-the-franke-function-with-resampling"]], "Exercise: Ordinary Least Square (OLS) on the Franke function": [[6, "exercise-ordinary-least-square-ols-on-the-franke-function"]], "Exercise: Ridge Regression on the Franke function with resampling": [[6, "exercise-ridge-regression-on-the-franke-function-with-resampling"]], "Exercises": [[0, "exercises"]], "Exercises and Projects": [[6, "exercises-and-projects"]], "Exercises week 34": [[15, null]], "Exercises week 35": [[16, null]], "Exercises week 36": [[17, null]], "Exercises week 37": [[18, null]], "Exercises week 38": [[19, null]], "Exercises week 39": [[20, null]], "Exercises week 41": [[21, null]], "Exercises week 42": [[22, null]], "Expectation value and variance": [[35, "expectation-value-and-variance"]], "Expectation value and variance for \\boldsymbol{\\theta}": [[35, "expectation-value-and-variance-for-boldsymbol-theta"]], "Expectation values": [[28, "expectation-values"]], "Explicit derivatives": [[38, "explicit-derivatives"], [39, "explicit-derivatives"]], "Exploding gradients": [[39, "exploding-gradients"]], "Extending to more predictors": [[36, "extending-to-more-predictors"], [37, "extending-to-more-predictors"]], "Extending to more than one variable": [[33, "extending-to-more-than-one-variable"]], "Extremely useful tools, strongly recommended": [[31, "extremely-useful-tools-strongly-recommended"]], "Feed-forward neural networks": [[12, "feed-forward-neural-networks"], [37, "feed-forward-neural-networks"], [38, "feed-forward-neural-networks"]], "Feed-forward pass": [[1, "feed-forward-pass"], [39, "feed-forward-pass"]], "Final back propagating equation": [[12, "final-back-propagating-equation"], [38, "final-back-propagating-equation"], [39, "final-back-propagating-equation"]], "Final derivatives": [[38, "final-derivatives"]], "Final expression": [[38, "final-expression"], [39, "final-expression"]], "Final expressions for the biases of the hidden layer": [[38, "final-expressions-for-the-biases-of-the-hidden-layer"], [39, "final-expressions-for-the-biases-of-the-hidden-layer"]], "Finding the Limit": [[35, "finding-the-limit"]], "Fine-tuning neural network hyperparameters": [[1, "fine-tuning-neural-network-hyperparameters"], [39, "fine-tuning-neural-network-hyperparameters"]], "First network example, simple percepetron with one input": [[38, "first-network-example-simple-percepetron-with-one-input"]], "Fitting an Equation of State for Dense Nuclear Matter": [[0, "fitting-an-equation-of-state-for-dense-nuclear-matter"]], "Fixing the singularity": [[32, "fixing-the-singularity"], [33, "fixing-the-singularity"]], "Format for electronic delivery of report and programs": [[25, "format-for-electronic-delivery-of-report-and-programs"], [26, "format-for-electronic-delivery-of-report-and-programs"]], "Forward and reverse modes": [[38, "forward-and-reverse-modes"]], "Frequently used scaling functions": [[32, "frequently-used-scaling-functions"], [34, "frequently-used-scaling-functions"]], "From OLS to Ridge and Lasso": [[33, "from-ols-to-ridge-and-lasso"]], "From one to many layers, the universal approximation theorem": [[12, "from-one-to-many-layers-the-universal-approximation-theorem"]], "Full object-oriented implementation": [[39, "full-object-oriented-implementation"]], "Functionality in Scikit-Learn": [[32, "functionality-in-scikit-learn"], [34, "functionality-in-scikit-learn"]], "Further Dimensionality Remarks": [[3, "further-dimensionality-remarks"]], "Further properties (important for our analyses later)": [[5, "further-properties-important-for-our-analyses-later"], [32, "further-properties-important-for-our-analyses-later"], [33, "further-properties-important-for-our-analyses-later"]], "Gaussian Elimination": [[24, "gaussian-elimination"]], "General Features": [[9, "general-features"]], "General linear models and linear algebra": [[31, "general-linear-models-and-linear-algebra"]], "Generalizing the fitting procedure as a linear algebra problem": [[31, "generalizing-the-fitting-procedure-as-a-linear-algebra-problem"], [31, "id1"]], "Generative Adversarial Networks": [[4, "generative-adversarial-networks"]], "Generative Models": [[4, "generative-models"]], "Generative Versus Discriminative Modeling": [[31, "generative-versus-discriminative-modeling"]], "Geometric Interpretation and link with Singular Value Decomposition": [[11, "geometric-interpretation-and-link-with-singular-value-decomposition"]], "Getting serious, the back propagation equations for a neural network": [[38, "getting-serious-the-back-propagation-equations-for-a-neural-network"]], "Getting started with project 1": [[20, "getting-started-with-project-1"]], "Gradient Boosting, Classification Example": [[10, "gradient-boosting-classification-example"]], "Gradient Boosting, Examples of Regression": [[10, "gradient-boosting-examples-of-regression"]], "Gradient Clipping": [[1, "gradient-clipping"], [39, "gradient-clipping"]], "Gradient Descent Example": [[33, "id1"], [34, "id1"]], "Gradient boosting: Basics with Steepest Descent/Functional Gradient Descent": [[10, "gradient-boosting-basics-with-steepest-descent-functional-gradient-descent"]], "Gradient descent": [[2, "gradient-descent"]], "Gradient descent and Ridge": [[33, "gradient-descent-and-ridge"], [34, "gradient-descent-and-ridge"]], "Gradient descent and revisiting Ordinary Least Squares from last week": [[34, "gradient-descent-and-revisiting-ordinary-least-squares-from-last-week"]], "Gradient descent example": [[33, "gradient-descent-example"], [34, "gradient-descent-example"]], "Gradient expressions": [[38, "gradient-expressions"], [39, "gradient-expressions"]], "Grading": [[29, "grading"], [29, "id2"], [31, "grading"]], "Hidden layers": [[39, "hidden-layers"]], "Homogeneous data": [[39, "homogeneous-data"]], "How to take derivatives of Matrix-Vector expressions": [[16, "how-to-take-derivatives-of-matrix-vector-expressions"]], "Hyperplanes and all that": [[8, "hyperplanes-and-all-that"]], "Identifying Terms": [[35, "identifying-terms"]], "Illustration of a single perceptron model and a multi-perceptron model": [[37, "illustration-of-a-single-perceptron-model-and-a-multi-perceptron-model"], [38, "illustration-of-a-single-perceptron-model-and-a-multi-perceptron-model"]], "Important Matrix and vector handling packages": [[24, "important-matrix-and-vector-handling-packages"]], "Important observations": [[38, "important-observations"], [39, "important-observations"]], "Important technicalities: More on Rescaling data": [[32, "important-technicalities-more-on-rescaling-data"]], "Improving gradient descent with momentum": [[34, "improving-gradient-descent-with-momentum"]], "Improving performance": [[1, "improving-performance"], [39, "improving-performance"]], "In general not this simple": [[38, "in-general-not-this-simple"]], "In summary": [[29, "in-summary"]], "Including Stochastic Gradient Descent with Autograd": [[13, "including-stochastic-gradient-descent-with-autograd"], [34, "including-stochastic-gradient-descent-with-autograd"]], "Including more classes": [[36, "including-more-classes"], [37, "including-more-classes"]], "Incremental PCA": [[11, "incremental-pca"]], "Independent and Identically Distributed (iid)": [[35, "independent-and-identically-distributed-iid"]], "Inputs to the activation function": [[38, "inputs-to-the-activation-function"], [39, "inputs-to-the-activation-function"]], "Insights from the paper by Glorot and Bengio": [[39, "insights-from-the-paper-by-glorot-and-bengio"]], "Installing R, C++, cython or Julia": [[31, "installing-r-c-cython-or-julia"]], "Installing R, C++, cython, Numba etc": [[31, "installing-r-c-cython-numba-etc"]], "Instructor information": [[29, "instructor-information"]], "Interpretations and optimizing our parameters": [[31, "interpretations-and-optimizing-our-parameters"], [31, "id2"], [31, "id3"], [32, "interpretations-and-optimizing-our-parameters"], [32, "id1"], [32, "id2"]], "Interpreting the Ridge results": [[32, "interpreting-the-ridge-results"], [33, "interpreting-the-ridge-results"], [33, "id4"]], "Introducing JAX": [[13, "introducing-jax"]], "Introducing the Covariance and Correlation functions": [[11, "introducing-the-covariance-and-correlation-functions"], [32, "introducing-the-covariance-and-correlation-functions"]], "Introduction": [[0, "introduction"], [6, "introduction"], [23, "introduction"], [24, "introduction"]], "Introduction to Neural networks": [[37, "introduction-to-neural-networks"], [38, "introduction-to-neural-networks"]], "Introduction to numerical projects": [[25, "introduction-to-numerical-projects"], [26, "introduction-to-numerical-projects"]], "Is the Logistic activation function (Sigmoid) our choice?": [[39, "is-the-logistic-activation-function-sigmoid-our-choice"]], "Iterative Fitting, Classification and AdaBoost": [[10, "iterative-fitting-classification-and-adaboost"]], "Iterative Fitting, Regression and Squared-error Cost Function": [[10, "iterative-fitting-regression-and-squared-error-cost-function"]], "Kernel PCA": [[11, "kernel-pca"]], "Kernels and non-linearity": [[8, "kernels-and-non-linearity"]], "LU Decomposition, the inverse of a matrix": [[24, "lu-decomposition-the-inverse-of-a-matrix"]], "Lab sessions Tuesday and Wednesday": [[37, "lab-sessions-tuesday-and-wednesday"]], "Lab sessions on Tuesday and Wednesday": [[38, "lab-sessions-on-tuesday-and-wednesday"]], "Lab sessions week 39": [[36, "lab-sessions-week-39"]], "Lasso Regression": [[33, "lasso-regression"]], "Lasso case": [[33, "lasso-case"]], "Layers": [[1, "layers"], [39, "layers"]], "Layers used to build CNNs": [[3, "layers-used-to-build-cnns"]], "Layout of a neural network with three hidden layers": [[38, "layout-of-a-neural-network-with-three-hidden-layers"]], "Layout of a neural network with three hidden layers (last layer = l=L=4, first layer l=0)": [[39, "layout-of-a-neural-network-with-three-hidden-layers-last-layer-l-l-4-first-layer-l-0"]], "Layout of a simple neural network with no hidden layer": [[38, "layout-of-a-simple-neural-network-with-no-hidden-layer"], [39, "layout-of-a-simple-neural-network-with-no-hidden-layer"]], "Layout of a simple neural network with one hidden layer": [[38, "layout-of-a-simple-neural-network-with-one-hidden-layer"], [39, "layout-of-a-simple-neural-network-with-one-hidden-layer"]], "Layout of a simple neural network with two input nodes, one hidden layer and one output node": [[38, "layout-of-a-simple-neural-network-with-two-input-nodes-one-hidden-layer-and-one-output-node"]], "Layout of a simple neural network with two input nodes, one hidden layer with two hidden noeds and one output node": [[39, "layout-of-a-simple-neural-network-with-two-input-nodes-one-hidden-layer-with-two-hidden-noeds-and-one-output-node"]], "Layout of input to first hidden layer l=1 from input layer l=0": [[39, "layout-of-input-to-first-hidden-layer-l-1-from-input-layer-l-0"]], "Learning goals": [[15, "learning-goals"], [16, "learning-goals"], [17, "learning-goals"], [18, "learning-goals"], [19, "learning-goals"], [20, "learning-goals"]], "Learning outcomes": [[23, "learning-outcomes"], [31, "learning-outcomes"]], "Learning rate methods": [[39, "learning-rate-methods"]], "Lecture Monday October 6": [[38, "lecture-monday-october-6"]], "Lecture Monday September 29, 2025": [[37, "lecture-monday-september-29-2025"]], "Lecture October 13, 2025": [[39, "lecture-october-13-2025"]], "Lecture material": [[36, "lecture-material"]], "Lecture material: Writing a code which implements a feed-forward neural network": [[39, "lecture-material-writing-a-code-which-implements-a-feed-forward-neural-network"]], "Lectures and ComputerLab": [[31, "lectures-and-computerlab"]], "Limitations of NNs": [[39, "limitations-of-nns"]], "Limitations of supervised learning with deep networks": [[1, "limitations-of-supervised-learning-with-deep-networks"], [39, "limitations-of-supervised-learning-with-deep-networks"]], "Linear Algebra, Handling of Arrays and more Python Features": [[24, null]], "Linear Regression": [[0, null]], "Linear Regression Problems": [[32, "linear-regression-problems"], [33, "linear-regression-problems"]], "Linear Regression and the SVD": [[33, "linear-regression-and-the-svd"]], "Linear Regression, basic elements": [[0, "linear-regression-basic-elements"]], "Linear classifier": [[36, "linear-classifier"]], "Linking Bayes\u2019 Theorem with Ridge and Lasso Regression": [[5, "linking-bayes-theorem-with-ridge-and-lasso-regression"]], "Linking the regression analysis with a statistical interpretation": [[5, "linking-the-regression-analysis-with-a-statistical-interpretation"], [35, "linking-the-regression-analysis-with-a-statistical-interpretation"]], "Linking with the SVD": [[5, "linking-with-the-svd"], [32, "linking-with-the-svd"]], "Links to relevant courses at the University of Oslo": [[30, "links-to-relevant-courses-at-the-university-of-oslo"]], "Logistic Regression": [[7, null], [7, "id1"], [36, "logistic-regression"]], "Logistic Regression, from last week": [[37, "logistic-regression-from-last-week"]], "Logistic function as the root of problems": [[39, "logistic-function-as-the-root-of-problems"]], "MNIST and GANs": [[4, "mnist-and-gans"]], "Machine Learning": [[31, "machine-learning"]], "Machine learning": [[23, "machine-learning"]], "Main textbooks": [[31, "main-textbooks"]], "Making a tree": [[9, "making-a-tree"]], "Making your own Bootstrap: Changing the Level of the Decision Tree": [[10, "making-your-own-bootstrap-changing-the-level-of-the-decision-tree"]], "Making your own test-train splitting": [[32, "making-your-own-test-train-splitting"]], "Material for exercises week 35": [[32, "material-for-exercises-week-35"]], "Material for lab sessions sessions Tuesday and Wednesday": [[33, "material-for-lab-sessions-sessions-tuesday-and-wednesday"]], "Material for lecture Monday September 2": [[33, "material-for-lecture-monday-september-2"]], "Material for lecture Monday September 8": [[34, "material-for-lecture-monday-september-8"]], "Material for the lab sessions": [[34, "material-for-the-lab-sessions"], [35, "material-for-the-lab-sessions"]], "Material for the lab sessions on Tuesday and Wednesday": [[39, "material-for-the-lab-sessions-on-tuesday-and-wednesday"]], "Material for the lecture on Monday October 6, 2025": [[38, "material-for-the-lecture-on-monday-october-6-2025"]], "Mathematical Interpretation of Ordinary Least Squares": [[5, "mathematical-interpretation-of-ordinary-least-squares"], [32, "mathematical-interpretation-of-ordinary-least-squares"], [33, "mathematical-interpretation-of-ordinary-least-squares"]], "Mathematical model": [[37, "mathematical-model"], [37, "id1"], [37, "id2"], [37, "id3"], [37, "id4"]], "Mathematical optimization of convex functions": [[8, "mathematical-optimization-of-convex-functions"]], "Mathematics of CNNs": [[3, "mathematics-of-cnns"]], "Mathematics of deep learning": [[38, "mathematics-of-deep-learning"], [39, "mathematics-of-deep-learning"]], "Mathematics of deep learning and neural networks": [[38, "mathematics-of-deep-learning-and-neural-networks"]], "Mathematics of the SVD and implications": [[5, "mathematics-of-the-svd-and-implications"], [32, "mathematics-of-the-svd-and-implications"], [33, "mathematics-of-the-svd-and-implications"]], "Matrices in Python": [[31, "matrices-in-python"]], "Matrix multiplication": [[1, "matrix-multiplication"], [39, "matrix-multiplication"]], "Matrix multiplications": [[39, "matrix-multiplications"]], "Matrix-vector notation": [[37, "matrix-vector-notation"]], "Matrix-vector notation and activation": [[12, "matrix-vector-notation-and-activation"], [37, "matrix-vector-notation-and-activation"]], "Maximum Likelihood Estimation (MLE)": [[35, "maximum-likelihood-estimation-mle"]], "Maximum likelihood": [[36, "maximum-likelihood"], [37, "maximum-likelihood"]], "Meet the covariance!": [[28, "meet-the-covariance"]], "Meet the Covariance Matrix": [[5, "meet-the-covariance-matrix"], [32, "meet-the-covariance-matrix"]], "Meet the Hessian Matrix": [[32, "meet-the-hessian-matrix"]], "Meet the Pandas": [[31, "meet-the-pandas"]], "Memory Usage and Scalability": [[34, "memory-usage-and-scalability"]], "Memory constraints": [[34, "memory-constraints"]], "Min-Max Scaling": [[32, "min-max-scaling"]], "Minimizing the cross entropy": [[36, "minimizing-the-cross-entropy"], [37, "minimizing-the-cross-entropy"]], "Momentum based GD": [[13, "momentum-based-gd"], [34, "momentum-based-gd"]], "More classes": [[36, "more-classes"], [37, "more-classes"]], "More complicated Example: The Ising model": [[6, "more-complicated-example-the-ising-model"]], "More complicated function": [[38, "more-complicated-function"]], "More considerations": [[38, "more-considerations"], [39, "more-considerations"]], "More examples on bootstrap and cross-validation and errors": [[35, "more-examples-on-bootstrap-and-cross-validation-and-errors"], [36, "more-examples-on-bootstrap-and-cross-validation-and-errors"]], "More interpretations": [[32, "more-interpretations"], [33, "more-interpretations"], [33, "id5"]], "More limitations": [[39, "more-limitations"]], "More on Dimensionalities": [[3, "more-on-dimensionalities"]], "More on Rescaling data": [[6, "more-on-rescaling-data"]], "More on Steepest descent": [[33, "more-on-steepest-descent"]], "More on activation functions, output layers": [[39, "more-on-activation-functions-output-layers"]], "More on convex functions": [[33, "more-on-convex-functions"]], "More on the general approximation theorem": [[38, "more-on-the-general-approximation-theorem"]], "More preprocessing": [[32, "more-preprocessing"], [34, "more-preprocessing"]], "More top-down perspectives": [[39, "more-top-down-perspectives"]], "Motivation for Adaptive Step Sizes": [[34, "motivation-for-adaptive-step-sizes"]], "Multiclass classification": [[39, "multiclass-classification"]], "Multilayer perceptrons": [[12, "multilayer-perceptrons"], [37, "multilayer-perceptrons"], [38, "multilayer-perceptrons"]], "Multivariable functions": [[38, "multivariable-functions"]], "Network requirements": [[2, "network-requirements"]], "Neural Networks vs CNNs": [[3, "neural-networks-vs-cnns"]], "Neural network types": [[37, "neural-network-types"], [38, "neural-network-types"]], "Neural networks": [[12, null]], "New expression for the derivative": [[38, "new-expression-for-the-derivative"]], "Non-Convex Problems": [[34, "non-convex-problems"]], "Note about SVD Calculations": [[32, "note-about-svd-calculations"], [33, "note-about-svd-calculations"]], "Note on Scikit-Learn": [[33, "note-on-scikit-learn"]], "Numerical experiments and the covariance, central limit theorem": [[28, "numerical-experiments-and-the-covariance-central-limit-theorem"]], "Numpy and arrays": [[24, "numpy-and-arrays"], [31, "numpy-and-arrays"]], "Numpy examples and Important Matrix and vector handling packages": [[31, "numpy-examples-and-important-matrix-and-vector-handling-packages"]], "Optimization and Deep learning": [[36, "optimization-and-deep-learning"], [37, "optimization-and-deep-learning"]], "Optimization and gradient descent, the central part of any Machine Learning algortithm": [[33, "optimization-and-gradient-descent-the-central-part-of-any-machine-learning-algortithm"]], "Optimization, the central part of any Machine Learning algortithm": [[13, null], [36, "optimization-the-central-part-of-any-machine-learning-algortithm"], [37, "optimization-the-central-part-of-any-machine-learning-algortithm"]], "Optimizing our parameters": [[31, "optimizing-our-parameters"]], "Optimizing our parameters, more details": [[31, "optimizing-our-parameters-more-details"]], "Optimizing the cost function": [[1, "optimizing-the-cost-function"], [39, "optimizing-the-cost-function"]], "Optimizing the parameters": [[38, "optimizing-the-parameters"], [39, "optimizing-the-parameters"]], "Organizing our data": [[0, "organizing-our-data"], [31, "organizing-our-data"]], "Other Matrix and Vector Operations": [[24, "other-matrix-and-vector-operations"]], "Other Types of Recurrent Neural Networks": [[4, "other-types-of-recurrent-neural-networks"]], "Other courses on Data science and Machine Learning at UiO": [[31, "other-courses-on-data-science-and-machine-learning-at-uio"]], "Other courses on Data science and Machine Learning at UiO, contn": [[31, "other-courses-on-data-science-and-machine-learning-at-uio-contn"]], "Other ingredients of a neural network": [[38, "other-ingredients-of-a-neural-network"]], "Other measures in classification studies": [[37, "other-measures-in-classification-studies"]], "Other parameters": [[38, "other-parameters"]], "Other popular texts": [[31, "other-popular-texts"]], "Other techniques": [[11, "other-techniques"]], "Other types of networks": [[12, "other-types-of-networks"], [37, "other-types-of-networks"], [38, "other-types-of-networks"]], "Other ways of visualizing the trees": [[9, "other-ways-of-visualizing-the-trees"]], "Our model for the nuclear binding energies": [[31, "our-model-for-the-nuclear-binding-energies"]], "Output layer": [[38, "output-layer"], [39, "output-layer"]], "Overarching aims of the exercises this week": [[21, "overarching-aims-of-the-exercises-this-week"], [22, "overarching-aims-of-the-exercises-this-week"]], "Overarching view of a neural network": [[38, "overarching-view-of-a-neural-network"]], "Overview of first week": [[31, "overview-of-first-week"]], "Overview video on Stochastic Gradient Descent (SGD)": [[34, "overview-video-on-stochastic-gradient-descent-sgd"]], "Own code for Ordinary Least Squares": [[31, "own-code-for-ordinary-least-squares"], [32, "own-code-for-ordinary-least-squares"]], "PCA and scikit-learn": [[11, "pca-and-scikit-learn"]], "Pandas AI": [[31, "pandas-ai"]], "Parameters of neural networks": [[38, "parameters-of-neural-networks"]], "Part a : Ordinary Least Square (OLS) for the Runge function": [[25, "part-a-ordinary-least-square-ols-for-the-runge-function"]], "Part a): Analytical warm-up": [[26, "part-a-analytical-warm-up"]], "Part b): Writing your own Neural Network code": [[26, "part-b-writing-your-own-neural-network-code"]], "Part b: Adding Ridge regression for the Runge function": [[25, "part-b-adding-ridge-regression-for-the-runge-function"]], "Part c): Testing against other software libraries": [[26, "part-c-testing-against-other-software-libraries"]], "Part c: Writing your own gradient descent code": [[25, "part-c-writing-your-own-gradient-descent-code"]], "Part d): Testing different activation functions and depths of the neural network": [[26, "part-d-testing-different-activation-functions-and-depths-of-the-neural-network"]], "Part d: Including momentum and more advanced ways to update the learning the rate": [[25, "part-d-including-momentum-and-more-advanced-ways-to-update-the-learning-the-rate"]], "Part e): Testing different norms": [[26, "part-e-testing-different-norms"]], "Part e: Writing our own code for Lasso regression": [[25, "part-e-writing-our-own-code-for-lasso-regression"]], "Part f): Classification analysis using neural networks": [[26, "part-f-classification-analysis-using-neural-networks"]], "Part f: Stochastic gradient descent": [[25, "part-f-stochastic-gradient-descent"]], "Part g) Critical evaluation of the various algorithms": [[26, "part-g-critical-evaluation-of-the-various-algorithms"]], "Part g: Bias-variance trade-off and resampling techniques": [[25, "part-g-bias-variance-trade-off-and-resampling-techniques"]], "Part h): Cross-validation as resampling techniques, adding more complexity": [[25, "part-h-cross-validation-as-resampling-techniques-adding-more-complexity"]], "Partial Differential Equations": [[2, "partial-differential-equations"]], "Plan for week 39, September 22-26, 2025": [[36, "plan-for-week-39-september-22-26-2025"]], "Plan for week 41, October 6-10": [[38, "plan-for-week-41-october-6-10"]], "Plans for week 35": [[32, "plans-for-week-35"]], "Plans for week 36": [[33, "plans-for-week-36"]], "Plans for week 37, lecture Monday": [[34, "plans-for-week-37-lecture-monday"]], "Plans for week 38, lecture Monday September 15": [[35, "plans-for-week-38-lecture-monday-september-15"]], "Plotting the Histogram": [[35, "plotting-the-histogram"]], "Plotting the mean value for each group": [[36, "plotting-the-mean-value-for-each-group"]], "Practical tips": [[13, "practical-tips"], [34, "practical-tips"]], "Practicalities": [[29, "practicalities"], [29, "id1"]], "Preamble: Note on writing reports, using reference material, AI and other tools": [[25, "preamble-note-on-writing-reports-using-reference-material-ai-and-other-tools"], [26, "preamble-note-on-writing-reports-using-reference-material-ai-and-other-tools"]], "Predicting New Points With A Trained Recurrent Neural Network": [[4, "predicting-new-points-with-a-trained-recurrent-neural-network"]], "Preprocessing our data": [[32, "preprocessing-our-data"]], "Prerequisites": [[31, "prerequisites"]], "Prerequisites and background": [[23, "prerequisites-and-background"]], "Prerequisites: Collect and pre-process data": [[3, "prerequisites-collect-and-pre-process-data"]], "Probability Distribution Functions": [[28, "probability-distribution-functions"]], "Program example for gradient descent with Ridge Regression": [[33, "program-example-for-gradient-descent-with-ridge-regression"], [34, "program-example-for-gradient-descent-with-ridge-regression"]], "Program for stochastic gradient": [[13, "program-for-stochastic-gradient"]], "Project 1 on Machine Learning, deadline October 6 (midnight), 2025": [[25, null]], "Project 2 on Machine Learning, deadline November 10 (Midnight)": [[26, null]], "Properties of PDFs": [[28, "properties-of-pdfs"]], "Pros and cons": [[34, "pros-and-cons"]], "Pros and cons of trees, pros": [[9, "pros-and-cons-of-trees-pros"]], "Python installers": [[23, "python-installers"], [31, "python-installers"]], "RMS prop": [[13, "rms-prop"]], "RMSProp algorithm, taken from Goodfellow et al": [[34, "rmsprop-algorithm-taken-from-goodfellow-et-al"]], "RMSProp: Adaptive Learning Rates": [[34, "rmsprop-adaptive-learning-rates"]], "RMSprop for adaptive learning rate with Stochastic Gradient Descent": [[34, "rmsprop-for-adaptive-learning-rate-with-stochastic-gradient-descent"]], "Random Numbers": [[28, "random-numbers"]], "Random forests": [[10, "random-forests"]], "Randomized PCA": [[11, "randomized-pca"]], "Reading material": [[31, "reading-material"]], "Reading recommendations": [[39, "reading-recommendations"]], "Reading recommendations:": [[32, "reading-recommendations"]], "Reading suggestions week 34": [[31, "reading-suggestions-week-34"]], "Readings and Videos": [[35, "readings-and-videos"]], "Readings and Videos, logistic regression": [[36, "readings-and-videos-logistic-regression"]], "Readings and Videos, resampling methods": [[36, "readings-and-videos-resampling-methods"]], "Readings and Videos:": [[34, "readings-and-videos"], [38, "readings-and-videos"]], "Readings and videos": [[39, "readings-and-videos"]], "Recurrent neural networks": [[12, "recurrent-neural-networks"], [37, "recurrent-neural-networks"], [38, "recurrent-neural-networks"]], "Recurrent neural networks: Overarching view": [[4, null]], "Reducing the number of degrees of freedom, overarching view": [[0, "reducing-the-number-of-degrees-of-freedom-overarching-view"], [32, "reducing-the-number-of-degrees-of-freedom-overarching-view"]], "Reducing the number of operations": [[38, "reducing-the-number-of-operations"]], "Reformulating the problem": [[2, "reformulating-the-problem"]], "Regression Case": [[10, "regression-case"]], "Regression analysis and resampling methods": [[25, "regression-analysis-and-resampling-methods"]], "Regression analysis, overarching aims": [[31, "regression-analysis-overarching-aims"]], "Regression analysis, overarching aims II": [[31, "regression-analysis-overarching-aims-ii"]], "Regularization": [[1, "regularization"], [39, "regularization"]], "Relevance": [[37, "relevance"], [39, "relevance"]], "Reminder about the gradient machinery from project 1": [[26, "reminder-about-the-gradient-machinery-from-project-1"]], "Reminder from last week": [[32, "reminder-from-last-week"]], "Reminder from last week: First network example, simple percepetron with one input": [[39, "reminder-from-last-week-first-network-example-simple-percepetron-with-one-input"]], "Reminder on Newton-Raphson\u2019s method": [[33, "reminder-on-newton-raphson-s-method"]], "Reminder on Statistics": [[6, "reminder-on-statistics"]], "Reminder on books with hands-on material and codes": [[38, "reminder-on-books-with-hands-on-material-and-codes"], [39, "reminder-on-books-with-hands-on-material-and-codes"]], "Reminder on different scaling methods": [[34, "reminder-on-different-scaling-methods"]], "Reminder on the chain rule and gradients": [[38, "reminder-on-the-chain-rule-and-gradients"]], "Replace or not": [[13, "replace-or-not"], [34, "replace-or-not"]], "Required Technologies": [[23, "required-technologies"]], "Resampling Methods": [[6, null]], "Resampling and the Bias-Variance Trade-off": [[19, "resampling-and-the-bias-variance-trade-off"]], "Resampling approaches can be computationally expensive": [[35, "resampling-approaches-can-be-computationally-expensive"], [36, "resampling-approaches-can-be-computationally-expensive"]], "Resampling methods": [[6, "id1"], [35, "resampling-methods"], [35, "id2"], [36, "resampling-methods"], [36, "id1"]], "Resampling methods: Bootstrap": [[35, "resampling-methods-bootstrap"], [36, "resampling-methods-bootstrap"]], "Resampling methods: Bootstrap approach": [[35, "resampling-methods-bootstrap-approach"]], "Resampling methods: Bootstrap background": [[35, "resampling-methods-bootstrap-background"]], "Resampling methods: Bootstrap steps": [[35, "resampling-methods-bootstrap-steps"]], "Resampling methods: More Bootstrap background": [[35, "resampling-methods-more-bootstrap-background"]], "Residual Error": [[32, "residual-error"], [33, "residual-error"]], "Resources on differential equations and deep learning": [[2, "resources-on-differential-equations-and-deep-learning"]], "Revisiting Ordinary Least Squares": [[33, "revisiting-ordinary-least-squares"]], "Revisiting our Linear Regression Solvers": [[13, "revisiting-our-linear-regression-solvers"]], "Revisiting our Logistic Regression case": [[36, "revisiting-our-logistic-regression-case"], [37, "revisiting-our-logistic-regression-case"]], "Rewriting the Covariance and/or Correlation Matrix": [[32, "rewriting-the-covariance-and-or-correlation-matrix"]], "Rewriting the \\delta-function": [[35, "rewriting-the-delta-function"]], "Rewriting the fitting procedure as a linear algebra problem": [[31, "rewriting-the-fitting-procedure-as-a-linear-algebra-problem"]], "Rewriting the fitting procedure as a linear algebra problem, more details": [[31, "rewriting-the-fitting-procedure-as-a-linear-algebra-problem-more-details"]], "Ridge Regression": [[33, "ridge-regression"]], "Ridge and LASSO Regression": [[32, "ridge-and-lasso-regression"], [33, "ridge-and-lasso-regression"], [33, "id2"]], "Ridge and Lasso Regression": [[5, null], [5, "id1"]], "SGD example": [[34, "sgd-example"]], "SGD vs Full-Batch GD: Convergence Speed and Memory Comparison": [[34, "sgd-vs-full-batch-gd-convergence-speed-and-memory-comparison"]], "SVD analysis": [[33, "svd-analysis"]], "Same code but now with momentum gradient descent": [[13, "same-code-but-now-with-momentum-gradient-descent"], [34, "same-code-but-now-with-momentum-gradient-descent"], [34, "id3"], [34, "id4"]], "Schedule first week": [[31, "schedule-first-week"]], "Schematic Regression Procedure": [[9, "schematic-regression-procedure"]], "Second moment of the gradient": [[34, "second-moment-of-the-gradient"]], "September 15-19": [[19, "september-15-19"]], "Setting up a Multi-layer perceptron model for classification": [[39, "setting-up-a-multi-layer-perceptron-model-for-classification"]], "Setting up the Back propagation algorithm": [[12, "setting-up-the-back-propagation-algorithm"]], "Setting up the Back propagation algorithm, part 3": [[38, "setting-up-the-back-propagation-algorithm-part-3"], [39, "setting-up-the-back-propagation-algorithm-part-3"]], "Setting up the Matrix to be inverted": [[32, "setting-up-the-matrix-to-be-inverted"], [33, "setting-up-the-matrix-to-be-inverted"]], "Setting up the back propagation algorithm": [[38, "setting-up-the-back-propagation-algorithm"]], "Setting up the back propagation algorithm and algorithm for a feed forward NN, initalizations": [[39, "setting-up-the-back-propagation-algorithm-and-algorithm-for-a-feed-forward-nn-initalizations"]], "Setting up the back propagation algorithm, part 1": [[39, "setting-up-the-back-propagation-algorithm-part-1"]], "Setting up the back propagation algorithm, part 2": [[38, "setting-up-the-back-propagation-algorithm-part-2"], [39, "setting-up-the-back-propagation-algorithm-part-2"]], "Setting up the equations for a neural network": [[38, "setting-up-the-equations-for-a-neural-network"], [39, "setting-up-the-equations-for-a-neural-network"]], "Setting up the network using Autograd; The full program": [[2, "setting-up-the-network-using-autograd-the-full-program"]], "Similar (second order function now) problem but now with AdaGrad": [[13, "similar-second-order-function-now-problem-but-now-with-adagrad"], [34, "similar-second-order-function-now-problem-but-now-with-adagrad"]], "Simple Python Code to read in Data and perform Classification": [[9, "simple-python-code-to-read-in-data-and-perform-classification"]], "Simple case": [[32, "simple-case"], [33, "simple-case"]], "Simple code for solving the above problem": [[33, "simple-code-for-solving-the-above-problem"]], "Simple example": [[36, "simple-example"], [38, "simple-example"]], "Simple example code": [[34, "simple-example-code"]], "Simple example to illustrate Ordinary Least Squares, Ridge and Lasso Regression": [[33, "simple-example-to-illustrate-ordinary-least-squares-ridge-and-lasso-regression"]], "Simple geometric interpretation": [[33, "simple-geometric-interpretation"]], "Simple linear regression model using scikit-learn": [[0, "simple-linear-regression-model-using-scikit-learn"], [31, "simple-linear-regression-model-using-scikit-learn"]], "Simple neural network and the back propagation equations": [[38, "simple-neural-network-and-the-back-propagation-equations"], [39, "simple-neural-network-and-the-back-propagation-equations"]], "Simple one-dimensional second-order polynomial": [[18, "simple-one-dimensional-second-order-polynomial"]], "Simple program": [[33, "simple-program"], [34, "simple-program"]], "Simpler examples first, and automatic differentiation": [[38, "simpler-examples-first-and-automatic-differentiation"]], "Slightly different approach": [[34, "slightly-different-approach"]], "Smarter way of evaluating the above function": [[38, "smarter-way-of-evaluating-the-above-function"]], "Sneaking in automatic differentiation using Autograd": [[34, "sneaking-in-automatic-differentiation-using-autograd"]], "Software and needed installations": [[25, "software-and-needed-installations"], [31, "software-and-needed-installations"]], "Solving Differential Equations with Deep Learning": [[2, null]], "Solving the one dimensional Poisson equation": [[2, "solving-the-one-dimensional-poisson-equation"]], "Solving the wave equation with Neural Networks": [[2, "solving-the-wave-equation-with-neural-networks"]], "Solving using Newton-Raphson\u2019s method": [[36, "solving-using-newton-raphson-s-method"], [37, "solving-using-newton-raphson-s-method"]], "Some famous Matrices": [[24, "some-famous-matrices"]], "Some parallels from real analysis": [[38, "some-parallels-from-real-analysis"]], "Some selected properties": [[36, "some-selected-properties"]], "Some simple problems": [[13, "some-simple-problems"], [33, "some-simple-problems"]], "Some useful matrix and vector expressions": [[32, "some-useful-matrix-and-vector-expressions"]], "Splitting our Data in Training and Test data": [[0, "splitting-our-data-in-training-and-test-data"], [32, "splitting-our-data-in-training-and-test-data"]], "Standard Approach based on the Normal Distribution": [[35, "standard-approach-based-on-the-normal-distribution"]], "Standard steepest descent": [[13, "standard-steepest-descent"]], "Statistical analysis": [[35, "statistical-analysis"], [36, "statistical-analysis"]], "Statistical analysis and optimization of data": [[23, "statistical-analysis-and-optimization-of-data"], [31, "statistical-analysis-and-optimization-of-data"]], "Steepest descent": [[13, "steepest-descent"], [33, "steepest-descent"]], "Stochastic Gradient Descent": [[34, "stochastic-gradient-descent"]], "Stochastic Gradient Descent (SGD)": [[13, "stochastic-gradient-descent-sgd"], [34, "stochastic-gradient-descent-sgd"]], "Stochastic variables and the main concepts, the discrete case": [[28, "stochastic-variables-and-the-main-concepts-the-discrete-case"]], "Strongly Convex Case": [[34, "strongly-convex-case"]], "Suggested readings and videos": [[37, "suggested-readings-and-videos"]], "Summing up": [[35, "summing-up"], [36, "summing-up"]], "Support Vector Machines, overarching aims": [[8, null]], "Synthetic data generation": [[36, "synthetic-data-generation"], [37, "synthetic-data-generation"]], "Systematic reduction": [[3, "systematic-reduction"]], "Teachers": [[31, "teachers"]], "Teachers and Grading": [[29, null]], "Teaching Assistants Fall semester 2023": [[29, "teaching-assistants-fall-semester-2023"]], "Tensorflow": [[39, "tensorflow"]], "Tentative deadllines for projects": [[29, "tentative-deadllines-for-projects"]], "Testing the Means Squared Error as function of Complexity": [[0, "testing-the-means-squared-error-as-function-of-complexity"], [32, "testing-the-means-squared-error-as-function-of-complexity"]], "Testing the XOR gate and other gates": [[39, "testing-the-xor-gate-and-other-gates"]], "Textbooks": [[30, null]], "The back propagation equations for a neural network": [[39, "the-back-propagation-equations-for-a-neural-network"]], "The Algorithm before theorem": [[11, "the-algorithm-before-theorem"]], "The Breast Cancer Data, now with Keras": [[1, "the-breast-cancer-data-now-with-keras"]], "The CART algorithm for Classification": [[9, "the-cart-algorithm-for-classification"]], "The CART algorithm for Regression": [[9, "the-cart-algorithm-for-regression"]], "The CIFAR01 data set": [[3, "the-cifar01-data-set"]], "The Central Limit Theorem": [[35, "the-central-limit-theorem"]], "The Hessian matrix": [[33, "the-hessian-matrix"], [34, "the-hessian-matrix"]], "The Hessian matrix for Ridge Regression": [[33, "the-hessian-matrix-for-ridge-regression"], [34, "the-hessian-matrix-for-ridge-regression"]], "The Jacobian": [[32, "the-jacobian"]], "The MNIST dataset again": [[3, "the-mnist-dataset-again"]], "The Neural Network": [[39, "the-neural-network"]], "The OLS case": [[33, "the-ols-case"]], "The RELU function family": [[1, "the-relu-function-family"], [39, "the-relu-function-family"]], "The Ridge case": [[33, "the-ridge-case"]], "The SVD, a Fantastic Algorithm": [[32, "the-svd-a-fantastic-algorithm"], [33, "the-svd-a-fantastic-algorithm"]], "The Softmax function": [[1, "the-softmax-function"], [39, "the-softmax-function"]], "The \\chi^2 function": [[0, "the-chi-2-function"], [31, "the-chi-2-function"], [31, "id4"], [31, "id5"], [31, "id6"], [31, "id7"], [31, "id8"]], "The approximation theorem in words": [[38, "the-approximation-theorem-in-words"]], "The bias-variance tradeoff": [[6, "the-bias-variance-tradeoff"], [35, "the-bias-variance-tradeoff"], [36, "the-bias-variance-tradeoff"]], "The code for solving the ODE": [[2, "the-code-for-solving-the-ode"]], "The complete code with a simple data set": [[32, "the-complete-code-with-a-simple-data-set"]], "The cost function rewritten": [[36, "the-cost-function-rewritten"], [37, "the-cost-function-rewritten"]], "The cost/loss function": [[32, "the-cost-loss-function"]], "The course has two central parts": [[23, "the-course-has-two-central-parts"]], "The derivative of the Logistic funtion": [[39, "the-derivative-of-the-logistic-funtion"]], "The derivative of the cost/loss function": [[33, "the-derivative-of-the-cost-loss-function"], [34, "the-derivative-of-the-cost-loss-function"]], "The derivatives": [[38, "the-derivatives"], [39, "the-derivatives"]], "The equations": [[33, "the-equations"]], "The equations for ordinary least squares": [[32, "the-equations-for-ordinary-least-squares"]], "The equations to solve": [[36, "the-equations-to-solve"], [37, "the-equations-to-solve"]], "The first Case": [[33, "the-first-case"]], "The gradient step": [[34, "the-gradient-step"]], "The ideal": [[33, "the-ideal"]], "The logistic function": [[7, "the-logistic-function"], [36, "the-logistic-function"]], "The mean squared error and its derivative": [[32, "the-mean-squared-error-and-its-derivative"]], "The moons example": [[8, "the-moons-example"]], "The multilayer perceptron (MLP)": [[12, "the-multilayer-perceptron-mlp"]], "The network with one input layer, specified number of hidden layers, and one output layer": [[2, "the-network-with-one-input-layer-specified-number-of-hidden-layers-and-one-output-layer"]], "The optimization problem": [[38, "the-optimization-problem"]], "The ouput layer": [[38, "the-ouput-layer"], [39, "the-ouput-layer"]], "The plethora of machine learning algorithms/methods": [[31, "the-plethora-of-machine-learning-algorithms-methods"]], "The same example but now with cross-validation": [[35, "the-same-example-but-now-with-cross-validation"], [36, "the-same-example-but-now-with-cross-validation"]], "The sensitiveness of the gradient descent": [[33, "the-sensitiveness-of-the-gradient-descent"]], "The singular value decomposition": [[5, "the-singular-value-decomposition"], [32, "the-singular-value-decomposition"], [33, "the-singular-value-decomposition"]], "The training": [[38, "the-training"], [39, "the-training"]], "The two-dimensional case": [[8, "the-two-dimensional-case"]], "Theoretical Convergence Speed and convex optimization": [[34, "theoretical-convergence-speed-and-convex-optimization"]], "Time decay rate": [[34, "time-decay-rate"]], "To our real data: nuclear binding energies. Brief reminder on masses and binding energies": [[31, "to-our-real-data-nuclear-binding-energies-brief-reminder-on-masses-and-binding-energies"]], "Topics covered in this course: Statistical analysis and optimization of data": [[31, "topics-covered-in-this-course-statistical-analysis-and-optimization-of-data"]], "Towards the PCA theorem": [[11, "towards-the-pca-theorem"]], "Train and test datasets": [[1, "train-and-test-datasets"], [39, "train-and-test-datasets"]], "Two parameters": [[36, "two-parameters"], [37, "two-parameters"]], "Two-dimensional Objects": [[3, "two-dimensional-objects"]], "Type of problem": [[2, "type-of-problem"]], "Types of Machine Learning": [[31, "types-of-machine-learning"]], "Understanding what happens": [[35, "understanding-what-happens"], [36, "understanding-what-happens"]], "Universal approximation theorem": [[38, "universal-approximation-theorem"]], "Updating the gradients": [[38, "updating-the-gradients"], [39, "updating-the-gradients"]], "Usage of the above learning rate schedulers": [[39, "usage-of-the-above-learning-rate-schedulers"]], "Use the books!": [[19, "use-the-books"]], "Useful Python libraries": [[23, "useful-python-libraries"], [31, "useful-python-libraries"]], "Using Autograd": [[13, "using-autograd"]], "Using Keras": [[39, "using-keras"]], "Using Scikit-learn": [[37, "using-scikit-learn"]], "Using forward Euler to solve the ODE": [[2, "using-forward-euler-to-solve-the-ode"]], "Using gradient descent methods, limitations": [[13, "using-gradient-descent-methods-limitations"], [33, "using-gradient-descent-methods-limitations"], [34, "using-gradient-descent-methods-limitations"]], "Using the chain rule and summing over all k entries": [[38, "using-the-chain-rule-and-summing-over-all-k-entries"], [39, "using-the-chain-rule-and-summing-over-all-k-entries"]], "Using the correlation matrix": [[37, "using-the-correlation-matrix"]], "Vanishing gradients": [[39, "vanishing-gradients"]], "Various steps in cross-validation": [[35, "various-steps-in-cross-validation"], [36, "various-steps-in-cross-validation"]], "Visualization": [[1, "visualization"], [1, "id1"], [39, "visualization"], [39, "id1"]], "Visualizing the Tree, Classification": [[9, "visualizing-the-tree-classification"]], "Week 34: Introduction to the course, Logistics and Practicalities": [[31, null]], "Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression": [[32, null]], "Week 36: Linear Regression and Gradient descent": [[33, null]], "Week 37: Gradient descent methods": [[34, null]], "Week 38: Statistical analysis, bias-variance tradeoff and resampling methods": [[35, null]], "Week 39: Resampling methods and logistic regression": [[36, null]], "Week 40: Gradient descent methods (continued) and start Neural networks": [[37, null]], "Week 41 Neural networks and constructing a neural network code": [[38, null]], "Week 42 Constructing a Neural Network code with examples": [[39, null]], "Weights and biases": [[39, "weights-and-biases"]], "What Is Generative Modeling?": [[31, "what-is-generative-modeling"]], "What does it mean?": [[32, "what-does-it-mean"], [33, "what-does-it-mean"]], "What is Machine Learning?": [[0, "what-is-machine-learning"]], "What is a good model?": [[0, "what-is-a-good-model"], [31, "what-is-a-good-model"]], "What is a good model? Can we define it?": [[31, "what-is-a-good-model-can-we-define-it"]], "When do we stop?": [[34, "when-do-we-stop"]], "Which activation function should I use?": [[1, "which-activation-function-should-i-use"]], "Which activation function should we use?": [[39, "which-activation-function-should-we-use"]], "Why Combine Momentum and RMSProp?": [[34, "why-combine-momentum-and-rmsprop"]], "Why Linear Regression (aka Ordinary Least Squares and family)": [[31, "why-linear-regression-aka-ordinary-least-squares-and-family"]], "Why multilayer perceptrons?": [[37, "why-multilayer-perceptrons"], [38, "why-multilayer-perceptrons"]], "Why resampling methods": [[35, "why-resampling-methods"]], "Why resampling methods ?": [[35, "id1"], [36, "why-resampling-methods"]], "Wisconsin Cancer Data": [[7, "wisconsin-cancer-data"]], "With Lasso Regression": [[33, "with-lasso-regression"]], "Wrapping it up": [[35, "wrapping-it-up"]], "Writing Our First Generative Adversarial Network": [[4, "writing-our-first-generative-adversarial-network"]], "Writing our own PCA code": [[11, "writing-our-own-pca-code"]], "Writing the Cost Function": [[33, "writing-the-cost-function"]], "XGBoost: Extreme Gradient Boosting": [[10, "xgboost-extreme-gradient-boosting"]], "Yet another Example": [[33, "yet-another-example"]], "a) Expression for Ridge regression": [[17, "a-expression-for-ridge-regression"]], "scikit-learn implementation": [[1, "scikit-learn-implementation"], [39, "scikit-learn-implementation"]]}, "docnames": ["chapter1", "chapter10", "chapter11", "chapter12", "chapter13", "chapter2", "chapter3", "chapter4", "chapter5", "chapter6", "chapter7", "chapter8", "chapter9", "chapteroptimization", "clustering", "exercisesweek34", "exercisesweek35", "exercisesweek36", "exercisesweek37", "exercisesweek38", "exercisesweek39", "exercisesweek41", "exercisesweek42", "intro", "linalg", "project1", "project2", "schedule", "statistics", "teachers", "textbooks", "week34", "week35", "week36", "week37", "week38", "week39", "week40", "week41", "week42"], "envversion": {"sphinx": 62, "sphinx.domains.c": 3, "sphinx.domains.changeset": 1, "sphinx.domains.citation": 1, "sphinx.domains.cpp": 9, "sphinx.domains.index": 1, "sphinx.domains.javascript": 3, "sphinx.domains.math": 2, "sphinx.domains.python": 4, "sphinx.domains.rst": 2, "sphinx.domains.std": 2, "sphinx.ext.intersphinx": 1}, "filenames": ["chapter1.ipynb", "chapter10.ipynb", "chapter11.ipynb", "chapter12.ipynb", "chapter13.ipynb", "chapter2.ipynb", "chapter3.ipynb", "chapter4.ipynb", "chapter5.ipynb", "chapter6.ipynb", "chapter7.ipynb", "chapter8.ipynb", "chapter9.ipynb", "chapteroptimization.ipynb", "clustering.ipynb", "exercisesweek34.ipynb", "exercisesweek35.ipynb", "exercisesweek36.ipynb", "exercisesweek37.ipynb", "exercisesweek38.ipynb", "exercisesweek39.ipynb", "exercisesweek41.ipynb", "exercisesweek42.ipynb", "intro.md", "linalg.ipynb", "project1.ipynb", "project2.ipynb", "schedule.md", "statistics.ipynb", "teachers.md", "textbooks.md", "week34.ipynb", "week35.ipynb", "week36.ipynb", "week37.ipynb", "week38.ipynb", "week39.ipynb", "week40.ipynb", "week41.ipynb", "week42.ipynb"], "indexentries": {}, "objects": {}, "objnames": {}, "objtypes": {}, "terms": {"": [0, 1, 2, 3, 4, 5, 6, 7, 9, 11, 12, 13, 15, 16, 17, 19, 21, 22, 23, 24, 25, 26, 28, 29, 31, 32, 38, 39], "0": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 21, 22, 24, 25, 26, 28, 29, 31, 32, 33, 34, 35, 36, 37, 38], "00": [0, 1, 5, 11, 31, 32, 38, 39], "000": [1, 3, 39], "000000": [], "00000000e": [], "001": [2, 8, 13, 21, 33, 34], "004": 5, "004113634617443131": 32, "004113634617443139": 32, "00411363461744314": 32, "004113634617443147": 32, "005b82": [], "00622f": [], "00727646693": [0, 31], "0072b2": [], "00749c": [], "0076268": 21, "008561": [], "0086649156": [0, 31], "00e0e0": [], "01": [0, 1, 2, 5, 9, 11, 13, 17, 30, 31, 32, 34, 36, 37, 38, 39], "010726": [], "0110": 28, "01719003e": [], "02": [0, 4, 7, 12, 31, 36, 37, 39], "02334824": [], "023b95": [], "024c1a": [], "025": 26, "02857": 4, "02f": 6, "03077640549": 4, "03097597e": [], "031": 5, "04": 11, "0458": 9, "05": [4, 6], "0550ae": [], "05767": 38, "062292565": 4, "062435": [], "06730814": [], "07": [], "0713": [0, 31], "07285": 3, "08": 28, "08078025e": [], "080808": [], "08336233266": 4, "08376632": 32, "083766322923899": 32, "0837663229239043": 32, "0917": 9, "0969da4a": [], "0d1117": [], "0n": [0, 31], "0x113e21950": 17, "1": [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 24, 27, 28, 29, 30, 31, 33, 34, 35, 36, 37], "10": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 16, 17, 18, 19, 21, 22, 24, 25, 27, 28, 29, 31, 32, 33, 34, 35, 36, 37, 39], "100": [0, 1, 2, 3, 4, 5, 6, 8, 9, 10, 11, 13, 14, 15, 16, 17, 18, 19, 21, 24, 26, 28, 29, 31, 32, 33, 34, 35, 36, 37, 38, 39], "1000": [0, 1, 2, 4, 5, 8, 11, 13, 14, 18, 19, 21, 23, 26, 28, 31, 33, 34, 36, 37, 39], "10000": [2, 5, 6, 10, 11, 13, 28, 35], "100000": 8, "10001": 10, "1001": 28, "1002": 28, "1003": 28, "1005": 28, "1007": [35, 36], "1009": 28, "101": 16, "1011": 28, "1013": 28, "1013904243": 28, "1015": 28, "102": 16, "1023": 28, "1024": 3, "1026": 28, "1027": 28, "103": [1, 39], "1030": 28, "1037": 28, "1038": 28, "1040": 28, "1047": 28, "107": 16, "108": [], "10e": 39, "10th": 9, "10x": [0, 26, 31], "10y": 26, "11": [0, 2, 5, 6, 7, 8, 9, 10, 11, 12, 13, 16, 24, 25, 26, 28, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39], "110": [], "1100": 28, "1101": 28, "111": [1, 7, 12, 36, 37, 38, 39], "112": 16, "11340253": [], "11590451": [], "116": 16, "116329": [], "116633": [], "117": 16, "118": 16, "12": [0, 1, 2, 3, 4, 5, 6, 8, 9, 11, 12, 18, 21, 24, 25, 26, 28, 30, 31, 32, 33, 34, 35, 37, 39], "120": 3, "121": [8, 9, 10, 16], "1215pm": [29, 31], "122": [8, 9, 10], "124": [0, 31], "125": 16, "127": [4, 16], "128": [3, 4, 13, 34], "129": 16, "1298": 9, "12pm": [29, 31], "13": [0, 2, 9, 12, 22, 24, 26, 28, 31, 37], "131": 16, "133": [7, 36], "135": 16, "136": 16, "14": [0, 2, 4, 6, 8, 9, 10, 12, 24, 26, 28, 30, 32, 35, 36], "141": 16, "1412": 34, "141414": [], "143": 16, "1446729567": 4, "149": 16, "14g": [6, 35], "15": [0, 2, 4, 6, 7, 8, 9, 12, 13, 25, 26, 28, 31, 33, 34, 36, 37], "150": [4, 8, 21, 36, 37], "1502": 38, "152": 16, "153760": [], "156": 16, "157": [], "158": [], "159": 16, "15g": [6, 35], "15pm": 31, "16": [1, 2, 3, 4, 5, 8, 9, 10, 21, 28, 31, 33, 35], "160": 16, "1603": 3, "161": 16, "162": 16, "16231451": 4, "163": 16, "16384": 3, "164": 16, "167": 16, "17": [1, 2, 8, 22, 28, 39], "172": 16, "173": 16, "175": [35, 36], "176": 16, "178": 16, "179": 16, "1797": [1, 39], "18": [2, 6, 7, 8, 9, 10, 28, 31, 35, 36], "1807": 4, "181036": [], "18392847": [], "18c1c4": [], "19": [2, 28, 31, 35], "192": [35, 36], "1940": [], "1943": [12, 37, 38], "19569961": 32, "19680801": [], "1970": [24, 31], "1973": 9, "1979": [6, 35], "1989": 38, "1991": 38, "1_1": [12, 37], "1_2": [12, 37], "1_3": [12, 37], "1cm": [0, 8, 10, 28, 31, 38, 39], "1d": [1, 2, 3, 36, 37, 39], "1e": [2, 4, 13, 14, 34, 36, 37, 39], "1e10": 14, "1e1e1": [], "1e4": 6, "1f": 1, "1ffvbn0xlhv": 22, "1k": 24, "1n": [0, 31], "1x": [0, 31], "1zkibvqf": 21, "2": [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 23, 24, 25, 28, 30, 34, 35, 36, 37], "20": [0, 1, 2, 6, 7, 8, 16, 17, 28, 29, 31, 32, 33, 34, 35, 36, 37, 38, 39], "200": [0, 2, 3, 4, 8, 9, 10, 36, 37], "2000": [0, 32], "2001": [], "2004": [13, 33], "2006": 30, "2007": [], "20072279": [], "2008": [31, 34], "2009": [], "2010": [1, 39], "2011": [1, 34, 39], "2012": 34, "2013": [], "2014": [4, 34], "2015": [1, 39], "2016": [0, 31], "2018": [0, 6, 32, 35, 36], "2019": [], "2020": [], "2021": [6, 14, 32, 34], "2022": [26, 31, 38, 39], "2023": 39, "2024": [21, 35], "2025": [18, 21, 22, 26, 31, 32, 33, 34, 35], "21": [0, 1, 5, 7, 9, 12, 24, 31, 32, 33, 36, 37, 38, 39], "2116753732": 4, "215pm": [29, 31], "2167072": [], "22": [0, 1, 5, 12, 13, 24, 31, 32, 33, 37, 39], "221": 8, "225": 4, "22948497": [], "23": [1, 12, 24, 37, 39], "24": [0, 1, 24, 31, 39], "242424": [], "24292f": [], "25": [2, 3, 4, 5, 6, 8, 9, 11, 32], "250": [2, 4, 7, 9, 36], "25000": [], "250154": [], "252124": [], "253775": [], "255": [3, 26], "256": [4, 34], "25x": [25, 26], "26": [], "26303845": [], "264": [], "265": [], "265109911": 4, "266": [], "269": [], "27": [1, 39], "270": [], "278": [33, 34], "27n_": 28, "28": [1, 3, 4, 39], "283": [33, 34], "2830637392": 4, "2861": 28, "2873": 9, "2882": 28, "2886": 28, "2890": [0, 31], "2892": 28, "29": 32, "2915": 28, "2931": 31, "29364655": [], "294399745619595": [], "296247": [], "2968": 31, "2980": [21, 31], "298273": [], "298375": [], "2990": 31, "2_": [12, 37], "2_1": [12, 37], "2_2": [12, 37], "2_3": [12, 37], "2_i": [12, 37], "2_m": [6, 28, 35], "2_t": 13, "2_x": 28, "2a": 17, "2a1968": [], "2b": 28, "2b2b2b": [], "2c8f433990d1": 34, "2cm": 8, "2d": [1, 3, 11, 12, 23, 31, 36, 37, 38, 39], "2e": [6, 35, 36], "2f": [0, 7, 9, 10, 11, 12, 31, 36, 37], "2g": 2, "2g_i": 2, "2k": 3, "2m": [6, 35], "2mvizaqfst8": 32, "2n": [0, 2, 3, 31, 32], "2nd": 9, "2p": [28, 38], "2pt": 4, "2x": [0, 3, 8, 13, 31, 38], "2x_ix_jy_iy_j": 8, "2x_j": 8, "2xb": 38, "2y_i": 10, "2y_j": 8, "3": [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 23, 24, 25, 26, 27, 28, 29, 31, 33, 34, 35, 36, 37], "30": [0, 1, 4, 6, 7, 10, 13, 29, 34, 35, 36, 37, 39], "300": [36, 37], "30000": [0, 31], "3072": 3, "31": [12, 24, 28, 37], "315": [6, 32, 34], "3155": [0, 5, 6, 32, 33, 34, 35, 36], "32": [3, 4, 6, 12, 13, 24, 28, 34, 37], "3200": [1, 39], "3250": [1, 39], "3297": [], "33": [12, 24, 29, 37], "3303": [], "3310": [], "332331": [], "333": [7, 36], "3331": [], "3337": [], "34": 24, "3436": [0, 31], "3437": [0, 31], "35": [0, 6, 25, 31, 33, 34], "3581341341": 4, "359": [5, 33], "36": [0, 5, 6, 18, 25, 28], "37": [25, 33, 35, 36], "370782966": 4, "38": [25, 28], "387": [35, 36], "39": [0, 25, 26, 29, 31], "3d": [2, 3, 4, 6, 13, 16, 35, 36], "3d73a9": [], "3f": [1, 3, 9, 39], "3n": 24, "3x": [2, 8], "3x_0x_1": 38, "3x_i": 2, "3y": 8, "4": [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 24, 25, 26, 28, 31, 33, 34, 35, 36, 37, 38], "40": [1, 6, 29, 31, 35, 36, 39], "400": 4, "4000": 31, "40008b9a5380fcacce3976bf7c08af5b": 34, "4050": [30, 31], "41": [24, 26], "4155": [2, 15], "41589548": [], "42": [1, 4, 8, 9, 10, 24, 26, 36, 37, 38], "43": [0, 7, 24], "4310": 31, "436462435": 4, "437a6b": [], "44": [0, 24, 33, 34], "45": [29, 31], "46": [29, 31], "462": [7, 36], "47": [29, 31], "473d18": [], "479465113": 4, "47958494": [], "48": [], "48257387": [29, 31], "49": [5, 6, 11], "49152": 3, "4940954": [0, 31], "4990": 28, "4992": 28, "4997": 28, "4c4b4be8": [], "4c4c7f": [9, 10], "4d": 3, "4f": [6, 26, 36, 37], "4pm": [29, 31], "4y": 8, "4y_i": 10, "5": [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 17, 24, 25, 26, 28, 31, 32, 33, 34, 35, 36, 37, 38, 39], "50": [1, 2, 3, 4, 6, 7, 8, 10, 13, 26, 31, 32, 34, 35, 37, 38, 39], "500": [1, 3, 4, 6, 9, 10, 13, 34, 35, 36, 39], "5000": [25, 26], "5018": 28, "506": [], "507d50": [9, 10], "50j": 13, "50x10": [1, 39], "51": 10, "510": [1, 39], "512132": [], "515151": [], "5177783846": 4, "52": 36, "53": [9, 36], "5391cf": [], "54": [6, 28], "5411205": [], "54894451": [], "55": [1, 39], "56": [1, 39], "56469864": 21, "56536": [0, 31], "569": 1, "57": [0, 8, 29, 31], "571": [5, 33], "576": 35, "58": [10, 29, 31], "58a6ff70": [], "591317992": 4, "5ca7e4": [], "5cm": 28, "5f": [8, 34], "5x": [8, 18], "5y": 8, "6": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 12, 13, 18, 24, 26, 28, 29, 31, 32, 33, 34, 35, 36, 37, 39], "60": [1, 3], "60000": 4, "6019067271": 4, "60610368": 21, "606439": [], "61362": 26, "622cbc": [], "625": [7, 36], "63": [1, 39], "64": [1, 3, 4, 13, 24, 31, 34, 39], "64x50": [1, 39], "65": [1, 8, 9, 39], "66666691": [], "66707b": [], "66ccee": [], "66e9ec": [], "6730c5": [], "6887363571": 4, "69": [16, 28], "69069n_": 28, "691": [], "6980": 34, "6e7681": [], "6e7781": [], "6f98b3": [], "6n_": 28, "7": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 11, 12, 13, 24, 25, 26, 28, 30, 31, 32, 34, 35, 36, 37, 38, 39], "70": [1, 7, 36, 39], "702c00": [], "70653767": 4, "71": [1, 39], "724": 3, "72f088": [], "73": [], "7304881": [], "737373": [], "75": [5, 6, 8, 11, 35], "76": [29, 31, 36], "765": [7, 36], "77": [29, 31], "7718": 9, "7782028952": 4, "77893972": [], "78": [], "797979": [], "7998f2": [], "79c0ff": [], "7d7d58": [9, 10], "7ee787": [], "7f4707": [], "8": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 12, 13, 14, 18, 19, 21, 24, 26, 28, 29, 31, 33, 36, 37, 39], "80": [0, 1, 5, 8, 17, 32, 39], "800": [4, 7, 36], "8045e5": [], "81": [1, 39], "815am": [29, 31], "81b19b": [], "8250df": [], "84858": [35, 36], "85": [1, 39], "8702784034": 4, "8786ac": [], "88": 31, "8a4600": [], "8b949e": [], "8c8c8c": [], "8f": [6, 35, 36], "8g": [6, 35], "8n": 24, "8x8": [1, 39], "9": [0, 1, 2, 4, 5, 6, 7, 8, 9, 11, 12, 13, 24, 28, 31, 34, 36, 37, 39], "90": 1, "9040": 9, "91": [29, 31], "912583": [], "91cbff": [], "92": [29, 31], "93": 16, "931": [0, 31], "933": [5, 33], "937": 28, "938": 28, "939": [0, 28, 31], "94": 28, "95": [1, 11, 35, 39], "953800": [], "954": 28, "955820c21e8b": 4, "9579870417283": 21, "96": [6, 35], "960": 28, "961": 28, "962": 28, "9649652536": 4, "96611194e": [], "974eb7": [], "978": [35, 36], "9780387310732": 30, "9780387848570": 30, "9781098134174": 31, "9781492032632": 30, "9781801819312": 31, "97898392": 32, "98": [0, 1, 16, 39], "985": 28, "986": 28, "98661b": [], "989": 28, "9898ff": [9, 10], "99": [13, 16, 34, 35], "991": 28, "992": 28, "993": 28, "996": 5, "996b00": [], "999": [9, 28, 34, 39], "999999": [], "9e86c8": [], "9e8741": [], "9f4e55": [], "9x": 6, "9y": 6, "A": [2, 3, 5, 6, 7, 10, 11, 12, 13, 15, 16, 19, 20, 23, 24, 25, 26, 27, 28, 29, 30, 32, 33, 34, 38], "AND": 2, "AS": [], "AT": [], "And": [0, 3, 4, 5, 6, 9, 13, 20, 22, 23, 25, 26, 28, 33], "As": [0, 1, 2, 3, 4, 5, 6, 8, 10, 12, 13, 15, 16, 24, 25, 26, 28, 31, 32, 33, 34, 35, 36, 37, 38, 39], "At": [0, 4, 6, 13, 20, 31, 34], "BE": [0, 31], "BUT": [], "BY": [], "Be": [2, 18, 23, 31], "Being": 13, "But": [0, 1, 2, 3, 5, 6, 9, 10, 16, 21, 26, 28, 32, 35, 36, 39], "By": [0, 3, 5, 6, 12, 13, 17, 19, 24, 31, 32, 33, 34, 35, 37], "FOR": [], "For": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 18, 19, 21, 22, 23, 24, 25, 26, 28, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39], "IF": [6, 32, 34], "IN": 30, "If": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 15, 16, 18, 21, 22, 23, 24, 25, 26, 28, 31, 32, 33, 34, 35, 36, 37, 38, 39], "In": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 18, 19, 21, 22, 23, 24, 25, 26, 28, 30, 31, 32, 33, 34, 35, 36, 37, 39], "Ising": [5, 12, 32, 33, 37, 38], "It": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 18, 20, 21, 22, 23, 24, 25, 26, 28, 31, 32, 33, 34, 35, 36, 37, 38, 39], "Its": [1, 2, 4, 11, 39], "NO": [], "NOT": [], "No": [6, 9, 31, 32, 34, 37, 39], "Not": [0, 1, 5, 6, 32, 33, 34, 35, 37, 39], "OF": [], "ON": [], "OR": 28, "Of": 28, "On": [0, 3, 25, 28, 29, 30, 31, 34, 35], "One": [0, 1, 3, 4, 5, 6, 7, 8, 11, 12, 13, 17, 28, 32, 33, 34, 35, 36, 37, 38, 39], "Or": [0, 1, 6, 31, 39], "SUCH": [], "Such": [0, 6, 12, 16, 28, 34, 35, 36, 37, 38, 39], "THE": [], "TO": 39, "That": [0, 5, 7, 10, 11, 12, 14, 25, 26, 28, 31, 35, 36, 37, 38, 39], "The": [4, 10, 13, 14, 16, 17, 18, 19, 20, 21, 22, 24, 25, 26, 27, 28, 29, 30], "Then": [0, 1, 6, 8, 9, 10, 11, 12, 13, 14, 15, 16, 18, 19, 20, 21, 24, 31, 33, 34, 35, 38, 39], "There": [0, 3, 4, 5, 6, 8, 9, 11, 12, 14, 15, 24, 25, 26, 28, 29, 31, 32, 33, 34, 37, 38], "These": [0, 3, 4, 5, 8, 9, 10, 11, 12, 13, 14, 17, 18, 22, 24, 25, 26, 28, 29, 31, 32, 33, 34, 38, 39], "To": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 15, 16, 20, 21, 22, 24, 26, 28, 32, 33, 34, 35, 36, 37, 38, 39], "WITH": [], "Will": [36, 37], "With": [0, 5, 6, 8, 9, 10, 11, 12, 14, 16, 19, 21, 24, 25, 26, 28, 31, 32, 35, 36, 37, 38, 39], "_": [0, 1, 2, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 16, 17, 18, 19, 21, 24, 25, 31, 32, 33, 34, 35, 36, 37, 39], "_0": [5, 8, 10, 11, 13, 32, 33], "_1": [2, 5, 6, 8, 10, 11, 12, 13, 14, 24, 32, 33, 34, 38, 39], "_2": [2, 5, 8, 11, 12, 13, 24, 32, 34, 37], "_3": 24, "_4": 24, "_9": [13, 34], "__array_finalize__": [], "__class__": [10, 39], "__doc__": [6, 35, 36], "__future__": [8, 9, 38], "__getattribute__": [], "__import__": [], "__init__": [1, 22, 36, 37, 39], "__main__": 2, "__name__": [2, 10, 39], "__new__": [], "__path__": [], "_accuraci": 39, "_add_intercept": [36, 37], "_auto1": [2, 3, 4, 5, 6, 7, 12, 13, 24, 28, 32, 33, 36, 37, 38, 39], "_auto10": [6, 12], "_auto11": 6, "_auto12": 6, "_auto2": [2, 3, 4, 5, 6, 12, 13, 24, 28, 37, 38, 39], "_auto3": [3, 4, 5, 6, 12, 13, 24, 37, 38, 39], "_auto4": [4, 6, 12, 13, 24, 37], "_auto5": [4, 6, 12, 13, 24, 37], "_auto6": [4, 6, 12, 24, 37], "_auto7": [4, 6, 12, 24, 37], "_auto8": [6, 12], "_auto9": [6, 12], "_backpropag": 39, "_build": [0, 23, 25, 26, 30, 31, 39], "_c": [1, 39], "_center": [], "_compile_transl": [], "_compon": 11, "_da": 22, "_data": [], "_depth": 9, "_export": [15, 16, 19], "_feed_forward_sav": 22, "_feedforward": 39, "_format": 39, "_fraction": 9, "_i": [0, 1, 2, 5, 6, 7, 8, 11, 12, 13, 19, 25, 31, 32, 33, 34, 35, 36, 37, 38, 39], "_j": [0, 1, 2, 3, 5, 6, 8, 13, 19, 25, 32, 33, 34, 35, 36, 39], "_k": [13, 33, 34, 39], "_l": [12, 37, 38, 39], "_lambda": 6, "_leaf": 9, "_m": 10, "_mask": [], "_multilayer_perceptron": [], "_n": [2, 5, 8, 11, 13, 32, 33, 34], "_node": 9, "_norm": [], "_p": [5, 8, 32, 33], "_parse_numpydoc_see_also_sect": [], "_progress_bar": 39, "_pydevd_bundl": [], "_ratio": 11, "_sampl": 9, "_set_classif": 39, "_sigmoid": [36, 37], "_softmax": [36, 37], "_split": [6, 9, 25], "_t": [13, 34], "_test": [6, 25], "_varianc": 11, "_weight": 9, "a0": 3, "a0111f": [], "a0faa0": [9, 10], "a1": [0, 21, 22, 31], "a11": [], "a12236": [], "a2": [0, 21, 22, 31], "a25e53": [], "a2bffc": [], "a3": [0, 31], "a4": [0, 31], "a5d6ff": [], "a_": [0, 1, 16, 24, 31, 32, 38, 39], "a_0": [0, 31, 38, 39], "a_1": [38, 39], "a_1a": [0, 31], "a_2": [38, 39], "a_2a": [0, 31], "a_3": [0, 31], "a_3a": [0, 31], "a_4": [0, 31], "a_4a": [0, 31], "a_h": [1, 39], "a_i": [0, 1, 2, 12, 31, 38, 39], "a_j": [1, 12, 38, 39], "a_k": [0, 1, 12, 38, 39], "a_matric": 39, "aa": [], "aaa": [], "aaron": 30, "ab": [0, 2, 5, 13, 14, 31, 32, 34, 38], "ab6369": [], "ab_channel": [23, 37, 38, 39], "abandon": [1, 39], "abe338": [], "abid": 28, "abil": [0, 10], "abl": [0, 1, 4, 5, 6, 7, 10, 12, 13, 16, 18, 20, 21, 25, 32, 33, 34, 36, 37, 38, 39], "about": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 15, 16, 19, 20, 22, 23, 24, 25, 29, 34, 35, 36, 37, 39], "abov": [0, 1, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 21, 22, 24, 26, 28, 30, 31, 32, 34, 35, 36, 37], "abovement": [6, 25, 31, 35, 36], "abscissa": [13, 33], "absent": 34, "absolut": [0, 2, 5, 6, 13, 31, 32, 33, 35, 36], "absorb": [32, 33], "abstract": [1, 34, 36, 39], "abund": 34, "ac": [], "acc_bin": [36, 37], "acc_multi": [36, 37], "acceler": [13, 34], "accept": [0, 3, 6, 9, 21, 25, 32, 34], "access": [3, 11, 28, 31, 34], "accid": [4, 6, 35, 36], "accompani": [0, 31, 32], "accomplish": [8, 9, 13, 34], "accord": [0, 1, 2, 5, 6, 9, 12, 13, 14, 28, 31, 33, 34, 35, 37, 38, 39], "accordingli": 11, "account": [0, 3, 5, 13, 15, 16, 20, 28, 31, 34], "accumul": [12, 13, 28, 34, 37, 38, 39], "accur": [0, 3, 4, 6, 10, 13, 34, 35, 36], "accuraci": [0, 1, 3, 4, 5, 6, 7, 9, 10, 11, 12, 21, 26, 31, 32, 33, 36, 37, 38, 39], "accuracy_scor": [0, 1, 10, 21, 22, 26, 31, 36, 37, 39], "accuracy_score_numpi": [1, 39], "acheiv": 21, "achiev": [0, 1, 5, 6, 8, 12, 24, 31, 34, 35, 36, 37, 38, 39], "aco": 28, "acquaint": 23, "acquir": [1, 23, 31, 39], "acr": [], "across": [1, 3, 6, 9, 17, 23, 31, 35, 39], "act": [1, 3, 24, 34, 39], "act_func": 39, "act_func_deriv": 39, "actic": 21, "action": 28, "activ": [0, 2, 3, 4, 9, 15, 22, 27, 29, 31, 34], "activation_d": 22, "activation_func": [21, 22], "activest": [], "actual": [0, 1, 4, 5, 6, 8, 11, 15, 16, 18, 21, 24, 28, 31, 32, 33, 34, 35, 39], "ad": [1, 3, 4, 5, 8, 13, 15, 16, 24, 33, 34, 35, 36], "ada_clf": 10, "adaboostclassifi": 10, "adadelta": [13, 34], "adagrad": [25, 35, 38, 39], "adagradmomentum": 39, "adam": [1, 3, 4, 21, 25, 26, 31, 35, 38, 39], "adam_schedul": 39, "adap": 38, "adapt": [4, 6, 13, 17, 30, 33, 35, 36, 38], "add": [0, 1, 2, 3, 4, 5, 6, 8, 10, 11, 12, 15, 16, 17, 18, 20, 21, 26, 28, 29, 31, 32, 33, 34, 35, 36, 37, 38, 39], "add6ff": [], "add_": [], "add_subplot": [1, 7, 12, 14, 36, 37, 39], "addendum": 5, "addeventlisten": [], "addit": [0, 2, 3, 5, 6, 7, 8, 9, 10, 12, 13, 15, 21, 23, 24, 25, 26, 28, 29, 30, 31, 32, 35, 36, 37, 38, 39], "addition": [12, 13, 33, 34, 37, 38], "address": [1, 9, 11, 13, 31, 34, 39], "adjac": [3, 12, 37, 38], "adjoint": [5, 32], "adjust": [0, 5, 12, 13, 33, 34, 37], "admir": [0, 31], "advanc": [4, 6, 12, 30, 31, 34, 35, 36, 37, 38], "advantag": [1, 3, 5, 6, 10, 13, 19, 24, 33, 34, 35, 36, 39], "adversari": 31, "advis": [], "afecionado": 31, "affect": [3, 15, 19, 39], "affin": [0, 3, 8, 11, 32, 38], "afford": 3, "aficionado": 31, "aforement": 14, "african": [], "after": [0, 1, 2, 4, 5, 6, 9, 11, 12, 13, 15, 16, 17, 18, 19, 20, 21, 23, 24, 25, 26, 28, 31, 32, 33, 34, 35, 38, 39], "afterward": [0, 31], "ag": [0, 7, 31, 32, 36], "ag_0": 2, "again": [0, 1, 4, 5, 6, 7, 8, 10, 11, 12, 13, 25, 26, 28, 31, 32, 33, 35, 36, 37, 38, 39], "against": [1, 4, 7, 10, 36, 39], "agegroup": [7, 36], "agegroupmean": [7, 36], "aggreg": [9, 10, 34], "agorithm": 10, "agre": [5, 6, 28, 32, 33, 34, 35], "agreement": [13, 34], "ahead": 9, "ai": [0, 30], "aid": [11, 20, 34], "aim": [0, 1, 4, 6, 7, 11, 14, 16, 17, 19, 20, 23, 24, 25, 26, 32, 35, 36, 37, 38, 39], "ainv": 5, "airplan": 3, "aka": [5, 26], "al": [0, 2, 4, 16, 17, 20, 26, 30, 31, 32, 33, 35, 36, 37, 38, 39], "alarm": [5, 7], "aldo": 32, "algebra": [0, 3, 5, 13, 23, 32, 33, 35], "algorithm": [0, 1, 2, 4, 5, 6, 7, 8, 13, 14, 16, 23, 24, 25, 28, 30, 35, 36, 37], "align": [0, 2, 5, 6, 7, 8, 13, 28, 31, 32, 33, 35, 36, 37], "all": [0, 1, 2, 3, 4, 5, 6, 7, 9, 10, 11, 12, 13, 14, 15, 18, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37], "allclos": 21, "allevi": [1, 13, 33, 39], "alloc": [3, 24], "allow": [0, 1, 2, 3, 5, 6, 8, 10, 13, 15, 23, 24, 25, 31, 32, 33, 34, 35, 36, 37, 38, 39], "almost": [0, 1, 6, 8, 11, 13, 28, 33, 34, 35, 36, 37, 39], "alon": [2, 9, 34], "along": [2, 3, 4, 5, 6, 9, 10, 11, 15, 20, 21, 22, 23, 24, 31, 32, 33, 35, 36, 39], "alpha": [0, 1, 2, 3, 4, 6, 7, 8, 9, 10, 13, 14, 28, 31, 32, 33, 34, 35, 36, 37, 39], "alpha_": [10, 34], "alpha_0": 3, "alpha_1": 3, "alpha_2": 3, "alpha_i": [3, 13], "alpha_k": 13, "alpha_m": 10, "alpha_n": 3, "alpha_opt": 13, "alreadi": [2, 3, 4, 5, 6, 10, 12, 15, 22, 23, 24, 28, 31, 32, 33, 36, 37, 38], "also": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 18, 19, 21, 22, 23, 24, 25, 26, 28, 31, 32, 33, 34, 35, 36, 37, 38, 39], "alter": [1, 39], "altern": [0, 1, 4, 5, 6, 8, 9, 11, 13, 15, 18, 24, 25, 31, 32, 34, 35, 36, 39], "although": [0, 1, 5, 6, 8, 10, 13, 16, 19, 20, 31, 34, 35, 36, 38, 39], "alwai": [0, 3, 5, 6, 12, 13, 16, 19, 21, 22, 25, 26, 28, 31, 32, 33, 34, 35, 37, 38], "am": 4, "ambit": [38, 39], "ame2016": [0, 31], "american": [], "amjith": [], "among": [0, 3, 5, 9, 10, 12, 24, 31, 32, 37, 38], "amongst": [5, 35], "amount": [0, 1, 3, 4, 6, 8, 10, 14, 23, 35, 36, 38, 39], "an": [1, 2, 3, 5, 6, 7, 8, 9, 11, 12, 13, 14, 16, 17, 18, 19, 21, 22, 23, 24, 25, 26, 28, 29, 30, 32, 33, 34, 35, 36, 37, 39], "an_": 28, "anaconda": [0, 1, 23, 25, 31, 39], "analogi": 13, "analys": [6, 35, 36], "analysi": [1, 3, 4, 7, 14, 19, 24, 30, 34, 37, 39], "analyt": [2, 3, 5, 6, 7, 12, 13, 17, 22, 23, 25, 31, 32, 33, 34, 35, 36, 37, 38], "analyz": [0, 1, 3, 4, 5, 6, 16, 25, 26, 28, 32, 33, 34], "andrew": [1, 39], "angl": [0, 3, 9, 32, 34], "anharmon": 3, "ani": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 12, 14, 15, 16, 19, 21, 28, 31, 32, 34, 35, 38, 39], "anim": [4, 12, 37, 38], "ann": [12, 37, 38], "annot": [0, 1, 3, 7, 8, 31, 37, 39], "announc": 31, "anom": [], "anomali": [], "anonym": 18, "anoth": [0, 1, 3, 4, 5, 6, 7, 8, 10, 11, 12, 13, 15, 24, 25, 26, 28, 31, 32, 34, 38, 39], "ansatz": [0, 18, 31], "answer": [0, 1, 3, 5, 6, 19, 22, 24, 25, 26, 29, 31, 35, 39], "antialias": [2, 6], "anticip": 4, "anymor": [1, 8, 39], "anyon": [4, 8, 15], "anyth": [1, 15, 16, 21, 22, 28, 39], "anytim": [29, 31], "anywai": [], "apach": [1, 39], "apart": [11, 13, 33, 34], "api": [1, 23, 31, 39], "appar": 2, "appear": [0, 1, 3, 13, 24, 28, 38, 39], "append": [1, 3, 4, 8, 9, 13, 19, 21, 22, 31, 34, 36, 37, 39], "appendic": [25, 26], "appendix": 25, "appli": [0, 1, 3, 4, 6, 7, 8, 9, 10, 11, 12, 13, 18, 25, 26, 28, 30, 31, 32, 34, 35, 36, 37, 38, 39], "applic": [0, 1, 3, 4, 5, 6, 7, 9, 12, 13, 16, 24, 28, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39], "apply_gradi": 4, "approach": [1, 2, 4, 5, 6, 9, 10, 11, 12, 13, 15, 16, 18, 21, 23, 25, 28, 30, 32, 33, 38, 39], "approch": 25, "appropri": [2, 6, 9, 12, 13, 17, 23, 28, 34, 35, 36, 37], "approv": 31, "approx": [0, 2, 3, 6, 10, 11, 13, 18, 25, 28, 31, 33, 34, 35], "approxim": [0, 1, 2, 3, 4, 5, 6, 7, 10, 11, 13, 19, 25, 26, 28, 31, 32, 33, 34, 35, 36, 37, 39], "apt": [0, 23, 25, 31], "aq": 28, "ar": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39], "aragorn": 31, "arang": [1, 3, 4, 6, 7, 9, 10, 12, 13, 31, 34, 36, 37, 39], "arbitrari": [1, 4, 6, 8, 12, 13, 28, 33, 35, 37, 38, 39], "arbitrarili": [0, 1, 11, 31, 34, 39], "arc": 6, "architectur": [3, 4, 12, 26, 38], "archiv": [25, 26], "area": [0, 3, 6, 30, 31], "argmax": [1, 11, 21, 36, 37, 39], "argmin": [4, 10, 14], "argsort": 11, "argu": [1, 13, 39], "arguement": 19, "argument": [0, 2, 3, 5, 11, 12, 13, 17, 21, 31, 32, 34, 35, 37, 38, 39], "aris": [0, 6, 12, 13, 28, 31, 33, 35, 36], "arithmet": [0, 13, 24, 31], "arm": [6, 32, 34], "armadillo": 24, "armin": [], "arnulf": [38, 39], "around": [0, 1, 4, 5, 6, 11, 18, 21, 22, 25, 26, 28, 31, 35, 36, 37, 38, 39], "arrai": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 12, 13, 14, 16, 18, 21, 23, 25, 28, 32, 33, 34, 35, 36, 37, 38, 39], "arrang": [3, 31], "array_equ": [36, 37], "arraybox": 13, "arriv": [0, 6, 9, 11, 19, 24, 28, 31, 35], "arrow": [12, 37, 38, 39], "arrowprop": 8, "art": [0, 1, 23, 39], "articl": [0, 3, 4, 6, 10, 19, 26, 31, 32, 33, 34, 35, 36], "artifici": [0, 2, 7, 12, 30, 31, 36], "artificialneuron": [12, 37, 38], "arug": 13, "arxiv": [3, 4, 34, 38], "as_fram": 26, "asarrai": [0, 6, 9, 32, 34], "asid": 32, "ask": [5, 6, 11, 12, 15, 19, 25, 26, 35, 38, 39], "aspect": [0, 6, 23, 31, 32, 38, 39], "assembl": 3, "assembli": [0, 31], "assert": [4, 39], "assess": [0, 6, 25, 31, 32, 35, 36], "asset": [], "assici": 4, "assign": [0, 7, 8, 9, 12, 13, 14, 15, 27, 29, 30, 31, 36, 37, 39], "associ": [0, 6, 9, 12, 14, 28, 31, 35, 36, 37, 38], "assum": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 17, 19, 24, 25, 28, 31, 32, 33, 34, 35, 36, 37, 38, 39], "assumpt": [0, 3, 5, 6, 9, 11, 28, 31, 32, 36], "ast": [0, 5, 6, 31, 35], "astyp": [4, 9, 10, 36, 37], "asymmetri": [0, 31], "asymptot": [4, 6, 34, 35, 36], "atom": [0, 31], "attain": 34, "attempt": [0, 4, 6, 7, 8, 10, 31, 32, 34, 36, 38, 39], "attend": 31, "attent": [0, 24, 31], "attract": [0, 10, 31], "attribut": [0, 9, 22, 31, 39], "audi": [0, 31], "audio": [3, 4], "august": [31, 32], "aurelien": [0, 30, 31], "austfjel": 6, "auth": 15, "authent": 15, "author": [0, 1, 10, 28, 39], "authour": 31, "auto": [9, 10, 26, 28, 39], "auto_exampl": [21, 25, 32], "autocor": 28, "autocorrelation_tim": 28, "autocorrelform": 28, "autocovari": 28, "autoencod": [4, 23, 31], "autoencond": 23, "autograd": [21, 23, 26, 31, 38, 39], "autograd_compliant_predict": 22, "autograd_gradi": 22, "autograd_one_lay": 22, "autom": [0, 23, 30, 31], "automac": 24, "automag": 31, "automat": [0, 1, 2, 3, 4, 11, 16, 21, 22, 23, 24, 26, 31, 37, 39], "automobil": 3, "autonom": 4, "avail": [0, 1, 4, 6, 10, 11, 23, 24, 25, 26, 27, 29, 30, 31, 35, 36, 39], "avali": [20, 25, 26], "averag": [0, 1, 3, 6, 9, 10, 13, 14, 28, 29, 31, 32, 35, 36, 39], "avoid": [0, 4, 5, 6, 9, 11, 13, 18, 21, 24, 32, 34, 35, 36, 39], "awai": [2, 3, 6, 32, 34, 38], "awar": [2, 10], "award": [29, 31], "ax": [0, 1, 2, 3, 4, 6, 7, 8, 9, 10, 11, 12, 13, 14, 20, 21, 24, 25, 26, 31, 35, 36, 37, 39], "axes3d": [2, 6, 13, 33, 34], "axes_grid1": 6, "axhlin": 8, "axi": [0, 1, 2, 3, 4, 6, 7, 8, 9, 10, 11, 12, 13, 14, 18, 21, 24, 28, 31, 32, 33, 34, 35, 36, 37, 38, 39], "axiom": 5, "axvlin": [4, 8], "axvspan": 4, "b": [0, 1, 3, 4, 5, 6, 8, 9, 10, 12, 13, 14, 15, 16, 17, 19, 20, 21, 22, 28, 29, 31, 32, 33, 34, 35, 36, 37, 38, 39], "b1": [8, 21, 22], "b19db4": [], "b1bac4": [], "b2": [8, 21, 22], "b3": 8, "b35900": [], "b89784": [], "b_": [0, 1, 24, 38, 39], "b_0": [0, 38], "b_1": [0, 2, 12, 13, 34, 37, 38, 39], "b_2": [0, 13, 38, 39], "b_5": [13, 34], "b_g": [21, 22], "b_group": 9, "b_i": [0, 1, 2, 12, 31, 37, 38, 39], "b_ia_": [0, 31], "b_ia_i": 0, "b_index": 9, "b_j": [1, 12, 37, 38, 39], "b_k": [0, 1, 12, 13, 34, 37, 38, 39], "b_m": [12, 37], "b_score": 9, "b_valu": 9, "ba": 34, "babcock": 31, "bach": 34, "bachelor": [27, 29], "back": [0, 3, 4, 5, 6, 8, 9, 10, 15, 16, 21, 24, 26, 28, 31, 34], "backbon": 24, "backend": [1, 4, 39], "background": [30, 31, 39], "backprogag": 22, "backpropag": [1, 21, 34, 38, 39], "backpropog": 22, "backslash": [], "backtrack": 9, "backup": 24, "backward": [1, 2, 4, 12, 22, 24, 34, 38, 39], "bad": [6, 17, 32, 39], "badli": 28, "bag": [9, 23, 31], "bag_clf": 10, "baggin": 31, "baggingboot": 10, "baggingclassifi": 10, "baggingtre": 10, "bailei": [], "balanc": [6, 34, 35, 36], "ballpark": 18, "band": 24, "bandwidth": 24, "banner": [], "bar": [0, 6, 11, 25, 31, 39], "barber": 30, "bare": [4, 10], "base": [0, 1, 3, 4, 5, 7, 8, 9, 10, 14, 15, 16, 17, 23, 28, 29, 30, 31, 32, 33, 36, 37, 38, 39], "basi": [5, 7, 8, 10, 11, 12, 13, 24, 32, 33, 36, 37, 38], "basic": [6, 8, 12, 13, 14, 15, 23, 25, 28, 31, 35, 39], "basin": 34, "batch": [3, 4, 11, 12, 13, 21, 33, 36, 37], "batch_shap": 4, "batch_siz": [1, 3, 4, 39], "batchnorm": 4, "bay": [7, 36, 37], "baydin": 38, "bayesian": [5, 23, 30, 31], "bbbbbb": [], "beauti": [], "becam": [], "becaus": [0, 1, 2, 3, 4, 5, 6, 8, 9, 12, 13, 14, 31, 32, 33, 34, 35, 36, 37, 39], "becom": [0, 1, 2, 5, 6, 7, 9, 12, 13, 19, 28, 31, 32, 33, 34, 35, 36, 37, 38, 39], "been": [0, 1, 2, 3, 4, 5, 6, 11, 12, 13, 19, 20, 23, 24, 25, 26, 31, 32, 34, 35, 37, 38, 39], "befor": [0, 1, 2, 3, 4, 5, 6, 7, 8, 12, 13, 14, 16, 17, 18, 19, 20, 21, 22, 24, 25, 28, 31, 32, 34, 35, 36, 37, 38, 39], "beforehand": [0, 28, 31], "began": [], "begin": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 11, 12, 13, 14, 15, 22, 24, 28, 29, 31, 32, 33, 34, 35, 36, 37, 38, 39], "behav": [1, 6, 13, 33, 35, 36, 39], "behavior": [0, 1, 13, 31, 33, 34, 39], "behaviour": [12, 34, 37, 38, 39], "behind": [0, 1, 6, 8, 13, 31, 33, 39], "being": [0, 1, 2, 3, 4, 5, 7, 8, 10, 11, 12, 13, 17, 20, 28, 31, 32, 33, 34, 36, 37, 38, 39], "believ": [9, 24], "belong": [7, 8, 9, 13, 14, 33, 36, 37, 39], "below": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 15, 18, 21, 22, 24, 25, 26, 28, 31, 32, 33, 34, 35, 36, 37, 38, 39], "benchmark": 10, "benefici": [1, 13, 39], "benefit": [0, 1, 4, 11, 13, 23, 31, 33, 34, 39], "bengio": [1, 26, 30, 31, 32, 34], "benign": [1, 7, 37], "benno": [38, 39], "berner": [38, 39], "besid": [4, 5, 33], "bessel": [5, 32, 35], "best": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 12, 13, 15, 16, 18, 21, 26, 29, 31, 32, 33, 34, 35, 36, 37, 39], "beta": [1, 3, 10, 11, 13, 16, 17, 19, 31, 32, 33, 39], "beta1": [], "beta2": [], "beta_": [3, 13, 17, 32], "beta_0": [1, 3, 13, 32, 39], "beta_1": [1, 3, 10, 13, 32, 34, 39], "beta_1m_": 34, "beta_1x_i": 13, "beta_2": [3, 13, 34], "beta_2v_": 34, "beta_3": 3, "beta_i": [3, 34], "beta_j": [13, 32], "beta_k": 13, "beta_linreg": 13, "beta_m": 10, "beta_mg_m": 10, "beta_n": 3, "better": [0, 1, 2, 3, 4, 6, 9, 10, 11, 12, 13, 19, 20, 22, 31, 32, 34, 35, 39], "between": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 11, 12, 13, 14, 15, 16, 17, 18, 19, 22, 25, 26, 28, 31, 32, 33, 34, 35, 36, 37, 38, 39], "beyond": [0, 1, 5, 6, 8, 13, 31, 32, 33, 34, 39], "bf": [13, 14, 24, 28, 33], "bf5400": [], "bg": 31, "bgd": [13, 34], "bia": [0, 1, 2, 3, 5, 8, 9, 10, 12, 13, 20, 21, 22, 26, 31, 32, 33, 37, 38, 39], "bias": [1, 2, 3, 5, 6, 9, 12, 19, 21, 22, 26, 34, 35, 37], "bib": [], "bibliographi": [25, 26], "bibtex": [], "big": [0, 1, 2, 5, 6, 14, 19, 34, 35, 39], "bigger": [1, 6, 32, 39], "bigr": [12, 37], "bike": 9, "bilbo": 31, "billion": [3, 12, 23, 34, 37, 38], "bin": [7, 28, 37], "binari": [0, 3, 5, 7, 9, 10, 12, 26, 31, 36, 37], "binary_cross_entropi": [36, 37], "binary_result": [36, 37], "binarycrossentropi": 4, "bind": 0, "binomi": [23, 28, 31], "binsboot": [6, 35], "bioinformat": 0, "biolog": [1, 12, 37, 38, 39], "bios1100": [23, 31], "bird": [0, 3], "birth": 31, "bishop": [30, 31], "bit": [1, 4, 19, 21, 24, 28, 31, 39], "bitwis": 28, "bivari": 2, "bk": [13, 34], "bla": [24, 31], "black": [8, 9, 14], "blame": [], "block": [6, 10, 23, 24, 28, 31, 35, 36], "blockquot": [], "blog": [26, 31], "blogpost": 4, "blue": [0, 3], "bm": [], "bmatrix": [0, 1, 3, 5, 7, 8, 11, 13, 24, 31, 32, 33, 34, 36, 37, 38, 39], "bmi": [1, 39], "bodi": [0, 1, 4, 12, 37, 38, 39], "bold": 1, "boldfac": [0, 5, 16, 32, 33], "boldsymbol": [0, 1, 2, 3, 5, 6, 7, 8, 10, 11, 13, 14, 16, 17, 19, 25, 31, 33, 34, 36, 37, 38, 39], "boltzmann": [12, 23, 31, 37, 38], "book": [17, 25, 26, 30, 31, 32, 35, 36], "book1": 30, "bool": [], "boolean": [4, 17], "boost": [1, 9, 23, 31, 39], "boostrap": 10, "bootstrap": [1, 13, 19, 23, 25, 31, 34, 39], "born": 34, "borrow": 31, "boston_dataset": [], "bot": 8, "both": [0, 1, 4, 5, 6, 8, 9, 10, 13, 14, 15, 16, 17, 19, 23, 24, 25, 26, 28, 29, 31, 32, 33, 34, 35, 36, 37, 39], "bottl": [7, 36, 37], "bottou": 34, "bound": [8, 12, 34, 37, 38, 39], "boundari": [2, 4, 8, 11, 12], "bousquet": 34, "bower": [], "box": [4, 9, 21, 22], "boyd": [8, 13, 33], "bracket": [4, 28], "brain": [1, 7, 12, 36, 37, 38, 39], "branch": [9, 31], "break": [0, 4, 6, 11, 14, 31, 34], "breast": [5, 7, 11, 37], "breviti": 13, "brew": [0, 23, 25, 31], "brg": 8, "brian": [], "brief": [25, 26, 32], "briefli": [0, 16, 19, 31, 35], "bring": [0, 5, 6, 10, 26, 32, 34], "britt": [29, 31], "broad": 0, "broadcast": 21, "broadli": 31, "brought": [13, 23, 31], "brownle": 4, "browser": [15, 31], "brute": [3, 5, 11, 32, 38], "bsd": [], "budget": 34, "buffer_s": 4, "bug": [], "bugfix": [], "bui": 4, "build": [0, 4, 5, 6, 10, 16, 22, 24, 28, 31, 35, 36, 37, 38], "built": [1, 3, 4, 6, 35, 36, 39], "bunch": 11, "bundl": [], "busi": [], "bxe2t": [37, 38, 39], "byte": [24, 31], "c": [0, 1, 2, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 19, 20, 21, 22, 23, 24, 27, 28, 29, 30, 32, 33, 34, 35, 36, 37, 38, 39], "c1": [8, 11], "c2": [8, 11], "c4a2f5": [], "c5e478": [], "c9d1d9": [], "c_": [8, 9, 10, 13, 28, 33, 34], "c_0": 28, "c_1": [12, 37], "c_2": [12, 37], "c_3": [12, 37], "c_4": [12, 37], "c_i": [12, 13, 34, 37], "c_k": 28, "ca": [1, 31], "caab6d": [], "cach": 10, "cal": [0, 8, 10, 12, 13, 33, 34, 38, 39], "calcul": [0, 1, 2, 4, 5, 6, 8, 9, 10, 11, 12, 13, 14, 16, 19, 22, 24, 26, 28, 31, 34, 35, 36, 37, 38, 39], "california": [25, 26], "call": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 18, 19, 21, 23, 24, 25, 26, 28, 29, 31, 32, 33, 34, 35, 36, 37, 38, 39], "callabl": 39, "calor": [0, 32], "caltech": [], "cambridg": [13, 30, 33, 38, 39], "can": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 28, 29, 30, 32, 33, 37, 39], "cancel": [0, 13, 31, 32], "cancer": [5, 10, 37], "cancerpd": [7, 37], "candid": [8, 9, 10, 34], "cannot": [0, 1, 4, 5, 6, 7, 8, 9, 25, 28, 32, 33, 34, 37, 39], "canopi": [0, 23, 25, 31], "canva": [15, 16, 19, 20, 25, 26, 31], "cap": 5, "capabl": [0, 1, 8, 13, 23, 31], "capac": [2, 29], "capita": [], "caption": [20, 25, 26], "captur": [4, 11, 12, 31, 37, 38], "car": [3, 4], "card": [0, 7, 31, 36, 37], "cardin": [1, 39], "care": [11, 15, 19, 22, 34], "carefulli": [13, 34], "carlo": [0, 6, 23, 28, 30, 31, 35, 36], "carri": [2, 6, 7, 25, 35, 36, 37], "cart": 10, "case": [0, 1, 2, 3, 4, 5, 6, 7, 11, 12, 13, 14, 15, 16, 23, 24, 25, 26, 31, 35, 38, 39], "casella": 30, "cast": [1, 39], "cat": [3, 4], "catch": 0, "categor": [0, 1, 3, 9, 11, 31, 36, 37, 39], "categori": [0, 1, 3, 7, 10, 12, 14, 31, 36, 37, 38, 39], "categorical_cross_entropi": [36, 37], "categorical_crossentropi": [1, 3, 39], "caus": [0, 5, 6, 28, 31, 32, 33, 34, 35, 36], "causal": 0, "causat": [0, 31], "cax": 1, "cb": [6, 31], "cbar": 1, "cc": [0, 1, 5, 13, 31, 32, 33, 34, 38, 39], "cc398b": [], "ccbb44": [], "ccc": [5, 12, 33, 37], "cdf": 28, "cdot": [0, 2, 6, 12, 13, 14, 24, 28, 31, 33, 34, 35, 37], "celebr": [13, 33], "cell": [4, 21, 22], "center": [0, 1, 6, 7, 8, 9, 11, 14, 18, 25, 28, 31, 32, 34, 35, 36, 37, 39], "central": [0, 3, 5, 6, 8, 16, 20, 24, 26, 31, 32, 38, 39], "centroid": [14, 28], "centroid_differ": 14, "centuri": 3, "certain": [0, 3, 6, 7, 9, 21, 28, 31, 32, 35, 36, 37], "certainti": 35, "cf": [], "cf222e": [], "cffi": [], "cg": 13, "cha": [], "chain": [0, 1, 13, 22, 23, 28, 31], "challeng": [15, 38], "chanc": [1, 5, 13, 28, 34, 39], "chang": [0, 1, 2, 3, 4, 5, 6, 8, 9, 11, 12, 13, 14, 15, 16, 19, 21, 22, 24, 25, 26, 28, 31, 32, 33, 34, 35, 36, 37, 38, 39], "changelog": [], "channel": 3, "chap4": [38, 39], "chapter": [0, 6, 10, 11, 16, 17, 19, 24, 25, 26, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39], "chapter3": [0, 25], "charact": [0, 3, 5, 31, 32, 33], "character": [8, 9, 10, 12, 28, 37, 39], "characterist": [0, 1, 3, 10, 13, 31, 39], "charg": [0, 31], "charl": [], "charset": [], "chase": 4, "chatgpt": [15, 25, 26], "chd": [7, 36], "chddata": [7, 36], "cheap": [5, 32, 33, 34], "cheaper": [1, 13, 34, 39], "check": [1, 3, 4, 5, 11, 13, 15, 16, 19, 21, 22, 24, 31, 34, 36, 37, 39], "checkmark": 3, "checkpoint": 4, "checkpoint_dir": 4, "checkpoint_prefix": 4, "chen": 10, "cheng": 32, "chiaramont": 2, "childcar": 16, "children": 16, "choic": [0, 1, 2, 3, 4, 6, 9, 12, 13, 14, 20, 24, 31, 32, 33, 34, 35, 36, 37], "choleski": [5, 24, 32, 33], "choos": [2, 3, 6, 9, 10, 11, 13, 14, 15, 18, 19, 21, 25, 26, 33, 35, 36, 37], "chosen": [0, 1, 2, 6, 8, 9, 10, 13, 16, 28, 31, 33, 34, 35, 36, 39], "chosen_datapoint": [1, 39], "christian": 30, "christoph": [30, 31], "chunk": 34, "cifar": 3, "cifar10": 3, "circ": [1, 12, 34, 38, 39], "circl": [0, 8, 12, 32, 34, 37, 38], "circuit": 3, "circumfer": 9, "circumv": [1, 5, 13, 32, 33, 34, 39], "citat": [], "cite": [20, 25, 26], "ckpt": 4, "cl": [36, 37], "claim": [], "clarifi": 21, "clariti": 28, "class": [0, 1, 3, 4, 6, 7, 8, 9, 11, 12, 13, 21, 22, 28, 31, 35, 39], "class0": [36, 37], "class1": [36, 37], "class_nam": [3, 9], "class_to_index": [36, 37], "class_val": 9, "class_valu": 9, "classic": [7, 9, 13, 26, 37], "classif": [0, 3, 5, 6, 7, 8, 11, 12, 21, 23, 25, 30, 31, 32, 35], "classifi": [0, 1, 4, 7, 9, 10, 11, 26, 31, 37, 39], "classificaton": [1, 39], "classifii": 10, "claus": [], "clean": [1, 39], "clear": [1, 5, 10, 12, 13, 34, 39], "clearli": [0, 3, 5, 6, 7, 8, 28, 32, 33, 35, 36, 37], "clever": [1, 10, 39], "clf": [0, 6, 8, 9, 10, 31, 32], "clf3": 0, "clf_lasso": 6, "clf_ridg": 6, "cli": 15, "click": [], "clip": [3, 28, 34, 36, 37], "clock": 34, "clone": [15, 29], "close": [0, 1, 2, 4, 6, 8, 9, 11, 12, 13, 14, 18, 28, 30, 31, 33, 34, 35, 37, 38, 39], "closer": [3, 5, 13, 32, 33, 34], "closest": [8, 11, 13, 14], "closur": [23, 31], "cloud": [23, 31], "cluster": [0, 1, 4, 6, 11, 23, 31, 35, 36, 37, 39], "cluster_label": 14, "cm": [1, 2, 3, 6, 8, 13, 33, 34, 39], "cmap": [0, 1, 2, 3, 4, 6, 8, 9, 10, 31, 39], "cmap_arg": 6, "cmd": [9, 15], "cn_": 28, "cnn": [12, 37, 38], "cnn_kera": 3, "cntk": [23, 31], "co": [0, 2, 3, 6, 9, 13, 31, 35, 36], "code": [0, 3, 4, 6, 7, 8, 18, 19, 21, 22, 23, 24, 28, 30], "codebas": 39, "codec": [], "coef": [0, 31], "coef0": 8, "coef_": [0, 5, 6, 8, 9, 13, 16, 31, 32, 33, 34], "coeff": 5, "coeffici": [0, 3, 5, 6, 7, 8, 9, 13, 18, 24, 31, 32, 34, 35, 36, 37], "coerc": [0, 6, 31, 35, 36], "coin": [10, 28], "coin_toss": 10, "col": [0, 11, 31, 32], "colab": [21, 22, 23, 31], "cold": 9, "colinear": [], "collabor": [20, 25, 26], "collaps": 8, "collect": [2, 6, 10, 11, 17, 23, 28, 30, 31, 35, 36, 38], "collinear": [5, 32, 33], "color": [0, 3, 4, 6, 8, 9, 10, 28, 34], "color_channel": 3, "color_cod": 6, "colorbar": [1, 6, 20], "coloumn": 39, "colsample_bytre": 10, "colsaobject": 10, "column": [0, 1, 2, 5, 6, 7, 8, 9, 11, 12, 16, 17, 18, 19, 24, 31, 32, 33, 34, 35, 36, 37, 38, 39], "columntransform": 9, "com": [4, 6, 15, 16, 19, 20, 21, 22, 23, 25, 26, 30, 31, 33, 34, 35, 36, 37, 38, 39], "combin": [1, 2, 5, 6, 7, 10, 15, 18, 22, 28, 35, 36, 39], "come": [0, 1, 3, 4, 5, 12, 13, 14, 15, 26, 31, 32, 33, 34, 37, 38, 39], "comfort": [], "command": [0, 1, 15, 39], "comment": [0, 4, 5, 6, 20, 25, 26], "commerci": [0, 23, 25, 31], "commit": 15, "commod": [0, 31], "common": [0, 1, 3, 5, 6, 7, 9, 11, 13, 14, 16, 25, 26, 28, 31, 32, 33, 34, 35, 36, 37, 38, 39], "commonli": [0, 1, 4, 6, 7, 9, 13, 14, 32, 34, 35, 36, 37, 39], "commonmark": [], "commun": [0, 12, 15, 25, 37, 38], "commut": 3, "commutatitav": 3, "compact": [0, 1, 3, 5, 6, 7, 9, 11, 12, 13, 14, 21, 31, 32, 35], "compair": 0, "compar": [0, 3, 4, 5, 6, 11, 13, 18, 24, 25, 26, 31, 32, 33, 34, 35, 36, 38], "comparison": [2, 4, 13, 26], "compat": [7, 36, 37], "compens": 34, "compet": 0, "competit": 10, "compil": [0, 1, 3, 4, 13, 23, 24, 31, 39], "compl": 21, "complet": [0, 2, 3, 4, 9, 12, 15, 16, 17, 18, 19, 20, 21, 22, 31, 37], "completenn": [12, 37], "complex": [1, 5, 8, 9, 11, 12, 13, 16, 19, 31, 33, 34, 35, 36, 39], "complianc": [], "complic": [0, 1, 9, 13, 25, 26, 31, 33, 34, 35, 36, 39], "compoment": 32, "compon": [0, 1, 3, 4, 5, 6, 7, 9, 14, 16, 23, 31, 32, 33, 35, 37, 38, 39], "components_": 11, "compos": [9, 12, 13, 14, 23, 31, 37, 38], "compphys": [0, 6, 16, 20, 23, 25, 26, 27, 29, 30, 31, 32, 33, 36, 37, 39], "compress": [0, 31, 32], "compris": 6, "compromis": [5, 32, 33], "compulsori": [23, 31], "comput": [0, 1, 2, 3, 4, 5, 6, 7, 8, 10, 11, 12, 13, 15, 16, 17, 18, 21, 22, 23, 24, 25, 27, 28, 30, 31, 32, 33, 35, 36, 37, 38, 39], "computation": [0, 3, 6, 9, 13, 28, 31, 33, 34, 38], "computationalscienceuio": 31, "compute_gradi": 22, "computerlab": [25, 26], "con": 26, "concaten": [2, 4, 6, 14, 36, 37], "concav": [1, 13, 32, 33], "concentr": 10, "concept": [0, 2, 23, 31, 32], "conceptu": [12, 13, 33, 37, 38], "concern": [0, 1, 4, 7, 31, 33, 36, 37, 39], "concic": 31, "conclud": [0, 5, 13, 34], "conclus": [1, 39], "cond": 2, "conda": [0, 1, 23, 25, 31, 39], "condis": 32, "condit": [0, 2, 4, 5, 6, 8, 9, 11, 13, 28, 31, 32, 34, 35], "conduct": 23, "condwav": 2, "confid": [0, 5, 6, 7, 8, 19, 31, 32, 36, 37], "configur": 3, "confirm": [5, 12, 21, 37], "conform": [], "confus": [5, 6, 7, 10, 24, 32, 35], "confusion_matrix": 9, "congruenti": 28, "conjug": [4, 8], "conjugaci": 13, "conjunct": 3, "connect": [0, 1, 3, 4, 9, 11, 12, 13, 24, 31, 32, 33, 37, 38, 39], "consensu": 34, "consequ": [5, 6, 8, 10, 12, 13, 32, 33, 34, 35], "consequenti": [], "conserv": [5, 14, 32, 33], "consid": [0, 1, 2, 3, 5, 6, 7, 8, 9, 10, 12, 13, 16, 19, 24, 25, 26, 28, 31, 32, 33, 34, 35, 36, 37, 38, 39], "consider": [0, 1, 5, 13, 31, 32, 33, 35], "consist": [1, 2, 3, 4, 6, 12, 13, 25, 26, 28, 32, 33, 35, 36, 37, 38, 39], "consol": [], "const": [], "constant": [0, 2, 4, 5, 6, 8, 12, 13, 16, 18, 28, 31, 32, 33, 34, 37, 38, 39], "constitu": [0, 31], "constitut": [2, 6, 35, 36], "constrain": [1, 3, 5, 7, 11, 33, 36, 39], "constraint": [5, 6, 8, 13, 32, 33, 35], "construct": [0, 1, 2, 3, 5, 6, 7, 8, 9, 10, 11, 24, 28, 31, 32, 35, 37], "constructor": [], "consult": 26, "consum": 34, "contact": [0, 31], "contain": [0, 2, 3, 4, 5, 6, 7, 8, 9, 11, 12, 13, 15, 18, 19, 21, 24, 25, 26, 28, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39], "contemporari": 31, "content": [1, 15, 20, 23, 24, 31, 33, 34], "context": [6, 10, 13, 22, 25, 33, 34, 35, 36, 38], "contigu": 24, "contin": 19, "continu": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 12, 13, 19, 24, 25, 26, 28, 31, 32, 33, 34, 35, 36, 38, 39], "contour": [9, 10, 13], "contourf": [8, 9, 10], "contract": [], "contrast": [1, 4, 9, 10, 12, 31, 34, 37, 38, 39], "contribut": [0, 3, 5, 13, 18, 28, 31, 32, 33, 34], "contributor": [0, 25], "control": [0, 1, 3, 9, 13, 15, 23, 31, 39], "conv": [3, 4], "conv2d": [3, 4], "conv2dtranspos": 4, "convei": 31, "conveni": [5, 6, 12, 13, 24, 25, 26, 31, 33, 34, 35, 37], "convent": [12, 32], "converg": [1, 2, 4, 5, 8, 13, 14, 18, 32, 33, 38, 39], "convergencewarn": [], "convers": [20, 34], "convert": [0, 1, 4, 5, 9, 11, 13, 24, 31, 32, 33, 36, 37], "converttomatrix": 4, "convex": [4, 5, 7, 32, 36, 37], "convinc": [13, 33], "convolut": [1, 4, 23, 31, 39], "cool": [4, 9], "coolwarm": 6, "coordin": [5, 12, 14, 32, 33, 34, 37], "coorel": [], "copi": [0, 1, 14, 15, 32, 36, 37, 39], "copyright": [], "core": 10, "corel": 31, "coronari": [7, 36], "corr": [5, 7, 11, 32, 37], "correalt": [11, 23], "correct": [0, 1, 2, 3, 4, 5, 7, 13, 15, 19, 20, 21, 22, 24, 28, 31, 32, 33, 35, 36, 37, 39], "correctli": [1, 2, 6, 7, 10, 18, 19, 21, 22, 25, 26, 35, 36, 39], "correl": [0, 1, 3, 5, 6, 7, 10, 12, 13, 23, 28, 31, 33, 34, 35, 38], "correlation_matrix": [5, 7, 11, 32, 37], "correspond": [0, 3, 5, 6, 8, 9, 11, 12, 23, 24, 25, 26, 28, 31, 32, 33, 35, 37, 38], "cortex": [12, 37, 38], "cosin": [3, 6, 35, 36], "cost": [0, 2, 3, 5, 6, 7, 8, 9, 12, 13, 16, 17, 18, 19, 21, 22, 25, 26, 31], "cost_autograd": 22, "cost_deep_grad": 2, "cost_der": 22, "cost_fun": 22, "cost_func": 39, "cost_func_deriv": 39, "cost_funct": 2, "cost_function_deep": 2, "cost_function_deep_grad": 2, "cost_function_grad": 2, "cost_function_train": 39, "cost_function_v": 39, "cost_grad": [2, 22], "cost_histori": [], "cost_ol": [], "cost_one_lay": 22, "cost_ridg": [], "cost_sum": 2, "cost_two_lay": 22, "costcrossentropi": 39, "costli": 34, "costlogreg": 39, "costol": [13, 34, 39], "could": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 15, 16, 17, 18, 24, 25, 26, 28, 31, 32, 33, 34, 35, 36, 37, 38, 39], "coulomb": [0, 31], "count": [0, 9, 15, 25, 26, 27, 28, 29, 31], "counter": [25, 26], "counteract": 34, "counterpart": 31, "countor": 13, "coupl": [4, 5, 6, 21, 35], "cours": [0, 1, 3, 5, 11, 15, 16, 17, 19, 20, 21, 25, 26, 29, 32, 35, 36, 39], "coursework": 15, "courvil": [26, 30, 31, 32, 34], "cov": [5, 6, 11, 24, 28, 31, 32, 35], "cov_xi": [5, 11, 32], "cov_xx": [5, 11, 32], "cov_yi": [5, 11, 32], "covari": [0, 7, 23, 24, 31, 33, 37], "covariance_matrix": [5, 11, 14], "cover": [0, 5, 23, 25, 26, 29, 30, 32, 33, 35], "covert": [0, 31], "covxi": 28, "covxx": 28, "covxz": 28, "covyi": 28, "covyz": 28, "covzz": 28, "cpu": [1, 39], "cqofi41lfdw": [38, 39], "craft": 3, "crash": 34, "creat": [1, 3, 4, 5, 9, 10, 11, 12, 15, 18, 19, 21, 22, 23, 31, 34, 36, 37, 38, 39], "create_biases_and_weight": [1, 39], "create_convolutional_neural_network_kera": 3, "create_lay": [21, 22], "create_layers_batch": 21, "create_neural_network_kera": [1, 39], "create_x": [5, 11, 39], "creation": [], "credit": [0, 7, 29, 31, 36, 37], "crim": [], "crime": [], "criteria": [0, 4, 9, 10, 14, 28, 31], "criterion": [9, 10, 13, 18, 33, 34, 38], "critic": [6, 25, 32], "critiqu": [25, 26], "cross": [0, 1, 3, 7, 9, 10, 13, 15, 21, 22, 23, 26, 28, 31, 32, 33, 34, 39], "cross_entropi": [4, 21], "cross_val_scor": [6, 35, 36], "cross_valid": [7, 10, 37], "crossvalid": [6, 35, 36], "crucial": [1, 28, 34, 39], "cs231": 3, "csr_matrix": [24, 31], "css": [], "csv": [0, 4, 6, 7, 9, 35, 36, 37], "ctnk": [1, 39], "cube": 38, "cubic": 0, "culprit": [], "cumbersom": [5, 35], "cumprod": [], "cumsum": [10, 11, 31], "cumul": [7, 10, 28, 34], "cumulative_heads_ratio": 10, "cup": 5, "current": [1, 2, 3, 4, 13, 14, 15, 16, 30, 33, 34, 36, 37, 39], "curs": [0, 32], "curv": [6, 7, 10, 12, 25, 36, 37, 39], "curvatur": [13, 33, 34], "custom": [6, 14], "custom_cmap": [9, 10], "custom_cmap2": [9, 10], "custom_lin": [], "cutpoint": 9, "cv": [6, 7, 10, 35, 36, 37], "cvxbook": [13, 33], "cvxopt": [5, 8, 32], "cybenko": 38, "cycl": [1, 12, 37, 38, 39], "cycler": [], "d": [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 13, 14, 15, 16, 17, 19, 20, 21, 22, 24, 28, 29, 31, 32, 33, 34, 35, 36, 37, 38, 39], "d1": [], "d166a3": [], "d2": [], "d2_g_t": 2, "d2a8ff": [], "d4d0ab": [], "d71835": [], "d9dee3": [], "d_f": [13, 33], "d_g_t": 2, "d_net_out": 2, "da": [3, 22, 38], "da_1": 22, "dagger": [5, 24, 32, 33], "dai": [1, 9, 23, 39], "damag": [], "damp": 3, "darget": 9, "darkr": 28, "dat": [0, 31], "dat_id": [0, 6, 7, 9, 31, 35, 36], "data": [2, 4, 5, 8, 10, 12, 13, 14, 16, 19, 20, 22, 24, 25, 26, 30, 33, 34, 35], "data1": 14, "data2": 14, "data3": 14, "data4": 14, "data_id": [0, 6, 7, 9, 31, 35, 36], "data_indic": [1, 39], "data_panda": 31, "data_path": [0, 6, 7, 9, 31, 35, 36], "databas": [1, 39], "datafil": [0, 6, 7, 9, 31, 35, 36], "datafram": [0, 4, 5, 7, 9, 11, 31, 32, 37], "datapoint": [1, 5, 6, 7, 11, 13, 16, 33, 34, 35, 36, 39], "datasci": [15, 16, 19], "dataset": [0, 4, 6, 7, 8, 9, 10, 11, 13, 14, 16, 21, 22, 25, 26, 31, 33, 34, 35, 36, 37], "datatyp": 4, "date": [15, 18, 21, 22, 25, 26, 31, 32, 33, 34, 35, 36, 37, 38, 39], "daughter": 10, "davi": [], "david": 30, "davison": [35, 36], "db": [22, 38], "db_1": 22, "dbb7ff": [], "dbh": [1, 39], "dbo": [1, 39], "dc": 22, "dc5e85cd93c3": 26, "dc_da": 22, "dc_da1": 22, "dc_da2": 22, "dc_db": 22, "dc_db1": 22, "dc_db2": 22, "dc_dw": 22, "dc_dw1": 22, "dc_dw2": 22, "dc_dz": 22, "dc_dz1": 22, "dc_dz2": 22, "dcc6e0": [], "dcomposit": 24, "ddot": 2, "de": 34, "dead": [1, 39], "deadlin": [15, 20, 21, 22], "deal": [0, 1, 3, 5, 6, 8, 11, 13, 14, 19, 24, 28, 31, 32, 33, 34, 38, 39], "dealt": 0, "debt": [7, 36, 37], "debug": [0, 5, 6, 32, 33, 34, 35, 36, 39], "debugg": [], "decad": [0, 3, 34], "decai": [0, 13, 28, 31], "decemb": [29, 31], "decent": 10, "decid": [0, 2, 3, 5, 6, 9, 18, 32, 33, 34, 35, 36, 39], "decim": [0, 31, 39], "decis": [0, 1, 8, 11, 23, 30, 31, 39], "decision_funct": 8, "decision_tre": 9, "decisiontreeclassifi": [9, 10], "decisiontreeregressor": [0, 9, 10], "declar": [0, 4, 20, 24, 31], "declare_namespac": [], "decompos": [5, 6, 24, 32, 33, 38], "decomposit": [0, 6, 12, 31, 37, 38], "decompost": [5, 32, 33], "deconvolut": 3, "decorrel": [10, 13, 34], "decreas": [1, 2, 4, 5, 6, 10, 11, 13, 19, 33, 34, 35, 36, 39], "dedic": 20, "deduc": [0, 31], "deep": [3, 7, 12, 13, 23, 26, 30, 32, 33], "deep_neural_network": 2, "deep_param": 2, "deep_tree_clf": [9, 10], "deep_tree_clf1": 9, "deep_tree_clf2": 9, "deepcopi": 39, "deepen": [5, 23, 31], "deeper": [0, 3, 4, 31], "deeplearningbook": [26, 30, 31, 33, 34], "deer": 3, "def": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 13, 14, 16, 17, 21, 22, 28, 31, 32, 33, 34, 35, 36, 37, 38, 39], "def_covari": 28, "default": [0, 1, 2, 4, 6, 7, 24, 26, 31, 32, 36, 37, 39], "default_tim": 4, "defect": [5, 32, 33], "defici": [5, 32, 33], "defin": [0, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 16, 17, 18, 19, 21, 22, 24, 25, 28, 32, 33, 34, 35, 36, 37], "definit": [1, 2, 5, 6, 7, 8, 10, 11, 12, 13, 24, 28, 32, 33, 34, 35, 36, 37], "defint": 28, "defualt": 39, "degre": [3, 5, 6, 8, 9, 10, 11, 15, 16, 19, 20, 25, 28, 31, 33, 34, 35, 36], "deisenroth": 32, "del": 1, "delet": [6, 15], "delimit": 4, "deliv": [15, 25, 26, 27, 31], "delta": [0, 2, 3, 6, 8, 12, 13, 14, 31, 34, 38, 39], "delta_": [1, 24, 38, 39], "delta_0": [3, 38], "delta_1": [3, 38, 39], "delta_2": [3, 38, 39], "delta_2a_1": [38, 39], "delta_3": 3, "delta_4": 3, "delta_5": 3, "delta_h": [0, 1, 31, 39], "delta_i": [38, 39], "delta_j": [3, 12, 38, 39], "delta_k": [12, 38, 39], "delta_l": [1, 3, 39], "delta_matrix": 39, "delta_momentum": [13, 34], "delta_n": [0, 3, 31], "delug": 23, "delv": 0, "demand": [13, 33], "demonstr": [0, 3, 5, 6, 7, 11, 12, 19, 23, 31, 32, 33, 34, 35, 36, 37, 39], "demystifi": [37, 38, 39], "den": 4, "denomin": [1, 5, 34, 39], "denot": [1, 2, 6, 7, 13, 28, 33, 34, 36, 37, 39], "dens": [1, 3, 4, 39], "densiti": [0, 2, 6, 28, 35, 36], "depart": [29, 31, 32, 33, 34, 35, 36, 37, 38, 39], "depend": [0, 1, 2, 4, 5, 6, 7, 8, 11, 12, 13, 15, 16, 23, 24, 25, 28, 31, 32, 33, 34, 36, 37, 38, 39], "depict": 28, "deploy": [0, 23, 25, 31], "depth": [0, 3, 9, 10, 24, 35, 39], "der": [], "deriv": [0, 1, 2, 6, 7, 8, 10, 11, 13, 18, 22, 23, 25, 26, 31, 36, 37], "derivati": 13, "derivative_fn": 13, "derivb1": [38, 39], "derivb2": [38, 39], "derivw1": [38, 39], "derivw2": [38, 39], "descend": [5, 9, 11, 32, 33], "descent": [0, 1, 3, 7, 8, 12, 22, 26, 31, 32, 36, 38, 39], "describ": [0, 2, 4, 5, 6, 8, 10, 11, 12, 13, 19, 20, 24, 25, 26, 31, 34, 35, 37, 38], "descript": [0, 8, 9, 20, 25, 26, 31, 39], "design": [0, 1, 3, 4, 5, 6, 7, 10, 11, 12, 13, 17, 18, 25, 26, 31, 33, 34, 35, 36, 37, 38, 39], "designmatrix": [0, 31], "desir": [0, 2, 4, 5, 13, 14, 31, 32, 33, 34, 39], "desktop": 15, "despit": [1, 12, 34, 37, 39], "destroi": 24, "det": [5, 24, 32, 33], "detail": [0, 6, 11, 13, 14, 18, 21, 22, 24, 25, 32, 33, 34, 39], "detect": [3, 8, 12, 37, 38], "determin": [0, 2, 3, 4, 5, 6, 8, 9, 10, 11, 12, 13, 18, 24, 28, 31, 32, 33, 34, 35, 36, 37, 38, 39], "determinist": [7, 13, 28, 33, 34, 36, 38], "deternin": 38, "dev": [1, 25, 26, 39], "develop": [0, 3, 5, 8, 10, 11, 12, 23, 24, 25, 26, 31, 32, 37, 38], "deviat": [0, 1, 2, 4, 5, 6, 17, 18, 19, 25, 28, 31, 32, 34, 35, 36, 39], "devis": [12, 37, 38], "df": [4, 8, 11, 13, 31, 38], "df1": 31, "di": [], "diag": [5, 8, 32, 33, 34], "diagnost": [1, 10, 39], "diagon": [0, 5, 7, 13, 18, 19, 24, 28, 31, 32, 33, 34, 36, 37, 39], "diagonaliz": [5, 32, 33], "diagram": 10, "diagsvd": 6, "dice": [6, 28, 35], "dict": [6, 8, 39], "dictionari": 39, "did": [0, 1, 5, 6, 7, 10, 11, 14, 16, 22, 25, 26, 31, 35, 36, 37, 39], "die": [1, 39], "diff": [2, 38], "diff1": 2, "diff2": 2, "diff_ag": 2, "diffeent": 8, "differ": [0, 1, 2, 3, 4, 5, 6, 7, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 21, 22, 23, 24, 25, 28, 30, 31, 32, 33, 35, 36, 37, 38, 39], "different": [38, 39], "differenti": [0, 3, 16, 21, 22, 23, 24, 26, 31, 32, 33, 37, 39], "difficult": [0, 1, 6, 10, 13, 28, 31, 34, 35, 36, 39], "difficulti": [0, 1, 13, 31, 33, 34, 39], "diffonedim": 2, "digit": [0, 1, 3, 4, 6, 26, 29, 31, 39], "digress": 38, "dilemma": [13, 34], "dilut": [1, 39], "dim": [4, 11, 14, 24, 39], "dimens": [0, 1, 2, 3, 4, 5, 8, 11, 14, 16, 24, 31, 32, 33, 38, 39], "dimension": [0, 4, 5, 6, 9, 11, 13, 14, 19, 23, 24, 25, 26, 31, 32, 33, 34, 35], "dimensionless": [0, 3, 31], "diment": 24, "diminish": 34, "dimnsion": 4, "diod": 3, "direct": [0, 1, 2, 4, 11, 12, 13, 14, 31, 32, 33, 34, 37, 38, 39], "directli": [1, 4, 5, 6, 18, 22, 28, 32, 33, 39], "directori": [], "disadvantag": [0, 31, 34], "disappear": [3, 6, 35], "disc_loss": 4, "disc_tap": 4, "discard": [6, 11, 34, 35, 36], "disciplin": [0, 3, 12, 37, 38], "disclaim": 28, "discontinu": 38, "discord": [21, 31], "discourag": [13, 15, 33], "discov": [0, 31], "discover": 5, "discret": [1, 3, 5, 7, 13, 36, 37, 39], "discrimin": [4, 7, 10, 11, 36, 37], "discriminator_loss": 4, "discriminator_loss_list": 4, "discriminator_model": 4, "discriminator_optim": 4, "discuss": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 16, 18, 19, 20, 23, 24, 25, 26, 28, 30, 31, 32, 33, 34, 35, 36, 38, 39], "diseas": [7, 36, 37], "disguis": [6, 32, 34], "disk": 34, "disord": [1, 7, 36, 37], "dispai": [37, 38], "displai": [0, 1, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 14, 25, 28, 31, 32, 34, 35, 36, 37, 38, 39], "displaystyl": [0, 5, 17, 31, 32, 33, 34], "disregard": [0, 31], "dissimilar": [11, 14], "dist": 14, "distanc": [8, 9, 11, 14, 28], "distance_list": 9, "distinct": [3, 7, 8, 9, 10, 14, 36, 37], "distinctli": 8, "distinguish": [0, 4, 7, 8, 28, 31, 37], "distplot": [], "distribut": [0, 1, 4, 6, 7, 10, 11, 13, 14, 18, 19, 21, 23, 24, 25, 26, 31, 32, 33, 34, 36, 39], "distrubut": [0, 23, 25, 31], "div": [], "dive": [0, 8, 24, 31], "diverg": [1, 13, 33, 34, 39], "divid": [0, 1, 3, 5, 6, 7, 8, 9, 11, 12, 18, 19, 26, 28, 31, 32, 34, 35, 36, 37, 38, 39], "divis": [6, 8, 9, 13, 18, 24, 28, 34, 35, 36, 38, 39], "dl": [], "dm": [], "dna": [7, 36, 37], "dnn": [0, 1, 2, 4, 12, 31, 37, 38, 39], "dnn1": 4, "dnn2_gru2": 4, "dnn_kera": [1, 39], "dnn_model": 1, "dnn_numpi": [1, 39], "dnn_scikit": [0, 1, 31, 39], "do": [0, 2, 3, 4, 5, 6, 8, 9, 10, 11, 12, 13, 14, 15, 16, 18, 19, 20, 21, 22, 24, 25, 26, 31, 32, 33, 35, 36, 38], "doc": [0, 15, 16, 19, 23, 25, 26, 27, 29, 30, 31, 39], "document": [4, 13, 15], "docutil": [], "doe": [0, 1, 2, 3, 4, 5, 6, 8, 10, 11, 12, 13, 15, 16, 17, 18, 19, 21, 22, 24, 25, 26, 28, 31, 34, 35, 36, 38, 39], "doesn": [3, 9, 12, 31, 34, 38, 39], "dog": [1, 3, 4, 39], "dollar": [], "domain": [5, 8, 13, 25, 26, 33, 35], "domcontentload": [], "domin": [0, 31], "don": [0, 1, 3, 5, 6, 8, 11, 13, 15, 16, 21, 23, 25, 26, 31, 32, 34, 39], "done": [0, 2, 3, 4, 5, 6, 9, 10, 11, 13, 16, 20, 22, 24, 25, 31, 32, 33, 34, 35, 36, 38, 39], "dot": [0, 2, 3, 5, 6, 7, 8, 9, 10, 11, 12, 13, 18, 24, 25, 28, 31, 32, 33, 34, 35, 36, 37, 38, 39], "doubl": [3, 4, 16, 24, 31], "doubli": [1, 39], "doubt": [25, 26], "down": [0, 3, 6, 9, 11, 12, 13, 33, 34, 37], "download": [0, 1, 3, 5, 6, 15, 20, 24, 30, 31, 39], "downsampl": 3, "downscal": 26, "dozen": [1, 39], "dq": [6, 35], "draft": 20, "drag": 13, "dragon": [], "dramat": 11, "drastic": 4, "draw": [4, 6, 10, 13, 33, 35, 36], "drawback": [0, 1, 3, 13, 32, 33, 34, 39], "drawn": [1, 4, 6, 7, 11, 28, 31, 35, 36, 37, 39], "drive": [3, 4, 21, 22], "driven": 3, "drop": [0, 1, 5, 6, 11, 13, 28, 31, 32, 33, 35, 39], "dropna": [0, 6, 31, 35, 36], "dropout": 4, "dt": [2, 3, 13, 28, 38], "dtype": [0, 1, 3, 4, 14, 24, 31, 36, 37, 38, 39], "dual": [], "dub": [0, 31], "duboi": [], "due": [1, 2, 5, 6, 8, 10, 12, 13, 18, 29, 31, 32, 33, 34, 35, 36, 37, 38, 39], "dugard": [], "dummi": [], "dure": [0, 1, 3, 4, 8, 9, 11, 20, 23, 25, 26, 31, 34, 35, 36, 37, 39], "dw": 22, "dw_1": 22, "dwell": [], "dwh": [1, 39], "dwo": [1, 39], "dx": [2, 3, 8, 28, 38], "dx_1": 28, "dx_1p": [6, 35], "dx_2p": [6, 35], "dx_mp": [6, 35], "dx_n": 28, "dxp": [6, 35], "dy": [1, 8, 28, 39], "dynam": 4, "dz": [8, 22], "dz_1": 22, "dz_2": 22, "e": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 28, 29, 31, 32, 33, 34, 35, 36, 37, 38, 39], "e1e1e1": [], "e_": [0, 2, 31], "e_z": 21, "each": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 18, 22, 23, 24, 26, 27, 28, 29, 31, 32, 33, 34, 35, 37, 38, 39], "eager": 35, "eapprox": [0, 31], "earli": [1, 13, 34, 39], "earlier": [0, 5, 7, 8, 9, 11, 12, 13, 19, 20, 21, 22, 31, 32, 36, 37, 38, 39], "earthexplor": 6, "eas": [6, 9, 14, 35], "easi": [0, 5, 6, 7, 8, 9, 10, 11, 12, 13, 15, 21, 22, 23, 24, 26, 31, 32, 33, 34, 35, 36, 37, 38, 39], "easier": [5, 6, 8, 9, 13, 15, 20, 21, 22, 25, 26, 28, 31, 32, 33, 35, 36], "easiest": [13, 18, 36, 37], "easili": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 24, 25, 31, 32, 33, 34, 35, 36, 37, 38, 39], "eastern": [29, 31], "ebind": [0, 31], "eblock": 9, "ec8e2c": [], "econom": [], "econometr": 31, "economi": 5, "ecosystem": [23, 31], "ect": 27, "edg": 3, "edgecolor": [6, 35, 36], "edit": [21, 22], "editor": [15, 20], "edu": [13, 25, 26, 33], "educ": [0, 25, 26, 31, 35], "ee6677": [], "eff": 28, "effect": [1, 4, 10, 13, 16, 17, 18, 28, 34, 39], "effic": [1, 39], "effici": [0, 3, 10, 13, 21, 22, 23, 24, 28, 31, 34, 36, 37, 38], "effort": 19, "efron": [6, 35, 36], "egrad": 13, "eig": [5, 11, 13, 24, 28, 31, 32, 33, 34], "eigen": 28, "eigenpair": [5, 11, 32, 33], "eigenvalu": [0, 5, 8, 11, 13, 24, 31, 32, 33, 34], "eigenvector": [5, 11, 13, 32, 33], "eight": [24, 31], "eigval": [24, 28, 31], "eigvalu": [11, 13, 33, 34], "eigvec": [24, 28, 31], "eigvector": [11, 13, 33, 34], "eir": [29, 31], "eispack": [24, 31], "either": [1, 5, 6, 7, 8, 9, 10, 11, 13, 18, 19, 25, 26, 28, 31, 32, 33, 34, 35, 36, 37, 38, 39], "eivind": 29, "eivinsto": 29, "ekstr\u00f8m": 4, "elabor": 28, "elarn": 3, "electr": [0, 3, 12, 31, 37, 38], "electron": 31, "eleg": 11, "element": [1, 2, 3, 4, 5, 6, 7, 8, 11, 12, 13, 19, 20, 21, 23, 24, 25, 26, 30, 32, 34, 35, 36, 37, 38, 39], "elementari": [10, 13, 24, 38], "elementwis": [3, 13], "elementwise_grad": [2, 13, 22, 39], "elessar": 31, "elif": [14, 39], "elim": 24, "elimin": [3, 8], "elin": [29, 31], "ell_": [], "ellipsi": 16, "els": [1, 3, 4, 7, 9, 12, 13, 16, 22, 24, 36, 37, 39], "elu": 1, "elus": [0, 31], "em": [], "email": [20, 21, 27, 29, 31], "emb": [], "embark": 38, "embed": [0, 11, 32], "embodi": [6, 25, 35, 36], "emit": 28, "emner": 30, "emph": 34, "emphas": [0, 10, 23, 31], "emphasi": [0, 23, 30, 31], "empir": [1, 11, 28, 39], "emploi": [0, 1, 5, 6, 11, 13, 26, 28, 31, 32, 33, 35, 39], "employ": 0, "empti": [6, 10, 15, 35, 36, 39], "emul": [12, 37, 38], "en": [23, 25, 30], "enabl": [11, 34, 39], "enbodi": [6, 35], "encod": [0, 3, 5, 9, 11, 14, 31, 32, 33, 36, 37], "encompass": [0, 25, 28], "encount": [0, 1, 5, 7, 13, 15, 21, 25, 28, 31, 32, 33, 34, 36, 37, 39], "encourag": [15, 25, 26], "end": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 20, 22, 24, 28, 29, 31, 32, 33, 34, 35, 36, 37, 38, 39], "endblock": [], "endfor": [], "endif": [], "endors": [], "endpoint": [3, 6], "energi": [0, 4, 6, 35, 36], "enforc": [12, 37, 38], "eng": 30, "engin": [0, 1, 3, 4, 23, 31, 39], "english": [25, 26], "enjoi": 34, "enocurag": [25, 26], "enorm": 3, "enough": [0, 6, 13, 26, 31, 33, 34, 35], "ensembl": [1, 9, 31, 39], "ensur": [0, 1, 2, 3, 5, 6, 11, 13, 18, 28, 32, 33, 34, 35, 36, 38, 39], "entail": 31, "enter": [5, 6, 32, 33, 34], "enthought": [0, 23, 25, 31], "entir": [1, 3, 7, 9, 21, 23, 28, 31, 34, 36, 39], "entireti": [], "entiti": [9, 12, 24, 31], "entri": [0, 5, 8, 11, 12, 24, 31, 32, 34, 35], "entropi": [1, 3, 7, 10, 13, 21, 22, 26, 31, 33, 34, 39], "enumer": [0, 1, 2, 3, 4, 6, 8, 21, 31, 32, 34, 36, 37, 39], "env": 28, "environ": [2, 21, 22, 23, 25, 31], "environemnt": 15, "eo": [0, 6, 35, 36], "eol": 0, "eosfit": 0, "epoch": [0, 1, 3, 4, 12, 13, 21, 31, 34, 36, 37, 39], "eppstein": [], "epsilon": [0, 5, 6, 7, 13, 25, 31, 32, 33, 34, 35, 36, 37, 38], "epsilon_": [0, 31], "epsilon_0": [0, 31], "epsilon_1": [0, 31], "epsilon_2": [0, 31], "epsilon_i": [0, 31, 32], "eq": [3, 13, 14, 24, 28, 33], "eqnarrai": [3, 5, 6, 35], "equal": [0, 1, 2, 3, 4, 5, 6, 8, 9, 11, 12, 13, 14, 16, 18, 24, 25, 26, 28, 31, 32, 33, 34, 35, 36, 38, 39], "equat": [1, 3, 4, 5, 6, 7, 8, 9, 10, 11, 13, 14, 17, 19, 24, 28, 31, 34, 35], "equilibrium": [2, 12, 37, 38], "equiv": [3, 13, 24, 28, 33, 34], "equival": [0, 1, 5, 7, 8, 11, 13, 23, 24, 26, 31, 32, 33, 34, 35, 39], "equivel": [19, 21, 22], "eras": [], "erf": 28, "eriador": 31, "eric": 39, "err": [0, 10], "err_": [6, 35, 36], "err_sqr": 2, "errat": [13, 33, 34], "erron": 2, "error": [1, 2, 4, 5, 6, 7, 9, 11, 12, 13, 15, 16, 17, 18, 19, 21, 23, 24, 25, 26, 28, 34, 37, 38, 39], "error_estimate_corr_tim": 28, "error_hidden": [1, 39], "error_output": [1, 39], "escap": [13, 33, 34], "escapehtml": [], "especi": [1, 3, 9, 12, 13, 15, 18, 25, 26, 34, 37, 38, 39], "essenti": [0, 5, 6, 9, 10, 12, 14, 15, 25, 26, 28, 32, 33, 34, 37, 38, 39], "establish": [0, 6, 10, 11, 16, 25, 26], "estim": [0, 1, 5, 6, 7, 10, 11, 13, 23, 28, 31, 32, 33, 34, 36, 37, 39], "estimated_mse_fold": [6, 35, 36], "estimated_mse_kfold": [6, 35, 36], "estimated_mse_sklearn": [6, 35, 36], "et": [0, 2, 4, 16, 17, 20, 26, 30, 31, 32, 33, 35, 36, 37, 38, 39], "eta": [0, 1, 3, 8, 12, 13, 18, 26, 31, 33, 34, 38, 39], "eta0": [8, 13], "eta_": 13, "eta_j": 34, "eta_t": [13, 34], "eta_v": [0, 1, 3, 31, 39], "etc": [0, 1, 3, 5, 7, 8, 9, 11, 12, 13, 14, 23, 24, 25, 26, 28, 32, 33, 34, 36, 37, 39], "ethic": 23, "etsim": 35, "euclidean": [0, 14, 32, 34], "euler": [], "evalu": [0, 2, 3, 4, 5, 6, 9, 13, 15, 16, 17, 19, 21, 25, 28, 31, 32, 33, 34, 35, 36, 37], "evalut": [13, 25], "even": [0, 1, 3, 4, 5, 6, 8, 9, 10, 11, 12, 13, 14, 22, 23, 24, 26, 28, 31, 32, 33, 34, 35, 36, 37, 38, 39], "evenli": 4, "event": [5, 7, 10, 28, 35, 36], "eventu": [0, 5, 6, 11, 12, 13, 25, 26, 29, 32, 33, 34, 35, 36, 37, 38], "everi": [0, 1, 2, 3, 4, 5, 6, 9, 10, 11, 12, 13, 14, 15, 21, 23, 28, 29, 31, 32, 33, 34, 35, 36, 37, 38, 39], "everyth": [4, 12, 16, 18, 21, 38, 39], "everywher": [4, 13, 33], "evolv": 0, "exact": [0, 5, 11, 12, 13, 24, 28, 31, 32, 34, 38, 39], "exactli": [0, 3, 4, 6, 12, 18, 23, 32, 34, 35, 37, 38, 39], "exam": 31, "examin": [6, 35, 36], "exampl": [0, 5, 11, 12, 13, 15, 16, 18, 20, 23, 24, 25, 26, 28, 30], "exce": [1, 12, 13, 34, 37, 38, 39], "exceed": 34, "excel": [0, 1, 4, 5, 10, 20, 25, 26, 31, 32, 39], "except": [3, 4, 6, 8, 9, 24, 39], "excess": [0, 31], "exchang": 34, "excit": 0, "exclud": [1, 6, 12, 25, 26, 32, 34, 35, 36, 37, 39], "exclus": [0, 1, 3, 6, 28, 31, 35, 36, 39], "execut": [2, 5, 13, 15, 32, 33, 34], "exemplari": [], "exemplifi": [13, 34], "exercic": [29, 31], "exercis": [5, 23, 25, 26, 27, 29, 31, 33, 34, 35, 36, 37, 39], "exercisesweek41": 26, "exercisesweek42": [26, 39], "exhaust": [6, 34, 35, 36], "exhibit": [0, 5, 6, 8, 31, 32, 35], "exist": [0, 1, 2, 3, 5, 6, 7, 8, 9, 13, 19, 24, 25, 26, 31, 33, 34, 35, 36, 39], "exit": [5, 24, 32, 33], "exp": [0, 1, 2, 5, 6, 7, 8, 10, 11, 12, 13, 16, 17, 19, 21, 22, 28, 32, 33, 34, 35, 36, 37, 38, 39], "exp_term": [1, 39], "exp_z": [36, 37], "expand": [5, 7, 11, 13, 33, 36, 37], "expans": [0, 3, 5, 8, 10, 12, 13, 31, 32, 33, 38], "expect": [0, 1, 5, 6, 7, 11, 12, 13, 15, 18, 23, 25, 26, 31, 32, 34, 36, 38, 39], "expectation_value_of_h_wrt_p": 28, "expens": [6, 10, 13, 16, 33, 34], "experi": [0, 1, 6, 8, 13, 15, 23, 25, 31, 32, 33, 34, 35, 36, 39], "experiment": [0, 4, 6, 9, 28, 31, 35, 36], "expert": [1, 9, 39], "explain": [0, 6, 9, 10, 11, 13, 16, 19, 25, 26, 31, 33, 36, 37], "explained_variance_ratio_": 11, "explan": [], "explanatori": [0, 31], "explicit": [0, 3, 6, 13, 24, 25, 31, 32, 33, 34], "explicitli": [0, 4, 21], "explod": [1, 38], "exploit": [0, 3, 12, 13, 31, 34, 37, 38], "explor": [1, 4, 6, 8, 13, 18, 23, 25, 26, 31, 33, 34, 39], "expon": [1, 39], "exponenti": [0, 1, 5, 6, 10, 13, 28, 31, 33, 38, 39], "export": [9, 15, 16, 19, 20, 36, 37], "export_graphviz": 9, "export_text": 9, "exporttext": 9, "expos": 23, "expr": 38, "express": [0, 2, 3, 5, 6, 7, 10, 12, 13, 18, 22, 24, 25, 26, 28, 31, 33, 34, 35], "exptmean": 28, "exptvari": 28, "extend": [0, 2, 7, 11, 13, 23, 31, 34], "extend_path": [], "extens": [0, 12, 15, 23, 26, 31, 37, 38], "extent": [0, 1, 6, 30, 35, 36, 39], "extern": [3, 6, 9], "extra": [1, 3, 5, 15, 29, 31, 32, 33, 39], "extract": [0, 3, 5, 6, 7, 8, 11, 13, 16, 17, 24, 26, 31, 32, 36, 37, 38], "extrapol": [0, 31], "extrem": [0, 1, 4, 5, 6, 7, 8, 9, 13, 15, 16, 24, 32, 33, 34, 36, 39], "extremum": [13, 33], "extrins": 11, "ey": [0, 5, 6, 13, 14, 18, 24, 31, 32, 33, 34], "f": [0, 1, 2, 3, 4, 5, 6, 7, 8, 10, 12, 13, 14, 15, 16, 17, 18, 19, 22, 24, 28, 29, 31, 32, 33, 34, 35, 36, 37, 38, 39], "f1": 13, "f11": [0, 31], "f12": [0, 31], "f13": [0, 31], "f1_grad": 13, "f1d": 13, "f2": 13, "f26196": [], "f2_grad_x1": 13, "f2_grad_x1_analyt": 13, "f2_grad_x2": 13, "f2_grad_x2_analyt": 13, "f2f2f2": [], "f3": 13, "f3_grad": 13, "f3_grad_analyt": 13, "f4": 13, "f4_grad": 13, "f4_grad_analyt": 13, "f5": 13, "f5_grad": 13, "f5a394": [], "f5ab35": [], "f5f5f5": [], "f6": 13, "f6_for": 13, "f6_for_grad": 13, "f6_grad_analyt": 13, "f6_while": 13, "f6_while_grad": 13, "f7": 13, "f78c6c": [], "f7_grad": 13, "f7_grad_analyt": 13, "f8": 13, "f8_grad": 13, "f8f8f2": [], "f9": [0, 13, 31], "f9_altern": 13, "f9_alternative_grad": 13, "f9_grad": 13, "f_": 10, "f_0": [3, 10], "f_1": [10, 13, 33], "f_2": [12, 13, 33, 37], "f_3": [12, 37], "f_d": 28, "f_grad": 13, "f_grad_analyt": 13, "f_i": [0, 6, 12, 16, 35, 36, 37], "f_m": [3, 10], "f_n": 3, "f_vec": 2, "face": [13, 31, 33], "facecolor": [6, 8, 28, 35], "facil": [0, 23], "facilit": [12, 37, 38], "fact": [0, 1, 3, 5, 9, 11, 12, 13, 22, 31, 32, 33, 34, 39], "facto": 34, "factor": [0, 1, 3, 5, 6, 9, 10, 11, 13, 24, 28, 31, 32, 33, 39], "factori": 13, "fad000": [], "fade": 6, "fae4c2": [], "fafab0": [9, 10], "fail": [0, 6, 13, 29, 31, 33, 35, 36, 38], "failur": [7, 36, 37], "fairli": [1, 2, 18, 28, 34, 39], "faisal": [16, 32], "fake": 4, "fake_loss": 4, "fake_output": 4, "fall": [8, 9, 27], "fals": [0, 1, 2, 3, 4, 5, 6, 7, 9, 10, 14, 16, 17, 24, 26, 31, 32, 33, 34, 35, 36, 37, 39], "famili": [0, 7, 8, 28, 32, 34, 36, 37, 38], "familiar": [0, 3, 5, 6, 8, 15, 23, 24, 25, 28, 31, 35, 38], "famou": [6, 12, 39], "far": [0, 3, 4, 5, 6, 8, 11, 12, 13, 14, 16, 20, 21, 22, 31, 32, 33, 34, 37, 38], "fashion": [0, 9, 10, 26, 31, 34], "fashionmnist": 26, "fast": [1, 3, 6, 10, 12, 13, 23, 28, 31, 33, 34, 35, 36, 38, 39], "faster": [1, 11, 13, 21, 34, 39], "fastest": [13, 24, 33], "fatal": [], "favor": [7, 34, 36], "favorit": 28, "fc": 3, "fcfcfc": [], "fdac54": [], "fdf2e2": [], "featur": [0, 1, 3, 5, 6, 7, 8, 10, 11, 12, 13, 15, 17, 18, 19, 21, 23, 26, 28, 31, 33, 34, 35, 36, 37, 38, 39], "feature_nam": [1, 7, 9, 21, 37], "feautur": 9, "fed": [1, 38, 39], "feed": [0, 2, 3, 11, 21, 23, 26, 31], "feed_forward": [1, 21, 22, 39], "feed_forward_all_relu": 21, "feed_forward_batch": 21, "feed_forward_one_lay": 22, "feed_forward_out": [1, 39], "feed_forward_sav": 22, "feed_forward_train": [1, 39], "feed_forward_two_lay": 22, "feedback": [4, 20, 31], "feeddorward": 4, "feedforward": [1, 4, 12, 39], "feel": [0, 5, 6, 11, 13, 15, 16, 18, 21, 22, 23, 25, 26, 29, 31, 38], "feet": [], "fefef": [], "fefeff": [], "felt": [25, 26], "fenc": [], "fernando": [], "fetch": [6, 15, 26], "fetch_openml": 26, "few": [1, 3, 4, 5, 9, 17, 18, 19, 22, 28, 31, 38, 39], "fewer": [0, 9, 11, 19, 31, 34], "ff7b72": [], "ff9492": [], "ffa07a": [], "ffa657": [], "ffb757": [], "ffd700": [], "ffd900": [], "ffd9002e": [], "ffffff": [], "ffnn": [1, 12, 26, 37, 38, 39], "fi": [], "field": [0, 3, 6, 12, 19, 23, 37, 38], "fieldmask": [], "fifteen": 38, "fifth": [0, 6, 31], "fig": [0, 1, 2, 3, 4, 6, 7, 12, 13, 14, 25, 31, 36, 37, 39], "fig_id": [0, 6, 7, 9, 31, 35, 36], "figaxi": 28, "figsiz": [0, 1, 2, 3, 4, 6, 7, 8, 9, 10, 31, 35, 36, 37, 39], "figur": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 12, 13, 14, 16, 23, 25, 26, 31, 32, 33, 34, 35, 36, 37, 38, 39], "figure_id": [0, 6, 7, 9, 31, 35, 36], "figurefil": [0, 6, 7, 9, 31, 35, 36], "file": [0, 4, 5, 6, 7, 9, 15, 20, 21, 22, 25, 26, 31, 35, 36], "file_prefix": 4, "filenam": 31, "fill": [5, 9, 18, 32, 33, 39], "fill_valu": [], "filter": [3, 4], "final": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 13, 14, 18, 20, 21, 22, 25, 26, 27, 28, 29, 31, 33, 35, 36, 37], "financ": 0, "find": [0, 1, 2, 3, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 21, 22, 23, 25, 26, 28, 31, 32, 33, 34, 36, 37, 38, 39], "fine": [0, 14], "finish": [2, 20, 21, 39], "finit": [3, 5, 6, 12, 13, 17, 28, 32, 33, 35, 36, 37, 38], "finnicki": 15, "fire": [], "first": [0, 1, 2, 3, 5, 6, 7, 8, 9, 10, 11, 13, 14, 15, 16, 18, 19, 21, 22, 24, 25, 26, 28, 29, 30, 32, 34, 35, 36, 37], "first_moment": 34, "first_term": 34, "firsteigvector": 11, "fit": [1, 3, 4, 5, 6, 7, 8, 9, 11, 12, 13, 17, 18, 19, 22, 25, 26, 28, 32, 34, 35, 36, 37, 38, 39], "fit_beta": 32, "fit_intercept": [0, 5, 6, 16, 32, 33, 34, 35, 36, 37], "fit_mod": 9, "fit_theta": [6, 34], "fit_transform": [0, 6, 8, 9, 11, 15, 19, 35, 36], "fiti": [0, 31], "five": [0, 9, 31, 32, 38], "fix": [0, 3, 4, 6, 10, 11, 12, 13, 25, 31, 35, 36, 37, 39], "flag": 4, "flat": [12, 13, 33, 34], "flatten": [1, 3, 4, 5, 24, 39], "flavor": [], "flexibl": [1, 6, 8, 10, 12, 26, 31, 34, 35, 36, 37, 39], "flip": [21, 29, 31], "float": [0, 3, 4, 5, 9, 11, 13, 14, 24, 31, 32, 33, 39], "float32": [4, 9, 39], "float64": [4, 24, 31, 37, 38, 39], "floatingpointerror": 39, "floor": 39, "flop": [5, 24, 32, 33], "flow": [1, 4, 12, 37, 38, 39], "flower": 21, "fluctuat": [5, 34], "flush": 39, "fly": 11, "fm": 0, "fmax": 3, "fmesh": 13, "fn": 7, "focu": [0, 3, 4, 5, 6, 15, 23, 25, 26, 30, 31, 32, 33, 34, 35, 36], "focus": [1, 6, 7, 24, 32, 34, 36, 37, 39], "fold": [6, 9, 25], "folder": [0, 4, 6, 15, 20, 25, 26, 31], "follow": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 19, 20, 21, 22, 23, 24, 25, 26, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39], "font": [7, 20, 28, 31, 36], "fontdict": 28, "fontsiz": [1, 6, 8, 9, 10, 28], "fontweight": 1, "footprint": [3, 34], "foral": [8, 32, 38], "forc": [0, 5, 6, 10, 11, 32, 33, 34, 38], "forcast": 4, "forcier": [], "forecast": [4, 12, 37, 38], "forest": [0, 1, 9, 23, 31, 39], "forget": [11, 34], "form": [0, 3, 4, 5, 6, 7, 8, 9, 11, 12, 13, 15, 16, 23, 24, 25, 26, 28, 31, 32, 33, 34, 35, 36, 37, 38, 39], "formal": [3, 4, 14, 18, 28, 38], "format": [0, 1, 3, 4, 6, 7, 8, 9, 10, 11, 20, 23, 28, 30, 35, 36, 37, 39], "format_data": 4, "formatstrformatt": [6, 13, 33, 34], "formatt": [], "formul": [4, 6, 11, 14], "formula": [3, 13, 28, 33, 38], "forth": [4, 12, 22, 37], "fortran": [0, 23, 24, 31], "fortran2003": [23, 31], "fortran2008": [25, 26], "fortran90": 28, "fortun": [0, 11, 32], "forward": [0, 3, 6, 21, 23, 24, 26, 31, 34, 35], "forwardpropag": [38, 39], "found": [1, 2, 4, 5, 6, 12, 13, 19, 20, 21, 22, 25, 31, 32, 34, 35, 36, 37, 38, 39], "foundat": [23, 31], "four": [4, 5, 6, 8, 12, 21, 24, 27, 29, 31, 33, 37, 38, 39], "fourier": [0, 31, 38], "fourierdef1": 3, "fourierdef2": 3, "fourierseriessign": 3, "fourth": [12, 31, 32], "fp": 7, "frac": [0, 1, 2, 3, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 16, 17, 19, 21, 22, 24, 25, 26, 28, 31, 32, 33, 34, 35, 36, 37, 38, 39], "fraction": [9, 36, 37], "frame": [7, 34, 37], "framework": [1, 8, 10, 28, 39], "frank": [5, 11], "frankefunct": [5, 6, 11], "fredli": [21, 29, 31], "free": [0, 6, 11, 13, 15, 16, 18, 21, 22, 23, 24, 25, 26, 28, 29, 30, 31, 38], "freecodecamp": 23, "freedom": [5, 33], "freeli": [0, 25], "freez": 15, "frequenc": [3, 6, 7, 28, 35, 37], "frequent": [0, 8, 9, 13, 33], "frequentist": 23, "fresh": 10, "fridai": [15, 21, 22, 29, 31], "friedman": [6, 19, 25, 30, 31], "friendli": 4, "fro": 25, "frodo": 31, "frog": 3, "from": [0, 1, 2, 3, 4, 6, 7, 8, 9, 11, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 28, 29, 30], "from_cod": 9, "from_logit": [3, 4], "from_tensor_slic": 4, "front": [0, 4, 5, 31, 32, 33], "frustrat": 15, "fulfil": [2, 5, 12, 32, 33, 37, 39], "full": [1, 3, 5, 7, 9, 10, 13, 21, 26, 28, 31, 32, 33, 36], "full_matric": [5, 32, 33], "fulli": [3, 6, 12, 28, 35, 36, 37, 38], "fullnam": [], "fun": [23, 31], "func": [2, 21, 39], "function": [2, 3, 4, 5, 9, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24], "functionali": 11, "fundament": [0, 6, 23, 31, 35, 36], "funtion": 2, "furnish": [], "furthemor": 38, "further": [2, 7, 9, 19, 31, 38], "furthermor": [0, 3, 5, 6, 7, 11, 12, 13, 23, 25, 26, 31, 32, 33, 34, 35, 36, 37, 39], "furthest": 22, "futur": [0, 4, 8, 9, 31], "fy": [15, 21, 25, 26, 27, 29, 30, 31], "fys4155": [25, 26], "fys5419": [30, 31], "fys5429": [30, 31], "f\u00f8470": [29, 31], "g": [0, 1, 2, 3, 4, 6, 8, 9, 10, 11, 13, 15, 18, 19, 28, 31, 32, 33, 34, 35, 36, 37, 39], "g0": 2, "g_": [2, 9, 10, 34], "g_0": 2, "g_1": [2, 10], "g_2": [2, 10], "g_3": 38, "g_analyt": 2, "g_dnn_ag": 2, "g_euler": 2, "g_i": [2, 38], "g_j": 38, "g_m": [3, 10], "g_n": 3, "g_re": 2, "g_t": [2, 34, 39], "g_t_d2t": 2, "g_t_d2x": 2, "g_t_dt": 2, "g_t_hessian": 2, "g_t_hessian_func": 2, "g_t_invers": 39, "g_t_jacobian": 2, "g_t_jacobian_func": 2, "g_trial": 2, "g_trial_deep": 2, "g_vec": 2, "gain": [1, 5, 7, 9, 10, 13, 32, 33, 39], "galleri": [0, 31], "game": 4, "gamge": 31, "gamma": [0, 2, 8, 9, 10, 11, 13, 31, 33], "gamma1": 8, "gamma2": 8, "gamma_": [0, 31], "gamma_0": 10, "gamma_1": 10, "gamma_1x": 10, "gamma_i": [0, 8, 28, 31], "gamma_j": 13, "gamma_k": [13, 33], "gamma_m": 10, "gamma_x": [0, 31], "gap": [8, 34], "gate": [4, 12, 38], "gather": [0, 1, 12, 32, 37, 38, 39], "gaug": [12, 37, 38], "gaussbacksub": 24, "gaussian": [4, 5, 6, 8, 14, 18, 28, 31, 35, 36, 37], "gaussian_point": 14, "gaussian_rbf": 8, "gave": [13, 26, 34], "gavra": 31, "gbc": 31, "gca": [2, 6, 8, 13], "gd": [1, 33, 38, 39], "gd_clf": 10, "gdclassiffiercgain": 10, "gdclassiffierconfus": 10, "gdclassiffierroc": 10, "gdm": 13, "gdregress": 10, "ge": [1, 5, 7, 28, 32, 33, 36, 39], "gen_loss": 4, "gen_tap": 4, "gender": [0, 31], "genener": 4, "gener": [0, 1, 2, 3, 5, 6, 8, 10, 11, 12, 13, 14, 15, 16, 18, 20, 21, 22, 24, 25, 26, 28, 30, 32, 33, 34, 35, 39], "generaliz": [16, 39], "generallay": [12, 37], "generate_and_save_imag": 4, "generate_binary_data": [36, 37], "generate_imag": 4, "generate_latent_point": 4, "generate_multiclass_data": [36, 37], "generate_simple_clustering_dataset": 14, "generated_imag": 4, "generator_loss": 4, "generator_loss_list": 4, "generator_model": 4, "generator_optim": 4, "genom": 23, "geodes": 11, "geoff": 34, "geometr": [0, 13, 31, 34], "geometri": 5, "georg": 30, "geotif": 6, "geq": [2, 5, 8, 9, 13, 32, 33, 34], "gerard": [], "geron": [0, 30, 31], "get": [0, 1, 2, 3, 4, 5, 6, 7, 9, 10, 11, 13, 15, 19, 21, 22, 23, 24, 25, 26, 28, 29, 31, 32, 33, 34, 35, 36, 39], "get_dummi": 9, "get_paramet": 2, "get_split": 9, "get_yaxi": 8, "get_yticklabel": 6, "getmask": [], "gh": 15, "giant": 34, "gibb": [23, 31], "gif": 4, "gini": 10, "gini_index": 9, "ginvers": 13, "git": [0, 15, 23, 31], "gitcdn": [], "giter": [13, 34], "github": [0, 20, 23, 25, 26, 27, 29, 30, 31, 32, 38, 39], "gitignor": 15, "gitlab": [0, 15, 23, 25, 26, 31], "gitta": [38, 39], "give": [0, 1, 2, 3, 5, 6, 7, 8, 9, 10, 12, 13, 14, 18, 19, 23, 25, 26, 28, 31, 32, 33, 34, 35, 36, 37, 38, 39], "given": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 17, 19, 21, 24, 26, 28, 31, 32, 33, 34, 35, 36, 37, 38, 39], "glkfgrjhtlnplbx4": 21, "global": [6, 7, 13, 33, 34, 36, 37], "gloriou": 26, "glorot": 1, "gmail": [], "gnew": 13, "go": [0, 1, 3, 5, 6, 8, 9, 11, 12, 13, 15, 16, 18, 21, 31, 32, 33, 35, 38, 39], "goal": [0, 7, 9, 31, 36, 37], "goe": [0, 1, 2, 5, 6, 13, 14, 15, 19, 24, 31, 32, 33, 34, 35, 39], "goessner": [], "golden": 13, "gone": [5, 32, 33], "gong": [1, 39], "good": [1, 3, 4, 5, 6, 9, 10, 11, 13, 15, 18, 21, 23, 28, 30, 32, 33, 34, 36, 38, 39], "goodfellow": [4, 26, 30, 31, 32, 33, 36, 37, 38, 39], "googl": [1, 4, 21, 22, 23, 31, 39], "got": [1, 6, 21, 22, 25, 26, 39], "gotten": [31, 39], "gov": 6, "govern": 31, "gp": 30, "gpu": [1, 13, 23, 31, 34, 39], "grad": [2, 13, 21, 22, 34, 39], "grad_analyt": 13, "grad_ol": 18, "grad_ridg": 18, "grad_two_lay": 22, "grade": [25, 26, 27], "gradient": [0, 3, 4, 7, 8, 9, 12, 21, 23, 31, 32, 36], "gradient_bia": 39, "gradient_desc": 34, "gradient_func": 21, "gradient_weight": 39, "gradientboostingclassifi": 10, "gradientboostingregressor": 10, "gradients_of_discrimin": 4, "gradients_of_gener": 4, "gradienttap": 4, "gradual": [1, 14, 39], "grai": [4, 6], "granger": [], "grant": [], "graph": [1, 9, 11, 12, 13, 16, 20, 33, 34, 37, 38, 39], "graph_from_dot_data": 9, "graphic": [0, 1, 9, 15, 31, 39], "grasp": 0, "gray_r": [1, 3, 39], "grayscal": 3, "great": [5, 13, 15, 21, 22, 33, 34, 38], "greater": [1, 7, 28, 32, 37, 39], "greatli": 13, "greedi": 9, "green": [0, 3, 9, 28], "gregor": 39, "grei": 4, "grid": [1, 3, 6, 7, 8, 12, 28, 32, 34, 35, 36, 37, 39], "groh": [38, 39], "grossli": [13, 33], "ground": [0, 31], "group": [0, 6, 7, 9, 14, 15, 20, 23, 25, 26, 27, 29, 31, 35], "groupbi": [0, 31], "grow": [1, 3, 9, 10, 34, 39], "growth": [0, 31], "gru": 4, "guarante": [0, 4, 13, 28, 31, 32, 33, 34], "guess": [1, 4, 10, 13, 14, 26, 33, 34, 39], "guestrin": 10, "gui": 15, "guid": [1, 21, 39], "guidelin": [20, 25, 26, 36, 37], "g\u00f6ssner": [], "h": [0, 1, 5, 6, 8, 13, 15, 19, 21, 28, 29, 30, 31, 32, 33, 34, 39], "h1": 2, "h_": [0, 13, 31, 33, 34], "h_0": 34, "h_1": [2, 13, 33], "h_2": [2, 13, 33], "h_m": 10, "h_t": 34, "ha": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 17, 18, 19, 20, 21, 22, 24, 25, 26, 28, 31, 32, 33, 34, 35, 36, 37, 38, 39], "haanen": [29, 31], "habit": [0, 32], "had": [0, 1, 6, 7, 13, 31, 33, 34, 35, 36, 39], "hadamard": [1, 12, 13, 34, 38, 39], "half": [1, 8, 9, 36, 37, 38, 39], "halv": 10, "hand": [0, 1, 2, 3, 5, 11, 12, 13, 23, 24, 25, 26, 28, 29, 30, 31, 32, 33, 34, 36, 37], "handi": [3, 25, 26], "handl": [0, 1, 2, 5, 9, 11, 15, 18, 22, 23, 32, 33, 34, 39], "handle_unknown": 9, "handsid": [12, 38, 39], "handwrit": [12, 37, 38], "handwritten": [1, 5, 39], "happen": [1, 2, 3, 4, 5, 6, 10, 13, 28, 32, 33, 34, 37, 39], "hard": [1, 7, 8, 10, 13, 21, 22, 33, 34, 36, 38, 39], "hardcopi": [23, 31], "harder": [0, 1, 19, 21, 32, 39], "harmon": 3, "hash": 34, "hasn": [], "hassl": [0, 23, 31], "hast": [23, 31], "hasti": [0, 6, 16, 17, 19, 20, 25, 30, 31, 32, 35, 36], "hat": [0, 1, 5, 6, 7, 9, 10, 11, 12, 13, 16, 17, 18, 19, 24, 32, 33, 34, 35, 37, 38], "hauser": [], "have": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 28, 29, 31, 32, 33, 34, 35, 36, 37, 38, 39], "have_sys_un_h": [], "haven": [1, 22, 39], "he": [7, 36, 37], "head": [4, 10, 28], "header": [0, 31], "heads_proba": 10, "health": [0, 32], "hear": [0, 13, 31, 34], "heart": [0, 7, 31, 36], "heatmap": [0, 1, 3, 7, 17, 20, 31, 37, 39], "heavi": 34, "heavili": 0, "heavisid": [1, 39], "height": [1, 3, 6, 32, 39], "held": [13, 34], "help": [0, 1, 4, 12, 13, 15, 16, 25, 26, 31, 34, 35, 37, 38, 39], "helper": [4, 14, 36, 37], "henc": [0, 5, 6, 8, 9, 10, 12, 13, 31, 32, 33, 34, 35, 36, 37], "henrik": [29, 31], "her": [7, 36, 37], "here": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 17, 18, 19, 21, 22, 23, 24, 25, 26, 28, 31, 32, 33, 34, 35, 36, 37, 38, 39], "hereaft": [0, 8, 12, 31], "herebi": [], "hermitian": 24, "hessenberg": 24, "hessian": [0, 2, 5, 13, 36, 37], "heterogen": [9, 10], "hex": [], "hi": [7, 36, 37], "hidden": [1, 3, 4, 12, 21, 26, 37], "hidden_bia": [1, 39], "hidden_bias_gradi": [1, 38, 39], "hidden_deriv": 39, "hidden_func": 39, "hidden_layer_s": [0, 1, 31, 39], "hidden_neuron": 4, "hidden_nodes1": 39, "hidden_nodes2": 39, "hidden_weight": [1, 39], "hidden_weights_gradi": [1, 38, 39], "hierarch": [5, 32, 33], "high": [0, 1, 2, 3, 4, 5, 6, 9, 10, 11, 13, 14, 21, 23, 24, 25, 31, 32, 33, 34, 35, 36, 39], "higher": [0, 1, 3, 5, 6, 8, 13, 18, 25, 31, 32, 33, 34, 35, 36, 39], "highest": [1, 2, 36, 37, 39], "highli": [0, 3, 4, 10, 19, 23, 24, 26, 30, 31, 32, 33, 34], "highlight": [], "highwai": [], "hing": 8, "hint": [13, 15, 16, 21, 22, 26, 32, 33], "hinton": 34, "hip": 23, "hire": 0, "hist": [4, 6, 7, 28, 35, 37], "histogram": [6, 7, 28, 37], "histor": [7, 11, 36], "histori": [3, 4, 12, 15, 34, 37, 38], "hitherto": 5, "hjorth": [29, 31, 32, 33, 34, 35, 36, 37, 38, 39], "hobbi": 28, "hoc": [5, 32, 33], "hoff": 30, "hojjatk": 26, "hold": [1, 3, 6, 13, 14, 33, 34, 35, 39], "holder": [0, 31], "holdgraf_evidence_2014": [], "home": [], "homepag": [25, 26, 31], "homework": [6, 13, 33, 34], "homogen": [1, 3, 9, 10, 13, 34], "honchar": 2, "hopefulli": [0, 11, 15, 19, 28, 31, 34], "horizont": 11, "horlyk": [29, 31], "hornik": 38, "hors": [3, 7, 31, 36, 37], "hot": [1, 9, 36, 37, 39], "hour": [1, 23, 27, 28, 29, 31, 34, 35, 39], "how": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 28, 31, 32, 33, 34, 35, 36, 37, 38, 39], "howev": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 21, 23, 24, 25, 26, 28, 31, 32, 33, 34, 35, 36, 37, 38, 39], "href": [], "hspace": [0, 4, 8, 10, 28, 31, 38, 39], "hstack": [1, 39], "htf": 31, "html": [0, 16, 20, 21, 23, 25, 26, 27, 29, 30, 31, 32, 33, 34, 38, 39], "http": [0, 3, 4, 6, 13, 15, 16, 19, 20, 21, 22, 23, 24, 25, 26, 27, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39], "huang": [0, 31], "huber": [0, 31], "huge": [1, 3, 4, 23, 34, 39], "human": [0, 1, 3, 6, 9, 12, 32, 37, 38, 39], "humid": 9, "hundr": [1, 39], "hungri": [1, 39], "hybrid": 27, "hydrogen": [0, 31], "hyper": 26, "hyperbol": [1, 4, 12], "hyperparam": 8, "hyperparamat": 38, "hyperparamet": [3, 4, 5, 6, 9, 13, 18, 25, 26, 32, 33, 34, 38], "hyperplan": 11, "h\u00f8rlyk": [29, 31], "i": [2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 32, 33, 34, 35, 36, 37, 38], "i0": [0, 31], "i1": [0, 6, 8, 12, 31, 32, 34, 37], "i2": [0, 8, 12, 31, 37], "i3": [0, 12, 31, 37], "i5": [0, 31], "i_": [13, 33, 34], "i_1": [5, 6, 35], "i_2": [5, 6, 35], "i_siz": [21, 22], "i_t": 34, "ian": 30, "iayaan2": 21, "ic": [1, 25, 26, 39], "id": [7, 13, 33, 34, 36], "ida": [29, 31], "idea": [0, 1, 2, 3, 4, 6, 9, 10, 12, 13, 20, 24, 25, 26, 32, 33, 34, 35, 36, 37, 38, 39], "ideal": [0, 2, 6, 8, 13, 28, 31, 34, 35, 36, 37, 39], "idem": [6, 35, 36], "ident": [5, 6, 12, 13, 17, 18, 24, 32, 33, 37, 39], "identical": 35, "identifi": [0, 1, 7, 9, 11, 12, 13, 14, 31, 32, 36, 37, 39], "idx": [36, 37], "ieor": 28, "ifi": 30, "ifs": [23, 31], "ignor": [0, 1, 3, 9, 15, 32, 34, 39], "ii": [24, 28, 39], "iii": [24, 31, 39], "ij": [0, 1, 3, 6, 8, 12, 14, 16, 24, 28, 31, 32, 34, 37, 38, 39], "ik": [0, 24, 31, 32], "iki": [], "ilg3ggewq5u": [38, 39], "ill": 34, "illinoi": [], "illustr": [5, 7, 10, 12, 13, 14, 20, 23, 31, 36, 39], "ilsvrc": 34, "im": 6, "imag": [1, 3, 4, 6, 9, 11, 12, 14, 30, 31, 37, 38, 39], "image_at_epoch_": 4, "image_batch": 4, "image_height": 3, "image_path": [0, 6, 7, 9, 31, 35, 36], "image_width": 3, "imageio": 6, "imagenet": 34, "images_from_seed_imag": 4, "imagin": [1, 39], "immedi": [0, 3, 4, 6, 23, 31, 34], "implement": [0, 2, 3, 4, 5, 6, 8, 9, 10, 11, 12, 13, 14, 19, 20, 21, 22, 25, 26, 28, 31, 32, 33, 34, 36, 37, 38], "impli": [3, 5, 6, 7, 13, 24, 32, 33, 34, 35, 36], "implicit": [3, 34], "implicitli": [11, 28], "import": [0, 1, 2, 3, 4, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 21, 22, 25, 26, 28, 34, 35, 36, 37], "importantli": 3, "importerror": [], "impos": [0, 6, 11, 12, 31, 37, 39], "imposs": [0, 5, 31, 32, 33], "impract": 34, "impress": [0, 12, 31, 37, 38], "improv": [0, 4, 5, 9, 10, 11, 13, 15, 21, 25, 26, 32, 33], "impur": 9, "imread": 6, "imshow": [1, 3, 4, 6, 39], "in3050": [30, 31], "in3310": 31, "in4080": [30, 31], "in4300": [30, 31], "in4310": 30, "in5400": 3, "in5550": 30, "in_out_neuron": 4, "inaccur": [13, 33], "inact": [12, 37, 38, 39], "inadequ": [0, 31], "inappropri": 34, "inch": [6, 32], "incident": [], "includ": [0, 1, 2, 3, 4, 5, 6, 7, 11, 12, 15, 16, 17, 18, 19, 20, 21, 22, 23, 26, 28, 29, 30, 31, 32, 33, 35, 39], "include_bia": [6, 9, 35, 36], "incom": [12, 16, 37, 38], "incorrect": [1, 39], "incoveni": 8, "increas": [0, 1, 3, 4, 5, 6, 9, 12, 13, 19, 25, 28, 31, 32, 34, 35, 36, 37, 38, 39], "increasingli": 28, "increment": 34, "ind": 6, "inde": [0, 2, 4, 5, 6, 13, 31, 32, 33, 38], "indefinit": 4, "independ": [0, 5, 6, 7, 8, 12, 13, 28, 31, 32, 33, 34, 36, 37], "index": [0, 1, 3, 4, 10, 14, 23, 24, 25, 26, 28, 30, 31, 39], "index_col": [0, 31], "indic": [0, 1, 3, 4, 5, 6, 9, 10, 11, 13, 16, 25, 26, 31, 32, 38, 39], "indirect": [], "indispens": [6, 35, 36], "individu": [1, 6, 7, 10, 12, 28, 31, 32, 34, 35, 36, 37, 38, 39], "indu": [], "indx": 24, "indx1": 2, "indx2": 2, "indx3": 2, "ineffici": [3, 13], "inequ": [8, 13], "inequaltii": 33, "inertia": 13, "inexperi": [], "inf": [], "inf1000": [23, 31], "inf1100": [23, 31], "inf1100l": [23, 31], "inf1110": [23, 31], "inf3000": 31, "infeas": [9, 34], "infer": [0, 1, 4, 6, 30, 31, 35, 36, 39], "inferenc": 1, "infil": [0, 6, 7, 9, 31, 35, 36], "infin": [5, 6, 7, 11, 19, 32, 33, 35, 36, 38, 39], "infinit": [3, 34], "infinitesim": 28, "influenc": [6, 10, 18, 35, 36], "influenti": [1, 39], "info": 31, "inform": [0, 1, 3, 4, 6, 9, 11, 12, 13, 14, 24, 25, 26, 30, 31, 33, 34, 35, 36, 37, 38, 39], "inforom": 15, "infrequ": 34, "infti": [3, 6, 13, 28, 33, 35, 38], "ingeni": [13, 33, 34], "ingredi": [0, 9, 31], "inher": [6, 34, 35, 36], "inherit": [24, 31, 34], "init": [], "initi": [0, 1, 2, 6, 10, 13, 14, 18, 24, 26, 28, 31, 33, 34, 35, 36, 37, 38, 39], "inititi": 39, "inject": 14, "inlin": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 24, 28, 31, 32, 33, 34, 35, 36, 37, 39], "inner": [0, 13, 32], "innerhtml": [], "inp": 4, "inplac": 13, "inpput": 38, "input": [0, 1, 3, 4, 5, 6, 7, 8, 12, 13, 14, 16, 25, 26, 28, 31, 32, 33, 34, 35, 36, 37], "input_dim": 1, "input_nod": 39, "input_s": 21, "input_shap": [3, 4], "inputs": 1, "inputs_shuffl": [0, 1, 32, 39], "inquiri": 20, "insert": [3, 5, 6, 8, 10, 28, 32, 33, 35], "insid": [4, 7, 21, 37], "insight": [0, 1, 5, 23, 26, 31, 32, 33, 35, 36, 38], "insist": [6, 13, 32, 34], "inspir": [0, 1, 12, 25, 26, 31, 37, 38, 39], "instabl": 2, "instal": [0, 1, 5, 6, 9, 15, 20, 26, 39], "instanc": [0, 1, 2, 4, 6, 9, 11, 13, 16, 31, 32, 33, 34, 35, 36, 39], "instanti": 10, "instead": [0, 1, 2, 3, 4, 5, 6, 8, 9, 11, 13, 14, 17, 20, 21, 22, 24, 28, 31, 32, 34, 35, 39], "institut": [1, 39], "instruct": [0, 1, 15, 39], "int": [0, 1, 2, 3, 4, 5, 6, 11, 13, 14, 24, 28, 32, 34, 35, 36, 37, 39], "int32": 10, "int_": [3, 6, 28, 35, 38], "int_0": 28, "int_a": 28, "intak": [0, 32], "integ": [1, 2, 13, 14, 24, 28, 31, 36, 37, 39], "integer_vector": [1, 39], "integr": [3, 6, 28, 31, 35], "intellig": [0, 14, 30, 31], "intend": 10, "intens": [1, 18, 39], "intention": 14, "interact": [0, 6, 9, 12, 23, 25, 26, 31, 37, 38], "intercept": [0, 6, 8, 11, 13, 16, 17, 18, 19, 31, 32, 33, 34, 35, 36, 37], "intercept_": [0, 6, 8, 9, 13, 31, 32, 34], "interchang": [5, 12, 24, 37, 38], "interconnect": [1, 39], "interesit": [], "interest": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 12, 19, 23, 25, 26, 28, 31, 32, 33, 35, 36, 37, 38, 39], "interfac": [0, 1, 15, 24, 32, 39], "interior": [0, 9, 31], "intermedi": [24, 32, 34], "intermediari": [21, 22], "intermeti": 22, "intermetidari": 22, "intern": [1, 10, 12, 22, 36, 37, 38, 39], "internation": [], "interpol": [1, 3, 4, 6, 12, 37, 38, 39], "interpr": [5, 32, 33], "interpret": [0, 1, 6, 9, 10, 12, 13, 15, 16, 21, 24, 25, 26, 28, 38, 39], "interrupt": [], "interv": [0, 3, 5, 6, 7, 13, 19, 28, 31, 32, 33, 36, 37], "intial": [13, 33], "intract": [0, 4, 32], "intrins": [3, 11, 24, 28, 31], "intro": [23, 30, 31], "introduc": [0, 1, 5, 6, 8, 10, 12, 24, 25, 28, 31, 33, 34, 35, 37, 38, 39], "introduct": [1, 2, 4, 13, 30, 32, 33, 34, 36, 39], "introductori": [0, 4, 24, 30, 31, 32], "intuit": [0, 5, 6, 8, 12, 13, 25, 31, 34, 35, 36, 37, 38, 39], "inv": [0, 5, 13, 17, 31, 32, 33, 34], "invalid": [], "invalu": [0, 13, 23, 31, 33], "invari": [1, 39], "invd": 5, "inver": [8, 37], "invers": [0, 3, 6, 13, 31, 32, 33, 34], "inverse_transform": 8, "invert": [0, 5, 7, 10, 13, 16, 18, 31, 34, 36, 37], "investig": [], "invh": [13, 34], "invok": 8, "involv": [0, 2, 6, 7, 11, 12, 31, 32, 34, 35, 36, 37, 38, 39], "io": [0, 23, 25, 26, 27, 29, 30, 31, 32, 39], "ion": [], "ip": [0, 8, 28, 31], "ipca": 11, "ipynb": [23, 31], "ipython": [0, 5, 7, 9, 11, 14, 23, 25, 26, 31, 32, 36], "iq": [6, 35], "iri": [8, 9, 21], "irreduc": [6, 35, 36], "irrelev": [5, 32, 33], "irrespect": [0, 31], "irvin": [25, 26], "isaac": [], "isaacmus": [], "iseffici": [], "isn": 5, "isnan": 39, "isnul": [], "isolo": 22, "isomap": 11, "issu": [1, 9, 15, 24, 34, 39], "it_arrai": 13, "item": [0, 13, 31], "items": [24, 31], "iter": [1, 2, 4, 6, 8, 13, 14, 18, 25, 28, 33, 34, 35, 36, 37, 38, 39], "its": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 20, 21, 23, 24, 25, 26, 28, 31, 33, 34, 35, 36, 37, 38, 39], "itself": [5, 6, 12, 25, 26, 28, 31, 32, 35, 38], "iv": 39, "ix": 39, "j": [0, 1, 2, 3, 4, 5, 6, 8, 9, 11, 12, 13, 14, 15, 16, 24, 25, 28, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39], "j1": 24, "j_": 6, "j_41hld6ttu": 35, "j_lasso_sk": 6, "j_ridge_sk": 6, "j_sk": 6, "jackknif": [6, 23, 31, 35, 36], "jacobian": [2, 13, 33], "janko": [], "jason": 4, "javascript": [], "jax": [23, 26, 31, 34, 38], "jeff": [], "jensen": [29, 31, 32, 33, 34, 35, 36, 37, 38, 39], "jentzen": [38, 39], "jerom": [19, 25, 30], "jhauser": [], "ji": [12, 24, 38, 39], "jit": 13, "jj": [0, 5, 6, 31, 35], "jk": [0, 1, 6, 12, 24, 31, 37, 38, 39], "jl": [0, 31], "jm": 24, "jnp": 13, "job": [2, 8, 10, 15], "join": [0, 4, 6, 7, 9, 25, 26, 31, 35, 36], "joint": [4, 5], "jonathan": [], "json": [], "judg": [13, 33, 36, 37], "judgement": 6, "julia": [23, 24, 25], "juliu": [38, 39], "jump": [28, 34], "junk": 4, "jupit": 31, "jupyt": [0, 15, 16, 19, 23, 25, 30, 31, 35, 38, 39], "jupyterbook": [], "jupytext": [], "just": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 20, 21, 22, 23, 26, 28, 31, 32, 33, 34, 35, 36, 37, 38, 39], "justif": 0, "justifi": [3, 10], "k": [0, 1, 3, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 21, 23, 24, 25, 28, 29, 31, 32, 33, 34, 37], "k0": [7, 36, 37], "k1": [7, 36, 37], "kaggl": [6, 25, 26], "kajda": 39, "kappa_d": 28, "karl": [29, 31], "karush": 8, "katex": [], "katrin": [29, 31], "keep": [0, 1, 4, 5, 6, 11, 13, 14, 15, 18, 21, 22, 24, 25, 26, 31, 32, 33, 34, 35, 36, 39], "keepdim": [1, 6, 10, 24, 35, 36, 37, 39], "kei": [1, 3, 6, 12, 34, 37, 39], "kellei": [], "kenneth": [], "kept": [4, 6, 14, 35, 36], "kera": [0, 4, 23, 25, 26, 31], "kernel": [0, 1, 3, 23, 31, 32, 39], "kernel_regular": [1, 3, 39], "kernel_s": 4, "kernelpca": 11, "kev": [0, 31], "kevin": [30, 31], "kevinsheppard": [], "keyboardinterrupt": 39, "keyword": [18, 24, 31, 39], "kfold": [6, 35, 36], "kg": [1, 39], "ki": 24, "kick": [1, 13, 34, 39], "kiener": 2, "kilomet": [6, 32], "kim": [], "kind": [0, 2, 3, 4, 8, 12, 13, 14, 31, 32, 37, 38, 39], "kingma": 34, "kj": [6, 12, 24, 32, 34, 38, 39], "kjm": [23, 31], "kkt": 8, "kl": 28, "km": [12, 31, 37], "kmean": 14, "kmeanspoint": 14, "kn_k": 14, "know": [0, 1, 2, 5, 6, 8, 13, 15, 16, 17, 19, 20, 23, 31, 32, 33, 39], "knowledg": [0, 23, 31], "known": [1, 3, 4, 5, 6, 7, 8, 9, 12, 18, 24, 25, 26, 28, 30, 32, 34, 35, 36, 37, 38, 39], "kondev": [0, 31], "kp": 28, "kpca": 11, "kramdown": [], "kroneck": 14, "kt": [], "kuckuck": [38, 39], "kuhn": 8, "kutyniok": [38, 39], "kvalsund": [29, 31], "kwarg": 39, "kwown": [0, 31], "l": [0, 1, 2, 3, 5, 6, 7, 8, 10, 11, 12, 13, 22, 24, 25, 28, 31, 33, 34, 36, 37], "l0": [7, 36, 37], "l1": [0, 1, 3, 7, 31, 36, 37, 39], "l1_l2": [1, 3, 39], "l1regl": 5, "l2": [1, 3, 39], "l_": [24, 34], "l_1": [7, 26, 36, 37, 38], "l_2": [7, 13, 26, 33, 34, 36, 37, 38], "l_i": 34, "l_j": [12, 38, 39], "l_ja": 39, "la": 13, "la_": [], "la_i": [12, 38, 39], "la_k": [12, 38], "lab": [20, 23, 25, 26, 31], "label": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 12, 13, 14, 15, 20, 23, 24, 25, 26, 28, 31, 32, 33, 34, 35, 36, 37, 38, 39], "labelencod": [7, 10, 37], "labels": [6, 8, 9], "labels_shuffl": [0, 1, 32, 39], "laboratori": 27, "lack": [0, 31, 34], "lagari": 2, "lagrang": [8, 11], "lam": [18, 39], "lambda": [0, 1, 2, 3, 5, 6, 7, 8, 10, 12, 13, 17, 18, 19, 20, 25, 26, 28, 31, 32, 33, 34, 35, 36, 37, 39], "lambda_": 11, "lambda_0": 11, "lambda_1": [5, 8, 11, 32, 33], "lambda_2": [8, 11], "lambda_i": [8, 11], "lambda_iy_i": 8, "lambda_jy_iy_j": 8, "lambda_k": 8, "lambda_n": [5, 8, 32, 33], "lamda": 1, "land": 8, "landmark": 8, "landscap": [13, 18, 33, 34], "langl": [0, 6, 11, 28, 31, 32], "languag": [0, 1, 4, 8, 23, 24, 25, 26, 30, 31, 39], "lapack": [24, 31], "laplac": 5, "laptop": [15, 23], "larg": [0, 1, 2, 4, 5, 6, 8, 9, 10, 11, 13, 18, 23, 24, 25, 28, 30, 31, 32, 33, 34, 35, 36, 38, 39], "larger": [0, 3, 5, 6, 8, 10, 11, 13, 17, 22, 28, 31, 32, 33, 34, 35], "largest": [4, 8, 11], "lasso": [0, 7, 23, 26, 31, 34, 35, 36, 37], "lasso_sk": 6, "last": [0, 1, 3, 4, 5, 6, 7, 8, 12, 16, 17, 19, 21, 22, 24, 25, 28, 29, 31, 33, 35, 36], "latent": 4, "latent_dim": 4, "latent_point": 4, "latent_space_value_rang": 4, "later": [0, 1, 4, 7, 8, 12, 13, 14, 15, 19, 21, 22, 23, 25, 26, 31, 34, 36, 37, 38, 39], "latest": [4, 15, 23], "latest_checkpoint": 4, "latex": [20, 31], "latexcodec": [], "latrpygrtttbnjr3znuhl": 22, "latter": [0, 3, 6, 7, 8, 11, 13, 24, 28, 31, 32, 33, 34, 35, 36, 37, 38], "lattic": [12, 37, 38], "law": 0, "layer": [0, 4, 13, 26, 31, 34, 37], "layer_grad": 22, "layer_input": 22, "layer_output_s": [21, 22], "layers_grad": 21, "lbfg": [7, 9, 10, 37], "lc_messag": [], "lcc": [5, 6, 35], "lda": 11, "ldot": [0, 6, 11, 25, 31, 35, 36], "le": [5, 7, 10, 13, 17, 28, 32, 33, 34, 36], "lead": [0, 1, 3, 5, 6, 7, 8, 9, 10, 11, 12, 13, 16, 17, 21, 22, 24, 28, 31, 32, 33, 34, 35, 36, 37, 38, 39], "leaf": 9, "leaki": [1, 26, 39], "leakyrelu": 4, "lear": [13, 33], "learn": [3, 4, 5, 6, 7, 8, 9, 10, 12, 21, 24, 29, 30], "learnabl": 3, "learner": 10, "learnig": 31, "learning_r": [8, 10, 21], "learning_rate_init": [0, 1, 31, 39], "learning_schedul": [13, 34], "learnt": [25, 26], "least": [0, 7, 8, 10, 11, 17, 18, 23, 24, 28, 35, 36, 37], "leat": [13, 34], "leav": [0, 1, 3, 5, 6, 9, 11, 21, 31, 33, 35, 36, 39], "lectur": [0, 1, 5, 10, 11, 12, 13, 23, 24, 25, 26, 27, 29, 30, 32], "lecturenot": [0, 23, 25, 26, 30, 31, 39], "left": [0, 1, 2, 3, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 16, 17, 19, 24, 25, 26, 28, 31, 32, 33, 34, 35, 36, 37, 38, 39], "leftarrow": [8, 12, 38, 39], "legend": [0, 2, 3, 4, 5, 6, 7, 8, 9, 10, 13, 15, 21, 31, 32, 33, 34, 35, 36, 37], "legend_el": 21, "leinonen": 31, "len": [0, 1, 2, 3, 4, 5, 6, 8, 9, 10, 11, 12, 16, 17, 21, 22, 24, 31, 32, 33, 34, 35, 36, 37, 39], "length": [0, 1, 3, 4, 8, 9, 13, 16, 21, 23, 31, 32, 33, 34, 39], "length_of_sequ": 4, "leq": [0, 5, 7, 8, 13, 14, 28, 31, 32, 33, 34, 36], "less": [0, 1, 3, 4, 5, 6, 8, 9, 13, 23, 28, 31, 32, 33, 34, 35, 36, 39], "lessen": [1, 39], "let": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 19, 22, 24, 28, 31, 32, 33, 34, 35, 36, 37, 38, 39], "letter": [0, 16, 24, 28, 31, 32], "level": [0, 1, 5, 6, 9, 23, 24, 25, 26, 27, 29, 31, 34, 35, 36, 38, 39], "leverag": 34, "lexer": [], "li": [8, 11], "liabil": [], "liabl": [], "lib": [], "liberti": 34, "liblinear": 10, "librari": [0, 1, 2, 3, 4, 5, 6, 9, 10, 11, 24, 25, 28, 30, 32, 33, 34, 39], "licenc": [], "licens": [0, 1, 23, 25, 31, 39], "lie": [0, 6, 11, 28, 31, 32, 35, 36], "life": [0, 1, 8, 12, 31, 37, 38, 39], "lifetim": 13, "light": [], "like": [0, 1, 2, 3, 4, 5, 6, 7, 9, 10, 11, 12, 13, 15, 16, 20, 21, 22, 23, 24, 25, 26, 28, 31, 32, 33, 34, 35, 36, 37, 38, 39], "likelihood": [0, 1, 5, 9, 31, 32, 39], "lim_": 28, "limit": [0, 5, 6, 8, 12, 24, 25, 26, 31, 32, 36, 37, 38], "lin_clf": 8, "lin_model": [], "lin_reg": 9, "linalg": [0, 2, 5, 6, 8, 11, 13, 17, 24, 28, 31, 32, 33, 34, 37], "line": [0, 3, 6, 8, 11, 13, 15, 16, 20, 21, 31, 33, 34, 35, 38, 39], "line1": 8, "line2": 8, "line2d": [], "line3": 8, "line_model": 15, "line_ms": 15, "line_predict": 15, "linear": [1, 3, 5, 6, 7, 9, 10, 11, 12, 16, 17, 18, 19, 21, 23, 25, 28, 34, 35, 37, 38, 39], "linear_model": [0, 5, 6, 7, 8, 9, 10, 11, 13, 15, 16, 19, 26, 31, 32, 33, 34, 35, 36, 37], "linear_regress": [6, 35, 36, 39], "linearli": [5, 32, 33, 34], "linearloc": [6, 13, 33, 34], "linearregress": [0, 6, 7, 9, 15, 16, 19, 31, 32, 34, 35, 36], "linearsvc": 8, "lineat": 33, "liner": [1, 3, 39], "linerar": 10, "linewidth": [0, 2, 4, 6, 8, 9, 10, 35], "link": [0, 4, 9, 12, 15, 20, 21, 23, 25, 26, 27, 29, 31, 36, 38], "linlag": 5, "linpack": [24, 31], "linreg": [0, 31], "linspac": [0, 2, 3, 4, 6, 8, 9, 10, 13, 16, 17, 19, 24, 28, 31, 32, 34, 35, 36], "linu": 4, "linux": [0, 1, 23, 25, 31, 39], "liquid": [0, 31], "list": [1, 2, 3, 4, 9, 15, 21, 22, 23, 25, 26, 31, 34, 37], "listedcolormap": [9, 10], "literatur": [1, 7, 14, 30, 35, 36, 39], "littl": [1, 3, 9, 12, 22, 34, 38, 39], "live": [8, 16], "ll": [0, 18, 28, 31, 32], "lle": [0, 32], "llm": 20, "lloyd": [4, 14], "lmb": [0, 2, 5, 6, 32, 33, 34, 35, 36], "lmbd": [0, 1, 3, 31, 39], "lmbd_val": [0, 1, 3, 31, 39], "lmbda": [13, 33, 34], "ln": [1, 13, 33, 39], "load": [1, 4, 6, 7, 9, 10, 34, 37], "load_boston": [], "load_breast_canc": [1, 7, 9, 10, 11, 37, 39], "load_data": [3, 4], "load_digit": [1, 3, 39], "load_iri": [8, 9, 21], "loc": [3, 6, 7, 8, 9, 10, 21, 31, 35, 36, 37], "local": [0, 1, 3, 7, 12, 13, 15, 21, 22, 32, 33, 34, 36, 37, 38, 39], "locat": [2, 3, 8, 15], "log": [0, 1, 2, 4, 5, 6, 7, 9, 10, 11, 13, 15, 20, 21, 24, 25, 26, 31, 34, 35, 36, 37, 39], "log10": [0, 5, 6, 32, 33, 34, 35, 36, 39], "log_": [0, 31], "log_clf": 10, "logarithm": [0, 5, 7, 17, 24, 31, 35, 36, 37], "logbook": [25, 26], "logic": [0, 1, 9, 31, 39], "logical_or": [], "login": 15, "logist": [0, 1, 2, 8, 9, 10, 11, 12, 13, 23, 26, 32, 33, 34, 38], "logisti": 26, "logistic_regress": 39, "logisticregress": [7, 9, 10, 11, 26, 36, 37], "logit": [7, 26, 36, 37], "logreg": [7, 9, 10, 11, 37], "logspac": [0, 1, 3, 5, 6, 31, 32, 33, 34, 35, 36, 39], "long": [0, 1, 3, 4, 12, 13, 21, 31, 33, 34, 37, 38, 39], "longer": [2, 3, 8, 10, 14, 24, 28, 31, 34], "loocv": [6, 35, 36], "look": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 13, 15, 16, 19, 20, 24, 25, 26, 28, 31, 32, 33, 34, 35, 36, 39], "loop": [1, 4, 6, 10, 12, 14, 16, 17, 18, 22, 23, 24, 31, 34, 35, 36, 39], "lose": [1, 39], "loss": [0, 1, 3, 4, 5, 6, 7, 8, 10, 11, 13, 18, 21, 24, 25, 26, 31, 35, 36, 37, 38, 39], "loss_bin": [36, 37], "loss_fil": 4, "loss_multi": [36, 37], "loss_vec": [36, 37], "lossfil": 4, "lost": 4, "lot": [1, 4, 6, 16, 19, 20, 34, 35, 39], "low": [0, 6, 9, 10, 11, 25, 31, 32, 35, 36], "lower": [0, 1, 3, 6, 9, 10, 16, 21, 24, 32, 34, 39], "lowercas": [24, 31], "lowest": [9, 13, 28, 34], "lr": [1, 3, 4, 10, 36, 37, 39], "lrelu": 39, "lstat": [], "lstm": 4, "lstm_2layer": 4, "lstsq": [0, 31, 32], "lt": [6, 35], "lu": [0, 5, 31, 32, 33], "lubksb": 24, "luckili": 2, "ludcmp": 24, "lux": 24, "lvert": [1, 39], "lw": [0, 31], "m": [0, 1, 2, 3, 5, 6, 8, 9, 10, 11, 12, 13, 15, 24, 28, 29, 30, 31, 32, 33, 34, 35, 37, 38, 39], "m_": [9, 12, 38, 39], "m_0": 34, "m_1": 14, "m_h": [0, 31], "m_k": 14, "m_l": [12, 38, 39], "m_n": [0, 31], "m_p": [0, 31], "m_t": [13, 34], "ma": 11, "machin": [1, 3, 4, 5, 6, 7, 9, 10, 11, 12, 15, 16, 24, 30, 32, 34, 35, 38, 39], "machinelearn": [0, 6, 16, 20, 23, 25, 26, 27, 29, 30, 31, 32, 33, 36, 37, 39], "machineri": [], "mackai": 30, "macro": [], "made": [0, 1, 3, 4, 5, 6, 7, 9, 11, 12, 25, 26, 31, 32, 34, 36, 37, 38, 39], "mae": [0, 31], "magic": 4, "magnitud": [1, 6, 7, 13, 21, 32, 34, 37, 38, 39], "mai": [0, 1, 2, 3, 5, 6, 7, 8, 9, 11, 12, 13, 19, 23, 24, 25, 26, 28, 31, 32, 33, 34, 35, 36, 37, 38, 39], "mail": [27, 29], "main": [0, 1, 3, 4, 5, 6, 7, 9, 24, 25, 26, 30, 32, 33, 34, 36, 37, 39], "mainli": [0, 5, 6, 7, 9, 31, 32, 35, 36, 37], "maintain": [6, 34, 35], "major": [1, 6, 9, 10, 13, 24, 31, 33, 34, 35, 36, 39], "make": [1, 2, 3, 4, 5, 6, 7, 8, 11, 12, 13, 15, 16, 18, 19, 21, 22, 23, 24, 25, 26, 28, 30, 31, 33, 34, 35, 36, 37, 38, 39], "make_axes_locat": 6, "make_classif": 37, "make_moon": [8, 9, 10], "make_pipelin": [0, 6, 10, 32, 35, 36], "makedir": [0, 6, 7, 9, 31, 35, 36], "malcondit": 24, "malign": [1, 7, 9, 37], "mammographi": 5, "manag": [0, 2, 3, 15, 23, 25, 31, 34], "mandatori": [29, 31], "mani": [0, 1, 3, 4, 5, 6, 7, 8, 9, 11, 13, 14, 15, 16, 17, 18, 19, 21, 22, 23, 24, 25, 26, 28, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39], "manifold": 11, "manner": 3, "manual": [6, 21, 22, 32, 34], "map": [0, 1, 2, 6, 7, 8, 11, 12, 14, 28, 31, 36, 37, 39], "marc": 32, "marchant": [], "margin": [0, 5, 8], "marit": [0, 31], "mark": 31, "markdownfil": [], "markdownit": [], "markdownitdeflist": [], "markedli": [], "marker": [7, 24, 31, 36], "markov": [23, 31], "markup": [], "marsaglia": 28, "mask_or": [], "masked_arrai": [], "maskedrecord": [], "mass": [0, 1, 5, 13, 32, 33, 39], "massag": [0, 31], "masses2016": [0, 31], "masses2016ol": [0, 31], "masses2016tre": 0, "masseval2016": [0, 31], "master": [27, 29], "mat": [23, 31], "mat1100": [23, 31], "mat1110": [23, 31], "mat1120": [23, 31], "match": [1, 4, 5, 13, 14, 15, 32, 33, 34, 39], "materi": [4, 5, 7, 13, 15, 24, 27, 29, 37], "math": [3, 7, 12, 13, 24, 28, 30, 31, 34, 36, 37, 39], "mathbb": [0, 4, 5, 6, 7, 8, 11, 12, 13, 14, 17, 19, 24, 25, 28, 31, 32, 33, 34, 35, 36, 37, 38], "mathbf": [0, 5, 6, 7, 8, 13, 19, 24, 25, 31, 32, 33, 34, 35, 36, 37, 38], "mathcal": [1, 5, 6, 7, 13, 25, 35, 36, 37, 39], "matheemat": 3, "mathemat": [0, 6, 11, 12, 13, 21, 23, 24, 28, 30, 31, 34], "mathemati": 31, "mathrm": [0, 1, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 17, 18, 19, 25, 28, 31, 32, 33, 34, 35, 36, 37, 38, 39], "matmul": [1, 2, 5, 38, 39], "matnat": 30, "matplotlib": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 17, 19, 21, 22, 23, 24, 25, 28, 31, 32, 33, 34, 35, 36, 37, 39], "matplotlibrc": [], "matric": [0, 1, 3, 4, 6, 7, 8, 11, 13, 16, 17, 23, 32, 33, 36, 37, 38, 39], "matrix": [0, 2, 3, 4, 6, 7, 8, 10, 13, 17, 18, 19, 21, 25, 26, 28, 35, 36, 38], "matshow": 1, "matter": [2, 3, 13, 32, 33, 34, 38], "matthia": [], "max": [0, 1, 2, 3, 4, 9, 10, 12, 13, 21, 29, 31, 33, 34, 36, 37, 38, 39], "max_depth": [0, 9, 10], "max_diff": 2, "max_diff1": 2, "max_diff2": 2, "max_it": [0, 1, 8, 13, 26, 31, 37, 39], "max_iter": 14, "max_leaf_nod": 10, "max_sampl": 10, "maxdegre": [0, 6, 10, 32, 35, 36], "maxdepth": 10, "maxim": [1, 4, 5, 7, 8, 11, 35, 36, 37, 39], "maximum": [0, 2, 3, 5, 7, 8, 9, 10, 13, 14, 31, 32, 33, 34], "maxpolydegre": [5, 6, 32, 33, 34, 35, 36], "maxpooling2d": 3, "mbox": [5, 6, 32, 33, 35], "mcculloch": [12, 37, 38], "md": 11, "mdoel": 4, "me": [], "mean": [1, 2, 3, 4, 5, 6, 7, 9, 10, 11, 12, 13, 14, 15, 17, 18, 19, 22, 23, 24, 25, 26, 28, 31, 34, 35, 37, 38, 39], "mean0": [36, 37], "mean1": [36, 37], "mean_absolute_error": [0, 31], "mean_divisor": 14, "mean_i": 28, "mean_matrix": 14, "mean_squared_error": [0, 4, 6, 7, 10, 15, 19, 31, 32, 35, 36], "mean_squared_log_error": [0, 31], "mean_vector": 14, "mean_x": 28, "meaning": [0, 4, 7, 31, 36], "meansquarederror": [0, 31], "meant": [3, 7, 10, 13, 36, 38], "meanwhil": 34, "measur": [0, 1, 2, 5, 6, 9, 11, 12, 14, 16, 18, 25, 26, 28, 31, 32, 34, 35, 36, 38, 39], "mechan": [0, 4, 28, 31, 34], "median": [0, 31, 32, 34], "medicin": [12, 37, 38], "medium": [4, 8, 13, 26, 34], "medv": [], "meet": [0, 29], "mehta": [0, 31, 32, 33], "member": [20, 25, 26], "memori": [3, 4, 11, 12, 13, 18, 24, 37, 38], "mentat": [], "mention": [0, 12, 13, 25, 26, 28, 31, 33, 34, 37, 38], "merchant": [], "mere": [0, 25, 26], "merg": [], "meshgrid": [2, 5, 6, 8, 9, 10, 11, 39], "mess": 15, "messag": [5, 13], "messi": 2, "messier": 22, "met": [0, 3, 8, 32], "meta": [], "meteorolog": 9, "meter": [6, 32], "method": [0, 1, 2, 3, 4, 5, 7, 8, 11, 12, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 26, 28, 30, 32, 38], "metion": 6, "metric": [0, 1, 3, 6, 7, 9, 10, 14, 15, 21, 22, 26, 31, 32, 35, 36, 37, 39], "metropoli": [23, 31], "mev": [0, 28, 31], "mgd": [13, 34], "mglearn": [23, 31], "mgrid": 13, "mhjensen": [], "mi": 10, "mia": [29, 31], "michael": [26, 38, 39], "microsoft": 30, "mid": [1, 39], "midel": 4, "midnight": [15, 21, 22], "midpoint": 9, "might": [0, 1, 2, 4, 6, 9, 13, 15, 17, 18, 22, 32, 33, 34, 39], "migth": 17, "mild": 9, "millimet": [6, 32], "million": [0, 31, 32, 34], "mimic": [12, 37, 38], "min": [0, 2, 5, 8, 9, 33], "min_": [0, 2, 5, 14, 17, 31, 32, 33], "min_samples_leaf": 9, "mind": [0, 6, 13, 15, 18, 21, 31, 32, 33, 34, 35], "mindboard": 4, "mine": [23, 31], "mini": [1, 11, 12, 13, 33, 39], "minibatch": [1, 11, 13, 39], "minibathc": [13, 34], "miniforge3": [], "minim": [0, 1, 2, 3, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 16, 17, 32, 33, 34, 35, 39], "minima": [0, 1, 7, 13, 31, 33, 34, 36, 37, 39], "minimum": [0, 1, 2, 6, 8, 9, 11, 13, 32, 33, 34, 35, 36, 37, 39], "minmaxscal": [0, 32, 34, 39], "minor": 28, "minst": [1, 39], "minu": [7, 36], "mirjalili": 31, "mirror": 9, "misc": 6, "misclassif": [8, 9, 10], "misclassifi": [8, 10], "miser": 0, "mismatch": [1, 39], "miss": [7, 10], "mistak": [4, 19], "mit": 30, "mitig": 34, "mix": [1, 2, 31, 39], "mixtur": [13, 34], "mk": [9, 24], "mkdir": [0, 6, 7, 9, 31, 35, 36], "ml": [0, 1, 10, 13, 24, 25, 26, 32, 33, 34, 39], "mlab": 28, "mle": [5, 7, 36, 37], "mlp": [1, 37, 38, 39], "mlpclassifi": [1, 37, 39], "mlpregressor": [0, 31], "mm": 24, "mml": 32, "mn": [12, 28, 37], "mnist": [1, 11, 26, 39], "mnist_784": 26, "mo": [], "mod": 28, "mode": [27, 29, 31, 36, 37, 39], "model": [2, 3, 5, 7, 8, 9, 10, 11, 13, 14, 16, 18, 19, 20, 21, 23, 25, 26, 28, 30, 32, 33, 34, 35, 36], "model_bin": [36, 37], "model_multi": [36, 37], "model_select": [0, 1, 3, 5, 6, 7, 9, 10, 11, 15, 16, 17, 19, 26, 31, 32, 33, 34, 35, 36, 37, 39], "moder": [10, 34], "modern": [0, 6, 7, 23, 31, 34, 35, 36, 37, 38, 39], "modest": 34, "modif": [2, 12, 13], "modifi": [0, 1, 3, 5, 7, 8, 10, 12, 13, 31, 32, 33, 34, 36, 37, 38, 39], "modul": [0, 16, 24, 31], "modular": 28, "modulo": 28, "moe": [11, 32], "moment": [5, 6, 13, 28, 35, 39], "moment_correct": 39, "momentum": [22, 38, 39], "momentum_schedul": 39, "mondai": [29, 31, 36], "monitor": [13, 34], "monoton": [5, 12, 28, 35, 37, 38, 39], "mont": [0, 6, 23, 28, 30, 31, 35, 36], "montli": 16, "moor": [5, 6], "more": [0, 1, 2, 4, 5, 7, 8, 9, 10, 11, 12, 13, 14, 16, 17, 19, 21, 22, 23, 26, 28], "moreov": [0, 3, 26], "morten": [29, 31, 32, 33, 34, 35, 36, 37, 38, 39], "mortenhj": 31, "most": [0, 1, 3, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 21, 22, 23, 25, 28, 31, 32, 33, 34, 35, 36, 37, 38, 39], "mostli": [1, 11, 18, 34, 39], "motion": [0, 13], "motiv": [1, 4, 38, 39], "moulin": 34, "move": [0, 4, 5, 6, 7, 9, 12, 13, 14, 15, 16, 21, 22, 25, 28, 32, 33, 35, 36, 37, 38, 39], "mpl": [7, 31, 36], "mpl_toolkit": [2, 6, 13, 33, 34], "mplot3d": [2, 6, 13, 33, 34], "mplregressor": [1, 39], "mr_": [], "mrecord": [], "ms3tv8fvar": 37, "mse": [0, 4, 5, 6, 9, 10, 15, 16, 17, 19, 20, 22, 25, 26, 31, 32, 33, 34, 35, 36, 39], "mse_der": 22, "mse_simpletre": 10, "mselassopredict": [5, 33], "mselassotrain": [5, 33], "mseownridgepredict": [6, 32, 33, 34], "msepredict": [5, 33], "mseridgepredict": [0, 5, 6, 32, 33, 34], "msetrain": [5, 33], "msg": [], "msle": [0, 31], "mt": [7, 12, 36, 37, 39], "mu": [0, 6, 11, 13, 28, 31, 34, 35], "mu0": 28, "mu1": 28, "mu2": 28, "mu_": [6, 28, 32, 34, 35], "mu_i": [6, 32, 34], "mu_n": 11, "mu_x": 28, "much": [0, 1, 2, 3, 4, 5, 6, 8, 9, 10, 11, 12, 13, 15, 20, 21, 22, 24, 25, 28, 31, 32, 33, 34, 35, 36, 38, 39], "multi": [0, 1, 3, 7, 23, 31, 36], "multi_class": [26, 36, 37], "multiclass": [1, 7, 26, 36, 37], "multiclass_result": [36, 37], "multidimension": [11, 12, 31, 37, 38], "multilay": [1, 39], "multinomi": [7, 26, 36, 37], "multipl": [2, 4, 5, 6, 7, 12, 13, 15, 22, 28, 32, 33, 34, 35, 36, 37, 38], "multipli": [3, 5, 6, 11, 13, 18, 22, 24, 28, 32, 33, 34], "multiplum": 8, "multivari": [0, 2, 10, 11, 23, 28, 31], "multivariate_norm": [11, 14], "multpli": 16, "murphi": [11, 30, 31], "muse": [], "must": [1, 2, 5, 6, 8, 10, 12, 13, 14, 15, 20, 22, 25, 26, 28, 32, 33, 34, 35, 36, 37, 38, 39], "mutat": [7, 36, 37], "mutual": [1, 3, 6, 13, 35, 36, 39], "mx_": 28, "my": 31, "myenv": [], "myriad": [0, 23, 31], "myself": [], "mz1": 28, "mz2": 28, "m\u00f8svatn": 6, "n": [0, 1, 2, 3, 4, 5, 6, 7, 8, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 24, 25, 26, 28, 31, 32, 33, 34, 35, 36, 37, 38, 39], "n0": [36, 37], "n1": [24, 36, 37], "n2": 24, "n8grai": [], "n_": [1, 2, 3, 8, 12, 28, 37, 39], "n_0": [12, 28, 37], "n_boostrap": [6, 10, 35, 36], "n_bootstrap": [6, 35], "n_categori": [1, 3, 39], "n_class": [36, 37], "n_cluster": 14, "n_compon": 11, "n_epoch": [13, 34, 39], "n_estim": 10, "n_examples_to_gener": 4, "n_featur": [1, 18, 36, 37, 38, 39], "n_filter": 3, "n_hidden": 2, "n_hidden_neuron": [0, 1, 31, 38, 39], "n_i": 28, "n_input": [0, 1, 3, 32, 38, 39], "n_instanc": 9, "n_iter": 34, "n_job": 10, "n_k": 14, "n_l": [12, 28, 37], "n_layer": 1, "n_m": 9, "n_neuron": 1, "n_neurons_connect": 3, "n_neurons_layer1": [1, 39], "n_neurons_layer2": [1, 39], "n_output": [38, 39], "n_point": 14, "n_sampl": [6, 8, 9, 10, 14, 18, 35, 36, 37], "n_split": [6, 35, 36], "n_step": 4, "n_t": 2, "n_x": 2, "nabla": [1, 13, 33, 34, 39], "nabla_": [2, 13, 33, 34], "nabla_w": 13, "nafter": 39, "nag": 13, "naimi": [0, 31], "naiv": [7, 36, 37], "naive_kmean": 14, "name": [0, 1, 3, 4, 5, 6, 7, 8, 9, 10, 12, 13, 14, 15, 18, 20, 21, 23, 24, 25, 26, 28, 29, 31, 32, 33, 35, 36, 37, 38, 39], "namespac": [], "nan": 39, "narrow": [13, 34], "nathaniel": [], "nation": [1, 5, 39], "nativ": [23, 31], "natur": [0, 1, 4, 8, 9, 12, 13, 25, 26, 28, 30, 31, 33, 34, 37, 38, 39], "navier": [12, 37, 38], "navig": [15, 34], "nb": 28, "nb_": 24, "nbconvert": 31, "nd": 14, "ndarrai": [6, 39], "nderiv": 39, "ne": [9, 10, 24, 28, 32, 33], "nearest": [1, 3, 6, 11, 39], "nearli": [13, 33], "neat": 31, "neccesari": [6, 35], "necess": 2, "necessari": [0, 1, 3, 4, 8, 14, 18, 31, 38, 39], "necessarili": [0, 4, 11, 28, 31], "necesserali": 5, "neck": [7, 36, 37], "need": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 18, 19, 20, 21, 22, 24, 26, 28, 32, 33, 34, 35, 36, 37, 38, 39], "neg": [0, 1, 3, 5, 6, 7, 10, 13, 24, 28, 31, 33, 35, 36, 37, 39], "neg_mean_squared_error": [6, 35, 36], "neglect": [28, 34], "neglig": 28, "neighbor": [3, 6, 11], "neither": [4, 13, 34], "neq": [13, 14, 28, 33], "nerual": 39, "nervou": [12, 37, 38], "nest": [9, 12, 37], "nesterov": 13, "net": [2, 4, 12, 37, 38], "netlib": [24, 31], "network": [0, 9, 13, 21, 22, 23, 30, 32], "network_input_s": [21, 22], "neural": [0, 13, 21, 22, 23, 30, 32, 36], "neural_network": [0, 1, 2, 31, 37, 39], "neuralnetwork": [1, 22, 39], "neuralnetworksanddeeplearn": [26, 38, 39], "neuron": [1, 2, 3, 4, 12, 39], "neutral": [0, 31], "neutron": [0, 31], "never": [1, 4, 6, 9, 28, 35, 36, 39], "new": [0, 1, 2, 3, 5, 6, 7, 8, 9, 10, 11, 13, 14, 15, 17, 20, 22, 24, 31, 32, 33, 34, 36, 37, 39], "new_chang": [13, 34], "new_hobbit": 31, "new_ma": [], "newaxi": [0, 3, 6, 9, 21, 35, 36], "newli": [0, 31], "newlin": [36, 37], "newton": [1, 7, 8, 13, 28, 38, 39], "next": [0, 1, 2, 3, 4, 5, 6, 8, 9, 13, 14, 15, 16, 21, 22, 31, 32, 33, 34, 35, 37, 38, 39], "next_guess": 13, "next_input": 4, "ng": [1, 39], "nhow": 39, "ni": 14, "nice": [0, 1, 5, 11, 22, 31, 32, 33, 39], "nicer": [18, 34], "nielsen": [26, 38, 39], "nine": [38, 39], "nip": 34, "niter": [13, 33, 34], "nitric": [], "nlambda": [0, 5, 6, 32, 33, 34, 35, 36], "nlp": 30, "nm": 28, "nm_n": [0, 31], "nmse": [6, 35, 36], "nn": [2, 5, 6, 12, 24, 31, 35, 37], "nn_model": 1, "nnmin": 2, "node": [1, 3, 9, 10, 12, 21, 26, 37], "nois": [0, 4, 5, 6, 8, 9, 10, 13, 18, 19, 25, 31, 32, 33, 34, 35, 36], "noise_dimens": 4, "noisi": [1, 6, 25, 34, 35, 36, 39], "nomask": [], "non": [0, 1, 3, 5, 6, 7, 9, 10, 11, 12, 13, 14, 18, 21, 24, 28, 31, 32, 33, 35, 36, 37, 38, 39], "nondifferenti": 34, "none": [0, 1, 2, 4, 5, 9, 10, 13, 28, 31, 32, 36, 37, 38, 39], "noninfring": [], "nonlinear": [3, 6, 8, 9, 11, 12, 35, 36, 37, 38], "nonneg": [6, 9, 13, 33, 35, 36], "nonparametr": 6, "nonsens": 28, "nonsingular": 24, "nonumb": [3, 7, 8, 13, 24, 36, 37], "nor": [1, 4, 13, 22, 34, 38, 39], "norm": [0, 1, 5, 6, 8, 11, 13, 18, 31, 32, 33, 34, 35, 38, 39], "normal": [3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 16, 17, 18, 19, 21, 23, 24, 25, 26, 28, 31, 32, 33, 34, 36, 37, 38], "normali": [24, 31], "norwai": [6, 25, 26, 31, 33, 34, 35, 37, 38, 39], "notabl": [], "notat": [0, 2, 5, 6, 13, 14, 28, 31, 32, 33, 35, 36, 38, 39], "note": [0, 1, 2, 3, 4, 5, 6, 7, 8, 11, 12, 13, 14, 15, 16, 18, 22, 23, 24, 28, 30, 31, 34, 35, 36, 37, 38, 39], "notebook": [0, 1, 3, 9, 15, 16, 19, 20, 21, 22, 23, 25, 26, 31, 35, 38, 39], "noteworthi": 34, "noth": [1, 2, 5, 8, 12, 14, 28, 32, 33, 37, 39], "notic": [4, 5, 12, 13, 22, 24, 28, 31, 38, 39], "notimplementederror": 39, "notion": 3, "noutput": 39, "novel": [3, 6, 10, 31], "novemb": [1, 29, 31, 39], "now": [0, 2, 4, 5, 6, 7, 8, 10, 11, 12, 14, 15, 16, 19, 21, 22, 23, 24, 25, 26, 28, 31, 32, 37, 38, 39], "nowadai": [0, 1, 3, 9, 23, 31, 39], "nox": [], "np": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 13, 14, 15, 16, 17, 18, 19, 21, 22, 24, 28, 31, 32, 33, 34, 35, 36, 37, 38, 39], "npm": [], "npr": 2, "nsampl": [6, 35, 36], "nt": 2, "nu": 28, "nuclear": [5, 32, 33], "nuclei": [0, 28, 31], "nucleon": [0, 31], "nucleu": [0, 31], "num": 4, "num_coordin": 2, "num_equ": 39, "num_hidden_neuron": 2, "num_it": [2, 18], "num_neuron": 2, "num_neurons_hidden": 2, "num_not": 39, "num_point": 2, "num_tre": 10, "num_valu": 2, "number": [1, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 16, 17, 18, 19, 21, 24, 25, 26, 27, 29, 31, 33, 35, 36, 37, 39], "numberid": [7, 36], "numberparamet": 3, "numer": [0, 5, 6, 9, 10, 11, 12, 13, 21, 23, 24, 30, 31, 32, 33, 34, 35, 36, 37, 38], "numpi": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 21, 22, 23, 25, 28, 32, 33, 34, 35, 36, 37, 38, 39], "numpydocstr": [], "nunmpi": [5, 32], "nve_frngahw": 33, "nx": 2, "ny": [28, 39], "o": [0, 1, 4, 5, 6, 7, 8, 9, 11, 24, 29, 30, 31, 32, 33, 34, 35, 36, 37], "obei": [6, 11, 13, 32, 34], "object": [0, 1, 4, 8, 10, 15, 19, 24, 31, 34, 38], "obliqu": [5, 32, 33], "observ": [0, 1, 3, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 28, 31, 33, 34, 35, 36, 37], "obtain": [0, 1, 5, 6, 7, 8, 9, 10, 12, 13, 14, 17, 24, 25, 26, 28, 31, 32, 33, 34, 35, 36, 37, 38, 39], "obviou": [5, 6, 11, 28, 32, 33], "obviouli": 31, "obvious": [0, 4, 5, 6, 24, 31, 35], "oc": [32, 33], "occupi": [], "occur": [0, 6, 8, 9, 24, 28, 31], "octob": [21, 22, 26, 29, 31, 37], "od": 0, "odd": [0, 3, 7, 31, 32, 34, 36, 37], "odenum": 2, "odesi": 2, "oen": 0, "off": [1, 3, 4, 5, 9, 13, 20, 26, 28, 34, 35, 39], "offer": [6, 11, 23, 24, 27, 29, 31, 35, 36], "offic": [29, 31], "offici": [27, 31], "offlin": [21, 22], "often": [0, 1, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 18, 19, 21, 23, 24, 25, 26, 28, 31, 32, 33, 34, 35, 36, 37, 38, 39], "ofter": [24, 31], "og": 39, "ol": [0, 13, 17, 19, 26, 32, 34, 36], "old": [1, 5, 10, 13, 15, 18, 36, 37, 39], "old_ma": [], "oliph": [], "ols_paramet": 16, "ols_sk": 6, "ols_svd": 6, "olsbeta": 33, "olstheta": [0, 5], "omega": [2, 3, 6], "omega_0": 3, "omit": [0, 5, 31, 32, 33, 35], "onc": [1, 6, 9, 11, 13, 20, 35, 36, 39], "one": [0, 1, 3, 4, 5, 6, 7, 8, 9, 10, 11, 13, 14, 15, 19, 20, 21, 23, 24, 25, 26, 28, 29, 31, 32, 34, 35, 36, 37], "one_hot": [36, 37], "one_hot_predict": 21, "onehot": [1, 39], "onehot_vector": [1, 39], "onehotencod": 9, "ones": [0, 2, 5, 6, 8, 9, 10, 11, 13, 16, 18, 21, 22, 24, 25, 31, 32, 33, 34, 35, 36, 38], "ones_lik": 4, "ong": 32, "onl": 3, "onli": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 17, 18, 19, 20, 21, 22, 24, 25, 26, 28, 31, 32, 33, 34, 35, 36, 37, 38, 39], "onlin": [11, 15, 20, 27, 34, 38, 39], "onto": [5, 11, 32, 33], "open": [0, 1, 4, 6, 7, 9, 15, 23, 25, 27, 29, 31, 35, 36, 37, 39], "oper": [0, 1, 3, 5, 6, 10, 11, 12, 13, 15, 16, 21, 22, 23, 28, 31, 32, 33, 34, 35, 37, 39], "operation": 28, "oplu": 28, "opmiz": [13, 34], "opportun": 0, "oppos": [6, 13], "opposit": [1, 5, 8, 32, 33, 39], "opt": [1, 5, 25, 26, 31, 33, 39], "optim": [0, 2, 3, 4, 5, 6, 7, 9, 10, 11, 14, 16, 17, 19, 21, 22, 25, 26, 35], "optimis": [1, 3, 39], "option": [0, 1, 3, 5, 6, 8, 11, 15, 18, 24, 26, 32, 34, 35, 39], "optmiz": [1, 8, 13, 32, 39], "oral": 31, "orang": 0, "order": [0, 1, 2, 3, 5, 6, 7, 8, 9, 10, 11, 12, 15, 19, 21, 24, 25, 26, 28, 31, 32, 33, 35, 36, 37, 38, 39], "ordinari": [0, 2, 3, 7, 11, 13, 17, 18, 23, 35, 36, 37], "oreilli": [30, 31], "org": [0, 3, 4, 16, 20, 21, 23, 24, 25, 26, 30, 31, 32, 33, 34, 38], "organ": [6, 7, 10, 24, 35, 36], "orgin": 38, "orient": [1, 5, 28, 32, 33], "origin": [0, 3, 5, 6, 8, 11, 12, 13, 15, 24, 31, 32, 33, 34, 35, 36, 37], "orthogn": [5, 32, 33], "orthogon": [0, 5, 6, 8, 11, 13, 24, 31, 32, 33], "orthonorm": [5, 32, 33], "os": [29, 31], "oscar": [1, 39], "oscil": [3, 13, 34], "oskar": 31, "oskarlei": 31, "osl": 18, "oslo": [0, 23, 25, 26, 27, 29, 31, 32, 33, 34, 35, 36, 37, 38, 39], "osx": [0, 23, 25, 31], "other": [0, 1, 2, 3, 5, 6, 7, 8, 10, 13, 14, 16, 19, 21, 22, 23, 27, 28, 29, 30, 32, 33, 34, 35, 36], "otherwis": [0, 1, 4, 7, 13, 24, 26, 31, 34, 36, 37, 39], "ouput": [5, 7, 12, 35, 36], "our": [1, 2, 3, 6, 7, 8, 9, 10, 12, 14, 15, 16, 17, 18, 19, 21, 23, 24, 28, 34, 35, 38], "ourmodel": 0, "ourselv": [0, 5, 6, 8, 11, 13, 31, 32, 33, 35], "out": [0, 1, 2, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 15, 16, 21, 22, 23, 24, 25, 26, 28, 31, 32, 34, 35, 36, 37, 38, 39], "out_deriv": 39, "out_fil": 9, "outcom": [0, 7, 9, 10, 12, 28, 32, 36, 37], "outdoor": 9, "outer": [6, 12, 13], "outfil": 4, "outlier": [0, 8, 31, 32, 34], "outlin": [6, 10, 11, 35, 36], "outlook": 9, "outperform": [10, 34], "output": [0, 1, 3, 4, 5, 6, 7, 8, 9, 10, 12, 13, 19, 21, 22, 24, 25, 26, 28, 31, 32, 33, 34, 35, 36, 37], "output_bia": [1, 39], "output_bias_gradi": [1, 38, 39], "output_func": 39, "output_nod": 39, "output_shap": 4, "output_weight": [1, 39], "output_weights_gradi": [1, 38, 39], "outputlayer1": [12, 37], "outputlayer2": [12, 37], "outsid": [4, 22], "over": [0, 1, 3, 4, 5, 6, 9, 10, 12, 13, 15, 16, 19, 22, 24, 25, 31, 32, 33, 34, 35, 36], "over1": 13, "overal": [1, 10, 34, 39], "overcast": 9, "overcom": [12, 13, 37, 38], "overdetermin": [0, 31], "overfit": [0, 1, 3, 6, 9, 10, 13, 26, 34, 35, 36, 39], "overflow": [5, 34, 35], "overflowerror": 39, "overhead": [12, 38, 39], "overlap": [3, 7, 8, 9, 37], "overleaf": [20, 25, 26], "overlin": [0, 5, 6, 9, 10, 11, 14, 24, 31, 32, 34], "overshoot": 34, "overst": 0, "overtrain": 4, "overview": [3, 20], "overwritten": 39, "own": [4, 5, 6, 8, 12, 13, 16, 18, 22, 23, 24, 33, 34, 35, 38, 39], "owner": [], "ownmsepredict": 0, "ownmsetrain": 0, "ownridgebeta": 32, "ownridgetheta": [0, 6, 32, 33, 34], "ownypredictridg": 0, "ownytilderidg": 0, "ox": [], "oxid": [], "p": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 13, 14, 15, 16, 17, 18, 19, 24, 28, 31, 32, 33, 34, 35, 36, 37, 39], "p0": 2, "p1": 2, "p_": [2, 4, 8, 9], "p_hidden": 2, "p_i": [5, 28], "p_j": 28, "p_n": 28, "p_output": 2, "p_x": 28, "pa": 38, "pack": [0, 31], "packag": [0, 1, 3, 4, 5, 8, 11, 13, 15, 20, 22, 23, 25, 26, 28, 32, 33, 34, 39], "packtpub": 31, "packtpublish": 31, "pad": [3, 4], "page": [0, 23, 25, 26, 31, 33, 34, 35, 36], "pai": [0, 1, 9, 13, 15, 34, 39], "pair": [0, 2, 3, 9, 23, 28, 31], "paltform": 15, "panda": [0, 4, 5, 6, 7, 9, 11, 23, 25, 33, 34, 35, 36, 37], "pandoc": [], "panel": 31, "paper": [1, 34], "paper_fil": 34, "paradigm": [0, 31], "paragraph": 20, "parallel": [10, 13, 23, 24, 31], "param": 2, "paramat": 2, "paramet": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 12, 13, 16, 17, 18, 19, 21, 22, 25, 26, 28, 33, 34, 35], "parameter": [0, 6, 10, 31, 32], "parametr": [0, 6, 31, 32, 35, 36], "paramt": [3, 5, 35, 38], "parent": 38, "parser": 26, "part": [0, 1, 3, 5, 6, 10, 17, 19, 20, 21, 22, 24, 27, 28, 29, 31, 32, 35], "partial": [0, 1, 5, 6, 7, 8, 10, 11, 12, 13, 16, 21, 28, 31, 32, 33, 34, 36, 37, 38, 39], "particip": [15, 23, 27, 29, 31], "particl": [0, 4, 13, 28, 31], "particular": [0, 1, 2, 3, 5, 6, 9, 10, 11, 12, 13, 16, 25, 28, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39], "particularli": [5, 6, 8, 11, 13, 28, 32, 33, 34, 35, 36], "partit": [1, 4, 9, 39], "partli": [6, 31], "partner": [15, 25, 26], "pass": [2, 3, 12, 14, 21, 34, 38], "password": [25, 26], "past": [10, 28, 34], "patch": [6, 28, 35], "path": [0, 4, 6, 7, 9, 23, 31, 34, 35, 36], "pathcollect": 17, "patholog": [], "patient": [7, 36, 37], "patter": 4, "pattern": [0, 3, 4, 12, 30, 31, 34, 37, 38], "paul": [], "pauli": [0, 31], "pav": [], "pc": [11, 15, 23], "pca": [0, 7, 23, 31, 32, 37], "pd": [0, 4, 5, 6, 7, 9, 11, 31, 32, 33, 34, 35, 36, 37], "pde": 2, "pdf": [0, 3, 4, 5, 6, 9, 15, 16, 19, 20, 25, 26, 30, 31, 35], "pedagog": [0, 31, 32], "penal": [6, 18, 32, 34], "penalti": [6, 13, 18, 25, 32, 34], "penros": [5, 6], "pentagon": [13, 33], "peopl": [1, 9, 13, 23, 25, 26, 34, 39], "per": [0, 1, 6, 21, 27, 29, 31, 34, 35, 36, 37, 39], "perc_print": 39, "percentag": [10, 11, 29, 39], "perceptron": [0, 1, 7, 31, 36], "peregrin": 31, "perez": [], "perfect": [0, 1, 13, 31, 34, 39], "perfectli": [4, 6, 35, 36], "perform": [0, 2, 3, 4, 5, 6, 8, 10, 11, 12, 13, 14, 16, 18, 19, 21, 22, 23, 24, 25, 26, 28, 31, 32, 33, 34, 35, 36, 37, 38], "performac": 4, "perhap": [0, 5, 13, 31, 32, 33, 34], "perimet": 1, "period": [1, 4, 28, 39], "permiss": 15, "permit": [], "permut": 11, "persist": 13, "person": [5, 6, 7, 16, 20, 27, 29, 31, 32, 36], "perspect": 30, "pertin": [12, 26, 31, 38, 39], "petal": [8, 9], "peter": [30, 32], "petersen": [38, 39], "phantom": 28, "phase": [6, 12, 37, 38], "phenomena": 28, "phenomenon": 34, "phi": 8, "phi_k": 8, "philipp": [38, 39], "philosophi": 13, "phone": [29, 31], "photo": [4, 31], "php": [25, 26], "phrase": [0, 31], "physic": [0, 1, 4, 7, 12, 13, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39], "pi": [2, 3, 5, 6, 7, 9, 12, 13, 28, 35, 36, 37, 39], "pick": [1, 9, 10, 11, 13, 14, 25, 26, 34, 39], "pickl": 1, "pictur": [0, 31], "pie": [23, 31], "piec": [11, 14, 21], "pierr": [], "pillow": [0, 23, 25, 31], "pinv": [5, 6, 13, 25, 32, 33, 34, 37], "pip": [0, 1, 15, 23, 25, 31, 39], "pip3": [0, 1, 25, 31, 39], "pipelin": [0, 6, 8, 10, 32, 35, 36], "pippin": 31, "pit": 4, "pitfal": [6, 32], "pitt": [12, 37, 38], "pixel": [1, 3, 4, 26, 31, 39], "pixel_height": [1, 3, 39], "pixel_width": [1, 3, 39], "pkg_resourc": [], "pkgutil": [], "place": [0, 4, 6, 8, 13, 15, 24, 25, 31, 33, 35], "plai": [0, 3, 4, 5, 6, 8, 11, 18, 22, 23, 25, 31, 32, 33, 35, 36, 38, 39], "plain": [8, 10, 12, 13, 14, 25, 26, 33, 34, 38, 39], "plan": [6, 9, 29, 30, 31, 39], "plane": [8, 9], "plateau": [5, 33, 34], "platform": [23, 31], "plausibl": [12, 37, 39], "pleas": [13, 25, 26, 29, 31], "plenti": [1, 39], "plethora": [3, 12, 37, 38], "pliahhy2ibx9hdharr6b7xevztgzra1p": [37, 38, 39], "plot": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 16, 17, 19, 20, 21, 23, 24, 25, 26, 28, 31, 32, 33, 34, 37, 39], "plot_all_sc": [25, 32], "plot_confusion_matrix": [7, 10, 37], "plot_count": 6, "plot_cumulative_gain": [7, 10, 37], "plot_data": 1, "plot_dataset": 8, "plot_decision_boundari": [9, 10], "plot_import": 10, "plot_iris_dataset": 21, "plot_max": 4, "plot_min": 4, "plot_model": 4, "plot_numb": 4, "plot_predict": 8, "plot_regression_predict": 9, "plot_result": 4, "plot_roc": [7, 10, 37], "plot_surfac": [2, 6, 13], "plot_train": 9, "plot_tre": [9, 10], "plqvvvaa0qudcjd5baw2dxe6of2tius3v3": [37, 38, 39], "plt": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 17, 19, 21, 22, 24, 28, 31, 32, 33, 34, 35, 36, 37, 39], "plu": [0, 3, 5, 7, 18, 31, 32, 36], "plugin": [], "pm": [8, 35], "pmatrix": 2, "pml": 30, "pn": 3, "png": [0, 4, 6, 7, 9, 31, 35, 36], "point": [0, 1, 2, 3, 5, 6, 7, 8, 9, 10, 11, 13, 14, 18, 19, 20, 24, 25, 28, 29, 31, 32, 33, 34, 35, 36, 37, 39], "point_1": 4, "point_2": 4, "poisson": [23, 28, 31], "poli": [6, 8, 35, 36], "poly100_kernel_svm_clf": 8, "poly3": 0, "poly3_plot": 0, "poly_degre": 39, "poly_featur": [8, 9, 15], "poly_features10": 9, "poly_fit": 9, "poly_fit10": 9, "poly_kernel_svm_clf": 8, "poly_model": 15, "poly_ms": 15, "poly_predict": 15, "polydegre": [0, 5, 6, 10, 32, 35, 36], "polygon": [13, 33], "polym": [12, 37, 38], "polymi": 25, "polynomi": [0, 5, 6, 7, 8, 9, 10, 11, 15, 17, 19, 20, 25, 26, 31, 32, 34, 35, 36, 37, 38], "polynomial_featur": [6, 15, 16, 17, 35, 36], "polynomial_svm_clf": 8, "polynomialfeatur": [0, 6, 8, 9, 15, 16, 19, 32, 35, 36], "polytrop": [0, 6, 35, 36], "pool": 3, "pool_siz": 3, "poor": [1, 13, 33, 34, 39], "poorli": [0, 32], "popul": [0, 5, 31, 32], "popular": [0, 1, 3, 6, 7, 8, 9, 11, 12, 15, 23, 24, 25, 28, 32, 36, 37, 39], "popularli": [0, 31], "portabl": 10, "portion": [11, 13, 34], "pose": [0, 4, 5, 6, 11, 28, 31, 35], "posit": [0, 1, 2, 3, 5, 7, 8, 10, 11, 13, 14, 21, 24, 28, 31, 32, 33, 34, 36, 37, 39], "possibl": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 21, 23, 24, 25, 26, 28, 29, 31, 32, 33, 34, 35, 36, 37, 38, 39], "possibli": [6, 8, 13, 25], "post": [], "posterior": 5, "postpon": [0, 32], "postscript": [25, 26], "postul": 5, "potenti": [0, 3, 5, 6, 12, 13, 32, 34, 35, 37, 38], "pott": [12, 37, 38], "power": [0, 1, 5, 6, 8, 9, 12, 13, 31, 32, 33, 34, 35, 36, 37, 38, 39], "pp": [5, 6, 19, 35, 38, 39], "practic": [0, 5, 6, 7, 8, 16, 18, 19, 21, 25, 26, 28, 32, 35, 36, 37], "practition": [0, 1, 3, 31, 34, 39], "pre": 31, "preambl": [], "precalcul": 38, "preced": [1, 11, 12, 28, 37, 39], "preceed": [4, 39], "preceq": 8, "precis": [0, 2, 5, 11, 13, 24, 25, 26, 28, 31, 32, 34, 35, 38], "pred": [6, 35, 36, 37], "pred_train": 39, "pred_val": 39, "predicit": 0, "prediciton": 39, "predict": [0, 1, 5, 6, 7, 8, 9, 10, 15, 16, 17, 19, 22, 23, 25, 26, 30, 31, 32, 33, 34, 35, 36, 37, 39], "predict_prob": [1, 36, 37, 39], "predict_proba": [7, 10, 37], "predictedlabel": [36, 37], "predictor": [0, 5, 6, 7, 9, 10, 11, 31, 32, 34], "prefer": [0, 1, 6, 8, 9, 11, 13, 15, 20, 23, 25, 26, 31, 39], "prefil": [], "prepar": [0, 6, 24, 25, 26, 31, 32], "preprocess": [0, 4, 6, 7, 8, 9, 10, 11, 15, 16, 17, 18, 19, 25, 35, 36, 37, 39], "prerequisit": 0, "prescript": [25, 26], "presenc": 13, "present": [0, 5, 6, 7, 9, 12, 13, 24, 25, 26, 28, 31, 32, 33, 34, 37, 38, 39], "preserv": [3, 11, 24], "press": [13, 15, 30, 33, 38, 39], "pretrain": [1, 4, 39], "pretti": [0, 4, 8, 9, 21, 23, 25, 31], "prettier": [], "prev_centroid": 14, "prevent": [13, 28, 34], "previou": [0, 1, 2, 3, 4, 5, 6, 8, 10, 11, 12, 13, 15, 16, 21, 22, 24, 25, 26, 28, 32, 33, 34, 37, 38, 39], "previous": [2, 3, 9, 10, 28], "price": [0, 4, 9, 13, 34], "primal": 8, "primari": [0, 7, 31, 36, 37], "prime": 28, "princip": [0, 5, 7, 23, 31, 32, 33, 37], "principl": [0, 6, 7, 8, 14, 31, 35, 36, 37], "print": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 13, 14, 18, 21, 22, 24, 26, 28, 31, 32, 33, 34, 35, 36, 37, 38, 39], "print_funct": [8, 9], "print_length": 39, "printout": [0, 31], "prior": [0, 5, 6, 31], "privat": 0, "pro": 26, "prob": [1, 28, 36, 37], "probabilist": [0, 30, 31, 32], "probabl": [0, 1, 3, 4, 6, 7, 10, 13, 21, 23, 31, 32, 34, 36, 37, 39], "problem": [0, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 17, 23, 24, 25, 26, 28, 35], "probml": 30, "proce": [0, 5, 6, 7, 8, 9, 10, 11, 13, 24, 31, 32, 35, 38], "procedur": [2, 4, 5, 6, 8, 10, 11, 13, 32, 33, 34, 35, 36], "proceed": 24, "process": [0, 2, 4, 6, 9, 10, 12, 13, 23, 24, 25, 28, 30, 31, 33, 34, 35, 36, 37, 38], "procur": [], "prod": 30, "prod_": [1, 5, 7, 35, 36, 37, 39], "produc": [0, 3, 4, 5, 6, 9, 10, 11, 12, 13, 18, 20, 23, 24, 25, 26, 28, 31, 32, 35, 37, 38], "product": [0, 1, 3, 5, 6, 7, 8, 12, 13, 16, 17, 23, 24, 31, 32, 34, 35, 36, 37, 38, 39], "profess": [0, 31], "profit": [], "progag": 26, "program": [0, 1, 4, 5, 6, 8, 12, 14, 15, 23, 24, 27, 28, 29, 31, 32, 37, 39], "programm": 24, "progress": [1, 4, 14, 34, 36, 37, 39], "prohibit": [6, 35, 36], "project": [0, 1, 2, 3, 5, 11, 13, 15, 19, 22, 23, 27, 32, 33, 34, 35, 36, 37, 39], "project_root_dir": [0, 6, 7, 9, 31, 35, 36], "promin": [12, 37, 38], "promis": 8, "promot": [29, 31], "prompt": 20, "prone": [9, 15, 21, 38], "pronounc": [13, 23, 31, 34], "proof": [0, 11, 12, 13, 31, 33, 35, 36, 38], "prop": [34, 39], "prop_cycl": [], "propag": [2, 3, 13, 21, 22, 26, 34], "proper": [0, 2, 6, 7, 20, 35, 36], "properli": [1, 6, 8, 10, 13, 18, 20, 25, 26, 34, 39], "properti": [0, 1, 3, 12, 13, 16, 24, 31, 35, 37, 39], "propgag": 38, "proport": [0, 1, 5, 9, 11, 13, 28, 31, 32, 39], "propos": [1, 4, 6, 10, 25, 26, 31, 34, 39], "propto": [5, 13, 33, 34], "proton": [0, 31], "prove": [3, 13, 33, 34], "provid": [0, 1, 3, 4, 5, 6, 8, 9, 10, 12, 13, 20, 21, 22, 23, 24, 25, 26, 28, 31, 32, 33, 34, 35, 36, 38, 39], "proxi": [1, 13, 34, 39], "prune": 9, "pseudo": [24, 28, 34], "pseudocod": [25, 26], "pseudoinv": 5, "pseudoinvers": [5, 6, 25], "pseudorandom": [6, 28, 35], "psychologi": [0, 31], "pt": 13, "public": [0, 15, 23, 31], "publish": [38, 39], "pull": 15, "punish": [0, 1, 31, 39], "pure": [3, 9, 28], "purest": 9, "puriti": 9, "purpos": [0, 3, 10, 12, 14, 21, 31, 37, 38], "push": 15, "put": [1, 20, 25, 26, 34], "putmask": [], "py": 5, "pybtex": [], "pycod": 31, "pydata": 23, "pydevd_extension_api": [], "pydevd_plugin": [], "pydevd_plugin_plugin_nam": [], "pydot": 9, "pygment": [], "pyhton2": 31, "pylab": [7, 31, 36], "pypi": 23, "pyplot": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 17, 19, 21, 22, 24, 28, 31, 32, 33, 34, 35, 36, 37, 39], "pythagora": 5, "python": [1, 2, 3, 5, 6, 8, 11, 12, 13, 14, 18, 20, 21, 22, 25, 26, 28, 32, 34, 38, 39], "python2": [0, 25], "python3": [0, 23, 25, 31], "pythonpath": [], "pytorch": [0, 23, 25, 26, 31, 38, 39], "pyzmq": [], "q": [5, 6, 8, 11, 28, 35, 39], "qp": 8, "qquad": [2, 11, 13, 24, 34], "qr": [5, 6, 24, 32, 33], "quad": [1, 13, 24, 39], "quadrat": [0, 8, 9, 13, 31], "qualit": [4, 9, 25, 26, 28], "qualiti": [0, 9, 23, 31, 32, 38], "quantifi": [1, 39], "quantil": 10, "quantit": [0, 6, 9, 25, 26, 31, 35, 36], "quantiti": [0, 2, 5, 6, 7, 9, 10, 11, 12, 14, 16, 24, 28, 31, 32, 33, 34, 35, 36, 37, 38, 39], "quantum": [4, 12, 30, 31, 37, 38], "quartil": [0, 32, 34], "quasi": 38, "quench": 5, "queri": 9, "question": [0, 5, 6, 9, 11, 12, 13, 25, 26, 29, 31, 32, 34, 35, 38, 39], "qugan": 4, "quick": [4, 28], "quicker": 34, "quickli": [1, 3, 9, 11, 13, 33, 34, 39], "quit": [1, 5, 6, 9, 10, 12, 15, 22, 32, 33, 35, 36, 37, 39], "quot": 4, "r": [0, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 17, 23, 24, 25, 28, 32, 33, 34, 35, 36, 37, 38, 39], "r2": [0, 5, 6, 19, 31, 32, 33], "r2_score": [0, 31], "r2score": [0, 31], "r_": 34, "r_0": 34, "r_1": 9, "r_2": 9, "r_j": 9, "r_m": 9, "r_t": 34, "rad": [], "rade": [], "radial": [8, 12, 37, 38], "radioact": 28, "radiu": [0, 1, 32, 34], "radziej": [], "ragan": [], "rain": 9, "rais": 39, "ram": 34, "ramanujam": [], "ramp": [1, 39], "ran0": 28, "ran1": 28, "ran2": 28, "ran3": 28, "rand": [0, 4, 5, 6, 9, 10, 13, 15, 19, 21, 22, 24, 31, 32, 33, 34, 35, 36, 39], "randint": [6, 9, 13, 34, 35], "randn": [0, 1, 2, 5, 6, 9, 11, 13, 15, 18, 21, 22, 31, 32, 33, 34, 35, 36, 37, 38, 39], "random": [0, 1, 2, 3, 4, 5, 6, 8, 9, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 31, 32, 33, 34, 35, 36, 37, 38, 39], "random_forest_model": 10, "random_index": [13, 34], "random_indic": [1, 3, 39], "random_st": [7, 8, 9, 10, 11, 26, 36, 37], "randomforestclassifi": 10, "randomli": [1, 6, 9, 13, 14, 18, 33, 34, 35, 36, 39], "randomst": [36, 37], "rang": [0, 1, 2, 3, 4, 5, 6, 7, 9, 10, 11, 12, 13, 14, 18, 19, 21, 22, 24, 26, 28, 31, 32, 33, 34, 35, 36, 37, 38, 39], "rangl": [0, 6, 11, 28, 31, 32], "rangle_x": 28, "rank": [5, 32, 33], "rankdir": 4, "raphson": [1, 8, 13, 39], "rapidli": [0, 34], "rare": [1, 13, 34, 39], "raschka": [26, 31, 32, 35, 36, 37], "rasckha": 31, "rashcka": [33, 34, 38, 39], "rashkca": [38, 39], "rate": [1, 2, 3, 4, 8, 9, 10, 12, 13, 18, 26, 33, 35, 36, 37, 38], "rather": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 24, 28, 31, 32, 33, 35, 36, 38, 39], "ratio": [4, 7, 9, 10, 11, 36, 37], "rational": [0, 31], "ravel": [5, 6, 7, 8, 9, 10, 11, 13, 24, 35, 36, 37, 39], "raw": [3, 34], "rbf": [8, 11, 12, 37, 38], "rbf_kernel_svm_clf": 8, "rbf_pca": 11, "rc": 28, "rcond": [0, 31, 32], "rcparam": [1, 3, 7, 8, 9, 10, 28, 31, 36, 39], "re": [2, 4, 13, 15, 33], "reach": [1, 4, 5, 6, 9, 10, 12, 13, 14, 33, 34, 35, 36, 38, 39], "react": [], "read": [0, 2, 3, 4, 5, 6, 7, 8, 11, 12, 16, 17, 19, 20, 24, 25, 26, 28, 30, 33], "read_csv": [0, 6, 7, 9, 35, 36], "read_fwf": [0, 31], "reader": [0, 6, 20, 24, 28, 31, 32, 34], "readi": [0, 1, 5, 6, 8, 10, 11, 12, 24, 31, 38, 39], "readili": [1, 39], "readm": [15, 20, 25, 26], "readthedoc": 23, "real": [0, 1, 4, 7, 10, 11, 12, 16, 18, 19, 24, 32, 35, 36, 37, 39], "real_loss": 4, "real_output": 4, "realist": [8, 31], "realiti": 28, "realiz": [1, 12, 37, 39], "realli": [0, 1, 31, 39], "rearrang": 13, "reason": [0, 1, 3, 4, 10, 13, 30, 31, 33, 34, 39], "reassign": 1, "reat": 39, "reber": 39, "recal": [5, 6, 9, 10, 11, 12, 22, 24, 28, 31, 32, 33, 34, 35, 36, 38, 39], "recarrai": [], "recast": 3, "receiv": [1, 3, 10, 12, 28, 37, 38, 39], "recent": [0, 6, 13, 30, 34, 35, 36, 38, 39], "recept": [3, 12, 37, 38], "receptive_field": 3, "recip": [0, 6, 7, 24, 25, 26, 31, 32, 36, 37], "reciproc": 5, "recogn": [0, 4, 5, 10, 31, 35], "recognit": [0, 1, 3, 12, 30, 31, 37, 38, 39], "recommen": 31, "recommend": [0, 2, 3, 4, 5, 6, 8, 13, 15, 19, 20, 21, 22, 23, 24, 25, 26, 30, 33, 34, 35, 36, 37, 38], "reconsid": 9, "reconstruct": 11, "record": [10, 25, 26, 27, 29, 31, 36, 37], "recreat": [15, 21], "rectangl": [9, 13, 33], "rectangular": [5, 32, 33], "rectifi": [1, 3, 12, 37, 39], "recur": [0, 23, 31], "recurr": [0, 1, 23, 31, 39], "recurs": [9, 23, 24, 31], "red": [0, 3, 4, 6, 8, 9, 34, 35], "redefin": [0, 10, 31, 32, 33], "redefinit": 33, "redistribut": [], "reduc": [1, 3, 5, 6, 9, 10, 11, 13, 21, 31, 33, 34, 35, 39], "reduct": [0, 10, 11, 23, 28, 31, 32], "reegress": 25, "ref": 20, "refer": [0, 1, 2, 3, 5, 6, 11, 12, 13, 14, 20, 24, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39], "referansestil": 20, "referenc": [2, 38, 39], "refin": [12, 37, 38], "refit": [6, 35, 36], "reflect": [0, 1, 4, 5, 25, 26, 28, 31, 39], "refresh": [23, 31], "refreshprogrammingskil": 31, "reg": [10, 11], "regard": [1, 9, 13, 39], "regardless": [12, 16, 37, 39], "regexp": [], "reggi": [], "regim": 34, "region": [3, 4, 6, 9, 12, 25, 34, 37, 38], "regist": [6, 28], "reglasso": [5, 33], "regr_1": [0, 9], "regr_2": [0, 9], "regr_3": [0, 9], "regress": [1, 8, 11, 12, 16, 20, 23, 24, 38, 39], "regressor": [0, 7, 10, 36, 39], "regret": [], "regridg": [0, 5, 6, 32, 33, 34], "regular": [0, 3, 4, 5, 6, 7, 9, 13, 17, 18, 26, 29, 31, 32, 33, 34, 35, 36, 37], "regularli": 15, "reilli": [0, 30, 31], "reinforc": [0, 8, 23, 31], "reiniti": 39, "reiter": 1, "reitz": [], "reject": 7, "rel": [0, 4, 6, 7, 9, 12, 13, 21, 28, 31, 32, 34, 35, 36, 37, 39], "relat": [0, 1, 3, 4, 5, 11, 13, 14, 19, 24, 28, 31, 32, 33, 35, 38, 39], "relationship": [0, 4, 9, 18, 31], "relativeerror": [0, 31, 32], "releas": [1, 23, 31, 39], "relev": [0, 1, 5, 7, 11, 23, 25, 26, 28, 31, 33, 34], "reli": [0, 6, 8, 34], "reliabilti": [25, 26], "reliabl": [7, 28, 36, 37], "relu": [3, 4, 21, 22, 26, 31], "relu_d": 22, "remain": [1, 2, 4, 6, 12, 24, 28, 32, 34, 35, 36, 37, 38, 39], "remaind": 28, "reman": 2, "remark": [1, 39], "rememb": [0, 8, 13, 20, 21, 22, 24, 25, 26, 31, 34], "remind": [0, 5, 11, 13, 19, 24, 28, 35], "remot": 15, "remov": [4, 5, 6, 18, 32, 33, 34], "renam": 15, "render": [0, 31, 32], "reorder": [5, 7, 32, 33, 36, 37], "reorgan": [0, 31], "repeat": [0, 1, 3, 4, 5, 6, 9, 10, 11, 13, 14, 24, 25, 28, 31, 32, 33, 34, 35, 36, 38, 39], "repeated": 31, "repeatedli": [0, 6, 10, 13, 35, 36], "repet": 3, "repetit": [6, 31, 32, 35, 36], "rephras": [13, 33], "replac": [0, 1, 3, 4, 5, 6, 10, 12, 14, 23, 25, 31, 32, 33, 35, 36, 38, 39], "replica": [6, 35], "repo": [15, 25, 26], "report": [31, 34, 36, 37], "repositori": [4, 20, 25, 26, 31], "reposotori": [], "repres": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 12, 13, 25, 26, 28, 31, 32, 33, 34, 35, 36, 37, 38, 39], "represent": [0, 1, 3, 6, 28, 31, 35, 36, 39], "representd": 3, "reproduc": [0, 5, 6, 9, 12, 15, 16, 18, 20, 23, 25, 26, 28, 31, 32, 38, 39], "repuls": [0, 31], "request": [0, 13, 34], "requir": [0, 1, 3, 4, 5, 6, 8, 9, 11, 12, 13, 15, 17, 18, 19, 20, 24, 25, 26, 31, 32, 33, 34, 35, 36, 37, 38, 39], "rerun": 39, "res1": 2, "res2": 2, "res3": 2, "res_analyt": 2, "res_analytical1": 2, "res_analytical2": 2, "res_analytical3": 2, "resaml": 6, "resampl": [0, 7, 10, 23, 31, 32, 39], "rescal": [0, 11, 12, 34, 37], "rescu": 5, "reseach": 6, "research": [0, 4, 13, 21, 22, 23, 26, 30, 31, 34], "resembl": [6, 28, 35], "reserv": [1, 5, 6, 28, 35, 36, 39], "reset": 39, "reset_weight": 39, "reshap": [0, 1, 2, 3, 4, 6, 8, 9, 10, 24, 31, 32, 35, 36, 39], "resid": 34, "residenti": [], "residu": [0, 5, 13, 31], "resiz": [5, 32, 33], "resnet": 34, "resort": 34, "resourc": [31, 34], "respect": [0, 1, 2, 3, 5, 6, 7, 8, 10, 11, 12, 13, 14, 16, 17, 18, 21, 25, 26, 28, 31, 32, 33, 34, 35, 36, 37, 38, 39], "respond": [12, 37, 38], "respons": [0, 7, 9, 12, 31, 32, 36, 37, 38], "rest": [0, 5, 18, 21, 22, 32, 33, 34], "restat": [0, 12, 31], "restor": 4, "restored_discrimin": 4, "restored_gener": 4, "restrict": [0, 3, 9, 12, 31, 37, 38, 39], "result": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 28, 31, 34, 35, 36, 37], "retail": [], "retain": [5, 6, 32, 33, 34, 35, 36], "rethink": 35, "return": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 11, 13, 14, 16, 17, 21, 22, 24, 28, 31, 32, 33, 34, 35, 36, 37, 38, 39], "return_data": 14, "return_sequ": 4, "return_x_i": 9, "reus": [1, 3, 6, 19, 20, 22, 25, 26, 38, 39], "reveal": [0, 12, 31, 37, 38], "revers": [1, 22, 24, 39], "review": [23, 24], "revis": [], "revisit": 14, "revolut": 31, "reward": [0, 4, 31], "rewrit": [0, 3, 5, 6, 7, 8, 10, 11, 12, 13, 16, 19, 24, 25, 28, 33, 34, 36, 37, 38, 39], "rewritten": [2, 6, 8, 10, 28, 35], "rewrot": [13, 36, 37], "rf": 10, "rgb": 3, "rgoj5yh7evk": 23, "rh": [6, 35], "rho": [0, 10, 13, 34, 39], "rho2": 39, "rho_1": 10, "rho_2": 10, "rho_m": 10, "rich": [0, 31], "rid": [], "ride": 9, "rideclass": 9, "ridedata": 9, "ridg": [7, 11, 13, 20, 23, 26, 31, 35, 36, 37], "ridge_paramet": 17, "ridge_sk": 6, "ridgebeta": 33, "ridgetheta": 5, "right": [0, 1, 2, 3, 5, 6, 7, 8, 9, 10, 12, 13, 14, 16, 17, 19, 21, 22, 24, 25, 26, 28, 31, 32, 33, 34, 35, 36, 37, 38, 39], "right_sid": 2, "rightarrow": [0, 1, 5, 6, 8, 11, 12, 13, 28, 31, 32, 33, 34, 35, 37, 38, 39], "rigor": [0, 31, 32, 33], "ring": 6, "rise": [0, 31], "risk": [0, 13, 31, 33, 34], "rival": 4, "river": [], "rlm": 31, "rm": [28, 34, 39], "rms_prop": 39, "rmse": [], "rmsporp": [13, 34], "rmsprop": [1, 3, 4, 13, 25, 26, 35, 38, 39], "rnd_clf": 10, "rng": [28, 36, 37], "rnn": [4, 12, 37, 38], "rnn1": 4, "rnn2": 4, "rnn_2layer": 4, "rnn_input": 4, "rnn_output": 4, "rnn_train": 4, "rntrick1": 28, "rntrick2": 28, "rntrick3": 28, "rntrick4": 28, "ro": [0, 13, 31, 33, 34], "robert": [19, 25, 30], "robust": [0, 31, 34], "robustscal": [0, 32, 34], "roc": [7, 10], "role": [0, 2, 5, 6, 8, 18, 23, 25, 31, 32, 33, 34, 35, 36, 38, 39], "roll": 6, "ronach": [], "room": [0, 29, 31], "root": [0, 5, 9, 13, 15, 28, 32, 33, 34, 38], "root_directori": [], "rot": 31, "rotat": [1, 8, 9, 10], "rotation_matrix": 9, "roughli": [1, 3, 18, 39], "round": [7, 9, 13, 37, 39], "routin": [13, 24, 31, 33], "row": [0, 1, 2, 5, 6, 9, 11, 16, 21, 24, 31, 32, 33, 35, 39], "rr": [5, 32, 33], "rrr": [5, 32, 33], "rubric": [], "rudg": [], "rug": [13, 33, 34], "rule": [0, 1, 5, 6, 13, 22, 25, 31, 32, 33, 37], "run": [0, 1, 2, 4, 5, 6, 8, 9, 11, 13, 15, 20, 21, 22, 23, 25, 26, 31, 32, 33, 34, 35, 36, 39], "rung": 26, "runtim": [1, 6, 14, 15, 39], "rust": [0, 23, 24, 31], "rvert": [1, 39], "rvert_2": [1, 39], "s41467": 26, "s_": [3, 6], "s_1": 6, "s_i": [6, 7, 36], "s_j": 6, "s_k": 6, "s_phenomenon": 25, "saddl": [13, 33, 34], "safeguard": [18, 34], "saga": 26, "sai": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 19, 24, 25, 26, 28, 31, 32, 33, 34, 35, 36, 37, 38, 39], "said": [6, 9, 13, 33], "sake": [0, 5, 7, 11, 31, 32, 33, 36, 37, 38, 39], "sale": [0, 31], "sam": 31, "same": [0, 1, 2, 3, 4, 5, 6, 8, 9, 11, 12, 14, 15, 16, 18, 20, 21, 22, 24, 25, 26, 28, 31, 32, 33, 37, 38, 39], "samm": 10, "sampl": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 13, 14, 18, 19, 23, 24, 25, 28, 31, 32, 34, 35, 36, 37, 39], "sample_vari": 14, "sampleexptvari": 28, "samples_per_class": [36, 37], "samwis": 31, "sandbox": [], "sandboxmod": [21, 22], "sasha": [], "sastri": 11, "satisfactori": [0, 31], "satisfi": [1, 2, 3, 6, 8, 13, 24, 28, 33, 35, 39], "satur": [1, 6, 35, 36, 39], "save": [0, 4, 6, 7, 9, 13, 20, 22, 31, 34, 35, 36], "save_fig": [0, 6, 7, 9, 10, 31, 35, 36], "savefig": [0, 4, 6, 7, 9, 28, 31, 35, 36], "savetxt": 4, "saw": [5, 32], "scalabl": 10, "scalar": [2, 5, 6, 10, 32, 35, 38, 39], "scale": [0, 1, 3, 5, 6, 7, 8, 9, 10, 11, 12, 13, 22, 23, 24, 25, 26, 29, 31, 33, 36, 37, 39], "scale_mean": 4, "scale_std": 4, "scaler": [0, 7, 8, 9, 10, 11, 17, 25, 32, 39], "scan": [5, 7, 36, 37], "scari": 5, "scatter": [0, 1, 6, 7, 8, 9, 14, 15, 17, 21, 31, 32, 34, 35, 36], "scenario": [6, 13, 33, 34], "schedul": [13, 34], "scheduler_arg": 39, "schedulers_bia": 39, "schedulers_weight": 39, "scheme": [1, 13, 33, 34, 36, 37, 39], "schrage": 28, "sch\u00f8yen": [6, 32, 34], "scienc": [0, 1, 10, 12, 13, 23, 27, 28, 29, 30, 33, 35, 36, 37, 38, 39], "scientif": [0, 20, 23, 25, 26, 31, 36, 37], "scientist": [0, 31], "scikit": [3, 5, 6, 8, 9, 10, 13, 15, 16, 20, 21, 23, 24, 25, 26, 30], "scikit_learn": [0, 37], "scikitlearn": 31, "scikitplot": [7, 10, 37], "scipi": [0, 3, 5, 6, 13, 23, 24, 25, 31, 32, 33, 35], "scl": 6, "scm": 15, "score": [0, 1, 3, 6, 7, 9, 10, 11, 15, 16, 19, 21, 25, 26, 29, 31, 32, 34, 35, 36, 37, 39], "scores_kfold": [6, 35, 36], "scratch": [1, 13, 16, 37, 38, 39], "script": [], "sdg": [13, 34], "sdv4f4s2sb8": [33, 34], "seaborn": [0, 1, 3, 6, 7, 26, 31, 37, 39], "seamless": [0, 23, 25, 31], "seamlessli": 39, "search": [0, 1, 3, 5, 9, 13, 15, 31, 33, 34, 39], "sebastian": [31, 38, 39], "sebastianraschka": [26, 31], "sec": 6, "second": [0, 2, 3, 4, 5, 6, 7, 8, 9, 11, 12, 14, 15, 16, 20, 21, 22, 23, 24, 28, 29, 31, 32, 33, 35, 36, 37, 38, 39], "second_correct": 39, "second_mo": 34, "second_term": 34, "secondari": 34, "secondeigvector": 11, "secondli": [12, 38, 39], "section": [4, 11, 16, 20, 24, 25, 28, 32, 34, 36], "sector": 0, "see": [0, 1, 2, 3, 4, 5, 6, 7, 8, 10, 11, 12, 13, 15, 16, 18, 19, 20, 21, 22, 23, 24, 25, 26, 28, 31, 32, 33, 34, 35, 36, 37, 38, 39], "seed": [0, 1, 2, 3, 4, 5, 6, 8, 9, 11, 13, 14, 18, 20, 21, 25, 26, 28, 31, 32, 33, 34, 35, 36, 38, 39], "seed_imag": 4, "seek": [1, 2, 8, 39], "seem": [1, 3, 4, 34, 39], "seemingli": [0, 31], "seen": [0, 1, 3, 5, 10, 12, 28, 39], "segment": [13, 33, 39], "seismic": 6, "seldomli": [0, 31], "select": [1, 5, 6, 8, 9, 10, 11, 15, 20, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 39], "selevet": 15, "self": [1, 5, 22, 32, 36, 37, 39], "sell": 4, "semest": [7, 27, 37], "semi": [8, 13, 33, 34], "semilogx": 6, "send": [5, 12, 13, 21, 22, 29, 31, 37, 38], "senior": [27, 29], "sens": [0, 4, 6, 8, 21, 31, 35], "sensibl": [3, 21], "sensit": [0, 5, 6, 9, 13, 31, 32, 34, 35, 36], "sent": [2, 21, 38, 39], "sentdex": [37, 38, 39], "sentenc": [4, 12, 37, 38], "separ": [0, 1, 2, 4, 6, 8, 9, 12, 14, 18, 21, 22, 23, 25, 28, 31, 34, 35, 37, 38, 39], "septemb": [18, 25, 31], "sequenc": [3, 4, 7, 9, 10, 12, 13, 23, 24, 28, 31, 33, 36, 37, 38], "sequenti": [1, 3, 4, 10, 12, 28, 37, 38, 39], "seri": [0, 1, 2, 3, 4, 5, 6, 10, 11, 12, 13, 24, 31, 32, 33, 35, 37, 38, 39], "serif": [7, 28, 31, 36], "serv": [0, 1, 2, 3, 5, 7, 13, 26, 30, 31, 32, 33, 34, 36, 37, 39], "servic": [25, 26], "session": [1, 15, 20, 25, 26, 27, 29, 31], "set": [1, 4, 5, 6, 7, 8, 10, 11, 13, 14, 16, 17, 18, 21, 22, 23, 24, 25, 26, 28, 29, 34, 35, 36, 37], "set_major_formatt": 6, "set_major_loc": 6, "set_tick": [1, 8], "set_ticklabel": 1, "set_titl": [0, 1, 2, 3, 7, 12, 14, 31, 36, 37, 39], "set_xlabel": [0, 1, 2, 3, 7, 12, 31, 36, 37, 39], "set_xlim": [7, 12, 36, 37, 39], "set_xticklabel": 1, "set_ylabel": [0, 1, 2, 3, 7, 31, 37, 39], "set_ylim": [7, 12, 36, 37, 39], "set_ytick": [7, 37], "set_yticklabel": [1, 6], "set_zlim": 6, "seth": 4, "setminu": 6, "setosa": [8, 9], "setosa_or_versicolor": 8, "setp": [6, 35, 36], "setup": [1, 4, 6, 8, 22, 23, 26, 31, 32, 33, 38, 39], "sever": [0, 3, 5, 6, 7, 8, 9, 11, 12, 13, 16, 23, 24, 25, 26, 28, 31, 32, 33, 34, 35, 36, 37, 38], "sgd": [1, 3, 33, 39], "sgd_clf": 8, "sgdclassifi": 8, "sgdreg": 13, "sgdregressor": 13, "sgn": [5, 32, 33], "shall": [], "shallow": [13, 34], "shape": [0, 1, 3, 4, 5, 6, 7, 8, 9, 10, 11, 13, 14, 15, 16, 18, 21, 22, 24, 31, 32, 33, 34, 35, 36, 37, 38, 39], "share": [1, 3, 15, 31, 39], "share_mask": [], "shareabl": 15, "she": [7, 36, 37], "sheppard": [], "shibukawa": [], "shift": [1, 6, 12, 15, 18, 28, 32, 34, 37, 39], "ship": 3, "shire": 31, "short": [4, 5, 20, 25, 26, 39], "shortcom": [13, 33, 34], "shorten": 4, "shorter": 28, "shorthand": [31, 35], "shortli": [24, 31], "should": [0, 2, 3, 5, 6, 8, 9, 11, 12, 15, 18, 19, 20, 21, 22, 24, 25, 26, 28, 31, 32, 34, 35, 36, 38], "shouldn": [], "show": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 19, 20, 24, 25, 28, 31, 32, 33, 34, 35, 36, 37, 38, 39], "show_shap": 4, "shown": [0, 4, 5, 8, 12, 13, 24, 32, 33, 34, 37, 38, 39], "shrink": [3, 5, 6, 8, 11, 32, 33, 34], "shrinkag": [5, 6, 32, 33], "shrunk": 11, "shuffl": [0, 1, 4, 6, 13, 32, 34, 35, 36, 39], "sickit": [38, 39], "side": [0, 2, 5, 8, 12, 13, 24, 25, 26, 31, 33, 36, 37, 39], "sigh": [23, 31], "sigma": [0, 1, 5, 6, 7, 10, 11, 12, 13, 19, 24, 25, 28, 31, 32, 33, 34, 35, 36, 37, 38, 39], "sigma0": 28, "sigma1": 28, "sigma2": 28, "sigma_": [5, 24, 31, 32, 33, 35], "sigma_0": [5, 32, 33], "sigma_1": [5, 32, 33, 38, 39], "sigma_2": [5, 32, 33, 38, 39], "sigma_fn": [7, 12, 36, 37, 39], "sigma_i": [0, 5, 31, 32, 33], "sigma_j": [5, 32, 33], "sigma_m": [6, 28, 35], "sigma_n": [11, 28], "sigma_t": 13, "sigma_x": 28, "sigmoid": [1, 2, 4, 7, 8, 10, 12, 21, 22, 26, 36, 37, 38], "sigmoid_autograd": 22, "sigmoid_d": 22, "sigmundson": [6, 32, 34], "sign": [1, 2, 7, 8, 10, 26, 28, 29, 36, 39], "signal": [1, 3, 10, 12, 34, 37, 38, 39], "signifi": 4, "signific": [1, 34, 39], "significantli": [1, 13, 18, 28, 33, 34, 39], "sim": [4, 5, 6, 13, 19, 28, 35], "similar": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 14, 18, 23, 24, 25, 26, 31, 33, 35, 36, 37, 38, 39], "similarli": [0, 1, 3, 5, 8, 10, 13, 28, 31, 32, 33, 34, 38, 39], "similiar": 39, "simpl": [1, 2, 3, 5, 6, 7, 8, 10, 11, 12, 14, 16, 17, 22, 23, 24, 26, 28, 35, 37], "simple_plot": [], "simplefilt": 39, "simplepredict": 10, "simpler": [0, 1, 5, 6, 7, 13, 16, 23, 25, 26, 31, 33, 34, 39], "simplernn": 4, "simplest": [0, 1, 3, 4, 9, 10, 12, 14, 25, 31, 37, 38, 39], "simpletre": 10, "simpli": [0, 1, 2, 4, 5, 6, 8, 9, 10, 11, 12, 23, 24, 25, 26, 28, 31, 32, 33, 34, 35, 36, 37, 38, 39], "simplic": [2, 5, 6, 7, 8, 9, 10, 11, 12, 14, 32, 33, 34, 36, 37, 38, 39], "simplicti": [5, 32, 33], "simplif": 38, "simplifi": [0, 6, 9, 18, 22, 23, 25, 31, 32, 34, 35, 36, 38], "simplist": [3, 6, 28, 35], "simul": [6, 18, 34, 35, 36], "simultan": [6, 34, 35, 36], "sin": [0, 1, 2, 3, 4, 9, 12, 13, 24, 31, 37, 39], "sinc": [0, 1, 2, 3, 5, 6, 7, 8, 9, 10, 11, 13, 16, 18, 21, 22, 24, 25, 28, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39], "sine": [3, 12, 37, 39], "singl": [0, 1, 2, 3, 5, 6, 7, 8, 9, 12, 13, 18, 19, 21, 22, 24, 28, 31, 32, 33, 34, 35, 36, 39], "singular": [0, 6, 13, 24, 31, 35], "sinusoid": 3, "site": [0, 25, 26, 27, 32], "situat": [0, 4, 5, 7, 13, 28, 31, 32, 33, 34, 36, 37], "six": [3, 28, 38], "size": [0, 1, 2, 3, 4, 5, 6, 8, 9, 10, 11, 13, 18, 20, 21, 24, 25, 28, 31, 35, 36, 37, 38, 39], "sizesp": 34, "skeleton": 22, "sketch": 10, "ski": 9, "skill": 0, "skip": 11, "skl": [0, 6, 31, 32, 34], "sklearn": [0, 1, 3, 5, 6, 7, 8, 9, 10, 11, 13, 14, 15, 17, 19, 20, 21, 22, 26, 31, 32, 33, 34, 35, 36, 37, 39], "skplt": [7, 10, 37], "skrankefunct": 39, "sl": [6, 32, 34], "slack": 8, "slender": [], "slice": [2, 24, 31], "slide": [0, 3, 16, 25, 26, 28, 31, 32, 33, 38, 39], "slight": [6, 13, 35, 36], "slightli": [1, 2, 3, 5, 6, 7, 10, 28, 32, 33, 35, 36, 37, 38, 39], "slope": [8, 11, 12, 37], "slow": [0, 2, 8, 13, 18, 32, 33, 34], "slower": [5, 24, 31, 32, 33, 34], "slowest": 24, "slowli": [12, 34], "slp": [1, 39], "small": [0, 1, 2, 3, 5, 6, 8, 9, 10, 11, 12, 13, 18, 21, 22, 23, 24, 28, 31, 32, 33, 34, 35, 36, 37, 38, 39], "smaller": [0, 1, 2, 5, 6, 8, 9, 11, 13, 21, 28, 31, 32, 33, 34, 35, 36, 39], "smallest": [0, 4, 14, 31], "smallest_row_index": 14, "smodin": [], "smooth": [0, 3, 6, 13, 25, 31, 33, 34], "smoother": 34, "sn": [0, 1, 3, 6, 7, 31, 37, 39], "sne": 11, "so": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 18, 19, 20, 21, 22, 23, 24, 25, 26, 28, 29, 31, 32, 33, 34, 35, 36, 37, 38, 39], "soar": 6, "social": 0, "soft": [1, 7, 10, 12, 36, 37, 38, 39], "soften": 8, "softmax": [3, 7, 21, 22, 26, 36, 37], "softmax_vec": 21, "softwar": [0, 8, 23, 24, 38], "sokogskriv": 20, "sol": 8, "sol1": 21, "sole": [0, 6, 31], "solid": [0, 7, 36, 37], "solut": [0, 1, 2, 3, 5, 6, 8, 10, 11, 13, 18, 21, 24, 25, 26, 28, 31, 32, 33, 34, 35, 39], "solution_ev": 34, "soluton": 2, "solv": [0, 1, 3, 5, 6, 8, 10, 11, 12, 13, 16, 24, 25, 26, 31, 32, 38, 39], "solve_expdec": 2, "solve_ode_deep_neural_network": 2, "solve_ode_neural_network": 2, "solve_pde_deep_neural_network": 2, "solveod": 2, "solveode_popul": 2, "solver": [2, 7, 8, 9, 10, 24, 26, 31, 37], "some": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 14, 15, 16, 18, 19, 21, 22, 25, 26, 28, 31, 34, 35, 37, 39], "some_model": [6, 32, 34], "somehow": 4, "someon": 16, "someth": [0, 1, 3, 4, 7, 9, 11, 15, 19, 20, 25, 26, 28, 31, 32, 37, 39], "sometim": [0, 1, 11, 12, 13, 14, 19, 32, 34, 37, 38, 39], "somewhat": [26, 37], "soon": [24, 29, 32], "sophist": [0, 31], "sopt": 13, "sort": [5, 6, 9, 11, 28, 35, 36], "sound": [3, 5], "sourc": [0, 1, 3, 6, 23, 24, 25, 26, 28, 31, 34, 35, 36, 39], "source1": 22, "source2": 22, "space": [0, 1, 4, 5, 8, 9, 11, 12, 13, 14, 28, 32, 33, 34, 36, 37, 38, 39], "span": [0, 3, 5, 9, 11, 24, 31, 32, 33], "spare": [1, 39], "spars": [3, 6, 18, 24, 31, 34], "sparse_mtx": [24, 31], "sparsecategoricalcrossentropi": 3, "sparsiti": [10, 18], "spatial": [1, 2, 3, 12, 37, 38, 39], "speak": 28, "special": [6, 7, 10, 12, 13, 24, 28, 31, 32, 33, 34, 36, 37, 38, 39], "specif": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 11, 12, 15, 16, 23, 24, 25, 26, 28, 30, 31, 32, 33, 35, 36, 37, 38, 39], "specifi": [0, 3, 5, 6, 7, 9, 11, 13, 14, 28, 31, 33, 34, 35, 36, 37, 39], "specifici": [0, 10, 31], "spectacular": 3, "spectral": 1, "speech": [0, 1, 3, 4, 12, 37, 38, 39], "speed": [1, 2, 4, 13, 39], "spend": [16, 28, 34], "spent": [25, 26], "sphere": [0, 32, 34], "sphinx": [], "sphinx_book_them": [], "sphinxcontrib": [], "spike": 34, "spin": 6, "spite": 0, "spitzer": [], "spline": 8, "split": [1, 3, 4, 5, 6, 8, 9, 10, 11, 14, 16, 17, 20, 21, 22, 25, 26, 28, 31, 33, 34, 35, 36, 39], "splite": 0, "splitter": [1, 10], "spoiler": [], "spontan": 28, "spot": 3, "spread": [0, 11, 28, 31, 32, 36, 37], "spring": 39, "springer": [19, 25, 30, 31, 35, 36], "spuriou": [13, 34], "sqquar": 33, "sqrsignal": 3, "sqrt": [3, 4, 5, 6, 8, 10, 11, 13, 28, 32, 33, 34, 35, 38, 39], "squar": [1, 2, 3, 4, 7, 8, 9, 11, 13, 14, 15, 17, 18, 23, 24, 26, 28, 35, 36, 37, 38, 39], "squarederror": 10, "squaredeuclidean": 14, "squash": [12, 37, 39], "src": [], "srtm": 6, "srtm_data_norway_1": 6, "sso": 20, "stabil": [5, 25, 26, 34, 36, 37], "stabl": [0, 4, 5, 6, 9, 16, 20, 23, 25, 31, 32, 33, 34], "stack": [3, 4], "stage": [5, 13, 15, 25, 26, 34, 38, 39], "stagnat": 34, "stai": [0, 2, 4, 5, 11, 31, 32, 34, 39], "stand": [0, 5, 9, 12, 31, 32, 33, 37], "standard": [0, 1, 4, 5, 6, 7, 8, 10, 12, 17, 18, 19, 24, 25, 26, 28, 31, 33, 34, 36, 37, 38, 39], "standardscal": [0, 6, 7, 8, 9, 10, 11, 17, 32, 34], "standpoint": 34, "stanford": [13, 33], "start": [0, 1, 2, 3, 4, 5, 6, 8, 9, 10, 11, 12, 13, 14, 15, 16, 18, 21, 22, 24, 26, 28, 29, 31, 32, 33, 34, 35, 36, 38, 39], "start_tim": 14, "starter": [], "stat": [6, 35], "state": [1, 2, 4, 5, 6, 7, 8, 10, 11, 12, 13, 23, 28, 31, 32, 33, 35, 36, 37, 38, 39], "statement": [0, 7, 24, 31, 37], "static": [], "stationari": [33, 34], "statist": [0, 1, 3, 4, 7, 9, 10, 11, 12, 13, 14, 19, 24, 25, 30, 32, 33, 34, 37, 38, 39], "statu": [0, 7, 15, 31, 36, 37], "stavang": 6, "stb": [], "std": [0, 4, 6, 18, 31, 32, 34, 35, 36], "stdout": 39, "steep": [13, 33, 34], "steepest": 34, "stefan": [], "step": [0, 1, 2, 4, 6, 7, 9, 10, 11, 12, 13, 14, 15, 18, 22, 24, 25, 31, 33, 37, 38, 39], "step_fn": [7, 12, 36, 37, 39], "step_length": [13, 34], "step_siz": 34, "steps_list": 9, "stereo": 3, "sticki": [], "still": [0, 2, 3, 5, 6, 11, 13, 21, 22, 26, 28, 32, 33, 34, 35, 36, 38], "stimuli": [12, 37, 38], "stk": [30, 31], "stk2100": [30, 31], "stk3155": [15, 25, 26, 27, 29], "stk4021": [30, 31], "stk4051": [30, 31], "stk4155": [27, 29], "stk5000": 30, "stochast": [0, 1, 5, 6, 8, 11, 12, 22, 26, 33, 35, 36, 38, 39], "stock": 4, "stoke": [12, 37, 38], "stone": [0, 7, 36, 37, 38], "stop": [1, 4, 9, 13, 14, 18, 33, 38, 39], "storag": [5, 32, 33], "store": [0, 1, 2, 3, 6, 11, 13, 22, 28, 31, 34, 39], "storehaug": [29, 31], "stori": [], "str": [1, 3, 4, 39], "straight": [0, 6, 8, 13, 31, 33, 35], "straightforward": [0, 2, 3, 5, 6, 8, 9, 10, 13, 24, 31, 32, 33, 35], "strategi": [0, 1, 9, 31, 39], "stratifi": [6, 35, 36], "stream": 34, "strength": [0, 5, 14, 32, 33], "stretch": 11, "strict": [8, 13, 33], "strictli": [8, 13, 33], "stride": [4, 24], "strike": 6, "string": [1, 39], "stroke": [7, 36, 37], "strong": [3, 6, 9, 10, 12, 24, 28, 34, 35, 37, 38], "strongli": [0, 8, 15, 20, 22, 23, 24, 26, 39], "stronli": [], "structur": [0, 1, 2, 3, 6, 9, 10, 12, 22, 23, 31, 35, 36, 37, 39], "stuck": [1, 13, 33, 34, 39], "student": [0, 15, 25, 26, 27, 29, 30, 31], "studi": [0, 3, 4, 5, 6, 7, 8, 11, 12, 13, 23, 25, 26, 30, 31, 32, 33, 34, 36, 38, 39], "studier": 30, "stuff": [21, 22], "style": [7, 9, 20, 24, 31], "stylesheet": [], "st\u00f8land": 29, "sub": [9, 12, 34, 37, 38], "subarrai": [], "subclass": [], "subdivid": [0, 24, 31], "subfield": 0, "subgradi": 34, "subject": [6, 8, 28], "sublicens": [], "sublinear": 34, "submit": 31, "subplot": [0, 1, 3, 4, 6, 7, 8, 9, 10, 14, 21, 31, 35, 36, 37, 39], "subplots_adjust": [8, 28], "subprogram": [24, 31], "subproject": [], "subract": [0, 32], "subroutin": [0, 31], "subscript": [1, 39], "subsequ": [1, 4, 5, 6, 12, 24, 28, 32, 33, 35, 37, 38], "subset": [1, 6, 9, 12, 13, 23, 31, 33, 34, 35, 36, 37, 38, 39], "subspac": [0, 8, 11, 32], "substanti": [9, 10, 34], "substep": 11, "substitut": [3, 6, 12, 16, 24, 35, 36, 37], "subsubset": 9, "subtask": 6, "subtl": [1, 39], "subtract": [0, 4, 5, 6, 11, 13, 18, 19, 24, 25, 28, 32, 34, 35, 36, 39], "subtre": 9, "succeed": [0, 4, 31], "success": [3, 7, 9, 13, 28, 36, 37], "successfulli": [4, 9], "succinctli": 34, "sudo": [0, 23, 25, 31], "suffer": [0, 1, 2, 5, 10, 31, 32, 33, 39], "suffici": [1, 6, 8, 11, 13, 33, 35, 36, 39], "suggest": [1, 13, 25, 26, 30, 33, 34, 39], "suit": [8, 12, 37, 38], "suitabl": [0, 15, 19, 28, 32, 34], "sum": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 19, 21, 24, 28, 31, 32, 33, 34, 37], "sum_": [0, 1, 2, 3, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 16, 19, 24, 25, 26, 28, 31, 32, 33, 34, 35, 36, 37, 38, 39], "sum_i": [0, 2, 5, 6, 8, 13, 19, 25, 32, 33, 34, 35, 36], "sum_j": [6, 18, 34], "sum_ja_": 0, "sum_k": [6, 8, 12, 24, 38, 39], "sum_logist": 13, "sum_m": 3, "sum_n": 3, "sum_nx_": 3, "summar": [5, 6, 9, 26, 35, 36], "summari": [1, 3, 4, 10, 27, 33, 34, 39], "summat": [0, 3, 16, 32, 33], "sunni": 9, "super": [5, 32, 33, 34, 39], "superfici": 3, "superscript": [1, 12, 37, 38, 39], "supervis": [0, 5, 6, 7, 9, 12, 23, 31, 32, 33, 35, 36, 37, 38], "supplement": [7, 25, 26, 36, 37], "supplementari": 26, "suppli": [], "support": [0, 1, 9, 10, 11, 13, 20, 21, 23, 31, 32, 34, 36, 37, 38, 39], "suppos": [0, 5, 6, 7, 8, 10, 11, 12, 13, 24, 31, 32, 33, 34, 35, 36, 37, 38], "suppress": [5, 13, 33], "sure": [0, 1, 4, 6, 16, 20, 21, 22, 25, 39], "surf": 6, "surfac": [0, 6, 31, 34], "surpass": 6, "surpris": [0, 31], "surround": [3, 23], "survei": [0, 5, 6, 31, 32], "svc": [8, 9, 10], "svd": [0, 6, 11, 31, 35], "svdinv": 5, "svm": [8, 9, 10, 11], "svm_clf": [8, 10], "svn": [], "swap": 21, "swath": [5, 32, 33], "switch": [0, 39], "sy": [13, 33, 34, 39], "symbol": [1, 5, 11, 13, 23, 28, 31, 32, 33, 38, 39], "symmeteri": 1, "symmetr": [0, 5, 8, 11, 12, 13, 24, 31, 32, 37, 38], "symmetri": 6, "sympi": [0, 23, 25, 31, 38], "synonim": 28, "syntax": 13, "system": [0, 1, 3, 4, 6, 7, 9, 10, 12, 13, 15, 23, 24, 25, 31, 33, 34, 36, 37, 38, 39], "systemat": [4, 6, 35, 36], "t": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 21, 22, 23, 24, 25, 26, 28, 29, 31, 33, 34, 35, 36, 37, 38, 39], "t0": [3, 6, 13, 34], "t1": [2, 13, 34], "t2": 2, "t3": 2, "t9jjwsmsd1o": 35, "t_": 2, "t_0": [2, 9, 13, 34], "t_1": [13, 34], "t_b": 10, "t_batch": 39, "t_i": [1, 2, 5, 12, 26, 32, 33, 39], "t_j": 12, "t_k": 9, "t_test": 39, "t_train": 39, "t_val": 39, "tabl": [9, 25, 26, 28, 29, 31, 37], "tabul": [0, 31], "tabular": 31, "tackl": 4, "tag": [2, 3, 4, 5, 6, 7, 12, 13, 14, 24, 28, 32, 33, 36, 37, 38, 39], "tagrget": 38, "taht": [0, 31], "tail": 28, "tailor": [2, 8, 11, 31, 38], "taiwan": [0, 31], "take": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 17, 19, 21, 22, 23, 24, 28, 31, 32, 33, 34, 35, 36, 37, 38, 39], "taken": [0, 1, 3, 6, 10, 13, 21, 24, 35, 39], "tan": 3, "tangent": [1, 4, 12, 13, 33, 37, 39], "tanh": [1, 4, 7, 8, 12, 36, 37, 39], "target": [0, 1, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 15, 16, 18, 19, 21, 22, 26, 31, 32, 33, 34, 35, 36, 37, 38, 39], "target_nam": [9, 21], "task": [0, 1, 3, 6, 9, 11, 12, 14, 21, 25, 31, 34, 35, 36, 37, 38, 39], "tau": [3, 5, 28], "taught": 31, "tax": [], "taylor": [2, 13, 33, 38], "taylornr": [13, 33], "tc": 8, "teach": [15, 27, 31, 35], "team": [1, 39], "teaser": 0, "technic": [0, 5, 6, 13, 25, 26, 33, 34, 35], "techniqu": [0, 1, 8, 10, 13, 23, 28, 30, 31, 32, 34, 35, 36, 39], "technologi": [0, 1, 39], "tell": [0, 4, 6, 10, 11, 13, 16, 28, 34, 35, 36], "temp": 1, "temp1": 1, "temp2": 1, "temperatur": [0, 9, 31], "templat": [18, 20], "temporari": [], "temporarili": [1, 39], "ten": [3, 31, 38], "tend": [3, 5, 6, 8, 9, 10, 12, 13, 14, 32, 34, 35, 36], "tendenc": [0, 31], "tension": [6, 35, 36], "tensor": 3, "tensorflow": [0, 2, 4, 8, 14, 23, 24, 25, 26, 30, 31, 32], "term": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 16, 18, 19, 22, 25, 26, 28, 31, 32, 33, 34, 36, 37], "term1": [5, 6, 11], "term2": [5, 6, 11], "term3": [5, 6, 11], "term4": [5, 6, 11], "termin": [0, 4, 5, 9, 10, 13, 15, 32, 33, 34], "terminarl": 15, "terrain": 6, "terrain1": 6, "test": [3, 4, 5, 6, 7, 8, 9, 10, 13, 16, 19, 20, 21, 24, 25, 28, 31, 33, 34, 35, 36, 37], "test_acc": 3, "test_accuraci": [1, 3, 39], "test_error": 6, "test_imag": [3, 4], "test_ind": [6, 35, 36], "test_input": 4, "test_label": [3, 4], "test_loss": 3, "test_pr": [1, 39], "test_predict": [1, 39], "test_rnn": 4, "test_scor": [7, 10, 37], "test_siz": [0, 1, 3, 5, 6, 10, 15, 17, 26, 32, 33, 34, 35, 36, 39], "test_split": 9, "testerror": [0, 6, 32, 35, 36], "testi": 4, "testpredict": 4, "testx": 4, "tex": [], "text": [0, 1, 2, 4, 5, 8, 9, 11, 13, 15, 18, 20, 24, 25, 26, 28, 30, 32, 33, 34, 35, 36, 39], "textbf": [], "textbook": [16, 25, 26, 32, 33, 35, 36], "textual": 9, "textur": 1, "tf": [1, 3, 4, 13, 14, 33, 39], "th": [0, 1, 2, 5, 6, 7, 9, 12, 13, 14, 24, 25, 28, 31, 32, 34, 35, 36, 37, 38, 39], "than": [0, 1, 2, 3, 4, 5, 6, 7, 9, 10, 11, 12, 13, 17, 21, 23, 28, 31, 32, 34, 35, 36, 37, 38, 39], "thank": [4, 6, 32, 34], "thats": 39, "theano": [1, 23, 31, 39], "thei": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 11, 12, 13, 15, 16, 18, 20, 22, 24, 25, 26, 28, 31, 32, 33, 34, 35, 36, 37, 38, 39], "them": [0, 1, 3, 4, 6, 8, 9, 10, 11, 12, 13, 18, 21, 24, 25, 26, 31, 32, 37, 38, 39], "theme": [0, 15, 31], "themselv": [0, 25, 26, 28, 31, 34], "thenc": [6, 35, 36], "theorem": [2, 6, 7, 32, 33, 36, 37, 39], "theoret": [0, 4, 10], "theori": [0, 1, 3, 8, 9, 12, 13, 19, 23, 25, 30, 31, 34, 37, 38, 39], "thereaft": [0, 5, 6, 11, 12, 24, 25, 31, 35, 36, 38, 39], "therebi": [0, 5, 7, 11, 25, 31, 32, 33, 36, 37, 38], "therefor": [0, 1, 2, 3, 4, 6, 7, 8, 11, 13, 19, 28, 31, 32, 33, 34, 35, 36, 37, 39], "therein": 11, "thereof": [0, 6, 13, 31, 34, 35], "theta": [0, 1, 4, 5, 6, 7, 13, 16, 25, 28, 31, 32, 33, 34, 36, 37, 38, 39], "theta1": 34, "theta2": 34, "theta_": [0, 1, 6, 7, 13, 31, 32, 33, 34, 36, 37, 39], "theta_0": [0, 5, 6, 7, 16, 31, 32, 33, 34, 36, 37], "theta_0x_": [0, 31, 32], "theta_1": [0, 5, 6, 7, 31, 32, 33, 34, 36, 37], "theta_1x_": [0, 31, 32], "theta_1x_0": [0, 31], "theta_1x_1": [0, 7, 31, 36, 37], "theta_1x_2": [0, 31], "theta_1x_i": [7, 32, 33, 34, 36, 37], "theta_2": [0, 31, 32], "theta_2x_": [0, 31, 32], "theta_2x_0": [0, 31], "theta_2x_1": [0, 31], "theta_2x_2": [0, 7, 31, 36, 37], "theta_2x_i": 32, "theta_3x_i": 32, "theta_4x_i": 32, "theta_closed_form": 18, "theta_closed_formol": 18, "theta_closed_formridg": 18, "theta_gdol": 18, "theta_gdridg": 18, "theta_i": [0, 1, 5, 31, 32, 33, 39], "theta_j": [0, 5, 6, 18, 31, 32, 34], "theta_k": [33, 34], "theta_linreg": [13, 33, 34], "theta_ol": 18, "theta_p": [7, 36, 37], "theta_px_p": [7, 36, 37], "theta_ridg": 18, "theta_t": [13, 34], "theta_tru": 18, "thetaand": 37, "thetaith": 34, "thetaor": 37, "thetavalu": 5, "thetaxor": 37, "thi": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 23, 24, 25, 26, 27, 28, 30, 32, 33, 34, 35, 36, 37, 39], "thing": [0, 1, 2, 4, 5, 7, 9, 15, 16, 18, 21, 22, 28, 31, 35, 37, 39], "think": [0, 1, 3, 4, 6, 9, 12, 13, 14, 28, 31, 32, 33, 34, 35, 37, 39], "third": [0, 3, 6, 13, 29, 31, 33, 34], "thirti": [7, 37], "thorughout": 31, "those": [0, 3, 5, 6, 8, 9, 10, 11, 24, 25, 26, 31, 32, 33, 34, 35, 36, 38], "though": [1, 2, 3, 4, 13, 16, 17, 19, 21, 22, 24, 28, 34, 39], "thought": [6, 14, 25, 26, 28, 35, 36], "thousand": [0, 1, 25, 32, 34, 39], "three": [0, 1, 3, 5, 6, 8, 9, 12, 21, 24, 25, 26, 27, 28, 29, 31, 32, 33, 35, 36, 37], "threshold": [1, 3, 9, 10, 11, 12, 13, 34, 36, 37, 38, 39], "through": [0, 1, 2, 3, 4, 5, 6, 8, 11, 12, 13, 14, 15, 21, 22, 23, 24, 25, 28, 31, 32, 33, 34, 35, 37, 39], "throughout": [0, 4, 5, 14, 15, 23, 24, 28, 31, 39], "throw": [3, 6, 28, 35], "thu": [0, 1, 2, 5, 6, 7, 8, 10, 11, 12, 13, 29, 31, 32, 33, 34, 35, 36, 37, 38, 39], "thumb": [0, 6, 25, 32], "thursdai": [], "tibshirani": [6, 19, 25, 30, 31, 35, 36], "tick_param": 6, "ticker": [6, 13, 28, 33, 34], "tif": 6, "tight_layout": [1, 7, 37], "tightli": 11, "tild": [0, 5, 6, 7, 11, 19, 25, 28, 31, 32, 33, 34, 35, 36, 38, 39], "till": [0, 4, 7, 8, 9, 10, 12, 24, 31, 32, 36, 37, 38, 39], "time": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 20, 21, 22, 23, 24, 25, 26, 28, 31, 32, 33, 35, 36, 37, 38, 39], "timeit": 4, "timer": 4, "timeseri": [], "tini": [1, 34, 39], "tip": 3, "titl": [0, 1, 2, 3, 4, 6, 7, 8, 9, 10, 13, 15, 20, 21, 28, 31, 33, 34, 35, 36, 39], "tm": [], "tmp": 13, "tn": [2, 3, 7], "to_categor": [1, 3, 4, 39], "to_categorical_numpi": [1, 39], "to_numer": [0, 6, 31, 35, 36], "todai": 3, "togeth": [0, 3, 6, 8, 11, 13, 22, 23, 31], "toi": 14, "token": [], "told": 13, "toler": [2, 14], "tolist": 4, "tomographi": [12, 37, 38], "too": [0, 2, 4, 5, 6, 9, 11, 13, 17, 18, 28, 30, 32, 33, 34, 35, 36], "took": [8, 31], "tool": [0, 1, 3, 6, 13, 15, 23, 32, 35, 36, 39], "toolbox": 8, "top": [0, 3, 5, 6, 9, 10, 19, 23, 31, 35], "topic": [0, 5, 6, 7, 8, 23, 25, 26, 32, 33, 35, 36, 37, 38], "topolog": [3, 12, 37, 38], "topologi": [1, 12, 39], "torkjellsdatt": [29, 31], "tort": [], "toss": [10, 28], "total": [0, 1, 2, 3, 4, 6, 7, 8, 10, 11, 12, 13, 14, 24, 26, 28, 29, 31, 32, 33, 34, 35, 36, 37, 38, 39], "total_loss": 4, "totalclustervari": 14, "totalscatt": 14, "toward": [1, 2, 7, 12, 13, 15, 33, 36, 37, 39], "towardsdatasci": 34, "town": [], "tp": [4, 7], "tpng": 9, "tpu": [13, 23, 31], "tqdm": 6, "tr": [], "track": [3, 13, 14, 15, 22, 24, 32, 33, 34], "tract": [], "tractabl": [0, 31, 32], "trade": [5, 9, 20, 26, 34, 35], "tradeoff": [0, 5, 19, 25, 31, 32, 33], "tradit": [0, 1, 4, 6, 31, 35, 36, 39], "train": [2, 3, 5, 6, 8, 9, 10, 11, 12, 13, 16, 17, 20, 25, 26, 33, 34, 35, 36, 37], "train_acc": 39, "train_accuraci": [0, 1, 3, 31, 39], "train_dataset": 4, "train_end": [0, 1, 32, 39], "train_error": [6, 39], "train_imag": [3, 4], "train_ind": [6, 35, 36], "train_label": [3, 4], "train_network": 21, "train_pr": [1, 39], "train_siz": [0, 1, 3, 32, 39], "train_step": 4, "train_test_split": [0, 1, 3, 5, 6, 7, 9, 10, 11, 15, 16, 17, 19, 26, 31, 32, 33, 34, 35, 36, 37, 39], "train_test_split_numpi": [0, 1, 32, 39], "trainable_vari": 4, "trained_model": [6, 32, 34], "trainerror": [0, 32], "traini": 4, "training_checkpoint": 4, "training_dataset": 4, "training_gradi": [13, 34], "trainingerror": [6, 35, 36], "trainpredict": 4, "trainscor": 4, "trainx": 4, "trait": [0, 31], "trajectori": [4, 34], "transfer": [9, 31], "transform": [0, 5, 6, 7, 8, 9, 10, 11, 12, 13, 17, 21, 23, 24, 31, 32, 33, 34, 35, 36, 37, 38, 39], "transit": [6, 12, 37, 38], "translat": [1, 4, 6, 10, 31, 32, 34, 39], "transpos": [1, 5, 11, 21, 24, 32, 33, 39], "travers": [0, 5], "travi": [], "treat": [0, 1, 3, 6, 12, 13, 18, 21, 28, 31, 32, 33, 34, 35, 36, 37, 38, 39], "tree": [0, 1, 23, 31, 39], "tree_clf": [9, 10], "tree_clf_": 9, "tree_clf_sr": 9, "tree_reg": 9, "tree_reg1": 9, "tree_reg2": 9, "trend": 28, "treue": 7, "trevor": [19, 25, 30], "tri": [2, 3, 4, 9, 13, 16, 34], "triain": 0, "trial": [0, 2, 4, 6, 13, 28, 31, 33, 34, 35, 36], "triangl": [13, 33], "triangular": 24, "trick": [3, 4, 8, 11, 13, 28, 34], "tricki": 22, "trickier": 28, "tridiagon": 24, "trillion": 23, "trim": [], "trivial": [0, 1, 5, 11, 28, 31, 33, 39], "troffa": [], "troubl": [0, 8, 12, 15, 21, 22, 32, 34, 38, 39], "truck": 3, "true": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 12, 13, 14, 16, 17, 18, 19, 21, 22, 24, 25, 28, 31, 32, 33, 34, 35, 36, 37, 39], "true_beta": 32, "true_fun": [6, 35, 36], "true_theta": [6, 34], "truelabel": [36, 37], "truli": 31, "truncat": 38, "try": [0, 1, 2, 4, 5, 6, 7, 8, 9, 10, 11, 13, 14, 15, 18, 21, 22, 23, 24, 25, 26, 28, 31, 32, 33, 34, 36, 37, 38, 39], "tr\u00f6ger": [], "tucker": 8, "tuesdai": [29, 31, 36], "tumor": [7, 9, 36, 37], "tumour": [7, 37], "tunabl": 1, "tune": [4, 9, 13, 24, 31, 34], "tupl": [21, 39], "turn": [0, 1, 5, 6, 7, 8, 9, 10, 11, 12, 13, 24, 25, 28, 31, 32, 33, 34, 35, 36, 37, 38, 39], "tutori": [1, 4, 39], "tv": 2, "tveito": 2, "tvw1zdmznwm": 37, "tweak": [1, 4, 10, 28, 39], "twice": [13, 33], "twist": 11, "two": [0, 1, 2, 4, 5, 6, 7, 9, 10, 11, 12, 13, 15, 17, 21, 24, 25, 26, 27, 28, 30, 31, 32, 33, 34, 35], "tx": [13, 33, 34, 37], "tx_1": [13, 33], "txt": [4, 15, 20, 25, 26], "ty": [13, 33], "type": [0, 1, 3, 6, 8, 10, 13, 21, 24, 28, 32, 33, 34, 35, 39], "typeset": 20, "typic": [0, 1, 2, 3, 4, 5, 7, 9, 10, 12, 13, 15, 16, 20, 28, 31, 32, 33, 34, 35, 36, 37, 38, 39], "typo": [25, 26], "u": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 21, 24, 25, 26, 28, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39], "u_": 24, "u_i": [12, 37], "u_m": 10, "ua": [0, 31], "ubuntu": [0, 23, 25, 31], "uci": [25, 26], "ufunc": [], "uio": [15, 20, 21, 25, 26, 29, 30], "uk": [], "un": 14, "unabl": [15, 21], "unari": [24, 31], "unbalanc": [6, 9, 35, 36], "unbias": [0, 5, 6, 31, 35], "uncent": [6, 32, 34], "uncertainti": [0, 5, 31], "uncertitud": 28, "unchang": [1, 3, 39], "uncom": [], "uncorrel": [10, 28], "undefin": [5, 32, 33], "under": [0, 1, 5, 6, 10, 13, 23, 25, 31, 32, 33, 34, 35, 39], "underdetermin": [0, 31], "underfit": [1, 6, 35, 36, 39], "underflowproblem": [5, 35], "undergo": [5, 21], "undergradu": [27, 29], "underli": [0, 1, 9, 13, 18, 28, 31, 34, 39], "underlin": [], "underscor": [], "underset": [4, 14], "understand": [0, 1, 3, 5, 6, 10, 13, 14, 15, 19, 20, 21, 23, 31, 32, 33, 34, 38, 39], "understood": [8, 13], "underwai": [], "undesir": 8, "undetermin": [5, 8, 35], "undo": 4, "unexpect": [6, 35], "unexpected": 28, "unexplain": 18, "unfair": [6, 32], "unfortun": [1, 8, 9, 10, 39], "unicode_liter": [8, 9], "uniform": [0, 1, 5, 6, 11, 13, 25, 28, 31, 33, 34, 36, 37, 39], "uniformli": [13, 28, 33, 34], "unifrompdf": 28, "unimport": [13, 33], "union": [5, 6, 35, 36], "uniqu": [0, 2, 6, 13, 14, 24, 31, 35, 36, 37], "unique_class": [36, 37], "unique_cluster_label": 14, "unit": [0, 1, 3, 4, 5, 10, 12, 18, 28, 31, 32, 33, 34, 37, 38, 39], "unitari": [5, 6, 24, 32, 33], "unitarili": [24, 31], "uniti": 28, "univari": 28, "univers": [0, 1, 2, 13, 23, 25, 26, 27, 29, 31, 32, 33, 34, 35, 36, 37, 39], "unix": [1, 39], "unknow": [0, 24, 31], "unknown": [0, 1, 3, 4, 5, 6, 8, 10, 13, 19, 24, 25, 31, 32, 33, 34, 35, 36, 38, 39], "unknowwn": 12, "unlabel": [1, 39], "unless": [0, 3, 6, 11, 13, 25, 26, 31, 33, 35, 38], "unlik": [1, 3, 8, 13, 33, 34, 39], "unnecessarili": 9, "unord": 3, "unpickl": [], "unpleas": [], "unpublish": 34, "unravel": [1, 39], "unrol": [3, 11], "unscal": 19, "unseen": [0, 7, 9, 15, 36, 37], "unstabl": [1, 39], "unsupervis": [0, 1, 4, 12, 23, 31, 37, 38, 39], "unsymmetr": [24, 31], "until": [1, 2, 4, 9, 12, 13, 14, 21, 33, 34, 37, 39], "untouch": 0, "unusu": [12, 37, 38], "up": [1, 3, 4, 5, 6, 8, 10, 11, 13, 14, 16, 18, 19, 20, 21, 22, 23, 24, 25, 28, 29, 34, 37], "updat": [1, 2, 10, 12, 13, 14, 15, 18, 19, 21, 22, 26, 35, 36, 37], "update_chang": 39, "update_matrix": 39, "update_weight": 22, "uploa": 31, "upload": [15, 20, 23, 25, 26, 30], "upon": [0, 1, 6, 7, 11, 24, 38, 39], "upper": [0, 8, 9, 16, 24, 32], "uppercas": [24, 31], "upsampl": 4, "upscal": 4, "uptad": 38, "upward": [], "url": [31, 32, 37], "us": [4, 5, 6, 8, 9, 10, 11, 12, 14, 15, 17, 20, 21, 24, 28, 30, 35], "usag": [0, 8, 23, 31, 32, 38], "usd": [], "usd10000": [], "use_bia": 4, "usecol": [0, 31], "useless": [1, 39], "user": [0, 1, 2, 4, 6, 7, 15, 23, 24, 25, 31, 32, 36, 37, 39], "usernam": [15, 25, 26], "usetex": 28, "usg": 6, "usr": 28, "usual": [0, 3, 4, 7, 12, 13, 14, 31, 34, 36, 37, 38], "ut": 5, "utf": [], "util": [1, 3, 4, 6, 7, 10, 14, 19, 31, 35, 36, 39], "ux": 24, "v": [2, 4, 5, 6, 11, 13, 15, 23, 32, 33, 35, 36, 37, 38, 39], "v0": 28, "v1": 28, "v2": 28, "v5": [], "v8xr": [37, 38, 39], "v_": 34, "v_0": [11, 34], "v_t": 34, "va": 1, "vahid": 31, "val": 13, "val_acc": 39, "val_accuraci": 3, "val_error": 39, "val_loss": 4, "val_set": 39, "vale": 2, "valid": [0, 1, 4, 7, 9, 10, 13, 23, 28, 31, 32, 34, 37, 39], "validation_data": 3, "validation_split": 4, "valu": [0, 1, 2, 3, 4, 6, 7, 8, 9, 10, 12, 13, 14, 16, 17, 18, 20, 21, 22, 23, 24, 25, 26, 31, 34, 37, 38, 39], "valuat": 9, "valued_at_a": 39, "valued_at_z": 39, "valueerror": [], "valy": 4, "van": [0, 19, 25, 31, 32, 33, 34], "vandenbergh": [8, 13, 33], "vandermond": [0, 31], "vanilla": [0, 6, 11, 14, 32, 34], "vanish": [1, 4, 13, 28, 33, 38], "var": [5, 6, 10, 11, 19, 25, 28, 32, 35, 36], "var_x": 28, "varabl": 8, "varepsilon": [5, 6, 19, 35], "varepsilon_": [5, 6, 35], "varepsilon_i": [5, 6, 35], "vari": [0, 1, 3, 5, 6, 10, 21, 31, 35, 36, 38, 39], "variabl": [0, 1, 2, 5, 6, 7, 8, 10, 11, 12, 13, 14, 21, 24, 31, 32, 34, 35, 36, 37, 38, 39], "varianc": [0, 1, 5, 7, 9, 10, 11, 13, 14, 18, 20, 23, 24, 26, 28, 31, 32, 33, 34, 37, 39], "variance_i": [5, 11, 32], "variance_x": [5, 11, 32], "variant": [0, 1, 6, 8, 12, 13, 26, 31, 32, 33, 34, 37, 38, 39], "variat": [3, 4, 11, 31], "varieti": [0, 3, 12, 23, 25, 31, 37, 38], "variou": [1, 3, 5, 6, 7, 8, 9, 11, 12, 13, 16, 19, 20, 23, 24, 25, 28, 31, 32, 33, 34, 37, 38, 39], "varydimens": 4, "vast": 34, "vastli": 3, "vaue": 1, "vault": 0, "vdot": [2, 13, 33, 34], "ve": [25, 26, 34], "vec": [6, 35], "vector": [0, 1, 2, 3, 4, 5, 6, 7, 9, 10, 11, 13, 14, 17, 18, 21, 22, 23, 33, 34, 35, 36, 38, 39], "vector_mean": 14, "ventur": [0, 8, 23, 31], "venv": 15, "verbos": [1, 3, 4, 36, 37, 39], "veri": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 16, 18, 21, 22, 25, 26, 28, 30, 31, 32, 33, 34, 35, 36, 39], "verifi": [3, 11, 24, 31], "versatil": [8, 31], "versicolor": [8, 9], "version": [0, 3, 10, 13, 14, 15, 21, 22, 23, 24, 25, 26, 28, 31], "versu": [1, 34, 39], "vert": [0, 1, 5, 6, 7, 8, 9, 11, 13, 16, 17, 31, 32, 33, 34, 35, 36, 37, 38, 39], "vert_1": [5, 6, 32, 33, 34], "vert_2": [5, 6, 11, 17, 32, 33, 34, 35], "vi": 39, "via": [0, 5, 6, 7, 8, 9, 10, 11, 12, 19, 23, 24, 25, 27, 28, 29, 31, 32, 33, 34, 35, 36, 37, 38, 39], "vidal": 11, "video": [0, 1, 12, 23, 27, 29, 31, 32, 33], "view": [1, 3, 5, 6, 12, 13, 28, 30, 31, 33, 34, 35, 37, 39], "vii": 39, "viii": 39, "violat": 8, "virginica": 9, "viridi": [0, 1, 2, 3, 31, 39], "virtanen": [], "virtual": [1, 34, 39], "viscos": 13, "viscou": 13, "visibl": 15, "vision": [0, 3], "visit": 34, "visual": [0, 3, 11, 12, 18, 23, 31, 32, 37, 38], "visualis": 1, "visualstudio": [15, 16, 19], "viz": [6, 8, 28], "vmap": 13, "vmax": [1, 6], "vmh0zpt0tli": 34, "vmin": [1, 6], "voic": 3, "volatil": 34, "volum": [0, 3, 31], "von": [38, 39], "vote": [10, 31], "voting_clf": 10, "votingclassifi": 10, "votingsimpl": 10, "vscode": [21, 22], "vstack": [5, 11, 24, 28, 31, 32, 36, 37, 39], "vt": [5, 32, 33], "w": [0, 1, 2, 3, 4, 5, 6, 7, 8, 10, 11, 12, 13, 14, 21, 22, 24, 28, 31, 32, 33, 34, 35, 36, 37, 38, 39], "w1": [8, 21, 22], "w2": [8, 11, 21, 22], "w3": 8, "w_": [1, 12, 37, 38, 39], "w_0": 38, "w_1": [8, 24, 38, 39], "w_1a_0": [38, 39], "w_1x": [38, 39], "w_1x_": 8, "w_1x_1": 8, "w_2": [8, 24, 38, 39], "w_2a_1": [38, 39], "w_2x_": 8, "w_2x_2": 8, "w_3": 24, "w_4": 24, "w_g": [21, 22], "w_hidden": 2, "w_i": [1, 2, 10, 38, 39], "w_ix_i": [12, 37, 38], "w_j": 24, "w_m": 24, "w_output": 2, "w_px_": 8, "w_px_p": 8, "w_t": [], "wa": [1, 3, 4, 5, 6, 7, 10, 11, 12, 14, 17, 19, 21, 24, 31, 32, 34, 35, 36, 37, 38, 39], "wai": [0, 1, 2, 3, 4, 5, 6, 7, 8, 10, 11, 12, 13, 14, 15, 18, 19, 21, 22, 24, 28, 31, 32, 33, 34, 37, 39], "walk": 9, "walker": 28, "wall": 34, "walt": [], "wang": [0, 31], "want": [0, 1, 2, 3, 4, 5, 6, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 20, 21, 22, 23, 25, 26, 28, 31, 32, 33, 34, 35, 36, 38, 39], "warn": [4, 39], "warrant": [6, 35, 36], "warranti": [], "wast": [3, 34], "watch": [23, 33, 34, 35, 37, 38, 39], "wave": 3, "wavelet": 8, "wcag": [], "we": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 28, 29, 30, 32, 33, 35, 36, 37], "weak": [9, 10, 14], "weaker": 34, "weather": [1, 12, 37, 38, 39], "web": [23, 27, 29, 31], "weblink": 26, "webpag": 31, "websit": [6, 24, 25, 26, 27, 31], "wedg": [8, 28, 38, 39], "wednesdai": [29, 31, 36], "wee": 11, "week": [0, 5, 6, 7, 25, 26, 27, 29], "week41": 26, "week42": 26, "weekli": [15, 16, 23, 25, 27, 29, 30, 31, 37], "weierstrass": 38, "weight": [1, 2, 3, 6, 7, 9, 10, 12, 13, 18, 21, 22, 26, 28, 34, 36, 37, 38], "weight_arrai": 39, "weigth": [2, 22], "welchlab": [37, 38, 39], "welcom": [8, 15, 23], "well": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 12, 13, 15, 16, 20, 21, 22, 23, 24, 25, 26, 28, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39], "went": 8, "were": [0, 1, 3, 4, 5, 6, 7, 8, 10, 11, 12, 14, 28, 31, 34, 35, 36, 37, 38, 39], "wessel": [0, 19, 25, 31, 32, 33, 34], "wg_nf1awssi": 38, "what": [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 19, 20, 21, 22, 23, 24, 25, 26, 28, 34, 37, 38, 39], "whatev": [3, 21], "when": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 18, 19, 21, 22, 24, 25, 26, 28, 31, 32, 33, 35, 36, 37, 38, 39], "whenev": [13, 15, 28, 34, 38], "where": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 20, 21, 22, 23, 24, 25, 26, 28, 29, 31, 32, 33, 34, 35, 36, 37, 38, 39], "wherea": [6, 28, 34, 35, 36], "wherefrom": [25, 26], "wherein": [1, 12, 37, 38, 39], "whether": [0, 3, 5, 7, 9, 25, 26, 28, 31, 36, 37], "which": [0, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 31, 32, 33, 35, 36, 37, 38], "whichev": [1, 3, 39], "while": [0, 1, 3, 4, 5, 6, 7, 8, 9, 11, 12, 13, 15, 16, 19, 20, 21, 22, 28, 31, 32, 33, 34, 35, 36, 37, 38, 39], "white": 9, "whiteboad": 34, "whiteboard": [32, 33, 34, 35, 36, 37, 38, 39], "who": [0, 15], "whole": [1, 3, 4, 5, 9, 11, 13, 21, 34, 39], "whom": [], "whose": [0, 6, 10, 26, 28, 32, 35, 36], "whow": [11, 32], "why": [0, 1, 3, 6, 13, 15, 16, 17, 19, 21, 25, 32, 33, 39], "wide": [0, 1, 3, 6, 7, 12, 23, 24, 25, 31, 35, 36, 37, 38, 39], "widehat": [6, 35], "width": [0, 3, 8, 9, 21, 31], "wieringen": [0, 19, 25, 31, 32, 33, 34], "wiki": 25, "wikipedia": 25, "win": [10, 34], "wind": 9, "window": [], "wing": [29, 31], "winther": 2, "wiothout": 6, "wiscons": 7, "wisconsin": [10, 37, 39], "wisdom": [6, 32, 34], "wise": [1, 5, 12, 13, 21, 32, 33, 34, 37, 39], "wish": [0, 2, 5, 7, 8, 11, 13, 14, 18, 24, 25, 26, 31, 32, 33, 34, 36, 37, 38, 39], "with_std": [0, 32], "wither": 6, "within": [0, 2, 3, 4, 7, 9, 12, 13, 14, 28, 30, 31, 33, 36, 37], "withinclust": 14, "without": [0, 1, 5, 6, 8, 9, 11, 12, 13, 15, 18, 25, 26, 31, 32, 33, 34, 35, 36, 37, 38, 39], "wo5dmep_bbi": [37, 38, 39], "won": [0, 15, 31, 38], "wonder": 8, "word": [0, 1, 3, 4, 5, 6, 7, 14, 19, 25, 26, 28, 31, 32, 33, 34, 39], "work": [0, 1, 4, 6, 7, 8, 9, 13, 15, 16, 18, 19, 20, 21, 22, 23, 25, 26, 27, 28, 29, 31, 32, 34, 35, 36, 37, 38, 39], "workabl": 34, "workaround": [], "workhors": 34, "workload": 34, "workshop": 31, "world": [0, 8, 16, 32], "worldwid": [0, 31], "worri": 15, "wors": [0, 1, 3, 4, 6, 31, 34, 35, 36, 39], "worth": [9, 19, 21], "worthi": [25, 26], "would": [0, 1, 3, 5, 6, 7, 8, 9, 10, 11, 12, 13, 16, 18, 20, 22, 24, 25, 26, 28, 31, 32, 33, 34, 35, 36, 37, 38, 39], "wouldn": [], "wrap": [6, 24, 31], "wrapper": [21, 22], "write": [0, 1, 2, 3, 5, 6, 7, 8, 12, 13, 15, 16, 18, 21, 24, 31, 32, 34, 35, 36, 37, 38], "writer": [36, 37], "writerow": [36, 37], "written": [0, 2, 3, 5, 11, 12, 13, 16, 23, 24, 25, 26, 28, 31, 32, 33, 34, 38, 39], "wrong": [1, 8, 15, 19, 39], "wrongli": 10, "wrote": [5, 11, 32], "wrt": [10, 13, 21, 22, 34, 38, 39], "wth": [10, 13, 34], "wurstemberg": [38, 39], "www": [20, 23, 24, 25, 26, 30, 31, 33, 34, 35, 37, 38, 39], "wx_1": 8, "x": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 21, 22, 24, 25, 26, 28, 31, 33, 34, 35, 36, 37, 38, 39], "x0": [8, 36, 37], "x1": [4, 8, 9, 10, 13, 36, 37], "x1_exampl": 8, "x1d": 8, "x2": [8, 9, 10, 13], "x2d": [8, 11], "x2d_train": 11, "x2dsl": 11, "x3": 8, "x_": [0, 2, 3, 5, 6, 8, 10, 11, 13, 14, 24, 28, 31, 32, 33, 34, 35, 36, 38], "x_0": [0, 5, 11, 18, 24, 31, 32, 35, 38], "x_1": [0, 2, 5, 6, 7, 8, 9, 10, 11, 13, 18, 24, 28, 31, 32, 33, 34, 35, 36, 37, 38, 39], "x_2": [0, 2, 5, 6, 7, 8, 9, 10, 11, 13, 24, 28, 31, 32, 33, 35, 36, 37, 38, 39], "x_3": [8, 24, 28, 38], "x_4": [24, 38], "x_5": 38, "x_6": 18, "x_batch": 39, "x_bin": [36, 37], "x_center": 11, "x_data": [1, 39], "x_data_ful": [1, 39], "x_hidden": 2, "x_i": [0, 1, 2, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 24, 28, 31, 32, 33, 34, 35, 36, 37, 38, 39], "x_input": 2, "x_ix_": [0, 31], "x_iy_i": 8, "x_j": [0, 2, 8, 9, 12, 16, 28, 32, 34, 37, 38], "x_jy_j": 8, "x_k": [12, 14, 24, 28, 32, 37], "x_l": [28, 38], "x_m": [6, 12, 24, 28, 35, 37], "x_mean": [18, 34], "x_multi": [36, 37], "x_n": [0, 2, 3, 6, 8, 11, 12, 13, 24, 28, 31, 33, 35, 37, 38], "x_new": [9, 10], "x_norm": [18, 34], "x_offset": [6, 32, 34], "x_output": 2, "x_p": [3, 7, 9, 36, 37], "x_poli": 9, "x_poly10": 9, "x_pred": 4, "x_prev": 2, "x_reduc": 11, "x_sampl": [], "x_scale": 8, "x_small": 13, "x_std": [18, 34], "x_t": 34, "x_test": [0, 1, 3, 5, 6, 7, 9, 10, 11, 15, 16, 17, 19, 26, 32, 33, 34, 35, 36, 37, 39], "x_test_": 17, "x_test_own": 6, "x_test_scal": [0, 6, 7, 9, 10, 11, 32, 34], "x_tot": 4, "x_train": [0, 1, 3, 4, 5, 6, 7, 9, 10, 11, 15, 16, 17, 19, 26, 31, 32, 33, 34, 35, 36, 37, 39], "x_train_": 17, "x_train_mean": [6, 32, 34], "x_train_own": 6, "x_train_r": 19, "x_train_scal": [0, 6, 7, 9, 10, 11, 32, 34], "x_val": [1, 39], "xarrai": [23, 31], "xavier": [1, 39], "xbnew": [13, 33, 34], "xcode": [0, 23, 25, 31], "xdclassiffierconfus": 10, "xdclassiffierroc": 10, "xg_clf": 10, "xgb": 10, "xgbclassifi": 10, "xgboost": 9, "xgboot": 10, "xgbregressor": 10, "xgparam": 10, "xgtree": 10, "xi": [8, 13, 34, 36, 37], "xi_": 8, "xi_1": 8, "xi_i": 8, "xinv": 37, "xk": 8, "xla": [13, 23, 31], "xlabel": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 13, 21, 28, 31, 32, 33, 34, 35, 36], "xlim": [6, 10, 35, 36], "xm": 9, "xmesh": 13, "xnew": [0, 13, 31, 33, 34], "xp": 28, "xpanda": [0, 32], "xpd": [5, 11, 32], "xplot": 0, "xscale": [0, 32], "xsr": 9, "xt_x": [13, 33, 34], "xtest": [6, 35, 36], "xtick": [3, 6, 8, 9, 35, 36], "xtrain": [6, 35, 36], "xu": [0, 31], "xx": [0, 24, 31], "xy": [0, 6, 8, 24, 31], "xytext": 8, "xyz": [], "xz": [24, 31], "y": [0, 1, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 24, 25, 26, 28, 31, 32, 33, 34, 35, 36, 37, 38, 39], "y1": 4, "y2": 4, "y3": 4, "y_": [0, 1, 5, 6, 10, 11, 24, 31, 32, 35, 36, 39], "y_0": [0, 5, 11, 24, 31, 32, 35], "y_1": [0, 5, 8, 9, 11, 13, 24, 31, 32, 33, 34, 35], "y_1y_1": 8, "y_1y_1k": 8, "y_1y_2": 8, "y_1y_2k": 8, "y_1y_n": 8, "y_1y_nk": 8, "y_2": [0, 5, 8, 9, 11, 24, 31, 32], "y_2y_1": 8, "y_2y_1k": 8, "y_2y_2": 8, "y_2y_2k": 8, "y_3": [0, 9, 24], "y_4": 24, "y_bin": [36, 37], "y_binari": [36, 37], "y_center": [18, 34], "y_data": [0, 1, 5, 6, 31, 32, 33, 34, 39], "y_data_ful": [1, 39], "y_decis": 8, "y_fit": [0, 32], "y_i": [0, 1, 5, 6, 7, 8, 9, 10, 11, 12, 13, 19, 24, 25, 26, 31, 32, 33, 34, 35, 36, 37, 38, 39], "y_if_": 10, "y_indic": [36, 37], "y_ix_": [0, 31], "y_ix_i": [7, 8, 13, 32, 33, 34, 36, 37], "y_iy_jk": 8, "y_j": [6, 8, 12, 25, 35, 36, 37, 38, 39], "y_k": [12, 37], "y_m": 24, "y_mean": [18, 34], "y_model": [0, 4, 5, 6, 31, 32, 33, 34], "y_multi": [36, 37], "y_n": [8, 13, 33, 34], "y_ny_1": 8, "y_ny_1k": 8, "y_ny_2": 8, "y_ny_2k": 8, "y_ny_n": 8, "y_ny_nk": 8, "y_offset": [6, 17, 32, 34], "y_onehot": [36, 37], "y_plot": 9, "y_pred": [0, 1, 4, 6, 7, 8, 9, 10, 26, 32, 34, 35, 36, 37, 39], "y_pred1": 9, "y_pred2": 9, "y_pred_bin": [36, 37], "y_pred_multi": [36, 37], "y_pred_rf": 10, "y_pred_tre": 10, "y_prob": [36, 37], "y_prob_bin": [36, 37], "y_prob_multi": [36, 37], "y_proba": [7, 10, 37], "y_sampl": [], "y_scaler": [6, 32, 34], "y_test": [0, 1, 3, 4, 5, 6, 7, 9, 10, 11, 15, 16, 17, 19, 26, 32, 33, 34, 35, 36, 37, 39], "y_test_onehot": [1, 39], "y_test_predict": [], "y_tot": 4, "y_train": [0, 1, 3, 4, 5, 6, 7, 9, 10, 11, 15, 16, 17, 19, 26, 31, 32, 33, 34, 35, 36, 37, 39], "y_train_mean": [6, 32, 34], "y_train_onehot": [1, 39], "y_train_predict": [], "y_train_r": 19, "y_train_scal": [6, 32, 34], "y_true": [36, 37], "y_val": 1, "yand": 37, "ye": [3, 6, 7, 35, 36, 37], "year": [0, 23, 31], "yet": [0, 1, 6, 8, 11, 13, 20, 21, 31, 36, 38, 39], "yi": [13, 34, 36, 37], "yield": [0, 2, 5, 6, 8, 10, 12, 13, 14, 24, 28, 31, 33, 34, 35, 37, 38, 39], "yk": 8, "ylabel": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 13, 21, 28, 31, 32, 33, 34, 35, 36], "ylim": [3, 6, 35, 36], "ym": 9, "ymesh": 13, "yn": 0, "yo": [8, 9, 10], "yor": 37, "yoshiki": [], "yoshua": [1, 30, 39], "you": [0, 1, 3, 4, 5, 6, 8, 9, 10, 11, 13, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 28, 29, 30, 31, 32, 33, 34, 35, 36, 38, 39], "young": 0, "your": [1, 2, 4, 5, 6, 8, 11, 13, 15, 17, 19, 20, 21, 22, 23, 24, 31, 33, 34, 35, 36, 37, 38, 39], "your_model_object": 16, "yourself": [11, 13, 31, 33], "youtu": [32, 33, 35, 37], "youtub": [23, 33, 34, 35, 37, 38, 39], "ypred": [6, 35, 36], "ypredict": [0, 13, 31, 32, 33, 34], "ypredict2": [13, 33, 34], "ypredictlasso": [5, 33], "ypredictol": [0, 5, 33], "ypredictown": [6, 32, 34], "ypredictownridg": [6, 32, 33, 34], "ypredictridg": [0, 5, 6, 32, 33, 34], "ypredictskl": [6, 32, 34], "ytest": [6, 35, 36], "ytick": [3, 6, 8, 9, 35, 36], "ytild": [0, 6, 31, 32, 35, 36], "ytildelasso": [5, 33], "ytildenp": [0, 31, 32], "ytildeol": [0, 5, 33], "ytildeownridg": [6, 32, 33, 34], "ytilderidg": [5, 6, 32, 33, 34], "ytrain": [6, 35, 36], "yuxi": 31, "yx": [24, 31], "yxor": [37, 39], "yy": [24, 31], "yz": [24, 31], "z": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 11, 12, 13, 21, 22, 24, 28, 31, 32, 35, 36, 37, 38, 39], "z1": [21, 22], "z2": [21, 22], "z_": [1, 2, 12, 24, 31, 38, 39], "z_0": [24, 31, 38], "z_1": [24, 31, 38, 39], "z_2": [22, 24, 31, 38, 39], "z_c": [1, 39], "z_h": [1, 39], "z_hidden": 2, "z_i": [1, 12, 37, 39], "z_j": [1, 12], "z_k": [12, 32, 38, 39], "z_m": [1, 39], "z_matric": 39, "z_mod": 9, "z_o": [1, 39], "z_output": 2, "za": [], "zalando": 26, "zaman": 28, "zaxi": 6, "zero": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 16, 17, 18, 19, 21, 24, 25, 26, 28, 31, 32, 33, 34, 35, 36, 37, 38, 39], "zeros_lik": [4, 36, 37], "zeroth": 32, "zfill": 4, "zip": [4, 6, 21, 22, 36, 37], "zm_h": [0, 31], "zn": [], "zone": [], "zoom": 31, "zscout": [], "zx": [24, 31], "zy": [24, 31], "zz": [24, 31], "\u00f8yvind": [6, 32, 34], "\u03b4": 39}, "titles": ["3. Linear Regression", "14. Building a Feed Forward Neural Network", "15. Solving Differential Equations with Deep Learning", "16. Convolutional Neural Networks", "17. Recurrent neural networks: Overarching view", "4. Ridge and Lasso Regression", "5. Resampling Methods", "6. Logistic Regression", "8. Support Vector Machines, overarching aims", "9. Decision trees, overarching aims", "10. Ensemble Methods: From a Single Tree to Many Trees and Extreme Boosting, Meet the Jungle of Methods", "11. Basic ideas of the Principal Component Analysis (PCA)", "13. Neural networks", "7. Optimization, the central part of any Machine Learning algortithm", "12. Clustering and Unsupervised Learning", "Exercises week 34", "Exercises week 35", "Exercises week 36", "Exercises week 37", "Exercises week 38", "Exercises week 39", "Exercises week 41", "Exercises week 42", "Applied Data Analysis and Machine Learning", "2. Linear Algebra, Handling of Arrays and more Python Features", "Project 1 on Machine Learning, deadline October 6 (midnight), 2025", "Project 2 on Machine Learning, deadline November 10 (Midnight)", "Course setting", "1. Elements of Probability Theory and Statistical Data Analysis", "Teachers and Grading", "Textbooks", "Week 34: Introduction to the course, Logistics and Practicalities", "Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression", "Week 36: Linear Regression and Gradient descent", "Week 37: Gradient descent methods", "Week 38: Statistical analysis, bias-variance tradeoff and resampling methods", "Week 39: Resampling methods and logistic regression", "Week 40: Gradient descent methods (continued) and start Neural networks", "Week 41 Neural networks and constructing a neural network code", "Week 42 Constructing a Neural Network code with examples"], "titleterms": {"": [8, 10, 33, 34, 35, 36, 37], "0": 39, "04": [], "05": [], "06": [], "07": [], "1": [0, 15, 16, 17, 18, 19, 20, 21, 22, 25, 26, 32, 38, 39], "10": [26, 38], "11": [], "13": 39, "15": [19, 35], "19": 19, "1a": 18, "2": [0, 15, 16, 17, 18, 19, 20, 21, 22, 26, 31, 32, 33, 38, 39], "20": [], "2017": [], "2018": [], "2019": [], "2023": 29, "2025": [25, 36, 37, 38, 39], "22": 36, "26": 36, "27": [], "29": 37, "2a": [], "2b": [], "3": [0, 15, 16, 17, 18, 19, 20, 21, 22, 32, 38, 39], "34": [15, 31], "35": [16, 32], "36": [17, 33], "37": [18, 34], "38": [19, 35], "39": [20, 36], "3a": 18, "3b": 18, "4": [0, 15, 16, 17, 18, 19, 20, 21, 22, 32, 39], "40": 37, "41": [21, 38], "42": [22, 39], "4a": 18, "4b": 18, "5": [0, 16, 18, 19, 20, 21, 22], "6": [21, 22, 25, 38], "7": [21, 22], "8": [22, 34], "A": [0, 1, 4, 8, 9, 31, 35, 36, 37, 39], "AND": 37, "And": [31, 32, 34], "But": 34, "In": [29, 38], "Ising": 6, "OR": 37, "The": [0, 1, 2, 3, 5, 6, 7, 8, 9, 11, 12, 15, 23, 31, 32, 33, 34, 35, 36, 37, 38, 39], "To": 31, "With": [4, 33], "a11i": [], "about": [26, 31, 32, 33], "abov": [33, 38, 39], "abstract": 20, "accuraci": 34, "across": 34, "activ": [1, 12, 21, 26, 37, 38, 39], "ad": [0, 6, 20, 25, 31, 32, 37, 38, 39], "adaboost": 10, "adagrad": [13, 34], "adam": [13, 34], "adapt": [10, 34], "add": [], "adjust": [1, 39], "advanc": 25, "adversari": 4, "again": [3, 9], "against": 26, "ai": [25, 26, 31], "aim": [8, 9, 21, 22, 31], "aka": 31, "al": 34, "algebra": [24, 31], "algorithm": [9, 10, 11, 12, 26, 31, 32, 33, 34, 38, 39], "algortithm": [13, 33, 36, 37], "all": [8, 38, 39], "an": [0, 4, 10, 15, 20, 31, 38], "analys": [5, 32, 33], "analysi": [0, 5, 6, 11, 23, 25, 26, 28, 31, 32, 33, 35, 36, 38], "analyt": [0, 16, 18, 26], "analyz": [38, 39], "ani": [13, 22, 33, 36, 37], "anoth": [9, 33, 35, 36], "api": [], "appli": 23, "approach": [0, 8, 14, 31, 34, 35, 36], "approxim": [12, 38], "architectur": [1, 39], "arrai": [24, 31], "artifici": [37, 38], "assist": 29, "assumpt": 35, "august": [], "author": [], "autocorrel": 28, "autograd": [2, 13, 22, 34], "automat": [13, 34, 38], "avail": 20, "avali": [], "averag": 34, "b": [25, 26], "back": [1, 11, 12, 32, 33, 38, 39], "background": [23, 25, 26, 35], "backpropag": 22, "bag": 10, "base": [13, 34, 35], "basic": [0, 5, 7, 9, 10, 11, 24, 32, 33, 36, 37, 38], "batch": [1, 22, 34, 39], "bay": 5, "befor": 11, "bengio": 39, "beta": [], "better": [8, 37], "bia": [6, 19, 25, 34, 35, 36], "bias": [38, 39], "binari": [1, 39], "bind": 31, "bird": 10, "blind": [], "block": [], "boldsymbol": [18, 32, 35], "book": [19, 38, 39], "boost": 10, "bootstrap": [6, 10, 35, 36], "boston": [], "breast": 1, "brief": [31, 35, 36], "bring": [12, 38, 39], "browser": [], "bsd": [], "build": [1, 3, 9, 39], "c": [25, 26, 31], "calcul": [18, 32, 33], "can": [31, 34, 35, 36, 38], "cancer": [1, 7, 9, 11], "cart": 9, "case": [8, 10, 28, 32, 33, 34, 36, 37], "cdn": [], "cell": [], "central": [13, 23, 28, 33, 35, 36, 37], "chain": [12, 38, 39], "challeng": 34, "chang": 10, "changelog": [], "channel": 31, "chi": [0, 31], "choic": [17, 39], "choos": [1, 34, 39], "cifar01": 3, "citat": [], "class": [36, 37, 38], "classic": 11, "classif": [1, 9, 10, 26, 36, 37, 39], "classifi": [8, 36], "claus": [], "clip": [1, 39], "cluster": 14, "cnn": 3, "code": [1, 2, 5, 9, 11, 12, 13, 14, 15, 16, 20, 25, 26, 31, 32, 33, 34, 35, 36, 37, 38, 39], "collect": [1, 3, 39], "color": [], "colorblind": [], "combin": 34, "commun": 31, "compact": [36, 37, 38, 39], "compar": [2, 10, 16], "comparison": [33, 34], "compet": 34, "compil": [], "complet": [32, 38, 39], "complex": [0, 6, 25, 32], "complic": [6, 38], "compon": 11, "comput": [9, 19, 34], "computation": [35, 36], "computerlab": 31, "con": [9, 34], "concept": 28, "condit": 33, "confid": 35, "conjug": 13, "consider": [38, 39], "constraint": 34, "construct": [38, 39], "contain": [], "content": [], "continu": 37, "contn": 31, "contrast": [], "contributor": [], "converg": 34, "convex": [8, 13, 33, 34], "convolut": [3, 12, 37, 38], "copyright": [], "core": [], "correct": 34, "correl": [11, 32, 37], "correspond": [], "cost": [1, 10, 32, 33, 34, 35, 36, 37, 38, 39], "count": 38, "cours": [23, 27, 30, 31], "covari": [5, 11, 28, 32], "cover": 31, "creat": [16, 20], "creator": [], "critic": 26, "cross": [6, 25, 35, 36, 37], "custom": 21, "cython": 31, "d": [25, 26], "dark": [], "data": [0, 1, 3, 6, 7, 9, 11, 15, 17, 18, 21, 23, 28, 31, 32, 36, 37, 38, 39], "dataset": [1, 3, 18, 39], "david": 31, "deadlin": [25, 26, 31], "deadllin": 29, "decai": [2, 34], "decis": [9, 10], "decomposit": [5, 11, 24, 32, 33], "deeep": [], "deep": [1, 2, 31, 34, 36, 37, 38, 39], "defin": [1, 31, 38, 39], "definit": [19, 38, 39], "deflist": [], "degre": [0, 17, 32], "deliver": [15, 16, 19, 20, 25, 26], "deliveri": [25, 26], "delta": 35, "dens": 0, "depend": [], "depth": 26, "deriv": [5, 12, 16, 17, 19, 32, 33, 34, 35, 38, 39], "descent": [2, 10, 13, 18, 25, 33, 34, 37], "design": 32, "detail": [3, 31], "develop": [1, 39], "diagon": 11, "differ": [8, 26, 34], "differenti": [2, 13, 34, 38], "diffus": 2, "dimens": 34, "dimension": [2, 3, 8, 18], "direct": [], "disadvantag": 9, "discret": 28, "discrimin": 31, "discuss": 37, "distribut": [5, 28, 35], "do": [1, 34, 37, 39], "document": 20, "doe": [32, 33, 37], "domain": 28, "down": [1, 39], "dropout": [1, 39], "e": [25, 26], "each": [21, 36], "economi": [32, 33], "electron": [25, 26], "element": [0, 28, 31], "elimin": 24, "elu": 39, "empir": 34, "energi": 31, "ensembl": 10, "entri": [38, 39], "entropi": [9, 36, 37], "environ": [0, 15], "equat": [0, 2, 12, 32, 33, 36, 37, 38, 39], "error": [0, 10, 31, 32, 33, 35, 36], "essenti": 31, "estim": 35, "et": 34, "etc": 31, "euler": 2, "evalu": [1, 26, 38, 39], "evid": 34, "exampl": [1, 2, 3, 4, 6, 7, 8, 9, 10, 31, 32, 33, 34, 35, 36, 37, 38, 39], "exercis": [0, 6, 15, 16, 17, 18, 19, 20, 21, 22, 32, 38], "expect": [19, 28, 35], "expens": [35, 36], "experi": 28, "explicit": [38, 39], "explod": 39, "explor": 0, "exponenti": [2, 34], "express": [16, 17, 19, 32, 36, 37, 38, 39], "extend": [33, 36, 37, 38], "extrapol": 4, "extrem": [10, 31], "ey": 10, "f": [25, 26], "fall": 29, "famili": [1, 31, 39], "famou": 24, "fantast": [32, 33], "faq": [], "featur": [9, 16, 24, 32], "februari": [], "feed": [1, 12, 22, 37, 38, 39], "figur": 20, "file": [], "fill": [], "final": [12, 32, 34, 38, 39], "find": [16, 18, 35], "fine": [1, 39], "first": [4, 12, 31, 33, 38, 39], "fit": [0, 10, 15, 16, 31, 33], "fix": [32, 33, 34], "float": 38, "fold": [35, 36], "forc": 3, "forest": 10, "form": 18, "format": [25, 26, 31], "formula": 18, "forward": [1, 2, 12, 22, 37, 38, 39], "foster": 31, "fourier": 3, "frank": 6, "freedom": [0, 17, 32], "frequent": [32, 34], "frequentist": [0, 31], "from": [5, 10, 12, 26, 31, 32, 33, 34, 35, 36, 37, 38, 39], "full": [2, 34, 39], "function": [0, 1, 6, 7, 8, 10, 11, 12, 13, 25, 26, 28, 31, 32, 33, 34, 35, 36, 37, 38, 39], "funtion": 39, "further": [3, 5, 32, 33], "g": [25, 26], "gan": 4, "gate": [37, 39], "gaussian": 24, "gd": [13, 34], "gener": [4, 9, 31, 36, 37, 38], "geometr": [11, 33], "get": [20, 38], "gini": 9, "github": 15, "glorot": 39, "goal": [15, 16, 17, 18, 19, 20], "good": [0, 20, 31], "goodfellow": 34, "gotthard": [], "grade": [29, 31], "gradient": [1, 2, 10, 13, 18, 22, 25, 26, 33, 34, 37, 38, 39], "greativ": [], "group": 36, "growth": 2, "guid": [], "h": 25, "ha": 23, "hand": [22, 38, 39], "handl": [24, 31], "happen": [35, 36], "hessian": [32, 33, 34], "hidden": [2, 38, 39], "high": [], "histogram": 35, "histori": [], "homogen": 39, "hous": [], "how": 16, "hyperbol": [37, 39], "hyperparamet": [1, 17, 39], "hyperplan": 8, "i": [0, 1, 31, 39], "id3": 9, "idea": 11, "ideal": 33, "ident": 35, "identifi": 35, "ii": 31, "iid": 35, "illustr": [33, 37, 38], "implement": [1, 16, 17, 18, 39], "implic": [5, 32, 33], "import": [5, 24, 31, 32, 33, 38, 39], "improv": [1, 34, 39], "includ": [13, 25, 34, 36, 37, 38], "incorpor": [], "increment": 11, "independ": 35, "index": 9, "inform": 29, "ingredi": 38, "init": 39, "input": [2, 21, 22, 38, 39], "insight": 39, "instal": [23, 25, 31], "instructor": 29, "intermedi": 38, "interpret": [5, 11, 19, 31, 32, 33, 35], "interv": 35, "introduc": [11, 13, 32], "introduct": [0, 6, 20, 23, 24, 25, 26, 31, 37, 38], "invers": [5, 24], "invert": [32, 33], "ipython": [], "iter": 10, "its": 32, "j": [], "jacobian": 32, "januari": [], "jax": 13, "job": 37, "julia": 31, "jungl": 10, "jupyt": [], "k": [35, 36, 38, 39], "kera": [1, 3, 39], "kernel": [8, 11], "l": [38, 39], "lab": [33, 34, 35, 36, 37, 38, 39], "lagrangian": 8, "lasso": [5, 6, 25, 32, 33], "last": [32, 34, 37, 38, 39], "later": [5, 32, 33], "layer": [1, 2, 3, 12, 21, 22, 38, 39], "layout": [38, 39], "learn": [0, 1, 2, 11, 13, 14, 15, 16, 17, 18, 19, 20, 23, 25, 26, 31, 32, 33, 34, 35, 36, 37, 38, 39], "least": [5, 6, 16, 19, 25, 31, 32, 33, 34], "lectur": [31, 33, 34, 35, 36, 37, 38, 39], "level": 10, "librari": [23, 26, 31], "licens": [], "light": [], "likelihood": [7, 35, 36, 37], "limit": [1, 13, 28, 33, 34, 35, 39], "linear": [0, 8, 13, 15, 24, 31, 32, 33, 36], "link": [5, 11, 30, 32, 35], "list": [38, 39], "literatur": [25, 26], "logist": [7, 31, 36, 37, 39], "loss": [32, 33, 34], "lu": 24, "ma": [], "machin": [0, 8, 13, 23, 25, 26, 31, 33, 36, 37], "machineri": 26, "made": 35, "main": [28, 31], "make": [0, 9, 10, 20, 32], "mani": [10, 12], "markdown": [], "mask": [], "maskedarrai": [], "mass": 31, "materi": [25, 26, 31, 32, 33, 34, 35, 36, 38, 39], "math": [5, 32, 33], "mathemat": [3, 5, 8, 32, 33, 37, 38, 39], "matplotlib": [], "matric": [5, 24, 31], "matrix": [1, 5, 11, 12, 16, 24, 31, 32, 33, 34, 37, 39], "matter": 0, "max": 32, "maximum": [35, 36, 37], "me": [], "mean": [0, 32, 33, 36], "measur": 37, "meet": [5, 10, 28, 31, 32], "memori": 34, "mercer": 8, "metadata": [], "method": [6, 9, 10, 13, 25, 31, 33, 34, 35, 36, 37, 39], "metric": 19, "midnight": [25, 26], "min": 32, "mini": 34, "minibatch": 34, "minim": [31, 36, 37], "mit": [], "ml": 31, "mle": 35, "mlp": 12, "mnist": [3, 4], "mode": 38, "model": [0, 1, 4, 6, 12, 15, 17, 31, 37, 38, 39], "moment": 34, "momentum": [13, 25, 34], "mondai": [33, 34, 35, 37, 38], "moon": [8, 9], "more": [3, 6, 24, 25, 31, 32, 33, 34, 35, 36, 37, 38, 39], "motiv": 34, "move": 34, "multi": [37, 38, 39], "multiclass": 39, "multilay": [12, 37, 38], "multipl": [1, 3, 17, 21, 39], "multipli": 8, "multivari": 38, "myst": [], "ncsa": [], "need": [25, 31], "network": [1, 2, 3, 4, 7, 12, 26, 31, 34, 36, 37, 38, 39], "neural": [1, 2, 3, 4, 7, 12, 26, 31, 34, 37, 38, 39], "neuron": [37, 38], "new": [4, 18, 35, 38], "newton": [33, 34, 36, 37], "nn": [38, 39], "node": [38, 39], "noeds": 39, "non": [8, 34], "none": 34, "norm": 26, "normal": [0, 1, 35, 39], "notat": [12, 37], "note": [25, 26, 32, 33], "notebook": [], "novemb": 26, "now": [1, 9, 13, 33, 34, 35, 36], "nuclear": [0, 31], "nueral": 36, "numba": 31, "number": [0, 2, 22, 28, 32, 34, 38], "numer": [2, 25, 26, 28], "numpi": [24, 31], "object": [3, 22, 39], "observ": [38, 39], "obtain": 11, "octob": [25, 38, 39], "od": 2, "off": [6, 19, 25], "ol": [5, 6, 15, 16, 18, 25, 33, 35], "onc": 21, "one": [2, 12, 18, 22, 33, 38, 39], "ones": [37, 39], "open": [], "oper": [24, 38], "optim": [1, 8, 13, 18, 23, 31, 32, 33, 34, 36, 37, 38, 39], "option": [21, 22], "order": [13, 18, 34], "ordinari": [5, 6, 16, 19, 25, 31, 32, 33, 34], "organ": [0, 31], "orient": [22, 39], "oslo": 30, "other": [4, 9, 11, 12, 24, 25, 26, 31, 37, 38, 39], "ouput": [38, 39], "our": [0, 4, 5, 11, 13, 25, 26, 31, 32, 33, 36, 37, 39], "outcom": [23, 31], "output": [2, 38, 39], "over": [38, 39], "overarch": [0, 4, 8, 9, 21, 22, 31, 32, 38], "overview": [10, 31, 34], "own": [0, 10, 11, 25, 26, 31, 32], "packag": [24, 31], "panda": [31, 32], "paper": 39, "parallel": 38, "paramet": [31, 32, 36, 37, 38, 39], "paramt": 18, "part": [13, 23, 25, 26, 33, 36, 37, 38, 39], "partial": 2, "pass": [1, 22, 39], "pca": 11, "pdf": 28, "percepetron": [38, 39], "perceptron": [12, 37, 38, 39], "perform": [1, 9, 39], "period": 3, "perspect": [1, 39], "pitaya": [], "plan": [32, 33, 34, 35, 36, 38], "plethora": 31, "plot": [35, 36], "point": [4, 38], "poisson": 2, "polici": [], "polynomi": [3, 16, 18, 33], "popul": 2, "popular": 31, "practic": [13, 29, 31, 34], "pre": [1, 3, 39], "preambl": [25, 26], "predict": [4, 21], "predictor": [36, 37], "preprocess": [32, 34], "prerequisit": [3, 23, 31], "present": 20, "princip": 11, "principl": 3, "pro": [9, 34], "probabl": [5, 28, 35], "problem": [1, 2, 13, 31, 32, 33, 34, 36, 37, 38, 39], "procedur": [9, 31], "process": [1, 3, 21, 39], "program": [2, 13, 25, 26, 33, 34, 38], "project": [6, 20, 25, 26, 29, 31], "prop": 13, "propag": [1, 12, 38, 39], "properti": [5, 28, 32, 33, 34, 36], "python": [0, 9, 15, 23, 24, 31], "quick": 8, "quickli": [], "r": 31, "random": [10, 11, 28], "raphson": [33, 36, 37], "rate": [25, 34, 39], "read": [9, 31, 32, 34, 35, 36, 37, 38, 39], "real": [6, 21, 31, 38], "recommend": [31, 32, 39], "record": [], "recurr": [4, 12, 37, 38], "reduc": [0, 32, 38], "reduct": 3, "refer": [25, 26], "referenc": 20, "reformul": 2, "regress": [0, 5, 6, 7, 9, 10, 13, 15, 17, 18, 19, 25, 26, 31, 32, 33, 34, 35, 36, 37], "regular": [1, 39], "relat": [], "relev": [30, 32, 37, 39], "relu": [1, 39], "remark": 3, "remind": [6, 8, 26, 31, 32, 33, 34, 38, 39], "replac": [13, 34], "report": [20, 25, 26], "repositori": [15, 35, 36], "requir": [2, 23], "resampl": [6, 19, 25, 35, 36], "rescal": [6, 32], "residu": [32, 33], "resourc": 2, "result": [32, 33, 38, 39], "revers": 38, "revis": [], "revisit": [13, 33, 34, 36, 37], "rewrit": [31, 32, 35], "rewritten": [36, 37], "ridg": [0, 5, 6, 17, 18, 19, 25, 32, 33, 34], "rm": 13, "rmsprop": 34, "role": [], "root": 39, "rule": [12, 34, 38, 39], "rung": 25, "same": [13, 34, 35, 36], "sampl": 11, "scalabl": 34, "scale": [17, 18, 19, 32, 34], "schedul": [31, 39], "schemat": 9, "scheme": 2, "scienc": 31, "scikit": [0, 1, 11, 31, 32, 33, 34, 35, 36, 37, 39], "second": [13, 18, 34], "select": 36, "semest": 29, "sensit": 33, "septemb": [19, 33, 34, 35, 36, 37], "seriou": 38, "session": [33, 34, 35, 36, 37, 38, 39], "set": [0, 2, 3, 9, 12, 15, 27, 31, 32, 33, 38, 39], "setup": 15, "sgd": [13, 34], "should": [1, 39], "show": [], "sigmoid": 39, "similar": [13, 34], "simpl": [0, 4, 9, 13, 18, 31, 32, 33, 34, 36, 38, 39], "simpler": 38, "simplest": 18, "singl": [10, 37, 38], "singular": [5, 11, 32, 33], "size": [32, 33, 34], "sklearn": 16, "slightli": 34, "smarter": 38, "smoothi": [], "sneak": 34, "soft": 8, "softmax": [1, 39], "softwar": [25, 26, 31], "solv": [2, 33, 36, 37], "solver": 13, "some": [13, 24, 32, 33, 36, 38], "sourc": [], "specifi": 2, "speed": 34, "sphinx": [], "split": [0, 15, 32], "squar": [0, 5, 6, 10, 16, 19, 25, 31, 32, 33, 34], "standard": [13, 32, 35], "start": [20, 37], "state": 0, "statist": [5, 6, 23, 28, 31, 35, 36], "steepest": [10, 13, 33], "step": [34, 35, 36], "stochast": [13, 25, 28, 34], "stop": 34, "strongli": [31, 34], "structur": [], "studi": 37, "suggest": [31, 37], "sum": [35, 36, 38, 39], "summari": [29, 31], "superposit": 3, "supervis": [1, 39], "support": 8, "svd": [5, 32, 33], "synthet": [18, 36, 37], "systemat": 3, "t": 32, "take": 16, "taken": [31, 34], "teach": 29, "teacher": [29, 31], "team": [], "technic": 32, "techniqu": [6, 11, 25], "technologi": 23, "tensorflow": [1, 3, 39], "tent": [29, 31], "term": [35, 38, 39], "test": [0, 1, 15, 17, 26, 32, 39], "texmath": [], "text": 31, "textbook": [30, 31], "than": 33, "thank": [], "theorem": [5, 8, 11, 12, 28, 35, 38], "theoret": 34, "theori": 28, "theta": [18, 35], "thi": [21, 22, 31, 38], "three": [38, 39], "through": 38, "time": 34, "tip": [13, 34], "todo": [], "togeth": [12, 38, 39], "tool": [25, 26, 31], "top": [1, 39], "topic": 31, "toward": 11, "trade": [6, 19, 25], "tradeoff": [6, 35, 36], "train": [0, 1, 4, 15, 21, 22, 31, 32, 38, 39], "transform": 3, "translat": [], "tree": [9, 10], "tuesdai": [33, 37, 38, 39], "tune": [1, 39], "two": [3, 8, 22, 23, 36, 37, 38, 39], "type": [2, 4, 12, 31, 37, 38], "uio": 31, "understand": [22, 35, 36], "univers": [12, 30, 38], "unsupervis": 14, "up": [0, 2, 9, 12, 15, 26, 31, 32, 33, 35, 36, 38, 39], "updat": [25, 34, 38, 39], "us": [0, 1, 2, 3, 7, 13, 16, 18, 19, 22, 23, 25, 26, 31, 32, 33, 34, 36, 37, 38, 39], "usag": [34, 39], "v": [3, 34], "valid": [6, 25, 35, 36], "valu": [5, 11, 19, 28, 32, 33, 35, 36], "vanish": 39, "vari": 34, "variabl": [28, 33], "varianc": [6, 19, 25, 35, 36], "variou": [0, 26, 35, 36], "vector": [8, 12, 16, 24, 31, 32, 37], "versu": 31, "video": [34, 35, 36, 37, 38, 39], "view": [0, 4, 10, 32, 38], "virtual": 15, "visual": [1, 9, 39], "wai": [9, 25, 35, 36, 38], "warm": 26, "wave": 2, "we": [31, 34, 38, 39], "wednesdai": [33, 37, 38, 39], "week": [15, 16, 17, 18, 19, 20, 21, 22, 31, 32, 33, 34, 35, 36, 37, 38, 39], "weekli": [], "weight": 39, "welcom": [], "what": [0, 31, 32, 33, 35, 36], "when": 34, "which": [1, 34, 39], "why": [31, 34, 35, 36, 37, 38], "wisconsin": 7, "word": 38, "workflow": [], "wrap": 35, "write": [4, 11, 20, 22, 25, 26, 33, 39], "x": 32, "xgboost": 10, "xor": [37, 39], "yaml": [], "yet": 33, "your": [0, 10, 16, 18, 25, 26, 32], "z_j": [38, 39]}}) \ No newline at end of file diff --git a/doc/LectureNotes/_build/html/statistics.html b/doc/LectureNotes/_build/html/statistics.html index 54f855838..97b1b2ccb 100644 --- a/doc/LectureNotes/_build/html/statistics.html +++ b/doc/LectureNotes/_build/html/statistics.html @@ -246,6 +246,7 @@ +
  • Week 42 Constructing a Neural Network code with examples
  • Exercises week 42
    • @@ -260,6 +261,7 @@

    Projects

    diff --git a/doc/LectureNotes/_build/html/teachers.html b/doc/LectureNotes/_build/html/teachers.html index bb23b9df1..00f6beed3 100644 --- a/doc/LectureNotes/_build/html/teachers.html +++ b/doc/LectureNotes/_build/html/teachers.html @@ -244,6 +244,7 @@ +
  • Week 42 Constructing a Neural Network code with examples
  • Exercises week 42
    • @@ -258,6 +259,7 @@

    Projects

    diff --git a/doc/LectureNotes/_build/html/textbooks.html b/doc/LectureNotes/_build/html/textbooks.html index f3ad80f00..92c689e82 100644 --- a/doc/LectureNotes/_build/html/textbooks.html +++ b/doc/LectureNotes/_build/html/textbooks.html @@ -244,6 +244,7 @@ +
  • Week 42 Constructing a Neural Network code with examples
  • Exercises week 42
    • @@ -258,6 +259,7 @@

    Projects

    diff --git a/doc/LectureNotes/_build/html/week34.html b/doc/LectureNotes/_build/html/week34.html index 6ef186ee7..7d35ecdf2 100644 --- a/doc/LectureNotes/_build/html/week34.html +++ b/doc/LectureNotes/_build/html/week34.html @@ -246,6 +246,7 @@ +
  • Week 42 Constructing a Neural Network code with examples
  • Exercises week 42
    • @@ -260,6 +261,7 @@

    Projects

    diff --git a/doc/LectureNotes/_build/html/week35.html b/doc/LectureNotes/_build/html/week35.html index 42ba8688b..6a6ed099e 100644 --- a/doc/LectureNotes/_build/html/week35.html +++ b/doc/LectureNotes/_build/html/week35.html @@ -246,6 +246,7 @@ +
  • Week 42 Constructing a Neural Network code with examples
  • Exercises week 42
    • @@ -260,6 +261,7 @@

    Projects

    diff --git a/doc/LectureNotes/_build/html/week36.html b/doc/LectureNotes/_build/html/week36.html index b70903025..609f8c732 100644 --- a/doc/LectureNotes/_build/html/week36.html +++ b/doc/LectureNotes/_build/html/week36.html @@ -246,6 +246,7 @@ +
  • Week 42 Constructing a Neural Network code with examples
  • Exercises week 42
    • @@ -260,6 +261,7 @@

    Projects

    diff --git a/doc/LectureNotes/_build/html/week37.html b/doc/LectureNotes/_build/html/week37.html index e91832625..680657a4f 100644 --- a/doc/LectureNotes/_build/html/week37.html +++ b/doc/LectureNotes/_build/html/week37.html @@ -246,6 +246,7 @@ +
  • Week 42 Constructing a Neural Network code with examples
  • Exercises week 42
    • @@ -260,6 +261,7 @@

    Projects

    diff --git a/doc/LectureNotes/_build/html/week38.html b/doc/LectureNotes/_build/html/week38.html index 0aedfa4c0..e404d608d 100644 --- a/doc/LectureNotes/_build/html/week38.html +++ b/doc/LectureNotes/_build/html/week38.html @@ -246,6 +246,7 @@ +
  • Week 42 Constructing a Neural Network code with examples
  • Exercises week 42
    • @@ -260,6 +261,7 @@

    Projects

    diff --git a/doc/LectureNotes/_build/html/week39.html b/doc/LectureNotes/_build/html/week39.html index 628f18664..a80aa8ece 100644 --- a/doc/LectureNotes/_build/html/week39.html +++ b/doc/LectureNotes/_build/html/week39.html @@ -246,6 +246,7 @@ +
  • Week 42 Constructing a Neural Network code with examples
  • Exercises week 42
    • @@ -260,6 +261,7 @@

    Projects

    diff --git a/doc/LectureNotes/_build/html/week40.html b/doc/LectureNotes/_build/html/week40.html index 8d629dfa6..743b05a6a 100644 --- a/doc/LectureNotes/_build/html/week40.html +++ b/doc/LectureNotes/_build/html/week40.html @@ -246,6 +246,7 @@ +
  • Week 42 Constructing a Neural Network code with examples
  • Exercises week 42
    • @@ -260,6 +261,7 @@

    Projects

    diff --git a/doc/LectureNotes/_build/html/week41.html b/doc/LectureNotes/_build/html/week41.html index 57d49ea36..c09c3caa6 100644 --- a/doc/LectureNotes/_build/html/week41.html +++ b/doc/LectureNotes/_build/html/week41.html @@ -246,6 +246,7 @@ +
  • Week 42 Constructing a Neural Network code with examples
  • Exercises week 42
    • @@ -260,6 +261,7 @@

    Projects

    diff --git a/doc/LectureNotes/_build/html/week42.html b/doc/LectureNotes/_build/html/week42.html new file mode 100644 index 000000000..bc75152dc --- /dev/null +++ b/doc/LectureNotes/_build/html/week42.html @@ -0,0 +1,4061 @@ + + + + + + + + + + + Week 42 Constructing a Neural Network code with examples — Applied Data Analysis and Machine Learning + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
    + + + + + + + + + + +
    +
    +
    +
    + + + Ctrl+K +
    +
    + +
    + +
    + + +
    +
    + + +
    +
    + + + +
    + + + +
    + + + + +
    + +
    +
    + + + + + +
    +
    + +
    +
    +
    + + +
    +
    + + + + +
    + +
    + + + +
    + +
    +
    + +
    +
    + +
    + +
    + +
    + + +
    + +
    + +
    + + + + + +
    + + +
    + + + + + + + + + + + + + +
    + +
    + +
    +
    + + + +
    +

    Week 42 Constructing a Neural Network code with examples

    + +
    +
    + +
    +

    Contents

    +
    + +
    +
    +
    + + + + +
    + + +
    +

    Week 42 Constructing a Neural Network code with examples#

    +

    Morten Hjorth-Jensen, Department of Physics, University of Oslo, Norway

    +

    Date: October 13-17, 2025

    +
    +

    Lecture October 13, 2025#

    +
      +

    1. Building our own Feed-forward Neural Network and discussion of project 2

      2. +

    2. Project 2 is available at CompPhysics/MachineLearning

      3. +
    +
    +
    +

    Readings and videos#

    +
      +

    1. These lecture notes

      2. +
    + + +
      +

    1. For a more in depth discussion on neural networks we recommend Goodfellow et al chapters 6 and 7. For the optimization part, see chapter 8.

      2. +

    2. Neural Networks demystified at https://www.youtube.com/watch?v=bxe2T-V8XRs&list=PLiaHhY2iBX9hdHaRr6b7XevZtgZRa1PoU&ab_channel=WelchLabs

      3. +

    3. Building Neural Networks from scratch at https://www.youtube.com/watch?v=Wo5dMEP_BbI&list=PLQVvvaa0QuDcjD5BAw2DxE6OF2tius3V3&ab_channel=sentdex

      4. +

    4. Video on Neural Networks at https://www.youtube.com/watch?v=CqOfi41LfDw

      5. +

    5. Video on the back propagation algorithm at https://www.youtube.com/watch?v=Ilg3gGewQ5U

      6. +
    +

    I also recommend Michael Nielsen’s intuitive approach to the neural networks and the universal approximation theorem, see the slides at http://neuralnetworksanddeeplearning.com/chap4.html.

    +
    +
    +

    Material for the lab sessions on Tuesday and Wednesday#

    +
      +

    1. Exercises on writing a code for neural networks, back propagation part, see exercises for week 42 at https://compphysics.github.io/MachineLearning/doc/LectureNotes/_build/html/exercisesweek42.html

      2. +

    2. Discussion of project 2

      3. +
    +
    +
    +

    Lecture material: Writing a code which implements a feed-forward neural network#

    +

    Last week we discussed the basics of neural networks and deep learning +and the basics of automatic differentiation. We looked also at +examples on how compute the parameters of a simple network with scalar +inputs and ouputs and no or just one hidden layers.

    +

    We ended our discussions with the derivation of the equations for a +neural network with one hidden layers and two input variables and two +hidden nodes but only one output node. We did almost finish the derivation of the back propagation algorithm.

    +
    +
    +

    Mathematics of deep learning#

    +

    Two recent books online.

    +
      +

    1. The Modern Mathematics of Deep Learning, by Julius Berner, Philipp Grohs, Gitta Kutyniok, Philipp Petersen, published as Mathematical Aspects of Deep Learning, pp. 1-111. Cambridge University Press, 2022

      2. +

    2. Mathematical Introduction to Deep Learning: Methods, Implementations, and Theory, Arnulf Jentzen, Benno Kuckuck, Philippe von Wurstemberger

      3. +
    +
    +
    +

    Reminder on books with hands-on material and codes#

    + +
    +
    +

    Reading recommendations#

    +
      +

    1. Rashkca et al., chapter 11, jupyter-notebook sent separately, from GitHub

      2. +

    2. Goodfellow et al, chapter 6 and 7 contain most of the neural network background.

      3. +
    +
    +
    +

    Reminder from last week: First network example, simple percepetron with one input#

    +

    As yet another example we define now a simple perceptron model with +all quantities given by scalars. We consider only one input variable +\(x\) and one target value \(y\). We define an activation function +\(\sigma_1\) which takes as input

    +
    +\[ +z_1 = w_1x+b_1, +\]
    +

    where \(w_1\) is the weight and \(b_1\) is the bias. These are the +parameters we want to optimize. The output is \(a_1=\sigma(z_1)\) (see +graph from whiteboard notes). This output is then fed into the +cost/loss function, which we here for the sake of simplicity just +define as the squared error

    +
    +\[ +C(x;w_1,b_1)=\frac{1}{2}(a_1-y)^2. +\]
    +
    +
    +

    Layout of a simple neural network with no hidden layer#

    + + +

    Figure 1:

    +
    +
    +

    Optimizing the parameters#

    +

    In setting up the feed forward and back propagation parts of the +algorithm, we need now the derivative of the various variables we want +to train.

    +

    We need

    +
    +\[ +\frac{\partial C}{\partial w_1} \hspace{0.1cm}\mathrm{and}\hspace{0.1cm}\frac{\partial C}{\partial b_1}. +\]
    +

    Using the chain rule we find

    +
    +\[ +\frac{\partial C}{\partial w_1}=\frac{\partial C}{\partial a_1}\frac{\partial a_1}{\partial z_1}\frac{\partial z_1}{\partial w_1}=(a_1-y)\sigma_1'x, +\]
    +

    and

    +
    +\[ +\frac{\partial C}{\partial b_1}=\frac{\partial C}{\partial a_1}\frac{\partial a_1}{\partial z_1}\frac{\partial z_1}{\partial b_1}=(a_1-y)\sigma_1', +\]
    +

    which we later will just define as

    +
    +\[ +\frac{\partial C}{\partial a_1}\frac{\partial a_1}{\partial z_1}=\delta_1. +\]
    +
    +
    +

    Adding a hidden layer#

    +

    We change our simple model to (see graph) +a network with just one hidden layer but with scalar variables only.

    +

    Our output variable changes to \(a_2\) and \(a_1\) is now the output from the hidden node and \(a_0=x\). +We have then

    +
    +\[ +z_1 = w_1a_0+b_1 \hspace{0.1cm} \wedge a_1 = \sigma_1(z_1), +\]
    +
    +\[ +z_2 = w_2a_1+b_2 \hspace{0.1cm} \wedge a_2 = \sigma_2(z_2), +\]
    +

    and the cost function

    +
    +\[ +C(x;\boldsymbol{\Theta})=\frac{1}{2}(a_2-y)^2, +\]
    +

    with \(\boldsymbol{\Theta}=[w_1,w_2,b_1,b_2]\).

    +
    +
    +

    Layout of a simple neural network with one hidden layer#

    + + +

    Figure 1:

    +
    +
    +

    The derivatives#

    +

    The derivatives are now, using the chain rule again

    +
    +\[ +\frac{\partial C}{\partial w_2}=\frac{\partial C}{\partial a_2}\frac{\partial a_2}{\partial z_2}\frac{\partial z_2}{\partial w_2}=(a_2-y)\sigma_2'a_1=\delta_2a_1, +\]
    +
    +\[ +\frac{\partial C}{\partial b_2}=\frac{\partial C}{\partial a_2}\frac{\partial a_2}{\partial z_2}\frac{\partial z_2}{\partial b_2}=(a_2-y)\sigma_2'=\delta_2, +\]
    +
    +\[ +\frac{\partial C}{\partial w_1}=\frac{\partial C}{\partial a_2}\frac{\partial a_2}{\partial z_2}\frac{\partial z_2}{\partial a_1}\frac{\partial a_1}{\partial z_1}\frac{\partial z_1}{\partial w_1}=(a_2-y)\sigma_2'a_1\sigma_1'a_0, +\]
    +
    +\[ +\frac{\partial C}{\partial b_1}=\frac{\partial C}{\partial a_2}\frac{\partial a_2}{\partial z_2}\frac{\partial z_2}{\partial a_1}\frac{\partial a_1}{\partial z_1}\frac{\partial z_1}{\partial b_1}=(a_2-y)\sigma_2'\sigma_1'=\delta_1. +\]
    +

    Can you generalize this to more than one hidden layer?

    +
    +
    +

    Important observations#

    +

    From the above equations we see that the derivatives of the activation +functions play a central role. If they vanish, the training may +stop. This is called the vanishing gradient problem, see discussions below. If they become +large, the parameters \(w_i\) and \(b_i\) may simply go to infinity. This +is referenced as the exploding gradient problem.

    +
    +
    +

    The training#

    +

    The training of the parameters is done through various gradient descent approximations with

    +
    +\[ +w_{i}\leftarrow w_{i}- \eta \delta_i a_{i-1}, +\]
    +

    and

    +
    +\[ +b_i \leftarrow b_i-\eta \delta_i, +\]
    +

    with \(\eta\) is the learning rate.

    +

    One iteration consists of one feed forward step and one back-propagation step. Each back-propagation step does one update of the parameters \(\boldsymbol{\Theta}\).

    +

    For the first hidden layer \(a_{i-1}=a_0=x\) for this simple model.

    +
    +
    +

    Code example#

    +

    The code here implements the above model with one hidden layer and +scalar variables for the same function we studied in the previous +example. The code is however set up so that we can add multiple +inputs \(x\) and target values \(y\). Note also that we have the +possibility of defining a feature matrix \(\boldsymbol{X}\) with more than just +one column for the input values. This will turn useful in our next example. We have also defined matrices and vectors for all of our operations although it is not necessary here.

    +
    +
    +
    import numpy as np
+# We use the Sigmoid function as activation function
+def sigmoid(z):
+    return 1.0/(1.0+np.exp(-z))
+
+def forwardpropagation(x):
+    # weighted sum of inputs to the hidden layer
+    z_1 = np.matmul(x, w_1) + b_1
+    # activation in the hidden layer
+    a_1 = sigmoid(z_1)
+    # weighted sum of inputs to the output layer
+    z_2 = np.matmul(a_1, w_2) + b_2
+    a_2 = z_2
+    return a_1, a_2
+
+def backpropagation(x, y):
+    a_1, a_2 = forwardpropagation(x)
+    # parameter delta for the output layer, note that a_2=z_2 and its derivative wrt z_2 is just 1
+    delta_2 = a_2 - y
+    print(0.5*((a_2-y)**2))
+    # delta for  the hidden layer
+    delta_1 = np.matmul(delta_2, w_2.T) * a_1 * (1 - a_1)
+    # gradients for the output layer
+    output_weights_gradient = np.matmul(a_1.T, delta_2)
+    output_bias_gradient = np.sum(delta_2, axis=0)
+    # gradient for the hidden layer
+    hidden_weights_gradient = np.matmul(x.T, delta_1)
+    hidden_bias_gradient = np.sum(delta_1, axis=0)
+    return output_weights_gradient, output_bias_gradient, hidden_weights_gradient, hidden_bias_gradient
+
+
+# ensure the same random numbers appear every time
+np.random.seed(0)
+# Input variable
+x = np.array([4.0],dtype=np.float64)
+# Target values
+y = 2*x+1.0 
+
+# Defining the neural network, only scalars here
+n_inputs = x.shape
+n_features = 1
+n_hidden_neurons = 1
+n_outputs = 1
+
+# Initialize the network
+# weights and bias in the hidden layer
+w_1 = np.random.randn(n_features, n_hidden_neurons)
+b_1 = np.zeros(n_hidden_neurons) + 0.01
+
+# weights and bias in the output layer
+w_2 = np.random.randn(n_hidden_neurons, n_outputs)
+b_2 = np.zeros(n_outputs) + 0.01
+
+eta = 0.1
+for i in range(50):
+    # calculate gradients
+    derivW2, derivB2, derivW1, derivB1 = backpropagation(x, y)
+    # update weights and biases
+    w_2 -= eta * derivW2
+    b_2 -= eta * derivB2
+    w_1 -= eta * derivW1
+    b_1 -= eta * derivB1
+
    +
    +
    +
    +

    We see that after some few iterations (the results do depend on the learning rate however), we get an error which is rather small.

    +
    +
    +

    Simple neural network and the back propagation equations#

    +

    Let us now try to increase our level of ambition and attempt at setting +up the equations for a neural network with two input nodes, one hidden +layer with two hidden nodes and one output layer with one output node/neuron only (see graph)..

    +

    We need to define the following parameters and variables with the input layer (layer \((0)\)) +where we label the nodes \(x_1\) and \(x_2\)

    +
    +\[ +x_1 = a_1^{(0)} \wedge x_2 = a_2^{(0)}. +\]
    +

    The hidden layer (layer \((1)\)) has nodes which yield the outputs \(a_1^{(1)}\) and \(a_2^{(1)}\)) with weight \(\boldsymbol{w}\) and bias \(\boldsymbol{b}\) parameters

    +
    +\[ +w_{ij}^{(1)}=\left\{w_{11}^{(1)},w_{12}^{(1)},w_{21}^{(1)},w_{22}^{(1)}\right\} \wedge b^{(1)}=\left\{b_1^{(1)},b_2^{(1)}\right\}. +\]
    +
    +
    +

    Layout of a simple neural network with two input nodes, one hidden layer with two hidden noeds and one output node#

    + + +

    Figure 1:

    +
    +
    +

    The ouput layer#

    +

    We have the ouput layer given by layer label \((2)\) with output \(a^{(2)}\) and weights and biases to be determined given by the variables

    +
    +\[ +w_{i}^{(2)}=\left\{w_{1}^{(2)},w_{2}^{(2)}\right\} \wedge b^{(2)}. +\]
    +

    Our output is \(\tilde{y}=a^{(2)}\) and we define a generic cost function \(C(a^{(2)},y;\boldsymbol{\Theta})\) where \(y\) is the target value (a scalar here). +The parameters we need to optimize are given by

    +
    +\[ +\boldsymbol{\Theta}=\left\{w_{11}^{(1)},w_{12}^{(1)},w_{21}^{(1)},w_{22}^{(1)},w_{1}^{(2)},w_{2}^{(2)},b_1^{(1)},b_2^{(1)},b^{(2)}\right\}. +\]
    +
    +
    +

    Compact expressions#

    +

    We can define the inputs to the activation functions for the various layers in terms of various matrix-vector multiplications and vector additions. +The inputs to the first hidden layer are

    +
    +\[\begin{split} +\begin{bmatrix}z_1^{(1)} \\ z_2^{(1)} \end{bmatrix}=\left(\begin{bmatrix}w_{11}^{(1)} & w_{12}^{(1)}\\ w_{21}^{(1)} &w_{22}^{(1)} \end{bmatrix}\right)^{T}\begin{bmatrix}a_1^{(0)} \\ a_2^{(0)} \end{bmatrix}+\begin{bmatrix}b_1^{(1)} \\ b_2^{(1)} \end{bmatrix}, +\end{split}\]
    +

    with outputs

    +
    +\[\begin{split} +\begin{bmatrix}a_1^{(1)} \\ a_2^{(1)} \end{bmatrix}=\begin{bmatrix}\sigma^{(1)}(z_1^{(1)}) \\ \sigma^{(1)}(z_2^{(1)}) \end{bmatrix}. +\end{split}\]
    +
    +
    +

    Output layer#

    +

    For the final output layer we have the inputs to the final activation function

    +
    +\[ +z^{(2)} = w_{1}^{(2)}a_1^{(1)} +w_{2}^{(2)}a_2^{(1)}+b^{(2)}, +\]
    +

    resulting in the output

    +
    +\[ +a^{(2)}=\sigma^{(2)}(z^{(2)}). +\]
    +
    +
    +

    Explicit derivatives#

    +

    In total we have nine parameters which we need to train. Using the +chain rule (or just the back-propagation algorithm) we can find all +derivatives. Since we will use automatic differentiation in reverse +mode, we start with the derivatives of the cost function with respect +to the parameters of the output layer, namely

    +
    +\[ +\frac{\partial C}{\partial w_{i}^{(2)}}=\frac{\partial C}{\partial a^{(2)}}\frac{\partial a^{(2)}}{\partial z^{(2)}}\frac{\partial z^{(2)}}{\partial w_{i}^{(2)}}=\delta^{(2)}a_i^{(1)}, +\]
    +

    with

    +
    +\[ +\delta^{(2)}=\frac{\partial C}{\partial a^{(2)}}\frac{\partial a^{(2)}}{\partial z^{(2)}} +\]
    +

    and finally

    +
    +\[ +\frac{\partial C}{\partial b^{(2)}}=\frac{\partial C}{\partial a^{(2)}}\frac{\partial a^{(2)}}{\partial z^{(2)}}\frac{\partial z^{(2)}}{\partial b^{(2)}}=\delta^{(2)}. +\]
    +
    +
    +

    Derivatives of the hidden layer#

    +

    Using the chain rule we have the following expressions for say one of the weight parameters (it is easy to generalize to the other weight parameters)

    +
    +\[ +\frac{\partial C}{\partial w_{11}^{(1)}}=\frac{\partial C}{\partial a^{(2)}}\frac{\partial a^{(2)}}{\partial z^{(2)}} +\frac{\partial z^{(2)}}{\partial z_1^{(1)}}\frac{\partial z_1^{(1)}}{\partial w_{11}^{(1)}}= \delta^{(2)}\frac{\partial z^{(2)}}{\partial z_1^{(1)}}\frac{\partial z_1^{(1)}}{\partial w_{11}^{(1)}}, +\]
    +

    which, noting that

    +
    +\[ +z^{(2)} =w_1^{(2)}a_1^{(1)}+w_2^{(2)}a_2^{(1)}+b^{(2)}, +\]
    +

    allows us to rewrite

    +
    +\[ +\frac{\partial z^{(2)}}{\partial z_1^{(1)}}\frac{\partial z_1^{(1)}}{\partial w_{11}^{(1)}}=w_1^{(2)}\frac{\partial a_1^{(1)}}{\partial z_1^{(1)}}a_1^{(1)}. +\]
    +
    +
    +

    Final expression#

    +

    Defining

    +
    +\[ +\delta_1^{(1)}=w_1^{(2)}\frac{\partial a_1^{(1)}}{\partial z_1^{(1)}}\delta^{(2)}, +\]
    +

    we have

    +
    +\[ +\frac{\partial C}{\partial w_{11}^{(1)}}=\delta_1^{(1)}a_1^{(1)}. +\]
    +

    Similarly, we obtain

    +
    +\[ +\frac{\partial C}{\partial w_{12}^{(1)}}=\delta_1^{(1)}a_2^{(1)}. +\]
    +
    +
    +

    Completing the list#

    +

    Similarly, we find

    +
    +\[ +\frac{\partial C}{\partial w_{21}^{(1)}}=\delta_2^{(1)}a_1^{(1)}, +\]
    +

    and

    +
    +\[ +\frac{\partial C}{\partial w_{22}^{(1)}}=\delta_2^{(1)}a_2^{(1)}, +\]
    +

    where we have defined

    +
    +\[ +\delta_2^{(1)}=w_2^{(2)}\frac{\partial a_2^{(1)}}{\partial z_2^{(1)}}\delta^{(2)}. +\]
    +
    +
    +

    Final expressions for the biases of the hidden layer#

    +

    For the sake of completeness, we list the derivatives of the biases, which are

    +
    +\[ +\frac{\partial C}{\partial b_{1}^{(1)}}=\delta_1^{(1)}, +\]
    +

    and

    +
    +\[ +\frac{\partial C}{\partial b_{2}^{(1)}}=\delta_2^{(1)}. +\]
    +

    As we will see below, these expressions can be generalized in a more compact form.

    +
    +
    +

    Gradient expressions#

    +

    For this specific model, with just one output node and two hidden +nodes, the gradient descent equations take the following form for output layer

    +
    +\[ +w_{i}^{(2)}\leftarrow w_{i}^{(2)}- \eta \delta^{(2)} a_{i}^{(1)}, +\]
    +

    and

    +
    +\[ +b^{(2)} \leftarrow b^{(2)}-\eta \delta^{(2)}, +\]
    +

    and

    +
    +\[ +w_{ij}^{(1)}\leftarrow w_{ij}^{(1)}- \eta \delta_{i}^{(1)} a_{j}^{(0)}, +\]
    +

    and

    +
    +\[ +b_{i}^{(1)} \leftarrow b_{i}^{(1)}-\eta \delta_{i}^{(1)}, +\]
    +

    where \(\eta\) is the learning rate.

    +
    +
    +

    Setting up the equations for a neural network#

    +

    The questions we want to ask are how do changes in the biases and the +weights in our network change the cost function and how can we use the +final output to modify the weights and biases?

    +

    To derive these equations let us start with a plain regression problem +and define our cost function as

    +
    +\[ +{\cal C}(\boldsymbol{\Theta}) = \frac{1}{2}\sum_{i=1}^n\left(y_i - \tilde{y}_i\right)^2, +\]
    +

    where the \(y_i\)s are our \(n\) targets (the values we want to +reproduce), while the outputs of the network after having propagated +all inputs \(\boldsymbol{x}\) are given by \(\boldsymbol{\tilde{y}}_i\).

    +
    +
    +

    Layout of a neural network with three hidden layers (last layer = \(l=L=4\), first layer \(l=0\))#

    + + +

    Figure 1:

    +
    +
    +

    Definitions#

    +

    With our definition of the targets \(\boldsymbol{y}\), the outputs of the +network \(\boldsymbol{\tilde{y}}\) and the inputs \(\boldsymbol{x}\) we +define now the activation \(z_j^l\) of node/neuron/unit \(j\) of the +\(l\)-th layer as a function of the bias, the weights which add up from +the previous layer \(l-1\) and the forward passes/outputs +\(\boldsymbol{a}^{l-1}\) from the previous layer as

    +
    +\[ +z_j^l = \sum_{i=1}^{M_{l-1}}w_{ij}^la_i^{l-1}+b_j^l, +\]
    +

    where \(b_k^l\) are the biases from layer \(l\). Here \(M_{l-1}\) +represents the total number of nodes/neurons/units of layer \(l-1\). The +figure in the whiteboard notes illustrates this equation. We can rewrite this in a more +compact form as the matrix-vector products we discussed earlier,

    +
    +\[ +\boldsymbol{z}^l = \left(\boldsymbol{W}^l\right)^T\boldsymbol{a}^{l-1}+\boldsymbol{b}^l. +\]
    +
    +
    +

    Inputs to the activation function#

    +

    With the activation values \(\boldsymbol{z}^l\) we can in turn define the +output of layer \(l\) as \(\boldsymbol{a}^l = \sigma(\boldsymbol{z}^l)\) where \(\sigma\) is our +activation function. In the examples here we will use the sigmoid +function discussed in our logistic regression lectures. We will also use the same activation function \(\sigma\) for all layers +and their nodes. It means we have

    +
    +\[ +a_j^l = \sigma(z_j^l) = \frac{1}{1+\exp{-(z_j^l)}}. +\]
    +
    +
    +

    Layout of input to first hidden layer \(l=1\) from input layer \(l=0\)#

    + + +

    Figure 1:

    +
    +
    +

    Derivatives and the chain rule#

    +

    From the definition of the input variable to the activation function, that is \(z_j^l\) we have

    +
    +\[ +\frac{\partial z_j^l}{\partial w_{ij}^l} = a_i^{l-1}, +\]
    +

    and

    +
    +\[ +\frac{\partial z_j^l}{\partial a_i^{l-1}} = w_{ji}^l. +\]
    +

    With our definition of the activation function we have that (note that this function depends only on \(z_j^l\))

    +
    +\[ +\frac{\partial a_j^l}{\partial z_j^{l}} = a_j^l(1-a_j^l)=\sigma(z_j^l)(1-\sigma(z_j^l)). +\]
    +
    +
    +

    Derivative of the cost function#

    +

    With these definitions we can now compute the derivative of the cost function in terms of the weights.

    +

    Let us specialize to the output layer \(l=L\). Our cost function is

    +
    +\[ +{\cal C}(\boldsymbol{\Theta}^L) = \frac{1}{2}\sum_{i=1}^n\left(y_i - \tilde{y}_i\right)^2=\frac{1}{2}\sum_{i=1}^n\left(a_i^L - y_i\right)^2, +\]
    +

    The derivative of this function with respect to the weights is

    +
    +\[ +\frac{\partial{\cal C}(\boldsymbol{\Theta}^L)}{\partial w_{ij}^L} = \left(a_j^L - y_j\right)\frac{\partial a_j^L}{\partial w_{ij}^{L}}, +\]
    +

    The last partial derivative can easily be computed and reads (by applying the chain rule)

    +
    +\[ +\frac{\partial a_j^L}{\partial w_{ij}^{L}} = \frac{\partial a_j^L}{\partial z_{j}^{L}}\frac{\partial z_j^L}{\partial w_{ij}^{L}}=a_j^L(1-a_j^L)a_i^{L-1}. +\]
    +
    +
    +

    The back propagation equations for a neural network#

    +

    We have thus

    +
    +\[ +\frac{\partial{\cal C}((\boldsymbol{\Theta}^L)}{\partial w_{ij}^L} = \left(a_j^L - y_j\right)a_j^L(1-a_j^L)a_i^{L-1}, +\]
    +

    Defining

    +
    +\[ +\delta_j^L = a_j^L(1-a_j^L)\left(a_j^L - y_j\right) = \sigma'(z_j^L)\frac{\partial {\cal C}}{\partial (a_j^L)}, +\]
    +

    and using the Hadamard product of two vectors we can write this as

    +
    +\[ +\boldsymbol{\delta}^L = \sigma'(\boldsymbol{z}^L)\circ\frac{\partial {\cal C}}{\partial (\boldsymbol{a}^L)}. +\]
    +
    +
    +

    Analyzing the last results#

    +

    This is an important expression. The second term on the right handside +measures how fast the cost function is changing as a function of the \(j\)th +output activation. If, for example, the cost function doesn’t depend +much on a particular output node \(j\), then \(\delta_j^L\) will be small, +which is what we would expect. The first term on the right, measures +how fast the activation function \(f\) is changing at a given activation +value \(z_j^L\).

    +
    +
    +

    More considerations#

    +

    Notice that everything in the above equations is easily computed. In +particular, we compute \(z_j^L\) while computing the behaviour of the +network, and it is only a small additional overhead to compute +\(\sigma'(z^L_j)\). The exact form of the derivative with respect to the +output depends on the form of the cost function. +However, provided the cost function is known there should be little +trouble in calculating

    +
    +\[ +\frac{\partial {\cal C}}{\partial (a_j^L)} +\]
    +

    With the definition of \(\delta_j^L\) we have a more compact definition of the derivative of the cost function in terms of the weights, namely

    +
    +\[ +\frac{\partial{\cal C}}{\partial w_{ij}^L} = \delta_j^La_i^{L-1}. +\]
    +
    +
    +

    Derivatives in terms of \(z_j^L\)#

    +

    It is also easy to see that our previous equation can be written as

    +
    +\[ +\delta_j^L =\frac{\partial {\cal C}}{\partial z_j^L}= \frac{\partial {\cal C}}{\partial a_j^L}\frac{\partial a_j^L}{\partial z_j^L}, +\]
    +

    which can also be interpreted as the partial derivative of the cost function with respect to the biases \(b_j^L\), namely

    +
    +\[ +\delta_j^L = \frac{\partial {\cal C}}{\partial b_j^L}\frac{\partial b_j^L}{\partial z_j^L}=\frac{\partial {\cal C}}{\partial b_j^L}, +\]
    +

    That is, the error \(\delta_j^L\) is exactly equal to the rate of change of the cost function as a function of the bias.

    +
    +
    +

    Bringing it together#

    +

    We have now three equations that are essential for the computations of the derivatives of the cost function at the output layer. These equations are needed to start the algorithm and they are

    + +
    +
    +\[ +\begin{equation} +\frac{\partial{\cal C}(\boldsymbol{W^L})}{\partial w_{ij}^L} = \delta_j^La_i^{L-1}, +\label{_auto1} \tag{1} +\end{equation} +\]
    +

    and

    + +
    +
    +\[ +\begin{equation} +\delta_j^L = \sigma'(z_j^L)\frac{\partial {\cal C}}{\partial (a_j^L)}, +\label{_auto2} \tag{2} +\end{equation} +\]
    +

    and

    + +
    +
    +\[ +\begin{equation} +\delta_j^L = \frac{\partial {\cal C}}{\partial b_j^L}, +\label{_auto3} \tag{3} +\end{equation} +\]
    +
    +
    +

    Final back propagating equation#

    +

    We have that (replacing \(L\) with a general layer \(l\))

    +
    +\[ +\delta_j^l =\frac{\partial {\cal C}}{\partial z_j^l}. +\]
    +

    We want to express this in terms of the equations for layer \(l+1\).

    +
    +
    +

    Using the chain rule and summing over all \(k\) entries#

    +

    We obtain

    +
    +\[ +\delta_j^l =\sum_k \frac{\partial {\cal C}}{\partial z_k^{l+1}}\frac{\partial z_k^{l+1}}{\partial z_j^{l}}=\sum_k \delta_k^{l+1}\frac{\partial z_k^{l+1}}{\partial z_j^{l}}, +\]
    +

    and recalling that

    +
    +\[ +z_j^{l+1} = \sum_{i=1}^{M_{l}}w_{ij}^{l+1}a_i^{l}+b_j^{l+1}, +\]
    +

    with \(M_l\) being the number of nodes in layer \(l\), we obtain

    +
    +\[ +\delta_j^l =\sum_k \delta_k^{l+1}w_{kj}^{l+1}\sigma'(z_j^l), +\]
    +

    This is our final equation.

    +

    We are now ready to set up the algorithm for back propagation and learning the weights and biases.

    +
    +
    +

    Setting up the back propagation algorithm and algorithm for a feed forward NN, initalizations#

    +

    The architecture (our model).

    +
      +

    1. Set up your inputs and outputs (scalars, vectors, matrices or higher-order arrays)

      2. +

    2. Define the number of hidden layers and hidden nodes

      3. +

    3. Define activation functions for hidden layers and output layers

      4. +

    4. Define optimizer (plan learning rate, momentum, ADAgrad, RMSprop, ADAM etc) and array of initial learning rates

      5. +

    5. Define cost function and possible regularization terms with hyperparameters

      6. +

    6. Initialize weights and biases

      7. +

    7. Fix number of iterations for the feed forward part and back propagation part

      8. +
    +
    +
    +

    Setting up the back propagation algorithm, part 1#

    +

    The four equations provide us with a way of computing the gradients of the cost function. Let us write this out in the form of an algorithm.

    +

    First, we set up the input data \(\boldsymbol{x}\) and the activations +\(\boldsymbol{z}_1\) of the input layer and compute the activation function and +the pertinent outputs \(\boldsymbol{a}^1\).

    +

    Secondly, we perform then the feed forward till we reach the output +layer and compute all \(\boldsymbol{z}_l\) of the input layer and compute the +activation function and the pertinent outputs \(\boldsymbol{a}^l\) for +\(l=1,2,3,\dots,L\).

    +

    Notation: The first hidden layer has \(l=1\) as label and the final output layer has \(l=L\).

    +
    +
    +

    Setting up the back propagation algorithm, part 2#

    +

    Thereafter we compute the ouput error \(\boldsymbol{\delta}^L\) by computing all

    +
    +\[ +\delta_j^L = \sigma'(z_j^L)\frac{\partial {\cal C}}{\partial (a_j^L)}. +\]
    +

    Then we compute the back propagate error for each \(l=L-1,L-2,\dots,1\) as

    +
    +\[ +\delta_j^l = \sum_k \delta_k^{l+1}w_{kj}^{l+1}\sigma'(z_j^l). +\]
    +
    +
    +

    Setting up the Back propagation algorithm, part 3#

    +

    Finally, we update the weights and the biases using gradient descent +for each \(l=L-1,L-2,\dots,1\) (the first hidden layer) and update the weights and biases +according to the rules

    +
    +\[ +w_{ij}^l\leftarrow = w_{ij}^l- \eta \delta_j^la_i^{l-1}, +\]
    +
    +\[ +b_j^l \leftarrow b_j^l-\eta \frac{\partial {\cal C}}{\partial b_j^l}=b_j^l-\eta \delta_j^l, +\]
    +

    with \(\eta\) being the learning rate.

    +
    +
    +

    Updating the gradients#

    +

    With the back propagate error for each \(l=L-1,L-2,\dots,1\) as

    +
    +\[ +\delta_j^l = \sum_k \delta_k^{l+1}w_{kj}^{l+1}\sigma'(z_j^l), +\]
    +

    we update the weights and the biases using gradient descent for each \(l=L-1,L-2,\dots,1\) and update the weights and biases according to the rules

    +
    +\[ +w_{ij}^l\leftarrow = w_{ij}^l- \eta \delta_j^la_i^{l-1}, +\]
    +
    +\[ +b_j^l \leftarrow b_j^l-\eta \frac{\partial {\cal C}}{\partial b_j^l}=b_j^l-\eta \delta_j^l, +\]
    +
    +
    +

    Activation functions#

    +

    A property that characterizes a neural network, other than its +connectivity, is the choice of activation function(s). The following +restrictions are imposed on an activation function for an FFNN to +fulfill the universal approximation theorem

    +
      +

    • Non-constant

      • +

    • Bounded

      • +

    • Monotonically-increasing

      • +

    • Continuous

      • +
    +
    +

    Activation functions, Logistic and Hyperbolic ones#

    +

    The second requirement excludes all linear functions. Furthermore, in +a MLP with only linear activation functions, each layer simply +performs a linear transformation of its inputs.

    +

    Regardless of the number of layers, the output of the NN will be +nothing but a linear function of the inputs. Thus we need to introduce +some kind of non-linearity to the NN to be able to fit non-linear +functions Typical examples are the logistic Sigmoid

    +
    +\[ +\sigma(x) = \frac{1}{1 + e^{-x}}, +\]
    +

    and the hyperbolic tangent function

    +
    +\[ +\sigma(x) = \tanh(x) +\]
    +
    +
    +
    +

    Relevance#

    +

    The sigmoid function are more biologically plausible because the +output of inactive neurons are zero. Such activation function are +called one-sided. However, it has been shown that the hyperbolic +tangent performs better than the sigmoid for training MLPs. has +become the most popular for deep neural networks

    +
    +
    +
    %matplotlib inline
+
+"""The sigmoid function (or the logistic curve) is a 
+function that takes any real number, z, and outputs a number (0,1).
+It is useful in neural networks for assigning weights on a relative scale.
+The value z is the weighted sum of parameters involved in the learning algorithm."""
+
+import numpy
+import matplotlib.pyplot as plt
+import math as mt
+
+z = numpy.arange(-5, 5, .1)
+sigma_fn = numpy.vectorize(lambda z: 1/(1+numpy.exp(-z)))
+sigma = sigma_fn(z)
+
+fig = plt.figure()
+ax = fig.add_subplot(111)
+ax.plot(z, sigma)
+ax.set_ylim([-0.1, 1.1])
+ax.set_xlim([-5,5])
+ax.grid(True)
+ax.set_xlabel('z')
+ax.set_title('sigmoid function')
+
+plt.show()
+
+"""Step Function"""
+z = numpy.arange(-5, 5, .02)
+step_fn = numpy.vectorize(lambda z: 1.0 if z >= 0.0 else 0.0)
+step = step_fn(z)
+
+fig = plt.figure()
+ax = fig.add_subplot(111)
+ax.plot(z, step)
+ax.set_ylim([-0.5, 1.5])
+ax.set_xlim([-5,5])
+ax.grid(True)
+ax.set_xlabel('z')
+ax.set_title('step function')
+
+plt.show()
+
+"""Sine Function"""
+z = numpy.arange(-2*mt.pi, 2*mt.pi, 0.1)
+t = numpy.sin(z)
+
+fig = plt.figure()
+ax = fig.add_subplot(111)
+ax.plot(z, t)
+ax.set_ylim([-1.0, 1.0])
+ax.set_xlim([-2*mt.pi,2*mt.pi])
+ax.grid(True)
+ax.set_xlabel('z')
+ax.set_title('sine function')
+
+plt.show()
+
+"""Plots a graph of the squashing function used by a rectified linear
+unit"""
+z = numpy.arange(-2, 2, .1)
+zero = numpy.zeros(len(z))
+y = numpy.max([zero, z], axis=0)
+
+fig = plt.figure()
+ax = fig.add_subplot(111)
+ax.plot(z, y)
+ax.set_ylim([-2.0, 2.0])
+ax.set_xlim([-2.0, 2.0])
+ax.grid(True)
+ax.set_xlabel('z')
+ax.set_title('Rectified linear unit')
+
+plt.show()
+
    +
    +
    +
    +
    +
    +

    Vanishing gradients#

    +

    The Back propagation algorithm we derived above works by going from +the output layer to the input layer, propagating the error gradient on +the way. Once the algorithm has computed the gradient of the cost +function with regards to each parameter in the network, it uses these +gradients to update each parameter with a Gradient Descent (GD) step.

    +

    Unfortunately for us, the gradients often get smaller and smaller as +the algorithm progresses down to the first hidden layers. As a result, +the GD update leaves the lower layer connection weights virtually +unchanged, and training never converges to a good solution. This is +known in the literature as the vanishing gradients problem.

    +
    +
    +

    Exploding gradients#

    +

    In other cases, the opposite can happen, namely the the gradients can +grow bigger and bigger. The result is that many of the layers get +large updates of the weights the algorithm diverges. This is the +exploding gradients problem, which is mostly encountered in +recurrent neural networks. More generally, deep neural networks suffer +from unstable gradients, different layers may learn at widely +different speeds

    +
    +
    +

    Is the Logistic activation function (Sigmoid) our choice?#

    +

    Although this unfortunate behavior has been empirically observed for +quite a while (it was one of the reasons why deep neural networks were +mostly abandoned for a long time), it is only around 2010 that +significant progress was made in understanding it.

    +

    A paper titled Understanding the Difficulty of Training Deep +Feedforward Neural Networks by Xavier Glorot and Yoshua Bengio found that +the problems with the popular logistic +sigmoid activation function and the weight initialization technique +that was most popular at the time, namely random initialization using +a normal distribution with a mean of 0 and a standard deviation of +1.

    +
    +
    +

    Logistic function as the root of problems#

    +

    They showed that with this activation function and this +initialization scheme, the variance of the outputs of each layer is +much greater than the variance of its inputs. Going forward in the +network, the variance keeps increasing after each layer until the +activation function saturates at the top layers. This is actually made +worse by the fact that the logistic function has a mean of 0.5, not 0 +(the hyperbolic tangent function has a mean of 0 and behaves slightly +better than the logistic function in deep networks).

    +
    +
    +

    The derivative of the Logistic funtion#

    +

    Looking at the logistic activation function, when inputs become large +(negative or positive), the function saturates at 0 or 1, with a +derivative extremely close to 0. Thus when backpropagation kicks in, +it has virtually no gradient to propagate back through the network, +and what little gradient exists keeps getting diluted as +backpropagation progresses down through the top layers, so there is +really nothing left for the lower layers.

    +

    In their paper, Glorot and Bengio propose a way to significantly +alleviate this problem. We need the signal to flow properly in both +directions: in the forward direction when making predictions, and in +the reverse direction when backpropagating gradients. We don’t want +the signal to die out, nor do we want it to explode and saturate. For +the signal to flow properly, the authors argue that we need the +variance of the outputs of each layer to be equal to the variance of +its inputs, and we also need the gradients to have equal variance +before and after flowing through a layer in the reverse direction.

    +
    +
    +

    Insights from the paper by Glorot and Bengio#

    +

    One of the insights in the 2010 paper by Glorot and Bengio was that +the vanishing/exploding gradients problems were in part due to a poor +choice of activation function. Until then most people had assumed that +if Nature had chosen to use roughly sigmoid activation functions in +biological neurons, they must be an excellent choice. But it turns out +that other activation functions behave much better in deep neural +networks, in particular the ReLU activation function, mostly because +it does not saturate for positive values (and also because it is quite +fast to compute).

    +
    +
    +

    The RELU function family#

    +

    The ReLU activation function suffers from a problem known as the dying +ReLUs: during training, some neurons effectively die, meaning they +stop outputting anything other than 0.

    +

    In some cases, you may find that half of your network’s neurons are +dead, especially if you used a large learning rate. During training, +if a neuron’s weights get updated such that the weighted sum of the +neuron’s inputs is negative, it will start outputting 0. When this +happen, the neuron is unlikely to come back to life since the gradient +of the ReLU function is 0 when its input is negative.

    +
    +
    +

    ELU function#

    +

    To solve this problem, nowadays practitioners use a variant of the +ReLU function, such as the leaky ReLU discussed above or the so-called +exponential linear unit (ELU) function

    +
    +\[\begin{split} +ELU(z) = \left\{\begin{array}{cc} \alpha\left( \exp{(z)}-1\right) & z < 0,\\ z & z \ge 0.\end{array}\right. +\end{split}\]
    +
    +
    +

    Which activation function should we use?#

    +

    In general it seems that the ELU activation function is better than +the leaky ReLU function (and its variants), which is better than +ReLU. ReLU performs better than \(\tanh\) which in turn performs better +than the logistic function.

    +

    If runtime performance is an issue, then you may opt for the leaky +ReLU function over the ELU function If you don’t want to tweak yet +another hyperparameter, you may just use the default \(\alpha\) of +\(0.01\) for the leaky ReLU, and \(1\) for ELU. If you have spare time and +computing power, you can use cross-validation or bootstrap to evaluate +other activation functions.

    +
    +
    +

    More on activation functions, output layers#

    +

    In most cases you can use the ReLU activation function in the hidden +layers (or one of its variants).

    +

    It is a bit faster to compute than other activation functions, and the +gradient descent optimization does in general not get stuck.

    +

    For the output layer:

    +
      +

    • For classification the softmax activation function is generally a good choice for classification tasks (when the classes are mutually exclusive).

      • +

    • For regression tasks, you can simply use no activation function at all.

      • +
    +
    +
    +

    Fine-tuning neural network hyperparameters#

    +

    The flexibility of neural networks is also one of their main +drawbacks: there are many hyperparameters to tweak. Not only can you +use any imaginable network topology (how neurons/nodes are +interconnected), but even in a simple FFNN you can change the number +of layers, the number of neurons per layer, the type of activation +function to use in each layer, the weight initialization logic, the +stochastic gradient optmized and much more. How do you know what +combination of hyperparameters is the best for your task?

    +
      +

    • You can use grid search with cross-validation to find the right hyperparameters.

      • +
    +

    However,since there are many hyperparameters to tune, and since +training a neural network on a large dataset takes a lot of time, you +will only be able to explore a tiny part of the hyperparameter space.

    +
      +

    • You can use randomized search.

      • +

    • Or use tools like Oscar, which implements more complex algorithms to help you find a good set of hyperparameters quickly.

      • +
    +
    +
    +

    Hidden layers#

    +

    For many problems you can start with just one or two hidden layers and +it will work just fine. For the MNIST data set discussed below you can easily get a +high accuracy using just one hidden layer with a few hundred neurons. +You can reach for this data set above 98% accuracy using two hidden +layers with the same total amount of neurons, in roughly the same +amount of training time.

    +

    For more complex problems, you can gradually ramp up the number of +hidden layers, until you start overfitting the training set. Very +complex tasks, such as large image classification or speech +recognition, typically require networks with dozens of layers and they +need a huge amount of training data. However, you will rarely have to +train such networks from scratch: it is much more common to reuse +parts of a pretrained state-of-the-art network that performs a similar +task.

    +
    +
    +

    Batch Normalization#

    +

    Batch Normalization aims to address the vanishing/exploding gradients +problems, and more generally the problem that the distribution of each +layer’s inputs changes during training, as the parameters of the +previous layers change.

    +

    The technique consists of adding an operation in the model just before +the activation function of each layer, simply zero-centering and +normalizing the inputs, then scaling and shifting the result using two +new parameters per layer (one for scaling, the other for shifting). In +other words, this operation lets the model learn the optimal scale and +mean of the inputs for each layer. In order to zero-center and +normalize the inputs, the algorithm needs to estimate the inputs’ mean +and standard deviation. It does so by evaluating the mean and standard +deviation of the inputs over the current mini-batch, from this the +name batch normalization.

    +
    +
    +

    Dropout#

    +

    It is a fairly simple algorithm: at every training step, every neuron +(including the input neurons but excluding the output neurons) has a +probability \(p\) of being temporarily dropped out, meaning it will be +entirely ignored during this training step, but it may be active +during the next step.

    +

    The hyperparameter \(p\) is called the dropout rate, and it is typically +set to 50%. After training, the neurons are not dropped anymore. It +is viewed as one of the most popular regularization techniques.

    +
    +
    +

    Gradient Clipping#

    +

    A popular technique to lessen the exploding gradients problem is to +simply clip the gradients during backpropagation so that they never +exceed some threshold (this is mostly useful for recurrent neural +networks).

    +

    This technique is called Gradient Clipping.

    +

    In general however, Batch +Normalization is preferred.

    +
    +
    +

    A top-down perspective on Neural networks#

    +

    The first thing we would like to do is divide the data into two or +three parts. A training set, a validation or dev (development) set, +and a test set. The test set is the data on which we want to make +predictions. The dev set is a subset of the training data we use to +check how well we are doing out-of-sample, after training the model on +the training dataset. We use the validation error as a proxy for the +test error in order to make tweaks to our model. It is crucial that we +do not use any of the test data to train the algorithm. This is a +cardinal sin in ML. Then:

    +
      +

    1. Estimate optimal error rate

      2. +

    2. Minimize underfitting (bias) on training data set.

      3. +

    3. Make sure you are not overfitting.

      4. +
    +
    +
    +

    More top-down perspectives#

    +

    If the validation and test sets are drawn from the same distributions, +then a good performance on the validation set should lead to similarly +good performance on the test set.

    +

    However, sometimes +the training data and test data differ in subtle ways because, for +example, they are collected using slightly different methods, or +because it is cheaper to collect data in one way versus another. In +this case, there can be a mismatch between the training and test +data. This can lead to the neural network overfitting these small +differences between the test and training sets, and a poor performance +on the test set despite having a good performance on the validation +set. To rectify this, Andrew Ng suggests making two validation or dev +sets, one constructed from the training data and one constructed from +the test data. The difference between the performance of the algorithm +on these two validation sets quantifies the train-test mismatch. This +can serve as another important diagnostic when using DNNs for +supervised learning.

    +
    +
    +

    Limitations of supervised learning with deep networks#

    +

    Like all statistical methods, supervised learning using neural +networks has important limitations. This is especially important when +one seeks to apply these methods, especially to physics problems. Like +all tools, DNNs are not a universal solution. Often, the same or +better performance on a task can be achieved by using a few +hand-engineered features (or even a collection of random +features).

    +
    +
    +

    Limitations of NNs#

    +

    Here we list some of the important limitations of supervised neural network based models.

    +
      +

    • Need labeled data. All supervised learning methods, DNNs for supervised learning require labeled data. Often, labeled data is harder to acquire than unlabeled data (e.g. one must pay for human experts to label images).

      • +

    • Supervised neural networks are extremely data intensive. DNNs are data hungry. They perform best when data is plentiful. This is doubly so for supervised methods where the data must also be labeled. The utility of DNNs is extremely limited if data is hard to acquire or the datasets are small (hundreds to a few thousand samples). In this case, the performance of other methods that utilize hand-engineered features can exceed that of DNNs.

      • +
    +
    +
    +

    Homogeneous data#

    +
      +

    • Homogeneous data. Almost all DNNs deal with homogeneous data of one type. It is very hard to design architectures that mix and match data types (i.e. some continuous variables, some discrete variables, some time series). In applications beyond images, video, and language, this is often what is required. In contrast, ensemble models like random forests or gradient-boosted trees have no difficulty handling mixed data types.

      • +
    +
    +
    +

    More limitations#

    +
      +

    • Many problems are not about prediction. In natural science we are often interested in learning something about the underlying distribution that generates the data. In this case, it is often difficult to cast these ideas in a supervised learning setting. While the problems are related, it is possible to make good predictions with a wrong model. The model might or might not be useful for understanding the underlying science.

      • +
    +

    Some of these remarks are particular to DNNs, others are shared by all supervised learning methods. This motivates the use of unsupervised methods which in part circumvent these problems.

    +
    +
    +

    Setting up a Multi-layer perceptron model for classification#

    +

    We are now gong to develop an example based on the MNIST data +base. This is a classification problem and we need to use our +cross-entropy function we discussed in connection with logistic +regression. The cross-entropy defines our cost function for the +classificaton problems with neural networks.

    +

    In binary classification with two classes \((0, 1)\) we define the +logistic/sigmoid function as the probability that a particular input +is in class \(0\) or \(1\). This is possible because the logistic +function takes any input from the real numbers and inputs a number +between 0 and 1, and can therefore be interpreted as a probability. It +also has other nice properties, such as a derivative that is simple to +calculate.

    +

    For an input \(\boldsymbol{a}\) from the hidden layer, the probability that the input \(\boldsymbol{x}\) +is in class 0 or 1 is just. We let \(\theta\) represent the unknown weights and biases to be adjusted by our equations). The variable \(x\) +represents our activation values \(z\). We have

    +
    +\[ +P(y = 0 \mid \boldsymbol{x}, \boldsymbol{\theta}) = \frac{1}{1 + \exp{(- \boldsymbol{x}})} , +\]
    +

    and

    +
    +\[ +P(y = 1 \mid \boldsymbol{x}, \boldsymbol{\theta}) = 1 - P(y = 0 \mid \boldsymbol{x}, \boldsymbol{\theta}) , +\]
    +

    where \(y \in \{0, 1\}\) and \(\boldsymbol{\theta}\) represents the weights and biases +of our network.

    +
    +
    +

    Defining the cost function#

    +

    Our cost function is given as (see the Logistic regression lectures)

    +
    +\[ +\mathcal{C}(\boldsymbol{\theta}) = - \ln P(\mathcal{D} \mid \boldsymbol{\theta}) = - \sum_{i=1}^n +y_i \ln[P(y_i = 0)] + (1 - y_i) \ln [1 - P(y_i = 0)] = \sum_{i=1}^n \mathcal{L}_i(\boldsymbol{\theta}) . +\]
    +

    This last equality means that we can interpret our cost function as a sum over the loss function +for each point in the dataset \(\mathcal{L}_i(\boldsymbol{\theta})\).
    +The negative sign is just so that we can think about our algorithm as minimizing a positive number, rather +than maximizing a negative number.

    +

    In multiclass classification it is common to treat each integer label as a so called one-hot vector:

    +

    \(y = 5 \quad \rightarrow \quad \boldsymbol{y} = (0, 0, 0, 0, 0, 1, 0, 0, 0, 0) ,\) and

    +

    \(y = 1 \quad \rightarrow \quad \boldsymbol{y} = (0, 1, 0, 0, 0, 0, 0, 0, 0, 0) ,\)

    +

    i.e. a binary bit string of length \(C\), where \(C = 10\) is the number of classes in the MNIST dataset (numbers from \(0\) to \(9\))..

    +

    If \(\boldsymbol{x}_i\) is the \(i\)-th input (image), \(y_{ic}\) refers to the \(c\)-th component of the \(i\)-th +output vector \(\boldsymbol{y}_i\).
    +The probability of \(\boldsymbol{x}_i\) being in class \(c\) will be given by the softmax function:

    +
    +\[ +P(y_{ic} = 1 \mid \boldsymbol{x}_i, \boldsymbol{\theta}) = \frac{\exp{((\boldsymbol{a}_i^{hidden})^T \boldsymbol{w}_c)}} +{\sum_{c'=0}^{C-1} \exp{((\boldsymbol{a}_i^{hidden})^T \boldsymbol{w}_{c'})}} , +\]
    +

    which reduces to the logistic function in the binary case.
    +The likelihood of this \(C\)-class classifier +is now given as:

    +
    +\[ +P(\mathcal{D} \mid \boldsymbol{\theta}) = \prod_{i=1}^n \prod_{c=0}^{C-1} [P(y_{ic} = 1)]^{y_{ic}} . +\]
    +

    Again we take the negative log-likelihood to define our cost function:

    +
    +\[ +\mathcal{C}(\boldsymbol{\theta}) = - \log{P(\mathcal{D} \mid \boldsymbol{\theta})}. +\]
    +

    See the logistic regression lectures for a full definition of the cost function.

    +

    The back propagation equations need now only a small change, namely the definition of a new cost function. We are thus ready to use the same equations as before!

    +
    +
    +

    Example: binary classification problem#

    +

    As an example of the above, relevant for project 2 as well, let us consider a binary class. As discussed in our logistic regression lectures, we defined a cost function in terms of the parameters \(\beta\) as

    +
    +\[ +\mathcal{C}(\boldsymbol{\beta}) = - \sum_{i=1}^n \left(y_i\log{p(y_i \vert x_i,\boldsymbol{\beta})}+(1-y_i)\log{1-p(y_i \vert x_i,\boldsymbol{\beta})}\right), +\]
    +

    where we had defined the logistic (sigmoid) function

    +
    +\[ +p(y_i =1\vert x_i,\boldsymbol{\beta})=\frac{\exp{(\beta_0+\beta_1 x_i)}}{1+\exp{(\beta_0+\beta_1 x_i)}}, +\]
    +

    and

    +
    +\[ +p(y_i =0\vert x_i,\boldsymbol{\beta})=1-p(y_i =1\vert x_i,\boldsymbol{\beta}). +\]
    +

    The parameters \(\boldsymbol{\beta}\) were defined using a minimization method like gradient descent or Newton-Raphson’s method.

    +

    Now we replace \(x_i\) with the activation \(z_i^l\) for a given layer \(l\) and the outputs as \(y_i=a_i^l=f(z_i^l)\), with \(z_i^l\) now being a function of the weights \(w_{ij}^l\) and biases \(b_i^l\). +We have then

    +
    +\[ +a_i^l = y_i = \frac{\exp{(z_i^l)}}{1+\exp{(z_i^l)}}, +\]
    +

    with

    +
    +\[ +z_i^l = \sum_{j}w_{ij}^l a_j^{l-1}+b_i^l, +\]
    +

    where the superscript \(l-1\) indicates that these are the outputs from layer \(l-1\). +Our cost function at the final layer \(l=L\) is now

    +
    +\[ +\mathcal{C}(\boldsymbol{W}) = - \sum_{i=1}^n \left(t_i\log{a_i^L}+(1-t_i)\log{(1-a_i^L)}\right), +\]
    +

    where we have defined the targets \(t_i\). The derivatives of the cost function with respect to the output \(a_i^L\) are then easily calculated and we get

    +
    +\[ +\frac{\partial \mathcal{C}(\boldsymbol{W})}{\partial a_i^L} = \frac{a_i^L-t_i}{a_i^L(1-a_i^L)}. +\]
    +

    In case we use another activation function than the logistic one, we need to evaluate other derivatives.

    +
    +
    +

    The Softmax function#

    +

    In case we employ the more general case given by the Softmax equation, we need to evaluate the derivative of the activation function with respect to the activation \(z_i^l\), that is we need

    +
    +\[ +\frac{\partial f(z_i^l)}{\partial w_{jk}^l} = +\frac{\partial f(z_i^l)}{\partial z_j^l} \frac{\partial z_j^l}{\partial w_{jk}^l}= \frac{\partial f(z_i^l)}{\partial z_j^l}a_k^{l-1}. +\]
    +

    For the Softmax function we have

    +
    +\[ +f(z_i^l) = \frac{\exp{(z_i^l)}}{\sum_{m=1}^K\exp{(z_m^l)}}. +\]
    +

    Its derivative with respect to \(z_j^l\) gives

    +
    +\[ +\frac{\partial f(z_i^l)}{\partial z_j^l}= f(z_i^l)\left(\delta_{ij}-f(z_j^l)\right), +\]
    +

    which in case of the simply binary model reduces to having \(i=j\).

    +
    +
    +

    Developing a code for doing neural networks with back propagation#

    +

    One can identify a set of key steps when using neural networks to solve supervised learning problems:

    +
      +

    1. Collect and pre-process data

      2. +

    2. Define model and architecture

      3. +

    3. Choose cost function and optimizer

      4. +

    4. Train the model

      5. +

    5. Evaluate model performance on test data

      6. +

    6. Adjust hyperparameters (if necessary, network architecture)

      7. +
    +
    +
    +

    Collect and pre-process data#

    +

    Here we will be using the MNIST dataset, which is readily available through the scikit-learn +package. You may also find it for example here.
    +The MNIST (Modified National Institute of Standards and Technology) database is a large database +of handwritten digits that is commonly used for training various image processing systems.
    +The MNIST dataset consists of 70 000 images of size \(28\times 28\) pixels, each labeled from 0 to 9.
    +The scikit-learn dataset we will use consists of a selection of 1797 images of size \(8\times 8\) collected and processed from this database.

    +

    To feed data into a feed-forward neural network we need to represent +the inputs as a design/feature matrix \(X = (n_{inputs}, n_{features})\). Each +row represents an input, in this case a handwritten digit, and +each column represents a feature, in this case a pixel. The +correct answers, also known as labels or targets are +represented as a 1D array of integers +\(Y = (n_{inputs}) = (5, 3, 1, 8,...)\).

    +

    As an example, say we want to build a neural network using supervised learning to predict Body-Mass Index (BMI) from +measurements of height (in m)
    +and weight (in kg). If we have measurements of 5 people the design/feature matrix could be for example:

    +
    +\[\begin{split} X = \begin{bmatrix} +1.85 & 81\\ +1.71 & 65\\ +1.95 & 103\\ +1.55 & 42\\ +1.63 & 56 +\end{bmatrix} ,\end{split}\]
    +

    and the targets would be:

    +
    +\[ Y = (23.7, 22.2, 27.1, 17.5, 21.1) \]
    +

    Since each input image is a 2D matrix, we need to flatten the image +(i.e. “unravel” the 2D matrix into a 1D array) to turn the data into a +design/feature matrix. This means we lose all spatial information in the +image, such as locality and translational invariance. More complicated +architectures such as Convolutional Neural Networks can take advantage +of such information, and are most commonly applied when analyzing +images.

    +
    +
    +
    # import necessary packages
+import numpy as np
+import matplotlib.pyplot as plt
+from sklearn import datasets
+
+
+# ensure the same random numbers appear every time
+np.random.seed(0)
+
+# display images in notebook
+%matplotlib inline
+plt.rcParams['figure.figsize'] = (12,12)
+
+
+# download MNIST dataset
+digits = datasets.load_digits()
+
+# define inputs and labels
+inputs = digits.images
+labels = digits.target
+
+print("inputs = (n_inputs, pixel_width, pixel_height) = " + str(inputs.shape))
+print("labels = (n_inputs) = " + str(labels.shape))
+
+
+# flatten the image
+# the value -1 means dimension is inferred from the remaining dimensions: 8x8 = 64
+n_inputs = len(inputs)
+inputs = inputs.reshape(n_inputs, -1)
+print("X = (n_inputs, n_features) = " + str(inputs.shape))
+
+
+# choose some random images to display
+indices = np.arange(n_inputs)
+random_indices = np.random.choice(indices, size=5)
+
+for i, image in enumerate(digits.images[random_indices]):
+    plt.subplot(1, 5, i+1)
+    plt.axis('off')
+    plt.imshow(image, cmap=plt.cm.gray_r, interpolation='nearest')
+    plt.title("Label: %d" % digits.target[random_indices[i]])
+plt.show()
+
    +
    +
    +
    +
    +
    +

    Train and test datasets#

    +

    Performing analysis before partitioning the dataset is a major error, that can lead to incorrect conclusions.

    +

    We will reserve \(80 \%\) of our dataset for training and \(20 \%\) for testing.

    +

    It is important that the train and test datasets are drawn randomly from our dataset, to ensure +no bias in the sampling.
    +Say you are taking measurements of weather data to predict the weather in the coming 5 days. +You don’t want to train your model on measurements taken from the hours 00.00 to 12.00, and then test it on data +collected from 12.00 to 24.00.

    +
    +
    +
    from sklearn.model_selection import train_test_split
+
+# one-liner from scikit-learn library
+train_size = 0.8
+test_size = 1 - train_size
+X_train, X_test, Y_train, Y_test = train_test_split(inputs, labels, train_size=train_size,
+                                                    test_size=test_size)
+
+# equivalently in numpy
+def train_test_split_numpy(inputs, labels, train_size, test_size):
+    n_inputs = len(inputs)
+    inputs_shuffled = inputs.copy()
+    labels_shuffled = labels.copy()
+    
+    np.random.shuffle(inputs_shuffled)
+    np.random.shuffle(labels_shuffled)
+    
+    train_end = int(n_inputs*train_size)
+    X_train, X_test = inputs_shuffled[:train_end], inputs_shuffled[train_end:]
+    Y_train, Y_test = labels_shuffled[:train_end], labels_shuffled[train_end:]
+    
+    return X_train, X_test, Y_train, Y_test
+
+#X_train, X_test, Y_train, Y_test = train_test_split_numpy(inputs, labels, train_size, test_size)
+
+print("Number of training images: " + str(len(X_train)))
+print("Number of test images: " + str(len(X_test)))
+
    +
    +
    +
    +
    +
    +

    Define model and architecture#

    +

    Our simple feed-forward neural network will consist of an input layer, a single hidden layer and an output layer. The activation \(y\) of each neuron is a weighted sum of inputs, passed through an activation function. In case of the simple perceptron model we have

    +
    +\[ z = \sum_{i=1}^n w_i a_i ,\]
    +
    +\[ y = f(z) ,\]
    +

    where \(f\) is the activation function, \(a_i\) represents input from neuron \(i\) in the preceding layer +and \(w_i\) is the weight to input \(i\).
    +The activation of the neurons in the input layer is just the features (e.g. a pixel value).

    +

    The simplest activation function for a neuron is the Heaviside function:

    +
    +\[\begin{split} f(z) = +\begin{cases} +1, & z > 0\\ +0, & \text{otherwise} +\end{cases} +\end{split}\]
    +

    A feed-forward neural network with this activation is known as a perceptron.
    +For a binary classifier (i.e. two classes, 0 or 1, dog or not-dog) we can also use this in our output layer.
    +This activation can be generalized to \(k\) classes (using e.g. the one-against-all strategy), +and we call these architectures multiclass perceptrons.

    +

    However, it is now common to use the terms Single Layer Perceptron (SLP) (1 hidden layer) and
    +Multilayer Perceptron (MLP) (2 or more hidden layers) to refer to feed-forward neural networks with any activation function.

    +

    Typical choices for activation functions include the sigmoid function, hyperbolic tangent, and Rectified Linear Unit (ReLU).
    +We will be using the sigmoid function \(\sigma(x)\):

    +
    +\[ f(x) = \sigma(x) = \frac{1}{1 + e^{-x}} ,\]
    +

    which is inspired by probability theory (see logistic regression) and was most commonly used until about 2011. See the discussion below concerning other activation functions.

    +
    +
    +

    Layers#

    +
      +

    • Input

      • +
    +

    Since each input image has 8x8 = 64 pixels or features, we have an input layer of 64 neurons.

    +
      +

    • Hidden layer

      • +
    +

    We will use 50 neurons in the hidden layer receiving input from the neurons in the input layer.
    +Since each neuron in the hidden layer is connected to the 64 inputs we have 64x50 = 3200 weights to the hidden layer.

    +
      +

    • Output

      • +
    +

    If we were building a binary classifier, it would be sufficient with a single neuron in the output layer, +which could output 0 or 1 according to the Heaviside function. This would be an example of a hard classifier, meaning it outputs the class of the input directly. However, if we are dealing with noisy data it is often beneficial to use a soft classifier, which outputs the probability of being in class 0 or 1.

    +

    For a soft binary classifier, we could use a single neuron and interpret the output as either being the probability of being in class 0 or the probability of being in class 1. Alternatively we could use 2 neurons, and interpret each neuron as the probability of being in each class.

    +

    Since we are doing multiclass classification, with 10 categories, it is natural to use 10 neurons in the output layer. We number the neurons \(j = 0,1,...,9\). The activation of each output neuron \(j\) will be according to the softmax function:

    +
    +\[ P(\text{class $j$} \mid \text{input $\boldsymbol{a}$}) = \frac{\exp{(\boldsymbol{a}^T \boldsymbol{w}_j)}} +{\sum_{c=0}^{9} \exp{(\boldsymbol{a}^T \boldsymbol{w}_c)}} ,\]
    +

    i.e. each neuron \(j\) outputs the probability of being in class \(j\) given an input from the hidden layer \(\boldsymbol{a}\), with \(\boldsymbol{w}_j\) the weights of neuron \(j\) to the inputs.
    +The denominator is a normalization factor to ensure the outputs (probabilities) sum up to 1.
    +The exponent is just the weighted sum of inputs as before:

    +
    +\[ z_j = \sum_{i=1}^n w_ {ij} a_i+b_j.\]
    +

    Since each neuron in the output layer is connected to the 50 inputs from the hidden layer we have 50x10 = 500 +weights to the output layer.

    +
    +
    +

    Weights and biases#

    +

    Typically weights are initialized with small values distributed around zero, drawn from a uniform +or normal distribution. Setting all weights to zero means all neurons give the same output, making the network useless.

    +

    Adding a bias value to the weighted sum of inputs allows the neural network to represent a greater range +of values. Without it, any input with the value 0 will be mapped to zero (before being passed through the activation). The bias unit has an output of 1, and a weight to each neuron \(j\), \(b_j\):

    +
    +\[ z_j = \sum_{i=1}^n w_ {ij} a_i + b_j.\]
    +

    The bias weights \(\boldsymbol{b}\) are often initialized to zero, but a small value like \(0.01\) ensures all neurons have some output which can be backpropagated in the first training cycle.

    +
    +
    +
    # building our neural network
+
+n_inputs, n_features = X_train.shape
+n_hidden_neurons = 50
+n_categories = 10
+
+# we make the weights normally distributed using numpy.random.randn
+
+# weights and bias in the hidden layer
+hidden_weights = np.random.randn(n_features, n_hidden_neurons)
+hidden_bias = np.zeros(n_hidden_neurons) + 0.01
+
+# weights and bias in the output layer
+output_weights = np.random.randn(n_hidden_neurons, n_categories)
+output_bias = np.zeros(n_categories) + 0.01
+
    +
    +
    +
    +
    +
    +

    Feed-forward pass#

    +

    Denote \(F\) the number of features, \(H\) the number of hidden neurons and \(C\) the number of categories.
    +For each input image we calculate a weighted sum of input features (pixel values) to each neuron \(j\) in the hidden layer \(l\):

    +
    +\[ z_{j}^{l} = \sum_{i=1}^{F} w_{ij}^{l} x_i + b_{j}^{l},\]
    +

    this is then passed through our activation function

    +
    +\[ a_{j}^{l} = f(z_{j}^{l}) .\]
    +

    We calculate a weighted sum of inputs (activations in the hidden layer) to each neuron \(j\) in the output layer:

    +
    +\[ z_{j}^{L} = \sum_{i=1}^{H} w_{ij}^{L} a_{i}^{l} + b_{j}^{L}.\]
    +

    Finally we calculate the output of neuron \(j\) in the output layer using the softmax function:

    +
    +\[ a_{j}^{L} = \frac{\exp{(z_j^{L})}} +{\sum_{c=0}^{C-1} \exp{(z_c^{L})}} .\]
    +
    +
    +

    Matrix multiplications#

    +

    Since our data has the dimensions \(X = (n_{inputs}, n_{features})\) and our weights to the hidden +layer have the dimensions
    +\(W_{hidden} = (n_{features}, n_{hidden})\), +we can easily feed the network all our training data in one go by taking the matrix product

    +
    +\[ X W^{h} = (n_{inputs}, n_{hidden}),\]
    +

    and obtain a matrix that holds the weighted sum of inputs to the hidden layer +for each input image and each hidden neuron.
    +We also add the bias to obtain a matrix of weighted sums to the hidden layer \(Z^{h}\):

    +
    +\[ \boldsymbol{z}^{l} = \boldsymbol{X} \boldsymbol{W}^{l} + \boldsymbol{b}^{l} ,\]
    +

    meaning the same bias (1D array with size equal number of hidden neurons) is added to each input image.
    +This is then passed through the activation:

    +
    +\[ \boldsymbol{a}^{l} = f(\boldsymbol{z}^l) .\]
    +

    This is fed to the output layer:

    +
    +\[ \boldsymbol{z}^{L} = \boldsymbol{a}^{L} \boldsymbol{W}^{L} + \boldsymbol{b}^{L} .\]
    +

    Finally we receive our output values for each image and each category by passing it through the softmax function:

    +
    +\[ output = softmax (\boldsymbol{z}^{L}) = (n_{inputs}, n_{categories}) .\]
    +
    +
    +
    # setup the feed-forward pass, subscript h = hidden layer
+
+def sigmoid(x):
+    return 1/(1 + np.exp(-x))
+
+def feed_forward(X):
+    # weighted sum of inputs to the hidden layer
+    z_h = np.matmul(X, hidden_weights) + hidden_bias
+    # activation in the hidden layer
+    a_h = sigmoid(z_h)
+    
+    # weighted sum of inputs to the output layer
+    z_o = np.matmul(a_h, output_weights) + output_bias
+    # softmax output
+    # axis 0 holds each input and axis 1 the probabilities of each category
+    exp_term = np.exp(z_o)
+    probabilities = exp_term / np.sum(exp_term, axis=1, keepdims=True)
+    
+    return probabilities
+
+probabilities = feed_forward(X_train)
+print("probabilities = (n_inputs, n_categories) = " + str(probabilities.shape))
+print("probability that image 0 is in category 0,1,2,...,9 = \n" + str(probabilities[0]))
+print("probabilities sum up to: " + str(probabilities[0].sum()))
+print()
+
+# we obtain a prediction by taking the class with the highest likelihood
+def predict(X):
+    probabilities = feed_forward(X)
+    return np.argmax(probabilities, axis=1)
+
+predictions = predict(X_train)
+print("predictions = (n_inputs) = " + str(predictions.shape))
+print("prediction for image 0: " + str(predictions[0]))
+print("correct label for image 0: " + str(Y_train[0]))
+
    +
    +
    +
    +
    +
    +

    Choose cost function and optimizer#

    +

    To measure how well our neural network is doing we need to introduce a cost function.
    +We will call the function that gives the error of a single sample output the loss function, and the function +that gives the total error of our network across all samples the cost function. +A typical choice for multiclass classification is the cross-entropy loss, also known as the negative log likelihood.

    +

    In multiclass classification it is common to treat each integer label as a so called one-hot vector:

    +
    +\[ y = 5 \quad \rightarrow \quad \boldsymbol{y} = (0, 0, 0, 0, 0, 1, 0, 0, 0, 0) ,\]
    +
    +\[ y = 1 \quad \rightarrow \quad \boldsymbol{y} = (0, 1, 0, 0, 0, 0, 0, 0, 0, 0) ,\]
    +

    i.e. a binary bit string of length \(C\), where \(C = 10\) is the number of classes in the MNIST dataset.

    +

    Let \(y_{ic}\) denote the \(c\)-th component of the \(i\)-th one-hot vector.
    +We define the cost function \(\mathcal{C}\) as a sum over the cross-entropy loss for each point \(\boldsymbol{x}_i\) in the dataset.

    +

    In the one-hot representation only one of the terms in the loss function is non-zero, namely the +probability of the correct category \(c'\)
    +(i.e. the category \(c'\) such that \(y_{ic'} = 1\)). This means that the cross entropy loss only punishes you for how wrong +you got the correct label. The probability of category \(c\) is given by the softmax function. The vector \(\boldsymbol{\theta}\) represents the parameters of our network, i.e. all the weights and biases.

    +
    +
    +

    Optimizing the cost function#

    +

    The network is trained by finding the weights and biases that minimize the cost function. One of the most widely used classes of methods is gradient descent and its generalizations. The idea behind gradient descent +is simply to adjust the weights in the direction where the gradient of the cost function is large and negative. This ensures we flow toward a local minimum of the cost function.
    +Each parameter \(\theta\) is iteratively adjusted according to the rule

    +
    +\[ \theta_{i+1} = \theta_i - \eta \nabla \mathcal{C}(\theta_i) ,\]
    +

    where \(\eta\) is known as the learning rate, which controls how big a step we take towards the minimum.
    +This update can be repeated for any number of iterations, or until we are satisfied with the result.

    +

    A simple and effective improvement is a variant called Batch Gradient Descent.
    +Instead of calculating the gradient on the whole dataset, we calculate an approximation of the gradient +on a subset of the data called a minibatch.
    +If there are \(N\) data points and we have a minibatch size of \(M\), the total number of batches +is \(N/M\).
    +We denote each minibatch \(B_k\), with \(k = 1, 2,...,N/M\). The gradient then becomes:

    +
    +\[ \nabla \mathcal{C}(\theta) = \frac{1}{N} \sum_{i=1}^N \nabla \mathcal{L}_i(\theta) \quad \rightarrow \quad +\frac{1}{M} \sum_{i \in B_k} \nabla \mathcal{L}_i(\theta) ,\]
    +

    i.e. instead of averaging the loss over the entire dataset, we average over a minibatch.

    +

    This has two important benefits:

    +
      +

    1. Introducing stochasticity decreases the chance that the algorithm becomes stuck in a local minima.

      2. +

    2. It significantly speeds up the calculation, since we do not have to use the entire dataset to calculate the gradient.

      3. +
    +

    The various optmization methods, with codes and algorithms, are discussed in our lectures on Gradient descent approaches.

    +
    +
    +

    Regularization#

    +

    It is common to add an extra term to the cost function, proportional +to the size of the weights. This is equivalent to constraining the +size of the weights, so that they do not grow out of control. +Constraining the size of the weights means that the weights cannot +grow arbitrarily large to fit the training data, and in this way +reduces overfitting.

    +

    We will measure the size of the weights using the so called L2-norm, meaning our cost function becomes:

    +
    +\[ \mathcal{C}(\theta) = \frac{1}{N} \sum_{i=1}^N \mathcal{L}_i(\theta) \quad \rightarrow \quad +\frac{1}{N} \sum_{i=1}^N \mathcal{L}_i(\theta) + \lambda \lvert \lvert \boldsymbol{w} \rvert \rvert_2^2 += \frac{1}{N} \sum_{i=1}^N \mathcal{L}(\theta) + \lambda \sum_{ij} w_{ij}^2,\]
    +

    i.e. we sum up all the weights squared. The factor \(\lambda\) is known as a regularization parameter.

    +

    In order to train the model, we need to calculate the derivative of +the cost function with respect to every bias and weight in the +network. In total our network has \((64 + 1)\times 50=3250\) weights in +the hidden layer and \((50 + 1)\times 10=510\) weights to the output +layer (\(+1\) for the bias), and the gradient must be calculated for +every parameter. We use the backpropagation algorithm discussed +above. This is a clever use of the chain rule that allows us to +calculate the gradient efficently.

    +
    +
    +

    Matrix multiplication#

    +

    To more efficently train our network these equations are implemented using matrix operations.
    +The error in the output layer is calculated simply as, with \(\boldsymbol{t}\) being our targets,

    +
    +\[ \delta_L = \boldsymbol{t} - \boldsymbol{y} = (n_{inputs}, n_{categories}) .\]
    +

    The gradient for the output weights is calculated as

    +
    +\[ \nabla W_{L} = \boldsymbol{a}^T \delta_L = (n_{hidden}, n_{categories}) ,\]
    +

    where \(\boldsymbol{a} = (n_{inputs}, n_{hidden})\). This simply means that we are summing up the gradients for each input.
    +Since we are going backwards we have to transpose the activation matrix.

    +

    The gradient with respect to the output bias is then

    +
    +\[ \nabla \boldsymbol{b}_{L} = \sum_{i=1}^{n_{inputs}} \delta_L = (n_{categories}) .\]
    +

    The error in the hidden layer is

    +
    +\[ \Delta_h = \delta_L W_{L}^T \circ f'(z_{h}) = \delta_L W_{L}^T \circ a_{h} \circ (1 - a_{h}) = (n_{inputs}, n_{hidden}) ,\]
    +

    where \(f'(a_{h})\) is the derivative of the activation in the hidden layer. The matrix products mean +that we are summing up the products for each neuron in the output layer. The symbol \(\circ\) denotes +the Hadamard product, meaning element-wise multiplication.

    +

    This again gives us the gradients in the hidden layer:

    +
    +\[ \nabla W_{h} = X^T \delta_h = (n_{features}, n_{hidden}) ,\]
    +
    +\[ \nabla b_{h} = \sum_{i=1}^{n_{inputs}} \delta_h = (n_{hidden}) .\]
    +
    +
    +
    # to categorical turns our integer vector into a onehot representation
+from sklearn.metrics import accuracy_score
+
+# one-hot in numpy
+def to_categorical_numpy(integer_vector):
+    n_inputs = len(integer_vector)
+    n_categories = np.max(integer_vector) + 1
+    onehot_vector = np.zeros((n_inputs, n_categories))
+    onehot_vector[range(n_inputs), integer_vector] = 1
+    
+    return onehot_vector
+
+#Y_train_onehot, Y_test_onehot = to_categorical(Y_train), to_categorical(Y_test)
+Y_train_onehot, Y_test_onehot = to_categorical_numpy(Y_train), to_categorical_numpy(Y_test)
+
+def feed_forward_train(X):
+    # weighted sum of inputs to the hidden layer
+    z_h = np.matmul(X, hidden_weights) + hidden_bias
+    # activation in the hidden layer
+    a_h = sigmoid(z_h)
+    
+    # weighted sum of inputs to the output layer
+    z_o = np.matmul(a_h, output_weights) + output_bias
+    # softmax output
+    # axis 0 holds each input and axis 1 the probabilities of each category
+    exp_term = np.exp(z_o)
+    probabilities = exp_term / np.sum(exp_term, axis=1, keepdims=True)
+    
+    # for backpropagation need activations in hidden and output layers
+    return a_h, probabilities
+
+def backpropagation(X, Y):
+    a_h, probabilities = feed_forward_train(X)
+    
+    # error in the output layer
+    error_output = probabilities - Y
+    # error in the hidden layer
+    error_hidden = np.matmul(error_output, output_weights.T) * a_h * (1 - a_h)
+    
+    # gradients for the output layer
+    output_weights_gradient = np.matmul(a_h.T, error_output)
+    output_bias_gradient = np.sum(error_output, axis=0)
+    
+    # gradient for the hidden layer
+    hidden_weights_gradient = np.matmul(X.T, error_hidden)
+    hidden_bias_gradient = np.sum(error_hidden, axis=0)
+
+    return output_weights_gradient, output_bias_gradient, hidden_weights_gradient, hidden_bias_gradient
+
+print("Old accuracy on training data: " + str(accuracy_score(predict(X_train), Y_train)))
+
+eta = 0.01
+lmbd = 0.01
+for i in range(1000):
+    # calculate gradients
+    dWo, dBo, dWh, dBh = backpropagation(X_train, Y_train_onehot)
+    
+    # regularization term gradients
+    dWo += lmbd * output_weights
+    dWh += lmbd * hidden_weights
+    
+    # update weights and biases
+    output_weights -= eta * dWo
+    output_bias -= eta * dBo
+    hidden_weights -= eta * dWh
+    hidden_bias -= eta * dBh
+
+print("New accuracy on training data: " + str(accuracy_score(predict(X_train), Y_train)))
+
    +
    +
    +
    +
    +
    +

    Improving performance#

    +

    As we can see the network does not seem to be learning at all. It seems to be just guessing the label for each image.
    +In order to obtain a network that does something useful, we will have to do a bit more work.

    +

    The choice of hyperparameters such as learning rate and regularization parameter is hugely influential for the performance of the network. Typically a grid-search is performed, wherein we test different hyperparameters separated by orders of magnitude. For example we could test the learning rates \(\eta = 10^{-6}, 10^{-5},...,10^{-1}\) with different regularization parameters \(\lambda = 10^{-6},...,10^{-0}\).

    +

    Next, we haven’t implemented minibatching yet, which introduces stochasticity and is though to act as an important regularizer on the weights. We call a feed-forward + backward pass with a minibatch an iteration, and a full training period +going through the entire dataset (\(n/M\) batches) an epoch.

    +

    If this does not improve network performance, you may want to consider altering the network architecture, adding more neurons or hidden layers.
    +Andrew Ng goes through some of these considerations in this video. You can find a summary of the video here.

    +
    +
    +

    Full object-oriented implementation#

    +

    It is very natural to think of the network as an object, with specific instances of the network +being realizations of this object with different hyperparameters. An implementation using Python classes provides a clean structure and interface, and the full implementation of our neural network is given below.

    +
    +
    +
    class NeuralNetwork:
+    def __init__(
+            self,
+            X_data,
+            Y_data,
+            n_hidden_neurons=50,
+            n_categories=10,
+            epochs=10,
+            batch_size=100,
+            eta=0.1,
+            lmbd=0.0):
+
+        self.X_data_full = X_data
+        self.Y_data_full = Y_data
+
+        self.n_inputs = X_data.shape[0]
+        self.n_features = X_data.shape[1]
+        self.n_hidden_neurons = n_hidden_neurons
+        self.n_categories = n_categories
+
+        self.epochs = epochs
+        self.batch_size = batch_size
+        self.iterations = self.n_inputs // self.batch_size
+        self.eta = eta
+        self.lmbd = lmbd
+
+        self.create_biases_and_weights()
+
+    def create_biases_and_weights(self):
+        self.hidden_weights = np.random.randn(self.n_features, self.n_hidden_neurons)
+        self.hidden_bias = np.zeros(self.n_hidden_neurons) + 0.01
+
+        self.output_weights = np.random.randn(self.n_hidden_neurons, self.n_categories)
+        self.output_bias = np.zeros(self.n_categories) + 0.01
+
+    def feed_forward(self):
+        # feed-forward for training
+        self.z_h = np.matmul(self.X_data, self.hidden_weights) + self.hidden_bias
+        self.a_h = sigmoid(self.z_h)
+
+        self.z_o = np.matmul(self.a_h, self.output_weights) + self.output_bias
+
+        exp_term = np.exp(self.z_o)
+        self.probabilities = exp_term / np.sum(exp_term, axis=1, keepdims=True)
+
+    def feed_forward_out(self, X):
+        # feed-forward for output
+        z_h = np.matmul(X, self.hidden_weights) + self.hidden_bias
+        a_h = sigmoid(z_h)
+
+        z_o = np.matmul(a_h, self.output_weights) + self.output_bias
+        
+        exp_term = np.exp(z_o)
+        probabilities = exp_term / np.sum(exp_term, axis=1, keepdims=True)
+        return probabilities
+
+    def backpropagation(self):
+        error_output = self.probabilities - self.Y_data
+        error_hidden = np.matmul(error_output, self.output_weights.T) * self.a_h * (1 - self.a_h)
+
+        self.output_weights_gradient = np.matmul(self.a_h.T, error_output)
+        self.output_bias_gradient = np.sum(error_output, axis=0)
+
+        self.hidden_weights_gradient = np.matmul(self.X_data.T, error_hidden)
+        self.hidden_bias_gradient = np.sum(error_hidden, axis=0)
+
+        if self.lmbd > 0.0:
+            self.output_weights_gradient += self.lmbd * self.output_weights
+            self.hidden_weights_gradient += self.lmbd * self.hidden_weights
+
+        self.output_weights -= self.eta * self.output_weights_gradient
+        self.output_bias -= self.eta * self.output_bias_gradient
+        self.hidden_weights -= self.eta * self.hidden_weights_gradient
+        self.hidden_bias -= self.eta * self.hidden_bias_gradient
+
+    def predict(self, X):
+        probabilities = self.feed_forward_out(X)
+        return np.argmax(probabilities, axis=1)
+
+    def predict_probabilities(self, X):
+        probabilities = self.feed_forward_out(X)
+        return probabilities
+
+    def train(self):
+        data_indices = np.arange(self.n_inputs)
+
+        for i in range(self.epochs):
+            for j in range(self.iterations):
+                # pick datapoints with replacement
+                chosen_datapoints = np.random.choice(
+                    data_indices, size=self.batch_size, replace=False
+                )
+
+                # minibatch training data
+                self.X_data = self.X_data_full[chosen_datapoints]
+                self.Y_data = self.Y_data_full[chosen_datapoints]
+
+                self.feed_forward()
+                self.backpropagation()
+
    +
    +
    +
    +
    +
    +

    Evaluate model performance on test data#

    +

    To measure the performance of our network we evaluate how well it does it data it has never seen before, i.e. the test data.
    +We measure the performance of the network using the accuracy score.
    +The accuracy is as you would expect just the number of images correctly labeled divided by the total number of images. A perfect classifier will have an accuracy score of \(1\).

    +
    +\[ \text{Accuracy} = \frac{\sum_{i=1}^n I(\tilde{y}_i = y_i)}{n} ,\]
    +

    where \(I\) is the indicator function, \(1\) if \(\tilde{y}_i = y_i\) and \(0\) otherwise.

    +
    +
    +
    epochs = 100
+batch_size = 100
+
+dnn = NeuralNetwork(X_train, Y_train_onehot, eta=eta, lmbd=lmbd, epochs=epochs, batch_size=batch_size,
+                    n_hidden_neurons=n_hidden_neurons, n_categories=n_categories)
+dnn.train()
+test_predict = dnn.predict(X_test)
+
+# accuracy score from scikit library
+print("Accuracy score on test set: ", accuracy_score(Y_test, test_predict))
+
+# equivalent in numpy
+def accuracy_score_numpy(Y_test, Y_pred):
+    return np.sum(Y_test == Y_pred) / len(Y_test)
+
+#print("Accuracy score on test set: ", accuracy_score_numpy(Y_test, test_predict))
+
    +
    +
    +
    +
    +
    +

    Adjust hyperparameters#

    +

    We now perform a grid search to find the optimal hyperparameters for the network.
    +Note that we are only using 1 layer with 50 neurons, and human performance is estimated to be around \(98\%\) (\(2\%\) error rate).

    +
    +
    +
    eta_vals = np.logspace(-5, 1, 7)
+lmbd_vals = np.logspace(-5, 1, 7)
+# store the models for later use
+DNN_numpy = np.zeros((len(eta_vals), len(lmbd_vals)), dtype=object)
+
+# grid search
+for i, eta in enumerate(eta_vals):
+    for j, lmbd in enumerate(lmbd_vals):
+        dnn = NeuralNetwork(X_train, Y_train_onehot, eta=eta, lmbd=lmbd, epochs=epochs, batch_size=batch_size,
+                            n_hidden_neurons=n_hidden_neurons, n_categories=n_categories)
+        dnn.train()
+        
+        DNN_numpy[i][j] = dnn
+        
+        test_predict = dnn.predict(X_test)
+        
+        print("Learning rate  = ", eta)
+        print("Lambda = ", lmbd)
+        print("Accuracy score on test set: ", accuracy_score(Y_test, test_predict))
+        print()
+
    +
    +
    +
    +
    +
    +

    Visualization#

    +
    +
    +
    # visual representation of grid search
+# uses seaborn heatmap, you can also do this with matplotlib imshow
+import seaborn as sns
+
+sns.set()
+
+train_accuracy = np.zeros((len(eta_vals), len(lmbd_vals)))
+test_accuracy = np.zeros((len(eta_vals), len(lmbd_vals)))
+
+for i in range(len(eta_vals)):
+    for j in range(len(lmbd_vals)):
+        dnn = DNN_numpy[i][j]
+        
+        train_pred = dnn.predict(X_train) 
+        test_pred = dnn.predict(X_test)
+
+        train_accuracy[i][j] = accuracy_score(Y_train, train_pred)
+        test_accuracy[i][j] = accuracy_score(Y_test, test_pred)
+
+        
+fig, ax = plt.subplots(figsize = (10, 10))
+sns.heatmap(train_accuracy, annot=True, ax=ax, cmap="viridis")
+ax.set_title("Training Accuracy")
+ax.set_ylabel("$\eta$")
+ax.set_xlabel("$\lambda$")
+plt.show()
+
+fig, ax = plt.subplots(figsize = (10, 10))
+sns.heatmap(test_accuracy, annot=True, ax=ax, cmap="viridis")
+ax.set_title("Test Accuracy")
+ax.set_ylabel("$\eta$")
+ax.set_xlabel("$\lambda$")
+plt.show()
+
    +
    +
    +
    +
    +
    +

    scikit-learn implementation#

    +

    scikit-learn focuses more +on traditional machine learning methods, such as regression, +clustering, decision trees, etc. As such, it has only two types of +neural networks: Multi Layer Perceptron outputting continuous values, +MPLRegressor, and Multi Layer Perceptron outputting labels, +MLPClassifier. We will see how simple it is to use these classes.

    +

    scikit-learn implements a few improvements from our neural network, +such as early stopping, a varying learning rate, different +optimization methods, etc. We would therefore expect a better +performance overall.

    +
    +
    +
    from sklearn.neural_network import MLPClassifier
+# store models for later use
+DNN_scikit = np.zeros((len(eta_vals), len(lmbd_vals)), dtype=object)
+
+for i, eta in enumerate(eta_vals):
+    for j, lmbd in enumerate(lmbd_vals):
+        dnn = MLPClassifier(hidden_layer_sizes=(n_hidden_neurons), activation='logistic',
+                            alpha=lmbd, learning_rate_init=eta, max_iter=epochs)
+        dnn.fit(X_train, Y_train)
+        
+        DNN_scikit[i][j] = dnn
+        
+        print("Learning rate  = ", eta)
+        print("Lambda = ", lmbd)
+        print("Accuracy score on test set: ", dnn.score(X_test, Y_test))
+        print()
+
    +
    +
    +
    +
    +
    +

    Visualization#

    +
    +
    +
    # optional
+# visual representation of grid search
+# uses seaborn heatmap, could probably do this in matplotlib
+import seaborn as sns
+
+sns.set()
+
+train_accuracy = np.zeros((len(eta_vals), len(lmbd_vals)))
+test_accuracy = np.zeros((len(eta_vals), len(lmbd_vals)))
+
+for i in range(len(eta_vals)):
+    for j in range(len(lmbd_vals)):
+        dnn = DNN_scikit[i][j]
+        
+        train_pred = dnn.predict(X_train) 
+        test_pred = dnn.predict(X_test)
+
+        train_accuracy[i][j] = accuracy_score(Y_train, train_pred)
+        test_accuracy[i][j] = accuracy_score(Y_test, test_pred)
+
+        
+fig, ax = plt.subplots(figsize = (10, 10))
+sns.heatmap(train_accuracy, annot=True, ax=ax, cmap="viridis")
+ax.set_title("Training Accuracy")
+ax.set_ylabel("$\eta$")
+ax.set_xlabel("$\lambda$")
+plt.show()
+
+fig, ax = plt.subplots(figsize = (10, 10))
+sns.heatmap(test_accuracy, annot=True, ax=ax, cmap="viridis")
+ax.set_title("Test Accuracy")
+ax.set_ylabel("$\eta$")
+ax.set_xlabel("$\lambda$")
+plt.show()
+
    +
    +
    +
    +
    +
    +

    Building neural networks in Tensorflow and Keras#

    +

    Now we want to build on the experience gained from our neural network implementation in NumPy and scikit-learn +and use it to construct a neural network in Tensorflow. Once we have constructed a neural network in NumPy +and Tensorflow, building one in Keras is really quite trivial, though the performance may suffer.

    +

    In our previous example we used only one hidden layer, and in this we will use two. From this it should be quite +clear how to build one using an arbitrary number of hidden layers, using data structures such as Python lists or +NumPy arrays.

    +
    +
    +

    Tensorflow#

    +

    Tensorflow is an open source library machine learning library +developed by the Google Brain team for internal use. It was released +under the Apache 2.0 open source license in November 9, 2015.

    +

    Tensorflow is a computational framework that allows you to construct +machine learning models at different levels of abstraction, from +high-level, object-oriented APIs like Keras, down to the C++ kernels +that Tensorflow is built upon. The higher levels of abstraction are +simpler to use, but less flexible, and our choice of implementation +should reflect the problems we are trying to solve.

    +

    Tensorflow uses so-called graphs to represent your computation +in terms of the dependencies between individual operations, such that you first build a Tensorflow graph +to represent your model, and then create a Tensorflow session to run the graph.

    +

    In this guide we will analyze the same data as we did in our NumPy and +scikit-learn tutorial, gathered from the MNIST database of images. We +will give an introduction to the lower level Python Application +Program Interfaces (APIs), and see how we use them to build our graph. +Then we will build (effectively) the same graph in Keras, to see just +how simple solving a machine learning problem can be.

    +

    To install tensorflow on Unix/Linux systems, use pip as

    +
    +
    +
    pip3 install tensorflow
+
    +
    +
    +
    +

    and/or if you use anaconda, just write (or install from the graphical user interface) +(current release of CPU-only TensorFlow)

    +
    +
    +
    conda create -n tf tensorflow
+conda activate tf
+
    +
    +
    +
    +

    To install the current release of GPU TensorFlow

    +
    +
    +
    conda create -n tf-gpu tensorflow-gpu
+conda activate tf-gpu
+
    +
    +
    +
    +
    +
    +

    Using Keras#

    +

    Keras is a high level neural network +that supports Tensorflow, CTNK and Theano as backends.
    +If you have Anaconda installed you may run the following command

    +
    +
    +
    conda install keras
+
    +
    +
    +
    +

    You can look up the instructions here for more information.

    +

    We will to a large extent use keras in this course.

    +
    +
    +

    Collect and pre-process data#

    +

    Let us look again at the MINST data set.

    +
    +
    +
    # import necessary packages
+import numpy as np
+import matplotlib.pyplot as plt
+import tensorflow as tf
+from sklearn import datasets
+
+
+# ensure the same random numbers appear every time
+np.random.seed(0)
+
+# display images in notebook
+%matplotlib inline
+plt.rcParams['figure.figsize'] = (12,12)
+
+
+# download MNIST dataset
+digits = datasets.load_digits()
+
+# define inputs and labels
+inputs = digits.images
+labels = digits.target
+
+print("inputs = (n_inputs, pixel_width, pixel_height) = " + str(inputs.shape))
+print("labels = (n_inputs) = " + str(labels.shape))
+
+
+# flatten the image
+# the value -1 means dimension is inferred from the remaining dimensions: 8x8 = 64
+n_inputs = len(inputs)
+inputs = inputs.reshape(n_inputs, -1)
+print("X = (n_inputs, n_features) = " + str(inputs.shape))
+
+
+# choose some random images to display
+indices = np.arange(n_inputs)
+random_indices = np.random.choice(indices, size=5)
+
+for i, image in enumerate(digits.images[random_indices]):
+    plt.subplot(1, 5, i+1)
+    plt.axis('off')
+    plt.imshow(image, cmap=plt.cm.gray_r, interpolation='nearest')
+    plt.title("Label: %d" % digits.target[random_indices[i]])
+plt.show()
+
    +
    +
    +
    +
    +
    +
    from tensorflow.keras.layers import Input
+from tensorflow.keras.models import Sequential      #This allows appending layers to existing models
+from tensorflow.keras.layers import Dense           #This allows defining the characteristics of a particular layer
+from tensorflow.keras import optimizers             #This allows using whichever optimiser we want (sgd,adam,RMSprop)
+from tensorflow.keras import regularizers           #This allows using whichever regularizer we want (l1,l2,l1_l2)
+from tensorflow.keras.utils import to_categorical   #This allows using categorical cross entropy as the cost function
+
+from sklearn.model_selection import train_test_split
+
+# one-hot representation of labels
+labels = to_categorical(labels)
+
+# split into train and test data
+train_size = 0.8
+test_size = 1 - train_size
+X_train, X_test, Y_train, Y_test = train_test_split(inputs, labels, train_size=train_size,
+                                                    test_size=test_size)
+
    +
    +
    +
    +
    +
    +
    
+epochs = 100
+batch_size = 100
+n_neurons_layer1 = 100
+n_neurons_layer2 = 50
+n_categories = 10
+eta_vals = np.logspace(-5, 1, 7)
+lmbd_vals = np.logspace(-5, 1, 7)
+def create_neural_network_keras(n_neurons_layer1, n_neurons_layer2, n_categories, eta, lmbd):
+    model = Sequential()
+    model.add(Dense(n_neurons_layer1, activation='sigmoid', kernel_regularizer=regularizers.l2(lmbd)))
+    model.add(Dense(n_neurons_layer2, activation='sigmoid', kernel_regularizer=regularizers.l2(lmbd)))
+    model.add(Dense(n_categories, activation='softmax'))
+    
+    sgd = optimizers.SGD(lr=eta)
+    model.compile(loss='categorical_crossentropy', optimizer=sgd, metrics=['accuracy'])
+    
+    return model
+
    +
    +
    +
    +
    +
    +
    DNN_keras = np.zeros((len(eta_vals), len(lmbd_vals)), dtype=object)
+        
+for i, eta in enumerate(eta_vals):
+    for j, lmbd in enumerate(lmbd_vals):
+        DNN = create_neural_network_keras(n_neurons_layer1, n_neurons_layer2, n_categories,
+                                         eta=eta, lmbd=lmbd)
+        DNN.fit(X_train, Y_train, epochs=epochs, batch_size=batch_size, verbose=0)
+        scores = DNN.evaluate(X_test, Y_test)
+        
+        DNN_keras[i][j] = DNN
+        
+        print("Learning rate = ", eta)
+        print("Lambda = ", lmbd)
+        print("Test accuracy: %.3f" % scores[1])
+        print()
+
    +
    +
    +
    +
    +
    +
    # optional
+# visual representation of grid search
+# uses seaborn heatmap, could probably do this in matplotlib
+import seaborn as sns
+
+sns.set()
+
+train_accuracy = np.zeros((len(eta_vals), len(lmbd_vals)))
+test_accuracy = np.zeros((len(eta_vals), len(lmbd_vals)))
+
+for i in range(len(eta_vals)):
+    for j in range(len(lmbd_vals)):
+        DNN = DNN_keras[i][j]
+
+        train_accuracy[i][j] = DNN.evaluate(X_train, Y_train)[1]
+        test_accuracy[i][j] = DNN.evaluate(X_test, Y_test)[1]
+
+        
+fig, ax = plt.subplots(figsize = (10, 10))
+sns.heatmap(train_accuracy, annot=True, ax=ax, cmap="viridis")
+ax.set_title("Training Accuracy")
+ax.set_ylabel("$\eta$")
+ax.set_xlabel("$\lambda$")
+plt.show()
+
+fig, ax = plt.subplots(figsize = (10, 10))
+sns.heatmap(test_accuracy, annot=True, ax=ax, cmap="viridis")
+ax.set_title("Test Accuracy")
+ax.set_ylabel("$\eta$")
+ax.set_xlabel("$\lambda$")
+plt.show()
+
    +
    +
    +
    +
    +
    +

    Building a neural network code#

    +

    Here we present a flexible object oriented codebase +for a feed forward neural network, along with a demonstration of how +to use it. Before we get into the details of the neural network, we +will first present some implementations of various schedulers, cost +functions and activation functions that can be used together with the +neural network.

    +

    The codes here were developed by Eric Reber and Gregor Kajda during spring 2023.

    +
    +

    Learning rate methods#

    +

    The code below shows object oriented implementations of the Constant, +Momentum, Adagrad, AdagradMomentum, RMS prop and Adam schedulers. All +of the classes belong to the shared abstract Scheduler class, and +share the update_change() and reset() methods allowing for any of the +schedulers to be seamlessly used during the training stage, as will +later be shown in the fit() method of the neural +network. Update_change() only has one parameter, the gradient +(\(δ^l_ja^{l−1}_k\)), and returns the change which will be subtracted +from the weights. The reset() function takes no parameters, and resets +the desired variables. For Constant and Momentum, reset does nothing.

    +
    +
    +
    import autograd.numpy as np
+
+class Scheduler:
+    """
+    Abstract class for Schedulers
+    """
+
+    def __init__(self, eta):
+        self.eta = eta
+
+    # should be overwritten
+    def update_change(self, gradient):
+        raise NotImplementedError
+
+    # overwritten if needed
+    def reset(self):
+        pass
+
+
+class Constant(Scheduler):
+    def __init__(self, eta):
+        super().__init__(eta)
+
+    def update_change(self, gradient):
+        return self.eta * gradient
+    
+    def reset(self):
+        pass
+
+
+class Momentum(Scheduler):
+    def __init__(self, eta: float, momentum: float):
+        super().__init__(eta)
+        self.momentum = momentum
+        self.change = 0
+
+    def update_change(self, gradient):
+        self.change = self.momentum * self.change + self.eta * gradient
+        return self.change
+
+    def reset(self):
+        pass
+
+
+class Adagrad(Scheduler):
+    def __init__(self, eta):
+        super().__init__(eta)
+        self.G_t = None
+
+    def update_change(self, gradient):
+        delta = 1e-8  # avoid division ny zero
+
+        if self.G_t is None:
+            self.G_t = np.zeros((gradient.shape[0], gradient.shape[0]))
+
+        self.G_t += gradient @ gradient.T
+
+        G_t_inverse = 1 / (
+            delta + np.sqrt(np.reshape(np.diagonal(self.G_t), (self.G_t.shape[0], 1)))
+        )
+        return self.eta * gradient * G_t_inverse
+
+    def reset(self):
+        self.G_t = None
+
+
+class AdagradMomentum(Scheduler):
+    def __init__(self, eta, momentum):
+        super().__init__(eta)
+        self.G_t = None
+        self.momentum = momentum
+        self.change = 0
+
+    def update_change(self, gradient):
+        delta = 1e-8  # avoid division ny zero
+
+        if self.G_t is None:
+            self.G_t = np.zeros((gradient.shape[0], gradient.shape[0]))
+
+        self.G_t += gradient @ gradient.T
+
+        G_t_inverse = 1 / (
+            delta + np.sqrt(np.reshape(np.diagonal(self.G_t), (self.G_t.shape[0], 1)))
+        )
+        self.change = self.change * self.momentum + self.eta * gradient * G_t_inverse
+        return self.change
+
+    def reset(self):
+        self.G_t = None
+
+
+class RMS_prop(Scheduler):
+    def __init__(self, eta, rho):
+        super().__init__(eta)
+        self.rho = rho
+        self.second = 0.0
+
+    def update_change(self, gradient):
+        delta = 1e-8  # avoid division ny zero
+        self.second = self.rho * self.second + (1 - self.rho) * gradient * gradient
+        return self.eta * gradient / (np.sqrt(self.second + delta))
+
+    def reset(self):
+        self.second = 0.0
+
+
+class Adam(Scheduler):
+    def __init__(self, eta, rho, rho2):
+        super().__init__(eta)
+        self.rho = rho
+        self.rho2 = rho2
+        self.moment = 0
+        self.second = 0
+        self.n_epochs = 1
+
+    def update_change(self, gradient):
+        delta = 1e-8  # avoid division ny zero
+
+        self.moment = self.rho * self.moment + (1 - self.rho) * gradient
+        self.second = self.rho2 * self.second + (1 - self.rho2) * gradient * gradient
+
+        moment_corrected = self.moment / (1 - self.rho**self.n_epochs)
+        second_corrected = self.second / (1 - self.rho2**self.n_epochs)
+
+        return self.eta * moment_corrected / (np.sqrt(second_corrected + delta))
+
+    def reset(self):
+        self.n_epochs += 1
+        self.moment = 0
+        self.second = 0
+
    +
    +
    +
    +
    +
    +

    Usage of the above learning rate schedulers#

    +

    To initalize a scheduler, simply create the object and pass in the +necessary parameters such as the learning rate and the momentum as +shown below. As the Scheduler class is an abstract class it should not +called directly, and will raise an error upon usage.

    +
    +
    +
    momentum_scheduler = Momentum(eta=1e-3, momentum=0.9)
+adam_scheduler = Adam(eta=1e-3, rho=0.9, rho2=0.999)
+
    +
    +
    +
    +

    Here is a small example for how a segment of code using schedulers +could look. Switching out the schedulers is simple.

    +
    +
    +
    weights = np.ones((3,3))
+print(f"Before scheduler:\n{weights=}")
+
+epochs = 10
+for e in range(epochs):
+    gradient = np.random.rand(3, 3)
+    change = adam_scheduler.update_change(gradient)
+    weights = weights - change
+    adam_scheduler.reset()
+
+print(f"\nAfter scheduler:\n{weights=}")
+
    +
    +
    +
    +
    +
    +

    Cost functions#

    +

    Here we discuss cost functions that can be used when creating the +neural network. Every cost function takes the target vector as its +parameter, and returns a function valued only at \(x\) such that it may +easily be differentiated.

    +
    +
    +
    import autograd.numpy as np
+
+def CostOLS(target):
+    
+    def func(X):
+        return (1.0 / target.shape[0]) * np.sum((target - X) ** 2)
+
+    return func
+
+
+def CostLogReg(target):
+
+    def func(X):
+        
+        return -(1.0 / target.shape[0]) * np.sum(
+            (target * np.log(X + 10e-10)) + ((1 - target) * np.log(1 - X + 10e-10))
+        )
+
+    return func
+
+
+def CostCrossEntropy(target):
+    
+    def func(X):
+        return -(1.0 / target.size) * np.sum(target * np.log(X + 10e-10))
+
+    return func
+
    +
    +
    +
    +

    Below we give a short example of how these cost function may be used +to obtain results if you wish to test them out on your own using +AutoGrad’s automatics differentiation.

    +
    +
    +
    from autograd import grad
+
+target = np.array([[1, 2, 3]]).T
+a = np.array([[4, 5, 6]]).T
+
+cost_func = CostCrossEntropy
+cost_func_derivative = grad(cost_func(target))
+
+valued_at_a = cost_func_derivative(a)
+print(f"Derivative of cost function {cost_func.__name__} valued at a:\n{valued_at_a}")
+
    +
    +
    +
    +
    +
    +

    Activation functions#

    +

    Finally, before we look at the neural network, we will look at the +activation functions which can be specified between the hidden layers +and as the output function. Each function can be valued for any given +vector or matrix X, and can be differentiated via derivate().

    +
    +
    +
    import autograd.numpy as np
+from autograd import elementwise_grad
+
+def identity(X):
+    return X
+
+
+def sigmoid(X):
+    try:
+        return 1.0 / (1 + np.exp(-X))
+    except FloatingPointError:
+        return np.where(X > np.zeros(X.shape), np.ones(X.shape), np.zeros(X.shape))
+
+
+def softmax(X):
+    X = X - np.max(X, axis=-1, keepdims=True)
+    delta = 10e-10
+    return np.exp(X) / (np.sum(np.exp(X), axis=-1, keepdims=True) + delta)
+
+
+def RELU(X):
+    return np.where(X > np.zeros(X.shape), X, np.zeros(X.shape))
+
+
+def LRELU(X):
+    delta = 10e-4
+    return np.where(X > np.zeros(X.shape), X, delta * X)
+
+
+def derivate(func):
+    if func.__name__ == "RELU":
+
+        def func(X):
+            return np.where(X > 0, 1, 0)
+
+        return func
+
+    elif func.__name__ == "LRELU":
+
+        def func(X):
+            delta = 10e-4
+            return np.where(X > 0, 1, delta)
+
+        return func
+
+    else:
+        return elementwise_grad(func)
+
    +
    +
    +
    +

    Below follows a short demonstration of how to use an activation +function. The derivative of the activation function will be important +when calculating the output delta term during backpropagation. Note +that derivate() can also be used for cost functions for a more +generalized approach.

    +
    +
    +
    z = np.array([[4, 5, 6]]).T
+print(f"Input to activation function:\n{z}")
+
+act_func = sigmoid
+a = act_func(z)
+print(f"\nOutput from {act_func.__name__} activation function:\n{a}")
+
+act_func_derivative = derivate(act_func)
+valued_at_z = act_func_derivative(a)
+print(f"\nDerivative of {act_func.__name__} activation function valued at z:\n{valued_at_z}")
+
    +
    +
    +
    +
    +
    +

    The Neural Network#

    +

    Now that we have gotten a good understanding of the implementation of +some important components, we can take a look at an object oriented +implementation of a feed forward neural network. The feed forward +neural network has been implemented as a class named FFNN, which can +be initiated as a regressor or classifier dependant on the choice of +cost function. The FFNN can have any number of input nodes, hidden +layers with any amount of hidden nodes, and any amount of output nodes +meaning it can perform multiclass classification as well as binary +classification and regression problems. Although there is a lot of +code present, it makes for an easy to use and generalizeable interface +for creating many types of neural networks as will be demonstrated +below.

    +
    +
    +
    import math
+import autograd.numpy as np
+import sys
+import warnings
+from autograd import grad, elementwise_grad
+from random import random, seed
+from copy import deepcopy, copy
+from typing import Tuple, Callable
+from sklearn.utils import resample
+
+warnings.simplefilter("error")
+
+
+class FFNN:
+    """
+    Description:
+    ------------
+        Feed Forward Neural Network with interface enabling flexible design of a
+        nerual networks architecture and the specification of activation function
+        in the hidden layers and output layer respectively. This model can be used
+        for both regression and classification problems, depending on the output function.
+
+    Attributes:
+    ------------
+        I   dimensions (tuple[int]): A list of positive integers, which specifies the
+            number of nodes in each of the networks layers. The first integer in the array
+            defines the number of nodes in the input layer, the second integer defines number
+            of nodes in the first hidden layer and so on until the last number, which
+            specifies the number of nodes in the output layer.
+        II  hidden_func (Callable): The activation function for the hidden layers
+        III output_func (Callable): The activation function for the output layer
+        IV  cost_func (Callable): Our cost function
+        V   seed (int): Sets random seed, makes results reproducible
+    """
+
+    def __init__(
+        self,
+        dimensions: tuple[int],
+        hidden_func: Callable = sigmoid,
+        output_func: Callable = lambda x: x,
+        cost_func: Callable = CostOLS,
+        seed: int = None,
+    ):
+        self.dimensions = dimensions
+        self.hidden_func = hidden_func
+        self.output_func = output_func
+        self.cost_func = cost_func
+        self.seed = seed
+        self.weights = list()
+        self.schedulers_weight = list()
+        self.schedulers_bias = list()
+        self.a_matrices = list()
+        self.z_matrices = list()
+        self.classification = None
+
+        self.reset_weights()
+        self._set_classification()
+
+    def fit(
+        self,
+        X: np.ndarray,
+        t: np.ndarray,
+        scheduler: Scheduler,
+        batches: int = 1,
+        epochs: int = 100,
+        lam: float = 0,
+        X_val: np.ndarray = None,
+        t_val: np.ndarray = None,
+    ):
+        """
+        Description:
+        ------------
+            This function performs the training the neural network by performing the feedforward and backpropagation
+            algorithm to update the networks weights.
+
+        Parameters:
+        ------------
+            I    X (np.ndarray) : training data
+            II   t (np.ndarray) : target data
+            III  scheduler (Scheduler) : specified scheduler (algorithm for optimization of gradient descent)
+            IV   scheduler_args (list[int]) : list of all arguments necessary for scheduler
+
+        Optional Parameters:
+        ------------
+            V    batches (int) : number of batches the datasets are split into, default equal to 1
+            VI   epochs (int) : number of iterations used to train the network, default equal to 100
+            VII  lam (float) : regularization hyperparameter lambda
+            VIII X_val (np.ndarray) : validation set
+            IX   t_val (np.ndarray) : validation target set
+
+        Returns:
+        ------------
+            I   scores (dict) : A dictionary containing the performance metrics of the model.
+                The number of the metrics depends on the parameters passed to the fit-function.
+
+        """
+
+        # setup 
+        if self.seed is not None:
+            np.random.seed(self.seed)
+
+        val_set = False
+        if X_val is not None and t_val is not None:
+            val_set = True
+
+        # creating arrays for score metrics
+        train_errors = np.empty(epochs)
+        train_errors.fill(np.nan)
+        val_errors = np.empty(epochs)
+        val_errors.fill(np.nan)
+
+        train_accs = np.empty(epochs)
+        train_accs.fill(np.nan)
+        val_accs = np.empty(epochs)
+        val_accs.fill(np.nan)
+
+        self.schedulers_weight = list()
+        self.schedulers_bias = list()
+
+        batch_size = X.shape[0] // batches
+
+        X, t = resample(X, t)
+
+        # this function returns a function valued only at X
+        cost_function_train = self.cost_func(t)
+        if val_set:
+            cost_function_val = self.cost_func(t_val)
+
+        # create schedulers for each weight matrix
+        for i in range(len(self.weights)):
+            self.schedulers_weight.append(copy(scheduler))
+            self.schedulers_bias.append(copy(scheduler))
+
+        print(f"{scheduler.__class__.__name__}: Eta={scheduler.eta}, Lambda={lam}")
+
+        try:
+            for e in range(epochs):
+                for i in range(batches):
+                    # allows for minibatch gradient descent
+                    if i == batches - 1:
+                        # If the for loop has reached the last batch, take all thats left
+                        X_batch = X[i * batch_size :, :]
+                        t_batch = t[i * batch_size :, :]
+                    else:
+                        X_batch = X[i * batch_size : (i + 1) * batch_size, :]
+                        t_batch = t[i * batch_size : (i + 1) * batch_size, :]
+
+                    self._feedforward(X_batch)
+                    self._backpropagate(X_batch, t_batch, lam)
+
+                # reset schedulers for each epoch (some schedulers pass in this call)
+                for scheduler in self.schedulers_weight:
+                    scheduler.reset()
+
+                for scheduler in self.schedulers_bias:
+                    scheduler.reset()
+
+                # computing performance metrics
+                pred_train = self.predict(X)
+                train_error = cost_function_train(pred_train)
+
+                train_errors[e] = train_error
+                if val_set:
+                    
+                    pred_val = self.predict(X_val)
+                    val_error = cost_function_val(pred_val)
+                    val_errors[e] = val_error
+
+                if self.classification:
+                    train_acc = self._accuracy(self.predict(X), t)
+                    train_accs[e] = train_acc
+                    if val_set:
+                        val_acc = self._accuracy(pred_val, t_val)
+                        val_accs[e] = val_acc
+
+                # printing progress bar
+                progression = e / epochs
+                print_length = self._progress_bar(
+                    progression,
+                    train_error=train_errors[e],
+                    train_acc=train_accs[e],
+                    val_error=val_errors[e],
+                    val_acc=val_accs[e],
+                )
+        except KeyboardInterrupt:
+            # allows for stopping training at any point and seeing the result
+            pass
+
+        # visualization of training progression (similiar to tensorflow progression bar)
+        sys.stdout.write("\r" + " " * print_length)
+        sys.stdout.flush()
+        self._progress_bar(
+            1,
+            train_error=train_errors[e],
+            train_acc=train_accs[e],
+            val_error=val_errors[e],
+            val_acc=val_accs[e],
+        )
+        sys.stdout.write("")
+
+        # return performance metrics for the entire run
+        scores = dict()
+
+        scores["train_errors"] = train_errors
+
+        if val_set:
+            scores["val_errors"] = val_errors
+
+        if self.classification:
+            scores["train_accs"] = train_accs
+
+            if val_set:
+                scores["val_accs"] = val_accs
+
+        return scores
+
+    def predict(self, X: np.ndarray, *, threshold=0.5):
+        """
+         Description:
+         ------------
+             Performs prediction after training of the network has been finished.
+
+         Parameters:
+        ------------
+             I   X (np.ndarray): The design matrix, with n rows of p features each
+
+         Optional Parameters:
+         ------------
+             II  threshold (float) : sets minimal value for a prediction to be predicted as the positive class
+                 in classification problems
+
+         Returns:
+         ------------
+             I   z (np.ndarray): A prediction vector (row) for each row in our design matrix
+                 This vector is thresholded if regression=False, meaning that classification results
+                 in a vector of 1s and 0s, while regressions in an array of decimal numbers
+
+        """
+
+        predict = self._feedforward(X)
+
+        if self.classification:
+            return np.where(predict > threshold, 1, 0)
+        else:
+            return predict
+
+    def reset_weights(self):
+        """
+        Description:
+        ------------
+            Resets/Reinitializes the weights in order to train the network for a new problem.
+
+        """
+        if self.seed is not None:
+            np.random.seed(self.seed)
+
+        self.weights = list()
+        for i in range(len(self.dimensions) - 1):
+            weight_array = np.random.randn(
+                self.dimensions[i] + 1, self.dimensions[i + 1]
+            )
+            weight_array[0, :] = np.random.randn(self.dimensions[i + 1]) * 0.01
+
+            self.weights.append(weight_array)
+
+    def _feedforward(self, X: np.ndarray):
+        """
+        Description:
+        ------------
+            Calculates the activation of each layer starting at the input and ending at the output.
+            Each following activation is calculated from a weighted sum of each of the preceeding
+            activations (except in the case of the input layer).
+
+        Parameters:
+        ------------
+            I   X (np.ndarray): The design matrix, with n rows of p features each
+
+        Returns:
+        ------------
+            I   z (np.ndarray): A prediction vector (row) for each row in our design matrix
+        """
+
+        # reset matrices
+        self.a_matrices = list()
+        self.z_matrices = list()
+
+        # if X is just a vector, make it into a matrix
+        if len(X.shape) == 1:
+            X = X.reshape((1, X.shape[0]))
+
+        # Add a coloumn of zeros as the first coloumn of the design matrix, in order
+        # to add bias to our data
+        bias = np.ones((X.shape[0], 1)) * 0.01
+        X = np.hstack([bias, X])
+
+        # a^0, the nodes in the input layer (one a^0 for each row in X - where the
+        # exponent indicates layer number).
+        a = X
+        self.a_matrices.append(a)
+        self.z_matrices.append(a)
+
+        # The feed forward algorithm
+        for i in range(len(self.weights)):
+            if i < len(self.weights) - 1:
+                z = a @ self.weights[i]
+                self.z_matrices.append(z)
+                a = self.hidden_func(z)
+                # bias column again added to the data here
+                bias = np.ones((a.shape[0], 1)) * 0.01
+                a = np.hstack([bias, a])
+                self.a_matrices.append(a)
+            else:
+                try:
+                    # a^L, the nodes in our output layers
+                    z = a @ self.weights[i]
+                    a = self.output_func(z)
+                    self.a_matrices.append(a)
+                    self.z_matrices.append(z)
+                except Exception as OverflowError:
+                    print(
+                        "OverflowError in fit() in FFNN\nHOW TO DEBUG ERROR: Consider lowering your learning rate or scheduler specific parameters such as momentum, or check if your input values need scaling"
+                    )
+
+        # this will be a^L
+        return a
+
+    def _backpropagate(self, X, t, lam):
+        """
+        Description:
+        ------------
+            Performs the backpropagation algorithm. In other words, this method
+            calculates the gradient of all the layers starting at the
+            output layer, and moving from right to left accumulates the gradient until
+            the input layer is reached. Each layers respective weights are updated while
+            the algorithm propagates backwards from the output layer (auto-differentation in reverse mode).
+
+        Parameters:
+        ------------
+            I   X (np.ndarray): The design matrix, with n rows of p features each.
+            II  t (np.ndarray): The target vector, with n rows of p targets.
+            III lam (float32): regularization parameter used to punish the weights in case of overfitting
+
+        Returns:
+        ------------
+            No return value.
+
+        """
+        out_derivative = derivate(self.output_func)
+        hidden_derivative = derivate(self.hidden_func)
+
+        for i in range(len(self.weights) - 1, -1, -1):
+            # delta terms for output
+            if i == len(self.weights) - 1:
+                # for multi-class classification
+                if (
+                    self.output_func.__name__ == "softmax"
+                ):
+                    delta_matrix = self.a_matrices[i + 1] - t
+                # for single class classification
+                else:
+                    cost_func_derivative = grad(self.cost_func(t))
+                    delta_matrix = out_derivative(
+                        self.z_matrices[i + 1]
+                    ) * cost_func_derivative(self.a_matrices[i + 1])
+
+            # delta terms for hidden layer
+            else:
+                delta_matrix = (
+                    self.weights[i + 1][1:, :] @ delta_matrix.T
+                ).T * hidden_derivative(self.z_matrices[i + 1])
+
+            # calculate gradient
+            gradient_weights = self.a_matrices[i][:, 1:].T @ delta_matrix
+            gradient_bias = np.sum(delta_matrix, axis=0).reshape(
+                1, delta_matrix.shape[1]
+            )
+
+            # regularization term
+            gradient_weights += self.weights[i][1:, :] * lam
+
+            # use scheduler
+            update_matrix = np.vstack(
+                [
+                    self.schedulers_bias[i].update_change(gradient_bias),
+                    self.schedulers_weight[i].update_change(gradient_weights),
+                ]
+            )
+
+            # update weights and bias
+            self.weights[i] -= update_matrix
+
+    def _accuracy(self, prediction: np.ndarray, target: np.ndarray):
+        """
+        Description:
+        ------------
+            Calculates accuracy of given prediction to target
+
+        Parameters:
+        ------------
+            I   prediction (np.ndarray): vector of predicitons output network
+                (1s and 0s in case of classification, and real numbers in case of regression)
+            II  target (np.ndarray): vector of true values (What the network ideally should predict)
+
+        Returns:
+        ------------
+            A floating point number representing the percentage of correctly classified instances.
+        """
+        assert prediction.size == target.size
+        return np.average((target == prediction))
+    def _set_classification(self):
+        """
+        Description:
+        ------------
+            Decides if FFNN acts as classifier (True) og regressor (False),
+            sets self.classification during init()
+        """
+        self.classification = False
+        if (
+            self.cost_func.__name__ == "CostLogReg"
+            or self.cost_func.__name__ == "CostCrossEntropy"
+        ):
+            self.classification = True
+
+    def _progress_bar(self, progression, **kwargs):
+        """
+        Description:
+        ------------
+            Displays progress of training
+        """
+        print_length = 40
+        num_equals = int(progression * print_length)
+        num_not = print_length - num_equals
+        arrow = ">" if num_equals > 0 else ""
+        bar = "[" + "=" * (num_equals - 1) + arrow + "-" * num_not + "]"
+        perc_print = self._format(progression * 100, decimals=5)
+        line = f"  {bar} {perc_print}% "
+
+        for key in kwargs:
+            if not np.isnan(kwargs[key]):
+                value = self._format(kwargs[key], decimals=4)
+                line += f"| {key}: {value} "
+        sys.stdout.write("\r" + line)
+        sys.stdout.flush()
+        return len(line)
+
+    def _format(self, value, decimals=4):
+        """
+        Description:
+        ------------
+            Formats decimal numbers for progress bar
+        """
+        if value > 0:
+            v = value
+        elif value < 0:
+            v = -10 * value
+        else:
+            v = 1
+        n = 1 + math.floor(math.log10(v))
+        if n >= decimals - 1:
+            return str(round(value))
+        return f"{value:.{decimals-n-1}f}"
+
    +
    +
    +
    +

    Before we make a model, we will quickly generate a dataset we can use +for our linear regression problem as shown below

    +
    +
    +
    import autograd.numpy as np
+from sklearn.model_selection import train_test_split
+
+def SkrankeFunction(x, y):
+    return np.ravel(0 + 1*x + 2*y + 3*x**2 + 4*x*y + 5*y**2)
+
+def create_X(x, y, n):
+    if len(x.shape) > 1:
+        x = np.ravel(x)
+        y = np.ravel(y)
+
+    N = len(x)
+    l = int((n + 1) * (n + 2) / 2)  # Number of elements in beta
+    X = np.ones((N, l))
+
+    for i in range(1, n + 1):
+        q = int((i) * (i + 1) / 2)
+        for k in range(i + 1):
+            X[:, q + k] = (x ** (i - k)) * (y**k)
+
+    return X
+
+step=0.5
+x = np.arange(0, 1, step)
+y = np.arange(0, 1, step)
+x, y = np.meshgrid(x, y)
+target = SkrankeFunction(x, y)
+target = target.reshape(target.shape[0], 1)
+
+poly_degree=3
+X = create_X(x, y, poly_degree)
+
+X_train, X_test, t_train, t_test = train_test_split(X, target)
+
    +
    +
    +
    +

    Now that we have our dataset ready for the regression, we can create +our regressor. Note that with the seed parameter, we can make sure our +results stay the same every time we run the neural network. For +inititialization, we simply specify the dimensions (we wish the amount +of input nodes to be equal to the datapoints, and the output to +predict one value).

    +
    +
    +
    input_nodes = X_train.shape[1]
+output_nodes = 1
+
+linear_regression = FFNN((input_nodes, output_nodes), output_func=identity, cost_func=CostOLS, seed=2023)
+
    +
    +
    +
    +

    We then fit our model with our training data using the scheduler of our choice.

    +
    +
    +
    linear_regression.reset_weights() # reset weights such that previous runs or reruns don't affect the weights
+
+scheduler = Constant(eta=1e-3)
+scores = linear_regression.fit(X_train, t_train, scheduler)
+
    +
    +
    +
    +

    Due to the progress bar we can see the MSE (train_error) throughout +the FFNN’s training. Note that the fit() function has some optional +parameters with defualt arguments. For example, the regularization +hyperparameter can be left ignored if not needed, and equally the FFNN +will by default run for 100 epochs. These can easily be changed, such +as for example:

    +
    +
    +
    linear_regression.reset_weights() # reset weights such that previous runs or reruns don't affect the weights
+
+scores = linear_regression.fit(X_train, t_train, scheduler, lam=1e-4, epochs=1000)
+
    +
    +
    +
    +

    We see that given more epochs to train on, the regressor reaches a lower MSE.

    +

    Let us then switch to a binary classification. We use a binary +classification dataset, and follow a similar setup to the regression +case.

    +
    +
    +
    from sklearn.datasets import load_breast_cancer
+from sklearn.preprocessing import MinMaxScaler
+
+wisconsin = load_breast_cancer()
+X = wisconsin.data
+target = wisconsin.target
+target = target.reshape(target.shape[0], 1)
+
+X_train, X_val, t_train, t_val = train_test_split(X, target)
+
+scaler = MinMaxScaler()
+scaler.fit(X_train)
+X_train = scaler.transform(X_train)
+X_val = scaler.transform(X_val)
+
    +
    +
    +
    +
    +
    +
    input_nodes = X_train.shape[1]
+output_nodes = 1
+
+logistic_regression = FFNN((input_nodes, output_nodes), output_func=sigmoid, cost_func=CostLogReg, seed=2023)
+
    +
    +
    +
    +

    We will now make use of our validation data by passing it into our fit function as a keyword argument

    +
    +
    +
    logistic_regression.reset_weights() # reset weights such that previous runs or reruns don't affect the weights
+
+scheduler = Adam(eta=1e-3, rho=0.9, rho2=0.999)
+scores = logistic_regression.fit(X_train, t_train, scheduler, epochs=1000, X_val=X_val, t_val=t_val)
+
    +
    +
    +
    +

    Finally, we will create a neural network with 2 hidden layers with activation functions.

    +
    +
    +
    input_nodes = X_train.shape[1]
+hidden_nodes1 = 100
+hidden_nodes2 = 30
+output_nodes = 1
+
+dims = (input_nodes, hidden_nodes1, hidden_nodes2, output_nodes)
+
+neural_network = FFNN(dims, hidden_func=RELU, output_func=sigmoid, cost_func=CostLogReg, seed=2023)
+
    +
    +
    +
    +
    +
    +
    neural_network.reset_weights() # reset weights such that previous runs or reruns don't affect the weights
+
+scheduler = Adam(eta=1e-4, rho=0.9, rho2=0.999)
+scores = neural_network.fit(X_train, t_train, scheduler, epochs=1000, X_val=X_val, t_val=t_val)
+
    +
    +
    +
    +
    +
    +

    Multiclass classification#

    +

    Finally, we will demonstrate the use case of multiclass classification +using our FFNN with the famous MNIST dataset, which contain images of +digits between the range of 0 to 9.

    +
    +
    +
    from sklearn.datasets import load_digits
+
+def onehot(target: np.ndarray):
+    onehot = np.zeros((target.size, target.max() + 1))
+    onehot[np.arange(target.size), target] = 1
+    return onehot
+
+digits = load_digits()
+
+X = digits.data
+target = digits.target
+target = onehot(target)
+
+input_nodes = 64
+hidden_nodes1 = 100
+hidden_nodes2 = 30
+output_nodes = 10
+
+dims = (input_nodes, hidden_nodes1, hidden_nodes2, output_nodes)
+
+multiclass = FFNN(dims, hidden_func=LRELU, output_func=softmax, cost_func=CostCrossEntropy)
+
+multiclass.reset_weights() # reset weights such that previous runs or reruns don't affect the weights
+
+scheduler = Adam(eta=1e-4, rho=0.9, rho2=0.999)
+scores = multiclass.fit(X, target, scheduler, epochs=1000)
+
    +
    +
    +
    +
    +
    +
    +

    Testing the XOR gate and other gates#

    +

    Let us now use our code to test the XOR gate.

    +
    +
    +
    X = np.array([ [0, 0], [0, 1], [1, 0],[1, 1]],dtype=np.float64)
+
+# The XOR gate
+yXOR = np.array( [[ 0], [1] ,[1], [0]])
+
+input_nodes = X.shape[1]
+output_nodes = 1
+
+logistic_regression = FFNN((input_nodes, output_nodes), output_func=sigmoid, cost_func=CostLogReg, seed=2023)
+logistic_regression.reset_weights() # reset weights such that previous runs or reruns don't affect the weights
+scheduler = Adam(eta=1e-1, rho=0.9, rho2=0.999)
+scores = logistic_regression.fit(X, yXOR, scheduler, epochs=1000)
+
    +
    +
    +
    +

    Not bad, but the results depend strongly on the learning reate. Try different learning rates.

    +
    +
    + + + + +
    + + + + + + + + +
    + + + +
    + + +
    +
    + Contents +
    +
    + +
    + + +
    +
    + +
    + +
    + +

    +By Morten Hjorth-Jensen +

    + +
    + +
    + + +

    + + © Copyright 2023. +
    + +

    + +
    + +
    + +
    + +
    + +
    + +
    +
    + + +
    +
    +
    + + + + + +
    +
    + + \ No newline at end of file diff --git a/doc/LectureNotes/_build/jupyter_execute/project2.ipynb b/doc/LectureNotes/_build/jupyter_execute/project2.ipynb new file mode 100644 index 000000000..50c44b123 --- /dev/null +++ b/doc/LectureNotes/_build/jupyter_execute/project2.ipynb @@ -0,0 +1,554 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "id": "061af572", + "metadata": { + "editable": true + }, + "source": [ + "\n", + "" + ] + }, + { + "cell_type": "markdown", + "id": "22578683", + "metadata": { + "editable": true + }, + "source": [ + "# Project 2 on Machine Learning, deadline November 10 (Midnight)\n", + "**[Data Analysis and Machine Learning FYS-STK3155/FYS4155](http://www.uio.no/studier/emner/matnat/fys/FYS3155/index-eng.html)**, University of Oslo, Norway\n", + "\n", + "Date: **October 14, 2025**" + ] + }, + { + "cell_type": "markdown", + "id": "61fb162f", + "metadata": { + "editable": true + }, + "source": [ + "## Deliverables\n", + "\n", + "First, join a group in canvas with your group partners. Pick an avaliable group for Project 2 in the **People** page.\n", + "\n", + "In canvas, deliver as a group and include:\n", + "\n", + "* A PDF of your report which follows the guidelines covered below and in the week 39 exercises. Additional requirements include:\n", + "\n", + " * It should be around 5000 words, use the word counter in Overleaf for this. This often corresponds to 10-12 pages. References and appendices are excluded from the word count\n", + "\n", + " * It should include around 10-15 figures. You can include more figures in appendices and/or as supplemental material in your repository.\n", + "\n", + "* A comment linking to your github repository (or folder in one of your github repositories) for this project. The repository must include\n", + "\n", + "A PDF file of the report\n", + " * A folder named Code, where you put python files for your functions and notebooks for reproducing your results. Remember to use a seed for generating random data and for train-test splits when generating final results.\n", + "\n", + " * A README file with the name of the group members\n", + "\n", + " * a short description of the project\n", + "\n", + " * a description of how to install the required packages to run your code from a requirements.txt file or similar (such as a plain text description) names and descriptions of the various notebooks in the Code folder and the results they produce" + ] + }, + { + "cell_type": "markdown", + "id": "104c69e1", + "metadata": { + "editable": true + }, + "source": [ + "### Preamble: Note on writing reports, using reference material, AI and other tools\n", + "\n", + "We want you to answer the three different projects by handing in\n", + "reports written like a standard scientific/technical report. The links\n", + "at\n", + "https://github.com/CompPhysics/MachineLearning/tree/master/doc/Projects\n", + "contain more information. There you can find examples of previous\n", + "reports, the projects themselves, how we grade reports etc. How to\n", + "write reports will also be discussed during the various lab\n", + "sessions. Please do ask us if you are in doubt.\n", + "\n", + "When using codes and material from other sources, you should refer to\n", + "these in the bibliography of your report, indicating wherefrom you for\n", + "example got the code, whether this is from the lecture notes,\n", + "softwares like Scikit-Learn, TensorFlow, PyTorch or other\n", + "sources. These sources should always be cited correctly. How to cite\n", + "some of the libraries is often indicated from their corresponding\n", + "GitHub sites or websites, see for example how to cite Scikit-Learn at\n", + "https://scikit-learn.org/dev/about.html.\n", + "\n", + "We enocurage you to use tools like ChatGPT or similar in writing the\n", + "report. If you use for example ChatGPT, please do cite it properly and\n", + "include (if possible) your questions and answers as an addition to the\n", + "report. This can be uploaded to for example your website,\n", + "GitHub/GitLab or similar as supplemental material.\n", + "\n", + "If you would like to study other data sets, feel free to propose other\n", + "sets. What we have proposed here are mere suggestions from our\n", + "side. If you opt for another data set, consider using a set which has\n", + "been studied in the scientific literature. This makes it easier for\n", + "you to compare and analyze your results. Comparing with existing\n", + "results from the scientific literature is also an essential element of\n", + "the scientific discussion. The University of California at Irvine with\n", + "its Machine Learning repository at\n", + "https://archive.ics.uci.edu/ml/index.php is an excellent site to look\n", + "up for examples and inspiration. Kaggle.com is an equally interesting\n", + "site. Feel free to explore these sites." + ] + }, + { + "cell_type": "markdown", + "id": "0d2c42e3", + "metadata": { + "editable": true + }, + "source": [ + "## Classification and Regression, writing our own neural network code\n", + "\n", + "The main aim of this project is to study both classification and\n", + "regression problems by developing our own \n", + "feed-forward neural network (FFNN) code. The exercises from week 41 and 42 (see and ) as well as the lecture material from the same weeks (see and ) should contain enough information for you to get started with writing your own code.\n", + "\n", + "We will also reuse our codes on gradient descent methods from project 1.\n", + "\n", + "The data sets that we propose here are (the default sets)\n", + "\n", + "* Regression (fitting a continuous function). In this part you will need to bring back your results from project 1 and compare these with what you get from your Neural Network code to be developed here. The data sets could be\n", + "\n", + " * The simple one-dimensional function Runge function from project 1, that is $f(x) = \\frac{1}{1+25x^2}$. We recommend using a simpler function when developing your neural network code for regression problems. Feel however free to discuss and study other functions, such as the the two-dimensional Runge function $f(x,y)=\\left[(10x - 5)^2 + (10y - 5)^2 + 1 \\right]^{-1}$, or even more complicated two-dimensional functions (see the supplementary material of for an extensive list of two-dimensional functions). \n", + "\n", + "* Classification.\n", + "\n", + " * We will consider a multiclass classification problem given by the full MNIST data set. The full data set is at .\n", + "\n", + "We will start with a regression problem and we will reuse our codes on gradient descent methods from project 1." + ] + }, + { + "cell_type": "markdown", + "id": "d8baab67", + "metadata": { + "editable": true + }, + "source": [ + "### Part a): Analytical warm-up\n", + "\n", + "When using our gradient machinery from project 1, we will need the expressions for the cost/loss functions and their respective\n", + "gradients. The functions whose gradients we need are:\n", + "1. The mean-squared error (MSE) with and without the $L_1$ and $L_2$ norms (regression problems)\n", + "\n", + "2. The binary cross entropy (aka log loss) for binary classification problems with and without $L_1$ and $L_2$ norms\n", + "\n", + "3. The multiclass cross entropy cost/loss function (aka Softmax cross entropy or just Softmax loss function)\n", + "\n", + "Set up these three cost/loss functions and their respective derivatives and explain the various terms. In this project you will however only use the MSE and the Softmax cross entropy.\n", + "\n", + "We will test three activation functions for our neural network setup, these are the \n", + "1. The Sigmoid (aka **logit**) function,\n", + "\n", + "2. the RELU function and\n", + "\n", + "3. the Leaky RELU function\n", + "\n", + "Set up their expressions and their first derivatives.\n", + "You may consult the lecture notes (with codes and more) from week 42 at ." + ] + }, + { + "cell_type": "markdown", + "id": "87e7ed71", + "metadata": { + "editable": true + }, + "source": [ + "### Reminder about the gradient machinery from project 1\n", + "\n", + "In the setup of a neural network code you will need your gradient descent codes from\n", + "project 1. For neural networks we will recommend using stochastic\n", + "gradient descent with either the RMSprop or the ADAM algorithms for\n", + "updating the learning rates. But you should feel free to try plain gradient descent as well.\n", + "\n", + "We recommend reading chapter 8 on optimization from the textbook of\n", + "Goodfellow, Bengio and Courville at\n", + ". This chapter contains many\n", + "useful insights and discussions on the optimization part of machine\n", + "learning. A useful reference on the back progagation algorithm is\n", + "Nielsen's book at . \n", + "\n", + "You will find the Python [Seaborn\n", + "package](https://seaborn.pydata.org/generated/seaborn.heatmap.html)\n", + "useful when plotting the results as function of the learning rate\n", + "$\\eta$ and the hyper-parameter $\\lambda$ ." + ] + }, + { + "cell_type": "markdown", + "id": "5a26b6ad", + "metadata": { + "editable": true + }, + "source": [ + "### Part b): Writing your own Neural Network code\n", + "\n", + "Your aim now, and this is the central part of this project, is to\n", + "write your own FFNN code implementing the back\n", + "propagation algorithm discussed in the lecture slides from week 41 at and week 42 at .\n", + "\n", + "We will focus on a regression problem first, using the one-dimensional Runge function" + ] + }, + { + "cell_type": "markdown", + "id": "096fe6c4", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "f(x) = \\frac{1}{1+25x^2},\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "fd986596", + "metadata": { + "editable": true + }, + "source": [ + "from project 1.\n", + "\n", + "Use only the mean-squared error as cost function (no regularization terms) and \n", + "write an FFNN code for a regression problem with a flexible number of hidden\n", + "layers and nodes using only the Sigmoid function as activation function for\n", + "the hidden layers. Initialize the weights using a normal\n", + "distribution. How would you initialize the biases? And which\n", + "activation function would you select for the final output layer?\n", + "And how would you set up your design/feature matrix? Hint: does it have to represent a polynomial approximation as you did in project 1? \n", + "\n", + "Train your network and compare the results with those from your OLS\n", + "regression code from project 1 using the one-dimensional Runge\n", + "function. When comparing your neural network code with the OLS\n", + "results from project 1, use the same data sets which gave you the best\n", + "MSE score. Moreover, use the polynomial order from project 1 that gave you the\n", + "best result. Compare these results with your neural network with one\n", + "and two hidden layers using $50$ and $100$ hidden nodes, respectively.\n", + "\n", + "Comment your results and give a critical discussion of the results\n", + "obtained with the OLS code from project 1 and your own neural network\n", + "code. Make an analysis of the learning rates employed to find the\n", + "optimal MSE score. Test both stochastic gradient descent\n", + "with RMSprop and ADAM and plain gradient descent with different\n", + "learning rates.\n", + "\n", + "You should, as you did in project 1, scale your data." + ] + }, + { + "cell_type": "markdown", + "id": "e853d4b6", + "metadata": { + "editable": true + }, + "source": [ + "### Part c): Testing against other software libraries\n", + "\n", + "You should test your results against a similar code using **Scikit-Learn** (see the examples in the above lecture notes from weeks 41 and 42) or **tensorflow/keras** or **Pytorch** (for Pytorch, see Raschka et al.'s text chapters 12 and 13). \n", + "\n", + "Furthermore, you should also test that your derivatives are correctly\n", + "calculated using automatic differentiation, using for example the\n", + "**Autograd** library or the **JAX** library. It is optional to implement\n", + "these libraries for the present project. In this project they serve as\n", + "useful tests of our derivatives." + ] + }, + { + "cell_type": "markdown", + "id": "fc2d413b", + "metadata": { + "editable": true + }, + "source": [ + "### Part d): Testing different activation functions and depths of the neural network\n", + "\n", + "You should also test different activation functions for the hidden\n", + "layers. Try out the Sigmoid, the RELU and the Leaky RELU functions and\n", + "discuss your results. Test your results as functions of the number of hidden layers and nodes. Do you see signs of overfitting?\n", + "It is optional in this project to perform a bias-variance trade-off analysis." + ] + }, + { + "cell_type": "markdown", + "id": "e6821051", + "metadata": { + "editable": true + }, + "source": [ + "### Part e): Testing different norms\n", + "\n", + "Finally, still using the one-dimensional Runge function, add now the\n", + "hyperparameters $\\lambda$ with the $L_2$ and $L_1$ norms. Find the\n", + "optimal results for the hyperparameters $\\lambda$ and the learning\n", + "rates $\\eta$ and neural network architecture and compare the $L_2$ results with Ridge regression from\n", + "project 1 and the $L_1$ results with the Lasso calculations of project 1.\n", + "Use again the same data sets and the best results from project 1 in your comparisons." + ] + }, + { + "cell_type": "markdown", + "id": "cba72d68", + "metadata": { + "editable": true + }, + "source": [ + "### Part f): Classification analysis using neural networks\n", + "\n", + "With a well-written code it should now be easy to change the\n", + "activation function for the output layer.\n", + "\n", + "Here we will change the cost function for our neural network code\n", + "developed in parts b), d) and e) in order to perform a classification\n", + "analysis. The classification problem we will study is the multiclass\n", + "MNIST problem, see the description of the full data set at\n", + ". We will use the Softmax cross entropy function discussed in a). \n", + "The MNIST data set discussed in the lecture notes from week 42 is a downscaled variant of the full dataset. \n", + "\n", + "Feel free to suggest other data sets. If you find the classic MNIST data set somewhat limited, feel free to try the \n", + "MNIST-Fashion data set at for example .\n", + "\n", + "To set up the data set, the following python programs may be useful" + ] + }, + { + "cell_type": "code", + "execution_count": 1, + "id": "e16fb528", + "metadata": { + "collapsed": false, + "editable": true + }, + "outputs": [], + "source": [ + "from sklearn.datasets import fetch_openml\n", + "\n", + "# Fetch the MNIST dataset\n", + "mnist = fetch_openml('mnist_784', version=1, as_frame=False, parser='auto')\n", + "\n", + "# Extract data (features) and target (labels)\n", + "X = mnist.data\n", + "y = mnist.target" + ] + }, + { + "cell_type": "markdown", + "id": "73599f42", + "metadata": { + "editable": true + }, + "source": [ + "You should consider scaling the data. The Pixel values in MNIST range from 0 to 255. Scaling them to a 0-1 range can improve the performance of some models. That is, you could implement the following scaling" + ] + }, + { + "cell_type": "code", + "execution_count": 2, + "id": "f1a639ef", + "metadata": { + "collapsed": false, + "editable": true + }, + "outputs": [], + "source": [ + "X = X / 255.0" + ] + }, + { + "cell_type": "markdown", + "id": "90fb7b41", + "metadata": { + "editable": true + }, + "source": [ + "And then perform the standard train-test splitting" + ] + }, + { + "cell_type": "code", + "execution_count": 3, + "id": "424af629", + "metadata": { + "collapsed": false, + "editable": true + }, + "outputs": [], + "source": [ + "from sklearn.model_selection import train_test_split\n", + "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)" + ] + }, + { + "cell_type": "markdown", + "id": "3c006080", + "metadata": { + "editable": true + }, + "source": [ + "To measure the performance of our classification problem we will use the\n", + "so-called *accuracy* score. The accuracy is as you would expect just\n", + "the number of correctly guessed targets $t_i$ divided by the total\n", + "number of targets, that is" + ] + }, + { + "cell_type": "markdown", + "id": "a18ddd54", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\text{Accuracy} = \\frac{\\sum_{i=1}^n I(t_i = y_i)}{n} ,\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "1a1afaf9", + "metadata": { + "editable": true + }, + "source": [ + "where $I$ is the indicator function, $1$ if $t_i = y_i$ and $0$\n", + "otherwise if we have a binary classification problem. Here $t_i$\n", + "represents the target and $y_i$ the outputs of your FFNN code and $n$ is simply the number of targets $t_i$.\n", + "\n", + "Discuss your results and give a critical analysis of the various parameters, including hyper-parameters like the learning rates and the regularization parameter $\\lambda$, various activation functions, number of hidden layers and nodes and activation functions. \n", + "\n", + "Again, we strongly recommend that you compare your own neural Network\n", + "code for classification and pertinent results against a similar code using **Scikit-Learn** or **tensorflow/keras** or **pytorch**.\n", + "\n", + "If you have time, you can use the functionality of **scikit-learn** and compare your neural network results with those from Logistic regression. This is optional.\n", + "The weblink here compares logistic regression and FFNN using the so-called MNIST data set. You may find several useful hints and ideas from this article. Your neural network code can implement the equivalent of logistic regression by simply setting the number of hidden layers to zero. \n", + "\n", + "If you wish to compare with say Logisti Regression from **scikit-learn**, the following code uses the above data set" + ] + }, + { + "cell_type": "code", + "execution_count": 4, + "id": "3c37cbaf", + "metadata": { + "collapsed": false, + "editable": true + }, + "outputs": [], + "source": [ + "from sklearn.linear_model import LogisticRegression\n", + "# Initialize the model\n", + "model = LogisticRegression(solver='saga', multi_class='multinomial', max_iter=1000, random_state=42)\n", + "# Train the model\n", + "model.fit(X_train, y_train)\n", + "from sklearn.metrics import accuracy_score\n", + "# Make predictions on the test set\n", + "y_pred = model.predict(X_test)\n", + "# Calculate accuracy\n", + "accuracy = accuracy_score(y_test, y_pred)\n", + "print(f\"Model Accuracy: {accuracy:.4f}\")" + ] + }, + { + "cell_type": "markdown", + "id": "106b9303", + "metadata": { + "editable": true + }, + "source": [ + "### Part g) Critical evaluation of the various algorithms\n", + "\n", + "After all these glorious calculations, you should now summarize the\n", + "various algorithms and come with a critical evaluation of their pros\n", + "and cons. Which algorithm works best for the regression case and which\n", + "is best for the classification case. These codes can also be part of\n", + "your final project 3, but now applied to other data sets." + ] + }, + { + "cell_type": "markdown", + "id": "55da0d7f", + "metadata": { + "editable": true + }, + "source": [ + "## Background literature\n", + "\n", + "1. The text of Michael Nielsen is highly recommended, see Nielsen's book at . It is an excellent read.\n", + "\n", + "2. Goodfellow, Bengio and Courville, Deep Learning at . Here we recommend chapters 6, 7 and 8\n", + "\n", + "3. Raschka et al. at . Here we recommend chapters 11, 12 and 13." + ] + }, + { + "cell_type": "markdown", + "id": "d3731e2c", + "metadata": { + "editable": true + }, + "source": [ + "## Introduction to numerical projects\n", + "\n", + "Here follows a brief recipe and recommendation on how to write a report for each\n", + "project.\n", + "\n", + " * Give a short description of the nature of the problem and the eventual numerical methods you have used.\n", + "\n", + " * Describe the algorithm you have used and/or developed. Here you may find it convenient to use pseudocoding. In many cases you can describe the algorithm in the program itself.\n", + "\n", + " * Include the source code of your program. Comment your program properly.\n", + "\n", + " * If possible, try to find analytic solutions, or known limits in order to test your program when developing the code.\n", + "\n", + " * Include your results either in figure form or in a table. Remember to label your results. All tables and figures should have relevant captions and labels on the axes.\n", + "\n", + " * Try to evaluate the reliabilty and numerical stability/precision of your results. If possible, include a qualitative and/or quantitative discussion of the numerical stability, eventual loss of precision etc.\n", + "\n", + " * Try to give an interpretation of you results in your answers to the problems.\n", + "\n", + " * Critique: if possible include your comments and reflections about the exercise, whether you felt you learnt something, ideas for improvements and other thoughts you've made when solving the exercise. We wish to keep this course at the interactive level and your comments can help us improve it.\n", + "\n", + " * Try to establish a practice where you log your work at the computerlab. You may find such a logbook very handy at later stages in your work, especially when you don't properly remember what a previous test version of your program did. Here you could also record the time spent on solving the exercise, various algorithms you may have tested or other topics which you feel worthy of mentioning." + ] + }, + { + "cell_type": "markdown", + "id": "6c7c5340", + "metadata": { + "editable": true + }, + "source": [ + "## Format for electronic delivery of report and programs\n", + "\n", + "The preferred format for the report is a PDF file. You can also use DOC or postscript formats or as an ipython notebook file. As programming language we prefer that you choose between C/C++, Fortran2008 or Python. The following prescription should be followed when preparing the report:\n", + "\n", + " * Use Canvas to hand in your projects, log in at with your normal UiO username and password.\n", + "\n", + " * Upload **only** the report file or the link to your GitHub/GitLab or similar typo of repos! For the source code file(s) you have developed please provide us with your link to your GitHub/GitLab or similar domain. The report file should include all of your discussions and a list of the codes you have developed. Do not include library files which are available at the course homepage, unless you have made specific changes to them.\n", + "\n", + " * In your GitHub/GitLab or similar repository, please include a folder which contains selected results. These can be in the form of output from your code for a selected set of runs and input parameters.\n", + "\n", + "Finally, \n", + "we encourage you to collaborate. Optimal working groups consist of \n", + "2-3 students. You can then hand in a common report." + ] + } + ], + "metadata": {}, + "nbformat": 4, + "nbformat_minor": 5 +} \ No newline at end of file diff --git a/doc/LectureNotes/_build/jupyter_execute/week42.ipynb b/doc/LectureNotes/_build/jupyter_execute/week42.ipynb new file mode 100644 index 000000000..50274c3c7 --- /dev/null +++ b/doc/LectureNotes/_build/jupyter_execute/week42.ipynb @@ -0,0 +1,5950 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "id": "7dcf8613", + "metadata": { + "editable": true + }, + "source": [ + "\n", + "" + ] + }, + { + "cell_type": "markdown", + "id": "98019924", + "metadata": { + "editable": true + }, + "source": [ + "# Week 42 Constructing a Neural Network code with examples\n", + "**Morten Hjorth-Jensen**, Department of Physics, University of Oslo, Norway\n", + "\n", + "Date: **October 13-17, 2025**" + ] + }, + { + "cell_type": "markdown", + "id": "ab0e660a", + "metadata": { + "editable": true + }, + "source": [ + "## Lecture October 13, 2025\n", + "1. Building our own Feed-forward Neural Network and discussion of project 2\n", + "\n", + "2. Project 2 is available at " + ] + }, + { + "cell_type": "markdown", + "id": "bcea021f", + "metadata": { + "editable": true + }, + "source": [ + "## Readings and videos\n", + "1. These lecture notes\n", + "\n", + "\n", + "\n", + "2. For a more in depth discussion on neural networks we recommend Goodfellow et al chapters 6 and 7. For the optimization part, see chapter 8. \n", + "\n", + "3. Neural Networks demystified at \n", + "\n", + "4. Building Neural Networks from scratch at \n", + "\n", + "5. Video on Neural Networks at \n", + "\n", + "6. Video on the back propagation algorithm at \n", + "\n", + "I also recommend Michael Nielsen's intuitive approach to the neural networks and the universal approximation theorem, see the slides at ." + ] + }, + { + "cell_type": "markdown", + "id": "fd77cd6b", + "metadata": { + "editable": true + }, + "source": [ + "## Material for the lab sessions on Tuesday and Wednesday\n", + "1. Exercises on writing a code for neural networks, back propagation part, see exercises for week 42 at \n", + "\n", + "2. Discussion of project 2" + ] + }, + { + "cell_type": "markdown", + "id": "7d93059a", + "metadata": { + "editable": true + }, + "source": [ + "## Lecture material: Writing a code which implements a feed-forward neural network\n", + "\n", + "Last week we discussed the basics of neural networks and deep learning\n", + "and the basics of automatic differentiation. We looked also at\n", + "examples on how compute the parameters of a simple network with scalar\n", + "inputs and ouputs and no or just one hidden layers.\n", + "\n", + "We ended our discussions with the derivation of the equations for a\n", + "neural network with one hidden layers and two input variables and two\n", + "hidden nodes but only one output node. We did almost finish the derivation of the back propagation algorithm." + ] + }, + { + "cell_type": "markdown", + "id": "5156bf60", + "metadata": { + "editable": true + }, + "source": [ + "## Mathematics of deep learning\n", + "\n", + "**Two recent books online.**\n", + "\n", + "1. [The Modern Mathematics of Deep Learning, by Julius Berner, Philipp Grohs, Gitta Kutyniok, Philipp Petersen](https://arxiv.org/abs/2105.04026), published as [Mathematical Aspects of Deep Learning, pp. 1-111. Cambridge University Press, 2022](https://doi.org/10.1017/9781009025096.002)\n", + "\n", + "2. [Mathematical Introduction to Deep Learning: Methods, Implementations, and Theory, Arnulf Jentzen, Benno Kuckuck, Philippe von Wurstemberger](https://doi.org/10.48550/arXiv.2310.20360)" + ] + }, + { + "cell_type": "markdown", + "id": "4f5d5b27", + "metadata": { + "editable": true + }, + "source": [ + "## Reminder on books with hands-on material and codes\n", + "* [Sebastian Rashcka et al, Machine learning with Sickit-Learn and PyTorch](https://sebastianraschka.com/blog/2022/ml-pytorch-book.html)" + ] + }, + { + "cell_type": "markdown", + "id": "0a490014", + "metadata": { + "editable": true + }, + "source": [ + "## Reading recommendations\n", + "\n", + "1. Rashkca et al., chapter 11, jupyter-notebook sent separately, from [GitHub](https://github.com/rasbt/machine-learning-book)\n", + "\n", + "2. Goodfellow et al, chapter 6 and 7 contain most of the neural network background." + ] + }, + { + "cell_type": "markdown", + "id": "9fa74383", + "metadata": { + "editable": true + }, + "source": [ + "## Reminder from last week: First network example, simple percepetron with one input\n", + "\n", + "As yet another example we define now a simple perceptron model with\n", + "all quantities given by scalars. We consider only one input variable\n", + "$x$ and one target value $y$. We define an activation function\n", + "$\\sigma_1$ which takes as input" + ] + }, + { + "cell_type": "markdown", + "id": "417ce076", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "z_1 = w_1x+b_1,\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "07ed6afb", + "metadata": { + "editable": true + }, + "source": [ + "where $w_1$ is the weight and $b_1$ is the bias. These are the\n", + "parameters we want to optimize. The output is $a_1=\\sigma(z_1)$ (see\n", + "graph from whiteboard notes). This output is then fed into the\n", + "**cost/loss** function, which we here for the sake of simplicity just\n", + "define as the squared error" + ] + }, + { + "cell_type": "markdown", + "id": "3b926536", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "C(x;w_1,b_1)=\\frac{1}{2}(a_1-y)^2.\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "0e1bd556", + "metadata": { + "editable": true + }, + "source": [ + "## Layout of a simple neural network with no hidden layer\n", + "\n", + "\n", + "\n", + "\n", + "

    Figure 1:

    \n", + "" + ] + }, + { + "cell_type": "markdown", + "id": "edebc935", + "metadata": { + "editable": true + }, + "source": [ + "## Optimizing the parameters\n", + "\n", + "In setting up the feed forward and back propagation parts of the\n", + "algorithm, we need now the derivative of the various variables we want\n", + "to train.\n", + "\n", + "We need" + ] + }, + { + "cell_type": "markdown", + "id": "3e2794ac", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\frac{\\partial C}{\\partial w_1} \\hspace{0.1cm}\\mathrm{and}\\hspace{0.1cm}\\frac{\\partial C}{\\partial b_1}.\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "54f91430", + "metadata": { + "editable": true + }, + "source": [ + "Using the chain rule we find" + ] + }, + { + "cell_type": "markdown", + "id": "f1e0a166", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\frac{\\partial C}{\\partial w_1}=\\frac{\\partial C}{\\partial a_1}\\frac{\\partial a_1}{\\partial z_1}\\frac{\\partial z_1}{\\partial w_1}=(a_1-y)\\sigma_1'x,\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "ad62ed90", + "metadata": { + "editable": true + }, + "source": [ + "and" + ] + }, + { + "cell_type": "markdown", + "id": "57dac362", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\frac{\\partial C}{\\partial b_1}=\\frac{\\partial C}{\\partial a_1}\\frac{\\partial a_1}{\\partial z_1}\\frac{\\partial z_1}{\\partial b_1}=(a_1-y)\\sigma_1',\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "a4dc4050", + "metadata": { + "editable": true + }, + "source": [ + "which we later will just define as" + ] + }, + { + "cell_type": "markdown", + "id": "b223c83d", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\frac{\\partial C}{\\partial a_1}\\frac{\\partial a_1}{\\partial z_1}=\\delta_1.\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "a1c9b5e7", + "metadata": { + "editable": true + }, + "source": [ + "## Adding a hidden layer\n", + "\n", + "We change our simple model to (see graph)\n", + "a network with just one hidden layer but with scalar variables only.\n", + "\n", + "Our output variable changes to $a_2$ and $a_1$ is now the output from the hidden node and $a_0=x$.\n", + "We have then" + ] + }, + { + "cell_type": "markdown", + "id": "9c3b03c0", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "z_1 = w_1a_0+b_1 \\hspace{0.1cm} \\wedge a_1 = \\sigma_1(z_1),\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "186485f2", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "z_2 = w_2a_1+b_2 \\hspace{0.1cm} \\wedge a_2 = \\sigma_2(z_2),\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "c2f1b30b", + "metadata": { + "editable": true + }, + "source": [ + "and the cost function" + ] + }, + { + "cell_type": "markdown", + "id": "13142fad", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "C(x;\\boldsymbol{\\Theta})=\\frac{1}{2}(a_2-y)^2,\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "b441f95a", + "metadata": { + "editable": true + }, + "source": [ + "with $\\boldsymbol{\\Theta}=[w_1,w_2,b_1,b_2]$." + ] + }, + { + "cell_type": "markdown", + "id": "46e57fd2", + "metadata": { + "editable": true + }, + "source": [ + "## Layout of a simple neural network with one hidden layer\n", + "\n", + "\n", + "\n", + "\n", + "

    Figure 1:

    \n", + "" + ] + }, + { + "cell_type": "markdown", + "id": "50158555", + "metadata": { + "editable": true + }, + "source": [ + "## The derivatives\n", + "\n", + "The derivatives are now, using the chain rule again" + ] + }, + { + "cell_type": "markdown", + "id": "4174ea9e", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\frac{\\partial C}{\\partial w_2}=\\frac{\\partial C}{\\partial a_2}\\frac{\\partial a_2}{\\partial z_2}\\frac{\\partial z_2}{\\partial w_2}=(a_2-y)\\sigma_2'a_1=\\delta_2a_1,\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "f7a29d74", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\frac{\\partial C}{\\partial b_2}=\\frac{\\partial C}{\\partial a_2}\\frac{\\partial a_2}{\\partial z_2}\\frac{\\partial z_2}{\\partial b_2}=(a_2-y)\\sigma_2'=\\delta_2,\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "cb1a387b", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\frac{\\partial C}{\\partial w_1}=\\frac{\\partial C}{\\partial a_2}\\frac{\\partial a_2}{\\partial z_2}\\frac{\\partial z_2}{\\partial a_1}\\frac{\\partial a_1}{\\partial z_1}\\frac{\\partial z_1}{\\partial w_1}=(a_2-y)\\sigma_2'a_1\\sigma_1'a_0,\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "125bcb29", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\frac{\\partial C}{\\partial b_1}=\\frac{\\partial C}{\\partial a_2}\\frac{\\partial a_2}{\\partial z_2}\\frac{\\partial z_2}{\\partial a_1}\\frac{\\partial a_1}{\\partial z_1}\\frac{\\partial z_1}{\\partial b_1}=(a_2-y)\\sigma_2'\\sigma_1'=\\delta_1.\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "76272ae9", + "metadata": { + "editable": true + }, + "source": [ + "Can you generalize this to more than one hidden layer?" + ] + }, + { + "cell_type": "markdown", + "id": "7e0d1157", + "metadata": { + "editable": true + }, + "source": [ + "## Important observations\n", + "\n", + "From the above equations we see that the derivatives of the activation\n", + "functions play a central role. If they vanish, the training may\n", + "stop. This is called the vanishing gradient problem, see discussions below. If they become\n", + "large, the parameters $w_i$ and $b_i$ may simply go to infinity. This\n", + "is referenced as the exploding gradient problem." + ] + }, + { + "cell_type": "markdown", + "id": "53dfc3bd", + "metadata": { + "editable": true + }, + "source": [ + "## The training\n", + "\n", + "The training of the parameters is done through various gradient descent approximations with" + ] + }, + { + "cell_type": "markdown", + "id": "126420bb", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "w_{i}\\leftarrow w_{i}- \\eta \\delta_i a_{i-1},\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "2114b4ba", + "metadata": { + "editable": true + }, + "source": [ + "and" + ] + }, + { + "cell_type": "markdown", + "id": "7d3549bb", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "b_i \\leftarrow b_i-\\eta \\delta_i,\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "80265b39", + "metadata": { + "editable": true + }, + "source": [ + "with $\\eta$ is the learning rate.\n", + "\n", + "One iteration consists of one feed forward step and one back-propagation step. Each back-propagation step does one update of the parameters $\\boldsymbol{\\Theta}$.\n", + "\n", + "For the first hidden layer $a_{i-1}=a_0=x$ for this simple model." + ] + }, + { + "cell_type": "markdown", + "id": "df2f78ac", + "metadata": { + "editable": true + }, + "source": [ + "## Code example\n", + "\n", + "The code here implements the above model with one hidden layer and\n", + "scalar variables for the same function we studied in the previous\n", + "example. The code is however set up so that we can add multiple\n", + "inputs $x$ and target values $y$. Note also that we have the\n", + "possibility of defining a feature matrix $\\boldsymbol{X}$ with more than just\n", + "one column for the input values. This will turn useful in our next example. We have also defined matrices and vectors for all of our operations although it is not necessary here." + ] + }, + { + "cell_type": "code", + "execution_count": 1, + "id": "0ec652bf", + "metadata": { + "collapsed": false, + "editable": true + }, + "outputs": [], + "source": [ + "import numpy as np\n", + "# We use the Sigmoid function as activation function\n", + "def sigmoid(z):\n", + " return 1.0/(1.0+np.exp(-z))\n", + "\n", + "def forwardpropagation(x):\n", + " # weighted sum of inputs to the hidden layer\n", + " z_1 = np.matmul(x, w_1) + b_1\n", + " # activation in the hidden layer\n", + " a_1 = sigmoid(z_1)\n", + " # weighted sum of inputs to the output layer\n", + " z_2 = np.matmul(a_1, w_2) + b_2\n", + " a_2 = z_2\n", + " return a_1, a_2\n", + "\n", + "def backpropagation(x, y):\n", + " a_1, a_2 = forwardpropagation(x)\n", + " # parameter delta for the output layer, note that a_2=z_2 and its derivative wrt z_2 is just 1\n", + " delta_2 = a_2 - y\n", + " print(0.5*((a_2-y)**2))\n", + " # delta for the hidden layer\n", + " delta_1 = np.matmul(delta_2, w_2.T) * a_1 * (1 - a_1)\n", + " # gradients for the output layer\n", + " output_weights_gradient = np.matmul(a_1.T, delta_2)\n", + " output_bias_gradient = np.sum(delta_2, axis=0)\n", + " # gradient for the hidden layer\n", + " hidden_weights_gradient = np.matmul(x.T, delta_1)\n", + " hidden_bias_gradient = np.sum(delta_1, axis=0)\n", + " return output_weights_gradient, output_bias_gradient, hidden_weights_gradient, hidden_bias_gradient\n", + "\n", + "\n", + "# ensure the same random numbers appear every time\n", + "np.random.seed(0)\n", + "# Input variable\n", + "x = np.array([4.0],dtype=np.float64)\n", + "# Target values\n", + "y = 2*x+1.0 \n", + "\n", + "# Defining the neural network, only scalars here\n", + "n_inputs = x.shape\n", + "n_features = 1\n", + "n_hidden_neurons = 1\n", + "n_outputs = 1\n", + "\n", + "# Initialize the network\n", + "# weights and bias in the hidden layer\n", + "w_1 = np.random.randn(n_features, n_hidden_neurons)\n", + "b_1 = np.zeros(n_hidden_neurons) + 0.01\n", + "\n", + "# weights and bias in the output layer\n", + "w_2 = np.random.randn(n_hidden_neurons, n_outputs)\n", + "b_2 = np.zeros(n_outputs) + 0.01\n", + "\n", + "eta = 0.1\n", + "for i in range(50):\n", + " # calculate gradients\n", + " derivW2, derivB2, derivW1, derivB1 = backpropagation(x, y)\n", + " # update weights and biases\n", + " w_2 -= eta * derivW2\n", + " b_2 -= eta * derivB2\n", + " w_1 -= eta * derivW1\n", + " b_1 -= eta * derivB1" + ] + }, + { + "cell_type": "markdown", + "id": "c382e2f2", + "metadata": { + "editable": true + }, + "source": [ + "We see that after some few iterations (the results do depend on the learning rate however), we get an error which is rather small." + ] + }, + { + "cell_type": "markdown", + "id": "1bec2330", + "metadata": { + "editable": true + }, + "source": [ + "## Simple neural network and the back propagation equations\n", + "\n", + "Let us now try to increase our level of ambition and attempt at setting \n", + "up the equations for a neural network with two input nodes, one hidden\n", + "layer with two hidden nodes and one output layer with one output node/neuron only (see graph)..\n", + "\n", + "We need to define the following parameters and variables with the input layer (layer $(0)$) \n", + "where we label the nodes $x_1$ and $x_2$" + ] + }, + { + "cell_type": "markdown", + "id": "165fb3f3", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "x_1 = a_1^{(0)} \\wedge x_2 = a_2^{(0)}.\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "7ae8de36", + "metadata": { + "editable": true + }, + "source": [ + "The hidden layer (layer $(1)$) has nodes which yield the outputs $a_1^{(1)}$ and $a_2^{(1)}$) with weight $\\boldsymbol{w}$ and bias $\\boldsymbol{b}$ parameters" + ] + }, + { + "cell_type": "markdown", + "id": "9a2729ab", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "w_{ij}^{(1)}=\\left\\{w_{11}^{(1)},w_{12}^{(1)},w_{21}^{(1)},w_{22}^{(1)}\\right\\} \\wedge b^{(1)}=\\left\\{b_1^{(1)},b_2^{(1)}\\right\\}.\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "372f0fac", + "metadata": { + "editable": true + }, + "source": [ + "## Layout of a simple neural network with two input nodes, one hidden layer with two hidden noeds and one output node\n", + "\n", + "\n", + "\n", + "\n", + "

    Figure 1:

    \n", + "" + ] + }, + { + "cell_type": "markdown", + "id": "d4f432cc", + "metadata": { + "editable": true + }, + "source": [ + "## The ouput layer\n", + "\n", + "We have the ouput layer given by layer label $(2)$ with output $a^{(2)}$ and weights and biases to be determined given by the variables" + ] + }, + { + "cell_type": "markdown", + "id": "064af49b", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "w_{i}^{(2)}=\\left\\{w_{1}^{(2)},w_{2}^{(2)}\\right\\} \\wedge b^{(2)}.\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "91115cc6", + "metadata": { + "editable": true + }, + "source": [ + "Our output is $\\tilde{y}=a^{(2)}$ and we define a generic cost function $C(a^{(2)},y;\\boldsymbol{\\Theta})$ where $y$ is the target value (a scalar here).\n", + "The parameters we need to optimize are given by" + ] + }, + { + "cell_type": "markdown", + "id": "c38d1990", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\boldsymbol{\\Theta}=\\left\\{w_{11}^{(1)},w_{12}^{(1)},w_{21}^{(1)},w_{22}^{(1)},w_{1}^{(2)},w_{2}^{(2)},b_1^{(1)},b_2^{(1)},b^{(2)}\\right\\}.\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "43de152e", + "metadata": { + "editable": true + }, + "source": [ + "## Compact expressions\n", + "\n", + "We can define the inputs to the activation functions for the various layers in terms of various matrix-vector multiplications and vector additions.\n", + "The inputs to the first hidden layer are" + ] + }, + { + "cell_type": "markdown", + "id": "f935912b", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\begin{bmatrix}z_1^{(1)} \\\\ z_2^{(1)} \\end{bmatrix}=\\left(\\begin{bmatrix}w_{11}^{(1)} & w_{12}^{(1)}\\\\ w_{21}^{(1)} &w_{22}^{(1)} \\end{bmatrix}\\right)^{T}\\begin{bmatrix}a_1^{(0)} \\\\ a_2^{(0)} \\end{bmatrix}+\\begin{bmatrix}b_1^{(1)} \\\\ b_2^{(1)} \\end{bmatrix},\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "15b380de", + "metadata": { + "editable": true + }, + "source": [ + "with outputs" + ] + }, + { + "cell_type": "markdown", + "id": "302da250", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\begin{bmatrix}a_1^{(1)} \\\\ a_2^{(1)} \\end{bmatrix}=\\begin{bmatrix}\\sigma^{(1)}(z_1^{(1)}) \\\\ \\sigma^{(1)}(z_2^{(1)}) \\end{bmatrix}.\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "3faf23cf", + "metadata": { + "editable": true + }, + "source": [ + "## Output layer\n", + "\n", + "For the final output layer we have the inputs to the final activation function" + ] + }, + { + "cell_type": "markdown", + "id": "4fe83376", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "z^{(2)} = w_{1}^{(2)}a_1^{(1)} +w_{2}^{(2)}a_2^{(1)}+b^{(2)},\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "9bee26b1", + "metadata": { + "editable": true + }, + "source": [ + "resulting in the output" + ] + }, + { + "cell_type": "markdown", + "id": "9affa08f", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "a^{(2)}=\\sigma^{(2)}(z^{(2)}).\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "1866378d", + "metadata": { + "editable": true + }, + "source": [ + "## Explicit derivatives\n", + "\n", + "In total we have nine parameters which we need to train. Using the\n", + "chain rule (or just the back-propagation algorithm) we can find all\n", + "derivatives. Since we will use automatic differentiation in reverse\n", + "mode, we start with the derivatives of the cost function with respect\n", + "to the parameters of the output layer, namely" + ] + }, + { + "cell_type": "markdown", + "id": "e6d27f9d", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\frac{\\partial C}{\\partial w_{i}^{(2)}}=\\frac{\\partial C}{\\partial a^{(2)}}\\frac{\\partial a^{(2)}}{\\partial z^{(2)}}\\frac{\\partial z^{(2)}}{\\partial w_{i}^{(2)}}=\\delta^{(2)}a_i^{(1)},\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "b33dac74", + "metadata": { + "editable": true + }, + "source": [ + "with" + ] + }, + { + "cell_type": "markdown", + "id": "27c367b4", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\delta^{(2)}=\\frac{\\partial C}{\\partial a^{(2)}}\\frac{\\partial a^{(2)}}{\\partial z^{(2)}}\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "18898e14", + "metadata": { + "editable": true + }, + "source": [ + "and finally" + ] + }, + { + "cell_type": "markdown", + "id": "31fcc7f9", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\frac{\\partial C}{\\partial b^{(2)}}=\\frac{\\partial C}{\\partial a^{(2)}}\\frac{\\partial a^{(2)}}{\\partial z^{(2)}}\\frac{\\partial z^{(2)}}{\\partial b^{(2)}}=\\delta^{(2)}.\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "76219fdd", + "metadata": { + "editable": true + }, + "source": [ + "## Derivatives of the hidden layer\n", + "\n", + "Using the chain rule we have the following expressions for say one of the weight parameters (it is easy to generalize to the other weight parameters)" + ] + }, + { + "cell_type": "markdown", + "id": "2cbccf94", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\frac{\\partial C}{\\partial w_{11}^{(1)}}=\\frac{\\partial C}{\\partial a^{(2)}}\\frac{\\partial a^{(2)}}{\\partial z^{(2)}}\n", + "\\frac{\\partial z^{(2)}}{\\partial z_1^{(1)}}\\frac{\\partial z_1^{(1)}}{\\partial w_{11}^{(1)}}= \\delta^{(2)}\\frac{\\partial z^{(2)}}{\\partial z_1^{(1)}}\\frac{\\partial z_1^{(1)}}{\\partial w_{11}^{(1)}},\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "ccdb071b", + "metadata": { + "editable": true + }, + "source": [ + "which, noting that" + ] + }, + { + "cell_type": "markdown", + "id": "55e4bed1", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "z^{(2)} =w_1^{(2)}a_1^{(1)}+w_2^{(2)}a_2^{(1)}+b^{(2)},\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "c00fd151", + "metadata": { + "editable": true + }, + "source": [ + "allows us to rewrite" + ] + }, + { + "cell_type": "markdown", + "id": "d49ae60d", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\frac{\\partial z^{(2)}}{\\partial z_1^{(1)}}\\frac{\\partial z_1^{(1)}}{\\partial w_{11}^{(1)}}=w_1^{(2)}\\frac{\\partial a_1^{(1)}}{\\partial z_1^{(1)}}a_1^{(1)}.\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "920db2dd", + "metadata": { + "editable": true + }, + "source": [ + "## Final expression\n", + "Defining" + ] + }, + { + "cell_type": "markdown", + "id": "62806f12", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\delta_1^{(1)}=w_1^{(2)}\\frac{\\partial a_1^{(1)}}{\\partial z_1^{(1)}}\\delta^{(2)},\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "f1d3e7b6", + "metadata": { + "editable": true + }, + "source": [ + "we have" + ] + }, + { + "cell_type": "markdown", + "id": "51fd0b81", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\frac{\\partial C}{\\partial w_{11}^{(1)}}=\\delta_1^{(1)}a_1^{(1)}.\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "ba9dd4ad", + "metadata": { + "editable": true + }, + "source": [ + "Similarly, we obtain" + ] + }, + { + "cell_type": "markdown", + "id": "91137440", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\frac{\\partial C}{\\partial w_{12}^{(1)}}=\\delta_1^{(1)}a_2^{(1)}.\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "8e2ecb8c", + "metadata": { + "editable": true + }, + "source": [ + "## Completing the list\n", + "\n", + "Similarly, we find" + ] + }, + { + "cell_type": "markdown", + "id": "8d91daf9", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\frac{\\partial C}{\\partial w_{21}^{(1)}}=\\delta_2^{(1)}a_1^{(1)},\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "8450f964", + "metadata": { + "editable": true + }, + "source": [ + "and" + ] + }, + { + "cell_type": "markdown", + "id": "1e662a52", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\frac{\\partial C}{\\partial w_{22}^{(1)}}=\\delta_2^{(1)}a_2^{(1)},\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "e90d34b5", + "metadata": { + "editable": true + }, + "source": [ + "where we have defined" + ] + }, + { + "cell_type": "markdown", + "id": "2d4b194d", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\delta_2^{(1)}=w_2^{(2)}\\frac{\\partial a_2^{(1)}}{\\partial z_2^{(1)}}\\delta^{(2)}.\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "8990fe0b", + "metadata": { + "editable": true + }, + "source": [ + "## Final expressions for the biases of the hidden layer\n", + "\n", + "For the sake of completeness, we list the derivatives of the biases, which are" + ] + }, + { + "cell_type": "markdown", + "id": "37e9e142", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\frac{\\partial C}{\\partial b_{1}^{(1)}}=\\delta_1^{(1)},\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "5f66c64a", + "metadata": { + "editable": true + }, + "source": [ + "and" + ] + }, + { + "cell_type": "markdown", + "id": "03e836ad", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\frac{\\partial C}{\\partial b_{2}^{(1)}}=\\delta_2^{(1)}.\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "1ea8a6ff", + "metadata": { + "editable": true + }, + "source": [ + "As we will see below, these expressions can be generalized in a more compact form." + ] + }, + { + "cell_type": "markdown", + "id": "84537cc9", + "metadata": { + "editable": true + }, + "source": [ + "## Gradient expressions\n", + "\n", + "For this specific model, with just one output node and two hidden\n", + "nodes, the gradient descent equations take the following form for output layer" + ] + }, + { + "cell_type": "markdown", + "id": "2bad6141", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "w_{i}^{(2)}\\leftarrow w_{i}^{(2)}- \\eta \\delta^{(2)} a_{i}^{(1)},\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "5150b413", + "metadata": { + "editable": true + }, + "source": [ + "and" + ] + }, + { + "cell_type": "markdown", + "id": "4166af55", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "b^{(2)} \\leftarrow b^{(2)}-\\eta \\delta^{(2)},\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "e5101f70", + "metadata": { + "editable": true + }, + "source": [ + "and" + ] + }, + { + "cell_type": "markdown", + "id": "0c662571", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "w_{ij}^{(1)}\\leftarrow w_{ij}^{(1)}- \\eta \\delta_{i}^{(1)} a_{j}^{(0)},\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "a7e1af99", + "metadata": { + "editable": true + }, + "source": [ + "and" + ] + }, + { + "cell_type": "markdown", + "id": "5a73623e", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "b_{i}^{(1)} \\leftarrow b_{i}^{(1)}-\\eta \\delta_{i}^{(1)},\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "b1d5327b", + "metadata": { + "editable": true + }, + "source": [ + "where $\\eta$ is the learning rate." + ] + }, + { + "cell_type": "markdown", + "id": "5cc5921c", + "metadata": { + "editable": true + }, + "source": [ + "## Setting up the equations for a neural network\n", + "\n", + "The questions we want to ask are how do changes in the biases and the\n", + "weights in our network change the cost function and how can we use the\n", + "final output to modify the weights and biases?\n", + "\n", + "To derive these equations let us start with a plain regression problem\n", + "and define our cost function as" + ] + }, + { + "cell_type": "markdown", + "id": "21633b14", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "{\\cal C}(\\boldsymbol{\\Theta}) = \\frac{1}{2}\\sum_{i=1}^n\\left(y_i - \\tilde{y}_i\\right)^2,\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "171782c4", + "metadata": { + "editable": true + }, + "source": [ + "where the $y_i$s are our $n$ targets (the values we want to\n", + "reproduce), while the outputs of the network after having propagated\n", + "all inputs $\\boldsymbol{x}$ are given by $\\boldsymbol{\\tilde{y}}_i$." + ] + }, + { + "cell_type": "markdown", + "id": "830e0cc3", + "metadata": { + "editable": true + }, + "source": [ + "## Layout of a neural network with three hidden layers (last layer = $l=L=4$, first layer $l=0$)\n", + "\n", + "\n", + "\n", + "\n", + "

    Figure 1:

    \n", + "" + ] + }, + { + "cell_type": "markdown", + "id": "3de97402", + "metadata": { + "editable": true + }, + "source": [ + "## Definitions\n", + "\n", + "With our definition of the targets $\\boldsymbol{y}$, the outputs of the\n", + "network $\\boldsymbol{\\tilde{y}}$ and the inputs $\\boldsymbol{x}$ we\n", + "define now the activation $z_j^l$ of node/neuron/unit $j$ of the\n", + "$l$-th layer as a function of the bias, the weights which add up from\n", + "the previous layer $l-1$ and the forward passes/outputs\n", + "$\\boldsymbol{a}^{l-1}$ from the previous layer as" + ] + }, + { + "cell_type": "markdown", + "id": "f4a20e55", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "z_j^l = \\sum_{i=1}^{M_{l-1}}w_{ij}^la_i^{l-1}+b_j^l,\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "e373e6cc", + "metadata": { + "editable": true + }, + "source": [ + "where $b_k^l$ are the biases from layer $l$. Here $M_{l-1}$\n", + "represents the total number of nodes/neurons/units of layer $l-1$. The\n", + "figure in the whiteboard notes illustrates this equation. We can rewrite this in a more\n", + "compact form as the matrix-vector products we discussed earlier," + ] + }, + { + "cell_type": "markdown", + "id": "a9fa6a69", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\boldsymbol{z}^l = \\left(\\boldsymbol{W}^l\\right)^T\\boldsymbol{a}^{l-1}+\\boldsymbol{b}^l.\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "f1730e5a", + "metadata": { + "editable": true + }, + "source": [ + "## Inputs to the activation function\n", + "\n", + "With the activation values $\\boldsymbol{z}^l$ we can in turn define the\n", + "output of layer $l$ as $\\boldsymbol{a}^l = \\sigma(\\boldsymbol{z}^l)$ where $\\sigma$ is our\n", + "activation function. In the examples here we will use the sigmoid\n", + "function discussed in our logistic regression lectures. We will also use the same activation function $\\sigma$ for all layers\n", + "and their nodes. It means we have" + ] + }, + { + "cell_type": "markdown", + "id": "59d7d98a", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "a_j^l = \\sigma(z_j^l) = \\frac{1}{1+\\exp{-(z_j^l)}}.\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "5a3d0931", + "metadata": { + "editable": true + }, + "source": [ + "## Layout of input to first hidden layer $l=1$ from input layer $l=0$\n", + "\n", + "\n", + "\n", + "\n", + "

    Figure 1:

    \n", + "" + ] + }, + { + "cell_type": "markdown", + "id": "c56d749e", + "metadata": { + "editable": true + }, + "source": [ + "## Derivatives and the chain rule\n", + "\n", + "From the definition of the input variable to the activation function, that is $z_j^l$ we have" + ] + }, + { + "cell_type": "markdown", + "id": "f3dd37e7", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\frac{\\partial z_j^l}{\\partial w_{ij}^l} = a_i^{l-1},\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "7654c47c", + "metadata": { + "editable": true + }, + "source": [ + "and" + ] + }, + { + "cell_type": "markdown", + "id": "cb1aa2a9", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\frac{\\partial z_j^l}{\\partial a_i^{l-1}} = w_{ji}^l.\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "11607454", + "metadata": { + "editable": true + }, + "source": [ + "With our definition of the activation function we have that (note that this function depends only on $z_j^l$)" + ] + }, + { + "cell_type": "markdown", + "id": "29a881cd", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\frac{\\partial a_j^l}{\\partial z_j^{l}} = a_j^l(1-a_j^l)=\\sigma(z_j^l)(1-\\sigma(z_j^l)).\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "1244c442", + "metadata": { + "editable": true + }, + "source": [ + "## Derivative of the cost function\n", + "\n", + "With these definitions we can now compute the derivative of the cost function in terms of the weights.\n", + "\n", + "Let us specialize to the output layer $l=L$. Our cost function is" + ] + }, + { + "cell_type": "markdown", + "id": "7d43f636", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "{\\cal C}(\\boldsymbol{\\Theta}^L) = \\frac{1}{2}\\sum_{i=1}^n\\left(y_i - \\tilde{y}_i\\right)^2=\\frac{1}{2}\\sum_{i=1}^n\\left(a_i^L - y_i\\right)^2,\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "c23708ab", + "metadata": { + "editable": true + }, + "source": [ + "The derivative of this function with respect to the weights is" + ] + }, + { + "cell_type": "markdown", + "id": "3a3d49b9", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\frac{\\partial{\\cal C}(\\boldsymbol{\\Theta}^L)}{\\partial w_{ij}^L} = \\left(a_j^L - y_j\\right)\\frac{\\partial a_j^L}{\\partial w_{ij}^{L}},\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "31b8b489", + "metadata": { + "editable": true + }, + "source": [ + "The last partial derivative can easily be computed and reads (by applying the chain rule)" + ] + }, + { + "cell_type": "markdown", + "id": "2cb11e87", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\frac{\\partial a_j^L}{\\partial w_{ij}^{L}} = \\frac{\\partial a_j^L}{\\partial z_{j}^{L}}\\frac{\\partial z_j^L}{\\partial w_{ij}^{L}}=a_j^L(1-a_j^L)a_i^{L-1}.\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "85b6783b", + "metadata": { + "editable": true + }, + "source": [ + "## The back propagation equations for a neural network\n", + "\n", + "We have thus" + ] + }, + { + "cell_type": "markdown", + "id": "1d9102e8", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\frac{\\partial{\\cal C}((\\boldsymbol{\\Theta}^L)}{\\partial w_{ij}^L} = \\left(a_j^L - y_j\\right)a_j^L(1-a_j^L)a_i^{L-1},\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "92547c80", + "metadata": { + "editable": true + }, + "source": [ + "Defining" + ] + }, + { + "cell_type": "markdown", + "id": "e06d7f47", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\delta_j^L = a_j^L(1-a_j^L)\\left(a_j^L - y_j\\right) = \\sigma'(z_j^L)\\frac{\\partial {\\cal C}}{\\partial (a_j^L)},\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "a15b2c15", + "metadata": { + "editable": true + }, + "source": [ + "and using the Hadamard product of two vectors we can write this as" + ] + }, + { + "cell_type": "markdown", + "id": "69a96bb6", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\boldsymbol{\\delta}^L = \\sigma'(\\boldsymbol{z}^L)\\circ\\frac{\\partial {\\cal C}}{\\partial (\\boldsymbol{a}^L)}.\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "6954d50b", + "metadata": { + "editable": true + }, + "source": [ + "## Analyzing the last results\n", + "\n", + "This is an important expression. The second term on the right handside\n", + "measures how fast the cost function is changing as a function of the $j$th\n", + "output activation. If, for example, the cost function doesn't depend\n", + "much on a particular output node $j$, then $\\delta_j^L$ will be small,\n", + "which is what we would expect. The first term on the right, measures\n", + "how fast the activation function $f$ is changing at a given activation\n", + "value $z_j^L$." + ] + }, + { + "cell_type": "markdown", + "id": "a8d2b720", + "metadata": { + "editable": true + }, + "source": [ + "## More considerations\n", + "\n", + "Notice that everything in the above equations is easily computed. In\n", + "particular, we compute $z_j^L$ while computing the behaviour of the\n", + "network, and it is only a small additional overhead to compute\n", + "$\\sigma'(z^L_j)$. The exact form of the derivative with respect to the\n", + "output depends on the form of the cost function.\n", + "However, provided the cost function is known there should be little\n", + "trouble in calculating" + ] + }, + { + "cell_type": "markdown", + "id": "b40feaea", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\frac{\\partial {\\cal C}}{\\partial (a_j^L)}\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "04d0916c", + "metadata": { + "editable": true + }, + "source": [ + "With the definition of $\\delta_j^L$ we have a more compact definition of the derivative of the cost function in terms of the weights, namely" + ] + }, + { + "cell_type": "markdown", + "id": "4067515f", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\frac{\\partial{\\cal C}}{\\partial w_{ij}^L} = \\delta_j^La_i^{L-1}.\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "572f5043", + "metadata": { + "editable": true + }, + "source": [ + "## Derivatives in terms of $z_j^L$\n", + "\n", + "It is also easy to see that our previous equation can be written as" + ] + }, + { + "cell_type": "markdown", + "id": "4272e56c", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\delta_j^L =\\frac{\\partial {\\cal C}}{\\partial z_j^L}= \\frac{\\partial {\\cal C}}{\\partial a_j^L}\\frac{\\partial a_j^L}{\\partial z_j^L},\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "3ffcc3a4", + "metadata": { + "editable": true + }, + "source": [ + "which can also be interpreted as the partial derivative of the cost function with respect to the biases $b_j^L$, namely" + ] + }, + { + "cell_type": "markdown", + "id": "87dc5557", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\delta_j^L = \\frac{\\partial {\\cal C}}{\\partial b_j^L}\\frac{\\partial b_j^L}{\\partial z_j^L}=\\frac{\\partial {\\cal C}}{\\partial b_j^L},\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "36f9508b", + "metadata": { + "editable": true + }, + "source": [ + "That is, the error $\\delta_j^L$ is exactly equal to the rate of change of the cost function as a function of the bias." + ] + }, + { + "cell_type": "markdown", + "id": "5e7f7ab6", + "metadata": { + "editable": true + }, + "source": [ + "## Bringing it together\n", + "\n", + "We have now three equations that are essential for the computations of the derivatives of the cost function at the output layer. These equations are needed to start the algorithm and they are" + ] + }, + { + "cell_type": "markdown", + "id": "b5caac62", + "metadata": { + "editable": true + }, + "source": [ + "\n", + "
    \n", + "\n", + "$$\n", + "\\begin{equation}\n", + "\\frac{\\partial{\\cal C}(\\boldsymbol{W^L})}{\\partial w_{ij}^L} = \\delta_j^La_i^{L-1},\n", + "\\label{_auto1} \\tag{1}\n", + "\\end{equation}\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "beffe853", + "metadata": { + "editable": true + }, + "source": [ + "and" + ] + }, + { + "cell_type": "markdown", + "id": "9ff991bb", + "metadata": { + "editable": true + }, + "source": [ + "\n", + "
    \n", + "\n", + "$$\n", + "\\begin{equation}\n", + "\\delta_j^L = \\sigma'(z_j^L)\\frac{\\partial {\\cal C}}{\\partial (a_j^L)},\n", + "\\label{_auto2} \\tag{2}\n", + "\\end{equation}\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "5c75d805", + "metadata": { + "editable": true + }, + "source": [ + "and" + ] + }, + { + "cell_type": "markdown", + "id": "df242f2d", + "metadata": { + "editable": true + }, + "source": [ + "\n", + "
    \n", + "\n", + "$$\n", + "\\begin{equation}\n", + "\\delta_j^L = \\frac{\\partial {\\cal C}}{\\partial b_j^L},\n", + "\\label{_auto3} \\tag{3}\n", + "\\end{equation}\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "18a65420", + "metadata": { + "editable": true + }, + "source": [ + "## Final back propagating equation\n", + "\n", + "We have that (replacing $L$ with a general layer $l$)" + ] + }, + { + "cell_type": "markdown", + "id": "04cb64d1", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\delta_j^l =\\frac{\\partial {\\cal C}}{\\partial z_j^l}.\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "c5320606", + "metadata": { + "editable": true + }, + "source": [ + "We want to express this in terms of the equations for layer $l+1$." + ] + }, + { + "cell_type": "markdown", + "id": "58b67295", + "metadata": { + "editable": true + }, + "source": [ + "## Using the chain rule and summing over all $k$ entries\n", + "\n", + "We obtain" + ] + }, + { + "cell_type": "markdown", + "id": "3b7c74a4", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\delta_j^l =\\sum_k \\frac{\\partial {\\cal C}}{\\partial z_k^{l+1}}\\frac{\\partial z_k^{l+1}}{\\partial z_j^{l}}=\\sum_k \\delta_k^{l+1}\\frac{\\partial z_k^{l+1}}{\\partial z_j^{l}},\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "76ff59e0", + "metadata": { + "editable": true + }, + "source": [ + "and recalling that" + ] + }, + { + "cell_type": "markdown", + "id": "60a7be03", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "z_j^{l+1} = \\sum_{i=1}^{M_{l}}w_{ij}^{l+1}a_i^{l}+b_j^{l+1},\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "27739d6d", + "metadata": { + "editable": true + }, + "source": [ + "with $M_l$ being the number of nodes in layer $l$, we obtain" + ] + }, + { + "cell_type": "markdown", + "id": "1efdb6f5", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\delta_j^l =\\sum_k \\delta_k^{l+1}w_{kj}^{l+1}\\sigma'(z_j^l),\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "b0b04f61", + "metadata": { + "editable": true + }, + "source": [ + "This is our final equation.\n", + "\n", + "We are now ready to set up the algorithm for back propagation and learning the weights and biases." + ] + }, + { + "cell_type": "markdown", + "id": "e6c4318c", + "metadata": { + "editable": true + }, + "source": [ + "## Setting up the back propagation algorithm and algorithm for a feed forward NN, initalizations\n", + "\n", + "**The architecture (our model).**\n", + "\n", + "1. Set up your inputs and outputs (scalars, vectors, matrices or higher-order arrays)\n", + "\n", + "2. Define the number of hidden layers and hidden nodes\n", + "\n", + "3. Define activation functions for hidden layers and output layers\n", + "\n", + "4. Define optimizer (plan learning rate, momentum, ADAgrad, RMSprop, ADAM etc) and array of initial learning rates\n", + "\n", + "5. Define cost function and possible regularization terms with hyperparameters\n", + "\n", + "6. Initialize weights and biases\n", + "\n", + "7. Fix number of iterations for the feed forward part and back propagation part" + ] + }, + { + "cell_type": "markdown", + "id": "4e58634a", + "metadata": { + "editable": true + }, + "source": [ + "## Setting up the back propagation algorithm, part 1\n", + "\n", + "The four equations provide us with a way of computing the gradients of the cost function. Let us write this out in the form of an algorithm.\n", + "\n", + "**First**, we set up the input data $\\boldsymbol{x}$ and the activations\n", + "$\\boldsymbol{z}_1$ of the input layer and compute the activation function and\n", + "the pertinent outputs $\\boldsymbol{a}^1$.\n", + "\n", + "**Secondly**, we perform then the feed forward till we reach the output\n", + "layer and compute all $\\boldsymbol{z}_l$ of the input layer and compute the\n", + "activation function and the pertinent outputs $\\boldsymbol{a}^l$ for\n", + "$l=1,2,3,\\dots,L$.\n", + "\n", + "**Notation**: The first hidden layer has $l=1$ as label and the final output layer has $l=L$." + ] + }, + { + "cell_type": "markdown", + "id": "690eb424", + "metadata": { + "editable": true + }, + "source": [ + "## Setting up the back propagation algorithm, part 2\n", + "\n", + "Thereafter we compute the ouput error $\\boldsymbol{\\delta}^L$ by computing all" + ] + }, + { + "cell_type": "markdown", + "id": "5f475d02", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\delta_j^L = \\sigma'(z_j^L)\\frac{\\partial {\\cal C}}{\\partial (a_j^L)}.\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "016ce549", + "metadata": { + "editable": true + }, + "source": [ + "Then we compute the back propagate error for each $l=L-1,L-2,\\dots,1$ as" + ] + }, + { + "cell_type": "markdown", + "id": "51f9ed82", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\delta_j^l = \\sum_k \\delta_k^{l+1}w_{kj}^{l+1}\\sigma'(z_j^l).\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "a8996176", + "metadata": { + "editable": true + }, + "source": [ + "## Setting up the Back propagation algorithm, part 3\n", + "\n", + "Finally, we update the weights and the biases using gradient descent\n", + "for each $l=L-1,L-2,\\dots,1$ (the first hidden layer) and update the weights and biases\n", + "according to the rules" + ] + }, + { + "cell_type": "markdown", + "id": "aaf186a8", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "w_{ij}^l\\leftarrow = w_{ij}^l- \\eta \\delta_j^la_i^{l-1},\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "3b8a0a79", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "b_j^l \\leftarrow b_j^l-\\eta \\frac{\\partial {\\cal C}}{\\partial b_j^l}=b_j^l-\\eta \\delta_j^l,\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "299cc7bb", + "metadata": { + "editable": true + }, + "source": [ + "with $\\eta$ being the learning rate." + ] + }, + { + "cell_type": "markdown", + "id": "3b10a1a0", + "metadata": { + "editable": true + }, + "source": [ + "## Updating the gradients\n", + "\n", + "With the back propagate error for each $l=L-1,L-2,\\dots,1$ as" + ] + }, + { + "cell_type": "markdown", + "id": "0f8769f8", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\delta_j^l = \\sum_k \\delta_k^{l+1}w_{kj}^{l+1}\\sigma'(z_j^l),\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "03860e2f", + "metadata": { + "editable": true + }, + "source": [ + "we update the weights and the biases using gradient descent for each $l=L-1,L-2,\\dots,1$ and update the weights and biases according to the rules" + ] + }, + { + "cell_type": "markdown", + "id": "00cea1da", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "w_{ij}^l\\leftarrow = w_{ij}^l- \\eta \\delta_j^la_i^{l-1},\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "fe5bbd2a", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "b_j^l \\leftarrow b_j^l-\\eta \\frac{\\partial {\\cal C}}{\\partial b_j^l}=b_j^l-\\eta \\delta_j^l,\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "e791fb56", + "metadata": { + "editable": true + }, + "source": [ + "## Activation functions\n", + "\n", + "A property that characterizes a neural network, other than its\n", + "connectivity, is the choice of activation function(s). The following\n", + "restrictions are imposed on an activation function for an FFNN to\n", + "fulfill the universal approximation theorem\n", + "\n", + " * Non-constant\n", + "\n", + " * Bounded\n", + "\n", + " * Monotonically-increasing\n", + "\n", + " * Continuous" + ] + }, + { + "cell_type": "markdown", + "id": "14ed4124", + "metadata": { + "editable": true + }, + "source": [ + "### Activation functions, Logistic and Hyperbolic ones\n", + "\n", + "The second requirement excludes all linear functions. Furthermore, in\n", + "a MLP with only linear activation functions, each layer simply\n", + "performs a linear transformation of its inputs.\n", + "\n", + "Regardless of the number of layers, the output of the NN will be\n", + "nothing but a linear function of the inputs. Thus we need to introduce\n", + "some kind of non-linearity to the NN to be able to fit non-linear\n", + "functions Typical examples are the logistic *Sigmoid*" + ] + }, + { + "cell_type": "markdown", + "id": "d6b31eb8", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\sigma(x) = \\frac{1}{1 + e^{-x}},\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "2be10af5", + "metadata": { + "editable": true + }, + "source": [ + "and the *hyperbolic tangent* function" + ] + }, + { + "cell_type": "markdown", + "id": "f45b9a99", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\sigma(x) = \\tanh(x)\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "865d802b", + "metadata": { + "editable": true + }, + "source": [ + "## Relevance\n", + "\n", + "The *sigmoid* function are more biologically plausible because the\n", + "output of inactive neurons are zero. Such activation function are\n", + "called *one-sided*. However, it has been shown that the hyperbolic\n", + "tangent performs better than the sigmoid for training MLPs. has\n", + "become the most popular for *deep neural networks*" + ] + }, + { + "cell_type": "code", + "execution_count": 2, + "id": "37773874", + "metadata": { + "collapsed": false, + "editable": true + }, + "outputs": [], + "source": [ + "%matplotlib inline\n", + "\n", + "\"\"\"The sigmoid function (or the logistic curve) is a \n", + "function that takes any real number, z, and outputs a number (0,1).\n", + "It is useful in neural networks for assigning weights on a relative scale.\n", + "The value z is the weighted sum of parameters involved in the learning algorithm.\"\"\"\n", + "\n", + "import numpy\n", + "import matplotlib.pyplot as plt\n", + "import math as mt\n", + "\n", + "z = numpy.arange(-5, 5, .1)\n", + "sigma_fn = numpy.vectorize(lambda z: 1/(1+numpy.exp(-z)))\n", + "sigma = sigma_fn(z)\n", + "\n", + "fig = plt.figure()\n", + "ax = fig.add_subplot(111)\n", + "ax.plot(z, sigma)\n", + "ax.set_ylim([-0.1, 1.1])\n", + "ax.set_xlim([-5,5])\n", + "ax.grid(True)\n", + "ax.set_xlabel('z')\n", + "ax.set_title('sigmoid function')\n", + "\n", + "plt.show()\n", + "\n", + "\"\"\"Step Function\"\"\"\n", + "z = numpy.arange(-5, 5, .02)\n", + "step_fn = numpy.vectorize(lambda z: 1.0 if z >= 0.0 else 0.0)\n", + "step = step_fn(z)\n", + "\n", + "fig = plt.figure()\n", + "ax = fig.add_subplot(111)\n", + "ax.plot(z, step)\n", + "ax.set_ylim([-0.5, 1.5])\n", + "ax.set_xlim([-5,5])\n", + "ax.grid(True)\n", + "ax.set_xlabel('z')\n", + "ax.set_title('step function')\n", + "\n", + "plt.show()\n", + "\n", + "\"\"\"Sine Function\"\"\"\n", + "z = numpy.arange(-2*mt.pi, 2*mt.pi, 0.1)\n", + "t = numpy.sin(z)\n", + "\n", + "fig = plt.figure()\n", + "ax = fig.add_subplot(111)\n", + "ax.plot(z, t)\n", + "ax.set_ylim([-1.0, 1.0])\n", + "ax.set_xlim([-2*mt.pi,2*mt.pi])\n", + "ax.grid(True)\n", + "ax.set_xlabel('z')\n", + "ax.set_title('sine function')\n", + "\n", + "plt.show()\n", + "\n", + "\"\"\"Plots a graph of the squashing function used by a rectified linear\n", + "unit\"\"\"\n", + "z = numpy.arange(-2, 2, .1)\n", + "zero = numpy.zeros(len(z))\n", + "y = numpy.max([zero, z], axis=0)\n", + "\n", + "fig = plt.figure()\n", + "ax = fig.add_subplot(111)\n", + "ax.plot(z, y)\n", + "ax.set_ylim([-2.0, 2.0])\n", + "ax.set_xlim([-2.0, 2.0])\n", + "ax.grid(True)\n", + "ax.set_xlabel('z')\n", + "ax.set_title('Rectified linear unit')\n", + "\n", + "plt.show()" + ] + }, + { + "cell_type": "markdown", + "id": "9cccb9ca", + "metadata": { + "editable": true + }, + "source": [ + "## Vanishing gradients\n", + "\n", + "The Back propagation algorithm we derived above works by going from\n", + "the output layer to the input layer, propagating the error gradient on\n", + "the way. Once the algorithm has computed the gradient of the cost\n", + "function with regards to each parameter in the network, it uses these\n", + "gradients to update each parameter with a Gradient Descent (GD) step.\n", + "\n", + "Unfortunately for us, the gradients often get smaller and smaller as\n", + "the algorithm progresses down to the first hidden layers. As a result,\n", + "the GD update leaves the lower layer connection weights virtually\n", + "unchanged, and training never converges to a good solution. This is\n", + "known in the literature as **the vanishing gradients problem**." + ] + }, + { + "cell_type": "markdown", + "id": "a42f7226", + "metadata": { + "editable": true + }, + "source": [ + "## Exploding gradients\n", + "\n", + "In other cases, the opposite can happen, namely the the gradients can\n", + "grow bigger and bigger. The result is that many of the layers get\n", + "large updates of the weights the algorithm diverges. This is the\n", + "**exploding gradients problem**, which is mostly encountered in\n", + "recurrent neural networks. More generally, deep neural networks suffer\n", + "from unstable gradients, different layers may learn at widely\n", + "different speeds" + ] + }, + { + "cell_type": "markdown", + "id": "0f57d23f", + "metadata": { + "editable": true + }, + "source": [ + "## Is the Logistic activation function (Sigmoid) our choice?\n", + "\n", + "Although this unfortunate behavior has been empirically observed for\n", + "quite a while (it was one of the reasons why deep neural networks were\n", + "mostly abandoned for a long time), it is only around 2010 that\n", + "significant progress was made in understanding it.\n", + "\n", + "A paper titled [Understanding the Difficulty of Training Deep\n", + "Feedforward Neural Networks by Xavier Glorot and Yoshua Bengio](http://proceedings.mlr.press/v9/glorot10a.html) found that\n", + "the problems with the popular logistic\n", + "sigmoid activation function and the weight initialization technique\n", + "that was most popular at the time, namely random initialization using\n", + "a normal distribution with a mean of 0 and a standard deviation of\n", + "1." + ] + }, + { + "cell_type": "markdown", + "id": "ad459faf", + "metadata": { + "editable": true + }, + "source": [ + "## Logistic function as the root of problems\n", + "\n", + "They showed that with this activation function and this\n", + "initialization scheme, the variance of the outputs of each layer is\n", + "much greater than the variance of its inputs. Going forward in the\n", + "network, the variance keeps increasing after each layer until the\n", + "activation function saturates at the top layers. This is actually made\n", + "worse by the fact that the logistic function has a mean of 0.5, not 0\n", + "(the hyperbolic tangent function has a mean of 0 and behaves slightly\n", + "better than the logistic function in deep networks)." + ] + }, + { + "cell_type": "markdown", + "id": "1e307870", + "metadata": { + "editable": true + }, + "source": [ + "## The derivative of the Logistic funtion\n", + "\n", + "Looking at the logistic activation function, when inputs become large\n", + "(negative or positive), the function saturates at 0 or 1, with a\n", + "derivative extremely close to 0. Thus when backpropagation kicks in,\n", + "it has virtually no gradient to propagate back through the network,\n", + "and what little gradient exists keeps getting diluted as\n", + "backpropagation progresses down through the top layers, so there is\n", + "really nothing left for the lower layers.\n", + "\n", + "In their paper, Glorot and Bengio propose a way to significantly\n", + "alleviate this problem. We need the signal to flow properly in both\n", + "directions: in the forward direction when making predictions, and in\n", + "the reverse direction when backpropagating gradients. We don’t want\n", + "the signal to die out, nor do we want it to explode and saturate. For\n", + "the signal to flow properly, the authors argue that we need the\n", + "variance of the outputs of each layer to be equal to the variance of\n", + "its inputs, and we also need the gradients to have equal variance\n", + "before and after flowing through a layer in the reverse direction." + ] + }, + { + "cell_type": "markdown", + "id": "06fe5be6", + "metadata": { + "editable": true + }, + "source": [ + "## Insights from the paper by Glorot and Bengio\n", + "\n", + "One of the insights in the 2010 paper by Glorot and Bengio was that\n", + "the vanishing/exploding gradients problems were in part due to a poor\n", + "choice of activation function. Until then most people had assumed that\n", + "if Nature had chosen to use roughly sigmoid activation functions in\n", + "biological neurons, they must be an excellent choice. But it turns out\n", + "that other activation functions behave much better in deep neural\n", + "networks, in particular the ReLU activation function, mostly because\n", + "it does not saturate for positive values (and also because it is quite\n", + "fast to compute)." + ] + }, + { + "cell_type": "markdown", + "id": "736441f3", + "metadata": { + "editable": true + }, + "source": [ + "## The RELU function family\n", + "\n", + "The ReLU activation function suffers from a problem known as the dying\n", + "ReLUs: during training, some neurons effectively die, meaning they\n", + "stop outputting anything other than 0.\n", + "\n", + "In some cases, you may find that half of your network’s neurons are\n", + "dead, especially if you used a large learning rate. During training,\n", + "if a neuron’s weights get updated such that the weighted sum of the\n", + "neuron’s inputs is negative, it will start outputting 0. When this\n", + "happen, the neuron is unlikely to come back to life since the gradient\n", + "of the ReLU function is 0 when its input is negative." + ] + }, + { + "cell_type": "markdown", + "id": "9ac6f21d", + "metadata": { + "editable": true + }, + "source": [ + "## ELU function\n", + "\n", + "To solve this problem, nowadays practitioners use a variant of the\n", + "ReLU function, such as the leaky ReLU discussed above or the so-called\n", + "exponential linear unit (ELU) function" + ] + }, + { + "cell_type": "markdown", + "id": "80ebd77c", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "ELU(z) = \\left\\{\\begin{array}{cc} \\alpha\\left( \\exp{(z)}-1\\right) & z < 0,\\\\ z & z \\ge 0.\\end{array}\\right.\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "2566a903", + "metadata": { + "editable": true + }, + "source": [ + "## Which activation function should we use?\n", + "\n", + "In general it seems that the ELU activation function is better than\n", + "the leaky ReLU function (and its variants), which is better than\n", + "ReLU. ReLU performs better than $\\tanh$ which in turn performs better\n", + "than the logistic function.\n", + "\n", + "If runtime performance is an issue, then you may opt for the leaky\n", + "ReLU function over the ELU function If you don’t want to tweak yet\n", + "another hyperparameter, you may just use the default $\\alpha$ of\n", + "$0.01$ for the leaky ReLU, and $1$ for ELU. If you have spare time and\n", + "computing power, you can use cross-validation or bootstrap to evaluate\n", + "other activation functions." + ] + }, + { + "cell_type": "markdown", + "id": "382f5fe1", + "metadata": { + "editable": true + }, + "source": [ + "## More on activation functions, output layers\n", + "\n", + "In most cases you can use the ReLU activation function in the hidden\n", + "layers (or one of its variants).\n", + "\n", + "It is a bit faster to compute than other activation functions, and the\n", + "gradient descent optimization does in general not get stuck.\n", + "\n", + "**For the output layer:**\n", + "\n", + "* For classification the softmax activation function is generally a good choice for classification tasks (when the classes are mutually exclusive).\n", + "\n", + "* For regression tasks, you can simply use no activation function at all." + ] + }, + { + "cell_type": "markdown", + "id": "fed07226", + "metadata": { + "editable": true + }, + "source": [ + "## Fine-tuning neural network hyperparameters\n", + "\n", + "The flexibility of neural networks is also one of their main\n", + "drawbacks: there are many hyperparameters to tweak. Not only can you\n", + "use any imaginable network topology (how neurons/nodes are\n", + "interconnected), but even in a simple FFNN you can change the number\n", + "of layers, the number of neurons per layer, the type of activation\n", + "function to use in each layer, the weight initialization logic, the\n", + "stochastic gradient optmized and much more. How do you know what\n", + "combination of hyperparameters is the best for your task?\n", + "\n", + "* You can use grid search with cross-validation to find the right hyperparameters.\n", + "\n", + "However,since there are many hyperparameters to tune, and since\n", + "training a neural network on a large dataset takes a lot of time, you\n", + "will only be able to explore a tiny part of the hyperparameter space.\n", + "\n", + "* You can use randomized search.\n", + "\n", + "* Or use tools like [Oscar](http://oscar.calldesk.ai/), which implements more complex algorithms to help you find a good set of hyperparameters quickly." + ] + }, + { + "cell_type": "markdown", + "id": "9ddc2586", + "metadata": { + "editable": true + }, + "source": [ + "## Hidden layers\n", + "\n", + "For many problems you can start with just one or two hidden layers and\n", + "it will work just fine. For the MNIST data set discussed below you can easily get a\n", + "high accuracy using just one hidden layer with a few hundred neurons.\n", + "You can reach for this data set above 98% accuracy using two hidden\n", + "layers with the same total amount of neurons, in roughly the same\n", + "amount of training time.\n", + "\n", + "For more complex problems, you can gradually ramp up the number of\n", + "hidden layers, until you start overfitting the training set. Very\n", + "complex tasks, such as large image classification or speech\n", + "recognition, typically require networks with dozens of layers and they\n", + "need a huge amount of training data. However, you will rarely have to\n", + "train such networks from scratch: it is much more common to reuse\n", + "parts of a pretrained state-of-the-art network that performs a similar\n", + "task." + ] + }, + { + "cell_type": "markdown", + "id": "d327b303", + "metadata": { + "editable": true + }, + "source": [ + "## Batch Normalization\n", + "\n", + "Batch Normalization aims to address the vanishing/exploding gradients\n", + "problems, and more generally the problem that the distribution of each\n", + "layer’s inputs changes during training, as the parameters of the\n", + "previous layers change.\n", + "\n", + "The technique consists of adding an operation in the model just before\n", + "the activation function of each layer, simply zero-centering and\n", + "normalizing the inputs, then scaling and shifting the result using two\n", + "new parameters per layer (one for scaling, the other for shifting). In\n", + "other words, this operation lets the model learn the optimal scale and\n", + "mean of the inputs for each layer. In order to zero-center and\n", + "normalize the inputs, the algorithm needs to estimate the inputs’ mean\n", + "and standard deviation. It does so by evaluating the mean and standard\n", + "deviation of the inputs over the current mini-batch, from this the\n", + "name batch normalization." + ] + }, + { + "cell_type": "markdown", + "id": "a3f4a57d", + "metadata": { + "editable": true + }, + "source": [ + "## Dropout\n", + "\n", + "It is a fairly simple algorithm: at every training step, every neuron\n", + "(including the input neurons but excluding the output neurons) has a\n", + "probability $p$ of being temporarily dropped out, meaning it will be\n", + "entirely ignored during this training step, but it may be active\n", + "during the next step.\n", + "\n", + "The hyperparameter $p$ is called the dropout rate, and it is typically\n", + "set to 50%. After training, the neurons are not dropped anymore. It\n", + "is viewed as one of the most popular regularization techniques." + ] + }, + { + "cell_type": "markdown", + "id": "da0c8dbe", + "metadata": { + "editable": true + }, + "source": [ + "## Gradient Clipping\n", + "\n", + "A popular technique to lessen the exploding gradients problem is to\n", + "simply clip the gradients during backpropagation so that they never\n", + "exceed some threshold (this is mostly useful for recurrent neural\n", + "networks).\n", + "\n", + "This technique is called Gradient Clipping.\n", + "\n", + "In general however, Batch\n", + "Normalization is preferred." + ] + }, + { + "cell_type": "markdown", + "id": "60e3da23", + "metadata": { + "editable": true + }, + "source": [ + "## A top-down perspective on Neural networks\n", + "\n", + "The first thing we would like to do is divide the data into two or\n", + "three parts. A training set, a validation or dev (development) set,\n", + "and a test set. The test set is the data on which we want to make\n", + "predictions. The dev set is a subset of the training data we use to\n", + "check how well we are doing out-of-sample, after training the model on\n", + "the training dataset. We use the validation error as a proxy for the\n", + "test error in order to make tweaks to our model. It is crucial that we\n", + "do not use any of the test data to train the algorithm. This is a\n", + "cardinal sin in ML. Then:\n", + "\n", + "1. Estimate optimal error rate\n", + "\n", + "2. Minimize underfitting (bias) on training data set.\n", + "\n", + "3. Make sure you are not overfitting." + ] + }, + { + "cell_type": "markdown", + "id": "17f8f5ed", + "metadata": { + "editable": true + }, + "source": [ + "## More top-down perspectives\n", + "\n", + "If the validation and test sets are drawn from the same distributions,\n", + "then a good performance on the validation set should lead to similarly\n", + "good performance on the test set. \n", + "\n", + "However, sometimes\n", + "the training data and test data differ in subtle ways because, for\n", + "example, they are collected using slightly different methods, or\n", + "because it is cheaper to collect data in one way versus another. In\n", + "this case, there can be a mismatch between the training and test\n", + "data. This can lead to the neural network overfitting these small\n", + "differences between the test and training sets, and a poor performance\n", + "on the test set despite having a good performance on the validation\n", + "set. To rectify this, Andrew Ng suggests making two validation or dev\n", + "sets, one constructed from the training data and one constructed from\n", + "the test data. The difference between the performance of the algorithm\n", + "on these two validation sets quantifies the train-test mismatch. This\n", + "can serve as another important diagnostic when using DNNs for\n", + "supervised learning." + ] + }, + { + "cell_type": "markdown", + "id": "06ba3896", + "metadata": { + "editable": true + }, + "source": [ + "## Limitations of supervised learning with deep networks\n", + "\n", + "Like all statistical methods, supervised learning using neural\n", + "networks has important limitations. This is especially important when\n", + "one seeks to apply these methods, especially to physics problems. Like\n", + "all tools, DNNs are not a universal solution. Often, the same or\n", + "better performance on a task can be achieved by using a few\n", + "hand-engineered features (or even a collection of random\n", + "features)." + ] + }, + { + "cell_type": "markdown", + "id": "72bc99e6", + "metadata": { + "editable": true + }, + "source": [ + "## Limitations of NNs\n", + "\n", + "Here we list some of the important limitations of supervised neural network based models. \n", + "\n", + "* **Need labeled data**. All supervised learning methods, DNNs for supervised learning require labeled data. Often, labeled data is harder to acquire than unlabeled data (e.g. one must pay for human experts to label images).\n", + "\n", + "* **Supervised neural networks are extremely data intensive.** DNNs are data hungry. They perform best when data is plentiful. This is doubly so for supervised methods where the data must also be labeled. The utility of DNNs is extremely limited if data is hard to acquire or the datasets are small (hundreds to a few thousand samples). In this case, the performance of other methods that utilize hand-engineered features can exceed that of DNNs." + ] + }, + { + "cell_type": "markdown", + "id": "d5f452a2", + "metadata": { + "editable": true + }, + "source": [ + "## Homogeneous data\n", + "\n", + "* **Homogeneous data.** Almost all DNNs deal with homogeneous data of one type. It is very hard to design architectures that mix and match data types (i.e. some continuous variables, some discrete variables, some time series). In applications beyond images, video, and language, this is often what is required. In contrast, ensemble models like random forests or gradient-boosted trees have no difficulty handling mixed data types." + ] + }, + { + "cell_type": "markdown", + "id": "63889ee1", + "metadata": { + "editable": true + }, + "source": [ + "## More limitations\n", + "\n", + "* **Many problems are not about prediction.** In natural science we are often interested in learning something about the underlying distribution that generates the data. In this case, it is often difficult to cast these ideas in a supervised learning setting. While the problems are related, it is possible to make good predictions with a *wrong* model. The model might or might not be useful for understanding the underlying science.\n", + "\n", + "Some of these remarks are particular to DNNs, others are shared by all supervised learning methods. This motivates the use of unsupervised methods which in part circumvent these problems." + ] + }, + { + "cell_type": "markdown", + "id": "5f47ef0f", + "metadata": { + "editable": true + }, + "source": [ + "## Setting up a Multi-layer perceptron model for classification\n", + "\n", + "We are now gong to develop an example based on the MNIST data\n", + "base. This is a classification problem and we need to use our\n", + "cross-entropy function we discussed in connection with logistic\n", + "regression. The cross-entropy defines our cost function for the\n", + "classificaton problems with neural networks.\n", + "\n", + "In binary classification with two classes $(0, 1)$ we define the\n", + "logistic/sigmoid function as the probability that a particular input\n", + "is in class $0$ or $1$. This is possible because the logistic\n", + "function takes any input from the real numbers and inputs a number\n", + "between 0 and 1, and can therefore be interpreted as a probability. It\n", + "also has other nice properties, such as a derivative that is simple to\n", + "calculate.\n", + "\n", + "For an input $\\boldsymbol{a}$ from the hidden layer, the probability that the input $\\boldsymbol{x}$\n", + "is in class 0 or 1 is just. We let $\\theta$ represent the unknown weights and biases to be adjusted by our equations). The variable $x$\n", + "represents our activation values $z$. We have" + ] + }, + { + "cell_type": "markdown", + "id": "b08a7b7e", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "P(y = 0 \\mid \\boldsymbol{x}, \\boldsymbol{\\theta}) = \\frac{1}{1 + \\exp{(- \\boldsymbol{x}})} ,\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "10220190", + "metadata": { + "editable": true + }, + "source": [ + "and" + ] + }, + { + "cell_type": "markdown", + "id": "0fb4d5ad", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "P(y = 1 \\mid \\boldsymbol{x}, \\boldsymbol{\\theta}) = 1 - P(y = 0 \\mid \\boldsymbol{x}, \\boldsymbol{\\theta}) ,\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "f8aa1bae", + "metadata": { + "editable": true + }, + "source": [ + "where $y \\in \\{0, 1\\}$ and $\\boldsymbol{\\theta}$ represents the weights and biases\n", + "of our network." + ] + }, + { + "cell_type": "markdown", + "id": "17de8e22", + "metadata": { + "editable": true + }, + "source": [ + "## Defining the cost function\n", + "\n", + "Our cost function is given as (see the Logistic regression lectures)" + ] + }, + { + "cell_type": "markdown", + "id": "ca8f4f94", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\mathcal{C}(\\boldsymbol{\\theta}) = - \\ln P(\\mathcal{D} \\mid \\boldsymbol{\\theta}) = - \\sum_{i=1}^n\n", + "y_i \\ln[P(y_i = 0)] + (1 - y_i) \\ln [1 - P(y_i = 0)] = \\sum_{i=1}^n \\mathcal{L}_i(\\boldsymbol{\\theta}) .\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "ca998f7f", + "metadata": { + "editable": true + }, + "source": [ + "This last equality means that we can interpret our *cost* function as a sum over the *loss* function\n", + "for each point in the dataset $\\mathcal{L}_i(\\boldsymbol{\\theta})$. \n", + "The negative sign is just so that we can think about our algorithm as minimizing a positive number, rather\n", + "than maximizing a negative number. \n", + "\n", + "In *multiclass* classification it is common to treat each integer label as a so called *one-hot* vector: \n", + "\n", + "$y = 5 \\quad \\rightarrow \\quad \\boldsymbol{y} = (0, 0, 0, 0, 0, 1, 0, 0, 0, 0) ,$ and\n", + "\n", + "$y = 1 \\quad \\rightarrow \\quad \\boldsymbol{y} = (0, 1, 0, 0, 0, 0, 0, 0, 0, 0) ,$ \n", + "\n", + "i.e. a binary bit string of length $C$, where $C = 10$ is the number of classes in the MNIST dataset (numbers from $0$ to $9$).. \n", + "\n", + "If $\\boldsymbol{x}_i$ is the $i$-th input (image), $y_{ic}$ refers to the $c$-th component of the $i$-th\n", + "output vector $\\boldsymbol{y}_i$. \n", + "The probability of $\\boldsymbol{x}_i$ being in class $c$ will be given by the softmax function:" + ] + }, + { + "cell_type": "markdown", + "id": "525b0573", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "P(y_{ic} = 1 \\mid \\boldsymbol{x}_i, \\boldsymbol{\\theta}) = \\frac{\\exp{((\\boldsymbol{a}_i^{hidden})^T \\boldsymbol{w}_c)}}\n", + "{\\sum_{c'=0}^{C-1} \\exp{((\\boldsymbol{a}_i^{hidden})^T \\boldsymbol{w}_{c'})}} ,\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "a9279995", + "metadata": { + "editable": true + }, + "source": [ + "which reduces to the logistic function in the binary case. \n", + "The likelihood of this $C$-class classifier\n", + "is now given as:" + ] + }, + { + "cell_type": "markdown", + "id": "e6cef751", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "P(\\mathcal{D} \\mid \\boldsymbol{\\theta}) = \\prod_{i=1}^n \\prod_{c=0}^{C-1} [P(y_{ic} = 1)]^{y_{ic}} .\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "0ec11e1d", + "metadata": { + "editable": true + }, + "source": [ + "Again we take the negative log-likelihood to define our cost function:" + ] + }, + { + "cell_type": "markdown", + "id": "e5d6a067", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\mathcal{C}(\\boldsymbol{\\theta}) = - \\log{P(\\mathcal{D} \\mid \\boldsymbol{\\theta})}.\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "add6e597", + "metadata": { + "editable": true + }, + "source": [ + "See the logistic regression lectures for a full definition of the cost function.\n", + "\n", + "The back propagation equations need now only a small change, namely the definition of a new cost function. We are thus ready to use the same equations as before!" + ] + }, + { + "cell_type": "markdown", + "id": "dc7ca42d", + "metadata": { + "editable": true + }, + "source": [ + "## Example: binary classification problem\n", + "\n", + "As an example of the above, relevant for project 2 as well, let us consider a binary class. As discussed in our logistic regression lectures, we defined a cost function in terms of the parameters $\\beta$ as" + ] + }, + { + "cell_type": "markdown", + "id": "80b56aa5", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\mathcal{C}(\\boldsymbol{\\beta}) = - \\sum_{i=1}^n \\left(y_i\\log{p(y_i \\vert x_i,\\boldsymbol{\\beta})}+(1-y_i)\\log{1-p(y_i \\vert x_i,\\boldsymbol{\\beta})}\\right),\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "72a87573", + "metadata": { + "editable": true + }, + "source": [ + "where we had defined the logistic (sigmoid) function" + ] + }, + { + "cell_type": "markdown", + "id": "ff8418ea", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "p(y_i =1\\vert x_i,\\boldsymbol{\\beta})=\\frac{\\exp{(\\beta_0+\\beta_1 x_i)}}{1+\\exp{(\\beta_0+\\beta_1 x_i)}},\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "dd978909", + "metadata": { + "editable": true + }, + "source": [ + "and" + ] + }, + { + "cell_type": "markdown", + "id": "ce5a0650", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "p(y_i =0\\vert x_i,\\boldsymbol{\\beta})=1-p(y_i =1\\vert x_i,\\boldsymbol{\\beta}).\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "dce8478b", + "metadata": { + "editable": true + }, + "source": [ + "The parameters $\\boldsymbol{\\beta}$ were defined using a minimization method like gradient descent or Newton-Raphson's method. \n", + "\n", + "Now we replace $x_i$ with the activation $z_i^l$ for a given layer $l$ and the outputs as $y_i=a_i^l=f(z_i^l)$, with $z_i^l$ now being a function of the weights $w_{ij}^l$ and biases $b_i^l$. \n", + "We have then" + ] + }, + { + "cell_type": "markdown", + "id": "92f7282e", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "a_i^l = y_i = \\frac{\\exp{(z_i^l)}}{1+\\exp{(z_i^l)}},\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "e83bc7a2", + "metadata": { + "editable": true + }, + "source": [ + "with" + ] + }, + { + "cell_type": "markdown", + "id": "04f9d7f2", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "z_i^l = \\sum_{j}w_{ij}^l a_j^{l-1}+b_i^l,\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "c70ab677", + "metadata": { + "editable": true + }, + "source": [ + "where the superscript $l-1$ indicates that these are the outputs from layer $l-1$.\n", + "Our cost function at the final layer $l=L$ is now" + ] + }, + { + "cell_type": "markdown", + "id": "801a6bd2", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\mathcal{C}(\\boldsymbol{W}) = - \\sum_{i=1}^n \\left(t_i\\log{a_i^L}+(1-t_i)\\log{(1-a_i^L)}\\right),\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "38eeee4b", + "metadata": { + "editable": true + }, + "source": [ + "where we have defined the targets $t_i$. The derivatives of the cost function with respect to the output $a_i^L$ are then easily calculated and we get" + ] + }, + { + "cell_type": "markdown", + "id": "d70ddf11", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\frac{\\partial \\mathcal{C}(\\boldsymbol{W})}{\\partial a_i^L} = \\frac{a_i^L-t_i}{a_i^L(1-a_i^L)}.\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "324ee5e0", + "metadata": { + "editable": true + }, + "source": [ + "In case we use another activation function than the logistic one, we need to evaluate other derivatives." + ] + }, + { + "cell_type": "markdown", + "id": "3366df27", + "metadata": { + "editable": true + }, + "source": [ + "## The Softmax function\n", + "In case we employ the more general case given by the Softmax equation, we need to evaluate the derivative of the activation function with respect to the activation $z_i^l$, that is we need" + ] + }, + { + "cell_type": "markdown", + "id": "c9a2bd62", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\frac{\\partial f(z_i^l)}{\\partial w_{jk}^l} =\n", + "\\frac{\\partial f(z_i^l)}{\\partial z_j^l} \\frac{\\partial z_j^l}{\\partial w_{jk}^l}= \\frac{\\partial f(z_i^l)}{\\partial z_j^l}a_k^{l-1}.\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "401183f6", + "metadata": { + "editable": true + }, + "source": [ + "For the Softmax function we have" + ] + }, + { + "cell_type": "markdown", + "id": "d4e6a79e", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "f(z_i^l) = \\frac{\\exp{(z_i^l)}}{\\sum_{m=1}^K\\exp{(z_m^l)}}.\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "61cfab52", + "metadata": { + "editable": true + }, + "source": [ + "Its derivative with respect to $z_j^l$ gives" + ] + }, + { + "cell_type": "markdown", + "id": "fc5cf891", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\frac{\\partial f(z_i^l)}{\\partial z_j^l}= f(z_i^l)\\left(\\delta_{ij}-f(z_j^l)\\right),\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "2e7f3950", + "metadata": { + "editable": true + }, + "source": [ + "which in case of the simply binary model reduces to having $i=j$." + ] + }, + { + "cell_type": "markdown", + "id": "144969f6", + "metadata": { + "editable": true + }, + "source": [ + "## Developing a code for doing neural networks with back propagation\n", + "\n", + "One can identify a set of key steps when using neural networks to solve supervised learning problems: \n", + "\n", + "1. Collect and pre-process data \n", + "\n", + "2. Define model and architecture \n", + "\n", + "3. Choose cost function and optimizer \n", + "\n", + "4. Train the model \n", + "\n", + "5. Evaluate model performance on test data \n", + "\n", + "6. Adjust hyperparameters (if necessary, network architecture)" + ] + }, + { + "cell_type": "markdown", + "id": "b6e503bd", + "metadata": { + "editable": true + }, + "source": [ + "## Collect and pre-process data\n", + "\n", + "Here we will be using the MNIST dataset, which is readily available through the **scikit-learn**\n", + "package. You may also find it for example [here](http://yann.lecun.com/exdb/mnist/). \n", + "The *MNIST* (Modified National Institute of Standards and Technology) database is a large database\n", + "of handwritten digits that is commonly used for training various image processing systems. \n", + "The MNIST dataset consists of 70 000 images of size $28\\times 28$ pixels, each labeled from 0 to 9. \n", + "The scikit-learn dataset we will use consists of a selection of 1797 images of size $8\\times 8$ collected and processed from this database. \n", + "\n", + "To feed data into a feed-forward neural network we need to represent\n", + "the inputs as a design/feature matrix $X = (n_{inputs}, n_{features})$. Each\n", + "row represents an *input*, in this case a handwritten digit, and\n", + "each column represents a *feature*, in this case a pixel. The\n", + "correct answers, also known as *labels* or *targets* are\n", + "represented as a 1D array of integers \n", + "$Y = (n_{inputs}) = (5, 3, 1, 8,...)$.\n", + "\n", + "As an example, say we want to build a neural network using supervised learning to predict Body-Mass Index (BMI) from\n", + "measurements of height (in m) \n", + "and weight (in kg). If we have measurements of 5 people the design/feature matrix could be for example: \n", + "\n", + "$$ X = \\begin{bmatrix}\n", + "1.85 & 81\\\\\n", + "1.71 & 65\\\\\n", + "1.95 & 103\\\\\n", + "1.55 & 42\\\\\n", + "1.63 & 56\n", + "\\end{bmatrix} ,$$ \n", + "\n", + "and the targets would be: \n", + "\n", + "$$ Y = (23.7, 22.2, 27.1, 17.5, 21.1) $$ \n", + "\n", + "Since each input image is a 2D matrix, we need to flatten the image\n", + "(i.e. \"unravel\" the 2D matrix into a 1D array) to turn the data into a\n", + "design/feature matrix. This means we lose all spatial information in the\n", + "image, such as locality and translational invariance. More complicated\n", + "architectures such as Convolutional Neural Networks can take advantage\n", + "of such information, and are most commonly applied when analyzing\n", + "images." + ] + }, + { + "cell_type": "code", + "execution_count": 3, + "id": "54230ef9", + "metadata": { + "collapsed": false, + "editable": true + }, + "outputs": [], + "source": [ + "# import necessary packages\n", + "import numpy as np\n", + "import matplotlib.pyplot as plt\n", + "from sklearn import datasets\n", + "\n", + "\n", + "# ensure the same random numbers appear every time\n", + "np.random.seed(0)\n", + "\n", + "# display images in notebook\n", + "%matplotlib inline\n", + "plt.rcParams['figure.figsize'] = (12,12)\n", + "\n", + "\n", + "# download MNIST dataset\n", + "digits = datasets.load_digits()\n", + "\n", + "# define inputs and labels\n", + "inputs = digits.images\n", + "labels = digits.target\n", + "\n", + "print(\"inputs = (n_inputs, pixel_width, pixel_height) = \" + str(inputs.shape))\n", + "print(\"labels = (n_inputs) = \" + str(labels.shape))\n", + "\n", + "\n", + "# flatten the image\n", + "# the value -1 means dimension is inferred from the remaining dimensions: 8x8 = 64\n", + "n_inputs = len(inputs)\n", + "inputs = inputs.reshape(n_inputs, -1)\n", + "print(\"X = (n_inputs, n_features) = \" + str(inputs.shape))\n", + "\n", + "\n", + "# choose some random images to display\n", + "indices = np.arange(n_inputs)\n", + "random_indices = np.random.choice(indices, size=5)\n", + "\n", + "for i, image in enumerate(digits.images[random_indices]):\n", + " plt.subplot(1, 5, i+1)\n", + " plt.axis('off')\n", + " plt.imshow(image, cmap=plt.cm.gray_r, interpolation='nearest')\n", + " plt.title(\"Label: %d\" % digits.target[random_indices[i]])\n", + "plt.show()" + ] + }, + { + "cell_type": "markdown", + "id": "6738e7b4", + "metadata": { + "editable": true + }, + "source": [ + "## Train and test datasets\n", + "\n", + "Performing analysis before partitioning the dataset is a major error, that can lead to incorrect conclusions. \n", + "\n", + "We will reserve $80 \\%$ of our dataset for training and $20 \\%$ for testing. \n", + "\n", + "It is important that the train and test datasets are drawn randomly from our dataset, to ensure\n", + "no bias in the sampling. \n", + "Say you are taking measurements of weather data to predict the weather in the coming 5 days.\n", + "You don't want to train your model on measurements taken from the hours 00.00 to 12.00, and then test it on data\n", + "collected from 12.00 to 24.00." + ] + }, + { + "cell_type": "code", + "execution_count": 4, + "id": "f369aa9d", + "metadata": { + "collapsed": false, + "editable": true + }, + "outputs": [], + "source": [ + "from sklearn.model_selection import train_test_split\n", + "\n", + "# one-liner from scikit-learn library\n", + "train_size = 0.8\n", + "test_size = 1 - train_size\n", + "X_train, X_test, Y_train, Y_test = train_test_split(inputs, labels, train_size=train_size,\n", + " test_size=test_size)\n", + "\n", + "# equivalently in numpy\n", + "def train_test_split_numpy(inputs, labels, train_size, test_size):\n", + " n_inputs = len(inputs)\n", + " inputs_shuffled = inputs.copy()\n", + " labels_shuffled = labels.copy()\n", + " \n", + " np.random.shuffle(inputs_shuffled)\n", + " np.random.shuffle(labels_shuffled)\n", + " \n", + " train_end = int(n_inputs*train_size)\n", + " X_train, X_test = inputs_shuffled[:train_end], inputs_shuffled[train_end:]\n", + " Y_train, Y_test = labels_shuffled[:train_end], labels_shuffled[train_end:]\n", + " \n", + " return X_train, X_test, Y_train, Y_test\n", + "\n", + "#X_train, X_test, Y_train, Y_test = train_test_split_numpy(inputs, labels, train_size, test_size)\n", + "\n", + "print(\"Number of training images: \" + str(len(X_train)))\n", + "print(\"Number of test images: \" + str(len(X_test)))" + ] + }, + { + "cell_type": "markdown", + "id": "f8bf8d16", + "metadata": { + "editable": true + }, + "source": [ + "## Define model and architecture\n", + "\n", + "Our simple feed-forward neural network will consist of an *input* layer, a single *hidden* layer and an *output* layer. The activation $y$ of each neuron is a weighted sum of inputs, passed through an activation function. In case of the simple perceptron model we have \n", + "\n", + "$$ z = \\sum_{i=1}^n w_i a_i ,$$\n", + "\n", + "$$ y = f(z) ,$$\n", + "\n", + "where $f$ is the activation function, $a_i$ represents input from neuron $i$ in the preceding layer\n", + "and $w_i$ is the weight to input $i$. \n", + "The activation of the neurons in the input layer is just the features (e.g. a pixel value). \n", + "\n", + "The simplest activation function for a neuron is the *Heaviside* function:\n", + "\n", + "$$ f(z) = \n", + "\\begin{cases}\n", + "1, & z > 0\\\\\n", + "0, & \\text{otherwise}\n", + "\\end{cases}\n", + "$$\n", + "\n", + "A feed-forward neural network with this activation is known as a *perceptron*. \n", + "For a binary classifier (i.e. two classes, 0 or 1, dog or not-dog) we can also use this in our output layer. \n", + "This activation can be generalized to $k$ classes (using e.g. the *one-against-all* strategy), \n", + "and we call these architectures *multiclass perceptrons*. \n", + "\n", + "However, it is now common to use the terms Single Layer Perceptron (SLP) (1 hidden layer) and \n", + "Multilayer Perceptron (MLP) (2 or more hidden layers) to refer to feed-forward neural networks with any activation function. \n", + "\n", + "Typical choices for activation functions include the sigmoid function, hyperbolic tangent, and Rectified Linear Unit (ReLU). \n", + "We will be using the sigmoid function $\\sigma(x)$: \n", + "\n", + "$$ f(x) = \\sigma(x) = \\frac{1}{1 + e^{-x}} ,$$\n", + "\n", + "which is inspired by probability theory (see logistic regression) and was most commonly used until about 2011. See the discussion below concerning other activation functions." + ] + }, + { + "cell_type": "markdown", + "id": "112b8cc3", + "metadata": { + "editable": true + }, + "source": [ + "## Layers\n", + "\n", + "* Input \n", + "\n", + "Since each input image has 8x8 = 64 pixels or features, we have an input layer of 64 neurons. \n", + "\n", + "* Hidden layer\n", + "\n", + "We will use 50 neurons in the hidden layer receiving input from the neurons in the input layer. \n", + "Since each neuron in the hidden layer is connected to the 64 inputs we have 64x50 = 3200 weights to the hidden layer. \n", + "\n", + "* Output\n", + "\n", + "If we were building a binary classifier, it would be sufficient with a single neuron in the output layer,\n", + "which could output 0 or 1 according to the Heaviside function. This would be an example of a *hard* classifier, meaning it outputs the class of the input directly. However, if we are dealing with noisy data it is often beneficial to use a *soft* classifier, which outputs the probability of being in class 0 or 1. \n", + "\n", + "For a soft binary classifier, we could use a single neuron and interpret the output as either being the probability of being in class 0 or the probability of being in class 1. Alternatively we could use 2 neurons, and interpret each neuron as the probability of being in each class. \n", + "\n", + "Since we are doing multiclass classification, with 10 categories, it is natural to use 10 neurons in the output layer. We number the neurons $j = 0,1,...,9$. The activation of each output neuron $j$ will be according to the *softmax* function: \n", + "\n", + "$$ P(\\text{class $j$} \\mid \\text{input $\\boldsymbol{a}$}) = \\frac{\\exp{(\\boldsymbol{a}^T \\boldsymbol{w}_j)}}\n", + "{\\sum_{c=0}^{9} \\exp{(\\boldsymbol{a}^T \\boldsymbol{w}_c)}} ,$$ \n", + "\n", + "i.e. each neuron $j$ outputs the probability of being in class $j$ given an input from the hidden layer $\\boldsymbol{a}$, with $\\boldsymbol{w}_j$ the weights of neuron $j$ to the inputs. \n", + "The denominator is a normalization factor to ensure the outputs (probabilities) sum up to 1. \n", + "The exponent is just the weighted sum of inputs as before: \n", + "\n", + "$$ z_j = \\sum_{i=1}^n w_ {ij} a_i+b_j.$$ \n", + "\n", + "Since each neuron in the output layer is connected to the 50 inputs from the hidden layer we have 50x10 = 500\n", + "weights to the output layer." + ] + }, + { + "cell_type": "markdown", + "id": "e4166103", + "metadata": { + "editable": true + }, + "source": [ + "## Weights and biases\n", + "\n", + "Typically weights are initialized with small values distributed around zero, drawn from a uniform\n", + "or normal distribution. Setting all weights to zero means all neurons give the same output, making the network useless. \n", + "\n", + "Adding a bias value to the weighted sum of inputs allows the neural network to represent a greater range\n", + "of values. Without it, any input with the value 0 will be mapped to zero (before being passed through the activation). The bias unit has an output of 1, and a weight to each neuron $j$, $b_j$: \n", + "\n", + "$$ z_j = \\sum_{i=1}^n w_ {ij} a_i + b_j.$$ \n", + "\n", + "The bias weights $\\boldsymbol{b}$ are often initialized to zero, but a small value like $0.01$ ensures all neurons have some output which can be backpropagated in the first training cycle." + ] + }, + { + "cell_type": "code", + "execution_count": 5, + "id": "36399cb5", + "metadata": { + "collapsed": false, + "editable": true + }, + "outputs": [], + "source": [ + "# building our neural network\n", + "\n", + "n_inputs, n_features = X_train.shape\n", + "n_hidden_neurons = 50\n", + "n_categories = 10\n", + "\n", + "# we make the weights normally distributed using numpy.random.randn\n", + "\n", + "# weights and bias in the hidden layer\n", + "hidden_weights = np.random.randn(n_features, n_hidden_neurons)\n", + "hidden_bias = np.zeros(n_hidden_neurons) + 0.01\n", + "\n", + "# weights and bias in the output layer\n", + "output_weights = np.random.randn(n_hidden_neurons, n_categories)\n", + "output_bias = np.zeros(n_categories) + 0.01" + ] + }, + { + "cell_type": "markdown", + "id": "5eaf095b", + "metadata": { + "editable": true + }, + "source": [ + "## Feed-forward pass\n", + "\n", + "Denote $F$ the number of features, $H$ the number of hidden neurons and $C$ the number of categories. \n", + "For each input image we calculate a weighted sum of input features (pixel values) to each neuron $j$ in the hidden layer $l$: \n", + "\n", + "$$ z_{j}^{l} = \\sum_{i=1}^{F} w_{ij}^{l} x_i + b_{j}^{l},$$\n", + "\n", + "this is then passed through our activation function \n", + "\n", + "$$ a_{j}^{l} = f(z_{j}^{l}) .$$ \n", + "\n", + "We calculate a weighted sum of inputs (activations in the hidden layer) to each neuron $j$ in the output layer: \n", + "\n", + "$$ z_{j}^{L} = \\sum_{i=1}^{H} w_{ij}^{L} a_{i}^{l} + b_{j}^{L}.$$ \n", + "\n", + "Finally we calculate the output of neuron $j$ in the output layer using the softmax function: \n", + "\n", + "$$ a_{j}^{L} = \\frac{\\exp{(z_j^{L})}}\n", + "{\\sum_{c=0}^{C-1} \\exp{(z_c^{L})}} .$$" + ] + }, + { + "cell_type": "markdown", + "id": "91eec6b8", + "metadata": { + "editable": true + }, + "source": [ + "## Matrix multiplications\n", + "\n", + "Since our data has the dimensions $X = (n_{inputs}, n_{features})$ and our weights to the hidden\n", + "layer have the dimensions \n", + "$W_{hidden} = (n_{features}, n_{hidden})$,\n", + "we can easily feed the network all our training data in one go by taking the matrix product \n", + "\n", + "$$ X W^{h} = (n_{inputs}, n_{hidden}),$$ \n", + "\n", + "and obtain a matrix that holds the weighted sum of inputs to the hidden layer\n", + "for each input image and each hidden neuron. \n", + "We also add the bias to obtain a matrix of weighted sums to the hidden layer $Z^{h}$: \n", + "\n", + "$$ \\boldsymbol{z}^{l} = \\boldsymbol{X} \\boldsymbol{W}^{l} + \\boldsymbol{b}^{l} ,$$\n", + "\n", + "meaning the same bias (1D array with size equal number of hidden neurons) is added to each input image. \n", + "This is then passed through the activation: \n", + "\n", + "$$ \\boldsymbol{a}^{l} = f(\\boldsymbol{z}^l) .$$ \n", + "\n", + "This is fed to the output layer: \n", + "\n", + "$$ \\boldsymbol{z}^{L} = \\boldsymbol{a}^{L} \\boldsymbol{W}^{L} + \\boldsymbol{b}^{L} .$$\n", + "\n", + "Finally we receive our output values for each image and each category by passing it through the softmax function: \n", + "\n", + "$$ output = softmax (\\boldsymbol{z}^{L}) = (n_{inputs}, n_{categories}) .$$" + ] + }, + { + "cell_type": "code", + "execution_count": 6, + "id": "ef497434", + "metadata": { + "collapsed": false, + "editable": true + }, + "outputs": [], + "source": [ + "# setup the feed-forward pass, subscript h = hidden layer\n", + "\n", + "def sigmoid(x):\n", + " return 1/(1 + np.exp(-x))\n", + "\n", + "def feed_forward(X):\n", + " # weighted sum of inputs to the hidden layer\n", + " z_h = np.matmul(X, hidden_weights) + hidden_bias\n", + " # activation in the hidden layer\n", + " a_h = sigmoid(z_h)\n", + " \n", + " # weighted sum of inputs to the output layer\n", + " z_o = np.matmul(a_h, output_weights) + output_bias\n", + " # softmax output\n", + " # axis 0 holds each input and axis 1 the probabilities of each category\n", + " exp_term = np.exp(z_o)\n", + " probabilities = exp_term / np.sum(exp_term, axis=1, keepdims=True)\n", + " \n", + " return probabilities\n", + "\n", + "probabilities = feed_forward(X_train)\n", + "print(\"probabilities = (n_inputs, n_categories) = \" + str(probabilities.shape))\n", + "print(\"probability that image 0 is in category 0,1,2,...,9 = \\n\" + str(probabilities[0]))\n", + "print(\"probabilities sum up to: \" + str(probabilities[0].sum()))\n", + "print()\n", + "\n", + "# we obtain a prediction by taking the class with the highest likelihood\n", + "def predict(X):\n", + " probabilities = feed_forward(X)\n", + " return np.argmax(probabilities, axis=1)\n", + "\n", + "predictions = predict(X_train)\n", + "print(\"predictions = (n_inputs) = \" + str(predictions.shape))\n", + "print(\"prediction for image 0: \" + str(predictions[0]))\n", + "print(\"correct label for image 0: \" + str(Y_train[0]))" + ] + }, + { + "cell_type": "markdown", + "id": "ab88238a", + "metadata": { + "editable": true + }, + "source": [ + "## Choose cost function and optimizer\n", + "\n", + "To measure how well our neural network is doing we need to introduce a cost function. \n", + "We will call the function that gives the error of a single sample output the *loss* function, and the function\n", + "that gives the total error of our network across all samples the *cost* function.\n", + "A typical choice for multiclass classification is the *cross-entropy* loss, also known as the negative log likelihood. \n", + "\n", + "In *multiclass* classification it is common to treat each integer label as a so called *one-hot* vector: \n", + "\n", + "$$ y = 5 \\quad \\rightarrow \\quad \\boldsymbol{y} = (0, 0, 0, 0, 0, 1, 0, 0, 0, 0) ,$$ \n", + "\n", + "$$ y = 1 \\quad \\rightarrow \\quad \\boldsymbol{y} = (0, 1, 0, 0, 0, 0, 0, 0, 0, 0) ,$$ \n", + "\n", + "i.e. a binary bit string of length $C$, where $C = 10$ is the number of classes in the MNIST dataset. \n", + "\n", + "Let $y_{ic}$ denote the $c$-th component of the $i$-th one-hot vector. \n", + "We define the cost function $\\mathcal{C}$ as a sum over the cross-entropy loss for each point $\\boldsymbol{x}_i$ in the dataset.\n", + "\n", + "In the one-hot representation only one of the terms in the loss function is non-zero, namely the\n", + "probability of the correct category $c'$ \n", + "(i.e. the category $c'$ such that $y_{ic'} = 1$). This means that the cross entropy loss only punishes you for how wrong\n", + "you got the correct label. The probability of category $c$ is given by the softmax function. The vector $\\boldsymbol{\\theta}$ represents the parameters of our network, i.e. all the weights and biases." + ] + }, + { + "cell_type": "markdown", + "id": "506145ff", + "metadata": { + "editable": true + }, + "source": [ + "## Optimizing the cost function\n", + "\n", + "The network is trained by finding the weights and biases that minimize the cost function. One of the most widely used classes of methods is *gradient descent* and its generalizations. The idea behind gradient descent\n", + "is simply to adjust the weights in the direction where the gradient of the cost function is large and negative. This ensures we flow toward a *local* minimum of the cost function. \n", + "Each parameter $\\theta$ is iteratively adjusted according to the rule \n", + "\n", + "$$ \\theta_{i+1} = \\theta_i - \\eta \\nabla \\mathcal{C}(\\theta_i) ,$$\n", + "\n", + "where $\\eta$ is known as the *learning rate*, which controls how big a step we take towards the minimum. \n", + "This update can be repeated for any number of iterations, or until we are satisfied with the result. \n", + "\n", + "A simple and effective improvement is a variant called *Batch Gradient Descent*. \n", + "Instead of calculating the gradient on the whole dataset, we calculate an approximation of the gradient\n", + "on a subset of the data called a *minibatch*. \n", + "If there are $N$ data points and we have a minibatch size of $M$, the total number of batches\n", + "is $N/M$. \n", + "We denote each minibatch $B_k$, with $k = 1, 2,...,N/M$. The gradient then becomes: \n", + "\n", + "$$ \\nabla \\mathcal{C}(\\theta) = \\frac{1}{N} \\sum_{i=1}^N \\nabla \\mathcal{L}_i(\\theta) \\quad \\rightarrow \\quad\n", + "\\frac{1}{M} \\sum_{i \\in B_k} \\nabla \\mathcal{L}_i(\\theta) ,$$\n", + "\n", + "i.e. instead of averaging the loss over the entire dataset, we average over a minibatch. \n", + "\n", + "This has two important benefits: \n", + "1. Introducing stochasticity decreases the chance that the algorithm becomes stuck in a local minima. \n", + "\n", + "2. It significantly speeds up the calculation, since we do not have to use the entire dataset to calculate the gradient. \n", + "\n", + "The various optmization methods, with codes and algorithms, are discussed in our lectures on [Gradient descent approaches](https://compphysics.github.io/MachineLearning/doc/pub/Splines/html/Splines-bs.html)." + ] + }, + { + "cell_type": "markdown", + "id": "e8138025", + "metadata": { + "editable": true + }, + "source": [ + "## Regularization\n", + "\n", + "It is common to add an extra term to the cost function, proportional\n", + "to the size of the weights. This is equivalent to constraining the\n", + "size of the weights, so that they do not grow out of control.\n", + "Constraining the size of the weights means that the weights cannot\n", + "grow arbitrarily large to fit the training data, and in this way\n", + "reduces *overfitting*.\n", + "\n", + "We will measure the size of the weights using the so called *L2-norm*, meaning our cost function becomes: \n", + "\n", + "$$ \\mathcal{C}(\\theta) = \\frac{1}{N} \\sum_{i=1}^N \\mathcal{L}_i(\\theta) \\quad \\rightarrow \\quad\n", + "\\frac{1}{N} \\sum_{i=1}^N \\mathcal{L}_i(\\theta) + \\lambda \\lvert \\lvert \\boldsymbol{w} \\rvert \\rvert_2^2 \n", + "= \\frac{1}{N} \\sum_{i=1}^N \\mathcal{L}(\\theta) + \\lambda \\sum_{ij} w_{ij}^2,$$ \n", + "\n", + "i.e. we sum up all the weights squared. The factor $\\lambda$ is known as a regularization parameter.\n", + "\n", + "In order to train the model, we need to calculate the derivative of\n", + "the cost function with respect to every bias and weight in the\n", + "network. In total our network has $(64 + 1)\\times 50=3250$ weights in\n", + "the hidden layer and $(50 + 1)\\times 10=510$ weights to the output\n", + "layer ($+1$ for the bias), and the gradient must be calculated for\n", + "every parameter. We use the *backpropagation* algorithm discussed\n", + "above. This is a clever use of the chain rule that allows us to\n", + "calculate the gradient efficently." + ] + }, + { + "cell_type": "markdown", + "id": "25becf76", + "metadata": { + "editable": true + }, + "source": [ + "## Matrix multiplication\n", + "\n", + "To more efficently train our network these equations are implemented using matrix operations. \n", + "The error in the output layer is calculated simply as, with $\\boldsymbol{t}$ being our targets, \n", + "\n", + "$$ \\delta_L = \\boldsymbol{t} - \\boldsymbol{y} = (n_{inputs}, n_{categories}) .$$ \n", + "\n", + "The gradient for the output weights is calculated as \n", + "\n", + "$$ \\nabla W_{L} = \\boldsymbol{a}^T \\delta_L = (n_{hidden}, n_{categories}) ,$$\n", + "\n", + "where $\\boldsymbol{a} = (n_{inputs}, n_{hidden})$. This simply means that we are summing up the gradients for each input. \n", + "Since we are going backwards we have to transpose the activation matrix. \n", + "\n", + "The gradient with respect to the output bias is then \n", + "\n", + "$$ \\nabla \\boldsymbol{b}_{L} = \\sum_{i=1}^{n_{inputs}} \\delta_L = (n_{categories}) .$$ \n", + "\n", + "The error in the hidden layer is \n", + "\n", + "$$ \\Delta_h = \\delta_L W_{L}^T \\circ f'(z_{h}) = \\delta_L W_{L}^T \\circ a_{h} \\circ (1 - a_{h}) = (n_{inputs}, n_{hidden}) ,$$ \n", + "\n", + "where $f'(a_{h})$ is the derivative of the activation in the hidden layer. The matrix products mean\n", + "that we are summing up the products for each neuron in the output layer. The symbol $\\circ$ denotes\n", + "the *Hadamard product*, meaning element-wise multiplication. \n", + "\n", + "This again gives us the gradients in the hidden layer: \n", + "\n", + "$$ \\nabla W_{h} = X^T \\delta_h = (n_{features}, n_{hidden}) ,$$ \n", + "\n", + "$$ \\nabla b_{h} = \\sum_{i=1}^{n_{inputs}} \\delta_h = (n_{hidden}) .$$" + ] + }, + { + "cell_type": "code", + "execution_count": 7, + "id": "657c604c", + "metadata": { + "collapsed": false, + "editable": true + }, + "outputs": [], + "source": [ + "# to categorical turns our integer vector into a onehot representation\n", + "from sklearn.metrics import accuracy_score\n", + "\n", + "# one-hot in numpy\n", + "def to_categorical_numpy(integer_vector):\n", + " n_inputs = len(integer_vector)\n", + " n_categories = np.max(integer_vector) + 1\n", + " onehot_vector = np.zeros((n_inputs, n_categories))\n", + " onehot_vector[range(n_inputs), integer_vector] = 1\n", + " \n", + " return onehot_vector\n", + "\n", + "#Y_train_onehot, Y_test_onehot = to_categorical(Y_train), to_categorical(Y_test)\n", + "Y_train_onehot, Y_test_onehot = to_categorical_numpy(Y_train), to_categorical_numpy(Y_test)\n", + "\n", + "def feed_forward_train(X):\n", + " # weighted sum of inputs to the hidden layer\n", + " z_h = np.matmul(X, hidden_weights) + hidden_bias\n", + " # activation in the hidden layer\n", + " a_h = sigmoid(z_h)\n", + " \n", + " # weighted sum of inputs to the output layer\n", + " z_o = np.matmul(a_h, output_weights) + output_bias\n", + " # softmax output\n", + " # axis 0 holds each input and axis 1 the probabilities of each category\n", + " exp_term = np.exp(z_o)\n", + " probabilities = exp_term / np.sum(exp_term, axis=1, keepdims=True)\n", + " \n", + " # for backpropagation need activations in hidden and output layers\n", + " return a_h, probabilities\n", + "\n", + "def backpropagation(X, Y):\n", + " a_h, probabilities = feed_forward_train(X)\n", + " \n", + " # error in the output layer\n", + " error_output = probabilities - Y\n", + " # error in the hidden layer\n", + " error_hidden = np.matmul(error_output, output_weights.T) * a_h * (1 - a_h)\n", + " \n", + " # gradients for the output layer\n", + " output_weights_gradient = np.matmul(a_h.T, error_output)\n", + " output_bias_gradient = np.sum(error_output, axis=0)\n", + " \n", + " # gradient for the hidden layer\n", + " hidden_weights_gradient = np.matmul(X.T, error_hidden)\n", + " hidden_bias_gradient = np.sum(error_hidden, axis=0)\n", + "\n", + " return output_weights_gradient, output_bias_gradient, hidden_weights_gradient, hidden_bias_gradient\n", + "\n", + "print(\"Old accuracy on training data: \" + str(accuracy_score(predict(X_train), Y_train)))\n", + "\n", + "eta = 0.01\n", + "lmbd = 0.01\n", + "for i in range(1000):\n", + " # calculate gradients\n", + " dWo, dBo, dWh, dBh = backpropagation(X_train, Y_train_onehot)\n", + " \n", + " # regularization term gradients\n", + " dWo += lmbd * output_weights\n", + " dWh += lmbd * hidden_weights\n", + " \n", + " # update weights and biases\n", + " output_weights -= eta * dWo\n", + " output_bias -= eta * dBo\n", + " hidden_weights -= eta * dWh\n", + " hidden_bias -= eta * dBh\n", + "\n", + "print(\"New accuracy on training data: \" + str(accuracy_score(predict(X_train), Y_train)))" + ] + }, + { + "cell_type": "markdown", + "id": "3c595805", + "metadata": { + "editable": true + }, + "source": [ + "## Improving performance\n", + "\n", + "As we can see the network does not seem to be learning at all. It seems to be just guessing the label for each image. \n", + "In order to obtain a network that does something useful, we will have to do a bit more work. \n", + "\n", + "The choice of *hyperparameters* such as learning rate and regularization parameter is hugely influential for the performance of the network. Typically a *grid-search* is performed, wherein we test different hyperparameters separated by orders of magnitude. For example we could test the learning rates $\\eta = 10^{-6}, 10^{-5},...,10^{-1}$ with different regularization parameters $\\lambda = 10^{-6},...,10^{-0}$. \n", + "\n", + "Next, we haven't implemented minibatching yet, which introduces stochasticity and is though to act as an important regularizer on the weights. We call a feed-forward + backward pass with a minibatch an *iteration*, and a full training period\n", + "going through the entire dataset ($n/M$ batches) an *epoch*.\n", + "\n", + "If this does not improve network performance, you may want to consider altering the network architecture, adding more neurons or hidden layers. \n", + "Andrew Ng goes through some of these considerations in this [video](https://youtu.be/F1ka6a13S9I). You can find a summary of the video [here](https://kevinzakka.github.io/2016/09/26/applying-deep-learning/)." + ] + }, + { + "cell_type": "markdown", + "id": "17d5f534", + "metadata": { + "editable": true + }, + "source": [ + "## Full object-oriented implementation\n", + "\n", + "It is very natural to think of the network as an object, with specific instances of the network\n", + "being realizations of this object with different hyperparameters. An implementation using Python classes provides a clean structure and interface, and the full implementation of our neural network is given below." + ] + }, + { + "cell_type": "code", + "execution_count": 8, + "id": "5aeaa0cd", + "metadata": { + "collapsed": false, + "editable": true + }, + "outputs": [], + "source": [ + "class NeuralNetwork:\n", + " def __init__(\n", + " self,\n", + " X_data,\n", + " Y_data,\n", + " n_hidden_neurons=50,\n", + " n_categories=10,\n", + " epochs=10,\n", + " batch_size=100,\n", + " eta=0.1,\n", + " lmbd=0.0):\n", + "\n", + " self.X_data_full = X_data\n", + " self.Y_data_full = Y_data\n", + "\n", + " self.n_inputs = X_data.shape[0]\n", + " self.n_features = X_data.shape[1]\n", + " self.n_hidden_neurons = n_hidden_neurons\n", + " self.n_categories = n_categories\n", + "\n", + " self.epochs = epochs\n", + " self.batch_size = batch_size\n", + " self.iterations = self.n_inputs // self.batch_size\n", + " self.eta = eta\n", + " self.lmbd = lmbd\n", + "\n", + " self.create_biases_and_weights()\n", + "\n", + " def create_biases_and_weights(self):\n", + " self.hidden_weights = np.random.randn(self.n_features, self.n_hidden_neurons)\n", + " self.hidden_bias = np.zeros(self.n_hidden_neurons) + 0.01\n", + "\n", + " self.output_weights = np.random.randn(self.n_hidden_neurons, self.n_categories)\n", + " self.output_bias = np.zeros(self.n_categories) + 0.01\n", + "\n", + " def feed_forward(self):\n", + " # feed-forward for training\n", + " self.z_h = np.matmul(self.X_data, self.hidden_weights) + self.hidden_bias\n", + " self.a_h = sigmoid(self.z_h)\n", + "\n", + " self.z_o = np.matmul(self.a_h, self.output_weights) + self.output_bias\n", + "\n", + " exp_term = np.exp(self.z_o)\n", + " self.probabilities = exp_term / np.sum(exp_term, axis=1, keepdims=True)\n", + "\n", + " def feed_forward_out(self, X):\n", + " # feed-forward for output\n", + " z_h = np.matmul(X, self.hidden_weights) + self.hidden_bias\n", + " a_h = sigmoid(z_h)\n", + "\n", + " z_o = np.matmul(a_h, self.output_weights) + self.output_bias\n", + " \n", + " exp_term = np.exp(z_o)\n", + " probabilities = exp_term / np.sum(exp_term, axis=1, keepdims=True)\n", + " return probabilities\n", + "\n", + " def backpropagation(self):\n", + " error_output = self.probabilities - self.Y_data\n", + " error_hidden = np.matmul(error_output, self.output_weights.T) * self.a_h * (1 - self.a_h)\n", + "\n", + " self.output_weights_gradient = np.matmul(self.a_h.T, error_output)\n", + " self.output_bias_gradient = np.sum(error_output, axis=0)\n", + "\n", + " self.hidden_weights_gradient = np.matmul(self.X_data.T, error_hidden)\n", + " self.hidden_bias_gradient = np.sum(error_hidden, axis=0)\n", + "\n", + " if self.lmbd > 0.0:\n", + " self.output_weights_gradient += self.lmbd * self.output_weights\n", + " self.hidden_weights_gradient += self.lmbd * self.hidden_weights\n", + "\n", + " self.output_weights -= self.eta * self.output_weights_gradient\n", + " self.output_bias -= self.eta * self.output_bias_gradient\n", + " self.hidden_weights -= self.eta * self.hidden_weights_gradient\n", + " self.hidden_bias -= self.eta * self.hidden_bias_gradient\n", + "\n", + " def predict(self, X):\n", + " probabilities = self.feed_forward_out(X)\n", + " return np.argmax(probabilities, axis=1)\n", + "\n", + " def predict_probabilities(self, X):\n", + " probabilities = self.feed_forward_out(X)\n", + " return probabilities\n", + "\n", + " def train(self):\n", + " data_indices = np.arange(self.n_inputs)\n", + "\n", + " for i in range(self.epochs):\n", + " for j in range(self.iterations):\n", + " # pick datapoints with replacement\n", + " chosen_datapoints = np.random.choice(\n", + " data_indices, size=self.batch_size, replace=False\n", + " )\n", + "\n", + " # minibatch training data\n", + " self.X_data = self.X_data_full[chosen_datapoints]\n", + " self.Y_data = self.Y_data_full[chosen_datapoints]\n", + "\n", + " self.feed_forward()\n", + " self.backpropagation()" + ] + }, + { + "cell_type": "markdown", + "id": "77f30e1e", + "metadata": { + "editable": true + }, + "source": [ + "## Evaluate model performance on test data\n", + "\n", + "To measure the performance of our network we evaluate how well it does it data it has never seen before, i.e. the test data. \n", + "We measure the performance of the network using the *accuracy* score. \n", + "The accuracy is as you would expect just the number of images correctly labeled divided by the total number of images. A perfect classifier will have an accuracy score of $1$. \n", + "\n", + "$$ \\text{Accuracy} = \\frac{\\sum_{i=1}^n I(\\tilde{y}_i = y_i)}{n} ,$$ \n", + "\n", + "where $I$ is the indicator function, $1$ if $\\tilde{y}_i = y_i$ and $0$ otherwise." + ] + }, + { + "cell_type": "code", + "execution_count": 9, + "id": "c1ab41e1", + "metadata": { + "collapsed": false, + "editable": true + }, + "outputs": [], + "source": [ + "epochs = 100\n", + "batch_size = 100\n", + "\n", + "dnn = NeuralNetwork(X_train, Y_train_onehot, eta=eta, lmbd=lmbd, epochs=epochs, batch_size=batch_size,\n", + " n_hidden_neurons=n_hidden_neurons, n_categories=n_categories)\n", + "dnn.train()\n", + "test_predict = dnn.predict(X_test)\n", + "\n", + "# accuracy score from scikit library\n", + "print(\"Accuracy score on test set: \", accuracy_score(Y_test, test_predict))\n", + "\n", + "# equivalent in numpy\n", + "def accuracy_score_numpy(Y_test, Y_pred):\n", + " return np.sum(Y_test == Y_pred) / len(Y_test)\n", + "\n", + "#print(\"Accuracy score on test set: \", accuracy_score_numpy(Y_test, test_predict))" + ] + }, + { + "cell_type": "markdown", + "id": "1b2b35c3", + "metadata": { + "editable": true + }, + "source": [ + "## Adjust hyperparameters\n", + "\n", + "We now perform a grid search to find the optimal hyperparameters for the network. \n", + "Note that we are only using 1 layer with 50 neurons, and human performance is estimated to be around $98\\%$ ($2\\%$ error rate)." + ] + }, + { + "cell_type": "code", + "execution_count": 10, + "id": "ac2ab4a6", + "metadata": { + "collapsed": false, + "editable": true + }, + "outputs": [], + "source": [ + "eta_vals = np.logspace(-5, 1, 7)\n", + "lmbd_vals = np.logspace(-5, 1, 7)\n", + "# store the models for later use\n", + "DNN_numpy = np.zeros((len(eta_vals), len(lmbd_vals)), dtype=object)\n", + "\n", + "# grid search\n", + "for i, eta in enumerate(eta_vals):\n", + " for j, lmbd in enumerate(lmbd_vals):\n", + " dnn = NeuralNetwork(X_train, Y_train_onehot, eta=eta, lmbd=lmbd, epochs=epochs, batch_size=batch_size,\n", + " n_hidden_neurons=n_hidden_neurons, n_categories=n_categories)\n", + " dnn.train()\n", + " \n", + " DNN_numpy[i][j] = dnn\n", + " \n", + " test_predict = dnn.predict(X_test)\n", + " \n", + " print(\"Learning rate = \", eta)\n", + " print(\"Lambda = \", lmbd)\n", + " print(\"Accuracy score on test set: \", accuracy_score(Y_test, test_predict))\n", + " print()" + ] + }, + { + "cell_type": "markdown", + "id": "f3e5dfec", + "metadata": { + "editable": true + }, + "source": [ + "## Visualization" + ] + }, + { + "cell_type": "code", + "execution_count": 11, + "id": "6d875555", + "metadata": { + "collapsed": false, + "editable": true + }, + "outputs": [], + "source": [ + "# visual representation of grid search\n", + "# uses seaborn heatmap, you can also do this with matplotlib imshow\n", + "import seaborn as sns\n", + "\n", + "sns.set()\n", + "\n", + "train_accuracy = np.zeros((len(eta_vals), len(lmbd_vals)))\n", + "test_accuracy = np.zeros((len(eta_vals), len(lmbd_vals)))\n", + "\n", + "for i in range(len(eta_vals)):\n", + " for j in range(len(lmbd_vals)):\n", + " dnn = DNN_numpy[i][j]\n", + " \n", + " train_pred = dnn.predict(X_train) \n", + " test_pred = dnn.predict(X_test)\n", + "\n", + " train_accuracy[i][j] = accuracy_score(Y_train, train_pred)\n", + " test_accuracy[i][j] = accuracy_score(Y_test, test_pred)\n", + "\n", + " \n", + "fig, ax = plt.subplots(figsize = (10, 10))\n", + "sns.heatmap(train_accuracy, annot=True, ax=ax, cmap=\"viridis\")\n", + "ax.set_title(\"Training Accuracy\")\n", + "ax.set_ylabel(\"$\\eta$\")\n", + "ax.set_xlabel(\"$\\lambda$\")\n", + "plt.show()\n", + "\n", + "fig, ax = plt.subplots(figsize = (10, 10))\n", + "sns.heatmap(test_accuracy, annot=True, ax=ax, cmap=\"viridis\")\n", + "ax.set_title(\"Test Accuracy\")\n", + "ax.set_ylabel(\"$\\eta$\")\n", + "ax.set_xlabel(\"$\\lambda$\")\n", + "plt.show()" + ] + }, + { + "cell_type": "markdown", + "id": "81c742e3", + "metadata": { + "editable": true + }, + "source": [ + "## scikit-learn implementation\n", + "\n", + "**scikit-learn** focuses more\n", + "on traditional machine learning methods, such as regression,\n", + "clustering, decision trees, etc. As such, it has only two types of\n", + "neural networks: Multi Layer Perceptron outputting continuous values,\n", + "*MPLRegressor*, and Multi Layer Perceptron outputting labels,\n", + "*MLPClassifier*. We will see how simple it is to use these classes.\n", + "\n", + "**scikit-learn** implements a few improvements from our neural network,\n", + "such as early stopping, a varying learning rate, different\n", + "optimization methods, etc. We would therefore expect a better\n", + "performance overall." + ] + }, + { + "cell_type": "code", + "execution_count": 12, + "id": "d21828bb", + "metadata": { + "collapsed": false, + "editable": true + }, + "outputs": [], + "source": [ + "from sklearn.neural_network import MLPClassifier\n", + "# store models for later use\n", + "DNN_scikit = np.zeros((len(eta_vals), len(lmbd_vals)), dtype=object)\n", + "\n", + "for i, eta in enumerate(eta_vals):\n", + " for j, lmbd in enumerate(lmbd_vals):\n", + " dnn = MLPClassifier(hidden_layer_sizes=(n_hidden_neurons), activation='logistic',\n", + " alpha=lmbd, learning_rate_init=eta, max_iter=epochs)\n", + " dnn.fit(X_train, Y_train)\n", + " \n", + " DNN_scikit[i][j] = dnn\n", + " \n", + " print(\"Learning rate = \", eta)\n", + " print(\"Lambda = \", lmbd)\n", + " print(\"Accuracy score on test set: \", dnn.score(X_test, Y_test))\n", + " print()" + ] + }, + { + "cell_type": "markdown", + "id": "4b40579d", + "metadata": { + "editable": true + }, + "source": [ + "## Visualization" + ] + }, + { + "cell_type": "code", + "execution_count": 13, + "id": "bfc97198", + "metadata": { + "collapsed": false, + "editable": true + }, + "outputs": [], + "source": [ + "# optional\n", + "# visual representation of grid search\n", + "# uses seaborn heatmap, could probably do this in matplotlib\n", + "import seaborn as sns\n", + "\n", + "sns.set()\n", + "\n", + "train_accuracy = np.zeros((len(eta_vals), len(lmbd_vals)))\n", + "test_accuracy = np.zeros((len(eta_vals), len(lmbd_vals)))\n", + "\n", + "for i in range(len(eta_vals)):\n", + " for j in range(len(lmbd_vals)):\n", + " dnn = DNN_scikit[i][j]\n", + " \n", + " train_pred = dnn.predict(X_train) \n", + " test_pred = dnn.predict(X_test)\n", + "\n", + " train_accuracy[i][j] = accuracy_score(Y_train, train_pred)\n", + " test_accuracy[i][j] = accuracy_score(Y_test, test_pred)\n", + "\n", + " \n", + "fig, ax = plt.subplots(figsize = (10, 10))\n", + "sns.heatmap(train_accuracy, annot=True, ax=ax, cmap=\"viridis\")\n", + "ax.set_title(\"Training Accuracy\")\n", + "ax.set_ylabel(\"$\\eta$\")\n", + "ax.set_xlabel(\"$\\lambda$\")\n", + "plt.show()\n", + "\n", + "fig, ax = plt.subplots(figsize = (10, 10))\n", + "sns.heatmap(test_accuracy, annot=True, ax=ax, cmap=\"viridis\")\n", + "ax.set_title(\"Test Accuracy\")\n", + "ax.set_ylabel(\"$\\eta$\")\n", + "ax.set_xlabel(\"$\\lambda$\")\n", + "plt.show()" + ] + }, + { + "cell_type": "markdown", + "id": "550680d5", + "metadata": { + "editable": true + }, + "source": [ + "## Building neural networks in Tensorflow and Keras\n", + "\n", + "Now we want to build on the experience gained from our neural network implementation in NumPy and scikit-learn\n", + "and use it to construct a neural network in Tensorflow. Once we have constructed a neural network in NumPy\n", + "and Tensorflow, building one in Keras is really quite trivial, though the performance may suffer. \n", + "\n", + "In our previous example we used only one hidden layer, and in this we will use two. From this it should be quite\n", + "clear how to build one using an arbitrary number of hidden layers, using data structures such as Python lists or\n", + "NumPy arrays." + ] + }, + { + "cell_type": "markdown", + "id": "bbade60e", + "metadata": { + "editable": true + }, + "source": [ + "## Tensorflow\n", + "\n", + "Tensorflow is an open source library machine learning library\n", + "developed by the Google Brain team for internal use. It was released\n", + "under the Apache 2.0 open source license in November 9, 2015.\n", + "\n", + "Tensorflow is a computational framework that allows you to construct\n", + "machine learning models at different levels of abstraction, from\n", + "high-level, object-oriented APIs like Keras, down to the C++ kernels\n", + "that Tensorflow is built upon. The higher levels of abstraction are\n", + "simpler to use, but less flexible, and our choice of implementation\n", + "should reflect the problems we are trying to solve.\n", + "\n", + "[Tensorflow uses](https://www.tensorflow.org/guide/graphs) so-called graphs to represent your computation\n", + "in terms of the dependencies between individual operations, such that you first build a Tensorflow *graph*\n", + "to represent your model, and then create a Tensorflow *session* to run the graph.\n", + "\n", + "In this guide we will analyze the same data as we did in our NumPy and\n", + "scikit-learn tutorial, gathered from the MNIST database of images. We\n", + "will give an introduction to the lower level Python Application\n", + "Program Interfaces (APIs), and see how we use them to build our graph.\n", + "Then we will build (effectively) the same graph in Keras, to see just\n", + "how simple solving a machine learning problem can be.\n", + "\n", + "To install tensorflow on Unix/Linux systems, use pip as" + ] + }, + { + "cell_type": "code", + "execution_count": 14, + "id": "e5505ed3", + "metadata": { + "collapsed": false, + "editable": true + }, + "outputs": [], + "source": [ + "pip3 install tensorflow" + ] + }, + { + "cell_type": "markdown", + "id": "98d4797a", + "metadata": { + "editable": true + }, + "source": [ + "and/or if you use **anaconda**, just write (or install from the graphical user interface)\n", + "(current release of CPU-only TensorFlow)" + ] + }, + { + "cell_type": "code", + "execution_count": 15, + "id": "95b05730", + "metadata": { + "collapsed": false, + "editable": true + }, + "outputs": [], + "source": [ + "conda create -n tf tensorflow\n", + "conda activate tf" + ] + }, + { + "cell_type": "markdown", + "id": "94170f18", + "metadata": { + "editable": true + }, + "source": [ + "To install the current release of GPU TensorFlow" + ] + }, + { + "cell_type": "code", + "execution_count": 16, + "id": "65ae6e69", + "metadata": { + "collapsed": false, + "editable": true + }, + "outputs": [], + "source": [ + "conda create -n tf-gpu tensorflow-gpu\n", + "conda activate tf-gpu" + ] + }, + { + "cell_type": "markdown", + "id": "df39c5c6", + "metadata": { + "editable": true + }, + "source": [ + "## Using Keras\n", + "\n", + "Keras is a high level [neural network](https://en.wikipedia.org/wiki/Application_programming_interface)\n", + "that supports Tensorflow, CTNK and Theano as backends. \n", + "If you have Anaconda installed you may run the following command" + ] + }, + { + "cell_type": "code", + "execution_count": 17, + "id": "d0d627f6", + "metadata": { + "collapsed": false, + "editable": true + }, + "outputs": [], + "source": [ + "conda install keras" + ] + }, + { + "cell_type": "markdown", + "id": "4124b5af", + "metadata": { + "editable": true + }, + "source": [ + "You can look up the [instructions here](https://keras.io/) for more information.\n", + "\n", + "We will to a large extent use **keras** in this course." + ] + }, + { + "cell_type": "markdown", + "id": "22dc977c", + "metadata": { + "editable": true + }, + "source": [ + "## Collect and pre-process data\n", + "\n", + "Let us look again at the MINST data set." + ] + }, + { + "cell_type": "code", + "execution_count": 18, + "id": "7df2e4d4", + "metadata": { + "collapsed": false, + "editable": true + }, + "outputs": [], + "source": [ + "# import necessary packages\n", + "import numpy as np\n", + "import matplotlib.pyplot as plt\n", + "import tensorflow as tf\n", + "from sklearn import datasets\n", + "\n", + "\n", + "# ensure the same random numbers appear every time\n", + "np.random.seed(0)\n", + "\n", + "# display images in notebook\n", + "%matplotlib inline\n", + "plt.rcParams['figure.figsize'] = (12,12)\n", + "\n", + "\n", + "# download MNIST dataset\n", + "digits = datasets.load_digits()\n", + "\n", + "# define inputs and labels\n", + "inputs = digits.images\n", + "labels = digits.target\n", + "\n", + "print(\"inputs = (n_inputs, pixel_width, pixel_height) = \" + str(inputs.shape))\n", + "print(\"labels = (n_inputs) = \" + str(labels.shape))\n", + "\n", + "\n", + "# flatten the image\n", + "# the value -1 means dimension is inferred from the remaining dimensions: 8x8 = 64\n", + "n_inputs = len(inputs)\n", + "inputs = inputs.reshape(n_inputs, -1)\n", + "print(\"X = (n_inputs, n_features) = \" + str(inputs.shape))\n", + "\n", + "\n", + "# choose some random images to display\n", + "indices = np.arange(n_inputs)\n", + "random_indices = np.random.choice(indices, size=5)\n", + "\n", + "for i, image in enumerate(digits.images[random_indices]):\n", + " plt.subplot(1, 5, i+1)\n", + " plt.axis('off')\n", + " plt.imshow(image, cmap=plt.cm.gray_r, interpolation='nearest')\n", + " plt.title(\"Label: %d\" % digits.target[random_indices[i]])\n", + "plt.show()" + ] + }, + { + "cell_type": "code", + "execution_count": 19, + "id": "4f5bff6f", + "metadata": { + "collapsed": false, + "editable": true + }, + "outputs": [], + "source": [ + "from tensorflow.keras.layers import Input\n", + "from tensorflow.keras.models import Sequential #This allows appending layers to existing models\n", + "from tensorflow.keras.layers import Dense #This allows defining the characteristics of a particular layer\n", + "from tensorflow.keras import optimizers #This allows using whichever optimiser we want (sgd,adam,RMSprop)\n", + "from tensorflow.keras import regularizers #This allows using whichever regularizer we want (l1,l2,l1_l2)\n", + "from tensorflow.keras.utils import to_categorical #This allows using categorical cross entropy as the cost function\n", + "\n", + "from sklearn.model_selection import train_test_split\n", + "\n", + "# one-hot representation of labels\n", + "labels = to_categorical(labels)\n", + "\n", + "# split into train and test data\n", + "train_size = 0.8\n", + "test_size = 1 - train_size\n", + "X_train, X_test, Y_train, Y_test = train_test_split(inputs, labels, train_size=train_size,\n", + " test_size=test_size)" + ] + }, + { + "cell_type": "code", + "execution_count": 20, + "id": "3ae517ea", + "metadata": { + "collapsed": false, + "editable": true + }, + "outputs": [], + "source": [ + "\n", + "epochs = 100\n", + "batch_size = 100\n", + "n_neurons_layer1 = 100\n", + "n_neurons_layer2 = 50\n", + "n_categories = 10\n", + "eta_vals = np.logspace(-5, 1, 7)\n", + "lmbd_vals = np.logspace(-5, 1, 7)\n", + "def create_neural_network_keras(n_neurons_layer1, n_neurons_layer2, n_categories, eta, lmbd):\n", + " model = Sequential()\n", + " model.add(Dense(n_neurons_layer1, activation='sigmoid', kernel_regularizer=regularizers.l2(lmbd)))\n", + " model.add(Dense(n_neurons_layer2, activation='sigmoid', kernel_regularizer=regularizers.l2(lmbd)))\n", + " model.add(Dense(n_categories, activation='softmax'))\n", + " \n", + " sgd = optimizers.SGD(lr=eta)\n", + " model.compile(loss='categorical_crossentropy', optimizer=sgd, metrics=['accuracy'])\n", + " \n", + " return model" + ] + }, + { + "cell_type": "code", + "execution_count": 21, + "id": "c880b209", + "metadata": { + "collapsed": false, + "editable": true + }, + "outputs": [], + "source": [ + "DNN_keras = np.zeros((len(eta_vals), len(lmbd_vals)), dtype=object)\n", + " \n", + "for i, eta in enumerate(eta_vals):\n", + " for j, lmbd in enumerate(lmbd_vals):\n", + " DNN = create_neural_network_keras(n_neurons_layer1, n_neurons_layer2, n_categories,\n", + " eta=eta, lmbd=lmbd)\n", + " DNN.fit(X_train, Y_train, epochs=epochs, batch_size=batch_size, verbose=0)\n", + " scores = DNN.evaluate(X_test, Y_test)\n", + " \n", + " DNN_keras[i][j] = DNN\n", + " \n", + " print(\"Learning rate = \", eta)\n", + " print(\"Lambda = \", lmbd)\n", + " print(\"Test accuracy: %.3f\" % scores[1])\n", + " print()" + ] + }, + { + "cell_type": "code", + "execution_count": 22, + "id": "cf8cea73", + "metadata": { + "collapsed": false, + "editable": true + }, + "outputs": [], + "source": [ + "# optional\n", + "# visual representation of grid search\n", + "# uses seaborn heatmap, could probably do this in matplotlib\n", + "import seaborn as sns\n", + "\n", + "sns.set()\n", + "\n", + "train_accuracy = np.zeros((len(eta_vals), len(lmbd_vals)))\n", + "test_accuracy = np.zeros((len(eta_vals), len(lmbd_vals)))\n", + "\n", + "for i in range(len(eta_vals)):\n", + " for j in range(len(lmbd_vals)):\n", + " DNN = DNN_keras[i][j]\n", + "\n", + " train_accuracy[i][j] = DNN.evaluate(X_train, Y_train)[1]\n", + " test_accuracy[i][j] = DNN.evaluate(X_test, Y_test)[1]\n", + "\n", + " \n", + "fig, ax = plt.subplots(figsize = (10, 10))\n", + "sns.heatmap(train_accuracy, annot=True, ax=ax, cmap=\"viridis\")\n", + "ax.set_title(\"Training Accuracy\")\n", + "ax.set_ylabel(\"$\\eta$\")\n", + "ax.set_xlabel(\"$\\lambda$\")\n", + "plt.show()\n", + "\n", + "fig, ax = plt.subplots(figsize = (10, 10))\n", + "sns.heatmap(test_accuracy, annot=True, ax=ax, cmap=\"viridis\")\n", + "ax.set_title(\"Test Accuracy\")\n", + "ax.set_ylabel(\"$\\eta$\")\n", + "ax.set_xlabel(\"$\\lambda$\")\n", + "plt.show()" + ] + }, + { + "cell_type": "markdown", + "id": "3024761c", + "metadata": { + "editable": true + }, + "source": [ + "## Building a neural network code\n", + "\n", + "Here we present a flexible object oriented codebase\n", + "for a feed forward neural network, along with a demonstration of how\n", + "to use it. Before we get into the details of the neural network, we\n", + "will first present some implementations of various schedulers, cost\n", + "functions and activation functions that can be used together with the\n", + "neural network.\n", + "\n", + "The codes here were developed by Eric Reber and Gregor Kajda during spring 2023." + ] + }, + { + "cell_type": "markdown", + "id": "53987675", + "metadata": { + "editable": true + }, + "source": [ + "### Learning rate methods\n", + "\n", + "The code below shows object oriented implementations of the Constant,\n", + "Momentum, Adagrad, AdagradMomentum, RMS prop and Adam schedulers. All\n", + "of the classes belong to the shared abstract Scheduler class, and\n", + "share the update_change() and reset() methods allowing for any of the\n", + "schedulers to be seamlessly used during the training stage, as will\n", + "later be shown in the fit() method of the neural\n", + "network. Update_change() only has one parameter, the gradient\n", + "($δ^l_ja^{l−1}_k$), and returns the change which will be subtracted\n", + "from the weights. The reset() function takes no parameters, and resets\n", + "the desired variables. For Constant and Momentum, reset does nothing." + ] + }, + { + "cell_type": "code", + "execution_count": 23, + "id": "5ed03631", + "metadata": { + "collapsed": false, + "editable": true + }, + "outputs": [], + "source": [ + "import autograd.numpy as np\n", + "\n", + "class Scheduler:\n", + " \"\"\"\n", + " Abstract class for Schedulers\n", + " \"\"\"\n", + "\n", + " def __init__(self, eta):\n", + " self.eta = eta\n", + "\n", + " # should be overwritten\n", + " def update_change(self, gradient):\n", + " raise NotImplementedError\n", + "\n", + " # overwritten if needed\n", + " def reset(self):\n", + " pass\n", + "\n", + "\n", + "class Constant(Scheduler):\n", + " def __init__(self, eta):\n", + " super().__init__(eta)\n", + "\n", + " def update_change(self, gradient):\n", + " return self.eta * gradient\n", + " \n", + " def reset(self):\n", + " pass\n", + "\n", + "\n", + "class Momentum(Scheduler):\n", + " def __init__(self, eta: float, momentum: float):\n", + " super().__init__(eta)\n", + " self.momentum = momentum\n", + " self.change = 0\n", + "\n", + " def update_change(self, gradient):\n", + " self.change = self.momentum * self.change + self.eta * gradient\n", + " return self.change\n", + "\n", + " def reset(self):\n", + " pass\n", + "\n", + "\n", + "class Adagrad(Scheduler):\n", + " def __init__(self, eta):\n", + " super().__init__(eta)\n", + " self.G_t = None\n", + "\n", + " def update_change(self, gradient):\n", + " delta = 1e-8 # avoid division ny zero\n", + "\n", + " if self.G_t is None:\n", + " self.G_t = np.zeros((gradient.shape[0], gradient.shape[0]))\n", + "\n", + " self.G_t += gradient @ gradient.T\n", + "\n", + " G_t_inverse = 1 / (\n", + " delta + np.sqrt(np.reshape(np.diagonal(self.G_t), (self.G_t.shape[0], 1)))\n", + " )\n", + " return self.eta * gradient * G_t_inverse\n", + "\n", + " def reset(self):\n", + " self.G_t = None\n", + "\n", + "\n", + "class AdagradMomentum(Scheduler):\n", + " def __init__(self, eta, momentum):\n", + " super().__init__(eta)\n", + " self.G_t = None\n", + " self.momentum = momentum\n", + " self.change = 0\n", + "\n", + " def update_change(self, gradient):\n", + " delta = 1e-8 # avoid division ny zero\n", + "\n", + " if self.G_t is None:\n", + " self.G_t = np.zeros((gradient.shape[0], gradient.shape[0]))\n", + "\n", + " self.G_t += gradient @ gradient.T\n", + "\n", + " G_t_inverse = 1 / (\n", + " delta + np.sqrt(np.reshape(np.diagonal(self.G_t), (self.G_t.shape[0], 1)))\n", + " )\n", + " self.change = self.change * self.momentum + self.eta * gradient * G_t_inverse\n", + " return self.change\n", + "\n", + " def reset(self):\n", + " self.G_t = None\n", + "\n", + "\n", + "class RMS_prop(Scheduler):\n", + " def __init__(self, eta, rho):\n", + " super().__init__(eta)\n", + " self.rho = rho\n", + " self.second = 0.0\n", + "\n", + " def update_change(self, gradient):\n", + " delta = 1e-8 # avoid division ny zero\n", + " self.second = self.rho * self.second + (1 - self.rho) * gradient * gradient\n", + " return self.eta * gradient / (np.sqrt(self.second + delta))\n", + "\n", + " def reset(self):\n", + " self.second = 0.0\n", + "\n", + "\n", + "class Adam(Scheduler):\n", + " def __init__(self, eta, rho, rho2):\n", + " super().__init__(eta)\n", + " self.rho = rho\n", + " self.rho2 = rho2\n", + " self.moment = 0\n", + " self.second = 0\n", + " self.n_epochs = 1\n", + "\n", + " def update_change(self, gradient):\n", + " delta = 1e-8 # avoid division ny zero\n", + "\n", + " self.moment = self.rho * self.moment + (1 - self.rho) * gradient\n", + " self.second = self.rho2 * self.second + (1 - self.rho2) * gradient * gradient\n", + "\n", + " moment_corrected = self.moment / (1 - self.rho**self.n_epochs)\n", + " second_corrected = self.second / (1 - self.rho2**self.n_epochs)\n", + "\n", + " return self.eta * moment_corrected / (np.sqrt(second_corrected + delta))\n", + "\n", + " def reset(self):\n", + " self.n_epochs += 1\n", + " self.moment = 0\n", + " self.second = 0" + ] + }, + { + "cell_type": "markdown", + "id": "8b98c385", + "metadata": { + "editable": true + }, + "source": [ + "### Usage of the above learning rate schedulers\n", + "\n", + "To initalize a scheduler, simply create the object and pass in the\n", + "necessary parameters such as the learning rate and the momentum as\n", + "shown below. As the Scheduler class is an abstract class it should not\n", + "called directly, and will raise an error upon usage." + ] + }, + { + "cell_type": "code", + "execution_count": 24, + "id": "54298abd", + "metadata": { + "collapsed": false, + "editable": true + }, + "outputs": [], + "source": [ + "momentum_scheduler = Momentum(eta=1e-3, momentum=0.9)\n", + "adam_scheduler = Adam(eta=1e-3, rho=0.9, rho2=0.999)" + ] + }, + { + "cell_type": "markdown", + "id": "c03dc1b1", + "metadata": { + "editable": true + }, + "source": [ + "Here is a small example for how a segment of code using schedulers\n", + "could look. Switching out the schedulers is simple." + ] + }, + { + "cell_type": "code", + "execution_count": 25, + "id": "5a2f7871", + "metadata": { + "collapsed": false, + "editable": true + }, + "outputs": [], + "source": [ + "weights = np.ones((3,3))\n", + "print(f\"Before scheduler:\\n{weights=}\")\n", + "\n", + "epochs = 10\n", + "for e in range(epochs):\n", + " gradient = np.random.rand(3, 3)\n", + " change = adam_scheduler.update_change(gradient)\n", + " weights = weights - change\n", + " adam_scheduler.reset()\n", + "\n", + "print(f\"\\nAfter scheduler:\\n{weights=}\")" + ] + }, + { + "cell_type": "markdown", + "id": "07fa559a", + "metadata": { + "editable": true + }, + "source": [ + "### Cost functions\n", + "\n", + "Here we discuss cost functions that can be used when creating the\n", + "neural network. Every cost function takes the target vector as its\n", + "parameter, and returns a function valued only at $x$ such that it may\n", + "easily be differentiated." + ] + }, + { + "cell_type": "code", + "execution_count": 26, + "id": "718450a2", + "metadata": { + "collapsed": false, + "editable": true + }, + "outputs": [], + "source": [ + "import autograd.numpy as np\n", + "\n", + "def CostOLS(target):\n", + " \n", + " def func(X):\n", + " return (1.0 / target.shape[0]) * np.sum((target - X) ** 2)\n", + "\n", + " return func\n", + "\n", + "\n", + "def CostLogReg(target):\n", + "\n", + " def func(X):\n", + " \n", + " return -(1.0 / target.shape[0]) * np.sum(\n", + " (target * np.log(X + 10e-10)) + ((1 - target) * np.log(1 - X + 10e-10))\n", + " )\n", + "\n", + " return func\n", + "\n", + "\n", + "def CostCrossEntropy(target):\n", + " \n", + " def func(X):\n", + " return -(1.0 / target.size) * np.sum(target * np.log(X + 10e-10))\n", + "\n", + " return func" + ] + }, + { + "cell_type": "markdown", + "id": "0db28b52", + "metadata": { + "editable": true + }, + "source": [ + "Below we give a short example of how these cost function may be used\n", + "to obtain results if you wish to test them out on your own using\n", + "AutoGrad's automatics differentiation." + ] + }, + { + "cell_type": "code", + "execution_count": 27, + "id": "1479de5a", + "metadata": { + "collapsed": false, + "editable": true + }, + "outputs": [], + "source": [ + "from autograd import grad\n", + "\n", + "target = np.array([[1, 2, 3]]).T\n", + "a = np.array([[4, 5, 6]]).T\n", + "\n", + "cost_func = CostCrossEntropy\n", + "cost_func_derivative = grad(cost_func(target))\n", + "\n", + "valued_at_a = cost_func_derivative(a)\n", + "print(f\"Derivative of cost function {cost_func.__name__} valued at a:\\n{valued_at_a}\")" + ] + }, + { + "cell_type": "markdown", + "id": "d42e1ccf", + "metadata": { + "editable": true + }, + "source": [ + "### Activation functions\n", + "\n", + "Finally, before we look at the neural network, we will look at the\n", + "activation functions which can be specified between the hidden layers\n", + "and as the output function. Each function can be valued for any given\n", + "vector or matrix X, and can be differentiated via derivate()." + ] + }, + { + "cell_type": "code", + "execution_count": 28, + "id": "2c2a8d32", + "metadata": { + "collapsed": false, + "editable": true + }, + "outputs": [], + "source": [ + "import autograd.numpy as np\n", + "from autograd import elementwise_grad\n", + "\n", + "def identity(X):\n", + " return X\n", + "\n", + "\n", + "def sigmoid(X):\n", + " try:\n", + " return 1.0 / (1 + np.exp(-X))\n", + " except FloatingPointError:\n", + " return np.where(X > np.zeros(X.shape), np.ones(X.shape), np.zeros(X.shape))\n", + "\n", + "\n", + "def softmax(X):\n", + " X = X - np.max(X, axis=-1, keepdims=True)\n", + " delta = 10e-10\n", + " return np.exp(X) / (np.sum(np.exp(X), axis=-1, keepdims=True) + delta)\n", + "\n", + "\n", + "def RELU(X):\n", + " return np.where(X > np.zeros(X.shape), X, np.zeros(X.shape))\n", + "\n", + "\n", + "def LRELU(X):\n", + " delta = 10e-4\n", + " return np.where(X > np.zeros(X.shape), X, delta * X)\n", + "\n", + "\n", + "def derivate(func):\n", + " if func.__name__ == \"RELU\":\n", + "\n", + " def func(X):\n", + " return np.where(X > 0, 1, 0)\n", + "\n", + " return func\n", + "\n", + " elif func.__name__ == \"LRELU\":\n", + "\n", + " def func(X):\n", + " delta = 10e-4\n", + " return np.where(X > 0, 1, delta)\n", + "\n", + " return func\n", + "\n", + " else:\n", + " return elementwise_grad(func)" + ] + }, + { + "cell_type": "markdown", + "id": "65e7caab", + "metadata": { + "editable": true + }, + "source": [ + "Below follows a short demonstration of how to use an activation\n", + "function. The derivative of the activation function will be important\n", + "when calculating the output delta term during backpropagation. Note\n", + "that derivate() can also be used for cost functions for a more\n", + "generalized approach." + ] + }, + { + "cell_type": "code", + "execution_count": 29, + "id": "75e9399d", + "metadata": { + "collapsed": false, + "editable": true + }, + "outputs": [], + "source": [ + "z = np.array([[4, 5, 6]]).T\n", + "print(f\"Input to activation function:\\n{z}\")\n", + "\n", + "act_func = sigmoid\n", + "a = act_func(z)\n", + "print(f\"\\nOutput from {act_func.__name__} activation function:\\n{a}\")\n", + "\n", + "act_func_derivative = derivate(act_func)\n", + "valued_at_z = act_func_derivative(a)\n", + "print(f\"\\nDerivative of {act_func.__name__} activation function valued at z:\\n{valued_at_z}\")" + ] + }, + { + "cell_type": "markdown", + "id": "0b0b82e8", + "metadata": { + "editable": true + }, + "source": [ + "### The Neural Network\n", + "\n", + "Now that we have gotten a good understanding of the implementation of\n", + "some important components, we can take a look at an object oriented\n", + "implementation of a feed forward neural network. The feed forward\n", + "neural network has been implemented as a class named FFNN, which can\n", + "be initiated as a regressor or classifier dependant on the choice of\n", + "cost function. The FFNN can have any number of input nodes, hidden\n", + "layers with any amount of hidden nodes, and any amount of output nodes\n", + "meaning it can perform multiclass classification as well as binary\n", + "classification and regression problems. Although there is a lot of\n", + "code present, it makes for an easy to use and generalizeable interface\n", + "for creating many types of neural networks as will be demonstrated\n", + "below." + ] + }, + { + "cell_type": "code", + "execution_count": 30, + "id": "5351bfd6", + "metadata": { + "collapsed": false, + "editable": true + }, + "outputs": [], + "source": [ + "import math\n", + "import autograd.numpy as np\n", + "import sys\n", + "import warnings\n", + "from autograd import grad, elementwise_grad\n", + "from random import random, seed\n", + "from copy import deepcopy, copy\n", + "from typing import Tuple, Callable\n", + "from sklearn.utils import resample\n", + "\n", + "warnings.simplefilter(\"error\")\n", + "\n", + "\n", + "class FFNN:\n", + " \"\"\"\n", + " Description:\n", + " ------------\n", + " Feed Forward Neural Network with interface enabling flexible design of a\n", + " nerual networks architecture and the specification of activation function\n", + " in the hidden layers and output layer respectively. This model can be used\n", + " for both regression and classification problems, depending on the output function.\n", + "\n", + " Attributes:\n", + " ------------\n", + " I dimensions (tuple[int]): A list of positive integers, which specifies the\n", + " number of nodes in each of the networks layers. The first integer in the array\n", + " defines the number of nodes in the input layer, the second integer defines number\n", + " of nodes in the first hidden layer and so on until the last number, which\n", + " specifies the number of nodes in the output layer.\n", + " II hidden_func (Callable): The activation function for the hidden layers\n", + " III output_func (Callable): The activation function for the output layer\n", + " IV cost_func (Callable): Our cost function\n", + " V seed (int): Sets random seed, makes results reproducible\n", + " \"\"\"\n", + "\n", + " def __init__(\n", + " self,\n", + " dimensions: tuple[int],\n", + " hidden_func: Callable = sigmoid,\n", + " output_func: Callable = lambda x: x,\n", + " cost_func: Callable = CostOLS,\n", + " seed: int = None,\n", + " ):\n", + " self.dimensions = dimensions\n", + " self.hidden_func = hidden_func\n", + " self.output_func = output_func\n", + " self.cost_func = cost_func\n", + " self.seed = seed\n", + " self.weights = list()\n", + " self.schedulers_weight = list()\n", + " self.schedulers_bias = list()\n", + " self.a_matrices = list()\n", + " self.z_matrices = list()\n", + " self.classification = None\n", + "\n", + " self.reset_weights()\n", + " self._set_classification()\n", + "\n", + " def fit(\n", + " self,\n", + " X: np.ndarray,\n", + " t: np.ndarray,\n", + " scheduler: Scheduler,\n", + " batches: int = 1,\n", + " epochs: int = 100,\n", + " lam: float = 0,\n", + " X_val: np.ndarray = None,\n", + " t_val: np.ndarray = None,\n", + " ):\n", + " \"\"\"\n", + " Description:\n", + " ------------\n", + " This function performs the training the neural network by performing the feedforward and backpropagation\n", + " algorithm to update the networks weights.\n", + "\n", + " Parameters:\n", + " ------------\n", + " I X (np.ndarray) : training data\n", + " II t (np.ndarray) : target data\n", + " III scheduler (Scheduler) : specified scheduler (algorithm for optimization of gradient descent)\n", + " IV scheduler_args (list[int]) : list of all arguments necessary for scheduler\n", + "\n", + " Optional Parameters:\n", + " ------------\n", + " V batches (int) : number of batches the datasets are split into, default equal to 1\n", + " VI epochs (int) : number of iterations used to train the network, default equal to 100\n", + " VII lam (float) : regularization hyperparameter lambda\n", + " VIII X_val (np.ndarray) : validation set\n", + " IX t_val (np.ndarray) : validation target set\n", + "\n", + " Returns:\n", + " ------------\n", + " I scores (dict) : A dictionary containing the performance metrics of the model.\n", + " The number of the metrics depends on the parameters passed to the fit-function.\n", + "\n", + " \"\"\"\n", + "\n", + " # setup \n", + " if self.seed is not None:\n", + " np.random.seed(self.seed)\n", + "\n", + " val_set = False\n", + " if X_val is not None and t_val is not None:\n", + " val_set = True\n", + "\n", + " # creating arrays for score metrics\n", + " train_errors = np.empty(epochs)\n", + " train_errors.fill(np.nan)\n", + " val_errors = np.empty(epochs)\n", + " val_errors.fill(np.nan)\n", + "\n", + " train_accs = np.empty(epochs)\n", + " train_accs.fill(np.nan)\n", + " val_accs = np.empty(epochs)\n", + " val_accs.fill(np.nan)\n", + "\n", + " self.schedulers_weight = list()\n", + " self.schedulers_bias = list()\n", + "\n", + " batch_size = X.shape[0] // batches\n", + "\n", + " X, t = resample(X, t)\n", + "\n", + " # this function returns a function valued only at X\n", + " cost_function_train = self.cost_func(t)\n", + " if val_set:\n", + " cost_function_val = self.cost_func(t_val)\n", + "\n", + " # create schedulers for each weight matrix\n", + " for i in range(len(self.weights)):\n", + " self.schedulers_weight.append(copy(scheduler))\n", + " self.schedulers_bias.append(copy(scheduler))\n", + "\n", + " print(f\"{scheduler.__class__.__name__}: Eta={scheduler.eta}, Lambda={lam}\")\n", + "\n", + " try:\n", + " for e in range(epochs):\n", + " for i in range(batches):\n", + " # allows for minibatch gradient descent\n", + " if i == batches - 1:\n", + " # If the for loop has reached the last batch, take all thats left\n", + " X_batch = X[i * batch_size :, :]\n", + " t_batch = t[i * batch_size :, :]\n", + " else:\n", + " X_batch = X[i * batch_size : (i + 1) * batch_size, :]\n", + " t_batch = t[i * batch_size : (i + 1) * batch_size, :]\n", + "\n", + " self._feedforward(X_batch)\n", + " self._backpropagate(X_batch, t_batch, lam)\n", + "\n", + " # reset schedulers for each epoch (some schedulers pass in this call)\n", + " for scheduler in self.schedulers_weight:\n", + " scheduler.reset()\n", + "\n", + " for scheduler in self.schedulers_bias:\n", + " scheduler.reset()\n", + "\n", + " # computing performance metrics\n", + " pred_train = self.predict(X)\n", + " train_error = cost_function_train(pred_train)\n", + "\n", + " train_errors[e] = train_error\n", + " if val_set:\n", + " \n", + " pred_val = self.predict(X_val)\n", + " val_error = cost_function_val(pred_val)\n", + " val_errors[e] = val_error\n", + "\n", + " if self.classification:\n", + " train_acc = self._accuracy(self.predict(X), t)\n", + " train_accs[e] = train_acc\n", + " if val_set:\n", + " val_acc = self._accuracy(pred_val, t_val)\n", + " val_accs[e] = val_acc\n", + "\n", + " # printing progress bar\n", + " progression = e / epochs\n", + " print_length = self._progress_bar(\n", + " progression,\n", + " train_error=train_errors[e],\n", + " train_acc=train_accs[e],\n", + " val_error=val_errors[e],\n", + " val_acc=val_accs[e],\n", + " )\n", + " except KeyboardInterrupt:\n", + " # allows for stopping training at any point and seeing the result\n", + " pass\n", + "\n", + " # visualization of training progression (similiar to tensorflow progression bar)\n", + " sys.stdout.write(\"\\r\" + \" \" * print_length)\n", + " sys.stdout.flush()\n", + " self._progress_bar(\n", + " 1,\n", + " train_error=train_errors[e],\n", + " train_acc=train_accs[e],\n", + " val_error=val_errors[e],\n", + " val_acc=val_accs[e],\n", + " )\n", + " sys.stdout.write(\"\")\n", + "\n", + " # return performance metrics for the entire run\n", + " scores = dict()\n", + "\n", + " scores[\"train_errors\"] = train_errors\n", + "\n", + " if val_set:\n", + " scores[\"val_errors\"] = val_errors\n", + "\n", + " if self.classification:\n", + " scores[\"train_accs\"] = train_accs\n", + "\n", + " if val_set:\n", + " scores[\"val_accs\"] = val_accs\n", + "\n", + " return scores\n", + "\n", + " def predict(self, X: np.ndarray, *, threshold=0.5):\n", + " \"\"\"\n", + " Description:\n", + " ------------\n", + " Performs prediction after training of the network has been finished.\n", + "\n", + " Parameters:\n", + " ------------\n", + " I X (np.ndarray): The design matrix, with n rows of p features each\n", + "\n", + " Optional Parameters:\n", + " ------------\n", + " II threshold (float) : sets minimal value for a prediction to be predicted as the positive class\n", + " in classification problems\n", + "\n", + " Returns:\n", + " ------------\n", + " I z (np.ndarray): A prediction vector (row) for each row in our design matrix\n", + " This vector is thresholded if regression=False, meaning that classification results\n", + " in a vector of 1s and 0s, while regressions in an array of decimal numbers\n", + "\n", + " \"\"\"\n", + "\n", + " predict = self._feedforward(X)\n", + "\n", + " if self.classification:\n", + " return np.where(predict > threshold, 1, 0)\n", + " else:\n", + " return predict\n", + "\n", + " def reset_weights(self):\n", + " \"\"\"\n", + " Description:\n", + " ------------\n", + " Resets/Reinitializes the weights in order to train the network for a new problem.\n", + "\n", + " \"\"\"\n", + " if self.seed is not None:\n", + " np.random.seed(self.seed)\n", + "\n", + " self.weights = list()\n", + " for i in range(len(self.dimensions) - 1):\n", + " weight_array = np.random.randn(\n", + " self.dimensions[i] + 1, self.dimensions[i + 1]\n", + " )\n", + " weight_array[0, :] = np.random.randn(self.dimensions[i + 1]) * 0.01\n", + "\n", + " self.weights.append(weight_array)\n", + "\n", + " def _feedforward(self, X: np.ndarray):\n", + " \"\"\"\n", + " Description:\n", + " ------------\n", + " Calculates the activation of each layer starting at the input and ending at the output.\n", + " Each following activation is calculated from a weighted sum of each of the preceeding\n", + " activations (except in the case of the input layer).\n", + "\n", + " Parameters:\n", + " ------------\n", + " I X (np.ndarray): The design matrix, with n rows of p features each\n", + "\n", + " Returns:\n", + " ------------\n", + " I z (np.ndarray): A prediction vector (row) for each row in our design matrix\n", + " \"\"\"\n", + "\n", + " # reset matrices\n", + " self.a_matrices = list()\n", + " self.z_matrices = list()\n", + "\n", + " # if X is just a vector, make it into a matrix\n", + " if len(X.shape) == 1:\n", + " X = X.reshape((1, X.shape[0]))\n", + "\n", + " # Add a coloumn of zeros as the first coloumn of the design matrix, in order\n", + " # to add bias to our data\n", + " bias = np.ones((X.shape[0], 1)) * 0.01\n", + " X = np.hstack([bias, X])\n", + "\n", + " # a^0, the nodes in the input layer (one a^0 for each row in X - where the\n", + " # exponent indicates layer number).\n", + " a = X\n", + " self.a_matrices.append(a)\n", + " self.z_matrices.append(a)\n", + "\n", + " # The feed forward algorithm\n", + " for i in range(len(self.weights)):\n", + " if i < len(self.weights) - 1:\n", + " z = a @ self.weights[i]\n", + " self.z_matrices.append(z)\n", + " a = self.hidden_func(z)\n", + " # bias column again added to the data here\n", + " bias = np.ones((a.shape[0], 1)) * 0.01\n", + " a = np.hstack([bias, a])\n", + " self.a_matrices.append(a)\n", + " else:\n", + " try:\n", + " # a^L, the nodes in our output layers\n", + " z = a @ self.weights[i]\n", + " a = self.output_func(z)\n", + " self.a_matrices.append(a)\n", + " self.z_matrices.append(z)\n", + " except Exception as OverflowError:\n", + " print(\n", + " \"OverflowError in fit() in FFNN\\nHOW TO DEBUG ERROR: Consider lowering your learning rate or scheduler specific parameters such as momentum, or check if your input values need scaling\"\n", + " )\n", + "\n", + " # this will be a^L\n", + " return a\n", + "\n", + " def _backpropagate(self, X, t, lam):\n", + " \"\"\"\n", + " Description:\n", + " ------------\n", + " Performs the backpropagation algorithm. In other words, this method\n", + " calculates the gradient of all the layers starting at the\n", + " output layer, and moving from right to left accumulates the gradient until\n", + " the input layer is reached. Each layers respective weights are updated while\n", + " the algorithm propagates backwards from the output layer (auto-differentation in reverse mode).\n", + "\n", + " Parameters:\n", + " ------------\n", + " I X (np.ndarray): The design matrix, with n rows of p features each.\n", + " II t (np.ndarray): The target vector, with n rows of p targets.\n", + " III lam (float32): regularization parameter used to punish the weights in case of overfitting\n", + "\n", + " Returns:\n", + " ------------\n", + " No return value.\n", + "\n", + " \"\"\"\n", + " out_derivative = derivate(self.output_func)\n", + " hidden_derivative = derivate(self.hidden_func)\n", + "\n", + " for i in range(len(self.weights) - 1, -1, -1):\n", + " # delta terms for output\n", + " if i == len(self.weights) - 1:\n", + " # for multi-class classification\n", + " if (\n", + " self.output_func.__name__ == \"softmax\"\n", + " ):\n", + " delta_matrix = self.a_matrices[i + 1] - t\n", + " # for single class classification\n", + " else:\n", + " cost_func_derivative = grad(self.cost_func(t))\n", + " delta_matrix = out_derivative(\n", + " self.z_matrices[i + 1]\n", + " ) * cost_func_derivative(self.a_matrices[i + 1])\n", + "\n", + " # delta terms for hidden layer\n", + " else:\n", + " delta_matrix = (\n", + " self.weights[i + 1][1:, :] @ delta_matrix.T\n", + " ).T * hidden_derivative(self.z_matrices[i + 1])\n", + "\n", + " # calculate gradient\n", + " gradient_weights = self.a_matrices[i][:, 1:].T @ delta_matrix\n", + " gradient_bias = np.sum(delta_matrix, axis=0).reshape(\n", + " 1, delta_matrix.shape[1]\n", + " )\n", + "\n", + " # regularization term\n", + " gradient_weights += self.weights[i][1:, :] * lam\n", + "\n", + " # use scheduler\n", + " update_matrix = np.vstack(\n", + " [\n", + " self.schedulers_bias[i].update_change(gradient_bias),\n", + " self.schedulers_weight[i].update_change(gradient_weights),\n", + " ]\n", + " )\n", + "\n", + " # update weights and bias\n", + " self.weights[i] -= update_matrix\n", + "\n", + " def _accuracy(self, prediction: np.ndarray, target: np.ndarray):\n", + " \"\"\"\n", + " Description:\n", + " ------------\n", + " Calculates accuracy of given prediction to target\n", + "\n", + " Parameters:\n", + " ------------\n", + " I prediction (np.ndarray): vector of predicitons output network\n", + " (1s and 0s in case of classification, and real numbers in case of regression)\n", + " II target (np.ndarray): vector of true values (What the network ideally should predict)\n", + "\n", + " Returns:\n", + " ------------\n", + " A floating point number representing the percentage of correctly classified instances.\n", + " \"\"\"\n", + " assert prediction.size == target.size\n", + " return np.average((target == prediction))\n", + " def _set_classification(self):\n", + " \"\"\"\n", + " Description:\n", + " ------------\n", + " Decides if FFNN acts as classifier (True) og regressor (False),\n", + " sets self.classification during init()\n", + " \"\"\"\n", + " self.classification = False\n", + " if (\n", + " self.cost_func.__name__ == \"CostLogReg\"\n", + " or self.cost_func.__name__ == \"CostCrossEntropy\"\n", + " ):\n", + " self.classification = True\n", + "\n", + " def _progress_bar(self, progression, **kwargs):\n", + " \"\"\"\n", + " Description:\n", + " ------------\n", + " Displays progress of training\n", + " \"\"\"\n", + " print_length = 40\n", + " num_equals = int(progression * print_length)\n", + " num_not = print_length - num_equals\n", + " arrow = \">\" if num_equals > 0 else \"\"\n", + " bar = \"[\" + \"=\" * (num_equals - 1) + arrow + \"-\" * num_not + \"]\"\n", + " perc_print = self._format(progression * 100, decimals=5)\n", + " line = f\" {bar} {perc_print}% \"\n", + "\n", + " for key in kwargs:\n", + " if not np.isnan(kwargs[key]):\n", + " value = self._format(kwargs[key], decimals=4)\n", + " line += f\"| {key}: {value} \"\n", + " sys.stdout.write(\"\\r\" + line)\n", + " sys.stdout.flush()\n", + " return len(line)\n", + "\n", + " def _format(self, value, decimals=4):\n", + " \"\"\"\n", + " Description:\n", + " ------------\n", + " Formats decimal numbers for progress bar\n", + " \"\"\"\n", + " if value > 0:\n", + " v = value\n", + " elif value < 0:\n", + " v = -10 * value\n", + " else:\n", + " v = 1\n", + " n = 1 + math.floor(math.log10(v))\n", + " if n >= decimals - 1:\n", + " return str(round(value))\n", + " return f\"{value:.{decimals-n-1}f}\"" + ] + }, + { + "cell_type": "markdown", + "id": "2d63ec5b", + "metadata": { + "editable": true + }, + "source": [ + "Before we make a model, we will quickly generate a dataset we can use\n", + "for our linear regression problem as shown below" + ] + }, + { + "cell_type": "code", + "execution_count": 31, + "id": "85a6b185", + "metadata": { + "collapsed": false, + "editable": true + }, + "outputs": [], + "source": [ + "import autograd.numpy as np\n", + "from sklearn.model_selection import train_test_split\n", + "\n", + "def SkrankeFunction(x, y):\n", + " return np.ravel(0 + 1*x + 2*y + 3*x**2 + 4*x*y + 5*y**2)\n", + "\n", + "def create_X(x, y, n):\n", + " if len(x.shape) > 1:\n", + " x = np.ravel(x)\n", + " y = np.ravel(y)\n", + "\n", + " N = len(x)\n", + " l = int((n + 1) * (n + 2) / 2) # Number of elements in beta\n", + " X = np.ones((N, l))\n", + "\n", + " for i in range(1, n + 1):\n", + " q = int((i) * (i + 1) / 2)\n", + " for k in range(i + 1):\n", + " X[:, q + k] = (x ** (i - k)) * (y**k)\n", + "\n", + " return X\n", + "\n", + "step=0.5\n", + "x = np.arange(0, 1, step)\n", + "y = np.arange(0, 1, step)\n", + "x, y = np.meshgrid(x, y)\n", + "target = SkrankeFunction(x, y)\n", + "target = target.reshape(target.shape[0], 1)\n", + "\n", + "poly_degree=3\n", + "X = create_X(x, y, poly_degree)\n", + "\n", + "X_train, X_test, t_train, t_test = train_test_split(X, target)" + ] + }, + { + "cell_type": "markdown", + "id": "f8842209", + "metadata": { + "editable": true + }, + "source": [ + "Now that we have our dataset ready for the regression, we can create\n", + "our regressor. Note that with the seed parameter, we can make sure our\n", + "results stay the same every time we run the neural network. For\n", + "inititialization, we simply specify the dimensions (we wish the amount\n", + "of input nodes to be equal to the datapoints, and the output to\n", + "predict one value)." + ] + }, + { + "cell_type": "code", + "execution_count": 32, + "id": "b7b14d8f", + "metadata": { + "collapsed": false, + "editable": true + }, + "outputs": [], + "source": [ + "input_nodes = X_train.shape[1]\n", + "output_nodes = 1\n", + "\n", + "linear_regression = FFNN((input_nodes, output_nodes), output_func=identity, cost_func=CostOLS, seed=2023)" + ] + }, + { + "cell_type": "markdown", + "id": "3e35cc67", + "metadata": { + "editable": true + }, + "source": [ + "We then fit our model with our training data using the scheduler of our choice." + ] + }, + { + "cell_type": "code", + "execution_count": 33, + "id": "c2b3e7a3", + "metadata": { + "collapsed": false, + "editable": true + }, + "outputs": [], + "source": [ + "linear_regression.reset_weights() # reset weights such that previous runs or reruns don't affect the weights\n", + "\n", + "scheduler = Constant(eta=1e-3)\n", + "scores = linear_regression.fit(X_train, t_train, scheduler)" + ] + }, + { + "cell_type": "markdown", + "id": "9d0eb8b2", + "metadata": { + "editable": true + }, + "source": [ + "Due to the progress bar we can see the MSE (train_error) throughout\n", + "the FFNN's training. Note that the fit() function has some optional\n", + "parameters with defualt arguments. For example, the regularization\n", + "hyperparameter can be left ignored if not needed, and equally the FFNN\n", + "will by default run for 100 epochs. These can easily be changed, such\n", + "as for example:" + ] + }, + { + "cell_type": "code", + "execution_count": 34, + "id": "19fd68fe", + "metadata": { + "collapsed": false, + "editable": true + }, + "outputs": [], + "source": [ + "linear_regression.reset_weights() # reset weights such that previous runs or reruns don't affect the weights\n", + "\n", + "scores = linear_regression.fit(X_train, t_train, scheduler, lam=1e-4, epochs=1000)" + ] + }, + { + "cell_type": "markdown", + "id": "ecbc4c88", + "metadata": { + "editable": true + }, + "source": [ + "We see that given more epochs to train on, the regressor reaches a lower MSE.\n", + "\n", + "Let us then switch to a binary classification. We use a binary\n", + "classification dataset, and follow a similar setup to the regression\n", + "case." + ] + }, + { + "cell_type": "code", + "execution_count": 35, + "id": "c62f0877", + "metadata": { + "collapsed": false, + "editable": true + }, + "outputs": [], + "source": [ + "from sklearn.datasets import load_breast_cancer\n", + "from sklearn.preprocessing import MinMaxScaler\n", + "\n", + "wisconsin = load_breast_cancer()\n", + "X = wisconsin.data\n", + "target = wisconsin.target\n", + "target = target.reshape(target.shape[0], 1)\n", + "\n", + "X_train, X_val, t_train, t_val = train_test_split(X, target)\n", + "\n", + "scaler = MinMaxScaler()\n", + "scaler.fit(X_train)\n", + "X_train = scaler.transform(X_train)\n", + "X_val = scaler.transform(X_val)" + ] + }, + { + "cell_type": "code", + "execution_count": 36, + "id": "5b955b7a", + "metadata": { + "collapsed": false, + "editable": true + }, + "outputs": [], + "source": [ + "input_nodes = X_train.shape[1]\n", + "output_nodes = 1\n", + "\n", + "logistic_regression = FFNN((input_nodes, output_nodes), output_func=sigmoid, cost_func=CostLogReg, seed=2023)" + ] + }, + { + "cell_type": "markdown", + "id": "e4c3e9cc", + "metadata": { + "editable": true + }, + "source": [ + "We will now make use of our validation data by passing it into our fit function as a keyword argument" + ] + }, + { + "cell_type": "code", + "execution_count": 37, + "id": "4e994594", + "metadata": { + "collapsed": false, + "editable": true + }, + "outputs": [], + "source": [ + "logistic_regression.reset_weights() # reset weights such that previous runs or reruns don't affect the weights\n", + "\n", + "scheduler = Adam(eta=1e-3, rho=0.9, rho2=0.999)\n", + "scores = logistic_regression.fit(X_train, t_train, scheduler, epochs=1000, X_val=X_val, t_val=t_val)" + ] + }, + { + "cell_type": "markdown", + "id": "690debd0", + "metadata": { + "editable": true + }, + "source": [ + "Finally, we will create a neural network with 2 hidden layers with activation functions." + ] + }, + { + "cell_type": "code", + "execution_count": 38, + "id": "ea04ae66", + "metadata": { + "collapsed": false, + "editable": true + }, + "outputs": [], + "source": [ + "input_nodes = X_train.shape[1]\n", + "hidden_nodes1 = 100\n", + "hidden_nodes2 = 30\n", + "output_nodes = 1\n", + "\n", + "dims = (input_nodes, hidden_nodes1, hidden_nodes2, output_nodes)\n", + "\n", + "neural_network = FFNN(dims, hidden_func=RELU, output_func=sigmoid, cost_func=CostLogReg, seed=2023)" + ] + }, + { + "cell_type": "code", + "execution_count": 39, + "id": "cd892db0", + "metadata": { + "collapsed": false, + "editable": true + }, + "outputs": [], + "source": [ + "neural_network.reset_weights() # reset weights such that previous runs or reruns don't affect the weights\n", + "\n", + "scheduler = Adam(eta=1e-4, rho=0.9, rho2=0.999)\n", + "scores = neural_network.fit(X_train, t_train, scheduler, epochs=1000, X_val=X_val, t_val=t_val)" + ] + }, + { + "cell_type": "markdown", + "id": "c85e6aa4", + "metadata": { + "editable": true + }, + "source": [ + "### Multiclass classification\n", + "\n", + "Finally, we will demonstrate the use case of multiclass classification\n", + "using our FFNN with the famous MNIST dataset, which contain images of\n", + "digits between the range of 0 to 9." + ] + }, + { + "cell_type": "code", + "execution_count": 40, + "id": "1dc44b4d", + "metadata": { + "collapsed": false, + "editable": true + }, + "outputs": [], + "source": [ + "from sklearn.datasets import load_digits\n", + "\n", + "def onehot(target: np.ndarray):\n", + " onehot = np.zeros((target.size, target.max() + 1))\n", + " onehot[np.arange(target.size), target] = 1\n", + " return onehot\n", + "\n", + "digits = load_digits()\n", + "\n", + "X = digits.data\n", + "target = digits.target\n", + "target = onehot(target)\n", + "\n", + "input_nodes = 64\n", + "hidden_nodes1 = 100\n", + "hidden_nodes2 = 30\n", + "output_nodes = 10\n", + "\n", + "dims = (input_nodes, hidden_nodes1, hidden_nodes2, output_nodes)\n", + "\n", + "multiclass = FFNN(dims, hidden_func=LRELU, output_func=softmax, cost_func=CostCrossEntropy)\n", + "\n", + "multiclass.reset_weights() # reset weights such that previous runs or reruns don't affect the weights\n", + "\n", + "scheduler = Adam(eta=1e-4, rho=0.9, rho2=0.999)\n", + "scores = multiclass.fit(X, target, scheduler, epochs=1000)" + ] + }, + { + "cell_type": "markdown", + "id": "785a6539", + "metadata": { + "editable": true + }, + "source": [ + "## Testing the XOR gate and other gates\n", + "\n", + "Let us now use our code to test the XOR gate." + ] + }, + { + "cell_type": "code", + "execution_count": 41, + "id": "f2137088", + "metadata": { + "collapsed": false, + "editable": true + }, + "outputs": [], + "source": [ + "X = np.array([ [0, 0], [0, 1], [1, 0],[1, 1]],dtype=np.float64)\n", + "\n", + "# The XOR gate\n", + "yXOR = np.array( [[ 0], [1] ,[1], [0]])\n", + "\n", + "input_nodes = X.shape[1]\n", + "output_nodes = 1\n", + "\n", + "logistic_regression = FFNN((input_nodes, output_nodes), output_func=sigmoid, cost_func=CostLogReg, seed=2023)\n", + "logistic_regression.reset_weights() # reset weights such that previous runs or reruns don't affect the weights\n", + "scheduler = Adam(eta=1e-1, rho=0.9, rho2=0.999)\n", + "scores = logistic_regression.fit(X, yXOR, scheduler, epochs=1000)" + ] + }, + { + "cell_type": "markdown", + "id": "6caa78c7", + "metadata": { + "editable": true + }, + "source": [ + "Not bad, but the results depend strongly on the learning reate. Try different learning rates." + ] + } + ], + "metadata": {}, + "nbformat": 4, + "nbformat_minor": 5 +} \ No newline at end of file diff --git a/doc/LectureNotes/_toc.yml b/doc/LectureNotes/_toc.yml index d1e427ac0..989e67705 100644 --- a/doc/LectureNotes/_toc.yml +++ b/doc/LectureNotes/_toc.yml @@ -56,8 +56,10 @@ parts: - file: week40.ipynb - file: week41.ipynb - file: exercisesweek41.ipynb + - file: week42.ipynb - file: exercisesweek42.ipynb - caption: Projects numbered: false chapters: - file: project1.ipynb + - file: project2.ipynb diff --git a/doc/LectureNotes/project2.ipynb b/doc/LectureNotes/project2.ipynb new file mode 100644 index 000000000..b22fbdd58 --- /dev/null +++ b/doc/LectureNotes/project2.ipynb @@ -0,0 +1,554 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "id": "061af572", + "metadata": { + "editable": true + }, + "source": [ + "\n", + "" + ] + }, + { + "cell_type": "markdown", + "id": "22578683", + "metadata": { + "editable": true + }, + "source": [ + "# Project 2 on Machine Learning, deadline November 10 (Midnight)\n", + "**[Data Analysis and Machine Learning FYS-STK3155/FYS4155](http://www.uio.no/studier/emner/matnat/fys/FYS3155/index-eng.html)**, University of Oslo, Norway\n", + "\n", + "Date: **October 14, 2025**" + ] + }, + { + "cell_type": "markdown", + "id": "61fb162f", + "metadata": { + "editable": true + }, + "source": [ + "## Deliverables\n", + "\n", + "First, join a group in canvas with your group partners. Pick an avaliable group for Project 2 in the **People** page.\n", + "\n", + "In canvas, deliver as a group and include:\n", + "\n", + "* A PDF of your report which follows the guidelines covered below and in the week 39 exercises. Additional requirements include:\n", + "\n", + " * It should be around 5000 words, use the word counter in Overleaf for this. This often corresponds to 10-12 pages. References and appendices are excluded from the word count\n", + "\n", + " * It should include around 10-15 figures. You can include more figures in appendices and/or as supplemental material in your repository.\n", + "\n", + "* A comment linking to your github repository (or folder in one of your github repositories) for this project. The repository must include\n", + "\n", + "A PDF file of the report\n", + " * A folder named Code, where you put python files for your functions and notebooks for reproducing your results. Remember to use a seed for generating random data and for train-test splits when generating final results.\n", + "\n", + " * A README file with the name of the group members\n", + "\n", + " * a short description of the project\n", + "\n", + " * a description of how to install the required packages to run your code from a requirements.txt file or similar (such as a plain text description) names and descriptions of the various notebooks in the Code folder and the results they produce" + ] + }, + { + "cell_type": "markdown", + "id": "104c69e1", + "metadata": { + "editable": true + }, + "source": [ + "### Preamble: Note on writing reports, using reference material, AI and other tools\n", + "\n", + "We want you to answer the three different projects by handing in\n", + "reports written like a standard scientific/technical report. The links\n", + "at\n", + "https://github.com/CompPhysics/MachineLearning/tree/master/doc/Projects\n", + "contain more information. There you can find examples of previous\n", + "reports, the projects themselves, how we grade reports etc. How to\n", + "write reports will also be discussed during the various lab\n", + "sessions. Please do ask us if you are in doubt.\n", + "\n", + "When using codes and material from other sources, you should refer to\n", + "these in the bibliography of your report, indicating wherefrom you for\n", + "example got the code, whether this is from the lecture notes,\n", + "softwares like Scikit-Learn, TensorFlow, PyTorch or other\n", + "sources. These sources should always be cited correctly. How to cite\n", + "some of the libraries is often indicated from their corresponding\n", + "GitHub sites or websites, see for example how to cite Scikit-Learn at\n", + "https://scikit-learn.org/dev/about.html.\n", + "\n", + "We enocurage you to use tools like ChatGPT or similar in writing the\n", + "report. If you use for example ChatGPT, please do cite it properly and\n", + "include (if possible) your questions and answers as an addition to the\n", + "report. This can be uploaded to for example your website,\n", + "GitHub/GitLab or similar as supplemental material.\n", + "\n", + "If you would like to study other data sets, feel free to propose other\n", + "sets. What we have proposed here are mere suggestions from our\n", + "side. If you opt for another data set, consider using a set which has\n", + "been studied in the scientific literature. This makes it easier for\n", + "you to compare and analyze your results. Comparing with existing\n", + "results from the scientific literature is also an essential element of\n", + "the scientific discussion. The University of California at Irvine with\n", + "its Machine Learning repository at\n", + "https://archive.ics.uci.edu/ml/index.php is an excellent site to look\n", + "up for examples and inspiration. Kaggle.com is an equally interesting\n", + "site. Feel free to explore these sites." + ] + }, + { + "cell_type": "markdown", + "id": "0d2c42e3", + "metadata": { + "editable": true + }, + "source": [ + "## Classification and Regression, writing our own neural network code\n", + "\n", + "The main aim of this project is to study both classification and\n", + "regression problems by developing our own \n", + "feed-forward neural network (FFNN) code. The exercises from week 41 and 42 (see and ) as well as the lecture material from the same weeks (see and ) should contain enough information for you to get started with writing your own code.\n", + "\n", + "We will also reuse our codes on gradient descent methods from project 1.\n", + "\n", + "The data sets that we propose here are (the default sets)\n", + "\n", + "* Regression (fitting a continuous function). In this part you will need to bring back your results from project 1 and compare these with what you get from your Neural Network code to be developed here. The data sets could be\n", + "\n", + " * The simple one-dimensional function Runge function from project 1, that is $f(x) = \\frac{1}{1+25x^2}$. We recommend using a simpler function when developing your neural network code for regression problems. Feel however free to discuss and study other functions, such as the the two-dimensional Runge function $f(x,y)=\\left[(10x - 5)^2 + (10y - 5)^2 + 1 \\right]^{-1}$, or even more complicated two-dimensional functions (see the supplementary material of for an extensive list of two-dimensional functions). \n", + "\n", + "* Classification.\n", + "\n", + " * We will consider a multiclass classification problem given by the full MNIST data set. The full data set is at .\n", + "\n", + "We will start with a regression problem and we will reuse our codes on gradient descent methods from project 1." + ] + }, + { + "cell_type": "markdown", + "id": "d8baab67", + "metadata": { + "editable": true + }, + "source": [ + "### Part a): Analytical warm-up\n", + "\n", + "When using our gradient machinery from project 1, we will need the expressions for the cost/loss functions and their respective\n", + "gradients. The functions whose gradients we need are:\n", + "1. The mean-squared error (MSE) with and without the $L_1$ and $L_2$ norms (regression problems)\n", + "\n", + "2. The binary cross entropy (aka log loss) for binary classification problems with and without $L_1$ and $L_2$ norms\n", + "\n", + "3. The multiclass cross entropy cost/loss function (aka Softmax cross entropy or just Softmax loss function)\n", + "\n", + "Set up these three cost/loss functions and their respective derivatives and explain the various terms. In this project you will however only use the MSE and the Softmax cross entropy.\n", + "\n", + "We will test three activation functions for our neural network setup, these are the \n", + "1. The Sigmoid (aka **logit**) function,\n", + "\n", + "2. the RELU function and\n", + "\n", + "3. the Leaky RELU function\n", + "\n", + "Set up their expressions and their first derivatives.\n", + "You may consult the lecture notes (with codes and more) from week 42 at ." + ] + }, + { + "cell_type": "markdown", + "id": "87e7ed71", + "metadata": { + "editable": true + }, + "source": [ + "### Reminder about the gradient machinery from project 1\n", + "\n", + "In the setup of a neural network code you will need your gradient descent codes from\n", + "project 1. For neural networks we will recommend using stochastic\n", + "gradient descent with either the RMSprop or the ADAM algorithms for\n", + "updating the learning rates. But you should feel free to try plain gradient descent as well.\n", + "\n", + "We recommend reading chapter 8 on optimization from the textbook of\n", + "Goodfellow, Bengio and Courville at\n", + ". This chapter contains many\n", + "useful insights and discussions on the optimization part of machine\n", + "learning. A useful reference on the back progagation algorithm is\n", + "Nielsen's book at . \n", + "\n", + "You will find the Python [Seaborn\n", + "package](https://seaborn.pydata.org/generated/seaborn.heatmap.html)\n", + "useful when plotting the results as function of the learning rate\n", + "$\\eta$ and the hyper-parameter $\\lambda$ ." + ] + }, + { + "cell_type": "markdown", + "id": "5a26b6ad", + "metadata": { + "editable": true + }, + "source": [ + "### Part b): Writing your own Neural Network code\n", + "\n", + "Your aim now, and this is the central part of this project, is to\n", + "write your own FFNN code implementing the back\n", + "propagation algorithm discussed in the lecture slides from week 41 at and week 42 at .\n", + "\n", + "We will focus on a regression problem first, using the one-dimensional Runge function" + ] + }, + { + "cell_type": "markdown", + "id": "096fe6c4", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "f(x) = \\frac{1}{1+25x^2},\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "fd986596", + "metadata": { + "editable": true + }, + "source": [ + "from project 1.\n", + "\n", + "Use only the mean-squared error as cost function (no regularization terms) and \n", + "write an FFNN code for a regression problem with a flexible number of hidden\n", + "layers and nodes using only the Sigmoid function as activation function for\n", + "the hidden layers. Initialize the weights using a normal\n", + "distribution. How would you initialize the biases? And which\n", + "activation function would you select for the final output layer?\n", + "And how would you set up your design/feature matrix? Hint: does it have to represent a polynomial approximation as you did in project 1? \n", + "\n", + "Train your network and compare the results with those from your OLS\n", + "regression code from project 1 using the one-dimensional Runge\n", + "function. When comparing your neural network code with the OLS\n", + "results from project 1, use the same data sets which gave you the best\n", + "MSE score. Moreover, use the polynomial order from project 1 that gave you the\n", + "best result. Compare these results with your neural network with one\n", + "and two hidden layers using $50$ and $100$ hidden nodes, respectively.\n", + "\n", + "Comment your results and give a critical discussion of the results\n", + "obtained with the OLS code from project 1 and your own neural network\n", + "code. Make an analysis of the learning rates employed to find the\n", + "optimal MSE score. Test both stochastic gradient descent\n", + "with RMSprop and ADAM and plain gradient descent with different\n", + "learning rates.\n", + "\n", + "You should, as you did in project 1, scale your data." + ] + }, + { + "cell_type": "markdown", + "id": "e853d4b6", + "metadata": { + "editable": true + }, + "source": [ + "### Part c): Testing against other software libraries\n", + "\n", + "You should test your results against a similar code using **Scikit-Learn** (see the examples in the above lecture notes from weeks 41 and 42) or **tensorflow/keras** or **Pytorch** (for Pytorch, see Raschka et al.'s text chapters 12 and 13). \n", + "\n", + "Furthermore, you should also test that your derivatives are correctly\n", + "calculated using automatic differentiation, using for example the\n", + "**Autograd** library or the **JAX** library. It is optional to implement\n", + "these libraries for the present project. In this project they serve as\n", + "useful tests of our derivatives." + ] + }, + { + "cell_type": "markdown", + "id": "fc2d413b", + "metadata": { + "editable": true + }, + "source": [ + "### Part d): Testing different activation functions and depths of the neural network\n", + "\n", + "You should also test different activation functions for the hidden\n", + "layers. Try out the Sigmoid, the RELU and the Leaky RELU functions and\n", + "discuss your results. Test your results as functions of the number of hidden layers and nodes. Do you see signs of overfitting?\n", + "It is optional in this project to perform a bias-variance trade-off analysis." + ] + }, + { + "cell_type": "markdown", + "id": "e6821051", + "metadata": { + "editable": true + }, + "source": [ + "### Part e): Testing different norms\n", + "\n", + "Finally, still using the one-dimensional Runge function, add now the\n", + "hyperparameters $\\lambda$ with the $L_2$ and $L_1$ norms. Find the\n", + "optimal results for the hyperparameters $\\lambda$ and the learning\n", + "rates $\\eta$ and neural network architecture and compare the $L_2$ results with Ridge regression from\n", + "project 1 and the $L_1$ results with the Lasso calculations of project 1.\n", + "Use again the same data sets and the best results from project 1 in your comparisons." + ] + }, + { + "cell_type": "markdown", + "id": "cba72d68", + "metadata": { + "editable": true + }, + "source": [ + "### Part f): Classification analysis using neural networks\n", + "\n", + "With a well-written code it should now be easy to change the\n", + "activation function for the output layer.\n", + "\n", + "Here we will change the cost function for our neural network code\n", + "developed in parts b), d) and e) in order to perform a classification\n", + "analysis. The classification problem we will study is the multiclass\n", + "MNIST problem, see the description of the full data set at\n", + ". We will use the Softmax cross entropy function discussed in a). \n", + "The MNIST data set discussed in the lecture notes from week 42 is a downscaled variant of the full dataset. \n", + "\n", + "Feel free to suggest other data sets. If you find the classic MNIST data set somewhat limited, feel free to try the \n", + "MNIST-Fashion data set at for example .\n", + "\n", + "To set up the data set, the following python programs may be useful" + ] + }, + { + "cell_type": "code", + "execution_count": 1, + "id": "e16fb528", + "metadata": { + "collapsed": false, + "editable": true + }, + "outputs": [], + "source": [ + "from sklearn.datasets import fetch_openml\n", + "\n", + "# Fetch the MNIST dataset\n", + "mnist = fetch_openml('mnist_784', version=1, as_frame=False, parser='auto')\n", + "\n", + "# Extract data (features) and target (labels)\n", + "X = mnist.data\n", + "y = mnist.target" + ] + }, + { + "cell_type": "markdown", + "id": "73599f42", + "metadata": { + "editable": true + }, + "source": [ + "You should consider scaling the data. The Pixel values in MNIST range from 0 to 255. Scaling them to a 0-1 range can improve the performance of some models. That is, you could implement the following scaling" + ] + }, + { + "cell_type": "code", + "execution_count": 2, + "id": "f1a639ef", + "metadata": { + "collapsed": false, + "editable": true + }, + "outputs": [], + "source": [ + "X = X / 255.0" + ] + }, + { + "cell_type": "markdown", + "id": "90fb7b41", + "metadata": { + "editable": true + }, + "source": [ + "And then perform the standard train-test splitting" + ] + }, + { + "cell_type": "code", + "execution_count": 3, + "id": "424af629", + "metadata": { + "collapsed": false, + "editable": true + }, + "outputs": [], + "source": [ + "from sklearn.model_selection import train_test_split\n", + "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)" + ] + }, + { + "cell_type": "markdown", + "id": "3c006080", + "metadata": { + "editable": true + }, + "source": [ + "To measure the performance of our classification problem we will use the\n", + "so-called *accuracy* score. The accuracy is as you would expect just\n", + "the number of correctly guessed targets $t_i$ divided by the total\n", + "number of targets, that is" + ] + }, + { + "cell_type": "markdown", + "id": "a18ddd54", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\text{Accuracy} = \\frac{\\sum_{i=1}^n I(t_i = y_i)}{n} ,\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "1a1afaf9", + "metadata": { + "editable": true + }, + "source": [ + "where $I$ is the indicator function, $1$ if $t_i = y_i$ and $0$\n", + "otherwise if we have a binary classification problem. Here $t_i$\n", + "represents the target and $y_i$ the outputs of your FFNN code and $n$ is simply the number of targets $t_i$.\n", + "\n", + "Discuss your results and give a critical analysis of the various parameters, including hyper-parameters like the learning rates and the regularization parameter $\\lambda$, various activation functions, number of hidden layers and nodes and activation functions. \n", + "\n", + "Again, we strongly recommend that you compare your own neural Network\n", + "code for classification and pertinent results against a similar code using **Scikit-Learn** or **tensorflow/keras** or **pytorch**.\n", + "\n", + "If you have time, you can use the functionality of **scikit-learn** and compare your neural network results with those from Logistic regression. This is optional.\n", + "The weblink here compares logistic regression and FFNN using the so-called MNIST data set. You may find several useful hints and ideas from this article. Your neural network code can implement the equivalent of logistic regression by simply setting the number of hidden layers to zero. \n", + "\n", + "If you wish to compare with say Logisti Regression from **scikit-learn**, the following code uses the above data set" + ] + }, + { + "cell_type": "code", + "execution_count": 4, + "id": "3c37cbaf", + "metadata": { + "collapsed": false, + "editable": true + }, + "outputs": [], + "source": [ + "from sklearn.linear_model import LogisticRegression\n", + "# Initialize the model\n", + "model = LogisticRegression(solver='saga', multi_class='multinomial', max_iter=1000, random_state=42)\n", + "# Train the model\n", + "model.fit(X_train, y_train)\n", + "from sklearn.metrics import accuracy_score\n", + "# Make predictions on the test set\n", + "y_pred = model.predict(X_test)\n", + "# Calculate accuracy\n", + "accuracy = accuracy_score(y_test, y_pred)\n", + "print(f\"Model Accuracy: {accuracy:.4f}\")" + ] + }, + { + "cell_type": "markdown", + "id": "106b9303", + "metadata": { + "editable": true + }, + "source": [ + "### Part g) Critical evaluation of the various algorithms\n", + "\n", + "After all these glorious calculations, you should now summarize the\n", + "various algorithms and come with a critical evaluation of their pros\n", + "and cons. Which algorithm works best for the regression case and which\n", + "is best for the classification case. These codes can also be part of\n", + "your final project 3, but now applied to other data sets." + ] + }, + { + "cell_type": "markdown", + "id": "55da0d7f", + "metadata": { + "editable": true + }, + "source": [ + "## Background literature\n", + "\n", + "1. The text of Michael Nielsen is highly recommended, see Nielsen's book at . It is an excellent read.\n", + "\n", + "2. Goodfellow, Bengio and Courville, Deep Learning at . Here we recommend chapters 6, 7 and 8\n", + "\n", + "3. Raschka et al. at . Here we recommend chapters 11, 12 and 13." + ] + }, + { + "cell_type": "markdown", + "id": "d3731e2c", + "metadata": { + "editable": true + }, + "source": [ + "## Introduction to numerical projects\n", + "\n", + "Here follows a brief recipe and recommendation on how to write a report for each\n", + "project.\n", + "\n", + " * Give a short description of the nature of the problem and the eventual numerical methods you have used.\n", + "\n", + " * Describe the algorithm you have used and/or developed. Here you may find it convenient to use pseudocoding. In many cases you can describe the algorithm in the program itself.\n", + "\n", + " * Include the source code of your program. Comment your program properly.\n", + "\n", + " * If possible, try to find analytic solutions, or known limits in order to test your program when developing the code.\n", + "\n", + " * Include your results either in figure form or in a table. Remember to label your results. All tables and figures should have relevant captions and labels on the axes.\n", + "\n", + " * Try to evaluate the reliabilty and numerical stability/precision of your results. If possible, include a qualitative and/or quantitative discussion of the numerical stability, eventual loss of precision etc.\n", + "\n", + " * Try to give an interpretation of you results in your answers to the problems.\n", + "\n", + " * Critique: if possible include your comments and reflections about the exercise, whether you felt you learnt something, ideas for improvements and other thoughts you've made when solving the exercise. We wish to keep this course at the interactive level and your comments can help us improve it.\n", + "\n", + " * Try to establish a practice where you log your work at the computerlab. You may find such a logbook very handy at later stages in your work, especially when you don't properly remember what a previous test version of your program did. Here you could also record the time spent on solving the exercise, various algorithms you may have tested or other topics which you feel worthy of mentioning." + ] + }, + { + "cell_type": "markdown", + "id": "6c7c5340", + "metadata": { + "editable": true + }, + "source": [ + "## Format for electronic delivery of report and programs\n", + "\n", + "The preferred format for the report is a PDF file. You can also use DOC or postscript formats or as an ipython notebook file. As programming language we prefer that you choose between C/C++, Fortran2008 or Python. The following prescription should be followed when preparing the report:\n", + "\n", + " * Use Canvas to hand in your projects, log in at with your normal UiO username and password.\n", + "\n", + " * Upload **only** the report file or the link to your GitHub/GitLab or similar typo of repos! For the source code file(s) you have developed please provide us with your link to your GitHub/GitLab or similar domain. The report file should include all of your discussions and a list of the codes you have developed. Do not include library files which are available at the course homepage, unless you have made specific changes to them.\n", + "\n", + " * In your GitHub/GitLab or similar repository, please include a folder which contains selected results. These can be in the form of output from your code for a selected set of runs and input parameters.\n", + "\n", + "Finally, \n", + "we encourage you to collaborate. Optimal working groups consist of \n", + "2-3 students. You can then hand in a common report." + ] + } + ], + "metadata": {}, + "nbformat": 4, + "nbformat_minor": 5 +} diff --git a/doc/LectureNotes/week42.ipynb b/doc/LectureNotes/week42.ipynb new file mode 100644 index 000000000..af01abbf4 --- /dev/null +++ b/doc/LectureNotes/week42.ipynb @@ -0,0 +1,5950 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "id": "7dcf8613", + "metadata": { + "editable": true + }, + "source": [ + "\n", + "" + ] + }, + { + "cell_type": "markdown", + "id": "98019924", + "metadata": { + "editable": true + }, + "source": [ + "# Week 42 Constructing a Neural Network code with examples\n", + "**Morten Hjorth-Jensen**, Department of Physics, University of Oslo, Norway\n", + "\n", + "Date: **October 13-17, 2025**" + ] + }, + { + "cell_type": "markdown", + "id": "ab0e660a", + "metadata": { + "editable": true + }, + "source": [ + "## Lecture October 13, 2025\n", + "1. Building our own Feed-forward Neural Network and discussion of project 2\n", + "\n", + "2. Project 2 is available at " + ] + }, + { + "cell_type": "markdown", + "id": "bcea021f", + "metadata": { + "editable": true + }, + "source": [ + "## Readings and videos\n", + "1. These lecture notes\n", + "\n", + "\n", + "\n", + "2. For a more in depth discussion on neural networks we recommend Goodfellow et al chapters 6 and 7. For the optimization part, see chapter 8. \n", + "\n", + "3. Neural Networks demystified at \n", + "\n", + "4. Building Neural Networks from scratch at \n", + "\n", + "5. Video on Neural Networks at \n", + "\n", + "6. Video on the back propagation algorithm at \n", + "\n", + "I also recommend Michael Nielsen's intuitive approach to the neural networks and the universal approximation theorem, see the slides at ." + ] + }, + { + "cell_type": "markdown", + "id": "fd77cd6b", + "metadata": { + "editable": true + }, + "source": [ + "## Material for the lab sessions on Tuesday and Wednesday\n", + "1. Exercises on writing a code for neural networks, back propagation part, see exercises for week 42 at \n", + "\n", + "2. Discussion of project 2" + ] + }, + { + "cell_type": "markdown", + "id": "7d93059a", + "metadata": { + "editable": true + }, + "source": [ + "## Lecture material: Writing a code which implements a feed-forward neural network\n", + "\n", + "Last week we discussed the basics of neural networks and deep learning\n", + "and the basics of automatic differentiation. We looked also at\n", + "examples on how compute the parameters of a simple network with scalar\n", + "inputs and ouputs and no or just one hidden layers.\n", + "\n", + "We ended our discussions with the derivation of the equations for a\n", + "neural network with one hidden layers and two input variables and two\n", + "hidden nodes but only one output node. We did almost finish the derivation of the back propagation algorithm." + ] + }, + { + "cell_type": "markdown", + "id": "5156bf60", + "metadata": { + "editable": true + }, + "source": [ + "## Mathematics of deep learning\n", + "\n", + "**Two recent books online.**\n", + "\n", + "1. [The Modern Mathematics of Deep Learning, by Julius Berner, Philipp Grohs, Gitta Kutyniok, Philipp Petersen](https://arxiv.org/abs/2105.04026), published as [Mathematical Aspects of Deep Learning, pp. 1-111. Cambridge University Press, 2022](https://doi.org/10.1017/9781009025096.002)\n", + "\n", + "2. [Mathematical Introduction to Deep Learning: Methods, Implementations, and Theory, Arnulf Jentzen, Benno Kuckuck, Philippe von Wurstemberger](https://doi.org/10.48550/arXiv.2310.20360)" + ] + }, + { + "cell_type": "markdown", + "id": "4f5d5b27", + "metadata": { + "editable": true + }, + "source": [ + "## Reminder on books with hands-on material and codes\n", + "* [Sebastian Rashcka et al, Machine learning with Sickit-Learn and PyTorch](https://sebastianraschka.com/blog/2022/ml-pytorch-book.html)" + ] + }, + { + "cell_type": "markdown", + "id": "0a490014", + "metadata": { + "editable": true + }, + "source": [ + "## Reading recommendations\n", + "\n", + "1. Rashkca et al., chapter 11, jupyter-notebook sent separately, from [GitHub](https://github.com/rasbt/machine-learning-book)\n", + "\n", + "2. Goodfellow et al, chapter 6 and 7 contain most of the neural network background." + ] + }, + { + "cell_type": "markdown", + "id": "9fa74383", + "metadata": { + "editable": true + }, + "source": [ + "## Reminder from last week: First network example, simple percepetron with one input\n", + "\n", + "As yet another example we define now a simple perceptron model with\n", + "all quantities given by scalars. We consider only one input variable\n", + "$x$ and one target value $y$. We define an activation function\n", + "$\\sigma_1$ which takes as input" + ] + }, + { + "cell_type": "markdown", + "id": "417ce076", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "z_1 = w_1x+b_1,\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "07ed6afb", + "metadata": { + "editable": true + }, + "source": [ + "where $w_1$ is the weight and $b_1$ is the bias. These are the\n", + "parameters we want to optimize. The output is $a_1=\\sigma(z_1)$ (see\n", + "graph from whiteboard notes). This output is then fed into the\n", + "**cost/loss** function, which we here for the sake of simplicity just\n", + "define as the squared error" + ] + }, + { + "cell_type": "markdown", + "id": "3b926536", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "C(x;w_1,b_1)=\\frac{1}{2}(a_1-y)^2.\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "0e1bd556", + "metadata": { + "editable": true + }, + "source": [ + "## Layout of a simple neural network with no hidden layer\n", + "\n", + "\n", + "\n", + "\n", + "

    Figure 1:

    \n", + "" + ] + }, + { + "cell_type": "markdown", + "id": "edebc935", + "metadata": { + "editable": true + }, + "source": [ + "## Optimizing the parameters\n", + "\n", + "In setting up the feed forward and back propagation parts of the\n", + "algorithm, we need now the derivative of the various variables we want\n", + "to train.\n", + "\n", + "We need" + ] + }, + { + "cell_type": "markdown", + "id": "3e2794ac", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\frac{\\partial C}{\\partial w_1} \\hspace{0.1cm}\\mathrm{and}\\hspace{0.1cm}\\frac{\\partial C}{\\partial b_1}.\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "54f91430", + "metadata": { + "editable": true + }, + "source": [ + "Using the chain rule we find" + ] + }, + { + "cell_type": "markdown", + "id": "f1e0a166", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\frac{\\partial C}{\\partial w_1}=\\frac{\\partial C}{\\partial a_1}\\frac{\\partial a_1}{\\partial z_1}\\frac{\\partial z_1}{\\partial w_1}=(a_1-y)\\sigma_1'x,\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "ad62ed90", + "metadata": { + "editable": true + }, + "source": [ + "and" + ] + }, + { + "cell_type": "markdown", + "id": "57dac362", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\frac{\\partial C}{\\partial b_1}=\\frac{\\partial C}{\\partial a_1}\\frac{\\partial a_1}{\\partial z_1}\\frac{\\partial z_1}{\\partial b_1}=(a_1-y)\\sigma_1',\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "a4dc4050", + "metadata": { + "editable": true + }, + "source": [ + "which we later will just define as" + ] + }, + { + "cell_type": "markdown", + "id": "b223c83d", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\frac{\\partial C}{\\partial a_1}\\frac{\\partial a_1}{\\partial z_1}=\\delta_1.\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "a1c9b5e7", + "metadata": { + "editable": true + }, + "source": [ + "## Adding a hidden layer\n", + "\n", + "We change our simple model to (see graph)\n", + "a network with just one hidden layer but with scalar variables only.\n", + "\n", + "Our output variable changes to $a_2$ and $a_1$ is now the output from the hidden node and $a_0=x$.\n", + "We have then" + ] + }, + { + "cell_type": "markdown", + "id": "9c3b03c0", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "z_1 = w_1a_0+b_1 \\hspace{0.1cm} \\wedge a_1 = \\sigma_1(z_1),\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "186485f2", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "z_2 = w_2a_1+b_2 \\hspace{0.1cm} \\wedge a_2 = \\sigma_2(z_2),\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "c2f1b30b", + "metadata": { + "editable": true + }, + "source": [ + "and the cost function" + ] + }, + { + "cell_type": "markdown", + "id": "13142fad", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "C(x;\\boldsymbol{\\Theta})=\\frac{1}{2}(a_2-y)^2,\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "b441f95a", + "metadata": { + "editable": true + }, + "source": [ + "with $\\boldsymbol{\\Theta}=[w_1,w_2,b_1,b_2]$." + ] + }, + { + "cell_type": "markdown", + "id": "46e57fd2", + "metadata": { + "editable": true + }, + "source": [ + "## Layout of a simple neural network with one hidden layer\n", + "\n", + "\n", + "\n", + "\n", + "

    Figure 1:

    \n", + "" + ] + }, + { + "cell_type": "markdown", + "id": "50158555", + "metadata": { + "editable": true + }, + "source": [ + "## The derivatives\n", + "\n", + "The derivatives are now, using the chain rule again" + ] + }, + { + "cell_type": "markdown", + "id": "4174ea9e", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\frac{\\partial C}{\\partial w_2}=\\frac{\\partial C}{\\partial a_2}\\frac{\\partial a_2}{\\partial z_2}\\frac{\\partial z_2}{\\partial w_2}=(a_2-y)\\sigma_2'a_1=\\delta_2a_1,\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "f7a29d74", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\frac{\\partial C}{\\partial b_2}=\\frac{\\partial C}{\\partial a_2}\\frac{\\partial a_2}{\\partial z_2}\\frac{\\partial z_2}{\\partial b_2}=(a_2-y)\\sigma_2'=\\delta_2,\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "cb1a387b", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\frac{\\partial C}{\\partial w_1}=\\frac{\\partial C}{\\partial a_2}\\frac{\\partial a_2}{\\partial z_2}\\frac{\\partial z_2}{\\partial a_1}\\frac{\\partial a_1}{\\partial z_1}\\frac{\\partial z_1}{\\partial w_1}=(a_2-y)\\sigma_2'a_1\\sigma_1'a_0,\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "125bcb29", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\frac{\\partial C}{\\partial b_1}=\\frac{\\partial C}{\\partial a_2}\\frac{\\partial a_2}{\\partial z_2}\\frac{\\partial z_2}{\\partial a_1}\\frac{\\partial a_1}{\\partial z_1}\\frac{\\partial z_1}{\\partial b_1}=(a_2-y)\\sigma_2'\\sigma_1'=\\delta_1.\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "76272ae9", + "metadata": { + "editable": true + }, + "source": [ + "Can you generalize this to more than one hidden layer?" + ] + }, + { + "cell_type": "markdown", + "id": "7e0d1157", + "metadata": { + "editable": true + }, + "source": [ + "## Important observations\n", + "\n", + "From the above equations we see that the derivatives of the activation\n", + "functions play a central role. If they vanish, the training may\n", + "stop. This is called the vanishing gradient problem, see discussions below. If they become\n", + "large, the parameters $w_i$ and $b_i$ may simply go to infinity. This\n", + "is referenced as the exploding gradient problem." + ] + }, + { + "cell_type": "markdown", + "id": "53dfc3bd", + "metadata": { + "editable": true + }, + "source": [ + "## The training\n", + "\n", + "The training of the parameters is done through various gradient descent approximations with" + ] + }, + { + "cell_type": "markdown", + "id": "126420bb", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "w_{i}\\leftarrow w_{i}- \\eta \\delta_i a_{i-1},\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "2114b4ba", + "metadata": { + "editable": true + }, + "source": [ + "and" + ] + }, + { + "cell_type": "markdown", + "id": "7d3549bb", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "b_i \\leftarrow b_i-\\eta \\delta_i,\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "80265b39", + "metadata": { + "editable": true + }, + "source": [ + "with $\\eta$ is the learning rate.\n", + "\n", + "One iteration consists of one feed forward step and one back-propagation step. Each back-propagation step does one update of the parameters $\\boldsymbol{\\Theta}$.\n", + "\n", + "For the first hidden layer $a_{i-1}=a_0=x$ for this simple model." + ] + }, + { + "cell_type": "markdown", + "id": "df2f78ac", + "metadata": { + "editable": true + }, + "source": [ + "## Code example\n", + "\n", + "The code here implements the above model with one hidden layer and\n", + "scalar variables for the same function we studied in the previous\n", + "example. The code is however set up so that we can add multiple\n", + "inputs $x$ and target values $y$. Note also that we have the\n", + "possibility of defining a feature matrix $\\boldsymbol{X}$ with more than just\n", + "one column for the input values. This will turn useful in our next example. We have also defined matrices and vectors for all of our operations although it is not necessary here." + ] + }, + { + "cell_type": "code", + "execution_count": 1, + "id": "0ec652bf", + "metadata": { + "collapsed": false, + "editable": true + }, + "outputs": [], + "source": [ + "import numpy as np\n", + "# We use the Sigmoid function as activation function\n", + "def sigmoid(z):\n", + " return 1.0/(1.0+np.exp(-z))\n", + "\n", + "def forwardpropagation(x):\n", + " # weighted sum of inputs to the hidden layer\n", + " z_1 = np.matmul(x, w_1) + b_1\n", + " # activation in the hidden layer\n", + " a_1 = sigmoid(z_1)\n", + " # weighted sum of inputs to the output layer\n", + " z_2 = np.matmul(a_1, w_2) + b_2\n", + " a_2 = z_2\n", + " return a_1, a_2\n", + "\n", + "def backpropagation(x, y):\n", + " a_1, a_2 = forwardpropagation(x)\n", + " # parameter delta for the output layer, note that a_2=z_2 and its derivative wrt z_2 is just 1\n", + " delta_2 = a_2 - y\n", + " print(0.5*((a_2-y)**2))\n", + " # delta for the hidden layer\n", + " delta_1 = np.matmul(delta_2, w_2.T) * a_1 * (1 - a_1)\n", + " # gradients for the output layer\n", + " output_weights_gradient = np.matmul(a_1.T, delta_2)\n", + " output_bias_gradient = np.sum(delta_2, axis=0)\n", + " # gradient for the hidden layer\n", + " hidden_weights_gradient = np.matmul(x.T, delta_1)\n", + " hidden_bias_gradient = np.sum(delta_1, axis=0)\n", + " return output_weights_gradient, output_bias_gradient, hidden_weights_gradient, hidden_bias_gradient\n", + "\n", + "\n", + "# ensure the same random numbers appear every time\n", + "np.random.seed(0)\n", + "# Input variable\n", + "x = np.array([4.0],dtype=np.float64)\n", + "# Target values\n", + "y = 2*x+1.0 \n", + "\n", + "# Defining the neural network, only scalars here\n", + "n_inputs = x.shape\n", + "n_features = 1\n", + "n_hidden_neurons = 1\n", + "n_outputs = 1\n", + "\n", + "# Initialize the network\n", + "# weights and bias in the hidden layer\n", + "w_1 = np.random.randn(n_features, n_hidden_neurons)\n", + "b_1 = np.zeros(n_hidden_neurons) + 0.01\n", + "\n", + "# weights and bias in the output layer\n", + "w_2 = np.random.randn(n_hidden_neurons, n_outputs)\n", + "b_2 = np.zeros(n_outputs) + 0.01\n", + "\n", + "eta = 0.1\n", + "for i in range(50):\n", + " # calculate gradients\n", + " derivW2, derivB2, derivW1, derivB1 = backpropagation(x, y)\n", + " # update weights and biases\n", + " w_2 -= eta * derivW2\n", + " b_2 -= eta * derivB2\n", + " w_1 -= eta * derivW1\n", + " b_1 -= eta * derivB1" + ] + }, + { + "cell_type": "markdown", + "id": "c382e2f2", + "metadata": { + "editable": true + }, + "source": [ + "We see that after some few iterations (the results do depend on the learning rate however), we get an error which is rather small." + ] + }, + { + "cell_type": "markdown", + "id": "1bec2330", + "metadata": { + "editable": true + }, + "source": [ + "## Simple neural network and the back propagation equations\n", + "\n", + "Let us now try to increase our level of ambition and attempt at setting \n", + "up the equations for a neural network with two input nodes, one hidden\n", + "layer with two hidden nodes and one output layer with one output node/neuron only (see graph)..\n", + "\n", + "We need to define the following parameters and variables with the input layer (layer $(0)$) \n", + "where we label the nodes $x_1$ and $x_2$" + ] + }, + { + "cell_type": "markdown", + "id": "165fb3f3", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "x_1 = a_1^{(0)} \\wedge x_2 = a_2^{(0)}.\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "7ae8de36", + "metadata": { + "editable": true + }, + "source": [ + "The hidden layer (layer $(1)$) has nodes which yield the outputs $a_1^{(1)}$ and $a_2^{(1)}$) with weight $\\boldsymbol{w}$ and bias $\\boldsymbol{b}$ parameters" + ] + }, + { + "cell_type": "markdown", + "id": "9a2729ab", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "w_{ij}^{(1)}=\\left\\{w_{11}^{(1)},w_{12}^{(1)},w_{21}^{(1)},w_{22}^{(1)}\\right\\} \\wedge b^{(1)}=\\left\\{b_1^{(1)},b_2^{(1)}\\right\\}.\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "372f0fac", + "metadata": { + "editable": true + }, + "source": [ + "## Layout of a simple neural network with two input nodes, one hidden layer with two hidden noeds and one output node\n", + "\n", + "\n", + "\n", + "\n", + "

    Figure 1:

    \n", + "" + ] + }, + { + "cell_type": "markdown", + "id": "d4f432cc", + "metadata": { + "editable": true + }, + "source": [ + "## The ouput layer\n", + "\n", + "We have the ouput layer given by layer label $(2)$ with output $a^{(2)}$ and weights and biases to be determined given by the variables" + ] + }, + { + "cell_type": "markdown", + "id": "064af49b", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "w_{i}^{(2)}=\\left\\{w_{1}^{(2)},w_{2}^{(2)}\\right\\} \\wedge b^{(2)}.\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "91115cc6", + "metadata": { + "editable": true + }, + "source": [ + "Our output is $\\tilde{y}=a^{(2)}$ and we define a generic cost function $C(a^{(2)},y;\\boldsymbol{\\Theta})$ where $y$ is the target value (a scalar here).\n", + "The parameters we need to optimize are given by" + ] + }, + { + "cell_type": "markdown", + "id": "c38d1990", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\boldsymbol{\\Theta}=\\left\\{w_{11}^{(1)},w_{12}^{(1)},w_{21}^{(1)},w_{22}^{(1)},w_{1}^{(2)},w_{2}^{(2)},b_1^{(1)},b_2^{(1)},b^{(2)}\\right\\}.\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "43de152e", + "metadata": { + "editable": true + }, + "source": [ + "## Compact expressions\n", + "\n", + "We can define the inputs to the activation functions for the various layers in terms of various matrix-vector multiplications and vector additions.\n", + "The inputs to the first hidden layer are" + ] + }, + { + "cell_type": "markdown", + "id": "f935912b", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\begin{bmatrix}z_1^{(1)} \\\\ z_2^{(1)} \\end{bmatrix}=\\left(\\begin{bmatrix}w_{11}^{(1)} & w_{12}^{(1)}\\\\ w_{21}^{(1)} &w_{22}^{(1)} \\end{bmatrix}\\right)^{T}\\begin{bmatrix}a_1^{(0)} \\\\ a_2^{(0)} \\end{bmatrix}+\\begin{bmatrix}b_1^{(1)} \\\\ b_2^{(1)} \\end{bmatrix},\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "15b380de", + "metadata": { + "editable": true + }, + "source": [ + "with outputs" + ] + }, + { + "cell_type": "markdown", + "id": "302da250", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\begin{bmatrix}a_1^{(1)} \\\\ a_2^{(1)} \\end{bmatrix}=\\begin{bmatrix}\\sigma^{(1)}(z_1^{(1)}) \\\\ \\sigma^{(1)}(z_2^{(1)}) \\end{bmatrix}.\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "3faf23cf", + "metadata": { + "editable": true + }, + "source": [ + "## Output layer\n", + "\n", + "For the final output layer we have the inputs to the final activation function" + ] + }, + { + "cell_type": "markdown", + "id": "4fe83376", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "z^{(2)} = w_{1}^{(2)}a_1^{(1)} +w_{2}^{(2)}a_2^{(1)}+b^{(2)},\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "9bee26b1", + "metadata": { + "editable": true + }, + "source": [ + "resulting in the output" + ] + }, + { + "cell_type": "markdown", + "id": "9affa08f", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "a^{(2)}=\\sigma^{(2)}(z^{(2)}).\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "1866378d", + "metadata": { + "editable": true + }, + "source": [ + "## Explicit derivatives\n", + "\n", + "In total we have nine parameters which we need to train. Using the\n", + "chain rule (or just the back-propagation algorithm) we can find all\n", + "derivatives. Since we will use automatic differentiation in reverse\n", + "mode, we start with the derivatives of the cost function with respect\n", + "to the parameters of the output layer, namely" + ] + }, + { + "cell_type": "markdown", + "id": "e6d27f9d", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\frac{\\partial C}{\\partial w_{i}^{(2)}}=\\frac{\\partial C}{\\partial a^{(2)}}\\frac{\\partial a^{(2)}}{\\partial z^{(2)}}\\frac{\\partial z^{(2)}}{\\partial w_{i}^{(2)}}=\\delta^{(2)}a_i^{(1)},\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "b33dac74", + "metadata": { + "editable": true + }, + "source": [ + "with" + ] + }, + { + "cell_type": "markdown", + "id": "27c367b4", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\delta^{(2)}=\\frac{\\partial C}{\\partial a^{(2)}}\\frac{\\partial a^{(2)}}{\\partial z^{(2)}}\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "18898e14", + "metadata": { + "editable": true + }, + "source": [ + "and finally" + ] + }, + { + "cell_type": "markdown", + "id": "31fcc7f9", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\frac{\\partial C}{\\partial b^{(2)}}=\\frac{\\partial C}{\\partial a^{(2)}}\\frac{\\partial a^{(2)}}{\\partial z^{(2)}}\\frac{\\partial z^{(2)}}{\\partial b^{(2)}}=\\delta^{(2)}.\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "76219fdd", + "metadata": { + "editable": true + }, + "source": [ + "## Derivatives of the hidden layer\n", + "\n", + "Using the chain rule we have the following expressions for say one of the weight parameters (it is easy to generalize to the other weight parameters)" + ] + }, + { + "cell_type": "markdown", + "id": "2cbccf94", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\frac{\\partial C}{\\partial w_{11}^{(1)}}=\\frac{\\partial C}{\\partial a^{(2)}}\\frac{\\partial a^{(2)}}{\\partial z^{(2)}}\n", + "\\frac{\\partial z^{(2)}}{\\partial z_1^{(1)}}\\frac{\\partial z_1^{(1)}}{\\partial w_{11}^{(1)}}= \\delta^{(2)}\\frac{\\partial z^{(2)}}{\\partial z_1^{(1)}}\\frac{\\partial z_1^{(1)}}{\\partial w_{11}^{(1)}},\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "ccdb071b", + "metadata": { + "editable": true + }, + "source": [ + "which, noting that" + ] + }, + { + "cell_type": "markdown", + "id": "55e4bed1", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "z^{(2)} =w_1^{(2)}a_1^{(1)}+w_2^{(2)}a_2^{(1)}+b^{(2)},\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "c00fd151", + "metadata": { + "editable": true + }, + "source": [ + "allows us to rewrite" + ] + }, + { + "cell_type": "markdown", + "id": "d49ae60d", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\frac{\\partial z^{(2)}}{\\partial z_1^{(1)}}\\frac{\\partial z_1^{(1)}}{\\partial w_{11}^{(1)}}=w_1^{(2)}\\frac{\\partial a_1^{(1)}}{\\partial z_1^{(1)}}a_1^{(1)}.\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "920db2dd", + "metadata": { + "editable": true + }, + "source": [ + "## Final expression\n", + "Defining" + ] + }, + { + "cell_type": "markdown", + "id": "62806f12", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\delta_1^{(1)}=w_1^{(2)}\\frac{\\partial a_1^{(1)}}{\\partial z_1^{(1)}}\\delta^{(2)},\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "f1d3e7b6", + "metadata": { + "editable": true + }, + "source": [ + "we have" + ] + }, + { + "cell_type": "markdown", + "id": "51fd0b81", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\frac{\\partial C}{\\partial w_{11}^{(1)}}=\\delta_1^{(1)}a_1^{(1)}.\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "ba9dd4ad", + "metadata": { + "editable": true + }, + "source": [ + "Similarly, we obtain" + ] + }, + { + "cell_type": "markdown", + "id": "91137440", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\frac{\\partial C}{\\partial w_{12}^{(1)}}=\\delta_1^{(1)}a_2^{(1)}.\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "8e2ecb8c", + "metadata": { + "editable": true + }, + "source": [ + "## Completing the list\n", + "\n", + "Similarly, we find" + ] + }, + { + "cell_type": "markdown", + "id": "8d91daf9", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\frac{\\partial C}{\\partial w_{21}^{(1)}}=\\delta_2^{(1)}a_1^{(1)},\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "8450f964", + "metadata": { + "editable": true + }, + "source": [ + "and" + ] + }, + { + "cell_type": "markdown", + "id": "1e662a52", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\frac{\\partial C}{\\partial w_{22}^{(1)}}=\\delta_2^{(1)}a_2^{(1)},\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "e90d34b5", + "metadata": { + "editable": true + }, + "source": [ + "where we have defined" + ] + }, + { + "cell_type": "markdown", + "id": "2d4b194d", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\delta_2^{(1)}=w_2^{(2)}\\frac{\\partial a_2^{(1)}}{\\partial z_2^{(1)}}\\delta^{(2)}.\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "8990fe0b", + "metadata": { + "editable": true + }, + "source": [ + "## Final expressions for the biases of the hidden layer\n", + "\n", + "For the sake of completeness, we list the derivatives of the biases, which are" + ] + }, + { + "cell_type": "markdown", + "id": "37e9e142", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\frac{\\partial C}{\\partial b_{1}^{(1)}}=\\delta_1^{(1)},\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "5f66c64a", + "metadata": { + "editable": true + }, + "source": [ + "and" + ] + }, + { + "cell_type": "markdown", + "id": "03e836ad", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\frac{\\partial C}{\\partial b_{2}^{(1)}}=\\delta_2^{(1)}.\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "1ea8a6ff", + "metadata": { + "editable": true + }, + "source": [ + "As we will see below, these expressions can be generalized in a more compact form." + ] + }, + { + "cell_type": "markdown", + "id": "84537cc9", + "metadata": { + "editable": true + }, + "source": [ + "## Gradient expressions\n", + "\n", + "For this specific model, with just one output node and two hidden\n", + "nodes, the gradient descent equations take the following form for output layer" + ] + }, + { + "cell_type": "markdown", + "id": "2bad6141", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "w_{i}^{(2)}\\leftarrow w_{i}^{(2)}- \\eta \\delta^{(2)} a_{i}^{(1)},\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "5150b413", + "metadata": { + "editable": true + }, + "source": [ + "and" + ] + }, + { + "cell_type": "markdown", + "id": "4166af55", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "b^{(2)} \\leftarrow b^{(2)}-\\eta \\delta^{(2)},\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "e5101f70", + "metadata": { + "editable": true + }, + "source": [ + "and" + ] + }, + { + "cell_type": "markdown", + "id": "0c662571", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "w_{ij}^{(1)}\\leftarrow w_{ij}^{(1)}- \\eta \\delta_{i}^{(1)} a_{j}^{(0)},\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "a7e1af99", + "metadata": { + "editable": true + }, + "source": [ + "and" + ] + }, + { + "cell_type": "markdown", + "id": "5a73623e", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "b_{i}^{(1)} \\leftarrow b_{i}^{(1)}-\\eta \\delta_{i}^{(1)},\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "b1d5327b", + "metadata": { + "editable": true + }, + "source": [ + "where $\\eta$ is the learning rate." + ] + }, + { + "cell_type": "markdown", + "id": "5cc5921c", + "metadata": { + "editable": true + }, + "source": [ + "## Setting up the equations for a neural network\n", + "\n", + "The questions we want to ask are how do changes in the biases and the\n", + "weights in our network change the cost function and how can we use the\n", + "final output to modify the weights and biases?\n", + "\n", + "To derive these equations let us start with a plain regression problem\n", + "and define our cost function as" + ] + }, + { + "cell_type": "markdown", + "id": "21633b14", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "{\\cal C}(\\boldsymbol{\\Theta}) = \\frac{1}{2}\\sum_{i=1}^n\\left(y_i - \\tilde{y}_i\\right)^2,\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "171782c4", + "metadata": { + "editable": true + }, + "source": [ + "where the $y_i$s are our $n$ targets (the values we want to\n", + "reproduce), while the outputs of the network after having propagated\n", + "all inputs $\\boldsymbol{x}$ are given by $\\boldsymbol{\\tilde{y}}_i$." + ] + }, + { + "cell_type": "markdown", + "id": "830e0cc3", + "metadata": { + "editable": true + }, + "source": [ + "## Layout of a neural network with three hidden layers (last layer = $l=L=4$, first layer $l=0$)\n", + "\n", + "\n", + "\n", + "\n", + "

    Figure 1:

    \n", + "" + ] + }, + { + "cell_type": "markdown", + "id": "3de97402", + "metadata": { + "editable": true + }, + "source": [ + "## Definitions\n", + "\n", + "With our definition of the targets $\\boldsymbol{y}$, the outputs of the\n", + "network $\\boldsymbol{\\tilde{y}}$ and the inputs $\\boldsymbol{x}$ we\n", + "define now the activation $z_j^l$ of node/neuron/unit $j$ of the\n", + "$l$-th layer as a function of the bias, the weights which add up from\n", + "the previous layer $l-1$ and the forward passes/outputs\n", + "$\\boldsymbol{a}^{l-1}$ from the previous layer as" + ] + }, + { + "cell_type": "markdown", + "id": "f4a20e55", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "z_j^l = \\sum_{i=1}^{M_{l-1}}w_{ij}^la_i^{l-1}+b_j^l,\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "e373e6cc", + "metadata": { + "editable": true + }, + "source": [ + "where $b_k^l$ are the biases from layer $l$. Here $M_{l-1}$\n", + "represents the total number of nodes/neurons/units of layer $l-1$. The\n", + "figure in the whiteboard notes illustrates this equation. We can rewrite this in a more\n", + "compact form as the matrix-vector products we discussed earlier," + ] + }, + { + "cell_type": "markdown", + "id": "a9fa6a69", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\boldsymbol{z}^l = \\left(\\boldsymbol{W}^l\\right)^T\\boldsymbol{a}^{l-1}+\\boldsymbol{b}^l.\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "f1730e5a", + "metadata": { + "editable": true + }, + "source": [ + "## Inputs to the activation function\n", + "\n", + "With the activation values $\\boldsymbol{z}^l$ we can in turn define the\n", + "output of layer $l$ as $\\boldsymbol{a}^l = \\sigma(\\boldsymbol{z}^l)$ where $\\sigma$ is our\n", + "activation function. In the examples here we will use the sigmoid\n", + "function discussed in our logistic regression lectures. We will also use the same activation function $\\sigma$ for all layers\n", + "and their nodes. It means we have" + ] + }, + { + "cell_type": "markdown", + "id": "59d7d98a", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "a_j^l = \\sigma(z_j^l) = \\frac{1}{1+\\exp{-(z_j^l)}}.\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "5a3d0931", + "metadata": { + "editable": true + }, + "source": [ + "## Layout of input to first hidden layer $l=1$ from input layer $l=0$\n", + "\n", + "\n", + "\n", + "\n", + "

    Figure 1:

    \n", + "" + ] + }, + { + "cell_type": "markdown", + "id": "c56d749e", + "metadata": { + "editable": true + }, + "source": [ + "## Derivatives and the chain rule\n", + "\n", + "From the definition of the input variable to the activation function, that is $z_j^l$ we have" + ] + }, + { + "cell_type": "markdown", + "id": "f3dd37e7", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\frac{\\partial z_j^l}{\\partial w_{ij}^l} = a_i^{l-1},\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "7654c47c", + "metadata": { + "editable": true + }, + "source": [ + "and" + ] + }, + { + "cell_type": "markdown", + "id": "cb1aa2a9", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\frac{\\partial z_j^l}{\\partial a_i^{l-1}} = w_{ji}^l.\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "11607454", + "metadata": { + "editable": true + }, + "source": [ + "With our definition of the activation function we have that (note that this function depends only on $z_j^l$)" + ] + }, + { + "cell_type": "markdown", + "id": "29a881cd", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\frac{\\partial a_j^l}{\\partial z_j^{l}} = a_j^l(1-a_j^l)=\\sigma(z_j^l)(1-\\sigma(z_j^l)).\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "1244c442", + "metadata": { + "editable": true + }, + "source": [ + "## Derivative of the cost function\n", + "\n", + "With these definitions we can now compute the derivative of the cost function in terms of the weights.\n", + "\n", + "Let us specialize to the output layer $l=L$. Our cost function is" + ] + }, + { + "cell_type": "markdown", + "id": "7d43f636", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "{\\cal C}(\\boldsymbol{\\Theta}^L) = \\frac{1}{2}\\sum_{i=1}^n\\left(y_i - \\tilde{y}_i\\right)^2=\\frac{1}{2}\\sum_{i=1}^n\\left(a_i^L - y_i\\right)^2,\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "c23708ab", + "metadata": { + "editable": true + }, + "source": [ + "The derivative of this function with respect to the weights is" + ] + }, + { + "cell_type": "markdown", + "id": "3a3d49b9", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\frac{\\partial{\\cal C}(\\boldsymbol{\\Theta}^L)}{\\partial w_{ij}^L} = \\left(a_j^L - y_j\\right)\\frac{\\partial a_j^L}{\\partial w_{ij}^{L}},\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "31b8b489", + "metadata": { + "editable": true + }, + "source": [ + "The last partial derivative can easily be computed and reads (by applying the chain rule)" + ] + }, + { + "cell_type": "markdown", + "id": "2cb11e87", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\frac{\\partial a_j^L}{\\partial w_{ij}^{L}} = \\frac{\\partial a_j^L}{\\partial z_{j}^{L}}\\frac{\\partial z_j^L}{\\partial w_{ij}^{L}}=a_j^L(1-a_j^L)a_i^{L-1}.\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "85b6783b", + "metadata": { + "editable": true + }, + "source": [ + "## The back propagation equations for a neural network\n", + "\n", + "We have thus" + ] + }, + { + "cell_type": "markdown", + "id": "1d9102e8", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\frac{\\partial{\\cal C}((\\boldsymbol{\\Theta}^L)}{\\partial w_{ij}^L} = \\left(a_j^L - y_j\\right)a_j^L(1-a_j^L)a_i^{L-1},\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "92547c80", + "metadata": { + "editable": true + }, + "source": [ + "Defining" + ] + }, + { + "cell_type": "markdown", + "id": "e06d7f47", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\delta_j^L = a_j^L(1-a_j^L)\\left(a_j^L - y_j\\right) = \\sigma'(z_j^L)\\frac{\\partial {\\cal C}}{\\partial (a_j^L)},\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "a15b2c15", + "metadata": { + "editable": true + }, + "source": [ + "and using the Hadamard product of two vectors we can write this as" + ] + }, + { + "cell_type": "markdown", + "id": "69a96bb6", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\boldsymbol{\\delta}^L = \\sigma'(\\boldsymbol{z}^L)\\circ\\frac{\\partial {\\cal C}}{\\partial (\\boldsymbol{a}^L)}.\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "6954d50b", + "metadata": { + "editable": true + }, + "source": [ + "## Analyzing the last results\n", + "\n", + "This is an important expression. The second term on the right handside\n", + "measures how fast the cost function is changing as a function of the $j$th\n", + "output activation. If, for example, the cost function doesn't depend\n", + "much on a particular output node $j$, then $\\delta_j^L$ will be small,\n", + "which is what we would expect. The first term on the right, measures\n", + "how fast the activation function $f$ is changing at a given activation\n", + "value $z_j^L$." + ] + }, + { + "cell_type": "markdown", + "id": "a8d2b720", + "metadata": { + "editable": true + }, + "source": [ + "## More considerations\n", + "\n", + "Notice that everything in the above equations is easily computed. In\n", + "particular, we compute $z_j^L$ while computing the behaviour of the\n", + "network, and it is only a small additional overhead to compute\n", + "$\\sigma'(z^L_j)$. The exact form of the derivative with respect to the\n", + "output depends on the form of the cost function.\n", + "However, provided the cost function is known there should be little\n", + "trouble in calculating" + ] + }, + { + "cell_type": "markdown", + "id": "b40feaea", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\frac{\\partial {\\cal C}}{\\partial (a_j^L)}\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "04d0916c", + "metadata": { + "editable": true + }, + "source": [ + "With the definition of $\\delta_j^L$ we have a more compact definition of the derivative of the cost function in terms of the weights, namely" + ] + }, + { + "cell_type": "markdown", + "id": "4067515f", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\frac{\\partial{\\cal C}}{\\partial w_{ij}^L} = \\delta_j^La_i^{L-1}.\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "572f5043", + "metadata": { + "editable": true + }, + "source": [ + "## Derivatives in terms of $z_j^L$\n", + "\n", + "It is also easy to see that our previous equation can be written as" + ] + }, + { + "cell_type": "markdown", + "id": "4272e56c", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\delta_j^L =\\frac{\\partial {\\cal C}}{\\partial z_j^L}= \\frac{\\partial {\\cal C}}{\\partial a_j^L}\\frac{\\partial a_j^L}{\\partial z_j^L},\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "3ffcc3a4", + "metadata": { + "editable": true + }, + "source": [ + "which can also be interpreted as the partial derivative of the cost function with respect to the biases $b_j^L$, namely" + ] + }, + { + "cell_type": "markdown", + "id": "87dc5557", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\delta_j^L = \\frac{\\partial {\\cal C}}{\\partial b_j^L}\\frac{\\partial b_j^L}{\\partial z_j^L}=\\frac{\\partial {\\cal C}}{\\partial b_j^L},\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "36f9508b", + "metadata": { + "editable": true + }, + "source": [ + "That is, the error $\\delta_j^L$ is exactly equal to the rate of change of the cost function as a function of the bias." + ] + }, + { + "cell_type": "markdown", + "id": "5e7f7ab6", + "metadata": { + "editable": true + }, + "source": [ + "## Bringing it together\n", + "\n", + "We have now three equations that are essential for the computations of the derivatives of the cost function at the output layer. These equations are needed to start the algorithm and they are" + ] + }, + { + "cell_type": "markdown", + "id": "b5caac62", + "metadata": { + "editable": true + }, + "source": [ + "\n", + "
    \n", + "\n", + "$$\n", + "\\begin{equation}\n", + "\\frac{\\partial{\\cal C}(\\boldsymbol{W^L})}{\\partial w_{ij}^L} = \\delta_j^La_i^{L-1},\n", + "\\label{_auto1} \\tag{1}\n", + "\\end{equation}\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "beffe853", + "metadata": { + "editable": true + }, + "source": [ + "and" + ] + }, + { + "cell_type": "markdown", + "id": "9ff991bb", + "metadata": { + "editable": true + }, + "source": [ + "\n", + "
    \n", + "\n", + "$$\n", + "\\begin{equation}\n", + "\\delta_j^L = \\sigma'(z_j^L)\\frac{\\partial {\\cal C}}{\\partial (a_j^L)},\n", + "\\label{_auto2} \\tag{2}\n", + "\\end{equation}\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "5c75d805", + "metadata": { + "editable": true + }, + "source": [ + "and" + ] + }, + { + "cell_type": "markdown", + "id": "df242f2d", + "metadata": { + "editable": true + }, + "source": [ + "\n", + "
    \n", + "\n", + "$$\n", + "\\begin{equation}\n", + "\\delta_j^L = \\frac{\\partial {\\cal C}}{\\partial b_j^L},\n", + "\\label{_auto3} \\tag{3}\n", + "\\end{equation}\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "18a65420", + "metadata": { + "editable": true + }, + "source": [ + "## Final back propagating equation\n", + "\n", + "We have that (replacing $L$ with a general layer $l$)" + ] + }, + { + "cell_type": "markdown", + "id": "04cb64d1", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\delta_j^l =\\frac{\\partial {\\cal C}}{\\partial z_j^l}.\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "c5320606", + "metadata": { + "editable": true + }, + "source": [ + "We want to express this in terms of the equations for layer $l+1$." + ] + }, + { + "cell_type": "markdown", + "id": "58b67295", + "metadata": { + "editable": true + }, + "source": [ + "## Using the chain rule and summing over all $k$ entries\n", + "\n", + "We obtain" + ] + }, + { + "cell_type": "markdown", + "id": "3b7c74a4", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\delta_j^l =\\sum_k \\frac{\\partial {\\cal C}}{\\partial z_k^{l+1}}\\frac{\\partial z_k^{l+1}}{\\partial z_j^{l}}=\\sum_k \\delta_k^{l+1}\\frac{\\partial z_k^{l+1}}{\\partial z_j^{l}},\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "76ff59e0", + "metadata": { + "editable": true + }, + "source": [ + "and recalling that" + ] + }, + { + "cell_type": "markdown", + "id": "60a7be03", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "z_j^{l+1} = \\sum_{i=1}^{M_{l}}w_{ij}^{l+1}a_i^{l}+b_j^{l+1},\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "27739d6d", + "metadata": { + "editable": true + }, + "source": [ + "with $M_l$ being the number of nodes in layer $l$, we obtain" + ] + }, + { + "cell_type": "markdown", + "id": "1efdb6f5", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\delta_j^l =\\sum_k \\delta_k^{l+1}w_{kj}^{l+1}\\sigma'(z_j^l),\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "b0b04f61", + "metadata": { + "editable": true + }, + "source": [ + "This is our final equation.\n", + "\n", + "We are now ready to set up the algorithm for back propagation and learning the weights and biases." + ] + }, + { + "cell_type": "markdown", + "id": "e6c4318c", + "metadata": { + "editable": true + }, + "source": [ + "## Setting up the back propagation algorithm and algorithm for a feed forward NN, initalizations\n", + "\n", + "**The architecture (our model).**\n", + "\n", + "1. Set up your inputs and outputs (scalars, vectors, matrices or higher-order arrays)\n", + "\n", + "2. Define the number of hidden layers and hidden nodes\n", + "\n", + "3. Define activation functions for hidden layers and output layers\n", + "\n", + "4. Define optimizer (plan learning rate, momentum, ADAgrad, RMSprop, ADAM etc) and array of initial learning rates\n", + "\n", + "5. Define cost function and possible regularization terms with hyperparameters\n", + "\n", + "6. Initialize weights and biases\n", + "\n", + "7. Fix number of iterations for the feed forward part and back propagation part" + ] + }, + { + "cell_type": "markdown", + "id": "4e58634a", + "metadata": { + "editable": true + }, + "source": [ + "## Setting up the back propagation algorithm, part 1\n", + "\n", + "The four equations provide us with a way of computing the gradients of the cost function. Let us write this out in the form of an algorithm.\n", + "\n", + "**First**, we set up the input data $\\boldsymbol{x}$ and the activations\n", + "$\\boldsymbol{z}_1$ of the input layer and compute the activation function and\n", + "the pertinent outputs $\\boldsymbol{a}^1$.\n", + "\n", + "**Secondly**, we perform then the feed forward till we reach the output\n", + "layer and compute all $\\boldsymbol{z}_l$ of the input layer and compute the\n", + "activation function and the pertinent outputs $\\boldsymbol{a}^l$ for\n", + "$l=1,2,3,\\dots,L$.\n", + "\n", + "**Notation**: The first hidden layer has $l=1$ as label and the final output layer has $l=L$." + ] + }, + { + "cell_type": "markdown", + "id": "690eb424", + "metadata": { + "editable": true + }, + "source": [ + "## Setting up the back propagation algorithm, part 2\n", + "\n", + "Thereafter we compute the ouput error $\\boldsymbol{\\delta}^L$ by computing all" + ] + }, + { + "cell_type": "markdown", + "id": "5f475d02", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\delta_j^L = \\sigma'(z_j^L)\\frac{\\partial {\\cal C}}{\\partial (a_j^L)}.\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "016ce549", + "metadata": { + "editable": true + }, + "source": [ + "Then we compute the back propagate error for each $l=L-1,L-2,\\dots,1$ as" + ] + }, + { + "cell_type": "markdown", + "id": "51f9ed82", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\delta_j^l = \\sum_k \\delta_k^{l+1}w_{kj}^{l+1}\\sigma'(z_j^l).\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "a8996176", + "metadata": { + "editable": true + }, + "source": [ + "## Setting up the Back propagation algorithm, part 3\n", + "\n", + "Finally, we update the weights and the biases using gradient descent\n", + "for each $l=L-1,L-2,\\dots,1$ (the first hidden layer) and update the weights and biases\n", + "according to the rules" + ] + }, + { + "cell_type": "markdown", + "id": "aaf186a8", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "w_{ij}^l\\leftarrow = w_{ij}^l- \\eta \\delta_j^la_i^{l-1},\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "3b8a0a79", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "b_j^l \\leftarrow b_j^l-\\eta \\frac{\\partial {\\cal C}}{\\partial b_j^l}=b_j^l-\\eta \\delta_j^l,\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "299cc7bb", + "metadata": { + "editable": true + }, + "source": [ + "with $\\eta$ being the learning rate." + ] + }, + { + "cell_type": "markdown", + "id": "3b10a1a0", + "metadata": { + "editable": true + }, + "source": [ + "## Updating the gradients\n", + "\n", + "With the back propagate error for each $l=L-1,L-2,\\dots,1$ as" + ] + }, + { + "cell_type": "markdown", + "id": "0f8769f8", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\delta_j^l = \\sum_k \\delta_k^{l+1}w_{kj}^{l+1}\\sigma'(z_j^l),\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "03860e2f", + "metadata": { + "editable": true + }, + "source": [ + "we update the weights and the biases using gradient descent for each $l=L-1,L-2,\\dots,1$ and update the weights and biases according to the rules" + ] + }, + { + "cell_type": "markdown", + "id": "00cea1da", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "w_{ij}^l\\leftarrow = w_{ij}^l- \\eta \\delta_j^la_i^{l-1},\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "fe5bbd2a", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "b_j^l \\leftarrow b_j^l-\\eta \\frac{\\partial {\\cal C}}{\\partial b_j^l}=b_j^l-\\eta \\delta_j^l,\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "e791fb56", + "metadata": { + "editable": true + }, + "source": [ + "## Activation functions\n", + "\n", + "A property that characterizes a neural network, other than its\n", + "connectivity, is the choice of activation function(s). The following\n", + "restrictions are imposed on an activation function for an FFNN to\n", + "fulfill the universal approximation theorem\n", + "\n", + " * Non-constant\n", + "\n", + " * Bounded\n", + "\n", + " * Monotonically-increasing\n", + "\n", + " * Continuous" + ] + }, + { + "cell_type": "markdown", + "id": "14ed4124", + "metadata": { + "editable": true + }, + "source": [ + "### Activation functions, Logistic and Hyperbolic ones\n", + "\n", + "The second requirement excludes all linear functions. Furthermore, in\n", + "a MLP with only linear activation functions, each layer simply\n", + "performs a linear transformation of its inputs.\n", + "\n", + "Regardless of the number of layers, the output of the NN will be\n", + "nothing but a linear function of the inputs. Thus we need to introduce\n", + "some kind of non-linearity to the NN to be able to fit non-linear\n", + "functions Typical examples are the logistic *Sigmoid*" + ] + }, + { + "cell_type": "markdown", + "id": "d6b31eb8", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\sigma(x) = \\frac{1}{1 + e^{-x}},\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "2be10af5", + "metadata": { + "editable": true + }, + "source": [ + "and the *hyperbolic tangent* function" + ] + }, + { + "cell_type": "markdown", + "id": "f45b9a99", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\sigma(x) = \\tanh(x)\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "865d802b", + "metadata": { + "editable": true + }, + "source": [ + "## Relevance\n", + "\n", + "The *sigmoid* function are more biologically plausible because the\n", + "output of inactive neurons are zero. Such activation function are\n", + "called *one-sided*. However, it has been shown that the hyperbolic\n", + "tangent performs better than the sigmoid for training MLPs. has\n", + "become the most popular for *deep neural networks*" + ] + }, + { + "cell_type": "code", + "execution_count": 2, + "id": "37773874", + "metadata": { + "collapsed": false, + "editable": true + }, + "outputs": [], + "source": [ + "%matplotlib inline\n", + "\n", + "\"\"\"The sigmoid function (or the logistic curve) is a \n", + "function that takes any real number, z, and outputs a number (0,1).\n", + "It is useful in neural networks for assigning weights on a relative scale.\n", + "The value z is the weighted sum of parameters involved in the learning algorithm.\"\"\"\n", + "\n", + "import numpy\n", + "import matplotlib.pyplot as plt\n", + "import math as mt\n", + "\n", + "z = numpy.arange(-5, 5, .1)\n", + "sigma_fn = numpy.vectorize(lambda z: 1/(1+numpy.exp(-z)))\n", + "sigma = sigma_fn(z)\n", + "\n", + "fig = plt.figure()\n", + "ax = fig.add_subplot(111)\n", + "ax.plot(z, sigma)\n", + "ax.set_ylim([-0.1, 1.1])\n", + "ax.set_xlim([-5,5])\n", + "ax.grid(True)\n", + "ax.set_xlabel('z')\n", + "ax.set_title('sigmoid function')\n", + "\n", + "plt.show()\n", + "\n", + "\"\"\"Step Function\"\"\"\n", + "z = numpy.arange(-5, 5, .02)\n", + "step_fn = numpy.vectorize(lambda z: 1.0 if z >= 0.0 else 0.0)\n", + "step = step_fn(z)\n", + "\n", + "fig = plt.figure()\n", + "ax = fig.add_subplot(111)\n", + "ax.plot(z, step)\n", + "ax.set_ylim([-0.5, 1.5])\n", + "ax.set_xlim([-5,5])\n", + "ax.grid(True)\n", + "ax.set_xlabel('z')\n", + "ax.set_title('step function')\n", + "\n", + "plt.show()\n", + "\n", + "\"\"\"Sine Function\"\"\"\n", + "z = numpy.arange(-2*mt.pi, 2*mt.pi, 0.1)\n", + "t = numpy.sin(z)\n", + "\n", + "fig = plt.figure()\n", + "ax = fig.add_subplot(111)\n", + "ax.plot(z, t)\n", + "ax.set_ylim([-1.0, 1.0])\n", + "ax.set_xlim([-2*mt.pi,2*mt.pi])\n", + "ax.grid(True)\n", + "ax.set_xlabel('z')\n", + "ax.set_title('sine function')\n", + "\n", + "plt.show()\n", + "\n", + "\"\"\"Plots a graph of the squashing function used by a rectified linear\n", + "unit\"\"\"\n", + "z = numpy.arange(-2, 2, .1)\n", + "zero = numpy.zeros(len(z))\n", + "y = numpy.max([zero, z], axis=0)\n", + "\n", + "fig = plt.figure()\n", + "ax = fig.add_subplot(111)\n", + "ax.plot(z, y)\n", + "ax.set_ylim([-2.0, 2.0])\n", + "ax.set_xlim([-2.0, 2.0])\n", + "ax.grid(True)\n", + "ax.set_xlabel('z')\n", + "ax.set_title('Rectified linear unit')\n", + "\n", + "plt.show()" + ] + }, + { + "cell_type": "markdown", + "id": "9cccb9ca", + "metadata": { + "editable": true + }, + "source": [ + "## Vanishing gradients\n", + "\n", + "The Back propagation algorithm we derived above works by going from\n", + "the output layer to the input layer, propagating the error gradient on\n", + "the way. Once the algorithm has computed the gradient of the cost\n", + "function with regards to each parameter in the network, it uses these\n", + "gradients to update each parameter with a Gradient Descent (GD) step.\n", + "\n", + "Unfortunately for us, the gradients often get smaller and smaller as\n", + "the algorithm progresses down to the first hidden layers. As a result,\n", + "the GD update leaves the lower layer connection weights virtually\n", + "unchanged, and training never converges to a good solution. This is\n", + "known in the literature as **the vanishing gradients problem**." + ] + }, + { + "cell_type": "markdown", + "id": "a42f7226", + "metadata": { + "editable": true + }, + "source": [ + "## Exploding gradients\n", + "\n", + "In other cases, the opposite can happen, namely the the gradients can\n", + "grow bigger and bigger. The result is that many of the layers get\n", + "large updates of the weights the algorithm diverges. This is the\n", + "**exploding gradients problem**, which is mostly encountered in\n", + "recurrent neural networks. More generally, deep neural networks suffer\n", + "from unstable gradients, different layers may learn at widely\n", + "different speeds" + ] + }, + { + "cell_type": "markdown", + "id": "0f57d23f", + "metadata": { + "editable": true + }, + "source": [ + "## Is the Logistic activation function (Sigmoid) our choice?\n", + "\n", + "Although this unfortunate behavior has been empirically observed for\n", + "quite a while (it was one of the reasons why deep neural networks were\n", + "mostly abandoned for a long time), it is only around 2010 that\n", + "significant progress was made in understanding it.\n", + "\n", + "A paper titled [Understanding the Difficulty of Training Deep\n", + "Feedforward Neural Networks by Xavier Glorot and Yoshua Bengio](http://proceedings.mlr.press/v9/glorot10a.html) found that\n", + "the problems with the popular logistic\n", + "sigmoid activation function and the weight initialization technique\n", + "that was most popular at the time, namely random initialization using\n", + "a normal distribution with a mean of 0 and a standard deviation of\n", + "1." + ] + }, + { + "cell_type": "markdown", + "id": "ad459faf", + "metadata": { + "editable": true + }, + "source": [ + "## Logistic function as the root of problems\n", + "\n", + "They showed that with this activation function and this\n", + "initialization scheme, the variance of the outputs of each layer is\n", + "much greater than the variance of its inputs. Going forward in the\n", + "network, the variance keeps increasing after each layer until the\n", + "activation function saturates at the top layers. This is actually made\n", + "worse by the fact that the logistic function has a mean of 0.5, not 0\n", + "(the hyperbolic tangent function has a mean of 0 and behaves slightly\n", + "better than the logistic function in deep networks)." + ] + }, + { + "cell_type": "markdown", + "id": "1e307870", + "metadata": { + "editable": true + }, + "source": [ + "## The derivative of the Logistic funtion\n", + "\n", + "Looking at the logistic activation function, when inputs become large\n", + "(negative or positive), the function saturates at 0 or 1, with a\n", + "derivative extremely close to 0. Thus when backpropagation kicks in,\n", + "it has virtually no gradient to propagate back through the network,\n", + "and what little gradient exists keeps getting diluted as\n", + "backpropagation progresses down through the top layers, so there is\n", + "really nothing left for the lower layers.\n", + "\n", + "In their paper, Glorot and Bengio propose a way to significantly\n", + "alleviate this problem. We need the signal to flow properly in both\n", + "directions: in the forward direction when making predictions, and in\n", + "the reverse direction when backpropagating gradients. We don’t want\n", + "the signal to die out, nor do we want it to explode and saturate. For\n", + "the signal to flow properly, the authors argue that we need the\n", + "variance of the outputs of each layer to be equal to the variance of\n", + "its inputs, and we also need the gradients to have equal variance\n", + "before and after flowing through a layer in the reverse direction." + ] + }, + { + "cell_type": "markdown", + "id": "06fe5be6", + "metadata": { + "editable": true + }, + "source": [ + "## Insights from the paper by Glorot and Bengio\n", + "\n", + "One of the insights in the 2010 paper by Glorot and Bengio was that\n", + "the vanishing/exploding gradients problems were in part due to a poor\n", + "choice of activation function. Until then most people had assumed that\n", + "if Nature had chosen to use roughly sigmoid activation functions in\n", + "biological neurons, they must be an excellent choice. But it turns out\n", + "that other activation functions behave much better in deep neural\n", + "networks, in particular the ReLU activation function, mostly because\n", + "it does not saturate for positive values (and also because it is quite\n", + "fast to compute)." + ] + }, + { + "cell_type": "markdown", + "id": "736441f3", + "metadata": { + "editable": true + }, + "source": [ + "## The RELU function family\n", + "\n", + "The ReLU activation function suffers from a problem known as the dying\n", + "ReLUs: during training, some neurons effectively die, meaning they\n", + "stop outputting anything other than 0.\n", + "\n", + "In some cases, you may find that half of your network’s neurons are\n", + "dead, especially if you used a large learning rate. During training,\n", + "if a neuron’s weights get updated such that the weighted sum of the\n", + "neuron’s inputs is negative, it will start outputting 0. When this\n", + "happen, the neuron is unlikely to come back to life since the gradient\n", + "of the ReLU function is 0 when its input is negative." + ] + }, + { + "cell_type": "markdown", + "id": "9ac6f21d", + "metadata": { + "editable": true + }, + "source": [ + "## ELU function\n", + "\n", + "To solve this problem, nowadays practitioners use a variant of the\n", + "ReLU function, such as the leaky ReLU discussed above or the so-called\n", + "exponential linear unit (ELU) function" + ] + }, + { + "cell_type": "markdown", + "id": "80ebd77c", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "ELU(z) = \\left\\{\\begin{array}{cc} \\alpha\\left( \\exp{(z)}-1\\right) & z < 0,\\\\ z & z \\ge 0.\\end{array}\\right.\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "2566a903", + "metadata": { + "editable": true + }, + "source": [ + "## Which activation function should we use?\n", + "\n", + "In general it seems that the ELU activation function is better than\n", + "the leaky ReLU function (and its variants), which is better than\n", + "ReLU. ReLU performs better than $\\tanh$ which in turn performs better\n", + "than the logistic function.\n", + "\n", + "If runtime performance is an issue, then you may opt for the leaky\n", + "ReLU function over the ELU function If you don’t want to tweak yet\n", + "another hyperparameter, you may just use the default $\\alpha$ of\n", + "$0.01$ for the leaky ReLU, and $1$ for ELU. If you have spare time and\n", + "computing power, you can use cross-validation or bootstrap to evaluate\n", + "other activation functions." + ] + }, + { + "cell_type": "markdown", + "id": "382f5fe1", + "metadata": { + "editable": true + }, + "source": [ + "## More on activation functions, output layers\n", + "\n", + "In most cases you can use the ReLU activation function in the hidden\n", + "layers (or one of its variants).\n", + "\n", + "It is a bit faster to compute than other activation functions, and the\n", + "gradient descent optimization does in general not get stuck.\n", + "\n", + "**For the output layer:**\n", + "\n", + "* For classification the softmax activation function is generally a good choice for classification tasks (when the classes are mutually exclusive).\n", + "\n", + "* For regression tasks, you can simply use no activation function at all." + ] + }, + { + "cell_type": "markdown", + "id": "fed07226", + "metadata": { + "editable": true + }, + "source": [ + "## Fine-tuning neural network hyperparameters\n", + "\n", + "The flexibility of neural networks is also one of their main\n", + "drawbacks: there are many hyperparameters to tweak. Not only can you\n", + "use any imaginable network topology (how neurons/nodes are\n", + "interconnected), but even in a simple FFNN you can change the number\n", + "of layers, the number of neurons per layer, the type of activation\n", + "function to use in each layer, the weight initialization logic, the\n", + "stochastic gradient optmized and much more. How do you know what\n", + "combination of hyperparameters is the best for your task?\n", + "\n", + "* You can use grid search with cross-validation to find the right hyperparameters.\n", + "\n", + "However,since there are many hyperparameters to tune, and since\n", + "training a neural network on a large dataset takes a lot of time, you\n", + "will only be able to explore a tiny part of the hyperparameter space.\n", + "\n", + "* You can use randomized search.\n", + "\n", + "* Or use tools like [Oscar](http://oscar.calldesk.ai/), which implements more complex algorithms to help you find a good set of hyperparameters quickly." + ] + }, + { + "cell_type": "markdown", + "id": "9ddc2586", + "metadata": { + "editable": true + }, + "source": [ + "## Hidden layers\n", + "\n", + "For many problems you can start with just one or two hidden layers and\n", + "it will work just fine. For the MNIST data set discussed below you can easily get a\n", + "high accuracy using just one hidden layer with a few hundred neurons.\n", + "You can reach for this data set above 98% accuracy using two hidden\n", + "layers with the same total amount of neurons, in roughly the same\n", + "amount of training time.\n", + "\n", + "For more complex problems, you can gradually ramp up the number of\n", + "hidden layers, until you start overfitting the training set. Very\n", + "complex tasks, such as large image classification or speech\n", + "recognition, typically require networks with dozens of layers and they\n", + "need a huge amount of training data. However, you will rarely have to\n", + "train such networks from scratch: it is much more common to reuse\n", + "parts of a pretrained state-of-the-art network that performs a similar\n", + "task." + ] + }, + { + "cell_type": "markdown", + "id": "d327b303", + "metadata": { + "editable": true + }, + "source": [ + "## Batch Normalization\n", + "\n", + "Batch Normalization aims to address the vanishing/exploding gradients\n", + "problems, and more generally the problem that the distribution of each\n", + "layer’s inputs changes during training, as the parameters of the\n", + "previous layers change.\n", + "\n", + "The technique consists of adding an operation in the model just before\n", + "the activation function of each layer, simply zero-centering and\n", + "normalizing the inputs, then scaling and shifting the result using two\n", + "new parameters per layer (one for scaling, the other for shifting). In\n", + "other words, this operation lets the model learn the optimal scale and\n", + "mean of the inputs for each layer. In order to zero-center and\n", + "normalize the inputs, the algorithm needs to estimate the inputs’ mean\n", + "and standard deviation. It does so by evaluating the mean and standard\n", + "deviation of the inputs over the current mini-batch, from this the\n", + "name batch normalization." + ] + }, + { + "cell_type": "markdown", + "id": "a3f4a57d", + "metadata": { + "editable": true + }, + "source": [ + "## Dropout\n", + "\n", + "It is a fairly simple algorithm: at every training step, every neuron\n", + "(including the input neurons but excluding the output neurons) has a\n", + "probability $p$ of being temporarily dropped out, meaning it will be\n", + "entirely ignored during this training step, but it may be active\n", + "during the next step.\n", + "\n", + "The hyperparameter $p$ is called the dropout rate, and it is typically\n", + "set to 50%. After training, the neurons are not dropped anymore. It\n", + "is viewed as one of the most popular regularization techniques." + ] + }, + { + "cell_type": "markdown", + "id": "da0c8dbe", + "metadata": { + "editable": true + }, + "source": [ + "## Gradient Clipping\n", + "\n", + "A popular technique to lessen the exploding gradients problem is to\n", + "simply clip the gradients during backpropagation so that they never\n", + "exceed some threshold (this is mostly useful for recurrent neural\n", + "networks).\n", + "\n", + "This technique is called Gradient Clipping.\n", + "\n", + "In general however, Batch\n", + "Normalization is preferred." + ] + }, + { + "cell_type": "markdown", + "id": "60e3da23", + "metadata": { + "editable": true + }, + "source": [ + "## A top-down perspective on Neural networks\n", + "\n", + "The first thing we would like to do is divide the data into two or\n", + "three parts. A training set, a validation or dev (development) set,\n", + "and a test set. The test set is the data on which we want to make\n", + "predictions. The dev set is a subset of the training data we use to\n", + "check how well we are doing out-of-sample, after training the model on\n", + "the training dataset. We use the validation error as a proxy for the\n", + "test error in order to make tweaks to our model. It is crucial that we\n", + "do not use any of the test data to train the algorithm. This is a\n", + "cardinal sin in ML. Then:\n", + "\n", + "1. Estimate optimal error rate\n", + "\n", + "2. Minimize underfitting (bias) on training data set.\n", + "\n", + "3. Make sure you are not overfitting." + ] + }, + { + "cell_type": "markdown", + "id": "17f8f5ed", + "metadata": { + "editable": true + }, + "source": [ + "## More top-down perspectives\n", + "\n", + "If the validation and test sets are drawn from the same distributions,\n", + "then a good performance on the validation set should lead to similarly\n", + "good performance on the test set. \n", + "\n", + "However, sometimes\n", + "the training data and test data differ in subtle ways because, for\n", + "example, they are collected using slightly different methods, or\n", + "because it is cheaper to collect data in one way versus another. In\n", + "this case, there can be a mismatch between the training and test\n", + "data. This can lead to the neural network overfitting these small\n", + "differences between the test and training sets, and a poor performance\n", + "on the test set despite having a good performance on the validation\n", + "set. To rectify this, Andrew Ng suggests making two validation or dev\n", + "sets, one constructed from the training data and one constructed from\n", + "the test data. The difference between the performance of the algorithm\n", + "on these two validation sets quantifies the train-test mismatch. This\n", + "can serve as another important diagnostic when using DNNs for\n", + "supervised learning." + ] + }, + { + "cell_type": "markdown", + "id": "06ba3896", + "metadata": { + "editable": true + }, + "source": [ + "## Limitations of supervised learning with deep networks\n", + "\n", + "Like all statistical methods, supervised learning using neural\n", + "networks has important limitations. This is especially important when\n", + "one seeks to apply these methods, especially to physics problems. Like\n", + "all tools, DNNs are not a universal solution. Often, the same or\n", + "better performance on a task can be achieved by using a few\n", + "hand-engineered features (or even a collection of random\n", + "features)." + ] + }, + { + "cell_type": "markdown", + "id": "72bc99e6", + "metadata": { + "editable": true + }, + "source": [ + "## Limitations of NNs\n", + "\n", + "Here we list some of the important limitations of supervised neural network based models. \n", + "\n", + "* **Need labeled data**. All supervised learning methods, DNNs for supervised learning require labeled data. Often, labeled data is harder to acquire than unlabeled data (e.g. one must pay for human experts to label images).\n", + "\n", + "* **Supervised neural networks are extremely data intensive.** DNNs are data hungry. They perform best when data is plentiful. This is doubly so for supervised methods where the data must also be labeled. The utility of DNNs is extremely limited if data is hard to acquire or the datasets are small (hundreds to a few thousand samples). In this case, the performance of other methods that utilize hand-engineered features can exceed that of DNNs." + ] + }, + { + "cell_type": "markdown", + "id": "d5f452a2", + "metadata": { + "editable": true + }, + "source": [ + "## Homogeneous data\n", + "\n", + "* **Homogeneous data.** Almost all DNNs deal with homogeneous data of one type. It is very hard to design architectures that mix and match data types (i.e. some continuous variables, some discrete variables, some time series). In applications beyond images, video, and language, this is often what is required. In contrast, ensemble models like random forests or gradient-boosted trees have no difficulty handling mixed data types." + ] + }, + { + "cell_type": "markdown", + "id": "63889ee1", + "metadata": { + "editable": true + }, + "source": [ + "## More limitations\n", + "\n", + "* **Many problems are not about prediction.** In natural science we are often interested in learning something about the underlying distribution that generates the data. In this case, it is often difficult to cast these ideas in a supervised learning setting. While the problems are related, it is possible to make good predictions with a *wrong* model. The model might or might not be useful for understanding the underlying science.\n", + "\n", + "Some of these remarks are particular to DNNs, others are shared by all supervised learning methods. This motivates the use of unsupervised methods which in part circumvent these problems." + ] + }, + { + "cell_type": "markdown", + "id": "5f47ef0f", + "metadata": { + "editable": true + }, + "source": [ + "## Setting up a Multi-layer perceptron model for classification\n", + "\n", + "We are now gong to develop an example based on the MNIST data\n", + "base. This is a classification problem and we need to use our\n", + "cross-entropy function we discussed in connection with logistic\n", + "regression. The cross-entropy defines our cost function for the\n", + "classificaton problems with neural networks.\n", + "\n", + "In binary classification with two classes $(0, 1)$ we define the\n", + "logistic/sigmoid function as the probability that a particular input\n", + "is in class $0$ or $1$. This is possible because the logistic\n", + "function takes any input from the real numbers and inputs a number\n", + "between 0 and 1, and can therefore be interpreted as a probability. It\n", + "also has other nice properties, such as a derivative that is simple to\n", + "calculate.\n", + "\n", + "For an input $\\boldsymbol{a}$ from the hidden layer, the probability that the input $\\boldsymbol{x}$\n", + "is in class 0 or 1 is just. We let $\\theta$ represent the unknown weights and biases to be adjusted by our equations). The variable $x$\n", + "represents our activation values $z$. We have" + ] + }, + { + "cell_type": "markdown", + "id": "b08a7b7e", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "P(y = 0 \\mid \\boldsymbol{x}, \\boldsymbol{\\theta}) = \\frac{1}{1 + \\exp{(- \\boldsymbol{x}})} ,\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "10220190", + "metadata": { + "editable": true + }, + "source": [ + "and" + ] + }, + { + "cell_type": "markdown", + "id": "0fb4d5ad", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "P(y = 1 \\mid \\boldsymbol{x}, \\boldsymbol{\\theta}) = 1 - P(y = 0 \\mid \\boldsymbol{x}, \\boldsymbol{\\theta}) ,\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "f8aa1bae", + "metadata": { + "editable": true + }, + "source": [ + "where $y \\in \\{0, 1\\}$ and $\\boldsymbol{\\theta}$ represents the weights and biases\n", + "of our network." + ] + }, + { + "cell_type": "markdown", + "id": "17de8e22", + "metadata": { + "editable": true + }, + "source": [ + "## Defining the cost function\n", + "\n", + "Our cost function is given as (see the Logistic regression lectures)" + ] + }, + { + "cell_type": "markdown", + "id": "ca8f4f94", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\mathcal{C}(\\boldsymbol{\\theta}) = - \\ln P(\\mathcal{D} \\mid \\boldsymbol{\\theta}) = - \\sum_{i=1}^n\n", + "y_i \\ln[P(y_i = 0)] + (1 - y_i) \\ln [1 - P(y_i = 0)] = \\sum_{i=1}^n \\mathcal{L}_i(\\boldsymbol{\\theta}) .\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "ca998f7f", + "metadata": { + "editable": true + }, + "source": [ + "This last equality means that we can interpret our *cost* function as a sum over the *loss* function\n", + "for each point in the dataset $\\mathcal{L}_i(\\boldsymbol{\\theta})$. \n", + "The negative sign is just so that we can think about our algorithm as minimizing a positive number, rather\n", + "than maximizing a negative number. \n", + "\n", + "In *multiclass* classification it is common to treat each integer label as a so called *one-hot* vector: \n", + "\n", + "$y = 5 \\quad \\rightarrow \\quad \\boldsymbol{y} = (0, 0, 0, 0, 0, 1, 0, 0, 0, 0) ,$ and\n", + "\n", + "$y = 1 \\quad \\rightarrow \\quad \\boldsymbol{y} = (0, 1, 0, 0, 0, 0, 0, 0, 0, 0) ,$ \n", + "\n", + "i.e. a binary bit string of length $C$, where $C = 10$ is the number of classes in the MNIST dataset (numbers from $0$ to $9$).. \n", + "\n", + "If $\\boldsymbol{x}_i$ is the $i$-th input (image), $y_{ic}$ refers to the $c$-th component of the $i$-th\n", + "output vector $\\boldsymbol{y}_i$. \n", + "The probability of $\\boldsymbol{x}_i$ being in class $c$ will be given by the softmax function:" + ] + }, + { + "cell_type": "markdown", + "id": "525b0573", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "P(y_{ic} = 1 \\mid \\boldsymbol{x}_i, \\boldsymbol{\\theta}) = \\frac{\\exp{((\\boldsymbol{a}_i^{hidden})^T \\boldsymbol{w}_c)}}\n", + "{\\sum_{c'=0}^{C-1} \\exp{((\\boldsymbol{a}_i^{hidden})^T \\boldsymbol{w}_{c'})}} ,\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "a9279995", + "metadata": { + "editable": true + }, + "source": [ + "which reduces to the logistic function in the binary case. \n", + "The likelihood of this $C$-class classifier\n", + "is now given as:" + ] + }, + { + "cell_type": "markdown", + "id": "e6cef751", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "P(\\mathcal{D} \\mid \\boldsymbol{\\theta}) = \\prod_{i=1}^n \\prod_{c=0}^{C-1} [P(y_{ic} = 1)]^{y_{ic}} .\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "0ec11e1d", + "metadata": { + "editable": true + }, + "source": [ + "Again we take the negative log-likelihood to define our cost function:" + ] + }, + { + "cell_type": "markdown", + "id": "e5d6a067", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\mathcal{C}(\\boldsymbol{\\theta}) = - \\log{P(\\mathcal{D} \\mid \\boldsymbol{\\theta})}.\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "add6e597", + "metadata": { + "editable": true + }, + "source": [ + "See the logistic regression lectures for a full definition of the cost function.\n", + "\n", + "The back propagation equations need now only a small change, namely the definition of a new cost function. We are thus ready to use the same equations as before!" + ] + }, + { + "cell_type": "markdown", + "id": "dc7ca42d", + "metadata": { + "editable": true + }, + "source": [ + "## Example: binary classification problem\n", + "\n", + "As an example of the above, relevant for project 2 as well, let us consider a binary class. As discussed in our logistic regression lectures, we defined a cost function in terms of the parameters $\\beta$ as" + ] + }, + { + "cell_type": "markdown", + "id": "80b56aa5", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\mathcal{C}(\\boldsymbol{\\beta}) = - \\sum_{i=1}^n \\left(y_i\\log{p(y_i \\vert x_i,\\boldsymbol{\\beta})}+(1-y_i)\\log{1-p(y_i \\vert x_i,\\boldsymbol{\\beta})}\\right),\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "72a87573", + "metadata": { + "editable": true + }, + "source": [ + "where we had defined the logistic (sigmoid) function" + ] + }, + { + "cell_type": "markdown", + "id": "ff8418ea", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "p(y_i =1\\vert x_i,\\boldsymbol{\\beta})=\\frac{\\exp{(\\beta_0+\\beta_1 x_i)}}{1+\\exp{(\\beta_0+\\beta_1 x_i)}},\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "dd978909", + "metadata": { + "editable": true + }, + "source": [ + "and" + ] + }, + { + "cell_type": "markdown", + "id": "ce5a0650", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "p(y_i =0\\vert x_i,\\boldsymbol{\\beta})=1-p(y_i =1\\vert x_i,\\boldsymbol{\\beta}).\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "dce8478b", + "metadata": { + "editable": true + }, + "source": [ + "The parameters $\\boldsymbol{\\beta}$ were defined using a minimization method like gradient descent or Newton-Raphson's method. \n", + "\n", + "Now we replace $x_i$ with the activation $z_i^l$ for a given layer $l$ and the outputs as $y_i=a_i^l=f(z_i^l)$, with $z_i^l$ now being a function of the weights $w_{ij}^l$ and biases $b_i^l$. \n", + "We have then" + ] + }, + { + "cell_type": "markdown", + "id": "92f7282e", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "a_i^l = y_i = \\frac{\\exp{(z_i^l)}}{1+\\exp{(z_i^l)}},\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "e83bc7a2", + "metadata": { + "editable": true + }, + "source": [ + "with" + ] + }, + { + "cell_type": "markdown", + "id": "04f9d7f2", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "z_i^l = \\sum_{j}w_{ij}^l a_j^{l-1}+b_i^l,\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "c70ab677", + "metadata": { + "editable": true + }, + "source": [ + "where the superscript $l-1$ indicates that these are the outputs from layer $l-1$.\n", + "Our cost function at the final layer $l=L$ is now" + ] + }, + { + "cell_type": "markdown", + "id": "801a6bd2", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\mathcal{C}(\\boldsymbol{W}) = - \\sum_{i=1}^n \\left(t_i\\log{a_i^L}+(1-t_i)\\log{(1-a_i^L)}\\right),\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "38eeee4b", + "metadata": { + "editable": true + }, + "source": [ + "where we have defined the targets $t_i$. The derivatives of the cost function with respect to the output $a_i^L$ are then easily calculated and we get" + ] + }, + { + "cell_type": "markdown", + "id": "d70ddf11", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\frac{\\partial \\mathcal{C}(\\boldsymbol{W})}{\\partial a_i^L} = \\frac{a_i^L-t_i}{a_i^L(1-a_i^L)}.\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "324ee5e0", + "metadata": { + "editable": true + }, + "source": [ + "In case we use another activation function than the logistic one, we need to evaluate other derivatives." + ] + }, + { + "cell_type": "markdown", + "id": "3366df27", + "metadata": { + "editable": true + }, + "source": [ + "## The Softmax function\n", + "In case we employ the more general case given by the Softmax equation, we need to evaluate the derivative of the activation function with respect to the activation $z_i^l$, that is we need" + ] + }, + { + "cell_type": "markdown", + "id": "c9a2bd62", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\frac{\\partial f(z_i^l)}{\\partial w_{jk}^l} =\n", + "\\frac{\\partial f(z_i^l)}{\\partial z_j^l} \\frac{\\partial z_j^l}{\\partial w_{jk}^l}= \\frac{\\partial f(z_i^l)}{\\partial z_j^l}a_k^{l-1}.\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "401183f6", + "metadata": { + "editable": true + }, + "source": [ + "For the Softmax function we have" + ] + }, + { + "cell_type": "markdown", + "id": "d4e6a79e", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "f(z_i^l) = \\frac{\\exp{(z_i^l)}}{\\sum_{m=1}^K\\exp{(z_m^l)}}.\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "61cfab52", + "metadata": { + "editable": true + }, + "source": [ + "Its derivative with respect to $z_j^l$ gives" + ] + }, + { + "cell_type": "markdown", + "id": "fc5cf891", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\frac{\\partial f(z_i^l)}{\\partial z_j^l}= f(z_i^l)\\left(\\delta_{ij}-f(z_j^l)\\right),\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "2e7f3950", + "metadata": { + "editable": true + }, + "source": [ + "which in case of the simply binary model reduces to having $i=j$." + ] + }, + { + "cell_type": "markdown", + "id": "144969f6", + "metadata": { + "editable": true + }, + "source": [ + "## Developing a code for doing neural networks with back propagation\n", + "\n", + "One can identify a set of key steps when using neural networks to solve supervised learning problems: \n", + "\n", + "1. Collect and pre-process data \n", + "\n", + "2. Define model and architecture \n", + "\n", + "3. Choose cost function and optimizer \n", + "\n", + "4. Train the model \n", + "\n", + "5. Evaluate model performance on test data \n", + "\n", + "6. Adjust hyperparameters (if necessary, network architecture)" + ] + }, + { + "cell_type": "markdown", + "id": "b6e503bd", + "metadata": { + "editable": true + }, + "source": [ + "## Collect and pre-process data\n", + "\n", + "Here we will be using the MNIST dataset, which is readily available through the **scikit-learn**\n", + "package. You may also find it for example [here](http://yann.lecun.com/exdb/mnist/). \n", + "The *MNIST* (Modified National Institute of Standards and Technology) database is a large database\n", + "of handwritten digits that is commonly used for training various image processing systems. \n", + "The MNIST dataset consists of 70 000 images of size $28\\times 28$ pixels, each labeled from 0 to 9. \n", + "The scikit-learn dataset we will use consists of a selection of 1797 images of size $8\\times 8$ collected and processed from this database. \n", + "\n", + "To feed data into a feed-forward neural network we need to represent\n", + "the inputs as a design/feature matrix $X = (n_{inputs}, n_{features})$. Each\n", + "row represents an *input*, in this case a handwritten digit, and\n", + "each column represents a *feature*, in this case a pixel. The\n", + "correct answers, also known as *labels* or *targets* are\n", + "represented as a 1D array of integers \n", + "$Y = (n_{inputs}) = (5, 3, 1, 8,...)$.\n", + "\n", + "As an example, say we want to build a neural network using supervised learning to predict Body-Mass Index (BMI) from\n", + "measurements of height (in m) \n", + "and weight (in kg). If we have measurements of 5 people the design/feature matrix could be for example: \n", + "\n", + "$$ X = \\begin{bmatrix}\n", + "1.85 & 81\\\\\n", + "1.71 & 65\\\\\n", + "1.95 & 103\\\\\n", + "1.55 & 42\\\\\n", + "1.63 & 56\n", + "\\end{bmatrix} ,$$ \n", + "\n", + "and the targets would be: \n", + "\n", + "$$ Y = (23.7, 22.2, 27.1, 17.5, 21.1) $$ \n", + "\n", + "Since each input image is a 2D matrix, we need to flatten the image\n", + "(i.e. \"unravel\" the 2D matrix into a 1D array) to turn the data into a\n", + "design/feature matrix. This means we lose all spatial information in the\n", + "image, such as locality and translational invariance. More complicated\n", + "architectures such as Convolutional Neural Networks can take advantage\n", + "of such information, and are most commonly applied when analyzing\n", + "images." + ] + }, + { + "cell_type": "code", + "execution_count": 3, + "id": "54230ef9", + "metadata": { + "collapsed": false, + "editable": true + }, + "outputs": [], + "source": [ + "# import necessary packages\n", + "import numpy as np\n", + "import matplotlib.pyplot as plt\n", + "from sklearn import datasets\n", + "\n", + "\n", + "# ensure the same random numbers appear every time\n", + "np.random.seed(0)\n", + "\n", + "# display images in notebook\n", + "%matplotlib inline\n", + "plt.rcParams['figure.figsize'] = (12,12)\n", + "\n", + "\n", + "# download MNIST dataset\n", + "digits = datasets.load_digits()\n", + "\n", + "# define inputs and labels\n", + "inputs = digits.images\n", + "labels = digits.target\n", + "\n", + "print(\"inputs = (n_inputs, pixel_width, pixel_height) = \" + str(inputs.shape))\n", + "print(\"labels = (n_inputs) = \" + str(labels.shape))\n", + "\n", + "\n", + "# flatten the image\n", + "# the value -1 means dimension is inferred from the remaining dimensions: 8x8 = 64\n", + "n_inputs = len(inputs)\n", + "inputs = inputs.reshape(n_inputs, -1)\n", + "print(\"X = (n_inputs, n_features) = \" + str(inputs.shape))\n", + "\n", + "\n", + "# choose some random images to display\n", + "indices = np.arange(n_inputs)\n", + "random_indices = np.random.choice(indices, size=5)\n", + "\n", + "for i, image in enumerate(digits.images[random_indices]):\n", + " plt.subplot(1, 5, i+1)\n", + " plt.axis('off')\n", + " plt.imshow(image, cmap=plt.cm.gray_r, interpolation='nearest')\n", + " plt.title(\"Label: %d\" % digits.target[random_indices[i]])\n", + "plt.show()" + ] + }, + { + "cell_type": "markdown", + "id": "6738e7b4", + "metadata": { + "editable": true + }, + "source": [ + "## Train and test datasets\n", + "\n", + "Performing analysis before partitioning the dataset is a major error, that can lead to incorrect conclusions. \n", + "\n", + "We will reserve $80 \\%$ of our dataset for training and $20 \\%$ for testing. \n", + "\n", + "It is important that the train and test datasets are drawn randomly from our dataset, to ensure\n", + "no bias in the sampling. \n", + "Say you are taking measurements of weather data to predict the weather in the coming 5 days.\n", + "You don't want to train your model on measurements taken from the hours 00.00 to 12.00, and then test it on data\n", + "collected from 12.00 to 24.00." + ] + }, + { + "cell_type": "code", + "execution_count": 4, + "id": "f369aa9d", + "metadata": { + "collapsed": false, + "editable": true + }, + "outputs": [], + "source": [ + "from sklearn.model_selection import train_test_split\n", + "\n", + "# one-liner from scikit-learn library\n", + "train_size = 0.8\n", + "test_size = 1 - train_size\n", + "X_train, X_test, Y_train, Y_test = train_test_split(inputs, labels, train_size=train_size,\n", + " test_size=test_size)\n", + "\n", + "# equivalently in numpy\n", + "def train_test_split_numpy(inputs, labels, train_size, test_size):\n", + " n_inputs = len(inputs)\n", + " inputs_shuffled = inputs.copy()\n", + " labels_shuffled = labels.copy()\n", + " \n", + " np.random.shuffle(inputs_shuffled)\n", + " np.random.shuffle(labels_shuffled)\n", + " \n", + " train_end = int(n_inputs*train_size)\n", + " X_train, X_test = inputs_shuffled[:train_end], inputs_shuffled[train_end:]\n", + " Y_train, Y_test = labels_shuffled[:train_end], labels_shuffled[train_end:]\n", + " \n", + " return X_train, X_test, Y_train, Y_test\n", + "\n", + "#X_train, X_test, Y_train, Y_test = train_test_split_numpy(inputs, labels, train_size, test_size)\n", + "\n", + "print(\"Number of training images: \" + str(len(X_train)))\n", + "print(\"Number of test images: \" + str(len(X_test)))" + ] + }, + { + "cell_type": "markdown", + "id": "f8bf8d16", + "metadata": { + "editable": true + }, + "source": [ + "## Define model and architecture\n", + "\n", + "Our simple feed-forward neural network will consist of an *input* layer, a single *hidden* layer and an *output* layer. The activation $y$ of each neuron is a weighted sum of inputs, passed through an activation function. In case of the simple perceptron model we have \n", + "\n", + "$$ z = \\sum_{i=1}^n w_i a_i ,$$\n", + "\n", + "$$ y = f(z) ,$$\n", + "\n", + "where $f$ is the activation function, $a_i$ represents input from neuron $i$ in the preceding layer\n", + "and $w_i$ is the weight to input $i$. \n", + "The activation of the neurons in the input layer is just the features (e.g. a pixel value). \n", + "\n", + "The simplest activation function for a neuron is the *Heaviside* function:\n", + "\n", + "$$ f(z) = \n", + "\\begin{cases}\n", + "1, & z > 0\\\\\n", + "0, & \\text{otherwise}\n", + "\\end{cases}\n", + "$$\n", + "\n", + "A feed-forward neural network with this activation is known as a *perceptron*. \n", + "For a binary classifier (i.e. two classes, 0 or 1, dog or not-dog) we can also use this in our output layer. \n", + "This activation can be generalized to $k$ classes (using e.g. the *one-against-all* strategy), \n", + "and we call these architectures *multiclass perceptrons*. \n", + "\n", + "However, it is now common to use the terms Single Layer Perceptron (SLP) (1 hidden layer) and \n", + "Multilayer Perceptron (MLP) (2 or more hidden layers) to refer to feed-forward neural networks with any activation function. \n", + "\n", + "Typical choices for activation functions include the sigmoid function, hyperbolic tangent, and Rectified Linear Unit (ReLU). \n", + "We will be using the sigmoid function $\\sigma(x)$: \n", + "\n", + "$$ f(x) = \\sigma(x) = \\frac{1}{1 + e^{-x}} ,$$\n", + "\n", + "which is inspired by probability theory (see logistic regression) and was most commonly used until about 2011. See the discussion below concerning other activation functions." + ] + }, + { + "cell_type": "markdown", + "id": "112b8cc3", + "metadata": { + "editable": true + }, + "source": [ + "## Layers\n", + "\n", + "* Input \n", + "\n", + "Since each input image has 8x8 = 64 pixels or features, we have an input layer of 64 neurons. \n", + "\n", + "* Hidden layer\n", + "\n", + "We will use 50 neurons in the hidden layer receiving input from the neurons in the input layer. \n", + "Since each neuron in the hidden layer is connected to the 64 inputs we have 64x50 = 3200 weights to the hidden layer. \n", + "\n", + "* Output\n", + "\n", + "If we were building a binary classifier, it would be sufficient with a single neuron in the output layer,\n", + "which could output 0 or 1 according to the Heaviside function. This would be an example of a *hard* classifier, meaning it outputs the class of the input directly. However, if we are dealing with noisy data it is often beneficial to use a *soft* classifier, which outputs the probability of being in class 0 or 1. \n", + "\n", + "For a soft binary classifier, we could use a single neuron and interpret the output as either being the probability of being in class 0 or the probability of being in class 1. Alternatively we could use 2 neurons, and interpret each neuron as the probability of being in each class. \n", + "\n", + "Since we are doing multiclass classification, with 10 categories, it is natural to use 10 neurons in the output layer. We number the neurons $j = 0,1,...,9$. The activation of each output neuron $j$ will be according to the *softmax* function: \n", + "\n", + "$$ P(\\text{class $j$} \\mid \\text{input $\\boldsymbol{a}$}) = \\frac{\\exp{(\\boldsymbol{a}^T \\boldsymbol{w}_j)}}\n", + "{\\sum_{c=0}^{9} \\exp{(\\boldsymbol{a}^T \\boldsymbol{w}_c)}} ,$$ \n", + "\n", + "i.e. each neuron $j$ outputs the probability of being in class $j$ given an input from the hidden layer $\\boldsymbol{a}$, with $\\boldsymbol{w}_j$ the weights of neuron $j$ to the inputs. \n", + "The denominator is a normalization factor to ensure the outputs (probabilities) sum up to 1. \n", + "The exponent is just the weighted sum of inputs as before: \n", + "\n", + "$$ z_j = \\sum_{i=1}^n w_ {ij} a_i+b_j.$$ \n", + "\n", + "Since each neuron in the output layer is connected to the 50 inputs from the hidden layer we have 50x10 = 500\n", + "weights to the output layer." + ] + }, + { + "cell_type": "markdown", + "id": "e4166103", + "metadata": { + "editable": true + }, + "source": [ + "## Weights and biases\n", + "\n", + "Typically weights are initialized with small values distributed around zero, drawn from a uniform\n", + "or normal distribution. Setting all weights to zero means all neurons give the same output, making the network useless. \n", + "\n", + "Adding a bias value to the weighted sum of inputs allows the neural network to represent a greater range\n", + "of values. Without it, any input with the value 0 will be mapped to zero (before being passed through the activation). The bias unit has an output of 1, and a weight to each neuron $j$, $b_j$: \n", + "\n", + "$$ z_j = \\sum_{i=1}^n w_ {ij} a_i + b_j.$$ \n", + "\n", + "The bias weights $\\boldsymbol{b}$ are often initialized to zero, but a small value like $0.01$ ensures all neurons have some output which can be backpropagated in the first training cycle." + ] + }, + { + "cell_type": "code", + "execution_count": 5, + "id": "36399cb5", + "metadata": { + "collapsed": false, + "editable": true + }, + "outputs": [], + "source": [ + "# building our neural network\n", + "\n", + "n_inputs, n_features = X_train.shape\n", + "n_hidden_neurons = 50\n", + "n_categories = 10\n", + "\n", + "# we make the weights normally distributed using numpy.random.randn\n", + "\n", + "# weights and bias in the hidden layer\n", + "hidden_weights = np.random.randn(n_features, n_hidden_neurons)\n", + "hidden_bias = np.zeros(n_hidden_neurons) + 0.01\n", + "\n", + "# weights and bias in the output layer\n", + "output_weights = np.random.randn(n_hidden_neurons, n_categories)\n", + "output_bias = np.zeros(n_categories) + 0.01" + ] + }, + { + "cell_type": "markdown", + "id": "5eaf095b", + "metadata": { + "editable": true + }, + "source": [ + "## Feed-forward pass\n", + "\n", + "Denote $F$ the number of features, $H$ the number of hidden neurons and $C$ the number of categories. \n", + "For each input image we calculate a weighted sum of input features (pixel values) to each neuron $j$ in the hidden layer $l$: \n", + "\n", + "$$ z_{j}^{l} = \\sum_{i=1}^{F} w_{ij}^{l} x_i + b_{j}^{l},$$\n", + "\n", + "this is then passed through our activation function \n", + "\n", + "$$ a_{j}^{l} = f(z_{j}^{l}) .$$ \n", + "\n", + "We calculate a weighted sum of inputs (activations in the hidden layer) to each neuron $j$ in the output layer: \n", + "\n", + "$$ z_{j}^{L} = \\sum_{i=1}^{H} w_{ij}^{L} a_{i}^{l} + b_{j}^{L}.$$ \n", + "\n", + "Finally we calculate the output of neuron $j$ in the output layer using the softmax function: \n", + "\n", + "$$ a_{j}^{L} = \\frac{\\exp{(z_j^{L})}}\n", + "{\\sum_{c=0}^{C-1} \\exp{(z_c^{L})}} .$$" + ] + }, + { + "cell_type": "markdown", + "id": "91eec6b8", + "metadata": { + "editable": true + }, + "source": [ + "## Matrix multiplications\n", + "\n", + "Since our data has the dimensions $X = (n_{inputs}, n_{features})$ and our weights to the hidden\n", + "layer have the dimensions \n", + "$W_{hidden} = (n_{features}, n_{hidden})$,\n", + "we can easily feed the network all our training data in one go by taking the matrix product \n", + "\n", + "$$ X W^{h} = (n_{inputs}, n_{hidden}),$$ \n", + "\n", + "and obtain a matrix that holds the weighted sum of inputs to the hidden layer\n", + "for each input image and each hidden neuron. \n", + "We also add the bias to obtain a matrix of weighted sums to the hidden layer $Z^{h}$: \n", + "\n", + "$$ \\boldsymbol{z}^{l} = \\boldsymbol{X} \\boldsymbol{W}^{l} + \\boldsymbol{b}^{l} ,$$\n", + "\n", + "meaning the same bias (1D array with size equal number of hidden neurons) is added to each input image. \n", + "This is then passed through the activation: \n", + "\n", + "$$ \\boldsymbol{a}^{l} = f(\\boldsymbol{z}^l) .$$ \n", + "\n", + "This is fed to the output layer: \n", + "\n", + "$$ \\boldsymbol{z}^{L} = \\boldsymbol{a}^{L} \\boldsymbol{W}^{L} + \\boldsymbol{b}^{L} .$$\n", + "\n", + "Finally we receive our output values for each image and each category by passing it through the softmax function: \n", + "\n", + "$$ output = softmax (\\boldsymbol{z}^{L}) = (n_{inputs}, n_{categories}) .$$" + ] + }, + { + "cell_type": "code", + "execution_count": 6, + "id": "ef497434", + "metadata": { + "collapsed": false, + "editable": true + }, + "outputs": [], + "source": [ + "# setup the feed-forward pass, subscript h = hidden layer\n", + "\n", + "def sigmoid(x):\n", + " return 1/(1 + np.exp(-x))\n", + "\n", + "def feed_forward(X):\n", + " # weighted sum of inputs to the hidden layer\n", + " z_h = np.matmul(X, hidden_weights) + hidden_bias\n", + " # activation in the hidden layer\n", + " a_h = sigmoid(z_h)\n", + " \n", + " # weighted sum of inputs to the output layer\n", + " z_o = np.matmul(a_h, output_weights) + output_bias\n", + " # softmax output\n", + " # axis 0 holds each input and axis 1 the probabilities of each category\n", + " exp_term = np.exp(z_o)\n", + " probabilities = exp_term / np.sum(exp_term, axis=1, keepdims=True)\n", + " \n", + " return probabilities\n", + "\n", + "probabilities = feed_forward(X_train)\n", + "print(\"probabilities = (n_inputs, n_categories) = \" + str(probabilities.shape))\n", + "print(\"probability that image 0 is in category 0,1,2,...,9 = \\n\" + str(probabilities[0]))\n", + "print(\"probabilities sum up to: \" + str(probabilities[0].sum()))\n", + "print()\n", + "\n", + "# we obtain a prediction by taking the class with the highest likelihood\n", + "def predict(X):\n", + " probabilities = feed_forward(X)\n", + " return np.argmax(probabilities, axis=1)\n", + "\n", + "predictions = predict(X_train)\n", + "print(\"predictions = (n_inputs) = \" + str(predictions.shape))\n", + "print(\"prediction for image 0: \" + str(predictions[0]))\n", + "print(\"correct label for image 0: \" + str(Y_train[0]))" + ] + }, + { + "cell_type": "markdown", + "id": "ab88238a", + "metadata": { + "editable": true + }, + "source": [ + "## Choose cost function and optimizer\n", + "\n", + "To measure how well our neural network is doing we need to introduce a cost function. \n", + "We will call the function that gives the error of a single sample output the *loss* function, and the function\n", + "that gives the total error of our network across all samples the *cost* function.\n", + "A typical choice for multiclass classification is the *cross-entropy* loss, also known as the negative log likelihood. \n", + "\n", + "In *multiclass* classification it is common to treat each integer label as a so called *one-hot* vector: \n", + "\n", + "$$ y = 5 \\quad \\rightarrow \\quad \\boldsymbol{y} = (0, 0, 0, 0, 0, 1, 0, 0, 0, 0) ,$$ \n", + "\n", + "$$ y = 1 \\quad \\rightarrow \\quad \\boldsymbol{y} = (0, 1, 0, 0, 0, 0, 0, 0, 0, 0) ,$$ \n", + "\n", + "i.e. a binary bit string of length $C$, where $C = 10$ is the number of classes in the MNIST dataset. \n", + "\n", + "Let $y_{ic}$ denote the $c$-th component of the $i$-th one-hot vector. \n", + "We define the cost function $\\mathcal{C}$ as a sum over the cross-entropy loss for each point $\\boldsymbol{x}_i$ in the dataset.\n", + "\n", + "In the one-hot representation only one of the terms in the loss function is non-zero, namely the\n", + "probability of the correct category $c'$ \n", + "(i.e. the category $c'$ such that $y_{ic'} = 1$). This means that the cross entropy loss only punishes you for how wrong\n", + "you got the correct label. The probability of category $c$ is given by the softmax function. The vector $\\boldsymbol{\\theta}$ represents the parameters of our network, i.e. all the weights and biases." + ] + }, + { + "cell_type": "markdown", + "id": "506145ff", + "metadata": { + "editable": true + }, + "source": [ + "## Optimizing the cost function\n", + "\n", + "The network is trained by finding the weights and biases that minimize the cost function. One of the most widely used classes of methods is *gradient descent* and its generalizations. The idea behind gradient descent\n", + "is simply to adjust the weights in the direction where the gradient of the cost function is large and negative. This ensures we flow toward a *local* minimum of the cost function. \n", + "Each parameter $\\theta$ is iteratively adjusted according to the rule \n", + "\n", + "$$ \\theta_{i+1} = \\theta_i - \\eta \\nabla \\mathcal{C}(\\theta_i) ,$$\n", + "\n", + "where $\\eta$ is known as the *learning rate*, which controls how big a step we take towards the minimum. \n", + "This update can be repeated for any number of iterations, or until we are satisfied with the result. \n", + "\n", + "A simple and effective improvement is a variant called *Batch Gradient Descent*. \n", + "Instead of calculating the gradient on the whole dataset, we calculate an approximation of the gradient\n", + "on a subset of the data called a *minibatch*. \n", + "If there are $N$ data points and we have a minibatch size of $M$, the total number of batches\n", + "is $N/M$. \n", + "We denote each minibatch $B_k$, with $k = 1, 2,...,N/M$. The gradient then becomes: \n", + "\n", + "$$ \\nabla \\mathcal{C}(\\theta) = \\frac{1}{N} \\sum_{i=1}^N \\nabla \\mathcal{L}_i(\\theta) \\quad \\rightarrow \\quad\n", + "\\frac{1}{M} \\sum_{i \\in B_k} \\nabla \\mathcal{L}_i(\\theta) ,$$\n", + "\n", + "i.e. instead of averaging the loss over the entire dataset, we average over a minibatch. \n", + "\n", + "This has two important benefits: \n", + "1. Introducing stochasticity decreases the chance that the algorithm becomes stuck in a local minima. \n", + "\n", + "2. It significantly speeds up the calculation, since we do not have to use the entire dataset to calculate the gradient. \n", + "\n", + "The various optmization methods, with codes and algorithms, are discussed in our lectures on [Gradient descent approaches](https://compphysics.github.io/MachineLearning/doc/pub/Splines/html/Splines-bs.html)." + ] + }, + { + "cell_type": "markdown", + "id": "e8138025", + "metadata": { + "editable": true + }, + "source": [ + "## Regularization\n", + "\n", + "It is common to add an extra term to the cost function, proportional\n", + "to the size of the weights. This is equivalent to constraining the\n", + "size of the weights, so that they do not grow out of control.\n", + "Constraining the size of the weights means that the weights cannot\n", + "grow arbitrarily large to fit the training data, and in this way\n", + "reduces *overfitting*.\n", + "\n", + "We will measure the size of the weights using the so called *L2-norm*, meaning our cost function becomes: \n", + "\n", + "$$ \\mathcal{C}(\\theta) = \\frac{1}{N} \\sum_{i=1}^N \\mathcal{L}_i(\\theta) \\quad \\rightarrow \\quad\n", + "\\frac{1}{N} \\sum_{i=1}^N \\mathcal{L}_i(\\theta) + \\lambda \\lvert \\lvert \\boldsymbol{w} \\rvert \\rvert_2^2 \n", + "= \\frac{1}{N} \\sum_{i=1}^N \\mathcal{L}(\\theta) + \\lambda \\sum_{ij} w_{ij}^2,$$ \n", + "\n", + "i.e. we sum up all the weights squared. The factor $\\lambda$ is known as a regularization parameter.\n", + "\n", + "In order to train the model, we need to calculate the derivative of\n", + "the cost function with respect to every bias and weight in the\n", + "network. In total our network has $(64 + 1)\\times 50=3250$ weights in\n", + "the hidden layer and $(50 + 1)\\times 10=510$ weights to the output\n", + "layer ($+1$ for the bias), and the gradient must be calculated for\n", + "every parameter. We use the *backpropagation* algorithm discussed\n", + "above. This is a clever use of the chain rule that allows us to\n", + "calculate the gradient efficently." + ] + }, + { + "cell_type": "markdown", + "id": "25becf76", + "metadata": { + "editable": true + }, + "source": [ + "## Matrix multiplication\n", + "\n", + "To more efficently train our network these equations are implemented using matrix operations. \n", + "The error in the output layer is calculated simply as, with $\\boldsymbol{t}$ being our targets, \n", + "\n", + "$$ \\delta_L = \\boldsymbol{t} - \\boldsymbol{y} = (n_{inputs}, n_{categories}) .$$ \n", + "\n", + "The gradient for the output weights is calculated as \n", + "\n", + "$$ \\nabla W_{L} = \\boldsymbol{a}^T \\delta_L = (n_{hidden}, n_{categories}) ,$$\n", + "\n", + "where $\\boldsymbol{a} = (n_{inputs}, n_{hidden})$. This simply means that we are summing up the gradients for each input. \n", + "Since we are going backwards we have to transpose the activation matrix. \n", + "\n", + "The gradient with respect to the output bias is then \n", + "\n", + "$$ \\nabla \\boldsymbol{b}_{L} = \\sum_{i=1}^{n_{inputs}} \\delta_L = (n_{categories}) .$$ \n", + "\n", + "The error in the hidden layer is \n", + "\n", + "$$ \\Delta_h = \\delta_L W_{L}^T \\circ f'(z_{h}) = \\delta_L W_{L}^T \\circ a_{h} \\circ (1 - a_{h}) = (n_{inputs}, n_{hidden}) ,$$ \n", + "\n", + "where $f'(a_{h})$ is the derivative of the activation in the hidden layer. The matrix products mean\n", + "that we are summing up the products for each neuron in the output layer. The symbol $\\circ$ denotes\n", + "the *Hadamard product*, meaning element-wise multiplication. \n", + "\n", + "This again gives us the gradients in the hidden layer: \n", + "\n", + "$$ \\nabla W_{h} = X^T \\delta_h = (n_{features}, n_{hidden}) ,$$ \n", + "\n", + "$$ \\nabla b_{h} = \\sum_{i=1}^{n_{inputs}} \\delta_h = (n_{hidden}) .$$" + ] + }, + { + "cell_type": "code", + "execution_count": 7, + "id": "657c604c", + "metadata": { + "collapsed": false, + "editable": true + }, + "outputs": [], + "source": [ + "# to categorical turns our integer vector into a onehot representation\n", + "from sklearn.metrics import accuracy_score\n", + "\n", + "# one-hot in numpy\n", + "def to_categorical_numpy(integer_vector):\n", + " n_inputs = len(integer_vector)\n", + " n_categories = np.max(integer_vector) + 1\n", + " onehot_vector = np.zeros((n_inputs, n_categories))\n", + " onehot_vector[range(n_inputs), integer_vector] = 1\n", + " \n", + " return onehot_vector\n", + "\n", + "#Y_train_onehot, Y_test_onehot = to_categorical(Y_train), to_categorical(Y_test)\n", + "Y_train_onehot, Y_test_onehot = to_categorical_numpy(Y_train), to_categorical_numpy(Y_test)\n", + "\n", + "def feed_forward_train(X):\n", + " # weighted sum of inputs to the hidden layer\n", + " z_h = np.matmul(X, hidden_weights) + hidden_bias\n", + " # activation in the hidden layer\n", + " a_h = sigmoid(z_h)\n", + " \n", + " # weighted sum of inputs to the output layer\n", + " z_o = np.matmul(a_h, output_weights) + output_bias\n", + " # softmax output\n", + " # axis 0 holds each input and axis 1 the probabilities of each category\n", + " exp_term = np.exp(z_o)\n", + " probabilities = exp_term / np.sum(exp_term, axis=1, keepdims=True)\n", + " \n", + " # for backpropagation need activations in hidden and output layers\n", + " return a_h, probabilities\n", + "\n", + "def backpropagation(X, Y):\n", + " a_h, probabilities = feed_forward_train(X)\n", + " \n", + " # error in the output layer\n", + " error_output = probabilities - Y\n", + " # error in the hidden layer\n", + " error_hidden = np.matmul(error_output, output_weights.T) * a_h * (1 - a_h)\n", + " \n", + " # gradients for the output layer\n", + " output_weights_gradient = np.matmul(a_h.T, error_output)\n", + " output_bias_gradient = np.sum(error_output, axis=0)\n", + " \n", + " # gradient for the hidden layer\n", + " hidden_weights_gradient = np.matmul(X.T, error_hidden)\n", + " hidden_bias_gradient = np.sum(error_hidden, axis=0)\n", + "\n", + " return output_weights_gradient, output_bias_gradient, hidden_weights_gradient, hidden_bias_gradient\n", + "\n", + "print(\"Old accuracy on training data: \" + str(accuracy_score(predict(X_train), Y_train)))\n", + "\n", + "eta = 0.01\n", + "lmbd = 0.01\n", + "for i in range(1000):\n", + " # calculate gradients\n", + " dWo, dBo, dWh, dBh = backpropagation(X_train, Y_train_onehot)\n", + " \n", + " # regularization term gradients\n", + " dWo += lmbd * output_weights\n", + " dWh += lmbd * hidden_weights\n", + " \n", + " # update weights and biases\n", + " output_weights -= eta * dWo\n", + " output_bias -= eta * dBo\n", + " hidden_weights -= eta * dWh\n", + " hidden_bias -= eta * dBh\n", + "\n", + "print(\"New accuracy on training data: \" + str(accuracy_score(predict(X_train), Y_train)))" + ] + }, + { + "cell_type": "markdown", + "id": "3c595805", + "metadata": { + "editable": true + }, + "source": [ + "## Improving performance\n", + "\n", + "As we can see the network does not seem to be learning at all. It seems to be just guessing the label for each image. \n", + "In order to obtain a network that does something useful, we will have to do a bit more work. \n", + "\n", + "The choice of *hyperparameters* such as learning rate and regularization parameter is hugely influential for the performance of the network. Typically a *grid-search* is performed, wherein we test different hyperparameters separated by orders of magnitude. For example we could test the learning rates $\\eta = 10^{-6}, 10^{-5},...,10^{-1}$ with different regularization parameters $\\lambda = 10^{-6},...,10^{-0}$. \n", + "\n", + "Next, we haven't implemented minibatching yet, which introduces stochasticity and is though to act as an important regularizer on the weights. We call a feed-forward + backward pass with a minibatch an *iteration*, and a full training period\n", + "going through the entire dataset ($n/M$ batches) an *epoch*.\n", + "\n", + "If this does not improve network performance, you may want to consider altering the network architecture, adding more neurons or hidden layers. \n", + "Andrew Ng goes through some of these considerations in this [video](https://youtu.be/F1ka6a13S9I). You can find a summary of the video [here](https://kevinzakka.github.io/2016/09/26/applying-deep-learning/)." + ] + }, + { + "cell_type": "markdown", + "id": "17d5f534", + "metadata": { + "editable": true + }, + "source": [ + "## Full object-oriented implementation\n", + "\n", + "It is very natural to think of the network as an object, with specific instances of the network\n", + "being realizations of this object with different hyperparameters. An implementation using Python classes provides a clean structure and interface, and the full implementation of our neural network is given below." + ] + }, + { + "cell_type": "code", + "execution_count": 8, + "id": "5aeaa0cd", + "metadata": { + "collapsed": false, + "editable": true + }, + "outputs": [], + "source": [ + "class NeuralNetwork:\n", + " def __init__(\n", + " self,\n", + " X_data,\n", + " Y_data,\n", + " n_hidden_neurons=50,\n", + " n_categories=10,\n", + " epochs=10,\n", + " batch_size=100,\n", + " eta=0.1,\n", + " lmbd=0.0):\n", + "\n", + " self.X_data_full = X_data\n", + " self.Y_data_full = Y_data\n", + "\n", + " self.n_inputs = X_data.shape[0]\n", + " self.n_features = X_data.shape[1]\n", + " self.n_hidden_neurons = n_hidden_neurons\n", + " self.n_categories = n_categories\n", + "\n", + " self.epochs = epochs\n", + " self.batch_size = batch_size\n", + " self.iterations = self.n_inputs // self.batch_size\n", + " self.eta = eta\n", + " self.lmbd = lmbd\n", + "\n", + " self.create_biases_and_weights()\n", + "\n", + " def create_biases_and_weights(self):\n", + " self.hidden_weights = np.random.randn(self.n_features, self.n_hidden_neurons)\n", + " self.hidden_bias = np.zeros(self.n_hidden_neurons) + 0.01\n", + "\n", + " self.output_weights = np.random.randn(self.n_hidden_neurons, self.n_categories)\n", + " self.output_bias = np.zeros(self.n_categories) + 0.01\n", + "\n", + " def feed_forward(self):\n", + " # feed-forward for training\n", + " self.z_h = np.matmul(self.X_data, self.hidden_weights) + self.hidden_bias\n", + " self.a_h = sigmoid(self.z_h)\n", + "\n", + " self.z_o = np.matmul(self.a_h, self.output_weights) + self.output_bias\n", + "\n", + " exp_term = np.exp(self.z_o)\n", + " self.probabilities = exp_term / np.sum(exp_term, axis=1, keepdims=True)\n", + "\n", + " def feed_forward_out(self, X):\n", + " # feed-forward for output\n", + " z_h = np.matmul(X, self.hidden_weights) + self.hidden_bias\n", + " a_h = sigmoid(z_h)\n", + "\n", + " z_o = np.matmul(a_h, self.output_weights) + self.output_bias\n", + " \n", + " exp_term = np.exp(z_o)\n", + " probabilities = exp_term / np.sum(exp_term, axis=1, keepdims=True)\n", + " return probabilities\n", + "\n", + " def backpropagation(self):\n", + " error_output = self.probabilities - self.Y_data\n", + " error_hidden = np.matmul(error_output, self.output_weights.T) * self.a_h * (1 - self.a_h)\n", + "\n", + " self.output_weights_gradient = np.matmul(self.a_h.T, error_output)\n", + " self.output_bias_gradient = np.sum(error_output, axis=0)\n", + "\n", + " self.hidden_weights_gradient = np.matmul(self.X_data.T, error_hidden)\n", + " self.hidden_bias_gradient = np.sum(error_hidden, axis=0)\n", + "\n", + " if self.lmbd > 0.0:\n", + " self.output_weights_gradient += self.lmbd * self.output_weights\n", + " self.hidden_weights_gradient += self.lmbd * self.hidden_weights\n", + "\n", + " self.output_weights -= self.eta * self.output_weights_gradient\n", + " self.output_bias -= self.eta * self.output_bias_gradient\n", + " self.hidden_weights -= self.eta * self.hidden_weights_gradient\n", + " self.hidden_bias -= self.eta * self.hidden_bias_gradient\n", + "\n", + " def predict(self, X):\n", + " probabilities = self.feed_forward_out(X)\n", + " return np.argmax(probabilities, axis=1)\n", + "\n", + " def predict_probabilities(self, X):\n", + " probabilities = self.feed_forward_out(X)\n", + " return probabilities\n", + "\n", + " def train(self):\n", + " data_indices = np.arange(self.n_inputs)\n", + "\n", + " for i in range(self.epochs):\n", + " for j in range(self.iterations):\n", + " # pick datapoints with replacement\n", + " chosen_datapoints = np.random.choice(\n", + " data_indices, size=self.batch_size, replace=False\n", + " )\n", + "\n", + " # minibatch training data\n", + " self.X_data = self.X_data_full[chosen_datapoints]\n", + " self.Y_data = self.Y_data_full[chosen_datapoints]\n", + "\n", + " self.feed_forward()\n", + " self.backpropagation()" + ] + }, + { + "cell_type": "markdown", + "id": "77f30e1e", + "metadata": { + "editable": true + }, + "source": [ + "## Evaluate model performance on test data\n", + "\n", + "To measure the performance of our network we evaluate how well it does it data it has never seen before, i.e. the test data. \n", + "We measure the performance of the network using the *accuracy* score. \n", + "The accuracy is as you would expect just the number of images correctly labeled divided by the total number of images. A perfect classifier will have an accuracy score of $1$. \n", + "\n", + "$$ \\text{Accuracy} = \\frac{\\sum_{i=1}^n I(\\tilde{y}_i = y_i)}{n} ,$$ \n", + "\n", + "where $I$ is the indicator function, $1$ if $\\tilde{y}_i = y_i$ and $0$ otherwise." + ] + }, + { + "cell_type": "code", + "execution_count": 9, + "id": "c1ab41e1", + "metadata": { + "collapsed": false, + "editable": true + }, + "outputs": [], + "source": [ + "epochs = 100\n", + "batch_size = 100\n", + "\n", + "dnn = NeuralNetwork(X_train, Y_train_onehot, eta=eta, lmbd=lmbd, epochs=epochs, batch_size=batch_size,\n", + " n_hidden_neurons=n_hidden_neurons, n_categories=n_categories)\n", + "dnn.train()\n", + "test_predict = dnn.predict(X_test)\n", + "\n", + "# accuracy score from scikit library\n", + "print(\"Accuracy score on test set: \", accuracy_score(Y_test, test_predict))\n", + "\n", + "# equivalent in numpy\n", + "def accuracy_score_numpy(Y_test, Y_pred):\n", + " return np.sum(Y_test == Y_pred) / len(Y_test)\n", + "\n", + "#print(\"Accuracy score on test set: \", accuracy_score_numpy(Y_test, test_predict))" + ] + }, + { + "cell_type": "markdown", + "id": "1b2b35c3", + "metadata": { + "editable": true + }, + "source": [ + "## Adjust hyperparameters\n", + "\n", + "We now perform a grid search to find the optimal hyperparameters for the network. \n", + "Note that we are only using 1 layer with 50 neurons, and human performance is estimated to be around $98\\%$ ($2\\%$ error rate)." + ] + }, + { + "cell_type": "code", + "execution_count": 10, + "id": "ac2ab4a6", + "metadata": { + "collapsed": false, + "editable": true + }, + "outputs": [], + "source": [ + "eta_vals = np.logspace(-5, 1, 7)\n", + "lmbd_vals = np.logspace(-5, 1, 7)\n", + "# store the models for later use\n", + "DNN_numpy = np.zeros((len(eta_vals), len(lmbd_vals)), dtype=object)\n", + "\n", + "# grid search\n", + "for i, eta in enumerate(eta_vals):\n", + " for j, lmbd in enumerate(lmbd_vals):\n", + " dnn = NeuralNetwork(X_train, Y_train_onehot, eta=eta, lmbd=lmbd, epochs=epochs, batch_size=batch_size,\n", + " n_hidden_neurons=n_hidden_neurons, n_categories=n_categories)\n", + " dnn.train()\n", + " \n", + " DNN_numpy[i][j] = dnn\n", + " \n", + " test_predict = dnn.predict(X_test)\n", + " \n", + " print(\"Learning rate = \", eta)\n", + " print(\"Lambda = \", lmbd)\n", + " print(\"Accuracy score on test set: \", accuracy_score(Y_test, test_predict))\n", + " print()" + ] + }, + { + "cell_type": "markdown", + "id": "f3e5dfec", + "metadata": { + "editable": true + }, + "source": [ + "## Visualization" + ] + }, + { + "cell_type": "code", + "execution_count": 11, + "id": "6d875555", + "metadata": { + "collapsed": false, + "editable": true + }, + "outputs": [], + "source": [ + "# visual representation of grid search\n", + "# uses seaborn heatmap, you can also do this with matplotlib imshow\n", + "import seaborn as sns\n", + "\n", + "sns.set()\n", + "\n", + "train_accuracy = np.zeros((len(eta_vals), len(lmbd_vals)))\n", + "test_accuracy = np.zeros((len(eta_vals), len(lmbd_vals)))\n", + "\n", + "for i in range(len(eta_vals)):\n", + " for j in range(len(lmbd_vals)):\n", + " dnn = DNN_numpy[i][j]\n", + " \n", + " train_pred = dnn.predict(X_train) \n", + " test_pred = dnn.predict(X_test)\n", + "\n", + " train_accuracy[i][j] = accuracy_score(Y_train, train_pred)\n", + " test_accuracy[i][j] = accuracy_score(Y_test, test_pred)\n", + "\n", + " \n", + "fig, ax = plt.subplots(figsize = (10, 10))\n", + "sns.heatmap(train_accuracy, annot=True, ax=ax, cmap=\"viridis\")\n", + "ax.set_title(\"Training Accuracy\")\n", + "ax.set_ylabel(\"$\\eta$\")\n", + "ax.set_xlabel(\"$\\lambda$\")\n", + "plt.show()\n", + "\n", + "fig, ax = plt.subplots(figsize = (10, 10))\n", + "sns.heatmap(test_accuracy, annot=True, ax=ax, cmap=\"viridis\")\n", + "ax.set_title(\"Test Accuracy\")\n", + "ax.set_ylabel(\"$\\eta$\")\n", + "ax.set_xlabel(\"$\\lambda$\")\n", + "plt.show()" + ] + }, + { + "cell_type": "markdown", + "id": "81c742e3", + "metadata": { + "editable": true + }, + "source": [ + "## scikit-learn implementation\n", + "\n", + "**scikit-learn** focuses more\n", + "on traditional machine learning methods, such as regression,\n", + "clustering, decision trees, etc. As such, it has only two types of\n", + "neural networks: Multi Layer Perceptron outputting continuous values,\n", + "*MPLRegressor*, and Multi Layer Perceptron outputting labels,\n", + "*MLPClassifier*. We will see how simple it is to use these classes.\n", + "\n", + "**scikit-learn** implements a few improvements from our neural network,\n", + "such as early stopping, a varying learning rate, different\n", + "optimization methods, etc. We would therefore expect a better\n", + "performance overall." + ] + }, + { + "cell_type": "code", + "execution_count": 12, + "id": "d21828bb", + "metadata": { + "collapsed": false, + "editable": true + }, + "outputs": [], + "source": [ + "from sklearn.neural_network import MLPClassifier\n", + "# store models for later use\n", + "DNN_scikit = np.zeros((len(eta_vals), len(lmbd_vals)), dtype=object)\n", + "\n", + "for i, eta in enumerate(eta_vals):\n", + " for j, lmbd in enumerate(lmbd_vals):\n", + " dnn = MLPClassifier(hidden_layer_sizes=(n_hidden_neurons), activation='logistic',\n", + " alpha=lmbd, learning_rate_init=eta, max_iter=epochs)\n", + " dnn.fit(X_train, Y_train)\n", + " \n", + " DNN_scikit[i][j] = dnn\n", + " \n", + " print(\"Learning rate = \", eta)\n", + " print(\"Lambda = \", lmbd)\n", + " print(\"Accuracy score on test set: \", dnn.score(X_test, Y_test))\n", + " print()" + ] + }, + { + "cell_type": "markdown", + "id": "4b40579d", + "metadata": { + "editable": true + }, + "source": [ + "## Visualization" + ] + }, + { + "cell_type": "code", + "execution_count": 13, + "id": "bfc97198", + "metadata": { + "collapsed": false, + "editable": true + }, + "outputs": [], + "source": [ + "# optional\n", + "# visual representation of grid search\n", + "# uses seaborn heatmap, could probably do this in matplotlib\n", + "import seaborn as sns\n", + "\n", + "sns.set()\n", + "\n", + "train_accuracy = np.zeros((len(eta_vals), len(lmbd_vals)))\n", + "test_accuracy = np.zeros((len(eta_vals), len(lmbd_vals)))\n", + "\n", + "for i in range(len(eta_vals)):\n", + " for j in range(len(lmbd_vals)):\n", + " dnn = DNN_scikit[i][j]\n", + " \n", + " train_pred = dnn.predict(X_train) \n", + " test_pred = dnn.predict(X_test)\n", + "\n", + " train_accuracy[i][j] = accuracy_score(Y_train, train_pred)\n", + " test_accuracy[i][j] = accuracy_score(Y_test, test_pred)\n", + "\n", + " \n", + "fig, ax = plt.subplots(figsize = (10, 10))\n", + "sns.heatmap(train_accuracy, annot=True, ax=ax, cmap=\"viridis\")\n", + "ax.set_title(\"Training Accuracy\")\n", + "ax.set_ylabel(\"$\\eta$\")\n", + "ax.set_xlabel(\"$\\lambda$\")\n", + "plt.show()\n", + "\n", + "fig, ax = plt.subplots(figsize = (10, 10))\n", + "sns.heatmap(test_accuracy, annot=True, ax=ax, cmap=\"viridis\")\n", + "ax.set_title(\"Test Accuracy\")\n", + "ax.set_ylabel(\"$\\eta$\")\n", + "ax.set_xlabel(\"$\\lambda$\")\n", + "plt.show()" + ] + }, + { + "cell_type": "markdown", + "id": "550680d5", + "metadata": { + "editable": true + }, + "source": [ + "## Building neural networks in Tensorflow and Keras\n", + "\n", + "Now we want to build on the experience gained from our neural network implementation in NumPy and scikit-learn\n", + "and use it to construct a neural network in Tensorflow. Once we have constructed a neural network in NumPy\n", + "and Tensorflow, building one in Keras is really quite trivial, though the performance may suffer. \n", + "\n", + "In our previous example we used only one hidden layer, and in this we will use two. From this it should be quite\n", + "clear how to build one using an arbitrary number of hidden layers, using data structures such as Python lists or\n", + "NumPy arrays." + ] + }, + { + "cell_type": "markdown", + "id": "bbade60e", + "metadata": { + "editable": true + }, + "source": [ + "## Tensorflow\n", + "\n", + "Tensorflow is an open source library machine learning library\n", + "developed by the Google Brain team for internal use. It was released\n", + "under the Apache 2.0 open source license in November 9, 2015.\n", + "\n", + "Tensorflow is a computational framework that allows you to construct\n", + "machine learning models at different levels of abstraction, from\n", + "high-level, object-oriented APIs like Keras, down to the C++ kernels\n", + "that Tensorflow is built upon. The higher levels of abstraction are\n", + "simpler to use, but less flexible, and our choice of implementation\n", + "should reflect the problems we are trying to solve.\n", + "\n", + "[Tensorflow uses](https://www.tensorflow.org/guide/graphs) so-called graphs to represent your computation\n", + "in terms of the dependencies between individual operations, such that you first build a Tensorflow *graph*\n", + "to represent your model, and then create a Tensorflow *session* to run the graph.\n", + "\n", + "In this guide we will analyze the same data as we did in our NumPy and\n", + "scikit-learn tutorial, gathered from the MNIST database of images. We\n", + "will give an introduction to the lower level Python Application\n", + "Program Interfaces (APIs), and see how we use them to build our graph.\n", + "Then we will build (effectively) the same graph in Keras, to see just\n", + "how simple solving a machine learning problem can be.\n", + "\n", + "To install tensorflow on Unix/Linux systems, use pip as" + ] + }, + { + "cell_type": "code", + "execution_count": 14, + "id": "e5505ed3", + "metadata": { + "collapsed": false, + "editable": true + }, + "outputs": [], + "source": [ + "pip3 install tensorflow" + ] + }, + { + "cell_type": "markdown", + "id": "98d4797a", + "metadata": { + "editable": true + }, + "source": [ + "and/or if you use **anaconda**, just write (or install from the graphical user interface)\n", + "(current release of CPU-only TensorFlow)" + ] + }, + { + "cell_type": "code", + "execution_count": 15, + "id": "95b05730", + "metadata": { + "collapsed": false, + "editable": true + }, + "outputs": [], + "source": [ + "conda create -n tf tensorflow\n", + "conda activate tf" + ] + }, + { + "cell_type": "markdown", + "id": "94170f18", + "metadata": { + "editable": true + }, + "source": [ + "To install the current release of GPU TensorFlow" + ] + }, + { + "cell_type": "code", + "execution_count": 16, + "id": "65ae6e69", + "metadata": { + "collapsed": false, + "editable": true + }, + "outputs": [], + "source": [ + "conda create -n tf-gpu tensorflow-gpu\n", + "conda activate tf-gpu" + ] + }, + { + "cell_type": "markdown", + "id": "df39c5c6", + "metadata": { + "editable": true + }, + "source": [ + "## Using Keras\n", + "\n", + "Keras is a high level [neural network](https://en.wikipedia.org/wiki/Application_programming_interface)\n", + "that supports Tensorflow, CTNK and Theano as backends. \n", + "If you have Anaconda installed you may run the following command" + ] + }, + { + "cell_type": "code", + "execution_count": 17, + "id": "d0d627f6", + "metadata": { + "collapsed": false, + "editable": true + }, + "outputs": [], + "source": [ + "conda install keras" + ] + }, + { + "cell_type": "markdown", + "id": "4124b5af", + "metadata": { + "editable": true + }, + "source": [ + "You can look up the [instructions here](https://keras.io/) for more information.\n", + "\n", + "We will to a large extent use **keras** in this course." + ] + }, + { + "cell_type": "markdown", + "id": "22dc977c", + "metadata": { + "editable": true + }, + "source": [ + "## Collect and pre-process data\n", + "\n", + "Let us look again at the MINST data set." + ] + }, + { + "cell_type": "code", + "execution_count": 18, + "id": "7df2e4d4", + "metadata": { + "collapsed": false, + "editable": true + }, + "outputs": [], + "source": [ + "# import necessary packages\n", + "import numpy as np\n", + "import matplotlib.pyplot as plt\n", + "import tensorflow as tf\n", + "from sklearn import datasets\n", + "\n", + "\n", + "# ensure the same random numbers appear every time\n", + "np.random.seed(0)\n", + "\n", + "# display images in notebook\n", + "%matplotlib inline\n", + "plt.rcParams['figure.figsize'] = (12,12)\n", + "\n", + "\n", + "# download MNIST dataset\n", + "digits = datasets.load_digits()\n", + "\n", + "# define inputs and labels\n", + "inputs = digits.images\n", + "labels = digits.target\n", + "\n", + "print(\"inputs = (n_inputs, pixel_width, pixel_height) = \" + str(inputs.shape))\n", + "print(\"labels = (n_inputs) = \" + str(labels.shape))\n", + "\n", + "\n", + "# flatten the image\n", + "# the value -1 means dimension is inferred from the remaining dimensions: 8x8 = 64\n", + "n_inputs = len(inputs)\n", + "inputs = inputs.reshape(n_inputs, -1)\n", + "print(\"X = (n_inputs, n_features) = \" + str(inputs.shape))\n", + "\n", + "\n", + "# choose some random images to display\n", + "indices = np.arange(n_inputs)\n", + "random_indices = np.random.choice(indices, size=5)\n", + "\n", + "for i, image in enumerate(digits.images[random_indices]):\n", + " plt.subplot(1, 5, i+1)\n", + " plt.axis('off')\n", + " plt.imshow(image, cmap=plt.cm.gray_r, interpolation='nearest')\n", + " plt.title(\"Label: %d\" % digits.target[random_indices[i]])\n", + "plt.show()" + ] + }, + { + "cell_type": "code", + "execution_count": 19, + "id": "4f5bff6f", + "metadata": { + "collapsed": false, + "editable": true + }, + "outputs": [], + "source": [ + "from tensorflow.keras.layers import Input\n", + "from tensorflow.keras.models import Sequential #This allows appending layers to existing models\n", + "from tensorflow.keras.layers import Dense #This allows defining the characteristics of a particular layer\n", + "from tensorflow.keras import optimizers #This allows using whichever optimiser we want (sgd,adam,RMSprop)\n", + "from tensorflow.keras import regularizers #This allows using whichever regularizer we want (l1,l2,l1_l2)\n", + "from tensorflow.keras.utils import to_categorical #This allows using categorical cross entropy as the cost function\n", + "\n", + "from sklearn.model_selection import train_test_split\n", + "\n", + "# one-hot representation of labels\n", + "labels = to_categorical(labels)\n", + "\n", + "# split into train and test data\n", + "train_size = 0.8\n", + "test_size = 1 - train_size\n", + "X_train, X_test, Y_train, Y_test = train_test_split(inputs, labels, train_size=train_size,\n", + " test_size=test_size)" + ] + }, + { + "cell_type": "code", + "execution_count": 20, + "id": "3ae517ea", + "metadata": { + "collapsed": false, + "editable": true + }, + "outputs": [], + "source": [ + "\n", + "epochs = 100\n", + "batch_size = 100\n", + "n_neurons_layer1 = 100\n", + "n_neurons_layer2 = 50\n", + "n_categories = 10\n", + "eta_vals = np.logspace(-5, 1, 7)\n", + "lmbd_vals = np.logspace(-5, 1, 7)\n", + "def create_neural_network_keras(n_neurons_layer1, n_neurons_layer2, n_categories, eta, lmbd):\n", + " model = Sequential()\n", + " model.add(Dense(n_neurons_layer1, activation='sigmoid', kernel_regularizer=regularizers.l2(lmbd)))\n", + " model.add(Dense(n_neurons_layer2, activation='sigmoid', kernel_regularizer=regularizers.l2(lmbd)))\n", + " model.add(Dense(n_categories, activation='softmax'))\n", + " \n", + " sgd = optimizers.SGD(lr=eta)\n", + " model.compile(loss='categorical_crossentropy', optimizer=sgd, metrics=['accuracy'])\n", + " \n", + " return model" + ] + }, + { + "cell_type": "code", + "execution_count": 21, + "id": "c880b209", + "metadata": { + "collapsed": false, + "editable": true + }, + "outputs": [], + "source": [ + "DNN_keras = np.zeros((len(eta_vals), len(lmbd_vals)), dtype=object)\n", + " \n", + "for i, eta in enumerate(eta_vals):\n", + " for j, lmbd in enumerate(lmbd_vals):\n", + " DNN = create_neural_network_keras(n_neurons_layer1, n_neurons_layer2, n_categories,\n", + " eta=eta, lmbd=lmbd)\n", + " DNN.fit(X_train, Y_train, epochs=epochs, batch_size=batch_size, verbose=0)\n", + " scores = DNN.evaluate(X_test, Y_test)\n", + " \n", + " DNN_keras[i][j] = DNN\n", + " \n", + " print(\"Learning rate = \", eta)\n", + " print(\"Lambda = \", lmbd)\n", + " print(\"Test accuracy: %.3f\" % scores[1])\n", + " print()" + ] + }, + { + "cell_type": "code", + "execution_count": 22, + "id": "cf8cea73", + "metadata": { + "collapsed": false, + "editable": true + }, + "outputs": [], + "source": [ + "# optional\n", + "# visual representation of grid search\n", + "# uses seaborn heatmap, could probably do this in matplotlib\n", + "import seaborn as sns\n", + "\n", + "sns.set()\n", + "\n", + "train_accuracy = np.zeros((len(eta_vals), len(lmbd_vals)))\n", + "test_accuracy = np.zeros((len(eta_vals), len(lmbd_vals)))\n", + "\n", + "for i in range(len(eta_vals)):\n", + " for j in range(len(lmbd_vals)):\n", + " DNN = DNN_keras[i][j]\n", + "\n", + " train_accuracy[i][j] = DNN.evaluate(X_train, Y_train)[1]\n", + " test_accuracy[i][j] = DNN.evaluate(X_test, Y_test)[1]\n", + "\n", + " \n", + "fig, ax = plt.subplots(figsize = (10, 10))\n", + "sns.heatmap(train_accuracy, annot=True, ax=ax, cmap=\"viridis\")\n", + "ax.set_title(\"Training Accuracy\")\n", + "ax.set_ylabel(\"$\\eta$\")\n", + "ax.set_xlabel(\"$\\lambda$\")\n", + "plt.show()\n", + "\n", + "fig, ax = plt.subplots(figsize = (10, 10))\n", + "sns.heatmap(test_accuracy, annot=True, ax=ax, cmap=\"viridis\")\n", + "ax.set_title(\"Test Accuracy\")\n", + "ax.set_ylabel(\"$\\eta$\")\n", + "ax.set_xlabel(\"$\\lambda$\")\n", + "plt.show()" + ] + }, + { + "cell_type": "markdown", + "id": "3024761c", + "metadata": { + "editable": true + }, + "source": [ + "## Building a neural network code\n", + "\n", + "Here we present a flexible object oriented codebase\n", + "for a feed forward neural network, along with a demonstration of how\n", + "to use it. Before we get into the details of the neural network, we\n", + "will first present some implementations of various schedulers, cost\n", + "functions and activation functions that can be used together with the\n", + "neural network.\n", + "\n", + "The codes here were developed by Eric Reber and Gregor Kajda during spring 2023." + ] + }, + { + "cell_type": "markdown", + "id": "53987675", + "metadata": { + "editable": true + }, + "source": [ + "### Learning rate methods\n", + "\n", + "The code below shows object oriented implementations of the Constant,\n", + "Momentum, Adagrad, AdagradMomentum, RMS prop and Adam schedulers. All\n", + "of the classes belong to the shared abstract Scheduler class, and\n", + "share the update_change() and reset() methods allowing for any of the\n", + "schedulers to be seamlessly used during the training stage, as will\n", + "later be shown in the fit() method of the neural\n", + "network. Update_change() only has one parameter, the gradient\n", + "($δ^l_ja^{l−1}_k$), and returns the change which will be subtracted\n", + "from the weights. The reset() function takes no parameters, and resets\n", + "the desired variables. For Constant and Momentum, reset does nothing." + ] + }, + { + "cell_type": "code", + "execution_count": 23, + "id": "5ed03631", + "metadata": { + "collapsed": false, + "editable": true + }, + "outputs": [], + "source": [ + "import autograd.numpy as np\n", + "\n", + "class Scheduler:\n", + " \"\"\"\n", + " Abstract class for Schedulers\n", + " \"\"\"\n", + "\n", + " def __init__(self, eta):\n", + " self.eta = eta\n", + "\n", + " # should be overwritten\n", + " def update_change(self, gradient):\n", + " raise NotImplementedError\n", + "\n", + " # overwritten if needed\n", + " def reset(self):\n", + " pass\n", + "\n", + "\n", + "class Constant(Scheduler):\n", + " def __init__(self, eta):\n", + " super().__init__(eta)\n", + "\n", + " def update_change(self, gradient):\n", + " return self.eta * gradient\n", + " \n", + " def reset(self):\n", + " pass\n", + "\n", + "\n", + "class Momentum(Scheduler):\n", + " def __init__(self, eta: float, momentum: float):\n", + " super().__init__(eta)\n", + " self.momentum = momentum\n", + " self.change = 0\n", + "\n", + " def update_change(self, gradient):\n", + " self.change = self.momentum * self.change + self.eta * gradient\n", + " return self.change\n", + "\n", + " def reset(self):\n", + " pass\n", + "\n", + "\n", + "class Adagrad(Scheduler):\n", + " def __init__(self, eta):\n", + " super().__init__(eta)\n", + " self.G_t = None\n", + "\n", + " def update_change(self, gradient):\n", + " delta = 1e-8 # avoid division ny zero\n", + "\n", + " if self.G_t is None:\n", + " self.G_t = np.zeros((gradient.shape[0], gradient.shape[0]))\n", + "\n", + " self.G_t += gradient @ gradient.T\n", + "\n", + " G_t_inverse = 1 / (\n", + " delta + np.sqrt(np.reshape(np.diagonal(self.G_t), (self.G_t.shape[0], 1)))\n", + " )\n", + " return self.eta * gradient * G_t_inverse\n", + "\n", + " def reset(self):\n", + " self.G_t = None\n", + "\n", + "\n", + "class AdagradMomentum(Scheduler):\n", + " def __init__(self, eta, momentum):\n", + " super().__init__(eta)\n", + " self.G_t = None\n", + " self.momentum = momentum\n", + " self.change = 0\n", + "\n", + " def update_change(self, gradient):\n", + " delta = 1e-8 # avoid division ny zero\n", + "\n", + " if self.G_t is None:\n", + " self.G_t = np.zeros((gradient.shape[0], gradient.shape[0]))\n", + "\n", + " self.G_t += gradient @ gradient.T\n", + "\n", + " G_t_inverse = 1 / (\n", + " delta + np.sqrt(np.reshape(np.diagonal(self.G_t), (self.G_t.shape[0], 1)))\n", + " )\n", + " self.change = self.change * self.momentum + self.eta * gradient * G_t_inverse\n", + " return self.change\n", + "\n", + " def reset(self):\n", + " self.G_t = None\n", + "\n", + "\n", + "class RMS_prop(Scheduler):\n", + " def __init__(self, eta, rho):\n", + " super().__init__(eta)\n", + " self.rho = rho\n", + " self.second = 0.0\n", + "\n", + " def update_change(self, gradient):\n", + " delta = 1e-8 # avoid division ny zero\n", + " self.second = self.rho * self.second + (1 - self.rho) * gradient * gradient\n", + " return self.eta * gradient / (np.sqrt(self.second + delta))\n", + "\n", + " def reset(self):\n", + " self.second = 0.0\n", + "\n", + "\n", + "class Adam(Scheduler):\n", + " def __init__(self, eta, rho, rho2):\n", + " super().__init__(eta)\n", + " self.rho = rho\n", + " self.rho2 = rho2\n", + " self.moment = 0\n", + " self.second = 0\n", + " self.n_epochs = 1\n", + "\n", + " def update_change(self, gradient):\n", + " delta = 1e-8 # avoid division ny zero\n", + "\n", + " self.moment = self.rho * self.moment + (1 - self.rho) * gradient\n", + " self.second = self.rho2 * self.second + (1 - self.rho2) * gradient * gradient\n", + "\n", + " moment_corrected = self.moment / (1 - self.rho**self.n_epochs)\n", + " second_corrected = self.second / (1 - self.rho2**self.n_epochs)\n", + "\n", + " return self.eta * moment_corrected / (np.sqrt(second_corrected + delta))\n", + "\n", + " def reset(self):\n", + " self.n_epochs += 1\n", + " self.moment = 0\n", + " self.second = 0" + ] + }, + { + "cell_type": "markdown", + "id": "8b98c385", + "metadata": { + "editable": true + }, + "source": [ + "### Usage of the above learning rate schedulers\n", + "\n", + "To initalize a scheduler, simply create the object and pass in the\n", + "necessary parameters such as the learning rate and the momentum as\n", + "shown below. As the Scheduler class is an abstract class it should not\n", + "called directly, and will raise an error upon usage." + ] + }, + { + "cell_type": "code", + "execution_count": 24, + "id": "54298abd", + "metadata": { + "collapsed": false, + "editable": true + }, + "outputs": [], + "source": [ + "momentum_scheduler = Momentum(eta=1e-3, momentum=0.9)\n", + "adam_scheduler = Adam(eta=1e-3, rho=0.9, rho2=0.999)" + ] + }, + { + "cell_type": "markdown", + "id": "c03dc1b1", + "metadata": { + "editable": true + }, + "source": [ + "Here is a small example for how a segment of code using schedulers\n", + "could look. Switching out the schedulers is simple." + ] + }, + { + "cell_type": "code", + "execution_count": 25, + "id": "5a2f7871", + "metadata": { + "collapsed": false, + "editable": true + }, + "outputs": [], + "source": [ + "weights = np.ones((3,3))\n", + "print(f\"Before scheduler:\\n{weights=}\")\n", + "\n", + "epochs = 10\n", + "for e in range(epochs):\n", + " gradient = np.random.rand(3, 3)\n", + " change = adam_scheduler.update_change(gradient)\n", + " weights = weights - change\n", + " adam_scheduler.reset()\n", + "\n", + "print(f\"\\nAfter scheduler:\\n{weights=}\")" + ] + }, + { + "cell_type": "markdown", + "id": "07fa559a", + "metadata": { + "editable": true + }, + "source": [ + "### Cost functions\n", + "\n", + "Here we discuss cost functions that can be used when creating the\n", + "neural network. Every cost function takes the target vector as its\n", + "parameter, and returns a function valued only at $x$ such that it may\n", + "easily be differentiated." + ] + }, + { + "cell_type": "code", + "execution_count": 26, + "id": "718450a2", + "metadata": { + "collapsed": false, + "editable": true + }, + "outputs": [], + "source": [ + "import autograd.numpy as np\n", + "\n", + "def CostOLS(target):\n", + " \n", + " def func(X):\n", + " return (1.0 / target.shape[0]) * np.sum((target - X) ** 2)\n", + "\n", + " return func\n", + "\n", + "\n", + "def CostLogReg(target):\n", + "\n", + " def func(X):\n", + " \n", + " return -(1.0 / target.shape[0]) * np.sum(\n", + " (target * np.log(X + 10e-10)) + ((1 - target) * np.log(1 - X + 10e-10))\n", + " )\n", + "\n", + " return func\n", + "\n", + "\n", + "def CostCrossEntropy(target):\n", + " \n", + " def func(X):\n", + " return -(1.0 / target.size) * np.sum(target * np.log(X + 10e-10))\n", + "\n", + " return func" + ] + }, + { + "cell_type": "markdown", + "id": "0db28b52", + "metadata": { + "editable": true + }, + "source": [ + "Below we give a short example of how these cost function may be used\n", + "to obtain results if you wish to test them out on your own using\n", + "AutoGrad's automatics differentiation." + ] + }, + { + "cell_type": "code", + "execution_count": 27, + "id": "1479de5a", + "metadata": { + "collapsed": false, + "editable": true + }, + "outputs": [], + "source": [ + "from autograd import grad\n", + "\n", + "target = np.array([[1, 2, 3]]).T\n", + "a = np.array([[4, 5, 6]]).T\n", + "\n", + "cost_func = CostCrossEntropy\n", + "cost_func_derivative = grad(cost_func(target))\n", + "\n", + "valued_at_a = cost_func_derivative(a)\n", + "print(f\"Derivative of cost function {cost_func.__name__} valued at a:\\n{valued_at_a}\")" + ] + }, + { + "cell_type": "markdown", + "id": "d42e1ccf", + "metadata": { + "editable": true + }, + "source": [ + "### Activation functions\n", + "\n", + "Finally, before we look at the neural network, we will look at the\n", + "activation functions which can be specified between the hidden layers\n", + "and as the output function. Each function can be valued for any given\n", + "vector or matrix X, and can be differentiated via derivate()." + ] + }, + { + "cell_type": "code", + "execution_count": 28, + "id": "2c2a8d32", + "metadata": { + "collapsed": false, + "editable": true + }, + "outputs": [], + "source": [ + "import autograd.numpy as np\n", + "from autograd import elementwise_grad\n", + "\n", + "def identity(X):\n", + " return X\n", + "\n", + "\n", + "def sigmoid(X):\n", + " try:\n", + " return 1.0 / (1 + np.exp(-X))\n", + " except FloatingPointError:\n", + " return np.where(X > np.zeros(X.shape), np.ones(X.shape), np.zeros(X.shape))\n", + "\n", + "\n", + "def softmax(X):\n", + " X = X - np.max(X, axis=-1, keepdims=True)\n", + " delta = 10e-10\n", + " return np.exp(X) / (np.sum(np.exp(X), axis=-1, keepdims=True) + delta)\n", + "\n", + "\n", + "def RELU(X):\n", + " return np.where(X > np.zeros(X.shape), X, np.zeros(X.shape))\n", + "\n", + "\n", + "def LRELU(X):\n", + " delta = 10e-4\n", + " return np.where(X > np.zeros(X.shape), X, delta * X)\n", + "\n", + "\n", + "def derivate(func):\n", + " if func.__name__ == \"RELU\":\n", + "\n", + " def func(X):\n", + " return np.where(X > 0, 1, 0)\n", + "\n", + " return func\n", + "\n", + " elif func.__name__ == \"LRELU\":\n", + "\n", + " def func(X):\n", + " delta = 10e-4\n", + " return np.where(X > 0, 1, delta)\n", + "\n", + " return func\n", + "\n", + " else:\n", + " return elementwise_grad(func)" + ] + }, + { + "cell_type": "markdown", + "id": "65e7caab", + "metadata": { + "editable": true + }, + "source": [ + "Below follows a short demonstration of how to use an activation\n", + "function. The derivative of the activation function will be important\n", + "when calculating the output delta term during backpropagation. Note\n", + "that derivate() can also be used for cost functions for a more\n", + "generalized approach." + ] + }, + { + "cell_type": "code", + "execution_count": 29, + "id": "75e9399d", + "metadata": { + "collapsed": false, + "editable": true + }, + "outputs": [], + "source": [ + "z = np.array([[4, 5, 6]]).T\n", + "print(f\"Input to activation function:\\n{z}\")\n", + "\n", + "act_func = sigmoid\n", + "a = act_func(z)\n", + "print(f\"\\nOutput from {act_func.__name__} activation function:\\n{a}\")\n", + "\n", + "act_func_derivative = derivate(act_func)\n", + "valued_at_z = act_func_derivative(a)\n", + "print(f\"\\nDerivative of {act_func.__name__} activation function valued at z:\\n{valued_at_z}\")" + ] + }, + { + "cell_type": "markdown", + "id": "0b0b82e8", + "metadata": { + "editable": true + }, + "source": [ + "### The Neural Network\n", + "\n", + "Now that we have gotten a good understanding of the implementation of\n", + "some important components, we can take a look at an object oriented\n", + "implementation of a feed forward neural network. The feed forward\n", + "neural network has been implemented as a class named FFNN, which can\n", + "be initiated as a regressor or classifier dependant on the choice of\n", + "cost function. The FFNN can have any number of input nodes, hidden\n", + "layers with any amount of hidden nodes, and any amount of output nodes\n", + "meaning it can perform multiclass classification as well as binary\n", + "classification and regression problems. Although there is a lot of\n", + "code present, it makes for an easy to use and generalizeable interface\n", + "for creating many types of neural networks as will be demonstrated\n", + "below." + ] + }, + { + "cell_type": "code", + "execution_count": 30, + "id": "5351bfd6", + "metadata": { + "collapsed": false, + "editable": true + }, + "outputs": [], + "source": [ + "import math\n", + "import autograd.numpy as np\n", + "import sys\n", + "import warnings\n", + "from autograd import grad, elementwise_grad\n", + "from random import random, seed\n", + "from copy import deepcopy, copy\n", + "from typing import Tuple, Callable\n", + "from sklearn.utils import resample\n", + "\n", + "warnings.simplefilter(\"error\")\n", + "\n", + "\n", + "class FFNN:\n", + " \"\"\"\n", + " Description:\n", + " ------------\n", + " Feed Forward Neural Network with interface enabling flexible design of a\n", + " nerual networks architecture and the specification of activation function\n", + " in the hidden layers and output layer respectively. This model can be used\n", + " for both regression and classification problems, depending on the output function.\n", + "\n", + " Attributes:\n", + " ------------\n", + " I dimensions (tuple[int]): A list of positive integers, which specifies the\n", + " number of nodes in each of the networks layers. The first integer in the array\n", + " defines the number of nodes in the input layer, the second integer defines number\n", + " of nodes in the first hidden layer and so on until the last number, which\n", + " specifies the number of nodes in the output layer.\n", + " II hidden_func (Callable): The activation function for the hidden layers\n", + " III output_func (Callable): The activation function for the output layer\n", + " IV cost_func (Callable): Our cost function\n", + " V seed (int): Sets random seed, makes results reproducible\n", + " \"\"\"\n", + "\n", + " def __init__(\n", + " self,\n", + " dimensions: tuple[int],\n", + " hidden_func: Callable = sigmoid,\n", + " output_func: Callable = lambda x: x,\n", + " cost_func: Callable = CostOLS,\n", + " seed: int = None,\n", + " ):\n", + " self.dimensions = dimensions\n", + " self.hidden_func = hidden_func\n", + " self.output_func = output_func\n", + " self.cost_func = cost_func\n", + " self.seed = seed\n", + " self.weights = list()\n", + " self.schedulers_weight = list()\n", + " self.schedulers_bias = list()\n", + " self.a_matrices = list()\n", + " self.z_matrices = list()\n", + " self.classification = None\n", + "\n", + " self.reset_weights()\n", + " self._set_classification()\n", + "\n", + " def fit(\n", + " self,\n", + " X: np.ndarray,\n", + " t: np.ndarray,\n", + " scheduler: Scheduler,\n", + " batches: int = 1,\n", + " epochs: int = 100,\n", + " lam: float = 0,\n", + " X_val: np.ndarray = None,\n", + " t_val: np.ndarray = None,\n", + " ):\n", + " \"\"\"\n", + " Description:\n", + " ------------\n", + " This function performs the training the neural network by performing the feedforward and backpropagation\n", + " algorithm to update the networks weights.\n", + "\n", + " Parameters:\n", + " ------------\n", + " I X (np.ndarray) : training data\n", + " II t (np.ndarray) : target data\n", + " III scheduler (Scheduler) : specified scheduler (algorithm for optimization of gradient descent)\n", + " IV scheduler_args (list[int]) : list of all arguments necessary for scheduler\n", + "\n", + " Optional Parameters:\n", + " ------------\n", + " V batches (int) : number of batches the datasets are split into, default equal to 1\n", + " VI epochs (int) : number of iterations used to train the network, default equal to 100\n", + " VII lam (float) : regularization hyperparameter lambda\n", + " VIII X_val (np.ndarray) : validation set\n", + " IX t_val (np.ndarray) : validation target set\n", + "\n", + " Returns:\n", + " ------------\n", + " I scores (dict) : A dictionary containing the performance metrics of the model.\n", + " The number of the metrics depends on the parameters passed to the fit-function.\n", + "\n", + " \"\"\"\n", + "\n", + " # setup \n", + " if self.seed is not None:\n", + " np.random.seed(self.seed)\n", + "\n", + " val_set = False\n", + " if X_val is not None and t_val is not None:\n", + " val_set = True\n", + "\n", + " # creating arrays for score metrics\n", + " train_errors = np.empty(epochs)\n", + " train_errors.fill(np.nan)\n", + " val_errors = np.empty(epochs)\n", + " val_errors.fill(np.nan)\n", + "\n", + " train_accs = np.empty(epochs)\n", + " train_accs.fill(np.nan)\n", + " val_accs = np.empty(epochs)\n", + " val_accs.fill(np.nan)\n", + "\n", + " self.schedulers_weight = list()\n", + " self.schedulers_bias = list()\n", + "\n", + " batch_size = X.shape[0] // batches\n", + "\n", + " X, t = resample(X, t)\n", + "\n", + " # this function returns a function valued only at X\n", + " cost_function_train = self.cost_func(t)\n", + " if val_set:\n", + " cost_function_val = self.cost_func(t_val)\n", + "\n", + " # create schedulers for each weight matrix\n", + " for i in range(len(self.weights)):\n", + " self.schedulers_weight.append(copy(scheduler))\n", + " self.schedulers_bias.append(copy(scheduler))\n", + "\n", + " print(f\"{scheduler.__class__.__name__}: Eta={scheduler.eta}, Lambda={lam}\")\n", + "\n", + " try:\n", + " for e in range(epochs):\n", + " for i in range(batches):\n", + " # allows for minibatch gradient descent\n", + " if i == batches - 1:\n", + " # If the for loop has reached the last batch, take all thats left\n", + " X_batch = X[i * batch_size :, :]\n", + " t_batch = t[i * batch_size :, :]\n", + " else:\n", + " X_batch = X[i * batch_size : (i + 1) * batch_size, :]\n", + " t_batch = t[i * batch_size : (i + 1) * batch_size, :]\n", + "\n", + " self._feedforward(X_batch)\n", + " self._backpropagate(X_batch, t_batch, lam)\n", + "\n", + " # reset schedulers for each epoch (some schedulers pass in this call)\n", + " for scheduler in self.schedulers_weight:\n", + " scheduler.reset()\n", + "\n", + " for scheduler in self.schedulers_bias:\n", + " scheduler.reset()\n", + "\n", + " # computing performance metrics\n", + " pred_train = self.predict(X)\n", + " train_error = cost_function_train(pred_train)\n", + "\n", + " train_errors[e] = train_error\n", + " if val_set:\n", + " \n", + " pred_val = self.predict(X_val)\n", + " val_error = cost_function_val(pred_val)\n", + " val_errors[e] = val_error\n", + "\n", + " if self.classification:\n", + " train_acc = self._accuracy(self.predict(X), t)\n", + " train_accs[e] = train_acc\n", + " if val_set:\n", + " val_acc = self._accuracy(pred_val, t_val)\n", + " val_accs[e] = val_acc\n", + "\n", + " # printing progress bar\n", + " progression = e / epochs\n", + " print_length = self._progress_bar(\n", + " progression,\n", + " train_error=train_errors[e],\n", + " train_acc=train_accs[e],\n", + " val_error=val_errors[e],\n", + " val_acc=val_accs[e],\n", + " )\n", + " except KeyboardInterrupt:\n", + " # allows for stopping training at any point and seeing the result\n", + " pass\n", + "\n", + " # visualization of training progression (similiar to tensorflow progression bar)\n", + " sys.stdout.write(\"\\r\" + \" \" * print_length)\n", + " sys.stdout.flush()\n", + " self._progress_bar(\n", + " 1,\n", + " train_error=train_errors[e],\n", + " train_acc=train_accs[e],\n", + " val_error=val_errors[e],\n", + " val_acc=val_accs[e],\n", + " )\n", + " sys.stdout.write(\"\")\n", + "\n", + " # return performance metrics for the entire run\n", + " scores = dict()\n", + "\n", + " scores[\"train_errors\"] = train_errors\n", + "\n", + " if val_set:\n", + " scores[\"val_errors\"] = val_errors\n", + "\n", + " if self.classification:\n", + " scores[\"train_accs\"] = train_accs\n", + "\n", + " if val_set:\n", + " scores[\"val_accs\"] = val_accs\n", + "\n", + " return scores\n", + "\n", + " def predict(self, X: np.ndarray, *, threshold=0.5):\n", + " \"\"\"\n", + " Description:\n", + " ------------\n", + " Performs prediction after training of the network has been finished.\n", + "\n", + " Parameters:\n", + " ------------\n", + " I X (np.ndarray): The design matrix, with n rows of p features each\n", + "\n", + " Optional Parameters:\n", + " ------------\n", + " II threshold (float) : sets minimal value for a prediction to be predicted as the positive class\n", + " in classification problems\n", + "\n", + " Returns:\n", + " ------------\n", + " I z (np.ndarray): A prediction vector (row) for each row in our design matrix\n", + " This vector is thresholded if regression=False, meaning that classification results\n", + " in a vector of 1s and 0s, while regressions in an array of decimal numbers\n", + "\n", + " \"\"\"\n", + "\n", + " predict = self._feedforward(X)\n", + "\n", + " if self.classification:\n", + " return np.where(predict > threshold, 1, 0)\n", + " else:\n", + " return predict\n", + "\n", + " def reset_weights(self):\n", + " \"\"\"\n", + " Description:\n", + " ------------\n", + " Resets/Reinitializes the weights in order to train the network for a new problem.\n", + "\n", + " \"\"\"\n", + " if self.seed is not None:\n", + " np.random.seed(self.seed)\n", + "\n", + " self.weights = list()\n", + " for i in range(len(self.dimensions) - 1):\n", + " weight_array = np.random.randn(\n", + " self.dimensions[i] + 1, self.dimensions[i + 1]\n", + " )\n", + " weight_array[0, :] = np.random.randn(self.dimensions[i + 1]) * 0.01\n", + "\n", + " self.weights.append(weight_array)\n", + "\n", + " def _feedforward(self, X: np.ndarray):\n", + " \"\"\"\n", + " Description:\n", + " ------------\n", + " Calculates the activation of each layer starting at the input and ending at the output.\n", + " Each following activation is calculated from a weighted sum of each of the preceeding\n", + " activations (except in the case of the input layer).\n", + "\n", + " Parameters:\n", + " ------------\n", + " I X (np.ndarray): The design matrix, with n rows of p features each\n", + "\n", + " Returns:\n", + " ------------\n", + " I z (np.ndarray): A prediction vector (row) for each row in our design matrix\n", + " \"\"\"\n", + "\n", + " # reset matrices\n", + " self.a_matrices = list()\n", + " self.z_matrices = list()\n", + "\n", + " # if X is just a vector, make it into a matrix\n", + " if len(X.shape) == 1:\n", + " X = X.reshape((1, X.shape[0]))\n", + "\n", + " # Add a coloumn of zeros as the first coloumn of the design matrix, in order\n", + " # to add bias to our data\n", + " bias = np.ones((X.shape[0], 1)) * 0.01\n", + " X = np.hstack([bias, X])\n", + "\n", + " # a^0, the nodes in the input layer (one a^0 for each row in X - where the\n", + " # exponent indicates layer number).\n", + " a = X\n", + " self.a_matrices.append(a)\n", + " self.z_matrices.append(a)\n", + "\n", + " # The feed forward algorithm\n", + " for i in range(len(self.weights)):\n", + " if i < len(self.weights) - 1:\n", + " z = a @ self.weights[i]\n", + " self.z_matrices.append(z)\n", + " a = self.hidden_func(z)\n", + " # bias column again added to the data here\n", + " bias = np.ones((a.shape[0], 1)) * 0.01\n", + " a = np.hstack([bias, a])\n", + " self.a_matrices.append(a)\n", + " else:\n", + " try:\n", + " # a^L, the nodes in our output layers\n", + " z = a @ self.weights[i]\n", + " a = self.output_func(z)\n", + " self.a_matrices.append(a)\n", + " self.z_matrices.append(z)\n", + " except Exception as OverflowError:\n", + " print(\n", + " \"OverflowError in fit() in FFNN\\nHOW TO DEBUG ERROR: Consider lowering your learning rate or scheduler specific parameters such as momentum, or check if your input values need scaling\"\n", + " )\n", + "\n", + " # this will be a^L\n", + " return a\n", + "\n", + " def _backpropagate(self, X, t, lam):\n", + " \"\"\"\n", + " Description:\n", + " ------------\n", + " Performs the backpropagation algorithm. In other words, this method\n", + " calculates the gradient of all the layers starting at the\n", + " output layer, and moving from right to left accumulates the gradient until\n", + " the input layer is reached. Each layers respective weights are updated while\n", + " the algorithm propagates backwards from the output layer (auto-differentation in reverse mode).\n", + "\n", + " Parameters:\n", + " ------------\n", + " I X (np.ndarray): The design matrix, with n rows of p features each.\n", + " II t (np.ndarray): The target vector, with n rows of p targets.\n", + " III lam (float32): regularization parameter used to punish the weights in case of overfitting\n", + "\n", + " Returns:\n", + " ------------\n", + " No return value.\n", + "\n", + " \"\"\"\n", + " out_derivative = derivate(self.output_func)\n", + " hidden_derivative = derivate(self.hidden_func)\n", + "\n", + " for i in range(len(self.weights) - 1, -1, -1):\n", + " # delta terms for output\n", + " if i == len(self.weights) - 1:\n", + " # for multi-class classification\n", + " if (\n", + " self.output_func.__name__ == \"softmax\"\n", + " ):\n", + " delta_matrix = self.a_matrices[i + 1] - t\n", + " # for single class classification\n", + " else:\n", + " cost_func_derivative = grad(self.cost_func(t))\n", + " delta_matrix = out_derivative(\n", + " self.z_matrices[i + 1]\n", + " ) * cost_func_derivative(self.a_matrices[i + 1])\n", + "\n", + " # delta terms for hidden layer\n", + " else:\n", + " delta_matrix = (\n", + " self.weights[i + 1][1:, :] @ delta_matrix.T\n", + " ).T * hidden_derivative(self.z_matrices[i + 1])\n", + "\n", + " # calculate gradient\n", + " gradient_weights = self.a_matrices[i][:, 1:].T @ delta_matrix\n", + " gradient_bias = np.sum(delta_matrix, axis=0).reshape(\n", + " 1, delta_matrix.shape[1]\n", + " )\n", + "\n", + " # regularization term\n", + " gradient_weights += self.weights[i][1:, :] * lam\n", + "\n", + " # use scheduler\n", + " update_matrix = np.vstack(\n", + " [\n", + " self.schedulers_bias[i].update_change(gradient_bias),\n", + " self.schedulers_weight[i].update_change(gradient_weights),\n", + " ]\n", + " )\n", + "\n", + " # update weights and bias\n", + " self.weights[i] -= update_matrix\n", + "\n", + " def _accuracy(self, prediction: np.ndarray, target: np.ndarray):\n", + " \"\"\"\n", + " Description:\n", + " ------------\n", + " Calculates accuracy of given prediction to target\n", + "\n", + " Parameters:\n", + " ------------\n", + " I prediction (np.ndarray): vector of predicitons output network\n", + " (1s and 0s in case of classification, and real numbers in case of regression)\n", + " II target (np.ndarray): vector of true values (What the network ideally should predict)\n", + "\n", + " Returns:\n", + " ------------\n", + " A floating point number representing the percentage of correctly classified instances.\n", + " \"\"\"\n", + " assert prediction.size == target.size\n", + " return np.average((target == prediction))\n", + " def _set_classification(self):\n", + " \"\"\"\n", + " Description:\n", + " ------------\n", + " Decides if FFNN acts as classifier (True) og regressor (False),\n", + " sets self.classification during init()\n", + " \"\"\"\n", + " self.classification = False\n", + " if (\n", + " self.cost_func.__name__ == \"CostLogReg\"\n", + " or self.cost_func.__name__ == \"CostCrossEntropy\"\n", + " ):\n", + " self.classification = True\n", + "\n", + " def _progress_bar(self, progression, **kwargs):\n", + " \"\"\"\n", + " Description:\n", + " ------------\n", + " Displays progress of training\n", + " \"\"\"\n", + " print_length = 40\n", + " num_equals = int(progression * print_length)\n", + " num_not = print_length - num_equals\n", + " arrow = \">\" if num_equals > 0 else \"\"\n", + " bar = \"[\" + \"=\" * (num_equals - 1) + arrow + \"-\" * num_not + \"]\"\n", + " perc_print = self._format(progression * 100, decimals=5)\n", + " line = f\" {bar} {perc_print}% \"\n", + "\n", + " for key in kwargs:\n", + " if not np.isnan(kwargs[key]):\n", + " value = self._format(kwargs[key], decimals=4)\n", + " line += f\"| {key}: {value} \"\n", + " sys.stdout.write(\"\\r\" + line)\n", + " sys.stdout.flush()\n", + " return len(line)\n", + "\n", + " def _format(self, value, decimals=4):\n", + " \"\"\"\n", + " Description:\n", + " ------------\n", + " Formats decimal numbers for progress bar\n", + " \"\"\"\n", + " if value > 0:\n", + " v = value\n", + " elif value < 0:\n", + " v = -10 * value\n", + " else:\n", + " v = 1\n", + " n = 1 + math.floor(math.log10(v))\n", + " if n >= decimals - 1:\n", + " return str(round(value))\n", + " return f\"{value:.{decimals-n-1}f}\"" + ] + }, + { + "cell_type": "markdown", + "id": "2d63ec5b", + "metadata": { + "editable": true + }, + "source": [ + "Before we make a model, we will quickly generate a dataset we can use\n", + "for our linear regression problem as shown below" + ] + }, + { + "cell_type": "code", + "execution_count": 31, + "id": "85a6b185", + "metadata": { + "collapsed": false, + "editable": true + }, + "outputs": [], + "source": [ + "import autograd.numpy as np\n", + "from sklearn.model_selection import train_test_split\n", + "\n", + "def SkrankeFunction(x, y):\n", + " return np.ravel(0 + 1*x + 2*y + 3*x**2 + 4*x*y + 5*y**2)\n", + "\n", + "def create_X(x, y, n):\n", + " if len(x.shape) > 1:\n", + " x = np.ravel(x)\n", + " y = np.ravel(y)\n", + "\n", + " N = len(x)\n", + " l = int((n + 1) * (n + 2) / 2) # Number of elements in beta\n", + " X = np.ones((N, l))\n", + "\n", + " for i in range(1, n + 1):\n", + " q = int((i) * (i + 1) / 2)\n", + " for k in range(i + 1):\n", + " X[:, q + k] = (x ** (i - k)) * (y**k)\n", + "\n", + " return X\n", + "\n", + "step=0.5\n", + "x = np.arange(0, 1, step)\n", + "y = np.arange(0, 1, step)\n", + "x, y = np.meshgrid(x, y)\n", + "target = SkrankeFunction(x, y)\n", + "target = target.reshape(target.shape[0], 1)\n", + "\n", + "poly_degree=3\n", + "X = create_X(x, y, poly_degree)\n", + "\n", + "X_train, X_test, t_train, t_test = train_test_split(X, target)" + ] + }, + { + "cell_type": "markdown", + "id": "f8842209", + "metadata": { + "editable": true + }, + "source": [ + "Now that we have our dataset ready for the regression, we can create\n", + "our regressor. Note that with the seed parameter, we can make sure our\n", + "results stay the same every time we run the neural network. For\n", + "inititialization, we simply specify the dimensions (we wish the amount\n", + "of input nodes to be equal to the datapoints, and the output to\n", + "predict one value)." + ] + }, + { + "cell_type": "code", + "execution_count": 32, + "id": "b7b14d8f", + "metadata": { + "collapsed": false, + "editable": true + }, + "outputs": [], + "source": [ + "input_nodes = X_train.shape[1]\n", + "output_nodes = 1\n", + "\n", + "linear_regression = FFNN((input_nodes, output_nodes), output_func=identity, cost_func=CostOLS, seed=2023)" + ] + }, + { + "cell_type": "markdown", + "id": "3e35cc67", + "metadata": { + "editable": true + }, + "source": [ + "We then fit our model with our training data using the scheduler of our choice." + ] + }, + { + "cell_type": "code", + "execution_count": 33, + "id": "c2b3e7a3", + "metadata": { + "collapsed": false, + "editable": true + }, + "outputs": [], + "source": [ + "linear_regression.reset_weights() # reset weights such that previous runs or reruns don't affect the weights\n", + "\n", + "scheduler = Constant(eta=1e-3)\n", + "scores = linear_regression.fit(X_train, t_train, scheduler)" + ] + }, + { + "cell_type": "markdown", + "id": "9d0eb8b2", + "metadata": { + "editable": true + }, + "source": [ + "Due to the progress bar we can see the MSE (train_error) throughout\n", + "the FFNN's training. Note that the fit() function has some optional\n", + "parameters with defualt arguments. For example, the regularization\n", + "hyperparameter can be left ignored if not needed, and equally the FFNN\n", + "will by default run for 100 epochs. These can easily be changed, such\n", + "as for example:" + ] + }, + { + "cell_type": "code", + "execution_count": 34, + "id": "19fd68fe", + "metadata": { + "collapsed": false, + "editable": true + }, + "outputs": [], + "source": [ + "linear_regression.reset_weights() # reset weights such that previous runs or reruns don't affect the weights\n", + "\n", + "scores = linear_regression.fit(X_train, t_train, scheduler, lam=1e-4, epochs=1000)" + ] + }, + { + "cell_type": "markdown", + "id": "ecbc4c88", + "metadata": { + "editable": true + }, + "source": [ + "We see that given more epochs to train on, the regressor reaches a lower MSE.\n", + "\n", + "Let us then switch to a binary classification. We use a binary\n", + "classification dataset, and follow a similar setup to the regression\n", + "case." + ] + }, + { + "cell_type": "code", + "execution_count": 35, + "id": "c62f0877", + "metadata": { + "collapsed": false, + "editable": true + }, + "outputs": [], + "source": [ + "from sklearn.datasets import load_breast_cancer\n", + "from sklearn.preprocessing import MinMaxScaler\n", + "\n", + "wisconsin = load_breast_cancer()\n", + "X = wisconsin.data\n", + "target = wisconsin.target\n", + "target = target.reshape(target.shape[0], 1)\n", + "\n", + "X_train, X_val, t_train, t_val = train_test_split(X, target)\n", + "\n", + "scaler = MinMaxScaler()\n", + "scaler.fit(X_train)\n", + "X_train = scaler.transform(X_train)\n", + "X_val = scaler.transform(X_val)" + ] + }, + { + "cell_type": "code", + "execution_count": 36, + "id": "5b955b7a", + "metadata": { + "collapsed": false, + "editable": true + }, + "outputs": [], + "source": [ + "input_nodes = X_train.shape[1]\n", + "output_nodes = 1\n", + "\n", + "logistic_regression = FFNN((input_nodes, output_nodes), output_func=sigmoid, cost_func=CostLogReg, seed=2023)" + ] + }, + { + "cell_type": "markdown", + "id": "e4c3e9cc", + "metadata": { + "editable": true + }, + "source": [ + "We will now make use of our validation data by passing it into our fit function as a keyword argument" + ] + }, + { + "cell_type": "code", + "execution_count": 37, + "id": "4e994594", + "metadata": { + "collapsed": false, + "editable": true + }, + "outputs": [], + "source": [ + "logistic_regression.reset_weights() # reset weights such that previous runs or reruns don't affect the weights\n", + "\n", + "scheduler = Adam(eta=1e-3, rho=0.9, rho2=0.999)\n", + "scores = logistic_regression.fit(X_train, t_train, scheduler, epochs=1000, X_val=X_val, t_val=t_val)" + ] + }, + { + "cell_type": "markdown", + "id": "690debd0", + "metadata": { + "editable": true + }, + "source": [ + "Finally, we will create a neural network with 2 hidden layers with activation functions." + ] + }, + { + "cell_type": "code", + "execution_count": 38, + "id": "ea04ae66", + "metadata": { + "collapsed": false, + "editable": true + }, + "outputs": [], + "source": [ + "input_nodes = X_train.shape[1]\n", + "hidden_nodes1 = 100\n", + "hidden_nodes2 = 30\n", + "output_nodes = 1\n", + "\n", + "dims = (input_nodes, hidden_nodes1, hidden_nodes2, output_nodes)\n", + "\n", + "neural_network = FFNN(dims, hidden_func=RELU, output_func=sigmoid, cost_func=CostLogReg, seed=2023)" + ] + }, + { + "cell_type": "code", + "execution_count": 39, + "id": "cd892db0", + "metadata": { + "collapsed": false, + "editable": true + }, + "outputs": [], + "source": [ + "neural_network.reset_weights() # reset weights such that previous runs or reruns don't affect the weights\n", + "\n", + "scheduler = Adam(eta=1e-4, rho=0.9, rho2=0.999)\n", + "scores = neural_network.fit(X_train, t_train, scheduler, epochs=1000, X_val=X_val, t_val=t_val)" + ] + }, + { + "cell_type": "markdown", + "id": "c85e6aa4", + "metadata": { + "editable": true + }, + "source": [ + "### Multiclass classification\n", + "\n", + "Finally, we will demonstrate the use case of multiclass classification\n", + "using our FFNN with the famous MNIST dataset, which contain images of\n", + "digits between the range of 0 to 9." + ] + }, + { + "cell_type": "code", + "execution_count": 40, + "id": "1dc44b4d", + "metadata": { + "collapsed": false, + "editable": true + }, + "outputs": [], + "source": [ + "from sklearn.datasets import load_digits\n", + "\n", + "def onehot(target: np.ndarray):\n", + " onehot = np.zeros((target.size, target.max() + 1))\n", + " onehot[np.arange(target.size), target] = 1\n", + " return onehot\n", + "\n", + "digits = load_digits()\n", + "\n", + "X = digits.data\n", + "target = digits.target\n", + "target = onehot(target)\n", + "\n", + "input_nodes = 64\n", + "hidden_nodes1 = 100\n", + "hidden_nodes2 = 30\n", + "output_nodes = 10\n", + "\n", + "dims = (input_nodes, hidden_nodes1, hidden_nodes2, output_nodes)\n", + "\n", + "multiclass = FFNN(dims, hidden_func=LRELU, output_func=softmax, cost_func=CostCrossEntropy)\n", + "\n", + "multiclass.reset_weights() # reset weights such that previous runs or reruns don't affect the weights\n", + "\n", + "scheduler = Adam(eta=1e-4, rho=0.9, rho2=0.999)\n", + "scores = multiclass.fit(X, target, scheduler, epochs=1000)" + ] + }, + { + "cell_type": "markdown", + "id": "785a6539", + "metadata": { + "editable": true + }, + "source": [ + "## Testing the XOR gate and other gates\n", + "\n", + "Let us now use our code to test the XOR gate." + ] + }, + { + "cell_type": "code", + "execution_count": 41, + "id": "f2137088", + "metadata": { + "collapsed": false, + "editable": true + }, + "outputs": [], + "source": [ + "X = np.array([ [0, 0], [0, 1], [1, 0],[1, 1]],dtype=np.float64)\n", + "\n", + "# The XOR gate\n", + "yXOR = np.array( [[ 0], [1] ,[1], [0]])\n", + "\n", + "input_nodes = X.shape[1]\n", + "output_nodes = 1\n", + "\n", + "logistic_regression = FFNN((input_nodes, output_nodes), output_func=sigmoid, cost_func=CostLogReg, seed=2023)\n", + "logistic_regression.reset_weights() # reset weights such that previous runs or reruns don't affect the weights\n", + "scheduler = Adam(eta=1e-1, rho=0.9, rho2=0.999)\n", + "scores = logistic_regression.fit(X, yXOR, scheduler, epochs=1000)" + ] + }, + { + "cell_type": "markdown", + "id": "6caa78c7", + "metadata": { + "editable": true + }, + "source": [ + "Not bad, but the results depend strongly on the learning reate. Try different learning rates." + ] + } + ], + "metadata": {}, + "nbformat": 4, + "nbformat_minor": 5 +}