FYS-STK4155/doc/pub/summary/ipynb/summary.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- dom:TITLE: Summary of course -->\n",
    "# Summary of course\n",
    "<!-- dom:AUTHOR: Morten Hjorth-Jensen   Email morten.hjorth-jensen@fys.uio.no at Department of Physics and Center of Mathematics for Applications, University of Oslo & National Superconducting Cyclotron Laboratory, Michigan State University -->\n",
    "<!-- Author: -->  \n",
    "**Morten Hjorth-Jensen   Email morten.hjorth-jensen@fys.uio.no**, Department of Physics and Center of Mathematics for Applications, University of Oslo and National Superconducting Cyclotron Laboratory, Michigan State University\n",
    "\n",
    "Date: **Nov 27, 2019**\n",
    "\n",
    "Copyright 1999-2019, Morten Hjorth-Jensen   Email morten.hjorth-jensen@fys.uio.no. Released under CC Attribution-NonCommercial 4.0 license\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "## What? Me worry? No final exam in this course!\n",
    "<!-- dom:FIGURE: [figures/exam1.jpeg, width=500 frac=0.6] -->\n",
    "<!-- begin figure -->\n",
    "\n",
    "<p></p>\n",
    "<img src=\"figures/exam1.jpeg\" width=500>\n",
    "\n",
    "<!-- end figure -->\n",
    "\n",
    "<!-- dom:FIGURE: [figures/whatmeworry.jpeg, width=500 frac=0.6] -->\n",
    "<!-- begin figure -->\n",
    "\n",
    "<p></p>\n",
    "<img src=\"figures/whatmeworry.jpeg\" width=500>\n",
    "\n",
    "<!-- end figure -->\n",
    "\n",
    "\n",
    "\n",
    "## What did I learn in school this year?\n",
    "\n",
    "[Our ideal about knowledge on computational science](http://hplgit.github.io/edu/py_vs_m/computing_competence.html)\n",
    "\n",
    "\n",
    "Does that match the experiences you have made this semester?\n",
    "<!-- dom:FIGURE: [figures/exam2.jpg, width=500 frac=0.7] -->\n",
    "<!-- begin figure -->\n",
    "\n",
    "<p></p>\n",
    "<img src=\"figures/exam2.jpg\" width=500>\n",
    "\n",
    "<!-- end figure -->\n",
    "\n",
    "\n",
    "## Topics we have covered this year\n",
    "\n",
    "The course has two central parts\n",
    "\n",
    "1. Statistical analysis and optimization of data\n",
    "\n",
    "2. Machine learning\n",
    "\n",
    "## Statistical analysis and optimization of data\n",
    "\n",
    "The following topics will be covered\n",
    "1. Basic concepts, expectation values, variance, covariance, correlation functions and errors;\n",
    "\n",
    "2. Simpler models, binomial distribution, the Poisson distribution, simple and multivariate normal distributions;\n",
    "\n",
    "3. Central elements of Bayesian statistics and modeling;\n",
    "\n",
    "4. Central elements from linear algebra\n",
    "\n",
    "5. Gradient methods for data optimization\n",
    "\n",
    "6. Estimation of errors using cross-validation, blocking, bootstrapping and jackknife methods;\n",
    "\n",
    "7. Practical optimization using Singular-value decomposition and least squares for parameterizing data.\n",
    "\n",
    "8. Principal Component Analysis.\n",
    "\n",
    "## Machine learning\n",
    "\n",
    "The following topics will be covered\n",
    "1. Linear methods for regression and classification;\n",
    "\n",
    "2. Neural networks;\n",
    "\n",
    "3. Decisions trees, random forests, boosting and bagging\n",
    "\n",
    "4. Support vector machines\n",
    "\n",
    "## Learning outcomes and overarching aims of this course\n",
    "\n",
    "The course introduces a variety of central algorithms and methods\n",
    "essential for studies of data analysis and machine learning. The\n",
    "course is project based and through the various projects, normally\n",
    "three, you will be exposed to fundamental research problems\n",
    "in these fields, with the aim to reproduce state of the art scientific\n",
    "results. The students will learn to develop and structure large codes\n",
    "for studying these systems, get acquainted with computing facilities\n",
    "and learn to handle large scientific projects. A good scientific and\n",
    "ethical conduct is emphasized throughout the course. \n",
    "\n",
    "* Understand linear methods for regression and classification;\n",
    "\n",
    "* Learn about neural network;\n",
    "\n",
    "* Learn about baggin, boosting and trees\n",
    "\n",
    "* Support vector machines\n",
    "\n",
    "* Learn about basic data analysis;\n",
    "\n",
    "* Be capable of extending the acquired knowledge to other systems and cases;\n",
    "\n",
    "* Have an understanding of central algorithms used in data analysis and machine learning;\n",
    "\n",
    "* Work on numerical projects to illustrate the theory. The projects play a central role and you are expected to know modern programming languages like Python or C++.\n",
    "\n",
    "## Perspective on Machine Learning\n",
    "\n",
    "1. Rapidly emerging application area\n",
    "\n",
    "2. Experiment AND theory are evolving in many many fields. Still many low-hanging fruits.\n",
    "\n",
    "3. Requires education/retraining for more widespread adoption\n",
    "\n",
    "4. A lot of “word-of-mouth” development methods\n",
    "\n",
    "Huge amounts of data sets require automation, classical analysis tools often inadequate. \n",
    "High energy physics hit this wall in the 90’s.\n",
    "In 2009 single top quark production was determined via [Boosted decision trees, Bayesian\n",
    "Neural Networks, etc.](https://arxiv.org/pdf/0903.0850.pdf)\n",
    "\n",
    "\n",
    "## Machine Learning Research\n",
    "\n",
    "Where to find recent results:\n",
    "1. Conference proceedings, arXiv and blog posts!\n",
    "\n",
    "2. **NIPS**: [Neural Information Processing Systems](https://papers.nips.cc)\n",
    "\n",
    "3. **ICLR**: [International Conference on Learning Representations](https://openreview.net/group?id=ICLR.cc/2018/Conference#accepted-oral-papers)\n",
    "\n",
    "4. **ICML**: International Conference on Machine Learning\n",
    "\n",
    "5. [Journal of Machine Learning Research](http://www.jmlr.org/papers/v19/) \n",
    "\n",
    "## Hot Topics Now\n",
    "\n",
    "1. Boosting techniques and complex neural networks\n",
    "\n",
    "2. [Adversarial examples](https://medium.com/@ml.at.berkeley/trickingneural-networks-create-your-own-adversarial-examples-a61eb7620fd8)\n",
    "\n",
    "3. [Zero shot learning](https://arxiv.org/pdf/1707.00600)\n",
    "\n",
    "4. Transfer learning\n",
    "\n",
    "5. [Model interpretability](https://christophm.github.io/interpretable-mlbook/interpretability.html)\n",
    "\n",
    "## Starting your Machine Learning Project\n",
    "\n",
    "1. Identify problem type: classification, generation, regression\n",
    "\n",
    "2. Consider your data carefully\n",
    "\n",
    "3. Choose a simple model that fits 1. and 2.\n",
    "\n",
    "4. Consider your data carefully again… data representation\n",
    "\n",
    "5. Based on results, feedback loop to earliest possible point\n",
    "\n",
    "## Choose a Model and Algorithm\n",
    "\n",
    "1. Supervised?\n",
    "\n",
    "2. Start with the simplest model that fits your problem\n",
    "\n",
    "3. Start with minimal processing of data\n",
    "\n",
    "## Preparing Your Data\n",
    "\n",
    "1. Shuffle your data\n",
    "\n",
    "2. Mean center your data\n",
    "\n",
    "  * Why?\n",
    "\n",
    "\n",
    "3. Normalize the variance\n",
    "\n",
    "  * Why?\n",
    "\n",
    "\n",
    "4. **Whitening**\n",
    "\n",
    "  * Decorrelates data\n",
    "\n",
    "  * Can be hit or miss\n",
    "\n",
    "\n",
    "5. When to do train/test split?\n",
    "\n",
    "## Which Activation and Weights to Choose in Neural Networks\n",
    "\n",
    "1. RELU? ELU?\n",
    "\n",
    "2. Sigmoid or Tanh?\n",
    "\n",
    "3. Set all weights to 0?\n",
    "\n",
    "  * Terrible idea\n",
    "\n",
    "\n",
    "4. Set all weights to random values?\n",
    "\n",
    "  * Small random values\n",
    "\n",
    "\n",
    "## Optimization Methods and Hyperparameters\n",
    "1. Stochastic gradient descent\n",
    "\n",
    "a. Stochastic gradient descent + momentum\n",
    "\n",
    "\n",
    "2. State-of-the-art approaches:\n",
    "\n",
    "  * RMSProp\n",
    "\n",
    "  * Adam\n",
    "\n",
    "\n",
    "Which regularization and hyperparameters? $L_1$ or $L_2$, soft classifiers, depths of trees and many other. Need to explore a large set of hyperparameters and regularization methods. \n",
    "\n",
    "\n",
    "## Resampling\n",
    "\n",
    "When do we resample?\n",
    "\n",
    "1. Bootstrap\n",
    "\n",
    "2. Cross-validation\n",
    "\n",
    "3. Jackknife and many other\n",
    "\n",
    "## Other courses on Data science and Machine Learning  at UiO\n",
    "\n",
    "The link here <https://www.mn.uio.no/english/research/about/centre-focus/innovation/data-science/studies/>  gives an excellent overview of courses on Machine learning at UiO.\n",
    "\n",
    "1. [STK2100 Machine learning and statistical methods for prediction and classification](http://www.uio.no/studier/emner/matnat/math/STK2100/index-eng.html). \n",
    "\n",
    "2. [IN3050 Introduction to Artificial Intelligence and Machine Learning](https://www.uio.no/studier/emner/matnat/ifi/IN3050/index-eng.html). Introductory course in machine learning and AI with an algorithmic approach. \n",
    "\n",
    "3. [STK-INF3000/4000 Selected Topics in Data Science](http://www.uio.no/studier/emner/matnat/math/STK-INF3000/index-eng.html). The course provides insight into selected contemporary relevant topics within Data Science. \n",
    "\n",
    "4. [IN4080 Natural Language Processing](https://www.uio.no/studier/emner/matnat/ifi/IN4080/index.html). Probabilistic and machine learning techniques applied to natural language processing. \n",
    "\n",
    "5. [STK-IN4300 – Statistical learning methods in Data Science](https://www.uio.no/studier/emner/matnat/math/STK-IN4300/index-eng.html). An advanced introduction to statistical and machine learning. For students with a good mathematics and statistics background.\n",
    "\n",
    "6. [INF4490 Biologically Inspired Computing](http://www.uio.no/studier/emner/matnat/ifi/INF4490/). An introduction to self-adapting methods also called artificial intelligence or machine learning. \n",
    "\n",
    "7. [IN-STK5000  Adaptive Methods for Data-Based Decision Making](https://www.uio.no/studier/emner/matnat/ifi/IN-STK5000/index-eng.html). Methods for adaptive collection and processing of data based on machine learning techniques. \n",
    "\n",
    "8. [IN5400/INF5860 – Machine Learning for Image Analysis](https://www.uio.no/studier/emner/matnat/ifi/IN5400/). An introduction to deep learning with particular emphasis on applications within Image analysis, but useful for other application areas too.\n",
    "\n",
    "9. [TEK5040 – Dyp læring for autonome systemer](https://www.uio.no/studier/emner/matnat/its/TEK5040/). The course addresses advanced algorithms and architectures for deep learning with neural networks. The course provides an introduction to how deep-learning techniques can be used in the construction of key parts of advanced autonomous systems that exist in physical environments and cyber environments.\n",
    "\n",
    "## Additional courses of interest\n",
    "\n",
    "1. [STK4051 Computational Statistics](https://www.uio.no/studier/emner/matnat/math/STK4051/index-eng.html)\n",
    "\n",
    "2. [STK4021 Applied Bayesian Analysis and Numerical Methods](https://www.uio.no/studier/emner/matnat/math/STK4021/index-eng.html)\n",
    "\n",
    "## Best wishes to you all and thanks so much for your heroic efforts this semester\n",
    "\n",
    "<!-- dom:FIGURE: [figures/Nebbdyr2.png, width=500 frac=0.6] -->\n",
    "<!-- begin figure -->\n",
    "\n",
    "<p></p>\n",
    "<img src=\"figures/Nebbdyr2.png\" width=500>\n",
    "\n",
    "<!-- end figure -->"
   ]
  }
 ],
 "metadata": {},
 "nbformat": 4,
 "nbformat_minor": 2
}