lars/FYS-STK4155
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

setting up week41

Browse Source
This commit is contained in:
Morten Hjorth-Jensen
2021-10-09 19:16:35 +02:00
parent 6cf9f698fb
commit 37f812763c
7 changed files with 876 additions and 659 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/pub/week41/html/week41-bs.html
+187 -133
View File
@@ -1,15 +1,15 @@
<!--
Automatically generated HTML file from DocOnce source
(https://github.com/hplgit/doconce/)
(https://github.com/doconce/doconce/)
-->
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<meta name="generator" content="DocOnce: https://github.com/hplgit/doconce/" />
<meta name="generator" content="DocOnce: https://github.com/doconce/doconce/" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<meta name="description" content="Week 41 Tensor flow and Deep Learning, Convolutional Neural Networks">
<meta name="description" content="Week 41 Constructing a Neural Network code, Tensor flow and start Convolutional Neural Networks">
<title>Week 41 Tensor flow and Deep Learning, Convolutional Neural Networks</title>
<title>Week 41 Constructing a Neural Network code, Tensor flow and start Convolutional Neural Networks</title>
<!-- Bootstrap style: bootstrap -->
<link href="https://netdna.bootstrapcdn.com/bootstrap/3.1.1/css/bootstrap.min.css" rel="stylesheet">
@@ -41,106 +41,160 @@ Automatically generated HTML file from DocOnce source
<!-- tocinfo
{'highest level': 2,
'sections': [('Plan for week 41', 2, None, '___sec0'),
('Setting up the Back propagation algorithm', 2, None, '___sec1'),
'sections': [('Plan for week 41', 2, None, 'plan-for-week-41'),
('Setting up the Back propagation algorithm',
2,
None,
'setting-up-the-back-propagation-algorithm'),
('Setting up a Multi-layer perceptron model for classification',
2,
None,
'___sec2'),
('Defining the cost function', 2, None, '___sec3'),
('Example: binary classification problem', 2, None, '___sec4'),
('The Softmax function', 2, None, '___sec5'),
'setting-up-a-multi-layer-perceptron-model-for-classification'),
('Defining the cost function',
2,
None,
'defining-the-cost-function'),
('Example: binary classification problem',
2,
None,
'example-binary-classification-problem'),
('The Softmax function', 2, None, 'the-softmax-function'),
('Developing a code for doing neural networks with back '
'propagation',
2,
None,
'___sec6'),
('Collect and pre-process data', 2, None, '___sec7'),
('Train and test datasets', 2, None, '___sec8'),
('Define model and architecture', 2, None, '___sec9'),
('Layers', 2, None, '___sec10'),
('Weights and biases', 2, None, '___sec11'),
('Feed-forward pass', 2, None, '___sec12'),
('Matrix multiplications', 2, None, '___sec13'),
('Choose cost function and optimizer', 2, None, '___sec14'),
('Optimizing the cost function', 2, None, '___sec15'),
('Regularization', 2, None, '___sec16'),
('Matrix multiplication', 2, None, '___sec17'),
('Improving performance', 2, None, '___sec18'),
('Full object-oriented implementation', 2, None, '___sec19'),
('Evaluate model performance on test data', 2, None, '___sec20'),
('Adjust hyperparameters', 2, None, '___sec21'),
('Visualization', 2, None, '___sec22'),
('scikit-learn implementation', 2, None, '___sec23'),
('Visualization', 2, None, '___sec24'),
'developing-a-code-for-doing-neural-networks-with-back-propagation'),
('Collect and pre-process data',
2,
None,
'collect-and-pre-process-data'),
('Train and test datasets', 2, None, 'train-and-test-datasets'),
('Define model and architecture',
2,
None,
'define-model-and-architecture'),
('Layers', 2, None, 'layers'),
('Weights and biases', 2, None, 'weights-and-biases'),
('Feed-forward pass', 2, None, 'feed-forward-pass'),
('Matrix multiplications', 2, None, 'matrix-multiplications'),
('Choose cost function and optimizer',
2,
None,
'choose-cost-function-and-optimizer'),
('Optimizing the cost function',
2,
None,
'optimizing-the-cost-function'),
('Regularization', 2, None, 'regularization'),
('Matrix multiplication', 2, None, 'matrix-multiplication'),
('Improving performance', 2, None, 'improving-performance'),
('Full object-oriented implementation',
2,
None,
'full-object-oriented-implementation'),
('Evaluate model performance on test data',
2,
None,
'evaluate-model-performance-on-test-data'),
('Adjust hyperparameters', 2, None, 'adjust-hyperparameters'),
('Visualization', 2, None, 'visualization'),
('scikit-learn implementation',
2,
None,
'scikit-learn-implementation'),
('Visualization', 2, None, 'visualization'),
('Building neural networks in Tensorflow and Keras',
2,
None,
'___sec25'),
('Tensorflow', 2, None, '___sec26'),
('Using Keras', 2, None, '___sec27'),
('Collect and pre-process data', 2, None, '___sec28'),
('The Breast Cancer Data, now with Keras', 2, None, '___sec29'),
'building-neural-networks-in-tensorflow-and-keras'),
('Tensorflow', 2, None, 'tensorflow'),
('Using Keras', 2, None, 'using-keras'),
('Collect and pre-process data',
2,
None,
'collect-and-pre-process-data'),
('The Breast Cancer Data, now with Keras',
2,
None,
'the-breast-cancer-data-now-with-keras'),
('Fine-tuning neural network hyperparameters',
2,
None,
'___sec30'),
('Hidden layers', 2, None, '___sec31'),
('Which activation function should I use?', 2, None, '___sec32'),
'fine-tuning-neural-network-hyperparameters'),
('Hidden layers', 2, None, 'hidden-layers'),
('Which activation function should I use?',
2,
None,
'which-activation-function-should-i-use'),
('Is the Logistic activation function (Sigmoid) our choice?',
2,
None,
'___sec33'),
('The derivative of the Logistic funtion', 2, None, '___sec34'),
('The RELU function family', 2, None, '___sec35'),
('Which activation function should we use?', 2, None, '___sec36'),
'is-the-logistic-activation-function-sigmoid-our-choice'),
('The derivative of the Logistic funtion',
2,
None,
'the-derivative-of-the-logistic-funtion'),
('The RELU function family', 2, None, 'the-relu-function-family'),
('Which activation function should we use?',
2,
None,
'which-activation-function-should-we-use'),
('More on activation functions, output layers',
2,
None,
'___sec37'),
('Batch Normalization', 2, None, '___sec38'),
('Dropout', 2, None, '___sec39'),
('Gradient Clipping', 2, None, '___sec40'),
('A very nice website on Neural Networks', 2, None, '___sec41'),
'more-on-activation-functions-output-layers'),
('Batch Normalization', 2, None, 'batch-normalization'),
('Dropout', 2, None, 'dropout'),
('Gradient Clipping', 2, None, 'gradient-clipping'),
('A very nice website on Neural Networks',
2,
None,
'a-very-nice-website-on-neural-networks'),
('A top-down perspective on Neural networks',
2,
None,
'___sec42'),
'a-top-down-perspective-on-neural-networks'),
('Limitations of supervised learning with deep networks',
2,
None,
'___sec43'),
'limitations-of-supervised-learning-with-deep-networks'),
('Convolutional Neural Networks (recognizing images)',
2,
None,
'___sec44'),
'convolutional-neural-networks-recognizing-images'),
('Regular NNs don’t scale well to full images',
2,
None,
'___sec45'),
('3D volumes of neurons', 2, None, '___sec46'),
('Layers used to build CNNs', 2, None, '___sec47'),
('Transforming images', 2, None, '___sec48'),
('CNNs in brief', 2, None, '___sec49'),
'regular-nns-don-t-scale-well-to-full-images'),
('3D volumes of neurons', 2, None, '3d-volumes-of-neurons'),
('Layers used to build CNNs',
2,
None,
'layers-used-to-build-cnns'),
('Transforming images', 2, None, 'transforming-images'),
('CNNs in brief', 2, None, 'cnns-in-brief'),
('CNNs in more detail, building convolutional neural networks in '
'Tensorflow and Keras',
2,
None,
'___sec50'),
('Setting it up', 2, None, '___sec51'),
('The MNIST dataset again', 2, None, '___sec52'),
('Strong correlations', 2, None, '___sec53'),
('Layers of a CNN', 2, None, '___sec54'),
('Systematic reduction', 2, None, '___sec55'),
'cnns-in-more-detail-building-convolutional-neural-networks-in-tensorflow-and-keras'),
('Setting it up', 2, None, 'setting-it-up'),
('The MNIST dataset again', 2, None, 'the-mnist-dataset-again'),
('Strong correlations', 2, None, 'strong-correlations'),
('Layers of a CNN', 2, None, 'layers-of-a-cnn'),
('Systematic reduction', 2, None, 'systematic-reduction'),
('Prerequisites: Collect and pre-process data',
2,
None,
'___sec56'),
('Importing Keras and Tensorflow', 2, None, '___sec57'),
('Running with Keras', 2, None, '___sec58'),
('Final part', 2, None, '___sec59'),
('Final visualization', 2, None, '___sec60'),
('Fun links', 2, None, '___sec61')]}
'prerequisites-collect-and-pre-process-data'),
('Importing Keras and Tensorflow',
2,
None,
'importing-keras-and-tensorflow'),
('Running with Keras', 2, None, 'running-with-keras'),
('Final part', 2, None, 'final-part'),
('Final visualization', 2, None, 'final-visualization'),
('Fun links', 2, None, 'fun-links')]}
end of tocinfo -->
<body>
@@ -170,7 +224,7 @@ MathJax.Hub.Config({
<span class="icon-bar"></span>
<span class="icon-bar"></span>
</button>
<a class="navbar-brand" href="week41-bs.html">Week 41 Tensor flow and Deep Learning, Convolutional Neural Networks</a>
<a class="navbar-brand" href="week41-bs.html">Week 41 Constructing a Neural Network code, Tensor flow and start Convolutional Neural Networks</a>
</div>
<div class="navbar-collapse collapse navbar-responsive-collapse">
@@ -178,68 +232,68 @@ MathJax.Hub.Config({
<li class="dropdown">
<a href="#" class="dropdown-toggle" data-toggle="dropdown">Contents <b class="caret"></b></a>
<ul class="dropdown-menu">
<!-- navigation toc: --> <li><a href="._week41-bs001.html#___sec0" style="font-size: 80%;">Plan for week 41</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs002.html#___sec1" style="font-size: 80%;">Setting up the Back propagation algorithm</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs003.html#___sec2" style="font-size: 80%;">Setting up a Multi-layer perceptron model for classification</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs004.html#___sec3" style="font-size: 80%;">Defining the cost function</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs005.html#___sec4" style="font-size: 80%;">Example: binary classification problem</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs006.html#___sec5" style="font-size: 80%;">The Softmax function</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs007.html#___sec6" style="font-size: 80%;">Developing a code for doing neural networks with back propagation</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs008.html#___sec7" style="font-size: 80%;">Collect and pre-process data</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs009.html#___sec8" style="font-size: 80%;">Train and test datasets</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs010.html#___sec9" style="font-size: 80%;">Define model and architecture</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs011.html#___sec10" style="font-size: 80%;">Layers</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs012.html#___sec11" style="font-size: 80%;">Weights and biases</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs013.html#___sec12" style="font-size: 80%;">Feed-forward pass</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs014.html#___sec13" style="font-size: 80%;">Matrix multiplications</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs015.html#___sec14" style="font-size: 80%;">Choose cost function and optimizer</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs016.html#___sec15" style="font-size: 80%;">Optimizing the cost function</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs017.html#___sec16" style="font-size: 80%;">Regularization</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs018.html#___sec17" style="font-size: 80%;">Matrix multiplication</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs019.html#___sec18" style="font-size: 80%;">Improving performance</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs020.html#___sec19" style="font-size: 80%;">Full object-oriented implementation</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs021.html#___sec20" style="font-size: 80%;">Evaluate model performance on test data</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs022.html#___sec21" style="font-size: 80%;">Adjust hyperparameters</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs023.html#___sec22" style="font-size: 80%;">Visualization</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs024.html#___sec23" style="font-size: 80%;">scikit-learn implementation</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs025.html#___sec24" style="font-size: 80%;">Visualization</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs026.html#___sec25" style="font-size: 80%;">Building neural networks in Tensorflow and Keras</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs027.html#___sec26" style="font-size: 80%;">Tensorflow</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs028.html#___sec27" style="font-size: 80%;">Using Keras</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs029.html#___sec28" style="font-size: 80%;">Collect and pre-process data</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs030.html#___sec29" style="font-size: 80%;">The Breast Cancer Data, now with Keras</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs031.html#___sec30" style="font-size: 80%;">Fine-tuning neural network hyperparameters</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs032.html#___sec31" style="font-size: 80%;">Hidden layers</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs033.html#___sec32" style="font-size: 80%;">Which activation function should I use?</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs034.html#___sec33" style="font-size: 80%;">Is the Logistic activation function (Sigmoid) our choice?</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs035.html#___sec34" style="font-size: 80%;">The derivative of the Logistic funtion</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs036.html#___sec35" style="font-size: 80%;">The RELU function family</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs037.html#___sec36" style="font-size: 80%;">Which activation function should we use?</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs038.html#___sec37" style="font-size: 80%;">More on activation functions, output layers</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs039.html#___sec38" style="font-size: 80%;">Batch Normalization</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs040.html#___sec39" style="font-size: 80%;">Dropout</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs041.html#___sec40" style="font-size: 80%;">Gradient Clipping</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs042.html#___sec41" style="font-size: 80%;">A very nice website on Neural Networks</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs043.html#___sec42" style="font-size: 80%;">A top-down perspective on Neural networks</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs044.html#___sec43" style="font-size: 80%;">Limitations of supervised learning with deep networks</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs045.html#___sec44" style="font-size: 80%;">Convolutional Neural Networks (recognizing images)</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs046.html#___sec45" style="font-size: 80%;">Regular NNs don’t scale well to full images</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs047.html#___sec46" style="font-size: 80%;">3D volumes of neurons</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs048.html#___sec47" style="font-size: 80%;">Layers used to build CNNs</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs049.html#___sec48" style="font-size: 80%;">Transforming images</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs050.html#___sec49" style="font-size: 80%;">CNNs in brief</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs051.html#___sec50" style="font-size: 80%;">CNNs in more detail, building convolutional neural networks in Tensorflow and Keras</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs052.html#___sec51" style="font-size: 80%;">Setting it up</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs053.html#___sec52" style="font-size: 80%;">The MNIST dataset again</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs054.html#___sec53" style="font-size: 80%;">Strong correlations</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs055.html#___sec54" style="font-size: 80%;">Layers of a CNN</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs056.html#___sec55" style="font-size: 80%;">Systematic reduction</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs057.html#___sec56" style="font-size: 80%;">Prerequisites: Collect and pre-process data</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs058.html#___sec57" style="font-size: 80%;">Importing Keras and Tensorflow</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs059.html#___sec58" style="font-size: 80%;">Running with Keras</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs060.html#___sec59" style="font-size: 80%;">Final part</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs061.html#___sec60" style="font-size: 80%;">Final visualization</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs062.html#___sec61" style="font-size: 80%;">Fun links</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs001.html#plan-for-week-41" style="font-size: 80%;">Plan for week 41</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs002.html#setting-up-the-back-propagation-algorithm" style="font-size: 80%;">Setting up the Back propagation algorithm</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs003.html#setting-up-a-multi-layer-perceptron-model-for-classification" style="font-size: 80%;">Setting up a Multi-layer perceptron model for classification</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs004.html#defining-the-cost-function" style="font-size: 80%;">Defining the cost function</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs005.html#example-binary-classification-problem" style="font-size: 80%;">Example: binary classification problem</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs006.html#the-softmax-function" style="font-size: 80%;">The Softmax function</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs007.html#developing-a-code-for-doing-neural-networks-with-back-propagation" style="font-size: 80%;">Developing a code for doing neural networks with back propagation</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs029.html#collect-and-pre-process-data" style="font-size: 80%;">Collect and pre-process data</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs009.html#train-and-test-datasets" style="font-size: 80%;">Train and test datasets</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs010.html#define-model-and-architecture" style="font-size: 80%;">Define model and architecture</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs011.html#layers" style="font-size: 80%;">Layers</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs012.html#weights-and-biases" style="font-size: 80%;">Weights and biases</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs013.html#feed-forward-pass" style="font-size: 80%;">Feed-forward pass</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs014.html#matrix-multiplications" style="font-size: 80%;">Matrix multiplications</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs015.html#choose-cost-function-and-optimizer" style="font-size: 80%;">Choose cost function and optimizer</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs016.html#optimizing-the-cost-function" style="font-size: 80%;">Optimizing the cost function</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs017.html#regularization" style="font-size: 80%;">Regularization</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs018.html#matrix-multiplication" style="font-size: 80%;">Matrix multiplication</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs019.html#improving-performance" style="font-size: 80%;">Improving performance</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs020.html#full-object-oriented-implementation" style="font-size: 80%;">Full object-oriented implementation</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs021.html#evaluate-model-performance-on-test-data" style="font-size: 80%;">Evaluate model performance on test data</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs022.html#adjust-hyperparameters" style="font-size: 80%;">Adjust hyperparameters</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs025.html#visualization" style="font-size: 80%;">Visualization</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs024.html#scikit-learn-implementation" style="font-size: 80%;">scikit-learn implementation</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs025.html#visualization" style="font-size: 80%;">Visualization</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs026.html#building-neural-networks-in-tensorflow-and-keras" style="font-size: 80%;">Building neural networks in Tensorflow and Keras</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs027.html#tensorflow" style="font-size: 80%;">Tensorflow</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs028.html#using-keras" style="font-size: 80%;">Using Keras</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs029.html#collect-and-pre-process-data" style="font-size: 80%;">Collect and pre-process data</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs030.html#the-breast-cancer-data-now-with-keras" style="font-size: 80%;">The Breast Cancer Data, now with Keras</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs031.html#fine-tuning-neural-network-hyperparameters" style="font-size: 80%;">Fine-tuning neural network hyperparameters</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs032.html#hidden-layers" style="font-size: 80%;">Hidden layers</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs033.html#which-activation-function-should-i-use" style="font-size: 80%;">Which activation function should I use?</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs034.html#is-the-logistic-activation-function-sigmoid-our-choice" style="font-size: 80%;">Is the Logistic activation function (Sigmoid) our choice?</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs035.html#the-derivative-of-the-logistic-funtion" style="font-size: 80%;">The derivative of the Logistic funtion</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs036.html#the-relu-function-family" style="font-size: 80%;">The RELU function family</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs037.html#which-activation-function-should-we-use" style="font-size: 80%;">Which activation function should we use?</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs038.html#more-on-activation-functions-output-layers" style="font-size: 80%;">More on activation functions, output layers</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs039.html#batch-normalization" style="font-size: 80%;">Batch Normalization</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs040.html#dropout" style="font-size: 80%;">Dropout</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs041.html#gradient-clipping" style="font-size: 80%;">Gradient Clipping</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs042.html#a-very-nice-website-on-neural-networks" style="font-size: 80%;">A very nice website on Neural Networks</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs043.html#a-top-down-perspective-on-neural-networks" style="font-size: 80%;">A top-down perspective on Neural networks</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs044.html#limitations-of-supervised-learning-with-deep-networks" style="font-size: 80%;">Limitations of supervised learning with deep networks</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs045.html#convolutional-neural-networks-recognizing-images" style="font-size: 80%;">Convolutional Neural Networks (recognizing images)</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs046.html#regular-nns-don-t-scale-well-to-full-images" style="font-size: 80%;">Regular NNs don’t scale well to full images</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs047.html#3d-volumes-of-neurons" style="font-size: 80%;">3D volumes of neurons</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs048.html#layers-used-to-build-cnns" style="font-size: 80%;">Layers used to build CNNs</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs049.html#transforming-images" style="font-size: 80%;">Transforming images</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs050.html#cnns-in-brief" style="font-size: 80%;">CNNs in brief</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs051.html#cnns-in-more-detail-building-convolutional-neural-networks-in-tensorflow-and-keras" style="font-size: 80%;">CNNs in more detail, building convolutional neural networks in Tensorflow and Keras</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs052.html#setting-it-up" style="font-size: 80%;">Setting it up</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs053.html#the-mnist-dataset-again" style="font-size: 80%;">The MNIST dataset again</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs054.html#strong-correlations" style="font-size: 80%;">Strong correlations</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs055.html#layers-of-a-cnn" style="font-size: 80%;">Layers of a CNN</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs056.html#systematic-reduction" style="font-size: 80%;">Systematic reduction</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs057.html#prerequisites-collect-and-pre-process-data" style="font-size: 80%;">Prerequisites: Collect and pre-process data</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs058.html#importing-keras-and-tensorflow" style="font-size: 80%;">Importing Keras and Tensorflow</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs059.html#running-with-keras" style="font-size: 80%;">Running with Keras</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs060.html#final-part" style="font-size: 80%;">Final part</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs061.html#final-visualization" style="font-size: 80%;">Final visualization</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs062.html#fun-links" style="font-size: 80%;">Fun links</a></li>
</ul>
</li>
@@ -258,7 +312,7 @@ MathJax.Hub.Config({
<div class="jumbotron">
<center><h1>Week 41 Tensor flow and Deep Learning, Convolutional Neural Networks</h1></center> <!-- document title -->
<center><h1>Week 41 Constructing a Neural Network code, Tensor flow and start Convolutional Neural Networks</h1></center> <!-- document title -->
<p>
<!-- author(s): Morten Hjorth-Jensen -->
@@ -274,7 +328,7 @@ MathJax.Hub.Config({
<center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
<br>
<p>
<center><h4>Oct 10, 2020</h4></center> <!-- date -->
<center><h4>Oct 9, 2021</h4></center> <!-- date -->
<br>
<p>
@@ -310,13 +364,13 @@ MathJax.Hub.Config({
<!-- Bootstrap footer
<footer>
<a href="http://..."><img width="250" align=right src="http://..."></a>
<a href="https://..."><img width="250" align=right src="https://..."></a>
</footer>
-->
<center style="font-size:80%">
<!-- copyright --> &copy; 1999-2020, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license
<!-- copyright --> &copy; 1999-2021, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license
</center>
doc/pub/week41/html/week41-reveal.html
+103 -102
View File
@@ -1,11 +1,11 @@
<!DOCTYPE html>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<meta name="generator" content="DocOnce: https://github.com/hplgit/doconce/" />
<meta name="generator" content="DocOnce: https://github.com/doconce/doconce/" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<meta name="description" content="Week 41 Tensor flow and Deep Learning, Convolutional Neural Networks">
<meta name="description" content="Week 41 Constructing a Neural Network code, Tensor flow and start Convolutional Neural Networks">
<title>Week 41 Tensor flow and Deep Learning, Convolutional Neural Networks</title>
<title>Week 41 Constructing a Neural Network code, Tensor flow and start Convolutional Neural Networks</title>
@@ -13,7 +13,7 @@
<!-- reveal.js: http://lab.hakim.se/reveal-js/ -->
<!-- reveal.js: https://lab.hakim.se/reveal-js/ -->
<meta name="viewport" content="width=device-width, initial-scale=1.0, maximum-scale=1.0, user-scalable=no">
@@ -76,10 +76,10 @@ document.getElementsByTagName( 'head' )[0].appendChild( link );
.reveal .alert-block > p, .alert-block > ul {margin-bottom:1em}
/*.reveal .alert li {margin-top: 1em}*/
.reveal .alert-block p+p {margin-top:5px}
/*.reveal .alert-notice { background-image: url(http://hplgit.github.io/doconce/bundled/html_images/small_gray_notice.png); }
.reveal .alert-summary { background-image:url(http://hplgit.github.io/doconce/bundled/html_images/small_gray_summary.png); }
.reveal .alert-warning { background-image: url(http://hplgit.github.io/doconce/bundled/html_images/small_gray_warning.png); }
.reveal .alert-question {background-image:url(http://hplgit.github.io/doconce/bundled/html_images/small_gray_question.png); } */
/*.reveal .alert-notice { background-image: url(https://hplgit.github.io/doconce/bundled/html_images/small_gray_notice.png); }
.reveal .alert-summary { background-image:url(https://hplgit.github.io/doconce/bundled/html_images/small_gray_summary.png); }
.reveal .alert-warning { background-image: url(https://hplgit.github.io/doconce/bundled/html_images/small_gray_warning.png); }
.reveal .alert-question {background-image:url(https://hplgit.github.io/doconce/bundled/html_images/small_gray_question.png); } */
</style>
@@ -132,7 +132,7 @@ MathJax.Hub.Config({
<center><h1 style="text-align: center;">Week 41 Tensor flow and Deep Learning, Convolutional Neural Networks</h1></center> <!-- document title -->
<center><h1 style="text-align: center;">Week 41 Constructing a Neural Network code, Tensor flow and start Convolutional Neural Networks</h1></center> <!-- document title -->
<p>
<!-- author(s): Morten Hjorth-Jensen -->
@@ -148,32 +148,33 @@ MathJax.Hub.Config({
<center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
<br>
<p>&nbsp;<br>
<center><h4>Oct 10, 2020</h4></center> <!-- date -->
<center><h4>Oct 9, 2021</h4></center> <!-- date -->
<br>
<p>
<center style="font-size:80%">
<!-- copyright --> &copy; 1999-2020, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license
<!-- copyright --> &copy; 1999-2021, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license
</center>
</section>
<section>
<h2 id="___sec0">Plan for week 41 </h2>
<h2 id="plan-for-week-41">Plan for week 41 </h2>
<ul>
<p><li> Thursday: Building our own Feed-forward Neural Network. <a href="https://www.uio.no/studier/emner/matnat/fys/FYS-STK4155/h20/forelesningsvideoer/LectureOctober8.mp4?vrtx=view-as-webpage" target="_blank">Video of Lecture</a></li>
<p><li> Friday: Playing around with our own Feed-forward Neural Network and introduction to TensorFlow. Start convolutional Neural Networks (CNN). <a href="https://www.uio.no/studier/emner/matnat/fys/FYS-STK4155/h20/forelesningsvideoer/LectureOcotber9.mp4?vrtx=view-as-webpage" target="_blank">Video of Lecture</a></li>
<p><li> Thursday: Building our own Feed-forward Neural Network.</li>
<p><li> Friday: Playing around with our own Feed-forward Neural Network and introduction to TensorFlow. Start convolutional Neural Networks (CNN).</li>
</ul>
<p>
Reading suggestions for both days: <a href="https://github.com/CompPhysics/MachineLearning/blob/master/doc/T\
extbooks/TensorflowML.pdf" target="_blank">Aurelien Geron's chapters 10-11</a> and Hastie et al chapter 11.
extbooks/TensorflowML.pdf" target="_blank">Aurelien Geron's chapters 10-11</a>.
For amore in depth discussion on neural networks we recommend Goodfellow et al chapters 6 and 7. For CNNs, see Goodfellow et al chapter 9. chapter 11 and 12 on practicalities and applications
</section>
<section>
<h2 id="___sec1">Setting up the Back propagation algorithm </h2>
<h2 id="setting-up-the-back-propagation-algorithm">Setting up the Back propagation algorithm </h2>
<p>
The four equations derived last week provide us with a way of computing the gradient of the cost function. Let us write this out in the form of an algorithm.
@@ -246,7 +247,7 @@ Here it is convenient to use stochastic gradient descent (see the examples below
<section>
<h2 id="___sec2">Setting up a Multi-layer perceptron model for classification </h2>
<h2 id="setting-up-a-multi-layer-perceptron-model-for-classification">Setting up a Multi-layer perceptron model for classification </h2>
<p>
We are now gong to develop an example based on the MNIST data
@@ -288,7 +289,7 @@ of our network.
<section>
<h2 id="___sec3">Defining the cost function </h2>
<h2 id="defining-the-cost-function">Defining the cost function </h2>
<p>
Our cost function is given as (see the Logistic regression lectures)
@@ -356,7 +357,7 @@ The back propagation equations need now only a small change, namely the definiti
<section>
<h2 id="___sec4">Example: binary classification problem </h2>
<h2 id="example-binary-classification-problem">Example: binary classification problem </h2>
<p>
As an example of the above, relevant for project 2 as well, let us consider a binary class. As discussed in our logistic regression lectures, we defined a cost function in terms of the parameters \( \beta \) as
@@ -418,7 +419,7 @@ In case we use another activation function than the logistic one, we need to eva
<section>
<h2 id="___sec5">The Softmax function </h2>
<h2 id="the-softmax-function">The Softmax function </h2>
In case we employ the more general case given by the Softmax equation, we need to evaluate the derivative of the activation function with respect to the activation \( z_i^l \), that is we need
<p>&nbsp;<br>
$$
@@ -446,7 +447,7 @@ which in case of the simply binary model reduces to having \( i=j \).
<section>
<h2 id="___sec6">Developing a code for doing neural networks with back propagation </h2>
<h2 id="developing-a-code-for-doing-neural-networks-with-back-propagation">Developing a code for doing neural networks with back propagation </h2>
<p>
One can identify a set of key steps when using neural networks to solve supervised learning problems:
@@ -468,7 +469,7 @@ One can identify a set of key steps when using neural networks to solve supervis
<section>
<h2 id="___sec7">Collect and pre-process data </h2>
<h2 id="collect-and-pre-process-data">Collect and pre-process data </h2>
<p>
Here we will be using the MNIST dataset, which is readily available through the <b>scikit-learn</b>
@@ -521,7 +522,7 @@ images.
<p>
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
<div class="highlight" style="background: #eeeedd"><pre style="font-size: 80%; line-height: 125%"><span></span><span style="color: #228B22"># import necessary packages</span>
<div class="highlight" style="background: #eeeedd"><pre style="font-size: 80%; line-height: 125%;"><span></span><span style="color: #228B22"># import necessary packages</span>
<span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">numpy</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">np</span>
<span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">matplotlib.pyplot</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">plt</span>
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">sklearn</span> <span style="color: #8B008B; font-weight: bold">import</span> datasets
@@ -568,7 +569,7 @@ plt.show()
<section>
<h2 id="___sec8">Train and test datasets </h2>
<h2 id="train-and-test-datasets">Train and test datasets </h2>
<p>
Performing analysis before partitioning the dataset is a major error, that can lead to incorrect conclusions.
@@ -586,7 +587,7 @@ collected from 12.00 to 24.00.
<p>
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
<div class="highlight" style="background: #eeeedd"><pre style="font-size: 80%; line-height: 125%"><span></span><span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">sklearn.model_selection</span> <span style="color: #8B008B; font-weight: bold">import</span> train_test_split
<div class="highlight" style="background: #eeeedd"><pre style="font-size: 80%; line-height: 125%;"><span></span><span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">sklearn.model_selection</span> <span style="color: #8B008B; font-weight: bold">import</span> train_test_split
<span style="color: #228B22"># one-liner from scikit-learn library</span>
train_size = <span style="color: #B452CD">0.8</span>
@@ -618,7 +619,7 @@ X_train, X_test, Y_train, Y_test = train_test_split(inputs, labels, train_size=t
<section>
<h2 id="___sec9">Define model and architecture </h2>
<h2 id="define-model-and-architecture">Define model and architecture </h2>
<p>
Our simple feed-forward neural network will consist of an <em>input</em> layer, a single <em>hidden</em> layer and an <em>output</em> layer. The activation \( y \) of each neuron is a weighted sum of inputs, passed through an activation function. In case of the simple perceptron model we have
@@ -672,7 +673,7 @@ which is inspired by probability theory (see logistic regression) and was most c
<section>
<h2 id="___sec10">Layers </h2>
<h2 id="layers">Layers </h2>
<ul>
<p><li> Input</li>
@@ -724,7 +725,7 @@ weights to the output layer.
<section>
<h2 id="___sec11">Weights and biases </h2>
<h2 id="weights-and-biases">Weights and biases </h2>
<p>
Typically weights are initialized with small values distributed around zero, drawn from a uniform
@@ -743,7 +744,7 @@ The bias weights \( \hat{b} \) are often initialized to zero, but a small value
<p>
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
<div class="highlight" style="background: #eeeedd"><pre style="font-size: 80%; line-height: 125%"><span></span><span style="color: #228B22"># building our neural network</span>
<div class="highlight" style="background: #eeeedd"><pre style="font-size: 80%; line-height: 125%;"><span></span><span style="color: #228B22"># building our neural network</span>
n_inputs, n_features = X_train.shape
n_hidden_neurons = <span style="color: #B452CD">50</span>
@@ -763,7 +764,7 @@ output_bias = np.zeros(n_categories) + <span style="color: #B452CD">0.01</span>
<section>
<h2 id="___sec12">Feed-forward pass </h2>
<h2 id="feed-forward-pass">Feed-forward pass </h2>
<p>
Denote \( F \) the number of features, \( H \) the number of hidden neurons and \( C \) the number of categories.
@@ -798,7 +799,7 @@ $$ a_{j}^{L} = \frac{\exp{(z_j^{L})}}
<section>
<h2 id="___sec13">Matrix multiplications </h2>
<h2 id="matrix-multiplications">Matrix multiplications </h2>
<p>
Since our data has the dimensions \( X = (n_{inputs}, n_{features}) \) and our weights to the hidden
@@ -844,7 +845,7 @@ $$ output = softmax (\hat{z}^{L}) = (n_{inputs}, n_{categories}) .$$
<p>
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
<div class="highlight" style="background: #eeeedd"><pre style="font-size: 80%; line-height: 125%"><span></span><span style="color: #228B22"># setup the feed-forward pass, subscript h = hidden layer</span>
<div class="highlight" style="background: #eeeedd"><pre style="font-size: 80%; line-height: 125%;"><span></span><span style="color: #228B22"># setup the feed-forward pass, subscript h = hidden layer</span>
<span style="color: #8B008B; font-weight: bold">def</span> <span style="color: #008b45">sigmoid</span>(x):
<span style="color: #8B008B; font-weight: bold">return</span> <span style="color: #B452CD">1</span>/(<span style="color: #B452CD">1</span> + np.exp(-x))
@@ -884,7 +885,7 @@ predictions = predict(X_train)
<section>
<h2 id="___sec14">Choose cost function and optimizer </h2>
<h2 id="choose-cost-function-and-optimizer">Choose cost function and optimizer </h2>
<p>
To measure how well our neural network is doing we need to introduce a cost function.
@@ -919,7 +920,7 @@ you got the correct label. The probability of category \( c \) is given by the s
<section>
<h2 id="___sec15">Optimizing the cost function </h2>
<h2 id="optimizing-the-cost-function">Optimizing the cost function </h2>
<p>
The network is trained by finding the weights and biases that minimize the cost function. One of the most widely used classes of methods is <em>gradient descent</em> and its generalizations. The idea behind gradient descent
@@ -965,7 +966,7 @@ The various optmization methods, with codes and algorithms, are discussed in o
<section>
<h2 id="___sec16">Regularization </h2>
<h2 id="regularization">Regularization </h2>
<p>
It is common to add an extra term to the cost function, proportional
@@ -1000,7 +1001,7 @@ calculate the gradient efficently.
<section>
<h2 id="___sec17">Matrix multiplication </h2>
<h2 id="matrix-multiplication">Matrix multiplication </h2>
<p>
To more efficently train our network these equations are implemented using matrix operations.
@@ -1054,7 +1055,7 @@ $$ \nabla b_{h} = \sum_{i=1}^{n_{inputs}} \delta_h = (n_{hidden}) .$$
<p>
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
<div class="highlight" style="background: #eeeedd"><pre style="font-size: 80%; line-height: 125%"><span></span><span style="color: #228B22"># to categorical turns our integer vector into a onehot representation</span>
<div class="highlight" style="background: #eeeedd"><pre style="font-size: 80%; line-height: 125%;"><span></span><span style="color: #228B22"># to categorical turns our integer vector into a onehot representation</span>
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">sklearn.metrics</span> <span style="color: #8B008B; font-weight: bold">import</span> accuracy_score
<span style="color: #228B22"># one-hot in numpy</span>
@@ -1127,7 +1128,7 @@ lmbd = <span style="color: #B452CD">0.01</span>
<section>
<h2 id="___sec18">Improving performance </h2>
<h2 id="improving-performance">Improving performance </h2>
<p>
As we can see the network does not seem to be learning at all. It seems to be just guessing the label for each image.
@@ -1147,7 +1148,7 @@ Andrew Ng goes through some of these considerations in this <a href="https://you
<section>
<h2 id="___sec19">Full object-oriented implementation </h2>
<h2 id="full-object-oriented-implementation">Full object-oriented implementation </h2>
<p>
It is very natural to think of the network as an object, with specific instances of the network
@@ -1156,7 +1157,7 @@ being realizations of this object with different hyperparameters. An implementat
<p>
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
<div class="highlight" style="background: #eeeedd"><pre style="font-size: 80%; line-height: 125%"><span></span><span style="color: #8B008B; font-weight: bold">class</span> <span style="color: #008b45; font-weight: bold">NeuralNetwork</span>:
<div class="highlight" style="background: #eeeedd"><pre style="font-size: 80%; line-height: 125%;"><span></span><span style="color: #8B008B; font-weight: bold">class</span> <span style="color: #008b45; font-weight: bold">NeuralNetwork</span>:
<span style="color: #8B008B; font-weight: bold">def</span> <span style="color: #008b45">__init__</span>(
<span style="color: #658b00">self</span>,
X_data,
@@ -1260,7 +1261,7 @@ being realizations of this object with different hyperparameters. An implementat
<section>
<h2 id="___sec20">Evaluate model performance on test data </h2>
<h2 id="evaluate-model-performance-on-test-data">Evaluate model performance on test data </h2>
<p>
To measure the performance of our network we evaluate how well it does it data it has never seen before, i.e. the test data.
@@ -1277,7 +1278,7 @@ where \( I \) is the indicator function, \( 1 \) if \( \hat{y}_i = y_i \) and \(
<p>
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
<div class="highlight" style="background: #eeeedd"><pre style="font-size: 80%; line-height: 125%"><span></span>epochs = <span style="color: #B452CD">100</span>
<div class="highlight" style="background: #eeeedd"><pre style="font-size: 80%; line-height: 125%;"><span></span>epochs = <span style="color: #B452CD">100</span>
batch_size = <span style="color: #B452CD">100</span>
dnn = NeuralNetwork(X_train, Y_train_onehot, eta=eta, lmbd=lmbd, epochs=epochs, batch_size=batch_size,
@@ -1298,7 +1299,7 @@ test_predict = dnn.predict(X_test)
<section>
<h2 id="___sec21">Adjust hyperparameters </h2>
<h2 id="adjust-hyperparameters">Adjust hyperparameters </h2>
<p>
We now perform a grid search to find the optimal hyperparameters for the network.
@@ -1307,7 +1308,7 @@ Note that we are only using 1 layer with 50 neurons, and human performance is es
<p>
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
<div class="highlight" style="background: #eeeedd"><pre style="font-size: 80%; line-height: 125%"><span></span>eta_vals = np.logspace(-<span style="color: #B452CD">5</span>, <span style="color: #B452CD">1</span>, <span style="color: #B452CD">7</span>)
<div class="highlight" style="background: #eeeedd"><pre style="font-size: 80%; line-height: 125%;"><span></span>eta_vals = np.logspace(-<span style="color: #B452CD">5</span>, <span style="color: #B452CD">1</span>, <span style="color: #B452CD">7</span>)
lmbd_vals = np.logspace(-<span style="color: #B452CD">5</span>, <span style="color: #B452CD">1</span>, <span style="color: #B452CD">7</span>)
<span style="color: #228B22"># store the models for later use</span>
DNN_numpy = np.zeros((<span style="color: #658b00">len</span>(eta_vals), <span style="color: #658b00">len</span>(lmbd_vals)), dtype=<span style="color: #658b00">object</span>)
@@ -1332,12 +1333,12 @@ DNN_numpy = np.zeros((<span style="color: #658b00">len</span>(eta_vals), <span s
<section>
<h2 id="___sec22">Visualization </h2>
<h2 id="visualization">Visualization </h2>
<p>
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
<div class="highlight" style="background: #eeeedd"><pre style="font-size: 80%; line-height: 125%"><span></span><span style="color: #228B22"># visual representation of grid search</span>
<div class="highlight" style="background: #eeeedd"><pre style="font-size: 80%; line-height: 125%;"><span></span><span style="color: #228B22"># visual representation of grid search</span>
<span style="color: #228B22"># uses seaborn heatmap, you can also do this with matplotlib imshow</span>
<span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">seaborn</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">sns</span>
@@ -1375,7 +1376,7 @@ plt.show()
<section>
<h2 id="___sec23">scikit-learn implementation </h2>
<h2 id="scikit-learn-implementation">scikit-learn implementation </h2>
<p>
<b>scikit-learn</b> focuses more
@@ -1394,7 +1395,7 @@ performance overall.
<p>
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
<div class="highlight" style="background: #eeeedd"><pre style="font-size: 80%; line-height: 125%"><span></span><span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">sklearn.neural_network</span> <span style="color: #8B008B; font-weight: bold">import</span> MLPClassifier
<div class="highlight" style="background: #eeeedd"><pre style="font-size: 80%; line-height: 125%;"><span></span><span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">sklearn.neural_network</span> <span style="color: #8B008B; font-weight: bold">import</span> MLPClassifier
<span style="color: #228B22"># store models for later use</span>
DNN_scikit = np.zeros((<span style="color: #658b00">len</span>(eta_vals), <span style="color: #658b00">len</span>(lmbd_vals)), dtype=<span style="color: #658b00">object</span>)
@@ -1415,11 +1416,11 @@ DNN_scikit = np.zeros((<span style="color: #658b00">len</span>(eta_vals), <span
<section>
<h2 id="___sec24">Visualization </h2>
<h2 id="visualization">Visualization </h2>
<p>
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
<div class="highlight" style="background: #eeeedd"><pre style="font-size: 80%; line-height: 125%"><span></span><span style="color: #228B22"># optional</span>
<div class="highlight" style="background: #eeeedd"><pre style="font-size: 80%; line-height: 125%;"><span></span><span style="color: #228B22"># optional</span>
<span style="color: #228B22"># visual representation of grid search</span>
<span style="color: #228B22"># uses seaborn heatmap, could probably do this in matplotlib</span>
<span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">seaborn</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">sns</span>
@@ -1458,7 +1459,7 @@ plt.show()
<section>
<h2 id="___sec25">Building neural networks in Tensorflow and Keras </h2>
<h2 id="building-neural-networks-in-tensorflow-and-keras">Building neural networks in Tensorflow and Keras </h2>
<p>
Now we want to build on the experience gained from our neural network implementation in NumPy and scikit-learn
@@ -1473,7 +1474,7 @@ NumPy arrays.
<section>
<h2 id="___sec26">Tensorflow </h2>
<h2 id="tensorflow">Tensorflow </h2>
<p>
Tensorflow is an open source library machine learning library
@@ -1506,7 +1507,7 @@ To install tensorflow on Unix/Linux systems, use pip as
<p>
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
<div class="highlight" style="background: #eeeedd"><pre style="font-size: 80%; line-height: 125%"><span></span>pip3 install tensorflow
<div class="highlight" style="background: #eeeedd"><pre style="font-size: 80%; line-height: 125%;"><span></span>pip3 install tensorflow
</pre></div>
<p>
and/or if you use <b>anaconda</b>, just write (or install from the graphical user interface)
@@ -1514,7 +1515,7 @@ and/or if you use <b>anaconda</b>, just write (or install from the graphical use
<p>
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
<div class="highlight" style="background: #eeeedd"><pre style="font-size: 80%; line-height: 125%"><span></span>conda create -n tf tensorflow
<div class="highlight" style="background: #eeeedd"><pre style="font-size: 80%; line-height: 125%;"><span></span>conda create -n tf tensorflow
conda activate tf
</pre></div>
<p>
@@ -1522,14 +1523,14 @@ To install the current release of GPU TensorFlow
<p>
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
<div class="highlight" style="background: #eeeedd"><pre style="font-size: 80%; line-height: 125%"><span></span>conda create -n tf-gpu tensorflow-gpu
<div class="highlight" style="background: #eeeedd"><pre style="font-size: 80%; line-height: 125%;"><span></span>conda create -n tf-gpu tensorflow-gpu
conda activate tf-gpu
</pre></div>
</section>
<section>
<h2 id="___sec27">Using Keras </h2>
<h2 id="using-keras">Using Keras </h2>
<p>
Keras is a high level <a href="https://en.wikipedia.org/wiki/Application_programming_interface" target="_blank">neural network</a>
@@ -1538,7 +1539,7 @@ If you have Anaconda installed you may run the following command
<p>
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
<div class="highlight" style="background: #eeeedd"><pre style="font-size: 80%; line-height: 125%"><span></span>conda install keras
<div class="highlight" style="background: #eeeedd"><pre style="font-size: 80%; line-height: 125%;"><span></span>conda install keras
</pre></div>
<p>
You can look up the <a href="https://keras.io/" target="_blank">instructions here</a> for more information.
@@ -1549,7 +1550,7 @@ We will to a large extent use <b>keras</b> in this course.
<section>
<h2 id="___sec28">Collect and pre-process data </h2>
<h2 id="collect-and-pre-process-data">Collect and pre-process data </h2>
<p>
Let us look again at the MINST data set.
@@ -1557,7 +1558,7 @@ Let us look again at the MINST data set.
<p>
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
<div class="highlight" style="background: #eeeedd"><pre style="font-size: 80%; line-height: 125%"><span></span><span style="color: #228B22"># import necessary packages</span>
<div class="highlight" style="background: #eeeedd"><pre style="font-size: 80%; line-height: 125%;"><span></span><span style="color: #228B22"># import necessary packages</span>
<span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">numpy</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">np</span>
<span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">matplotlib.pyplot</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">plt</span>
<span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">tensorflow</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">tf</span>
@@ -1604,7 +1605,7 @@ plt.show()
<p>
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
<div class="highlight" style="background: #eeeedd"><pre style="font-size: 80%; line-height: 125%"><span></span><span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">tensorflow.keras.layers</span> <span style="color: #8B008B; font-weight: bold">import</span> Input
<div class="highlight" style="background: #eeeedd"><pre style="font-size: 80%; line-height: 125%;"><span></span><span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">tensorflow.keras.layers</span> <span style="color: #8B008B; font-weight: bold">import</span> Input
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">tensorflow.keras.models</span> <span style="color: #8B008B; font-weight: bold">import</span> Sequential <span style="color: #228B22">#This allows appending layers to existing models</span>
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">tensorflow.keras.layers</span> <span style="color: #8B008B; font-weight: bold">import</span> Dense <span style="color: #228B22">#This allows defining the characteristics of a particular layer</span>
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">tensorflow.keras</span> <span style="color: #8B008B; font-weight: bold">import</span> optimizers <span style="color: #228B22">#This allows using whichever optimiser we want (sgd,adam,RMSprop)</span>
@@ -1625,7 +1626,7 @@ X_train, X_test, Y_train, Y_test = train_test_split(inputs, labels, train_size=t
<p>
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
<div class="highlight" style="background: #eeeedd"><pre style="font-size: 80%; line-height: 125%"><span></span>epochs = <span style="color: #B452CD">100</span>
<div class="highlight" style="background: #eeeedd"><pre style="font-size: 80%; line-height: 125%;"><span></span>epochs = <span style="color: #B452CD">100</span>
batch_size = <span style="color: #B452CD">100</span>
n_neurons_layer1 = <span style="color: #B452CD">100</span>
n_neurons_layer2 = <span style="color: #B452CD">50</span>
@@ -1646,7 +1647,7 @@ lmbd_vals = np.logspace(-<span style="color: #B452CD">5</span>, <span style="col
<p>
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
<div class="highlight" style="background: #eeeedd"><pre style="font-size: 80%; line-height: 125%"><span></span>DNN_keras = np.zeros((<span style="color: #658b00">len</span>(eta_vals), <span style="color: #658b00">len</span>(lmbd_vals)), dtype=<span style="color: #658b00">object</span>)
<div class="highlight" style="background: #eeeedd"><pre style="font-size: 80%; line-height: 125%;"><span></span>DNN_keras = np.zeros((<span style="color: #658b00">len</span>(eta_vals), <span style="color: #658b00">len</span>(lmbd_vals)), dtype=<span style="color: #658b00">object</span>)
<span style="color: #8B008B; font-weight: bold">for</span> i, eta <span style="color: #8B008B">in</span> <span style="color: #658b00">enumerate</span>(eta_vals):
<span style="color: #8B008B; font-weight: bold">for</span> j, lmbd <span style="color: #8B008B">in</span> <span style="color: #658b00">enumerate</span>(lmbd_vals):
@@ -1665,7 +1666,7 @@ lmbd_vals = np.logspace(-<span style="color: #B452CD">5</span>, <span style="col
<p>
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
<div class="highlight" style="background: #eeeedd"><pre style="font-size: 80%; line-height: 125%"><span></span><span style="color: #228B22"># optional</span>
<div class="highlight" style="background: #eeeedd"><pre style="font-size: 80%; line-height: 125%;"><span></span><span style="color: #228B22"># optional</span>
<span style="color: #228B22"># visual representation of grid search</span>
<span style="color: #228B22"># uses seaborn heatmap, could probably do this in matplotlib</span>
<span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">seaborn</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">sns</span>
@@ -1701,12 +1702,12 @@ plt.show()
<section>
<h2 id="___sec29">The Breast Cancer Data, now with Keras </h2>
<h2 id="the-breast-cancer-data-now-with-keras">The Breast Cancer Data, now with Keras </h2>
<p>
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
<div class="highlight" style="background: #eeeedd"><pre style="font-size: 80%; line-height: 125%"><span></span><span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">tensorflow</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">tf</span>
<div class="highlight" style="background: #eeeedd"><pre style="font-size: 80%; line-height: 125%;"><span></span><span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">tensorflow</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">tf</span>
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">tensorflow.keras.layers</span> <span style="color: #8B008B; font-weight: bold">import</span> Input
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">tensorflow.keras.models</span> <span style="color: #8B008B; font-weight: bold">import</span> Sequential <span style="color: #228B22">#This allows appending layers to existing models</span>
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">tensorflow.keras.layers</span> <span style="color: #8B008B; font-weight: bold">import</span> Dense <span style="color: #228B22">#This allows defining the characteristics of a particular layer</span>
@@ -1876,7 +1877,7 @@ plot_data(eta,n_neuron,Test_accuracy, <span style="color: #CD5555">&#39;testing&
<section>
<h2 id="___sec30">Fine-tuning neural network hyperparameters </h2>
<h2 id="fine-tuning-neural-network-hyperparameters">Fine-tuning neural network hyperparameters </h2>
<p>
The flexibility of neural networks is also one of their main
@@ -1904,7 +1905,7 @@ will only be able to explore a tiny part of the hyperparameter space.
<section>
<h2 id="___sec31">Hidden layers </h2>
<h2 id="hidden-layers">Hidden layers </h2>
<p>
For many problems you can start with just one or two hidden layers and it will work just fine.
@@ -1924,7 +1925,7 @@ common to reuse parts of a pretrained state-of-the-art network that performs a s
<section>
<h2 id="___sec32">Which activation function should I use? </h2>
<h2 id="which-activation-function-should-i-use">Which activation function should I use? </h2>
<p>
The Back propagation algorithm we derived above works by going from
@@ -1953,7 +1954,7 @@ learn at widely different speeds
<section>
<h2 id="___sec33">Is the Logistic activation function (Sigmoid) our choice? </h2>
<h2 id="is-the-logistic-activation-function-sigmoid-our-choice">Is the Logistic activation function (Sigmoid) our choice? </h2>
<p>
Although this unfortunate behavior has been empirically observed for
@@ -1983,7 +1984,7 @@ better than the logistic function in deep networks).
<section>
<h2 id="___sec34">The derivative of the Logistic funtion </h2>
<h2 id="the-derivative-of-the-logistic-funtion">The derivative of the Logistic funtion </h2>
<p>
Looking at the logistic activation function, when inputs become large
@@ -2019,7 +2020,7 @@ fast to compute).
<section>
<h2 id="___sec35">The RELU function family </h2>
<h2 id="the-relu-function-family">The RELU function family </h2>
<p>
The ReLU activation function suffers from a problem known as the dying
@@ -2048,7 +2049,7 @@ $$
<section>
<h2 id="___sec36">Which activation function should we use? </h2>
<h2 id="which-activation-function-should-we-use">Which activation function should we use? </h2>
<p>
In general it seems that the ELU activation function is better than
@@ -2068,7 +2069,7 @@ bootstrap to evaluate other activation functions.
<section>
<h2 id="___sec37">More on activation functions, output layers </h2>
<h2 id="more-on-activation-functions-output-layers">More on activation functions, output layers </h2>
<p>
In most cases you can use the ReLU activation function in the hidden layers (or one of its variants).
@@ -2087,7 +2088,7 @@ It is a bit faster to compute than other activation functions, and the gradient
<section>
<h2 id="___sec38">Batch Normalization </h2>
<h2 id="batch-normalization">Batch Normalization </h2>
<p>
Batch Normalization
@@ -2106,7 +2107,7 @@ mini-batch, from this the name batch normalization.
<section>
<h2 id="___sec39">Dropout </h2>
<h2 id="dropout">Dropout </h2>
<p>
It is a fairly simple algorithm: at every training step, every neuron (including the input neurons but
@@ -2121,7 +2122,7 @@ hyperparameter \( p \) is called the dropout rate, and it is typically set to 50
<section>
<h2 id="___sec40">Gradient Clipping </h2>
<h2 id="gradient-clipping">Gradient Clipping </h2>
<p>
A popular technique to lessen the exploding gradients problem is to simply clip the gradients during
@@ -2138,7 +2139,7 @@ Normalization is preferred.
<section>
<h2 id="___sec41">A very nice website on Neural Networks </h2>
<h2 id="a-very-nice-website-on-neural-networks">A very nice website on Neural Networks </h2>
<p>
You may find this <a href="https://playground.tensorflow.org/#activation=tanh&batchSize=10&dataset=circle&regDataset=reg-plane&learningRate=0.03&regularizationRate=0&noise=0&networkShape=4,2&seed=0.29243&showTestData=false&discretize=false&percTrainData=50&x=true&y=true&xTimesY=false&xSquared=false&ySquared=false&cosX=false&sinX=false&cosY=false&sinY=false&collectStats=false&problem=classification&initZero=false&hideText=false" target="_blank">website</a> very useful. Thx a million to Ghadi for sharing.
@@ -2146,7 +2147,7 @@ You may find this <a href="https://playground.tensorflow.org/#activation=tanh&ba
<section>
<h2 id="___sec42">A top-down perspective on Neural networks </h2>
<h2 id="a-top-down-perspective-on-neural-networks">A top-down perspective on Neural networks </h2>
<p>
The first thing we would like to do is divide the data into two or three
@@ -2189,7 +2190,7 @@ supervised learning.
<section>
<h2 id="___sec43">Limitations of supervised learning with deep networks </h2>
<h2 id="limitations-of-supervised-learning-with-deep-networks">Limitations of supervised learning with deep networks </h2>
<p>
Like all statistical methods, supervised learning using neural
@@ -2216,7 +2217,7 @@ Some of these remarks are particular to DNNs, others are shared by all supervise
<section>
<h2 id="___sec44">Convolutional Neural Networks (recognizing images) </h2>
<h2 id="convolutional-neural-networks-recognizing-images">Convolutional Neural Networks (recognizing images) </h2>
<p>
Convolutional neural networks (CNNs) were developed during the last
@@ -2255,7 +2256,7 @@ Another good read is the article here <a href="https://arxiv.org/pdf/1603.07285.
<section>
<h2 id="___sec45">Regular NNs don&#8217;t scale well to full images </h2>
<h2 id="regular-nns-don-t-scale-well-to-full-images">Regular NNs don&#8217;t scale well to full images </h2>
<p>
As an example, consider
@@ -2283,7 +2284,7 @@ would quickly lead to possible overfitting.
<section>
<h2 id="___sec46">3D volumes of neurons </h2>
<h2 id="3d-volumes-of-neurons">3D volumes of neurons </h2>
<p>
Convolutional Neural Networks take advantage of the fact that the
@@ -2323,7 +2324,7 @@ dimension.
<section>
<h2 id="___sec47">Layers used to build CNNs </h2>
<h2 id="layers-used-to-build-cnns">Layers used to build CNNs </h2>
<p>
A simple CNN is a sequence of layers, and every layer of a CNN
@@ -2347,7 +2348,7 @@ A simple CNN for image classification could have the architecture:
<section>
<h2 id="___sec48">Transforming images </h2>
<h2 id="transforming-images">Transforming images </h2>
<p>
CNNs transform the original image layer by layer from the original
@@ -2366,7 +2367,7 @@ are consistent with the labels in the training set for each image.
<section>
<h2 id="___sec49">CNNs in brief </h2>
<h2 id="cnns-in-brief">CNNs in brief </h2>
<p>
In summary:
@@ -2388,7 +2389,7 @@ and the slides of <a href="http://cs231n.github.io/convolutional-networks/" targ
<section>
<h2 id="___sec50">CNNs in more detail, building convolutional neural networks in Tensorflow and Keras </h2>
<h2 id="cnns-in-more-detail-building-convolutional-neural-networks-in-tensorflow-and-keras">CNNs in more detail, building convolutional neural networks in Tensorflow and Keras </h2>
<p>
As discussed above, CNNs are neural networks built from the assumption that the inputs
@@ -2404,7 +2405,7 @@ matrices, typically 1 for each color dimension (Red, Green, Blue).
<section>
<h2 id="___sec51">Setting it up </h2>
<h2 id="setting-it-up">Setting it up </h2>
<p>
It means that to represent the entire
@@ -2419,7 +2420,7 @@ $$
<section>
<h2 id="___sec52">The MNIST dataset again </h2>
<h2 id="the-mnist-dataset-again">The MNIST dataset again </h2>
<p>
The MNIST dataset consists of grayscale images with a pixel size of
@@ -2437,7 +2438,7 @@ single neuron in the first hidden layer.
<section>
<h2 id="___sec53">Strong correlations </h2>
<h2 id="strong-correlations">Strong correlations </h2>
<p>
Images typically have strong local correlations, meaning that a small
@@ -2455,7 +2456,7 @@ fixed, and known as a <a href="https://en.wikipedia.org/wiki/Receptive_field" ta
<section>
<h2 id="___sec54">Layers of a CNN </h2>
<h2 id="layers-of-a-cnn">Layers of a CNN </h2>
The layers of a convolutional neural network arrange neurons in 3D: width, height and depth.
The input image is typically a square matrix of depth 3.
@@ -2477,7 +2478,7 @@ layer.
<section>
<h2 id="___sec55">Systematic reduction </h2>
<h2 id="systematic-reduction">Systematic reduction </h2>
<p>
By systematically reducing the size of the input volume, through
@@ -2492,11 +2493,11 @@ classification.
<section>
<h2 id="___sec56">Prerequisites: Collect and pre-process data </h2>
<h2 id="prerequisites-collect-and-pre-process-data">Prerequisites: Collect and pre-process data </h2>
<p>
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
<div class="highlight" style="background: #eeeedd"><pre style="font-size: 80%; line-height: 125%"><span></span><span style="color: #228B22"># import necessary packages</span>
<div class="highlight" style="background: #eeeedd"><pre style="font-size: 80%; line-height: 125%;"><span></span><span style="color: #228B22"># import necessary packages</span>
<span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">numpy</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">np</span>
<span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">matplotlib.pyplot</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">plt</span>
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">sklearn</span> <span style="color: #8B008B; font-weight: bold">import</span> datasets
@@ -2541,11 +2542,11 @@ plt.show()
<section>
<h2 id="___sec57">Importing Keras and Tensorflow </h2>
<h2 id="importing-keras-and-tensorflow">Importing Keras and Tensorflow </h2>
<p>
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
<div class="highlight" style="background: #eeeedd"><pre style="font-size: 80%; line-height: 125%"><span></span><span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">tensorflow.keras</span> <span style="color: #8B008B; font-weight: bold">import</span> datasets, layers, models
<div class="highlight" style="background: #eeeedd"><pre style="font-size: 80%; line-height: 125%;"><span></span><span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">tensorflow.keras</span> <span style="color: #8B008B; font-weight: bold">import</span> datasets, layers, models
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">tensorflow.keras.layers</span> <span style="color: #8B008B; font-weight: bold">import</span> Input
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">tensorflow.keras.models</span> <span style="color: #8B008B; font-weight: bold">import</span> Sequential <span style="color: #228B22">#This allows appending layers to existing models</span>
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">tensorflow.keras.layers</span> <span style="color: #8B008B; font-weight: bold">import</span> Dense <span style="color: #228B22">#This allows defining the characteristics of a particular layer</span>
@@ -2572,12 +2573,12 @@ X_train, X_test, Y_train, Y_test = train_test_split(inputs, labels, train_size=t
<section>
<h2 id="___sec58">Running with Keras </h2>
<h2 id="running-with-keras">Running with Keras </h2>
<p>
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
<div class="highlight" style="background: #eeeedd"><pre style="font-size: 80%; line-height: 125%"><span></span><span style="color: #8B008B; font-weight: bold">def</span> <span style="color: #008b45">create_convolutional_neural_network_keras</span>(input_shape, receptive_field,
<div class="highlight" style="background: #eeeedd"><pre style="font-size: 80%; line-height: 125%;"><span></span><span style="color: #8B008B; font-weight: bold">def</span> <span style="color: #008b45">create_convolutional_neural_network_keras</span>(input_shape, receptive_field,
n_filters, n_neurons_connected, n_categories,
eta, lmbd):
model = Sequential()
@@ -2608,12 +2609,12 @@ lmbd_vals = np.logspace(-<span style="color: #B452CD">5</span>, <span style="col
<section>
<h2 id="___sec59">Final part </h2>
<h2 id="final-part">Final part </h2>
<p>
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
<div class="highlight" style="background: #eeeedd"><pre style="font-size: 80%; line-height: 125%"><span></span>CNN_keras = np.zeros((<span style="color: #658b00">len</span>(eta_vals), <span style="color: #658b00">len</span>(lmbd_vals)), dtype=<span style="color: #658b00">object</span>)
<div class="highlight" style="background: #eeeedd"><pre style="font-size: 80%; line-height: 125%;"><span></span>CNN_keras = np.zeros((<span style="color: #658b00">len</span>(eta_vals), <span style="color: #658b00">len</span>(lmbd_vals)), dtype=<span style="color: #658b00">object</span>)
<span style="color: #8B008B; font-weight: bold">for</span> i, eta <span style="color: #8B008B">in</span> <span style="color: #658b00">enumerate</span>(eta_vals):
<span style="color: #8B008B; font-weight: bold">for</span> j, lmbd <span style="color: #8B008B">in</span> <span style="color: #658b00">enumerate</span>(lmbd_vals):
@@ -2634,12 +2635,12 @@ lmbd_vals = np.logspace(-<span style="color: #B452CD">5</span>, <span style="col
<section>
<h2 id="___sec60">Final visualization </h2>
<h2 id="final-visualization">Final visualization </h2>
<p>
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
<div class="highlight" style="background: #eeeedd"><pre style="font-size: 80%; line-height: 125%"><span></span><span style="color: #228B22"># visual representation of grid search</span>
<div class="highlight" style="background: #eeeedd"><pre style="font-size: 80%; line-height: 125%;"><span></span><span style="color: #228B22"># visual representation of grid search</span>
<span style="color: #228B22"># uses seaborn heatmap, could probably do this in matplotlib</span>
<span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">seaborn</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">sns</span>
@@ -2674,7 +2675,7 @@ plt.show()
<section>
<h2 id="___sec61">Fun links </h2>
<h2 id="fun-links">Fun links </h2>
<ol>
<p><li> <a href="https://arxiv.org/abs/1604.07316" target="_blank">Self-Driving cars using a convolutional neural network</a></li>
doc/pub/week41/html/week41-solarized.html
+221 -166
View File
@@ -1,19 +1,19 @@
<!--
Automatically generated HTML file from DocOnce source
(https://github.com/hplgit/doconce/)
(https://github.com/doconce/doconce/)
-->
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<meta name="generator" content="DocOnce: https://github.com/hplgit/doconce/" />
<meta name="generator" content="DocOnce: https://github.com/doconce/doconce/" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<meta name="description" content="Week 41 Tensor flow and Deep Learning, Convolutional Neural Networks">
<meta name="description" content="Week 41 Constructing a Neural Network code, Tensor flow and start Convolutional Neural Networks">
<title>Week 41 Tensor flow and Deep Learning, Convolutional Neural Networks</title>
<title>Week 41 Constructing a Neural Network code, Tensor flow and start Convolutional Neural Networks</title>
<link href="https://cdn.rawgit.com/hplgit/doconce/master/bundled/html_styles/style_solarized_box/css/solarized_light_code.css" rel="stylesheet" type="text/css" title="light"/>
<script src="https://cdn.rawgit.com/hplgit/doconce/master/bundled/html_styles/style_solarized_box/js/highlight.pack.js"></script>
<link href="https://cdn.rawgit.com/doconce/doconce/master/bundled/html_styles/style_solarized_box/css/solarized_light_code.css" rel="stylesheet" type="text/css" title="light"/>
<script src="https://cdn.rawgit.com/doconce/doconce/master/bundled/html_styles/style_solarized_box/js/highlight.pack.js"></script>
<script>hljs.initHighlightingOnLoad();</script>
<link href="https://thomasf.github.io/solarized-css/solarized-light.min.css" rel="stylesheet">
@@ -48,10 +48,10 @@ pre {
.alert-block > p, .alert-block > ul {margin-bottom:1em}
.alert li {margin-top: 1em}
.alert-block p+p {margin-top:5px}
.alert-notice { background-image: url(https://cdn.rawgit.com/hplgit/doconce/master/bundled/html_images/small_yellow_notice.png); }
.alert-summary { background-image:url(https://cdn.rawgit.com/hplgit/doconce/master/bundled/html_images/small_yellow_summary.png); }
.alert-warning { background-image: url(https://cdn.rawgit.com/hplgit/doconce/master/bundled/html_images/small_yellow_warning.png); }
.alert-question {background-image:url(https://cdn.rawgit.com/hplgit/doconce/master/bundled/html_images/small_yellow_question.png); }
.alert-notice { background-image: url(https://cdn.rawgit.com/doconce/doconce/master/bundled/html_images/small_yellow_notice.png); }
.alert-summary { background-image:url(https://cdn.rawgit.com/doconce/doconce/master/bundled/html_images/small_yellow_summary.png); }
.alert-warning { background-image: url(https://cdn.rawgit.com/doconce/doconce/master/bundled/html_images/small_yellow_warning.png); }
.alert-question {background-image:url(https://cdn.rawgit.com/doconce/doconce/master/bundled/html_images/small_yellow_question.png); }
div { text-align: justify; text-justify: inter-word; }
</style>
@@ -61,106 +61,160 @@ div { text-align: justify; text-justify: inter-word; }
<!-- tocinfo
{'highest level': 2,
'sections': [('Plan for week 41', 2, None, '___sec0'),
('Setting up the Back propagation algorithm', 2, None, '___sec1'),
'sections': [('Plan for week 41', 2, None, 'plan-for-week-41'),
('Setting up the Back propagation algorithm',
2,
None,
'setting-up-the-back-propagation-algorithm'),
('Setting up a Multi-layer perceptron model for classification',
2,
None,
'___sec2'),
('Defining the cost function', 2, None, '___sec3'),
('Example: binary classification problem', 2, None, '___sec4'),
('The Softmax function', 2, None, '___sec5'),
'setting-up-a-multi-layer-perceptron-model-for-classification'),
('Defining the cost function',
2,
None,
'defining-the-cost-function'),
('Example: binary classification problem',
2,
None,
'example-binary-classification-problem'),
('The Softmax function', 2, None, 'the-softmax-function'),
('Developing a code for doing neural networks with back '
'propagation',
2,
None,
'___sec6'),
('Collect and pre-process data', 2, None, '___sec7'),
('Train and test datasets', 2, None, '___sec8'),
('Define model and architecture', 2, None, '___sec9'),
('Layers', 2, None, '___sec10'),
('Weights and biases', 2, None, '___sec11'),
('Feed-forward pass', 2, None, '___sec12'),
('Matrix multiplications', 2, None, '___sec13'),
('Choose cost function and optimizer', 2, None, '___sec14'),
('Optimizing the cost function', 2, None, '___sec15'),
('Regularization', 2, None, '___sec16'),
('Matrix multiplication', 2, None, '___sec17'),
('Improving performance', 2, None, '___sec18'),
('Full object-oriented implementation', 2, None, '___sec19'),
('Evaluate model performance on test data', 2, None, '___sec20'),
('Adjust hyperparameters', 2, None, '___sec21'),
('Visualization', 2, None, '___sec22'),
('scikit-learn implementation', 2, None, '___sec23'),
('Visualization', 2, None, '___sec24'),
'developing-a-code-for-doing-neural-networks-with-back-propagation'),
('Collect and pre-process data',
2,
None,
'collect-and-pre-process-data'),
('Train and test datasets', 2, None, 'train-and-test-datasets'),
('Define model and architecture',
2,
None,
'define-model-and-architecture'),
('Layers', 2, None, 'layers'),
('Weights and biases', 2, None, 'weights-and-biases'),
('Feed-forward pass', 2, None, 'feed-forward-pass'),
('Matrix multiplications', 2, None, 'matrix-multiplications'),
('Choose cost function and optimizer',
2,
None,
'choose-cost-function-and-optimizer'),
('Optimizing the cost function',
2,
None,
'optimizing-the-cost-function'),
('Regularization', 2, None, 'regularization'),
('Matrix multiplication', 2, None, 'matrix-multiplication'),
('Improving performance', 2, None, 'improving-performance'),
('Full object-oriented implementation',
2,
None,
'full-object-oriented-implementation'),
('Evaluate model performance on test data',
2,
None,
'evaluate-model-performance-on-test-data'),
('Adjust hyperparameters', 2, None, 'adjust-hyperparameters'),
('Visualization', 2, None, 'visualization'),
('scikit-learn implementation',
2,
None,
'scikit-learn-implementation'),
('Visualization', 2, None, 'visualization'),
('Building neural networks in Tensorflow and Keras',
2,
None,
'___sec25'),
('Tensorflow', 2, None, '___sec26'),
('Using Keras', 2, None, '___sec27'),
('Collect and pre-process data', 2, None, '___sec28'),
('The Breast Cancer Data, now with Keras', 2, None, '___sec29'),
'building-neural-networks-in-tensorflow-and-keras'),
('Tensorflow', 2, None, 'tensorflow'),
('Using Keras', 2, None, 'using-keras'),
('Collect and pre-process data',
2,
None,
'collect-and-pre-process-data'),
('The Breast Cancer Data, now with Keras',
2,
None,
'the-breast-cancer-data-now-with-keras'),
('Fine-tuning neural network hyperparameters',
2,
None,
'___sec30'),
('Hidden layers', 2, None, '___sec31'),
('Which activation function should I use?', 2, None, '___sec32'),
'fine-tuning-neural-network-hyperparameters'),
('Hidden layers', 2, None, 'hidden-layers'),
('Which activation function should I use?',
2,
None,
'which-activation-function-should-i-use'),
('Is the Logistic activation function (Sigmoid) our choice?',
2,
None,
'___sec33'),
('The derivative of the Logistic funtion', 2, None, '___sec34'),
('The RELU function family', 2, None, '___sec35'),
('Which activation function should we use?', 2, None, '___sec36'),
'is-the-logistic-activation-function-sigmoid-our-choice'),
('The derivative of the Logistic funtion',
2,
None,
'the-derivative-of-the-logistic-funtion'),
('The RELU function family', 2, None, 'the-relu-function-family'),
('Which activation function should we use?',
2,
None,
'which-activation-function-should-we-use'),
('More on activation functions, output layers',
2,
None,
'___sec37'),
('Batch Normalization', 2, None, '___sec38'),
('Dropout', 2, None, '___sec39'),
('Gradient Clipping', 2, None, '___sec40'),
('A very nice website on Neural Networks', 2, None, '___sec41'),
'more-on-activation-functions-output-layers'),
('Batch Normalization', 2, None, 'batch-normalization'),
('Dropout', 2, None, 'dropout'),
('Gradient Clipping', 2, None, 'gradient-clipping'),
('A very nice website on Neural Networks',
2,
None,
'a-very-nice-website-on-neural-networks'),
('A top-down perspective on Neural networks',
2,
None,
'___sec42'),
'a-top-down-perspective-on-neural-networks'),
('Limitations of supervised learning with deep networks',
2,
None,
'___sec43'),
'limitations-of-supervised-learning-with-deep-networks'),
('Convolutional Neural Networks (recognizing images)',
2,
None,
'___sec44'),
'convolutional-neural-networks-recognizing-images'),
('Regular NNs don’t scale well to full images',
2,
None,
'___sec45'),
('3D volumes of neurons', 2, None, '___sec46'),
('Layers used to build CNNs', 2, None, '___sec47'),
('Transforming images', 2, None, '___sec48'),
('CNNs in brief', 2, None, '___sec49'),
'regular-nns-don-t-scale-well-to-full-images'),
('3D volumes of neurons', 2, None, '3d-volumes-of-neurons'),
('Layers used to build CNNs',
2,
None,
'layers-used-to-build-cnns'),
('Transforming images', 2, None, 'transforming-images'),
('CNNs in brief', 2, None, 'cnns-in-brief'),
('CNNs in more detail, building convolutional neural networks in '
'Tensorflow and Keras',
2,
None,
'___sec50'),
('Setting it up', 2, None, '___sec51'),
('The MNIST dataset again', 2, None, '___sec52'),
('Strong correlations', 2, None, '___sec53'),
('Layers of a CNN', 2, None, '___sec54'),
('Systematic reduction', 2, None, '___sec55'),
'cnns-in-more-detail-building-convolutional-neural-networks-in-tensorflow-and-keras'),
('Setting it up', 2, None, 'setting-it-up'),
('The MNIST dataset again', 2, None, 'the-mnist-dataset-again'),
('Strong correlations', 2, None, 'strong-correlations'),
('Layers of a CNN', 2, None, 'layers-of-a-cnn'),
('Systematic reduction', 2, None, 'systematic-reduction'),
('Prerequisites: Collect and pre-process data',
2,
None,
'___sec56'),
('Importing Keras and Tensorflow', 2, None, '___sec57'),
('Running with Keras', 2, None, '___sec58'),
('Final part', 2, None, '___sec59'),
('Final visualization', 2, None, '___sec60'),
('Fun links', 2, None, '___sec61')]}
'prerequisites-collect-and-pre-process-data'),
('Importing Keras and Tensorflow',
2,
None,
'importing-keras-and-tensorflow'),
('Running with Keras', 2, None, 'running-with-keras'),
('Final part', 2, None, 'final-part'),
('Final visualization', 2, None, 'final-visualization'),
('Fun links', 2, None, 'fun-links')]}
end of tocinfo -->
<body>
@@ -186,7 +240,7 @@ MathJax.Hub.Config({
<center><h1>Week 41 Tensor flow and Deep Learning, Convolutional Neural Networks</h1></center> <!-- document title -->
<center><h1>Week 41 Constructing a Neural Network code, Tensor flow and start Convolutional Neural Networks</h1></center> <!-- document title -->
<p>
<!-- author(s): Morten Hjorth-Jensen -->
@@ -202,25 +256,26 @@ MathJax.Hub.Config({
<center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
<br>
<p>
<center><h4>Oct 10, 2020</h4></center> <!-- date -->
<center><h4>Oct 9, 2021</h4></center> <!-- date -->
<br>
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec0">Plan for week 41 </h2>
<h2 id="plan-for-week-41">Plan for week 41 </h2>
<ul>
<li> Thursday: Building our own Feed-forward Neural Network. <a href="https://www.uio.no/studier/emner/matnat/fys/FYS-STK4155/h20/forelesningsvideoer/LectureOctober8.mp4?vrtx=view-as-webpage" target="_blank">Video of Lecture</a></li>
<li> Friday: Playing around with our own Feed-forward Neural Network and introduction to TensorFlow. Start convolutional Neural Networks (CNN). <a href="https://www.uio.no/studier/emner/matnat/fys/FYS-STK4155/h20/forelesningsvideoer/LectureOcotber9.mp4?vrtx=view-as-webpage" target="_blank">Video of Lecture</a></li>
<li> Thursday: Building our own Feed-forward Neural Network.</li>
<li> Friday: Playing around with our own Feed-forward Neural Network and introduction to TensorFlow. Start convolutional Neural Networks (CNN).</li>
</ul>
Reading suggestions for both days: <a href="https://github.com/CompPhysics/MachineLearning/blob/master/doc/T\
extbooks/TensorflowML.pdf" target="_blank">Aurelien Geron's chapters 10-11</a> and Hastie et al chapter 11.
extbooks/TensorflowML.pdf" target="_blank">Aurelien Geron's chapters 10-11</a>.
For amore in depth discussion on neural networks we recommend Goodfellow et al chapters 6 and 7. For CNNs, see Goodfellow et al chapter 9. chapter 11 and 12 on practicalities and applications
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec1">Setting up the Back propagation algorithm </h2>
<h2 id="setting-up-the-back-propagation-algorithm">Setting up the Back propagation algorithm </h2>
<p>
The four equations derived last week provide us with a way of computing the gradient of the cost function. Let us write this out in the form of an algorithm.
@@ -291,7 +346,7 @@ Here it is convenient to use stochastic gradient descent (see the examples below
<p>
<!-- !split -->
<h2 id="___sec2">Setting up a Multi-layer perceptron model for classification </h2>
<h2 id="setting-up-a-multi-layer-perceptron-model-for-classification">Setting up a Multi-layer perceptron model for classification </h2>
<p>
We are now gong to develop an example based on the MNIST data
@@ -329,7 +384,7 @@ of our network.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec3">Defining the cost function </h2>
<h2 id="defining-the-cost-function">Defining the cost function </h2>
<p>
Our cost function is given as (see the Logistic regression lectures)
@@ -389,7 +444,7 @@ The back propagation equations need now only a small change, namely the definiti
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec4">Example: binary classification problem </h2>
<h2 id="example-binary-classification-problem">Example: binary classification problem </h2>
<p>
As an example of the above, relevant for project 2 as well, let us consider a binary class. As discussed in our logistic regression lectures, we defined a cost function in terms of the parameters \( \beta \) as
@@ -437,7 +492,7 @@ In case we use another activation function than the logistic one, we need to eva
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec5">The Softmax function </h2>
<h2 id="the-softmax-function">The Softmax function </h2>
In case we employ the more general case given by the Softmax equation, we need to evaluate the derivative of the activation function with respect to the activation \( z_i^l \), that is we need
$$
\frac{\partial f(z_i^l)}{\partial w_{jk}^l} =
@@ -459,7 +514,7 @@ which in case of the simply binary model reduces to having \( i=j \).
<p>
<!-- !split -->
<h2 id="___sec6">Developing a code for doing neural networks with back propagation </h2>
<h2 id="developing-a-code-for-doing-neural-networks-with-back-propagation">Developing a code for doing neural networks with back propagation </h2>
<p>
One can identify a set of key steps when using neural networks to solve supervised learning problems:
@@ -475,7 +530,7 @@ One can identify a set of key steps when using neural networks to solve supervis
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec7">Collect and pre-process data </h2>
<h2 id="collect-and-pre-process-data">Collect and pre-process data </h2>
<p>
Here we will be using the MNIST dataset, which is readily available through the <b>scikit-learn</b>
@@ -524,7 +579,7 @@ images.
<p>
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
<div class="highlight" style="background: #eeeedd"><pre style="line-height: 125%"><span></span><span style="color: #228B22"># import necessary packages</span>
<div class="highlight" style="background: #eeeedd"><pre style="line-height: 125%;"><span></span><span style="color: #228B22"># import necessary packages</span>
<span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">numpy</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">np</span>
<span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">matplotlib.pyplot</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">plt</span>
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">sklearn</span> <span style="color: #8B008B; font-weight: bold">import</span> datasets
@@ -570,7 +625,7 @@ plt.show()
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec8">Train and test datasets </h2>
<h2 id="train-and-test-datasets">Train and test datasets </h2>
<p>
Performing analysis before partitioning the dataset is a major error, that can lead to incorrect conclusions.
@@ -588,7 +643,7 @@ collected from 12.00 to 24.00.
<p>
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
<div class="highlight" style="background: #eeeedd"><pre style="line-height: 125%"><span></span><span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">sklearn.model_selection</span> <span style="color: #8B008B; font-weight: bold">import</span> train_test_split
<div class="highlight" style="background: #eeeedd"><pre style="line-height: 125%;"><span></span><span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">sklearn.model_selection</span> <span style="color: #8B008B; font-weight: bold">import</span> train_test_split
<span style="color: #228B22"># one-liner from scikit-learn library</span>
train_size = <span style="color: #B452CD">0.8</span>
@@ -619,7 +674,7 @@ X_train, X_test, Y_train, Y_test = train_test_split(inputs, labels, train_size=t
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec9">Define model and architecture </h2>
<h2 id="define-model-and-architecture">Define model and architecture </h2>
<p>
Our simple feed-forward neural network will consist of an <em>input</em> layer, a single <em>hidden</em> layer and an <em>output</em> layer. The activation \( y \) of each neuron is a weighted sum of inputs, passed through an activation function. In case of the simple perceptron model we have
@@ -665,7 +720,7 @@ which is inspired by probability theory (see logistic regression) and was most c
<p>
<!-- !split -->
<h2 id="___sec10">Layers </h2>
<h2 id="layers">Layers </h2>
<ul>
<li> Input</li>
@@ -710,7 +765,7 @@ weights to the output layer.
<p>
<!-- !split -->
<h2 id="___sec11">Weights and biases </h2>
<h2 id="weights-and-biases">Weights and biases </h2>
<p>
Typically weights are initialized with small values distributed around zero, drawn from a uniform
@@ -727,7 +782,7 @@ The bias weights \( \hat{b} \) are often initialized to zero, but a small value
<p>
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
<div class="highlight" style="background: #eeeedd"><pre style="line-height: 125%"><span></span><span style="color: #228B22"># building our neural network</span>
<div class="highlight" style="background: #eeeedd"><pre style="line-height: 125%;"><span></span><span style="color: #228B22"># building our neural network</span>
n_inputs, n_features = X_train.shape
n_hidden_neurons = <span style="color: #B452CD">50</span>
@@ -746,7 +801,7 @@ output_bias = np.zeros(n_categories) + <span style="color: #B452CD">0.01</span>
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec12">Feed-forward pass </h2>
<h2 id="feed-forward-pass">Feed-forward pass </h2>
<p>
Denote \( F \) the number of features, \( H \) the number of hidden neurons and \( C \) the number of categories.
@@ -773,7 +828,7 @@ $$ a_{j}^{L} = \frac{\exp{(z_j^{L})}}
<p>
<!-- !split -->
<h2 id="___sec13">Matrix multiplications </h2>
<h2 id="matrix-multiplications">Matrix multiplications </h2>
<p>
Since our data has the dimensions \( X = (n_{inputs}, n_{features}) \) and our weights to the hidden
@@ -809,7 +864,7 @@ $$ output = softmax (\hat{z}^{L}) = (n_{inputs}, n_{categories}) .$$
<p>
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
<div class="highlight" style="background: #eeeedd"><pre style="line-height: 125%"><span></span><span style="color: #228B22"># setup the feed-forward pass, subscript h = hidden layer</span>
<div class="highlight" style="background: #eeeedd"><pre style="line-height: 125%;"><span></span><span style="color: #228B22"># setup the feed-forward pass, subscript h = hidden layer</span>
<span style="color: #8B008B; font-weight: bold">def</span> <span style="color: #008b45">sigmoid</span>(x):
<span style="color: #8B008B; font-weight: bold">return</span> <span style="color: #B452CD">1</span>/(<span style="color: #B452CD">1</span> + np.exp(-x))
@@ -848,7 +903,7 @@ predictions = predict(X_train)
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec14">Choose cost function and optimizer </h2>
<h2 id="choose-cost-function-and-optimizer">Choose cost function and optimizer </h2>
<p>
To measure how well our neural network is doing we need to introduce a cost function.
@@ -879,7 +934,7 @@ you got the correct label. The probability of category \( c \) is given by the s
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec15">Optimizing the cost function </h2>
<h2 id="optimizing-the-cost-function">Optimizing the cost function </h2>
<p>
The network is trained by finding the weights and biases that minimize the cost function. One of the most widely used classes of methods is <em>gradient descent</em> and its generalizations. The idea behind gradient descent
@@ -919,7 +974,7 @@ The various optmization methods, with codes and algorithms, are discussed in o
<p>
<!-- !split -->
<h2 id="___sec16">Regularization </h2>
<h2 id="regularization">Regularization </h2>
<p>
It is common to add an extra term to the cost function, proportional
@@ -952,7 +1007,7 @@ calculate the gradient efficently.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec17">Matrix multiplication </h2>
<h2 id="matrix-multiplication">Matrix multiplication </h2>
<p>
To more efficently train our network these equations are implemented using matrix operations.
@@ -994,7 +1049,7 @@ $$ \nabla b_{h} = \sum_{i=1}^{n_{inputs}} \delta_h = (n_{hidden}) .$$
<p>
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
<div class="highlight" style="background: #eeeedd"><pre style="line-height: 125%"><span></span><span style="color: #228B22"># to categorical turns our integer vector into a onehot representation</span>
<div class="highlight" style="background: #eeeedd"><pre style="line-height: 125%;"><span></span><span style="color: #228B22"># to categorical turns our integer vector into a onehot representation</span>
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">sklearn.metrics</span> <span style="color: #8B008B; font-weight: bold">import</span> accuracy_score
<span style="color: #228B22"># one-hot in numpy</span>
@@ -1066,7 +1121,7 @@ lmbd = <span style="color: #B452CD">0.01</span>
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec18">Improving performance </h2>
<h2 id="improving-performance">Improving performance </h2>
<p>
As we can see the network does not seem to be learning at all. It seems to be just guessing the label for each image.
@@ -1086,7 +1141,7 @@ Andrew Ng goes through some of these considerations in this <a href="https://you
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec19">Full object-oriented implementation </h2>
<h2 id="full-object-oriented-implementation">Full object-oriented implementation </h2>
<p>
It is very natural to think of the network as an object, with specific instances of the network
@@ -1095,7 +1150,7 @@ being realizations of this object with different hyperparameters. An implementat
<p>
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
<div class="highlight" style="background: #eeeedd"><pre style="line-height: 125%"><span></span><span style="color: #8B008B; font-weight: bold">class</span> <span style="color: #008b45; font-weight: bold">NeuralNetwork</span>:
<div class="highlight" style="background: #eeeedd"><pre style="line-height: 125%;"><span></span><span style="color: #8B008B; font-weight: bold">class</span> <span style="color: #008b45; font-weight: bold">NeuralNetwork</span>:
<span style="color: #8B008B; font-weight: bold">def</span> <span style="color: #008b45">__init__</span>(
<span style="color: #658b00">self</span>,
X_data,
@@ -1198,7 +1253,7 @@ being realizations of this object with different hyperparameters. An implementat
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec20">Evaluate model performance on test data </h2>
<h2 id="evaluate-model-performance-on-test-data">Evaluate model performance on test data </h2>
<p>
To measure the performance of our network we evaluate how well it does it data it has never seen before, i.e. the test data.
@@ -1213,7 +1268,7 @@ where \( I \) is the indicator function, \( 1 \) if \( \hat{y}_i = y_i \) and \(
<p>
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
<div class="highlight" style="background: #eeeedd"><pre style="line-height: 125%"><span></span>epochs = <span style="color: #B452CD">100</span>
<div class="highlight" style="background: #eeeedd"><pre style="line-height: 125%;"><span></span>epochs = <span style="color: #B452CD">100</span>
batch_size = <span style="color: #B452CD">100</span>
dnn = NeuralNetwork(X_train, Y_train_onehot, eta=eta, lmbd=lmbd, epochs=epochs, batch_size=batch_size,
@@ -1233,7 +1288,7 @@ test_predict = dnn.predict(X_test)
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec21">Adjust hyperparameters </h2>
<h2 id="adjust-hyperparameters">Adjust hyperparameters </h2>
<p>
We now perform a grid search to find the optimal hyperparameters for the network.
@@ -1242,7 +1297,7 @@ Note that we are only using 1 layer with 50 neurons, and human performance is es
<p>
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
<div class="highlight" style="background: #eeeedd"><pre style="line-height: 125%"><span></span>eta_vals = np.logspace(-<span style="color: #B452CD">5</span>, <span style="color: #B452CD">1</span>, <span style="color: #B452CD">7</span>)
<div class="highlight" style="background: #eeeedd"><pre style="line-height: 125%;"><span></span>eta_vals = np.logspace(-<span style="color: #B452CD">5</span>, <span style="color: #B452CD">1</span>, <span style="color: #B452CD">7</span>)
lmbd_vals = np.logspace(-<span style="color: #B452CD">5</span>, <span style="color: #B452CD">1</span>, <span style="color: #B452CD">7</span>)
<span style="color: #228B22"># store the models for later use</span>
DNN_numpy = np.zeros((<span style="color: #658b00">len</span>(eta_vals), <span style="color: #658b00">len</span>(lmbd_vals)), dtype=<span style="color: #658b00">object</span>)
@@ -1266,12 +1321,12 @@ DNN_numpy = np.zeros((<span style="color: #658b00">len</span>(eta_vals), <span s
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec22">Visualization </h2>
<h2 id="visualization">Visualization </h2>
<p>
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
<div class="highlight" style="background: #eeeedd"><pre style="line-height: 125%"><span></span><span style="color: #228B22"># visual representation of grid search</span>
<div class="highlight" style="background: #eeeedd"><pre style="line-height: 125%;"><span></span><span style="color: #228B22"># visual representation of grid search</span>
<span style="color: #228B22"># uses seaborn heatmap, you can also do this with matplotlib imshow</span>
<span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">seaborn</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">sns</span>
@@ -1308,7 +1363,7 @@ plt.show()
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec23">scikit-learn implementation </h2>
<h2 id="scikit-learn-implementation">scikit-learn implementation </h2>
<p>
<b>scikit-learn</b> focuses more
@@ -1327,7 +1382,7 @@ performance overall.
<p>
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
<div class="highlight" style="background: #eeeedd"><pre style="line-height: 125%"><span></span><span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">sklearn.neural_network</span> <span style="color: #8B008B; font-weight: bold">import</span> MLPClassifier
<div class="highlight" style="background: #eeeedd"><pre style="line-height: 125%;"><span></span><span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">sklearn.neural_network</span> <span style="color: #8B008B; font-weight: bold">import</span> MLPClassifier
<span style="color: #228B22"># store models for later use</span>
DNN_scikit = np.zeros((<span style="color: #658b00">len</span>(eta_vals), <span style="color: #658b00">len</span>(lmbd_vals)), dtype=<span style="color: #658b00">object</span>)
@@ -1347,11 +1402,11 @@ DNN_scikit = np.zeros((<span style="color: #658b00">len</span>(eta_vals), <span
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec24">Visualization </h2>
<h2 id="visualization">Visualization </h2>
<p>
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
<div class="highlight" style="background: #eeeedd"><pre style="line-height: 125%"><span></span><span style="color: #228B22"># optional</span>
<div class="highlight" style="background: #eeeedd"><pre style="line-height: 125%;"><span></span><span style="color: #228B22"># optional</span>
<span style="color: #228B22"># visual representation of grid search</span>
<span style="color: #228B22"># uses seaborn heatmap, could probably do this in matplotlib</span>
<span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">seaborn</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">sns</span>
@@ -1389,7 +1444,7 @@ plt.show()
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec25">Building neural networks in Tensorflow and Keras </h2>
<h2 id="building-neural-networks-in-tensorflow-and-keras">Building neural networks in Tensorflow and Keras </h2>
<p>
Now we want to build on the experience gained from our neural network implementation in NumPy and scikit-learn
@@ -1404,7 +1459,7 @@ NumPy arrays.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec26">Tensorflow </h2>
<h2 id="tensorflow">Tensorflow </h2>
<p>
Tensorflow is an open source library machine learning library
@@ -1437,7 +1492,7 @@ To install tensorflow on Unix/Linux systems, use pip as
<p>
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
<div class="highlight" style="background: #eeeedd"><pre style="line-height: 125%"><span></span>pip3 install tensorflow
<div class="highlight" style="background: #eeeedd"><pre style="line-height: 125%;"><span></span>pip3 install tensorflow
</pre></div>
<p>
and/or if you use <b>anaconda</b>, just write (or install from the graphical user interface)
@@ -1445,7 +1500,7 @@ and/or if you use <b>anaconda</b>, just write (or install from the graphical use
<p>
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
<div class="highlight" style="background: #eeeedd"><pre style="line-height: 125%"><span></span>conda create -n tf tensorflow
<div class="highlight" style="background: #eeeedd"><pre style="line-height: 125%;"><span></span>conda create -n tf tensorflow
conda activate tf
</pre></div>
<p>
@@ -1453,13 +1508,13 @@ To install the current release of GPU TensorFlow
<p>
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
<div class="highlight" style="background: #eeeedd"><pre style="line-height: 125%"><span></span>conda create -n tf-gpu tensorflow-gpu
<div class="highlight" style="background: #eeeedd"><pre style="line-height: 125%;"><span></span>conda create -n tf-gpu tensorflow-gpu
conda activate tf-gpu
</pre></div>
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec27">Using Keras </h2>
<h2 id="using-keras">Using Keras </h2>
<p>
Keras is a high level <a href="https://en.wikipedia.org/wiki/Application_programming_interface" target="_blank">neural network</a>
@@ -1468,7 +1523,7 @@ If you have Anaconda installed you may run the following command
<p>
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
<div class="highlight" style="background: #eeeedd"><pre style="line-height: 125%"><span></span>conda install keras
<div class="highlight" style="background: #eeeedd"><pre style="line-height: 125%;"><span></span>conda install keras
</pre></div>
<p>
You can look up the <a href="https://keras.io/" target="_blank">instructions here</a> for more information.
@@ -1479,7 +1534,7 @@ We will to a large extent use <b>keras</b> in this course.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec28">Collect and pre-process data </h2>
<h2 id="collect-and-pre-process-data">Collect and pre-process data </h2>
<p>
Let us look again at the MINST data set.
@@ -1487,7 +1542,7 @@ Let us look again at the MINST data set.
<p>
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
<div class="highlight" style="background: #eeeedd"><pre style="line-height: 125%"><span></span><span style="color: #228B22"># import necessary packages</span>
<div class="highlight" style="background: #eeeedd"><pre style="line-height: 125%;"><span></span><span style="color: #228B22"># import necessary packages</span>
<span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">numpy</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">np</span>
<span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">matplotlib.pyplot</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">plt</span>
<span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">tensorflow</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">tf</span>
@@ -1534,7 +1589,7 @@ plt.show()
<p>
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
<div class="highlight" style="background: #eeeedd"><pre style="line-height: 125%"><span></span><span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">tensorflow.keras.layers</span> <span style="color: #8B008B; font-weight: bold">import</span> Input
<div class="highlight" style="background: #eeeedd"><pre style="line-height: 125%;"><span></span><span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">tensorflow.keras.layers</span> <span style="color: #8B008B; font-weight: bold">import</span> Input
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">tensorflow.keras.models</span> <span style="color: #8B008B; font-weight: bold">import</span> Sequential <span style="color: #228B22">#This allows appending layers to existing models</span>
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">tensorflow.keras.layers</span> <span style="color: #8B008B; font-weight: bold">import</span> Dense <span style="color: #228B22">#This allows defining the characteristics of a particular layer</span>
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">tensorflow.keras</span> <span style="color: #8B008B; font-weight: bold">import</span> optimizers <span style="color: #228B22">#This allows using whichever optimiser we want (sgd,adam,RMSprop)</span>
@@ -1555,7 +1610,7 @@ X_train, X_test, Y_train, Y_test = train_test_split(inputs, labels, train_size=t
<p>
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
<div class="highlight" style="background: #eeeedd"><pre style="line-height: 125%"><span></span>epochs = <span style="color: #B452CD">100</span>
<div class="highlight" style="background: #eeeedd"><pre style="line-height: 125%;"><span></span>epochs = <span style="color: #B452CD">100</span>
batch_size = <span style="color: #B452CD">100</span>
n_neurons_layer1 = <span style="color: #B452CD">100</span>
n_neurons_layer2 = <span style="color: #B452CD">50</span>
@@ -1576,7 +1631,7 @@ lmbd_vals = np.logspace(-<span style="color: #B452CD">5</span>, <span style="col
<p>
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
<div class="highlight" style="background: #eeeedd"><pre style="line-height: 125%"><span></span>DNN_keras = np.zeros((<span style="color: #658b00">len</span>(eta_vals), <span style="color: #658b00">len</span>(lmbd_vals)), dtype=<span style="color: #658b00">object</span>)
<div class="highlight" style="background: #eeeedd"><pre style="line-height: 125%;"><span></span>DNN_keras = np.zeros((<span style="color: #658b00">len</span>(eta_vals), <span style="color: #658b00">len</span>(lmbd_vals)), dtype=<span style="color: #658b00">object</span>)
<span style="color: #8B008B; font-weight: bold">for</span> i, eta <span style="color: #8B008B">in</span> <span style="color: #658b00">enumerate</span>(eta_vals):
<span style="color: #8B008B; font-weight: bold">for</span> j, lmbd <span style="color: #8B008B">in</span> <span style="color: #658b00">enumerate</span>(lmbd_vals):
@@ -1595,7 +1650,7 @@ lmbd_vals = np.logspace(-<span style="color: #B452CD">5</span>, <span style="col
<p>
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
<div class="highlight" style="background: #eeeedd"><pre style="line-height: 125%"><span></span><span style="color: #228B22"># optional</span>
<div class="highlight" style="background: #eeeedd"><pre style="line-height: 125%;"><span></span><span style="color: #228B22"># optional</span>
<span style="color: #228B22"># visual representation of grid search</span>
<span style="color: #228B22"># uses seaborn heatmap, could probably do this in matplotlib</span>
<span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">seaborn</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">sns</span>
@@ -1630,12 +1685,12 @@ plt.show()
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec29">The Breast Cancer Data, now with Keras </h2>
<h2 id="the-breast-cancer-data-now-with-keras">The Breast Cancer Data, now with Keras </h2>
<p>
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
<div class="highlight" style="background: #eeeedd"><pre style="line-height: 125%"><span></span><span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">tensorflow</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">tf</span>
<div class="highlight" style="background: #eeeedd"><pre style="line-height: 125%;"><span></span><span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">tensorflow</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">tf</span>
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">tensorflow.keras.layers</span> <span style="color: #8B008B; font-weight: bold">import</span> Input
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">tensorflow.keras.models</span> <span style="color: #8B008B; font-weight: bold">import</span> Sequential <span style="color: #228B22">#This allows appending layers to existing models</span>
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">tensorflow.keras.layers</span> <span style="color: #8B008B; font-weight: bold">import</span> Dense <span style="color: #228B22">#This allows defining the characteristics of a particular layer</span>
@@ -1804,7 +1859,7 @@ plot_data(eta,n_neuron,Test_accuracy, <span style="color: #CD5555">&#39;testing&
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec30">Fine-tuning neural network hyperparameters </h2>
<h2 id="fine-tuning-neural-network-hyperparameters">Fine-tuning neural network hyperparameters </h2>
<p>
The flexibility of neural networks is also one of their main
@@ -1830,7 +1885,7 @@ will only be able to explore a tiny part of the hyperparameter space.
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec31">Hidden layers </h2>
<h2 id="hidden-layers">Hidden layers </h2>
<p>
For many problems you can start with just one or two hidden layers and it will work just fine.
@@ -1850,7 +1905,7 @@ common to reuse parts of a pretrained state-of-the-art network that performs a s
<p>
<!-- !split -->
<h2 id="___sec32">Which activation function should I use? </h2>
<h2 id="which-activation-function-should-i-use">Which activation function should I use? </h2>
<p>
The Back propagation algorithm we derived above works by going from
@@ -1879,7 +1934,7 @@ learn at widely different speeds
<p>
<!-- !split -->
<h2 id="___sec33">Is the Logistic activation function (Sigmoid) our choice? </h2>
<h2 id="is-the-logistic-activation-function-sigmoid-our-choice">Is the Logistic activation function (Sigmoid) our choice? </h2>
<p>
Although this unfortunate behavior has been empirically observed for
@@ -1909,7 +1964,7 @@ better than the logistic function in deep networks).
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec34">The derivative of the Logistic funtion </h2>
<h2 id="the-derivative-of-the-logistic-funtion">The derivative of the Logistic funtion </h2>
<p>
Looking at the logistic activation function, when inputs become large
@@ -1945,7 +2000,7 @@ fast to compute).
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec35">The RELU function family </h2>
<h2 id="the-relu-function-family">The RELU function family </h2>
<p>
The ReLU activation function suffers from a problem known as the dying
@@ -1972,7 +2027,7 @@ $$
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec36">Which activation function should we use? </h2>
<h2 id="which-activation-function-should-we-use">Which activation function should we use? </h2>
<p>
In general it seems that the ELU activation function is better than
@@ -1992,7 +2047,7 @@ bootstrap to evaluate other activation functions.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec37">More on activation functions, output layers </h2>
<h2 id="more-on-activation-functions-output-layers">More on activation functions, output layers </h2>
<p>
In most cases you can use the ReLU activation function in the hidden layers (or one of its variants).
@@ -2010,7 +2065,7 @@ It is a bit faster to compute than other activation functions, and the gradient
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec38">Batch Normalization </h2>
<h2 id="batch-normalization">Batch Normalization </h2>
<p>
Batch Normalization
@@ -2029,7 +2084,7 @@ mini-batch, from this the name batch normalization.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec39">Dropout </h2>
<h2 id="dropout">Dropout </h2>
<p>
It is a fairly simple algorithm: at every training step, every neuron (including the input neurons but
@@ -2044,7 +2099,7 @@ hyperparameter \( p \) is called the dropout rate, and it is typically set to 50
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec40">Gradient Clipping </h2>
<h2 id="gradient-clipping">Gradient Clipping </h2>
<p>
A popular technique to lessen the exploding gradients problem is to simply clip the gradients during
@@ -2061,7 +2116,7 @@ Normalization is preferred.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec41">A very nice website on Neural Networks </h2>
<h2 id="a-very-nice-website-on-neural-networks">A very nice website on Neural Networks </h2>
<p>
You may find this <a href="https://playground.tensorflow.org/#activation=tanh&batchSize=10&dataset=circle&regDataset=reg-plane&learningRate=0.03&regularizationRate=0&noise=0&networkShape=4,2&seed=0.29243&showTestData=false&discretize=false&percTrainData=50&x=true&y=true&xTimesY=false&xSquared=false&ySquared=false&cosX=false&sinX=false&cosY=false&sinY=false&collectStats=false&problem=classification&initZero=false&hideText=false" target="_blank">website</a> very useful. Thx a million to Ghadi for sharing.
@@ -2069,7 +2124,7 @@ You may find this <a href="https://playground.tensorflow.org/#activation=tanh&ba
<p>
<!-- !split -->
<h2 id="___sec42">A top-down perspective on Neural networks </h2>
<h2 id="a-top-down-perspective-on-neural-networks">A top-down perspective on Neural networks </h2>
<p>
The first thing we would like to do is divide the data into two or three
@@ -2111,7 +2166,7 @@ supervised learning.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec43">Limitations of supervised learning with deep networks </h2>
<h2 id="limitations-of-supervised-learning-with-deep-networks">Limitations of supervised learning with deep networks </h2>
<p>
Like all statistical methods, supervised learning using neural
@@ -2137,7 +2192,7 @@ Some of these remarks are particular to DNNs, others are shared by all supervise
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec44">Convolutional Neural Networks (recognizing images) </h2>
<h2 id="convolutional-neural-networks-recognizing-images">Convolutional Neural Networks (recognizing images) </h2>
<p>
Convolutional neural networks (CNNs) were developed during the last
@@ -2176,7 +2231,7 @@ Another good read is the article here <a href="https://arxiv.org/pdf/1603.07285.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec45">Regular NNs don&#8217;t scale well to full images </h2>
<h2 id="regular-nns-don-t-scale-well-to-full-images">Regular NNs don&#8217;t scale well to full images </h2>
<p>
As an example, consider
@@ -2204,7 +2259,7 @@ would quickly lead to possible overfitting.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec46">3D volumes of neurons </h2>
<h2 id="3d-volumes-of-neurons">3D volumes of neurons </h2>
<p>
Convolutional Neural Networks take advantage of the fact that the
@@ -2244,7 +2299,7 @@ dimension.
<p>
<!-- !split -->
<h2 id="___sec47">Layers used to build CNNs </h2>
<h2 id="layers-used-to-build-cnns">Layers used to build CNNs </h2>
<p>
A simple CNN is a sequence of layers, and every layer of a CNN
@@ -2267,7 +2322,7 @@ A simple CNN for image classification could have the architecture:
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec48">Transforming images </h2>
<h2 id="transforming-images">Transforming images </h2>
<p>
CNNs transform the original image layer by layer from the original
@@ -2286,7 +2341,7 @@ are consistent with the labels in the training set for each image.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec49">CNNs in brief </h2>
<h2 id="cnns-in-brief">CNNs in brief </h2>
<p>
In summary:
@@ -2307,7 +2362,7 @@ and the slides of <a href="http://cs231n.github.io/convolutional-networks/" targ
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec50">CNNs in more detail, building convolutional neural networks in Tensorflow and Keras </h2>
<h2 id="cnns-in-more-detail-building-convolutional-neural-networks-in-tensorflow-and-keras">CNNs in more detail, building convolutional neural networks in Tensorflow and Keras </h2>
<p>
As discussed above, CNNs are neural networks built from the assumption that the inputs
@@ -2323,7 +2378,7 @@ matrices, typically 1 for each color dimension (Red, Green, Blue).
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec51">Setting it up </h2>
<h2 id="setting-it-up">Setting it up </h2>
<p>
It means that to represent the entire
@@ -2335,7 +2390,7 @@ $$
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec52">The MNIST dataset again </h2>
<h2 id="the-mnist-dataset-again">The MNIST dataset again </h2>
<p>
The MNIST dataset consists of grayscale images with a pixel size of
@@ -2353,7 +2408,7 @@ single neuron in the first hidden layer.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec53">Strong correlations </h2>
<h2 id="strong-correlations">Strong correlations </h2>
<p>
Images typically have strong local correlations, meaning that a small
@@ -2371,7 +2426,7 @@ fixed, and known as a <a href="https://en.wikipedia.org/wiki/Receptive_field" ta
<p>
<!-- !split -->
<h2 id="___sec54">Layers of a CNN </h2>
<h2 id="layers-of-a-cnn">Layers of a CNN </h2>
The layers of a convolutional neural network arrange neurons in 3D: width, height and depth.
The input image is typically a square matrix of depth 3.
@@ -2393,7 +2448,7 @@ layer.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec55">Systematic reduction </h2>
<h2 id="systematic-reduction">Systematic reduction </h2>
<p>
By systematically reducing the size of the input volume, through
@@ -2408,11 +2463,11 @@ classification.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec56">Prerequisites: Collect and pre-process data </h2>
<h2 id="prerequisites-collect-and-pre-process-data">Prerequisites: Collect and pre-process data </h2>
<p>
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
<div class="highlight" style="background: #eeeedd"><pre style="line-height: 125%"><span></span><span style="color: #228B22"># import necessary packages</span>
<div class="highlight" style="background: #eeeedd"><pre style="line-height: 125%;"><span></span><span style="color: #228B22"># import necessary packages</span>
<span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">numpy</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">np</span>
<span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">matplotlib.pyplot</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">plt</span>
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">sklearn</span> <span style="color: #8B008B; font-weight: bold">import</span> datasets
@@ -2456,11 +2511,11 @@ plt.show()
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec57">Importing Keras and Tensorflow </h2>
<h2 id="importing-keras-and-tensorflow">Importing Keras and Tensorflow </h2>
<p>
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
<div class="highlight" style="background: #eeeedd"><pre style="line-height: 125%"><span></span><span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">tensorflow.keras</span> <span style="color: #8B008B; font-weight: bold">import</span> datasets, layers, models
<div class="highlight" style="background: #eeeedd"><pre style="line-height: 125%;"><span></span><span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">tensorflow.keras</span> <span style="color: #8B008B; font-weight: bold">import</span> datasets, layers, models
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">tensorflow.keras.layers</span> <span style="color: #8B008B; font-weight: bold">import</span> Input
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">tensorflow.keras.models</span> <span style="color: #8B008B; font-weight: bold">import</span> Sequential <span style="color: #228B22">#This allows appending layers to existing models</span>
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">tensorflow.keras.layers</span> <span style="color: #8B008B; font-weight: bold">import</span> Dense <span style="color: #228B22">#This allows defining the characteristics of a particular layer</span>
@@ -2486,12 +2541,12 @@ X_train, X_test, Y_train, Y_test = train_test_split(inputs, labels, train_size=t
<p>
<!-- !split -->
<h2 id="___sec58">Running with Keras </h2>
<h2 id="running-with-keras">Running with Keras </h2>
<p>
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
<div class="highlight" style="background: #eeeedd"><pre style="line-height: 125%"><span></span><span style="color: #8B008B; font-weight: bold">def</span> <span style="color: #008b45">create_convolutional_neural_network_keras</span>(input_shape, receptive_field,
<div class="highlight" style="background: #eeeedd"><pre style="line-height: 125%;"><span></span><span style="color: #8B008B; font-weight: bold">def</span> <span style="color: #008b45">create_convolutional_neural_network_keras</span>(input_shape, receptive_field,
n_filters, n_neurons_connected, n_categories,
eta, lmbd):
model = Sequential()
@@ -2521,12 +2576,12 @@ lmbd_vals = np.logspace(-<span style="color: #B452CD">5</span>, <span style="col
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec59">Final part </h2>
<h2 id="final-part">Final part </h2>
<p>
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
<div class="highlight" style="background: #eeeedd"><pre style="line-height: 125%"><span></span>CNN_keras = np.zeros((<span style="color: #658b00">len</span>(eta_vals), <span style="color: #658b00">len</span>(lmbd_vals)), dtype=<span style="color: #658b00">object</span>)
<div class="highlight" style="background: #eeeedd"><pre style="line-height: 125%;"><span></span>CNN_keras = np.zeros((<span style="color: #658b00">len</span>(eta_vals), <span style="color: #658b00">len</span>(lmbd_vals)), dtype=<span style="color: #658b00">object</span>)
<span style="color: #8B008B; font-weight: bold">for</span> i, eta <span style="color: #8B008B">in</span> <span style="color: #658b00">enumerate</span>(eta_vals):
<span style="color: #8B008B; font-weight: bold">for</span> j, lmbd <span style="color: #8B008B">in</span> <span style="color: #658b00">enumerate</span>(lmbd_vals):
@@ -2546,12 +2601,12 @@ lmbd_vals = np.logspace(-<span style="color: #B452CD">5</span>, <span style="col
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec60">Final visualization </h2>
<h2 id="final-visualization">Final visualization </h2>
<p>
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
<div class="highlight" style="background: #eeeedd"><pre style="line-height: 125%"><span></span><span style="color: #228B22"># visual representation of grid search</span>
<div class="highlight" style="background: #eeeedd"><pre style="line-height: 125%;"><span></span><span style="color: #228B22"># visual representation of grid search</span>
<span style="color: #228B22"># uses seaborn heatmap, could probably do this in matplotlib</span>
<span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">seaborn</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">sns</span>
@@ -2585,7 +2640,7 @@ plt.show()
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec61">Fun links </h2>
<h2 id="fun-links">Fun links </h2>
<ol>
<li> <a href="https://arxiv.org/abs/1604.07316" target="_blank">Self-Driving cars using a convolutional neural network</a></li>
@@ -2597,7 +2652,7 @@ plt.show()
<center style="font-size:80%">
<!-- copyright --> &copy; 1999-2020, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license
<!-- copyright --> &copy; 1999-2021, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license
</center>
doc/pub/week41/html/week41.html
+219 -164
View File
@@ -1,15 +1,15 @@
<!--
Automatically generated HTML file from DocOnce source
(https://github.com/hplgit/doconce/)
(https://github.com/doconce/doconce/)
-->
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<meta name="generator" content="DocOnce: https://github.com/hplgit/doconce/" />
<meta name="generator" content="DocOnce: https://github.com/doconce/doconce/" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<meta name="description" content="Week 41 Tensor flow and Deep Learning, Convolutional Neural Networks">
<meta name="description" content="Week 41 Constructing a Neural Network code, Tensor flow and start Convolutional Neural Networks">
<title>Week 41 Tensor flow and Deep Learning, Convolutional Neural Networks</title>
<title>Week 41 Constructing a Neural Network code, Tensor flow and start Convolutional Neural Networks</title>
<style type="text/css">
@@ -53,10 +53,10 @@ hr.figure { border: 0; width: 80%; border-bottom: 1px solid #aaa}
.alert-block > p, .alert-block > ul {margin-bottom:1em}
.alert li {margin-top: 1em}
.alert-block p+p {margin-top:5px}
.alert-notice { background-image: url(https://cdn.rawgit.com/hplgit/doconce/master/bundled/html_images/small_gray_notice.png); }
.alert-summary { background-image:url(https://cdn.rawgit.com/hplgit/doconce/master/bundled/html_images/small_gray_summary.png); }
.alert-warning { background-image: url(https://cdn.rawgit.com/hplgit/doconce/master/bundled/html_images/small_gray_warning.png); }
.alert-question {background-image:url(https://cdn.rawgit.com/hplgit/doconce/master/bundled/html_images/small_gray_question.png); }
.alert-notice { background-image: url(https://cdn.rawgit.com/doconce/doconce/master/bundled/html_images/small_gray_notice.png); }
.alert-summary { background-image:url(https://cdn.rawgit.com/doconce/doconce/master/bundled/html_images/small_gray_summary.png); }
.alert-warning { background-image: url(https://cdn.rawgit.com/doconce/doconce/master/bundled/html_images/small_gray_warning.png); }
.alert-question {background-image:url(https://cdn.rawgit.com/doconce/doconce/master/bundled/html_images/small_gray_question.png); }
div { text-align: justify; text-justify: inter-word; }
</style>
@@ -66,106 +66,160 @@ div { text-align: justify; text-justify: inter-word; }
<!-- tocinfo
{'highest level': 2,
'sections': [('Plan for week 41', 2, None, '___sec0'),
('Setting up the Back propagation algorithm', 2, None, '___sec1'),
'sections': [('Plan for week 41', 2, None, 'plan-for-week-41'),
('Setting up the Back propagation algorithm',
2,
None,
'setting-up-the-back-propagation-algorithm'),
('Setting up a Multi-layer perceptron model for classification',
2,
None,
'___sec2'),
('Defining the cost function', 2, None, '___sec3'),
('Example: binary classification problem', 2, None, '___sec4'),
('The Softmax function', 2, None, '___sec5'),
'setting-up-a-multi-layer-perceptron-model-for-classification'),
('Defining the cost function',
2,
None,
'defining-the-cost-function'),
('Example: binary classification problem',
2,
None,
'example-binary-classification-problem'),
('The Softmax function', 2, None, 'the-softmax-function'),
('Developing a code for doing neural networks with back '
'propagation',
2,
None,
'___sec6'),
('Collect and pre-process data', 2, None, '___sec7'),
('Train and test datasets', 2, None, '___sec8'),
('Define model and architecture', 2, None, '___sec9'),
('Layers', 2, None, '___sec10'),
('Weights and biases', 2, None, '___sec11'),
('Feed-forward pass', 2, None, '___sec12'),
('Matrix multiplications', 2, None, '___sec13'),
('Choose cost function and optimizer', 2, None, '___sec14'),
('Optimizing the cost function', 2, None, '___sec15'),
('Regularization', 2, None, '___sec16'),
('Matrix multiplication', 2, None, '___sec17'),
('Improving performance', 2, None, '___sec18'),
('Full object-oriented implementation', 2, None, '___sec19'),
('Evaluate model performance on test data', 2, None, '___sec20'),
('Adjust hyperparameters', 2, None, '___sec21'),
('Visualization', 2, None, '___sec22'),
('scikit-learn implementation', 2, None, '___sec23'),
('Visualization', 2, None, '___sec24'),
'developing-a-code-for-doing-neural-networks-with-back-propagation'),
('Collect and pre-process data',
2,
None,
'collect-and-pre-process-data'),
('Train and test datasets', 2, None, 'train-and-test-datasets'),
('Define model and architecture',
2,
None,
'define-model-and-architecture'),
('Layers', 2, None, 'layers'),
('Weights and biases', 2, None, 'weights-and-biases'),
('Feed-forward pass', 2, None, 'feed-forward-pass'),
('Matrix multiplications', 2, None, 'matrix-multiplications'),
('Choose cost function and optimizer',
2,
None,
'choose-cost-function-and-optimizer'),
('Optimizing the cost function',
2,
None,
'optimizing-the-cost-function'),
('Regularization', 2, None, 'regularization'),
('Matrix multiplication', 2, None, 'matrix-multiplication'),
('Improving performance', 2, None, 'improving-performance'),
('Full object-oriented implementation',
2,
None,
'full-object-oriented-implementation'),
('Evaluate model performance on test data',
2,
None,
'evaluate-model-performance-on-test-data'),
('Adjust hyperparameters', 2, None, 'adjust-hyperparameters'),
('Visualization', 2, None, 'visualization'),
('scikit-learn implementation',
2,
None,
'scikit-learn-implementation'),
('Visualization', 2, None, 'visualization'),
('Building neural networks in Tensorflow and Keras',
2,
None,
'___sec25'),
('Tensorflow', 2, None, '___sec26'),
('Using Keras', 2, None, '___sec27'),
('Collect and pre-process data', 2, None, '___sec28'),
('The Breast Cancer Data, now with Keras', 2, None, '___sec29'),
'building-neural-networks-in-tensorflow-and-keras'),
('Tensorflow', 2, None, 'tensorflow'),
('Using Keras', 2, None, 'using-keras'),
('Collect and pre-process data',
2,
None,
'collect-and-pre-process-data'),
('The Breast Cancer Data, now with Keras',
2,
None,
'the-breast-cancer-data-now-with-keras'),
('Fine-tuning neural network hyperparameters',
2,
None,
'___sec30'),
('Hidden layers', 2, None, '___sec31'),
('Which activation function should I use?', 2, None, '___sec32'),
'fine-tuning-neural-network-hyperparameters'),
('Hidden layers', 2, None, 'hidden-layers'),
('Which activation function should I use?',
2,
None,
'which-activation-function-should-i-use'),
('Is the Logistic activation function (Sigmoid) our choice?',
2,
None,
'___sec33'),
('The derivative of the Logistic funtion', 2, None, '___sec34'),
('The RELU function family', 2, None, '___sec35'),
('Which activation function should we use?', 2, None, '___sec36'),
'is-the-logistic-activation-function-sigmoid-our-choice'),
('The derivative of the Logistic funtion',
2,
None,
'the-derivative-of-the-logistic-funtion'),
('The RELU function family', 2, None, 'the-relu-function-family'),
('Which activation function should we use?',
2,
None,
'which-activation-function-should-we-use'),
('More on activation functions, output layers',
2,
None,
'___sec37'),
('Batch Normalization', 2, None, '___sec38'),
('Dropout', 2, None, '___sec39'),
('Gradient Clipping', 2, None, '___sec40'),
('A very nice website on Neural Networks', 2, None, '___sec41'),
'more-on-activation-functions-output-layers'),
('Batch Normalization', 2, None, 'batch-normalization'),
('Dropout', 2, None, 'dropout'),
('Gradient Clipping', 2, None, 'gradient-clipping'),
('A very nice website on Neural Networks',
2,
None,
'a-very-nice-website-on-neural-networks'),
('A top-down perspective on Neural networks',
2,
None,
'___sec42'),
'a-top-down-perspective-on-neural-networks'),
('Limitations of supervised learning with deep networks',
2,
None,
'___sec43'),
'limitations-of-supervised-learning-with-deep-networks'),
('Convolutional Neural Networks (recognizing images)',
2,
None,
'___sec44'),
'convolutional-neural-networks-recognizing-images'),
('Regular NNs don’t scale well to full images',
2,
None,
'___sec45'),
('3D volumes of neurons', 2, None, '___sec46'),
('Layers used to build CNNs', 2, None, '___sec47'),
('Transforming images', 2, None, '___sec48'),
('CNNs in brief', 2, None, '___sec49'),
'regular-nns-don-t-scale-well-to-full-images'),
('3D volumes of neurons', 2, None, '3d-volumes-of-neurons'),
('Layers used to build CNNs',
2,
None,
'layers-used-to-build-cnns'),
('Transforming images', 2, None, 'transforming-images'),
('CNNs in brief', 2, None, 'cnns-in-brief'),
('CNNs in more detail, building convolutional neural networks in '
'Tensorflow and Keras',
2,
None,
'___sec50'),
('Setting it up', 2, None, '___sec51'),
('The MNIST dataset again', 2, None, '___sec52'),
('Strong correlations', 2, None, '___sec53'),
('Layers of a CNN', 2, None, '___sec54'),
('Systematic reduction', 2, None, '___sec55'),
'cnns-in-more-detail-building-convolutional-neural-networks-in-tensorflow-and-keras'),
('Setting it up', 2, None, 'setting-it-up'),
('The MNIST dataset again', 2, None, 'the-mnist-dataset-again'),
('Strong correlations', 2, None, 'strong-correlations'),
('Layers of a CNN', 2, None, 'layers-of-a-cnn'),
('Systematic reduction', 2, None, 'systematic-reduction'),
('Prerequisites: Collect and pre-process data',
2,
None,
'___sec56'),
('Importing Keras and Tensorflow', 2, None, '___sec57'),
('Running with Keras', 2, None, '___sec58'),
('Final part', 2, None, '___sec59'),
('Final visualization', 2, None, '___sec60'),
('Fun links', 2, None, '___sec61')]}
'prerequisites-collect-and-pre-process-data'),
('Importing Keras and Tensorflow',
2,
None,
'importing-keras-and-tensorflow'),
('Running with Keras', 2, None, 'running-with-keras'),
('Final part', 2, None, 'final-part'),
('Final visualization', 2, None, 'final-visualization'),
('Fun links', 2, None, 'fun-links')]}
end of tocinfo -->
<body>
@@ -191,7 +245,7 @@ MathJax.Hub.Config({
<center><h1>Week 41 Tensor flow and Deep Learning, Convolutional Neural Networks</h1></center> <!-- document title -->
<center><h1>Week 41 Constructing a Neural Network code, Tensor flow and start Convolutional Neural Networks</h1></center> <!-- document title -->
<p>
<!-- author(s): Morten Hjorth-Jensen -->
@@ -207,25 +261,26 @@ MathJax.Hub.Config({
<center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
<br>
<p>
<center><h4>Oct 10, 2020</h4></center> <!-- date -->
<center><h4>Oct 9, 2021</h4></center> <!-- date -->
<br>
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec0">Plan for week 41 </h2>
<h2 id="plan-for-week-41">Plan for week 41 </h2>
<ul>
<li> Thursday: Building our own Feed-forward Neural Network. <a href="https://www.uio.no/studier/emner/matnat/fys/FYS-STK4155/h20/forelesningsvideoer/LectureOctober8.mp4?vrtx=view-as-webpage" target="_blank">Video of Lecture</a></li>
<li> Friday: Playing around with our own Feed-forward Neural Network and introduction to TensorFlow. Start convolutional Neural Networks (CNN). <a href="https://www.uio.no/studier/emner/matnat/fys/FYS-STK4155/h20/forelesningsvideoer/LectureOcotber9.mp4?vrtx=view-as-webpage" target="_blank">Video of Lecture</a></li>
<li> Thursday: Building our own Feed-forward Neural Network.</li>
<li> Friday: Playing around with our own Feed-forward Neural Network and introduction to TensorFlow. Start convolutional Neural Networks (CNN).</li>
</ul>
Reading suggestions for both days: <a href="https://github.com/CompPhysics/MachineLearning/blob/master/doc/T\
extbooks/TensorflowML.pdf" target="_blank">Aurelien Geron's chapters 10-11</a> and Hastie et al chapter 11.
extbooks/TensorflowML.pdf" target="_blank">Aurelien Geron's chapters 10-11</a>.
For amore in depth discussion on neural networks we recommend Goodfellow et al chapters 6 and 7. For CNNs, see Goodfellow et al chapter 9. chapter 11 and 12 on practicalities and applications
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec1">Setting up the Back propagation algorithm </h2>
<h2 id="setting-up-the-back-propagation-algorithm">Setting up the Back propagation algorithm </h2>
<p>
The four equations derived last week provide us with a way of computing the gradient of the cost function. Let us write this out in the form of an algorithm.
@@ -296,7 +351,7 @@ Here it is convenient to use stochastic gradient descent (see the examples below
<p>
<!-- !split -->
<h2 id="___sec2">Setting up a Multi-layer perceptron model for classification </h2>
<h2 id="setting-up-a-multi-layer-perceptron-model-for-classification">Setting up a Multi-layer perceptron model for classification </h2>
<p>
We are now gong to develop an example based on the MNIST data
@@ -334,7 +389,7 @@ of our network.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec3">Defining the cost function </h2>
<h2 id="defining-the-cost-function">Defining the cost function </h2>
<p>
Our cost function is given as (see the Logistic regression lectures)
@@ -394,7 +449,7 @@ The back propagation equations need now only a small change, namely the definiti
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec4">Example: binary classification problem </h2>
<h2 id="example-binary-classification-problem">Example: binary classification problem </h2>
<p>
As an example of the above, relevant for project 2 as well, let us consider a binary class. As discussed in our logistic regression lectures, we defined a cost function in terms of the parameters \( \beta \) as
@@ -442,7 +497,7 @@ In case we use another activation function than the logistic one, we need to eva
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec5">The Softmax function </h2>
<h2 id="the-softmax-function">The Softmax function </h2>
In case we employ the more general case given by the Softmax equation, we need to evaluate the derivative of the activation function with respect to the activation \( z_i^l \), that is we need
$$
\frac{\partial f(z_i^l)}{\partial w_{jk}^l} =
@@ -464,7 +519,7 @@ which in case of the simply binary model reduces to having \( i=j \).
<p>
<!-- !split -->
<h2 id="___sec6">Developing a code for doing neural networks with back propagation </h2>
<h2 id="developing-a-code-for-doing-neural-networks-with-back-propagation">Developing a code for doing neural networks with back propagation </h2>
<p>
One can identify a set of key steps when using neural networks to solve supervised learning problems:
@@ -480,7 +535,7 @@ One can identify a set of key steps when using neural networks to solve supervis
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec7">Collect and pre-process data </h2>
<h2 id="collect-and-pre-process-data">Collect and pre-process data </h2>
<p>
Here we will be using the MNIST dataset, which is readily available through the <b>scikit-learn</b>
@@ -529,7 +584,7 @@ images.
<p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span><span style="color: #408080; font-style: italic"># import necessary packages</span>
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%;"><span></span><span style="color: #408080; font-style: italic"># import necessary packages</span>
<span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">numpy</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">np</span>
<span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">matplotlib.pyplot</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">plt</span>
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn</span> <span style="color: #008000; font-weight: bold">import</span> datasets
@@ -575,7 +630,7 @@ plt<span style="color: #666666">.</span>show()
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec8">Train and test datasets </h2>
<h2 id="train-and-test-datasets">Train and test datasets </h2>
<p>
Performing analysis before partitioning the dataset is a major error, that can lead to incorrect conclusions.
@@ -593,7 +648,7 @@ collected from 12.00 to 24.00.
<p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span><span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn.model_selection</span> <span style="color: #008000; font-weight: bold">import</span> train_test_split
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%;"><span></span><span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn.model_selection</span> <span style="color: #008000; font-weight: bold">import</span> train_test_split
<span style="color: #408080; font-style: italic"># one-liner from scikit-learn library</span>
train_size <span style="color: #666666">=</span> <span style="color: #666666">0.8</span>
@@ -624,7 +679,7 @@ X_train, X_test, Y_train, Y_test <span style="color: #666666">=</span> train_tes
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec9">Define model and architecture </h2>
<h2 id="define-model-and-architecture">Define model and architecture </h2>
<p>
Our simple feed-forward neural network will consist of an <em>input</em> layer, a single <em>hidden</em> layer and an <em>output</em> layer. The activation \( y \) of each neuron is a weighted sum of inputs, passed through an activation function. In case of the simple perceptron model we have
@@ -670,7 +725,7 @@ which is inspired by probability theory (see logistic regression) and was most c
<p>
<!-- !split -->
<h2 id="___sec10">Layers </h2>
<h2 id="layers">Layers </h2>
<ul>
<li> Input</li>
@@ -715,7 +770,7 @@ weights to the output layer.
<p>
<!-- !split -->
<h2 id="___sec11">Weights and biases </h2>
<h2 id="weights-and-biases">Weights and biases </h2>
<p>
Typically weights are initialized with small values distributed around zero, drawn from a uniform
@@ -732,7 +787,7 @@ The bias weights \( \hat{b} \) are often initialized to zero, but a small value
<p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span><span style="color: #408080; font-style: italic"># building our neural network</span>
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%;"><span></span><span style="color: #408080; font-style: italic"># building our neural network</span>
n_inputs, n_features <span style="color: #666666">=</span> X_train<span style="color: #666666">.</span>shape
n_hidden_neurons <span style="color: #666666">=</span> <span style="color: #666666">50</span>
@@ -751,7 +806,7 @@ output_bias <span style="color: #666666">=</span> np<span style="color: #666666"
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec12">Feed-forward pass </h2>
<h2 id="feed-forward-pass">Feed-forward pass </h2>
<p>
Denote \( F \) the number of features, \( H \) the number of hidden neurons and \( C \) the number of categories.
@@ -778,7 +833,7 @@ $$ a_{j}^{L} = \frac{\exp{(z_j^{L})}}
<p>
<!-- !split -->
<h2 id="___sec13">Matrix multiplications </h2>
<h2 id="matrix-multiplications">Matrix multiplications </h2>
<p>
Since our data has the dimensions \( X = (n_{inputs}, n_{features}) \) and our weights to the hidden
@@ -814,7 +869,7 @@ $$ output = softmax (\hat{z}^{L}) = (n_{inputs}, n_{categories}) .$$
<p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span><span style="color: #408080; font-style: italic"># setup the feed-forward pass, subscript h = hidden layer</span>
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%;"><span></span><span style="color: #408080; font-style: italic"># setup the feed-forward pass, subscript h = hidden layer</span>
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">sigmoid</span>(x):
<span style="color: #008000; font-weight: bold">return</span> <span style="color: #666666">1/</span>(<span style="color: #666666">1</span> <span style="color: #666666">+</span> np<span style="color: #666666">.</span>exp(<span style="color: #666666">-</span>x))
@@ -853,7 +908,7 @@ predictions <span style="color: #666666">=</span> predict(X_train)
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec14">Choose cost function and optimizer </h2>
<h2 id="choose-cost-function-and-optimizer">Choose cost function and optimizer </h2>
<p>
To measure how well our neural network is doing we need to introduce a cost function.
@@ -884,7 +939,7 @@ you got the correct label. The probability of category \( c \) is given by the s
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec15">Optimizing the cost function </h2>
<h2 id="optimizing-the-cost-function">Optimizing the cost function </h2>
<p>
The network is trained by finding the weights and biases that minimize the cost function. One of the most widely used classes of methods is <em>gradient descent</em> and its generalizations. The idea behind gradient descent
@@ -924,7 +979,7 @@ The various optmization methods, with codes and algorithms, are discussed in o
<p>
<!-- !split -->
<h2 id="___sec16">Regularization </h2>
<h2 id="regularization">Regularization </h2>
<p>
It is common to add an extra term to the cost function, proportional
@@ -957,7 +1012,7 @@ calculate the gradient efficently.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec17">Matrix multiplication </h2>
<h2 id="matrix-multiplication">Matrix multiplication </h2>
<p>
To more efficently train our network these equations are implemented using matrix operations.
@@ -999,7 +1054,7 @@ $$ \nabla b_{h} = \sum_{i=1}^{n_{inputs}} \delta_h = (n_{hidden}) .$$
<p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span><span style="color: #408080; font-style: italic"># to categorical turns our integer vector into a onehot representation</span>
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%;"><span></span><span style="color: #408080; font-style: italic"># to categorical turns our integer vector into a onehot representation</span>
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn.metrics</span> <span style="color: #008000; font-weight: bold">import</span> accuracy_score
<span style="color: #408080; font-style: italic"># one-hot in numpy</span>
@@ -1071,7 +1126,7 @@ lmbd <span style="color: #666666">=</span> <span style="color: #666666">0.01</sp
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec18">Improving performance </h2>
<h2 id="improving-performance">Improving performance </h2>
<p>
As we can see the network does not seem to be learning at all. It seems to be just guessing the label for each image.
@@ -1091,7 +1146,7 @@ Andrew Ng goes through some of these considerations in this <a href="https://you
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec19">Full object-oriented implementation </h2>
<h2 id="full-object-oriented-implementation">Full object-oriented implementation </h2>
<p>
It is very natural to think of the network as an object, with specific instances of the network
@@ -1100,7 +1155,7 @@ being realizations of this object with different hyperparameters. An implementat
<p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span><span style="color: #008000; font-weight: bold">class</span> <span style="color: #0000FF; font-weight: bold">NeuralNetwork</span>:
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%;"><span></span><span style="color: #008000; font-weight: bold">class</span> <span style="color: #0000FF; font-weight: bold">NeuralNetwork</span>:
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">__init__</span>(
<span style="color: #008000">self</span>,
X_data,
@@ -1203,7 +1258,7 @@ being realizations of this object with different hyperparameters. An implementat
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec20">Evaluate model performance on test data </h2>
<h2 id="evaluate-model-performance-on-test-data">Evaluate model performance on test data </h2>
<p>
To measure the performance of our network we evaluate how well it does it data it has never seen before, i.e. the test data.
@@ -1218,7 +1273,7 @@ where \( I \) is the indicator function, \( 1 \) if \( \hat{y}_i = y_i \) and \(
<p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span>epochs <span style="color: #666666">=</span> <span style="color: #666666">100</span>
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%;"><span></span>epochs <span style="color: #666666">=</span> <span style="color: #666666">100</span>
batch_size <span style="color: #666666">=</span> <span style="color: #666666">100</span>
dnn <span style="color: #666666">=</span> NeuralNetwork(X_train, Y_train_onehot, eta<span style="color: #666666">=</span>eta, lmbd<span style="color: #666666">=</span>lmbd, epochs<span style="color: #666666">=</span>epochs, batch_size<span style="color: #666666">=</span>batch_size,
@@ -1238,7 +1293,7 @@ test_predict <span style="color: #666666">=</span> dnn<span style="color: #66666
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec21">Adjust hyperparameters </h2>
<h2 id="adjust-hyperparameters">Adjust hyperparameters </h2>
<p>
We now perform a grid search to find the optimal hyperparameters for the network.
@@ -1247,7 +1302,7 @@ Note that we are only using 1 layer with 50 neurons, and human performance is es
<p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span>eta_vals <span style="color: #666666">=</span> np<span style="color: #666666">.</span>logspace(<span style="color: #666666">-5</span>, <span style="color: #666666">1</span>, <span style="color: #666666">7</span>)
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%;"><span></span>eta_vals <span style="color: #666666">=</span> np<span style="color: #666666">.</span>logspace(<span style="color: #666666">-5</span>, <span style="color: #666666">1</span>, <span style="color: #666666">7</span>)
lmbd_vals <span style="color: #666666">=</span> np<span style="color: #666666">.</span>logspace(<span style="color: #666666">-5</span>, <span style="color: #666666">1</span>, <span style="color: #666666">7</span>)
<span style="color: #408080; font-style: italic"># store the models for later use</span>
DNN_numpy <span style="color: #666666">=</span> np<span style="color: #666666">.</span>zeros((<span style="color: #008000">len</span>(eta_vals), <span style="color: #008000">len</span>(lmbd_vals)), dtype<span style="color: #666666">=</span><span style="color: #008000">object</span>)
@@ -1271,12 +1326,12 @@ DNN_numpy <span style="color: #666666">=</span> np<span style="color: #666666">.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec22">Visualization </h2>
<h2 id="visualization">Visualization </h2>
<p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span><span style="color: #408080; font-style: italic"># visual representation of grid search</span>
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%;"><span></span><span style="color: #408080; font-style: italic"># visual representation of grid search</span>
<span style="color: #408080; font-style: italic"># uses seaborn heatmap, you can also do this with matplotlib imshow</span>
<span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">seaborn</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">sns</span>
@@ -1313,7 +1368,7 @@ plt<span style="color: #666666">.</span>show()
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec23">scikit-learn implementation </h2>
<h2 id="scikit-learn-implementation">scikit-learn implementation </h2>
<p>
<b>scikit-learn</b> focuses more
@@ -1332,7 +1387,7 @@ performance overall.
<p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span><span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn.neural_network</span> <span style="color: #008000; font-weight: bold">import</span> MLPClassifier
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%;"><span></span><span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn.neural_network</span> <span style="color: #008000; font-weight: bold">import</span> MLPClassifier
<span style="color: #408080; font-style: italic"># store models for later use</span>
DNN_scikit <span style="color: #666666">=</span> np<span style="color: #666666">.</span>zeros((<span style="color: #008000">len</span>(eta_vals), <span style="color: #008000">len</span>(lmbd_vals)), dtype<span style="color: #666666">=</span><span style="color: #008000">object</span>)
@@ -1352,11 +1407,11 @@ DNN_scikit <span style="color: #666666">=</span> np<span style="color: #666666">
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec24">Visualization </h2>
<h2 id="visualization">Visualization </h2>
<p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span><span style="color: #408080; font-style: italic"># optional</span>
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%;"><span></span><span style="color: #408080; font-style: italic"># optional</span>
<span style="color: #408080; font-style: italic"># visual representation of grid search</span>
<span style="color: #408080; font-style: italic"># uses seaborn heatmap, could probably do this in matplotlib</span>
<span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">seaborn</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">sns</span>
@@ -1394,7 +1449,7 @@ plt<span style="color: #666666">.</span>show()
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec25">Building neural networks in Tensorflow and Keras </h2>
<h2 id="building-neural-networks-in-tensorflow-and-keras">Building neural networks in Tensorflow and Keras </h2>
<p>
Now we want to build on the experience gained from our neural network implementation in NumPy and scikit-learn
@@ -1409,7 +1464,7 @@ NumPy arrays.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec26">Tensorflow </h2>
<h2 id="tensorflow">Tensorflow </h2>
<p>
Tensorflow is an open source library machine learning library
@@ -1442,7 +1497,7 @@ To install tensorflow on Unix/Linux systems, use pip as
<p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span>pip3 install tensorflow
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%;"><span></span>pip3 install tensorflow
</pre></div>
<p>
and/or if you use <b>anaconda</b>, just write (or install from the graphical user interface)
@@ -1450,7 +1505,7 @@ and/or if you use <b>anaconda</b>, just write (or install from the graphical use
<p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span>conda create <span style="color: #666666">-</span>n tf tensorflow
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%;"><span></span>conda create <span style="color: #666666">-</span>n tf tensorflow
conda activate tf
</pre></div>
<p>
@@ -1458,13 +1513,13 @@ To install the current release of GPU TensorFlow
<p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span>conda create <span style="color: #666666">-</span>n tf<span style="color: #666666">-</span>gpu tensorflow<span style="color: #666666">-</span>gpu
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%;"><span></span>conda create <span style="color: #666666">-</span>n tf<span style="color: #666666">-</span>gpu tensorflow<span style="color: #666666">-</span>gpu
conda activate tf<span style="color: #666666">-</span>gpu
</pre></div>
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec27">Using Keras </h2>
<h2 id="using-keras">Using Keras </h2>
<p>
Keras is a high level <a href="https://en.wikipedia.org/wiki/Application_programming_interface" target="_blank">neural network</a>
@@ -1473,7 +1528,7 @@ If you have Anaconda installed you may run the following command
<p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span>conda install keras
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%;"><span></span>conda install keras
</pre></div>
<p>
You can look up the <a href="https://keras.io/" target="_blank">instructions here</a> for more information.
@@ -1484,7 +1539,7 @@ We will to a large extent use <b>keras</b> in this course.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec28">Collect and pre-process data </h2>
<h2 id="collect-and-pre-process-data">Collect and pre-process data </h2>
<p>
Let us look again at the MINST data set.
@@ -1492,7 +1547,7 @@ Let us look again at the MINST data set.
<p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span><span style="color: #408080; font-style: italic"># import necessary packages</span>
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%;"><span></span><span style="color: #408080; font-style: italic"># import necessary packages</span>
<span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">numpy</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">np</span>
<span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">matplotlib.pyplot</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">plt</span>
<span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">tensorflow</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">tf</span>
@@ -1539,7 +1594,7 @@ plt<span style="color: #666666">.</span>show()
<p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span><span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">tensorflow.keras.layers</span> <span style="color: #008000; font-weight: bold">import</span> Input
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%;"><span></span><span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">tensorflow.keras.layers</span> <span style="color: #008000; font-weight: bold">import</span> Input
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">tensorflow.keras.models</span> <span style="color: #008000; font-weight: bold">import</span> Sequential <span style="color: #408080; font-style: italic">#This allows appending layers to existing models</span>
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">tensorflow.keras.layers</span> <span style="color: #008000; font-weight: bold">import</span> Dense <span style="color: #408080; font-style: italic">#This allows defining the characteristics of a particular layer</span>
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">tensorflow.keras</span> <span style="color: #008000; font-weight: bold">import</span> optimizers <span style="color: #408080; font-style: italic">#This allows using whichever optimiser we want (sgd,adam,RMSprop)</span>
@@ -1560,7 +1615,7 @@ X_train, X_test, Y_train, Y_test <span style="color: #666666">=</span> train_tes
<p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span>epochs <span style="color: #666666">=</span> <span style="color: #666666">100</span>
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%;"><span></span>epochs <span style="color: #666666">=</span> <span style="color: #666666">100</span>
batch_size <span style="color: #666666">=</span> <span style="color: #666666">100</span>
n_neurons_layer1 <span style="color: #666666">=</span> <span style="color: #666666">100</span>
n_neurons_layer2 <span style="color: #666666">=</span> <span style="color: #666666">50</span>
@@ -1581,7 +1636,7 @@ lmbd_vals <span style="color: #666666">=</span> np<span style="color: #666666">.
<p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span>DNN_keras <span style="color: #666666">=</span> np<span style="color: #666666">.</span>zeros((<span style="color: #008000">len</span>(eta_vals), <span style="color: #008000">len</span>(lmbd_vals)), dtype<span style="color: #666666">=</span><span style="color: #008000">object</span>)
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%;"><span></span>DNN_keras <span style="color: #666666">=</span> np<span style="color: #666666">.</span>zeros((<span style="color: #008000">len</span>(eta_vals), <span style="color: #008000">len</span>(lmbd_vals)), dtype<span style="color: #666666">=</span><span style="color: #008000">object</span>)
<span style="color: #008000; font-weight: bold">for</span> i, eta <span style="color: #AA22FF; font-weight: bold">in</span> <span style="color: #008000">enumerate</span>(eta_vals):
<span style="color: #008000; font-weight: bold">for</span> j, lmbd <span style="color: #AA22FF; font-weight: bold">in</span> <span style="color: #008000">enumerate</span>(lmbd_vals):
@@ -1600,7 +1655,7 @@ lmbd_vals <span style="color: #666666">=</span> np<span style="color: #666666">.
<p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span><span style="color: #408080; font-style: italic"># optional</span>
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%;"><span></span><span style="color: #408080; font-style: italic"># optional</span>
<span style="color: #408080; font-style: italic"># visual representation of grid search</span>
<span style="color: #408080; font-style: italic"># uses seaborn heatmap, could probably do this in matplotlib</span>
<span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">seaborn</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">sns</span>
@@ -1635,12 +1690,12 @@ plt<span style="color: #666666">.</span>show()
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec29">The Breast Cancer Data, now with Keras </h2>
<h2 id="the-breast-cancer-data-now-with-keras">The Breast Cancer Data, now with Keras </h2>
<p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span><span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">tensorflow</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">tf</span>
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%;"><span></span><span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">tensorflow</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">tf</span>
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">tensorflow.keras.layers</span> <span style="color: #008000; font-weight: bold">import</span> Input
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">tensorflow.keras.models</span> <span style="color: #008000; font-weight: bold">import</span> Sequential <span style="color: #408080; font-style: italic">#This allows appending layers to existing models</span>
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">tensorflow.keras.layers</span> <span style="color: #008000; font-weight: bold">import</span> Dense <span style="color: #408080; font-style: italic">#This allows defining the characteristics of a particular layer</span>
@@ -1809,7 +1864,7 @@ plot_data(eta,n_neuron,Test_accuracy, <span style="color: #BA2121">&#39;testing&
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec30">Fine-tuning neural network hyperparameters </h2>
<h2 id="fine-tuning-neural-network-hyperparameters">Fine-tuning neural network hyperparameters </h2>
<p>
The flexibility of neural networks is also one of their main
@@ -1835,7 +1890,7 @@ will only be able to explore a tiny part of the hyperparameter space.
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec31">Hidden layers </h2>
<h2 id="hidden-layers">Hidden layers </h2>
<p>
For many problems you can start with just one or two hidden layers and it will work just fine.
@@ -1855,7 +1910,7 @@ common to reuse parts of a pretrained state-of-the-art network that performs a s
<p>
<!-- !split -->
<h2 id="___sec32">Which activation function should I use? </h2>
<h2 id="which-activation-function-should-i-use">Which activation function should I use? </h2>
<p>
The Back propagation algorithm we derived above works by going from
@@ -1884,7 +1939,7 @@ learn at widely different speeds
<p>
<!-- !split -->
<h2 id="___sec33">Is the Logistic activation function (Sigmoid) our choice? </h2>
<h2 id="is-the-logistic-activation-function-sigmoid-our-choice">Is the Logistic activation function (Sigmoid) our choice? </h2>
<p>
Although this unfortunate behavior has been empirically observed for
@@ -1914,7 +1969,7 @@ better than the logistic function in deep networks).
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec34">The derivative of the Logistic funtion </h2>
<h2 id="the-derivative-of-the-logistic-funtion">The derivative of the Logistic funtion </h2>
<p>
Looking at the logistic activation function, when inputs become large
@@ -1950,7 +2005,7 @@ fast to compute).
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec35">The RELU function family </h2>
<h2 id="the-relu-function-family">The RELU function family </h2>
<p>
The ReLU activation function suffers from a problem known as the dying
@@ -1977,7 +2032,7 @@ $$
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec36">Which activation function should we use? </h2>
<h2 id="which-activation-function-should-we-use">Which activation function should we use? </h2>
<p>
In general it seems that the ELU activation function is better than
@@ -1997,7 +2052,7 @@ bootstrap to evaluate other activation functions.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec37">More on activation functions, output layers </h2>
<h2 id="more-on-activation-functions-output-layers">More on activation functions, output layers </h2>
<p>
In most cases you can use the ReLU activation function in the hidden layers (or one of its variants).
@@ -2015,7 +2070,7 @@ It is a bit faster to compute than other activation functions, and the gradient
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec38">Batch Normalization </h2>
<h2 id="batch-normalization">Batch Normalization </h2>
<p>
Batch Normalization
@@ -2034,7 +2089,7 @@ mini-batch, from this the name batch normalization.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec39">Dropout </h2>
<h2 id="dropout">Dropout </h2>
<p>
It is a fairly simple algorithm: at every training step, every neuron (including the input neurons but
@@ -2049,7 +2104,7 @@ hyperparameter \( p \) is called the dropout rate, and it is typically set to 50
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec40">Gradient Clipping </h2>
<h2 id="gradient-clipping">Gradient Clipping </h2>
<p>
A popular technique to lessen the exploding gradients problem is to simply clip the gradients during
@@ -2066,7 +2121,7 @@ Normalization is preferred.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec41">A very nice website on Neural Networks </h2>
<h2 id="a-very-nice-website-on-neural-networks">A very nice website on Neural Networks </h2>
<p>
You may find this <a href="https://playground.tensorflow.org/#activation=tanh&batchSize=10&dataset=circle&regDataset=reg-plane&learningRate=0.03&regularizationRate=0&noise=0&networkShape=4,2&seed=0.29243&showTestData=false&discretize=false&percTrainData=50&x=true&y=true&xTimesY=false&xSquared=false&ySquared=false&cosX=false&sinX=false&cosY=false&sinY=false&collectStats=false&problem=classification&initZero=false&hideText=false" target="_blank">website</a> very useful. Thx a million to Ghadi for sharing.
@@ -2074,7 +2129,7 @@ You may find this <a href="https://playground.tensorflow.org/#activation=tanh&ba
<p>
<!-- !split -->
<h2 id="___sec42">A top-down perspective on Neural networks </h2>
<h2 id="a-top-down-perspective-on-neural-networks">A top-down perspective on Neural networks </h2>
<p>
The first thing we would like to do is divide the data into two or three
@@ -2116,7 +2171,7 @@ supervised learning.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec43">Limitations of supervised learning with deep networks </h2>
<h2 id="limitations-of-supervised-learning-with-deep-networks">Limitations of supervised learning with deep networks </h2>
<p>
Like all statistical methods, supervised learning using neural
@@ -2142,7 +2197,7 @@ Some of these remarks are particular to DNNs, others are shared by all supervise
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec44">Convolutional Neural Networks (recognizing images) </h2>
<h2 id="convolutional-neural-networks-recognizing-images">Convolutional Neural Networks (recognizing images) </h2>
<p>
Convolutional neural networks (CNNs) were developed during the last
@@ -2181,7 +2236,7 @@ Another good read is the article here <a href="https://arxiv.org/pdf/1603.07285.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec45">Regular NNs don&#8217;t scale well to full images </h2>
<h2 id="regular-nns-don-t-scale-well-to-full-images">Regular NNs don&#8217;t scale well to full images </h2>
<p>
As an example, consider
@@ -2209,7 +2264,7 @@ would quickly lead to possible overfitting.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec46">3D volumes of neurons </h2>
<h2 id="3d-volumes-of-neurons">3D volumes of neurons </h2>
<p>
Convolutional Neural Networks take advantage of the fact that the
@@ -2249,7 +2304,7 @@ dimension.
<p>
<!-- !split -->
<h2 id="___sec47">Layers used to build CNNs </h2>
<h2 id="layers-used-to-build-cnns">Layers used to build CNNs </h2>
<p>
A simple CNN is a sequence of layers, and every layer of a CNN
@@ -2272,7 +2327,7 @@ A simple CNN for image classification could have the architecture:
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec48">Transforming images </h2>
<h2 id="transforming-images">Transforming images </h2>
<p>
CNNs transform the original image layer by layer from the original
@@ -2291,7 +2346,7 @@ are consistent with the labels in the training set for each image.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec49">CNNs in brief </h2>
<h2 id="cnns-in-brief">CNNs in brief </h2>
<p>
In summary:
@@ -2312,7 +2367,7 @@ and the slides of <a href="http://cs231n.github.io/convolutional-networks/" targ
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec50">CNNs in more detail, building convolutional neural networks in Tensorflow and Keras </h2>
<h2 id="cnns-in-more-detail-building-convolutional-neural-networks-in-tensorflow-and-keras">CNNs in more detail, building convolutional neural networks in Tensorflow and Keras </h2>
<p>
As discussed above, CNNs are neural networks built from the assumption that the inputs
@@ -2328,7 +2383,7 @@ matrices, typically 1 for each color dimension (Red, Green, Blue).
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec51">Setting it up </h2>
<h2 id="setting-it-up">Setting it up </h2>
<p>
It means that to represent the entire
@@ -2340,7 +2395,7 @@ $$
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec52">The MNIST dataset again </h2>
<h2 id="the-mnist-dataset-again">The MNIST dataset again </h2>
<p>
The MNIST dataset consists of grayscale images with a pixel size of
@@ -2358,7 +2413,7 @@ single neuron in the first hidden layer.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec53">Strong correlations </h2>
<h2 id="strong-correlations">Strong correlations </h2>
<p>
Images typically have strong local correlations, meaning that a small
@@ -2376,7 +2431,7 @@ fixed, and known as a <a href="https://en.wikipedia.org/wiki/Receptive_field" ta
<p>
<!-- !split -->
<h2 id="___sec54">Layers of a CNN </h2>
<h2 id="layers-of-a-cnn">Layers of a CNN </h2>
The layers of a convolutional neural network arrange neurons in 3D: width, height and depth.
The input image is typically a square matrix of depth 3.
@@ -2398,7 +2453,7 @@ layer.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec55">Systematic reduction </h2>
<h2 id="systematic-reduction">Systematic reduction </h2>
<p>
By systematically reducing the size of the input volume, through
@@ -2413,11 +2468,11 @@ classification.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec56">Prerequisites: Collect and pre-process data </h2>
<h2 id="prerequisites-collect-and-pre-process-data">Prerequisites: Collect and pre-process data </h2>
<p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span><span style="color: #408080; font-style: italic"># import necessary packages</span>
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%;"><span></span><span style="color: #408080; font-style: italic"># import necessary packages</span>
<span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">numpy</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">np</span>
<span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">matplotlib.pyplot</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">plt</span>
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn</span> <span style="color: #008000; font-weight: bold">import</span> datasets
@@ -2461,11 +2516,11 @@ plt<span style="color: #666666">.</span>show()
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec57">Importing Keras and Tensorflow </h2>
<h2 id="importing-keras-and-tensorflow">Importing Keras and Tensorflow </h2>
<p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span><span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">tensorflow.keras</span> <span style="color: #008000; font-weight: bold">import</span> datasets, layers, models
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%;"><span></span><span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">tensorflow.keras</span> <span style="color: #008000; font-weight: bold">import</span> datasets, layers, models
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">tensorflow.keras.layers</span> <span style="color: #008000; font-weight: bold">import</span> Input
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">tensorflow.keras.models</span> <span style="color: #008000; font-weight: bold">import</span> Sequential <span style="color: #408080; font-style: italic">#This allows appending layers to existing models</span>
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">tensorflow.keras.layers</span> <span style="color: #008000; font-weight: bold">import</span> Dense <span style="color: #408080; font-style: italic">#This allows defining the characteristics of a particular layer</span>
@@ -2491,12 +2546,12 @@ X_train, X_test, Y_train, Y_test <span style="color: #666666">=</span> train_tes
<p>
<!-- !split -->
<h2 id="___sec58">Running with Keras </h2>
<h2 id="running-with-keras">Running with Keras </h2>
<p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span><span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">create_convolutional_neural_network_keras</span>(input_shape, receptive_field,
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%;"><span></span><span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">create_convolutional_neural_network_keras</span>(input_shape, receptive_field,
n_filters, n_neurons_connected, n_categories,
eta, lmbd):
model <span style="color: #666666">=</span> Sequential()
@@ -2526,12 +2581,12 @@ lmbd_vals <span style="color: #666666">=</span> np<span style="color: #666666">.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec59">Final part </h2>
<h2 id="final-part">Final part </h2>
<p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span>CNN_keras <span style="color: #666666">=</span> np<span style="color: #666666">.</span>zeros((<span style="color: #008000">len</span>(eta_vals), <span style="color: #008000">len</span>(lmbd_vals)), dtype<span style="color: #666666">=</span><span style="color: #008000">object</span>)
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%;"><span></span>CNN_keras <span style="color: #666666">=</span> np<span style="color: #666666">.</span>zeros((<span style="color: #008000">len</span>(eta_vals), <span style="color: #008000">len</span>(lmbd_vals)), dtype<span style="color: #666666">=</span><span style="color: #008000">object</span>)
<span style="color: #008000; font-weight: bold">for</span> i, eta <span style="color: #AA22FF; font-weight: bold">in</span> <span style="color: #008000">enumerate</span>(eta_vals):
<span style="color: #008000; font-weight: bold">for</span> j, lmbd <span style="color: #AA22FF; font-weight: bold">in</span> <span style="color: #008000">enumerate</span>(lmbd_vals):
@@ -2551,12 +2606,12 @@ lmbd_vals <span style="color: #666666">=</span> np<span style="color: #666666">.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec60">Final visualization </h2>
<h2 id="final-visualization">Final visualization </h2>
<p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span><span style="color: #408080; font-style: italic"># visual representation of grid search</span>
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%;"><span></span><span style="color: #408080; font-style: italic"># visual representation of grid search</span>
<span style="color: #408080; font-style: italic"># uses seaborn heatmap, could probably do this in matplotlib</span>
<span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">seaborn</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">sns</span>
@@ -2590,7 +2645,7 @@ plt<span style="color: #666666">.</span>show()
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec61">Fun links </h2>
<h2 id="fun-links">Fun links </h2>
<ol>
<li> <a href="https://arxiv.org/abs/1604.07316" target="_blank">Self-Driving cars using a convolutional neural network</a></li>
@@ -2602,7 +2657,7 @@ plt<span style="color: #666666">.</span>show()
<center style="font-size:80%">
<!-- copyright --> &copy; 1999-2020, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license
<!-- copyright --> &copy; 1999-2021, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license
</center>
doc/pub/week41/ipynb/ipynb-week41-src.tar.gz
BIN
View File
Binary file not shown.
doc/pub/week41/ipynb/week41.ipynb
+141 -89
View File
@@ -4,28 +4,28 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"<!-- dom:TITLE: Week 41 Tensor flow and Deep Learning, Convolutional Neural Networks -->\n",
"# Week 41 Tensor flow and Deep Learning, Convolutional Neural Networks\n",
"<!-- dom:TITLE: Week 41 Constructing a Neural Network code, Tensor flow and start Convolutional Neural Networks -->\n",
"# Week 41 Constructing a Neural Network code, Tensor flow and start Convolutional Neural Networks\n",
"<!-- dom:AUTHOR: Morten Hjorth-Jensen at Department of Physics, University of Oslo & Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University -->\n",
"<!-- Author: --> \n",
"**Morten Hjorth-Jensen**, Department of Physics, University of Oslo and Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University\n",
"\n",
"Date: **Oct 10, 2020**\n",
"Date: **Oct 9, 2021**\n",
"\n",
"Copyright 1999-2020, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license\n",
"Copyright 1999-2021, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license\n",
"\n",
"\n",
"\n",
"\n",
"## Plan for week 41\n",
"\n",
"* Thursday: Building our own Feed-forward Neural Network. [Video of Lecture](https://www.uio.no/studier/emner/matnat/fys/FYS-STK4155/h20/forelesningsvideoer/LectureOctober8.mp4?vrtx=view-as-webpage)\n",
"* Thursday: Building our own Feed-forward Neural Network. \n",
"\n",
"* Friday: Playing around with our own Feed-forward Neural Network and introduction to TensorFlow. Start convolutional Neural Networks (CNN). [Video of Lecture](https://www.uio.no/studier/emner/matnat/fys/FYS-STK4155/h20/forelesningsvideoer/LectureOcotber9.mp4?vrtx=view-as-webpage)\n",
"* Friday: Playing around with our own Feed-forward Neural Network and introduction to TensorFlow. Start convolutional Neural Networks (CNN). \n",
"\n",
"Reading suggestions for both days: [Aurelien Geron's chapters 10-11](https://github.com/CompPhysics/MachineLearning/blob/master/doc/T\\\n",
"extbooks/TensorflowML.pdf) and Hastie et al chapter 11.\n",
"\n",
"extbooks/TensorflowML.pdf).\n",
"For amore in depth discussion on neural networks we recommend Goodfellow et al chapters 6 and 7. For CNNs, see Goodfellow et al chapter 9. chapter 11 and 12 on practicalities and applications\n",
"\n",
"## Setting up the Back propagation algorithm\n",
"\n",
@@ -490,8 +490,11 @@
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"execution_count": null,
"metadata": {
"collapsed": false,
"editable": true
},
"outputs": [],
"source": [
"%matplotlib inline\n",
@@ -559,8 +562,11 @@
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"execution_count": null,
"metadata": {
"collapsed": false,
"editable": true
},
"outputs": [],
"source": [
"from sklearn.model_selection import train_test_split\n",
@@ -681,8 +687,11 @@
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"execution_count": null,
"metadata": {
"collapsed": false,
"editable": true
},
"outputs": [],
"source": [
"# building our neural network\n",
@@ -758,8 +767,11 @@
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"execution_count": null,
"metadata": {
"collapsed": false,
"editable": true
},
"outputs": [],
"source": [
"# setup the feed-forward pass, subscript h = hidden layer\n",
@@ -923,8 +935,11 @@
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {},
"execution_count": null,
"metadata": {
"collapsed": false,
"editable": true
},
"outputs": [],
"source": [
"# to categorical turns our integer vector into a onehot representation\n",
@@ -1022,8 +1037,11 @@
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {},
"execution_count": null,
"metadata": {
"collapsed": false,
"editable": true
},
"outputs": [],
"source": [
"class NeuralNetwork:\n",
@@ -1144,8 +1162,11 @@
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {},
"execution_count": null,
"metadata": {
"collapsed": false,
"editable": true
},
"outputs": [],
"source": [
"epochs = 100\n",
@@ -1178,8 +1199,11 @@
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {},
"execution_count": null,
"metadata": {
"collapsed": false,
"editable": true
},
"outputs": [],
"source": [
"eta_vals = np.logspace(-5, 1, 7)\n",
@@ -1213,8 +1237,11 @@
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {},
"execution_count": null,
"metadata": {
"collapsed": false,
"editable": true
},
"outputs": [],
"source": [
"# visual representation of grid search\n",
@@ -1273,8 +1300,11 @@
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {},
"execution_count": null,
"metadata": {
"collapsed": false,
"editable": true
},
"outputs": [],
"source": [
"from sklearn.neural_network import MLPClassifier\n",
@@ -1304,8 +1334,11 @@
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {},
"execution_count": null,
"metadata": {
"collapsed": false,
"editable": true
},
"outputs": [],
"source": [
"# optional\n",
@@ -1387,8 +1420,11 @@
},
{
"cell_type": "code",
"execution_count": 12,
"metadata": {},
"execution_count": null,
"metadata": {
"collapsed": false,
"editable": true
},
"outputs": [],
"source": [
"pip3 install tensorflow"
@@ -1404,8 +1440,11 @@
},
{
"cell_type": "code",
"execution_count": 13,
"metadata": {},
"execution_count": null,
"metadata": {
"collapsed": false,
"editable": true
},
"outputs": [],
"source": [
"conda create -n tf tensorflow\n",
@@ -1421,8 +1460,11 @@
},
{
"cell_type": "code",
"execution_count": 14,
"metadata": {},
"execution_count": null,
"metadata": {
"collapsed": false,
"editable": true
},
"outputs": [],
"source": [
"conda create -n tf-gpu tensorflow-gpu\n",
@@ -1442,8 +1484,11 @@
},
{
"cell_type": "code",
"execution_count": 15,
"metadata": {},
"execution_count": null,
"metadata": {
"collapsed": false,
"editable": true
},
"outputs": [],
"source": [
"conda install keras"
@@ -1464,8 +1509,11 @@
},
{
"cell_type": "code",
"execution_count": 16,
"metadata": {},
"execution_count": null,
"metadata": {
"collapsed": false,
"editable": true
},
"outputs": [],
"source": [
"# import necessary packages\n",
@@ -1515,8 +1563,11 @@
},
{
"cell_type": "code",
"execution_count": 17,
"metadata": {},
"execution_count": null,
"metadata": {
"collapsed": false,
"editable": true
},
"outputs": [],
"source": [
"from tensorflow.keras.layers import Input\n",
@@ -1540,8 +1591,11 @@
},
{
"cell_type": "code",
"execution_count": 18,
"metadata": {},
"execution_count": null,
"metadata": {
"collapsed": false,
"editable": true
},
"outputs": [],
"source": [
"\n",
@@ -1566,8 +1620,11 @@
},
{
"cell_type": "code",
"execution_count": 19,
"metadata": {},
"execution_count": null,
"metadata": {
"collapsed": false,
"editable": true
},
"outputs": [],
"source": [
"DNN_keras = np.zeros((len(eta_vals), len(lmbd_vals)), dtype=object)\n",
@@ -1589,8 +1646,11 @@
},
{
"cell_type": "code",
"execution_count": 20,
"metadata": {},
"execution_count": null,
"metadata": {
"collapsed": false,
"editable": true
},
"outputs": [],
"source": [
"# optional\n",
@@ -1635,8 +1695,11 @@
},
{
"cell_type": "code",
"execution_count": 21,
"metadata": {},
"execution_count": null,
"metadata": {
"collapsed": false,
"editable": true
},
"outputs": [],
"source": [
"\n",
@@ -2155,11 +2218,7 @@
"\n",
"<!-- dom:FIGURE: [figslides/nn.jpeg, width=500 frac=0.6] A regular 3-layer Neural Network. -->\n",
"<!-- begin figure -->\n",
"\n",
"<p>A regular 3-layer Neural Network.</p>\n",
"<img src=\"figslides/nn.jpeg\" width=500>\n",
"\n",
"<!-- end figure -->\n",
"<img src=\"figslides/nn.jpeg\" width=500><p style=\"font-size: 0.9em\"><i>Figure 1: A regular 3-layer Neural Network.</i></p><!-- end figure -->\n",
"\n",
"\n",
"## 3D volumes of neurons\n",
@@ -2190,11 +2249,7 @@
"\n",
"<!-- dom:FIGURE: [figslides/cnn.jpeg, width=500 frac=0.6] A CNN arranges its neurons in three dimensions (width, height, depth), as visualized in one of the layers. Every layer of a CNN transforms the 3D input volume to a 3D output volume of neuron activations. In this example, the red input layer holds the image, so its width and height would be the dimensions of the image, and the depth would be 3 (Red, Green, Blue channels). -->\n",
"<!-- begin figure -->\n",
"\n",
"<p>A CNN arranges its neurons in three dimensions (width, height, depth), as visualized in one of the layers. Every layer of a CNN transforms the 3D input volume to a 3D output volume of neuron activations. In this example, the red input layer holds the image, so its width and height would be the dimensions of the image, and the depth would be 3 (Red, Green, Blue channels).</p>\n",
"<img src=\"figslides/cnn.jpeg\" width=500>\n",
"\n",
"<!-- end figure -->\n",
"<img src=\"figslides/cnn.jpeg\" width=500><p style=\"font-size: 0.9em\"><i>Figure 1: A CNN arranges its neurons in three dimensions (width, height, depth), as visualized in one of the layers. Every layer of a CNN transforms the 3D input volume to a 3D output volume of neuron activations. In this example, the red input layer holds the image, so its width and height would be the dimensions of the image, and the depth would be 3 (Red, Green, Blue channels).</i></p><!-- end figure -->\n",
"\n",
"\n",
"\n",
@@ -2355,8 +2410,11 @@
},
{
"cell_type": "code",
"execution_count": 22,
"metadata": {},
"execution_count": null,
"metadata": {
"collapsed": false,
"editable": true
},
"outputs": [],
"source": [
"# import necessary packages\n",
@@ -2410,8 +2468,11 @@
},
{
"cell_type": "code",
"execution_count": 23,
"metadata": {},
"execution_count": null,
"metadata": {
"collapsed": false,
"editable": true
},
"outputs": [],
"source": [
"from tensorflow.keras import datasets, layers, models\n",
@@ -2448,8 +2509,11 @@
},
{
"cell_type": "code",
"execution_count": 24,
"metadata": {},
"execution_count": null,
"metadata": {
"collapsed": false,
"editable": true
},
"outputs": [],
"source": [
"def create_convolutional_neural_network_keras(input_shape, receptive_field,\n",
@@ -2489,8 +2553,11 @@
},
{
"cell_type": "code",
"execution_count": 25,
"metadata": {},
"execution_count": null,
"metadata": {
"collapsed": false,
"editable": true
},
"outputs": [],
"source": [
"CNN_keras = np.zeros((len(eta_vals), len(lmbd_vals)), dtype=object)\n",
@@ -2520,8 +2587,11 @@
},
{
"cell_type": "code",
"execution_count": 26,
"metadata": {},
"execution_count": null,
"metadata": {
"collapsed": false,
"editable": true
},
"outputs": [],
"source": [
"# visual representation of grid search\n",
@@ -2568,25 +2638,7 @@
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.8.5"
}
},
"metadata": {},
"nbformat": 4,
"nbformat_minor": 4
}
doc/src/week41/week41.do.txt
+5 -5
View File
@@ -1,4 +1,4 @@
TITLE: Week 41 Tensor flow and Deep Learning, Convolutional Neural Networks
TITLE: Week 41 Constructing a Neural Network code, Tensor flow and start Convolutional Neural Networks
AUTHOR: Morten Hjorth-Jensen {copyright, 1999-present|CC BY-NC} at Department of Physics, University of Oslo & Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University
DATE: today
@@ -6,12 +6,12 @@ DATE: today
!split
===== Plan for week 41 =====
* Thursday: Building our own Feed-forward Neural Network. "Video of Lecture":"https://www.uio.no/studier/emner/matnat/fys/FYS-STK4155/h20/forelesningsvideoer/LectureOctober8.mp4?vrtx=view-as-webpage"
* Friday: Playing around with our own Feed-forward Neural Network and introduction to TensorFlow. Start convolutional Neural Networks (CNN). "Video of Lecture":"https://www.uio.no/studier/emner/matnat/fys/FYS-STK4155/h20/forelesningsvideoer/LectureOcotber9.mp4?vrtx=view-as-webpage"
* Thursday: Building our own Feed-forward Neural Network.
* Friday: Playing around with our own Feed-forward Neural Network and introduction to TensorFlow. Start convolutional Neural Networks (CNN).
Reading suggestions for both days: "Aurelien Geron's chapters 10-11":"https://github.com/CompPhysics/MachineLearning/blob/master/doc/T\
extbooks/TensorflowML.pdf" and Hastie et al chapter 11.
extbooks/TensorflowML.pdf".
For amore in depth discussion on neural networks we recommend Goodfellow et al chapters 6 and 7. For CNNs, see Goodfellow et al chapter 9. chapter 11 and 12 on practicalities and applications
!split
===== Setting up the Back propagation algorithm =====