-
@@ -297,6 +316,13 @@ Some of these remarks are particular to DNNs, others are shared by all supervise
- 65
- 66
- 67 +
- 68 +
- 69 +
- 70 +
- 71 +
- 72 +
- 73 +
- »
+ + + + +
Convolutional Neural Networks (recognizing images)
+ ++Convolutional Neural Networks (CNN) are very similar to ordinary Neural Networks. + +
+They are made up of neurons that have learnable weights and +biases. Each neuron receives some inputs, performs a dot product and +optionally follows it with a non-linearity. The whole network still +expresses a single differentiable score function: from the raw image +pixels on one end to class scores at the other. And they still have a +loss function (for example Softmax) on the last (fully-connected) layer +and all the tips/tricks we developed for learning regular Neural +Networks still apply (back propagation, gradient descent etc etc). + +
+What is the difference? CNN architectures make the explicit assumption that +the inputs are images, which allows us to encode certain properties +into the architecture. These then make the forward function more +efficient to implement and vastly reduce the amount of parameters in +the network. + +
+Here we provide only a superficial overview, for the more interested, we recommend highly the course +IN5400 – Machine Learning for Image Analysis +and the slides of CS231. + +
+
+ +
+ + +
+ + + + +
Regular NNs don’t scale well to full images
+ ++As an example, consider +an image of size \( 32\times 32\times 3 \) (32 wide, 32 high, 3 color channels), so a +single fully-connected neuron in a first hidden layer of a regular +Neural Network would have \( 32*32*3 = 3072 \) weights. This amount still +seems manageable, but clearly this fully-connected structure does not +scale to larger images. For example, an image of more respectable +size, say \( 200\times 200\times 3 \), would lead to neurons that have +\( 200*200*3 =120,000 \) weights. Moreover, we would almost certainly want to have +several such neurons, so the parameters would add up quickly! Clearly, +this full connectivity is wasteful and the huge number of parameters +would quickly lead to overfitting. + +
+
+
Figure 1: A regular 3-layer Neural Network.
+
+ +
+ + +
+ + + + +
3D volumes of neurons
+ ++Convolutional Neural Networks take advantage of the fact that the +input consists of images and they constrain the architecture in a more +sensible way. + +
+In particular, unlike a regular Neural Network, the +layers of a CNN have neurons arranged in 3 dimensions: width, +height, depth. (Note that the word depth here refers to the third +dimension of an activation volume, not to the depth of a full Neural +Network, which can refer to the total number of layers in a network.) + +
+To understand it better, the above example of an image +with an input volume of +activations has dimensions \( 32\times 32\times 3 \) (width, height, +depth respectively). + +
+The neurons in a layer will +only be connected to a small region of the layer before it, instead of +all of the neurons in a fully-connected manner. Moreover, the final +output layer could for this specific image have dimensions \( 1\times 1 times 10 \), +because by the +end of the CNN architecture we will reduce the full image into a +single vector of class scores, arranged along the depth +dimension. + +
+
+
Figure 2: A CNN arranges its neurons in three dimensions (width, height, depth), as visualized in one of the layers. Every layer of a CNN transforms the 3D input volume to a 3D output volume of neuron activations. In this example, the red input layer holds the image, so its width and height would be the dimensions of the image, and the depth would be 3 (Red, Green, Blue channels).
+
+ +
+ + +
+ + + + +
Layers used to build CNNs
+ ++A simple CNN is a sequence of layers, and every layer of a CNN +transforms one volume of activations to another through a +differentiable function. We use three main types of layers to build +CNN architectures: Convolutional Layer, Pooling Layer, and +Fully-Connected Layer (exactly as seen in regular Neural Networks). We +will stack these layers to form a full CNN architecture. + +
+A simple CNN for image classification could have the architecture: + +
-
+
- INPUT (\( 32\times 32 \times 3 \)) will hold the raw pixel values of the image, in this case an image of width 32, height 32, and with three color channels R,G,B. +
- CONV layer will compute the output of neurons that are connected to local regions in the input, each computing a dot product between their weights and a small region they are connected to in the input volume. This may result in volume such as \( [32\times 32\times 12] \) if we decided to use 12 filters. +
- RELU layer will apply an elementwise activation function, such as the \( max(0,x) \) thresholding at zero. This leaves the size of the volume unchanged (\( [32\times 32\times 12] \)). +
- POOL layer will perform a downsampling operation along the spatial dimensions (width, height), resulting in volume such as \( [16\times 16\times 12] \). +
- FC (i.e. fully-connected) layer will compute the class scores, resulting in volume of size \( [1\times 1\times 10] \), where each of the 10 numbers correspond to a class score, such as among the 10 categories of the MNIST images we considered above . As with ordinary Neural Networks and as the name implies, each neuron in this layer will be connected to all the numbers in the previous volume. +
+
+ +
+ + +
+ + + + +
CNNs in brief
+ ++In summary: + +
-
+
- A CNN architecture is in the simplest case a list of Layers that transform the image volume into an output volume (e.g. holding the class scores) +
- There are a few distinct types of Layers (e.g. CONV/FC/RELU/POOL are by far the most popular) +
- Each Layer accepts an input 3D volume and transforms it to an output 3D volume through a differentiable function +
- Each Layer may or may not have parameters (e.g. CONV/FC do, RELU/POOL don’t) +
- Each Layer may or may not have additional hyperparameters (e.g. CONV/FC/POOL do, RELU doesn’t) +
+ +
+ + +-
Oct 16, 2018
Oct 18, 2018
@@ -302,7 +320,7 @@ MathJax.Hub.Config({
-
Oct 16, 2018
Oct 18, 2018
@@ -3085,7 +3085,153 @@ Here we list some of the important limitations of supervised neural network base
-Some of these remarks are particular to DNNs, others are shared by all supervised learning methods. This motivates the use of unsupervised methods which in part circumnavigate these problems.
+Some of these remarks are particular to DNNs, others are shared by all supervised learning methods. This motivates the use of unsupervised methods which in part circumvent these problems.
+
+
+
+
+Convolutional Neural Networks (CNN) are very similar to ordinary Neural Networks.
+
+
+They are made up of neurons that have learnable weights and
+biases. Each neuron receives some inputs, performs a dot product and
+optionally follows it with a non-linearity. The whole network still
+expresses a single differentiable score function: from the raw image
+pixels on one end to class scores at the other. And they still have a
+loss function (for example Softmax) on the last (fully-connected) layer
+and all the tips/tricks we developed for learning regular Neural
+Networks still apply (back propagation, gradient descent etc etc).
+
+
+What is the difference? CNN architectures make the explicit assumption that
+the inputs are images, which allows us to encode certain properties
+into the architecture. These then make the forward function more
+efficient to implement and vastly reduce the amount of parameters in
+the network.
+
+
+Here we provide only a superficial overview, for the more interested, we recommend highly the course
+IN5400 – Machine Learning for Image Analysis
+and the slides of CS231.
+
+As an example, consider
+an image of size \( 32\times 32\times 3 \) (32 wide, 32 high, 3 color channels), so a
+single fully-connected neuron in a first hidden layer of a regular
+Neural Network would have \( 32*32*3 = 3072 \) weights. This amount still
+seems manageable, but clearly this fully-connected structure does not
+scale to larger images. For example, an image of more respectable
+size, say \( 200\times 200\times 3 \), would lead to neurons that have
+\( 200*200*3 =120,000 \) weights. Moreover, we would almost certainly want to have
+several such neurons, so the parameters would add up quickly! Clearly,
+this full connectivity is wasteful and the huge number of parameters
+would quickly lead to overfitting.
+
+
+ Figure 1: A regular 3-layer Neural Network.
+Convolutional Neural Networks take advantage of the fact that the
+input consists of images and they constrain the architecture in a more
+sensible way.
+
+
+In particular, unlike a regular Neural Network, the
+layers of a CNN have neurons arranged in 3 dimensions: width,
+height, depth. (Note that the word depth here refers to the third
+dimension of an activation volume, not to the depth of a full Neural
+Network, which can refer to the total number of layers in a network.)
+
+
+To understand it better, the above example of an image
+with an input volume of
+activations has dimensions \( 32\times 32\times 3 \) (width, height,
+depth respectively).
+
+
+The neurons in a layer will
+only be connected to a small region of the layer before it, instead of
+all of the neurons in a fully-connected manner. Moreover, the final
+output layer could for this specific image have dimensions \( 1\times 1 times 10 \),
+because by the
+end of the CNN architecture we will reduce the full image into a
+single vector of class scores, arranged along the depth
+dimension.
+
+
+ Figure 2: A CNN arranges its neurons in three dimensions (width, height, depth), as visualized in one of the layers. Every layer of a CNN transforms the 3D input volume to a 3D output volume of neuron activations. In this example, the red input layer holds the image, so its width and height would be the dimensions of the image, and the depth would be 3 (Red, Green, Blue channels).
+A simple CNN is a sequence of layers, and every layer of a CNN
+transforms one volume of activations to another through a
+differentiable function. We use three main types of layers to build
+CNN architectures: Convolutional Layer, Pooling Layer, and
+Fully-Connected Layer (exactly as seen in regular Neural Networks). We
+will stack these layers to form a full CNN architecture.
+
+
+A simple CNN for image classification could have the architecture:
+
+
+
+In this way, CNNs transform the original image layer by layer from the original pixel values to the final class scores. Note that some layers contain parameters and other don’t. In particular, the CONV/FC layers perform transformations that are a function of not only the activations in the input volume, but also of the parameters (the weights and biases of the neurons). On the other hand, the RELU/POOL layers will implement a fixed function. The parameters in the CONV/FC layers will be trained with gradient descent so that the class scores that the CNN computes are consistent with the labels in the training set for each image.
+
+In summary:
+
+
+More material to come with examples.
Convolutional Neural Networks (recognizing images)
+
+Regular NNs don’t scale well to full images
+
+
+3D volumes of neurons
+
+
+Layers used to build CNNs
+
+
+
+CNNs in brief
+
+
+
+CNNs in more detail
+
+
-
Oct 16, 2018
Oct 18, 2018
@@ -2923,7 +2935,151 @@ Here we list some of the important limitations of supervised neural network base
+
+
+
Convolutional Neural Networks (recognizing images)
+ ++Convolutional Neural Networks (CNN) are very similar to ordinary Neural Networks. + +
+They are made up of neurons that have learnable weights and +biases. Each neuron receives some inputs, performs a dot product and +optionally follows it with a non-linearity. The whole network still +expresses a single differentiable score function: from the raw image +pixels on one end to class scores at the other. And they still have a +loss function (for example Softmax) on the last (fully-connected) layer +and all the tips/tricks we developed for learning regular Neural +Networks still apply (back propagation, gradient descent etc etc). + +
+What is the difference? CNN architectures make the explicit assumption that +the inputs are images, which allows us to encode certain properties +into the architecture. These then make the forward function more +efficient to implement and vastly reduce the amount of parameters in +the network. + +
+Here we provide only a superficial overview, for the more interested, we recommend highly the course +IN5400 – Machine Learning for Image Analysis +and the slides of CS231. + +
+
+
+
Regular NNs don’t scale well to full images
+ ++As an example, consider +an image of size \( 32\times 32\times 3 \) (32 wide, 32 high, 3 color channels), so a +single fully-connected neuron in a first hidden layer of a regular +Neural Network would have \( 32*32*3 = 3072 \) weights. This amount still +seems manageable, but clearly this fully-connected structure does not +scale to larger images. For example, an image of more respectable +size, say \( 200\times 200\times 3 \), would lead to neurons that have +\( 200*200*3 =120,000 \) weights. Moreover, we would almost certainly want to have +several such neurons, so the parameters would add up quickly! Clearly, +this full connectivity is wasteful and the huge number of parameters +would quickly lead to overfitting. + +
+
+
Figure 1: A regular 3-layer Neural Network.
+
+
+
3D volumes of neurons
+ ++Convolutional Neural Networks take advantage of the fact that the +input consists of images and they constrain the architecture in a more +sensible way. + +
+In particular, unlike a regular Neural Network, the +layers of a CNN have neurons arranged in 3 dimensions: width, +height, depth. (Note that the word depth here refers to the third +dimension of an activation volume, not to the depth of a full Neural +Network, which can refer to the total number of layers in a network.) + +
+To understand it better, the above example of an image +with an input volume of +activations has dimensions \( 32\times 32\times 3 \) (width, height, +depth respectively). + +
+The neurons in a layer will +only be connected to a small region of the layer before it, instead of +all of the neurons in a fully-connected manner. Moreover, the final +output layer could for this specific image have dimensions \( 1\times 1 times 10 \), +because by the +end of the CNN architecture we will reduce the full image into a +single vector of class scores, arranged along the depth +dimension. + +
+
+
Figure 2: A CNN arranges its neurons in three dimensions (width, height, depth), as visualized in one of the layers. Every layer of a CNN transforms the 3D input volume to a 3D output volume of neuron activations. In this example, the red input layer holds the image, so its width and height would be the dimensions of the image, and the depth would be 3 (Red, Green, Blue channels).
+ + +
Layers used to build CNNs
+ ++A simple CNN is a sequence of layers, and every layer of a CNN +transforms one volume of activations to another through a +differentiable function. We use three main types of layers to build +CNN architectures: Convolutional Layer, Pooling Layer, and +Fully-Connected Layer (exactly as seen in regular Neural Networks). We +will stack these layers to form a full CNN architecture. + +
+A simple CNN for image classification could have the architecture: + +
-
+
- INPUT (\( 32\times 32 \times 3 \)) will hold the raw pixel values of the image, in this case an image of width 32, height 32, and with three color channels R,G,B. +
- CONV layer will compute the output of neurons that are connected to local regions in the input, each computing a dot product between their weights and a small region they are connected to in the input volume. This may result in volume such as \( [32\times 32\times 12] \) if we decided to use 12 filters. +
- RELU layer will apply an elementwise activation function, such as the \( max(0,x) \) thresholding at zero. This leaves the size of the volume unchanged (\( [32\times 32\times 12] \)). +
- POOL layer will perform a downsampling operation along the spatial dimensions (width, height), resulting in volume such as \( [16\times 16\times 12] \). +
- FC (i.e. fully-connected) layer will compute the class scores, resulting in volume of size \( [1\times 1\times 10] \), where each of the 10 numbers correspond to a class score, such as among the 10 categories of the MNIST images we considered above . As with ordinary Neural Networks and as the name implies, each neuron in this layer will be connected to all the numbers in the previous volume. +
+
+
+
CNNs in brief
+ ++In summary: + +
-
+
- A CNN architecture is in the simplest case a list of Layers that transform the image volume into an output volume (e.g. holding the class scores) +
- There are a few distinct types of Layers (e.g. CONV/FC/RELU/POOL are by far the most popular) +
- Each Layer accepts an input 3D volume and transforms it to an output 3D volume through a differentiable function +
- Each Layer may or may not have parameters (e.g. CONV/FC do, RELU/POOL don’t) +
- Each Layer may or may not have additional hyperparameters (e.g. CONV/FC/POOL do, RELU doesn’t) +
+ +
CNNs in more detail
+ ++More material to come with examples. diff --git a/doc/pub/NeuralNet/html/NeuralNet.html b/doc/pub/NeuralNet/html/NeuralNet.html index 6c767d090..f38aa6a9d 100644 --- a/doc/pub/NeuralNet/html/NeuralNet.html +++ b/doc/pub/NeuralNet/html/NeuralNet.html @@ -165,7 +165,19 @@ div { text-align: justify; text-justify: inter-word; } ('Limitations of supervised learning with deep networks', 2, None, - '___sec65')]} + '___sec65'), + ('Convolutional Neural Networks (recognizing images)', + 2, + None, + '___sec66'), + ('Regular NNs don’t scale well to full images', + 2, + None, + '___sec67'), + ('3D volumes of neurons', 2, None, '___sec68'), + ('Layers used to build CNNs', 2, None, '___sec69'), + ('CNNs in brief', 2, None, '___sec70'), + ('CNNs in more detail', 2, None, '___sec71')]} end of tocinfo -->
@@ -207,7 +219,7 @@ MathJax.Hub.Config({-
Oct 16, 2018
Oct 18, 2018
@@ -2928,7 +2940,151 @@ Here we list some of the important limitations of supervised neural network base
+
+
+
Convolutional Neural Networks (recognizing images)
+ ++Convolutional Neural Networks (CNN) are very similar to ordinary Neural Networks. + +
+They are made up of neurons that have learnable weights and +biases. Each neuron receives some inputs, performs a dot product and +optionally follows it with a non-linearity. The whole network still +expresses a single differentiable score function: from the raw image +pixels on one end to class scores at the other. And they still have a +loss function (for example Softmax) on the last (fully-connected) layer +and all the tips/tricks we developed for learning regular Neural +Networks still apply (back propagation, gradient descent etc etc). + +
+What is the difference? CNN architectures make the explicit assumption that +the inputs are images, which allows us to encode certain properties +into the architecture. These then make the forward function more +efficient to implement and vastly reduce the amount of parameters in +the network. + +
+Here we provide only a superficial overview, for the more interested, we recommend highly the course +IN5400 – Machine Learning for Image Analysis +and the slides of CS231. + +
+
+
+
Regular NNs don’t scale well to full images
+ ++As an example, consider +an image of size \( 32\times 32\times 3 \) (32 wide, 32 high, 3 color channels), so a +single fully-connected neuron in a first hidden layer of a regular +Neural Network would have \( 32*32*3 = 3072 \) weights. This amount still +seems manageable, but clearly this fully-connected structure does not +scale to larger images. For example, an image of more respectable +size, say \( 200\times 200\times 3 \), would lead to neurons that have +\( 200*200*3 =120,000 \) weights. Moreover, we would almost certainly want to have +several such neurons, so the parameters would add up quickly! Clearly, +this full connectivity is wasteful and the huge number of parameters +would quickly lead to overfitting. + +
+
+
Figure 1: A regular 3-layer Neural Network.
+
+
+
3D volumes of neurons
+ ++Convolutional Neural Networks take advantage of the fact that the +input consists of images and they constrain the architecture in a more +sensible way. + +
+In particular, unlike a regular Neural Network, the +layers of a CNN have neurons arranged in 3 dimensions: width, +height, depth. (Note that the word depth here refers to the third +dimension of an activation volume, not to the depth of a full Neural +Network, which can refer to the total number of layers in a network.) + +
+To understand it better, the above example of an image +with an input volume of +activations has dimensions \( 32\times 32\times 3 \) (width, height, +depth respectively). + +
+The neurons in a layer will +only be connected to a small region of the layer before it, instead of +all of the neurons in a fully-connected manner. Moreover, the final +output layer could for this specific image have dimensions \( 1\times 1 times 10 \), +because by the +end of the CNN architecture we will reduce the full image into a +single vector of class scores, arranged along the depth +dimension. + +
+
+
Figure 2: A CNN arranges its neurons in three dimensions (width, height, depth), as visualized in one of the layers. Every layer of a CNN transforms the 3D input volume to a 3D output volume of neuron activations. In this example, the red input layer holds the image, so its width and height would be the dimensions of the image, and the depth would be 3 (Red, Green, Blue channels).
+ + +
Layers used to build CNNs
+ ++A simple CNN is a sequence of layers, and every layer of a CNN +transforms one volume of activations to another through a +differentiable function. We use three main types of layers to build +CNN architectures: Convolutional Layer, Pooling Layer, and +Fully-Connected Layer (exactly as seen in regular Neural Networks). We +will stack these layers to form a full CNN architecture. + +
+A simple CNN for image classification could have the architecture: + +
-
+
- INPUT (\( 32\times 32 \times 3 \)) will hold the raw pixel values of the image, in this case an image of width 32, height 32, and with three color channels R,G,B. +
- CONV layer will compute the output of neurons that are connected to local regions in the input, each computing a dot product between their weights and a small region they are connected to in the input volume. This may result in volume such as \( [32\times 32\times 12] \) if we decided to use 12 filters. +
- RELU layer will apply an elementwise activation function, such as the \( max(0,x) \) thresholding at zero. This leaves the size of the volume unchanged (\( [32\times 32\times 12] \)). +
- POOL layer will perform a downsampling operation along the spatial dimensions (width, height), resulting in volume such as \( [16\times 16\times 12] \). +
- FC (i.e. fully-connected) layer will compute the class scores, resulting in volume of size \( [1\times 1\times 10] \), where each of the 10 numbers correspond to a class score, such as among the 10 categories of the MNIST images we considered above . As with ordinary Neural Networks and as the name implies, each neuron in this layer will be connected to all the numbers in the previous volume. +
+
+
+
CNNs in brief
+ ++In summary: + +
-
+
- A CNN architecture is in the simplest case a list of Layers that transform the image volume into an output volume (e.g. holding the class scores) +
- There are a few distinct types of Layers (e.g. CONV/FC/RELU/POOL are by far the most popular) +
- Each Layer accepts an input 3D volume and transforms it to an output 3D volume through a differentiable function +
- Each Layer may or may not have parameters (e.g. CONV/FC do, RELU/POOL don’t) +
- Each Layer may or may not have additional hyperparameters (e.g. CONV/FC/POOL do, RELU doesn’t) +
+ +
CNNs in more detail
+ ++More material to come with examples. diff --git a/doc/pub/NeuralNet/ipynb/NeuralNet.ipynb b/doc/pub/NeuralNet/ipynb/NeuralNet.ipynb index f4a238955..85ac78395 100644 --- a/doc/pub/NeuralNet/ipynb/NeuralNet.ipynb +++ b/doc/pub/NeuralNet/ipynb/NeuralNet.ipynb @@ -10,7 +10,7 @@ " \n", "**Morten Hjorth-Jensen**, Department of Physics, University of Oslo and Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University\n", "\n", - "Date: **Oct 16, 2018**\n", + "Date: **Oct 18, 2018**\n", "\n", "Copyright 1999-2018, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license\n", "\n", @@ -3307,7 +3307,135 @@ "\n", "* **Many problems are not about prediction.** In natural science we are often interested in learning something about the underlying distribution that generates the data. In this case, it is often difficult to cast these ideas in a supervised learning setting. While the problems are related, it is possible to make good predictions with a *wrong* model. The model might or might not be useful for understanding the underlying science.\n", "\n", - "Some of these remarks are particular to DNNs, others are shared by all supervised learning methods. This motivates the use of unsupervised methods which in part circumnavigate these problems." + "Some of these remarks are particular to DNNs, others are shared by all supervised learning methods. This motivates the use of unsupervised methods which in part circumvent these problems.\n", + "\n", + "\n", + "## Convolutional Neural Networks (recognizing images)\n", + "\n", + "Convolutional Neural Networks (CNN) are very similar to ordinary Neural Networks.\n", + "\n", + "They are made up of neurons that have learnable weights and\n", + "biases. Each neuron receives some inputs, performs a dot product and\n", + "optionally follows it with a non-linearity. The whole network still\n", + "expresses a single differentiable score function: from the raw image\n", + "pixels on one end to class scores at the other. And they still have a\n", + "loss function (for example Softmax) on the last (fully-connected) layer\n", + "and all the tips/tricks we developed for learning regular Neural\n", + "Networks still apply (back propagation, gradient descent etc etc).\n", + "\n", + "What is the difference? **CNN architectures make the explicit assumption that\n", + "the inputs are images, which allows us to encode certain properties\n", + "into the architecture. These then make the forward function more\n", + "efficient to implement and vastly reduce the amount of parameters in\n", + "the network.**\n", + "\n", + "Here we provide only a superficial overview, for the more interested, we recommend highly the course\n", + "[IN5400 – Machine Learning for Image Analysis](https://www.uio.no/studier/emner/matnat/ifi/IN5400/index-eng.html)\n", + "and the slides of [CS231](http://cs231n.github.io/convolutional-networks/).\n", + "\n", + "## Regular NNs don’t scale well to full images\n", + "\n", + "As an example, consider\n", + "an image of size $32\\times 32\\times 3$ (32 wide, 32 high, 3 color channels), so a\n", + "single fully-connected neuron in a first hidden layer of a regular\n", + "Neural Network would have $32*32*3 = 3072$ weights. This amount still\n", + "seems manageable, but clearly this fully-connected structure does not\n", + "scale to larger images. For example, an image of more respectable\n", + "size, say $200\\times 200\\times 3$, would lead to neurons that have \n", + "$200*200*3 =120,000$ weights. Moreover, we would almost certainly want to have\n", + "several such neurons, so the parameters would add up quickly! Clearly,\n", + "this full connectivity is wasteful and the huge number of parameters\n", + "would quickly lead to overfitting.\n", + "\n", + "\n", + "\n", + "\n", + "
A regular 3-layer Neural Network.
\n", + "![](\"figslides/nn.jpeg\")
\n",
+ "\n",
+ "\n",
+ "\n",
+ "\n",
+ "## 3D volumes of neurons\n",
+ "\n",
+ "Convolutional Neural Networks take advantage of the fact that the\n",
+ "input consists of images and they constrain the architecture in a more\n",
+ "sensible way. \n",
+ "\n",
+ "In particular, unlike a regular Neural Network, the\n",
+ "layers of a CNN have neurons arranged in 3 dimensions: width,\n",
+ "height, depth. (Note that the word depth here refers to the third\n",
+ "dimension of an activation volume, not to the depth of a full Neural\n",
+ "Network, which can refer to the total number of layers in a network.)\n",
+ "\n",
+ "To understand it better, the above example of an image \n",
+ "with an input volume of\n",
+ "activations has dimensions $32\\times 32\\times 3$ (width, height,\n",
+ "depth respectively). \n",
+ "\n",
+ "The neurons in a layer will\n",
+ "only be connected to a small region of the layer before it, instead of\n",
+ "all of the neurons in a fully-connected manner. Moreover, the final\n",
+ "output layer could for this specific image have dimensions $1\\times 1 times 10$, \n",
+ "because by the\n",
+ "end of the CNN architecture we will reduce the full image into a\n",
+ "single vector of class scores, arranged along the depth\n",
+ "dimension. \n",
+ "\n",
+ "\n",
+ "\n",
+ "\n",
+ "A CNN arranges its neurons in three dimensions (width, height, depth), as visualized in one of the layers. Every layer of a CNN transforms the 3D input volume to a 3D output volume of neuron activations. In this example, the red input layer holds the image, so its width and height would be the dimensions of the image, and the depth would be 3 (Red, Green, Blue channels).
\n", + "![](\"figslides/cnn.jpeg\")
\n",
+ "\n",
+ "\n",
+ "\n",
+ "\n",
+ "\n",
+ "\n",
+ "\n",
+ "## Layers used to build CNNs\n",
+ "\n",
+ "\n",
+ "A simple CNN is a sequence of layers, and every layer of a CNN\n",
+ "transforms one volume of activations to another through a\n",
+ "differentiable function. We use three main types of layers to build\n",
+ "CNN architectures: Convolutional Layer, Pooling Layer, and\n",
+ "Fully-Connected Layer (exactly as seen in regular Neural Networks). We\n",
+ "will stack these layers to form a full CNN architecture.\n",
+ "\n",
+ "A simple CNN for image classification could have the architecture:\n",
+ "\n",
+ "* INPUT ($32\\times 32 \\times 3$) will hold the raw pixel values of the image, in this case an image of width 32, height 32, and with three color channels R,G,B.\n",
+ "\n",
+ "* CONV layer will compute the output of neurons that are connected to local regions in the input, each computing a dot product between their weights and a small region they are connected to in the input volume. This may result in volume such as $[32\\times 32\\times 12]$ if we decided to use 12 filters.\n",
+ "\n",
+ "* RELU layer will apply an elementwise activation function, such as the $max(0,x)$ thresholding at zero. This leaves the size of the volume unchanged ($[32\\times 32\\times 12]$).\n",
+ "\n",
+ "* POOL layer will perform a downsampling operation along the spatial dimensions (width, height), resulting in volume such as $[16\\times 16\\times 12]$.\n",
+ "\n",
+ "* FC (i.e. fully-connected) layer will compute the class scores, resulting in volume of size $[1\\times 1\\times 10]$, where each of the 10 numbers correspond to a class score, such as among the 10 categories of the MNIST images we considered above . As with ordinary Neural Networks and as the name implies, each neuron in this layer will be connected to all the numbers in the previous volume.\n",
+ "\n",
+ "In this way, CNNs transform the original image layer by layer from the original pixel values to the final class scores. Note that some layers contain parameters and other don’t. In particular, the CONV/FC layers perform transformations that are a function of not only the activations in the input volume, but also of the parameters (the weights and biases of the neurons). On the other hand, the RELU/POOL layers will implement a fixed function. The parameters in the CONV/FC layers will be trained with gradient descent so that the class scores that the CNN computes are consistent with the labels in the training set for each image.\n",
+ "\n",
+ "\n",
+ "## CNNs in brief\n",
+ "\n",
+ "In summary:\n",
+ "\n",
+ "* A CNN architecture is in the simplest case a list of Layers that transform the image volume into an output volume (e.g. holding the class scores)\n",
+ "\n",
+ "* There are a few distinct types of Layers (e.g. CONV/FC/RELU/POOL are by far the most popular)\n",
+ "\n",
+ "* Each Layer accepts an input 3D volume and transforms it to an output 3D volume through a differentiable function\n",
+ "\n",
+ "* Each Layer may or may not have parameters (e.g. CONV/FC do, RELU/POOL don’t)\n",
+ "\n",
+ "* Each Layer may or may not have additional hyperparameters (e.g. CONV/FC/POOL do, RELU doesn’t)\n",
+ "\n",
+ "## CNNs in more detail\n",
+ "\n",
+ "More material to come with examples."
]
}
],
diff --git a/doc/pub/NeuralNet/ipynb/ipynb-NeuralNet-src.tar.gz b/doc/pub/NeuralNet/ipynb/ipynb-NeuralNet-src.tar.gz
index af4eab7ec..44dcf094e 100644
Binary files a/doc/pub/NeuralNet/ipynb/ipynb-NeuralNet-src.tar.gz and b/doc/pub/NeuralNet/ipynb/ipynb-NeuralNet-src.tar.gz differ
diff --git a/doc/pub/NeuralNet/pdf/NeuralNet-minted.pdf b/doc/pub/NeuralNet/pdf/NeuralNet-minted.pdf
index 6a0d241ab..d6f1918c0 100644
Binary files a/doc/pub/NeuralNet/pdf/NeuralNet-minted.pdf and b/doc/pub/NeuralNet/pdf/NeuralNet-minted.pdf differ
diff --git a/doc/pub/Regression/html/._Regression-bs000.html b/doc/pub/Regression/html/._Regression-bs000.html
index e247df55f..f2952f17c 100644
--- a/doc/pub/Regression/html/._Regression-bs000.html
+++ b/doc/pub/Regression/html/._Regression-bs000.html
@@ -433,7 +433,7 @@ MathJax.Hub.Config({
-
Oct 11, 2018
Oct 18, 2018
diff --git a/doc/pub/Regression/html/._Regression-bs109.html b/doc/pub/Regression/html/._Regression-bs109.html index 344f96c72..59012db88 100644 --- a/doc/pub/Regression/html/._Regression-bs109.html +++ b/doc/pub/Regression/html/._Regression-bs109.html @@ -453,7 +453,7 @@ on the left hand side. return x.T @ y @ scl.inv( x.T @ x + np.eye(x.shape[1], x.shape[1]) * _lambda ) -lambda = 0.1 +_lambda = 0.1 omega_ridge = get_ridge_weights(X_train_own, y_train, np.array([_lambda])) clf_ridge = skl.Ridge(alpha=_lambda).fit(X_train, y_train) J_ridge_own = omega_ridge[1:].reshape(L, L) diff --git a/doc/pub/Regression/html/Regression-bs.html b/doc/pub/Regression/html/Regression-bs.html index e247df55f..f2952f17c 100644 --- a/doc/pub/Regression/html/Regression-bs.html +++ b/doc/pub/Regression/html/Regression-bs.html @@ -433,7 +433,7 @@ MathJax.Hub.Config({
-
Oct 11, 2018
Oct 18, 2018
diff --git a/doc/pub/Regression/html/Regression-reveal.html b/doc/pub/Regression/html/Regression-reveal.html index dda913849..b5a627cb8 100644 --- a/doc/pub/Regression/html/Regression-reveal.html +++ b/doc/pub/Regression/html/Regression-reveal.html @@ -148,7 +148,7 @@ MathJax.Hub.Config({
-
Oct 11, 2018
Oct 18, 2018
@@ -3908,7 +3908,7 @@ on the left hand side. return x.T @ y @ scl.inv( x.T @ x + np.eye(x.shape[1], x.shape[1]) * _lambda ) -lambda = 0.1 +_lambda = 0.1 omega_ridge = get_ridge_weights(X_train_own, y_train, np.array([_lambda])) clf_ridge = skl.Ridge(alpha=_lambda).fit(X_train, y_train) J_ridge_own = omega_ridge[1:].reshape(L, L) diff --git a/doc/pub/Regression/html/Regression-solarized.html b/doc/pub/Regression/html/Regression-solarized.html index 6808aa209..fc93c73e8 100644 --- a/doc/pub/Regression/html/Regression-solarized.html +++ b/doc/pub/Regression/html/Regression-solarized.html @@ -310,7 +310,7 @@ MathJax.Hub.Config({
-
Oct 11, 2018
Oct 18, 2018
@@ -3833,7 +3833,7 @@ on the left hand side.
return x.T @ y @ scl.inv(
x.T @ x + np.eye(x.shape[1], x.shape[1]) * _lambda
)
-lambda = 0.1
+_lambda = 0.1
omega_ridge = get_ridge_weights(X_train_own, y_train, np.array([_lambda]))
clf_ridge = skl.Ridge(alpha=_lambda).fit(X_train, y_train)
J_ridge_own = omega_ridge[1:].reshape(L, L)
diff --git a/doc/pub/Regression/html/Regression.html b/doc/pub/Regression/html/Regression.html
index d19243051..bd9eb1210 100644
--- a/doc/pub/Regression/html/Regression.html
+++ b/doc/pub/Regression/html/Regression.html
@@ -315,7 +315,7 @@ MathJax.Hub.Config({
-
Oct 11, 2018
Oct 18, 2018
@@ -3838,7 +3838,7 @@ on the left hand side.
return x.T @ y @ scl.inv(
x.T @ x + np.eye(x.shape[1], x.shape[1]) * _lambda
)
-lambda = 0.1
+_lambda = 0.1
omega_ridge = get_ridge_weights(X_train_own, y_train, np.array([_lambda]))
clf_ridge = skl.Ridge(alpha=_lambda).fit(X_train, y_train)
J_ridge_own = omega_ridge[1:].reshape(L, L)
diff --git a/doc/pub/Regression/ipynb/Regression.ipynb b/doc/pub/Regression/ipynb/Regression.ipynb
index 3600fe608..b6a28684b 100644
--- a/doc/pub/Regression/ipynb/Regression.ipynb
+++ b/doc/pub/Regression/ipynb/Regression.ipynb
@@ -10,7 +10,7 @@
" \n",
"**Morten Hjorth-Jensen**, Department of Physics, University of Oslo and Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University\n",
"\n",
- "Date: **Oct 11, 2018**\n",
+ "Date: **Oct 18, 2018**\n",
"\n",
"Copyright 1999-2018, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license\n",
"\n",
@@ -5083,7 +5083,7 @@
" return x.T @ y @ scl.inv(\n",
" x.T @ x + np.eye(x.shape[1], x.shape[1]) * _lambda\n",
" )\n",
- "lambda = 0.1\n",
+ "_lambda = 0.1\n",
"omega_ridge = get_ridge_weights(X_train_own, y_train, np.array([_lambda]))\n",
"clf_ridge = skl.Ridge(alpha=_lambda).fit(X_train, y_train)\n",
"J_ridge_own = omega_ridge[1:].reshape(L, L)\n",
diff --git a/doc/pub/Regression/ipynb/ipynb-Regression-src.tar.gz b/doc/pub/Regression/ipynb/ipynb-Regression-src.tar.gz
index 43dd2e263..7da7a75aa 100644
Binary files a/doc/pub/Regression/ipynb/ipynb-Regression-src.tar.gz and b/doc/pub/Regression/ipynb/ipynb-Regression-src.tar.gz differ
diff --git a/doc/pub/Regression/pdf/Regression-beamer-handouts2x3.pdf b/doc/pub/Regression/pdf/Regression-beamer-handouts2x3.pdf
index 876912a1a..50fc490c0 100644
Binary files a/doc/pub/Regression/pdf/Regression-beamer-handouts2x3.pdf and b/doc/pub/Regression/pdf/Regression-beamer-handouts2x3.pdf differ
diff --git a/doc/pub/Regression/pdf/Regression-beamer.pdf b/doc/pub/Regression/pdf/Regression-beamer.pdf
index 685138f6d..afc4f2dd9 100644
Binary files a/doc/pub/Regression/pdf/Regression-beamer.pdf and b/doc/pub/Regression/pdf/Regression-beamer.pdf differ
diff --git a/doc/pub/Regression/pdf/Regression-minted.pdf b/doc/pub/Regression/pdf/Regression-minted.pdf
index 8478d424b..9ed537841 100644
Binary files a/doc/pub/Regression/pdf/Regression-minted.pdf and b/doc/pub/Regression/pdf/Regression-minted.pdf differ
diff --git a/doc/pub/Splines/html/._Splines-bs000.html b/doc/pub/Splines/html/._Splines-bs000.html
index e716541a9..9941cf5dd 100644
--- a/doc/pub/Splines/html/._Splines-bs000.html
+++ b/doc/pub/Splines/html/._Splines-bs000.html
@@ -289,7 +289,7 @@ MathJax.Hub.Config({
-
Oct 12, 2018
Oct 18, 2018
diff --git a/doc/pub/Splines/html/Splines-bs.html b/doc/pub/Splines/html/Splines-bs.html index e716541a9..9941cf5dd 100644 --- a/doc/pub/Splines/html/Splines-bs.html +++ b/doc/pub/Splines/html/Splines-bs.html @@ -289,7 +289,7 @@ MathJax.Hub.Config({
-
Oct 12, 2018
Oct 18, 2018
diff --git a/doc/pub/Splines/html/Splines-reveal.html b/doc/pub/Splines/html/Splines-reveal.html index cd16912c3..e3efed135 100644 --- a/doc/pub/Splines/html/Splines-reveal.html +++ b/doc/pub/Splines/html/Splines-reveal.html @@ -148,7 +148,7 @@ MathJax.Hub.Config({
-
Oct 12, 2018
Oct 18, 2018
diff --git a/doc/pub/Splines/html/Splines-solarized.html b/doc/pub/Splines/html/Splines-solarized.html index c60ec43bb..380b5b670 100644 --- a/doc/pub/Splines/html/Splines-solarized.html +++ b/doc/pub/Splines/html/Splines-solarized.html @@ -208,7 +208,7 @@ MathJax.Hub.Config({
-
Oct 12, 2018
Oct 18, 2018
diff --git a/doc/pub/Splines/html/Splines.html b/doc/pub/Splines/html/Splines.html
index 6fc0d1e6f..f8973a018 100644
--- a/doc/pub/Splines/html/Splines.html
+++ b/doc/pub/Splines/html/Splines.html
@@ -213,7 +213,7 @@ MathJax.Hub.Config({
-
Oct 12, 2018
Oct 18, 2018
diff --git a/doc/pub/Splines/ipynb/Splines.ipynb b/doc/pub/Splines/ipynb/Splines.ipynb
index 45e35ec62..ed1aa84c5 100644
--- a/doc/pub/Splines/ipynb/Splines.ipynb
+++ b/doc/pub/Splines/ipynb/Splines.ipynb
@@ -10,7 +10,7 @@
" \n",
"**Morten Hjorth-Jensen**, Department of Physics, University of Oslo and Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University\n",
"\n",
- "Date: **Oct 12, 2018**\n",
+ "Date: **Oct 18, 2018**\n",
"\n",
"Copyright 1999-2018, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license\n",
"\n",
diff --git a/doc/pub/Splines/ipynb/ipynb-Splines-src.tar.gz b/doc/pub/Splines/ipynb/ipynb-Splines-src.tar.gz
index f23e679fa..4e934655f 100644
Binary files a/doc/pub/Splines/ipynb/ipynb-Splines-src.tar.gz and b/doc/pub/Splines/ipynb/ipynb-Splines-src.tar.gz differ
diff --git a/doc/pub/Splines/pdf/Splines-minted.pdf b/doc/pub/Splines/pdf/Splines-minted.pdf
index 1658bdf4e..7f259d0bf 100644
Binary files a/doc/pub/Splines/pdf/Splines-minted.pdf and b/doc/pub/Splines/pdf/Splines-minted.pdf differ
diff --git a/doc/src/LogisticRegression/LogReg.do.txt b/doc/src/LogisticRegression/LogReg.do.txt
index a3c08ca14..67812e97c 100644
--- a/doc/src/LogisticRegression/LogReg.do.txt
+++ b/doc/src/LogisticRegression/LogReg.do.txt
@@ -487,7 +487,7 @@ X_train, X_test, y_train, y_test = skms.train_test_split(
)
!ec
-Using a small training set yields a better accuracy. This will be discussed in the end.
+
!split
===== Logistic regression =====
@@ -672,8 +672,4 @@ transition and is therefore highly non-linear. The ROC curve for the
training data looks quite good, but as the testing data is so far off
we see that we are dealing with an overfit model.
-A previous run with $50\%$ of the data used for training yielded a
-worse performance than using a smaller training set. This again gives
-confidence to the fact that logistic regression is not able to
-correctly fit the Ising model as it is not a linear model.
diff --git a/doc/src/NeuralNet/NeuralNet.do.txt b/doc/src/NeuralNet/NeuralNet.do.txt
index 6d2e4ef48..9fa665949 100644
--- a/doc/src/NeuralNet/NeuralNet.do.txt
+++ b/doc/src/NeuralNet/NeuralNet.do.txt
@@ -2450,4 +2450,115 @@ Here we list some of the important limitations of supervised neural network base
* _Homogeneous data._ Almost all DNNs deal with homogeneous data of one type. It is very hard to design architectures that mix and match data types (i.e.~some continuous variables, some discrete variables, some time series). In applications beyond images, video, and language, this is often what is required. In contrast, ensemble models like random forests or gradient-boosted trees have no difficulty handling mixed data types.
* _Many problems are not about prediction._ In natural science we are often interested in learning something about the underlying distribution that generates the data. In this case, it is often difficult to cast these ideas in a supervised learning setting. While the problems are related, it is possible to make good predictions with a *wrong* model. The model might or might not be useful for understanding the underlying science.
-Some of these remarks are particular to DNNs, others are shared by all supervised learning methods. This motivates the use of unsupervised methods which in part circumnavigate these problems.
+Some of these remarks are particular to DNNs, others are shared by all supervised learning methods. This motivates the use of unsupervised methods which in part circumvent these problems.
+
+
+!split
+===== Convolutional Neural Networks (recognizing images) =====
+
+Convolutional Neural Networks (CNN) are very similar to ordinary Neural Networks.
+
+They are made up of neurons that have learnable weights and
+biases. Each neuron receives some inputs, performs a dot product and
+optionally follows it with a non-linearity. The whole network still
+expresses a single differentiable score function: from the raw image
+pixels on one end to class scores at the other. And they still have a
+loss function (for example Softmax) on the last (fully-connected) layer
+and all the tips/tricks we developed for learning regular Neural
+Networks still apply (back propagation, gradient descent etc etc).
+
+What is the difference? _CNN architectures make the explicit assumption that
+the inputs are images, which allows us to encode certain properties
+into the architecture. These then make the forward function more
+efficient to implement and vastly reduce the amount of parameters in
+the network._
+
+Here we provide only a superficial overview, for the more interested, we recommend highly the course
+"IN5400 – Machine Learning for Image Analysis":"https://www.uio.no/studier/emner/matnat/ifi/IN5400/index-eng.html"
+and the slides of "CS231":"http://cs231n.github.io/convolutional-networks/".
+
+!split
+===== Regular NNs don’t scale well to full images =====
+
+As an example, consider
+an image of size $32\times 32\times 3$ (32 wide, 32 high, 3 color channels), so a
+single fully-connected neuron in a first hidden layer of a regular
+Neural Network would have $32*32*3 = 3072$ weights. This amount still
+seems manageable, but clearly this fully-connected structure does not
+scale to larger images. For example, an image of more respectable
+size, say $200\times 200\times 3$, would lead to neurons that have
+$200*200*3 =120,000$ weights. Moreover, we would almost certainly want to have
+several such neurons, so the parameters would add up quickly! Clearly,
+this full connectivity is wasteful and the huge number of parameters
+would quickly lead to overfitting.
+
+FIGURE: [figslides/nn.jpeg, width=500 frac=0.6] A regular 3-layer Neural Network.
+
+!split
+===== 3D volumes of neurons =====
+
+Convolutional Neural Networks take advantage of the fact that the
+input consists of images and they constrain the architecture in a more
+sensible way.
+
+In particular, unlike a regular Neural Network, the
+layers of a CNN have neurons arranged in 3 dimensions: width,
+height, depth. (Note that the word depth here refers to the third
+dimension of an activation volume, not to the depth of a full Neural
+Network, which can refer to the total number of layers in a network.)
+
+To understand it better, the above example of an image
+with an input volume of
+activations has dimensions $32\times 32\times 3$ (width, height,
+depth respectively).
+
+The neurons in a layer will
+only be connected to a small region of the layer before it, instead of
+all of the neurons in a fully-connected manner. Moreover, the final
+output layer could for this specific image have dimensions $1\times 1 times 10$,
+because by the
+end of the CNN architecture we will reduce the full image into a
+single vector of class scores, arranged along the depth
+dimension.
+
+FIGURE: [figslides/cnn.jpeg, width=500 frac=0.6] A CNN arranges its neurons in three dimensions (width, height, depth), as visualized in one of the layers. Every layer of a CNN transforms the 3D input volume to a 3D output volume of neuron activations. In this example, the red input layer holds the image, so its width and height would be the dimensions of the image, and the depth would be 3 (Red, Green, Blue channels).
+
+
+
+!split
+===== Layers used to build CNNs =====
+
+
+A simple CNN is a sequence of layers, and every layer of a CNN
+transforms one volume of activations to another through a
+differentiable function. We use three main types of layers to build
+CNN architectures: Convolutional Layer, Pooling Layer, and
+Fully-Connected Layer (exactly as seen in regular Neural Networks). We
+will stack these layers to form a full CNN architecture.
+
+A simple CNN for image classification could have the architecture:
+
+* INPUT ($32\times 32 \times 3$) will hold the raw pixel values of the image, in this case an image of width 32, height 32, and with three color channels R,G,B.
+* CONV layer will compute the output of neurons that are connected to local regions in the input, each computing a dot product between their weights and a small region they are connected to in the input volume. This may result in volume such as $[32\times 32\times 12]$ if we decided to use 12 filters.
+* RELU layer will apply an elementwise activation function, such as the $max(0,x)$ thresholding at zero. This leaves the size of the volume unchanged ($[32\times 32\times 12]$).
+* POOL layer will perform a downsampling operation along the spatial dimensions (width, height), resulting in volume such as $[16\times 16\times 12]$.
+* FC (i.e. fully-connected) layer will compute the class scores, resulting in volume of size $[1\times 1\times 10]$, where each of the 10 numbers correspond to a class score, such as among the 10 categories of the MNIST images we considered above . As with ordinary Neural Networks and as the name implies, each neuron in this layer will be connected to all the numbers in the previous volume.
+
+In this way, CNNs transform the original image layer by layer from the original pixel values to the final class scores. Note that some layers contain parameters and other don’t. In particular, the CONV/FC layers perform transformations that are a function of not only the activations in the input volume, but also of the parameters (the weights and biases of the neurons). On the other hand, the RELU/POOL layers will implement a fixed function. The parameters in the CONV/FC layers will be trained with gradient descent so that the class scores that the CNN computes are consistent with the labels in the training set for each image.
+
+
+!split
+===== CNNs in brief =====
+
+In summary:
+
+* A CNN architecture is in the simplest case a list of Layers that transform the image volume into an output volume (e.g. holding the class scores)
+* There are a few distinct types of Layers (e.g. CONV/FC/RELU/POOL are by far the most popular)
+* Each Layer accepts an input 3D volume and transforms it to an output 3D volume through a differentiable function
+* Each Layer may or may not have parameters (e.g. CONV/FC do, RELU/POOL don’t)
+* Each Layer may or may not have additional hyperparameters (e.g. CONV/FC/POOL do, RELU doesn’t)
+
+!split
+===== CNNs in more detail =====
+
+More material to come with examples.
diff --git a/doc/src/NeuralNet/figslides/cnn.jpeg b/doc/src/NeuralNet/figslides/cnn.jpeg
new file mode 100644
index 000000000..67bf3ced7
Binary files /dev/null and b/doc/src/NeuralNet/figslides/cnn.jpeg differ
diff --git a/doc/src/NeuralNet/figslides/nn.jpeg b/doc/src/NeuralNet/figslides/nn.jpeg
new file mode 100644
index 000000000..0a495cfe4
Binary files /dev/null and b/doc/src/NeuralNet/figslides/nn.jpeg differ
diff --git a/doc/src/Regression/Regression.do.txt b/doc/src/Regression/Regression.do.txt
index d09a0c0b7..9e6ccdde7 100644
--- a/doc/src/Regression/Regression.do.txt
+++ b/doc/src/Regression/Regression.do.txt
@@ -3165,7 +3165,7 @@ def get_ridge_weights(x: np.ndarray, y: np.ndarray, _lambda: float) -> np.ndarra
return x.T @ y @ scl.inv(
x.T @ x + np.eye(x.shape[1], x.shape[1]) * _lambda
)
-lambda = 0.1
+_lambda = 0.1
omega_ridge = get_ridge_weights(X_train_own, y_train, np.array([_lambda]))
clf_ridge = skl.Ridge(alpha=_lambda).fit(X_train, y_train)
J_ridge_own = omega_ridge[1:].reshape(L, L)
diff --git a/doc/src/Splines/Splines.do.txt b/doc/src/Splines/Splines.do.txt
index 8a8096fd7..57a5d4059 100644
--- a/doc/src/Splines/Splines.do.txt
+++ b/doc/src/Splines/Splines.do.txt
@@ -1704,8 +1704,6 @@ plt.show()
-
-
!split
===== Using gradient descent methods, limitations =====