lars/FYS-STK4155
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

extra notebooks

Browse Source
This commit is contained in:
Morten Hjorth-Jensen
2024-10-15 06:12:08 +02:00
parent b4d34273d9
commit 549a4515c6
2 changed files with 1648 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/pub/week42/ipynb/additionweek42.ipynb
+844
View File
@@ -0,0 +1,844 @@
{
"cells": [
{
"cell_type": "markdown",
"id": "206229a7",
"metadata": {
"editable": true
},
"source": [
"<!-- HTML file automatically generated from DocOnce source (https://github.com/doconce/doconce/)\n",
"doconce format html additionweek42.do.txt --no_mako -->\n",
"<!-- dom:TITLE: Exercises Week 42: Logistic Regression and Optimization, reminders from week 38 and week 40 -->"
]
},
{
"cell_type": "markdown",
"id": "3fd06200",
"metadata": {
"editable": true
},
"source": [
"# Exercises Week 42: Logistic Regression and Optimization, reminders from week 38 and week 40\n",
"**Morten Hjorth-Jensen**, Department of Physics and Center for Computing in Science Education, University of Oslo and Department of Physics and Astronomy and Facility for Rare Isotope Beams, Michigan State University\n",
"\n",
"Date: **October 14-18, 2024**"
]
},
{
"cell_type": "markdown",
"id": "22f09329",
"metadata": {
"editable": true
},
"source": [
"## The logistic function\n",
"\n",
"A widely studied model, is the \n",
"perceptron model, which is an example of a \"hard classification\" model. We\n",
"have used this model when we discussed neural networks as\n",
"well. Each datapoint is deterministically assigned to a category (i.e\n",
"$y_i=0$ or $y_i=1$). In many cases it is favorable to have a \"soft\"\n",
"classifier that outputs the probability of a given category rather\n",
"than a single value. For example, given $x_i$, the classifier\n",
"outputs the probability of being in a category $k$. Logistic regression\n",
"is the most common example of a so-called soft classifier. In logistic\n",
"regression, the probability that a data point $x_i$\n",
"belongs to a category $y_i=\\{0,1\\}$ is given by the so-called logit function (or Sigmoid) which is meant to represent the likelihood for a given event,"
]
},
{
"cell_type": "markdown",
"id": "02e3244f",
"metadata": {
"editable": true
},
"source": [
"$$\n",
"p(t) = \\frac{1}{1+\\mathrm \\exp{-t}}=\\frac{\\exp{t}}{1+\\mathrm \\exp{t}}.\n",
"$$"
]
},
{
"cell_type": "markdown",
"id": "c999f9a3",
"metadata": {
"editable": true
},
"source": [
"Note that $1-p(t)= p(-t)$."
]
},
{
"cell_type": "markdown",
"id": "e6d247bf",
"metadata": {
"editable": true
},
"source": [
"## Examples of likelihood functions used in logistic regression and nueral networks\n",
"\n",
"The following code plots the logistic function, the step function and other functions we will encounter from here and on."
]
},
{
"cell_type": "code",
"execution_count": 1,
"id": "40e247ae",
"metadata": {
"collapsed": false,
"editable": true
},
"outputs": [],
"source": [
"%matplotlib inline\n",
"\n",
"\"\"\"The sigmoid function (or the logistic curve) is a\n",
"function that takes any real number, z, and outputs a number (0,1).\n",
"It is useful in neural networks for assigning weights on a relative scale.\n",
"The value z is the weighted sum of parameters involved in the learning algorithm.\"\"\"\n",
"\n",
"import numpy\n",
"import matplotlib.pyplot as plt\n",
"import math as mt\n",
"\n",
"z = numpy.arange(-5, 5, .1)\n",
"sigma_fn = numpy.vectorize(lambda z: 1/(1+numpy.exp(-z)))\n",
"sigma = sigma_fn(z)\n",
"\n",
"fig = plt.figure()\n",
"ax = fig.add_subplot(111)\n",
"ax.plot(z, sigma)\n",
"ax.set_ylim([-0.1, 1.1])\n",
"ax.set_xlim([-5,5])\n",
"ax.grid(True)\n",
"ax.set_xlabel('z')\n",
"ax.set_title('sigmoid function')\n",
"\n",
"plt.show()\n",
"\n",
"\"\"\"Step Function\"\"\"\n",
"z = numpy.arange(-5, 5, .02)\n",
"step_fn = numpy.vectorize(lambda z: 1.0 if z >= 0.0 else 0.0)\n",
"step = step_fn(z)\n",
"\n",
"fig = plt.figure()\n",
"ax = fig.add_subplot(111)\n",
"ax.plot(z, step)\n",
"ax.set_ylim([-0.5, 1.5])\n",
"ax.set_xlim([-5,5])\n",
"ax.grid(True)\n",
"ax.set_xlabel('z')\n",
"ax.set_title('step function')\n",
"\n",
"plt.show()\n",
"\n",
"\"\"\"tanh Function\"\"\"\n",
"z = numpy.arange(-2*mt.pi, 2*mt.pi, 0.1)\n",
"t = numpy.tanh(z)\n",
"\n",
"fig = plt.figure()\n",
"ax = fig.add_subplot(111)\n",
"ax.plot(z, t)\n",
"ax.set_ylim([-1.0, 1.0])\n",
"ax.set_xlim([-2*mt.pi,2*mt.pi])\n",
"ax.grid(True)\n",
"ax.set_xlabel('z')\n",
"ax.set_title('tanh function')\n",
"\n",
"plt.show()"
]
},
{
"cell_type": "markdown",
"id": "c0072e88",
"metadata": {
"editable": true
},
"source": [
"## Two parameters\n",
"\n",
"We assume now that we have two classes with $y_i$ either $0$ or $1$. Furthermore we assume also that we have only two parameters $\\beta$ in our fitting of the Sigmoid function, that is we define probabilities"
]
},
{
"cell_type": "markdown",
"id": "47c8d6f0",
"metadata": {
"editable": true
},
"source": [
"$$\n",
"\\begin{align*}\n",
"p(y_i=1|x_i,\\boldsymbol{\\beta}) &= \\frac{\\exp{(\\beta_0+\\beta_1x_i)}}{1+\\exp{(\\beta_0+\\beta_1x_i)}},\\nonumber\\\\\n",
"p(y_i=0|x_i,\\boldsymbol{\\beta}) &= 1 - p(y_i=1|x_i,\\boldsymbol{\\beta}),\n",
"\\end{align*}\n",
"$$"
]
},
{
"cell_type": "markdown",
"id": "7f0dbf70",
"metadata": {
"editable": true
},
"source": [
"where $\\boldsymbol{\\beta}$ are the weights we wish to extract from data, in our case $\\beta_0$ and $\\beta_1$. \n",
"\n",
"Note that we used"
]
},
{
"cell_type": "markdown",
"id": "5666ab6f",
"metadata": {
"editable": true
},
"source": [
"$$\n",
"p(y_i=0\\vert x_i, \\boldsymbol{\\beta}) = 1-p(y_i=1\\vert x_i, \\boldsymbol{\\beta}).\n",
"$$"
]
},
{
"cell_type": "markdown",
"id": "3a5bf836",
"metadata": {
"editable": true
},
"source": [
"## The cost function\n",
"\n",
"Reordering the logarithms, we can rewrite the **cost/loss** function as"
]
},
{
"cell_type": "markdown",
"id": "865c1168",
"metadata": {
"editable": true
},
"source": [
"$$\n",
"\\mathcal{C}(\\boldsymbol{\\beta}) = \\sum_{i=1}^n \\left(y_i(\\beta_0+\\beta_1x_i) -\\log{(1+\\exp{(\\beta_0+\\beta_1x_i)})}\\right).\n",
"$$"
]
},
{
"cell_type": "markdown",
"id": "a319692e",
"metadata": {
"editable": true
},
"source": [
"The maximum likelihood estimator is defined as the set of parameters that maximize the log-likelihood where we maximize with respect to $\\beta$.\n",
"Since the cost (error) function is just the negative log-likelihood, for logistic regression we have that"
]
},
{
"cell_type": "markdown",
"id": "c6a573cc",
"metadata": {
"editable": true
},
"source": [
"$$\n",
"\\mathcal{C}(\\boldsymbol{\\beta})=-\\sum_{i=1}^n \\left(y_i(\\beta_0+\\beta_1x_i) -\\log{(1+\\exp{(\\beta_0+\\beta_1x_i)})}\\right).\n",
"$$"
]
},
{
"cell_type": "markdown",
"id": "9a2cce7c",
"metadata": {
"editable": true
},
"source": [
"This equation is known in statistics as the **cross entropy**. Finally, we note that just as in linear regression, \n",
"in practice we often supplement the cross-entropy with additional regularization terms, usually $L_1$ and $L_2$ regularization as we did for Ridge and Lasso regression."
]
},
{
"cell_type": "markdown",
"id": "5b52cf1c",
"metadata": {
"editable": true
},
"source": [
"## Minimizing the cross entropy\n",
"\n",
"The cross entropy is a convex function of the weights $\\boldsymbol{\\beta}$ and,\n",
"therefore, any local minimizer is a global minimizer. \n",
"\n",
"Minimizing this\n",
"cost function with respect to the two parameters $\\beta_0$ and $\\beta_1$ we obtain"
]
},
{
"cell_type": "markdown",
"id": "e48db3ac",
"metadata": {
"editable": true
},
"source": [
"$$\n",
"\\frac{\\partial \\mathcal{C}(\\boldsymbol{\\beta})}{\\partial \\beta_0} = -\\sum_{i=1}^n \\left(y_i -\\frac{\\exp{(\\beta_0+\\beta_1x_i)}}{1+\\exp{(\\beta_0+\\beta_1x_i)}}\\right),\n",
"$$"
]
},
{
"cell_type": "markdown",
"id": "9844a743",
"metadata": {
"editable": true
},
"source": [
"and"
]
},
{
"cell_type": "markdown",
"id": "bc707870",
"metadata": {
"editable": true
},
"source": [
"$$\n",
"\\frac{\\partial \\mathcal{C}(\\boldsymbol{\\beta})}{\\partial \\beta_1} = -\\sum_{i=1}^n \\left(y_ix_i -x_i\\frac{\\exp{(\\beta_0+\\beta_1x_i)}}{1+\\exp{(\\beta_0+\\beta_1x_i)}}\\right).\n",
"$$"
]
},
{
"cell_type": "markdown",
"id": "f67203cd",
"metadata": {
"editable": true
},
"source": [
"## A more compact expression\n",
"\n",
"Let us now define a vector $\\boldsymbol{y}$ with $n$ elements $y_i$, an\n",
"$n\\times p$ matrix $\\boldsymbol{X}$ which contains the $x_i$ values and a\n",
"vector $\\boldsymbol{p}$ of fitted probabilities $p(y_i\\vert x_i,\\boldsymbol{\\beta})$. We can rewrite in a more compact form the first\n",
"derivative of cost function as"
]
},
{
"cell_type": "markdown",
"id": "d863b540",
"metadata": {
"editable": true
},
"source": [
"$$\n",
"\\frac{\\partial \\mathcal{C}(\\boldsymbol{\\beta})}{\\partial \\boldsymbol{\\beta}} = -\\boldsymbol{X}^T\\left(\\boldsymbol{y}-\\boldsymbol{p}\\right).\n",
"$$"
]
},
{
"cell_type": "markdown",
"id": "e769467d",
"metadata": {
"editable": true
},
"source": [
"If we in addition define a diagonal matrix $\\boldsymbol{W}$ with elements \n",
"$p(y_i\\vert x_i,\\boldsymbol{\\beta})(1-p(y_i\\vert x_i,\\boldsymbol{\\beta})$, we can obtain a compact expression of the second derivative as"
]
},
{
"cell_type": "markdown",
"id": "074ba1e7",
"metadata": {
"editable": true
},
"source": [
"$$\n",
"\\frac{\\partial^2 \\mathcal{C}(\\boldsymbol{\\beta})}{\\partial \\boldsymbol{\\beta}\\partial \\boldsymbol{\\beta}^T} = \\boldsymbol{X}^T\\boldsymbol{W}\\boldsymbol{X}.\n",
"$$"
]
},
{
"cell_type": "markdown",
"id": "784aef3a",
"metadata": {
"editable": true
},
"source": [
"## Extending to more predictors\n",
"\n",
"Within a binary classification problem, we can easily expand our model to include multiple predictors. Our ratio between likelihoods is then with $p$ predictors"
]
},
{
"cell_type": "markdown",
"id": "4d71c488",
"metadata": {
"editable": true
},
"source": [
"$$\n",
"\\log{ \\frac{p(\\boldsymbol{\\beta}\\boldsymbol{x})}{1-p(\\boldsymbol{\\beta}\\boldsymbol{x})}} = \\beta_0+\\beta_1x_1+\\beta_2x_2+\\dots+\\beta_px_p.\n",
"$$"
]
},
{
"cell_type": "markdown",
"id": "2a4a1497",
"metadata": {
"editable": true
},
"source": [
"Here we defined $\\boldsymbol{x}=[1,x_1,x_2,\\dots,x_p]$ and $\\boldsymbol{\\beta}=[\\beta_0, \\beta_1, \\dots, \\beta_p]$ leading to"
]
},
{
"cell_type": "markdown",
"id": "7435cbe3",
"metadata": {
"editable": true
},
"source": [
"$$\n",
"p(\\boldsymbol{\\beta}\\boldsymbol{x})=\\frac{ \\exp{(\\beta_0+\\beta_1x_1+\\beta_2x_2+\\dots+\\beta_px_p)}}{1+\\exp{(\\beta_0+\\beta_1x_1+\\beta_2x_2+\\dots+\\beta_px_p)}}.\n",
"$$"
]
},
{
"cell_type": "markdown",
"id": "1cb32178",
"metadata": {
"editable": true
},
"source": [
"## Including more classes\n",
"\n",
"Till now we have mainly focused on two classes, the so-called binary\n",
"system. Suppose we wish to extend to $K$ classes. Let us for the sake\n",
"of simplicity assume we have only two predictors. We have then following model"
]
},
{
"cell_type": "markdown",
"id": "e620bb11",
"metadata": {
"editable": true
},
"source": [
"$$\n",
"\\log{\\frac{p(C=1\\vert x)}{p(K\\vert x)}} = \\beta_{10}+\\beta_{11}x_1,\n",
"$$"
]
},
{
"cell_type": "markdown",
"id": "b4851e37",
"metadata": {
"editable": true
},
"source": [
"and"
]
},
{
"cell_type": "markdown",
"id": "3999e7f2",
"metadata": {
"editable": true
},
"source": [
"$$\n",
"\\log{\\frac{p(C=2\\vert x)}{p(K\\vert x)}} = \\beta_{20}+\\beta_{21}x_1,\n",
"$$"
]
},
{
"cell_type": "markdown",
"id": "7200162c",
"metadata": {
"editable": true
},
"source": [
"and so on till the class $C=K-1$ class"
]
},
{
"cell_type": "markdown",
"id": "4795f8a9",
"metadata": {
"editable": true
},
"source": [
"$$\n",
"\\log{\\frac{p(C=K-1\\vert x)}{p(K\\vert x)}} = \\beta_{(K-1)0}+\\beta_{(K-1)1}x_1,\n",
"$$"
]
},
{
"cell_type": "markdown",
"id": "c55dc7f6",
"metadata": {
"editable": true
},
"source": [
"and the model is specified in term of $K-1$ so-called log-odds or\n",
"**logit** transformations."
]
},
{
"cell_type": "markdown",
"id": "c8cdb1d4",
"metadata": {
"editable": true
},
"source": [
"## More classes\n",
"\n",
"In our discussion of neural networks we will encounter the above again\n",
"in terms of a slightly modified function, the so-called **Softmax** function.\n",
"\n",
"The softmax function is used in various multiclass classification\n",
"methods, such as multinomial logistic regression (also known as\n",
"softmax regression), multiclass linear discriminant analysis, naive\n",
"Bayes classifiers, and artificial neural networks. Specifically, in\n",
"multinomial logistic regression and linear discriminant analysis, the\n",
"input to the function is the result of $K$ distinct linear functions,\n",
"and the predicted probability for the $k$-th class given a sample\n",
"vector $\\boldsymbol{x}$ and a weighting vector $\\boldsymbol{\\beta}$ is (with two\n",
"predictors):"
]
},
{
"cell_type": "markdown",
"id": "89c9a8d4",
"metadata": {
"editable": true
},
"source": [
"$$\n",
"p(C=k\\vert \\mathbf {x} )=\\frac{\\exp{(\\beta_{k0}+\\beta_{k1}x_1)}}{1+\\sum_{l=1}^{K-1}\\exp{(\\beta_{l0}+\\beta_{l1}x_1)}}.\n",
"$$"
]
},
{
"cell_type": "markdown",
"id": "3b4446bd",
"metadata": {
"editable": true
},
"source": [
"It is easy to extend to more predictors. The final class is"
]
},
{
"cell_type": "markdown",
"id": "801a59a0",
"metadata": {
"editable": true
},
"source": [
"$$\n",
"p(C=K\\vert \\mathbf {x} )=\\frac{1}{1+\\sum_{l=1}^{K-1}\\exp{(\\beta_{l0}+\\beta_{l1}x_1)}},\n",
"$$"
]
},
{
"cell_type": "markdown",
"id": "a0ae9ae9",
"metadata": {
"editable": true
},
"source": [
"and they sum to one."
]
},
{
"cell_type": "markdown",
"id": "ace600ee",
"metadata": {
"editable": true
},
"source": [
"## Wisconsin Cancer Data\n",
"\n",
"We show here how we can use a simple regression case on the breast\n",
"cancer data using Logistic regression as our algorithm for\n",
"classification."
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "56c9f070",
"metadata": {
"collapsed": false,
"editable": true
},
"outputs": [],
"source": [
"import matplotlib.pyplot as plt\n",
"import numpy as np\n",
"from sklearn.model_selection import train_test_split \n",
"from sklearn.datasets import load_breast_cancer\n",
"from sklearn.linear_model import LogisticRegression\n",
"\n",
"# Load the data\n",
"cancer = load_breast_cancer()\n",
"\n",
"X_train, X_test, y_train, y_test = train_test_split(cancer.data,cancer.target,random_state=0)\n",
"print(X_train.shape)\n",
"print(X_test.shape)\n",
"# Logistic Regression\n",
"logreg = LogisticRegression(solver='lbfgs')\n",
"logreg.fit(X_train, y_train)\n",
"print(\"Test set accuracy with Logistic Regression: {:.2f}\".format(logreg.score(X_test,y_test)))"
]
},
{
"cell_type": "markdown",
"id": "31b448d5",
"metadata": {
"editable": true
},
"source": [
"## Using the correlation matrix\n",
"\n",
"In addition to the above scores, we could also study the covariance (and the correlation matrix).\n",
"We use **Pandas** to compute the correlation matrix."
]
},
{
"cell_type": "code",
"execution_count": 3,
"id": "9b7aa971",
"metadata": {
"collapsed": false,
"editable": true
},
"outputs": [],
"source": [
"import matplotlib.pyplot as plt\n",
"import numpy as np\n",
"from sklearn.model_selection import train_test_split \n",
"from sklearn.datasets import load_breast_cancer\n",
"from sklearn.linear_model import LogisticRegression\n",
"cancer = load_breast_cancer()\n",
"import pandas as pd\n",
"# Making a data frame\n",
"cancerpd = pd.DataFrame(cancer.data, columns=cancer.feature_names)\n",
"\n",
"fig, axes = plt.subplots(15,2,figsize=(10,20))\n",
"malignant = cancer.data[cancer.target == 0]\n",
"benign = cancer.data[cancer.target == 1]\n",
"ax = axes.ravel()\n",
"\n",
"for i in range(30):\n",
" _, bins = np.histogram(cancer.data[:,i], bins =50)\n",
" ax[i].hist(malignant[:,i], bins = bins, alpha = 0.5)\n",
" ax[i].hist(benign[:,i], bins = bins, alpha = 0.5)\n",
" ax[i].set_title(cancer.feature_names[i])\n",
" ax[i].set_yticks(())\n",
"ax[0].set_xlabel(\"Feature magnitude\")\n",
"ax[0].set_ylabel(\"Frequency\")\n",
"ax[0].legend([\"Malignant\", \"Benign\"], loc =\"best\")\n",
"fig.tight_layout()\n",
"plt.show()\n",
"\n",
"import seaborn as sns\n",
"correlation_matrix = cancerpd.corr().round(1)\n",
"# use the heatmap function from seaborn to plot the correlation matrix\n",
"# annot = True to print the values inside the square\n",
"plt.figure(figsize=(15,8))\n",
"sns.heatmap(data=correlation_matrix, annot=True)\n",
"plt.show()"
]
},
{
"cell_type": "markdown",
"id": "e48192a2",
"metadata": {
"editable": true
},
"source": [
"## Discussing the correlation data\n",
"\n",
"In the above example we note two things. In the first plot we display\n",
"the overlap of benign and malignant tumors as functions of the various\n",
"features in the Wisconsing breast cancer data set. We see that for\n",
"some of the features we can distinguish clearly the benign and\n",
"malignant cases while for other features we cannot. This can point to\n",
"us which features may be of greater interest when we wish to classify\n",
"a benign or not benign tumour.\n",
"\n",
"In the second figure we have computed the so-called correlation\n",
"matrix, which in our case with thirty features becomes a $30\\times 30$\n",
"matrix.\n",
"\n",
"We constructed this matrix using **pandas** via the statements"
]
},
{
"cell_type": "code",
"execution_count": 4,
"id": "be07af84",
"metadata": {
"collapsed": false,
"editable": true
},
"outputs": [],
"source": [
"cancerpd = pd.DataFrame(cancer.data, columns=cancer.feature_names)"
]
},
{
"cell_type": "markdown",
"id": "1c5f852a",
"metadata": {
"editable": true
},
"source": [
"and then"
]
},
{
"cell_type": "code",
"execution_count": 5,
"id": "2925cdd6",
"metadata": {
"collapsed": false,
"editable": true
},
"outputs": [],
"source": [
"correlation_matrix = cancerpd.corr().round(1)"
]
},
{
"cell_type": "markdown",
"id": "7fdedbd7",
"metadata": {
"editable": true
},
"source": [
"Diagonalizing this matrix we can in turn say something about which\n",
"features are of relevance and which are not. This leads us to\n",
"the classical Principal Component Analysis (PCA) theorem with\n",
"applications. This will be discussed later this semester ([week 43](https://compphysics.github.io/MachineLearning/doc/pub/week43/html/week43-bs.html))."
]
},
{
"cell_type": "markdown",
"id": "62d43eb5",
"metadata": {
"editable": true
},
"source": [
"## Other measures in classification studies: Cancer Data again"
]
},
{
"cell_type": "code",
"execution_count": 6,
"id": "371f0a26",
"metadata": {
"collapsed": false,
"editable": true
},
"outputs": [],
"source": [
"import matplotlib.pyplot as plt\n",
"import numpy as np\n",
"from sklearn.model_selection import train_test_split \n",
"from sklearn.datasets import load_breast_cancer\n",
"from sklearn.linear_model import LogisticRegression\n",
"\n",
"# Load the data\n",
"cancer = load_breast_cancer()\n",
"\n",
"X_train, X_test, y_train, y_test = train_test_split(cancer.data,cancer.target,random_state=0)\n",
"print(X_train.shape)\n",
"print(X_test.shape)\n",
"# Logistic Regression\n",
"logreg = LogisticRegression(solver='lbfgs')\n",
"logreg.fit(X_train, y_train)\n",
"\n",
"from sklearn.preprocessing import LabelEncoder\n",
"from sklearn.model_selection import cross_validate\n",
"#Cross validation\n",
"accuracy = cross_validate(logreg,X_test,y_test,cv=10)['test_score']\n",
"print(accuracy)\n",
"print(\"Test set accuracy with Logistic Regression: {:.2f}\".format(logreg.score(X_test,y_test)))\n",
"\n",
"import scikitplot as skplt\n",
"y_pred = logreg.predict(X_test)\n",
"skplt.metrics.plot_confusion_matrix(y_test, y_pred, normalize=True)\n",
"plt.show()\n",
"y_probas = logreg.predict_proba(X_test)\n",
"skplt.metrics.plot_roc(y_test, y_probas)\n",
"plt.show()\n",
"skplt.metrics.plot_cumulative_gain(y_test, y_probas)\n",
"plt.show()"
]
},
{
"cell_type": "markdown",
"id": "448d70b5",
"metadata": {
"editable": true
},
"source": [
"## Gradient descent and Logistic regression\n",
"\n",
"We complete these examples by adding a simple code for\n",
"Logistic regression. Note the more general approach with a class for\n",
"the method. Here we use a so-called **AND** gate for our data set."
]
},
{
"cell_type": "code",
"execution_count": 7,
"id": "1ce6966d",
"metadata": {
"collapsed": false,
"editable": true
},
"outputs": [],
"source": [
"import numpy as np\n",
"class LogisticRegression:\n",
" def __init__(self, learning_rate=0.01, num_iterations=1000):\n",
" self.learning_rate = learning_rate\n",
" self.num_iterations = num_iterations\n",
" self.beta_logreg = None\n",
" def sigmoid(self, z):\n",
" return 1 / (1 + np.exp(-z))\n",
" def GDfit(self, X, y):\n",
" n_data, num_features = X.shape\n",
" self.beta_logreg = np.zeros(num_features)\n",
" for _ in range(self.num_iterations):\n",
" linear_model = X @ self.beta_logreg\n",
" y_predicted = self.sigmoid(linear_model)\n",
" # Gradient calculation\n",
" gradient = (X.T @ (y_predicted - y))/n_data\n",
" # Update beta_logreg\n",
" self.beta_logreg -= self.learning_rate*gradient\n",
" def predict(self, X):\n",
" linear_model = X @ self.beta_logreg\n",
" y_predicted = self.sigmoid(linear_model)\n",
" return [1 if i >= 0.5 else 0 for i in y_predicted]\n",
"# Example usage\n",
"if __name__ == \"__main__\":\n",
" # Sample data\n",
" X = np.array([[0, 0], [1, 0], [0, 1], [1, 1]])\n",
" y = np.array([0, 0, 0, 1]) # This is an AND gate\n",
" model = LogisticRegression(learning_rate=0.01, num_iterations=1000)\n",
" model.GDfit(X, y)\n",
" predictions = model.predict(X)\n",
" print(\"Predictions:\", predictions)"
]
}
],
"metadata": {},
"nbformat": 4,
"nbformat_minor": 5
}
doc/pub/week42/ipynb/exercisesweek42.ipynb
+804
View File
@@ -0,0 +1,804 @@
{
"cells": [
{
"cell_type": "markdown",
"id": "4b4c06bc",
"metadata": {},
"source": [
"<!-- HTML file automatically generated from DocOnce source (https://github.com/doconce/doconce/)\n",
"doconce format html exercisesweek41.do.txt -->\n",
"<!-- dom:TITLE: Exercises week 41 -->\n"
]
},
{
"cell_type": "markdown",
"id": "bcb25e64",
"metadata": {},
"source": [
"# Exercises week 42\n",
"\n",
"**October 14-18, 2024**\n",
"\n",
"Date: **Deadline is Friday October 18 at midnight**\n"
]
},
{
"cell_type": "markdown",
"id": "bb01f126",
"metadata": {},
"source": [
"# Overarching aims of the exercises this week\n",
"\n",
"This week, you will implement the entire feed-forward pass of a neural network! Next week you will compute the gradient of the network by implementing back-propagation manually, and by using autograd which does back-propagation for you (much easier!). Next week, you will also use the gradient to optimize the network with a gradient method! However, there is an optional exercise this week to get started on training the network and getting good results!\n",
"\n",
"We recommend that you do the exercises this week by editing and running this notebook file, as it includes some checks along the way that you have implemented the pieces of the feed-forward pass correctly, and running small parts of the code at a time will be important for understanding the methods.\n",
"\n",
"If you have trouble running a notebook, you can run this notebook in google colab instead (https://colab.research.google.com/drive/1zKibVQf-iAYaAn2-GlKfgRjHtLnPlBX4#offline=true&sandboxMode=true), an updated link will be provided on the course discord (you can also send an email to k.h.fredly@fys.uio.no if you encounter any trouble), though we recommend that you set up VSCode and your python environment to run code like this locally.\n",
"\n",
"First, here are some functions you are going to need, don't change this cell. If you are unable to import autograd, just swap in normal numpy until you want to do the final optional exercise.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "c6f61b09",
"metadata": {},
"outputs": [],
"source": [
"import autograd.numpy as np # We need to use this numpy wrapper to make automatic differentiation work later\n",
"from sklearn import datasets\n",
"import matplotlib.pyplot as plt\n",
"from sklearn.metrics import accuracy_score\n",
"\n",
"\n",
"# Defining some activation functions\n",
"def ReLU(z):\n",
" return np.where(z > 0, z, 0)\n",
"\n",
"\n",
"def sigmoid(z):\n",
" return 1 / (1 + np.exp(-z))\n",
"\n",
"\n",
"def softmax(z):\n",
" \"\"\"Compute softmax values for each set of scores in the rows of the matrix z.\n",
" Used with batched input data.\"\"\"\n",
" e_z = np.exp(z - np.max(z, axis=0))\n",
" return e_z / np.sum(e_z, axis=1)[:, np.newaxis]\n",
"\n",
"\n",
"def softmax_vec(z):\n",
" \"\"\"Compute softmax values for each set of scores in the vector z.\n",
" Use this function when you use the activation function on one vector at a time\"\"\"\n",
" e_z = np.exp(z - np.max(z))\n",
" return e_z / np.sum(e_z)"
]
},
{
"cell_type": "markdown",
"id": "6248ec53",
"metadata": {},
"source": [
"# Exercise 1\n",
"\n",
"In this exercise you will compute the activation of the first layer. You only need to change the code in the cells right below an exercise, the rest works out of the box. Feel free to make changes and see how stuff works though!\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "37f30740",
"metadata": {},
"outputs": [],
"source": [
"np.random.seed(2024)\n",
"\n",
"x = np.random.randn(2) # network input. This is a single input with two features\n",
"W1 = np.random.randn(4, 2) # first layer weights"
]
},
{
"cell_type": "markdown",
"id": "4ed2cf3d",
"metadata": {},
"source": [
"**a)** Given the shape of the first layer weight matrix, what is the input shape of the neural network? What is the output shape of the first layer?\n"
]
},
{
"cell_type": "markdown",
"id": "edf7217b",
"metadata": {},
"source": [
"**b)** Define the bias of the first layer, `b1`with the correct shape. (Run the next cell right after the previous to get the random generated values to line up with the test solution below)\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "2129c19f",
"metadata": {},
"outputs": [],
"source": [
"b1 = ..."
]
},
{
"cell_type": "markdown",
"id": "09e8d453",
"metadata": {},
"source": [
"**c)** Compute the intermediary `z1` for the first layer\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "6837119b",
"metadata": {},
"outputs": [],
"source": [
"z1 = ..."
]
},
{
"cell_type": "markdown",
"id": "6f71374e",
"metadata": {},
"source": [
"**d)** Compute the activation `a1` for the first layer using the ReLU activation function defined earlier.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "8d41ed19",
"metadata": {},
"outputs": [],
"source": [
"a1 = ..."
]
},
{
"cell_type": "markdown",
"id": "088710c0",
"metadata": {},
"source": [
"Confirm that you got the correct activation with the test below. Make sure that you define `b1` with the randn function right after you define `W1`.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "4d2f54b4",
"metadata": {},
"outputs": [],
"source": [
"sol1 = np.array([0.60610368, 4.0076268, 0.0, 0.56469864])\n",
"\n",
"print(np.allclose(a1, sol1))"
]
},
{
"cell_type": "markdown",
"id": "7fb0cf46",
"metadata": {},
"source": [
"# Exercise 2\n",
"\n",
"Now we will add a layer to the network with an output of length 8 and ReLU activation.\n",
"\n",
"**a)** What is the input of the second layer? What is its shape?\n",
"\n",
"**b)** Define the weight and bias of the second layer with the right shapes.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "00063acf",
"metadata": {},
"outputs": [],
"source": [
"W2 = ...\n",
"b2 = ..."
]
},
{
"cell_type": "markdown",
"id": "5bd7d84b",
"metadata": {},
"source": [
"**c)** Compute the intermediary `z2` and activation `a2` for the second layer.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "2fd0383d",
"metadata": {},
"outputs": [],
"source": [
"z2 = ...\n",
"a2 = ..."
]
},
{
"cell_type": "markdown",
"id": "1b5daae5",
"metadata": {},
"source": [
"Confirm that you got the correct activation shape with the test below.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "f7f2f8a1",
"metadata": {},
"outputs": [],
"source": [
"print(\n",
" np.allclose(np.exp(len(a2)), 2980.9579870417283)\n",
") # This should evaluate to True if a2 has the correct shape :)"
]
},
{
"cell_type": "markdown",
"id": "3759620d",
"metadata": {},
"source": [
"# Exercise 3\n",
"\n",
"We often want our neural networks to have many layers of varying sizes. To avoid writing very long and error-prone code where we explicitly define and evaluate each layer we should keep all our layers in a single variable which is easy to create and use.\n",
"\n",
"**a)** Complete the function below so that it returns a list `layers` of weight and bias tuples `(W, b)` for each layer, in order, with the correct shapes that we can use later as our network parameters.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "c58f10f9",
"metadata": {},
"outputs": [],
"source": [
"def create_layers(network_input_size, layer_output_sizes):\n",
" layers = []\n",
"\n",
" i_size = network_input_size\n",
" for layer_output_size in layer_output_sizes:\n",
" W = ...\n",
" b = ...\n",
" layers.append((W, b))\n",
"\n",
" i_size = layer_output_size\n",
" return layers"
]
},
{
"cell_type": "markdown",
"id": "bdc0cda2",
"metadata": {},
"source": [
"**b)** Comple the function below so that it evaluates the intermediary `z` and activation `a` for each layer, with ReLU actication, and returns the final activation `a`. This is the complete feed-forward pass, a full neural network!\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "5262df05",
"metadata": {},
"outputs": [],
"source": [
"def feed_forward_all_relu(layers, input):\n",
" a = input\n",
" for W, b in layers:\n",
" z = ...\n",
" a = ...\n",
" return a"
]
},
{
"cell_type": "markdown",
"id": "245adbcb",
"metadata": {},
"source": [
"**c)** Create a network with input size 8 and layers with output sizes 10, 16, 6, 2. Evaluate it and make sure that you get the correct size vectors along the way.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "89a8f70d",
"metadata": {},
"outputs": [],
"source": [
"input_size = ...\n",
"layer_output_sizes = [...]\n",
"\n",
"x = np.random.rand(input_size)\n",
"layers = ...\n",
"predict = ...\n",
"print(predict)"
]
},
{
"cell_type": "markdown",
"id": "0da7fd52",
"metadata": {},
"source": [
"**d)** Why is a neural network with no activation functions always mathematically equivelent to a neural network with only one layer?\n"
]
},
{
"cell_type": "markdown",
"id": "306d8b7c",
"metadata": {},
"source": [
"# Exercise 4 - Custom activation for each layer\n"
]
},
{
"cell_type": "markdown",
"id": "221c7b6c",
"metadata": {},
"source": [
"So far, every layer has used the same activation, ReLU. We often want to use other types of activation however, so we need to update our code to support multiple types of activation functions. Make sure that you have completed every previous exercise before trying this one.\n"
]
},
{
"cell_type": "markdown",
"id": "10896d06",
"metadata": {},
"source": [
"**a)** Complete the `feed_forward` function which accepts a list of activation functions as an argument, and which evaluates these activation functions at each layer.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "de062369",
"metadata": {},
"outputs": [],
"source": [
"def feed_forward(input, layers, activation_funcs):\n",
" a = input\n",
" for (W, b), activation_func in zip(layers, activation_funcs):\n",
" z = ...\n",
" a = ...\n",
" return a"
]
},
{
"cell_type": "markdown",
"id": "8f7df363",
"metadata": {},
"source": [
"**b)** You are now given a list with three activation functions, two ReLU and one sigmoid. (Don't call them yet! you can make a list with function names as elements, and then call these elements of the list later. If you add other functions than the ones defined at the start of the notebook, make sure everything is defined using autograd's numpy wrapper, like above, since we want to use automatic differentiation on all of these functions later.)\n",
"\n",
"Evaluate a network with three layers and these activation functions.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "301b46dc",
"metadata": {},
"outputs": [],
"source": [
"network_input_size = ...\n",
"layer_output_sizes = [...]\n",
"activation_funcs = [ReLU, ReLU, sigmoid]\n",
"layers = ...\n",
"\n",
"x = np.random.randn(network_input_size)\n",
"feed_forward(x, layers, activation_funcs)"
]
},
{
"cell_type": "markdown",
"id": "9c914fd0",
"metadata": {},
"source": [
"**c)** How does the output of the network change if you use sigmoid in the hidden layers and ReLU in the output layer?\n"
]
},
{
"cell_type": "markdown",
"id": "a8d6c425",
"metadata": {},
"source": [
"# Exercise 5 - Processing multiple inputs at once\n"
]
},
{
"cell_type": "markdown",
"id": "0f4330a4",
"metadata": {},
"source": [
"So far, the feed forward function has taken one input vector as an input. This vector then undergoes a linear transformation and then an element-wise non-linear operation for each layer. This approach of sending one vector in at a time is great for interpreting how the network transforms data with its linear and non-linear operations, but not the best for numerical efficiency. Now, we want to be able to send many inputs through the network at once. This will make the code a bit harder to understand, but it will make it faster, and more compact. It will be worth the trouble.\n",
"\n",
"To process multiple inputs at once, while still performing the same operations, you will only need to flip a couple things around.\n"
]
},
{
"cell_type": "markdown",
"id": "17023bb7",
"metadata": {},
"source": [
"**a)** Complete the function `create_layers_batch` so that the weight matrix is the transpose of what it was when you only sent in one input at a time.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "a241fd79",
"metadata": {},
"outputs": [],
"source": [
"def create_layers_batch(network_input_size, layer_output_sizes):\n",
" layers = []\n",
"\n",
" i_size = network_input_size\n",
" for layer_output_size in layer_output_sizes:\n",
" W = ...\n",
" b = ...\n",
" layers.append((W, b))\n",
"\n",
" i_size = layer_output_size\n",
" return layers"
]
},
{
"cell_type": "markdown",
"id": "a6349db6",
"metadata": {},
"source": [
"**b)** Make a matrix of inputs with the shape (number of features, number of inputs), you choose the number of inputs and features per input. Then complete the function `feed_forward_batch` so that you can process this matrix of inputs with only one matrix multiplication and one broadcasted vector addition per layer. (Hint: You will only need to swap two variable around from your previous implementation, but remember to test that you get the same results for equivelent inputs!)\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "425f3bcc",
"metadata": {},
"outputs": [],
"source": [
"inputs = np.random.rand(1000, 4)\n",
"\n",
"\n",
"def feed_forward_batch(inputs, layers, activation_funcs):\n",
" a = inputs\n",
" for (W, b), activation_func in zip(layers, activation_funcs):\n",
" z = ...\n",
" a = ...\n",
" return a"
]
},
{
"cell_type": "markdown",
"id": "efd07b4e",
"metadata": {},
"source": [
"**c)** Create and evaluate a neural network with 4 inputs and layers with output sizes 12, 10, 3 and activations ReLU, ReLU, softmax.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "ce6fcc2f",
"metadata": {},
"outputs": [],
"source": [
"network_input_size = ...\n",
"layer_output_sizes = [...]\n",
"activation_funcs = [...]\n",
"layers = create_layers_batch(network_input_size, layer_output_sizes)\n",
"\n",
"x = np.random.randn(network_input_size)\n",
"feed_forward_batch(inputs, layers, activation_funcs)"
]
},
{
"cell_type": "markdown",
"id": "87999271",
"metadata": {},
"source": [
"You should use this batched approach moving forward, as it will lead to much more compact code. However, remember that each input is still treated separately, and that you will need to keep in mind the transposed weight matrix and other details when implementing backpropagation.\n"
]
},
{
"cell_type": "markdown",
"id": "237eb782",
"metadata": {},
"source": [
"# Exercise 6 - Predicting on real data\n"
]
},
{
"cell_type": "markdown",
"id": "54d5fde7",
"metadata": {},
"source": [
"You will now evaluate your neural network on the iris data set (https://scikit-learn.org/1.5/auto_examples/datasets/plot_iris_dataset.html).\n",
"\n",
"This dataset contains data on 150 flowers of 3 different types which can be separated pretty well using the four features given for each flower, which includes the width and length of their leaves. You are will later train your network to actually make good predictions.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "6bd4c148",
"metadata": {},
"outputs": [],
"source": [
"iris = datasets.load_iris()\n",
"\n",
"_, ax = plt.subplots()\n",
"scatter = ax.scatter(iris.data[:, 0], iris.data[:, 1], c=iris.target)\n",
"ax.set(xlabel=iris.feature_names[0], ylabel=iris.feature_names[1])\n",
"_ = ax.legend(\n",
" scatter.legend_elements()[0], iris.target_names, loc=\"lower right\", title=\"Classes\"\n",
")"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "ed3e2fc9",
"metadata": {},
"outputs": [],
"source": [
"inputs = iris.data\n",
"\n",
"# Since each prediction is a vector with a score for each of the three types of flowers,\n",
"# we need to make each target a vector with a 1 for the correct flower and a 0 for the others.\n",
"targets = np.zeros((len(iris.data), 3))\n",
"for i, t in enumerate(iris.target):\n",
" targets[i, t] = 1\n",
"\n",
"\n",
"def accuracy(predictions, targets):\n",
" one_hot_predictions = np.zeros(predictions.shape)\n",
"\n",
" for i, prediction in enumerate(predictions):\n",
" one_hot_predictions[i, np.argmax(prediction)] = 1\n",
" return accuracy_score(one_hot_predictions, targets)"
]
},
{
"cell_type": "markdown",
"id": "0362c4a9",
"metadata": {},
"source": [
"**a)** What should the input size for the network be with this dataset? What should the output size of the last layer be?\n"
]
},
{
"cell_type": "markdown",
"id": "bf62607e",
"metadata": {},
"source": [
"**b)** Create a network with two hidden layers, the first with sigmoid activation and the last with softmax, the first layer should have 8 \"nodes\", the second has the number of nodes you found in exercise a). Softmax returns a \"probability distribution\", in the sense that the numbers in the output are positive and add up to 1 and, their magnitude are in some sense relative to their magnitude before going through the softmax function. Remember to use the batched version of the create_layers and feed forward functions.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "5366d4ae",
"metadata": {},
"outputs": [],
"source": [
"...\n",
"layers = ..."
]
},
{
"cell_type": "markdown",
"id": "c528846f",
"metadata": {},
"source": [
"**c)** Evaluate your model on the entire iris dataset! For later purposes, we will split the data into train and test sets, and compute gradients on smaller batches of the training data. But for now, evaluate the network on the whole thing at once.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "6c783105",
"metadata": {},
"outputs": [],
"source": [
"predictions = feed_forward_batch(inputs, layers, activation_funcs)"
]
},
{
"cell_type": "markdown",
"id": "01a3caa8",
"metadata": {},
"source": [
"**d)** Compute the accuracy of your model using the accuracy function defined above. Recreate your model a couple times and see how the accuracy changes.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "a2612b82",
"metadata": {},
"outputs": [],
"source": [
"print(accuracy(predictions, targets))"
]
},
{
"cell_type": "markdown",
"id": "334560b6",
"metadata": {},
"source": [
"# Exercise 7 - Training on real data (Optional)\n",
"\n",
"To be able to actually do anything useful with your neural network, you need to train it. For this, we need a cost function and a way to take the gradient of the cost function wrt. the network parameters. The following exercises guide you through taking the gradient using autograd, and updating the network parameters using the gradient. Feel free to implement gradient methods like ADAM if you finish everything.\n"
]
},
{
"cell_type": "markdown",
"id": "700cabe4",
"metadata": {},
"source": [
"Since we are doing a classification task with multiple output classes, we use the cross-entropy loss function, which can evaluate performance on classification tasks. It sees if your prediction is \"most certain\" on the correct target.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "f30e6e2c",
"metadata": {},
"outputs": [],
"source": [
"def cross_entropy(predict, target):\n",
" return np.sum(-target * np.log(predict))\n",
"\n",
"\n",
"def cost(input, layers, activation_funcs, target):\n",
" predict = feed_forward_batch(input, layers, activation_funcs)\n",
" return cross_entropy(predict, target)"
]
},
{
"cell_type": "markdown",
"id": "7ea9c1a4",
"metadata": {},
"source": [
"To improve our network on whatever prediction task we have given it, we need to use a sensible cost function, take the gradient of that cost function with respect to our network parameters, the weights and biases, and then update the weights and biases using these gradients. To clarify, we need to find and use these\n",
"\n",
"$$\n",
"\\frac{\\partial C}{\\partial W}, \\frac{\\partial C}{\\partial b}\n",
"$$\n"
]
},
{
"cell_type": "markdown",
"id": "6c753e3b",
"metadata": {},
"source": [
"Now we need to compute these gradients. This is pretty hard to do for a neural network, we will use most of next week to do this, but we can also use autograd to just do it for us, which is what we always do in practice. With the code cell below, we create a function which takes all of these gradients for us.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "56bef776",
"metadata": {},
"outputs": [],
"source": [
"from autograd import grad\n",
"\n",
"\n",
"gradient_func = grad(\n",
" cost, 1\n",
") # Taking the gradient wrt. the second input to the cost function, i.e. the layers"
]
},
{
"cell_type": "markdown",
"id": "7b1b74bc",
"metadata": {},
"source": [
"**a)** What shape should the gradient of the cost function wrt. weights and biases be?\n",
"\n",
"**b)** Use the `gradient_func` function to take the gradient of the cross entropy wrt. the weights and biases of the network. Check the shapes of what's inside. What does the `grad` func from autograd actually do?\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "841c9e87",
"metadata": {},
"outputs": [],
"source": [
"layers_grad = gradient_func(\n",
" inputs, layers, activation_funcs, targets\n",
") # Don't change this"
]
},
{
"cell_type": "markdown",
"id": "adc9e9be",
"metadata": {},
"source": [
"**c)** Finish the `train_network` function.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "6e4d38d3",
"metadata": {},
"outputs": [],
"source": [
"def train_network(\n",
" inputs, layers, activation_funcs, targets, learning_rate=0.001, epochs=100\n",
"):\n",
" for i in range(epochs):\n",
" layers_grad = gradient_func(inputs, layers, activation_funcs, targets)\n",
" for (W, b), (W_g, b_g) in zip(layers, layers_grad):\n",
" W -= ...\n",
" b -= ..."
]
},
{
"cell_type": "markdown",
"id": "2f65d663",
"metadata": {},
"source": [
"**e)** What do we call the gradient method used above?\n"
]
},
{
"cell_type": "markdown",
"id": "7059dd8c",
"metadata": {},
"source": [
"**d)** Train your network and see how the accuracy changes! Make a plot if you want.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "5027c7a5",
"metadata": {},
"outputs": [],
"source": [
"..."
]
},
{
"cell_type": "markdown",
"id": "3bc77016",
"metadata": {},
"source": [
"**e)** How high of an accuracy is it possible to acheive with a neural network on this dataset, if we use the whole thing as training data?\n"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.9.15"
}
},
"nbformat": 4,
"nbformat_minor": 5
}