mhjensen
1342 lines
57 KiB
Plaintext
1342 lines
57 KiB
Plaintext
{
|
||
"cells": [
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"<!-- dom:TITLE: Data Analysis and Machine Learning Lectures: Optimization and Gradient Methods -->\n",
|
||
"# Data Analysis and Machine Learning Lectures: Optimization and Gradient Methods\n",
|
||
"<!-- dom:AUTHOR: Morten Hjorth-Jensen at Department of Physics, University of Oslo & Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University -->\n",
|
||
"<!-- Author: --> \n",
|
||
"**Morten Hjorth-Jensen**, Department of Physics, University of Oslo and Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University\n",
|
||
"\n",
|
||
"Date: **Sep 20, 2018**\n",
|
||
"\n",
|
||
"Copyright 1999-2018, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license\n",
|
||
"\n",
|
||
"\n",
|
||
"\n",
|
||
"\n",
|
||
"## Optimization, the central part of any Machine Learning algortithm\n",
|
||
"\n",
|
||
"Almost every problem in machine learning and data science starts with\n",
|
||
"a dataset $X$, a model $g(\\beta)$, which is a function of the\n",
|
||
"parameters $\\beta$ and a cost function $C(X, g(\\beta))$ that allows\n",
|
||
"us to judge how well the model $g(\\beta)$ explains the observations\n",
|
||
"$X$. The model is fit by finding the values of $\\beta$ that minimize\n",
|
||
"the cost function. Ideally we would be able to solve for $\\beta$\n",
|
||
"analytically, however this is not possible in general and we must use\n",
|
||
"some approximative/numerical method to compute the minimum.\n",
|
||
"\n",
|
||
"## Steepest descent\n",
|
||
"\n",
|
||
"The method of steepest descent The basic idea of gradient descent is\n",
|
||
"that a function $F(\\mathbf{x})$, \n",
|
||
"$\\mathbf{x} \\equiv (x_1,\\cdots,x_n)$, decreases fastest if one goes from $\\bf {x}$ in the\n",
|
||
"direction of the negative gradient $-\\nabla F(\\mathbf{x})$.\n",
|
||
"\n",
|
||
"It can be shown that if"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"$$\n",
|
||
"\\mathbf{x}_{k+1} = \\mathbf{x}_k - \\gamma_k \\nabla F(\\mathbf{x}_k), \\ \\ \\gamma_k > 0\n",
|
||
"$$"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"for $\\gamma_k$ small enough, then $F(\\mathbf{x}_{k+1}) \\leq\n",
|
||
"F(\\mathbf{x}_k)$. This means that for a sufficiently small $\\gamma_k$\n",
|
||
"we are always moving towards smaller function values, i.e a minimum.\n",
|
||
"\n",
|
||
"<!-- !split -->\n",
|
||
"## More on Steepest descent\n",
|
||
"\n",
|
||
"The previous observation is the basis of the method of steepest\n",
|
||
"descent, which is also referred to as just gradient descent (GD). One\n",
|
||
"starts with an initial guess $\\mathbf{x}_0$ for a minimum of $F$ and\n",
|
||
"computes new approximations according to"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"$$\n",
|
||
"\\mathbf{x}_{k+1} = \\mathbf{x}_k - \\gamma_k \\nabla F(\\mathbf{x}_k), \\ \\ k \\geq 0.\n",
|
||
"$$"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"The parameter $\\gamma_k$ is often referred to as the step length or\n",
|
||
"the learning rate within the context of Machine Learning.\n",
|
||
"\n",
|
||
"<!-- !split -->\n",
|
||
"## The ideal\n",
|
||
"\n",
|
||
"Ideally the sequence $\\{ \\mathbf{x}_k \\}_{k=0}$ converges to a global\n",
|
||
"minimum of the function $F$. In general we do not know if we are in a\n",
|
||
"global or local minimum. In the special case when $F$ is a convex\n",
|
||
"function, all local minima are also global minima, so in this case\n",
|
||
"gradient descent can converge to the global solution. The advantage of\n",
|
||
"this scheme is that it is conceptually simple and straightforward to\n",
|
||
"implement. However the method in this form has some severe\n",
|
||
"limitations:\n",
|
||
"\n",
|
||
"In machine learing we are often faced with non-convex high dimensional\n",
|
||
"cost functions with many local minima. Since GD is deterministic we\n",
|
||
"will get stuck in a local minimum, if the method converges, unless we\n",
|
||
"have a very good intial guess. This also implies that the scheme is\n",
|
||
"sensitive to the chosen initial condition.\n",
|
||
"\n",
|
||
"Note that the gradient is a function of $\\mathbf{x} =\n",
|
||
"(x_1,\\cdots,x_n)$ which makes it expensive to compute numerically.\n",
|
||
"\n",
|
||
"\n",
|
||
"<!-- !split -->\n",
|
||
"## The sensitiveness of the gradient descent\n",
|
||
"\n",
|
||
"GD is sensitive to the choice of learning rate $\\gamma_k$. This is due\n",
|
||
"to the fact that we are only guaranteed that $F(\\mathbf{x}_{k+1}) \\leq\n",
|
||
"F(\\mathbf{x}_k)$ for sufficiently small $\\gamma_k$. The problem is to\n",
|
||
"determine an optimal learning rate. If the learning rate is chosen too\n",
|
||
"small the method will take a long time to converge and if it is too\n",
|
||
"large we can experience erratic behavior.\n",
|
||
"\n",
|
||
"Many of these shortcomings can be alleviated by introducing\n",
|
||
"randomness. One such method is that of Stochastic Gradient Descent\n",
|
||
"(SGD), see below.\n",
|
||
"\n",
|
||
"## Gradient Descent Example\n",
|
||
"\n",
|
||
"We revisit now our simple linear regression example with a linear polynomial."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 1,
|
||
"metadata": {},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"[[3.79420435]\n",
|
||
" [3.13985512]]\n",
|
||
"[[3.79420435]\n",
|
||
" [3.13985512]]\n"
|
||
]
|
||
},
|
||
{
|
||
"data": {
|
||
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAYwAAAEWCAYAAAB1xKBvAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3XmYHGW1x/HvmckeQvZAQpIJSwirbEFWISwJiCjqdQEDFxCNRK8rylXjgkhUXMAVERGIJCIYFxTwmgiThASSMNkgK4SQjYTs+56Zc/+omrEzdPfUTFd3dc/8Ps/Tz3TXeqq65z1V7/tWlbk7IiIiDSlLOgARESkNShgiIhKJEoaIiESihCEiIpEoYYiISCRKGCIiEokShhScmS03s8vD9183swcTimOIma1OYt0thZk9YmZ3JR2HxEMJQw5hZtea2Qwz22Vm68P3nzYzy8f63P177v6JXJdjZgPMzM2sVRxxJU0FrRQjJQypY2a3AT8DfgQcCRwB3ApcALTJME95wQIUkUQpYQgAZtYZuBP4tLuPd/cdHpjj7sPdfV843SNm9msze8bMdgGXmNl7zGyOmW03s1Vmdke9Zd9gZivMbJOZjao37g4zG5vy+Vwze8HMtprZPDMbkjJukpl918ymmdkOM5tgZj3C0VPCv1vNbKeZnZdmG9uH8W8xs4XA2fXG9zGzP5vZBjN7w8w+lzLunWZWFW7jOjO7J2XchSkxrzKzm8Lhbc3sx2a2MpznfjNrH44bYmarzey28ExurZndHI4bAQwHbg+35R8ZvrMTzGyimW02syVm9pFweBszm2tmnw0/l4f77Fsp2/JiGO9aM/ulmbVJWa6HZ5Wvhfv5u2Z2bLiN283sidrpU7bj62a2MaxuHJ4u3nD6q8PYtobLe0emaaUIubteegFcCRwEWjUw3SPANoKzjjKgHTAEODX8/A5gHfD+cPqTgJ3ARUBb4J5wPZeH4+8AxobvjwI2AVeFyxoafu4Zjp8EvA4cD7QPP/8gHDcA8GzxAz8Ange6Af2A+cDqcFwZMAv4FsHZ1DHAMuCKcPyLwA3h+8OAc8P3FcAO4DqgNdAdOD0cdy/w93B9nYB/AN8Pxw0J98Od4XxXAbuBrin7+a4s29IRWAXcDLQCzgA2AieF408BtgAnAqOA6UB5OO4s4NxwvgHAIuALKct24EngcOBkYB/wbLhPOgMLgRvrbcc94fd7MbALGFR/O8IY1wPnAOXAjcByoG3Sv3+9or10hiG1egAb3f1g7YCUo+Y9ZnZRyrRPuvs0d69x973uPsndXwk/vww8RlBwAHwIeMrdp3hwlvJNoCZDDNcDz7j7M+GyJgJVBIVprYfd/VV33wM8AZzeiG38CDDa3Te7+yrg5ynjziZITHe6+353Xwb8Frg2HH8AOM7Merj7TnefHg7/GPBvd3/M3Q+4+yZ3nxu2+YwAvhiubwfwvZTl1S7zznC+ZwgS66CI23I1sNzdH3b3g+4+B/gz8GEAd58P3AX8DfgyQbKrDsfNcvfp4XzLgd/wn++r1g/dfbu7LyBIrBPcfZm7bwP+SVD4p/qmu+9z98nA0+G+rm8E8Bt3n+Hu1e4+hiAZnRtxmyVhShhSaxPQI7XR2N3Pd/cu4bjU38qq1BnN7BwzqwyrcrYRtHvUVhX1SZ3e3XeFy0unAvhwmKS2mtlW4EKgd8o0b6W8301wtB/VIbEAK+qtu0+9dX+doB0H4BaCM5vFZvaSmV0dDu9HcNZTX0+gAzArZXn/Fw6vtSk1QTdyeyqAc+rFO5yg7anWmHC6Z9z9tdqBZna8mT1lZm+Z2XaCRNaDQ61Leb8nzefUOLeE32utFQT7Ol3Mt9WLuV+GaaUIKWFIrRcJjvauiTBt/Vsc/4Gg6qWfu3cG7gdqe1WtJSgUADCzDgTVNumsAh519y4pr47u/oMmxJTOIbEA/eut+4166+7k7lcBuPtr7n4d0Au4GxhvZrXVQsemWddGgoL15JTldXb3qAmhoe1ZBUyuF+9h7j4yZZr7gKeAK8zswpThvwYWAwPd/XCCxJhLL7iu4b6o1R9YkyHm0fVi7uDuj+WwbikgJQwBwN23At8B7jOzD5lZJzMrM7PTCerLs+kEbHb3vWb2ToJqmlrjgavDhuE2BHX2mX53Y4H3mtkVYUNtu7BRtW+ETdhAUNV1TJZpngC+ZmZdw2V+NmXcTGCHmf1v2DhebmanmNnZAGZ2vZn1dPcaYGs4Tw0wDrjczD5iZq3MrLuZnR5O91vgXjPrFS7jKDO7IsK2QHBEn21bngKOt6BDQevwdbaZnRiu6waCtoqbgM8BY8ysNll1ArYDO83sBGDk2xffaN8JG9vfRVBd9qc00/wWuDU8IzUz62hBh4lOMaxfCkAJQ+q4+w+BLwG3ExRY6wjqt/8XeCHLrJ8G7jSzHQSNxk+kLHMB8BmCs5C1BA2xaS+WC9sVriE44t1AcET6FSL8Tt19NzAamBZWd6SrF/8OQXXJG8AE4NGU+asJCrrTw/EbgQcJGnkh6BSwwMx2EnQ9vtbd97j7SoI2ltuAzcBc4LRwnv8FlgLTw6qffxO9jeJ3wEnhtvwtzfbuAIYRtImsIaiquxtoa2b9gZ8C/x22t/yBoC3o3nD2LxMk9R0EhfjjEWPK5C2C73UNQQK91d0Xp4m5Cvgk8Mtw+qUECU1KhLnrAUoi0jQWdHse6+5RzgKlxOkMQ0REIlHCEBGRSFQlJSIikegMQ0REIimpO3v26NHDBwwYkHQYIiIlZdasWRvdvWfDU2ZXUgljwIABVFVVJR2GiEhJMbMVDU/VMFVJiYhIJEoYIiISiRKGiIhEooQhIiKRKGGIiEgkShgiIhKJEoaIiESihCEiIpEoYYiISCRKGCIiEkneE4aZPWRm681sfppxt5mZm1n9B9CLiEiRKcQZxiMEj7c8hJn1I3jE5MoCxCAiIjnKe8Jw9ykEzzqu716CZ0frgRwiIiUgkTYMM7sGeNPd50WYdoSZVZlZ1YYNGwoQnYiIpFPwhGFmHYCvA9+KMr27P+Dug919cM+eOd/OXUREmiiJM4xjgaOBeWa2HOgLzDazIxOIRUREIir4A5Tc/RWgV+3nMGkMdveNhY5FRESiK0S32seAF4FBZrbazG7J9zpFRCR+eT/DcPfrGhg/IN8xiIhI7nSlt4iIRKKEISIikShhiIhIJEoYIiISiRKGiIhEooQhIiKRKGGIiEgkShgiIhKJEoaIiESihCEiIpEoYYiISCRKGCIiEokShoiIRKKEISIikShhiIhIJEoYIiISiRKGiIhEooQhIiKRKGGIiEgkShgiIhJJ3hOGmT1kZuvNbH7KsB+Z2WIze9nM/mpmXfIdh4iI5KYQZxiPAFfWGzYROMXd3wG8CnytAHGIiEgO8p4w3H0KsLnesAnufjD8OB3om+84REQkN8XQhvFx4J+ZRprZCDOrMrOqDRs2FDAsERFJlWjCMLNRwEFgXKZp3P0Bdx/s7oN79uxZuOBEROQQrZJasZndBFwNXObunlQcIiISTSIJw8yuBG4HLnb33UnEICIijVOIbrWPAS8Cg8xstZndAvwS6ARMNLO5ZnZ/vuMQEWlWxo2DAQOgrCz4Oy5jzX5sCtFL6jp37+3urd29r7v/zt2Pc/d+7n56+Lo133GIiBS1xiSAceNgxAhYsQLcg78jRuQ9aRRDLykRkZatsQlg1CjYXa82f/fuYHgeKWGIiCStsQlg5crGDY+JEoaISNIamwD692/c8JgoYYiIJK2xCWD0aOjQ4dBhHToEw/NICUNEJGmNTQDDh8MDD0BFBZhB9+7Qvj3ccENee0wpYYiIJK1+AqioCD4PH559nuXL4dFHYc8e2LQp7z2mrJQush48eLBXVVUlHYaISPEYMCBIEvVVVAQJBTCzWe4+ONdV6QxDRKSUFbDHlBKGiEgpK2CPKSUMEZFSVsAeU0oYItIyJXAvprystykN5k2kRm8RaXlqb8WRenV1hw55K2iTXm9cjd5KGCLS8kToWVTQ9ZaXw5gxeUsa6iUlItJUCd2LKePyq6sLcrfZXClhiEjLk9C9mLIuvwB3m82VEoaIJC9bQ3A+GqcTuhdT2vWmyvcZTo6UMEQkWdmeBZGvBwXls2dRtgRXu97y8vTz5vsMJ0dq9BaRZGVrgIZkGqebKmovqAL3llIvKRFpHsrKgrOH+syCv5nG1dTkN66maEzvq3HjgjaLlSuDM4vRo9VLSkQkq2wN0JnGlZUV/oK7KBrT+6r2brM1NcHffF7/EZO8Jwwze8jM1pvZ/JRh3cxsopm9Fv7tmu84RKRIZWuAztRIXF2d91t5N0lTel/l+Yrz6v3VsS2rEGcYjwBX1hv2VeBZdx8IPBt+FpGWKFsDdP1x6RqLd++GG28sjjOOxva+ylOj/htTVvGb4VP4UN8X6dFuR07LOoS75/0FDADmp3xeAvQO3/cGlkRZzllnneUi0oKZuQdFa+ZXhw7uY8cmF+PYse4VFUGsFRXZY6moyLwdDc2bYuuKrf7Xr073kSdP9mNbLa9bRL/y1f7xgVMcqPIYyvKCNHqb2QDgKXc/Jfy81d27hO8N2FL7Oc28I4ARAP379z9rRboGJRFpGTI1KtdXrL2o6svU4F8rQ8+pg3sPMnPMIiY8tomJs7sxY8dJVNOKw9jBkF6LGPauPQz9eD8GXXk0VmbNp9Hbg4yVcY+5+wPuPtjdB/fs2bOAkYlI0WnowrdacV0Al+872jZ03UXK1d+vP7eCX183hQ/0mU739ru44NZTuXPyRRysKeOrF0xl8s/nsWlHW/6x7p18dvzFnHDVMViZxRpuq1iXFt06M+vt7mvNrDewPqE4RKSU1B5p13ZHLSsLGsDri+MCuPrXStS2L6TGkavRo99+PUY9vmIlx7ZeyRsHK4AKKspX89ET5jH03a257DMn0O3YU+KJJYKkzjD+DtwYvr8ReDKhOESk1KR2Rx0zJn+3+Bg16u0Fedz3exo+PGiwt8xnAqvoy6k91vLLD0/m1QnLeWP/UTyw6CI+fM95dDu2sB1M836GYWaPAUOAHma2Gvg28APgCTO7BVgBfCTfcYhIM1T/jCPOC+DyfEdbr3Fem7icI8f8hcMztGNUt25Hn9/cxZM3nxPLOnOlK71FJD4FvHo57/LwzIzNr2/h2V8tZsIzB5i49BhWVPelmjLKMjXjjh0by/5rNo3eItJM5OtGgXFpbAN2ugb21q1h587Iy9i/cz+TfzaXb1w4iXcetoAex3XmI/eexxNLTuPMXqv59XVTqO7VJ/3MFRXFl2zj6JtbqJeuwxApYpmuKaioSDqy4HqGDh0af71G6jUV3bu7t2mTdRk11TW+6OnX/WcfnORX95rhHdnh4F7OAb+g0zy/Y0ilv/Cbl/3AngO5x9YIlNJ1GHFRlZRIEct2E8GkbxQYR/VShmVU9+nL+A89zoR/VjNh2bGsrg7OGI5rvZxhg1Yw9Op2XDLyBDr375x52XmuytPdakWkuCT1nOwosl0gF7WdIMMyajDKqaGLbeWyPosYNuQAQ0cczdEX9csx6PioDUNEiktST7GLItt1GQ20s3iNs+DJpWxv1yv9eIwaytjc7zTG372MEWMvKqpkESclDBGJRz6fYperbFeIp7m2Yv2CDfzhM9O4eeDz9G39Fqe8/zhu3fMT9tD+bbOXU4Ph2MqVxdXInweqkhKRlmHcOLj++rSj3Izn7q5iwvjtTHzlCObsORGAbraZy/suZugl1QwdcTQVyyc3fJV5rlVweWjPUJWUiEhjDB/+n8e+1rPK+3L57Wdy78zzObzNPkYPncTMRxay4eGnebzsY3zi0YupGH5hMHHtVeaZGvLrX9jXmO68Rd41WQlDRFqEt15ez9TjbmSftT1k+B7aManff/PUt19i89r9TNp6Ol+fMISzW82h7NO3Zi68ozwsqbEJoBC3I8mBqqREpFnas3kPz/9mIRP/soMJ83vz8t5BAHyS3zC6/Nv0qF5P9ZF9aPXju9NX+TTU66v+zQnh7bcjb2zPsTx1TVa3WhGRFDUHa3jlL68xYcxaJk7vxJTNJ7OPdrRhHxd0Wciwc7Yx7IYjOP2jgyhrFaFyJUrh3VB7Q2MTQJ66JseVMJK6vbmIxKU53b+pkdbMfot/37+UCRONiSuOZ70PAgZxctvX+PSZ0xl6TUcuuvUkOvY6o/EL798/feGdWuVU+xjZXJaRKt3tzoulazLo1iAiJS3O20o05tGiCdm1YZf/87sv+RfPrPRT2r5at8k9bb1/rGKqP/KJ5331S2viWVkc+3bkyPS3Sxk5Mvt6Y/4eiOnWIIkngca8lDBE6onr/k0FuJ9RU1QfqPZZYxf6D66s9Eu7zvI27HVwb8sev7xbld/97kqf88fFXn2gOj8B5Fp4F8n9tZQwRCQoyNIVSGaNW06RFGzu7qtmrvGHbp7i11VM9R62oS6UU9st8dvOqvR/fa/Kd23Y1fQVZEsCjU0QDU0f1/eTIyUMEQnuoBpHQZ9gwbZz3U5/+o6Z/vnTJ/mJbZbWrfqIsnV+/dHP++8/9byvmfNWPCtLdyZlFlQRNfYsK8r0DSXiAlUDKmGItHRjx779dtvg3rp1UVedVB+o9pfGLPDvDav0IV1me2v2Obi3Y7cP6/6S//jqSp/3pyVeU10T+7ozbmft7csbsw+i7LNsSaWA1YBKGCItXaYCq3v3xi9r5Mi3n2XEWHiteGG1P3jjFP9Iv2ne3TbWreL09ov8K2dX+sS7Z/meLXtiWVdWmc6ksr0ynWVFPSvLdBZRwCSthCHS0sVRjTR2bPoj69pqmvrTRqw+2f7mdv/7N2b4Z98xyQe1eb1usSP5lW9odYTXYH6wd9/CN6pnKqSzvXI5w8imgNWAShgixapQ3VNzLbDSVYk0pWrF3Q/uO+gzHprv372s0i/qPMdbsd/BvT27/N09Z/o911T6yi/+xGuS7ok1dmzjzjJybcPIRmcYjVw5fBFYAMwHHgPaZZteCUOKXiG7p+arwEp3pJth2u0dj/APHfWCd7XNdYPPbL/Qv3pupT/749m+d9vehtdX6J5Y6arfMsWVay+phuZVG0bkZHEU8AbQPvz8BHBTtnmUMKToFbpQzKXAaqjQTI05w7TVmPctf9NvHjjFH/vsNF+/cEPj11fgLqbu/p/91thqqHzFoV5SkRLGKqAbwS1KngKGZZtHCaPIlMCVwQVXTIViQ7IVmB06+MGHf+8v/vYV/84llb7Weqedbl/PPtF7MxXLGUaqIr1gMW4lnzCCbeDzwE5gAzAuwzQjgCqgqn///rHuRMlBC/lHa7S4rosohDTfYQ347taH+z1d7vDObA1yHdU+qvXdvq+sbW7fd7H+ZlrAgU/JJwygK/Ac0BNoDfwNuD7bPDrDKCLFeLSYtFyui0io0Nr509/6rsOP9BrMV3OUX8dYB/f+5av8E4Mm++NfmOYbX90UX4wtoHAuRs0hYXwY+F3K5/8G7ss2jxJGESmlqpfGyKVAa+p1EQU88t6/a79PvW+ef/viSj/vsJe9jIMO7oex3d97xHT/xYcm+eJnXs/PRXOSmOaQMM4Je0h1AAwYA3w22zxKGBGUSpfOYpRrwd3UJJrHfVlTXeOvTnjDf/XRSX7NkdP98LCaqYyDfk7HV/wbF1b6lF/M9f279ue8LileJZ8wgm3gO8DisFvto0DbbNMrYTSglLp0FqNcCu6xY93Ly5s2f5RE04gDgc3LtvifvvSCjzhhsg9otbJucQNarfQRJ0z28V9+wTcv29LwNknh5emAr1kkjMa+lDAaUEpdOotRU88Qsl0AFyWJRrlBXZbkvH/Xfp/yi7n+jQsr/ZyOr9RVMx3OVr/myOn+q49O8tf+vVzVTMUujwdhBUsYwETgtDhWlutLCaMBzbVdoVCamnAzzVdeHu2fvaGCIsPyq63Mf3T4HX4Y2+uqmc477GX/9sWVPvW+eapmKjV5POArZMI4E6gEHgZ6x7HSpr6UMBrQHNsVCqmpR3hxVCllG5/lArtdtPff9fmG/+X2F33L8q3x7AdJRh4P+ApeJQX8FzAP+Hbt1dmFfilhNKA5tis0VVOry5oyX45VSuns27HPK++d418/v9LXZLhormQPCJpbVWZcmsMZRrAuDDgFuBXYCKwGbogjgMa8lDAi0D9j4RNnE6uUUguCmuoaX/iPpf7TD0zy9/Sa4R3ZEdRqccC/3fZ7b79oLt1RaCl83zqoyayZtGFMA9aEbRnfBa4GjgN+ATwQRxBRX0oYEkkSVXNNqFKqMfPHPjvNbx44xY8qW1M3amDrZf6ZUyf537423beu2Pqf5WfqhVVKha+qTbMr8l5SFiwrMzM7GVjoaSY0s0XufmLWBcRo8ODBXlVVVajVSakqKwuKoXTGjoXhwwsbz4ABsGLF2wYvp4KjWU5X28JlfRYz7NIDDB1xDAMu7Jt+OePGwYgRsHt39vVVVMDy5TmHnReZvhszqKkpfDwthJnNcvfBuS6nrKEJ3H1BumQRek+uAYjErn//zONGjAgK3gLwGmfBk0t5pvv17KXdIeP20o45J13PjIcWsGHv4fxp9Xl88vcXZU4WECS6Bx4IEoJZ5ulWrmx8sOPGBYmtrCz4m699lOm7yfadSfGI4zSlUC9VSUkkjXkwUMzeemW9jx051W889nnvk1LN9KXye3xT6+BJc9VH9YunqiGu6h1d8NnsoQv3RLIYOzZzwojxupQ9W/b4xLtn+VfOrvTT2y+qW0U32+Qf7TfNH7xxiq94YXVs6ztEXIWvLvhs9pQwJF7N8Z84DwVhTXWNz/vTEv/x1ZU+rPtL3o7dDu6t2edDusz27w2r9JfGLPCD+w7GthlZxfG9FfsFn83xt1lgShgSn+ZaTRDTdq2dt85//6nn/fqjn/cjy96qW9RJbV/zz58+yZ++Y6bvWLsjvpgLXTgWc8+l5vrbLDAlDIlPMRcY9TW2QG1CAbxrwy7/v7te8tvOqvRT2y2p2x09bINfVzHVH7p5iq+auSaGjUkTaxKFYzEXyqX02yxiShjSeJkKz2KvkqiVp4Kt+kC1z/njYr/73ZV+ebcqb8seB/c27PVLu87yH1xZ6bP/sMirD1THtCEZJFk4Fmu1T6n8NotcXAmjweswiomuw8hBuj78HToE3TRHjUp7nUDR9efPcD1DU+JcM/stJv56KRMmGv9eeTzrvScAp7R9jWGnvMnQ9x/GRbeeRIceHXKPOypdo/B2MX7nLVlc12EkftbQmJfOMHKQ7ei12Kok8nAmtHPdTn/mOzP9C2dM8pPbvlo3ay9b78MHTPUxn3ze35y1tmlxxUXVL29XbL/NEoWqpBKSj0IjyjJzXW9DhW2xVEmkKyDM3EeObFSBWn2g2meNXejfv6LSL+ky29uw18G9LXt8aLcq/+FVlT738cXRq5kaU3DlcuNDFY5vVyy/zRKmhJGEfPxDR1lmHOvN59FrnP/QmeKsTRpZ9sPK6W/6726a4tf2n+Y9bEPdJO9ot9i/PLjS//W9Kt+9aXe8cdXff7l+VyocJQ+UMJKQj0I3yjLjWG++jl7jXm6WZz/UFaBhgVrdt5/P/uCd/rnTJvkJbZbWTXZk2Vt+wzHP+6O3TvW189bltn0NxVW/Oqw5VSspeTUbShhJyEePjSjLzOXRoan/8LXVOnEWAHEXkJmWR3B315mPLPDRQyv94s5zvDX7HNzbsduv6P6S/+R9lf7y+CX5eRRp1O1sLr16VD3WrChhNJcLnPJ1hlGof/i4C8ixYzMucyV96z6e0X6h3/7OSv/3D2f5ni174t2mTHFF2Z/N5QyjuWyHuHszSRhAF2A8sBhYBJyXbfq6hNGcLnDKVxtG9+6F+YePuWDZtmqbv37mB72m3vJ20d7v7/VNH/fpqb5u/vpYNyGyqJ0TmsOReXM5UxJ3bz4JYwzwifB9G6BLtunrEkZzu8Ap7l5SBbrxXt26ciggD+476NMffMXvvLTS33X4XG/Ffgf3m/idv1V2pNdgvq9nH695tIQK3OZQ968zjGal5BMG0Bl4A4KLB6O86hJGMR79FFMhkaUdIC//8I3c9mWTV/pvhk/2/zrqBe9iW4Kvjmo/q8MC/+q5lf7cT2b73m17449TomsuZ0ri7s0jYZwOzAQeAeYADwId00w3AqgCqvr37x9sfbEd/RTbP1e2nka1MRUwwW1dsdX/+tXp/ulTJvlxrd+oC6Vv+Zv+8YFT/I+fm+YbFm/M2/qliYrpIEhy0hwSxmDgIHBO+PlnwHezzZN4G0YmxZbAMsXTvXswPs/778CeAz7t/pf9jiGVfn6neV7OAQf3juzwq3vN8J99cJIvevr1/PRmiktLKSxbyna2cM0hYRwJLE/5/C7g6WzzJN5LKpNiqyJrKCHkIcEtfXa533ftZP9A7xe9M1vrqpnO7jjfR11Q6ZN/Ptf37dgXy+blXbEdkORLS9lOKf2EEWwDzwODwvd3AD/KNn3i12FkUmxnGO7ZE2oMCW7L8q3+56+86J86cbIf02p53SL6l6/yT54w2Z/44gu+8dVNsW9WQTT0fRbTwUouivF3K3kRV8JI9G61ZnY6QdtFG2AZcLO7b8k0fdHerTbbnWCHD08urkyacAfQA7sPMOORRUx8fDMTZndn5s6TqKGcTmznkiMXMexdexl6S38GDh2AlVlew8+7bHeNffTR0vqus9HdcVsM3a222JTSUWeEqoia6hp/dcIb/ssPT/L3HTndO7HNwb2Mg37uYS/7N99V6c//ap7v37U/ue3Il2xH3s3pqLw5bYtkRXOokmrsK6eEEVeBXkqJIZs027Fp6Wb/05de8E+eMNkrylfVlR9Ht1rhnzpxso//8gu+edmWoog17+vLlFCLrb0qF2rDaDGUMBojrn+MZvYPtm/HPp/887k+6oJKf2fHV9yodnA/nK3+/t4v+n3XTvalzy5v+griKOiTvKo/9Wr57t3/sz3N6ai8uRwASVZKGI0R1z95iRcWNdU1vujp1/3n/zXJr+41wzuyw8G9nAN+fqd5fseQSp92/8t+YM+B3FcWV0Gf1D7PFH8Dt1gXKUZxJYyW8YjWuBr3SrCRcOOSTTx73xIm/LOaicuOYVX1UQB5pX+pAAAPjUlEQVQc13o5Q49fybD3tuWSkSfQuX/neFcc16M1k9rn2eIfPTp4rO3KldC/f/C51Bq8pUWJq9G7ZSSMuAqvEni+8L7t+3jhwYVMHL+NCfN6MXv3CThldLGtXNZnEUMvPsDQTw7gmCH98xtIXAV9Uvu8BA8ORDKJK2GUxRFM0Rs9Ouj6mKpDh2B4EsvJ1bhxQUFaVoZXVLD6tnv46Qcm855eL9Gt80Euve0MfvTiBbRvdYDvXDqF6Q/OZ8Puwxi/+jw+Ne6i/CcLCI68GzM8k6T2eVzxizQncdRrFeqlXlLB+mvatT+kDn0nHfw6xvrxrZf5Z06d5E9+fbpvW7WtsHGliTO2uv4k9nkz6+AgLRtqw2g59m7dy7TfLmTin7fzuZkfo4+vfds01Yd3oXxbxmsekzFuXGnX9Zd6/CIhVUk1Y17jzP/ra9xzzSSu7FFFt641XH77mfxkxgUcmSZZAJRv3xoUcNmkVGUxYEDD0+dq+PCgnaGmJvjb1MK20HHXiit+kWZCCSNpYWHoZWXs7HQkv+71LY5qvY5TPziQ2/4+hBU7uvLJ017iH9+cyeY391JWXp55WaNGZV/PiBFBA7J78HfEiMIUvrkU+EnGLSKHUJVUQvZs3sPSkT9h0Pi7aFOzr274btrzu2630/F9lzH01mPpd06fQ2e0LPdpytaDJ6neRrneZ6sEeqaJFDt1qy0xXuO8PP5VJv5+LRNe7MTzm09iEScygEYWhpkK0IbmK8brGaIU+OreKpIztWGUgLVz1/H7EVO54Zip9G69gdM/OoivPD2EN3d15tYzZlLByvQzrswwHIKG1zZt3j68devsXU2T6iaaaVuybWMqdW8VKRpKGDHavXE3/xpdxW2DJ3Fqu9foc8YR3PjbC/nX8kFc1u81Hr5lKqtfWsuCvcdx7+yLsYomFIbDh8NDD0H37v8Z1r07PPxw9iqeUr2eoViufRGRFnQdRh5UH6j22X9Y5D+4stIv6zrL27DXwb0te/yyrrP87ndX+pw/LvbqA9XpF1Dovv7FfD1DttiSvvZFpMShmw8mY/VLa/zhW5736yqmek9bX1cGntpuid92VqX/310v+a4Nu6IvsCUUhg1toy6SE8mruBJGq6TPcIrdrvW7mPzrhUx8cjcTFh7Fwn3HAb05omw9VwxYwrBhS7j81oH0Pv144Pikwy1Ow4dnry4bNerQXlQQfB41Stc+iBQRJYx6ag7WMOePS5jw6DomzOzMtK0nc4CzacceLuq+gJsvn8Swm/pw6gcHYmW9cltZ/S6ntdcYQMsqKHNtGBeRglC3WmDVjDVM+PXrTHyunH+vHsQmDxqUT2u3hGGnrmXoBztx4YiTaN+tfbwr1jUGAe0HkbxSt9oc7Fizg6e+NZPPnTaZE9u+Tv9z+/CJMe9iypvH8J6jFzF25DTeemUDc/cM4oczhzD0q2c1nCyacjWzjqwDhewJldRtRkSagzgaQnJ5AeXAHOCphqZtaqP3wX0HfcZD8/2uyyv9os5zvBX7Hdzbs8uv7DHT77mm0l/5y6teU13TpOU3qtE2tQG4vPzQeUrsCX6xitr4n0snATWuSwtFc7lbrZl9CRgMHO7uV2ebtjFVUsunrmbiA8uY8Fxrnl1zAlu8KwBntl/E0HesY9iHO3P+LSfSrku7XDchepVKuttk1NeY22a0NLrNiEiTxFUllfTZRV/gWeBScjzD2LZqmz/59en+mVMn+cDWy+oOII8qW+M3D5zif/ifab5+4YbGJuZozNKfKZgdOl2m51OXl5dOt9okuwHn+nzvqN+TSDNDM+lW+1PgdqBTpgnMbAQwAqB/ytXBB/cepGrsYib8YSMTZ3Xlxe0nU805dGAXQ3ou5DMXrmToTUdx4tXHYmW987sV/funP3KtfzVzpraJmprivi9S7XMhVqwI7uFUe1Za6F5dcdxmJMr3JCLpxZF1mvICrgbuC98PIcIZxinHn+r3f2yyf7DPi96ZrcHBIdU+uMMC/9p5lV557xzfu21vfGk5qqh14005Qk76wr5025ZUm0uuZxhqw5AWilK/0hv4PrAaWA68BewGxmaf5ywH937lq/2W46f441+Y5hsWb4xzvzZdlIK9sQVWUwu4OJNMpkI6iSqdOAr8pBOwSAJKPmEcEkTEM4x+XY73xc+83vTeTMWgMQVWU89I4jyKzlTvn1SvLhX4Io0WV8JIvJcUgJkNAb7sMfaSahaa8iyIuHsCZXv+BqhXl0gJaFYX7rn7pIaSRYvUlFuDx30xYLqL6mqf+ldRoWQh0oIURcKQDJpyBXTcDxwaPjxIChUVQaKoqIBHHw3OfJYvV7IQaUGUMLJJ+jYS6Qrrho7o83GbjeHDg+RQU6MkIdKCJX0dRvEqljvJNnRr8HTTQ3DdxMqVwZnF6NEq5EUkZ0XR6B1VQRu9dRsJEWkmmlWjd1HSnWRFRA6hhJFJ3I3HIiIlTgkjkyiNx0k3iouIFJASRiYN9VCqbRRfsSLoYlrbKK6kISLNlBq9m0qN4iJSItTonTQ1iotIC6OE0VRqFBeRFqY0E0YxNDbn44pqEZEiVnoJo1gam5ty2w4RkRJWeo3eGzeqsVlEpBFabqO3GptFRBJReglDjc0iIokovYShxmYRkUSUXsJQY7OISCJK83kYjX1GhIiI5Kz0zjBERCQRiSUMM+tnZpVmttDMFpjZ55OKRUREGpZkldRB4DZ3n21mnYBZZjbR3RcmGJOIiGSQ2BmGu69199nh+x3AIuCopOIREZHsiqINw8wGAGcAM9KMG2FmVWZWtWHDhkKHJiIiocQThpkdBvwZ+IK7b68/3t0fcPfB7j64Z8+ehQ9QRESAhBOGmbUmSBbj3P0vScYiIiLZJdlLyoDfAYvc/Z6k4hARkWiSPMO4ALgBuNTM5oavqxKMR0REskisW627TwUsqfWLiEjjJN7oLSIipUEJQ0REIlHCEBGRSJQwREQkEiUMERGJRAlDREQiUcIQEZFIlDBERCQSJQwREYlECUNERCJRwhARkUiUMEREJBIlDBERiUQJQ0REIlHCEBGRSJQwREQkEiUMERGJRAlDREQiUcIQEZFIlDBERCSSRBOGmV1pZkvMbKmZfTXJWEREJLvEEoaZlQO/At4NnARcZ2YnJRWPiIhkl+QZxjuBpe6+zN33A38ErkkwHhERyaJVgus+CliV8nk1cE79icxsBDAi/LjPzOYXILZc9QA2Jh1EBIozPqUQIyjOuJVKnIPiWEiSCSMSd38AeADAzKrcfXDCITVIccarFOIshRhBccatlOKMYzlJVkm9CfRL+dw3HCYiIkUoyYTxEjDQzI42szbAtcDfE4xHRESySKxKyt0Pmtn/AP8CyoGH3H1BA7M9kP/IYqE441UKcZZCjKA449ai4jR3j2M5IiLSzOlKbxERiUQJQ0REIimahNHQbULMrK2ZPR6On2FmA1LGfS0cvsTMrkgwxi+Z2UIze9nMnjWzipRx1WY2N3zltXE/Qpw3mdmGlHg+kTLuRjN7LXzdmHCc96bE+KqZbU0ZV5D9aWYPmdn6TNf/WODn4Ta8bGZnpowr5L5sKM7hYXyvmNkLZnZayrjl4fC5cXW/zCHOIWa2LeW7/VbKuILdSihCnF9JiXF++HvsFo4ryP40s35mVhmWOQvM7PNppon39+nuib8IGr1fB44B2gDzgJPqTfNp4P7w/bXA4+H7k8Lp2wJHh8spTyjGS4AO4fuRtTGGn3cW0b68Cfhlmnm7AcvCv13D912TirPe9J8l6BhR6P15EXAmMD/D+KuAfwIGnAvMKPS+jBjn+bXrJ7gdz4yUccuBHkWyP4cAT+X6e8l3nPWmfS/wXKH3J9AbODN83wl4Nc3/eqy/z2I5w4hym5BrgDHh+/HAZWZm4fA/uvs+d38DWBour+Axunulu+8OP04nuLak0HK55coVwER33+zuW4CJwJVFEud1wGN5iiUjd58CbM4yyTXA7z0wHehiZr0p7L5sME53fyGMA5L7bUbZn5kU9FZCjYwzqd/mWnefHb7fASwiuINGqlh/n8WSMNLdJqT+htdN4+4HgW1A94jzFirGVLcQZPZa7cysysymm9n78xBfrahx/ld4ijrezGovoCzUvmzUusKqvaOB51IGF2p/NiTTdhRyXzZW/d+mAxPMbJYFt+JJ2nlmNs/M/mlmJ4fDinJ/mlkHgoL2zymDC74/LaiiPwOYUW9UrL/Por81SCkys+uBwcDFKYMr3P1NMzsGeM7MXnH315OJkH8Aj7n7PjP7FMGZ26UJxRLFtcB4d69OGVZM+7NkmNklBAnjwpTBF4b7shcw0cwWh0fYSZhN8N3uNLOrgL8BAxOKJYr3AtPcPfVspKD708wOI0hYX3D37flaDxTPGUaU24TUTWNmrYDOwKaI8xYqRszscmAU8D5331c73N3fDP8uAyYRHA3kQ4NxuvumlNgeBM6KOm8h40xxLfVO+Qu4PxuSaTuK7tY3ZvYOgu/7GnffVDs8ZV+uB/5Kfqp0I3H37e6+M3z/DNDazHpQhPszlO23mff9aWatCZLFOHf/S5pJ4v195rthJmLjTSuCRpej+U+D1sn1pvkMhzZ6PxG+P5lDG72XkZ9G7ygxnkHQMDew3vCuQNvwfQ/gNfLUYBcxzt4p7z8ATPf/NIS9EcbbNXzfLak4w+lOIGhEtCT2Z7iOAWRupH0PhzYqziz0vowYZ3+C9r3z6w3vCHRKef8CcGWCcR5Z+10TFLQrw30b6fdSqDjD8Z0J2jk6JrE/w/3ye+CnWaaJ9feZt53dhI2/iqCV/3VgVDjsToIjdYB2wJ/CH/1M4JiUeUeF8y0B3p1gjP8G1gFzw9ffw+HnA6+EP/JXgFsS3pffBxaE8VQCJ6TM+/FwHy8Fbk4yzvDzHcAP6s1XsP1JcPS4FjhAUM97C3ArcGs43ggeBPZ6GMvghPZlQ3E+CGxJ+W1WhcOPCffjvPA3MSrhOP8n5bc5nZQEl+73klSc4TQ3EXS4SZ2vYPuToFrRgZdTvter8vn71K1BREQkkmJpwxARkSKnhCEiIpEoYYiISCRKGCIiEokShoiIRKKEISIikShhiIhIJEoYIjkIn0cwNHx/l5n9IumYRPJFNx8Uyc23gTvDG82dAbwv4XhE8kZXeovkyMwmA4cBQzx4LoFIs6QqKZEcmNmpBE8+269kIc2dEoZIE4VPLhtH8FSznWaWtyfqiRQDJQyRJgiftPYX4DZ3XwR8l6A9Q6TZUhuGiIhEojMMERGJRAlDREQiUcIQEZFIlDBERCQSJQwREYlECUNERCJRwhARkUj+Hz+GMgV/Yq0JAAAAAElFTkSuQmCC\n",
|
||
"text/plain": [
|
||
"<Figure size 432x288 with 1 Axes>"
|
||
]
|
||
},
|
||
"metadata": {},
|
||
"output_type": "display_data"
|
||
}
|
||
],
|
||
"source": [
|
||
"%matplotlib inline\n",
|
||
"\n",
|
||
"\n",
|
||
"# Importing various packages\n",
|
||
"from random import random, seed\n",
|
||
"import numpy as np\n",
|
||
"import matplotlib.pyplot as plt\n",
|
||
"from mpl_toolkits.mplot3d import Axes3D\n",
|
||
"from matplotlib import cm\n",
|
||
"from matplotlib.ticker import LinearLocator, FormatStrFormatter\n",
|
||
"import sys\n",
|
||
"\n",
|
||
"x = 2*np.random.rand(100,1)\n",
|
||
"y = 4+3*x+np.random.randn(100,1)\n",
|
||
"\n",
|
||
"xb = np.c_[np.ones((100,1)), x]\n",
|
||
"beta_linreg = np.linalg.inv(xb.T.dot(xb)).dot(xb.T).dot(y)\n",
|
||
"print(beta_linreg)\n",
|
||
"beta = np.random.randn(2,1)\n",
|
||
"\n",
|
||
"eta = 0.1\n",
|
||
"Niterations = 1000\n",
|
||
"m = 100\n",
|
||
"\n",
|
||
"for iter in range(Niterations):\n",
|
||
" gradients = 2.0/m*xb.T.dot(xb.dot(beta)-y)\n",
|
||
" beta -= eta*gradients\n",
|
||
"\n",
|
||
"print(beta)\n",
|
||
"xnew = np.array([[0],[2]])\n",
|
||
"xbnew = np.c_[np.ones((2,1)), xnew]\n",
|
||
"ypredict = xbnew.dot(beta)\n",
|
||
"ypredict2 = xbnew.dot(beta_linreg)\n",
|
||
"plt.plot(xnew, ypredict, \"r-\")\n",
|
||
"plt.plot(xnew, ypredict2, \"b-\")\n",
|
||
"plt.plot(x, y ,'ro')\n",
|
||
"plt.axis([0,2.0,0, 15.0])\n",
|
||
"plt.xlabel(r'$x$')\n",
|
||
"plt.ylabel(r'$y$')\n",
|
||
"plt.title(r'Gradient descent example')\n",
|
||
"plt.show()"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"## And a corresponding example using **scikit-learn**"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 2,
|
||
"metadata": {},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"[[4.20374363]\n",
|
||
" [2.83470272]]\n",
|
||
"[4.21610176] [2.82633808]\n"
|
||
]
|
||
},
|
||
{
|
||
"name": "stderr",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"/usr/local/lib/python3.7/site-packages/sklearn/linear_model/stochastic_gradient.py:117: DeprecationWarning: n_iter parameter is deprecated in 0.19 and will be removed in 0.21. Use max_iter and tol instead.\n",
|
||
" DeprecationWarning)\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"# Importing various packages\n",
|
||
"from random import random, seed\n",
|
||
"import numpy as np\n",
|
||
"import matplotlib.pyplot as plt\n",
|
||
"from sklearn.linear_model import SGDRegressor\n",
|
||
"\n",
|
||
"x = 2*np.random.rand(100,1)\n",
|
||
"y = 4+3*x+np.random.randn(100,1)\n",
|
||
"\n",
|
||
"xb = np.c_[np.ones((100,1)), x]\n",
|
||
"beta_linreg = np.linalg.inv(xb.T.dot(xb)).dot(xb.T).dot(y)\n",
|
||
"print(beta_linreg)\n",
|
||
"sgdreg = SGDRegressor(n_iter = 50, penalty=None, eta0=0.1)\n",
|
||
"sgdreg.fit(x,y.ravel())\n",
|
||
"print(sgdreg.intercept_, sgdreg.coef_)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"<!-- !split -->\n",
|
||
"## Convex functions\n",
|
||
"\n",
|
||
"Ideally we want our cost/loss function to be convex(concave).\n",
|
||
"\n",
|
||
"First we give the definition of a convex set: A set $C$ in\n",
|
||
"$\\mathbb{R}^n$ is said to be convex if, for all $x$ and $y$ in $C$ and\n",
|
||
"all $t \\in (0,1)$ , the point $(1 − t)x + ty$ also belongs to\n",
|
||
"C. Geometrically this means that every point on the line segment\n",
|
||
"connecting $x$ and $y$ is in $C$ as discussed below.\n",
|
||
"\n",
|
||
"The convex subsets of $\\mathbb{R}$ are the intervals of\n",
|
||
"$\\mathbb{R}$. Examples of convex sets of $\\mathbb{R}^2$ are the\n",
|
||
"regular polygons (triangles, rectangles, pentagons, etc...).\n",
|
||
"\n",
|
||
"## Convex function\n",
|
||
"\n",
|
||
"**Convex function**: Let $X \\subset \\mathbb{R}^n$ be a convex set. Assume that the function $f: X \\rightarrow \\mathbb{R}$ is continuous, then $f$ is said to be convex if $$f(tx_1 + (1-t)x_2) \\leq tf(x_1) + (1-t)f(x_2) $$ for all $x_1, x_2 \\in X$ and for all $t \\in [0,1]$. If $\\leq$ is replaced with a strict inequaltiy in the definition, we demand $x_1 \\neq x_2$ and $t\\in(0,1)$ then $f$ is said to be strictly convex. For a single variable function, convexity means that if you draw a straight line connecting $f(x_1)$ and $f(x_2)$, the value of the function on the interval $[x_1,x_2]$ is always below the line as illustrated below.\n",
|
||
"\n",
|
||
"## Conditions on convex functions\n",
|
||
"\n",
|
||
"In the following we state first and second-order conditions which\n",
|
||
"ensures convexity of a function $f$. We write $D_f$ to denote the\n",
|
||
"domain of $f$, i.e the subset of $R^n$ where $f$ is defined. For more\n",
|
||
"details and proofs we refer to: [S. Boyd and L. Vandenberghe. Convex Optimization. Cambridge University Press](http://stanford.edu/boyd/cvxbook/, 2004).\n",
|
||
"\n",
|
||
"**First order condition.**\n",
|
||
"\n",
|
||
"Suppose $f$ is differentiable (i.e $\\nabla f(x)$ is well defined for\n",
|
||
"all $x$ in the domain of $f$). Then $f$ is convex if and only if $D_f$\n",
|
||
"is a convex set and $$f(y) \\geq f(x) + \\nabla f(x)^T (y-x) $$ holds\n",
|
||
"for all $x,y \\in D_f$. This condition means that for a convex function\n",
|
||
"the first order Taylor expansion (right hand side above) at any point\n",
|
||
"a global under estimator of the function. To convince yourself you can\n",
|
||
"make a drawing of f(x) = x^2+1 and draw the tangent line to $f(x)$ and\n",
|
||
"note that it is always below the graph.\n",
|
||
"\n",
|
||
"\n",
|
||
"\n",
|
||
"**Second order condition.**\n",
|
||
"\n",
|
||
"Assume that $f$ is twice\n",
|
||
"differentiable, i.e the Hessian matrix exists at each point in\n",
|
||
"$D_f$. Then $f$ is convex if and only if $D_f$ is a convex set and its\n",
|
||
"Hessian is positive semi-definite for all $x\\in D_f$. For a\n",
|
||
"single-variable function this reduces to $f''(x) \\geq\n",
|
||
"0$. Geometrically this means that $f$ has nonnegative curvature\n",
|
||
"everywhere.\n",
|
||
"\n",
|
||
"\n",
|
||
"\n",
|
||
"This condition is particularly useful since it gives us an procedure for determining if the function under consideration is convex, apart from using the definition.\n",
|
||
"\n",
|
||
"## More on convex functions\n",
|
||
"\n",
|
||
"The next result is of great importance to us and the reason why we are\n",
|
||
"going on about convex functions. In machine learning we frequently\n",
|
||
"have to minimize a loss/cost function in order to find the best\n",
|
||
"parameters for the model we are considering. \n",
|
||
"\n",
|
||
"Ideally we want the\n",
|
||
"global minimum (for high-dimensional models it is hard to know\n",
|
||
"if we have local or global minimum). However, if the cost/loss function\n",
|
||
"is convex the following result provides invaluable information:\n",
|
||
"\n",
|
||
"**Any minimum is global for convex functions.**\n",
|
||
"\n",
|
||
"Consider the problem of finding $x \\in \\mathbb{R}^n$ such that $f(x)$\n",
|
||
"is minimal, where $f$ is convex and differentiable. Then, any point\n",
|
||
"$x^*$ that satisfies $\\nabla f(x^*) = 0$ is a global minimum.\n",
|
||
"\n",
|
||
"\n",
|
||
"\n",
|
||
"This result means that if we know that the cost/loss function is convex and we are able to find a minimum, we are guaranteed that it is a global minimum.\n",
|
||
"\n",
|
||
"## Some simple problems\n",
|
||
"\n",
|
||
"1. Show that $f(x)=x^2$ is convex for $x \\in \\mathbb{R}$ using the definition of convexity. Hint: If you re-write the definition, $f$ is convex if the following holds for all $x,y \\in D_f$ and any $\\lambda \\in [0,1] $ $\\lambda f(x) + (1-\\lambda)f(y) - f(\\lambda x + (1-\\lambda) y ) \\geq 0. $\n",
|
||
"\n",
|
||
"2. Using the second order condition show that the following functions are convex on the specified domain.\n",
|
||
"\n",
|
||
" * $f(x) = e^x$ is convex for $x \\in \\mathbb{R}$.\n",
|
||
"\n",
|
||
" * $g(x) = -\\ln(x)$ is convex for $x \\in (0,\\infty)$.\n",
|
||
"\n",
|
||
"\n",
|
||
"3. Let $f(x) = x^2$ and $g(x) = e^x$. Show that $f(g(x))$ and $g(f(x))$ is convex for $x \\in \\mathbb{R}$. Also show that if $f(x)$ is any convex function than $h(x) = e^{f(x)}$ is convex.\n",
|
||
"\n",
|
||
"4. A norm is any function that satisfy the following properties\n",
|
||
"\n",
|
||
" * $f(\\alpha x) = |\\alpha| f(x)$ for all $\\alpha \\in \\mathbb{R}$.\n",
|
||
"\n",
|
||
" * $f(x+y) \\leq f(x) + f(y)$\n",
|
||
"\n",
|
||
" * $f(x) \\leq 0$ for all $x \\in \\mathbb{R}^n$ with equality if and only if $x = 0$\n",
|
||
"\n",
|
||
"\n",
|
||
"Using the definition of convexity, try to show that a function satisfying the properties above is convex (the third condition is not needed to show this).\n",
|
||
"\n",
|
||
"<!-- !split -->\n",
|
||
"## Revisiting our first homework\n",
|
||
"\n",
|
||
"We will use linear regression as a case study for the gradient descent\n",
|
||
"methods. Linear regression is a great test case for the gradient\n",
|
||
"descent methods discussed in the lectures since it has several\n",
|
||
"desirable properties such as:\n",
|
||
"\n",
|
||
"1. An analytical solution (recall homework set 1).\n",
|
||
"\n",
|
||
"2. The gradient can be computed analytically.\n",
|
||
"\n",
|
||
"3. The cost function is convex which guarantees that gradient descent converges for small enough learning rates\n",
|
||
"\n",
|
||
"We revisit the example from homework set 1 where we had"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"$$\n",
|
||
"y_i = 5x_i^2 + 0.1\\xi_i, \\ i=1,\\cdots,100\n",
|
||
"$$"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"with $x_i \\in [0,1] $ chosen randomly with a uniform distribution. Additionally $\\xi_i$ represents stochastic noise chosen according to a normal distribution $\\cal {N}(0,1)$. \n",
|
||
"The linear regression model is given by"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"$$\n",
|
||
"h_\\beta(x) = \\hat{y} = \\beta_0 + \\beta_1 x,\n",
|
||
"$$"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"such that"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"$$\n",
|
||
"\\hat{y}_i = \\beta_0 + \\beta_1 x_i.\n",
|
||
"$$"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"<!-- !split -->\n",
|
||
"## Gradient descent example\n",
|
||
"\n",
|
||
"Let $\\mathbf{y} = (y_1,\\cdots,y_n)^T$, $\\mathbf{\\hat{y}} = (\\hat{y}_1,\\cdots,\\hat{y}_n)^T$ and $\\beta = (\\beta_0, \\beta_1)^T$\n",
|
||
"\n",
|
||
"t is convenient to write $\\mathbf{\\hat{y}} = X\\beta$ where $X \\in \\mathbb{R}^{100 \\times 2} $ is the design matrix given by"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"<!-- Equation labels as ordinary links -->\n",
|
||
"<div id=\"_auto1\"></div>\n",
|
||
"\n",
|
||
"$$\n",
|
||
"\\begin{equation}\n",
|
||
"X \\equiv \\begin{bmatrix}\n",
|
||
"1 & x_1 \\\\\n",
|
||
"\\vdots & \\vdots \\\\\n",
|
||
"1 & x_{100} & \\\\\n",
|
||
"\\end{bmatrix}.\n",
|
||
"\\label{_auto1} \\tag{1}\n",
|
||
"\\end{equation}\n",
|
||
"$$"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"The loss function is given by"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"$$\n",
|
||
"C(\\beta) = ||X\\beta-\\mathbf{y}||^2 = ||X\\beta||^2 - 2 \\mathbf{y}^T X\\beta + ||\\mathbf{y}||^2 = \\sum_{i=1}^{100} (\\beta_0 + \\beta_1 x_i)^2 - 2 y_i (\\beta_0 + \\beta_1 x_i) + y_i^2\n",
|
||
"$$"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"and we want to find $\\beta$ such that $C(\\beta)$ is minimized.\n",
|
||
"\n",
|
||
"## The derivative of the cost/loss function\n",
|
||
"\n",
|
||
"Computing $\\partial C(\\beta) / \\partial \\beta_0$ and $\\partial C(\\beta) / \\partial \\beta_1$ we can show that the gradient can be written as"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"$$\n",
|
||
"\\nabla_\\beta C(\\beta) = (\\partial C(\\beta) / \\partial \\beta_0, \\partial C(\\beta) / \\partial \\beta_1)^T = 2\\begin{bmatrix} \\sum_{i=1}^{100} \\left(\\beta_0+\\beta_1x_i-y_i\\right) \\\\\n",
|
||
"\\sum_{i=1}^{100}\\left( x_i (\\beta_0+\\beta_1x_i)-y_ix_i\\right) \\\\\n",
|
||
"\\end{bmatrix} = 2X^T(X\\beta - \\mathbf{y}),\n",
|
||
"$$"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"where $X$ is the design matrix defined above.\n",
|
||
"\n",
|
||
"## The Hessian matrix\n",
|
||
"The Hessian matrix of $C(\\beta)$ is given by"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"$$\n",
|
||
"\\hat{H} \\equiv \\begin{bmatrix}\n",
|
||
"\\frac{\\partial^2 C(\\beta)}{\\partial \\beta_0^2} & \\frac{\\partial^2 C(\\beta)}{\\partial \\beta_0 \\partial \\beta_1} \\\\\n",
|
||
"\\frac{\\partial^2 C(\\beta)}{\\partial \\beta_0 \\partial \\beta_1} & \\frac{\\partial^2 C(\\beta)}{\\partial \\beta_1^2} & \\\\\n",
|
||
"\\end{bmatrix} = 2X^T X.\n",
|
||
"$$"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"This result implies that $C(\\beta)$ is a convex function since the matrix $X^T X$ always is positive semi-definite.\n",
|
||
"\n",
|
||
"## Simple program\n",
|
||
"\n",
|
||
"We can now write a program that minimizes $C(\\beta)$ using the gradient descent method with a constant learning rate $\\gamma$ according to"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"$$\n",
|
||
"\\beta_{k+1} = \\beta_k - \\gamma \\nabla_\\beta C(\\beta_k), \\ k=0,1,\\cdots\n",
|
||
"$$"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"We can use the expression we computed for the gradient and let use a\n",
|
||
"$\\beta_0$ be chosen randomly and let $\\gamma = 0.001$. Stop iterating\n",
|
||
"when $||\\nabla_\\beta C(\\beta_k) || < \\epsilon = 10^{-8}$. \n",
|
||
"\n",
|
||
"And finally we can compare our solution for $\\beta$ with the analytic result given by \n",
|
||
"$\\beta= (X^TX)^{-1} X^T \\mathbf{y}$."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 3,
|
||
"metadata": {},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"[-0.72135172 4.78496149]\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"import numpy as np\n",
|
||
"\n",
|
||
"\"\"\"\n",
|
||
"The following setup is just a suggestion, feel free to write it the way you like.\n",
|
||
"\"\"\"\n",
|
||
"\n",
|
||
"#Setup problem described in the exercise\n",
|
||
"N = 100 #Nr of datapoints\n",
|
||
"M = 2 #Nr of features\n",
|
||
"x = np.random.rand(N) #Uniformly generated x-values in [0,1]\n",
|
||
"y = 5*x**2 + 0.1*np.random.randn(N)\n",
|
||
"X = np.c_[np.ones(N),x] #Construct design matrix\n",
|
||
"\n",
|
||
"#Compute beta according to normal equations to compare with GD solution\n",
|
||
"Xt_X_inv = np.linalg.inv(np.dot(X.T,X))\n",
|
||
"Xt_y = np.dot(X.transpose(),y)\n",
|
||
"beta_NE = np.dot(Xt_X_inv,Xt_y)\n",
|
||
"print(beta_NE)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"<!-- !split -->\n",
|
||
"## Gradient descent and Ridge\n",
|
||
"\n",
|
||
"We have also discussed Ridge regression where the loss function contains a regularized given by the $L_2$ norm of $\\beta$,"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"$$\n",
|
||
"C_{\\text{ridge}}(\\beta) = ||X\\beta -\\mathbf{y}||^2 + \\lambda ||\\beta||^2, \\ \\lambda \\geq 0.\n",
|
||
"$$"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"In order to minimize $C_{\\text{ridge}}(\\beta)$ using GD we only have adjust the gradient as follows"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"$$\n",
|
||
"\\nabla_\\beta C_{\\text{ridge}}(\\beta) = 2\\begin{bmatrix} \\sum_{i=1}^{100} \\left(\\beta_0+\\beta_1x_i-y_i\\right) \\\\\n",
|
||
"\\sum_{i=1}^{100}\\left( x_i (\\beta_0+\\beta_1x_i)-y_ix_i\\right) \\\\\n",
|
||
"\\end{bmatrix} + 2\\lambda\\begin{bmatrix} \\beta_0 \\\\ \\beta_1\\end{bmatrix} = 2 (X^T(X\\beta - \\mathbf{y})+\\lambda \\beta).\n",
|
||
"$$"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"We can now extend our program to minimize $C_{\\text{ridge}}(\\beta)$ using gradient descent and compare with the analytical solution given by"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"$$\n",
|
||
"\\beta_{\\text{ridge}} = \\left(X^T X + \\lambda I_{2 \\times 2} \\right)^{-1} X^T \\mathbf{y},\n",
|
||
"$$"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"for $\\lambda = {0,1,10,50,100}$ ($\\lambda = 0$ corresponds to ordinary least squares). \n",
|
||
"We can then compute $||\\beta_{\\text{ridge}}||$ for each $\\lambda$."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 4,
|
||
"metadata": {},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"[-0.76259836 4.89939464]\n",
|
||
"4.958389265060428\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"import numpy as np\n",
|
||
"\n",
|
||
"\"\"\"\n",
|
||
"The following setup is just a suggestion, feel free to write it the way you like.\n",
|
||
"\"\"\"\n",
|
||
"\n",
|
||
"#Setup problem described in the exercise\n",
|
||
"N = 100 #Nr of datapoints\n",
|
||
"M = 2 #Nr of features\n",
|
||
"x = np.random.rand(N)\n",
|
||
"y = 5*x**2 + 0.1*np.random.randn(N)\n",
|
||
"\n",
|
||
"\n",
|
||
"#Compute analytic beta for Ridge regression \n",
|
||
"X = np.c_[np.ones(N),x]\n",
|
||
"XT_X = np.dot(X.T,X)\n",
|
||
"\n",
|
||
"l = 0.1 #Ridge parameter lambda\n",
|
||
"Id = np.eye(XT_X.shape[0])\n",
|
||
"\n",
|
||
"Z = np.linalg.inv(XT_X+l*Id)\n",
|
||
"beta_ridge = np.dot(Z,np.dot(X.T,y))\n",
|
||
"\n",
|
||
"print(beta_ridge)\n",
|
||
"print(np.linalg.norm(beta_ridge)) #||beta||"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"## Stochastic Gradient Descent\n",
|
||
"\n",
|
||
"Stochastic gradient descent (SGD) and variants thereof address some of\n",
|
||
"the shortcomings of the Gradient descent method discussed above.\n",
|
||
"\n",
|
||
"The underlying idea of SGD comes from the observation that the cost\n",
|
||
"function, which we want to minimize, can almost always be written as a\n",
|
||
"sum over $n$ datapoints $\\{\\mathbf{x}_i\\}_{i=1}^n$,"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"$$\n",
|
||
"C(\\mathbf{\\beta}) = \\sum_{i=1}^n c_i(\\mathbf{x}_i,\n",
|
||
"\\mathbf{\\beta}).\n",
|
||
"$$"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"## Computation of gradients\n",
|
||
"\n",
|
||
"This in turn means that the gradient can be\n",
|
||
"computed as a sum over $i$-gradients"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"$$\n",
|
||
"\\nabla_\\beta C(\\mathbf{\\beta}) = \\sum_i^n \\nabla_\\beta c_i(\\mathbf{x}_i,\n",
|
||
"\\mathbf{\\beta}).\n",
|
||
"$$"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"Stochasticity/randomness is introduced by only taking the\n",
|
||
"gradient on a subset of the data called minibatches. If there are $n$\n",
|
||
"datapoints and the size of each minibatch is $M$, there will be $n/M$\n",
|
||
"minibatches. We denote these minibatches by $B_k$ where\n",
|
||
"$k=1,\\cdots,n/M$.\n",
|
||
"\n",
|
||
"## SGD example\n",
|
||
"As an example, suppose we have $10$ datapoints $( \\mathbf{x}_1,\n",
|
||
"\\cdots, \\mathbf{x}_{10} )$ and we choose to have $M=5$ minibathces,\n",
|
||
"then each minibatch contains two datapoints. In particular we have\n",
|
||
"$B_1 = (\\mathbf{x}_1,\\mathbf{x}_2), \\cdots, B_5 =\n",
|
||
"(\\mathbf{x}_9,\\mathbf{x}_{10})$. Note that if you choose $M=1$ you\n",
|
||
"have only a single batch with all datapoints and on the other extreme,\n",
|
||
"you may choose $M=n$ resulting in a minibatch for each datapoint, i.e\n",
|
||
"$B_k = \\mathbf{x}_k$.\n",
|
||
"\n",
|
||
"The idea is now to approximate the gradient by replacing the sum over\n",
|
||
"all datapoints with a sum over the datapoints in one the minibatches\n",
|
||
"picked at random in each gradient descent step"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"$$\n",
|
||
"\\nabla_\\beta\n",
|
||
"C(\\mathbf{\\beta}) = \\sum_{i=1}^n \\nabla_\\beta c_i(\\mathbf{x}_i,\n",
|
||
"\\mathbf{\\beta}) \\rightarrow \\sum_{i \\in B_k}^n \\nabla_\\beta\n",
|
||
"c_i(\\mathbf{x}_i, \\mathbf{\\beta}).\n",
|
||
"$$"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"## The gradient step\n",
|
||
"\n",
|
||
"Thus a gradient descent step now looks like"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"$$\n",
|
||
"\\beta_{j+1} = \\beta_j - \\gamma_j \\sum_{i \\in B_k}^n \\nabla_\\beta c_i(\\mathbf{x}_i,\n",
|
||
"\\mathbf{\\beta})\n",
|
||
"$$"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"where $k$ is picked at random with equal\n",
|
||
"probability from $[1,n/M]$. An iteration over the number of\n",
|
||
"minibathces (n/M) is commonly referred to as an epoch. Thus it is\n",
|
||
"typical to choose a number of epochs and for each epoch iterate over\n",
|
||
"the number of minibatches, as exemplified in the code below.\n",
|
||
"\n",
|
||
"## Simple example code"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 5,
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"import numpy as np \n",
|
||
"\n",
|
||
"n = 100 #100 datapoints \n",
|
||
"M = 5 #size of each minibatch\n",
|
||
"m = int(n/M) #number of minibatches\n",
|
||
"n_epochs = 10 #number of epochs\n",
|
||
"\n",
|
||
"j = 0\n",
|
||
"for epoch in range(1,n_epochs+1):\n",
|
||
" for i in range(m):\n",
|
||
" k = np.random.randint(m) #Pick the k-th minibatch at random\n",
|
||
" #Compute the gradient using the data in minibatch Bk\n",
|
||
" #Compute new suggestion for \n",
|
||
" j += 1"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"Taking the gradient only on a subset of the data has two important\n",
|
||
"benefits. First, it introduces randomness which decreases the chance\n",
|
||
"that our opmization scheme gets stuck in a local minima. Second, if\n",
|
||
"the size of the minibatches are small relative to the number of\n",
|
||
"datapoints ($M < n$), the computation of the gradient is much\n",
|
||
"cheaper since we sum over the datapoints in the k-th minibatch and not\n",
|
||
"all $n$ datapoints.\n",
|
||
"\n",
|
||
"## When do we stop?\n",
|
||
"\n",
|
||
"A natural question is when do we stop the search for a new minimum?\n",
|
||
"One possibility is to compute the full gradient after a given number\n",
|
||
"of epochs and check if the norm of the gradient is smaller than some\n",
|
||
"threshold and stop if true. However, the condition that the gradient\n",
|
||
"is zero is valid also for local minima, so this would only tell us\n",
|
||
"that we are close to a local/global minimum. However, we could also\n",
|
||
"evaluate the cost function at this point, store the result and\n",
|
||
"continue the search. If the test kicks in at a later stage we can\n",
|
||
"compare the values of the cost function and keep the $\\beta$ that\n",
|
||
"gave the lowest value.\n",
|
||
"\n",
|
||
"## Slightly different approach\n",
|
||
"\n",
|
||
"Another approach is to let the step length $\\gamma_j$ depend on the\n",
|
||
"number of epochs in such a way that it becomes very small after a\n",
|
||
"reasonable time such that we do not move at all.\n",
|
||
"\n",
|
||
"As an example, let $e = 0,1,2,3,\\cdots$ denote the current epoch and let $t_0, t_1 > 0$ be two fixed numbers. Furthermore, let $t = e \\cdot m + i$ where $m$ is the number of minibatches and $i=0,\\cdots,m-1$. Then the function $$\\gamma_j(t; t_0, t_1) = \\frac{t_0}{t+t_1} $$ goes to zero as the number of epochs gets large. I.e. we start with a step length $\\gamma_j (0; t_0, t_1) = t_0/t_1$ which decays in *time* $t$.\n",
|
||
"\n",
|
||
"In this way we can fix the number of epochs, compute $\\beta$ and\n",
|
||
"evaluate the cost function at the end. Repeating the computation will\n",
|
||
"give a different result since the scheme is random by design. Then we\n",
|
||
"pick the final $\\beta$ that gives the lowest value of the cost\n",
|
||
"function."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 6,
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"import numpy as np \n",
|
||
"\n",
|
||
"def step_length(t,t0,t1):\n",
|
||
" return t0/(t+t1)\n",
|
||
"\n",
|
||
"n = 100 #100 datapoints \n",
|
||
"M = 5 #size of each minibatch\n",
|
||
"m = int(n/M) #number of minibatches\n",
|
||
"n_epochs = 500 #number of epochs\n",
|
||
"t0 = 1.0\n",
|
||
"t1 = 10\n",
|
||
"\n",
|
||
"gamma_j = t0/t1\n",
|
||
"j = 0\n",
|
||
"for epoch in range(1,n_epochs+1):\n",
|
||
" for i in range(m):\n",
|
||
" k = np.random.randint(m) #Pick the k-th minibatch at random\n",
|
||
" #Compute the gradient using the data in minibatch Bk\n",
|
||
" #Compute new suggestion for beta\n",
|
||
" t = epoch*m+i\n",
|
||
" gamma_j = step_length(t,t0,t1)\n",
|
||
" j += 1\n",
|
||
"\n",
|
||
"print(\"gamma_j after %d epochs: %g\" % (n_epochs,gamma_j))"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"## Conjugate gradient (CG) method\n",
|
||
"The success of the CG method for finding solutions of non-linear problems is based\n",
|
||
"on the theory of conjugate gradients for linear systems of equations. It belongs\n",
|
||
"to the class of iterative methods for solving problems from linear algebra of the type"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"$$\n",
|
||
"\\hat{A}\\hat{x} = \\hat{b}.\n",
|
||
"$$"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"In the iterative process we end up with a problem like"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"$$\n",
|
||
"\\hat{r}= \\hat{b}-\\hat{A}\\hat{x},\n",
|
||
"$$"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"where $\\hat{r}$ is the so-called residual or error in the iterative process.\n",
|
||
"\n",
|
||
"When we have found the exact solution, $\\hat{r}=0$.\n",
|
||
"\n",
|
||
"\n",
|
||
"\n",
|
||
"\n",
|
||
"## Conjugate gradient method\n",
|
||
"\n",
|
||
"The residual is zero when we reach the minimum of the quadratic equation"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"$$\n",
|
||
"P(\\hat{x})=\\frac{1}{2}\\hat{x}^T\\hat{A}\\hat{x} - \\hat{x}^T\\hat{b},\n",
|
||
"$$"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"with the constraint that the matrix $\\hat{A}$ is positive definite and symmetric.\n",
|
||
"If we search for a minimum of the quantum mechanical variance, then the matrix \n",
|
||
"$\\hat{A}$, which is called the Hessian, is given by the second-derivative of the function we want to minimize. This quantity is always positive definite. In our case this corresponds normally to the second derivative of the energy.\n",
|
||
"\n",
|
||
"\n",
|
||
"\n",
|
||
"\n",
|
||
"\n",
|
||
"\n",
|
||
"\n",
|
||
"## Conjugate gradient method, Newton's method first\n",
|
||
"We seek the minimum of the energy or the variance as function of various variational parameters. \n",
|
||
"In our case we have thus a function $f$ whose minimum we are seeking.\n",
|
||
"In Newton's method we set $\\nabla f = 0$ and we can thus compute the next iteration point"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"$$\n",
|
||
"\\hat{x}-\\hat{x}_i=\\hat{A}^{-1}\\nabla f(\\hat{x}_i).\n",
|
||
"$$"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"Subtracting this equation from that of $\\hat{x}_{i+1}$ we have"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"$$\n",
|
||
"\\hat{x}_{i+1}-\\hat{x}_i=\\hat{A}^{-1}(\\nabla f(\\hat{x}_{i+1})-\\nabla f(\\hat{x}_i)).\n",
|
||
"$$"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"## Conjugate gradient method\n",
|
||
"In the CG method we define so-called conjugate directions and two vectors \n",
|
||
"$\\hat{s}$ and $\\hat{t}$\n",
|
||
"are said to be\n",
|
||
"conjugate if"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"$$\n",
|
||
"\\hat{s}^T\\hat{A}\\hat{t}= 0.\n",
|
||
"$$"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"The philosophy of the CG method is to perform searches in various conjugate directions\n",
|
||
"of our vectors $\\hat{x}_i$ obeying the above criterion, namely"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"$$\n",
|
||
"\\hat{x}_i^T\\hat{A}\\hat{x}_j= 0.\n",
|
||
"$$"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"Two vectors are conjugate if they are orthogonal with respect to \n",
|
||
"this inner product. Being conjugate is a symmetric relation: if $\\hat{s}$ is conjugate to $\\hat{t}$, then $\\hat{t}$ is conjugate to $\\hat{s}$.\n",
|
||
"\n",
|
||
"\n",
|
||
"\n",
|
||
"## Conjugate gradient method\n",
|
||
"An example is given by the eigenvectors of the matrix"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"$$\n",
|
||
"\\hat{v}_i^T\\hat{A}\\hat{v}_j= \\lambda\\hat{v}_i^T\\hat{v}_j,\n",
|
||
"$$"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"which is zero unless $i=j$.\n",
|
||
"\n",
|
||
"\n",
|
||
"\n",
|
||
"\n",
|
||
"## Conjugate gradient method\n",
|
||
"Assume now that we have a symmetric positive-definite matrix $\\hat{A}$ of size\n",
|
||
"$n\\times n$. At each iteration $i+1$ we obtain the conjugate direction of a vector"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"$$\n",
|
||
"\\hat{x}_{i+1}=\\hat{x}_{i}+\\alpha_i\\hat{p}_{i}.\n",
|
||
"$$"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"We assume that $\\hat{p}_{i}$ is a sequence of $n$ mutually conjugate directions. \n",
|
||
"Then the $\\hat{p}_{i}$ form a basis of $R^n$ and we can expand the solution \n",
|
||
"$ \\hat{A}\\hat{x} = \\hat{b}$ in this basis, namely"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"$$\n",
|
||
"\\hat{x} = \\sum^{n}_{i=1} \\alpha_i \\hat{p}_i.\n",
|
||
"$$"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"## Conjugate gradient method\n",
|
||
"The coefficients are given by"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"$$\n",
|
||
"\\mathbf{A}\\mathbf{x} = \\sum^{n}_{i=1} \\alpha_i \\mathbf{A} \\mathbf{p}_i = \\mathbf{b}.\n",
|
||
"$$"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"Multiplying with $\\hat{p}_k^T$ from the left gives"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"$$\n",
|
||
"\\hat{p}_k^T \\hat{A}\\hat{x} = \\sum^{n}_{i=1} \\alpha_i\\hat{p}_k^T \\hat{A}\\hat{p}_i= \\hat{p}_k^T \\hat{b},\n",
|
||
"$$"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"and we can define the coefficients $\\alpha_k$ as"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"$$\n",
|
||
"\\alpha_k = \\frac{\\hat{p}_k^T \\hat{b}}{\\hat{p}_k^T \\hat{A} \\hat{p}_k}\n",
|
||
"$$"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"## Conjugate gradient method and iterations\n",
|
||
"\n",
|
||
"If we choose the conjugate vectors $\\hat{p}_k$ carefully, \n",
|
||
"then we may not need all of them to obtain a good approximation to the solution \n",
|
||
"$\\hat{x}$. \n",
|
||
"We want to regard the conjugate gradient method as an iterative method. \n",
|
||
"This will us to solve systems where $n$ is so large that the direct \n",
|
||
"method would take too much time.\n",
|
||
"\n",
|
||
"We denote the initial guess for $\\hat{x}$ as $\\hat{x}_0$. \n",
|
||
"We can assume without loss of generality that"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"$$\n",
|
||
"\\hat{x}_0=0,\n",
|
||
"$$"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"or consider the system"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"$$\n",
|
||
"\\hat{A}\\hat{z} = \\hat{b}-\\hat{A}\\hat{x}_0,\n",
|
||
"$$"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"instead.\n",
|
||
"\n",
|
||
"\n",
|
||
"\n",
|
||
"\n",
|
||
"## Conjugate gradient method\n",
|
||
"One can show that the solution $\\hat{x}$ is also the unique minimizer of the quadratic form"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"$$\n",
|
||
"f(\\hat{x}) = \\frac{1}{2}\\hat{x}^T\\hat{A}\\hat{x} - \\hat{x}^T \\hat{x} , \\quad \\hat{x}\\in\\mathbf{R}^n.\n",
|
||
"$$"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"This suggests taking the first basis vector $\\hat{p}_1$ \n",
|
||
"to be the gradient of $f$ at $\\hat{x}=\\hat{x}_0$, \n",
|
||
"which equals"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"$$\n",
|
||
"\\hat{A}\\hat{x}_0-\\hat{b},\n",
|
||
"$$"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"and \n",
|
||
"$\\hat{x}_0=0$ it is equal $-\\hat{b}$.\n",
|
||
"The other vectors in the basis will be conjugate to the gradient, \n",
|
||
"hence the name conjugate gradient method.\n",
|
||
"\n",
|
||
"\n",
|
||
"\n",
|
||
"\n",
|
||
"## Conjugate gradient method\n",
|
||
"Let $\\hat{r}_k$ be the residual at the $k$-th step:"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"$$\n",
|
||
"\\hat{r}_k=\\hat{b}-\\hat{A}\\hat{x}_k.\n",
|
||
"$$"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"Note that $\\hat{r}_k$ is the negative gradient of $f$ at \n",
|
||
"$\\hat{x}=\\hat{x}_k$, \n",
|
||
"so the gradient descent method would be to move in the direction $\\hat{r}_k$. \n",
|
||
"Here, we insist that the directions $\\hat{p}_k$ are conjugate to each other, \n",
|
||
"so we take the direction closest to the gradient $\\hat{r}_k$ \n",
|
||
"under the conjugacy constraint. \n",
|
||
"This gives the following expression"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"$$\n",
|
||
"\\hat{p}_{k+1}=\\hat{r}_k-\\frac{\\hat{p}_k^T \\hat{A}\\hat{r}_k}{\\hat{p}_k^T\\hat{A}\\hat{p}_k} \\hat{p}_k.\n",
|
||
"$$"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"## Conjugate gradient method\n",
|
||
"We can also compute the residual iteratively as"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"$$\n",
|
||
"\\hat{r}_{k+1}=\\hat{b}-\\hat{A}\\hat{x}_{k+1},\n",
|
||
"$$"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"which equals"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"$$\n",
|
||
"\\hat{b}-\\hat{A}(\\hat{x}_k+\\alpha_k\\hat{p}_k),\n",
|
||
"$$"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"or"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"$$\n",
|
||
"(\\hat{b}-\\hat{A}\\hat{x}_k)-\\alpha_k\\hat{A}\\hat{p}_k,\n",
|
||
"$$"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"which gives"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"$$\n",
|
||
"\\hat{r}_{k+1}=\\hat{r}_k-\\hat{A}\\hat{p}_{k},\n",
|
||
"$$"
|
||
]
|
||
}
|
||
],
|
||
"metadata": {
|
||
"kernelspec": {
|
||
"display_name": "Python 3",
|
||
"language": "python",
|
||
"name": "python3"
|
||
},
|
||
"language_info": {
|
||
"codemirror_mode": {
|
||
"name": "ipython",
|
||
"version": 3
|
||
},
|
||
"file_extension": ".py",
|
||
"mimetype": "text/x-python",
|
||
"name": "python",
|
||
"nbconvert_exporter": "python",
|
||
"pygments_lexer": "ipython3",
|
||
"version": "3.7.0"
|
||
}
|
||
},
|
||
"nbformat": 4,
|
||
"nbformat_minor": 2
|
||
}
|