update on gradient descent
This commit is contained in:
mhjensen
File diff suppressed because it is too large
Load Diff
Binary file not shown.
|
Before Width: | Height: | Size: 20 KiB After Width: | Height: | Size: 21 KiB |
File diff suppressed because it is too large
Load Diff
Binary file not shown.
|
Before Width: | Height: | Size: 41 KiB After Width: | Height: | Size: 39 KiB |
File diff suppressed because it is too large
Load Diff
Binary file not shown.
|
Before Width: | Height: | Size: 31 KiB After Width: | Height: | Size: 32 KiB |
@@ -1,7 +1,7 @@
|
||||
%!PS-Adobe-3.0 EPSF-3.0
|
||||
%%Title: gaussian/hist.eps
|
||||
%%Creator: matplotlib version 2.0.2, http://matplotlib.org/
|
||||
%%CreationDate: Thu Sep 13 11:46:15 2018
|
||||
%%CreationDate: Wed Sep 19 12:27:13 2018
|
||||
%%Orientation: portrait
|
||||
%%BoundingBox: 75 223 536 568
|
||||
%%EndComments
|
||||
@@ -42,7 +42,7 @@ newpath
|
||||
/FontMatrix[.001 0 0 .001 0 0]def
|
||||
/FontBBox[-1021 -463 1793 1232]def
|
||||
/FontType 3 def
|
||||
/Encoding [ /period /zero /one /two /three /four /five /six /seven /eight /nine ] def
|
||||
/Encoding [ /period /zero /one /two /three /four /five /eight /nine ] def
|
||||
/FontInfo 10 dict dup begin
|
||||
/FamilyName (DejaVu Sans) def
|
||||
/FullName (DejaVu Sans) def
|
||||
@@ -54,7 +54,7 @@ newpath
|
||||
/UnderlinePosition -130 def
|
||||
/UnderlineThickness 90 def
|
||||
end readonly def
|
||||
/CharStrings 11 dict dup begin
|
||||
/CharStrings 9 dict dup begin
|
||||
/period{318 0 107 0 210 124 _sc
|
||||
107 124 _m
|
||||
210 124 _l
|
||||
@@ -195,45 +195,6 @@ _cl}_d
|
||||
162 384 135 375 108 363 _c
|
||||
108 729 _l
|
||||
_cl}_e}_d
|
||||
/six{{636 0 70 -13 573 742 _sc
|
||||
330 404 _m
|
||||
286 404 251 388 225 358 _c
|
||||
199 328 186 286 186 234 _c
|
||||
186 181 199 139 225 109 _c
|
||||
251 79 286 64 330 64 _c
|
||||
374 64 409 79 435 109 _c
|
||||
461 139 474 181 474 234 _c
|
||||
474 286 461 328 435 358 _c
|
||||
409 388 374 404 330 404 _c
|
||||
526 713 _m
|
||||
526 623 _l
|
||||
501 635 476 644 451 650 _c
|
||||
425 656 400 659 376 659 _c
|
||||
310 659 260 637 226 593 _c
|
||||
}_e{192 549 172 482 168 394 _c
|
||||
187 422 211 444 240 459 _c
|
||||
269 474 301 482 336 482 _c
|
||||
409 482 467 459 509 415 _c
|
||||
551 371 573 310 573 234 _c
|
||||
573 159 550 99 506 54 _c
|
||||
462 9 403 -13 330 -13 _c
|
||||
246 -13 181 19 137 83 _c
|
||||
92 147 70 241 70 364 _c
|
||||
70 479 97 571 152 639 _c
|
||||
206 707 280 742 372 742 _c
|
||||
396 742 421 739 447 735 _c
|
||||
472 730 498 723 526 713 _c
|
||||
_cl}_e}_d
|
||||
/seven{636 0 82 0 551 729 _sc
|
||||
82 729 _m
|
||||
551 729 _l
|
||||
551 687 _l
|
||||
286 0 _l
|
||||
183 0 _l
|
||||
432 646 _l
|
||||
82 646 _l
|
||||
82 729 _l
|
||||
_cl}_d
|
||||
/eight{{636 0 68 -13 568 742 _sc
|
||||
318 346 _m
|
||||
271 346 234 333 207 308 _c
|
||||
@@ -782,8 +743,8 @@ gsave
|
||||
357.1 266.1 57.6 38.02 clipbox
|
||||
87.083933 38.016 m
|
||||
93.709536 38.016 l
|
||||
93.709536 38.080818 l
|
||||
87.083933 38.080818 l
|
||||
93.709536 38.016 l
|
||||
87.083933 38.016 l
|
||||
cl
|
||||
0.122 0.467 0.706 setrgbcolor
|
||||
fill
|
||||
@@ -792,8 +753,8 @@ gsave
|
||||
357.1 266.1 57.6 38.02 clipbox
|
||||
93.709536 38.016 m
|
||||
100.335139 38.016 l
|
||||
100.335139 38.048409 l
|
||||
93.709536 38.048409 l
|
||||
100.335139 38.142419 l
|
||||
93.709536 38.142419 l
|
||||
cl
|
||||
0.122 0.467 0.706 setrgbcolor
|
||||
fill
|
||||
@@ -812,8 +773,8 @@ gsave
|
||||
357.1 266.1 57.6 38.02 clipbox
|
||||
106.960742 38.016 m
|
||||
113.586345 38.016 l
|
||||
113.586345 38.080818 l
|
||||
106.960742 38.080818 l
|
||||
113.586345 38.07921 l
|
||||
106.960742 38.07921 l
|
||||
cl
|
||||
0.122 0.467 0.706 setrgbcolor
|
||||
fill
|
||||
@@ -822,8 +783,8 @@ gsave
|
||||
357.1 266.1 57.6 38.02 clipbox
|
||||
113.586345 38.016 m
|
||||
120.211948 38.016 l
|
||||
120.211948 38.307683 l
|
||||
113.586345 38.307683 l
|
||||
120.211948 38.07921 l
|
||||
113.586345 38.07921 l
|
||||
cl
|
||||
0.122 0.467 0.706 setrgbcolor
|
||||
fill
|
||||
@@ -832,8 +793,8 @@ gsave
|
||||
357.1 266.1 57.6 38.02 clipbox
|
||||
120.211948 38.016 m
|
||||
126.837551 38.016 l
|
||||
126.837551 38.469729 l
|
||||
120.211948 38.469729 l
|
||||
126.837551 38.458467 l
|
||||
120.211948 38.458467 l
|
||||
cl
|
||||
0.122 0.467 0.706 setrgbcolor
|
||||
fill
|
||||
@@ -842,8 +803,8 @@ gsave
|
||||
357.1 266.1 57.6 38.02 clipbox
|
||||
126.837551 38.016 m
|
||||
133.463154 38.016 l
|
||||
133.463154 39.442005 l
|
||||
126.837551 39.442005 l
|
||||
133.463154 39.3434 l
|
||||
126.837551 39.3434 l
|
||||
cl
|
||||
0.122 0.467 0.706 setrgbcolor
|
||||
fill
|
||||
@@ -852,8 +813,8 @@ gsave
|
||||
357.1 266.1 57.6 38.02 clipbox
|
||||
133.463154 38.016 m
|
||||
140.088757 38.016 l
|
||||
140.088757 41.224512 l
|
||||
133.463154 41.224512 l
|
||||
140.088757 40.923637 l
|
||||
133.463154 40.923637 l
|
||||
cl
|
||||
0.122 0.467 0.706 setrgbcolor
|
||||
fill
|
||||
@@ -862,8 +823,8 @@ gsave
|
||||
357.1 266.1 57.6 38.02 clipbox
|
||||
140.088757 38.016 m
|
||||
146.71436 38.016 l
|
||||
146.71436 42.423652 l
|
||||
140.088757 42.423652 l
|
||||
146.71436 42.314246 l
|
||||
140.088757 42.314246 l
|
||||
cl
|
||||
0.122 0.467 0.706 setrgbcolor
|
||||
fill
|
||||
@@ -872,8 +833,8 @@ gsave
|
||||
357.1 266.1 57.6 38.02 clipbox
|
||||
146.71436 38.016 m
|
||||
153.339963 38.016 l
|
||||
153.339963 45.178435 l
|
||||
146.71436 45.178435 l
|
||||
153.339963 45.158674 l
|
||||
146.71436 45.158674 l
|
||||
cl
|
||||
0.122 0.467 0.706 setrgbcolor
|
||||
fill
|
||||
@@ -882,8 +843,8 @@ gsave
|
||||
357.1 266.1 57.6 38.02 clipbox
|
||||
153.339963 38.016 m
|
||||
159.965566 38.016 l
|
||||
159.965566 51.012092 l
|
||||
153.339963 51.012092 l
|
||||
159.965566 51.100367 l
|
||||
153.339963 51.100367 l
|
||||
cl
|
||||
0.122 0.467 0.706 setrgbcolor
|
||||
fill
|
||||
@@ -892,8 +853,8 @@ gsave
|
||||
357.1 266.1 57.6 38.02 clipbox
|
||||
159.965566 38.016 m
|
||||
166.591169 38.016 l
|
||||
166.591169 59.27644 l
|
||||
159.965566 59.27644 l
|
||||
166.591169 58.116621 l
|
||||
159.965566 58.116621 l
|
||||
cl
|
||||
0.122 0.467 0.706 setrgbcolor
|
||||
fill
|
||||
@@ -902,8 +863,8 @@ gsave
|
||||
357.1 266.1 57.6 38.02 clipbox
|
||||
166.591169 38.016 m
|
||||
173.216772 38.016 l
|
||||
173.216772 72.013258 l
|
||||
166.591169 72.013258 l
|
||||
173.216772 71.390617 l
|
||||
166.591169 71.390617 l
|
||||
cl
|
||||
0.122 0.467 0.706 setrgbcolor
|
||||
fill
|
||||
@@ -912,8 +873,8 @@ gsave
|
||||
357.1 266.1 57.6 38.02 clipbox
|
||||
173.216772 38.016 m
|
||||
179.842375 38.016 l
|
||||
179.842375 86.953903 l
|
||||
173.216772 86.953903 l
|
||||
179.842375 85.486335 l
|
||||
173.216772 85.486335 l
|
||||
cl
|
||||
0.122 0.467 0.706 setrgbcolor
|
||||
fill
|
||||
@@ -922,8 +883,8 @@ gsave
|
||||
357.1 266.1 57.6 38.02 clipbox
|
||||
179.842375 38.016 m
|
||||
186.467978 38.016 l
|
||||
186.467978 107.598568 l
|
||||
179.842375 107.598568 l
|
||||
186.467978 108.557803 l
|
||||
179.842375 108.557803 l
|
||||
cl
|
||||
0.122 0.467 0.706 setrgbcolor
|
||||
fill
|
||||
@@ -932,8 +893,8 @@ gsave
|
||||
357.1 266.1 57.6 38.02 clipbox
|
||||
186.467978 38.016 m
|
||||
193.093581 38.016 l
|
||||
193.093581 134.07689 l
|
||||
186.467978 134.07689 l
|
||||
193.093581 129.859405 l
|
||||
186.467978 129.859405 l
|
||||
cl
|
||||
0.122 0.467 0.706 setrgbcolor
|
||||
fill
|
||||
@@ -942,8 +903,8 @@ gsave
|
||||
357.1 266.1 57.6 38.02 clipbox
|
||||
193.093581 38.016 m
|
||||
199.719184 38.016 l
|
||||
199.719184 163.569269 l
|
||||
193.093581 163.569269 l
|
||||
199.719184 162.349089 l
|
||||
193.093581 162.349089 l
|
||||
cl
|
||||
0.122 0.467 0.706 setrgbcolor
|
||||
fill
|
||||
@@ -952,8 +913,8 @@ gsave
|
||||
357.1 266.1 57.6 38.02 clipbox
|
||||
199.719184 38.016 m
|
||||
206.344787 38.016 l
|
||||
206.344787 195.881248 l
|
||||
199.719184 195.881248 l
|
||||
206.344787 191.109412 l
|
||||
199.719184 191.109412 l
|
||||
cl
|
||||
0.122 0.467 0.706 setrgbcolor
|
||||
fill
|
||||
@@ -962,8 +923,8 @@ gsave
|
||||
357.1 266.1 57.6 38.02 clipbox
|
||||
206.344787 38.016 m
|
||||
212.97039 38.016 l
|
||||
212.97039 226.669995 l
|
||||
206.344787 226.669995 l
|
||||
212.97039 223.662305 l
|
||||
206.344787 223.662305 l
|
||||
cl
|
||||
0.122 0.467 0.706 setrgbcolor
|
||||
fill
|
||||
@@ -972,8 +933,8 @@ gsave
|
||||
357.1 266.1 57.6 38.02 clipbox
|
||||
212.97039 38.016 m
|
||||
219.595993 38.016 l
|
||||
219.595993 252.532542 l
|
||||
212.97039 252.532542 l
|
||||
219.595993 245.84884 l
|
||||
212.97039 245.84884 l
|
||||
cl
|
||||
0.122 0.467 0.706 setrgbcolor
|
||||
fill
|
||||
@@ -982,8 +943,8 @@ gsave
|
||||
357.1 266.1 57.6 38.02 clipbox
|
||||
219.595993 38.016 m
|
||||
226.221596 38.016 l
|
||||
226.221596 280.598916 l
|
||||
219.595993 280.598916 l
|
||||
226.221596 270.184498 l
|
||||
219.595993 270.184498 l
|
||||
cl
|
||||
0.122 0.467 0.706 setrgbcolor
|
||||
fill
|
||||
@@ -992,8 +953,8 @@ gsave
|
||||
357.1 266.1 57.6 38.02 clipbox
|
||||
226.221596 38.016 m
|
||||
232.847199 38.016 l
|
||||
232.847199 297.289657 l
|
||||
226.221596 297.289657 l
|
||||
232.847199 287.756739 l
|
||||
226.221596 287.756739 l
|
||||
cl
|
||||
0.122 0.467 0.706 setrgbcolor
|
||||
fill
|
||||
@@ -1002,8 +963,8 @@ gsave
|
||||
357.1 266.1 57.6 38.02 clipbox
|
||||
232.847199 38.016 m
|
||||
239.472801 38.016 l
|
||||
239.472801 303.803908 l
|
||||
232.847199 303.803908 l
|
||||
239.472801 303.495905 l
|
||||
232.847199 303.495905 l
|
||||
cl
|
||||
0.122 0.467 0.706 setrgbcolor
|
||||
fill
|
||||
@@ -1012,8 +973,8 @@ gsave
|
||||
357.1 266.1 57.6 38.02 clipbox
|
||||
239.472801 38.016 m
|
||||
246.098404 38.016 l
|
||||
246.098404 291.87732 l
|
||||
239.472801 291.87732 l
|
||||
246.098404 284.469845 l
|
||||
239.472801 284.469845 l
|
||||
cl
|
||||
0.122 0.467 0.706 setrgbcolor
|
||||
fill
|
||||
@@ -1022,8 +983,8 @@ gsave
|
||||
357.1 266.1 57.6 38.02 clipbox
|
||||
246.098404 38.016 m
|
||||
252.724007 38.016 l
|
||||
252.724007 279.237729 l
|
||||
246.098404 279.237729 l
|
||||
252.724007 276.379029 l
|
||||
246.098404 276.379029 l
|
||||
cl
|
||||
0.122 0.467 0.706 setrgbcolor
|
||||
fill
|
||||
@@ -1032,8 +993,8 @@ gsave
|
||||
357.1 266.1 57.6 38.02 clipbox
|
||||
252.724007 38.016 m
|
||||
259.34961 38.016 l
|
||||
259.34961 257.977289 l
|
||||
252.724007 257.977289 l
|
||||
259.34961 252.359419 l
|
||||
252.724007 252.359419 l
|
||||
cl
|
||||
0.122 0.467 0.706 setrgbcolor
|
||||
fill
|
||||
@@ -1042,8 +1003,8 @@ gsave
|
||||
357.1 266.1 57.6 38.02 clipbox
|
||||
259.34961 38.016 m
|
||||
265.975213 38.016 l
|
||||
265.975213 229.813688 l
|
||||
259.34961 229.813688 l
|
||||
265.975213 226.633152 l
|
||||
259.34961 226.633152 l
|
||||
cl
|
||||
0.122 0.467 0.706 setrgbcolor
|
||||
fill
|
||||
@@ -1052,8 +1013,8 @@ gsave
|
||||
357.1 266.1 57.6 38.02 clipbox
|
||||
265.975213 38.016 m
|
||||
272.600816 38.016 l
|
||||
272.600816 194.390425 l
|
||||
265.975213 194.390425 l
|
||||
272.600816 187.064004 l
|
||||
265.975213 187.064004 l
|
||||
cl
|
||||
0.122 0.467 0.706 setrgbcolor
|
||||
fill
|
||||
@@ -1062,8 +1023,8 @@ gsave
|
||||
357.1 266.1 57.6 38.02 clipbox
|
||||
272.600816 38.016 m
|
||||
279.226419 38.016 l
|
||||
279.226419 162.467355 l
|
||||
272.600816 162.467355 l
|
||||
279.226419 156.723443 l
|
||||
272.600816 156.723443 l
|
||||
cl
|
||||
0.122 0.467 0.706 setrgbcolor
|
||||
fill
|
||||
@@ -1072,8 +1033,8 @@ gsave
|
||||
357.1 266.1 57.6 38.02 clipbox
|
||||
279.226419 38.016 m
|
||||
285.852022 38.016 l
|
||||
285.852022 133.428706 l
|
||||
279.226419 133.428706 l
|
||||
285.852022 130.112243 l
|
||||
279.226419 130.112243 l
|
||||
cl
|
||||
0.122 0.467 0.706 setrgbcolor
|
||||
fill
|
||||
@@ -1082,8 +1043,8 @@ gsave
|
||||
357.1 266.1 57.6 38.02 clipbox
|
||||
285.852022 38.016 m
|
||||
292.477625 38.016 l
|
||||
292.477625 107.015202 l
|
||||
285.852022 107.015202 l
|
||||
292.477625 105.586957 l
|
||||
285.852022 105.586957 l
|
||||
cl
|
||||
0.122 0.467 0.706 setrgbcolor
|
||||
fill
|
||||
@@ -1092,8 +1053,8 @@ gsave
|
||||
357.1 266.1 57.6 38.02 clipbox
|
||||
292.477625 38.016 m
|
||||
299.103228 38.016 l
|
||||
299.103228 87.115949 l
|
||||
292.477625 87.115949 l
|
||||
299.103228 87.951506 l
|
||||
292.477625 87.951506 l
|
||||
cl
|
||||
0.122 0.467 0.706 setrgbcolor
|
||||
fill
|
||||
@@ -1102,8 +1063,8 @@ gsave
|
||||
357.1 266.1 57.6 38.02 clipbox
|
||||
299.103228 38.016 m
|
||||
305.728831 38.016 l
|
||||
305.728831 72.304941 l
|
||||
299.103228 72.304941 l
|
||||
305.728831 72.970854 l
|
||||
299.103228 72.970854 l
|
||||
cl
|
||||
0.122 0.467 0.706 setrgbcolor
|
||||
fill
|
||||
@@ -1112,8 +1073,8 @@ gsave
|
||||
357.1 266.1 57.6 38.02 clipbox
|
||||
305.728831 38.016 m
|
||||
312.354434 38.016 l
|
||||
312.354434 58.887529 l
|
||||
305.728831 58.887529 l
|
||||
312.354434 59.064764 l
|
||||
305.728831 59.064764 l
|
||||
cl
|
||||
0.122 0.467 0.706 setrgbcolor
|
||||
fill
|
||||
@@ -1122,8 +1083,8 @@ gsave
|
||||
357.1 266.1 57.6 38.02 clipbox
|
||||
312.354434 38.016 m
|
||||
318.980037 38.016 l
|
||||
318.980037 51.10932 l
|
||||
312.354434 51.10932 l
|
||||
318.980037 50.468272 l
|
||||
312.354434 50.468272 l
|
||||
cl
|
||||
0.122 0.467 0.706 setrgbcolor
|
||||
fill
|
||||
@@ -1132,8 +1093,8 @@ gsave
|
||||
357.1 266.1 57.6 38.02 clipbox
|
||||
318.980037 38.016 m
|
||||
325.60564 38.016 l
|
||||
325.60564 46.247939 l
|
||||
318.980037 46.247939 l
|
||||
325.60564 44.905836 l
|
||||
318.980037 44.905836 l
|
||||
cl
|
||||
0.122 0.467 0.706 setrgbcolor
|
||||
fill
|
||||
@@ -1142,8 +1103,8 @@ gsave
|
||||
357.1 266.1 57.6 38.02 clipbox
|
||||
325.60564 38.016 m
|
||||
332.231243 38.016 l
|
||||
332.231243 42.456061 l
|
||||
325.60564 42.456061 l
|
||||
332.231243 42.503875 l
|
||||
325.60564 42.503875 l
|
||||
cl
|
||||
0.122 0.467 0.706 setrgbcolor
|
||||
fill
|
||||
@@ -1152,8 +1113,8 @@ gsave
|
||||
357.1 266.1 57.6 38.02 clipbox
|
||||
332.231243 38.016 m
|
||||
338.856846 38.016 l
|
||||
338.856846 39.830916 l
|
||||
332.231243 39.830916 l
|
||||
338.856846 39.659447 l
|
||||
332.231243 39.659447 l
|
||||
cl
|
||||
0.122 0.467 0.706 setrgbcolor
|
||||
fill
|
||||
@@ -1162,8 +1123,8 @@ gsave
|
||||
357.1 266.1 57.6 38.02 clipbox
|
||||
338.856846 38.016 m
|
||||
345.482449 38.016 l
|
||||
345.482449 39.701279 l
|
||||
338.856846 39.701279 l
|
||||
345.482449 39.533028 l
|
||||
338.856846 39.533028 l
|
||||
cl
|
||||
0.122 0.467 0.706 setrgbcolor
|
||||
fill
|
||||
@@ -1172,8 +1133,8 @@ gsave
|
||||
357.1 266.1 57.6 38.02 clipbox
|
||||
345.482449 38.016 m
|
||||
352.108052 38.016 l
|
||||
352.108052 38.534547 l
|
||||
345.482449 38.534547 l
|
||||
352.108052 38.648095 l
|
||||
345.482449 38.648095 l
|
||||
cl
|
||||
0.122 0.467 0.706 setrgbcolor
|
||||
fill
|
||||
@@ -1182,8 +1143,8 @@ gsave
|
||||
357.1 266.1 57.6 38.02 clipbox
|
||||
352.108052 38.016 m
|
||||
358.733655 38.016 l
|
||||
358.733655 38.307683 l
|
||||
352.108052 38.307683 l
|
||||
358.733655 38.268838 l
|
||||
352.108052 38.268838 l
|
||||
cl
|
||||
0.122 0.467 0.706 setrgbcolor
|
||||
fill
|
||||
@@ -1192,8 +1153,8 @@ gsave
|
||||
357.1 266.1 57.6 38.02 clipbox
|
||||
358.733655 38.016 m
|
||||
365.359258 38.016 l
|
||||
365.359258 38.242864 l
|
||||
358.733655 38.242864 l
|
||||
365.359258 38.268838 l
|
||||
358.733655 38.268838 l
|
||||
cl
|
||||
0.122 0.467 0.706 setrgbcolor
|
||||
fill
|
||||
@@ -1202,8 +1163,8 @@ gsave
|
||||
357.1 266.1 57.6 38.02 clipbox
|
||||
365.359258 38.016 m
|
||||
371.984861 38.016 l
|
||||
371.984861 38.113228 l
|
||||
365.359258 38.113228 l
|
||||
371.984861 38.142419 l
|
||||
365.359258 38.142419 l
|
||||
cl
|
||||
0.122 0.467 0.706 setrgbcolor
|
||||
fill
|
||||
@@ -1271,13 +1232,13 @@ grestore
|
||||
stroke
|
||||
grestore
|
||||
} bind def
|
||||
106.48 38.016 o
|
||||
106.602 38.016 o
|
||||
grestore
|
||||
/DejaVuSans findfont
|
||||
10.000 scalefont
|
||||
setfont
|
||||
gsave
|
||||
98.526656 23.422250 translate
|
||||
98.648800 23.422250 translate
|
||||
0.000000 rotate
|
||||
0.000000 0.000000 m /nine glyphshow
|
||||
6.362305 0.000000 m /period glyphshow
|
||||
@@ -1301,10 +1262,10 @@ grestore
|
||||
stroke
|
||||
grestore
|
||||
} bind def
|
||||
171.308 38.016 o
|
||||
171.362 38.016 o
|
||||
grestore
|
||||
gsave
|
||||
163.354947 23.422250 translate
|
||||
163.408693 23.422250 translate
|
||||
0.000000 rotate
|
||||
0.000000 0.000000 m /nine glyphshow
|
||||
6.362305 0.000000 m /period glyphshow
|
||||
@@ -1328,10 +1289,10 @@ grestore
|
||||
stroke
|
||||
grestore
|
||||
} bind def
|
||||
236.136 38.016 o
|
||||
236.122 38.016 o
|
||||
grestore
|
||||
gsave
|
||||
225.003550 23.422250 translate
|
||||
224.988897 23.422250 translate
|
||||
0.000000 rotate
|
||||
0.000000 0.000000 m /one glyphshow
|
||||
6.362305 0.000000 m /zero glyphshow
|
||||
@@ -1356,10 +1317,10 @@ grestore
|
||||
stroke
|
||||
grestore
|
||||
} bind def
|
||||
300.965 38.016 o
|
||||
300.882 38.016 o
|
||||
grestore
|
||||
gsave
|
||||
289.831840 23.422250 translate
|
||||
289.748790 23.422250 translate
|
||||
0.000000 rotate
|
||||
0.000000 0.000000 m /one glyphshow
|
||||
6.362305 0.000000 m /zero glyphshow
|
||||
@@ -1384,10 +1345,10 @@ grestore
|
||||
stroke
|
||||
grestore
|
||||
} bind def
|
||||
365.793 38.016 o
|
||||
365.641 38.016 o
|
||||
grestore
|
||||
gsave
|
||||
354.660131 23.422250 translate
|
||||
354.508682 23.422250 translate
|
||||
0.000000 rotate
|
||||
0.000000 0.000000 m /one glyphshow
|
||||
6.362305 0.000000 m /zero glyphshow
|
||||
@@ -1455,10 +1416,37 @@ grestore
|
||||
stroke
|
||||
grestore
|
||||
} bind def
|
||||
57.6 70.4252 o
|
||||
57.6 69.6208 o
|
||||
grestore
|
||||
gsave
|
||||
25.162500 66.628332 translate
|
||||
31.521875 65.823876 translate
|
||||
0.000000 rotate
|
||||
0.000000 0.000000 m /five glyphshow
|
||||
6.362305 0.000000 m /zero glyphshow
|
||||
12.724609 0.000000 m /zero glyphshow
|
||||
grestore
|
||||
gsave
|
||||
/o {
|
||||
gsave
|
||||
newpath
|
||||
translate
|
||||
0.8 setlinewidth
|
||||
1 setlinejoin
|
||||
0 setlinecap
|
||||
0 0 m
|
||||
-3.5 0 l
|
||||
|
||||
gsave
|
||||
0.000 setgray
|
||||
fill
|
||||
grestore
|
||||
stroke
|
||||
grestore
|
||||
} bind def
|
||||
57.6 101.226 o
|
||||
grestore
|
||||
gsave
|
||||
25.162500 97.428626 translate
|
||||
0.000000 rotate
|
||||
0.000000 0.000000 m /one glyphshow
|
||||
6.362305 0.000000 m /zero glyphshow
|
||||
@@ -1483,10 +1471,38 @@ grestore
|
||||
stroke
|
||||
grestore
|
||||
} bind def
|
||||
57.6 102.834 o
|
||||
57.6 132.83 o
|
||||
grestore
|
||||
gsave
|
||||
25.162500 99.037539 translate
|
||||
25.162500 129.033377 translate
|
||||
0.000000 rotate
|
||||
0.000000 0.000000 m /one glyphshow
|
||||
6.362305 0.000000 m /five glyphshow
|
||||
12.724609 0.000000 m /zero glyphshow
|
||||
19.086914 0.000000 m /zero glyphshow
|
||||
grestore
|
||||
gsave
|
||||
/o {
|
||||
gsave
|
||||
newpath
|
||||
translate
|
||||
0.8 setlinewidth
|
||||
1 setlinejoin
|
||||
0 setlinecap
|
||||
0 0 m
|
||||
-3.5 0 l
|
||||
|
||||
gsave
|
||||
0.000 setgray
|
||||
fill
|
||||
grestore
|
||||
stroke
|
||||
grestore
|
||||
} bind def
|
||||
57.6 164.435 o
|
||||
grestore
|
||||
gsave
|
||||
25.162500 160.638127 translate
|
||||
0.000000 rotate
|
||||
0.000000 0.000000 m /two glyphshow
|
||||
6.362305 0.000000 m /zero glyphshow
|
||||
@@ -1511,10 +1527,38 @@ grestore
|
||||
stroke
|
||||
grestore
|
||||
} bind def
|
||||
57.6 135.244 o
|
||||
57.6 196.04 o
|
||||
grestore
|
||||
gsave
|
||||
25.162500 131.446746 translate
|
||||
25.162500 192.242878 translate
|
||||
0.000000 rotate
|
||||
0.000000 0.000000 m /two glyphshow
|
||||
6.362305 0.000000 m /five glyphshow
|
||||
12.724609 0.000000 m /zero glyphshow
|
||||
19.086914 0.000000 m /zero glyphshow
|
||||
grestore
|
||||
gsave
|
||||
/o {
|
||||
gsave
|
||||
newpath
|
||||
translate
|
||||
0.8 setlinewidth
|
||||
1 setlinejoin
|
||||
0 setlinecap
|
||||
0 0 m
|
||||
-3.5 0 l
|
||||
|
||||
gsave
|
||||
0.000 setgray
|
||||
fill
|
||||
grestore
|
||||
stroke
|
||||
grestore
|
||||
} bind def
|
||||
57.6 227.645 o
|
||||
grestore
|
||||
gsave
|
||||
25.162500 223.847629 translate
|
||||
0.000000 rotate
|
||||
0.000000 0.000000 m /three glyphshow
|
||||
6.362305 0.000000 m /zero glyphshow
|
||||
@@ -1539,128 +1583,44 @@ grestore
|
||||
stroke
|
||||
grestore
|
||||
} bind def
|
||||
57.6 167.653 o
|
||||
57.6 259.249 o
|
||||
grestore
|
||||
gsave
|
||||
25.162500 163.855954 translate
|
||||
25.162500 255.452379 translate
|
||||
0.000000 rotate
|
||||
0.000000 0.000000 m /three glyphshow
|
||||
6.362305 0.000000 m /five glyphshow
|
||||
12.724609 0.000000 m /zero glyphshow
|
||||
19.086914 0.000000 m /zero glyphshow
|
||||
grestore
|
||||
gsave
|
||||
/o {
|
||||
gsave
|
||||
newpath
|
||||
translate
|
||||
0.8 setlinewidth
|
||||
1 setlinejoin
|
||||
0 setlinecap
|
||||
0 0 m
|
||||
-3.5 0 l
|
||||
|
||||
gsave
|
||||
0.000 setgray
|
||||
fill
|
||||
grestore
|
||||
stroke
|
||||
grestore
|
||||
} bind def
|
||||
57.6 290.854 o
|
||||
grestore
|
||||
gsave
|
||||
25.162500 287.057130 translate
|
||||
0.000000 rotate
|
||||
0.000000 0.000000 m /four glyphshow
|
||||
6.362305 0.000000 m /zero glyphshow
|
||||
12.724609 0.000000 m /zero glyphshow
|
||||
19.086914 0.000000 m /zero glyphshow
|
||||
grestore
|
||||
gsave
|
||||
/o {
|
||||
gsave
|
||||
newpath
|
||||
translate
|
||||
0.8 setlinewidth
|
||||
1 setlinejoin
|
||||
0 setlinecap
|
||||
0 0 m
|
||||
-3.5 0 l
|
||||
|
||||
gsave
|
||||
0.000 setgray
|
||||
fill
|
||||
grestore
|
||||
stroke
|
||||
grestore
|
||||
} bind def
|
||||
57.6 200.062 o
|
||||
grestore
|
||||
gsave
|
||||
25.162500 196.265161 translate
|
||||
0.000000 rotate
|
||||
0.000000 0.000000 m /five glyphshow
|
||||
6.362305 0.000000 m /zero glyphshow
|
||||
12.724609 0.000000 m /zero glyphshow
|
||||
19.086914 0.000000 m /zero glyphshow
|
||||
grestore
|
||||
gsave
|
||||
/o {
|
||||
gsave
|
||||
newpath
|
||||
translate
|
||||
0.8 setlinewidth
|
||||
1 setlinejoin
|
||||
0 setlinecap
|
||||
0 0 m
|
||||
-3.5 0 l
|
||||
|
||||
gsave
|
||||
0.000 setgray
|
||||
fill
|
||||
grestore
|
||||
stroke
|
||||
grestore
|
||||
} bind def
|
||||
57.6 232.471 o
|
||||
grestore
|
||||
gsave
|
||||
25.162500 228.674368 translate
|
||||
0.000000 rotate
|
||||
0.000000 0.000000 m /six glyphshow
|
||||
6.362305 0.000000 m /zero glyphshow
|
||||
12.724609 0.000000 m /zero glyphshow
|
||||
19.086914 0.000000 m /zero glyphshow
|
||||
grestore
|
||||
gsave
|
||||
/o {
|
||||
gsave
|
||||
newpath
|
||||
translate
|
||||
0.8 setlinewidth
|
||||
1 setlinejoin
|
||||
0 setlinecap
|
||||
0 0 m
|
||||
-3.5 0 l
|
||||
|
||||
gsave
|
||||
0.000 setgray
|
||||
fill
|
||||
grestore
|
||||
stroke
|
||||
grestore
|
||||
} bind def
|
||||
57.6 264.88 o
|
||||
grestore
|
||||
gsave
|
||||
25.162500 261.083575 translate
|
||||
0.000000 rotate
|
||||
0.000000 0.000000 m /seven glyphshow
|
||||
6.362305 0.000000 m /zero glyphshow
|
||||
12.724609 0.000000 m /zero glyphshow
|
||||
19.086914 0.000000 m /zero glyphshow
|
||||
grestore
|
||||
gsave
|
||||
/o {
|
||||
gsave
|
||||
newpath
|
||||
translate
|
||||
0.8 setlinewidth
|
||||
1 setlinejoin
|
||||
0 setlinecap
|
||||
0 0 m
|
||||
-3.5 0 l
|
||||
|
||||
gsave
|
||||
0.000 setgray
|
||||
fill
|
||||
grestore
|
||||
stroke
|
||||
grestore
|
||||
} bind def
|
||||
57.6 297.29 o
|
||||
grestore
|
||||
gsave
|
||||
25.162500 293.492782 translate
|
||||
0.000000 rotate
|
||||
0.000000 0.000000 m /eight glyphshow
|
||||
6.362305 0.000000 m /zero glyphshow
|
||||
12.724609 0.000000 m /zero glyphshow
|
||||
19.086914 0.000000 m /zero glyphshow
|
||||
grestore
|
||||
/DejaVuSerif findfont
|
||||
10.000 scalefont
|
||||
setfont
|
||||
@@ -1698,56 +1658,56 @@ grestore
|
||||
1.000 0.000 0.000 setrgbcolor
|
||||
gsave
|
||||
357.1 266.1 57.6 38.02 clipbox
|
||||
73.832727 38.016983 m
|
||||
80.45833 38.018672 l
|
||||
87.083933 38.022964 l
|
||||
93.709536 38.033409 l
|
||||
100.335139 38.057747 l
|
||||
106.960742 38.112026 l
|
||||
113.586345 38.227866 l
|
||||
120.211948 38.464381 l
|
||||
126.837551 38.926216 l
|
||||
133.463154 39.788366 l
|
||||
140.088757 41.326354 l
|
||||
146.71436 43.946719 l
|
||||
153.339963 48.20783 l
|
||||
159.965566 54.815988 l
|
||||
166.591169 64.579036 l
|
||||
173.216772 78.302393 l
|
||||
179.842375 96.623201 l
|
||||
186.467978 119.797584 l
|
||||
193.093581 147.480169 l
|
||||
199.719184 178.556141 l
|
||||
206.344787 211.093541 l
|
||||
212.97039 242.468763 l
|
||||
219.595993 269.679271 l
|
||||
226.221596 289.803008 l
|
||||
232.847199 300.511211 l
|
||||
239.472801 300.511211 l
|
||||
246.098404 289.803008 l
|
||||
252.724007 269.679271 l
|
||||
259.34961 242.468763 l
|
||||
265.975213 211.093541 l
|
||||
272.600816 178.556141 l
|
||||
279.226419 147.480169 l
|
||||
285.852022 119.797584 l
|
||||
292.477625 96.623201 l
|
||||
299.103228 78.302393 l
|
||||
305.728831 64.579036 l
|
||||
312.354434 54.815988 l
|
||||
318.980037 48.20783 l
|
||||
325.60564 43.946719 l
|
||||
332.231243 41.326354 l
|
||||
338.856846 39.788366 l
|
||||
345.482449 38.926216 l
|
||||
352.108052 38.464381 l
|
||||
358.733655 38.227866 l
|
||||
365.359258 38.112026 l
|
||||
371.984861 38.057747 l
|
||||
378.610464 38.033409 l
|
||||
385.236067 38.022964 l
|
||||
391.86167 38.018672 l
|
||||
398.487273 38.016983 l
|
||||
73.832727 38.016959 m
|
||||
80.45833 38.018606 l
|
||||
87.083933 38.022791 l
|
||||
93.709536 38.032977 l
|
||||
100.335139 38.056711 l
|
||||
106.960742 38.109642 l
|
||||
113.586345 38.222607 l
|
||||
120.211948 38.453251 l
|
||||
126.837551 38.903622 l
|
||||
133.463154 39.744372 l
|
||||
140.088757 41.244184 l
|
||||
146.71436 43.799507 l
|
||||
153.339963 47.95485 l
|
||||
159.965566 54.398982 l
|
||||
166.591169 63.919692 l
|
||||
173.216772 77.30241 l
|
||||
179.842375 95.168462 l
|
||||
186.467978 117.767613 l
|
||||
193.093581 144.763065 l
|
||||
199.719184 175.067674 l
|
||||
206.344787 206.797436 l
|
||||
212.97039 237.393867 l
|
||||
219.595993 263.928959 l
|
||||
226.221596 283.553188 l
|
||||
232.847199 293.995593 l
|
||||
239.472801 293.995593 l
|
||||
246.098404 283.553188 l
|
||||
252.724007 263.928959 l
|
||||
259.34961 237.393867 l
|
||||
265.975213 206.797436 l
|
||||
272.600816 175.067674 l
|
||||
279.226419 144.763065 l
|
||||
285.852022 117.767613 l
|
||||
292.477625 95.168462 l
|
||||
299.103228 77.30241 l
|
||||
305.728831 63.919692 l
|
||||
312.354434 54.398982 l
|
||||
318.980037 47.95485 l
|
||||
325.60564 43.799507 l
|
||||
332.231243 41.244184 l
|
||||
338.856846 39.744372 l
|
||||
345.482449 38.903622 l
|
||||
352.108052 38.453251 l
|
||||
358.733655 38.222607 l
|
||||
365.359258 38.109642 l
|
||||
371.984861 38.056711 l
|
||||
378.610464 38.032977 l
|
||||
385.236067 38.022791 l
|
||||
391.86167 38.018606 l
|
||||
398.487273 38.016959 l
|
||||
stroke
|
||||
grestore
|
||||
0.800 setlinewidth
|
||||
|
||||
Binary file not shown.
|
Before Width: | Height: | Size: 28 KiB After Width: | Height: | Size: 27 KiB |
@@ -6,6 +6,116 @@ DATE: today
|
||||
!split
|
||||
===== Optimization, the central part of any Machine Learning algortithm =====
|
||||
|
||||
Almost every problem in machine learning and data science starts with
|
||||
a dataset $X$, a model $g(\theta)$, which is a function of the
|
||||
parameters $\theta$ and a cost function $C(X, g(\theta))$ that allows
|
||||
us to judge how well the model $g(\theta)$ explains the observations
|
||||
$X$. The model is fit by finding the values of $\theta$ that minimize
|
||||
the cost function. Ideally we would be able to solve for $\theta$
|
||||
analytically, however this is not possible in general and we must use
|
||||
some approximative/numerical method to compute the minimum.
|
||||
|
||||
The method of steepest descent
|
||||
The basic idea of gradient descent is that a function $F(\mathbf{x})$, $ \mathbf{x} \equiv (x_1,\cdots,x_n)$, decreases fastest if one goes from $\bf {x}$ in the direction of the negative gradient $-\nabla F(\mathbf{x})$.
|
||||
|
||||
It can be shown that if $$
|
||||
\mathbf{x}_{k+1} = \mathbf{x}_k - \gamma_k \nabla F(\mathbf{x}_k), \ \ \gamma_k > 0
|
||||
$$ for $\gamma_k$ small enough, then $F(\mathbf{x}_{k+1}) \leq F(\mathbf{x}_k)$. This means that for a sufficiently small $\gamma_k$ we are always moving towards smaller function values, i.e a minimum.
|
||||
|
||||
This observation is the basis of the method of steepest descent, which is also referred to as just gradient descent (GD). One starts with an initial guess $\mathbf{x}_0$ for a minimum of $F$ and compute new approximations according to
|
||||
|
||||
$$
|
||||
\mathbf{x}_{k+1} = \mathbf{x}_k - \gamma_k \nabla F(\mathbf{x}_k), \ \ k \geq 0.
|
||||
$$
|
||||
The parameter $\gamma_k$ is often referred to as the step length or the learning rate in the context of ML.
|
||||
|
||||
Ideally the sequence $\{ \mathbf{x}_k \}_{k=0}$ converges to a global minimum of the function $F$. In general we do not know if we are in a global or local minimum. In the special case when $F$ is a convex function, all local minima are also global minima, so in this case gradient descent can converge to the global solution. The advantage of this scheme is that it is conceptually simple and straightforward to implement. However the method in this form has some severe limitations:
|
||||
|
||||
In machine learing we are often faced with non-convex high dimensional cost functions with many local minimum. Since GD is deterministic we will get stuck in a local minimum, if the method converges, unless we have a very good intial guess. This also implies that the scheme is sensitive to the chosen initial condition.
|
||||
|
||||
Note that gradient is a function of $\mathbf{x} = (x_1,\cdots,x_n)$ which makes it expensive to compute numerically.
|
||||
|
||||
GD is sensitive to the choice of learning rate $\gamma_k$. This is due to the fact that we are only guaranteed that $F(\mathbf{x}_{k+1}) \leq F(\mathbf{x}_k)$ for sufficiently small $\gamma_k$. The problem is to determine an optimal learning rate. If the learning rate is chosen to small the method will take a long to converge and if it is to large we can experience erratic behavior.
|
||||
|
||||
Many of these shortcomings can be alleviated by introducing randomness. One such method is that of Stochastic Gradient Descent (SGD).
|
||||
|
||||
|
||||
Stochastic Gradient Descent
|
||||
Stochastic gradient descent (SGD) and variants thereof address some of the shortcomings of the Gradient descent method discussed above.
|
||||
|
||||
The underlying idea of SGD comes from the observation that the cost function, which we want to minimize, can almost always be written as a sum over $n$ datapoints $\{\mathbf{x}_i\}_{i=1}^n$, $$C(\mathbf{\theta}) = \sum_{i=1}^n c_i(\mathbf{x}_i, \mathbf{\theta}). $$ This in turn means that the gradient can be computed as a sum over $i$-gradients $$\nabla_\theta C(\mathbf{\theta}) = \sum_i^n \nabla_\theta c_i(\mathbf{x}_i, \mathbf{\theta}). $$
|
||||
|
||||
Now, stochasticity/randomness is introduced by only taking the gradient on a subset of the data called minibatches.
|
||||
If there are $n$ datapoints and the size of each minibatch is $M$, there will be $n/M$ minibatches. We denote these minibatches by $B_k$ where $k=1,\cdots,n/M$.
|
||||
|
||||
As an example, suppose we have $10$ datapoints $( \mathbf{x}_1, \cdots, \mathbf{x}_{10} )$ and we choose to have $M=5$ minibathces, then each minibatch contains two datapoints. In particular we have $B_1 = (\mathbf{x}_1,\mathbf{x}_2), \cdots, B_5 = (\mathbf{x}_9,\mathbf{x}_{10})$. Note that if you choose $M=1$ you have only a single batch with all datapoints and on the other extreme, you may choose $M=n$ resulting in a minibatch for each datapoint, i.e $B_k = \mathbf{x}_k$.
|
||||
|
||||
The idea is now to approximate the gradient by replacing the sum over all datapoints with a sum over the datapoints in one the minibatches picked at random in each gradient descent step $$\nabla_\theta C(\mathbf{\theta}) = \sum_{i=1}^n \nabla_\theta c_i(\mathbf{x}_i, \mathbf{\theta}) \rightarrow \sum_{i \in B_k}^n \nabla_\theta c_i(\mathbf{x}_i, \mathbf{\theta}). $$
|
||||
|
||||
Thus a gradient descent step now looks like $$ \theta_{j+1} = \theta_j - \gamma_j \sum_{i \in B_k}^n \nabla_\theta c_i(\mathbf{x}_i, \mathbf{\theta}) $$ where $k$ is picked at random with equal probability from $[1,n/M]$. An iteration over the number of minibathces (n/M) is commonly referred to as an epoch. Thus it is typical to choose a number of epochs and for each epoch iterate over the number of minibatches, as exemplified in the code below.
|
||||
|
||||
|
||||
!bc pycod
|
||||
import numpy as np
|
||||
|
||||
n = 100 #100 datapoints
|
||||
M = 5 #size of each minibatch
|
||||
m = int(n/M) #number of minibatches
|
||||
n_epochs = 10 #number of epochs
|
||||
|
||||
j = 0
|
||||
for epoch in range(1,n_epochs+1):
|
||||
for i in range(m):
|
||||
k = np.random.randint(m) #Pick the k-th minibatch at random
|
||||
#Compute the gradient using the data in minibatch Bk
|
||||
#Compute new suggestion for theta
|
||||
j += 1
|
||||
!ec
|
||||
|
||||
Taking the gradient only on a subset of the data has two important benefits. First, it introduces randomness which decreases the chance that our opmization scheme gets stuck in a local minima. Second, if the size of the minibatches are small relative to the number of datapoints ($M < n$), the computation of the gradient is much cheaper since we sum over the datapoints in the k-th minibatch and not all $n$ datapoints.
|
||||
|
||||
A natural question is when do we stop the search for a new minimum? One possibility is to compute the full gradient after a given number of epochs and check if the norm of the gradient is smaller than some threshold and stop if true. However, the condition that the gradient is zero is valid also for local minima, so this would only tell us that we are close to a local/global minimum. However, we could also evaluate the cost function at this point, store the result and continue the search. If the test kicks in at a later stage we can compare the values of the cost function and keep the $\theta$ that gave the lowest value.
|
||||
|
||||
Another approach is to let the step length $\gamma_j$ depend on the number of epochs in such a way that it becomes very small after a reasonable time such that we do not move at all.
|
||||
|
||||
As an example, let $e = 0,1,2,3,\cdots$ denote the current epoch and let $t_0, t_1 > 0$ be two fixed numbers. Furthermore, let $t = e \cdot m + i$ where $m$ is the number of minibatches and $i=0,\cdots,m-1$. Then the function $$\gamma_j(t; t_0, t_1) = \frac{t_0}{t+t_1} $$ goes to zero as the number of epochs gets large. I.e. we start with a step length $\gamma_j (0; t_0, t_1) = t_0/t_1$ which decays in "time" t.
|
||||
|
||||
In this way we can fix the number of epochs, compute $\theta$ and evaluate the cost function at the end. Repeating the computation will give a different result since the scheme is random by design. Then we pick the final $\theta$ that gives the lowest value of the cost function.
|
||||
|
||||
!bc pycod
|
||||
import numpy as np
|
||||
|
||||
def step_length(t,t0,t1):
|
||||
return t0/(t+t1)
|
||||
|
||||
n = 100 #100 datapoints
|
||||
M = 5 #size of each minibatch
|
||||
m = int(n/M) #number of minibatches
|
||||
n_epochs = 500 #number of epochs
|
||||
t0 = 1.0
|
||||
t1 = 10
|
||||
|
||||
gamma_j = t0/t1
|
||||
j = 0
|
||||
for epoch in range(1,n_epochs+1):
|
||||
for i in range(m):
|
||||
k = np.random.randint(m) #Pick the k-th minibatch at random
|
||||
#Compute the gradient using the data in minibatch Bk
|
||||
#Compute new suggestion for theta
|
||||
t = epoch*m+i
|
||||
gamma_j = step_length(t,t0,t1)
|
||||
j += 1
|
||||
|
||||
print("gamma_j after %d epochs: %g" % (n_epochs,gamma_j))
|
||||
!ec
|
||||
|
||||
|
||||
We will look at a class of methods for computing minima of functions known as gradient descent and its generalizations. This note is meant as a quick introduction/crash course on gradient methods and mathematical details are kept brief.
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
We will explain the general approach for minimizing a multi-variate
|
||||
function and set the terminology in this section.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user