small addition

2021-11-05 06:31:22 +01:00
parent c763c3b13a
commit 4565722caf
71 changed files with 6014 additions and 5836 deletions
@@ -146,7 +146,22 @@ plt.show()

 !ec

+!split
+===== Learning Rate and more =====

+When developing your own gradient descent code, it is useful to test
+it first on a standard ordinary least squares problem. Then the
+Hessian matrix is determined by the design matrix only, namely
+$\bm{H}\propto \bm{X}^T\bm{X}$.
+
+The optimal learning rate is determined by the inverse of the largest
+eigenvalue of $\bm{H}$. This can be used as a guideline for the
+learning rate guess.
+
+Keeping this fixed, can aid in studyng the dependence on say the mean
+square value for OLS as function of the number of batches and epochs
+in your stochastic gradient descent code. See for example the code
+examples for week 40 (right before the neural network material).