From c7c7ed20fb86c9bd3945dae42297acadc27d2eba Mon Sep 17 00:00:00 2001 From: Morten Hjorth-Jensen Date: Fri, 23 Sep 2022 07:26:39 +0200 Subject: [PATCH] added video on bootstrap --- doc/pub/week38/html/._week38-bs000.html | 28 +- doc/pub/week38/html/._week38-bs001.html | 30 +- doc/pub/week38/html/._week38-bs002.html | 28 +- doc/pub/week38/html/._week38-bs003.html | 28 +- doc/pub/week38/html/._week38-bs004.html | 28 +- doc/pub/week38/html/._week38-bs005.html | 28 +- doc/pub/week38/html/._week38-bs006.html | 28 +- doc/pub/week38/html/._week38-bs007.html | 28 +- doc/pub/week38/html/._week38-bs008.html | 28 +- doc/pub/week38/html/._week38-bs009.html | 28 +- doc/pub/week38/html/._week38-bs010.html | 28 +- doc/pub/week38/html/._week38-bs011.html | 28 +- doc/pub/week38/html/._week38-bs012.html | 28 +- doc/pub/week38/html/._week38-bs013.html | 28 +- doc/pub/week38/html/._week38-bs014.html | 28 +- doc/pub/week38/html/._week38-bs015.html | 28 +- doc/pub/week38/html/._week38-bs016.html | 28 +- doc/pub/week38/html/._week38-bs017.html | 28 +- doc/pub/week38/html/._week38-bs018.html | 28 +- doc/pub/week38/html/._week38-bs019.html | 28 +- doc/pub/week38/html/._week38-bs020.html | 28 +- doc/pub/week38/html/._week38-bs021.html | 28 +- doc/pub/week38/html/._week38-bs022.html | 28 +- doc/pub/week38/html/._week38-bs023.html | 28 +- doc/pub/week38/html/._week38-bs024.html | 28 +- doc/pub/week38/html/._week38-bs025.html | 28 +- doc/pub/week38/html/._week38-bs026.html | 28 +- doc/pub/week38/html/._week38-bs027.html | 28 +- doc/pub/week38/html/._week38-bs028.html | 28 +- doc/pub/week38/html/._week38-bs029.html | 28 +- doc/pub/week38/html/._week38-bs030.html | 42 +-- doc/pub/week38/html/._week38-bs031.html | 55 ++-- doc/pub/week38/html/._week38-bs032.html | 56 ++-- doc/pub/week38/html/._week38-bs033.html | 50 ++-- doc/pub/week38/html/._week38-bs034.html | 54 ++-- doc/pub/week38/html/._week38-bs035.html | 67 +++-- doc/pub/week38/html/._week38-bs036.html | 69 ++--- doc/pub/week38/html/._week38-bs037.html | 89 ++++-- doc/pub/week38/html/._week38-bs038.html | 90 ++---- doc/pub/week38/html/._week38-bs039.html | 49 ++-- doc/pub/week38/html/._week38-bs040.html | 54 ++-- doc/pub/week38/html/._week38-bs041.html | 57 ++-- doc/pub/week38/html/week38-bs.html | 28 +- doc/pub/week38/html/week38-reveal.html | 6 +- doc/pub/week38/html/week38-solarized.html | 6 +- doc/pub/week38/html/week38.html | 6 +- doc/pub/week38/ipynb/ipynb-week38-src.tar.gz | Bin 192 -> 193 bytes doc/pub/week38/ipynb/week38.ipynb | 286 +++++++++---------- doc/src/week38/week38.do.txt | 6 +- 49 files changed, 901 insertions(+), 1011 deletions(-) diff --git a/doc/pub/week38/html/._week38-bs000.html b/doc/pub/week38/html/._week38-bs000.html index 4888028fe..72a3d698d 100644 --- a/doc/pub/week38/html/._week38-bs000.html +++ b/doc/pub/week38/html/._week38-bs000.html @@ -112,7 +112,6 @@ doconce format html week38.do.txt --html_style=bootstrap --pygments_html_style=d 2, None, 'other-measures-in-classification-studies-cancer-data-again'), - ('Friday September 25', 2, None, 'friday-september-25'), ('Optimization, the central part of any Machine Learning ' 'algortithm', 2, @@ -210,19 +209,18 @@ MathJax.Hub.Config({
  • Using the correlation matrix
  • Discussing the correlation data
  • Other measures in classification studies: Cancer Data again
    • -
  • Friday September 25
    • -
  • Optimization, the central part of any Machine Learning algortithm
    • -
  • Revisiting our Logistic Regression case
    • -
  • The equations to solve
    • -
  • Solving using Newton-Raphson's method
    • -
  • Brief reminder on Newton-Raphson's method
    • -
  • The equations
    • -
  • Simple geometric interpretation
    • -
  • Extending to more than one variable
    • -
  • Steepest descent
    • -
  • More on Steepest descent
    • -
  • The ideal
    • -
  • The sensitiveness of the gradient descent
    • +
  • Optimization, the central part of any Machine Learning algortithm
    • +
  • Revisiting our Logistic Regression case
    • +
  • The equations to solve
    • +
  • Solving using Newton-Raphson's method
    • +
  • Brief reminder on Newton-Raphson's method
    • +
  • The equations
    • +
  • Simple geometric interpretation
    • +
  • Extending to more than one variable
    • +
  • Steepest descent
    • +
  • More on Steepest descent
    • +
  • The ideal
    • +
  • The sensitiveness of the gradient descent
    • @@ -277,7 +275,7 @@ MathJax.Hub.Config({
  • 9
  • 10
  • ...
    • -
  • 43
    • +
  • 42
  • »
    • diff --git a/doc/pub/week38/html/._week38-bs001.html b/doc/pub/week38/html/._week38-bs001.html index 9ff52a9a8..2ae737fac 100644 --- a/doc/pub/week38/html/._week38-bs001.html +++ b/doc/pub/week38/html/._week38-bs001.html @@ -112,7 +112,6 @@ doconce format html week38.do.txt --html_style=bootstrap --pygments_html_style=d 2, None, 'other-measures-in-classification-studies-cancer-data-again'), - ('Friday September 25', 2, None, 'friday-september-25'), ('Optimization, the central part of any Machine Learning ' 'algortithm', 2, @@ -210,19 +209,18 @@ MathJax.Hub.Config({
  • Using the correlation matrix
  • Discussing the correlation data
  • Other measures in classification studies: Cancer Data again
    • -
  • Friday September 25
    • -
  • Optimization, the central part of any Machine Learning algortithm
    • -
  • Revisiting our Logistic Regression case
    • -
  • The equations to solve
    • -
  • Solving using Newton-Raphson's method
    • -
  • Brief reminder on Newton-Raphson's method
    • -
  • The equations
    • -
  • Simple geometric interpretation
    • -
  • Extending to more than one variable
    • -
  • Steepest descent
    • -
  • More on Steepest descent
    • -
  • The ideal
    • -
  • The sensitiveness of the gradient descent
    • +
  • Optimization, the central part of any Machine Learning algortithm
    • +
  • Revisiting our Logistic Regression case
    • +
  • The equations to solve
    • +
  • Solving using Newton-Raphson's method
    • +
  • Brief reminder on Newton-Raphson's method
    • +
  • The equations
    • +
  • Simple geometric interpretation
    • +
  • Extending to more than one variable
    • +
  • Steepest descent
    • +
  • More on Steepest descent
    • +
  • The ideal
    • +
  • The sensitiveness of the gradient descent
    • @@ -238,7 +236,7 @@ MathJax.Hub.Config({ diff --git a/doc/pub/week38/html/._week38-bs002.html b/doc/pub/week38/html/._week38-bs002.html index e34156536..daeea76de 100644 --- a/doc/pub/week38/html/._week38-bs002.html +++ b/doc/pub/week38/html/._week38-bs002.html @@ -112,7 +112,6 @@ doconce format html week38.do.txt --html_style=bootstrap --pygments_html_style=d 2, None, 'other-measures-in-classification-studies-cancer-data-again'), - ('Friday September 25', 2, None, 'friday-september-25'), ('Optimization, the central part of any Machine Learning ' 'algortithm', 2, @@ -210,19 +209,18 @@ MathJax.Hub.Config({
  • Using the correlation matrix
  • Discussing the correlation data
  • Other measures in classification studies: Cancer Data again
    • -
  • Friday September 25
    • -
  • Optimization, the central part of any Machine Learning algortithm
    • -
  • Revisiting our Logistic Regression case
    • -
  • The equations to solve
    • -
  • Solving using Newton-Raphson's method
    • -
  • Brief reminder on Newton-Raphson's method
    • -
  • The equations
    • -
  • Simple geometric interpretation
    • -
  • Extending to more than one variable
    • -
  • Steepest descent
    • -
  • More on Steepest descent
    • -
  • The ideal
    • -
  • The sensitiveness of the gradient descent
    • +
  • Optimization, the central part of any Machine Learning algortithm
    • +
  • Revisiting our Logistic Regression case
    • +
  • The equations to solve
    • +
  • Solving using Newton-Raphson's method
    • +
  • Brief reminder on Newton-Raphson's method
    • +
  • The equations
    • +
  • Simple geometric interpretation
    • +
  • Extending to more than one variable
    • +
  • Steepest descent
    • +
  • More on Steepest descent
    • +
  • The ideal
    • +
  • The sensitiveness of the gradient descent
    • @@ -305,7 +303,7 @@ $$
  • 11
  • 12
  • ...
    • -
  • 43
    • +
  • 42
  • »
    • diff --git a/doc/pub/week38/html/._week38-bs003.html b/doc/pub/week38/html/._week38-bs003.html index ab905464b..3000e0895 100644 --- a/doc/pub/week38/html/._week38-bs003.html +++ b/doc/pub/week38/html/._week38-bs003.html @@ -112,7 +112,6 @@ doconce format html week38.do.txt --html_style=bootstrap --pygments_html_style=d 2, None, 'other-measures-in-classification-studies-cancer-data-again'), - ('Friday September 25', 2, None, 'friday-september-25'), ('Optimization, the central part of any Machine Learning ' 'algortithm', 2, @@ -210,19 +209,18 @@ MathJax.Hub.Config({
  • Using the correlation matrix
  • Discussing the correlation data
  • Other measures in classification studies: Cancer Data again
    • -
  • Friday September 25
    • -
  • Optimization, the central part of any Machine Learning algortithm
    • -
  • Revisiting our Logistic Regression case
    • -
  • The equations to solve
    • -
  • Solving using Newton-Raphson's method
    • -
  • Brief reminder on Newton-Raphson's method
    • -
  • The equations
    • -
  • Simple geometric interpretation
    • -
  • Extending to more than one variable
    • -
  • Steepest descent
    • -
  • More on Steepest descent
    • -
  • The ideal
    • -
  • The sensitiveness of the gradient descent
    • +
  • Optimization, the central part of any Machine Learning algortithm
    • +
  • Revisiting our Logistic Regression case
    • +
  • The equations to solve
    • +
  • Solving using Newton-Raphson's method
    • +
  • Brief reminder on Newton-Raphson's method
    • +
  • The equations
    • +
  • Simple geometric interpretation
    • +
  • Extending to more than one variable
    • +
  • Steepest descent
    • +
  • More on Steepest descent
    • +
  • The ideal
    • +
  • The sensitiveness of the gradient descent
    • @@ -270,7 +268,7 @@ cross-validation (LOOCV).
  • 12
  • 13
  • ...
    • -
  • 43
    • +
  • 42
  • »
    • diff --git a/doc/pub/week38/html/._week38-bs004.html b/doc/pub/week38/html/._week38-bs004.html index 401c59cd6..edb6d7631 100644 --- a/doc/pub/week38/html/._week38-bs004.html +++ b/doc/pub/week38/html/._week38-bs004.html @@ -112,7 +112,6 @@ doconce format html week38.do.txt --html_style=bootstrap --pygments_html_style=d 2, None, 'other-measures-in-classification-studies-cancer-data-again'), - ('Friday September 25', 2, None, 'friday-september-25'), ('Optimization, the central part of any Machine Learning ' 'algortithm', 2, @@ -210,19 +209,18 @@ MathJax.Hub.Config({
  • Using the correlation matrix
  • Discussing the correlation data
  • Other measures in classification studies: Cancer Data again
    • -
  • Friday September 25
    • -
  • Optimization, the central part of any Machine Learning algortithm
    • -
  • Revisiting our Logistic Regression case
    • -
  • The equations to solve
    • -
  • Solving using Newton-Raphson's method
    • -
  • Brief reminder on Newton-Raphson's method
    • -
  • The equations
    • -
  • Simple geometric interpretation
    • -
  • Extending to more than one variable
    • -
  • Steepest descent
    • -
  • More on Steepest descent
    • -
  • The ideal
    • -
  • The sensitiveness of the gradient descent
    • +
  • Optimization, the central part of any Machine Learning algortithm
    • +
  • Revisiting our Logistic Regression case
    • +
  • The equations to solve
    • +
  • Solving using Newton-Raphson's method
    • +
  • Brief reminder on Newton-Raphson's method
    • +
  • The equations
    • +
  • Simple geometric interpretation
    • +
  • Extending to more than one variable
    • +
  • Steepest descent
    • +
  • More on Steepest descent
    • +
  • The ideal
    • +
  • The sensitiveness of the gradient descent
    • @@ -274,7 +272,7 @@ $$
  • 13
  • 14
  • ...
    • -
  • 43
    • +
  • 42
  • »
    • diff --git a/doc/pub/week38/html/._week38-bs005.html b/doc/pub/week38/html/._week38-bs005.html index 0ef58d866..7a5e66296 100644 --- a/doc/pub/week38/html/._week38-bs005.html +++ b/doc/pub/week38/html/._week38-bs005.html @@ -112,7 +112,6 @@ doconce format html week38.do.txt --html_style=bootstrap --pygments_html_style=d 2, None, 'other-measures-in-classification-studies-cancer-data-again'), - ('Friday September 25', 2, None, 'friday-september-25'), ('Optimization, the central part of any Machine Learning ' 'algortithm', 2, @@ -210,19 +209,18 @@ MathJax.Hub.Config({
  • Using the correlation matrix
  • Discussing the correlation data
  • Other measures in classification studies: Cancer Data again
    • -
  • Friday September 25
    • -
  • Optimization, the central part of any Machine Learning algortithm
    • -
  • Revisiting our Logistic Regression case
    • -
  • The equations to solve
    • -
  • Solving using Newton-Raphson's method
    • -
  • Brief reminder on Newton-Raphson's method
    • -
  • The equations
    • -
  • Simple geometric interpretation
    • -
  • Extending to more than one variable
    • -
  • Steepest descent
    • -
  • More on Steepest descent
    • -
  • The ideal
    • -
  • The sensitiveness of the gradient descent
    • +
  • Optimization, the central part of any Machine Learning algortithm
    • +
  • Revisiting our Logistic Regression case
    • +
  • The equations to solve
    • +
  • Solving using Newton-Raphson's method
    • +
  • Brief reminder on Newton-Raphson's method
    • +
  • The equations
    • +
  • Simple geometric interpretation
    • +
  • Extending to more than one variable
    • +
  • Steepest descent
    • +
  • More on Steepest descent
    • +
  • The ideal
    • +
  • The sensitiveness of the gradient descent
    • @@ -270,7 +268,7 @@ MathJax.Hub.Config({
  • 14
  • 15
  • ...
    • -
  • 43
    • +
  • 42
  • »
    • diff --git a/doc/pub/week38/html/._week38-bs006.html b/doc/pub/week38/html/._week38-bs006.html index 1c66a6600..0da959d00 100644 --- a/doc/pub/week38/html/._week38-bs006.html +++ b/doc/pub/week38/html/._week38-bs006.html @@ -112,7 +112,6 @@ doconce format html week38.do.txt --html_style=bootstrap --pygments_html_style=d 2, None, 'other-measures-in-classification-studies-cancer-data-again'), - ('Friday September 25', 2, None, 'friday-september-25'), ('Optimization, the central part of any Machine Learning ' 'algortithm', 2, @@ -210,19 +209,18 @@ MathJax.Hub.Config({
  • Using the correlation matrix
  • Discussing the correlation data
  • Other measures in classification studies: Cancer Data again
    • -
  • Friday September 25
    • -
  • Optimization, the central part of any Machine Learning algortithm
    • -
  • Revisiting our Logistic Regression case
    • -
  • The equations to solve
    • -
  • Solving using Newton-Raphson's method
    • -
  • Brief reminder on Newton-Raphson's method
    • -
  • The equations
    • -
  • Simple geometric interpretation
    • -
  • Extending to more than one variable
    • -
  • Steepest descent
    • -
  • More on Steepest descent
    • -
  • The ideal
    • -
  • The sensitiveness of the gradient descent
    • +
  • Optimization, the central part of any Machine Learning algortithm
    • +
  • Revisiting our Logistic Regression case
    • +
  • The equations to solve
    • +
  • Solving using Newton-Raphson's method
    • +
  • Brief reminder on Newton-Raphson's method
    • +
  • The equations
    • +
  • Simple geometric interpretation
    • +
  • Extending to more than one variable
    • +
  • Steepest descent
    • +
  • More on Steepest descent
    • +
  • The ideal
    • +
  • The sensitiveness of the gradient descent
    • @@ -370,7 +368,7 @@ plt.show()
  • 15
  • 16
  • ...
    • -
  • 43
    • +
  • 42
  • »
    • diff --git a/doc/pub/week38/html/._week38-bs007.html b/doc/pub/week38/html/._week38-bs007.html index 1216d69a7..bd4fce807 100644 --- a/doc/pub/week38/html/._week38-bs007.html +++ b/doc/pub/week38/html/._week38-bs007.html @@ -112,7 +112,6 @@ doconce format html week38.do.txt --html_style=bootstrap --pygments_html_style=d 2, None, 'other-measures-in-classification-studies-cancer-data-again'), - ('Friday September 25', 2, None, 'friday-september-25'), ('Optimization, the central part of any Machine Learning ' 'algortithm', 2, @@ -210,19 +209,18 @@ MathJax.Hub.Config({
  • Using the correlation matrix
  • Discussing the correlation data
  • Other measures in classification studies: Cancer Data again
    • -
  • Friday September 25
    • -
  • Optimization, the central part of any Machine Learning algortithm
    • -
  • Revisiting our Logistic Regression case
    • -
  • The equations to solve
    • -
  • Solving using Newton-Raphson's method
    • -
  • Brief reminder on Newton-Raphson's method
    • -
  • The equations
    • -
  • Simple geometric interpretation
    • -
  • Extending to more than one variable
    • -
  • Steepest descent
    • -
  • More on Steepest descent
    • -
  • The ideal
    • -
  • The sensitiveness of the gradient descent
    • +
  • Optimization, the central part of any Machine Learning algortithm
    • +
  • Revisiting our Logistic Regression case
    • +
  • The equations to solve
    • +
  • Solving using Newton-Raphson's method
    • +
  • Brief reminder on Newton-Raphson's method
    • +
  • The equations
    • +
  • Simple geometric interpretation
    • +
  • Extending to more than one variable
    • +
  • Steepest descent
    • +
  • More on Steepest descent
    • +
  • The ideal
    • +
  • The sensitiveness of the gradient descent
    • @@ -271,7 +269,7 @@ simple recipe for fitting our data.
  • 16
  • 17
  • ...
    • -
  • 43
    • +
  • 42
  • »
    • diff --git a/doc/pub/week38/html/._week38-bs008.html b/doc/pub/week38/html/._week38-bs008.html index 302eefe13..e623d4cb9 100644 --- a/doc/pub/week38/html/._week38-bs008.html +++ b/doc/pub/week38/html/._week38-bs008.html @@ -112,7 +112,6 @@ doconce format html week38.do.txt --html_style=bootstrap --pygments_html_style=d 2, None, 'other-measures-in-classification-studies-cancer-data-again'), - ('Friday September 25', 2, None, 'friday-september-25'), ('Optimization, the central part of any Machine Learning ' 'algortithm', 2, @@ -210,19 +209,18 @@ MathJax.Hub.Config({
  • Using the correlation matrix
  • Discussing the correlation data
  • Other measures in classification studies: Cancer Data again
    • -
  • Friday September 25
    • -
  • Optimization, the central part of any Machine Learning algortithm
    • -
  • Revisiting our Logistic Regression case
    • -
  • The equations to solve
    • -
  • Solving using Newton-Raphson's method
    • -
  • Brief reminder on Newton-Raphson's method
    • -
  • The equations
    • -
  • Simple geometric interpretation
    • -
  • Extending to more than one variable
    • -
  • Steepest descent
    • -
  • More on Steepest descent
    • -
  • The ideal
    • -
  • The sensitiveness of the gradient descent
    • +
  • Optimization, the central part of any Machine Learning algortithm
    • +
  • Revisiting our Logistic Regression case
    • +
  • The equations to solve
    • +
  • Solving using Newton-Raphson's method
    • +
  • Brief reminder on Newton-Raphson's method
    • +
  • The equations
    • +
  • Simple geometric interpretation
    • +
  • Extending to more than one variable
    • +
  • Steepest descent
    • +
  • More on Steepest descent
    • +
  • The ideal
    • +
  • The sensitiveness of the gradient descent
    • @@ -277,7 +275,7 @@ failure etc.
  • 17
  • 18
  • ...
    • -
  • 43
    • +
  • 42
  • »
    • diff --git a/doc/pub/week38/html/._week38-bs009.html b/doc/pub/week38/html/._week38-bs009.html index 583433d9a..8923bf7e8 100644 --- a/doc/pub/week38/html/._week38-bs009.html +++ b/doc/pub/week38/html/._week38-bs009.html @@ -112,7 +112,6 @@ doconce format html week38.do.txt --html_style=bootstrap --pygments_html_style=d 2, None, 'other-measures-in-classification-studies-cancer-data-again'), - ('Friday September 25', 2, None, 'friday-september-25'), ('Optimization, the central part of any Machine Learning ' 'algortithm', 2, @@ -210,19 +209,18 @@ MathJax.Hub.Config({
  • Using the correlation matrix
  • Discussing the correlation data
  • Other measures in classification studies: Cancer Data again
    • -
  • Friday September 25
    • -
  • Optimization, the central part of any Machine Learning algortithm
    • -
  • Revisiting our Logistic Regression case
    • -
  • The equations to solve
    • -
  • Solving using Newton-Raphson's method
    • -
  • Brief reminder on Newton-Raphson's method
    • -
  • The equations
    • -
  • Simple geometric interpretation
    • -
  • Extending to more than one variable
    • -
  • Steepest descent
    • -
  • More on Steepest descent
    • -
  • The ideal
    • -
  • The sensitiveness of the gradient descent
    • +
  • Optimization, the central part of any Machine Learning algortithm
    • +
  • Revisiting our Logistic Regression case
    • +
  • The equations to solve
    • +
  • Solving using Newton-Raphson's method
    • +
  • Brief reminder on Newton-Raphson's method
    • +
  • The equations
    • +
  • Simple geometric interpretation
    • +
  • Extending to more than one variable
    • +
  • Steepest descent
    • +
  • More on Steepest descent
    • +
  • The ideal
    • +
  • The sensitiveness of the gradient descent
    • @@ -276,7 +274,7 @@ models, as we will see later.
  • 18
  • 19
  • ...
    • -
  • 43
    • +
  • 42
  • »
    • diff --git a/doc/pub/week38/html/._week38-bs010.html b/doc/pub/week38/html/._week38-bs010.html index 4d3fb3221..fb667c912 100644 --- a/doc/pub/week38/html/._week38-bs010.html +++ b/doc/pub/week38/html/._week38-bs010.html @@ -112,7 +112,6 @@ doconce format html week38.do.txt --html_style=bootstrap --pygments_html_style=d 2, None, 'other-measures-in-classification-studies-cancer-data-again'), - ('Friday September 25', 2, None, 'friday-september-25'), ('Optimization, the central part of any Machine Learning ' 'algortithm', 2, @@ -210,19 +209,18 @@ MathJax.Hub.Config({
  • Using the correlation matrix
  • Discussing the correlation data
  • Other measures in classification studies: Cancer Data again
    • -
  • Friday September 25
    • -
  • Optimization, the central part of any Machine Learning algortithm
    • -
  • Revisiting our Logistic Regression case
    • -
  • The equations to solve
    • -
  • Solving using Newton-Raphson's method
    • -
  • Brief reminder on Newton-Raphson's method
    • -
  • The equations
    • -
  • Simple geometric interpretation
    • -
  • Extending to more than one variable
    • -
  • Steepest descent
    • -
  • More on Steepest descent
    • -
  • The ideal
    • -
  • The sensitiveness of the gradient descent
    • +
  • Optimization, the central part of any Machine Learning algortithm
    • +
  • Revisiting our Logistic Regression case
    • +
  • The equations to solve
    • +
  • Solving using Newton-Raphson's method
    • +
  • Brief reminder on Newton-Raphson's method
    • +
  • The equations
    • +
  • Simple geometric interpretation
    • +
  • Extending to more than one variable
    • +
  • Steepest descent
    • +
  • More on Steepest descent
    • +
  • The ideal
    • +
  • The sensitiveness of the gradient descent
    • @@ -284,7 +282,7 @@ $$
  • 19
  • 20
  • ...
    • -
  • 43
    • +
  • 42
  • »
    • diff --git a/doc/pub/week38/html/._week38-bs011.html b/doc/pub/week38/html/._week38-bs011.html index 956f87955..0c9ab2836 100644 --- a/doc/pub/week38/html/._week38-bs011.html +++ b/doc/pub/week38/html/._week38-bs011.html @@ -112,7 +112,6 @@ doconce format html week38.do.txt --html_style=bootstrap --pygments_html_style=d 2, None, 'other-measures-in-classification-studies-cancer-data-again'), - ('Friday September 25', 2, None, 'friday-september-25'), ('Optimization, the central part of any Machine Learning ' 'algortithm', 2, @@ -210,19 +209,18 @@ MathJax.Hub.Config({
  • Using the correlation matrix
  • Discussing the correlation data
  • Other measures in classification studies: Cancer Data again
    • -
  • Friday September 25
    • -
  • Optimization, the central part of any Machine Learning algortithm
    • -
  • Revisiting our Logistic Regression case
    • -
  • The equations to solve
    • -
  • Solving using Newton-Raphson's method
    • -
  • Brief reminder on Newton-Raphson's method
    • -
  • The equations
    • -
  • Simple geometric interpretation
    • -
  • Extending to more than one variable
    • -
  • Steepest descent
    • -
  • More on Steepest descent
    • -
  • The ideal
    • -
  • The sensitiveness of the gradient descent
    • +
  • Optimization, the central part of any Machine Learning algortithm
    • +
  • Revisiting our Logistic Regression case
    • +
  • The equations to solve
    • +
  • Solving using Newton-Raphson's method
    • +
  • Brief reminder on Newton-Raphson's method
    • +
  • The equations
    • +
  • Simple geometric interpretation
    • +
  • Extending to more than one variable
    • +
  • Steepest descent
    • +
  • More on Steepest descent
    • +
  • The ideal
    • +
  • The sensitiveness of the gradient descent
    • @@ -281,7 +279,7 @@ $$
  • 20
  • 21
  • ...
    • -
  • 43
    • +
  • 42
  • »
    • diff --git a/doc/pub/week38/html/._week38-bs012.html b/doc/pub/week38/html/._week38-bs012.html index 1585f0664..5519c9c71 100644 --- a/doc/pub/week38/html/._week38-bs012.html +++ b/doc/pub/week38/html/._week38-bs012.html @@ -112,7 +112,6 @@ doconce format html week38.do.txt --html_style=bootstrap --pygments_html_style=d 2, None, 'other-measures-in-classification-studies-cancer-data-again'), - ('Friday September 25', 2, None, 'friday-september-25'), ('Optimization, the central part of any Machine Learning ' 'algortithm', 2, @@ -210,19 +209,18 @@ MathJax.Hub.Config({
  • Using the correlation matrix
  • Discussing the correlation data
  • Other measures in classification studies: Cancer Data again
    • -
  • Friday September 25
    • -
  • Optimization, the central part of any Machine Learning algortithm
    • -
  • Revisiting our Logistic Regression case
    • -
  • The equations to solve
    • -
  • Solving using Newton-Raphson's method
    • -
  • Brief reminder on Newton-Raphson's method
    • -
  • The equations
    • -
  • Simple geometric interpretation
    • -
  • Extending to more than one variable
    • -
  • Steepest descent
    • -
  • More on Steepest descent
    • -
  • The ideal
    • -
  • The sensitiveness of the gradient descent
    • +
  • Optimization, the central part of any Machine Learning algortithm
    • +
  • Revisiting our Logistic Regression case
    • +
  • The equations to solve
    • +
  • Solving using Newton-Raphson's method
    • +
  • Brief reminder on Newton-Raphson's method
    • +
  • The equations
    • +
  • Simple geometric interpretation
    • +
  • Extending to more than one variable
    • +
  • Steepest descent
    • +
  • More on Steepest descent
    • +
  • The ideal
    • +
  • The sensitiveness of the gradient descent
    • @@ -281,7 +279,7 @@ the probability of a given category. This leads us to the logistic function.
  • 21
  • 22
  • ...
    • -
  • 43
    • +
  • 42
  • »
    • diff --git a/doc/pub/week38/html/._week38-bs013.html b/doc/pub/week38/html/._week38-bs013.html index b7387c3ba..c6009c8ff 100644 --- a/doc/pub/week38/html/._week38-bs013.html +++ b/doc/pub/week38/html/._week38-bs013.html @@ -112,7 +112,6 @@ doconce format html week38.do.txt --html_style=bootstrap --pygments_html_style=d 2, None, 'other-measures-in-classification-studies-cancer-data-again'), - ('Friday September 25', 2, None, 'friday-september-25'), ('Optimization, the central part of any Machine Learning ' 'algortithm', 2, @@ -210,19 +209,18 @@ MathJax.Hub.Config({
  • Using the correlation matrix
  • Discussing the correlation data
  • Other measures in classification studies: Cancer Data again
    • -
  • Friday September 25
    • -
  • Optimization, the central part of any Machine Learning algortithm
    • -
  • Revisiting our Logistic Regression case
    • -
  • The equations to solve
    • -
  • Solving using Newton-Raphson's method
    • -
  • Brief reminder on Newton-Raphson's method
    • -
  • The equations
    • -
  • Simple geometric interpretation
    • -
  • Extending to more than one variable
    • -
  • Steepest descent
    • -
  • More on Steepest descent
    • -
  • The ideal
    • -
  • The sensitiveness of the gradient descent
    • +
  • Optimization, the central part of any Machine Learning algortithm
    • +
  • Revisiting our Logistic Regression case
    • +
  • The equations to solve
    • +
  • Solving using Newton-Raphson's method
    • +
  • Brief reminder on Newton-Raphson's method
    • +
  • The equations
    • +
  • Simple geometric interpretation
    • +
  • Extending to more than one variable
    • +
  • Steepest descent
    • +
  • More on Steepest descent
    • +
  • The ideal
    • +
  • The sensitiveness of the gradient descent
    • @@ -340,7 +338,7 @@ plt.show()
  • 22
  • 23
  • ...
    • -
  • 43
    • +
  • 42
  • »
    • diff --git a/doc/pub/week38/html/._week38-bs014.html b/doc/pub/week38/html/._week38-bs014.html index 464a44b98..3fcd3c8e9 100644 --- a/doc/pub/week38/html/._week38-bs014.html +++ b/doc/pub/week38/html/._week38-bs014.html @@ -112,7 +112,6 @@ doconce format html week38.do.txt --html_style=bootstrap --pygments_html_style=d 2, None, 'other-measures-in-classification-studies-cancer-data-again'), - ('Friday September 25', 2, None, 'friday-september-25'), ('Optimization, the central part of any Machine Learning ' 'algortithm', 2, @@ -210,19 +209,18 @@ MathJax.Hub.Config({
  • Using the correlation matrix
  • Discussing the correlation data
  • Other measures in classification studies: Cancer Data again
    • -
  • Friday September 25
    • -
  • Optimization, the central part of any Machine Learning algortithm
    • -
  • Revisiting our Logistic Regression case
    • -
  • The equations to solve
    • -
  • Solving using Newton-Raphson's method
    • -
  • Brief reminder on Newton-Raphson's method
    • -
  • The equations
    • -
  • Simple geometric interpretation
    • -
  • Extending to more than one variable
    • -
  • Steepest descent
    • -
  • More on Steepest descent
    • -
  • The ideal
    • -
  • The sensitiveness of the gradient descent
    • +
  • Optimization, the central part of any Machine Learning algortithm
    • +
  • Revisiting our Logistic Regression case
    • +
  • The equations to solve
    • +
  • Solving using Newton-Raphson's method
    • +
  • Brief reminder on Newton-Raphson's method
    • +
  • The equations
    • +
  • Simple geometric interpretation
    • +
  • Extending to more than one variable
    • +
  • Steepest descent
    • +
  • More on Steepest descent
    • +
  • The ideal
    • +
  • The sensitiveness of the gradient descent
    • @@ -312,7 +310,7 @@ representing the probability for finding a value of \( y_i \) with a given
  • 23
  • 24
  • ...
    • -
  • 43
    • +
  • 42
  • »
    • diff --git a/doc/pub/week38/html/._week38-bs015.html b/doc/pub/week38/html/._week38-bs015.html index a981b7ebd..55ec26504 100644 --- a/doc/pub/week38/html/._week38-bs015.html +++ b/doc/pub/week38/html/._week38-bs015.html @@ -112,7 +112,6 @@ doconce format html week38.do.txt --html_style=bootstrap --pygments_html_style=d 2, None, 'other-measures-in-classification-studies-cancer-data-again'), - ('Friday September 25', 2, None, 'friday-september-25'), ('Optimization, the central part of any Machine Learning ' 'algortithm', 2, @@ -210,19 +209,18 @@ MathJax.Hub.Config({
  • Using the correlation matrix
  • Discussing the correlation data
  • Other measures in classification studies: Cancer Data again
    • -
  • Friday September 25
    • -
  • Optimization, the central part of any Machine Learning algortithm
    • -
  • Revisiting our Logistic Regression case
    • -
  • The equations to solve
    • -
  • Solving using Newton-Raphson's method
    • -
  • Brief reminder on Newton-Raphson's method
    • -
  • The equations
    • -
  • Simple geometric interpretation
    • -
  • Extending to more than one variable
    • -
  • Steepest descent
    • -
  • More on Steepest descent
    • -
  • The ideal
    • -
  • The sensitiveness of the gradient descent
    • +
  • Optimization, the central part of any Machine Learning algortithm
    • +
  • Revisiting our Logistic Regression case
    • +
  • The equations to solve
    • +
  • Solving using Newton-Raphson's method
    • +
  • Brief reminder on Newton-Raphson's method
    • +
  • The equations
    • +
  • Simple geometric interpretation
    • +
  • Extending to more than one variable
    • +
  • Steepest descent
    • +
  • More on Steepest descent
    • +
  • The ideal
    • +
  • The sensitiveness of the gradient descent
    • @@ -279,7 +277,7 @@ $$
  • 24
  • 25
  • ...
    • -
  • 43
    • +
  • 42
  • »
    • diff --git a/doc/pub/week38/html/._week38-bs016.html b/doc/pub/week38/html/._week38-bs016.html index f4204eb4f..973dc12fe 100644 --- a/doc/pub/week38/html/._week38-bs016.html +++ b/doc/pub/week38/html/._week38-bs016.html @@ -112,7 +112,6 @@ doconce format html week38.do.txt --html_style=bootstrap --pygments_html_style=d 2, None, 'other-measures-in-classification-studies-cancer-data-again'), - ('Friday September 25', 2, None, 'friday-september-25'), ('Optimization, the central part of any Machine Learning ' 'algortithm', 2, @@ -210,19 +209,18 @@ MathJax.Hub.Config({
  • Using the correlation matrix
  • Discussing the correlation data
  • Other measures in classification studies: Cancer Data again
    • -
  • Friday September 25
    • -
  • Optimization, the central part of any Machine Learning algortithm
    • -
  • Revisiting our Logistic Regression case
    • -
  • The equations to solve
    • -
  • Solving using Newton-Raphson's method
    • -
  • Brief reminder on Newton-Raphson's method
    • -
  • The equations
    • -
  • Simple geometric interpretation
    • -
  • Extending to more than one variable
    • -
  • Steepest descent
    • -
  • More on Steepest descent
    • -
  • The ideal
    • -
  • The sensitiveness of the gradient descent
    • +
  • Optimization, the central part of any Machine Learning algortithm
    • +
  • Revisiting our Logistic Regression case
    • +
  • The equations to solve
    • +
  • Solving using Newton-Raphson's method
    • +
  • Brief reminder on Newton-Raphson's method
    • +
  • The equations
    • +
  • Simple geometric interpretation
    • +
  • Extending to more than one variable
    • +
  • Steepest descent
    • +
  • More on Steepest descent
    • +
  • The ideal
    • +
  • The sensitiveness of the gradient descent
    • @@ -340,7 +338,7 @@ plt.show()
  • 25
  • 26
  • ...
    • -
  • 43
    • +
  • 42
  • »
    • diff --git a/doc/pub/week38/html/._week38-bs017.html b/doc/pub/week38/html/._week38-bs017.html index 329d11a09..523d66dec 100644 --- a/doc/pub/week38/html/._week38-bs017.html +++ b/doc/pub/week38/html/._week38-bs017.html @@ -112,7 +112,6 @@ doconce format html week38.do.txt --html_style=bootstrap --pygments_html_style=d 2, None, 'other-measures-in-classification-studies-cancer-data-again'), - ('Friday September 25', 2, None, 'friday-september-25'), ('Optimization, the central part of any Machine Learning ' 'algortithm', 2, @@ -210,19 +209,18 @@ MathJax.Hub.Config({
  • Using the correlation matrix
  • Discussing the correlation data
  • Other measures in classification studies: Cancer Data again
    • -
  • Friday September 25
    • -
  • Optimization, the central part of any Machine Learning algortithm
    • -
  • Revisiting our Logistic Regression case
    • -
  • The equations to solve
    • -
  • Solving using Newton-Raphson's method
    • -
  • Brief reminder on Newton-Raphson's method
    • -
  • The equations
    • -
  • Simple geometric interpretation
    • -
  • Extending to more than one variable
    • -
  • Steepest descent
    • -
  • More on Steepest descent
    • -
  • The ideal
    • -
  • The sensitiveness of the gradient descent
    • +
  • Optimization, the central part of any Machine Learning algortithm
    • +
  • Revisiting our Logistic Regression case
    • +
  • The equations to solve
    • +
  • Solving using Newton-Raphson's method
    • +
  • Brief reminder on Newton-Raphson's method
    • +
  • The equations
    • +
  • Simple geometric interpretation
    • +
  • Extending to more than one variable
    • +
  • Steepest descent
    • +
  • More on Steepest descent
    • +
  • The ideal
    • +
  • The sensitiveness of the gradient descent
    • @@ -277,7 +275,7 @@ $$
  • 26
  • 27
  • ...
    • -
  • 43
    • +
  • 42
  • »
    • diff --git a/doc/pub/week38/html/._week38-bs018.html b/doc/pub/week38/html/._week38-bs018.html index 5909abf9c..9e63c9a4f 100644 --- a/doc/pub/week38/html/._week38-bs018.html +++ b/doc/pub/week38/html/._week38-bs018.html @@ -112,7 +112,6 @@ doconce format html week38.do.txt --html_style=bootstrap --pygments_html_style=d 2, None, 'other-measures-in-classification-studies-cancer-data-again'), - ('Friday September 25', 2, None, 'friday-september-25'), ('Optimization, the central part of any Machine Learning ' 'algortithm', 2, @@ -210,19 +209,18 @@ MathJax.Hub.Config({
  • Using the correlation matrix
  • Discussing the correlation data
  • Other measures in classification studies: Cancer Data again
    • -
  • Friday September 25
    • -
  • Optimization, the central part of any Machine Learning algortithm
    • -
  • Revisiting our Logistic Regression case
    • -
  • The equations to solve
    • -
  • Solving using Newton-Raphson's method
    • -
  • Brief reminder on Newton-Raphson's method
    • -
  • The equations
    • -
  • Simple geometric interpretation
    • -
  • Extending to more than one variable
    • -
  • Steepest descent
    • -
  • More on Steepest descent
    • -
  • The ideal
    • -
  • The sensitiveness of the gradient descent
    • +
  • Optimization, the central part of any Machine Learning algortithm
    • +
  • Revisiting our Logistic Regression case
    • +
  • The equations to solve
    • +
  • Solving using Newton-Raphson's method
    • +
  • Brief reminder on Newton-Raphson's method
    • +
  • The equations
    • +
  • Simple geometric interpretation
    • +
  • Extending to more than one variable
    • +
  • Steepest descent
    • +
  • More on Steepest descent
    • +
  • The ideal
    • +
  • The sensitiveness of the gradient descent
    • @@ -280,7 +278,7 @@ $$
  • 27
  • 28
  • ...
    • -
  • 43
    • +
  • 42
  • »
    • diff --git a/doc/pub/week38/html/._week38-bs019.html b/doc/pub/week38/html/._week38-bs019.html index c8234e504..ab4a754e9 100644 --- a/doc/pub/week38/html/._week38-bs019.html +++ b/doc/pub/week38/html/._week38-bs019.html @@ -112,7 +112,6 @@ doconce format html week38.do.txt --html_style=bootstrap --pygments_html_style=d 2, None, 'other-measures-in-classification-studies-cancer-data-again'), - ('Friday September 25', 2, None, 'friday-september-25'), ('Optimization, the central part of any Machine Learning ' 'algortithm', 2, @@ -210,19 +209,18 @@ MathJax.Hub.Config({
  • Using the correlation matrix
  • Discussing the correlation data
  • Other measures in classification studies: Cancer Data again
    • -
  • Friday September 25
    • -
  • Optimization, the central part of any Machine Learning algortithm
    • -
  • Revisiting our Logistic Regression case
    • -
  • The equations to solve
    • -
  • Solving using Newton-Raphson's method
    • -
  • Brief reminder on Newton-Raphson's method
    • -
  • The equations
    • -
  • Simple geometric interpretation
    • -
  • Extending to more than one variable
    • -
  • Steepest descent
    • -
  • More on Steepest descent
    • -
  • The ideal
    • -
  • The sensitiveness of the gradient descent
    • +
  • Optimization, the central part of any Machine Learning algortithm
    • +
  • Revisiting our Logistic Regression case
    • +
  • The equations to solve
    • +
  • Solving using Newton-Raphson's method
    • +
  • Brief reminder on Newton-Raphson's method
    • +
  • The equations
    • +
  • Simple geometric interpretation
    • +
  • Extending to more than one variable
    • +
  • Steepest descent
    • +
  • More on Steepest descent
    • +
  • The ideal
    • +
  • The sensitiveness of the gradient descent
    • @@ -277,7 +275,7 @@ in practice we often supplement the cross-entropy with additional regularization
  • 28
  • 29
  • ...
    • -
  • 43
    • +
  • 42
  • »
    • diff --git a/doc/pub/week38/html/._week38-bs020.html b/doc/pub/week38/html/._week38-bs020.html index a8eee0a96..c8ca0f1e6 100644 --- a/doc/pub/week38/html/._week38-bs020.html +++ b/doc/pub/week38/html/._week38-bs020.html @@ -112,7 +112,6 @@ doconce format html week38.do.txt --html_style=bootstrap --pygments_html_style=d 2, None, 'other-measures-in-classification-studies-cancer-data-again'), - ('Friday September 25', 2, None, 'friday-september-25'), ('Optimization, the central part of any Machine Learning ' 'algortithm', 2, @@ -210,19 +209,18 @@ MathJax.Hub.Config({
  • Using the correlation matrix
  • Discussing the correlation data
  • Other measures in classification studies: Cancer Data again
    • -
  • Friday September 25
    • -
  • Optimization, the central part of any Machine Learning algortithm
    • -
  • Revisiting our Logistic Regression case
    • -
  • The equations to solve
    • -
  • Solving using Newton-Raphson's method
    • -
  • Brief reminder on Newton-Raphson's method
    • -
  • The equations
    • -
  • Simple geometric interpretation
    • -
  • Extending to more than one variable
    • -
  • Steepest descent
    • -
  • More on Steepest descent
    • -
  • The ideal
    • -
  • The sensitiveness of the gradient descent
    • +
  • Optimization, the central part of any Machine Learning algortithm
    • +
  • Revisiting our Logistic Regression case
    • +
  • The equations to solve
    • +
  • Solving using Newton-Raphson's method
    • +
  • Brief reminder on Newton-Raphson's method
    • +
  • The equations
    • +
  • Simple geometric interpretation
    • +
  • Extending to more than one variable
    • +
  • Steepest descent
    • +
  • More on Steepest descent
    • +
  • The ideal
    • +
  • The sensitiveness of the gradient descent
    • @@ -279,7 +277,7 @@ $$
  • 29
  • 30
  • ...
    • -
  • 43
    • +
  • 42
  • »
    • diff --git a/doc/pub/week38/html/._week38-bs021.html b/doc/pub/week38/html/._week38-bs021.html index daac5ae47..aeaddcc88 100644 --- a/doc/pub/week38/html/._week38-bs021.html +++ b/doc/pub/week38/html/._week38-bs021.html @@ -112,7 +112,6 @@ doconce format html week38.do.txt --html_style=bootstrap --pygments_html_style=d 2, None, 'other-measures-in-classification-studies-cancer-data-again'), - ('Friday September 25', 2, None, 'friday-september-25'), ('Optimization, the central part of any Machine Learning ' 'algortithm', 2, @@ -210,19 +209,18 @@ MathJax.Hub.Config({
  • Using the correlation matrix
  • Discussing the correlation data
  • Other measures in classification studies: Cancer Data again
    • -
  • Friday September 25
    • -
  • Optimization, the central part of any Machine Learning algortithm
    • -
  • Revisiting our Logistic Regression case
    • -
  • The equations to solve
    • -
  • Solving using Newton-Raphson's method
    • -
  • Brief reminder on Newton-Raphson's method
    • -
  • The equations
    • -
  • Simple geometric interpretation
    • -
  • Extending to more than one variable
    • -
  • Steepest descent
    • -
  • More on Steepest descent
    • -
  • The ideal
    • -
  • The sensitiveness of the gradient descent
    • +
  • Optimization, the central part of any Machine Learning algortithm
    • +
  • Revisiting our Logistic Regression case
    • +
  • The equations to solve
    • +
  • Solving using Newton-Raphson's method
    • +
  • Brief reminder on Newton-Raphson's method
    • +
  • The equations
    • +
  • Simple geometric interpretation
    • +
  • Extending to more than one variable
    • +
  • Steepest descent
    • +
  • More on Steepest descent
    • +
  • The ideal
    • +
  • The sensitiveness of the gradient descent
    • @@ -280,7 +278,7 @@ $$
  • 30
  • 31
  • ...
    • -
  • 43
    • +
  • 42
  • »
    • diff --git a/doc/pub/week38/html/._week38-bs022.html b/doc/pub/week38/html/._week38-bs022.html index 64c55fd44..2183ecb3a 100644 --- a/doc/pub/week38/html/._week38-bs022.html +++ b/doc/pub/week38/html/._week38-bs022.html @@ -112,7 +112,6 @@ doconce format html week38.do.txt --html_style=bootstrap --pygments_html_style=d 2, None, 'other-measures-in-classification-studies-cancer-data-again'), - ('Friday September 25', 2, None, 'friday-september-25'), ('Optimization, the central part of any Machine Learning ' 'algortithm', 2, @@ -210,19 +209,18 @@ MathJax.Hub.Config({
  • Using the correlation matrix
  • Discussing the correlation data
  • Other measures in classification studies: Cancer Data again
    • -
  • Friday September 25
    • -
  • Optimization, the central part of any Machine Learning algortithm
    • -
  • Revisiting our Logistic Regression case
    • -
  • The equations to solve
    • -
  • Solving using Newton-Raphson's method
    • -
  • Brief reminder on Newton-Raphson's method
    • -
  • The equations
    • -
  • Simple geometric interpretation
    • -
  • Extending to more than one variable
    • -
  • Steepest descent
    • -
  • More on Steepest descent
    • -
  • The ideal
    • -
  • The sensitiveness of the gradient descent
    • +
  • Optimization, the central part of any Machine Learning algortithm
    • +
  • Revisiting our Logistic Regression case
    • +
  • The equations to solve
    • +
  • Solving using Newton-Raphson's method
    • +
  • Brief reminder on Newton-Raphson's method
    • +
  • The equations
    • +
  • Simple geometric interpretation
    • +
  • Extending to more than one variable
    • +
  • Steepest descent
    • +
  • More on Steepest descent
    • +
  • The ideal
    • +
  • The sensitiveness of the gradient descent
    • @@ -272,7 +270,7 @@ $$
  • 31
  • 32
  • ...
    • -
  • 43
    • +
  • 42
  • »
    • diff --git a/doc/pub/week38/html/._week38-bs023.html b/doc/pub/week38/html/._week38-bs023.html index 943d7bad2..92460bd82 100644 --- a/doc/pub/week38/html/._week38-bs023.html +++ b/doc/pub/week38/html/._week38-bs023.html @@ -112,7 +112,6 @@ doconce format html week38.do.txt --html_style=bootstrap --pygments_html_style=d 2, None, 'other-measures-in-classification-studies-cancer-data-again'), - ('Friday September 25', 2, None, 'friday-september-25'), ('Optimization, the central part of any Machine Learning ' 'algortithm', 2, @@ -210,19 +209,18 @@ MathJax.Hub.Config({
  • Using the correlation matrix
  • Discussing the correlation data
  • Other measures in classification studies: Cancer Data again
    • -
  • Friday September 25
    • -
  • Optimization, the central part of any Machine Learning algortithm
    • -
  • Revisiting our Logistic Regression case
    • -
  • The equations to solve
    • -
  • Solving using Newton-Raphson's method
    • -
  • Brief reminder on Newton-Raphson's method
    • -
  • The equations
    • -
  • Simple geometric interpretation
    • -
  • Extending to more than one variable
    • -
  • Steepest descent
    • -
  • More on Steepest descent
    • -
  • The ideal
    • -
  • The sensitiveness of the gradient descent
    • +
  • Optimization, the central part of any Machine Learning algortithm
    • +
  • Revisiting our Logistic Regression case
    • +
  • The equations to solve
    • +
  • Solving using Newton-Raphson's method
    • +
  • Brief reminder on Newton-Raphson's method
    • +
  • The equations
    • +
  • Simple geometric interpretation
    • +
  • Extending to more than one variable
    • +
  • Steepest descent
    • +
  • More on Steepest descent
    • +
  • The ideal
    • +
  • The sensitiveness of the gradient descent
    • @@ -284,7 +282,7 @@ $$
  • 32
  • 33
  • ...
    • -
  • 43
    • +
  • 42
  • »
    • diff --git a/doc/pub/week38/html/._week38-bs024.html b/doc/pub/week38/html/._week38-bs024.html index 0927f9e0d..f3d585f1f 100644 --- a/doc/pub/week38/html/._week38-bs024.html +++ b/doc/pub/week38/html/._week38-bs024.html @@ -112,7 +112,6 @@ doconce format html week38.do.txt --html_style=bootstrap --pygments_html_style=d 2, None, 'other-measures-in-classification-studies-cancer-data-again'), - ('Friday September 25', 2, None, 'friday-september-25'), ('Optimization, the central part of any Machine Learning ' 'algortithm', 2, @@ -210,19 +209,18 @@ MathJax.Hub.Config({
  • Using the correlation matrix
  • Discussing the correlation data
  • Other measures in classification studies: Cancer Data again
    • -
  • Friday September 25
    • -
  • Optimization, the central part of any Machine Learning algortithm
    • -
  • Revisiting our Logistic Regression case
    • -
  • The equations to solve
    • -
  • Solving using Newton-Raphson's method
    • -
  • Brief reminder on Newton-Raphson's method
    • -
  • The equations
    • -
  • Simple geometric interpretation
    • -
  • Extending to more than one variable
    • -
  • Steepest descent
    • -
  • More on Steepest descent
    • -
  • The ideal
    • -
  • The sensitiveness of the gradient descent
    • +
  • Optimization, the central part of any Machine Learning algortithm
    • +
  • Revisiting our Logistic Regression case
    • +
  • The equations to solve
    • +
  • Solving using Newton-Raphson's method
    • +
  • Brief reminder on Newton-Raphson's method
    • +
  • The equations
    • +
  • Simple geometric interpretation
    • +
  • Extending to more than one variable
    • +
  • Steepest descent
    • +
  • More on Steepest descent
    • +
  • The ideal
    • +
  • The sensitiveness of the gradient descent
    • @@ -296,7 +294,7 @@ methods.
  • 33
  • 34
  • ...
    • -
  • 43
    • +
  • 42
  • »
    • diff --git a/doc/pub/week38/html/._week38-bs025.html b/doc/pub/week38/html/._week38-bs025.html index 5449e772a..a8acf9929 100644 --- a/doc/pub/week38/html/._week38-bs025.html +++ b/doc/pub/week38/html/._week38-bs025.html @@ -112,7 +112,6 @@ doconce format html week38.do.txt --html_style=bootstrap --pygments_html_style=d 2, None, 'other-measures-in-classification-studies-cancer-data-again'), - ('Friday September 25', 2, None, 'friday-september-25'), ('Optimization, the central part of any Machine Learning ' 'algortithm', 2, @@ -210,19 +209,18 @@ MathJax.Hub.Config({
  • Using the correlation matrix
  • Discussing the correlation data
  • Other measures in classification studies: Cancer Data again
    • -
  • Friday September 25
    • -
  • Optimization, the central part of any Machine Learning algortithm
    • -
  • Revisiting our Logistic Regression case
    • -
  • The equations to solve
    • -
  • Solving using Newton-Raphson's method
    • -
  • Brief reminder on Newton-Raphson's method
    • -
  • The equations
    • -
  • Simple geometric interpretation
    • -
  • Extending to more than one variable
    • -
  • Steepest descent
    • -
  • More on Steepest descent
    • -
  • The ideal
    • -
  • The sensitiveness of the gradient descent
    • +
  • Optimization, the central part of any Machine Learning algortithm
    • +
  • Revisiting our Logistic Regression case
    • +
  • The equations to solve
    • +
  • Solving using Newton-Raphson's method
    • +
  • Brief reminder on Newton-Raphson's method
    • +
  • The equations
    • +
  • Simple geometric interpretation
    • +
  • Extending to more than one variable
    • +
  • Steepest descent
    • +
  • More on Steepest descent
    • +
  • The ideal
    • +
  • The sensitiveness of the gradient descent
    • @@ -261,7 +259,7 @@ MathJax.Hub.Config({
  • 34
  • 35
  • ...
    • -
  • 43
    • +
  • 42
  • »
    • diff --git a/doc/pub/week38/html/._week38-bs026.html b/doc/pub/week38/html/._week38-bs026.html index 06b664b52..5fd0e6c7d 100644 --- a/doc/pub/week38/html/._week38-bs026.html +++ b/doc/pub/week38/html/._week38-bs026.html @@ -112,7 +112,6 @@ doconce format html week38.do.txt --html_style=bootstrap --pygments_html_style=d 2, None, 'other-measures-in-classification-studies-cancer-data-again'), - ('Friday September 25', 2, None, 'friday-september-25'), ('Optimization, the central part of any Machine Learning ' 'algortithm', 2, @@ -210,19 +209,18 @@ MathJax.Hub.Config({
  • Using the correlation matrix
  • Discussing the correlation data
  • Other measures in classification studies: Cancer Data again
    • -
  • Friday September 25
    • -
  • Optimization, the central part of any Machine Learning algortithm
    • -
  • Revisiting our Logistic Regression case
    • -
  • The equations to solve
    • -
  • Solving using Newton-Raphson's method
    • -
  • Brief reminder on Newton-Raphson's method
    • -
  • The equations
    • -
  • Simple geometric interpretation
    • -
  • Extending to more than one variable
    • -
  • Steepest descent
    • -
  • More on Steepest descent
    • -
  • The ideal
    • -
  • The sensitiveness of the gradient descent
    • +
  • Optimization, the central part of any Machine Learning algortithm
    • +
  • Revisiting our Logistic Regression case
    • +
  • The equations to solve
    • +
  • Solving using Newton-Raphson's method
    • +
  • Brief reminder on Newton-Raphson's method
    • +
  • The equations
    • +
  • Simple geometric interpretation
    • +
  • Extending to more than one variable
    • +
  • Steepest descent
    • +
  • More on Steepest descent
    • +
  • The ideal
    • +
  • The sensitiveness of the gradient descent
    • @@ -314,7 +312,7 @@ logreg.fit(X_train_scaled, y_train)
  • 35
  • 36
  • ...
    • -
  • 43
    • +
  • 42
  • »
    • diff --git a/doc/pub/week38/html/._week38-bs027.html b/doc/pub/week38/html/._week38-bs027.html index 9c3461aad..bb5becf55 100644 --- a/doc/pub/week38/html/._week38-bs027.html +++ b/doc/pub/week38/html/._week38-bs027.html @@ -112,7 +112,6 @@ doconce format html week38.do.txt --html_style=bootstrap --pygments_html_style=d 2, None, 'other-measures-in-classification-studies-cancer-data-again'), - ('Friday September 25', 2, None, 'friday-september-25'), ('Optimization, the central part of any Machine Learning ' 'algortithm', 2, @@ -210,19 +209,18 @@ MathJax.Hub.Config({
  • Using the correlation matrix
  • Discussing the correlation data
  • Other measures in classification studies: Cancer Data again
    • -
  • Friday September 25
    • -
  • Optimization, the central part of any Machine Learning algortithm
    • -
  • Revisiting our Logistic Regression case
    • -
  • The equations to solve
    • -
  • Solving using Newton-Raphson's method
    • -
  • Brief reminder on Newton-Raphson's method
    • -
  • The equations
    • -
  • Simple geometric interpretation
    • -
  • Extending to more than one variable
    • -
  • Steepest descent
    • -
  • More on Steepest descent
    • -
  • The ideal
    • -
  • The sensitiveness of the gradient descent
    • +
  • Optimization, the central part of any Machine Learning algortithm
    • +
  • Revisiting our Logistic Regression case
    • +
  • The equations to solve
    • +
  • Solving using Newton-Raphson's method
    • +
  • Brief reminder on Newton-Raphson's method
    • +
  • The equations
    • +
  • Simple geometric interpretation
    • +
  • Extending to more than one variable
    • +
  • Steepest descent
    • +
  • More on Steepest descent
    • +
  • The ideal
    • +
  • The sensitiveness of the gradient descent
    • @@ -321,7 +319,7 @@ plt.show()
  • 36
  • 37
  • ...
    • -
  • 43
    • +
  • 42
  • »
    • diff --git a/doc/pub/week38/html/._week38-bs028.html b/doc/pub/week38/html/._week38-bs028.html index d529b3cee..ac98b8d96 100644 --- a/doc/pub/week38/html/._week38-bs028.html +++ b/doc/pub/week38/html/._week38-bs028.html @@ -112,7 +112,6 @@ doconce format html week38.do.txt --html_style=bootstrap --pygments_html_style=d 2, None, 'other-measures-in-classification-studies-cancer-data-again'), - ('Friday September 25', 2, None, 'friday-september-25'), ('Optimization, the central part of any Machine Learning ' 'algortithm', 2, @@ -210,19 +209,18 @@ MathJax.Hub.Config({
  • Using the correlation matrix
  • Discussing the correlation data
  • Other measures in classification studies: Cancer Data again
    • -
  • Friday September 25
    • -
  • Optimization, the central part of any Machine Learning algortithm
    • -
  • Revisiting our Logistic Regression case
    • -
  • The equations to solve
    • -
  • Solving using Newton-Raphson's method
    • -
  • Brief reminder on Newton-Raphson's method
    • -
  • The equations
    • -
  • Simple geometric interpretation
    • -
  • Extending to more than one variable
    • -
  • Steepest descent
    • -
  • More on Steepest descent
    • -
  • The ideal
    • -
  • The sensitiveness of the gradient descent
    • +
  • Optimization, the central part of any Machine Learning algortithm
    • +
  • Revisiting our Logistic Regression case
    • +
  • The equations to solve
    • +
  • Solving using Newton-Raphson's method
    • +
  • Brief reminder on Newton-Raphson's method
    • +
  • The equations
    • +
  • Simple geometric interpretation
    • +
  • Extending to more than one variable
    • +
  • Steepest descent
    • +
  • More on Steepest descent
    • +
  • The ideal
    • +
  • The sensitiveness of the gradient descent
    • @@ -329,7 +327,7 @@ applications. This will be discussed later this semester (37
  • 38
  • ...
    • -
  • 43
    • +
  • 42
  • »
    • diff --git a/doc/pub/week38/html/._week38-bs029.html b/doc/pub/week38/html/._week38-bs029.html index e0a5c2338..e5254da59 100644 --- a/doc/pub/week38/html/._week38-bs029.html +++ b/doc/pub/week38/html/._week38-bs029.html @@ -112,7 +112,6 @@ doconce format html week38.do.txt --html_style=bootstrap --pygments_html_style=d 2, None, 'other-measures-in-classification-studies-cancer-data-again'), - ('Friday September 25', 2, None, 'friday-september-25'), ('Optimization, the central part of any Machine Learning ' 'algortithm', 2, @@ -210,19 +209,18 @@ MathJax.Hub.Config({
  • Using the correlation matrix
  • Discussing the correlation data
  • Other measures in classification studies: Cancer Data again
    • -
  • Friday September 25
    • -
  • Optimization, the central part of any Machine Learning algortithm
    • -
  • Revisiting our Logistic Regression case
    • -
  • The equations to solve
    • -
  • Solving using Newton-Raphson's method
    • -
  • Brief reminder on Newton-Raphson's method
    • -
  • The equations
    • -
  • Simple geometric interpretation
    • -
  • Extending to more than one variable
    • -
  • Steepest descent
    • -
  • More on Steepest descent
    • -
  • The ideal
    • -
  • The sensitiveness of the gradient descent
    • +
  • Optimization, the central part of any Machine Learning algortithm
    • +
  • Revisiting our Logistic Regression case
    • +
  • The equations to solve
    • +
  • Solving using Newton-Raphson's method
    • +
  • Brief reminder on Newton-Raphson's method
    • +
  • The equations
    • +
  • Simple geometric interpretation
    • +
  • Extending to more than one variable
    • +
  • Steepest descent
    • +
  • More on Steepest descent
    • +
  • The ideal
    • +
  • The sensitiveness of the gradient descent
    • @@ -327,7 +325,7 @@ plt.show()
  • 38
  • 39
  • ...
    • -
  • 43
    • +
  • 42
  • »
    • diff --git a/doc/pub/week38/html/._week38-bs030.html b/doc/pub/week38/html/._week38-bs030.html index 39a47e6e1..5751878b5 100644 --- a/doc/pub/week38/html/._week38-bs030.html +++ b/doc/pub/week38/html/._week38-bs030.html @@ -112,7 +112,6 @@ doconce format html week38.do.txt --html_style=bootstrap --pygments_html_style=d 2, None, 'other-measures-in-classification-studies-cancer-data-again'), - ('Friday September 25', 2, None, 'friday-september-25'), ('Optimization, the central part of any Machine Learning ' 'algortithm', 2, @@ -210,19 +209,18 @@ MathJax.Hub.Config({
  • Using the correlation matrix
  • Discussing the correlation data
  • Other measures in classification studies: Cancer Data again
    • -
  • Friday September 25
    • -
  • Optimization, the central part of any Machine Learning algortithm
    • -
  • Revisiting our Logistic Regression case
    • -
  • The equations to solve
    • -
  • Solving using Newton-Raphson's method
    • -
  • Brief reminder on Newton-Raphson's method
    • -
  • The equations
    • -
  • Simple geometric interpretation
    • -
  • Extending to more than one variable
    • -
  • Steepest descent
    • -
  • More on Steepest descent
    • -
  • The ideal
    • -
  • The sensitiveness of the gradient descent
    • +
  • Optimization, the central part of any Machine Learning algortithm
    • +
  • Revisiting our Logistic Regression case
    • +
  • The equations to solve
    • +
  • Solving using Newton-Raphson's method
    • +
  • Brief reminder on Newton-Raphson's method
    • +
  • The equations
    • +
  • Simple geometric interpretation
    • +
  • Extending to more than one variable
    • +
  • Steepest descent
    • +
  • More on Steepest descent
    • +
  • The ideal
    • +
  • The sensitiveness of the gradient descent
    • @@ -234,7 +232,19 @@ MathJax.Hub.Config({

     

     

     

    -

    Friday September 25

    +

    Optimization, the central part of any Machine Learning algortithm

    + +Overview Video, why do we care about gradient methods? + +

    Almost every problem in machine learning and data science starts with +a dataset \( X \), a model \( g(\beta) \), which is a function of the +parameters \( \beta \) and a cost function \( C(X, g(\beta)) \) that allows +us to judge how well the model \( g(\beta) \) explains the observations +\( X \). The model is fit by finding the values of \( \beta \) that minimize +the cost function. Ideally we would be able to solve for \( \beta \) +analytically, however this is not possible in general and we must use +some approximative/numerical method to compute the minimum. +

    @@ -261,7 +271,7 @@ MathJax.Hub.Config({

  • 39
  • 40
  • ...
    • -
  • 43
    • +
  • 42
  • »
    • diff --git a/doc/pub/week38/html/._week38-bs031.html b/doc/pub/week38/html/._week38-bs031.html index a7c01321c..be0290b7f 100644 --- a/doc/pub/week38/html/._week38-bs031.html +++ b/doc/pub/week38/html/._week38-bs031.html @@ -112,7 +112,6 @@ doconce format html week38.do.txt --html_style=bootstrap --pygments_html_style=d 2, None, 'other-measures-in-classification-studies-cancer-data-again'), - ('Friday September 25', 2, None, 'friday-september-25'), ('Optimization, the central part of any Machine Learning ' 'algortithm', 2, @@ -210,19 +209,18 @@ MathJax.Hub.Config({
  • Using the correlation matrix
  • Discussing the correlation data
  • Other measures in classification studies: Cancer Data again
    • -
  • Friday September 25
    • -
  • Optimization, the central part of any Machine Learning algortithm
    • -
  • Revisiting our Logistic Regression case
    • -
  • The equations to solve
    • -
  • Solving using Newton-Raphson's method
    • -
  • Brief reminder on Newton-Raphson's method
    • -
  • The equations
    • -
  • Simple geometric interpretation
    • -
  • Extending to more than one variable
    • -
  • Steepest descent
    • -
  • More on Steepest descent
    • -
  • The ideal
    • -
  • The sensitiveness of the gradient descent
    • +
  • Optimization, the central part of any Machine Learning algortithm
    • +
  • Revisiting our Logistic Regression case
    • +
  • The equations to solve
    • +
  • Solving using Newton-Raphson's method
    • +
  • Brief reminder on Newton-Raphson's method
    • +
  • The equations
    • +
  • Simple geometric interpretation
    • +
  • Extending to more than one variable
    • +
  • Steepest descent
    • +
  • More on Steepest descent
    • +
  • The ideal
    • +
  • The sensitiveness of the gradient descent
    • @@ -234,20 +232,25 @@ MathJax.Hub.Config({

     

     

     

    -

    Optimization, the central part of any Machine Learning algortithm

    +

    Revisiting our Logistic Regression case

    -Overview Video, why do we care about gradient methods? - -

    Almost every problem in machine learning and data science starts with -a dataset \( X \), a model \( g(\beta) \), which is a function of the -parameters \( \beta \) and a cost function \( C(X, g(\beta)) \) that allows -us to judge how well the model \( g(\beta) \) explains the observations -\( X \). The model is fit by finding the values of \( \beta \) that minimize -the cost function. Ideally we would be able to solve for \( \beta \) -analytically, however this is not possible in general and we must use -some approximative/numerical method to compute the minimum. +

    In our discussion on Logistic Regression we studied the +case of +two classes, with \( y_i \) either +\( 0 \) or \( 1 \). Furthermore we assumed also that we have only two +parameters \( \beta \) in our fitting, that is we +defined probabilities

    +$$ +\begin{align*} +p(y_i=1|x_i,\boldsymbol{\beta}) &= \frac{\exp{(\beta_0+\beta_1x_i)}}{1+\exp{(\beta_0+\beta_1x_i)}},\nonumber\\ +p(y_i=0|x_i,\boldsymbol{\beta}) &= 1 - p(y_i=1|x_i,\boldsymbol{\beta}), +\end{align*} +$$ + +

    where \( \boldsymbol{\beta} \) are the weights we wish to extract from data, in our case \( \beta_0 \) and \( \beta_1 \).

    +

    diff --git a/doc/pub/week38/html/._week38-bs032.html b/doc/pub/week38/html/._week38-bs032.html index 7c95b1d67..8df35c3c3 100644 --- a/doc/pub/week38/html/._week38-bs032.html +++ b/doc/pub/week38/html/._week38-bs032.html @@ -112,7 +112,6 @@ doconce format html week38.do.txt --html_style=bootstrap --pygments_html_style=d 2, None, 'other-measures-in-classification-studies-cancer-data-again'), - ('Friday September 25', 2, None, 'friday-september-25'), ('Optimization, the central part of any Machine Learning ' 'algortithm', 2, @@ -210,19 +209,18 @@ MathJax.Hub.Config({
  • Using the correlation matrix
  • Discussing the correlation data
  • Other measures in classification studies: Cancer Data again
    • -
  • Friday September 25
    • -
  • Optimization, the central part of any Machine Learning algortithm
    • -
  • Revisiting our Logistic Regression case
    • -
  • The equations to solve
    • -
  • Solving using Newton-Raphson's method
    • -
  • Brief reminder on Newton-Raphson's method
    • -
  • The equations
    • -
  • Simple geometric interpretation
    • -
  • Extending to more than one variable
    • -
  • Steepest descent
    • -
  • More on Steepest descent
    • -
  • The ideal
    • -
  • The sensitiveness of the gradient descent
    • +
  • Optimization, the central part of any Machine Learning algortithm
    • +
  • Revisiting our Logistic Regression case
    • +
  • The equations to solve
    • +
  • Solving using Newton-Raphson's method
    • +
  • Brief reminder on Newton-Raphson's method
    • +
  • The equations
    • +
  • Simple geometric interpretation
    • +
  • Extending to more than one variable
    • +
  • Steepest descent
    • +
  • More on Steepest descent
    • +
  • The ideal
    • +
  • The sensitiveness of the gradient descent
    • @@ -234,24 +232,28 @@ MathJax.Hub.Config({

     

     

     

    -

    Revisiting our Logistic Regression case

    +

    The equations to solve

    -

    In our discussion on Logistic Regression we studied the -case of -two classes, with \( y_i \) either -\( 0 \) or \( 1 \). Furthermore we assumed also that we have only two -parameters \( \beta \) in our fitting, that is we -defined probabilities +

    Our compact equations used a definition of a vector \( \boldsymbol{y} \) with \( n \) +elements \( y_i \), an \( n\times p \) matrix \( \boldsymbol{X} \) which contains the +\( x_i \) values and a vector \( \boldsymbol{p} \) of fitted probabilities +\( p(y_i\vert x_i,\boldsymbol{\beta}) \). We rewrote in a more compact form +the first derivative of the cost function as

    $$ -\begin{align*} -p(y_i=1|x_i,\boldsymbol{\beta}) &= \frac{\exp{(\beta_0+\beta_1x_i)}}{1+\exp{(\beta_0+\beta_1x_i)}},\nonumber\\ -p(y_i=0|x_i,\boldsymbol{\beta}) &= 1 - p(y_i=1|x_i,\boldsymbol{\beta}), -\end{align*} +\frac{\partial \mathcal{C}(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}} = -\boldsymbol{X}^T\left(\boldsymbol{y}-\boldsymbol{p}\right). $$ -

    where \( \boldsymbol{\beta} \) are the weights we wish to extract from data, in our case \( \beta_0 \) and \( \beta_1 \).

    +

    If we in addition define a diagonal matrix \( \boldsymbol{W} \) with elements +\( p(y_i\vert x_i,\boldsymbol{\beta})(1-p(y_i\vert x_i,\boldsymbol{\beta}) \), we can obtain a compact expression of the second derivative as +

    + +$$ +\frac{\partial^2 \mathcal{C}(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}\partial \boldsymbol{\beta}^T} = \boldsymbol{X}^T\boldsymbol{W}\boldsymbol{X}. +$$ + +

    This defines what is called the Hessian matrix.

    @@ -277,8 +279,6 @@ $$

  • 40
  • 41
  • 42
    • -
  • ...
    • -
  • 43
  • »
    • diff --git a/doc/pub/week38/html/._week38-bs033.html b/doc/pub/week38/html/._week38-bs033.html index 8c8df75f2..c4184c30b 100644 --- a/doc/pub/week38/html/._week38-bs033.html +++ b/doc/pub/week38/html/._week38-bs033.html @@ -112,7 +112,6 @@ doconce format html week38.do.txt --html_style=bootstrap --pygments_html_style=d 2, None, 'other-measures-in-classification-studies-cancer-data-again'), - ('Friday September 25', 2, None, 'friday-september-25'), ('Optimization, the central part of any Machine Learning ' 'algortithm', 2, @@ -210,19 +209,18 @@ MathJax.Hub.Config({
  • Using the correlation matrix
  • Discussing the correlation data
  • Other measures in classification studies: Cancer Data again
    • -
  • Friday September 25
    • -
  • Optimization, the central part of any Machine Learning algortithm
    • -
  • Revisiting our Logistic Regression case
    • -
  • The equations to solve
    • -
  • Solving using Newton-Raphson's method
    • -
  • Brief reminder on Newton-Raphson's method
    • -
  • The equations
    • -
  • Simple geometric interpretation
    • -
  • Extending to more than one variable
    • -
  • Steepest descent
    • -
  • More on Steepest descent
    • -
  • The ideal
    • -
  • The sensitiveness of the gradient descent
    • +
  • Optimization, the central part of any Machine Learning algortithm
    • +
  • Revisiting our Logistic Regression case
    • +
  • The equations to solve
    • +
  • Solving using Newton-Raphson's method
    • +
  • Brief reminder on Newton-Raphson's method
    • +
  • The equations
    • +
  • Simple geometric interpretation
    • +
  • Extending to more than one variable
    • +
  • Steepest descent
    • +
  • More on Steepest descent
    • +
  • The ideal
    • +
  • The sensitiveness of the gradient descent
    • @@ -234,28 +232,25 @@ MathJax.Hub.Config({

     

     

     

    -

    The equations to solve

    +

    Solving using Newton-Raphson's method

    -

    Our compact equations used a definition of a vector \( \boldsymbol{y} \) with \( n \) -elements \( y_i \), an \( n\times p \) matrix \( \boldsymbol{X} \) which contains the -\( x_i \) values and a vector \( \boldsymbol{p} \) of fitted probabilities -\( p(y_i\vert x_i,\boldsymbol{\beta}) \). We rewrote in a more compact form -the first derivative of the cost function as -

    +

    If we can set up these equations, Newton-Raphson's iterative method is normally the method of choice. It requires however that we can compute in an efficient way the matrices that define the first and second derivatives.

    + +

    Our iterative scheme is then given by

    $$ -\frac{\partial \mathcal{C}(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}} = -\boldsymbol{X}^T\left(\boldsymbol{y}-\boldsymbol{p}\right). +\boldsymbol{\beta}^{\mathrm{new}} = \boldsymbol{\beta}^{\mathrm{old}}-\left(\frac{\partial^2 \mathcal{C}(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}\partial \boldsymbol{\beta}^T}\right)^{-1}_{\boldsymbol{\beta}^{\mathrm{old}}}\times \left(\frac{\partial \mathcal{C}(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}}\right)_{\boldsymbol{\beta}^{\mathrm{old}}}, $$ -

    If we in addition define a diagonal matrix \( \boldsymbol{W} \) with elements -\( p(y_i\vert x_i,\boldsymbol{\beta})(1-p(y_i\vert x_i,\boldsymbol{\beta}) \), we can obtain a compact expression of the second derivative as -

    +

    or in matrix form as

    $$ -\frac{\partial^2 \mathcal{C}(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}\partial \boldsymbol{\beta}^T} = \boldsymbol{X}^T\boldsymbol{W}\boldsymbol{X}. +\boldsymbol{\beta}^{\mathrm{new}} = \boldsymbol{\beta}^{\mathrm{old}}-\left(\boldsymbol{X}^T\boldsymbol{W}\boldsymbol{X} \right)^{-1}\times \left(-\boldsymbol{X}^T(\boldsymbol{y}-\boldsymbol{p}) \right)_{\boldsymbol{\beta}^{\mathrm{old}}}. $$ -

    This defines what is called the Hessian matrix.

    +

    The right-hand side is computed with the old values of \( \beta \).

    + +

    If we can compute these matrices, in particular the Hessian, the above is often the easiest method to implement.

    @@ -280,7 +275,6 @@ $$

  • 40
  • 41
  • 42
    • -
  • 43
  • »
    • diff --git a/doc/pub/week38/html/._week38-bs034.html b/doc/pub/week38/html/._week38-bs034.html index 9feabd083..b223ec106 100644 --- a/doc/pub/week38/html/._week38-bs034.html +++ b/doc/pub/week38/html/._week38-bs034.html @@ -112,7 +112,6 @@ doconce format html week38.do.txt --html_style=bootstrap --pygments_html_style=d 2, None, 'other-measures-in-classification-studies-cancer-data-again'), - ('Friday September 25', 2, None, 'friday-september-25'), ('Optimization, the central part of any Machine Learning ' 'algortithm', 2, @@ -210,19 +209,18 @@ MathJax.Hub.Config({
  • Using the correlation matrix
  • Discussing the correlation data
  • Other measures in classification studies: Cancer Data again
    • -
  • Friday September 25
    • -
  • Optimization, the central part of any Machine Learning algortithm
    • -
  • Revisiting our Logistic Regression case
    • -
  • The equations to solve
    • -
  • Solving using Newton-Raphson's method
    • -
  • Brief reminder on Newton-Raphson's method
    • -
  • The equations
    • -
  • Simple geometric interpretation
    • -
  • Extending to more than one variable
    • -
  • Steepest descent
    • -
  • More on Steepest descent
    • -
  • The ideal
    • -
  • The sensitiveness of the gradient descent
    • +
  • Optimization, the central part of any Machine Learning algortithm
    • +
  • Revisiting our Logistic Regression case
    • +
  • The equations to solve
    • +
  • Solving using Newton-Raphson's method
    • +
  • Brief reminder on Newton-Raphson's method
    • +
  • The equations
    • +
  • Simple geometric interpretation
    • +
  • Extending to more than one variable
    • +
  • Steepest descent
    • +
  • More on Steepest descent
    • +
  • The ideal
    • +
  • The sensitiveness of the gradient descent
    • @@ -234,25 +232,18 @@ MathJax.Hub.Config({

     

     

     

    -

    Solving using Newton-Raphson's method

    +

    Brief reminder on Newton-Raphson's method

    -

    If we can set up these equations, Newton-Raphson's iterative method is normally the method of choice. It requires however that we can compute in an efficient way the matrices that define the first and second derivatives.

    +

    Let us quickly remind ourselves how we derive the above method.

    -

    Our iterative scheme is then given by

    - -$$ -\boldsymbol{\beta}^{\mathrm{new}} = \boldsymbol{\beta}^{\mathrm{old}}-\left(\frac{\partial^2 \mathcal{C}(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}\partial \boldsymbol{\beta}^T}\right)^{-1}_{\boldsymbol{\beta}^{\mathrm{old}}}\times \left(\frac{\partial \mathcal{C}(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}}\right)_{\boldsymbol{\beta}^{\mathrm{old}}}, -$$ - -

    or in matrix form as

    - -$$ -\boldsymbol{\beta}^{\mathrm{new}} = \boldsymbol{\beta}^{\mathrm{old}}-\left(\boldsymbol{X}^T\boldsymbol{W}\boldsymbol{X} \right)^{-1}\times \left(-\boldsymbol{X}^T(\boldsymbol{y}-\boldsymbol{p}) \right)_{\boldsymbol{\beta}^{\mathrm{old}}}. -$$ - -

    The right-hand side is computed with the old values of \( \beta \).

    - -

    If we can compute these matrices, in particular the Hessian, the above is often the easiest method to implement.

    +

    Perhaps the most celebrated of all one-dimensional root-finding +routines is Newton's method, also called the Newton-Raphson +method. This method requires the evaluation of both the +function \( f \) and its derivative \( f' \) at arbitrary points. +If you can only calculate the derivative +numerically and/or your function is not of the smooth type, we +normally discourage the use of this method. +

    @@ -276,7 +267,6 @@ $$

  • 40
  • 41
  • 42
    • -
  • 43
  • »
    • diff --git a/doc/pub/week38/html/._week38-bs035.html b/doc/pub/week38/html/._week38-bs035.html index 9ea7e5542..d2b29e53a 100644 --- a/doc/pub/week38/html/._week38-bs035.html +++ b/doc/pub/week38/html/._week38-bs035.html @@ -112,7 +112,6 @@ doconce format html week38.do.txt --html_style=bootstrap --pygments_html_style=d 2, None, 'other-measures-in-classification-studies-cancer-data-again'), - ('Friday September 25', 2, None, 'friday-september-25'), ('Optimization, the central part of any Machine Learning ' 'algortithm', 2, @@ -210,19 +209,18 @@ MathJax.Hub.Config({
  • Using the correlation matrix
  • Discussing the correlation data
  • Other measures in classification studies: Cancer Data again
    • -
  • Friday September 25
    • -
  • Optimization, the central part of any Machine Learning algortithm
    • -
  • Revisiting our Logistic Regression case
    • -
  • The equations to solve
    • -
  • Solving using Newton-Raphson's method
    • -
  • Brief reminder on Newton-Raphson's method
    • -
  • The equations
    • -
  • Simple geometric interpretation
    • -
  • Extending to more than one variable
    • -
  • Steepest descent
    • -
  • More on Steepest descent
    • -
  • The ideal
    • -
  • The sensitiveness of the gradient descent
    • +
  • Optimization, the central part of any Machine Learning algortithm
    • +
  • Revisiting our Logistic Regression case
    • +
  • The equations to solve
    • +
  • Solving using Newton-Raphson's method
    • +
  • Brief reminder on Newton-Raphson's method
    • +
  • The equations
    • +
  • Simple geometric interpretation
    • +
  • Extending to more than one variable
    • +
  • Steepest descent
    • +
  • More on Steepest descent
    • +
  • The ideal
    • +
  • The sensitiveness of the gradient descent
    • @@ -234,19 +232,39 @@ MathJax.Hub.Config({

     

     

     

    -

    Brief reminder on Newton-Raphson's method

    +

    The equations

    -

    Let us quickly remind ourselves how we derive the above method.

    - -

    Perhaps the most celebrated of all one-dimensional root-finding -routines is Newton's method, also called the Newton-Raphson -method. This method requires the evaluation of both the -function \( f \) and its derivative \( f' \) at arbitrary points. -If you can only calculate the derivative -numerically and/or your function is not of the smooth type, we -normally discourage the use of this method. +

    The Newton-Raphson formula consists geometrically of extending the +tangent line at a current point until it crosses zero, then setting +the next guess to the abscissa of that zero-crossing. The mathematics +behind this method is rather simple. Employing a Taylor expansion for +\( x \) sufficiently close to the solution \( s \), we have

    +$$ + f(s)=0=f(x)+(s-x)f'(x)+\frac{(s-x)^2}{2}f''(x) +\dots. + \tag{2} +$$ + +

    For small enough values of the function and for well-behaved +functions, the terms beyond linear are unimportant, hence we obtain +

    + +$$ + f(x)+(s-x)f'(x)\approx 0, +$$ + +

    yielding

    +$$ + s\approx x-\frac{f(x)}{f'(x)}. +$$ + +

    Having in mind an iterative procedure, it is natural to start iterating with

    +$$ + x_{n+1}=x_n-\frac{f(x_n)}{f'(x_n)}. +$$ + +

    diff --git a/doc/pub/week38/html/._week38-bs036.html b/doc/pub/week38/html/._week38-bs036.html index 84052c007..34dedcd05 100644 --- a/doc/pub/week38/html/._week38-bs036.html +++ b/doc/pub/week38/html/._week38-bs036.html @@ -112,7 +112,6 @@ doconce format html week38.do.txt --html_style=bootstrap --pygments_html_style=d 2, None, 'other-measures-in-classification-studies-cancer-data-again'), - ('Friday September 25', 2, None, 'friday-september-25'), ('Optimization, the central part of any Machine Learning ' 'algortithm', 2, @@ -210,19 +209,18 @@ MathJax.Hub.Config({
  • Using the correlation matrix
  • Discussing the correlation data
  • Other measures in classification studies: Cancer Data again
    • -
  • Friday September 25
    • -
  • Optimization, the central part of any Machine Learning algortithm
    • -
  • Revisiting our Logistic Regression case
    • -
  • The equations to solve
    • -
  • Solving using Newton-Raphson's method
    • -
  • Brief reminder on Newton-Raphson's method
    • -
  • The equations
    • -
  • Simple geometric interpretation
    • -
  • Extending to more than one variable
    • -
  • Steepest descent
    • -
  • More on Steepest descent
    • -
  • The ideal
    • -
  • The sensitiveness of the gradient descent
    • +
  • Optimization, the central part of any Machine Learning algortithm
    • +
  • Revisiting our Logistic Regression case
    • +
  • The equations to solve
    • +
  • Solving using Newton-Raphson's method
    • +
  • Brief reminder on Newton-Raphson's method
    • +
  • The equations
    • +
  • Simple geometric interpretation
    • +
  • Extending to more than one variable
    • +
  • Steepest descent
    • +
  • More on Steepest descent
    • +
  • The ideal
    • +
  • The sensitiveness of the gradient descent
    • @@ -234,39 +232,21 @@ MathJax.Hub.Config({

     

     

     

    -

    The equations

    +

    Simple geometric interpretation

    -

    The Newton-Raphson formula consists geometrically of extending the -tangent line at a current point until it crosses zero, then setting -the next guess to the abscissa of that zero-crossing. The mathematics -behind this method is rather simple. Employing a Taylor expansion for -\( x \) sufficiently close to the solution \( s \), we have +

    The above is Newton-Raphson's method. It has a simple geometric +interpretation, namely \( x_{n+1} \) is the point where the tangent from +\( (x_n,f(x_n)) \) crosses the \( x \)-axis. Close to the solution, +Newton-Raphson converges fast to the desired result. However, if we +are far from a root, where the higher-order terms in the series are +important, the Newton-Raphson formula can give grossly inaccurate +results. For instance, the initial guess for the root might be so far +from the true root as to let the search interval include a local +maximum or minimum of the function. If an iteration places a trial +guess near such a local extremum, so that the first derivative nearly +vanishes, then Newton-Raphson may fail totally

    -$$ - f(s)=0=f(x)+(s-x)f'(x)+\frac{(s-x)^2}{2}f''(x) +\dots. - \tag{2} -$$ - -

    For small enough values of the function and for well-behaved -functions, the terms beyond linear are unimportant, hence we obtain -

    - -$$ - f(x)+(s-x)f'(x)\approx 0, -$$ - -

    yielding

    -$$ - s\approx x-\frac{f(x)}{f'(x)}. -$$ - -

    Having in mind an iterative procedure, it is natural to start iterating with

    -$$ - x_{n+1}=x_n-\frac{f(x_n)}{f'(x_n)}. -$$ - -

    diff --git a/doc/pub/week38/html/._week38-bs037.html b/doc/pub/week38/html/._week38-bs037.html index 97bc41a29..dbbd9e692 100644 --- a/doc/pub/week38/html/._week38-bs037.html +++ b/doc/pub/week38/html/._week38-bs037.html @@ -112,7 +112,6 @@ doconce format html week38.do.txt --html_style=bootstrap --pygments_html_style=d 2, None, 'other-measures-in-classification-studies-cancer-data-again'), - ('Friday September 25', 2, None, 'friday-september-25'), ('Optimization, the central part of any Machine Learning ' 'algortithm', 2, @@ -210,19 +209,18 @@ MathJax.Hub.Config({
  • Using the correlation matrix
  • Discussing the correlation data
  • Other measures in classification studies: Cancer Data again
    • -
  • Friday September 25
    • -
  • Optimization, the central part of any Machine Learning algortithm
    • -
  • Revisiting our Logistic Regression case
    • -
  • The equations to solve
    • -
  • Solving using Newton-Raphson's method
    • -
  • Brief reminder on Newton-Raphson's method
    • -
  • The equations
    • -
  • Simple geometric interpretation
    • -
  • Extending to more than one variable
    • -
  • Steepest descent
    • -
  • More on Steepest descent
    • -
  • The ideal
    • -
  • The sensitiveness of the gradient descent
    • +
  • Optimization, the central part of any Machine Learning algortithm
    • +
  • Revisiting our Logistic Regression case
    • +
  • The equations to solve
    • +
  • Solving using Newton-Raphson's method
    • +
  • Brief reminder on Newton-Raphson's method
    • +
  • The equations
    • +
  • Simple geometric interpretation
    • +
  • Extending to more than one variable
    • +
  • Steepest descent
    • +
  • More on Steepest descent
    • +
  • The ideal
    • +
  • The sensitiveness of the gradient descent
    • @@ -234,19 +232,57 @@ MathJax.Hub.Config({

     

     

     

    -

    Simple geometric interpretation

    +

    Extending to more than one variable

    -

    The above is Newton-Raphson's method. It has a simple geometric -interpretation, namely \( x_{n+1} \) is the point where the tangent from -\( (x_n,f(x_n)) \) crosses the \( x \)-axis. Close to the solution, -Newton-Raphson converges fast to the desired result. However, if we -are far from a root, where the higher-order terms in the series are -important, the Newton-Raphson formula can give grossly inaccurate -results. For instance, the initial guess for the root might be so far -from the true root as to let the search interval include a local -maximum or minimum of the function. If an iteration places a trial -guess near such a local extremum, so that the first derivative nearly -vanishes, then Newton-Raphson may fail totally +

    Newton's method can be generalized to systems of several non-linear equations +and variables. Consider the case with two equations +

    +$$ + \begin{array}{cc} f_1(x_1,x_2) &=0\\ + f_2(x_1,x_2) &=0,\end{array} +$$ + +

    which we Taylor expand to obtain

    + +$$ + \begin{array}{cc} 0=f_1(x_1+h_1,x_2+h_2)=&f_1(x_1,x_2)+h_1 + \partial f_1/\partial x_1+h_2 + \partial f_1/\partial x_2+\dots\\ + 0=f_2(x_1+h_1,x_2+h_2)=&f_2(x_1,x_2)+h_1 + \partial f_2/\partial x_1+h_2 + \partial f_2/\partial x_2+\dots + \end{array}. +$$ + +

    Defining the Jacobian matrix \( {\bf \boldsymbol{J}} \) we have

    +$$ + {\bf \boldsymbol{J}}=\left( \begin{array}{cc} + \partial f_1/\partial x_1 & \partial f_1/\partial x_2 \\ + \partial f_2/\partial x_1 &\partial f_2/\partial x_2 + \end{array} \right), +$$ + +

    we can rephrase Newton's method as

    +$$ +\left(\begin{array}{c} x_1^{n+1} \\ x_2^{n+1} \end{array} \right)= +\left(\begin{array}{c} x_1^{n} \\ x_2^{n} \end{array} \right)+ +\left(\begin{array}{c} h_1^{n} \\ h_2^{n} \end{array} \right), +$$ + +

    where we have defined

    +$$ + \left(\begin{array}{c} h_1^{n} \\ h_2^{n} \end{array} \right)= + -{\bf \boldsymbol{J}}^{-1} + \left(\begin{array}{c} f_1(x_1^{n},x_2^{n}) \\ f_2(x_1^{n},x_2^{n}) \end{array} \right). +$$ + +

    We need thus to compute the inverse of the Jacobian matrix and it +is to understand that difficulties may +arise in case \( {\bf \boldsymbol{J}} \) is nearly singular. +

    + +

    It is rather straightforward to extend the above scheme to systems of +more than two non-linear equations. In our case, the Jacobian matrix is given by the Hessian that represents the second derivative of cost function.

    @@ -268,7 +304,6 @@ vanishes, then Newton-Raphson may fail totally

  • 40
  • 41
  • 42
    • -
  • 43
  • »
    • diff --git a/doc/pub/week38/html/._week38-bs038.html b/doc/pub/week38/html/._week38-bs038.html index 87916bcbf..f1bdd342c 100644 --- a/doc/pub/week38/html/._week38-bs038.html +++ b/doc/pub/week38/html/._week38-bs038.html @@ -112,7 +112,6 @@ doconce format html week38.do.txt --html_style=bootstrap --pygments_html_style=d 2, None, 'other-measures-in-classification-studies-cancer-data-again'), - ('Friday September 25', 2, None, 'friday-september-25'), ('Optimization, the central part of any Machine Learning ' 'algortithm', 2, @@ -210,19 +209,18 @@ MathJax.Hub.Config({
  • Using the correlation matrix
  • Discussing the correlation data
  • Other measures in classification studies: Cancer Data again
    • -
  • Friday September 25
    • -
  • Optimization, the central part of any Machine Learning algortithm
    • -
  • Revisiting our Logistic Regression case
    • -
  • The equations to solve
    • -
  • Solving using Newton-Raphson's method
    • -
  • Brief reminder on Newton-Raphson's method
    • -
  • The equations
    • -
  • Simple geometric interpretation
    • -
  • Extending to more than one variable
    • -
  • Steepest descent
    • -
  • More on Steepest descent
    • -
  • The ideal
    • -
  • The sensitiveness of the gradient descent
    • +
  • Optimization, the central part of any Machine Learning algortithm
    • +
  • Revisiting our Logistic Regression case
    • +
  • The equations to solve
    • +
  • Solving using Newton-Raphson's method
    • +
  • Brief reminder on Newton-Raphson's method
    • +
  • The equations
    • +
  • Simple geometric interpretation
    • +
  • Extending to more than one variable
    • +
  • Steepest descent
    • +
  • More on Steepest descent
    • +
  • The ideal
    • +
  • The sensitiveness of the gradient descent
    • @@ -234,57 +232,24 @@ MathJax.Hub.Config({

     

     

     

    -

    Extending to more than one variable

    +

    Steepest descent

    -

    Newton's method can be generalized to systems of several non-linear equations -and variables. Consider the case with two equations -

    -$$ - \begin{array}{cc} f_1(x_1,x_2) &=0\\ - f_2(x_1,x_2) &=0,\end{array} -$$ - -

    which we Taylor expand to obtain

    - -$$ - \begin{array}{cc} 0=f_1(x_1+h_1,x_2+h_2)=&f_1(x_1,x_2)+h_1 - \partial f_1/\partial x_1+h_2 - \partial f_1/\partial x_2+\dots\\ - 0=f_2(x_1+h_1,x_2+h_2)=&f_2(x_1,x_2)+h_1 - \partial f_2/\partial x_1+h_2 - \partial f_2/\partial x_2+\dots - \end{array}. -$$ - -

    Defining the Jacobian matrix \( {\bf \boldsymbol{J}} \) we have

    -$$ - {\bf \boldsymbol{J}}=\left( \begin{array}{cc} - \partial f_1/\partial x_1 & \partial f_1/\partial x_2 \\ - \partial f_2/\partial x_1 &\partial f_2/\partial x_2 - \end{array} \right), -$$ - -

    we can rephrase Newton's method as

    -$$ -\left(\begin{array}{c} x_1^{n+1} \\ x_2^{n+1} \end{array} \right)= -\left(\begin{array}{c} x_1^{n} \\ x_2^{n} \end{array} \right)+ -\left(\begin{array}{c} h_1^{n} \\ h_2^{n} \end{array} \right), -$$ - -

    where we have defined

    -$$ - \left(\begin{array}{c} h_1^{n} \\ h_2^{n} \end{array} \right)= - -{\bf \boldsymbol{J}}^{-1} - \left(\begin{array}{c} f_1(x_1^{n},x_2^{n}) \\ f_2(x_1^{n},x_2^{n}) \end{array} \right). -$$ - -

    We need thus to compute the inverse of the Jacobian matrix and it -is to understand that difficulties may -arise in case \( {\bf \boldsymbol{J}} \) is nearly singular. +

    The basic idea of gradient descent is +that a function \( F(\mathbf{x}) \), +\( \mathbf{x} \equiv (x_1,\cdots,x_n) \), decreases fastest if one goes from \( \bf {x} \) in the +direction of the negative gradient \( -\nabla F(\mathbf{x}) \).

    -

    It is rather straightforward to extend the above scheme to systems of -more than two non-linear equations. In our case, the Jacobian matrix is given by the Hessian that represents the second derivative of cost function. +

    It can be shown that if

    +$$ +\mathbf{x}_{k+1} = \mathbf{x}_k - \gamma_k \nabla F(\mathbf{x}_k), +$$ + +

    with \( \gamma_k > 0 \).

    + +

    For \( \gamma_k \) small enough, then \( F(\mathbf{x}_{k+1}) \leq +F(\mathbf{x}_k) \). This means that for a sufficiently small \( \gamma_k \) +we are always moving towards smaller function values, i.e a minimum.

    @@ -305,7 +270,6 @@ more than two non-linear equations. In our case, the Jacobian matrix is given by

  • 40
  • 41
  • 42
    • -
  • 43
  • »
    • diff --git a/doc/pub/week38/html/._week38-bs039.html b/doc/pub/week38/html/._week38-bs039.html index 5991ebc73..837d6c637 100644 --- a/doc/pub/week38/html/._week38-bs039.html +++ b/doc/pub/week38/html/._week38-bs039.html @@ -112,7 +112,6 @@ doconce format html week38.do.txt --html_style=bootstrap --pygments_html_style=d 2, None, 'other-measures-in-classification-studies-cancer-data-again'), - ('Friday September 25', 2, None, 'friday-september-25'), ('Optimization, the central part of any Machine Learning ' 'algortithm', 2, @@ -210,19 +209,18 @@ MathJax.Hub.Config({
  • Using the correlation matrix
  • Discussing the correlation data
  • Other measures in classification studies: Cancer Data again
    • -
  • Friday September 25
    • -
  • Optimization, the central part of any Machine Learning algortithm
    • -
  • Revisiting our Logistic Regression case
    • -
  • The equations to solve
    • -
  • Solving using Newton-Raphson's method
    • -
  • Brief reminder on Newton-Raphson's method
    • -
  • The equations
    • -
  • Simple geometric interpretation
    • -
  • Extending to more than one variable
    • -
  • Steepest descent
    • -
  • More on Steepest descent
    • -
  • The ideal
    • -
  • The sensitiveness of the gradient descent
    • +
  • Optimization, the central part of any Machine Learning algortithm
    • +
  • Revisiting our Logistic Regression case
    • +
  • The equations to solve
    • +
  • Solving using Newton-Raphson's method
    • +
  • Brief reminder on Newton-Raphson's method
    • +
  • The equations
    • +
  • Simple geometric interpretation
    • +
  • Extending to more than one variable
    • +
  • Steepest descent
    • +
  • More on Steepest descent
    • +
  • The ideal
    • +
  • The sensitiveness of the gradient descent
    • @@ -233,25 +231,21 @@ MathJax.Hub.Config({

     

     

     

    - -

    Steepest descent

    + +

    More on Steepest descent

    -

    The basic idea of gradient descent is -that a function \( F(\mathbf{x}) \), -\( \mathbf{x} \equiv (x_1,\cdots,x_n) \), decreases fastest if one goes from \( \bf {x} \) in the -direction of the negative gradient \( -\nabla F(\mathbf{x}) \). +

    The previous observation is the basis of the method of steepest +descent, which is also referred to as just gradient descent (GD). One +starts with an initial guess \( \mathbf{x}_0 \) for a minimum of \( F \) and +computes new approximations according to

    -

    It can be shown that if

    $$ -\mathbf{x}_{k+1} = \mathbf{x}_k - \gamma_k \nabla F(\mathbf{x}_k), +\mathbf{x}_{k+1} = \mathbf{x}_k - \gamma_k \nabla F(\mathbf{x}_k), \ \ k \geq 0. $$ -

    with \( \gamma_k > 0 \).

    - -

    For \( \gamma_k \) small enough, then \( F(\mathbf{x}_{k+1}) \leq -F(\mathbf{x}_k) \). This means that for a sufficiently small \( \gamma_k \) -we are always moving towards smaller function values, i.e a minimum. +

    The parameter \( \gamma_k \) is often referred to as the step length or +the learning rate within the context of Machine Learning.

    @@ -271,7 +265,6 @@ we are always moving towards smaller function values, i.e a minimum.

  • 40
  • 41
  • 42
    • -
  • 43
  • »
    • diff --git a/doc/pub/week38/html/._week38-bs040.html b/doc/pub/week38/html/._week38-bs040.html index 64f0d1189..53207ded4 100644 --- a/doc/pub/week38/html/._week38-bs040.html +++ b/doc/pub/week38/html/._week38-bs040.html @@ -112,7 +112,6 @@ doconce format html week38.do.txt --html_style=bootstrap --pygments_html_style=d 2, None, 'other-measures-in-classification-studies-cancer-data-again'), - ('Friday September 25', 2, None, 'friday-september-25'), ('Optimization, the central part of any Machine Learning ' 'algortithm', 2, @@ -210,19 +209,18 @@ MathJax.Hub.Config({
  • Using the correlation matrix
  • Discussing the correlation data
  • Other measures in classification studies: Cancer Data again
    • -
  • Friday September 25
    • -
  • Optimization, the central part of any Machine Learning algortithm
    • -
  • Revisiting our Logistic Regression case
    • -
  • The equations to solve
    • -
  • Solving using Newton-Raphson's method
    • -
  • Brief reminder on Newton-Raphson's method
    • -
  • The equations
    • -
  • Simple geometric interpretation
    • -
  • Extending to more than one variable
    • -
  • Steepest descent
    • -
  • More on Steepest descent
    • -
  • The ideal
    • -
  • The sensitiveness of the gradient descent
    • +
  • Optimization, the central part of any Machine Learning algortithm
    • +
  • Revisiting our Logistic Regression case
    • +
  • The equations to solve
    • +
  • Solving using Newton-Raphson's method
    • +
  • Brief reminder on Newton-Raphson's method
    • +
  • The equations
    • +
  • Simple geometric interpretation
    • +
  • Extending to more than one variable
    • +
  • Steepest descent
    • +
  • More on Steepest descent
    • +
  • The ideal
    • +
  • The sensitiveness of the gradient descent
    • @@ -234,20 +232,27 @@ MathJax.Hub.Config({

     

     

     

    -

    More on Steepest descent

    +

    The ideal

    -

    The previous observation is the basis of the method of steepest -descent, which is also referred to as just gradient descent (GD). One -starts with an initial guess \( \mathbf{x}_0 \) for a minimum of \( F \) and -computes new approximations according to +

    Ideally the sequence \( \{\mathbf{x}_k \}_{k=0} \) converges to a global +minimum of the function \( F \). In general we do not know if we are in a +global or local minimum. In the special case when \( F \) is a convex +function, all local minima are also global minima, so in this case +gradient descent can converge to the global solution. The advantage of +this scheme is that it is conceptually simple and straightforward to +implement. However the method in this form has some severe +limitations:

    -$$ -\mathbf{x}_{k+1} = \mathbf{x}_k - \gamma_k \nabla F(\mathbf{x}_k), \ \ k \geq 0. -$$ +

    In machine learing we are often faced with non-convex high dimensional +cost functions with many local minima. Since GD is deterministic we +will get stuck in a local minimum, if the method converges, unless we +have a very good intial guess. This also implies that the scheme is +sensitive to the chosen initial condition. +

    -

    The parameter \( \gamma_k \) is often referred to as the step length or -the learning rate within the context of Machine Learning. +

    Note that the gradient is a function of \( \mathbf{x} = +(x_1,\cdots,x_n) \) which makes it expensive to compute numerically.

    @@ -266,7 +271,6 @@ the learning rate within the context of Machine Learning.

  • 40
  • 41
  • 42
    • -
  • 43
  • »
    • diff --git a/doc/pub/week38/html/._week38-bs041.html b/doc/pub/week38/html/._week38-bs041.html index 89dfdd787..b4098c011 100644 --- a/doc/pub/week38/html/._week38-bs041.html +++ b/doc/pub/week38/html/._week38-bs041.html @@ -112,7 +112,6 @@ doconce format html week38.do.txt --html_style=bootstrap --pygments_html_style=d 2, None, 'other-measures-in-classification-studies-cancer-data-again'), - ('Friday September 25', 2, None, 'friday-september-25'), ('Optimization, the central part of any Machine Learning ' 'algortithm', 2, @@ -210,19 +209,18 @@ MathJax.Hub.Config({
  • Using the correlation matrix
  • Discussing the correlation data
  • Other measures in classification studies: Cancer Data again
    • -
  • Friday September 25
    • -
  • Optimization, the central part of any Machine Learning algortithm
    • -
  • Revisiting our Logistic Regression case
    • -
  • The equations to solve
    • -
  • Solving using Newton-Raphson's method
    • -
  • Brief reminder on Newton-Raphson's method
    • -
  • The equations
    • -
  • Simple geometric interpretation
    • -
  • Extending to more than one variable
    • -
  • Steepest descent
    • -
  • More on Steepest descent
    • -
  • The ideal
    • -
  • The sensitiveness of the gradient descent
    • +
  • Optimization, the central part of any Machine Learning algortithm
    • +
  • Revisiting our Logistic Regression case
    • +
  • The equations to solve
    • +
  • Solving using Newton-Raphson's method
    • +
  • Brief reminder on Newton-Raphson's method
    • +
  • The equations
    • +
  • Simple geometric interpretation
    • +
  • Extending to more than one variable
    • +
  • Steepest descent
    • +
  • More on Steepest descent
    • +
  • The ideal
    • +
  • The sensitiveness of the gradient descent
    • @@ -234,27 +232,20 @@ MathJax.Hub.Config({

     

     

     

    -

    The ideal

    +

    The sensitiveness of the gradient descent

    -

    Ideally the sequence \( \{\mathbf{x}_k \}_{k=0} \) converges to a global -minimum of the function \( F \). In general we do not know if we are in a -global or local minimum. In the special case when \( F \) is a convex -function, all local minima are also global minima, so in this case -gradient descent can converge to the global solution. The advantage of -this scheme is that it is conceptually simple and straightforward to -implement. However the method in this form has some severe -limitations: +

    The gradient descent method +is sensitive to the choice of learning rate \( \gamma_k \). This is due +to the fact that we are only guaranteed that \( F(\mathbf{x}_{k+1}) \leq +F(\mathbf{x}_k) \) for sufficiently small \( \gamma_k \). The problem is to +determine an optimal learning rate. If the learning rate is chosen too +small the method will take a long time to converge and if it is too +large we can experience erratic behavior.

    -

    In machine learing we are often faced with non-convex high dimensional -cost functions with many local minima. Since GD is deterministic we -will get stuck in a local minimum, if the method converges, unless we -have a very good intial guess. This also implies that the scheme is -sensitive to the chosen initial condition. -

    - -

    Note that the gradient is a function of \( \mathbf{x} = -(x_1,\cdots,x_n) \) which makes it expensive to compute numerically. +

    Many of these shortcomings can be alleviated by introducing +randomness. One such method is that of Stochastic Gradient Descent +(SGD), see below.

    @@ -272,8 +263,6 @@ sensitive to the chosen initial condition.

  • 40
  • 41
  • 42
    • -
  • 43
    • -
  • »
    • diff --git a/doc/pub/week38/html/week38-bs.html b/doc/pub/week38/html/week38-bs.html index 4888028fe..72a3d698d 100644 --- a/doc/pub/week38/html/week38-bs.html +++ b/doc/pub/week38/html/week38-bs.html @@ -112,7 +112,6 @@ doconce format html week38.do.txt --html_style=bootstrap --pygments_html_style=d 2, None, 'other-measures-in-classification-studies-cancer-data-again'), - ('Friday September 25', 2, None, 'friday-september-25'), ('Optimization, the central part of any Machine Learning ' 'algortithm', 2, @@ -210,19 +209,18 @@ MathJax.Hub.Config({
  • Using the correlation matrix
  • Discussing the correlation data
  • Other measures in classification studies: Cancer Data again
    • -
  • Friday September 25
    • -
  • Optimization, the central part of any Machine Learning algortithm
    • -
  • Revisiting our Logistic Regression case
    • -
  • The equations to solve
    • -
  • Solving using Newton-Raphson's method
    • -
  • Brief reminder on Newton-Raphson's method
    • -
  • The equations
    • -
  • Simple geometric interpretation
    • -
  • Extending to more than one variable
    • -
  • Steepest descent
    • -
  • More on Steepest descent
    • -
  • The ideal
    • -
  • The sensitiveness of the gradient descent
    • +
  • Optimization, the central part of any Machine Learning algortithm
    • +
  • Revisiting our Logistic Regression case
    • +
  • The equations to solve
    • +
  • Solving using Newton-Raphson's method
    • +
  • Brief reminder on Newton-Raphson's method
    • +
  • The equations
    • +
  • Simple geometric interpretation
    • +
  • Extending to more than one variable
    • +
  • Steepest descent
    • +
  • More on Steepest descent
    • +
  • The ideal
    • +
  • The sensitiveness of the gradient descent
    • @@ -277,7 +275,7 @@ MathJax.Hub.Config({
  • 9
  • 10
  • ...
    • -
  • 43
    • +
  • 42
  • »
    • diff --git a/doc/pub/week38/html/week38-reveal.html b/doc/pub/week38/html/week38-reveal.html index 99995e895..42a926b75 100644 --- a/doc/pub/week38/html/week38-reveal.html +++ b/doc/pub/week38/html/week38-reveal.html @@ -199,7 +199,7 @@ MathJax.Hub.Config({