lars/FYS-STK4155
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
7f918abf673d3869916b2ec7ca3fd15fac1fbebb
FYS-STK4155/doc/LectureNotes/_build/html/week39.html
T
Morten Hjorth-Jensen 7f918abf67 update book
2025-09-29 07:26:11 +02:00

2044 lines
112 KiB
HTML
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
<!DOCTYPE html>
<html lang="en" data-content_root="./" >
<head>
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" /><meta name="viewport" content="width=device-width, initial-scale=1" />
<title>Week 39: Resampling methods and logistic regression &#8212; Applied Data Analysis and Machine Learning</title>
<script data-cfasync="false">
document.documentElement.dataset.mode = localStorage.getItem("mode") || "";
document.documentElement.dataset.theme = localStorage.getItem("theme") || "";
</script>
<!-- Loaded before other Sphinx assets -->
<link href="_static/styles/theme.css?digest=dfe6caa3a7d634c4db9b" rel="stylesheet" />
<link href="_static/styles/bootstrap.css?digest=dfe6caa3a7d634c4db9b" rel="stylesheet" />
<link href="_static/styles/pydata-sphinx-theme.css?digest=dfe6caa3a7d634c4db9b" rel="stylesheet" />
<link href="_static/vendor/fontawesome/6.5.2/css/all.min.css?digest=dfe6caa3a7d634c4db9b" rel="stylesheet" />
<link rel="preload" as="font" type="font/woff2" crossorigin href="_static/vendor/fontawesome/6.5.2/webfonts/fa-solid-900.woff2" />
<link rel="preload" as="font" type="font/woff2" crossorigin href="_static/vendor/fontawesome/6.5.2/webfonts/fa-brands-400.woff2" />
<link rel="preload" as="font" type="font/woff2" crossorigin href="_static/vendor/fontawesome/6.5.2/webfonts/fa-regular-400.woff2" />
<link rel="stylesheet" type="text/css" href="_static/pygments.css?v=fa44fd50" />
<link rel="stylesheet" type="text/css" href="_static/styles/sphinx-book-theme.css?v=eba8b062" />
<link rel="stylesheet" type="text/css" href="_static/togglebutton.css?v=13237357" />
<link rel="stylesheet" type="text/css" href="_static/copybutton.css?v=76b2166b" />
<link rel="stylesheet" type="text/css" href="_static/mystnb.8ecb98da25f57f5357bf6f572d296f466b2cfe2517ffebfabe82451661e28f02.css?v=6644e6bb" />
<link rel="stylesheet" type="text/css" href="_static/sphinx-thebe.css?v=4fa983c6" />
<link rel="stylesheet" type="text/css" href="_static/sphinx-design.min.css?v=95c83b7e" />
<!-- Pre-loaded scripts that we'll load fully later -->
<link rel="preload" as="script" href="_static/scripts/bootstrap.js?digest=dfe6caa3a7d634c4db9b" />
<link rel="preload" as="script" href="_static/scripts/pydata-sphinx-theme.js?digest=dfe6caa3a7d634c4db9b" />
<script src="_static/vendor/fontawesome/6.5.2/js/all.min.js?digest=dfe6caa3a7d634c4db9b"></script>
<script src="_static/documentation_options.js?v=9eb32ce0"></script>
<script src="_static/doctools.js?v=9a2dae69"></script>
<script src="_static/sphinx_highlight.js?v=dc90522c"></script>
<script src="_static/clipboard.min.js?v=a7894cd8"></script>
<script src="_static/copybutton.js?v=f281be69"></script>
<script src="_static/scripts/sphinx-book-theme.js?v=887ef09a"></script>
<script>let toggleHintShow = 'Click to show';</script>
<script>let toggleHintHide = 'Click to hide';</script>
<script>let toggleOpenOnPrint = 'true';</script>
<script src="_static/togglebutton.js?v=4a39c7ea"></script>
<script>var togglebuttonSelector = '.toggle, .admonition.dropdown';</script>
<script src="_static/design-tabs.js?v=f930bc37"></script>
<script>const THEBE_JS_URL = "https://unpkg.com/thebe@0.8.2/lib/index.js"; const thebe_selector = ".thebe,.cell"; const thebe_selector_input = "pre"; const thebe_selector_output = ".output, .cell_output"</script>
<script async="async" src="_static/sphinx-thebe.js?v=c100c467"></script>
<script>var togglebuttonSelector = '.toggle, .admonition.dropdown';</script>
<script>const THEBE_JS_URL = "https://unpkg.com/thebe@0.8.2/lib/index.js"; const thebe_selector = ".thebe,.cell"; const thebe_selector_input = "pre"; const thebe_selector_output = ".output, .cell_output"</script>
<script>window.MathJax = {"options": {"processHtmlClass": "tex2jax_process|mathjax_process|math|output_area"}}</script>
<script defer="defer" src="https://cdn.jsdelivr.net/npm/mathjax@3/es5/tex-mml-chtml.js"></script>
<script>DOCUMENTATION_OPTIONS.pagename = 'week39';</script>
<link rel="index" title="Index" href="genindex.html" />
<link rel="search" title="Search" href="search.html" />
<link rel="next" title="Week 40: Gradient descent methods (continued) and start Neural networks" href="week40.html" />
<link rel="prev" title="Exercises week 39" href="exercisesweek39.html" />
<meta name="viewport" content="width=device-width, initial-scale=1"/>
<meta name="docsearch:language" content="en"/>
</head>
<body data-bs-spy="scroll" data-bs-target=".bd-toc-nav" data-offset="180" data-bs-root-margin="0px 0px -60%" data-default-mode="">
<div id="pst-skip-link" class="skip-link d-print-none"><a href="#main-content">Skip to main content</a></div>
<div id="pst-scroll-pixel-helper"></div>
<button type="button" class="btn rounded-pill" id="pst-back-to-top">
<i class="fa-solid fa-arrow-up"></i>Back to top</button>
<input type="checkbox"
class="sidebar-toggle"
id="pst-primary-sidebar-checkbox"/>
<label class="overlay overlay-primary" for="pst-primary-sidebar-checkbox"></label>
<input type="checkbox"
class="sidebar-toggle"
id="pst-secondary-sidebar-checkbox"/>
<label class="overlay overlay-secondary" for="pst-secondary-sidebar-checkbox"></label>
<div class="search-button__wrapper">
<div class="search-button__overlay"></div>
<div class="search-button__search-container">
<form class="bd-search d-flex align-items-center"
action="search.html"
method="get">
<i class="fa-solid fa-magnifying-glass"></i>
<input type="search"
class="form-control"
name="q"
id="search-input"
placeholder="Search this book..."
aria-label="Search this book..."
autocomplete="off"
autocorrect="off"
autocapitalize="off"
spellcheck="false"/>
<span class="search-button__kbd-shortcut"><kbd class="kbd-shortcut__modifier">Ctrl</kbd>+<kbd>K</kbd></span>
</form></div>
</div>
<div class="pst-async-banner-revealer d-none">
<aside id="bd-header-version-warning" class="d-none d-print-none" aria-label="Version warning"></aside>
</div>
<header class="bd-header navbar navbar-expand-lg bd-navbar d-print-none">
</header>
<div class="bd-container">
<div class="bd-container__inner bd-page-width">
<div class="bd-sidebar-primary bd-sidebar">
<div class="sidebar-header-items sidebar-primary__section">
</div>
<div class="sidebar-primary-items__start sidebar-primary__section">
<div class="sidebar-primary-item">
<a class="navbar-brand logo" href="intro.html">
<img src="_static/logo.png" class="logo__image only-light" alt="Applied Data Analysis and Machine Learning - Home"/>
<script>document.write(`<img src="_static/logo.png" class="logo__image only-dark" alt="Applied Data Analysis and Machine Learning - Home"/>`);</script>
</a></div>
<div class="sidebar-primary-item">
<script>
document.write(`
<button class="btn search-button-field search-button__button" title="Search" aria-label="Search" data-bs-placement="bottom" data-bs-toggle="tooltip">
<i class="fa-solid fa-magnifying-glass"></i>
<span class="search-button__default-text">Search</span>
<span class="search-button__kbd-shortcut"><kbd class="kbd-shortcut__modifier">Ctrl</kbd>+<kbd class="kbd-shortcut__modifier">K</kbd></span>
</button>
`);
</script></div>
<div class="sidebar-primary-item"><nav class="bd-links bd-docs-nav" aria-label="Main">
<div class="bd-toc-item navbar-nav active">
<ul class="nav bd-sidenav bd-sidenav__home-link">
<li class="toctree-l1">
<a class="reference internal" href="intro.html">
Applied Data Analysis and Machine Learning
</a>
</li>
</ul>
<p aria-level="2" class="caption" role="heading"><span class="caption-text">About the course</span></p>
<ul class="nav bd-sidenav">
<li class="toctree-l1"><a class="reference internal" href="schedule.html">Course setting</a></li>
<li class="toctree-l1"><a class="reference internal" href="teachers.html">Teachers and Grading</a></li>
<li class="toctree-l1"><a class="reference internal" href="textbooks.html">Textbooks</a></li>
</ul>
<p aria-level="2" class="caption" role="heading"><span class="caption-text">Review of Statistics with Resampling Techniques and Linear Algebra</span></p>
<ul class="nav bd-sidenav">
<li class="toctree-l1"><a class="reference internal" href="statistics.html">1. Elements of Probability Theory and Statistical Data Analysis</a></li>
<li class="toctree-l1"><a class="reference internal" href="linalg.html">2. Linear Algebra, Handling of Arrays and more Python Features</a></li>
</ul>
<p aria-level="2" class="caption" role="heading"><span class="caption-text">From Regression to Support Vector Machines</span></p>
<ul class="nav bd-sidenav">
<li class="toctree-l1"><a class="reference internal" href="chapter1.html">3. Linear Regression</a></li>
<li class="toctree-l1"><a class="reference internal" href="chapter2.html">4. Ridge and Lasso Regression</a></li>
<li class="toctree-l1"><a class="reference internal" href="chapter3.html">5. Resampling Methods</a></li>
<li class="toctree-l1"><a class="reference internal" href="chapter4.html">6. Logistic Regression</a></li>
<li class="toctree-l1"><a class="reference internal" href="chapteroptimization.html">7. Optimization, the central part of any Machine Learning algortithm</a></li>
<li class="toctree-l1"><a class="reference internal" href="chapter5.html">8. Support Vector Machines, overarching aims</a></li>
</ul>
<p aria-level="2" class="caption" role="heading"><span class="caption-text">Decision Trees, Ensemble Methods and Boosting</span></p>
<ul class="nav bd-sidenav">
<li class="toctree-l1"><a class="reference internal" href="chapter6.html">9. Decision trees, overarching aims</a></li>
<li class="toctree-l1"><a class="reference internal" href="chapter7.html">10. Ensemble Methods: From a Single Tree to Many Trees and Extreme Boosting, Meet the Jungle of Methods</a></li>
</ul>
<p aria-level="2" class="caption" role="heading"><span class="caption-text">Dimensionality Reduction</span></p>
<ul class="nav bd-sidenav">
<li class="toctree-l1"><a class="reference internal" href="chapter8.html">11. Basic ideas of the Principal Component Analysis (PCA)</a></li>
<li class="toctree-l1"><a class="reference internal" href="clustering.html">12. Clustering and Unsupervised Learning</a></li>
</ul>
<p aria-level="2" class="caption" role="heading"><span class="caption-text">Deep Learning Methods</span></p>
<ul class="nav bd-sidenav">
<li class="toctree-l1"><a class="reference internal" href="chapter9.html">13. Neural networks</a></li>
<li class="toctree-l1"><a class="reference internal" href="chapter10.html">14. Building a Feed Forward Neural Network</a></li>
<li class="toctree-l1"><a class="reference internal" href="chapter11.html">15. Solving Differential Equations with Deep Learning</a></li>
<li class="toctree-l1"><a class="reference internal" href="chapter12.html">16. Convolutional Neural Networks</a></li>
<li class="toctree-l1"><a class="reference internal" href="chapter13.html">17. Recurrent neural networks: Overarching view</a></li>
</ul>
<p aria-level="2" class="caption" role="heading"><span class="caption-text">Weekly material, notes and exercises</span></p>
<ul class="current nav bd-sidenav">
<li class="toctree-l1"><a class="reference internal" href="exercisesweek34.html">Exercises week 34</a></li>
<li class="toctree-l1"><a class="reference internal" href="week34.html">Week 34: Introduction to the course, Logistics and Practicalities</a></li>
<li class="toctree-l1"><a class="reference internal" href="exercisesweek35.html">Exercises week 35</a></li>
<li class="toctree-l1"><a class="reference internal" href="week35.html">Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression</a></li>
<li class="toctree-l1"><a class="reference internal" href="exercisesweek36.html">Exercises week 36</a></li>
<li class="toctree-l1"><a class="reference internal" href="week36.html">Week 36: Linear Regression and Gradient descent</a></li>
<li class="toctree-l1"><a class="reference internal" href="exercisesweek37.html">Exercises week 37</a></li>
<li class="toctree-l1"><a class="reference internal" href="week37.html">Week 37: Gradient descent methods</a></li>
<li class="toctree-l1"><a class="reference internal" href="exercisesweek38.html">Exercises week 38</a></li>
<li class="toctree-l1"><a class="reference internal" href="week38.html">Week 38: Statistical analysis, bias-variance tradeoff and resampling methods</a></li>
<li class="toctree-l1"><a class="reference internal" href="exercisesweek39.html">Exercises week 39</a></li>
<li class="toctree-l1 current active"><a class="current reference internal" href="#">Week 39: Resampling methods and logistic regression</a></li>
<li class="toctree-l1"><a class="reference internal" href="week40.html">Week 40: Gradient descent methods (continued) and start Neural networks</a></li>
</ul>
<p aria-level="2" class="caption" role="heading"><span class="caption-text">Projects</span></p>
<ul class="nav bd-sidenav">
<li class="toctree-l1"><a class="reference internal" href="project1.html">Project 1 on Machine Learning, deadline October 6 (midnight), 2025</a></li>
</ul>
</div>
</nav></div>
</div>
<div class="sidebar-primary-items__end sidebar-primary__section">
</div>
<div id="rtd-footer-container"></div>
</div>
<main id="main-content" class="bd-main" role="main">
<div class="sbt-scroll-pixel-helper"></div>
<div class="bd-content">
<div class="bd-article-container">
<div class="bd-header-article d-print-none">
<div class="header-article-items header-article__inner">
<div class="header-article-items__start">
<div class="header-article-item"><button class="sidebar-toggle primary-toggle btn btn-sm" title="Toggle primary sidebar" data-bs-placement="bottom" data-bs-toggle="tooltip">
<span class="fa-solid fa-bars"></span>
</button></div>
</div>
<div class="header-article-items__end">
<div class="header-article-item">
<div class="article-header-buttons">
<div class="dropdown dropdown-download-buttons">
<button class="btn dropdown-toggle" type="button" data-bs-toggle="dropdown" aria-expanded="false" aria-label="Download this page">
<i class="fas fa-download"></i>
</button>
<ul class="dropdown-menu">
<li><a href="_sources/week39.ipynb" target="_blank"
class="btn btn-sm btn-download-source-button dropdown-item"
title="Download source file"
data-bs-placement="left" data-bs-toggle="tooltip"
>
<span class="btn__icon-container">
<i class="fas fa-file"></i>
</span>
<span class="btn__text-container">.ipynb</span>
</a>
</li>
<li>
<button onclick="window.print()"
class="btn btn-sm btn-download-pdf-button dropdown-item"
title="Print to PDF"
data-bs-placement="left" data-bs-toggle="tooltip"
>
<span class="btn__icon-container">
<i class="fas fa-file-pdf"></i>
</span>
<span class="btn__text-container">.pdf</span>
</button>
</li>
</ul>
</div>
<button onclick="toggleFullScreen()"
class="btn btn-sm btn-fullscreen-button"
title="Fullscreen mode"
data-bs-placement="bottom" data-bs-toggle="tooltip"
>
<span class="btn__icon-container">
<i class="fas fa-expand"></i>
</span>
</button>
<script>
document.write(`
<button class="btn btn-sm nav-link pst-navbar-icon theme-switch-button" title="light/dark" aria-label="light/dark" data-bs-placement="bottom" data-bs-toggle="tooltip">
<i class="theme-switch fa-solid fa-sun fa-lg" data-mode="light"></i>
<i class="theme-switch fa-solid fa-moon fa-lg" data-mode="dark"></i>
<i class="theme-switch fa-solid fa-circle-half-stroke fa-lg" data-mode="auto"></i>
</button>
`);
</script>
<script>
document.write(`
<button class="btn btn-sm pst-navbar-icon search-button search-button__button" title="Search" aria-label="Search" data-bs-placement="bottom" data-bs-toggle="tooltip">
<i class="fa-solid fa-magnifying-glass fa-lg"></i>
</button>
`);
</script>
<button class="sidebar-toggle secondary-toggle btn btn-sm" title="Toggle secondary sidebar" data-bs-placement="bottom" data-bs-toggle="tooltip">
<span class="fa-solid fa-list"></span>
</button>
</div></div>
</div>
</div>
</div>
<div id="jb-print-docs-body" class="onlyprint">
<h1>Week 39: Resampling methods and logistic regression</h1>
<!-- Table of contents -->
<div id="print-main-content">
<div id="jb-print-toc">
<div>
<h2> Contents </h2>
</div>
<nav aria-label="Page">
<ul class="visible nav section-nav flex-column">
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#plan-for-week-39-september-22-26-2025">Plan for week 39, September 22-26, 2025</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#readings-and-videos-resampling-methods">Readings and Videos, resampling methods</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#readings-and-videos-logistic-regression">Readings and Videos, logistic regression</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#lab-sessions-week-39">Lab sessions week 39</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#lecture-material">Lecture material</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#resampling-methods">Resampling methods</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#resampling-approaches-can-be-computationally-expensive">Resampling approaches can be computationally expensive</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#why-resampling-methods">Why resampling methods ?</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#statistical-analysis">Statistical analysis</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#id1">Resampling methods</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#resampling-methods-bootstrap">Resampling methods: Bootstrap</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#the-bias-variance-tradeoff">The bias-variance tradeoff</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#a-way-to-read-the-bias-variance-tradeoff">A way to Read the Bias-Variance Tradeoff</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#understanding-what-happens">Understanding what happens</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#summing-up">Summing up</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#another-example-from-scikit-learn-s-repository">Another Example from Scikit-Learn’s Repository</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#various-steps-in-cross-validation">Various steps in cross-validation</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#cross-validation-in-brief">Cross-validation in brief</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#code-example-for-cross-validation-and-k-fold-cross-validation">Code Example for Cross-validation and <span class="math notranslate nohighlight">\(k\)</span>-fold Cross-validation</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#more-examples-on-bootstrap-and-cross-validation-and-errors">More examples on bootstrap and cross-validation and errors</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#the-same-example-but-now-with-cross-validation">The same example but now with cross-validation</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#logistic-regression">Logistic Regression</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#classification-problems">Classification problems</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#optimization-and-deep-learning">Optimization and Deep learning</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#basics">Basics</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#linear-classifier">Linear classifier</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#some-selected-properties">Some selected properties</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#simple-example">Simple example</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#plotting-the-mean-value-for-each-group">Plotting the mean value for each group</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#the-logistic-function">The logistic function</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#examples-of-likelihood-functions-used-in-logistic-regression-and-nueral-networks">Examples of likelihood functions used in logistic regression and nueral networks</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#two-parameters">Two parameters</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#maximum-likelihood">Maximum likelihood</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#the-cost-function-rewritten">The cost function rewritten</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#minimizing-the-cross-entropy">Minimizing the cross entropy</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#a-more-compact-expression">A more compact expression</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#extending-to-more-predictors">Extending to more predictors</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#including-more-classes">Including more classes</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#more-classes">More classes</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#optimization-the-central-part-of-any-machine-learning-algortithm">Optimization, the central part of any Machine Learning algortithm</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#revisiting-our-logistic-regression-case">Revisiting our Logistic Regression case</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#the-equations-to-solve">The equations to solve</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#solving-using-newton-raphson-s-method">Solving using Newton-Raphson’s method</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#example-code-for-logistic-regression">Example code for Logistic Regression</a><ul class="nav section-nav flex-column">
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#synthetic-data-generation">Synthetic data generation</a></li>
</ul>
</li>
</ul>
</nav>
</div>
</div>
</div>
<div id="searchbox"></div>
<article class="bd-article">
<!-- HTML file automatically generated from DocOnce source (https://github.com/doconce/doconce/)
doconce format html week39.do.txt --no_mako -->
<!-- dom:TITLE: Week 39: Resampling methods and logistic regression --><section class="tex2jax_ignore mathjax_ignore" id="week-39-resampling-methods-and-logistic-regression">
<h1>Week 39: Resampling methods and logistic regression<a class="headerlink" href="#week-39-resampling-methods-and-logistic-regression" title="Link to this heading">#</a></h1>
<p><strong>Morten Hjorth-Jensen</strong>, Department of Physics, University of Oslo</p>
<p>Date: <strong>Week 39</strong></p>
<section id="plan-for-week-39-september-22-26-2025">
<h2>Plan for week 39, September 22-26, 2025<a class="headerlink" href="#plan-for-week-39-september-22-26-2025" title="Link to this heading">#</a></h2>
<p><strong>Material for the lecture on Monday September 22.</strong></p>
<ol class="arabic simple">
<li><p>Resampling techniques, Bootstrap and cross validation and bias-variance tradeoff</p></li>
<li><p>Logistic regression, our first classification encounter and a stepping stone towards neural networks</p></li>
<li><p><a class="reference external" href="https://youtu.be/OVouJyhoksY">Video of lecture</a></p></li>
<li><p><a class="reference external" href="https://github.com/CompPhysics/MachineLearning/blob/master/doc/HandWrittenNotes/2024/FYSSTKweek39.pdf">Whiteboard notes</a></p></li>
</ol>
</section>
<section id="readings-and-videos-resampling-methods">
<h2>Readings and Videos, resampling methods<a class="headerlink" href="#readings-and-videos-resampling-methods" title="Link to this heading">#</a></h2>
<ol class="arabic simple">
<li><p>Raschka et al, pages 175-192</p></li>
<li><p>Hastie et al Chapter 7, here we recommend 7.1-7.5 and 7.10 (cross-validation) and 7.11 (bootstrap). See <a class="reference external" href="https://link.springer.com/book/10.1007/978-0-387-84858-7">https://link.springer.com/book/10.1007/978-0-387-84858-7</a>.</p></li>
<li><p><a class="reference external" href="https://www.youtube.com/watch?v=EuBBz3bI-aA">Video on bias-variance tradeoff</a></p></li>
<li><p><a class="reference external" href="https://www.youtube.com/watch?v=Xz0x-8-cgaQ">Video on Bootstrapping</a></p></li>
<li><p><a class="reference external" href="https://www.youtube.com/watch?v=fSytzGwwBVw">Video on cross validation</a></p></li>
</ol>
</section>
<section id="readings-and-videos-logistic-regression">
<h2>Readings and Videos, logistic regression<a class="headerlink" href="#readings-and-videos-logistic-regression" title="Link to this heading">#</a></h2>
<ol class="arabic simple">
<li><p>Hastie et al 4.1, 4.2 and 4.3 on logistic regression</p></li>
<li><p>Raschka et al, pages 53-76 on Logistic regression and pages 37-52 on gradient optimization</p></li>
<li><p><a class="reference external" href="https://www.youtube.com/watch?v=C5268D9t9Ak">Video on Logistic regression</a></p></li>
<li><p><a class="reference external" href="https://www.youtube.com/watch?v=yIYKR4sgzI8">Yet another video on logistic regression</a></p></li>
</ol>
</section>
<section id="lab-sessions-week-39">
<h2>Lab sessions week 39<a class="headerlink" href="#lab-sessions-week-39" title="Link to this heading">#</a></h2>
<p><strong>Material for the lab sessions on Tuesday and Wednesday.</strong></p>
<ol class="arabic simple">
<li><p>Discussions on how to structure your report for the first project</p></li>
<li><p>Exercise for week 39 on how to write the abstract and the introduction of the report and how to include references.</p></li>
<li><p>Work on project 1, in particular resampling methods like cross-validation and bootstrap. <strong>For more discussions of project 1, chapter 5 of Goodfellow et al is a good read, in particular sections 5.1-5.5 and 5.7-5.11</strong>.</p></li>
<li><p><a class="reference external" href="https://youtu.be/tVW1ZDmZnwM">Video on how to write scientific reports recorded during one of the lab sessions</a></p></li>
<li><p>A general guideline can be found at <a class="github reference external" href="https://github.com/CompPhysics/MachineLearning/blob/master/doc/Projects/EvaluationGrading/EvaluationForm.md">CompPhysics/MachineLearning</a>.</p></li>
</ol>
</section>
<section id="lecture-material">
<h2>Lecture material<a class="headerlink" href="#lecture-material" title="Link to this heading">#</a></h2>
</section>
<section id="resampling-methods">
<h2>Resampling methods<a class="headerlink" href="#resampling-methods" title="Link to this heading">#</a></h2>
<p>Resampling methods are an indispensable tool in modern
statistics. They involve repeatedly drawing samples from a training
set and refitting a model of interest on each sample in order to
obtain additional information about the fitted model. For example, in
order to estimate the variability of a linear regression fit, we can
repeatedly draw different samples from the training data, fit a linear
regression to each new sample, and then examine the extent to which
the resulting fits differ. Such an approach may allow us to obtain
information that would not be available from fitting the model only
once using the original training sample.</p>
<p>Two resampling methods are often used in Machine Learning analyses,</p>
<ol class="arabic simple">
<li><p>The <strong>bootstrap method</strong></p></li>
<li><p>and <strong>Cross-Validation</strong></p></li>
</ol>
<p>In addition there are several other methods such as the Jackknife and the Blocking methods. This week will repeat some of the elements of the bootstrap method and focus more on cross-validation.</p>
</section>
<section id="resampling-approaches-can-be-computationally-expensive">
<h2>Resampling approaches can be computationally expensive<a class="headerlink" href="#resampling-approaches-can-be-computationally-expensive" title="Link to this heading">#</a></h2>
<p>Resampling approaches can be computationally expensive, because they
involve fitting the same statistical method multiple times using
different subsets of the training data. However, due to recent
advances in computing power, the computational requirements of
resampling methods generally are not prohibitive. In this chapter, we
discuss two of the most commonly used resampling methods,
cross-validation and the bootstrap. Both methods are important tools
in the practical application of many statistical learning
procedures. For example, cross-validation can be used to estimate the
test error associated with a given statistical learning method in
order to evaluate its performance, or to select the appropriate level
of flexibility. The process of evaluating a model’s performance is
known as model assessment, whereas the process of selecting the proper
level of flexibility for a model is known as model selection. The
bootstrap is widely used.</p>
</section>
<section id="why-resampling-methods">
<h2>Why resampling methods ?<a class="headerlink" href="#why-resampling-methods" title="Link to this heading">#</a></h2>
<p><strong>Statistical analysis.</strong></p>
<ul class="simple">
<li><p>Our simulations can be treated as <em>computer experiments</em>. This is particularly the case for Monte Carlo methods which are widely used in statistical analyses.</p></li>
<li><p>The results can be analysed with the same statistical tools as we would use when analysing experimental data.</p></li>
<li><p>As in all experiments, we are looking for expectation values and an estimate of how accurate they are, i.e., possible sources for errors.</p></li>
</ul>
</section>
<section id="statistical-analysis">
<h2>Statistical analysis<a class="headerlink" href="#statistical-analysis" title="Link to this heading">#</a></h2>
<ul class="simple">
<li><p>As in other experiments, many numerical experiments have two classes of errors:</p>
<ul>
<li><p>Statistical errors</p></li>
<li><p>Systematical errors</p></li>
</ul>
</li>
<li><p>Statistical errors can be estimated using standard tools from statistics</p></li>
<li><p>Systematical errors are method specific and must be treated differently from case to case.</p></li>
</ul>
</section>
<section id="id1">
<h2>Resampling methods<a class="headerlink" href="#id1" title="Link to this heading">#</a></h2>
<p>With all these analytical equations for both the OLS and Ridge
regression, we will now outline how to assess a given model. This will
lead to a discussion of the so-called bias-variance tradeoff (see
below) and so-called resampling methods.</p>
<p>One of the quantities we have discussed as a way to measure errors is
the mean-squared error (MSE), mainly used for fitting of continuous
functions. Another choice is the absolute error.</p>
<p>In the discussions below we will focus on the MSE and in particular since we will split the data into test and training data,
we discuss the</p>
<ol class="arabic simple">
<li><p>prediction error or simply the <strong>test error</strong> <span class="math notranslate nohighlight">\(\mathrm{Err_{Test}}\)</span>, where we have a fixed training set and the test error is the MSE arising from the data reserved for testing. We discuss also the</p></li>
<li><p>training error <span class="math notranslate nohighlight">\(\mathrm{Err_{Train}}\)</span>, which is the average loss over the training data.</p></li>
</ol>
<p>As our model becomes more and more complex, more of the training data tends to used. The training may thence adapt to more complicated structures in the data. This may lead to a decrease in the bias (see below for code example) and a slight increase of the variance for the test error.
For a certain level of complexity the test error will reach minimum, before starting to increase again. The
training error reaches a saturation.</p>
</section>
<section id="resampling-methods-bootstrap">
<h2>Resampling methods: Bootstrap<a class="headerlink" href="#resampling-methods-bootstrap" title="Link to this heading">#</a></h2>
<p>Bootstrapping is a <a class="reference external" href="https://en.wikipedia.org/wiki/Nonparametric_statistics">non-parametric approach</a> to statistical inference
that substitutes computation for more traditional distributional
assumptions and asymptotic results. Bootstrapping offers a number of
advantages:</p>
<ol class="arabic simple">
<li><p>The bootstrap is quite general, although there are some cases in which it fails.</p></li>
<li><p>Because it does not require distributional assumptions (such as normally distributed errors), the bootstrap can provide more accurate inferences when the data are not well behaved or when the sample size is small.</p></li>
<li><p>It is possible to apply the bootstrap to statistics with sampling distributions that are difficult to derive, even asymptotically.</p></li>
<li><p>It is relatively simple to apply the bootstrap to complex data-collection plans (such as stratified and clustered samples).</p></li>
</ol>
<p>The textbook by <a class="reference external" href="https://www.cambridge.org/core/books/bootstrap-methods-and-their-application/ED2FD043579F27952363566DC09CBD6A">Davison on the Bootstrap Methods and their Applications</a> provides many more insights and proofs. In this course we will take a more practical approach and use the results and theorems provided in the literature. For those interested in reading more about the bootstrap methods, we recommend the above text and the one by <a class="reference external" href="https://www.routledge.com/An-Introduction-to-the-Bootstrap/Efron-Tibshirani/p/book/9780412042317">Efron and Tibshirani</a>.</p>
</section>
<section id="the-bias-variance-tradeoff">
<h2>The bias-variance tradeoff<a class="headerlink" href="#the-bias-variance-tradeoff" title="Link to this heading">#</a></h2>
<p>We will discuss the bias-variance tradeoff in the context of
continuous predictions such as regression. However, many of the
intuitions and ideas discussed here also carry over to classification
tasks. Consider a dataset <span class="math notranslate nohighlight">\(\mathcal{D}\)</span> consisting of the data
<span class="math notranslate nohighlight">\(\mathbf{X}_\mathcal{D}=\{(y_j, \boldsymbol{x}_j), j=0\ldots n-1\}\)</span>.</p>
<p>Let us assume that the true data is generated from a noisy model</p>
<div class="math notranslate nohighlight">
\[
\boldsymbol{y}=f(\boldsymbol{x}) + \boldsymbol{\epsilon}
\]</div>
<p>where <span class="math notranslate nohighlight">\(\epsilon\)</span> is normally distributed with mean zero and standard deviation <span class="math notranslate nohighlight">\(\sigma^2\)</span>.</p>
<p>In our derivation of the ordinary least squares method we defined then
an approximation to the function <span class="math notranslate nohighlight">\(f\)</span> in terms of the parameters
<span class="math notranslate nohighlight">\(\boldsymbol{\theta}\)</span> and the design matrix <span class="math notranslate nohighlight">\(\boldsymbol{X}\)</span> which embody our model,
that is <span class="math notranslate nohighlight">\(\boldsymbol{\tilde{y}}=\boldsymbol{X}\boldsymbol{\theta}\)</span>.</p>
<p>Thereafter we found the parameters <span class="math notranslate nohighlight">\(\boldsymbol{\theta}\)</span> by optimizing the means squared error via the so-called cost function</p>
<div class="math notranslate nohighlight">
\[
C(\boldsymbol{X},\boldsymbol{\theta}) =\frac{1}{n}\sum_{i=0}^{n-1}(y_i-\tilde{y}_i)^2=\mathbb{E}\left[(\boldsymbol{y}-\boldsymbol{\tilde{y}})^2\right].
\]</div>
<p>We can rewrite this as</p>
<div class="math notranslate nohighlight">
\[
\mathbb{E}\left[(\boldsymbol{y}-\boldsymbol{\tilde{y}})^2\right]=\frac{1}{n}\sum_i(f_i-\mathbb{E}\left[\boldsymbol{\tilde{y}}\right])^2+\frac{1}{n}\sum_i(\tilde{y}_i-\mathbb{E}\left[\boldsymbol{\tilde{y}}\right])^2+\sigma^2.
\]</div>
<p>The three terms represent the square of the bias of the learning
method, which can be thought of as the error caused by the simplifying
assumptions built into the method. The second term represents the
variance of the chosen model and finally the last terms is variance of
the error <span class="math notranslate nohighlight">\(\boldsymbol{\epsilon}\)</span>.</p>
<p>To derive this equation, we need to recall that the variance of <span class="math notranslate nohighlight">\(\boldsymbol{y}\)</span> and <span class="math notranslate nohighlight">\(\boldsymbol{\epsilon}\)</span> are both equal to <span class="math notranslate nohighlight">\(\sigma^2\)</span>. The mean value of <span class="math notranslate nohighlight">\(\boldsymbol{\epsilon}\)</span> is by definition equal to zero. Furthermore, the function <span class="math notranslate nohighlight">\(f\)</span> is not a stochastics variable, idem for <span class="math notranslate nohighlight">\(\boldsymbol{\tilde{y}}\)</span>.
We use a more compact notation in terms of the expectation value</p>
<div class="math notranslate nohighlight">
\[
\mathbb{E}\left[(\boldsymbol{y}-\boldsymbol{\tilde{y}})^2\right]=\mathbb{E}\left[(\boldsymbol{f}+\boldsymbol{\epsilon}-\boldsymbol{\tilde{y}})^2\right],
\]</div>
<p>and adding and subtracting <span class="math notranslate nohighlight">\(\mathbb{E}\left[\boldsymbol{\tilde{y}}\right]\)</span> we get</p>
<div class="math notranslate nohighlight">
\[
\mathbb{E}\left[(\boldsymbol{y}-\boldsymbol{\tilde{y}})^2\right]=\mathbb{E}\left[(\boldsymbol{f}+\boldsymbol{\epsilon}-\boldsymbol{\tilde{y}}+\mathbb{E}\left[\boldsymbol{\tilde{y}}\right]-\mathbb{E}\left[\boldsymbol{\tilde{y}}\right])^2\right],
\]</div>
<p>which, using the abovementioned expectation values can be rewritten as</p>
<div class="math notranslate nohighlight">
\[
\mathbb{E}\left[(\boldsymbol{y}-\boldsymbol{\tilde{y}})^2\right]=\mathbb{E}\left[(\boldsymbol{y}-\mathbb{E}\left[\boldsymbol{\tilde{y}}\right])^2\right]+\mathrm{Var}\left[\boldsymbol{\tilde{y}}\right]+\sigma^2,
\]</div>
<p>that is the rewriting in terms of the so-called bias, the variance of the model <span class="math notranslate nohighlight">\(\boldsymbol{\tilde{y}}\)</span> and the variance of <span class="math notranslate nohighlight">\(\boldsymbol{\epsilon}\)</span>.</p>
<p><strong>Note that in order to derive these equations we have assumed we can replace the unknown function <span class="math notranslate nohighlight">\(\boldsymbol{f}\)</span> with the target/output data <span class="math notranslate nohighlight">\(\boldsymbol{y}\)</span>.</strong></p>
</section>
<section id="a-way-to-read-the-bias-variance-tradeoff">
<h2>A way to Read the Bias-Variance Tradeoff<a class="headerlink" href="#a-way-to-read-the-bias-variance-tradeoff" title="Link to this heading">#</a></h2>
<!-- dom:FIGURE: [figures/BiasVariance.png, width=600 frac=0.9] -->
<!-- begin figure -->
<p><img src="figures/BiasVariance.png" width="600"><p style="font-size: 0.9em"><i>Figure 1: </i></p></p>
<!-- end figure --></section>
<section id="understanding-what-happens">
<h2>Understanding what happens<a class="headerlink" href="#understanding-what-happens" title="Link to this heading">#</a></h2>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span>%matplotlib inline
import matplotlib.pyplot as plt
import numpy as np
from sklearn.linear_model import LinearRegression, Ridge, Lasso
from sklearn.preprocessing import PolynomialFeatures
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.utils import resample
np.random.seed(2018)
n = 40
n_boostraps = 100
maxdegree = 14
# Make data set.
x = np.linspace(-3, 3, n).reshape(-1, 1)
y = np.exp(-x**2) + 1.5 * np.exp(-(x-2)**2)+ np.random.normal(0, 0.1, x.shape)
error = np.zeros(maxdegree)
bias = np.zeros(maxdegree)
variance = np.zeros(maxdegree)
polydegree = np.zeros(maxdegree)
x_train, x_test, y_train, y_test = train_test_split(x, y, test_size=0.2)
for degree in range(maxdegree):
model = make_pipeline(PolynomialFeatures(degree=degree), LinearRegression(fit_intercept=False))
y_pred = np.empty((y_test.shape[0], n_boostraps))
for i in range(n_boostraps):
x_, y_ = resample(x_train, y_train)
y_pred[:, i] = model.fit(x_, y_).predict(x_test).ravel()
polydegree[degree] = degree
error[degree] = np.mean( np.mean((y_test - y_pred)**2, axis=1, keepdims=True) )
bias[degree] = np.mean( (y_test - np.mean(y_pred, axis=1, keepdims=True))**2 )
variance[degree] = np.mean( np.var(y_pred, axis=1, keepdims=True) )
print(&#39;Polynomial degree:&#39;, degree)
print(&#39;Error:&#39;, error[degree])
print(&#39;Bias^2:&#39;, bias[degree])
print(&#39;Var:&#39;, variance[degree])
print(&#39;{} &gt;= {} + {} = {}&#39;.format(error[degree], bias[degree], variance[degree], bias[degree]+variance[degree]))
plt.plot(polydegree, error, label=&#39;Error&#39;)
plt.plot(polydegree, bias, label=&#39;bias&#39;)
plt.plot(polydegree, variance, label=&#39;Variance&#39;)
plt.legend()
plt.show()
</pre></div>
</div>
</div>
</div>
</section>
<section id="summing-up">
<h2>Summing up<a class="headerlink" href="#summing-up" title="Link to this heading">#</a></h2>
<p>The bias-variance tradeoff summarizes the fundamental tension in
machine learning, particularly supervised learning, between the
complexity of a model and the amount of training data needed to train
it. Since data is often limited, in practice it is often useful to
use a less-complex model with higher bias, that is a model whose asymptotic
performance is worse than another model because it is easier to
train and less sensitive to sampling noise arising from having a
finite-sized training dataset (smaller variance).</p>
<p>The above equations tell us that in
order to minimize the expected test error, we need to select a
statistical learning method that simultaneously achieves low variance
and low bias. Note that variance is inherently a nonnegative quantity,
and squared bias is also nonnegative. Hence, we see that the expected
test MSE can never lie below <span class="math notranslate nohighlight">\(Var(\epsilon)\)</span>, the irreducible error.</p>
<p>What do we mean by the variance and bias of a statistical learning
method? The variance refers to the amount by which our model would change if we
estimated it using a different training data set. Since the training
data are used to fit the statistical learning method, different
training data sets will result in a different estimate. But ideally the
estimate for our model should not vary too much between training
sets. However, if a method has high variance then small changes in
the training data can result in large changes in the model. In general, more
flexible statistical methods have higher variance.</p>
<p>You may also find this recent <a class="reference external" href="https://www.pnas.org/content/116/32/15849">article</a> of interest.</p>
</section>
<section id="another-example-from-scikit-learn-s-repository">
<h2>Another Example from Scikit-Learn’s Repository<a class="headerlink" href="#another-example-from-scikit-learn-s-repository" title="Link to this heading">#</a></h2>
<p>This example demonstrates the problems of underfitting and overfitting and
how we can use linear regression with polynomial features to approximate
nonlinear functions. The plot shows the function that we want to approximate,
which is a part of the cosine function. In addition, the samples from the
real function and the approximations of different models are displayed. The
models have polynomial features of different degrees. We can see that a
linear function (polynomial with degree 1) is not sufficient to fit the
training samples. This is called <strong>underfitting</strong>. A polynomial of degree 4
approximates the true function almost perfectly. However, for higher degrees
the model will <strong>overfit</strong> the training data, i.e. it learns the noise of the
training data.
We evaluate quantitatively overfitting and underfitting by using
cross-validation. We calculate the mean squared error (MSE) on the validation
set, the higher, the less likely the model generalizes correctly from the
training data.</p>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span>
#print(__doc__)
import numpy as np
import matplotlib.pyplot as plt
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import PolynomialFeatures
from sklearn.linear_model import LinearRegression
from sklearn.model_selection import cross_val_score
def true_fun(X):
return np.cos(1.5 * np.pi * X)
np.random.seed(0)
n_samples = 30
degrees = [1, 4, 15]
X = np.sort(np.random.rand(n_samples))
y = true_fun(X) + np.random.randn(n_samples) * 0.1
plt.figure(figsize=(14, 5))
for i in range(len(degrees)):
ax = plt.subplot(1, len(degrees), i + 1)
plt.setp(ax, xticks=(), yticks=())
polynomial_features = PolynomialFeatures(degree=degrees[i],
include_bias=False)
linear_regression = LinearRegression()
pipeline = Pipeline([(&quot;polynomial_features&quot;, polynomial_features),
(&quot;linear_regression&quot;, linear_regression)])
pipeline.fit(X[:, np.newaxis], y)
# Evaluate the models using crossvalidation
scores = cross_val_score(pipeline, X[:, np.newaxis], y,
scoring=&quot;neg_mean_squared_error&quot;, cv=10)
X_test = np.linspace(0, 1, 100)
plt.plot(X_test, pipeline.predict(X_test[:, np.newaxis]), label=&quot;Model&quot;)
plt.plot(X_test, true_fun(X_test), label=&quot;True function&quot;)
plt.scatter(X, y, edgecolor=&#39;b&#39;, s=20, label=&quot;Samples&quot;)
plt.xlabel(&quot;x&quot;)
plt.ylabel(&quot;y&quot;)
plt.xlim((0, 1))
plt.ylim((-2, 2))
plt.legend(loc=&quot;best&quot;)
plt.title(&quot;Degree {}\nMSE = {:.2e}(+/- {:.2e})&quot;.format(
degrees[i], -scores.mean(), scores.std()))
plt.show()
</pre></div>
</div>
</div>
</div>
</section>
<section id="various-steps-in-cross-validation">
<h2>Various steps in cross-validation<a class="headerlink" href="#various-steps-in-cross-validation" title="Link to this heading">#</a></h2>
<p>When the repetitive splitting of the data set is done randomly,
samples may accidently end up in a fast majority of the splits in
either training or test set. Such samples may have an unbalanced
influence on either model building or prediction evaluation. To avoid
this <span class="math notranslate nohighlight">\(k\)</span>-fold cross-validation structures the data splitting. The
samples are divided into <span class="math notranslate nohighlight">\(k\)</span> more or less equally sized exhaustive and
mutually exclusive subsets. In turn (at each split) one of these
subsets plays the role of the test set while the union of the
remaining subsets constitutes the training set. Such a splitting
warrants a balanced representation of each sample in both training and
test set over the splits. Still the division into the <span class="math notranslate nohighlight">\(k\)</span> subsets
involves a degree of randomness. This may be fully excluded when
choosing <span class="math notranslate nohighlight">\(k=n\)</span>. This particular case is referred to as leave-one-out
cross-validation (LOOCV).</p>
</section>
<section id="cross-validation-in-brief">
<h2>Cross-validation in brief<a class="headerlink" href="#cross-validation-in-brief" title="Link to this heading">#</a></h2>
<p>For the various values of <span class="math notranslate nohighlight">\(k\)</span></p>
<ol class="arabic simple">
<li><p>shuffle the dataset randomly.</p></li>
<li><p>Split the dataset into <span class="math notranslate nohighlight">\(k\)</span> groups.</p></li>
<li><p>For each unique group:</p></li>
</ol>
<p>a. Decide which group to use as set for test data</p>
<p>b. Take the remaining groups as a training data set</p>
<p>c. Fit a model on the training set and evaluate it on the test set</p>
<p>d. Retain the evaluation score and discard the model</p>
<ol class="arabic simple" start="5">
<li><p>Summarize the model using the sample of model evaluation scores</p></li>
</ol>
</section>
<section id="code-example-for-cross-validation-and-k-fold-cross-validation">
<h2>Code Example for Cross-validation and <span class="math notranslate nohighlight">\(k\)</span>-fold Cross-validation<a class="headerlink" href="#code-example-for-cross-validation-and-k-fold-cross-validation" title="Link to this heading">#</a></h2>
<p>The code here uses Ridge regression with cross-validation (CV) resampling and <span class="math notranslate nohighlight">\(k\)</span>-fold CV in order to fit a specific polynomial.</p>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span>import numpy as np
import matplotlib.pyplot as plt
from sklearn.model_selection import KFold
from sklearn.linear_model import Ridge
from sklearn.model_selection import cross_val_score
from sklearn.preprocessing import PolynomialFeatures
# A seed just to ensure that the random numbers are the same for every run.
# Useful for eventual debugging.
np.random.seed(3155)
# Generate the data.
nsamples = 100
x = np.random.randn(nsamples)
y = 3*x**2 + np.random.randn(nsamples)
## Cross-validation on Ridge regression using KFold only
# Decide degree on polynomial to fit
poly = PolynomialFeatures(degree = 6)
# Decide which values of lambda to use
nlambdas = 500
lambdas = np.logspace(-3, 5, nlambdas)
# Initialize a KFold instance
k = 5
kfold = KFold(n_splits = k)
# Perform the cross-validation to estimate MSE
scores_KFold = np.zeros((nlambdas, k))
i = 0
for lmb in lambdas:
ridge = Ridge(alpha = lmb)
j = 0
for train_inds, test_inds in kfold.split(x):
xtrain = x[train_inds]
ytrain = y[train_inds]
xtest = x[test_inds]
ytest = y[test_inds]
Xtrain = poly.fit_transform(xtrain[:, np.newaxis])
ridge.fit(Xtrain, ytrain[:, np.newaxis])
Xtest = poly.fit_transform(xtest[:, np.newaxis])
ypred = ridge.predict(Xtest)
scores_KFold[i,j] = np.sum((ypred - ytest[:, np.newaxis])**2)/np.size(ypred)
j += 1
i += 1
estimated_mse_KFold = np.mean(scores_KFold, axis = 1)
## Cross-validation using cross_val_score from sklearn along with KFold
# kfold is an instance initialized above as:
# kfold = KFold(n_splits = k)
estimated_mse_sklearn = np.zeros(nlambdas)
i = 0
for lmb in lambdas:
ridge = Ridge(alpha = lmb)
X = poly.fit_transform(x[:, np.newaxis])
estimated_mse_folds = cross_val_score(ridge, X, y[:, np.newaxis], scoring=&#39;neg_mean_squared_error&#39;, cv=kfold)
# cross_val_score return an array containing the estimated negative mse for every fold.
# we have to the the mean of every array in order to get an estimate of the mse of the model
estimated_mse_sklearn[i] = np.mean(-estimated_mse_folds)
i += 1
## Plot and compare the slightly different ways to perform cross-validation
plt.figure()
plt.plot(np.log10(lambdas), estimated_mse_sklearn, label = &#39;cross_val_score&#39;)
#plt.plot(np.log10(lambdas), estimated_mse_KFold, &#39;r--&#39;, label = &#39;KFold&#39;)
plt.xlabel(&#39;log10(lambda)&#39;)
plt.ylabel(&#39;mse&#39;)
plt.legend()
plt.show()
</pre></div>
</div>
</div>
</div>
</section>
<section id="more-examples-on-bootstrap-and-cross-validation-and-errors">
<h2>More examples on bootstrap and cross-validation and errors<a class="headerlink" href="#more-examples-on-bootstrap-and-cross-validation-and-errors" title="Link to this heading">#</a></h2>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span># Common imports
import os
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from sklearn.linear_model import LinearRegression, Ridge, Lasso
from sklearn.model_selection import train_test_split
from sklearn.utils import resample
from sklearn.metrics import mean_squared_error
# Where to save the figures and data files
PROJECT_ROOT_DIR = &quot;Results&quot;
FIGURE_ID = &quot;Results/FigureFiles&quot;
DATA_ID = &quot;DataFiles/&quot;
if not os.path.exists(PROJECT_ROOT_DIR):
os.mkdir(PROJECT_ROOT_DIR)
if not os.path.exists(FIGURE_ID):
os.makedirs(FIGURE_ID)
if not os.path.exists(DATA_ID):
os.makedirs(DATA_ID)
def image_path(fig_id):
return os.path.join(FIGURE_ID, fig_id)
def data_path(dat_id):
return os.path.join(DATA_ID, dat_id)
def save_fig(fig_id):
plt.savefig(image_path(fig_id) + &quot;.png&quot;, format=&#39;png&#39;)
infile = open(data_path(&quot;EoS.csv&quot;),&#39;r&#39;)
# Read the EoS data as csv file and organize the data into two arrays with density and energies
EoS = pd.read_csv(infile, names=(&#39;Density&#39;, &#39;Energy&#39;))
EoS[&#39;Energy&#39;] = pd.to_numeric(EoS[&#39;Energy&#39;], errors=&#39;coerce&#39;)
EoS = EoS.dropna()
Energies = EoS[&#39;Energy&#39;]
Density = EoS[&#39;Density&#39;]
# The design matrix now as function of various polytrops
Maxpolydegree = 30
X = np.zeros((len(Density),Maxpolydegree))
X[:,0] = 1.0
testerror = np.zeros(Maxpolydegree)
trainingerror = np.zeros(Maxpolydegree)
polynomial = np.zeros(Maxpolydegree)
trials = 100
for polydegree in range(1, Maxpolydegree):
polynomial[polydegree] = polydegree
for degree in range(polydegree):
X[:,degree] = Density**(degree/3.0)
# loop over trials in order to estimate the expectation value of the MSE
testerror[polydegree] = 0.0
trainingerror[polydegree] = 0.0
for samples in range(trials):
x_train, x_test, y_train, y_test = train_test_split(X, Energies, test_size=0.2)
model = LinearRegression(fit_intercept=False).fit(x_train, y_train)
ypred = model.predict(x_train)
ytilde = model.predict(x_test)
testerror[polydegree] += mean_squared_error(y_test, ytilde)
trainingerror[polydegree] += mean_squared_error(y_train, ypred)
testerror[polydegree] /= trials
trainingerror[polydegree] /= trials
print(&quot;Degree of polynomial: %3d&quot;% polynomial[polydegree])
print(&quot;Mean squared error on training data: %.8f&quot; % trainingerror[polydegree])
print(&quot;Mean squared error on test data: %.8f&quot; % testerror[polydegree])
plt.plot(polynomial, np.log10(trainingerror), label=&#39;Training Error&#39;)
plt.plot(polynomial, np.log10(testerror), label=&#39;Test Error&#39;)
plt.xlabel(&#39;Polynomial degree&#39;)
plt.ylabel(&#39;log10[MSE]&#39;)
plt.legend()
plt.show()
</pre></div>
</div>
</div>
</div>
<p>Note that we kept the intercept column in the fitting here. This means that we need to set the <strong>intercept</strong> in the call to the <strong>Scikit-Learn</strong> function as <strong>False</strong>. Alternatively, we could have set up the design matrix <span class="math notranslate nohighlight">\(X\)</span> without the first column of ones.</p>
</section>
<section id="the-same-example-but-now-with-cross-validation">
<h2>The same example but now with cross-validation<a class="headerlink" href="#the-same-example-but-now-with-cross-validation" title="Link to this heading">#</a></h2>
<p>In this example we keep the intercept column again but add cross-validation in order to estimate the best possible value of the means squared error.</p>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span># Common imports
import os
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from sklearn.linear_model import LinearRegression, Ridge, Lasso
from sklearn.metrics import mean_squared_error
from sklearn.model_selection import KFold
from sklearn.model_selection import cross_val_score
# Where to save the figures and data files
PROJECT_ROOT_DIR = &quot;Results&quot;
FIGURE_ID = &quot;Results/FigureFiles&quot;
DATA_ID = &quot;DataFiles/&quot;
if not os.path.exists(PROJECT_ROOT_DIR):
os.mkdir(PROJECT_ROOT_DIR)
if not os.path.exists(FIGURE_ID):
os.makedirs(FIGURE_ID)
if not os.path.exists(DATA_ID):
os.makedirs(DATA_ID)
def image_path(fig_id):
return os.path.join(FIGURE_ID, fig_id)
def data_path(dat_id):
return os.path.join(DATA_ID, dat_id)
def save_fig(fig_id):
plt.savefig(image_path(fig_id) + &quot;.png&quot;, format=&#39;png&#39;)
infile = open(data_path(&quot;EoS.csv&quot;),&#39;r&#39;)
# Read the EoS data as csv file and organize the data into two arrays with density and energies
EoS = pd.read_csv(infile, names=(&#39;Density&#39;, &#39;Energy&#39;))
EoS[&#39;Energy&#39;] = pd.to_numeric(EoS[&#39;Energy&#39;], errors=&#39;coerce&#39;)
EoS = EoS.dropna()
Energies = EoS[&#39;Energy&#39;]
Density = EoS[&#39;Density&#39;]
# The design matrix now as function of various polytrops
Maxpolydegree = 30
X = np.zeros((len(Density),Maxpolydegree))
X[:,0] = 1.0
estimated_mse_sklearn = np.zeros(Maxpolydegree)
polynomial = np.zeros(Maxpolydegree)
k =5
kfold = KFold(n_splits = k)
for polydegree in range(1, Maxpolydegree):
polynomial[polydegree] = polydegree
for degree in range(polydegree):
X[:,degree] = Density**(degree/3.0)
OLS = LinearRegression(fit_intercept=False)
# loop over trials in order to estimate the expectation value of the MSE
estimated_mse_folds = cross_val_score(OLS, X, Energies, scoring=&#39;neg_mean_squared_error&#39;, cv=kfold)
#[:, np.newaxis]
estimated_mse_sklearn[polydegree] = np.mean(-estimated_mse_folds)
plt.plot(polynomial, np.log10(estimated_mse_sklearn), label=&#39;Test Error&#39;)
plt.xlabel(&#39;Polynomial degree&#39;)
plt.ylabel(&#39;log10[MSE]&#39;)
plt.legend()
plt.show()
</pre></div>
</div>
</div>
</div>
</section>
<section id="logistic-regression">
<h2>Logistic Regression<a class="headerlink" href="#logistic-regression" title="Link to this heading">#</a></h2>
<p>In linear regression our main interest was centered on learning the
coefficients of a functional fit (say a polynomial) in order to be
able to predict the response of a continuous variable on some unseen
data. The fit to the continuous variable <span class="math notranslate nohighlight">\(y_i\)</span> is based on some
independent variables <span class="math notranslate nohighlight">\(\boldsymbol{x}_i\)</span>. Linear regression resulted in
analytical expressions for standard ordinary Least Squares or Ridge
regression (in terms of matrices to invert) for several quantities,
ranging from the variance and thereby the confidence intervals of the
parameters <span class="math notranslate nohighlight">\(\boldsymbol{\theta}\)</span> to the mean squared error. If we can invert
the product of the design matrices, linear regression gives then a
simple recipe for fitting our data.</p>
</section>
<section id="classification-problems">
<h2>Classification problems<a class="headerlink" href="#classification-problems" title="Link to this heading">#</a></h2>
<p>Classification problems, however, are concerned with outcomes taking
the form of discrete variables (i.e. categories). We may for example,
on the basis of DNA sequencing for a number of patients, like to find
out which mutations are important for a certain disease; or based on
scans of various patients’ brains, figure out if there is a tumor or
not; or given a specific physical system, we’d like to identify its
state, say whether it is an ordered or disordered system (typical
situation in solid state physics); or classify the status of a
patient, whether she/he has a stroke or not and many other similar
situations.</p>
<p>The most common situation we encounter when we apply logistic
regression is that of two possible outcomes, normally denoted as a
binary outcome, true or false, positive or negative, success or
failure etc.</p>
</section>
<section id="optimization-and-deep-learning">
<h2>Optimization and Deep learning<a class="headerlink" href="#optimization-and-deep-learning" title="Link to this heading">#</a></h2>
<p>Logistic regression will also serve as our stepping stone towards
neural network algorithms and supervised deep learning. For logistic
learning, the minimization of the cost function leads to a non-linear
equation in the parameters <span class="math notranslate nohighlight">\(\boldsymbol{\theta}\)</span>. The optimization of the
problem calls therefore for minimization algorithms. This forms the
bottle neck of all machine learning algorithms, namely how to find
reliable minima of a multi-variable function. This leads us to the
family of gradient descent methods. The latter are the working horses
of basically all modern machine learning algorithms.</p>
<p>We note also that many of the topics discussed here on logistic
regression are also commonly used in modern supervised Deep Learning
models, as we will see later.</p>
</section>
<section id="basics">
<h2>Basics<a class="headerlink" href="#basics" title="Link to this heading">#</a></h2>
<p>We consider the case where the outputs/targets, also called the
responses or the outcomes, <span class="math notranslate nohighlight">\(y_i\)</span> are discrete and only take values
from <span class="math notranslate nohighlight">\(k=0,\dots,K-1\)</span> (i.e. <span class="math notranslate nohighlight">\(K\)</span> classes).</p>
<p>The goal is to predict the
output classes from the design matrix <span class="math notranslate nohighlight">\(\boldsymbol{X}\in\mathbb{R}^{n\times p}\)</span>
made of <span class="math notranslate nohighlight">\(n\)</span> samples, each of which carries <span class="math notranslate nohighlight">\(p\)</span> features or predictors. The
primary goal is to identify the classes to which new unseen samples
belong.</p>
<p>Let us specialize to the case of two classes only, with outputs
<span class="math notranslate nohighlight">\(y_i=0\)</span> and <span class="math notranslate nohighlight">\(y_i=1\)</span>. Our outcomes could represent the status of a
credit card user that could default or not on her/his credit card
debt. That is</p>
<div class="math notranslate nohighlight">
\[\begin{split}
y_i = \begin{bmatrix} 0 &amp; \mathrm{no}\\ 1 &amp; \mathrm{yes} \end{bmatrix}.
\end{split}\]</div>
</section>
<section id="linear-classifier">
<h2>Linear classifier<a class="headerlink" href="#linear-classifier" title="Link to this heading">#</a></h2>
<p>Before moving to the logistic model, let us try to use our linear
regression model to classify these two outcomes. We could for example
fit a linear model to the default case if <span class="math notranslate nohighlight">\(y_i &gt; 0.5\)</span> and the no
default case <span class="math notranslate nohighlight">\(y_i \leq 0.5\)</span>.</p>
<p>We would then have our
weighted linear combination, namely</p>
<!-- Equation labels as ordinary links -->
<div id="_auto1"></div>
<div class="math notranslate nohighlight">
\[
\begin{equation}
\boldsymbol{y} = \boldsymbol{X}^T\boldsymbol{\theta} + \boldsymbol{\epsilon},
\label{_auto1} \tag{1}
\end{equation}
\]</div>
<p>where <span class="math notranslate nohighlight">\(\boldsymbol{y}\)</span> is a vector representing the possible outcomes, <span class="math notranslate nohighlight">\(\boldsymbol{X}\)</span> is our
<span class="math notranslate nohighlight">\(n\times p\)</span> design matrix and <span class="math notranslate nohighlight">\(\boldsymbol{\theta}\)</span> represents our estimators/predictors.</p>
</section>
<section id="some-selected-properties">
<h2>Some selected properties<a class="headerlink" href="#some-selected-properties" title="Link to this heading">#</a></h2>
<p>The main problem with our function is that it takes values on the
entire real axis. In the case of logistic regression, however, the
labels <span class="math notranslate nohighlight">\(y_i\)</span> are discrete variables. A typical example is the credit
card data discussed below here, where we can set the state of
defaulting the debt to <span class="math notranslate nohighlight">\(y_i=1\)</span> and not to <span class="math notranslate nohighlight">\(y_i=0\)</span> for one the persons
in the data set (see the full example below).</p>
<p>One simple way to get a discrete output is to have sign
functions that map the output of a linear regressor to values <span class="math notranslate nohighlight">\(\{0,1\}\)</span>,
<span class="math notranslate nohighlight">\(f(s_i)=sign(s_i)=1\)</span> if <span class="math notranslate nohighlight">\(s_i\ge 0\)</span> and 0 if otherwise.
We will encounter this model in our first demonstration of neural networks.</p>
<p>Historically it is called the <strong>perceptron</strong> model in the machine learning
literature. This model is extremely simple. However, in many cases it is more
favorable to use a ``soft” classifier that outputs
the probability of a given category. This leads us to the logistic function.</p>
</section>
<section id="simple-example">
<h2>Simple example<a class="headerlink" href="#simple-example" title="Link to this heading">#</a></h2>
<p>The following example on data for coronary heart disease (CHD) as function of age may serve as an illustration. In the code here we read and plot whether a person has had CHD (output = 1) or not (output = 0). This ouput is plotted the person’s against age. Clearly, the figure shows that attempting to make a standard linear regression fit may not be very meaningful.</p>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span># Common imports
import os
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from sklearn.linear_model import LinearRegression, Ridge, Lasso
from sklearn.model_selection import train_test_split
from sklearn.utils import resample
from sklearn.metrics import mean_squared_error
from IPython.display import display
from pylab import plt, mpl
mpl.rcParams[&#39;font.family&#39;] = &#39;serif&#39;
# Where to save the figures and data files
PROJECT_ROOT_DIR = &quot;Results&quot;
FIGURE_ID = &quot;Results/FigureFiles&quot;
DATA_ID = &quot;DataFiles/&quot;
if not os.path.exists(PROJECT_ROOT_DIR):
os.mkdir(PROJECT_ROOT_DIR)
if not os.path.exists(FIGURE_ID):
os.makedirs(FIGURE_ID)
if not os.path.exists(DATA_ID):
os.makedirs(DATA_ID)
def image_path(fig_id):
return os.path.join(FIGURE_ID, fig_id)
def data_path(dat_id):
return os.path.join(DATA_ID, dat_id)
def save_fig(fig_id):
plt.savefig(image_path(fig_id) + &quot;.png&quot;, format=&#39;png&#39;)
infile = open(data_path(&quot;chddata.csv&quot;),&#39;r&#39;)
# Read the chd data as csv file and organize the data into arrays with age group, age, and chd
chd = pd.read_csv(infile, names=(&#39;ID&#39;, &#39;Age&#39;, &#39;Agegroup&#39;, &#39;CHD&#39;))
chd.columns = [&#39;ID&#39;, &#39;Age&#39;, &#39;Agegroup&#39;, &#39;CHD&#39;]
output = chd[&#39;CHD&#39;]
age = chd[&#39;Age&#39;]
agegroup = chd[&#39;Agegroup&#39;]
numberID = chd[&#39;ID&#39;]
display(chd)
plt.scatter(age, output, marker=&#39;o&#39;)
plt.axis([18,70.0,-0.1, 1.2])
plt.xlabel(r&#39;Age&#39;)
plt.ylabel(r&#39;CHD&#39;)
plt.title(r&#39;Age distribution and Coronary heart disease&#39;)
plt.show()
</pre></div>
</div>
</div>
</div>
</section>
<section id="plotting-the-mean-value-for-each-group">
<h2>Plotting the mean value for each group<a class="headerlink" href="#plotting-the-mean-value-for-each-group" title="Link to this heading">#</a></h2>
<p>What we could attempt however is to plot the mean value for each group.</p>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span>agegroupmean = np.array([0.1, 0.133, 0.250, 0.333, 0.462, 0.625, 0.765, 0.800])
group = np.array([1, 2, 3, 4, 5, 6, 7, 8])
plt.plot(group, agegroupmean, &quot;r-&quot;)
plt.axis([0,9,0, 1.0])
plt.xlabel(r&#39;Age group&#39;)
plt.ylabel(r&#39;CHD mean values&#39;)
plt.title(r&#39;Mean values for each age group&#39;)
plt.show()
</pre></div>
</div>
</div>
</div>
<p>We are now trying to find a function <span class="math notranslate nohighlight">\(f(y\vert x)\)</span>, that is a function which gives us an expected value for the output <span class="math notranslate nohighlight">\(y\)</span> with a given input <span class="math notranslate nohighlight">\(x\)</span>.
In standard linear regression with a linear dependence on <span class="math notranslate nohighlight">\(x\)</span>, we would write this in terms of our model</p>
<div class="math notranslate nohighlight">
\[
f(y_i\vert x_i)=\theta_0+\theta_1 x_i.
\]</div>
<p>This expression implies however that <span class="math notranslate nohighlight">\(f(y_i\vert x_i)\)</span> could take any
value from minus infinity to plus infinity. If we however let
<span class="math notranslate nohighlight">\(f(y\vert y)\)</span> be represented by the mean value, the above example
shows us that we can constrain the function to take values between
zero and one, that is we have <span class="math notranslate nohighlight">\(0 \le f(y_i\vert x_i) \le 1\)</span>. Looking
at our last curve we see also that it has an S-shaped form. This leads
us to a very popular model for the function <span class="math notranslate nohighlight">\(f\)</span>, namely the so-called
Sigmoid function or logistic model. We will consider this function as
representing the probability for finding a value of <span class="math notranslate nohighlight">\(y_i\)</span> with a given
<span class="math notranslate nohighlight">\(x_i\)</span>.</p>
</section>
<section id="the-logistic-function">
<h2>The logistic function<a class="headerlink" href="#the-logistic-function" title="Link to this heading">#</a></h2>
<p>Another widely studied model, is the so-called
perceptron model, which is an example of a “hard classification” model. We
will encounter this model when we discuss neural networks as
well. Each datapoint is deterministically assigned to a category (i.e
<span class="math notranslate nohighlight">\(y_i=0\)</span> or <span class="math notranslate nohighlight">\(y_i=1\)</span>). In many cases, and the coronary heart disease data forms one of many such examples, it is favorable to have a “soft”
classifier that outputs the probability of a given category rather
than a single value. For example, given <span class="math notranslate nohighlight">\(x_i\)</span>, the classifier
outputs the probability of being in a category <span class="math notranslate nohighlight">\(k\)</span>. Logistic regression
is the most common example of a so-called soft classifier. In logistic
regression, the probability that a data point <span class="math notranslate nohighlight">\(x_i\)</span>
belongs to a category <span class="math notranslate nohighlight">\(y_i=\{0,1\}\)</span> is given by the so-called logit function (or Sigmoid) which is meant to represent the likelihood for a given event,</p>
<div class="math notranslate nohighlight">
\[
p(t) = \frac{1}{1+\mathrm \exp{-t}}=\frac{\exp{t}}{1+\mathrm \exp{t}}.
\]</div>
<p>Note that <span class="math notranslate nohighlight">\(1-p(t)= p(-t)\)</span>.</p>
</section>
<section id="examples-of-likelihood-functions-used-in-logistic-regression-and-nueral-networks">
<h2>Examples of likelihood functions used in logistic regression and nueral networks<a class="headerlink" href="#examples-of-likelihood-functions-used-in-logistic-regression-and-nueral-networks" title="Link to this heading">#</a></h2>
<p>The following code plots the logistic function, the step function and other functions we will encounter from here and on.</p>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span>&quot;&quot;&quot;The sigmoid function (or the logistic curve) is a
function that takes any real number, z, and outputs a number (0,1).
It is useful in neural networks for assigning weights on a relative scale.
The value z is the weighted sum of parameters involved in the learning algorithm.&quot;&quot;&quot;
import numpy
import matplotlib.pyplot as plt
import math as mt
z = numpy.arange(-5, 5, .1)
sigma_fn = numpy.vectorize(lambda z: 1/(1+numpy.exp(-z)))
sigma = sigma_fn(z)
fig = plt.figure()
ax = fig.add_subplot(111)
ax.plot(z, sigma)
ax.set_ylim([-0.1, 1.1])
ax.set_xlim([-5,5])
ax.grid(True)
ax.set_xlabel(&#39;z&#39;)
ax.set_title(&#39;sigmoid function&#39;)
plt.show()
&quot;&quot;&quot;Step Function&quot;&quot;&quot;
z = numpy.arange(-5, 5, .02)
step_fn = numpy.vectorize(lambda z: 1.0 if z &gt;= 0.0 else 0.0)
step = step_fn(z)
fig = plt.figure()
ax = fig.add_subplot(111)
ax.plot(z, step)
ax.set_ylim([-0.5, 1.5])
ax.set_xlim([-5,5])
ax.grid(True)
ax.set_xlabel(&#39;z&#39;)
ax.set_title(&#39;step function&#39;)
plt.show()
&quot;&quot;&quot;tanh Function&quot;&quot;&quot;
z = numpy.arange(-2*mt.pi, 2*mt.pi, 0.1)
t = numpy.tanh(z)
fig = plt.figure()
ax = fig.add_subplot(111)
ax.plot(z, t)
ax.set_ylim([-1.0, 1.0])
ax.set_xlim([-2*mt.pi,2*mt.pi])
ax.grid(True)
ax.set_xlabel(&#39;z&#39;)
ax.set_title(&#39;tanh function&#39;)
plt.show()
</pre></div>
</div>
</div>
</div>
</section>
<section id="two-parameters">
<h2>Two parameters<a class="headerlink" href="#two-parameters" title="Link to this heading">#</a></h2>
<p>We assume now that we have two classes with <span class="math notranslate nohighlight">\(y_i\)</span> either <span class="math notranslate nohighlight">\(0\)</span> or <span class="math notranslate nohighlight">\(1\)</span>. Furthermore we assume also that we have only two parameters <span class="math notranslate nohighlight">\(\theta\)</span> in our fitting of the Sigmoid function, that is we define probabilities</p>
<div class="math notranslate nohighlight">
\[\begin{split}
\begin{align*}
p(y_i=1|x_i,\boldsymbol{\theta}) &amp;= \frac{\exp{(\theta_0+\theta_1x_i)}}{1+\exp{(\theta_0+\theta_1x_i)}},\nonumber\\
p(y_i=0|x_i,\boldsymbol{\theta}) &amp;= 1 - p(y_i=1|x_i,\boldsymbol{\theta}),
\end{align*}
\end{split}\]</div>
<p>where <span class="math notranslate nohighlight">\(\boldsymbol{\theta}\)</span> are the weights we wish to extract from data, in our case <span class="math notranslate nohighlight">\(\theta_0\)</span> and <span class="math notranslate nohighlight">\(\theta_1\)</span>.</p>
<p>Note that we used</p>
<div class="math notranslate nohighlight">
\[
p(y_i=0\vert x_i, \boldsymbol{\theta}) = 1-p(y_i=1\vert x_i, \boldsymbol{\theta}).
\]</div>
</section>
<section id="maximum-likelihood">
<h2>Maximum likelihood<a class="headerlink" href="#maximum-likelihood" title="Link to this heading">#</a></h2>
<p>In order to define the total likelihood for all possible outcomes from a<br />
dataset <span class="math notranslate nohighlight">\(\mathcal{D}=\{(y_i,x_i)\}\)</span>, with the binary labels
<span class="math notranslate nohighlight">\(y_i\in\{0,1\}\)</span> and where the data points are drawn independently, we use the so-called <a class="reference external" href="https://en.wikipedia.org/wiki/Maximum_likelihood_estimation">Maximum Likelihood Estimation</a> (MLE) principle.
We aim thus at maximizing
the probability of seeing the observed data. We can then approximate the
likelihood in terms of the product of the individual probabilities of a specific outcome <span class="math notranslate nohighlight">\(y_i\)</span>, that is</p>
<div class="math notranslate nohighlight">
\[\begin{split}
\begin{align*}
P(\mathcal{D}|\boldsymbol{\theta})&amp; = \prod_{i=1}^n \left[p(y_i=1|x_i,\boldsymbol{\theta})\right]^{y_i}\left[1-p(y_i=1|x_i,\boldsymbol{\theta}))\right]^{1-y_i}\nonumber \\
\end{align*}
\end{split}\]</div>
<p>from which we obtain the log-likelihood and our <strong>cost/loss</strong> function</p>
<div class="math notranslate nohighlight">
\[
\mathcal{C}(\boldsymbol{\theta}) = \sum_{i=1}^n \left( y_i\log{p(y_i=1|x_i,\boldsymbol{\theta})} + (1-y_i)\log\left[1-p(y_i=1|x_i,\boldsymbol{\theta}))\right]\right).
\]</div>
</section>
<section id="the-cost-function-rewritten">
<h2>The cost function rewritten<a class="headerlink" href="#the-cost-function-rewritten" title="Link to this heading">#</a></h2>
<p>Reordering the logarithms, we can rewrite the <strong>cost/loss</strong> function as</p>
<div class="math notranslate nohighlight">
\[
\mathcal{C}(\boldsymbol{\theta}) = \sum_{i=1}^n \left(y_i(\theta_0+\theta_1x_i) -\log{(1+\exp{(\theta_0+\theta_1x_i)})}\right).
\]</div>
<p>The maximum likelihood estimator is defined as the set of parameters that maximize the log-likelihood where we maximize with respect to <span class="math notranslate nohighlight">\(\theta\)</span>.
Since the cost (error) function is just the negative log-likelihood, for logistic regression we have that</p>
<div class="math notranslate nohighlight">
\[
\mathcal{C}(\boldsymbol{\theta})=-\sum_{i=1}^n \left(y_i(\theta_0+\theta_1x_i) -\log{(1+\exp{(\theta_0+\theta_1x_i)})}\right).
\]</div>
<p>This equation is known in statistics as the <strong>cross entropy</strong>. Finally, we note that just as in linear regression,
in practice we often supplement the cross-entropy with additional regularization terms, usually <span class="math notranslate nohighlight">\(L_1\)</span> and <span class="math notranslate nohighlight">\(L_2\)</span> regularization as we did for Ridge and Lasso regression.</p>
</section>
<section id="minimizing-the-cross-entropy">
<h2>Minimizing the cross entropy<a class="headerlink" href="#minimizing-the-cross-entropy" title="Link to this heading">#</a></h2>
<p>The cross entropy is a convex function of the weights <span class="math notranslate nohighlight">\(\boldsymbol{\theta}\)</span> and,
therefore, any local minimizer is a global minimizer.</p>
<p>Minimizing this
cost function with respect to the two parameters <span class="math notranslate nohighlight">\(\theta_0\)</span> and <span class="math notranslate nohighlight">\(\theta_1\)</span> we obtain</p>
<div class="math notranslate nohighlight">
\[
\frac{\partial \mathcal{C}(\boldsymbol{\theta})}{\partial \theta_0} = -\sum_{i=1}^n \left(y_i -\frac{\exp{(\theta_0+\theta_1x_i)}}{1+\exp{(\theta_0+\theta_1x_i)}}\right),
\]</div>
<p>and</p>
<div class="math notranslate nohighlight">
\[
\frac{\partial \mathcal{C}(\boldsymbol{\theta})}{\partial \theta_1} = -\sum_{i=1}^n \left(y_ix_i -x_i\frac{\exp{(\theta_0+\theta_1x_i)}}{1+\exp{(\theta_0+\theta_1x_i)}}\right).
\]</div>
</section>
<section id="a-more-compact-expression">
<h2>A more compact expression<a class="headerlink" href="#a-more-compact-expression" title="Link to this heading">#</a></h2>
<p>Let us now define a vector <span class="math notranslate nohighlight">\(\boldsymbol{y}\)</span> with <span class="math notranslate nohighlight">\(n\)</span> elements <span class="math notranslate nohighlight">\(y_i\)</span>, an
<span class="math notranslate nohighlight">\(n\times p\)</span> matrix <span class="math notranslate nohighlight">\(\boldsymbol{X}\)</span> which contains the <span class="math notranslate nohighlight">\(x_i\)</span> values and a
vector <span class="math notranslate nohighlight">\(\boldsymbol{p}\)</span> of fitted probabilities <span class="math notranslate nohighlight">\(p(y_i\vert x_i,\boldsymbol{\theta})\)</span>. We can rewrite in a more compact form the first
derivative of the cost function as</p>
<div class="math notranslate nohighlight">
\[
\frac{\partial \mathcal{C}(\boldsymbol{\theta})}{\partial \boldsymbol{\theta}} = -\boldsymbol{X}^T\left(\boldsymbol{y}-\boldsymbol{p}\right).
\]</div>
<p>If we in addition define a diagonal matrix <span class="math notranslate nohighlight">\(\boldsymbol{W}\)</span> with elements
<span class="math notranslate nohighlight">\(p(y_i\vert x_i,\boldsymbol{\theta})(1-p(y_i\vert x_i,\boldsymbol{\theta})\)</span>, we can obtain a compact expression of the second derivative as</p>
<div class="math notranslate nohighlight">
\[
\frac{\partial^2 \mathcal{C}(\boldsymbol{\theta})}{\partial \boldsymbol{\theta}\partial \boldsymbol{\theta}^T} = \boldsymbol{X}^T\boldsymbol{W}\boldsymbol{X}.
\]</div>
</section>
<section id="extending-to-more-predictors">
<h2>Extending to more predictors<a class="headerlink" href="#extending-to-more-predictors" title="Link to this heading">#</a></h2>
<p>Within a binary classification problem, we can easily expand our model to include multiple predictors. Our ratio between likelihoods is then with <span class="math notranslate nohighlight">\(p\)</span> predictors</p>
<div class="math notranslate nohighlight">
\[
\log{ \frac{p(\boldsymbol{\theta}\boldsymbol{x})}{1-p(\boldsymbol{\theta}\boldsymbol{x})}} = \theta_0+\theta_1x_1+\theta_2x_2+\dots+\theta_px_p.
\]</div>
<p>Here we defined <span class="math notranslate nohighlight">\(\boldsymbol{x}=[1,x_1,x_2,\dots,x_p]\)</span> and <span class="math notranslate nohighlight">\(\boldsymbol{\theta}=[\theta_0, \theta_1, \dots, \theta_p]\)</span> leading to</p>
<div class="math notranslate nohighlight">
\[
p(\boldsymbol{\theta}\boldsymbol{x})=\frac{ \exp{(\theta_0+\theta_1x_1+\theta_2x_2+\dots+\theta_px_p)}}{1+\exp{(\theta_0+\theta_1x_1+\theta_2x_2+\dots+\theta_px_p)}}.
\]</div>
</section>
<section id="including-more-classes">
<h2>Including more classes<a class="headerlink" href="#including-more-classes" title="Link to this heading">#</a></h2>
<p>Till now we have mainly focused on two classes, the so-called binary
system. Suppose we wish to extend to <span class="math notranslate nohighlight">\(K\)</span> classes. Let us for the sake
of simplicity assume we have only two predictors. We have then following model</p>
<div class="math notranslate nohighlight">
\[
\log{\frac{p(C=1\vert x)}{p(K\vert x)}} = \theta_{10}+\theta_{11}x_1,
\]</div>
<p>and</p>
<div class="math notranslate nohighlight">
\[
\log{\frac{p(C=2\vert x)}{p(K\vert x)}} = \theta_{20}+\theta_{21}x_1,
\]</div>
<p>and so on till the class <span class="math notranslate nohighlight">\(C=K-1\)</span> class</p>
<div class="math notranslate nohighlight">
\[
\log{\frac{p(C=K-1\vert x)}{p(K\vert x)}} = \theta_{(K-1)0}+\theta_{(K-1)1}x_1,
\]</div>
<p>and the model is specified in term of <span class="math notranslate nohighlight">\(K-1\)</span> so-called log-odds or
<strong>logit</strong> transformations.</p>
</section>
<section id="more-classes">
<h2>More classes<a class="headerlink" href="#more-classes" title="Link to this heading">#</a></h2>
<p>In our discussion of neural networks we will encounter the above again
in terms of a slightly modified function, the so-called <strong>Softmax</strong> function.</p>
<p>The softmax function is used in various multiclass classification
methods, such as multinomial logistic regression (also known as
softmax regression), multiclass linear discriminant analysis, naive
Bayes classifiers, and artificial neural networks. Specifically, in
multinomial logistic regression and linear discriminant analysis, the
input to the function is the result of <span class="math notranslate nohighlight">\(K\)</span> distinct linear functions,
and the predicted probability for the <span class="math notranslate nohighlight">\(k\)</span>-th class given a sample
vector <span class="math notranslate nohighlight">\(\boldsymbol{x}\)</span> and a weighting vector <span class="math notranslate nohighlight">\(\boldsymbol{\theta}\)</span> is (with two
predictors):</p>
<div class="math notranslate nohighlight">
\[
p(C=k\vert \mathbf {x} )=\frac{\exp{(\theta_{k0}+\theta_{k1}x_1)}}{1+\sum_{l=1}^{K-1}\exp{(\theta_{l0}+\theta_{l1}x_1)}}.
\]</div>
<p>It is easy to extend to more predictors. The final class is</p>
<div class="math notranslate nohighlight">
\[
p(C=K\vert \mathbf {x} )=\frac{1}{1+\sum_{l=1}^{K-1}\exp{(\theta_{l0}+\theta_{l1}x_1)}},
\]</div>
<p>and they sum to one. Our earlier discussions were all specialized to
the case with two classes only. It is easy to see from the above that
what we derived earlier is compatible with these equations.</p>
<p>To find the optimal parameters we would typically use a gradient
descent method. Newton’s method and gradient descent methods are
discussed in the material on <a class="reference external" href="https://compphysics.github.io/MachineLearning/doc/pub/Splines/html/Splines-bs.html">optimization
methods</a>.</p>
</section>
<section id="optimization-the-central-part-of-any-machine-learning-algortithm">
<h2>Optimization, the central part of any Machine Learning algortithm<a class="headerlink" href="#optimization-the-central-part-of-any-machine-learning-algortithm" title="Link to this heading">#</a></h2>
<p>Almost every problem in machine learning and data science starts with
a dataset <span class="math notranslate nohighlight">\(X\)</span>, a model <span class="math notranslate nohighlight">\(g(\theta)\)</span>, which is a function of the
parameters <span class="math notranslate nohighlight">\(\theta\)</span> and a cost function <span class="math notranslate nohighlight">\(C(X, g(\theta))\)</span> that allows
us to judge how well the model <span class="math notranslate nohighlight">\(g(\theta)\)</span> explains the observations
<span class="math notranslate nohighlight">\(X\)</span>. The model is fit by finding the values of <span class="math notranslate nohighlight">\(\theta\)</span> that minimize
the cost function. Ideally we would be able to solve for <span class="math notranslate nohighlight">\(\theta\)</span>
analytically, however this is not possible in general and we must use
some approximative/numerical method to compute the minimum.</p>
</section>
<section id="revisiting-our-logistic-regression-case">
<h2>Revisiting our Logistic Regression case<a class="headerlink" href="#revisiting-our-logistic-regression-case" title="Link to this heading">#</a></h2>
<p>In our discussion on Logistic Regression we studied the
case of
two classes, with <span class="math notranslate nohighlight">\(y_i\)</span> either
<span class="math notranslate nohighlight">\(0\)</span> or <span class="math notranslate nohighlight">\(1\)</span>. Furthermore we assumed also that we have only two
parameters <span class="math notranslate nohighlight">\(\theta\)</span> in our fitting, that is we
defined probabilities</p>
<div class="math notranslate nohighlight">
\[\begin{split}
\begin{align*}
p(y_i=1|x_i,\boldsymbol{\theta}) &amp;= \frac{\exp{(\theta_0+\theta_1x_i)}}{1+\exp{(\theta_0+\theta_1x_i)}},\nonumber\\
p(y_i=0|x_i,\boldsymbol{\theta}) &amp;= 1 - p(y_i=1|x_i,\boldsymbol{\theta}),
\end{align*}
\end{split}\]</div>
<p>where <span class="math notranslate nohighlight">\(\boldsymbol{\theta}\)</span> are the weights we wish to extract from data, in our case <span class="math notranslate nohighlight">\(\theta_0\)</span> and <span class="math notranslate nohighlight">\(\theta_1\)</span>.</p>
</section>
<section id="the-equations-to-solve">
<h2>The equations to solve<a class="headerlink" href="#the-equations-to-solve" title="Link to this heading">#</a></h2>
<p>Our compact equations used a definition of a vector <span class="math notranslate nohighlight">\(\boldsymbol{y}\)</span> with <span class="math notranslate nohighlight">\(n\)</span>
elements <span class="math notranslate nohighlight">\(y_i\)</span>, an <span class="math notranslate nohighlight">\(n\times p\)</span> matrix <span class="math notranslate nohighlight">\(\boldsymbol{X}\)</span> which contains the
<span class="math notranslate nohighlight">\(x_i\)</span> values and a vector <span class="math notranslate nohighlight">\(\boldsymbol{p}\)</span> of fitted probabilities
<span class="math notranslate nohighlight">\(p(y_i\vert x_i,\boldsymbol{\theta})\)</span>. We rewrote in a more compact form
the first derivative of the cost function as</p>
<div class="math notranslate nohighlight">
\[
\frac{\partial \mathcal{C}(\boldsymbol{\theta})}{\partial \boldsymbol{\theta}} = -\boldsymbol{X}^T\left(\boldsymbol{y}-\boldsymbol{p}\right).
\]</div>
<p>If we in addition define a diagonal matrix <span class="math notranslate nohighlight">\(\boldsymbol{W}\)</span> with elements
<span class="math notranslate nohighlight">\(p(y_i\vert x_i,\boldsymbol{\theta})(1-p(y_i\vert x_i,\boldsymbol{\theta})\)</span>, we can obtain a compact expression of the second derivative as</p>
<div class="math notranslate nohighlight">
\[
\frac{\partial^2 \mathcal{C}(\boldsymbol{\theta})}{\partial \boldsymbol{\theta}\partial \boldsymbol{\theta}^T} = \boldsymbol{X}^T\boldsymbol{W}\boldsymbol{X}.
\]</div>
<p>This defines what is called the Hessian matrix.</p>
</section>
<section id="solving-using-newton-raphson-s-method">
<h2>Solving using Newton-Raphson’s method<a class="headerlink" href="#solving-using-newton-raphson-s-method" title="Link to this heading">#</a></h2>
<p>If we can set up these equations, Newton-Raphson’s iterative method is normally the method of choice. It requires however that we can compute in an efficient way the matrices that define the first and second derivatives.</p>
<p>Our iterative scheme is then given by</p>
<div class="math notranslate nohighlight">
\[
\boldsymbol{\theta}^{\mathrm{new}} = \boldsymbol{\theta}^{\mathrm{old}}-\left(\frac{\partial^2 \mathcal{C}(\boldsymbol{\theta})}{\partial \boldsymbol{\theta}\partial \boldsymbol{\theta}^T}\right)^{-1}_{\boldsymbol{\theta}^{\mathrm{old}}}\times \left(\frac{\partial \mathcal{C}(\boldsymbol{\theta})}{\partial \boldsymbol{\theta}}\right)_{\boldsymbol{\theta}^{\mathrm{old}}},
\]</div>
<p>or in matrix form as</p>
<div class="math notranslate nohighlight">
\[
\boldsymbol{\theta}^{\mathrm{new}} = \boldsymbol{\theta}^{\mathrm{old}}-\left(\boldsymbol{X}^T\boldsymbol{W}\boldsymbol{X} \right)^{-1}\times \left(-\boldsymbol{X}^T(\boldsymbol{y}-\boldsymbol{p}) \right)_{\boldsymbol{\theta}^{\mathrm{old}}}.
\]</div>
<p>The right-hand side is computed with the old values of <span class="math notranslate nohighlight">\(\theta\)</span>.</p>
<p>If we can compute these matrices, in particular the Hessian, the above is often the easiest method to implement.</p>
</section>
<section id="example-code-for-logistic-regression">
<h2>Example code for Logistic Regression<a class="headerlink" href="#example-code-for-logistic-regression" title="Link to this heading">#</a></h2>
<p>Here we make a class for Logistic regression. The code uses a simple data set and includes both a binary case and a multiclass case.</p>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span>import numpy as np
class LogisticRegression:
&quot;&quot;&quot;
Logistic Regression for binary and multiclass classification.
&quot;&quot;&quot;
def __init__(self, lr=0.01, epochs=1000, fit_intercept=True, verbose=False):
self.lr = lr # Learning rate for gradient descent
self.epochs = epochs # Number of iterations
self.fit_intercept = fit_intercept # Whether to add intercept (bias)
self.verbose = verbose # Print loss during training if True
self.weights = None
self.multi_class = False # Will be determined at fit time
def _add_intercept(self, X):
&quot;&quot;&quot;Add intercept term (column of ones) to feature matrix.&quot;&quot;&quot;
intercept = np.ones((X.shape[0], 1))
return np.concatenate((intercept, X), axis=1)
def _sigmoid(self, z):
&quot;&quot;&quot;Sigmoid function for binary logistic.&quot;&quot;&quot;
return 1 / (1 + np.exp(-z))
def _softmax(self, Z):
&quot;&quot;&quot;Softmax function for multiclass logistic.&quot;&quot;&quot;
exp_Z = np.exp(Z - np.max(Z, axis=1, keepdims=True))
return exp_Z / np.sum(exp_Z, axis=1, keepdims=True)
def fit(self, X, y):
&quot;&quot;&quot;
Train the logistic regression model using gradient descent.
Supports binary (sigmoid) and multiclass (softmax) based on y.
&quot;&quot;&quot;
X = np.array(X)
y = np.array(y)
n_samples, n_features = X.shape
# Add intercept if needed
if self.fit_intercept:
X = self._add_intercept(X)
n_features += 1
# Determine classes and mode (binary vs multiclass)
unique_classes = np.unique(y)
if len(unique_classes) &gt; 2:
self.multi_class = True
else:
self.multi_class = False
# ----- Multiclass case -----
if self.multi_class:
n_classes = len(unique_classes)
# Map original labels to 0...n_classes-1
class_to_index = {c: idx for idx, c in enumerate(unique_classes)}
y_indices = np.array([class_to_index[c] for c in y])
# Initialize weight matrix (features x classes)
self.weights = np.zeros((n_features, n_classes))
# One-hot encode y
Y_onehot = np.zeros((n_samples, n_classes))
Y_onehot[np.arange(n_samples), y_indices] = 1
# Gradient descent
for epoch in range(self.epochs):
scores = X.dot(self.weights) # Linear scores (n_samples x n_classes)
probs = self._softmax(scores) # Probabilities (n_samples x n_classes)
# Compute gradient (features x classes)
gradient = (1 / n_samples) * X.T.dot(probs - Y_onehot)
# Update weights
self.weights -= self.lr * gradient
if self.verbose and epoch % 100 == 0:
# Compute current loss (categorical cross-entropy)
loss = -np.sum(Y_onehot * np.log(probs + 1e-15)) / n_samples
print(f&quot;[Epoch {epoch}] Multiclass loss: {loss:.4f}&quot;)
# ----- Binary case -----
else:
# Convert y to 0/1 if not already
if not np.array_equal(unique_classes, [0, 1]):
# Map the two classes to 0 and 1
class0, class1 = unique_classes
y_binary = np.where(y == class1, 1, 0)
else:
y_binary = y.copy().astype(int)
# Initialize weights vector (features,)
self.weights = np.zeros(n_features)
# Gradient descent
for epoch in range(self.epochs):
linear_model = X.dot(self.weights) # (n_samples,)
probs = self._sigmoid(linear_model) # (n_samples,)
# Gradient for binary cross-entropy
gradient = (1 / n_samples) * X.T.dot(probs - y_binary)
self.weights -= self.lr * gradient
if self.verbose and epoch % 100 == 0:
# Compute binary cross-entropy loss
loss = -np.mean(
y_binary * np.log(probs + 1e-15) +
(1 - y_binary) * np.log(1 - probs + 1e-15)
)
print(f&quot;[Epoch {epoch}] Binary loss: {loss:.4f}&quot;)
def predict_prob(self, X):
&quot;&quot;&quot;
Compute probability estimates. Returns a 1D array for binary or
a 2D array (n_samples x n_classes) for multiclass.
&quot;&quot;&quot;
X = np.array(X)
# Add intercept if the model used it
if self.fit_intercept:
X = self._add_intercept(X)
scores = X.dot(self.weights)
if self.multi_class:
return self._softmax(scores)
else:
return self._sigmoid(scores)
def predict(self, X):
&quot;&quot;&quot;
Predict class labels for samples in X.
Returns integer class labels (0,1 for binary, or 0...C-1 for multiclass).
&quot;&quot;&quot;
probs = self.predict_prob(X)
if self.multi_class:
# Choose class with highest probability
return np.argmax(probs, axis=1)
else:
# Threshold at 0.5 for binary
return (probs &gt;= 0.5).astype(int)
</pre></div>
</div>
</div>
</div>
<p>The class implements the sigmoid and softmax internally. During fit(),
we check the number of classes: if more than 2, we set
self.multi_class=True and perform multinomial logistic regression. We
one-hot encode the target vector and update a weight matrix with
softmax probabilities. Otherwise, we do standard binary logistic
regression, converting labels to 0/1 if needed and updating a weight
vector. In both cases we use batch gradient descent on the
cross-entropy loss (we add a small epsilon 1e-15 to logs for numerical
stability). Progress (loss) can be printed if verbose=True.</p>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span># Evaluation Metrics
#We define helper functions for accuracy and cross-entropy loss. Accuracy is the fraction of correct predictions . For loss, we compute the appropriate cross-entropy:
def accuracy_score(y_true, y_pred):
&quot;&quot;&quot;Accuracy = (# correct predictions) / (total samples).&quot;&quot;&quot;
y_true = np.array(y_true)
y_pred = np.array(y_pred)
return np.mean(y_true == y_pred)
def binary_cross_entropy(y_true, y_prob):
&quot;&quot;&quot;
Binary cross-entropy loss.
y_true: true binary labels (0 or 1), y_prob: predicted probabilities for class 1.
&quot;&quot;&quot;
y_true = np.array(y_true)
y_prob = np.clip(np.array(y_prob), 1e-15, 1-1e-15)
return -np.mean(y_true * np.log(y_prob) + (1 - y_true) * np.log(1 - y_prob))
def categorical_cross_entropy(y_true, y_prob):
&quot;&quot;&quot;
Categorical cross-entropy loss for multiclass.
y_true: true labels (0...C-1), y_prob: array of predicted probabilities (n_samples x C).
&quot;&quot;&quot;
y_true = np.array(y_true, dtype=int)
y_prob = np.clip(np.array(y_prob), 1e-15, 1-1e-15)
# One-hot encode true labels
n_samples, n_classes = y_prob.shape
one_hot = np.zeros_like(y_prob)
one_hot[np.arange(n_samples), y_true] = 1
# Compute cross-entropy
loss_vec = -np.sum(one_hot * np.log(y_prob), axis=1)
return np.mean(loss_vec)
</pre></div>
</div>
</div>
</div>
<section id="synthetic-data-generation">
<h3>Synthetic data generation<a class="headerlink" href="#synthetic-data-generation" title="Link to this heading">#</a></h3>
<p>Binary classification data: Create two Gaussian clusters in 2D. For example, class 0 around mean [-2,-2] and class 1 around [2,2].
Multiclass data: Create several Gaussian clusters (one per class) spread out in feature space.</p>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span>import numpy as np
def generate_binary_data(n_samples=100, n_features=2, random_state=None):
&quot;&quot;&quot;
Generate synthetic binary classification data.
Returns (X, y) where X is (n_samples x n_features), y in {0,1}.
&quot;&quot;&quot;
rng = np.random.RandomState(random_state)
# Half samples for class 0, half for class 1
n0 = n_samples // 2
n1 = n_samples - n0
# Class 0 around mean -2, class 1 around +2
mean0 = -2 * np.ones(n_features)
mean1 = 2 * np.ones(n_features)
X0 = rng.randn(n0, n_features) + mean0
X1 = rng.randn(n1, n_features) + mean1
X = np.vstack((X0, X1))
y = np.array([0]*n0 + [1]*n1)
return X, y
def generate_multiclass_data(n_samples=150, n_features=2, n_classes=3, random_state=None):
&quot;&quot;&quot;
Generate synthetic multiclass data with n_classes Gaussian clusters.
&quot;&quot;&quot;
rng = np.random.RandomState(random_state)
X = []
y = []
samples_per_class = n_samples // n_classes
for cls in range(n_classes):
# Random cluster center for each class
center = rng.uniform(-5, 5, size=n_features)
Xi = rng.randn(samples_per_class, n_features) + center
yi = [cls] * samples_per_class
X.append(Xi)
y.extend(yi)
X = np.vstack(X)
y = np.array(y)
return X, y
# Generate and test on binary data
X_bin, y_bin = generate_binary_data(n_samples=200, n_features=2, random_state=42)
model_bin = LogisticRegression(lr=0.1, epochs=1000)
model_bin.fit(X_bin, y_bin)
y_prob_bin = model_bin.predict_prob(X_bin) # probabilities for class 1
y_pred_bin = model_bin.predict(X_bin) # predicted classes 0 or 1
acc_bin = accuracy_score(y_bin, y_pred_bin)
loss_bin = binary_cross_entropy(y_bin, y_prob_bin)
print(f&quot;Binary Classification - Accuracy: {acc_bin:.2f}, Cross-Entropy Loss: {loss_bin:.2f}&quot;)
#For multiclass:
# Generate and test on multiclass data
X_multi, y_multi = generate_multiclass_data(n_samples=300, n_features=2, n_classes=3, random_state=1)
model_multi = LogisticRegression(lr=0.1, epochs=1000)
model_multi.fit(X_multi, y_multi)
y_prob_multi = model_multi.predict_prob(X_multi) # (n_samples x 3) probabilities
y_pred_multi = model_multi.predict(X_multi) # predicted labels 0,1,2
acc_multi = accuracy_score(y_multi, y_pred_multi)
loss_multi = categorical_cross_entropy(y_multi, y_prob_multi)
print(f&quot;Multiclass Classification - Accuracy: {acc_multi:.2f}, Cross-Entropy Loss: {loss_multi:.2f}&quot;)
# CSV Export
import csv
# Export binary results
with open(&#39;binary_results.csv&#39;, mode=&#39;w&#39;, newline=&#39;&#39;) as f:
writer = csv.writer(f)
writer.writerow([&quot;TrueLabel&quot;, &quot;PredictedLabel&quot;])
for true, pred in zip(y_bin, y_pred_bin):
writer.writerow([true, pred])
# Export multiclass results
with open(&#39;multiclass_results.csv&#39;, mode=&#39;w&#39;, newline=&#39;&#39;) as f:
writer = csv.writer(f)
writer.writerow([&quot;TrueLabel&quot;, &quot;PredictedLabel&quot;])
for true, pred in zip(y_multi, y_pred_multi):
writer.writerow([true, pred])
</pre></div>
</div>
</div>
</div>
</section>
</section>
</section>
<script type="text/x-thebe-config">
{
requestKernel: true,
binderOptions: {
repo: "binder-examples/jupyter-stacks-datascience",
ref: "master",
},
codeMirrorConfig: {
theme: "abcdef",
mode: "python"
},
kernelOptions: {
name: "python3",
path: "./."
},
predefinedOutput: true
}
</script>
<script>kernelName = 'python3'</script>
</article>
<footer class="prev-next-footer d-print-none">
<div class="prev-next-area">
<a class="left-prev"
href="exercisesweek39.html"
title="previous page">
<i class="fa-solid fa-angle-left"></i>
<div class="prev-next-info">
<p class="prev-next-subtitle">previous</p>
<p class="prev-next-title">Exercises week 39</p>
</div>
</a>
<a class="right-next"
href="week40.html"
title="next page">
<div class="prev-next-info">
<p class="prev-next-subtitle">next</p>
<p class="prev-next-title">Week 40: Gradient descent methods (continued) and start Neural networks</p>
</div>
<i class="fa-solid fa-angle-right"></i>
</a>
</div>
</footer>
</div>
<div class="bd-sidebar-secondary bd-toc"><div class="sidebar-secondary-items sidebar-secondary__inner">
<div class="sidebar-secondary-item">
<div class="page-toc tocsection onthispage">
<i class="fa-solid fa-list"></i> Contents
</div>
<nav class="bd-toc-nav page-toc">
<ul class="visible nav section-nav flex-column">
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#plan-for-week-39-september-22-26-2025">Plan for week 39, September 22-26, 2025</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#readings-and-videos-resampling-methods">Readings and Videos, resampling methods</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#readings-and-videos-logistic-regression">Readings and Videos, logistic regression</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#lab-sessions-week-39">Lab sessions week 39</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#lecture-material">Lecture material</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#resampling-methods">Resampling methods</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#resampling-approaches-can-be-computationally-expensive">Resampling approaches can be computationally expensive</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#why-resampling-methods">Why resampling methods ?</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#statistical-analysis">Statistical analysis</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#id1">Resampling methods</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#resampling-methods-bootstrap">Resampling methods: Bootstrap</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#the-bias-variance-tradeoff">The bias-variance tradeoff</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#a-way-to-read-the-bias-variance-tradeoff">A way to Read the Bias-Variance Tradeoff</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#understanding-what-happens">Understanding what happens</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#summing-up">Summing up</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#another-example-from-scikit-learn-s-repository">Another Example from Scikit-Learn’s Repository</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#various-steps-in-cross-validation">Various steps in cross-validation</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#cross-validation-in-brief">Cross-validation in brief</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#code-example-for-cross-validation-and-k-fold-cross-validation">Code Example for Cross-validation and <span class="math notranslate nohighlight">\(k\)</span>-fold Cross-validation</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#more-examples-on-bootstrap-and-cross-validation-and-errors">More examples on bootstrap and cross-validation and errors</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#the-same-example-but-now-with-cross-validation">The same example but now with cross-validation</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#logistic-regression">Logistic Regression</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#classification-problems">Classification problems</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#optimization-and-deep-learning">Optimization and Deep learning</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#basics">Basics</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#linear-classifier">Linear classifier</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#some-selected-properties">Some selected properties</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#simple-example">Simple example</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#plotting-the-mean-value-for-each-group">Plotting the mean value for each group</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#the-logistic-function">The logistic function</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#examples-of-likelihood-functions-used-in-logistic-regression-and-nueral-networks">Examples of likelihood functions used in logistic regression and nueral networks</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#two-parameters">Two parameters</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#maximum-likelihood">Maximum likelihood</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#the-cost-function-rewritten">The cost function rewritten</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#minimizing-the-cross-entropy">Minimizing the cross entropy</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#a-more-compact-expression">A more compact expression</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#extending-to-more-predictors">Extending to more predictors</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#including-more-classes">Including more classes</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#more-classes">More classes</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#optimization-the-central-part-of-any-machine-learning-algortithm">Optimization, the central part of any Machine Learning algortithm</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#revisiting-our-logistic-regression-case">Revisiting our Logistic Regression case</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#the-equations-to-solve">The equations to solve</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#solving-using-newton-raphson-s-method">Solving using Newton-Raphson’s method</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#example-code-for-logistic-regression">Example code for Logistic Regression</a><ul class="nav section-nav flex-column">
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#synthetic-data-generation">Synthetic data generation</a></li>
</ul>
</li>
</ul>
</nav></div>
</div></div>
</div>
<footer class="bd-footer-content">
<div class="bd-footer-content__inner container">
<div class="footer-item">
<p class="component-author">
By Morten Hjorth-Jensen
</p>
</div>
<div class="footer-item">
<p class="copyright">
© Copyright 2023.
<br/>
</p>
</div>
<div class="footer-item">
</div>
<div class="footer-item">
</div>
</div>
</footer>
</main>
</div>
</div>
<!-- Scripts loaded after <body> so the DOM is not blocked -->
<script src="_static/scripts/bootstrap.js?digest=dfe6caa3a7d634c4db9b"></script>
<script src="_static/scripts/pydata-sphinx-theme.js?digest=dfe6caa3a7d634c4db9b"></script>
<footer class="bd-footer">
</footer>
</body>
</html>
Reference in New Issue View Git Blame Copy Permalink