lars/FYS-STK4155
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
00a2bd1a7efde5fbfd9b9d6d6c365dcd82fe8baf
FYS-STK4155/doc/pub/week34/html/._week34-bs037.html
T
Morten Hjorth-Jensen 0158a5b4e3 update
2023-08-21 13:40:37 +02:00

654 lines
40 KiB
HTML
Raw Blame History

<!--
HTML file automatically generated from DocOnce source
(https://github.com/doconce/doconce/)
doconce format html week34.do.txt --html_style=bootstrap --pygments_html_style=default --html_admon=bootstrap_panel --html_output=week34-bs --no_mako
-->
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<meta name="generator" content="DocOnce: https://github.com/doconce/doconce/" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<meta name="description" content="Week 34: Introduction to the course, Logistics and Practicalities">
<title>Week 34: Introduction to the course, Logistics and Practicalities</title>
<!-- Bootstrap style: bootstrap -->
<!-- doconce format html week34.do.txt --html_style=bootstrap --pygments_html_style=default --html_admon=bootstrap_panel --html_output=week34-bs --no_mako -->
<link href="https://netdna.bootstrapcdn.com/bootstrap/3.1.1/css/bootstrap.min.css" rel="stylesheet">
<!-- not necessary
<link href="https://netdna.bootstrapcdn.com/font-awesome/4.0.3/css/font-awesome.css" rel="stylesheet">
-->
<style type="text/css">
/* Add scrollbar to dropdown menus in bootstrap navigation bar */
.dropdown-menu {
height: auto;
max-height: 400px;
overflow-x: hidden;
}
/* Adds an invisible element before each target to offset for the navigation
bar */
.anchor::before {
content:"";
display:block;
height:50px; /* fixed header height for style bootstrap */
margin:-50px 0 0; /* negative fixed header height */
}
</style>
</head>
<!-- tocinfo
{'highest level': 2,
'sections': [('Overview of first week', 2, None, 'overview-of-first-week'),
('Schedule first week', 2, None, 'schedule-first-week'),
('Lectures and ComputerLab', 2, None, 'lectures-and-computerlab'),
('Communication channels', 2, None, 'communication-channels'),
('Course Format', 2, None, 'course-format'),
('Teachers', 2, None, 'teachers'),
('Deadlines for projects (tentative)',
2,
None,
'deadlines-for-projects-tentative'),
('Grading', 2, None, 'grading'),
('Reading material', 2, None, 'reading-material'),
('Textbooks', 2, None, 'textbooks'),
('Reading suggestions week 34',
2,
None,
'reading-suggestions-week-34'),
('Prerequisites', 2, None, 'prerequisites'),
('Topics covered in this course: Statistical analysis and '
'optimization of data',
2,
None,
'topics-covered-in-this-course-statistical-analysis-and-optimization-of-data'),
('Statistical analysis and optimization of data',
2,
None,
'statistical-analysis-and-optimization-of-data'),
('Machine Learning', 2, None, 'machine-learning'),
('Extremely useful tools, strongly recommended',
2,
None,
'extremely-useful-tools-strongly-recommended'),
('Other courses on Data science and Machine Learning at UiO',
2,
None,
'other-courses-on-data-science-and-machine-learning-at-uio'),
('Other courses on Data science and Machine Learning at UiO, '
'contn',
2,
None,
'other-courses-on-data-science-and-machine-learning-at-uio-contn'),
('Learning outcomes', 2, None, 'learning-outcomes'),
('Introduction', 2, None, 'introduction'),
('AI/ML and some statements you may have heard (and what do they '
'mean?)',
2,
None,
'ai-ml-and-some-statements-you-may-have-heard-and-what-do-they-mean'),
('What is Machine Learning?',
2,
None,
'what-is-machine-learning'),
('Types of Machine Learning',
2,
None,
'types-of-machine-learning'),
('Essential elements of ML', 2, None, 'essential-elements-of-ml'),
('An optimization/minimization problem',
2,
None,
'an-optimization-minimization-problem'),
('A Frequentist approach to data analysis',
2,
None,
'a-frequentist-approach-to-data-analysis'),
('What is a good model?', 2, None, 'what-is-a-good-model'),
('What is a good model? Can we define it?',
2,
None,
'what-is-a-good-model-can-we-define-it'),
('Software and needed installations',
2,
None,
'software-and-needed-installations'),
('Python installers', 2, None, 'python-installers'),
('Useful Python libraries', 2, None, 'useful-python-libraries'),
('Installing R, C++, cython or Julia',
2,
None,
'installing-r-c-cython-or-julia'),
('Installing R, C++, cython, Numba etc',
2,
None,
'installing-r-c-cython-numba-etc'),
('Numpy examples and Important Matrix and vector handling '
'packages',
2,
None,
'numpy-examples-and-important-matrix-and-vector-handling-packages'),
('Numpy and arrays', 2, None, 'numpy-and-arrays'),
('Matrices in Python', 2, None, 'matrices-in-python'),
('Meet the Pandas', 2, None, 'meet-the-pandas'),
('Simple linear regression model using _scikit-learn_',
3,
None,
'simple-linear-regression-model-using-scikit-learn'),
('To our real data: nuclear binding energies. Brief reminder on '
'masses and binding energies',
3,
None,
'to-our-real-data-nuclear-binding-energies-brief-reminder-on-masses-and-binding-energies'),
('Organizing our data', 3, None, 'organizing-our-data'),
('And what about using neural networks?',
3,
None,
'and-what-about-using-neural-networks'),
('A first summary', 2, None, 'a-first-summary'),
('Why Linear Regression (aka Ordinary Least Squares and family)',
2,
None,
'why-linear-regression-aka-ordinary-least-squares-and-family'),
('Regression analysis, overarching aims',
2,
None,
'regression-analysis-overarching-aims'),
('Regression analysis, overarching aims II',
2,
None,
'regression-analysis-overarching-aims-ii'),
('Examples', 2, None, 'examples'),
('General linear models and linear algebra',
2,
None,
'general-linear-models-and-linear-algebra'),
('Rewriting the fitting procedure as a linear algebra problem',
2,
None,
'rewriting-the-fitting-procedure-as-a-linear-algebra-problem'),
('Rewriting the fitting procedure as a linear algebra problem, '
'more details',
2,
None,
'rewriting-the-fitting-procedure-as-a-linear-algebra-problem-more-details'),
('Generalizing the fitting procedure as a linear algebra problem',
2,
None,
'generalizing-the-fitting-procedure-as-a-linear-algebra-problem'),
('Generalizing the fitting procedure as a linear algebra problem',
2,
None,
'generalizing-the-fitting-procedure-as-a-linear-algebra-problem'),
('Optimizing our parameters',
2,
None,
'optimizing-our-parameters'),
('Our model for the nuclear binding energies',
2,
None,
'our-model-for-the-nuclear-binding-energies'),
('Optimizing our parameters, more details',
2,
None,
'optimizing-our-parameters-more-details'),
('Interpretations and optimizing our parameters',
2,
None,
'interpretations-and-optimizing-our-parameters'),
('Interpretations and optimizing our parameters',
2,
None,
'interpretations-and-optimizing-our-parameters'),
('Some useful matrix and vector expressions',
2,
None,
'some-useful-matrix-and-vector-expressions'),
('Interpretations and optimizing our parameters',
2,
None,
'interpretations-and-optimizing-our-parameters'),
('Own code for Ordinary Least Squares',
2,
None,
'own-code-for-ordinary-least-squares'),
('Adding error analysis and training set up',
2,
None,
'adding-error-analysis-and-training-set-up'),
('The $\\chi^2$ function', 2, None, 'the-chi-2-function'),
('The $\\chi^2$ function', 2, None, 'the-chi-2-function'),
('The $\\chi^2$ function', 2, None, 'the-chi-2-function'),
('The $\\chi^2$ function', 2, None, 'the-chi-2-function'),
('The $\\chi^2$ function', 2, None, 'the-chi-2-function'),
('The $\\chi^2$ function', 2, None, 'the-chi-2-function'),
('Fitting an Equation of State for Dense Nuclear Matter',
2,
None,
'fitting-an-equation-of-state-for-dense-nuclear-matter'),
('The code', 2, None, 'the-code'),
('Splitting our Data in Training and Test data',
2,
None,
'splitting-our-data-in-training-and-test-data'),
('Exercises', 2, None, 'exercises'),
('Exercise 1: Setting up various Python environments',
2,
None,
'exercise-1-setting-up-various-python-environments'),
('Exercise 2: making your own data and exploring scikit-learn',
2,
None,
'exercise-2-making-your-own-data-and-exploring-scikit-learn'),
('Exercise 3: Split data in test and training data',
2,
None,
'exercise-3-split-data-in-test-and-training-data')]}
end of tocinfo -->
<body>
<script type="text/x-mathjax-config">
MathJax.Hub.Config({
TeX: {
equationNumbers: { autoNumber: "none" },
extensions: ["AMSmath.js", "AMSsymbols.js", "autobold.js", "color.js"]
}
});
</script>
<script type="text/javascript" async
src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.1/MathJax.js?config=TeX-AMS-MML_HTMLorMML">
</script>
<!-- Bootstrap navigation bar -->
<div class="navbar navbar-default navbar-fixed-top">
<div class="navbar-header">
<button type="button" class="navbar-toggle" data-toggle="collapse" data-target=".navbar-responsive-collapse">
<span class="icon-bar"></span>
<span class="icon-bar"></span>
<span class="icon-bar"></span>
</button>
<a class="navbar-brand" href="week34-bs.html">Week 34: Introduction to the course, Logistics and Practicalities</a>
</div>
<div class="navbar-collapse collapse navbar-responsive-collapse">
<ul class="nav navbar-nav navbar-right">
<li class="dropdown">
<a href="#" class="dropdown-toggle" data-toggle="dropdown">Contents <b class="caret"></b></a>
<ul class="dropdown-menu">
<!-- navigation toc: --> <li><a href="._week34-bs001.html#overview-of-first-week" style="font-size: 80%;"><b>Overview of first week</b></a></li>
<!-- navigation toc: --> <li><a href="._week34-bs002.html#schedule-first-week" style="font-size: 80%;"><b>Schedule first week</b></a></li>
<!-- navigation toc: --> <li><a href="._week34-bs003.html#lectures-and-computerlab" style="font-size: 80%;"><b>Lectures and ComputerLab</b></a></li>
<!-- navigation toc: --> <li><a href="._week34-bs004.html#communication-channels" style="font-size: 80%;"><b>Communication channels</b></a></li>
<!-- navigation toc: --> <li><a href="._week34-bs005.html#course-format" style="font-size: 80%;"><b>Course Format</b></a></li>
<!-- navigation toc: --> <li><a href="._week34-bs006.html#teachers" style="font-size: 80%;"><b>Teachers</b></a></li>
<!-- navigation toc: --> <li><a href="._week34-bs007.html#deadlines-for-projects-tentative" style="font-size: 80%;"><b>Deadlines for projects (tentative)</b></a></li>
<!-- navigation toc: --> <li><a href="._week34-bs008.html#grading" style="font-size: 80%;"><b>Grading</b></a></li>
<!-- navigation toc: --> <li><a href="._week34-bs009.html#reading-material" style="font-size: 80%;"><b>Reading material</b></a></li>
<!-- navigation toc: --> <li><a href="._week34-bs010.html#textbooks" style="font-size: 80%;"><b>Textbooks</b></a></li>
<!-- navigation toc: --> <li><a href="._week34-bs011.html#reading-suggestions-week-34" style="font-size: 80%;"><b>Reading suggestions week 34</b></a></li>
<!-- navigation toc: --> <li><a href="._week34-bs012.html#prerequisites" style="font-size: 80%;"><b>Prerequisites</b></a></li>
<!-- navigation toc: --> <li><a href="._week34-bs013.html#topics-covered-in-this-course-statistical-analysis-and-optimization-of-data" style="font-size: 80%;"><b>Topics covered in this course: Statistical analysis and optimization of data</b></a></li>
<!-- navigation toc: --> <li><a href="._week34-bs014.html#statistical-analysis-and-optimization-of-data" style="font-size: 80%;"><b>Statistical analysis and optimization of data</b></a></li>
<!-- navigation toc: --> <li><a href="._week34-bs015.html#machine-learning" style="font-size: 80%;"><b>Machine Learning</b></a></li>
<!-- navigation toc: --> <li><a href="._week34-bs016.html#extremely-useful-tools-strongly-recommended" style="font-size: 80%;"><b>Extremely useful tools, strongly recommended</b></a></li>
<!-- navigation toc: --> <li><a href="._week34-bs017.html#other-courses-on-data-science-and-machine-learning-at-uio" style="font-size: 80%;"><b>Other courses on Data science and Machine Learning at UiO</b></a></li>
<!-- navigation toc: --> <li><a href="._week34-bs018.html#other-courses-on-data-science-and-machine-learning-at-uio-contn" style="font-size: 80%;"><b>Other courses on Data science and Machine Learning at UiO, contn</b></a></li>
<!-- navigation toc: --> <li><a href="._week34-bs019.html#learning-outcomes" style="font-size: 80%;"><b>Learning outcomes</b></a></li>
<!-- navigation toc: --> <li><a href="._week34-bs020.html#introduction" style="font-size: 80%;"><b>Introduction</b></a></li>
<!-- navigation toc: --> <li><a href="._week34-bs021.html#ai-ml-and-some-statements-you-may-have-heard-and-what-do-they-mean" style="font-size: 80%;"><b>AI/ML and some statements you may have heard (and what do they mean?)</b></a></li>
<!-- navigation toc: --> <li><a href="._week34-bs022.html#what-is-machine-learning" style="font-size: 80%;"><b>What is Machine Learning?</b></a></li>
<!-- navigation toc: --> <li><a href="._week34-bs023.html#types-of-machine-learning" style="font-size: 80%;"><b>Types of Machine Learning</b></a></li>
<!-- navigation toc: --> <li><a href="._week34-bs024.html#essential-elements-of-ml" style="font-size: 80%;"><b>Essential elements of ML</b></a></li>
<!-- navigation toc: --> <li><a href="._week34-bs025.html#an-optimization-minimization-problem" style="font-size: 80%;"><b>An optimization/minimization problem</b></a></li>
<!-- navigation toc: --> <li><a href="._week34-bs026.html#a-frequentist-approach-to-data-analysis" style="font-size: 80%;"><b>A Frequentist approach to data analysis</b></a></li>
<!-- navigation toc: --> <li><a href="._week34-bs027.html#what-is-a-good-model" style="font-size: 80%;"><b>What is a good model?</b></a></li>
<!-- navigation toc: --> <li><a href="._week34-bs028.html#what-is-a-good-model-can-we-define-it" style="font-size: 80%;"><b>What is a good model? Can we define it?</b></a></li>
<!-- navigation toc: --> <li><a href="._week34-bs029.html#software-and-needed-installations" style="font-size: 80%;"><b>Software and needed installations</b></a></li>
<!-- navigation toc: --> <li><a href="._week34-bs030.html#python-installers" style="font-size: 80%;"><b>Python installers</b></a></li>
<!-- navigation toc: --> <li><a href="._week34-bs031.html#useful-python-libraries" style="font-size: 80%;"><b>Useful Python libraries</b></a></li>
<!-- navigation toc: --> <li><a href="._week34-bs032.html#installing-r-c-cython-or-julia" style="font-size: 80%;"><b>Installing R, C++, cython or Julia</b></a></li>
<!-- navigation toc: --> <li><a href="._week34-bs033.html#installing-r-c-cython-numba-etc" style="font-size: 80%;"><b>Installing R, C++, cython, Numba etc</b></a></li>
<!-- navigation toc: --> <li><a href="._week34-bs034.html#numpy-examples-and-important-matrix-and-vector-handling-packages" style="font-size: 80%;"><b>Numpy examples and Important Matrix and vector handling packages</b></a></li>
<!-- navigation toc: --> <li><a href="._week34-bs035.html#numpy-and-arrays" style="font-size: 80%;"><b>Numpy and arrays</b></a></li>
<!-- navigation toc: --> <li><a href="._week34-bs036.html#matrices-in-python" style="font-size: 80%;"><b>Matrices in Python</b></a></li>
<!-- navigation toc: --> <li><a href="#meet-the-pandas" style="font-size: 80%;"><b>Meet the Pandas</b></a></li>
<!-- navigation toc: --> <li><a href="._week34-bs038.html#simple-linear-regression-model-using-scikit-learn" style="font-size: 80%;">&nbsp;&nbsp;&nbsp;Simple linear regression model using <b>scikit-learn</b></a></li>
<!-- navigation toc: --> <li><a href="._week34-bs038.html#to-our-real-data-nuclear-binding-energies-brief-reminder-on-masses-and-binding-energies" style="font-size: 80%;">&nbsp;&nbsp;&nbsp;To our real data: nuclear binding energies. Brief reminder on masses and binding energies</a></li>
<!-- navigation toc: --> <li><a href="._week34-bs038.html#organizing-our-data" style="font-size: 80%;">&nbsp;&nbsp;&nbsp;Organizing our data</a></li>
<!-- navigation toc: --> <li><a href="._week34-bs038.html#and-what-about-using-neural-networks" style="font-size: 80%;">&nbsp;&nbsp;&nbsp;And what about using neural networks?</a></li>
<!-- navigation toc: --> <li><a href="._week34-bs038.html#a-first-summary" style="font-size: 80%;"><b>A first summary</b></a></li>
<!-- navigation toc: --> <li><a href="._week34-bs039.html#why-linear-regression-aka-ordinary-least-squares-and-family" style="font-size: 80%;"><b>Why Linear Regression (aka Ordinary Least Squares and family)</b></a></li>
<!-- navigation toc: --> <li><a href="._week34-bs040.html#regression-analysis-overarching-aims" style="font-size: 80%;"><b>Regression analysis, overarching aims</b></a></li>
<!-- navigation toc: --> <li><a href="._week34-bs041.html#regression-analysis-overarching-aims-ii" style="font-size: 80%;"><b>Regression analysis, overarching aims II</b></a></li>
<!-- navigation toc: --> <li><a href="._week34-bs042.html#examples" style="font-size: 80%;"><b>Examples</b></a></li>
<!-- navigation toc: --> <li><a href="._week34-bs043.html#general-linear-models-and-linear-algebra" style="font-size: 80%;"><b>General linear models and linear algebra</b></a></li>
<!-- navigation toc: --> <li><a href="._week34-bs044.html#rewriting-the-fitting-procedure-as-a-linear-algebra-problem" style="font-size: 80%;"><b>Rewriting the fitting procedure as a linear algebra problem</b></a></li>
<!-- navigation toc: --> <li><a href="._week34-bs045.html#rewriting-the-fitting-procedure-as-a-linear-algebra-problem-more-details" style="font-size: 80%;"><b>Rewriting the fitting procedure as a linear algebra problem, more details</b></a></li>
<!-- navigation toc: --> <li><a href="._week34-bs047.html#generalizing-the-fitting-procedure-as-a-linear-algebra-problem" style="font-size: 80%;"><b>Generalizing the fitting procedure as a linear algebra problem</b></a></li>
<!-- navigation toc: --> <li><a href="._week34-bs047.html#generalizing-the-fitting-procedure-as-a-linear-algebra-problem" style="font-size: 80%;"><b>Generalizing the fitting procedure as a linear algebra problem</b></a></li>
<!-- navigation toc: --> <li><a href="._week34-bs048.html#optimizing-our-parameters" style="font-size: 80%;"><b>Optimizing our parameters</b></a></li>
<!-- navigation toc: --> <li><a href="._week34-bs049.html#our-model-for-the-nuclear-binding-energies" style="font-size: 80%;"><b>Our model for the nuclear binding energies</b></a></li>
<!-- navigation toc: --> <li><a href="._week34-bs050.html#optimizing-our-parameters-more-details" style="font-size: 80%;"><b>Optimizing our parameters, more details</b></a></li>
<!-- navigation toc: --> <li><a href="._week34-bs054.html#interpretations-and-optimizing-our-parameters" style="font-size: 80%;"><b>Interpretations and optimizing our parameters</b></a></li>
<!-- navigation toc: --> <li><a href="._week34-bs054.html#interpretations-and-optimizing-our-parameters" style="font-size: 80%;"><b>Interpretations and optimizing our parameters</b></a></li>
<!-- navigation toc: --> <li><a href="._week34-bs053.html#some-useful-matrix-and-vector-expressions" style="font-size: 80%;"><b>Some useful matrix and vector expressions</b></a></li>
<!-- navigation toc: --> <li><a href="._week34-bs054.html#interpretations-and-optimizing-our-parameters" style="font-size: 80%;"><b>Interpretations and optimizing our parameters</b></a></li>
<!-- navigation toc: --> <li><a href="._week34-bs055.html#own-code-for-ordinary-least-squares" style="font-size: 80%;"><b>Own code for Ordinary Least Squares</b></a></li>
<!-- navigation toc: --> <li><a href="._week34-bs056.html#adding-error-analysis-and-training-set-up" style="font-size: 80%;"><b>Adding error analysis and training set up</b></a></li>
<!-- navigation toc: --> <li><a href="._week34-bs062.html#the-chi-2-function" style="font-size: 80%;"><b>The \( \chi^2 \) function</b></a></li>
<!-- navigation toc: --> <li><a href="._week34-bs062.html#the-chi-2-function" style="font-size: 80%;"><b>The \( \chi^2 \) function</b></a></li>
<!-- navigation toc: --> <li><a href="._week34-bs062.html#the-chi-2-function" style="font-size: 80%;"><b>The \( \chi^2 \) function</b></a></li>
<!-- navigation toc: --> <li><a href="._week34-bs062.html#the-chi-2-function" style="font-size: 80%;"><b>The \( \chi^2 \) function</b></a></li>
<!-- navigation toc: --> <li><a href="._week34-bs062.html#the-chi-2-function" style="font-size: 80%;"><b>The \( \chi^2 \) function</b></a></li>
<!-- navigation toc: --> <li><a href="._week34-bs062.html#the-chi-2-function" style="font-size: 80%;"><b>The \( \chi^2 \) function</b></a></li>
<!-- navigation toc: --> <li><a href="._week34-bs063.html#fitting-an-equation-of-state-for-dense-nuclear-matter" style="font-size: 80%;"><b>Fitting an Equation of State for Dense Nuclear Matter</b></a></li>
<!-- navigation toc: --> <li><a href="._week34-bs064.html#the-code" style="font-size: 80%;"><b>The code</b></a></li>
<!-- navigation toc: --> <li><a href="._week34-bs065.html#splitting-our-data-in-training-and-test-data" style="font-size: 80%;"><b>Splitting our Data in Training and Test data</b></a></li>
<!-- navigation toc: --> <li><a href="._week34-bs066.html#exercises" style="font-size: 80%;"><b>Exercises</b></a></li>
<!-- navigation toc: --> <li><a href="._week34-bs066.html#exercise-1-setting-up-various-python-environments" style="font-size: 80%;"><b>Exercise 1: Setting up various Python environments</b></a></li>
<!-- navigation toc: --> <li><a href="._week34-bs066.html#exercise-2-making-your-own-data-and-exploring-scikit-learn" style="font-size: 80%;"><b>Exercise 2: making your own data and exploring scikit-learn</b></a></li>
<!-- navigation toc: --> <li><a href="._week34-bs066.html#exercise-3-split-data-in-test-and-training-data" style="font-size: 80%;"><b>Exercise 3: Split data in test and training data</b></a></li>
</ul>
</li>
</ul>
</div>
</div>
</div> <!-- end of navigation bar -->
<div class="container">
<p>&nbsp;</p><p>&nbsp;</p><p>&nbsp;</p> <!-- add vertical space -->
<a name="part0037"></a>
<!-- !split -->
<h2 id="meet-the-pandas" class="anchor">Meet the Pandas </h2>
<br/><br/>
<center>
<p><img src="fig/pandas.jpg" width="600" align="bottom"></p>
</center>
<br/><br/>
<p>Another useful Python package is
<a href="https://pandas.pydata.org/" target="_self">pandas</a>, which is an open source library
providing high-performance, easy-to-use data structures and data
analysis tools for Python. <b>pandas</b> stands for panel data, a term borrowed from econometrics and is an efficient library for data analysis with an emphasis on tabular data.
<b>pandas</b> has two major classes, the <b>DataFrame</b> class with two-dimensional data objects and tabular data organized in columns and the class <b>Series</b> with a focus on one-dimensional data objects. Both classes allow you to index data easily as we will see in the examples below.
<b>pandas</b> allows you also to perform mathematical operations on the data, spanning from simple reshapings of vectors and matrices to statistical operations.
</p>
<p>The following simple example shows how we can, in an easy way make tables of our data. Here we define a data set which includes names, place of birth and date of birth, and displays the data in an easy to read way. We will see repeated use of <b>pandas</b>, in particular in connection with classification of data. </p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="inner_cell">
<div class="input_area">
<div class="highlight" style="background: #f8f8f8">
<pre style="line-height: 125%;"><span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">pandas</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">pd</span>
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">IPython.display</span> <span style="color: #008000; font-weight: bold">import</span> display
data <span style="color: #666666">=</span> {<span style="color: #BA2121">&#39;First Name&#39;</span>: [<span style="color: #BA2121">&quot;Frodo&quot;</span>, <span style="color: #BA2121">&quot;Bilbo&quot;</span>, <span style="color: #BA2121">&quot;Aragorn II&quot;</span>, <span style="color: #BA2121">&quot;Samwise&quot;</span>],
<span style="color: #BA2121">&#39;Last Name&#39;</span>: [<span style="color: #BA2121">&quot;Baggins&quot;</span>, <span style="color: #BA2121">&quot;Baggins&quot;</span>,<span style="color: #BA2121">&quot;Elessar&quot;</span>,<span style="color: #BA2121">&quot;Gamgee&quot;</span>],
<span style="color: #BA2121">&#39;Place of birth&#39;</span>: [<span style="color: #BA2121">&quot;Shire&quot;</span>, <span style="color: #BA2121">&quot;Shire&quot;</span>, <span style="color: #BA2121">&quot;Eriador&quot;</span>, <span style="color: #BA2121">&quot;Shire&quot;</span>],
<span style="color: #BA2121">&#39;Date of Birth T.A.&#39;</span>: [<span style="color: #666666">2968</span>, <span style="color: #666666">2890</span>, <span style="color: #666666">2931</span>, <span style="color: #666666">2980</span>]
}
data_pandas <span style="color: #666666">=</span> pd<span style="color: #666666">.</span>DataFrame(data)
display(data_pandas)
</pre>
</div>
</div>
</div>
</div>
<div class="output_wrapper">
<div class="output">
<div class="output_area">
<div class="output_subarea output_stream output_stdout output_text">
</div>
</div>
</div>
</div>
</div>
<p>In the above we have imported <b>pandas</b> with the shorthand <b>pd</b>, the latter has become the standard way we import <b>pandas</b>. We make then a list of various variables
and reorganize the aboves lists into a <b>DataFrame</b> and then print out a neat table with specific column labels as <em>Name</em>, <em>place of birth</em> and <em>date of birth</em>.
Displaying these results, we see that the indices are given by the default numbers from zero to three.
<b>pandas</b> is extremely flexible and we can easily change the above indices by defining a new type of indexing as
</p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="inner_cell">
<div class="input_area">
<div class="highlight" style="background: #f8f8f8">
<pre style="line-height: 125%;">data_pandas <span style="color: #666666">=</span> pd<span style="color: #666666">.</span>DataFrame(data,index<span style="color: #666666">=</span>[<span style="color: #BA2121">&#39;Frodo&#39;</span>,<span style="color: #BA2121">&#39;Bilbo&#39;</span>,<span style="color: #BA2121">&#39;Aragorn&#39;</span>,<span style="color: #BA2121">&#39;Sam&#39;</span>])
display(data_pandas)
</pre>
</div>
</div>
</div>
</div>
<div class="output_wrapper">
<div class="output">
<div class="output_area">
<div class="output_subarea output_stream output_stdout output_text">
</div>
</div>
</div>
</div>
</div>
<p>Thereafter we display the content of the row which begins with the index <b>Aragorn</b></p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="inner_cell">
<div class="input_area">
<div class="highlight" style="background: #f8f8f8">
<pre style="line-height: 125%;">display(data_pandas<span style="color: #666666">.</span>loc[<span style="color: #BA2121">&#39;Aragorn&#39;</span>])
</pre>
</div>
</div>
</div>
</div>
<div class="output_wrapper">
<div class="output">
<div class="output_area">
<div class="output_subarea output_stream output_stdout output_text">
</div>
</div>
</div>
</div>
</div>
<p>We can easily append data to this, for example</p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="inner_cell">
<div class="input_area">
<div class="highlight" style="background: #f8f8f8">
<pre style="line-height: 125%;">new_hobbit <span style="color: #666666">=</span> {<span style="color: #BA2121">&#39;First Name&#39;</span>: [<span style="color: #BA2121">&quot;Peregrin&quot;</span>],
<span style="color: #BA2121">&#39;Last Name&#39;</span>: [<span style="color: #BA2121">&quot;Took&quot;</span>],
<span style="color: #BA2121">&#39;Place of birth&#39;</span>: [<span style="color: #BA2121">&quot;Shire&quot;</span>],
<span style="color: #BA2121">&#39;Date of Birth T.A.&#39;</span>: [<span style="color: #666666">2990</span>]
}
data_pandas<span style="color: #666666">=</span>data_pandas<span style="color: #666666">.</span>append(pd<span style="color: #666666">.</span>DataFrame(new_hobbit, index<span style="color: #666666">=</span>[<span style="color: #BA2121">&#39;Pippin&#39;</span>]))
display(data_pandas)
</pre>
</div>
</div>
</div>
</div>
<div class="output_wrapper">
<div class="output">
<div class="output_area">
<div class="output_subarea output_stream output_stdout output_text">
</div>
</div>
</div>
</div>
</div>
<p>Here are other examples where we use the <b>DataFrame</b> functionality to handle arrays, now with more interesting features for us, namely numbers. We set up a matrix
of dimensionality \( 10\times 5 \) and compute the mean value and standard deviation of each column. Similarly, we can perform mathematial operations like squaring the matrix elements and many other operations.
</p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="inner_cell">
<div class="input_area">
<div class="highlight" style="background: #f8f8f8">
<pre style="line-height: 125%;"><span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">numpy</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">np</span>
<span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">pandas</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">pd</span>
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">IPython.display</span> <span style="color: #008000; font-weight: bold">import</span> display
np<span style="color: #666666">.</span>random<span style="color: #666666">.</span>seed(<span style="color: #666666">100</span>)
<span style="color: #408080; font-style: italic"># setting up a 10 x 5 matrix</span>
rows <span style="color: #666666">=</span> <span style="color: #666666">10</span>
cols <span style="color: #666666">=</span> <span style="color: #666666">5</span>
a <span style="color: #666666">=</span> np<span style="color: #666666">.</span>random<span style="color: #666666">.</span>randn(rows,cols)
df <span style="color: #666666">=</span> pd<span style="color: #666666">.</span>DataFrame(a)
display(df)
<span style="color: #008000">print</span>(df<span style="color: #666666">.</span>mean())
<span style="color: #008000">print</span>(df<span style="color: #666666">.</span>std())
display(df<span style="color: #666666">**2</span>)
</pre>
</div>
</div>
</div>
</div>
<div class="output_wrapper">
<div class="output">
<div class="output_area">
<div class="output_subarea output_stream output_stdout output_text">
</div>
</div>
</div>
</div>
</div>
<p>Thereafter we can select specific columns only and plot final results</p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="inner_cell">
<div class="input_area">
<div class="highlight" style="background: #f8f8f8">
<pre style="line-height: 125%;">df<span style="color: #666666">.</span>columns <span style="color: #666666">=</span> [<span style="color: #BA2121">&#39;First&#39;</span>, <span style="color: #BA2121">&#39;Second&#39;</span>, <span style="color: #BA2121">&#39;Third&#39;</span>, <span style="color: #BA2121">&#39;Fourth&#39;</span>, <span style="color: #BA2121">&#39;Fifth&#39;</span>]
df<span style="color: #666666">.</span>index <span style="color: #666666">=</span> np<span style="color: #666666">.</span>arange(<span style="color: #666666">10</span>)
display(df)
<span style="color: #008000">print</span>(df[<span style="color: #BA2121">&#39;Second&#39;</span>]<span style="color: #666666">.</span>mean() )
<span style="color: #008000">print</span>(df<span style="color: #666666">.</span>info())
<span style="color: #008000">print</span>(df<span style="color: #666666">.</span>describe())
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">pylab</span> <span style="color: #008000; font-weight: bold">import</span> plt, mpl
plt<span style="color: #666666">.</span>style<span style="color: #666666">.</span>use(<span style="color: #BA2121">&#39;seaborn&#39;</span>)
mpl<span style="color: #666666">.</span>rcParams[<span style="color: #BA2121">&#39;font.family&#39;</span>] <span style="color: #666666">=</span> <span style="color: #BA2121">&#39;serif&#39;</span>
df<span style="color: #666666">.</span>cumsum()<span style="color: #666666">.</span>plot(lw<span style="color: #666666">=2.0</span>, figsize<span style="color: #666666">=</span>(<span style="color: #666666">10</span>,<span style="color: #666666">6</span>))
plt<span style="color: #666666">.</span>show()
df<span style="color: #666666">.</span>plot<span style="color: #666666">.</span>bar(figsize<span style="color: #666666">=</span>(<span style="color: #666666">10</span>,<span style="color: #666666">6</span>), rot<span style="color: #666666">=15</span>)
plt<span style="color: #666666">.</span>show()
</pre>
</div>
</div>
</div>
</div>
<div class="output_wrapper">
<div class="output">
<div class="output_area">
<div class="output_subarea output_stream output_stdout output_text">
</div>
</div>
</div>
</div>
</div>
<p>We can produce a \( 4\times 4 \) matrix</p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="inner_cell">
<div class="input_area">
<div class="highlight" style="background: #f8f8f8">
<pre style="line-height: 125%;">b <span style="color: #666666">=</span> np<span style="color: #666666">.</span>arange(<span style="color: #666666">16</span>)<span style="color: #666666">.</span>reshape((<span style="color: #666666">4</span>,<span style="color: #666666">4</span>))
<span style="color: #008000">print</span>(b)
df1 <span style="color: #666666">=</span> pd<span style="color: #666666">.</span>DataFrame(b)
<span style="color: #008000">print</span>(df1)
</pre>
</div>
</div>
</div>
</div>
<div class="output_wrapper">
<div class="output">
<div class="output_area">
<div class="output_subarea output_stream output_stdout output_text">
</div>
</div>
</div>
</div>
</div>
<p>and many other operations. </p>
<p>The <b>Series</b> class is another important class included in
<b>pandas</b>. You can view it as a specialization of <b>DataFrame</b> but where
we have just a single column of data. It shares many of the same features as <b>DataFrame</b>. As with <b>DataFrame</b>,
most operations are vectorized, achieving thereby a high performance when dealing with computations of arrays, in particular labeled arrays.
As we will see below it leads also to a very concice code close to the mathematical operations we may be interested in.
For multidimensional arrays, we recommend strongly <a href="http://xarray.pydata.org/en/stable/" target="_self">xarray</a>. <b>xarray</b> has much of the same flexibility as <b>pandas</b>, but allows for the extension to higher dimensions than two. We will see examples later of the usage of both <b>pandas</b> and <b>xarray</b>.
</p>
<p>
<!-- navigation buttons at the bottom of the page -->
<ul class="pagination">
<li><a href="._week34-bs036.html">&laquo;</a></li>
<li><a href="._week34-bs000.html">1</a></li>
<li><a href="">...</a></li>
<li><a href="._week34-bs029.html">30</a></li>
<li><a href="._week34-bs030.html">31</a></li>
<li><a href="._week34-bs031.html">32</a></li>
<li><a href="._week34-bs032.html">33</a></li>
<li><a href="._week34-bs033.html">34</a></li>
<li><a href="._week34-bs034.html">35</a></li>
<li><a href="._week34-bs035.html">36</a></li>
<li><a href="._week34-bs036.html">37</a></li>
<li class="active"><a href="._week34-bs037.html">38</a></li>
<li><a href="._week34-bs038.html">39</a></li>
<li><a href="._week34-bs039.html">40</a></li>
<li><a href="._week34-bs040.html">41</a></li>
<li><a href="._week34-bs041.html">42</a></li>
<li><a href="._week34-bs042.html">43</a></li>
<li><a href="._week34-bs043.html">44</a></li>
<li><a href="._week34-bs044.html">45</a></li>
<li><a href="._week34-bs045.html">46</a></li>
<li><a href="._week34-bs046.html">47</a></li>
<li><a href="">...</a></li>
<li><a href="._week34-bs066.html">67</a></li>
<li><a href="._week34-bs038.html">&raquo;</a></li>
</ul>
<!-- ------------------- end of main content --------------- -->
</div> <!-- end container -->
<!-- include javascript, jQuery *first* -->
<script src="https://ajax.googleapis.com/ajax/libs/jquery/1.10.2/jquery.min.js"></script>
<script src="https://netdna.bootstrapcdn.com/bootstrap/3.0.0/js/bootstrap.min.js"></script>
<!-- Bootstrap footer
<footer>
<a href="https://..."><img width="250" align=right src="https://..."></a>
</footer>
-->
<center style="font-size:80%">
<!-- copyright only on the titlepage -->
</center>
</body>
</html>
Reference in New Issue View Git Blame Copy Permalink