Files
FYS-STK4155/doc/pub/week34/html/._week34-bs028.html
T
2020-09-16 13:58:39 +02:00

364 lines
24 KiB
HTML

<!--
Automatically generated HTML file from DocOnce source
(https://github.com/hplgit/doconce/)
-->
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<meta name="generator" content="DocOnce: https://github.com/hplgit/doconce/" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<meta name="description" content="Week 34: Introduction to the course, Logistics and Practicalities">
<title>Week 34: Introduction to the course, Logistics and Practicalities</title>
<!-- Bootstrap style: bootstrap -->
<link href="https://netdna.bootstrapcdn.com/bootstrap/3.1.1/css/bootstrap.min.css" rel="stylesheet">
<!-- not necessary
<link href="https://netdna.bootstrapcdn.com/font-awesome/4.0.3/css/font-awesome.css" rel="stylesheet">
-->
<style type="text/css">
/* Add scrollbar to dropdown menus in bootstrap navigation bar */
.dropdown-menu {
height: auto;
max-height: 400px;
overflow-x: hidden;
}
/* Adds an invisible element before each target to offset for the navigation
bar */
.anchor::before {
content:"";
display:block;
height:50px; /* fixed header height for style bootstrap */
margin:-50px 0 0; /* negative fixed header height */
}
</style>
</head>
<!-- tocinfo
{'highest level': 2,
'sections': [('Overview of first week', 2, None, '___sec0'),
('Thursday August 20', 2, None, '___sec1'),
('Lectures and ComputerLab', 2, None, '___sec2'),
('Course Format', 2, None, '___sec3'),
('Teachers', 2, None, '___sec4'),
('Deadlines for projects (tentative)', 2, None, '___sec5'),
('Recommended textbooks', 2, None, '___sec6'),
('Prerequisites', 2, None, '___sec7'),
('Learning outcomes', 2, None, '___sec8'),
('Topics covered in this course: Statistical analysis and '
'optimization of data',
2,
None,
'___sec9'),
('Topics covered in this course: Machine Learning',
2,
None,
'___sec10'),
('Extremely useful tools, strongly recommended',
2,
None,
'___sec11'),
('Other courses on Data science and Machine Learning at UiO',
2,
None,
'___sec12'),
('Introduction', 2, None, '___sec13'),
('What is Machine Learning?', 2, None, '___sec14'),
('Types of Machine Learning', 2, None, '___sec15'),
('Software and needed installations', 2, None, '___sec16'),
('Python installers', 2, None, '___sec17'),
('Useful Python libraries', 2, None, '___sec18'),
('Installing R, C++, cython or Julia', 2, None, '___sec19'),
('Installing R, C++, cython, Numba etc', 2, None, '___sec20'),
('Numpy examples and Important Matrix and vector handling '
'packages',
2,
None,
'___sec21'),
('Basic Matrix Features', 2, None, '___sec22'),
('Some famous Matrices', 3, None, '___sec23'),
('More Basic Matrix Features', 3, None, '___sec24'),
('Numpy and arrays', 2, None, '___sec25'),
('Matrices in Python', 2, None, '___sec26'),
('Meet the Pandas', 2, None, '___sec27'),
('Friday August 21', 2, None, '___sec28'),
('Reading Data and fitting', 2, None, '___sec29'),
('Friday August 21', 2, None, '___sec30'),
('Simple linear regression model using _scikit-learn_',
3,
None,
'___sec31'),
('To our real data: nuclear binding energies. Brief reminder on '
'masses and binding energies',
3,
None,
'___sec32'),
('Organizing our data', 3, None, '___sec33'),
('Seeing the wood for the trees', 3, None, '___sec34'),
('And what about using neural networks?', 3, None, '___sec35'),
('A first summary', 2, None, '___sec36')]}
end of tocinfo -->
<body>
<script type="text/x-mathjax-config">
MathJax.Hub.Config({
TeX: {
equationNumbers: { autoNumber: "none" },
extensions: ["AMSmath.js", "AMSsymbols.js", "autobold.js", "color.js"]
}
});
</script>
<script type="text/javascript" async
src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.1/MathJax.js?config=TeX-AMS-MML_HTMLorMML">
</script>
<!-- Bootstrap navigation bar -->
<div class="navbar navbar-default navbar-fixed-top">
<div class="navbar-header">
<button type="button" class="navbar-toggle" data-toggle="collapse" data-target=".navbar-responsive-collapse">
<span class="icon-bar"></span>
<span class="icon-bar"></span>
<span class="icon-bar"></span>
</button>
<a class="navbar-brand" href="week34-bs.html">Week 34: Introduction to the course, Logistics and Practicalities</a>
</div>
<div class="navbar-collapse collapse navbar-responsive-collapse">
<ul class="nav navbar-nav navbar-right">
<li class="dropdown">
<a href="#" class="dropdown-toggle" data-toggle="dropdown">Contents <b class="caret"></b></a>
<ul class="dropdown-menu">
<!-- navigation toc: --> <li><a href="._week34-bs001.html#___sec0" style="font-size: 80%;"><b>Overview of first week</b></a></li>
<!-- navigation toc: --> <li><a href="._week34-bs002.html#___sec1" style="font-size: 80%;"><b>Thursday August 20</b></a></li>
<!-- navigation toc: --> <li><a href="._week34-bs003.html#___sec2" style="font-size: 80%;"><b>Lectures and ComputerLab</b></a></li>
<!-- navigation toc: --> <li><a href="._week34-bs004.html#___sec3" style="font-size: 80%;"><b>Course Format</b></a></li>
<!-- navigation toc: --> <li><a href="._week34-bs005.html#___sec4" style="font-size: 80%;"><b>Teachers</b></a></li>
<!-- navigation toc: --> <li><a href="._week34-bs006.html#___sec5" style="font-size: 80%;"><b>Deadlines for projects (tentative)</b></a></li>
<!-- navigation toc: --> <li><a href="._week34-bs007.html#___sec6" style="font-size: 80%;"><b>Recommended textbooks</b></a></li>
<!-- navigation toc: --> <li><a href="._week34-bs008.html#___sec7" style="font-size: 80%;"><b>Prerequisites</b></a></li>
<!-- navigation toc: --> <li><a href="._week34-bs009.html#___sec8" style="font-size: 80%;"><b>Learning outcomes</b></a></li>
<!-- navigation toc: --> <li><a href="._week34-bs010.html#___sec9" style="font-size: 80%;"><b>Topics covered in this course: Statistical analysis and optimization of data</b></a></li>
<!-- navigation toc: --> <li><a href="._week34-bs011.html#___sec10" style="font-size: 80%;"><b>Topics covered in this course: Machine Learning</b></a></li>
<!-- navigation toc: --> <li><a href="._week34-bs012.html#___sec11" style="font-size: 80%;"><b>Extremely useful tools, strongly recommended</b></a></li>
<!-- navigation toc: --> <li><a href="._week34-bs013.html#___sec12" style="font-size: 80%;"><b>Other courses on Data science and Machine Learning at UiO</b></a></li>
<!-- navigation toc: --> <li><a href="._week34-bs014.html#___sec13" style="font-size: 80%;"><b>Introduction</b></a></li>
<!-- navigation toc: --> <li><a href="._week34-bs015.html#___sec14" style="font-size: 80%;"><b>What is Machine Learning?</b></a></li>
<!-- navigation toc: --> <li><a href="._week34-bs016.html#___sec15" style="font-size: 80%;"><b>Types of Machine Learning</b></a></li>
<!-- navigation toc: --> <li><a href="._week34-bs017.html#___sec16" style="font-size: 80%;"><b>Software and needed installations</b></a></li>
<!-- navigation toc: --> <li><a href="._week34-bs018.html#___sec17" style="font-size: 80%;"><b>Python installers</b></a></li>
<!-- navigation toc: --> <li><a href="._week34-bs019.html#___sec18" style="font-size: 80%;"><b>Useful Python libraries</b></a></li>
<!-- navigation toc: --> <li><a href="._week34-bs020.html#___sec19" style="font-size: 80%;"><b>Installing R, C++, cython or Julia</b></a></li>
<!-- navigation toc: --> <li><a href="._week34-bs021.html#___sec20" style="font-size: 80%;"><b>Installing R, C++, cython, Numba etc</b></a></li>
<!-- navigation toc: --> <li><a href="._week34-bs022.html#___sec21" style="font-size: 80%;"><b>Numpy examples and Important Matrix and vector handling packages</b></a></li>
<!-- navigation toc: --> <li><a href="._week34-bs023.html#___sec22" style="font-size: 80%;"><b>Basic Matrix Features</b></a></li>
<!-- navigation toc: --> <li><a href="._week34-bs024.html#___sec23" style="font-size: 80%;">&nbsp;&nbsp;&nbsp;Some famous Matrices</a></li>
<!-- navigation toc: --> <li><a href="._week34-bs025.html#___sec24" style="font-size: 80%;">&nbsp;&nbsp;&nbsp;More Basic Matrix Features</a></li>
<!-- navigation toc: --> <li><a href="._week34-bs026.html#___sec25" style="font-size: 80%;"><b>Numpy and arrays</b></a></li>
<!-- navigation toc: --> <li><a href="._week34-bs027.html#___sec26" style="font-size: 80%;"><b>Matrices in Python</b></a></li>
<!-- navigation toc: --> <li><a href="#___sec27" style="font-size: 80%;"><b>Meet the Pandas</b></a></li>
<!-- navigation toc: --> <li><a href="._week34-bs029.html#___sec28" style="font-size: 80%;"><b>Friday August 21</b></a></li>
<!-- navigation toc: --> <li><a href="._week34-bs030.html#___sec29" style="font-size: 80%;"><b>Reading Data and fitting</b></a></li>
<!-- navigation toc: --> <li><a href="._week34-bs031.html#___sec30" style="font-size: 80%;"><b>Friday August 21</b></a></li>
<!-- navigation toc: --> <li><a href="._week34-bs032.html#___sec31" style="font-size: 80%;">&nbsp;&nbsp;&nbsp;Simple linear regression model using <b>scikit-learn</b></a></li>
<!-- navigation toc: --> <li><a href="._week34-bs032.html#___sec32" style="font-size: 80%;">&nbsp;&nbsp;&nbsp;To our real data: nuclear binding energies. Brief reminder on masses and binding energies</a></li>
<!-- navigation toc: --> <li><a href="._week34-bs032.html#___sec33" style="font-size: 80%;">&nbsp;&nbsp;&nbsp;Organizing our data</a></li>
<!-- navigation toc: --> <li><a href="._week34-bs032.html#___sec34" style="font-size: 80%;">&nbsp;&nbsp;&nbsp;Seeing the wood for the trees</a></li>
<!-- navigation toc: --> <li><a href="._week34-bs032.html#___sec35" style="font-size: 80%;">&nbsp;&nbsp;&nbsp;And what about using neural networks?</a></li>
<!-- navigation toc: --> <li><a href="._week34-bs032.html#___sec36" style="font-size: 80%;"><b>A first summary</b></a></li>
</ul>
</li>
</ul>
</div>
</div>
</div> <!-- end of navigation bar -->
<div class="container">
<p>&nbsp;</p><p>&nbsp;</p><p>&nbsp;</p> <!-- add vertical space -->
<a name="part0028"></a>
<!-- !split -->
<h2 id="___sec27" class="anchor">Meet the Pandas </h2>
<p>
<br /><br /><center><p><img src="fig/pandas.jpg" align="bottom" width=600></p></center><br /><br />
<p>
Another useful Python package is
<a href="https://pandas.pydata.org/" target="_self">pandas</a>, which is an open source library
providing high-performance, easy-to-use data structures and data
analysis tools for Python. <b>pandas</b> stands for panel data, a term borrowed from econometrics and is an efficient library for data analysis with an emphasis on tabular data.
<b>pandas</b> has two major classes, the <b>DataFrame</b> class with two-dimensional data objects and tabular data organized in columns and the class <b>Series</b> with a focus on one-dimensional data objects. Both classes allow you to index data easily as we will see in the examples below.
<b>pandas</b> allows you also to perform mathematical operations on the data, spanning from simple reshapings of vectors and matrices to statistical operations.
<p>
The following simple example shows how we can, in an easy way make tables of our data. Here we define a data set which includes names, place of birth and date of birth, and displays the data in an easy to read way. We will see repeated use of <b>pandas</b>, in particular in connection with classification of data.
<p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span><span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">pandas</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">pd</span>
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">IPython.display</span> <span style="color: #008000; font-weight: bold">import</span> display
data <span style="color: #666666">=</span> {<span style="color: #BA2121">&#39;First Name&#39;</span>: [<span style="color: #BA2121">&quot;Frodo&quot;</span>, <span style="color: #BA2121">&quot;Bilbo&quot;</span>, <span style="color: #BA2121">&quot;Aragorn II&quot;</span>, <span style="color: #BA2121">&quot;Samwise&quot;</span>],
<span style="color: #BA2121">&#39;Last Name&#39;</span>: [<span style="color: #BA2121">&quot;Baggins&quot;</span>, <span style="color: #BA2121">&quot;Baggins&quot;</span>,<span style="color: #BA2121">&quot;Elessar&quot;</span>,<span style="color: #BA2121">&quot;Gamgee&quot;</span>],
<span style="color: #BA2121">&#39;Place of birth&#39;</span>: [<span style="color: #BA2121">&quot;Shire&quot;</span>, <span style="color: #BA2121">&quot;Shire&quot;</span>, <span style="color: #BA2121">&quot;Eriador&quot;</span>, <span style="color: #BA2121">&quot;Shire&quot;</span>],
<span style="color: #BA2121">&#39;Date of Birth T.A.&#39;</span>: [<span style="color: #666666">2968</span>, <span style="color: #666666">2890</span>, <span style="color: #666666">2931</span>, <span style="color: #666666">2980</span>]
}
data_pandas <span style="color: #666666">=</span> pd<span style="color: #666666">.</span>DataFrame(data)
display(data_pandas)
</pre></div>
<p>
In the above we have imported <b>pandas</b> with the shorthand <b>pd</b>, the latter has become the standard way we import <b>pandas</b>. We make then a list of various variables
and reorganize the aboves lists into a <b>DataFrame</b> and then print out a neat table with specific column labels as <em>Name</em>, <em>place of birth</em> and <em>date of birth</em>.
Displaying these results, we see that the indices are given by the default numbers from zero to three.
<b>pandas</b> is extremely flexible and we can easily change the above indices by defining a new type of indexing as
<p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span>data_pandas <span style="color: #666666">=</span> pd<span style="color: #666666">.</span>DataFrame(data,index<span style="color: #666666">=</span>[<span style="color: #BA2121">&#39;Frodo&#39;</span>,<span style="color: #BA2121">&#39;Bilbo&#39;</span>,<span style="color: #BA2121">&#39;Aragorn&#39;</span>,<span style="color: #BA2121">&#39;Sam&#39;</span>])
display(data_pandas)
</pre></div>
<p>
Thereafter we display the content of the row which begins with the index <b>Aragorn</b>
<p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span>display(data_pandas<span style="color: #666666">.</span>loc[<span style="color: #BA2121">&#39;Aragorn&#39;</span>])
</pre></div>
<p>
We can easily append data to this, for example
<p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span>new_hobbit <span style="color: #666666">=</span> {<span style="color: #BA2121">&#39;First Name&#39;</span>: [<span style="color: #BA2121">&quot;Peregrin&quot;</span>],
<span style="color: #BA2121">&#39;Last Name&#39;</span>: [<span style="color: #BA2121">&quot;Took&quot;</span>],
<span style="color: #BA2121">&#39;Place of birth&#39;</span>: [<span style="color: #BA2121">&quot;Shire&quot;</span>],
<span style="color: #BA2121">&#39;Date of Birth T.A.&#39;</span>: [<span style="color: #666666">2990</span>]
}
data_pandas<span style="color: #666666">=</span>data_pandas<span style="color: #666666">.</span>append(pd<span style="color: #666666">.</span>DataFrame(new_hobbit, index<span style="color: #666666">=</span>[<span style="color: #BA2121">&#39;Pippin&#39;</span>]))
display(data_pandas)
</pre></div>
<p>
Here are other examples where we use the <b>DataFrame</b> functionality to handle arrays, now with more interesting features for us, namely numbers. We set up a matrix
of dimensionality \( 10\times 5 \) and compute the mean value and standard deviation of each column. Similarly, we can perform mathematial operations like squaring the matrix elements and many other operations.
<p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span><span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">numpy</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">np</span>
<span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">pandas</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">pd</span>
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">IPython.display</span> <span style="color: #008000; font-weight: bold">import</span> display
np<span style="color: #666666">.</span>random<span style="color: #666666">.</span>seed(<span style="color: #666666">100</span>)
<span style="color: #408080; font-style: italic"># setting up a 10 x 5 matrix</span>
rows <span style="color: #666666">=</span> <span style="color: #666666">10</span>
cols <span style="color: #666666">=</span> <span style="color: #666666">5</span>
a <span style="color: #666666">=</span> np<span style="color: #666666">.</span>random<span style="color: #666666">.</span>randn(rows,cols)
df <span style="color: #666666">=</span> pd<span style="color: #666666">.</span>DataFrame(a)
display(df)
<span style="color: #008000; font-weight: bold">print</span>(df<span style="color: #666666">.</span>mean())
<span style="color: #008000; font-weight: bold">print</span>(df<span style="color: #666666">.</span>std())
display(df<span style="color: #666666">**2</span>)
</pre></div>
<p>
Thereafter we can select specific columns only and plot final results
<p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span>df<span style="color: #666666">.</span>columns <span style="color: #666666">=</span> [<span style="color: #BA2121">&#39;First&#39;</span>, <span style="color: #BA2121">&#39;Second&#39;</span>, <span style="color: #BA2121">&#39;Third&#39;</span>, <span style="color: #BA2121">&#39;Fourth&#39;</span>, <span style="color: #BA2121">&#39;Fifth&#39;</span>]
df<span style="color: #666666">.</span>index <span style="color: #666666">=</span> np<span style="color: #666666">.</span>arange(<span style="color: #666666">10</span>)
display(df)
<span style="color: #008000; font-weight: bold">print</span>(df[<span style="color: #BA2121">&#39;Second&#39;</span>]<span style="color: #666666">.</span>mean() )
<span style="color: #008000; font-weight: bold">print</span>(df<span style="color: #666666">.</span>info())
<span style="color: #008000; font-weight: bold">print</span>(df<span style="color: #666666">.</span>describe())
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">pylab</span> <span style="color: #008000; font-weight: bold">import</span> plt, mpl
plt<span style="color: #666666">.</span>style<span style="color: #666666">.</span>use(<span style="color: #BA2121">&#39;seaborn&#39;</span>)
mpl<span style="color: #666666">.</span>rcParams[<span style="color: #BA2121">&#39;font.family&#39;</span>] <span style="color: #666666">=</span> <span style="color: #BA2121">&#39;serif&#39;</span>
df<span style="color: #666666">.</span>cumsum()<span style="color: #666666">.</span>plot(lw<span style="color: #666666">=2.0</span>, figsize<span style="color: #666666">=</span>(<span style="color: #666666">10</span>,<span style="color: #666666">6</span>))
plt<span style="color: #666666">.</span>show()
df<span style="color: #666666">.</span>plot<span style="color: #666666">.</span>bar(figsize<span style="color: #666666">=</span>(<span style="color: #666666">10</span>,<span style="color: #666666">6</span>), rot<span style="color: #666666">=15</span>)
plt<span style="color: #666666">.</span>show()
</pre></div>
<p>
We can produce a \( 4\times 4 \) matrix
<p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span>b <span style="color: #666666">=</span> np<span style="color: #666666">.</span>arange(<span style="color: #666666">16</span>)<span style="color: #666666">.</span>reshape((<span style="color: #666666">4</span>,<span style="color: #666666">4</span>))
<span style="color: #008000; font-weight: bold">print</span>(b)
df1 <span style="color: #666666">=</span> pd<span style="color: #666666">.</span>DataFrame(b)
<span style="color: #008000; font-weight: bold">print</span>(df1)
</pre></div>
<p>
and many other operations.
<p>
The <b>Series</b> class is another important class included in
<b>pandas</b>. You can view it as a specialization of <b>DataFrame</b> but where
we have just a single column of data. It shares many of the same features as _DataFrame. As with <b>DataFrame</b>,
most operations are vectorized, achieving thereby a high performance when dealing with computations of arrays, in particular labeled arrays.
As we will see below it leads also to a very concice code close to the mathematical operations we may be interested in.
For multidimensional arrays, we recommend strongly <a href="http://xarray.pydata.org/en/stable/" target="_self">xarray</a>. <b>xarray</b> has much of the same flexibility as <b>pandas</b>, but allows for the extension to higher dimensions than two. We will see examples later of the usage of both <b>pandas</b> and <b>xarray</b>.
<p>
<p>
<!-- navigation buttons at the bottom of the page -->
<ul class="pagination">
<li><a href="._week34-bs027.html">&laquo;</a></li>
<li><a href="._week34-bs000.html">1</a></li>
<li><a href="">...</a></li>
<li><a href="._week34-bs020.html">21</a></li>
<li><a href="._week34-bs021.html">22</a></li>
<li><a href="._week34-bs022.html">23</a></li>
<li><a href="._week34-bs023.html">24</a></li>
<li><a href="._week34-bs024.html">25</a></li>
<li><a href="._week34-bs025.html">26</a></li>
<li><a href="._week34-bs026.html">27</a></li>
<li><a href="._week34-bs027.html">28</a></li>
<li class="active"><a href="._week34-bs028.html">29</a></li>
<li><a href="._week34-bs029.html">30</a></li>
<li><a href="._week34-bs030.html">31</a></li>
<li><a href="._week34-bs031.html">32</a></li>
<li><a href="._week34-bs032.html">33</a></li>
<li><a href="._week34-bs029.html">&raquo;</a></li>
</ul>
<!-- ------------------- end of main content --------------- -->
</div> <!-- end container -->
<!-- include javascript, jQuery *first* -->
<script src="https://ajax.googleapis.com/ajax/libs/jquery/1.10.2/jquery.min.js"></script>
<script src="https://netdna.bootstrapcdn.com/bootstrap/3.0.0/js/bootstrap.min.js"></script>
<!-- Bootstrap footer
<footer>
<a href="http://..."><img width="250" align=right src="http://..."></a>
</footer>
-->
<center style="font-size:80%">
<!-- copyright only on the titlepage -->
</center>
</body>
</html>