lars/FYS-STK4155
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
5c6c05efbe058f033f3f35def695f64a7c70b9ce
FYS-STK4155/doc/pub/Reinforce/html/._Reinforce-bs002.html
T
mhjensen 73f987a013 update on getting started
2019-08-13 23:10:37 +02:00

228 lines
17 KiB
HTML
Raw Blame History

<!--
Automatically generated HTML file from DocOnce source
(https://github.com/hplgit/doconce/)
-->
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<meta name="generator" content="DocOnce: https://github.com/hplgit/doconce/" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<meta name="description" content="Data Analysis and Machine Learning: Reinforcement Learning">
<title>Data Analysis and Machine Learning: Reinforcement Learning</title>
<!-- Bootstrap style: bootstrap -->
<link href="https://netdna.bootstrapcdn.com/bootstrap/3.1.1/css/bootstrap.min.css" rel="stylesheet">
<!-- not necessary
<link href="https://netdna.bootstrapcdn.com/font-awesome/4.0.3/css/font-awesome.css" rel="stylesheet">
-->
<style type="text/css">
/* Add scrollbar to dropdown menus in bootstrap navigation bar */
.dropdown-menu {
height: auto;
max-height: 400px;
overflow-x: hidden;
}
/* Adds an invisible element before each target to offset for the navigation
bar */
.anchor::before {
content:"";
display:block;
height:50px; /* fixed header height for style bootstrap */
margin:-50px 0 0; /* negative fixed header height */
}
</style>
</head>
<!-- tocinfo
{'highest level': 2,
'sections': [('Reinforcement Learning: Overarching view', 2, None, '___sec0'),
('Code example', 2, None, '___sec1')]}
end of tocinfo -->
<body>
<!-- Bootstrap navigation bar -->
<div class="navbar navbar-default navbar-fixed-top">
<div class="navbar-header">
<button type="button" class="navbar-toggle" data-toggle="collapse" data-target=".navbar-responsive-collapse">
<span class="icon-bar"></span>
<span class="icon-bar"></span>
<span class="icon-bar"></span>
</button>
<a class="navbar-brand" href="Reinforce-bs.html">Data Analysis and Machine Learning: Reinforcement Learning</a>
</div>
<div class="navbar-collapse collapse navbar-responsive-collapse">
<ul class="nav navbar-nav navbar-right">
<li class="dropdown">
<a href="#" class="dropdown-toggle" data-toggle="dropdown">Contents <b class="caret"></b></a>
<ul class="dropdown-menu">
<!-- navigation toc: --> <li><a href="._Reinforce-bs001.html#___sec0" style="font-size: 80%;">Reinforcement Learning: Overarching view</a></li>
<!-- navigation toc: --> <li><a href="#___sec1" style="font-size: 80%;">Code example</a></li>
</ul>
</li>
</ul>
</div>
</div>
</div> <!-- end of navigation bar -->
<div class="container">
<p>&nbsp;</p><p>&nbsp;</p><p>&nbsp;</p> <!-- add vertical space -->
<a name="part0002"></a>
<!-- !split -->
<h2 id="___sec1" class="anchor">Code example </h2>
<p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span><span style="color: #BA2121; font-style: italic">&quot;&quot;&quot;</span>
<span style="color: #BA2121; font-style: italic">A simple example for Reinforcement Learning using table lookup Q-learning method.</span>
<span style="color: #BA2121; font-style: italic">An agent &quot;o&quot; is on the left of a 1 dimensional world, the treasure is on the rightmost location.</span>
<span style="color: #BA2121; font-style: italic">Run this program and to see how the agent will improve its strategy of finding the treasure.</span>
<span style="color: #BA2121; font-style: italic">View more on my tutorial page: https://morvanzhou.github.io/tutorials/</span>
<span style="color: #BA2121; font-style: italic">&quot;&quot;&quot;</span>
<span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">numpy</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">np</span>
<span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">pandas</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">pd</span>
<span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">time</span>
np<span style="color: #666666">.</span>random<span style="color: #666666">.</span>seed(<span style="color: #666666">2</span>) <span style="color: #408080; font-style: italic"># reproducible</span>
N_STATES <span style="color: #666666">=</span> <span style="color: #666666">6</span> <span style="color: #408080; font-style: italic"># the length of the 1 dimensional world</span>
ACTIONS <span style="color: #666666">=</span> [<span style="color: #BA2121">&#39;left&#39;</span>, <span style="color: #BA2121">&#39;right&#39;</span>] <span style="color: #408080; font-style: italic"># available actions</span>
EPSILON <span style="color: #666666">=</span> <span style="color: #666666">0.9</span> <span style="color: #408080; font-style: italic"># greedy police</span>
ALPHA <span style="color: #666666">=</span> <span style="color: #666666">0.1</span> <span style="color: #408080; font-style: italic"># learning rate</span>
GAMMA <span style="color: #666666">=</span> <span style="color: #666666">0.9</span> <span style="color: #408080; font-style: italic"># discount factor</span>
MAX_EPISODES <span style="color: #666666">=</span> <span style="color: #666666">13</span> <span style="color: #408080; font-style: italic"># maximum episodes</span>
FRESH_TIME <span style="color: #666666">=</span> <span style="color: #666666">0.3</span> <span style="color: #408080; font-style: italic"># fresh time for one move</span>
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">build_q_table</span>(n_states, actions):
table <span style="color: #666666">=</span> pd<span style="color: #666666">.</span>DataFrame(
np<span style="color: #666666">.</span>zeros((n_states, <span style="color: #008000">len</span>(actions))), <span style="color: #408080; font-style: italic"># q_table initial values</span>
columns<span style="color: #666666">=</span>actions, <span style="color: #408080; font-style: italic"># actions&#39;s name</span>
)
<span style="color: #408080; font-style: italic"># print(table) # show table</span>
<span style="color: #008000; font-weight: bold">return</span> table
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">choose_action</span>(state, q_table):
<span style="color: #408080; font-style: italic"># This is how to choose an action</span>
state_actions <span style="color: #666666">=</span> q_table<span style="color: #666666">.</span>iloc[state, :]
<span style="color: #008000; font-weight: bold">if</span> (np<span style="color: #666666">.</span>random<span style="color: #666666">.</span>uniform() <span style="color: #666666">&gt;</span> EPSILON) <span style="color: #AA22FF; font-weight: bold">or</span> ((state_actions <span style="color: #666666">==</span> <span style="color: #666666">0</span>)<span style="color: #666666">.</span>all()): <span style="color: #408080; font-style: italic"># act non-greedy or state-action have no value</span>
action_name <span style="color: #666666">=</span> np<span style="color: #666666">.</span>random<span style="color: #666666">.</span>choice(ACTIONS)
<span style="color: #008000; font-weight: bold">else</span>: <span style="color: #408080; font-style: italic"># act greedy</span>
action_name <span style="color: #666666">=</span> state_actions<span style="color: #666666">.</span>idxmax() <span style="color: #408080; font-style: italic"># replace argmax to idxmax as argmax means a different function in newer version of pandas</span>
<span style="color: #008000; font-weight: bold">return</span> action_name
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">get_env_feedback</span>(S, A):
<span style="color: #408080; font-style: italic"># This is how agent will interact with the environment</span>
<span style="color: #008000; font-weight: bold">if</span> A <span style="color: #666666">==</span> <span style="color: #BA2121">&#39;right&#39;</span>: <span style="color: #408080; font-style: italic"># move right</span>
<span style="color: #008000; font-weight: bold">if</span> S <span style="color: #666666">==</span> N_STATES <span style="color: #666666">-</span> <span style="color: #666666">2</span>: <span style="color: #408080; font-style: italic"># terminate</span>
S_ <span style="color: #666666">=</span> <span style="color: #BA2121">&#39;terminal&#39;</span>
R <span style="color: #666666">=</span> <span style="color: #666666">1</span>
<span style="color: #008000; font-weight: bold">else</span>:
S_ <span style="color: #666666">=</span> S <span style="color: #666666">+</span> <span style="color: #666666">1</span>
R <span style="color: #666666">=</span> <span style="color: #666666">0</span>
<span style="color: #008000; font-weight: bold">else</span>: <span style="color: #408080; font-style: italic"># move left</span>
R <span style="color: #666666">=</span> <span style="color: #666666">0</span>
<span style="color: #008000; font-weight: bold">if</span> S <span style="color: #666666">==</span> <span style="color: #666666">0</span>:
S_ <span style="color: #666666">=</span> S <span style="color: #408080; font-style: italic"># reach the wall</span>
<span style="color: #008000; font-weight: bold">else</span>:
S_ <span style="color: #666666">=</span> S <span style="color: #666666">-</span> <span style="color: #666666">1</span>
<span style="color: #008000; font-weight: bold">return</span> S_, R
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">update_env</span>(S, episode, step_counter):
<span style="color: #408080; font-style: italic"># This is how environment be updated</span>
env_list <span style="color: #666666">=</span> [<span style="color: #BA2121">&#39;-&#39;</span>]<span style="color: #666666">*</span>(N_STATES<span style="color: #666666">-1</span>) <span style="color: #666666">+</span> [<span style="color: #BA2121">&#39;T&#39;</span>] <span style="color: #408080; font-style: italic"># &#39;---------T&#39; our environment</span>
<span style="color: #008000; font-weight: bold">if</span> S <span style="color: #666666">==</span> <span style="color: #BA2121">&#39;terminal&#39;</span>:
interaction <span style="color: #666666">=</span> <span style="color: #BA2121">&#39;Episode </span><span style="color: #BB6688; font-weight: bold">%s</span><span style="color: #BA2121">: total_steps = </span><span style="color: #BB6688; font-weight: bold">%s</span><span style="color: #BA2121">&#39;</span> <span style="color: #666666">%</span> (episode<span style="color: #666666">+1</span>, step_counter)
<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">&#39;</span><span style="color: #BB6622; font-weight: bold">\r</span><span style="color: #BA2121">{}&#39;</span><span style="color: #666666">.</span>format(interaction), end<span style="color: #666666">=</span><span style="color: #BA2121">&#39;&#39;</span>)
time<span style="color: #666666">.</span>sleep(<span style="color: #666666">2</span>)
<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">&#39;</span><span style="color: #BB6622; font-weight: bold">\r</span><span style="color: #BA2121"> &#39;</span>, end<span style="color: #666666">=</span><span style="color: #BA2121">&#39;&#39;</span>)
<span style="color: #008000; font-weight: bold">else</span>:
env_list[S] <span style="color: #666666">=</span> <span style="color: #BA2121">&#39;o&#39;</span>
interaction <span style="color: #666666">=</span> <span style="color: #BA2121">&#39;&#39;</span><span style="color: #666666">.</span>join(env_list)
<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">&#39;</span><span style="color: #BB6622; font-weight: bold">\r</span><span style="color: #BA2121">{}&#39;</span><span style="color: #666666">.</span>format(interaction), end<span style="color: #666666">=</span><span style="color: #BA2121">&#39;&#39;</span>)
time<span style="color: #666666">.</span>sleep(FRESH_TIME)
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">rl</span>():
<span style="color: #408080; font-style: italic"># main part of RL loop</span>
q_table <span style="color: #666666">=</span> build_q_table(N_STATES, ACTIONS)
<span style="color: #008000; font-weight: bold">for</span> episode <span style="color: #AA22FF; font-weight: bold">in</span> <span style="color: #008000">range</span>(MAX_EPISODES):
step_counter <span style="color: #666666">=</span> <span style="color: #666666">0</span>
S <span style="color: #666666">=</span> <span style="color: #666666">0</span>
is_terminated <span style="color: #666666">=</span> <span style="color: #008000">False</span>
update_env(S, episode, step_counter)
<span style="color: #008000; font-weight: bold">while</span> <span style="color: #AA22FF; font-weight: bold">not</span> is_terminated:
A <span style="color: #666666">=</span> choose_action(S, q_table)
S_, R <span style="color: #666666">=</span> get_env_feedback(S, A) <span style="color: #408080; font-style: italic"># take action &amp; get next state and reward</span>
q_predict <span style="color: #666666">=</span> q_table<span style="color: #666666">.</span>loc[S, A]
<span style="color: #008000; font-weight: bold">if</span> S_ <span style="color: #666666">!=</span> <span style="color: #BA2121">&#39;terminal&#39;</span>:
q_target <span style="color: #666666">=</span> R <span style="color: #666666">+</span> GAMMA <span style="color: #666666">*</span> q_table<span style="color: #666666">.</span>iloc[S_, :]<span style="color: #666666">.</span>max() <span style="color: #408080; font-style: italic"># next state is not terminal</span>
<span style="color: #008000; font-weight: bold">else</span>:
q_target <span style="color: #666666">=</span> R <span style="color: #408080; font-style: italic"># next state is terminal</span>
is_terminated <span style="color: #666666">=</span> <span style="color: #008000">True</span> <span style="color: #408080; font-style: italic"># terminate this episode</span>
q_table<span style="color: #666666">.</span>loc[S, A] <span style="color: #666666">+=</span> ALPHA <span style="color: #666666">*</span> (q_target <span style="color: #666666">-</span> q_predict) <span style="color: #408080; font-style: italic"># update</span>
S <span style="color: #666666">=</span> S_ <span style="color: #408080; font-style: italic"># move to next state</span>
update_env(S, episode, step_counter<span style="color: #666666">+1</span>)
step_counter <span style="color: #666666">+=</span> <span style="color: #666666">1</span>
<span style="color: #008000; font-weight: bold">return</span> q_table
<span style="color: #008000; font-weight: bold">if</span> <span style="color: #19177C">__name__</span> <span style="color: #666666">==</span> <span style="color: #BA2121">&quot;__main__&quot;</span>:
q_table <span style="color: #666666">=</span> rl()
<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">&#39;</span><span style="color: #BB6622; font-weight: bold">\r\n</span><span style="color: #BA2121">Q-table:</span><span style="color: #BB6622; font-weight: bold">\n</span><span style="color: #BA2121">&#39;</span>)
<span style="color: #008000; font-weight: bold">print</span>(q_table)
</pre></div>
<p>
<p>
<!-- navigation buttons at the bottom of the page -->
<ul class="pagination">
<li><a href="._Reinforce-bs001.html">&laquo;</a></li>
<li><a href="._Reinforce-bs000.html">1</a></li>
<li><a href="._Reinforce-bs001.html">2</a></li>
<li class="active"><a href="._Reinforce-bs002.html">3</a></li>
</ul>
<!-- ------------------- end of main content --------------- -->
</div> <!-- end container -->
<!-- include javascript, jQuery *first* -->
<script src="https://ajax.googleapis.com/ajax/libs/jquery/1.10.2/jquery.min.js"></script>
<script src="https://netdna.bootstrapcdn.com/bootstrap/3.0.0/js/bootstrap.min.js"></script>
<!-- Bootstrap footer
<footer>
<a href="http://..."><img width="250" align=right src="http://..."></a>
</footer>
-->
<center style="font-size:80%">
<!-- copyright only on the titlepage -->
</center>
</body>
</html>
Reference in New Issue View Git Blame Copy Permalink