lars/FYS-STK4155
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
270b0beff9b052d3cf776769c8f3eb18c05c6ea9
FYS-STK4155/doc/pub/Reinforce/html/Reinforce-solarized.html
T
mhjensen ec1d118cde added code
2018-12-26 09:53:15 +01:00

204 lines
12 KiB
HTML
Raw Blame History

<!--
Automatically generated HTML file from DocOnce source
(https://github.com/hplgit/doconce/)
-->
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<meta name="generator" content="DocOnce: https://github.com/hplgit/doconce/" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<meta name="description" content="Data Analysis and Machine Learning: Reinforcement Learning">
<title>Data Analysis and Machine Learning: Reinforcement Learning</title>
<link href="https://cdn.rawgit.com/hplgit/doconce/master/bundled/html_styles/style_solarized_box/css/solarized_light_code.css" rel="stylesheet" type="text/css" title="light"/>
<script src="https://cdn.rawgit.com/hplgit/doconce/master/bundled/html_styles/style_solarized_box/js/highlight.pack.js"></script>
<script>hljs.initHighlightingOnLoad();</script>
<link href="https://thomasf.github.io/solarized-css/solarized-light.min.css" rel="stylesheet">
<style type="text/css">
h1 {color: #b58900;} /* yellow */
/* h1 {color: #cb4b16;} orange */
/* h1 {color: #d33682;} magenta, the original choice of thomasf */
code { padding: 0px; background-color: inherit; }
pre {
border: 0pt solid #93a1a1;
box-shadow: none;
}
div { text-align: justify; text-justify: inter-word; }
</style>
</head>
<!-- tocinfo
{'highest level': 2,
'sections': [('Reinforcement Learning: Overarching view', 2, None, '___sec0'),
('Code example', 2, None, '___sec1')]}
end of tocinfo -->
<body>
<!-- ------------------- main content ---------------------- -->
<center><h1>Data Analysis and Machine Learning: Reinforcement Learning </h1></center> <!-- document title -->
<p>
<!-- author(s): Morten Hjorth-Jensen -->
<center>
<b>Morten Hjorth-Jensen</b> [1, 2]
</center>
<p>
<!-- institution(s) -->
<center>[1] <b>Department of Physics, University of Oslo</b></center>
<center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
<br>
<p>
<center><h4>Dec 26, 2018</h4></center> <!-- date -->
<br>
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec0">Reinforcement Learning: Overarching view </h2>
<p>
Reinforcement Learning (RL) is one of the most exciting fields of Machine Learning today, and also one
of the oldest. It has been around since the 1950s, producing many interesting applications over the years.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec1">Code example </h2>
<p>
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
<div class="highlight" style="background: #eeeedd"><pre style="line-height: 125%"><span></span><span style="color: #CD5555">&quot;&quot;&quot;</span>
<span style="color: #CD5555">A simple example for Reinforcement Learning using table lookup Q-learning method.</span>
<span style="color: #CD5555">An agent &quot;o&quot; is on the left of a 1 dimensional world, the treasure is on the rightmost location.</span>
<span style="color: #CD5555">Run this program and to see how the agent will improve its strategy of finding the treasure.</span>
<span style="color: #CD5555">View more on my tutorial page: https://morvanzhou.github.io/tutorials/</span>
<span style="color: #CD5555">&quot;&quot;&quot;</span>
<span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">numpy</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">np</span>
<span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">pandas</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">pd</span>
<span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">time</span>
np.random.seed(<span style="color: #B452CD">2</span>) <span style="color: #228B22"># reproducible</span>
N_STATES = <span style="color: #B452CD">6</span> <span style="color: #228B22"># the length of the 1 dimensional world</span>
ACTIONS = [<span style="color: #CD5555">&#39;left&#39;</span>, <span style="color: #CD5555">&#39;right&#39;</span>] <span style="color: #228B22"># available actions</span>
EPSILON = <span style="color: #B452CD">0.9</span> <span style="color: #228B22"># greedy police</span>
ALPHA = <span style="color: #B452CD">0.1</span> <span style="color: #228B22"># learning rate</span>
GAMMA = <span style="color: #B452CD">0.9</span> <span style="color: #228B22"># discount factor</span>
MAX_EPISODES = <span style="color: #B452CD">13</span> <span style="color: #228B22"># maximum episodes</span>
FRESH_TIME = <span style="color: #B452CD">0.3</span> <span style="color: #228B22"># fresh time for one move</span>
<span style="color: #8B008B; font-weight: bold">def</span> <span style="color: #008b45">build_q_table</span>(n_states, actions):
table = pd.DataFrame(
np.zeros((n_states, <span style="color: #658b00">len</span>(actions))), <span style="color: #228B22"># q_table initial values</span>
columns=actions, <span style="color: #228B22"># actions&#39;s name</span>
)
<span style="color: #228B22"># print(table) # show table</span>
<span style="color: #8B008B; font-weight: bold">return</span> table
<span style="color: #8B008B; font-weight: bold">def</span> <span style="color: #008b45">choose_action</span>(state, q_table):
<span style="color: #228B22"># This is how to choose an action</span>
state_actions = q_table.iloc[state, :]
<span style="color: #8B008B; font-weight: bold">if</span> (np.random.uniform() &gt; EPSILON) <span style="color: #8B008B">or</span> ((state_actions == <span style="color: #B452CD">0</span>).all()): <span style="color: #228B22"># act non-greedy or state-action have no value</span>
action_name = np.random.choice(ACTIONS)
<span style="color: #8B008B; font-weight: bold">else</span>: <span style="color: #228B22"># act greedy</span>
action_name = state_actions.idxmax() <span style="color: #228B22"># replace argmax to idxmax as argmax means a different function in newer version of pandas</span>
<span style="color: #8B008B; font-weight: bold">return</span> action_name
<span style="color: #8B008B; font-weight: bold">def</span> <span style="color: #008b45">get_env_feedback</span>(S, A):
<span style="color: #228B22"># This is how agent will interact with the environment</span>
<span style="color: #8B008B; font-weight: bold">if</span> A == <span style="color: #CD5555">&#39;right&#39;</span>: <span style="color: #228B22"># move right</span>
<span style="color: #8B008B; font-weight: bold">if</span> S == N_STATES - <span style="color: #B452CD">2</span>: <span style="color: #228B22"># terminate</span>
S_ = <span style="color: #CD5555">&#39;terminal&#39;</span>
R = <span style="color: #B452CD">1</span>
<span style="color: #8B008B; font-weight: bold">else</span>:
S_ = S + <span style="color: #B452CD">1</span>
R = <span style="color: #B452CD">0</span>
<span style="color: #8B008B; font-weight: bold">else</span>: <span style="color: #228B22"># move left</span>
R = <span style="color: #B452CD">0</span>
<span style="color: #8B008B; font-weight: bold">if</span> S == <span style="color: #B452CD">0</span>:
S_ = S <span style="color: #228B22"># reach the wall</span>
<span style="color: #8B008B; font-weight: bold">else</span>:
S_ = S - <span style="color: #B452CD">1</span>
<span style="color: #8B008B; font-weight: bold">return</span> S_, R
<span style="color: #8B008B; font-weight: bold">def</span> <span style="color: #008b45">update_env</span>(S, episode, step_counter):
<span style="color: #228B22"># This is how environment be updated</span>
env_list = [<span style="color: #CD5555">&#39;-&#39;</span>]*(N_STATES-<span style="color: #B452CD">1</span>) + [<span style="color: #CD5555">&#39;T&#39;</span>] <span style="color: #228B22"># &#39;---------T&#39; our environment</span>
<span style="color: #8B008B; font-weight: bold">if</span> S == <span style="color: #CD5555">&#39;terminal&#39;</span>:
interaction = <span style="color: #CD5555">&#39;Episode %s: total_steps = %s&#39;</span> % (episode+<span style="color: #B452CD">1</span>, step_counter)
<span style="color: #8B008B; font-weight: bold">print</span>(<span style="color: #CD5555">&#39;\r{}&#39;</span>.format(interaction), end=<span style="color: #CD5555">&#39;&#39;</span>)
time.sleep(<span style="color: #B452CD">2</span>)
<span style="color: #8B008B; font-weight: bold">print</span>(<span style="color: #CD5555">&#39;\r &#39;</span>, end=<span style="color: #CD5555">&#39;&#39;</span>)
<span style="color: #8B008B; font-weight: bold">else</span>:
env_list[S] = <span style="color: #CD5555">&#39;o&#39;</span>
interaction = <span style="color: #CD5555">&#39;&#39;</span>.join(env_list)
<span style="color: #8B008B; font-weight: bold">print</span>(<span style="color: #CD5555">&#39;\r{}&#39;</span>.format(interaction), end=<span style="color: #CD5555">&#39;&#39;</span>)
time.sleep(FRESH_TIME)
<span style="color: #8B008B; font-weight: bold">def</span> <span style="color: #008b45">rl</span>():
<span style="color: #228B22"># main part of RL loop</span>
q_table = build_q_table(N_STATES, ACTIONS)
<span style="color: #8B008B; font-weight: bold">for</span> episode <span style="color: #8B008B">in</span> <span style="color: #658b00">range</span>(MAX_EPISODES):
step_counter = <span style="color: #B452CD">0</span>
S = <span style="color: #B452CD">0</span>
is_terminated = <span style="color: #658b00">False</span>
update_env(S, episode, step_counter)
<span style="color: #8B008B; font-weight: bold">while</span> <span style="color: #8B008B">not</span> is_terminated:
A = choose_action(S, q_table)
S_, R = get_env_feedback(S, A) <span style="color: #228B22"># take action &amp; get next state and reward</span>
q_predict = q_table.loc[S, A]
<span style="color: #8B008B; font-weight: bold">if</span> S_ != <span style="color: #CD5555">&#39;terminal&#39;</span>:
q_target = R + GAMMA * q_table.iloc[S_, :].max() <span style="color: #228B22"># next state is not terminal</span>
<span style="color: #8B008B; font-weight: bold">else</span>:
q_target = R <span style="color: #228B22"># next state is terminal</span>
is_terminated = <span style="color: #658b00">True</span> <span style="color: #228B22"># terminate this episode</span>
q_table.loc[S, A] += ALPHA * (q_target - q_predict) <span style="color: #228B22"># update</span>
S = S_ <span style="color: #228B22"># move to next state</span>
update_env(S, episode, step_counter+<span style="color: #B452CD">1</span>)
step_counter += <span style="color: #B452CD">1</span>
<span style="color: #8B008B; font-weight: bold">return</span> q_table
<span style="color: #8B008B; font-weight: bold">if</span> <span style="color: #00688B">__name__</span> == <span style="color: #CD5555">&quot;__main__&quot;</span>:
q_table = rl()
<span style="color: #8B008B; font-weight: bold">print</span>(<span style="color: #CD5555">&#39;\r\nQ-table:\n&#39;</span>)
<span style="color: #8B008B; font-weight: bold">print</span>(q_table)
</pre></div>
<p>
<!-- ------------------- end of main content --------------- -->
<center style="font-size:80%">
<!-- copyright --> &copy; 1999-2018, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license
</center>
</body>
</html>
Reference in New Issue View Git Blame Copy Permalink